Lightweight memory leak debugging on linux
Solution 1
Surprisingly, I was unable to find anything like Microsoft's UMDH in open-source domain or available for immediate download. (I also looked at Google Heap Leak Checker but it is more like Valgrind rather than to UMDH). So I ended up writing the tool myself using malloc instrumentation project as a reference point:
https://github.com/glagolig/heapwatch
The tool has number of limitations, but it worked just fine for my purposes.
Solution 2
GNU libc has built-in malloc debugging:
http://www.gnu.org/software/libc/manual/html_node/Allocation-Debugging.html
Use LD_PRELOAD to call mtrace()
from your own .so:
#include <mcheck.h>
static void prepare(void) __attribute__((constructor));
static void prepare(void)
{
mtrace();
}
Compile it with:
gcc -shared -fPIC dbg.c -o dbg.so
Run it with:
export MALLOC_TRACE=out.txt
LD_PRELOAD=./dbg.so ./my-leaky-program
Later inspect the output file:
mtrace ./my-leaky-program out.txt
And you will get something like:
Memory not freed:
-----------------
Address Size Caller
0x0000000001bda460 0x96 at /tmp/test/src/test.c:7
Of course, feel free to write your own malloc hooks that dump the entire stack (calling backtrace() if you think that's going to help).
Lines numbers and/or function names will be obtainable if you kept debug info for the binary somewhere (e.g. the binary has some debug info built in, or you did objcopy --only-keep-debug my-leaky-program my-leaky-program.debug
).
Also, you could try Boehm's GC, it works as a leak detector too:
http://www.hpl.hp.com/personal/Hans_Boehm/gc/leak.html
Solution 3
memleax should work for you.
It debugs memory leak of a running process by attaching it, without recompiling program or restarting target process. It's very convenient and suitable for production environment.
It TRAPs only for malloc/free() calls, so it should bring less performace impact than Vagrild.
It works on GNU/Linux-x86_64 and FreeBSD-amd64.
NOTE: I'm the author, any suggestion is welcomed
Solution 4
I would like to advertise my just announced heaptrack utility, which should be just what you where looking for back then. You can find more information here: http://milianw.de/blog/heaptrack-a-heap-memory-profiler-for-linux
Compared to your heapwatch tool, the performance should be far better, as I use libunwind and later libbacktrace to delay the annotation of the backtrace with DWARF debug information.
I'd love to get more feedback on it, so try it out!
Solution 5
MemoryScape would meet your needs. This is the dynamic memory debugging tool that comes with the TotalView debugger.
Related videos on Youtube
glagolig
Updated on July 31, 2020Comments
-
glagolig almost 4 years
I looked for existing answers first and saw that Valgrind is everyone’s favorite tool for memory leak debugging on linux. Unfortunately Valgrind does not seem to work for my purposes. I will try to explain why.
Constraints:
- The leak reproduces only in customer’s environment. Due to certain legal restrictions we have to work with existing binary. No rebuilds.
- In regular environment our application consumes ~10% CPU. Say, we can tolerate up to 10x CPU usage increase. Valgrind with default memcheck settings does much worse making our application unresponsive for long periods of time.
What I need is an equivalent of Microsoft’s UMDH: turn on stack tracing for each heap allocation, then at certain point of time dump all allocations grouped by stacks and ordered by allocation count in descending order. Our app ships on both Windows and Linux platforms, so I know that performance on Windows under UMDH is still tolerable.
Here are the tools/methods I considered
-
Valgrind's -memcheck and –massif tools They do much more than needed (like scanning whole process memory for every allocation
pointer), they are too slow, and they still don’t do exactly what I
need (dump callstacks sorted by counts), so I will have to write some scripts parsing the output - dmalloc library (dmalloc.com) requires new binary
- LeakTracer (http://www.andreasen.org/LeakTracer/) Works only with C++ new/delete (I need malloc/free as well), does not have group-by-stack and sort functionality
- Implementing the tool myself as .so library using LD_PRELOAD mechanism (Overriding 'malloc' using the LD_PRELOAD mechanism) That will take at least a week given my coding-for-Linux skills and it feels like inventing a bicycle.
Did I miss anything? Are there any lightweight Valgrind options or existing LD_PRELOAD tool?
-
WhozCraig over 10 yearsYou know your code better than anyone. Can you get a process dump and look at the heap to find out what is slowly chewing memory off the cob ? Start with common structure allocations that are likely in the customer's environment.
-
glagolig over 10 yearsI can get dumps, but there are too many allocations from normal execution flow in addition to the ones contributed by the leak. Some automatic tool is still required. In any case stacks would be much more informative than structures alone.
-
pcworld about 2 yearsThis appears fitting: github.com/jrfonseca/memtrail "A LD_PRELOAD based memory profiler and leak detector for Linux "
-
glagolig over 10 yearsThanks. This is a valid approach. But there are some problems though. Disk space: say, with 10^4 allocations per sec and 50 bytes per record we will have 5G file in about 3 hours. The leak becomes visible after several days. Performance: hopefully file writes are asynchronous, but I think they are still more expensive than storing stack trace along with allocation. I have to do some perf tests first. Coding: I still have to implement group-by-stack-trace and order-by-count functionality.
-
glagolig over 10 yearsYes, it looks like the tool does the job. I will play with the trial version and mark this as an answer if the tool really does by-stack grouping. The software is not free though; getting purchase approved is no faster/easier than writing my own code...
-
glagolig over 10 yearsOops... Trial version not available for automatic download (they evaluate the request for trial version)
-
DanielKO over 10 yearsThen I would keep an fixed-size hash table around; insert a record for every
malloc()
, delete it for everyfree()
, and only output to the file if a collision happened. With the appropriate size it should be enough to discard short-lived allocations from appearing in the output file. Writing to the file can be done in a dedicated thread, with the help of a concurrent queue. -
glagolig over 10 yearsThanks. I will send you an e-mail.
-
undercat about 5 yearsSadly in many modern projects that use
xmalloc
and othermalloc
wrappersmtrace
becomes useless as it will point to that generic wrapper rather than the function calling it. Luckily, there are other options described in the other answers that obviate that issue by means of stack unwinding etc. To put it short:mtrace
is—alas!—obsolete for most practical purposes. -
mpr about 5 yearsI enjoy being able to use this tool without any recompilation. When interpreting the results though, it doesn't have any concept of "old" buffers, correct? If I see
alloc=200295
andfree=200293
, those two missing free() calls could be right around the corner right? -
Bingzheng Wu about 5 yearsNot exactly.
may-leak
is better. Assume your program alloc() 1 block first, and would free() it 3 second later. But you terminate memleax just after 1 second, before the free(). So this is not a leak, but just no time to free. In this case,callc=1
,free=0
,may-leak=0
. -
Bingzheng Wu about 5 yearsBTW, there is a bug in
memleax
to debugging multi-thread program. While I write a new tool libleak, which is better and without recompliation too. -
Jan Hudec about 3 years@undercat, it's enough if the project is in C++ where
operator new
is that generic wrapper. -
Pa_ over 2 yearsI have tried it, together with the companion analysis tool heaptrack_gui, for a memory tracing session of 15 minutes. when opening the trace with my laptop, it crashed the machine OOM. So i retried on another machine with 64gb(!) of ram. On this one it managed to run with 28gb of memory used. Unfortunately, after a bit of leaving it alone (although it kept at 100% cpu nevertheless), it almost froze the machine as well, which i managed to salvage via a remote shell i had open there so i could kill the tool.
-
milianw over 2 years@Pa_ please report bugs on bugs.kde.org/enter_bug.cgi?product=Heaptrack and ideally attach or upload the heaptrack data file, such that I can reproduce the analysis bug locally and fix it. Reporting bugs should not be done in a forum like SO.
-
Pa_ over 2 yearsWell, so feel free to forward it to kde.org. I can't report since i don't know if this is a bug or it's by design, and so i merely reported my experience. Perhaps the author expects user to have 1TB of ram for using this tool
-
milianw over 2 years@Pa_ I am the author, and no I don't expect you to have so much memory. Please report bugs properly, not on a forum.