How can I find a memory leak of a running process?

Question

Is there a way, I can find the memory leak of a running process? I can use Valgrind for finding memory leaks before the start of a process. I can use GDB to attach it to a running process. How could I debug a memory leaks of a running process?

Solution 1

Here are the steps that almost guarantee to find what is leaking memory:

1. Find out the PID of the process which causing memory leak.

   ps -aux
   

2. capture the /proc/PID/smaps and save into some file like BeforeMemInc.txt.

3. wait till memory gets increased.
4. capture again /proc/PID/smaps and save it has afterMemInc.txt

5. find the difference between first smaps and 2nd smaps, e. g. with

   diff -u beforeMemInc.txt afterMemInc.txt

6. note down the address range where memory got increased, for example:

      beforeMemInc.txt            afterMemInc.txt
   ---------------------------------------------------
   2b3289290000-2b3289343000   2b3289290000-2b3289343000  #ADDRESS
   Shared_Clean:    0 kB       Shared_Clean:    0 kB          
   Shared_Dirty:    0 kB       Shared_Dirty:    0 kB
   Private_Clean:   0 kB       Private_Clean:   0 kB
   Private_Dirty:  28 kB       Private_Dirty:  36 kB  
   Referenced:     28 kB       Referenced:     36 kB
   Anonymous:      28 kB       Anonymous:      36 kB  #INCREASE MEM
   AnonHugePages:   0 kB       AnonHugePages:   0 kB
   Swap:            0 kB       Swap:            0 kB
   KernelPageSize:  4 kB       KernelPageSize:  4 kB
   MMUPageSize:     4 kB       MMUPageSize:     4 kB
   Locked:          0 kB       Locked:          0 kB
   VmFlags: rd wr mr mw me ac  VmFlags: rd wr mr mw me ac
   

7. use GDB to dump memory on running process or get the coredump using gcore -o process

8. I used gdb on running process to dump the memory to some file.

   gdb -p PID
   dump memory ./dump_outputfile.dump 0x2b3289290000 0x2b3289343000
   

9. now, use strings command or hexdump -C to print the dump_outputfile.dump

   strings outputfile.dump
   

10. You get readable form where you can locate those strings into your source code.

11. Analyze your source to find the leak.

Solution 2

I think memleax is exact what you want.

It debugs memory leak of a running process by attaching it, without recompiling program or restarting target process. It's very convenient and suitable for production environment.

It works on GNU/Linux and FreeBSD.

NOTE: I'm the author, any suggestion is welcome.

== EDIT ==

I also wrote another tool libleak, which hooks memory functions by LD_PRELOAD.

There is no need to modify the target program. Although you have to restart the progress with LD_PRELOAD, you can enable/disable the detection during running.

There is much less impact on performance since no signal trap.

Compared with similar tools (such as mtrace), it print the full call-stack at suspicious memory leak point.

Solution 3

On Linux, you could enable mtrace in your program, but it is a code change.

On OpenBSD, you could try the malloc stats.

Google's leak checker might also be worth a look, and unlike mtrace you may be able to use LD_PRELOAD to avoid recompilation.

Comments

[user400344]

Valgrind is very useful, I would even call it intuitive.

[stiv]

strings command doesn't give me any hint where to look at my source files. for me i just see some garbage strings.