What can cause an increase in inactive memory and how to reclaim it?

Question

I have heavy application running on a CentOS server and I'm seeing a strange memory behavior. Here is a snapshot of a munin graph:

As you can see the amount of committed memory increases gradually causing the swap file to be use. What strikes me odd is that the amount of inactive memory keeps growing as well. It is my understanding that the inactive memory is actually memory freed up but not yet clean by the OS and put back in the free memory pool. It seems that running out of memory is acutally caused by this lack of clean up, but I may be wrong.

Can you give some tips to find the cause of the problem and/or cause CentOS to reclaim the inactive memory?

Thanks.

Some extra info:

1) I have a tmpfs mounted on /tmp and the number of files stored there grows (but it is double the amount of the inactive memory).

2) cat /proc/meminfo (at a later stage than the image) gives:

MemTotal:     14371428 kB
MemFree:       1207108 kB
Buffers:         35440 kB
Cached:        4276628 kB
SwapCached:     785316 kB
Active:        9038924 kB
Inactive:      3902876 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     14371428 kB
LowFree:       1207108 kB
SwapTotal:    10223608 kB
SwapFree:      6438320 kB
Dirty:          627792 kB
Writeback:           0 kB
AnonPages:     7844560 kB
Mapped:          49304 kB
Slab:           146676 kB
PageTables:      27480 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  17409320 kB
Committed_AS: 16471488 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    275852 kB
VmallocChunk: 34359462007 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

3) The application is a combination of MySQL, Heritrix (http://crawler.archive.org/ ) and a Tomcat based Java servlet to manage things.

Solution

It is my understanding that the inactive memory is actually memory freed up but not yet clean by the OS and put back in the free memory pool.

This is false. 'inactive' memory is actively mapped memory which has not been utilized by any application for some time. When its time to swap, memory is taken from pages marked like this and swapped out. It can also be used to swap out in favour of page cache.

As you can see the amount of committed memory increases gradually causing the swap file to be use. What strikes me odd is that the amount of inactive memory keeps growing as well.

The two dont necessarily correlate, but to me this strongly looks as if something is leaking memory. The fact that you have pages not being accessed by any applications growing, and swap growing too suggests something is allocating memory, forgetting about it then not freeing it afterwards.

Memory could be 'inactive', for example if malloc() is called. This is a libc call that may allocate a chunk of memory, but only a portion of it is actually utilized to do any work (less than the number of pages allocated anyway). Even if you free in malloc it doesnt actually mean you free the memory by asking the operating system to do so, its just mallocs tables might mark is as 'reusable', it might free it after.

Comments

[Kyle Smith]

I think that graph has safely entered the realm of 'so detailed it's not helpful.' :(