Very high memory usage, but not claimed by any process?

Question

While stress-testing LVM on one of our Debian servers, I came across this issue where memory would fill up a lot to the point where it would run the server out of memory, but no process would claim the memory. See http://i.imgur.com/cLn5ZHS.png, and see https://serverfault.com/a/449102/125894 for an explanation on the colors used in htop.

Why is this happening?
And is there any way to see what process is using the memory?
Htop is configured not to hide any processes, so what is it that htop is missing?

In this particular case, I can fairly certainly say that it is caused, directly or indirectly, by lvmcreate, lvmremove or dmsetup, as I was stress-testing that. Do note that this question is not about solving the LVM problem, but about why the memory isn't claimed by any process. Stopping all LVM commands does bring the memory back down to <600MB.

Output of free -m:

             total       used       free     shared    buffers     cached
Mem:         32153      31958        194          0         52       3830
-/+ buffers/cache:      28075       4077
Swap:          975          0        975

Output of top, on pastebin due to the length of it: http://pastebin.com/WchrpF7W

Solution 1

Since you seem to be running I/O related tests (you didn't explain with details what you did with the LVM volumes), is there any chance that the behavior you noticed is related to the natural OS page cache?

I'm thinking about the Linux standard behavior of using (sometimes almost all) available memory (that is, memory that is not reserved by applications) to cache disk regions that were recently accessed, in order to speed up IOs.

Of course, active applications have priority over this cache, so in case a process needs more ram, the cache is dynamically shrinked.

There is a lot of documentation about Linux page cache on the web, here is one.

Solution 2

I have the same problem, but I checked the kernel parameters located under /proc/sys/vm/ : overcommit_memory = Default 0 overcommit_ratio = Default 50 max_map_count = Default 65530

And in my case the parameter, root of my problems nr_hugepages, i haved the custom value, but change it for default

nr_hugepages = 0

For more information see https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/6/html/performance_tuning_guide/s-memory-captun

Sorry for my english :(

Solution 3

For me it seemed to be the page cache not clearing itself.

sync; echo 1 | sudo tee /proc/sys/vm/drop_caches

Gave me 30gB back after a few seconds.

Comments

[user2299634]

I'd be interested by a free -m and a regular top.

[SharkWipf]

@user2299634 I have reproduced the situation and added the output of free -m and top, as requested.

[Nehal Dattani]

What is the output from /proc/meminfo ?

[user2299634]

What tool are you using to do your stress test ? Do you have customs sysctl relative to memory management, buffers, storage ? Anything like bcache ou dm-cache ?

[SharkWipf]

I do not have access to the server for 2 weeks, so I cannot reproduce anything for a while. Nothing special in sysctl afaik, will doublecheck when I have access. The "stress test" is basically a bash loop creating/deleting snapshots, while a dd if=/dev/urandom is constantly changing the lv that's being snapshotted. The dd+snapshot is what makes the RAM fill up, which makes sense, but it not showing does not make sense.

[charlie_pl]

You are a lifesaver Jeant! Hugepages were an issue also for me.