Why does MemTotal in /proc/meminfo change?

Question

I monitor value from /proc/meminfo file, namely the MemTotal: number. It changes if a ram module breaks, roughly by size of the memory module - this is obvious.

I know the definition for the field from kernel documentation:

MemTotal: Total usable ram (i.e. physical ram minus a few reserved
          bits and the kernel binary code)

The dmesg also lists kernel data. What other particular actions make the MemTotal number change if I omit hardware failure of the memory module?

This happens on both physical & virtual systems. I monitor hundreds of physical, thousands of virtual systems. Although the change is rather rare, it does happen.

Solution 1

I was not comfortable with having bug in kernel or a module, so I digged further and found out... that MemTotal can regularly change, downwards, or upwards. It is not a constant and this value is definitely modified by kernel code on many places, under various circumstances.

E.g. virtio_balloon kmod can decrease MemTotal as well as increase it back again. Then of course, mm/memory_hotplug.c is exporting [add|remove]_memory, both of which are used by lot of drivers too.

Solution 2

That should never change. It could be an indication of some faulty locations within RAM. You can diagnose your RAM using Memtest86+.

Other than that the only other way that number would change might be if you were analyzing a virtual machine guest. Perhaps the host the VM guest is running on is adjusting the RAM allocated to the guest.

Edit #1

I did dig up this forum post which discusses the potential leaking of memory by Kernel modules which could lead to the MemTotal changing.

excerpt from: Re: How do you determine the amount of system memory?

On 7/30/08, Bill McGonigle wrote:

On Jul 30, 2008, at 10:03, Thomas Charron wrote:

MemTotal can change. MemTotal doesn't show memory utilized by the kernel binaries themselves in memory, however there are situations where modules can leak memory which also removes from MemTotal. I don't recall the specifics of this happening, but I do know what Paul is talking about now. Well, that's only a bug if you define MemTotal as the physical memory present in a machine. If you view it as total memory available to the kernel, it's performing correctly, right?

Yes, and the 'bug' is when that number goes down, it means somethings leaking, aka, a 'bad kernel'. At least potentially 'bad' (read potentially buggy) kernel modules.

Comments

[Matej Kovac]

memory is tested after such things happen, on physical systems, no bad signs... we do not dynamically change allocated ram to guests.

[slm]

@MatejKovac - Passing the tests doesn't completely exclude you from hardware that might be starting to fail.

[Matej Kovac]

ACK. but servers do operate normally. is there a possibilty that loading a module or changing some parameters makes kernel allocate more memory, changing MemTotal?

[slm]

@MatejKovac - not that I've ever seen. Those values are, to my knowledge and experience, always derived from the the actual hardware present.

[slm]

@MatejKovac - see update.

[Matej Kovac]

this was my worst guess. I was looking around and was seeing similar hints, but this is quite clear. thanks..

[slm]

@MatejKovac - I too was quite surprised that this was the case as well. I've monitored that value for years using tools such as Nagios and haven't ever seen that value change, but you peaked my curiosity with it, so if I have time I might go back through some databases to see if that value has changed.

[Matej Kovac]

I have accepted my own answer now, but still count with possibility of a bug - this can probably be never omitted. But I digged in kernel code for a while and found MemTotal can change in regular occasions too, so I think it is more 'correct' answer.

[slm]

@MatejKovac - your Q so you get to do as you see fit 8-), but in this situation the things you've linked to are likely not the reasons you were seeing your MemTotals fluctuate. virtio is the virtual network interface that's used for guest VMs, so that hardly seems to be a real reason in your circumstance for the fluctuation on a physical host.

[slm]

@MatejKovac - The link to memory_hotplug.c is related to hot add/remove of memory from a system, see this: events.linuxfoundation.org/sites/events/files/… as well as to the add/removal of memory to a running VM when you adjust its balloon assignment. Given the behavior you described with the MemTotal, a leaking/buggy module, IMO, is still the most likely candidate.

[Matej Kovac]

I understand what MemTotal is and that it does not show exact physical memory size. As I explained in my own answer, I went briefly through kernel code and looked for modification of totalram_pages variable, what is actually value displayed in MemTotal here. So there are legitimate means that it changes, not just (possible) bugs. I cannot rely on monitoring this value (so much).