When does Linux run out of memory?

Question

I'm trying to establish why a process keeps unexpectedly stopping, and one possibility is that the server is running out of memory, but I'm not sure I understand memory and Linux properly.

Take the following output from the free command (taken from the same machine with the issue I'm trying to fix):

             total       used       free     shared    buffers     cached
Mem:           991        827        163          0        107        361
-/+ buffers/cache:        358        633
Swap:            0          0          0

The top line suggests that this machine is indeed using a lot of memory (only 163 Mb free) and so if a few more processes fire up (which they do) then we could have an out of memory situation with various processes being killed.

However, I've always been led to believe that as Linux makes great use of buffers and caches in memory which can be used if needed, that the figure I should be paying more attention to is the second line which suggests that ~633 Mb is free, in which case I think it's unlikely that this machine is running out of memory.

So can you clarify my understand of Linux and memory, and help me understand when Linux actually runs out of memory?

PS - this machine is a single purpose machine - it runs the background process for a large web application, that's all it does. No web server, no database, just one huge Ruby on Rails app running as a background process. Occasional cron jobs fire up for specific application tasks which would temporarily create another instance of the rails app in memory.

Solution

The kernel uses ram that is not otherwise utilized to cache data that's been read in from storage - this is called the buffer cache. Since the buffer-cache is just a copy of what's on the harddrive, if the kernel ever needs memory for something more important, it can just "forget" the copy of that data in memory and use it for something else. If it ever needed that data, it'd just reread it from the drive.

Keep in mind there is the buffer-cache and then there are regular kernel buffers (for tcp sockets, routing table, etc.)

So, the free command on the first line is showing you:

• Total: How much ram you got, period.
• Used: How much of it actually has anything of value in it.
• Free: Absolutely unused - may not be empty, but whatever there would be garbage.
• Shared: Things shared between processes - largely shared libraries.
• Buffers: regular kernel buffers.
• Cached: ostensibly, the buffer-cache.

Now, the second line shows you what's "really used." as in, what is in ram that cannot be forgotten without consequences. And the "free" field on the second line is the sum of unused ram and buffer-cache.

The last line would include swap space - storage space the kernel can write important information in memory to, so that it can use that memory location for something else. But, by your screenshot, you don't have any configured.

Right now, it looks like you're using a solid ~360MB out of your 1GB. If you're concerned or suspect you're running out of memory, check the kernel log messages. The kernel will log when it has reached an absolutely-out-of-memory-and-have-to-do-something situation. It will invoke the out of memory killer with the hope that it can salvage the situation by killing off one process.

I initially thought this was a VPS. If its a physical server and you don't have extra RAM you can add to it, then consider adding swap space!

Otherwise, check /var/log for log messages or at the command line, just use dmesg to check the latest kernel messages.

Comments

[mulaz]

read this for more info: linuxatemyram.com

[aaronrussell]

Thanks, the linked question does answer my question (and confirm that my understanding on Linux and memory is accurate).... doesn't help me understand why this damned process keeps stopping :p

[voretaq7]

@aaronrussell Have you checked your log files? Is the process being killed because Linux thinks it's out of RAM?