How to know whether enough memory is free to deploy a new application on a Linux machine?

Question

I have got a Linux machine whose memory snapshot (according to /proc/meminfo) is as follows:

MemTotal:     16413388 kB
MemFree:         48296 kB
Buffers:        193600 kB
Cached:        1986448 kB
SwapCached:     874512 kB
Active:       15034264 kB
Inactive:       713672 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     16413388 kB
LowFree:         48296 kB
SwapTotal:     8385920 kB
SwapFree:      4682408 kB
Dirty:            3124 kB
Writeback:           0 kB
Mapped:       13005560 kB
Slab:           257784 kB
CommitLimit:  16592612 kB
Committed_AS: 59624324 kB
PageTables:     233748 kB
VmallocTotal: 536870911 kB
VmallocUsed:    267064 kB
VmallocChunk: 536603555 kB
HugePages_Total:     0
HugePages_Free:      0
Hugepagesize:     2048 kB

This is a 16 GB-machine, and I have a Java application to deploy on it, which will use 3 JVM instances whose typical combined memory requirement will be close to 1 GB.

How can I make sure that it will be safe to deploy the said application without affecting other applications currently running on that machine. Is it possible to find that out from the above memory snapshot?

What other statistics may help me to decide that, and how can I collect those statistics?

Solution

(It's probably a bit late for the OP, but this is asked quite often, so I'll give it a shot)

free normally shows something like this:

             total       used       free     shared    buffers     cached
Mem:       8195284    8137708      57576          0    1232328    2651156
-/+ buffers/cache:    4254224    3941060
Swap:     18892216     759852   18132364

People tend to look at the Mem: line when trying to find out how much free memory they have. Unfortunately that line is quite misleading, because the Linux kernel tries to make optimal use of the available memory in (at least) these ways:

• It will cache data from the I/O subsystem (e.g. the disk), so that it will be readily available if needed.

• It will actively evict processes that have been inactive for some time to the swap space, in favour of caching data for active processes. This tends to favour throughput over responsiveness, so some people tune their kernel to change this behaviour.

The first point is the source of confusion regarding free, because the Mem: line includes the memory used for caching in the used memory amount. The kernel, though, will cache as much as possible for performance reasons. In fact, on any Linux system that has been up for some time, the free memory tends to be close to zero - unused memory is wasted memory.

The cache memory, though, can be freed by the kernel if needed by another process. While it will impact I/O performance to a degree, other processes can have more memory without using the swap space. Therefore, for most intents and purposes, that memory is free.

That's why free includes a second line, where the cache memory is considered free:

-/+ buffers/cache:    4254224    3941060

This second line is what people should be looking at when they want to know if they have enough free memory for a certain purpose.

In the example above, according to the Mem: line there are ~57 MB of free memory. If one reads the second line, though, there are in fact about 3.9 GB that can be used without forcing active processes to swap. As a sidenote, there are also about 760 MB of rarely-used data that have been swapped out, to make more space in the main memory for processes and caching.

At roughly the same time, the contents of /proc/meminfo:

MemTotal:        8195284 kB
MemFree:           57660 kB
Buffers:         1232352 kB
Cached:          2651156 kB
SwapCached:       119936 kB
.
.
.

MemTotal: the available physical memory detected by the kernel.

MemFree: the unused physical memory - the free memory shown in the Mem: line of free.

Buffers: relatively temporary storage of raw disk blocks.

Cached: in-memory cache for files read from the disk. It does not include SwapCached memory.

SwapCached: memory that was once swapped out, then swapped back in but is still in the swap space. If needed, its contents can be just discarded (very fast!), without having to swap them out (slower).

So, to have a semi-accurate estimate of the memory that is actually available

MemFree + Buffers + Cached + SwapCached

is a good starting point - and the one free shows in that second line.

Naturally, memory management and the related statistics and measurements are more complicated than this. The numbers shown by free are mere estimates at best, since there are a lot of other variables to take into account if you want to go deeper. For people who regularly perform memory usage optimization, this is almost a form of art.

EDIT:

A somewhat humorous link about this "issue":

http://www.linuxatemyram.com/

EDIT 2:

To confirm the comment about memory use analysis almost being a form of art:

Even free misses a major chunk of cached data on modern Linux systems. From /proc/meminfo on my system:

SReclaimable:    2253576 kB

That's about 2GB of memory that is used by the system slab allocator for caching directory entries and such and it is reclaimable (i.e. it can be cleared and used by processes if necessary). Yet free does not consider it cache memory and does not enter it in any of its calculations and therefore it shows up as used memory.

The slabtop utility, if available, allows the system administrator to find out what the slab cache is used for.

A way (for the root user only) to have free show the actual memory use of the system is the following:

# swapoff -a
# sync
# echo 3 > /proc/sys/vm/drop_caches 
# free
             total       used       free     shared    buffers     cached
Mem:       8195284    3181468    5013816          0       8656     228832
-/+ buffers/cache:    2943980    5251304
Swap:            0          0          0
# swapon -a

The first command disables the swap space. It should not be issued if the available memory may not be enough to hold the data that have been swapped out - in that case one has to take into account the Swap: line of free in their memory usage calculations.

The second command pushes all buffered data to the disk. It allows more cache memory to be freed in the next step.

The third command is the most important of the set - it forces the kernel to discard as much cached data as possible (page cache, directory entries, inodes etc).

Then free finally shows what the running processes actually use in its -/+ buffers/cache: line. It is quite noticeable that even after dropping all cached data the kernel quickly starts caching again - in this case it has already reached almost 250MB of cached data within a few seconds.

The final command enables the swap space again - it is only necessary if the first command was used too.

It should be noted that these commands should be executed by the root user in order to have the necessary privileges.

Comments

[Manish]

free command's stats are same as that given by /proc/meminfo