Linux running slow with 0 swap left

Question

Our Linux server is responding very slowly. top doesn't show any excessive CPU usage. I noticed that even though there's about 5GB of free memory, the system is using all of swap with no free swap left. Could this be reason for system running slow? Is there any solution other than reducing the number of processes?

Secondly, even though there is free memory available, why is Linux swapping already? I thought swap is used only when there’s not actual memory left.

free -m
             total       used       free     shared    buffers     cached
Mem:         32045      26218       5826          0        127        123
-/+ buffers/cache:      25967       6077
Swap:        16387      16387          0

Update:

• swapiness is at default level: 60
• I don't think it's a numa system
• I see a couple of java processes running with 8GB heap: -Xms8000m -Xmx8000m

Seems nuts on first look, but possibly someone had reasons for doing this. I think this is what's eating up most of swap, but I need to further investigate how the java heap affects Linux swap and memory/performance in general. Any pointers on impact of above java heap config on system performance will be really helpful.

Solution 1

How many Java processes have been initialized with the 8GB heap? Perhaps keep the maximum at 8 and lower the initial heap size. If you have the disk space, experiment with increasing the swap size as well

Solution 2

Do you have tmpfs mounts? Those are backed up by swap and may be the cause for zero swap space available. Do syslog or dmesg show anything unexpected? Are you sure that your system is not constantly swapping? Check vmstat -SK 1 and verify columns si and so (swapped pages in and out in KB/s). If you need to know which processes are swapping, run sudo iotop -od5. If you're running Ubuntu, you need delayacct kernel flag for full statistics. For me, your system seems to have really small cache and buffers for such a big system memory.

The swap is used to improve whole system performance. Example of this would be to swap out parts of a big program that's currently sleeping to allow using more memory for disk cache or file buffers. If you were to disable the swap in this case, system would effectively have smaller disk cache and file buffers. Because your swap space is full, you may effectively suffer from the same issue.

The swap space is especially important if you have any programs that "leak" memory (that is, a program acquires blocks of memory that are not really used). Such a "leak" may not be an error in the program, it may just happen that the memory is so seldom used that it may be better treated as leaked. Without swap space, such leaked memory cannot be used as cache or buffers.

A correctly running OS with virtual memory is using swap pretty much always unless the whole working set (all the accessed files, all memory reserved by applications and all write buffers) fits in the memory fully for the entire runtime of the system. In most cases this is only true if the system is constantly rebooted or system has huge amount of RAM compared to whole filesystem size.

Comments

[Admin]

Is this a numa system? Maybe the resources are not distributed evenly (e.g. one process taking a lot of memory) and one node has no memory left?

[Admin]

Using swap is not necessarily bad. Use vmstat or iotop to verify that there actually is swap activity.

[Admin]

Running with no swap left is likely to slow a system down, as it no longer has enough RAM left to comfortably keep disk files in cache.

[Mikko Rantalainen]

Update in 2019: Looking at the question again I would now guess that your system runs some short-lived processes that require around 5 GB RAM. This causes your buffers and cached values in free output to go really slow, fill the swap and cause overall slowness. The fact that you have such a big amount for free suggests that the moment you're investigating the problem the process that required huge amount of RAM is already gone. If you cannot easily figure out which process is causing the trouble, it's time to install sar and read the documentation: man sar.