Why is swap being used even though I have plenty of free RAM?

Question

I thought the whole essence of swap was to act as a temporary storage safety net when RAM was full but my swap partition is constantly being used even though I sometimes have as much as 3GB free RAM. Is this normal?

Solution 1

You could try changing your "swappiness" value:

From the Ubuntu's Swap FAQ:

What is swappiness and how do I change it?

The swappiness parameter controls the tendency of the kernel to move processes out of physical memory and onto the swap disk. Because disks are much slower than RAM, this can lead to slower response times for system and applications if processes are too aggressively moved out of memory.

1. swappiness can have a value of between 0 and 100

2. swappiness=0 tells the kernel to avoid swapping processes out of physical memory for as long as possible. For Kernel version 3.5 and newer it disables swappiness.

3. swappiness=100 tells the kernel to aggressively swap processes out of physical memory and move them to swap cache

The default setting in Ubuntu is swappiness=60. Reducing the default value of swappiness will probably improve overall performance for a typical Ubuntu desktop installation. A value of swappiness=10 is recommended, but feel free to experiment. Note: Ubuntu server installations have different performance requirements to desktop systems, and the default value of 60 is likely more suitable.

To check the swappiness value

cat /proc/sys/vm/swappiness

To change the swappiness value A temporary change (lost on reboot) with a swappiness value of 10 can be made with

sudo sysctl vm.swappiness=10

To make a change permanent, edit the configuration file with your favorite editor:

gksudo gedit /etc/sysctl.conf

Search for vm.swappiness and change its value as desired. If vm.swappiness does not exist, add it to the end of the file like so:

vm.swappiness=10

Save the file and reboot.

Run sudo sysctl --load=/etc/sysctl.conf after editing the file to apply the changes

Also you can check out: https://askubuntu.com/a/103916/54187

Solution 2

There are a few different aspects to your question.

Firstly, what your definition of "free" is. It is actually not as simple as it sounds in Linux (or any modern operating system).

How Linux uses RAM (very simplified)

Each application can use some of your memory. Linux uses all otherwise unoccupied memory (except for the last few Mb) as "cache". This includes the page cache, inode caches, etc. This is a good thing - it helps speed things up heaps. Both writing to disk and reading from disk can be sped up immensely by cache.

Ideally, you have enough memory for all your applications, and you still have several hundred Mb left for cache. In this situation, as long as your applications don't increase their memory use and the system isn't struggling to get enough space for cache, there is no need for any swap.

Once applications claim more RAM, it simply goes into some of the space that was used by cache, shrinking the cache. De-allocating cache is cheap and easy enough that it is simply done in real time - everything that sits in the cache is either just a second copy of something that's already on disk, so can just be deallocated instantly, or it's something that we would have had to flush to disk within the next few seconds anyway.

This is not a situation that is specific to Linux - all modern operating systems work this way. The different operating systems might just report free RAM differently: some include the cache as part of what they consider "free" and some may not.

When you talk about free RAM, it's a lot more meaningful to include cache, because it practically is free - it's available should any application request it. On Linux, the free command reports it both ways - the first line includes cache in the used RAM column, and the second line includes cache (and buffers) in the free column.

How Linux uses swap (even more simplified)

Once you have used up enough memory that there is not enough left for a smooth-running cache, Linux may decide to re-allocate some unused application memory from RAM to swap.

It doesn't do this according to a definite cut-off. It's not like you reach a certain percentage of allocation then Linux starts swapping. It has a rather "fuzzy" algorithm. It takes a lot of things into account, which can best be described by "how much pressure is there for memory allocation". If there is a lot of "pressure" to allocate new memory, then it will increase the chances some will be swapped to make more room. If there is less "pressure" then it will decrease these chances.

Your system has a "swappiness" setting which helps you tweak how this "pressure" is calculated. It's normally not recommended to alter this at all, and I would not recommend you alter it. Swapping is overall a very good thing - although there are a few edge cases where it harms performance, if you look at overall system performance it's a net benefit for a wide range of tasks. If you reduce the swappiness, you let the amount of cache memory shrink a little bit more than it would otherwise, even when it may really be useful. Whether this is a good enough trade-off for whatever problem you're having with swapping is up to you. You should just know what you're doing, that's all.

There is a well-known situation in which swap really harms perceived performance on a desktop system, and that's in how quickly applications can respond to user input again after being left idle for a long time and having background processes heavy in IO (such as an overnight backup) run. This is a very visible sluggishness, but not enough to justify turning off swap all together and very hard to prevent in any operating system. Turn off swap and this initial sluggishness after the backup/virus scan may not happen, but the system may run a little bit slower all day long. This is not a situation that's limited to Linux, either.

When choosing what is to be swapped to disk, the system tries to pick memory that is not actually being used - read to or written from. It has a pretty simple algorithm for calculating this that chooses well most of the time.

If you have a system where you have a huge amount of RAM (at time of writing, 8GB is a huge amount for a typical Linux distro), then you will very rarely ever hit a situation where swap is needed at all. You may even try turning swap off. I never recommend doing that, but only because you never know when more RAM may save you from some application crashing. But if you know you're not going to need it, you can do it.

But how can swap speed up my system? Doesn't swapping slow things down?

The act of transferring data from RAM to swap is a slow operation, but it's only taken when the kernel is pretty sure the overall benefit will outweigh this. For example, if your application memory has risen to the point that you have almost no cache left and your I/O is very inefficient because of this, you can actually get a lot more speed out of your system by freeing up some memory, even after the initial expense of swapping data in order to free it up.

It's also a last resort should your applications actually request more memory than you actually have. In this case, swapping is necessary to prevent an out-of-memory situation which will often result in an application crashing or having to be forcibly killed.

Swapping is only associated with times where your system is performing poorly because it happens at times when you are running out of usable RAM, which would slow your system down (or make it unstable) even if you didn't have swap. So to simplify things, swapping happens because your system is becoming bogged down, rather than the other way around.

Once data is in swap, when does it come out again?

Transferring data out of swap is (for traditional hard disks, at least) just as time-consuming as putting it in there. So understandably, your kernel will be just as reluctant to remove data from swap, especially if it's not actually being used (ie read from or written to). If you have data in swap and it's not being used, then it's actually a good thing that it remains in swap, since it leaves more memory for other things that are being used, potentially speeding up your system.

Solution 3

Setting the swappiness value doesn't work in every situation. If it works for you, great. If not, I've written a script to periodically clear out swap by turning it off and back on again.

Toggling swap is a bit risky if you're not careful. If you don't have enough free RAM to hold everything in RAM plus everything in swap, trying to disable swap will cause your system to become unresponsive. My script first checks whether there's enough free RAM (which takes a bit of doing, as the actual amount of free RAM is different from what free reports as free), then only toggles swap if so. But, if you're a bit short on RAM, don't start another major process while the script is running. Here it is:

#!/bin/bash

# Make sure that all text is parsed in the same language
export LC_MESSAGES=en_US.UTF-8
export LC_COLLATE=en_US.UTF-8
export LANG=en_US.utf8
export LANGUAGE=en_US:en
export LC_CTYPE=en_US.UTF-8

# Calculate how much memory and swap is free
free_data="$(free)"
mem_data="$(echo "$free_data" | grep 'Mem:')"
free_mem="$(echo "$mem_data" | awk '{print $4}')"
buffers="$(echo "$mem_data" | awk '{print $6}')"
cache="$(echo "$mem_data" | awk '{print $7}')"
total_free=$((free_mem + buffers + cache))
used_swap="$(echo "$free_data" | grep 'Swap:' | awk '{print $3}')"

echo -e "Free memory:\t$total_free kB ($((total_free / 1024)) MB)\nUsed swap:\t$used_swap kB ($((used_swap / 1024)) MB)"

# Do the work
if [[ $used_swap -eq 0 ]]; then
    echo "Congratulations! No swap is in use."
elif [[ $used_swap -lt $total_free ]]; then
    echo "Freeing swap..."
    swapoff -a
    swapon -a
else
    echo "Not enough free memory. Exiting."
    exit 1
fi

You must run this script as root (e.g., with sudo). This script won't leave your system unresponsive; if you've got insufficient RAM, it will refuse to toggle swap. I've used this script without problems for close to five years now.

Solution 4

I edited the script of Scott Severance to match the newer versions of free which already include a total available memory field.

#!/bin/bash

free_mem="$(free | grep 'Mem:' | awk '{print $7}')"
used_swap="$(free | grep 'Swap:' | awk '{print $3}')"

echo -e "Free memory:\t$free_mem kB ($((free_mem / 1024)) MiB)\nUsed swap:\t$used_swap kB ($((used_swap / 1024)) MiB)"
if [[ $used_swap -eq 0 ]]; then
    echo "Congratulations! No swap is in use."
elif [[ $used_swap -lt $free_mem ]]; then
    echo "Freeing swap..."
    sudo swapoff -a
    sudo swapon -a
else
    echo "Not enough free memory. Exiting."
    exit 1
fi

Solution 5

Generally swap remains unused on now-a-days systems. In my experience, the processes which are running for a long time without intensive operations are shifted to swap by the linux.
It makes a few affected programs run slow.
If you have lots of ram free, you may switch the swap off by running the command:
swapoff -av (you'll need sudo rights for it.)
If you don't like the swap off, you may switch it on, using symmetrical command:
swapon -av (again sudo required).

View more solutions

Comments

[Admin]

Will update the question with pastebins of those commands tomorrow. It's 12 am right now :(

[Admin]

I should edit the title to add "...but I'm too sleepy to care right now, so you guys take a nap too" :P

[Brendan Long]

@SimonRichter Yes that's the idea. Swappiness of 60 is probably too agressive on modern systems though, since that means you're starting to swap out when you still have several gigs of free memory.

[Brendan Long]

Turning swap off entirely is overkill since swapiness=0 does the same thing, but safer (if you do end up with too much stuff in memory, it will throw something in swap).

[Joe]

+1 for script. You might want to modernize it a bit by changing MB to MiB in the outputs. I wrote a similar script without all the (good) safety tests you include. On my old notebook where swap was used heavily, it would take awhile to run - a minute or two, but didn't hurt anything. Once in awhile it would fail because there wasn't enough free memory for it to run, but that didn't cause any additional problems. I had to reboot to clear things up when that happened. It had to do with what looked like a memory leak in Transmission (since fixed, I believe) and having only 2GiB ram.

[Joe]

+1 for detailed explanation. I also like to keep 2GiB swap on my computers. If something goes bad - memory leaks, etc., then you can see swap going up and look for the problem before out of memory occurs. It's a safety net as well as a performance tool.

[Scott Severance]

On my netbook, which has only 2GB of RAM, I keep 4GB of swap. This is because I use the netbook for a lot more that it was intended for and my programs frequently require more than 2GB of memory. Since Linux behaves very poorly in out-of-memory situations, I prefer to have a large safety net. And I have plenty of disk space.

[thomasrutter]

If your programs frequently require more than 2GB memory I'd consider chucking more RAM in there (it's ultra-cheap!) because exhausting RAM is gonna hurt performance. For the benefit of other readers: make that you're not just looking at cache memory which is supposed to hog available RAM - see other references to free -m.

[Scott Severance]

@neon_overload: I should just note that according to HP, my netbook's maximum RAM is 1GB. I've got 2GB in there now, and internet sources suggest that 2GB is the actual maximum. Thus the need for these sorts of workarounds.

[thomasrutter]

If something has been moved to swap and it remains in swap, that's normally a good thing - it means the data is not actually being used, and it's freed up some of your RAM for other stuff, resulting in a potential boost in speed. What has lead you to believe that this data shouldn't be in swap and should be taking up more valuable RAM instead? Usually the kernel's pretty good at knowing what is unused, and which can safely remain in swap to free up RAM.

[Scott Severance]

@neon: Bear in mind that my machines have 2GB and 3GB of RAM, respectively, which is quite little these days. The evidence is in performance. For example, bringing up menus or switching programs can be slow due to swapping, as can Compos visoal effects. This slowness is cured by toggling the swap, thus demonstrating that at least in some situations the kernel's optimization algorithms are suboptimal. I can't argue with repeated experience.

[AzkerM]

Phew!! This just boosted down to 0% of swap.... I didn't had much time to look into this complete question and its answer but I Favorited it to read it when I get some free time.

[chrisinmtown]

@thomasrutter: How does your answer change for large servers, one of mine has 1TB (yes one full terabyte) of ram and yet is still using swap; what swappiness setting and what size swap partition would you recommend?

[thomasrutter]

I'd recommend first doing more analysis of what on that server is using up that 1TB and wanting more, and addressing that. Swapping is a symptom of a potential problem (well, low memory) rather than a cause and adjusting swappiness in that setting would be like re-arranging deck chairs on the titanic. It's wise to have swap in case it's needed sometimes, but if you're typically seeing many gigabytes swapped on a server something isn't right. For a server with relatively even load, you'd be aiming for little or no swap actually being used.

[Dan Barron]

Clearest explanation of memory usage in Linux I've found anywhere. Thanks!

[jhfrontz]

Why wouldn't you just set swappiness to a low value? That would seem to be doing the same thing as your script, but in a much more controlled fashion.

[Scott Severance]

@jhfrontz: Actually, swappiness only affects how aggressively the kernel swaps stuff out. When I wrote this script years ago, setting swappiness to a low value had no noticeable effect (and especially not a quick one) on a machine with limited RAM.

[jhfrontz]

Right-- I was thinking that setting swappiness to 0 (at boot) would say "don't swap unless you really, really, really have to". And thus, the only thing that would be swapped would be that required for continued operation of the system.

[Scott Severance]

Sure, but that doesn't address the problem of what to do once memory is freed and stuff is still in swap despite there now being enough free memory. The kernel won't pull it out of swap until it's needed, which can cause major issues with system responsiveness. (Consider when the system menu is all swapped out: it doesn't get swapped back in until you click the menu button, which means you suddenly have to wait a while for the menu to pop up.) That's why manually pulling it out of swap at an opportune time can improve system responsiveness.

[jhfrontz]

Good point; I wasn't thinking of the post-OOM-preventing-swap recovery scenario.

[sudo]

I feel like disabling swap altogether. I have a situation now where I have 64GiB of RAM, mostly only for Postgres to use, and I've set Postgres's shared buffer size to 30GiB. There are 10 postgres processes running, each with SHR = 30GiB. My system is swapping in and out like crazy, lagging badly. There's 45GiB of "cached Mem" (aka free memory) sitting there that it's not using! I think the system is being fooled by all the SHR memory of the postgres processes and thinks I'm out of memory.

[thomasrutter]

If your system is swapping badly, disabling swap is probably going to create more problems. You may have unintentionally configured Postgres incorrectly or something may be using significantly more memory than you realise. The swap system does not get "fooled" - it knows exactly what pages are allocated for what purpose because it's all in the kernel. Firstly, identify if it really is swapping or if it's disk activity that is not swapping. Then, identify why your system is struggling to allocate RAM. I don't know enough about how Postgres uses memory to help more.

[Tmanok]

I just want to point out that this is a life saver with SSDs that have a maximum number of bytes writtable (aka endurance), I don't want things swapping unless I'm out of memory and once I have enough memory I want everything back in memory to stop killing my SSD. This applies to servers too, I could care less about the 2 minutes of purging swap if the remainder of the day my users are all performing extremely fast.

[Tmanok]

One thing to note with these kinds of scripts, is that they are effective at removing swap, but you will start swapping immediately after if you don't purge your buffers and caches. For example less than a second after running this script, I started accumulating more swap. This is very easily shown in htop. There are many arguments about caches, cache pressures, how quickly buffers and caches will "move out of the way" for working memory, and more. But for me, I simply eliminate them with "sync" and telling the kernel to drop caches.

[france1]

All these things are tunable (look into /proc/sys/vm/)

[alper]

Shouldn't language be: export LANGUAGE=en_US.UTF-8?