Memory runs full over time, high "buffer/cache" usage, low "available" memory

Question

Whenever I reboot my laptop, everything runs amazingly and I have a maximum of 40% memory usage (out of 8GB). However over time (~ 1 day of usage), memory usage goes up to 90%+, and the system starts swapping.

Right now, free -mh returns this:

              total        used        free      shared  buff/cache   available
Mem:           7,7G        1,3G        141M        223M        6,3G        246M
Swap:          7,5G        530M        6,9G

I was assuming that buff/cache memory is free to be reallocated if processes require it, but it seems to mostly be unavailable.

cat /proc/meminfo:

MemTotal:        8055268 kB
MemFree:          145184 kB
MemAvailable:     247984 kB
Buffers:           49092 kB
Cached:           423724 kB
SwapCached:        38652 kB
Active:           881184 kB
Inactive:         791552 kB
Active(anon):     708420 kB
Inactive(anon):   725564 kB
Active(file):     172764 kB
Inactive(file):    65988 kB
Unevictable:         252 kB
Mlocked:             252 kB
SwapTotal:       7812092 kB
SwapFree:        7267624 kB
Dirty:               352 kB
Writeback:             0 kB
AnonPages:       1195320 kB
Mapped:           235860 kB
Shmem:            234068 kB
Slab:            6117796 kB
SReclaimable:     167260 kB
SUnreclaim:      5950536 kB
KernelStack:       10352 kB
PageTables:        30312 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:    11839724 kB
Committed_AS:    6410728 kB
VmallocTotal:   34359738367 kB
VmallocUsed:           0 kB
VmallocChunk:          0 kB
HardwareCorrupted:     0 kB
AnonHugePages:    104448 kB
CmaTotal:              0 kB
CmaFree:               0 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
DirectMap4k:     1361472 kB
DirectMap2M:     5859328 kB
DirectMap1G:     1048576 kB

I found these values especially interesting, as they correlate a lot with the buff/cache usage from free, but I don't know what to do with them or where to look next:

SReclaimable:     167260 kB
SUnreclaim:      5950536 kB
Slab:            6117796 kB

Where can I look next? What is the slab, and is there a way to reduce it's memory usage?

Solution 1

You should check with top if something is actually using your RAM or not, sort by memory usage, or check the memory usage in the System Monitor.

Linux will borrow unused memory for disk caching. This makes it looks like you are low on memory, but you are not. Check this webpage for more explanation : https://www.linuxatemyram.com/

You have actually around 6.5Gb unused memory on the example which was posted. You can also see that the swap amount is very low (540Mb).

You can release the cache(s) as explained here and then free will display you the free memory in the available field :

• To free pagecache:

    # echo 1 > /proc/sys/vm/drop_caches
  

• To free dentries and inodes:

    # echo 2 > /proc/sys/vm/drop_caches
  

• To free pagecache, dentries and inodes:

    # echo 3 > /proc/sys/vm/drop_caches
  

Or with this command :

free && sync && echo 3 > /proc/sys/vm/drop_caches && free

Regarding Slab:
Slab, SReclaimable, SUnreclaim
The kernel does a lot of repetition during the time it is running. Some objects, like asking for the specific inode of a file may be performed thousand times a day. In such case, it would be wise to store it in a quick reference list, or cache. Slab are the caches for kernel objects, to optimize those activities that happen the most.

The Slab field is the total of SReclaimable and SUnreclaim.

Try to troubleshoot what is using the Slab SUnreclaim memory amount with slabtop.

The above are meant to be run as root.

Solution 2

So after some somewhat overengineered troubleshooting I found out that xflux seems to be causing the kmalloc-4096 slab to grow slowly but steadily. I will try to find out why and update this anwser.

Comments

[Patrick Mevzek]

But what is your real problem? The system works as expected, nothing to fear from these numbers. And if you dislike the swapping you can run your system without swap at all. Otherwise see if you are a long running process that hogs the RAM. On my systems it is often the browser...

[Max Hollmann]

It gets sluggish, because of swapping. But the memory genuinely seems to be used, so swap is necessary. Swappiness is already pretty low (10). None of my processes seem to take up a lot of memory. The most is used by firefox, and that's less than 1GB. Even when I close most processes, most of the RAM is still unavailable.

[Patrick Mevzek]

Remove the swap then. In top hit M and processes will be ordered by amount of RAM they use. RAM is not unavailable. Cache/Buffer part will shrink automatically as needed. And as counter-intuitive as it may be the kernel may decide it is better to put things in swap (pages rarely touched, like a sleeping process) than take memory from the buffer cache.

[Alecz]

@PatrickMevzek, 6GB of unreclaimable slab memory when the active processes use only 1.6 GB sounds like an issue to me because the Kernel is not freeing the memory to be used by the processes, so the system becomes sluggish and needs a reboot.

[NickD]

Just to be clear: freeing the various caches is counterproductive. It will "improve" the numbers and perhaps make the OP feel better for a little while, but the caches will be repopulated with time and everything will go back to the way it was before the caches were freed: that is normal. Unless there is some other indication that something is wrong, it is best to leave this alone.

[phemmer]

Additionally the provided meminfo shows that the memory is sitting in slab unreclaimable, so this won't do anything anyway.

[Max Hollmann]

I've tried that, it releases at most a couple of MB. Like Patrick says, it seems to be the slab, and I have no idea what that is or how to troubleshoot it.

[NickD]

Why do you need to troubleshoot it? What do you think is wrong?

[magor]

@Nick it is counterproductive but if the OP wants to see the free memory at the available field, that's what he should do. I have not noticed the SUnreclaim amount....try to troubleshoot it with slabtop, updated answer

[HopeKing]

@mazs - in the link you provided (+1 for the great link), it says "To see how much ram your applications could use without swapping, run free -m and look at the "available" column. Now the OP has only 246M in the available column - But you mentioned in your answer that there is actually 6.5GB. Seems like there is a contradiction ?