kswapd often uses 100% CPU when swap is in use

linux cpu-usage linux-kernel swap

1,878

Solution 1

When the problem is happening, can you check if issuing the following command stops it: echo 1 > /proc/sys/vm/drop_caches

If it works, you can schedule it as a periodic cron job as a workaround.

Solution 2

I have a C720 running Linux Kernel 4.4.0 on Ubuntu 14.04.1 LTS with 2 GB RAM and 2 GB swap.

Assuming heavy Chrome/Chromium usage, here are some ways to make your system more performant:

Edit /etc/default/grub and add the following kernel parameters to the GRUB_CMDLINE_LINUX_DEFAULT line:
- elevator=noop
- zswap.enabled=1
- transparent_hugepage=madvise
Run sudo update-grub2.
Edit /etc/sysctl.conf and append the following:
- vm.swappiness=25 # ~ max(RES in top)*2 / RAM = 500 MB / 2 GB
- vm.vfs_cache_pressure=1000 # safer than periodically dropping caches
Reboot.

You can verify the changes like so:

$ dmesg | grep -i noop
[    0.694680] io scheduler noop registered (default)
$ dmesg | grep -i zswap
[    0.724855] zswap: loaded using pool lzo/zbud
$ cat /sys/kernel/mm/transparent_hugepage/enabled
always [madvise] never
$ sysctl vm.swappiness
vm.swappiness = 25
$ sysctl vm.vfs_cache_pressure
vm.vfs_cache_pressure = 1000

Update

Increasing vm.min_free_kbytes in step #3 may be beneficial. Try a value of 131072 (128 MB). The final takeaway is that Linux on the desktop doesn't perform very well in low-memory situations. Some have suggested placing Chrome/Chromium in a cgroup, but that's beyond the scope of this answer.

Solution 3

I am not sure why this answer has not been suggested: killall -9 kswapd0

I came across this problem where the kswapd0 process was running as a non-root user who had not logged in for a long while. I have killed this process and the issue hasn't returned.

No, this does not address the root issue (how did it even get 100% in the first place) but allows you to quickly recover usage of the system.

Solution 4

kswap kernel are used to allocate and release memory pages, if your swap are used in you see this kernel threads using so much cpu time, that would say the kswap kernel threads are scanning the memory pages for swaping some pages and serve the memory allocation request.

I think drop the cache in this case doesn't help, because the kernel reclaim the cache when the OS is tight memory situation automatically.

If you don't have any memory problem and use the free command, you will see so much memory used as cache, but if you have a memory problem, Linux reduce the cache for serve the memory allocation requests, without any need to drop cache

you can use sar -B and looking for majft and pgscank values, for other values man sar

View more solutions

1,878

fvosberg

Updated on September 18, 2022

Comments

fvosberg almost 2 years

i'm fairly new to Google Analytics and I'm starting with the new Google Analytics 4. I've set it up via Google Tag Manager.

I have two custom events:

cta_visible (event visible)
click_meeting_link (outbound click)

When I debug my page with the https://tagassistant.google.com/, I can see both events beeing triggered.

In the debug view of Google Analytics, the cta_visible event is displayed, the click_meeting_link is missing. I thought, that it's maybe a bug, caused by the fact, that as I'm clicking the link, my browser is leaving the page.

But I can see the event cta_visible in my reports, click_meeting_link is also missing there.

In the network tab I see both events being sent to GA (with a response code of 204).

curl 'https://www.google-analytics.com/g/collect?v=2&tid=G-NKBZG0FK64&gtm=2oead0&_p=1988538019&sr=1792x1120&gcs=G100&gdid=dOThhZD&ul=en-gb&cid=1495603155.1634555573&_s=5&dl=https%3A%2F%2Finnovation.tarent.de%2Fsparring&dt=Innovation%20Sparring%20%7C%20tarent&sid=1634555572&sct=1&seg=0&en=click_meeting_link&_c=1&_et=2&ep.debug_mode=true&ep.click_url=https%3A%2F%2Fmeetings.hubspot.com%2Ffrederik-vosberg%2Finnovation-sparring' \
  -X 'POST' \
  -H 'authority: www.google-analytics.com' \
  -H 'content-length: 0' \
  -H 'pragma: no-cache' \
  -H 'cache-control: no-cache' \
  -H 'sec-ch-ua: "Chromium";v="94", "Google Chrome";v="94", ";Not A Brand";v="99"' \
  -H 'sec-ch-ua-mobile: ?0' \
  -H 'user-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/94.0.4606.81 Safari/537.36' \
  -H 'sec-ch-ua-platform: "macOS"' \
  -H 'content-type: text/plain;charset=UTF-8' \
  -H 'accept: */*' \
  -H 'origin: https://innovation.tarent.de' \
  -H 'sec-fetch-site: cross-site' \
  -H 'sec-fetch-mode: no-cors' \
  -H 'sec-fetch-dest: empty' \
  -H 'referer: https://innovation.tarent.de/' \
  -H 'accept-language: en-GB,en-US;q=0.9,en;q=0.8,de;q=0.7' \
  --compressed



  curl 'https://www.google-analytics.com/g/collect?v=2&tid=G-NKBZG0FK64&gtm=2oead0&_p=1800931673&sr=1792x1120&gcs=G100&ul=en-gb&cid=175794657.1634555667&_s=2&dl=https%3A%2F%2Finnovation.tarent.de%2Fsparring&dt=Innovation%20Sparring%20%7C%20tarent&sid=1634555666&sct=1&seg=0&en=cta_visible&_fv=1&_nsi=1&_ss=1&_eu=C&ep.debug_mode=true' \
  -X 'POST' \
  -H 'authority: www.google-analytics.com' \
  -H 'content-length: 0' \
  -H 'pragma: no-cache' \
  -H 'cache-control: no-cache' \
  -H 'sec-ch-ua: "Chromium";v="94", "Google Chrome";v="94", ";Not A Brand";v="99"' \
  -H 'sec-ch-ua-mobile: ?0' \
  -H 'user-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/94.0.4606.81 Safari/537.36' \
  -H 'sec-ch-ua-platform: "macOS"' \
  -H 'content-type: text/plain;charset=UTF-8' \
  -H 'accept: */*' \
  -H 'origin: https://innovation.tarent.de' \
  -H 'sec-fetch-site: cross-site' \
  -H 'sec-fetch-mode: no-cors' \
  -H 'sec-fetch-dest: empty' \
  -H 'referer: https://innovation.tarent.de/' \
  -H 'accept-language: en-GB,en-US;q=0.9,en;q=0.8,de;q=0.7' \
  --compressed

Any suggestions, what can cause this?

Thanks in advance

fgbreel about 9 years

take a look if irqbalance is running, and see /proc/interrupts if interruptions are well balanced.

Zaz about 9 years

I added the information to my question. Arch Linux has an AUR package for building Linux with UltraKSM, so I might try that. Thanks for the help!
Zaz about 9 years

Regarding SSDs, I think their fragility compared to HDDs is often exaggerated; even with flat out writing and erasing, it can take years to wear out an SSD. Even if I'm wrong and wear my SSD out, I'll should still be able to recover my data, and will only have lost a $30 SSD.
Zaz about 9 years

Unfortunately that has no effect, and /proc/sys/vm/drop_caches stays at 1.
Zaz about 9 years

What are sar majft and pgscank? I couldn't find them on my machine or in my package manager (pacman).
Zaz about 9 years

free shows 1.0G used, 545M cache, 425M available. Seems pretty normal to me.
poige about 9 years

ok, try changing hugepages → madvise, swappiness → 0, vfs_cache_pressure → 5000
c4f4t0r about 9 years

what did you see in sar -B?
Zaz about 9 years

zsh: command not found: sar
Zaz about 9 years

Doesn't seem to make any difference. Sorry.
poige about 9 years

Did you do 1st item in the list above?
Startec almost 8 years

This works and is very helpful, but what is it doing? I am on a very low resource machine and this issue comes up all the time (and makes it hard to use my computer over ssh) but I do not know what is going on. Scheduling this as a cron job seems like a said way to fix the glitch...
shodanshok almost 8 years

The above command drops the cached information that Linux pagecache subsystem keep to speedup I/O access. For example, when you read a file from the disks, it is also stored on the pagecache to speedup further access to file content. By dropping caches, you decrease memory pressure and this seems sufficient to avoid the kswapd problem.
josephdpurcell about 7 years

This fixed the high CPU usage for me--night and day difference! kswapd0 went from 100% CPU to 0%. An explanation for why and a permanent solution would be great. (Side note: I'm running linux kernel 4.8.0-36-generic with 16 GB mem and 16 GB swap.)
Peter Krauss over 6 years

See enhanced solution at askubuntu.com/a/736956/439867
user1089802 over 6 years

@Zaz You need to install it then.
SharkWipf over 3 years

I'm... pretty sure that's a rootkit disguising itself as a kernel process, not actual kswapd0. Pretty sure your user has been compromised. kswapd0 does not run as regular user and can't be killed like that (I hope)
Newton_Jose over 3 years

This happens exactly with me and your solution solve my problem. Using 4.19.0-9-amd64 kernel.
reukiodo over 3 years

@SharkWipf, that might be true. I've since removed that user as a precaution anyway since it was no longer needed.
djdomi almost 3 years

The question is, if you ran ever out of memory, or have to change swappiness or a less swapping factor, i.e. only swap if really not possible like value 1 or 0 . As A value of 0 instructs the kernel not to initiate swap until the amount of free and file-backed pages is less than the high water mark in a zone. Reference: kernel.org
ddm-j almost 3 years

kswapd0 tries to determine what makes more sense: dropping caches or swapping out processes. Usually, swapping out processes wins, as it gets rid of data that hasn't been accessed in ages in favor of data that has been accessed recently. If you drop caches, you communicate that it is more important to keep unused data in RAM and re-fetch data that is actually used from disk. That will show nice low CPU usage, because processes will be in iowait instead, but it's not useful behavior.