Linux – kswapd often uses 100% CPU when swap is in use

cpu-usagelinuxlinux-kernelswap

Most of the time when my computer starts to need swap, I see a massive spike in CPU usage (kswapd0 is consistently using 99%-100% CPU). According to top, the time is spent in sy (system/kernel) not wa (IO wait).

I am running Linux 4.0.4-2-ARCH on a C720 with 2GB RAM, and 6GB swap on an SSD.

I seem to have this problem with or without discard pages (TRIM) turned on.

Are there any setting I should inspect or tweak to see if I can fix this?

Is there any way to debug the problem? Something like strace for kernel threads?

Running with default Arch Linux settings:

/proc/sys/vm/swappiness = 60
/proc/sys/vm/vfs_cache_pressure = 100
/sys/kernel/mm/transparent_hugepage/enabled = [always] madvise never

Best Answer

It seems a relatively common problem

When the problem is happening, can you check if issuing the following command stops it: echo 1 > /proc/sys/vm/drop_caches

If it works, you can schedule it as a periodic cron job as a workaround.

Related Solutions

Linux – How to tell what process is causing kswapd to be in use

kswapd is managing swap space in response to memory demands greater than physically available for all processes.

It is process agnostic, it is only interested in what pages are access and when (it is more complex than this of course but to keep things simple we may as well view it this way).

So the real question is "what processes have the greatest burden on memory that are causing kswapd to need to page all the time".

That is most easily answered using 'top' and switching to memory usage sort mode.

Linux – why is kswapd using high CPU on an idle system

AFAICS this is neither related to free RAM nor SWAP. We have the same problem here which sometimes hits production machines and there is plenty of RAM free, quite often more than 700 MB with no dirty buffers to sync and 0 bytes SWAP used. It definitively looks like a severe Kernel BUG due to some unknown race condition.

Currently we run CentOS Kernel 2.6.18-194.el5 and will try to replace it by some newer kernel, because we think, this might help.

Update:

RedHat had confirmed that it is a kernel issue for 2.6.18-194.el5

Solutions:

Minimum: kernel-2.6.18-194.32.1.el5 contains the immediate bugfix
Better: kernel-2.6.18-238.el5 contains additional kswapd-related bugfixes
Best: kernel-2.6.18-348.4.1.el5 latest kernel which runs with RHEL 5.5 without change

In the meanwhile there is a script, which is able to detect the 100% CPU situation quite well. It is called by our monitoring each minute to inform us about the situation. If the situation stays for too long, affected machines would lock up completely due to more and more unkillable processes using 100% CPU, until the machine becomes completely unmanageable.

Currently the only way known to solve the problem is to manually hard reboot the affected machine. /sbin/reboot fails, because the machine hangs on shutdown quite too often.

To hard-reboot a machine from any root shell commandline without direct access to Console do:

echo 10 > /proc/sys/kernel/panic
echo 1 > /proc/sys/kernel/sysrq
echo s > /proc/sysrq-trigger
sleep 5
echo s > /proc/sysrq-trigger
sleep 1
echo b > /proc/sysrq-trigger

Keep in mind, do this after quiescing the machine, such that there is no more process writing to the disks. This shall prevent that fsck runs in severe trouble after reboot.

Sorry, no real solution, but HTH. And keep in mind, perhaps there might be other things which cause a 100% CPU situation on kswapd than described here. So automating a reboot in this case perhaps is a bad idea.

Best Answer

Related Solutions

Linux – How to tell what process is causing kswapd to be in use

Linux – why is kswapd using high CPU on an idle system

Related Topic