kswapd0 thread 100% cpu usage with recent kernels

kswapd0 thread 100% cpu usage with recent kernels

[Mike Kazantsev]

[-- Attachment #1: Type: text/plain, Size: 4099 bytes --]

Good day,

I have a strange situation with "kswapd0" thread using 100% of one
cpu core, reproducable with (maybe specific) combination of CPU and I/O
load.

Recently I've updated OS (from old gentoo linux to exherbo linux) on
fairly low-spec mini-ITX machine and noticed a strange problem.

Machine was showing higher load averages than before and the top
process with highest cpu usage seem to be quite often kswapd0 kernel
thread.

Judging from atop and top utils' output, it seem to have 100% (one core,
or close to that) cpu usage and very high disk write activity.

atop:
  PID RUID     EUID      THR   SYSCPU  USRCPU  VGROW   RGROW  RDDSK  WRDSK  ST EXC S  CPUNR  CPU CMD
   38 root     root        1    9.49s   0.00s     0K      0K     0K 83280K  --   - R      2  95% kswapd0

top:
  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
   38 root      20   0     0    0    0 R   92  0.0 291:20.79 kswapd0

iotop:
  TID  PRIO  USER     DISK READ  DISK WRITE  SWAPIN     IO>    COMMAND
   38 be/4 root        0.00 B/s    5.06 M/s  0.00 %  0.00 % [kswapd0]


Activity doesn't seem to be swapping-out though, as swap is empty and
there's still enough RAM to go around:

  MEM | tot     1.9G | free   77.6M  | cache 462.1M | dirty   0.1M | buff   10.3M |  slab  286.3M |
  SWP | tot     4.0G | free    4.0G  | vmcom   1.5G | vmlim   5.0G |

  ~# free
               total       used       free     shared    buffers     cached
  Mem:       2040412    1941784      98628          0      10580     454048
  -/+ buffers/cache:    1477156     563256
  Swap:      4194300        536    4193764

  ~# swapon -s
  Filename                                Type            Size    Used    Priority
  /dev/mapper/prime-swap                  partition       4194300 572     -1

Furthermore, these "disk writes" seem to be purely virtual, as neither
iostat nor atop show any writes to disks at the time kswapd0 works like
that.

Also, I've noticed that kswapd0 reproducibly acts like that when read
I/O activity is high (not 100% disk utilization, but about 60-70) in
the system along with some significant CPU load (about 100% of one
core), although I tend to think that latter is probably irrelevant.

And that's actually what drew my attention to it: trying to checksum a
lot of data from distributed fs (moosefs) with local chunkserver
resulted invariably in kswapd0 getting the first place in "top" output.

As I've mentioned, system in question is quite low-spec mini-ITX
platform featuring dualcore atom cpu with hyper threading and 2G of RAM
plus fairly slow sata (WD Green) disks, so "high I/O activity" here
means no more than 10 MBytes/s.

I'm observing this behavior now with 2.6.38 kernel (from kernel.org,
arch is x86_64), but I'm fairly sure I've seen this before with 2.6.37
and 2.6.37.3 releases.
Kernel .config file: http://goo.gl/Msk8I

To do something about it, I've tried to disable swap (swapoff), set
vm.swappiness to 0 (otherwise it's set to 20), lower
vm.vfs_cache_pressure to 20, raise vm.vfs_cache_pressure to 200.
None of these actions seemed to have any effect within a few minutes.
Checksumming processes I ran spanned for hours with kswapd0 being there
the whole time, while not consuming any resources before that and
becoming dormant again right after they were finished.

What can possibly result in such behavior of kswapd0?
Is there anything else I can tweak to lessen cpu usage of this thread?

I've found similar reports from much older kernels and indications that
the problem has been fixed since, but not much details to make sure if
it's the really the same problem or not.

Is there anything I can do to provide more info on the issue?

I'm afraid I'm not familiar with kernel debugging, but if anyone is
interested in finding out what's going on there and will point me to
the right tools, I'll be happy to provide any information I can get.

Will appreciate any advice or insight on that.
Thank you for your attention.


-- 
Mike Kazantsev // fraggod.net

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]