Re: 2.6.14 kswapd eating too much CPU

[Marcelo Tosatti <marcelo.tosatti@cyclades.com>]

On Sun, Nov 27, 2005 at 09:39:24PM +0100, Jan Kasprzak wrote:
> Marcelo Tosatti wrote:
> : Out of curiosity, what is the size of the inode cache from /proc/slabinfo 
> : at this moment? 
> : 
> : Because even though the traces shows kswapd trying to reclaim i-cache, the percentage
> : of i-cache is really small:
> : 
> : inode_cache         1164   1224    600    6    1 : tunables   54   27    8 : slabdata    204    204      0
> : dentry_cache       44291  48569    224   17    1 : tunables  120   60    8 : slabdata   2857   2857     51
> : 
> 	The size of icache is similar to that shown above:
> 
> # egrep '^(inode|dentry)_cache' /proc/slabinfo
> 
> inode_cache         1189   1326    600    6    1 : tunables   54   27    8 : slabdata    221    221      0
> dentry_cache       41845  45509    224   17    1 : tunables  120   60    8 : slabdata   2677   2677      0
> 
> inode_cache         1212   1326    600    6    1 : tunables   54   27    8 : slabdata    221    221      0
> dentry_cache       42662  48892    224   17    1 : tunables  120   60    8 : slabdata   2876   2876    288
> 
> 	However, the traces I have sent are traces of kswapd1, which at that
> time was eating around 50% of CPU, while kswapd0 was at >95%. I have not
> managed to get the trace of kswapd0 yet.
> 
> 	I have tried to bind the serial IRQ to CPU0 to get the trace of
> kswapd0 (echo 1 >/proc/irq/4/smp_affinity). After sysrq-p I get the dump
> of registers at CPU0, but the strange thing is, that I get the stack trace
> of kacpid instead of kswapd0 (kacpid is not even visible in top(1) output,
> and it has a total of 0 seconds of CPU time consumed since boot, while kswapd0
> is first in top(1) eating >95% of CPU). Why kacpid? When I bind the serial IRQ
> to CPU1, I get the trace of PID 0 (swapper).

Have no idea, sorry.

> 	The task that probably triggers this problem is a cron job
> doing full-text indexing of mailing list archive, so it accesses lots
> of small files, and then recreates the inverted index, which is one big
> file. So maybe inode cache shrinking or something may be the problem there.
> However, the cron job does an incremental reindexing only, so I think it
> reads less than 100 files per each run.
> : 
> : Maybe you should also try profile/oprofile during the kswapd peeks?
> : 
> 	Do you have any details on it? I can of course RTFdocs of oprofile,
> but should I try to catch something special?

Try to isolate the profile trace to the period which kswapd goes mad.

That is, reset the profiling buffer when you notice kswapd going crazy,
let it profile for a few minutes (no idea how long the load takes),
and stop before kswapd is back to idleness (preferably while its still
burning the CPU).

I think you can try the old profiler first (CONFIG_PROFILE=y, boot with
readprofile=2, use readprofile..).

If that does not give meaningful data then try oprofile.