2.4.28 - kswapd excessive cpu usage under heavy IO

2.4.28 - kswapd excessive cpu usage under heavy IO

[Per Jessen]

(apologies if this is sent more than once)
I've found similar incidences in the archives, but none that indicates that a
solution was found. 
I'm seeing excessive cpu usage by kswapd on a 4way 500MHZ Xeon with 2GB RAM.  A
find in a directory containing perhaps 6-700,000 files makes the box almost
grind to a halt.  In 12days uptime, kswap has used 590:43.82, and during the
find-exercise usually runs with 90-100% util.
The file-system is 150GB with JFS117 on a software-RAID5 - not exactly optimal,
I agree, but reasonably workable.

I've read that 2.6 has significant improvements in this area, but upgrading is
not currently an option.  

All hints & suggestions much appreciated.

-- 
Per Jessen, Zurich
http://www.spamcek.com - let your spam stop here.

Re: 2.4.28 - kswapd excessive cpu usage under heavy IO

[Marcelo Tosatti]

Hi,

On Mon, Dec 06, 2004 at 12:14:48PM +0100, Per Jessen wrote:
> (apologies if this is sent more than once)
> I've found similar incidences in the archives, but none that indicates that a
> solution was found. 
> I'm seeing excessive cpu usage by kswapd on a 4way 500MHZ Xeon with 2GB RAM.

What is your workload? Except the massive amount of inode's/dentries in your 
system do you have anonymous memory hungry applications running? 

If you do have a significant amount of anonymous pages on your system you might
want to try 2.4.29-pre1 which contains a VM change which should decrease their 
impact on the VM page freeing efforts (including kswapd CPU usage).

Since you have massive amount of inodes/dentries you might want to tweak
/proc/sys/vm/vm_vfs_scan_ratio (increase from 6 to 10 and so on...)

>  A find in a directory containing perhaps 6-700,000 files makes the box almost
> grind to a halt.  In 12days uptime, kswap has used 590:43.82, and during the
> find-exercise usually runs with 90-100% util.

Can you boot with profile=2 and use readprofile tool to read the functions which 
are using most CPU time with

readprofile | sort -nr +2 | head -20

Do that on a 2.4.28 and 2.4.29-pre1 kernels.

> The file-system is 150GB with JFS117 on a software-RAID5 - not exactly optimal,
> I agree, but reasonably workable.
> 
> I've read that 2.6 has significant improvements in this area, but upgrading is
> not currently an option.  

Yes, v2.6 VM design is way better than v2.4's in many aspects.

Re: 2.4.28 - kswapd excessive cpu usage under heavy IO

[Per Jessen]

Marcelo Tosatti wrote:

> What is your workload? Except the massive amount of inode's/dentries in your
> system do you have anonymous memory hungry applications running?

Nope, workload is typically 6-7, with probably named being the most memory-hungry. 

> If you do have a significant amount of anonymous pages on your system you
> might want to try 2.4.29-pre1 which contains a VM change which should decrease
> their impact on the VM page freeing efforts (including kswapd CPU usage).
> 
> Since you have massive amount of inodes/dentries you might want to tweak
> /proc/sys/vm/vm_vfs_scan_ratio (increase from 6 to 10 and so on...)

OK, will try that.

>>A find in a directory containing perhaps 6-700,000 files makes the box almost
>> grind to a halt.  In 12days uptime, kswap has used 590:43.82, and during the
>> find-exercise usually runs with 90-100% util.
> 
> Can you boot with profile=2 and use readprofile tool to read the functions
> which are using most CPU time with
> 
> readprofile | sort -nr +2 | head -20
> Do that on a 2.4.28 and 2.4.29-pre1 kernels.

Will be difficult - I'll keep this in mind, but a reboot is not possible at the moment.

Many thanks for getting back to me - much appreciated!  


-- 
Per Jessen, Zurich
http://www.spamcek.com - let your spam stop here.