kswapd using all cpu for long periods in 2.4.9-pre4

kswapd using all cpu for long periods in 2.4.9-pre4

[Nick Piggin]

Hi,
Somewhere between 2.4.8 and 2.4.9-pre4 kswapd developed a problem. Actually
it looks like it is related to the changes in do_try_to_free_pages...

kswapd is using all cpu for long periods (100-200 seconds then 100-200
seconds break....) there is very little disk activity (heres a vmstat while
its happening)

0 0 1 18372 1872 1628 38540 1 0 4 2 343 127 1 96 3
0 0 1 18360 1992 1628 38416 0 0 0 3 282 120 0 100 0
0 0 1 18312 2188 1632 38244 1 0 31 6 376 147 1 89 10
0 0 1 18324 1908 1632 38252 0 0 0 3 411 130 0 99 0
0 0 1 18324 1996 1632 38260 0 0 1 1 175 115 0 100 0
0 0 1 18324 1852 1632 38336 17 0 36 8 150 163 0 100 0
0 0 1 18324 2028 1632 38336 0 0 0 8 107 119 0 100 0

kswapd task:
Trace; c0145328 <shrink_icache_memory+18/30>
Trace; c012b850 <do_try_to_free_pages+60/c0>
Trace; c012b90e <kswapd+5e/c0>
Trace; c0105000 <_stext+0/0>
Trace; c0105000 <_stext+0/0>
Trace; c01054d6 <kernel_thread+26/30>
Trace; c012b8b0 <kswapd+0/c0>

meminfo just before kswapd goes crazy:
Mem-info:
Free pages: 4616kB ( 0kB HighMem)
( Active: 6707, inactive_dirty: 4312, inactive_clean: 895, free: 1154 (256
512 2048) )
5*4kB 1*8kB 1*16kB 1*32kB 1*64kB 1*128kB 1*256kB 1*512kB 0*1024kB 0*2048kB =
1036kB)
483*4kB 104*8kB 5*16kB 1*32kB 1*64kB 1*128kB 0*256kB 1*512kB 0*1024kB
0*2048kB = 3580kB)
= 0kB)
Swap cache: add 16566, delete 14047, find 17323/32269
Free swap: 110016kB
16368 pages of RAM
0 pages of HIGHMEM
689 reserved pages
10205 pages shared
2519 pages swap cached
0 pages in page table cache
Buffer memory: 1472kB

mem info during:
SysRq: Show Memory
Mem-info:
Free pages: 3744kB ( 0kB HighMem)
( Active: 6946, inactive_dirty: 4296, inactive_clean: 895, free: 936 (256
512 2048) )
5*4kB 1*8kB 1*16kB 1*32kB 1*64kB 1*128kB 1*256kB 1*512kB 0*1024kB 0*2048kB =
1036kB)
263*4kB 105*8kB 5*16kB 1*32kB 1*64kB 1*128kB 0*256kB 1*512kB 0*1024kB
0*2048kB = 2708kB)
= 0kB)
Swap cache: add 16680, delete 14047, find 17408/32591
Free swap: 110016kB
16368 pages of RAM
0 pages of HIGHMEM
689 reserved pages
10453 pages shared
2633 pages swap cached
0 pages in page table cache
Buffer memory: 1480kB

Re: kswapd using all cpu for long periods in 2.4.9-pre4

[Linus Torvalds]

In article <000c01c12569$65581630$0200a8c0@W2K> you write:
>
>Somewhere between 2.4.8 and 2.4.9-pre4 kswapd developed a problem. Actually
>it looks like it is related to the changes in do_try_to_free_pages...
>
>kswapd is using all cpu for long periods (100-200 seconds then 100-200
>seconds break....) there is very little disk activity (heres a vmstat while
>its happening)

For some reason do_try_to_free_pages() doesn't end up ever being happy
with how much memory there is free (that's why kswapd will loop).  But
you actually have a reasonable amount of free memory, so all the memory
free'ers will actually refuse to do anything.  The problem seems to be
that either inactive_shortage() or free_shortage() will just continually
claim that you have a shortage of memory. 

However, you actually do have memory:

>mem info during:
>SysRq: Show Memory
>Mem-info:
>Free pages: 3744kB ( 0kB HighMem)
>( Active: 6946, inactive_dirty: 4296, inactive_clean: 895, free: 936 (256
>512 2048) )
>5*4kB 1*8kB 1*16kB 1*32kB 1*64kB 1*128kB 1*256kB 1*512kB 0*1024kB 0*2048kB = 1036kB)
>263*4kB 105*8kB 5*16kB 1*32kB 1*64kB 1*128kB 0*256kB 1*512kB 0*1024kB 0*2048kB = 2708kB)
> = 0kB)
>Swap cache: add 16680, delete 14047, find 17408/32591
>Free swap: 110016kB
>16368 pages of RAM
>0 pages of HIGHMEM

I bet this is it.

It probably decides that you have a shortage of HIGHMEM pages. Which is
understandable, since you don't have any ;)

Does the following trivial patch fix it for you?

(If it doesn't, please try just removing the "continue" at line 930 or
so in the middle of the kswapd loop.  But I think there's something else
that makes kswapd just think it _should_ loop, and the lack of
zone->size testing in the inactive_shortage() calculations looks like
it).

	Thanks,

		Linus

-----
--- pre4/linux/mm/vmscan.c	Wed Aug 15 02:39:44 2001
+++ linux/mm/vmscan.c	Wed Aug 15 02:37:07 2001
@@ -788,6 +788,9 @@
 			zone_t *zone = pgdat->node_zones + i;
 			unsigned int inactive;
 
+			if (!zone->size)
+				continue;
+
 			inactive  = zone->inactive_dirty_pages;
 			inactive += zone->inactive_clean_pages;
 			inactive += zone->free_pages;

Re: kswapd using all cpu for long periods in 2.4.9-pre4

[Linus Torvalds]

I just noticed something..

In article <000c01c12569$65581630$0200a8c0@W2K> you write:
>
>kswapd is using all cpu for long periods (100-200 seconds then 100-200
>seconds break....) there is very little disk activity (heres a vmstat while
>its happening)
>
>mem info during:
>SysRq: Show Memory
>Mem-info:
>Free pages: 3744kB ( 0kB HighMem)
>( Active: 6946, inactive_dirty: 4296, inactive_clean: 895, free: 936 (256
>512 2048) )

This shows your freepages.min low and high: 256. 512 and 2048
respectively.

And that's WRONG. Your "freepages.high" is _way_ too high. You must have
changed it through the /proc interface somehow, which should be
impossible (the /proc/sys/vm/freepages file is read-only), but I suspect
that maybe there's a sysctl backdoor.

How do I know that "freepages.high" is too big? Because it should always
be 3*freepages.min. Yours is 8*min.

And why does this matter? Because what has apparently happened is:
 - your setup somehow modifies the global free target
 - but the per-zone targets are still the old ones

The end result: we always think that we are under a global shortage, but
the actual page freeing routines refuse to free pages because all the
_zones_ think that they are fine.

Which gives you _exactly_ the behaviour you're describing. "kswapd" will
run forever, and never make any progress at all.

I wonder if this is more common? Does any of the distributions maybe
have some "system tuning" script that modifies the freepages values
through sysctl? That would certainly explain why some people see the
problem and others do not..

How DID you change that value?

		Linus

Re: kswapd using all cpu for long periods in 2.4.9-pre4

[Nick Piggin]

Yes you're right, my fault sorry. Leaving it as default does fix the
problem. The values were changed using sysctl(8). It writes straight to
/proc - my freepages is not readonly for some reason (source has been
changed)... anyway, no sysctl(2) backdoors found here.

On related topic - is /Documentation/sysctl/vm.txt up-to-date? Is there a
better document to use?

Nick

----- Original Message -----
From: "Linus Torvalds" <torvalds@transmeta.com>
Newsgroups: linux.dev.kernel
To: <s3293115@student.anu.edu.au>; <linux-kernel@vger.kernel.org>; "Marcelo
Tosatti" <marcelo@conectiva.com.br>; "Alan Cox" <alan@lxorguk.ukuu.org.uk>
Sent: Thursday, August 16, 2001 3:19 AM
Subject: Re: kswapd using all cpu for long periods in 2.4.9-pre4


> I just noticed something..
>
> In article <000c01c12569$65581630$0200a8c0@W2K> you write:
> >
> >kswapd is using all cpu for long periods (100-200 seconds then 100-200
> >seconds break....) there is very little disk activity (heres a vmstat
while
> >its happening)
> >
> >mem info during:
> >SysRq: Show Memory
> >Mem-info:
> >Free pages: 3744kB ( 0kB HighMem)
> >( Active: 6946, inactive_dirty: 4296, inactive_clean: 895, free: 936 (256
> >512 2048) )
>
> This shows your freepages.min low and high: 256. 512 and 2048
> respectively.
>
> And that's WRONG. Your "freepages.high" is _way_ too high. You must have
> changed it through the /proc interface somehow, which should be
> impossible (the /proc/sys/vm/freepages file is read-only), but I suspect
> that maybe there's a sysctl backdoor.
>
> How do I know that "freepages.high" is too big? Because it should always
> be 3*freepages.min. Yours is 8*min.
>
> And why does this matter? Because what has apparently happened is:
>  - your setup somehow modifies the global free target
>  - but the per-zone targets are still the old ones
>
> The end result: we always think that we are under a global shortage, but
> the actual page freeing routines refuse to free pages because all the
> _zones_ think that they are fine.
>
> Which gives you _exactly_ the behaviour you're describing. "kswapd" will
> run forever, and never make any progress at all.
>
> I wonder if this is more common? Does any of the distributions maybe
> have some "system tuning" script that modifies the freepages values
> through sysctl? That would certainly explain why some people see the
> problem and others do not..
>
> How DID you change that value?
>
> Linus
>