kswapd eats the cpu without swap

kswapd eats the cpu without swap

[BERECZ Szabolcs]

Hi!

kernel is 2.4.7-ac3 with the used-once patch.
I have 160M of ram, and I don't use swap at all, but some minutes before
kswapd was eating lot's of cpu (98-100%). the system did not responded for
some 10 seconds, then it worked for some seconds, then it did not responded
again, until I did a swapon. after adding the swap to the system, it
swapped out 148k then everything was ok. kswapd still eat some of the cpu
(0.5% of a k6-2/450). then swapoff, and everything is OK, again.

szabi@traktor:~# cat /proc/modules
nls_iso8859-1           2880   0 (autoclean)
sr_mod                 13408   0 (autoclean)
ide-scsi                7680   0
cdrom                  27488   0 (autoclean) [sr_mod]
scsi_mod               80112   2 (autoclean) [sr_mod ide-scsi]
isofs                  18000   0 (autoclean)
ppp_deflate            39264   0 (autoclean)
bsd_comp                4160   0 (autoclean)
ppp_async               6416   1 (autoclean)
snd-pcm-oss            36768   1 (autoclean)
snd-mixer-oss           8912   0 (autoclean) [snd-pcm-oss]
ppp_generic            14240   3 (autoclean) [ppp_deflate bsd_comp
ppp_async]
slhc                    4800   0 (autoclean) [ppp_generic]
agpgart                12704   2 (autoclean)
snd-card-ymfpci         3520   1
snd-opl3                5712   0 [snd-card-ymfpci]
snd-hwdep               3760   0 [snd-opl3]
snd-ymfpci             36272   0 [snd-card-ymfpci]
snd-ac97-codec         21712   0 [snd-ymfpci]
snd-pcm                45888   0 [snd-pcm-oss snd-ymfpci]
snd-timer               8848   0 [snd-opl3 snd-pcm]
snd-mpu401-uart         2848   0 [snd-card-ymfpci]
snd-rawmidi            11744   0 [snd-mpu401-uart]
snd-seq-device          4080   0 [snd-opl3 snd-rawmidi]
snd                    25760   0 [snd-pcm-oss snd-mixer-oss
snd-card-ymfpci snd-opl3 snd-hwdep snd-ymfpci snd-ac97-codec snd-pcm
snd-timer snd-mpu401-uart snd-rawmidi snd-seq-device]
soundcore               3984   3 [snd]
iptable_filter          2048   0 (autoclean) (unused)
ip_tables              10592   1 [iptable_filter]
reiserfs              149904   4 (autoclean)
ne2k-pci                5152   1
8390                    6176   0 [ne2k-pci]
szabi@traktor:~#

tell me what to do. anything to try? or something about the system?
I will try to reproduce it, but I don't think I will succeed.

Bye,
Szabi

Re: kswapd eats the cpu without swap

[BERECZ Szabolcs]

Hi!

sorry, I forgot something.
when kswapd was eating the cpu, there was 3m of free ram, and 19m of
buffer cache.
I thought it was about the network, but it seems not.

szabi@traktor:~# free
             total       used       free     shared    buffers     cached
Mem:        159356     155480       3876        684       1132      15464
-/+ buffers/cache:     138884      20472
Swap:            0          0          0
szabi@traktor:~#

Bye,
Szabi

Re: kswapd eats the cpu without swap

[BERECZ Szabolcs]

Hi!

some notes again.
when kswapd was working, there was no hdd activity at all.
every interrup was handled after kswapd finished the 'work'.
after a reboot everything looks ok with the same modules, and
approximately the same load.

oh, I almost forgot, the swapfile is on a reiserfs partition.

Bye,
Szabi

Re: kswapd eats the cpu without swap

[Peter Wächtler]

BERECZ Szabolcs wrote:
> 
> some notes again.
> when kswapd was working, there was no hdd activity at all.
> every interrup was handled after kswapd finished the 'work'.
> after a reboot everything looks ok with the same modules, and
> approximately the same load.
> 
> oh, I almost forgot, the swapfile is on a reiserfs partition.
> 
In a former message you wrote:
"I have 160M of ram, and I don't use swap at all,"

Then you meant: no single page was swapped out?

I thought you was observing the same as me:

when the system runs low on memory (on a 64MB setop box like device
with _no_ swap partition/file), the harddisk gets very active and
the system does not respond for 1-5 seconds.
The VM (in mm/oom_kill.c) is killing the "memory hog" (simple program 
that calls malloc() in a loop and touching the mem). I think the
amount of "busyness" depends on the size of malloc chunks. If they
are bigger the process gets killed faster.

Until now, I don't understand what is happening. Several subsystems
in the kernel compete for memory: dentry cache, buffer cache, VM
that wants at least /proc/sys/vm/freepages free.
Is demand loading involved? Does the VM quashes text pages when
running low on memory? What about relocation then?

Or simpler: what keeps the harddisk so busy?
It is on 2.4.2 with a single ext2 "/" mounted with noatime,sync
but the question is meant more general.

Anybody?

Re: kswapd eats the cpu without swap

[Helge Hafting]

Peter Wächtler wrote:

> when the system runs low on memory (on a 64MB setop box like device
> with _no_ swap partition/file), the harddisk gets very active and
> the system does not respond for 1-5 seconds.
> The VM (in mm/oom_kill.c) is killing the "memory hog" (simple program
> that calls malloc() in a loop and touching the mem). I think the
> amount of "busyness" depends on the size of malloc chunks. If they
> are bigger the process gets killed faster.
> 
> Until now, I don't understand what is happening. Several subsystems
> in the kernel compete for memory: dentry cache, buffer cache, VM
> that wants at least /proc/sys/vm/freepages free.
> Is demand loading involved? Does the VM quashes text pages when
> running low on memory? What about relocation then?
> 
> Or simpler: what keeps the harddisk so busy?

The VM system drop unwriteable pages (i.e. program code) when out of
memory.  This is the only kind of "swapping" when you don't
have a swap partition/file.  It will of course also empty the
various disk caches (page cache, buffer cache, inode/dentry caches)
so every little file operation might need several disk accesses.

Getting really low on memory with no swap device means that you
have almost no program code loaded.  Data is unswappable without
a device and fills memory, every little piece of code that needs
execution have to be re-loaded from the executables on disk.

This is why your disk get busy.  Your machine is trashing.
Trashin happens both with and without a swap device.

Having swap can help a lot - the VM subsystem will then be able to
page out old unused data leaving more room for often-used code.
You will then get a lot better performance even though you
don't have more RAM.  Of course you may run into trashing with
a swap device too - but it happens later after using much more
memory.  With luck you don't get that far with normal use.  If 
you do - get more RAM.

Helge Hafting

Re: kswapd eats the cpu without swap

[Marcelo Tosatti]

On Thu, 2 Aug 2001, BERECZ Szabolcs wrote:

> Hi!
> 
> kernel is 2.4.7-ac3 with the used-once patch.

> I have 160M of ram, and I don't use swap at all, but some minutes before
> kswapd was eating lot's of cpu (98-100%). the system did not responded for
> some 10 seconds, then it worked for some seconds, then it did not responded
> again, until I did a swapon. after adding the swap to the system, it
> swapped out 148k then everything was ok. kswapd still eat some of the cpu
> (0.5% of a k6-2/450). then swapoff, and everything is OK, again.


Does the problem happen only with the used-once patch ? 

If it also happens without the used-once patch, can you reproduce the
problem with 2.4.6 ? 

Thanks 

Re: kswapd eats the cpu without swap

[BERECZ Szabolcs]

On Fri, 3 Aug 2001, Marcelo Tosatti wrote:

> Does the problem happen only with the used-once patch ?
>
> If it also happens without the used-once patch, can you reproduce the
> problem with 2.4.6 ?
The problem happened about 4 times, with the used-once patch,
but I don't know exactly what triggered it.

now I use 2.4.7-ac5, and I have not seen the problem, yet.

I will try with the used-once patch, if it appears again.

Bye,
Szabi

Re: kswapd eats the cpu without swap

[Daniel Phillips]

On Tuesday 07 August 2001 00:48, BERECZ Szabolcs wrote:
> On Fri, 3 Aug 2001, Marcelo Tosatti wrote:
> > Does the problem happen only with the used-once patch ?
> >
> > If it also happens without the used-once patch, can you reproduce
> > the problem with 2.4.6 ?
>
> The problem happened about 4 times, with the used-once patch,
> but I don't know exactly what triggered it.
>
> now I use 2.4.7-ac5, and I have not seen the problem, yet.
>
> I will try with the used-once patch, if it appears again.

Please note the additional patch, to be applied after the used-once 
patch for 2.4.7 and 2.4.7-ac*, or directly to 2.4.8-pre*.  This was 
posted on lkml and linux-mm on Aug 5 under the subject:

    [PATCH] Unlazy activate

which adds the additional behaviour of moving used-twice pages to the 
active list.

Here it is again:

--- ../2.4.7.clean/mm/filemap.c	Sat Aug  4 14:27:16 2001
+++ ./mm/filemap.c	Sat Aug  4 23:41:00 2001
@@ -979,9 +979,13 @@
 
 static inline void check_used_once (struct page *page)
 {
-	if (!page->age) {
-		page->age = PAGE_AGE_START;
-		ClearPageReferenced(page);
+	if (!PageActive(page)) {
+		if (page->age)
+			activate_page(page);
+		else {
+			page->age = PAGE_AGE_START;
+			ClearPageReferenced(page);
+		}
 	}
 }
 

Re: kswapd eats the cpu without swap

[Marcelo Tosatti]

On Tue, 7 Aug 2001, Daniel Phillips wrote:

> On Tuesday 07 August 2001 00:48, BERECZ Szabolcs wrote:
> > On Fri, 3 Aug 2001, Marcelo Tosatti wrote:
> > > Does the problem happen only with the used-once patch ?
> > >
> > > If it also happens without the used-once patch, can you reproduce
> > > the problem with 2.4.6 ?
> >
> > The problem happened about 4 times, with the used-once patch,
> > but I don't know exactly what triggered it.
> >
> > now I use 2.4.7-ac5, and I have not seen the problem, yet.
> >
> > I will try with the used-once patch, if it appears again.
> 
> Please note the additional patch, to be applied after the used-once 
> patch for 2.4.7 and 2.4.7-ac*, or directly to 2.4.8-pre*.  This was 
> posted on lkml and linux-mm on Aug 5 under the subject:
> 
>     [PATCH] Unlazy activate
> 
> which adds the additional behaviour of moving used-twice pages to the 
> active list.


Daniel,

I would like to identify the reason why kswapd is looping like mad instead
trying _any_ fix.

I can't see why unlazy activation would make kswapd not loop like mad
anymore.

Re: kswapd eats the cpu without swap

[BERECZ Szabolcs]

On Thu, 2 Aug 2001, Peter [iso-8859-1] Wächtler wrote:

> In a former message you wrote:
> "I have 160M of ram, and I don't use swap at all,"
>
> Then you meant: no single page was swapped out?

yes. I don't attach any swap file to the system.

> I thought you was observing the same as me:
>
> when the system runs low on memory (on a 64MB setop box like device
> with _no_ swap partition/file), the harddisk gets very active and
				      ^^^^^^^^^^^^^^^^^^^^^^^^^
there was _no_ harddisk activity.
and the size of the buffer cache was 19MB.

the bug is reproducable, but I don't know how to do that :/
it showed up 3 times.

Bye,
Szabi