2.6.0-test4-mm1 - kswap hogs cpu OO takes ages to start!

2.6.0-test4-mm1 - kswap hogs cpu OO takes ages to start!

[warudkar]

Trying out 2.6.0-test4-mm1. Inside KDE, I start OpenOffice.org, Rational Rose and Konsole at a time. All of these take extremely long time to startup. (approx > 5 minutes). Kswapd hogs the CPU all the time.
X becomes unusable till all of them startup, although I can telnet and run top.
Same thing run under 2.4.18 starts up in 3 minutes, X stays usable and kswapd never take more than 2% CPU.


Here a snapshot of Top output when running 2.6.0-test4-mm1:
==============================
  4:24pm  up 21:19,  6 users,  load average: 3.47, 2.04, 1.49
96 processes: 88 sleeping, 8 running, 0 zombie, 0 stopped
CPU states: 11.3% user, 88.6% system,  0.0% nice,  0.0% idle
Mem:   124632K av,  122664K used,    1968K free,       0K shrd,     160K buff
Swap: 1052248K av,   71256K used,  980992K free                   43540K cached

  PID USER     PRI  NI  SIZE  RSS SHARE STAT %CPU %MEM   TIME COMMAND
    8 root      15   0     0    0     0 SW   66.6  0.0   4:59 kswapd0
 2087 wipro     17   0 29576  10M 27408 S     4.6  8.9   0:03 kdeinit
 2044 wipro     17   0  118M  22M  103M D     2.7 18.8   0:04 soffice.bin
 2292 wipro     18   0  3856  912  3716 S     1.8  0.7   0:02 top
 2312 wipro     18   0  3852  912  3716 R     1.8  0.7   0:00 top
  540 wipro     15   0 28988 6640 26672 S     0.9  5.3  10:40 kdeinit
 1254 wipro     15   0 26340 4916 24716 S     0.9  3.9   0:17 kdeinit
    1 root      17   0  1320  268  1288 S     0.0  0.2   0:04 init
    2 root      0K   0     0    0     0 SW    0.0  0.0   0:00 migration/0
    3 root      34  19     0    0     0 SWN   0.0  0.0   0:00 ksoftirqd/0
    4 root       5 -10     0    0     0 SW<   0.0  0.0   0:00 events/0
    5 root       5 -10     0    0     0 SW<   0.0  0.0   0:01 kblockd/0
    6 root      15   0     0    0     0 SW    0.0  0.0   0:00 pdflush
    7 root      15   0     0    0     0 SW    0.0  0.0   0:00 pdflush
    9 root      10 -10     0    0     0 SW<   0.0  0.0   0:00 aio/0
   10 root      10 -10     0    0     0 SW<   0.0  0.0   0:00 aio_fput/0
   11 root      15   0     0    0     0 SW    0.0  0.0   0:00 kseriod
   12 root      15   0     0    0     0 SW    0.0  0.0   0:00 kjournald

Re: 2.6.0-test4-mm1 - kswap hogs cpu OO takes ages to start!

[William Lee Irwin III]

On Wed, Aug 27, 2003 at 04:38:44PM -0500, warudkar@vsnl.net wrote:
> Here a snapshot of Top output when running 2.6.0-test4-mm1:
> ==============================
>   4:24pm  up 21:19,  6 users,  load average: 3.47, 2.04, 1.49
> 96 processes: 88 sleeping, 8 running, 0 zombie, 0 stopped
> CPU states: 11.3% user, 88.6% system,  0.0% nice,  0.0% idle
> Mem:   124632K av,  122664K used,    1968K free,       0K shrd,     160K buff
> Swap: 1052248K av,   71256K used,  980992K free                   43540K cached
> 
>   PID USER     PRI  NI  SIZE  RSS SHARE STAT %CPU %MEM   TIME COMMAND
>     8 root      15   0     0    0     0 SW   66.6  0.0   4:59 kswapd0
>  2087 wipro     17   0 29576  10M 27408 S     4.6  8.9   0:03 kdeinit
>  2044 wipro     17   0  118M  22M  103M D     2.7 18.8   0:04 soffice.bin

This looks odd. At any rate, the 71MB swap should be a big hint you're
in for slow times. 2.4.x likely does okay here because the virtualscan
acts as implicit load control; 2.6.x will likely need explicit load
control to handle that workload on that machine.


-- wli

Re: 2.6.0-test4-mm1 - kswap hogs cpu OO takes ages to start!

[William Lee Irwin III]

On Wed, Aug 27, 2003 at 04:38:44PM -0500, warudkar@vsnl.net wrote:
> Trying out 2.6.0-test4-mm1. Inside KDE, I start OpenOffice.org,
> Rational Rose and Konsole at a time. All of these take extremely long
> time to startup. (approx > 5 minutes). Kswapd hogs the CPU all the time.
> X becomes unusable till all of them startup, although I can telnet
> and run top. Same thing run under 2.4.18 starts up in 3 minutes, X
> stays usable and kswapd never take more than 2% CPU.

Also, can you capture vmstat 1 with this?


-- wli

Re: 2.6.0-test4-mm1 - kswap hogs cpu OO takes ages to start!

[Con Kolivas]

On Thu, 28 Aug 2003 07:38, warudkar@vsnl.net wrote:
> Trying out 2.6.0-test4-mm1. Inside KDE, I start OpenOffice.org, Rational
> Rose and Konsole at a time. All of these take extremely long time to
> startup. (approx > 5 minutes). Kswapd hogs the CPU all the time. X becomes
> unusable till all of them startup, although I can telnet and run top. Same
> thing run under 2.4.18 starts up in 3 minutes, X stays usable and kswapd
> never take more than 2% CPU.

Yes I can reproduce this with a memory heavy load as well on low memory 
(linking at the end of a big kernel compile is standard problem). I actually 
found the best workaround was to increase the swappiness instead of 
decreasing it.

Try 
echo 100 > /proc/sys/vm/swappiness

time it

then try
echo 0 > /proc/sys/vm/swappiness

you'll see that at low swappiness kswapd0 can use ridiculous amounts of cpu 
trying to avoid swap. The default is 60.

Con

Re: 2.6.0-test4-mm1 - kswap hogs cpu OO takes ages to start!

[Andrew Morton]

Con Kolivas <kernel@kolivas.org> wrote:
>
> On Thu, 28 Aug 2003 07:38, warudkar@vsnl.net wrote:
> > Trying out 2.6.0-test4-mm1. Inside KDE, I start OpenOffice.org, Rational
> > Rose and Konsole at a time. All of these take extremely long time to
> > startup. (approx > 5 minutes). Kswapd hogs the CPU all the time. X becomes
> > unusable till all of them startup, although I can telnet and run top. Same
> > thing run under 2.4.18 starts up in 3 minutes, X stays usable and kswapd
> > never take more than 2% CPU.
> 
> Yes I can reproduce this with a memory heavy load as well on low memory 
> (linking at the end of a big kernel compile is standard problem).

It could be that recent changes to page reclaim which improve I/O
scheduling have exacerbated this.

Does this make a difference?

diff -puN mm/vmscan.c~a mm/vmscan.c
--- 25/mm/vmscan.c~a	Wed Aug 27 12:51:36 2003
+++ 25-akpm/mm/vmscan.c	Wed Aug 27 12:51:48 2003
@@ -360,8 +360,6 @@ shrink_list(struct list_head *page_list,
 		 * See swapfile.c:page_queue_congested().
 		 */
 		if (PageDirty(page)) {
-			if (referenced)
-				goto keep_locked;
 			if (!is_page_cache_freeable(page))
 				goto keep_locked;
 			if (!mapping)

_

Re: 2.6.0-test4-mm1 - kswap hogs cpu OO takes ages to start!

[Con Kolivas]

On Thu, 28 Aug 2003 05:53, Andrew Morton wrote:
> Con Kolivas <kernel@kolivas.org> wrote:
> > On Thu, 28 Aug 2003 07:38, warudkar@vsnl.net wrote:
> > > Trying out 2.6.0-test4-mm1. Inside KDE, I start OpenOffice.org,
> > > Rational Rose and Konsole at a time. All of these take extremely long
> > > time to startup. (approx > 5 minutes). Kswapd hogs the CPU all the
> > > time. X becomes unusable till all of them startup, although I can
> > > telnet and run top. Same thing run under 2.4.18 starts up in 3 minutes,
> > > X stays usable and kswapd never take more than 2% CPU.
> >
> > Yes I can reproduce this with a memory heavy load as well on low memory
> > (linking at the end of a big kernel compile is standard problem).
>
> It could be that recent changes to page reclaim which improve I/O
> scheduling have exacerbated this.
>
> Does this make a difference?

Tried it. No change. 

kswapd0 can hit 90% cpu at times unless the swappiness is increased.

Con

Re: 2.6.0-test4-mm1 - kswap hogs cpu OO takes ages to start!

[Rik van Riel]

On Thu, 28 Aug 2003, Con Kolivas wrote:

> > Does this make a difference?
> 
> Tried it. No change. 
> 
> kswapd0 can hit 90% cpu at times unless the swappiness is increased.

Looks like the problem is that cache and process pages are on
the same lists, forcing kswapd to scan the lists endlessly.

One thing you could try is splitting the lists, at least the
active list, like done in 2.4-rmap15...

-- 
"Debugging is twice as hard as writing the code in the first place.
Therefore, if you write the code as cleverly as possible, you are,
by definition, not smart enough to debug it." - Brian W. Kernighan

Re: 2.6.0-test4-mm1 - kswap hogs cpu OO takes ages to start!

[Tariq Firoz]

I faced a similar problem with 2.6.0-test4-mm2 while using unstable kde
3.1.9x(?)
In fact I was not able to do anything and my system became unresponsive,
with kswapd0
hogging 60% CPU. Later X got killed ... I did not face such a situation in
any of the previous
releases (I have been using 2.5 since 2.5.70 )

[ I have 256 MB RAM and 256 SWAP with P4 2.0 GHz]

Tariq Firoz


----- Original Message -----
From: "Andrew Morton" <akpm@osdl.org>
To: "Con Kolivas" <kernel@kolivas.org>
Cc: <warudkar@vsnl.net>; <linux-kernel@vger.kernel.org>
Sent: Thursday, August 28, 2003 1:23 AM
Subject: Re: 2.6.0-test4-mm1 - kswap hogs cpu OO takes ages to start!


> Con Kolivas <kernel@kolivas.org> wrote:
> >
> > On Thu, 28 Aug 2003 07:38, warudkar@vsnl.net wrote:
> > > Trying out 2.6.0-test4-mm1. Inside KDE, I start OpenOffice.org,
Rational
> > > Rose and Konsole at a time. All of these take extremely long time to
> > > startup. (approx > 5 minutes). Kswapd hogs the CPU all the time. X
becomes
> > > unusable till all of them startup, although I can telnet and run top.
Same
> > > thing run under 2.4.18 starts up in 3 minutes, X stays usable and
kswapd
> > > never take more than 2% CPU.
> >
> > Yes I can reproduce this with a memory heavy load as well on low memory
> > (linking at the end of a big kernel compile is standard problem).
>
> It could be that recent changes to page reclaim which improve I/O
> scheduling have exacerbated this.
>
> Does this make a difference?
>
> diff -puN mm/vmscan.c~a mm/vmscan.c
> --- 25/mm/vmscan.c~a Wed Aug 27 12:51:36 2003
> +++ 25-akpm/mm/vmscan.c Wed Aug 27 12:51:48 2003
> @@ -360,8 +360,6 @@ shrink_list(struct list_head *page_list,
>   * See swapfile.c:page_queue_congested().
>   */
>   if (PageDirty(page)) {
> - if (referenced)
> - goto keep_locked;
>   if (!is_page_cache_freeable(page))
>   goto keep_locked;
>   if (!mapping)
>
> _
>
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
>

Re: 2.6.0-test4-mm1 - kswap hogs cpu OO takes ages to start!

[Jose Luis Domingo Lopez]

On Wednesday, 27 August 2003, at 16:38:44 -0500,
warudkar@vsnl.net wrote:

> Same thing run under 2.4.18 starts up in 3 minutes, X stays usable and kswapd never take more than 2% CPU.
> 
OpenOffice is know to have severe performance problems with kernels
2.5.x and 2.6.x kernels in the presence of any CPU hog in the box. It is
a known problem that seems to have been corrected in OO 1.1.0-RC3. Note
I am talking about the symptom, not about the cause of your overall
system slowness.

Regards,

-- 
Jose Luis Domingo Lopez
Linux Registered User #189436     Debian Linux Sid (Linux 2.6.0-test4-mm1)