* kswapd CPU usage and heavy disk IO
@ 2003-01-09 13:31 Russell Coker
2003-01-09 16:02 ` Brian Tinsley
2003-01-09 16:56 ` Oleg Drokin
0 siblings, 2 replies; 7+ messages in thread
From: Russell Coker @ 2003-01-09 13:31 UTC (permalink / raw)
To: ReiserFS; +Cc: Rik van Riel
I have a server with 4G of RAM running ReiserFS for everything that matters.
It has 2G of swap space free, but so far I have not seen swap usage go above
1.6M (so in normal use I could turn off swap entirely and expect not to see
much difference).
When it's under really heavy load (when I have a maintenance task involving a
"find /" and there are lots of POP/IMAP clients hitting the server as well as
mail delivery) and the load average gets to about 40, the "kswapd" kernel
thread starts using excessive CPU time. It will stay on ~4% but have spikes
of up to 45%!!! This is a two-processor machine so 45% CPU reported by top
means 90% of a single CPU I guess. 90% of a 1.8GHz P4 CPU is a lot of CPU
and I think that something is wrong.
In the meager documentation in the kernel source kswapd is described as being
involved in paging to disk. I don't think that this is what it is doing as
there is no noticable paging activity (it generally has at least 600M of
"buffers" so there is no real shortage of memory).
Would the activity of kswapd be involved with ReiserFS in any way? What can I
do to improve this situation?
--
http://www.coker.com.au/selinux/ My NSA Security Enhanced Linux packages
http://www.coker.com.au/bonnie++/ Bonnie++ hard drive benchmark
http://www.coker.com.au/postal/ Postal SMTP/POP benchmark
http://www.coker.com.au/~russell/ My home page
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: kswapd CPU usage and heavy disk IO
2003-01-09 13:31 kswapd CPU usage and heavy disk IO Russell Coker
@ 2003-01-09 16:02 ` Brian Tinsley
2003-01-09 16:42 ` Dieter Nützel
2003-01-09 16:56 ` Oleg Drokin
1 sibling, 1 reply; 7+ messages in thread
From: Brian Tinsley @ 2003-01-09 16:02 UTC (permalink / raw)
To: Russell Coker; +Cc: ReiserFS, Rik van Riel
I've been seeing the exact same thing on the same type of system in the
same situations. This has been causing all kinds of problems on our
clusters: the system live-locks for a minute or two, causes cluster
heartbeats to not be received, and falsely fails over when the system
recovers from the live-lock. The only thing I can find after the
live-lock is that the runtime for kswapd is abnormally high.
We started running sar (60 second collection interval) and were able to
capture some stats during this live-lock period. I've snipped some I
believe may be of interest. Note the missing stats between 03:59:43 and
04:02:03
Oh BTW, this is on a stock 2.4.20 kernel (dual P3, 4GB), but I have seen
the same behavior on 2.4.19 and 2.4.17.
1. sar -f sa09 -r
03:53:43 AM kbmemfree kbmemused %memused kbmemshrd kbbuffers kbcached
kbswpfree kbswpused %swpused
03:54:43 AM 411888 3888264 90.42 0 629520
2666968 209713
6 0 0.00
03:55:43 AM 396684 3903468 90.78 0 658656
2667160 209713
6 0 0.00
03:56:43 AM 331360 3968792 92.29 0 675008
2733476 209713
6 0 0.00
03:57:43 AM 231588 4068564 94.61 0 683680
2832816 209713
6 0 0.00
03:58:43 AM 209740 4090412 95.12 0 702148
2854332 209713
6 0 0.00
03:59:43 AM 211016 4089136 95.09 0 712580
2854508 209713
6 0 0.00
04:02:03 AM 207828 4092324 95.17 0 715180
2854596 209713
6 0 0.00
04:04:30 AM 2581956 1718196 39.96 0 662320
874536 209713
6 0 0.00
04:05:30 AM 4013000 287152 6.68 0 27012
84084 209713
6 0 0.00
2. sar -f sa09 -R
03:53:43 AM frmpg/s shmpg/s bufpg/s campg/s
03:54:43 AM -263.02 0.00 91.67 299.50
03:55:43 AM -63.35 0.00 121.40 0.80
03:56:43 AM -272.18 0.00 68.13 276.32
03:57:43 AM -415.72 0.00 36.13 413.92
03:58:43 AM -91.03 0.00 76.95 89.65
03:59:43 AM 5.32 0.00 43.47 0.73
04:02:03 AM -4.74 0.00 3.86 0.13
04:04:30 AM 5013.36 0.00 -111.62 -4181.22
04:05:30 AM 5962.68 0.00 -2647.12 -3293.55
04:06:30 AM -8.10 0.00 0.02 6.50
3. sar -f sa09 -b
03:53:43 AM tps rtps wtps bread/s bwrtn/s
03:54:43 AM 161.52 156.32 5.20 3156.67 119.60
03:55:43 AM 148.37 129.35 19.02 1034.80 377.33
03:56:43 AM 146.32 128.48 17.83 2732.80 360.40
03:57:43 AM 107.32 84.62 22.70 3743.60 447.07
03:58:43 AM 91.73 82.03 9.70 1312.40 194.80
03:59:43 AM 75.62 54.22 21.40 433.73 350.00
04:02:03 AM 4.97 4.83 0.14 38.65 1.24
04:04:30 AM 82.68 9.44 73.24 78.45 958.39
04:05:30 AM 2.93 0.00 2.93 0.00 29.33
04:06:30 AM 0.22 0.00 0.22 0.00 1.73
4. sar -f sa09 -i
03:53:43 AM dentunusd file-sz %file-sz inode-sz super-sz %super-sz
dquot-sz %dquot-sz rtsig-sz %rtsig-sz
03:54:43 AM 57361 134 0.01 61318 0 0.00
0 0.00 0 0.00
03:55:43 AM 58318 124 0.01 62006 0 0.00
0 0.00 0 0.00
03:56:43 AM 44384 135 0.01 47145 0 0.00
0 0.00 0 0.00
03:57:43 AM 42565 135 0.01 45983 0 0.00
0 0.00 0 0.00
03:58:43 AM 18901 134 0.01 22408 0 0.00
0 0.00 0 0.00
03:59:43 AM 607 135 0.01 1173 0 0.00
0 0.00 0 0.00
04:02:03 AM 4294967295 113 0.01 417 0
0.00 0 0.00 4 0.39
04:04:30 AM 49 247 0.02 6316 0 0.00
0 0.00 0 0.00
04:05:30 AM 121 311 0.03 365 0 0.00
0 0.00 0 0.00
5. sar -f sa09 -u
03:53:43 AM CPU %user %nice %system %idle
03:54:43 AM all 7.52 0.00 25.15 67.33
03:55:43 AM all 8.97 0.00 25.28 65.75
03:56:43 AM all 6.07 0.00 23.82 70.11
03:57:43 AM all 5.08 0.00 23.54 71.38
03:58:43 AM all 6.77 0.00 22.88 70.36
03:59:43 AM all 7.18 0.00 25.82 67.00
04:02:03 AM all 0.77 0.00 96.32 2.91
04:04:30 AM all 4.20 0.00 95.11 0.69
04:05:30 AM all 1.88 0.00 5.29 92.83
04:06:30 AM all 2.01 0.00 2.81 95.18
Russell Coker wrote:
>I have a server with 4G of RAM running ReiserFS for everything that matters.
>
>It has 2G of swap space free, but so far I have not seen swap usage go above
>1.6M (so in normal use I could turn off swap entirely and expect not to see
>much difference).
>
>When it's under really heavy load (when I have a maintenance task involving a
>"find /" and there are lots of POP/IMAP clients hitting the server as well as
>mail delivery) and the load average gets to about 40, the "kswapd" kernel
>thread starts using excessive CPU time. It will stay on ~4% but have spikes
>of up to 45%!!! This is a two-processor machine so 45% CPU reported by top
>means 90% of a single CPU I guess. 90% of a 1.8GHz P4 CPU is a lot of CPU
>and I think that something is wrong.
>
>In the meager documentation in the kernel source kswapd is described as being
>involved in paging to disk. I don't think that this is what it is doing as
>there is no noticable paging activity (it generally has at least 600M of
>"buffers" so there is no real shortage of memory).
>
>Would the activity of kswapd be involved with ReiserFS in any way? What can I
>do to improve this situation?
>
>
>
--
-[========================]-
-[ Brian Tinsley ]-
-[ Chief Systems Engineer ]-
-[ Emageon ]-
-[========================]-
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: kswapd CPU usage and heavy disk IO
2003-01-09 16:02 ` Brian Tinsley
@ 2003-01-09 16:42 ` Dieter Nützel
2003-01-09 16:48 ` Brian Tinsley
` (2 more replies)
0 siblings, 3 replies; 7+ messages in thread
From: Dieter Nützel @ 2003-01-09 16:42 UTC (permalink / raw)
To: Brian Tinsley, Russell Coker
Cc: ReiserFS, Rik van Riel, Andrea Arcangeli, Linux Kernel List
Am Donnerstag, 9. Januar 2003 17:02 schrieb Brian Tinsley:
> I've been seeing the exact same thing on the same type of system in the
> same situations. This has been causing all kinds of problems on our
> clusters: the system live-locks for a minute or two, causes cluster
> heartbeats to not be received, and falsely fails over when the system
> recovers from the live-lock. The only thing I can find after the
> live-lock is that the runtime for kswapd is abnormally high.
>
> We started running sar (60 second collection interval) and were able to
> capture some stats during this live-lock period. I've snipped some I
> believe may be of interest. Note the missing stats between 03:59:43 and
> 04:02:03
>
> Oh BTW, this is on a stock 2.4.20 kernel (dual P3, 4GB), but I have seen
> the same behavior on 2.4.19 and 2.4.17.
I think you should have cc'ed Andrea Arcangeli <andrea@suse.de>, LKM and try
2.4.20-aa1.
Are you sure it is a ReiserFS and not a kernel thing?
Regards,
Dieter
> 1. sar -f sa09 -r
>
> 03:53:43 AM kbmemfree kbmemused %memused kbmemshrd kbbuffers kbcached
> kbswpfree kbswpused %swpused
> 03:54:43 AM 411888 3888264 90.42 0 629520
> 2666968 209713
> 6 0 0.00
> 03:55:43 AM 396684 3903468 90.78 0 658656
> 2667160 209713
> 6 0 0.00
> 03:56:43 AM 331360 3968792 92.29 0 675008
> 2733476 209713
> 6 0 0.00
> 03:57:43 AM 231588 4068564 94.61 0 683680
> 2832816 209713
> 6 0 0.00
> 03:58:43 AM 209740 4090412 95.12 0 702148
> 2854332 209713
> 6 0 0.00
> 03:59:43 AM 211016 4089136 95.09 0 712580
> 2854508 209713
> 6 0 0.00
> 04:02:03 AM 207828 4092324 95.17 0 715180
> 2854596 209713
> 6 0 0.00
> 04:04:30 AM 2581956 1718196 39.96 0 662320
> 874536 209713
> 6 0 0.00
> 04:05:30 AM 4013000 287152 6.68 0 27012
> 84084 209713
> 6 0 0.00
>
> 2. sar -f sa09 -R
>
> 03:53:43 AM frmpg/s shmpg/s bufpg/s campg/s
> 03:54:43 AM -263.02 0.00 91.67 299.50
> 03:55:43 AM -63.35 0.00 121.40 0.80
> 03:56:43 AM -272.18 0.00 68.13 276.32
> 03:57:43 AM -415.72 0.00 36.13 413.92
> 03:58:43 AM -91.03 0.00 76.95 89.65
> 03:59:43 AM 5.32 0.00 43.47 0.73
> 04:02:03 AM -4.74 0.00 3.86 0.13
> 04:04:30 AM 5013.36 0.00 -111.62 -4181.22
> 04:05:30 AM 5962.68 0.00 -2647.12 -3293.55
> 04:06:30 AM -8.10 0.00 0.02 6.50
>
> 3. sar -f sa09 -b
>
> 03:53:43 AM tps rtps wtps bread/s bwrtn/s
> 03:54:43 AM 161.52 156.32 5.20 3156.67 119.60
> 03:55:43 AM 148.37 129.35 19.02 1034.80 377.33
> 03:56:43 AM 146.32 128.48 17.83 2732.80 360.40
> 03:57:43 AM 107.32 84.62 22.70 3743.60 447.07
> 03:58:43 AM 91.73 82.03 9.70 1312.40 194.80
> 03:59:43 AM 75.62 54.22 21.40 433.73 350.00
> 04:02:03 AM 4.97 4.83 0.14 38.65 1.24
> 04:04:30 AM 82.68 9.44 73.24 78.45 958.39
> 04:05:30 AM 2.93 0.00 2.93 0.00 29.33
> 04:06:30 AM 0.22 0.00 0.22 0.00 1.73
>
> 4. sar -f sa09 -i
>
> 03:53:43 AM dentunusd file-sz %file-sz inode-sz super-sz %super-sz
> dquot-sz %dquot-sz rtsig-sz %rtsig-sz
> 03:54:43 AM 57361 134 0.01 61318 0 0.00
> 0 0.00 0 0.00
> 03:55:43 AM 58318 124 0.01 62006 0 0.00
> 0 0.00 0 0.00
> 03:56:43 AM 44384 135 0.01 47145 0 0.00
> 0 0.00 0 0.00
> 03:57:43 AM 42565 135 0.01 45983 0 0.00
> 0 0.00 0 0.00
> 03:58:43 AM 18901 134 0.01 22408 0 0.00
> 0 0.00 0 0.00
> 03:59:43 AM 607 135 0.01 1173 0 0.00
> 0 0.00 0 0.00
> 04:02:03 AM 4294967295 113 0.01 417 0
> 0.00 0 0.00 4 0.39
> 04:04:30 AM 49 247 0.02 6316 0 0.00
> 0 0.00 0 0.00
> 04:05:30 AM 121 311 0.03 365 0 0.00
> 0 0.00 0 0.00
>
> 5. sar -f sa09 -u
>
> 03:53:43 AM CPU %user %nice %system %idle
> 03:54:43 AM all 7.52 0.00 25.15 67.33
> 03:55:43 AM all 8.97 0.00 25.28 65.75
> 03:56:43 AM all 6.07 0.00 23.82 70.11
> 03:57:43 AM all 5.08 0.00 23.54 71.38
> 03:58:43 AM all 6.77 0.00 22.88 70.36
> 03:59:43 AM all 7.18 0.00 25.82 67.00
> 04:02:03 AM all 0.77 0.00 96.32 2.91
> 04:04:30 AM all 4.20 0.00 95.11 0.69
> 04:05:30 AM all 1.88 0.00 5.29 92.83
> 04:06:30 AM all 2.01 0.00 2.81 95.18
>
> Russell Coker wrote:
> >I have a server with 4G of RAM running ReiserFS for everything that
> > matters.
> >
> >It has 2G of swap space free, but so far I have not seen swap usage go
> > above 1.6M (so in normal use I could turn off swap entirely and expect
> > not to see much difference).
> >
> >When it's under really heavy load (when I have a maintenance task
> > involving a "find /" and there are lots of POP/IMAP clients hitting the
> > server as well as mail delivery) and the load average gets to about 40,
> > the "kswapd" kernel thread starts using excessive CPU time. It will stay
> > on ~4% but have spikes of up to 45%!!! This is a two-processor machine
> > so 45% CPU reported by top means 90% of a single CPU I guess. 90% of a
> > 1.8GHz P4 CPU is a lot of CPU and I think that something is wrong.
> >
> >In the meager documentation in the kernel source kswapd is described as
> > being involved in paging to disk. I don't think that this is what it is
> > doing as there is no noticable paging activity (it generally has at least
> > 600M of "buffers" so there is no real shortage of memory).
> >
> >Would the activity of kswapd be involved with ReiserFS in any way? What
> > can I do to improve this situation?
--
Dieter Nützel
Graduate Student, Computer Science
University of Hamburg
Department of Computer Science
@home: Dieter.Nuetzel at hamburg.de (replace at with @)
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: kswapd CPU usage and heavy disk IO
2003-01-09 16:42 ` Dieter Nützel
@ 2003-01-09 16:48 ` Brian Tinsley
2003-01-09 16:50 ` Anders Widman
2003-01-10 0:46 ` William Lee Irwin III
2 siblings, 0 replies; 7+ messages in thread
From: Brian Tinsley @ 2003-01-09 16:48 UTC (permalink / raw)
To: Dieter Nützel
Cc: Russell Coker, ReiserFS, Rik van Riel, Andrea Arcangeli,
Linux Kernel List
Dieter Nützel wrote:
>I think you should have cc'ed Andrea Arcangeli <andrea@suse.de>, LKM and try
>2.4.20-aa1.
>
I've got the -aa1 patch, but I have not been able to build the Linux
Virtual Server code with it yet. I absolutely depend on this and have a
request for assistance posted to that mailing list.
>Are you sure it is a ReiserFS and not a kernel thing?
>
I don't believe it's a reiserfs issue. That's just where this thread
started. IMHO, it's a kernel issue.
>
>
--
-[========================]-
-[ Brian Tinsley ]-
-[ Chief Systems Engineer ]-
-[ Emageon ]-
-[========================]-
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: kswapd CPU usage and heavy disk IO
2003-01-09 16:42 ` Dieter Nützel
2003-01-09 16:48 ` Brian Tinsley
@ 2003-01-09 16:50 ` Anders Widman
2003-01-10 0:46 ` William Lee Irwin III
2 siblings, 0 replies; 7+ messages in thread
From: Anders Widman @ 2003-01-09 16:50 UTC (permalink / raw)
To: Dieter Nützel
Cc: Brian Tinsley, Russell Coker, ReiserFS, Rik van Riel,
Andrea Arcangeli, Linux Kernel List
> Are you sure it is a ReiserFS and not a kernel thing?
I would think it is probably not. I have seen this also when running
things like "badblocks /dev/hdb" and the kswapd eats up all CPU
recourses. Then again I am always using ReiserFS so I do not know if
the ReiserFS is the cause or not.. But judging from badblocks is not
FS dependantI think there is no wrong with ReiserFS =)
//Anders
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: kswapd CPU usage and heavy disk IO
2003-01-09 13:31 kswapd CPU usage and heavy disk IO Russell Coker
2003-01-09 16:02 ` Brian Tinsley
@ 2003-01-09 16:56 ` Oleg Drokin
1 sibling, 0 replies; 7+ messages in thread
From: Oleg Drokin @ 2003-01-09 16:56 UTC (permalink / raw)
To: Russell Coker; +Cc: ReiserFS, Rik van Riel
Hello!
On Thu, Jan 09, 2003 at 02:31:54PM +0100, Russell Coker wrote:
> I have a server with 4G of RAM running ReiserFS for everything that matters.
> It has 2G of swap space free, but so far I have not seen swap usage go above
> 1.6M (so in normal use I could turn off swap entirely and expect not to see
> much difference).
> When it's under really heavy load (when I have a maintenance task involving a
> "find /" and there are lots of POP/IMAP clients hitting the server as well as
> mail delivery) and the load average gets to about 40, the "kswapd" kernel
> thread starts using excessive CPU time. It will stay on ~4% but have spikes
> of up to 45%!!! This is a two-processor machine so 45% CPU reported by top
> means 90% of a single CPU I guess. 90% of a 1.8GHz P4 CPU is a lot of CPU
> and I think that something is wrong.
Sounds exactly like yesterday/todays topic on lkml.
You have highmem box, during heavy IO all of the lowmempages are
occupied with bounce buffers and bh's.
Kernel needs more low memory and tries to free some with no much success though.
Known non-reiserfs related problem. Not easy to fix unfortunatelly.
Relevant lkml topic was "2.4.20, .text.lock.swap cpu usage? (ibm x440)"
Mail from Andrew Morton with msgid <3E1A16C5.87EDE35A@digeo.com>
He recommended to try http://www.kernel.org/pub/linux/kernel/people/andrea/kernels/v2.4/2.4.20aa1.bz2
"and send a report on the outcome"
Bye,
Oleg
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: kswapd CPU usage and heavy disk IO
2003-01-09 16:42 ` Dieter Nützel
2003-01-09 16:48 ` Brian Tinsley
2003-01-09 16:50 ` Anders Widman
@ 2003-01-10 0:46 ` William Lee Irwin III
2 siblings, 0 replies; 7+ messages in thread
From: William Lee Irwin III @ 2003-01-10 0:46 UTC (permalink / raw)
To: Dieter N?tzel
Cc: Brian Tinsley, Russell Coker, ReiserFS, Rik van Riel,
Andrea Arcangeli, Linux Kernel List
[-- Attachment #1: brief message --]
[-- Type: text/plain, Size: 1604 bytes --]
Am Donnerstag, 9. Januar 2003 17:02 schrieb Brian Tinsley:
>> I've been seeing the exact same thing on the same type of system in the
>> same situations. This has been causing all kinds of problems on our
>> clusters: the system live-locks for a minute or two, causes cluster
>> heartbeats to not be received, and falsely fails over when the system
>> recovers from the live-lock. The only thing I can find after the
>> live-lock is that the runtime for kswapd is abnormally high.
>> We started running sar (60 second collection interval) and were able to
>> capture some stats during this live-lock period. I've snipped some I
>> believe may be of interest. Note the missing stats between 03:59:43 and
>> 04:02:03
>> Oh BTW, this is on a stock 2.4.20 kernel (dual P3, 4GB), but I have seen
>> the same behavior on 2.4.19 and 2.4.17.
On Thu, Jan 09, 2003 at 05:42:51PM +0100, Dieter N?tzel wrote:
> I think you should have cc'ed Andrea Arcangeli <andrea@suse.de>,
> LKM and try 2.4.20-aa1. Are you sure it is a ReiserFS and not a
> kernel thing?
There simply aren't enough scenarios for this to be a mystery. Both
-aa and 2.5.x should have something in there for it: memclass-related
buffer_head stuff in -aa, and bh-stripping + "bh-less" operation (for
ext2) in 2.5.x + fewer (if any) bh's outside of actual dirty data.
Bloat monitoring scripts attached, which might provide somewhat more
useful output to capture, though they certainly don't eliminate the
need for /proc/meminfo logging. I'll also see if some of the accounting
patches can be backported and send those to Marcelo and Andrea.
Bill
[-- Attachment #2: bloatmon --]
[-- Type: text/plain, Size: 413 bytes --]
#!/usr/bin/awk -f
BEGIN {
printf "%18s %8s %8s %8s\n", "cache", "active", "alloc", "%util";
}
{
if ($3 != 0.0) {
pct = 100.0 * $2 / $3;
frac = (10000.0 * $2 / $3) % 100;
} else {
pct = 100.0;
frac = 0.0;
}
active = ($2 * $4)/1024;
alloc = ($3 * $4)/1024;
if ((alloc - active) < 1.0) {
pct = 100.0;
frac = 0.0;
}
printf "%18s: %8dKB %8dKB %3d.%-2d\n", $1, active, alloc, pct, frac;
}
[-- Attachment #3: bloatmeter --]
[-- Type: text/plain, Size: 133 bytes --]
#!/bin/sh
while : ; do
grep -v '^slabinfo' /proc/slabinfo \
| bloatmon \
| sort -n -k 4,4 \
| head -22
sleep 5
echo
done
[-- Attachment #4: bloatmost --]
[-- Type: text/plain, Size: 110 bytes --]
#!/bin/sh
while true
do
bloatmon < /proc/slabinfo \
| sort -rn -k 3,3 \
| head -22
sleep 60
echo
done
^ permalink raw reply [flat|nested] 7+ messages in thread
end of thread, other threads:[~2003-01-10 0:46 UTC | newest]
Thread overview: 7+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2003-01-09 13:31 kswapd CPU usage and heavy disk IO Russell Coker
2003-01-09 16:02 ` Brian Tinsley
2003-01-09 16:42 ` Dieter Nützel
2003-01-09 16:48 ` Brian Tinsley
2003-01-09 16:50 ` Anders Widman
2003-01-10 0:46 ` William Lee Irwin III
2003-01-09 16:56 ` Oleg Drokin
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.