Re: 2.6.0 performance problems

[William Lee Irwin III <wli@holomorphy.com>]

On Mon, Dec 29, 2003 at 08:37:53PM -0500, Thomas Molina wrote:
>> Right.  I have 120MB RAM and 256MB swap partition.  That corresponds to 
>> the 85 to 90 percent top says I am spending in iowait.

On Tue, Dec 30, 2003 at 01:21:45PM -0600, Andy Isaacson wrote:
> Yeah, right now BK needs about 140-160MB of working set to do a
> consistency check on the 2.5 tree.  That means you're paging, and it
> sounds like paging sucks on 2.6?
> (Actually, BK is even happier if the kernel can keep all the sfiles in
> cache, so a half-gig is a comfortable amount for working with the
> current 2.5 tree, although 256MB should be enough to avoid paging hell.
> With a full gig, you can keep two full trees in "checkout:get" mode in
> cache, which is nice.)

Well, it's not supposed to suck.

Something to try that affects paging directly would be adjusting
/proc/sys/vm/swappiness to, say, 0 and 100 and trying it at both levels.

More intelligent solutions require more instrumentation to address.

I generally recommend:
(1) logging top(1) running in batch mode
(2) logging vmstat(1)
(3) snapshotting /proc/meminfo
(4) snapshotting /prov/vmstat

I recommend an interval of 5s and logging with things like
top b d 5 > /tmp/top.log 2>&1 &
vmstat 5 > /tmp/vmstat.log 2>&1 &
(while true; do cat /proc/meminfo; sleep 5; done) > /tmp/meminfo.log 2>&1 &
(while true; do cat /proc/vmstat; sleep 5; done) > /tmp/proc_vmstat.log 2>&1 &

Thus far interpretations of information collected this way have been
somewhat lacking. Roger Luethi has identified various points at which
regressions happened over the course of 2.5, but it appears that
information hasn't yet been and still needs to be acted on.

If you could also try to identify points in time when the system has
become less responsive I'd be much obliged.


-- wli