Re: dramatic I/O slowdown after upgrading 2.6.32->3.0

[Michael Tokarev <mjt@tls.msk.ru>]

On 06.04.2012 03:29, Jan Kara wrote:

> On Fri 30-03-12 20:50:54, Michael Tokarev wrote:
>> I'm observing a dramatic slowdown of several hosts after upgrading
>> from 2.6.32.y to 3.0.x i686 kernels (in both cases from kernel.org,
>> on both cases the last version is relatively latest).
[]
>> What's the way to debug this issue?
>   Identifying a particular kernel where things regresses might help as Jon
> wrote. Just from top of my head, 3.0 had a bug in device plugging so
> readahead was broken. I think it was addressed in -stable series so you

That's definitely not readahead, it since writes are painfully slow
too.  I found one more example -- extlinux --once="test kernel" with
3.0 takes about 20 seconds to complete on an idle system.

> might want to check out latest 3.0-stable.

I did mention this in my initial email (that part quoted above) --
both 2.6.32 and 3.0 are relatively latest from each series,
right now it is 3.0.27.

Yesterday I tried to do some bisection, but ended up in an unbootable
system (it is remote production server), so now I'm waiting for remote
hands to repair it (I don't yet know what went wrong, we'll figure it
out).  I've some time during nights when I can do anything with that
machine, but I have to keep it reachable/working on each reboot.

Apparently I was wrong saying that there's another machine which
suffers from the same issue -- nope, the other machine had an unrelated
issue which I fixed.  So it turns out that from about 200 different
machines, I've just one machine which does not run 3.0 kernel properly.
I especially tried 3.0 on a few more - different - machines last
weekend, in order to see what other machines has this problem, but
found nothing.

So I'll try to continue (or actually _start_) the bisection on this
very server, the way it will be possible having in mind the difficult
conditions.

I just thoght I'd ask first, maybe someone knows offhand what may be
the problem.. ;)

Thank you!

/mjt