Re: dramatic I/O slowdown after upgrading 2.6.38->3.0+

[Suresh Jayaraman <sjayaraman@suse.com>]

On 04/10/2012 08:43 PM, Jan Kara wrote:
> On Tue 10-04-12 10:00:38, Michael Tokarev wrote:
>> On 10.04.2012 06:26, Dave Chinner wrote:
>>
>>> Barriers. Turn them off, and see if that fixes your problem.
>>
>> Thank you Dave for a hint.  And nope, that's not it, not at all... ;)
>> While turning off barriers helps a tiny bit, to gain a few %% from
>> the huge slowdown, it does not cure the issue.
>>
>> Meanwhile, I observed the following:
>>
>> 1) the issue persists on more recent kernels too, I tried 3.3
>>    and it is also as slow as 3.0.
>>
>> 2) at least 2.6.38 kernel works fine, as fast as 2.6.32, I'll
>>    try 2.6.39 next.
>>
>>    I updated $subject accordingly.
>>
>> 3) the most important thing I think: this is general I/O speed
>>    issue.  Here's why:
>>
>>   2.6.38:
>>   # dd if=/dev/sdb of=/dev/null bs=1M iflag=direct count=100
>>   100+0 records in
>>   100+0 records out
>>   104857600 bytes (105 MB) copied, 1.73126 s, 60.6 MB/s
>>
>>   3.0:
>>   # dd if=/dev/sdb of=/dev/null bs=1M iflag=direct count=100
>>   100+0 records in
>>   100+0 records out
>>   104857600 bytes (105 MB) copied, 29.4508 s, 3.6 MB/s
>>
>> That's about 20 times difference on direct read from the
>> same - idle - device!!
>   Huh, that's a huge difference for such a trivial load. So we can rule out
> filesystems, writeback, mm. I also wouldn't think it's IO scheduler but
> you can always check by comparing dd numbers after
>   echo none >/sys/block/sdb/queue/scheduler

s/none/noop
you meant noop, of course?


Suresh