fio rand read/write regression with 2.6.32-rc3

fio rand read/write regression with 2.6.32-rc3

[Zhang, Yanmin]

Comparing with 2.6.23-rc1's result, fio rand read write has regression
on my 2*4 core stoakley machine (8GB memory) with a JBOD of 12 disks.

Every disk has 8 1-GB files. Start 8 sub-processes per disk and every
process random chooses a file on the disk to do 36 times of file read or
write on the file and then choose another file.


fio_mmap_rand_read_4k regresion is about 35%.
Bisect down to patch:
1d2235152dc745c6d94bedb550fea84cffdbf768 is first bad commit
commit 1d2235152dc745c6d94bedb550fea84cffdbf768
Author: Jens Axboe <jens.axboe@oracle.com>
Date:   Fri Oct 2 19:27:04 2009 +0200

    cfq-iosched: add a knob for desktop interactiveness


After I revert the patch against 2.6.23-rc3, fio_mmap_rand_read_4k regression
disappears.

fio_mmap_randrw_4k has less than 20% regression and reverting the patch
could restore performance.

On another stoakley machine with another JBOD, we see the similiar regression.

ffsb rand read/write has a bigger regression and reverting the patch
could restore performance.

Yanmin

Re: fio rand read/write regression with 2.6.32-rc3

[Jens Axboe]

On Sat, Oct 10 2009, Zhang, Yanmin wrote:
> Comparing with 2.6.23-rc1's result, fio rand read write has regression
> on my 2*4 core stoakley machine (8GB memory) with a JBOD of 12 disks.
> 
> Every disk has 8 1-GB files. Start 8 sub-processes per disk and every
> process random chooses a file on the disk to do 36 times of file read or
> write on the file and then choose another file.
> 
> 
> fio_mmap_rand_read_4k regresion is about 35%.

Heh, I seem to recollect I told Linus that this would cost is 30-40%
performance. So not totally crazy.

So yes, this isn't hugely unexpected. If you send me your fio job files,
I'll try and see what I can do about it.

> Bisect down to patch:
> 1d2235152dc745c6d94bedb550fea84cffdbf768 is first bad commit
> commit 1d2235152dc745c6d94bedb550fea84cffdbf768
> Author: Jens Axboe <jens.axboe@oracle.com>
> Date:   Fri Oct 2 19:27:04 2009 +0200
> 
>     cfq-iosched: add a knob for desktop interactiveness
> 
> 
> After I revert the patch against 2.6.23-rc3, fio_mmap_rand_read_4k regression
> disappears.
> 
> fio_mmap_randrw_4k has less than 20% regression and reverting the patch
> could restore performance.
> 
> On another stoakley machine with another JBOD, we see the similiar regression.
> 
> ffsb rand read/write has a bigger regression and reverting the patch
> could restore performance.
> 
> Yanmin
> 
> 

-- 
Jens Axboe

Re: fio rand read/write regression with 2.6.32-rc3

[Jens Axboe]

On Sat, Oct 10 2009, Jens Axboe wrote:
> On Sat, Oct 10 2009, Zhang, Yanmin wrote:
> > Comparing with 2.6.23-rc1's result, fio rand read write has regression
> > on my 2*4 core stoakley machine (8GB memory) with a JBOD of 12 disks.
> > 
> > Every disk has 8 1-GB files. Start 8 sub-processes per disk and every
> > process random chooses a file on the disk to do 36 times of file read or
> > write on the file and then choose another file.
> > 
> > 
> > fio_mmap_rand_read_4k regresion is about 35%.
> 
> Heh, I seem to recollect I told Linus that this would cost is 30-40%
> performance. So not totally crazy.
> 
> So yes, this isn't hugely unexpected. If you send me your fio job files,
> I'll try and see what I can do about it.

Oh, and can you check whether setting

/sys/block/sdX/queue/iosched/low_latency

to 0 for the involved devices makes the regression go away?

-- 
Jens Axboe

Re: fio rand read/write regression with 2.6.32-rc3

[Zhang, Yanmin]

On Sat, 2009-10-10 at 10:02 +0200, Jens Axboe wrote:
> On Sat, Oct 10 2009, Jens Axboe wrote:
> > On Sat, Oct 10 2009, Zhang, Yanmin wrote:
> > > Comparing with 2.6.23-rc1's result, fio rand read write has regression
> > > on my 2*4 core stoakley machine (8GB memory) with a JBOD of 12 disks.
> > > 
> > > Every disk has 8 1-GB files. Start 8 sub-processes per disk and every
> > > process random chooses a file on the disk to do 36 times of file read or
> > > write on the file and then choose another file.
> > > 
> > > 
> > > fio_mmap_rand_read_4k regresion is about 35%.
> > 
> > Heh, I seem to recollect I told Linus that this would cost is 30-40%
> > performance. So not totally crazy.
> > 
> > So yes, this isn't hugely unexpected. If you send me your fio job files,
> > I'll try and see what I can do about it.
> 
> Oh, and can you check whether setting
> 
> /sys/block/sdX/queue/iosched/low_latency
> 
> to 0 for the involved devices makes the regression go away?
Echo 0 to low_latency makes the regression become less than 3%.

Re: fio rand read/write regression with 2.6.32-rc3

[Zhang, Yanmin]

[-- Attachment #1: Type: text/plain, Size: 1513 bytes --]

On Sat, 2009-10-10 at 10:01 +0200, Jens Axboe wrote:
> On Sat, Oct 10 2009, Zhang, Yanmin wrote:
> > Comparing with 2.6.23-rc1's result, fio rand read write has regression
> > on my 2*4 core stoakley machine (8GB memory) with a JBOD of 12 disks.
> > 
> > Every disk has 8 1-GB files. Start 8 sub-processes per disk and every
> > process random chooses a file on the disk to do 36 times of file read or
> > write on the file and then choose another file.
> > 
> > 
> > fio_mmap_rand_read_4k regresion is about 35%.
> 
> Heh, I seem to recollect I told Linus that this would cost is 30-40%
> performance. So not totally crazy.
> 
> So yes, this isn't hugely unexpected. If you send me your fio job files,
> I'll try and see what I can do about it.
See the attachment.

> 
> > Bisect down to patch:
> > 1d2235152dc745c6d94bedb550fea84cffdbf768 is first bad commit
> > commit 1d2235152dc745c6d94bedb550fea84cffdbf768
> > Author: Jens Axboe <jens.axboe@oracle.com>
> > Date:   Fri Oct 2 19:27:04 2009 +0200
> > 
> >     cfq-iosched: add a knob for desktop interactiveness
> > 
> > 
> > After I revert the patch against 2.6.23-rc3, fio_mmap_rand_read_4k regression
> > disappears.
> > 
> > fio_mmap_randrw_4k has less than 20% regression and reverting the patch
> > could restore performance.
> > 
> > On another stoakley machine with another JBOD, we see the similiar regression.
> > 
> > ffsb rand read/write has a bigger regression and reverting the patch
> > could restore performance.
> > 
> > Yanmin
> > 
> > 
> 

[-- Attachment #2: job_file_st01_fio_mmap_randread_4k.gz --]
[-- Type: application/x-gzip, Size: 1125 bytes --]

Re: fio rand read/write regression with 2.6.32-rc3

[Corrado Zoccolo]

On Sat, Oct 10, 2009 at 11:52 AM, Zhang, Yanmin
<yanmin_zhang@linux.intel.com> wrote:
> On Sat, 2009-10-10 at 10:01 +0200, Jens Axboe wrote:
>> On Sat, Oct 10 2009, Zhang, Yanmin wrote:
>> > Comparing with 2.6.23-rc1's result, fio rand read write has regression
>> > on my 2*4 core stoakley machine (8GB memory) with a JBOD of 12 disks.
>> >
>> > Every disk has 8 1-GB files. Start 8 sub-processes per disk and every
>> > process random chooses a file on the disk to do 36 times of file read or
>> > write on the file and then choose another file.
>> >
>> >
>> > fio_mmap_rand_read_4k regresion is about 35%.
>>
>> Heh, I seem to recollect I told Linus that this would cost is 30-40%
>> performance. So not totally crazy.
>>
>> So yes, this isn't hugely unexpected. If you send me your fio job files,
>> I'll try and see what I can do about it.
> See the attachment.
>
Hi Yanmin,
the fio test you sent just performs random read, no write seems involved here.
I suspect that you should be able to observe the same regression if
you just run on a single disk. Can you confirm?
Is your disk a SATA2 rotational disk with NCQ?

Thanks,
Corrado

-- 
__________________________________________________________________________

dott. Corrado Zoccolo                          mailto:czoccolo@gmail.com
PhD - Department of Computer Science - University of Pisa, Italy
--------------------------------------------------------------------------

Re: fio rand read/write regression with 2.6.32-rc3

[Zhang, Yanmin]

On Sun, 2009-10-11 at 10:23 +0200, Corrado Zoccolo wrote:
> On Sat, Oct 10, 2009 at 11:52 AM, Zhang, Yanmin
> <yanmin_zhang@linux.intel.com> wrote:
> > On Sat, 2009-10-10 at 10:01 +0200, Jens Axboe wrote:
> >> On Sat, Oct 10 2009, Zhang, Yanmin wrote:
> >> > Comparing with 2.6.23-rc1's result, fio rand read write has regression
> >> > on my 2*4 core stoakley machine (8GB memory) with a JBOD of 12 disks.
> >> >
> >> > Every disk has 8 1-GB files. Start 8 sub-processes per disk and every
> >> > process random chooses a file on the disk to do 36 times of file read or
> >> > write on the file and then choose another file.
> >> >
> >> >
> >> > fio_mmap_rand_read_4k regresion is about 35%.
> >>
> >> Heh, I seem to recollect I told Linus that this would cost is 30-40%
> >> performance. So not totally crazy.
> >>
> >> So yes, this isn't hugely unexpected. If you send me your fio job files,
> >> I'll try and see what I can do about it.
> > See the attachment.
> >
> Hi Yanmin,
> the fio test you sent just performs random read, no write seems involved here.

Sorry for not sending all the job files because they are similiar and big.
You could change the job file to run rand write and readwrite.

> I suspect that you should be able to observe the same regression if
> you just run on a single disk. Can you confirm?
> Is your disk a SATA2 rotational disk with NCQ?
It's a JOBD with 12 SAS disks.

Another machine's JBOD has 13 SCSI disks.

Based on your suggestion, I tested it on a single SAS disk (create 8 1-GB file and
start 8 processes) and get 30% regression. On a single SCSI disk, there is also about
30% regression. On a single SATA (SATA link up 1.5 Gbps) disk with NCQ, the regression
is about 20%.

Perhaps the job file causes fio to deliver too many I/O requests, sometimes kernel
print out some blocking info like below:

Apr  8 23:54:19 lkp-tulsa01-x8664 kernel: INFO: task kjournald:6261 blocked for more than 120 seconds.
Apr  8 23:54:19 lkp-tulsa01-x8664 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Apr  8 23:54:19 lkp-tulsa01-x8664 kernel: kjournald     D ffff88002f228180     0  6261      2
Apr  8 23:54:19 lkp-tulsa01-x8664 kernel:  ffff88027ab09910 0000000000000046 0000000000000000 ffff88027cc20440
Apr  8 23:54:19 lkp-tulsa01-x8664 kernel:  ffff88027f0c66a0 ffff88027ab09bb0 0000000100000e56 000000007cc20440
Apr  8 23:54:19 lkp-tulsa01-x8664 kernel:  ffff88027a915480 ffffffff803678a9 0000000000000000 0000000100000000
Apr  8 23:54:19 lkp-tulsa01-x8664 kernel: Call Trace:
Apr  8 23:54:19 lkp-tulsa01-x8664 kernel:  [<ffffffff803678a9>] ? __make_request+0x310/0x40b
Apr  8 23:54:19 lkp-tulsa01-x8664 kernel:  [<ffffffff802b5571>] ? sync_buffer+0x0/0x3f
Apr  8 23:54:19 lkp-tulsa01-x8664 kernel:  [<ffffffff805aeadf>] ? schedule+0x9/0x1d