uneven IOPS among HD

uneven IOPS among HD

[john smith]

I have a couple of questions:

1) if it is normal (and if so, why) that while running a number of identical jobs (e.g 4), each bound to a different CPU (cpu_allowed=i) and each performing sequential (bs=512) reads on (e.g. 4) different HD, fio reports IOPS numbers that are highly unbalanced among HD with one HD reporting more than 7x of the IOPS of the minimum IOPS HD.
Different runs may show most of the IOPS on a different HD, so it's not that one HD is faster.

Here are some possibly interesting parameters:

ioengine=libaio
iodepth=32
direct=1

2) why increasing "iodepth' (eg to 512) decreses the IOPS results significantly (by 10x)?

thanks,
John  


      

Re: uneven IOPS among HD

[Jens Axboe]

On 2011-02-17 03:14, john smith wrote:
> I have a couple of questions:
> 
> 1) if it is normal (and if so, why) that while running a number of identical jobs (e.g 4), each bound to a different CPU (cpu_allowed=i) and each performing sequential (bs=512) reads on (e.g. 4) different HD, fio reports IOPS numbers that are highly unbalanced among HD with one HD reporting more than 7x of the IOPS of the minimum IOPS HD.
> Different runs may show most of the IOPS on a different HD, so it's not that one HD is faster.
> 
> Here are some possibly interesting parameters:
> 
> ioengine=libaio
> iodepth=32
> direct=1

There should be very little variance between runs, and between jobs.
Each are separate threads/processes, so most of the variance you will
see will be due to system "artifacts". So I'd suggest you look there. If
you don't pin to specific CPUs, do you see smaller or larger
differences?

Are you doing random or sequential IO? Could also be timing differences
in submission and dispatch from the IO scheduler.

> 2) why increasing "iodepth' (eg to 512) decreses the IOPS results significantly (by 10x)?

You probably don't want a larger depth in fio than the system supports.

-- 
Jens Axboe

Re: uneven IOPS among HD

[Josh Aune]

On Sunday, February 20, 2011, Jens Axboe <jaxboe@fusionio.com> wrote:
> On 2011-02-17 03:14, john smith wrote:
>> I have a couple of questions:
>>
>> 1) if it is normal (and if so, why) that while running a number of identical jobs (e.g 4), each bound to a different CPU (cpu_allowed=i) and each performing sequential (bs=512) reads on (e.g. 4) different HD, fio reports IOPS numbers that are highly unbalanced among HD with one HD reporting more than 7x of the IOPS of the minimum IOPS HD.
>> Different runs may show most of the IOPS on a different HD, so it's not that one HD is faster.
>>
>> Here are some possibly interesting parameters:
>>
>> ioengine=libaio
>> iodepth=32
>> direct=1
>
> There should be very little variance between runs, and between jobs.
> Each are separate threads/processes, so most of the variance you will
> see will be due to system "artifacts". So I'd suggest you look there. If
> you don't pin to specific CPUs, do you see smaller or larger
> differences?
>
> Are you doing random or sequential IO? Could also be timing differences
> in submission and dispatch from the IO scheduler.

Is hyper threading enabled?  Could be running some of the jobs on the
same CPU core.

Does it track to different sockets?


>
>> 2) why increasing "iodepth' (eg to 512) decreses the IOPS results significantly (by 10x)?
>
> You probably don't want a larger depth in fio than the system supports.
>
> --
> Jens Axboe
>
> --
> To unsubscribe from this list: send the line "unsubscribe fio" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at �http://vger.kernel.org/majordomo-info.html
>

Re: uneven IOPS among HD

[Jens Axboe]

On 2011-02-26 15:31, Josh Aune wrote:
> On Sunday, February 20, 2011, Jens Axboe <jaxboe@fusionio.com> wrote:
>> On 2011-02-17 03:14, john smith wrote:
>>> I have a couple of questions:
>>>
>>> 1) if it is normal (and if so, why) that while running a number of identical jobs (e.g 4), each bound to a different CPU (cpu_allowed=i) and each performing sequential (bs=512) reads on (e.g. 4) different HD, fio reports IOPS numbers that are highly unbalanced among HD with one HD reporting more than 7x of the IOPS of the minimum IOPS HD.
>>> Different runs may show most of the IOPS on a different HD, so it's not that one HD is faster.
>>>
>>> Here are some possibly interesting parameters:
>>>
>>> ioengine=libaio
>>> iodepth=32
>>> direct=1
>>
>> There should be very little variance between runs, and between jobs.
>> Each are separate threads/processes, so most of the variance you will
>> see will be due to system "artifacts". So I'd suggest you look there. If
>> you don't pin to specific CPUs, do you see smaller or larger
>> differences?
>>
>> Are you doing random or sequential IO? Could also be timing differences
>> in submission and dispatch from the IO scheduler.
> 
> Is hyper threading enabled?  Could be running some of the jobs on the
> same CPU core.
> 
> Does it track to different sockets?

No, the replay doesn't factor in CPU placement of the apps submitting
IO.

-- 
Jens Axboe