Re: Measuring IOPS

[Jens Axboe <axboe@kernel.dk>]

On 2011-08-04 10:51, Martin Steigerwald wrote:
> Am Mittwoch, 3. August 2011 schrieb Martin Steigerwald:
>> Am Mittwoch, 3. August 2011 schrieb Martin Steigerwald:
>>> Am Mittwoch, 3. August 2011 schrieben Sie:
>>>> Martin Steigerwald <Martin@lichtvoll.de> writes:
>> [...]
>>
>>> Does using iodepth > 1 need ioengine=libaio? Let�s see the manpage:
>>>        iodepth=int
>>>        
>>>               Number  of I/O units to keep in flight against the
>>>               file. Note that increasing iodepth beyond  1  will
>>>               not affect synchronous ioengines (except for small
>>>               degress when verify_async is in use).  Even  async
>>>               engines  my  impose  OS  restrictions  causing the
>>>               desired depth not to be achieved.  This may happen
>>>               on   Linux  when  using  libaio  and  not  setting
>>>               direct=1, since buffered IO is not async  on  that
>>>               OS.  Keep  an  eye on the IO depth distribution in
>>>               the fio output to verify that the  achieved  depth
>>>               is as expected. Default: 1.
>>>
>>> Okay, yes, it does. I start getting a hang on it. Its a bit puzzling
>>> to have two concepts of synchronous I/O around:
>>>
>>> 1) synchronous system call interfaces aka fio I/O engine
>>>
>>> 2) synchronous I/O requests aka O_SYNC
>>
>> But isn�t this a case for iodepth=1 if buffered I/O on Linux is
>> synchronous? I bet most regular applications except some databases use
>> buffered I/O.
> 
> Thanks a lot for your answers, Jens, Jeff, DongJin.
> 
> Now what about the above one?
> 
> In what cases is iodepth > 1 relevant, when Linux buffered I/O is 
> synchronous? For mutiple threads or processes?

iodepth controls what depth fio operates at, not the OS. You are right
in that with iodepth=1, for buffered writes you could be seeing a much
higher depth on the device side.

So think of iodepth as how many IO units fio can have in flight, nothing
else.

> One process / thread can only submit one I/O at a time with synchronous 
> system call I/O, but the function returns when the stuff is in the page 
> cache. So first why can�t Linux use iodepth > 1 when there is lots of stuff 
> in the page cache to be written out? That should help the single process 
> case.

Since the IO unit is done when the system call returns, you can never
have more than the one in flight for a sync engine. So iodepth > 1 makes
no sense for a sync engine.

> On the mutiple process/threadsa case Linux gets several I/O requests from 
> mutiple processes/threads and thus iodepth > 1 does make sense?

No.

> Maybe it helps getting clear where in the stack iodepth is located at, is 
> it
> 
> process / thread
> systemcall
> pagecache
> blocklayer
> iodepth
> device driver
> device
> 
> ? If so, why can�t Linux  not make use of iodepth > 1 with synchronous 
> system call I/O? Or is it further up on the system call level? But then 

Because it is sync. The very nature of the sync system calls is that
submission and completion are one event. For libaio, you could submit a
bunch of requests before retrieving or waiting for completion of any one
of them.

The only example where a sync engine could drive a higher queue depth on
the device side is buffered writes. For any other case (reads, direct
writes), you need async submission to build up a higher queue depth.

> what sense would it make there, when using system calls that are 
> asynchronous already?
> (Is that ordering above correct at all?)

Your ordering looks OK. Now consider where and how you end up waiting
for issued IO, that should tell you where queue depth could build up or
not.

-- 
Jens Axboe