Re: confusion about nr of pending I/O requests

[Jens Axboe <axboe@kernel.dk>]

On 12/18/18 5:45 AM, Paolo Valente wrote:
> Hi Jens,
> sorry for the following silly question, but maybe you can solve very
> quickly a doubt for which I'd spend much more time investigating.
> 
> While doing some tests with scsi_debug, I've just seen that (at least)
> with direct I/O, the maximum number of pending I/O requests (at least
> in the I/O schedulers) is equal, unexpectedly, to the queue depth of
> the drive and not to
> /sys/block/<dev>/queue/nr_requests
> 
> For example, after:
> 
> sudo modprobe scsi_debug max_queue=4
> 
> and with fio executed as follows:
> 
> job: (g=0): rw=read, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=20
> 
> I get this periodic trace, where four insertions are followed by four
> completions, and so on, till the end of the I/O.  This trace is taken
> with none, but the result is the same with bfq.
> 
>              fio-20275 [001] d...  7560.655213:   8,48   I   R 281088 + 8 [fio]
>              fio-20275 [001] d...  7560.655288:   8,48   I   R 281096 + 8 [fio]
>              fio-20275 [001] d...  7560.655311:   8,48   I   R 281104 + 8 [fio]
>              fio-20275 [001] d...  7560.655331:   8,48   I   R 281112 + 8 [fio]
>           <idle>-0     [001] d.h.  7560.749868:   8,48   C   R 281088 + 8 [0]
>           <idle>-0     [001] dNh.  7560.749912:   8,48   C   R 281096 + 8 [0]
>           <idle>-0     [001] dNh.  7560.749928:   8,48   C   R 281104 + 8 [0]
>           <idle>-0     [001] dNh.  7560.749934:   8,48   C   R 281112 + 8 [0]
>              fio-20275 [001] d...  7560.750023:   8,48   I   R 281120 + 8 [fio]
>              fio-20275 [001] d...  7560.750196:   8,48   I   R 281128 + 8 [fio]
>              fio-20275 [001] d...  7560.750229:   8,48   I   R 281136 + 8 [fio]
>              fio-20275 [001] d...  7560.750250:   8,48   I   R 281144 + 8 [fio]
>           <idle>-0     [001] d.h.  7560.842510:   8,48   C   R 281120 + 8 [0]
>           <idle>-0     [001] dNh.  7560.842551:   8,48   C   R 281128 + 8 [0]
>           <idle>-0     [001] dNh.  7560.842556:   8,48   C   R 281136 + 8 [0]
>           <idle>-0     [001] dNh.  7560.842562:   8,48   C   R 281144 + 8 [0]
> 
> Shouldn't the total number of pending requests reach
> /sys/block/<dev>/queue/nr_requests ?
> 
> The latter is of course equal to 8.

With a scheduler, the depth is what the scheduler provides. You cannot
exceed the hardware queue depth in any situation. You just have 8
requests available for scheduling, with a max of 4 being inflight on
the device side.

If both were 4, for instance, then you would have nothing to schedule
with, as all of them could reside on the hardware side. That's why
the scheduler defaults to twice the hardware queue depth.

-- 
Jens Axboe