How to get more sequential IO merged at elevator

How to get more sequential IO merged at elevator

[Desai, Kashyap]

Jens,

While working on one issue of less IOPs for sequential READ/WRITE, I found interesting stuffs which was causing performance drop for sequential IO.  I did some reverse engineering on Block layer code to understand how to get benefit from any sysfs parameters settings, but could not find anything useful to solve this issue. 

I have described problem statement and root cause of this issue in this mail thread. 

Problem statement - "Cannot achieve Sequential read/write performance, because of back merge is not happening frequently"

Here is my understanding of back merge done in elevator.

Linux block layer is responsible for merging/sorting of the IO with the help of Elevator hook in OS + IO scheduler.
IO scheduler does not have any role in merging sequential IO. It is done in Elevator hook, so choosing any IO scheduler in Linux will not help (OR you can consider that behavior will be unchanged irrespective of IO scheduler). Any sequential IO will be merge at Elevator code path.

1. When IO comes from upper layer, it will be queued at Elevator/IO scheduler level.  It will also add IO into hash look up which will be used for merge and other purpose.
2. Elevator code will search any outstanding IO (in the queue of the same layer). If there is any chance to merge it, it will perform  (BACK MERGE) merge 
3. If there is no merge possible, IO will be queued to the next level (which is IO scheduler).
4. In IO completion Path, IO scheduler will post IO to the Driver queue, if at all there is any outstanding IO. (There are many other condition, but this is very common code path)

To merge more command, #2 should find more outstanding in hash table look up. This is possible if flow control start either at block layer/Driver level.
It means, driver/block layer forcefully delay IO submission to next level and give more chance at elevator code to merge more IO via accumulating more IO from user space.

If I manually change queue depth of the device to lower value (between 1-8), which is doing only Sequential IO.. I am able to see maximum IO coming to the driver after merge and it eventually increase the IOPs.

Is there any way to increase possibility of merged IO coming from block layer to the Low level driver ?

Thanks, Kashyap
--
To unsubscribe from this list: send the line "unsubscribe linux-scsi" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: How to get more sequential IO merged at elevator

[Desai, Kashyap]

I got some clue on what was going on while doing 4K sequential read using fio.

If I use ioengine in fio script as "libaio", I see " do_io_submit" call plug/unplug the queue before submitting the IO.
It means after every IO, we expect to send IO immediately to the next layer. If at all there are any pending IO do merge.. but not due to plugging.

This is what happens on my test. Every time IO comes from application with libaio engine, it send down to the elevator/io-scheduler because queue was unplugged in 
do_io_submit(). Moment I reduce the queue depth of the block device, merge start because of congestion at scsi mid layer.

If I use, mmap engine, I see merged IO coming to the device driver because of plugging. I really don't know how it works, but gave a try and found merge happen because of plugging. ( I confirm using blktrace)

Is there any ioengine in <fio> (or any other parameter setting), which can use plugging mechanism of block layer to merge more IO other than mmap ?

~ Kashyap

> -----Original Message-----
> From: Desai, Kashyap
> Sent: Wednesday, April 30, 2014 12:33 PM
> To: 'axboe@kernel.dk'
> Cc: linux-scsi@vger.kernel.org
> Subject: How to get more sequential IO merged at elevator
> 
> Jens,
> 
> While working on one issue of less IOPs for sequential READ/WRITE, I found
> interesting stuffs which was causing performance drop for sequential IO.  I
> did some reverse engineering on Block layer code to understand how to get
> benefit from any sysfs parameters settings, but could not find anything
> useful to solve this issue.
> 
> I have described problem statement and root cause of this issue in this mail
> thread.
> 
> Problem statement - "Cannot achieve Sequential read/write performance,
> because of back merge is not happening frequently"
> 
> Here is my understanding of back merge done in elevator.
> 
> Linux block layer is responsible for merging/sorting of the IO with the help of
> Elevator hook in OS + IO scheduler.
> IO scheduler does not have any role in merging sequential IO. It is done in
> Elevator hook, so choosing any IO scheduler in Linux will not help (OR you can
> consider that behavior will be unchanged irrespective of IO scheduler). Any
> sequential IO will be merge at Elevator code path.
> 
> 1. When IO comes from upper layer, it will be queued at Elevator/IO
> scheduler level.  It will also add IO into hash look up which will be used for
> merge and other purpose.
> 2. Elevator code will search any outstanding IO (in the queue of the same
> layer). If there is any chance to merge it, it will perform  (BACK MERGE)
> merge 3. If there is no merge possible, IO will be queued to the next level
> (which is IO scheduler).
> 4. In IO completion Path, IO scheduler will post IO to the Driver queue, if at all
> there is any outstanding IO. (There are many other condition, but this is very
> common code path)
> 
> To merge more command, #2 should find more outstanding in hash table
> look up. This is possible if flow control start either at block layer/Driver level.
> It means, driver/block layer forcefully delay IO submission to next level and
> give more chance at elevator code to merge more IO via accumulating more
> IO from user space.
> 
> If I manually change queue depth of the device to lower value (between 1-
> 8), which is doing only Sequential IO.. I am able to see maximum IO coming to
> the driver after merge and it eventually increase the IOPs.
> 
> Is there any way to increase possibility of merged IO coming from block layer
> to the Low level driver ?
> 
> Thanks, Kashyap
--
To unsubscribe from this list: send the line "unsubscribe linux-scsi" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: How to get more sequential IO merged at elevator

[Jens Axboe]

On 05/06/2014 04:06 AM, Desai, Kashyap wrote:
> I got some clue on what was going on while doing 4K sequential read using fio.
> 
> If I use ioengine in fio script as "libaio", I see " do_io_submit" call plug/unplug the queue before submitting the IO.
> It means after every IO, we expect to send IO immediately to the next layer. If at all there are any pending IO do merge.. but not due to plugging.
> 
> This is what happens on my test. Every time IO comes from application with libaio engine, it send down to the elevator/io-scheduler because queue was unplugged in 
> do_io_submit(). Moment I reduce the queue depth of the block device, merge start because of congestion at scsi mid layer.
> 
> If I use, mmap engine, I see merged IO coming to the device driver because of plugging. I really don't know how it works, but gave a try and found merge happen because of plugging. ( I confirm using blktrace)
> 
> Is there any ioengine in <fio> (or any other parameter setting), which can use plugging mechanism of block layer to merge more IO other than mmap ?

O_DIRECT IO is sync by nature, which is why it is sent off immediately
instead of held for potentially merging. mmap is not. You should be able
to provoke merging by submitting more than 1 IO at the time. See the
iodepth_batch settings for fio.

-- 
Jens Axboe

RE: How to get more sequential IO merged at elevator

[Kashyap Desai]

> -----Original Message-----
> From: Jens Axboe [mailto:axboe@kernel.dk]
> Sent: Tuesday, May 06, 2014 7:47 PM
> To: Desai, Kashyap
> Cc: linux-scsi@vger.kernel.org
> Subject: Re: How to get more sequential IO merged at elevator
>
> On 05/06/2014 04:06 AM, Desai, Kashyap wrote:
> > I got some clue on what was going on while doing 4K sequential read
using
> fio.
> >
> > If I use ioengine in fio script as "libaio", I see " do_io_submit"
call
> plug/unplug the queue before submitting the IO.
> > It means after every IO, we expect to send IO immediately to the next
> layer. If at all there are any pending IO do merge.. but not due to
plugging.
> >
> > This is what happens on my test. Every time IO comes from application
> > with libaio engine, it send down to the elevator/io-scheduler because
> queue was unplugged in do_io_submit(). Moment I reduce the queue depth
> of the block device, merge start because of congestion at scsi mid
layer.
> >
> > If I use, mmap engine, I see merged IO coming to the device driver
> > because of plugging. I really don't know how it works, but gave a try
> > and found merge happen because of plugging. ( I confirm using
> > blktrace)
> >
> > Is there any ioengine in <fio> (or any other parameter setting), which
can
> use plugging mechanism of block layer to merge more IO other than mmap ?
>
> O_DIRECT IO is sync by nature, which is why it is sent off immediately
instead
> of held for potentially merging. mmap is not. You should be able to
provoke
> merging by submitting more than 1 IO at the time. See the iodepth_batch
> settings for fio.

Thanks Jens. I got your point about O_DIRECT IO. When I read <man> page of
<fio> it mentioned default iodepth_batch and iodepth_low will be same as
iodepth value.. but in my case I have to explicitly provide those
parameters. It looks like default value for ipdepth_bacch/low is 1.

Once I provide fio parameter "iodepth_batch=32" and "iodepth_low=32", I
see IO in batch before plug and after unplug from application.
I am able to get now max merged IO. Thanks for helping me on this.

~ Kashyap

>
> --
> Jens Axboe
>

RE: How to get more sequential IO merged at elevator

[Kashyap Desai]

> -----Original Message-----
> From: Jens Axboe [mailto:axboe@kernel.dk]
> Sent: Tuesday, May 06, 2014 7:47 PM
> To: Desai, Kashyap
> Cc: linux-scsi@vger.kernel.org
> Subject: Re: How to get more sequential IO merged at elevator
>
> On 05/06/2014 04:06 AM, Desai, Kashyap wrote:
> > I got some clue on what was going on while doing 4K sequential read
using
> fio.
> >
> > If I use ioengine in fio script as "libaio", I see " do_io_submit"
call
> plug/unplug the queue before submitting the IO.
> > It means after every IO, we expect to send IO immediately to the next
> layer. If at all there are any pending IO do merge.. but not due to
plugging.
> >
> > This is what happens on my test. Every time IO comes from application
> > with libaio engine, it send down to the elevator/io-scheduler because
> queue was unplugged in do_io_submit(). Moment I reduce the queue depth
> of the block device, merge start because of congestion at scsi mid
layer.
> >
> > If I use, mmap engine, I see merged IO coming to the device driver
> > because of plugging. I really don't know how it works, but gave a try
> > and found merge happen because of plugging. ( I confirm using
> > blktrace)
> >
> > Is there any ioengine in <fio> (or any other parameter setting), which
can
> use plugging mechanism of block layer to merge more IO other than mmap ?
>
> O_DIRECT IO is sync by nature, which is why it is sent off immediately
instead
> of held for potentially merging. mmap is not. You should be able to
provoke
> merging by submitting more than 1 IO at the time. See the iodepth_batch
> settings for fio.

Thanks Jens. I got your point about O_DIRECT IO. When I read <man> page of
<fio> it mentioned default iodepth_batch and iodepth_low will be same as
iodepth value.. but in my case I have to explicitly provide those
parameters. (I explore this after you pointed out ). It looks like default
value for iodepth_batch/low is not same as iodepth.

Once I provide fio parameter "iodepth_batch=32" and "iodepth_low=32", I
see IO in batch before plug and after unplug from application.
I am able to get now max merged IO. Thanks for helping me on this.

~ Kashyap
>
> --
> Jens Axboe
>