From: Douglas Gilbert <dgilbert@interlog.com>
To: Jack Wang <xjtuwjp@gmail.com>
Cc: Alireza Haghdoost <alireza@cs.umn.edu>,
linux-fsdevel@vger.kernel.org, linux-scsi@vger.kernel.org,
Jerry Fredin <jerry.fredin@netapp.com>
Subject: Re: Bypass block layer and Fill SCSI lower layer driver queue
Date: Wed, 18 Sep 2013 10:07:53 -0400 [thread overview]
Message-ID: <5239B3B9.8050209@interlog.com> (raw)
In-Reply-To: <52395D14.6010103@gmail.com>
On 13-09-18 03:58 AM, Jack Wang wrote:
> On 09/18/2013 08:41 AM, Alireza Haghdoost wrote:
>> Hi
>>
>> I am working on a high throughput and low latency application which
>> does not tolerate block layer overhead to send IO request directly to
>> fiber channel lower layer SCSI driver. I used to work with libaio but
>> currently I am looking for a way to by pass the block layer and send
>> SCSI commands from the application layer directly to the SCSI driver
>> using /dev/sgX device and ioctl() system call.
>>
>> I have noticed that sending IO request through sg device even with
>> nonblocking and direct IO flags is quite slow and does not fill up
>> lower layer SCSI driver TCQ queue. i.e IO depth or
>> /sys/block/sdX/in_flight is always ZERO. Therefore the application
>> throughput is even lower that sending IO request through block layer
>> with libaio and io_submit() system call. In both cases I used only one
>> IO context (or fd) and single threaded.
>>
> Hi Alireza,
>
> I think what you want is in_flight command scsi dispatch to low level
> device.
> I submit a simple patch to export device_busy
>
> http://www.spinics.net/lists/linux-scsi/msg68697.html
>
> I also notice fio sg engine will not fill queue properly, but haven't
> look into deeper.
>
> Cheers
> Jack
>
>> I have noticed that some well known benchmarking tools like fio does
>> not support IO depth for sg devices as well. Therefore, I was
>> wondering if it is feasible to bypass block layer and achieve higher
>> throughput and lower latency (for sending IO request only).
>>
>>
>> Any comment on my issue is highly appreciated.
I'm not sure if this is relevant to your problem but by
default both the bsg and sg drivers "queue at head"
when they inject SCSI commands into the block layer.
The bsg driver has a BSG_FLAG_Q_AT_TAIL flag to change
that queueing to what may be preferable for your purposes.
The sg driver could, but does not, support that flag.
Doug Gilbert
next prev parent reply other threads:[~2013-09-18 14:08 UTC|newest]
Thread overview: 8+ messages / expand[flat|nested] mbox.gz Atom feed top
2013-09-18 6:41 Bypass block layer and Fill SCSI lower layer driver queue Alireza Haghdoost
2013-09-18 7:58 ` Jack Wang
2013-09-18 14:07 ` Douglas Gilbert [this message]
2013-09-18 14:31 ` Boaz Harrosh
2013-09-27 6:06 ` Vladislav Bolkhovitin
2013-09-18 21:00 ` Nicholas A. Bellinger
2013-09-19 2:05 ` Alireza Haghdoost
2013-09-19 21:49 ` Nicholas A. Bellinger
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=5239B3B9.8050209@interlog.com \
--to=dgilbert@interlog.com \
--cc=alireza@cs.umn.edu \
--cc=jerry.fredin@netapp.com \
--cc=linux-fsdevel@vger.kernel.org \
--cc=linux-scsi@vger.kernel.org \
--cc=xjtuwjp@gmail.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).