Re: [RFC PATCH 1/4] scsi: Allow drivers to set BLK_MQ_F_BLOCKING

[Ming Lei <ming.lei@redhat.com>]

On Tue, Mar 08, 2022 at 07:17:13PM -0600, Mike Christie wrote:
> On 3/8/22 6:53 PM, Ming Lei wrote:
> > On Mon, Mar 07, 2022 at 06:39:54PM -0600, Mike Christie wrote:
> >> The software iscsi driver's queuecommand can block and taking the extra
> >> hop from kblockd to its workqueue results in a performance hit. Allowing
> >> it to set BLK_MQ_F_BLOCKING and transmit from that context directly
> >> results in a 20-30% improvement in IOPs for workloads like:
> >>
> >> fio --filename=/dev/sdb --direct=1 --rw=randrw --bs=4k --ioengine=libaio
> >> --iodepth=128  --numjobs=1
> >>
> >> and for all write workloads.
> > 
> > This single patch shouldn't make any difference for iscsi, so please
> > make it as last one if performance improvement data is provided
> > in commit log.
> 
> Ok.
> 
> > 
> > Also is there performance effect for other worloads? such as multiple
> > jobs? iscsi is SQ hardware, so if driver is blocked in ->queuecommand()
> > via BLK_MQ_F_BLOCKING, other contexts can't submit IO to scsi ML any more.
> 
> If you mean multiple jobs running on the same connection/session then
> they are all serialized now. A connection can only do 1 cmd at a time.
> There's a big mutex around it in the network layer, so multiple jobs
> just suck no matter what.

I guess one block device can only bind to one isci connection, given the
1 cmd per connection limit, so looks multiple jobs is fine.

> 
> If you mean multiple jobs from different connection/sessions, then the
> iscsi code with this patchset blocks only because the network layer
> takes a mutex for a short time. We configure it to not block for things
> like socket space, memory allocations, we do zero copy IO normally, etc
> so it's quick.
> 
> We also can do up to workqueues max_active limit worth of calls so
> other things can normally send IO. We haven't found a need to increase
> it yet.
 
I meant that hctx->run_work is required for blk-mq to dispatch IO, iscsi is
SQ HBA, so there is only single work_struct. If one context is blocked in
->queue_rq or ->queuecommand, other contexts can't submit IO to driver any
more.


Thanks,
Ming