[ATTEND] scsi-mq prototype discussion

[ATTEND] scsi-mq prototype discussion

[Nicholas A. Bellinger]

Hello,

I would like to attend the 2013 Kernel Summit.

At the summit, I would like to discuss scsi-mq, a high performance SCSI
initiator prototype that utilizes the next-generation blk-mq effort by
Jens Axboe.  The long-term goal is a path to move beyond the
long-standing small block random I/O limitations vs. raw make_request
based drivers of the existing Linux/SCSI client stack.

Along with using blk-mq's excellent native per-cpu primitive + NUMA
local friendly queuing of pre-allocated struct request descriptor
memory, the scsi-mq prototype currently avoids all I/O fast-path access
of legacy scsi_host->host_lock, and bypasses existing scsi_request_fn()
dispatch into scsi-mq enabled LLD code.

It also allows scsi-core to eliminate all fast-path memory allocations
using struct scsi_cmnd + $LLD_CMD pre-allocations based on a per struct
blk_mq_hw_ctx -> scsi_device->sdev_mq_req context, along with per
scsi_cmnd descriptor pre-allocation of SGL and sense buffer memory.

So far the initial conversion of virtio-scsi + scsi-debug LLDs has been
completed.  Also, the intention is to keep the conversion requirements
for existing LLDs to scsi-mq as simple as possible.

There are still many areas that have been conveniently left out of the
initial prototype, including proper fast-path get_device() +
put_device() reference counting, a functioning scsi-generic IOCTL,
anything close to per struct scsi_device error handling, amongst other
things..

Drilling down the work items ahead of a real mainline push is high on
priority list for discussion.

The parties to be included in such a discussion are:

  - Jens Axboe (blk-mq author)
  - James Bottomley (scsi maintainer)
  - Christoph Hellwig (scsi)
  - Martin Petersen (scsi)
  - Tejun Heo (block + libata)
  - Hannes Reinecke (scsi error recovery)
  - Kent Overstreet (block, per-cpu ida)
  - Stephen Cameron (scsi-over-pcie driver)
  - Andrew Vasquez (qla2xxx LLD)
  - James Smart (lpfc LLD)

Thank you,

--nab

Re: [Ksummit-2013-discuss] [ATTEND] scsi-mq prototype discussion

[Greg KH]

On Thu, Jul 11, 2013 at 05:23:32PM -0700, Nicholas A. Bellinger wrote:
> Drilling down the work items ahead of a real mainline push is high on
> priority list for discussion.
> 
> The parties to be included in such a discussion are:
> 
>   - Jens Axboe (blk-mq author)
>   - James Bottomley (scsi maintainer)
>   - Christoph Hellwig (scsi)
>   - Martin Petersen (scsi)
>   - Tejun Heo (block + libata)
>   - Hannes Reinecke (scsi error recovery)
>   - Kent Overstreet (block, per-cpu ida)
>   - Stephen Cameron (scsi-over-pcie driver)
>   - Andrew Vasquez (qla2xxx LLD)
>   - James Smart (lpfc LLD)

Isn't this something that should have been discussed at the storage
mini-summit a few months ago?  It seems very specific to one subsystem
to be a kernel summit topic, don't you think?

thanks,

greg k-h

Re: [Ksummit-2013-discuss] [ATTEND] scsi-mq prototype discussion

[Nicholas A. Bellinger]

On Thu, 2013-07-11 at 18:02 -0700, Greg KH wrote:
> On Thu, Jul 11, 2013 at 05:23:32PM -0700, Nicholas A. Bellinger wrote:
> > Drilling down the work items ahead of a real mainline push is high on
> > priority list for discussion.
> > 
> > The parties to be included in such a discussion are:
> > 
> >   - Jens Axboe (blk-mq author)
> >   - James Bottomley (scsi maintainer)
> >   - Christoph Hellwig (scsi)
> >   - Martin Petersen (scsi)
> >   - Tejun Heo (block + libata)
> >   - Hannes Reinecke (scsi error recovery)
> >   - Kent Overstreet (block, per-cpu ida)
> >   - Stephen Cameron (scsi-over-pcie driver)
> >   - Andrew Vasquez (qla2xxx LLD)
> >   - James Smart (lpfc LLD)
> 
> Isn't this something that should have been discussed at the storage
> mini-summit a few months ago?

The scsi-mq prototype, along with blk-mq (in it's current form) did not
exist a few short months ago.  ;)

>  It seems very specific to one subsystem to be a kernel summit topic,
> don't you think?

It's no more subsystem specific than half of the other proposals so far,
and given it's reach across multiple subsystems (block, scsi, target),
and the amount of off-list interest on the topic, I think it would make
a good candidate for discussion.

Thanks,

--nab 

Re: [Ksummit-2013-discuss] [ATTEND] scsi-mq prototype discussion

[Hannes Reinecke]

On 07/12/2013 03:33 AM, Nicholas A. Bellinger wrote:
> On Thu, 2013-07-11 at 18:02 -0700, Greg KH wrote:
>> On Thu, Jul 11, 2013 at 05:23:32PM -0700, Nicholas A. Bellinger wrote:
>>> Drilling down the work items ahead of a real mainline push is high on
>>> priority list for discussion.
>>>
>>> The parties to be included in such a discussion are:
>>>
>>>   - Jens Axboe (blk-mq author)
>>>   - James Bottomley (scsi maintainer)
>>>   - Christoph Hellwig (scsi)
>>>   - Martin Petersen (scsi)
>>>   - Tejun Heo (block + libata)
>>>   - Hannes Reinecke (scsi error recovery)
>>>   - Kent Overstreet (block, per-cpu ida)
>>>   - Stephen Cameron (scsi-over-pcie driver)
>>>   - Andrew Vasquez (qla2xxx LLD)
>>>   - James Smart (lpfc LLD)
>>
>> Isn't this something that should have been discussed at the storage
>> mini-summit a few months ago?
> 
> The scsi-mq prototype, along with blk-mq (in it's current form) did not
> exist a few short months ago.  ;)
> 
>>  It seems very specific to one subsystem to be a kernel summit topic,
>> don't you think?
> 
> It's no more subsystem specific than half of the other proposals so far,
> and given it's reach across multiple subsystems (block, scsi, target),
> and the amount of off-list interest on the topic, I think it would make
> a good candidate for discussion.
> 
And it'll open up new approaches which previously were dismissed,
like re-implementing multipathing on top of scsi-mq, giving us the
single scsi device like other UNIX systems.

Also I do think there's quite some synergy to be had, as with blk-mq
we could nail each queue to a processor, which would eliminate the
need for locking.
Which could be useful for other subsystems, too.

Cheers,

Hannes
-- 
Dr. Hannes Reinecke		      zSeries & Storage
hare@suse.de			      +49 911 74053 688
SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg
GF: J. Hawn, J. Guild, F. Imendörffer, HRB 16746 (AG Nürnberg)

Re: [Ksummit-2013-discuss] [ATTEND] scsi-mq prototype discussion

[James Bottomley]

On Fri, 2013-07-12 at 12:52 +0200, Hannes Reinecke wrote:
> On 07/12/2013 03:33 AM, Nicholas A. Bellinger wrote:
> > On Thu, 2013-07-11 at 18:02 -0700, Greg KH wrote:
> >> On Thu, Jul 11, 2013 at 05:23:32PM -0700, Nicholas A. Bellinger wrote:
> >>> Drilling down the work items ahead of a real mainline push is high on
> >>> priority list for discussion.
> >>>
> >>> The parties to be included in such a discussion are:
> >>>
> >>>   - Jens Axboe (blk-mq author)
> >>>   - James Bottomley (scsi maintainer)
> >>>   - Christoph Hellwig (scsi)
> >>>   - Martin Petersen (scsi)
> >>>   - Tejun Heo (block + libata)
> >>>   - Hannes Reinecke (scsi error recovery)
> >>>   - Kent Overstreet (block, per-cpu ida)
> >>>   - Stephen Cameron (scsi-over-pcie driver)
> >>>   - Andrew Vasquez (qla2xxx LLD)
> >>>   - James Smart (lpfc LLD)
> >>
> >> Isn't this something that should have been discussed at the storage
> >> mini-summit a few months ago?
> > 
> > The scsi-mq prototype, along with blk-mq (in it's current form) did not
> > exist a few short months ago.  ;)
> > 
> >>  It seems very specific to one subsystem to be a kernel summit topic,
> >> don't you think?
> > 
> > It's no more subsystem specific than half of the other proposals so far,
> > and given it's reach across multiple subsystems (block, scsi, target),
> > and the amount of off-list interest on the topic, I think it would make
> > a good candidate for discussion.
> > 
> And it'll open up new approaches which previously were dismissed,
> like re-implementing multipathing on top of scsi-mq, giving us the
> single scsi device like other UNIX systems.
> 
> Also I do think there's quite some synergy to be had, as with blk-mq
> we could nail each queue to a processor, which would eliminate the
> need for locking.
> Which could be useful for other subsystems, too.

Lets start with discussing this on the list, please, and then see where
we go from there ...

James

Re: [Ksummit-2013-discuss] [ATTEND] scsi-mq prototype discussion

[Nicholas A. Bellinger]

On Sat, 2013-07-13 at 06:53 +0000, James Bottomley wrote:
> On Fri, 2013-07-12 at 12:52 +0200, Hannes Reinecke wrote:
> > On 07/12/2013 03:33 AM, Nicholas A. Bellinger wrote:
> > > On Thu, 2013-07-11 at 18:02 -0700, Greg KH wrote:
> > >> On Thu, Jul 11, 2013 at 05:23:32PM -0700, Nicholas A. Bellinger wrote:
> > >>> Drilling down the work items ahead of a real mainline push is high on
> > >>> priority list for discussion.
> > >>>
> > >>> The parties to be included in such a discussion are:
> > >>>
> > >>>   - Jens Axboe (blk-mq author)
> > >>>   - James Bottomley (scsi maintainer)
> > >>>   - Christoph Hellwig (scsi)
> > >>>   - Martin Petersen (scsi)
> > >>>   - Tejun Heo (block + libata)
> > >>>   - Hannes Reinecke (scsi error recovery)
> > >>>   - Kent Overstreet (block, per-cpu ida)
> > >>>   - Stephen Cameron (scsi-over-pcie driver)
> > >>>   - Andrew Vasquez (qla2xxx LLD)
> > >>>   - James Smart (lpfc LLD)
> > >>
> > >> Isn't this something that should have been discussed at the storage
> > >> mini-summit a few months ago?
> > > 
> > > The scsi-mq prototype, along with blk-mq (in it's current form) did not
> > > exist a few short months ago.  ;)
> > > 
> > >>  It seems very specific to one subsystem to be a kernel summit topic,
> > >> don't you think?
> > > 
> > > It's no more subsystem specific than half of the other proposals so far,
> > > and given it's reach across multiple subsystems (block, scsi, target),
> > > and the amount of off-list interest on the topic, I think it would make
> > > a good candidate for discussion.
> > > 
> > And it'll open up new approaches which previously were dismissed,
> > like re-implementing multipathing on top of scsi-mq, giving us the
> > single scsi device like other UNIX systems.
> > 
> > Also I do think there's quite some synergy to be had, as with blk-mq
> > we could nail each queue to a processor, which would eliminate the
> > need for locking.
> > Which could be useful for other subsystems, too.
> 
> Lets start with discussing this on the list, please, and then see where
> we go from there ...
> 

Yes, the discussion is beginning to make it's way to the list.  I've
mostly been waiting for blk-mq to get a wider review before taking the
early scsi-mq prototype driver to a larger public audience.

Primarily, I'm now reaching out to the people most effected by existing
scsi_request_fn() based performance limitations.  Most of them have
abandoned existing scsi_request_fn() based logic in favor of raw block
make_request() based drivers, and are now estimating the amount of
effort to move to an scsi-mq based approach.

Regardless, as the prototype progresses over the next months, having a
face-to-face discussion with the key parties in the room would be very
helpful given the large amount of effort involved to actually make this
type of generational shift in SCSI actually happen.

--nab

Re: [Ksummit-2013-discuss] [ATTEND] scsi-mq prototype discussion

[Jens Axboe]

On Tue, Jul 16 2013, Nicholas A. Bellinger wrote:
> On Sat, 2013-07-13 at 06:53 +0000, James Bottomley wrote:
> > On Fri, 2013-07-12 at 12:52 +0200, Hannes Reinecke wrote:
> > > On 07/12/2013 03:33 AM, Nicholas A. Bellinger wrote:
> > > > On Thu, 2013-07-11 at 18:02 -0700, Greg KH wrote:
> > > >> On Thu, Jul 11, 2013 at 05:23:32PM -0700, Nicholas A. Bellinger wrote:
> > > >>> Drilling down the work items ahead of a real mainline push is high on
> > > >>> priority list for discussion.
> > > >>>
> > > >>> The parties to be included in such a discussion are:
> > > >>>
> > > >>>   - Jens Axboe (blk-mq author)
> > > >>>   - James Bottomley (scsi maintainer)
> > > >>>   - Christoph Hellwig (scsi)
> > > >>>   - Martin Petersen (scsi)
> > > >>>   - Tejun Heo (block + libata)
> > > >>>   - Hannes Reinecke (scsi error recovery)
> > > >>>   - Kent Overstreet (block, per-cpu ida)
> > > >>>   - Stephen Cameron (scsi-over-pcie driver)
> > > >>>   - Andrew Vasquez (qla2xxx LLD)
> > > >>>   - James Smart (lpfc LLD)
> > > >>
> > > >> Isn't this something that should have been discussed at the storage
> > > >> mini-summit a few months ago?
> > > > 
> > > > The scsi-mq prototype, along with blk-mq (in it's current form) did not
> > > > exist a few short months ago.  ;)
> > > > 
> > > >>  It seems very specific to one subsystem to be a kernel summit topic,
> > > >> don't you think?
> > > > 
> > > > It's no more subsystem specific than half of the other proposals so far,
> > > > and given it's reach across multiple subsystems (block, scsi, target),
> > > > and the amount of off-list interest on the topic, I think it would make
> > > > a good candidate for discussion.
> > > > 
> > > And it'll open up new approaches which previously were dismissed,
> > > like re-implementing multipathing on top of scsi-mq, giving us the
> > > single scsi device like other UNIX systems.
> > > 
> > > Also I do think there's quite some synergy to be had, as with blk-mq
> > > we could nail each queue to a processor, which would eliminate the
> > > need for locking.
> > > Which could be useful for other subsystems, too.
> > 
> > Lets start with discussing this on the list, please, and then see where
> > we go from there ...
> > 
> 
> Yes, the discussion is beginning to make it's way to the list.  I've
> mostly been waiting for blk-mq to get a wider review before taking the
> early scsi-mq prototype driver to a larger public audience.
> 
> Primarily, I'm now reaching out to the people most effected by existing
> scsi_request_fn() based performance limitations.  Most of them have
> abandoned existing scsi_request_fn() based logic in favor of raw block
> make_request() based drivers, and are now estimating the amount of
> effort to move to an scsi-mq based approach.
> 
> Regardless, as the prototype progresses over the next months, having a
> face-to-face discussion with the key parties in the room would be very
> helpful given the large amount of effort involved to actually make this
> type of generational shift in SCSI actually happen.

There's a certain amount of overlap with the aio/O_DIRECT work as well.
But if it's not a general session, could always be a BOF or something.

I'll second the argument that most technical topics probably DO belong
in a topic related workshop. But that leaves us with basically only
process related topics at KS, I don't think it hurts to have a bit of
tech meat on the bone too. At least I personally miss that part of KS
from years gone by.

-- 
Jens Axboe

Re: [Ksummit-2013-discuss] [ATTEND] scsi-mq prototype discussion

[James Bottomley]

On Tue, 2013-07-16 at 15:15 -0600, Jens Axboe wrote:
> On Tue, Jul 16 2013, Nicholas A. Bellinger wrote:
> > On Sat, 2013-07-13 at 06:53 +0000, James Bottomley wrote:
> > > On Fri, 2013-07-12 at 12:52 +0200, Hannes Reinecke wrote:
> > > > On 07/12/2013 03:33 AM, Nicholas A. Bellinger wrote:
> > > > > On Thu, 2013-07-11 at 18:02 -0700, Greg KH wrote:
> > > > >> On Thu, Jul 11, 2013 at 05:23:32PM -0700, Nicholas A. Bellinger wrote:
> > > > >>> Drilling down the work items ahead of a real mainline push is high on
> > > > >>> priority list for discussion.
> > > > >>>
> > > > >>> The parties to be included in such a discussion are:
> > > > >>>
> > > > >>>   - Jens Axboe (blk-mq author)
> > > > >>>   - James Bottomley (scsi maintainer)
> > > > >>>   - Christoph Hellwig (scsi)
> > > > >>>   - Martin Petersen (scsi)
> > > > >>>   - Tejun Heo (block + libata)
> > > > >>>   - Hannes Reinecke (scsi error recovery)
> > > > >>>   - Kent Overstreet (block, per-cpu ida)
> > > > >>>   - Stephen Cameron (scsi-over-pcie driver)
> > > > >>>   - Andrew Vasquez (qla2xxx LLD)
> > > > >>>   - James Smart (lpfc LLD)
> > > > >>
> > > > >> Isn't this something that should have been discussed at the storage
> > > > >> mini-summit a few months ago?
> > > > > 
> > > > > The scsi-mq prototype, along with blk-mq (in it's current form) did not
> > > > > exist a few short months ago.  ;)
> > > > > 
> > > > >>  It seems very specific to one subsystem to be a kernel summit topic,
> > > > >> don't you think?
> > > > > 
> > > > > It's no more subsystem specific than half of the other proposals so far,
> > > > > and given it's reach across multiple subsystems (block, scsi, target),
> > > > > and the amount of off-list interest on the topic, I think it would make
> > > > > a good candidate for discussion.
> > > > > 
> > > > And it'll open up new approaches which previously were dismissed,
> > > > like re-implementing multipathing on top of scsi-mq, giving us the
> > > > single scsi device like other UNIX systems.
> > > > 
> > > > Also I do think there's quite some synergy to be had, as with blk-mq
> > > > we could nail each queue to a processor, which would eliminate the
> > > > need for locking.
> > > > Which could be useful for other subsystems, too.
> > > 
> > > Lets start with discussing this on the list, please, and then see where
> > > we go from there ...
> > > 
> > 
> > Yes, the discussion is beginning to make it's way to the list.  I've
> > mostly been waiting for blk-mq to get a wider review before taking the
> > early scsi-mq prototype driver to a larger public audience.
> > 
> > Primarily, I'm now reaching out to the people most effected by existing
> > scsi_request_fn() based performance limitations.  Most of them have
> > abandoned existing scsi_request_fn() based logic in favor of raw block
> > make_request() based drivers, and are now estimating the amount of
> > effort to move to an scsi-mq based approach.
> > 
> > Regardless, as the prototype progresses over the next months, having a
> > face-to-face discussion with the key parties in the room would be very
> > helpful given the large amount of effort involved to actually make this
> > type of generational shift in SCSI actually happen.
> 
> There's a certain amount of overlap with the aio/O_DIRECT work as well.
> But if it's not a general session, could always be a BOF or something.
> 
> I'll second the argument that most technical topics probably DO belong
> in a topic related workshop. But that leaves us with basically only
> process related topics at KS, I don't think it hurts to have a bit of
> tech meat on the bone too. At least I personally miss that part of KS
> from years gone by.

Heh well, given that most of the block mq discussions at LSF have been
you saying you really should get around to cleaning up and posting the
code, you'll understand my wanting to see that happen first ...

I suppose we could try to run a storage workshop within KS, but I think
most of the mini summit slots have already gone.  There's also plumbers
if all slots are gone (I would say that, being biased and on the
programme committee) Ric is running the storage and Filesystems MC

http://www.linuxplumbersconf.org/2013/ocw/events/LPC2013/tracks/159

James

Re: [Ksummit-2013-discuss] [ATTEND] scsi-mq prototype discussion

[Ric Wheeler]

On 07/17/2013 12:52 AM, James Bottomley wrote:
> On Tue, 2013-07-16 at 15:15 -0600, Jens Axboe wrote:
>> On Tue, Jul 16 2013, Nicholas A. Bellinger wrote:
>>> On Sat, 2013-07-13 at 06:53 +0000, James Bottomley wrote:
>>>> On Fri, 2013-07-12 at 12:52 +0200, Hannes Reinecke wrote:
>>>>> On 07/12/2013 03:33 AM, Nicholas A. Bellinger wrote:
>>>>>> On Thu, 2013-07-11 at 18:02 -0700, Greg KH wrote:
>>>>>>> On Thu, Jul 11, 2013 at 05:23:32PM -0700, Nicholas A. Bellinger wrote:
>>>>>>>> Drilling down the work items ahead of a real mainline push is high on
>>>>>>>> priority list for discussion.
>>>>>>>>
>>>>>>>> The parties to be included in such a discussion are:
>>>>>>>>
>>>>>>>>    - Jens Axboe (blk-mq author)
>>>>>>>>    - James Bottomley (scsi maintainer)
>>>>>>>>    - Christoph Hellwig (scsi)
>>>>>>>>    - Martin Petersen (scsi)
>>>>>>>>    - Tejun Heo (block + libata)
>>>>>>>>    - Hannes Reinecke (scsi error recovery)
>>>>>>>>    - Kent Overstreet (block, per-cpu ida)
>>>>>>>>    - Stephen Cameron (scsi-over-pcie driver)
>>>>>>>>    - Andrew Vasquez (qla2xxx LLD)
>>>>>>>>    - James Smart (lpfc LLD)
>>>>>>> Isn't this something that should have been discussed at the storage
>>>>>>> mini-summit a few months ago?
>>>>>> The scsi-mq prototype, along with blk-mq (in it's current form) did not
>>>>>> exist a few short months ago.  ;)
>>>>>>
>>>>>>>   It seems very specific to one subsystem to be a kernel summit topic,
>>>>>>> don't you think?
>>>>>> It's no more subsystem specific than half of the other proposals so far,
>>>>>> and given it's reach across multiple subsystems (block, scsi, target),
>>>>>> and the amount of off-list interest on the topic, I think it would make
>>>>>> a good candidate for discussion.
>>>>>>
>>>>> And it'll open up new approaches which previously were dismissed,
>>>>> like re-implementing multipathing on top of scsi-mq, giving us the
>>>>> single scsi device like other UNIX systems.
>>>>>
>>>>> Also I do think there's quite some synergy to be had, as with blk-mq
>>>>> we could nail each queue to a processor, which would eliminate the
>>>>> need for locking.
>>>>> Which could be useful for other subsystems, too.
>>>> Lets start with discussing this on the list, please, and then see where
>>>> we go from there ...
>>>>
>>> Yes, the discussion is beginning to make it's way to the list.  I've
>>> mostly been waiting for blk-mq to get a wider review before taking the
>>> early scsi-mq prototype driver to a larger public audience.
>>>
>>> Primarily, I'm now reaching out to the people most effected by existing
>>> scsi_request_fn() based performance limitations.  Most of them have
>>> abandoned existing scsi_request_fn() based logic in favor of raw block
>>> make_request() based drivers, and are now estimating the amount of
>>> effort to move to an scsi-mq based approach.
>>>
>>> Regardless, as the prototype progresses over the next months, having a
>>> face-to-face discussion with the key parties in the room would be very
>>> helpful given the large amount of effort involved to actually make this
>>> type of generational shift in SCSI actually happen.
>> There's a certain amount of overlap with the aio/O_DIRECT work as well.
>> But if it's not a general session, could always be a BOF or something.
>>
>> I'll second the argument that most technical topics probably DO belong
>> in a topic related workshop. But that leaves us with basically only
>> process related topics at KS, I don't think it hurts to have a bit of
>> tech meat on the bone too. At least I personally miss that part of KS
>> from years gone by.
> Heh well, given that most of the block mq discussions at LSF have been
> you saying you really should get around to cleaning up and posting the
> code, you'll understand my wanting to see that happen first ...
>
> I suppose we could try to run a storage workshop within KS, but I think
> most of the mini summit slots have already gone.  There's also plumbers
> if all slots are gone (I would say that, being biased and on the
> programme committee) Ric is running the storage and Filesystems MC
>
> http://www.linuxplumbersconf.org/2013/ocw/events/LPC2013/tracks/159
>
> James
>

And we still are looking for suggested topics - it would be great to have the 
multi-queue work at plumbers.

You can post a proposal for it (or other topics) here:

http://www.linuxplumbersconf.org/2013/ocw/events/LPC2013/proposals

Ric

Re: [Ksummit-2013-discuss] [ATTEND] scsi-mq prototype discussion

[Jens Axboe]

On Fri, Jul 19 2013, Ric Wheeler wrote:
> down the work items ahead of a real mainline push is high on
> >>>>>>>>priority list for discussion.
> >>>>>>>>
> >>>>>>>>The parties to be included in such a discussion are:
> >>>>>>>>
> >>>>>>>>   - Jens Axboe (blk-mq author)
> >>>>>>>>   - James Bottomley (scsi maintainer)
> >>>>>>>>   - Christoph Hellwig (scsi)
> >>>>>>>>   - Martin Petersen (scsi)
> >>>>>>>>   - Tejun Heo (block + libata)
> >>>>>>>>   - Hannes Reinecke (scsi error recovery)
> >>>>>>>>   - Kent Overstreet (block, per-cpu ida)
> >>>>>>>>   - Stephen Cameron (scsi-over-pcie driver)
> >>>>>>>>   - Andrew Vasquez (qla2xxx LLD)
> >>>>>>>>   - James Smart (lpfc LLD)
> >>>>>>>Isn't this something that should have been discussed at the storage
> >>>>>>>mini-summit a few months ago?
> >>>>>>The scsi-mq prototype, along with blk-mq (in it's current form) did not
> >>>>>>exist a few short months ago.  ;)
> >>>>>>
> >>>>>>>  It seems very specific to one subsystem to be a kernel summit topic,
> >>>>>>>don't you think?
> >>>>>>It's no more subsystem specific than half of the other proposals so far,
> >>>>>>and given it's reach across multiple subsystems (block, scsi, target),
> >>>>>>and the amount of off-list interest on the topic, I think it would make
> >>>>>>a good candidate for discussion.
> >>>>>>
> >>>>>And it'll open up new approaches which previously were dismissed,
> >>>>>like re-implementing multipathing on top of scsi-mq, giving us the
> >>>>>single scsi device like other UNIX systems.
> >>>>>
> >>>>>Also I do think there's quite some synergy to be had, as with blk-mq
> >>>>>we could nail each queue to a processor, which would eliminate the
> >>>>>need for locking.
> >>>>>Which could be useful for other subsystems, too.
> >>>>Lets start with discussing this on the list, please, and then see where
> >>>>we go from there ...
> >>>>
> >>>Yes, the discussion is beginning to make it's way to the list.  I've
> >>>mostly been waiting for blk-mq to get a wider review before taking the
> >>>early scsi-mq prototype driver to a larger public audience.
> >>>
> >>>Primarily, I'm now reaching out to the people most effected by existing
> >>>scsi_request_fn() based performance limitations.  Most of them have
> >>>abandoned existing scsi_request_fn() based logic in favor of raw block
> >>>make_request() based drivers, and are now estimating the amount of
> >>>effort to move to an scsi-mq based approach.
> >>>
> >>>Regardless, as the prototype progresses over the next months, having a
> >>>face-to-face discussion with the key parties in the room would be very
> >>>helpful given the large amount of effort involved to actually make this
> >>>type of generational shift in SCSI actually happen.
> >>There's a certain amount of overlap with the aio/O_DIRECT work as well.
> >>But if it's not a general session, could always be a BOF or something.
> >>
> >>I'll second the argument that most technical topics probably DO belong
> >>in a topic related workshop. But that leaves us with basically only
> >>process related topics at KS, I don't think it hurts to have a bit of
> >>tech meat on the bone too. At least I personally miss that part of KS
> >>from years gone by.
> >Heh well, given that most of the block mq discussions at LSF have been
> >you saying you really should get around to cleaning up and posting the
> >code, you'll understand my wanting to see that happen first ...
> >
> >I suppose we could try to run a storage workshop within KS, but I think
> >most of the mini summit slots have already gone.  There's also plumbers
> >if all slots are gone (I would say that, being biased and on the
> >programme committee) Ric is running the storage and Filesystems MC
> >
> >http://www.linuxplumbersconf.org/2013/ocw/events/LPC2013/tracks/159
> >
> >James
> >
> 
> And we still are looking for suggested topics - it would be great to have
> the multi-queue work at plumbers.
> 
> You can post a proposal for it (or other topics) here:
> 
> http://www.linuxplumbersconf.org/2013/ocw/events/LPC2013/proposals

FWIW, I can't make Plumbers this year, unfortunately.

-- 
Jens Axboe

Re: [Ksummit-2013-discuss] [ATTEND] scsi-mq prototype discussion

[Nicholas A. Bellinger]

On Wed, 2013-07-17 at 04:52 +0000, James Bottomley wrote:
> On Tue, 2013-07-16 at 15:15 -0600, Jens Axboe wrote:
> > On Tue, Jul 16 2013, Nicholas A. Bellinger wrote:
> > > On Sat, 2013-07-13 at 06:53 +0000, James Bottomley wrote:

<SNIP>

> > > > Lets start with discussing this on the list, please, and then see where
> > > > we go from there ...
> > > > 
> > > 
> > > Yes, the discussion is beginning to make it's way to the list.  I've
> > > mostly been waiting for blk-mq to get a wider review before taking the
> > > early scsi-mq prototype driver to a larger public audience.
> > > 
> > > Primarily, I'm now reaching out to the people most effected by existing
> > > scsi_request_fn() based performance limitations.  Most of them have
> > > abandoned existing scsi_request_fn() based logic in favor of raw block
> > > make_request() based drivers, and are now estimating the amount of
> > > effort to move to an scsi-mq based approach.
> > > 
> > > Regardless, as the prototype progresses over the next months, having a
> > > face-to-face discussion with the key parties in the room would be very
> > > helpful given the large amount of effort involved to actually make this
> > > type of generational shift in SCSI actually happen.
> > 
> > There's a certain amount of overlap with the aio/O_DIRECT work as well.
> > But if it's not a general session, could always be a BOF or something.
> > 
> > I'll second the argument that most technical topics probably DO belong
> > in a topic related workshop. But that leaves us with basically only
> > process related topics at KS, I don't think it hurts to have a bit of
> > tech meat on the bone too. At least I personally miss that part of KS
> > from years gone by.
> 
> Heh well, given that most of the block mq discussions at LSF have been
> you saying you really should get around to cleaning up and posting the
> code, you'll understand my wanting to see that happen first ...
> 
> I suppose we could try to run a storage workshop within KS, but I think
> most of the mini summit slots have already gone.

That would be great, given there is a reasonable level of interest from
various parities, and the pain threshold for existing scsi small block
random I/O performance is high..

When will we know if there is a workshop / mini summit slot available..?

(CC'ing Mike Christie as well for open-iscsi/scsi-mq bits)

> There's also plumbers
> if all slots are gone (I would say that, being biased and on the
> programme committee) Ric is running the storage and Filesystems MC
> 
> http://www.linuxplumbersconf.org/2013/ocw/events/LPC2013/tracks/159

FYI, I'm not trying to 'sell' scsi-mq to the larger community yet, but
rather interested in getting the right scsi/block/LLD people in the same
room at KS for an hour or two to discuss implementation details, given
the scope of the effort involved.

--nab

Re: [Ksummit-2013-discuss] [ATTEND] scsi-mq prototype discussion

[James Bottomley]

On Fri, 2013-07-19 at 14:22 -0700, Nicholas A. Bellinger wrote:
> On Wed, 2013-07-17 at 04:52 +0000, James Bottomley wrote:
> > On Tue, 2013-07-16 at 15:15 -0600, Jens Axboe wrote:
> > > On Tue, Jul 16 2013, Nicholas A. Bellinger wrote:
> > > > On Sat, 2013-07-13 at 06:53 +0000, James Bottomley wrote:
> 
> <SNIP>
> 
> > > > > Lets start with discussing this on the list, please, and then see where
> > > > > we go from there ...
> > > > > 
> > > > 
> > > > Yes, the discussion is beginning to make it's way to the list.  I've
> > > > mostly been waiting for blk-mq to get a wider review before taking the
> > > > early scsi-mq prototype driver to a larger public audience.
> > > > 
> > > > Primarily, I'm now reaching out to the people most effected by existing
> > > > scsi_request_fn() based performance limitations.  Most of them have
> > > > abandoned existing scsi_request_fn() based logic in favor of raw block
> > > > make_request() based drivers, and are now estimating the amount of
> > > > effort to move to an scsi-mq based approach.
> > > > 
> > > > Regardless, as the prototype progresses over the next months, having a
> > > > face-to-face discussion with the key parties in the room would be very
> > > > helpful given the large amount of effort involved to actually make this
> > > > type of generational shift in SCSI actually happen.
> > > 
> > > There's a certain amount of overlap with the aio/O_DIRECT work as well.
> > > But if it's not a general session, could always be a BOF or something.
> > > 
> > > I'll second the argument that most technical topics probably DO belong
> > > in a topic related workshop. But that leaves us with basically only
> > > process related topics at KS, I don't think it hurts to have a bit of
> > > tech meat on the bone too. At least I personally miss that part of KS
> > > from years gone by.
> > 
> > Heh well, given that most of the block mq discussions at LSF have been
> > you saying you really should get around to cleaning up and posting the
> > code, you'll understand my wanting to see that happen first ...
> > 
> > I suppose we could try to run a storage workshop within KS, but I think
> > most of the mini summit slots have already gone.
> 
> That would be great, given there is a reasonable level of interest from
> various parities, and the pain threshold for existing scsi small block
> random I/O performance is high..
> 
> When will we know if there is a workshop / mini summit slot available..?
> 
> (CC'ing Mike Christie as well for open-iscsi/scsi-mq bits)
> 
> > There's also plumbers
> > if all slots are gone (I would say that, being biased and on the
> > programme committee) Ric is running the storage and Filesystems MC
> > 
> > http://www.linuxplumbersconf.org/2013/ocw/events/LPC2013/tracks/159
> 
> FYI, I'm not trying to 'sell' scsi-mq to the larger community yet, but
> rather interested in getting the right scsi/block/LLD people in the same
> room at KS for an hour or two to discuss implementation details, given
> the scope of the effort involved.

Well, so that's why I think plumbers might be a better venue: we'll have
more of the actual storage people there.  Usually we get at most 2-3
storage people to KS compared to the 25 or so we usually have at LSF ...
that makes KS not a very good venue for storage centric discussions.

James

Re: [Ksummit-2013-discuss] [ATTEND] scsi-mq prototype discussion

[Nicholas A. Bellinger]

On Fri, 2013-07-19 at 21:46 +0000, James Bottomley wrote:
> On Fri, 2013-07-19 at 14:22 -0700, Nicholas A. Bellinger wrote:
> > On Wed, 2013-07-17 at 04:52 +0000, James Bottomley wrote:
> > > On Tue, 2013-07-16 at 15:15 -0600, Jens Axboe wrote:
> > > > On Tue, Jul 16 2013, Nicholas A. Bellinger wrote:
> > > > > On Sat, 2013-07-13 at 06:53 +0000, James Bottomley wrote:
> > 
> > <SNIP>
> > 
> > > > > > Lets start with discussing this on the list, please, and then see where
> > > > > > we go from there ...
> > > > > > 
> > > > > 
> > > > > Yes, the discussion is beginning to make it's way to the list.  I've
> > > > > mostly been waiting for blk-mq to get a wider review before taking the
> > > > > early scsi-mq prototype driver to a larger public audience.
> > > > > 
> > > > > Primarily, I'm now reaching out to the people most effected by existing
> > > > > scsi_request_fn() based performance limitations.  Most of them have
> > > > > abandoned existing scsi_request_fn() based logic in favor of raw block
> > > > > make_request() based drivers, and are now estimating the amount of
> > > > > effort to move to an scsi-mq based approach.
> > > > > 
> > > > > Regardless, as the prototype progresses over the next months, having a
> > > > > face-to-face discussion with the key parties in the room would be very
> > > > > helpful given the large amount of effort involved to actually make this
> > > > > type of generational shift in SCSI actually happen.
> > > > 
> > > > There's a certain amount of overlap with the aio/O_DIRECT work as well.
> > > > But if it's not a general session, could always be a BOF or something.
> > > > 
> > > > I'll second the argument that most technical topics probably DO belong
> > > > in a topic related workshop. But that leaves us with basically only
> > > > process related topics at KS, I don't think it hurts to have a bit of
> > > > tech meat on the bone too. At least I personally miss that part of KS
> > > > from years gone by.
> > > 
> > > Heh well, given that most of the block mq discussions at LSF have been
> > > you saying you really should get around to cleaning up and posting the
> > > code, you'll understand my wanting to see that happen first ...
> > > 
> > > I suppose we could try to run a storage workshop within KS, but I think
> > > most of the mini summit slots have already gone.
> > 
> > That would be great, given there is a reasonable level of interest from
> > various parities, and the pain threshold for existing scsi small block
> > random I/O performance is high..
> > 
> > When will we know if there is a workshop / mini summit slot available..?
> > 
> > (CC'ing Mike Christie as well for open-iscsi/scsi-mq bits)
> > 
> > > There's also plumbers
> > > if all slots are gone (I would say that, being biased and on the
> > > programme committee) Ric is running the storage and Filesystems MC
> > > 
> > > http://www.linuxplumbersconf.org/2013/ocw/events/LPC2013/tracks/159
> > 
> > FYI, I'm not trying to 'sell' scsi-mq to the larger community yet, but
> > rather interested in getting the right scsi/block/LLD people in the same
> > room at KS for an hour or two to discuss implementation details, given
> > the scope of the effort involved.
> 
> Well, so that's why I think plumbers might be a better venue: we'll have
> more of the actual storage people there.  Usually we get at most 2-3
> storage people to KS compared to the 25 or so we usually have at LSF ...
> that makes KS not a very good venue for storage centric discussions.
> 

The most relevant people for the discussion are Jens, Hannes, Christoph,
Tejun, Martin, Mike, and you.

I know these folks are regular attendees for KS, but typically not for
plumbers, which is why I made this KS topic proposal in the first place.

--nab

Re: [Ksummit-2013-discuss] [ATTEND] scsi-mq prototype discussion

[scameron]

On Tue, Jul 16, 2013 at 02:07:29PM -0700, Nicholas A. Bellinger wrote:
> On Sat, 2013-07-13 at 06:53 +0000, James Bottomley wrote:
> > On Fri, 2013-07-12 at 12:52 +0200, Hannes Reinecke wrote:
> > > On 07/12/2013 03:33 AM, Nicholas A. Bellinger wrote:
> > > > On Thu, 2013-07-11 at 18:02 -0700, Greg KH wrote:
> > > >> On Thu, Jul 11, 2013 at 05:23:32PM -0700, Nicholas A. Bellinger wrote:
> > > >>> Drilling down the work items ahead of a real mainline push is high on
> > > >>> priority list for discussion.
> > > >>>
> > > >>> The parties to be included in such a discussion are:
> > > >>>
> > > >>>   - Jens Axboe (blk-mq author)
> > > >>>   - James Bottomley (scsi maintainer)
> > > >>>   - Christoph Hellwig (scsi)
> > > >>>   - Martin Petersen (scsi)
> > > >>>   - Tejun Heo (block + libata)
> > > >>>   - Hannes Reinecke (scsi error recovery)
> > > >>>   - Kent Overstreet (block, per-cpu ida)
> > > >>>   - Stephen Cameron (scsi-over-pcie driver)
> > > >>>   - Andrew Vasquez (qla2xxx LLD)
> > > >>>   - James Smart (lpfc LLD)
> > > >>
> > > >> Isn't this something that should have been discussed at the storage
> > > >> mini-summit a few months ago?
> > > > 
> > > > The scsi-mq prototype, along with blk-mq (in it's current form) did not
> > > > exist a few short months ago.  ;)
> > > > 
> > > >>  It seems very specific to one subsystem to be a kernel summit topic,
> > > >> don't you think?
> > > > 
> > > > It's no more subsystem specific than half of the other proposals so far,
> > > > and given it's reach across multiple subsystems (block, scsi, target),
> > > > and the amount of off-list interest on the topic, I think it would make
> > > > a good candidate for discussion.
> > > > 
> > > And it'll open up new approaches which previously were dismissed,
> > > like re-implementing multipathing on top of scsi-mq, giving us the
> > > single scsi device like other UNIX systems.
> > > 
> > > Also I do think there's quite some synergy to be had, as with blk-mq
> > > we could nail each queue to a processor, which would eliminate the
> > > need for locking.
> > > Which could be useful for other subsystems, too.
> > 
> > Lets start with discussing this on the list, please, and then see where
> > we go from there ...
> > 
> 
> Yes, the discussion is beginning to make it's way to the list.  I've
> mostly been waiting for blk-mq to get a wider review before taking the
> early scsi-mq prototype driver to a larger public audience.
> 
> Primarily, I'm now reaching out to the people most effected by existing
> scsi_request_fn() based performance limitations.  Most of them have
> abandoned existing scsi_request_fn() based logic in favor of raw block
> make_request() based drivers, and are now estimating the amount of
> effort to move to an scsi-mq based approach.
> 
> Regardless, as the prototype progresses over the next months, having a
> face-to-face discussion with the key parties in the room would be very
> helpful given the large amount of effort involved to actually make this
> type of generational shift in SCSI actually happen.

I'd be very interested in attending this, if invited.

-- steve

Re: scsi-mq prototype

[Bart Van Assche]

On 07/12/13 02:23, Nicholas A. Bellinger wrote:
> [ ... ] I would like to discuss scsi-mq, a high performance SCSI
> initiator prototype that utilizes blk-mq [ ... ]

(replying to an e-mail of four months ago)

It took a while but I finally found some time to look further into
blk-mq and scsi-mq. Did I see correctly that the scsi-mq prototype
implements the so-called hw queues in the SCSI core and not in the SCSI
LLDs ? Sorry but that approach looks wrong to me. Several high-end
storage adapters already support multiple queues today. E.g. the NVMe
specification supports up to 64.000 I/O queues. With InfiniBand HCA's it
is possible to create multiple queues between initiator and target for
e.g. the iSER and SRP protocols to increase throughput. When there are
multiple queues between initiator and target there is a potential that
SCSI commands get reordered. The SCSI specifications define whether or
not it is allowed to change the submission order of commands. As an
example, if two atomic writes are submitted by the same CPU neither the
block layer nor the SCSI core nor the SCSI LLD nor the SCSI target is
allowed to reorder these. With the scsi-mq prototype present in your
repo the only way for a SCSI LLD to guarantee that SCSI commands for
which the submission order must be preserved do not get reordered is
either to limit the queue depth to one or to use a single queue for
communication with the target. I don't think this is what we want.

In other words, the so-called hw queue concept must be mapped onto
queues managed by the SCSI LLD and not onto queues managed by the SCSI
core. That will allow the block layer and the SCSI core to preserve SCSI
command submission order when necessary by queueing ordered commands on
the same hardware queue.

See also
https://git.kernel.org/cgit/linux/kernel/git/nab/target-pending.git/log/?h=scsi-mq.

Bart.

Re: scsi-mq prototype

[Nicholas A. Bellinger]

On Fri, 2013-11-29 at 15:08 +0100, Bart Van Assche wrote:
> On 07/12/13 02:23, Nicholas A. Bellinger wrote:
> > [ ... ] I would like to discuss scsi-mq, a high performance SCSI
> > initiator prototype that utilizes blk-mq [ ... ]
> 
> (replying to an e-mail of four months ago)
> 
> It took a while but I finally found some time to look further into
> blk-mq and scsi-mq. Did I see correctly that the scsi-mq prototype
> implements the so-called hw queues in the SCSI core and not in the SCSI
> LLDs ?

No.  The hw queues are coming from blk-mq, and there is no extra queues
implemented by scsi-mq.

>  Sorry but that approach looks wrong to me. Several high-end
> storage adapters already support multiple queues today. E.g. the NVMe
> specification supports up to 64.000 I/O queues. With InfiniBand HCA's it
> is possible to create multiple queues between initiator and target for
> e.g. the iSER and SRP protocols to increase throughput. When there are
> multiple queues between initiator and target there is a potential that
> SCSI commands get reordered. The SCSI specifications define whether or
> not it is allowed to change the submission order of commands. As an
> example, if two atomic writes are submitted by the same CPU neither the
> block layer nor the SCSI core nor the SCSI LLD nor the SCSI target is
> allowed to reorder these. With the scsi-mq prototype present in your
> repo the only way for a SCSI LLD to guarantee that SCSI commands for
> which the submission order must be preserved do not get reordered is
> either to limit the queue depth to one or to use a single queue for
> communication with the target. I don't think this is what we want.
> 

In case you haven't already noticed, all of the scsi-mq logic is running
with nr_hw_queues=1, which by itself gives a huge performance + latency
improvement vs. existing scsi_request_fn() based logic.

Frankly, getting this to work properly for all cases is currently higher
on my priority list than enabling nr_hw_queues > 1.

> In other words, the so-called hw queue concept must be mapped onto
> queues managed by the SCSI LLD and not onto queues managed by the SCSI
> core. That will allow the block layer and the SCSI core to preserve SCSI
> command submission order when necessary by queueing ordered commands on
> the same hardware queue.
> 

Perhaps once nr_hw_queues > 1 logic is enabled in scsi-mq, you can send
a patch series to address your concerns.

Without patches, your observations aren't particular helpful.

--nab