[LSF/MM ATTEND] interest in blk-mq, scsi-mq, dm-cache, dm-thinp, dm-*

[LSF/MM ATTEND] interest in blk-mq, scsi-mq, dm-cache, dm-thinp, dm-*

[Mike Snitzer]

I would like to attend to participate in discussions related to topics
listed in the subject.  As a maintainer of DM I'd be interested to
learn/discuss areas that should become a development focus in the months
following LSF.

Thanks,
Mike

Re: [LSF/MM ATTEND] interest in blk-mq, scsi-mq, dm-cache, dm-thinp, dm-*

[Hannes Reinecke]

On 01/10/2014 07:27 PM, Mike Snitzer wrote:
> I would like to attend to participate in discussions related to topics
> listed in the subject.  As a maintainer of DM I'd be interested to
> learn/discuss areas that should become a development focus in the months
> following LSF.
> 
+1

I've been thinking on (re-) implementing multipathing on top of
blk-mq, and would like to discuss the probability of which.
There are some design decisions in blk-mq (eg statically allocating
the number of queues) which do not play well with that.

Cheers,

Hannes
-- 
Dr. Hannes Reinecke		      zSeries & Storage
hare@suse.de			      +49 911 74053 688
SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg
GF: J. Hawn, J. Guild, F. Imendörffer, HRB 16746 (AG Nürnberg)
--
To unsubscribe from this list: send the line "unsubscribe linux-scsi" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [LSF/MM ATTEND] interest in blk-mq, scsi-mq, dm-cache, dm-thinp, dm-*

[Mike Christie]

On 01/13/2014 05:36 AM, Hannes Reinecke wrote:
> On 01/10/2014 07:27 PM, Mike Snitzer wrote:
>> I would like to attend to participate in discussions related to topics
>> listed in the subject.  As a maintainer of DM I'd be interested to
>> learn/discuss areas that should become a development focus in the months
>> following LSF.
>>
> +1
> 
> I've been thinking on (re-) implementing multipathing on top of
> blk-mq, and would like to discuss the probability of which.
> There are some design decisions in blk-mq (eg statically allocating
> the number of queues) which do not play well with that.
> 

I think I have been thinking about going a completely different direction.

The thing about dm-multipath is that request based adds the extra queue
locking and that of course is bad. In our testing it is a major perf
issue. We got things like ioscheduling though.

If we went back to bio based multipathing then it turns out that when
scsi also supports multiqueue then it all works pretty nicely. There is
room for improvement in general like with some dm allocations being
numa/cpu aware, but the request_queue locking issues we have go away and
it is very simple code wise.

We could go the route of making request based dm-multipath:

1. aware of underlying multiqueue devices. So just basically keep what
we have more or less but then have dm-multipath make a request that can
be sent to a multiqueue device then call blk_mq_insert_request. This
would all be hidden by nice interfaces that hide if it is multiqueue
underlying device or not.

2. make dm-multipath do multiqueue (so implement map_queue, queue_rq,
etc) and also making it aware of underlying multiqueue devices.

#1 just keeps the existing request spin_lock problem so there is not
much point other than just getting things working.

#2 is a good deal of work and what does it end up buying us over just
making multipath bio based. We lose iosched support. If we are going to
make advanced multiqueue ioschedulers that rely on request structs then
#2 could be useful.

Re: [LSF/MM ATTEND] interest in blk-mq, scsi-mq, dm-cache, dm-thinp, dm-*

[Hannes Reinecke]

On 01/24/2014 03:37 AM, Mike Christie wrote:
> On 01/13/2014 05:36 AM, Hannes Reinecke wrote:
>> On 01/10/2014 07:27 PM, Mike Snitzer wrote:
>>> I would like to attend to participate in discussions related to topics
>>> listed in the subject.  As a maintainer of DM I'd be interested to
>>> learn/discuss areas that should become a development focus in the months
>>> following LSF.
>>>
>> +1
>>
>> I've been thinking on (re-) implementing multipathing on top of
>> blk-mq, and would like to discuss the probability of which.
>> There are some design decisions in blk-mq (eg statically allocating
>> the number of queues) which do not play well with that.
>>
> 
> I think I have been thinking about going a completely different direction.
> 
> The thing about dm-multipath is that request based adds the extra queue
> locking and that of course is bad. In our testing it is a major perf
> issue. We got things like ioscheduling though.
> 
Indeed. And without that we cannot do true load-balancing.

> If we went back to bio based multipathing then it turns out that when
> scsi also supports multiqueue then it all works pretty nicely. There is
> room for improvement in general like with some dm allocations being
> numa/cpu aware, but the request_queue locking issues we have go away and
> it is very simple code wise.
> 
If and when.

The main issue I see with that is that it might take some time (if
ever) for SCSI LLDDs to go fully multiqueue.
In fact, I strongly suspect that only newer LLDDs will ever support
multiqueue; for the older cards the HW interface it too much tied
to single queue operations.

> We could go the route of making request based dm-multipath:
> 
> 1. aware of underlying multiqueue devices. So just basically keep what
> we have more or less but then have dm-multipath make a request that can
> be sent to a multiqueue device then call blk_mq_insert_request. This
> would all be hidden by nice interfaces that hide if it is multiqueue
> underlying device or not.
> 
> 2. make dm-multipath do multiqueue (so implement map_queue, queue_rq,
> etc) and also making it aware of underlying multiqueue devices.
> 
> #1 just keeps the existing request spin_lock problem so there is not
> much point other than just getting things working.
> 
> #2 is a good deal of work and what does it end up buying us over just
> making multipath bio based. We lose iosched support. If we are going to
> make advanced multiqueue ioschedulers that rely on request structs then
> #2 could be useful.
> 

Obviously we need iosched support when going multiqueue.
I wouldn't dream of dropping them.

So my overall idea here is to move multipath over to block-mq,
making each path identical to one queue.
(As mentioned above, currently every single FC HBA exposes a single
HW queue anyway)
The ioschedulers would be moved to the map_queue function.

This approach has several issues which I would like to discuss:
- block-mq ctx allocation currently is static. This doesn't play
  well with multipathing, were paths (=queues) might get configured
  on-the-fly.
- Queues might be coming from different HBAs; one would need to
  audit the block-mq stuff if that's possible.

Cheers,

Hannes
-- 
Dr. Hannes Reinecke		      zSeries & Storage
hare@suse.de			      +49 911 74053 688
SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg
GF: J. Hawn, J. Guild, F. Imendörffer, HRB 16746 (AG Nürnberg)
--
To unsubscribe from this list: send the line "unsubscribe linux-scsi" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [LSF/MM ATTEND] interest in blk-mq, scsi-mq, dm-cache, dm-thinp, dm-*

[Mike Christie]

On 01/24/2014 01:12 AM, Hannes Reinecke wrote:
> On 01/24/2014 03:37 AM, Mike Christie wrote:
>> On 01/13/2014 05:36 AM, Hannes Reinecke wrote:
>>> On 01/10/2014 07:27 PM, Mike Snitzer wrote:
>>>> I would like to attend to participate in discussions related to topics
>>>> listed in the subject.  As a maintainer of DM I'd be interested to
>>>> learn/discuss areas that should become a development focus in the months
>>>> following LSF.
>>>>
>>> +1
>>>
>>> I've been thinking on (re-) implementing multipathing on top of
>>> blk-mq, and would like to discuss the probability of which.
>>> There are some design decisions in blk-mq (eg statically allocating
>>> the number of queues) which do not play well with that.
>>>
>>
>> I think I have been thinking about going a completely different direction.
>>
>> The thing about dm-multipath is that request based adds the extra queue
>> locking and that of course is bad. In our testing it is a major perf
>> issue. We got things like ioscheduling though.
>>
> Indeed. And without that we cannot do true load-balancing.

I think I might be using some terms differently. You do not need
ioscheduling like deadline and cfq for path load balancing in multipath
do you? Do you mean IO merging?

With noop load balancing works as expected.

> 
>> If we went back to bio based multipathing then it turns out that when
>> scsi also supports multiqueue then it all works pretty nicely. There is
>> room for improvement in general like with some dm allocations being
>> numa/cpu aware, but the request_queue locking issues we have go away and
>> it is very simple code wise.
>>
> If and when.
> 
> The main issue I see with that is that it might take some time (if
> ever) for SCSI LLDDs to go fully multiqueue.
> In fact, I strongly suspect that only newer LLDDs will ever support
> multiqueue; for the older cards the HW interface it too much tied
> to single queue operations.

Maybe we are talking about different things. We will do software/partial
blk/scsi mq support soon and dm-multipath is not compatible with
underlying devices that do multiqueue as it is now. I am just saying
that dm cannot do its current bio/request cloning and then call
blk_insert_cloned_request on a request when the underlying device does
even the partial/software multiqueue that people are making patches for
scsi for.

We can hack in support like I mentioned or we can do a bigger change
like yours or we can do bio based or something else. The point is that
something will need to be done and it is not dependent on hw multiqueue
support.

However, on that subject of hw multiqueue support, Bart is experimenting
with some tricks for SRP so it is not that far off. That is why he keeps
asking about hw mutiqueue support for scsi.

> 
>> We could go the route of making request based dm-multipath:
>>
>> 1. aware of underlying multiqueue devices. So just basically keep what
>> we have more or less but then have dm-multipath make a request that can
>> be sent to a multiqueue device then call blk_mq_insert_request. This
>> would all be hidden by nice interfaces that hide if it is multiqueue
>> underlying device or not.
>>
>> 2. make dm-multipath do multiqueue (so implement map_queue, queue_rq,
>> etc) and also making it aware of underlying multiqueue devices.
>>
>> #1 just keeps the existing request spin_lock problem so there is not
>> much point other than just getting things working.
>>
>> #2 is a good deal of work and what does it end up buying us over just
>> making multipath bio based. We lose iosched support. If we are going to
>> make advanced multiqueue ioschedulers that rely on request structs then
>> #2 could be useful.
>>
> 
> Obviously we need iosched support when going multiqueue.
> I wouldn't dream of dropping them.

I was just saying that blk-mq already dropped support for advanced
ioscheds like deadline and cfq. It basically only does noop where there
is a list/fifo.

> 
> So my overall idea here is to move multipath over to block-mq,
> making each path identical to one queue.

There are too many queues now :) Do you mean request_queue or
blk_mq_hw_ctx or blk_mq_ctx?


> (As mentioned above, currently every single FC HBA exposes a single
> HW queue anyway)
> The ioschedulers would be moved to the map_queue function.
> 

Are you maybe talking about path selectors here? map_queue would be too
early for ioscheduling like deadline/cfq or you would need more
callouts. Depending on how you are implementing blk multipath multiqueue
I think map_queue could be good for path selection.


> This approach has several issues which I would like to discuss:

Hey, yeah, maybe best to discuss at LSF.

Re: [LSF/MM ATTEND] interest in blk-mq, scsi-mq, dm-cache, dm-thinp, dm-*

[Nicholas A. Bellinger]

On Fri, 2014-01-10 at 13:27 -0500, Mike Snitzer wrote:
> I would like to attend to participate in discussions related to topics
> listed in the subject.  As a maintainer of DM I'd be interested to
> learn/discuss areas that should become a development focus in the months
> following LSF.

+1 on the scsi-mq bit.

--nab

Re: [LSF/MM ATTEND] interest in blk-mq, scsi-mq, dm-cache, dm-thinp, dm-*

[Sagi Grimberg]

On 1/10/2014 8:27 PM, Mike Snitzer wrote:
> I would like to attend to participate in discussions related to topics
> listed in the subject.  As a maintainer of DM I'd be interested to
> learn/discuss areas that should become a development focus in the months
> following LSF.
>

+1 for scsi-mq.

Re: [LSF/MM ATTEND] interest in blk-mq, scsi-mq, dm-cache, dm-thinp, dm-*

[Sagi Grimberg]

On 1/16/2014 6:34 PM, Sagi Grimberg wrote:
> On 1/10/2014 8:27 PM, Mike Snitzer wrote:
>> I would like to attend to participate in discussions related to topics
>> listed in the subject.  As a maintainer of DM I'd be interested to
>> learn/discuss areas that should become a development focus in the months
>> following LSF.
>>
>
> +1 for scsi-mq.

Sparing a few words on why I voted on this topic,
As part of me being involved in fast RDMA based initiators (iSER, SRP) I 
can say that SCSI layer
became a real performance bottleneck. Implementations bypassing SCSI has 
started to pop-up lately.
scsi-mq can and should boost performance and it is important that we 
work also on LLDs to
provide a total solution.

Sagi.

Re: [LSF/MM ATTEND] interest in blk-mq, scsi-mq, dm-cache, dm-thinp, dm-*

[Giridhar Malavali]

[-- Attachment #1: Type: text/plain, Size: 562 bytes --]



On 1/10/14 10:27 AM, "Mike Snitzer" <snitzer@redhat.com> wrote:

>I would like to attend to participate in discussions related to topics
>listed in the subject.  As a maintainer of DM I'd be interested to
>learn/discuss areas that should become a development focus in the months
>following LSF.

+1 for scsi-mq. 

-- Giri

>
>Thanks,
>Mike
>--
>To unsubscribe from this list: send the line "unsubscribe linux-scsi" in
>the body of a message to majordomo@vger.kernel.org
>More majordomo info at  http://vger.kernel.org/majordomo-info.html


[-- Attachment #2: winmail.dat --]
[-- Type: application/ms-tnef, Size: 4034 bytes --]

Re: [LSF/MM ATTEND] interest in blk-mq, scsi-mq, dm-cache, dm-thinp, dm-*

[Bart Van Assche]

On 01/16/14 23:35, Giridhar Malavali wrote:
> On 1/10/14 10:27 AM, "Mike Snitzer" <snitzer@redhat.com> wrote:
>> I would like to attend to participate in discussions related to topics
>> listed in the subject.  As a maintainer of DM I'd be interested to
>> learn/discuss areas that should become a development focus in the months
>> following LSF.
> 
> +1 for scsi-mq. 

(removed lsf-pc from CC-list)

Hello Giridhar,

I definitely appreciate QLogic's continuous efforts to improve
performance of their initiator and target drivers. Via your e-mail I
learned that QLogic is looking at scsi-mq, which is great news. However,
the following issue might have to be addressed before the qla2xxx driver
can fully take advantage of the scsi-mq work:
https://bugzilla.kernel.org/show_bug.cgi?id=69201. It would be
appreciated if someone could have a look at the measurements described
in that bugzilla entry.

Thanks,

Bart.

Re: [LSF/MM ATTEND] interest in blk-mq, scsi-mq, dm-cache, dm-thinp, dm-*

[Giridhar Malavali]

[-- Attachment #1: Type: text/plain, Size: 1269 bytes --]



On 1/22/14 1:54 AM, "Bart Van Assche" <bvanassche@acm.org> wrote:

>On 01/16/14 23:35, Giridhar Malavali wrote:
>> On 1/10/14 10:27 AM, "Mike Snitzer" <snitzer@redhat.com> wrote:
>>> I would like to attend to participate in discussions related to topics
>>> listed in the subject.  As a maintainer of DM I'd be interested to
>>> learn/discuss areas that should become a development focus in the
>>>months
>>> following LSF.
>> 
>> +1 for scsi-mq. 
>
>(removed lsf-pc from CC-list)
>
>Hello Giridhar,
>
>I definitely appreciate QLogic's continuous efforts to improve
>performance of their initiator and target drivers. Via your e-mail I
>learned that QLogic is looking at scsi-mq, which is great news. However,
>the following issue might have to be addressed before the qla2xxx driver
>can fully take advantage of the scsi-mq work:
>https://bugzilla.kernel.org/show_bug.cgi?id=69201. It would be
>appreciated if someone could have a look at the measurements described
>in that bugzilla entry.

One of our internal runs have indicated lock contention issues and we are
working on this actively.
We will be posting the patches once we are done with it.

Thanks for bringing this to our attention.

-- Giri


>
>Thanks,
>
>Bart.


[-- Attachment #2: winmail.dat --]
[-- Type: application/ms-tnef, Size: 4526 bytes --]