dm-multipath low performance with blk-mq

[hare@suse.de (Hannes Reinecke)]

On 02/04/2016 03:09 PM, Mike Snitzer wrote:
> On Thu, Feb 04 2016 at  8:58am -0500,
> Hannes Reinecke <hare@suse.de> wrote:
> 
>> On 02/04/2016 02:54 PM, Mike Snitzer wrote:
>>> On Thu, Feb 04 2016 at  1:54am -0500,
>>> Hannes Reinecke <hare@suse.de> wrote:
>>>
>> [ .. ]
>>>> But anyway, I'll be looking at your patches.
>>>
>>> Thanks, sadly none of the patches are going to fix the performance
>>> problems but I do think they are a step forward.
>>>
>> Hmm. I've got a slew of patches converting dm-mpath to use atomic_t
>> and bitops; with that we should be able to move to rcu for path
>> lookup and do away with most of the locking.
>> Quite raw, though; drop me a mail if you're interested.
> 
> Hmm, ok I just switched m->lock from spinlock_t to rwlock_t, see:
> https://git.kernel.org/cgit/linux/kernel/git/device-mapper/linux-dm.git/commit/?h=dm-4.6&id=a5226e23a6958ac9b7ade13a983604c43d232c7d
> 
> So any patch you'd have in this area would need rebasing.  I'll gladly
> look at what you have (even if it isn't rebased).  So yes please share.
> 
> (it could be that there isn't a big enough win associated with switching
> to rwlock_t -- that we could get away without doing that particular
> churn.. open to that if you think rwlock_t pointless given we'll take
> the write lock after repeat_count drops to 0)
> 
personally, I don't think the switching to a rwlock_t will buy us
anything; for a decent performance you have to set rq_min_io to 1
anyway, thereby defeating the purpose of the rwlock.

My thinking was rather a different direction:
Move the crucial bits of the multipath structure to atomics, and
split off the path selection code into one bit for selecting the
path within a path group, and another which switches the path groups.
When we do that we could use rcus for the paths themselves, and
would only need to take the spinlock if we need to switch path
groups. Which should be okay as switching path groups is
(potentially) a rather slow operation anyway.

Cheers,

Hannes
-- 
Dr. Hannes Reinecke		   Teamlead Storage & Networking
hare at suse.de			               +49 911 74053 688
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 N?rnberg
GF: F. Imend?rffer, J. Smithard, J. Guild, D. Upmanyu, G. Norton
HRB 21284 (AG N?rnberg)
-------------- next part --------------
A non-text attachment was scrubbed...
Name: dm-atomic.tar.gz
Type: application/x-gzip
Size: 10584 bytes
Desc: not available
URL: <http://lists.infradead.org/pipermail/linux-nvme/attachments/20160204/b81a6d8b/attachment.bin>