Re: [LSF/MM ATTEND] multipath redesign and dm blk-mq issues

[Mike Snitzer <snitzer@redhat.com>]

On Thu, Jan 28 2016 at  9:11pm -0500,
Benjamin Marzinski <bmarzins@redhat.com> wrote:

> On Thu, Jan 28, 2016 at 07:33:16PM -0600, Benjamin Marzinski wrote:
> > On Thu, Jan 28, 2016 at 05:37:33PM -0500, Mike Snitzer wrote:
> > > On Thu, Jan 28 2016 at  4:23pm -0500,
> > > Benjamin Marzinski <bmarzins@redhat.com> wrote:
> 
> > > blk-mq's .queue_rq hook is the logical place to do the mpath mapping, as
> > > it deals with getting a request from the underlying paths.
> > > 
> > > blk-mq's .map_queue is all about mapping sw to hw queues.  It is very
> > > blk-mq specific and isn't something DM has a roll in -- cannot yet see
> > > why it'd need to.
> > 
> > At the moment, we only have one hwqueue.  But we could have one hwqueue
> > per path. Then queue_rq would just be in charge of handing the requst
> > down to the underlying device.  In that setup, instead using a default
> > mapping of all swqueues to one hwqueue in .map_queue, we would be
> > mapping to the hardware queue for the path.  I'd have to look through
> > the blk-mq code more to know if one of these methods has an obvious
> > advantage, but it seems like this way, if different cpus were using
> > different paths (with the per-cpu load-balancing), you wouldn't
> > constantly be accessing the hwqueue from different cpus. Although I
> > suppose you may do better just by leaving multipath_map where it is now,
> > and just adjusting the number of hardware queues. Speaking of which,
> > have you tried fiddling around with that in your tests?
> > 
> 
> O.k. a quick look shows that map_queue get called so often that any sort
> of dynamic mapping there would be a pain.  But constantly having all the
> cpus accessing one hwqueue seems like it could be part of the
> performance issue. So, it would definitely be worth playing around with
> that.

Yeah, I have a patch that makes both hw_queues and queue_depth tunable:
http://git.kernel.org/cgit/linux/kernel/git/snitzer/linux.git/commit/?h=devel2&id=99ebcaf36d9d1fa3acec98492c36664d57ba8fbd

Increasing nr_hw_queues doesn't help (in fact it hurts, going from 1 to
2 results in a drop from ~970K to ~945K IOPs, to 4 I get ~930K).

Will need to revisit the blk-mq code in general to appreciate how the
sw -> hw mapping will scale, etc.

And verify assumptions like: the top-level dm-mpath rq->mq_ctx->cpu
matches the underlying path's clone->mq_ctx->cpu