Re: [LSF/MM TOPIC] multiqueue and interrupt assignment

[Sagi Grimberg <sagig@dev.mellanox.co.il>]

Hi Bart and Hannes,

> This topic indeed needs further attention. I also encountered this
> challenge while adding scsi-mq support to the SRP initiator driver. What
> I learned while working on the SRP driver is the following:
> - Although I agree that requests and interrupts should be processed on
>    the same processor (same physical chip) if the request has been
>    submitted from the CPU closest to the HBA, I'm not convinced that
>    processing request completions and interrupts on the same CPU core
>    yields the best performance. I would appreciate it if there would
>    remain some freedom in how to assign interrupts to CPU cores.

This is true not only for this reason. Some block storage transports
(e.g. srp/iser) share the HBA with the networking stack and possibly
with user-space workloads in the case of RDMA. This is why I don't see
how would MSIX assignments can be done anywhere other than user-space.

However what I think we can do is have blk-mq ask the drivers
information about the MSIX mappings. This concept was introduced in
2011 by Ben Hutchings with the CPU affinity reverse-mapping API [1].

Perhaps we'd want to have drivers provide blk-mq a struct cpu_rmap when
assigning the hctx mappings, or possibly per I/O if we want to be
agnostic to MSIX topology changes. I think this approach would solve
Hannes is experiencing.

Thoughts?

[1]:
commit c39649c331c70952700f99832b03f87e9d7f5b4b
Author: Ben Hutchings <bhutchings@solarflare.com>
Date:   Wed Jan 19 11:03:25 2011 +0000

     lib: cpu_rmap: CPU affinity reverse-mapping

     When initiating I/O on a multiqueue and multi-IRQ device, we may want
     to select a queue for which the response will be handled on the same
     or a nearby CPU.  This requires a reverse-map of IRQ affinity.  Add
     library functions to support a generic reverse-mapping from CPUs to
     objects with affinity and the specific case where the objects are
     IRQs.

     Signed-off-by: Ben Hutchings <bhutchings@solarflare.com>
     Signed-off-by: David S. Miller <davem@davemloft.net>