Why NVMe MSIx vectors affinity set across NUMA nodes?

Why NVMe MSIx vectors affinity set across NUMA nodes?

[Ganapatrao Kulkarni]

Hi,

I  have observed that NVMe driver splitting interrupt affinity of MSIx
vectors among available NUMA nodes,
any specific reason for that?

i see this is happening due to pci flag PCI_IRQ_AFFINITY is set in
function nvme_setup_io_queues

   nr_io_queues = pci_alloc_irq_vectors(pdev, 1, nr_io_queues,
                        PCI_IRQ_ALL_TYPES | PCI_IRQ_AFFINITY);

IMO, having all vectors on same node CPUs improves interrupt latency
than distributing among all nodes.


thanks
Ganapat

Why NVMe MSIx vectors affinity set across NUMA nodes?

[Keith Busch]

On Mon, Jan 22, 2018@09:55:55AM +0530, Ganapatrao Kulkarni wrote:
> Hi,
> 
> I  have observed that NVMe driver splitting interrupt affinity of MSIx
> vectors among available NUMA nodes,
> any specific reason for that?
> 
> i see this is happening due to pci flag PCI_IRQ_AFFINITY is set in
> function nvme_setup_io_queues
> 
>    nr_io_queues = pci_alloc_irq_vectors(pdev, 1, nr_io_queues,
>                         PCI_IRQ_ALL_TYPES | PCI_IRQ_AFFINITY);
> 
> IMO, having all vectors on same node CPUs improves interrupt latency
> than distributing among all nodes.

What affinity maps are you seeing? It's not supposed to share one vector
across two NUMA nodes, unless you simply don't have enough vectors.

Why NVMe MSIx vectors affinity set across NUMA nodes?

[Ganapatrao Kulkarni]

On Mon, Jan 22, 2018@10:44 PM, Keith Busch <keith.busch@intel.com> wrote:
> On Mon, Jan 22, 2018@09:55:55AM +0530, Ganapatrao Kulkarni wrote:
>> Hi,
>>
>> I  have observed that NVMe driver splitting interrupt affinity of MSIx
>> vectors among available NUMA nodes,
>> any specific reason for that?
>>
>> i see this is happening due to pci flag PCI_IRQ_AFFINITY is set in
>> function nvme_setup_io_queues
>>
>>    nr_io_queues = pci_alloc_irq_vectors(pdev, 1, nr_io_queues,
>>                         PCI_IRQ_ALL_TYPES | PCI_IRQ_AFFINITY);
>>
>> IMO, having all vectors on same node CPUs improves interrupt latency
>> than distributing among all nodes.
>
> What affinity maps are you seeing? It's not supposed to share one vector
> across two NUMA nodes, unless you simply don't have enough vectors.

There are 31 MSIx vectors getting initialised(one per NVMe queue) and
out of it, 0-15 are getting
affinity set to node 0 CPUs and vectors 16-30 are getting set to node 1.
My question is, why not set affinity to all vectors from same node
CPUs, what was need to use flag PCI_IRQ_AFFINITY?

thanks
Ganapat

Why NVMe MSIx vectors affinity set across NUMA nodes?

[Keith Busch]

On Mon, Jan 22, 2018@10:52:59PM +0530, Ganapatrao Kulkarni wrote:
> 
> There are 31 MSIx vectors getting initialised(one per NVMe queue) and
> out of it, 0-15 are getting
> affinity set to node 0 CPUs and vectors 16-30 are getting set to node 1.
> My question is, why not set affinity to all vectors from same node
> CPUs, what was need to use flag PCI_IRQ_AFFINITY?

I'm sorry, but I am not able to parse this question.

Why NVMe MSIx vectors affinity set across NUMA nodes?

[Ganapatrao Kulkarni]

On Mon, Jan 22, 2018@11:02 PM, Keith Busch <keith.busch@intel.com> wrote:
> On Mon, Jan 22, 2018@10:52:59PM +0530, Ganapatrao Kulkarni wrote:
>>
>> There are 31 MSIx vectors getting initialised(one per NVMe queue) and
>> out of it, 0-15 are getting
>> affinity set to node 0 CPUs and vectors 16-30 are getting set to node 1.
>> My question is, why not set affinity to all vectors from same node
>> CPUs, what was need to use flag PCI_IRQ_AFFINITY?
>
> I'm sorry, but I am not able to parse this question.

ok, let me rephrase,
what was the need to use flag PCI_IRQ_AFFINITY in NVMe driver?
i dont see this flag being used widely.

Why NVMe MSIx vectors affinity set across NUMA nodes?

[Keith Busch]

On Mon, Jan 22, 2018@11:25:45PM +0530, Ganapatrao Kulkarni wrote:
> what was the need to use flag PCI_IRQ_AFFINITY in NVMe driver?
> i dont see this flag being used widely.

The flag is how we get the affinity set in the first place. Without it,
we'd have to rely on user space to set irq affinities.

Why NVMe MSIx vectors affinity set across NUMA nodes?

[Ganapatrao Kulkarni]

On Mon, Jan 22, 2018@11:35 PM, Keith Busch <keith.busch@intel.com> wrote:
> On Mon, Jan 22, 2018@11:25:45PM +0530, Ganapatrao Kulkarni wrote:
>> what was the need to use flag PCI_IRQ_AFFINITY in NVMe driver?
>> i dont see this flag being used widely.
>
> The flag is how we get the affinity set in the first place. Without it,
> we'd have to rely on user space to set irq affinities.

AFAIK, usually drivers sets default affinity and it is likely be node
affinity for NUMA systems.
Later, it is the user-space(like irqbalance etc) which decides the
affinity not the driver.

Why NVMe MSIx vectors affinity set across NUMA nodes?

[Keith Busch]

On Mon, Jan 22, 2018@11:42:45PM +0530, Ganapatrao Kulkarni wrote:
> 
> AFAIK, usually drivers sets default affinity and it is likely be node
> affinity for NUMA systems.
> Later, it is the user-space(like irqbalance etc) which decides the
> affinity not the driver.

Relying on userspace to provide an optimal setting is a bad idea,
especially for NVMe where we have submission queue cpu affinity that
doesn't work very efficiently if the completion affinity doesn't match.

Why NVMe MSIx vectors affinity set across NUMA nodes?

[Sagi Grimberg]

>> AFAIK, usually drivers sets default affinity and it is likely be node
>> affinity for NUMA systems.
>> Later, it is the user-space(like irqbalance etc) which decides the
>> affinity not the driver.
> 
> Relying on userspace to provide an optimal setting is a bad idea,
> especially for NVMe where we have submission queue cpu affinity that
> doesn't work very efficiently if the completion affinity doesn't match.

I tend to agree, also application locality is equally as important as
device locality. so spreading across numa nodes will help applications
running on the far numa node as well.

Also, a recent thread [1] related to PCI_IRQ_AFFINITY not allowing
userspace to modify irq affinity suggested that maybe that can be
supported, but not sure what happened to it.

[1] https://www.spinics.net/lists/netdev/msg464301.html

Why NVMe MSIx vectors affinity set across NUMA nodes?

[Ganapatrao Kulkarni]

On Tue, Jan 23, 2018@7:00 PM, Sagi Grimberg <sagi@grimberg.me> wrote:
>
>>> AFAIK, usually drivers sets default affinity and it is likely be node
>>> affinity for NUMA systems.
>>> Later, it is the user-space(like irqbalance etc) which decides the
>>> affinity not the driver.
>>
>>
>> Relying on userspace to provide an optimal setting is a bad idea,
>> especially for NVMe where we have submission queue cpu affinity that
>> doesn't work very efficiently if the completion affinity doesn't match.
>
>
> I tend to agree, also application locality is equally as important as
> device locality. so spreading across numa nodes will help applications
> running on the far numa node as well.
>
> Also, a recent thread [1] related to PCI_IRQ_AFFINITY not allowing
> userspace to modify irq affinity suggested that maybe that can be
> supported, but not sure what happened to it.
>

application uses libnuma to align to numa locality.
here the driver is breaking the affinity.
certainly having affinity with remote node cpu will add latency to
interrupt response time.
here it is for some NVMe queues.

> [1] https://www.spinics.net/lists/netdev/msg464301.html

thanks
Ganapat

Why NVMe MSIx vectors affinity set across NUMA nodes?

[Keith Busch]

On Wed, Jan 24, 2018@07:47:44AM +0530, Ganapatrao Kulkarni wrote:
> application uses libnuma to align to numa locality.
> here the driver is breaking the affinity.
> certainly having affinity with remote node cpu will add latency to
> interrupt response time.
> here it is for some NVMe queues.

I bet you can't come up with an IRQ CPU affinity map that performs better
than the current setup.

Why NVMe MSIx vectors affinity set across NUMA nodes?

[Sagi Grimberg]

>> application uses libnuma to align to numa locality.
>> here the driver is breaking the affinity.
>> certainly having affinity with remote node cpu will add latency to
>> interrupt response time.
>> here it is for some NVMe queues.
> 
> I bet you can't come up with an IRQ CPU affinity map that performs better
> than the current setup.

:)

While I agree that managed affinity will probably get the optimal
affinitization in 99% of the cases, this is the second complaint we've
had that managed affinity breaks an existing user interface (even though
it was a sure way to allow userspace to screw up for years).

My mlx5 conversion ended up reverted due to that...

Why NVMe MSIx vectors affinity set across NUMA nodes?

[Keith Busch]

On Wed, Jan 24, 2018@09:39:02PM +0200, Sagi Grimberg wrote:
> 
> > > application uses libnuma to align to numa locality.
> > > here the driver is breaking the affinity.
> > > certainly having affinity with remote node cpu will add latency to
> > > interrupt response time.
> > > here it is for some NVMe queues.
> > 
> > I bet you can't come up with an IRQ CPU affinity map that performs better
> > than the current setup.
> 
> :)
> 
> While I agree that managed affinity will probably get the optimal
> affinitization in 99% of the cases, this is the second complaint we've
> had that managed affinity breaks an existing user interface (even though
> it was a sure way to allow userspace to screw up for years).

Well, this the only complaint for NVMe, and it doesn't seem aware of
how this work. If libnuma is used to run on a specific node, interrupts
will occur only on that node. An interrupt sent to a remote node means
you submitted a command from there, and handling the interrupt there is
cheaper than bouncing hot cache lines across nodes.