Re: [RFC] PCI_IRQ_AFFINITY limits MSI-X allocation on 384 CPU / 1000+ NVMe system

[Thomas Gleixner <tglx@linutronix.de>]

On Thu, Jul 30 2026 at 07:28, Keith Busch wrote:
> On Thu, Jul 30, 2026 at 03:16:46PM +0200, Thomas Gleixner wrote:
>> As I told you before it's simple math and resource exhaustion due to the
>> way how NVME is implemented and the x86 vector space limitations.
>> 
>> If you want that to behave differently, then you have to talk to the
>> NVME people as I told you before.
>
> We can introduce a module parameter to throttle down the maximum number
> of IO queues to allocate per controller. I don't think the driver can
> automatically reason out what the correct number should be because it
> doesn't know how many devices it's going to see.

I might be missing something here as usual, but the number of io queues
per controller does not affect that at all as I explained before.

The number of io queues just influences the spreading, but not the
number of vectors required. The queues segment the system into IO zones
from 1 up to min(max_queues, number of CPUs).

What influences the vector allocation is the number of sets, i.e. when
the PCI controller has enough queues to support read queues, then you
have one set for the default queues and one set for the read queues. If
not it's only the default set. This is the per set resource consumption:

   nr queues ==  | nr_masks | CPUs per mask | vectors | vectors
   nr_interrupts |          |               | per CPU | total
   ------------------------------------------------------------------------
   1             | 1        | 64            | 1       | 64
   2             | 2        | 32            | 1       | 64
   4             | 4        | 16            | 1       | 64
   ...
   32            | 32       | 2             | 1       | 64
   64            | 64       | 1             | 1       | 64

So it does not matter whether you limit the number of IO queues per PCI
controller because the per CPU vector consumption is always _one_ per
set.

The number of interrupts is irrelevant because they are just virtual
Linux software interrupt numbers. They are only limited by memory, but
if those tiny allocations fail then the NVME device is the least of your
worries.

The only relevant limiting factor is the x86 vector space, but who needs
more than 640K RAM and 256 interrupt vectors :)

This all was designed to make multi queue operations simpler in terms of
affinity management and also to guarantee that CPU hot-unplug, which is
required for suspend/hibernate, does not end up trying to move hundreds
of queues to the boot CPU and ends up in vector exhaustion on x86.

The whole concept of multi-queue is built around that.

If the last CPU in the affinity mask of a queue goes down, then the
multi-queue core has also disabled request queueing for that queue,
drained the outstanding requests and shut down the queue in the
controller. Then the interrupt core deals with the still requested
interrupt and instead of moving it to an online CPU it shuts the
interrupt down. When the first CPU in the mask comes online again, the
interrupt core starts up the interrupt again and the multi-queue core
reenables the queue and makes it available for request queueing again.

What Christoph and me wanted to avoid was the insanity to have drivers
managing all of this, each driver on it's own. Guess how well that would
work in general and especially with CPU hotplug...

So there is a very good reason why this was implemented that
way. Christoph and myself were well aware of the limitations of x86, but
neither did we expect such insanities, nor has this come up until now,
which is a whopping 10 years later.

Santosh clearly pointed to PCI_IRQ_AFFINITY, which is only used in
drivers/nvme/host/pci.c:

  nvme_probe()
    nvme_setup_io_queues()
      nvme_setup_irqs()
       	pci_alloc_irq_vectors_affinity(PCI_IRQ_AFFINITY)

That means he actually wants to instantiate 1000+ PCIe devices, where
each of them consumes one or two vectors per CPU independent of the
number of IO queues the device supports, which is according to my
understanding of math not possible on x86.

So when he removes PCI_IRQ_AFFINITY from the allocation call, then this
won't go into the spreading code and therefore neither ends up with
managed interrupts nor with separate read queues. It will allocate one
for the management queue and one for an io queue, which means each PCI
device consumes two vectors total and 384 CPUs will share that IO
queue. And that obviously fits into the available vector space of 384
CPUs.

So the only knob you can provide is a 'no-multi-queue' knob, which
limits the device to one system wide IO queue and skips the managed
interrupt allocation. That obviously will fit into the available vector
space across all 384 CPUs, but will provide absymal performance and is
guaranteed to fail CPU hotplug operations.

If you want to support this 'I want a pony' use case, feel free to do
so.

Thanks,

        tglx