[RFC] PCI_IRQ_AFFINITY limits MSI-X allocation on 384 CPU / 1000+ NVMe system

[RFC] PCI_IRQ_AFFINITY limits MSI-X allocation on 384 CPU / 1000+ NVMe system

[santhosh kumar]

Observed:
- Linux 6.13
- 384 CPUs
- 1000+ NVMe devices


With PCI_IRQ_AFFINITY:
- some NVMe devices fail to obtain dedicated MSI-X vectors


Without PCI_IRQ_AFFINITY:
- all NVMe devices obtain 2 MSI-X vectors

Investigation suggests an interaction between:
- group_cpus_evenly()
- irq_create_affinity_masks()
- x86 vector allocation



Looking for feedback on whether affinity-constrained vector allocation
could explain this behavior.

Re: [RFC] PCI_IRQ_AFFINITY limits MSI-X allocation on 384 CPU / 1000+ NVMe system

[Thomas Gleixner]

On Tue, Jul 28 2026 at 12:12, santhosh kumar wrote:
> Observed:
> - Linux 6.13
> - 384 CPUs
> - 1000+ NVMe devices
>
>
> With PCI_IRQ_AFFINITY:
> - some NVMe devices fail to obtain dedicated MSI-X vectors
>
>
> Without PCI_IRQ_AFFINITY:
> - all NVMe devices obtain 2 MSI-X vectors
>
> Investigation suggests an interaction between:
> - group_cpus_evenly()
> - irq_create_affinity_masks()
> - x86 vector allocation
>
> Looking for feedback on whether affinity-constrained vector allocation
> could explain this behavior.

Math perhaps?

Number of CPUs:  384

Total number of available device vectors: ~ (200 * 384) = 76800

NVMe devices try to allocate min(nr_queues, NR_CPUS) queues where each
queue requires a dedicated interrupt. Add the managament queue to it and
then it's obvious that the total number of required vectors is larger
than the number of available vectors in the system when the number of
devices gets large enough.

Nothing to see here. It's simply resource exhaustion.

If you want that odd setup to be supported you have to talk to the NVME
people and not to a random list of folks which have absolutely nothing
to do with NVME.

Thanks,

        tglx

Re: [RFC] PCI_IRQ_AFFINITY limits MSI-X allocation on 384 CPU / 1000+ NVMe system

[santhosh kumar]

On Wed, Jul 29, 2026 at 4:22 AM Thomas Gleixner <tglx@linutronix.de> wrote:
>
> On Tue, Jul 28 2026 at 12:12, santhosh kumar wrote:
> > Observed:
> > - Linux 6.13
> > - 384 CPUs
> > - 1000+ NVMe devices
> >
> >
> > With PCI_IRQ_AFFINITY:
> > - some NVMe devices fail to obtain dedicated MSI-X vectors
> >
> >
> > Without PCI_IRQ_AFFINITY:
> > - all NVMe devices obtain 2 MSI-X vectors
> >
> > Investigation suggests an interaction between:
> > - group_cpus_evenly()
> > - irq_create_affinity_masks()
> > - x86 vector allocation
> >
> > Looking for feedback on whether affinity-constrained vector allocation
> > could explain this behavior.
>
> Math perhaps?
>
> Number of CPUs:  384
>
> Total number of available device vectors: ~ (200 * 384) = 76800
>
> NVMe devices try to allocate min(nr_queues, NR_CPUS) queues where each
> queue requires a dedicated interrupt. Add the managament queue to it and
> then it's obvious that the total number of required vectors is larger
> than the number of available vectors in the system when the number of
> devices gets large enough.
>
> Nothing to see here. It's simply resource exhaustion.
>
> If you want that odd setup to be supported you have to talk to the NVME
> people and not to a random list of folks which have absolutely nothing
> to do with NVME.
>
I am trying to create 1024 nvme devices , each device requesting 2 msix vectors.
Total we need 1024*2=2048 msix vectors
Total number of available device vectors: ~ (200 * 384) = 76800
But still seeing following messages:
  [ 9166.370948] Interrupt reservation exceeds available resources
  [ 9167.455029] Interrupt reservation exceeds available resources
  [ 9167.455572] Interrupt reservation exceeds available resources
  [ 9167.455888] Interrupt reservation exceeds available resources

● Summary: Affinity Mask Generation in Linux 6.13
  The affinity mask is generated in multiple layers, here's the exact flow:
  Key Files:
  1. lib/group_cpus.c:347 - group_cpus_evenly()
    - This function creates per-interrupt CPU masks
    - Spreads interrupts across CPUs in a NUMA-aware manner
    - Each interrupt gets assigned to 1 or a small set of specific CPUs
  2. lib/group_cpus.c:14 - grp_spread_init_one()
    - Actually picks which specific CPUs go into each mask
    - Tries to use sibling CPUs (same core) for cache locality
  3. kernel/irq/affinity.c:26 - irq_create_affinity_masks()
    - Calls group_cpus_evenly() to get CPU assignments
    - Creates affinity descriptor array with masks
    - Marks interrupts as "managed" (cannot be changed from userspace)
  4. drivers/pci/msi/msi.c:669 - msix_setup_interrupts()
    - Calls irq_create_affinity_masks() when PCI_IRQ_AFFINITY is set
    - Attaches the masks to MSI-X descriptors

  5. arch/x86/kernel/apic/vector.c:295 - assign_irq_vector_any_locked()
    - Reads the affinity mask created above
    - ONLY allocates from CPUs in that specific mask
    - No fallback to other CPUs!


  The Problem Code:
  // lib/group_cpus.c - Creates restricted CPU masks
  grp_spread_init_one()
  {
      cpu = cpumask_first(nmsk);  // Pick CPU 68 for nvme0q0
      cpumask_set_cpu(cpu, irqmsk);  // Mask = {68} only!
  }

  // arch/x86/kernel/apic/vector.c - Can only use that mask
  assign_irq_vector_any_locked()
  {
      affmsk = irq_data_get_affinity_mask(irqd);  // Gets {68}
      assign_vector_locked(irqd, affmsk);  // ONLY tries CPU 68!
      // ❌ No fallback to CPU 200-383 even if they have free vectors
  }
  This is why disabling PCI_IRQ_AFFINITY fixes the issue - it prevents
the restrictive masks from being created in the first place!
   In Linux 6.13, PCI_IRQ_AFFINITY causes MSI-X vectors to receive
affinity masks generated through irq_create_affinity_masks() and
group_cpus_evenly(). On very large systems with hundreds of CPUs and
thousands of devices, these masks can       become sufficiently
restrictive that vector allocation is limited to a relatively small
subset of CPUs. When those CPUs become vector-constrained, some NVMe
devices receive fewer MSI-X vectors or fall back to shared interrupt
usage. Disabling  PCI_IRQ_AFFINITY removes these affinity constraints
and allows the allocator to place vectors more flexibly, which
explains why all NVMe devices successfully obtain two MSI-X vectors in
the same configuration

> Thanks,
>
>         tglx
>
>
>
>

Re: [RFC] PCI_IRQ_AFFINITY limits MSI-X allocation on 384 CPU / 1000+ NVMe system

[Thomas Gleixner]

On Wed, Jul 29 2026 at 11:59, santhosh kumar wrote:
> On Wed, Jul 29, 2026 at 4:22 AM Thomas Gleixner <tglx@linutronix.de> wrote:
>> Nothing to see here. It's simply resource exhaustion.
>>
>> If you want that odd setup to be supported you have to talk to the NVME
>> people and not to a random list of folks which have absolutely nothing
>> to do with NVME.

I've cleaned up the CC list for you because your AI buddy ignored that
request.

> I am trying to create 1024 nvme devices , each device requesting 2 msix vectors.
> Total we need 1024*2=2048 msix vectors
> Total number of available device vectors: ~ (200 * 384) = 76800
> But still seeing following messages:
>   [ 9166.370948] Interrupt reservation exceeds available resources
>   [ 9167.455029] Interrupt reservation exceeds available resources
>   [ 9167.455572] Interrupt reservation exceeds available resources
>   [ 9167.455888] Interrupt reservation exceeds available resources
>
> ● Summary: Affinity Mask Generation in Linux 6.13

Copying and pasting the output of your AI buddy is a pretty useless
exercise simply because your AI buddy does not understand how all of
this works. Neither did you actually validate that his slop makes any
sense.

Aside of that reports want to be against the latest upstream kernel and
not against something which is 8 revisions behind and eventually lacks a
ton of updates.

But let me explain you why your AI buddy gets it wrong.

>   The affinity mask is generated in multiple layers, here's the exact flow:

We all know how that works even without the wisdom of your AI buddy.

>   The Problem Code:
>   // lib/group_cpus.c - Creates restricted CPU masks
>   grp_spread_init_one()
>   {
>       cpu = cpumask_first(nmsk);  // Pick CPU 68 for nvme0q0
>       cpumask_set_cpu(cpu, irqmsk);  // Mask = {68} only!
>   }

See below.

>   // arch/x86/kernel/apic/vector.c - Can only use that mask
>   assign_irq_vector_any_locked()
>   {
>       affmsk = irq_data_get_affinity_mask(irqd);  // Gets {68}
>       assign_vector_locked(irqd, affmsk);  // ONLY tries CPU 68!
>       // ❌ No fallback to CPU 200-383 even if they have free vectors

Again your AI buddy is wrong.

assign_irq_vector_any_locked() has a fallback if the mask does not
result in an allocatable interrupt. It tries to get a close vector and
falls through to the very end of the function, which does:

	/* Try the full online mask */
	return assign_vector_locked(irqd, cpu_online_mask);

That's why the function has _any_ in the name. This function has a full
fallback. and that function is irrelevant for managed interrupts. It
only is invoked for non-managed interrupts.

Managed interrupts go through a different code path and that has no
fallback by design because that's the fundamental property of managed
interrupts.

So if group_spread_one() inits the mask for the managed interrupt with
only CPU 68 set then there is no fallback because there is no other
choice. But that also means that the driver did allocate more than two
vectors. Why?

If it really only allocates only two, then one is non-managed and the
other one is managed. In that case both end up with the a cpumask of
0-383, because spreading _ONE_ interrupt over 384 CPUs simply results in
a affinity mask with all CPUs set.

Just for illustration with a system with 64 CPUs and two nodes, where
CPU 0-15 and 32-47 are on node 0, CPU 16-31 and 48-63 are on node 1, the
spreading results in:

nr interrupts | nr_masks | masks
-----------------------------------------------------------------------
1             | 1        | [0-63]
2             | 2        | [0-15,32-47] [16-31,48-63]
4             | 4        | [0-7,32-39] [8-15,40-47] [16-23,48-55] ...
...
32            | 32       | [0,32] [1,33] ... [16,48] [17,49] ...
64            | 64       | [0] [1] [2] ....

That's the basic principle of the spreading mechanism. It's pretty
obvious, no?

Can you now explain me how that mechanism ends up creating a affinity
mask with a single CPU set if there is only _ONE_ interrupt to be
spreaded out?

I doubt it, but I can explain to you what happens in principle with NVME
and managed interrupts independent of the number of queues per device.

Managed interrupts ensure on startup, that for each CPU in the affinity
mask of each managed interrupt there is a vector guaranteed
available. That means:

nr interrupts | nr_masks | CPUs per mask | vectors | vectors
              |          |               | per CPU | total
------------------------------------------------------------------------
1             | 1        | 64            | 1       | 64
2             | 2        | 32            | 1       | 64
4             | 4        | 16            | 1       | 64
...
32            | 32       | 2             | 1       | 64
64            | 64       | 1             | 1       | 64

So depending on the other interrupts allocated this will fail somewhere
around 200 devices for sure.

I'm sure it's the same on your side because that's independent of the
number of CPUs, even if you failed to provide that information.

As I told you before it's simple math and resource exhaustion due to the
way how NVME is implemented and the x86 vector space limitations.

If you want that to behave differently, then you have to talk to the
NVME people as I told you before.

Thanks,

        tglx

Re: [RFC] PCI_IRQ_AFFINITY limits MSI-X allocation on 384 CPU / 1000+ NVMe system

[Keith Busch]

On Thu, Jul 30, 2026 at 03:16:46PM +0200, Thomas Gleixner wrote:
> As I told you before it's simple math and resource exhaustion due to the
> way how NVME is implemented and the x86 vector space limitations.
> 
> If you want that to behave differently, then you have to talk to the
> NVME people as I told you before.

We can introduce a module parameter to throttle down the maximum number
of IO queues to allocate per controller. I don't think the driver can
automatically reason out what the correct number should be because it
doesn't know how many devices it's going to see.

Re: [RFC] PCI_IRQ_AFFINITY limits MSI-X allocation on 384 CPU / 1000+ NVMe system

[Thomas Gleixner]

On Thu, Jul 30 2026 at 07:28, Keith Busch wrote:
> On Thu, Jul 30, 2026 at 03:16:46PM +0200, Thomas Gleixner wrote:
>> As I told you before it's simple math and resource exhaustion due to the
>> way how NVME is implemented and the x86 vector space limitations.
>> 
>> If you want that to behave differently, then you have to talk to the
>> NVME people as I told you before.
>
> We can introduce a module parameter to throttle down the maximum number
> of IO queues to allocate per controller. I don't think the driver can
> automatically reason out what the correct number should be because it
> doesn't know how many devices it's going to see.

I might be missing something here as usual, but the number of io queues
per controller does not affect that at all as I explained before.

The number of io queues just influences the spreading, but not the
number of vectors required. The queues segment the system into IO zones
from 1 up to min(max_queues, number of CPUs).

What influences the vector allocation is the number of sets, i.e. when
the PCI controller has enough queues to support read queues, then you
have one set for the default queues and one set for the read queues. If
not it's only the default set. This is the per set resource consumption:

   nr queues ==  | nr_masks | CPUs per mask | vectors | vectors
   nr_interrupts |          |               | per CPU | total
   ------------------------------------------------------------------------
   1             | 1        | 64            | 1       | 64
   2             | 2        | 32            | 1       | 64
   4             | 4        | 16            | 1       | 64
   ...
   32            | 32       | 2             | 1       | 64
   64            | 64       | 1             | 1       | 64

So it does not matter whether you limit the number of IO queues per PCI
controller because the per CPU vector consumption is always _one_ per
set.

The number of interrupts is irrelevant because they are just virtual
Linux software interrupt numbers. They are only limited by memory, but
if those tiny allocations fail then the NVME device is the least of your
worries.

The only relevant limiting factor is the x86 vector space, but who needs
more than 640K RAM and 256 interrupt vectors :)

This all was designed to make multi queue operations simpler in terms of
affinity management and also to guarantee that CPU hot-unplug, which is
required for suspend/hibernate, does not end up trying to move hundreds
of queues to the boot CPU and ends up in vector exhaustion on x86.

The whole concept of multi-queue is built around that.

If the last CPU in the affinity mask of a queue goes down, then the
multi-queue core has also disabled request queueing for that queue,
drained the outstanding requests and shut down the queue in the
controller. Then the interrupt core deals with the still requested
interrupt and instead of moving it to an online CPU it shuts the
interrupt down. When the first CPU in the mask comes online again, the
interrupt core starts up the interrupt again and the multi-queue core
reenables the queue and makes it available for request queueing again.

What Christoph and me wanted to avoid was the insanity to have drivers
managing all of this, each driver on it's own. Guess how well that would
work in general and especially with CPU hotplug...

So there is a very good reason why this was implemented that
way. Christoph and myself were well aware of the limitations of x86, but
neither did we expect such insanities, nor has this come up until now,
which is a whopping 10 years later.

Santosh clearly pointed to PCI_IRQ_AFFINITY, which is only used in
drivers/nvme/host/pci.c:

  nvme_probe()
    nvme_setup_io_queues()
      nvme_setup_irqs()
       	pci_alloc_irq_vectors_affinity(PCI_IRQ_AFFINITY)

That means he actually wants to instantiate 1000+ PCIe devices, where
each of them consumes one or two vectors per CPU independent of the
number of IO queues the device supports, which is according to my
understanding of math not possible on x86.

So when he removes PCI_IRQ_AFFINITY from the allocation call, then this
won't go into the spreading code and therefore neither ends up with
managed interrupts nor with separate read queues. It will allocate one
for the management queue and one for an io queue, which means each PCI
device consumes two vectors total and 384 CPUs will share that IO
queue. And that obviously fits into the available vector space of 384
CPUs.

So the only knob you can provide is a 'no-multi-queue' knob, which
limits the device to one system wide IO queue and skips the managed
interrupt allocation. That obviously will fit into the available vector
space across all 384 CPUs, but will provide absymal performance and is
guaranteed to fail CPU hotplug operations.

If you want to support this 'I want a pony' use case, feel free to do
so.

Thanks,

        tglx

Re: [RFC] PCI_IRQ_AFFINITY limits MSI-X allocation on 384 CPU / 1000+ NVMe system

[Keith Busch]

On Thu, Jul 30, 2026 at 09:44:21PM +0200, Thomas Gleixner wrote:
> On Thu, Jul 30 2026 at 07:28, Keith Busch wrote:
> > We can introduce a module parameter to throttle down the maximum number
> > of IO queues to allocate per controller. I don't think the driver can
> > automatically reason out what the correct number should be because it
> > doesn't know how many devices it's going to see.
> 
> I might be missing something here as usual, but the number of io queues
> per controller does not affect that at all as I explained before.

Oh, you did explain that. I skipped to the end as I had too many
distractions last week. Sorry about that, and thanks for the
re-explanation.

I mistakenly was thinking the effective_affinity was the only hardware
resource used. Back to the drawing board.

Re: [RFC] PCI_IRQ_AFFINITY limits MSI-X allocation on 384 CPU / 1000+ NVMe system

[Christoph Hellwig]

On Mon, Aug 03, 2026 at 05:43:59AM -1000, Keith Busch wrote:
> On Thu, Jul 30, 2026 at 09:44:21PM +0200, Thomas Gleixner wrote:
> > On Thu, Jul 30 2026 at 07:28, Keith Busch wrote:
> > > We can introduce a module parameter to throttle down the maximum number
> > > of IO queues to allocate per controller. I don't think the driver can
> > > automatically reason out what the correct number should be because it
> > > doesn't know how many devices it's going to see.
> > 
> > I might be missing something here as usual, but the number of io queues
> > per controller does not affect that at all as I explained before.
> 
> Oh, you did explain that. I skipped to the end as I had too many
> distractions last week. Sorry about that, and thanks for the
> re-explanation.
> 
> I mistakenly was thinking the effective_affinity was the only hardware
> resource used. Back to the drawing board.

But given that we don't use multiple tag_set by default this can't
be the issue of the reported unless they didnt report enabling
multiple tag sets.

Re: [RFC] PCI_IRQ_AFFINITY limits MSI-X allocation on 384 CPU / 1000+ NVMe system

[Keith Busch]

On Tue, Aug 04, 2026 at 04:10:23PM +0200, Christoph Hellwig wrote:
> On Mon, Aug 03, 2026 at 05:43:59AM -1000, Keith Busch wrote:
> > On Thu, Jul 30, 2026 at 09:44:21PM +0200, Thomas Gleixner wrote:
> > > On Thu, Jul 30 2026 at 07:28, Keith Busch wrote:
> > > > We can introduce a module parameter to throttle down the maximum number
> > > > of IO queues to allocate per controller. I don't think the driver can
> > > > automatically reason out what the correct number should be because it
> > > > doesn't know how many devices it's going to see.
> > > 
> > > I might be missing something here as usual, but the number of io queues
> > > per controller does not affect that at all as I explained before.
> > 
> > Oh, you did explain that. I skipped to the end as I had too many
> > distractions last week. Sorry about that, and thanks for the
> > re-explanation.
> > 
> > I mistakenly was thinking the effective_affinity was the only hardware
> > resource used. Back to the drawing board.
> 
> But given that we don't use multiple tag_set by default this can't
> be the issue of the reported unless they didnt report enabling
> multiple tag sets.

I believe the use of additional sets makes the problem worse, but I
think we still have a problem even if we have only the one default set.
This is the quote from Thomas that I'm reading into:

  "the per CPU vector consumption is always _one_ per set."

So just the default set with only 1 NVMe IO queue managed IRQ will still
consume 384 vectors for the reported system even though the effective
affinity is pinned to 1 CPU.

Re: [RFC] PCI_IRQ_AFFINITY limits MSI-X allocation on 384 CPU / 1000+ NVMe system

[Thomas Gleixner]

On Tue, Aug 04 2026 at 04:56, Keith Busch wrote:
> On Tue, Aug 04, 2026 at 04:10:23PM +0200, Christoph Hellwig wrote:
>> On Mon, Aug 03, 2026 at 05:43:59AM -1000, Keith Busch wrote:
>> > On Thu, Jul 30, 2026 at 09:44:21PM +0200, Thomas Gleixner wrote:
>> > > On Thu, Jul 30 2026 at 07:28, Keith Busch wrote:
>> > > > We can introduce a module parameter to throttle down the maximum number
>> > > > of IO queues to allocate per controller. I don't think the driver can
>> > > > automatically reason out what the correct number should be because it
>> > > > doesn't know how many devices it's going to see.
>> > > 
>> > > I might be missing something here as usual, but the number of io queues
>> > > per controller does not affect that at all as I explained before.
>> > 
>> > Oh, you did explain that. I skipped to the end as I had too many
>> > distractions last week. Sorry about that, and thanks for the
>> > re-explanation.
>> > 
>> > I mistakenly was thinking the effective_affinity was the only hardware
>> > resource used. Back to the drawing board.
>> 
>> But given that we don't use multiple tag_set by default this can't
>> be the issue of the reported unless they didnt report enabling
>> multiple tag sets.
>
> I believe the use of additional sets makes the problem worse, but I
> think we still have a problem even if we have only the one default set.
> This is the quote from Thomas that I'm reading into:
>
>   "the per CPU vector consumption is always _one_ per set."
>
> So just the default set with only 1 NVMe IO queue managed IRQ will still
> consume 384 vectors for the reported system even though the effective
> affinity is pinned to 1 CPU.

Correct. We don't treat the one I/O queue case differently than the
multi I/O queue case.

We could come up with some less restrictive mechanism, but that would at
the end run into the vector limitation on hotplug/hibernate because you
can't fit more than ~200 vectors into the last online CPU.

OTOH. With 1000 devices which consume also one non-managed interrupt for
their management queues, i.e. a total of 1000, that's not going to work
anyway.

The restriction given by the x86 vector limitations when ignoring that
there might be network cards and other devices which consume interrupts
as well for the current managed model is:

nr_sets nr_nonmanaged   device limit
1       1               200/2 = 100
2       1               200/3 =  66

Subtract the interrupts consumed by other devices and your number might
become significantly smaller.

FRED will lift that restriction in the non-foreseeable future, but until
then you can only create a mode which effectively prevents hotplug
either upfront or in the hotplug case (see below).

The spreading mechanism would be basically the same as for managed mode,
so you can spread out several queues per device and let the vector
allocation pick one CPU out of the provided mask, but without guarantees
that in the hotplug case there is a free vector on another online CPU
within the mask.

Right now there is no mechanism to figure this out, so
lapic_can_unplug_cpu() would have to walk the vectors, look at the
affinity mask in each 'light managed' descriptor and validate that _all_
of them can be moved within their masks to prevent hotplug if not. The
hackery needed for that will be disgusting at best.

The only simple case are devices with one I/O queue per set. There you
could just do what Santhosh did: make the interrupts not be managed
at all and hotplug will fail at some point when the still online CPUs do
not have the capacity to take over the ones of the outgoing CPU. That
needs no change and works already today. If you would use two sets with
one queue each that works too, but then hotplug will simply fail
earlier.

Thanks,

        tglx