[RFC] PCI_IRQ_AFFINITY limits MSI-X allocation on 384 CPU / 1000+ NVMe system

[RFC] PCI_IRQ_AFFINITY limits MSI-X allocation on 384 CPU / 1000+ NVMe system

[santhosh kumar]

Observed:
- Linux 6.13
- 384 CPUs
- 1000+ NVMe devices


With PCI_IRQ_AFFINITY:
- some NVMe devices fail to obtain dedicated MSI-X vectors


Without PCI_IRQ_AFFINITY:
- all NVMe devices obtain 2 MSI-X vectors

Investigation suggests an interaction between:
- group_cpus_evenly()
- irq_create_affinity_masks()
- x86 vector allocation



Looking for feedback on whether affinity-constrained vector allocation
could explain this behavior.

Re: [RFC] PCI_IRQ_AFFINITY limits MSI-X allocation on 384 CPU / 1000+ NVMe system

[Thomas Gleixner]

On Tue, Jul 28 2026 at 12:12, santhosh kumar wrote:
> Observed:
> - Linux 6.13
> - 384 CPUs
> - 1000+ NVMe devices
>
>
> With PCI_IRQ_AFFINITY:
> - some NVMe devices fail to obtain dedicated MSI-X vectors
>
>
> Without PCI_IRQ_AFFINITY:
> - all NVMe devices obtain 2 MSI-X vectors
>
> Investigation suggests an interaction between:
> - group_cpus_evenly()
> - irq_create_affinity_masks()
> - x86 vector allocation
>
> Looking for feedback on whether affinity-constrained vector allocation
> could explain this behavior.

Math perhaps?

Number of CPUs:  384

Total number of available device vectors: ~ (200 * 384) = 76800

NVMe devices try to allocate min(nr_queues, NR_CPUS) queues where each
queue requires a dedicated interrupt. Add the managament queue to it and
then it's obvious that the total number of required vectors is larger
than the number of available vectors in the system when the number of
devices gets large enough.

Nothing to see here. It's simply resource exhaustion.

If you want that odd setup to be supported you have to talk to the NVME
people and not to a random list of folks which have absolutely nothing
to do with NVME.

Thanks,

        tglx