Re: [RFC] PCI_IRQ_AFFINITY limits MSI-X allocation on 384 CPU / 1000+ NVMe system

[Thomas Gleixner <tglx@linutronix.de>]

On Mon, Aug 10 2026 at 15:11, Keith Busch wrote:
> On Tue, Aug 04, 2026 at 11:27:04PM +0200, Thomas Gleixner wrote:
>> We could come up with some less restrictive mechanism, but that would at
>> the end run into the vector limitation on hotplug/hibernate because you
>> can't fit more than ~200 vectors into the last online CPU.
>> 
>> OTOH. With 1000 devices which consume also one non-managed interrupt for
>> their management queues, i.e. a total of 1000, that's not going to work
>> anyway.
>
> I think we'd have to at some point declare that extremely mismatched
> setups just can't be reasonably supported. :)

Then let the guy who wants 1000+ NVME devices have a parameter to turn off
managed and multiqueue mode and leave everything else alone. :)

> My understanding is you've got the managed IRQ's allocating a vector up
> front for every possible CPU they could be migrated to primarily to
> ensure CPU unplug can always move the effective affinity to an active
> CPU.

Until the last CPU in the assigned affinity mask of a particular
interrupt goes offline, which shuts down the interrupt after the
multiqueue management drained and silenced the queue.

> I totally get that not having this guarantee can cause a device to
> appear unresponsive, but maybe this is unlikely enough to accept? Or

Unlikely does not exist at scale, so that's a non-argument to begin
with. I have exactly ZERO interest to debug such nonsense.

> maybe not, I am not sure; it just seems like a fringe case that trying
> to mitigate causes a more likely problem.

Which fringe case? CPU hotplug and hibernation are actively used for
dynamic scaling and it's not so fringe as you think. Just because your
use case does not need it does not mean there are no relevant users who
depend on it.

Throwing 1000+ NVME devices at a machine is more fringe to me and as I
explained before that also means that CPU hotplug down to a single CPU
is not possible at all.

The real fringe case is x86 because any other sane and relevant
architecture just has a large enough vector space to work with. Don't
tell me about loongarch as those clowns just repeated the x86 nonsense.

> Anyway, if we can tolerate dynamic vector allocation that only happens
> on effective CPU assigment, here's a PoC I did some basic sanity testing
> with:

That breaks the basic guarantees of managed interrupts and the
multi-queue magic:

  The interrupt and the queue are local to the auto-configured
  affinity set which got established by the spreading algorithm.

You can't guarantee that anymore. A lot of people really want to have
that guarantee and worked hard to make the spreading algorithm work
accross various topologies for optimal performance.

This needs a lot more thoughts than just ripping it out and pretending
that it works by some definition of works in order to support a use case
which has not even demonstrated any real world relevance. Just because
someone want's to instantiate 1000+ devices does not mean that's
something actually useful and practical. I used Qemu myself to push the
limits just to verify that stuff works as designed.

You cannot rip out the managed code on x86 and the related matrix bits
without taking care of the consequences for CPU hotplug under vector
pressure. With your change, failing to move the interrupt during hotplug
within the given affinity mask will result in a non-working
interrupt. Of course you can "fix" that in the interrupt hotplug
handling and break affinity instead of failing, but that will
permanently screw up the affinity mask and therefore the relationship
between queues and interrupts and you might end up with a queue on one
and the related interrupt on another node.

The problem here is that the interrupt affinity handling is done late in
the hotplug operations, i.e. after the CPU marked itself offline, which
is the point of no return. There have been discussions to move that
earlier, but that's a non trivial effort.

That's why the hotplug machinery validates that the actual required
affinity movement can be handled _before_ this stage is reached and if
not the hotplug operation is aborted.

But that code does not take managed interrupts into account under the
assumption that there is a guaranteed free vector for those interrupts
within the associated affinity mask.

To make this work you have to do something like this:

   1) Change the managed code in the matrix allocator to reserve a
      vector only on one CPU within the affinity mask of each queue and
      ensure that they are balanced out between the CPUs properly, which
      the matrix allocator already supports. The resulting single CPU
      reservation needs to be returned and stored e.g. in irq_cfg

   2) Let the set_affinity() callback, which is invoked when the
      interrupt is activated, use that cached CPU information to
      actually allocate the interrupt on that CPU.
      
   3) Make lapic_can_unplug_cpu() aware of the managed magic. That means
      you have to:

      - Walk the per CPU vector descriptor array

      - Find the managed interrupts

      - Check their affinity mask and validate that they can be
        moved within the mask unless it's the last CPU going down

        - If that's not possible, reject the hotplug operation after
          cleaning up the already made reservations.
         
        - If it's not the last CPU going down the reserve a vector on
          one of the remaining online CPUs, which is the same as #1)

So if #3 succeeds the vector space is not exhausted and then
migrate_one_irq() should just do the right thing.

The reservation magic is required because otherwise the allocator could
exhaust the vector space on one of the still online CPUs by moving
non-managed interrupts to them instead of moving them to some other
CPU as the actual hotplug code just walks the interrupt space linearly
from 0 to max and does not pick the managed ones first.

That should pretty much preserve the current behavior and guarantees and
if exceeded actually fail the hotplug operations (e.g. on hibernate)
gracefully and not rely on insane "unlikely to happen" assumptions.

With that the only limiting factor for the non-hotplug case will be the
number of queues per device which you spread assuming an even spreading,
which is not guaranteed on assymetric node topologies. And of course
ignoring other devices in the system which consume interrupt vectors.

    Ndev = Nvec * NCPUs / (M_per_device + Nqueues_per_device * Nsets_per_device)

Where 'M_per_device' stands for the non-managed 'management' interrupts
per device, which is usually one.

So assuming 256 CPUs with 200 available vectors (Nvec) you get the
following limits:

Nqueues_per_device | Nsets_per_device | M_per_device | Ndev
  1                | 1                | 1            | 25600
  2                | 1                | 1            | 17067
  4                | 1                | 1            | 10240
  8                | 1                | 1            |  5689
 16                | 1                | 1            |  3012
 32                | 1                | 1            |  1552
 64                | 1                | 1            |   788
128                | 1                | 1            |   397
256                | 1                | 1            |   199

With two sets per device that obviously just shifts the whole business
up by one row and the 199 devices limit happens already at 128 write
queues plus 128 read qeues per device and the 256 queues limit is
reached with 100 devices.

The point where hibernation remains possible reduces that number
significantly, but that's obviously not at all different from todays
limitations:

  Ndev = Nvec / (Nqueues_per_device * Nsets_per_device)

Nqueues_per_device | Nsets_per_device | M_per_device | Ndev
  X                | 1                | 1            |   100
  X                | 2                | 1            |    50

As I told the original poster before, it all just boils down to
trivial math :)

I have zero objections to make that change, but I'm not going to accept
anything which does not completely address the points I made because I
have _ZERO_ interrest to deal with the fallout. Correctness first is
still the only valid engineering principle.

Thanks,

        tglx