Re: Control of IRQ Affinities from Userspace

[Thomas Gleixner <tglx@linutronix.de>]

On Wed, Nov 26 2025 at 16:07, Florian Bezdeka wrote:
> On Wed Nov 26, 2025 at 3:26 PM CET, Thomas Gleixner wrote:
>> The question is whether that affinity hint has a functional requirement
>> to be applied or not. I don't think so because those interrupts can be
>> moved by userspace as it sees fit.
>
> The background seems performance. Those NICs support link speeds up to
> (or even above) 2.5Gbit/s. Seems it's hard to fully utilize the link
> when all queues are routed - IRQ wise - to a single core.
>
> This is now the point where the IRQ chips matters. Some (like APIC for
> x86) have the IRQ balancing implemented in SW, while others don't have
> that. So the driver does that manually by ignoring all the RT settings.

Hardware interrupt balancing never worked right :)

APIC "supports" it in logical/cluster mode, but in fact 99% of the
interrupts ended up on the lowest APIC in the logical/cluster mask. So
we gave up on it because the benefit was close to zero and the
complexity for multi-CPU affinity management with the limited vector
space was just not worth it. In high performance setups the interrupts
were anyway steered to a single CPU by the admin or irqbalanced :)

ARM64 would support that too IIRC, but they decided to avoid the whole
multi-CPU affinity mess as well :)

>> So it's easy enough to make this "set" part conditional and restrict it
>> to some TBD mask (housekeeping, default ...) under some isolation magic.
>>
>
> For now I would be happy if I could modify the stmmac in a way that its
> balancing takes the default affinity into account. I couldn't find any
> available API that allows me to do so from a module.
>
> Are there any strong reasons for not exporting the default affinity from
> the IRQ core? Read-only would be enough.

Default affinity is yet another piece which is disconnected from all the
other isolation mechanics. So we are not exporting it for some quick and
dirty hack. You can do that of course in your own kernel, but please
don't send the result to my inbox :)

> In addition I'm quite sure that the housekeeping infrastructure would
> not help in the area of networking as nobody (except one driver) is
> based on the managed IRQ API.

Managed interrupts are not user steerable and due to their strict
CPU/CPUgroup relationship they are not required to be steerable. NVME &
al have a strict command/response on the same queue scheme, which is
obviously most efficient when you have per CPU queues. The nice thing
about that concept is that the queues are only active (and having
interrupts) when an application on a given CPU issues a R/W operation.

Networking does not have that by default as their strategy of routing
packages to queues is way more complicated and can be affected by
hardware filtering etc.

But why can't housekeeping help in general and why do you want to hack
around the problem in random drivers?

What's wrong with providing a new irq_set_affinity_hint_xxx() variant
which takes a additional queue number as argument and let that do:

    if (isolate) {
        weight = cpumask_weight(housekeeping);
        qnr %= weight;
        cpu = cpumask_nth(qnr, housekeeping);
        mask = cpumask_of(cpu);
    }
    return irq_set_affinity_hint(mask);
    
or something like that. From a quick glance over the drivers this could
maybe be based on a queue number alone as most drivers do:

      mask = cpumask_of(qnr % num_online_cpus());

or something daft like that, which is obviously broken, but who cares.
So that would become:

    if (isolate) {
        weight = cpumask_weight(housekeeping);
        qnr %= weight;
        cpu = cpumask_nth(qnr, housekeeping);
    } else {
        guard(cpus_read_lock)();
        qnr %= num_online_cpus();
        cpu = cpumask_nth(qnr, cpu_online_mask);
    }
    	
    return irq_set_affinity_hint(cpumask_of(cpu));

See?

That lets userspace still override the hint but does at least initial
spreading within the housekeeping mask. Which ever mask that is out of
the zoo of masks you best debate with Frederic. :)

Thanks,

        tglx