Re: Control of IRQ Affinities from Userspace

[Sebastian Andrzej Siewior <bigeasy@linutronix.de>]

On 2025-11-03 18:12:48 [+0100], Florian Bezdeka wrote:
> I'm trying to jump in and adding some thoughts and results we got while
> analyzing this issue:
> 
> What stmmac (and some more drivers) are trying to achieve here is some
> kind of handcrafted IRQ balancing, like the good old irqbalanced did in
> the past from usermode. Turns out that the situation about IRQ balancing
> is a bit inconsistent. Some IRQ chips (like the APIC on x86 do that
> "automatically" on driver level, many others don't. So drivers end up
> fiddling with affinities.

Doing it once during startup is probably okay. The problem is probably
that it forgets everything while it removes the IRQ and requests it
again during down/ up. It guess this is simpler because the number of
interrupts can change if the networking queues have been changed. And
this is probably also invoked in that case.

> We can nicely tune IRQs and affected affinities that that have been
> requested during system boot. Tools like tuned can configure them using
> the APIs Tobias described. IRQs that are requested / setup after boot,
> during runtime, are kind of "problematic" for us, as there is no API
> that informs about new IRQ. We would have to rescan /proc. But even if
> there would be such an API: That would be too late. The IRQ might have
> fired already.
> 
> Once an affinity has been set (e.g. by tuned) this affinity is being
> restored when the IRQ comes back after a link up/down or bpf load. But:
> It might have happened that the situation on the system has changed.
> Even the default affinity could be different now. In case of the stmmac
> - and probably way more drivers - the default affinity is not taken into
> account anymore. The previous affinity is being restored
> unconditionally.
> 
> I tried to modify stmmac and let it evaluate the default affinity while
> doing the IRQ balancing dance. That turned out to be working at the end,
> but each line violated several coding/style/abstraction rules. There is
> no API at driver level to read the current default affinity - or I
> missed it. I could sent that hack out as RFC if requested. Just let me
> know.

Several driver tune the affinity based on what they think is best. The
usual is we start with current CPU and increment the CPU with each
queue. This is not unique to networking but also happen with storage.

But we do have the "managed API" already.

> Thinking more about this problem - and trying to abstract that in a
> generalized way - triggered some ideas about "IRQ namespaces", similar
> to what we have for CPUs/Memory/... in the cgroup world. Devices, or
> classes of devices could be moved into namespaces, instead of
> configuring them one by one. Thoughts welcome. The main challenge here
> is that we do not think about rt vs. non-rt. It's more about multiple RT
> applications running in parallel, well isolated from each other and the
> non-rt world.

The excluded "affinity" would be a good place to start. So if you have
16 CPUs but declare only two CPU as housekeeping it would sense to limit
it to two interrupts if possible. Otherwise shuffle them among the two
available CPUs.

> Florian
Sebastian