Linux-ARM-Kernel Archive on lore.kernel.org
 help / color / mirror / Atom feed
* [PATCH RFC 0/3] genirq: Allow drivers to respect userspace IRQ affinities
@ 2026-08-19 14:30 Florian Bezdeka
  2026-08-19 14:30 ` [PATCH RFC 1/3] cpumask: Honor irq_default_affinity in cpumask_local_spread() Florian Bezdeka
                   ` (5 more replies)
  0 siblings, 6 replies; 12+ messages in thread
From: Florian Bezdeka @ 2026-08-19 14:30 UTC (permalink / raw)
  To: Maxime Chevallier, Andrew Lunn, David S. Miller, Eric Dumazet,
	Jakub Kicinski, Paolo Abeni, Maxime Coquelin, Alexandre Torgue,
	Yury Norov, Rasmus Villemoes, Andrew Morton,
	Sebastian Andrzej Siewior, Clark Williams, Steven Rostedt,
	Thomas Gleixner
  Cc: Jan Kiszka, netdev, linux-stm32, linux-arm-kernel, linux-kernel,
	linux-rt-devel, Florian Bezdeka

Hi all,

I'm trying to demonstrate a real problem for PREEMPT_RT here, using the
stmmac driver as example. There are more drivers "affected" but let's
ignore that for one moment. Let's discuss the underlying problem first.

To achieve the best throughput most network devices are based on
multiple queues. Each queue (pair) is equipped with one device IRQ. To
reach the maximum throughput, spreading / balancing them between all the
available CPUs makes sense.

While some IRQ chips - and with that some architectures - implement the
necessary spreading (or balancing) at IRQ chip level others don't do that.
If a device driver wants to make sure that balancing happens as intended
it has to implement that on his own (again).

The typical shortcoming of those implementations: They do not honor RT
relevant settings like the smp_default_affinity or isolated CPU cores.
Device IRQs are balanced over "all" or "all online CPUs".

That's not a problem for "normal" systems, but for RT systems - or
systems running cpu-isolating workloads - it is.

IRQ affinities can be controlled via /proc/irq/<n>/smp_affinity{_list}
for existing IRQs and via /proc/irq/default_smp_affinity for "new" or 
"not yet registered" IRQs. Those settings - as written by user space - 
must be honored. Always.

In our case the settings were bypassed by the following sequence:

    - system boot (all CPUs available, no isolation yet)
    - deployment of the first RT application
        - writing a new default smp affinity (remove RT isolated cores)
        - migrating away all that were targeting the now isolated cores
        - creating a cgroup with RT cores as the only usable cores
    - RT application running fine for some time
    - stmmac network interface went up for the first time
        - driver balances IRQs over all CPUs, ignoring the existing
	  affinities
    - Too much IRQ traffic on RT cores

Even with series applied there is (at least) one problem remaining:

The /proc/irq/<n> interface is populated on the first request_irq() call. 
That means that we can not control affinities from userspace until the 
IRQ gets requested.

The problem for network interfaces: We have to bring up the interfaces
once, to be able to control such affinities.

That raises the question why request_irq() is called on "link up" time,
while the low level vector allocation takes place during device probing.
At least that seems to be the common pattern. Can someone tell me why
this is done this way? Shouldn't we call request_irq() at the same time?

So, let's hope that all of this was short enough that somebody reads it
and precise enough to make the problem clear. The idea behind this series
is not fixing or applying the series as is. I'm expecting a discussion
first. So, input welcome!

Signed-off-by: Florian Bezdeka <florian.bezdeka@siemens.com>
---
Florian Bezdeka (3):
      cpumask: Honor irq_default_affinity in cpumask_local_spread()
      genirq: Honor existing IRQ affinities when setting affinity hints
      net: stmmac: Migrate IRQ balancing to cpumask_local_spread()

 drivers/net/ethernet/stmicro/stmmac/stmmac_main.c | 21 +++++++++++++++++----
 kernel/irq/manage.c                               | 10 ++++++++++
 lib/cpumask.c                                     | 10 ++++++----
 3 files changed, 33 insertions(+), 8 deletions(-)
---
base-commit: aa2e13ae8d3cbe2c15ef4f7e971b2de0832794aa
change-id: 20260810-flo-net-7-2-make-stmmac-default-affinity-aware-a145a4941d8f

Best regards,
-- 
Florian Bezdeka <florian.bezdeka@siemens.com>



^ permalink raw reply	[flat|nested] 12+ messages in thread

* [PATCH RFC 1/3] cpumask: Honor irq_default_affinity in cpumask_local_spread()
  2026-08-19 14:30 [PATCH RFC 0/3] genirq: Allow drivers to respect userspace IRQ affinities Florian Bezdeka
@ 2026-08-19 14:30 ` Florian Bezdeka
  2026-08-19 18:38   ` Yury Norov
  2026-08-19 14:30 ` [PATCH RFC 2/3] genirq: Honor existing IRQ affinities when setting affinity hints Florian Bezdeka
                   ` (4 subsequent siblings)
  5 siblings, 1 reply; 12+ messages in thread
From: Florian Bezdeka @ 2026-08-19 14:30 UTC (permalink / raw)
  To: Maxime Chevallier, Andrew Lunn, David S. Miller, Eric Dumazet,
	Jakub Kicinski, Paolo Abeni, Maxime Coquelin, Alexandre Torgue,
	Yury Norov, Rasmus Villemoes, Andrew Morton,
	Sebastian Andrzej Siewior, Clark Williams, Steven Rostedt,
	Thomas Gleixner
  Cc: Jan Kiszka, netdev, linux-stm32, linux-arm-kernel, linux-kernel,
	linux-rt-devel, Florian Bezdeka

Many drivers call cpumask_local_spread() to spread IRQs to several
CPUs, mainly to get best performance and balance CPU load. For
realtime (PREEMPT_RT) and other cpu-isolating workloads the old
implementation was triggering an IRQ placement problem. IRQs were
targeting CPUs that were isolated for those sensitive workloads.

Userland will tell the kernel about the desired IRQ configuration
for new interrupts by writing a proper cpumask to
/proc/irq/default_smp_affinity. This cpu mask has to be honored to
avoid IRQ noise on isolated cores.

The default for irq_default_affinity is "all CPUs". So all CPUs will
be taken into account for spreading when userland did not configure
something special.
---
 lib/cpumask.c | 10 ++++++----
 1 file changed, 6 insertions(+), 4 deletions(-)

diff --git a/lib/cpumask.c b/lib/cpumask.c
index 5adb9874fbd0f5a42ea8cd9e6c3729a70599781f..73e7b60a9201174f83df6067effbe7d1889dcdb9 100644
--- a/lib/cpumask.c
+++ b/lib/cpumask.c
@@ -6,6 +6,7 @@
 #include <linux/export.h>
 #include <linux/memblock.h>
 #include <linux/numa.h>
+#include <linux/interrupt.h>
 
 /* These are not inline because of header tangles. */
 #ifdef CONFIG_CPUMASK_OFFSTACK
@@ -81,8 +82,9 @@ void __init free_bootmem_cpumask_var(cpumask_var_t mask)
  * @i: index number
  * @node: local numa_node
  *
- * Return: online CPU according to a numa aware policy; local cpus are returned
- * first, followed by non-local ones, then it wraps around.
+ * Return: online CPU according to the default IRQ affinity and a numa aware
+ * policy; local cpus are returned first, followed by non-local ones, then it
+ * wraps around.
  *
  * For those who wants to enumerate all CPUs based on their NUMA distances,
  * i.e. call this function in a loop, like:
@@ -110,9 +112,9 @@ unsigned int cpumask_local_spread(unsigned int i, int node)
 	unsigned int cpu;
 
 	/* Wrap: we always want a cpu. */
-	i %= num_online_cpus();
+	i %= cpumask_weight(irq_default_affinity);
 
-	cpu = sched_numa_find_nth_cpu(cpu_online_mask, i, node);
+	cpu = sched_numa_find_nth_cpu(irq_default_affinity, i, node);
 
 	WARN_ON(cpu >= nr_cpu_ids);
 	return cpu;

-- 
2.55.0



^ permalink raw reply related	[flat|nested] 12+ messages in thread

* [PATCH RFC 2/3] genirq: Honor existing IRQ affinities when setting affinity hints
  2026-08-19 14:30 [PATCH RFC 0/3] genirq: Allow drivers to respect userspace IRQ affinities Florian Bezdeka
  2026-08-19 14:30 ` [PATCH RFC 1/3] cpumask: Honor irq_default_affinity in cpumask_local_spread() Florian Bezdeka
@ 2026-08-19 14:30 ` Florian Bezdeka
  2026-08-20 16:41   ` Jakub Kicinski
  2026-08-19 14:30 ` [PATCH RFC 3/3] net: stmmac: Migrate IRQ balancing to cpumask_local_spread() Florian Bezdeka
                   ` (3 subsequent siblings)
  5 siblings, 1 reply; 12+ messages in thread
From: Florian Bezdeka @ 2026-08-19 14:30 UTC (permalink / raw)
  To: Maxime Chevallier, Andrew Lunn, David S. Miller, Eric Dumazet,
	Jakub Kicinski, Paolo Abeni, Maxime Coquelin, Alexandre Torgue,
	Yury Norov, Rasmus Villemoes, Andrew Morton,
	Sebastian Andrzej Siewior, Clark Williams, Steven Rostedt,
	Thomas Gleixner
  Cc: Jan Kiszka, netdev, linux-stm32, linux-arm-kernel, linux-kernel,
	linux-rt-devel, Florian Bezdeka

Some device drivers implement an IRQ balancing / spreading mechanism
based on cpumask_local_spread() and irq_set_affinity_hint() (deprecated)
or irq_set_affinity_and_hint().

__irq_apply_affinity_hint() was overwriting already configured
affinities and with that violating user defined affinities.

This was especially a problem on systems with isolated CPUs, a common
pattern on PREEMPT_RT enabled systems. Device IRQs were balanced over
CPUs that were isolated for RT workloads.

Note that only the first call of irq_set_affinity_hint() and
irq_set_affinity_and_hint() - for each IRQ - will have any effect.
Afterward affinities might be controlled from userspace using the
/proc/irq/<n>/smp_affinity{_list} interface.

Signed-off-by: Florian Bezdeka <florian.bezdeka@siemens.com>
---
 kernel/irq/manage.c | 10 ++++++++++
 1 file changed, 10 insertions(+)

diff --git a/kernel/irq/manage.c b/kernel/irq/manage.c
index 7eb07e3bdb4c2419e0a58a75c29d88b4b00b8287..3937ba24df6d63375dda88164cf92fd49e70a3eb 100644
--- a/kernel/irq/manage.c
+++ b/kernel/irq/manage.c
@@ -503,11 +503,21 @@ EXPORT_SYMBOL_GPL(irq_force_affinity);
 
 int __irq_apply_affinity_hint(unsigned int irq, const struct cpumask *m, bool setaffinity)
 {
+	struct irq_data *irq_data;
 	int ret = -EINVAL;
 
 	scoped_irqdesc_get_and_lock(irq, IRQ_GET_DESC_CHECK_GLOBAL) {
 		scoped_irqdesc->affinity_hint = m;
+		irq_data = irq_desc_get_irq_data(scoped_irqdesc);
 		ret = 0;
+
+		/*
+		 * Do not overwrite the affinity in case an affinity for this
+		 * IRQ has already been requested, e.g. by writing to
+		 * /proc/irq/<n>/smp_affinity
+		 */
+		if (irqd_affinity_was_set(irq_data))
+			setaffinity = false;
 	}
 
 	if (!ret && m && setaffinity)

-- 
2.55.0



^ permalink raw reply related	[flat|nested] 12+ messages in thread

* [PATCH RFC 3/3] net: stmmac: Migrate IRQ balancing to cpumask_local_spread()
  2026-08-19 14:30 [PATCH RFC 0/3] genirq: Allow drivers to respect userspace IRQ affinities Florian Bezdeka
  2026-08-19 14:30 ` [PATCH RFC 1/3] cpumask: Honor irq_default_affinity in cpumask_local_spread() Florian Bezdeka
  2026-08-19 14:30 ` [PATCH RFC 2/3] genirq: Honor existing IRQ affinities when setting affinity hints Florian Bezdeka
@ 2026-08-19 14:30 ` Florian Bezdeka
  2026-08-19 18:45   ` Yury Norov
  2026-08-19 18:28 ` [PATCH RFC 0/3] genirq: Allow drivers to respect userspace IRQ affinities Yury Norov
                   ` (2 subsequent siblings)
  5 siblings, 1 reply; 12+ messages in thread
From: Florian Bezdeka @ 2026-08-19 14:30 UTC (permalink / raw)
  To: Maxime Chevallier, Andrew Lunn, David S. Miller, Eric Dumazet,
	Jakub Kicinski, Paolo Abeni, Maxime Coquelin, Alexandre Torgue,
	Yury Norov, Rasmus Villemoes, Andrew Morton,
	Sebastian Andrzej Siewior, Clark Williams, Steven Rostedt,
	Thomas Gleixner
  Cc: Jan Kiszka, netdev, linux-stm32, linux-arm-kernel, linux-kernel,
	linux-rt-devel, Florian Bezdeka

The previous balancing mechanism was based on num_online_cpus(), which
is a problem for systems cpu-isolating workloads. IRQs were targeting
CPUs that were isolated for those sensitive workloads.

With a migration to cpumask_local_spread() we
    - get NUMA locality
    - honor the default SMP affinity mask, which avoids targeting
      isolated CPUs.

This also aligns with the pattern used by most network drivers dealing
with IRQ affinities / affinity hints.

Signed-off-by: Florian Bezdeka <florian.bezdeka@siemens.com>
---
 drivers/net/ethernet/stmicro/stmmac/stmmac_main.c | 21 +++++++++++++++++----
 1 file changed, 17 insertions(+), 4 deletions(-)

diff --git a/drivers/net/ethernet/stmicro/stmmac/stmmac_main.c b/drivers/net/ethernet/stmicro/stmmac/stmmac_main.c
index a71f0df263785dd8badc45292ca3067ab33bda05..949ced7e46d2814b57c6bd86b4886ac3bf33996c 100644
--- a/drivers/net/ethernet/stmicro/stmmac/stmmac_main.c
+++ b/drivers/net/ethernet/stmicro/stmmac/stmmac_main.c
@@ -3826,11 +3826,16 @@ static int stmmac_request_irq_multi_msi(struct net_device *dev)
 	struct stmmac_priv *priv = netdev_priv(dev);
 	struct stmmac_msi *msi = priv->msi;
 	enum request_irq_err irq_err;
+	cpumask_var_t affinity;
 	int irq_idx = 0;
 	char *int_name;
+	int node;
 	int ret;
 	int i;
 
+	if (!zalloc_cpumask_var(&affinity, GFP_KERNEL))
+		return -ENOMEM;
+
 	/* For common interrupt */
 	int_name = msi->int_name_mac;
 	sprintf(int_name, "%s:%s", dev->name, "mac");
@@ -3916,6 +3921,7 @@ static int stmmac_request_irq_multi_msi(struct net_device *dev)
 	}
 
 	/* Request Rx MSI irq */
+	node = dev_to_node(&priv->dev->dev);
 	for (i = 0; i < priv->plat->rx_queues_to_use; i++) {
 		if (i >= MTL_MAX_RX_QUEUES)
 			break;
@@ -3935,8 +3941,10 @@ static int stmmac_request_irq_multi_msi(struct net_device *dev)
 			irq_idx = i;
 			goto irq_error;
 		}
-		irq_set_affinity_hint(msi->rx_irq[i],
-				      cpumask_of(i % num_online_cpus()));
+
+		cpumask_clear(affinity);
+		cpumask_set_cpu(cpumask_local_spread(i, node), affinity);
+		irq_set_affinity_and_hint(msi->rx_irq[i], affinity);
 	}
 
 	/* Request Tx MSI irq */
@@ -3959,13 +3967,18 @@ static int stmmac_request_irq_multi_msi(struct net_device *dev)
 			irq_idx = i;
 			goto irq_error;
 		}
-		irq_set_affinity_hint(msi->tx_irq[i],
-				      cpumask_of(i % num_online_cpus()));
+
+		cpumask_clear(affinity);
+		cpumask_set_cpu(cpumask_local_spread(i, node), affinity);
+		irq_set_affinity_and_hint(msi->tx_irq[i], affinity);
 	}
 
+	free_cpumask_var(affinity);
+
 	return 0;
 
 irq_error:
+	free_cpumask_var(affinity);
 	stmmac_free_irq(dev, irq_err, irq_idx);
 	return ret;
 }

-- 
2.55.0



^ permalink raw reply related	[flat|nested] 12+ messages in thread

* Re: [PATCH RFC 0/3] genirq: Allow drivers to respect userspace IRQ affinities
  2026-08-19 14:30 [PATCH RFC 0/3] genirq: Allow drivers to respect userspace IRQ affinities Florian Bezdeka
                   ` (2 preceding siblings ...)
  2026-08-19 14:30 ` [PATCH RFC 3/3] net: stmmac: Migrate IRQ balancing to cpumask_local_spread() Florian Bezdeka
@ 2026-08-19 18:28 ` Yury Norov
  2026-08-19 23:54 ` Andrew Lunn
  2026-08-20 15:12 ` Sebastian Andrzej Siewior
  5 siblings, 0 replies; 12+ messages in thread
From: Yury Norov @ 2026-08-19 18:28 UTC (permalink / raw)
  To: Florian Bezdeka
  Cc: Maxime Chevallier, Andrew Lunn, David S. Miller, Eric Dumazet,
	Jakub Kicinski, Paolo Abeni, Maxime Coquelin, Alexandre Torgue,
	Yury Norov, Rasmus Villemoes, Andrew Morton,
	Sebastian Andrzej Siewior, Clark Williams, Steven Rostedt,
	Thomas Gleixner, Jan Kiszka, netdev, linux-stm32,
	linux-arm-kernel, linux-kernel, linux-rt-devel

On Wed, Aug 19, 2026 at 04:30:29PM +0200, Florian Bezdeka wrote:
> Hi all,
> 
> I'm trying to demonstrate a real problem for PREEMPT_RT here, using the
> stmmac driver as example. There are more drivers "affected" but let's
> ignore that for one moment. Let's discuss the underlying problem first.
> 
> To achieve the best throughput most network devices are based on
> multiple queues. Each queue (pair) is equipped with one device IRQ. To
> reach the maximum throughput, spreading / balancing them between all the
> available CPUs makes sense.
> 
> While some IRQ chips - and with that some architectures - implement the
> necessary spreading (or balancing) at IRQ chip level others don't do that.
> If a device driver wants to make sure that balancing happens as intended
> it has to implement that on his own (again).
> 
> The typical shortcoming of those implementations: They do not honor RT
> relevant settings like the smp_default_affinity or isolated CPU cores.
> Device IRQs are balanced over "all" or "all online CPUs".
> 
> That's not a problem for "normal" systems, but for RT systems - or
> systems running cpu-isolating workloads - it is.
> 
> IRQ affinities can be controlled via /proc/irq/<n>/smp_affinity{_list}
> for existing IRQs and via /proc/irq/default_smp_affinity for "new" or 
> "not yet registered" IRQs. Those settings - as written by user space - 
> must be honored. Always.
> 
> In our case the settings were bypassed by the following sequence:
> 
>     - system boot (all CPUs available, no isolation yet)
>     - deployment of the first RT application
>         - writing a new default smp affinity (remove RT isolated cores)
>         - migrating away all that were targeting the now isolated cores
>         - creating a cgroup with RT cores as the only usable cores
>     - RT application running fine for some time
>     - stmmac network interface went up for the first time
>         - driver balances IRQs over all CPUs, ignoring the existing
> 	  affinities
>     - Too much IRQ traffic on RT cores
> 
> Even with series applied there is (at least) one problem remaining:
> 
> The /proc/irq/<n> interface is populated on the first request_irq() call. 
> That means that we can not control affinities from userspace until the 
> IRQ gets requested.
> 
> The problem for network interfaces: We have to bring up the interfaces
> once, to be able to control such affinities.
> 
> That raises the question why request_irq() is called on "link up" time,
> while the low level vector allocation takes place during device probing.
> At least that seems to be the common pattern. Can someone tell me why
> this is done this way? Shouldn't we call request_irq() at the same time?
> 
> So, let's hope that all of this was short enough that somebody reads it

I did!

> and precise enough to make the problem clear. The idea behind this series
> is not fixing or applying the series as is. I'm expecting a discussion
> first. So, input welcome!

Not sure I understand the full scope, but it's not because of your
description. And I want to learn more about the RT business. I'll
comment the cpumasks part, and please keep me in CC.

Thanks,
Yury

> Signed-off-by: Florian Bezdeka <florian.bezdeka@siemens.com>
> ---
> Florian Bezdeka (3):
>       cpumask: Honor irq_default_affinity in cpumask_local_spread()
>       genirq: Honor existing IRQ affinities when setting affinity hints
>       net: stmmac: Migrate IRQ balancing to cpumask_local_spread()
> 
>  drivers/net/ethernet/stmicro/stmmac/stmmac_main.c | 21 +++++++++++++++++----
>  kernel/irq/manage.c                               | 10 ++++++++++
>  lib/cpumask.c                                     | 10 ++++++----
>  3 files changed, 33 insertions(+), 8 deletions(-)
> ---
> base-commit: aa2e13ae8d3cbe2c15ef4f7e971b2de0832794aa
> change-id: 20260810-flo-net-7-2-make-stmmac-default-affinity-aware-a145a4941d8f
> 
> Best regards,
> -- 
> Florian Bezdeka <florian.bezdeka@siemens.com>


^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: [PATCH RFC 1/3] cpumask: Honor irq_default_affinity in cpumask_local_spread()
  2026-08-19 14:30 ` [PATCH RFC 1/3] cpumask: Honor irq_default_affinity in cpumask_local_spread() Florian Bezdeka
@ 2026-08-19 18:38   ` Yury Norov
  2026-08-20 16:09     ` Sebastian Andrzej Siewior
  0 siblings, 1 reply; 12+ messages in thread
From: Yury Norov @ 2026-08-19 18:38 UTC (permalink / raw)
  To: Florian Bezdeka
  Cc: Maxime Chevallier, Andrew Lunn, David S. Miller, Eric Dumazet,
	Jakub Kicinski, Paolo Abeni, Maxime Coquelin, Alexandre Torgue,
	Yury Norov, Rasmus Villemoes, Andrew Morton,
	Sebastian Andrzej Siewior, Clark Williams, Steven Rostedt,
	Thomas Gleixner, Jan Kiszka, netdev, linux-stm32,
	linux-arm-kernel, linux-kernel, linux-rt-devel

On Wed, Aug 19, 2026 at 04:30:30PM +0200, Florian Bezdeka wrote:
> Many drivers call cpumask_local_spread() to spread IRQs to several
> CPUs, mainly to get best performance and balance CPU load. For
> realtime (PREEMPT_RT) and other cpu-isolating workloads the old
> implementation was triggering an IRQ placement problem. IRQs were
> targeting CPUs that were isolated for those sensitive workloads.
> 
> Userland will tell the kernel about the desired IRQ configuration
> for new interrupts by writing a proper cpumask to
> /proc/irq/default_smp_affinity. This cpu mask has to be honored to
> avoid IRQ noise on isolated cores.
> 
> The default for irq_default_affinity is "all CPUs". So all CPUs will
> be taken into account for spreading when userland did not configure
> something special.
> ---
>  lib/cpumask.c | 10 ++++++----
>  1 file changed, 6 insertions(+), 4 deletions(-)
> 
> diff --git a/lib/cpumask.c b/lib/cpumask.c
> index 5adb9874fbd0f5a42ea8cd9e6c3729a70599781f..73e7b60a9201174f83df6067effbe7d1889dcdb9 100644
> --- a/lib/cpumask.c
> +++ b/lib/cpumask.c
> @@ -6,6 +6,7 @@
>  #include <linux/export.h>
>  #include <linux/memblock.h>
>  #include <linux/numa.h>
> +#include <linux/interrupt.h>
>  
>  /* These are not inline because of header tangles. */
>  #ifdef CONFIG_CPUMASK_OFFSTACK
> @@ -81,8 +82,9 @@ void __init free_bootmem_cpumask_var(cpumask_var_t mask)
>   * @i: index number
>   * @node: local numa_node
>   *
> - * Return: online CPU according to a numa aware policy; local cpus are returned
> - * first, followed by non-local ones, then it wraps around.
> + * Return: online CPU according to the default IRQ affinity and a numa aware
> + * policy; local cpus are returned first, followed by non-local ones, then it
> + * wraps around.
>   *
>   * For those who wants to enumerate all CPUs based on their NUMA distances,
>   * i.e. call this function in a loop, like:
> @@ -110,9 +112,9 @@ unsigned int cpumask_local_spread(unsigned int i, int node)

Please don't touch this function. There's ~40 users, and we don't want
to inspect every caller for their intention.

Looking at ionic_get_preferred_cpu(), the cpumask_local_spread() is
used there as a fallback for the affinity IRQ search:

 static int ionic_get_preferred_cpu(struct ionic *ionic,
                                    struct ionic_intr_info *intr)
 {
         int cpu;
 
         cpu = cpumask_first_and(*intr->affinity_mask, cpu_online_mask);
         if (cpu >= nr_cpu_ids)
                 cpu = cpumask_local_spread(0, dev_to_node(ionic->dev));
 
         return cpu;
 }

Your change may affect the logic, seemingly.

Instead, please create something like:

        unsigned int cpumask_spread(struct cpumask cpus,
                                    unsigned int i, int node);

Thanks,
Yury

>  	unsigned int cpu;
>  
>  	/* Wrap: we always want a cpu. */
> -	i %= num_online_cpus();
> +	i %= cpumask_weight(irq_default_affinity);
>  
> -	cpu = sched_numa_find_nth_cpu(cpu_online_mask, i, node);
> +	cpu = sched_numa_find_nth_cpu(irq_default_affinity, i, node);
>  
>  	WARN_ON(cpu >= nr_cpu_ids);
>  	return cpu;
> 
> -- 
> 2.55.0


^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: [PATCH RFC 3/3] net: stmmac: Migrate IRQ balancing to cpumask_local_spread()
  2026-08-19 14:30 ` [PATCH RFC 3/3] net: stmmac: Migrate IRQ balancing to cpumask_local_spread() Florian Bezdeka
@ 2026-08-19 18:45   ` Yury Norov
  0 siblings, 0 replies; 12+ messages in thread
From: Yury Norov @ 2026-08-19 18:45 UTC (permalink / raw)
  To: Florian Bezdeka
  Cc: Maxime Chevallier, Andrew Lunn, David S. Miller, Eric Dumazet,
	Jakub Kicinski, Paolo Abeni, Maxime Coquelin, Alexandre Torgue,
	Yury Norov, Rasmus Villemoes, Andrew Morton,
	Sebastian Andrzej Siewior, Clark Williams, Steven Rostedt,
	Thomas Gleixner, Jan Kiszka, netdev, linux-stm32,
	linux-arm-kernel, linux-kernel, linux-rt-devel

On Wed, Aug 19, 2026 at 04:30:32PM +0200, Florian Bezdeka wrote:
> The previous balancing mechanism was based on num_online_cpus(), which
> is a problem for systems cpu-isolating workloads. IRQs were targeting
> CPUs that were isolated for those sensitive workloads.
> 
> With a migration to cpumask_local_spread() we
>     - get NUMA locality
>     - honor the default SMP affinity mask, which avoids targeting
>       isolated CPUs.
> 
> This also aligns with the pattern used by most network drivers dealing
> with IRQ affinities / affinity hints.
> 
> Signed-off-by: Florian Bezdeka <florian.bezdeka@siemens.com>
> ---
>  drivers/net/ethernet/stmicro/stmmac/stmmac_main.c | 21 +++++++++++++++++----
>  1 file changed, 17 insertions(+), 4 deletions(-)
> 
> diff --git a/drivers/net/ethernet/stmicro/stmmac/stmmac_main.c b/drivers/net/ethernet/stmicro/stmmac/stmmac_main.c
> index a71f0df263785dd8badc45292ca3067ab33bda05..949ced7e46d2814b57c6bd86b4886ac3bf33996c 100644
> --- a/drivers/net/ethernet/stmicro/stmmac/stmmac_main.c
> +++ b/drivers/net/ethernet/stmicro/stmmac/stmmac_main.c
> @@ -3826,11 +3826,16 @@ static int stmmac_request_irq_multi_msi(struct net_device *dev)
>  	struct stmmac_priv *priv = netdev_priv(dev);
>  	struct stmmac_msi *msi = priv->msi;
>  	enum request_irq_err irq_err;
> +	cpumask_var_t affinity;
>  	int irq_idx = 0;
>  	char *int_name;
> +	int node;
>  	int ret;
>  	int i;
>  
> +	if (!zalloc_cpumask_var(&affinity, GFP_KERNEL))
> +		return -ENOMEM;
> +
>  	/* For common interrupt */
>  	int_name = msi->int_name_mac;
>  	sprintf(int_name, "%s:%s", dev->name, "mac");
> @@ -3916,6 +3921,7 @@ static int stmmac_request_irq_multi_msi(struct net_device *dev)
>  	}
>  
>  	/* Request Rx MSI irq */
> +	node = dev_to_node(&priv->dev->dev);
>  	for (i = 0; i < priv->plat->rx_queues_to_use; i++) {
>  		if (i >= MTL_MAX_RX_QUEUES)
>  			break;
> @@ -3935,8 +3941,10 @@ static int stmmac_request_irq_multi_msi(struct net_device *dev)
>  			irq_idx = i;
>  			goto irq_error;
>  		}
> -		irq_set_affinity_hint(msi->rx_irq[i],
> -				      cpumask_of(i % num_online_cpus()));
> +
> +		cpumask_clear(affinity);
> +		cpumask_set_cpu(cpumask_local_spread(i, node), affinity);
> +		irq_set_affinity_and_hint(msi->rx_irq[i], affinity);

If you want to spread more than one IRQ, you'd better convert your
loop into for_each_numa_hop_mask(). That way you don't need to
introduce new function. See the comment on top of
cpumask_local_spread():

 * For those who wants to enumerate all CPUs based on their NUMA distances,
 * i.e. call this function in a loop, like:
 *
 * for (i = 0; i < num_online_cpus(); i++) {
 *      cpu = cpumask_local_spread(i, node);
 *      do_something(cpu);
 * }
 *
 * There's a better alternative based on for_each()-like iterators:
 *
 *      for_each_numa_hop_mask(mask, node) {
 *              for_each_cpu_andnot(cpu, mask, prev)
 *                      do_something(cpu);
 *              prev = mask;
 *      }

Thanks,
Yury

>  	}
>  
>  	/* Request Tx MSI irq */
> @@ -3959,13 +3967,18 @@ static int stmmac_request_irq_multi_msi(struct net_device *dev)
>  			irq_idx = i;
>  			goto irq_error;
>  		}
> -		irq_set_affinity_hint(msi->tx_irq[i],
> -				      cpumask_of(i % num_online_cpus()));
> +
> +		cpumask_clear(affinity);
> +		cpumask_set_cpu(cpumask_local_spread(i, node), affinity);
> +		irq_set_affinity_and_hint(msi->tx_irq[i], affinity);
>  	}
>  
> +	free_cpumask_var(affinity);
> +
>  	return 0;
>  
>  irq_error:
> +	free_cpumask_var(affinity);
>  	stmmac_free_irq(dev, irq_err, irq_idx);
>  	return ret;
>  }
> 
> -- 
> 2.55.0


^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: [PATCH RFC 0/3] genirq: Allow drivers to respect userspace IRQ affinities
  2026-08-19 14:30 [PATCH RFC 0/3] genirq: Allow drivers to respect userspace IRQ affinities Florian Bezdeka
                   ` (3 preceding siblings ...)
  2026-08-19 18:28 ` [PATCH RFC 0/3] genirq: Allow drivers to respect userspace IRQ affinities Yury Norov
@ 2026-08-19 23:54 ` Andrew Lunn
  2026-08-20  0:10   ` Andrew Lunn
  2026-08-20 15:12 ` Sebastian Andrzej Siewior
  5 siblings, 1 reply; 12+ messages in thread
From: Andrew Lunn @ 2026-08-19 23:54 UTC (permalink / raw)
  To: Florian Bezdeka
  Cc: Maxime Chevallier, Andrew Lunn, David S. Miller, Eric Dumazet,
	Jakub Kicinski, Paolo Abeni, Maxime Coquelin, Alexandre Torgue,
	Yury Norov, Rasmus Villemoes, Andrew Morton,
	Sebastian Andrzej Siewior, Clark Williams, Steven Rostedt,
	Thomas Gleixner, Jan Kiszka, netdev, linux-stm32,
	linux-arm-kernel, linux-kernel, linux-rt-devel

> That raises the question why request_irq() is called on "link up" time,
> while the low level vector allocation takes place during device probing.

If the interface is admin down, the hardware should not be generating
any interrupts. So there is no need to request them.

> At least that seems to be the common pattern. Can someone tell me why
> this is done this way? Shouldn't we call request_irq() at the same time?

If you want to change anything, move the low level vector allocation
into open(). But you need to be careful of EPROBE_DEFER. If the
interrupt controller has not loaded yet, i _guess_ the low level
vector allocation will return EPROBE_DEFER, and the MAC driver will
try to probe again later. If you get EPROBE_DEFER in open(), there is
nothing you can do about it.

	Andrew


^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: [PATCH RFC 0/3] genirq: Allow drivers to respect userspace IRQ affinities
  2026-08-19 23:54 ` Andrew Lunn
@ 2026-08-20  0:10   ` Andrew Lunn
  0 siblings, 0 replies; 12+ messages in thread
From: Andrew Lunn @ 2026-08-20  0:10 UTC (permalink / raw)
  To: Florian Bezdeka
  Cc: Maxime Chevallier, Andrew Lunn, David S. Miller, Eric Dumazet,
	Jakub Kicinski, Paolo Abeni, Maxime Coquelin, Alexandre Torgue,
	Yury Norov, Rasmus Villemoes, Andrew Morton,
	Sebastian Andrzej Siewior, Clark Williams, Steven Rostedt,
	Thomas Gleixner, Jan Kiszka, netdev, linux-stm32,
	linux-arm-kernel, linux-kernel, linux-rt-devel

The problem with using stmmac as an example is that it is very old. It
was added to Linux in 2009. I guess most of the SoCs at that time were
single or dual core. CPU affinity was not something developers thought
about for that class of SoC.

Over time the number of cores has gone up and stmmac has got faster
link speeds. But is the architecture correct? Is the driver following
best practices?

I would suggest you look at more modern MAC drivers and see how they
do CPU affinity, etc. Is there anything which can be learned from them
and implemented in stmmac?

    Andrew


^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: [PATCH RFC 0/3] genirq: Allow drivers to respect userspace IRQ affinities
  2026-08-19 14:30 [PATCH RFC 0/3] genirq: Allow drivers to respect userspace IRQ affinities Florian Bezdeka
                   ` (4 preceding siblings ...)
  2026-08-19 23:54 ` Andrew Lunn
@ 2026-08-20 15:12 ` Sebastian Andrzej Siewior
  5 siblings, 0 replies; 12+ messages in thread
From: Sebastian Andrzej Siewior @ 2026-08-20 15:12 UTC (permalink / raw)
  To: Florian Bezdeka
  Cc: Maxime Chevallier, Andrew Lunn, David S. Miller, Eric Dumazet,
	Jakub Kicinski, Paolo Abeni, Maxime Coquelin, Alexandre Torgue,
	Yury Norov, Rasmus Villemoes, Andrew Morton, Clark Williams,
	Steven Rostedt, Thomas Gleixner, Jan Kiszka, netdev, linux-stm32,
	linux-arm-kernel, linux-kernel, linux-rt-devel

On 2026-08-19 16:30:29 [+0200], Florian Bezdeka wrote:
…
> The typical shortcoming of those implementations: They do not honor RT
> relevant settings like the smp_default_affinity or isolated CPU cores.

"irqaffinity" if you refer to the boot command argument.
"default_smp_affinity" if you refer to the proc file.

> Device IRQs are balanced over "all" or "all online CPUs".
> That raises the question why request_irq() is called on "link up" time,
> while the low level vector allocation takes place during device probing.
> At least that seems to be the common pattern. Can someone tell me why
> this is done this way? Shouldn't we call request_irq() at the same time?

The IRQ vector is created while the system enumerates the IRQ-chips.
Once the devices are enumerated (such as the NICs) the devices is linked
with its IRQ. I think an exception are MSI-X devices which could ask for
one or more interrupt and then (at device's probe time) the PCI core
will link the requested amount of interrupts so their actual number
could change.
The driver _could_ request a "managed interrupt" which would be mapped
to a specific CPU. The difference to a "regular interrupt" is that if
that CPU goes down, the interrupt is not "moved" to another CPU. Instead
is remains off and the driver needs to deal with this (this is common
for NVME devices).

If the device is not programmed (as in IP address has been assigned,
link is up) then it should not create any interrupts. So it might be
reasonable to not request an interrupt either.
I *think* uarts do the same.

Sebastian


^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: [PATCH RFC 1/3] cpumask: Honor irq_default_affinity in cpumask_local_spread()
  2026-08-19 18:38   ` Yury Norov
@ 2026-08-20 16:09     ` Sebastian Andrzej Siewior
  0 siblings, 0 replies; 12+ messages in thread
From: Sebastian Andrzej Siewior @ 2026-08-20 16:09 UTC (permalink / raw)
  To: Yury Norov
  Cc: Florian Bezdeka, Maxime Chevallier, Andrew Lunn, David S. Miller,
	Eric Dumazet, Jakub Kicinski, Paolo Abeni, Maxime Coquelin,
	Alexandre Torgue, Yury Norov, Rasmus Villemoes, Andrew Morton,
	Clark Williams, Steven Rostedt, Thomas Gleixner, Jan Kiszka,
	netdev, linux-stm32, linux-arm-kernel, linux-kernel,
	linux-rt-devel

On 2026-08-19 14:38:22 [-0400], Yury Norov wrote:
> > --- a/lib/cpumask.c
> > +++ b/lib/cpumask.c
> > @@ -81,8 +82,9 @@ void __init free_bootmem_cpumask_var(cpumask_var_t mask)
> >   * @i: index number
> >   * @node: local numa_node
> >   *
> > - * Return: online CPU according to a numa aware policy; local cpus are returned
> > - * first, followed by non-local ones, then it wraps around.
> > + * Return: online CPU according to the default IRQ affinity and a numa aware
> > + * policy; local cpus are returned first, followed by non-local ones, then it
> > + * wraps around.
> >   *
> >   * For those who wants to enumerate all CPUs based on their NUMA distances,
> >   * i.e. call this function in a loop, like:
> > @@ -110,9 +112,9 @@ unsigned int cpumask_local_spread(unsigned int i, int node)
> 
> Please don't touch this function. There's ~40 users, and we don't want
> to inspect every caller for their intention.

While this might be okay for the NICs, the usage of PMUs shouldn't be
changed. So yes, Yury's suggestion to create your own function and then
move the drivers one by one makes sense.
You might also want to verify that your setup/ design works on some
"newer" multi queue NICs even after the number of queues is reconfigured
at runtime.

There is also netif_set_affinity_auto().

Sebastian


^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: [PATCH RFC 2/3] genirq: Honor existing IRQ affinities when setting affinity hints
  2026-08-19 14:30 ` [PATCH RFC 2/3] genirq: Honor existing IRQ affinities when setting affinity hints Florian Bezdeka
@ 2026-08-20 16:41   ` Jakub Kicinski
  0 siblings, 0 replies; 12+ messages in thread
From: Jakub Kicinski @ 2026-08-20 16:41 UTC (permalink / raw)
  To: Florian Bezdeka
  Cc: Maxime Chevallier, Andrew Lunn, David S. Miller, Eric Dumazet,
	Paolo Abeni, Maxime Coquelin, Alexandre Torgue, Yury Norov,
	Rasmus Villemoes, Andrew Morton, Sebastian Andrzej Siewior,
	Clark Williams, Steven Rostedt, Thomas Gleixner, Jan Kiszka,
	netdev, linux-stm32, linux-arm-kernel, linux-kernel,
	linux-rt-devel

On Wed, 19 Aug 2026 16:30:31 +0200 Florian Bezdeka wrote:
> Some device drivers implement an IRQ balancing / spreading mechanism
> based on cpumask_local_spread() and irq_set_affinity_hint() (deprecated)
> or irq_set_affinity_and_hint().

Not an expert on IRQs, but this may make the API even more confusing.
irq_set_affinity_and_hint() not setting the affinity seems odd.

I'd go the other way and either have irq_update_affinity_hint() apply
the affinity if it's safe to do so, or create a new function for
"defaults".

Really, the drivers are dumb here, they are just trying to pass to 
the core a hint which is: This is Nth of X IRQs, please spread them on
the NUMA node where the device is attached. Ideally this (Nth of X)
should even be part of request_irq() args. IRQ core already knows the
device/NUMA node. Drivers juggling multiple flavors of uAPI-shaped API
is... not very scalable. People rarely get this right.


^ permalink raw reply	[flat|nested] 12+ messages in thread

end of thread, other threads:[~2026-08-20 16:42 UTC | newest]

Thread overview: 12+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2026-08-19 14:30 [PATCH RFC 0/3] genirq: Allow drivers to respect userspace IRQ affinities Florian Bezdeka
2026-08-19 14:30 ` [PATCH RFC 1/3] cpumask: Honor irq_default_affinity in cpumask_local_spread() Florian Bezdeka
2026-08-19 18:38   ` Yury Norov
2026-08-20 16:09     ` Sebastian Andrzej Siewior
2026-08-19 14:30 ` [PATCH RFC 2/3] genirq: Honor existing IRQ affinities when setting affinity hints Florian Bezdeka
2026-08-20 16:41   ` Jakub Kicinski
2026-08-19 14:30 ` [PATCH RFC 3/3] net: stmmac: Migrate IRQ balancing to cpumask_local_spread() Florian Bezdeka
2026-08-19 18:45   ` Yury Norov
2026-08-19 18:28 ` [PATCH RFC 0/3] genirq: Allow drivers to respect userspace IRQ affinities Yury Norov
2026-08-19 23:54 ` Andrew Lunn
2026-08-20  0:10   ` Andrew Lunn
2026-08-20 15:12 ` Sebastian Andrzej Siewior

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox