[PATCH] genirq: Limit irq_calc_affinity_vectors to housekeeping CPUs

[PATCH] genirq: Limit irq_calc_affinity_vectors to housekeeping CPUs

[Liu Chao]

Replace the cpumask used in irq_calc_affinity_vectors from all possible
CPUs to only housekeeping CPUs.

When we have isolated CPUs used by real-time tasks, IRQs will be move to
housekeeping CPUs.

If there are too many IRQ vectors, moving the all to housekeeping CPUs may
exceed per-CPU vector limits. For example, when I only have two
housekeeping CPUs, there are dozens of IRQs on two CPUs, but actually one
IRQ per housekeeping CPU is enough.

Signed-off-by: Liu Chao <liuchao173@huawei.com>
---
 kernel/irq/affinity.c | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/kernel/irq/affinity.c b/kernel/irq/affinity.c
index 4d89ad4fae3b..3f22e3314e1b 100644
--- a/kernel/irq/affinity.c
+++ b/kernel/irq/affinity.c
@@ -8,6 +8,7 @@
 #include <linux/slab.h>
 #include <linux/cpu.h>
 #include <linux/sort.h>
+#include <linux/sched/isolation.h>
 
 static void irq_spread_init_one(struct cpumask *irqmsk, struct cpumask *nmsk,
 				unsigned int cpus_per_vec)
@@ -506,7 +507,8 @@ unsigned int irq_calc_affinity_vectors(unsigned int minvec, unsigned int maxvec,
 		set_vecs = maxvec - resv;
 	} else {
 		get_online_cpus();
-		set_vecs = cpumask_weight(cpu_possible_mask);
+		set_vecs = cpumask_weight(
+				housekeeping_cpumask(HK_FLAG_MANAGED_IRQ));
 		put_online_cpus();
 	}
 
-- 
2.23.0

Re: [PATCH] genirq: Limit irq_calc_affinity_vectors to housekeeping CPUs

[Thomas Gleixner]

Liu,

On Fri, Jan 22 2021 at 08:47, Liu Chao wrote:
> Replace the cpumask used in irq_calc_affinity_vectors from all possible
> CPUs to only housekeeping CPUs.
>
> When we have isolated CPUs used by real-time tasks, IRQs will be move to
> housekeeping CPUs.

No.

> If there are too many IRQ vectors, moving the all to housekeeping CPUs may
> exceed per-CPU vector limits. For example, when I only have two
> housekeeping CPUs, there are dozens of IRQs on two CPUs, but actually one
> IRQ per housekeeping CPU is enough.

This does not make any sense.

Assumed a system has 256 possible CPUs and the device allows 256 queues
then each possible CPU will have ONE queue and ONE associated
interrupt. Nothing will change the affinity of any of those interrupts
ever.

The only way how the housekeeping mask interacts with multiqueue
affinities is when there are less queues and therefore less interrupts
than CPUs which means that one queue and it's interrupt is associated to
multiple CPUs. So the resulting affinity mask might spawn housekeeping
and isolated CPUs. When the affinity is set up for the interrupt in
hardware then the selection logic which choses a target CPU based on the
interrupt affinity mask will prefer the housekeeping CPUs and avoid the
isolated ones. But that does not cause vector exhaustion except you have
more devices than vectors.

So for a single multiqueue device a housekeeping CPU will not have more
than one managed queue interrupt ever.

I'm not understanding which problem are you trying to solve.

Thanks,

        tglx

[PATCH] genirq: Limit irq_calc_affinity_vectors to housekeeping CPUs

[Liu Chao]

Replace the cpumask used in irq_calc_affinity_vectors from all possible
CPUs to only housekeeping CPUs.

When we have isolated CPUs used by real-time tasks, IRQs will be move to
housekeeping CPUs.

If there are too many IRQ vectors, moving the all to housekeeping CPUs may
exceed per-CPU vector limits. For example, when I only have two
housekeeping CPUs, there are dozens of IRQs on two CPUs, but actually one
IRQ per housekeeping CPU is enough.

Signed-off-by: Liu Chao <liuchao173@huawei.com>
---
 kernel/irq/affinity.c | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/kernel/irq/affinity.c b/kernel/irq/affinity.c
index 4d89ad4fae3b..3f22e3314e1b 100644
--- a/kernel/irq/affinity.c
+++ b/kernel/irq/affinity.c
@@ -8,6 +8,7 @@
 #include <linux/slab.h>
 #include <linux/cpu.h>
 #include <linux/sort.h>
+#include <linux/sched/isolation.h>
 
 static void irq_spread_init_one(struct cpumask *irqmsk, struct cpumask *nmsk,
 				unsigned int cpus_per_vec)
@@ -506,7 +507,8 @@ unsigned int irq_calc_affinity_vectors(unsigned int minvec, unsigned int maxvec,
 		set_vecs = maxvec - resv;
 	} else {
 		get_online_cpus();
-		set_vecs = cpumask_weight(cpu_possible_mask);
+		set_vecs = cpumask_weight(
+				housekeeping_cpumask(HK_FLAG_MANAGED_IRQ));
 		put_online_cpus();
 	}
 
-- 
2.23.0