[PATCH 1/6] genirq: allow assigning affinity to present but not online CPUs

[PATCH 1/6] genirq: allow assigning affinity to present but not online CPUs

[Christoph Hellwig]

This will allow us to spread MSI/MSI-X affinity over all present CPUs and
thus better deal with systems where cpus are take on and offline all the
time.

Signed-off-by: Christoph Hellwig <hch@lst.de>
---
 kernel/irq/manage.c | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/kernel/irq/manage.c b/kernel/irq/manage.c
index 6b669593e7eb..7a1424330f9c 100644
--- a/kernel/irq/manage.c
+++ b/kernel/irq/manage.c
@@ -354,22 +354,22 @@ static int setup_affinity(struct irq_desc *desc, struct cpumask *mask)
 
 	/*
 	 * Preserve the managed affinity setting and an userspace affinity
-	 * setup, but make sure that one of the targets is online.
+	 * setup, but make sure that one of the targets is present.
 	 */
 	if (irqd_affinity_is_managed(&desc->irq_data) ||
 	    irqd_has_set(&desc->irq_data, IRQD_AFFINITY_SET)) {
 		if (cpumask_intersects(desc->irq_common_data.affinity,
-				       cpu_online_mask))
+				       cpu_present_mask))
 			set = desc->irq_common_data.affinity;
 		else
 			irqd_clear(&desc->irq_data, IRQD_AFFINITY_SET);
 	}
 
-	cpumask_and(mask, cpu_online_mask, set);
+	cpumask_and(mask, cpu_present_mask, set);
 	if (node != NUMA_NO_NODE) {
 		const struct cpumask *nodemask = cpumask_of_node(node);
 
-		/* make sure at least one of the cpus in nodemask is online */
+		/* make sure at least one of the cpus in nodemask is present */
 		if (cpumask_intersects(mask, nodemask))
 			cpumask_and(mask, mask, nodemask);
 	}
-- 
2.11.0

Re: [PATCH 1/6] genirq: allow assigning affinity to present but not online CPUs

[Christoph Hellwig]

Hi Joe,

On Fri, Feb 03, 2017 at 08:58:09PM -0500, Joe Korty wrote:
> IIRC, some years ago I ran across a customer system where
> the #cpus_present was twice as big as #cpus_possible.
> 
> Hyperthreading was turned off in the BIOS so it was not
> entirely out of line for the extra cpus to be declared
> present, even though none of them would ever be available
> for use.

This sounds like a system we should quirk around instead of optimizing
for it.  Unless I totally misunderstand the idea behind cpu_possible
and cpu_present.

Re: [PATCH 1/6] genirq: allow assigning affinity to present but not online CPUs

[Keith Busch]

On Sun, Feb 05, 2017 at 05:40:23PM +0100, Christoph Hellwig wrote:
> Hi Joe,
> 
> On Fri, Feb 03, 2017 at 08:58:09PM -0500, Joe Korty wrote:
> > IIRC, some years ago I ran across a customer system where
> > the #cpus_present was twice as big as #cpus_possible.
> > 
> > Hyperthreading was turned off in the BIOS so it was not
> > entirely out of line for the extra cpus to be declared
> > present, even though none of them would ever be available
> > for use.
> 
> This sounds like a system we should quirk around instead of optimizing
> for it.  Unless I totally misunderstand the idea behind cpu_possible
> and cpu_present.

Can we use the online CPUs and create a new hot-cpu notifier to the nvme
driver to free/reallocate as needed? We were doing that before blk-mq. Now
blk-mq can change the number hardware contexts on a live queue, so we
can reintroduce that behavior to nvme and only allocate what we need.

Re: [PATCH 1/6] genirq: allow assigning affinity to present but not online CPUs

[Christoph Hellwig]

On Mon, Feb 06, 2017 at 12:03:05PM -0500, Keith Busch wrote:
> Can we use the online CPUs and create a new hot-cpu notifier to the nvme
> driver to free/reallocate as needed? We were doing that before blk-mq. Now
> blk-mq can change the number hardware contexts on a live queue, so we
> can reintroduce that behavior to nvme and only allocate what we need.

That could be a next step, but given how badly I keep messing up core
IRQ bits we'll probably want to wait for two rewrites of those bits
from Thomas before moving on :)