Re: [PATCH 1/6] genirq: allow assigning affinity to present but not online CPUs

Re: [PATCH 1/6] genirq: allow assigning affinity to present but not online CPUs

[Christoph Hellwig]

Hi Joe,

On Fri, Feb 03, 2017 at 08:58:09PM -0500, Joe Korty wrote:
> IIRC, some years ago I ran across a customer system where
> the #cpus_present was twice as big as #cpus_possible.
> 
> Hyperthreading was turned off in the BIOS so it was not
> entirely out of line for the extra cpus to be declared
> present, even though none of them would ever be available
> for use.

This sounds like a system we should quirk around instead of optimizing
for it.  Unless I totally misunderstand the idea behind cpu_possible
and cpu_present.

Re: [PATCH 1/6] genirq: allow assigning affinity to present but not online CPUs

[Keith Busch]

On Sun, Feb 05, 2017 at 05:40:23PM +0100, Christoph Hellwig wrote:
> Hi Joe,
> 
> On Fri, Feb 03, 2017 at 08:58:09PM -0500, Joe Korty wrote:
> > IIRC, some years ago I ran across a customer system where
> > the #cpus_present was twice as big as #cpus_possible.
> > 
> > Hyperthreading was turned off in the BIOS so it was not
> > entirely out of line for the extra cpus to be declared
> > present, even though none of them would ever be available
> > for use.
> 
> This sounds like a system we should quirk around instead of optimizing
> for it.  Unless I totally misunderstand the idea behind cpu_possible
> and cpu_present.

Can we use the online CPUs and create a new hot-cpu notifier to the nvme
driver to free/reallocate as needed? We were doing that before blk-mq. Now
blk-mq can change the number hardware contexts on a live queue, so we
can reintroduce that behavior to nvme and only allocate what we need.

Re: [PATCH 1/6] genirq: allow assigning affinity to present but not online CPUs

[Christoph Hellwig]

On Mon, Feb 06, 2017 at 12:03:05PM -0500, Keith Busch wrote:
> Can we use the online CPUs and create a new hot-cpu notifier to the nvme
> driver to free/reallocate as needed? We were doing that before blk-mq. Now
> blk-mq can change the number hardware contexts on a live queue, so we
> can reintroduce that behavior to nvme and only allocate what we need.

That could be a next step, but given how badly I keep messing up core
IRQ bits we'll probably want to wait for two rewrites of those bits
from Thomas before moving on :)

spread MSI(-X) vectors to all possible CPUs

[Christoph Hellwig]

Hi all,

this series changes our automatic MSI-X vector assignment so that it
takes all present CPUs into account instead of all online ones.  This
allows to better deal with cpu hotplug events, which could happen
frequently due to power management for example.

[PATCH 1/6] genirq: allow assigning affinity to present but not online CPUs

[Christoph Hellwig]

This will allow us to spread MSI/MSI-X affinity over all present CPUs and
thus better deal with systems where cpus are take on and offline all the
time.

Signed-off-by: Christoph Hellwig <hch@lst.de>
---
 kernel/irq/manage.c | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/kernel/irq/manage.c b/kernel/irq/manage.c
index 6b669593e7eb..7a1424330f9c 100644
--- a/kernel/irq/manage.c
+++ b/kernel/irq/manage.c
@@ -354,22 +354,22 @@ static int setup_affinity(struct irq_desc *desc, struct cpumask *mask)
 
 	/*
 	 * Preserve the managed affinity setting and an userspace affinity
-	 * setup, but make sure that one of the targets is online.
+	 * setup, but make sure that one of the targets is present.
 	 */
 	if (irqd_affinity_is_managed(&desc->irq_data) ||
 	    irqd_has_set(&desc->irq_data, IRQD_AFFINITY_SET)) {
 		if (cpumask_intersects(desc->irq_common_data.affinity,
-				       cpu_online_mask))
+				       cpu_present_mask))
 			set = desc->irq_common_data.affinity;
 		else
 			irqd_clear(&desc->irq_data, IRQD_AFFINITY_SET);
 	}
 
-	cpumask_and(mask, cpu_online_mask, set);
+	cpumask_and(mask, cpu_present_mask, set);
 	if (node != NUMA_NO_NODE) {
 		const struct cpumask *nodemask = cpumask_of_node(node);
 
-		/* make sure at least one of the cpus in nodemask is online */
+		/* make sure at least one of the cpus in nodemask is present */
 		if (cpumask_intersects(mask, nodemask))
 			cpumask_and(mask, mask, nodemask);
 	}
-- 
2.11.0