[PATCH 1/4] 2.6.2-rc2-mm2 CPU Hotplug: cpu_active_map

[PATCH 1/4] 2.6.2-rc2-mm2 CPU Hotplug: cpu_active_map

[Rusty Russell]

Hi Andrew et al.

	Forward ported to -mm tree, so there might still be bugs.
Wanted to get these aired more widely.  Especially get Nick exposed to
the sched parts.

Name: Create cpu_active_map
Author: Rusty Russell
Status: Booted on 2.6.2-rc2-bk2

D: When CPUs are going down, there is a time when cpu_online(cpu) is
D: false, but they are still scheduling and responding to interrupts
D: (we are migrating things off the CPU, shutting down per-cpu
D: threads, etc).  It turns out that RCU cares about these CPUs, so
D: the decision was made to expose this mask (previously internal to x86,
D: and only used for IPIs).
D: 
D: The semantics of this mask are as follows:
D: 1) For platforms without hot unplug of CPUs: cpu_active_map ==
D:    cpu_online_map.
D: 2) For the others, they are equal except for a CPU which is going
D:    down: cpu_online_map gets cleared by __cpu_disable(), cpu_ipi_map
D:    gets cleared in __cpu_die() once the CPU is no longer responding
D:    to interrupts.

diff -urpN --exclude TAGS -X /home/rusty/devel/kernel/kernel-patches/current-dontdiff --minimal .25707-linux-2.6.0-test11-bk7/include/linux/cpumask.h .25707-linux-2.6.0-test11-bk7.updated/include/linux/cpumask.h
--- .25707-linux-2.6.0-test11-bk7/include/linux/cpumask.h	2003-12-10 13:58:18.000000000 +1100
+++ .25707-linux-2.6.0-test11-bk7.updated/include/linux/cpumask.h	2003-12-10 14:01:29.000000000 +1100
@@ -43,6 +43,13 @@ typedef unsigned long cpumask_t;
 extern cpumask_t cpu_online_map;
 extern cpumask_t cpu_possible_map;
 
+#ifdef CONFIG_HOTPLUG_CPU
+/* Online, or on its way down but still receiving interrupts. */
+extern cpumask_t cpu_active_map;
+#else
+#define cpu_active_map cpu_online_map
+#endif
+
 #define num_online_cpus()		cpus_weight(cpu_online_map)
 #define cpu_online(cpu)			cpu_isset(cpu, cpu_online_map)
 #define cpu_possible(cpu)		cpu_isset(cpu, cpu_possible_map)
diff -urpN --exclude TAGS -X /home/rusty/devel/kernel/kernel-patches/current-dontdiff --minimal .25707-linux-2.6.0-test11-bk7/kernel/rcupdate.c .25707-linux-2.6.0-test11-bk7.updated/kernel/rcupdate.c
--- .25707-linux-2.6.0-test11-bk7/kernel/rcupdate.c	2003-12-10 13:58:19.000000000 +1100
+++ .25707-linux-2.6.0-test11-bk7.updated/kernel/rcupdate.c	2003-12-10 14:01:03.000000000 +1100
@@ -110,7 +110,7 @@ static void rcu_start_batch(long newbatc
 	    !cpus_empty(rcu_ctrlblk.rcu_cpu_mask)) {
 		return;
 	}
-	rcu_ctrlblk.rcu_cpu_mask = cpu_online_map;
+	rcu_ctrlblk.rcu_cpu_mask = cpu_active_map;
 }
 
 /*
--
  Anyone who quotes me in their sig is an idiot. -- Rusty Russell.

Re: [PATCH 1/4] 2.6.2-rc2-mm2 CPU Hotplug: cpu_active_map

[Ingo Molnar]

* Rusty Russell <rusty@rustcorp.com.au> wrote:

> D: When CPUs are going down, there is a time when cpu_online(cpu) is
> D: false, but they are still scheduling and responding to interrupts
> D: (we are migrating things off the CPU, shutting down per-cpu
> D: threads, etc).  It turns out that RCU cares about these CPUs, so
> D: the decision was made to expose this mask (previously internal to x86,
> D: and only used for IPIs).

these kinds of problems could be avoided by making the CPU-off as much
of an atomic operation as possible. The less atomic it is, the more
kernel code is exposed to the transitional state - and since this is a
rare situation it will always have quality problems. Is there any killer
argument that makes it impossible to down a CPU atomically? Kernel
threads can get their callbacks on other CPUs just fine. Other tasks
should not care. If the migrate-off operation is done 100% atomically
then zero knowledge is needed by unrelated scheduler code about the act
of disabling a CPU.

	Ingo

Re: [PATCH 1/4] 2.6.2-rc2-mm2 CPU Hotplug: cpu_active_map

[Rusty Russell]

In message <20040202100827.GA28870@elte.hu> you write:
> 
> * Rusty Russell <rusty@rustcorp.com.au> wrote:
> 
> > D: When CPUs are going down, there is a time when cpu_online(cpu) is
> > D: false, but they are still scheduling and responding to interrupts
> > D: (we are migrating things off the CPU, shutting down per-cpu
> > D: threads, etc).  It turns out that RCU cares about these CPUs, so
> > D: the decision was made to expose this mask (previously internal to x86,
> > D: and only used for IPIs).
> 
> these kinds of problems could be avoided by making the CPU-off as much
> of an atomic operation as possible. The less atomic it is, the more
> kernel code is exposed to the transitional state - and since this is a
> rare situation it will always have quality problems. Is there any killer
> argument that makes it impossible to down a CPU atomically? Kernel
> threads can get their callbacks on other CPUs just fine.

Well, we could grab control of the machine, unconditionally move all
the threads and kill the cpu.

We'd need a lot of locks, but we could just use a bogolock (a-la
module.c's schedule a thread per cpu and tell them all to disable
interrupts) which has the same effect of grabbing every spinlock.

It means that kernel threads must not use smp_processor_id(), because
it might change under them (get_cpu() would protect from this).  You'd
still need explicit shutdown code.

> Other tasks should not care. If the migrate-off operation is done
> 100% atomically then zero knowledge is needed by unrelated scheduler
> code about the act of disabling a CPU.

No, migration code still needs to check the cpu is still up after
you've slept.

I'll sleep on it and see if it still sounds like a good idea in the
morning 8)
Rusty.
--
  Anyone who quotes me in their sig is an idiot. -- Rusty Russell.

Re: [PATCH 1/4] 2.6.2-rc2-mm2 CPU Hotplug: cpu_active_map

[Matthew Dobson]

Rusty Russell wrote:
> D: The semantics of this mask are as follows:
> D: 1) For platforms without hot unplug of CPUs: cpu_active_map ==
> D:    cpu_online_map.
> D: 2) For the others, they are equal except for a CPU which is going
> D:    down: cpu_online_map gets cleared by __cpu_disable(), cpu_ipi_map
> D:    gets cleared in __cpu_die() once the CPU is no longer responding
> D:    to interrupts.

Small typo, but I believe you meant to type 'cpu_active_map' in part 2, 
not 'cpu_ipi_map'.

Cheers!

-Matt