[RFC][PATCH] node hotplug : register_cpu() changes [0/3]

[RFC][PATCH] node hotplug : register_cpu() changes [0/3]

[KAMEZAWA Hiroyuki]

With Goto-san's patch, we can add new pgdat/node in runtime.
I'm now considering node-hot-add with cpu + memory on ACPI.

I found acpi container, which describes node, could evaluate cpu before
memory. This means cpu-hot-add occurs before memory hot add.

In most part, cpu-hot-add doesn't depend on node hot add.
But register_cpu, which creates symbolic link from node to cpu, requires
that node should be onlined before register_cpu().
When a node is onlined, its pgdat should be there.

This patch-set holds off creating symbolic link from node to cpu
until node is onlined. 

[1/3] modifies register_cpu
[2/3] changes caller of register cpu
[3/3] changes register_node. create symbolic link fron node to cpu.

By these patch, (cpu + memory) node hot add will succeed.
(cpu-only)/(IO-only) node hot add will need more fixes. (But we need this ?)

And ia64 needs more fixes. I'll post later.

-Kame

[RFC][PATCH] node hotplug [1/3] register-cpu-remove-node-struct

[KAMEZAWA Hiroyuki]

This removes node arguments from register_cpu().

Now, register_cpu() requires 'struct node' as its argument.
But the array of struct node is now unified in driver/base/node.c now
(By Goto's node hotplug patch).
We can get struct node in generic way. So, this argument is not necessary now. 

This patch also guarantees add cpu under node only when node is onlined.
It is necessary for node-hot-add vs. cpu-hot-add patch following this.

Moreover, register_cpu calculates cpu->node_id by cpu_to_node() without regard
to its 'struct node *root' argument. This patch removes it.

Patches for fixing the caller of register_cpu() for each arch will follow this.

Signed-Off-By : KAMEZAWA Hiroyuki <kamezawa.hiroyu@jp.fujitsu.com>

 drivers/base/cpu.c   |   18 ++++++++----------
 drivers/base/node.c  |   32 ++++++++++++++++++++++++++++++++
 include/linux/cpu.h  |    2 +-
 include/linux/node.h |   13 +++++++++++++
 4 files changed, 54 insertions(+), 11 deletions(-)

Index: linux-2.6.17-rc4-mm3/drivers/base/cpu.c
===================================================================
--- linux-2.6.17-rc4-mm3.orig/drivers/base/cpu.c	2006-05-23 17:31:49.000000000 +0900
+++ linux-2.6.17-rc4-mm3/drivers/base/cpu.c	2006-05-23 19:18:18.000000000 +0900
@@ -8,6 +8,7 @@
 #include <linux/cpu.h>
 #include <linux/topology.h>
 #include <linux/device.h>
+#include <linux/node.h>
 
 #include "base.h"
 
@@ -57,13 +58,12 @@
 {
 	sysdev_create_file(&cpu->sysdev, &attr_online);
 }
-void unregister_cpu(struct cpu *cpu, struct node *root)
+void unregister_cpu(struct cpu *cpu)
 {
 	int logical_cpu = cpu->sysdev.id;
 
-	if (root)
-		sysfs_remove_link(&root->sysdev.kobj,
-				  kobject_name(&cpu->sysdev.kobj));
+	unregister_cpu_under_node(logical_cpu, cpu_to_node(logical_cpu));
+
 	sysdev_remove_file(&cpu->sysdev, &attr_online);
 
 	sysdev_unregister(&cpu->sysdev);
@@ -109,23 +109,21 @@
  *
  * Initialize and register the CPU device.
  */
-int __devinit register_cpu(struct cpu *cpu, int num, struct node *root)
+int __devinit register_cpu(struct cpu *cpu, int num)
 {
 	int error;
-
 	cpu->node_id = cpu_to_node(num);
 	cpu->sysdev.id = num;
 	cpu->sysdev.cls = &cpu_sysdev_class;
 
 	error = sysdev_register(&cpu->sysdev);
-	if (!error && root)
-		error = sysfs_create_link(&root->sysdev.kobj,
-					  &cpu->sysdev.kobj,
-					  kobject_name(&cpu->sysdev.kobj));
+
 	if (!error && !cpu->no_control)
 		register_cpu_control(cpu);
 	if (!error)
 		cpu_sys_devices[num] = &cpu->sysdev;
+	if (!error)
+		register_cpu_under_node(num, cpu_to_node(num));
 
 #ifdef CONFIG_KEXEC
 	if (!error)
Index: linux-2.6.17-rc4-mm3/drivers/base/node.c
===================================================================
--- linux-2.6.17-rc4-mm3.orig/drivers/base/node.c	2006-05-23 17:33:00.000000000 +0900
+++ linux-2.6.17-rc4-mm3/drivers/base/node.c	2006-05-23 19:21:26.000000000 +0900
@@ -11,6 +11,7 @@
 #include <linux/cpumask.h>
 #include <linux/topology.h>
 #include <linux/nodemask.h>
+#include <linux/cpu.h>
 
 static struct sysdev_class node_class = {
 	set_kset_name("node"),
@@ -192,6 +193,37 @@
 
 struct node node_devices[MAX_NUMNODES];
 
+/*
+ * register cpu under node
+ */
+int register_cpu_under_node(unsigned int cpu, unsigned int nid)
+{
+	struct sys_device *obj;
+	if (node_online(nid)) {
+		obj = get_cpu_sysdev(cpu);
+		if (!obj)
+			return 0;
+		return sysfs_create_link(&node_devices[nid].sysdev.kobj,
+					 &obj->kobj,
+					 kobject_name(&obj->kobj));
+	 }
+
+	return 0;
+}
+
+int unregister_cpu_under_node(unsigned int cpu, unsigned int nid)
+{
+	struct sys_device *obj;
+	if (node_online(nid)) {
+		obj = get_cpu_sysdev(cpu);
+		if (!obj)
+			return 0;
+		return sysfs_remove_link(&node_devices[nid].sysdev.kobj,
+					 kobject_name(&obj->kobj));
+	}
+	return 0;
+}
+
 int register_one_node(int nid)
 {
 	int error = 0;
Index: linux-2.6.17-rc4-mm3/include/linux/node.h
===================================================================
--- linux-2.6.17-rc4-mm3.orig/include/linux/node.h	2006-05-23 17:33:06.000000000 +0900
+++ linux-2.6.17-rc4-mm3/include/linux/node.h	2006-05-23 19:14:30.000000000 +0900
@@ -32,6 +32,19 @@
 extern void unregister_node(struct node *node);
 extern int register_one_node(int nid);
 extern void unregister_one_node(int nid);
+#ifdef CONFIG_NUMA
+extern int register_cpu_under_node(unsigned int cpu, unsigned int nid);
+extern int unregister_cpu_under_node(unsigned int cpu, unsigned int nid);
+#else
+static inline int register_cpu_under_node(unsigned int cpu, unsigned int nid)
+{
+	return 0;
+}
+static inline int unregister_cpu_under_node(unsigned int cpu, unsigned int nid)
+{
+	return 0;
+}
+#endif
 
 #define to_node(sys_device) container_of(sys_device, struct node, sysdev)
 
Index: linux-2.6.17-rc4-mm3/include/linux/cpu.h
===================================================================
--- linux-2.6.17-rc4-mm3.orig/include/linux/cpu.h	2006-05-23 17:31:58.000000000 +0900
+++ linux-2.6.17-rc4-mm3/include/linux/cpu.h	2006-05-23 19:24:23.000000000 +0900
@@ -31,10 +31,10 @@
 	struct sys_device sysdev;
 };
 
-extern int register_cpu(struct cpu *, int, struct node *);
+extern int register_cpu(struct cpu *, int);
 extern struct sys_device *get_cpu_sysdev(unsigned cpu);
 #ifdef CONFIG_HOTPLUG_CPU
-extern void unregister_cpu(struct cpu *, struct node *);
+extern void unregister_cpu(struct cpu *);
 #endif
 struct notifier_block;
 

[RFC][PATCH] node hotplug [2/3] fixes callres of register_cpu

[KAMEZAWA Hiroyuki]

This patch modifies caller of register_cpu()/unregister_cpu, whose args are
changed by register-cpu-remove-node-struct patch.

Signed-Off-By: KAMEZAWA Hiroyuki <kamezawa.hiroyu@jp.fujitsu.ocm>

 arch/arm/kernel/setup.c        |    2 +-
 arch/i386/kernel/topology.c    |   18 ++----------------
 arch/ia64/kernel/topology.c    |   17 ++---------------
 arch/m32r/kernel/setup.c       |    2 +-
 arch/mips/kernel/smp.c         |    2 +-
 arch/parisc/kernel/topology.c  |    3 +--
 arch/powerpc/kernel/setup_32.c |    2 +-
 arch/powerpc/kernel/sysfs.c    |   10 +---------
 arch/ppc/kernel/setup.c        |    2 +-
 arch/s390/kernel/smp.c         |    2 +-
 arch/sh/kernel/setup.c         |    2 +-
 arch/sh64/kernel/setup.c       |    2 +-
 arch/sparc64/kernel/setup.c    |    2 +-
 13 files changed, 15 insertions(+), 51 deletions(-)

Index: linux-2.6.17-rc4-mm3/arch/arm/kernel/setup.c
===================================================================
--- linux-2.6.17-rc4-mm3.orig/arch/arm/kernel/setup.c	2006-05-23 19:23:35.000000000 +0900
+++ linux-2.6.17-rc4-mm3/arch/arm/kernel/setup.c	2006-05-23 19:27:16.000000000 +0900
@@ -838,7 +838,7 @@
 	int cpu;
 
 	for_each_possible_cpu(cpu)
-		register_cpu(&per_cpu(cpu_data, cpu).cpu, cpu, NULL);
+		register_cpu(&per_cpu(cpu_data, cpu).cpu, cpu);
 
 	return 0;
 }
Index: linux-2.6.17-rc4-mm3/arch/i386/kernel/topology.c
===================================================================
--- linux-2.6.17-rc4-mm3.orig/arch/i386/kernel/topology.c	2006-05-23 19:23:35.000000000 +0900
+++ linux-2.6.17-rc4-mm3/arch/i386/kernel/topology.c	2006-05-23 19:27:16.000000000 +0900
@@ -35,12 +35,6 @@
 int arch_register_cpu(int num){
 	struct node *parent = NULL;
 
-#ifdef CONFIG_NUMA
-	int node = cpu_to_node(num);
-	if (node_online(node))
-		parent = &node_devices[parent_node(node)];
-#endif /* CONFIG_NUMA */
-
 	/*
 	 * CPU0 cannot be offlined due to several
 	 * restrictions and assumptions in kernel. This basically
@@ -50,21 +44,13 @@
 	if (!num)
 		cpu_devices[num].cpu.no_control = 1;
 
-	return register_cpu(&cpu_devices[num].cpu, num, parent);
+	return register_cpu(&cpu_devices[num].cpu, num);
 }
 
 #ifdef CONFIG_HOTPLUG_CPU
 
 void arch_unregister_cpu(int num) {
-	struct node *parent = NULL;
-
-#ifdef CONFIG_NUMA
-	int node = cpu_to_node(num);
-	if (node_online(node))
-		parent = &node_devices[parent_node(node)];
-#endif /* CONFIG_NUMA */
-
-	return unregister_cpu(&cpu_devices[num].cpu, parent);
+	return unregister_cpu(&cpu_devices[num].cpu);
 }
 EXPORT_SYMBOL(arch_register_cpu);
 EXPORT_SYMBOL(arch_unregister_cpu);
Index: linux-2.6.17-rc4-mm3/arch/m32r/kernel/setup.c
===================================================================
--- linux-2.6.17-rc4-mm3.orig/arch/m32r/kernel/setup.c	2006-05-23 19:23:35.000000000 +0900
+++ linux-2.6.17-rc4-mm3/arch/m32r/kernel/setup.c	2006-05-23 19:27:16.000000000 +0900
@@ -275,7 +275,7 @@
 	int i;
 
 	for_each_present_cpu(i)
-		register_cpu(&cpu_devices[i], i, NULL);
+		register_cpu(&cpu_devices[i], i);
 
 	return 0;
 }
Index: linux-2.6.17-rc4-mm3/arch/mips/kernel/smp.c
===================================================================
--- linux-2.6.17-rc4-mm3.orig/arch/mips/kernel/smp.c	2006-05-23 19:23:35.000000000 +0900
+++ linux-2.6.17-rc4-mm3/arch/mips/kernel/smp.c	2006-05-23 19:27:16.000000000 +0900
@@ -443,7 +443,7 @@
 	int ret;
 
 	for_each_possible_cpu(cpu) {
-		ret = register_cpu(&per_cpu(cpu_devices, cpu), cpu, NULL);
+		ret = register_cpu(&per_cpu(cpu_devices, cpu), cpu);
 		if (ret)
 			printk(KERN_WARNING "topology_init: register_cpu %d "
 			       "failed (%d)\n", cpu, ret);
Index: linux-2.6.17-rc4-mm3/arch/parisc/kernel/topology.c
===================================================================
--- linux-2.6.17-rc4-mm3.orig/arch/parisc/kernel/topology.c	2006-05-23 19:23:35.000000000 +0900
+++ linux-2.6.17-rc4-mm3/arch/parisc/kernel/topology.c	2006-05-23 19:27:16.000000000 +0900
@@ -26,11 +26,10 @@
 
 static int __init topology_init(void)
 {
-	struct node *parent = NULL;
 	int num;
 
 	for_each_present_cpu(num) {
-		register_cpu(&cpu_devices[num], num, parent);
+		register_cpu(&cpu_devices[num], num);
 	}
 	return 0;
 }
Index: linux-2.6.17-rc4-mm3/arch/powerpc/kernel/setup_32.c
===================================================================
--- linux-2.6.17-rc4-mm3.orig/arch/powerpc/kernel/setup_32.c	2006-05-23 19:23:35.000000000 +0900
+++ linux-2.6.17-rc4-mm3/arch/powerpc/kernel/setup_32.c	2006-05-23 19:27:16.000000000 +0900
@@ -215,7 +215,7 @@
 
 	/* register CPU devices */
 	for_each_possible_cpu(i)
-		register_cpu(&cpu_devices[i], i, NULL);
+		register_cpu(&cpu_devices[i], i);
 
 	/* call platform init */
 	if (ppc_md.init != NULL) {
Index: linux-2.6.17-rc4-mm3/arch/powerpc/kernel/sysfs.c
===================================================================
--- linux-2.6.17-rc4-mm3.orig/arch/powerpc/kernel/sysfs.c	2006-05-23 19:23:35.000000000 +0900
+++ linux-2.6.17-rc4-mm3/arch/powerpc/kernel/sysfs.c	2006-05-23 19:27:16.000000000 +0900
@@ -357,14 +357,6 @@
 	for_each_possible_cpu(cpu) {
 		struct cpu *c = &per_cpu(cpu_devices, cpu);
 
-#ifdef CONFIG_NUMA
-		/* The node to which a cpu belongs can't be known
-		 * until the cpu is made present.
-		 */
-		parent = NULL;
-		if (cpu_present(cpu))
-			parent = &node_devices[cpu_to_node(cpu)];
-#endif
 		/*
 		 * For now, we just see if the system supports making
 		 * the RTAS calls for CPU hotplug.  But, there may be a
@@ -376,7 +368,7 @@
 			c->no_control = 1;
 
 		if (cpu_online(cpu) || (c->no_control == 0)) {
-			register_cpu(c, cpu, parent);
+			register_cpu(c, cpu);
 
 			sysdev_create_file(&c->sysdev, &attr_physical_id);
 		}
Index: linux-2.6.17-rc4-mm3/arch/ppc/kernel/setup.c
===================================================================
--- linux-2.6.17-rc4-mm3.orig/arch/ppc/kernel/setup.c	2006-05-23 19:23:35.000000000 +0900
+++ linux-2.6.17-rc4-mm3/arch/ppc/kernel/setup.c	2006-05-23 19:27:16.000000000 +0900
@@ -475,7 +475,7 @@
 
 	/* register CPU devices */
 	for_each_possible_cpu(i)
-		register_cpu(&cpu_devices[i], i, NULL);
+		register_cpu(&cpu_devices[i], i);
 
 	/* call platform init */
 	if (ppc_md.init != NULL) {
Index: linux-2.6.17-rc4-mm3/arch/s390/kernel/smp.c
===================================================================
--- linux-2.6.17-rc4-mm3.orig/arch/s390/kernel/smp.c	2006-05-23 19:23:35.000000000 +0900
+++ linux-2.6.17-rc4-mm3/arch/s390/kernel/smp.c	2006-05-23 19:27:16.000000000 +0900
@@ -869,7 +869,7 @@
 	int ret;
 
 	for_each_possible_cpu(cpu) {
-		ret = register_cpu(&per_cpu(cpu_devices, cpu), cpu, NULL);
+		ret = register_cpu(&per_cpu(cpu_devices, cpu), cpu);
 		if (ret)
 			printk(KERN_WARNING "topology_init: register_cpu %d "
 			       "failed (%d)\n", cpu, ret);
Index: linux-2.6.17-rc4-mm3/arch/sh/kernel/setup.c
===================================================================
--- linux-2.6.17-rc4-mm3.orig/arch/sh/kernel/setup.c	2006-05-23 19:23:35.000000000 +0900
+++ linux-2.6.17-rc4-mm3/arch/sh/kernel/setup.c	2006-05-23 19:27:16.000000000 +0900
@@ -402,7 +402,7 @@
 	int cpu_id;
 
 	for_each_possible_cpu(cpu_id)
-		register_cpu(&cpu[cpu_id], cpu_id, NULL);
+		register_cpu(&cpu[cpu_id], cpu_id);
 
 	return 0;
 }
Index: linux-2.6.17-rc4-mm3/arch/sh64/kernel/setup.c
===================================================================
--- linux-2.6.17-rc4-mm3.orig/arch/sh64/kernel/setup.c	2006-05-23 19:23:35.000000000 +0900
+++ linux-2.6.17-rc4-mm3/arch/sh64/kernel/setup.c	2006-05-23 19:27:16.000000000 +0900
@@ -309,7 +309,7 @@
 
 static int __init topology_init(void)
 {
-	return register_cpu(cpu, 0, NULL);
+	return register_cpu(cpu, 0);
 }
 
 subsys_initcall(topology_init);
Index: linux-2.6.17-rc4-mm3/arch/sparc64/kernel/setup.c
===================================================================
--- linux-2.6.17-rc4-mm3.orig/arch/sparc64/kernel/setup.c	2006-05-23 19:23:35.000000000 +0900
+++ linux-2.6.17-rc4-mm3/arch/sparc64/kernel/setup.c	2006-05-23 19:27:16.000000000 +0900
@@ -514,7 +514,7 @@
 	for_each_possible_cpu(i) {
 		struct cpu *p = kzalloc(sizeof(*p), GFP_KERNEL);
 		if (p) {
-			register_cpu(p, i, NULL);
+			register_cpu(p, i);
 			err = 0;
 		}
 	}
Index: linux-2.6.17-rc4-mm3/arch/ia64/kernel/topology.c
===================================================================
--- linux-2.6.17-rc4-mm3.orig/arch/ia64/kernel/topology.c	2006-05-23 19:23:35.000000000 +0900
+++ linux-2.6.17-rc4-mm3/arch/ia64/kernel/topology.c	2006-05-23 19:27:16.000000000 +0900
@@ -30,12 +30,6 @@
 
 int arch_register_cpu(int num)
 {
-	struct node *parent = NULL;
-	
-#ifdef CONFIG_NUMA
-	parent = &node_devices[cpu_to_node(num)];
-#endif /* CONFIG_NUMA */
-
 #if defined (CONFIG_ACPI) && defined (CONFIG_HOTPLUG_CPU)
 	/*
 	 * If CPEI cannot be re-targetted, and this is
@@ -45,21 +39,14 @@
 		sysfs_cpus[num].cpu.no_control = 1;
 #endif
 
-	return register_cpu(&sysfs_cpus[num].cpu, num, parent);
+	return register_cpu(&sysfs_cpus[num].cpu, num);
 }
 
 #ifdef CONFIG_HOTPLUG_CPU
 
 void arch_unregister_cpu(int num)
 {
-	struct node *parent = NULL;
-
-#ifdef CONFIG_NUMA
-	int node = cpu_to_node(num);
-	parent = &node_devices[node];
-#endif /* CONFIG_NUMA */
-
-	return unregister_cpu(&sysfs_cpus[num].cpu, parent);
+	return unregister_cpu(&sysfs_cpus[num].cpu);
 }
 EXPORT_SYMBOL(arch_register_cpu);
 EXPORT_SYMBOL(arch_unregister_cpu);

[RFC][PATCH] node hotplug [3/3] register_node fix

[KAMEZAWA Hiroyuki]

create symblic link from newly-added-node to its cpus.

Ideally, node should be onlined before cpu-hot-add. But it cannot be
always achieved (at node-hot-add.)

This patch registers cpus to its node when the node is registered.

Signed-Off-By: KAMEZAWA Hiroyuki <kamezawa.hiroyu@jp.fujitsu.com>

 drivers/base/node.c |    7 +++++++
 1 files changed, 7 insertions(+)

Index: linux-2.6.17-rc4-mm3/drivers/base/node.c
===================================================================
--- linux-2.6.17-rc4-mm3.orig/drivers/base/node.c	2006-05-23 19:21:26.000000000 +0900
+++ linux-2.6.17-rc4-mm3/drivers/base/node.c	2006-05-23 19:27:27.000000000 +0900
@@ -227,6 +227,7 @@
 int register_one_node(int nid)
 {
 	int error = 0;
+	int cpu;
 
 	if (node_online(nid)) {
 		int p_node = parent_node(nid);
@@ -236,6 +237,12 @@
 			parent = &node_devices[p_node];
 
 		error = register_node(&node_devices[nid], nid, parent);
+
+		/* link cpu under this node */
+		for_each_present_cpu(cpu) {
+			if (cpu_to_node(cpu) == nid)
+				register_cpu_under_node(cpu, nid);
+		}
 	}
 
 	return error;

Re: [RFC][PATCH] node hotplug : register_cpu() changes [0/3]

[Ashok Raj]

On Tue, May 23, 2006 at 07:56:36PM +0900, KAMEZAWA Hiroyuki wrote:
> I found acpi container, which describes node, could evaluate cpu before
> memory. This means cpu-hot-add occurs before memory hot add.
> 

Is it possible to process memory before cpu in container hot-add code?

> In most part, cpu-hot-add doesn't depend on node hot add.
> But register_cpu, which creates symbolic link from node to cpu, requires

Dont you need all per-cpu allocated on that node? Or is it from node0 or 
something for all hotpluggable cpus?

If node is online first, then all allocations come from that node, thought you
*want* to ensure node/mem is online before cpu is up to get that benefit.

> that node should be onlined before register_cpu().
> When a node is onlined, its pgdat should be there.

Re: [RFC][PATCH] node hotplug : register_cpu() changes [0/3]

[KAMEZAWA Hiroyuki]

On Tue, 23 May 2006 07:52:03 -0700
Ashok Raj <ashok.raj@intel.com> wrote:

> On Tue, May 23, 2006 at 07:56:36PM +0900, KAMEZAWA Hiroyuki wrote:
> > I found acpi container, which describes node, could evaluate cpu before
> > memory. This means cpu-hot-add occurs before memory hot add.
> > 
> 
> Is it possible to process memory before cpu in container hot-add code?
> 

Maybe No. I know ACPI people doesn't want to add special handling for cpu/memory
in a container. It complicates the code very much.

> > In most part, cpu-hot-add doesn't depend on node hot add.
> > But register_cpu, which creates symbolic link from node to cpu, requires
> 
> Dont you need all per-cpu allocated on that node? Or is it from node0 or 
> something for all hotpluggable cpus?
> 
I want to allocate on-node. But it's impossible now because per-cpu-pages
has to be allocated at boot-time as for possible cpus. They has to be
off-node now. They are from node0 (ia64) now.

I want to migrate per-cpu when a cpu is enabled (if I can). But maybe there
is a code which has pointer to object in per-cpu area.


> If node is online first, then all allocations come from that node, thought you
> *want* to ensure node/mem is online before cpu is up to get that benefit.
> 
Yes. But per-cpu should be allocated at boot-time now. I'd like to go step-by-step.

-Kame

Re: [RFC][PATCH] node hotplug : register_cpu() changes [0/3]

[KAMEZAWA Hiroyuki]

On Wed, 24 May 2006 09:18:15 +0900
KAMEZAWA Hiroyuki <kamezawa.hiroyu@jp.fujitsu.com> wrote:

> I want to migrate per-cpu when a cpu is enabled (if I can). But maybe there
> is a code which has pointer to object in per-cpu area.
> 
I read robust-VM-per_cpu variables thread. If it is merged, I think we can
migrate pages for per_cpu variables (of possible cpus) to suitable nodes,
by remapping.

Thanks,
-Kame

Re: [RFC][PATCH] node hotplug : register_cpu() changes [0/3]

[Ashok Raj]

On Wed, May 24, 2006 at 09:18:15AM +0900, KAMEZAWA Hiroyuki wrote:
> On Tue, 23 May 2006 07:52:03 -0700
> Ashok Raj <ashok.raj@intel.com> wrote:
> 
> > On Tue, May 23, 2006 at 07:56:36PM +0900, KAMEZAWA Hiroyuki wrote:
> > > I found acpi container, which describes node, could evaluate cpu before
> > > memory. This means cpu-hot-add occurs before memory hot add.
> > > 
> > 
> > Is it possible to process memory before cpu in container hot-add code?
> > 
> 
> Maybe No. I know ACPI people doesn't want to add special handling for cpu/memory
> in a container. It complicates the code very much.

Iam not attached to the API change, so disassociating the node from cpu is ok.
It just feels a bit weird to say node is not online. 

Probably ACPI doesnt give a precise way to put the dependencies (in a certain
order), there are EDL/EJD, but they have limitations and not directly imply
a dependency like this. But container code having to deal with this dependency
for add/remove is probably ok. 

i hope container would eventually perform onlining the pieces from user space
scripts via udev like mechanisms.

I remember folks from SGI posting patches to cpu only nodes in the past. 
Same way there are probably IO only nodes. Are you sure we cover these cases 
as well.
> 
> > > In most part, cpu-hot-add doesn't depend on node hot add.
> > > But register_cpu, which creates symbolic link from node to cpu, requires
> > 
> > Dont you need all per-cpu allocated on that node? Or is it from node0 or 
> > something for all hotpluggable cpus?
> > 

-- 
Cheers,
Ashok Raj
- Open Source Technology Center

Re: [RFC][PATCH] node hotplug : register_cpu() changes [0/3]

[KAMEZAWA Hiroyuki]

On Wed, 24 May 2006 07:51:28 -0700
Ashok Raj <ashok.raj@intel.com> wrote:

> Probably ACPI doesnt give a precise way to put the dependencies (in a certain
> order), there are EDL/EJD, but they have limitations and not directly imply
> a dependency like this. But container code having to deal with this dependency
> for add/remove is probably ok. 
> 
I know fujitsu's engineers posted codes to add devices under container in past
and it knew dependency of memory -> cpu. It was rejected.
So, the current container driver just uses acpi_bus_add/start() to add/start all
devices under the container.

> i hope container would eventually perform onlining the pieces from user space
> scripts via udev like mechanisms.
> 
> I remember folks from SGI posting patches to cpu only nodes in the past. 
> Same way there are probably IO only nodes. Are you sure we cover these cases 
> as well.

Currently, I think  creating new empty node (pgdat) at onlining can work.
by cpu hotplug notifier chain, dangling cpus will create new pgdat.
If people want to see node <-> cpu relationship before onlining cpu, I' ll have
to do complicated work.

I also expects we can migrate per-cpu in notifier chain if  VM-per_cpu is 
implemented.

Thanks, 
-Kame

Re: [RFC][PATCH] node hotplug : register_cpu() changes [0/3]

[Ashok Raj]

On Thu, May 25, 2006 at 09:34:18AM +0900, KAMEZAWA Hiroyuki wrote:
> 
> Currently, I think  creating new empty node (pgdat) at onlining can work.
> by cpu hotplug notifier chain, dangling cpus will create new pgdat.
> If people want to see node <-> cpu relationship before onlining cpu, I' ll have
> to do complicated work.
> 


Iam ok with this patch, no need to get anything complicated without good 
reason.

Just want to make sure assumptions for cpu only nodes are not broken, as long
as we have your implementation take care of those, this is good.

-- 
Cheers,
Ashok Raj
- Open Source Technology Center