[BUGFIX][PATCH] cpu to node relationship fixup take2 [1/2]

[BUGFIX][PATCH] cpu to node relationship fixup take2 [1/2]

[KAMEZAWA Hiroyuki]

I rewrote the whoe patch..

Problem description:
We have additional_cpus= option for allocating possible_cpus. But nid for
possible cpus are not fixed at boot time. cpus which is offlined at boot 
or cpus which is not on SRAT is not tied to its node.
This will cause panic at cpu onlining.

Changelog V1 -> V2
- divded patch into 2 pathces.
- move cpu_to_node relationship fixup before cpu onlining.

Tested on ia64/NUMA system, which has *physical* node-hot-add function.

Future work:  node-hot-add by cpu-hot-add.

-Kame

=
In usual, pxm_to_nid() mapping is fixed at boot time by SRAT.

But, unfortunatelly, some system (my system!) do not include
full SRAT table for possible cpus. (Then, I use additiona_cpus= option.)

For such possible cpus, pxm<->nid should be fixed at hot-add.
We now have acpi_map_pxm_to_node() which is also used at boot. It's suitable
here.

Signed-Off-By: KAMEZAWA Hiroyuki <kamezawa.hiroyu@jp.fujitsu.com>

 arch/ia64/kernel/acpi.c |   13 ++++++++-----
 1 file changed, 8 insertions(+), 5 deletions(-)

Index: linux-2.6.18/arch/ia64/kernel/acpi.c
=================================--- linux-2.6.18.orig/arch/ia64/kernel/acpi.c	2006-09-22 13:39:12.000000000 +0900
+++ linux-2.6.18/arch/ia64/kernel/acpi.c	2006-09-22 14:24:48.000000000 +0900
@@ -771,16 +771,19 @@
 {
 #ifdef CONFIG_ACPI_NUMA
 	int pxm_id;
+	int nid;
 
 	pxm_id = acpi_get_pxm(handle);
-
 	/*
-	 * Assuming that the container driver would have set the proximity
-	 * domain and would have initialized pxm_to_node(pxm_id) && pxm_flag
+	 * We don't have cpu-only-node hotadd. But if the system equips
+	 * SRAT table, pxm is already found and node is ready.
+  	 * So, just pxm_to_nid(pxm) is OK.
+	 * This code here is for the system which doesn't have full SRAT
+  	 * table for possible cpus.
 	 */
-	node_cpuid[cpu].nid = (pxm_id < 0) ? 0 : pxm_to_node(pxm_id);
-
+	nid = acpi_map_pxm_to_node(pxm_id);
 	node_cpuid[cpu].phys_id = physid;
+	node_cpuid[cpu].nid = nid;
 #endif
 	return (0);
 }

[BUGFIX][PATCH] cpu to node relationship fixup take2 [1/2] acpi_map_cpu2node

[KAMEZAWA Hiroyuki]

I rewrote the whoe patch..

Problem description:
We have additional_cpus= option for allocating possible_cpus. But nid for
possible cpus are not fixed at boot time. cpus which is offlined at boot 
or cpus which is not on SRAT is not tied to its node.
This will cause panic at cpu onlining.

Changelog V1 -> V2
- divded patch into 2 pathces.
- move cpu_to_node relationship fixup before cpu onlining.

Tested on ia64/NUMA system, which has *physical* node-hot-add function.

Future work:  node-hot-add by cpu-hot-add.

-Kame

==
In usual, pxm_to_nid() mapping is fixed at boot time by SRAT.

But, unfortunatelly, some system (my system!) do not include
full SRAT table for possible cpus. (Then, I use additiona_cpus= option.)

For such possible cpus, pxm<->nid should be fixed at hot-add.
We now have acpi_map_pxm_to_node() which is also used at boot. It's suitable
here.

Signed-Off-By: KAMEZAWA Hiroyuki <kamezawa.hiroyu@jp.fujitsu.com>

 arch/ia64/kernel/acpi.c |   13 ++++++++-----
 1 file changed, 8 insertions(+), 5 deletions(-)

Index: linux-2.6.18/arch/ia64/kernel/acpi.c
===================================================================
--- linux-2.6.18.orig/arch/ia64/kernel/acpi.c	2006-09-22 13:39:12.000000000 +0900
+++ linux-2.6.18/arch/ia64/kernel/acpi.c	2006-09-22 14:24:48.000000000 +0900
@@ -771,16 +771,19 @@
 {
 #ifdef CONFIG_ACPI_NUMA
 	int pxm_id;
+	int nid;
 
 	pxm_id = acpi_get_pxm(handle);
-
 	/*
-	 * Assuming that the container driver would have set the proximity
-	 * domain and would have initialized pxm_to_node(pxm_id) && pxm_flag
+	 * We don't have cpu-only-node hotadd. But if the system equips
+	 * SRAT table, pxm is already found and node is ready.
+  	 * So, just pxm_to_nid(pxm) is OK.
+	 * This code here is for the system which doesn't have full SRAT
+  	 * table for possible cpus.
 	 */
-	node_cpuid[cpu].nid = (pxm_id < 0) ? 0 : pxm_to_node(pxm_id);
-
+	nid = acpi_map_pxm_to_node(pxm_id);
 	node_cpuid[cpu].phys_id = physid;
+	node_cpuid[cpu].nid = nid;
 #endif
 	return (0);
 }

[BUGFIX][PATCH] cpu to node relationship fixup take2 [2/2] map cpu

[KAMEZAWA Hiroyuki]

Assume that a cpu is *physically* offlined at boot time...

Because smpboot.c::smp_boot_cpu_map() canoot find cpu's sapicid,
numa.c::build_cpu_to_node_map() cannot build cpu<->node map for
offlined cpu.

For such cpus, cpu_to_node map should be fixed at cpu-hot-add.
This mapping should be done before cpu onlining.

This patch also handles cpu hotremove case.

Signed-Off-By: KAMEZAWA Hiroyuki <kamezawa.hiroyu@jp.fujitsu.com>


 arch/ia64/kernel/numa.c     |   34 +++++++++++++++++++++++++++++++---
 arch/ia64/kernel/topology.c |    6 +++++-
 include/asm-ia64/numa.h     |    4 ++++
 3 files changed, 40 insertions(+), 4 deletions(-)

Index: linux-2.6.18/arch/ia64/kernel/numa.c
=================================--- linux-2.6.18.orig/arch/ia64/kernel/numa.c	2006-09-22 14:22:44.000000000 +0900
+++ linux-2.6.18/arch/ia64/kernel/numa.c	2006-09-22 14:44:46.000000000 +0900
@@ -29,6 +29,36 @@
 
 cpumask_t node_to_cpu_mask[MAX_NUMNODES] __cacheline_aligned;
 
+void __cpuinit map_cpu_to_node(int cpu, int nid)
+{
+	int oldnid;
+	if (nid < 0) { /* just initialize by zero */
+		cpu_to_node_map[cpu] = 0;
+		return;
+	}
+	/* sanity check first */
+	oldnid = cpu_to_node_map[cpu];
+	if (cpu_isset(cpu, node_to_cpu_mask[oldnid])) {
+		return; /* nothing to do */
+	}
+	/* we don't have cpu-driven node hot add yet...
+	   In usual case, node is created from SRAT at boot time. */
+	if (!node_online(nid))
+		nid = first_online_node;
+	cpu_to_node_map[cpu] = nid;
+	cpu_set(cpu, node_to_cpu_mask[nid]);
+	return;
+}
+
+void __cpuinit unmap_cpu_from_node(int cpu, int nid)
+{
+	WARN_ON(!cpu_isset(cpu, node_to_cpu_mask[nid]));
+	WARN_ON(cpu_to_node_map[cpu] != nid);
+	cpu_to_node_map[cpu] = 0;
+	cpu_clear(cpu, node_to_cpu_mask[nid]);
+}
+
+
 /**
  * build_cpu_to_node_map - setup cpu to node and node to cpumask arrays
  *
@@ -49,8 +79,6 @@
 				node = node_cpuid[i].nid;
 				break;
 			}
-		cpu_to_node_map[cpu] = (node >= 0) ? node : 0;
-		if (node >= 0)
-			cpu_set(cpu, node_to_cpu_mask[node]);
+		map_cpu_to_node(cpu, node);
 	}
 }
Index: linux-2.6.18/include/asm-ia64/numa.h
=================================--- linux-2.6.18.orig/include/asm-ia64/numa.h	2006-09-22 14:22:44.000000000 +0900
+++ linux-2.6.18/include/asm-ia64/numa.h	2006-09-22 14:25:07.000000000 +0900
@@ -64,6 +64,10 @@
 
 #define local_nodeid (cpu_to_node_map[smp_processor_id()])
 
+extern void map_cpu_to_node(int cpu, int nid);
+extern void unmap_cpu_from_node(int cpu, int nid);
+
+
 #else /* !CONFIG_NUMA */
 
 #define paddr_to_nid(addr)	0
Index: linux-2.6.18/arch/ia64/kernel/topology.c
=================================--- linux-2.6.18.orig/arch/ia64/kernel/topology.c	2006-09-22 14:22:44.000000000 +0900
+++ linux-2.6.18/arch/ia64/kernel/topology.c	2006-09-22 14:25:07.000000000 +0900
@@ -36,6 +36,9 @@
 	 */
 	if (!can_cpei_retarget() && is_cpu_cpei_target(num))
 		sysfs_cpus[num].cpu.no_control = 1;
+#ifdef CONFIG_NUMA
+	map_cpu_to_node(num, node_cpuid[num].nid);
+#endif
 #endif
 
 	return register_cpu(&sysfs_cpus[num].cpu, num);
@@ -45,7 +48,8 @@
 
 void arch_unregister_cpu(int num)
 {
-	return unregister_cpu(&sysfs_cpus[num].cpu);
+	unregister_cpu(&sysfs_cpus[num].cpu);
+	unmap_cpu_from_node(num, cpu_to_node(num));
 }
 EXPORT_SYMBOL(arch_register_cpu);
 EXPORT_SYMBOL(arch_unregister_cpu);

[BUGFIX][PATCH] cpu to node relationship fixup take2 [2/2] map cpu to node

[KAMEZAWA Hiroyuki]

Assume that a cpu is *physically* offlined at boot time...

Because smpboot.c::smp_boot_cpu_map() canoot find cpu's sapicid,
numa.c::build_cpu_to_node_map() cannot build cpu<->node map for
offlined cpu.

For such cpus, cpu_to_node map should be fixed at cpu-hot-add.
This mapping should be done before cpu onlining.

This patch also handles cpu hotremove case.

Signed-Off-By: KAMEZAWA Hiroyuki <kamezawa.hiroyu@jp.fujitsu.com>


 arch/ia64/kernel/numa.c     |   34 +++++++++++++++++++++++++++++++---
 arch/ia64/kernel/topology.c |    6 +++++-
 include/asm-ia64/numa.h     |    4 ++++
 3 files changed, 40 insertions(+), 4 deletions(-)

Index: linux-2.6.18/arch/ia64/kernel/numa.c
===================================================================
--- linux-2.6.18.orig/arch/ia64/kernel/numa.c	2006-09-22 14:22:44.000000000 +0900
+++ linux-2.6.18/arch/ia64/kernel/numa.c	2006-09-22 14:44:46.000000000 +0900
@@ -29,6 +29,36 @@
 
 cpumask_t node_to_cpu_mask[MAX_NUMNODES] __cacheline_aligned;
 
+void __cpuinit map_cpu_to_node(int cpu, int nid)
+{
+	int oldnid;
+	if (nid < 0) { /* just initialize by zero */
+		cpu_to_node_map[cpu] = 0;
+		return;
+	}
+	/* sanity check first */
+	oldnid = cpu_to_node_map[cpu];
+	if (cpu_isset(cpu, node_to_cpu_mask[oldnid])) {
+		return; /* nothing to do */
+	}
+	/* we don't have cpu-driven node hot add yet...
+	   In usual case, node is created from SRAT at boot time. */
+	if (!node_online(nid))
+		nid = first_online_node;
+	cpu_to_node_map[cpu] = nid;
+	cpu_set(cpu, node_to_cpu_mask[nid]);
+	return;
+}
+
+void __cpuinit unmap_cpu_from_node(int cpu, int nid)
+{
+	WARN_ON(!cpu_isset(cpu, node_to_cpu_mask[nid]));
+	WARN_ON(cpu_to_node_map[cpu] != nid);
+	cpu_to_node_map[cpu] = 0;
+	cpu_clear(cpu, node_to_cpu_mask[nid]);
+}
+
+
 /**
  * build_cpu_to_node_map - setup cpu to node and node to cpumask arrays
  *
@@ -49,8 +79,6 @@
 				node = node_cpuid[i].nid;
 				break;
 			}
-		cpu_to_node_map[cpu] = (node >= 0) ? node : 0;
-		if (node >= 0)
-			cpu_set(cpu, node_to_cpu_mask[node]);
+		map_cpu_to_node(cpu, node);
 	}
 }
Index: linux-2.6.18/include/asm-ia64/numa.h
===================================================================
--- linux-2.6.18.orig/include/asm-ia64/numa.h	2006-09-22 14:22:44.000000000 +0900
+++ linux-2.6.18/include/asm-ia64/numa.h	2006-09-22 14:25:07.000000000 +0900
@@ -64,6 +64,10 @@
 
 #define local_nodeid (cpu_to_node_map[smp_processor_id()])
 
+extern void map_cpu_to_node(int cpu, int nid);
+extern void unmap_cpu_from_node(int cpu, int nid);
+
+
 #else /* !CONFIG_NUMA */
 
 #define paddr_to_nid(addr)	0
Index: linux-2.6.18/arch/ia64/kernel/topology.c
===================================================================
--- linux-2.6.18.orig/arch/ia64/kernel/topology.c	2006-09-22 14:22:44.000000000 +0900
+++ linux-2.6.18/arch/ia64/kernel/topology.c	2006-09-22 14:25:07.000000000 +0900
@@ -36,6 +36,9 @@
 	 */
 	if (!can_cpei_retarget() && is_cpu_cpei_target(num))
 		sysfs_cpus[num].cpu.no_control = 1;
+#ifdef CONFIG_NUMA
+	map_cpu_to_node(num, node_cpuid[num].nid);
+#endif
 #endif
 
 	return register_cpu(&sysfs_cpus[num].cpu, num);
@@ -45,7 +48,8 @@
 
 void arch_unregister_cpu(int num)
 {
-	return unregister_cpu(&sysfs_cpus[num].cpu);
+	unregister_cpu(&sysfs_cpus[num].cpu);
+	unmap_cpu_from_node(num, cpu_to_node(num));
 }
 EXPORT_SYMBOL(arch_register_cpu);
 EXPORT_SYMBOL(arch_unregister_cpu);

Re: [BUGFIX][PATCH] cpu to node relationship fixup take2 [2/2] map

[bibo,mao]

KAMEZAWA Hiroyuki wrote:
> Assume that a cpu is *physically* offlined at boot time...
> 
> Because smpboot.c::smp_boot_cpu_map() canoot find cpu's sapicid,
> numa.c::build_cpu_to_node_map() cannot build cpu<->node map for
> offlined cpu.
> 
> For such cpus, cpu_to_node map should be fixed at cpu-hot-add.
> This mapping should be done before cpu onlining.
> 
> This patch also handles cpu hotremove case.
> 
> Signed-Off-By: KAMEZAWA Hiroyuki <kamezawa.hiroyu@jp.fujitsu.com>
> 
> 
>  arch/ia64/kernel/numa.c     |   34 +++++++++++++++++++++++++++++++---
>  arch/ia64/kernel/topology.c |    6 +++++-
>  include/asm-ia64/numa.h     |    4 ++++
>  3 files changed, 40 insertions(+), 4 deletions(-)
> 
> Index: linux-2.6.18/arch/ia64/kernel/numa.c
> =================================> --- linux-2.6.18.orig/arch/ia64/kernel/numa.c	2006-09-22 14:22:44.000000000 +0900
> +++ linux-2.6.18/arch/ia64/kernel/numa.c	2006-09-22 14:44:46.000000000 +0900
> @@ -29,6 +29,36 @@
>  
>  cpumask_t node_to_cpu_mask[MAX_NUMNODES] __cacheline_aligned;
>  
> +void __cpuinit map_cpu_to_node(int cpu, int nid)
> +{
> +	int oldnid;
> +	if (nid < 0) { /* just initialize by zero */
> +		cpu_to_node_map[cpu] = 0;
> +		return;
> +	}
> +	/* sanity check first */
> +	oldnid = cpu_to_node_map[cpu];
> +	if (cpu_isset(cpu, node_to_cpu_mask[oldnid])) {
> +		return; /* nothing to do */
> +	}
> +	/* we don't have cpu-driven node hot add yet...
> +	   In usual case, node is created from SRAT at boot time. */
> +	if (!node_online(nid))
> +		nid = first_online_node;
> +	cpu_to_node_map[cpu] = nid;
> +	cpu_set(cpu, node_to_cpu_mask[nid]);
> +	return;
> +}
> +
> +void __cpuinit unmap_cpu_from_node(int cpu, int nid)
> +{
> +	WARN_ON(!cpu_isset(cpu, node_to_cpu_mask[nid]));
> +	WARN_ON(cpu_to_node_map[cpu] != nid);
> +	cpu_to_node_map[cpu] = 0;
> +	cpu_clear(cpu, node_to_cpu_mask[nid]);
> +}
> +
> +
>  /**
>   * build_cpu_to_node_map - setup cpu to node and node to cpumask arrays
>   *
> @@ -49,8 +79,6 @@
>  				node = node_cpuid[i].nid;
>  				break;
>  			}
> -		cpu_to_node_map[cpu] = (node >= 0) ? node : 0;
> -		if (node >= 0)
> -			cpu_set(cpu, node_to_cpu_mask[node]);
> +		map_cpu_to_node(cpu, node);
>  	}
>  }
> Index: linux-2.6.18/include/asm-ia64/numa.h
> =================================> --- linux-2.6.18.orig/include/asm-ia64/numa.h	2006-09-22 14:22:44.000000000 +0900
> +++ linux-2.6.18/include/asm-ia64/numa.h	2006-09-22 14:25:07.000000000 +0900
> @@ -64,6 +64,10 @@
>  
>  #define local_nodeid (cpu_to_node_map[smp_processor_id()])
>  
> +extern void map_cpu_to_node(int cpu, int nid);
> +extern void unmap_cpu_from_node(int cpu, int nid);
> +
> +
>  #else /* !CONFIG_NUMA */
>  
>  #define paddr_to_nid(addr)	0
> Index: linux-2.6.18/arch/ia64/kernel/topology.c
> =================================> --- linux-2.6.18.orig/arch/ia64/kernel/topology.c	2006-09-22 14:22:44.000000000 +0900
> +++ linux-2.6.18/arch/ia64/kernel/topology.c	2006-09-22 14:25:07.000000000 +0900
> @@ -36,6 +36,9 @@
>  	 */
>  	if (!can_cpei_retarget() && is_cpu_cpei_target(num))
>  		sysfs_cpus[num].cpu.no_control = 1;
> +#ifdef CONFIG_NUMA
> +	map_cpu_to_node(num, node_cpuid[num].nid);
> +#endif
>  #endif
>  
>  	return register_cpu(&sysfs_cpus[num].cpu, num);
> @@ -45,7 +48,8 @@
>  
>  void arch_unregister_cpu(int num)
>  {
> -	return unregister_cpu(&sysfs_cpus[num].cpu);
> +	unregister_cpu(&sysfs_cpus[num].cpu);
> +	unmap_cpu_from_node(num, cpu_to_node(num));
This patch failed to compile in my IA64 box, my machine is not NUMA machine,
there is unmap_cpu_from_node not defined error like this.

arch/ia64/kernel/built-in.o: In function `arch_unregister_cpu':
arch/ia64/kernel/topology.c:52: undefined reference to `unmap_cpu_from_node'
make: *** [.tmp_vmlinux1] Error 1

thanks
bibo,mao

>  }
>  EXPORT_SYMBOL(arch_register_cpu);
>  EXPORT_SYMBOL(arch_unregister_cpu);
> 
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
> 

Re: [BUGFIX][PATCH] cpu to node relationship fixup take2 [2/2] map cpu to node

[bibo,mao]

KAMEZAWA Hiroyuki wrote:
> Assume that a cpu is *physically* offlined at boot time...
> 
> Because smpboot.c::smp_boot_cpu_map() canoot find cpu's sapicid,
> numa.c::build_cpu_to_node_map() cannot build cpu<->node map for
> offlined cpu.
> 
> For such cpus, cpu_to_node map should be fixed at cpu-hot-add.
> This mapping should be done before cpu onlining.
> 
> This patch also handles cpu hotremove case.
> 
> Signed-Off-By: KAMEZAWA Hiroyuki <kamezawa.hiroyu@jp.fujitsu.com>
> 
> 
>  arch/ia64/kernel/numa.c     |   34 +++++++++++++++++++++++++++++++---
>  arch/ia64/kernel/topology.c |    6 +++++-
>  include/asm-ia64/numa.h     |    4 ++++
>  3 files changed, 40 insertions(+), 4 deletions(-)
> 
> Index: linux-2.6.18/arch/ia64/kernel/numa.c
> ===================================================================
> --- linux-2.6.18.orig/arch/ia64/kernel/numa.c	2006-09-22 14:22:44.000000000 +0900
> +++ linux-2.6.18/arch/ia64/kernel/numa.c	2006-09-22 14:44:46.000000000 +0900
> @@ -29,6 +29,36 @@
>  
>  cpumask_t node_to_cpu_mask[MAX_NUMNODES] __cacheline_aligned;
>  
> +void __cpuinit map_cpu_to_node(int cpu, int nid)
> +{
> +	int oldnid;
> +	if (nid < 0) { /* just initialize by zero */
> +		cpu_to_node_map[cpu] = 0;
> +		return;
> +	}
> +	/* sanity check first */
> +	oldnid = cpu_to_node_map[cpu];
> +	if (cpu_isset(cpu, node_to_cpu_mask[oldnid])) {
> +		return; /* nothing to do */
> +	}
> +	/* we don't have cpu-driven node hot add yet...
> +	   In usual case, node is created from SRAT at boot time. */
> +	if (!node_online(nid))
> +		nid = first_online_node;
> +	cpu_to_node_map[cpu] = nid;
> +	cpu_set(cpu, node_to_cpu_mask[nid]);
> +	return;
> +}
> +
> +void __cpuinit unmap_cpu_from_node(int cpu, int nid)
> +{
> +	WARN_ON(!cpu_isset(cpu, node_to_cpu_mask[nid]));
> +	WARN_ON(cpu_to_node_map[cpu] != nid);
> +	cpu_to_node_map[cpu] = 0;
> +	cpu_clear(cpu, node_to_cpu_mask[nid]);
> +}
> +
> +
>  /**
>   * build_cpu_to_node_map - setup cpu to node and node to cpumask arrays
>   *
> @@ -49,8 +79,6 @@
>  				node = node_cpuid[i].nid;
>  				break;
>  			}
> -		cpu_to_node_map[cpu] = (node >= 0) ? node : 0;
> -		if (node >= 0)
> -			cpu_set(cpu, node_to_cpu_mask[node]);
> +		map_cpu_to_node(cpu, node);
>  	}
>  }
> Index: linux-2.6.18/include/asm-ia64/numa.h
> ===================================================================
> --- linux-2.6.18.orig/include/asm-ia64/numa.h	2006-09-22 14:22:44.000000000 +0900
> +++ linux-2.6.18/include/asm-ia64/numa.h	2006-09-22 14:25:07.000000000 +0900
> @@ -64,6 +64,10 @@
>  
>  #define local_nodeid (cpu_to_node_map[smp_processor_id()])
>  
> +extern void map_cpu_to_node(int cpu, int nid);
> +extern void unmap_cpu_from_node(int cpu, int nid);
> +
> +
>  #else /* !CONFIG_NUMA */
>  
>  #define paddr_to_nid(addr)	0
> Index: linux-2.6.18/arch/ia64/kernel/topology.c
> ===================================================================
> --- linux-2.6.18.orig/arch/ia64/kernel/topology.c	2006-09-22 14:22:44.000000000 +0900
> +++ linux-2.6.18/arch/ia64/kernel/topology.c	2006-09-22 14:25:07.000000000 +0900
> @@ -36,6 +36,9 @@
>  	 */
>  	if (!can_cpei_retarget() && is_cpu_cpei_target(num))
>  		sysfs_cpus[num].cpu.no_control = 1;
> +#ifdef CONFIG_NUMA
> +	map_cpu_to_node(num, node_cpuid[num].nid);
> +#endif
>  #endif
>  
>  	return register_cpu(&sysfs_cpus[num].cpu, num);
> @@ -45,7 +48,8 @@
>  
>  void arch_unregister_cpu(int num)
>  {
> -	return unregister_cpu(&sysfs_cpus[num].cpu);
> +	unregister_cpu(&sysfs_cpus[num].cpu);
> +	unmap_cpu_from_node(num, cpu_to_node(num));
This patch failed to compile in my IA64 box, my machine is not NUMA machine,
there is unmap_cpu_from_node not defined error like this.

arch/ia64/kernel/built-in.o: In function `arch_unregister_cpu':
arch/ia64/kernel/topology.c:52: undefined reference to `unmap_cpu_from_node'
make: *** [.tmp_vmlinux1] Error 1

thanks
bibo,mao

>  }
>  EXPORT_SYMBOL(arch_register_cpu);
>  EXPORT_SYMBOL(arch_unregister_cpu);
> 
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
> 

Re: [BUGFIX][PATCH] cpu to node relationship fixup take2 [2/2] map

[KAMEZAWA Hiroyuki]

On Tue, 26 Sep 2006 19:23:05 +0800
"bibo,mao" <bibo.mao@intel.com> wrote:

> KAMEZAWA Hiroyuki wrote:
> > Assume that a cpu is *physically* offlined at boot time...
> > 
> > Because smpboot.c::smp_boot_cpu_map() canoot find cpu's sapicid,
> > numa.c::build_cpu_to_node_map() cannot build cpu<->node map for
> > offlined cpu.
> > 
> > For such cpus, cpu_to_node map should be fixed at cpu-hot-add.
> > This mapping should be done before cpu onlining.
> > 
> > This patch also handles cpu hotremove case.
> > 
> > Signed-Off-By: KAMEZAWA Hiroyuki <kamezawa.hiroyu@jp.fujitsu.com>
> > 
> > 
> >  arch/ia64/kernel/numa.c     |   34 +++++++++++++++++++++++++++++++---
> >  arch/ia64/kernel/topology.c |    6 +++++-
> >  include/asm-ia64/numa.h     |    4 ++++
> >  3 files changed, 40 insertions(+), 4 deletions(-)
> > 
> > Index: linux-2.6.18/arch/ia64/kernel/numa.c
> > =================================> > --- linux-2.6.18.orig/arch/ia64/kernel/numa.c	2006-09-22 14:22:44.000000000 +0900
> > +++ linux-2.6.18/arch/ia64/kernel/numa.c	2006-09-22 14:44:46.000000000 +0900
> > @@ -29,6 +29,36 @@
> >  
> >  cpumask_t node_to_cpu_mask[MAX_NUMNODES] __cacheline_aligned;
> >  
> > +void __cpuinit map_cpu_to_node(int cpu, int nid)
> > +{
> > +	int oldnid;
> > +	if (nid < 0) { /* just initialize by zero */
> > +		cpu_to_node_map[cpu] = 0;
> > +		return;
> > +	}
> > +	/* sanity check first */
> > +	oldnid = cpu_to_node_map[cpu];
> > +	if (cpu_isset(cpu, node_to_cpu_mask[oldnid])) {
> > +		return; /* nothing to do */
> > +	}
> > +	/* we don't have cpu-driven node hot add yet...
> > +	   In usual case, node is created from SRAT at boot time. */
> > +	if (!node_online(nid))
> > +		nid = first_online_node;
> > +	cpu_to_node_map[cpu] = nid;
> > +	cpu_set(cpu, node_to_cpu_mask[nid]);
> > +	return;
> > +}
> > +
> > +void __cpuinit unmap_cpu_from_node(int cpu, int nid)
> > +{
> > +	WARN_ON(!cpu_isset(cpu, node_to_cpu_mask[nid]));
> > +	WARN_ON(cpu_to_node_map[cpu] != nid);
> > +	cpu_to_node_map[cpu] = 0;
> > +	cpu_clear(cpu, node_to_cpu_mask[nid]);
> > +}
> > +
> > +
> >  /**
> >   * build_cpu_to_node_map - setup cpu to node and node to cpumask arrays
> >   *
> > @@ -49,8 +79,6 @@
> >  				node = node_cpuid[i].nid;
> >  				break;
> >  			}
> > -		cpu_to_node_map[cpu] = (node >= 0) ? node : 0;
> > -		if (node >= 0)
> > -			cpu_set(cpu, node_to_cpu_mask[node]);
> > +		map_cpu_to_node(cpu, node);
> >  	}
> >  }
> > Index: linux-2.6.18/include/asm-ia64/numa.h
> > =================================> > --- linux-2.6.18.orig/include/asm-ia64/numa.h	2006-09-22 14:22:44.000000000 +0900
> > +++ linux-2.6.18/include/asm-ia64/numa.h	2006-09-22 14:25:07.000000000 +0900
> > @@ -64,6 +64,10 @@
> >  
> >  #define local_nodeid (cpu_to_node_map[smp_processor_id()])
> >  
> > +extern void map_cpu_to_node(int cpu, int nid);
> > +extern void unmap_cpu_from_node(int cpu, int nid);
> > +
> > +
> >  #else /* !CONFIG_NUMA */
> >  
> >  #define paddr_to_nid(addr)	0
> > Index: linux-2.6.18/arch/ia64/kernel/topology.c
> > =================================> > --- linux-2.6.18.orig/arch/ia64/kernel/topology.c	2006-09-22 14:22:44.000000000 +0900
> > +++ linux-2.6.18/arch/ia64/kernel/topology.c	2006-09-22 14:25:07.000000000 +0900
> > @@ -36,6 +36,9 @@
> >  	 */
> >  	if (!can_cpei_retarget() && is_cpu_cpei_target(num))
> >  		sysfs_cpus[num].cpu.no_control = 1;
> > +#ifdef CONFIG_NUMA
> > +	map_cpu_to_node(num, node_cpuid[num].nid);
> > +#endif
> >  #endif
> >  
> >  	return register_cpu(&sysfs_cpus[num].cpu, num);
> > @@ -45,7 +48,8 @@
> >  
> >  void arch_unregister_cpu(int num)
> >  {
> > -	return unregister_cpu(&sysfs_cpus[num].cpu);
> > +	unregister_cpu(&sysfs_cpus[num].cpu);
> > +	unmap_cpu_from_node(num, cpu_to_node(num));
> This patch failed to compile in my IA64 box, my machine is not NUMA machine,
> there is unmap_cpu_from_node not defined error like this.
> 
> arch/ia64/kernel/built-in.o: In function `arch_unregister_cpu':
> arch/ia64/kernel/topology.c:52: undefined reference to `unmap_cpu_from_node'
> make: *** [.tmp_vmlinux1] Error 1
> 
> thanks
> bibo,mao
> 

Uh... sorry. How about this hot-fix ?

-Kame
====

Non-NUMA case should be handled...

Signed-Off-By: KAMEZAWA Hiroyuki <kamezawa.hiroyu@jp.fujitsu.com>

Index: linux-2.6.18/arch/ia64/kernel/topology.c
=================================--- linux-2.6.18.orig/arch/ia64/kernel/topology.c	2006-09-26 20:31:25.000000000 +0900
+++ linux-2.6.18/arch/ia64/kernel/topology.c	2006-09-26 20:34:03.000000000 +0900
@@ -36,10 +36,8 @@
 	 */
 	if (!can_cpei_retarget() && is_cpu_cpei_target(num))
 		sysfs_cpus[num].cpu.no_control = 1;
-#ifdef CONFIG_NUMA
 	map_cpu_to_node(num, node_cpuid[num].nid);
 #endif
-#endif
 
 	return register_cpu(&sysfs_cpus[num].cpu, num);
 }
Index: linux-2.6.18/include/asm-ia64/numa.h
=================================--- linux-2.6.18.orig/include/asm-ia64/numa.h	2006-09-26 20:31:25.000000000 +0900
+++ linux-2.6.18/include/asm-ia64/numa.h	2006-09-26 20:33:49.000000000 +0900
@@ -69,6 +69,8 @@
 
 
 #else /* !CONFIG_NUMA */
+#define map_cpu_to_node(cpu, nid)	do{}while(0)
+#define unmap_cpu_from_node(cpu, nid)	do{}while(0)
 
 #define paddr_to_nid(addr)	0
 

Re: [BUGFIX][PATCH] cpu to node relationship fixup take2 [2/2] map cpu to node

[KAMEZAWA Hiroyuki]

On Tue, 26 Sep 2006 19:23:05 +0800
"bibo,mao" <bibo.mao@intel.com> wrote:

> KAMEZAWA Hiroyuki wrote:
> > Assume that a cpu is *physically* offlined at boot time...
> > 
> > Because smpboot.c::smp_boot_cpu_map() canoot find cpu's sapicid,
> > numa.c::build_cpu_to_node_map() cannot build cpu<->node map for
> > offlined cpu.
> > 
> > For such cpus, cpu_to_node map should be fixed at cpu-hot-add.
> > This mapping should be done before cpu onlining.
> > 
> > This patch also handles cpu hotremove case.
> > 
> > Signed-Off-By: KAMEZAWA Hiroyuki <kamezawa.hiroyu@jp.fujitsu.com>
> > 
> > 
> >  arch/ia64/kernel/numa.c     |   34 +++++++++++++++++++++++++++++++---
> >  arch/ia64/kernel/topology.c |    6 +++++-
> >  include/asm-ia64/numa.h     |    4 ++++
> >  3 files changed, 40 insertions(+), 4 deletions(-)
> > 
> > Index: linux-2.6.18/arch/ia64/kernel/numa.c
> > ===================================================================
> > --- linux-2.6.18.orig/arch/ia64/kernel/numa.c	2006-09-22 14:22:44.000000000 +0900
> > +++ linux-2.6.18/arch/ia64/kernel/numa.c	2006-09-22 14:44:46.000000000 +0900
> > @@ -29,6 +29,36 @@
> >  
> >  cpumask_t node_to_cpu_mask[MAX_NUMNODES] __cacheline_aligned;
> >  
> > +void __cpuinit map_cpu_to_node(int cpu, int nid)
> > +{
> > +	int oldnid;
> > +	if (nid < 0) { /* just initialize by zero */
> > +		cpu_to_node_map[cpu] = 0;
> > +		return;
> > +	}
> > +	/* sanity check first */
> > +	oldnid = cpu_to_node_map[cpu];
> > +	if (cpu_isset(cpu, node_to_cpu_mask[oldnid])) {
> > +		return; /* nothing to do */
> > +	}
> > +	/* we don't have cpu-driven node hot add yet...
> > +	   In usual case, node is created from SRAT at boot time. */
> > +	if (!node_online(nid))
> > +		nid = first_online_node;
> > +	cpu_to_node_map[cpu] = nid;
> > +	cpu_set(cpu, node_to_cpu_mask[nid]);
> > +	return;
> > +}
> > +
> > +void __cpuinit unmap_cpu_from_node(int cpu, int nid)
> > +{
> > +	WARN_ON(!cpu_isset(cpu, node_to_cpu_mask[nid]));
> > +	WARN_ON(cpu_to_node_map[cpu] != nid);
> > +	cpu_to_node_map[cpu] = 0;
> > +	cpu_clear(cpu, node_to_cpu_mask[nid]);
> > +}
> > +
> > +
> >  /**
> >   * build_cpu_to_node_map - setup cpu to node and node to cpumask arrays
> >   *
> > @@ -49,8 +79,6 @@
> >  				node = node_cpuid[i].nid;
> >  				break;
> >  			}
> > -		cpu_to_node_map[cpu] = (node >= 0) ? node : 0;
> > -		if (node >= 0)
> > -			cpu_set(cpu, node_to_cpu_mask[node]);
> > +		map_cpu_to_node(cpu, node);
> >  	}
> >  }
> > Index: linux-2.6.18/include/asm-ia64/numa.h
> > ===================================================================
> > --- linux-2.6.18.orig/include/asm-ia64/numa.h	2006-09-22 14:22:44.000000000 +0900
> > +++ linux-2.6.18/include/asm-ia64/numa.h	2006-09-22 14:25:07.000000000 +0900
> > @@ -64,6 +64,10 @@
> >  
> >  #define local_nodeid (cpu_to_node_map[smp_processor_id()])
> >  
> > +extern void map_cpu_to_node(int cpu, int nid);
> > +extern void unmap_cpu_from_node(int cpu, int nid);
> > +
> > +
> >  #else /* !CONFIG_NUMA */
> >  
> >  #define paddr_to_nid(addr)	0
> > Index: linux-2.6.18/arch/ia64/kernel/topology.c
> > ===================================================================
> > --- linux-2.6.18.orig/arch/ia64/kernel/topology.c	2006-09-22 14:22:44.000000000 +0900
> > +++ linux-2.6.18/arch/ia64/kernel/topology.c	2006-09-22 14:25:07.000000000 +0900
> > @@ -36,6 +36,9 @@
> >  	 */
> >  	if (!can_cpei_retarget() && is_cpu_cpei_target(num))
> >  		sysfs_cpus[num].cpu.no_control = 1;
> > +#ifdef CONFIG_NUMA
> > +	map_cpu_to_node(num, node_cpuid[num].nid);
> > +#endif
> >  #endif
> >  
> >  	return register_cpu(&sysfs_cpus[num].cpu, num);
> > @@ -45,7 +48,8 @@
> >  
> >  void arch_unregister_cpu(int num)
> >  {
> > -	return unregister_cpu(&sysfs_cpus[num].cpu);
> > +	unregister_cpu(&sysfs_cpus[num].cpu);
> > +	unmap_cpu_from_node(num, cpu_to_node(num));
> This patch failed to compile in my IA64 box, my machine is not NUMA machine,
> there is unmap_cpu_from_node not defined error like this.
> 
> arch/ia64/kernel/built-in.o: In function `arch_unregister_cpu':
> arch/ia64/kernel/topology.c:52: undefined reference to `unmap_cpu_from_node'
> make: *** [.tmp_vmlinux1] Error 1
> 
> thanks
> bibo,mao
> 

Uh... sorry. How about this hot-fix ?

-Kame
=========


Non-NUMA case should be handled...

Signed-Off-By: KAMEZAWA Hiroyuki <kamezawa.hiroyu@jp.fujitsu.com>

Index: linux-2.6.18/arch/ia64/kernel/topology.c
===================================================================
--- linux-2.6.18.orig/arch/ia64/kernel/topology.c	2006-09-26 20:31:25.000000000 +0900
+++ linux-2.6.18/arch/ia64/kernel/topology.c	2006-09-26 20:34:03.000000000 +0900
@@ -36,10 +36,8 @@
 	 */
 	if (!can_cpei_retarget() && is_cpu_cpei_target(num))
 		sysfs_cpus[num].cpu.no_control = 1;
-#ifdef CONFIG_NUMA
 	map_cpu_to_node(num, node_cpuid[num].nid);
 #endif
-#endif
 
 	return register_cpu(&sysfs_cpus[num].cpu, num);
 }
Index: linux-2.6.18/include/asm-ia64/numa.h
===================================================================
--- linux-2.6.18.orig/include/asm-ia64/numa.h	2006-09-26 20:31:25.000000000 +0900
+++ linux-2.6.18/include/asm-ia64/numa.h	2006-09-26 20:33:49.000000000 +0900
@@ -69,6 +69,8 @@
 
 
 #else /* !CONFIG_NUMA */
+#define map_cpu_to_node(cpu, nid)	do{}while(0)
+#define unmap_cpu_from_node(cpu, nid)	do{}while(0)
 
 #define paddr_to_nid(addr)	0
 

Re: [BUGFIX][PATCH] cpu to node relationship fixup take2 [2/2] map

[bibo,mao]

yes, the hot-fix works for me now :)

thanks
bibo,mao

KAMEZAWA Hiroyuki wrote:
> 
> Uh... sorry. How about this hot-fix ?
> 
> -Kame
> ====> 
> 
> Non-NUMA case should be handled...
> 
> Signed-Off-By: KAMEZAWA Hiroyuki <kamezawa.hiroyu@jp.fujitsu.com>
> 
> Index: linux-2.6.18/arch/ia64/kernel/topology.c
> =================================> --- linux-2.6.18.orig/arch/ia64/kernel/topology.c	2006-09-26 20:31:25.000000000 +0900
> +++ linux-2.6.18/arch/ia64/kernel/topology.c	2006-09-26 20:34:03.000000000 +0900
> @@ -36,10 +36,8 @@
>  	 */
>  	if (!can_cpei_retarget() && is_cpu_cpei_target(num))
>  		sysfs_cpus[num].cpu.no_control = 1;
> -#ifdef CONFIG_NUMA
>  	map_cpu_to_node(num, node_cpuid[num].nid);
>  #endif
> -#endif
>  
>  	return register_cpu(&sysfs_cpus[num].cpu, num);
>  }
> Index: linux-2.6.18/include/asm-ia64/numa.h
> =================================> --- linux-2.6.18.orig/include/asm-ia64/numa.h	2006-09-26 20:31:25.000000000 +0900
> +++ linux-2.6.18/include/asm-ia64/numa.h	2006-09-26 20:33:49.000000000 +0900
> @@ -69,6 +69,8 @@
>  
>  
>  #else /* !CONFIG_NUMA */
> +#define map_cpu_to_node(cpu, nid)	do{}while(0)
> +#define unmap_cpu_from_node(cpu, nid)	do{}while(0)
>  
>  #define paddr_to_nid(addr)	0
>  
> 
> 
> 

Re: [BUGFIX][PATCH] cpu to node relationship fixup take2 [2/2] map cpu to node

[bibo,mao]

yes, the hot-fix works for me now :)

thanks
bibo,mao

KAMEZAWA Hiroyuki wrote:
> 
> Uh... sorry. How about this hot-fix ?
> 
> -Kame
> =========
> 
> 
> Non-NUMA case should be handled...
> 
> Signed-Off-By: KAMEZAWA Hiroyuki <kamezawa.hiroyu@jp.fujitsu.com>
> 
> Index: linux-2.6.18/arch/ia64/kernel/topology.c
> ===================================================================
> --- linux-2.6.18.orig/arch/ia64/kernel/topology.c	2006-09-26 20:31:25.000000000 +0900
> +++ linux-2.6.18/arch/ia64/kernel/topology.c	2006-09-26 20:34:03.000000000 +0900
> @@ -36,10 +36,8 @@
>  	 */
>  	if (!can_cpei_retarget() && is_cpu_cpei_target(num))
>  		sysfs_cpus[num].cpu.no_control = 1;
> -#ifdef CONFIG_NUMA
>  	map_cpu_to_node(num, node_cpuid[num].nid);
>  #endif
> -#endif
>  
>  	return register_cpu(&sysfs_cpus[num].cpu, num);
>  }
> Index: linux-2.6.18/include/asm-ia64/numa.h
> ===================================================================
> --- linux-2.6.18.orig/include/asm-ia64/numa.h	2006-09-26 20:31:25.000000000 +0900
> +++ linux-2.6.18/include/asm-ia64/numa.h	2006-09-26 20:33:49.000000000 +0900
> @@ -69,6 +69,8 @@
>  
>  
>  #else /* !CONFIG_NUMA */
> +#define map_cpu_to_node(cpu, nid)	do{}while(0)
> +#define unmap_cpu_from_node(cpu, nid)	do{}while(0)
>  
>  #define paddr_to_nid(addr)	0
>  
> 
> 
> 

Re: [BUGFIX][PATCH] cpu to node relationship fixup take2 [1/2]

[Andrew Morton]

On Fri, 22 Sep 2006 15:24:47 +0900
KAMEZAWA Hiroyuki <kamezawa.hiroyu@jp.fujitsu.com> wrote:

> I rewrote the whoe patch..
> 

Well I don't recall ever having seen a "cpu to node relationship fixup
take1" and I have generally lost the plot regarding these fixes.

What I have now is:

cpu-to-node-relationship-fixup-take2.patch
cpu-to-node-relationship-fixup-map-cpu-to-node.patch

I shall send those patches in reply to this email.  Please confirm that
these are correct, sufficient, complete, etc.

Do you believe these are needed in 2.6.18.x?

Please follow the guidelines in
http://www.zip.com.au/~akpm/linux/patches/stuff/tpp.txt a little more
closely, especially regarding Subject:s

Please bear in mind that I'm sitting on thousands of patches from hundreds
of developers, that I process sometimes hundreds of patches per day and I
am very easily confused.  Retaining consistent and well-thought out
Subject:s and referring to previous patches via their precise Subject:s
really helps, thanks.

Re: [BUGFIX][PATCH] cpu to node relationship fixup take2 [1/2] acpi_map_cpu2node

[Andrew Morton]

On Fri, 22 Sep 2006 15:24:47 +0900
KAMEZAWA Hiroyuki <kamezawa.hiroyu@jp.fujitsu.com> wrote:

> I rewrote the whoe patch..
> 

Well I don't recall ever having seen a "cpu to node relationship fixup
take1" and I have generally lost the plot regarding these fixes.

What I have now is:

cpu-to-node-relationship-fixup-take2.patch
cpu-to-node-relationship-fixup-map-cpu-to-node.patch

I shall send those patches in reply to this email.  Please confirm that
these are correct, sufficient, complete, etc.

Do you believe these are needed in 2.6.18.x?

Please follow the guidelines in
http://www.zip.com.au/~akpm/linux/patches/stuff/tpp.txt a little more
closely, especially regarding Subject:s

Please bear in mind that I'm sitting on thousands of patches from hundreds
of developers, that I process sometimes hundreds of patches per day and I
am very easily confused.  Retaining consistent and well-thought out
Subject:s and referring to previous patches via their precise Subject:s
really helps, thanks.

Re: [BUGFIX][PATCH] cpu to node relationship fixup take2 [1/2]

[Andrew Morton]

On Fri, 22 Sep 2006 17:06:04 -0700
Andrew Morton <akpm@osdl.org> wrote:

> cpu-to-node-relationship-fixup-take2.patch

From: KAMEZAWA Hiroyuki <kamezawa.hiroyu@jp.fujitsu.com>

Problem description:

  We have additional_cpus= option for allocating possible_cpus.  But nid
  for possible cpus are not fixed at boot time.  cpus which is offlined at
  boot or cpus which is not on SRAT is not tied to its node.  This will
  cause panic at cpu onlining.

Usually, pxm_to_nid() mapping is fixed at boot time by SRAT.

But, unfortunately, some system (my system!) do not include
full SRAT table for possible cpus.  (Then, I use
additiona_cpus= option.)

For such possible cpus, pxm<->nid should be fixed at
hot-add.  We now have acpi_map_pxm_to_node() which is also
used at boot.  It's suitable here.

Signed-off-by: KAMEZAWA Hiroyuki <kamezawa.hiroyu@jp.fujitsu.com>
Cc: "Luck, Tony" <tony.luck@intel.com>
Cc: <stable@kernel.org>
Signed-off-by: Andrew Morton <akpm@osdl.org>
---

 arch/ia64/kernel/acpi.c |   13 ++++++++-----
 1 file changed, 8 insertions(+), 5 deletions(-)

diff -puN arch/ia64/kernel/acpi.c~cpu-to-node-relationship-fixup-take2 arch/ia64/kernel/acpi.c
--- a/arch/ia64/kernel/acpi.c~cpu-to-node-relationship-fixup-take2
+++ a/arch/ia64/kernel/acpi.c
@@ -771,16 +771,19 @@ int acpi_map_cpu2node(acpi_handle handle
 {
 #ifdef CONFIG_ACPI_NUMA
 	int pxm_id;
+	int nid;
 
 	pxm_id = acpi_get_pxm(handle);
-
 	/*
-	 * Assuming that the container driver would have set the proximity
-	 * domain and would have initialized pxm_to_node(pxm_id) && pxm_flag
+	 * We don't have cpu-only-node hotadd. But if the system equips
+	 * SRAT table, pxm is already found and node is ready.
+  	 * So, just pxm_to_nid(pxm) is OK.
+	 * This code here is for the system which doesn't have full SRAT
+  	 * table for possible cpus.
 	 */
-	node_cpuid[cpu].nid = (pxm_id < 0) ? 0 : pxm_to_node(pxm_id);
-
+	nid = acpi_map_pxm_to_node(pxm_id);
 	node_cpuid[cpu].phys_id = physid;
+	node_cpuid[cpu].nid = nid;
 #endif
 	return (0);
 }
_

Re: [BUGFIX][PATCH] cpu to node relationship fixup take2 [1/2] acpi_map_cpu2node

[Andrew Morton]

On Fri, 22 Sep 2006 17:06:04 -0700
Andrew Morton <akpm@osdl.org> wrote:

> cpu-to-node-relationship-fixup-take2.patch

From: KAMEZAWA Hiroyuki <kamezawa.hiroyu@jp.fujitsu.com>

Problem description:

  We have additional_cpus= option for allocating possible_cpus.  But nid
  for possible cpus are not fixed at boot time.  cpus which is offlined at
  boot or cpus which is not on SRAT is not tied to its node.  This will
  cause panic at cpu onlining.

Usually, pxm_to_nid() mapping is fixed at boot time by SRAT.

But, unfortunately, some system (my system!) do not include
full SRAT table for possible cpus.  (Then, I use
additiona_cpus= option.)

For such possible cpus, pxm<->nid should be fixed at
hot-add.  We now have acpi_map_pxm_to_node() which is also
used at boot.  It's suitable here.

Signed-off-by: KAMEZAWA Hiroyuki <kamezawa.hiroyu@jp.fujitsu.com>
Cc: "Luck, Tony" <tony.luck@intel.com>
Cc: <stable@kernel.org>
Signed-off-by: Andrew Morton <akpm@osdl.org>
---

 arch/ia64/kernel/acpi.c |   13 ++++++++-----
 1 file changed, 8 insertions(+), 5 deletions(-)

diff -puN arch/ia64/kernel/acpi.c~cpu-to-node-relationship-fixup-take2 arch/ia64/kernel/acpi.c
--- a/arch/ia64/kernel/acpi.c~cpu-to-node-relationship-fixup-take2
+++ a/arch/ia64/kernel/acpi.c
@@ -771,16 +771,19 @@ int acpi_map_cpu2node(acpi_handle handle
 {
 #ifdef CONFIG_ACPI_NUMA
 	int pxm_id;
+	int nid;
 
 	pxm_id = acpi_get_pxm(handle);
-
 	/*
-	 * Assuming that the container driver would have set the proximity
-	 * domain and would have initialized pxm_to_node(pxm_id) && pxm_flag
+	 * We don't have cpu-only-node hotadd. But if the system equips
+	 * SRAT table, pxm is already found and node is ready.
+  	 * So, just pxm_to_nid(pxm) is OK.
+	 * This code here is for the system which doesn't have full SRAT
+  	 * table for possible cpus.
 	 */
-	node_cpuid[cpu].nid = (pxm_id < 0) ? 0 : pxm_to_node(pxm_id);
-
+	nid = acpi_map_pxm_to_node(pxm_id);
 	node_cpuid[cpu].phys_id = physid;
+	node_cpuid[cpu].nid = nid;
 #endif
 	return (0);
 }
_

Re: [BUGFIX][PATCH] cpu to node relationship fixup take2 [1/2]

[Andrew Morton]

On Fri, 22 Sep 2006 17:06:04 -0700
Andrew Morton <akpm@osdl.org> wrote:

> cpu-to-node-relationship-fixup-map-cpu-to-node.patch

From: KAMEZAWA Hiroyuki <kamezawa.hiroyu@jp.fujitsu.com>

Assume that a cpu is *physically* offlined at boot time...

Because smpboot.c::smp_boot_cpu_map() canoot find cpu's sapicid,
numa.c::build_cpu_to_node_map() cannot build cpu<->node map for
offlined cpu.

For such cpus, cpu_to_node map should be fixed at cpu-hot-add.
This mapping should be done before cpu onlining.

This patch also handles cpu hotremove case.

Signed-off-by: KAMEZAWA Hiroyuki <kamezawa.hiroyu@jp.fujitsu.com>
Cc: "Luck, Tony" <tony.luck@intel.com>
Cc: <stable@kernel.org>
Signed-off-by: Andrew Morton <akpm@osdl.org>
---

 arch/ia64/kernel/numa.c     |   34 +++++++++++++++++++++++++++++++---
 arch/ia64/kernel/topology.c |    6 +++++-
 include/asm-ia64/numa.h     |    4 ++++
 3 files changed, 40 insertions(+), 4 deletions(-)

diff -puN arch/ia64/kernel/numa.c~cpu-to-node-relationship-fixup-map-cpu-to-node arch/ia64/kernel/numa.c
--- a/arch/ia64/kernel/numa.c~cpu-to-node-relationship-fixup-map-cpu-to-node
+++ a/arch/ia64/kernel/numa.c
@@ -29,6 +29,36 @@ EXPORT_SYMBOL(cpu_to_node_map);
 
 cpumask_t node_to_cpu_mask[MAX_NUMNODES] __cacheline_aligned;
 
+void __cpuinit map_cpu_to_node(int cpu, int nid)
+{
+	int oldnid;
+	if (nid < 0) { /* just initialize by zero */
+		cpu_to_node_map[cpu] = 0;
+		return;
+	}
+	/* sanity check first */
+	oldnid = cpu_to_node_map[cpu];
+	if (cpu_isset(cpu, node_to_cpu_mask[oldnid])) {
+		return; /* nothing to do */
+	}
+	/* we don't have cpu-driven node hot add yet...
+	   In usual case, node is created from SRAT at boot time. */
+	if (!node_online(nid))
+		nid = first_online_node;
+	cpu_to_node_map[cpu] = nid;
+	cpu_set(cpu, node_to_cpu_mask[nid]);
+	return;
+}
+
+void __cpuinit unmap_cpu_from_node(int cpu, int nid)
+{
+	WARN_ON(!cpu_isset(cpu, node_to_cpu_mask[nid]));
+	WARN_ON(cpu_to_node_map[cpu] != nid);
+	cpu_to_node_map[cpu] = 0;
+	cpu_clear(cpu, node_to_cpu_mask[nid]);
+}
+
+
 /**
  * build_cpu_to_node_map - setup cpu to node and node to cpumask arrays
  *
@@ -49,8 +79,6 @@ void __init build_cpu_to_node_map(void)
 				node = node_cpuid[i].nid;
 				break;
 			}
-		cpu_to_node_map[cpu] = (node >= 0) ? node : 0;
-		if (node >= 0)
-			cpu_set(cpu, node_to_cpu_mask[node]);
+		map_cpu_to_node(cpu, node);
 	}
 }
diff -puN arch/ia64/kernel/topology.c~cpu-to-node-relationship-fixup-map-cpu-to-node arch/ia64/kernel/topology.c
--- a/arch/ia64/kernel/topology.c~cpu-to-node-relationship-fixup-map-cpu-to-node
+++ a/arch/ia64/kernel/topology.c
@@ -36,6 +36,9 @@ int arch_register_cpu(int num)
 	 */
 	if (!can_cpei_retarget() && is_cpu_cpei_target(num))
 		sysfs_cpus[num].cpu.no_control = 1;
+#ifdef CONFIG_NUMA
+	map_cpu_to_node(num, node_cpuid[num].nid);
+#endif
 #endif
 
 	return register_cpu(&sysfs_cpus[num].cpu, num);
@@ -45,7 +48,8 @@ int arch_register_cpu(int num)
 
 void arch_unregister_cpu(int num)
 {
-	return unregister_cpu(&sysfs_cpus[num].cpu);
+	unregister_cpu(&sysfs_cpus[num].cpu);
+	unmap_cpu_from_node(num, cpu_to_node(num));
 }
 EXPORT_SYMBOL(arch_register_cpu);
 EXPORT_SYMBOL(arch_unregister_cpu);
diff -puN include/asm-ia64/numa.h~cpu-to-node-relationship-fixup-map-cpu-to-node include/asm-ia64/numa.h
--- a/include/asm-ia64/numa.h~cpu-to-node-relationship-fixup-map-cpu-to-node
+++ a/include/asm-ia64/numa.h
@@ -64,6 +64,10 @@ extern int paddr_to_nid(unsigned long pa
 
 #define local_nodeid (cpu_to_node_map[smp_processor_id()])
 
+extern void map_cpu_to_node(int cpu, int nid);
+extern void unmap_cpu_from_node(int cpu, int nid);
+
+
 #else /* !CONFIG_NUMA */
 
 #define paddr_to_nid(addr)	0
_

Re: [BUGFIX][PATCH] cpu to node relationship fixup take2 [1/2] acpi_map_cpu2node

[Andrew Morton]

On Fri, 22 Sep 2006 17:06:04 -0700
Andrew Morton <akpm@osdl.org> wrote:

> cpu-to-node-relationship-fixup-map-cpu-to-node.patch

From: KAMEZAWA Hiroyuki <kamezawa.hiroyu@jp.fujitsu.com>

Assume that a cpu is *physically* offlined at boot time...

Because smpboot.c::smp_boot_cpu_map() canoot find cpu's sapicid,
numa.c::build_cpu_to_node_map() cannot build cpu<->node map for
offlined cpu.

For such cpus, cpu_to_node map should be fixed at cpu-hot-add.
This mapping should be done before cpu onlining.

This patch also handles cpu hotremove case.

Signed-off-by: KAMEZAWA Hiroyuki <kamezawa.hiroyu@jp.fujitsu.com>
Cc: "Luck, Tony" <tony.luck@intel.com>
Cc: <stable@kernel.org>
Signed-off-by: Andrew Morton <akpm@osdl.org>
---

 arch/ia64/kernel/numa.c     |   34 +++++++++++++++++++++++++++++++---
 arch/ia64/kernel/topology.c |    6 +++++-
 include/asm-ia64/numa.h     |    4 ++++
 3 files changed, 40 insertions(+), 4 deletions(-)

diff -puN arch/ia64/kernel/numa.c~cpu-to-node-relationship-fixup-map-cpu-to-node arch/ia64/kernel/numa.c
--- a/arch/ia64/kernel/numa.c~cpu-to-node-relationship-fixup-map-cpu-to-node
+++ a/arch/ia64/kernel/numa.c
@@ -29,6 +29,36 @@ EXPORT_SYMBOL(cpu_to_node_map);
 
 cpumask_t node_to_cpu_mask[MAX_NUMNODES] __cacheline_aligned;
 
+void __cpuinit map_cpu_to_node(int cpu, int nid)
+{
+	int oldnid;
+	if (nid < 0) { /* just initialize by zero */
+		cpu_to_node_map[cpu] = 0;
+		return;
+	}
+	/* sanity check first */
+	oldnid = cpu_to_node_map[cpu];
+	if (cpu_isset(cpu, node_to_cpu_mask[oldnid])) {
+		return; /* nothing to do */
+	}
+	/* we don't have cpu-driven node hot add yet...
+	   In usual case, node is created from SRAT at boot time. */
+	if (!node_online(nid))
+		nid = first_online_node;
+	cpu_to_node_map[cpu] = nid;
+	cpu_set(cpu, node_to_cpu_mask[nid]);
+	return;
+}
+
+void __cpuinit unmap_cpu_from_node(int cpu, int nid)
+{
+	WARN_ON(!cpu_isset(cpu, node_to_cpu_mask[nid]));
+	WARN_ON(cpu_to_node_map[cpu] != nid);
+	cpu_to_node_map[cpu] = 0;
+	cpu_clear(cpu, node_to_cpu_mask[nid]);
+}
+
+
 /**
  * build_cpu_to_node_map - setup cpu to node and node to cpumask arrays
  *
@@ -49,8 +79,6 @@ void __init build_cpu_to_node_map(void)
 				node = node_cpuid[i].nid;
 				break;
 			}
-		cpu_to_node_map[cpu] = (node >= 0) ? node : 0;
-		if (node >= 0)
-			cpu_set(cpu, node_to_cpu_mask[node]);
+		map_cpu_to_node(cpu, node);
 	}
 }
diff -puN arch/ia64/kernel/topology.c~cpu-to-node-relationship-fixup-map-cpu-to-node arch/ia64/kernel/topology.c
--- a/arch/ia64/kernel/topology.c~cpu-to-node-relationship-fixup-map-cpu-to-node
+++ a/arch/ia64/kernel/topology.c
@@ -36,6 +36,9 @@ int arch_register_cpu(int num)
 	 */
 	if (!can_cpei_retarget() && is_cpu_cpei_target(num))
 		sysfs_cpus[num].cpu.no_control = 1;
+#ifdef CONFIG_NUMA
+	map_cpu_to_node(num, node_cpuid[num].nid);
+#endif
 #endif
 
 	return register_cpu(&sysfs_cpus[num].cpu, num);
@@ -45,7 +48,8 @@ int arch_register_cpu(int num)
 
 void arch_unregister_cpu(int num)
 {
-	return unregister_cpu(&sysfs_cpus[num].cpu);
+	unregister_cpu(&sysfs_cpus[num].cpu);
+	unmap_cpu_from_node(num, cpu_to_node(num));
 }
 EXPORT_SYMBOL(arch_register_cpu);
 EXPORT_SYMBOL(arch_unregister_cpu);
diff -puN include/asm-ia64/numa.h~cpu-to-node-relationship-fixup-map-cpu-to-node include/asm-ia64/numa.h
--- a/include/asm-ia64/numa.h~cpu-to-node-relationship-fixup-map-cpu-to-node
+++ a/include/asm-ia64/numa.h
@@ -64,6 +64,10 @@ extern int paddr_to_nid(unsigned long pa
 
 #define local_nodeid (cpu_to_node_map[smp_processor_id()])
 
+extern void map_cpu_to_node(int cpu, int nid);
+extern void unmap_cpu_from_node(int cpu, int nid);
+
+
 #else /* !CONFIG_NUMA */
 
 #define paddr_to_nid(addr)	0
_

Re: [BUGFIX][PATCH] cpu to node relationship fixup take2 [1/2]

[KAMEZAWA Hiroyuki]

On Fri, 22 Sep 2006 17:06:04 -0700
Andrew Morton <akpm@osdl.org> wrote:

> On Fri, 22 Sep 2006 15:24:47 +0900
> KAMEZAWA Hiroyuki <kamezawa.hiroyu@jp.fujitsu.com> wrote:
> 
> > I rewrote the whoe patch..
> > 
> 
> Well I don't recall ever having seen a "cpu to node relationship fixup
> take1" and I have generally lost the plot regarding these fixes.
> 
Sorry....I noticed cpu-to-node relation ship should be fixed before onlining.
About subject, I should be more carefull.

> What I have now is:
> 
> cpu-to-node-relationship-fixup-take2.patch
> cpu-to-node-relationship-fixup-map-cpu-to-node.patch
> 
> I shall send those patches in reply to this email.  Please confirm that
> these are correct, sufficient, complete, etc.
> 
will do.

> Do you believe these are needed in 2.6.18.x?
> 
yes, I do, becasue my system panics now at cpu onlining.

Thank you.

-Kame

Re: [BUGFIX][PATCH] cpu to node relationship fixup take2 [1/2] acpi_map_cpu2node

[KAMEZAWA Hiroyuki]

On Fri, 22 Sep 2006 17:06:04 -0700
Andrew Morton <akpm@osdl.org> wrote:

> On Fri, 22 Sep 2006 15:24:47 +0900
> KAMEZAWA Hiroyuki <kamezawa.hiroyu@jp.fujitsu.com> wrote:
> 
> > I rewrote the whoe patch..
> > 
> 
> Well I don't recall ever having seen a "cpu to node relationship fixup
> take1" and I have generally lost the plot regarding these fixes.
> 
Sorry....I noticed cpu-to-node relation ship should be fixed before onlining.
About subject, I should be more carefull.

> What I have now is:
> 
> cpu-to-node-relationship-fixup-take2.patch
> cpu-to-node-relationship-fixup-map-cpu-to-node.patch
> 
> I shall send those patches in reply to this email.  Please confirm that
> these are correct, sufficient, complete, etc.
> 
will do.

> Do you believe these are needed in 2.6.18.x?
> 
yes, I do, becasue my system panics now at cpu onlining.

Thank you.

-Kame

Re: [BUGFIX][PATCH] cpu to node relationship fixup take2 [1/2]

[KAMEZAWA Hiroyuki]

On Sat, 23 Sep 2006 10:06:03 +0900
KAMEZAWA Hiroyuki <kamezawa.hiroyu@jp.fujitsu.com> wrote:

> > What I have now is:
> > 
> > cpu-to-node-relationship-fixup-take2.patch
> > cpu-to-node-relationship-fixup-map-cpu-to-node.patch
> > 
> > I shall send those patches in reply to this email.  Please confirm that
> > these are correct, sufficient, complete, etc.
> > 
> will do.
> 
Thank you, Andrew-san. I confirmed that all pathces I need was included in your
replied e-mails.

Regards,
-Kame

Re: [BUGFIX][PATCH] cpu to node relationship fixup take2 [1/2] acpi_map_cpu2node

[KAMEZAWA Hiroyuki]

On Sat, 23 Sep 2006 10:06:03 +0900
KAMEZAWA Hiroyuki <kamezawa.hiroyu@jp.fujitsu.com> wrote:

> > What I have now is:
> > 
> > cpu-to-node-relationship-fixup-take2.patch
> > cpu-to-node-relationship-fixup-map-cpu-to-node.patch
> > 
> > I shall send those patches in reply to this email.  Please confirm that
> > these are correct, sufficient, complete, etc.
> > 
> will do.
> 
Thank you, Andrew-san. I confirmed that all pathces I need was included in your
replied e-mails.

Regards,
-Kame