[PATCH] NUMA-aware per-cpu allocation

[PATCH] NUMA-aware per-cpu allocation

[Rusty Russell]

Stolen from x86_64: if we have NUMA, then use alloc_bootmem_node.

Can't hurt, I figure.

In 2.7, my cunning plan is to require contiguous sections, and allocate
similar sections for the dynamic per-cpu allocations, so this won't work
(archs will need to use tricks to create virtually-contiguous memory
from different NUMA nodes).  I have a bitrotted implementation.

Meanwhile, this compiles: does it help performance?

Name: Per-CPU Memory On Right Nodes
Status: Untested

Stolen from x86_64: for NUMA machines, allocate the per-cpu memory
using alloc_bootmem_node so it's optimally placed.

diff -urpN --exclude TAGS -X /home/rusty/devel/kernel/kernel-patches/current-dontdiff --minimal .1784-linux-2.6.7-rc2-bk4/include/asm-ppc64/percpu.h .1784-linux-2.6.7-rc2-bk4.updated/include/asm-ppc64/percpu.h
--- .1784-linux-2.6.7-rc2-bk4/include/asm-ppc64/percpu.h	2004-02-04 15:39:11.000000000 +1100
+++ .1784-linux-2.6.7-rc2-bk4.updated/include/asm-ppc64/percpu.h	2004-06-04 14:49:59.000000000 +1000
@@ -1,6 +1,8 @@
 #ifndef __ARCH_PPC64_PERCPU__
 #define __ARCH_PPC64_PERCPU__
 
+/* Big sloppy arch needs more per-cpu room than usual */
+#define PERCPU_ENOUGH_ROOM 65536
 #include <asm-generic/percpu.h>
 
 #endif /* __ARCH_PPC64_PERCPU__ */
diff -urpN --exclude TAGS -X /home/rusty/devel/kernel/kernel-patches/current-dontdiff --minimal .1784-linux-2.6.7-rc2-bk4/init/main.c .1784-linux-2.6.7-rc2-bk4.updated/init/main.c
--- .1784-linux-2.6.7-rc2-bk4/init/main.c	2004-05-31 09:57:40.000000000 +1000
+++ .1784-linux-2.6.7-rc2-bk4.updated/init/main.c	2004-06-04 14:53:17.000000000 +1000
@@ -338,12 +338,26 @@ static void __init setup_per_cpu_areas(v
 		size = PERCPU_ENOUGH_ROOM;
 #endif
 
+	/* Multiple allocs for non-NUMA just wastes memory. */
+#ifdef CONFIG_NUMA
+	for (i = 0; i < NR_CPUS; i++) { 
+		pg_data_t *pgdat;
+
+		pgdat = NODE_DATA(cpu_to_node(i));
+		ptr = alloc_bootmem_node(pgdat, size);
+		if (!ptr)
+			panic("Cannot allocate cpu data for CPU %ld\n", i);
+		__per_cpu_offset[i] = ptr - __per_cpu_start;
+		memcpy(ptr, __per_cpu_start, __per_cpu_end - __per_cpu_start);
+	}
+#else
 	ptr = alloc_bootmem(size * NR_CPUS);
 
 	for (i = 0; i < NR_CPUS; i++, ptr += size) {
 		__per_cpu_offset[i] = ptr - __per_cpu_start;
 		memcpy(ptr, __per_cpu_start, __per_cpu_end - __per_cpu_start);
 	}
+#endif /* CONFIG_NUMA */
 }
 #endif /* !__GENERIC_PER_CPU */
 

-- 
Anyone who quotes me in their signature is an idiot -- Rusty Russell