NUMA Questions

NUMA Questions

[Scott Robert Ladd]

I'm running a Tyan K8W 2885, with two Opteron 240s and 512MB in each
memory bank. I've compiled kernel 2.6.7-rc1 with the following in my config:

CONFIG_K8_NUMA=y
CONFIG_DISCONTIGMEM=y
CONFIG_NUMA=y

In /var/log/messages, I see:

May 24 14:45:11 Corwin Scanning NUMA topology in Northbridge 24
May 24 14:45:11 Corwin Number of nodes 2 (10010)
May 24 14:45:11 Corwin Node 0 MemBase 0000000000000000 Limit 
000000001fffffff
May 24 14:45:11 Corwin Node 1 MemBase 0000000020000000 Limit 
000000003fff0000
May 24 14:45:11 Corwin Using node hash shift of 24
May 24 14:45:11 Corwin Bootmem setup node 0 
0000000000000000-000000001fffffff
May 24 14:45:11 Corwin Bootmem setup node 1 
0000000020000000-000000003fff0000
May 24 14:45:11 Corwin ACPI: have wakeup address 0x10020005000
May 24 14:45:11 Corwin No mptable found.
May 24 14:45:11 Corwin No mptable found.
May 24 14:45:11 Corwin setting up node 0 0-1ffff
May 24 14:45:11 Corwin On node 0 totalpages: 131071
May 24 14:45:11 Corwin DMA zone: 4096 pages, LIFO batch:1
May 24 14:45:11 Corwin Normal zone: 126975 pages, LIFO batch:16
May 24 14:45:11 Corwin HighMem zone: 0 pages, LIFO batch:1
May 24 14:45:11 Corwin setting up node 1 20000-3fff0
May 24 14:45:11 Corwin On node 1 totalpages: 131056
May 24 14:45:11 Corwin DMA zone: 0 pages, LIFO batch:1
May 24 14:45:11 Corwin Normal zone: 131056 pages, LIFO batch:16
May 24 14:45:11 Corwin HighMem zone: 0 pages, LIFO batch:1

So am I correct in assuming that NUMA is up and running? I note that 
Andi's numastat displays:

		         node1         node0
numa_hit	        304539        472556
numa_miss	             0             0
numa_foreign	             0             0
interleave_hit	             0             0
local_node	        303607        472549
other_node	           932             7

Yet when I run Andi's numactl, it states:

Corwin /usr/src/numactl-0.6.3 # numactl --show
No NUMA support available on this system.

A quick test program, and I see:

numa_available() returns -1
numa_max_node() return 1

Can anyone shed light on this?

-- 
Scott Robert Ladd
Coyote Gulch Productions (http://www.coyotegulch.com)
Software Invention for High-Performance Computing

Re: NUMA Questions

[Andi Kleen]

Scott Robert Ladd <coyote@coyotegulch.com> writes:

[hmm, didn't I answer this already?]

>
> Can anyone shed light on this?

The system calls were still missing on x86-64 in 2.6.7rc1. 

Apply this small patch.

-Andi

--- linux-2.6.7rc1/include/asm-x86_64/unistd.h	2004-05-23 15:41:56.000000000 +0200
+++ linux-2.6.7rc1-amd64/include/asm-x86_64/unistd.h	2004-05-24 01:31:42.000000000 +0200
@@ -535,11 +535,11 @@
 #define __NR_vserver		236
 __SYSCALL(__NR_vserver, sys_ni_syscall)
 #define __NR_mbind 		237
-__SYSCALL(__NR_mbind, sys_ni_syscall)
+__SYSCALL(__NR_mbind, sys_mbind)
 #define __NR_set_mempolicy 	238
-__SYSCALL(__NR_set_mempolicy, sys_ni_syscall)
+__SYSCALL(__NR_set_mempolicy, sys_set_mempolicy)
 #define __NR_get_mempolicy 	239
-__SYSCALL(__NR_get_mempolicy, sys_ni_syscall)
+__SYSCALL(__NR_get_mempolicy, sys_get_mempolicy)
 #define __NR_mq_open 		240
 __SYSCALL(__NR_mq_open, sys_mq_open)
 #define __NR_mq_unlink 		241

Re: NUMA Questions

[Scott Robert Ladd]

Andi Kleen wrote:
> [hmm, didn't I answer this already?]

Yes, you did, and I applied the patch earlier, prior to rebuilding the 
kernel and (of course) updating the kernel headers. Then I recompiled 
the numa utilities, with the result I detailed in the original post. 
I've just double-checked that /usr/include/asm/unistd.h includes your patch.

..Scott

-- 
Scott Robert Ladd
Coyote Gulch Productions (http://www.coyotegulch.com)
Software Invention for High-Performance Computing