Questions about the numa support implementation of Linux

Questions about the numa support implementation of Linux

[Christos Margiolas]

Hello,
Few weeks now, I'm working on a map reduce implementation for numa
systems(cell be and amd64 arch) and I have searched the web and irc
channels for information about numa support on Linux. I study about
the libnuma v2.1 library and I wrote some test programs and I
understood enough good the library api. But I still have enough
questions and I think they're kernel support relative. I will try
indite my questions as simple as possible in order to save your time.

I know that the operating system(Linux) is  running on a single node
and it's not distributed over the nodes.

a)When a process or a process's thread is executing on a node(not the
same with the kernel), the local memory(execution node of process or
thread) has a copy of the text and data segments or there are always
references to the memory of the node with the os where the data and
text segments would legacy be? If the final is true, this is a big
bottleneck.

b)Memory allocation system calls which is used by standard c library
api (malloc, calloc, realloc) are aware about numa policy on memory
systems? Or only with the libnuma calls is possible to allocate memory
effectively with respect on numa bind restrictions?

c)If a strict memory  policy was specified, which designates specific
nodes, and the nodes  are out of memory resources, the system will
kill the process or it will will use the swap as it does for a normal
uniform memory system?

Now a question relative with scheduler conduct:

d)When a processor binding function(scheduler or numa api call)
returns the schedulers has applied the requested policy or the changes
will take effect after the context switch? I have the same question
about memory policy appliance.

e)Also the shared libs running only on the kernel's node, right?

I will appreciate also any link or study material about numa on Linux.

Chris Margiolas
Undergraduate Student
Computer Science Department
University of Crete
Ics Forth Computer Architecture
and VLSI Systems Lab

Re: Questions about the numa support implementation of Linux

[Andi Kleen]

On Sun, Nov 22, 2009 at 04:07:27AM +0200, Christos Margiolas wrote:
> Hello,
> Few weeks now, I'm working on a map reduce implementation for numa
> systems(cell be and amd64 arch) and I have searched the web and irc

Single socket cell is not really a classical cc/numa system, if you refer to 
the SPEs, because they don't have a common address space with the other CPUs
NUMA policy is only for systems with cache coherent common address
space.


> channels for information about numa support on Linux. I study about
> the libnuma v2.1 library and I wrote some test programs and I
> understood enough good the library api. But I still have enough
> questions and I think they're kernel support relative. I will try
> indite my questions as simple as possible in order to save your time.
> 
> I know that the operating system(Linux) is  running on a single node
> and it's not distributed over the nodes.

That's not correct; 64bit Linux uses all nodes. 32bit Linux has some
limitations, but it's normally not used in NUMA mode.

> 
> a)When a process or a process's thread is executing on a node(not the
> same with the kernel), the local memory(execution node of process or
> thread) has a copy of the text and data segments or there are always

Linux has no user text duplication in the standard kernel. There were
some experimental out of tree patches. Some non x86 ports do text duplication
for the kernel text.

In practice it can be emulated by running multiple copies of the
executables though.

> references to the memory of the node with the os where the data and
> text segments would legacy be? If the final is true, this is a big
> bottleneck.

It is not.

Remember that most NUMA systems today have large caches, fast
interconnects and relatively low NUMA factors (< 1:2). Also CPUs are 
quite good at prefetching code

A lot of the classical NUMA papers you might have read
were written for systems with much worse NUMA factors and slow
interconnects, and the wisdom in there does not necessarily 
apply.


> b)Memory allocation system calls which is used by standard c library
> api (malloc, calloc, realloc) are aware about numa policy on memory
> systems? Or only with the libnuma calls is possible to allocate memory
> effectively with respect on numa bind restrictions?

When malloc.et.al. allocate fresh memory they are bound by NUMA policies
for the thread. For reallocated/freed objects it's a bit more difficult
although as long as you don't malloc in one thread and free in others
it tends to work out too, because malloc has per thread pools.

There are some custom allocators which are NUMA aware.

> 
> c)If a strict memory  policy was specified, which designates specific
> nodes, and the nodes  are out of memory resources, the system will
> kill the process or it will will use the swap as it does for a normal
> uniform memory system?

It will swap or otherwise free memory.

> d)When a processor binding function(scheduler or numa api call)
> returns the schedulers has applied the requested policy or the changes
> will take effect after the context switch? I have the same question
> about memory policy appliance.

It takes effect immediately.

> 
> e)Also the shared libs running only on the kernel's node, right?

There's no kernel node.

> I will appreciate also any link or study material about numa on Linux.

http://halobates.de/numaapi3.pdf is a somewhat outdated introduction.

-Andi

-- 
ak@linux.intel.com -- Speaking for myself only.