Re: [2/8,v3] NUMA Hotplug Emulator: infrastructure of NUMA hotplug emulation

David Rientjes
2010-11-17T21:11:02Z

On Wed, 17 Nov 2010, Shaohui Zheng wrote:

> > Hmm, why can't you use numa=hide to hide a specified quantity of memory
> > from the kernel and then use the add_memory() interface to hot-add the
> > offlined memory in the desired quantity? In other words, why do you need
> > to track the offlined nodes with a state?
> >
> > The userspace interface would take a desired size of hidden memory to
> > hot-add and the node id would be the first_unset_node(node_online_map).
> Yes, it is a good idea, your solution is what we indeed do in our first 2
> versions. We use mem=memsize to hide memory, and we call add_memory interface
> to hot-add offlined memory with desired quantity, and we can also add to
> desired nodes(even through the nodes does not exists). it is very flexible
> solution.
>
> However, this solution was denied since we notice NUMA emulation, we should
> reuse it.
>

I don't understand why that's a requirement, NUMA emulation is a seperate
feature. Although both are primarily used to test and instrument other VM
and kernel code, NUMA emulation is restricted to only being used at boot
to fake nodes on smaller machines and can be used to test things like the
slab allocator. The NUMA hotplug emulator that you're developing here is
primarily used to test the hotplug callbacks; for that use-case, it seems
particularly helpful if nodes can be hotplugged of various sizes and node
ids rather than having static characteristics that cannot be changed with
a reboot.

> Currently, our solution creates static nodes when OS boots, only the node with
> state N_HIDDEN can be hot-added with node/probe interface, and we can query
>

The idea that I've proposed (and you've apparently thought about and even
implemented at one point) is much more powerful than that. We need not
query the state of hidden nodes that we've setup at boot but can rather
use the amount of hidden memory to setup the nodes in any way that we want
at runtime (various sizes, interleaved node ids, etc).