[v2, 0/8] NUMA Hotplug emulator

[v2, 0/8] NUMA Hotplug emulator

[Shaohui Zheng]

Hi, All

	This patchset introduces NUMA hotplug emulator for x86. we already sent out
an early version in LKML (http ://lwn.net/Articles/387571/). This is 4th version
in internal, and 2nd time to sent to LKML. 

Compare with last version, we accept the comments and feedback from the mailing
list, and did more testing and bug fixing on different hardwares. It is relative
stable version.

* WHAT IS HOTPLUG EMULATOR 

NUMA hotplug emulator is collectively named for the hotplug emulation it is
able to emulate NUMA Node Hotplug thru a pure software way. It intends to help
people easily debug and test node/cpu/memory hotplug related stuff on a
none-numa-hotplug-support machine, even an UMA machine.

The emulator provides mechanism to emulate the process of physcial cpu/mem
hotadd, it provides possibility to debug CPU and memory hotplug on the machines
without NUMA support for kenrel developers. It offers an interface for cpu and
memory hotplug test purpose.

* WHY DO WE USE HOTPLUG EMULATOR

We are focusing on the hotplug emualation for a few months. The emualor helps
 team to reproduce all the major hotplug bugs. It plays an important role to
the hotplug code quality assuirance. Because of the hotplug emulator, we already
move most of the debug working to virtual evironment.

* Principles & Usages 

NUMA hotplug emulator include 3 different parts, We add a menu item to the
menuconfig to enable/disable them.

1) Node hotplug emulation:

The emulator firstly hides RAM via E820 table, and then it can
fake offlined nodes with the hidden RAM.

After system bootup, user is able to hotplug-add these offlined
nodes, which is just similar to a real hotplug hardware behavior.

Using boot option "numa=hide=N*size" to fake offlined nodes:
	- N is the number of hidden nodes
	- size is the memory size (in MB) per hidden node.

There is a sysfs entry "probe" under /sys/devices/system/node/ for user
to hotplug the fake offlined nodes:

 - to show all fake offlined nodes:
    $ cat /sys/devices/system/node/probe

 - to hotadd a fake offlined node, e.g. nodeid is N:
    $ echo N > /sys/devices/system/node/probe

2) CPU hotplug emulation:

The emulator reserve CPUs throu grub parameter, the reserved CPUs can be
hot-add/hot-remove in software method.

When hotplug a CPU with emulator, we are using a logical CPU to emulate the CPU
hotplug process. For the CPU supported SMT, some logical CPUs are in the same
socket, but it may located in different NUMA node after we have emulator.  We
put the logical CPU into a fake CPU socket, and assign it an unique
phys_proc_id. For the fake socket, we put one logical CPU in only.

 - to hide CPUs
	- Using boot option "maxcpus=N" hide CPUs
	  N is the number of initialize CPUs
	- Using boot option "cpu_hpe=on" to enable cpu hotplug emulation
      when cpu_hpe is enabled, the rest CPUs will not be initialized 

 - to hot-add CPU to node
	$ echo nid > cpu/probe

 - to hot-remove CPU
	$ echo nid > cpu/release

3) Memory hotplug emulation:

The emulator reserve memory before OS booting, the reserved memory region
is remove from e820 table, and they can be hot-added via the probe interface,
this interface was extend to support add memory to the specified node, It
maintains backwards compatibility.

The difficulty of Memory Release is well-known, we have no plan for it until now.

 - reserve memory throu grub parameter
 	mem\x1024m

 - add a memory section to node 3
    $ echo 0x40000000,3 > memory/probe
	OR
    $ echo 1024m,3 > memory/probe

* ACKNOWLEDGMENT 

hotplug emulator includes a team's efforts, thanks all of them.
They are:
Andi Kleen, Haicheng Li, Shaohui Zheng, Fengguang Wu and Yongkang You

-- 
Thanks & Regards,
Shaohui

Re: [v2, 0/8] NUMA Hotplug emulator

[Greg KH]

On Sat, Nov 13, 2010 at 01:37:14PM +0800, Shaohui Zheng wrote:
> Hi, All
> 
> 	This patchset introduces NUMA hotplug emulator for x86. we already sent out
> an early version in LKML (http ://lwn.net/Articles/387571/). This is 4th version
> in internal, and 2nd time to sent to LKML. 

Please send this to lkml, the linux-hotplug mailing list is primarily
for userspace hotplug tools (like udev) and not for kernel stuff like
this where you want the core kernel developers to review your patches.

Please use the script, scripts/get_maintainer.pl to determine who best
to send your patches to, it will tell you the mailing list as well as
the people involved.

good luck,

greg k-h

Re: [v2, 0/8] NUMA Hotplug emulator

[Wu Fengguang]

On Sat, Nov 13, 2010 at 10:42:52PM +0800, Greg KH wrote:
> On Sat, Nov 13, 2010 at 01:37:14PM +0800, Shaohui Zheng wrote:
> > Hi, All
> > 
> > 	This patchset introduces NUMA hotplug emulator for x86. we already sent out
> > an early version in LKML (http ://lwn.net/Articles/387571/). This is 4th version
> > in internal, and 2nd time to sent to LKML. 
> 
> Please send this to lkml, the linux-hotplug mailing list is primarily
> for userspace hotplug tools (like udev) and not for kernel stuff like
> this where you want the core kernel developers to review your patches.

And try the git/quilt tools when resending the emails to LKML.

> Please use the script, scripts/get_maintainer.pl to determine who best
> to send your patches to, it will tell you the mailing list as well as
> the people involved.

I run a handy script (based on your git-authors) to help me create the CC
list for given source files:

        [ $# -lt "1" ] && { echo 'usage: git authors <files...>'; exit -1; }

        git log $@ | grep Author: | cut -d: -f2 | sort | uniq -c | sort -n | tail -5
        echo
        git log $@ | grep Author: | cut -d: -f2 | sort | uniq -c | sort -n | tail -5 \
                | sed 's/ *[0-9]\+ /CC:/' \
                | sed 's/.*clameter@sgi.com.*/CC: Christoph Lameter <cl@linux-foundation.org>/' \
                | sed 's/.*hugh@veritas.com.*/CC: Hugh Dickins <hugh.dickins@tiscali.co.uk>/' \
                | sed 's/.*npiggin@suse.de.*/CC: Nick Piggin <npiggin@kernel.dk>/' \
                | sed 's/.*nickpiggin@yahoo.com.au.*/CC: Nick Piggin <npiggin@kernel.dk>/' \
                | sed 's/.*jens.axboe@oracle.com.*/CC: Jens Axboe <axboe@kernel.dk>/' \
                | sed 's/.*dgc@sgi.com.*/CC: Dave Chinner <david@fromorbit.com>/'

The output looks like

        wfg@bee ~/cc/linux-2.6% git-authors-email mm/memory_hotplug.c 
              3  Keith Mannthey <kmannth@us.ibm.com>
              3  Minchan Kim <minchan.kim@gmail.com>
              4  Dave Hansen <haveblue@us.ibm.com>
             14  KAMEZAWA Hiroyuki <kamezawa.hiroyu@jp.fujitsu.com>
             14  Yasunori Goto <y-goto@jp.fujitsu.com>

        CC: Keith Mannthey <kmannth@us.ibm.com>
        CC: Minchan Kim <minchan.kim@gmail.com>
        CC: Dave Hansen <haveblue@us.ibm.com>
        CC: KAMEZAWA Hiroyuki <kamezawa.hiroyu@jp.fujitsu.com>
        CC: Yasunori Goto <y-goto@jp.fujitsu.com>

Thanks,
Fengguang

Re: [v2, 0/8] NUMA Hotplug emulator

[Shaohui Zheng]

On Sat, Nov 13, 2010 at 06:42:52AM -0800, Greg KH wrote:
> On Sat, Nov 13, 2010 at 01:37:14PM +0800, Shaohui Zheng wrote:
> 
> Please use the script, scripts/get_maintainer.pl to determine who best
> to send your patches to, it will tell you the mailing list as well as
> the people involved.
> 
thanks greg, I misunderstand the linux-hotplug mailing list is relate to
 CPU/Memory hotplug. I will recheck it with the script, and send them
to the correct list.

> good luck,
> 
> greg k-h

-- 
Thanks & Regards,
Shaohui