[ANNOUNCE] [PATCH] Node Hotplug Support

[ANNOUNCE] [PATCH] Node Hotplug Support

[Keiichiro Tokunaga]

Hi,

I'm announcing a new project "LHNS (Linux Hotplug Node Support)"
and patches for 2.6.5:

    http://sourceforge.net/projects/lhns/



Goal
==
The main goal of LHNS is to support "node hotplug" in Linux.

Where, "node" is a container device that contains CPU, memory,
and/or I/O devices.  The definition of "node" is different from one
of "NUMA node".

  - "NUMA node" is: a block of memory and the CPUs, I/O, etc.
    physically on the same bus as the memory (according to NUMA
    project website, this is a common definition).

  - "node" used here is: a hotpluggable hardware that contains
    CPUs, memory, and/or I/O.

The "node" and "NUMA node" are not always the same.  (I'm
sorry for the inconvenient :(  For instance, the "node" could be a
device that contains:

  - processors, memory, and I/O devices
  - processors and memory
  - processors only
  - memory only
  - I/O devices only
  - etc

"node hotplug" allows you to hot-add and hot-remove "node"
without stopping the system.



Why?
==
Someone might think like "Why don't we invoke CPU, memory, IO
hotplug individually without node hotplug?".  However, CPU, 
memory, and IO hotplug cannot remove a node (container device)
from the system physically.  That's a node hotplug's job.  Also,
when hotplug request occurs on a node, node hotplug searchs
resouces on the node, and invokes CPU, memory, and IO hotplug
in proper order.   Actually, the order is very important.  For instance,
when hot-adding a NUMA node, memory should be added first
so that CPUs could allocate data in the memory while the CPUs is
being added.  Otherwise, CPUs need to allocate it in other memory
on other node.  This might cause performance issue.



Design
===
ACPI is used to do some hardware manipulation.
There is no general purpose interface to get hardware information
and manipulate hardware today, but hardware proprietary interfaces.
ACPI is one of them, and I decided to use it because:

  - Its spec is open.
  - I can use it without any hardware special knowledge:)

The following assumptions are necessary for node hotplug:

  - The system has hotpluggable "node" (hardware).
  - Each of "node" is defined as a container device (HID=PNP0A05)
    in ACPI namespace.
  - Existing CPU, memory, I/O hotplug (LHCS, LHMS, PCI HotPlug
    for Linux, etc) produce a hook (function) for Node hotplug.

"node hotplug" consists of one main part and three sub parts so far:

  - ACPI container device hotplug (main)
  - ACPI based CPU hotplug (sub)
  - ACPI based memory hotplug (sub)
  - ACPI based I/O hotplug (sub)

  "ACPI container device hotplug" is: the main part of node hotplug,
  which is supposed to do:

    [hot-addition]
    1. Its handler is invoked when ACPI notify is occured by a node
       attached to the system.
    2. This creates data structures for the node.
       If CONFIG_ACPI_NUMA_CONTAINER=y, this handles NUMA
       related data also (not implemented yet).
    3. This invokes ACPI based CPU/memory/IO hotplug in proper
       order with proper arguments (e.g. acpi_handle of the added node)
    4. This notifies userland that the node is added/removed.
    5. This writes results of hotplug processing to a log file.

    [hot-removal]
    1. Its handler is invoked when ACPI notify is occured or user issues
       hot-removal request via sysfs.
    2. This deletes data structures of the node.
       If CONFIG_ACPI_NUMA_CONTAINER=y, this handles NUMA
       related data also (not implemented yet).
    3. This invokes ACPI based CPU/memory/IO hotplug in proper
       order with proper arguments (e.g. acpi_handle of the node)
    4. This evaluates _EJ0 method of the node.
    5. This notifies userland that the node is added/removed.
    6. This writes results of hotplugging into log file.

  "ACPI based CPU/memory/IO hotplug" are: the sub part, which are
  supposed to do:

    1. They are invoked by the main part with the argument.
    2. They search ACPI namespace to check if there are any devices
       on the node that they should handle.  For instance, if ACPI
       based CPU hotplug searches ACPI namespace to see if there
       are any CPUs on an added node.
    3. If they find devices, they do something and call existing hotplug
       features:
           o LHCS (Linux Hotplug CPU Support)
           o LHMS (Linux Hotplug Memory Support)
           o PCI HotPlug for Linux
           o etc.
       with some appropriate arguments (TBD).

  Then each hotplug handles each device hotplug.

See http://lhns.sourceforge.net/ for more information and figures
(module and component diagram).



Patches
===The following nine patches are available at:

    http://sourceforge.net/projects/lhns/
    (Actually, these files are packed into .bz2 file.)

  - p00001_sci_emu.patch:
      SCI interrupt emulation.

  - p00002_over-20030109.patch:
      Override an original BIOS's DSDT with a DSDT that you make up.

  - p00003_procfs_util.patch
      Extension for procfs.
      A new function remove_recursive_proc_entry().

  - p00004_acpi_processor_bug.patch
      Replace remove_proc_entry() to remove_recursive_proc_entry()
      in drivers/acpi/processor.c to avoid getting Call Trace.

  - p00005_acpi_hp_config.patch
      Kconfigs and Makefiles for ACPI hotplug.

  - p00006_acpi_core.patch
      Changes to ACPI core part for hotplug.
      A new function acpi_bus_scan_free(), etc...

  - p00007_acpi_hp_util.patch
      Utility functions for ACPI hotplug.

  - p00008_acpi_container_hp.patch
      The main part of node hotplug.

  - p00009_acpi_container_hp_cpu.patch
      The sub part (CPU) of node hotplug.



Current Status
=======
The first patches have been tested in an emulation and a virtual
environment without LHCS, LHMS, etc's codes.  However, they are
very early versions of the code, so there are still many rough codes,
not Linux style codes, and TBDs.  The first patches are just nice
to show my idea and how they work.  Some sub parts and NUMA
node support have not been implemented yet.  (Please don't
enable "Hotplug memory" and "Hotplug IO" when you're doing
"make config":)



Usage
==See "Documents" section at http://lhns.sourceforge.net/.  There
are some documents and instructions:

  - How to build the kernel with LHNS patches
  - How to make a fake DSDT
  - How to generate SCI interrupt (emulation)
  - How to invoke hot-addition and hot-removal for container hotplug

Please feel free to take it for a spin!



Plan
==
First of all, I'd like to discuss the design of node hotplug and
interface with hotplug folks.  After I get feedback from the people,
I'll update my patches.  Also I'll release the following features in
the near future.  Then I'll make them work all together and test
them.

  - NUMA node support
  - ACPI based memory hotplug
  - ACPI based IO hotplug
  - H2P bridge hotplug
  - P2P brdige hotplug
  - IOSAPIC hotplug



Any comments, bug reports, or suggestions are welcome.

Thanks,
Keiichiro Tokunaga


-------------------------------------------------------
This SF.Net email is sponsored by Sleepycat Software
Learn developer strategies Cisco, Motorola, Ericsson & Lucent use to 
deliver higher performing products faster, at low TCO.
http://www.sleepycat.com/telcomwpreg.php?From=osdnemail3
_______________________________________________
Linux-hotplug-devel mailing list  http://linux-hotplug.sourceforge.net
Linux-hotplug-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/linux-hotplug-devel

Re: [ANNOUNCE] [PATCH] Node Hotplug Support

[Dave Hansen]

On Fri, 2004-05-07 at 08:39, Keiichiro Tokunaga wrote:
> First of all, I'd like to discuss the design of node hotplug and
> interface with hotplug folks.  After I get feedback from the people,
> I'll update my patches.  Also I'll release the following features in
> the near future.  Then I'll make them work all together and test
> them.
> 
>   - NUMA node support
>   - ACPI based memory hotplug
>   - ACPI based IO hotplug
>   - H2P bridge hotplug
>   - P2P brdige hotplug
>   - IOSAPIC hotplug

How does this interoperate with the current NUMA topology already in
sysfs today?  I don't see any references at all to the current code.  

-- Dave



-------------------------------------------------------
This SF.Net email is sponsored by Sleepycat Software
Learn developer strategies Cisco, Motorola, Ericsson & Lucent use to 
deliver higher performing products faster, at low TCO.
http://www.sleepycat.com/telcomwpreg.php?From=osdnemail3
_______________________________________________
Linux-hotplug-devel mailing list  http://linux-hotplug.sourceforge.net
Linux-hotplug-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/linux-hotplug-devel

Re: [ANNOUNCE] [PATCH] Node Hotplug Support

[Martin J. Bligh]

> ACPI is used to do some hardware manipulation.
> There is no general purpose interface to get hardware information
> and manipulate hardware today, but hardware proprietary interfaces.
> ACPI is one of them, and I decided to use it because:
> 
>   - Its spec is open.
>   - I can use it without any hardware special knowledge:)

You can't base platform-independant Linux code on ACPI, when not all
NUMA boxes will support it. The fact that your particular box may
support it doesn't make it a generally applicable idea ;-)

You need a better abstraction (and preferably one without the massive 
complexity whilst you're at it).

M.



-------------------------------------------------------
This SF.Net email is sponsored by Sleepycat Software
Learn developer strategies Cisco, Motorola, Ericsson & Lucent use to 
deliver higher performing products faster, at low TCO.
http://www.sleepycat.com/telcomwpreg.php?From=osdnemail3
_______________________________________________
Linux-hotplug-devel mailing list  http://linux-hotplug.sourceforge.net
Linux-hotplug-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/linux-hotplug-devel

Re: [ANNOUNCE] [PATCH] Node Hotplug Support

[Keiichiro Tokunaga]

On Fri, 07 May 2004 08:49:05 -0700
Dave Hansen <haveblue@us.ibm.com> wrote:

> On Fri, 2004-05-07 at 08:39, Keiichiro Tokunaga wrote:
> > First of all, I'd like to discuss the design of node hotplug and
> > interface with hotplug folks.  After I get feedback from the people,
> > I'll update my patches.  Also I'll release the following features in
> > the near future.  Then I'll make them work all together and test
> > them.
> > 
> >   - NUMA node support
> >   - ACPI based memory hotplug
> >   - ACPI based IO hotplug
> >   - H2P bridge hotplug
> >   - P2P brdige hotplug
> >   - IOSAPIC hotplug
> 
> How does this interoperate with the current NUMA topology already in
> sysfs today?  I don't see any references at all to the current code.  

There is no NUMA support in the current code yet.  I'll post a
rough patch to show my idea soon.  I'm thinking to regard a
container device that has PXM as a NUMA node so far.

Thanks,
Kei


-------------------------------------------------------
This SF.Net email is sponsored by Sleepycat Software
Learn developer strategies Cisco, Motorola, Ericsson & Lucent use to 
deliver higher performing products faster, at low TCO.
http://www.sleepycat.com/telcomwpreg.php?From=osdnemail3
_______________________________________________
Linux-hotplug-devel mailing list  http://linux-hotplug.sourceforge.net
Linux-hotplug-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/linux-hotplug-devel

Re: [ANNOUNCE] [PATCH] Node Hotplug Support

[Keiichiro Tokunaga]

On Fri, 07 May 2004 09:16:51 -0700
"Martin J. Bligh" <mbligh@aracnet.com> wrote:

> > ACPI is used to do some hardware manipulation.
> > There is no general purpose interface to get hardware information
> > and manipulate hardware today, but hardware proprietary interfaces.
> > ACPI is one of them, and I decided to use it because:
> > 
> >   - Its spec is open.
> >   - I can use it without any hardware special knowledge:)
> 
> You can't base platform-independant Linux code on ACPI, when not all
> NUMA boxes will support it. The fact that your particular box may
> support it doesn't make it a generally applicable idea ;-)

I'm not trying to base everything on ACPI, this happens in the first
patces though.  I would separate them in the future release.
(Actually I put comments about it in the first patch:)

There will be the dependent and independent codes.  Any platform
can share the independent code, and dependent code needs to
be made by each platform.  I think that PCI hotplug takes the same
way.

Thanks,
Kei


-------------------------------------------------------
This SF.Net email is sponsored by Sleepycat Software
Learn developer strategies Cisco, Motorola, Ericsson & Lucent use to 
deliver higher performing products faster, at low TCO.
http://www.sleepycat.com/telcomwpreg.php?From=osdnemail3
_______________________________________________
Linux-hotplug-devel mailing list  http://linux-hotplug.sourceforge.net
Linux-hotplug-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/linux-hotplug-devel

Re: [Lhns-devel] Re: [ANNOUNCE] [PATCH] Node Hotplug Support

[Takayoshi Kochi]

From: Keiichiro Tokunaga <tokunaga.keiich@jp.fujitsu.com>
Subject: [Lhns-devel] Re: [ANNOUNCE] [PATCH] Node Hotplug Support
Date: Mon, 10 May 2004 10:47:25 +0900

> > How does this interoperate with the current NUMA topology already in
> > sysfs today?  I don't see any references at all to the current code.  
> 
> There is no NUMA support in the current code yet.  I'll post a
> rough patch to show my idea soon.  I'm thinking to regard a
> container device that has PXM as a NUMA node so far.

I've not looking closely into the code, but why do you use "PNP0A05"
for container device?
"PNP0A05" is defined as "Generic ISA devie" in the ACPI spec.

I think "module device (ACPI0004)"  sounds more suitable for the
purpose, though I don't know whether your hardware will support it
or not.

Also, assuming devices that have _PXM are nodes sounds a bit too
aggressive for me.  For example, something like below is possible.

Device(\_SB) {
  Processor(CPU0...) {
     Name(_PXM, 0)
  }
  Processor(CPU1...) {
     Name(_PXM, 1)
  }

  Device(PCI0) {
     Name(_PXM, 0)
  }
  Device(PCI1) {
     Name(_PXM,1)
  }
}

(I don't know if such an implementation exists, but from the spec,
it is possible)
In this case, OS has to group devices by same number.

Please don't assume specific ACPI AML implementatin as a generic
rule.

---
Takayoshi Kochi


-------------------------------------------------------
This SF.Net email is sponsored by Sleepycat Software
Learn developer strategies Cisco, Motorola, Ericsson & Lucent use to 
deliver higher performing products faster, at low TCO.
http://www.sleepycat.com/telcomwpreg.php?From=osdnemail3
_______________________________________________
Linux-hotplug-devel mailing list  http://linux-hotplug.sourceforge.net
Linux-hotplug-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/linux-hotplug-devel

Re: [ANNOUNCE] [PATCH] Node Hotplug Support

[Dave Hansen]

On Sun, 2004-05-09 at 18:47, Keiichiro Tokunaga wrote:
> There is no NUMA support in the current code yet.  I'll post a
> rough patch to show my idea soon.  I'm thinking to regard a
> container device that has PXM as a NUMA node so far.

Don't you think it would be a good idea to work with some of the current
code, instead of trying to wrap around it?  

I'm sure Matt Dobson can give you some great ideas about things in the
current NUMA code that aren't hotplug safe.  That really needs to be
done before any other work, anyway.  

-- Dave



-------------------------------------------------------
This SF.Net email is sponsored by Sleepycat Software
Learn developer strategies Cisco, Motorola, Ericsson & Lucent use to 
deliver higher performing products faster, at low TCO.
http://www.sleepycat.com/telcomwpreg.php?From=osdnemail3
_______________________________________________
Linux-hotplug-devel mailing list  http://linux-hotplug.sourceforge.net
Linux-hotplug-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/linux-hotplug-devel

Re: [Lhns-devel] Re: [ANNOUNCE] [PATCH] Node Hotplug Support

[Keiichiro Tokunaga]

On Mon, 10 May 2004 11:12:40 +0900 (JST)
Takayoshi Kochi <t-kochi@bq.jp.nec.com> wrote:

> From: Keiichiro Tokunaga <tokunaga.keiich@jp.fujitsu.com>
> Subject: [Lhns-devel] Re: [ANNOUNCE] [PATCH] Node Hotplug Support
> Date: Mon, 10 May 2004 10:47:25 +0900
> 
> > > How does this interoperate with the current NUMA topology already in
> > > sysfs today?  I don't see any references at all to the current code.  
> > 
> > There is no NUMA support in the current code yet.  I'll post a
> > rough patch to show my idea soon.  I'm thinking to regard a
> > container device that has PXM as a NUMA node so far.
> 
> I've not looking closely into the code, but why do you use "PNP0A05"
> for container device?
> "PNP0A05" is defined as "Generic ISA devie" in the ACPI spec.
> 
> I think "module device (ACPI0004)"  sounds more suitable for the
> purpose, though I don't know whether your hardware will support it
> or not.

Yes.  ACPI0004 could be used as well.  Actually, PNP0A05 is
redefined as "Generic Container Device" in ACPI spec2.0c.
According to that, both PNP0A05 and ACPI0004 behave
almost the same way.  PNP0A05 also sounds nice for me:)
 
> Also, assuming devices that have _PXM are nodes sounds a bit too
> aggressive for me.  For example, something like below is possible.
> 
> Device(\_SB) {
>   Processor(CPU0...) {
>      Name(_PXM, 0)
>   }
>   Processor(CPU1...) {
>      Name(_PXM, 1)
>   }
> 
>   Device(PCI0) {
>      Name(_PXM, 0)
>   }
>   Device(PCI1) {
>      Name(_PXM,1)
>   }
> }
> 
> (I don't know if such an implementation exists, but from the spec,
> it is possible)
> In this case, OS has to group devices by same number.
> Please don't assume specific ACPI AML implementatin as a generic
> rule.

If ACPI ASL is written that way, LHNS cannot do anything since
there is no container device (this is not ACPI term here) in the
system.  LHNS's target is a container *device* (again, not ACPI
term:) and hotplugs it physically (of course, the device needs to
be hotpluggable).  So, LHNS can handle a NUMA node if all
CPUs and memory of the node are on a container device. I know
that it's a big restriction :(

Thanks,
kei


-------------------------------------------------------
This SF.Net email is sponsored by Sleepycat Software
Learn developer strategies Cisco, Motorola, Ericsson & Lucent use to 
deliver higher performing products faster, at low TCO.
http://www.sleepycat.com/telcomwpreg.php?From=osdnemail3
_______________________________________________
Linux-hotplug-devel mailing list  http://linux-hotplug.sourceforge.net
Linux-hotplug-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/linux-hotplug-devel

RE: [Lhns-devel] Re: [ANNOUNCE] [PATCH] Node Hotplug Support

[Rajesh]

On Mon, May 10, 2004 at 04:20:36AM -0700, Keiichiro Tokunaga wrote: 
>
>On Mon, 10 May 2004 11:12:40 +0900 (JST)
>Takayoshi Kochi <t-kochi@bq.jp.nec.com> wrote:
>
>> From: Keiichiro Tokunaga <tokunaga.keiich@jp.fujitsu.com>
>> 
>> I've not looking closely into the code, but why do you use "PNP0A05"
>> for container device?
>> "PNP0A05" is defined as "Generic ISA devie" in the ACPI spec.
>> 
>> I think "module device (ACPI0004)"  sounds more suitable for the
>
>Yes.  ACPI0004 could be used as well.  Actually, PNP0A05 is

I think the code should work regardless of whether the ACPI firmware
issues the Notify() on the ACPI0004 or PNP0A05 container object.
That is, it should register for Notify() on both container objects.

>> Also, assuming devices that have _PXM are nodes sounds a bit too

In addition, we should not assume that proximity (ACPI _PXM) implies 
a unit of hot-plug. ACPI already requires that all hot-pluggable 
objects must have the _EJ0 method. We can use the scope of _EJ0 to 
determine what is ejectable.

>> 
>> Device(\_SB) {
>>   Processor(CPU0...) {
>>      Name(_PXM, 0)
>>   }
>>   Processor(CPU1...) {
>>      Name(_PXM, 1)
>>   }
>> 
>>   Device(PCI0) {
>>      Name(_PXM, 0)
>>   }
>>   Device(PCI1) {
>>      Name(_PXM,1)
>>   }
>> }
>> 
>> (I don't know if such an implementation exists, but from the spec,
>> it is possible)
>> In this case, OS has to group devices by same number.
>> Please don't assume specific ACPI AML implementatin as a generic
>> rule.
>
>If ACPI ASL is written that way, LHNS cannot do anything since
>there is no container device (this is not ACPI term here) in the
>system.  LHNS's target is a container *device* (again, not ACPI
>term:) and hotplugs it physically (of course, the device needs to

To clarify: the scope of LHNS may be restricted to supporting ACPI
based hot-plug of resources within an ACPI container only. However,
we are also working on supporting ACPI Notify() issued directly to
the processor and memory device objects, so this will also be
supported. The overall ACPI hot-plug solution must not require the
user (administrator, kernel builder) to know the details of whether
the ACPI firmware issues a Notify() directly on the processor/memory
object or on a container object that includes processor and/or
memory objects.

Note: I am moving this discussion to acpi-devel list and away from
lkml. We are really discussing extending the current ACPI hot-plug
code to trigger CPU and memory hot-plug (similar to ACPI glue code
that helps PCI hot-plug). LHNS is not trying to define
the correct way to write the kernel code that actually implements CPU 
or memory hot-plug. Obviously, we will need to make sure that hot-plug
triggered by ACPI plays well with hot-plug triggered by non-ACPI means.

Rajesh



-------------------------------------------------------
This SF.Net email is sponsored by Sleepycat Software
Learn developer strategies Cisco, Motorola, Ericsson & Lucent use to 
deliver higher performing products faster, at low TCO.
http://www.sleepycat.com/telcomwpreg.php?From=osdnemail3
_______________________________________________
Linux-hotplug-devel mailing list  http://linux-hotplug.sourceforge.net
Linux-hotplug-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/linux-hotplug-devel

Re: [ANNOUNCE] [PATCH] Node Hotplug Support

[Keiichiro Tokunaga]

On Sun, 09 May 2004 22:45:42 -0700
Dave Hansen <haveblue@us.ibm.com> wrote:

> On Sun, 2004-05-09 at 18:47, Keiichiro Tokunaga wrote:
> > There is no NUMA support in the current code yet.  I'll post a
> > rough patch to show my idea soon.  I'm thinking to regard a
> > container device that has PXM as a NUMA node so far.
> 
> Don't you think it would be a good idea to work with some of the current
> code, instead of trying to wrap around it?  

Are you saying that LHNS should use the current NUMA code
(or coming code in the future) to support NUMA node hotplug?

> I'm sure Matt Dobson can give you some great ideas about things in the
> current NUMA code that aren't hotplug safe.  That really needs to be
> done before any other work, anyway.  

Thanks,
Kei


-------------------------------------------------------
This SF.Net email is sponsored by: SourceForge.net Broadband
Sign-up now for SourceForge Broadband and get the fastest
6.0/768 connection for only $19.95/mo for the first 3 months!
http://ads.osdn.com/?ad_id%62&alloc_ida84&op=click
_______________________________________________
Linux-hotplug-devel mailing list  http://linux-hotplug.sourceforge.net
Linux-hotplug-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/linux-hotplug-devel

Re: [ANNOUNCE] [PATCH] Node Hotplug Support

[Dave Hansen]

On Wed, 2004-05-12 at 18:27, Keiichiro Tokunaga wrote:
> On Sun, 09 May 2004 22:45:42 -0700
> Dave Hansen <haveblue@us.ibm.com> wrote:
> 
> > On Sun, 2004-05-09 at 18:47, Keiichiro Tokunaga wrote:
> > > There is no NUMA support in the current code yet.  I'll post a
> > > rough patch to show my idea soon.  I'm thinking to regard a
> > > container device that has PXM as a NUMA node so far.
> > 
> > Don't you think it would be a good idea to work with some of the current
> > code, instead of trying to wrap around it?  
> 
> Are you saying that LHNS should use the current NUMA code
> (or coming code in the future) to support NUMA node hotplug?

Absolutely.  Why do we need wrappers when we can offline entire nodes
with 6-line shell scripts?  The CPU hotplug interfaces are here today
and the memory stuff will be here soon.  Perhaps you could help with the
NUMA part.

#!/bin/sh
NODENUM=$1
NODEDIR=/sys/devices/system/node/node${NODENUM}
for i in $NODEDIR/cpu* $NODEDIR/memory*; do
	echo 0 > $i/control/online
fi
echo 0 > $NODEDIR/control/online

We don't currently export bus to node mappings in sysfs, but we have
them in the kernel, so that won't be too hard to export as well.  

-- Dave



-------------------------------------------------------
This SF.Net email is sponsored by: SourceForge.net Broadband
Sign-up now for SourceForge Broadband and get the fastest
6.0/768 connection for only $19.95/mo for the first 3 months!
http://ads.osdn.com/?ad_id%62&alloc_ida84&op=click
_______________________________________________
Linux-hotplug-devel mailing list  http://linux-hotplug.sourceforge.net
Linux-hotplug-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/linux-hotplug-devel

Re: [ANNOUNCE] [PATCH] Node Hotplug Support

[Keiichiro Tokunaga]

On Wed, 12 May 2004 19:04:47 -0700
Dave Hansen <haveblue@us.ibm.com> wrote:

> On Wed, 2004-05-12 at 18:27, Keiichiro Tokunaga wrote:
> > On Sun, 09 May 2004 22:45:42 -0700
> > Dave Hansen <haveblue@us.ibm.com> wrote:
> > 
> > > On Sun, 2004-05-09 at 18:47, Keiichiro Tokunaga wrote:
> > > > There is no NUMA support in the current code yet.  I'll post a
> > > > rough patch to show my idea soon.  I'm thinking to regard a
> > > > container device that has PXM as a NUMA node so far.
> > > 
> > > Don't you think it would be a good idea to work with some of the current
> > > code, instead of trying to wrap around it?  
> > 
> > Are you saying that LHNS should use the current NUMA code
> > (or coming code in the future) to support NUMA node hotplug?
> 
> Absolutely.  Why do we need wrappers when we can offline entire nodes
> with 6-line shell scripts?  The CPU hotplug interfaces are here today
> and the memory stuff will be here soon.  Perhaps you could help with the
> NUMA part.
> 
> #!/bin/sh
> NODENUM=$1
> NODEDIR=/sys/devices/system/node/node${NODENUM}
> for i in $NODEDIR/cpu* $NODEDIR/memory*; do
> 	echo 0 > $i/control/online
> fi
> echo 0 > $NODEDIR/control/online
> 
> We don't currently export bus to node mappings in sysfs, but we have
> them in the kernel, so that won't be too hard to export as well.  

LHNS is focusing on "container device hotplug". Container device
could contain CPUs, memory, and/or IO devices.  Container device
could contain only IO devices.  In this case, LHNS cannot use
$NODED/control/online (NUMA stuff) for the container device.

By the way, what happen when you issue
"echo 0 > $NODEDIR/control/online"?  Can you detach it
from the system after echo-ing?

Thanks,
Kei


-------------------------------------------------------
This SF.Net email is sponsored by: SourceForge.net Broadband
Sign-up now for SourceForge Broadband and get the fastest
6.0/768 connection for only $19.95/mo for the first 3 months!
http://ads.osdn.com/?ad_id%62&alloc_ida84&op=click
_______________________________________________
Linux-hotplug-devel mailing list  http://linux-hotplug.sourceforge.net
Linux-hotplug-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/linux-hotplug-devel

Re: [Lhns-devel] Re: [ANNOUNCE] [PATCH] Node Hotplug Support

[Keiichiro Tokunaga]

On Tue, 11 May 2004 11:51:53 -0700
Rajesh <rajesh.shah@intel.com> wrote:

> On Mon, May 10, 2004 at 04:20:36AM -0700, Keiichiro Tokunaga wrote: 
> >
> >On Mon, 10 May 2004 11:12:40 +0900 (JST)
> >Takayoshi Kochi <t-kochi@bq.jp.nec.com> wrote:
> >
> >> From: Keiichiro Tokunaga <tokunaga.keiich@jp.fujitsu.com>
> >> 
> >> I've not looking closely into the code, but why do you use "PNP0A05"
> >> for container device?
> >> "PNP0A05" is defined as "Generic ISA devie" in the ACPI spec.
> >> 
> >> I think "module device (ACPI0004)"  sounds more suitable for the
> >
> >Yes.  ACPI0004 could be used as well.  Actually, PNP0A05 is
> 
> I think the code should work regardless of whether the ACPI firmware
> issues the Notify() on the ACPI0004 or PNP0A05 container object.
> That is, it should register for Notify() on both container objects.

That makes more sense.
 
> >> Also, assuming devices that have _PXM are nodes sounds a bit too
> 
> In addition, we should not assume that proximity (ACPI _PXM) implies 
> a unit of hot-plug. ACPI already requires that all hot-pluggable 
> objects must have the _EJ0 method. We can use the scope of _EJ0 to 
> determine what is ejectable.

This is right.  Referring to _PXM doesn't work...

> >> Device(\_SB) {
> >>   Processor(CPU0...) {
> >>      Name(_PXM, 0)
> >>   }
> >>   Processor(CPU1...) {
> >>      Name(_PXM, 1)
> >>   }
> >> 
> >>   Device(PCI0) {
> >>      Name(_PXM, 0)
> >>   }
> >>   Device(PCI1) {
> >>      Name(_PXM,1)
> >>   }
> >> }
> >> 
> >> (I don't know if such an implementation exists, but from the spec,
> >> it is possible)
> >> In this case, OS has to group devices by same number.
> >> Please don't assume specific ACPI AML implementatin as a generic
> >> rule.
> >
> >If ACPI ASL is written that way, LHNS cannot do anything since
> >there is no container device (this is not ACPI term here) in the
> >system.  LHNS's target is a container *device* (again, not ACPI
> >term:) and hotplugs it physically (of course, the device needs to
> 
> To clarify: the scope of LHNS may be restricted to supporting ACPI
> based hot-plug of resources within an ACPI container only. However,
> we are also working on supporting ACPI Notify() issued directly to
> the processor and memory device objects, so this will also be
> supported.

That'll be nice :)

> The overall ACPI hot-plug solution must not require the
> user (administrator, kernel builder) to know the details of whether
> the ACPI firmware issues a Notify() directly on the processor/memory
> object or on a container object that includes processor and/or
> memory objects.
> Note: I am moving this discussion to acpi-devel list and away from
> lkml. We are really discussing extending the current ACPI hot-plug
> code to trigger CPU and memory hot-plug (similar to ACPI glue code
> that helps PCI hot-plug). LHNS is not trying to define
> the correct way to write the kernel code that actually implements CPU 
> or memory hot-plug. Obviously, we will need to make sure that hot-plug
> triggered by ACPI plays well with hot-plug triggered by non-ACPI means.

Thank you for coordinating.  

Thanks,
Kei


-------------------------------------------------------
This SF.Net email is sponsored by: SourceForge.net Broadband
Sign-up now for SourceForge Broadband and get the fastest
6.0/768 connection for only $19.95/mo for the first 3 months!
http://ads.osdn.com/?ad_id%62&alloc_ida84&op=click
_______________________________________________
Linux-hotplug-devel mailing list  http://linux-hotplug.sourceforge.net
Linux-hotplug-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/linux-hotplug-devel

Re: Node Hotplug Support

[Dave Hansen]

On Wed, 2004-05-12 at 23:35, Keiichiro Tokunaga wrote:
> LHNS is focusing on "container device hotplug". Container device
> could contain CPUs, memory, and/or IO devices.  Container device
> could contain only IO devices.  In this case, LHNS cannot use
> $NODED/control/online (NUMA stuff) for the container device.

So, why not expose your containers in the same way that all of the other
NUMA node information is exported?  What makes your NUMA containers
different from all of the other flavors of NUMA implementations in
Linux?

> By the way, what happen when you issue
> "echo 0 > $NODEDIR/control/online"?  Can you detach it
> from the system after echo-ing?

Well, since it doesn't exist yet... Sure :)

-- Dave



-------------------------------------------------------
This SF.Net email is sponsored by: SourceForge.net Broadband
Sign-up now for SourceForge Broadband and get the fastest
6.0/768 connection for only $19.95/mo for the first 3 months!
http://ads.osdn.com/?ad_id%62&alloc_ida84&op=click
_______________________________________________
Linux-hotplug-devel mailing list  http://linux-hotplug.sourceforge.net
Linux-hotplug-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/linux-hotplug-devel

Re: [Lhns-devel] Re: Node Hotplug Support

[Dave Hansen]

On Wed, 2004-05-12 at 23:45, Dave Hansen wrote:
> On Wed, 2004-05-12 at 23:35, Keiichiro Tokunaga wrote:
> > By the way, what happen when you issue
> > "echo 0 > $NODEDIR/control/online"?  Can you detach it
> > from the system after echo-ing?
> 
> Well, since it doesn't exist yet... Sure :)

What we really need is something like eject(1) for system devices.  

-- Dave



-------------------------------------------------------
This SF.Net email is sponsored by: SourceForge.net Broadband
Sign-up now for SourceForge Broadband and get the fastest
6.0/768 connection for only $19.95/mo for the first 3 months!
http://ads.osdn.com/?ad_id%62&alloc_ida84&op=click
_______________________________________________
Linux-hotplug-devel mailing list  http://linux-hotplug.sourceforge.net
Linux-hotplug-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/linux-hotplug-devel

Re: Node Hotplug Support

[Keiichiro Tokunaga]

On Wed, 12 May 2004 23:45:38 -0700
Dave Hansen <haveblue@us.ibm.com> wrote:

> On Wed, 2004-05-12 at 23:35, Keiichiro Tokunaga wrote:
> > LHNS is focusing on "container device hotplug". Container device
> > could contain CPUs, memory, and/or IO devices.  Container device
> > could contain only IO devices.  In this case, LHNS cannot use
> > $NODED/control/online (NUMA stuff) for the container device.
> 
> So, why not expose your containers in the same way that all of the other
> NUMA node information is exported?  What makes your NUMA containers
> different from all of the other flavors of NUMA implementations in
> Linux?
> 
> > By the way, what happen when you issue
> > "echo 0 > $NODEDIR/control/online"?  Can you detach it
> > from the system after echo-ing?
> 
> Well, since it doesn't exist yet... Sure :)

The same way as you described in your previous email seems
work for a container device if we have $CONTAINERD/online.
Anyway, I feel we need to be more specific about implementation.
Is there already any information that I can access?

Thanks,
Kei


-------------------------------------------------------
This SF.Net email is sponsored by: SourceForge.net Broadband
Sign-up now for SourceForge Broadband and get the fastest
6.0/768 connection for only $19.95/mo for the first 3 months!
http://ads.osdn.com/?ad_id%62&alloc_ida84&op=click
_______________________________________________
Linux-hotplug-devel mailing list  http://linux-hotplug.sourceforge.net
Linux-hotplug-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/linux-hotplug-devel

Re: Node Hotplug Support

[Dave Hansen]

On Thu, 2004-05-13 at 18:13, Keiichiro Tokunaga wrote:
> The same way as you described in your previous email seems
> work for a container device if we have $CONTAINERD/online.
> Anyway, I feel we need to be more specific about implementation.
> Is there already any information that I can access?

I don't think I understand.  We already have the concept of a node.  Why
do we also need a container?

-- Dave



-------------------------------------------------------
This SF.Net email is sponsored by: SourceForge.net Broadband
Sign-up now for SourceForge Broadband and get the fastest
6.0/768 connection for only $19.95/mo for the first 3 months!
http://ads.osdn.com/?ad_id%62&alloc_ida84&op=click
_______________________________________________
Linux-hotplug-devel mailing list  http://linux-hotplug.sourceforge.net
Linux-hotplug-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/linux-hotplug-devel