Re: [Lhns-devel] Re: Who's doing what with cpu/memory/node hotplug?

Re: [Lhns-devel] Re: Who's doing what with cpu/memory/node hotplug?

[Yasunori Goto]

Hello.

> > Iam not sure what specifically you are looking for ...
> 
> I was looking for the short and sweet overview such as you just
> presented ;).  Thanks.
> 
> I need to start following this hotplug work because it is next door to
> the cpu/memory placement work that I am most focused on, such as
> cpusets, led by Simon Derr of Bull, sitting on top of sched_setaffinity
> and Andi Kleen's numa work.  A cpuset is a set of cpu and memory
> resources that several tasks share, perhaps exclusively (all other
> tasks kept out).
> 
> At some point, questions such as "what happens when a node is unplugged
> from a cpuset" will need to be answered.  And even earlier, "what happens
> when a cpu is unplugged that was in a tasks cpus_allowed, or a memory
> node unplugged, that was in a vma's list of allowed memory zones."
> 
> Or perhaps the answers to these questions are already known ??

At least, there aren't enough answer about memory hotplug yet
if there is a process that is using memory affinity.

But did you see Iwamoto-san's patch ?
This is infrastructure that contents of memory can 
migrate from a node to other node.
If remap_onepage() of his patch execute on same cpus of that process 
or 'node id' is specified, it will migrate to appriciate node.
I think it can be your help.

But, probably you might think many other works are necessary, right?

Thanks.

-- 
Yasunori Goto <ygoto at us.fujitsu.com>




-------------------------------------------------------
This SF.Net email is sponsored by: SourceForge.net Broadband
Sign-up now for SourceForge Broadband and get the fastest
6.0/768 connection for only $19.95/mo for the first 3 months!
http://ads.osdn.com/?ad_id%62&alloc_ida84&op=click
_______________________________________________
Linux-hotplug-devel mailing list  http://linux-hotplug.sourceforge.net
Linux-hotplug-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/linux-hotplug-devel

Re: [Lhns-devel] Re: Who's doing what with cpu/memory/node hotplug?

[Paul Jackson]

> But did you see Iwamoto-san's patch ? ... remap_onepage() ...

Thank-you for the pointer.  I will have to consider that
work when I get to the right point in my efforts.

-- 
                          I won't rest till it's the best ...
                          Programmer, Linux Scalability
                          Paul Jackson <pj@sgi.com> 1.650.933.1373


-------------------------------------------------------
This SF.Net email is sponsored by: SourceForge.net Broadband
Sign-up now for SourceForge Broadband and get the fastest
6.0/768 connection for only $19.95/mo for the first 3 months!
http://ads.osdn.com/?ad_id%62&alloc_ida84&op=click
_______________________________________________
Linux-hotplug-devel mailing list  http://linux-hotplug.sourceforge.net
Linux-hotplug-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/linux-hotplug-devel

Re: [Lhns-devel] Re: Who's doing what with cpu/memory/node hotplug?

[Dave Hansen]

On Thu, 2004-05-13 at 18:14, Keiichiro Tokunaga wrote:
> On Wed, 12 May 2004 23:50:58 -0700 Dave Hansen <haveblue@us.ibm.com> wrote:
> > Why not?  We already export the information about which PCI busses are
> > attached to which NUMA nodes.  How are I/O devices different from CPU or
> > memory?  
> 
> I have just one concern.
> How about a container device that contains IO devices only?
>   - Does it count as a NUMA node?
>   - Is it having a NUMA node ID?

I don't see any real reasons why we can't make I/O only nodes.  I know
that we've run into and fixed a few situations where a node came up with
either no cpus or no memory for various reasons before.  Does anybody
have any philosophical objections to having I/O only nodes?

In any case, I'd bet we'll need some form of this eventually.  On my
hardware,  It's probably possible to hotplug individual CPUs or sections
of memory in such a way that a node has no online cpus or memory.  So,
we'll probably have to handle those cases anyway.  

This seems familliar somehow.  I have the feeling it's been discussed
before.  Matt, does this ring a bell?

-- Dave



-------------------------------------------------------
This SF.Net email is sponsored by: SourceForge.net Broadband
Sign-up now for SourceForge Broadband and get the fastest
6.0/768 connection for only $19.95/mo for the first 3 months!
http://ads.osdn.com/?ad_id%62&alloc_ida84&op=click
_______________________________________________
Linux-hotplug-devel mailing list  http://linux-hotplug.sourceforge.net
Linux-hotplug-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/linux-hotplug-devel

Re: [Lhns-devel] Re: Who's doing what with cpu/memory/node hotplug?

[Jesse Barnes]

On Thursday, May 13, 2004 6:28 pm, Dave Hansen wrote:
> I don't see any real reasons why we can't make I/O only nodes.  I know
> that we've run into and fixed a few situations where a node came up with
> either no cpus or no memory for various reasons before.  Does anybody
> have any philosophical objections to having I/O only nodes?

Well, it's kind of odd conceptually (to me a node implies memory) but SGI has 
such nodes.  They're simply numalink<->pci bridges, and as such are 
individually addressable and have physical NUMA node ids.

> In any case, I'd bet we'll need some form of this eventually.  On my
> hardware,  It's probably possible to hotplug individual CPUs or sections
> of memory in such a way that a node has no online cpus or memory.  So,
> we'll probably have to handle those cases anyway.

Yep, I suppose so.

Jesse


-------------------------------------------------------
This SF.Net email is sponsored by: SourceForge.net Broadband
Sign-up now for SourceForge Broadband and get the fastest
6.0/768 connection for only $19.95/mo for the first 3 months!
http://ads.osdn.com/?ad_id%62&alloc_ida84&op=click
_______________________________________________
Linux-hotplug-devel mailing list  http://linux-hotplug.sourceforge.net
Linux-hotplug-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/linux-hotplug-devel

Re: [Lhns-devel] Re: Who's doing what with cpu/memory/node hotplug?

[Keiichiro Tokunaga]

On Thu, 13 May 2004 18:28:37 -0700
Dave Hansen <haveblue@us.ibm.com> wrote:

> On Thu, 2004-05-13 at 18:14, Keiichiro Tokunaga wrote:
> > On Wed, 12 May 2004 23:50:58 -0700 Dave Hansen <haveblue@us.ibm.com> wrote:
> > > Why not?  We already export the information about which PCI busses are
> > > attached to which NUMA nodes.  How are I/O devices different from CPU or
> > > memory?  
> > 
> > I have just one concern.
> > How about a container device that contains IO devices only?
> >   - Does it count as a NUMA node?
> >   - Is it having a NUMA node ID?
> 
> I don't see any real reasons why we can't make I/O only nodes.  I know
> that we've run into and fixed a few situations where a node came up with
> either no cpus or no memory for various reasons before.  Does anybody
> have any philosophical objections to having I/O only nodes?

I'm afraid I didn't know that the issue had discussed.  I just
wondered if it's possible or not.  It was more like a question
and I don't have any objection to having I/O only nodes.  Can
you direct me to the discussion so that I could follow you :)?

Thanks,
Kei


-------------------------------------------------------
This SF.Net email is sponsored by: SourceForge.net Broadband
Sign-up now for SourceForge Broadband and get the fastest
6.0/768 connection for only $19.95/mo for the first 3 months!
http://ads.osdn.com/?ad_id%62&alloc_ida84&op=click
_______________________________________________
Linux-hotplug-devel mailing list  http://linux-hotplug.sourceforge.net
Linux-hotplug-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/linux-hotplug-devel

Re: [Lhns-devel] Re: Who's doing what with cpu/memory/node hotplug?

[Dave Hansen]

On Thu, 2004-05-13 at 20:02, Keiichiro Tokunaga wrote:
> On Thu, 13 May 2004 18:28:37 -0700
> Dave Hansen <haveblue@us.ibm.com> wrote:
> > I don't see any real reasons why we can't make I/O only nodes.  I know
> > that we've run into and fixed a few situations where a node came up with
> > either no cpus or no memory for various reasons before.  Does anybody
> > have any philosophical objections to having I/O only nodes?
> 
> I'm afraid I didn't know that the issue had discussed.  I just
> wondered if it's possible or not.  It was more like a question
> and I don't have any objection to having I/O only nodes.  Can
> you direct me to the discussion so that I could follow you :)?

It's just a vague impression that the topic seems familliar.  I'd say:
if no one speaks too stronly about the whole topic, consider it an OK.
:)

-- Dave



-------------------------------------------------------
This SF.Net email is sponsored by: SourceForge.net Broadband
Sign-up now for SourceForge Broadband and get the fastest
6.0/768 connection for only $19.95/mo for the first 3 months!
http://ads.osdn.com/?ad_id%62&alloc_ida84&op=click
_______________________________________________
Linux-hotplug-devel mailing list  http://linux-hotplug.sourceforge.net
Linux-hotplug-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/linux-hotplug-devel

RE: [Lhns-devel] Re: Who's doing what with cpu/memory/node hotplug?

[Kenji Kaneshige]

Hi,

> > I have just one concern.
> > How about a container device that contains IO devices only?
> >   - Does it count as a NUMA node?
> >   - Is it having a NUMA node ID?
>
> Yes, it would count as NUMA node with a NUMA ID.
>
> So a NUMA node could have CPUs, memory, and I/O,
> where each of those may or may not be present.
>
> There could be a node without any CPU, memory, or I/O.
> An example would be an I/O only node with PCI devices
> that are hot plug.  The PCI devices may or may not be
> connected at any point in time.

I don't know about NUMA much, but I think NUMA node ID is based
on the SRAT at the boot time. But SRAT doesn't have entries for I/O
devices. How should we create the NUMA node ID for I/O nodes at the
boot time?

Thanks,
Kenji Kaneshige


> -----Original Message-----
> From: lhns-devel-admin@lists.sourceforge.net
> [mailto:lhns-devel-admin@lists.sourceforge.net]On Behalf Of Russ Anderson
> Sent: Friday, May 14, 2004 10:49 PM
> To: Keiichiro Tokunaga
> Cc: Dave Hansen; tokunaga.keiich@jp.fujitsu.com; pj@sgi.com;
> ashok.raj@intel.com; lhns-devel@lists.sourceforge.net;
> linux-hotplug-devel@lists.sourceforge.net; colpatch@us.ibm.com;
> linux-ia64@vger.kernel.org; greg@kroah.com; pfg@sgi.com
> Subject: [Lhns-devel] Re: Who's doing what with cpu/memory/node hotplug?
>
>
> Keiichiro Tokunaga wrote:
> > Dave Hansen <haveblue@us.ibm.com> wrote:
> > > On Wed, 2004-05-12 at 23:35, Keiichiro Tokunaga wrote:
> > > >
> > > > I'm rather focusing on ACPI based container device hotplug.
> > > > A scope of LHNS includes a container device that contains
> > > > IO devices.  I think such a devices couldn't be handled by NUMA
> > > > hotplug.
> > >
> > > Why not?  We already export the information about which PCI busses are
> > > attached to which NUMA nodes.  How are I/O devices different
> from CPU or
> > > memory?
> >
> > I have just one concern.
> > How about a container device that contains IO devices only?
> >   - Does it count as a NUMA node?
> >   - Is it having a NUMA node ID?
>
> Yes, it would count as NUMA node with a NUMA ID.
>
> So a NUMA node could have CPUs, memory, and I/O,
> where each of those may or may not be present.
>
> There could be a node without any CPU, memory, or I/O.
> An example would be an I/O only node with PCI devices
> that are hot plug.  The PCI devices may or may not be
> connected at any point in time.
>
> And Yes, SGI does have I/O only nodes (and memory only nodes).
>
> Thanks,
> --
> Russ Anderson, OS RAS/Partitioning Project Lead
> SGI - Silicon Graphics Inc          rja@sgi.com
>
>
> -------------------------------------------------------
> This SF.Net email is sponsored by: SourceForge.net Broadband
> Sign-up now for SourceForge Broadband and get the fastest
> 6.0/768 connection for only $19.95/mo for the first 3 months!
> http://ads.osdn.com/?ad_id%62&alloc_ida84&op=click
> _______________________________________________
> Lhns-devel mailing list
> Lhns-devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/lhns-devel



-------------------------------------------------------
This SF.Net email is sponsored by: SourceForge.net Broadband
Sign-up now for SourceForge Broadband and get the fastest
6.0/768 connection for only $19.95/mo for the first 3 months!
http://ads.osdn.com/?ad_id%62&alloc_ida84&op=click
_______________________________________________
Linux-hotplug-devel mailing list  http://linux-hotplug.sourceforge.net
Linux-hotplug-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/linux-hotplug-devel

Re: [Lhns-devel] Re: Who's doing what with cpu/memory/node hotplug?

[Dave Hansen]

On Thu, 2004-05-13 at 17:13, Yasunori Goto wrote:
> > At some point, questions such as "what happens when a node is unplugged
> > from a cpuset" will need to be answered.  And even earlier, "what happens
> > when a cpu is unplugged that was in a tasks cpus_allowed, or a memory
> > node unplugged, that was in a vma's list of allowed memory zones."
> > 
> > Or perhaps the answers to these questions are already known ??
> 
> At least, there aren't enough answer about memory hotplug yet
> if there is a process that is using memory affinity.

There's been a lot of talk of what to do with hotplug CPUs as well.  No
matter what, it's going to be really easy to notify whatever app you
want with the hotplug scripts whenever a memory or CPU event happens. 
Just place your app script in /etc/hotplug.d.

-- Dave



-------------------------------------------------------
This SF.Net email is sponsored by: SourceForge.net Broadband
Sign-up now for SourceForge Broadband and get the fastest
6.0/768 connection for only $19.95/mo for the first 3 months!
http://ads.osdn.com/?ad_id%62&alloc_ida84&op=click
_______________________________________________
Linux-hotplug-devel mailing list  http://linux-hotplug.sourceforge.net
Linux-hotplug-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/linux-hotplug-devel

Re: [Lhns-devel] Re: Who's doing what with cpu/memory/node hotplug?

[Matthias Fouquet-Lapar]

> There's been a lot of talk of what to do with hotplug CPUs as well.  No
> matter what, it's going to be really easy to notify whatever app you
> want with the hotplug scripts whenever a memory or CPU event happens. 
> Just place your app script in /etc/hotplug.d.

Just curious, but what's the underlying reason for providing CPU hotplug ?
If the background is taking a "bad" CPU offline, what kind of CPU failures
are expected to be covered ? 

Thanks

Matthias Fouquet-Lapar  Core Platform Software    mfl@sgi.com  VNET 521-8213
Principal Engineer      Silicon Graphics          Home Office (+33) 1 3047 4127



-------------------------------------------------------
This SF.Net email is sponsored by: SourceForge.net Broadband
Sign-up now for SourceForge Broadband and get the fastest
6.0/768 connection for only $19.95/mo for the first 3 months!
http://ads.osdn.com/?ad_id%62&alloc_ida84&op=click
_______________________________________________
Linux-hotplug-devel mailing list  http://linux-hotplug.sourceforge.net
Linux-hotplug-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/linux-hotplug-devel

Re: [Lhns-devel] Re: Who's doing what with cpu/memory/node hotplug?

[Dave Hansen]

On Mon, 2004-05-17 at 00:35, Matthias Fouquet-Lapar wrote:
> > There's been a lot of talk of what to do with hotplug CPUs as well.  No
> > matter what, it's going to be really easy to notify whatever app you
> > want with the hotplug scripts whenever a memory or CPU event happens. 
> > Just place your app script in /etc/hotplug.d.
> 
> Just curious, but what's the underlying reason for providing CPU hotplug ?
> If the background is taking a "bad" CPU offline, what kind of CPU failures
> are expected to be covered ? 

In the ppc64 case, there is some partitioning that can be done of a
large system.  CPU hotplug lets us balance cpu power between
partitions.  You can also buy a machine with 32 cpus, but only have 16
of them turned on. There's a switch you can flip to get more horsepower
only when you need it.  (for a price, of course)

There's a quick mention of failure recovery here:
http://www-1.ibm.com/servers/eserver/pseries/hardware/whitepapers/lpar_3.html

But, the story on ppc64 is pretty much that the firmware detects the
kinds of hardware failures that we handle, Linux is just notified of it
afterword.  

Remember, chips designs like the POWER4 tend to have a lot of cost and
design overhead in error correction and detection that are quite absent
from consumer type cpus.

-- Dave



-------------------------------------------------------
This SF.Net email is sponsored by: SourceForge.net Broadband
Sign-up now for SourceForge Broadband and get the fastest
6.0/768 connection for only $19.95/mo for the first 3 months!
http://ads.osdn.com/?ad_id%62&alloc_ida84&op=click
_______________________________________________
Linux-hotplug-devel mailing list  http://linux-hotplug.sourceforge.net
Linux-hotplug-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/linux-hotplug-devel

Re: [Lhns-devel] Re: Who's doing what with cpu/memory/node hotplug?

[Matthias Fouquet-Lapar]

> In the ppc64 case, there is some partitioning that can be done of a
> large system.  CPU hotplug lets us balance cpu power between
> partitions.  You can also buy a machine with 32 cpus, but only have 16
> of them turned on. There's a switch you can flip to get more horsepower
> only when you need it.  (for a price, of course)
> 
> There's a quick mention of failure recovery here:
> http://www-1.ibm.com/servers/eserver/pseries/hardware/whitepapers/lpar_3.html
> 
> But, the story on ppc64 is pretty much that the firmware detects the
> kinds of hardware failures that we handle, Linux is just notified of it
> afterword.  

I think I'm not quite sure why you need hotplug to balance CPUs between
partitons. (You are not moving physical components for this, right ?)

> Remember, chips designs like the POWER4 tend to have a lot of cost and
> design overhead in error correction and detection that are quite absent
> from consumer type cpus.

Taken beside which CPU architecture has best the RAS features :-), which are
CPU error types on ppc64 (besides cache SBE etc) which allow gracefully
recovery (i.e. the OS continues operating and non-affected user processes
continue to run) and hotplug replacement 

Thanks

Matthias Fouquet-Lapar  Core Platform Software    mfl@sgi.com  VNET 521-8213
Principal Engineer      Silicon Graphics          Home Office (+33) 1 3047 4127



-------------------------------------------------------
This SF.Net email is sponsored by: SourceForge.net Broadband
Sign-up now for SourceForge Broadband and get the fastest
6.0/768 connection for only $19.95/mo for the first 3 months!
http://ads.osdn.com/?ad_id%62&alloc_ida84&op=click
_______________________________________________
Linux-hotplug-devel mailing list  http://linux-hotplug.sourceforge.net
Linux-hotplug-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/linux-hotplug-devel

Re: [Lhns-devel] Re: Who's doing what with cpu/memory/node hotplug?

[Grant Grundler]

On Mon, May 17, 2004 at 10:38:49AM +0200, Matthias Fouquet-Lapar wrote:
> I think I'm not quite sure why you need hotplug to balance CPUs between
> partitons. (You are not moving physical components for this, right ?)

It can look like it. Eg moving a "cell" on a superdome from
one hard partition to another completely dissassociates the 
"cell" from the previous partition as if it had been physically
removed.

grant


-------------------------------------------------------
This SF.Net email is sponsored by: SourceForge.net Broadband
Sign-up now for SourceForge Broadband and get the fastest
6.0/768 connection for only $19.95/mo for the first 3 months!
http://ads.osdn.com/?ad_id%62&alloc_ida84&op=click
_______________________________________________
Linux-hotplug-devel mailing list  http://linux-hotplug.sourceforge.net
Linux-hotplug-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/linux-hotplug-devel

Re: [Lhns-devel] Re: Who's doing what with cpu/memory/node hotplug?

[Dave Hansen]

On Mon, 2004-05-17 at 01:38, Matthias Fouquet-Lapar wrote:
> I think I'm not quite sure why you need hotplug to balance CPUs between
> partitons. (You are not moving physical components for this, right ?)

Right, there is no movement of physical components. But, from Linux's
perspective, it really doesn't matter.  There is a cpu that I'm using,
it won't be there in a minute, I better stop using it.  That means that
Linux has to be careful with things like the for_each_cpu().   Take a
look at the calls to hotcpu_notifier() for more things that we keep an
eye on when CPUs go down.

That's pretty much what the CPU hotplug code does, and *that* doesn't
change whether the CPUs are real or more virtualized.

> > Remember, chips designs like the POWER4 tend to have a lot of cost and
> > design overhead in error correction and detection that are quite absent
> > from consumer type cpus.
> 
> Taken beside which CPU architecture has best the RAS features :-), which are
> CPU error types on ppc64 (besides cache SBE etc) which allow gracefully
> recovery (i.e. the OS continues operating and non-affected user processes
> continue to run) and hotplug replacement 

Sorry, I'm just a software guy.  No one tells me such things :)  

Rusty, have I butchered explaining CPU hotplug enough for you to step
in, yet?

-- Dave



-------------------------------------------------------
This SF.Net email is sponsored by: SourceForge.net Broadband
Sign-up now for SourceForge Broadband and get the fastest
6.0/768 connection for only $19.95/mo for the first 3 months!
http://ads.osdn.com/?ad_id%62&alloc_ida84&op=click
_______________________________________________
Linux-hotplug-devel mailing list  http://linux-hotplug.sourceforge.net
Linux-hotplug-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/linux-hotplug-devel

Re: [Lhns-devel] Re: Who's doing what with cpu/memory/node hotplug?

[Ashok Raj]

On Mon, May 17, 2004 at 10:38:49AM +0200, Matthias Fouquet-Lapar wrote:
> > But, the story on ppc64 is pretty much that the firmware detects the
> > kinds of hardware failures that we handle, Linux is just notified of it
> > afterword.  
> 
> I think I'm not quite sure why you need hotplug to balance CPUs between
> partitons. (You are not moving physical components for this, right ?)

Don't always equate HOTPLUG with physical removal/addition.

In the above case of balancing between partitions, its a still a hotplug
wrt the kernel, where the cpu is removed from os image (say scheduler etc)
and reintroduce the new resource to another running partition.

Physical removal/addition requires one extra step in addition to what is done
currently, i.e firmware interaction with the OS to handle device arrival and
eject notifications... these mechanisms are often platoform independent, one
example of such a mechanism would be ACPI.

Cheers,
ashok


-------------------------------------------------------
This SF.Net email is sponsored by: SourceForge.net Broadband
Sign-up now for SourceForge Broadband and get the fastest
6.0/768 connection for only $19.95/mo for the first 3 months!
http://ads.osdn.com/?ad_id%62&alloc_ida84&op=click
_______________________________________________
Linux-hotplug-devel mailing list  http://linux-hotplug.sourceforge.net
Linux-hotplug-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/linux-hotplug-devel

Re: [Lhns-devel] Re: Who's doing what with cpu/memory/node hotplug?

[Matthias Fouquet-Lapar]

> Don't always equate HOTPLUG with physical removal/addition.
> 
> In the above case of balancing between partitions, its a still a hotplug
> wrt the kernel, where the cpu is removed from os image (say scheduler etc)
> and reintroduce the new resource to another running partition.

Thanks, this make sense. I was previously working on systems which allowed
real physical hot swap, i.e. physical replacement of CPU, memory etc without
OS reboot, hence my questions.

[ Just as a site-note, I always wondered if hotplug would be a useful feature
  for larger configurations to reduce boot-time. For example, on a 64P system 
  you start the kernel on some "root" CPU with a relativly small amount of
  memory, so it should boot relativly fast. While the system is going into
  the desired run-level, hotplug/add now adds in parallel all available CPUs,
  main memory, I/O etc. The idea is that you probably don't need 64P and
  1 Terabyte to start system services
]


Matthias Fouquet-Lapar  Core Platform Software    mfl@sgi.com  VNET 521-8213
Principal Engineer      Silicon Graphics          Home Office (+33) 1 3047 4127



-------------------------------------------------------
This SF.Net email is sponsored by: SourceForge.net Broadband
Sign-up now for SourceForge Broadband and get the fastest
6.0/768 connection for only $19.95/mo for the first 3 months!
http://ads.osdn.com/?ad_id%62&alloc_ida84&op=click
_______________________________________________
Linux-hotplug-devel mailing list  http://linux-hotplug.sourceforge.net
Linux-hotplug-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/linux-hotplug-devel

RE: [Lhns-devel] Re: Who's doing what with cpu/memory/node hotplug?

[Matthew Dobson]

On Sun, 2004-05-16 at 19:34, Kenji Kaneshige wrote:
> Hi,
> 
> > > I have just one concern.
> > > How about a container device that contains IO devices only?
> > >   - Does it count as a NUMA node?
> > >   - Is it having a NUMA node ID?
> >
> > Yes, it would count as NUMA node with a NUMA ID.
> >
> > So a NUMA node could have CPUs, memory, and I/O,
> > where each of those may or may not be present.
> >
> > There could be a node without any CPU, memory, or I/O.
> > An example would be an I/O only node with PCI devices
> > that are hot plug.  The PCI devices may or may not be
> > connected at any point in time.
> 
> I don't know about NUMA much, but I think NUMA node ID is based
> on the SRAT at the boot time. But SRAT doesn't have entries for I/O
> devices. How should we create the NUMA node ID for I/O nodes at the
> boot time?
> 
> Thanks,
> Kenji Kaneshige

The NUMA node ID *can* be based on the SRAT, but doesn't *have* to be. 
It's really up to each individual architecture how they want to number
their nodes.  Or at least that's the way things should work.  That was a
large part of the motivation behind the nodemask patches I submitted a
while ago.  We would like individual architectures to be able to number
their nodes in any manner they deem appropriate, as long as 0 <= nid <
MAX_NUMNODES.  Currently there is an additional requirement (that the
nodemask code would eliminate) that nodes must be numbered sequentially
without any gaps in the numbering, ie: 0 <= nid < numnodes.  There is
really no good reason that this has to hold, so we should get rid of it.

We want macros like for_each_node_with_cpus(),
for_each_node_with_memory(), etc, that people can use to iterate over
the type of nodes they care about.

I'd look at SGI's IA64 code for examples of how to create NUMA nodes
with only I/O on them.

-Matt



-------------------------------------------------------
This SF.Net email is sponsored by: SourceForge.net Broadband
Sign-up now for SourceForge Broadband and get the fastest
6.0/768 connection for only $19.95/mo for the first 3 months!
http://ads.osdn.com/?ad_id%62&alloc_ida84&op=click
_______________________________________________
Linux-hotplug-devel mailing list  http://linux-hotplug.sourceforge.net
Linux-hotplug-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/linux-hotplug-devel

RE: [Lhns-devel] Re: Who's doing what with cpu/memory/node hotplug?

[Luck, Tony]

>I'd look at SGI's IA64 code for examples of how to create NUMA nodes
>with only I/O on them.

The 'M' in NUMA stands for 'Memory' ... so the pedant in me says
that these I/O only nodes aren't NUMA nodes, perhaps NUIOA?

Is anyone doing anything to optimize access to such beasts?  Binding
processes that are accessing such I/O devices to nearby nodes ...
sounds possible for NIC, and ugly for disks.

-Tony


-------------------------------------------------------
This SF.Net email is sponsored by: SourceForge.net Broadband
Sign-up now for SourceForge Broadband and get the fastest
6.0/768 connection for only $19.95/mo for the first 3 months!
http://ads.osdn.com/?ad_id%62&alloc_ida84&opÌk
_______________________________________________
Linux-hotplug-devel mailing list  http://linux-hotplug.sourceforge.net
Linux-hotplug-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/linux-hotplug-devel

RE: [Lhns-devel] Re: Who's doing what with cpu/memory/node hotplug?

[Dave Hansen]

On Mon, 2004-05-17 at 16:18, Luck, Tony wrote:
> >I'd look at SGI's IA64 code for examples of how to create NUMA nodes
> >with only I/O on them.
> 
> The 'M' in NUMA stands for 'Memory' ... so the pedant in me says
> that these I/O only nodes aren't NUMA nodes, perhaps NUIOA?

And the 'A' is Access (or architecture I guess).  Those I/O nodes'
access to memory is non-uniform :)

> Is anyone doing anything to optimize access to such beasts?  Binding
> processes that are accessing such I/O devices to nearby nodes ...
> sounds possible for NIC, and ugly for disks.

Well, binding isn't the best thing to do, but simple preference and
error recovery would be great.  The NUMAQ (and I'm sure others) used to
have a few fiber channel cards in each node that allowed multiple paths
to I/O devices.  You could be fast, and recover from errors since
everything was multiply connected.  We do this on some drivers today,
but not everything.  

-- Dave



-------------------------------------------------------
This SF.Net email is sponsored by: SourceForge.net Broadband
Sign-up now for SourceForge Broadband and get the fastest
6.0/768 connection for only $19.95/mo for the first 3 months!
http://ads.osdn.com/?ad_id%62&alloc_ida84&op=click
_______________________________________________
Linux-hotplug-devel mailing list  http://linux-hotplug.sourceforge.net
Linux-hotplug-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/linux-hotplug-devel

Re: [Lhns-devel] Re: Who's doing what with cpu/memory/node hotplug?

[Jesse Barnes]

On Monday, May 17, 2004 4:28 pm, Dave Hansen wrote:
> > Is anyone doing anything to optimize access to such beasts?  Binding
> > processes that are accessing such I/O devices to nearby nodes ...
> > sounds possible for NIC, and ugly for disks.
>
> Well, binding isn't the best thing to do, but simple preference and
> error recovery would be great.  The NUMAQ (and I'm sure others) used to
> have a few fiber channel cards in each node that allowed multiple paths
> to I/O devices.  You could be fast, and recover from errors since
> everything was multiply connected.  We do this on some drivers today,
> but not everything.

We do a few more things on Altix.  IRQs are already routed to close CPUs, and 
I have patches to allocate DMA memory from the node closest to a given 
devices.  Drivers can use pci_to_nodemask (currently pcibus_to_cpumask) to 
get a list of nodes close to their device, and there are patches floating 
about to link PCI busses to nodes in /sys, which would allow user level 
applications to intelligently place themselves.

Jesse


-------------------------------------------------------
This SF.Net email is sponsored by: SourceForge.net Broadband
Sign-up now for SourceForge Broadband and get the fastest
6.0/768 connection for only $19.95/mo for the first 3 months!
http://ads.osdn.com/?ad_id%62&alloc_ida84&op=click
_______________________________________________
Linux-hotplug-devel mailing list  http://linux-hotplug.sourceforge.net
Linux-hotplug-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/linux-hotplug-devel

Re: [Lhns-devel] Re: Who's doing what with cpu/memory/node hotplug?

[Grant Grundler]

On Mon, May 17, 2004 at 09:15:50PM +0200, Matthias Fouquet-Lapar wrote:
...
> I always wondered if hotplug would be a useful feature
>   for larger configurations to reduce boot-time.

yes. definitely. We also considering this for parisc port just to
simplify the boot code.

grant


-------------------------------------------------------
This SF.Net email is sponsored by: SourceForge.net Broadband
Sign-up now for SourceForge Broadband and get the fastest
6.0/768 connection for only $19.95/mo for the first 3 months!
http://ads.osdn.com/?ad_id%62&alloc_ida84&op=click
_______________________________________________
Linux-hotplug-devel mailing list  http://linux-hotplug.sourceforge.net
Linux-hotplug-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/linux-hotplug-devel

Re: [Lhns-devel] Re: Who's doing what with cpu/memory/node hotplug?

[Russ Anderson]

Matthias Fouquet-Lapar wrote:
> 
> [ Just as a site-note, I always wondered if hotplug would be a useful feature
>   for larger configurations to reduce boot-time. For example, on a 64P system 
>   you start the kernel on some "root" CPU with a relativly small amount of
>   memory, so it should boot relativly fast. While the system is going into
>   the desired run-level, hotplug/add now adds in parallel all available CPUs,
>   main memory, I/O etc. The idea is that you probably don't need 64P and
>   1 Terabyte to start system services
> ]

I can certainly think of cases (customers) where minimizing the total
system down time is a huge issue.  Getting a subset of the system
back into production as quickly as possible would be very desirable.

Another advantage of that type boot process is that it would catch
any bugs (or regressions) in the hot add code.  At a minumum, it 
would be a very good test of the hot add functionality.

I think a boot option to allow that type of booting process
would be a good idea.

-- 
Russ Anderson, OS RAS/Partitioning Project Lead  
SGI - Silicon Graphics Inc          rja@sgi.com


-------------------------------------------------------
This SF.Net email is sponsored by: SourceForge.net Broadband
Sign-up now for SourceForge Broadband and get the fastest
6.0/768 connection for only $19.95/mo for the first 3 months!
http://ads.osdn.com/?ad_id%62&alloc_ida84&op=click
_______________________________________________
Linux-hotplug-devel mailing list  http://linux-hotplug.sourceforge.net
Linux-hotplug-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/linux-hotplug-devel

Re: [Lhns-devel] Re: Who's doing what with cpu/memory/node hotplug?

[Ashok Raj]

On Tue, May 18, 2004 at 09:58:38AM -0500, Russ Anderson wrote:
> Matthias Fouquet-Lapar wrote:
> > 
> > [ Just as a site-note, I always wondered if hotplug would be a useful feature
> >   for larger configurations to reduce boot-time. For example, on a 64P system 
> >   you start the kernel on some "root" CPU with a relativly small amount of
> >   memory, so it should boot relativly fast. While the system is going into
> >   the desired run-level, hotplug/add now adds in parallel all available CPUs,
> >   main memory, I/O etc. The idea is that you probably don't need 64P and
> >   1 Terabyte to start system services
> > ]
> 
> I can certainly think of cases (customers) where minimizing the total
> system down time is a huge issue.  Getting a subset of the system
> back into production as quickly as possible would be very desirable.
> 
> Another advantage of that type boot process is that it would catch
> any bugs (or regressions) in the hot add code.  At a minumum, it 
> would be a very good test of the hot add functionality.
> 
> I think a boot option to allow that type of booting process
> would be a good idea.

The boot already follows the same methodology, the boot cpu comes up, and the rest of the cpu's are brought up. for e.g. smp_init() does call cpu_up() which is the same path for hotplug.

Not sure if cpu_up is a time consuming process, i think if you have more cpu's
the startup could be faster as you have now more resources to complete the 
statup of kernel.

Often what i have seen is that the IO part of the probe is the most time
consuming process. Splitting the discovery of controllers and boot resources
first, and then deferring the rest if the io discovery process would 
speedup boot. Also its your firmware that would do the same work as OS 
in identifying all the controllers and disks upfront that contributes to 
slow boot.

bringup all cpu() and then defer part of the probe/discovery (especially
disks) in parallel would provide real benefit i think.

Cheers,
ashok


-------------------------------------------------------
This SF.Net email is sponsored by: SourceForge.net Broadband
Sign-up now for SourceForge Broadband and get the fastest
6.0/768 connection for only $19.95/mo for the first 3 months!
http://ads.osdn.com/?ad_id%62&alloc_ida84&op=click
_______________________________________________
Linux-hotplug-devel mailing list  http://linux-hotplug.sourceforge.net
Linux-hotplug-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/linux-hotplug-devel

Re: [Lhns-devel] Re: Who's doing what with cpu/memory/node hotplug?

[Jack Steiner]

On Tue, May 18, 2004 at 01:16:30PM -0700, Ashok Raj wrote:
> On Tue, May 18, 2004 at 09:58:38AM -0500, Russ Anderson wrote:
> > Matthias Fouquet-Lapar wrote:
> > > 
> > > [ Just as a site-note, I always wondered if hotplug would be a useful feature
> > >   for larger configurations to reduce boot-time. For example, on a 64P system 
> > >   you start the kernel on some "root" CPU with a relativly small amount of
> > >   memory, so it should boot relativly fast. While the system is going into
> > >   the desired run-level, hotplug/add now adds in parallel all available CPUs,
> > >   main memory, I/O etc. The idea is that you probably don't need 64P and
> > >   1 Terabyte to start system services
> > > ]
> > 
> > I can certainly think of cases (customers) where minimizing the total
> > system down time is a huge issue.  Getting a subset of the system
> > back into production as quickly as possible would be very desirable.
> > 
> > Another advantage of that type boot process is that it would catch
> > any bugs (or regressions) in the hot add code.  At a minumum, it 
> > would be a very good test of the hot add functionality.
> > 
> > I think a boot option to allow that type of booting process
> > would be a good idea.
> 
> The boot already follows the same methodology, the boot cpu comes up, and the rest of the cpu's are brought up. for e.g. smp_init() does call cpu_up() which is the same path for hotplug.
> 
> Not sure if cpu_up is a time consuming process, i think if you have more cpu's
> the startup could be faster as you have now more resources to complete the 
> statup of kernel.
> 
> Often what i have seen is that the IO part of the probe is the most time
> consuming process. Splitting the discovery of controllers and boot resources
> first, and then deferring the rest if the io discovery process would 
> speedup boot. Also its your firmware that would do the same work as OS 
> in identifying all the controllers and disks upfront that contributes to 
> slow boot.
> 
> bringup all cpu() and then defer part of the probe/discovery (especially
> disks) in parallel would provide real benefit i think.

I agree. 

On the large SGI systems, starting all cpus is fairly fast. The time
consuming parts of boot are:
	- probing disks (depends, of course, on the number)
	- VM init (mem_init, free_bootmem_core)
	- initializing memory structures (arch_memmap_init)

By far, probing is the slowest part of boot on systems with a lot of
disks.



> 
> Cheers,
> ashok
> -
> To unsubscribe from this list: send the line "unsubscribe linux-ia64" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

-- 
Thanks

Jack Steiner (steiner@sgi.com)          651-683-5302
Principal Engineer                      SGI - Silicon Graphics, Inc.




-------------------------------------------------------
This SF.Net email is sponsored by: SourceForge.net Broadband
Sign-up now for SourceForge Broadband and get the fastest
6.0/768 connection for only $19.95/mo for the first 3 months!
http://ads.osdn.com/?ad_id%62&alloc_ida84&op=click
_______________________________________________
Linux-hotplug-devel mailing list  http://linux-hotplug.sourceforge.net
Linux-hotplug-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/linux-hotplug-devel

Re: [Lhns-devel] Re: Who's doing what with cpu/memory/node hotplug?

[Andi Kleen]

> On the large SGI systems, starting all cpus is fairly fast. The time
> consuming parts of boot are:
> 	- probing disks (depends, of course, on the number)
> 	- VM init (mem_init, free_bootmem_core)
> 	- initializing memory structures (arch_memmap_init)
> 
> By far, probing is the slowest part of boot on systems with a lot of
> disks.

We should really have parallel scanning for disks ...
That would help on smaller system too.

-Andi


-------------------------------------------------------
This SF.Net email is sponsored by: SourceForge.net Broadband
Sign-up now for SourceForge Broadband and get the fastest
6.0/768 connection for only $19.95/mo for the first 3 months!
http://ads.osdn.com/?ad_id%62&alloc_ida84&op=click
_______________________________________________
Linux-hotplug-devel mailing list  http://linux-hotplug.sourceforge.net
Linux-hotplug-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/linux-hotplug-devel

Re: [Lhns-devel] Re: Who's doing what with cpu/memory/node hotplug?

[Greg KH]

On Tue, May 18, 2004 at 11:05:01PM +0200, Andi Kleen wrote:
> 
> We should really have parallel scanning for disks ...
> That would help on smaller system too.

Will happen in 2.7, I've been saying it for over a year...

greg k-h


-------------------------------------------------------
This SF.Net email is sponsored by: SourceForge.net Broadband
Sign-up now for SourceForge Broadband and get the fastest
6.0/768 connection for only $19.95/mo for the first 3 months!
http://ads.osdn.com/?ad_id%62&alloc_ida84&op=click
_______________________________________________
Linux-hotplug-devel mailing list  http://linux-hotplug.sourceforge.net
Linux-hotplug-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/linux-hotplug-devel

Re: [Lhns-devel] Re: Who's doing what with cpu/memory/node hotplug?

[Matthias Fouquet-Lapar]

> The boot already follows the same methodology, the boot cpu comes up, and the rest of the cpu's are brought up. for e.g. smp_init() does call cpu_up() which is the same path for hotplug.

My point was that if it would make sense to get one CPU with a mininmal
amount of memory through the entire boot sequence, so it can start system 
daemons etc. Depending on the configuration, this typically takes some time. 

While this is happening, HW resources can be added on the fly using hotplug

> Not sure if cpu_up is a time consuming process, i think if you have more cpu's
> the startup could be faster as you have now more resources to complete the 
> statup of kernel.

Again my suggestion was to complete the boot of the kernel with a minimal
set of resources (TBD). To me it's seems like overkill to have all CPUs
and main memory ready before you an start sendmail, printer daemon etc.

Of course, you might want to have I/O ready before mounting disk drives

> bringup all cpu() and then defer part of the probe/discovery (especially
> disks) in parallel would provide real benefit i think.

I would take it one step farther in that the system already can go for
example to multi-user before all HW resources are attached

Thanks

Matthias Fouquet-Lapar  Core Platform Software    mfl@sgi.com  VNET 521-8213
Principal Engineer      Silicon Graphics          Home Office (+33) 1 3047 4127



-------------------------------------------------------
This SF.Net email is sponsored by: Oracle 10g
Get certified on the hottest thing ever to hit the market... Oracle 10g. 
Take an Oracle 10g class now, and we'll give you the exam FREE.
http://ads.osdn.com/?ad_id149&alloc_idÅ66&op=click
_______________________________________________
Linux-hotplug-devel mailing list  http://linux-hotplug.sourceforge.net
Linux-hotplug-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/linux-hotplug-devel

Re: [Lhns-devel] Re: Who's doing what with cpu/memory/node hotplug?

[Paul Jackson]

> My point was that if it would make sense to get one CPU with a mininmal
> amount of memory through the entire boot sequence, so it can start system
> daemons ...

The placement and initialization of system daemons on large systems
sometimes depends on the system configuration.  Besides the obvious
set of daemon kernel threads that are established one per cpu or one
per node, there may be an intentional placement of classic Unix daemons
to run on a subset of the available cpus or memory, in order to free
up the rest of the system for dedicated application use.

Also, I suspect that certain critical system resources (kernel arrays)
are sized according to how many cpus or how much memory or such is
available.

Such placement and sizing can adapt to a few cpus and nodes coming and
going later on in the life of that system boot, but having what looks
like a one-lung system turn into a 512 cpu monster sometime after init
has gone multi-user could result in considerable misconfiguraiton.

-- 
                          I won't rest till it's the best ...
                          Programmer, Linux Scalability
                          Paul Jackson <pj@sgi.com> 1.650.933.1373


-------------------------------------------------------
This SF.Net email is sponsored by: Oracle 10g
Get certified on the hottest thing ever to hit the market... Oracle 10g. 
Take an Oracle 10g class now, and we'll give you the exam FREE.
http://ads.osdn.com/?ad_id149&alloc_idÅ66&op=click
_______________________________________________
Linux-hotplug-devel mailing list  http://linux-hotplug.sourceforge.net
Linux-hotplug-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/linux-hotplug-devel

Re: [Lhns-devel] Re: Who's doing what with cpu/memory/node hotplug?

[Matthias Fouquet-Lapar]

> Also, I suspect that certain critical system resources (kernel arrays)
> are sized according to how many cpus or how much memory or such is
> available.
> 
> Such placement and sizing can adapt to a few cpus and nodes coming and
> going later on in the life of that system boot, but having what looks
> like a one-lung system turn into a 512 cpu monster sometime after init
> has gone multi-user could result in considerable misconfiguraiton.

I also wondered about this. But doesn't a true hotplug implementation
used for partitions (or cell) reconfiguration require this flexibility
for "on-the-fly scaling" ?

Thanks

Matthias Fouquet-Lapar  Core Platform Software    mfl@sgi.com  VNET 521-8213
Principal Engineer      Silicon Graphics          Home Office (+33) 1 3047 4127



-------------------------------------------------------
This SF.Net email is sponsored by: Oracle 10g
Get certified on the hottest thing ever to hit the market... Oracle 10g. 
Take an Oracle 10g class now, and we'll give you the exam FREE.
http://ads.osdn.com/?ad_id149&alloc_idÅ66&op=click
_______________________________________________
Linux-hotplug-devel mailing list  http://linux-hotplug.sourceforge.net
Linux-hotplug-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/linux-hotplug-devel

Re: [Lhns-devel] Re: Who's doing what with cpu/memory/node hotplug?

[Paul Jackson]

Matthias wrote:
> But doesn't a true hotplug implementation ... require ... "on-the-fly scaling" ?

You'd have to ask the hotplug folks that.  I'd imagine that they will
have some work to do before they can go from one to 512 cpus in a single
partition boot and still keep a sensible configuration.

-- 
                          I won't rest till it's the best ...
                          Programmer, Linux Scalability
                          Paul Jackson <pj@sgi.com> 1.650.933.1373


-------------------------------------------------------
This SF.Net email is sponsored by: Oracle 10g
Get certified on the hottest thing ever to hit the market... Oracle 10g. 
Take an Oracle 10g class now, and we'll give you the exam FREE.
http://ads.osdn.com/?ad_id149&alloc_idÅ66&op=click
_______________________________________________
Linux-hotplug-devel mailing list  http://linux-hotplug.sourceforge.net
Linux-hotplug-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/linux-hotplug-devel

RE: [Lhns-devel] Re: Who's doing what with cpu/memory/node hotplug?

[Howell, David P]

Seems like to do this right you would need a dependence based init 
script implementation, i.e. do step D when steps A, B, and C have 
successfully completed. This would permit more granular bringup of 
services when the hardware/driver initialization has completed.

Thanks,
Dave Howell

These are my opinions and not official opinions of Intel Corp.
 
David Howell
Intel Corporation
Telco Server Development
Server Products Division
Voice: (803) 216-2359  Fax: (803) 216-2178
 
Intel Corporation
Columbia Design Center, CBA-1
100 Center Point Circle, Suite 210
Columbia, SC 29210
 
david.p.howell@intel.com
 

-----Original Message-----
From: linux-ia64-owner@vger.kernel.org
[mailto:linux-ia64-owner@vger.kernel.org] On Behalf Of Matthias
Fouquet-Lapar
Sent: Wednesday, May 19, 2004 1:25 AM
To: Raj, Ashok
Cc: Russ Anderson; Matthias Fouquet-Lapar; Raj, Ashok; Dave Hansen;
Yasunori Goto; Paul Jackson; lhns-devel@lists.sourceforge.net; hotplug
devel; colpatch@us.ibm.com; linux-ia64@vger.kernel.org; Greg KH;
tokunaga.keiich@jp.fujitsu.com; pfg@SGI.COM; IWAMOTO Toshihiro
Subject: Re: [Lhns-devel] Re: Who's doing what with cpu/memory/node
hotplug?

> The boot already follows the same methodology, the boot cpu comes up,
and the rest of the cpu's are brought up. for e.g. smp_init() does call
cpu_up() which is the same path for hotplug.

My point was that if it would make sense to get one CPU with a mininmal
amount of memory through the entire boot sequence, so it can start
system 
daemons etc. Depending on the configuration, this typically takes some
time. 

While this is happening, HW resources can be added on the fly using
hotplug

> Not sure if cpu_up is a time consuming process, i think if you have
more cpu's
> the startup could be faster as you have now more resources to complete
the 
> statup of kernel.

Again my suggestion was to complete the boot of the kernel with a
minimal
set of resources (TBD). To me it's seems like overkill to have all CPUs
and main memory ready before you an start sendmail, printer daemon etc.

Of course, you might want to have I/O ready before mounting disk drives

> bringup all cpu() and then defer part of the probe/discovery
(especially
> disks) in parallel would provide real benefit i think.

I would take it one step farther in that the system already can go for
example to multi-user before all HW resources are attached

Thanks

Matthias Fouquet-Lapar  Core Platform Software    mfl@sgi.com  VNET
521-8213
Principal Engineer      Silicon Graphics          Home Office (+33) 1
3047 4127

-
To unsubscribe from this list: send the line "unsubscribe linux-ia64" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html


-------------------------------------------------------
This SF.Net email is sponsored by: Oracle 10g
Get certified on the hottest thing ever to hit the market... Oracle 10g. 
Take an Oracle 10g class now, and we'll give you the exam FREE.
http://ads.osdn.com/?ad_id149&alloc_idÅ66&opÃk
_______________________________________________
Linux-hotplug-devel mailing list  http://linux-hotplug.sourceforge.net
Linux-hotplug-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/linux-hotplug-devel

Re: [Lhns-devel] Re: Who's doing what with cpu/memory/node hotplug?

[Matthias Fouquet-Lapar]

> Seems like to do this right you would need a dependence based init 
> script implementation, i.e. do step D when steps A, B, and C have 
> successfully completed. This would permit more granular bringup of 
> services when the hardware/driver initialization has completed.

Probably something like this, although I'm not sure if it would need
to be hierarchically. Maybe split into device categories, disk, network
etc. The goal to keep in mind is the minimization of customer visible
down-time either due to a system crash (HW/SW), hotplug configuration 
changes, power outage etc

Or let's say you need to take a system dump. Maybe we can dump the
first 128 MB within a few seconds and then start rebooting the minimal
configuration. System dumping continues and while memory is saved for
post-mortem dump analysis, saved memory pages are added hot-plug 
back into operation.  (or maybe on a DIMM level to make life easier)

Thanks

Matthias Fouquet-Lapar  Core Platform Software    mfl@sgi.com  VNET 521-8213
Principal Engineer      Silicon Graphics          Home Office (+33) 1 3047 4127



-------------------------------------------------------
This SF.Net email is sponsored by: Oracle 10g
Get certified on the hottest thing ever to hit the market... Oracle 10g. 
Take an Oracle 10g class now, and we'll give you the exam FREE.
http://ads.osdn.com/?ad_id149&alloc_idÅ66&op=click
_______________________________________________
Linux-hotplug-devel mailing list  http://linux-hotplug.sourceforge.net
Linux-hotplug-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/linux-hotplug-devel