Hotplug CPU Boot Changes: BEWARE

Hotplug CPU Boot Changes: BEWARE

[Rusty Russell]

Hi all (esp port maintainers),

	In writing the hotplug CPU stuff, Linus asked me to alter the
boot sequence to "plug in" CPUs.  I am shortly going to be sending
these patches to him now I have got my x86 box to boot with the
changes.

The changes are as follows:
1) Non-linear CPU support.  No more number/logical map, or
   smp_num_cpus.  This is easy to change over to.

2) The boot sequence used to be:
	smp_boot_cpus()
	smp_commence()
	start idle thread, do initfuncs...

   Now it is:
	start idle thread
	smp_prepare_cpus(int max_cpus)
		=> Probes for cpus, sets up cpu_possible() macro to work.
	do initfuncs
	For each cpu for which cpu_possible(cpu) is true:
		cpu_up(int cpunum)
	smp_cpus_done(max_cpus);

There are two ways to transition: one is to do the minimal hacks so
that the new boot code works (as per my x86 patch).  The other is to
take into account that the next stage (optional by arch) is to
actually bring cpus up and down on the fly, and hence actually write
code that will work after boot as well (as per my ppc patch).

For the patches, see:
	http://www.kernel.org/pub/linux/kernel/people/rusty

Rusty.	
--
  Anyone who quotes me in their sig is an idiot. -- Rusty Russell.

Re: Hotplug CPU Boot Changes: BEWARE

[Eric W. Biederman]

Rusty Russell <rusty@rustcorp.com.au> writes:

> Hi all (esp port maintainers),
> 
> 	In writing the hotplug CPU stuff, Linus asked me to alter the
> boot sequence to "plug in" CPUs.  I am shortly going to be sending
> these patches to him now I have got my x86 box to boot with the
> changes.

If to the general SMP case is added the ability to dynamically enable
and disable cpus at runtime, this infrastructure work appears to have
general applicability now.  Allowing for example dynamic
enable/disable of HT on P4-Xeons at runtime for example.


> There are two ways to transition: one is to do the minimal hacks so
> that the new boot code works (as per my x86 patch).  The other is to
> take into account that the next stage (optional by arch) is to
> actually bring cpus up and down on the fly, and hence actually write
> code that will work after boot as well (as per my ppc patch).

Thinking in terms of physically hot-plugging cpus has me doubt the
actual utility of this code.  Instead thinking of dynamically enabling
and disabling processors for debugging sounds very reasonable.

But for the latter something just a little more than minimal hacks
must be implemented.  But dynamic cpu enable/disable is definitely
worth it.

Eric

Re: Hotplug CPU Boot Changes: BEWARE

[Pavel Machek]

Hi!

> > 	In writing the hotplug CPU stuff, Linus asked me to alter the
> > boot sequence to "plug in" CPUs.  I am shortly going to be sending
> > these patches to him now I have got my x86 box to boot with the
> > changes.
> 
> If to the general SMP case is added the ability to dynamically enable
> and disable cpus at runtime, this infrastructure work appears to have
> general applicability now.  Allowing for example dynamic
> enable/disable of HT on P4-Xeons at runtime for example.

Well, this way we can get suspend-to-{RAM,disk} for SMP... Offline all
other CPUS, do suspend, put them back online.
								Pavel
-- 
Philips Velo 1: 1"x4"x8", 300gram, 60, 12MB, 40bogomips, linux, mutt,
details at http://atrey.karlin.mff.cuni.cz/~pavel/velo/index.html.

Re: Hotplug CPU Boot Changes: BEWARE

[Keith Owens]

On 07 Jun 2002 08:51:32 -0600, 
ebiederm@xmission.com (Eric W. Biederman) wrote:
>Thinking in terms of physically hot-plugging cpus has me doubt the
>actual utility of this code.  Instead thinking of dynamically enabling
>and disabling processors for debugging sounds very reasonable.

IBM (S390) and SGI hardware.  Virtually disable the cpu so the cpu
partition manager migrates work off the cpu[*].  Vary the cpu offline.
Physically replace the cpu.  Reverse the process.  It all works fine if
the hardware supports it and[*] the OS supports migration off a cpu.

Re: Hotplug CPU Boot Changes: BEWARE

[Rusty Russell]

In message <m1elfjw39n.fsf@frodo.biederman.org> you write:
> But for the latter something just a little more than minimal hacks
> must be implemented.  But dynamic cpu enable/disable is definitely
> worth it.

Perhaps I didn't make myself clear: hotplugging does not neccessarily
mean physically removing or adding the CPU.  And as to whether they
offer full support, or stub support, architectures can decide that for
themselves, as they need.  It's not my call.

I don't know how much of a win it is to disable HT on cpus, but I can
tell you that adding & subtracting CPUs is a fairly heavy-weight
operation in this design (I don't think we really want to lock around
every cpu iteration).

Rusty.
--
  Anyone who quotes me in their sig is an idiot. -- Rusty Russell.

Re: Hotplug CPU Boot Changes: BEWARE

[Eric W. Biederman]

Rusty Russell <rusty@rustcorp.com.au> writes:

> In message <m1elfjw39n.fsf@frodo.biederman.org> you write:
> > But for the latter something just a little more than minimal hacks
> > must be implemented.  But dynamic cpu enable/disable is definitely
> > worth it.
> 
> Perhaps I didn't make myself clear: hotplugging does not neccessarily
> mean physically removing or adding the CPU.  And as to whether they
> offer full support, or stub support, architectures can decide that for
> themselves, as they need.  It's not my call.

I guess as long as the interface to user space is consistent I don't
mind.
 
> I don't know how much of a win it is to disable HT on cpus, but I can
> tell you that adding & subtracting CPUs is a fairly heavy-weight
> operation in this design (I don't think we really want to lock around
> every cpu iteration).

Agreed.  However it should be lighter than a full system reboot,
which is what is needed now.  And if you can disable the extra
cpu you can benchmark on the same machine without rebooting
and see what kind of a win it is.

My basic point was that with a name like hotplug it was not immediately
clear this code could be useful for anything.  And that there are a lot
of debugging uses of being able stop or start a cpu.  Things like is
testing if HT/SMT worth it.

For the kernel I fully expect this to be a heavy weight operation.
But it is almost certainly lighter than swapoff /dev/hda1.  Which
makes it a lightweight operation for a system administrator.  

So it is my belief there is general usefulness of this code, on any
SMP target.  And with general applicability should come a common user
space interface.  

On x86  the code to do dynamic start/stop should already exist,
because we need it at kernel startup and shutdown.  The code paths just
need to be made safe to run at relatively arbitrary times.  The
generic part of which you are handling.

Eric