CPU hotplug on i386

CPU hotplug on i386

[Pavel Machek]

Hi!

I tried to solve long-standing uglyness in swsusp cmp code by calling
cpu hotplug... only to find out that CONFIG_CPU_HOTPLUG is not
available on i386. Is there way to enable CPU_HOTPLUG on i386?

								Pavel

--- clean/kernel/power/smp.c	2004-08-15 19:15:06.000000000 +0200
+++ linux/kernel/power/smp.c	2005-03-16 14:16:00.000000000 +0100
@@ -16,70 +16,39 @@
 #include <asm/atomic.h>
 #include <asm/tlbflush.h>
 
-static atomic_t cpu_counter, freeze;
-
-
-static void smp_pause(void * data)
-{
-	struct saved_context ctxt;
-	__save_processor_state(&ctxt);
-	printk("Sleeping in:\n");
-	dump_stack();
-	atomic_inc(&cpu_counter);
-	while (atomic_read(&freeze)) {
-		/* FIXME: restore takes place at random piece inside this.
-		   This should probably be written in assembly, and
-		   preserve general-purpose registers, too
-
-		   What about stack? We may need to move to new stack here.
-
-		   This should better be ran with interrupts disabled.
-		 */
-		cpu_relax();
-		barrier();
-	}
-	atomic_dec(&cpu_counter);
-	__restore_processor_state(&ctxt);
-}
-
-cpumask_t oldmask;
+cpumask_t frozen_cpus;
 
 void disable_nonboot_cpus(void)
 {
-	printk("Freezing CPUs (at %d)", smp_processor_id());
-	oldmask = current->cpus_allowed;
-	set_cpus_allowed(current, cpumask_of_cpu(0));
-	current->state = TASK_INTERRUPTIBLE;
-	schedule_timeout(HZ);
-	printk("...");
-	BUG_ON(smp_processor_id() != 0);
-
-	/* FIXME: for this to work, all the CPUs must be running
-	 * "idle" thread (or we deadlock). Is that guaranteed? */
-
-	atomic_set(&cpu_counter, 0);
-	atomic_set(&freeze, 1);
-	smp_call_function(smp_pause, NULL, 0, 0);
-	while (atomic_read(&cpu_counter) < (num_online_cpus() - 1)) {
-		cpu_relax();
-		barrier();
+	int cpu, error;
+	cpus_clear(frozen_cpus);
+	printk("Freezing cpus...\n");
+	for_each_online_cpu(cpu) {
+		if (!cpu)
+			continue;
+		cpu_set(cpu, frozen_cpus);
+		error = cpu_down(cpu);
+		if (!error)
+			continue;
+		printk("Error taking cpu %d down: %d\n", cpu, error);
+		panic("Too many cpus");
 	}
-	printk("ok\n");
+	BUG_ON(smp_processor_id() != 0);
 }
 
 void enable_nonboot_cpus(void)
 {
-	printk("Restarting CPUs");
-	atomic_set(&freeze, 0);
-	while (atomic_read(&cpu_counter)) {
-		cpu_relax();
-		barrier();
+	int cpu, error;
+	printk("Thawing cpus...\n");
+	for_each_cpu_mask(cpu, frozen_cpus) {
+		if (!cpu)
+			continue;
+		error = cpu_up(cpu);
+		if (!error)
+			continue;
+		printk("Error taking cpu %d up: %d\n", cpu, error);
+		panic("Not enough cpus");
 	}
-	printk("...");
-	set_cpus_allowed(current, oldmask);
-	schedule();
-	printk("ok\n");
-
 }
 
 

-- 
People were complaining that M$ turns users into beta-testers...
...jr ghea gurz vagb qrirybcref, naq gurl frrz gb yvxr vg gung jnl!

Re: CPU hotplug on i386

[Rafael J. Wysocki]

Hi,

On Wednesday, 16 of March 2005 14:21, Pavel Machek wrote:
> Hi!
> 
> I tried to solve long-standing uglyness in swsusp cmp code by calling
> cpu hotplug... only to find out that CONFIG_CPU_HOTPLUG is not
> available on i386. Is there way to enable CPU_HOTPLUG on i386?

Heh, that's exactly what I was thinking about.  ;-)

AFAICS we don't need the full CPU hotplug to do this.  For suspend, we need to
enable the CPU hotplug-related code in sched.c and cpu.c, and we need to
implement the functions __cpu_disable() and __cpu_die() (called from within
cpu.c) on each architecture for which we want swsusp to work on SMP.

If that's acceptable, the CPU hotplug code in sched.c and cpu.c may be
enabled by changing some #ifdefs there.  For example, we could replace the

#idef CONFIG_CPU_HOTPLUG

with

#if defined(CONFIG_CPU_HOTPLUG) || (defined(CONFIG_SOFTWARE_SUSPEND)
	&& defined(CONFIG_SMP))

wherever necessary.

The implementation of __cpu_disable() and __cpu_die() may be a bit more
tricky, however.  In __cpu_disable() we need to save the settings of the local
APIC of each CPU (on x86-64, at least) etc.  In __cpu_die() we should give the
"frozen" CPU some "neutral" code to execute, it seems (or "hlt" it?).

I've looked at the ia64 implementation of these functions, but I haven't fully
understood it yet.

Greets,
Rafael


-- 
- Would you tell me, please, which way I ought to go from here?
- That depends a good deal on where you want to get to.
		-- Lewis Carroll "Alice's Adventures in Wonderland"

Re: CPU hotplug on i386

[Nigel Cunningham]

Hi.

On Thu, 2005-03-17 at 01:40, Rafael J. Wysocki wrote:
> On Wednesday, 16 of March 2005 14:21, Pavel Machek wrote:
> > Hi!
> > 
> > I tried to solve long-standing uglyness in swsusp cmp code by calling
> > cpu hotplug... only to find out that CONFIG_CPU_HOTPLUG is not
> > available on i386. Is there way to enable CPU_HOTPLUG on i386?
> 
> Heh, that's exactly what I was thinking about.  ;-)
> 
> AFAICS we don't need the full CPU hotplug to do this.  For suspend, we need to
> enable the CPU hotplug-related code in sched.c and cpu.c, and we need to
> implement the functions __cpu_disable() and __cpu_die() (called from within
> cpu.c) on each architecture for which we want swsusp to work on SMP.
> 
> If that's acceptable, the CPU hotplug code in sched.c and cpu.c may be
> enabled by changing some #ifdefs there.  For example, we could replace the
> 
> #idef CONFIG_CPU_HOTPLUG
> 
> with
> 
> #if defined(CONFIG_CPU_HOTPLUG) || (defined(CONFIG_SOFTWARE_SUSPEND)
> 	&& defined(CONFIG_SMP))
> 
> wherever necessary.
> 
> The implementation of __cpu_disable() and __cpu_die() may be a bit more
> tricky, however.  In __cpu_disable() we need to save the settings of the local
> APIC of each CPU (on x86-64, at least) etc.  In __cpu_die() we should give the
> "frozen" CPU some "neutral" code to execute, it seems (or "hlt" it?).
> 
> I've looked at the ia64 implementation of these functions, but I haven't fully
> understood it yet.

It would be good (32 bit) if you handled MTRRs as well. Putting their
save/restore code in the driver model is racy (SMP calls). I've worked
around that in Suspend2 by removing the driver model support and making
the save/restore calls from the lowlevel code. I think this should be
considered the right thing to do.

By the way, I'm keen to test this with Suspend2 as well. I would have
done so before now, but I didn't realise Zwane had the support done.

Regards,

Nigel
-- 
Nigel Cunningham
Software Engineer, Canberra, Australia
http://www.cyclades.com
Bus: +61 (2) 6291 9554; Hme: +61 (2) 6292 8028;  Mob: +61 (417) 100 574

Maintainer of Suspend2 Kernel Patches http://suspend2.net

Re: CPU hotplug on i386

[Nathan Lynch]

On Wed, Mar 16, 2005 at 02:21:52PM +0100, Pavel Machek wrote:
> Hi!
> 
> I tried to solve long-standing uglyness in swsusp cmp code by calling
> cpu hotplug... only to find out that CONFIG_CPU_HOTPLUG is not
> available on i386. Is there way to enable CPU_HOTPLUG on i386?

i386 cpu hotplug has been in -mm for a while.  Don't know when (if
ever) it will get merged.

Nathan

Re: CPU hotplug on i386

[Zwane Mwaikambo]

On Wed, 16 Mar 2005, Nathan Lynch wrote:

> On Wed, Mar 16, 2005 at 02:21:52PM +0100, Pavel Machek wrote:
>
> > I tried to solve long-standing uglyness in swsusp cmp code by calling
> > cpu hotplug... only to find out that CONFIG_CPU_HOTPLUG is not
> > available on i386. Is there way to enable CPU_HOTPLUG on i386?
> 
> i386 cpu hotplug has been in -mm for a while.  Don't know when (if
> ever) it will get merged.

Well this is a good user =)

Re: CPU hotplug on i386

[Rafael J. Wysocki]

Hi,

On Wednesday, 16 of March 2005 18:09, Nathan Lynch wrote:
> On Wed, Mar 16, 2005 at 02:21:52PM +0100, Pavel Machek wrote:
> > Hi!
> > 
> > I tried to solve long-standing uglyness in swsusp cmp code by calling
> > cpu hotplug... only to find out that CONFIG_CPU_HOTPLUG is not
> > available on i386. Is there way to enable CPU_HOTPLUG on i386?
> 
> i386 cpu hotplug has been in -mm for a while.  Don't know when (if
> ever) it will get merged.

Thanks a lot for this hint! ;-)

Pavel, I've ported the basic i386 CPU hotplug stuff, without the sysfs
interface, to x86-64 (a cut'n'paste kind of work, mostly).  For now, I've
made HOTPLUG_CPU on x86-64 depend on SMP and SOFTWARE_SUSPEND and be set
automatically.

I'm going to test it together with your patch tomorrow.

Greets,
Rafael


-- 
- Would you tell me, please, which way I ought to go from here?
- That depends a good deal on where you want to get to.
		-- Lewis Carroll "Alice's Adventures in Wonderland"

Re: CPU hotplug on i386

[Pavel Machek]

Hi!

> > > I tried to solve long-standing uglyness in swsusp cmp code by calling
> > > cpu hotplug... only to find out that CONFIG_CPU_HOTPLUG is not
> > > available on i386. Is there way to enable CPU_HOTPLUG on i386?
> > 
> > i386 cpu hotplug has been in -mm for a while.  Don't know when (if
> > ever) it will get merged.
> 
> Thanks a lot for this hint! ;-)
> 
> Pavel, I've ported the basic i386 CPU hotplug stuff, without the sysfs
> interface, to x86-64 (a cut'n'paste kind of work, mostly).  For now, I've
> made HOTPLUG_CPU on x86-64 depend on SMP and SOFTWARE_SUSPEND and be set
> automatically.
> 
> I'm going to test it together with your patch tomorrow.

Hey, don't count on my patch. It is first shot, never tested.

									Pavel
-- 
People were complaining that M$ turns users into beta-testers...
...jr ghea gurz vagb qrirybcref, naq gurl frrz gb yvxr vg gung jnl!

Re: CPU hotplug on i386

[Rafael J. Wysocki]

Hi,

On Thursday, 17 of March 2005 01:04, Pavel Machek wrote:
> Hi!
> 
> > > > I tried to solve long-standing uglyness in swsusp cmp code by calling
> > > > cpu hotplug... only to find out that CONFIG_CPU_HOTPLUG is not
> > > > available on i386. Is there way to enable CPU_HOTPLUG on i386?
> > > 
> > > i386 cpu hotplug has been in -mm for a while.  Don't know when (if
> > > ever) it will get merged.
> > 
> > Thanks a lot for this hint! ;-)
> > 
> > Pavel, I've ported the basic i386 CPU hotplug stuff, without the sysfs
> > interface, to x86-64 (a cut'n'paste kind of work, mostly).  For now, I've
> > made HOTPLUG_CPU on x86-64 depend on SMP and SOFTWARE_SUSPEND and be set
> > automatically.
> > 
> > I'm going to test it together with your patch tomorrow.
> 
> Hey, don't count on my patch. It is first shot, never tested.

It'll be the first time, then.  I'm not afraid. ;-)

Rafael


-- 
- Would you tell me, please, which way I ought to go from here?
- That depends a good deal on where you want to get to.
		-- Lewis Carroll "Alice's Adventures in Wonderland"

Re: CPU hotplug on i386

[Nathan Lynch]

Hi again-

On Wed, Mar 16, 2005 at 02:21:52PM +0100, Pavel Machek wrote:
> +	int cpu, error;
> +	cpus_clear(frozen_cpus);
> +	printk("Freezing cpus...\n");
> +	for_each_online_cpu(cpu) {
> +		if (!cpu)
> +			continue;
> +		cpu_set(cpu, frozen_cpus);
> +		error = cpu_down(cpu);
> +		if (!error)
> +			continue;
> +		printk("Error taking cpu %d down: %d\n", cpu, error);
> +		panic("Too many cpus");
>  	}
> -	printk("ok\n");
> +	BUG_ON(smp_processor_id() != 0);
>  }
>  
>  void enable_nonboot_cpus(void)
>  {
> -	printk("Restarting CPUs");
> -	atomic_set(&freeze, 0);
> -	while (atomic_read(&cpu_counter)) {
> -		cpu_relax();
> -		barrier();
> +	int cpu, error;
> +	printk("Thawing cpus...\n");
> +	for_each_cpu_mask(cpu, frozen_cpus) {
> +		if (!cpu)
> +			continue;
> +		error = cpu_up(cpu);
> +		if (!error)
> +			continue;
> +		printk("Error taking cpu %d up: %d\n", cpu, error);
> +		panic("Not enough cpus");
>  	}

If you're going to depend on CONFIG_HOTPLUG_CPU for swsusp on smp, why
not offline the non-boot cpus from userspace?  Same goes for onlining
additional cpus during resume.

Nathan

Re: CPU hotplug on i386

[Pavel Machek]

Hi!

> On Wed, Mar 16, 2005 at 02:21:52PM +0100, Pavel Machek wrote:
> > +	int cpu, error;
> > +	cpus_clear(frozen_cpus);
> > +	printk("Freezing cpus...\n");
> > +	for_each_online_cpu(cpu) {
> > +		if (!cpu)
> > +			continue;
> > +		cpu_set(cpu, frozen_cpus);
> > +		error = cpu_down(cpu);
> > +		if (!error)
> > +			continue;
> > +		printk("Error taking cpu %d down: %d\n", cpu, error);
> > +		panic("Too many cpus");
> >  	}
> > -	printk("ok\n");
> > +	BUG_ON(smp_processor_id() != 0);
> >  }
> >  
> >  void enable_nonboot_cpus(void)
> >  {
> > -	printk("Restarting CPUs");
> > -	atomic_set(&freeze, 0);
> > -	while (atomic_read(&cpu_counter)) {
> > -		cpu_relax();
> > -		barrier();
> > +	int cpu, error;
> > +	printk("Thawing cpus...\n");
> > +	for_each_cpu_mask(cpu, frozen_cpus) {
> > +		if (!cpu)
> > +			continue;
> > +		error = cpu_up(cpu);
> > +		if (!error)
> > +			continue;
> > +		printk("Error taking cpu %d up: %d\n", cpu, error);
> > +		panic("Not enough cpus");
> >  	}
> 
> If you're going to depend on CONFIG_HOTPLUG_CPU for swsusp on smp, why
> not offline the non-boot cpus from userspace?  Same goes for onlining
> additional cpus during resume.

I'd like to stay in control. I may want to stop drivers first,
or something like that. Plus I need to do that during resume, too, when
userspace is not available, yet.
								Pavel
-- 
People were complaining that M$ turns users into beta-testers...
...jr ghea gurz vagb qrirybcref, naq gurl frrz gb yvxr vg gung jnl!

Re: CPU hotplug on i386

[Pavel Machek]

Hi!

> I tried to solve long-standing uglyness in swsusp cmp code by calling
> cpu hotplug... only to find out that CONFIG_CPU_HOTPLUG is not
> available on i386. Is there way to enable CPU_HOTPLUG on i386?

BTW Li Shaohua has prototype smp/S3 implementation. I'll take detailed
look at that one.
								Pavel
-- 
People were complaining that M$ turns users into beta-testers...
...jr ghea gurz vagb qrirybcref, naq gurl frrz gb yvxr vg gung jnl!