SMP cpu deep sleep

SMP cpu deep sleep

[Hans Kristian Rosbach]

While reading through hotplug and speedstep patches
I came to think of a feature I think might be useful.

In an SMP system there are several cpus, this generates
extra heat and power consuption even on idle load.
Is there a way to put all cpus but cpu1 into a kind of
deep sleep? Cpu1 would have to do all work (including irqs)
of course.

We have a lot of SMP systems that we host, and they
are heavily used ~10 hours of the day, the rest they are
mostly idle. They could run on only 1 cpu during lenghty
idle periods.

If it is possible to put cpus to a deeper sleep than
just the simple idle, then the kernel could make use of this.

It would be a cool feature.

Sincerly
    Hans K. Rosbach

Re: SMP cpu deep sleep

[Jeff Sipek]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Friday 20 August 2004 04:06, Hans Kristian Rosbach wrote:
> While reading through hotplug and speedstep patches
> I came to think of a feature I think might be useful.
>
> In an SMP system there are several cpus, this generates
> extra heat and power consuption even on idle load.
> Is there a way to put all cpus but cpu1 into a kind of
> deep sleep? Cpu1 would have to do all work (including irqs)
> of course.

With Rusty's Hotplug CPU, a userspace script should be able to do this by 
cat'ing 1 or 0 into the appropriate sysfs file.

Jeff.

- -- 
"If I have trouble installing Linux, something is wrong. Very wrong."
  - Linus Torvalds
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.5 (GNU/Linux)

iD8DBQFBJb08wFP0+seVj/4RAotfAJ9gvudaaAQbYjSiky78rkkRnlZHAACfb8oR
Whj349hrd6ZnUPf27JwjJxY=
=mL1E
-----END PGP SIGNATURE-----

Re: SMP cpu deep sleep

[Hans Kristian Rosbach]

On Fri, 2004-08-20 at 10:58, Jeff Sipek wrote:
> On Friday 20 August 2004 04:06, Hans Kristian Rosbach wrote:
> > While reading through hotplug and speedstep patches
> > I came to think of a feature I think might be useful.
> >
> > In an SMP system there are several cpus, this generates
> > extra heat and power consuption even on idle load.
> > Is there a way to put all cpus but cpu1 into a kind of
> > deep sleep? Cpu1 would have to do all work (including irqs)
> > of course.
> 
> With Rusty's Hotplug CPU, a userspace script should be able to do this
by 
> cat'ing 1 or 0 into the appropriate sysfs file.

Nice, this can be done on non-hotplug motherboards as well?
How much time does it take to take the cpu up again?

It would need to be awakened whenever the activity increases
again, and that delay should not be too long.

Also, what effect does this have on cpu power consumtion
and thermal output? Does it lower it below normal idle?

-HK

RE: SMP cpu deep sleep

[Nakajima, Jun]

>From: linux-kernel-owner@vger.kernel.org [mailto:linux-kernel-
>owner@vger.kernel.org] On Behalf Of Hans Kristian Rosbach
>Sent: Friday, August 20, 2004 1:07 AM
>To: linux-kernel@vger.kernel.org
>Subject: SMP cpu deep sleep
>
>While reading through hotplug and speedstep patches
>I came to think of a feature I think might be useful.
>
>In an SMP system there are several cpus, this generates
>extra heat and power consuption even on idle load.
>Is there a way to put all cpus but cpu1 into a kind of
>deep sleep? Cpu1 would have to do all work (including irqs)
>of course.
>
>We have a lot of SMP systems that we host, and they
>are heavily used ~10 hours of the day, the rest they are
>mostly idle. They could run on only 1 cpu during lenghty
>idle periods.
>
>If it is possible to put cpus to a deeper sleep than
>just the simple idle, then the kernel could make use of this.

Most SMP server systems today don't support the C-states beyond C1 (at
least on x86 or x86-64, as far as I know). If your system supports that,
I think it would be nice to do that. 

Jun

>
>It would be a cool feature.
>
>Sincerly
>    Hans K. Rosbach
>

Re: SMP cpu deep sleep

[Zwane Mwaikambo]

On Fri, 20 Aug 2004, Hans Kristian Rosbach wrote:

> Nice, this can be done on non-hotplug motherboards as well?
> How much time does it take to take the cpu up again?

Yes it works on non hotplug motherboards, it's only a logical offline.
Returning from offline;

root@morocco cpu1 {0:0} time echo 0 > online
0.000u 0.290s 0:01.77 16.3%     0+0k 0+0io 0pf+0w
root@morocco cpu1 {0:0} time echo 1 > online
0.000u 0.001s 0:00.10 0.0%      0+0k 0+0io 0pf+0w

> It would need to be awakened whenever the activity increases
> again, and that delay should not be too long.
>
> Also, what effect does this have on cpu power consumtion
> and thermal output? Does it lower it below normal idle?

None at the moment as we busy loop, so if anything, power consumption
would go up. What could be done is perhaps 'hlt' the processor and wait
for a RETURN_FROM_OFFLINE IPI. Or perhaps with more work, graft in the
ACPI C state transitions.

Re: SMP cpu deep sleep

[Wes Felter]

On Fri, 20 Aug 2004 10:06:47 +0200, Hans Kristian Rosbach wrote:

> While reading through hotplug and speedstep patches
> I came to think of a feature I think might be useful.
> 
> In an SMP system there are several cpus, this generates
> extra heat and power consuption even on idle load.
> Is there a way to put all cpus but cpu1 into a kind of
> deep sleep? Cpu1 would have to do all work (including irqs)
> of course.
> 
> We have a lot of SMP systems that we host, and they
> are heavily used ~10 hours of the day, the rest they are
> mostly idle. They could run on only 1 cpu during lenghty
> idle periods.
> 
> If it is possible to put cpus to a deeper sleep than
> just the simple idle, then the kernel could make use of this.

I worked on this last year (I call it CPU packing, because the idea is to
pack the load onto the fewest number of CPUs).

The CPU hotplug patch is the way to go, but the hardware is the problem. I
talked to an Intel CPU architect at MICRO last year and he confirmed that
SMP Intel systems don't support any low-power modes besides HLT. AMD's
documentation says that Opterons support voltage/frequency scaling (aka
Cool 'n' Quiet), but AFAICT the documentation is wrong. In summary, you
are doomed.

-- 
Wes Felter
Power-Aware Systems Department
IBM Austin Research Lab
11400 Burnet Road, Austin, TX 78758
Tel 512-838-7933

Re: SMP cpu deep sleep

[Benjamin Herrenschmidt]

On Sat, 2004-08-21 at 02:44, Wes Felter wrote:

> The CPU hotplug patch is the way to go, but the hardware is the problem. I
> talked to an Intel CPU architect at MICRO last year and he confirmed that
> SMP Intel systems don't support any low-power modes besides HLT. AMD's
> documentation says that Opterons support voltage/frequency scaling (aka
> Cool 'n' Quiet), but AFAICT the documentation is wrong. In summary, you
> are doomed.

Well, with intel CPUs maybe ;) On PPC we can put some CPUs (at least the
desktop ones and the 970, I don't know about POWER3/4/5) into SLEEP mode
after flushing all caches & masking all IRQs to them, and get them out
via an IPI, or we can mask EE as well and get them out via a soft reset
if we have access to it.

Ben.

Re: SMP cpu deep sleep

[Bill Davidsen]

Wes Felter wrote:

> I worked on this last year (I call it CPU packing, because the idea is to
> pack the load onto the fewest number of CPUs).
> 
> The CPU hotplug patch is the way to go, but the hardware is the problem. I
> talked to an Intel CPU architect at MICRO last year and he confirmed that
> SMP Intel systems don't support any low-power modes besides HLT. AMD's
> documentation says that Opterons support voltage/frequency scaling (aka
> Cool 'n' Quiet), but AFAICT the documentation is wrong. In summary, you
> are doomed.
> 

For power saving, HLT is hard to beat ;-) You note HLT as if there was 
some good reason not to use it... Mask everything except some BACK2WORK 
int from the night watchman CPU. I would really like this on some 
machines which seem to leave all CPUs generating heat even when booted 
with a uni kernel.

Whilst thinking about this, *if* using HLT is practical in therms of 
power saving, perhaps all but the last CPU could HLT if the run queue 
was empty, and only be awakened by the "last" CPU, in some case where 
the run queue length was longer than {some_value}.

-- 
    -bill davidsen (davidsen@tmr.com)
"The secret to procrastination is to put things off until the
  last possible moment - but no longer"  -me

Re: SMP cpu deep sleep

[Pavel Machek]

Hi!

> SMP Intel systems don't support any low-power modes besides HLT. AMD's
> documentation says that Opterons support voltage/frequency scaling (aka
> Cool 'n' Quiet), but AFAICT the documentation is wrong. In summary, you

I believe docs is right here. See archives, it looks like Paul
 is actually testing pn-k8 on smp machines.

				Pavel

-- 
64 bytes from 195.113.31.123: icmp_seq=28 ttl=51 time=448769.1 ms         

Re: SMP cpu deep sleep

[Alan Cox]

On Llu, 2004-08-23 at 22:30, Bill Davidsen wrote:
> For power saving, HLT is hard to beat ;-) You note HLT as if there was 
> some good reason not to use it...

On a large number of processors HLT has only small effect because the
processor still has to do bus arbitration , still has large leakage
currents and has other circuitry running. The PowerNow stuff seems to
make a much bigger difference but has limits to how low you can go.