Mwait usage on AMD processors

Mwait usage on AMD processors

[Huang Rui]

Hi Len, Rafael, and all,

AMD proposed a new instruction named mwaitx. This is an extension of
mwait with a configurable timer (mwaitx = mwait + timer). And mwaitx
will act as mwait if timer is disabled. However, mwait/mwaitx cannot
let cpu core go to C1 state at current AMD processors, but has less
power consumption even at C0 while core is waiting.
As you know, mwait/mwaitx would have better performance than halt. So
could we propose an implementation to use mwaitx at idle call in boot
phase and cpuidle driver after boot phase. And the mwaitx idle is
exposed to user as an optional kernel parameter(idle=...), and decided
by user.

Any comments are warm for me :)

Thanks,
Rui

Re: Mwait usage on AMD processors

[Borislav Petkov]

On Thu, May 14, 2015 at 02:54:52PM +0800, Huang Rui wrote:
> Hi Len, Rafael, and all,
> 
> AMD proposed a new instruction named mwaitx. This is an extension of
> mwait with a configurable timer (mwaitx = mwait + timer). And mwaitx
> will act as mwait if timer is disabled. However, mwait/mwaitx cannot
> let cpu core go to C1 state at current AMD processors, but has less
> power consumption even at C0 while core is waiting.
> As you know, mwait/mwaitx would have better performance than halt. So
> could we propose an implementation to use mwaitx at idle call in boot
> phase and cpuidle driver after boot phase.

I'd select it in select_idle_routine()...

> And the mwaitx idle is exposed to user as an optional kernel
> parameter(idle=...), and decided by user.

If MWAITX is better than C1 (and C1E, which we're doing now AFAIR)
then I'd enable it by default on those machines so that it is used
automatically.

-- 
Regards/Gruss,
    Boris.

ECO tip #101: Trim your mails when you reply.
--

Re: Mwait usage on AMD processors

[Huang Rui]

On Thu, May 14, 2015 at 11:21:37AM +0200, Borislav Petkov wrote:
> On Thu, May 14, 2015 at 02:54:52PM +0800, Huang Rui wrote:
> > Hi Len, Rafael, and all,
> > 
> > AMD proposed a new instruction named mwaitx. This is an extension of
> > mwait with a configurable timer (mwaitx = mwait + timer). And mwaitx
> > will act as mwait if timer is disabled. However, mwait/mwaitx cannot
> > let cpu core go to C1 state at current AMD processors, but has less
> > power consumption even at C0 while core is waiting.
> > As you know, mwait/mwaitx would have better performance than halt. So
> > could we propose an implementation to use mwaitx at idle call in boot
> > phase and cpuidle driver after boot phase.
> 
> I'd select it in select_idle_routine()...
> 

I know, actually, I already completed it in my local side, and
mwaitx_idle will be called with x86_idle at boot phase. :)

I plan to continue to implement it with cpuidle driver if mwaitx_idle
can be accepted.

> > And the mwaitx idle is exposed to user as an optional kernel
> > parameter(idle=...), and decided by user.
> 
> If MWAITX is better than C1 (and C1E, which we're doing now AFAIR)
> then I'd enable it by default on those machines so that it is used
> automatically.
> 

At current processors, MWAITX cannot go to C1 state, but has faster
waiting exit speed. MWAITX is still at C0 but less power consumption.
So for now, I just want to set it optionally.
HW designer will do deeper low power state (C1, and more) in future
processors. At that time, we can enable it by default.

Thanks,
Rui

Re: Mwait usage on AMD processors

[Borislav Petkov]

On Thu, May 14, 2015 at 06:17:28PM +0800, Huang Rui wrote:
> At current processors, MWAITX cannot go to C1 state, but has faster
> waiting exit speed. MWAITX is still at C0 but less power consumption.
> So for now, I just want to set it optionally.
> HW designer will do deeper low power state (C1, and more) in future
> processors. At that time, we can enable it by default.

The only thing that matters IMO is which brings more power savings. If
C1E is still better, we should keep it the default. We can still use
MWAITX if someone boots with idle=mwait, as you say.

-- 
Regards/Gruss,
    Boris.

ECO tip #101: Trim your mails when you reply.
--

Re: Mwait usage on AMD processors

[Huang Rui]

On Thu, May 14, 2015 at 01:23:03PM +0200, Borislav Petkov wrote:
> On Thu, May 14, 2015 at 06:17:28PM +0800, Huang Rui wrote:
> > At current processors, MWAITX cannot go to C1 state, but has faster
> > waiting exit speed. MWAITX is still at C0 but less power consumption.
> > So for now, I just want to set it optionally.
> > HW designer will do deeper low power state (C1, and more) in future
> > processors. At that time, we can enable it by default.
> 
> The only thing that matters IMO is which brings more power savings. If
> C1E is still better, we should keep it the default. We can still use
> MWAITX if someone boots with idle=mwait, as you say.
> 

Is C1E here you mentioned is waiting state that use mwaitx enters at
AMD platform? If yes, please see below comments:

Current processor:
Power saving: C0 < C1E (AMD) < C1
Performance: Halt < Mwait <= Mwaitx
Halt -> C1, and Mwaitx/Mwait -> C1E (AMD)

Consider about the balance between power consumption and performance,
so we want to expose the interface. And mwaitx has different opcode
with traditional mwait. Due to C1E (AMD) less power saving that real
C1, so you can think it still in C0 at current.

In furture processor:
HW designer would do Mwaitx -> C1 or deeper low power state.

BTW, could I expose it and send the patches to review?

Thanks,
Rui

Re: Mwait usage on AMD processors

[Borislav Petkov]

On Thu, May 14, 2015 at 09:38:57PM +0800, Huang Rui wrote:
> Is C1E here you mentioned is waiting state that use mwaitx enters at
> AMD platform? If yes, please see below comments:
> 
> Current processor:
> Power saving: C0 < C1E (AMD) < C1

How is C1 > C1E ?

C1E is the Enhanced C1.

> Performance: Halt < Mwait <= Mwaitx

What performance? You're idle.

> Halt -> C1, and Mwaitx/Mwait -> C1E (AMD)

Huh? Right now we do HLT on all AMD and the hw enters C1E after a bunch
of stuff is fulfilled first. Are the plans to enter C1E from MWAIT now?

> Consider about the balance between power consumption and performance,
> so we want to expose the interface. And mwaitx has different opcode
> with traditional mwait.

There's alternative()'s for that.

> Due to C1E (AMD) less power saving that real C1, so you can think it
> still in C0 at current.

Which CPUs, current or upcoming?

If you want to do MWAITX for upcoming CPUs, simply define your own idle
routine and select it in select_idle_routine().

> In furture processor:
> HW designer would do Mwaitx -> C1 or deeper low power state.
> 
> BTW, could I expose it and send the patches to review?

You can always send out the patches, sure.

-- 
Regards/Gruss,
    Boris.

ECO tip #101: Trim your mails when you reply.
--

Re: Mwait usage on AMD processors

[Huang Rui]

On Thu, May 14, 2015 at 10:20:51PM +0800, Borislav Petkov wrote:
> On Thu, May 14, 2015 at 09:38:57PM +0800, Huang Rui wrote:
> > Is C1E here you mentioned is waiting state that use mwaitx enters at
> > AMD platform? If yes, please see below comments:
> > 
> > Current processor:
> > Power saving: C0 < C1E (AMD) < C1
> 
> How is C1 > C1E ?
> 
> C1E is the Enhanced C1.
> 

Apology that cause to misunderstand. It's not as same as intel.
Intel is able to go to C1E like you said, the C1E has less power
consumption than C1 on Intel platform.

But on AMD platform, mwaitx/mwait cannot go to C1 or C1E
like intel. The power consumption of waiting phase is somewhere in
between (C0 and C1). Actually, it's still in C0 but less power
consumption than normal C0.

> > Performance: Halt < Mwait <= Mwaitx
> 
> What performance? You're idle.
> 

The faster waiting exit speed. But it's hard to test the improvement,
do you have any idea? It's told by HW designer.

> > Halt -> C1, and Mwaitx/Mwait -> C1E (AMD)
> 
> Huh? Right now we do HLT on all AMD and the hw enters C1E after a bunch
> of stuff is fulfilled first. Are the plans to enter C1E from MWAIT now?
> 

Yes, I see all AMD platform only use HLT at current.

> > Consider about the balance between power consumption and performance,
> > so we want to expose the interface. And mwaitx has different opcode
> > with traditional mwait.
> 
> There's alternative()'s for that.
> 
> > Due to C1E (AMD) less power saving that real C1, so you can think it
> > still in C0 at current.
> 
> Which CPUs, current or upcoming?
> 

Current CPU, power consumption cannot go to deeper low power state
(C1) via mwaitx/mwait. But HW designers will implement it in future
processors.

Thanks,
Rui

Re: Mwait usage on AMD processors

[Borislav Petkov]

On Tue, May 19, 2015 at 03:25:53PM +0800, Huang Rui wrote:
> Apology that cause to misunderstand. It's not as same as intel.
> Intel is able to go to C1E like you said, the C1E has less power
> consumption than C1 on Intel platform.

You still misunderstand - I'm not talking about Intel platforms here but
AMD ones. On AMD we never enter idle with MWAIT - we do HLT which enters
C1 and then the hw enters C1E when a bunch of conditions are fulfilled.

> The faster waiting exit speed. But it's hard to test the improvement,
> do you have any idea? It's told by HW designer.

You can test the improvement with a special setup only. Unless you can
read out power consumption of a box while it is idle.

The exit-idle speed only does not suffice to switch to MWAIT though,
IMHO. I think power consumption in idle should be the relevant metric
here.

> Current CPU, power consumption cannot go to deeper low power state
> (C1) via mwaitx/mwait. But HW designers will implement it in future
> processors.

So future CPUs we will switch to MWAIT. I don't see a problem with that.

-- 
Regards/Gruss,
    Boris.

ECO tip #101: Trim your mails when you reply.
--

Re: Mwait usage on AMD processors

[Huang Rui]

On Tue, May 19, 2015 at 04:50:17PM +0800, Borislav Petkov wrote:
> On Tue, May 19, 2015 at 03:25:53PM +0800, Huang Rui wrote:
> > Apology that cause to misunderstand. It's not as same as intel.
> > Intel is able to go to C1E like you said, the C1E has less power
> > consumption than C1 on Intel platform.
> 
> You still misunderstand - I'm not talking about Intel platforms here but
> AMD ones. On AMD we never enter idle with MWAIT - we do HLT which enters
> C1 and then the hw enters C1E when a bunch of conditions are fulfilled.
> 

OK, got it. I see on AMD platforms, we all use default_idle (HLT).

> > The faster waiting exit speed. But it's hard to test the improvement,
> > do you have any idea? It's told by HW designer.
> 
> You can test the improvement with a special setup only. Unless you can
> read out power consumption of a box while it is idle.
> 

Could you please explain how to create the "special setup"? Actually,
that's my difficulty.

> The exit-idle speed only does not suffice to switch to MWAIT though,
> IMHO. I think power consumption in idle should be the relevant metric
> here.
> 

Yes, I agree with you. So that's why I was asking to provide an
optional parameter, not set it default. 

> > Current CPU, power consumption cannot go to deeper low power state
> > (C1) via mwaitx/mwait. But HW designers will implement it in future
> > processors.
> 
> So future CPUs we will switch to MWAIT. I don't see a problem with that.
> 

Yes, at that time, we would like to use mwaitx/mwait as default idle
routine for AMD.

Thanks,
Rui