questions of cpuidle

questions of cpuidle

[Alex Shi]

Sorry for a idiot of cpuidle.

I just find few cpu set TIMER_STOP on cpuidle, like omap4 and big.Little
driver. Does that mean other ARM cpu or x86 cpu can get the timer
interrupt in cpuidle?

If the timer stopped during cpuidle, does that means at least one cpu
cann't get into deep c-state since system need a cpu to wake up other
deep c-state cpu? Or sth I missed?

If the cpu stopped the interrupt during deep c-state and without
monitor/mwait support, which kind of ipi can wake the cpu? I mean like a
x86 cpu, APIC stopped in c3 mode, but actually ipi send via apic bus. So
I don't know which ipi work?

-- 
Thanks
    Alex

Re: questions of cpuidle

[Daniel Lezcano]

On 12/09/2013 02:40 PM, Alex Shi wrote:
>
> Sorry for a idiot of cpuidle.
>
> I just find few cpu set TIMER_STOP on cpuidle, like omap4 and big.Little
> driver. Does that mean other ARM cpu or x86 cpu can get the timer
> interrupt in cpuidle?
>
> If the timer stopped during cpuidle, does that means at least one cpu
> cann't get into deep c-state since system need a cpu to wake up other
> deep c-state cpu? Or sth I missed?
>
> If the cpu stopped the interrupt during deep c-state and without
> monitor/mwait support, which kind of ipi can wake the cpu? I mean like a
> x86 cpu, APIC stopped in c3 mode, but actually ipi send via apic bus. So
> I don't know which ipi work?

Hi Alex,

there are several timers on the system, the local and the external. On 
ARM we find the local timers twd. Depending on the depth of the idle 
state, they can be shutdown.

All the cpus can go to a deep idle state because the timer expiration 
will be delegated to a timer which does not belong to the power domain 
of the idle state.

The idle state contains the flag TIMER_STOP when the idle state power 
downs the local timer.

The cpuidle common framework checks this flag when entering the idle 
state and calls clock_events_notify(ENTER/EXIT). That tells the time 
framework to switch to another timer on the system in replacement of the 
local one.

The external timer will be programmed to the next event and will store 
the cpu which is concerned by this expiration. This timer is called the 
broadcast timer.

When the broadcast timer expires there are two cases:

* if the timer has the CLOCK_EVT_FEAT_DYNIRQ set, when the time 
framework switched to this timer as described above, it also set the irq 
affinity to the cpu. So no IPI happens because the interrupt occurs 
directly to the targeted cpu

* if the timer has *no* CLOCK_EVT_FEAT_DYNIRQ set, then the interrupt 
will occur to any cpus belonging to the cpumask (usually cpu0), and if 
the timer must expires for another cpu then an IPI is emitted to the 
cpu. The IPI is the IPI_TIMER.

Concerning the wake up of the cpu: the cpu disabled the irq and goes to 
sleep, it is up to the firmware to wake up the cpu when an interrupt 
occurs. It will exits its sleep state, call clock_events_notify(EXIT), 
by this way re-switching to the local timer, and then re-enabling the 
local interrupt which leads to the interrupt handler.

There are some more informations in the wiki page [1].

   -- Daniel

[1] https://wiki.linaro.org/WorkingGroups/PowerManagement/Doc/WakeUpSources

-- 
  <http://www.linaro.org/> Linaro.org │ Open source software for ARM SoCs

Follow Linaro:  <http://www.facebook.com/pages/Linaro> Facebook |
<http://twitter.com/#!/linaroorg> Twitter |
<http://www.linaro.org/linaro-blog/> Blog

Re: questions of cpuidle

[Alex Shi]

On 12/09/2013 10:17 PM, Daniel Lezcano wrote:
> 
> Concerning the wake up of the cpu: the cpu disabled the irq and goes to
> sleep, it is up to the firmware to wake up the cpu when an interrupt
> occurs. It will exits its sleep state, call clock_events_notify(EXIT),
> by this way re-switching to the local timer, and then re-enabling the
> local interrupt which leads to the interrupt handler.

Thanks a lots for excellent article and detailed explains!

So, if the firmware is in response to wake up cpu. that means there is a
unit which control the firmware and it can not be power down. Do you
know which unit running the firmware to wake up deep idle CPU.
And does the wake up pass via GIC to CPU? If so, does the GIC need keep
awake when all cpu idle? If not, how the firmware give the interrupt to
CPU? And I am wondering if the deep idle cpu voltage get to near 0.
How the cpu get the interrupt signal?

> 
> There are some more informations in the wiki page [1].
> 
>   -- Daniel
> 
> [1] https://wiki.linaro.org/WorkingGroups/PowerManagement/Doc/WakeUpSources


-- 
Thanks
    Alex

Re: questions of cpuidle

[Daniel Lezcano]

On 12/10/2013 07:33 AM, Alex Shi wrote:
> On 12/09/2013 10:17 PM, Daniel Lezcano wrote:
>>
>> Concerning the wake up of the cpu: the cpu disabled the irq and
>> goes to sleep, it is up to the firmware to wake up the cpu when an
>> interrupt occurs. It will exits its sleep state, call
>> clock_events_notify(EXIT), by this way re-switching to the local
>> timer, and then re-enabling the local interrupt which leads to the
>> interrupt handler.
>
> Thanks a lots for excellent article and detailed explains!
>
> So, if the firmware is in response to wake up cpu. that means there
> is a unit which control the firmware and it can not be power down.

Correct.

> Do you know which unit running the firmware to wake up deep idle
> CPU.

That depends on the SoC implementation.

Some SoC have a "Power Management Unit". The PMU has several idle states
defined, each of them are described in the technical reference manual
(TRM) with the wake up sources.

Some SoC don't have any PMU and the idle states are very few, keeping
most of the logic on.

Some other SoC hide the PMU behind PSCI calls.

> And does the wake up pass via GIC to CPU? If so, does the GIC need
> keep awake when all cpu idle? If not, how the firmware give the
> interrupt to CPU? And I am wondering if the deep idle cpu voltage get
> to near 0. How the cpu get the interrupt signal?

If a deep idle state powers down the GIC, it is up to the PMU to proxy
the interrupts. When an interrupt occurs, the PMU powers up the logic,
including the GIC. The notifier call chain with cpu_suspend / cpu_resume
will save and restore the GIC registers.

But this is hardware specific and will depend on how the PMU is
implemented and how far it goes in the power management.

You have a good example in the drivers/cpuidle/cpuidle-ux500.c to
understand with the comments how the interrupts are handled through the
power management unit.

In the Xillinx documentation available on the web [1], the chapter 24.4
gives the information about one kind of PMU.

I believe the mechanism is pretty similar on all the hardware but it is
obfuscated by a generic power instruction like mwait.

   -- Daniel

[1]
http://www.xilinx.com/support/documentation/user_guides/ug585-Zynq-7000-TRM.pdf


>> There are some more informations in the wiki page [1].
>>
>> -- Daniel
>>
>> [1]
>> https://wiki.linaro.org/WorkingGroups/PowerManagement/Doc/WakeUpSources
>
>>
>


-- 
  <http://www.linaro.org/> Linaro.org │ Open source software for ARM SoCs

Follow Linaro:  <http://www.facebook.com/pages/Linaro> Facebook |
<http://twitter.com/#!/linaroorg> Twitter |
<http://www.linaro.org/linaro-blog/> Blog

Re: questions of cpuidle

[anish singh]

On Mon, Dec 9, 2013 at 11:27 PM, Daniel Lezcano
<daniel.lezcano@linaro.org> wrote:
> On 12/10/2013 07:33 AM, Alex Shi wrote:
>>
>> On 12/09/2013 10:17 PM, Daniel Lezcano wrote:
>>>
>>>
>>> Concerning the wake up of the cpu: the cpu disabled the irq and
>>> goes to sleep, it is up to the firmware to wake up the cpu when an
>>> interrupt occurs. It will exits its sleep state, call
>>> clock_events_notify(EXIT), by this way re-switching to the local
>>> timer, and then re-enabling the local interrupt which leads to the
>>> interrupt handler.
>>
>>
>> Thanks a lots for excellent article and detailed explains!
>>
>> So, if the firmware is in response to wake up cpu. that means there
>> is a unit which control the firmware and it can not be power down.
>
>
> Correct.
>
>
>> Do you know which unit running the firmware to wake up deep idle
>> CPU.
>
>
> That depends on the SoC implementation.
and which is intentionally kept hidden away.
>
> Some SoC have a "Power Management Unit". The PMU has several idle states
> defined, each of them are described in the technical reference manual
> (TRM) with the wake up sources.
PMU is intentionally kept hidden by OEM companies as this way
they protect there hardware IP.
>
> Some SoC don't have any PMU and the idle states are very few, keeping
> most of the logic on.
>
> Some other SoC hide the PMU behind PSCI calls.
which is intentional.
>
>
>> And does the wake up pass via GIC to CPU? If so, does the GIC need
>> keep awake when all cpu idle? If not, how the firmware give the
>> interrupt to CPU? And I am wondering if the deep idle cpu voltage get
>> to near 0. How the cpu get the interrupt signal?
>
>
> If a deep idle state powers down the GIC, it is up to the PMU to proxy
> the interrupts. When an interrupt occurs, the PMU powers up the logic,
> including the GIC. The notifier call chain with cpu_suspend / cpu_resume
> will save and restore the GIC registers.
>
> But this is hardware specific and will depend on how the PMU is
> implemented and how far it goes in the power management.
>
> You have a good example in the drivers/cpuidle/cpuidle-ux500.c to
> understand with the comments how the interrupts are handled through the
> power management unit.
>
> In the Xillinx documentation available on the web [1], the chapter 24.4
> gives the information about one kind of PMU.
>
> I believe the mechanism is pretty similar on all the hardware but it is
> obfuscated by a generic power instruction like mwait.
>
>   -- Daniel
>
> [1]
> http://www.xilinx.com/support/documentation/user_guides/ug585-Zynq-7000-TRM.pdf
>
>
>
>>> There are some more informations in the wiki page [1].
>>>
>>> -- Daniel
>>>
>>> [1]
>>> https://wiki.linaro.org/WorkingGroups/PowerManagement/Doc/WakeUpSources
>>
>>
>>>
>>
>
>
> --
>  <http://www.linaro.org/> Linaro.org │ Open source software for ARM SoCs
>
> Follow Linaro:  <http://www.facebook.com/pages/Linaro> Facebook |
> <http://twitter.com/#!/linaroorg> Twitter |
> <http://www.linaro.org/linaro-blog/> Blog
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/

Re: questions of cpuidle

[Daniel Lezcano]

On 12/10/2013 09:07 AM, anish singh wrote:
> On Mon, Dec 9, 2013 at 11:27 PM, Daniel Lezcano
> <daniel.lezcano@linaro.org> wrote:
>> On 12/10/2013 07:33 AM, Alex Shi wrote:
>>>
>>> On 12/09/2013 10:17 PM, Daniel Lezcano wrote:
>>>>
>>>>
>>>> Concerning the wake up of the cpu: the cpu disabled the irq and
>>>> goes to sleep, it is up to the firmware to wake up the cpu when an
>>>> interrupt occurs. It will exits its sleep state, call
>>>> clock_events_notify(EXIT), by this way re-switching to the local
>>>> timer, and then re-enabling the local interrupt which leads to the
>>>> interrupt handler.
>>>
>>>
>>> Thanks a lots for excellent article and detailed explains!
>>>
>>> So, if the firmware is in response to wake up cpu. that means there
>>> is a unit which control the firmware and it can not be power down.
>>
>>
>> Correct.
>>
>>
>>> Do you know which unit running the firmware to wake up deep idle
>>> CPU.
>>
>>
>> That depends on the SoC implementation.
> and which is intentionally kept hidden away.
>>
>> Some SoC have a "Power Management Unit". The PMU has several idle states
>> defined, each of them are described in the technical reference manual
>> (TRM) with the wake up sources.
> PMU is intentionally kept hidden by OEM companies as this way
> they protect there hardware IP.

Unfortunately yes and beside that hiding the bugs in a black box letting 
the user/developer to bang its head against the walls :)

>> Some SoC don't have any PMU and the idle states are very few, keeping
>> most of the logic on.
>>
>> Some other SoC hide the PMU behind PSCI calls.
> which is intentional.
>>> And does the wake up pass via GIC to CPU? If so, does the GIC need
>>> keep awake when all cpu idle? If not, how the firmware give the
>>> interrupt to CPU? And I am wondering if the deep idle cpu voltage get
>>> to near 0. How the cpu get the interrupt signal?
>>
>>
>> If a deep idle state powers down the GIC, it is up to the PMU to proxy
>> the interrupts. When an interrupt occurs, the PMU powers up the logic,
>> including the GIC. The notifier call chain with cpu_suspend / cpu_resume
>> will save and restore the GIC registers.
>>
>> But this is hardware specific and will depend on how the PMU is
>> implemented and how far it goes in the power management.
>>
>> You have a good example in the drivers/cpuidle/cpuidle-ux500.c to
>> understand with the comments how the interrupts are handled through the
>> power management unit.
>>
>> In the Xillinx documentation available on the web [1], the chapter 24.4
>> gives the information about one kind of PMU.
>>
>> I believe the mechanism is pretty similar on all the hardware but it is
>> obfuscated by a generic power instruction like mwait.
>>
>>    -- Daniel
>>
>> [1]
>> http://www.xilinx.com/support/documentation/user_guides/ug585-Zynq-7000-TRM.pdf
>>
>>
>>
>>>> There are some more informations in the wiki page [1].
>>>>
>>>> -- Daniel
>>>>
>>>> [1]
>>>> https://wiki.linaro.org/WorkingGroups/PowerManagement/Doc/WakeUpSources
>>>
>>>
>>>>
>>>
>>
>>
>> --
>>   <http://www.linaro.org/> Linaro.org │ Open source software for ARM SoCs
>>
>> Follow Linaro:  <http://www.facebook.com/pages/Linaro> Facebook |
>> <http://twitter.com/#!/linaroorg> Twitter |
>> <http://www.linaro.org/linaro-blog/> Blog
>>
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>> Please read the FAQ at  http://www.tux.org/lkml/


-- 
  <http://www.linaro.org/> Linaro.org │ Open source software for ARM SoCs

Follow Linaro:  <http://www.facebook.com/pages/Linaro> Facebook |
<http://twitter.com/#!/linaroorg> Twitter |
<http://www.linaro.org/linaro-blog/> Blog

Re: questions of cpuidle

[Alex Shi]

On 12/10/2013 03:27 PM, Daniel Lezcano wrote:
> 
>> And does the wake up pass via GIC to CPU? If so, does the GIC need
>> keep awake when all cpu idle? If not, how the firmware give the
>> interrupt to CPU? And I am wondering if the deep idle cpu voltage get
>> to near 0. How the cpu get the interrupt signal?
> 
> If a deep idle state powers down the GIC, it is up to the PMU to proxy
> the interrupts. When an interrupt occurs, the PMU powers up the logic,
> including the GIC. The notifier call chain with cpu_suspend / cpu_resume
> will save and restore the GIC registers.
> 
> But this is hardware specific and will depend on how the PMU is
> implemented and how far it goes in the power management.
> 
> You have a good example in the drivers/cpuidle/cpuidle-ux500.c to
> understand with the comments how the interrupts are handled through the
> power management unit.
> 
> In the Xillinx documentation available on the web [1], the chapter 24.4
> gives the information about one kind of PMU.
> 
> I believe the mechanism is pretty similar on all the hardware but it is
> obfuscated by a generic power instruction like mwait.
> 
>   -- Daniel
> 
> [1]
> http://www.xilinx.com/support/documentation/user_guides/ug585-Zynq-7000-TRM.pdf
> 

Thanks a lot, Daniel. You are a expert of ARM cpu idle! :)

-- 
Thanks
    Alex

Re: questions of cpuidle

[Preeti U Murthy]

Hi Alex,

On 12/09/2013 07:10 PM, Alex Shi wrote:
> 
> Sorry for a idiot of cpuidle.
> 
> I just find few cpu set TIMER_STOP on cpuidle, like omap4 and big.Little
> driver. Does that mean other ARM cpu or x86 cpu can get the timer
> interrupt in cpuidle?

There could be. Its only in some architectures and certain processors in
them that in a deep idle state(note that this happens only in a *deep*
idle state, *not* every idle state), the local timers stop as a side
effect and hence the CPUs can't get timer interrupts.

> 
> If the timer stopped during cpuidle, does that means at least one cpu
> cann't get into deep c-state since system need a cpu to wake up other
> deep c-state cpu? Or sth I missed?

Every CPU can get into a deep idle state since its possible to wake them
up using external interrupts. These CPUs do not necessarily need an IPI
to wake them up, any external interrupt will do. But for convenience,
instead of waking them all up using some external timer, we wake up only
one CPU or a few CPUs using an external timer and these CPUs inturn send
IPIs to the other CPUs in deep idle state since IPIs are a more
convenient way of sending interrupts to CPUs.

That is precisely the use of the broadcast timer. *Every* CPU can go
into deep idle state since they can be woken up by any external
interrupt, In case of architectures where there is an external clock
device, this external timer interrupt wakes up one CPU and it sends IPIs
to the other CPUs.

> 
> If the cpu stopped the interrupt during deep c-state and without
> monitor/mwait support, which kind of ipi can wake the cpu? I mean like a
> x86 cpu, APIC stopped in c3 mode, but actually ipi send via apic bus. So
> I don't know which ipi work?
> 

As far as my understanding goes, an external interrupt sent via the apic
bus wakes up a core in deep idle state first, meaning powers on the core
and hence the local apic. It does not yet acknowledge the interrupt,
meaning it cannot invoke the interrupt handler immediately.
   After the core goes through some initialization steps after wakeup,
it will be in a position to acknowledge the external interrupt and
service it accordingly.

Ideally the interrupt handler of this external interrupt should be that
of the local timer itself since it was meant to act on the behalf of the
local timer interrupt.

Thanks

Regards
Preeti U Murthy

Re: questions of cpuidle

[Alex Shi]

On 12/09/2013 11:26 PM, Preeti U Murthy wrote:
>> > If the cpu stopped the interrupt during deep c-state and without
>> > monitor/mwait support, which kind of ipi can wake the cpu? I mean like a
>> > x86 cpu, APIC stopped in c3 mode, but actually ipi send via apic bus. So
>> > I don't know which ipi work?
>> > 
> As far as my understanding goes, an external interrupt sent via the apic
> bus wakes up a core in deep idle state first,

Is there some evidence for this? Documents or some explanation?
 meaning powers on the core
> and hence the local apic. It does not yet acknowledge the interrupt,
> meaning it cannot invoke the interrupt handler immediately.
>    After the core goes through some initialization steps after wakeup,
> it will be in a position to acknowledge the external interrupt and
> service it accordingly.
> 
> Ideally the interrupt handler of this external interrupt should be that
> of the local timer itself since it was meant to act on the behalf of the
> local timer interrupt.
> 

Added more Intel experts.

Many thanks for response, Preeti!

But I still don't know how to get external/internal interrupt by a deep
c-state cpu.
In Intel Architecture Software Developer's Manual Vol.3A, Figure 10-3.
Local APICs and I/O APIC When P6 Family Processors Are Used in
Multiple-Processor Systems.
The Local APIC is response for the the external/internal interrupt
receiving. and It is included in CPU.

And some explanation often be used in wikipedia.
(http://www.hardwaresecrets.com/article/Everything-You-Need-to-Know-About-the-CPU-C-States-Power-Saving-Modes/611/4)
It said the APIC clock was stopped in deep c-state, So I am wondering
how can the nonfunctional LAPIC pass interrupt to CPU?

And for monitor/mwait idle method, seems deep c-state cpu need to keep a
eye on memory bus. So seems the memory controller in cpu package is
impossible to get into sleep, right?
-- 
Thanks
    Alex

Re: questions of cpuidle

[Arjan van de Ven]

On 12/9/2013 10:08 PM, Alex Shi wrote:
> And for monitor/mwait idle method, seems deep c-state cpu need to keep a
> eye on memory bus. So seems the memory controller in cpu package is
> impossible to get into sleep, right?
not the memory bus
the cache coherency logic ;-)

Re: questions of cpuidle

[Alex Shi]

On 12/10/2013 10:53 PM, Arjan van de Ven wrote:
> On 12/9/2013 10:08 PM, Alex Shi wrote:
>> And for monitor/mwait idle method, seems deep c-state cpu need to keep a
>> eye on memory bus. So seems the memory controller in cpu package is
>> impossible to get into sleep, right?
> not the memory bus
> the cache coherency logic ;-)
> 

Thanks a lot, Arjan!

But in Intel c7 state, l3 cache also flushed. Are there some other unit
monitor this?

-- 
Thanks
    Alex