[Question] About handling PMU context loss in the deepest idle state where the core is powered down

[Question] About handling PMU context loss in the deepest idle state where the core is powered down

[Xiongfeng Wang]

Hi Will, Mark,

Sorry to bother you. It's just that we have come across some problems about PMU recently.

We are working on deep power state on CPU cores. In the deepest idle state, the core will be
powered down. In our implementation, the PMU and the core are in the same power domain,
so the PMU will also be powered down. But I didn't find where we saved the PMU context
in kernel before entering the deepest idle state.

Before we enter the system sleep state, we update the kernel PMU counter and stop the PMU
in 'cpu_pm_pmu_notify()'. But we didn't do that before we enter idle state.
I only find some system registers saving in 'psci_cpu_suspend_enter()->cpu_susend()->cpu_do_suspend()'

Do you have some suggestion about how we can handle this problem ?
Should we save and restore the PMU context before and after the deepest idle state . I don't know
if we can take care of the PMU  counters and interrupts properly by saving and restoring
the PMU registers.
Or we should update the kernel counter and stop the PMU, like what we did in system sleep situation.
Or we should let the firmware to handle the saving and restoring problem.

Thanks,
Xiongfeng


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [Question] About handling PMU context loss in the deepest idle state where the core is powered down

[Will Deacon]

[+Lorenzo]

On Thu, Jan 09, 2020 at 10:43:40AM +0800, Xiongfeng Wang wrote:
> Sorry to bother you. It's just that we have come across some problems
> about PMU recently.

No bother, and thanks for including the mailing list.

> We are working on deep power state on CPU cores. In the deepest idle state, the core will be
> powered down. In our implementation, the PMU and the core are in the same power domain,
> so the PMU will also be powered down. But I didn't find where we saved the PMU context
> in kernel before entering the deepest idle state.
> 
> Before we enter the system sleep state, we update the kernel PMU counter and stop the PMU
> in 'cpu_pm_pmu_notify()'. But we didn't do that before we enter idle state.
> I only find some system registers saving in 'psci_cpu_suspend_enter()->cpu_susend()->cpu_do_suspend()'

I'm not sure what you mean by "system sleep state", but if you're putting
the CPU into a state where register contents is lost, then I think you need
those PM notifiers to run. My understanding was that CPUidle took care of
this. Why is that not sufficient for you?

> Do you have some suggestion about how we can handle this problem ?
> Should we save and restore the PMU context before and after the deepest idle state . I don't know
> if we can take care of the PMU  counters and interrupts properly by saving and restoring
> the PMU registers.
> Or we should update the kernel counter and stop the PMU, like what we did in system sleep situation.
> Or we should let the firmware to handle the saving and restoring problem.

If the idle state is somehow autonomous (i.e. the kernel is not aware of
it), then I suppose firmware has to save/restore the register state that
is not otherwise preserved. However, I'd like to understand if this is
really what's happening and why CPUidle isn't just doing the right thing.

Thanks,

Will

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [Question] About handling PMU context loss in the deepest idle state where the core is powered down

[Lorenzo Pieralisi]

On Thu, Jan 09, 2020 at 03:03:19PM +0000, Will Deacon wrote:
> [+Lorenzo]
> 
> On Thu, Jan 09, 2020 at 10:43:40AM +0800, Xiongfeng Wang wrote:
> > Sorry to bother you. It's just that we have come across some problems
> > about PMU recently.
> 
> No bother, and thanks for including the mailing list.
> 
> > We are working on deep power state on CPU cores. In the deepest idle
> > state, the core will be powered down. In our implementation, the PMU
> > and the core are in the same power domain, so the PMU will also be
> > powered down. But I didn't find where we saved the PMU context in
> > kernel before entering the deepest idle state.
> > 
> > Before we enter the system sleep state, we update the kernel PMU
> > counter and stop the PMU in 'cpu_pm_pmu_notify()'. But we didn't do
> > that before we enter idle state.

ACPI or DT firmware ? I suspect that's ACPI, with LPI idle state
flags set to 0x0 (3.1.3 - save and restore flags):

http://infocenter.arm.com/help/topic/com.arm.doc.den0048a/DEN0048A_ARM_FFH_Specification.pdf

If that's the case a firmware update is needed (ie currently the kernel
expects the PMU state to be retained).

arch/arm64/kernel/cpuidle.c

ARM64_LPI_IS_RETENTION_STATE()

In DT in the PSCI CPUidle driver we run the notifiers irrespective
of the idle state depth so I don't think this behaviour can happen
in a DT bootstrapped system.

I am just guessing - please let me know if my assumption is correct.

> > I only find some system registers saving in 'psci_cpu_suspend_enter()->cpu_susend()->cpu_do_suspend()'
> 
> I'm not sure what you mean by "system sleep state"

I think they mean suspend-to-RAM - in suspend-to-RAM the notifiers
are run through syscore operations which are decoupled from CPUidle.

Regardless, CPUidle should call the notifiers if instructed by firmware
correctly.

Thanks,
Lorenzo

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [Question] About handling PMU context loss in the deepest idle state where the core is powered down

[Xiongfeng Wang]

On 2020/1/10 0:46, Lorenzo Pieralisi wrote:
> On Thu, Jan 09, 2020 at 03:03:19PM +0000, Will Deacon wrote:
>> [+Lorenzo]
>>
>> On Thu, Jan 09, 2020 at 10:43:40AM +0800, Xiongfeng Wang wrote:
>>> Sorry to bother you. It's just that we have come across some problems
>>> about PMU recently.
>>
>> No bother, and thanks for including the mailing list.
>>
>>> We are working on deep power state on CPU cores. In the deepest idle
>>> state, the core will be powered down. In our implementation, the PMU
>>> and the core are in the same power domain, so the PMU will also be
>>> powered down. But I didn't find where we saved the PMU context in
>>> kernel before entering the deepest idle state.
>>>
>>> Before we enter the system sleep state, we update the kernel PMU
>>> counter and stop the PMU in 'cpu_pm_pmu_notify()'. But we didn't do
>>> that before we enter idle state.
> 
> ACPI or DT firmware ? I suspect that's ACPI, with LPI idle state
> flags set to 0x0 (3.1.3 - save and restore flags):
> 
> http://infocenter.arm.com/help/topic/com.arm.doc.den0048a/DEN0048A_ARM_FFH_Specification.pdf
> 
> If that's the case a firmware update is needed (ie currently the kernel
> expects the PMU state to be retained).
> 
> arch/arm64/kernel/cpuidle.c
> 
> ARM64_LPI_IS_RETENTION_STATE()

That totally solved my problem.
I set the LPI idle state flag in firmware, and the 'cpu_pm_pmu_notify()' can be called
before I enter the context-lost idle state. Thanks a lot !

> 
> In DT in the PSCI CPUidle driver we run the notifiers irrespective
> of the idle state depth so I don't think this behaviour can happen
> in a DT bootstrapped system.
> 
> I am just guessing - please let me know if my assumption is correct.

Yes, it's correct. We are using ACPI.

> 
>>> I only find some system registers saving in 'psci_cpu_suspend_enter()->cpu_susend()->cpu_do_suspend()'
>>
>> I'm not sure what you mean by "system sleep state"
> 
> I think they mean suspend-to-RAM - in suspend-to-RAM the notifiers
> are run through syscore operations which are decoupled from CPUidle.

Yes, I mean suspend-to-RAM.

Thanks,
Xiongfeng

> 
> Regardless, CPUidle should call the notifiers if instructed by firmware
> correctly.
> 
> Thanks,
> Lorenzo
> 
> .
> 


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [Question] About handling PMU context loss in the deepest idle state where the core is powered down

[Xiongfeng Wang]

On 2020/1/9 23:03, Will Deacon wrote:
> [+Lorenzo]
> 
> On Thu, Jan 09, 2020 at 10:43:40AM +0800, Xiongfeng Wang wrote:
>> Sorry to bother you. It's just that we have come across some problems
>> about PMU recently.
> 
> No bother, and thanks for including the mailing list.
> 
>> We are working on deep power state on CPU cores. In the deepest idle state, the core will be
>> powered down. In our implementation, the PMU and the core are in the same power domain,
>> so the PMU will also be powered down. But I didn't find where we saved the PMU context
>> in kernel before entering the deepest idle state.
>>
>> Before we enter the system sleep state, we update the kernel PMU counter and stop the PMU
>> in 'cpu_pm_pmu_notify()'. But we didn't do that before we enter idle state.
>> I only find some system registers saving in 'psci_cpu_suspend_enter()->cpu_susend()->cpu_do_suspend()'
> 
> I'm not sure what you mean by "system sleep state", but if you're putting
> the CPU into a state where register contents is lost, then I think you need
> those PM notifiers to run. My understanding was that CPUidle took care of
> this. Why is that not sufficient for you?

Sorry, I didn't notice the PM notifiers are called in CPUidle when 'arch_flags' is set.
I thought it's only called in 'cpu_pm_syscore_ops', I mean Susepnd-to-RAM.

Thanks,
Xiongfeng

> 
>> Do you have some suggestion about how we can handle this problem ?
>> Should we save and restore the PMU context before and after the deepest idle state . I don't know
>> if we can take care of the PMU  counters and interrupts properly by saving and restoring
>> the PMU registers.
>> Or we should update the kernel counter and stop the PMU, like what we did in system sleep situation.
>> Or we should let the firmware to handle the saving and restoring problem.
> 
> If the idle state is somehow autonomous (i.e. the kernel is not aware of
> it), then I suppose firmware has to save/restore the register state that
> is not otherwise preserved. However, I'd like to understand if this is
> really what's happening and why CPUidle isn't just doing the right thing.
> 
> Thanks,
> 
> Will
> 
> .
> 


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel