[PSCI DISCUSS] How to implement standby and suspend-to-ram by PSCI

[PSCI DISCUSS] How to implement standby and suspend-to-ram by PSCI

[Hongbo Zhang]

Hi Guys,
We know in the legacy kernel suspend codes, a typical suspend_ops->enter 
callback is like this:

static int imx6q_pm_enter(suspend_state_t state)
{
	switch (state) {
	case PM_SUSPEND_STANDBY:
		do this;
		break;
	case PM_SUSPEND_MEM:
		do that;
		break;
	default:
		return -EINVAL;
	}
	return 0;
}

That means we implement standby and suspend-to-ram in one callback, and 
these two state can be identified by an input parameter 'state'.

While now PSCI is used for ARM to implement cores operations and power 
management, for the above suspend_ops->enter we choose SYSTEM_SUSPEND 
function, and this function is defined for suspend-to-ram, then here 
comes the problem:

How can we implement the standby state following the PSCI framework?

I couldn't find other PSCI function which fits for standby well.
In the spec there isn't such a 'state' parameter passed for 
SYSTEM_SUSPEND (only entry_point_address and context_id), can we change 
the spec a bit, e.g. using one more register to pass another 'state' 
parameter? is that acceptable? (similar like CPU_SUSPEND then)

Any other ideas, suggestions?

Thanks.

Re: [PSCI DISCUSS] How to implement standby and suspend-to-ram by PSCI

[Sudeep Holla]

Hi Hongbo,

On 06/05/16 11:11, Hongbo Zhang wrote:
> Hi Guys,
> We know in the legacy kernel suspend codes, a typical suspend_ops->enter
> callback is like this:
>

[...]

>
> That means we implement standby and suspend-to-ram in one callback, and
> these two state can be identified by an input parameter 'state'.
>
> While now PSCI is used for ARM to implement cores operations and power
> management, for the above suspend_ops->enter we choose SYSTEM_SUSPEND
> function, and this function is defined for suspend-to-ram, then here
> comes the problem:
>
> How can we implement the standby state following the PSCI framework?
>

Jisheng Zhang raised similar question few months back and we had a
discussion. Please have a look at the thread [1]

> I couldn't find other PSCI function which fits for standby well.
> In the spec there isn't such a 'state' parameter passed for
> SYSTEM_SUSPEND (only entry_point_address and context_id), can we change
> the spec a bit, e.g. using one more register to pass another 'state'
> parameter? is that acceptable? (similar like CPU_SUSPEND then)
>

Since the standby state is not well defined on ARM systems, it's hard to
define the conditions like system suspend and hence difficult to
generalize in the spec. I think suspend-to-idle should suffice in your
case. Anyways its better to get more information on the definition of
standby on your platform.

-- 
Regards,
Sudeep

[1] 
http://lists.infradead.org/pipermail/linux-arm-kernel/2015-July/356467.html

Re: [PSCI DISCUSS] How to implement standby and suspend-to-ram by PSCI

[Hongbo Zhang]

Hi Sudeep,

On 2016年05月06日 21:28, Sudeep Holla wrote:
> Hi Hongbo,
>
> On 06/05/16 11:11, Hongbo Zhang wrote:
>> Hi Guys,
>> We know in the legacy kernel suspend codes, a typical suspend_ops->enter
>> callback is like this:
>>
>
> [...]
>
>>
>> That means we implement standby and suspend-to-ram in one callback, and
>> these two state can be identified by an input parameter 'state'.
>>
>> While now PSCI is used for ARM to implement cores operations and power
>> management, for the above suspend_ops->enter we choose SYSTEM_SUSPEND
>> function, and this function is defined for suspend-to-ram, then here
>> comes the problem:
>>
>> How can we implement the standby state following the PSCI framework?
>>
>
> Jisheng Zhang raised similar question few months back and we had a
> discussion. Please have a look at the thread [1]
>
I looked through that discussion, it seems better I continue to reply 
there but I didn't book that mailing list, so I add linux-arm-kernel 
list and Jisheng here.

>> I couldn't find other PSCI function which fits for standby well.
>> In the spec there isn't such a 'state' parameter passed for
>> SYSTEM_SUSPEND (only entry_point_address and context_id), can we change
>> the spec a bit, e.g. using one more register to pass another 'state'
>> parameter? is that acceptable? (similar like CPU_SUSPEND then)
>>
>
> Since the standby state is not well defined on ARM systems, it's hard to
> define the conditions like system suspend and hence difficult to
> generalize in the spec. I think suspend-to-idle should suffice in your
> case. Anyways its better to get more information on the definition of
> standby on your platform.
>

On our platform we have some shallow states which maps to cpuidle well, 
and we have two more deeper states:

Sleep state:
All cores are in WFI state,
Core cluster is in standby state, e.g. L2 cache is flushed then stops 
snooping,
Only those modules which are required to wake up the device still have a 
running clock, other device modules in chip are clock gated.

Deep sleep state:
ARM cores and DDR controller are switched off,
DDR is in self-refresh mode,
VDD power to off domain IPs are removed, Only the blocks needed to 
detect wake up and sequence the chip out of deep sleep are on.

For the two states, except for the wake-up devices, all the other device 
dirver's suspend callback needs to be called because those devices will 
be clock or power gated off. So, the kernel suspend process (freeze 
tasks, suspend devices etc) are needed for our sleep and deep sleep states.
In fact we've already implemented these two stated by legacy way, e.g. 
mapping sleep state to STANDBY and deep sleep to MEM, both states are 
implemented in one traditional .enter callback and can be indexed by a 
'state' parameter.

While transferring to PSCI, we met the problem I mentioned.

As to the freeze(suspend-to-idle), it is described as "a generic, pure 
software, light-weight, system sleep state" in the Document, and the 
fact is only acpi is using it, what's more, I don't think it satisfy us 
well, because the freeze_enter() is executed before 
disable_nonboot_cpus() and syscore_suspend(), while the later two are 
necessary for us.

So it seems another 'state' parameter for SYSTEM_SUSPEND can settle this 
problem, but it is lack in the spec.

There are diverse kinds of ARM SOC, and divers kind of peripheral 
devices in each SOC, so even for the SYSTEM_SUSPEND, there may be 
different states too(deeper or shallower states of each device, more or 
less of devices are put to idle), the more the PSCI spec supports, the 
better.

Re: [PSCI DISCUSS] How to implement standby and suspend-to-ram by PSCI

[Sudeep Holla]

On 12/05/16 10:42, Hongbo Zhang wrote:
> Hi Sudeep,
> 

[...]

> 
> Sleep state:
> All cores are in WFI state,
> Core cluster is in standby state, e.g. L2 cache is flushed then stops
> snooping,
> Only those modules which are required to wake up the device still have a
> running clock, other device modules in chip are clock gated.
> 
> Deep sleep state:
> ARM cores and DDR controller are switched off,
> DDR is in self-refresh mode,
> VDD power to off domain IPs are removed, Only the blocks needed to
> detect wake up and sequence the chip out of deep sleep are on.
> 
> For the two states, except for the wake-up devices, all the other device
> dirver's suspend callback needs to be called because those devices will
> be clock or power gated off. So, the kernel suspend process (freeze
> tasks, suspend devices etc) are needed for our sleep and deep sleep states.
> In fact we've already implemented these two stated by legacy way, e.g.
> mapping sleep state to STANDBY and deep sleep to MEM, both states are
> implemented in one traditional .enter callback and can be indexed by a
> 'state' parameter.
> 

Yes I understand that.

> While transferring to PSCI, we met the problem I mentioned.
> 
> As to the freeze(suspend-to-idle), it is described as "a generic, pure
> software, light-weight, system sleep state" in the Document, and the
> fact is only acpi is using it, 

No, it independent of ACPI. It can be used on any platform with CPUIdle
support.

> what's more, I don't think it satisfy us
> well, because the freeze_enter() is executed before
> disable_nonboot_cpus() and syscore_suspend(), while the later two are
> necessary for us.
> 

Care to explain ? With suspend-to-idle, wakeup latency is much lesser as
the CPUs are not hotplugged out. I don't think it makes huge different
with power on ARM platforms. So its is much better than standby which
does CPU hotplugging IMO.

> So it seems another 'state' parameter for SYSTEM_SUSPEND can settle this
> problem, but it is lack in the spec.
> 

If you still think you can't make use of suspend-to-idle with added
advantage of reduced wakeup latency, please send email with all the
details to errata@arm.com as suggested in the PSCI specification.

> There are diverse kinds of ARM SOC, and divers kind of peripheral
> devices in each SOC, so even for the SYSTEM_SUSPEND, there may be
> different states too(deeper or shallower states of each device, more or
> less of devices are put to idle), the more the PSCI spec supports, the
> better.
> 

With runtime PM, you can work out on the peripheral device PM directly,
so I can't imagine how the diversity there adds to SYSTEM_SUSPEND
complexity.

-- 
Regards,
Sudeep

RE: [PSCI DISCUSS] How to implement standby and suspend-to-ram by PSCI

[Hongbo Zhang]

> -----Original Message-----
> From: Sudeep Holla [mailto:sudeep.holla@arm.com]
> Sent: Thursday, May 12, 2016 6:09 PM
> To: Hongbo Zhang <hongbo.zhang@nxp.com>; linux-pm@vger.kernel.org
> Cc: Sudeep Holla <sudeep.holla@arm.com>; Mark Rutland
> <mark.rutland@arm.com>; jszhang@marvell.com; Lorenzo Pieralisi
> <lorenzo.pieralisi@arm.com>; Frank Li <Frank.Li@freescale.com>; Marc Zyngier
> <marc.zyngier@arm.com>; Jan Kiszka <jan.kiszka@siemens.com>; Daniel
> Lezcano <daniel.lezcano@linaro.org>; Peng Fan <Peng.Fan@freescale.com>;
> Hans de Goede <hdegoede@redhat.com>; Chen-Yu Tsai <wens@csie.org>;
> Tom Warren <twarren@nvidia.com>; Vincent Guittot
> <vincent.guittot@linaro.org>; linux-arm-kernel@lists.infradead.org
> Subject: Re: [PSCI DISCUSS] How to implement standby and suspend-to-ram by
> PSCI
> 
> 
> 
> On 12/05/16 10:42, Hongbo Zhang wrote:
> > Hi Sudeep,
> >
> 
> [...]
> 
> >
> > Sleep state:
> > All cores are in WFI state,
> > Core cluster is in standby state, e.g. L2 cache is flushed then stops
> > snooping,
> > Only those modules which are required to wake up the device still have a
> > running clock, other device modules in chip are clock gated.
> >
> > Deep sleep state:
> > ARM cores and DDR controller are switched off,
> > DDR is in self-refresh mode,
> > VDD power to off domain IPs are removed, Only the blocks needed to
> > detect wake up and sequence the chip out of deep sleep are on.
> >
> > For the two states, except for the wake-up devices, all the other device
> > dirver's suspend callback needs to be called because those devices will
> > be clock or power gated off. So, the kernel suspend process (freeze
> > tasks, suspend devices etc) are needed for our sleep and deep sleep states.
> > In fact we've already implemented these two stated by legacy way, e.g.
> > mapping sleep state to STANDBY and deep sleep to MEM, both states are
> > implemented in one traditional .enter callback and can be indexed by a
> > 'state' parameter.
> >
> 
> Yes I understand that.
> 
> > While transferring to PSCI, we met the problem I mentioned.
> >
> > As to the freeze(suspend-to-idle), it is described as "a generic, pure
> > software, light-weight, system sleep state" in the Document, and the
> > fact is only acpi is using it,
> 
> No, it independent of ACPI. It can be used on any platform with CPUIdle
> support.
> 
> > what's more, I don't think it satisfy us
> > well, because the freeze_enter() is executed before
> > disable_nonboot_cpus() and syscore_suspend(), while the later two are
> > necessary for us.
> >
> 
> Care to explain ? With suspend-to-idle, wakeup latency is much lesser as
> the CPUs are not hotplugged out. I don't think it makes huge different
> with power on ARM platforms. So its is much better than standby which
> does CPU hotplugging IMO.
> 

Sorry for response late, I was investigating the possibility of implementing suspend-to-idle on our platform.
Our user manual says the core1 should enter wfi first and the core0 is the last core which is responsible for some cleanup such as triggering the clock gating off for non-wakeup devices.
If we use suspend-to-idle (no disable none boot cpu in this routine), which core enters idle last cannot be guaranteed, I don't know if core1 can do the cleanup instead, this cannot be confirmed from any document, so have to do a test later, this needs some time.

> > So it seems another 'state' parameter for SYSTEM_SUSPEND can settle this
> > problem, but it is lack in the spec.
> >
> 
> If you still think you can't make use of suspend-to-idle with added
> advantage of reduced wakeup latency, please send email with all the
> details to errata@arm.com as suggested in the PSCI specification.

Will consider this after my test I mention above, thanks for the info.

> 
> > There are diverse kinds of ARM SOC, and divers kind of peripheral
> > devices in each SOC, so even for the SYSTEM_SUSPEND, there may be
> > different states too(deeper or shallower states of each device, more or
> > less of devices are put to idle), the more the PSCI spec supports, the
> > better.
> >
> 
> With runtime PM, you can work out on the peripheral device PM directly,
> so I can't imagine how the diversity there adds to SYSTEM_SUSPEND
> complexity.

Our SoC isn't designed for mobile device, so the peripheral PM is quite rough, except for the wakeup devices, all the other devices can only be power or clock gated off by one operation at one time, they cannot be treated separately, so the runtime PM doesn't fit well.

> 
> --
> Regards,
> Sudeep

Re: [PSCI DISCUSS] How to implement standby and suspend-to-ram by PSCI

[Oliver Neukum]

On Thu, 2016-05-12 at 09:42 +0000, Hongbo Zhang wrote:
> On our platform we have some shallow states which maps to cpuidle
> well, 
> and we have two more deeper states:
> 
> Sleep state:
> All cores are in WFI state,
> Core cluster is in standby state, e.g. L2 cache is flushed then stops 
> snooping,
> Only those modules which are required to wake up the device still have
> a 
> running clock, other device modules in chip are clock gated.
> 
> Deep sleep state:
> ARM cores and DDR controller are switched off,
> DDR is in self-refresh mode,
> VDD power to off domain IPs are removed, Only the blocks needed to 
> detect wake up and sequence the chip out of deep sleep are on.

For the pm module of the core kernel a low power state is a sleep
state if even one of three conditions is met:

a) a device that can be used for input is not a source of wake ups
b) tasks need to be frozen to enter it
c) the display goes off, while it should be on

Generally runtime PM is better if you can get the same results.
Going to a system sleep state is a necessary evil.

	Regards
		Oliver

RE: [PSCI DISCUSS] How to implement standby and suspend-to-ram by PSCI

[Hongbo Zhang]

> -----Original Message-----
> From: Oliver Neukum [mailto:oneukum@suse.com]
> Sent: Tuesday, May 17, 2016 7:54 PM
> To: Hongbo Zhang <hongbo.zhang@nxp.com>
> Cc: Sudeep Holla <sudeep.holla@arm.com>; linux-pm@vger.kernel.org;
> Lorenzo Pieralisi <lorenzo.pieralisi@arm.com>; Marc Zyngier
> <marc.zyngier@arm.com>; Jan Kiszka <jan.kiszka@siemens.com>; Mark
> Rutland <mark.rutland@arm.com>; Daniel Lezcano
> <daniel.lezcano@linaro.org>; Chen-Yu Tsai <wens@csie.org>; Hans de Goede
> <hdegoede@redhat.com>; Frank Li <Frank.Li@freescale.com>; Peng Fan
> <Peng.Fan@freescale.com>; Tom Warren <twarren@nvidia.com>; Vincent
> Guittot <vincent.guittot@linaro.org>; jszhang@marvell.com; linux-arm-
> kernel@lists.infradead.org
> Subject: Re: [PSCI DISCUSS] How to implement standby and suspend-to-ram by
> PSCI
> 
> On Thu, 2016-05-12 at 09:42 +0000, Hongbo Zhang wrote:
> > On our platform we have some shallow states which maps to cpuidle
> > well,
> > and we have two more deeper states:
> >
> > Sleep state:
> > All cores are in WFI state,
> > Core cluster is in standby state, e.g. L2 cache is flushed then stops
> > snooping,
> > Only those modules which are required to wake up the device still have
> > a
> > running clock, other device modules in chip are clock gated.
> >
> > Deep sleep state:
> > ARM cores and DDR controller are switched off,
> > DDR is in self-refresh mode,
> > VDD power to off domain IPs are removed, Only the blocks needed to
> > detect wake up and sequence the chip out of deep sleep are on.
> 
> For the pm module of the core kernel a low power state is a sleep
> state if even one of three conditions is met:
> 
> a) a device that can be used for input is not a source of wake ups
> b) tasks need to be frozen to enter it
> c) the display goes off, while it should be on
> 
> Generally runtime PM is better if you can get the same results.
> Going to a system sleep state is a necessary evil.
> 
> 	Regards
> 		Oliver
> 
As I just explained in previous reply, many devices in our Soc cannot be physically clock or power gated, all the non-wakeup devices can only be gated off at same time by a PM module, so runtime PM doesn't fit for us well.

Thanks for your advice. 

Re: [PSCI DISCUSS] How to implement standby and suspend-to-ram by PSCI

[Oliver Neukum]

On Thu, 2016-05-19 at 10:27 +0000, Hongbo Zhang wrote:
> > For the pm module of the core kernel a low power state is a sleep
> > state if even one of three conditions is met:
> > 
> > a) a device that can be used for input is not a source of wake ups
> > b) tasks need to be frozen to enter it
> > c) the display goes off, while it should be on
> > 
> > Generally runtime PM is better if you can get the same results.
> > Going to a system sleep state is a necessary evil.


> As I just explained in previous reply, many devices in our Soc cannot
> be physically clock or power gated, all the non-wakeup devices can
> only be gated off at same time by a PM module, so runtime PM doesn't
> fit for us well.

You are looking for a contradiction where none is to be found. We have
PM domains which allow you to model common clocks and gates. You simply
use the driver core and it will use the common gate when every device is
idle. Obviously this is also true when the system is suspended. But not
necessarily only when the system is suspended.

	Regards
		Oliver

RE: [PSCI DISCUSS] How to implement standby and suspend-to-ram by PSCI

[Hongbo Zhang]

> -----Original Message-----
> From: Oliver Neukum [mailto:oneukum@suse.com]
> Sent: Thursday, May 19, 2016 8:01 PM
> To: Hongbo Zhang <hongbo.zhang@nxp.com>
> Cc: Sudeep Holla <sudeep.holla@arm.com>; linux-pm@vger.kernel.org;
> Lorenzo Pieralisi <lorenzo.pieralisi@arm.com>; Marc Zyngier
> <marc.zyngier@arm.com>; Jan Kiszka <jan.kiszka@siemens.com>; Mark
> Rutland <mark.rutland@arm.com>; Daniel Lezcano
> <daniel.lezcano@linaro.org>; Chen-Yu Tsai <wens@csie.org>; Hans de Goede
> <hdegoede@redhat.com>; Frank Li <Frank.Li@freescale.com>; Peng Fan
> <Peng.Fan@freescale.com>; Tom Warren <twarren@nvidia.com>; Vincent
> Guittot <vincent.guittot@linaro.org>; jszhang@marvell.com; linux-arm-
> kernel@lists.infradead.org
> Subject: Re: [PSCI DISCUSS] How to implement standby and suspend-to-ram by
> PSCI
> 
> On Thu, 2016-05-19 at 10:27 +0000, Hongbo Zhang wrote:
> > > For the pm module of the core kernel a low power state is a sleep
> > > state if even one of three conditions is met:
> > >
> > > a) a device that can be used for input is not a source of wake ups
> > > b) tasks need to be frozen to enter it
> > > c) the display goes off, while it should be on
> > >
> > > Generally runtime PM is better if you can get the same results.
> > > Going to a system sleep state is a necessary evil.
> 
> 
> > As I just explained in previous reply, many devices in our Soc cannot
> > be physically clock or power gated, all the non-wakeup devices can
> > only be gated off at same time by a PM module, so runtime PM doesn't
> > fit for us well.
> 
> You are looking for a contradiction where none is to be found. We have
> PM domains which allow you to model common clocks and gates. You simply
> use the driver core and it will use the common gate when every device is
> idle. Obviously this is also true when the system is suspended. But not
> necessarily only when the system is suspended.
> 
> 	Regards
> 		Oliver
> 

Understand.
Well, currently our situation is that we've already implemented our sleep/deep sleep mode (mapping to kernel terms standby/suspend to ram), and now while transferring to PSCI we meet such a problem.
If runtime PM is used, many device driver's developers have to be involved to offer their runtime callbacks, a lot of efforts needed in whole system level. But yes, runtime PM has its own advantages, will consider to apply it in the coming SOCs.
Thanks for advice.

Hongbo Zhang @ NXP