[PATCH 00/04][RFC] PM: Runtime platform device PM

[PATCH 00/04][RFC] PM: Runtime platform device PM

[Magnus Damm]

PM: Runtime platform device power management

[PATCH 01/04] Driver Core: Add platform device arch data
[PATCH 02/04] Driver Core: Add idle and wakeup functions
[PATCH 03/04] PM: Add platform bus runtime dev_pm_ops
[PATCH 04/04] sh: Runtime platform device PM mockup

These patches extend platform device and driver interfaces to
allow architectures to implement platform device runtime pm.

Upstream runtime power management needs to be improved to fully
make use of hardware power saving features found on embedded
platforms and in common SoCs.

For runtime power management we today have cpuidle, the clock
framework and qos. This allows the cpu core to enter various
forms of deep sleep, and for devices we may stop clocks to save
power. Modern SoCs however allow disabling of power to parts of
the chip, and we have no upstream interface to handle that today.

I propose adding the following simple platform device functions:
 - platform_device_wakeup()
 - platform_device_idle()

The idle function is used by the platform driver to let the 
architecture power management code know that from now on the
device is in idle state. When the device is marked as idle
the architecture specific runtime power management may decide
to do various levels of device power management, ranging from
stopping clocks to turning off power. The dev_pm_ops callbacks
may be invoked by the runtime pm code to save and restore state
whenever the device is marked as idle.

The wakeup function is used by the platform driver to notify the
architecture code that the driver wants to make use of the hardware
device. If the device has been put in sleep then it needs to be
woken up. This wakeup call may invoke dev_pm_ops callbacks.

Have a look at the last patch for a SuperH mockup that shows how
it all fits together. We need this or a similar interface to be able
to enter deep sleep states on SuperH. I believe other architectures
have similar requirements.

Signed-off-by: Magnus Damm <damm@igel.co.jp>
---

 arch/Kconfig                               |    9 +
 arch/sh/Kconfig                            |    3 
 arch/sh/boards/mach-migor/setup.c          |    9 +
 arch/sh/include/asm/device.h               |    6 
 arch/sh/include/cpu-sh4/cpu/sh7722.h       |   14 ++
 arch/sh/kernel/cpu/sh4a/setup-sh7722.c     |   31 ++++
 arch/sh/kernel/cpu/shmobile/pm.c           |  118 +++++++++++++++++
 drivers/base/platform.c                    |  193 +++++++++++++++++++++++++++-
 drivers/clocksource/sh_cmt.c               |    2 
 drivers/clocksource/sh_tmu.c               |    2 
 drivers/i2c/busses/i2c-sh_mobile.c         |    2 
 drivers/media/video/sh_mobile_ceu_camera.c |    2 
 drivers/video/sh_mobile_lcdcfb.c           |    4 
 include/linux/platform_device.h            |   21 +++
 14 files changed, 415 insertions(+), 1 deletion(-)

Re: [linux-pm] [PATCH 00/04][RFC] PM: Runtime platform device PM

[Mark Brown]

On Wed, May 27, 2009 at 07:06:25PM +0900, Magnus Damm wrote:

> These patches extend platform device and driver interfaces to
> allow architectures to implement platform device runtime pm.

> Have a look at the last patch for a SuperH mockup that shows how
> it all fits together. We need this or a similar interface to be able
> to enter deep sleep states on SuperH. I believe other architectures
> have similar requirements.

ARM definitely does.  The OMAP SoCs are probably furthest along in
software support for this sort of functionality but it's pretty much
standard for modern ARMs.  I'd CC in the ARM list but it's subscribers
only, though the OMAP folks are also over on linux-omap@vger.kernel.org
which isn't.

Re: [PATCH 00/04][RFC] PM: Runtime platform device PM

[Alan Stern]

On Wed, 27 May 2009, Magnus Damm wrote:

> PM: Runtime platform device power management
> 
> [PATCH 01/04] Driver Core: Add platform device arch data
> [PATCH 02/04] Driver Core: Add idle and wakeup functions
> [PATCH 03/04] PM: Add platform bus runtime dev_pm_ops
> [PATCH 04/04] sh: Runtime platform device PM mockup
> 
> These patches extend platform device and driver interfaces to
> allow architectures to implement platform device runtime pm.
> 
> Upstream runtime power management needs to be improved to fully
> make use of hardware power saving features found on embedded
> platforms and in common SoCs.
> 
> For runtime power management we today have cpuidle, the clock
> framework and qos. This allows the cpu core to enter various
> forms of deep sleep, and for devices we may stop clocks to save
> power. Modern SoCs however allow disabling of power to parts of
> the chip, and we have no upstream interface to handle that today.
> 
> I propose adding the following simple platform device functions:
>  - platform_device_wakeup()
>  - platform_device_idle()
> 
> The idle function is used by the platform driver to let the 
> architecture power management code know that from now on the
> device is in idle state. When the device is marked as idle
> the architecture specific runtime power management may decide
> to do various levels of device power management, ranging from
> stopping clocks to turning off power. The dev_pm_ops callbacks
> may be invoked by the runtime pm code to save and restore state
> whenever the device is marked as idle.
> 
> The wakeup function is used by the platform driver to notify the
> architecture code that the driver wants to make use of the hardware
> device. If the device has been put in sleep then it needs to be
> woken up. This wakeup call may invoke dev_pm_ops callbacks.
> 
> Have a look at the last patch for a SuperH mockup that shows how
> it all fits together. We need this or a similar interface to be able
> to enter deep sleep states on SuperH. I believe other architectures
> have similar requirements.

Have you given any thought as to whether the platform_device_wakeup and 
platform_device_idle calls should be restricted to process context?

You should consider using a special workqueue for this stuff instead of 
relying on the default workqueue.  It can help avoid deadlocks, and it 
has the advantage that you can define the workqueue to be freezable.  
(Generally speaking, you don't want platform-level wakeup and idle 
calls to start running spontaneously in the middle of a system sleep 
transition.)

Alan Stern

Re: [PATCH 00/04][RFC] PM: Runtime platform device PM

[Kevin Hilman]

Magnus Damm <magnus.damm@gmail.com> writes:

> PM: Runtime platform device power management
>
> [PATCH 01/04] Driver Core: Add platform device arch data
> [PATCH 02/04] Driver Core: Add idle and wakeup functions
> [PATCH 03/04] PM: Add platform bus runtime dev_pm_ops
> [PATCH 04/04] sh: Runtime platform device PM mockup
>
> These patches extend platform device and driver interfaces to
> allow architectures to implement platform device runtime pm.
>
> Upstream runtime power management needs to be improved to fully
> make use of hardware power saving features found on embedded
> platforms and in common SoCs.
>
> For runtime power management we today have cpuidle, the clock
> framework and qos. This allows the cpu core to enter various
> forms of deep sleep, and for devices we may stop clocks to save
> power. Modern SoCs however allow disabling of power to parts of
> the chip, and we have no upstream interface to handle that today.
>
> I propose adding the following simple platform device functions:
>  - platform_device_wakeup()
>  - platform_device_idle()
>
> The idle function is used by the platform driver to let the 
> architecture power management code know that from now on the
> device is in idle state. When the device is marked as idle
> the architecture specific runtime power management may decide
> to do various levels of device power management, ranging from
> stopping clocks to turning off power. The dev_pm_ops callbacks
> may be invoked by the runtime pm code to save and restore state
> whenever the device is marked as idle.
>
> The wakeup function is used by the platform driver to notify the
> architecture code that the driver wants to make use of the hardware
> device. If the device has been put in sleep then it needs to be
> woken up. This wakeup call may invoke dev_pm_ops callbacks.
>
> Have a look at the last patch for a SuperH mockup that shows how
> it all fits together. We need this or a similar interface to be able
> to enter deep sleep states on SuperH. I believe other architectures
> have similar requirements.
>
> Signed-off-by: Magnus Damm <damm@igel.co.jp>

Speaking on behalf of OMAP PM developers, and as maintainer of the
current OMAP PM infrastructure, this is a major step towards a generic
runtime PM that we would use on OMAP as well.

In the PM branch of the linux-omap tree, we currently have an ad-hoc
but working infrastructure for all of this, including device-specific
save/restore hooks for supporting some or all of the on-chip
powerdomains to go off during idle.

However, our current approach hides most of the details behind the
clock framework, custom save/restore hooks and in some cases custom
platform hooks passed using platform_data.

I like this proposed approach much better as it generalizes the
approach and allows us to share the infrastructure across arches
as well as decouple device idleness from clock management.

Kevin

Re: [linux-pm] [PATCH 00/04][RFC] PM: Runtime platform device PM

[Magnus Damm]

On Wed, May 27, 2009 at 9:10 PM, Mark Brown
<broonie@opensource.wolfsonmicro.com> wrote:
> On Wed, May 27, 2009 at 07:06:25PM +0900, Magnus Damm wrote:
>
>> These patches extend platform device and driver interfaces to
>> allow architectures to implement platform device runtime pm.
>
>> Have a look at the last patch for a SuperH mockup that shows how
>> it all fits together. We need this or a similar interface to be able
>> to enter deep sleep states on SuperH. I believe other architectures
>> have similar requirements.
>
> ARM definitely does.  The OMAP SoCs are probably furthest along in
> software support for this sort of functionality but it's pretty much
> standard for modern ARMs.  I'd CC in the ARM list but it's subscribers
> only, though the OMAP folks are also over on linux-omap@vger.kernel.org
> which isn't.

I'll include linux-omap next time. Thanks for your suggestion!

/ magnus

Re: [PATCH 00/04][RFC] PM: Runtime platform device PM

[Magnus Damm]

On Wed, May 27, 2009 at 11:30 PM, Alan Stern <stern@rowland.harvard.edu> wrote:
> On Wed, 27 May 2009, Magnus Damm wrote:
>> PM: Runtime platform device power management

> Have you given any thought as to whether the platform_device_wakeup and
> platform_device_idle calls should be restricted to process context?

Good question. The first thing that pops into my mind is to have same
restrictions as clk_enable() and clk_disable(). To keep things simple
I'd say that it's unlikely that any embedded platform driver would
need to sleep during the dev_pm_ops callbacks. Maybe it makes sense to
only allow _noirq variants of dev_pm_ops, not sure. Any ideas?

> You should consider using a special workqueue for this stuff instead of
> relying on the default workqueue.  It can help avoid deadlocks, and it
> has the advantage that you can define the workqueue to be freezable.
> (Generally speaking, you don't want platform-level wakeup and idle
> calls to start running spontaneously in the middle of a system sleep
> transition.)

Yeah, the mockup code needs more work. =) For SuperH Mobile I suspect
that we don't need any workqueues since the freeze() and disabling of
power has to be done from cpuidle context. For SoCs with more complex
power domain layouts it may make sense handle each power domain
independently. And on top of this we probably want to have some QoS
information so the runtime pm code can chose which device sleep mode
to enter. Like cpuidle but for devices or power domains.

Thanks for your comments!

/ magnus

Re: [PATCH 00/04][RFC] PM: Runtime platform device PM

[Rafael J. Wysocki]

On Thursday 28 May 2009, Magnus Damm wrote:
> On Wed, May 27, 2009 at 11:30 PM, Alan Stern <stern@rowland.harvard.edu> wrote:
> > On Wed, 27 May 2009, Magnus Damm wrote:
> >> PM: Runtime platform device power management
> 
> > Have you given any thought as to whether the platform_device_wakeup and
> > platform_device_idle calls should be restricted to process context?
> 
> Good question. The first thing that pops into my mind is to have same
> restrictions as clk_enable() and clk_disable(). To keep things simple
> I'd say that it's unlikely that any embedded platform driver would
> need to sleep during the dev_pm_ops callbacks. Maybe it makes sense to
> only allow _noirq variants of dev_pm_ops, not sure. Any ideas?
> 
> > You should consider using a special workqueue for this stuff instead of
> > relying on the default workqueue.  It can help avoid deadlocks, and it
> > has the advantage that you can define the workqueue to be freezable.
> > (Generally speaking, you don't want platform-level wakeup and idle
> > calls to start running spontaneously in the middle of a system sleep
> > transition.)
> 
> Yeah, the mockup code needs more work. =) For SuperH Mobile I suspect
> that we don't need any workqueues since the freeze() and disabling of
> power has to be done from cpuidle context. For SoCs with more complex
> power domain layouts it may make sense handle each power domain
> independently. And on top of this we probably want to have some QoS
> information so the runtime pm code can chose which device sleep mode
> to enter. Like cpuidle but for devices or power domains.

I think we'll need one separate workqueue for run-time PM in general, so that
bus types don't introduce their own workqueues for this purpose.  IMO one
system-wide run-time PM workqueue should be sufficient (it could also be
used for the suspend blockers BTW).

So, perhaps it makes sense to implement such a workqueue at the core level?
Thoughts?

Rafael

Re: [PATCH 00/04][RFC] PM: Runtime platform device PM

[Alan Stern]

On Thu, 28 May 2009, Rafael J. Wysocki wrote:

> I think we'll need one separate workqueue for run-time PM in general, so that
> bus types don't introduce their own workqueues for this purpose.  IMO one
> system-wide run-time PM workqueue should be sufficient (it could also be
> used for the suspend blockers BTW).
> 
> So, perhaps it makes sense to implement such a workqueue at the core level?
> Thoughts?

That's fine with me.  This new workqueue can take over the job of the 
"ksuspend_usbd" workqueue.  But it would have to be marked as 
freezable.

Alan Stern

Re: [PATCH 00/04][RFC] PM: Runtime platform device PM

[Alan Stern]

On Thu, 28 May 2009, Magnus Damm wrote:

> On Wed, May 27, 2009 at 11:30 PM, Alan Stern <stern@rowland.harvard.edu> wrote:
> > On Wed, 27 May 2009, Magnus Damm wrote:
> >> PM: Runtime platform device power management
> 
> > Have you given any thought as to whether the platform_device_wakeup and
> > platform_device_idle calls should be restricted to process context?
> 
> Good question. The first thing that pops into my mind is to have same
> restrictions as clk_enable() and clk_disable(). To keep things simple
> I'd say that it's unlikely that any embedded platform driver would
> need to sleep during the dev_pm_ops callbacks. Maybe it makes sense to
> only allow _noirq variants of dev_pm_ops, not sure. Any ideas?

I don't know what would be appropriate for you.  USB power transitions
cannot be made in an atomic context, but the situation could well be
different for platform devices.  It's just something to be aware of.

(And by the way, the _noirq ops don't run in atomic context.  They run 
in process context with most interrupt deliveries disabled.  It's not 
the same thing -- they are allowed to sleep.)

Alan Stern

Re: [PATCH 00/04][RFC] PM: Runtime platform device PM

[Kevin Hilman]

Magnus Damm <magnus.damm@gmail.com> writes:

> For runtime power management we today have cpuidle, the clock
> framework and qos. This allows the cpu core to enter various
> forms of deep sleep, and for devices we may stop clocks to save
> power. Modern SoCs however allow disabling of power to parts of
> the chip, and we have no upstream interface to handle that today.
>
> I propose adding the following simple platform device functions:
>  - platform_device_wakeup()
>  - platform_device_idle()
>
> The idle function is used by the platform driver to let the 
> architecture power management code know that from now on the
> device is in idle state. When the device is marked as idle
> the architecture specific runtime power management may decide
> to do various levels of device power management, ranging from
> stopping clocks to turning off power. The dev_pm_ops callbacks
> may be invoked by the runtime pm code to save and restore state
> whenever the device is marked as idle.
>
> The wakeup function is used by the platform driver to notify the
> architecture code that the driver wants to make use of the hardware
> device. If the device has been put in sleep then it needs to be
> woken up. This wakeup call may invoke dev_pm_ops callbacks.

A couple comments/questions:

Wouldn't the 'wakeup' hook be better named platform_device_enable().
The _wakeup name seems to imply that the call will perform some sort
of wakeup, but all it's really doing is enabling it by turning it on
or taking it out of idle.

In the process of discussing a similar interface for OMAP, we dicussed
that having 3 states would be more useful.  Specifically, and
_enable(), _idle() and _disable() hook.  The _enable() and _idle()
hooks being exactly what you proposed above, but with the addition of
a _disable() hook which says not only can the device go idle, but that
the driver is really finished with the device.  In this case, more
aggresive PM measures could be taken, such as turning of regulators
that may have long latencies that may not be appropriate to turn off
in idle.

I'm interested in any thoughts there because the line between _idle()
and _disable() remains a little fuzzy to me still.  For exmaple, maybe
the _idle() call in combination with latency constraints in PM QoS
could do effectively the same thing as _disable().

Kevin

Re: [PATCH 00/04][RFC] PM: Runtime platform device PM

[Magnus Damm]

On Fri, May 29, 2009 at 12:33 AM, Alan Stern <stern@rowland.harvard.edu> wrote:
> On Thu, 28 May 2009, Magnus Damm wrote:
>
>> On Wed, May 27, 2009 at 11:30 PM, Alan Stern <stern@rowland.harvard.edu> wrote:
>> > On Wed, 27 May 2009, Magnus Damm wrote:
>> >> PM: Runtime platform device power management
>>
>> > Have you given any thought as to whether the platform_device_wakeup and
>> > platform_device_idle calls should be restricted to process context?
>>
>> Good question. The first thing that pops into my mind is to have same
>> restrictions as clk_enable() and clk_disable(). To keep things simple
>> I'd say that it's unlikely that any embedded platform driver would
>> need to sleep during the dev_pm_ops callbacks. Maybe it makes sense to
>> only allow _noirq variants of dev_pm_ops, not sure. Any ideas?
>
> I don't know what would be appropriate for you.  USB power transitions
> cannot be made in an atomic context, but the situation could well be
> different for platform devices.  It's just something to be aware of.

For our SuperH platform devices there should be no need to sleep.

> (And by the way, the _noirq ops don't run in atomic context.  They run
> in process context with most interrupt deliveries disabled.  It's not
> the same thing -- they are allowed to sleep.)

Huh, so if they are allowed to sleep then clock events are still
running. And probably some clocksource as well. I guess that's why you
say "most" interrupts disabled instead of all. Sharing the timer IRQ
is not allowed then?

I wonder if the PM code assumes that the clock event is a sys device?
We use platform drivers for clock events on SuperH...

/ magnus

Re: [PATCH 00/04][RFC] PM: Runtime platform device PM

[Alan Stern]

On Fri, 29 May 2009, Magnus Damm wrote:

> > (And by the way, the _noirq ops don't run in atomic context.  They run
> > in process context with most interrupt deliveries disabled.  It's not
> > the same thing -- they are allowed to sleep.)
> 
> Huh, so if they are allowed to sleep then clock events are still
> running. And probably some clocksource as well. I guess that's why you
> say "most" interrupts disabled instead of all. Sharing the timer IRQ
> is not allowed then?

Rafael can tell you.  But notice I didn't say the interrupts were
disabled -- I said that interrupt _delivery_ was disabled.  In general
the interrupts themselves _are_ enabled; the kernel fields them but
then does not call the drivers' interrupt handler routines.  (Except
perhaps for IRQs which are explicitly marked as wakeup sources...)

> I wonder if the PM code assumes that the clock event is a sys device?
> We use platform drivers for clock events on SuperH...

Ask Rafael.

Alan Stern

Re: [PATCH 00/04][RFC] PM: Runtime platform device PM

[Magnus Damm]

On Fri, May 29, 2009 at 2:14 AM, Kevin Hilman
<khilman@deeprootsystems.com> wrote:
> Magnus Damm <magnus.damm@gmail.com> writes:
>> I propose adding the following simple platform device functions:
>>  - platform_device_wakeup()
>>  - platform_device_idle()

> A couple comments/questions:
>
> Wouldn't the 'wakeup' hook be better named platform_device_enable().
> The _wakeup name seems to imply that the call will perform some sort
> of wakeup, but all it's really doing is enabling it by turning it on
> or taking it out of idle.

The names platform_device_enable() and platform_device_disable() were
my first pick. Similar to pci_enable_device() and
pci_disable_device(). People may mixup platform_device_wakeup() and
pci_enable_wake() so avoiding using wakeup as name may be a good plan.
Not sure what would be a better name though.

Now why I didn't go with enable() and disable() was that dev_pm_ops
callbacks may be called for a device that has been "disabled". This
seems a bit strange to me. Making the interface clear and show that
this is not a normal enable() and disable() interface may be a good
idea.

Or maybe it's perfectly acceptable that a "disabled" device may get
it's dev_pm_ops callbacks exercised?

> In the process of discussing a similar interface for OMAP, we dicussed
> that having 3 states would be more useful.  Specifically, and
> _enable(), _idle() and _disable() hook.  The _enable() and _idle()
> hooks being exactly what you proposed above, but with the addition of
> a _disable() hook which says not only can the device go idle, but that
> the driver is really finished with the device.  In this case, more
> aggresive PM measures could be taken, such as turning of regulators
> that may have long latencies that may not be appropriate to turn off
> in idle.

Hm.. I wonder when the driver is really finished with the device
though. Only when the module is unloaded? If so then we could deal
with the hard disable using platform bus notifiers (like in [04/04]).

Or is it common that the driver does disable() at some point and
depending on the work load may have to perform a enable() soon again?

disable() seems like a more heavy weight clk_disable()? Maybe we can
let i2c drivers use enable() and disable() in their ->master_xfer()
callback. So the natural state of the i2c master device is off, and
enable() is called before data transfer in ->master_xfer(), followed
by a disable(). I do exactly this but with the clock framework in
i2c-sh_mobile.c.

The problem with the i2c example above is that the driver writer has
no clue about the frequency of the enable() and disable() calls. So
maybe disable() should be replaced with idle()? But if so when is the
real use for disable(). I'd say that the runtime PM core should decide
which state to enter, and the driver can can help out by giving
latency requirements as input. That's it.

So I'm not really sure when we need this hard disable(). Do you have
any special case in mind?

> I'm interested in any thoughts there because the line between _idle()
> and _disable() remains a little fuzzy to me still.  For exmaple, maybe
> the _idle() call in combination with latency constraints in PM QoS
> could do effectively the same thing as _disable().

Aren't two functions (wakeup()/idle() or enable()/disable()) together
with PM QoS latency enough?

Yeah, the fuzzy line is interesting. =)

/ magnus

Re: [PATCH 00/04][RFC] PM: Runtime platform device PM

[Rafael J. Wysocki]

On Friday 29 May 2009, Alan Stern wrote:
> On Fri, 29 May 2009, Magnus Damm wrote:
> 
> > > (And by the way, the _noirq ops don't run in atomic context.  They run
> > > in process context with most interrupt deliveries disabled.  It's not
> > > the same thing -- they are allowed to sleep.)
> > 
> > Huh, so if they are allowed to sleep then clock events are still
> > running. And probably some clocksource as well. I guess that's why you
> > say "most" interrupts disabled instead of all. Sharing the timer IRQ
> > is not allowed then?
> 
> Rafael can tell you.  But notice I didn't say the interrupts were
> disabled -- I said that interrupt _delivery_ was disabled.  In general
> the interrupts themselves _are_ enabled; the kernel fields them but
> then does not call the drivers' interrupt handler routines.

That's correct, except for the platforms where __disable_irq() actually masks
the IRQ.  I'm not sure if SuperH is one of these.

> (Except perhaps for IRQs which are explicitly marked as wakeup sources...)

No, they are treated just like the other ones except that we check if any of
them are pending right at the beginning of sysdev_suspend().

> > I wonder if the PM code assumes that the clock event is a sys device?

Yes, it does.

> > We use platform drivers for clock events on SuperH...

Argh.  This is going to hurt, but I'm not sure how badly.  At least on
multiprocessor.

Best,
Rafael

Re: [PATCH 00/04][RFC] PM: Runtime platform device PM

[Rafael J. Wysocki]

On Thursday 28 May 2009, Alan Stern wrote:
> On Thu, 28 May 2009, Rafael J. Wysocki wrote:
> 
> > I think we'll need one separate workqueue for run-time PM in general, so that
> > bus types don't introduce their own workqueues for this purpose.  IMO one
> > system-wide run-time PM workqueue should be sufficient (it could also be
> > used for the suspend blockers BTW).
> > 
> > So, perhaps it makes sense to implement such a workqueue at the core level?
> > Thoughts?
> 
> That's fine with me.  This new workqueue can take over the job of the 
> "ksuspend_usbd" workqueue.  But it would have to be marked as 
> freezable.

Having considered it for a while I'm not sure if freezing is the right approach
here.  Namely, the work items put into the workqueue need not be useful after
the resume any more, so perhaps we should deqeue them during suspend?

Best,
Rafael

Re: [PATCH 00/04][RFC] PM: Runtime platform device PM

[Alan Stern]

On Mon, 1 Jun 2009, Rafael J. Wysocki wrote:

> On Thursday 28 May 2009, Alan Stern wrote:
> > On Thu, 28 May 2009, Rafael J. Wysocki wrote:
> > 
> > > I think we'll need one separate workqueue for run-time PM in general, so that
> > > bus types don't introduce their own workqueues for this purpose.  IMO one
> > > system-wide run-time PM workqueue should be sufficient (it could also be
> > > used for the suspend blockers BTW).
> > > 
> > > So, perhaps it makes sense to implement such a workqueue at the core level?
> > > Thoughts?
> > 
> > That's fine with me.  This new workqueue can take over the job of the 
> > "ksuspend_usbd" workqueue.  But it would have to be marked as 
> > freezable.
> 
> Having considered it for a while I'm not sure if freezing is the right approach
> here.  Namely, the work items put into the workqueue need not be useful after
> the resume any more, so perhaps we should deqeue them during suspend?

With USB at least, this isn't true.  Some work items shouldn't be
dequeued, namely, various sorts of resume requests generated while the
driver is in interrupt context.  If they were dequeued during a system
suspend then the device wouldn't ever be woken up, because usbcore
doesn't pass system-resume messages to an autosuspended device.  The
idea is if a device was autosuspended before the system sleep then it
should remain autosuspended after the system wakes up.

Furthermore, even if the existing work items were dequeued, you'd still 
have to worry about new work items added on by drivers before they get 
suspended.  If the workqueue were allowed to run freely, we might find 
devices being autoresumed in the middle of the sleep transition!

Alan Stern

Re: [PATCH 00/04][RFC] PM: Runtime platform device PM

[Rafael J. Wysocki]

On Monday 01 June 2009, Alan Stern wrote:
> On Mon, 1 Jun 2009, Rafael J. Wysocki wrote:
> 
> > On Thursday 28 May 2009, Alan Stern wrote:
> > > On Thu, 28 May 2009, Rafael J. Wysocki wrote:
> > > 
> > > > I think we'll need one separate workqueue for run-time PM in general, so that
> > > > bus types don't introduce their own workqueues for this purpose.  IMO one
> > > > system-wide run-time PM workqueue should be sufficient (it could also be
> > > > used for the suspend blockers BTW).
> > > > 
> > > > So, perhaps it makes sense to implement such a workqueue at the core level?
> > > > Thoughts?
> > > 
> > > That's fine with me.  This new workqueue can take over the job of the 
> > > "ksuspend_usbd" workqueue.  But it would have to be marked as 
> > > freezable.
> > 
> > Having considered it for a while I'm not sure if freezing is the right approach
> > here.  Namely, the work items put into the workqueue need not be useful after
> > the resume any more, so perhaps we should deqeue them during suspend?
> 
> With USB at least, this isn't true.  Some work items shouldn't be
> dequeued, namely, various sorts of resume requests generated while the
> driver is in interrupt context.  If they were dequeued during a system
> suspend then the device wouldn't ever be woken up, because usbcore
> doesn't pass system-resume messages to an autosuspended device.  The
> idea is if a device was autosuspended before the system sleep then it
> should remain autosuspended after the system wakes up.

Hmm, well, I'm not sure, really.  During a system-wide resume, does it really
matter if devices were autosuspended before the preceding suspend or they have
been suspended by the PM core?  Also, what happens if the platform firmware
resumes the autosuspended devices before passing control to the kernel during
system-wide resume?  How do we handle that?

Moreover, what if the autosuspended device is no longer present in the system
after a system-wide resume.  Are we still going to attempt to autoresume it?

> Furthermore, even if the existing work items were dequeued, you'd still 
> have to worry about new work items added on by drivers before they get 
> suspended.  If the workqueue were allowed to run freely, we might find 
> devices being autoresumed in the middle of the sleep transition!

Unless there's some other kind of synchronization between the workqueue and the
core suspend-resume code.

IMO, it would be convenient to treat every system-wide suspend (or hibernation)
as a cancellation point for all of the pending autosuspend and autoresume
requests and to treat devices autosuspended before that point as just
suspended.  Of course, the bus type and device driver suspend callbacks would
have to be prepared to handle this situation cleanly.

Best,
Rafael

Re: [PATCH 00/04][RFC] PM: Runtime platform device PM

[Alan Stern]

On Mon, 1 Jun 2009, Rafael J. Wysocki wrote:

> > With USB at least, this isn't true.  Some work items shouldn't be
> > dequeued, namely, various sorts of resume requests generated while the
> > driver is in interrupt context.  If they were dequeued during a system
> > suspend then the device wouldn't ever be woken up, because usbcore
> > doesn't pass system-resume messages to an autosuspended device.  The
> > idea is if a device was autosuspended before the system sleep then it
> > should remain autosuspended after the system wakes up.
> 
> Hmm, well, I'm not sure, really.  During a system-wide resume, does it really
> matter if devices were autosuspended before the preceding suspend or they have
> been suspended by the PM core?

It's partly a question of avoiding unnecessary activity.  Is there any
reason to resume a device if it's just going to be autosuspended again
a few seconds later?

It's also a matter of principle.  A system sleep should be as
transparent as possible (except for the passage of time, of course).  
When the sleep is over, devices should be in the same state as before
it began.

>  Also, what happens if the platform firmware
> resumes the autosuspended devices before passing control to the kernel during
> system-wide resume?  How do we handle that?

In the case of USB, it's not possible to resume a suspended device
without first resuming and using a host controller.  The host
controller driver will realize that the firmware has taken over the
controller and will reset the controller.

For devices that were not autosuspended, this action converts an
ordinary resume into a reset-resume.  A different driver method is
called, and drivers that don't support reset-resume are automatically
unbound from their devices (and then reprobed afterward).

For devices that were autosuspended, the controller reset will put them 
back into a suspended state.  They will remain that way until they are 
autoresumed -- and of course it will be an "auto-reset-resume".

What happens if the firmware is sufficiently sneaky that it manages to
fool the host controller driver?  In that case we end up with a device
that really is at full power even though the kernel thinks it is
autosuspended.  When we finally try to autoresume it, we will realize
that it wasn't suspended after all.

> Moreover, what if the autosuspended device is no longer present in the system
> after a system-wide resume.  Are we still going to attempt to autoresume it?

When the device or perhaps its parent hub is resumed, we will realize 
the device is gone.  An attempted autoresume will simply fail.  Until 
that happens, the kernel may believe the device is present when in fact 
it isn't.

> > Furthermore, even if the existing work items were dequeued, you'd still 
> > have to worry about new work items added on by drivers before they get 
> > suspended.  If the workqueue were allowed to run freely, we might find 
> > devices being autoresumed in the middle of the sleep transition!
> 
> Unless there's some other kind of synchronization between the workqueue and the
> core suspend-resume code.

So the work routine would somehow be aware when a system sleep is in
progress, at which point it would return immediately without doing
anything?  That seems racey.  Isn't it easier just to freeze the
workqueue?

> IMO, it would be convenient to treat every system-wide suspend (or hibernation)
> as a cancellation point for all of the pending autosuspend and autoresume
> requests and to treat devices autosuspended before that point as just
> suspended.  Of course, the bus type and device driver suspend callbacks would
> have to be prepared to handle this situation cleanly.

For pending autosuspend, okay.  The device was powered up before the
system sleep, it remains powered up afterward, and another autosuspend
will occur in due course.

For pending autoresume requests coming from user I/O calls, again okay.  
There can't be any of these because userspace is frozen, and besides,
they wouldn't go through the workqueue anyway.

For autoresume requests coming from an interrupt handler, I'm not so 
sure.  I don't know of any good examples -- in fact there don't seem to 
be any examples at the moment.  (The USB core supports autoresume 
requests in interrupt context, AFAICS no driver uses them.)  So maybe 
it's premature to worry about this case.

Lastly, we have resume requests coming from a device -- i.e., remote
wakeup requests.  This is a little tricky.  Suppose you have remote
wakeup enabled, and you press a key on your USB keyboard just as the
computer is going to sleep.  Should this cause the sleep to be aborted?  
Should it cause the computer to wake back up again right away?  Should
the keystroke be ignored?

More to the point, suppose you plug in a new USB device as the computer
is going to sleep.  When the computer wakes up, shouldn't it realize
that a new device has been connected?  This is the same problem that
other people have been discussing -- what should happen to wakeup IRQs
during the time the system is suspending?  Will they get delivered 
later on (and when drivers are ready to receive them)?

Alan Stern

Re: [PATCH 00/04][RFC] PM: Runtime platform device PM

[Rafael J. Wysocki]

On Tuesday 02 June 2009, Alan Stern wrote:
> On Mon, 1 Jun 2009, Rafael J. Wysocki wrote:
> 
> > > With USB at least, this isn't true.  Some work items shouldn't be
> > > dequeued, namely, various sorts of resume requests generated while the
> > > driver is in interrupt context.  If they were dequeued during a system
> > > suspend then the device wouldn't ever be woken up, because usbcore
> > > doesn't pass system-resume messages to an autosuspended device.  The
> > > idea is if a device was autosuspended before the system sleep then it
> > > should remain autosuspended after the system wakes up.
> > 
> > Hmm, well, I'm not sure, really.  During a system-wide resume, does it really
> > matter if devices were autosuspended before the preceding suspend or they have
> > been suspended by the PM core?
> 
> It's partly a question of avoiding unnecessary activity.  Is there any
> reason to resume a device if it's just going to be autosuspended again
> a few seconds later?

Well, I'm not sure if we know in advance what's going to happen to the device
right after a system-wide resume.  It may be a totally different situation from
the device activity point of view.

> It's also a matter of principle.  A system sleep should be as
> transparent as possible (except for the passage of time, of course).  
> When the sleep is over, devices should be in the same state as before
> it began.

As a matter of fact, this need not be true at all.  For example, consider a
wireless network adapter that was assosicated with specific access point
before a system-wide suspend, but this access point is not present any more
after the subsequent system-wide resume.

I really don't think we can predict what's going to happen to a device after
a system-wide resume, so IMO we should assume that the device will be necessary
immediately after the resume and we should resume it.  It still can autosuspend
normally afterwards if it's not needed after all.

> >  Also, what happens if the platform firmware
> > resumes the autosuspended devices before passing control to the kernel during
> > system-wide resume?  How do we handle that?
> 
> In the case of USB, it's not possible to resume a suspended device
> without first resuming and using a host controller.  The host
> controller driver will realize that the firmware has taken over the
> controller and will reset the controller.
> 
> For devices that were not autosuspended, this action converts an
> ordinary resume into a reset-resume.  A different driver method is
> called, and drivers that don't support reset-resume are automatically
> unbound from their devices (and then reprobed afterward).
> 
> For devices that were autosuspended, the controller reset will put them 
> back into a suspended state.  They will remain that way until they are 
> autoresumed -- and of course it will be an "auto-reset-resume".

I'm not sure if this is doable so easily for PCI devices, for one example.
Most probably, they will come up in D0 after the system-wide suspend and
the PCI resume code will have to suspend them directly so that they remain
"autosuspended" after the resume.

> What happens if the firmware is sufficiently sneaky that it manages to
> fool the host controller driver?  In that case we end up with a device
> that really is at full power even though the kernel thinks it is
> autosuspended.  When we finally try to autoresume it, we will realize
> that it wasn't suspended after all.
> 
> > Moreover, what if the autosuspended device is no longer present in the system
> > after a system-wide resume.  Are we still going to attempt to autoresume it?
> 
> When the device or perhaps its parent hub is resumed, we will realize 
> the device is gone.  An attempted autoresume will simply fail.  Until 
> that happens, the kernel may believe the device is present when in fact 
> it isn't.
> 
> > > Furthermore, even if the existing work items were dequeued, you'd still 
> > > have to worry about new work items added on by drivers before they get 
> > > suspended.  If the workqueue were allowed to run freely, we might find 
> > > devices being autoresumed in the middle of the sleep transition!
> > 
> > Unless there's some other kind of synchronization between the workqueue and the
> > core suspend-resume code.
> 
> So the work routine would somehow be aware when a system sleep is in
> progress, at which point it would return immediately without doing
> anything?  That seems racey.  Isn't it easier just to freeze the
> workqueue?

Well, yes and no.  Namely, if we freeze the workqueue, all of the bus types
using it will have to check the vaildity of operations carried out by it,
because of the possibility that system-wide power transition took place in the
meantime and changed the state of the device.  Also, please take the PCI
example into consideration.

> > IMO, it would be convenient to treat every system-wide suspend (or hibernation)
> > as a cancellation point for all of the pending autosuspend and autoresume
> > requests and to treat devices autosuspended before that point as just
> > suspended.  Of course, the bus type and device driver suspend callbacks would
> > have to be prepared to handle this situation cleanly.
> 
> For pending autosuspend, okay.  The device was powered up before the
> system sleep, it remains powered up afterward, and another autosuspend
> will occur in due course.
> 
> For pending autoresume requests coming from user I/O calls, again okay.  
> There can't be any of these because userspace is frozen, and besides,
> they wouldn't go through the workqueue anyway.
> 
> For autoresume requests coming from an interrupt handler, I'm not so 
> sure.  I don't know of any good examples -- in fact there don't seem to 
> be any examples at the moment.  (The USB core supports autoresume 
> requests in interrupt context, AFAICS no driver uses them.)  So maybe 
> it's premature to worry about this case.

Hmm.  Can you please describe a theoretical situation in which this kind
of autoresume request will appear?  I'm not really sure what you mean here. :-)

> Lastly, we have resume requests coming from a device -- i.e., remote
> wakeup requests.  This is a little tricky.  Suppose you have remote
> wakeup enabled, and you press a key on your USB keyboard just as the
> computer is going to sleep.  Should this cause the sleep to be aborted?  
> Should it cause the computer to wake back up again right away?  Should
> the keystroke be ignored?

This is a difficult problem in general.  I _think_ we should ignore such
autoresume requests for devices that are not marked as wake-up ones
and we should abort the suspend if there's a request from a wake-up device.
 
> More to the point, suppose you plug in a new USB device as the computer
> is going to sleep.  When the computer wakes up, shouldn't it realize
> that a new device has been connected?  This is the same problem that
> other people have been discussing -- what should happen to wakeup IRQs
> during the time the system is suspending?  Will they get delivered 
> later on (and when drivers are ready to receive them)?

I think they should abort the suspend unless it's too late (eg. the suspend
sequence has reached sysdev_suspend()).

Anyway, below is a prototype implementation of a generic runtime PM workqueue,
just for discussion.

pm_schedule_suspend() and pm_schedule_resume() are to be called by bus types
that are requesting the autosuspend or autoresume, respectively, of a device.
The supposed control flow is like this:
* bus type calls pm_schedule_suspend() to schedule autosuspend of a device
* at the right time the PM workqueue executes the bus type's ->autosuspend()
  callback which is responsible for calling the device driver to suspend the
  device (the way in which this is done is totally bus type-specific)
* if there's a resume event for the device, it's delivered to the bus type
  driver (in a bus type-specific way) and the bus type schedules autoresume of
  the device by calling pm_schedule_resume()
* the PM workqueue executes the bus type's ->autoresume() as soon as it can
  and this callback is responsible for calling the device driver to resume the
  device (the way in which this is done is totally bus type-specific).

Thanks,
Rafael

---
 drivers/base/power/main.c |   11 ++++--
 include/linux/pm.h        |   16 +++++++++
 kernel/power/main.c       |   75 ++++++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 99 insertions(+), 3 deletions(-)

Index: linux-2.6/include/linux/pm.h
=================================--- linux-2.6.orig/include/linux/pm.h
+++ linux-2.6/include/linux/pm.h
@@ -22,6 +22,7 @@
 #define _LINUX_PM_H
 
 #include <linux/list.h>
+#include <linux/workqueue.h>
 
 /*
  * Callbacks for platform drivers to implement.
@@ -182,6 +183,8 @@ struct dev_pm_ops {
 	int (*thaw_noirq)(struct device *dev);
 	int (*poweroff_noirq)(struct device *dev);
 	int (*restore_noirq)(struct device *dev);
+	int (*autosuspend)(struct device *dev);
+	int (*autoresume)(struct device *dev);
 };
 
 /**
@@ -323,8 +326,21 @@ struct dev_pm_info {
 #ifdef	CONFIG_PM_SLEEP
 	struct list_head	entry;
 #endif
+	struct delayed_work	dwork;
 };
 
+extern void pm_schedule_suspend(struct device *dev, unsigned long delay);
+extern void pm_schedule_resume(struct device *dev);
+
+extern bool transition_started;
+
+static inline bool pm_transition_in_progress(void)
+{
+	return transition_started;
+}
+
+extern void pm_flush_workqueue(void);
+
 /*
  * The PM_EVENT_ messages are also used by drivers implementing the legacy
  * suspend framework, based on the ->suspend() and ->resume() callbacks common
Index: linux-2.6/kernel/power/main.c
=================================--- linux-2.6.orig/kernel/power/main.c
+++ linux-2.6/kernel/power/main.c
@@ -22,6 +22,8 @@
 #include <linux/freezer.h>
 #include <linux/vmstat.h>
 #include <linux/syscalls.h>
+#include <linux/workqueue.h>
+#include <linux/device.h>
 
 #include "power.h"
 
@@ -578,12 +580,20 @@ static struct attribute_group attr_group
 	.attrs = g,
 };
 
+static int __init pm_start_workqueue(void);
 
 static int __init pm_init(void)
 {
+	int error;
+
+	error = pm_start_workqueue();
+	if (error)
+		return error;
+
 	power_kobj = kobject_create_and_add("power", NULL);
 	if (!power_kobj)
 		return -ENOMEM;
+
 	return sysfs_create_group(power_kobj, &attr_group);
 }
 
@@ -729,3 +739,68 @@ done:
 late_initcall(test_suspend);
 
 #endif /* CONFIG_PM_TEST_SUSPEND */
+
+
+static struct workqueue_struct *pm_wq;
+
+static int __init pm_start_workqueue(void)
+{
+	pm_wq = create_singlethread_workqueue("pm");
+
+	return pm_wq ? 0 : -ENOMEM;
+}
+
+void pm_flush_workqueue(void)
+{
+	flush_workqueue(pm_wq);
+}
+
+static inline struct device *pm_dwork_to_device(struct delayed_work *dwork)
+{
+	struct dev_pm_info *d = container_of(dwork, struct dev_pm_info, dwork);
+	return container_of(d, struct device, power);
+}
+
+static void pm_autosuspend(struct work_struct *work)
+{
+	struct delayed_work *dwork = to_delayed_work(work);
+	struct device *dev = pm_dwork_to_device(dwork);
+
+	device_pm_lock();
+	if (dev && dev->bus && dev->bus->pm && dev->bus->pm->autosuspend
+	    && !pm_transition_in_progress())
+		dev->bus->pm->autosuspend(dev);
+	device_pm_unlock();
+}
+
+void pm_schedule_suspend(struct device *dev, unsigned long delay)
+{
+	device_pm_lock();
+	if (!pm_transition_in_progress()) {
+		INIT_DELAYED_WORK(&dev->power.dwork, pm_autosuspend);
+		queue_delayed_work(pm_wq, &dev->power.dwork, delay);
+	}
+	device_pm_unlock();
+}
+
+static void pm_autoresume(struct work_struct *work)
+{
+	struct delayed_work *dwork = to_delayed_work(work);
+	struct device *dev = pm_dwork_to_device(dwork);
+
+	device_pm_lock();
+	if (dev && dev->bus && dev->bus->pm && dev->bus->pm->autoresume
+	    && !pm_transition_in_progress())
+		dev->bus->pm->autoresume(dev);
+	device_pm_unlock();
+}
+
+void pm_schedule_resume(struct device *dev)
+{
+	device_pm_lock();
+	if (!pm_transition_in_progress()) {
+		INIT_WORK(&dev->power.dwork.work, pm_autoresume);
+		queue_work(pm_wq, &dev->power.dwork.work);
+	}
+	device_pm_unlock();
+}
Index: linux-2.6/drivers/base/power/main.c
=================================--- linux-2.6.orig/drivers/base/power/main.c
+++ linux-2.6/drivers/base/power/main.c
@@ -46,7 +46,7 @@ static DEFINE_MUTEX(dpm_list_mtx);
  * Set once the preparation of devices for a PM transition has started, reset
  * before starting to resume devices.  Protected by dpm_list_mtx.
  */
-static bool transition_started;
+bool transition_started;
 
 /**
  *	device_pm_lock - lock the list of active devices used by the PM core
@@ -436,7 +436,6 @@ static void dpm_resume(pm_message_t stat
 
 	INIT_LIST_HEAD(&list);
 	mutex_lock(&dpm_list_mtx);
-	transition_started = false;
 	while (!list_empty(&dpm_list)) {
 		struct device *dev = to_device(dpm_list.next);
 
@@ -461,6 +460,7 @@ static void dpm_resume(pm_message_t stat
 		put_device(dev);
 	}
 	list_splice(&list, &dpm_list);
+	transition_started = false;
 	mutex_unlock(&dpm_list_mtx);
 }
 
@@ -752,9 +752,14 @@ static int dpm_prepare(pm_message_t stat
 	struct list_head list;
 	int error = 0;
 
-	INIT_LIST_HEAD(&list);
 	mutex_lock(&dpm_list_mtx);
 	transition_started = true;
+	mutex_unlock(&dpm_list_mtx);
+
+	pm_flush_workqueue();
+
+	INIT_LIST_HEAD(&list);
+	mutex_lock(&dpm_list_mtx);
 	while (!list_empty(&dpm_list)) {
 		struct device *dev = to_device(dpm_list.next);
 

Re: [PATCH 00/04][RFC] PM: Runtime platform device PM

[Magnus Damm]

On Sat, May 30, 2009 at 3:18 AM, Rafael J. Wysocki <rjw@sisk.pl> wrote:
> On Friday 29 May 2009, Alan Stern wrote:
>> On Fri, 29 May 2009, Magnus Damm wrote:
>>
>> > > (And by the way, the _noirq ops don't run in atomic context.  They run
>> > > in process context with most interrupt deliveries disabled.  It's not
>> > > the same thing -- they are allowed to sleep.)
>> >
>> > Huh, so if they are allowed to sleep then clock events are still
>> > running. And probably some clocksource as well. I guess that's why you
>> > say "most" interrupts disabled instead of all. Sharing the timer IRQ
>> > is not allowed then?
>>
>> Rafael can tell you.  But notice I didn't say the interrupts were
>> disabled -- I said that interrupt _delivery_ was disabled.  In general
>> the interrupts themselves _are_ enabled; the kernel fields them but
>> then does not call the drivers' interrupt handler routines.
>
> That's correct, except for the platforms where __disable_irq() actually masks
> the IRQ.  I'm not sure if SuperH is one of these.

We make use of CONFIG_GENERIC_HARDIRQS so the __disable_irq() suspend
logic should work for us as well.

>> (Except perhaps for IRQs which are explicitly marked as wakeup sources...)
>
> No, they are treated just like the other ones except that we check if any of
> them are pending right at the beginning of sysdev_suspend().
>
>> > I wonder if the PM code assumes that the clock event is a sys device?
>
> Yes, it does.

I note the IRQF_TIMER check in __disable_irq(). I wonder if we can
make a second not bus-specific round where we disable only IRQF_TIMER
interrupts.

>> > We use platform drivers for clock events on SuperH...
>
> Argh.  This is going to hurt, but I'm not sure how badly.  At least on
> multiprocessor.

Luckily most boards around here are still UP. =)

/ magnus

Re: [PATCH 00/04][RFC] PM: Runtime platform device PM

[Alan Stern]

On Tue, 2 Jun 2009, Rafael J. Wysocki wrote:

> > It's partly a question of avoiding unnecessary activity.  Is there any
> > reason to resume a device if it's just going to be autosuspended again
> > a few seconds later?
> 
> Well, I'm not sure if we know in advance what's going to happen to the device
> right after a system-wide resume.  It may be a totally different situation from
> the device activity point of view.

We _never_ know in advance what's going to happen.  An autosuspended
device may have to wake up at any time in order to carry out some work
-- that's true whether or not a system sleep has taken place.  IMO it
doesn't make sense to wake up an autosuspended device for no reason
other than that it _may_ need to do something; we should let it stay in
a low-power mode until we _know_ it has to wake up.

> > It's also a matter of principle.  A system sleep should be as
> > transparent as possible (except for the passage of time, of course).  
> > When the sleep is over, devices should be in the same state as before
> > it began.
> 
> As a matter of fact, this need not be true at all.  For example, consider a
> wireless network adapter that was assosicated with specific access point
> before a system-wide suspend, but this access point is not present any more
> after the subsequent system-wide resume.

This example doesn't prove your point.  Sure, conditions may have
changed while the system was asleep.  They might also change while the
system is awake.  In general, whether or not the system was asleep
shouldn't make any difference -- to as great an extent as possible we
should strive to pretend that the entire sleep took no time at all (or
simply didn't happen).

So: Suppose the system had been awake when the access point
disappeared.  What would the wireless adapter driver do then?  It
should try to do exactly the same thing if the access point disappears
while the system is asleep.

You see?  The sleep should be transparent.

> I really don't think we can predict what's going to happen to a device after
> a system-wide resume, so IMO we should assume that the device will be necessary
> immediately after the resume and we should resume it.  It still can autosuspend
> normally afterwards if it's not needed after all.

This is highly debatable.  I'd like to hear some thoughts from other 
people.


> > >  Also, what happens if the platform firmware
> > > resumes the autosuspended devices before passing control to the kernel during
> > > system-wide resume?  How do we handle that?
> > 
> > In the case of USB, it's not possible to resume a suspended device
> > without first resuming and using a host controller.  The host
> > controller driver will realize that the firmware has taken over the
> > controller and will reset the controller.
...

> I'm not sure if this is doable so easily for PCI devices, for one example.
> Most probably, they will come up in D0 after the system-wide suspend and
> the PCI resume code will have to suspend them directly so that they remain
> "autosuspended" after the resume.

Clearly the behavior will need to vary among buses.  What's appropriate
for USB won't be appropriate for PCI.  Furthermore, the overhead of
waking a PCI device unnecessarily might be a lot lower than the
overhead of waking a USB device, so it might make sense always to
resume autosuspended PCI devices during a system resume.


> > So the work routine would somehow be aware when a system sleep is in
> > progress, at which point it would return immediately without doing
> > anything?  That seems racey.  Isn't it easier just to freeze the
> > workqueue?
> 
> Well, yes and no.  Namely, if we freeze the workqueue, all of the bus types
> using it will have to check the vaildity of operations carried out by it,
> because of the possibility that system-wide power transition took place in the
> meantime and changed the state of the device.  Also, please take the PCI
> example into consideration.

Freezability doesn't matter; you can prevent items from being added to
the workqueue during a power transition just as easily regardless of
whether it is freezable or not.  In fact, it's easier if the workqueue
is freezable.  You don't have to prevent anything from being added; you
just have to cancel everything on the workqueue before unfreezing it.

One solution would be to have the bus types cancel their own pending
work items explicitly.  Another is to have two freezable PM workqueues,
one with automatic cancellation and the other without.


> > For autoresume requests coming from an interrupt handler, I'm not so 
> > sure.  I don't know of any good examples -- in fact there don't seem to 
> > be any examples at the moment.  (The USB core supports autoresume 
> > requests in interrupt context, AFAICS no driver uses them.)  So maybe 
> > it's premature to worry about this case.
> 
> Hmm.  Can you please describe a theoretical situation in which this kind
> of autoresume request will appear?  I'm not really sure what you mean here. :-)

Well, here's something people discussed but which hasn't been
implemented (and may never be implemented).  A suspended USB keyboard
doesn't have enough power to turn on any LEDs.  So suppose a system has
two keyboards, and kbd-A is autosuspended when the user engages
CapsLock on kbd-B.  The input layer wants to turn on the CapsLock LED
on all the keyboards, which means it has to autoresume kbd-A.  But all
this action takes place in interrupt context, so an autoresume request
would have to be queued.


> > Lastly, we have resume requests coming from a device -- i.e., remote
> > wakeup requests.  This is a little tricky.  Suppose you have remote
> > wakeup enabled, and you press a key on your USB keyboard just as the
> > computer is going to sleep.  Should this cause the sleep to be aborted?  
> > Should it cause the computer to wake back up again right away?  Should
> > the keystroke be ignored?
> 
> This is a difficult problem in general.  I _think_ we should ignore such
> autoresume requests for devices that are not marked as wake-up ones
> and we should abort the suspend if there's a request from a wake-up device.

It _is_ difficult.  Right now USB ignores such requests, because the
threads that handle them are all freezable.  But the information isn't
lost; it is stored in the hardware and when the system wakes up the
requests will be seen.

Except for wakeup requests from root hubs.  If they arrive at just the 
right time then they are not stored in the hardware -- instead they are 
stored as work items on the ksuspend_usbd workqueue!  In essence, the 
workqueue serves as a software mechanism for saving IRQs which were 
delivered but couldn't be acted on.

That's why I don't want to erase the work items; it would be logically 
equivalent to dropping interrupts.


> Anyway, below is a prototype implementation of a generic runtime PM workqueue,
> just for discussion.
> 
> pm_schedule_suspend() and pm_schedule_resume() are to be called by bus types
> that are requesting the autosuspend or autoresume, respectively, of a device.
> The supposed control flow is like this:
> * bus type calls pm_schedule_suspend() to schedule autosuspend of a device
> * at the right time the PM workqueue executes the bus type's ->autosuspend()
>   callback which is responsible for calling the device driver to suspend the
>   device (the way in which this is done is totally bus type-specific)
> * if there's a resume event for the device, it's delivered to the bus type
>   driver (in a bus type-specific way) and the bus type schedules autoresume of
>   the device by calling pm_schedule_resume()
> * the PM workqueue executes the bus type's ->autoresume() as soon as it can
>   and this callback is responsible for calling the device driver to resume the
>   device (the way in which this is done is totally bus type-specific).

It looks like the only benefits provided by this patch are the
workqueue itself and the embedded delayed_work structure.  The ability
to cancel the work items (or prevent them from being added) during a
system sleep is questionable, as discussed above.  There isn't any
value added by the new functions or methods.

In short, it would be just as useful simply to add the delayed_work 
structure and export two PM workqueues, both freezable, with automatic
cancellation for one of them.

Alan Stern

Re: [linux-pm] [PATCH 00/04][RFC] PM: Runtime platform device PM

[Pavel Machek]

Hi!


> > In the process of discussing a similar interface for OMAP, we dicussed
> > that having 3 states would be more useful.  Specifically, and
> > _enable(), _idle() and _disable() hook.  The _enable() and _idle()
> > hooks being exactly what you proposed above, but with the addition of
> > a _disable() hook which says not only can the device go idle, but that
> > the driver is really finished with the device.  In this case, more
> > aggresive PM measures could be taken, such as turning of regulators
> > that may have long latencies that may not be appropriate to turn off
> > in idle.
> 
> Hm.. I wonder when the driver is really finished with the device
> though. Only when the module is unloaded? If so then we could deal
> with the hard disable using platform bus notifiers (like in [04/04]).

I'd guess that they call disable() when /dev node is closed... and
idle() when device is opened but inactive.

-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

Re: [linux-pm] [PATCH 00/04][RFC] PM: Runtime platform device PM

[Magnus Damm]

Hi Pavel!

On Wed, Jun 3, 2009 at 6:37 AM, Pavel Machek <pavel@ucw.cz> wrote:
>> > In the process of discussing a similar interface for OMAP, we dicussed
>> > that having 3 states would be more useful.  Specifically, and
>> > _enable(), _idle() and _disable() hook.  The _enable() and _idle()
>> > hooks being exactly what you proposed above, but with the addition of
>> > a _disable() hook which says not only can the device go idle, but that
>> > the driver is really finished with the device.  In this case, more
>> > aggresive PM measures could be taken, such as turning of regulators
>> > that may have long latencies that may not be appropriate to turn off
>> > in idle.
>>
>> Hm.. I wonder when the driver is really finished with the device
>> though. Only when the module is unloaded? If so then we could deal
>> with the hard disable using platform bus notifiers (like in [04/04]).
>
> I'd guess that they call disable() when /dev node is closed... and
> idle() when device is opened but inactive.

But the time from close() to next open() is work load dependent. So
the difference between _disable() and _idle() is pretty fuzzy in my
opinion. I'm trying to think of a case where hard _disable() is
needed, but nothing comes to mind.

In my mind a single disable()/idle() function together with latency
information is enough. It covers the entire range from _idle() to hard
_disable().

/ magnus

Re: [PATCH 00/04][RFC] PM: Runtime platform device PM

[Rafael J. Wysocki]

On Tuesday 02 June 2009, Alan Stern wrote:
> On Tue, 2 Jun 2009, Rafael J. Wysocki wrote:
> 
> > > It's partly a question of avoiding unnecessary activity.  Is there any
> > > reason to resume a device if it's just going to be autosuspended again
> > > a few seconds later?
> > 
> > Well, I'm not sure if we know in advance what's going to happen to the device
> > right after a system-wide resume.  It may be a totally different situation from
> > the device activity point of view.
> 
> We _never_ know in advance what's going to happen.  An autosuspended
> device may have to wake up at any time in order to carry out some work
> -- that's true whether or not a system sleep has taken place.  IMO it
> doesn't make sense to wake up an autosuspended device for no reason
> other than that it _may_ need to do something; we should let it stay in
> a low-power mode until we _know_ it has to wake up.

OK, that's a valid point.

I also think that if the BIOS has already woke up the device during system-wide
resume, we shouldn't try to put it to a low power state again unless we know
that this is desirable.

> > > It's also a matter of principle.  A system sleep should be as
> > > transparent as possible (except for the passage of time, of course).  
> > > When the sleep is over, devices should be in the same state as before
> > > it began.
> > 
> > As a matter of fact, this need not be true at all.  For example, consider a
> > wireless network adapter that was assosicated with specific access point
> > before a system-wide suspend, but this access point is not present any more
> > after the subsequent system-wide resume.
> 
> This example doesn't prove your point.

No, it doesn't.  Still, it illustrates the problem I had in mind, that it may
not be desirable to restore the pre-suspend settings in all cases.

> Sure, conditions may have changed while the system was asleep.  They might
> also change while the system is awake.  In general, whether or not the system
> was asleep shouldn't make any difference -- to as great an extent as possible
> we should strive to pretend that the entire sleep took no time at all (or
> simply didn't happen).
> 
> So: Suppose the system had been awake when the access point
> disappeared.  What would the wireless adapter driver do then?  It
> should try to do exactly the same thing if the access point disappears
> while the system is asleep.

Not really.  Namely, when the access point _vanishes_ at run time, this is an
error condition from which we have to recover, while if it's _not_ _present_
after a resume, it's a normal situation and we shouldn't be recovering from
that (the user might have changed the physical location while suspended).

On my system the wireless always remembers the network it was associated with
before suspend and it tries to reassociate with it after the subsequent resume,
even if the network is not there any more, so it fails and I have to choose
another one manually.  This isn't a desirable behavior and I'd even say it's
not correct, because after a fresh boot it always scans for networks and
associates with one that's actually present.  I'd like it to do the same after
the resume.

> You see?  The sleep should be transparent.

In some cases it won't be, because the hardware state changes while sleeping.
For example, a laptop battery may be depleted while suspended, so that after
resume it's in the 'low' condition, while it was in the 'good' condition before
the preceding suspend.  Arguably, you can't drain a battery from 90% to 10%,
for example, momentarily.  Also, a CPU fan might be 100% on before suspend,
because the CPU was hot at that time, but that doesn't mean the fan should be
100% on after the subsequent resume, because it's likely that the CPU will be
cold (on some systems trying to make the fan spin too fast may lead to
general problems with thermal management afterwards).

> > I really don't think we can predict what's going to happen to a device after
> > a system-wide resume, so IMO we should assume that the device will be necessary
> > immediately after the resume and we should resume it.  It still can autosuspend
> > normally afterwards if it's not needed after all.
> 
> This is highly debatable.  I'd like to hear some thoughts from other 
> people.

Agreed, but no one has spoken so far. :-)

> > > >  Also, what happens if the platform firmware
> > > > resumes the autosuspended devices before passing control to the kernel during
> > > > system-wide resume?  How do we handle that?
> > > 
> > > In the case of USB, it's not possible to resume a suspended device
> > > without first resuming and using a host controller.  The host
> > > controller driver will realize that the firmware has taken over the
> > > controller and will reset the controller.
> ...
> 
> > I'm not sure if this is doable so easily for PCI devices, for one example.
> > Most probably, they will come up in D0 after the system-wide suspend and
> > the PCI resume code will have to suspend them directly so that they remain
> > "autosuspended" after the resume.
> 
> Clearly the behavior will need to vary among buses.

Agreed.

> What's appropriate for USB won't be appropriate for PCI.  Furthermore, the
> overhead of waking a PCI device unnecessarily might be a lot lower than the
> overhead of waking a USB device, so it might make sense always to
> resume autosuspended PCI devices during a system resume.

Sure.  Also, as I said earlier, the BIOS usually wakes up PCI devices (in fact
a bus reset causes them to wake up and the buses are ususally reset during
resume by the BIOS) and it doesn't really make sense to put them into low
power states just because they were in low power states before the suspend.

> > > So the work routine would somehow be aware when a system sleep is in
> > > progress, at which point it would return immediately without doing
> > > anything?  That seems racey.  Isn't it easier just to freeze the
> > > workqueue?
> > 
> > Well, yes and no.  Namely, if we freeze the workqueue, all of the bus types
> > using it will have to check the vaildity of operations carried out by it,
> > because of the possibility that system-wide power transition took place in the
> > meantime and changed the state of the device.  Also, please take the PCI
> > example into consideration.
> 
> Freezability doesn't matter; you can prevent items from being added to
> the workqueue during a power transition just as easily regardless of
> whether it is freezable or not.  In fact, it's easier if the workqueue
> is freezable.  You don't have to prevent anything from being added; you
> just have to cancel everything on the workqueue before unfreezing it.

Good idea.

> One solution would be to have the bus types cancel their own pending
> work items explicitly.  Another is to have two freezable PM workqueues,
> one with automatic cancellation and the other without.

What about having a flag in struct dev_pm_info (along with the delayed
work structure) indicating whether or not the work should be automatically
cancelled before unfreezing the queue?  [It seems, in that case we'd need
another flag to indicate what is to cancel, the delayed work or just the work.]

> > > For autoresume requests coming from an interrupt handler, I'm not so 
> > > sure.  I don't know of any good examples -- in fact there don't seem to 
> > > be any examples at the moment.  (The USB core supports autoresume 
> > > requests in interrupt context, AFAICS no driver uses them.)  So maybe 
> > > it's premature to worry about this case.
> > 
> > Hmm.  Can you please describe a theoretical situation in which this kind
> > of autoresume request will appear?  I'm not really sure what you mean here. :-)
> 
> Well, here's something people discussed but which hasn't been
> implemented (and may never be implemented).  A suspended USB keyboard
> doesn't have enough power to turn on any LEDs.  So suppose a system has
> two keyboards, and kbd-A is autosuspended when the user engages
> CapsLock on kbd-B.  The input layer wants to turn on the CapsLock LED
> on all the keyboards, which means it has to autoresume kbd-A.  But all
> this action takes place in interrupt context, so an autoresume request
> would have to be queued.

I see, but this is similar to the remote wake-up in that we have to queue
the autoresume request.

> > > Lastly, we have resume requests coming from a device -- i.e., remote
> > > wakeup requests.  This is a little tricky.  Suppose you have remote
> > > wakeup enabled, and you press a key on your USB keyboard just as the
> > > computer is going to sleep.  Should this cause the sleep to be aborted?  
> > > Should it cause the computer to wake back up again right away?  Should
> > > the keystroke be ignored?
> > 
> > This is a difficult problem in general.  I _think_ we should ignore such
> > autoresume requests for devices that are not marked as wake-up ones
> > and we should abort the suspend if there's a request from a wake-up device.
> 
> It _is_ difficult.  Right now USB ignores such requests, because the
> threads that handle them are all freezable.  But the information isn't
> lost; it is stored in the hardware and when the system wakes up the
> requests will be seen.
> 
> Except for wakeup requests from root hubs.  If they arrive at just the 
> right time then they are not stored in the hardware -- instead they are 
> stored as work items on the ksuspend_usbd workqueue!  In essence, the 
> workqueue serves as a software mechanism for saving IRQs which were 
> delivered but couldn't be acted on.
> 
> That's why I don't want to erase the work items; it would be logically 
> equivalent to dropping interrupts.

I see.

Still, we'll need to think about a mechanism allowing us to abort suspend
if there's a wake-up request from a wake-up device.

> > Anyway, below is a prototype implementation of a generic runtime PM workqueue,
> > just for discussion.
> > 
> > pm_schedule_suspend() and pm_schedule_resume() are to be called by bus types
> > that are requesting the autosuspend or autoresume, respectively, of a device.
> > The supposed control flow is like this:
> > * bus type calls pm_schedule_suspend() to schedule autosuspend of a device
> > * at the right time the PM workqueue executes the bus type's ->autosuspend()
> >   callback which is responsible for calling the device driver to suspend the
> >   device (the way in which this is done is totally bus type-specific)
> > * if there's a resume event for the device, it's delivered to the bus type
> >   driver (in a bus type-specific way) and the bus type schedules autoresume of
> >   the device by calling pm_schedule_resume()
> > * the PM workqueue executes the bus type's ->autoresume() as soon as it can
> >   and this callback is responsible for calling the device driver to resume the
> >   device (the way in which this is done is totally bus type-specific).
> 
> It looks like the only benefits provided by this patch are the
> workqueue itself and the embedded delayed_work structure.  The ability
> to cancel the work items (or prevent them from being added) during a
> system sleep is questionable, as discussed above.  There isn't any
> value added by the new functions or methods.

The value added is that since the delayed work member is to be used for
queueing work, each bus type will have to do the same casting from
'struct work_struct' to 'struct delayed_work' and next to 'struct device'
to get to the device object, which means quite a bit of code duplication. 

Also, IMO, if there's a set of functions for doing that, the probability
that someone gets things wrong is smaller (slightly).  Moreover, we can
add variants for the works that will be automatically cancelled during
system-wide resume, so that bus types don't have to worry about the details
related to that.

Thanks,
Rafael

Re: [linux-pm] [PATCH 00/04][RFC] PM: Runtime platform device PM

[Pavel Machek]

Hi!

> > Sure, conditions may have changed while the system was asleep.  They might
> > also change while the system is awake.  In general, whether or not the system
> > was asleep shouldn't make any difference -- to as great an extent as possible
> > we should strive to pretend that the entire sleep took no time at all (or
> > simply didn't happen).
> > 
> > So: Suppose the system had been awake when the access point
> > disappeared.  What would the wireless adapter driver do then?  It
> > should try to do exactly the same thing if the access point disappears
> > while the system is asleep.
> 
> Not really.  Namely, when the access point _vanishes_ at run time, this is an
> error condition from which we have to recover, while if it's _not_ _present_
> after a resume, it's a normal situation and we shouldn't be recovering from
> that (the user might have changed the physical location while
> suspended).

I disagree here. User can just walk away with powered notebook.


> > You see?  The sleep should be transparent.
> 
> In some cases it won't be, because the hardware state changes while sleeping.
> For example, a laptop battery may be depleted while suspended, so that after
> resume it's in the 'low' condition, while it was in the 'good' condition before
> the preceding suspend.  Arguably, you can't drain a battery from 90% to 10%,
> for example, momentarily.  Also, a CPU fan might be 100% on before
> suspend,

Actually... going 90%->10% immediately is common behaviour of old
li-ion batteries.

> because the CPU was hot at that time, but that doesn't mean the fan should be
> 100% on after the subsequent resume, because it's likely that the CPU will be
> cold (on some systems trying to make the fan spin too fast may lead to
> general problems with thermal management afterwards).

100% is indeed sane default for fan. Yes you should slow the fan down
when you read the real temperature. And yes some systems play it safe
and do 100% fan during boot.

(Otherwise you may have heat problems if you suspend and immediately
resume).
									Pavel
-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html