System sleep vs. runtime PM

System sleep vs. runtime PM

[Alan Stern]

Rafael and Oliver:

I'm in the midst of converting USB over to the runtime PM framework,
and I'm getting stuck on the interaction between system sleep and
runtime PM.  In fact, I'm so confused about it that it's hard to
formulate a coherent set of questions.  So bear with me...

If a driver gets an output request and its device is runtime-suspended,
then the driver should do a runtime-resume before sending the output to
the device.  If the output request arrives during a system sleep
transition then the output should be refused or left to sit in a queue
(presumably this won't happen, but it might).

However this requires the driver to know when a system sleep is in
progress -- it's not enough merely to know whether the device is
suspended.  We could make things easier for the driver by failing
runtime-resume requests when the device's power.status isn't DPM_ON or
DPM_RESUMING.  Does that make sense?

But then what about remote wakeup?  The arrival of a remote-wakeup
request during a sleep transition has always been problematic.  Should
the request be allowed to abort the sleep transition?  (Once the kernel
is aware of the request, it is probably no longer stored in the
hardware so it won't wake the system back up as soon as we go to
sleep.)

What if the device is enabled for runtime remote wakeup and not for
wakeup from system sleep?  There will be a window during which the
wakeup settings are wrong.  Should a wakeup request from such a device
be allowed to abort a sleep transition?

If a remote-wakeup request should abort the system sleep, how do we
make this happen?

If a remote-wakeup request doesn't abort the system sleep, then what 
happens to it?  Obviously it should get serviced when the system wakes 
up again.  But what if the device was runtime-suspended when the sleep 
began?  How do we make sure it gets runtime-resumed when the sleep 
ends?

I can see two possibilities here.  One is that the remote-wakeup
request triggered an asynchronous pm_request_wakeup().  In this case
the request is sitting on the PM workqueue and it will automatically be
satisfied after the system wakes back up.  The other possibility is
that the request triggered a synchronous pm_runtime_resume().  If this
fails because a sleep transition is in progress, the request will get
lost.  (The same sort of thing happens when output requests get stuck
in a queue and the runtime-resume fails.)

This suggests that the safest course is for dpm_resume() to
automatically do a pm_runtime_resume() on every device.  Is this
drastic overkill?  Should we keep a record when we fail a
runtime-resume during a sleep transition and then runtime-resume only
those devices?

Then there's the whole sticky question about how to handle devices
whose runtime-wakeup and system-sleep-wakeup settings need to be
different.  In many cases the setting can't be changed unless the
device is at full power.  Such devices would always need to be
runtime-resumed both before and after every system sleep (during 
the prepare and complete phases).  Should the bus subsystem be 
responsible for this?  (Don't forget that during the prepare phase, the 
device's state will be DPM_PREPARING and so the bus subsystem won't be 
allowed to do a runtime resume.)

There's probably a good deal more to discuss, but this is all I can 
think of at the moment.

Alan Stern

Re: System sleep vs. runtime PM

[Oliver Neukum]

Am Mittwoch, 2. Dezember 2009 19:35:34 schrieb Alan Stern:
> Rafael and Oliver:
> 
> I'm in the midst of converting USB over to the runtime PM framework,
> and I'm getting stuck on the interaction between system sleep and
> runtime PM.  In fact, I'm so confused about it that it's hard to
> formulate a coherent set of questions.  So bear with me...
> 
> If a driver gets an output request and its device is runtime-suspended,
> then the driver should do a runtime-resume before sending the output to
> the device.  If the output request arrives during a system sleep
> transition then the output should be refused or left to sit in a queue
> (presumably this won't happen, but it might).

How can it happen? As the freezer has run before drivers learn
about a sleep transition, the request would come from a kernel thread.

> However this requires the driver to know when a system sleep is in
> progress -- it's not enough merely to know whether the device is
> suspended.  We could make things easier for the driver by failing
> runtime-resume requests when the device's power.status isn't DPM_ON or
> DPM_RESUMING.  Does that make sense?

That is not a good idea as it needs error handling in all drivers
for a rare corner case. That's bound to cause many bugs.

> But then what about remote wakeup?  The arrival of a remote-wakeup
> request during a sleep transition has always been problematic.  Should
> the request be allowed to abort the sleep transition?  (Once the kernel
> is aware of the request, it is probably no longer stored in the
> hardware so it won't wake the system back up as soon as we go to
> sleep.)

It should abort the sleep if the device would wake the system from sleep.

> What if the device is enabled for runtime remote wakeup and not for
> wakeup from system sleep?  There will be a window during which the
> wakeup settings are wrong.  Should a wakeup request from such a device
> be allowed to abort a sleep transition?

The core should prevent that window. If necessary devices with
different wakeup settings should be resumed before starting system sleep.
 
> If a remote-wakeup request should abort the system sleep, how do we
> make this happen?

Good question. I have no answer.

> If a remote-wakeup request doesn't abort the system sleep, then what
> happens to it?  Obviously it should get serviced when the system wakes
> up again.  But what if the device was runtime-suspended when the sleep
> began?  How do we make sure it gets runtime-resumed when the sleep
> ends?
> 
> I can see two possibilities here.  One is that the remote-wakeup
> request triggered an asynchronous pm_request_wakeup().  In this case
> the request is sitting on the PM workqueue and it will automatically be

That sounds very reasonable.

> satisfied after the system wakes back up.  The other possibility is
> that the request triggered a synchronous pm_runtime_resume().  If this
> fails because a sleep transition is in progress, the request will get
> lost.  (The same sort of thing happens when output requests get stuck
> in a queue and the runtime-resume fails.)

That must not happen.

	Regards
		Oliver

Re: System sleep vs. runtime PM

[Alan Stern]

On Wed, 2 Dec 2009, Oliver Neukum wrote:

> Am Mittwoch, 2. Dezember 2009 19:35:34 schrieb Alan Stern:
> > Rafael and Oliver:
> > 
> > I'm in the midst of converting USB over to the runtime PM framework,
> > and I'm getting stuck on the interaction between system sleep and
> > runtime PM.  In fact, I'm so confused about it that it's hard to
> > formulate a coherent set of questions.  So bear with me...
> > 
> > If a driver gets an output request and its device is runtime-suspended,
> > then the driver should do a runtime-resume before sending the output to
> > the device.  If the output request arrives during a system sleep
> > transition then the output should be refused or left to sit in a queue
> > (presumably this won't happen, but it might).
> 
> How can it happen? As the freezer has run before drivers learn
> about a sleep transition, the request would come from a kernel thread.

Exactly.  Or a timer routine, or an interrupt handler, ...

> > However this requires the driver to know when a system sleep is in
> > progress -- it's not enough merely to know whether the device is
> > suspended.  We could make things easier for the driver by failing
> > runtime-resume requests when the device's power.status isn't DPM_ON or
> > DPM_RESUMING.  Does that make sense?
> 
> That is not a good idea as it needs error handling in all drivers
> for a rare corner case. That's bound to cause many bugs.

Drivers are likely to need error handling for failed resumes
regardless, rare though they are.  Besides, the alternative is for
drivers to be aware of the distinction between runtime-suspended and
waiting-for-system-sleep, which is also a rare corner case.

How would you solve this problem?

> > But then what about remote wakeup?  The arrival of a remote-wakeup
> > request during a sleep transition has always been problematic.  Should
> > the request be allowed to abort the sleep transition?  (Once the kernel
> > is aware of the request, it is probably no longer stored in the
> > hardware so it won't wake the system back up as soon as we go to
> > sleep.)
> 
> It should abort the sleep if the device would wake the system from sleep.

In other words, if the device is enabled for remote wakeup during 
system sleep (which is -- or soon will be -- different from supporting 
remote wakeup during runtime suspend).

> > What if the device is enabled for runtime remote wakeup and not for
> > wakeup from system sleep?  There will be a window during which the
> > wakeup settings are wrong.  Should a wakeup request from such a device
> > be allowed to abort a sleep transition?
> 
> The core should prevent that window. If necessary devices with
> different wakeup settings should be resumed before starting system sleep.

This is a little difficult.  Only the bus subsystem or the driver --
not the PM core -- knows if the extra wakeup is needed, and the core
doesn't send out a general announcement prior to starting the sleep.  
The closest we come is the "prepare" stage.

I suppose during "prepare", the PM core could runtime-resume any device 
with differing wakeup settings, even the ones that don't need it.

> > If a remote-wakeup request should abort the system sleep, how do we
> > make this happen?
> 
> Good question. I have no answer.

Clearly it will depend on how the request is handled.  No doubt a
common approach will be to submit a runtime-resume request.  (That's
not what USB will do though; it will queue its own work item on the
runtime PM workqueue).  We could detect at such points whether a sleep
transition has started and set a flag to abort it.  The PM core could
even export a routine for drivers to call when they want to abort a
system sleep.

> > If a remote-wakeup request doesn't abort the system sleep, then what
> > happens to it?  Obviously it should get serviced when the system wakes
> > up again.  But what if the device was runtime-suspended when the sleep
> > began?  How do we make sure it gets runtime-resumed when the sleep
> > ends?
> > 
> > I can see two possibilities here.  One is that the remote-wakeup
> > request triggered an asynchronous pm_request_wakeup().  In this case
> > the request is sitting on the PM workqueue and it will automatically be
> 
> That sounds very reasonable.

Actually this question no longer applies.  We have agreed that a
remote-wakeup request received during a sleep transition should always
abort the sleep, because otherwise the device wouldn't be enabled for 
remote wakeup.

There's yet another issue to discuss.  Suppose a device is
runtime-suspended when a system sleep starts.  When the system wakes
up, should the device be runtime-resumed?  (Assume that the wakeup
settings don't differ; otherwise it has to be.)

Alan Stern

Re: System sleep vs. runtime PM

[Oliver Neukum]

Am Mittwoch, 2. Dezember 2009 23:20:24 schrieb Alan Stern:
> On Wed, 2 Dec 2009, Oliver Neukum wrote:
> > Am Mittwoch, 2. Dezember 2009 19:35:34 schrieb Alan Stern:
> > > Rafael and Oliver:
> > >
> > > I'm in the midst of converting USB over to the runtime PM framework,
> > > and I'm getting stuck on the interaction between system sleep and
> > > runtime PM.  In fact, I'm so confused about it that it's hard to
> > > formulate a coherent set of questions.  So bear with me...
> > >
> > > If a driver gets an output request and its device is runtime-suspended,
> > > then the driver should do a runtime-resume before sending the output to
> > > the device.  If the output request arrives during a system sleep
> > > transition then the output should be refused or left to sit in a queue
> > > (presumably this won't happen, but it might).
> >
> > How can it happen? As the freezer has run before drivers learn
> > about a sleep transition, the request would come from a kernel thread.
> 
> Exactly.  Or a timer routine, or an interrupt handler, ...

Yes, but in these cases you cannot sleep anyway and must use an
asynchronous resume. These cases are rare. HID mainly. Generally
the freezer is your guarantee.

If you must use an asynchronous resumption request, the problem
becomes moot. You queue IO in all cases and the resumption will
be deferred until after the system will have woken up as the work
queue is frozen.

> > > However this requires the driver to know when a system sleep is in
> > > progress -- it's not enough merely to know whether the device is
> > > suspended.  We could make things easier for the driver by failing
> > > runtime-resume requests when the device's power.status isn't DPM_ON or
> > > DPM_RESUMING.  Does that make sense?
> >
> > That is not a good idea as it needs error handling in all drivers
> > for a rare corner case. That's bound to cause many bugs.
> 
> Drivers are likely to need error handling for failed resumes
> regardless, rare though they are.  Besides, the alternative is for

But then they just return -EIO. That's simple. Queuing data isn't.

> drivers to be aware of the distinction between runtime-suspended and
> waiting-for-system-sleep, which is also a rare corner case.
> 
> How would you solve this problem?

I just love the freezer.

> In other words, if the device is enabled for remote wakeup during
> system sleep (which is -- or soon will be -- different from supporting
> remote wakeup during runtime suspend).

Yes.

> > > What if the device is enabled for runtime remote wakeup and not for
> > > wakeup from system sleep?  There will be a window during which the
> > > wakeup settings are wrong.  Should a wakeup request from such a device
> > > be allowed to abort a sleep transition?
> >
> > The core should prevent that window. If necessary devices with
> > different wakeup settings should be resumed before starting system sleep.
> 
> This is a little difficult.  Only the bus subsystem or the driver --
> not the PM core -- knows if the extra wakeup is needed, and the core
> doesn't send out a general announcement prior to starting the sleep.
> The closest we come is the "prepare" stage.
> 
> I suppose during "prepare", the PM core could runtime-resume any device
> with differing wakeup settings, even the ones that don't need it.

This is necessary.

> > > If a remote-wakeup request should abort the system sleep, how do we
> > > make this happen?
> >
> > Good question. I have no answer.
> 
> Clearly it will depend on how the request is handled.  No doubt a
> common approach will be to submit a runtime-resume request.  (That's
> not what USB will do though; it will queue its own work item on the
> runtime PM workqueue).  We could detect at such points whether a sleep
> transition has started and set a flag to abort it.  The PM core could
> even export a routine for drivers to call when they want to abort a
> system sleep.

Yes, I have no thoughts on that.

> > > If a remote-wakeup request doesn't abort the system sleep, then what
> > > happens to it?  Obviously it should get serviced when the system wakes
> > > up again.  But what if the device was runtime-suspended when the sleep
> > > began?  How do we make sure it gets runtime-resumed when the sleep
> > > ends?
> > >
> > > I can see two possibilities here.  One is that the remote-wakeup
> > > request triggered an asynchronous pm_request_wakeup().  In this case
> > > the request is sitting on the PM workqueue and it will automatically be
> >
> > That sounds very reasonable.
> 
> Actually this question no longer applies.  We have agreed that a
> remote-wakeup request received during a sleep transition should always
> abort the sleep, because otherwise the device wouldn't be enabled for
> remote wakeup.

An evil thought, do we have devices that will always request remote wakeup,
without an option to switch it off? What about OTG?

> There's yet another issue to discuss.  Suppose a device is
> runtime-suspended when a system sleep starts.  When the system wakes
> up, should the device be runtime-resumed?  (Assume that the wakeup
> settings don't differ; otherwise it has to be.)

That depends on how sure we are to never drop a remote wakeup,
not even if requests race or for any other reason. And we must always
know what woke us up.

	Regards
		Oliver

Re: System sleep vs. runtime PM

[Rafael J. Wysocki]

On Thursday 03 December 2009, Oliver Neukum wrote:
> Am Mittwoch, 2. Dezember 2009 23:20:24 schrieb Alan Stern:
> > On Wed, 2 Dec 2009, Oliver Neukum wrote:
> > > Am Mittwoch, 2. Dezember 2009 19:35:34 schrieb Alan Stern:
> > > > Rafael and Oliver:
> > > >
> > > > I'm in the midst of converting USB over to the runtime PM framework,
> > > > and I'm getting stuck on the interaction between system sleep and
> > > > runtime PM.  In fact, I'm so confused about it that it's hard to
> > > > formulate a coherent set of questions.  So bear with me...
> > > >
> > > > If a driver gets an output request and its device is runtime-suspended,
> > > > then the driver should do a runtime-resume before sending the output to
> > > > the device.  If the output request arrives during a system sleep
> > > > transition then the output should be refused or left to sit in a queue
> > > > (presumably this won't happen, but it might).
> > >
> > > How can it happen? As the freezer has run before drivers learn
> > > about a sleep transition, the request would come from a kernel thread.
> > 
> > Exactly.  Or a timer routine, or an interrupt handler, ...
> 
> Yes, but in these cases you cannot sleep anyway and must use an
> asynchronous resume. These cases are rare. HID mainly. Generally
> the freezer is your guarantee.
> 
> If you must use an asynchronous resumption request, the problem
> becomes moot. You queue IO in all cases and the resumption will
> be deferred until after the system will have woken up as the work
> queue is frozen.
> 
> > > > However this requires the driver to know when a system sleep is in
> > > > progress -- it's not enough merely to know whether the device is
> > > > suspended.  We could make things easier for the driver by failing
> > > > runtime-resume requests when the device's power.status isn't DPM_ON or
> > > > DPM_RESUMING.  Does that make sense?
> > >
> > > That is not a good idea as it needs error handling in all drivers
> > > for a rare corner case. That's bound to cause many bugs.
> > 
> > Drivers are likely to need error handling for failed resumes
> > regardless, rare though they are.  Besides, the alternative is for
> 
> But then they just return -EIO. That's simple. Queuing data isn't.
> 
> > drivers to be aware of the distinction between runtime-suspended and
> > waiting-for-system-sleep, which is also a rare corner case.
> > 
> > How would you solve this problem?

I think such drivers will need to implement the ->prepare() and ->complete()
suspend/resume callbacks and make them handle this.  That also may be done at
the bus type level.

> I just love the freezer.
> 
> > In other words, if the device is enabled for remote wakeup during
> > system sleep (which is -- or soon will be -- different from supporting
> > remote wakeup during runtime suspend).
> 
> Yes.
> 
> > > > What if the device is enabled for runtime remote wakeup and not for
> > > > wakeup from system sleep?  There will be a window during which the
> > > > wakeup settings are wrong.  Should a wakeup request from such a device
> > > > be allowed to abort a sleep transition?
> > >
> > > The core should prevent that window. If necessary devices with
> > > different wakeup settings should be resumed before starting system sleep.
> > 
> > This is a little difficult.  Only the bus subsystem or the driver --
> > not the PM core -- knows if the extra wakeup is needed, and the core
> > doesn't send out a general announcement prior to starting the sleep.
> > The closest we come is the "prepare" stage.
> > 
> > I suppose during "prepare", the PM core could runtime-resume any device
> > with differing wakeup settings, even the ones that don't need it.
> 
> This is necessary.

In fact, I'd leave that to the bus type level code, because the core doesn't
even know what "differing wakeup settings" actually mean.

> > > > If a remote-wakeup request should abort the system sleep, how do we
> > > > make this happen?
> > >
> > > Good question. I have no answer.
> > 
> > Clearly it will depend on how the request is handled.  No doubt a
> > common approach will be to submit a runtime-resume request.  (That's
> > not what USB will do though; it will queue its own work item on the
> > runtime PM workqueue).  We could detect at such points whether a sleep
> > transition has started and set a flag to abort it.  The PM core could
> > even export a routine for drivers to call when they want to abort a
> > system sleep.
> 
> Yes, I have no thoughts on that.
> 
> > > > If a remote-wakeup request doesn't abort the system sleep, then what
> > > > happens to it?  Obviously it should get serviced when the system wakes
> > > > up again.  But what if the device was runtime-suspended when the sleep
> > > > began?  How do we make sure it gets runtime-resumed when the sleep
> > > > ends?
> > > >
> > > > I can see two possibilities here.  One is that the remote-wakeup
> > > > request triggered an asynchronous pm_request_wakeup().  In this case
> > > > the request is sitting on the PM workqueue and it will automatically be
> > >
> > > That sounds very reasonable.
> > 
> > Actually this question no longer applies.  We have agreed that a
> > remote-wakeup request received during a sleep transition should always
> > abort the sleep, because otherwise the device wouldn't be enabled for
> > remote wakeup.
> 
> An evil thought, do we have devices that will always request remote wakeup,
> without an option to switch it off? What about OTG?
> 
> > There's yet another issue to discuss.  Suppose a device is
> > runtime-suspended when a system sleep starts.  When the system wakes
> > up, should the device be runtime-resumed?  (Assume that the wakeup
> > settings don't differ; otherwise it has to be.)
> 
> That depends on how sure we are to never drop a remote wakeup,
> not even if requests race or for any other reason. And we must always
> know what woke us up.

On PCI there is the problem that you may need to reprogram devices for
system wakeup, so generally you need to resume them during system suspend
if they have been run-time suspended.  In such a case there's no way we can
handle a wakeup request that comes in after the device has been resumed
(during the system suspend) due to the way in which ACPI wakeup GPEs are set
up (it happens after the I/O devices' suspend and suspend_noirq callbacks have
been executed).

Thanks,
Rafael

Re: System sleep vs. runtime PM

[Alan Stern]

On Thu, 3 Dec 2009, Oliver Neukum wrote:

> An evil thought, do we have devices that will always request remote wakeup,
> without an option to switch it off? What about OTG?

I don't know of any such devices.  USB OTG will depend on the 
controller design, but I would be surprised to find a controller that 
didn't let you turn off wakeup-IRQ generation.

> > There's yet another issue to discuss.  Suppose a device is
> > runtime-suspended when a system sleep starts.  When the system wakes
> > up, should the device be runtime-resumed?  (Assume that the wakeup
> > settings don't differ; otherwise it has to be.)
> 
> That depends on how sure we are to never drop a remote wakeup,
> not even if requests race or for any other reason. And we must always
> know what woke us up.

This is a complicated question and I will discuss it at greater length 
in another email.

However, I don't see any reason to know what woke us up.  All we really 
need to know is what devices have wakeup requests outstanding when the 
system resume is finished.  It doesn't matter which request came first 
(presumably that was the one which woke us up).

Alan Stern

Re: System sleep vs. runtime PM

[Alan Stern]

On Thu, 3 Dec 2009, Rafael J. Wysocki wrote:

> > > > > However this requires the driver to know when a system sleep is in
> > > > > progress -- it's not enough merely to know whether the device is
> > > > > suspended.  We could make things easier for the driver by failing
> > > > > runtime-resume requests when the device's power.status isn't DPM_ON or
> > > > > DPM_RESUMING.  Does that make sense?
> > > >
> > > > That is not a good idea as it needs error handling in all drivers
> > > > for a rare corner case. That's bound to cause many bugs.
> > > 
> > > Drivers are likely to need error handling for failed resumes
> > > regardless, rare though they are.  Besides, the alternative is for
> > 
> > But then they just return -EIO. That's simple. Queuing data isn't.
> > 
> > > drivers to be aware of the distinction between runtime-suspended and
> > > waiting-for-system-sleep, which is also a rare corner case.
> > > 
> > > How would you solve this problem?
> 
> I think such drivers will need to implement the ->prepare() and ->complete()
> suspend/resume callbacks and make them handle this.  That also may be done at
> the bus type level.

I prefer Oliver's argument.  Once processes are frozen, trouble can be
caused only by kernel threads submitting or carrying out I/O requests.  
(For example, the block layer could ask usb-storage's thread to
transfer data to/from a USB drive after the drive has been suspended.)  
The question is: Do we need to worry about this or can we assume it
won't arise?

The simplest solution is to make pm_runtime_resume() abort the sleep
transition if it is called for a device not in the DPM_ON or
DPM_PREPARING states -- and do a WARN_ON if the device isn't
wakeup-enabled to let people know what went wrong.

Alternatively, if pm_runtime_resume() is called for a device that isn't
in DPM_ON or DPM_PREPARING and isn't wakeup-enabled, it could block
until the system sleep is over.  But this is likely to deadlock with
the system resume, so I don't think it will work.


> > > I suppose during "prepare", the PM core could runtime-resume any device
> > > with differing wakeup settings, even the ones that don't need it.
> > 
> > This is necessary.
> 
> In fact, I'd leave that to the bus type level code, because the core doesn't
> even know what "differing wakeup settings" actually mean.

But the core does know how to tell whether the settings are different.  
Centralizing this seems like a good idea, and I don't think too many 
devices would be affected.  Individual bus subsystems could still 
runtime-resume devices that the PM core missed, if they wanted to.


> > > > > If a remote-wakeup request should abort the system sleep, how do we
> > > > > make this happen?
> > > >
> > > > Good question. I have no answer.
> > > 
> > > Clearly it will depend on how the request is handled.  No doubt a
> > > common approach will be to submit a runtime-resume request.  (That's
> > > not what USB will do though; it will queue its own work item on the
> > > runtime PM workqueue).  We could detect at such points whether a sleep
> > > transition has started and set a flag to abort it.  The PM core could
> > > even export a routine for drivers to call when they want to abort a
> > > system sleep.
> > 
> > Yes, I have no thoughts on that.

I had some thoughts last night.  Starting just before processes are
frozen, we should make a check every time a wakeup request is added to
the PM workqueue.  If the device is enabled for remote wakeup, we abort
the system sleep.

It's easy enough to check workqueue additions in runtime.c, but drivers
may want to add their own work items to the PM workqueue.  There are
two ways to handle this.  First, we could make the workqueue private
and export functions to enqueue suspend and resume work items.  
Second, we could make the workqueue public and export a function for
drivers to call when they enqueue a resume request (the function would
check if the device is wakeup-enabled and abort the system sleep if it
is).  I don't know which alternative is better.  Any suggestions?

The details of how the sleep transition gets aborted don't matter.  
It can be something as simple as a flag that gets checked at various 
stages.


> On PCI there is the problem that you may need to reprogram devices for
> system wakeup, so generally you need to resume them during system suspend
> if they have been run-time suspended.  In such a case there's no way we can
> handle a wakeup request that comes in after the device has been resumed
> (during the system suspend) due to the way in which ACPI wakeup GPEs are set
> up (it happens after the I/O devices' suspend and suspend_noirq callbacks have
> been executed).

Wouldn't the GPE cause the system to wake up immediately, as soon as it
goes to sleep?  That's just as good as aborting the sleep.


> > > There's yet another issue to discuss.  Suppose a device is
> > > runtime-suspended when a system sleep starts.  When the system wakes
> > > up, should the device be runtime-resumed?  (Assume that the wakeup
> > > settings don't differ; otherwise it has to be.)

Let me discuss this is some detail.  I'm not so much concerned about 
missing wakeup requests but rather about making sure that (1) we don't 
resume devices unnecessarily and (2) the device's physical power state 
agrees with what the system thinks it is.

None of this matters for devices that are runtime-active when a system
sleep starts.  This is only about devices that are runtime-suspended.  
Also, I expect the decisions discussed here will be made at the bus
subsystem or driver level, not by the PM core.

First, consider resume from hibernation (the RESTORE transition).  In
this case we have no idea what the device's physical state is, so the
only safe thing to do is make sure it is powered on and runtime-active.  
The normal restore method call should leave the device powered on; the
question is how to get it into RPM_ACTIVE without confusing the driver,
the bus subsystem, or the PM core.

Now consider the THAW and RESUME transitions.  Here the driver does
know the device's physical state.  Under what circumstances should it
avoid powering-up the device?  As I see it:

	The device might need to be active in order to change the
	wakeup settings.

	If the usage_count > 1 then the device will likely be used 
	soon, so it should be powered up.  (Remember that the 
	usage_count is always at least 1 because the PM core increments 
	it during a system sleep.)

There might be other device-specific or bus-specific reasons (e.g.,
reset-resume needed for USB), but in general this seems good enough.  
If none of the exceptions applies then we might as well leave the
device turned off.  But if the driver does decide to power-up the
device, we face the same problem as above: How to get it into
RPM_ACTIVE without confusing everybody?

Probably the best answer is for the bus subsystem or driver to call
pm_runtime_resume().  Of course, they would then have to be smart
enough to handle runtime_resume method calls correctly when the device
is already active.

Is it reasonable for the resume and thaw methods to inform the PM core
as to whether they skipped the power-up?  For instance, they could
return -ERESTART in such cases.  Then the PM core could automatically
call pm_runtime_resume() when needed.

There's one issue I skipped over.  Suppose a device is left suspended,
but one of its children needs to be resumed.  The only way to do it
would be to do a runtime resume on the child, because only then would
the parent device get woken up first.  Does this mean that the best way
to implement resume and thaw is to have them simply call
pm_runtime_resume()?

Alan Stern

Re: System sleep vs. runtime PM

[Oliver Neukum]

Am Donnerstag, 3. Dezember 2009 16:25:30 schrieb Alan Stern:
> On Thu, 3 Dec 2009, Oliver Neukum wrote:

> > > There's yet another issue to discuss.  Suppose a device is
> > > runtime-suspended when a system sleep starts.  When the system wakes
> > > up, should the device be runtime-resumed?  (Assume that the wakeup
> > > settings don't differ; otherwise it has to be.)
> >
> > That depends on how sure we are to never drop a remote wakeup,
> > not even if requests race or for any other reason. And we must always
> > know what woke us up.
> 
> This is a complicated question and I will discuss it at greater length
> in another email.
> 
> However, I don't see any reason to know what woke us up.  All we really
> need to know is what devices have wakeup requests outstanding when the
> system resume is finished.  It doesn't matter which request came first
> (presumably that was the one which woke us up).

That assumes that the firmware doesn't do anything stupid with
pending remote wakeups as it resumes the system. I'd call this
unwarranted optimism. The conservative solution would be to resume
every device whose driver has requested remote wakeup be enabled.

	Regards
		Oliver

Re: System sleep vs. runtime PM

[Alan Stern]

On Thu, 3 Dec 2009, Oliver Neukum wrote:

> > However, I don't see any reason to know what woke us up.  All we really
> > need to know is what devices have wakeup requests outstanding when the
> > system resume is finished.  It doesn't matter which request came first
> > (presumably that was the one which woke us up).
> 
> That assumes that the firmware doesn't do anything stupid with
> pending remote wakeups as it resumes the system. I'd call this
> unwarranted optimism. The conservative solution would be to resume
> every device whose driver has requested remote wakeup be enabled.

Drivers certainly can use that as one of their criteria for whether to 
power-up a device during system resume.

Alan Stern

Re: System sleep vs. runtime PM

[Oliver Neukum]

Am Donnerstag, 3. Dezember 2009 18:50:28 schrieb Alan Stern:
> On Thu, 3 Dec 2009, Oliver Neukum wrote:
> > > However, I don't see any reason to know what woke us up.  All we really
> > > need to know is what devices have wakeup requests outstanding when the
> > > system resume is finished.  It doesn't matter which request came first
> > > (presumably that was the one which woke us up).
> >
> > That assumes that the firmware doesn't do anything stupid with
> > pending remote wakeups as it resumes the system. I'd call this
> > unwarranted optimism. The conservative solution would be to resume
> > every device whose driver has requested remote wakeup be enabled.
> 
> Drivers certainly can use that as one of their criteria for whether to
> power-up a device during system resume.

Leaving this decision to the drivers won't work because they don't
know enough.
For reliable operation we must guarantee no remote wakeup is lost.
But remote wakeups must travel a whole chain of busses and the
guarantees is of the weakest link apply. Drivers must not know
over which busses they are connected higher up.

Therefore you must make the decision in core. But the core doesn't
know specifics. Unless you really want to overengineer this and compute
the reliability of each path, resuming only those whose drivers have
requested that remote wakeup be enabled is the best you can do.

	Regards
		Oliver

Re: System sleep vs. runtime PM

[Alan Stern]

On Thu, 3 Dec 2009, Oliver Neukum wrote:

> Am Donnerstag, 3. Dezember 2009 18:50:28 schrieb Alan Stern:
> > On Thu, 3 Dec 2009, Oliver Neukum wrote:
> > > > However, I don't see any reason to know what woke us up.  All we really
> > > > need to know is what devices have wakeup requests outstanding when the
> > > > system resume is finished.  It doesn't matter which request came first
> > > > (presumably that was the one which woke us up).
> > >
> > > That assumes that the firmware doesn't do anything stupid with
> > > pending remote wakeups as it resumes the system. I'd call this
> > > unwarranted optimism. The conservative solution would be to resume
> > > every device whose driver has requested remote wakeup be enabled.
> > 
> > Drivers certainly can use that as one of their criteria for whether to
> > power-up a device during system resume.
> 
> Leaving this decision to the drivers won't work because they don't
> know enough.
> For reliable operation we must guarantee no remote wakeup is lost.

That's not necessarily so.  If remote wakeup is disabled at a device
between the CPU and the source device, then wakeup events are indeed
allowed to get lost.  For example, even though a USB hub may be enabled
for remote wakeup, if its host controller isn't then a wakeup event
won't generate an IRQ and so won't awaken the system.  And in fact this
behavior may be desired by the user.  After all, who would want their
laptop to wake up merely because a USB mouse was unplugged?

> But remote wakeups must travel a whole chain of busses and the
> guarantees is of the weakest link apply. Drivers must not know
> over which busses they are connected higher up.
> 
> Therefore you must make the decision in core.

This doesn't follow.  The decision is made individually for each 
device, based on whether or not the device is enabled for remote 
wakeup.  There's no need to know anything about higher-up buses.

>  But the core doesn't
> know specifics. Unless you really want to overengineer this and compute
> the reliability of each path, resuming only those whose drivers have
> requested that remote wakeup be enabled is the best you can do.

Isn't that what I agreed drivers should do?

Alan Stern

Re: System sleep vs. runtime PM

[Oliver Neukum]

Am Donnerstag, 3. Dezember 2009 20:52:15 schrieb Alan Stern:

> That's not necessarily so.  If remote wakeup is disabled at a device
> between the CPU and the source device, then wakeup events are indeed
> allowed to get lost.  For example, even though a USB hub may be enabled
> for remote wakeup, if its host controller isn't then a wakeup event
> won't generate an IRQ and so won't awaken the system.  And in fact this
> behavior may be desired by the user.  After all, who would want their
> laptop to wake up merely because a USB mouse was unplugged?

That is no problem while the system is asleep. It is no good once the system
wakes up.

> >  But the core doesn't
> > know specifics. Unless you really want to overengineer this and compute
> > the reliability of each path, resuming only those whose drivers have
> > requested that remote wakeup be enabled is the best you can do.
> 
> Isn't that what I agreed drivers should do?

If the algorithm is clear and based only on remote wakeup, why
would you want to involve drivers?

I am afraid we are having a misunderstanding. Could you elaborate?

	Regards
		Oliver

Re: System sleep vs. runtime PM

[Alan Stern]

On Thu, 3 Dec 2009, Oliver Neukum wrote:

> Am Donnerstag, 3. Dezember 2009 20:52:15 schrieb Alan Stern:
> 
> > That's not necessarily so.  If remote wakeup is disabled at a device
> > between the CPU and the source device, then wakeup events are indeed
> > allowed to get lost.  For example, even though a USB hub may be enabled
> > for remote wakeup, if its host controller isn't then a wakeup event
> > won't generate an IRQ and so won't awaken the system.  And in fact this
> > behavior may be desired by the user.  After all, who would want their
> > laptop to wake up merely because a USB mouse was unplugged?
> 
> That is no problem while the system is asleep. It is no good once the system
> wakes up.

Of course.  That's part of the reason why we will have two separate
controls for remote wakeup: one for runtime PM and one for system
sleep.

> > >  But the core doesn't
> > > know specifics. Unless you really want to overengineer this and compute
> > > the reliability of each path, resuming only those whose drivers have
> > > requested that remote wakeup be enabled is the best you can do.
> > 
> > Isn't that what I agreed drivers should do?
> 
> If the algorithm is clear and based only on remote wakeup, why
> would you want to involve drivers?

1: This criterion is based only on remote wakeup, but others aren't.  
There were two other criteria I mentioned in an earlier email message 
(power-up needed to change wakeup settings, usage count > 1).

2: The core can't make the decision about whether to power-up a device.  
All it can do is notify drivers that the system sleep is over; the 
decision about what to do then is up to the drivers.

> I am afraid we are having a misunderstanding. Could you elaborate?

In a nutshell:  The PM core _has_ to call thaw and resume methods; 
otherwise drivers won't know the system sleep is over.  The question is 
what those methods should do if the device was runtime-suspended before 
the sleep began.  Should they power-up the device (and somehow change 
its runtime_status)?  Or should they leave it powered down?

I tried to list conditions under which a driver would have to power-up 
the device.  If none of the conditions hold then the device may remain 
powered down.

Alan Stern

System sleep vs. runtime PM

[Alan Stern]

Rafael:

What should happen during system wakeup if a device was
runtime-suspended when the sleep began?

Although there are circumstances where the device could remain
suspended, on the whole it seems a lot safer and easier always to go
back to full power.  Certainly it's necessary during the RESTORE phase
of hibernation, because then the device's physical state is unknown.  
And even during the RESUME phase of system suspend, the firmware could
have made some unwelcome changes to the device.  In the THAW phase of
hibernation it may not be necessary -- but consider the disk drive
that will be used for storing the memory image.  If it was
runtime-suspended then it will need a runtime-resume, and it will
probably have to use an async pm_request_resume() call, but the PM
workqueue will be frozen so nothing will happen.  Not to mention all
the issues involved with remote wakeup.

So okay.  This should be mentioned in the documentation since it is
important, especially for drivers that have different pathways for
system resume and runtime resume.

This means that when a driver's system-resume routine is finished, the
device will be at full power and so dev->power.runtime_status needs to
be changed to RPM_ACTIVE.  Hence a runtime-PM-aware driver will have to
do something like:

	pm_runtime_disable(dev);
	pm_runtime_set_active(dev);
	pm_runtime_enable(dev);

in its system-resume routine.  There probably should be a function in
the PM core to encapsulate this.  Whether there is or not, drivers must
know to do it so it should be mentioned in the documentation.  (The PM
core can't do it automatically, because it would mess up drivers that
aren't runtime-PM-aware.)

Also, when the system resume is finished, drivers need a chance to 
runtime-suspend again.  Hence the pm_runtime_put_noidle() call in 
dpm_complete() should be changed to pm_runtime_put_sync().

Alan Stern

Re: System sleep vs. runtime PM

[Rafael J. Wysocki]

On Saturday 12 December 2009, Alan Stern wrote:
> Rafael:

Hi,

> What should happen during system wakeup if a device was
> runtime-suspended when the sleep began?
> 
> Although there are circumstances where the device could remain
> suspended, on the whole it seems a lot safer and easier always to go
> back to full power.

I think we should do this, at least to start with.  For things like PCI there's
no other choice really, because we usually get pully powered devices from
the BIOS.

> Certainly it's necessary during the RESTORE phase
> of hibernation, because then the device's physical state is unknown.  
> And even during the RESUME phase of system suspend, the firmware could
> have made some unwelcome changes to the device.  In the THAW phase of
> hibernation it may not be necessary -- but consider the disk drive
> that will be used for storing the memory image.  If it was
> runtime-suspended then it will need a runtime-resume, and it will
> probably have to use an async pm_request_resume() call, but the PM
> workqueue will be frozen so nothing will happen.  Not to mention all
> the issues involved with remote wakeup.
> 
> So okay.  This should be mentioned in the documentation since it is
> important, especially for drivers that have different pathways for
> system resume and runtime resume.

Agreed.

> This means that when a driver's system-resume routine is finished, the
> device will be at full power and so dev->power.runtime_status needs to
> be changed to RPM_ACTIVE.  Hence a runtime-PM-aware driver will have to
> do something like:
> 
> 	pm_runtime_disable(dev);
> 	pm_runtime_set_active(dev);
> 	pm_runtime_enable(dev);
> 
> in its system-resume routine.

Unless it does runtime resume during system suspend, that is.

> There probably should be a function in the PM core to encapsulate this.

Very likely, but I'd defer introducing all of these helper functions to the
when somebody really needs them, ie. introduce them along with the first
user.

> Whether there is or not, drivers must know to do it so it should be mentioned
> in the documentation.  (The PM core can't do it automatically, because it
> would mess up drivers that aren't runtime-PM-aware.)

Agreed.

> Also, when the system resume is finished, drivers need a chance to 
> runtime-suspend again.  Hence the pm_runtime_put_noidle() call in 
> dpm_complete() should be changed to pm_runtime_put_sync().

OK

Are you going to send a patch or do you want me to prepare one?

Rafael

Re: System sleep vs. runtime PM

[Alan Stern]

On Sat, 12 Dec 2009, Rafael J. Wysocki wrote:

> > Also, when the system resume is finished, drivers need a chance to 
> > runtime-suspend again.  Hence the pm_runtime_put_noidle() call in 
> > dpm_complete() should be changed to pm_runtime_put_sync().
> 
> OK
> 
> Are you going to send a patch or do you want me to prepare one?

We're talking about mostly documentation changes, right?  I can write 
them.

But the code changes will present a small problem.  I'll need them for
the USB development.  Will it be okay to ask Greg KH to put them in his
tree after you have accepted them into yours?  That will avoid
cross-subsystem build errors.  (We just got through one of them 
involving David Miller and I'd rather avoid any more.)

There's one other thing (should have brought it up earlier): Do you
mind having the runtime-PM callbacks invoke the bus type or the bus
class methods instead of the bus methods, if they are defined?  That
is, do them in the same way the system-PM callbacks work.  The USB code
will definitely want different methods for the different types.  In 
fact, I've already written a version that tests the bus type within the 
callback routines.

Alan Stern

Re: System sleep vs. runtime PM

[Rafael J. Wysocki]

On Saturday 12 December 2009, Alan Stern wrote:
> On Sat, 12 Dec 2009, Rafael J. Wysocki wrote:
> 
> > > Also, when the system resume is finished, drivers need a chance to 
> > > runtime-suspend again.  Hence the pm_runtime_put_noidle() call in 
> > > dpm_complete() should be changed to pm_runtime_put_sync().
> > 
> > OK
> > 
> > Are you going to send a patch or do you want me to prepare one?
> 
> We're talking about mostly documentation changes, right?  I can write 
> them.

OK

> But the code changes will present a small problem.  I'll need them for
> the USB development.  Will it be okay to ask Greg KH to put them in his
> tree after you have accepted them into yours?  That will avoid
> cross-subsystem build errors.  (We just got through one of them 
> involving David Miller and I'd rather avoid any more.)

I'm fine with that.

> There's one other thing (should have brought it up earlier): Do you
> mind having the runtime-PM callbacks invoke the bus type or the bus

I guess you mean device type and device class?

> class methods instead of the bus methods, if they are defined?  That
> is, do them in the same way the system-PM callbacks work.  The USB code
> will definitely want different methods for the different types.  In 
> fact, I've already written a version that tests the bus type within the 
> callback routines.

That's a good question.  I didn't anticipate that, so it's hard to tell right
now.  I don't really see any fundamental obstacles now, though.

Rafael