Re: [PATCH] Remove process freezer from suspend to RAM pathway

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Alan Stern]

On Sun, 8 Jul 2007, Rafael J. Wysocki wrote:

> I'm all for changing this infrastructure, but in an organized way (ie. we
> discuss what to do next, we do that and then we go to the next step) and in
> the order that everyone will be comfortable with.
> 
> So, let's finish this thread and start over from discussing what needs to be
> done, how (ie. in what order etc.) we are going to do that and who is going to
> do what.  Shall we?

IMO we should start by using the new notifier chain and by implementing 
a central "icebox" routine.  Then we can forbid device registration 
during suspend.

It might also be a good idea to add a freezable keventd-like workqueue 
specifically intended for things that need to block during a suspend.  
Although maybe this will end up being unnecessary; it's too soon to 
tell.

Alan Stern

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Pavel Machek]

Hi!

> > I'm all for changing this infrastructure, but in an organized way (ie. we
> > discuss what to do next, we do that and then we go to the next step) and in
> > the order that everyone will be comfortable with.
> > 
> > So, let's finish this thread and start over from discussing what needs to be
> > done, how (ie. in what order etc.) we are going to do that and who is going to
> > do what.  Shall we?
> 
> IMO we should start by using the new notifier chain and by implementing 
> a central "icebox" routine.  Then we can forbid device registration 
> during suspend.
> 
> It might also be a good idea to add a freezable keventd-like workqueue 
> specifically intended for things that need to block during a suspend.  
> Although maybe this will end up being unnecessary; it's too soon to 
> tell.

We actually had patches for freezeable workqueues at one point.
									Pavel
-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Benjamin Herrenschmidt]

On Sun, 2007-07-08 at 16:22 -0400, Alan Stern wrote:
> On Sun, 8 Jul 2007, Rafael J. Wysocki wrote:
> 
> > I'm all for changing this infrastructure, but in an organized way (ie. we
> > discuss what to do next, we do that and then we go to the next step) and in
> > the order that everyone will be comfortable with.
> > 
> > So, let's finish this thread and start over from discussing what needs to be
> > done, how (ie. in what order etc.) we are going to do that and who is going to
> > do what.  Shall we?
> 
> IMO we should start by using the new notifier chain and by implementing 
> a central "icebox" routine.  Then we can forbid device registration 
> during suspend.
> 
> It might also be a good idea to add a freezable keventd-like workqueue 
> specifically intended for things that need to block during a suspend.  
> Although maybe this will end up being unnecessary; it's too soon to 
> tell.

I think looking at the kmallo/gfp issue I mentioned earlier should be
done asap. I can try to spare some time for it this week but I can't
promise, I'm actually swamped by something totally unrelated at the
moment.

Another issue that's been a problem forever with suspend is the
synchronous request_firmware interface. Lots of drivers do that in
resume() which will generally not work.

Ben.

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Oliver Neukum]

Am Sonntag, 8. Juli 2007 schrieb Benjamin Herrenschmidt:
> Another issue that's been a problem forever with suspend is the
> synchronous request_firmware interface. Lots of drivers do that in
> resume() which will generally not work.

Unfortunately yes they do. We now have the notifier chains, which
can be used. Firmware, especially if it is large, must be requested
while the system is still fully functional and paging will work.

Asynchronocity will not help much because the devices must work
after resumingand delaying that will not help with respect to needing
devices to get at the firmware.

	Regards
		Oliver

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Benjamin Herrenschmidt]

On Mon, 2007-07-09 at 08:52 +0200, Oliver Neukum wrote:
> Am Sonntag, 8. Juli 2007 schrieb Benjamin Herrenschmidt:
> > Another issue that's been a problem forever with suspend is the
> > synchronous request_firmware interface. Lots of drivers do that in
> > resume() which will generally not work.
> 
> Unfortunately yes they do. We now have the notifier chains, which
> can be used. Firmware, especially if it is large, must be requested
> while the system is still fully functional and paging will work.
> 
> Asynchronocity will not help much because the devices must work
> after resumingand delaying that will not help with respect to needing
> devices to get at the firmware.

True, pre-loading is the best approach.

Ben.

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Rafael J. Wysocki]

On Monday, 9 July 2007 11:14, Benjamin Herrenschmidt wrote:
> On Mon, 2007-07-09 at 08:52 +0200, Oliver Neukum wrote:
> > Am Sonntag, 8. Juli 2007 schrieb Benjamin Herrenschmidt:
> > > Another issue that's been a problem forever with suspend is the
> > > synchronous request_firmware interface. Lots of drivers do that in
> > > resume() which will generally not work.
> > 
> > Unfortunately yes they do. We now have the notifier chains, which
> > can be used. Firmware, especially if it is large, must be requested
> > while the system is still fully functional and paging will work.
> > 
> > Asynchronocity will not help much because the devices must work
> > after resumingand delaying that will not help with respect to needing
> > devices to get at the firmware.
> 
> True, pre-loading is the best approach.

Agreed.

Greetings,
Rafael


-- 
"Premature optimization is the root of all evil." - Donald Knuth

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Alan Stern]

On Sun, 8 Jul 2007, Benjamin Herrenschmidt wrote:

> Note that we could have also a per-device "icebox" (just a waitqueue).
> Might be nice for a device to resume whatever it froze just after it
> resumed itself, might even be necessary in case whatever thread got
> frozen is necessary for handling child devices.

This wouldn't need any special infrastructure.  Just a regular 
waitqueue defined in the device's private data structure.

Alan Stern

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Benjamin Herrenschmidt]

> In my defense, you should realize that until Rafael's notifier chain
> was added (just a few weeks ago, still not in mainline I believe) there
> was no other way to do it.  Plug activity needs to be stopped before
> the child devices are suspended, and the PM core does not send any
> notification to drivers at that time.  All it does is activate the 
> freezer.

That's true. That was one of the reason I've always wanted the
pre-suspend and post-resume hooks. (I prefer keeping the ordering there
too, rather than a notifier, but a notifier is fine I suppose).

Among the clients we want here the firmware stuff, the allocators,
etc... to get themselves in conditions that won't deadlock during
suspend cycle.

I think that's a much bigger issue overall than freezer vs. no
freezer :-)

Ben.

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Rafael J. Wysocki]

On Sunday, 8 July 2007 06:39, Benjamin Herrenschmidt wrote:
> 
> > In my defense, you should realize that until Rafael's notifier chain
> > was added (just a few weeks ago, still not in mainline I believe) there
> > was no other way to do it.  Plug activity needs to be stopped before
> > the child devices are suspended, and the PM core does not send any
> > notification to drivers at that time.  All it does is activate the 
> > freezer.
> 
> That's true. That was one of the reason I've always wanted the
> pre-suspend and post-resume hooks. (I prefer keeping the ordering there
> too, rather than a notifier, but a notifier is fine I suppose).
> 
> Among the clients we want here the firmware stuff, the allocators,
> etc... to get themselves in conditions that won't deadlock during
> suspend cycle.
> 
> I think that's a much bigger issue overall than freezer vs. no
> freezer :-)

Yes.  The freezer is only one part of the PM infrastructure that we have
currently and that is not very sophisticated, so to speak.

I'm all for changing this infrastructure, but in an organized way (ie. we
discuss what to do next, we do that and then we go to the next step) and in
the order that everyone will be comfortable with.

So, let's finish this thread and start over from discussing what needs to be
done, how (ie. in what order etc.) we are going to do that and who is going to
do what.  Shall we?

Greetings,
Rafael


-- 
"Premature optimization is the root of all evil." - Donald Knuth

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Rafael J. Wysocki]

On Sunday, 8 July 2007 04:24, Alan Stern wrote:
> On Sun, 8 Jul 2007, Benjamin Herrenschmidt wrote:
> 
> > > Which is exactly my point.  It _doesn't_ work fine without a freezer, 
> > > because the USB stack currently relies on the freezer to prevent plug 
> > > activity.
> > 
> > Putting on hold plug activity has nothing, NOTHING, to do with the half
> > assed piece of deadlocking crap we have now we call a freezer.
> 
> You're wrong; it _does_ have something to do with the freezer.  The 
> connection is that the code uses the freezer to put plug activity on 
> hold.
> 
> You probably meant to say that it _should_ have nothing to do with the 
> freezer.  That's a different matter.  But in any case you should write 
> what you actually mean, rather than just putting down something as 
> inflammatory as possible.
> 
> > As long as you guys keep mixing up all the issues and coming up with
> > totally bogus solutions that cannot work, we won't have a useful suspend
> > (either to RAM or to disk) in linux.
> 
> In my defense, you should realize that until Rafael's notifier chain
> was added (just a few weeks ago, still not in mainline I believe) there
> was no other way to do it.  Plug activity needs to be stopped before
> the child devices are suspended, and the PM core does not send any
> notification to drivers at that time.  All it does is activate the 
> freezer.

Exactly.

And if we really want to get rid of the freezer, we need to _prepare_ for
that.  At least, we need to have an idea of how we will do thing when the
freezer is not there.

For example, how will we check if the system is not in a state in which the
resume is likely to fail?

There are more questions like that, IMO, and we should at least have a list
of them _before_ we remove the freezer.

Greetings,
Rafael


-- 
"Premature optimization is the root of all evil." - Donald Knuth

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Benjamin Herrenschmidt]

> You agree that drivers need to block various activities during suspend.  
> Principally I/O requests, but other things as well.  So when one of 
> these requests arrives, the driver has to make it wait somehow and then 
> has to allow it to proceed at the appropriate time.

Yes. The main thing I see here is that here is nothing common among
drivers in what a "request" is and how it's processed.

For example, for block drivers, it's actually fairly simple to just stop
processing the request queue and wait for pending ones to complete.

For network, in a similar vein, we can mostly just tell the network
stack to stop sending us packets.

That's what I call the "main path". This is often the trivial part to
deal with, mostly because for a whole lot of drivers, it can be done via
a couple of helpers in the subsystem that the driver provides a service
too, via a helper, asking that subsystem to stop calling into the said
driver (the asking should be done by the driver itself of course, for
ordering reasons).

We have some helpers, but I think not enough, and that's where we should
focus imho. For example, I added fb_set_suspend() so that fbdev's can
request fbcon to stop accessing them (it doesn't solve the problem of
userland mmap's, that will have to be done, if we want to do it, in a
more sneaky way, using VM tricks, but the DRM nowadays has the
infrastructure to do it).

But that's only the "main" path. Aside for that, almost all drivers also
have sideband "request" input and some driver don't actually live behind
a subsystem. That ranges from ioctl, to direct read/write on a char dev
from userland.

I think many of those cases can fairly well deal with just taking a PM
semaphore, that's how I did for a couple of things in the past, provided
that the request path isn't deadlocking with the semaphore held because
of the system suspending of course.  

But in a whole lot of cases, it's, I beleive, perfectly kosher to just
return an error. You're trying to capture frame from your camera while
the machine is suspended ? error. At worst, your capture app will be
unhappy when you wakeup, nothing terrible and totally fixable in
userland if it's a problem.

In some cases, we could use a little bit more help from the subsystem.
Network for example, could have some explicit knowledge of the suspend
state, and in addition to stopping the queue would also stop calling
into things like change_mtu or set_multicast, provided it's agreed that
the driver will account for those changes on resume (the actual MTU
values or multicast lists are still updated in the netdev).

> Normally a waitqueue or a struct completion would be used for this
> purpose.  

I think there's no "normal" scenario, each driver or family of drivers
will do things very differently.

> But either one puts the burden on the driver of defining a
> data structure and signalling it at the right time.

Yes.

> That time is generally when the device is resumed, but there's nothing
> wrong with delaying it slightly, to after all the devices have been resumed (i.e.,
> the time when the current PM code takes everything out of the freezer).

In fact, the best is to have parallel suspend/resume of drivers and asynchronous resume but that's out of topic :-) (For the record, I did some bits like ADB resume like that, that is asynchronous, to speed up wakeup time).
>   
> In fact, we definitely don't want to unblock plug events until this
> later time.

There are two things I believe. There's a generic issue with usermode helpers that make no sense to call between pre-suspend and post-resume, and there's the specific issue of adding/removing devices.

I believe that "bus" drivers such as USB should indeed get a first round of notifications to tell them to stop performing bus plug/unplug operations (it's debatable whether we want to keep unplug going provided we can stack up the usermode events and re-send them later though, but let's say no for the sake of simplicity).

> So instead, why not have the PM core take care of all this?  There
> could be a block_task_until_suspend_is_over() routine available for all
> drivers to use.  Its effect would be exactly the same as sending the
> current task into the freezer, but it wouldn't be the freezer that
> exists now.  It would just be some routine that blocks until the system 
> suspend is over.  We could call it "the icebox" instead of "the 
> freezer".  :-)

I'm not totally sure about that. I like some of it, but I think it's
fairly different conceptually from the freezer (and the implementation
could be as trivial as a single system wide wait queue). 

Basically it has a very big difference to the current freezer, and I
like that, which is that we don't have some 3rd party trying to find out
what to freeze and what not (the freezer), but instead, we have
explicitely drivers or kernel threads sending -themselves- to the
"icebox" when they think it's a good idea. Think of it as lazy freezing
-> you only freeze lazy tasks that are trying to do something that
cannot be done because of suspend.

> Does that make you happier?

I think it's a fairly significant change from the current freezer and I
also think it's a very good idea. The more I think about it, the more I
like it, in the sense that it's a simple drop-in that you could put in a
lot of the ioctl path of drivers to just block tasks that are trying to
call in while suspending, and could be used selectively by things like
the USB hub threads.

> User tasks can cause driver binding by writing to sysfs.  Binding 
> _can't_ be blocked in the driver; by then it's already too late.  If 
> it is going to be blocked at all, it has to be blocked earlier.  One 
> possibility is in the sysfs attribute code; another is to block all 
> sysfs access.
> 
> Of course, another possibility is simply to fail the bind.  But that's 
> not very satisfying, since suspends should be transparent.

I don't think suspend has to be -that- transparent (though there is some
debate on whether it should be if we're gonna do some kind of fast
"light" suspend for things like OLPC) but overall, I agree that a bind
operation on sysfs should probably block until resume, and it does make
sense to have the logic to do that in sysfs itself. It could perfectly
use the above icebox thingy you came up with.

> Ben, you haven't given enough thought to the work needed to avoid 
> locking problems.
> 
> For instance, you agree that during suspend we must not allow device or
> driver registration or unregistration, right? 

Registration is fine, binding/ubinding is not (no problem putting the
driver on/off a list, though of course, you can't unregister a bound
driver without unbinding, it's prefectly find to unregister an unbound
driver).

> And we must not allow driver binding or unbinding. 

That's where the meat is.

> But these events generally involve acquiring a device semaphore, in the
> driver core and quite often in the core's caller.  Since that semaphore
> is also needed for calling the suspend and resume methods, we have to be
> very careful about blocking binding/unbinding/registration/unregistration.

> It has to be done at a time when no device semaphores are held.

I think that should indeed be handled within the driver core. Though I
tend to think our driver core is a bit of a locking mess at the
moment :-) (Who says it could use more RCU-like constructs ?).

Regarding unbinding, that's debatable, it might be perfectly allright to
unbind while suspended, though then, there is the question of a driver
being later on bound to a piece of HW that is suspended (though I
suppose that could happen today... some machines have their firmware
leave some devices off at boot).

As a general matter, If we have those pre-suspend/post-resume notifiers
and we adapt the few (there isn't that much) bus drivers so that they
stop all probe/unprobe operations before the suspend dance starts, we
avoid a lot of that problem. At this point, it becomes fair enough for
bind/unbind, in the core, to return an error (and maybe even stack trace
in dmesg to catch the culprits) while suspend in in progress.

The only remaining annoying case is manual bind/unbind via sysfs which
is a good candidate for the above described icebox.

> You also agree that kernel threads and workqueues must be allowed to 
> operate during suspend.

Yes, unless kernel threads explicitely decide to stop themselves (for
example, khubd is a good candidate for that). Again, not a 3rd party
trying to decide what to freeze and what not, but the drivers or kernel
threads themselves deciding it.
 
> But consider this: By writing the appropriate 
> sysfs attribute, a user task can cause a workqueue item to be queued to 
> keventd that tries to unregister a device.  That really puts you on the 
> spot: Unregistration can't be allowed to fail, it can't be allowed to 
> succeed during a suspend, and keventd can't be blocked!  So what should 
> we do?

We can either stop it at the sysfs write level, or we can have the
workqueue task reschedule itself later until we are resumed. In fact,
worqueue items being what they are (queue items), we could imagine
having a special list where they enqueue themselves to rescheduled after
resume.

Don't get me wrong, I never said we don't need generic infrastructure
and utilities, such as your proposed icebox scheme, or some of those
workqueue bits, helpers in subsystems, etc...

I just think that the freezer approach, as it is, is backward. We can't
have a 3rd party try to discriminate what to freeze and what not, it
will always get something wrong, and in some cases with the wrong timing
or ordering.

Cheers,
Ben.

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Benjamin Herrenschmidt]

> I think it's a fairly significant change from the current freezer and I
> also think it's a very good idea. The more I think about it, the more I
> like it, in the sense that it's a simple drop-in that you could put in a
> lot of the ioctl path of drivers to just block tasks that are trying to
> call in while suspending, and could be used selectively by things like
> the USB hub threads.

Note that we could have also a per-device "icebox" (just a waitqueue).
Might be nice for a device to resume whatever it froze just after it
resumed itself, might even be necessary in case whatever thread got
frozen is necessary for handling child devices.

Cheers,
Ben.

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Rafael J. Wysocki]

On Sunday, 8 July 2007 07:14, Benjamin Herrenschmidt wrote:
[--snip--]
> 
> I just think that the freezer approach, as it is, is backward. We can't
> have a 3rd party try to discriminate what to freeze and what not, it
> will always get something wrong, and in some cases with the wrong timing
> or ordering.

Nice discussion, except for one thing: the freezer doesn't decide what to
freeze.  For example, even right now kernel threads decide if they want to be
frozen.

Greetings,
Rafael


-- 
"Premature optimization is the root of all evil." - Donald Knuth

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Alan Stern]

I'll make this reply short by agreeing up front with most of what you 
say.

On Sun, 8 Jul 2007, Benjamin Herrenschmidt wrote:

> But that's only the "main" path. Aside for that, almost all drivers also
> have sideband "request" input and some driver don't actually live behind
> a subsystem. That ranges from ioctl, to direct read/write on a char dev
> from userland.

Yes, these are the problem cases.

> I think many of those cases can fairly well deal with just taking a PM
> semaphore, that's how I did for a couple of things in the past, provided
> that the request path isn't deadlocking with the semaphore held because
> of the system suspending of course.  

That's what USB does as well (for the drivers which have runtime PM
support -- at the moment only a few of them).

> But in a whole lot of cases, it's, I beleive, perfectly kosher to just
> return an error. You're trying to capture frame from your camera while
> the machine is suspended ? error. At worst, your capture app will be
> unhappy when you wakeup, nothing terrible and totally fixable in
> userland if it's a problem.

We can try falling back on this approach for now.  If the drivers are
smart enough to fail cleanly when the device is already suspended, it
should work.

But I'm not sure it's a good idea in the long run.  Think of a printer 
daemon, for example.  It shouldn't have to experience unexpected I/O 
problems merely because someone has decided to put the system to sleep.

> In some cases, we could use a little bit more help from the subsystem.
> Network for example, could have some explicit knowledge of the suspend
> state, and in addition to stopping the queue would also stop calling
> into things like change_mtu or set_multicast, provided it's agreed that
> the driver will account for those changes on resume (the actual MTU
> values or multicast lists are still updated in the netdev).

This will be up to the people responsible for the subsystems.  I can 
take care of USB.

> There are two things I believe. There's a generic issue with usermode
> helpers that make no sense to call between pre-suspend and
> post-resume, and there's the specific issue of adding/removing
> devices.
> 
> I believe that "bus" drivers such as USB should indeed get a first
> round of notifications to tell them to stop performing bus
> plug/unplug operations (it's debatable whether we want to keep unplug
> going provided we can stack up the usermode events and re-send them
> later though, but let's say no for the sake of simplicity).

Yes.  Rafael, how close is your new notifier chain to mainline?  Can it 
at least be added to Greg KH's development tree so that I can start 
using it?

> > So instead, why not have the PM core take care of all this?  There
> > could be a block_task_until_suspend_is_over() routine available for all
> > drivers to use.  Its effect would be exactly the same as sending the
> > current task into the freezer, but it wouldn't be the freezer that
> > exists now.  It would just be some routine that blocks until the system 
> > suspend is over.  We could call it "the icebox" instead of "the 
> > freezer".  :-)
> 
> I'm not totally sure about that. I like some of it, but I think it's
> fairly different conceptually from the freezer (and the implementation
> could be as trivial as a single system wide wait queue). 

Exactly.

> Basically it has a very big difference to the current freezer, and I
> like that, which is that we don't have some 3rd party trying to find out
> what to freeze and what not (the freezer), but instead, we have
> explicitely drivers or kernel threads sending -themselves- to the
> "icebox" when they think it's a good idea. Think of it as lazy freezing
> -> you only freeze lazy tasks that are trying to do something that
> cannot be done because of suspend.
> 
> > Does that make you happier?
> 
> I think it's a fairly significant change from the current freezer and I
> also think it's a very good idea. The more I think about it, the more I
> like it, in the sense that it's a simple drop-in that you could put in a
> lot of the ioctl path of drivers to just block tasks that are trying to
> call in while suspending, and could be used selectively by things like
> the USB hub threads.

That's what I had in mind.  Rafael, can we add an "icebox" routine?  
Like Ben says, it doesn't need to be much more than a waitqueue
that the current task puts itself on if a suspend is in progress.  
Callers arriving at a time when the icebox isn't activated should
simply return without blocking.  Basically the icebox should be active 
at the same times as the existing freezer.

> > Of course, another possibility is simply to fail the bind.  But that's 
> > not very satisfying, since suspends should be transparent.
> 
> I don't think suspend has to be -that- transparent (though there is some
> debate on whether it should be if we're gonna do some kind of fast
> "light" suspend for things like OLPC) but overall, I agree that a bind
> operation on sysfs should probably block until resume, and it does make
> sense to have the logic to do that in sysfs itself. It could perfectly
> use the above icebox thingy you came up with.

Here's a wacky idea which just might work:

In order to prevent binding and unbinding, while suspending devices all
the PM core has to do is avoid dropping the device semaphores!  It can
release the semaphores as it resumes the devices.

Of course, for this to work it's necessary to avoid changes to the 
device list during the suspend.  However I believe the iteration can be 
made safe against unregistration, so we only have to prevent device 
registration.  (And anyway, it won't be possible to unregister a device 
while the PM core is holding its semaphore.)

If we are willing to be somewhat non-transparent, this is easy to
accomplish.  After the notifier chain has been alerted about the
upcoming suspend, we tell the driver core to disallow adding new
devices.  Maybe use SRCU to synchronize with registration calls that
are in progress.  Thus, until the suspend is over device_add() will
immediately return an error.  We could even add a new ESUSPENDING code
to errno.h; it would come in handy in a few places.

Drivers are already prepared for device registration to fail (or they
ought to be), so this change shouldn't knock the bottom out of things.  
device_add() isn't on a hot path, so adding an extra check and
srcu_read_lock() won't hurt.

> Regarding unbinding, that's debatable, it might be perfectly allright to
> unbind while suspended, though then, there is the question of a driver
> being later on bound to a piece of HW that is suspended (though I
> suppose that could happen today... some machines have their firmware
> leave some devices off at boot).

I have had the same thought, that unbinding and unregistration would be 
easier to handle than binding and registration.  As it happens, holding 
the device semaphore will block both all three -- which makes life 
simpler.

> As a general matter, If we have those pre-suspend/post-resume notifiers
> and we adapt the few (there isn't that much) bus drivers so that they
> stop all probe/unprobe operations before the suspend dance starts, we
> avoid a lot of that problem. At this point, it becomes fair enough for
> bind/unbind, in the core, to return an error (and maybe even stack trace
> in dmesg to catch the culprits) while suspend in in progress.

Putting a WARN_ON() in device_add() would be a good idea.

> > But consider this: By writing the appropriate 
> > sysfs attribute, a user task can cause a workqueue item to be queued to 
> > keventd that tries to unregister a device.  That really puts you on the 
> > spot: Unregistration can't be allowed to fail, it can't be allowed to 
> > succeed during a suspend, and keventd can't be blocked!  So what should 
> > we do?
> 
> We can either stop it at the sysfs write level, or we can have the
> workqueue task reschedule itself later until we are resumed. In fact,
> worqueue items being what they are (queue items), we could imagine
> having a special list where they enqueue themselves to rescheduled after
> resume.

There are other possibilities too.  For example, instead of using
keventd these attributes could use a separate workqueue which would put
itself in the icebox during a suspend.  Or maybe sysfs can be reworked 
so that they don't need to use a workqueue at all.

> Don't get me wrong, I never said we don't need generic infrastructure
> and utilities, such as your proposed icebox scheme, or some of those
> workqueue bits, helpers in subsystems, etc...

I hope that everyone will agree that now is a good time to get started 
on them.  There shouldn't be any problem about having them present 
along with the freezer, and then it will be all that much easier to 
remove the freezer later on if that's what we decide to do.

Alan Stern

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Pavel Machek]

Hi!

> But in a whole lot of cases, it's, I beleive, perfectly kosher to just
> return an error. You're trying to capture frame from your camera while
> the machine is suspended ? error. At worst, your capture app will be
> unhappy when you wakeup, nothing terrible and totally fixable in
> userland if it's a problem.

Well, that way you'd have to teach applications about suspend... Which
is quite bad.  You mentioned it -- returning random errors will be
very bad for machines like OLPC that want to suspend
automatically. Plus it is a step back from current implementation, and
ABI change, too...

> > So instead, why not have the PM core take care of all this?  There
> > could be a block_task_until_suspend_is_over() routine available for all
> > drivers to use.  Its effect would be exactly the same as sending the
> > current task into the freezer, but it wouldn't be the freezer that
> > exists now.  It would just be some routine that blocks until the system 
> > suspend is over.  We could call it "the icebox" instead of "the 
> > freezer".  :-)
> 
> I'm not totally sure about that. I like some of it, but I think it's
> fairly different conceptually from the freezer (and the implementation
> could be as trivial as a single system wide wait queue). 
> 
> Basically it has a very big difference to the current freezer, and I
> like that, which is that we don't have some 3rd party trying to find out
> what to freeze and what not (the freezer), but instead, we have
> explicitely drivers or kernel threads sending -themselves- to the
> "icebox" when they think it's a good idea. Think of it as lazy
> freezing

Kernel threads already send _themselves_ to the refrigerator. [Plus we
put all the userland there, which is what you don't like, but kernel
can not rely on userland after suspend starts, anyway, so it should
not hurt].

Anyway.. PPC currently suspends without freezer, which puts rules on
drivers. ("Must handle i/o requests after .suspend() method is ran,
must not use GFP_KERNEL to do so, must not try to synchronously
communicate with userspace before _all_ devices are unfrozen") I am
not certain what the exact rules are, but you seem to know them. Could
we get Doc*/power/suspend_wo_freezer.txt describing them for driver
authors? That way we can make sure drivers work on ppc, too, and maybe
get rid of freezer in the long run.

> > You also agree that kernel threads and workqueues must be allowed to 
> > operate during suspend.
> 
> Yes, unless kernel threads explicitely decide to stop themselves (for
> example, khubd is a good candidate for that). Again, not a 3rd party
> trying to decide what to freeze and what not, but the drivers or kernel
> threads themselves deciding it.

This is how it works currently in -mm.

(Plus, the rule is that threads that decide _not to_ stop themselves
should not do any I/O.)
									Pavel
-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Benjamin Herrenschmidt]

On Sun, 2007-07-08 at 21:15 +0200, Rafael J. Wysocki wrote:
> On Sunday, 8 July 2007 07:14, Benjamin Herrenschmidt wrote:
> [--snip--]
> > 
> > I just think that the freezer approach, as it is, is backward. We can't
> > have a 3rd party try to discriminate what to freeze and what not, it
> > will always get something wrong, and in some cases with the wrong timing
> > or ordering.
> 
> Nice discussion, except for one thing: the freezer doesn't decide what to
> freeze.  For example, even right now kernel threads decide if they want to be
> frozen.

Somewhat... userspace doesn't and workqueues are a gray area.

Also, I've been thinking this "icebox" idea a bit more and it seems in
fact a bit racy in some areas, at least for use by things like drivers,
unless we end up doing something aking to an RCU on suspend, waiting for
all tasks to reach userland once, but that has the same annoyances as
the current freezer.

Thus I'm tempted to go back to saying that driver can handle things
locally :-)

Ben.

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Pavel Machek]

Hi!

> > > I just think that the freezer approach, as it is, is backward. We can't
> > > have a 3rd party try to discriminate what to freeze and what not, it
> > > will always get something wrong, and in some cases with the wrong timing
> > > or ordering.
> > 
> > Nice discussion, except for one thing: the freezer doesn't decide what to
> > freeze.  For example, even right now kernel threads decide if they want to be
> > frozen.
> 
> Somewhat... userspace doesn't and workqueues are a gray area.

But userspace must not be neccessary for kernel functioning, so that's
quite okay. And we do need to solve the workqueues.

> Also, I've been thinking this "icebox" idea a bit more and it seems in
> fact a bit racy in some areas, at least for use by things like drivers,
> unless we end up doing something aking to an RCU on suspend, waiting for
> all tasks to reach userland once, but that has the same annoyances as
> the current freezer.
> 
> Thus I'm tempted to go back to saying that driver can handle things
> locally :-)

:-). Or perhaps freezer is not _that_ evil after all?
									Pavel
-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Rafael J. Wysocki]

On Sunday, 8 July 2007 23:03, Benjamin Herrenschmidt wrote:
> On Sun, 2007-07-08 at 21:15 +0200, Rafael J. Wysocki wrote:
> > On Sunday, 8 July 2007 07:14, Benjamin Herrenschmidt wrote:
> > [--snip--]
> > > 
> > > I just think that the freezer approach, as it is, is backward. We can't
> > > have a 3rd party try to discriminate what to freeze and what not, it
> > > will always get something wrong, and in some cases with the wrong timing
> > > or ordering.
> > 
> > Nice discussion, except for one thing: the freezer doesn't decide what to
> > freeze.  For example, even right now kernel threads decide if they want to be
> > frozen.
> 
> Somewhat... userspace doesn't and workqueues are a gray area.

Workqueues are kernel threads and the creator decides if they are going to
freeze.  There are only two freezable worqueues in the entire tree right now.

> Also, I've been thinking this "icebox" idea a bit more and it seems in
> fact a bit racy in some areas, at least for use by things like drivers,
> unless we end up doing something aking to an RCU on suspend, waiting for
> all tasks to reach userland once, but that has the same annoyances as
> the current freezer.
> 
> Thus I'm tempted to go back to saying that driver can handle things
> locally :-)

Actaully, I'm perfectly fine with that, as long as each task blocked by the
driver due to suspend has PF_FROZEN (or something similar) set.  Then, at
least theoretically, we'll be able to drop the freezer from the suspend code
path and move it after device_suspend() (or the hibernation-specific
equivalent) for hibernation (in that case there shouldn't be a problem with
any task waiting on I/O while the freezer is running ;-)).

Greetings,
Rafael


-- 
"Premature optimization is the root of all evil." - Donald Knuth

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Benjamin Herrenschmidt]

On Sun, 2007-07-08 at 23:45 +0200, Rafael J. Wysocki wrote:
> 
> Workqueues are kernel threads and the creator decides if they are going to
> freeze.  There are only two freezable worqueues in the entire tree right now.

That and keventd workqueues... my point is you may well end up with
something in a workqueue doing things that should have been blocked
after the freeze.

> Actaully, I'm perfectly fine with that, as long as each task blocked by the
> driver due to suspend has PF_FROZEN (or something similar) set.  Then, at
> least theoretically, we'll be able to drop the freezer from the suspend code
> path and move it after device_suspend() (or the hibernation-specific
> equivalent) for hibernation (in that case there shouldn't be a problem with
> any task waiting on I/O while the freezer is running ;-)).

I don't see the need for a freezer for snapshot but that's a different
issue. (stop_machine looks good enough to me).

Ben

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Pavel Machek]

Hi!

> > There are two things I believe. There's a generic issue with usermode
> > helpers that make no sense to call between pre-suspend and
> > post-resume, and there's the specific issue of adding/removing
> > devices.
> > 
> > I believe that "bus" drivers such as USB should indeed get a first
> > round of notifications to tell them to stop performing bus
> > plug/unplug operations (it's debatable whether we want to keep unplug
> > going provided we can stack up the usermode events and re-send them
> > later though, but let's say no for the sake of simplicity).
> 
> Yes.  Rafael, how close is your new notifier chain to mainline?  Can it 
> at least be added to Greg KH's development tree so that I can start 
> using it?

It should be in -mm, IIRC.

> > I think it's a fairly significant change from the current freezer and I
> > also think it's a very good idea. The more I think about it, the more I
> > like it, in the sense that it's a simple drop-in that you could put in a
> > lot of the ioctl path of drivers to just block tasks that are trying to
> > call in while suspending, and could be used selectively by things like
> > the USB hub threads.
> 
> That's what I had in mind.  Rafael, can we add an "icebox" routine?  
> Like Ben says, it doesn't need to be much more than a waitqueue
> that the current task puts itself on if a suspend is in progress.  
> Callers arriving at a time when the icebox isn't activated should
> simply return without blocking.  Basically the icebox should be active 
> at the same times as the existing freezer.

You could use try_to_freeze(), but you want it to be separate routine
so it can be grepped for... Can you #define icebox_me try_to_freeze
for testing?
								Pavel
-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Benjamin Herrenschmidt]

> But I'm not sure it's a good idea in the long run.  Think of a printer 
> daemon, for example.  It shouldn't have to experience unexpected I/O 
> problems merely because someone has decided to put the system to sleep.

Why not ? Printer is offline when machine is asleep... trying to print
errors out, I don't see the problem there. At one point, we'll need a
cleaner way to also notify userland in which case our daemon could
become more intelligent and stop servicings things before sleep and
resume afterward :-)
 
> This will be up to the people responsible for the subsystems.  I can 
> take care of USB.

USB is not that much of a problem in the sense that for most "leaf"
drivers, USB is a provider (ie, the bus they sit on), not the client
(like the network stack is to network drivers).

In most cases, that "helper" thing would sit on the client subsystem,
since it's the one feeding drivers with requests. The main ones I see at
hand are block, alsa, net, fb/drm... Some of them already have
infrastructure to do it, some my need some more work.

> > I think it's a fairly significant change from the current freezer and I
> > also think it's a very good idea. The more I think about it, the more I
> > like it, in the sense that it's a simple drop-in that you could put in a
> > lot of the ioctl path of drivers to just block tasks that are trying to
> > call in while suspending, and could be used selectively by things like
> > the USB hub threads.
> 
> That's what I had in mind.  Rafael, can we add an "icebox" routine?  
> Like Ben says, it doesn't need to be much more than a waitqueue
> that the current task puts itself on if a suspend is in progress.  
> Callers arriving at a time when the icebox isn't activated should
> simply return without blocking.  Basically the icebox should be active 
> at the same times as the existing freezer.

There is still the race of:

	drivers_sysfs_write()
		try_to_icebox()
		<---- <sleep request gets here>
		hit hardware

Those are akin, in some ways, to the freezer races. Some kind of RCU
might take care of them if we enable the icebox, then wait for all tasks
to hit an explicit schedule point once (or return to userland). That
would mean that drivers need to try_to_icebox() again if they do
something that may schedule (such as __get_user). So it's not a magic
solution, it has issues, but it can handle a lot of the simplest cases. 

> Here's a wacky idea which just might work:
> 
> In order to prevent binding and unbinding, while suspending devices all
> the PM core has to do is avoid dropping the device semaphores!  It can
> release the semaphores as it resumes the devices.
> 
> Of course, for this to work it's necessary to avoid changes to the 
> device list during the suspend.  However I believe the iteration can be 
> made safe against unregistration, so we only have to prevent device 
> registration.  (And anyway, it won't be possible to unregister a device 
> while the PM core is holding its semaphore.)
> 
> If we are willing to be somewhat non-transparent, this is easy to
> accomplish.  After the notifier chain has been alerted about the
> upcoming suspend, we tell the driver core to disallow adding new
> devices.  Maybe use SRCU to synchronize with registration calls that
> are in progress.  Thus, until the suspend is over device_add() will
> immediately return an error.  We could even add a new ESUSPENDING code
> to errno.h; it would come in handy in a few places.
> 
> Drivers are already prepared for device registration to fail (or they
> ought to be), so this change shouldn't knock the bottom out of things.  
> device_add() isn't on a hot path, so adding an extra check and
> srcu_read_lock() won't hurt.

True. Also, bus drivers could just flag the port with something saying
"try registering again later". Don't underestimate the power of "try
later" constructs :-)

> I have had the same thought, that unbinding and unregistration would be 
> easier to handle than binding and registration.  As it happens, holding 
> the device semaphore will block both all three -- which makes life 
> simpler.

Yup. Fair and simple.

> There are other possibilities too.  For example, instead of using
> keventd these attributes could use a separate workqueue which would put
> itself in the icebox during a suspend.  Or maybe sysfs can be reworked 
> so that they don't need to use a workqueue at all.

I think having a facility for a given workqueue entry to requeue itself
for after resume might be of use for drivers too.

> > Don't get me wrong, I never said we don't need generic infrastructure
> > and utilities, such as your proposed icebox scheme, or some of those
> > workqueue bits, helpers in subsystems, etc...
> 
> I hope that everyone will agree that now is a good time to get started 
> on them.  There shouldn't be any problem about having them present 
> along with the freezer, and then it will be all that much easier to 
> remove the freezer later on if that's what we decide to do.

Ben.

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Rafael J. Wysocki]

On Sunday, 8 July 2007 22:16, Alan Stern wrote:
> I'll make this reply short by agreeing up front with most of what you 
> say.
> 
> On Sun, 8 Jul 2007, Benjamin Herrenschmidt wrote:
> 
> > But that's only the "main" path. Aside for that, almost all drivers also
> > have sideband "request" input and some driver don't actually live behind
> > a subsystem. That ranges from ioctl, to direct read/write on a char dev
> > from userland.
> 
> Yes, these are the problem cases.
> 
> > I think many of those cases can fairly well deal with just taking a PM
> > semaphore, that's how I did for a couple of things in the past, provided
> > that the request path isn't deadlocking with the semaphore held because
> > of the system suspending of course.  
> 
> That's what USB does as well (for the drivers which have runtime PM
> support -- at the moment only a few of them).
> 
> > But in a whole lot of cases, it's, I beleive, perfectly kosher to just
> > return an error. You're trying to capture frame from your camera while
> > the machine is suspended ? error. At worst, your capture app will be
> > unhappy when you wakeup, nothing terrible and totally fixable in
> > userland if it's a problem.
> 
> We can try falling back on this approach for now.  If the drivers are
> smart enough to fail cleanly when the device is already suspended, it
> should work.

And are they?

> But I'm not sure it's a good idea in the long run.  Think of a printer 
> daemon, for example.  It shouldn't have to experience unexpected I/O 
> problems merely because someone has decided to put the system to sleep.

Agreed.

> > In some cases, we could use a little bit more help from the subsystem.
> > Network for example, could have some explicit knowledge of the suspend
> > state, and in addition to stopping the queue would also stop calling
> > into things like change_mtu or set_multicast, provided it's agreed that
> > the driver will account for those changes on resume (the actual MTU
> > values or multicast lists are still updated in the netdev).
> 
> This will be up to the people responsible for the subsystems.  I can 
> take care of USB.
> 
> > There are two things I believe. There's a generic issue with usermode
> > helpers that make no sense to call between pre-suspend and
> > post-resume, and there's the specific issue of adding/removing
> > devices.
> > 
> > I believe that "bus" drivers such as USB should indeed get a first
> > round of notifications to tell them to stop performing bus
> > plug/unplug operations (it's debatable whether we want to keep unplug
> > going provided we can stack up the usermode events and re-send them
> > later though, but let's say no for the sake of simplicity).
> 
> Yes.  Rafael, how close is your new notifier chain to mainline?  Can it 
> at least be added to Greg KH's development tree so that I can start 
> using it?

It's in -mm.  The patches queued up in -mm are also in my patchset at
http://www.sisk.pl/kernel/hibernation_and_suspend/2.6.22-rc7/patches/

> > > So instead, why not have the PM core take care of all this?  There
> > > could be a block_task_until_suspend_is_over() routine available for all
> > > drivers to use.  Its effect would be exactly the same as sending the
> > > current task into the freezer, but it wouldn't be the freezer that
> > > exists now.  It would just be some routine that blocks until the system 
> > > suspend is over.  We could call it "the icebox" instead of "the 
> > > freezer".  :-)
> > 
> > I'm not totally sure about that. I like some of it, but I think it's
> > fairly different conceptually from the freezer (and the implementation
> > could be as trivial as a single system wide wait queue). 
> 
> Exactly.
> 
> > Basically it has a very big difference to the current freezer, and I
> > like that, which is that we don't have some 3rd party trying to find out
> > what to freeze and what not (the freezer), but instead, we have
> > explicitely drivers or kernel threads sending -themselves- to the
> > "icebox" when they think it's a good idea. Think of it as lazy freezing
> > -> you only freeze lazy tasks that are trying to do something that
> > cannot be done because of suspend.
> > 
> > > Does that make you happier?
> > 
> > I think it's a fairly significant change from the current freezer and I
> > also think it's a very good idea. The more I think about it, the more I
> > like it, in the sense that it's a simple drop-in that you could put in a
> > lot of the ioctl path of drivers to just block tasks that are trying to
> > call in while suspending, and could be used selectively by things like
> > the USB hub threads.
> 
> That's what I had in mind.  Rafael, can we add an "icebox" routine?  

Yes, I think so.

> Like Ben says, it doesn't need to be much more than a waitqueue
> that the current task puts itself on if a suspend is in progress.  
> Callers arriving at a time when the icebox isn't activated should
> simply return without blocking.  Basically the icebox should be active 
> at the same times as the existing freezer.

Hmm, I think this would be functionally equivalent to a modified freezer
with the following changes:

(1) the actual freezing functionality is not hooked up to the signal code,
but instead there are many invocations of try_to_freeze() all over the place,

(2) the freezer doesn't wait for tasks to freeze, only sets their TIF_FREEZE
flags and exits, where TIF_FREEZE now means "there's a suspend in
progress, please freeze yourself if need be".

BTW, this is similar to what I'd like to have for kernel threads RSN, except
for the "doesn't wait" part.

IMO, something like this might be used for hibernation too, but in that case
we need to do something about the tasks that haven't gone to the icebox.

I have some implementation-related questions:
* do we want each task to be notified individually or do we want a global
  variable, say "pm_entering_sleep_state", that will mean "now you're supposed
  to go to the icebox"
* at what point exactly this should be activated (ie. before the freezer or
  after it)
* how do we release tasks from the icebox (all at a time or individually, and
  if the latter then how exactly).

> > > Of course, another possibility is simply to fail the bind.  But that's 
> > > not very satisfying, since suspends should be transparent.
> > 
> > I don't think suspend has to be -that- transparent (though there is some
> > debate on whether it should be if we're gonna do some kind of fast
> > "light" suspend for things like OLPC) but overall, I agree that a bind
> > operation on sysfs should probably block until resume, and it does make
> > sense to have the logic to do that in sysfs itself. It could perfectly
> > use the above icebox thingy you came up with.
> 
> Here's a wacky idea which just might work:
> 
> In order to prevent binding and unbinding, while suspending devices all
> the PM core has to do is avoid dropping the device semaphores!  It can
> release the semaphores as it resumes the devices.
> 
> Of course, for this to work it's necessary to avoid changes to the 
> device list during the suspend.  However I believe the iteration can be 
> made safe against unregistration, so we only have to prevent device 
> registration.  (And anyway, it won't be possible to unregister a device 
> while the PM core is holding its semaphore.)
> 
> If we are willing to be somewhat non-transparent, this is easy to
> accomplish.  After the notifier chain has been alerted about the
> upcoming suspend, we tell the driver core to disallow adding new
> devices.  Maybe use SRCU to synchronize with registration calls that
> are in progress.  Thus, until the suspend is over device_add() will
> immediately return an error.  We could even add a new ESUSPENDING code
> to errno.h; it would come in handy in a few places.

This last thing is a good idea, IMHO.

> Drivers are already prepared for device registration to fail (or they
> ought to be), so this change shouldn't knock the bottom out of things.  
> device_add() isn't on a hot path, so adding an extra check and
> srcu_read_lock() won't hurt.
> 
> > Regarding unbinding, that's debatable, it might be perfectly allright to
> > unbind while suspended, though then, there is the question of a driver
> > being later on bound to a piece of HW that is suspended (though I
> > suppose that could happen today... some machines have their firmware
> > leave some devices off at boot).
> 
> I have had the same thought, that unbinding and unregistration would be 
> easier to handle than binding and registration.  As it happens, holding 
> the device semaphore will block both all three -- which makes life 
> simpler.

I think we should do something like that in the beginning and probably try to
add some flexibility later on.

> > As a general matter, If we have those pre-suspend/post-resume notifiers
> > and we adapt the few (there isn't that much) bus drivers so that they
> > stop all probe/unprobe operations before the suspend dance starts, we
> > avoid a lot of that problem. At this point, it becomes fair enough for
> > bind/unbind, in the core, to return an error (and maybe even stack trace
> > in dmesg to catch the culprits) while suspend in in progress.
> 
> Putting a WARN_ON() in device_add() would be a good idea.
> 
> > > But consider this: By writing the appropriate 
> > > sysfs attribute, a user task can cause a workqueue item to be queued to 
> > > keventd that tries to unregister a device.  That really puts you on the 
> > > spot: Unregistration can't be allowed to fail, it can't be allowed to 
> > > succeed during a suspend, and keventd can't be blocked!  So what should 
> > > we do?
> > 
> > We can either stop it at the sysfs write level, or we can have the
> > workqueue task reschedule itself later until we are resumed. In fact,
> > worqueue items being what they are (queue items), we could imagine
> > having a special list where they enqueue themselves to rescheduled after
> > resume.
> 
> There are other possibilities too.  For example, instead of using
> keventd these attributes could use a separate workqueue which would put
> itself in the icebox during a suspend.  Or maybe sysfs can be reworked 
> so that they don't need to use a workqueue at all.
> 
> > Don't get me wrong, I never said we don't need generic infrastructure
> > and utilities, such as your proposed icebox scheme, or some of those
> > workqueue bits, helpers in subsystems, etc...
> 
> I hope that everyone will agree that now is a good time to get started 
> on them.  There shouldn't be any problem about having them present 
> along with the freezer, and then it will be all that much easier to 
> remove the freezer later on if that's what we decide to do.

Agreed.

Greetings,
Rafael


-- 
"Premature optimization is the root of all evil." - Donald Knuth

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Rafael J. Wysocki]

On Sunday, 8 July 2007 23:20, Benjamin Herrenschmidt wrote:
> 
> > But I'm not sure it's a good idea in the long run.  Think of a printer 
> > daemon, for example.  It shouldn't have to experience unexpected I/O 
> > problems merely because someone has decided to put the system to sleep.
> 
> Why not ? Printer is offline when machine is asleep... trying to print
> errors out, I don't see the problem there. At one point, we'll need a
> cleaner way to also notify userland in which case our daemon could
> become more intelligent and stop servicings things before sleep and
> resume afterward :-)
>  
> > This will be up to the people responsible for the subsystems.  I can 
> > take care of USB.
> 
> USB is not that much of a problem in the sense that for most "leaf"
> drivers, USB is a provider (ie, the bus they sit on), not the client
> (like the network stack is to network drivers).
> 
> In most cases, that "helper" thing would sit on the client subsystem,
> since it's the one feeding drivers with requests. The main ones I see at
> hand are block, alsa, net, fb/drm... Some of them already have
> infrastructure to do it, some my need some more work.
> 
> > > I think it's a fairly significant change from the current freezer and I
> > > also think it's a very good idea. The more I think about it, the more I
> > > like it, in the sense that it's a simple drop-in that you could put in a
> > > lot of the ioctl path of drivers to just block tasks that are trying to
> > > call in while suspending, and could be used selectively by things like
> > > the USB hub threads.
> > 
> > That's what I had in mind.  Rafael, can we add an "icebox" routine?  
> > Like Ben says, it doesn't need to be much more than a waitqueue
> > that the current task puts itself on if a suspend is in progress.  
> > Callers arriving at a time when the icebox isn't activated should
> > simply return without blocking.  Basically the icebox should be active 
> > at the same times as the existing freezer.
> 
> There is still the race of:
> 
> 	drivers_sysfs_write()
> 		try_to_icebox()
> 		<---- <sleep request gets here>
> 		hit hardware
> 
> Those are akin, in some ways, to the freezer races.

I'm not sure what races you're referring to, but never mind. :-)

This is the reason why the freezer waits for tasks to freeze, actually.

> Some kind of RCU might take care of them if we enable the icebox,
> then wait for all tasks to hit an explicit schedule point once (or return to
> userland).

This is what the freezer does, isn't it? ;-)

> That would mean that drivers need to try_to_icebox() again if they do
> something that may schedule (such as __get_user). So it's not a magic
> solution, it has issues, but it can handle a lot of the simplest cases. 

Well, can't we do:

 	drivers_sysfs_write()
		while(!suspend_trylock())
			try_to_icebox() --> or even try_to_freeze(), what's the difference?

 		hit hardware
		unlock_suspend()

where the PM core must wait for the suspend lock to get released (say with a
timeout)?
 
> > Here's a wacky idea which just might work:
> > 
> > In order to prevent binding and unbinding, while suspending devices all
> > the PM core has to do is avoid dropping the device semaphores!  It can
> > release the semaphores as it resumes the devices.
> > 
> > Of course, for this to work it's necessary to avoid changes to the 
> > device list during the suspend.  However I believe the iteration can be 
> > made safe against unregistration, so we only have to prevent device 
> > registration.  (And anyway, it won't be possible to unregister a device 
> > while the PM core is holding its semaphore.)
> > 
> > If we are willing to be somewhat non-transparent, this is easy to
> > accomplish.  After the notifier chain has been alerted about the
> > upcoming suspend, we tell the driver core to disallow adding new
> > devices.  Maybe use SRCU to synchronize with registration calls that
> > are in progress.  Thus, until the suspend is over device_add() will
> > immediately return an error.  We could even add a new ESUSPENDING code
> > to errno.h; it would come in handy in a few places.
> > 
> > Drivers are already prepared for device registration to fail (or they
> > ought to be), so this change shouldn't knock the bottom out of things.  
> > device_add() isn't on a hot path, so adding an extra check and
> > srcu_read_lock() won't hurt.
> 
> True. Also, bus drivers could just flag the port with something saying
> "try registering again later". Don't underestimate the power of "try
> later" constructs :-)
> 
> > I have had the same thought, that unbinding and unregistration would be 
> > easier to handle than binding and registration.  As it happens, holding 
> > the device semaphore will block both all three -- which makes life 
> > simpler.
> 
> Yup. Fair and simple.
> 
> > There are other possibilities too.  For example, instead of using
> > keventd these attributes could use a separate workqueue which would put
> > itself in the icebox during a suspend.  Or maybe sysfs can be reworked 
> > so that they don't need to use a workqueue at all.
> 
> I think having a facility for a given workqueue entry to requeue itself
> for after resume might be of use for drivers too.

Hmm, perhaps we can use a special workqueue that's created before the suspend
(say by the PM core) and starts processing jobs after the resume?

Greetings,
Rafael


-- 
"Premature optimization is the root of all evil." - Donald Knuth

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Benjamin Herrenschmidt]

>  	drivers_sysfs_write()
> 		while(!suspend_trylock())
> 			try_to_icebox() --> or even try_to_freeze(), what's the difference?
> 
>  		hit hardware
> 		unlock_suspend()
> 
> where the PM core must wait for the suspend lock to get released (say with a
> timeout)?

Somewhat. What I'd like is to have that a construct of that sort on a
per-driver basis though :-) Now the question is, in what way would the
above be different from a simple suspend mutex ? I've been using mutexes
in the past for a couple of drivers (iirc, that's how I did it for
dmasound_pmac, though that driver is long past obsolescence now).

But yes, that's the idea, something under control of drivers, instead of
a 3rd party freezer that tries to hit at processes. 

> Hmm, perhaps we can use a special workqueue that's created before the suspend
> (say by the PM core) and starts processing jobs after the resume?

Yeah, whatever the implementation is, though, the interface should be
transparent. But then, I see little use of that. Mostly things like this
sysfs write to unbind Alan was talking about. Driver workqueues probably
need to be functioning all the way until the driver itself is suspended,
in which case, the driver may just flush_worqueues in it's suspend path
at the right time to make sure it doesn't exit with something still
pending.

Ben.

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Kyle Moffett]

On Jul 08, 2007, at 20:33:37, Benjamin Herrenschmidt wrote:
>>  	drivers_sysfs_write()
>> 		while(!suspend_trylock())
>> 			try_to_icebox() --> or even try_to_freeze(), what's the  
>> difference?
>>
>>  		hit hardware
>> 		unlock_suspend()
>>
>> where the PM core must wait for the suspend lock to get released  
>> (say with a timeout)?
>
> Somewhat. What I'd like is to have that a construct of that sort on  
> a per-driver basis though :-) Now the question is, in what way  
> would the above be different from a simple suspend mutex ? I've  
> been using mutexes in the past for a couple of drivers (iirc,  
> that's how I did it for dmasound_pmac, though that driver is long  
> past obsolescence now).

I agree completely.  It's a bit trickier if you want to do work in  
uninterruptable contexts:

driver_suspend_callback(...)
{
	dev_suspend_lock(dev);
	put_hardware_to_sleep(dev);
}

driver_resume_callback(...)
{
	wake_hardware_up(dev);
	dev_suspend_unlock(dev);
}

Then for sleep-capable contexts:
	dev_suspend_lock(dev);
	dev_suspend_unlock(dev);

And for no-sleep contexts like interrupts etc:
	if (!dev_suspend_trylock(dev))
		return postpone_work_for_later(dev, ...);
	do_stuff_with(dev);
	dev_suspend_unlock(dev);

You could do this with a straight mutex except for the  
dev_suspend_trylock/unlock bit in uninterruptable contexts, but I  
seem to recall somebody saying that could be made to work if there  
was a real need for it.  Alternatively you could just drive the  
"Generic Mutex" guys up the wall by inventing your own pseudo-mutex  
with a spinlock, a boolean value, and a waitqueue.

Then when you put your driver to sleep, things trying to do IO will  
automatically "freeze" themselves exactly until the device is woken  
again.

Assuming the driver model and subsystem get the ordering right for  
which devices to suspend/resume, then it's impossible to deadlock or  
cause hardware confusion.  And even further, if you manage to make  
the automagic mutex-debugging code work with the noninterruptable  
trylock it will yell at you when the driver model does nasty deadlock- 
y things.

On the other hand, if the driver model *doesn't* get the ordering  
right then it's fundamentally impossible to reliably suspend and resume.

Cheers,
Kyle Moffett

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Benjamin Herrenschmidt]

> You could do this with a straight mutex except for the  
> dev_suspend_trylock/unlock bit in uninterruptable contexts, but I  
> seem to recall somebody saying that could be made to work if there  
> was a real need for it.  Alternatively you could just drive the  
> "Generic Mutex" guys up the wall by inventing your own pseudo-mutex  
> with a spinlock, a boolean value, and a waitqueue.

Most of the time, the interrupt/atomic contexts are part of what I call
the "main path" of the driver, which doesn't need most of this stuff.

Depending on what is "upstream" from the driver, you typically set a
condition stopping the flow of "interrupt" activity and wait for it to
drain in suspend. For example, a block driver would instruct the driver
to stop processing the request queue and wait for pending in flight
requests to drain, or a network driver would stop the tx queue and wait
for pending/concurrent xmit() callbacks to complete (basically sync. bh
and sync. with NAPI poll).

In both examples, helpers can be provided in the block layer or network
stack to make that easier.

In many cases, the driver may just turn off something in the chip in
"suspend" (using a low level spinlock that it already has if it has an
irq path to sync. HW access) and set some flag/state to instruct the irq
handler to not restart, then synchronize_irq().

All those techniques are fairly simple and well known. The main source
of problems are the "sideband" channels, such as sysfs accesses to
driver tunables, ioctl's, etc... for which I believe mutexes can solve
90% of them.

Ben.

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Oliver Neukum]

Am Sonntag, 8. Juli 2007 schrieb Benjamin Herrenschmidt:
> > But I'm not sure it's a good idea in the long run.  Think of a printer 
> > daemon, for example.  It shouldn't have to experience unexpected I/O 
> > problems merely because someone has decided to put the system to sleep.
> 
> Why not ? Printer is offline when machine is asleep... trying to print

Not necessarily. The machine must survive going to sleep while you are
printing. Any other error return than -ERESTARTSYS is not an option.
We can't simply change the ABI.

	Regards
		Oliver

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Benjamin Herrenschmidt]

On Mon, 2007-07-09 at 08:47 +0200, Oliver Neukum wrote:
> Am Sonntag, 8. Juli 2007 schrieb Benjamin Herrenschmidt:
> > > But I'm not sure it's a good idea in the long run.  Think of a printer 
> > > daemon, for example.  It shouldn't have to experience unexpected I/O 
> > > problems merely because someone has decided to put the system to sleep.
> > 
> > Why not ? Printer is offline when machine is asleep... trying to print
> 
> Not necessarily. The machine must survive going to sleep while you are
> printing. Any other error return than -ERESTARTSYS is not an option.
> We can't simply change the ABI.

Ugh ? Why returning an error from the printer driver to the userland
print server/daemon would prevent the machine from "surviving" ? I would
be happy with -EIO personally :-)

Ben.

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Oliver Neukum]

Am Montag, 9. Juli 2007 schrieb Benjamin Herrenschmidt:
> On Mon, 2007-07-09 at 08:47 +0200, Oliver Neukum wrote:
> > Am Sonntag, 8. Juli 2007 schrieb Benjamin Herrenschmidt:
> > > > But I'm not sure it's a good idea in the long run.  Think of a printer 
> > > > daemon, for example.  It shouldn't have to experience unexpected I/O 
> > > > problems merely because someone has decided to put the system to sleep.
> > > 
> > > Why not ? Printer is offline when machine is asleep... trying to print
> > 
> > Not necessarily. The machine must survive going to sleep while you are
> > printing. Any other error return than -ERESTARTSYS is not an option.
> > We can't simply change the ABI.
> 
> Ugh ? Why returning an error from the printer driver to the userland
> print server/daemon would prevent the machine from "surviving" ? I would
> be happy with -EIO personally :-)

Surviving is a bit strongly worded.

Suspension is to be transparent. Apart from a jump in the system clock
user space must not notice, thus returning errors due to suspension is
not an option.

	Regards
		Oliver

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Benjamin Herrenschmidt]

On Mon, 2007-07-09 at 11:23 +0200, Oliver Neukum wrote:
> Suspension is to be transparent. Apart from a jump in the system clock
> user space must not notice, thus returning errors due to suspension is
> not an option.

Who says ?

If I'm talking to a USB printer, it will notice the USB bus being
suspended, believe me. It's actually likely to screw up whatever job is
in progress. It make perfect sense to error out attempts to talk to it
when suspended.

I don't think "transparent" is an absolute requirement. In some cases,
it doesn't make sense and the printer is a good example of that. The
printing daemon is typically something that will ideally need to grow
knowledge about suspend/resume to interrupt things at the right time.

That's one of the thing... we don't even have the kernel right, so we
are light-years from having the rest of userland right.

For example, from a user point of view, what should happen if you
suspend while printing ?

Well, it's a matter of policy (thus should be configurable), but I would
exect (as a former Mac user) something around the lines of this as a
default:

 - If the laptop's clamshell has been closed, it's likely that the
user's just picking up the laptop for a ride, makes sense to abort jobs
in progress.

 - If this is a manual action via the menus, displaying a dialog asking
if you really want to suspend before your page is finished makes sense
(with that dialog disappearing and the machine going to sleep if you
just wait and let it print).

That sort of thing requires much more than just kernel collaboration of
course, but a whole infrastructure in userland as well.

So no, suspend is -not- transparent, doesn't have to be, it's something
that is to be decided on a case-by-case basis (or rather,
driver-per-driver and possibly via policy settings).

Ben.

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Adrian Bunk]

On Mon, Jul 09, 2007 at 07:33:56PM +1000, Benjamin Herrenschmidt wrote:
> On Mon, 2007-07-09 at 11:23 +0200, Oliver Neukum wrote:
> > Suspension is to be transparent. Apart from a jump in the system clock
> > user space must not notice, thus returning errors due to suspension is
> > not an option.
> 
> Who says ?
> 
> If I'm talking to a USB printer, it will notice the USB bus being
> suspended, believe me. It's actually likely to screw up whatever job is
> in progress. It make perfect sense to error out attempts to talk to it
> when suspended.
> 
> I don't think "transparent" is an absolute requirement. In some cases,
> it doesn't make sense and the printer is a good example of that. The
> printing daemon is typically something that will ideally need to grow
> knowledge about suspend/resume to interrupt things at the right time.
> 
> That's one of the thing... we don't even have the kernel right, so we
> are light-years from having the rest of userland right.
> 
> For example, from a user point of view, what should happen if you
> suspend while printing ?

There might be other existing things with a semantics that might be 
worth looking at, e.g.:

I don't have an USB printer, but what happens if you unplug the printer 
while printing, wait 30 seconds, and plug it in again?
Wait 3 hours?

Could this work theoretically?
Does it work in practice?

This are non-theoretical use cases kernel and userspace already somehow 
handle today, and for userspace an USB printer disconnected for 3 hours 
and a 3 hour suspend might simply be the same with the latter not 
requiring any extra userspace handling.

Another example might be mounted NFS file systems:

The case that a client's ethernet cable gets unplugged for 24 hours and 
plugged in again is a real life use case NFS and all applications 
already have to handle.

> Well, it's a matter of policy (thus should be configurable), but I would
> exect (as a former Mac user) something around the lines of this as a
> default:
> 
>  - If the laptop's clamshell has been closed, it's likely that the
> user's just picking up the laptop for a ride, makes sense to abort jobs
> in progress.
> 
>  - If this is a manual action via the menus, displaying a dialog asking
> if you really want to suspend before your page is finished makes sense
> (with that dialog disappearing and the machine going to sleep if you
> just wait and let it print).
> 
> That sort of thing requires much more than just kernel collaboration of
> course, but a whole infrastructure in userland as well.
> 
> So no, suspend is -not- transparent, doesn't have to be, it's something
> that is to be decided on a case-by-case basis (or rather,
> driver-per-driver and possibly via policy settings).

But keep in mind that every piece that does not work transparently 
might result in less use cases for suspend.

There are cases like CD burning while suspending that might not be 
solvable, but as much as possible should work as transparently as 
possible.

> Ben.

cu
Adrian

-- 

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Pavel Machek]

Hi!

> > > But I'm not sure it's a good idea in the long run.  Think of a printer 
> > > daemon, for example.  It shouldn't have to experience unexpected I/O 
> > > problems merely because someone has decided to put the system to sleep.
> > 
> > Why not ? Printer is offline when machine is asleep... trying to print

...filesystems are offline, too, when the machine is asleep. Yet,
unmounting everything on suspend would not result in useful suspend
support.

Yes, I believe we should be transparent.
								Pavel
-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Benjamin Herrenschmidt]

On Mon, 2007-07-09 at 12:02 +0200, Pavel Machek wrote:
> Hi!
> 
> > > > But I'm not sure it's a good idea in the long run.  Think of a printer 
> > > > daemon, for example.  It shouldn't have to experience unexpected I/O 
> > > > problems merely because someone has decided to put the system to sleep.
> > > 
> > > Why not ? Printer is offline when machine is asleep... trying to print
> 
> ...filesystems are offline, too, when the machine is asleep. Yet,
> unmounting everything on suspend would not result in useful suspend
> support.
> 
> Yes, I believe we should be transparent.

You just compared apple and oranges... Try printing and half way through
the page, suspend your USB bus, and see how the printer reacts.

Ben.

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Alan Stern]

On Mon, 9 Jul 2007, Benjamin Herrenschmidt wrote:

> > This will be up to the people responsible for the subsystems.  I can 
> > take care of USB.
> 
> USB is not that much of a problem in the sense that for most "leaf"
> drivers, USB is a provider (ie, the bus they sit on), not the client
> (like the network stack is to network drivers).

That's why I thought I would be able to handle it!  :-)

> There is still the race of:
> 
> 	drivers_sysfs_write()
> 		try_to_icebox()
> 		<---- <sleep request gets here>
> 		hit hardware
> 
> Those are akin, in some ways, to the freezer races. Some kind of RCU
> might take care of them if we enable the icebox, then wait for all tasks
> to hit an explicit schedule point once (or return to userland). That
> would mean that drivers need to try_to_icebox() again if they do
> something that may schedule (such as __get_user). So it's not a magic
> solution, it has issues, but it can handle a lot of the simplest cases. 

It depends on what kind of race you're worried about.  In general a 
sysfs access that causes I/O isn't much different from a char device's 
read or write.  For such cases the driver would have to do something 
like this:

	drivers_sysfs_write()
	{
	 restart:
		mutex_lock(private_io_mutex);
		if (device is suspended) {
			mutex_unlock(private_io_mutex);
			icebox();
			goto restart;
		}
		... hit hardware ...
		mutex_unlock(private_io_mutex);
	}

And of course the suspend method would have to acquire the
private_io_mutex.  Some drivers might be able to use the device
semaphore instead of adding a new mutex.

I'm more concerned about races involving device registration.  The best 
solution I can come up with is to use an rwsem:

	drivers_sysfs_write()
	{
		down_read(private_rwsem);
		... register/unregister devices ...
		up_read(private_rwsem);
	}

where the suspend notifer callout would do down_write() and the resume
callout would do up_write().  Can you suggest anything better?

> > Drivers are already prepared for device registration to fail (or they
> > ought to be), so this change shouldn't knock the bottom out of things.  
> > device_add() isn't on a hot path, so adding an extra check and
> > srcu_read_lock() won't hurt.
> 
> True. Also, bus drivers could just flag the port with something saying
> "try registering again later". Don't underestimate the power of "try
> later" constructs :-)

-ESUSPENDING would naturally imply that the caller should retry after 
the resume.

Alan Stern

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Alan Stern]

On Sun, 8 Jul 2007, Benjamin Herrenschmidt wrote:

> > (And rather than trying to manage a waitqueue or struct completion, it
> > would be easiest to jump directly into the freezer!  The driver or the 
> > core wouldn't have to worry about waking up all these blocked threads.)
> 
> That's wrong. The freezer is NOT a solution for that sort of thing. Just
> because you guys can't get your locking right.

You misunderstood.  Stop trying to incite riot, calm down, and pay
attention to what I actually wrote.  I'll explain it again in more
explicit terms:

You agree that drivers need to block various activities during suspend.  
Principally I/O requests, but other things as well.  So when one of 
these requests arrives, the driver has to make it wait somehow and then 
has to allow it to proceed at the appropriate time.

Normally a waitqueue or a struct completion would be used for this
purpose.  But either one puts the burden on the driver of defining a
data structure and signalling it at the right time.  That time is
generally when the device is resumed, but there's nothing wrong with
delaying it slightly, to after all the devices have been resumed (i.e.,
the time when the current PM code takes everything out of the freezer).  
In fact, we definitely don't want to unblock plug events until this
later time.

So instead, why not have the PM core take care of all this?  There
could be a block_task_until_suspend_is_over() routine available for all
drivers to use.  Its effect would be exactly the same as sending the
current task into the freezer, but it wouldn't be the freezer that
exists now.  It would just be some routine that blocks until the system 
suspend is over.  We could call it "the icebox" instead of "the 
freezer".  :-)

Does that make you happier?


> > > Face it, we should seriously look into doing suspend/resume without a
> > > freezer.
> > 
> > I'm willing to try, although I think it will be a tremendous amount of 
> > work to verify that every driver does the right thing.  There's lots of 
> > support missing.  For example, don't you think we should block all 
> > sysfs I/O during suspend?  And likewise for insmod/rmmod?
> 
> sysfs is a matter of driver. If a sysfs read/write callback in a driver
> is hitting the HW, it most certainly already has some kind of locking.
> That locking can/should be extended to deal with blocking when the HW is
> suspended.

User tasks can cause driver binding by writing to sysfs.  Binding 
_can't_ be blocked in the driver; by then it's already too late.  If 
it is going to be blocked at all, it has to be blocked earlier.  One 
possibility is in the sysfs attribute code; another is to block all 
sysfs access.

Of course, another possibility is simply to fail the bind.  But that's 
not very satisfying, since suspends should be transparent.

> However, since it seems that people universally consider it very hard to
> get right (I don't but heh), Linus and Paul have come up with a solution
> for most simple enough directly-mapped drivers such as PCI (ok, that
> doesn't include USB) which is to simply do the HW suspend in a late
> callback after IRQs are off, and not bother with the rest.

Ben, you haven't given enough thought to the work needed to avoid 
locking problems.

For instance, you agree that during suspend we must not allow device or
driver registration or unregistration, right?  And we must not allow
driver binding or unbinding.  But these events generally involve
acquiring a device semaphore, in the driver core and quite often in the
core's caller.  Since that semaphore is also needed for calling the
suspend and resume methods, we have to be very careful about blocking
binding/unbinding/registration/unregistration.  It has to be done at a 
time when no device semaphores are held.

You also agree that kernel threads and workqueues must be allowed to 
operate during suspend.  But consider this: By writing the appropriate 
sysfs attribute, a user task can cause a workqueue item to be queued to 
keventd that tries to unregister a device.  That really puts you on the 
spot: Unregistration can't be allowed to fail, it can't be allowed to 
succeed during a suspend, and keventd can't be blocked!  So what should 
we do?

Alan Stern

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Benjamin Herrenschmidt]

> Spinning in the driver with the lock not held is impossible, since the 
> driver is called with the lock already acquired.
> 
> Failing with -ERETRY is non-transparent.  I would prefer to block such 
> requests at their source, before the lock is acquired.  Perhaps in the 
> driver core, perhaps even earlier.
> 
> (And rather than trying to manage a waitqueue or struct completion, it
> would be easiest to jump directly into the freezer!  The driver or the 
> core wouldn't have to worry about waking up all these blocked threads.)

That's wrong. The freezer is NOT a solution for that sort of thing. Just
because you guys can't get your locking right.

> You're missing the point.  If the driver and the freezer are both 
> solid, there's no reason they can't share the work.  If many drivers 
> can pass off part of their workload to the single freezer, it's a net 
> win.

I've explained already multiple times that the freezer will not do what
you guys expect it to do. IOs can be submited at non-task time and there
is no clear distinction between IO generating threads that must and
those that must not be frozen.

I really can't understand why you guys work so hard at trying to avoid
the right solutions systematically.

> So it isn't a question of how solid the drivers are; it's a question 
> of how solid the freezer is.  And bear in mind that if you convince 
> people the freezer is not solid enough to be used, then you will have 
> to find an alternative for purposes of hibernation.

Because I'm intimately convinced that the freezer is a wrong approach
that cannot be made solid enough.

> > The freezer is a flawed concept in the first place.
> 
> <... Long and cogent argument which I will skip over for now ...>

Too bad, that's where the interesting points that show that the freezer
cannot work are..

> > Face it, we should seriously look into doing suspend/resume without a
> > freezer.
> 
> I'm willing to try, although I think it will be a tremendous amount of 
> work to verify that every driver does the right thing.  There's lots of 
> support missing.  For example, don't you think we should block all 
> sysfs I/O during suspend?  And likewise for insmod/rmmod?

sysfs is a matter of driver. If a sysfs read/write callback in a driver
is hitting the HW, it most certainly already has some kind of locking.
That locking can/should be extended to deal with blocking when the HW is
suspended.

However, since it seems that people universally consider it very hard to
get right (I don't but heh), Linus and Paul have come up with a solution
for most simple enough directly-mapped drivers such as PCI (ok, that
doesn't include USB) which is to simply do the HW suspend in a late
callback after IRQs are off, and not bother with the rest.

> > I even tend to think that we could do STD that way too, in
> > fact, while Linus is right saying it's a different problem than STR, we
> > could even probably re-use some of the STR infrastructure in some
> > hackish way, still without a freezer. We could have ways to block page
> > cache writeout, for example, to prevent new post-snapshot dirty data
> > from hitting the platter, and use direct BIOs for writeout. That's just
> > an example.
> 
> What about systems with no BIOS?  I think this would be very hard or 
> even impossible to make work.

You made the same mistake I did when reading Nigel's mail ... BIOs ->
Block IO requests, not BIOS :-)

Ben.

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Alan Stern]

On Sat, 7 Jul 2007, Benjamin Herrenschmidt wrote:

> > > so the suspend process will wait for it.  When binding  
> > > is done the suspend_device() code will take the device lock and tell  
> > > everything else to postpone further bind requests as above.
> > 
> > My question referred to drivers trying to bind or unbind a device
> > _after_ the device has been suspended.  I suppose you'll say that's
> > covered by the NO_BIND flag.  But now we have the locking problem
> > mentioned above: The thread trying to bind is holding a lock which is
> > needed for resuming.
> 
> Why would it ? Just make it fail, maybe with some kind of -ERETRY... Or
> it can spin with the lock not held if it want. That's a detail really.

Spinning in the driver with the lock not held is impossible, since the 
driver is called with the lock already acquired.

Failing with -ERETRY is non-transparent.  I would prefer to block such 
requests at their source, before the lock is acquired.  Perhaps in the 
driver core, perhaps even earlier.

(And rather than trying to manage a waitqueue or struct completion, it
would be easiest to jump directly into the freezer!  The driver or the 
core wouldn't have to worry about waking up all these blocked threads.)

> > As one of the people responsible for the USB power management 
> > implementation, I would appreciate more details about this.  For 
> > example, a dmesg log with CONFIG_USB_DEBUG turned on together with a
> > complete description of the actions you took to provoke the bug.
> > 
> > (I wonder how much of this "buginess" is caused by the lack of the 
> > freezer in PPC.)
> 
> No. The freezer will hide some of those problems under the carpet, but
> not solve the basic issue which is the driver should be solid. Period.

You're missing the point.  If the driver and the freezer are both 
solid, there's no reason they can't share the work.  If many drivers 
can pass off part of their workload to the single freezer, it's a net 
win.

So it isn't a question of how solid the drivers are; it's a question 
of how solid the freezer is.  And bear in mind that if you convince 
people the freezer is not solid enough to be used, then you will have 
to find an alternative for purposes of hibernation.

> The freezer is a flawed concept in the first place.

<... Long and cogent argument which I will skip over for now ...>

> Face it, we should seriously look into doing suspend/resume without a
> freezer.

I'm willing to try, although I think it will be a tremendous amount of 
work to verify that every driver does the right thing.  There's lots of 
support missing.  For example, don't you think we should block all 
sysfs I/O during suspend?  And likewise for insmod/rmmod?

> I even tend to think that we could do STD that way too, in
> fact, while Linus is right saying it's a different problem than STR, we
> could even probably re-use some of the STR infrastructure in some
> hackish way, still without a freezer. We could have ways to block page
> cache writeout, for example, to prevent new post-snapshot dirty data
> from hitting the platter, and use direct BIOs for writeout. That's just
> an example.

What about systems with no BIOS?  I think this would be very hard or 
even impossible to make work.

Alan Stern

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Benjamin Herrenschmidt]

> > so the suspend process will wait for it.  When binding  
> > is done the suspend_device() code will take the device lock and tell  
> > everything else to postpone further bind requests as above.
> 
> My question referred to drivers trying to bind or unbind a device
> _after_ the device has been suspended.  I suppose you'll say that's
> covered by the NO_BIND flag.  But now we have the locking problem
> mentioned above: The thread trying to bind is holding a lock which is
> needed for resuming.

Why would it ? Just make it fail, maybe with some kind of -ERETRY... Or
it can spin with the lock not held if it want. That's a detail really.

> As one of the people responsible for the USB power management 
> implementation, I would appreciate more details about this.  For 
> example, a dmesg log with CONFIG_USB_DEBUG turned on together with a
> complete description of the actions you took to provoke the bug.
> 
> (I wonder how much of this "buginess" is caused by the lack of the 
> freezer in PPC.)

No. The freezer will hide some of those problems under the carpet, but
not solve the basic issue which is the driver should be solid. Period.

The freezer is a flawed concept in the first place. If you go back to
square one, what is the basic idea of it ? I'll basically expose the
idea and go down all of the path I have in mind where it stops working
and becomes an incredibly difficult thing that in the end doesn't even
solve all the problems it's supposed to.

So first thing first...

I want a quiescent system with no new "IO requests" (whatever that mean
in the context of drivers) issued to avoid races during suspend/resume.

That sounds like a nice idea. Yeah. Sounds... only. Problem is. How do
you define that quiescent system ? First idea is ... let's stop
userland. There are various ways of doing that, but the freezer hooking
into the signal code is not necessarily a bad one.

No, I'm purposefully putting aside all the cases where the above doesn't
work (user process in the kernel in some uninterruptible wait, etc...),
which are the first big setback imho... our simple idea is suddenly not
so simple anymore, but we can bring those back later.

Now, there is still a problem... kernel threads. In fact, there is no
fundamental distinction between a kernel thread and a user process...
one has an MM and the other doesn't but as far as we are concerned, it's
the same. Kernel threads can issues IOs, or like khubd, detect devices,
plug/unplug them, etc etc.... all over the place.

Easy answer that comes to mind -> freeze them too. Heh, but kernel
threads don't do signals, so we end up with all those try_to_freeze().
Then what about the fact that drivers may need those kernel threads to
proceed ? Some drivers queue up their IO requests to a kernel thread to
process them and suspend() might need to flush those down, issue a
couple more such as "spin down disks" before that kernel thread can
actually be frozen... Hrm.. maybe not all of them then.

But how do you decide ? What defines that a kernel can issue an IO ? In
fact, if you look closely, anything doing kmalloc(...,GFP_KERNEL) for
example can trigger an IO... implicitely, via the VM pushing things out.
And that's just one example.

In some case, those same threads that may need to be kept non-frozen are
-also- the ones that will potentially submit new IOs or bring in new
devices.

And then, there is keventd ... what do you do about work queues ? You
have everybody pouring things at workqueues... some of these things may
well hit your driver, some may not. Same goes in some cases with
interrupt time stuff, such as timers or tasklets.. think about
networking.

In the end, the nice idea that "threads/tasks cause requests, so we just
stop them" basically falls appart. Half of the kernel can cause a driver
to be hit somewhere and a given time, it can be from a thread context,
directly caused by userland, or from some timer due to some subsystem
having a keepalive thing ticking in or whatever else.

Now, we go back to the previous issue of what do we do about
uninterruptible sleep... You want to abort suspend because, for example,
somethign called a driver that does an msleep(200) or so ? Are you aware
that 99% of laptop users close their laptops and shove it in the bag not
even waiting for the disk to spin down ? And you want suspend to abort
because some random "happen all the time" even such as a process being
somewhere temporarily in uninterruptible state in the kernel ?

So let's say we freeze them from within the scheduler even when they are
uininterruptible.. ouch... you just caused the deadlocks we talked about
before. While without a freezer, suspend() can at least rely on the fact
that it can wait for processes that have such pending locked constructs
waiting will ultimately wakeupm and wait for them (or even explicitely
wake them), it can't if they've been frozen. So what was a perfectly
solvable moderate driver synchronisation issue becomes a deadlock
nightmare.

And those are just example. During this discussion, we also brought the
example of FUSE which is a big stab at the whole freezer concept. And
I'm sure we can find more everyday.

Face it, we should seriously look into doing suspend/resume without a
freezer. I even tend to think that we could do STD that way too, in
fact, while Linus is right saying it's a different problem than STR, we
could even probably re-use some of the STR infrastructure in some
hackish way, still without a freezer. We could have ways to block page
cache writeout, for example, to prevent new post-snapshot dirty data
from hitting the platter, and use direct BIOs for writeout. That's just
an example.

Ben.

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Benjamin Herrenschmidt]

On Fri, 2007-07-06 at 10:38 -0400, Alan Stern wrote:
> On Fri, 6 Jul 2007, Benjamin Herrenschmidt wrote:
> 
> > What you propose is basically a slightly over-simplistic version of what
> > I think (and Paulus think) should be done. We do need to do it via
> > driver callbacks down the tree since only drivers can know how to deal
> > with their DMA etc... and ordering need to be respected, but that's
> > basically it.
> > 
> > And guess what ? It's what we do on powerbooks, and it works fine,
> > without a freezer :-)
> 
> I wish you'd stop saying that.  Have you ever done any serious testing?
> 
> Here's something to try:  Add a time delay to the end of hub_suspend in
> drivers/usb/core/hub.c, so you can provoke a race manually.  Then while
> one of your root hubs is being suspended and the system is waiting in
> that delay, either plug in a new USB device to that hub or unplug an
> existing device.
> 
> Be sure that CONFIG_USB_DEBUG is on so that we can figure out what 
> happened after the fact.

If you remember, one of the things I've been advocating has always been
that we should put on hold all plug activity (unplug might be alright as
long as the user events are just delayed) when we start suspending. No
new devices, no new bindings. "hub" type devices are respondible for
bringing in the new stuff after resume.

Ben.

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Pavel Machek]

Hi!

> > > And guess what ? It's what we do on powerbooks, and it works fine,
> > > without a freezer :-)

Well, issue is, you should stop claiming it works fine until issue
below is fixed... please?

And anyway I believe that current issue (fuse deadlocks with s2ram)
should be present on powerbooks, too... it is just way harder to
trigger. All that is neccessary is fused (or one of its helpers) to
get frozen by accessing suspended device.


									Pavel

> > I wish you'd stop saying that.  Have you ever done any serious testing?
> > 
> > Here's something to try:  Add a time delay to the end of hub_suspend in
> > drivers/usb/core/hub.c, so you can provoke a race manually.  Then while
> > one of your root hubs is being suspended and the system is waiting in
> > that delay, either plug in a new USB device to that hub or unplug an
> > existing device.
> > 
> > Be sure that CONFIG_USB_DEBUG is on so that we can figure out what 
> > happened after the fact.
> 
> If you remember, one of the things I've been advocating has always been
> that we should put on hold all plug activity (unplug might be alright as
> long as the user events are just delayed) when we start suspending. No
> new devices, no new bindings. "hub" type devices are respondible for
> bringing in the new stuff after resume.



-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Benjamin Herrenschmidt]

On Sat, 2007-07-07 at 13:49 +0200, Pavel Machek wrote:

> And anyway I believe that current issue (fuse deadlocks with s2ram)
> should be present on powerbooks, too... it is just way harder to
> trigger. All that is neccessary is fused (or one of its helpers) to
> get frozen by accessing suspended device.

I don't see any fuse specific issue without a freezer, but I see generic
issues such as kmalloc blocking in a driver with a mutex held etc... but
I've talked about these already. They have nothing to do with the
freezer though.

Ben.

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Alan Stern]

On Sat, 7 Jul 2007, Benjamin Herrenschmidt wrote:

> > > And guess what ? It's what we do on powerbooks, and it works fine,
> > > without a freezer :-)

> If you remember, one of the things I've been advocating has always been
> that we should put on hold all plug activity (unplug might be alright as
> long as the user events are just delayed) when we start suspending. No
> new devices, no new bindings. "hub" type devices are respondible for
> bringing in the new stuff after resume.

Which is exactly my point.  It _doesn't_ work fine without a freezer, 
because the USB stack currently relies on the freezer to prevent plug 
activity.

Alan Stern

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Benjamin Herrenschmidt]

On Sat, 2007-07-07 at 12:17 -0400, Alan Stern wrote:
> On Sat, 7 Jul 2007, Benjamin Herrenschmidt wrote:
> 
> > > > And guess what ? It's what we do on powerbooks, and it works fine,
> > > > without a freezer :-)
> 
> > If you remember, one of the things I've been advocating has always been
> > that we should put on hold all plug activity (unplug might be alright as
> > long as the user events are just delayed) when we start suspending. No
> > new devices, no new bindings. "hub" type devices are respondible for
> > bringing in the new stuff after resume.
> 
> Which is exactly my point.  It _doesn't_ work fine without a freezer, 
> because the USB stack currently relies on the freezer to prevent plug 
> activity.

Putting on hold plug activity has nothing, NOTHING, to do with the half
assed piece of deadlocking crap we have now we call a freezer.

As long as you guys keep mixing up all the issues and coming up with
totally bogus solutions that cannot work, we won't have a useful suspend
(either to RAM or to disk) in linux.

Ben.

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Alan Stern]

On Sun, 8 Jul 2007, Benjamin Herrenschmidt wrote:

> > Which is exactly my point.  It _doesn't_ work fine without a freezer, 
> > because the USB stack currently relies on the freezer to prevent plug 
> > activity.
> 
> Putting on hold plug activity has nothing, NOTHING, to do with the half
> assed piece of deadlocking crap we have now we call a freezer.

You're wrong; it _does_ have something to do with the freezer.  The 
connection is that the code uses the freezer to put plug activity on 
hold.

You probably meant to say that it _should_ have nothing to do with the 
freezer.  That's a different matter.  But in any case you should write 
what you actually mean, rather than just putting down something as 
inflammatory as possible.

> As long as you guys keep mixing up all the issues and coming up with
> totally bogus solutions that cannot work, we won't have a useful suspend
> (either to RAM or to disk) in linux.

In my defense, you should realize that until Rafael's notifier chain
was added (just a few weeks ago, still not in mainline I believe) there
was no other way to do it.  Plug activity needs to be stopped before
the child devices are suspended, and the PM core does not send any
notification to drivers at that time.  All it does is activate the 
freezer.

Alan Stern

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Rafael J. Wysocki]

On Sunday, 8 July 2007 02:42, Benjamin Herrenschmidt wrote:
> On Sat, 2007-07-07 at 12:17 -0400, Alan Stern wrote:
> > On Sat, 7 Jul 2007, Benjamin Herrenschmidt wrote:
> > 
> > > > > And guess what ? It's what we do on powerbooks, and it works fine,
> > > > > without a freezer :-)
> > 
> > > If you remember, one of the things I've been advocating has always been
> > > that we should put on hold all plug activity (unplug might be alright as
> > > long as the user events are just delayed) when we start suspending. No
> > > new devices, no new bindings. "hub" type devices are respondible for
> > > bringing in the new stuff after resume.
> > 
> > Which is exactly my point.  It _doesn't_ work fine without a freezer, 
> > because the USB stack currently relies on the freezer to prevent plug 
> > activity.
> 
> Putting on hold plug activity has nothing, NOTHING, to do with the half
> assed piece of deadlocking crap we have now we call a freezer.

Well, no matter how much bad names you will call it, that's what in use
right now and it's not _that_ easy to get rid of.

I'm perfectly fine with readying drivers etc. for dropping the freezer, but
in the meantime bugs in the freezer need to be fixed, if possible.

And if it deadlocks, there is a bug in it.

> As long as you guys keep mixing up all the issues and coming up with
> totally bogus solutions that cannot work, we won't have a useful suspend
> (either to RAM or to disk) in linux. 

Discussion at this level would be totally counterproductive and wouldn't lead
to anything constructive, so let me avoid commenting your abusive remarks.

Greetings,
Rafael


-- 
"Premature optimization is the root of all evil." - Donald Knuth

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Alan Stern]

On Fri, 6 Jul 2007, Kyle Moffett wrote:

> Except Linus already decreed (and I heartily agree) that hibernation  
> and suspend-to-RAM were fundamentally completely different operations  
> and therefore any attempts to share code were basically just making a  
> big muddy mess of things.  Would a thread "Remove phase-of-the-moon  
> calculations from network-recv code" be relevant to lunar observation  
> just because the two had to do with the phases of the moon?  No!

For that matter, shutting down a CPU and hibernation are fundamentally
different operations -- but they both use the freezer.  Is that a big
muddy mess?  But never mind; I won't discuss hibernation.

> > A)  Decide what to do about remote wakeup requests.
> 
> Why do we care?  If the wakeup request arrives before we go to sleep,  
> we obviously aren't asleep and so can't wake up.  If it arrives after  
> we go to sleep then it will wake us up.  Anything that depends on a  
> wakeup arriving mid-sequence is 100% masochistic race condition.

You don't understand my point.  If a wakeup request arrives before the
system goes to sleep, and it is serviced, then a device which ought to
have been suspended will in fact be awake.  This will (if the parent's
driver is written correctly) cause the sleep transition to abort.

Not that there's necessarily anything wrong with that.  I just wanted 
to be sure you were aware of the potential problems.

> > C)  Prevent devices and drivers from being registered or  
> > unregistered; in particular decide what to do about hot-plug or hot- 
> > unplug events.
> 
> (1b) Again, that's where the NO_BIND flag comes in.  If its set then  
> any device probe events must sleep, otherwise they can go through.

I didn't say "bind", I said "registered".  Admittedly, they are rather
similar.

Still, there are difficulties.  Let's say a driver has set the NO_BIND 
flag for one of its devices.  A bind request comes in, and the driver 
puts it on a waitqueue.  Note that the binding thread holds the device 
semaphore; this is always true when a driver's probe routine is called.

Later on it comes time for the PM core to resume the device, which will
start up the threads on the waitqueue.  Before doing so it must acquire
the device semaphore.  Deadlock!

> If any of those things screw up suspend-to-RAM then it is 100% the  
> drivers fault and no "process freezer" is going to fix it, end of  
> story.

Why do you say that?  A "process freezer" can prevent bind and
registration calls from occurring, since these calls have to run in
process context.  Ergo a freezer _can_ fix some of these problems.

>  And "A" cannot be made reliable.  At some point you shut off  
> interrupts right before going to sleep, and at that point any remote  
> wakeup event is just going to get dropped until you actually enter  
> sleep mode and the hardware takes over again.  If you miss a wakeup  
> event then whatever sent it should just retry, just as with *every*  
> other kind of network packet.

Who mentioned network packets?  And who says a remote wakeup event will 
get dropped once interrupts are disabled?  More likely it will set a 
bit somewhere that causes the system to wake up immediately after it 
has gone to sleep.

> > So what happens if a new subdevice arrives at the wrong time?  Do  
> > you block instead of binding it?  While holding a mutex needed to  
> > suspend the parent device?
> 
> That would be a driver bug.  If you have asynchronous probing then  
> proper suspend handling includes being able to postpone driver probe  
> events until after resume.

One of your conditions (embodied in the pseudocode you posted earlier) 
was that drivers should be told to prevent binding and registration 
before the child devices are suspended.  Currently the PM core doesn't 
do anything like that.  You can't blame the drivers for this lack.

Of course it could be added.  Or perhaps more easily, the drivers that 
support asynchronous probing could be notified when a suspend is about 
to start so they could begin blocking bindings/registrations then.

> > What about drivers trying to bind to existing devices?
> 
> While binding it will clearly be holding a mutex/spinlock on the  
> parent device,

("Parent device"?  Do you mean the device being bound?  If so then I
agree.  Or do you mean the device's parent?  If so then your statement
is not clear at all.  There is special-case code in the driver core to
make sure it is true for USB devices, and it looks ugly as can be.)

> so the suspend process will wait for it.  When binding  
> is done the suspend_device() code will take the device lock and tell  
> everything else to postpone further bind requests as above.

My question referred to drivers trying to bind or unbind a device
_after_ the device has been suspended.  I suppose you'll say that's
covered by the NO_BIND flag.  But now we have the locking problem
mentioned above: The thread trying to bind is holding a lock which is
needed for resuming.

> Most drivers have an implicit NO_BIND flag:  The device's interrupts  
> are off and/or its in a low-power state.

That won't cause binding to block or be postponed; it will cause it to
fail.  Not the same thing at all.

>  USB is already terribly  
> buggy with regards to suspend:  If you hotplug a device during  
> suspend (like the touchpad in my powerbook powering down/up), then  
> the USB stack will basically hang that controller.  The device is off  
> and the hotplug triggers interrupts and IO, *EVEN* *WITHOUT*  
> *USERSPACE*.

As one of the people responsible for the USB power management 
implementation, I would appreciate more details about this.  For 
example, a dmesg log with CONFIG_USB_DEBUG turned on together with a
complete description of the actions you took to provoke the bug.

(I wonder how much of this "buginess" is caused by the lack of the 
freezer in PPC.)

> So if your driver doesn't already have a proper way of blocking IO  
> during suspend then it probably doesn't suspend 50% of the time  
> anyways.  A bug which bites *every* *time* is easy to fix, one which  
> only bites when things hit a race condition is much harder.

The USB drivers (at least, the ones with runtime PM support) rely on 
the freezer to block I/O during suspend.  As far as I know, they do 
suspend properly, on systems where the freezer is used.

> That responsibility has been there ever since suspend-to-RAM support  
> was added.  Nobody ever denied that writing a proper driver wasn't  
> tricky.  You have to simultaneously be able to handle handle hot- 
> unplug, IO errors, interrupts, IO requests, suspend-to-RAM, and  
> hibernation.  If your driver mutual-exclusion is buggy then it  
> probably already bites you during hotplug or other similar  
> scenarios.  Let's at least make the problems much more reproducible  
> so we can fix the drivers properly instead of continuing to kludge  
> around it for all eternity.

Is reproducibility really a problem at this stage?  A bug which bites 
50% of the time might not be quite as easy to fix as one which occurs 
every time, but it isn't terribly bad either.

Alan Stern

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Kyle Moffett]

On Jul 06, 2007, at 11:42:33, Alan Stern wrote:
> On Thu, 5 Jul 2007, Kyle Moffett wrote:
>> Umm, this thread is NOT ABOUT HIBERNATING!!!  Please go back and  
>> read the subject, specifically the "suspend to RAM" parts :-D.
>
> But it _is_ about the freezer; see the "Remove process freezer"  
> part. :-)  Since the freezer is used during hibernation,  
> hibernation is a legitimate topic.

Except Linus already decreed (and I heartily agree) that hibernation  
and suspend-to-RAM were fundamentally completely different operations  
and therefore any attempts to share code were basically just making a  
big muddy mess of things.  Would a thread "Remove phase-of-the-moon  
calculations from network-recv code" be relevant to lunar observation  
just because the two had to do with the phases of the moon?  No!

>> When your hardware can put itself to sleep and atomically preserve  
>> memory as it does so, you don't need an atomic copy.  For Real  
>> Suspend(TM) (IE: Suspend-to-RAM), the list of things to do is  
>> short and simple:
>>
>> 1)  Stop DMA and put most hardware into low-power states (stops  
>> all interrupt sources)
>> 2)  Ensure that the other CPUs have finished any trailing  
>> interrupt handlers and put them to sleep
>> 3)  Put the interrupt-controllers into low-power stat
>> 4)  Go to sleep
>
> Your short and simple list omits a few crucial items:
>
> A)  Decide what to do about remote wakeup requests.

Why do we care?  If the wakeup request arrives before we go to sleep,  
we obviously aren't asleep and so can't wake up.  If it arrives after  
we go to sleep then it will wake us up.  Anything that depends on a  
wakeup arriving mid-sequence is 100% masochistic race condition.

> B)  Prevent I/O requests from resuming devices that have been  
> suspended.

(1a) As I describe below, step (1) includes setting NO_BIND and NO_IO  
flags on devices as they are processed.  Anybody who wants to do IO  
while those flags are set should just go sleep on a waitqueue.

> C)  Prevent devices and drivers from being registered or  
> unregistered; in particular decide what to do about hot-plug or hot- 
> unplug events.

(1b) Again, that's where the NO_BIND flag comes in.  If its set then  
any device probe events must sleep, otherwise they can go through.

> D)  Block driver bind or unbind calls.

See points (1a) and (1b) above.

> Any of these things is capable of screwing up the course of  
> events.  (In fact A _should_ be allowed to abort a suspend.)

If any of those things screw up suspend-to-RAM then it is 100% the  
drivers fault and no "process freezer" is going to fix it, end of  
story.  And "A" cannot be made reliable.  At some point you shut off  
interrupts right before going to sleep, and at that point any remote  
wakeup event is just going to get dropped until you actually enter  
sleep mode and the hardware takes over again.  If you miss a wakeup  
event then whatever sent it should just retry, just as with *every*  
other kind of network packet.

>> How about a freezer whose job it is to "wait for pending hard  
>> interrupts to complete when we have already guaranteed that we  
>> won't get any more"?  That part should be really *REALLY* easy.   
>> You don't need to care about either userspace processes or kernel  
>> threads at all.  Specifically, Step 1 consists of:
>>
>> suspend_device(dev)
>> {
>> 	set_no_bind_flag(dev);
>> 	for (dev->subdevices)
>> 		suspend_device(dev);
>> 	set_no_io_flag(dev);
>> 	wait_for_in_progress_dma(dev);
>> 	turn_off_interrupts(dev);
>> 	go_to_low_power_state(dev);
>> }
>>
>> After you've set the "no_bind" flag, you won't get any *new*  
>> subdevices trying to bind,
>
> So what happens if a new subdevice arrives at the wrong time?  Do  
> you block instead of binding it?  While holding a mutex needed to  
> suspend the parent device?

That would be a driver bug.  If you have asynchronous probing then  
proper suspend handling includes being able to postpone driver probe  
events until after resume.  If you have synchronous probing then the  
problem doesn't exist because "set_no_bind_flag" is just telling the  
device not to raise any more device probe interrupts.

> What about drivers trying to bind to existing devices?

While binding it will clearly be holding a mutex/spinlock on the  
parent device, so the suspend process will wait for it.  When binding  
is done the suspend_device() code will take the device lock and tell  
everything else to postpone further bind requests as above.

> What happens to I/O requests submitted after the "no_io" flag is  
> set?  The driver will have to block them, effectively creating its  
> own little "freezer".

Oh, so you're calling every waitqueue in the kernel a "freezer" now?   
We do these things at the driver level *all* *the* *time*.  For  
instance, you can't submit new IOs to an ATA controller while it's  
renegotiating the bus speed, but that's never been a problem before.

>> When all the leaf devices are off, the parent device can be turned  
>> off because everything waiting on the leaf devices is blocked on  
>> them and won't unblock until the parent device *AND* the leaf  
>> device are turned on again, in that order.
>
> This is a lot like what we already do.  The differences are:
>
> There is nothing corresponding to your "no-bind" flag.
>
> Most drivers don't have anything like your "no_io" flag; they  
> assume that nobody will be around to submit an I/O request.

Most drivers have an implicit NO_BIND flag:  The device's interrupts  
are off and/or its in a low-power state.  USB is already terribly  
buggy with regards to suspend:  If you hotplug a device during  
suspend (like the touchpad in my powerbook powering down/up), then  
the USB stack will basically hang that controller.  The device is off  
and the hotplug triggers interrupts and IO, *EVEN* *WITHOUT*  
*USERSPACE*.

So if your driver doesn't already have a proper way of blocking IO  
during suspend then it probably doesn't suspend 50% of the time  
anyways.  A bug which bites *every* *time* is easy to fix, one which  
only bites when things hit a race condition is much harder.

>> Resuming is basically running the whole process in reverse.   
>> Runtime-suspend is achieved by not setting the 'no_io' or  
>> 'no_bind' flags and putting selective device-subtrees to sleep  
>> without doing anything to the rest of the system.
>
> Nobody doubts that suspend can be made to work without the freezer.  
> The point is that doing it this way dumps a bunch of extra  
> responsibility on drivers.

That responsibility has been there ever since suspend-to-RAM support  
was added.  Nobody ever denied that writing a proper driver wasn't  
tricky.  You have to simultaneously be able to handle handle hot- 
unplug, IO errors, interrupts, IO requests, suspend-to-RAM, and  
hibernation.  If your driver mutual-exclusion is buggy then it  
probably already bites you during hotplug or other similar  
scenarios.  Let's at least make the problems much more reproducible  
so we can fix the drivers properly instead of continuing to kludge  
around it for all eternity.

Cheers,
Kyle Moffett

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Miklos Szeredi]

> > Alan Stern writes:
> > 
> > > Remember what I wrote a few minutes ago about khubd and ksuspend_usbd
> > > wanting to resume devices during a system suspend transition?  This is
> > > exactly what happens when those threads aren't frozen.
> > 
> > So, I wonder why I don't see that error on my powerbook?
> 
> That's a good question.  Miklos, can you please reproduce the suspend 
> error using a kernel built with CONFIG_USB_DEBUG turned on?

Here's the full dmesg.  First there was a successful suspend/resume,
then a couple of unsuccessful ones:

Linux version 2.6.22-rc6 (mszeredi@tucsk) (gcc version 4.1.2 20061115 (prerelease) (SUSE Linux)) #19 SMP Thu Jul 5 18:07:34 CEST 2007
Command line: vga=775 root=/dev/sda2
BIOS-provided physical RAM map:
 BIOS-e820: 0000000000000000 - 000000000009f000 (usable)
 BIOS-e820: 000000000009f000 - 00000000000a0000 (reserved)
 BIOS-e820: 00000000000d2000 - 00000000000d4000 (reserved)
 BIOS-e820: 00000000000dc000 - 0000000000100000 (reserved)
 BIOS-e820: 0000000000100000 - 000000003fed0000 (usable)
 BIOS-e820: 000000003fed0000 - 000000003fedf000 (ACPI data)
 BIOS-e820: 000000003fedf000 - 000000003ff00000 (ACPI NVS)
 BIOS-e820: 000000003ff00000 - 0000000040000000 (reserved)
 BIOS-e820: 00000000f0000000 - 00000000f4000000 (reserved)
 BIOS-e820: 00000000fec00000 - 00000000fec10000 (reserved)
 BIOS-e820: 00000000fed00000 - 00000000fed00400 (reserved)
 BIOS-e820: 00000000fed14000 - 00000000fed1a000 (reserved)
 BIOS-e820: 00000000fed1c000 - 00000000fed90000 (reserved)
 BIOS-e820: 00000000fee00000 - 00000000fee01000 (reserved)
 BIOS-e820: 00000000ff800000 - 0000000100000000 (reserved)
Entering add_active_range(0, 0, 159) 0 entries of 256 used
Entering add_active_range(0, 256, 261840) 1 entries of 256 used
end_pfn_map = 1048576
DMI present.
ACPI: RSDP 000F67D0, 0024 (r2 LENOVO)
ACPI: XSDT 3FED1308, 008C (r1 LENOVO TP-79        2110  LTP        0)
ACPI: FACP 3FED1400, 00F4 (r3 LENOVO TP-79        2110 LNVO        1)
ACPI Warning (tbfadt-0434): Optional field "Gpe1Block" has zero address or length: 000000000000102C/0 [20070126]
ACPI: DSDT 3FED175E, D481 (r1 LENOVO TP-79        2110 MSFT  100000E)
ACPI: FACS 3FEF4000, 0040
ACPI: SSDT 3FED15B4, 01AA (r1 LENOVO TP-79        2110 MSFT  100000E)
ACPI: ECDT 3FEDEBDF, 0052 (r1 LENOVO TP-79        2110 LNVO        1)
ACPI: TCPA 3FEDEC31, 0032 (r2 LENOVO TP-79        2110 LNVO        1)
ACPI: APIC 3FEDEC63, 0068 (r1 LENOVO TP-79        2110 LNVO        1)
ACPI: MCFG 3FEDECCB, 003C (r1 LENOVO TP-79        2110 LNVO        1)
ACPI: HPET 3FEDED07, 0038 (r1 LENOVO TP-79        2110 LNVO        1)
ACPI: SLIC 3FEDEE62, 0176 (r1 LENOVO TP-79        2110  LTP        0)
ACPI: BOOT 3FEDEFD8, 0028 (r1 LENOVO TP-79        2110  LTP        1)
ACPI: SSDT 3FEF2655, 025F (r1 LENOVO TP-79        2110 INTL 20050513)
ACPI: SSDT 3FEF28B4, 00A6 (r1 LENOVO TP-79        2110 INTL 20050513)
ACPI: SSDT 3FEF295A, 04F7 (r1 LENOVO TP-79        2110 INTL 20050513)
ACPI: SSDT 3FEF2E51, 01D8 (r1 LENOVO TP-79        2110 INTL 20050513)
Entering add_active_range(0, 0, 159) 0 entries of 256 used
Entering add_active_range(0, 256, 261840) 1 entries of 256 used
Zone PFN ranges:
  DMA             0 ->     4096
  DMA32        4096 ->  1048576
  Normal    1048576 ->  1048576
early_node_map[2] active PFN ranges
    0:        0 ->      159
    0:      256 ->   261840
On node 0 totalpages: 261743
  DMA zone: 56 pages used for memmap
  DMA zone: 1144 pages reserved
  DMA zone: 2799 pages, LIFO batch:0
  DMA32 zone: 3523 pages used for memmap
  DMA32 zone: 254221 pages, LIFO batch:31
  Normal zone: 0 pages used for memmap
ACPI: PM-Timer IO Port: 0x1008
ACPI: Local APIC address 0xfee00000
ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled)
Processor #0 (Bootup-CPU)
ACPI: LAPIC (acpi_id[0x01] lapic_id[0x01] enabled)
Processor #1
ACPI: LAPIC_NMI (acpi_id[0x00] high edge lint[0x1])
ACPI: LAPIC_NMI (acpi_id[0x01] high edge lint[0x1])
ACPI: IOAPIC (id[0x01] address[0xfec00000] gsi_base[0])
IOAPIC[0]: apic_id 1, address 0xfec00000, GSI 0-23
ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)
ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)
ACPI: IRQ0 used by override.
ACPI: IRQ2 used by override.
ACPI: IRQ9 used by override.
Setting APIC routing to flat
ACPI: HPET id: 0x8086a201 base: 0xfed00000
Using ACPI (MADT) for SMP configuration information
swsusp: Registered nosave memory region: 000000000009f000 - 00000000000a0000
swsusp: Registered nosave memory region: 00000000000a0000 - 00000000000d2000
swsusp: Registered nosave memory region: 00000000000d2000 - 00000000000d4000
swsusp: Registered nosave memory region: 00000000000d4000 - 00000000000dc000
swsusp: Registered nosave memory region: 00000000000dc000 - 0000000000100000
Allocating PCI resources starting at 50000000 (gap: 40000000:b0000000)
SMP: Allowing 2 CPUs, 0 hotplug CPUs
PERCPU: Allocating 32616 bytes of per cpu data
Built 1 zonelists.  Total pages: 257020
Kernel command line: vga=775 root=/dev/sda2
Initializing CPU#0
PID hash table entries: 4096 (order: 12, 32768 bytes)
Extended CMOS year: 2000
Marking TSC unstable due to TSCs unsynchronized
time.c: Detected 1828.744 MHz processor.
Console: colour dummy device 80x25
Dentry cache hash table entries: 131072 (order: 8, 1048576 bytes)
Inode-cache hash table entries: 65536 (order: 7, 524288 bytes)
Checking aperture...
Memory: 1026200k/1047360k available (2316k kernel code, 20600k reserved, 1434k data, 188k init)
Calibrating delay using timer specific routine.. 3661.49 BogoMIPS (lpj=7322983)
Mount-cache hash table entries: 256
CPU: L1 I cache: 32K, L1 D cache: 32K
CPU: L2 cache: 2048K
using mwait in idle threads.
CPU: Physical Processor ID: 0
CPU: Processor Core ID: 0
CPU0: Thermal monitoring enabled (TM2)
SMP alternatives: switching to UP code
ACPI: Core revision 20070126
Using local APIC timer interrupts.
result 10390586
Detected 10.390 MHz APIC timer.
SMP alternatives: switching to SMP code
Booting processor 1/2 APIC 0x1
Initializing CPU#1
Calibrating delay using timer specific routine.. 3657.60 BogoMIPS (lpj=7315219)
CPU: L1 I cache: 32K, L1 D cache: 32K
CPU: L2 cache: 2048K
CPU: Physical Processor ID: 0
CPU: Processor Core ID: 1
CPU1: Thermal monitoring enabled (TM2)
Intel(R) Core(TM)2 CPU         T5600  @ 1.83GHz stepping 06
Brought up 2 CPUs
migration_cost=39
NET: Registered protocol family 16
ACPI: bus type pci registered
PCI: Using configuration type 1
ACPI: Interpreter enabled
ACPI: (supports S0 S3 S4 S5)
ACPI: Using IOAPIC for interrupt routing
ACPI: PCI Root Bridge [PCI0] (0000:00)
PCI: Probing PCI hardware (bus 00)
PCI quirk: region 1000-107f claimed by ICH6 ACPI/GPIO/TCO
PCI quirk: region 1180-11bf claimed by ICH6 GPIO
PCI: Transparent bridge - 0000:00:1e.0
ACPI: PCI Interrupt Routing Table [\_SB_.PCI0._PRT]
ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.AGP_._PRT]
ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.EXP0._PRT]
ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.EXP1._PRT]
ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.EXP2._PRT]
ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.EXP3._PRT]
ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.PCI1._PRT]
ACPI: PCI Interrupt Link [LNKA] (IRQs 3 4 5 6 7 9 10 *11)
ACPI: PCI Interrupt Link [LNKB] (IRQs 3 4 5 6 7 9 10 *11)
ACPI: PCI Interrupt Link [LNKC] (IRQs 3 4 5 6 7 9 10 *11)
ACPI: PCI Interrupt Link [LNKD] (IRQs 3 4 5 6 7 9 10 *11)
ACPI: PCI Interrupt Link [LNKE] (IRQs 3 4 5 6 7 9 10 *11)
ACPI: PCI Interrupt Link [LNKF] (IRQs 3 4 5 6 7 9 10 *11)
ACPI: PCI Interrupt Link [LNKG] (IRQs 3 4 5 6 7 9 10 *11)
ACPI: PCI Interrupt Link [LNKH] (IRQs 3 4 5 6 7 9 10 *11)
ACPI: Power Resource [PUBS] (on)
Linux Plug and Play Support v0.97 (c) Adam Belay
pnp: PnP ACPI init
ACPI: bus type pnp registered
pnp: PnP ACPI: found 12 devices
ACPI: ACPI bus type pnp unregistered
SCSI subsystem initialized
libata version 2.21 loaded.
usbcore: registered new interface driver usbfs
usbcore: registered new interface driver hub
usbcore: registered new device driver usb
PCI: Using ACPI for IRQ routing
PCI: If a device doesn't work, try "pci=routeirq".  If it helps, post a report
PCI-GART: No AMD northbridge found.
hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0
hpet0: 3 64-bit timers, 14318180 Hz
pnp: 00:00: iomem range 0x0-0x9ffff could not be reserved
pnp: 00:00: iomem range 0xc0000-0xc3fff has been reserved
pnp: 00:00: iomem range 0xc4000-0xc7fff has been reserved
pnp: 00:00: iomem range 0xc8000-0xcbfff has been reserved
pnp: 00:02: iomem range 0xf0000000-0xf3ffffff could not be reserved
pnp: 00:02: iomem range 0xfed1c000-0xfed1ffff could not be reserved
pnp: 00:02: iomem range 0xfed14000-0xfed17fff could not be reserved
pnp: 00:02: iomem range 0xfed18000-0xfed18fff could not be reserved
PCI: Bridge: 0000:00:01.0
  IO window: 2000-2fff
  MEM window: ee100000-ee1fffff
  PREFETCH window: d8000000-dfffffff
PCI: Bridge: 0000:00:1c.0
  IO window: 3000-3fff
  MEM window: ee000000-ee0fffff
  PREFETCH window: disabled.
PCI: Bridge: 0000:00:1c.1
  IO window: 4000-5fff
  MEM window: ec000000-edffffff
  PREFETCH window: e4000000-e40fffff
PCI: Bridge: 0000:00:1c.2
  IO window: 6000-7fff
  MEM window: e8000000-e9ffffff
  PREFETCH window: e4100000-e41fffff
PCI: Bridge: 0000:00:1c.3
  IO window: 8000-9fff
  MEM window: ea000000-ebffffff
  PREFETCH window: e4200000-e42fffff
PCI: Bus 22, cardbus bridge: 0000:15:00.0
  IO window: 0000a000-0000a0ff
  IO window: 0000a400-0000a4ff
  PREFETCH window: e0000000-e3ffffff
  MEM window: 50000000-53ffffff
PCI: Bridge: 0000:00:1e.0
  IO window: a000-dfff
  MEM window: e4300000-e7ffffff
  PREFETCH window: e0000000-e3ffffff
ACPI: PCI Interrupt 0000:00:01.0[A] -> GSI 16 (level, low) -> IRQ 16
PCI: Setting latency timer of device 0000:00:01.0 to 64
ACPI: PCI Interrupt 0000:00:1c.0[A] -> GSI 20 (level, low) -> IRQ 20
PCI: Setting latency timer of device 0000:00:1c.0 to 64
ACPI: PCI Interrupt 0000:00:1c.1[B] -> GSI 21 (level, low) -> IRQ 21
PCI: Setting latency timer of device 0000:00:1c.1 to 64
ACPI: PCI Interrupt 0000:00:1c.2[C] -> GSI 22 (level, low) -> IRQ 22
PCI: Setting latency timer of device 0000:00:1c.2 to 64
ACPI: PCI Interrupt 0000:00:1c.3[D] -> GSI 23 (level, low) -> IRQ 23
PCI: Setting latency timer of device 0000:00:1c.3 to 64
PCI: Enabling device 0000:00:1e.0 (0005 -> 0007)
PCI: Setting latency timer of device 0000:00:1e.0 to 64
ACPI: PCI Interrupt 0000:15:00.0[A] -> GSI 16 (level, low) -> IRQ 16
NET: Registered protocol family 2
Time: hpet clocksource has been installed.
IP route cache hash table entries: 32768 (order: 6, 262144 bytes)
TCP established hash table entries: 131072 (order: 9, 3145728 bytes)
TCP bind hash table entries: 65536 (order: 8, 1048576 bytes)
TCP: Hash tables configured (established 131072 bind 65536)
TCP reno registered
Simple Boot Flag at 0x35 set to 0x1
io scheduler noop registered
io scheduler cfq registered (default)
pci 0000:00:1d.0: uhci_check_and_reset_hc: legsup = 0x2000
pci 0000:00:1d.0: Performing full reset
pci 0000:00:1d.1: uhci_check_and_reset_hc: legsup = 0x2000
pci 0000:00:1d.1: Performing full reset
pci 0000:00:1d.2: uhci_check_and_reset_hc: legsup = 0x2000
pci 0000:00:1d.2: Performing full reset
pci 0000:00:1d.3: uhci_check_and_reset_hc: legsup = 0x2000
pci 0000:00:1d.3: Performing full reset
Boot video device is 0000:01:00.0
PCI: Setting latency timer of device 0000:00:01.0 to 64
assign_interrupt_mode Found MSI capability
Allocate Port Service[0000:00:01.0:pcie00]
PCI: Setting latency timer of device 0000:00:1c.0 to 64
assign_interrupt_mode Found MSI capability
Allocate Port Service[0000:00:1c.0:pcie00]
Allocate Port Service[0000:00:1c.0:pcie02]
PCI: Setting latency timer of device 0000:00:1c.1 to 64
assign_interrupt_mode Found MSI capability
Allocate Port Service[0000:00:1c.1:pcie00]
Allocate Port Service[0000:00:1c.1:pcie02]
PCI: Setting latency timer of device 0000:00:1c.2 to 64
assign_interrupt_mode Found MSI capability
Allocate Port Service[0000:00:1c.2:pcie00]
Allocate Port Service[0000:00:1c.2:pcie02]
PCI: Setting latency timer of device 0000:00:1c.3 to 64
assign_interrupt_mode Found MSI capability
Allocate Port Service[0000:00:1c.3:pcie00]
Allocate Port Service[0000:00:1c.3:pcie02]
vesafb: framebuffer at 0xd8000000, mapped to 0xffffc20000080000, using 2560k, total 16384k
vesafb: mode is 1280x1024x8, linelength=1280, pages=11
vesafb: scrolling: redraw
vesafb: Pseudocolor: size=8:8:8:8, shift=0:0:0:0
Console: switching to colour frame buffer device 160x64
fb0: VESA VGA frame buffer device
ACPI: AC Adapter [AC] (on-line)
ACPI: Battery Slot [BAT0] (battery absent)
input: Power Button (FF) as /class/input/input0
ACPI: Power Button (FF) [PWRF]
input: Lid Switch as /class/input/input1
ACPI: Lid Switch [LID]
input: Sleep Button (CM) as /class/input/input2
ACPI: Sleep Button (CM) [SLPB]
ACPI: SSDT 3FEF1D36, 0240 (r1  PmRef  Cpu0Ist      100 INTL 20050513)
ACPI: SSDT 3FEF1FFB, 065A (r1  PmRef  Cpu0Cst      100 INTL 20050513)
Monitor-Mwait will be used to enter C-1 state
Monitor-Mwait will be used to enter C-2 state
Monitor-Mwait will be used to enter C-3 state
ACPI: CPU0 (power states: C1[C1] C2[C2] C3[C3])
ACPI: Processor [CPU0] (supports 8 throttling states)
ACPI: SSDT 3FEF1C6E, 00C8 (r1  PmRef  Cpu1Ist      100 INTL 20050513)
ACPI: SSDT 3FEF1F76, 0085 (r1  PmRef  Cpu1Cst      100 INTL 20050513)
ACPI: CPU1 (power states: C1[C1] C2[C2] C3[C3])
ACPI: Processor [CPU1] (supports 8 throttling states)
ACPI: Thermal Zone [THM0] (40 C)
ACPI: Thermal Zone [THM1] (41 C)
Real Time Clock Driver v1.12ac
hpet_resources: 0xfed00000 is busy
Linux agpgart interface v0.102 (c) Dave Jones
Serial: 8250/16550 driver $Revision: 1.90 $ 4 ports, IRQ sharing disabled
loop: module loaded
thinkpad_acpi: ThinkPad ACPI Extras v0.14
thinkpad_acpi: http://ibm-acpi.sf.net/
thinkpad_acpi: ThinkPad EC firmware 79HT50WW-1.07
tun: Universal TUN/TAP device driver, 1.6
tun: (C) 1999-2004 Max Krasnyansky <maxk@qualcomm.com>
Uniform Multi-Platform E-IDE driver Revision: 7.00alpha2
ide: Assuming 33MHz system bus speed for PIO modes; override with idebus=xx
ICH7: IDE controller at PCI slot 0000:00:1f.1
ACPI: PCI Interrupt 0000:00:1f.1[C] -> GSI 16 (level, low) -> IRQ 16
ICH7: chipset revision 2
ICH7: not 100% native mode: will probe irqs later
    ide0: BM-DMA at 0x1880-0x1887, BIOS settings: hda:DMA, hdb:pio
Probing IDE interface ide0...
hda: MATSHITADVD-RAM UJ-842, ATAPI CD/DVD-ROM drive
hda: selected mode 0x42
ide0 at 0x1f0-0x1f7,0x3f6 on irq 14
Probing IDE interface ide1...
hda: ATAPI 24X DVD-ROM DVD-R-RAM CD-R/RW drive, 2048kB Cache, UDMA(33)
Uniform CD-ROM driver Revision: 3.20
ahci 0000:00:1f.2: version 2.2
ACPI: PCI Interrupt 0000:00:1f.2[B] -> GSI 16 (level, low) -> IRQ 16
ahci 0000:00:1f.2: AHCI 0001.0100 32 slots 4 ports 1.5 Gbps 0xf impl SATA mode
ahci 0000:00:1f.2: flags: 64bit ncq pm led clo pio slum part 
PCI: Setting latency timer of device 0000:00:1f.2 to 64
scsi0 : ahci
scsi1 : ahci
scsi2 : ahci
scsi3 : ahci
ata1: SATA max UDMA/133 cmd 0xffffc2000003a500 ctl 0x0000000000000000 bmdma 0x0000000000000000 irq 0
ata2: SATA max UDMA/133 cmd 0xffffc2000003a580 ctl 0x0000000000000000 bmdma 0x0000000000000000 irq 0
ata3: SATA max UDMA/133 cmd 0xffffc2000003a600 ctl 0x0000000000000000 bmdma 0x0000000000000000 irq 0
ata4: SATA max UDMA/133 cmd 0xffffc2000003a680 ctl 0x0000000000000000 bmdma 0x0000000000000000 irq 0
ata1: SATA link up 1.5 Gbps (SStatus 113 SControl 300)
ata1.00: ata_hpa_resize 1: sectors = 156301488, hpa_sectors = 156301488
ata1.00: ATA-7: HTS541080G9SA00, MB4IC60R, max UDMA/100
ata1.00: 156301488 sectors, multi 16: LBA48 
ata1.00: ata_hpa_resize 1: sectors = 156301488, hpa_sectors = 156301488
ata1.00: configured for UDMA/100
ata2: SATA link down (SStatus 0 SControl 0)
ata3: SATA link down (SStatus 0 SControl 0)
ata4: SATA link down (SStatus 0 SControl 0)
scsi 0:0:0:0: Direct-Access     ATA      HTS541080G9SA00  MB4I PQ: 0 ANSI: 5
sd 0:0:0:0: [sda] 156301488 512-byte hardware sectors (80026 MB)
sd 0:0:0:0: [sda] Write Protect is off
sd 0:0:0:0: [sda] Mode Sense: 00 3a 00 00
sd 0:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
sd 0:0:0:0: [sda] 156301488 512-byte hardware sectors (80026 MB)
sd 0:0:0:0: [sda] Write Protect is off
sd 0:0:0:0: [sda] Mode Sense: 00 3a 00 00
sd 0:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
 sda: sda1 sda2 sda3 < sda5 >
sd 0:0:0:0: [sda] Attached SCSI disk
sd 0:0:0:0: Attached scsi generic sg0 type 0
usbmon: debugfs is not available
ehci_hcd: block sizes: qh 160 qtd 96 itd 192 sitd 96
ACPI: PCI Interrupt 0000:00:1d.7[D] -> GSI 19 (level, low) -> IRQ 19
PCI: Setting latency timer of device 0000:00:1d.7 to 64
ehci_hcd 0000:00:1d.7: EHCI Host Controller
ehci_hcd 0000:00:1d.7: new USB bus registered, assigned bus number 1
ehci_hcd 0000:00:1d.7: reset hcs_params 0x104208 dbg=1 cc=4 pcc=2 ordered !ppc ports=8
ehci_hcd 0000:00:1d.7: reset hcc_params 6871 thresh 7 uframes 1024 64 bit addr
ehci_hcd 0000:00:1d.7: debug port 1
PCI: cache line size of 32 is not supported by device 0000:00:1d.7
ehci_hcd 0000:00:1d.7: supports USB remote wakeup
ehci_hcd 0000:00:1d.7: irq 19, io mem 0xee404000
ehci_hcd 0000:00:1d.7: reset command 080022 (park)=0 ithresh=8 Async period=1024 Reset HALT
ehci_hcd 0000:00:1d.7: init command 010001 (park)=0 ithresh=1 period=1024 RUN
ehci_hcd 0000:00:1d.7: USB 2.0 started, EHCI 1.00, driver 10 Dec 2004
usb usb1: default language 0x0409
usb usb1: new device strings: Mfr=3, Product=2, SerialNumber=1
usb usb1: Product: EHCI Host Controller
usb usb1: Manufacturer: Linux 2.6.22-rc6 ehci_hcd
usb usb1: SerialNumber: 0000:00:1d.7
usb usb1: uevent
usb usb1: usb_probe_device
usb usb1: configuration #1 chosen from 1 choice
usb usb1: adding 1-0:1.0 (config #1, interface 0)
usb 1-0:1.0: uevent
usb 1-0:1.0: uevent
hub 1-0:1.0: usb_probe_interface
hub 1-0:1.0: usb_probe_interface - got id
hub 1-0:1.0: USB hub found
hub 1-0:1.0: 8 ports detected
hub 1-0:1.0: standalone hub
hub 1-0:1.0: no power switching (usb 1.0)
hub 1-0:1.0: individual port over-current protection
hub 1-0:1.0: Single TT
hub 1-0:1.0: TT requires at most 8 FS bit times (666 ns)
hub 1-0:1.0: power on to power good time: 20ms
hub 1-0:1.0: local power source is good
hub 1-0:1.0: trying to enable port power on non-switchable hub
hub 1-0:1.0: state 7 ports 8 chg 0000 evt 0000
ehci_hcd 0000:00:1d.7: GetStatus port 1 status 001403 POWER sig=k CSC CONNECT
USB Universal Host Controller Interface driver v3.0
hub 1-0:1.0: port 1, status 0501, change 0001, 480 Mb/s
ACPI: PCI Interrupt 0000:00:1d.0[A] -> GSI 16 (level, low) -> IRQ 16
PCI: Setting latency timer of device 0000:00:1d.0 to 64
uhci_hcd 0000:00:1d.0: UHCI Host Controller
uhci_hcd 0000:00:1d.0: new USB bus registered, assigned bus number 2
uhci_hcd 0000:00:1d.0: detected 2 ports
uhci_hcd 0000:00:1d.0: uhci_check_and_reset_hc: cmd = 0x0000
uhci_hcd 0000:00:1d.0: Performing full reset
uhci_hcd 0000:00:1d.0: irq 16, io base 0x00001800
usb usb2: default language 0x0409
usb usb2: new device strings: Mfr=3, Product=2, SerialNumber=1
usb usb2: Product: UHCI Host Controller
usb usb2: Manufacturer: Linux 2.6.22-rc6 uhci_hcd
usb usb2: SerialNumber: 0000:00:1d.0
usb usb2: uevent
usb usb2: usb_probe_device
usb usb2: configuration #1 chosen from 1 choice
usb usb2: adding 2-0:1.0 (config #1, interface 0)
usb 2-0:1.0: uevent
usb 2-0:1.0: uevent
hub 2-0:1.0: usb_probe_interface
hub 2-0:1.0: usb_probe_interface - got id
hub 2-0:1.0: USB hub found
hub 2-0:1.0: 2 ports detected
hub 2-0:1.0: standalone hub
hub 2-0:1.0: no power switching (usb 1.0)
hub 2-0:1.0: individual port over-current protection
hub 2-0:1.0: power on to power good time: 2ms
hub 2-0:1.0: local power source is good
hub 2-0:1.0: trying to enable port power on non-switchable hub
hub 1-0:1.0: debounce: port 1: total 100ms stable 100ms status 0x501
ehci_hcd 0000:00:1d.7: port 1 low speed --> companion
ACPI: PCI Interrupt 0000:00:1d.1[B] -> GSI 17 (level, low) -> IRQ 17
PCI: Setting latency timer of device 0000:00:1d.1 to 64
uhci_hcd 0000:00:1d.1: UHCI Host Controller
uhci_hcd 0000:00:1d.1: new USB bus registered, assigned bus number 3
uhci_hcd 0000:00:1d.1: detected 2 ports
uhci_hcd 0000:00:1d.1: uhci_check_and_reset_hc: cmd = 0x0000
uhci_hcd 0000:00:1d.1: Performing full reset
uhci_hcd 0000:00:1d.1: irq 17, io base 0x00001820
usb usb3: default language 0x0409
usb usb3: new device strings: Mfr=3, Product=2, SerialNumber=1
usb usb3: Product: UHCI Host Controller
usb usb3: Manufacturer: Linux 2.6.22-rc6 uhci_hcd
usb usb3: SerialNumber: 0000:00:1d.1
usb usb3: uevent
usb usb3: usb_probe_device
usb usb3: configuration #1 chosen from 1 choice
usb usb3: adding 3-0:1.0 (config #1, interface 0)
ehci_hcd 0000:00:1d.7: GetStatus port 1 status 003002 POWER OWNER sig=se0 CSC
usb 3-0:1.0: uevent
usb 3-0:1.0: uevent
ehci_hcd 0000:00:1d.7: GetStatus port 8 status 001803 POWER sig=j CSC CONNECT
hub 1-0:1.0: port 8, status 0501, change 0001, 480 Mb/s
hub 3-0:1.0: usb_probe_interface
hub 3-0:1.0: usb_probe_interface - got id
hub 3-0:1.0: USB hub found
hub 3-0:1.0: 2 ports detected
hub 3-0:1.0: standalone hub
hub 3-0:1.0: no power switching (usb 1.0)
hub 3-0:1.0: individual port over-current protection
hub 3-0:1.0: power on to power good time: 2ms
hub 3-0:1.0: local power source is good
hub 3-0:1.0: trying to enable port power on non-switchable hub
ACPI: PCI Interrupt 0000:00:1d.2[C] -> GSI 18 (level, low) -> IRQ 18
PCI: Setting latency timer of device 0000:00:1d.2 to 64
uhci_hcd 0000:00:1d.2: UHCI Host Controller
uhci_hcd 0000:00:1d.2: new USB bus registered, assigned bus number 4
uhci_hcd 0000:00:1d.2: detected 2 ports
uhci_hcd 0000:00:1d.2: uhci_check_and_reset_hc: cmd = 0x0000
uhci_hcd 0000:00:1d.2: Performing full reset
uhci_hcd 0000:00:1d.2: irq 18, io base 0x00001840
usb usb4: default language 0x0409
usb usb4: new device strings: Mfr=3, Product=2, SerialNumber=1
usb usb4: Product: UHCI Host Controller
usb usb4: Manufacturer: Linux 2.6.22-rc6 uhci_hcd
usb usb4: SerialNumber: 0000:00:1d.2
usb usb4: uevent
usb usb4: usb_probe_device
usb usb4: configuration #1 chosen from 1 choice
usb usb4: adding 4-0:1.0 (config #1, interface 0)
usb 4-0:1.0: uevent
usb 4-0:1.0: uevent
hub 4-0:1.0: usb_probe_interface
hub 4-0:1.0: usb_probe_interface - got id
hub 4-0:1.0: USB hub found
hub 4-0:1.0: 2 ports detected
hub 1-0:1.0: debounce: port 8: total 100ms stable 100ms status 0x501
hub 4-0:1.0: standalone hub
hub 4-0:1.0: no power switching (usb 1.0)
hub 4-0:1.0: individual port over-current protection
hub 4-0:1.0: power on to power good time: 2ms
hub 4-0:1.0: local power source is good
hub 4-0:1.0: trying to enable port power on non-switchable hub
ehci_hcd 0000:00:1d.7: port 8 full speed --> companion
ehci_hcd 0000:00:1d.7: GetStatus port 8 status 003801 POWER OWNER sig=j CONNECT
hub 1-0:1.0: port 8 not reset yet, waiting 50ms
ACPI: PCI Interrupt 0000:00:1d.3[D] -> GSI 19 (level, low) -> IRQ 19
PCI: Setting latency timer of device 0000:00:1d.3 to 64
uhci_hcd 0000:00:1d.3: UHCI Host Controller
ehci_hcd 0000:00:1d.7: GetStatus port 8 status 003002 POWER OWNER sig=se0 CSC
hub 1-0:1.0: state 7 ports 8 chg 0000 evt 0100
hub 2-0:1.0: state 7 ports 2 chg 0000 evt 0002
uhci_hcd 0000:00:1d.0: port 1 portsc 01ab,00
hub 2-0:1.0: port 1, status 0301, change 0003, 1.5 Mb/s
uhci_hcd 0000:00:1d.3: new USB bus registered, assigned bus number 5
uhci_hcd 0000:00:1d.3: detected 2 ports
uhci_hcd 0000:00:1d.3: uhci_check_and_reset_hc: cmd = 0x0000
uhci_hcd 0000:00:1d.3: Performing full reset
uhci_hcd 0000:00:1d.3: irq 19, io base 0x00001860
usb usb5: default language 0x0409
usb usb5: new device strings: Mfr=3, Product=2, SerialNumber=1
usb usb5: Product: UHCI Host Controller
usb usb5: Manufacturer: Linux 2.6.22-rc6 uhci_hcd
usb usb5: SerialNumber: 0000:00:1d.3
usb usb5: uevent
usb usb5: usb_probe_device
usb usb5: configuration #1 chosen from 1 choice
usb usb5: adding 5-0:1.0 (config #1, interface 0)
usb 5-0:1.0: uevent
usb 5-0:1.0: uevent
hub 5-0:1.0: usb_probe_interface
hub 5-0:1.0: usb_probe_interface - got id
hub 5-0:1.0: USB hub found
hub 5-0:1.0: 2 ports detected
hub 5-0:1.0: standalone hub
hub 5-0:1.0: no power switching (usb 1.0)
hub 5-0:1.0: individual port over-current protection
hub 5-0:1.0: power on to power good time: 2ms
hub 5-0:1.0: local power source is good
hub 5-0:1.0: trying to enable port power on non-switchable hub
hub 2-0:1.0: debounce: port 1: total 100ms stable 100ms status 0x301
PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12
serio: i8042 KBD port at 0x60,0x64 irq 1
serio: i8042 AUX port at 0x60,0x64 irq 12
mice: PS/2 mouse device common for all mice
input: AT Translated Set 2 keyboard as /class/input/input3
usb 2-1: new low speed USB device using uhci_hcd and address 2
usb 2-1: skipped 1 descriptor after interface
usb 2-1: default language 0x0409
usb 2-1: new device strings: Mfr=1, Product=2, SerialNumber=0
usb 2-1: Product: Basic Optical Mouse
usb 2-1: Manufacturer: Microsoft
usb 2-1: uevent
usb 2-1: usb_probe_device
usb 2-1: configuration #1 chosen from 1 choice
usb 2-1: adding 2-1:1.0 (config #1, interface 0)
usb 2-1:1.0: uevent
usb 2-1:1.0: uevent
hub 3-0:1.0: state 7 ports 2 chg 0000 evt 0000
hub 4-0:1.0: state 7 ports 2 chg 0000 evt 0000
hub 5-0:1.0: state 7 ports 2 chg 0000 evt 0004
uhci_hcd 0000:00:1d.3: port 2 portsc 009b,00
hub 5-0:1.0: port 2, status 0101, change 0003, 12 Mb/s
hub 5-0:1.0: debounce: port 2: total 100ms stable 100ms status 0x101
usb 5-2: new full speed USB device using uhci_hcd and address 2
Synaptics Touchpad, model: 1, fw: 6.2, id: 0x81a0b1, caps: 0xa04793/0x300000
serio: Synaptics pass-through port at isa0060/serio1/input0
usb 5-2: ep0 maxpacket = 8
input: SynPS/2 Synaptics TouchPad as /class/input/input4
usb 5-2: default language 0x0409
usb 5-2: new device strings: Mfr=1, Product=2, SerialNumber=0
usb 5-2: Product: Biometric Coprocessor
usb 5-2: Manufacturer: STMicroelectronics
usb usb3: suspend_rh (auto-stop)
usb 5-2: uevent
usb 5-2: usb_probe_device
usb 5-2: configuration #1 chosen from 1 choice
usb 5-2: adding 5-2:1.0 (config #1, interface 0)
usb 5-2:1.0: uevent
usb 5-2:1.0: uevent
usbhid 2-1:1.0: usb_probe_interface
usbhid 2-1:1.0: usb_probe_interface - got id
input: Microsoft Basic Optical Mouse as /class/input/input5
input: USB HID v1.10 Mouse [Microsoft Basic Optical Mouse] on usb-0000:00:1d.0-1
usbcore: registered new interface driver usbhid
drivers/hid/usbhid/hid-core.c: v2.6:USB HID core driver
Advanced Linux Sound Architecture Driver Version 1.0.14 (Thu May 31 09:03:25 2007 UTC).
ACPI: PCI Interrupt 0000:00:1b.0[B] -> GSI 17 (level, low) -> IRQ 17
PCI: Setting latency timer of device 0000:00:1b.0 to 64
usb usb4: suspend_rh (auto-stop)
ALSA device list:
  #0: HDA Intel at 0xee400000 irq 17
Netfilter messages via NETLINK v0.30.
nf_conntrack version 0.5.0 (4091 buckets, 32728 max)
ip_tables: (C) 2000-2006 Netfilter Core Team
TCP cubic registered
NET: Registered protocol family 1
NET: Registered protocol family 17
IBM TrackPoint firmware: 0x0e, buttons: 3/3
input: TPPS/2 IBM TrackPoint as /class/input/input6
kjournald starting.  Commit interval 5 seconds
EXT3-fs: mounted filesystem with ordered data mode.
VFS: Mounted root (ext3 filesystem) readonly.
Freeing unused kernel memory: 188k freed
usb usb2: uevent
usb 2-0:1.0: uevent
usb 2-0:1.0: uevent
usb 2-1: uevent
usb 2-1:1.0: uevent
usb 2-1:1.0: uevent
usb usb3: uevent
usb 3-0:1.0: uevent
usb 3-0:1.0: uevent
usb usb4: uevent
usb 4-0:1.0: uevent
usb 4-0:1.0: uevent
usb usb5: uevent
usb 5-0:1.0: uevent
usb 5-0:1.0: uevent
usb 5-2: uevent
usb 5-2:1.0: uevent
usb 5-2:1.0: uevent
usb usb1: uevent
usb 1-0:1.0: uevent
usb 1-0:1.0: uevent
Intel(R) PRO/1000 Network Driver - version 7.3.20-k2
Copyright (c) 1999-2006 Intel Corporation.
ACPI: PCI Interrupt 0000:02:00.0[A] -> GSI 16 (level, low) -> IRQ 16
PCI: Setting latency timer of device 0000:02:00.0 to 64
e1000: 0000:02:00.0: e1000_validate_option: Receive Interrupt Delay set to 32
e1000: 0000:02:00.0: e1000_probe: (PCI Express:2.5Gb/s:Width x1) 00:16:41:e3:2c:76
e1000: eth0: e1000_probe: Intel(R) PRO/1000 Network Connection
Adding 1542196k swap on /dev/sda1.  Priority:-1 extents:1 across:1542196k
EXT3 FS on sda2, internal journal
fuse init (API version 7.8)
e1000: eth0: e1000_watchdog: NIC Link is Up 100 Mbps Full Duplex, Flow Control: RX/TX
e1000: eth0: e1000_watchdog: 10/100 speed: disabling TSO
uhci_hcd 0000:00:1d.0: reserve dev 2 ep81-INT, period 8, phase 4, 93 us
uhci_hcd 0000:00:1d.0: release dev 2 ep81-INT, period 8, phase 4, 93 us
uhci_hcd 0000:00:1d.0: reserve dev 2 ep81-INT, period 8, phase 4, 93 us
uhci_hcd 0000:00:1d.0: release dev 2 ep81-INT, period 8, phase 4, 93 us
uhci_hcd 0000:00:1d.0: reserve dev 2 ep81-INT, period 8, phase 4, 93 us
uhci_hcd 0000:00:1d.0: release dev 2 ep81-INT, period 8, phase 4, 93 us
IA-32 Microcode Update Driver: v1.14a <tigran@aivazian.fsnet.co.uk>
uhci_hcd 0000:00:1d.0: reserve dev 2 ep81-INT, period 8, phase 4, 93 us
uhci_hcd 0000:00:1d.0: release dev 2 ep81-INT, period 8, phase 4, 93 us
uhci_hcd 0000:00:1d.0: reserve dev 2 ep81-INT, period 8, phase 4, 93 us
uhci_hcd 0000:00:1d.0: release dev 2 ep81-INT, period 8, phase 4, 93 us
Suspending console(s)
hub 5-0:1.0: hub_suspend
usb usb5: suspend_rh
hub 4-0:1.0: hub_suspend
usb usb4: suspend_rh
hub 3-0:1.0: hub_suspend
usb usb3: suspend_rh
hub 2-0:1.0: hub_suspend
usb usb2: suspend_rh
hub 1-0:1.0: hub_suspend
ehci_hcd 0000:00:1d.7: suspend root hub
sd 0:0:0:0: [sda] Synchronizing SCSI cache
sd 0:0:0:0: [sda] Stopping disk
ACPI: PCI interrupt for device 0000:02:00.0 disabled
ACPI: PCI interrupt for device 0000:00:1d.7 disabled
ehci_hcd 0000:00:1d.7: --> PCI D3/wakeup
uhci_hcd 0000:00:1d.3: uhci_suspend
ACPI: PCI interrupt for device 0000:00:1d.3 disabled
uhci_hcd 0000:00:1d.3: --> PCI D0/legacy
uhci_hcd 0000:00:1d.2: uhci_suspend
ACPI: PCI interrupt for device 0000:00:1d.2 disabled
uhci_hcd 0000:00:1d.2: --> PCI D0/legacy
uhci_hcd 0000:00:1d.1: uhci_suspend
ACPI: PCI interrupt for device 0000:00:1d.1 disabled
uhci_hcd 0000:00:1d.1: --> PCI D0/legacy
uhci_hcd 0000:00:1d.0: uhci_suspend
ACPI: PCI interrupt for device 0000:00:1d.0 disabled
uhci_hcd 0000:00:1d.0: --> PCI D0/legacy
ACPI: PCI interrupt for device 0000:00:1b.0 disabled
Disabling non-boot CPUs ...
CPU 1 is now offline
SMP alternatives: switching to UP code
CPU1 is down
Extended CMOS year: 2000
Back to C!
Extended CMOS year: 2000
Enabling non-boot CPUs ...
SMP alternatives: switching to SMP code
Booting processor 1/2 APIC 0x1
Initializing CPU#1
Calibrating delay using timer specific routine.. 3657.62 BogoMIPS (lpj=7315250)
CPU: L1 I cache: 32K, L1 D cache: 32K
CPU: L2 cache: 2048K
CPU: Physical Processor ID: 0
CPU: Processor Core ID: 1
Intel(R) Core(TM)2 CPU         T5600  @ 1.83GHz stepping 06
CPU1 is up
PM: Writing back config space on device 0000:00:01.0 at offset 7 (was 20002020, writing 2020)
PM: Writing back config space on device 0000:00:01.0 at offset 1 (was 100107, writing 100507)
PCI: Setting latency timer of device 0000:00:01.0 to 64
PM: Writing back config space on device 0000:00:1b.0 at offset 1 (was 100106, writing 100102)
ACPI: PCI Interrupt 0000:00:1b.0[B] -> GSI 17 (level, low) -> IRQ 17
PCI: Setting latency timer of device 0000:00:1b.0 to 64
PM: Writing back config space on device 0000:00:1c.0 at offset 1 (was 180107, writing 100507)
PCI: Setting latency timer of device 0000:00:1c.0 to 64
PM: Writing back config space on device 0000:00:1c.1 at offset 1 (was 100107, writing 100507)
PCI: Setting latency timer of device 0000:00:1c.1 to 64
PM: Writing back config space on device 0000:00:1c.2 at offset 7 (was 20007060, writing 7060)
PM: Writing back config space on device 0000:00:1c.2 at offset 1 (was 100107, writing 100507)
PCI: Setting latency timer of device 0000:00:1c.2 to 64
PM: Writing back config space on device 0000:00:1c.3 at offset f (was 40400, writing 4040b)
PM: Writing back config space on device 0000:00:1c.3 at offset 9 (was 10001, writing e421e421)
PM: Writing back config space on device 0000:00:1c.3 at offset 8 (was 0, writing ebf0ea00)
PM: Writing back config space on device 0000:00:1c.3 at offset 7 (was 20000000, writing 9080)
PM: Writing back config space on device 0000:00:1c.3 at offset 3 (was 810000, writing 810010)
PM: Writing back config space on device 0000:00:1c.3 at offset 1 (was 100000, writing 100507)
PCI: Setting latency timer of device 0000:00:1c.3 to 64
uhci_hcd 0000:00:1d.0: PCI legacy resume
ACPI: PCI Interrupt 0000:00:1d.0[A] -> GSI 16 (level, low) -> IRQ 16
PCI: Setting latency timer of device 0000:00:1d.0 to 64
uhci_hcd 0000:00:1d.0: uhci_resume
uhci_hcd 0000:00:1d.0: uhci_check_and_reset_hc: legsup = 0x2f00
uhci_hcd 0000:00:1d.0: Performing full reset
usb usb2: root hub lost power or was reset
usb usb2: suspend_rh
hub 2-0:1.0: hub_resume
usb usb2: wakeup_rh
uhci_hcd 0000:00:1d.1: PCI legacy resume
ACPI: PCI Interrupt 0000:00:1d.1[B] -> GSI 17 (level, low) -> IRQ 17
PCI: Setting latency timer of device 0000:00:1d.1 to 64
uhci_hcd 0000:00:1d.1: uhci_resume
uhci_hcd 0000:00:1d.1: uhci_check_and_reset_hc: legsup = 0x2000
uhci_hcd 0000:00:1d.1: Performing full reset
usb usb3: root hub lost power or was reset
usb usb3: suspend_rh
uhci_hcd 0000:00:1d.2: PCI legacy resume
ACPI: PCI Interrupt 0000:00:1d.2[C] -> GSI 18 (level, low) -> IRQ 18
PCI: Setting latency timer of device 0000:00:1d.2 to 64
uhci_hcd 0000:00:1d.2: uhci_resume
uhci_hcd 0000:00:1d.2: uhci_check_and_reset_hc: legsup = 0x2000
uhci_hcd 0000:00:1d.2: Performing full reset
usb usb4: root hub lost power or was reset
usb usb4: suspend_rh
uhci_hcd 0000:00:1d.3: PCI legacy resume
ACPI: PCI Interrupt 0000:00:1d.3[D] -> GSI 19 (level, low) -> IRQ 19
PCI: Setting latency timer of device 0000:00:1d.3 to 64
uhci_hcd 0000:00:1d.3: uhci_resume
uhci_hcd 0000:00:1d.3: uhci_check_and_reset_hc: legsup = 0x2000
uhci_hcd 0000:00:1d.3: Performing full reset
usb usb5: root hub lost power or was reset
usb usb5: suspend_rh
ehci_hcd 0000:00:1d.7: PCI D0, from previous PCI D3
ACPI: PCI Interrupt 0000:00:1d.7[D] -> GSI 19 (level, low) -> IRQ 19
PCI: Setting latency timer of device 0000:00:1d.7 to 64
PM: Writing back config space on device 0000:00:1e.0 at offset 1 (was 100005, writing 100007)
PCI: Setting latency timer of device 0000:00:1e.0 to 64
ACPI: PCI Interrupt 0000:00:1f.1[C] -> GSI 16 (level, low) -> IRQ 16
PM: Writing back config space on device 0000:00:1f.2 at offset 1 (was 2b00007, writing 2b00407)
PCI: Setting latency timer of device 0000:00:1f.2 to 64
hub 5-0:1.0: hub_resume
usb usb5: wakeup_rh
hub 2-0:1.0: state 7 ports 2 chg 0002 evt 0000
uhci_hcd 0000:00:1d.0: port 1 portsc 008a,00
hub 2-0:1.0: port 1, status 0100, change 0003, 12 Mb/s
usb 2-1: USB disconnect, address 2
usb 2-1: unregistering device
usb 2-1: usb_disable_device nuking all URBs
usb 2-1: unregistering interface 2-1:1.0
usb_endpoint usbdev2.2_ep81: ep_device_release called for usbdev2.2_ep81
usb 2-1:1.0: uevent
usb 2-1:1.0: uevent
usb_endpoint usbdev2.2_ep00: ep_device_release called for usbdev2.2_ep00
usb 2-1: uevent
hub 1-0:1.0: hub_resume
ehci_hcd 0000:00:1d.7: resume root hub
hub 2-0:1.0: debounce: port 1: total 100ms stable 100ms status 0x100
hub 5-0:1.0: state 7 ports 2 chg 0004 evt 0004
uhci_hcd 0000:00:1d.3: port 2 portsc 008a,00
hub 5-0:1.0: port 2, status 0100, change 0003, 12 Mb/s
usb 5-2: USB disconnect, address 2
usb 5-2: unregistering device
usb 5-2: usb_disable_device nuking all URBs
usb 5-2: unregistering interface 5-2:1.0
usb_endpoint usbdev5.2_ep81: ep_device_release called for usbdev5.2_ep81
usb_endpoint usbdev5.2_ep02: ep_device_release called for usbdev5.2_ep02
usb_endpoint usbdev5.2_ep83: ep_device_release called for usbdev5.2_ep83
usb 5-2:1.0: uevent
usb 5-2:1.0: uevent
usb_endpoint usbdev5.2_ep00: ep_device_release called for usbdev5.2_ep00
usb 5-2: uevent
hub 5-0:1.0: debounce: port 2: total 100ms stable 100ms status 0x100
hub 1-0:1.0: state 7 ports 8 chg 0000 evt 0102
ehci_hcd 0000:00:1d.7: GetStatus port 1 status 001403 POWER sig=k CSC CONNECT
hub 1-0:1.0: port 1, status 0501, change 0001, 480 Mb/s
hub 1-0:1.0: debounce: port 1: total 100ms stable 100ms status 0x501
ehci_hcd 0000:00:1d.7: port 1 low speed --> companion
ehci_hcd 0000:00:1d.7: GetStatus port 1 status 003002 POWER OWNER sig=se0 CSC
ehci_hcd 0000:00:1d.7: GetStatus port 8 status 001803 POWER sig=j CSC CONNECT
hub 1-0:1.0: port 8, status 0501, change 0001, 480 Mb/s
hub 1-0:1.0: debounce: port 8: total 100ms stable 100ms status 0x501
ehci_hcd 0000:00:1d.7: port 8 full speed --> companion
ehci_hcd 0000:00:1d.7: GetStatus port 8 status 003801 POWER OWNER sig=j CONNECT
hub 1-0:1.0: port 8 not reset yet, waiting 50ms
ehci_hcd 0000:00:1d.7: GetStatus port 8 status 003002 POWER OWNER sig=se0 CSC
hub 1-0:1.0: state 7 ports 8 chg 0000 evt 0100
hub 2-0:1.0: state 7 ports 2 chg 0000 evt 0002
uhci_hcd 0000:00:1d.0: port 1 portsc 01a3,00
hub 2-0:1.0: port 1, status 0301, change 0001, 1.5 Mb/s
hub 2-0:1.0: debounce: port 1: total 100ms stable 100ms status 0x301
usb 2-1: new low speed USB device using uhci_hcd and address 3
PM: Writing back config space on device 0000:01:00.0 at offset 1 (was 100103, writing 100107)
PM: Writing back config space on device 0000:02:00.0 at offset 1 (was 100107, writing 100507)
ACPI: PCI Interrupt 0000:02:00.0[A] -> GSI 16 (level, low) -> IRQ 16
PCI: Setting latency timer of device 0000:02:00.0 to 64
PM: Writing back config space on device 0000:15:00.0 at offset f (was 34001ff, writing 3c0010b)
PM: Writing back config space on device 0000:15:00.0 at offset e (was 0, writing a4fc)
PM: Writing back config space on device 0000:15:00.0 at offset d (was 0, writing a400)
PM: Writing back config space on device 0000:15:00.0 at offset c (was 0, writing a0fc)
PM: Writing back config space on device 0000:15:00.0 at offset b (was 0, writing a000)
PM: Writing back config space on device 0000:15:00.0 at offset a (was 0, writing 53fff000)
PM: Writing back config space on device 0000:15:00.0 at offset 9 (was 0, writing 50000000)
PM: Writing back config space on device 0000:15:00.0 at offset 8 (was 0, writing e3fff000)
PM: Writing back config space on device 0000:15:00.0 at offset 7 (was 0, writing e0000000)
PM: Writing back config space on device 0000:15:00.0 at offset 6 (was 0, writing b0171615)
PM: Writing back config space on device 0000:15:00.0 at offset 4 (was 0, writing e4300000)
PM: Writing back config space on device 0000:15:00.0 at offset 3 (was 20000, writing 2a808)
PM: Writing back config space on device 0000:15:00.0 at offset 1 (was 2100000, writing 2100007)
ACPI: PCI Interrupt 0000:15:00.0[A] -> GSI 16 (level, low) -> IRQ 16
usb 2-1: skipped 1 descriptor after interface
hda: selected mode 0x42
usb 2-1: default language 0x0409
sd 0:0:0:0: [sda] Starting disk
usb 2-1: new device strings: Mfr=1, Product=2, SerialNumber=0
usb 2-1: Product: Basic Optical Mouse
usb 2-1: Manufacturer: Microsoft
usb 2-1: uevent
usb 2-1: usb_probe_device
usb 2-1: configuration #1 chosen from 1 choice
usb 2-1: adding 2-1:1.0 (config #1, interface 0)
usb 2-1:1.0: uevent
usb 2-1:1.0: uevent
usbhid 2-1:1.0: usb_probe_interface
usbhid 2-1:1.0: usb_probe_interface - got id
input: Microsoft Basic Optical Mouse as /class/input/input7
input: USB HID v1.10 Mouse [Microsoft Basic Optical Mouse] on usb-0000:00:1d.0-1
hub 5-0:1.0: state 7 ports 2 chg 0000 evt 0004
uhci_hcd 0000:00:1d.3: port 2 portsc 0093,00
hub 5-0:1.0: port 2, status 0101, change 0001, 12 Mb/s
uhci_hcd 0000:00:1d.0: reserve dev 3 ep81-INT, period 8, phase 4, 93 us
uhci_hcd 0000:00:1d.0: release dev 3 ep81-INT, period 8, phase 4, 93 us
uhci_hcd 0000:00:1d.0: reserve dev 3 ep81-INT, period 8, phase 4, 93 us
uhci_hcd 0000:00:1d.0: release dev 3 ep81-INT, period 8, phase 4, 93 us
hub 5-0:1.0: debounce: port 2: total 100ms stable 100ms status 0x101
ata2: SATA link down (SStatus 0 SControl 0)
ata3: SATA link down (SStatus 0 SControl 0)
ata4: SATA link down (SStatus 0 SControl 0)
usb 5-2: new full speed USB device using uhci_hcd and address 3
ata1: SATA link up 1.5 Gbps (SStatus 113 SControl 300)
usb 5-2: ep0 maxpacket = 8
ata1.00: ata_hpa_resize 1: sectors = 156301488, hpa_sectors = 156301488
ata1.00: ata_hpa_resize 1: sectors = 156301488, hpa_sectors = 156301488
ata1.00: configured for UDMA/100
usb 5-2: default language 0x0409
sd 0:0:0:0: [sda] 156301488 512-byte hardware sectors (80026 MB)
hub 3-0:1.0: hub_resume
usb usb3: wakeup_rh
sd 0:0:0:0: [sda] Write Protect is off
sd 0:0:0:0: [sda] Mode Sense: 00 3a 00 00
usb 5-2: new device strings: Mfr=1, Product=2, SerialNumber=0
usb 5-2: Product: Biometric Coprocessor
usb 5-2: Manufacturer: STMicroelectronics
usb 5-2: uevent
usb 5-2: usb_probe_device
usb 5-2: configuration #1 chosen from 1 choice
usb 5-2: adding 5-2:1.0 (config #1, interface 0)
usb 5-2:1.0: uevent
usb 5-2:1.0: uevent
sd 0:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
hub 3-0:1.0: state 7 ports 2 chg 0000 evt 0000
hub 4-0:1.0: hub_resume
usb usb4: wakeup_rh
hub 4-0:1.0: state 7 ports 2 chg 0000 evt 0000
usb usb3: suspend_rh (auto-stop)
usb usb4: suspend_rh (auto-stop)
e1000: eth0: e1000_watchdog: NIC Link is Up 100 Mbps Full Duplex, Flow Control: RX/TX
e1000: eth0: e1000_watchdog: 10/100 speed: disabling TSO
ata4: soft resetting port
ata4: SATA link down (SStatus 0 SControl 0)
ata4: EH complete
ata3: soft resetting port
ata3: SATA link down (SStatus 0 SControl 0)
ata3: EH complete
ata2: soft resetting port
ata2: SATA link down (SStatus 0 SControl 0)
ata2: EH complete
Uhhuh. NMI received for unknown reason a0.
You have some hardware problem, likely on the PCI bus.
Dazed and confused, but trying to continue
uhci_hcd 0000:00:1d.0: reserve dev 3 ep81-INT, period 8, phase 4, 93 us
uhci_hcd 0000:00:1d.0: release dev 3 ep81-INT, period 8, phase 4, 93 us
Suspending console(s)
hub 5-0:1.0: port 2 nyet suspended
hub 5-0:1.0: suspend error -16
suspend_device(): usb_suspend+0x0/0x1c() returns -16
Could not suspend device usb5: error -16
Some devices failed to suspend
ata4: soft resetting port
ata4: SATA link down (SStatus 0 SControl 0)
ata4: EH complete
ata3: soft resetting port
ata3: SATA link down (SStatus 0 SControl 0)
ata3: EH complete
ata2: soft resetting port
ata2: SATA link down (SStatus 0 SControl 0)
ata2: EH complete
uhci_hcd 0000:00:1d.0: reserve dev 3 ep81-INT, period 8, phase 4, 93 us
uhci_hcd 0000:00:1d.0: release dev 3 ep81-INT, period 8, phase 4, 93 us
Suspending console(s)
hub 5-0:1.0: port 2 nyet suspended
hub 5-0:1.0: suspend error -16
suspend_device(): usb_suspend+0x0/0x1c() returns -16
Could not suspend device usb5: error -16
Some devices failed to suspend
ata4: soft resetting port
ata4: SATA link down (SStatus 0 SControl 0)
ata4: EH complete
ata3: soft resetting port
ata3: SATA link down (SStatus 0 SControl 0)
ata3: EH complete
ata2: soft resetting port
ata2: SATA link down (SStatus 0 SControl 0)
ata2: EH complete
uhci_hcd 0000:00:1d.0: reserve dev 3 ep81-INT, period 8, phase 4, 93 us

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Alan Stern]

On Thu, 5 Jul 2007, Miklos Szeredi wrote:

> > > Alan Stern writes:
> > > 
> > > > Remember what I wrote a few minutes ago about khubd and ksuspend_usbd
> > > > wanting to resume devices during a system suspend transition?  This is
> > > > exactly what happens when those threads aren't frozen.
> > > 
> > > So, I wonder why I don't see that error on my powerbook?
> > 
> > That's a good question.  Miklos, can you please reproduce the suspend 
> > error using a kernel built with CONFIG_USB_DEBUG turned on?
> 
> Here's the full dmesg.  First there was a successful suspend/resume,
> then a couple of unsuccessful ones:

Okay, good.  It's not a coincidence that the first one worked and later 
ones didn't.  Here's what happened:

You suspended the system.  Upon resuming, the USB host controller 
drivers detected that their device sessions had been interrupted and 
consequently marked your two USB devices (the optical mouse and the 
biometric thingy) for removal.  Normally that removal would take place 
when khubd wakes up, i.e., after the resume is complete and the thread 
is taken out of the freezer.

But in this case khubd was already awake.  It resumed the two root hubs
in response to their remote wakeup requests and saw that the devices
were gone.  So it unregistered the device structures immediately --
even before the PM core tried to resume them -- and then registered new
data structures to embody the new device connections.  As far as I can
tell, the fact that these new device structures were created while the
suspend-list was in flux must have caused one of them (the biometric
coprocessor) not to be added to the appropriate list.

Thus the next time you tried to suspend, the PM core didn't call the
device's suspend method.  It was left awake, and as a result its parent 
hub refused to suspend (since it had an unsuspended child).  This 
caused the overall system suspend to fail.

This illustrates the folly of allowing other threads to perform 
driver-core-type operations in the middle of a system sleep transition.

Alan Stern

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Alan Stern]

On Tue, 3 Jul 2007, Matthew Garrett wrote:

> On Tue, Jul 03, 2007 at 06:21:42PM -0400, Alan Stern wrote:
> > On Tue, 3 Jul 2007, Matthew Garrett wrote:
> > > We're used to the idea of applications blocking when a resource they're 
> > > using goes away - NFS has done it forever. 
> > 
> > You persist in evading my point.  I'm not worried about applications;  
> > I'm worried about drivers.
> > 
> > Let me put it explicitly: You're writing a driver.  You're working on
> > the read, write, or probe method.  You add code to check if a system
> > sleep is underway.  Suppose the answer is Yes -- what does your driver
> > do next?
> 
> Leave the process blocked and defer any i/o until after resume. Why does 
> it need to be any more complicated than that?

(1) The driver will undoubtedly hold some mutex or semaphore at the 
time it checks whether a system sleep is underway.  You will have to 
drop it before blocking and then reacquire it afterward.

(2) The driver may have been called by some other routine which holds a 
mutex needed for resuming the device.  In this case the driver _can't_ 
drop the mutex and so the resume will deadlock.


Okay, I agree that (1) can be handled without too much effort.  But 
doing it adds an extra test to _every_ driver's I/O pathway.  Freezing 
userspace does not incur all this additional overhead.

(2) shouldn't arise during normal read and write operations, but it
certainly _will_ arise during probe.

Alan Stern

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Matthew Garrett]

On Wed, Jul 04, 2007 at 10:38:47AM -0400, Alan Stern wrote:

> Okay, I agree that (1) can be handled without too much effort.  But 
> doing it adds an extra test to _every_ driver's I/O pathway.  Freezing 
> userspace does not incur all this additional overhead.

For runtime PM to work it's already necessary to have a test in that 
path to check if the device is suspended. I can't see how this adds any 
overhead to the common case.

-- 
Matthew Garrett | mjg59@srcf.ucam.org

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Oliver Neukum]

Am Mittwoch, 4. Juli 2007 schrieb Matthew Garrett:
> On Wed, Jul 04, 2007 at 10:38:47AM -0400, Alan Stern wrote:
> 
> > Okay, I agree that (1) can be handled without too much effort.  But 
> > doing it adds an extra test to _every_ driver's I/O pathway.  Freezing 
> > userspace does not incur all this additional overhead.
> 
> For runtime PM to work it's already necessary to have a test in that 
> path to check if the device is suspended. I can't see how this adds any 
> overhead to the common case.

No,

you just make sure the device reports to upper layers when it might
be busy. The USB layer manages this quite well without burdening
the common case.

	Regards
		Oliver

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Alan Stern]

On Wed, 4 Jul 2007, Matthew Garrett wrote:

> On Wed, Jul 04, 2007 at 10:38:47AM -0400, Alan Stern wrote:
> 
> > Okay, I agree that (1) can be handled without too much effort.  But 
> > doing it adds an extra test to _every_ driver's I/O pathway.  Freezing 
> > userspace does not incur all this additional overhead.
> 
> For runtime PM to work it's already necessary to have a test in that 
> path to check if the device is suspended. I can't see how this adds any 
> overhead to the common case.

Actually it isn't necessary to have a test to check if the device is 
suspended.  We simply call the autoresume routine; if the device isn't 
suspended that routine doesn't have to do anything.

I agree that that the extra test (for system-wide suspend underway) is
needed only if the autoresume fails, which isn't part of the main
pathway.  So it doesn't add runtime overhead -- but handling it does
add code overhead.

Alan Stern

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Robert Hancock]

Matthew Garrett wrote:
> On Tue, Jul 03, 2007 at 06:21:42PM -0400, Alan Stern wrote:
>> On Tue, 3 Jul 2007, Matthew Garrett wrote:
>>> We're used to the idea of applications blocking when a resource they're 
>>> using goes away - NFS has done it forever. 
>> You persist in evading my point.  I'm not worried about applications;  
>> I'm worried about drivers.
>>
>> Let me put it explicitly: You're writing a driver.  You're working on
>> the read, write, or probe method.  You add code to check if a system
>> sleep is underway.  Suppose the answer is Yes -- what does your driver
>> do next?
> 
> Leave the process blocked and defer any i/o until after resume. Why does 
> it need to be any more complicated than that?

It gets complicated when this has to be added and TESTED in EVERY 
driver. The implied contract for drivers previously was that their 
device would not get accessed after it was suspended until it had been 
resumed first. This proposed change violates that.

I don't think this sort of handling is something that individual drivers 
should have to deal with (at least not ones that are part of a framework 
like USB, libata, etc.)

-- 
Robert Hancock      Saskatoon, SK, Canada
To email, remove "nospam" from hancockr@nospamshaw.ca
Home Page: http://www.roberthancock.com/

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Matthew Garrett]

On Tue, Jul 03, 2007 at 06:17:04PM -0600, Robert Hancock wrote:
> Matthew Garrett wrote:
> >Leave the process blocked and defer any i/o until after resume. Why does 
> >it need to be any more complicated than that?
> 
> It gets complicated when this has to be added and TESTED in EVERY 
> driver. The implied contract for drivers previously was that their 
> device would not get accessed after it was suspended until it had been 
> resumed first. This proposed change violates that.

No, that's only ever been true for ACPI systems. It's never been true 
elsewhere, and it won't be true for anything implementing any sort of 
runtime power management.

> I don't think this sort of handling is something that individual drivers 
> should have to deal with (at least not ones that are part of a framework 
> like USB, libata, etc.)

I'd agree there. Driver midlayers (where they exist) are the appropriate 
place to handle this.

-- 
Matthew Garrett | mjg59@srcf.ucam.org

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Rafael J. Wysocki]

On Wednesday, 4 July 2007 02:34, Matthew Garrett wrote:
> On Tue, Jul 03, 2007 at 06:17:04PM -0600, Robert Hancock wrote:
> > Matthew Garrett wrote:
> > >Leave the process blocked and defer any i/o until after resume. Why does 
> > >it need to be any more complicated than that?
> > 
> > It gets complicated when this has to be added and TESTED in EVERY 
> > driver. The implied contract for drivers previously was that their 
> > device would not get accessed after it was suspended until it had been 
> > resumed first. This proposed change violates that.
> 
> No, that's only ever been true for ACPI systems.

Sorry, but you're mistaken.

Some ARM platforms also use the suspend code in kernel/power and the 52xx
powerpc too (BTW, I don't know of any ARM platforms that support suspend and
don't use it).

> It's never been true elsewhere, and it won't be true for anything
> implementing any sort of runtime power management.

There are not many drivers in the tree supporting that right now.

> > I don't think this sort of handling is something that individual drivers 
> > should have to deal with (at least not ones that are part of a framework 
> > like USB, libata, etc.)
> 
> I'd agree there. Driver midlayers (where they exist) are the appropriate 
> place to handle this.

So perhaps we should modify them first and we can drop the freezer?

Perhaps we can schedule the freezer removal (from the suspend code path), so
that people have the time to rework the driver midlayers to take that into
account?

Greetings,
Rafael


-- 
"Premature optimization is the root of all evil." - Donald Knuth

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Alan Stern]

On Tue, 3 Jul 2007, Matthew Garrett wrote:

> On Tue, Jul 03, 2007 at 05:10:08PM -0400, Alan Stern wrote:
> 
> > No, no -- you have it exactly backwards.  Removing the freezer turns 
> > STR into something _less_ like runtime suspend, because it adds the 
> > requirement that devices must not automatically be resumed when an I/O 
> > request arrives.
> 
> But that's fine - "Are we undergoing a systemwide suspend" is an easy 
> question to ask. Freezing processes instead means that most of those 
> paths will never be tested.

The question is easy to ask, but it's not so easy to figure out what
you should do if the answer is Yes.  Freezing processes instead means
that those "untested" paths -- in many, many drivers -- won't have to 
exist at all.

Alan Stern

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Matthew Garrett]

On Tue, Jul 03, 2007 at 05:16:37PM -0400, Alan Stern wrote:
> On Tue, 3 Jul 2007, Matthew Garrett wrote:
> > But that's fine - "Are we undergoing a systemwide suspend" is an easy 
> > question to ask. Freezing processes instead means that most of those 
> > paths will never be tested.
> 
> The question is easy to ask, but it's not so easy to figure out what
> you should do if the answer is Yes.  Freezing processes instead means
> that those "untested" paths -- in many, many drivers -- won't have to 
> exist at all.

We're used to the idea of applications blocking when a resource they're 
using goes away - NFS has done it forever. 

-- 
Matthew Garrett | mjg59@srcf.ucam.org

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Matthew Garrett]

On Tue, Jul 03, 2007 at 11:37:51PM +0200, Rafael J. Wysocki wrote:
> On Tuesday, 3 July 2007 23:20, Matthew Garrett wrote:
> > We're used to the idea of applications blocking when a resource they're 
> > using goes away - NFS has done it forever. 
> 
> Now, please tell me how many driver writers even thought that something
> might try to access their devices after .suspend() had been executed (or
> even whilie it was being executed)?

Every single driver that fails under those conditions is already broken, 
and has been forever. It's likely that they're broken under run-time 
suspend, too.

-- 
Matthew Garrett | mjg59@srcf.ucam.org

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Oliver Neukum]

Am Dienstag, 3. Juli 2007 schrieb Matthew Garrett:
> On Tue, Jul 03, 2007 at 11:37:51PM +0200, Rafael J. Wysocki wrote:
> > On Tuesday, 3 July 2007 23:20, Matthew Garrett wrote:
> > > We're used to the idea of applications blocking when a resource they're 
> > > using goes away - NFS has done it forever. 
> > 
> > Now, please tell me how many driver writers even thought that something
> > might try to access their devices after .suspend() had been executed (or
> > even whilie it was being executed)?
> 
> Every single driver that fails under those conditions is already broken, 
> and has been forever. It's likely that they're broken under run-time 
> suspend, too.

There _is_ no runtime syspend in working condition. The interface
to do that has been scheduled for removal.

	Oliver

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Rafael J. Wysocki]

On Tuesday, 3 July 2007 23:36, Matthew Garrett wrote:
> On Tue, Jul 03, 2007 at 11:37:51PM +0200, Rafael J. Wysocki wrote:
> > On Tuesday, 3 July 2007 23:20, Matthew Garrett wrote:
> > > We're used to the idea of applications blocking when a resource they're 
> > > using goes away - NFS has done it forever. 
> > 
> > Now, please tell me how many driver writers even thought that something
> > might try to access their devices after .suspend() had been executed (or
> > even whilie it was being executed)?
> 
> Every single driver that fails under those conditions is already broken, 
> and has been forever. It's likely that they're broken under run-time 
> suspend, too.

Well, I won't argue with that, but do you actually know how many drivers are
broken this way?

Greetings,
Rafael


-- 
"Premature optimization is the root of all evil." - Donald Knuth

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Paul Mackerras]

Rafael J. Wysocki writes:

> Now, please tell me how many driver writers even thought that something
> might try to access their devices after .suspend() had been executed (or
> even whilie it was being executed)?

Well, I believe that the USB framework copes with this, except
possibly for some corner cases like the example that Alan Stern
posted.  The fact that powerbooks suspend and resume without the
freezer implies that the IDE framework, the console code and the
framebuffer code cope correctly (though possibly not all chipset
drivers).

So I think that a lot of the frameworks already get it right.  Of
course the quality of the low-level chipset drivers has always been
pretty variable. :)

Paul.

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Rafael J. Wysocki]

On Wednesday, 4 July 2007 05:38, Paul Mackerras wrote:
> Rafael J. Wysocki writes:
> 
> > Now, please tell me how many driver writers even thought that something
> > might try to access their devices after .suspend() had been executed (or
> > even whilie it was being executed)?
> 
> Well, I believe that the USB framework copes with this, except
> possibly for some corner cases like the example that Alan Stern
> posted.  The fact that powerbooks suspend and resume without the
> freezer implies that the IDE framework, the console code and the
> framebuffer code cope correctly (though possibly not all chipset
> drivers).

Okay, so in fact you don't know.

And that's my point in this thread.

I won't fight for the freezer for what it's worth, but let's do things in the
_right_ _order_.  For example, let's make sure that by making the $subject
change we won't introduce (too many) regressions and fix the frameworks
that don't get it right.

Using the problems with FUSE as an argument for making this change immediately
doesn't seem to be right to me.

> So I think that a lot of the frameworks already get it right.  Of
> course the quality of the low-level chipset drivers has always been
> pretty variable. :)

Yes, but that's another issue.

Greetings,
Rafael


-- 
"Premature optimization is the root of all evil." - Donald Knuth

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Paul Mackerras]

Rafael J. Wysocki writes:

> Okay, so in fact you don't know.

Don't know what exactly?

It has been a while since I had my head in the USB code.  I assume
it's being maintained by competent people. :)

> And that's my point in this thread.

Well, I'd be interested in hearing from Matthew whether he has
actually been using his patch in Ubuntu, and if so, what bug reports
he has been receiving related to it?

> I won't fight for the freezer for what it's worth, but let's do things in the
> _right_ _order_.  For example, let's make sure that by making the $subject
> change we won't introduce (too many) regressions and fix the frameworks
> that don't get it right.
> 
> Using the problems with FUSE as an argument for making this change immediately
> doesn't seem to be right to me.

I can see your point, but I won't be moving powermac over to use the
generic suspend path until the freezer is gone, since I am pretty
confident that the drivers we care about behave sensibly, and I have
seen a lot of traffic on linux-pm and lkml about problems caused by
the freezer.

Also, no-one has yet answered my fundamental objection to the freezer,
which is that the very kernel threads we would want to freeze are
often the same ones that we must not freeze, namely the threads that
issue I/O requests in order to satisfy incoming I/O requests.

If there was an automatic way to construct the graph of dependencies
(including data flows) between tasks, and derive an ordering for
freezing that guarantees that all I/Os will get completed without
deadlocks, then I could accept the freezer.  But we don't have
anything like that.

Paul.

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Rafael J. Wysocki]

On Wednesday, 4 July 2007 12:58, Paul Mackerras wrote:
> Rafael J. Wysocki writes:
> 
> > Okay, so in fact you don't know.
> 
> Don't know what exactly?

How many drivers will be adversely affected by the $subject change.

> It has been a while since I had my head in the USB code.  I assume
> it's being maintained by competent people. :)

I'm not talking about USB.
 
> > And that's my point in this thread.
> 
> Well, I'd be interested in hearing from Matthew whether he has
> actually been using his patch in Ubuntu, and if so, what bug reports
> he has been receiving related to it?

Me too.

> > I won't fight for the freezer for what it's worth, but let's do things in the
> > _right_ _order_.  For example, let's make sure that by making the $subject
> > change we won't introduce (too many) regressions and fix the frameworks
> > that don't get it right.
> > 
> > Using the problems with FUSE as an argument for making this change immediately
> > doesn't seem to be right to me.
> 
> I can see your point, but I won't be moving powermac over to use the
> generic suspend path until the freezer is gone, since I am pretty
> confident that the drivers we care about behave sensibly, and I have
> seen a lot of traffic on linux-pm and lkml about problems caused by
> the freezer.

They are mostly related to kernel threads, that we've already agreed no to
freeze (except for the ones that want that, but they will be responsible for
getting everything right).  The initial patches for that are in -mm and more
will come.

> Also, no-one has yet answered my fundamental objection to the freezer,
> which is that the very kernel threads we would want to freeze are
> often the same ones that we must not freeze, namely the threads that
> issue I/O requests in order to satisfy incoming I/O requests.

See above.  We're moving away from freezing kernel threads.

> If there was an automatic way to construct the graph of dependencies
> (including data flows) between tasks, and derive an ordering for
> freezing that guarantees that all I/Os will get completed without
> deadlocks, then I could accept the freezer.  But we don't have
> anything like that.

No we don't.

Still, my position is this:

1) The freezer (in the modified form, with the freezing of kernel threads
limited to the ones that want to be frozen) is needed for hibernation.

2) The freezer is generally not needed for suspend, _but_ there are drivers
in the tree that rely on it being used.  Thus, at some point in time we can
remove the freezer from the suspend code path, _but_ no sooner than we are
sure that the majority of drivers is prepared for that.

3) In the meantime, if there are freezer-related problems, they should be
fixed rather than used as arguments for immediate removal of it, because of 2).

Greetings,
Rafael


-- 
"Premature optimization is the root of all evil." - Donald Knuth

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Paul Mackerras]

Rafael J. Wysocki writes:

> They are mostly related to kernel threads, that we've already agreed no to
> freeze (except for the ones that want that, but they will be responsible for
> getting everything right).  The initial patches for that are in -mm and more
> will come.

Serious question: which kernel threads would actually want to be
frozen?

Threads that do no I/O at all don't care about suspend/resume and
don't need to be frozen in any case.  Threads that issue I/O requests
in order to service incoming I/O requests can't be frozen because of
the possibility of deadlock.  Which leaves threads that do I/O just
for the fun of it. :)

What am I missing?

> > Also, no-one has yet answered my fundamental objection to the freezer,
> > which is that the very kernel threads we would want to freeze are
> > often the same ones that we must not freeze, namely the threads that
> > issue I/O requests in order to satisfy incoming I/O requests.
> 
> See above.  We're moving away from freezing kernel threads.

I believe the distinction between threads and user processes is a
false one, because user processes can now do things that were formerly
only doable by kernel threads.

> > If there was an automatic way to construct the graph of dependencies
> > (including data flows) between tasks, and derive an ordering for
> > freezing that guarantees that all I/Os will get completed without
> > deadlocks, then I could accept the freezer.  But we don't have
> > anything like that.
> 
> No we don't.
> 
> Still, my position is this:
> 
> 1) The freezer (in the modified form, with the freezing of kernel threads
> limited to the ones that want to be frozen) is needed for hibernation.
> 
> 2) The freezer is generally not needed for suspend, _but_ there are drivers
> in the tree that rely on it being used.  Thus, at some point in time we can

Do you know which drivers they are?  I'm happy to help hack things
into shape.

> remove the freezer from the suspend code path, _but_ no sooner than we are
> sure that the majority of drivers is prepared for that.
> 
> 3) In the meantime, if there are freezer-related problems, they should be
> fixed rather than used as arguments for immediate removal of it, because of 2).

I don't know how you can make the freezer completely deadlock-free
while still providing the guarantee that some drivers currently need,
without constructing the dependency graph I mentioned.

Paul.

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Miklos Szeredi]

> Still, my position is this:
> 
> 1) The freezer (in the modified form, with the freezing of kernel threads
> limited to the ones that want to be frozen) is needed for hibernation.
> 
> 2) The freezer is generally not needed for suspend, _but_ there are drivers
> in the tree that rely on it being used.  Thus, at some point in time we can
> remove the freezer from the suspend code path, _but_ no sooner than we are
> sure that the majority of drivers is prepared for that.

And we won't know if drivers are OK until we remove the freezer,
catch-22.

So I think we need to disable the freezer at least in -mm and/or
optionally in -linus.

I applied Matthew's patch, and suspend did in fact stop working
(thinkpad t60), but there was nothing catastrophic.  Here's the dmesg
if somebody is interested:

Suspending console(s)
usb_endpoint usbdev5.3_ep83: PM: suspend 0->2, parent 5-2:1.0 already 2
usb_endpoint usbdev5.3_ep02: PM: suspend 0->2, parent 5-2:1.0 already 2
usb_endpoint usbdev5.3_ep81: PM: suspend 0->2, parent 5-2:1.0 already 2
hub 2-0:1.0: suspend error -16
suspend_device(): usb_suspend+0x0/0x1c() returns -16
Could not suspend device usb2: error -16
usb_endpoint usbdev5.3_ep81: PM: resume from 0, parent 5-2:1.0 still 2
usb_endpoint usbdev5.3_ep02: PM: resume from 0, parent 5-2:1.0 still 2
usb_endpoint usbdev5.3_ep83: PM: resume from 0, parent 5-2:1.0 still 2
Some devices failed to suspend

Miklos

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Theodore Tso]

On Wed, Jul 04, 2007 at 01:25:55PM +0200, Rafael J. Wysocki wrote:
> > Don't know what exactly?
> 
> How many drivers will be adversely affected by the $subject change.

Ok, so how about a CONFIG option which removes the freezer, so we can
find out experimentally how many people without it?  We can make it be
experimental at first, or (my preference) make it be the default
initially, and if people complain that their laptop's suspend feature
is broken, we can tell them how to turn back on the
CONFIG_FREEZER_DEPRECATED option.

							- Ted

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Rafael J. Wysocki]

On Wednesday, 4 July 2007 14:41, Theodore Tso wrote:
> On Wed, Jul 04, 2007 at 01:25:55PM +0200, Rafael J. Wysocki wrote:
> > > Don't know what exactly?
> > 
> > How many drivers will be adversely affected by the $subject change.
> 
> Ok, so how about a CONFIG option which removes the freezer, so we can
> find out experimentally how many people without it?  We can make it be
> experimental at first, or (my preference) make it be the default
> initially, and if people complain that their laptop's suspend feature
> is broken, we can tell them how to turn back on the
> CONFIG_FREEZER_DEPRECATED option.

The freezer is generally still necessary for hibernation, so I won't call
it "DEPRECATED'.

Moreover, I'd prefer to make drivers use different callbacks for hibernation
and suspend before we do that.

Greetings,
Rafael


-- 
"Premature optimization is the root of all evil." - Donald Knuth

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Rafael J. Wysocki]

On Wednesday, 4 July 2007 13:51, Miklos Szeredi wrote:
> > Still, my position is this:
> > 
> > 1) The freezer (in the modified form, with the freezing of kernel threads
> > limited to the ones that want to be frozen) is needed for hibernation.
> > 
> > 2) The freezer is generally not needed for suspend, _but_ there are drivers
> > in the tree that rely on it being used.  Thus, at some point in time we can
> > remove the freezer from the suspend code path, _but_ no sooner than we are
> > sure that the majority of drivers is prepared for that.
> 
> And we won't know if drivers are OK until we remove the freezer,
> catch-22.

I disagree.  We can learn that by auditing the drivers.

> So I think we need to disable the freezer at least in -mm and/or
> optionally in -linus.
> 
> I applied Matthew's patch, and suspend did in fact stop working
> (thinkpad t60), but there was nothing catastrophic.  Here's the dmesg
> if somebody is interested:
> 
> Suspending console(s)
> usb_endpoint usbdev5.3_ep83: PM: suspend 0->2, parent 5-2:1.0 already 2
> usb_endpoint usbdev5.3_ep02: PM: suspend 0->2, parent 5-2:1.0 already 2
> usb_endpoint usbdev5.3_ep81: PM: suspend 0->2, parent 5-2:1.0 already 2
> hub 2-0:1.0: suspend error -16
> suspend_device(): usb_suspend+0x0/0x1c() returns -16
> Could not suspend device usb2: error -16
> usb_endpoint usbdev5.3_ep81: PM: resume from 0, parent 5-2:1.0 still 2
> usb_endpoint usbdev5.3_ep02: PM: resume from 0, parent 5-2:1.0 still 2
> usb_endpoint usbdev5.3_ep83: PM: resume from 0, parent 5-2:1.0 still 2
> Some devices failed to suspend

No, it's not catastrophic, but something like this will result in a bug report
with "regression" in the subject.

Greetings,
Rafael


-- 
"Premature optimization is the root of all evil." - Donald Knuth

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Miklos Szeredi]

> On Wednesday, 4 July 2007 13:51, Miklos Szeredi wrote:
> > > Still, my position is this:
> > > 
> > > 1) The freezer (in the modified form, with the freezing of kernel threads
> > > limited to the ones that want to be frozen) is needed for hibernation.
> > > 
> > > 2) The freezer is generally not needed for suspend, _but_ there are drivers
> > > in the tree that rely on it being used.  Thus, at some point in time we can
> > > remove the freezer from the suspend code path, _but_ no sooner than we are
> > > sure that the majority of drivers is prepared for that.
> > 
> > And we won't know if drivers are OK until we remove the freezer,
> > catch-22.
> 
> I disagree.  We can learn that by auditing the drivers.

In theory, yes.  But it scales far worse than letting everyone
experiment/report/fix problems as they crop up.

> > So I think we need to disable the freezer at least in -mm and/or
> > optionally in -linus.
> > 
> > I applied Matthew's patch, and suspend did in fact stop working
> > (thinkpad t60), but there was nothing catastrophic.  Here's the dmesg
> > if somebody is interested:
> > 
> > Suspending console(s)
> > usb_endpoint usbdev5.3_ep83: PM: suspend 0->2, parent 5-2:1.0 already 2
> > usb_endpoint usbdev5.3_ep02: PM: suspend 0->2, parent 5-2:1.0 already 2
> > usb_endpoint usbdev5.3_ep81: PM: suspend 0->2, parent 5-2:1.0 already 2
> > hub 2-0:1.0: suspend error -16
> > suspend_device(): usb_suspend+0x0/0x1c() returns -16
> > Could not suspend device usb2: error -16
> > usb_endpoint usbdev5.3_ep81: PM: resume from 0, parent 5-2:1.0 still 2
> > usb_endpoint usbdev5.3_ep02: PM: resume from 0, parent 5-2:1.0 still 2
> > usb_endpoint usbdev5.3_ep83: PM: resume from 0, parent 5-2:1.0 still 2
> > Some devices failed to suspend
> 
> No, it's not catastrophic, but something like this will result in a
> bug report with "regression" in the subject.

If it was due to a config option marked experimental, then it's not a
regression.

It's a bug, and it needs looking at, but while the freezer is not
completely removed, it's not a serious problem.

So I agree with Ted, a config option (or maybe a runtime sysctl
tunable) to turn off the freezer for suspend should only have positive
effects.

Miklos

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Oliver Neukum]

Am Mittwoch, 4. Juli 2007 schrieb Miklos Szeredi:
> > > And we won't know if drivers are OK until we remove the freezer,
> > > catch-22.
> > 
> > I disagree.  We can learn that by auditing the drivers.
> 
> In theory, yes.  But it scales far worse than letting everyone
> experiment/report/fix problems as they crop up.

You will open many but small races. This would be very painfull
to debug.

	Regards
		Oliver

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Rafael J. Wysocki]

On Wednesday, 4 July 2007 17:03, Oliver Neukum wrote:
> Am Mittwoch, 4. Juli 2007 schrieb Miklos Szeredi:
> > > > And we won't know if drivers are OK until we remove the freezer,
> > > > catch-22.
> > > 
> > > I disagree.  We can learn that by auditing the drivers.
> > 
> > In theory, yes.  But it scales far worse than letting everyone
> > experiment/report/fix problems as they crop up.
> 
> You will open many but small races. This would be very painfull
> to debug.

Agreed.

They will not trigger 100% of the time, but sporadically and generally at
random.

At least the freezer problems are reproducible. ;-)

Greetings,
Rafael


-- 
"Premature optimization is the root of all evil." - Donald Knuth

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Paul Mackerras]

Rafael J. Wysocki writes:

> They will not trigger 100% of the time, but sporadically and generally at
> random.
> 
> At least the freezer problems are reproducible. ;-)

Our experience with powermacs has been that it isn't actually all that
hard to get it right for the drivers you care about.

Paul.

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Rafael J. Wysocki]

On Thursday, 5 July 2007 02:29, Paul Mackerras wrote:
> Rafael J. Wysocki writes:
> 
> > They will not trigger 100% of the time, but sporadically and generally at
> > random.
> > 
> > At least the freezer problems are reproducible. ;-)
> 
> Our experience with powermacs has been that it isn't actually all that
> hard to get it right for the drivers you care about.

Well, I'm a bit suspicious about that. ;-)

Namely, if you run your suspend code on one CPU and (do I remeber correctly
that?) with kernel preemption disabled, then you practically prevent user land
processes from being scheduled when your suspend code is running.

If that is the case (of which I'm not sure), the freezer is obviously
unnecessary, because you've taken processes out of the equation in a different
way ...

Greetings,
Rafael


-- 
"Premature optimization is the root of all evil." - Donald Knuth

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Pavel Machek]

Hi!

> > They will not trigger 100% of the time, but sporadically and generally at
> > random.
> > 
> > At least the freezer problems are reproducible. ;-)
> 
> Our experience with powermacs has been that it isn't actually all that
> hard to get it right for the drivers you care about.

So... will you fix input and usb? Their maintainers seem to prefer
simplicity of frozen system.

Also, can you write down rules for drivers to follow?

Like... 'if you get a request after system suspend started, you may
not kmalloc(GFP_KERNEL)'

 - because disk driver may be already suspended

- hmm, what happens if disk driver _is_ suspended? Let's say I'm v4l
driver on cpu1. Someone may talk to me _while_ disk driver is being
suspended... and if I'm after disk driver in pci order, I may only
learn about suspend _after_ I do the wrong kmalloc.

							Pavel
-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Alan Stern]

On Wed, 4 Jul 2007, Miklos Szeredi wrote:

> And we won't know if drivers are OK until we remove the freezer,
> catch-22.
> 
> So I think we need to disable the freezer at least in -mm and/or
> optionally in -linus.
> 
> I applied Matthew's patch, and suspend did in fact stop working
> (thinkpad t60), but there was nothing catastrophic.  Here's the dmesg
> if somebody is interested:
> 
> Suspending console(s)
> usb_endpoint usbdev5.3_ep83: PM: suspend 0->2, parent 5-2:1.0 already 2
> usb_endpoint usbdev5.3_ep02: PM: suspend 0->2, parent 5-2:1.0 already 2
> usb_endpoint usbdev5.3_ep81: PM: suspend 0->2, parent 5-2:1.0 already 2
> hub 2-0:1.0: suspend error -16
> suspend_device(): usb_suspend+0x0/0x1c() returns -16
> Could not suspend device usb2: error -16
> usb_endpoint usbdev5.3_ep81: PM: resume from 0, parent 5-2:1.0 still 2
> usb_endpoint usbdev5.3_ep02: PM: resume from 0, parent 5-2:1.0 still 2
> usb_endpoint usbdev5.3_ep83: PM: resume from 0, parent 5-2:1.0 still 2
> Some devices failed to suspend

Remember what I wrote a few minutes ago about khubd and ksuspend_usbd
wanting to resume devices during a system suspend transition?  This is
exactly what happens when those threads aren't frozen.

Alan Stern

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Miklos Szeredi]

> Remember what I wrote a few minutes ago about khubd and ksuspend_usbd
> wanting to resume devices during a system suspend transition?  This is
> exactly what happens when those threads aren't frozen.

OK, let me summarize the situation as I see it now: there are two
camps, the pro-freezers and the anti-freezers.

Pro-freezers say:

  - don't remove the freezer, otherwise we'll have to deal with
    numerous problems in drivers

Anti-freezers say:

  - let's remove the freezer, which causes numerous problems

Alan summerized the pro-freezer arguments well I think.  What are the
anti-freezer arguments then?

After having looked at the freezer and done some experiments with it,
the most obvious problem looks to be, that it can get stuck on a
process doing uninterruptible sleep.  And yes, this can happen if a
fuse filesystem daemon is frozen before a filesystem user is.  And
this is not something that can be fixed in fuse, some filesystem calls
(rename(2) for example) are simply not restartable.

This doesn't explain the deadlocks, but it could cause failure to
suspend which would be pretty annoying.

Does this affect other things than fuse?

Can this be fixed?

It seems to be a fundamental problem with the freezer: while it does
make sure that user processes are not calling into drivers during
suspend, it also disallows perfectly harmless non-driver calls as
well.

Miklos

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Paul Mackerras]

Alan Stern writes:

> Remember what I wrote a few minutes ago about khubd and ksuspend_usbd
> wanting to resume devices during a system suspend transition?  This is
> exactly what happens when those threads aren't frozen.

So, I wonder why I don't see that error on my powerbook?

Paul.

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Rafael J. Wysocki]

On Wednesday, 4 July 2007 21:25, Miklos Szeredi wrote:
> > Remember what I wrote a few minutes ago about khubd and ksuspend_usbd
> > wanting to resume devices during a system suspend transition?  This is
> > exactly what happens when those threads aren't frozen.
> 
> OK, let me summarize the situation as I see it now: there are two
> camps, the pro-freezers and the anti-freezers.
> 
> Pro-freezers say:
> 
>   - don't remove the freezer, otherwise we'll have to deal with
>     numerous problems in drivers

And these problems will generally be difficult to reproduce reliably and debug.

> Anti-freezers say:
> 
>   - let's remove the freezer, which causes numerous problems
> 
> Alan summerized the pro-freezer arguments well I think.  What are the
> anti-freezer arguments then?
> 
> After having looked at the freezer and done some experiments with it,
> the most obvious problem looks to be, that it can get stuck on a
> process doing uninterruptible sleep.

That's correct.

> And yes, this can happen if a fuse filesystem daemon is frozen before a
> filesystem user is.  And this is not something that can be fixed in fuse,
> some filesystem calls (rename(2) for example) are simply not restartable.
>
> This doesn't explain the deadlocks, but it could cause failure to
> suspend which would be pretty annoying.

I think the only thing that can deadlock in that context is the sync.  At
least, I don't see anything else.

> Does this affect other things than fuse?

Not that I know of.  It may affect user space drivers, but there's no data on
that.

> Can this be fixed?
> 
> It seems to be a fundamental problem with the freezer: while it does
> make sure that user processes are not calling into drivers during
> suspend, it also disallows perfectly harmless non-driver calls as
> well.

The problem is that when the freezer was designed (I didn't do that, BTW),
there was no FUSE and similar things, so it's not prepared to cope with
such interdependencies between user space tasks.

We had an analogous problem with vfork() and it was solved by using the
PF_FREEZER_SKIP flag.  Perhaps we can do similar thing with FUSE.

Greetings,
Rafael


-- 
"Premature optimization is the root of all evil." - Donald Knuth

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Miklos Szeredi]

> > Pro-freezers say:
> > 
> >   - don't remove the freezer, otherwise we'll have to deal with
> >     numerous problems in drivers
> 
> And these problems will generally be difficult to reproduce reliably
> and debug.

I see exactly the opposite.

With the freezer I can have very rarely occuring failures, due to
freeze ordering effects.

And without the freezer I have a 100% reproducable problem, that is
not hard to fix according to Alan Stern.  OK, I don't know what the
next problem would be, but the powermac experience shows, that it's
not nearly as bad as you and Oliver try to make it out.

> > Can this be fixed?
> > 
> > It seems to be a fundamental problem with the freezer: while it does
> > make sure that user processes are not calling into drivers during
> > suspend, it also disallows perfectly harmless non-driver calls as
> > well.
> 
> The problem is that when the freezer was designed (I didn't do that, BTW),
> there was no FUSE and similar things, so it's not prepared to cope with
> such interdependencies between user space tasks.
> 
> We had an analogous problem with vfork() and it was solved by using the
> PF_FREEZER_SKIP flag.  Perhaps we can do similar thing with FUSE.

It cannot be just worked around in fuse, as a task might be sleeping
on a number of VFS mutexes as well (i_mutex, s_vfs_rename_mutex, etc).
It would be a gigantic hack, possible at all.

Miklos

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Rafael J. Wysocki]

On Thursday, 5 July 2007 10:37, Miklos Szeredi wrote:
> > > Pro-freezers say:
> > > 
> > >   - don't remove the freezer, otherwise we'll have to deal with
> > >     numerous problems in drivers
> > 
> > And these problems will generally be difficult to reproduce reliably
> > and debug.
> 
> I see exactly the opposite.
> 
> With the freezer I can have very rarely occuring failures, due to
> freeze ordering effects.

The "freezer ordering effects" only affect uninterruptible tasks.
 
> And without the freezer I have a 100% reproducable problem, that is
> not hard to fix according to Alan Stern.  OK, I don't know what the
> next problem would be, but the powermac experience shows, that it's
> not nearly as bad as you and Oliver try to make it out.

The powermac need not be a good example due to the different code ordering
(it's run on one CPU only, which makes the probability of triggering a race
be quite low).

> > > Can this be fixed?
> > > 
> > > It seems to be a fundamental problem with the freezer: while it does
> > > make sure that user processes are not calling into drivers during
> > > suspend, it also disallows perfectly harmless non-driver calls as
> > > well.
> > 
> > The problem is that when the freezer was designed (I didn't do that, BTW),
> > there was no FUSE and similar things, so it's not prepared to cope with
> > such interdependencies between user space tasks.
> > 
> > We had an analogous problem with vfork() and it was solved by using the
> > PF_FREEZER_SKIP flag.  Perhaps we can do similar thing with FUSE.
> 
> It cannot be just worked around in fuse, as a task might be sleeping
> on a number of VFS mutexes as well (i_mutex, s_vfs_rename_mutex, etc).
> It would be a gigantic hack, possible at all.

Well, obviously FUSE and the freezer don't play well together, but that's FUSE
who's late in the game (the freezer was here before). ;-)

If you give me some time, I'll see what can be done.

Greetings,
Rafael


-- 
"Premature optimization is the root of all evil." - Donald Knuth

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Miklos Szeredi]

> > > PF_FREEZER_SKIP flag.  Perhaps we can do similar thing with FUSE.
> > 
> > It cannot be just worked around in fuse, as a task might be sleeping
> > on a number of VFS mutexes as well (i_mutex, s_vfs_rename_mutex, etc).
> > It would be a gigantic hack, possible at all.
> 
> Well, obviously FUSE and the freezer don't play well together, but
> that's FUSE who's late in the game (the freezer was here
> before). ;-)

Umm, and CODA which is _very_ similar to fuse was there long before
fuse or the freezer ;)

> If you give me some time, I'll see what can be done.

I give you all the time in the world.  I'd also be willing to help
with drivers for which I have the hardware.

Miklos

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Jeremy Fitzhardinge]

Miklos Szeredi wrote:
> Umm, and CODA which is _very_ similar to fuse was there long before
> fuse or the freezer ;)
>   

I did userfs around 1994-5.

    J

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Miklos Szeredi]

> > Umm, and CODA which is _very_ similar to fuse was there long before
> > fuse or the freezer ;)
> >   
> 
> I did userfs around 1994-5.

Yes, fuse didn't in fact have very much original idea in it.  It was
just putting all the pieces together to make a stable and useful
userspace filesystem framework.

Miklos

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Paul Mackerras]

Miklos Szeredi writes:

> OK, let me summarize the situation as I see it now: there are two
> camps, the pro-freezers and the anti-freezers.
> 
> Pro-freezers say:
> 
>   - don't remove the freezer, otherwise we'll have to deal with
>     numerous problems in drivers
> 
> Anti-freezers say:
> 
>   - let's remove the freezer, which causes numerous problems
> 
> Alan summerized the pro-freezer arguments well I think.  What are the
> anti-freezer arguments then?

1. The freezer cannot be guaranteed deadlock-free without constructing
   a dependency graph between tasks (both user and kernel), which is
   virtually impossible since the dependencies are not externally
   observable.

2. As a consequence of (1), we try to make a crude approximation of
   the graph by saying "only kernel threads that want to be frozen
   will be frozen" or some other similar statement.

3. However, (2) means that we can no longer guarantee that drivers
   will not get any I/O requests after their suspend method has been
   called, and therefore the freezer fails in its main objective.

4. We have an existence proof that reliable suspend can be achieved
   without the freezer.

To summarize, the argument is that the freezer is deadlock-prone and
ineffective.

Paul.

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Rafael J. Wysocki]

On Thursday, 5 July 2007 02:43, Paul Mackerras wrote:
> Miklos Szeredi writes:
> 
> > OK, let me summarize the situation as I see it now: there are two
> > camps, the pro-freezers and the anti-freezers.
> > 
> > Pro-freezers say:
> > 
> >   - don't remove the freezer, otherwise we'll have to deal with
> >     numerous problems in drivers
> > 
> > Anti-freezers say:
> > 
> >   - let's remove the freezer, which causes numerous problems
> > 
> > Alan summerized the pro-freezer arguments well I think.  What are the
> > anti-freezer arguments then?
> 
> 1. The freezer cannot be guaranteed deadlock-free without constructing
>    a dependency graph between tasks (both user and kernel), which is
>    virtually impossible since the dependencies are not externally
>    observable.

I don't agree with that.

The freezer only fails to handle uninterruptible tasks, so we need to take
the situations in which an uninterruptible task waits for a frozen task into
consideration.  Now, if both tasks are from the user land, this is highly
unusual.

> 2. As a consequence of (1), we try to make a crude approximation of
>    the graph by saying "only kernel threads that want to be frozen
>    will be frozen" or some other similar statement.

No.  The rule is that kernel threads should not be freezable, but there are
some for which that is useful.

> 3. However, (2) means that we can no longer guarantee that drivers
>    will not get any I/O requests after their suspend method has been
>    called, and therefore the freezer fails in its main objective.

This is a very general statement.  Can you please give some examples?

> 4. We have an existence proof that reliable suspend can be achieved
>    without the freezer.

No.  We only know that it might work if the nonboot CPUs are disabled before
suspending devices, which is not the case in the generic suspend code.

> To summarize, the argument is that the freezer is deadlock-prone and
> ineffective.

I remain unconvinced. ;-)

Greetings,
Rafael


-- 
"Premature optimization is the root of all evil." - Donald Knuth

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Johannes Berg]

[-- Attachment #1.1: Type: text/plain, Size: 736 bytes --]

On Thu, 2007-07-05 at 14:51 +0200, Rafael J. Wysocki wrote:

> > > Remember what I wrote a few minutes ago about khubd and ksuspend_usbd
> > > wanting to resume devices during a system suspend transition?  This is
> > > exactly what happens when those threads aren't frozen.
> > 
> > So, I wonder why I don't see that error on my powerbook?
> 
> Because you have only one CPU running while your suspend code is being
> executed?

If that's really all the problem then what's wrong with just unplugging
the other CPUs earlier? Sure, that makes suspend no longer perfectly
transparent since userspace might notice the CPUs being unplugged, but
it has to cope with that anyway since a user can do it manually...

johannes

[-- Attachment #1.2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 190 bytes --]

[-- Attachment #2: Type: text/plain, Size: 0 bytes --]

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Rafael J. Wysocki]

On Thursday, 5 July 2007 14:50, Johannes Berg wrote:
> On Thu, 2007-07-05 at 14:51 +0200, Rafael J. Wysocki wrote:
> 
> > > > Remember what I wrote a few minutes ago about khubd and ksuspend_usbd
> > > > wanting to resume devices during a system suspend transition?  This is
> > > > exactly what happens when those threads aren't frozen.
> > > 
> > > So, I wonder why I don't see that error on my powerbook?
> > 
> > Because you have only one CPU running while your suspend code is being
> > executed?
> 
> If that's really all the problem then what's wrong with just unplugging
> the other CPUs earlier? Sure, that makes suspend no longer perfectly
> transparent since userspace might notice the CPUs being unplugged, but
> it has to cope with that anyway since a user can do it manually...

This is a bit complicated, but I'll try to explain.

We used to disable the nonboot CPUs before suspending devices and enable
them after resuming devices, but that turned out to lead to resume problems on
some ACPI systems.  Namely, it turned out that this code ordering was not in
line with the ACPI spec that assumed specific ordering of events during a
suspend.  For this reason, we changed the code ordering and now it is more or
less in agreement with ACPI (for the first time, AFAICS).  I don't think that
anyone would like to revert that change right now.

Moreover, in the meantime we learned that the CPU hotplug code that we use
for disabling the nonboot CPUs, is generally problematic on x86 and only works
for us because we have the majority of interrupt sources disabled when it's
invoked.

Greetings,
Rafael


-- 
"Premature optimization is the root of all evil." - Donald Knuth

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Rafael J. Wysocki]

On Thursday, 5 July 2007 02:36, Paul Mackerras wrote:
> Alan Stern writes:
> 
> > Remember what I wrote a few minutes ago about khubd and ksuspend_usbd
> > wanting to resume devices during a system suspend transition?  This is
> > exactly what happens when those threads aren't frozen.
> 
> So, I wonder why I don't see that error on my powerbook?

Because you have only one CPU running while your suspend code is being
executed?

Greetings,
Rafael


-- 
"Premature optimization is the root of all evil." - Donald Knuth

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Alan Stern]

On Thu, 5 Jul 2007, Paul Mackerras wrote:

> Alan Stern writes:
> 
> > Remember what I wrote a few minutes ago about khubd and ksuspend_usbd
> > wanting to resume devices during a system suspend transition?  This is
> > exactly what happens when those threads aren't frozen.
> 
> So, I wonder why I don't see that error on my powerbook?

That's a good question.  Miklos, can you please reproduce the suspend 
error using a kernel built with CONFIG_USB_DEBUG turned on?

Alan Stern

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Dmitry Torokhov]

On 7/4/07, Paul Mackerras <paulus@samba.org> wrote:
> Rafael J. Wysocki writes:
>
> > They are mostly related to kernel threads, that we've already agreed no to
> > freeze (except for the ones that want that, but they will be responsible for
> > getting everything right).  The initial patches for that are in -mm and more
> > will come.
>
> Serious question: which kernel threads would actually want to be
> frozen?
>
> Threads that do no I/O at all don't care about suspend/resume and
> don't need to be frozen in any case.  Threads that issue I/O requests
> in order to service incoming I/O requests can't be frozen because of
> the possibility of deadlock.  Which leaves threads that do I/O just
> for the fun of it. :)
>
> What am I missing?
>

I like kseriod and kgameportd to be frozen. If fast resume of serio
port fails we kick the task of full resume to kseriod. But when we
hibernate and all the devices are resumed again so snapshot can be
written I don't care if touchpad failed to resume - it can be dealt
with later, when system is actually resumes. So I like that by
freezing I do not delay hibernation and all requests get simply
discarded as they are not in the image. This mostly affects
hibernation though.

Having these threads frozen also guarantees that serio/gameport device
tree is stable during susped/resume and I can process outstanding
requests later, when system is fully up.

-- 
Dmitry

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Alan Stern]

On Wed, 4 Jul 2007, Paul Mackerras wrote:

> Rafael J. Wysocki writes:
> 
> > They are mostly related to kernel threads, that we've already agreed no to
> > freeze (except for the ones that want that, but they will be responsible for
> > getting everything right).  The initial patches for that are in -mm and more
> > will come.
> 
> Serious question: which kernel threads would actually want to be
> frozen?

khubd and ksuspend_usbd.

> Threads that do no I/O at all don't care about suspend/resume and
> don't need to be frozen in any case.  Threads that issue I/O requests
> in order to service incoming I/O requests can't be frozen because of
> the possibility of deadlock.  Which leaves threads that do I/O just
> for the fun of it. :)
> 
> What am I missing?

Those two threads will try to resume USB devices in response to wakeup
requests.  Such requests arrive during a suspend or resume transition
more often than one would expect.

If the resume attempt occurs before the host controller has been 
suspended, it will abort the system suspend.  If it occurs after the 
host controller is suspended (and before the controller resumes) it 
will fail and try to unregister the USB device -- something else we 
don't like happening while the sytem is only partially up (not to 
mention the annoyance caused by the unregistration of a perfectly 
functional device).

Alan Stern

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Alan Stern]

On Wed, 4 Jul 2007, Alan Stern wrote:

> > Threads that do no I/O at all don't care about suspend/resume and
> > don't need to be frozen in any case.  Threads that issue I/O requests
> > in order to service incoming I/O requests can't be frozen because of
> > the possibility of deadlock.  Which leaves threads that do I/O just
> > for the fun of it. :)
> > 
> > What am I missing?
> 
> Those two threads will try to resume USB devices in response to wakeup
> requests.  Such requests arrive during a suspend or resume transition
> more often than one would expect.
> 
> If the resume attempt occurs before the host controller has been 
> suspended, it will abort the system suspend.  If it occurs after the 
> host controller is suspended (and before the controller resumes) it 
> will fail and try to unregister the USB device -- something else we 
> don't like happening while the sytem is only partially up (not to 
> mention the annoyance caused by the unregistration of a perfectly 
> functional device).

Actually the situation may not be quite this bad any more.  It's been a 
while since I tried suspending a system without freezing khubd and 
ksuspend_usbd.  But Miklos's mail shows that problems can and will 
occur.

Alan Stern

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Rafael J. Wysocki]

On Tuesday, 3 July 2007 23:20, Matthew Garrett wrote:
> On Tue, Jul 03, 2007 at 05:16:37PM -0400, Alan Stern wrote:
> > On Tue, 3 Jul 2007, Matthew Garrett wrote:
> > > But that's fine - "Are we undergoing a systemwide suspend" is an easy 
> > > question to ask. Freezing processes instead means that most of those 
> > > paths will never be tested.
> > 
> > The question is easy to ask, but it's not so easy to figure out what
> > you should do if the answer is Yes.  Freezing processes instead means
> > that those "untested" paths -- in many, many drivers -- won't have to 
> > exist at all.
> 
> We're used to the idea of applications blocking when a resource they're 
> using goes away - NFS has done it forever. 

Now, please tell me how many driver writers even thought that something
might try to access their devices after .suspend() had been executed (or
even whilie it was being executed)?

Greetings,
Rafael


-- 
"Premature optimization is the root of all evil." - Donald Knuth

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Alan Stern]

On Tue, 3 Jul 2007, Matthew Garrett wrote:

> On Tue, Jul 03, 2007 at 05:16:37PM -0400, Alan Stern wrote:
> > On Tue, 3 Jul 2007, Matthew Garrett wrote:
> > > But that's fine - "Are we undergoing a systemwide suspend" is an easy 
> > > question to ask. Freezing processes instead means that most of those 
> > > paths will never be tested.
> > 
> > The question is easy to ask, but it's not so easy to figure out what
> > you should do if the answer is Yes.  Freezing processes instead means
> > that those "untested" paths -- in many, many drivers -- won't have to 
> > exist at all.
> 
> We're used to the idea of applications blocking when a resource they're 
> using goes away - NFS has done it forever. 

You persist in evading my point.  I'm not worried about applications;  
I'm worried about drivers.

Let me put it explicitly: You're writing a driver.  You're working on
the read, write, or probe method.  You add code to check if a system
sleep is underway.  Suppose the answer is Yes -- what does your driver
do next?

Make your answer as detailed as you reasonably can.  And be careful to 
arrange things so that an ongoing I/O operation doesn't get messed up 
when your suspend method is called.

Alan Stern

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Matthew Garrett]

On Tue, Jul 03, 2007 at 06:21:42PM -0400, Alan Stern wrote:
> On Tue, 3 Jul 2007, Matthew Garrett wrote:
> > We're used to the idea of applications blocking when a resource they're 
> > using goes away - NFS has done it forever. 
> 
> You persist in evading my point.  I'm not worried about applications;  
> I'm worried about drivers.
> 
> Let me put it explicitly: You're writing a driver.  You're working on
> the read, write, or probe method.  You add code to check if a system
> sleep is underway.  Suppose the answer is Yes -- what does your driver
> do next?

Leave the process blocked and defer any i/o until after resume. Why does 
it need to be any more complicated than that?

-- 
Matthew Garrett | mjg59@srcf.ucam.org

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Alan Stern]

On Tue, 3 Jul 2007, Matthew Garrett wrote:

> See the start of this thread. It's just not clear what the freezer buys 
> us - removing it gets rid of a load of subtle issues and complexity, and 
> turns system suspend into something that looks more like runtime 
> suspend (which might then encourage people to get runtime suspend 
> right...)

No, no -- you have it exactly backwards.  Removing the freezer turns 
STR into something _less_ like runtime suspend, because it adds the 
requirement that devices must not automatically be resumed when an I/O 
request arrives.

Alan Stern

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Matthew Garrett]

On Tue, Jul 03, 2007 at 05:10:08PM -0400, Alan Stern wrote:

> No, no -- you have it exactly backwards.  Removing the freezer turns 
> STR into something _less_ like runtime suspend, because it adds the 
> requirement that devices must not automatically be resumed when an I/O 
> request arrives.

But that's fine - "Are we undergoing a systemwide suspend" is an easy 
question to ask. Freezing processes instead means that most of those 
paths will never be tested.

-- 
Matthew Garrett | mjg59@srcf.ucam.org

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Nigel Cunningham]

[-- Attachment #1.1: Type: text/plain, Size: 840 bytes --]

On Tuesday 03 July 2007 14:29:18 Matthew Garrett wrote:
> Suspend to RAM on a machine with / on a fuse filesystem turns out to be 
> a screaming nightmare - either the suspend fails because syslog (for 
> instance) can't be frozen, or the machine deadlocks for some other 
> reason I haven't tracked down. We could "fix" fuse, or alternatively we 
> could do what we do for suspend to RAM on other platforms (PPC and APM) 
> and just not use the freezer.
> 
> Signed-off-by: Matthew Garrett <mjg59@srcf.ucam.org>

Note, though, that this won't help at all when people use the "suspend-to-ram 
instead of powering down after writing a hibernation image" feature in 
(uswsusp | tuxonice). Fuse is just a broken idea in the first place, but 
given that it exists, we still need to find the underlying cause.

Regards,

Nigel

[-- Attachment #1.2: Type: application/pgp-signature, Size: 189 bytes --]

[-- Attachment #2: Type: text/plain, Size: 0 bytes --]

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Matthew Garrett]

On Tue, Jul 03, 2007 at 02:54:41PM +1000, Nigel Cunningham wrote:

> Note, though, that this won't help at all when people use the "suspend-to-ram 
> instead of powering down after writing a hibernation image" feature in 
> (uswsusp | tuxonice). Fuse is just a broken idea in the first place, but 
> given that it exists, we still need to find the underlying cause.

If / is on fuse it's unlikely that hibernation is high on your list of 
priorities right now.

-- 
Matthew Garrett | mjg59@srcf.ucam.org

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Nigel Cunningham]

[-- Attachment #1.1: Type: text/plain, Size: 701 bytes --]

Hi.

On Tuesday 03 July 2007 15:21:30 Matthew Garrett wrote:
> On Tue, Jul 03, 2007 at 02:54:41PM +1000, Nigel Cunningham wrote:
> 
> > Note, though, that this won't help at all when people use 
the "suspend-to-ram 
> > instead of powering down after writing a hibernation image" feature in 
> > (uswsusp | tuxonice). Fuse is just a broken idea in the first place, but 
> > given that it exists, we still need to find the underlying cause.
> 
> If / is on fuse it's unlikely that hibernation is high on your list of 
> priorities right now.

Yeah, well... what can you say to that? :)

Nigel
-- 
See http://www.tuxonice.net for Howtos, FAQs, mailing
lists, wiki and bugzilla info.

[-- Attachment #1.2: Type: application/pgp-signature, Size: 189 bytes --]

[-- Attachment #2: Type: text/plain, Size: 0 bytes --]

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Benjamin Herrenschmidt]

> Note, though, that this won't help at all when people use the "suspend-to-ram 
> instead of powering down after writing a hibernation image" feature in 
> (uswsusp | tuxonice). Fuse is just a broken idea in the first place, but 
> given that it exists, we still need to find the underlying cause.

No, Fuse is not a broken idea in the first place. It's the freezer that
is a totally broken idea. It has proven many times to be racy by design
and cannot be made right. Ther usermode helper mess is just part of
that, fuse is another example, etc etc ...

So I think Matthew is totally right. In fact, the presence of the
freezer is the main reason why Paulus so far NACKed Johannes attempts at
merging the PPC PM code with the generic code in kernel/power.c

We've been doing fine without it so far and intend to continue to do so.

As for suspend-to-disk, I refer you to the discussions we had in the
past with Linus, where he explains I think quite clearly how wrong the
current implementation of STR is :-)

Thing is, if you're going to do snapshots, you should probably not sync
after you have "frozen" anyway.

Cheers,
Ben.

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Nigel Cunningham]

[-- Attachment #1.1: Type: text/plain, Size: 1958 bytes --]

Hi.

On Tuesday 03 July 2007 15:48:26 Benjamin Herrenschmidt wrote:
> 
> > Note, though, that this won't help at all when people use 
the "suspend-to-ram 
> > instead of powering down after writing a hibernation image" feature in 
> > (uswsusp | tuxonice). Fuse is just a broken idea in the first place, but 
> > given that it exists, we still need to find the underlying cause.
> 
> No, Fuse is not a broken idea in the first place. It's the freezer that
> is a totally broken idea. It has proven many times to be racy by design
> and cannot be made right. Ther usermode helper mess is just part of
> that, fuse is another example, etc etc ...

To some extent, I agree. I think the ideal solution would be to simply not 
schedule processes that are supposed to be frozen. But who wants to play with 
scheduler code? Not me!

> So I think Matthew is totally right. In fact, the presence of the
> freezer is the main reason why Paulus so far NACKed Johannes attempts at
> merging the PPC PM code with the generic code in kernel/power.c
> 
> We've been doing fine without it so far and intend to continue to do so.

Fuse depends on !PPC?
 
> As for suspend-to-disk, I refer you to the discussions we had in the
> past with Linus, where he explains I think quite clearly how wrong the
> current implementation of STR is :-)

I assume you mean STD. The problem there is that Linus doesn't care about STD. 
If he did, I dare say he'd think through the issues more thoroughly than he 
apparently has.
 
> Thing is, if you're going to do snapshots, you should probably not sync
> after you have "frozen" anyway.

Fully agree. But how do you stop things syncing while you're writing the image 
if you don't have a freezer or equivalent? (scheduler based, kexec.. they're 
all workarounds for this issue).

Regards,

Nigel
-- 
Nigel, Michelle and Alisdair Cunningham
5 Mitchell Street
Cobden 3266
Victoria, Australia

[-- Attachment #1.2: Type: application/pgp-signature, Size: 189 bytes --]

[-- Attachment #2: Type: text/plain, Size: 0 bytes --]

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Benjamin Herrenschmidt]

On Tue, 2007-07-03 at 16:08 +1000, Nigel Cunningham wrote:
> 
> > So I think Matthew is totally right. In fact, the presence of the
> > freezer is the main reason why Paulus so far NACKed Johannes attempts at
> > merging the PPC PM code with the generic code in kernel/power.c
> > 
> > We've been doing fine without it so far and intend to continue to do so.
> 
> Fuse depends on !PPC?

No, that's not what I'm saying. I'm saying we've been doing STR without
the freezer and that's the way to go imho.

> > As for suspend-to-disk, I refer you to the discussions we had in the
> > past with Linus, where he explains I think quite clearly how wrong the
> > current implementation of STR is :-)
> 
> I assume you mean STD.

Oops, yeah, sorry.

> The problem there is that Linus doesn't care about STD. 
> If he did, I dare say he'd think through the issues more thoroughly than he 
> apparently has.

Heh, that might be the case :-)
 
> > Thing is, if you're going to do snapshots, you should probably not sync
> > after you have "frozen" anyway.
> 
> Fully agree. But how do you stop things syncing while you're writing the image 
> if you don't have a freezer or equivalent? (scheduler based, kexec.. they're 
> all workarounds for this issue).

Well, I was saying that in the context of the -current- snapshotting
mechanism which is based on the freezer, then you should not
sys_sync(). 

Some random user or kernel thread doing a sync is not a problem. It will
stop in the middle of sync and resume on wakeup.

The problem is currently because STD -itself- attempts to sync after it
has frozen things.

I think that should be changed. If you want to sync for whatever reason,
(mostly save RAM ?) do it before the freeze. That means you may get new
dirty data in memory that isn't written out by the sync before you
freeze, but that's allright, that data will be in the suspend image
anyway. If you fail to wakeup, that's akin to a normal crash, the user
will only lose the last data written at the time of the suspend and
journaling fs'es should take care of fs metadata integrity.

So to summarize, the plan that makes things work with fuse is:

 - For STR, don't do the freezer thing.

 - For STD, don't sys_sync() after you froze

There might be -other- issues, but that should get you through some of
them at least. Of course, you'll be in trouble if you try to do things
like STD-to-a-file which sits on a fuse FS but there's a limit to
insanity :-)

Cheers,
Ben.

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Rafael J. Wysocki]

On Tuesday, 3 July 2007 09:19, Benjamin Herrenschmidt wrote:
> On Tue, 2007-07-03 at 16:08 +1000, Nigel Cunningham wrote:
> > 
> > > So I think Matthew is totally right. In fact, the presence of the
> > > freezer is the main reason why Paulus so far NACKed Johannes attempts at
> > > merging the PPC PM code with the generic code in kernel/power.c
> > > 
> > > We've been doing fine without it so far and intend to continue to do so.
> > 
> > Fuse depends on !PPC?
> 
> No, that's not what I'm saying. I'm saying we've been doing STR without
> the freezer and that's the way to go imho.
> 
> > > As for suspend-to-disk, I refer you to the discussions we had in the
> > > past with Linus, where he explains I think quite clearly how wrong the
> > > current implementation of STR is :-)
> > 
> > I assume you mean STD.
> 
> Oops, yeah, sorry.
> 
> > The problem there is that Linus doesn't care about STD. 
> > If he did, I dare say he'd think through the issues more thoroughly than he 
> > apparently has.
> 
> Heh, that might be the case :-)
>  
> > > Thing is, if you're going to do snapshots, you should probably not sync
> > > after you have "frozen" anyway.
> > 
> > Fully agree. But how do you stop things syncing while you're writing the image 
> > if you don't have a freezer or equivalent? (scheduler based, kexec.. they're 
> > all workarounds for this issue).
> 
> Well, I was saying that in the context of the -current- snapshotting
> mechanism which is based on the freezer, then you should not
> sys_sync(). 
> 
> Some random user or kernel thread doing a sync is not a problem. It will
> stop in the middle of sync and resume on wakeup.
> 
> The problem is currently because STD -itself- attempts to sync after it
> has frozen things.

To be precise, it tries to sync after it has frozen the user land.
 
> I think that should be changed. If you want to sync for whatever reason,
> (mostly save RAM ?) do it before the freeze. That means you may get new
> dirty data in memory that isn't written out by the sync before you
> freeze, but that's allright, that data will be in the suspend image
> anyway. If you fail to wakeup, that's akin to a normal crash, the user
> will only lose the last data written at the time of the suspend and
> journaling fs'es should take care of fs metadata integrity.
> 
> So to summarize, the plan that makes things work with fuse is:
> 
>  - For STR, don't do the freezer thing.

In the long run, I agree.

Still, can you please read this post from Alan Stern:

https://lists.linux-foundation.org/pipermail/linux-pm/2007-June/012847.html

?  I don't think I'm able to repeat the arguments given in there in a
convincing way.

>  - For STD, don't sys_sync() after you froze

Yeah, I think we can move the syncing before the freezing, so to speak.

And it need not be called from within the freezer, BTW.

> There might be -other- issues, but that should get you through some of
> them at least. Of course, you'll be in trouble if you try to do things
> like STD-to-a-file which sits on a fuse FS but there's a limit to
> insanity :-)

Yes. :-)

Ggreetings,
Rafael


-- 
"Premature optimization is the root of all evil." - Donald Knuth

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Benjamin Herrenschmidt]

> >  - For STR, don't do the freezer thing.
> 
> In the long run, I agree.
> 
> Still, can you please read this post from Alan Stern:
> 
> https://lists.linux-foundation.org/pipermail/linux-pm/2007-June/012847.html
> 
> ?  I don't think I'm able to repeat the arguments given in there in a
> convincing way.

That's the same crackpot I've been hearing for the past 3 years or
so ...

Both Paulus and I think the freezer is just a way to try to put your
head in the sand and ignore the problem. It causes as many problems as
it solves on its own, and is just not a solution that will be of any use
once you start implementing dynamic PM schemes etc...

In many cases, having proper support for "live" suspend of devices is
just a matter of having a couple of helpers in whatever subsystem those
drivers hookup with. In the case of network, for example, it's mostly
trivial (stop the queue). For block, it's not terribly hard neither,
though you want to have some orderign/atomicity between the blocking of
the incoming request queue and the sending of things like spindown &
flush commands to the disk. For old-style IDE, that was fairly easily
solved by piping suspend/resume command down the request queue itself
and have the queue block/unblbock itself after processing them. Some of
that logic could maybe be moved to the block layer for all block drivers
to benefit.

But yes, overall, there is work to do on drivers and I'm doing the ones
I hit on the platforms I use. I don't think the freezer is any kind of
remotely good solution, just a way to continue avoiding the problem.

Ben.

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Rafael J. Wysocki]

On Tuesday, 3 July 2007 23:14, Benjamin Herrenschmidt wrote:
> 
> > >  - For STR, don't do the freezer thing.
> > 
> > In the long run, I agree.
> > 
> > Still, can you please read this post from Alan Stern:
> > 
> > https://lists.linux-foundation.org/pipermail/linux-pm/2007-June/012847.html
> > 
> > ?  I don't think I'm able to repeat the arguments given in there in a
> > convincing way.
> 
> That's the same crackpot I've been hearing for the past 3 years or
> so ...
> 
> Both Paulus and I think the freezer is just a way to try to put your
> head in the sand and ignore the problem. It causes as many problems as
> it solves on its own, and is just not a solution that will be of any use
> once you start implementing dynamic PM schemes etc...
> 
> In many cases, having proper support for "live" suspend of devices is
> just a matter of having a couple of helpers in whatever subsystem those
> drivers hookup with. In the case of network, for example, it's mostly
> trivial (stop the queue). For block, it's not terribly hard neither,
> though you want to have some orderign/atomicity between the blocking of
> the incoming request queue and the sending of things like spindown &
> flush commands to the disk. For old-style IDE, that was fairly easily
> solved by piping suspend/resume command down the request queue itself
> and have the queue block/unblbock itself after processing them. Some of
> that logic could maybe be moved to the block layer for all block drivers
> to benefit.
> 
> But yes, overall, there is work to do on drivers and I'm doing the ones
> I hit on the platforms I use. I don't think the freezer is any kind of
> remotely good solution, just a way to continue avoiding the problem.

Still, do you really think that we're ready to drop it _right_ _now_ (I'm
referring to suspend only) and if so than on what basis (except that you
don't like it, which falls short of being a techical argument)?

Greetings,
Rafael


-- 
"Premature optimization is the root of all evil." - Donald Knuth

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Benjamin Herrenschmidt]

On Tue, 2007-07-03 at 23:32 +0200, Rafael J. Wysocki wrote:
> 
> Still, do you really think that we're ready to drop it _right_ _now_
> (I'm
> referring to suspend only) and if so than on what basis (except that
> you
> don't like it, which falls short of being a techical argument)?

Works fine for me without it ;-)

Ben.

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Rafael J. Wysocki]

On Tuesday, 3 July 2007 23:35, Benjamin Herrenschmidt wrote:
> On Tue, 2007-07-03 at 23:32 +0200, Rafael J. Wysocki wrote:
> > 
> > Still, do you really think that we're ready to drop it _right_ _now_
> > (I'm
> > referring to suspend only) and if so than on what basis (except that
> > you
> > don't like it, which falls short of being a techical argument)?
> 
> Works fine for me without it ;-)

Yeah, that makes sense. ;-)

Still, someone needs to take care of bug reports from unlucky people and I
wouldn't like to increase the number of these by doing such a high-level change
without any preparations ...

Greetings,
Rafael


-- 
"Premature optimization is the root of all evil." - Donald Knuth

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Pavel Machek]

Hi!

> > >  - For STR, don't do the freezer thing.
> > 
> > In the long run, I agree.
> > 
> > Still, can you please read this post from Alan Stern:
> > 
> > https://lists.linux-foundation.org/pipermail/linux-pm/2007-June/012847.html
> > 
> > ?  I don't think I'm able to repeat the arguments given in there in a
> > convincing way.
> 
> That's the same crackpot I've been hearing for the past 3 years or
> so ...
> 
> Both Paulus and I think the freezer is just a way to try to put your
> head in the sand and ignore the problem. It causes as many problems as
> it solves on its own, and is just not a solution that will be of any use
> once you start implementing dynamic PM schemes etc...

Well, yep, you can view freezer as a head in the sand...

> In many cases, having proper support for "live" suspend of devices is
> just a matter of having a couple of helpers in whatever subsystem those
> drivers hookup with. In the case of network, for example, it's mostly
> trivial (stop the queue). For block, it's not terribly hard neither,
> though you want to have some orderign/atomicity between the blocking of
> the incoming request queue and the sending of things like spindown &
> flush commands to the disk. For old-style IDE, that was fairly easily
> solved by piping suspend/resume command down the request queue itself
> and have the queue block/unblbock itself after processing them. Some of
> that logic could maybe be moved to the block layer for all block drivers
> to benefit.

...but the moment you start blocking tasks that done driver request,
you _do_ have mini-freezer of your own, with pretty much the same
problems.

In another message I shown that removing freezer will not help with
FUSE in general case.

It probably does not help with firmware, too; as soon as udev attempts
to do something with your wireless card, it is blocked, and if the
wireless card needs the firmware from udev, you are deadlocked.
									Pavel
-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Benjamin Herrenschmidt]

On Thu, 2007-07-05 at 11:30 +0200, Pavel Machek wrote:
> 
> ...but the moment you start blocking tasks that done driver request,
> you _do_ have mini-freezer of your own, with pretty much the same
> problems.

No, not at all the same problems. Those tasks will block, but that will
be harmless because we won't have some "freezer" things waiting for all
tasks to reach a "stable" point (calling try_to_freeze()). We just let
them block wherever we want, as long as it doesn't prevent a -driver-
from suspending, which should be allright, we have no problem.

> In another message I shown that removing freezer will not help with
> FUSE in general case.

I disagree.

> It probably does not help with firmware, too; as soon as udev attempts
> to do something with your wireless card, it is blocked, and if the
> wireless card needs the firmware from udev, you are deadlocked.

Firmware load has been a problem since day 1, I've talked about it
multiple times, it's broken with or without the freezer, and so far, the
reaction of pretty much everybody has been to dig their head deeper in
the mud and ignore the problem.

There are other issues (again, with or without freezer) that should be
dealt with. For example, drivers that haven't yet got their suspend()
callback or already have got their resume() may rely on services of the
kernel that are still blocked, that's where things may go hairy.
request_firmware() within resume() is a typical example of that.

There are a few things we should do in that area. For example, once we
start to call driver suspend's, we should probably set a system wide
flag that will do things such as:

 - block usermode helpers (either make call_usermodehelper return
something like -EBUSY or have it queue up the calls and issue them later
when thing are resuming, we need to look closely at what semantic we
want here).

 - Silently add GFP_NOIO to all allocations, to avoid having things
blocking in kmalloc() with a mutex held that will deadlock with
suspend() in a driver for example. Or set some way to have all GFP
waiters wakeup and fail rather than wait for IOs. It's hard/bizarre but
necessary, again, with or without a freezer.

 - Deal with the firmware problem. The best way is probably to have an
async request_firmware interface(). Another thing is, drivers may want
to cache their firmware in main memory, that sort of thing...

And that's just a small list off the top of my mind, of known problems
that will cause deadlocks or misbehaviours today, with or without the
freezer, and that need to be addressed.

Ben.

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Nigel Cunningham]

[-- Attachment #1.1: Type: text/plain, Size: 3426 bytes --]

Hi.

On Friday 06 July 2007 08:46:54 Benjamin Herrenschmidt wrote:
> On Thu, 2007-07-05 at 11:30 +0200, Pavel Machek wrote:
> > 
> > ...but the moment you start blocking tasks that done driver request,
> > you _do_ have mini-freezer of your own, with pretty much the same
> > problems.
> 
> No, not at all the same problems. Those tasks will block, but that will
> be harmless because we won't have some "freezer" things waiting for all
> tasks to reach a "stable" point (calling try_to_freeze()). We just let
> them block wherever we want, as long as it doesn't prevent a -driver-
> from suspending, which should be allright, we have no problem.

Will you be able to guarantee that every place where a task can/will block 
will be harmless place? If so, how will you guarantee that? How will you 
debug issues where a task occasionally doesn't block in the right place, 
particularly instances where it is some less than obvious interaction with 
other tasks?

This is the whole point to having the freezer. It makes things more 
predictable and testable. It shows us, clearly, when process X is the one 
that is causing problems.

> > In another message I shown that removing freezer will not help with
> > FUSE in general case.
> 
> I disagree.

Why?
 
> > It probably does not help with firmware, too; as soon as udev attempts
> > to do something with your wireless card, it is blocked, and if the
> > wireless card needs the firmware from udev, you are deadlocked.
> 
> Firmware load has been a problem since day 1, I've talked about it
> multiple times, it's broken with or without the freezer, and so far, the
> reaction of pretty much everybody has been to dig their head deeper in
> the mud and ignore the problem.
> 
> There are other issues (again, with or without freezer) that should be
> dealt with. For example, drivers that haven't yet got their suspend()
> callback or already have got their resume() may rely on services of the
> kernel that are still blocked, that's where things may go hairy.
> request_firmware() within resume() is a typical example of that.
> 
> There are a few things we should do in that area. For example, once we
> start to call driver suspend's, we should probably set a system wide
> flag that will do things such as:
> 
>  - block usermode helpers (either make call_usermodehelper return
> something like -EBUSY or have it queue up the calls and issue them later
> when thing are resuming, we need to look closely at what semantic we
> want here).
> 
>  - Silently add GFP_NOIO to all allocations, to avoid having things
> blocking in kmalloc() with a mutex held that will deadlock with
> suspend() in a driver for example. Or set some way to have all GFP
> waiters wakeup and fail rather than wait for IOs. It's hard/bizarre but
> necessary, again, with or without a freezer.

GFP_ATOMIC? (In driver suspend, they shouldn't be sleeping either, right?)
 
>  - Deal with the firmware problem. The best way is probably to have an
> async request_firmware interface(). Another thing is, drivers may want
> to cache their firmware in main memory, that sort of thing...
> 
> And that's just a small list off the top of my mind, of known problems
> that will cause deadlocks or misbehaviours today, with or without the
> freezer, and that need to be addressed.

Userspace device drivers too?

Regards,

Nigel

[-- Attachment #1.2: Type: application/pgp-signature, Size: 189 bytes --]

[-- Attachment #2: Type: text/plain, Size: 0 bytes --]

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Benjamin Herrenschmidt]

> Will you be able to guarantee that every place where a task can/will block 
> will be harmless place? If so, how will you guarantee that? How will you 
> debug issues where a task occasionally doesn't block in the right place, 
> particularly instances where it is some less than obvious interaction with 
> other tasks?

Which places aren't harmless if you don't have a freezer ?

> This is the whole point to having the freezer. It makes things more 
> predictable and testable. It shows us, clearly, when process X is the one 
> that is causing problems.

No, the freezer creates all those places what are harmful for a task to
block because they will break the freezer :-)

> >  - Silently add GFP_NOIO to all allocations, to avoid having things
> > blocking in kmalloc() with a mutex held that will deadlock with
> > suspend() in a driver for example. Or set some way to have all GFP
> > waiters wakeup and fail rather than wait for IOs. It's hard/bizarre but
> > necessary, again, with or without a freezer.
> 
> GFP_ATOMIC? (In driver suspend, they shouldn't be sleeping either, right?)

NOIO should be enough I think but ATOMIC would do).
 
That's one of the reason why I used to have the pre-suspend and
post-resume hooks in my original powermac implementation, for those few
drivers complicated enough to require some pre-allocations.
 
> >  - Deal with the firmware problem. The best way is probably to have an
> > async request_firmware interface(). Another thing is, drivers may want
> > to cache their firmware in main memory, that sort of thing...
> >

Note that the above firmware problem could be dealt with also with the
pre-suspend/post-resume. Allowing to pre-request firmware etc... and
keep it around until after resume, because we know we will need it.
Gives a chance to drivers to perform things while the system is still
live, filesystems still working, etc... (big memory allocations for
example).

> > And that's just a small list off the top of my mind, of known problems
> > that will cause deadlocks or misbehaviours today, with or without the
> > freezer, and that need to be addressed.
> 
> Userspace device drivers too?

Maybe but they are less of an issue, most of the time, they don't do DMA
or whatever harmful things. If they are USB drivers, for example, they
are an non-issues at that level.

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Nigel Cunningham]

[-- Attachment #1.1: Type: text/plain, Size: 3217 bytes --]

Hi.

On Friday 06 July 2007 09:20:43 Benjamin Herrenschmidt wrote:
> 
> > Will you be able to guarantee that every place where a task can/will block 
> > will be harmless place? If so, how will you guarantee that? How will you 
> > debug issues where a task occasionally doesn't block in the right place, 
> > particularly instances where it is some less than obvious interaction with 
> > other tasks?
> 
> Which places aren't harmless if you don't have a freezer ?

If I knew that, I wouldn't be asking the question.

> > This is the whole point to having the freezer. It makes things more 
> > predictable and testable. It shows us, clearly, when process X is the one 
> > that is causing problems.
> 
> No, the freezer creates all those places what are harmful for a task to
> block because they will break the freezer :-)

Nice try :) Okay then, you remove the freezer, try hibernating, then get back 
to me after you've fixed your filesystem because some process that wasn't 
frozen started writing things after the atomic copy (making the on disk 
filesystem inconsistent with the snapshot).

As Pavel rightly said, you can get rid of the freezer, but you're only going 
to have to implement another one that does the essentially the same thing, 
even if it is at some other level.
 
> > >  - Silently add GFP_NOIO to all allocations, to avoid having things
> > > blocking in kmalloc() with a mutex held that will deadlock with
> > > suspend() in a driver for example. Or set some way to have all GFP
> > > waiters wakeup and fail rather than wait for IOs. It's hard/bizarre but
> > > necessary, again, with or without a freezer.
> > 
> > GFP_ATOMIC? (In driver suspend, they shouldn't be sleeping either, right?)
> 
> NOIO should be enough I think but ATOMIC would do).
>  
> That's one of the reason why I used to have the pre-suspend and
> post-resume hooks in my original powermac implementation, for those few
> drivers complicated enough to require some pre-allocations.
>  
> > >  - Deal with the firmware problem. The best way is probably to have an
> > > async request_firmware interface(). Another thing is, drivers may want
> > > to cache their firmware in main memory, that sort of thing...
> > >
> 
> Note that the above firmware problem could be dealt with also with the
> pre-suspend/post-resume. Allowing to pre-request firmware etc... and
> keep it around until after resume, because we know we will need it.
> Gives a chance to drivers to perform things while the system is still
> live, filesystems still working, etc... (big memory allocations for
> example).
> 
> > > And that's just a small list off the top of my mind, of known problems
> > > that will cause deadlocks or misbehaviours today, with or without the
> > > freezer, and that need to be addressed.
> > 
> > Userspace device drivers too?
> 
> Maybe but they are less of an issue, most of the time, they don't do DMA
> or whatever harmful things. If they are USB drivers, for example, they
> are an non-issues at that level.

(Leaving the rest of the message intact so we don't have to fragment the 
discussion into a million subthreads).

Regards,

Nigel

[-- Attachment #1.2: Type: application/pgp-signature, Size: 189 bytes --]

[-- Attachment #2: Type: text/plain, Size: 0 bytes --]

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Kyle Moffett]

On Jul 05, 2007, at 19:35:11, Nigel Cunningham wrote:
> On Friday 06 July 2007 09:20:43 Benjamin Herrenschmidt wrote:
>> No, the freezer creates all those places what are harmful for a  
>> task to block because they will break the freezer :-)
>
> Nice try :) Okay then, you remove the freezer, try hibernating,  
> then get back to me after you've fixed your filesystem because some  
> process that wasn't frozen started writing things after the atomic  
> copy (making the on disk filesystem inconsistent with the snapshot).

Umm, this thread is NOT ABOUT HIBERNATING!!!  Please go back and read  
the subject, specifically the "suspend to RAM" parts :-D.  When your  
hardware can put itself to sleep and atomically preserve memory as it  
does so, you don't need an atomic copy.  For Real Suspend(TM) (IE:  
Suspend-to-RAM), the list of things to do is short and simple:

1)  Stop DMA and put most hardware into low-power states (stops all  
interrupt sources)
2)  Ensure that the other CPUs have finished any trailing interrupt  
handlers and put them to sleep
3)  Put the interrupt-controllers into low-power state
4)  Go to sleep

> As Pavel rightly said, you can get rid of the freezer, but you're  
> only going to have to implement another one that does the  
> essentially the same thing, even if it is at some other level.

How about a freezer whose job it is to "wait for pending hard  
interrupts to complete when we have already guaranteed that we won't  
get any more"?  That part should be really *REALLY* easy.  You don't  
need to care about either userspace processes or kernel threads at  
all.  Specifically, Step 1 consists of:

suspend_device(dev)
{
	set_no_bind_flag(dev);
	for (dev->subdevices)
		suspend_device(dev);
	set_no_io_flag(dev);
	wait_for_in_progress_dma(dev);
	turn_off_interrupts(dev);
	go_to_low_power_state(dev);
}

After you've set the "no_bind" flag, you won't get any *new*  
subdevices trying to bind, therefore it's safe to iterate over the  
list of present sub-devices and suspend them.  Once those are  
suspended and in low-power states you can set a "no_io" flag to  
prevent the driver from submitting more IO.  At that point you can  
lazily wait for existing DMA/IO/interrupts to finish on the device,  
since *NOBODY* will be submitting them anymore, and we certainly  
aren't probing for new devices.  Then you can just turn off the power  
to the device.  When all the leaf devices are off, the parent device  
can be turned off because everything waiting on the leaf devices is  
blocked on them and won't unblock until the parent device *AND* the  
leaf device are turned on again, in that order.

Scheduling and userspace are all still fully enabled in this  
scenario.  Once all your devices are turned off, the only remaining  
running threads will be those which haven't done IO since the  
beginning of the suspend.  We can then disable preemption, turn off  
the timer interrupts, and tell the other CPUs to park all their  
remaining threads in schedule() and sleep.  Then we put the IRQ  
controller to sleep and go to sleep ourselves.  If our driver model  
locking is sufficient to handle putting a parent device to sleep  
while threads are sleeping on a child device then there are exactly 0  
problems.

Resuming is basically running the whole process in reverse.  Runtime- 
suspend is achieved by not setting the 'no_io' or 'no_bind' flags and  
putting selective device-subtrees to sleep without doing anything to  
the rest of the system.

Cheers,
Kyle Moffett

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Benjamin Herrenschmidt]

On Fri, 2007-07-06 at 09:35 +1000, Nigel Cunningham wrote:
> 
> Nice try :) Okay then, you remove the freezer, try hibernating, then get back 
> to me after you've fixed your filesystem because some process that wasn't 
> frozen started writing things after the atomic copy (making the on disk 
> filesystem inconsistent with the snapshot).
> 
> As Pavel rightly said, you can get rid of the freezer, but you're only going 
> to have to implement another one that does the essentially the same thing, 
> even if it is at some other level.

I was mostly talking about STR... Regarding STD, we have a different
problem and we all know it. The freezer is one somewhat horrible way to
get it working for now, I would prefer something more along the way that
blocks the page cache from writing out new dirty pages though, except
those specifically flagged by the snapshot.

That is, some kind of proper snapshotting facility, as linus was
describing some time ago.

Ben.

> > > >  - Silently add GFP_NOIO to all allocations, to avoid having things
> > > > blocking in kmalloc() with a mutex held that will deadlock with
> > > > suspend() in a driver for example. Or set some way to have all GFP
> > > > waiters wakeup and fail rather than wait for IOs. It's hard/bizarre but
> > > > necessary, again, with or without a freezer.
> > > 
> > > GFP_ATOMIC? (In driver suspend, they shouldn't be sleeping either, right?)
> > 
> > NOIO should be enough I think but ATOMIC would do).
> >  
> > That's one of the reason why I used to have the pre-suspend and
> > post-resume hooks in my original powermac implementation, for those few
> > drivers complicated enough to require some pre-allocations.
> >  
> > > >  - Deal with the firmware problem. The best way is probably to have an
> > > > async request_firmware interface(). Another thing is, drivers may want
> > > > to cache their firmware in main memory, that sort of thing...
> > > >
> > 
> > Note that the above firmware problem could be dealt with also with the
> > pre-suspend/post-resume. Allowing to pre-request firmware etc... and
> > keep it around until after resume, because we know we will need it.
> > Gives a chance to drivers to perform things while the system is still
> > live, filesystems still working, etc... (big memory allocations for
> > example).
> > 
> > > > And that's just a small list off the top of my mind, of known problems
> > > > that will cause deadlocks or misbehaviours today, with or without the
> > > > freezer, and that need to be addressed.
> > > 
> > > Userspace device drivers too?
> > 
> > Maybe but they are less of an issue, most of the time, they don't do DMA
> > or whatever harmful things. If they are USB drivers, for example, they
> > are an non-issues at that level.
> 
> (Leaving the rest of the message intact so we don't have to fragment the 
> discussion into a million subthreads).
> 
> Regards,
> 
> Nigel
> 
> 
> Invalid signature
> 
> 

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Nigel Cunningham]

[-- Attachment #1.1: Type: text/plain, Size: 2087 bytes --]

Hi.

On Friday 06 July 2007 13:54:15 Benjamin Herrenschmidt wrote:
> On Fri, 2007-07-06 at 09:35 +1000, Nigel Cunningham wrote:
> > 
> > Nice try :) Okay then, you remove the freezer, try hibernating, then get 
back 
> > to me after you've fixed your filesystem because some process that wasn't 
> > frozen started writing things after the atomic copy (making the on disk 
> > filesystem inconsistent with the snapshot).
> > 
> > As Pavel rightly said, you can get rid of the freezer, but you're only 
going 
> > to have to implement another one that does the essentially the same thing, 
> > even if it is at some other level.
> 
> I was mostly talking about STR... Regarding STD, we have a different
> problem and we all know it. The freezer is one somewhat horrible way to
> get it working for now, I would prefer something more along the way that
> blocks the page cache from writing out new dirty pages though, except
> those specifically flagged by the snapshot.
> 
> That is, some kind of proper snapshotting facility, as linus was
> describing some time ago.

The kind of thing Linus was talking about would limit you (as swsusp and 
uswsusp do now) to only half the amount of memory. I suppose you could lzf 
compress as you did the snapshot. That would generally get you up to 2/3rds, 
but then again you can't know what compression ratio you'll get until you 
try, so reliability would suffer or it would take longer because of retrying.

I/O from swsusp and suspend2 use bios directly, so the page cache isn't an 
issue for them (apart from the fact that Suspend2 saves the page cache 
separately so it can get a full image). Not sure about uswsusp.

Only having half the amount of memory doesn't sound like a big limitation for 
modern desktops & laptops, but don't forget that there are embedded guys 
wanting to hbernate too :)

Regards,

Nigel
-- 
Nigel Cunningham
Christian Reformed Church of Cobden
103 Curdie Street, Cobden 3266, Victoria, Australia
Ph. +61 3 5595 1185 / +61 417 100 574
Communal Worship: 11 am Sunday.

[-- Attachment #1.2: Type: application/pgp-signature, Size: 189 bytes --]

[-- Attachment #2: Type: text/plain, Size: 0 bytes --]

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Benjamin Herrenschmidt]

> I/O from swsusp and suspend2 use bios directly, so the page cache isn't an 
> issue for them (apart from the fact that Suspend2 saves the page cache 
> separately so it can get a full image). Not sure about uswsusp.
> 
> Only having half the amount of memory doesn't sound like a big limitation for 
> modern desktops & laptops, but don't forget that there are embedded guys 
> wanting to hbernate too :)

Wait wait wait ... uses the BIOS ? what do you mean ?

I know that for example, things like MacOS X use a separate polled path to
the storage driver for suspend (works fine for the built-in IDE, but more
complicated in large scale). If you can use BIOS calls to write your
suspend image, that is, if you don't need any of the normal block
infrastructure, then you don't need a freezer ! not at all !

You just do like STR ... and at the end of the day, once you have stopped
all your driver, you shut interrupts off and do the BIOS thing....

I fail to see how processes could dirty pages while/after the BIOS thingy :-)

But then, the problem with that approach is that of course, you need a BIOS
capable of doing that (or a special sideband path to the "blessed" block
driver that will be used for suspend ... not necessarily a hard thing to do,
would be trivial to add support to drivers/ide or libata for that sort of
things I suppose).

Ben.

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Nigel Cunningham]

[-- Attachment #1.1: Type: text/plain, Size: 754 bytes --]

Hi.

On Friday 06 July 2007 14:41:40 Benjamin Herrenschmidt wrote:
> 
> > I/O from swsusp and suspend2 use bios directly, so the page cache isn't an 
> > issue for them (apart from the fact that Suspend2 saves the page cache 
> > separately so it can get a full image). Not sure about uswsusp.
> > 
> > Only having half the amount of memory doesn't sound like a big limitation 
for 
> > modern desktops & laptops, but don't forget that there are embedded guys 
> > wanting to hbernate too :)
> 
> Wait wait wait ... uses the BIOS ? what do you mean ?

You misread me, Ben. Sorry for not being clearer. bios as in struct bio.

Regards,

Nigel
-- 
See http://www.tuxonice.net for Howtos, FAQs, mailing
lists, wiki and bugzilla info.

[-- Attachment #1.2: Type: application/pgp-signature, Size: 189 bytes --]

[-- Attachment #2: Type: text/plain, Size: 0 bytes --]

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Kyle Moffett]

On Jul 06, 2007, at 00:03:15, Nigel Cunningham wrote:
> On Friday 06 July 2007 13:54:15 Benjamin Herrenschmidt wrote:
>> On Fri, 2007-07-06 at 09:35 +1000, Nigel Cunningham wrote:
>>>
>>> Nice try :) Okay then, you remove the freezer, try hibernating,  
>>> then get back to me after you've fixed your filesystem because  
>>> some process that wasn't frozen started writing things after the  
>>> atomic copy (making the on disk filesystem inconsistent with the  
>>> snapshot).
>>>
>>> As Pavel rightly said, you can get rid of the freezer, but you're  
>>> only going to have to implement another one that does the  
>>> essentially the same thing, even if it is at some other level.
>>
>> I was mostly talking about STR... Regarding STD, we have a  
>> different problem and we all know it. The freezer is one somewhat  
>> horrible way to get it working for now, I would prefer something  
>> more along the way that blocks the page cache from writing out new  
>> dirty pages though, except those specifically flagged by the  
>> snapshot.
>>
>> That is, some kind of proper snapshotting facility, as linus was  
>> describing some time ago.
>
> The kind of thing Linus was talking about would limit you (as  
> swsusp and uswsusp do now) to only half the amount of memory.

How so?  Suppose hibernate is implemented like this:

(1) Userspace program calls sys_freeze_processes()
   (a) Pokes all CPUs with IPMIs and tells them to finish the  
currently running timeslot then stop
   (b) Atomically sends SIGSTOP to all userspace processes in a non- 
trappable way, except the calling process and any process which is  
ptracing it.
   (c) Returns to the calling process.

(2) Userspace process sends SIGCONT to only those processes which are  
necessary for sync and a device-mapper snapshot.

(3) Userspace calls sys_snapshot_kernel(snapshot_overhead_pages)
   (a) Kernel starts freeing memory and swapping stuff out to make  
room for a copy of *kernel* memory (not pagecache, not process RAM).   
It does the same for at least snapshot_overhead_pages extra (used by  
userspace later).  It then allocates this memory to keep it from  
going away.  Since most processes are stopped we won't have much else  
competing with us for the RAM.
   (a) Kernel uses the device-mapper up-call-into-filesystem  
machinery to get all mounted filesystems synced and ready for a DM  
snapshot.  This may include sending data via the userspace processes  
resumed in (2).  Any deadlocks here are userspace's fault (see (2)).   
Will need some modification to handle doing multiple blockdevs at a  
time.  Anything using FUSE is basically perma-synced anyways (no dep- 
handling needed), and anything using loop should already be handled  
by DM.  This includes allocating memory for the basic snapshot  
datastructures.
   (b) At this point all blockdev operations should be halted and  
disk caches flushed; that's all we care about.
   (c) Go through the device tree and quiesce DMA and shut off  
interrupts.  Since all the disks are synced this is easy.
   (d) Use IPMIs again to get all the CPUs together, which should be  
easy as most processes are sleeping in IO or SIGSTOPed, and we're  
getting no interrupts.
   (e) One CPU turns off all interrupts on itself and takes an atomic  
snapshot of kernel memory into the previously allocated storage.   
Once again, does not include pagecache.  The kernel also records a  
list of what pages *are* included in the pagecache.  It then marks  
all userspace pages as copy-on-write.
   (f) That CPU finalizes the modified DM snapshot using the  
previously-allocated memory.
   (g) That CPU frees up the snapshot_overhead_pages memory allocated  
during step (a) for userspace to use.
   (h) The CPU does the equivalent of a "swapoff -a" without  
overwriting any data already on any swap device(s).
   (i) The CPU then IPMI-signals the other CPUs to wake them up
   (j) The kernel returns a FD-reference to the snapshot and the read- 
only halves of the CoW pagecache to the process which called  
sys_snapshot_kernel().

(4) The userspace process now has a reference to the copy of the  
kernel pages and the unmodified pagecache pages.  Since 99% of the  
processes aren't running, we aren't going to be having to CoW many of  
the pagecache pages.

(5) The userspace process uses read() or other syscalls to get data  
out of the kernel-snapshot FD in small chunks, within its  
snapshot_overhead_pages limit.  It compresses these and writes them  
out to the snapshot-storage blockdev (must not be mounted during  
snapshot), or to any network server.

(6) The userspace process syncs the disks and halts the system.  Any  
changed filesystem pages after the pseudo-DM-snapshot should have  
been stored in semi-volatile storage somewhere and will be discarded  
on the next reboot.

So basically your hibernate-overhead would consist of:
   (1) The pages necessary for the atomic snapshot of kernel memory  
and the list of pagecache pages at that time
   (2) A little memory necessary for the kernel non-persistent DM  
snapshot datastructures.
   (3) The snapshot_overhead_pages needed by userspace.

If you're using swap devices then you can save 99% of the state of  
the running kernel with an initial swapout overhead of virtually  
nothing beyond the size of the unswappable kernel memory.

Cheers,
Kyle Moffett

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Nigel Cunningham]

[-- Attachment #1.1: Type: text/plain, Size: 762 bytes --]

Hi Kyle.

On Friday 06 July 2007 15:01:48 Kyle Moffett wrote:
> On Jul 06, 2007, at 00:03:15, Nigel Cunningham wrote:
> > The kind of thing Linus was talking about would limit you (as  
> > swsusp and uswsusp do now) to only half the amount of memory.
> 
> How so?  Suppose hibernate is implemented like this:

You're not talking about the same thing Linus was suggesting. He was just 
wanting a result = sys_snapshot() sort of call. That would limit us to half 
the amount of memory.

I've looked over what you've written below and want to consider it in detail. 
Right now though, I don't have the time. I'll try to get back to you 
promptly.

Nigel
-- 
See http://www.tuxonice.net for Howtos, FAQs, mailing
lists, wiki and bugzilla info.

[-- Attachment #1.2: Type: application/pgp-signature, Size: 189 bytes --]

[-- Attachment #2: Type: text/plain, Size: 0 bytes --]

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Nigel Cunningham]

[-- Attachment #1.1: Type: text/plain, Size: 9010 bytes --]

Hi.

Sorry for the long delay. Busy weekend and my motivation for working on 
programming is almost zero at the moment...

On Friday 06 July 2007 15:01:48 Kyle Moffett wrote:
> On Jul 06, 2007, at 00:03:15, Nigel Cunningham wrote:
> > On Friday 06 July 2007 13:54:15 Benjamin Herrenschmidt wrote:
> >> On Fri, 2007-07-06 at 09:35 +1000, Nigel Cunningham wrote:
> >>>
> >>> Nice try :) Okay then, you remove the freezer, try hibernating,  
> >>> then get back to me after you've fixed your filesystem because  
> >>> some process that wasn't frozen started writing things after the  
> >>> atomic copy (making the on disk filesystem inconsistent with the  
> >>> snapshot).
> >>>
> >>> As Pavel rightly said, you can get rid of the freezer, but you're  
> >>> only going to have to implement another one that does the  
> >>> essentially the same thing, even if it is at some other level.
> >>
> >> I was mostly talking about STR... Regarding STD, we have a  
> >> different problem and we all know it. The freezer is one somewhat  
> >> horrible way to get it working for now, I would prefer something  
> >> more along the way that blocks the page cache from writing out new  
> >> dirty pages though, except those specifically flagged by the  
> >> snapshot.
> >>
> >> That is, some kind of proper snapshotting facility, as linus was  
> >> describing some time ago.
> >
> > The kind of thing Linus was talking about would limit you (as  
> > swsusp and uswsusp do now) to only half the amount of memory.
> 
> How so?  Suppose hibernate is implemented like this:
> 
> (1) Userspace program calls sys_freeze_processes()
>    (a) Pokes all CPUs with IPMIs and tells them to finish the  
> currently running timeslot then stop
>    (b) Atomically sends SIGSTOP to all userspace processes in a non- 
> trappable way, except the calling process and any process which is  
> ptracing it.
>    (c) Returns to the calling process.

Ok. First, I'll ignore the specification that userspace does this - I don't 
think it matters whether it's userspace or kernel that does the suspending 
and I'm yet to see a good reason for it to be [required to be] done from 
userspace.

In this first step, you've reinvented the first part of the current freezer 
implementation. The reason we don't use a real signal is precisely so we can 
have an untrappable SIGSTOP. In this regard, I particularly remember Win4Lin 
from a few years ago. It would die if you sent it a real signal, so we had to 
do it this way. No doubt there are other instances I'm not aware of.
 
> (2) Userspace process sends SIGCONT to only those processes which are  
> necessary for sync and a device-mapper snapshot.

How do you determine which ones are needed? Why stop them in the first place?
 
> (3) Userspace calls sys_snapshot_kernel(snapshot_overhead_pages)
>    (a) Kernel starts freeing memory and swapping stuff out to make  
> room for a copy of *kernel* memory (not pagecache, not process RAM).   
> It does the same for at least snapshot_overhead_pages extra (used by  
> userspace later).  It then allocates this memory to keep it from  
> going away.  Since most processes are stopped we won't have much else  
> competing with us for the RAM.

Ok. So now you also need processes running that are needed for swapping, 
because freeing that memory might involve swapping. Fully agree with the 
logic though (not really surprising - this is what I do in 
Suspend2^wTuxOnIce).

>    (a) Kernel uses the device-mapper up-call-into-filesystem  
> machinery to get all mounted filesystems synced and ready for a DM  
> snapshot.  This may include sending data via the userspace processes  
> resumed in (2).  Any deadlocks here are userspace's fault (see (2)).   
> Will need some modification to handle doing multiple blockdevs at a  
> time.  Anything using FUSE is basically perma-synced anyways (no dep- 
> handling needed), and anything using loop should already be handled  
> by DM.  This includes allocating memory for the basic snapshot  
> datastructures.
>    (b) At this point all blockdev operations should be halted and  
> disk caches flushed; that's all we care about.
>    (c) Go through the device tree and quiesce DMA and shut off  
> interrupts.  Since all the disks are synced this is easy.
>    (d) Use IPMIs again to get all the CPUs together, which should be  
> easy as most processes are sleeping in IO or SIGSTOPed, and we're  
> getting no interrupts.
>    (e) One CPU turns off all interrupts on itself and takes an atomic  
> snapshot of kernel memory into the previously allocated storage.   
> Once again, does not include pagecache.  The kernel also records a  
> list of what pages *are* included in the pagecache.  It then marks  
> all userspace pages as copy-on-write.

Hotplugging cpus (when all those locking issues are taken care of) is simpler. 
Prior to cpu hotplugging, I used IMPIs to put secondary cpus into a tight 
loop, so I know it's possible to do it this way too. That way, though, you 
have less flexibility. What if a cpu really is plugged in between hibernate 
and resume? With cpu hotplugging, it's handled properly and transparently. 
Without cpu hotplugging, you could be using uninitialised data after the 
atomic restore.

Marking userspace as COW makes things more complicated, too. You then have to 
add code to the COW handling to update the list of pages that need to be 
saved, and you reduce the reliability of the whole process. You can't predict 
beforehand how many of these COW pages are going to be needed, and therefore 
can't know how much memory to free earlier on in the process. If you run out 
of memory, what will be the effect?

>    (f) That CPU finalizes the modified DM snapshot using the  
> previously-allocated memory.
>    (g) That CPU frees up the snapshot_overhead_pages memory allocated  
> during step (a) for userspace to use.
>    (h) The CPU does the equivalent of a "swapoff -a" without  
> overwriting any data already on any swap device(s).

You still need to remember what swap you're going to use to write the image. 
You'll probably want to get this information (and allocate the swap) sooner 
rather than later so that you're not racing against the memory freeing 
earlier, and don't run into issues with bmapping the pages or having enough 
memory to record the bdevs & sector numbers (not usually an issue, but if 
swap is highly fragmented...).

>    (i) The CPU then IPMI-signals the other CPUs to wake them up
>    (j) The kernel returns a FD-reference to the snapshot and the read- 
> only halves of the CoW pagecache to the process which called  
> sys_snapshot_kernel().

Readonly halves? I don't get that, sorry.
 
> (4) The userspace process now has a reference to the copy of the  
> kernel pages and the unmodified pagecache pages.  Since 99% of the  
> processes aren't running, we aren't going to be having to CoW many of  
> the pagecache pages.

Mmm, but you still don't know how many.
 
> (5) The userspace process uses read() or other syscalls to get data  
> out of the kernel-snapshot FD in small chunks, within its  
> snapshot_overhead_pages limit.  It compresses these and writes them  
> out to the snapshot-storage blockdev (must not be mounted during  
> snapshot), or to any network server.
> 
> (6) The userspace process syncs the disks and halts the system.  Any  
> changed filesystem pages after the pseudo-DM-snapshot should have  
> been stored in semi-volatile storage somewhere and will be discarded  
> on the next reboot.

Are you thinking the changed filesystem pages are caught by COW? (AFAIUI, 
kernel writes aren't). If (as I expect), you're thinking about filesystem 
writes to DM based storage, what about non DM-based filesystem pages?
 
> So basically your hibernate-overhead would consist of:
>    (1) The pages necessary for the atomic snapshot of kernel memory  
> and the list of pagecache pages at that time
>    (2) A little memory necessary for the kernel non-persistent DM  
> snapshot datastructures.
>    (3) The snapshot_overhead_pages needed by userspace.
> 
> If you're using swap devices then you can save 99% of the state of  
> the running kernel with an initial swapout overhead of virtually  
> nothing beyond the size of the unswappable kernel memory.

FWIW, let me note an important variation from how Suspend2 works; it might 
provide food for thought. In Suspend2, we treat the processes that remain 
stopped throughout the whole process specially. We write their data to disk 
before the atomic copy (usually 70 or 80% of memory), and then use the memory 
they occupy for the destination of the atomic copy. This further reduces the 
amount of memory that has to be freed, almost always to zero.

Regards,

Nigel
-- 
See http://www.tuxonice.net for Howtos, FAQs, mailing
lists, wiki and bugzilla info.

[-- Attachment #1.2: Type: application/pgp-signature, Size: 189 bytes --]

[-- Attachment #2: Type: text/plain, Size: 0 bytes --]

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Kyle Moffett]

Thanks for the detailed reply!

On Jul 09, 2007, at 22:07:15, Nigel Cunningham wrote:
> On Friday 06 July 2007 15:01:48 Kyle Moffett wrote:
>> Suppose hibernate is implemented like this:
>>
>> (1) Userspace program calls sys_freeze_processes()
>>    (a) Pokes all CPUs with IPMIs and tells them to finish the  
>> currently running timeslot then stop
>>    (b) Atomically sends SIGSTOP to all userspace processes in a  
>> non-trappable way, except the calling process and any process  
>> which is ptracing it.
>>    (c) Returns to the calling process.
>
> Ok. First, I'll ignore the specification that userspace does this -  
> I don't think it matters whether it's userspace or kernel that does  
> the suspending and I'm yet to see a good reason for it to be  
> [required to be] done from userspace.

The reason it's _required_ to be done from userspace is that  
userspace is the only one which can figure out "These processes need  
to run for suspend to work", and then let those processes continue  
running after the freeze.  The *ONLY* reason this even stops  
processes at all is so we can do the post-device-mapper-snapshot code  
with very little usably-free RAM (IE: only about 1MB for a standard  
desktop system).


> In this first step, you've reinvented the first part of the current  
> freezer implementation. The reason we don't use a real signal is  
> precisely so we can have an untrappable SIGSTOP. In this regard, I  
> particularly remember Win4Lin from a few years ago. It would die if  
> you sent it a real signal, so we had to do it this way. No doubt  
> there are other instances I'm not aware of.

Well, you *do* want it to have semi-signal semantics, processes which  
receive it must not get back to userspace code so that they don't  
start allocating more memory when we're trying to do the freeze.  You  
also don't want a process to be able to trap it (IE: like SIGSTOP or  
SIGKILL).

On the other hand, it should be delivered asynchronously (IE: It  
doesn't break an interruptable sleep or respond to most is-a-signal- 
present checks).  You don't actually care if its sleeping in the  
kernel somewhere, just as long as it doesn't allocate much memory.

You would probably need a new signal "SIGFREEZE" which causes the  
process to be ignored as runnable the next time they schedule but  
never actually gets delivered, and a "SIGUNFREEZE" which does the  
reverse.  That way userspace could selectively resume processes based  
on its policy of "this needs to run for hibernation".


>> (2) Userspace process sends SIGCONT to only those processes which  
>> are necessary for sync and a device-mapper snapshot.
>
> How do you determine which ones are needed?

It's userspace's job to know which ones are needed.  For example, if  
you are hibernating over NFS then you need to resume the various NFS/ 
RPC daemons and threads.

> Why stop them in the first place?

So they aren't allocating memory when we are doing the device-mapper  
snapshot.


>> (3) Userspace calls sys_snapshot_kernel(snapshot_overhead_pages)
>>    (a) Kernel starts freeing memory and swapping stuff out to make  
>> room for a copy of *kernel* memory (not pagecache, not process  
>> RAM).   It does the same for at least snapshot_overhead_pages  
>> extra (used by userspace later).  It then allocates this memory to  
>> keep it from going away.  Since most processes are stopped we  
>> won't have much else competing with us for the RAM.
>
> Ok. So now you also need processes running that are needed for  
> swapping, because freeing that memory might involve swapping. Fully  
> agree with the logic though (not really surprising - this is what I  
> do in Suspend2^wTuxOnIce).
>
>>    (a) Kernel uses the device-mapper up-call-into-filesystem  
>> machinery to get all mounted filesystems synced and ready for a DM  
>> snapshot.  This may include sending data via the userspace  
>> processes resumed in (2).  Any deadlocks here are userspace's  
>> fault (see (2)).  Will need some modification to handle doing  
>> multiple blockdevs at a time.  Anything using FUSE is basically  
>> perma-synced anyways (no dep-handling needed), and anything using  
>> loop should already be handled by DM.  This includes allocating  
>> memory for the basic snapshot
>> datastructures.
>>    (b) At this point all blockdev operations should be halted and  
>> disk caches flushed; that's all we care about.
>>    (c) Go through the device tree and quiesce DMA and shut off  
>> interrupts.  Since all the disks are synced this is easy.
>>    (d) Use IPMIs again to get all the CPUs together, which should  
>> be easy as most processes are sleeping in IO or SIGSTOPed, and  
>> we're getting no interrupts.
>>    (e) One CPU turns off all interrupts on itself and takes an  
>> atomic snapshot of kernel memory into the previously allocated  
>> storage.  Once again, does not include pagecache.  The kernel also  
>> records a list of what pages *are* included in the pagecache.  It  
>> then marks all userspace pages as copy-on-write.
>
> Hotplugging cpus (when all those locking issues are taken care of)  
> is simpler.  Prior to cpu hotplugging, I used IMPIs to put  
> secondary cpus into a tight loop, so I know it's possible to do it  
> this way too.

It may be simpler, but it really screws up things like cpusets,  
processor affinity, etc.  It also ties hibernation to the presently  
very-flakey CPU-hotplug support, which is probably not what we want.


> That way, though, you have less flexibility. What if a cpu really  
> is plugged in between hibernate and resume? With cpu hotplugging,  
> it's handled properly and transparently.  Without cpu hotplugging,  
> you could be using uninitialised data after the atomic restore.

IMHO if the user pulls a CPU while the box is hibernated, then he/she  
gets what he/she deserves.  If you really want to support that, then  
the user must do the hotplug operation *manually* before suspending.   
Anything else is just going to be shooting ourselves in the foot  
repeatedly.

> Marking userspace as COW makes things more complicated, too. You  
> then have to add code to the COW handling to update the list of  
> pages that need to be saved, and you reduce the reliability of the  
> whole process. You can't predict beforehand how many of these COW  
> pages are going to be needed, and therefore can't know how much  
> memory to free earlier on in the process. If you run out of memory,  
> what will be the effect?

You could pretty easily have a spare 128MB swap partition somewhere  
which is not used during system operation but is "swapon"ed by  
userspace after the COW snapshot to provide extra backing store.

>>    (f) That CPU finalizes the modified DM snapshot using the  
>> previously-allocated memory.
>>    (g) That CPU frees up the snapshot_overhead_pages memory  
>> allocated during step (a) for userspace to use.
>>    (h) The CPU does the equivalent of a "swapoff -a" without  
>> overwriting any data already on any swap device(s).
>
> You still need to remember what swap you're going to use to write  
> the image. You'll probably want to get this information (and  
> allocate the swap) sooner rather than later so that you're not  
> racing against the memory freeing earlier, and don't run into  
> issues with bmapping the pages or having enough memory to record  
> the bdevs & sector numbers (not usually an issue, but if swap is  
> highly fragmented...).

Who says we have to use this swap to write the image?  That may be  
the default use-case, but it's certainly shouldn't be mandatory.   
Really, for the write-image-to-swap case you would just need to  
preallocate sufficient memory for the bmap tables beforehand, then  
populate them at this phase.

>>    (i) The CPU then IPMI-signals the other CPUs to wake them up
>>    (j) The kernel returns a FD-reference to the snapshot and the  
>> read-only halves of the CoW pagecache to the process which called  
>> sys_snapshot_kernel().
>
> Readonly halves? I don't get that, sorry.

Well, each page is copy-on-write, so the FD reference would always  
provide access to the original page data, whereas the processes may  
end up copying the page so they can write to it.  The trick would be  
that shared pages need to remain shared between processes even after  
the copy-on-write.  This is likely to be the trickiest part.

>> (4) The userspace process now has a reference to the copy of the  
>> kernel pages and the unmodified pagecache pages.  Since 99% of the  
>> processes aren't running, we aren't going to be having to CoW many  
>> of the pagecache pages.
>
> Mmm, but you still don't know how many.

Yes, but at this point we're basically running a segment of userspace  
with full kernel services available.  Like I said above we can just  
add a dedicated 128MB swap device to provide some spare backing  
store.  When you start running low on memory it might even page out  
to the swap device a part of the atomic copy of kernel memory.

>> (5) The userspace process uses read() or other syscalls to get  
>> data out of the kernel-snapshot FD in small chunks, within its  
>> snapshot_overhead_pages limit.  It compresses these and writes  
>> them out to the snapshot-storage blockdev (must not be mounted  
>> during snapshot), or to any network server.
>>
>> (6) The userspace process syncs the disks and halts the system.   
>> Any changed filesystem pages after the pseudo-DM-snapshot should  
>> have been stored in semi-volatile storage somewhere and will be  
>> discarded on the next reboot.
>
> Are you thinking the changed filesystem pages are caught by COW?  
> (AFAIUI, kernel writes aren't). If (as I expect), you're thinking  
> about filesystem writes to DM based storage, what about non DM- 
> based filesystem pages?

No, but a kernel write to a DM-snapshot calls the DM-snapshot code  
which copies the segment(s) to the snapshot device and modifies them  
there.  Basically disk-filesystem pagecache pages would be synced and  
protected by the DM snapshot, while anonymous memory pages would be  
CoW-ed.  And it might not be an unreasonable requirement to state  
that the disk-based filesystems must all be mapped through DM- 
snapshot devices (even just straight 1:1 linear mappings), so that  
they can be trivially snapshotted.

>> So basically your hibernate-overhead would consist of:
>>    (1) The pages necessary for the atomic snapshot of kernel  
>> memory and the list of pagecache pages at that time
>>    (2) A little memory necessary for the kernel non-persistent DM  
>> snapshot datastructures.
>>    (3) The snapshot_overhead_pages needed by userspace.
>>
>> If you're using swap devices then you can save 99% of the state of  
>> the running kernel with an initial swapout overhead of virtually  
>> nothing beyond the size of the unswappable kernel memory.
>
> FWIW, let me note an important variation from how Suspend2 works;  
> it might provide food for thought. In Suspend2, we treat the  
> processes that remain stopped throughout the whole process  
> specially. We write their data to disk before the atomic copy  
> (usually 70 or 80% of memory), and then use the memory they occupy  
> for the destination of the atomic copy. This further reduces the  
> amount of memory that has to be freed, almost always to zero.

I suppose you could record swap-outs done to SIGFREEZEd processes  
specially, so that they would be swapped in again before resuming  
userspace.  That would effectively result in the same thing.

Cheers,
Kyle Moffett

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Nigel Cunningham]

[-- Attachment #1.1: Type: text/plain, Size: 4877 bytes --]

Hi.

On Friday 06 July 2007 11:19:32 Kyle Moffett wrote:
> On Jul 05, 2007, at 19:35:11, Nigel Cunningham wrote:
> > On Friday 06 July 2007 09:20:43 Benjamin Herrenschmidt wrote:
> >> No, the freezer creates all those places what are harmful for a  
> >> task to block because they will break the freezer :-)
> >
> > Nice try :) Okay then, you remove the freezer, try hibernating,  
> > then get back to me after you've fixed your filesystem because some  
> > process that wasn't frozen started writing things after the atomic  
> > copy (making the on disk filesystem inconsistent with the snapshot).
> 
> Umm, this thread is NOT ABOUT HIBERNATING!!!  Please go back and read  
> the subject, specifically the "suspend to RAM" parts :-D.  When your  
> hardware can put itself to sleep and atomically preserve memory as it  
> does so, you don't need an atomic copy.  For Real Suspend(TM) (IE:  
> Suspend-to-RAM), the list of things to do is short and simple:

We agreed a while back that you don't need the freezer for suspend to ram. As 
far as I was aware, we went off-topic, so the topic is out of date.

> 1)  Stop DMA and put most hardware into low-power states (stops all  
> interrupt sources)
> 2)  Ensure that the other CPUs have finished any trailing interrupt  
> handlers and put them to sleep
> 3)  Put the interrupt-controllers into low-power state
> 4)  Go to sleep
> 
> > As Pavel rightly said, you can get rid of the freezer, but you're  
> > only going to have to implement another one that does the  
> > essentially the same thing, even if it is at some other level.
> 
> How about a freezer whose job it is to "wait for pending hard  
> interrupts to complete when we have already guaranteed that we won't  
> get any more"?  That part should be really *REALLY* easy.  You don't  
> need to care about either userspace processes or kernel threads at  
> all.  Specifically, Step 1 consists of:
> 
> suspend_device(dev)
> {
> 	set_no_bind_flag(dev);
> 	for (dev->subdevices)
> 		suspend_device(dev);
> 	set_no_io_flag(dev);
> 	wait_for_in_progress_dma(dev);
> 	turn_off_interrupts(dev);
> 	go_to_low_power_state(dev);
> }
> 
> After you've set the "no_bind" flag, you won't get any *new*  
> subdevices trying to bind, therefore it's safe to iterate over the  
> list of present sub-devices and suspend them.  Once those are  
> suspended and in low-power states you can set a "no_io" flag to  
> prevent the driver from submitting more IO.  At that point you can  
> lazily wait for existing DMA/IO/interrupts to finish on the device,  
> since *NOBODY* will be submitting them anymore, and we certainly  
> aren't probing for new devices.  Then you can just turn off the power  
> to the device.  When all the leaf devices are off, the parent device  
> can be turned off because everything waiting on the leaf devices is  
> blocked on them and won't unblock until the parent device *AND* the  
> leaf device are turned on again, in that order.

For suspending, yes. For hibernating, that's not enough, because other 
processes can still be happily allocating and freeing memory, and will only 
get stopped when they try to do i/o or such like. If you're trying to make 
hibernation reliable, you need to be able to reliably check whether you're 
going to have enough storage for the image you're preparing, and enough 
memory for the atomic copy and so on. That's why the freezer is needed for 
hibernation. If you don't have it, any hibernation implementation you make is 
going to be only as reliable as the extent to which the system is otherwise 
idle.
 
> Scheduling and userspace are all still fully enabled in this  
> scenario.  Once all your devices are turned off, the only remaining  
> running threads will be those which haven't done IO since the  
> beginning of the suspend.  We can then disable preemption, turn off  
> the timer interrupts, and tell the other CPUs to park all their  
> remaining threads in schedule() and sleep.  Then we put the IRQ  
> controller to sleep and go to sleep ourselves.  If our driver model  
> locking is sufficient to handle putting a parent device to sleep  
> while threads are sleeping on a child device then there are exactly 0  
> problems.
> 
> Resuming is basically running the whole process in reverse.  Runtime- 
> suspend is achieved by not setting the 'no_io' or 'no_bind' flags and  
> putting selective device-subtrees to sleep without doing anything to  
> the rest of the system.

Fully agree when it comes to suspend to ram. There, this process should work, 
so far as I can see. But as I said, we went - to one degree or another - off 
topic, and did discuss hibernation too.

Regards,

Nigel
-- 
See http://www.tuxonice.net for Howtos, FAQs, mailing
lists, wiki and bugzilla info.

[-- Attachment #1.2: Type: application/pgp-signature, Size: 189 bytes --]

[-- Attachment #2: Type: text/plain, Size: 0 bytes --]

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Benjamin Herrenschmidt]

> How about a freezer whose job it is to "wait for pending hard  
> interrupts to complete when we have already guaranteed that we won't  
> get any more"?  That part should be really *REALLY* easy.  You don't  
> need to care about either userspace processes or kernel threads at  
> all.  Specifically, Step 1 consists of:

Well, waiting for pending DMA and making sure to not trigger more
activity is what driver suspend() is supposed to do. With the ability
for simple drivers that can cope with it to just basically use a
late_suspend(), called after IRQs are off, that basically does what you
describe: wait for pending HW tasks to complete (polling) and turn the
damn thing off.

Note that the later is really a shortcut for somewhat dump and directly
accessible devices (PCI comes to mind). Things like USB has to use the
"normal" mechanism of blocking IOs etc... at suspend(), at least, USB
devices have to since the USB HC will not issue any new URBs. (And will
return them with a nice error code which is a perfect way to deal with
it in driver, been there, it works fine, once we fixed the races in the
USB host code itself, which I think we pretty much did by now).

> Scheduling and userspace are all still fully enabled in this  
> scenario.  Once all your devices are turned off, the only remaining  
> running threads will be those which haven't done IO since the  
> beginning of the suspend.  We can then disable preemption, turn off  
> the timer interrupts, and tell the other CPUs to park all their  
> remaining threads in schedule() and sleep.  Then we put the IRQ  
> controller to sleep and go to sleep ourselves.  If our driver model  
> locking is sufficient to handle putting a parent device to sleep  
> while threads are sleeping on a child device then there are exactly 0  
> problems.

What you propose is basically a slightly over-simplistic version of what
I think (and Paulus think) should be done. We do need to do it via
driver callbacks down the tree since only drivers can know how to deal
with their DMA etc... and ordering need to be respected, but that's
basically it.

And guess what ? It's what we do on powerbooks, and it works fine,
without a freezer :-)

Ben.

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Rafael J. Wysocki]

On Friday, 6 July 2007 05:59, Benjamin Herrenschmidt wrote:
> 
> > How about a freezer whose job it is to "wait for pending hard  
> > interrupts to complete when we have already guaranteed that we won't  
> > get any more"?  That part should be really *REALLY* easy.  You don't  
> > need to care about either userspace processes or kernel threads at  
> > all.  Specifically, Step 1 consists of:
> 
> Well, waiting for pending DMA and making sure to not trigger more
> activity is what driver suspend() is supposed to do. With the ability
> for simple drivers that can cope with it to just basically use a
> late_suspend(), called after IRQs are off, that basically does what you
> describe: wait for pending HW tasks to complete (polling) and turn the
> damn thing off.
> 
> Note that the later is really a shortcut for somewhat dump and directly
> accessible devices (PCI comes to mind). Things like USB has to use the
> "normal" mechanism of blocking IOs etc... at suspend(), at least, USB
> devices have to since the USB HC will not issue any new URBs. (And will
> return them with a nice error code which is a perfect way to deal with
> it in driver, been there, it works fine, once we fixed the races in the
> USB host code itself, which I think we pretty much did by now).
> 
> > Scheduling and userspace are all still fully enabled in this  
> > scenario.  Once all your devices are turned off, the only remaining  
> > running threads will be those which haven't done IO since the  
> > beginning of the suspend.  We can then disable preemption, turn off  
> > the timer interrupts, and tell the other CPUs to park all their  
> > remaining threads in schedule() and sleep.  Then we put the IRQ  
> > controller to sleep and go to sleep ourselves.  If our driver model  
> > locking is sufficient to handle putting a parent device to sleep  
> > while threads are sleeping on a child device then there are exactly 0  
> > problems.
> 
> What you propose is basically a slightly over-simplistic version of what
> I think (and Paulus think) should be done. We do need to do it via
> driver callbacks down the tree since only drivers can know how to deal
> with their DMA etc... and ordering need to be respected, but that's
> basically it.
> 
> And guess what ? It's what we do on powerbooks, and it works fine,
> without a freezer :-)

On powerbooks you disable the nonboot CPUs before suspending devices, which
simplifies things _a_ _lot_ in comparison with the general case.

If you additionally disable kernel preemption, then you don't need anything
like the freezer anyway in that case (except for detecting situations in which
the suspend process can deadlock with a task stuck in the D state holding a
lock, but that's overkill).

Greetings,
Rafael


-- 
"Premature optimization is the root of all evil." - Donald Knuth

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Benjamin Herrenschmidt]

> On powerbooks you disable the nonboot CPUs before suspending devices, which
> simplifies things _a_ _lot_ in comparison with the general case.

Not that much actually. It avoids having to gather them all in the last
stage at the low level but that isn't really related to what we are
talking about right now.

> If you additionally disable kernel preemption, then you don't need anything
> like the freezer anyway in that case (except for detecting situations in which
> the suspend process can deadlock with a task stuck in the D state holding a
> lock, but that's overkill).

kernel preemption and SMP have nothing to do with it. You can still
schedule because one of the driver suspend() callback is schedul'ing,
maybe waiting for some requests to complete, etc... Happens typically
with the disk suspend callbacks.

I agree that it does lower the likelyness of having a userspace process
pound on the wrong driver at the wrong time though.

Ben.

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Alan Stern]

On Fri, 6 Jul 2007, Benjamin Herrenschmidt wrote:

> What you propose is basically a slightly over-simplistic version of what
> I think (and Paulus think) should be done. We do need to do it via
> driver callbacks down the tree since only drivers can know how to deal
> with their DMA etc... and ordering need to be respected, but that's
> basically it.
> 
> And guess what ? It's what we do on powerbooks, and it works fine,
> without a freezer :-)

I wish you'd stop saying that.  Have you ever done any serious testing?

Here's something to try:  Add a time delay to the end of hub_suspend in
drivers/usb/core/hub.c, so you can provoke a race manually.  Then while
one of your root hubs is being suspended and the system is waiting in
that delay, either plug in a new USB device to that hub or unplug an
existing device.

Be sure that CONFIG_USB_DEBUG is on so that we can figure out what 
happened after the fact.

Alan Stern

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Alan Stern]

On Thu, 5 Jul 2007, Kyle Moffett wrote:

> Umm, this thread is NOT ABOUT HIBERNATING!!!  Please go back and read  
> the subject, specifically the "suspend to RAM" parts :-D.

But it _is_ about the freezer; see the "Remove process freezer" part.  
:-)  Since the freezer is used during hibernation, hibernation is a
legitimate topic.

>  When your  
> hardware can put itself to sleep and atomically preserve memory as it  
> does so, you don't need an atomic copy.  For Real Suspend(TM) (IE:  
> Suspend-to-RAM), the list of things to do is short and simple:
> 
> 1)  Stop DMA and put most hardware into low-power states (stops all  
> interrupt sources)
> 2)  Ensure that the other CPUs have finished any trailing interrupt  
> handlers and put them to sleep
> 3)  Put the interrupt-controllers into low-power state
> 4)  Go to sleep

Your short and simple list omits a few crucial items:

A)  Decide what to do about remote wakeup requests.
B)  Prevent I/O requests from resuming devices that have been 
suspended.
C)  Prevent devices and drivers from being registered or unregistered; 
in particular decide what to do about hot-plug or hot-unplug events.
D)  Block driver bind or unbind calls.

Any of these things is capable of screwing up the course of events.
(In fact A _should_ be allowed to abort a suspend.)

> > As Pavel rightly said, you can get rid of the freezer, but you're  
> > only going to have to implement another one that does the  
> > essentially the same thing, even if it is at some other level.
> 
> How about a freezer whose job it is to "wait for pending hard  
> interrupts to complete when we have already guaranteed that we won't  
> get any more"?  That part should be really *REALLY* easy.  You don't  
> need to care about either userspace processes or kernel threads at  
> all.  Specifically, Step 1 consists of:
> 
> suspend_device(dev)
> {
> 	set_no_bind_flag(dev);
> 	for (dev->subdevices)
> 		suspend_device(dev);
> 	set_no_io_flag(dev);
> 	wait_for_in_progress_dma(dev);
> 	turn_off_interrupts(dev);
> 	go_to_low_power_state(dev);
> }
> 
> After you've set the "no_bind" flag, you won't get any *new*  
> subdevices trying to bind, 

So what happens if a new subdevice arrives at the wrong time?  Do you 
block instead of binding it?  While holding a mutex needed to suspend 
the parent device?

What about drivers trying to bind to existing devices?

What happens to I/O requests submitted after the "no_io" flag is set?  
The driver will have to block them, effectively creating its own little 
"freezer".

> therefore it's safe to iterate over the  
> list of present sub-devices and suspend them.  Once those are  
> suspended and in low-power states you can set a "no_io" flag to  
> prevent the driver from submitting more IO.  At that point you can  
> lazily wait for existing DMA/IO/interrupts to finish on the device,  
> since *NOBODY* will be submitting them anymore, and we certainly  
> aren't probing for new devices.  Then you can just turn off the power  
> to the device.  When all the leaf devices are off, the parent device  
> can be turned off because everything waiting on the leaf devices is  
> blocked on them and won't unblock until the parent device *AND* the  
> leaf device are turned on again, in that order.

This is a lot like what we already do.  The differences are:

	There is nothing corresponding to your "no-bind" flag.

	Most drivers don't have anything like your "no_io" flag;
	they assume that nobody will be around to submit an I/O
	request.

> Scheduling and userspace are all still fully enabled in this  
> scenario.  Once all your devices are turned off, the only remaining  
> running threads will be those which haven't done IO since the  
> beginning of the suspend.  We can then disable preemption, turn off  
> the timer interrupts, and tell the other CPUs to park all their  
> remaining threads in schedule() and sleep.  Then we put the IRQ  
> controller to sleep and go to sleep ourselves.  If our driver model  
> locking is sufficient to handle putting a parent device to sleep  
> while threads are sleeping on a child device then there are exactly 0  
> problems.
> 
> Resuming is basically running the whole process in reverse.  Runtime- 
> suspend is achieved by not setting the 'no_io' or 'no_bind' flags and  
> putting selective device-subtrees to sleep without doing anything to  
> the rest of the system.

Nobody doubts that suspend can be made to work without the freezer.  
The point is that doing it this way dumps a bunch of extra 
responsibility on drivers.

Alan Stern

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Pavel Machek]

Hi!

> So I think Matthew is totally right. In fact, the presence of the
> freezer is the main reason why Paulus so far NACKed Johannes attempts at
> merging the PPC PM code with the generic code in kernel/power.c
> 
> We've been doing fine without it so far and intend to continue to do
> so.

How well does it work on SMP PPC?
									Pavel

-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Benjamin Herrenschmidt]

On Tue, 2007-07-03 at 05:29 +0100, Matthew Garrett wrote:
> Suspend to RAM on a machine with / on a fuse filesystem turns out to be 
> a screaming nightmare - either the suspend fails because syslog (for 
> instance) can't be frozen, or the machine deadlocks for some other 
> reason I haven't tracked down. We could "fix" fuse, or alternatively we 
> could do what we do for suspend to RAM on other platforms (PPC and APM) 
> and just not use the freezer.
> 
> Signed-off-by: Matthew Garrett <mjg59@srcf.ucam.org>

Acked-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>

(And with much pleasure :-)

Cheers,
Ben.

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Benjamin Herrenschmidt]

On Tue, 2007-07-03 at 05:29 +0100, Matthew Garrett wrote:
> Suspend to RAM on a machine with / on a fuse filesystem turns out to be 
> a screaming nightmare - either the suspend fails because syslog (for 
> instance) can't be frozen, or the machine deadlocks for some other 
> reason I haven't tracked down. We could "fix" fuse, or alternatively we 
> could do what we do for suspend to RAM on other platforms (PPC and APM) 
> and just not use the freezer.

The main reason for deadlocks is because we do a sys_sync() after the
freeze, which we shouldn't do.

Ben.

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Oliver Neukum]

Am Dienstag, 3. Juli 2007 schrieb Matthew Garrett:
> Suspend to RAM on a machine with / on a fuse filesystem turns out to be 
> a screaming nightmare - either the suspend fails because syslog (for 
> instance) can't be frozen, or the machine deadlocks for some other 
> reason I haven't tracked down. We could "fix" fuse, or alternatively we 
> could do what we do for suspend to RAM on other platforms (PPC and APM) 
> and just not use the freezer.

Only if you want to audit all character devices' read() and write()
methods for races against suspend().
/ on fuse is a bad idea.

	Regards
		Oliver

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Romano Giannetti]

On Tue, 2007-07-03 at 05:29 +0100, Matthew Garrett wrote:
> or alternatively we  could do what we do for suspend to RAM on other
> platforms (PPC and APM) and just not use the freezer.

As a data point, I am running with this patch on top of 2.6.21.2 the
last 3+ weeks, with an average of 5/6 STR cycles a day, and had no
problems at all. (Sony vaio pcg-fx701). Just normal work, I didn't try
to stress the thing, but I have quite a few times suspended/resumed over
a big compile without a glitch.

What are the risks of this patch supposed to be?

Romano



--
La presente comunicación tiene carácter confidencial y es para el exclusivo uso del destinatario indicado en la misma. Si Ud. no es el destinatario indicado, le informamos que cualquier forma de distribución, reproducción o uso de esta comunicación y/o de la información contenida en la misma están estrictamente prohibidos por la ley. Si Ud. ha recibido esta comunicación por error, por favor, notifíquelo inmediatamente al remitente contestando a este mensaje y proceda a continuación a destruirlo. Gracias por su colaboración.

This communication contains confidential information. It is for the exclusive use of the intended addressee. If you are not the intended addressee, please note that any form of distribution, copying or use of this communication or the information in it is strictly prohibited by law. If you have received this communication in error, please immediately notify the sender by reply e-mail and destroy this message. Thank you for your cooperation.

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Oliver Neukum]

Am Dienstag, 3. Juli 2007 schrieb Romano Giannetti:
> On Tue, 2007-07-03 at 05:29 +0100, Matthew Garrett wrote:
> > or alternatively we  could do what we do for suspend to RAM on other
> > platforms (PPC and APM) and just not use the freezer.
> 
> As a data point, I am running with this patch on top of 2.6.21.2 the
> last 3+ weeks, with an average of 5/6 STR cycles a day, and had no
> problems at all. (Sony vaio pcg-fx701). Just normal work, I didn't try
> to stress the thing, but I have quite a few times suspended/resumed over
> a big compile without a glitch.
> 
> What are the risks of this patch supposed to be?

You did this test only while stressing normal block devices. Try
provoking races in character devices.

	Regards
		Oliver

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Rafael J. Wysocki]

On Tuesday, 3 July 2007 09:37, Romano Giannetti wrote:
> On Tue, 2007-07-03 at 05:29 +0100, Matthew Garrett wrote:
> > or alternatively we  could do what we do for suspend to RAM on other
> > platforms (PPC and APM) and just not use the freezer.
> 
> As a data point, I am running with this patch on top of 2.6.21.2 the
> last 3+ weeks, with an average of 5/6 STR cycles a day, and had no
> problems at all. (Sony vaio pcg-fx701). Just normal work, I didn't try
> to stress the thing, but I have quite a few times suspended/resumed over
> a big compile without a glitch.
> 
> What are the risks of this patch supposed to be?

See https://lists.linux-foundation.org/pipermail/linux-pm/2007-June/012847.html

Greetings,
Rafael


-- 
"Premature optimization is the root of all evil." - Donald Knuth

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Miklos Szeredi]

> > Suspend to RAM on a machine with / on a fuse filesystem turns out to be 
> > a screaming nightmare - either the suspend fails because syslog (for 
> > instance) can't be frozen, or the machine deadlocks for some other 
> > reason I haven't tracked down. We could "fix" fuse, or alternatively we 
> > could do what we do for suspend to RAM on other platforms (PPC and APM) 
> > and just not use the freezer.
> 
> Only if you want to audit all character devices' read() and write()
> methods for races against suspend().
> / on fuse is a bad idea.

What makes / special?  Why aren't all fuse filesystems affected?
Suspend isn't trying to do I/O on the root fs, is it?

Miklos

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Matthew Garrett]

On Tue, Jul 03, 2007 at 08:13:53AM +0200, Oliver Neukum wrote:

> Only if you want to audit all character devices' read() and write()
> methods for races against suspend().
> / on fuse is a bad idea.

Any driver that assumes that userspace will be frozen during suspend has 
been broken forever. That behaviour has never been guaranteed.
-- 
Matthew Garrett | mjg59@srcf.ucam.org

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Rafael J. Wysocki]

On Tuesday, 3 July 2007 14:13, Matthew Garrett wrote:
> On Tue, Jul 03, 2007 at 08:13:53AM +0200, Oliver Neukum wrote:
> 
> > Only if you want to audit all character devices' read() and write()
> > methods for races against suspend().
> > / on fuse is a bad idea.
> 
> Any driver that assumes that userspace will be frozen during suspend has 
> been broken forever. That behaviour has never been guaranteed.

Can we please fix those drivers _first_, then?

Rafael


-- 
"Premature optimization is the root of all evil." - Donald Knuth

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Rafael J. Wysocki]

On Tuesday, 3 July 2007 06:29, Matthew Garrett wrote:
> Suspend to RAM on a machine with / on a fuse filesystem turns out to be 
> a screaming nightmare - either the suspend fails because syslog (for 
> instance) can't be frozen, or the machine deadlocks for some other 
> reason I haven't tracked down. We could "fix" fuse, or alternatively we 
> could do what we do for suspend to RAM on other platforms (PPC and APM) 
> and just not use the freezer.
> 
> Signed-off-by: Matthew Garrett <mjg59@srcf.ucam.org>

Could you please rediff against the current -mm tree?

There are some patches in there that this will clash with.

I still think that this is a mistake, BTW.  Please see the Alan's post at

https://lists.linux-foundation.org/pipermail/linux-pm/2007-June/012847.html

Greetings,
Rafael


-- 
"Premature optimization is the root of all evil." - Donald Knuth

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Rafael J. Wysocki]

On Tuesday, 3 July 2007 07:49, Benjamin Herrenschmidt wrote:
> On Tue, 2007-07-03 at 05:29 +0100, Matthew Garrett wrote:
> > Suspend to RAM on a machine with / on a fuse filesystem turns out to be 
> > a screaming nightmare - either the suspend fails because syslog (for 
> > instance) can't be frozen, or the machine deadlocks for some other 
> > reason I haven't tracked down. We could "fix" fuse, or alternatively we 
> > could do what we do for suspend to RAM on other platforms (PPC and APM) 
> > and just not use the freezer.
> > 
> > Signed-off-by: Matthew Garrett <mjg59@srcf.ucam.org>
> 
> Acked-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
> 
> (And with much pleasure :-)

Clashes with some code already in -mm.

Greetings,
Rafael


-- 
"Premature optimization is the root of all evil." - Donald Knuth

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Rafael J. Wysocki]

On Tuesday, 3 July 2007 07:51, Benjamin Herrenschmidt wrote:
> On Tue, 2007-07-03 at 05:29 +0100, Matthew Garrett wrote:
> > Suspend to RAM on a machine with / on a fuse filesystem turns out to be 
> > a screaming nightmare - either the suspend fails because syslog (for 
> > instance) can't be frozen, or the machine deadlocks for some other 
> > reason I haven't tracked down. We could "fix" fuse, or alternatively we 
> > could do what we do for suspend to RAM on other platforms (PPC and APM) 
> > and just not use the freezer.
> 
> The main reason for deadlocks is because we do a sys_sync() after the
> freeze, which we shouldn't do.

So why don't we remove the sys_sync() from freeze_processes() instead?

Greetings,
Rafael


-- 
"Premature optimization is the root of all evil." - Donald Knuth

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Rafael J. Wysocki]

On Tuesday, 3 July 2007 15:08, Rafael J. Wysocki wrote:
> On Tuesday, 3 July 2007 07:51, Benjamin Herrenschmidt wrote:
> > On Tue, 2007-07-03 at 05:29 +0100, Matthew Garrett wrote:
> > > Suspend to RAM on a machine with / on a fuse filesystem turns out to be 
> > > a screaming nightmare - either the suspend fails because syslog (for 
> > > instance) can't be frozen, or the machine deadlocks for some other 
> > > reason I haven't tracked down. We could "fix" fuse, or alternatively we 
> > > could do what we do for suspend to RAM on other platforms (PPC and APM) 
> > > and just not use the freezer.
> > 
> > The main reason for deadlocks is because we do a sys_sync() after the
> > freeze, which we shouldn't do.
> 
> So why don't we remove the sys_sync() from freeze_processes() instead?

The patch follows (untested).

Greetings,
Rafael


---
From: Rafael J. Wysocki <rjw@sisk.pl>

We shouldn't sync filesystems from within the freezer, because it's not needed
for suspend to RAM and leads to problems with FUSE.

Signed-off-by: Rafael J. Wysocki <rjw@sisk.pl>
---
 kernel/power/disk.c    |    4 ++++
 kernel/power/process.c |    1 -
 2 files changed, 4 insertions(+), 1 deletion(-)

Index: linux-2.6.22-rc7/kernel/power/disk.c
===================================================================
--- linux-2.6.22-rc7.orig/kernel/power/disk.c
+++ linux-2.6.22-rc7/kernel/power/disk.c
@@ -296,6 +296,10 @@ int hibernate(void)
 {
 	int error;
 
+	printk("Syncing filesystems ... \n");
+	sys_sync();
+	printk("done.\n");
+
 	mutex_lock(&pm_mutex);
 	/* The snapshot device should not be opened while we're running */
 	if (!atomic_add_unless(&snapshot_device_available, -1, 0)) {
Index: linux-2.6.22-rc7/kernel/power/process.c
===================================================================
--- linux-2.6.22-rc7.orig/kernel/power/process.c
+++ linux-2.6.22-rc7/kernel/power/process.c
@@ -190,7 +190,6 @@ int freeze_processes(void)
 	if (error)
 		return error;
 
-	sys_sync();
 	error = try_to_freeze_tasks(FREEZER_KERNEL_THREADS);
 	if (error)
 		return error;

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Oliver Neukum]

Am Dienstag, 3. Juli 2007 schrieb Rafael J. Wysocki:
> On Tuesday, 3 July 2007 15:08, Rafael J. Wysocki wrote:
> > On Tuesday, 3 July 2007 07:51, Benjamin Herrenschmidt wrote:
> > > On Tue, 2007-07-03 at 05:29 +0100, Matthew Garrett wrote:
> > > > Suspend to RAM on a machine with / on a fuse filesystem turns out to be 
> > > > a screaming nightmare - either the suspend fails because syslog (for 
> > > > instance) can't be frozen, or the machine deadlocks for some other 
> > > > reason I haven't tracked down. We could "fix" fuse, or alternatively we 
> > > > could do what we do for suspend to RAM on other platforms (PPC and APM) 
> > > > and just not use the freezer.
> > > 
> > > The main reason for deadlocks is because we do a sys_sync() after the
> > > freeze, which we shouldn't do.
> > 
> > So why don't we remove the sys_sync() from freeze_processes() instead?
> 
> The patch follows (untested).
> 
> Greetings,
> Rafael
> 
> 
> ---
> From: Rafael J. Wysocki <rjw@sisk.pl>
> 
> We shouldn't sync filesystems from within the freezer, because it's not needed
> for suspend to RAM and leads to problems with FUSE.

This seems fishy. Swsusp needs enough clean memory to make enough
room for the image. If you sync before you freeze, the running tasks can
redirty memory.
What makes you sure that you don't die as shrink_all_memory() writes out
pages?

	Regards
		Oliver

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Rafael J. Wysocki]

On Tuesday, 3 July 2007 19:20, Oliver Neukum wrote:
> Am Dienstag, 3. Juli 2007 schrieb Rafael J. Wysocki:
> > On Tuesday, 3 July 2007 15:08, Rafael J. Wysocki wrote:
> > > On Tuesday, 3 July 2007 07:51, Benjamin Herrenschmidt wrote:
> > > > On Tue, 2007-07-03 at 05:29 +0100, Matthew Garrett wrote:
> > > > > Suspend to RAM on a machine with / on a fuse filesystem turns out to be 
> > > > > a screaming nightmare - either the suspend fails because syslog (for 
> > > > > instance) can't be frozen, or the machine deadlocks for some other 
> > > > > reason I haven't tracked down. We could "fix" fuse, or alternatively we 
> > > > > could do what we do for suspend to RAM on other platforms (PPC and APM) 
> > > > > and just not use the freezer.
> > > > 
> > > > The main reason for deadlocks is because we do a sys_sync() after the
> > > > freeze, which we shouldn't do.
> > > 
> > > So why don't we remove the sys_sync() from freeze_processes() instead?
> > 
> > The patch follows (untested).
> > 
> > Greetings,
> > Rafael
> > 
> > 
> > ---
> > From: Rafael J. Wysocki <rjw@sisk.pl>
> > 
> > We shouldn't sync filesystems from within the freezer, because it's not needed
> > for suspend to RAM and leads to problems with FUSE.
> 
> This seems fishy. Swsusp needs enough clean memory to make enough
> room for the image. If you sync before you freeze, the running tasks can
> redirty memory.
> What makes you sure that you don't die as shrink_all_memory() writes out
> pages?

I don't think that would matter.

Still, I can remove the sync from the suspend code path only, leaving it in
the hibernation code path.  The patch will be bigger, but well.

Any objection to that?

Greetings,
Rafael


-- 
"Premature optimization is the root of all evil." - Donald Knuth

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Benjamin Herrenschmidt]

> I don't think that would matter.
> 
> Still, I can remove the sync from the suspend code path only, leaving it in
> the hibernation code path.  The patch will be bigger, but well.
> 
> Any objection to that?

Makes sense to sync before suspend tho, to limit the amount of dirty
non-written pages in case things go wrong.

Ben.

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Rafael J. Wysocki]

On Tuesday, 3 July 2007 23:35, Benjamin Herrenschmidt wrote:
> 
> > I don't think that would matter.
> > 
> > Still, I can remove the sync from the suspend code path only, leaving it in
> > the hibernation code path.  The patch will be bigger, but well.
> > 
> > Any objection to that?
> 
> Makes sense to sync before suspend tho, to limit the amount of dirty
> non-written pages in case things go wrong.

OK, below is the updated patch.

Greetings,
Rafael


---
From: Rafael J. Wysocki <rjw@sisk.pl>

The syncing of filesystems from within the freezer in not needed for suspend to
RAM and leads to problems with FUSE.  Change freeze_processes() so that it
doesn't execute sys_sync() and introduce the "syncing" version of it to be
called from the hibernation code paths.

Signed-off-by: Rafael J. Wysocki <rjw@sisk.pl>
---
 include/linux/freezer.h |   14 ++++++++++++--
 kernel/power/disk.c     |    2 +-
 kernel/power/main.c     |    6 ++++++
 kernel/power/process.c  |    8 +++++---
 kernel/power/user.c     |    2 +-
 5 files changed, 25 insertions(+), 7 deletions(-)

Index: linux-2.6.22-rc7/include/linux/freezer.h
===================================================================
--- linux-2.6.22-rc7.orig/include/linux/freezer.h	2007-07-04 00:21:26.000000000 +0200
+++ linux-2.6.22-rc7/include/linux/freezer.h	2007-07-04 00:22:37.000000000 +0200
@@ -62,7 +62,7 @@ static inline int thaw_process(struct ta
 }
 
 extern void refrigerator(void);
-extern int freeze_processes(void);
+extern int __freeze_processes(int sync_filesystems);
 extern void thaw_processes(void);
 
 static inline int try_to_freeze(void)
@@ -134,7 +134,7 @@ static inline void clear_freeze_flag(str
 static inline int thaw_process(struct task_struct *p) { return 1; }
 
 static inline void refrigerator(void) {}
-static inline int freeze_processes(void) { BUG(); return 0; }
+static inline int __freeze_processes(int s) { BUG(); return 0; }
 static inline void thaw_processes(void) {}
 
 static inline int try_to_freeze(void) { return 0; }
@@ -145,4 +145,14 @@ static inline int freezer_should_skip(st
 static inline void set_freezable(void) {}
 #endif
 
+static inline int freeze_processes(void)
+{
+	return __freeze_processes(0);
+}
+
+static inline int freeze_processes_with_sync(void)
+{
+	return __freeze_processes(1);
+}
+
 #endif	/* FREEZER_H_INCLUDED */
Index: linux-2.6.22-rc7/kernel/power/disk.c
===================================================================
--- linux-2.6.22-rc7.orig/kernel/power/disk.c	2007-07-04 00:21:26.000000000 +0200
+++ linux-2.6.22-rc7/kernel/power/disk.c	2007-07-04 00:21:44.000000000 +0200
@@ -281,7 +281,7 @@ static int prepare_processes(void)
 	int error = 0;
 
 	pm_prepare_console();
-	if (freeze_processes()) {
+	if (freeze_processes_with_sync()) {
 		error = -EBUSY;
 		unprepare_processes();
 	}
Index: linux-2.6.22-rc7/kernel/power/main.c
===================================================================
--- linux-2.6.22-rc7.orig/kernel/power/main.c	2007-07-04 00:21:26.000000000 +0200
+++ linux-2.6.22-rc7/kernel/power/main.c	2007-07-04 00:23:40.000000000 +0200
@@ -20,6 +20,7 @@
 #include <linux/resume-trace.h>
 #include <linux/freezer.h>
 #include <linux/vmstat.h>
+#include <linux/syscalls.h>
 
 #include "power.h"
 
@@ -231,6 +232,11 @@ static int enter_state(suspend_state_t s
 
 	if (!valid_state(state))
 		return -ENODEV;
+
+	printk("Syncing filesystems ... ");
+	sys_sync();
+	printk("done.\n");
+
 	if (!mutex_trylock(&pm_mutex))
 		return -EBUSY;
 
Index: linux-2.6.22-rc7/kernel/power/process.c
===================================================================
--- linux-2.6.22-rc7.orig/kernel/power/process.c	2007-07-04 00:21:26.000000000 +0200
+++ linux-2.6.22-rc7/kernel/power/process.c	2007-07-04 00:21:44.000000000 +0200
@@ -179,9 +179,9 @@ static int try_to_freeze_tasks(int freez
 }
 
 /**
- *	freeze_processes - tell processes to enter the refrigerator
+ *	__freeze_processes - tell processes to enter the refrigerator
  */
-int freeze_processes(void)
+int __freeze_processes(int sync_filesystems)
 {
 	int error;
 
@@ -190,7 +190,9 @@ int freeze_processes(void)
 	if (error)
 		return error;
 
-	sys_sync();
+	if (sync_filesystems)
+		sys_sync();
+
 	error = try_to_freeze_tasks(FREEZER_KERNEL_THREADS);
 	if (error)
 		return error;
Index: linux-2.6.22-rc7/kernel/power/user.c
===================================================================
--- linux-2.6.22-rc7.orig/kernel/power/user.c	2007-07-04 00:21:26.000000000 +0200
+++ linux-2.6.22-rc7/kernel/power/user.c	2007-07-04 00:21:44.000000000 +0200
@@ -153,7 +153,7 @@ static int snapshot_ioctl(struct inode *
 		mutex_lock(&pm_mutex);
 		error = pm_notifier_call_chain(PM_HIBERNATION_PREPARE);
 		if (!error) {
-			error = freeze_processes();
+			error = freeze_processes_with_sync();
 			if (error)
 				thaw_processes();
 		}

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Pavel Machek]

On Tue 2007-07-03 19:20:59, Oliver Neukum wrote:
> Am Dienstag, 3. Juli 2007 schrieb Rafael J. Wysocki:
> > On Tuesday, 3 July 2007 15:08, Rafael J. Wysocki wrote:
> > > On Tuesday, 3 July 2007 07:51, Benjamin Herrenschmidt wrote:
> > > > On Tue, 2007-07-03 at 05:29 +0100, Matthew Garrett wrote:
> > > > > Suspend to RAM on a machine with / on a fuse filesystem turns out to be 
> > > > > a screaming nightmare - either the suspend fails because syslog (for 
> > > > > instance) can't be frozen, or the machine deadlocks for some other 
> > > > > reason I haven't tracked down. We could "fix" fuse, or alternatively we 
> > > > > could do what we do for suspend to RAM on other platforms (PPC and APM) 
> > > > > and just not use the freezer.
> > > > 
> > > > The main reason for deadlocks is because we do a sys_sync() after the
> > > > freeze, which we shouldn't do.
> > > 
> > > So why don't we remove the sys_sync() from freeze_processes() instead?
> > 
> > The patch follows (untested).
> > 
> > Greetings,
> > Rafael
> > 
> > 
> > ---
> > From: Rafael J. Wysocki <rjw@sisk.pl>
> > 
> > We shouldn't sync filesystems from within the freezer, because it's not needed
> > for suspend to RAM and leads to problems with FUSE.
> 
> This seems fishy. Swsusp needs enough clean memory to make enough
> room for the image. If you sync before you freeze, the running tasks can
> redirty memory.
> What makes you sure that you don't die as shrink_all_memory() writes out
> pages?

Shrink_all_memory should just free enough memory, what's the problem?
Yes, we can have dirty memory, shrink_all_memory() can write that out
just fine.

-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Oliver Neukum]

Am Donnerstag, 5. Juli 2007 schrieb Pavel Machek:
> On Tue 2007-07-03 19:20:59, Oliver Neukum wrote:
> > Am Dienstag, 3. Juli 2007 schrieb Rafael J. Wysocki:
> > > On Tuesday, 3 July 2007 15:08, Rafael J. Wysocki wrote:
> > > > On Tuesday, 3 July 2007 07:51, Benjamin Herrenschmidt wrote:
> > > > > On Tue, 2007-07-03 at 05:29 +0100, Matthew Garrett wrote:
> > > > > > Suspend to RAM on a machine with / on a fuse filesystem turns out to be 
> > > > > > a screaming nightmare - either the suspend fails because syslog (for 
> > > > > > instance) can't be frozen, or the machine deadlocks for some other 
> > > > > > reason I haven't tracked down. We could "fix" fuse, or alternatively we 
> > > > > > could do what we do for suspend to RAM on other platforms (PPC and APM) 
> > > > > > and just not use the freezer.
> > > > > 
> > > > > The main reason for deadlocks is because we do a sys_sync() after the
> > > > > freeze, which we shouldn't do.
> > > > 
> > > > So why don't we remove the sys_sync() from freeze_processes() instead?
> > > 
> > > The patch follows (untested).
> > > 
> > > Greetings,
> > > Rafael
> > > 
> > > 
> > > ---
> > > From: Rafael J. Wysocki <rjw@sisk.pl>
> > > 
> > > We shouldn't sync filesystems from within the freezer, because it's not needed
> > > for suspend to RAM and leads to problems with FUSE.
> > 
> > This seems fishy. Swsusp needs enough clean memory to make enough
> > room for the image. If you sync before you freeze, the running tasks can
> > redirty memory.
> > What makes you sure that you don't die as shrink_all_memory() writes out
> > pages?
> 
> Shrink_all_memory should just free enough memory, what's the problem?
> Yes, we can have dirty memory, shrink_all_memory() can write that out
> just fine.

If there's any dirty memory to be written out through fuse, this will not
work as user space is already frozen. Now I am told that with fuse writes
are synchronous. Therefore I don't understand why having a call to sys_sync()
can make a difference. IMHO removing it to make fuse work covers over
a symptom but hides the bug.

	Regards
		Oliver

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Oliver Neukum]

Am Dienstag, 3. Juli 2007 schrieb Rafael J. Wysocki:
> > > The main reason for deadlocks is because we do a sys_sync() after the
> > > freeze, which we shouldn't do.
> > 
> > So why don't we remove the sys_sync() from freeze_processes() instead?
> 
> The patch follows (untested).

And a further question. The freezer is not atomic. What do you do
if a task not yet frozen calls sys_sync(), but fuse is already frozen?

	Regards
		Oliver

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Miklos Szeredi]

> And a further question. The freezer is not atomic. What do you do
> if a task not yet frozen calls sys_sync(), but fuse is already frozen?

What do you do if a task not yet frozen writes to a pipe, on the other
end of which is a task already frozen?

It doesn't matter.  The only thing that should matter during suspend
(not hibernate) is saving the state of devices to ram, and putting the
devices to sleep.

I'm not sure why this can't be made atomic, but assuming, that it
can't, fuse should still not need to be implicated.  If it is, that's
an indication about something wrong in the suspend procedure.

Miklos

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Oliver Neukum]

Am Dienstag, 3. Juli 2007 schrieb Miklos Szeredi:
> > And a further question. The freezer is not atomic. What do you do
> > if a task not yet frozen calls sys_sync(), but fuse is already frozen?
> 
> What do you do if a task not yet frozen writes to a pipe, on the other
> end of which is a task already frozen?

The same as you do with a pipe when the reader is not ready.

> It doesn't matter.  The only thing that should matter during suspend
> (not hibernate) is saving the state of devices to ram, and putting the
> devices to sleep.

Well, but you did remove sys_sync() from the freezer, which is
and must be called in the hibernate path.
 
> I'm not sure why this can't be made atomic, but assuming, that it
> can't, fuse should still not need to be implicated.  If it is, that's
> an indication about something wrong in the suspend procedure.

Nope, something's wrong in fuse. You must be able to deal with sync
until every task is frozen.

	Regards
		Oliver

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Miklos Szeredi]

> Am Dienstag, 3. Juli 2007 schrieb Miklos Szeredi:
> > > And a further question. The freezer is not atomic. What do you do
> > > if a task not yet frozen calls sys_sync(), but fuse is already frozen?
> > 
> > What do you do if a task not yet frozen writes to a pipe, on the other
> > end of which is a task already frozen?
> 
> The same as you do with a pipe when the reader is not ready.

Yes.  And if the fuse process is not ready, what will sys_sync() do?

(As a side note, sys_sync() actually does nothing in fuse, so these
examples are slightly silly, but the concept applies to any filesystem
operation).

> > It doesn't matter.  The only thing that should matter during suspend
> > (not hibernate) is saving the state of devices to ram, and putting the
> > devices to sleep.
> 
> Well, but you did remove sys_sync() from the freezer, which is
> and must be called in the hibernate path.

Since I don't know why sync() needs to be called from hibernate, I
can't argue about this.  But for _suspend_ it definitely should not be
needed.

> > I'm not sure why this can't be made atomic, but assuming, that it
> > can't, fuse should still not need to be implicated.  If it is, that's
> > an indication about something wrong in the suspend procedure.
> 
> Nope, something's wrong in fuse. You must be able to deal with sync
> until every task is frozen.

Why exactly?  That's like saying the pipe must be able to deal with
writes until every task is frozen.

As I've said, fuse is just special kind of IPC.  The task calling into
fuse can assume nothing about when it's request will be served.  Just
like a task writing into a pipe can assume nothing about when that
write will return.

Miklos

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Benjamin Herrenschmidt]

On Tue, 2007-07-03 at 21:32 +0200, Oliver Neukum wrote:
> > I'm not sure why this can't be made atomic, but assuming, that it
> > can't, fuse should still not need to be implicated.  If it is,
> that's
> > an indication about something wrong in the suspend procedure.
> 
> Nope, something's wrong in fuse. You must be able to deal with sync
> until every task is frozen. 

Pipe dream

Ben.

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Oliver Neukum]

Am Dienstag, 3. Juli 2007 schrieb Benjamin Herrenschmidt:
> On Tue, 2007-07-03 at 21:32 +0200, Oliver Neukum wrote:
> > > I'm not sure why this can't be made atomic, but assuming, that it
> > > can't, fuse should still not need to be implicated.  If it is,
> > that's
> > > an indication about something wrong in the suspend procedure.
> > 
> > Nope, something's wrong in fuse. You must be able to deal with sync
> > until every task is frozen. 
> 
> Pipe dream

Then tell me how you want to avoid that condition.

	Oliver

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Benjamin Herrenschmidt]

On Tue, 2007-07-03 at 23:48 +0200, Oliver Neukum wrote:
> Am Dienstag, 3. Juli 2007 schrieb Benjamin Herrenschmidt:
> > On Tue, 2007-07-03 at 21:32 +0200, Oliver Neukum wrote:
> > > > I'm not sure why this can't be made atomic, but assuming, that it
> > > > can't, fuse should still not need to be implicated.  If it is,
> > > that's
> > > > an indication about something wrong in the suspend procedure.
> > > 
> > > Nope, something's wrong in fuse. You must be able to deal with sync
> > > until every task is frozen. 
> > 
> > Pipe dream
> 
> Then tell me how you want to avoid that condition.

Don't freeze :-)

Ben.

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Oliver Neukum]

Am Dienstag, 3. Juli 2007 schrieb Benjamin Herrenschmidt:
> On Tue, 2007-07-03 at 23:48 +0200, Oliver Neukum wrote:
> > Am Dienstag, 3. Juli 2007 schrieb Benjamin Herrenschmidt:
> > > On Tue, 2007-07-03 at 21:32 +0200, Oliver Neukum wrote:
> > > > > I'm not sure why this can't be made atomic, but assuming, that it
> > > > > can't, fuse should still not need to be implicated.  If it is,
> > > > that's
> > > > > an indication about something wrong in the suspend procedure.
> > > > 
> > > > Nope, something's wrong in fuse. You must be able to deal with sync
> > > > until every task is frozen. 
> > > 
> > > Pipe dream
> > 
> > Then tell me how you want to avoid that condition.
> 
> Don't freeze :-)

Then you will have to deal with all syscalls unfrozen tasks can make.

	Regards
		Oliver

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Benjamin Herrenschmidt]

On Wed, 2007-07-04 at 00:04 +0200, Oliver Neukum wrote:
> Am Dienstag, 3. Juli 2007 schrieb Benjamin Herrenschmidt:
> > On Tue, 2007-07-03 at 23:48 +0200, Oliver Neukum wrote:
> > > Am Dienstag, 3. Juli 2007 schrieb Benjamin Herrenschmidt:
> > > > On Tue, 2007-07-03 at 21:32 +0200, Oliver Neukum wrote:
> > > > > > I'm not sure why this can't be made atomic, but assuming, that it
> > > > > > can't, fuse should still not need to be implicated.  If it is,
> > > > > that's
> > > > > > an indication about something wrong in the suspend procedure.
> > > > > 
> > > > > Nope, something's wrong in fuse. You must be able to deal with sync
> > > > > until every task is frozen. 
> > > > 
> > > > Pipe dream
> > > 
> > > Then tell me how you want to avoid that condition.
> > 
> > Don't freeze :-)
> 
> Then you will have to deal with all syscalls unfrozen tasks can make.

Yup, and the majority of them is totally harmless. Looks like people
around here have a problem with the idea of writing robust drivers ...

Ben.

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Oliver Neukum]

Am Mittwoch, 4. Juli 2007 schrieb Benjamin Herrenschmidt:
> > > > > > Nope, something's wrong in fuse. You must be able to deal with sync
> > > > > > until every task is frozen. 
> > > > > 
> > > > > Pipe dream
> > > > 
> > > > Then tell me how you want to avoid that condition.
> > > 
> > > Don't freeze :-)
> > 
> > Then you will have to deal with all syscalls unfrozen tasks can make.
> 
> Yup, and the majority of them is totally harmless. Looks like people
> around here have a problem with the idea of writing robust drivers ...

The majority is meaningless here. The subsystem works or works not.
Reliably failing to work is better than working most of the time.

What I am having a problem with is the rest of the system changing its
behavior and expecting the drivers to cope with that out of the blue.

	Oliver

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Pavel Machek]

On Tue 2007-07-03 21:32:20, Oliver Neukum wrote:
> Am Dienstag, 3. Juli 2007 schrieb Miklos Szeredi:
> > > And a further question. The freezer is not atomic. What do you do
> > > if a task not yet frozen calls sys_sync(), but fuse is already frozen?
> > 
> > What do you do if a task not yet frozen writes to a pipe, on the other
> > end of which is a task already frozen?

There's some difference between uninterruptible and interruptible
sleep I'd say.

> > It doesn't matter.  The only thing that should matter during suspend
> > (not hibernate) is saving the state of devices to ram, and putting the
> > devices to sleep.
> 
> Well, but you did remove sys_sync() from the freezer, which is
> and must be called in the hibernate path.

Not "must". In fact, hibernation should be safe without sys_sync(). It
is just user un-friendly.

									Pavel
-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Rafael J. Wysocki]

On Thursday, 5 July 2007 01:45, Pavel Machek wrote:
> On Tue 2007-07-03 21:32:20, Oliver Neukum wrote:
> > Am Dienstag, 3. Juli 2007 schrieb Miklos Szeredi:
> > > > And a further question. The freezer is not atomic. What do you do
> > > > if a task not yet frozen calls sys_sync(), but fuse is already frozen?
> > > 
> > > What do you do if a task not yet frozen writes to a pipe, on the other
> > > end of which is a task already frozen?
> 
> There's some difference between uninterruptible and interruptible
> sleep I'd say.
> 
> > > It doesn't matter.  The only thing that should matter during suspend
> > > (not hibernate) is saving the state of devices to ram, and putting the
> > > devices to sleep.
> > 
> > Well, but you did remove sys_sync() from the freezer, which is
> > and must be called in the hibernate path.
> 
> Not "must". In fact, hibernation should be safe without sys_sync(). It
> is just user un-friendly.

In fact, I'd like to remove the sys_sync() from the freezer entirely, because
it just doesn't belong in there.

The only advantege of having sys_sync() in freeze_processes() is that we
have a chance to write out everything when applications cannot produce more
data to write, but there are filesystems which don't do that anyway (eg. XFS),
so generally there's no reason to bother.

Greetings,
Rafael


-- 
"Premature optimization is the root of all evil." - Donald Knuth

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Nigel Cunningham]

[-- Attachment #1.1: Type: text/plain, Size: 2199 bytes --]

Hi.

On Thursday 05 July 2007 22:25:06 Rafael J. Wysocki wrote:
> On Thursday, 5 July 2007 01:45, Pavel Machek wrote:
> > On Tue 2007-07-03 21:32:20, Oliver Neukum wrote:
> > > Am Dienstag, 3. Juli 2007 schrieb Miklos Szeredi:
> > > > > And a further question. The freezer is not atomic. What do you do
> > > > > if a task not yet frozen calls sys_sync(), but fuse is already 
frozen?
> > > > 
> > > > What do you do if a task not yet frozen writes to a pipe, on the other
> > > > end of which is a task already frozen?
> > 
> > There's some difference between uninterruptible and interruptible
> > sleep I'd say.
> > 
> > > > It doesn't matter.  The only thing that should matter during suspend
> > > > (not hibernate) is saving the state of devices to ram, and putting the
> > > > devices to sleep.
> > > 
> > > Well, but you did remove sys_sync() from the freezer, which is
> > > and must be called in the hibernate path.
> > 
> > Not "must". In fact, hibernation should be safe without sys_sync(). It
> > is just user un-friendly.
> 
> In fact, I'd like to remove the sys_sync() from the freezer entirely, 
because
> it just doesn't belong in there.
> 
> The only advantege of having sys_sync() in freeze_processes() is that we
> have a chance to write out everything when applications cannot produce more
> data to write, but there are filesystems which don't do that anyway (eg. 
XFS),
> so generally there's no reason to bother.

Shouldn't XFS - and fuse - be considered to be broken? Sync should sync data 
and if XFS isn't doing that, it's wrong.

In the case of fuse, we should have a mechanism by which fuse processes can be 
made to sync if they do have any pending I/O, and by which they can be frozen 
later than other userspace processes.

I'd like to see the sync stay, because it improves reliability and data 
integrity in the fail-to-resume case. Calling scripts would probably invoke 
sync themselves if they don't already, but that's racy. As it is at the 
moment, we know userspace is stopped, so syncing isn't racy.

Regards,

Nigel
-- 
See http://www.tuxonice.net for Howtos, FAQs, mailing
lists, wiki and bugzilla info.

[-- Attachment #1.2: Type: application/pgp-signature, Size: 189 bytes --]

[-- Attachment #2: Type: text/plain, Size: 0 bytes --]

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Rafael J. Wysocki]

On Thursday, 5 July 2007 14:38, Nigel Cunningham wrote:
> Hi.
> 
> On Thursday 05 July 2007 22:25:06 Rafael J. Wysocki wrote:
> > On Thursday, 5 July 2007 01:45, Pavel Machek wrote:
> > > On Tue 2007-07-03 21:32:20, Oliver Neukum wrote:
> > > > Am Dienstag, 3. Juli 2007 schrieb Miklos Szeredi:
> > > > > > And a further question. The freezer is not atomic. What do you do
> > > > > > if a task not yet frozen calls sys_sync(), but fuse is already 
> frozen?
> > > > > 
> > > > > What do you do if a task not yet frozen writes to a pipe, on the other
> > > > > end of which is a task already frozen?
> > > 
> > > There's some difference between uninterruptible and interruptible
> > > sleep I'd say.
> > > 
> > > > > It doesn't matter.  The only thing that should matter during suspend
> > > > > (not hibernate) is saving the state of devices to ram, and putting the
> > > > > devices to sleep.
> > > > 
> > > > Well, but you did remove sys_sync() from the freezer, which is
> > > > and must be called in the hibernate path.
> > > 
> > > Not "must". In fact, hibernation should be safe without sys_sync(). It
> > > is just user un-friendly.
> > 
> > In fact, I'd like to remove the sys_sync() from the freezer entirely, 
> because
> > it just doesn't belong in there.
> > 
> > The only advantege of having sys_sync() in freeze_processes() is that we
> > have a chance to write out everything when applications cannot produce more
> > data to write, but there are filesystems which don't do that anyway (eg. 
> XFS),
> > so generally there's no reason to bother.
> 
> Shouldn't XFS - and fuse - be considered to be broken? Sync should sync data 
> and if XFS isn't doing that, it's wrong.
> 
> In the case of fuse, we should have a mechanism by which fuse processes can be 
> made to sync if they do have any pending I/O, and by which they can be frozen 
> later than other userspace processes.
> 
> I'd like to see the sync stay, because it improves reliability and data 
> integrity in the fail-to-resume case. Calling scripts would probably invoke 
> sync themselves if they don't already, but that's racy. As it is at the 
> moment, we know userspace is stopped, so syncing isn't racy.

I'd like to move the sync out of the freezer, but to call it from the
suspend/hibernation code, so that we do

sys_sync();
error = freeze_processes();

etc.

Greetings,
Rafael


-- 
"Premature optimization is the root of all evil." - Donald Knuth

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Nigel Cunningham]

[-- Attachment #1.1: Type: text/plain, Size: 3253 bytes --]

Hi.

On Thursday 05 July 2007 23:35:45 Rafael J. Wysocki wrote:
> On Thursday, 5 July 2007 14:38, Nigel Cunningham wrote:
> > On Thursday 05 July 2007 22:25:06 Rafael J. Wysocki wrote:
> > > On Thursday, 5 July 2007 01:45, Pavel Machek wrote:
> > > > On Tue 2007-07-03 21:32:20, Oliver Neukum wrote:
> > > > > Am Dienstag, 3. Juli 2007 schrieb Miklos Szeredi:
> > > > > > > And a further question. The freezer is not atomic. What do you 
do
> > > > > > > if a task not yet frozen calls sys_sync(), but fuse is already 
> > frozen?
> > > > > > 
> > > > > > What do you do if a task not yet frozen writes to a pipe, on the 
other
> > > > > > end of which is a task already frozen?
> > > > 
> > > > There's some difference between uninterruptible and interruptible
> > > > sleep I'd say.
> > > > 
> > > > > > It doesn't matter.  The only thing that should matter during 
suspend
> > > > > > (not hibernate) is saving the state of devices to ram, and putting 
the
> > > > > > devices to sleep.
> > > > > 
> > > > > Well, but you did remove sys_sync() from the freezer, which is
> > > > > and must be called in the hibernate path.
> > > > 
> > > > Not "must". In fact, hibernation should be safe without sys_sync(). It
> > > > is just user un-friendly.
> > > 
> > > In fact, I'd like to remove the sys_sync() from the freezer entirely, 
> > because
> > > it just doesn't belong in there.
> > > 
> > > The only advantege of having sys_sync() in freeze_processes() is that we
> > > have a chance to write out everything when applications cannot produce 
more
> > > data to write, but there are filesystems which don't do that anyway (eg. 
> > XFS),
> > > so generally there's no reason to bother.
> > 
> > Shouldn't XFS - and fuse - be considered to be broken? Sync should sync 
data 
> > and if XFS isn't doing that, it's wrong.
> > 
> > In the case of fuse, we should have a mechanism by which fuse processes 
can be 
> > made to sync if they do have any pending I/O, and by which they can be 
frozen 
> > later than other userspace processes.
> > 
> > I'd like to see the sync stay, because it improves reliability and data 
> > integrity in the fail-to-resume case. Calling scripts would probably 
invoke 
> > sync themselves if they don't already, but that's racy. As it is at the 
> > moment, we know userspace is stopped, so syncing isn't racy.
> 
> I'd like to move the sync out of the freezer, but to call it from the
> suspend/hibernation code, so that we do
> 
> sys_sync();
> error = freeze_processes();

Yeah, I understand that. The problem then is that you're racing against 
userspace. That's not usually a problem, but that doesn't mean it's never a 
problem. Try running the stress suite while testing hibernating and you'll 
see what I mean. If something is submitting lots of I/O when you try to 
suspend, your sync call will race against that process if it's not yet 
frozen, and its continued activity will make your sync pointless (there'll be 
more unsynced data when you sys_sync call finishes). Stopping userspace 
before syncing removes that race.

Regards,

Nigel
-- 
See http://www.tuxonice.net for Howtos, FAQs, mailing
lists, wiki and bugzilla info.

[-- Attachment #1.2: Type: application/pgp-signature, Size: 189 bytes --]

[-- Attachment #2: Type: text/plain, Size: 0 bytes --]

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Rafael J. Wysocki]

Hi,

On Thursday, 5 July 2007 15:36, Nigel Cunningham wrote:
> Hi.
> 
> On Thursday 05 July 2007 23:35:45 Rafael J. Wysocki wrote:
> > On Thursday, 5 July 2007 14:38, Nigel Cunningham wrote:
> > > On Thursday 05 July 2007 22:25:06 Rafael J. Wysocki wrote:
> > > > On Thursday, 5 July 2007 01:45, Pavel Machek wrote:
> > > > > On Tue 2007-07-03 21:32:20, Oliver Neukum wrote:
> > > > > > Am Dienstag, 3. Juli 2007 schrieb Miklos Szeredi:
> > > > > > > > And a further question. The freezer is not atomic. What do you 
> do
> > > > > > > > if a task not yet frozen calls sys_sync(), but fuse is already 
> > > frozen?
> > > > > > > 
> > > > > > > What do you do if a task not yet frozen writes to a pipe, on the 
> other
> > > > > > > end of which is a task already frozen?
> > > > > 
> > > > > There's some difference between uninterruptible and interruptible
> > > > > sleep I'd say.
> > > > > 
> > > > > > > It doesn't matter.  The only thing that should matter during 
> suspend
> > > > > > > (not hibernate) is saving the state of devices to ram, and putting 
> the
> > > > > > > devices to sleep.
> > > > > > 
> > > > > > Well, but you did remove sys_sync() from the freezer, which is
> > > > > > and must be called in the hibernate path.
> > > > > 
> > > > > Not "must". In fact, hibernation should be safe without sys_sync(). It
> > > > > is just user un-friendly.
> > > > 
> > > > In fact, I'd like to remove the sys_sync() from the freezer entirely, 
> > > because
> > > > it just doesn't belong in there.
> > > > 
> > > > The only advantege of having sys_sync() in freeze_processes() is that we
> > > > have a chance to write out everything when applications cannot produce 
> more
> > > > data to write, but there are filesystems which don't do that anyway (eg. 
> > > XFS),
> > > > so generally there's no reason to bother.
> > > 
> > > Shouldn't XFS - and fuse - be considered to be broken? Sync should sync 
> data 
> > > and if XFS isn't doing that, it's wrong.
> > > 
> > > In the case of fuse, we should have a mechanism by which fuse processes 
> can be 
> > > made to sync if they do have any pending I/O, and by which they can be 
> frozen 
> > > later than other userspace processes.
> > > 
> > > I'd like to see the sync stay, because it improves reliability and data 
> > > integrity in the fail-to-resume case. Calling scripts would probably 
> invoke 
> > > sync themselves if they don't already, but that's racy. As it is at the 
> > > moment, we know userspace is stopped, so syncing isn't racy.
> > 
> > I'd like to move the sync out of the freezer, but to call it from the
> > suspend/hibernation code, so that we do
> > 
> > sys_sync();
> > error = freeze_processes();
> 
> Yeah, I understand that. The problem then is that you're racing against 
> userspace. That's not usually a problem, but that doesn't mean it's never a 
> problem. Try running the stress suite while testing hibernating and you'll 
> see what I mean. If something is submitting lots of I/O when you try to 
> suspend, your sync call will race against that process if it's not yet 
> frozen, and its continued activity will make your sync pointless (there'll be 
> more unsynced data when you sys_sync call finishes). Stopping userspace 
> before syncing removes that race.

Yes, that will make the suspend/hibernation less reliable in case the resume
fails (some data, written after the sync, may be lost).  However, the sync done
from within the freezer doesn't guarantee that there are no data lost anyway,
so we don't lose much by not doing it.

Now, there's a question how much data may be lost, potentially, if we do the
sync before the freezer and I don't think that's a lot.

Greetings,
Rafael


-- 
"Premature optimization is the root of all evil." - Donald Knuth

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Nigel Cunningham]

[-- Attachment #1.1: Type: text/plain, Size: 4382 bytes --]

Good morning!

On Thursday 05 July 2007 23:59:57 Rafael J. Wysocki wrote:
> On Thursday, 5 July 2007 15:36, Nigel Cunningham wrote:
> > On Thursday 05 July 2007 23:35:45 Rafael J. Wysocki wrote:
> > > On Thursday, 5 July 2007 14:38, Nigel Cunningham wrote:
> > > > On Thursday 05 July 2007 22:25:06 Rafael J. Wysocki wrote:
> > > > > On Thursday, 5 July 2007 01:45, Pavel Machek wrote:
> > > > > > On Tue 2007-07-03 21:32:20, Oliver Neukum wrote:
> > > > > > > Am Dienstag, 3. Juli 2007 schrieb Miklos Szeredi:
> > > > > > > > > And a further question. The freezer is not atomic. What do 
you 
> > do
> > > > > > > > > if a task not yet frozen calls sys_sync(), but fuse is 
already 
> > > > frozen?
> > > > > > > > 
> > > > > > > > What do you do if a task not yet frozen writes to a pipe, on 
the 
> > other
> > > > > > > > end of which is a task already frozen?
> > > > > > 
> > > > > > There's some difference between uninterruptible and interruptible
> > > > > > sleep I'd say.
> > > > > > 
> > > > > > > > It doesn't matter.  The only thing that should matter during 
> > suspend
> > > > > > > > (not hibernate) is saving the state of devices to ram, and 
putting 
> > the
> > > > > > > > devices to sleep.
> > > > > > > 
> > > > > > > Well, but you did remove sys_sync() from the freezer, which is
> > > > > > > and must be called in the hibernate path.
> > > > > > 
> > > > > > Not "must". In fact, hibernation should be safe without 
sys_sync(). It
> > > > > > is just user un-friendly.
> > > > > 
> > > > > In fact, I'd like to remove the sys_sync() from the freezer 
entirely, 
> > > > because
> > > > > it just doesn't belong in there.
> > > > > 
> > > > > The only advantege of having sys_sync() in freeze_processes() is 
that we
> > > > > have a chance to write out everything when applications cannot 
produce 
> > more
> > > > > data to write, but there are filesystems which don't do that anyway 
(eg. 
> > > > XFS),
> > > > > so generally there's no reason to bother.
> > > > 
> > > > Shouldn't XFS - and fuse - be considered to be broken? Sync should 
sync 
> > data 
> > > > and if XFS isn't doing that, it's wrong.
> > > > 
> > > > In the case of fuse, we should have a mechanism by which fuse 
processes 
> > can be 
> > > > made to sync if they do have any pending I/O, and by which they can be 
> > frozen 
> > > > later than other userspace processes.
> > > > 
> > > > I'd like to see the sync stay, because it improves reliability and 
data 
> > > > integrity in the fail-to-resume case. Calling scripts would probably 
> > invoke 
> > > > sync themselves if they don't already, but that's racy. As it is at 
the 
> > > > moment, we know userspace is stopped, so syncing isn't racy.
> > > 
> > > I'd like to move the sync out of the freezer, but to call it from the
> > > suspend/hibernation code, so that we do
> > > 
> > > sys_sync();
> > > error = freeze_processes();
> > 
> > Yeah, I understand that. The problem then is that you're racing against 
> > userspace. That's not usually a problem, but that doesn't mean it's never 
a 
> > problem. Try running the stress suite while testing hibernating and you'll 
> > see what I mean. If something is submitting lots of I/O when you try to 
> > suspend, your sync call will race against that process if it's not yet 
> > frozen, and its continued activity will make your sync pointless (there'll 
be 
> > more unsynced data when you sys_sync call finishes). Stopping userspace 
> > before syncing removes that race.
> 
> Yes, that will make the suspend/hibernation less reliable in case the resume
> fails (some data, written after the sync, may be lost).  However, the sync 
done
> from within the freezer doesn't guarantee that there are no data lost 
anyway,
> so we don't lose much by not doing it.
> 
> Now, there's a question how much data may be lost, potentially, if we do the
> sync before the freezer and I don't think that's a lot.

You're missing the point. I'm arguing that a sync from within the freezer 
should guarantee that there is no data loss. As I said about, XFS should be 
fixed to properly sync its data, and something should be done about fuse 
filesystems too.

Regards,

Nigel
-- 
See http://www.tuxonice.net for Howtos, FAQs, mailing
lists, wiki and bugzilla info.

[-- Attachment #1.2: Type: application/pgp-signature, Size: 189 bytes --]

[-- Attachment #2: Type: text/plain, Size: 0 bytes --]

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Miklos Szeredi]

> > You're missing the point. I'm arguing that a sync from within the freezer 
> > should guarantee that there is no data loss.
> 
> Well, it should, but it doesn't ...
> 
> Moreover, if FUSE implements syncing, then the sync from within the freezer
> will almost certainly deadlock.

Rafael, think positively: by the time fuse implements sync(), the
freezer will be long gone ;)

Miklos

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Oliver Neukum]

Am Freitag, 6. Juli 2007 schrieb Miklos Szeredi:
> > > You're missing the point. I'm arguing that a sync from within the freezer 
> > > should guarantee that there is no data loss.
> > 
> > Well, it should, but it doesn't ...
> > 
> > Moreover, if FUSE implements syncing, then the sync from within the freezer
> > will almost certainly deadlock.
> 
> Rafael, think positively: by the time fuse implements sync(), the
> freezer will be long gone ;)

Now you are entering really dangerous territory.
If you can implement a meaningfull sync method, you must have dirty
pages in the page cache. That means you are in the page freeing path
of the vm. Then we are in real trouble. Don't even think about it.

As far as suspend/hibernate is concerned, get yourself on the new
notifying chain and revert to synchronous operations when notified.

	Regards
		Oliver

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Miklos Szeredi]

> > > Moreover, if FUSE implements syncing, then the sync from within the freezer
> > > will almost certainly deadlock.
> > 
> > Rafael, think positively: by the time fuse implements sync(), the
> > freezer will be long gone ;)
> 
> Now you are entering really dangerous territory.
> If you can implement a meaningfull sync method, you must have dirty
> pages in the page cache. That means you are in the page freeing path
> of the vm. Then we are in real trouble. Don't even think about it.

VM induced deadlocks are real nasty.  I have thought about them a lot
already.  Suspend shouldn't introduce any big surprises.

Miklos

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Nigel Cunningham]

[-- Attachment #1.1: Type: text/plain, Size: 885 bytes --]

Hi.

On Friday 06 July 2007 19:09:43 Miklos Szeredi wrote:
> > > > Moreover, if FUSE implements syncing, then the sync from within the 
freezer
> > > > will almost certainly deadlock.
> > > 
> > > Rafael, think positively: by the time fuse implements sync(), the
> > > freezer will be long gone ;)
> > 
> > Now you are entering really dangerous territory.
> > If you can implement a meaningfull sync method, you must have dirty
> > pages in the page cache. That means you are in the page freeing path
> > of the vm. Then we are in real trouble. Don't even think about it.
> 
> VM induced deadlocks are real nasty.  I have thought about them a lot
> already.  Suspend shouldn't introduce any big surprises.

Suspend isn't introducing the surprises. Fuse is. It creates the potential 
deadlocks simply by existing (this isn't suspend or hibernate specific).

Nigel

[-- Attachment #1.2: Type: application/pgp-signature, Size: 189 bytes --]

[-- Attachment #2: Type: text/plain, Size: 0 bytes --]

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Miklos Szeredi]

> > VM induced deadlocks are real nasty.  I have thought about them a lot
> > already.  Suspend shouldn't introduce any big surprises.
> 
> Suspend isn't introducing the surprises. Fuse is. It creates the potential 
> deadlocks simply by existing (this isn't suspend or hibernate specific).

Yes.  And fuse is not going away any time soon.  Get over it ;)

Miklos

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Rafael J. Wysocki]

Hi,

On Thursday, 5 July 2007 23:49, Nigel Cunningham wrote:
> Good morning!
> 
> On Thursday 05 July 2007 23:59:57 Rafael J. Wysocki wrote:
> > On Thursday, 5 July 2007 15:36, Nigel Cunningham wrote:
> > > On Thursday 05 July 2007 23:35:45 Rafael J. Wysocki wrote:
> > > > On Thursday, 5 July 2007 14:38, Nigel Cunningham wrote:
> > > > > On Thursday 05 July 2007 22:25:06 Rafael J. Wysocki wrote:
> > > > > > On Thursday, 5 July 2007 01:45, Pavel Machek wrote:
> > > > > > > On Tue 2007-07-03 21:32:20, Oliver Neukum wrote:
> > > > > > > > Am Dienstag, 3. Juli 2007 schrieb Miklos Szeredi:
> > > > > > > > > > And a further question. The freezer is not atomic. What do 
> you 
> > > do
> > > > > > > > > > if a task not yet frozen calls sys_sync(), but fuse is 
> already 
> > > > > frozen?
> > > > > > > > > 
> > > > > > > > > What do you do if a task not yet frozen writes to a pipe, on 
> the 
> > > other
> > > > > > > > > end of which is a task already frozen?
> > > > > > > 
> > > > > > > There's some difference between uninterruptible and interruptible
> > > > > > > sleep I'd say.
> > > > > > > 
> > > > > > > > > It doesn't matter.  The only thing that should matter during 
> > > suspend
> > > > > > > > > (not hibernate) is saving the state of devices to ram, and 
> putting 
> > > the
> > > > > > > > > devices to sleep.
> > > > > > > > 
> > > > > > > > Well, but you did remove sys_sync() from the freezer, which is
> > > > > > > > and must be called in the hibernate path.
> > > > > > > 
> > > > > > > Not "must". In fact, hibernation should be safe without 
> sys_sync(). It
> > > > > > > is just user un-friendly.
> > > > > > 
> > > > > > In fact, I'd like to remove the sys_sync() from the freezer 
> entirely, 
> > > > > because
> > > > > > it just doesn't belong in there.
> > > > > > 
> > > > > > The only advantege of having sys_sync() in freeze_processes() is 
> that we
> > > > > > have a chance to write out everything when applications cannot 
> produce 
> > > more
> > > > > > data to write, but there are filesystems which don't do that anyway 
> (eg. 
> > > > > XFS),
> > > > > > so generally there's no reason to bother.
> > > > > 
> > > > > Shouldn't XFS - and fuse - be considered to be broken? Sync should 
> sync 
> > > data 
> > > > > and if XFS isn't doing that, it's wrong.
> > > > > 
> > > > > In the case of fuse, we should have a mechanism by which fuse 
> processes 
> > > can be 
> > > > > made to sync if they do have any pending I/O, and by which they can be 
> > > frozen 
> > > > > later than other userspace processes.
> > > > > 
> > > > > I'd like to see the sync stay, because it improves reliability and 
> data 
> > > > > integrity in the fail-to-resume case. Calling scripts would probably 
> > > invoke 
> > > > > sync themselves if they don't already, but that's racy. As it is at 
> the 
> > > > > moment, we know userspace is stopped, so syncing isn't racy.
> > > > 
> > > > I'd like to move the sync out of the freezer, but to call it from the
> > > > suspend/hibernation code, so that we do
> > > > 
> > > > sys_sync();
> > > > error = freeze_processes();
> > > 
> > > Yeah, I understand that. The problem then is that you're racing against 
> > > userspace. That's not usually a problem, but that doesn't mean it's never 
> a 
> > > problem. Try running the stress suite while testing hibernating and you'll 
> > > see what I mean. If something is submitting lots of I/O when you try to 
> > > suspend, your sync call will race against that process if it's not yet 
> > > frozen, and its continued activity will make your sync pointless (there'll 
> be 
> > > more unsynced data when you sys_sync call finishes). Stopping userspace 
> > > before syncing removes that race.
> > 
> > Yes, that will make the suspend/hibernation less reliable in case the resume
> > fails (some data, written after the sync, may be lost).  However, the sync 
> done
> > from within the freezer doesn't guarantee that there are no data lost 
> anyway,
> > so we don't lose much by not doing it.
> > 
> > Now, there's a question how much data may be lost, potentially, if we do the
> > sync before the freezer and I don't think that's a lot.
> 
> You're missing the point. I'm arguing that a sync from within the freezer 
> should guarantee that there is no data loss.

Well, it should, but it doesn't ...

Moreover, if FUSE implements syncing, then the sync from within the freezer
will almost certainly deadlock.

> As I said about, XFS should be fixed to properly sync its data, and
> something should be done about fuse filesystems too.

I think that we can't do anything about it, so we should just live with it.

Greetings,
Rafael


-- 
"Premature optimization is the root of all evil." - Donald Knuth

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Rafael J. Wysocki]

On Tuesday, 3 July 2007 20:26, Oliver Neukum wrote:
> Am Dienstag, 3. Juli 2007 schrieb Rafael J. Wysocki:
> > > > The main reason for deadlocks is because we do a sys_sync() after the
> > > > freeze, which we shouldn't do.
> > > 
> > > So why don't we remove the sys_sync() from freeze_processes() instead?
> > 
> > The patch follows (untested).
> 
> And a further question. The freezer is not atomic. What do you do
> if a task not yet frozen calls sys_sync(), but fuse is already frozen?

Hmm, if the sync is interruptible (I'm not sure), the task should be frozen while
waiting for it to complete.

Otherwise, the freezing of tasks will fail (no deadlock).

Greetings,
Rafael


-- 
"Premature optimization is the root of all evil." - Donald Knuth

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Pavel Machek]

Hi!

> > > The main reason for deadlocks is because we do a sys_sync() after the
> > > freeze, which we shouldn't do.
> > 
> > So why don't we remove the sys_sync() from freeze_processes() instead?
> 
> The patch follows (untested).
> 
> Greetings,
> Rafael
> 
> 
> ---
> From: Rafael J. Wysocki <rjw@sisk.pl>
> 
> We shouldn't sync filesystems from within the freezer, because it's not needed
> for suspend to RAM and leads to problems with FUSE.

Actually... It is not _needed_ for suspend to disk, either. Snapshot is
atomic, so it should be okay to suspend with filesystems dirty.

_But_, if anything goes wrong, we'd prefer to have at least
filesystems synced. Battery running out during s2ram is not quite
uncommon, so we perhaps should do sync somewhere there. (But we can do
it before freezer just fine).
								Pavel

-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Rafael J. Wysocki]

On Tuesday, 3 July 2007 21:27, Pavel Machek wrote:
> Hi!
> 
> > > > The main reason for deadlocks is because we do a sys_sync() after the
> > > > freeze, which we shouldn't do.
> > > 
> > > So why don't we remove the sys_sync() from freeze_processes() instead?
> > 
> > The patch follows (untested).
> > 
> > Greetings,
> > Rafael
> > 
> > 
> > ---
> > From: Rafael J. Wysocki <rjw@sisk.pl>
> > 
> > We shouldn't sync filesystems from within the freezer, because it's not needed
> > for suspend to RAM and leads to problems with FUSE.
> 
> Actually... It is not _needed_ for suspend to disk, either. Snapshot is
> atomic, so it should be okay to suspend with filesystems dirty.
> 
> _But_, if anything goes wrong, we'd prefer to have at least
> filesystems synced. Battery running out during s2ram is not quite
> uncommon, so we perhaps should do sync somewhere there. (But we can do
> it before freezer just fine).

OK

So, should I add the sync() to suspend_prepare(), before freeze_processes()
(in analogy with hibernate())?

Greetings
Rafael


-- 
"Premature optimization is the root of all evil." - Donald Knuth

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Benjamin Herrenschmidt]

On Tue, 2007-07-03 at 15:08 +0200, Rafael J. Wysocki wrote:
> On Tuesday, 3 July 2007 07:51, Benjamin Herrenschmidt wrote:
> > On Tue, 2007-07-03 at 05:29 +0100, Matthew Garrett wrote:
> > > Suspend to RAM on a machine with / on a fuse filesystem turns out to be 
> > > a screaming nightmare - either the suspend fails because syslog (for 
> > > instance) can't be frozen, or the machine deadlocks for some other 
> > > reason I haven't tracked down. We could "fix" fuse, or alternatively we 
> > > could do what we do for suspend to RAM on other platforms (PPC and APM) 
> > > and just not use the freezer.
> > 
> > The main reason for deadlocks is because we do a sys_sync() after the
> > freeze, which we shouldn't do.
> 
> So why don't we remove the sys_sync() from freeze_processes() instead?

Or rather, move it to before freeze_processes() so that you get at least
some level of sync'ing just in case you never wake up. Not perfect
(things may still get dirty) but it will help the immediate issue.

Ben.

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Alan Stern]

On Tue, 3 Jul 2007, Matthew Garrett wrote:

> Suspend to RAM on a machine with / on a fuse filesystem turns out to be 
> a screaming nightmare - either the suspend fails because syslog (for 
> instance) can't be frozen, or the machine deadlocks for some other 
> reason I haven't tracked down. We could "fix" fuse, or alternatively we 
> could do what we do for suspend to RAM on other platforms (PPC and APM) 
> and just not use the freezer.

Quite apart from the sync() matter, _any_ synchronous call to a FUSE 
filesystem during STR will cause trouble.  Even if the user task 
implementing the filesystem isn't frozen, when it tries to carry out 
some I/O to a suspended device it will either:

	block until the system wakes up, or

	cause the suspend to abort.

Neither outcome is desirable.

Alan Stern

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Matthew Garrett]

On Tue, Jul 03, 2007 at 12:03:33PM -0400, Alan Stern wrote:
> On Tue, 3 Jul 2007, Matthew Garrett wrote:
> 
> > Suspend to RAM on a machine with / on a fuse filesystem turns out to be 
> > a screaming nightmare - either the suspend fails because syslog (for 
> > instance) can't be frozen, or the machine deadlocks for some other 
> > reason I haven't tracked down. We could "fix" fuse, or alternatively we 
> > could do what we do for suspend to RAM on other platforms (PPC and APM) 
> > and just not use the freezer.
> 
> Quite apart from the sync() matter, _any_ synchronous call to a FUSE 
> filesystem during STR will cause trouble.  Even if the user task 
> implementing the filesystem isn't frozen, when it tries to carry out 
> some I/O to a suspended device it will either:
> 
> 	block until the system wakes up, or

For the suspend to RAM case, that sounds absolutely fine.

-- 
Matthew Garrett | mjg59@srcf.ucam.org

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Alan Stern]

On Tue, 3 Jul 2007, Matthew Garrett wrote:

> On Tue, Jul 03, 2007 at 12:03:33PM -0400, Alan Stern wrote:
> > On Tue, 3 Jul 2007, Matthew Garrett wrote:
> > 
> > > Suspend to RAM on a machine with / on a fuse filesystem turns out to be 
> > > a screaming nightmare - either the suspend fails because syslog (for 
> > > instance) can't be frozen, or the machine deadlocks for some other 
> > > reason I haven't tracked down. We could "fix" fuse, or alternatively we 
> > > could do what we do for suspend to RAM on other platforms (PPC and APM) 
> > > and just not use the freezer.
> > 
> > Quite apart from the sync() matter, _any_ synchronous call to a FUSE 
> > filesystem during STR will cause trouble.  Even if the user task 
> > implementing the filesystem isn't frozen, when it tries to carry out 
> > some I/O to a suspended device it will either:
> > 
> > 	block until the system wakes up, or
> 
> For the suspend to RAM case, that sounds absolutely fine.

It's not so good when your suspend process has to wait for the call to 
complete!

Alan Stern

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Matthew Garrett]

On Tue, Jul 03, 2007 at 12:57:17PM -0400, Alan Stern wrote:
> On Tue, 3 Jul 2007, Matthew Garrett wrote:
> > On Tue, Jul 03, 2007 at 12:03:33PM -0400, Alan Stern wrote:
> > > Quite apart from the sync() matter, _any_ synchronous call to a FUSE 
> > > filesystem during STR will cause trouble.  Even if the user task 
> > > implementing the filesystem isn't frozen, when it tries to carry out 
> > > some I/O to a suspended device it will either:
> > > 
> > > 	block until the system wakes up, or
> > 
> > For the suspend to RAM case, that sounds absolutely fine.
> 
> It's not so good when your suspend process has to wait for the call to 
> complete!

Why would it have to? Sorry, I suspect I'm missing something obvious 
here.

-- 
Matthew Garrett | mjg59@srcf.ucam.org

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Alan Stern]

On Tue, 3 Jul 2007, Matthew Garrett wrote:

> On Tue, Jul 03, 2007 at 12:57:17PM -0400, Alan Stern wrote:
> > On Tue, 3 Jul 2007, Matthew Garrett wrote:
> > > On Tue, Jul 03, 2007 at 12:03:33PM -0400, Alan Stern wrote:
> > > > Quite apart from the sync() matter, _any_ synchronous call to a FUSE 
> > > > filesystem during STR will cause trouble.  Even if the user task 
> > > > implementing the filesystem isn't frozen, when it tries to carry out 
> > > > some I/O to a suspended device it will either:
> > > > 
> > > > 	block until the system wakes up, or
> > > 
> > > For the suspend to RAM case, that sounds absolutely fine.
> > 
> > It's not so good when your suspend process has to wait for the call to 
> > complete!
> 
> Why would it have to? Sorry, I suspect I'm missing something obvious 
> here.

Well, the sys_sync() that caused your original problem did exactly 
that.  It's the reason you get deadlocks, right?

I agree that in general the suspend process should not have to wait for 
a userspace callback to complete.  Indeed, there's no particular 
reason that anything running during STR should have to wait for 
something in userspace to complete.  Given that fact, I don't see 
anything wrong with freezing userspace when doing STR.

Alan Stern

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Matthew Garrett]

On Tue, Jul 03, 2007 at 03:33:40PM -0400, Alan Stern wrote:
> On Tue, 3 Jul 2007, Matthew Garrett wrote:
> 
> > On Tue, Jul 03, 2007 at 12:57:17PM -0400, Alan Stern wrote:
> > > On Tue, 3 Jul 2007, Matthew Garrett wrote:
> > > > For the suspend to RAM case, that sounds absolutely fine.
> > > 
> > > It's not so good when your suspend process has to wait for the call to 
> > > complete!
> > 
> > Why would it have to? Sorry, I suspect I'm missing something obvious 
> > here.
> 
> Well, the sys_sync() that caused your original problem did exactly 
> that.  It's the reason you get deadlocks, right?

The sys_sync is unnecessary in the first case. There shouldn't be 
anything in the suspend path that's going to require userspace access to 
a device after that device has been suspended.

> I agree that in general the suspend process should not have to wait for 
> a userspace callback to complete.  Indeed, there's no particular 
> reason that anything running during STR should have to wait for 
> something in userspace to complete.  Given that fact, I don't see 
> anything wrong with freezing userspace when doing STR.

There's nothing wrong with it as such, it's just that our implementation 
appears to suck in a myriad of small ways that keep cropping up and 
biting people. Even without the sys_sync(), freezing processes results 
in the suspend failing because syslog is stuck in D state and won't go 
into the refrigerator.

-- 
Matthew Garrett | mjg59@srcf.ucam.org

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Alan Stern]

On Tue, 3 Jul 2007, Matthew Garrett wrote:

> > I agree that in general the suspend process should not have to wait for 
> > a userspace callback to complete.  Indeed, there's no particular 
> > reason that anything running during STR should have to wait for 
> > something in userspace to complete.  Given that fact, I don't see 
> > anything wrong with freezing userspace when doing STR.
> 
> There's nothing wrong with it as such, it's just that our implementation 
> appears to suck in a myriad of small ways that keep cropping up and 
> biting people. Even without the sys_sync(), freezing processes results 
> in the suspend failing because syslog is stuck in D state and won't go 
> into the refrigerator.

Okay, I can believe that.  The proper response then is to fix the 
freezer, not eliminate it.  Has the syslog problem been reported on 
linux-pm?  I don't recall hearing of it before.

Alan Stern

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Matthew Garrett]

On Tue, Jul 03, 2007 at 03:54:55PM -0400, Alan Stern wrote:
> On Tue, 3 Jul 2007, Matthew Garrett wrote:
> > There's nothing wrong with it as such, it's just that our implementation 
> > appears to suck in a myriad of small ways that keep cropping up and 
> > biting people. Even without the sys_sync(), freezing processes results 
> > in the suspend failing because syslog is stuck in D state and won't go 
> > into the refrigerator.
> 
> Okay, I can believe that.  The proper response then is to fix the 
> freezer, not eliminate it.  Has the syslog problem been reported on 
> linux-pm?  I don't recall hearing of it before.

See the start of this thread. It's just not clear what the freezer buys 
us - removing it gets rid of a load of subtle issues and complexity, and 
turns system suspend into something that looks more like runtime 
suspend (which might then encourage people to get runtime suspend 
right...)

-- 
Matthew Garrett | mjg59@srcf.ucam.org

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Pavel Machek]

Hi!

> Suspend to RAM on a machine with / on a fuse filesystem turns out to be 
> a screaming nightmare - either the suspend fails because syslog (for 
> instance) can't be frozen, or the machine deadlocks for some other 
> reason I haven't tracked down. We could "fix" fuse, or alternatively we 
> could do what we do for suspend to RAM on other platforms (PPC and APM) 
> and just not use the freezer.
> 
> Signed-off-by: Matthew Garrett <mjg59@srcf.ucam.org>

Sorry, no.

* this needs audit of all drivers. Or we can just merge it and then
fix all the problems it causes. If you are willing to become suspend
maintainer and handle all that mess, perhaps we can do this.

* it does not solve FUSE vs. hibernation

* it does not solve FUSE vs. suspend-to-both

* userspace will now see CPUs going up and down at minimum

Now, we want to do something like this long-term, but I do not think
we can just remove the freezer like this.


								Pavel
-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

[PATCH] Remove process freezer from suspend to RAM pathway

[Matthew Garrett]

Suspend to RAM on a machine with / on a fuse filesystem turns out to be 
a screaming nightmare - either the suspend fails because syslog (for 
instance) can't be frozen, or the machine deadlocks for some other 
reason I haven't tracked down. We could "fix" fuse, or alternatively we 
could do what we do for suspend to RAM on other platforms (PPC and APM) 
and just not use the freezer.

Signed-off-by: Matthew Garrett <mjg59@srcf.ucam.org>

diff --git a/kernel/power/main.c b/kernel/power/main.c
index 8812985..5f109d5 100644
--- a/kernel/power/main.c
+++ b/kernel/power/main.c
@@ -19,7 +19,6 @@
 #include <linux/console.h>
 #include <linux/cpu.h>
 #include <linux/resume-trace.h>
-#include <linux/freezer.h>
 #include <linux/vmstat.h>
 
 #include "power.h"
@@ -81,11 +80,6 @@ static int suspend_prepare(suspend_state_t state)
 
 	pm_prepare_console();
 
-	if (freeze_processes()) {
-		error = -EAGAIN;
-		goto Thaw;
-	}
-
 	if ((free_pages = global_page_state(NR_FREE_PAGES))
 			< FREE_PAGE_NUMBER) {
 		pr_debug("PM: free some memory\n");
@@ -93,7 +87,7 @@ static int suspend_prepare(suspend_state_t state)
 		if (nr_free_pages() < FREE_PAGE_NUMBER) {
 			error = -ENOMEM;
 			printk(KERN_ERR "PM: No enough memory\n");
-			goto Thaw;
+			goto Restore_console;
 		}
 	}
 
@@ -118,8 +112,7 @@ static int suspend_prepare(suspend_state_t state)
 	device_resume();
  Resume_console:
 	resume_console();
- Thaw:
-	thaw_processes();
+ Restore_console:
 	pm_restore_console();
 	return error;
 }
@@ -170,7 +163,6 @@ static void suspend_finish(suspend_state_t state)
 	pm_finish(state);
 	device_resume();
 	resume_console();
-	thaw_processes();
 	pm_restore_console();
 }

-- 
Matthew Garrett | mjg59@srcf.ucam.org