Re: [PATCH] Remove process freezer from suspend to RAM pathway

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Nigel Cunningham]

[-- Attachment #1.1: Type: text/plain, Size: 840 bytes --]

On Tuesday 03 July 2007 14:29:18 Matthew Garrett wrote:
> Suspend to RAM on a machine with / on a fuse filesystem turns out to be 
> a screaming nightmare - either the suspend fails because syslog (for 
> instance) can't be frozen, or the machine deadlocks for some other 
> reason I haven't tracked down. We could "fix" fuse, or alternatively we 
> could do what we do for suspend to RAM on other platforms (PPC and APM) 
> and just not use the freezer.
> 
> Signed-off-by: Matthew Garrett <mjg59@srcf.ucam.org>

Note, though, that this won't help at all when people use the "suspend-to-ram 
instead of powering down after writing a hibernation image" feature in 
(uswsusp | tuxonice). Fuse is just a broken idea in the first place, but 
given that it exists, we still need to find the underlying cause.

Regards,

Nigel

[-- Attachment #1.2: Type: application/pgp-signature, Size: 189 bytes --]

[-- Attachment #2: Type: text/plain, Size: 0 bytes --]

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Matthew Garrett]

On Tue, Jul 03, 2007 at 02:54:41PM +1000, Nigel Cunningham wrote:

> Note, though, that this won't help at all when people use the "suspend-to-ram 
> instead of powering down after writing a hibernation image" feature in 
> (uswsusp | tuxonice). Fuse is just a broken idea in the first place, but 
> given that it exists, we still need to find the underlying cause.

If / is on fuse it's unlikely that hibernation is high on your list of 
priorities right now.

-- 
Matthew Garrett | mjg59@srcf.ucam.org

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Nigel Cunningham]

[-- Attachment #1.1: Type: text/plain, Size: 701 bytes --]

Hi.

On Tuesday 03 July 2007 15:21:30 Matthew Garrett wrote:
> On Tue, Jul 03, 2007 at 02:54:41PM +1000, Nigel Cunningham wrote:
> 
> > Note, though, that this won't help at all when people use 
the "suspend-to-ram 
> > instead of powering down after writing a hibernation image" feature in 
> > (uswsusp | tuxonice). Fuse is just a broken idea in the first place, but 
> > given that it exists, we still need to find the underlying cause.
> 
> If / is on fuse it's unlikely that hibernation is high on your list of 
> priorities right now.

Yeah, well... what can you say to that? :)

Nigel
-- 
See http://www.tuxonice.net for Howtos, FAQs, mailing
lists, wiki and bugzilla info.

[-- Attachment #1.2: Type: application/pgp-signature, Size: 189 bytes --]

[-- Attachment #2: Type: text/plain, Size: 0 bytes --]

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Benjamin Herrenschmidt]

> Note, though, that this won't help at all when people use the "suspend-to-ram 
> instead of powering down after writing a hibernation image" feature in 
> (uswsusp | tuxonice). Fuse is just a broken idea in the first place, but 
> given that it exists, we still need to find the underlying cause.

No, Fuse is not a broken idea in the first place. It's the freezer that
is a totally broken idea. It has proven many times to be racy by design
and cannot be made right. Ther usermode helper mess is just part of
that, fuse is another example, etc etc ...

So I think Matthew is totally right. In fact, the presence of the
freezer is the main reason why Paulus so far NACKed Johannes attempts at
merging the PPC PM code with the generic code in kernel/power.c

We've been doing fine without it so far and intend to continue to do so.

As for suspend-to-disk, I refer you to the discussions we had in the
past with Linus, where he explains I think quite clearly how wrong the
current implementation of STR is :-)

Thing is, if you're going to do snapshots, you should probably not sync
after you have "frozen" anyway.

Cheers,
Ben.

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Nigel Cunningham]

[-- Attachment #1.1: Type: text/plain, Size: 1958 bytes --]

Hi.

On Tuesday 03 July 2007 15:48:26 Benjamin Herrenschmidt wrote:
> 
> > Note, though, that this won't help at all when people use 
the "suspend-to-ram 
> > instead of powering down after writing a hibernation image" feature in 
> > (uswsusp | tuxonice). Fuse is just a broken idea in the first place, but 
> > given that it exists, we still need to find the underlying cause.
> 
> No, Fuse is not a broken idea in the first place. It's the freezer that
> is a totally broken idea. It has proven many times to be racy by design
> and cannot be made right. Ther usermode helper mess is just part of
> that, fuse is another example, etc etc ...

To some extent, I agree. I think the ideal solution would be to simply not 
schedule processes that are supposed to be frozen. But who wants to play with 
scheduler code? Not me!

> So I think Matthew is totally right. In fact, the presence of the
> freezer is the main reason why Paulus so far NACKed Johannes attempts at
> merging the PPC PM code with the generic code in kernel/power.c
> 
> We've been doing fine without it so far and intend to continue to do so.

Fuse depends on !PPC?
 
> As for suspend-to-disk, I refer you to the discussions we had in the
> past with Linus, where he explains I think quite clearly how wrong the
> current implementation of STR is :-)

I assume you mean STD. The problem there is that Linus doesn't care about STD. 
If he did, I dare say he'd think through the issues more thoroughly than he 
apparently has.
 
> Thing is, if you're going to do snapshots, you should probably not sync
> after you have "frozen" anyway.

Fully agree. But how do you stop things syncing while you're writing the image 
if you don't have a freezer or equivalent? (scheduler based, kexec.. they're 
all workarounds for this issue).

Regards,

Nigel
-- 
Nigel, Michelle and Alisdair Cunningham
5 Mitchell Street
Cobden 3266
Victoria, Australia

[-- Attachment #1.2: Type: application/pgp-signature, Size: 189 bytes --]

[-- Attachment #2: Type: text/plain, Size: 0 bytes --]

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Benjamin Herrenschmidt]

On Tue, 2007-07-03 at 16:08 +1000, Nigel Cunningham wrote:
> 
> > So I think Matthew is totally right. In fact, the presence of the
> > freezer is the main reason why Paulus so far NACKed Johannes attempts at
> > merging the PPC PM code with the generic code in kernel/power.c
> > 
> > We've been doing fine without it so far and intend to continue to do so.
> 
> Fuse depends on !PPC?

No, that's not what I'm saying. I'm saying we've been doing STR without
the freezer and that's the way to go imho.

> > As for suspend-to-disk, I refer you to the discussions we had in the
> > past with Linus, where he explains I think quite clearly how wrong the
> > current implementation of STR is :-)
> 
> I assume you mean STD.

Oops, yeah, sorry.

> The problem there is that Linus doesn't care about STD. 
> If he did, I dare say he'd think through the issues more thoroughly than he 
> apparently has.

Heh, that might be the case :-)
 
> > Thing is, if you're going to do snapshots, you should probably not sync
> > after you have "frozen" anyway.
> 
> Fully agree. But how do you stop things syncing while you're writing the image 
> if you don't have a freezer or equivalent? (scheduler based, kexec.. they're 
> all workarounds for this issue).

Well, I was saying that in the context of the -current- snapshotting
mechanism which is based on the freezer, then you should not
sys_sync(). 

Some random user or kernel thread doing a sync is not a problem. It will
stop in the middle of sync and resume on wakeup.

The problem is currently because STD -itself- attempts to sync after it
has frozen things.

I think that should be changed. If you want to sync for whatever reason,
(mostly save RAM ?) do it before the freeze. That means you may get new
dirty data in memory that isn't written out by the sync before you
freeze, but that's allright, that data will be in the suspend image
anyway. If you fail to wakeup, that's akin to a normal crash, the user
will only lose the last data written at the time of the suspend and
journaling fs'es should take care of fs metadata integrity.

So to summarize, the plan that makes things work with fuse is:

 - For STR, don't do the freezer thing.

 - For STD, don't sys_sync() after you froze

There might be -other- issues, but that should get you through some of
them at least. Of course, you'll be in trouble if you try to do things
like STD-to-a-file which sits on a fuse FS but there's a limit to
insanity :-)

Cheers,
Ben.

Re: Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Oliver Neukum]

Am Dienstag, 3. Juli 2007 schrieb Benjamin Herrenschmidt:
> So to summarize, the plan that makes things work with fuse is:
> 
>  - For STR, don't do the freezer thing.
> 
>  - For STD, don't sys_sync() after you froze
> 
> There might be -other- issues, but that should get you through some of

At the risk of repeating myself. Character device drivers are written
with the assumption that normal io and suspend/resume do not race
with each other due to the freezer.
What do you intend to do about that?

	Regards
		Oliver

Re: Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Miklos Szeredi]

> > So to summarize, the plan that makes things work with fuse is:
> > 
> >  - For STR, don't do the freezer thing.
> > 
> >  - For STD, don't sys_sync() after you froze
> > 
> > There might be -other- issues, but that should get you through some of
> 
> At the risk of repeating myself. Character device drivers are written
> with the assumption that normal io and suspend/resume do not race
> with each other due to the freezer.
> What do you intend to do about that?

Oliver, can you please explain your worries in a bit more detail?

I don't claim to know anything about how STR or hibernate works, but
neither seem to have any problem with I/O on the fuse device "racing"
with them.

And conceptually I can't see anything that would cause trouble either.
The fuse kernel module just provides a specialized IPC mechanism,
where one userspace process communicates with another using file
operations and a char dev.

Miklos

Re: Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Oliver Neukum]

Am Dienstag, 3. Juli 2007 schrieb Miklos Szeredi:
> > > So to summarize, the plan that makes things work with fuse is:
> > > 
> > >  - For STR, don't do the freezer thing.
> > > 
> > >  - For STD, don't sys_sync() after you froze
> > > 
> > > There might be -other- issues, but that should get you through some of
> > 
> > At the risk of repeating myself. Character device drivers are written
> > with the assumption that normal io and suspend/resume do not race
> > with each other due to the freezer.
> > What do you intend to do about that?
> 
> Oliver, can you please explain your worries in a bit more detail?
> 
> I don't claim to know anything about how STR or hibernate works, but
> neither seem to have any problem with I/O on the fuse device "racing"
> with them.

The problem is not with fuse. The problem is generic in nature.

If you remove the freezer, user space remains active until the last CPU
goes into suspend. It can do syscalls. Or do you know a clean way to exempt
only the tasks fuse might use?

Now device drivers have a guaranteed temporal sequence:

last io -> suspend() -> resume() [or disconnect()] -> new io

This is because suspend() is called after the freezer goes into action. If
you remove the freezer, you need to deal with

1. io to suspended devices
2. resume() assuming that the device is in the state suspend() left it
3. io changing a device's state while suspend is saving it

and you need to fix this for all device drivers, not just those fuse is
involved with. Removing the freezer means doing a more or less full
audit of every driver and additional locking in many drivers.

	Regards
		Oliver

Re: Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Miklos Szeredi]

> > > > So to summarize, the plan that makes things work with fuse is:
> > > > 
> > > >  - For STR, don't do the freezer thing.
> > > > 
> > > >  - For STD, don't sys_sync() after you froze
> > > > 
> > > > There might be -other- issues, but that should get you through some of
> > > 
> > > At the risk of repeating myself. Character device drivers are written
> > > with the assumption that normal io and suspend/resume do not race
> > > with each other due to the freezer.
> > > What do you intend to do about that?
> > 
> > Oliver, can you please explain your worries in a bit more detail?
> > 
> > I don't claim to know anything about how STR or hibernate works, but
> > neither seem to have any problem with I/O on the fuse device "racing"
> > with them.
> 
> The problem is not with fuse. The problem is generic in nature.
> 
> If you remove the freezer, user space remains active until the last CPU
> goes into suspend. It can do syscalls. Or do you know a clean way to exempt
> only the tasks fuse might use?

You are talking about hibernate, right?  Suspending (to ram) is
instantaneous, in that _after_ suspend no CPU is active obviously.

> Now device drivers have a guaranteed temporal sequence:
> 
> last io -> suspend() -> resume() [or disconnect()] -> new io
> 
> This is because suspend() is called after the freezer goes into action. If
> you remove the freezer, you need to deal with
> 
> 1. io to suspended devices
> 2. resume() assuming that the device is in the state suspend() left it
> 3. io changing a device's state while suspend is saving it
> 
> and you need to fix this for all device drivers, not just those fuse is
> involved with. Removing the freezer means doing a more or less full
> audit of every driver and additional locking in many drivers.

OK, this has _nothing_ at all to do with fuse then, and everything to
do with disk I/O.

Just because fuse is a filesystem, it doesn't have to do anything with
block devices, just like procfs doesn't either.

So removing the freezer from the hibernate path would be problematic,
but as I understand this is not what has been proposed, only removing
the freezer from the STR path, which should be OK.

Miklos

Re: Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Oliver Neukum]

Am Dienstag, 3. Juli 2007 schrieb Miklos Szeredi:
> > > > > So to summarize, the plan that makes things work with fuse is:
> > > > > 
> > > > >  - For STR, don't do the freezer thing.
> > > > > 
> > > > >  - For STD, don't sys_sync() after you froze
> > > > > 
> > > > > There might be -other- issues, but that should get you through some of
> > > > 
> > > > At the risk of repeating myself. Character device drivers are written
> > > > with the assumption that normal io and suspend/resume do not race
> > > > with each other due to the freezer.
> > > > What do you intend to do about that?
> > > 
> > > Oliver, can you please explain your worries in a bit more detail?
> > > 
> > > I don't claim to know anything about how STR or hibernate works, but
> > > neither seem to have any problem with I/O on the fuse device "racing"
> > > with them.
> > 
> > The problem is not with fuse. The problem is generic in nature.
> > 
> > If you remove the freezer, user space remains active until the last CPU
> > goes into suspend. It can do syscalls. Or do you know a clean way to exempt
> > only the tasks fuse might use?
> 
> You are talking about hibernate, right?  Suspending (to ram) is
> instantaneous, in that _after_ suspend no CPU is active obviously.

If that is so, why do you care? If it is really atomic, fuse has no chance
to call out to its component in user space either. Removing the freezer
cannot make a difference.

Something is fishy here.

	Regards
		Oliver

Re: Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Benjamin Herrenschmidt]

On Tue, 2007-07-03 at 13:27 +0200, Oliver Neukum wrote:
> > You are talking about hibernate, right?  Suspending (to ram) is
> > instantaneous, in that _after_ suspend no CPU is active obviously.
> 
> If that is so, why do you care? If it is really atomic, fuse has no
> chance
> to call out to its component in user space either. Removing the
> freezer
> cannot make a difference.

It's not atomic. You will get called after suspend() in drivers. The
thing is ... you just have to deal with it :-)

Ben.

Re: Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Oliver Neukum]

Am Dienstag, 3. Juli 2007 schrieb Benjamin Herrenschmidt:
> On Tue, 2007-07-03 at 13:27 +0200, Oliver Neukum wrote:
> > > You are talking about hibernate, right?  Suspending (to ram) is
> > > instantaneous, in that _after_ suspend no CPU is active obviously.
> > 
> > If that is so, why do you care? If it is really atomic, fuse has no
> > chance
> > to call out to its component in user space either. Removing the
> > freezer
> > cannot make a difference.
> 
> It's not atomic. You will get called after suspend() in drivers. The
> thing is ... you just have to deal with it :-)

So you are volunteering to go through all drivers >:-> ?

	Regards
		Oliver

Re: Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Pavel Machek]

Hi!

> > > I don't claim to know anything about how STR or hibernate works, but
> > > neither seem to have any problem with I/O on the fuse device "racing"
> > > with them.
> > 
> > The problem is not with fuse. The problem is generic in nature.
> > 
> > If you remove the freezer, user space remains active until the last CPU
> > goes into suspend. It can do syscalls. Or do you know a clean way to exempt
> > only the tasks fuse might use?
> 
> You are talking about hibernate, right?  Suspending (to ram) is
> instantaneous, in that _after_ suspend no CPU is active obviously.

No, suspend to ram is not instantaneous.

We may have 16 cpus, and we may have 250 disks that need to be spun
down. That takes time, and is really not atomic operation.
								Pavel

-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

Re: Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Benjamin Herrenschmidt]

> The problem is not with fuse. The problem is generic in nature.
> 
> If you remove the freezer, user space remains active until the last CPU
> goes into suspend. It can do syscalls. Or do you know a clean way to exempt
> only the tasks fuse might use?
>
> Now device drivers have a guaranteed temporal sequence:
> 
> last io -> suspend() -> resume() [or disconnect()] -> new io

No, that's always been bullshit. You can have IOs emitted by kernel
threads (think knfsd, and that's just one among many others). Beside,
relying on having userland frozen means that your driver will be unable
to be "live" suspended/resumed for more ambitious dynamic power
management schemes.

So it's always been wrong, imho, to rely on that. I've had powermac STR
work fine without the freezer for years, and few drivers have been a
problem, and we just fixed them.

The freezer thingy, at best, hides problems, causing them not to be
fixed.

> This is because suspend() is called after the freezer goes into action. If
> you remove the freezer, you need to deal with
> 
> 1. io to suspended devices
> 2. resume() assuming that the device is in the state suspend() left it
> 3. io changing a device's state while suspend is saving it
> 
> and you need to fix this for all device drivers, not just those fuse is
> involved with. Removing the freezer means doing a more or less full
> audit of every driver and additional locking in many drivers.

Yes, more or less.

The good news is that a whole lot of drivers don't really care much, and
in some cases, things can be done trivially with a bit of help from the
upper layers. But yeah, as I've been explaining over and over again, the
lazy approach here doesn't work.

Ben.

Re: Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Oliver Neukum]

Am Dienstag, 3. Juli 2007 schrieb Benjamin Herrenschmidt:
> > Now device drivers have a guaranteed temporal sequence:
> > 
> > last io -> suspend() -> resume() [or disconnect()] -> new io
> 
> No, that's always been bullshit. You can have IOs emitted by kernel
> threads (think knfsd, and that's just one among many others). Beside,

That's why we have the problem of freezing the kernel threads or not.
Short of knfsd very few kernel threads really operate on their own and
those can use the new notifier chain.

> relying on having userland frozen means that your driver will be unable
> to be "live" suspended/resumed for more ambitious dynamic power
> management schemes.

Only if they work without cooperation and idle detection in the drivers.

> So it's always been wrong, imho, to rely on that. I've had powermac STR
> work fine without the freezer for years, and few drivers have been a
> problem, and we just fixed them.

Powermacs are somewhat limited in hardware used with them (OK, powerbook
have PCMCIA, but how often is that used?)

You want to have all that pain for fuse?

	Regards
		Oliver

Re: Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Paul Mackerras]

Oliver Neukum writes:

> That's why we have the problem of freezing the kernel threads or not.

That problem is a symptom of the deeper conceptual problem, as is the
problem with FUSE.

> You want to have all that pain for fuse?

I'd certainly rather get the drivers right, and maybe have an
occasional deadlock if I miss something, than have a GUARANTEED
deadlock every time I suspend with a FUSE filesystem mounted (which is
pretty much every time, since I use encfs regularly).

Paul.

Re: Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Miklos Szeredi]

> > That's why we have the problem of freezing the kernel threads or not.
> 
> That problem is a symptom of the deeper conceptual problem, as is the
> problem with FUSE.
> 
> > You want to have all that pain for fuse?
> 
> I'd certainly rather get the drivers right, and maybe have an
> occasional deadlock if I miss something, than have a GUARANTEED
> deadlock every time I suspend with a FUSE filesystem mounted (which is
> pretty much every time, since I use encfs regularly).

That's weird, I never had a suspend problem due to a fuse mount,
though I have them all the time.  And I suspect, that even the sync()
thing that suspend does is not the real cause, because sync() actually
does nothing in fuse filesystems.

So there's something else going on, which obviously has to do with
freezing user processes, but it's not clear what.

Miklos

Re: Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Paul Mackerras]

Miklos Szeredi writes:

> That's weird, I never had a suspend problem due to a fuse mount,
> though I have them all the time.  And I suspect, that even the sync()

Well, I don't either, because we don't freeze processes on
powerbooks.  But I have heard that other people have problems with
suspending with a fuse filesystem mounted.  Maybe the difference is
whether or not the filesystem is writable?

> thing that suspend does is not the real cause, because sync() actually
> does nothing in fuse filesystems.

It's not the filesystem sync method, as I understand it, it's that if
there are dirty pages in the page cache for files on the fuse
filesystem, the system will initiate a write-out on them and wait for
it to finish.  But if the fuse userspace is frozen, the write-out will
never complete.

Paul.

Re: Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Miklos Szeredi]

> > That's weird, I never had a suspend problem due to a fuse mount,
> > though I have them all the time.  And I suspect, that even the sync()
> 
> Well, I don't either, because we don't freeze processes on
> powerbooks.  But I have heard that other people have problems with
> suspending with a fuse filesystem mounted.  Maybe the difference is
> whether or not the filesystem is writable?
> 
> > thing that suspend does is not the real cause, because sync() actually
> > does nothing in fuse filesystems.
> 
> It's not the filesystem sync method, as I understand it, it's that if
> there are dirty pages in the page cache for files on the fuse
> filesystem,

Currently fuse doesn't produce dirty pages.  Normal writes are done
synchronously, and writable mmap is not supported.  So sync() should
really be a no-op for fuse.

> the system will initiate a write-out on them and wait for it to
> finish.  But if the fuse userspace is frozen, the write-out will
> never complete.

Maybe there is some other fs operation being done, possibly not
directly, but by waiting for a kernel thread, that does that.

It would be nice, if someone who can reproduce the deadlock could
debug it.  Does sysrq still work during suspend?

Miklos

Re: Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Rafael J. Wysocki]

On Wednesday, 4 July 2007 10:26, Miklos Szeredi wrote:
> > > That's weird, I never had a suspend problem due to a fuse mount,
> > > though I have them all the time.  And I suspect, that even the sync()
> > 
> > Well, I don't either, because we don't freeze processes on
> > powerbooks.  But I have heard that other people have problems with
> > suspending with a fuse filesystem mounted.  Maybe the difference is
> > whether or not the filesystem is writable?
> > 
> > > thing that suspend does is not the real cause, because sync() actually
> > > does nothing in fuse filesystems.
> > 
> > It's not the filesystem sync method, as I understand it, it's that if
> > there are dirty pages in the page cache for files on the fuse
> > filesystem,
> 
> Currently fuse doesn't produce dirty pages.  Normal writes are done
> synchronously, and writable mmap is not supported.  So sync() should
> really be a no-op for fuse.
> 
> > the system will initiate a write-out on them and wait for it to
> > finish.  But if the fuse userspace is frozen, the write-out will
> > never complete.
> 
> Maybe there is some other fs operation being done, possibly not
> directly, but by waiting for a kernel thread, that does that.

We're going to limit the freezing of kernel threads to the ones that explicitly
want to be frozen, so if that's the case, then I think it'll be fixed soon.

> It would be nice, if someone who can reproduce the deadlock could
> debug it.

Agreed.

> Does sysrq still work during suspend? 

Yes, it should work.

Greetings,
Rafael


-- 
"Premature optimization is the root of all evil." - Donald Knuth

Re: Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Rafael J. Wysocki]

On Tuesday, 3 July 2007 13:07, Oliver Neukum wrote:
> Am Dienstag, 3. Juli 2007 schrieb Miklos Szeredi:
> > > > So to summarize, the plan that makes things work with fuse is:
> > > > 
> > > >  - For STR, don't do the freezer thing.
> > > > 
> > > >  - For STD, don't sys_sync() after you froze
> > > > 
> > > > There might be -other- issues, but that should get you through some of
> > > 
> > > At the risk of repeating myself. Character device drivers are written
> > > with the assumption that normal io and suspend/resume do not race
> > > with each other due to the freezer.
> > > What do you intend to do about that?
> > 
> > Oliver, can you please explain your worries in a bit more detail?
> > 
> > I don't claim to know anything about how STR or hibernate works, but
> > neither seem to have any problem with I/O on the fuse device "racing"
> > with them.
> 
> The problem is not with fuse. The problem is generic in nature.
> 
> If you remove the freezer, user space remains active until the last CPU
> goes into suspend. It can do syscalls. Or do you know a clean way to exempt
> only the tasks fuse might use?
> 
> Now device drivers have a guaranteed temporal sequence:
> 
> last io -> suspend() -> resume() [or disconnect()] -> new io
> 
> This is because suspend() is called after the freezer goes into action. If
> you remove the freezer, you need to deal with
> 
> 1. io to suspended devices
> 2. resume() assuming that the device is in the state suspend() left it
> 3. io changing a device's state while suspend is saving it
> 
> and you need to fix this for all device drivers, not just those fuse is
> involved with. Removing the freezer means doing a more or less full
> audit of every driver and additional locking in many drivers.

Agreed.

Greetings,
Rafael


-- 
"Premature optimization is the root of all evil." - Donald Knuth

Re: Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Miklos Szeredi]

> > > > So to summarize, the plan that makes things work with fuse is:
> > > > 
> > > >  - For STR, don't do the freezer thing.
> > > > 
> > > >  - For STD, don't sys_sync() after you froze
> > > > 
> > > > There might be -other- issues, but that should get you through some of
> > > 
> > > At the risk of repeating myself. Character device drivers are written
> > > with the assumption that normal io and suspend/resume do not race
> > > with each other due to the freezer.
> > > What do you intend to do about that?
> > 
> > Oliver, can you please explain your worries in a bit more detail?
> > 
> > I don't claim to know anything about how STR or hibernate works, but
> > neither seem to have any problem with I/O on the fuse device "racing"
> > with them.
> 
> The problem is not with fuse. The problem is generic in nature.
> 
> If you remove the freezer, user space remains active until the last CPU
> goes into suspend. It can do syscalls. Or do you know a clean way to exempt
> only the tasks fuse might use?
> 
> Now device drivers have a guaranteed temporal sequence:
> 
> last io -> suspend() -> resume() [or disconnect()] -> new io
> 
> This is because suspend() is called after the freezer goes into action. If
> you remove the freezer, you need to deal with
> 
> 1. io to suspended devices
> 2. resume() assuming that the device is in the state suspend() left it
> 3. io changing a device's state while suspend is saving it
> 
> and you need to fix this for all device drivers, not just those fuse is
> involved with.

Fuse is not involved with _any_ device drivers.  It is fully unaware
of suspend issues and I think that's how it should stay ;)

> Removing the freezer means doing a more or less full
> audit of every driver and additional locking in many drivers.

How about a "CONFIG_NOFREEZE (experimental): only turn this on if you
want to fix buggy drivers that can fail during suspend with the
freezer turned off"?

I'm guessing quite a few kernel developers would be willing to turn on
such an option.

Miklos

Re: Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Paul Mackerras]

Oliver Neukum writes:

> At the risk of repeating myself. Character device drivers are written
> with the assumption that normal io and suspend/resume do not race
> with each other due to the freezer.
> What do you intend to do about that?

Going back to the old powerbook sleep code, we had a two-phase
suspend: drivers got notified once when userspace is still running,
with interrupts enabled, in process context; and then a second time
with interrupts disabled and with only one CPU up, so the process
that is initiating the suspend is the only process running (since
interrupts are disabled and nothing it does can sleep, no other
process can get to run).

I still believe that is the right way to go, although we currently
only have a single-phase suspend.

Most drivers suspended their hardware in the second call.  If they are
in the middle of a conversation with their device that *has* to be
completed, they can do that by polling.  If it's a character device, a
better approach would be to set a flag or whatever in the first
suspend call to make sure that no new conversations get started with
the device, sleeping if necessary.

I'm actually having a hard time thinking of how to test your assertion
since there are so few things on a typical computer that are plain
character devices driving real hardware.  A serial port would be about
the only one; keyboards and mice (and serial ports :) are USB these
days, or ADB on older powerbooks.

Or did you mean to include drivers for pseudo-devices (e.g. ptys)?  I
don't see why they would have a suspend method at all.

Paul.

Re: Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Benjamin Herrenschmidt]

On Tue, 2007-07-03 at 09:44 +0200, Oliver Neukum wrote:
> Am Dienstag, 3. Juli 2007 schrieb Benjamin Herrenschmidt:
> > So to summarize, the plan that makes things work with fuse is:
> > 
> >  - For STR, don't do the freezer thing.
> > 
> >  - For STD, don't sys_sync() after you froze
> > 
> > There might be -other- issues, but that should get you through some of
> 
> At the risk of repeating myself. Character device drivers are written
> with the assumption that normal io and suspend/resume do not race
> with each other due to the freezer.
> What do you intend to do about that?

Ugh ... "character devices" ... that's a pretty wide statement...
there's lots of those and very different one from the other...

Any sane device-driver will have to cope with being suspended in a
"live" system. I've demonstrated multiple times in the past why this is
necessary anyway, for things like dynamic power management, among
others.

The whole freezer thing is a hack job to avoid fixing drivers that need
fixing. Unfortunately, I believe in that area, it's simply not
sustainable. Besides, getting drivers to behave properly isn't very hard
in most cases.

Ben.

Re: Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Oliver Neukum]

Am Dienstag, 3. Juli 2007 schrieb Benjamin Herrenschmidt:
> On Tue, 2007-07-03 at 09:44 +0200, Oliver Neukum wrote:
> > Am Dienstag, 3. Juli 2007 schrieb Benjamin Herrenschmidt:
> > > So to summarize, the plan that makes things work with fuse is:
> > > 
> > >  - For STR, don't do the freezer thing.
> > > 
> > >  - For STD, don't sys_sync() after you froze
> > > 
> > > There might be -other- issues, but that should get you through some of
> > 
> > At the risk of repeating myself. Character device drivers are written
> > with the assumption that normal io and suspend/resume do not race
> > with each other due to the freezer.
> > What do you intend to do about that?
> 
> Ugh ... "character devices" ... that's a pretty wide statement...
> there's lots of those and very different one from the other...

That is a good summary of the problem ;-(

> Any sane device-driver will have to cope with being suspended in a
> "live" system. I've demonstrated multiple times in the past why this is
> necessary anyway, for things like dynamic power management, among
> others.

That is an interesting notion. I'd rather see device drivers reporting
their devices idle and requsting to be suspended.
But in any case it doesn't solve the problem.

	Regards
		Oliver

Re: Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Rafael J. Wysocki]

On Tuesday, 3 July 2007 13:46, Oliver Neukum wrote:
> Am Dienstag, 3. Juli 2007 schrieb Benjamin Herrenschmidt:
> > On Tue, 2007-07-03 at 09:44 +0200, Oliver Neukum wrote:
> > > Am Dienstag, 3. Juli 2007 schrieb Benjamin Herrenschmidt:
> > > > So to summarize, the plan that makes things work with fuse is:
> > > > 
> > > >  - For STR, don't do the freezer thing.
> > > > 
> > > >  - For STD, don't sys_sync() after you froze
> > > > 
> > > > There might be -other- issues, but that should get you through some of
> > > 
> > > At the risk of repeating myself. Character device drivers are written
> > > with the assumption that normal io and suspend/resume do not race
> > > with each other due to the freezer.
> > > What do you intend to do about that?
> > 
> > Ugh ... "character devices" ... that's a pretty wide statement...
> > there's lots of those and very different one from the other...
> 
> That is a good summary of the problem ;-(
> 
> > Any sane device-driver will have to cope with being suspended in a
> > "live" system. I've demonstrated multiple times in the past why this is
> > necessary anyway, for things like dynamic power management, among
> > others.
> 
> That is an interesting notion. I'd rather see device drivers reporting
> their devices idle and requsting to be suspended.
> But in any case it doesn't solve the problem.

Agreed.

What I think will solve the problem in the long run is to:

1) Separate the hibernation code from the suspend code (ie. hibernation-related
callbacks should generally be different from suspend-related callback for each
driver).
2) Remove the freezing of kernel threads from each of them (in the hibernation
case, if possible) an fix the things that get broken.
3) Remove the freezing of user space from the suspend code path and fix the
things that get broken.

Going to step 3) before doing 1) and 2) doesn't seem to be the right thing to
me.

Greetings,
Rafael


-- 
"Premature optimization is the root of all evil." - Donald Knuth

Re: Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Oliver Neukum]

Am Dienstag, 3. Juli 2007 schrieb Paul Mackerras:
> I'm actually having a hard time thinking of how to test your assertion
> since there are so few things on a typical computer that are plain
> character devices driving real hardware.  A serial port would be about
> the only one; keyboards and mice (and serial ports :) are USB these
> days, or ADB on older powerbooks.

USB devices certainly have suspend methods.

	Regards
		Oliver

Re: Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Paul Mackerras]

Oliver Neukum writes:

> USB devices certainly have suspend methods.

Indeed, and the USB framework has code to know when the host
controller is suspended and avoid trying to send out urbs in that
case.  Or at least it did last time I looked at it in any detail; it's
been "just working" - including suspending and resuming, without the
freezer - for quite a while now.

Paul.

Re: Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Oliver Neukum]

Am Mittwoch, 4. Juli 2007 schrieb Paul Mackerras:
> Oliver Neukum writes:
> 
> > USB devices certainly have suspend methods.
> 
> Indeed, and the USB framework has code to know when the host
> controller is suspended and avoid trying to send out urbs in that
> case.  Or at least it did last time I looked at it in any detail; it's
> been "just working" - including suspending and resuming, without the
> freezer - for quite a while now.

And what happens to that IO? I suspended and lost data is not an
acceptable behavior.

	Oliver

Re: Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Paul Mackerras]

Oliver Neukum writes:

> > Indeed, and the USB framework has code to know when the host
> > controller is suspended and avoid trying to send out urbs in that
> > case.  Or at least it did last time I looked at it in any detail; it's
> > been "just working" - including suspending and resuming, without the
> > freezer - for quite a while now.
> 
> And what happens to that IO? I suspended and lost data is not an
> acceptable behavior.

It's not lost, it's sitting in RAM, and will be sent out when you
resume.

Paul.

Re: Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Oliver Neukum]

Am Mittwoch, 4. Juli 2007 schrieb Paul Mackerras:
> Oliver Neukum writes:
> 
> > > Indeed, and the USB framework has code to know when the host
> > > controller is suspended and avoid trying to send out urbs in that
> > > case.  Or at least it did last time I looked at it in any detail; it's
> > > been "just working" - including suspending and resuming, without the
> > > freezer - for quite a while now.
> > 
> > And what happens to that IO? I suspended and lost data is not an
> > acceptable behavior.
> 
> It's not lost, it's sitting in RAM, and will be sent out when you
> resume.

Unfortunately this is not the case. The URB will error out.

	Oliver

Re: Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Paul Mackerras]

Oliver Neukum writes:

> > It's not lost, it's sitting in RAM, and will be sent out when you
> > resume.
> 
> Unfortunately this is not the case. The URB will error out.

So the higher-level driver needs to do the sensible thing, i.e.,
resubmit the URB after resume.  It's not rocket science.  The data is
not lost, it's sitting in RAM, and the higher-level driver will send
it out when you resume.  If not, then we fix the higher-level driver.

Of course with USB there is the interesting question of whether the
device is still there when we resume.  But if it isn't, the situation
is no different to the user asynchronously unplugging the device
during operation, and if we lose data in that situation, we can only
blame the user. :)

Paul.

Re: Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Oliver Neukum]

Am Mittwoch, 4. Juli 2007 schrieb Paul Mackerras:
> Oliver Neukum writes:
> 
> > > It's not lost, it's sitting in RAM, and will be sent out when you
> > > resume.
> > 
> > Unfortunately this is not the case. The URB will error out.
> 
> So the higher-level driver needs to do the sensible thing, i.e.,
> resubmit the URB after resume.  It's not rocket science.  The data is
> not lost, it's sitting in RAM, and the higher-level driver will send
> it out when you resume.  If not, then we fix the higher-level driver.

You cannot simply restart the URB without thinking.
The device after resumption may or may not be in the stage
you left it. It needs to be rechecked and some settings must be
renewed. You cannot simple throw an URB from an arbitrary
stage of the protocol at it.
Suspension of devices can only happen at some points
in the protocol.

	Oliver

Re: Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Paul Mackerras]

Oliver Neukum writes:

> You cannot simply restart the URB without thinking.
> The device after resumption may or may not be in the stage
> you left it. It needs to be rechecked and some settings must be
> renewed. You cannot simple throw an URB from an arbitrary
> stage of the protocol at it.
> Suspension of devices can only happen at some points
> in the protocol.

Yeah, and?

I said "the higher-level driver needs to do the sensible thing".

Paul.

Re: Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Oliver Neukum]

Am Mittwoch, 4. Juli 2007 schrieb Paul Mackerras:
> Oliver Neukum writes:
> 
> > You cannot simply restart the URB without thinking.
> > The device after resumption may or may not be in the stage
> > you left it. It needs to be rechecked and some settings must be
> > renewed. You cannot simple throw an URB from an arbitrary
> > stage of the protocol at it.
> > Suspension of devices can only happen at some points
> > in the protocol.
> 
> Yeah, and?
> 
> I said "the higher-level driver needs to do the sensible thing".

They can't. Device specific protocols are known to the drivers only.
The fact remains, remove the freezer and you need to go through
all drivers.

	Oliver

Re: Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Paul Mackerras]

Oliver Neukum writes:

> They can't. Device specific protocols are known to the drivers only.
> The fact remains, remove the freezer and you need to go through
> all drivers.

The freezer does not actually mean that you don't have to get the
drivers right, because kernel threads can issue I/O requests.

Paul.

Re: Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Oliver Neukum]

Am Mittwoch, 4. Juli 2007 schrieb Paul Mackerras:
> Oliver Neukum writes:
> 
> > They can't. Device specific protocols are known to the drivers only.
> > The fact remains, remove the freezer and you need to go through
> > all drivers.
> 
> The freezer does not actually mean that you don't have to get the
> drivers right, because kernel threads can issue I/O requests.

Kernel threads do not issue requests for the hell of it. And yes,
kernel threads must be aware of suspension. Threads are few,
drivers many.

	Oliver

Re: Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Alan Stern]

On Wed, 4 Jul 2007, Paul Mackerras wrote:

> Oliver Neukum writes:
> 
> > USB devices certainly have suspend methods.
> 
> Indeed, and the USB framework has code to know when the host
> controller is suspended and avoid trying to send out urbs in that
> case.  Or at least it did last time I looked at it in any detail; it's
> been "just working" - including suspending and resuming, without the
> freezer - for quite a while now.

Evidently you haven't been stressing it.  You might try suspending
while printing to a USB printer.

Alan Stern

Re: Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Alan Stern]

On Tue, 3 Jul 2007, Paul Mackerras wrote:

> Going back to the old powerbook sleep code, we had a two-phase
> suspend: drivers got notified once when userspace is still running,
> with interrupts enabled, in process context; and then a second time
> with interrupts disabled and with only one CPU up, so the process
> that is initiating the suspend is the only process running (since
> interrupts are disabled and nothing it does can sleep, no other
> process can get to run).
> 
> I still believe that is the right way to go, although we currently
> only have a single-phase suspend.
> 
> Most drivers suspended their hardware in the second call.  If they are
> in the middle of a conversation with their device that *has* to be
> completed, they can do that by polling.

Ugh.  That will cause problems when you try to integrate runtime 
suspend.  In fact this whole approach is unsuitable for runtime PM and 
it obscures the similarities between runtime PM and STR.

>  If it's a character device, a
> better approach would be to set a flag or whatever in the first
> suspend call to make sure that no new conversations get started with
> the device, sleeping if necessary.
> 
> I'm actually having a hard time thinking of how to test your assertion
> since there are so few things on a typical computer that are plain
> character devices driving real hardware.  A serial port would be about
> the only one; keyboards and mice (and serial ports :) are USB these
> days, or ADB on older powerbooks.

You don't have to restrict yourself to character devices driving real 
hardware.  The same issues apply to USB and other buses.

Alan Stern

Re: Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Paul Mackerras]

Alan Stern writes:

> > Most drivers suspended their hardware in the second call.  If they are
> > in the middle of a conversation with their device that *has* to be
> > completed, they can do that by polling.
> 
> Ugh.  That will cause problems when you try to integrate runtime 
> suspend.  In fact this whole approach is unsuitable for runtime PM and 
> it obscures the similarities between runtime PM and STR.

Yes there are similarities, but it would be a big mistake to say that
a requirement for STR is that all drivers do runtime PM.

If a driver does runtime PM, that's great, and it is useful for
implementing STR.  However, there are a class of devices for which
runtime PM is not possible or not useful, but which can suspend/resume
just fine as part of suspending/resuming the complete system, and for
which all that is needed is some small amount of simple hardware
poking just before the system as a whole is put into suspend.  For
those a late-suspend call with interrupts off is the simplest and best
way to go.

Think of a serial port on a motherboard for instance, where the only
power control is the overall power control for the system.  All that
is needed is to poll for the transmitter being empty (with timeout, of
course) in the late-suspend call (and possibly also turn off output
drivers, perhaps), and to reinitialize some registers in an
early-resume call.

The main attraction of the late-suspend call is that it really does,
reliably, guarantee that the driver's I/O request methods won't get
called between the late-suspend call and the early-resume call.

Paul.

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Rafael J. Wysocki]

On Tuesday, 3 July 2007 09:19, Benjamin Herrenschmidt wrote:
> On Tue, 2007-07-03 at 16:08 +1000, Nigel Cunningham wrote:
> > 
> > > So I think Matthew is totally right. In fact, the presence of the
> > > freezer is the main reason why Paulus so far NACKed Johannes attempts at
> > > merging the PPC PM code with the generic code in kernel/power.c
> > > 
> > > We've been doing fine without it so far and intend to continue to do so.
> > 
> > Fuse depends on !PPC?
> 
> No, that's not what I'm saying. I'm saying we've been doing STR without
> the freezer and that's the way to go imho.
> 
> > > As for suspend-to-disk, I refer you to the discussions we had in the
> > > past with Linus, where he explains I think quite clearly how wrong the
> > > current implementation of STR is :-)
> > 
> > I assume you mean STD.
> 
> Oops, yeah, sorry.
> 
> > The problem there is that Linus doesn't care about STD. 
> > If he did, I dare say he'd think through the issues more thoroughly than he 
> > apparently has.
> 
> Heh, that might be the case :-)
>  
> > > Thing is, if you're going to do snapshots, you should probably not sync
> > > after you have "frozen" anyway.
> > 
> > Fully agree. But how do you stop things syncing while you're writing the image 
> > if you don't have a freezer or equivalent? (scheduler based, kexec.. they're 
> > all workarounds for this issue).
> 
> Well, I was saying that in the context of the -current- snapshotting
> mechanism which is based on the freezer, then you should not
> sys_sync(). 
> 
> Some random user or kernel thread doing a sync is not a problem. It will
> stop in the middle of sync and resume on wakeup.
> 
> The problem is currently because STD -itself- attempts to sync after it
> has frozen things.

To be precise, it tries to sync after it has frozen the user land.
 
> I think that should be changed. If you want to sync for whatever reason,
> (mostly save RAM ?) do it before the freeze. That means you may get new
> dirty data in memory that isn't written out by the sync before you
> freeze, but that's allright, that data will be in the suspend image
> anyway. If you fail to wakeup, that's akin to a normal crash, the user
> will only lose the last data written at the time of the suspend and
> journaling fs'es should take care of fs metadata integrity.
> 
> So to summarize, the plan that makes things work with fuse is:
> 
>  - For STR, don't do the freezer thing.

In the long run, I agree.

Still, can you please read this post from Alan Stern:

https://lists.linux-foundation.org/pipermail/linux-pm/2007-June/012847.html

?  I don't think I'm able to repeat the arguments given in there in a
convincing way.

>  - For STD, don't sys_sync() after you froze

Yeah, I think we can move the syncing before the freezing, so to speak.

And it need not be called from within the freezer, BTW.

> There might be -other- issues, but that should get you through some of
> them at least. Of course, you'll be in trouble if you try to do things
> like STD-to-a-file which sits on a fuse FS but there's a limit to
> insanity :-)

Yes. :-)

Ggreetings,
Rafael


-- 
"Premature optimization is the root of all evil." - Donald Knuth

Re: Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Johannes Berg]

[-- Attachment #1.1: Type: text/plain, Size: 1073 bytes --]

On Tue, 2007-07-03 at 14:56 +0200, Rafael J. Wysocki wrote:

> Still, can you please read this post from Alan Stern:
> 
> https://lists.linux-foundation.org/pipermail/linux-pm/2007-June/012847.html
> 
> ?  I don't think I'm able to repeat the arguments given in there in a
> convincing way.

As I read it, Alan basically has two objections:
 (1) drivers shouldn't need to worry about this
 (2) suspend should be transparent to userspace

His proposed solution (freezing tasks when they cross the kernel
boundary) helps for the s-t-r case, but in fact doesn't solve (1)
because devices can be suspended at runtime and then you certainly do
not want to freeze tasks that try to access the device.

(2) is related but not identical, what if you have a device suspended at
runtime and some tasks tries to access it; should the task block until
you wake up that device?

I think the core of the discussion isn't appreciated by everybody here
yet---we need to solve both run-time and suspend-to-ram-time device
suspend, not just one of them.

johannes

[-- Attachment #1.2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 190 bytes --]

[-- Attachment #2: Type: text/plain, Size: 0 bytes --]

Re: Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Alan Stern]

On Tue, 3 Jul 2007, Johannes Berg wrote:

> On Tue, 2007-07-03 at 14:56 +0200, Rafael J. Wysocki wrote:
> 
> > Still, can you please read this post from Alan Stern:
> > 
> > https://lists.linux-foundation.org/pipermail/linux-pm/2007-June/012847.html
> > 
> > ?  I don't think I'm able to repeat the arguments given in there in a
> > convincing way.
> 
> As I read it, Alan basically has two objections:
>  (1) drivers shouldn't need to worry about this
>  (2) suspend should be transparent to userspace
> 
> His proposed solution (freezing tasks when they cross the kernel
> boundary) helps for the s-t-r case, but in fact doesn't solve (1)
> because devices can be suspended at runtime and then you certainly do
> not want to freeze tasks that try to access the device.
> 
> (2) is related but not identical, what if you have a device suspended at
> runtime and some tasks tries to access it; should the task block until
> you wake up that device?

Time for me to jump in.

USB already implements runtime PM.  If a device is suspended at runtime
and a task tries to access it, the device is automatically resumed.  
No problem there.

The problem comes when the system is doing a STR.  Right now the code
doesn't keep track of the difference between a runtime suspend and a
system suspend -- once the device is suspended, it's suspended, period.  
Consequently, a non-frozen user task trying to do I/O to a suspended
device during STR will cause that device to resume, thereby forcing the
system suspend to abort.  Something much like this has actually
happened and been reported as a bug on LKML (I don't have a URL handy,
and it was actually a non-frozen kernel thread interfering with
hibernate rather than a non-frozen user task interfering with STR, but
the principle is the same).

Yes, the code could be changed to keep track of the reason for a device
suspend.  But that just raises the old problem of what to do when
there's an I/O request for a suspended device during STR.

> I think the core of the discussion isn't appreciated by everybody here
> yet---we need to solve both run-time and suspend-to-ram-time device
> suspend, not just one of them.

Runtime suspend isn't a problem.  Only STR.

Consider a particularly troublesome case: During STR, a non-frozen task
writes to /sys/bus/BBB/drivers/DDD/bind.  The sysfs core grabs the
device semaphore and calls the driver's probe routine.  If the driver
isn't PM-aware it simply tries to initialize the device and fails
because the device is already suspended.  That's no good; it isn't
transparent.

So assume the driver is PM-aware.  It tries to resume the device, which
fails because STR is underway.  Now what can it do?  There's only one 
possibility: It must block until the resume call can succeed.  But when 
is that?

It has to be before the PM core tries to resume the device, because the 
core will try to acquire the device semaphore and will block waiting 
for the probe call to complete.  But it has to be after the PM core 
resumes the device's parent, because obviously the device can't resume 
until its parent is awake.

As you can see, this is a very difficult problem to solve.

Alan Stern

Re: Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Johannes Berg]

[-- Attachment #1.1: Type: text/plain, Size: 3019 bytes --]

On Tue, 2007-07-03 at 10:50 -0400, Alan Stern wrote:

> Time for me to jump in.

:)

> USB already implements runtime PM.  If a device is suspended at runtime
> and a task tries to access it, the device is automatically resumed.  
> No problem there.

Right.

> The problem comes when the system is doing a STR.  Right now the code
> doesn't keep track of the difference between a runtime suspend and a
> system suspend -- once the device is suspended, it's suspended, period.  
> Consequently, a non-frozen user task trying to do I/O to a suspended
> device during STR will cause that device to resume, thereby forcing the
> system suspend to abort.  Something much like this has actually
> happened and been reported as a bug on LKML (I don't have a URL handy,
> and it was actually a non-frozen kernel thread interfering with
> hibernate rather than a non-frozen user task interfering with STR, but
> the principle is the same).

Yeah, I can see that happen.

> Yes, the code could be changed to keep track of the reason for a device
> suspend.  But that just raises the old problem of what to do when
> there's an I/O request for a suspended device during STR.
> 
> > I think the core of the discussion isn't appreciated by everybody here
> > yet---we need to solve both run-time and suspend-to-ram-time device
> > suspend, not just one of them.
> 
> Runtime suspend isn't a problem.  Only STR.

Ah but for all those character devices people were saying are the
problem we haven't even solved runtime suspend as far as I can tell from
the discussion.

> Consider a particularly troublesome case: During STR, a non-frozen task
> writes to /sys/bus/BBB/drivers/DDD/bind.  The sysfs core grabs the
> device semaphore and calls the driver's probe routine.  If the driver
> isn't PM-aware it simply tries to initialize the device and fails
> because the device is already suspended.  That's no good; it isn't
> transparent.
> 
> So assume the driver is PM-aware.  It tries to resume the device, which
> fails because STR is underway.  Now what can it do?  There's only one 
> possibility: It must block until the resume call can succeed.  But when 
> is that?
> 
> It has to be before the PM core tries to resume the device, because the 
> core will try to acquire the device semaphore and will block waiting 
> for the probe call to complete.  But it has to be after the PM core 
> resumes the device's parent, because obviously the device can't resume 
> until its parent is awake.
> 
> As you can see, this is a very difficult problem to solve.

Indeed. Actually, one could argue that it's impossible to solve the
problem as long as we try to call out to userspace during suspend and
need to wait until that's finished, like in the case of sys_sync() and
fuse filesystems, and probably other cases. Maybe we should make *those*
calls return a failure so that the suspend isn't transparent inside the
kernel but is transparent to userspace.

johannes

[-- Attachment #1.2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 190 bytes --]

[-- Attachment #2: Type: text/plain, Size: 0 bytes --]

Re: Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Rafael J. Wysocki]

On Tuesday, 3 July 2007 16:59, Johannes Berg wrote:
> On Tue, 2007-07-03 at 10:50 -0400, Alan Stern wrote:
> 
> > Time for me to jump in.
> 
> :)
> 
> > USB already implements runtime PM.  If a device is suspended at runtime
> > and a task tries to access it, the device is automatically resumed.  
> > No problem there.
> 
> Right.
> 
> > The problem comes when the system is doing a STR.  Right now the code
> > doesn't keep track of the difference between a runtime suspend and a
> > system suspend -- once the device is suspended, it's suspended, period.  
> > Consequently, a non-frozen user task trying to do I/O to a suspended
> > device during STR will cause that device to resume, thereby forcing the
> > system suspend to abort.  Something much like this has actually
> > happened and been reported as a bug on LKML (I don't have a URL handy,
> > and it was actually a non-frozen kernel thread interfering with
> > hibernate rather than a non-frozen user task interfering with STR, but
> > the principle is the same).
> 
> Yeah, I can see that happen.
> 
> > Yes, the code could be changed to keep track of the reason for a device
> > suspend.  But that just raises the old problem of what to do when
> > there's an I/O request for a suspended device during STR.
> > 
> > > I think the core of the discussion isn't appreciated by everybody here
> > > yet---we need to solve both run-time and suspend-to-ram-time device
> > > suspend, not just one of them.
> > 
> > Runtime suspend isn't a problem.  Only STR.
> 
> Ah but for all those character devices people were saying are the
> problem we haven't even solved runtime suspend as far as I can tell from
> the discussion.
> 
> > Consider a particularly troublesome case: During STR, a non-frozen task
> > writes to /sys/bus/BBB/drivers/DDD/bind.  The sysfs core grabs the
> > device semaphore and calls the driver's probe routine.  If the driver
> > isn't PM-aware it simply tries to initialize the device and fails
> > because the device is already suspended.  That's no good; it isn't
> > transparent.
> > 
> > So assume the driver is PM-aware.  It tries to resume the device, which
> > fails because STR is underway.  Now what can it do?  There's only one 
> > possibility: It must block until the resume call can succeed.  But when 
> > is that?
> > 
> > It has to be before the PM core tries to resume the device, because the 
> > core will try to acquire the device semaphore and will block waiting 
> > for the probe call to complete.  But it has to be after the PM core 
> > resumes the device's parent, because obviously the device can't resume 
> > until its parent is awake.
> > 
> > As you can see, this is a very difficult problem to solve.
> 
> Indeed. Actually, one could argue that it's impossible to solve the
> problem as long as we try to call out to userspace during suspend and
> need to wait until that's finished, like in the case of sys_sync() and
> fuse filesystems, and probably other cases. Maybe we should make *those*
> calls return a failure so that the suspend isn't transparent inside the
> kernel but is transparent to userspace.

Well, it generally needs more consideration. :-)

I think that we should introduce mechanisms that will allow us to notify all
kernel subsystems, including FUSE and similar, that the system is going to
enter a sleep state (one of those is the notifier chain introduced recently).

Then, they may react to such a notification by entering a "suspend" mode
of operation in which they will return errors from some callbacks that
otherwise should have succeeded etc.  That depends on the subsystem in
question.

Greetings,
Rafael


-- 
"Premature optimization is the root of all evil." - Donald Knuth

Re: Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Miklos Szeredi]

> > Indeed. Actually, one could argue that it's impossible to solve the
> > problem as long as we try to call out to userspace during suspend and
> > need to wait until that's finished, like in the case of sys_sync() and
> > fuse filesystems, and probably other cases. Maybe we should make *those*
> > calls return a failure so that the suspend isn't transparent inside the
> > kernel but is transparent to userspace.
> 
> Well, it generally needs more consideration. :-)
> 
> I think that we should introduce mechanisms that will allow us to notify all
> kernel subsystems, including FUSE and similar, that the system is going to
> enter a sleep state (one of those is the notifier chain introduced recently).

Ugh, please no.

Believe me, fuse is doing _nothing_ out of the ordinary, and should
not need special treatment during suspend/resume.  If suspend itself
is doing something that triggers fuse activity, then that's a bug,
such as the sync() thing that started this thread.

> Then, they may react to such a notification by entering a "suspend" mode
> of operation in which they will return errors from some callbacks that
> otherwise should have succeeded etc.  That depends on the subsystem in
> question.

Sounds horrible.

Why do we need to deal with subsystem interdependencies during
suspend?  Isn't it about saving device state to ram?  That definitely
_should not_ need to trigger anything that touches filesystems or
other subsystems.

Miklos

Re: Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Rafael J. Wysocki]

On Tuesday, 3 July 2007 19:38, Miklos Szeredi wrote:
> > > Indeed. Actually, one could argue that it's impossible to solve the
> > > problem as long as we try to call out to userspace during suspend and
> > > need to wait until that's finished, like in the case of sys_sync() and
> > > fuse filesystems, and probably other cases. Maybe we should make *those*
> > > calls return a failure so that the suspend isn't transparent inside the
> > > kernel but is transparent to userspace.
> > 
> > Well, it generally needs more consideration. :-)
> > 
> > I think that we should introduce mechanisms that will allow us to notify all
> > kernel subsystems, including FUSE and similar, that the system is going to
> > enter a sleep state (one of those is the notifier chain introduced recently).
> 
> Ugh, please no.
> 
> Believe me, fuse is doing _nothing_ out of the ordinary, and should
> not need special treatment during suspend/resume.  If suspend itself
> is doing something that triggers fuse activity, then that's a bug,
> such as the sync() thing that started this thread.

Apart from the sync, it shouldn't trigger any fs activity.  Still, some other
task running concurrently with the suspend code may do that and _if_
we are going to allow that to happen (and we do, if we remove the freezer
from the suspend code path), we will have to take that into consideration.

> > Then, they may react to such a notification by entering a "suspend" mode
> > of operation in which they will return errors from some callbacks that
> > otherwise should have succeeded etc.  That depends on the subsystem in
> > question.
> 
> Sounds horrible.
> 
> Why do we need to deal with subsystem interdependencies during
> suspend?  Isn't it about saving device state to ram?

No.  In addition, the devices should be prevented from generating interrupts
or initiating DMA transfers and put into low power states before we actually
suspend the system (platform).  This is a bit difficult to do while all user
space is running.

> That definitely _should not_ need to trigger anything that touches
> filesystems or other subsystems.

Right, but the subsystems may do something that affects devices.

Greetings,
Rafael


-- 
"Premature optimization is the root of all evil." - Donald Knuth

Re: Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Alan Stern]

On Tue, 3 Jul 2007, Johannes Berg wrote:

> > Runtime suspend isn't a problem.  Only STR.
> 
> Ah but for all those character devices people were saying are the
> problem we haven't even solved runtime suspend as far as I can tell from
> the discussion.

The technique used by USB for runtime PM should work fine with other
devices.  They may not have implemented it yet, but I consider the
matter more-or-less solved.


> > As you can see, this is a very difficult problem to solve.
> 
> Indeed. Actually, one could argue that it's impossible to solve the
> problem as long as we try to call out to userspace during suspend and
> need to wait until that's finished, like in the case of sys_sync() and
> fuse filesystems, and probably other cases. Maybe we should make *those*
> calls return a failure so that the suspend isn't transparent inside the
> kernel but is transparent to userspace.

I disagree.  The problem isn't the kernel calling userspace; it's
userspace trying to do I/O at a time when everything is supposed to be
quiescing.  Detecting that and blocking it in drivers is hard and
error-prone; preventing it by freezing userspace is easy and cheap.

The reasons why the PPC people dislike the whole idea aren't clear to
me.  If it were necessary to have some user task running in order to
carry out the STR then their objection would make sense -- obviously
that task couldn't do its job if it were frozen.  But it isn't
necessary, or at least it should not be.

Userspace will be effectively "frozen" while the system as a whole is 
suspended.  So what's wrong with freezing it a little early?  Despite 
Ben's comments, it seems to me that the freezer doesn't hide problems 
-- it prevents them.

Now people may claim that the freezer implementation itself is buggy.  
I wouldn't dispute it.  But the bugs should be fixable; nobody has 
pointed out anything fundamentally wrong with the idea AFAICT.

Alan Stern

Re: Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Paul Mackerras]

Alan Stern writes:

> I disagree.  The problem isn't the kernel calling userspace; it's
> userspace trying to do I/O at a time when everything is supposed to be
> quiescing.  Detecting that and blocking it in drivers is hard and
> error-prone; preventing it by freezing userspace is easy and cheap.

And unreliable, and prone to deadlocks, and invasive - requiring
changes to kernel threads that have nothing to do with drivers or
suspend/resume.

> The reasons why the PPC people dislike the whole idea aren't clear to
> me. 

Our experience is that it isn't necessary.  It's extra code that in
practice causes deadlocks and added maintenance burden for no
discernable benefit.

> If it were necessary to have some user task running in order to
> carry out the STR then their objection would make sense -- obviously
> that task couldn't do its job if it were frozen.  But it isn't
> necessary, or at least it should not be.

The freezer doesn't achieve its stated goal of preventing drivers from
getting I/O requests after suspend, since kernel threads can (and do)
initiate I/O.  So then we say that some kernel threads need to be
frozen and others don't, but making that decision is difficult and
error-prone.

Besides, any kernel thread that does I/O is potentially doing that in
order to complete some other I/O request.  So we want to freeze it in
order to prevent new I/O requests from being initiated, but we don't
want to freeze it so that existing I/O requests can be completed.
Thus we have a fundamental conflict in the notion of the freezer.

In fact I believe that making a distinction between user and kernel
threads is wrong and likely to lead to problems, since userspace can
be involved in doing I/O (e.g. FUSE or the user-space driver
framework).  So the argument of the previous paragraph also applies to
some userspace processes.

> Userspace will be effectively "frozen" while the system as a whole is 
> suspended.  So what's wrong with freezing it a little early?  Despite 
> Ben's comments, it seems to me that the freezer doesn't hide problems 
> -- it prevents them.

No, it appears to prevent them, but doesn't in fact.

I remain convinced that the right approach is to fix the drivers to do
one of two things; either do something in the suspend call to block
further requests to the device, or use a late-suspend call to put
their device into a low-power state.  Of course, correctly-written
frameworks can do a lot to help the chipset drivers here.

> Now people may claim that the freezer implementation itself is buggy.  
> I wouldn't dispute it.  But the bugs should be fixable; nobody has 
> pointed out anything fundamentally wrong with the idea AFAICT.

The fundamental problem is the kernel threads and user processes that
we need running to complete existing I/O requests, but which may
initiate new I/O requests in doing so.

The right way to solve the problem is to do the request blocking in
the drivers.

Paul.

Re: Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Alan Stern]

On Wed, 4 Jul 2007, Paul Mackerras wrote:

> Alan Stern writes:
> 
> > I disagree.  The problem isn't the kernel calling userspace; it's
> > userspace trying to do I/O at a time when everything is supposed to be
> > quiescing.  Detecting that and blocking it in drivers is hard and
> > error-prone; preventing it by freezing userspace is easy and cheap.
> 
> And unreliable, and prone to deadlocks, and invasive - requiring
> changes to kernel threads that have nothing to do with drivers or
> suspend/resume.

Let's agree the kernel threads and the freezer are a separate issue.  
In the most recent kernels, the freezer does not suspend kernel threads 
by default.

I agree the kernel threads which try to do I/O during a suspend will 
need extra attention.  However if these threads are necessary for the 
suspend procedure, then blocking them (which is how people on this 
thread have been saying driver should treat I/O requests during a 
suspend) will cause additional problems.  There's no way around it; 
these threads _will_ require more work.

> > The reasons why the PPC people dislike the whole idea aren't clear to
> > me. 
> 
> Our experience is that it isn't necessary.  It's extra code that in
> practice causes deadlocks and added maintenance burden for no
> discernable benefit.

I have discussed the benefits elsewhere.  As for the deadlocks -- do 
you still observe them if you use the version of the freezer which 
doesn't freeze kernel threads?

> The freezer doesn't achieve its stated goal of preventing drivers from
> getting I/O requests after suspend, since kernel threads can (and do)
> initiate I/O.  So then we say that some kernel threads need to be
> frozen and others don't, but making that decision is difficult and
> error-prone.

No -- we say that the kernel threads which generate I/O requests during 
suspend need to be changed.

> In fact I believe that making a distinction between user and kernel
> threads is wrong and likely to lead to problems, since userspace can
> be involved in doing I/O (e.g. FUSE or the user-space driver
> framework).  So the argument of the previous paragraph also applies to
> some userspace processes.

Userspace cannot do I/O directly on its own, apart from some
exceptional situations where a privileged task directly twiddles some
I/O ports or the equivalent.

There remains the problem of user tasks whose assistance is required to 
carry out some I/O (as with FUSE).  If the I/O can be deferred until 
after the resume, then there's no problem.  If the I/O can be carried 
out before the suspend, then it should be.  And finally, if the I/O 
must be done during the suspend, you're in real trouble -- how do you 
do I/O to a suspended device?

> I remain convinced that the right approach is to fix the drivers to do
> one of two things; either do something in the suspend call to block
> further requests to the device, or use a late-suspend call to put
> their device into a low-power state.  Of course, correctly-written
> frameworks can do a lot to help the chipset drivers here.

The first alternative is a possibility.  My argument all along has been 
that it is difficult and error-prone, and it adds more overhead to 
system operation (even when not suspending!) than simply freezing 
userspace.

Alan Stern

Re: Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Paul Mackerras]

Alan Stern writes:

> Let's agree the kernel threads and the freezer are a separate issue.  

No, I don't think they are a separate issue, because I think the
distinction the freezer makes between kernel threads and user threads
is a false and misleading distinction.

> In the most recent kernels, the freezer does not suspend kernel threads 
> by default.

And therefore doesn't guarantee that drivers won't get I/O requests
after being suspended, as far as I can see...

> I agree the kernel threads which try to do I/O during a suspend will 
> need extra attention.  However if these threads are necessary for the 
> suspend procedure, then blocking them (which is how people on this 
> thread have been saying driver should treat I/O requests during a 
> suspend) will cause additional problems.  There's no way around it; 
> these threads _will_ require more work.

There is a way around it; do the request blocking in the drivers,
where it belongs.

> > > The reasons why the PPC people dislike the whole idea aren't clear to
> > > me. 
> > 
> > Our experience is that it isn't necessary.  It's extra code that in
> > practice causes deadlocks and added maintenance burden for no
> > discernable benefit.
> 
> I have discussed the benefits elsewhere.  As for the deadlocks -- do 
> you still observe them if you use the version of the freezer which 
> doesn't freeze kernel threads?

In general the only way to guarantee there are no deadlocks is to
construct the graph of dependencies between tasks.  Those dependencies
are not in practice observable from outside the tasks, so it is
virtually impossible to construct the graph.

The "don't freeze kernel threads" thing is an attempt to make a crude
approximation to the dependency graph (by saying kernel threads only
depend on other kernel threads), but the approximation breaks down
when you have FUSE or user-level device drivers.

> Userspace cannot do I/O directly on its own, apart from some
> exceptional situations where a privileged task directly twiddles some
> I/O ports or the equivalent.

Userspace can be involved in servicing I/O requests; not just FUSE,
but also user-level nfsd and user-level PPP demonstrate that.

> There remains the problem of user tasks whose assistance is required to 
> carry out some I/O (as with FUSE).  If the I/O can be deferred until 
> after the resume, then there's no problem.  If the I/O can be carried 
> out before the suspend, then it should be.  And finally, if the I/O 
> must be done during the suspend, you're in real trouble -- how do you 
> do I/O to a suspended device?

So why doesn't that argument apply to kernel threads? :)

> > I remain convinced that the right approach is to fix the drivers to do
> > one of two things; either do something in the suspend call to block
> > further requests to the device, or use a late-suspend call to put
> > their device into a low-power state.  Of course, correctly-written
> > frameworks can do a lot to help the chipset drivers here.
> 
> The first alternative is a possibility.  My argument all along has been 
> that it is difficult and error-prone, and it adds more overhead to 
> system operation (even when not suspending!) than simply freezing 
> userspace.

It does actually provably solve the problem though, which is more than
the freezer does.

Paul.

Re: Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Oliver Neukum]

Am Donnerstag, 5. Juli 2007 schrieb Paul Mackerras:
> > I have discussed the benefits elsewhere.  As for the deadlocks -- do 
> > you still observe them if you use the version of the freezer which 
> > doesn't freeze kernel threads?
> 
> In general the only way to guarantee there are no deadlocks is to
> construct the graph of dependencies between tasks.  Those dependencies
> are not in practice observable from outside the tasks, so it is
> virtually impossible to construct the graph.

In which way can user space tasks depend on each other in a way that
allows a them members of that cycle to be in uninterruptible sleep?

	Regards
		Oliver

Re: Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Miklos Szeredi]

> > > I have discussed the benefits elsewhere.  As for the deadlocks -- do 
> > > you still observe them if you use the version of the freezer which 
> > > doesn't freeze kernel threads?
> > 
> > In general the only way to guarantee there are no deadlocks is to
> > construct the graph of dependencies between tasks.  Those dependencies
> > are not in practice observable from outside the tasks, so it is
> > virtually impossible to construct the graph.
> 
> In which way can user space tasks depend on each other in a way that
> allows a them members of that cycle to be in uninterruptible sleep?

 - process A calls rename() on a fuse fs
 - process B, the fuse server, starts to process the rename request
 - process B is frozen before it can reply

Now process A is unfreezable.  We cannot make rename() restartable,
hence it cannot be interruptible.

Miklos

Re: Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Oliver Neukum]

Am Donnerstag, 5. Juli 2007 schrieb Miklos Szeredi:
> > > > I have discussed the benefits elsewhere.  As for the deadlocks -- do 
> > > > you still observe them if you use the version of the freezer which 
> > > > doesn't freeze kernel threads?
> > > 
> > > In general the only way to guarantee there are no deadlocks is to
> > > construct the graph of dependencies between tasks.  Those dependencies
> > > are not in practice observable from outside the tasks, so it is
> > > virtually impossible to construct the graph.
> > 
> > In which way can user space tasks depend on each other in a way that
> > allows a them members of that cycle to be in uninterruptible sleep?
> 
>  - process A calls rename() on a fuse fs
>  - process B, the fuse server, starts to process the rename request
>  - process B is frozen before it can reply
> 
> Now process A is unfreezable.  We cannot make rename() restartable,
> hence it cannot be interruptible.

Then this is a problem specific to fuse. You should teach fuse to block
suspension while such operations are being performed.

	Regards
		Oliver

Re: Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Miklos Szeredi]

> > > > > I have discussed the benefits elsewhere.  As for the deadlocks -- do 
> > > > > you still observe them if you use the version of the freezer which 
> > > > > doesn't freeze kernel threads?
> > > > 
> > > > In general the only way to guarantee there are no deadlocks is to
> > > > construct the graph of dependencies between tasks.  Those dependencies
> > > > are not in practice observable from outside the tasks, so it is
> > > > virtually impossible to construct the graph.
> > > 
> > > In which way can user space tasks depend on each other in a way that
> > > allows a them members of that cycle to be in uninterruptible sleep?
> > 
> >  - process A calls rename() on a fuse fs
> >  - process B, the fuse server, starts to process the rename request
> >  - process B is frozen before it can reply
> > 
> > Now process A is unfreezable.  We cannot make rename() restartable,
> > hence it cannot be interruptible.
> 
> Then this is a problem specific to fuse. You should teach fuse to block
> suspension while such operations are being performed.

And teach VFS to block suspension, while waiting on a mutex held by
another process performing a fuse operation.

I can already hear the beautiful praise from Al Viro at the sight of
that ;)

Miklos

Re: Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Oliver Neukum]

Am Donnerstag, 5. Juli 2007 schrieb Miklos Szeredi:
> > > > > > I have discussed the benefits elsewhere.  As for the deadlocks -- do 
> > > > > > you still observe them if you use the version of the freezer which 
> > > > > > doesn't freeze kernel threads?
> > > > > 
> > > > > In general the only way to guarantee there are no deadlocks is to
> > > > > construct the graph of dependencies between tasks.  Those dependencies
> > > > > are not in practice observable from outside the tasks, so it is
> > > > > virtually impossible to construct the graph.
> > > > 
> > > > In which way can user space tasks depend on each other in a way that
> > > > allows a them members of that cycle to be in uninterruptible sleep?
> > > 
> > >  - process A calls rename() on a fuse fs
> > >  - process B, the fuse server, starts to process the rename request
> > >  - process B is frozen before it can reply
> > > 
> > > Now process A is unfreezable.  We cannot make rename() restartable,
> > > hence it cannot be interruptible.
> > 
> > Then this is a problem specific to fuse. You should teach fuse to block
> > suspension while such operations are being performed.
> 
> And teach VFS to block suspension, while waiting on a mutex held by
> another process performing a fuse operation.
> 
> I can already hear the beautiful praise from Al Viro at the sight of
> that ;)

There is that.

OK, bite the bullet. Tasks involved in fuse are special. Give them a flag
and teach the freezer to put them on ice only after all other task are
frozen. In a way they are kernel, there's no use denying that.

	Regards
		Oliver

Re: Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Miklos Szeredi]

> > And teach VFS to block suspension, while waiting on a mutex held by
> > another process performing a fuse operation.
> > 
> > I can already hear the beautiful praise from Al Viro at the sight of
> > that ;)
> 
> There is that.
> 
> OK, bite the bullet. Tasks involved in fuse are special. Give them a flag
> and teach the freezer to put them on ice only after all other task are
> frozen. In a way they are kernel, there's no use denying that.

And flag every other process, that the flagged process is
communicating with?  How are you proposing to do that?

Quoting Paul:

"1. The freezer cannot be guaranteed deadlock-free without constructing
   a dependency graph between tasks (both user and kernel), which is
   virtually impossible since the dependencies are not externally
   observable."

Miklos

Re: Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Oliver Neukum]

Am Donnerstag, 5. Juli 2007 schrieb Miklos Szeredi:
> > > And teach VFS to block suspension, while waiting on a mutex held by
> > > another process performing a fuse operation.
> > > 
> > > I can already hear the beautiful praise from Al Viro at the sight of
> > > that ;)
> > 
> > There is that.
> > 
> > OK, bite the bullet. Tasks involved in fuse are special. Give them a flag
> > and teach the freezer to put them on ice only after all other task are
> > frozen. In a way they are kernel, there's no use denying that.
> 
> And flag every other process, that the flagged process is
> communicating with?  How are you proposing to do that?
> 
> Quoting Paul:
> 
> "1. The freezer cannot be guaranteed deadlock-free without constructing
>    a dependency graph between tasks (both user and kernel), which is
>    virtually impossible since the dependencies are not externally
>    observable."

A deadlock requires that the circular wait is uninterruptible. Normal IPC
isn't.

What are you doing in the userland portions of fuse? Some kind of IPC
with other tasks? There is a limit to which you can push kernel functionality
into user space.

	Regards
		Oliver

Re: Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Miklos Szeredi]

> > > > And teach VFS to block suspension, while waiting on a mutex held by
> > > > another process performing a fuse operation.
> > > > 
> > > > I can already hear the beautiful praise from Al Viro at the sight of
> > > > that ;)
> > > 
> > > There is that.
> > > 
> > > OK, bite the bullet. Tasks involved in fuse are special. Give them a flag
> > > and teach the freezer to put them on ice only after all other task are
> > > frozen. In a way they are kernel, there's no use denying that.
> > 
> > And flag every other process, that the flagged process is
> > communicating with?  How are you proposing to do that?
> > 
> > Quoting Paul:
> > 
> > "1. The freezer cannot be guaranteed deadlock-free without constructing
> >    a dependency graph between tasks (both user and kernel), which is
> >    virtually impossible since the dependencies are not externally
> >    observable."
> 
> A deadlock requires that the circular wait is uninterruptible. Normal IPC
> isn't.
> 
> What are you doing in the userland portions of fuse? Some kind of IPC
> with other tasks?

Anything, writing to a file, writing to shared memory, sending things
over the network.  There's no limit to what a filesystem daemon may
do.  It's a perfectly ordinary unprivileged userspace process.  And
this is a feature not a bug.

> There is a limit to which you can push kernel functionality into
> user space.

Limiting what a userspace filesystem can do would defeat the whole
purpose of the bloody thing.  This is not negotiable ;)

Miklos

Re: Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Rafael J. Wysocki]

On Thursday, 5 July 2007 12:14, Miklos Szeredi wrote:
> > > > > And teach VFS to block suspension, while waiting on a mutex held by
> > > > > another process performing a fuse operation.
> > > > > 
> > > > > I can already hear the beautiful praise from Al Viro at the sight of
> > > > > that ;)
> > > > 
> > > > There is that.
> > > > 
> > > > OK, bite the bullet. Tasks involved in fuse are special. Give them a flag
> > > > and teach the freezer to put them on ice only after all other task are
> > > > frozen. In a way they are kernel, there's no use denying that.
> > > 
> > > And flag every other process, that the flagged process is
> > > communicating with?  How are you proposing to do that?
> > > 
> > > Quoting Paul:
> > > 
> > > "1. The freezer cannot be guaranteed deadlock-free without constructing
> > >    a dependency graph between tasks (both user and kernel), which is
> > >    virtually impossible since the dependencies are not externally
> > >    observable."

This statement is ganarally false.

There is the limitation in the freezer that it cannot handle uninterruptible
tasks and that's all.

Now, this is not usual for user space tasks to make other user space
tasks become uninterruptible.  I'd say this is a little strange.

> > A deadlock requires that the circular wait is uninterruptible. Normal IPC
> > isn't.
> > 
> > What are you doing in the userland portions of fuse? Some kind of IPC
> > with other tasks?
> 
> Anything, writing to a file, writing to shared memory, sending things
> over the network.  There's no limit to what a filesystem daemon may
> do.  It's a perfectly ordinary unprivileged userspace process.  And
> this is a feature not a bug.
> 
> > There is a limit to which you can push kernel functionality into
> > user space.
> 
> Limiting what a userspace filesystem can do would defeat the whole
> purpose of the bloody thing.  This is not negotiable ;)

Which doesn't change the fact that FUSE _is_ special, because it adds
dependencies between processed that were not present before.

Greetings,
Rafael


-- 
"Premature optimization is the root of all evil." - Donald Knuth

Re: Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Miklos Szeredi]

> > Limiting what a userspace filesystem can do would defeat the whole
> > purpose of the bloody thing.  This is not negotiable ;)
> 
> Which doesn't change the fact that FUSE _is_ special, because it adds
> dependencies between processed that were not present before.

OK, fuse is special.  So is the userspace driver framework (UIO)
proposed by Greg KH and co.  Now what can be done about these?

 - making them not-special is not an option due to the established
   interfaces, which don't allow restartability.

 - fixing the freezer is pretty much impossible because the
   dependencies between the tasks cannot be known.

 - removing the freezer and fixing the drivers seems workable, we
   already have a prototype in the form of the powermac architecture.

It seems pretty clear cut.  Whining about how much problems this will
cause won't get us nearer to a solution.

Miklos

Re: Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Rafael J. Wysocki]

On Thursday, 5 July 2007 13:54, Miklos Szeredi wrote:
> > > Limiting what a userspace filesystem can do would defeat the whole
> > > purpose of the bloody thing.  This is not negotiable ;)
> > 
> > Which doesn't change the fact that FUSE _is_ special, because it adds
> > dependencies between processed that were not present before.
> 
> OK, fuse is special.  So is the userspace driver framework (UIO)
> proposed by Greg KH and co.  Now what can be done about these?
> 
>  - making them not-special is not an option due to the established
>    interfaces, which don't allow restartability.
> 
>  - fixing the freezer is pretty much impossible because the
>    dependencies between the tasks cannot be known.
> 
>  - removing the freezer and fixing the drivers seems workable, we
>    already have a prototype in the form of the powermac architecture.
> 
> It seems pretty clear cut.  Whining about how much problems this will
> cause won't get us nearer to a solution.

Yes, that's pretty clear cut, but we should start from fixing the drivers. :-)

Greetings,
Rafael


-- 
"Premature optimization is the root of all evil." - Donald Knuth

Re: Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Oliver Neukum]

Am Donnerstag, 5. Juli 2007 schrieb Rafael J. Wysocki:
> > It seems pretty clear cut.  Whining about how much problems this will
> > cause won't get us nearer to a solution.
> 
> Yes, that's pretty clear cut, but we should start from fixing the drivers. :-)

If, at a minimum, we can determine that we can STD without a freezer.
It makes no sense to invest a lot of work to face the same problem
again with STD.

	Regards
		Oliver

Re: Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Matthew Garrett]

On Thu, Jul 05, 2007 at 03:28:33PM +0200, Oliver Neukum wrote:

> If, at a minimum, we can determine that we can STD without a freezer.
> It makes no sense to invest a lot of work to face the same problem
> again with STD.

I have a model for STD that avoids the need to freeze the entirity of 
userspace, but I need to find some more time to flesh it out.

-- 
Matthew Garrett | mjg59@srcf.ucam.org

Re: Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Rafael J. Wysocki]

On Thursday, 5 July 2007 15:28, Oliver Neukum wrote:
> Am Donnerstag, 5. Juli 2007 schrieb Rafael J. Wysocki:
> > > It seems pretty clear cut.  Whining about how much problems this will
> > > cause won't get us nearer to a solution.
> > 
> > Yes, that's pretty clear cut, but we should start from fixing the drivers. :-)
> 
> If, at a minimum, we can determine that we can STD without a freezer.

No, we can't.

> It makes no sense to invest a lot of work to face the same problem
> again with STD.

Arguably, it does make sense, because for many platforms the hibernation
is irrelevant and we're going to separate the suspend and hibernation
frameworks anyway.

Greetings,
Rafael


-- 
"Premature optimization is the root of all evil." - Donald Knuth

Re: Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Rafael J. Wysocki]

On Thursday, 5 July 2007 15:46, Matthew Garrett wrote:
> On Thu, Jul 05, 2007 at 03:28:33PM +0200, Oliver Neukum wrote:
> 
> > If, at a minimum, we can determine that we can STD without a freezer.
> > It makes no sense to invest a lot of work to face the same problem
> > again with STD.
> 
> I have a model for STD that avoids the need to freeze the entirity of 
> userspace, but I need to find some more time to flesh it out.

You can just describe it, as far as I'm concerned. :-)

Greetings,
Rafael


-- 
"Premature optimization is the root of all evil." - Donald Knuth

Re: Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Matthew Garrett]

On Thu, Jul 05, 2007 at 04:09:24PM +0200, Rafael J. Wysocki wrote:
> On Thursday, 5 July 2007 15:46, Matthew Garrett wrote:
> > I have a model for STD that avoids the need to freeze the entirity of 
> > userspace, but I need to find some more time to flesh it out.
> 
> You can just describe it, as far as I'm concerned. :-)

The basic model is that nobody's really described a use-case where we 
actually care about restoring system state. What people want is to be 
able to restore application state. So, arguably, what we want isn't to 
save the entire kernel state and application state in one go because we 
can reconstruct a huge amount of that afterwards.

This isn't too much of a problem. All we actually need to be able to do 
is to atomically dump process state (which requires the freezer, but 
doesn't require freezing the entire system), shut down, get the system 
back into approximately the correct state (remount filesystems, start X, 
whatever) and then restore the processes.

Now, obviously, there's actually quite a lot of complexity here that I'm 
neatly eliding :) The biggest issue is restoring hardware state. We'd 
require quite a different model to the existing one, but I think there 
are arguments there for it being helpful anywy. Keeping state in the 
midlevels rather than the low-level drivers would give us much more 
ability to deal with hardware issues, and potentially allow the 
replacement of faulty hardware without userspace caring (freeze your 
mission-critical application, hotplug the network card, let the kernel 
restore state and resume it)

There's other advantages to this. As long as the kernel hasn't changed 
too much it would be possible to restore userspace across kernel 
security upgrades. You end up saving less to disk so performance should 
be better. Touching filesystems between suspend and resume doesn't 
result in the entire world ending.

I've mocked up a basic implementation using cryopid, but it's somewhat 
limited by the lack of support for sockets. I'd like to move more of the 
smarts into the kernel (Hurray, checkpointing!) and then see how much 
hardware support ends up horifically broken.
-- 
Matthew Garrett | mjg59@srcf.ucam.org

Re: Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Ray Lee]

On 7/5/07, Matthew Garrett <mjg59@srcf.ucam.org> wrote:
> On Thu, Jul 05, 2007 at 04:09:24PM +0200, Rafael J. Wysocki wrote:
> > On Thursday, 5 July 2007 15:46, Matthew Garrett wrote:
> > > I have a model for STD that avoids the need to freeze the entirity of
> > > userspace, but I need to find some more time to flesh it out.
> >
> > You can just describe it, as far as I'm concerned. :-)
>
> The basic model is that nobody's really described a use-case where we
> actually care about restoring system state. What people want is to be
> able to restore application state.

Hmm, careful. There are a bunch of people who use suspend2 exactly
because it saves and restores the page cache, leaving the system in a
usable state without waiting for the universe to swap back in from
disk. It makes a big difference on older laptops with slow drives.
While the other advantages you list for process cryogenics are pretty
neat, let's remember that the 99% use case for STD is laptops.

Re: Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Rafael J. Wysocki]

On Thursday, 5 July 2007 16:23, Matthew Garrett wrote:
> On Thu, Jul 05, 2007 at 04:09:24PM +0200, Rafael J. Wysocki wrote:
> > On Thursday, 5 July 2007 15:46, Matthew Garrett wrote:
> > > I have a model for STD that avoids the need to freeze the entirity of 
> > > userspace, but I need to find some more time to flesh it out.
> > 
> > You can just describe it, as far as I'm concerned. :-)
> 
> The basic model is that nobody's really described a use-case where we 
> actually care about restoring system state. What people want is to be 
> able to restore application state. So, arguably, what we want isn't to 
> save the entire kernel state and application state in one go because we 
> can reconstruct a huge amount of that afterwards.

Hmm, I think that will take more time than just restoring the entire system
state.

> This isn't too much of a problem. All we actually need to be able to do 
> is to atomically dump process state (which requires the freezer, but 
> doesn't require freezing the entire system),

Once you've frozen processes, the freezing of the rest of the system is
pretty straightforward.  Currently, we do a bit too much for that, because we
suspend devices instead of just quiescing them before creating the image,
bu that's going to change (I hope).

> shut down, get the system back into approximately the correct state (remount
> filesystems, start X, whatever) and then restore the processes.
>
> Now, obviously, there's actually quite a lot of complexity here that I'm 
> neatly eliding :) The biggest issue is restoring hardware state. We'd 
> require quite a different model to the existing one, but I think there 
> are arguments there for it being helpful anywy. Keeping state in the 
> midlevels rather than the low-level drivers would give us much more 
> ability to deal with hardware issues, and potentially allow the 
> replacement of faulty hardware without userspace caring (freeze your 
> mission-critical application, hotplug the network card, let the kernel 
> restore state and resume it)

Sounds neat, but what about the processes that depend on the hardware
(like hal)?

> There's other advantages to this. As long as the kernel hasn't changed 
> too much it would be possible to restore userspace across kernel 
> security upgrades. You end up saving less to disk so performance should 
> be better.

Well, not that much less. :-)

> Touching filesystems between suspend and resume doesn't result in the entire
> world ending.

Yes, that would be an advantege.

> I've mocked up a basic implementation using cryopid, but it's somewhat 
> limited by the lack of support for sockets. I'd like to move more of the 
> smarts into the kernel (Hurray, checkpointing!) and then see how much 
> hardware support ends up horifically broken.

There's one more thing, I'm not sure if that's possible to separate the kernel
state from the processes state entirely (think shared memory, LRU lists,
situations in which the application has been frozen while waiting for an
event in the kernel space etc.).

Greetings,
Rafael


-- 
"Premature optimization is the root of all evil." - Donald Knuth

Re: Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Matthew Garrett]

On Thu, Jul 05, 2007 at 07:46:01AM -0700, Ray Lee wrote:

> Hmm, careful. There are a bunch of people who use suspend2 exactly
> because it saves and restores the page cache, leaving the system in a
> usable state without waiting for the universe to swap back in from
> disk. It makes a big difference on older laptops with slow drives.
> While the other advantages you list for process cryogenics are pretty
> neat, let's remember that the 99% use case for STD is laptops.

Saving the processes means that you're implicitly saving the interesting 
chunks of the page cache. Removing the need for the atomic copy saves 
you from pushing a pile of stuff out to swap in the first place.

-- 
Matthew Garrett | mjg59@srcf.ucam.org

Re: Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Jeremy Maitin-Shepard]

Matthew Garrett <mjg59@srcf.ucam.org> writes:

> On Thu, Jul 05, 2007 at 04:09:24PM +0200, Rafael J. Wysocki wrote:
>> On Thursday, 5 July 2007 15:46, Matthew Garrett wrote:
>> > I have a model for STD that avoids the need to freeze the entirity of 
>> > userspace, but I need to find some more time to flesh it out.
>> 
>> You can just describe it, as far as I'm concerned. :-)

[snip: new hibernate idea]

I think my kexec-based hibernate idea is simpler and more feasible than
this approach, and also avoids the freezer.

-- 
Jeremy Maitin-Shepard

Re: Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Daniel Pittman]

Matthew Garrett <mjg59@srcf.ucam.org> writes:
> On Thu, Jul 05, 2007 at 04:09:24PM +0200, Rafael J. Wysocki wrote:
>> On Thursday, 5 July 2007 15:46, Matthew Garrett wrote:
>> > I have a model for STD that avoids the need to freeze the entirity of 
>> > userspace, but I need to find some more time to flesh it out.
>> 
>> You can just describe it, as far as I'm concerned. :-)
>
> The basic model is that nobody's really described a use-case where we 
> actually care about restoring system state. What people want is to be 
> able to restore application state. So, arguably, what we want isn't to 
> save the entire kernel state and application state in one go because we 
> can reconstruct a huge amount of that afterwards.

[...]

> I've mocked up a basic implementation using cryopid, but it's somewhat
> limited by the lack of support for sockets. I'd like to move more of
> the smarts into the kernel (Hurray, checkpointing!) and then see how
> much hardware support ends up horifically broken.

You might want to look at the checkpoint / migration support in the
OpenVZ kernel in relation to this.  That does work to dump the state of
a running "virtual environment" complete with applications to disk, move
it to another running kernel and restore the content.

That might, perhaps, help with the prototype of this?

Regards,
	Daniel

Re: Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Benjamin Herrenschmidt]

> There is that.
> 
> OK, bite the bullet. Tasks involved in fuse are special. Give them a flag
> and teach the freezer to put them on ice only after all other task are
> frozen. In a way they are kernel, there's no use denying that.

Yet another ugly hack to work around the fact that the freezer cannot
work reliably ... yuck

Why not spend that energy fixing drivers to properly block requests
instead ?

Ben.

Re: Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Rafael J. Wysocki]

On Friday, 6 July 2007 00:38, Benjamin Herrenschmidt wrote:
> 
> > There is that.
> > 
> > OK, bite the bullet. Tasks involved in fuse are special. Give them a flag
> > and teach the freezer to put them on ice only after all other task are
> > frozen. In a way they are kernel, there's no use denying that.
> 
> Yet another ugly hack to work around the fact that the freezer cannot
> work reliably ... yuck
> 
> Why not spend that energy fixing drivers to properly block requests
> instead ?

Because it's more difficult? :-)

Greetings,
Rafael


-- 
"Premature optimization is the root of all evil." - Donald Knuth

Re: Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Oliver Neukum]

Am Freitag, 6. Juli 2007 schrieb Benjamin Herrenschmidt:
> 
> > There is that.
> > 
> > OK, bite the bullet. Tasks involved in fuse are special. Give them a flag
> > and teach the freezer to put them on ice only after all other task are
> > frozen. In a way they are kernel, there's no use denying that.
> 
> Yet another ugly hack to work around the fact that the freezer cannot
> work reliably ... yuck
> 
> Why not spend that energy fixing drivers to properly block requests
> instead ?

Because we will be unable to escape that job. Let's assume that
we remove the freezer from the STR path. The next complaint would
be that we cannot do STD with fuse. "Then don't do that" would not
be taken kindly as answer.

So we would be under pressure to remove the freezer from STD, too,
or find a way to make the freezer work. If we'd have to make the freezer
work anyway, we can do it right now.

	Regards
		Oliver

Re: Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Pavel Machek]

On Thu 2007-07-05 10:17:17, Miklos Szeredi wrote:
> > > > I have discussed the benefits elsewhere.  As for the deadlocks -- do 
> > > > you still observe them if you use the version of the freezer which 
> > > > doesn't freeze kernel threads?
> > > 
> > > In general the only way to guarantee there are no deadlocks is to
> > > construct the graph of dependencies between tasks.  Those dependencies
> > > are not in practice observable from outside the tasks, so it is
> > > virtually impossible to construct the graph.
> > 
> > In which way can user space tasks depend on each other in a way that
> > allows a them members of that cycle to be in uninterruptible sleep?
> 
>  - process A calls rename() on a fuse fs
>  - process B, the fuse server, starts to process the rename request
>  - process B is frozen before it can reply
> 
> Now process A is unfreezable.  We cannot make rename() restartable,
> hence it cannot be interruptible.

Yes, we are claiming fuse is very special in this regard, and perhaps
even broken.

Let's see. If I SIGSTOP the fuse server, I can get unrelated tasks
unkillable (even for SIGKILL!) forever. That's very special, and maybe
even a FUSE bug. And that is also what makes FUSE special
w.r.t. s2ram.

So no, you can't claim "FUSE is just IPC". It is very special IPC.
								Pavel
-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

Re: Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Miklos Szeredi]

> > > > > I have discussed the benefits elsewhere.  As for the deadlocks -- do 
> > > > > you still observe them if you use the version of the freezer which 
> > > > > doesn't freeze kernel threads?
> > > > 
> > > > In general the only way to guarantee there are no deadlocks is to
> > > > construct the graph of dependencies between tasks.  Those dependencies
> > > > are not in practice observable from outside the tasks, so it is
> > > > virtually impossible to construct the graph.
> > > 
> > > In which way can user space tasks depend on each other in a way that
> > > allows a them members of that cycle to be in uninterruptible sleep?
> > 
> >  - process A calls rename() on a fuse fs
> >  - process B, the fuse server, starts to process the rename request
> >  - process B is frozen before it can reply
> > 
> > Now process A is unfreezable.  We cannot make rename() restartable,
> > hence it cannot be interruptible.
> 
> Yes, we are claiming fuse is very special in this regard, and perhaps
> even broken.
> 
> Let's see. If I SIGSTOP the fuse server, I can get unrelated tasks
> unkillable (even for SIGKILL!) forever.

Actually fuse allows SIGKILL, because it's always fatal, and the
syscall may not be restarted.

> That's very special, and maybe even a FUSE bug. And that is also
> what makes FUSE special w.r.t. s2ram.

What makes fuse special is that some file operations are synchronous
and non-restartable.  That's just how the UNIX filesystem API works
and is hardly a bug in fuse.

> So no, you can't claim "FUSE is just IPC". It is very special IPC.

I did say it's special.  Sure, it has some "interesting" properties,
and with a bit of malice you can do very ugly things with it.  If you
are interested, read Documentation/filesystems/fuse.txt, especially
the "Tricky deadlock" section ;)

Miklos

Re: Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Pavel Machek]

Hi!

> > > > In which way can user space tasks depend on each other in a way that
> > > > allows a them members of that cycle to be in uninterruptible sleep?
> > > 
> > >  - process A calls rename() on a fuse fs
> > >  - process B, the fuse server, starts to process the rename request
> > >  - process B is frozen before it can reply
> > > 
> > > Now process A is unfreezable.  We cannot make rename() restartable,
> > > hence it cannot be interruptible.
> > 
> > Yes, we are claiming fuse is very special in this regard, and perhaps
> > even broken.
> > 
> > Let's see. If I SIGSTOP the fuse server, I can get unrelated tasks
> > unkillable (even for SIGKILL!) forever.
> 
> Actually fuse allows SIGKILL, because it's always fatal, and the
> syscall may not be restarted.

I think you want to stick try_to_freeze() at the same places where you
do SIGKILL handling. That should solve the 'syslogd is unfreezeable'
problem.

Plus, it would be nice to find out where suspend/hibernation is
triggering fuse activity. We can then decide where to fix it -- in
fuse or in suspend parts. You said sys_sync is not implemented... so
where is the problem?

> > That's very special, and maybe even a FUSE bug. And that is also
> > what makes FUSE special w.r.t. s2ram.
> 
> What makes fuse special is that some file operations are synchronous
> and non-restartable.  That's just how the UNIX filesystem API works
> and is hardly a bug in fuse.

Well, unix is not plan9, and maybe userland filesystems are impossible
in unix. But that is hardly a bug in unix :-).

							Pavel

-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

Re: Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Rafael J. Wysocki]

On Thursday, 5 July 2007 11:31, Miklos Szeredi wrote:
> > > > > > I have discussed the benefits elsewhere.  As for the deadlocks -- do 
> > > > > > you still observe them if you use the version of the freezer which 
> > > > > > doesn't freeze kernel threads?
> > > > > 
> > > > > In general the only way to guarantee there are no deadlocks is to
> > > > > construct the graph of dependencies between tasks.  Those dependencies
> > > > > are not in practice observable from outside the tasks, so it is
> > > > > virtually impossible to construct the graph.
> > > > 
> > > > In which way can user space tasks depend on each other in a way that
> > > > allows a them members of that cycle to be in uninterruptible sleep?
> > > 
> > >  - process A calls rename() on a fuse fs
> > >  - process B, the fuse server, starts to process the rename request
> > >  - process B is frozen before it can reply
> > > 
> > > Now process A is unfreezable.  We cannot make rename() restartable,
> > > hence it cannot be interruptible.
> > 
> > Yes, we are claiming fuse is very special in this regard, and perhaps
> > even broken.
> > 
> > Let's see. If I SIGSTOP the fuse server, I can get unrelated tasks
> > unkillable (even for SIGKILL!) forever.
> 
> Actually fuse allows SIGKILL, because it's always fatal, and the
> syscall may not be restarted.
> 
> > That's very special, and maybe even a FUSE bug. And that is also
> > what makes FUSE special w.r.t. s2ram.
> 
> What makes fuse special is that some file operations are synchronous
> and non-restartable.  That's just how the UNIX filesystem API works
> and is hardly a bug in fuse.
> 
> > So no, you can't claim "FUSE is just IPC". It is very special IPC.
> 
> I did say it's special.  Sure, it has some "interesting" properties,
> and with a bit of malice you can do very ugly things with it.  If you
> are interested, read Documentation/filesystems/fuse.txt, especially
> the "Tricky deadlock" section ;)

Very well.

Don't you think, however, that it can be modified a little to play well,
for example, with the freezer?

Greetings,
Rafael


-- 
"Premature optimization is the root of all evil." - Donald Knuth

Re: Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Miklos Szeredi]

> > Actually fuse allows SIGKILL, because it's always fatal, and the
> > syscall may not be restarted.
> 
> I think you want to stick try_to_freeze() at the same places where you
> do SIGKILL handling. That should solve the 'syslogd is unfreezeable'
> problem.

I could, but it would not solve the general problem.  Namely, that the
presence of fuse imposes a certain ordering in which userspace tasks
have to be frozen.  And it is not possible to know this ordering.

And even if the ordering were solved, the freezer would still not work
if the filesystem is not responding due to external events, such as a
lost network (this affects NFS, CIFS, whatever just the same as fuse).

> Plus, it would be nice to find out where suspend/hibernation is
> triggering fuse activity. We can then decide where to fix it -- in
> fuse or in suspend parts. You said sys_sync is not implemented... so
> where is the problem?

I cannot say without having a sysrq-t of the situation.

> > > That's very special, and maybe even a FUSE bug. And that is also
> > > what makes FUSE special w.r.t. s2ram.
> > 
> > What makes fuse special is that some file operations are synchronous
> > and non-restartable.  That's just how the UNIX filesystem API works
> > and is hardly a bug in fuse.
> 
> Well, unix is not plan9, and maybe userland filesystems are impossible
> in unix. But that is hardly a bug in unix :-).

I'd rather say, reliable suspend to ram is impossible in the presense
of userspace filesystems, iff people are too lazy to fix the suspend
framework and the drivers to work without the freezer.

This has nothing to do with unix, if plan9 would need to support STR
or hibernate, it would face exactly the same problems.

Miklos

Re: Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Rafael J. Wysocki]

On Thursday, 5 July 2007 14:07, Miklos Szeredi wrote:
> > > Actually fuse allows SIGKILL, because it's always fatal, and the
> > > syscall may not be restarted.
> > 
> > I think you want to stick try_to_freeze() at the same places where you
> > do SIGKILL handling. That should solve the 'syslogd is unfreezeable'
> > problem.
> 
> I could, but it would not solve the general problem.  Namely, that the
> presence of fuse imposes a certain ordering in which userspace tasks
> have to be frozen.  And it is not possible to know this ordering.
> 
> And even if the ordering were solved, the freezer would still not work
> if the filesystem is not responding due to external events, such as a
> lost network (this affects NFS, CIFS, whatever just the same as fuse).
> 
> > Plus, it would be nice to find out where suspend/hibernation is
> > triggering fuse activity. We can then decide where to fix it -- in
> > fuse or in suspend parts. You said sys_sync is not implemented... so
> > where is the problem?
> 
> I cannot say without having a sysrq-t of the situation.
> 
> > > > That's very special, and maybe even a FUSE bug. And that is also
> > > > what makes FUSE special w.r.t. s2ram.
> > > 
> > > What makes fuse special is that some file operations are synchronous
> > > and non-restartable.  That's just how the UNIX filesystem API works
> > > and is hardly a bug in fuse.
> > 
> > Well, unix is not plan9, and maybe userland filesystems are impossible
> > in unix. But that is hardly a bug in unix :-).
> 
> I'd rather say, reliable suspend to ram is impossible in the presense
> of userspace filesystems, iff people are too lazy to fix the suspend
> framework and the drivers to work without the freezer.

Well, I don't think it has anything to do with laziness or things like that.
Rather, people have limited time and this requires some knowledge about
drivers you're modifying, so not many people really can do that.

Also, you've already assumed that there's no other solution, but I'm not
convinced about that yet.

Greetings,
Rafael


-- 
"Premature optimization is the root of all evil." - Donald Knuth

Re: Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Oliver Neukum]

Am Donnerstag, 5. Juli 2007 schrieb Miklos Szeredi:
> > > Actually fuse allows SIGKILL, because it's always fatal, and the
> > > syscall may not be restarted.
> > 
> > I think you want to stick try_to_freeze() at the same places where you
> > do SIGKILL handling. That should solve the 'syslogd is unfreezeable'
> > problem.
> 
> I could, but it would not solve the general problem.  Namely, that the
> presence of fuse imposes a certain ordering in which userspace tasks
> have to be frozen.  And it is not possible to know this ordering.

Actually, why do you need this? There is no absolute need that you
finish the request. You must either finish the request or let yourself
be frozen.

A quick look through fuse reveals principally request_wait_answer()
And maybe a few other places. Is there some hidden reason you cannot
handle being frozen here?

	Regards
		Oliver

Re: Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Miklos Szeredi]

> Am Donnerstag, 5. Juli 2007 schrieb Miklos Szeredi:
> > > > Actually fuse allows SIGKILL, because it's always fatal, and the
> > > > syscall may not be restarted.
> > > 
> > > I think you want to stick try_to_freeze() at the same places where you
> > > do SIGKILL handling. That should solve the 'syslogd is unfreezeable'
> > > problem.
> > 
> > I could, but it would not solve the general problem.  Namely, that the
> > presence of fuse imposes a certain ordering in which userspace tasks
> > have to be frozen.  And it is not possible to know this ordering.
> 
> Actually, why do you need this? There is no absolute need that you
> finish the request. You must either finish the request or let yourself
> be frozen.
> 
> A quick look through fuse reveals principally request_wait_answer()
> And maybe a few other places. Is there some hidden reason you cannot
> handle being frozen here?

Yes, fuse could handle being frozen there.  However that would only
solve part of the problem: an operation waiting for a reply could be
holding a VFS mutex and some other task may be blocked on that mutex.

How would you solve freezing those tasks?

Miklos

Re: Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Rafael J. Wysocki]

On Thursday, 5 July 2007 21:44, Miklos Szeredi wrote:
> > Am Donnerstag, 5. Juli 2007 schrieb Miklos Szeredi:
> > > > > Actually fuse allows SIGKILL, because it's always fatal, and the
> > > > > syscall may not be restarted.
> > > > 
> > > > I think you want to stick try_to_freeze() at the same places where you
> > > > do SIGKILL handling. That should solve the 'syslogd is unfreezeable'
> > > > problem.
> > > 
> > > I could, but it would not solve the general problem.  Namely, that the
> > > presence of fuse imposes a certain ordering in which userspace tasks
> > > have to be frozen.  And it is not possible to know this ordering.
> > 
> > Actually, why do you need this? There is no absolute need that you
> > finish the request. You must either finish the request or let yourself
> > be frozen.
> > 
> > A quick look through fuse reveals principally request_wait_answer()
> > And maybe a few other places. Is there some hidden reason you cannot
> > handle being frozen here?
> 
> Yes, fuse could handle being frozen there.  However that would only
> solve part of the problem: an operation waiting for a reply could be
> holding a VFS mutex and some other task may be blocked on that mutex.
> 
> How would you solve freezing those tasks?

How probable is this situation?

Greetings,
Rafael


-- 
"Premature optimization is the root of all evil." - Donald Knuth

Re: Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Oliver Neukum]

Am Donnerstag, 5. Juli 2007 schrieb Miklos Szeredi:
> > Am Donnerstag, 5. Juli 2007 schrieb Miklos Szeredi:
> > > > > Actually fuse allows SIGKILL, because it's always fatal, and the
> > > > > syscall may not be restarted.
> > > > 
> > > > I think you want to stick try_to_freeze() at the same places where you
> > > > do SIGKILL handling. That should solve the 'syslogd is unfreezeable'
> > > > problem.
> > > 
> > > I could, but it would not solve the general problem.  Namely, that the
> > > presence of fuse imposes a certain ordering in which userspace tasks
> > > have to be frozen.  And it is not possible to know this ordering.
> > 
> > Actually, why do you need this? There is no absolute need that you
> > finish the request. You must either finish the request or let yourself
> > be frozen.
> > 
> > A quick look through fuse reveals principally request_wait_answer()
> > And maybe a few other places. Is there some hidden reason you cannot
> > handle being frozen here?
> 
> Yes, fuse could handle being frozen there.  However that would only
> solve part of the problem: an operation waiting for a reply could be
> holding a VFS mutex and some other task may be blocked on that mutex.
> 
> How would you solve freezing those tasks?

OK, you made me reach for literatur on theoretical computer science.

IMHO the range of actions a fuse server is inherently limited.
You must never ever block on a lock one of your clients is holding. In
this case the limitation is not influenced by the freezer.

The freezer introduces a further limitation in that the server can freeze
before the client, which must not be. You can prevent that by freezing
the servers last.

In principle you might have dependencies between servers and you won't
catch that, true. You won't catch servers blocking on IPC, but you are
balancing on the edge of deadlock with fuse anyway.

	Regards
		Oliver

Re: Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Miklos Szeredi]

> > Yes, fuse could handle being frozen there.  However that would only
> > solve part of the problem: an operation waiting for a reply could be
> > holding a VFS mutex and some other task may be blocked on that mutex.
> > 
> > How would you solve freezing those tasks?
> 
> OK, you made me reach for literatur on theoretical computer science.
> 
> IMHO the range of actions a fuse server is inherently limited.
> You must never ever block on a lock one of your clients is holding. In
> this case the limitation is not influenced by the freezer.

Obviously.  But I wasn't about the server trying to acquire a lock
held by a client.  I was talking about a client trying to acquire a
lock held by _another_ client.

If this coincides with the server (or some other task which the server
is depending on) being frozen before the clients, the freezer has a
problem.

> The freezer introduces a further limitation in that the server can freeze
> before the client, which must not be. You can prevent that by freezing
> the servers last.
> 
> In principle you might have dependencies between servers and you won't
> catch that, true. You won't catch servers blocking on IPC, but you are
> balancing on the edge of deadlock with fuse anyway.

Huh?

Miklos

Re: Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Oliver Neukum]

Am Donnerstag, 5. Juli 2007 schrieb Miklos Szeredi:
> > > Yes, fuse could handle being frozen there.  However that would only
> > > solve part of the problem: an operation waiting for a reply could be
> > > holding a VFS mutex and some other task may be blocked on that mutex.
> > > 
> > > How would you solve freezing those tasks?
> > 
> > OK, you made me reach for literatur on theoretical computer science.
> > 
> > IMHO the range of actions a fuse server is inherently limited.
> > You must never ever block on a lock one of your clients is holding. In
> > this case the limitation is not influenced by the freezer.
> 
> Obviously.  But I wasn't about the server trying to acquire a lock
> held by a client.  I was talking about a client trying to acquire a
> lock held by _another_ client.
> 
> If this coincides with the server (or some other task which the server
> is depending on) being frozen before the clients, the freezer has a
> problem.

True, but that case can only happen if servers are frozen before clients.
You don't need a full dependency graph. A simple set sequence of two
classes of tasks will do.

	Regards
		Oliver

Re: Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Oliver Neukum]

Am Donnerstag, 5. Juli 2007 schrieb Oliver Neukum:
> Am Donnerstag, 5. Juli 2007 schrieb Miklos Szeredi:
> > > > Yes, fuse could handle being frozen there.  However that would only
> > > > solve part of the problem: an operation waiting for a reply could be
> > > > holding a VFS mutex and some other task may be blocked on that mutex.
> > > > 
> > > > How would you solve freezing those tasks?
> > > 
> > > OK, you made me reach for literatur on theoretical computer science.
> > > 
> > > IMHO the range of actions a fuse server is inherently limited.
> > > You must never ever block on a lock one of your clients is holding. In
> > > this case the limitation is not influenced by the freezer.
> > 
> > Obviously.  But I wasn't about the server trying to acquire a lock
> > held by a client.  I was talking about a client trying to acquire a
> > lock held by _another_ client.
> > 
> > If this coincides with the server (or some other task which the server
> > is depending on) being frozen before the clients, the freezer has a
> > problem.
> 
> True, but that case can only happen if servers are frozen before clients.
> You don't need a full dependency graph. A simple set sequence of two
> classes of tasks will do.

Any replying to myself. A deadlock here is not fatal. You can and will
timeout in the freezer and can try again.

	REegards
		Oliver

Re: Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Alan Stern]

On Thu, 5 Jul 2007, Oliver Neukum wrote:

> > Obviously.  But I wasn't about the server trying to acquire a lock
> > held by a client.  I was talking about a client trying to acquire a
> > lock held by _another_ client.
> > 
> > If this coincides with the server (or some other task which the server
> > is depending on) being frozen before the clients, the freezer has a
> > problem.
> 
> True, but that case can only happen if servers are frozen before clients.
> You don't need a full dependency graph. A simple set sequence of two
> classes of tasks will do.

Just to make things more complicated...  Since a server isn't
restricted in what it can do, what happens when one server depends on
another server?

Alan Stern

Re: Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Miklos Szeredi]

> > > > Yes, fuse could handle being frozen there.  However that would only
> > > > solve part of the problem: an operation waiting for a reply could be
> > > > holding a VFS mutex and some other task may be blocked on that mutex.
> > > > 
> > > > How would you solve freezing those tasks?
> > > 
> > > OK, you made me reach for literatur on theoretical computer science.
> > > 
> > > IMHO the range of actions a fuse server is inherently limited.
> > > You must never ever block on a lock one of your clients is holding. In
> > > this case the limitation is not influenced by the freezer.
> > 
> > Obviously.  But I wasn't about the server trying to acquire a lock
> > held by a client.  I was talking about a client trying to acquire a
> > lock held by _another_ client.
> > 
> > If this coincides with the server (or some other task which the server
> > is depending on) being frozen before the clients, the freezer has a
> > problem.
> 
> True, but that case can only happen if servers are frozen before clients.
> You don't need a full dependency graph. A simple set sequence of two
> classes of tasks will do.

Umm, let's take sshfs, it has a separate "reply processing thread",
that reads replies from the sftp server and then wakes up the relevant
request thread.  The reply processing thread has no direct contact
with the fuse kernel module.  How is freezer supposed to know that
that task in fact also belongs to the server and needs to be kept from
freezing?

Given, it's in the same thread group, but using that is a rather weak
heuristic, as it could easily be a separate process, and likely in
some filesystems it is.

I fear, that your efforts to "save" the freezer are in vain.  It is
already moderately hackish with that PF_FREEZER_SKIP and the kernel
dotted randomly with try_to_freeze() calls, but adding bandaids to try
to order freezing userspace processes in the right order would just
make it a horrible mess.

Miklos

Re: Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Miklos Szeredi]

> > > > > > Actually fuse allows SIGKILL, because it's always fatal, and the
> > > > > > syscall may not be restarted.
> > > > > 
> > > > > I think you want to stick try_to_freeze() at the same places where you
> > > > > do SIGKILL handling. That should solve the 'syslogd is unfreezeable'
> > > > > problem.
> > > > 
> > > > I could, but it would not solve the general problem.  Namely, that the
> > > > presence of fuse imposes a certain ordering in which userspace tasks
> > > > have to be frozen.  And it is not possible to know this ordering.
> > > 
> > > Actually, why do you need this? There is no absolute need that you
> > > finish the request. You must either finish the request or let yourself
> > > be frozen.
> > > 
> > > A quick look through fuse reveals principally request_wait_answer()
> > > And maybe a few other places. Is there some hidden reason you cannot
> > > handle being frozen here?
> > 
> > Yes, fuse could handle being frozen there.  However that would only
> > solve part of the problem: an operation waiting for a reply could be
> > holding a VFS mutex and some other task may be blocked on that mutex.
> > 
> > How would you solve freezing those tasks?
> 
> How probable is this situation?

I guess it depends on usage patterns.

I don't remember seeing any such cases, even though I have a permanent
fuse mount, and I suspend regularly.  But the fs is probably totally
idle during suspend in my case.

But some people use fuse as a home or root filesystem and in those
cases it can become quite likely to cause problems.

Miklos

Re: Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Rafael J. Wysocki]

On Thursday, 5 July 2007 22:38, Miklos Szeredi wrote:
> > > > > > > Actually fuse allows SIGKILL, because it's always fatal, and the
> > > > > > > syscall may not be restarted.
> > > > > > 
> > > > > > I think you want to stick try_to_freeze() at the same places where you
> > > > > > do SIGKILL handling. That should solve the 'syslogd is unfreezeable'
> > > > > > problem.
> > > > > 
> > > > > I could, but it would not solve the general problem.  Namely, that the
> > > > > presence of fuse imposes a certain ordering in which userspace tasks
> > > > > have to be frozen.  And it is not possible to know this ordering.
> > > > 
> > > > Actually, why do you need this? There is no absolute need that you
> > > > finish the request. You must either finish the request or let yourself
> > > > be frozen.
> > > > 
> > > > A quick look through fuse reveals principally request_wait_answer()
> > > > And maybe a few other places. Is there some hidden reason you cannot
> > > > handle being frozen here?
> > > 
> > > Yes, fuse could handle being frozen there.  However that would only
> > > solve part of the problem: an operation waiting for a reply could be
> > > holding a VFS mutex and some other task may be blocked on that mutex.
> > > 
> > > How would you solve freezing those tasks?
> > 
> > How probable is this situation?
> 
> I guess it depends on usage patterns.
> 
> I don't remember seeing any such cases, even though I have a permanent
> fuse mount, and I suspend regularly.  But the fs is probably totally
> idle during suspend in my case.
> 
> But some people use fuse as a home or root filesystem and in those
> cases it can become quite likely to cause problems.

That means we need to prevent the freezer from sending freeze requests FUSE's
filesystem servers too early.

What about this (more or less):
1) We add a list of userland processes to the freezer that should not be frozen
in the first phase (ie. during the freezing of user space).  They will be frozen
anyway along with the freezable kernel threads.
2) FUSE adds its filesystem servers to this list upon connecting to them
3) FUSE removes its filesystem servers from this list upon disconnecting

Greetings,
Rafael


-- 
"Premature optimization is the root of all evil." - Donald Knuth

Re: Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Benjamin Herrenschmidt]

> Yes, fuse could handle being frozen there.  However that would only
> solve part of the problem: an operation waiting for a reply could be
> holding a VFS mutex and some other task may be blocked on that mutex.
> 
> How would you solve freezing those tasks?

That task is implicitely frozen... but the kernel doesn't know it and
thus the freezer timeouts or fails or deadlocks or whatever.

The freezer could be made to ignore tasks that are sleeping in the
kernel assuming that if they go out of it, they'll ultimately reach
do_signal and freeze, but that means they can potentially still issues
IOs which is what the freezer tries to avoid ...

Or the kernel could start tracking dependencies, but then, good luck
implementing that crap.

At the end of the day, I stand my ground, the freezer cannot be made
reliable without massive infrastructure changes or giving up on very
useful features such as fuse among others. Besides, it only partially
"hides" the problem of requests going to drivers, thus it's a bad
solutions.

We would be much better off spending time fixing the drivers to properly
block requests after suspended, and that also gives for free the ability
to do dynamic runtime suspend on them.

And for "trivial" drivers where we don't care, using late_suspend to
power the chip off later when IRQs are off is an easy enough way to
solve it with very little code (though won't help with dynamic PM but
that's not necessarily an issue). No need for a freezer either way.

Ben.

Re: Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Jeremy Maitin-Shepard]

Benjamin Herrenschmidt <benh@kernel.crashing.org> writes:

[snip]

> At the end of the day, I stand my ground, the freezer cannot be made
> reliable without massive infrastructure changes or giving up on very
> useful features such as fuse among others. Besides, it only partially
> "hides" the problem of requests going to drivers, thus it's a bad
> solutions.

I agree that the freezer absolutely should not be used for suspend to
ram ("suspend"), since it is unnecessary with properly written drivers,
which are important to have anyway.  It seems that it is indeed the
consensus that it will be phased out sooner or later.

It does seem that the current device suspend interface does not tell the
drivers enough, since as discussed, they need to know whether to merely
block if they receive a request while suspended (as should be done while
initiating a suspend to ram), or if they should wake up the device (as
should be done if a suspend to ram is not in progress).  Clearly these
two cases need to be addressed by every driver supporting suspend/resume
(but possibly indirectly if the subsystem handles it for them).

The current hibernate approach used by all of the existing
implementations for Linux seems to depend fundamentally on the freezer,
though, in order to actually save the system state.  Thus, it will still
be necessary to fix all of the issues with the freezer, or adopt an
alternate hibernate approach (which is unlikely).  Unfortunately, even
leaving kernel threads and certain drivers running after the snapshot is
taken means that the saved image isn't completely correct, and the
freezer cannot help with these issues.

[snip]

-- 
Jeremy Maitin-Shepard

Re: Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Oliver Neukum]

Am Freitag, 6. Juli 2007 schrieb Benjamin Herrenschmidt:
> > Yes, fuse could handle being frozen there.  However that would only
> > solve part of the problem: an operation waiting for a reply could be
> > holding a VFS mutex and some other task may be blocked on that mutex.
> > 
> > How would you solve freezing those tasks?
> 
> That task is implicitely frozen... but the kernel doesn't know it and
> thus the freezer timeouts or fails or deadlocks or whatever.
> 
> The freezer could be made to ignore tasks that are sleeping in the
> kernel assuming that if they go out of it, they'll ultimately reach
> do_signal and freeze, but that means they can potentially still issues
> IOs which is what the freezer tries to avoid ...
> 
> Or the kernel could start tracking dependencies, but then, good luck
> implementing that crap.

Do we need dependencies? Don't we know that fuse can deadlock only
on a limited number of locks in VFS?

	Regards
		Oliver

Re: Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Pavel Machek]

Hi!

> > > Actually fuse allows SIGKILL, because it's always fatal, and the
> > > syscall may not be restarted.
> > 
> > I think you want to stick try_to_freeze() at the same places where you
> > do SIGKILL handling. That should solve the 'syslogd is unfreezeable'
> > problem.
> 
> I could, but it would not solve the general problem.  Namely, that the
> presence of fuse imposes a certain ordering in which userspace tasks
> have to be frozen.  And it is not possible to know this ordering.

We can just wait for all fuse requests to be serviced before
proceeding further with freeze, right?

> And even if the ordering were solved, the freezer would still not work
> if the filesystem is not responding due to external events, such as a
> lost network (this affects NFS, CIFS, whatever just the same as
> fuse).

That's ok, you can't suspend if your hdd is dead, and in the same way
you can't suspend if your NFS server is dead. I agree it is ugly, but
we seem to live ok with that.

We could (and should?) handle that, probably by realizing that NFS is
not a disk and using interruptible sleep, but...

> > Plus, it would be nice to find out where suspend/hibernation is
> > triggering fuse activity. We can then decide where to fix it -- in
> > fuse or in suspend parts. You said sys_sync is not implemented... so
> > where is the problem?
> 
> I cannot say without having a sysrq-t of the situation.

Yes please. Can someone affected please produce sysrq-t?
								Pavel

-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

Re: Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Miklos Szeredi]

> We can just wait for all fuse requests to be serviced before
> proceeding further with freeze, right?

Right.  Nice way to slow down or stop the suspend with an unprivileged
process.  Avoiding that sort of DoS is one of the design goals of
fuse.

Look at it this way: the task of the freezer is to stop new I/O
hitting the hardware.  But it is totally indiscriminate about what it
stops, it tries to stop _everything_ even things which have nothing to
do with hardware.

Not nice.

Miklos

malicious filesystems (was Re: Re: [PATCH] Remove process freezer from suspend to RAM pathway)

[Pavel Machek]

Hi!

> > We can just wait for all fuse requests to be serviced before
> > proceeding further with freeze, right?
> 
> Right.  Nice way to slow down or stop the suspend with an unprivileged
> process.  Avoiding that sort of DoS is one of the design goals of
> fuse.

So you want me to handle _malicious_ filesystems now?

That should be easy... :-). You already have nasty deadlocks in FUSE,
and you solve them by "root can echo 1 > abort"... so allow me the
same possibility.

We can tell fused we are freezing, and if all the requests are not
serviced within, say, 30 seconds, we call the filesystem malicious and
do echo 1 > abort.

Not ideal, but neither is allowing malicious filesystems in the first
place...

> Look at it this way: the task of the freezer is to stop new I/O
> hitting the hardware.  But it is totally indiscriminate about what it
> stops, it tries to stop _everything_ even things which have nothing to
> do with hardware.
> 
> Not nice.

Not nice, but we don't know any better for now. "Just fix all the
drivers" basically means "just fix 90% of kernel".
									Pavel
-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

Re: malicious filesystems (was Re: Re: [PATCH] Remove process freezer from suspend to RAM pathway)

[Miklos Szeredi]

> > > We can just wait for all fuse requests to be serviced before
> > > proceeding further with freeze, right?
> > 
> > Right.  Nice way to slow down or stop the suspend with an unprivileged
> > process.  Avoiding that sort of DoS is one of the design goals of
> > fuse.
> 
> So you want me to handle _malicious_ filesystems now?

What I'd like, is a suspend, that works reliably, regardless of the
state of any userspace filesystem, network servers and such.

> That should be easy... :-). You already have nasty deadlocks in FUSE,
> and you solve them by "root can echo 1 > abort"... so allow me the
> same possibility.
> 
> We can tell fused we are freezing, and if all the requests are not
> serviced within, say, 30 seconds, we call the filesystem malicious and
> do echo 1 > abort.

Arbitrary time limits, nice.  Not.

This freezer is like an old house that's close to collapsing, and you
are basically just thinking of where to prop it up further.  To
continute this brilliant analogy, Rafael's patch at least demolishes
the worst part of the house, where bricks are already falling on our
head ;)

> Not ideal, but neither is allowing malicious filesystems in the first
> place...

Malicious programs are not something specific to fuse.  A lot of the
multiuser/multitasking OS design is about isolating things, so such a
program is limited in the damage it can do.

> > Look at it this way: the task of the freezer is to stop new I/O
> > hitting the hardware.  But it is totally indiscriminate about what it
> > stops, it tries to stop _everything_ even things which have nothing to
> > do with hardware.
> > 
> > Not nice.
> 
> Not nice, but we don't know any better for now. "Just fix all the
> drivers" basically means "just fix 90% of kernel".

And how much of that 90% currently has any power management?

Miklos

Re: malicious filesystems (was Re: Re: [PATCH] Remove process freezer from suspend to RAM pathway)

[Pavel Machek]

Hi!

> > > > We can just wait for all fuse requests to be serviced before
> > > > proceeding further with freeze, right?
> > > 
> > > Right.  Nice way to slow down or stop the suspend with an unprivileged
> > > process.  Avoiding that sort of DoS is one of the design goals of
> > > fuse.
> > 
> > So you want me to handle _malicious_ filesystems now?
> 
> What I'd like, is a suspend, that works reliably, regardless of the
> state of any userspace filesystem, network servers and such.

Well, fix userspace filesystems and maybe NFS. If they react to
sigstop in timely manner, they will work with suspend properly, too.

> > That should be easy... :-). You already have nasty deadlocks in FUSE,
> > and you solve them by "root can echo 1 > abort"... so allow me the
> > same possibility.
> > 
> > We can tell fused we are freezing, and if all the requests are not
> > serviced within, say, 30 seconds, we call the filesystem malicious and
> > do echo 1 > abort.
> 
> Arbitrary time limits, nice.  Not.
> 
> This freezer is like an old house that's close to collapsing, and you

Nice way to have useful discussion. Not.

Look, Linux was not designed with malicious filesystems in mind. In
particular, suspend was not designed with malicious filesystems in
mind, and VFS was not designed with malicious filesystems in mind.
That's why you have the nastiness with deadlocks... which you are
unable/unwilling to solve because you'd have to redesign VFS and meet
Al Viro.

Now.. freezer already includes timeout; if some part of kernel knows
nothing about suspend or had crashed, it will abort after some time,
telling you which part to blame. Another timeout for detecting
malicious userspace filesystems will not hurt much in this old house.
(Maybe you don't want to auto abort them, just tell user what happened.

We are stuck with refrigerator for now, and at least for hibernation,
I don't see any feasible alternative.

And now, can we get that sysrq-t trace? Dealing with 'unable to
freeze' may be hard, but the deadlock Matthew saw should be simple bug
for us to fix.

> > Not ideal, but neither is allowing malicious filesystems in the first
> > place...
> 
> Malicious programs are not something specific to fuse.  A lot of the
> multiuser/multitasking OS design is about isolating things, so such a
> program is limited in the damage it can do.

I'm talking malicious _filesystems_ here, and yes, fuse is first of
this kind. We want to handle unresponding NFS, but I believe handling
malicious NFS server nicely is slightly out of scope.

> > Not nice, but we don't know any better for now. "Just fix all the
> > drivers" basically means "just fix 90% of kernel".
> 
> And how much of that 90% currently has any power management?

Anything that's used in today's notebooks, I'd say...
							Pavel
-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

Re: malicious filesystems (was Re: Re: [PATCH] Remove process freezer from suspend to RAM pathway)

[Al Viro]

On Sun, Jul 08, 2007 at 12:37:48PM +0000, Pavel Machek wrote:
> I'm talking malicious _filesystems_ here, and yes, fuse is first of
> this kind. We want to handle unresponding NFS, but I believe handling
> malicious NFS server nicely is slightly out of scope.

If your variant doesn't handle compromised NFS server, your variant is
needs to be fixed...

Re: malicious filesystems (was Re: Re: [PATCH] Remove process freezer from suspend to RAM pathway)

[David Brownell]

On Sunday 08 July 2007, Al Viro wrote:
> On Sun, Jul 08, 2007 at 12:37:48PM +0000, Pavel Machek wrote:
> > I'm talking malicious _filesystems_ here, and yes, fuse is first of
> > this kind. We want to handle unresponding NFS, but I believe handling
> > malicious NFS server nicely is slightly out of scope.
> 
> If your variant doesn't handle compromised NFS server, your variant is
> needs to be fixed...

That would depend on the type of compromise, right?

Remember that the fundamental contract between a client and
a server includes the client extending some trust to that
server.  Whether it's appropriate to extend that trust (at
any given moment) is an out-of-band security issue.  Trust
is not a protocol issue ... more like an operational issue,
and only slightly an implementation issue.

A malicious filesystem could do many things.  It could send
private data off to someone who wasn't intended to receive
that data.  It could return falsified data.  It could do any
variety of things outside the protocol specification...

And *MOST* of those would be impractical to defend against
in code.  Which is why I have such a hard time agreeing
with your comment about "fixing" a client that doesn't try
to defend itself.  (Unless by "client" you also include the
whole operational side of the client, including regular
re-validation of the trust extended to that server... which
would at best minimize damage caused by compromises.)

Re: malicious filesystems (was Re: Re: [PATCH] Remove process freezer from suspend to RAM pathway)

[Miklos Szeredi]

> > > > > We can just wait for all fuse requests to be serviced before
> > > > > proceeding further with freeze, right?
> > > > 
> > > > Right.  Nice way to slow down or stop the suspend with an unprivileged
> > > > process.  Avoiding that sort of DoS is one of the design goals of
> > > > fuse.
> > > 
> > > So you want me to handle _malicious_ filesystems now?
> > 
> > What I'd like, is a suspend, that works reliably, regardless of the
> > state of any userspace filesystem, network servers and such.
> 
> Well, fix userspace filesystems and maybe NFS. If they react to
> sigstop in timely manner, they will work with suspend properly, too.

Which is pretty much impossible, given the unix filesystem API.  To be
able to react to sigstop, the operations in question need to be
restartable.  Which they are not, so they can't react to sigstop.  End
of story.

> > > That should be easy... :-). You already have nasty deadlocks in FUSE,
> > > and you solve them by "root can echo 1 > abort"... so allow me the
> > > same possibility.
> > > 
> > > We can tell fused we are freezing, and if all the requests are not
> > > serviced within, say, 30 seconds, we call the filesystem malicious and
> > > do echo 1 > abort.
> > 
> > Arbitrary time limits, nice.  Not.
> > 
> > This freezer is like an old house that's close to collapsing, and you
> 
> Nice way to have useful discussion. Not.

Sorry, didn't mean to offend you.

> Look, Linux was not designed with malicious filesystems in mind. In
> particular, suspend was not designed with malicious filesystems in
> mind, and VFS was not designed with malicious filesystems in mind.
> That's why you have the nastiness with deadlocks... which you are
> unable/unwilling to solve because you'd have to redesign VFS and meet
> Al Viro.

No, I don't think we need to redesign the VFS, and also I think the
malicious filesystem thing has been adequately taken care of in fuse.

You may not like the fact that one process can cause another to go
into uninterruptible sleep, but in fact there's nothing wrong with
that.  The effects are localized to the mount owner, it won't cause
any system-wide ill, it's in fact no worse than a malicious process
doing ptrace().

And as explained above it's unavoidable due to the well established
userspace API.  The _point_ of fuse is to let this API be used by
unmodified programs and the filesystem be provided by a userspace
process.  After showing me the right way to do this with Podfuk, this
should not come as a surprise to you.

So the fact that the freezer can't handle this is unfortunate, but
it's just a symptom of the brokenness of it, not something that fuse
introduced.  Not being able to suspend with NFS (or other network
filesystems) when the network is lost shows that this is a deeper
problem.

And for some reason you seem not to accept that.  You think that the
problem is with fuse, NFS, CIFS, whatever, and not the freezer, when
in fact it's quite clear, that neither of the above should have
anything to do with power management.  Even if it was possible to fix
them, it would still be just fixing the symptoms.

> Now.. freezer already includes timeout; if some part of kernel knows
> nothing about suspend or had crashed, it will abort after some time,
> telling you which part to blame. Another timeout for detecting
> malicious userspace filesystems will not hurt much in this old house.
> (Maybe you don't want to auto abort them, just tell user what happened.
> 
> We are stuck with refrigerator for now, and at least for hibernation,
> I don't see any feasible alternative.

Even for hibernation, I don't see, why we would need all processes
being effectively in a stopped state.

As stated otherwise in the thread, suspend2 in fact allowed processes
to be in uninterruptible sleep instead, without negative side effects.

The current freezer does too much for suspend and for hibernate too.

> > Malicious programs are not something specific to fuse.  A lot of the
> > multiuser/multitasking OS design is about isolating things, so such a
> > program is limited in the damage it can do.
> 
> I'm talking malicious _filesystems_ here, and yes, fuse is first of
> this kind. We want to handle unresponding NFS, but I believe handling
> malicious NFS server nicely is slightly out of scope.

It doesn't have to be malicious, it's enough if the server crashes, or
the network connection is lost.

Do we want to maintain the status quo, just because we can live with
it?

> > > Not nice, but we don't know any better for now. "Just fix all the
> > > drivers" basically means "just fix 90% of kernel".
> > 
> > And how much of that 90% currently has any power management?
> 
> Anything that's used in today's notebooks, I'd say...

Which is what, 10% of all the drivers?  Then it really not as bad as
you try to make it sound.  And with the late suspend call (whatever it
does) that can take care of most of those, it really becomes just a
few drivers and subsystems to fix.

Miklos

Re: malicious filesystems (was Re: Re: [PATCH] Remove process freezer from suspend to RAM pathway)

[Rafael J. Wysocki]

On Sunday, 8 July 2007 16:23, Miklos Szeredi wrote:
> > > > > > We can just wait for all fuse requests to be serviced before
> > > > > > proceeding further with freeze, right?
> > > > > 
> > > > > Right.  Nice way to slow down or stop the suspend with an unprivileged
> > > > > process.  Avoiding that sort of DoS is one of the design goals of
> > > > > fuse.
> > > > 
> > > > So you want me to handle _malicious_ filesystems now?
> > > 
> > > What I'd like, is a suspend, that works reliably, regardless of the
> > > state of any userspace filesystem, network servers and such.
> > 
> > Well, fix userspace filesystems and maybe NFS. If they react to
> > sigstop in timely manner, they will work with suspend properly, too.
> 
> Which is pretty much impossible, given the unix filesystem API.  To be
> able to react to sigstop, the operations in question need to be
> restartable.  Which they are not, so they can't react to sigstop.  End
> of story.

Or not.  That depends on your willingness to cooperate, I'd say. :-)

> > > > That should be easy... :-). You already have nasty deadlocks in FUSE,
> > > > and you solve them by "root can echo 1 > abort"... so allow me the
> > > > same possibility.
> > > > 
> > > > We can tell fused we are freezing, and if all the requests are not
> > > > serviced within, say, 30 seconds, we call the filesystem malicious and
> > > > do echo 1 > abort.
> > > 
> > > Arbitrary time limits, nice.  Not.
> > > 
> > > This freezer is like an old house that's close to collapsing, and you
> > 
> > Nice way to have useful discussion. Not.
> 
> Sorry, didn't mean to offend you.
> 
> > Look, Linux was not designed with malicious filesystems in mind. In
> > particular, suspend was not designed with malicious filesystems in
> > mind, and VFS was not designed with malicious filesystems in mind.
> > That's why you have the nastiness with deadlocks... which you are
> > unable/unwilling to solve because you'd have to redesign VFS and meet
> > Al Viro.
> 
> No, I don't think we need to redesign the VFS, and also I think the
> malicious filesystem thing has been adequately taken care of in fuse.
> 
> You may not like the fact that one process can cause another to go
> into uninterruptible sleep, but in fact there's nothing wrong with
> that.

Well, this introduces interdependencies between processes that do not exist
otherwise.  Even if that isn't wrong per se, it's something that needs
consideration in any case.

IMO, FUSE breaks one of the assumptions that the freezer is based on and
saying that the freezer is broken because of that is unfair.

> The effects are localized to the mount owner, it won't cause 
> any system-wide ill, it's in fact no worse than a malicious process
> doing ptrace().
> 
> And as explained above it's unavoidable due to the well established
> userspace API.  The _point_ of fuse is to let this API be used by
> unmodified programs and the filesystem be provided by a userspace
> process.  After showing me the right way to do this with Podfuk, this
> should not come as a surprise to you.
> 
> So the fact that the freezer can't handle this is unfortunate, but
> it's just a symptom of the brokenness of it, not something that fuse
> introduced.  Not being able to suspend with NFS (or other network
> filesystems) when the network is lost shows that this is a deeper
> problem.

Well, the system that cannot access its filesystems is not in a consistent
state, so it generally is not reasonable to suspend or hibernate it.

In fact, NFS and similar filesystems should always be unmounted before the
suspend/hibernation to avoid problems that may arise after the resume, if
they become unavailable at that time.

> And for some reason you seem not to accept that.  You think that the
> problem is with fuse, NFS, CIFS, whatever, and not the freezer, when
> in fact it's quite clear, that neither of the above should have
> anything to do with power management.

IMHO, it's not _that_ clear.

> Even if it was possible to fix them, it would still be just fixing the
> symptoms. 

I don't think we need to fix the network filesystems.

If there's a mounted filesystem and we have at least one process waiting for
it to become available in the D state (ie. uninterruptible), then the system is
not in a consistent state and should not be suspended.

> > Now.. freezer already includes timeout; if some part of kernel knows
> > nothing about suspend or had crashed, it will abort after some time,
> > telling you which part to blame. Another timeout for detecting
> > malicious userspace filesystems will not hurt much in this old house.
> > (Maybe you don't want to auto abort them, just tell user what happened.
> > 
> > We are stuck with refrigerator for now, and at least for hibernation,
> > I don't see any feasible alternative.
> 
> Even for hibernation, I don't see, why we would need all processes
> being effectively in a stopped state.
> 
> As stated otherwise in the thread, suspend2 in fact allowed processes
> to be in uninterruptible sleep instead, without negative side effects.

And yet, Nigel thinks that the freezer is necessary for the hibernation.
Strange, no?

> The current freezer does too much for suspend and for hibernate too.

Yes, it does.  It's quilte similar to the BKL in that.  Still, for now, there's
no infrastructure for a more fine grained approach and instead of working
on introducing one, we're losing time in this thread.

> > > Malicious programs are not something specific to fuse.  A lot of the
> > > multiuser/multitasking OS design is about isolating things, so such a
> > > program is limited in the damage it can do.
> > 
> > I'm talking malicious _filesystems_ here, and yes, fuse is first of
> > this kind. We want to handle unresponding NFS, but I believe handling
> > malicious NFS server nicely is slightly out of scope.
> 
> It doesn't have to be malicious, it's enough if the server crashes, or
> the network connection is lost.
> 
> Do we want to maintain the status quo, just because we can live with
> it?

Say you're standing on a cliff and you have three choices:
(1) stay where you are (well, this is not attractive in the long run)
(2) jump down
(3) look for a path to go down slowly?

Which one will you choose?

> > > > Not nice, but we don't know any better for now. "Just fix all the
> > > > drivers" basically means "just fix 90% of kernel".
> > > 
> > > And how much of that 90% currently has any power management?
> > 
> > Anything that's used in today's notebooks, I'd say...
> 
> Which is what, 10% of all the drivers?  Then it really not as bad as
> you try to make it sound.  And with the late suspend call (whatever it
> does) that can take care of most of those, it really becomes just a
> few drivers and subsystems to fix.

Are they fixed _now_?

Greetings,
Rafael


-- 
"Premature optimization is the root of all evil." - Donald Knuth

Re: malicious filesystems (was Re: Re: [PATCH] Remove process freezer from suspend to RAM pathway)

[Al Viro]

On Sun, Jul 08, 2007 at 08:08:56PM +0200, Rafael J. Wysocki wrote:
> Well, the system that cannot access its filesystems is not in a consistent
> state, so it generally is not reasonable to suspend or hibernate it.
> 
> In fact, NFS and similar filesystems should always be unmounted before the
> suspend/hibernation to avoid problems that may arise after the resume, if
> they become unavailable at that time.

Ah, thanks for clarifying that. [mentally putting the entire thing into
"unusable due to idiotic design imposing idiotic limitations" bin and
stepping out of this thread]

Re: malicious filesystems (was Re: Re: [PATCH] Remove process freezer from suspend to RAM pathway)

[Miklos Szeredi]

> > > Well, fix userspace filesystems and maybe NFS. If they react to
> > > sigstop in timely manner, they will work with suspend properly, too.
> > 
> > Which is pretty much impossible, given the unix filesystem API.  To be
> > able to react to sigstop, the operations in question need to be
> > restartable.  Which they are not, so they can't react to sigstop.  End
> > of story.
> 
> Or not.  That depends on your willingness to cooperate, I'd say. :-)

Do you actually understand what I'm talking about?  Because it sure
doesn't depend on my cooperation.

Maybe I'm stupid, and I'm missing something obvious.  In that case
please explain how you propose to make filesystem operations, like
rename() restartable.

> > You may not like the fact that one process can cause another to go
> > into uninterruptible sleep, but in fact there's nothing wrong with
> > that.
> 
> Well, this introduces interdependencies between processes that do not exist
> otherwise.  Even if that isn't wrong per se, it's something that needs
> consideration in any case.
> 
> IMO, FUSE breaks one of the assumptions that the freezer is based on and
> saying that the freezer is broken because of that is unfair.

The freezer is not broken because of that, it's broken anyway.  What
we are seeing is a _symptom_ of it being broken.

And by broken, I don't mean it's buggy or that it was badly designed.
I just mean, that it's simply not what suspend should depend on, to
protect drivers.

> > So the fact that the freezer can't handle this is unfortunate, but
> > it's just a symptom of the brokenness of it, not something that fuse
> > introduced.  Not being able to suspend with NFS (or other network
> > filesystems) when the network is lost shows that this is a deeper
> > problem.
> 
> Well, the system that cannot access its filesystems is not in a consistent
> state, so it generally is not reasonable to suspend or hibernate it.

Saying the system must be in a "consistent" state, and defining
consistent as "every process is stopped", is just an arbitrary
limitation that fits what the freezer does now.  Yes the _hardware_
state must be consistent, but that has nothing to do with either fuse
or NFS.  Can't you see that?

> > As stated otherwise in the thread, suspend2 in fact allowed processes
> > to be in uninterruptible sleep instead, without negative side effects.
> 
> And yet, Nigel thinks that the freezer is necessary for the hibernation.
> Strange, no?

I'm totally ignorant about why the freezer is necessary for hibernate.
Please explain.

Yes, we need to make sure, that nothing is scheduled during (and
possibly after) taking the snapshot.  But AFAICS that could be
achieved by unplugging all but one CPU.

Miklos

Re: malicious filesystems (was Re: Re: [PATCH] Remove process freezer from suspend to RAM pathway)

[Rafael J. Wysocki]

On Sunday, 8 July 2007 21:50, Miklos Szeredi wrote:
> > > > Well, fix userspace filesystems and maybe NFS. If they react to
> > > > sigstop in timely manner, they will work with suspend properly, too.
> > > 
> > > Which is pretty much impossible, given the unix filesystem API.  To be
> > > able to react to sigstop, the operations in question need to be
> > > restartable.  Which they are not, so they can't react to sigstop.  End
> > > of story.
> > 
> > Or not.  That depends on your willingness to cooperate, I'd say. :-)
> 
> Do you actually understand what I'm talking about?  Because it sure
> doesn't depend on my cooperation.
>
> Maybe I'm stupid, and I'm missing something obvious.  In that case
> please explain how you propose to make filesystem operations, like
> rename() restartable.

I'm not proposing that, sorry.  I'm not a filesystem expert, so don't think
I can propose a working solution here, right now.  Still, maybe something
like in fork.c, line 1409-1411 might work (I know that it won't solve the
"other tasks may be waiting on VFS mutexes" issue, but at least it may
decrease the probability of freezer failure).

> > > You may not like the fact that one process can cause another to go
> > > into uninterruptible sleep, but in fact there's nothing wrong with
> > > that.
> > 
> > Well, this introduces interdependencies between processes that do not exist
> > otherwise.  Even if that isn't wrong per se, it's something that needs
> > consideration in any case.
> > 
> > IMO, FUSE breaks one of the assumptions that the freezer is based on and
> > saying that the freezer is broken because of that is unfair.
> 
> The freezer is not broken because of that, it's broken anyway.  What
> we are seeing is a _symptom_ of it being broken.
> 
> And by broken, I don't mean it's buggy or that it was badly designed.
> I just mean, that it's simply not what suspend should depend on, to
> protect drivers.

Please have a look at the documentation update at the bottom of this patch:

http://www.sisk.pl/kernel/hibernation_and_suspend/2.6.22-rc7/patches/15-freezer-make-kernel-threads-nonfreezable-by-default.patch

It says what the freezer is for in the first place. :-)
 
> > > So the fact that the freezer can't handle this is unfortunate, but
> > > it's just a symptom of the brokenness of it, not something that fuse
> > > introduced.  Not being able to suspend with NFS (or other network
> > > filesystems) when the network is lost shows that this is a deeper
> > > problem.
> > 
> > Well, the system that cannot access its filesystems is not in a consistent
> > state, so it generally is not reasonable to suspend or hibernate it.
> 
> Saying the system must be in a "consistent" state, and defining
> consistent as "every process is stopped", is just an arbitrary
> limitation that fits what the freezer does now.  Yes the _hardware_
> state must be consistent, but that has nothing to do with either fuse
> or NFS.  Can't you see that?

Well, I can agree as far as FUSE is concerned.  Still, imagine that you have
an NFS share mounted, which is unavailable at the moment and one of the
tasks waits for an I/O on it to complete (it might hold some locks needed
for suspending, but say it doesn't).  Then you suspend, take the box somewhere
else and attach to a different network with a different NFS server.  Now, you
resume and you have a mess to recover from.  Yes, it _should_ be recoverable,
but refusing to suspend in such a case is not okerkill, IMHO.

> > > As stated otherwise in the thread, suspend2 in fact allowed processes
> > > to be in uninterruptible sleep instead, without negative side effects.
> > 
> > And yet, Nigel thinks that the freezer is necessary for the hibernation.
> > Strange, no?
> 
> I'm totally ignorant about why the freezer is necessary for hibernate.
> Please explain.

See above. :-)

> Yes, we need to make sure, that nothing is scheduled during (and
> possibly after) taking the snapshot.  But AFAICS that could be
> achieved by unplugging all but one CPU.

Actually, we want things to get scheduled, because we need some of them to
save the image (to make things more difficult, we don't know what will be
needed to save the image in advance ;-)).

Greetings,
Rafael


-- 
"Premature optimization is the root of all evil." - Donald Knuth

Re: malicious filesystems (was Re: Re: [PATCH] Remove process freezer from suspend to RAM pathway)

[Miklos Szeredi]

> Please have a look at the documentation update at the bottom of this patch:
> 
> http://www.sisk.pl/kernel/hibernation_and_suspend/2.6.22-rc7/patches/15-freezer-make-kernel-threads-nonfreezable-by-default.patch
> 
> It says what the freezer is for in the first place. :-)

Thanks, good description.

> > Yes, we need to make sure, that nothing is scheduled during (and
> > possibly after) taking the snapshot.  But AFAICS that could be
> > achieved by unplugging all but one CPU.
> 
> Actually, we want things to get scheduled, because we need some of them to
> save the image (to make things more difficult, we don't know what will be
> needed to save the image in advance ;-)).

Well, kexecing a new kernel (as being pursued in one of the
subthreads) to do the saving should take care of that.

In that case the "we need suspend to be invisible to userspace" as a
reason to use the freezer would also be moot, since if you don't
schedule userspace after offlining the CPUs, it can't notice this.

Miklos

Re: malicious filesystems (was Re: Re: [PATCH] Remove process freezer from suspend to RAM pathway)

[Oliver Neukum]

Am Montag, 9. Juli 2007 schrieb Miklos Szeredi:
> In that case the "we need suspend to be invisible to userspace" as a
> reason to use the freezer would also be moot, since if you don't
> schedule userspace after offlining the CPUs, it can't notice this.

After? Can you do the offlining atomically?

	Regards
		Oliver

Re: malicious filesystems (was Re: Re: [PATCH] Remove process freezer from suspend to RAM pathway)

[Miklos Szeredi]

> > In that case the "we need suspend to be invisible to userspace" as a
> > reason to use the freezer would also be moot, since if you don't
> > schedule userspace after offlining the CPUs, it can't notice this.
> 
> After? Can you do the offlining atomically?

Don't know.  Wait for all CPUs to reach a scheduling point and then
take them offline?  Doesn't sound too difficult.

Miklos

Re: malicious filesystems (was Re: Re: [PATCH] Remove process freezer from suspend to RAM pathway)

[Rafael J. Wysocki]

On Sunday, 8 July 2007 09:21, Miklos Szeredi wrote:
> > > > We can just wait for all fuse requests to be serviced before
> > > > proceeding further with freeze, right?
> > > 
> > > Right.  Nice way to slow down or stop the suspend with an unprivileged
> > > process.  Avoiding that sort of DoS is one of the design goals of
> > > fuse.
> > 
> > So you want me to handle _malicious_ filesystems now?
> 
> What I'd like, is a suspend, that works reliably, regardless of the
> state of any userspace filesystem, network servers and such.

And then you also would like to have a consistent state of the system after
the resume, but if the state were inconsistent before the suspend that
would be difficult to get.

> > That should be easy... :-). You already have nasty deadlocks in FUSE,
> > and you solve them by "root can echo 1 > abort"... so allow me the
> > same possibility.
> > 
> > We can tell fused we are freezing, and if all the requests are not
> > serviced within, say, 30 seconds, we call the filesystem malicious and
> > do echo 1 > abort.
> 
> Arbitrary time limits, nice.  Not.
>
> This freezer is like an old house that's close to collapsing, and you
> are basically just thinking of where to prop it up further.  To
> continute this brilliant analogy, Rafael's patch at least demolishes
> the worst part of the house, where bricks are already falling on our
> head ;)

Actually, this isn't correct and I have some testing problems with this
patch (as I expected, BTW), so it's not going anywhere.  Alternative ideas
will be appreciated. :-)

> > Not ideal, but neither is allowing malicious filesystems in the first
> > place...
> 
> Malicious programs are not something specific to fuse.  A lot of the
> multiuser/multitasking OS design is about isolating things, so such a
> program is limited in the damage it can do.
>
> > > Look at it this way: the task of the freezer is to stop new I/O
> > > hitting the hardware.  But it is totally indiscriminate about what it
> > > stops, it tries to stop _everything_ even things which have nothing to
> > > do with hardware.
> > > 
> > > Not nice.
> > 
> > Not nice, but we don't know any better for now. "Just fix all the
> > drivers" basically means "just fix 90% of kernel".
> 
> And how much of that 90% currently has any power management?

Well, you know why that actually started to be a real problem?  Because
sufficiently many people have started to use suspend/hibernation and it
actually works for quite a lot of them.  This makes people to try exotic
combinations like suspend+FUSE and they find that doesn't work.

Greetings,
Rafael


-- 
"Premature optimization is the root of all evil." - Donald Knuth

Re: malicious filesystems (was Re: Re: [PATCH] Remove process freezer from suspend to RAM pathway)

[Miklos Szeredi]

> > > We can just wait for all fuse requests to be serviced before
> > > proceeding further with freeze, right?
> > 
> > Right.  Nice way to slow down or stop the suspend with an unprivileged
> > process.  Avoiding that sort of DoS is one of the design goals of
> > fuse.
> 
> So you want me to handle _malicious_ filesystems now?
> 
> That should be easy... :-). You already have nasty deadlocks in FUSE,
> and you solve them by "root can echo 1 > abort"... so allow me the
> same possibility.
> 
> We can tell fused we are freezing, and if all the requests are not
> serviced within, say, 30 seconds, we call the filesystem malicious and
> do echo 1 > abort.
> 
> Not ideal, but neither is allowing malicious filesystems in the first
> place...

Actually there's also a non-malicious case in which waiting for
requests to finish won't work: when one fuse filesystem is accessing
another.

Since we are blocking new fuse requests, that might block a fuse
daemon, which in turn makes it impossible to finish the pending
request.

And this is not at all theoretical, I know that encfs is used over
various other fuse filesystems like sshfs or ntfs-3g.

Yeah, stacking userspace filesystems could be done entirely in
userspace, and I'm actually working on that (with fuse-2.7.0 already
supporting some basic stacking).

But the point is, that the "wait for fuse to quiescence" hack would
not work in todays environment.

Miklos

Re: malicious filesystems (was Re: Re: [PATCH] Remove process freezer from suspend to RAM pathway)

[Pavel Machek]

Hi!

> > So you want me to handle _malicious_ filesystems now?
> > 
> > That should be easy... :-). You already have nasty deadlocks in FUSE,
> > and you solve them by "root can echo 1 > abort"... so allow me the
> > same possibility.
> > 
> > We can tell fused we are freezing, and if all the requests are not
> > serviced within, say, 30 seconds, we call the filesystem malicious and
> > do echo 1 > abort.
> > 
> > Not ideal, but neither is allowing malicious filesystems in the first
> > place...
> 
> Actually there's also a non-malicious case in which waiting for
> requests to finish won't work: when one fuse filesystem is accessing
> another.
> 
> Since we are blocking new fuse requests, that might block a fuse
> daemon, which in turn makes it impossible to finish the pending
> request.

Hmm, yes, this is ugly. But will it hurt? We'll just wait for number
of active fuse requests to go down to 0 before proceeding. It may
still livelock on system with busy fused-s, but we'll detect that with
timeout, and I believe it is 'don't do that' situation.

> And this is not at all theoretical, I know that encfs is used over
> various other fuse filesystems like sshfs or ntfs-3g.
> 
> Yeah, stacking userspace filesystems could be done entirely in
> userspace, and I'm actually working on that (with fuse-2.7.0 already
> supporting some basic stacking).

Great.

> But the point is, that the "wait for fuse to quiescence" hack would
> not work in todays environment.

Ok, I'll just blame fuse here. 'You have to write to /sys
files for SIGKILL to work' is not funny.
							Pavel
-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

Re: malicious filesystems (was Re: Re: [PATCH] Remove process freezer from suspend to RAM pathway)

[Miklos Szeredi]

> > Actually there's also a non-malicious case in which waiting for
> > requests to finish won't work: when one fuse filesystem is accessing
> > another.
> > 
> > Since we are blocking new fuse requests, that might block a fuse
> > daemon, which in turn makes it impossible to finish the pending
> > request.
> 
> Hmm, yes, this is ugly. But will it hurt? We'll just wait for number
> of active fuse requests to go down to 0 before proceeding.

I'm thinking of the following situation, immediately after we begin to
wait for fuse requests to subside:

  - task A is performing file operation O1
  - O1 is being served by task B
  - task B needs to perform O2 before finishing O1

Since we don't allow new requests, O2 will never start.  So O1 will
never finish.  So the number of fuse requests will never go to 0.

> It may still livelock on system with busy fused-s, but we'll detect
> that with timeout, and I believe it is 'don't do that' situation.

There are no busy fused's.  If we allowed O2 to go through, everything
would be nice and dandy.  But we cannot know that O2 needs to be let
through, and other fuse requests must not.

> > And this is not at all theoretical, I know that encfs is used over
> > various other fuse filesystems like sshfs or ntfs-3g.
> > 
> > Yeah, stacking userspace filesystems could be done entirely in
> > userspace, and I'm actually working on that (with fuse-2.7.0 already
> > supporting some basic stacking).
> 
> Great.
> 
> > But the point is, that the "wait for fuse to quiescence" hack would
> > not work in todays environment.
> 
> Ok, I'll just blame fuse here. 'You have to write to /sys
> files for SIGKILL to work' is not funny.

SIGKILL won't work on a stopped task.  Neither on a traced task.
Neither on a zombie (how many newbies are thoroughly confused about
that ;)

And it won't work on a task that is hanging on an NFS mount, when the
network broke.

Oh, and it won't work on a deadlocked fuse filesystem.  How many of
these have you met?  How many of the others?

Miklos

Re: malicious filesystems (was Re: Re: [PATCH] Remove process freezer from suspend to RAM pathway)

[Miklos Szeredi]

> > Ok, I'll just blame fuse here. 'You have to write to /sys
> > files for SIGKILL to work' is not funny.
> 
> SIGKILL won't work on a stopped task.  Neither on a traced task.
> Neither on a zombie (how many newbies are thoroughly confused about
> that ;)
> 
> And it won't work on a task that is hanging on an NFS mount, when the
> network broke.
> 
> Oh, and it won't work on a deadlocked fuse filesystem.  How many of
> these have you met?  How many of the others?

And maybe someday, when I'm feeling _really_ bored, I may start
hacking the VFS to make it possible for tasks to be killed while
waiting on a mutex.  Can a task doing a page fault be interrupted?  I
don't see why not.

But, it's _not_ going to happen now, to fix the problems with suspend.
Hacking fuse (however broken it may be) or the VFS is not the right
way to fix those problems.  It may be easier than fixing the drivers,
though I doubt it.  But even if it's the easier way it's not the
_right_ way.

We want the various subsystems in the kernel to be less dependent on
each other not more.  And the freezer is just adding unnecessary
dependence on totally unrelated code.  And that does not only impact
fuse, but a lot of other code in the kernel, in the form of
try_to_freeze() calls all over the place.

Is that so hard to understand?

You can bash fuse all you want, I really don't mind.  But please stop
trying to prove, that this situation is best handled in fuse.  It just
won't happen, not because I'm lazy to do the work but because I don't
have the guts to do something so obviously wrong.

Miklos

Re: Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Miklos Szeredi]

> Don't you think, however, that it can be modified a little to play well,
> for example, with the freezer?

I could stick a couple of try_to_freeze()s into fuse, and that would
make suspend failure less likely.  But making problems less easy to
reproduce is not a good thing.

Miklos

Re: Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Rafael J. Wysocki]

On Thursday, 5 July 2007 14:24, Miklos Szeredi wrote:
> > Don't you think, however, that it can be modified a little to play well,
> > for example, with the freezer?
> 
> I could stick a couple of try_to_freeze()s into fuse, and that would
> make suspend failure less likely.  But making problems less easy to
> reproduce is not a good thing.

So, how about eliminating them?

Greetings,
Rafael


-- 
"Premature optimization is the root of all evil." - Donald Knuth

Re: Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Miklos Szeredi]

> > > Don't you think, however, that it can be modified a little to play well,
> > > for example, with the freezer?
> > 
> > I could stick a couple of try_to_freeze()s into fuse, and that would
> > make suspend failure less likely.  But making problems less easy to
> > reproduce is not a good thing.
> 
> So, how about eliminating them?

That can't be done just within fuse, a process might be sleeping on a
VFS mutex.  Do we want to hack VFS as well?

I guess I know your answer.  But it ain't gonna work.  Suspend code
really doesn't belong in VFS, and I'm pretty sure the maintainers of
that little piece of code would agree with me on this.

Miklos

Re: Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Rafael J. Wysocki]

On Thursday, 5 July 2007 15:50, Miklos Szeredi wrote:
> > > > Don't you think, however, that it can be modified a little to play well,
> > > > for example, with the freezer?
> > > 
> > > I could stick a couple of try_to_freeze()s into fuse, and that would
> > > make suspend failure less likely.  But making problems less easy to
> > > reproduce is not a good thing.
> > 
> > So, how about eliminating them?
> 
> That can't be done just within fuse, a process might be sleeping on a
> VFS mutex.  Do we want to hack VFS as well?

No.

> I guess I know your answer.  But it ain't gonna work.  Suspend code
> really doesn't belong in VFS, and I'm pretty sure the maintainers of
> that little piece of code would agree with me on this.

Surprise, surprise.  Not that I'm scared of the VFS maintainers, though. ;-)

Greetings,
Rafael


-- 
"Premature optimization is the root of all evil." - Donald Knuth

Re: Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Miklos Szeredi]

> > I guess I know your answer.  But it ain't gonna work.  Suspend code
> > really doesn't belong in VFS, and I'm pretty sure the maintainers of
> > that little piece of code would agree with me on this.
> 
> Surprise, surprise.  Not that I'm scared of the VFS maintainers, though. ;-)

Probably you haven't had a close encounter with them yet.  They are
pretty much the _most_ dangerous people in kernel land ;)

Miklos

Re: Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Pavel Machek]

Hi!

> > > > > > I have discussed the benefits elsewhere.  As for the deadlocks -- do 
> > > > > > you still observe them if you use the version of the freezer which 
> > > > > > doesn't freeze kernel threads?
> > > > > 
> > > > > In general the only way to guarantee there are no deadlocks is to
> > > > > construct the graph of dependencies between tasks.  Those dependencies
> > > > > are not in practice observable from outside the tasks, so it is
> > > > > virtually impossible to construct the graph.
> > > > 
> > > > In which way can user space tasks depend on each other in a way that
> > > > allows a them members of that cycle to be in uninterruptible sleep?
> > > 
> > >  - process A calls rename() on a fuse fs
> > >  - process B, the fuse server, starts to process the rename request
> > >  - process B is frozen before it can reply
> > > 
> > > Now process A is unfreezable.  We cannot make rename() restartable,
> > > hence it cannot be interruptible.
> > 
> > Yes, we are claiming fuse is very special in this regard, and perhaps
> > even broken.
> > 
> > Let's see. If I SIGSTOP the fuse server, I can get unrelated tasks
> > unkillable (even for SIGKILL!) forever.
> 
> Actually fuse allows SIGKILL, because it's always fatal, and the
> syscall may not be restarted.

Okay, and you should handle refrigerator in the same paths where you
handle SIGKILL. Just add try_to_freeze() there...
								Pavel
-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

Re: Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Miklos Szeredi]

> > Actually fuse allows SIGKILL, because it's always fatal, and the
> > syscall may not be restarted.
> 
> Okay, and you should handle refrigerator in the same paths where you
> handle SIGKILL. Just add try_to_freeze() there...

It's the fourth time I'm repeating this in this thread:

Yes adding try_to_freeze() there would partially solve the probelem.

But another task can be sleeping on a mutex held by the task waiting
for the reply.  And the freezer won't be able to handle that one.

Generally, calling try_to_freeze() with mutexes held is not a good
idea.

Miklos

Re: Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Pavel Machek]

On Fri 2007-07-06 09:07:38, Miklos Szeredi wrote:
> > > Actually fuse allows SIGKILL, because it's always fatal, and the
> > > syscall may not be restarted.
> > 
> > Okay, and you should handle refrigerator in the same paths where you
> > handle SIGKILL. Just add try_to_freeze() there...
> 
> It's the fourth time I'm repeating this in this thread:
> 
> Yes adding try_to_freeze() there would partially solve the probelem.
> 
> But another task can be sleeping on a mutex held by the task waiting
> for the reply.  And the freezer won't be able to handle that one.
> 
> Generally, calling try_to_freeze() with mutexes held is not a good
> idea.

Agreed, calling try_to_freeze() with mutex held is no-no, and it is
even documented somewhere.
								Pavel

-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

Re: Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Rafael J. Wysocki]

On Friday, 6 July 2007 00:04, Pavel Machek wrote:
> Hi!
> 
> > > > > > > I have discussed the benefits elsewhere.  As for the deadlocks -- do 
> > > > > > > you still observe them if you use the version of the freezer which 
> > > > > > > doesn't freeze kernel threads?
> > > > > > 
> > > > > > In general the only way to guarantee there are no deadlocks is to
> > > > > > construct the graph of dependencies between tasks.  Those dependencies
> > > > > > are not in practice observable from outside the tasks, so it is
> > > > > > virtually impossible to construct the graph.
> > > > > 
> > > > > In which way can user space tasks depend on each other in a way that
> > > > > allows a them members of that cycle to be in uninterruptible sleep?
> > > > 
> > > >  - process A calls rename() on a fuse fs
> > > >  - process B, the fuse server, starts to process the rename request
> > > >  - process B is frozen before it can reply
> > > > 
> > > > Now process A is unfreezable.  We cannot make rename() restartable,
> > > > hence it cannot be interruptible.
> > > 
> > > Yes, we are claiming fuse is very special in this regard, and perhaps
> > > even broken.
> > > 
> > > Let's see. If I SIGSTOP the fuse server, I can get unrelated tasks
> > > unkillable (even for SIGKILL!) forever.
> > 
> > Actually fuse allows SIGKILL, because it's always fatal, and the
> > syscall may not be restarted.
> 
> Okay, and you should handle refrigerator in the same paths where you
> handle SIGKILL. Just add try_to_freeze() there...

In fact the problem is more complicated than that, because some tasks may
be waiting on VFS locks related to FUSE.

Greetings,
Rafael


-- 
"Premature optimization is the root of all evil." - Donald Knuth

Re: Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Alan Stern]

On Thu, 5 Jul 2007, Paul Mackerras wrote:

> Alan Stern writes:
> 
> > Let's agree the kernel threads and the freezer are a separate issue.  
> 
> No, I don't think they are a separate issue, because I think the
> distinction the freezer makes between kernel threads and user threads
> is a false and misleading distinction.

That's a little strong.  "Misleading" I could understand, but "false"?  
Isn't the distinction between a kernel thread and a user task pretty 
clear-cut (except for a few borderline cases which aren't at issue just 
now)?

> > I agree the kernel threads which try to do I/O during a suspend will 
> > need extra attention.  However if these threads are necessary for the 
> > suspend procedure, then blocking them (which is how people on this 
> > thread have been saying driver should treat I/O requests during a 
> > suspend) will cause additional problems.  There's no way around it; 
> > these threads _will_ require more work.
> 
> There is a way around it; do the request blocking in the drivers,
> where it belongs.

How will that help?  Block the kernel thread in the freezer or block it 
in the driver -- either way it is blocked.  So how do your deadlocks 
get resolved?

> In general the only way to guarantee there are no deadlocks is to
> construct the graph of dependencies between tasks.  Those dependencies
> are not in practice observable from outside the tasks, so it is
> virtually impossible to construct the graph.
> 
> The "don't freeze kernel threads" thing is an attempt to make a crude
> approximation to the dependency graph (by saying kernel threads only
> depend on other kernel threads), but the approximation breaks down
> when you have FUSE or user-level device drivers.

I disagree with your analysis -- not that it's completely wrong, but it 
points out an existing basic problem in the kernel.  The kernel should 
never depend on userspace!  More correctly, a task executing in the 
kernel should never block with any sort of mutex or other lock held (in 
a way that would preclude it from being frozen, let's say) while 
waiting for a response from userspace.

Then the dependency graph would be easy to construct: User tasks can
depend on whatever they want, and kernel threads never depend on a user
task.

If this contradicts the existing implementations and APIs for userspace 
filesystems, then so be it.  My conclusion would be that the 
implementations and APIs should be changed.

> > There remains the problem of user tasks whose assistance is required to 
> > carry out some I/O (as with FUSE).  If the I/O can be deferred until 
> > after the resume, then there's no problem.  If the I/O can be carried 
> > out before the suspend, then it should be.  And finally, if the I/O 
> > must be done during the suspend, you're in real trouble -- how do you 
> > do I/O to a suspended device?
> 
> So why doesn't that argument apply to kernel threads? :)

It _does_ apply to kernel threads.  That's exactly why I wrote above 
that kernel threads which try to do I/O during a suspend will need 
extra attention.

Alan Stern

Re: Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Paul Mackerras]

Alan Stern writes:

> USB already implements runtime PM.  If a device is suspended at runtime
> and a task tries to access it, the device is automatically resumed.  
> No problem there.
> 
> The problem comes when the system is doing a STR.  Right now the code
> doesn't keep track of the difference between a runtime suspend and a
> system suspend -- once the device is suspended, it's suspended, period.  

Whether or not to resume a suspended device when an I/O request comes
in is a policy decision, and there could be cases where the user wants
I/O requests to be blocked, or to fail, or to be dropped while the
device is suspended, even for runtime power management.  For example,
a sound card could be suspended due to a low-battery condition, and in
that case you would want the driver to just drop any data that
userspace tries to write to the soundcard.

> Yes, the code could be changed to keep track of the reason for a device
> suspend.  But that just raises the old problem of what to do when
> there's an I/O request for a suspended device during STR.

Is this actually a real problem?  I would think the policy would be
"block" for block devices (pun not intended :), "drop" for network
devices, etc.

> Consider a particularly troublesome case: During STR, a non-frozen task
> writes to /sys/bus/BBB/drivers/DDD/bind.  The sysfs core grabs the
> device semaphore and calls the driver's probe routine.  If the driver
> isn't PM-aware it simply tries to initialize the device and fails
> because the device is already suspended.  That's no good; it isn't
> transparent.

How did the device get suspended if it didn't have a driver?  If it
did have a driver, why didn't the bind attempt fail?

> So assume the driver is PM-aware.  It tries to resume the device, which
> fails because STR is underway.  Now what can it do?  There's only one 
> possibility: It must block until the resume call can succeed.  But when 
> is that?
> 
> It has to be before the PM core tries to resume the device, because the 
> core will try to acquire the device semaphore and will block waiting 
> for the probe call to complete.  But it has to be after the PM core 
> resumes the device's parent, because obviously the device can't resume 
> until its parent is awake.

Suppose the device-model core code simply blocked all bind and unbind
requests while suspend is under way, until resume is finished.
Wouldn't that solve the problem?

Paul.

Re: Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Rafael J. Wysocki]

On Tuesday, 3 July 2007 16:21, Johannes Berg wrote:
> On Tue, 2007-07-03 at 14:56 +0200, Rafael J. Wysocki wrote:
> 
> > Still, can you please read this post from Alan Stern:
> > 
> > https://lists.linux-foundation.org/pipermail/linux-pm/2007-June/012847.html
> > 
> > ?  I don't think I'm able to repeat the arguments given in there in a
> > convincing way.
> 
> As I read it, Alan basically has two objections:
>  (1) drivers shouldn't need to worry about this
>  (2) suspend should be transparent to userspace
> 
> His proposed solution (freezing tasks when they cross the kernel
> boundary) helps for the s-t-r case, but in fact doesn't solve (1)
> because devices can be suspended at runtime

This is a different thing and a different infrastructure is needed for it (not
present at the moment).

> and then you certainly do not want to freeze tasks that try to access the
> device. 
> 
> (2) is related but not identical, what if you have a device suspended at
> runtime and some tasks tries to access it; should the task block until
> you wake up that device?

I think the device should be woken up in that case.

> I think the core of the discussion isn't appreciated by everybody here
> yet---we need to solve both run-time and suspend-to-ram-time device
> suspend, not just one of them.

For now, we're discussing the suspend-to-ram-time suspend only, for which
we have (some) infrastrcuture (and which should be supported by all drivers,
IMO).

Greetings,
Rafael


-- 
"Premature optimization is the root of all evil." - Donald Knuth

Re: Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Johannes Berg]

[-- Attachment #1.1: Type: text/plain, Size: 1367 bytes --]

On Tue, 2007-07-03 at 16:51 +0200, Rafael J. Wysocki wrote:

> > His proposed solution (freezing tasks when they cross the kernel
> > boundary) helps for the s-t-r case, but in fact doesn't solve (1)
> > because devices can be suspended at runtime
> 
> This is a different thing and a different infrastructure is needed for it (not
> present at the moment).

Yeah I should've said "any of his proposed solutions"

> > and then you certainly do not want to freeze tasks that try to access the
> > device. 
> > 
> > (2) is related but not identical, what if you have a device suspended at
> > runtime and some tasks tries to access it; should the task block until
> > you wake up that device?
> 
> I think the device should be woken up in that case.

Ah, but that also means the device has to actually know about it.

> > I think the core of the discussion isn't appreciated by everybody here
> > yet---we need to solve both run-time and suspend-to-ram-time device
> > suspend, not just one of them.
> 
> For now, we're discussing the suspend-to-ram-time suspend only, for which
> we have (some) infrastrcuture (and which should be supported by all drivers,
> IMO).

Right but if we solve the run-time suspend case in favour of having the
device driver know about it then the suspend-to-ram-time suspend case
solves itself.

johannes

[-- Attachment #1.2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 190 bytes --]

[-- Attachment #2: Type: text/plain, Size: 0 bytes --]

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Benjamin Herrenschmidt]

> >  - For STR, don't do the freezer thing.
> 
> In the long run, I agree.
> 
> Still, can you please read this post from Alan Stern:
> 
> https://lists.linux-foundation.org/pipermail/linux-pm/2007-June/012847.html
> 
> ?  I don't think I'm able to repeat the arguments given in there in a
> convincing way.

That's the same crackpot I've been hearing for the past 3 years or
so ...

Both Paulus and I think the freezer is just a way to try to put your
head in the sand and ignore the problem. It causes as many problems as
it solves on its own, and is just not a solution that will be of any use
once you start implementing dynamic PM schemes etc...

In many cases, having proper support for "live" suspend of devices is
just a matter of having a couple of helpers in whatever subsystem those
drivers hookup with. In the case of network, for example, it's mostly
trivial (stop the queue). For block, it's not terribly hard neither,
though you want to have some orderign/atomicity between the blocking of
the incoming request queue and the sending of things like spindown &
flush commands to the disk. For old-style IDE, that was fairly easily
solved by piping suspend/resume command down the request queue itself
and have the queue block/unblbock itself after processing them. Some of
that logic could maybe be moved to the block layer for all block drivers
to benefit.

But yes, overall, there is work to do on drivers and I'm doing the ones
I hit on the platforms I use. I don't think the freezer is any kind of
remotely good solution, just a way to continue avoiding the problem.

Ben.

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Rafael J. Wysocki]

On Tuesday, 3 July 2007 23:14, Benjamin Herrenschmidt wrote:
> 
> > >  - For STR, don't do the freezer thing.
> > 
> > In the long run, I agree.
> > 
> > Still, can you please read this post from Alan Stern:
> > 
> > https://lists.linux-foundation.org/pipermail/linux-pm/2007-June/012847.html
> > 
> > ?  I don't think I'm able to repeat the arguments given in there in a
> > convincing way.
> 
> That's the same crackpot I've been hearing for the past 3 years or
> so ...
> 
> Both Paulus and I think the freezer is just a way to try to put your
> head in the sand and ignore the problem. It causes as many problems as
> it solves on its own, and is just not a solution that will be of any use
> once you start implementing dynamic PM schemes etc...
> 
> In many cases, having proper support for "live" suspend of devices is
> just a matter of having a couple of helpers in whatever subsystem those
> drivers hookup with. In the case of network, for example, it's mostly
> trivial (stop the queue). For block, it's not terribly hard neither,
> though you want to have some orderign/atomicity between the blocking of
> the incoming request queue and the sending of things like spindown &
> flush commands to the disk. For old-style IDE, that was fairly easily
> solved by piping suspend/resume command down the request queue itself
> and have the queue block/unblbock itself after processing them. Some of
> that logic could maybe be moved to the block layer for all block drivers
> to benefit.
> 
> But yes, overall, there is work to do on drivers and I'm doing the ones
> I hit on the platforms I use. I don't think the freezer is any kind of
> remotely good solution, just a way to continue avoiding the problem.

Still, do you really think that we're ready to drop it _right_ _now_ (I'm
referring to suspend only) and if so than on what basis (except that you
don't like it, which falls short of being a techical argument)?

Greetings,
Rafael


-- 
"Premature optimization is the root of all evil." - Donald Knuth

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Benjamin Herrenschmidt]

On Tue, 2007-07-03 at 23:32 +0200, Rafael J. Wysocki wrote:
> 
> Still, do you really think that we're ready to drop it _right_ _now_
> (I'm
> referring to suspend only) and if so than on what basis (except that
> you
> don't like it, which falls short of being a techical argument)?

Works fine for me without it ;-)

Ben.

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Rafael J. Wysocki]

On Tuesday, 3 July 2007 23:35, Benjamin Herrenschmidt wrote:
> On Tue, 2007-07-03 at 23:32 +0200, Rafael J. Wysocki wrote:
> > 
> > Still, do you really think that we're ready to drop it _right_ _now_
> > (I'm
> > referring to suspend only) and if so than on what basis (except that
> > you
> > don't like it, which falls short of being a techical argument)?
> 
> Works fine for me without it ;-)

Yeah, that makes sense. ;-)

Still, someone needs to take care of bug reports from unlucky people and I
wouldn't like to increase the number of these by doing such a high-level change
without any preparations ...

Greetings,
Rafael


-- 
"Premature optimization is the root of all evil." - Donald Knuth

Re: Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Paul Mackerras]

Rafael J. Wysocki writes:

> Still, do you really think that we're ready to drop it _right_ _now_ (I'm
> referring to suspend only) and if so than on what basis (except that you
> don't like it, which falls short of being a techical argument)?

The basis is that it (the freezer) causes more deadlocks and other
problems than it avoids, so it's a net win to remove it.

Paul.

Re: Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Rafael J. Wysocki]

On Wednesday, 4 July 2007 05:29, Paul Mackerras wrote:
> Rafael J. Wysocki writes:
> 
> > Still, do you really think that we're ready to drop it _right_ _now_ (I'm
> > referring to suspend only) and if so than on what basis (except that you
> > don't like it, which falls short of being a techical argument)?
> 
> The basis is that it (the freezer) causes more deadlocks and other
> problems than it avoids, so it's a net win to remove it.

So, I gather, you're volunteering to handle suspend-related bug reports
from the point in which we drop the freezer from the suspend code path?

Greetings,
Rafael


-- 
"Premature optimization is the root of all evil." - Donald Knuth

Re: Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Paul Mackerras]

Rafael J. Wysocki writes:

> So, I gather, you're volunteering to handle suspend-related bug reports
> from the point in which we drop the freezer from the suspend code path?

Ben and I are happy to handle all the ones for the platform we
maintain, which currently does suspend without freezing processes. :)

Paul.

Re: Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Rafael J. Wysocki]

On Wednesday, 4 July 2007 12:48, Paul Mackerras wrote:
> Rafael J. Wysocki writes:
> 
> > So, I gather, you're volunteering to handle suspend-related bug reports
> > from the point in which we drop the freezer from the suspend code path?
> 
> Ben and I are happy to handle all the ones for the platform we
> maintain, which currently does suspend without freezing processes. :)

I mean all platforms.  After all, the $subject change won't affect yours.

BTW, does your platform's suspend work on SMP systems?

Rafael


-- 
"Premature optimization is the root of all evil." - Donald Knuth

Re: Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Paul Mackerras]

Rafael J. Wysocki writes:

> BTW, does your platform's suspend work on SMP systems?

Yes; currently we require userspace to offline all cpus other than the
boot cpu before initiating the suspend.

The main difficulty is actually that SMP powermacs that can suspend
tend to have video cards that get powered off in suspend.  We know how
to re-initialize one (the Radeon RV100 QW) but not others.  That's an
orthogonal issue to the issues we have been discussing, though.

Paul.

Re: Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Rafael J. Wysocki]

On Wednesday, 4 July 2007 13:24, Paul Mackerras wrote:
> Rafael J. Wysocki writes:
> 
> > BTW, does your platform's suspend work on SMP systems?
> 
> Yes; currently we require userspace to offline all cpus other than the
> boot cpu before initiating the suspend.

This is incompatible with the code in kernel/power/main.c, since we only
disable the nonboot CPUs after devices have been suspended.  Do you think that
your framework can be modified to work without disabling the nonboot CPUs
by the user space?

> The main difficulty is actually that SMP powermacs that can suspend
> tend to have video cards that get powered off in suspend.  We know how
> to re-initialize one (the Radeon RV100 QW) but not others.  That's an
> orthogonal issue to the issues we have been discussing, though.

Sure.

Greetings,
Rafael


-- 
"Premature optimization is the root of all evil." - Donald Knuth

Re: Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Paul Mackerras]

Rafael J. Wysocki writes:

> This is incompatible with the code in kernel/power/main.c, since we only
> disable the nonboot CPUs after devices have been suspended.  Do you think that
> your framework can be modified to work without disabling the nonboot CPUs
> by the user space?

Sure.  It was a "if it can be done in userspace, do it in userspace"
kind of decision, but I'm not wedded to it.

I actually do want to converge to using the generic suspend-to-ram
code on powerbooks.  I just want to avoid causing regressions for
powerbook users, including myself. :)

Paul.

Re: Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Rafael J. Wysocki]

On Thursday, 5 July 2007 02:15, Paul Mackerras wrote:
> Rafael J. Wysocki writes:
> 
> > This is incompatible with the code in kernel/power/main.c, since we only
> > disable the nonboot CPUs after devices have been suspended.  Do you think that
> > your framework can be modified to work without disabling the nonboot CPUs
> > by the user space?
> 
> Sure.  It was a "if it can be done in userspace, do it in userspace"
> kind of decision, but I'm not wedded to it.
> 
> I actually do want to converge to using the generic suspend-to-ram
> code on powerbooks.  I just want to avoid causing regressions for
> powerbook users, including myself. :)

Okay, but my question is this: Would that be possible, within your framework,
to disable the nonboot CPUs _after_ suspending devices?

Can you please point me to your high-level suspend code?

Greetings,
Rafael


-- 
"Premature optimization is the root of all evil." - Donald Knuth

Re: Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Pavel Machek]

On Thu 2007-07-05 10:15:01, Paul Mackerras wrote:
> Rafael J. Wysocki writes:
> 
> > This is incompatible with the code in kernel/power/main.c, since we only
> > disable the nonboot CPUs after devices have been suspended.  Do you think that
> > your framework can be modified to work without disabling the nonboot CPUs
> > by the user space?
> 
> Sure.  It was a "if it can be done in userspace, do it in userspace"
> kind of decision, but I'm not wedded to it.
> 
> I actually do want to converge to using the generic suspend-to-ram
> code on powerbooks.  I just want to avoid causing regressions for
> powerbook users, including myself. :)

Curious, do you actually use fuse? Can you try it _with_ freezer and
produce sysrq-t trace of deadlock?
									Pavel
-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

Re: Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Paul Mackerras]

Rafael J. Wysocki writes:

> > > So, I gather, you're volunteering to handle suspend-related bug reports
> > > from the point in which we drop the freezer from the suspend code path?
> > 
> > Ben and I are happy to handle all the ones for the platform we
> > maintain, which currently does suspend without freezing processes. :)
> 
> I mean all platforms.  After all, the $subject change won't affect yours.

Well, I can't commit to handling all bug reports, but I am happy to
help out where I can...

Paul.

The big suspend mess

[Adrian Bunk]

On Wed, Jul 04, 2007 at 01:29:49PM +1000, Paul Mackerras wrote:
> Rafael J. Wysocki writes:
> 
> > Still, do you really think that we're ready to drop it _right_ _now_ (I'm
> > referring to suspend only) and if so than on what basis (except that you
> > don't like it, which falls short of being a techical argument)?
> 
> The basis is that it (the freezer) causes more deadlocks and other
> problems than it avoids, so it's a net win to remove it.

You forget one important point:
Regressions are much worse than normal bugs.

It's not about the question whether the driver was "correct" or
"buggy" - for a user the important question is whether it worked in 
practice, and if yes, whether it continues to work.

And my impression is that every time someone touches some part of the 
suspend code some drivers break.

During 2.6.21-rc, it wasn't unusual for users to run with one kernel 
into up to three distinct suspend regressions on one machine.

And no, it wasn't always the same set of distinct regressions.

IMHO the suspend code is currently way too much of a moving target which 
results in this mess.

The correct order seems to be:
1. agree on what the suspend code as a whole should look like
2. implement this
3. fix ALL drivers to work at least as good as they do today
4. get it tested in -mm
5. fix all bugs people run into
6. submit it for inclusion in Linus' tree
7. quickly work on the most likely big amount of bug reports

Step 1 is the most important one - evolving code is often something 
good, but in this case with different people trying to evolve the 
suspend code in different directions it simply results in a big mess.

> Paul.

cu
Adrian

-- 

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

Re: The big suspend mess

[Pavel Machek]

> IMHO the suspend code is currently way too much of a moving target which 
> results in this mess.
> 
> The correct order seems to be:

0. Get someone to sign up as a maintainer for suspend, so we have
someone to blame for the mess? :-)

> 1. agree on what the suspend code as a whole should look like
> 2. implement this
> 3. fix ALL drivers to work at least as good as they do today
> 4. get it tested in -mm
> 5. fix all bugs people run into
> 6. submit it for inclusion in Linus' tree
> 7. quickly work on the most likely big amount of bug reports
> 
> Step 1 is the most important one - evolving code is often something 
> good, but in this case with different people trying to evolve the 
> suspend code in different directions it simply results in a big mess.

									Pavel
-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

Re: The big suspend mess

[Paul Mackerras]

Pavel Machek writes:

> 0. Get someone to sign up as a maintainer for suspend, so we have
> someone to blame for the mess? :-)

I thought that was Rafael?

Paul.

Re: The big suspend mess

[Adrian Bunk]

On Thu, Jul 05, 2007 at 02:27:47AM +0200, Pavel Machek wrote:
> 
> > IMHO the suspend code is currently way too much of a moving target which 
> > results in this mess.
> > 
> > The correct order seems to be:
> 
> 0. Get someone to sign up as a maintainer for suspend, so we have
> someone to blame for the mess? :-)

The "SOFTWARE SUSPEND" entry in MAINTAINERS already contains a victim...  ;-)

My impression is that suspend is an area of the kernel that does not 
lack maintainers - you and Rafael are doing a good job, and there's e.g. 
also the maintained code formerly known as suspend2.

But some basic questions like e.g.
- What should be done in the kernel and what in userspace?
- How should this be implemented?
- What must subsystems and drivers do?
- What must subsystems and drivers not do?
seem to be in a constant flux because the big picture everyone agrees 
upon seems to be missing.

> > 1. agree on what the suspend code as a whole should look like
> > 2. implement this
> > 3. fix ALL drivers to work at least as good as they do today
> > 4. get it tested in -mm
> > 5. fix all bugs people run into
> > 6. submit it for inclusion in Linus' tree
> > 7. quickly work on the most likely big amount of bug reports
> > 
> > Step 1 is the most important one - evolving code is often something 
> > good, but in this case with different people trying to evolve the 
> > suspend code in different directions it simply results in a big mess.
> 
> 									Pavel

cu
Adrian

-- 

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

Re: The big suspend mess

[Pavel Machek]

On Thu 2007-07-05 10:53:37, Paul Mackerras wrote:
> Pavel Machek writes:
> 
> > 0. Get someone to sign up as a maintainer for suspend, so we have
> > someone to blame for the mess? :-)
> 
> I thought that was Rafael?

Rafael is good job trying to fix suspend when he can, but we do not
actually have "SUSPEND" entry in MAINTAINERS file.
								Pavel
-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

Re: The big suspend mess

[Gabriel C]

Pavel Machek wrote:
> On Thu 2007-07-05 10:53:37, Paul Mackerras wrote:
>   
>> Pavel Machek writes:
>>
>>     
>>> 0. Get someone to sign up as a maintainer for suspend, so we have
>>> someone to blame for the mess? :-)
>>>       
>> I thought that was Rafael?
>>     
>
> Rafael is good job trying to fix suspend when he can, but we do not
> actually have "SUSPEND" entry in MAINTAINERS file.
> 								Pavel
>   

Huch ?;)

We do :p

...

SOFTWARE SUSPEND:
P: Pavel Machek
M: pavel@suse.cz
L: linux-pm@lists.linux-foundation.org
S: Maintained


...

Re: The big suspend mess

[Fabio Comolli]

Hi.

On 7/5/07, Gabriel C <nix.or.die@googlemail.com> wrote:
> Pavel Machek wrote:
> > On Thu 2007-07-05 10:53:37, Paul Mackerras wrote:
> >
> >> Pavel Machek writes:
> >>
> >>
> >>> 0. Get someone to sign up as a maintainer for suspend, so we have
> >>> someone to blame for the mess? :-)
> >>>
> >> I thought that was Rafael?
> >>
> >
> > Rafael is good job trying to fix suspend when he can, but we do not
> > actually have "SUSPEND" entry in MAINTAINERS file.
> >                                                               Pavel
> >
>
> Huch ?;)
>
> We do :p
>
> ...
>
> SOFTWARE SUSPEND:
> P: Pavel Machek
> M: pavel@suse.cz
> L: linux-pm@lists.linux-foundation.org
> S: Maintained
>

I think he means SUSPEND (aka suspend-to-ram) not "SOFTWARE SUSPEND"
(aka suspend-to-disk)

Re: The big suspend mess

[Rafael J. Wysocki]

On Thursday, 5 July 2007 03:22, Adrian Bunk wrote:
> On Thu, Jul 05, 2007 at 02:27:47AM +0200, Pavel Machek wrote:
> > 
> > > IMHO the suspend code is currently way too much of a moving target which 
> > > results in this mess.
> > > 
> > > The correct order seems to be:
> > 
> > 0. Get someone to sign up as a maintainer for suspend, so we have
> > someone to blame for the mess? :-)
> 
> The "SOFTWARE SUSPEND" entry in MAINTAINERS already contains a victim...  ;-)
> 
> My impression is that suspend is an area of the kernel that does not 
> lack maintainers - you and Rafael are doing a good job, and there's e.g. 
> also the maintained code formerly known as suspend2.
> 
> But some basic questions like e.g.
> - What should be done in the kernel and what in userspace?
> - How should this be implemented?
> - What must subsystems and drivers do?
> - What must subsystems and drivers not do?
> seem to be in a constant flux because the big picture everyone agrees 
> upon seems to be missing.

Well, to some extent I agree, but that's because it's difficult to reach such
an agreement and if we have one, new poeple come with new ideas and we need to
start all over again. :-)

In the meantime, problems are reported that need to be fixed and the fixes
often break other systems and we learn about that too late ect.

As far as the big picture is concerned, we have already agreed, or at least
this is my impression, that we need to separate the drivers' suspend callbacks
(as they exist today) from their hibernation callbacks (not present yet).

This seems to be needed for a couple of reasons, the first of them being that
the desired functionality of the hibernation code may be substantially
different from the desired functionality of the suspend code.  For instance,
it generally is not necessary to put devices into low power states, or to power
them off, in order to create the hibernation image, while it is necessary to do
that for a suspend.

It will require us to redesign the core quite a bit and I've already started to
prepare for that.  I really would like to do that in the first place and _then_
to think of improving both the suspend and hibernation code paths
_separately_.  This way, we can do things more easily and in a much more clean
way.

Greetings,
Rafael


-- 
"Premature optimization is the root of all evil." - Donald Knuth

Re: The big suspend mess

[Alan Stern]

On Thu, 5 Jul 2007, Adrian Bunk wrote:

> IMHO the suspend code is currently way too much of a moving target which 
> results in this mess.
> 
> The correct order seems to be:
> 1. agree on what the suspend code as a whole should look like
> 2. implement this
> 3. fix ALL drivers to work at least as good as they do today
> 4. get it tested in -mm
> 5. fix all bugs people run into
> 6. submit it for inclusion in Linus' tree
> 7. quickly work on the most likely big amount of bug reports
> 
> Step 1 is the most important one - evolving code is often something 
> good, but in this case with different people trying to evolve the 
> suspend code in different directions it simply results in a big mess.

Isn't Step 1 exactly what we are in the midst of right now?

Alan Stern

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Pavel Machek]

Hi!

> > >  - For STR, don't do the freezer thing.
> > 
> > In the long run, I agree.
> > 
> > Still, can you please read this post from Alan Stern:
> > 
> > https://lists.linux-foundation.org/pipermail/linux-pm/2007-June/012847.html
> > 
> > ?  I don't think I'm able to repeat the arguments given in there in a
> > convincing way.
> 
> That's the same crackpot I've been hearing for the past 3 years or
> so ...
> 
> Both Paulus and I think the freezer is just a way to try to put your
> head in the sand and ignore the problem. It causes as many problems as
> it solves on its own, and is just not a solution that will be of any use
> once you start implementing dynamic PM schemes etc...

Well, yep, you can view freezer as a head in the sand...

> In many cases, having proper support for "live" suspend of devices is
> just a matter of having a couple of helpers in whatever subsystem those
> drivers hookup with. In the case of network, for example, it's mostly
> trivial (stop the queue). For block, it's not terribly hard neither,
> though you want to have some orderign/atomicity between the blocking of
> the incoming request queue and the sending of things like spindown &
> flush commands to the disk. For old-style IDE, that was fairly easily
> solved by piping suspend/resume command down the request queue itself
> and have the queue block/unblbock itself after processing them. Some of
> that logic could maybe be moved to the block layer for all block drivers
> to benefit.

...but the moment you start blocking tasks that done driver request,
you _do_ have mini-freezer of your own, with pretty much the same
problems.

In another message I shown that removing freezer will not help with
FUSE in general case.

It probably does not help with firmware, too; as soon as udev attempts
to do something with your wireless card, it is blocked, and if the
wireless card needs the firmware from udev, you are deadlocked.
									Pavel
-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Benjamin Herrenschmidt]

On Thu, 2007-07-05 at 11:30 +0200, Pavel Machek wrote:
> 
> ...but the moment you start blocking tasks that done driver request,
> you _do_ have mini-freezer of your own, with pretty much the same
> problems.

No, not at all the same problems. Those tasks will block, but that will
be harmless because we won't have some "freezer" things waiting for all
tasks to reach a "stable" point (calling try_to_freeze()). We just let
them block wherever we want, as long as it doesn't prevent a -driver-
from suspending, which should be allright, we have no problem.

> In another message I shown that removing freezer will not help with
> FUSE in general case.

I disagree.

> It probably does not help with firmware, too; as soon as udev attempts
> to do something with your wireless card, it is blocked, and if the
> wireless card needs the firmware from udev, you are deadlocked.

Firmware load has been a problem since day 1, I've talked about it
multiple times, it's broken with or without the freezer, and so far, the
reaction of pretty much everybody has been to dig their head deeper in
the mud and ignore the problem.

There are other issues (again, with or without freezer) that should be
dealt with. For example, drivers that haven't yet got their suspend()
callback or already have got their resume() may rely on services of the
kernel that are still blocked, that's where things may go hairy.
request_firmware() within resume() is a typical example of that.

There are a few things we should do in that area. For example, once we
start to call driver suspend's, we should probably set a system wide
flag that will do things such as:

 - block usermode helpers (either make call_usermodehelper return
something like -EBUSY or have it queue up the calls and issue them later
when thing are resuming, we need to look closely at what semantic we
want here).

 - Silently add GFP_NOIO to all allocations, to avoid having things
blocking in kmalloc() with a mutex held that will deadlock with
suspend() in a driver for example. Or set some way to have all GFP
waiters wakeup and fail rather than wait for IOs. It's hard/bizarre but
necessary, again, with or without a freezer.

 - Deal with the firmware problem. The best way is probably to have an
async request_firmware interface(). Another thing is, drivers may want
to cache their firmware in main memory, that sort of thing...

And that's just a small list off the top of my mind, of known problems
that will cause deadlocks or misbehaviours today, with or without the
freezer, and that need to be addressed.

Ben.

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Nigel Cunningham]

[-- Attachment #1.1: Type: text/plain, Size: 3426 bytes --]

Hi.

On Friday 06 July 2007 08:46:54 Benjamin Herrenschmidt wrote:
> On Thu, 2007-07-05 at 11:30 +0200, Pavel Machek wrote:
> > 
> > ...but the moment you start blocking tasks that done driver request,
> > you _do_ have mini-freezer of your own, with pretty much the same
> > problems.
> 
> No, not at all the same problems. Those tasks will block, but that will
> be harmless because we won't have some "freezer" things waiting for all
> tasks to reach a "stable" point (calling try_to_freeze()). We just let
> them block wherever we want, as long as it doesn't prevent a -driver-
> from suspending, which should be allright, we have no problem.

Will you be able to guarantee that every place where a task can/will block 
will be harmless place? If so, how will you guarantee that? How will you 
debug issues where a task occasionally doesn't block in the right place, 
particularly instances where it is some less than obvious interaction with 
other tasks?

This is the whole point to having the freezer. It makes things more 
predictable and testable. It shows us, clearly, when process X is the one 
that is causing problems.

> > In another message I shown that removing freezer will not help with
> > FUSE in general case.
> 
> I disagree.

Why?
 
> > It probably does not help with firmware, too; as soon as udev attempts
> > to do something with your wireless card, it is blocked, and if the
> > wireless card needs the firmware from udev, you are deadlocked.
> 
> Firmware load has been a problem since day 1, I've talked about it
> multiple times, it's broken with or without the freezer, and so far, the
> reaction of pretty much everybody has been to dig their head deeper in
> the mud and ignore the problem.
> 
> There are other issues (again, with or without freezer) that should be
> dealt with. For example, drivers that haven't yet got their suspend()
> callback or already have got their resume() may rely on services of the
> kernel that are still blocked, that's where things may go hairy.
> request_firmware() within resume() is a typical example of that.
> 
> There are a few things we should do in that area. For example, once we
> start to call driver suspend's, we should probably set a system wide
> flag that will do things such as:
> 
>  - block usermode helpers (either make call_usermodehelper return
> something like -EBUSY or have it queue up the calls and issue them later
> when thing are resuming, we need to look closely at what semantic we
> want here).
> 
>  - Silently add GFP_NOIO to all allocations, to avoid having things
> blocking in kmalloc() with a mutex held that will deadlock with
> suspend() in a driver for example. Or set some way to have all GFP
> waiters wakeup and fail rather than wait for IOs. It's hard/bizarre but
> necessary, again, with or without a freezer.

GFP_ATOMIC? (In driver suspend, they shouldn't be sleeping either, right?)
 
>  - Deal with the firmware problem. The best way is probably to have an
> async request_firmware interface(). Another thing is, drivers may want
> to cache their firmware in main memory, that sort of thing...
> 
> And that's just a small list off the top of my mind, of known problems
> that will cause deadlocks or misbehaviours today, with or without the
> freezer, and that need to be addressed.

Userspace device drivers too?

Regards,

Nigel

[-- Attachment #1.2: Type: application/pgp-signature, Size: 189 bytes --]

[-- Attachment #2: Type: text/plain, Size: 0 bytes --]

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Benjamin Herrenschmidt]

> Will you be able to guarantee that every place where a task can/will block 
> will be harmless place? If so, how will you guarantee that? How will you 
> debug issues where a task occasionally doesn't block in the right place, 
> particularly instances where it is some less than obvious interaction with 
> other tasks?

Which places aren't harmless if you don't have a freezer ?

> This is the whole point to having the freezer. It makes things more 
> predictable and testable. It shows us, clearly, when process X is the one 
> that is causing problems.

No, the freezer creates all those places what are harmful for a task to
block because they will break the freezer :-)

> >  - Silently add GFP_NOIO to all allocations, to avoid having things
> > blocking in kmalloc() with a mutex held that will deadlock with
> > suspend() in a driver for example. Or set some way to have all GFP
> > waiters wakeup and fail rather than wait for IOs. It's hard/bizarre but
> > necessary, again, with or without a freezer.
> 
> GFP_ATOMIC? (In driver suspend, they shouldn't be sleeping either, right?)

NOIO should be enough I think but ATOMIC would do).
 
That's one of the reason why I used to have the pre-suspend and
post-resume hooks in my original powermac implementation, for those few
drivers complicated enough to require some pre-allocations.
 
> >  - Deal with the firmware problem. The best way is probably to have an
> > async request_firmware interface(). Another thing is, drivers may want
> > to cache their firmware in main memory, that sort of thing...
> >

Note that the above firmware problem could be dealt with also with the
pre-suspend/post-resume. Allowing to pre-request firmware etc... and
keep it around until after resume, because we know we will need it.
Gives a chance to drivers to perform things while the system is still
live, filesystems still working, etc... (big memory allocations for
example).

> > And that's just a small list off the top of my mind, of known problems
> > that will cause deadlocks or misbehaviours today, with or without the
> > freezer, and that need to be addressed.
> 
> Userspace device drivers too?

Maybe but they are less of an issue, most of the time, they don't do DMA
or whatever harmful things. If they are USB drivers, for example, they
are an non-issues at that level.

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Nigel Cunningham]

[-- Attachment #1.1: Type: text/plain, Size: 3217 bytes --]

Hi.

On Friday 06 July 2007 09:20:43 Benjamin Herrenschmidt wrote:
> 
> > Will you be able to guarantee that every place where a task can/will block 
> > will be harmless place? If so, how will you guarantee that? How will you 
> > debug issues where a task occasionally doesn't block in the right place, 
> > particularly instances where it is some less than obvious interaction with 
> > other tasks?
> 
> Which places aren't harmless if you don't have a freezer ?

If I knew that, I wouldn't be asking the question.

> > This is the whole point to having the freezer. It makes things more 
> > predictable and testable. It shows us, clearly, when process X is the one 
> > that is causing problems.
> 
> No, the freezer creates all those places what are harmful for a task to
> block because they will break the freezer :-)

Nice try :) Okay then, you remove the freezer, try hibernating, then get back 
to me after you've fixed your filesystem because some process that wasn't 
frozen started writing things after the atomic copy (making the on disk 
filesystem inconsistent with the snapshot).

As Pavel rightly said, you can get rid of the freezer, but you're only going 
to have to implement another one that does the essentially the same thing, 
even if it is at some other level.
 
> > >  - Silently add GFP_NOIO to all allocations, to avoid having things
> > > blocking in kmalloc() with a mutex held that will deadlock with
> > > suspend() in a driver for example. Or set some way to have all GFP
> > > waiters wakeup and fail rather than wait for IOs. It's hard/bizarre but
> > > necessary, again, with or without a freezer.
> > 
> > GFP_ATOMIC? (In driver suspend, they shouldn't be sleeping either, right?)
> 
> NOIO should be enough I think but ATOMIC would do).
>  
> That's one of the reason why I used to have the pre-suspend and
> post-resume hooks in my original powermac implementation, for those few
> drivers complicated enough to require some pre-allocations.
>  
> > >  - Deal with the firmware problem. The best way is probably to have an
> > > async request_firmware interface(). Another thing is, drivers may want
> > > to cache their firmware in main memory, that sort of thing...
> > >
> 
> Note that the above firmware problem could be dealt with also with the
> pre-suspend/post-resume. Allowing to pre-request firmware etc... and
> keep it around until after resume, because we know we will need it.
> Gives a chance to drivers to perform things while the system is still
> live, filesystems still working, etc... (big memory allocations for
> example).
> 
> > > And that's just a small list off the top of my mind, of known problems
> > > that will cause deadlocks or misbehaviours today, with or without the
> > > freezer, and that need to be addressed.
> > 
> > Userspace device drivers too?
> 
> Maybe but they are less of an issue, most of the time, they don't do DMA
> or whatever harmful things. If they are USB drivers, for example, they
> are an non-issues at that level.

(Leaving the rest of the message intact so we don't have to fragment the 
discussion into a million subthreads).

Regards,

Nigel

[-- Attachment #1.2: Type: application/pgp-signature, Size: 189 bytes --]

[-- Attachment #2: Type: text/plain, Size: 0 bytes --]

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Kyle Moffett]

On Jul 05, 2007, at 19:35:11, Nigel Cunningham wrote:
> On Friday 06 July 2007 09:20:43 Benjamin Herrenschmidt wrote:
>> No, the freezer creates all those places what are harmful for a  
>> task to block because they will break the freezer :-)
>
> Nice try :) Okay then, you remove the freezer, try hibernating,  
> then get back to me after you've fixed your filesystem because some  
> process that wasn't frozen started writing things after the atomic  
> copy (making the on disk filesystem inconsistent with the snapshot).

Umm, this thread is NOT ABOUT HIBERNATING!!!  Please go back and read  
the subject, specifically the "suspend to RAM" parts :-D.  When your  
hardware can put itself to sleep and atomically preserve memory as it  
does so, you don't need an atomic copy.  For Real Suspend(TM) (IE:  
Suspend-to-RAM), the list of things to do is short and simple:

1)  Stop DMA and put most hardware into low-power states (stops all  
interrupt sources)
2)  Ensure that the other CPUs have finished any trailing interrupt  
handlers and put them to sleep
3)  Put the interrupt-controllers into low-power state
4)  Go to sleep

> As Pavel rightly said, you can get rid of the freezer, but you're  
> only going to have to implement another one that does the  
> essentially the same thing, even if it is at some other level.

How about a freezer whose job it is to "wait for pending hard  
interrupts to complete when we have already guaranteed that we won't  
get any more"?  That part should be really *REALLY* easy.  You don't  
need to care about either userspace processes or kernel threads at  
all.  Specifically, Step 1 consists of:

suspend_device(dev)
{
	set_no_bind_flag(dev);
	for (dev->subdevices)
		suspend_device(dev);
	set_no_io_flag(dev);
	wait_for_in_progress_dma(dev);
	turn_off_interrupts(dev);
	go_to_low_power_state(dev);
}

After you've set the "no_bind" flag, you won't get any *new*  
subdevices trying to bind, therefore it's safe to iterate over the  
list of present sub-devices and suspend them.  Once those are  
suspended and in low-power states you can set a "no_io" flag to  
prevent the driver from submitting more IO.  At that point you can  
lazily wait for existing DMA/IO/interrupts to finish on the device,  
since *NOBODY* will be submitting them anymore, and we certainly  
aren't probing for new devices.  Then you can just turn off the power  
to the device.  When all the leaf devices are off, the parent device  
can be turned off because everything waiting on the leaf devices is  
blocked on them and won't unblock until the parent device *AND* the  
leaf device are turned on again, in that order.

Scheduling and userspace are all still fully enabled in this  
scenario.  Once all your devices are turned off, the only remaining  
running threads will be those which haven't done IO since the  
beginning of the suspend.  We can then disable preemption, turn off  
the timer interrupts, and tell the other CPUs to park all their  
remaining threads in schedule() and sleep.  Then we put the IRQ  
controller to sleep and go to sleep ourselves.  If our driver model  
locking is sufficient to handle putting a parent device to sleep  
while threads are sleeping on a child device then there are exactly 0  
problems.

Resuming is basically running the whole process in reverse.  Runtime- 
suspend is achieved by not setting the 'no_io' or 'no_bind' flags and  
putting selective device-subtrees to sleep without doing anything to  
the rest of the system.

Cheers,
Kyle Moffett

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Benjamin Herrenschmidt]

On Fri, 2007-07-06 at 09:35 +1000, Nigel Cunningham wrote:
> 
> Nice try :) Okay then, you remove the freezer, try hibernating, then get back 
> to me after you've fixed your filesystem because some process that wasn't 
> frozen started writing things after the atomic copy (making the on disk 
> filesystem inconsistent with the snapshot).
> 
> As Pavel rightly said, you can get rid of the freezer, but you're only going 
> to have to implement another one that does the essentially the same thing, 
> even if it is at some other level.

I was mostly talking about STR... Regarding STD, we have a different
problem and we all know it. The freezer is one somewhat horrible way to
get it working for now, I would prefer something more along the way that
blocks the page cache from writing out new dirty pages though, except
those specifically flagged by the snapshot.

That is, some kind of proper snapshotting facility, as linus was
describing some time ago.

Ben.

> > > >  - Silently add GFP_NOIO to all allocations, to avoid having things
> > > > blocking in kmalloc() with a mutex held that will deadlock with
> > > > suspend() in a driver for example. Or set some way to have all GFP
> > > > waiters wakeup and fail rather than wait for IOs. It's hard/bizarre but
> > > > necessary, again, with or without a freezer.
> > > 
> > > GFP_ATOMIC? (In driver suspend, they shouldn't be sleeping either, right?)
> > 
> > NOIO should be enough I think but ATOMIC would do).
> >  
> > That's one of the reason why I used to have the pre-suspend and
> > post-resume hooks in my original powermac implementation, for those few
> > drivers complicated enough to require some pre-allocations.
> >  
> > > >  - Deal with the firmware problem. The best way is probably to have an
> > > > async request_firmware interface(). Another thing is, drivers may want
> > > > to cache their firmware in main memory, that sort of thing...
> > > >
> > 
> > Note that the above firmware problem could be dealt with also with the
> > pre-suspend/post-resume. Allowing to pre-request firmware etc... and
> > keep it around until after resume, because we know we will need it.
> > Gives a chance to drivers to perform things while the system is still
> > live, filesystems still working, etc... (big memory allocations for
> > example).
> > 
> > > > And that's just a small list off the top of my mind, of known problems
> > > > that will cause deadlocks or misbehaviours today, with or without the
> > > > freezer, and that need to be addressed.
> > > 
> > > Userspace device drivers too?
> > 
> > Maybe but they are less of an issue, most of the time, they don't do DMA
> > or whatever harmful things. If they are USB drivers, for example, they
> > are an non-issues at that level.
> 
> (Leaving the rest of the message intact so we don't have to fragment the 
> discussion into a million subthreads).
> 
> Regards,
> 
> Nigel
> 
> 
> Invalid signature
> 
> 

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Nigel Cunningham]

[-- Attachment #1.1: Type: text/plain, Size: 2087 bytes --]

Hi.

On Friday 06 July 2007 13:54:15 Benjamin Herrenschmidt wrote:
> On Fri, 2007-07-06 at 09:35 +1000, Nigel Cunningham wrote:
> > 
> > Nice try :) Okay then, you remove the freezer, try hibernating, then get 
back 
> > to me after you've fixed your filesystem because some process that wasn't 
> > frozen started writing things after the atomic copy (making the on disk 
> > filesystem inconsistent with the snapshot).
> > 
> > As Pavel rightly said, you can get rid of the freezer, but you're only 
going 
> > to have to implement another one that does the essentially the same thing, 
> > even if it is at some other level.
> 
> I was mostly talking about STR... Regarding STD, we have a different
> problem and we all know it. The freezer is one somewhat horrible way to
> get it working for now, I would prefer something more along the way that
> blocks the page cache from writing out new dirty pages though, except
> those specifically flagged by the snapshot.
> 
> That is, some kind of proper snapshotting facility, as linus was
> describing some time ago.

The kind of thing Linus was talking about would limit you (as swsusp and 
uswsusp do now) to only half the amount of memory. I suppose you could lzf 
compress as you did the snapshot. That would generally get you up to 2/3rds, 
but then again you can't know what compression ratio you'll get until you 
try, so reliability would suffer or it would take longer because of retrying.

I/O from swsusp and suspend2 use bios directly, so the page cache isn't an 
issue for them (apart from the fact that Suspend2 saves the page cache 
separately so it can get a full image). Not sure about uswsusp.

Only having half the amount of memory doesn't sound like a big limitation for 
modern desktops & laptops, but don't forget that there are embedded guys 
wanting to hbernate too :)

Regards,

Nigel
-- 
Nigel Cunningham
Christian Reformed Church of Cobden
103 Curdie Street, Cobden 3266, Victoria, Australia
Ph. +61 3 5595 1185 / +61 417 100 574
Communal Worship: 11 am Sunday.

[-- Attachment #1.2: Type: application/pgp-signature, Size: 189 bytes --]

[-- Attachment #2: Type: text/plain, Size: 0 bytes --]

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Benjamin Herrenschmidt]

> I/O from swsusp and suspend2 use bios directly, so the page cache isn't an 
> issue for them (apart from the fact that Suspend2 saves the page cache 
> separately so it can get a full image). Not sure about uswsusp.
> 
> Only having half the amount of memory doesn't sound like a big limitation for 
> modern desktops & laptops, but don't forget that there are embedded guys 
> wanting to hbernate too :)

Wait wait wait ... uses the BIOS ? what do you mean ?

I know that for example, things like MacOS X use a separate polled path to
the storage driver for suspend (works fine for the built-in IDE, but more
complicated in large scale). If you can use BIOS calls to write your
suspend image, that is, if you don't need any of the normal block
infrastructure, then you don't need a freezer ! not at all !

You just do like STR ... and at the end of the day, once you have stopped
all your driver, you shut interrupts off and do the BIOS thing....

I fail to see how processes could dirty pages while/after the BIOS thingy :-)

But then, the problem with that approach is that of course, you need a BIOS
capable of doing that (or a special sideband path to the "blessed" block
driver that will be used for suspend ... not necessarily a hard thing to do,
would be trivial to add support to drivers/ide or libata for that sort of
things I suppose).

Ben.

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Nigel Cunningham]

[-- Attachment #1.1: Type: text/plain, Size: 754 bytes --]

Hi.

On Friday 06 July 2007 14:41:40 Benjamin Herrenschmidt wrote:
> 
> > I/O from swsusp and suspend2 use bios directly, so the page cache isn't an 
> > issue for them (apart from the fact that Suspend2 saves the page cache 
> > separately so it can get a full image). Not sure about uswsusp.
> > 
> > Only having half the amount of memory doesn't sound like a big limitation 
for 
> > modern desktops & laptops, but don't forget that there are embedded guys 
> > wanting to hbernate too :)
> 
> Wait wait wait ... uses the BIOS ? what do you mean ?

You misread me, Ben. Sorry for not being clearer. bios as in struct bio.

Regards,

Nigel
-- 
See http://www.tuxonice.net for Howtos, FAQs, mailing
lists, wiki and bugzilla info.

[-- Attachment #1.2: Type: application/pgp-signature, Size: 189 bytes --]

[-- Attachment #2: Type: text/plain, Size: 0 bytes --]

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Kyle Moffett]

On Jul 06, 2007, at 00:03:15, Nigel Cunningham wrote:
> On Friday 06 July 2007 13:54:15 Benjamin Herrenschmidt wrote:
>> On Fri, 2007-07-06 at 09:35 +1000, Nigel Cunningham wrote:
>>>
>>> Nice try :) Okay then, you remove the freezer, try hibernating,  
>>> then get back to me after you've fixed your filesystem because  
>>> some process that wasn't frozen started writing things after the  
>>> atomic copy (making the on disk filesystem inconsistent with the  
>>> snapshot).
>>>
>>> As Pavel rightly said, you can get rid of the freezer, but you're  
>>> only going to have to implement another one that does the  
>>> essentially the same thing, even if it is at some other level.
>>
>> I was mostly talking about STR... Regarding STD, we have a  
>> different problem and we all know it. The freezer is one somewhat  
>> horrible way to get it working for now, I would prefer something  
>> more along the way that blocks the page cache from writing out new  
>> dirty pages though, except those specifically flagged by the  
>> snapshot.
>>
>> That is, some kind of proper snapshotting facility, as linus was  
>> describing some time ago.
>
> The kind of thing Linus was talking about would limit you (as  
> swsusp and uswsusp do now) to only half the amount of memory.

How so?  Suppose hibernate is implemented like this:

(1) Userspace program calls sys_freeze_processes()
   (a) Pokes all CPUs with IPMIs and tells them to finish the  
currently running timeslot then stop
   (b) Atomically sends SIGSTOP to all userspace processes in a non- 
trappable way, except the calling process and any process which is  
ptracing it.
   (c) Returns to the calling process.

(2) Userspace process sends SIGCONT to only those processes which are  
necessary for sync and a device-mapper snapshot.

(3) Userspace calls sys_snapshot_kernel(snapshot_overhead_pages)
   (a) Kernel starts freeing memory and swapping stuff out to make  
room for a copy of *kernel* memory (not pagecache, not process RAM).   
It does the same for at least snapshot_overhead_pages extra (used by  
userspace later).  It then allocates this memory to keep it from  
going away.  Since most processes are stopped we won't have much else  
competing with us for the RAM.
   (a) Kernel uses the device-mapper up-call-into-filesystem  
machinery to get all mounted filesystems synced and ready for a DM  
snapshot.  This may include sending data via the userspace processes  
resumed in (2).  Any deadlocks here are userspace's fault (see (2)).   
Will need some modification to handle doing multiple blockdevs at a  
time.  Anything using FUSE is basically perma-synced anyways (no dep- 
handling needed), and anything using loop should already be handled  
by DM.  This includes allocating memory for the basic snapshot  
datastructures.
   (b) At this point all blockdev operations should be halted and  
disk caches flushed; that's all we care about.
   (c) Go through the device tree and quiesce DMA and shut off  
interrupts.  Since all the disks are synced this is easy.
   (d) Use IPMIs again to get all the CPUs together, which should be  
easy as most processes are sleeping in IO or SIGSTOPed, and we're  
getting no interrupts.
   (e) One CPU turns off all interrupts on itself and takes an atomic  
snapshot of kernel memory into the previously allocated storage.   
Once again, does not include pagecache.  The kernel also records a  
list of what pages *are* included in the pagecache.  It then marks  
all userspace pages as copy-on-write.
   (f) That CPU finalizes the modified DM snapshot using the  
previously-allocated memory.
   (g) That CPU frees up the snapshot_overhead_pages memory allocated  
during step (a) for userspace to use.
   (h) The CPU does the equivalent of a "swapoff -a" without  
overwriting any data already on any swap device(s).
   (i) The CPU then IPMI-signals the other CPUs to wake them up
   (j) The kernel returns a FD-reference to the snapshot and the read- 
only halves of the CoW pagecache to the process which called  
sys_snapshot_kernel().

(4) The userspace process now has a reference to the copy of the  
kernel pages and the unmodified pagecache pages.  Since 99% of the  
processes aren't running, we aren't going to be having to CoW many of  
the pagecache pages.

(5) The userspace process uses read() or other syscalls to get data  
out of the kernel-snapshot FD in small chunks, within its  
snapshot_overhead_pages limit.  It compresses these and writes them  
out to the snapshot-storage blockdev (must not be mounted during  
snapshot), or to any network server.

(6) The userspace process syncs the disks and halts the system.  Any  
changed filesystem pages after the pseudo-DM-snapshot should have  
been stored in semi-volatile storage somewhere and will be discarded  
on the next reboot.

So basically your hibernate-overhead would consist of:
   (1) The pages necessary for the atomic snapshot of kernel memory  
and the list of pagecache pages at that time
   (2) A little memory necessary for the kernel non-persistent DM  
snapshot datastructures.
   (3) The snapshot_overhead_pages needed by userspace.

If you're using swap devices then you can save 99% of the state of  
the running kernel with an initial swapout overhead of virtually  
nothing beyond the size of the unswappable kernel memory.

Cheers,
Kyle Moffett

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Nigel Cunningham]

[-- Attachment #1.1: Type: text/plain, Size: 762 bytes --]

Hi Kyle.

On Friday 06 July 2007 15:01:48 Kyle Moffett wrote:
> On Jul 06, 2007, at 00:03:15, Nigel Cunningham wrote:
> > The kind of thing Linus was talking about would limit you (as  
> > swsusp and uswsusp do now) to only half the amount of memory.
> 
> How so?  Suppose hibernate is implemented like this:

You're not talking about the same thing Linus was suggesting. He was just 
wanting a result = sys_snapshot() sort of call. That would limit us to half 
the amount of memory.

I've looked over what you've written below and want to consider it in detail. 
Right now though, I don't have the time. I'll try to get back to you 
promptly.

Nigel
-- 
See http://www.tuxonice.net for Howtos, FAQs, mailing
lists, wiki and bugzilla info.

[-- Attachment #1.2: Type: application/pgp-signature, Size: 189 bytes --]

[-- Attachment #2: Type: text/plain, Size: 0 bytes --]

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Nigel Cunningham]

[-- Attachment #1.1: Type: text/plain, Size: 9010 bytes --]

Hi.

Sorry for the long delay. Busy weekend and my motivation for working on 
programming is almost zero at the moment...

On Friday 06 July 2007 15:01:48 Kyle Moffett wrote:
> On Jul 06, 2007, at 00:03:15, Nigel Cunningham wrote:
> > On Friday 06 July 2007 13:54:15 Benjamin Herrenschmidt wrote:
> >> On Fri, 2007-07-06 at 09:35 +1000, Nigel Cunningham wrote:
> >>>
> >>> Nice try :) Okay then, you remove the freezer, try hibernating,  
> >>> then get back to me after you've fixed your filesystem because  
> >>> some process that wasn't frozen started writing things after the  
> >>> atomic copy (making the on disk filesystem inconsistent with the  
> >>> snapshot).
> >>>
> >>> As Pavel rightly said, you can get rid of the freezer, but you're  
> >>> only going to have to implement another one that does the  
> >>> essentially the same thing, even if it is at some other level.
> >>
> >> I was mostly talking about STR... Regarding STD, we have a  
> >> different problem and we all know it. The freezer is one somewhat  
> >> horrible way to get it working for now, I would prefer something  
> >> more along the way that blocks the page cache from writing out new  
> >> dirty pages though, except those specifically flagged by the  
> >> snapshot.
> >>
> >> That is, some kind of proper snapshotting facility, as linus was  
> >> describing some time ago.
> >
> > The kind of thing Linus was talking about would limit you (as  
> > swsusp and uswsusp do now) to only half the amount of memory.
> 
> How so?  Suppose hibernate is implemented like this:
> 
> (1) Userspace program calls sys_freeze_processes()
>    (a) Pokes all CPUs with IPMIs and tells them to finish the  
> currently running timeslot then stop
>    (b) Atomically sends SIGSTOP to all userspace processes in a non- 
> trappable way, except the calling process and any process which is  
> ptracing it.
>    (c) Returns to the calling process.

Ok. First, I'll ignore the specification that userspace does this - I don't 
think it matters whether it's userspace or kernel that does the suspending 
and I'm yet to see a good reason for it to be [required to be] done from 
userspace.

In this first step, you've reinvented the first part of the current freezer 
implementation. The reason we don't use a real signal is precisely so we can 
have an untrappable SIGSTOP. In this regard, I particularly remember Win4Lin 
from a few years ago. It would die if you sent it a real signal, so we had to 
do it this way. No doubt there are other instances I'm not aware of.
 
> (2) Userspace process sends SIGCONT to only those processes which are  
> necessary for sync and a device-mapper snapshot.

How do you determine which ones are needed? Why stop them in the first place?
 
> (3) Userspace calls sys_snapshot_kernel(snapshot_overhead_pages)
>    (a) Kernel starts freeing memory and swapping stuff out to make  
> room for a copy of *kernel* memory (not pagecache, not process RAM).   
> It does the same for at least snapshot_overhead_pages extra (used by  
> userspace later).  It then allocates this memory to keep it from  
> going away.  Since most processes are stopped we won't have much else  
> competing with us for the RAM.

Ok. So now you also need processes running that are needed for swapping, 
because freeing that memory might involve swapping. Fully agree with the 
logic though (not really surprising - this is what I do in 
Suspend2^wTuxOnIce).

>    (a) Kernel uses the device-mapper up-call-into-filesystem  
> machinery to get all mounted filesystems synced and ready for a DM  
> snapshot.  This may include sending data via the userspace processes  
> resumed in (2).  Any deadlocks here are userspace's fault (see (2)).   
> Will need some modification to handle doing multiple blockdevs at a  
> time.  Anything using FUSE is basically perma-synced anyways (no dep- 
> handling needed), and anything using loop should already be handled  
> by DM.  This includes allocating memory for the basic snapshot  
> datastructures.
>    (b) At this point all blockdev operations should be halted and  
> disk caches flushed; that's all we care about.
>    (c) Go through the device tree and quiesce DMA and shut off  
> interrupts.  Since all the disks are synced this is easy.
>    (d) Use IPMIs again to get all the CPUs together, which should be  
> easy as most processes are sleeping in IO or SIGSTOPed, and we're  
> getting no interrupts.
>    (e) One CPU turns off all interrupts on itself and takes an atomic  
> snapshot of kernel memory into the previously allocated storage.   
> Once again, does not include pagecache.  The kernel also records a  
> list of what pages *are* included in the pagecache.  It then marks  
> all userspace pages as copy-on-write.

Hotplugging cpus (when all those locking issues are taken care of) is simpler. 
Prior to cpu hotplugging, I used IMPIs to put secondary cpus into a tight 
loop, so I know it's possible to do it this way too. That way, though, you 
have less flexibility. What if a cpu really is plugged in between hibernate 
and resume? With cpu hotplugging, it's handled properly and transparently. 
Without cpu hotplugging, you could be using uninitialised data after the 
atomic restore.

Marking userspace as COW makes things more complicated, too. You then have to 
add code to the COW handling to update the list of pages that need to be 
saved, and you reduce the reliability of the whole process. You can't predict 
beforehand how many of these COW pages are going to be needed, and therefore 
can't know how much memory to free earlier on in the process. If you run out 
of memory, what will be the effect?

>    (f) That CPU finalizes the modified DM snapshot using the  
> previously-allocated memory.
>    (g) That CPU frees up the snapshot_overhead_pages memory allocated  
> during step (a) for userspace to use.
>    (h) The CPU does the equivalent of a "swapoff -a" without  
> overwriting any data already on any swap device(s).

You still need to remember what swap you're going to use to write the image. 
You'll probably want to get this information (and allocate the swap) sooner 
rather than later so that you're not racing against the memory freeing 
earlier, and don't run into issues with bmapping the pages or having enough 
memory to record the bdevs & sector numbers (not usually an issue, but if 
swap is highly fragmented...).

>    (i) The CPU then IPMI-signals the other CPUs to wake them up
>    (j) The kernel returns a FD-reference to the snapshot and the read- 
> only halves of the CoW pagecache to the process which called  
> sys_snapshot_kernel().

Readonly halves? I don't get that, sorry.
 
> (4) The userspace process now has a reference to the copy of the  
> kernel pages and the unmodified pagecache pages.  Since 99% of the  
> processes aren't running, we aren't going to be having to CoW many of  
> the pagecache pages.

Mmm, but you still don't know how many.
 
> (5) The userspace process uses read() or other syscalls to get data  
> out of the kernel-snapshot FD in small chunks, within its  
> snapshot_overhead_pages limit.  It compresses these and writes them  
> out to the snapshot-storage blockdev (must not be mounted during  
> snapshot), or to any network server.
> 
> (6) The userspace process syncs the disks and halts the system.  Any  
> changed filesystem pages after the pseudo-DM-snapshot should have  
> been stored in semi-volatile storage somewhere and will be discarded  
> on the next reboot.

Are you thinking the changed filesystem pages are caught by COW? (AFAIUI, 
kernel writes aren't). If (as I expect), you're thinking about filesystem 
writes to DM based storage, what about non DM-based filesystem pages?
 
> So basically your hibernate-overhead would consist of:
>    (1) The pages necessary for the atomic snapshot of kernel memory  
> and the list of pagecache pages at that time
>    (2) A little memory necessary for the kernel non-persistent DM  
> snapshot datastructures.
>    (3) The snapshot_overhead_pages needed by userspace.
> 
> If you're using swap devices then you can save 99% of the state of  
> the running kernel with an initial swapout overhead of virtually  
> nothing beyond the size of the unswappable kernel memory.

FWIW, let me note an important variation from how Suspend2 works; it might 
provide food for thought. In Suspend2, we treat the processes that remain 
stopped throughout the whole process specially. We write their data to disk 
before the atomic copy (usually 70 or 80% of memory), and then use the memory 
they occupy for the destination of the atomic copy. This further reduces the 
amount of memory that has to be freed, almost always to zero.

Regards,

Nigel
-- 
See http://www.tuxonice.net for Howtos, FAQs, mailing
lists, wiki and bugzilla info.

[-- Attachment #1.2: Type: application/pgp-signature, Size: 189 bytes --]

[-- Attachment #2: Type: text/plain, Size: 0 bytes --]

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Kyle Moffett]

Thanks for the detailed reply!

On Jul 09, 2007, at 22:07:15, Nigel Cunningham wrote:
> On Friday 06 July 2007 15:01:48 Kyle Moffett wrote:
>> Suppose hibernate is implemented like this:
>>
>> (1) Userspace program calls sys_freeze_processes()
>>    (a) Pokes all CPUs with IPMIs and tells them to finish the  
>> currently running timeslot then stop
>>    (b) Atomically sends SIGSTOP to all userspace processes in a  
>> non-trappable way, except the calling process and any process  
>> which is ptracing it.
>>    (c) Returns to the calling process.
>
> Ok. First, I'll ignore the specification that userspace does this -  
> I don't think it matters whether it's userspace or kernel that does  
> the suspending and I'm yet to see a good reason for it to be  
> [required to be] done from userspace.

The reason it's _required_ to be done from userspace is that  
userspace is the only one which can figure out "These processes need  
to run for suspend to work", and then let those processes continue  
running after the freeze.  The *ONLY* reason this even stops  
processes at all is so we can do the post-device-mapper-snapshot code  
with very little usably-free RAM (IE: only about 1MB for a standard  
desktop system).


> In this first step, you've reinvented the first part of the current  
> freezer implementation. The reason we don't use a real signal is  
> precisely so we can have an untrappable SIGSTOP. In this regard, I  
> particularly remember Win4Lin from a few years ago. It would die if  
> you sent it a real signal, so we had to do it this way. No doubt  
> there are other instances I'm not aware of.

Well, you *do* want it to have semi-signal semantics, processes which  
receive it must not get back to userspace code so that they don't  
start allocating more memory when we're trying to do the freeze.  You  
also don't want a process to be able to trap it (IE: like SIGSTOP or  
SIGKILL).

On the other hand, it should be delivered asynchronously (IE: It  
doesn't break an interruptable sleep or respond to most is-a-signal- 
present checks).  You don't actually care if its sleeping in the  
kernel somewhere, just as long as it doesn't allocate much memory.

You would probably need a new signal "SIGFREEZE" which causes the  
process to be ignored as runnable the next time they schedule but  
never actually gets delivered, and a "SIGUNFREEZE" which does the  
reverse.  That way userspace could selectively resume processes based  
on its policy of "this needs to run for hibernation".


>> (2) Userspace process sends SIGCONT to only those processes which  
>> are necessary for sync and a device-mapper snapshot.
>
> How do you determine which ones are needed?

It's userspace's job to know which ones are needed.  For example, if  
you are hibernating over NFS then you need to resume the various NFS/ 
RPC daemons and threads.

> Why stop them in the first place?

So they aren't allocating memory when we are doing the device-mapper  
snapshot.


>> (3) Userspace calls sys_snapshot_kernel(snapshot_overhead_pages)
>>    (a) Kernel starts freeing memory and swapping stuff out to make  
>> room for a copy of *kernel* memory (not pagecache, not process  
>> RAM).   It does the same for at least snapshot_overhead_pages  
>> extra (used by userspace later).  It then allocates this memory to  
>> keep it from going away.  Since most processes are stopped we  
>> won't have much else competing with us for the RAM.
>
> Ok. So now you also need processes running that are needed for  
> swapping, because freeing that memory might involve swapping. Fully  
> agree with the logic though (not really surprising - this is what I  
> do in Suspend2^wTuxOnIce).
>
>>    (a) Kernel uses the device-mapper up-call-into-filesystem  
>> machinery to get all mounted filesystems synced and ready for a DM  
>> snapshot.  This may include sending data via the userspace  
>> processes resumed in (2).  Any deadlocks here are userspace's  
>> fault (see (2)).  Will need some modification to handle doing  
>> multiple blockdevs at a time.  Anything using FUSE is basically  
>> perma-synced anyways (no dep-handling needed), and anything using  
>> loop should already be handled by DM.  This includes allocating  
>> memory for the basic snapshot
>> datastructures.
>>    (b) At this point all blockdev operations should be halted and  
>> disk caches flushed; that's all we care about.
>>    (c) Go through the device tree and quiesce DMA and shut off  
>> interrupts.  Since all the disks are synced this is easy.
>>    (d) Use IPMIs again to get all the CPUs together, which should  
>> be easy as most processes are sleeping in IO or SIGSTOPed, and  
>> we're getting no interrupts.
>>    (e) One CPU turns off all interrupts on itself and takes an  
>> atomic snapshot of kernel memory into the previously allocated  
>> storage.  Once again, does not include pagecache.  The kernel also  
>> records a list of what pages *are* included in the pagecache.  It  
>> then marks all userspace pages as copy-on-write.
>
> Hotplugging cpus (when all those locking issues are taken care of)  
> is simpler.  Prior to cpu hotplugging, I used IMPIs to put  
> secondary cpus into a tight loop, so I know it's possible to do it  
> this way too.

It may be simpler, but it really screws up things like cpusets,  
processor affinity, etc.  It also ties hibernation to the presently  
very-flakey CPU-hotplug support, which is probably not what we want.


> That way, though, you have less flexibility. What if a cpu really  
> is plugged in between hibernate and resume? With cpu hotplugging,  
> it's handled properly and transparently.  Without cpu hotplugging,  
> you could be using uninitialised data after the atomic restore.

IMHO if the user pulls a CPU while the box is hibernated, then he/she  
gets what he/she deserves.  If you really want to support that, then  
the user must do the hotplug operation *manually* before suspending.   
Anything else is just going to be shooting ourselves in the foot  
repeatedly.

> Marking userspace as COW makes things more complicated, too. You  
> then have to add code to the COW handling to update the list of  
> pages that need to be saved, and you reduce the reliability of the  
> whole process. You can't predict beforehand how many of these COW  
> pages are going to be needed, and therefore can't know how much  
> memory to free earlier on in the process. If you run out of memory,  
> what will be the effect?

You could pretty easily have a spare 128MB swap partition somewhere  
which is not used during system operation but is "swapon"ed by  
userspace after the COW snapshot to provide extra backing store.

>>    (f) That CPU finalizes the modified DM snapshot using the  
>> previously-allocated memory.
>>    (g) That CPU frees up the snapshot_overhead_pages memory  
>> allocated during step (a) for userspace to use.
>>    (h) The CPU does the equivalent of a "swapoff -a" without  
>> overwriting any data already on any swap device(s).
>
> You still need to remember what swap you're going to use to write  
> the image. You'll probably want to get this information (and  
> allocate the swap) sooner rather than later so that you're not  
> racing against the memory freeing earlier, and don't run into  
> issues with bmapping the pages or having enough memory to record  
> the bdevs & sector numbers (not usually an issue, but if swap is  
> highly fragmented...).

Who says we have to use this swap to write the image?  That may be  
the default use-case, but it's certainly shouldn't be mandatory.   
Really, for the write-image-to-swap case you would just need to  
preallocate sufficient memory for the bmap tables beforehand, then  
populate them at this phase.

>>    (i) The CPU then IPMI-signals the other CPUs to wake them up
>>    (j) The kernel returns a FD-reference to the snapshot and the  
>> read-only halves of the CoW pagecache to the process which called  
>> sys_snapshot_kernel().
>
> Readonly halves? I don't get that, sorry.

Well, each page is copy-on-write, so the FD reference would always  
provide access to the original page data, whereas the processes may  
end up copying the page so they can write to it.  The trick would be  
that shared pages need to remain shared between processes even after  
the copy-on-write.  This is likely to be the trickiest part.

>> (4) The userspace process now has a reference to the copy of the  
>> kernel pages and the unmodified pagecache pages.  Since 99% of the  
>> processes aren't running, we aren't going to be having to CoW many  
>> of the pagecache pages.
>
> Mmm, but you still don't know how many.

Yes, but at this point we're basically running a segment of userspace  
with full kernel services available.  Like I said above we can just  
add a dedicated 128MB swap device to provide some spare backing  
store.  When you start running low on memory it might even page out  
to the swap device a part of the atomic copy of kernel memory.

>> (5) The userspace process uses read() or other syscalls to get  
>> data out of the kernel-snapshot FD in small chunks, within its  
>> snapshot_overhead_pages limit.  It compresses these and writes  
>> them out to the snapshot-storage blockdev (must not be mounted  
>> during snapshot), or to any network server.
>>
>> (6) The userspace process syncs the disks and halts the system.   
>> Any changed filesystem pages after the pseudo-DM-snapshot should  
>> have been stored in semi-volatile storage somewhere and will be  
>> discarded on the next reboot.
>
> Are you thinking the changed filesystem pages are caught by COW?  
> (AFAIUI, kernel writes aren't). If (as I expect), you're thinking  
> about filesystem writes to DM based storage, what about non DM- 
> based filesystem pages?

No, but a kernel write to a DM-snapshot calls the DM-snapshot code  
which copies the segment(s) to the snapshot device and modifies them  
there.  Basically disk-filesystem pagecache pages would be synced and  
protected by the DM snapshot, while anonymous memory pages would be  
CoW-ed.  And it might not be an unreasonable requirement to state  
that the disk-based filesystems must all be mapped through DM- 
snapshot devices (even just straight 1:1 linear mappings), so that  
they can be trivially snapshotted.

>> So basically your hibernate-overhead would consist of:
>>    (1) The pages necessary for the atomic snapshot of kernel  
>> memory and the list of pagecache pages at that time
>>    (2) A little memory necessary for the kernel non-persistent DM  
>> snapshot datastructures.
>>    (3) The snapshot_overhead_pages needed by userspace.
>>
>> If you're using swap devices then you can save 99% of the state of  
>> the running kernel with an initial swapout overhead of virtually  
>> nothing beyond the size of the unswappable kernel memory.
>
> FWIW, let me note an important variation from how Suspend2 works;  
> it might provide food for thought. In Suspend2, we treat the  
> processes that remain stopped throughout the whole process  
> specially. We write their data to disk before the atomic copy  
> (usually 70 or 80% of memory), and then use the memory they occupy  
> for the destination of the atomic copy. This further reduces the  
> amount of memory that has to be freed, almost always to zero.

I suppose you could record swap-outs done to SIGFREEZEd processes  
specially, so that they would be swapped in again before resuming  
userspace.  That would effectively result in the same thing.

Cheers,
Kyle Moffett

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Nigel Cunningham]

[-- Attachment #1.1: Type: text/plain, Size: 4877 bytes --]

Hi.

On Friday 06 July 2007 11:19:32 Kyle Moffett wrote:
> On Jul 05, 2007, at 19:35:11, Nigel Cunningham wrote:
> > On Friday 06 July 2007 09:20:43 Benjamin Herrenschmidt wrote:
> >> No, the freezer creates all those places what are harmful for a  
> >> task to block because they will break the freezer :-)
> >
> > Nice try :) Okay then, you remove the freezer, try hibernating,  
> > then get back to me after you've fixed your filesystem because some  
> > process that wasn't frozen started writing things after the atomic  
> > copy (making the on disk filesystem inconsistent with the snapshot).
> 
> Umm, this thread is NOT ABOUT HIBERNATING!!!  Please go back and read  
> the subject, specifically the "suspend to RAM" parts :-D.  When your  
> hardware can put itself to sleep and atomically preserve memory as it  
> does so, you don't need an atomic copy.  For Real Suspend(TM) (IE:  
> Suspend-to-RAM), the list of things to do is short and simple:

We agreed a while back that you don't need the freezer for suspend to ram. As 
far as I was aware, we went off-topic, so the topic is out of date.

> 1)  Stop DMA and put most hardware into low-power states (stops all  
> interrupt sources)
> 2)  Ensure that the other CPUs have finished any trailing interrupt  
> handlers and put them to sleep
> 3)  Put the interrupt-controllers into low-power state
> 4)  Go to sleep
> 
> > As Pavel rightly said, you can get rid of the freezer, but you're  
> > only going to have to implement another one that does the  
> > essentially the same thing, even if it is at some other level.
> 
> How about a freezer whose job it is to "wait for pending hard  
> interrupts to complete when we have already guaranteed that we won't  
> get any more"?  That part should be really *REALLY* easy.  You don't  
> need to care about either userspace processes or kernel threads at  
> all.  Specifically, Step 1 consists of:
> 
> suspend_device(dev)
> {
> 	set_no_bind_flag(dev);
> 	for (dev->subdevices)
> 		suspend_device(dev);
> 	set_no_io_flag(dev);
> 	wait_for_in_progress_dma(dev);
> 	turn_off_interrupts(dev);
> 	go_to_low_power_state(dev);
> }
> 
> After you've set the "no_bind" flag, you won't get any *new*  
> subdevices trying to bind, therefore it's safe to iterate over the  
> list of present sub-devices and suspend them.  Once those are  
> suspended and in low-power states you can set a "no_io" flag to  
> prevent the driver from submitting more IO.  At that point you can  
> lazily wait for existing DMA/IO/interrupts to finish on the device,  
> since *NOBODY* will be submitting them anymore, and we certainly  
> aren't probing for new devices.  Then you can just turn off the power  
> to the device.  When all the leaf devices are off, the parent device  
> can be turned off because everything waiting on the leaf devices is  
> blocked on them and won't unblock until the parent device *AND* the  
> leaf device are turned on again, in that order.

For suspending, yes. For hibernating, that's not enough, because other 
processes can still be happily allocating and freeing memory, and will only 
get stopped when they try to do i/o or such like. If you're trying to make 
hibernation reliable, you need to be able to reliably check whether you're 
going to have enough storage for the image you're preparing, and enough 
memory for the atomic copy and so on. That's why the freezer is needed for 
hibernation. If you don't have it, any hibernation implementation you make is 
going to be only as reliable as the extent to which the system is otherwise 
idle.
 
> Scheduling and userspace are all still fully enabled in this  
> scenario.  Once all your devices are turned off, the only remaining  
> running threads will be those which haven't done IO since the  
> beginning of the suspend.  We can then disable preemption, turn off  
> the timer interrupts, and tell the other CPUs to park all their  
> remaining threads in schedule() and sleep.  Then we put the IRQ  
> controller to sleep and go to sleep ourselves.  If our driver model  
> locking is sufficient to handle putting a parent device to sleep  
> while threads are sleeping on a child device then there are exactly 0  
> problems.
> 
> Resuming is basically running the whole process in reverse.  Runtime- 
> suspend is achieved by not setting the 'no_io' or 'no_bind' flags and  
> putting selective device-subtrees to sleep without doing anything to  
> the rest of the system.

Fully agree when it comes to suspend to ram. There, this process should work, 
so far as I can see. But as I said, we went - to one degree or another - off 
topic, and did discuss hibernation too.

Regards,

Nigel
-- 
See http://www.tuxonice.net for Howtos, FAQs, mailing
lists, wiki and bugzilla info.

[-- Attachment #1.2: Type: application/pgp-signature, Size: 189 bytes --]

[-- Attachment #2: Type: text/plain, Size: 0 bytes --]

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Benjamin Herrenschmidt]

> How about a freezer whose job it is to "wait for pending hard  
> interrupts to complete when we have already guaranteed that we won't  
> get any more"?  That part should be really *REALLY* easy.  You don't  
> need to care about either userspace processes or kernel threads at  
> all.  Specifically, Step 1 consists of:

Well, waiting for pending DMA and making sure to not trigger more
activity is what driver suspend() is supposed to do. With the ability
for simple drivers that can cope with it to just basically use a
late_suspend(), called after IRQs are off, that basically does what you
describe: wait for pending HW tasks to complete (polling) and turn the
damn thing off.

Note that the later is really a shortcut for somewhat dump and directly
accessible devices (PCI comes to mind). Things like USB has to use the
"normal" mechanism of blocking IOs etc... at suspend(), at least, USB
devices have to since the USB HC will not issue any new URBs. (And will
return them with a nice error code which is a perfect way to deal with
it in driver, been there, it works fine, once we fixed the races in the
USB host code itself, which I think we pretty much did by now).

> Scheduling and userspace are all still fully enabled in this  
> scenario.  Once all your devices are turned off, the only remaining  
> running threads will be those which haven't done IO since the  
> beginning of the suspend.  We can then disable preemption, turn off  
> the timer interrupts, and tell the other CPUs to park all their  
> remaining threads in schedule() and sleep.  Then we put the IRQ  
> controller to sleep and go to sleep ourselves.  If our driver model  
> locking is sufficient to handle putting a parent device to sleep  
> while threads are sleeping on a child device then there are exactly 0  
> problems.

What you propose is basically a slightly over-simplistic version of what
I think (and Paulus think) should be done. We do need to do it via
driver callbacks down the tree since only drivers can know how to deal
with their DMA etc... and ordering need to be respected, but that's
basically it.

And guess what ? It's what we do on powerbooks, and it works fine,
without a freezer :-)

Ben.

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Rafael J. Wysocki]

On Friday, 6 July 2007 05:59, Benjamin Herrenschmidt wrote:
> 
> > How about a freezer whose job it is to "wait for pending hard  
> > interrupts to complete when we have already guaranteed that we won't  
> > get any more"?  That part should be really *REALLY* easy.  You don't  
> > need to care about either userspace processes or kernel threads at  
> > all.  Specifically, Step 1 consists of:
> 
> Well, waiting for pending DMA and making sure to not trigger more
> activity is what driver suspend() is supposed to do. With the ability
> for simple drivers that can cope with it to just basically use a
> late_suspend(), called after IRQs are off, that basically does what you
> describe: wait for pending HW tasks to complete (polling) and turn the
> damn thing off.
> 
> Note that the later is really a shortcut for somewhat dump and directly
> accessible devices (PCI comes to mind). Things like USB has to use the
> "normal" mechanism of blocking IOs etc... at suspend(), at least, USB
> devices have to since the USB HC will not issue any new URBs. (And will
> return them with a nice error code which is a perfect way to deal with
> it in driver, been there, it works fine, once we fixed the races in the
> USB host code itself, which I think we pretty much did by now).
> 
> > Scheduling and userspace are all still fully enabled in this  
> > scenario.  Once all your devices are turned off, the only remaining  
> > running threads will be those which haven't done IO since the  
> > beginning of the suspend.  We can then disable preemption, turn off  
> > the timer interrupts, and tell the other CPUs to park all their  
> > remaining threads in schedule() and sleep.  Then we put the IRQ  
> > controller to sleep and go to sleep ourselves.  If our driver model  
> > locking is sufficient to handle putting a parent device to sleep  
> > while threads are sleeping on a child device then there are exactly 0  
> > problems.
> 
> What you propose is basically a slightly over-simplistic version of what
> I think (and Paulus think) should be done. We do need to do it via
> driver callbacks down the tree since only drivers can know how to deal
> with their DMA etc... and ordering need to be respected, but that's
> basically it.
> 
> And guess what ? It's what we do on powerbooks, and it works fine,
> without a freezer :-)

On powerbooks you disable the nonboot CPUs before suspending devices, which
simplifies things _a_ _lot_ in comparison with the general case.

If you additionally disable kernel preemption, then you don't need anything
like the freezer anyway in that case (except for detecting situations in which
the suspend process can deadlock with a task stuck in the D state holding a
lock, but that's overkill).

Greetings,
Rafael


-- 
"Premature optimization is the root of all evil." - Donald Knuth

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Benjamin Herrenschmidt]

> On powerbooks you disable the nonboot CPUs before suspending devices, which
> simplifies things _a_ _lot_ in comparison with the general case.

Not that much actually. It avoids having to gather them all in the last
stage at the low level but that isn't really related to what we are
talking about right now.

> If you additionally disable kernel preemption, then you don't need anything
> like the freezer anyway in that case (except for detecting situations in which
> the suspend process can deadlock with a task stuck in the D state holding a
> lock, but that's overkill).

kernel preemption and SMP have nothing to do with it. You can still
schedule because one of the driver suspend() callback is schedul'ing,
maybe waiting for some requests to complete, etc... Happens typically
with the disk suspend callbacks.

I agree that it does lower the likelyness of having a userspace process
pound on the wrong driver at the wrong time though.

Ben.

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Alan Stern]

On Fri, 6 Jul 2007, Benjamin Herrenschmidt wrote:

> What you propose is basically a slightly over-simplistic version of what
> I think (and Paulus think) should be done. We do need to do it via
> driver callbacks down the tree since only drivers can know how to deal
> with their DMA etc... and ordering need to be respected, but that's
> basically it.
> 
> And guess what ? It's what we do on powerbooks, and it works fine,
> without a freezer :-)

I wish you'd stop saying that.  Have you ever done any serious testing?

Here's something to try:  Add a time delay to the end of hub_suspend in
drivers/usb/core/hub.c, so you can provoke a race manually.  Then while
one of your root hubs is being suspended and the system is waiting in
that delay, either plug in a new USB device to that hub or unplug an
existing device.

Be sure that CONFIG_USB_DEBUG is on so that we can figure out what 
happened after the fact.

Alan Stern

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Alan Stern]

On Thu, 5 Jul 2007, Kyle Moffett wrote:

> Umm, this thread is NOT ABOUT HIBERNATING!!!  Please go back and read  
> the subject, specifically the "suspend to RAM" parts :-D.

But it _is_ about the freezer; see the "Remove process freezer" part.  
:-)  Since the freezer is used during hibernation, hibernation is a
legitimate topic.

>  When your  
> hardware can put itself to sleep and atomically preserve memory as it  
> does so, you don't need an atomic copy.  For Real Suspend(TM) (IE:  
> Suspend-to-RAM), the list of things to do is short and simple:
> 
> 1)  Stop DMA and put most hardware into low-power states (stops all  
> interrupt sources)
> 2)  Ensure that the other CPUs have finished any trailing interrupt  
> handlers and put them to sleep
> 3)  Put the interrupt-controllers into low-power state
> 4)  Go to sleep

Your short and simple list omits a few crucial items:

A)  Decide what to do about remote wakeup requests.
B)  Prevent I/O requests from resuming devices that have been 
suspended.
C)  Prevent devices and drivers from being registered or unregistered; 
in particular decide what to do about hot-plug or hot-unplug events.
D)  Block driver bind or unbind calls.

Any of these things is capable of screwing up the course of events.
(In fact A _should_ be allowed to abort a suspend.)

> > As Pavel rightly said, you can get rid of the freezer, but you're  
> > only going to have to implement another one that does the  
> > essentially the same thing, even if it is at some other level.
> 
> How about a freezer whose job it is to "wait for pending hard  
> interrupts to complete when we have already guaranteed that we won't  
> get any more"?  That part should be really *REALLY* easy.  You don't  
> need to care about either userspace processes or kernel threads at  
> all.  Specifically, Step 1 consists of:
> 
> suspend_device(dev)
> {
> 	set_no_bind_flag(dev);
> 	for (dev->subdevices)
> 		suspend_device(dev);
> 	set_no_io_flag(dev);
> 	wait_for_in_progress_dma(dev);
> 	turn_off_interrupts(dev);
> 	go_to_low_power_state(dev);
> }
> 
> After you've set the "no_bind" flag, you won't get any *new*  
> subdevices trying to bind, 

So what happens if a new subdevice arrives at the wrong time?  Do you 
block instead of binding it?  While holding a mutex needed to suspend 
the parent device?

What about drivers trying to bind to existing devices?

What happens to I/O requests submitted after the "no_io" flag is set?  
The driver will have to block them, effectively creating its own little 
"freezer".

> therefore it's safe to iterate over the  
> list of present sub-devices and suspend them.  Once those are  
> suspended and in low-power states you can set a "no_io" flag to  
> prevent the driver from submitting more IO.  At that point you can  
> lazily wait for existing DMA/IO/interrupts to finish on the device,  
> since *NOBODY* will be submitting them anymore, and we certainly  
> aren't probing for new devices.  Then you can just turn off the power  
> to the device.  When all the leaf devices are off, the parent device  
> can be turned off because everything waiting on the leaf devices is  
> blocked on them and won't unblock until the parent device *AND* the  
> leaf device are turned on again, in that order.

This is a lot like what we already do.  The differences are:

	There is nothing corresponding to your "no-bind" flag.

	Most drivers don't have anything like your "no_io" flag;
	they assume that nobody will be around to submit an I/O
	request.

> Scheduling and userspace are all still fully enabled in this  
> scenario.  Once all your devices are turned off, the only remaining  
> running threads will be those which haven't done IO since the  
> beginning of the suspend.  We can then disable preemption, turn off  
> the timer interrupts, and tell the other CPUs to park all their  
> remaining threads in schedule() and sleep.  Then we put the IRQ  
> controller to sleep and go to sleep ourselves.  If our driver model  
> locking is sufficient to handle putting a parent device to sleep  
> while threads are sleeping on a child device then there are exactly 0  
> problems.
> 
> Resuming is basically running the whole process in reverse.  Runtime- 
> suspend is achieved by not setting the 'no_io' or 'no_bind' flags and  
> putting selective device-subtrees to sleep without doing anything to  
> the rest of the system.

Nobody doubts that suspend can be made to work without the freezer.  
The point is that doing it this way dumps a bunch of extra 
responsibility on drivers.

Alan Stern

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Pavel Machek]

Hi!

> So I think Matthew is totally right. In fact, the presence of the
> freezer is the main reason why Paulus so far NACKed Johannes attempts at
> merging the PPC PM code with the generic code in kernel/power.c
> 
> We've been doing fine without it so far and intend to continue to do
> so.

How well does it work on SMP PPC?
									Pavel

-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

Re: Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Paul Mackerras]

Pavel Machek writes:

> How well does it work on SMP PPC?

Just fine, on those machines where we know how to reinitialize the
video card.  We currently require userspace to offline all except the
boot cpu before suspending, but that could be moved into the kernel.
I have no particular attachment to that way of doing it; it was just a
"don't do things in the kernel that can be reasonably be done in
userspace" kind of thing.

Paul.

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Benjamin Herrenschmidt]

On Tue, 2007-07-03 at 05:29 +0100, Matthew Garrett wrote:
> Suspend to RAM on a machine with / on a fuse filesystem turns out to be 
> a screaming nightmare - either the suspend fails because syslog (for 
> instance) can't be frozen, or the machine deadlocks for some other 
> reason I haven't tracked down. We could "fix" fuse, or alternatively we 
> could do what we do for suspend to RAM on other platforms (PPC and APM) 
> and just not use the freezer.
> 
> Signed-off-by: Matthew Garrett <mjg59@srcf.ucam.org>

Acked-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>

(And with much pleasure :-)

Cheers,
Ben.

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Benjamin Herrenschmidt]

On Tue, 2007-07-03 at 05:29 +0100, Matthew Garrett wrote:
> Suspend to RAM on a machine with / on a fuse filesystem turns out to be 
> a screaming nightmare - either the suspend fails because syslog (for 
> instance) can't be frozen, or the machine deadlocks for some other 
> reason I haven't tracked down. We could "fix" fuse, or alternatively we 
> could do what we do for suspend to RAM on other platforms (PPC and APM) 
> and just not use the freezer.

The main reason for deadlocks is because we do a sys_sync() after the
freeze, which we shouldn't do.

Ben.

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Oliver Neukum]

Am Dienstag, 3. Juli 2007 schrieb Matthew Garrett:
> Suspend to RAM on a machine with / on a fuse filesystem turns out to be 
> a screaming nightmare - either the suspend fails because syslog (for 
> instance) can't be frozen, or the machine deadlocks for some other 
> reason I haven't tracked down. We could "fix" fuse, or alternatively we 
> could do what we do for suspend to RAM on other platforms (PPC and APM) 
> and just not use the freezer.

Only if you want to audit all character devices' read() and write()
methods for races against suspend().
/ on fuse is a bad idea.

	Regards
		Oliver

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Romano Giannetti]

On Tue, 2007-07-03 at 05:29 +0100, Matthew Garrett wrote:
> or alternatively we  could do what we do for suspend to RAM on other
> platforms (PPC and APM) and just not use the freezer.

As a data point, I am running with this patch on top of 2.6.21.2 the
last 3+ weeks, with an average of 5/6 STR cycles a day, and had no
problems at all. (Sony vaio pcg-fx701). Just normal work, I didn't try
to stress the thing, but I have quite a few times suspended/resumed over
a big compile without a glitch.

What are the risks of this patch supposed to be?

Romano



--
La presente comunicación tiene carácter confidencial y es para el exclusivo uso del destinatario indicado en la misma. Si Ud. no es el destinatario indicado, le informamos que cualquier forma de distribución, reproducción o uso de esta comunicación y/o de la información contenida en la misma están estrictamente prohibidos por la ley. Si Ud. ha recibido esta comunicación por error, por favor, notifíquelo inmediatamente al remitente contestando a este mensaje y proceda a continuación a destruirlo. Gracias por su colaboración.

This communication contains confidential information. It is for the exclusive use of the intended addressee. If you are not the intended addressee, please note that any form of distribution, copying or use of this communication or the information in it is strictly prohibited by law. If you have received this communication in error, please immediately notify the sender by reply e-mail and destroy this message. Thank you for your cooperation.

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Oliver Neukum]

Am Dienstag, 3. Juli 2007 schrieb Romano Giannetti:
> On Tue, 2007-07-03 at 05:29 +0100, Matthew Garrett wrote:
> > or alternatively we  could do what we do for suspend to RAM on other
> > platforms (PPC and APM) and just not use the freezer.
> 
> As a data point, I am running with this patch on top of 2.6.21.2 the
> last 3+ weeks, with an average of 5/6 STR cycles a day, and had no
> problems at all. (Sony vaio pcg-fx701). Just normal work, I didn't try
> to stress the thing, but I have quite a few times suspended/resumed over
> a big compile without a glitch.
> 
> What are the risks of this patch supposed to be?

You did this test only while stressing normal block devices. Try
provoking races in character devices.

	Regards
		Oliver

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Rafael J. Wysocki]

On Tuesday, 3 July 2007 09:37, Romano Giannetti wrote:
> On Tue, 2007-07-03 at 05:29 +0100, Matthew Garrett wrote:
> > or alternatively we  could do what we do for suspend to RAM on other
> > platforms (PPC and APM) and just not use the freezer.
> 
> As a data point, I am running with this patch on top of 2.6.21.2 the
> last 3+ weeks, with an average of 5/6 STR cycles a day, and had no
> problems at all. (Sony vaio pcg-fx701). Just normal work, I didn't try
> to stress the thing, but I have quite a few times suspended/resumed over
> a big compile without a glitch.
> 
> What are the risks of this patch supposed to be?

See https://lists.linux-foundation.org/pipermail/linux-pm/2007-June/012847.html

Greetings,
Rafael


-- 
"Premature optimization is the root of all evil." - Donald Knuth

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Miklos Szeredi]

> > Suspend to RAM on a machine with / on a fuse filesystem turns out to be 
> > a screaming nightmare - either the suspend fails because syslog (for 
> > instance) can't be frozen, or the machine deadlocks for some other 
> > reason I haven't tracked down. We could "fix" fuse, or alternatively we 
> > could do what we do for suspend to RAM on other platforms (PPC and APM) 
> > and just not use the freezer.
> 
> Only if you want to audit all character devices' read() and write()
> methods for races against suspend().
> / on fuse is a bad idea.

What makes / special?  Why aren't all fuse filesystems affected?
Suspend isn't trying to do I/O on the root fs, is it?

Miklos

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Matthew Garrett]

On Tue, Jul 03, 2007 at 08:13:53AM +0200, Oliver Neukum wrote:

> Only if you want to audit all character devices' read() and write()
> methods for races against suspend().
> / on fuse is a bad idea.

Any driver that assumes that userspace will be frozen during suspend has 
been broken forever. That behaviour has never been guaranteed.
-- 
Matthew Garrett | mjg59@srcf.ucam.org

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Rafael J. Wysocki]

On Tuesday, 3 July 2007 14:13, Matthew Garrett wrote:
> On Tue, Jul 03, 2007 at 08:13:53AM +0200, Oliver Neukum wrote:
> 
> > Only if you want to audit all character devices' read() and write()
> > methods for races against suspend().
> > / on fuse is a bad idea.
> 
> Any driver that assumes that userspace will be frozen during suspend has 
> been broken forever. That behaviour has never been guaranteed.

Can we please fix those drivers _first_, then?

Rafael


-- 
"Premature optimization is the root of all evil." - Donald Knuth

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Rafael J. Wysocki]

On Tuesday, 3 July 2007 06:29, Matthew Garrett wrote:
> Suspend to RAM on a machine with / on a fuse filesystem turns out to be 
> a screaming nightmare - either the suspend fails because syslog (for 
> instance) can't be frozen, or the machine deadlocks for some other 
> reason I haven't tracked down. We could "fix" fuse, or alternatively we 
> could do what we do for suspend to RAM on other platforms (PPC and APM) 
> and just not use the freezer.
> 
> Signed-off-by: Matthew Garrett <mjg59@srcf.ucam.org>

Could you please rediff against the current -mm tree?

There are some patches in there that this will clash with.

I still think that this is a mistake, BTW.  Please see the Alan's post at

https://lists.linux-foundation.org/pipermail/linux-pm/2007-June/012847.html

Greetings,
Rafael


-- 
"Premature optimization is the root of all evil." - Donald Knuth

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Rafael J. Wysocki]

On Tuesday, 3 July 2007 07:49, Benjamin Herrenschmidt wrote:
> On Tue, 2007-07-03 at 05:29 +0100, Matthew Garrett wrote:
> > Suspend to RAM on a machine with / on a fuse filesystem turns out to be 
> > a screaming nightmare - either the suspend fails because syslog (for 
> > instance) can't be frozen, or the machine deadlocks for some other 
> > reason I haven't tracked down. We could "fix" fuse, or alternatively we 
> > could do what we do for suspend to RAM on other platforms (PPC and APM) 
> > and just not use the freezer.
> > 
> > Signed-off-by: Matthew Garrett <mjg59@srcf.ucam.org>
> 
> Acked-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
> 
> (And with much pleasure :-)

Clashes with some code already in -mm.

Greetings,
Rafael


-- 
"Premature optimization is the root of all evil." - Donald Knuth

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Rafael J. Wysocki]

On Tuesday, 3 July 2007 07:51, Benjamin Herrenschmidt wrote:
> On Tue, 2007-07-03 at 05:29 +0100, Matthew Garrett wrote:
> > Suspend to RAM on a machine with / on a fuse filesystem turns out to be 
> > a screaming nightmare - either the suspend fails because syslog (for 
> > instance) can't be frozen, or the machine deadlocks for some other 
> > reason I haven't tracked down. We could "fix" fuse, or alternatively we 
> > could do what we do for suspend to RAM on other platforms (PPC and APM) 
> > and just not use the freezer.
> 
> The main reason for deadlocks is because we do a sys_sync() after the
> freeze, which we shouldn't do.

So why don't we remove the sys_sync() from freeze_processes() instead?

Greetings,
Rafael


-- 
"Premature optimization is the root of all evil." - Donald Knuth

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Rafael J. Wysocki]

On Tuesday, 3 July 2007 15:08, Rafael J. Wysocki wrote:
> On Tuesday, 3 July 2007 07:51, Benjamin Herrenschmidt wrote:
> > On Tue, 2007-07-03 at 05:29 +0100, Matthew Garrett wrote:
> > > Suspend to RAM on a machine with / on a fuse filesystem turns out to be 
> > > a screaming nightmare - either the suspend fails because syslog (for 
> > > instance) can't be frozen, or the machine deadlocks for some other 
> > > reason I haven't tracked down. We could "fix" fuse, or alternatively we 
> > > could do what we do for suspend to RAM on other platforms (PPC and APM) 
> > > and just not use the freezer.
> > 
> > The main reason for deadlocks is because we do a sys_sync() after the
> > freeze, which we shouldn't do.
> 
> So why don't we remove the sys_sync() from freeze_processes() instead?

The patch follows (untested).

Greetings,
Rafael


---
From: Rafael J. Wysocki <rjw@sisk.pl>

We shouldn't sync filesystems from within the freezer, because it's not needed
for suspend to RAM and leads to problems with FUSE.

Signed-off-by: Rafael J. Wysocki <rjw@sisk.pl>
---
 kernel/power/disk.c    |    4 ++++
 kernel/power/process.c |    1 -
 2 files changed, 4 insertions(+), 1 deletion(-)

Index: linux-2.6.22-rc7/kernel/power/disk.c
===================================================================
--- linux-2.6.22-rc7.orig/kernel/power/disk.c
+++ linux-2.6.22-rc7/kernel/power/disk.c
@@ -296,6 +296,10 @@ int hibernate(void)
 {
 	int error;
 
+	printk("Syncing filesystems ... \n");
+	sys_sync();
+	printk("done.\n");
+
 	mutex_lock(&pm_mutex);
 	/* The snapshot device should not be opened while we're running */
 	if (!atomic_add_unless(&snapshot_device_available, -1, 0)) {
Index: linux-2.6.22-rc7/kernel/power/process.c
===================================================================
--- linux-2.6.22-rc7.orig/kernel/power/process.c
+++ linux-2.6.22-rc7/kernel/power/process.c
@@ -190,7 +190,6 @@ int freeze_processes(void)
 	if (error)
 		return error;
 
-	sys_sync();
 	error = try_to_freeze_tasks(FREEZER_KERNEL_THREADS);
 	if (error)
 		return error;

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Oliver Neukum]

Am Dienstag, 3. Juli 2007 schrieb Rafael J. Wysocki:
> On Tuesday, 3 July 2007 15:08, Rafael J. Wysocki wrote:
> > On Tuesday, 3 July 2007 07:51, Benjamin Herrenschmidt wrote:
> > > On Tue, 2007-07-03 at 05:29 +0100, Matthew Garrett wrote:
> > > > Suspend to RAM on a machine with / on a fuse filesystem turns out to be 
> > > > a screaming nightmare - either the suspend fails because syslog (for 
> > > > instance) can't be frozen, or the machine deadlocks for some other 
> > > > reason I haven't tracked down. We could "fix" fuse, or alternatively we 
> > > > could do what we do for suspend to RAM on other platforms (PPC and APM) 
> > > > and just not use the freezer.
> > > 
> > > The main reason for deadlocks is because we do a sys_sync() after the
> > > freeze, which we shouldn't do.
> > 
> > So why don't we remove the sys_sync() from freeze_processes() instead?
> 
> The patch follows (untested).
> 
> Greetings,
> Rafael
> 
> 
> ---
> From: Rafael J. Wysocki <rjw@sisk.pl>
> 
> We shouldn't sync filesystems from within the freezer, because it's not needed
> for suspend to RAM and leads to problems with FUSE.

This seems fishy. Swsusp needs enough clean memory to make enough
room for the image. If you sync before you freeze, the running tasks can
redirty memory.
What makes you sure that you don't die as shrink_all_memory() writes out
pages?

	Regards
		Oliver

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Rafael J. Wysocki]

On Tuesday, 3 July 2007 19:20, Oliver Neukum wrote:
> Am Dienstag, 3. Juli 2007 schrieb Rafael J. Wysocki:
> > On Tuesday, 3 July 2007 15:08, Rafael J. Wysocki wrote:
> > > On Tuesday, 3 July 2007 07:51, Benjamin Herrenschmidt wrote:
> > > > On Tue, 2007-07-03 at 05:29 +0100, Matthew Garrett wrote:
> > > > > Suspend to RAM on a machine with / on a fuse filesystem turns out to be 
> > > > > a screaming nightmare - either the suspend fails because syslog (for 
> > > > > instance) can't be frozen, or the machine deadlocks for some other 
> > > > > reason I haven't tracked down. We could "fix" fuse, or alternatively we 
> > > > > could do what we do for suspend to RAM on other platforms (PPC and APM) 
> > > > > and just not use the freezer.
> > > > 
> > > > The main reason for deadlocks is because we do a sys_sync() after the
> > > > freeze, which we shouldn't do.
> > > 
> > > So why don't we remove the sys_sync() from freeze_processes() instead?
> > 
> > The patch follows (untested).
> > 
> > Greetings,
> > Rafael
> > 
> > 
> > ---
> > From: Rafael J. Wysocki <rjw@sisk.pl>
> > 
> > We shouldn't sync filesystems from within the freezer, because it's not needed
> > for suspend to RAM and leads to problems with FUSE.
> 
> This seems fishy. Swsusp needs enough clean memory to make enough
> room for the image. If you sync before you freeze, the running tasks can
> redirty memory.
> What makes you sure that you don't die as shrink_all_memory() writes out
> pages?

I don't think that would matter.

Still, I can remove the sync from the suspend code path only, leaving it in
the hibernation code path.  The patch will be bigger, but well.

Any objection to that?

Greetings,
Rafael


-- 
"Premature optimization is the root of all evil." - Donald Knuth

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Benjamin Herrenschmidt]

> I don't think that would matter.
> 
> Still, I can remove the sync from the suspend code path only, leaving it in
> the hibernation code path.  The patch will be bigger, but well.
> 
> Any objection to that?

Makes sense to sync before suspend tho, to limit the amount of dirty
non-written pages in case things go wrong.

Ben.

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Rafael J. Wysocki]

On Tuesday, 3 July 2007 23:35, Benjamin Herrenschmidt wrote:
> 
> > I don't think that would matter.
> > 
> > Still, I can remove the sync from the suspend code path only, leaving it in
> > the hibernation code path.  The patch will be bigger, but well.
> > 
> > Any objection to that?
> 
> Makes sense to sync before suspend tho, to limit the amount of dirty
> non-written pages in case things go wrong.

OK, below is the updated patch.

Greetings,
Rafael


---
From: Rafael J. Wysocki <rjw@sisk.pl>

The syncing of filesystems from within the freezer in not needed for suspend to
RAM and leads to problems with FUSE.  Change freeze_processes() so that it
doesn't execute sys_sync() and introduce the "syncing" version of it to be
called from the hibernation code paths.

Signed-off-by: Rafael J. Wysocki <rjw@sisk.pl>
---
 include/linux/freezer.h |   14 ++++++++++++--
 kernel/power/disk.c     |    2 +-
 kernel/power/main.c     |    6 ++++++
 kernel/power/process.c  |    8 +++++---
 kernel/power/user.c     |    2 +-
 5 files changed, 25 insertions(+), 7 deletions(-)

Index: linux-2.6.22-rc7/include/linux/freezer.h
===================================================================
--- linux-2.6.22-rc7.orig/include/linux/freezer.h	2007-07-04 00:21:26.000000000 +0200
+++ linux-2.6.22-rc7/include/linux/freezer.h	2007-07-04 00:22:37.000000000 +0200
@@ -62,7 +62,7 @@ static inline int thaw_process(struct ta
 }
 
 extern void refrigerator(void);
-extern int freeze_processes(void);
+extern int __freeze_processes(int sync_filesystems);
 extern void thaw_processes(void);
 
 static inline int try_to_freeze(void)
@@ -134,7 +134,7 @@ static inline void clear_freeze_flag(str
 static inline int thaw_process(struct task_struct *p) { return 1; }
 
 static inline void refrigerator(void) {}
-static inline int freeze_processes(void) { BUG(); return 0; }
+static inline int __freeze_processes(int s) { BUG(); return 0; }
 static inline void thaw_processes(void) {}
 
 static inline int try_to_freeze(void) { return 0; }
@@ -145,4 +145,14 @@ static inline int freezer_should_skip(st
 static inline void set_freezable(void) {}
 #endif
 
+static inline int freeze_processes(void)
+{
+	return __freeze_processes(0);
+}
+
+static inline int freeze_processes_with_sync(void)
+{
+	return __freeze_processes(1);
+}
+
 #endif	/* FREEZER_H_INCLUDED */
Index: linux-2.6.22-rc7/kernel/power/disk.c
===================================================================
--- linux-2.6.22-rc7.orig/kernel/power/disk.c	2007-07-04 00:21:26.000000000 +0200
+++ linux-2.6.22-rc7/kernel/power/disk.c	2007-07-04 00:21:44.000000000 +0200
@@ -281,7 +281,7 @@ static int prepare_processes(void)
 	int error = 0;
 
 	pm_prepare_console();
-	if (freeze_processes()) {
+	if (freeze_processes_with_sync()) {
 		error = -EBUSY;
 		unprepare_processes();
 	}
Index: linux-2.6.22-rc7/kernel/power/main.c
===================================================================
--- linux-2.6.22-rc7.orig/kernel/power/main.c	2007-07-04 00:21:26.000000000 +0200
+++ linux-2.6.22-rc7/kernel/power/main.c	2007-07-04 00:23:40.000000000 +0200
@@ -20,6 +20,7 @@
 #include <linux/resume-trace.h>
 #include <linux/freezer.h>
 #include <linux/vmstat.h>
+#include <linux/syscalls.h>
 
 #include "power.h"
 
@@ -231,6 +232,11 @@ static int enter_state(suspend_state_t s
 
 	if (!valid_state(state))
 		return -ENODEV;
+
+	printk("Syncing filesystems ... ");
+	sys_sync();
+	printk("done.\n");
+
 	if (!mutex_trylock(&pm_mutex))
 		return -EBUSY;
 
Index: linux-2.6.22-rc7/kernel/power/process.c
===================================================================
--- linux-2.6.22-rc7.orig/kernel/power/process.c	2007-07-04 00:21:26.000000000 +0200
+++ linux-2.6.22-rc7/kernel/power/process.c	2007-07-04 00:21:44.000000000 +0200
@@ -179,9 +179,9 @@ static int try_to_freeze_tasks(int freez
 }
 
 /**
- *	freeze_processes - tell processes to enter the refrigerator
+ *	__freeze_processes - tell processes to enter the refrigerator
  */
-int freeze_processes(void)
+int __freeze_processes(int sync_filesystems)
 {
 	int error;
 
@@ -190,7 +190,9 @@ int freeze_processes(void)
 	if (error)
 		return error;
 
-	sys_sync();
+	if (sync_filesystems)
+		sys_sync();
+
 	error = try_to_freeze_tasks(FREEZER_KERNEL_THREADS);
 	if (error)
 		return error;
Index: linux-2.6.22-rc7/kernel/power/user.c
===================================================================
--- linux-2.6.22-rc7.orig/kernel/power/user.c	2007-07-04 00:21:26.000000000 +0200
+++ linux-2.6.22-rc7/kernel/power/user.c	2007-07-04 00:21:44.000000000 +0200
@@ -153,7 +153,7 @@ static int snapshot_ioctl(struct inode *
 		mutex_lock(&pm_mutex);
 		error = pm_notifier_call_chain(PM_HIBERNATION_PREPARE);
 		if (!error) {
-			error = freeze_processes();
+			error = freeze_processes_with_sync();
 			if (error)
 				thaw_processes();
 		}

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Pavel Machek]

On Tue 2007-07-03 19:20:59, Oliver Neukum wrote:
> Am Dienstag, 3. Juli 2007 schrieb Rafael J. Wysocki:
> > On Tuesday, 3 July 2007 15:08, Rafael J. Wysocki wrote:
> > > On Tuesday, 3 July 2007 07:51, Benjamin Herrenschmidt wrote:
> > > > On Tue, 2007-07-03 at 05:29 +0100, Matthew Garrett wrote:
> > > > > Suspend to RAM on a machine with / on a fuse filesystem turns out to be 
> > > > > a screaming nightmare - either the suspend fails because syslog (for 
> > > > > instance) can't be frozen, or the machine deadlocks for some other 
> > > > > reason I haven't tracked down. We could "fix" fuse, or alternatively we 
> > > > > could do what we do for suspend to RAM on other platforms (PPC and APM) 
> > > > > and just not use the freezer.
> > > > 
> > > > The main reason for deadlocks is because we do a sys_sync() after the
> > > > freeze, which we shouldn't do.
> > > 
> > > So why don't we remove the sys_sync() from freeze_processes() instead?
> > 
> > The patch follows (untested).
> > 
> > Greetings,
> > Rafael
> > 
> > 
> > ---
> > From: Rafael J. Wysocki <rjw@sisk.pl>
> > 
> > We shouldn't sync filesystems from within the freezer, because it's not needed
> > for suspend to RAM and leads to problems with FUSE.
> 
> This seems fishy. Swsusp needs enough clean memory to make enough
> room for the image. If you sync before you freeze, the running tasks can
> redirty memory.
> What makes you sure that you don't die as shrink_all_memory() writes out
> pages?

Shrink_all_memory should just free enough memory, what's the problem?
Yes, we can have dirty memory, shrink_all_memory() can write that out
just fine.

-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Oliver Neukum]

Am Donnerstag, 5. Juli 2007 schrieb Pavel Machek:
> On Tue 2007-07-03 19:20:59, Oliver Neukum wrote:
> > Am Dienstag, 3. Juli 2007 schrieb Rafael J. Wysocki:
> > > On Tuesday, 3 July 2007 15:08, Rafael J. Wysocki wrote:
> > > > On Tuesday, 3 July 2007 07:51, Benjamin Herrenschmidt wrote:
> > > > > On Tue, 2007-07-03 at 05:29 +0100, Matthew Garrett wrote:
> > > > > > Suspend to RAM on a machine with / on a fuse filesystem turns out to be 
> > > > > > a screaming nightmare - either the suspend fails because syslog (for 
> > > > > > instance) can't be frozen, or the machine deadlocks for some other 
> > > > > > reason I haven't tracked down. We could "fix" fuse, or alternatively we 
> > > > > > could do what we do for suspend to RAM on other platforms (PPC and APM) 
> > > > > > and just not use the freezer.
> > > > > 
> > > > > The main reason for deadlocks is because we do a sys_sync() after the
> > > > > freeze, which we shouldn't do.
> > > > 
> > > > So why don't we remove the sys_sync() from freeze_processes() instead?
> > > 
> > > The patch follows (untested).
> > > 
> > > Greetings,
> > > Rafael
> > > 
> > > 
> > > ---
> > > From: Rafael J. Wysocki <rjw@sisk.pl>
> > > 
> > > We shouldn't sync filesystems from within the freezer, because it's not needed
> > > for suspend to RAM and leads to problems with FUSE.
> > 
> > This seems fishy. Swsusp needs enough clean memory to make enough
> > room for the image. If you sync before you freeze, the running tasks can
> > redirty memory.
> > What makes you sure that you don't die as shrink_all_memory() writes out
> > pages?
> 
> Shrink_all_memory should just free enough memory, what's the problem?
> Yes, we can have dirty memory, shrink_all_memory() can write that out
> just fine.

If there's any dirty memory to be written out through fuse, this will not
work as user space is already frozen. Now I am told that with fuse writes
are synchronous. Therefore I don't understand why having a call to sys_sync()
can make a difference. IMHO removing it to make fuse work covers over
a symptom but hides the bug.

	Regards
		Oliver

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Oliver Neukum]

Am Dienstag, 3. Juli 2007 schrieb Rafael J. Wysocki:
> > > The main reason for deadlocks is because we do a sys_sync() after the
> > > freeze, which we shouldn't do.
> > 
> > So why don't we remove the sys_sync() from freeze_processes() instead?
> 
> The patch follows (untested).

And a further question. The freezer is not atomic. What do you do
if a task not yet frozen calls sys_sync(), but fuse is already frozen?

	Regards
		Oliver

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Miklos Szeredi]

> And a further question. The freezer is not atomic. What do you do
> if a task not yet frozen calls sys_sync(), but fuse is already frozen?

What do you do if a task not yet frozen writes to a pipe, on the other
end of which is a task already frozen?

It doesn't matter.  The only thing that should matter during suspend
(not hibernate) is saving the state of devices to ram, and putting the
devices to sleep.

I'm not sure why this can't be made atomic, but assuming, that it
can't, fuse should still not need to be implicated.  If it is, that's
an indication about something wrong in the suspend procedure.

Miklos

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Oliver Neukum]

Am Dienstag, 3. Juli 2007 schrieb Miklos Szeredi:
> > And a further question. The freezer is not atomic. What do you do
> > if a task not yet frozen calls sys_sync(), but fuse is already frozen?
> 
> What do you do if a task not yet frozen writes to a pipe, on the other
> end of which is a task already frozen?

The same as you do with a pipe when the reader is not ready.

> It doesn't matter.  The only thing that should matter during suspend
> (not hibernate) is saving the state of devices to ram, and putting the
> devices to sleep.

Well, but you did remove sys_sync() from the freezer, which is
and must be called in the hibernate path.
 
> I'm not sure why this can't be made atomic, but assuming, that it
> can't, fuse should still not need to be implicated.  If it is, that's
> an indication about something wrong in the suspend procedure.

Nope, something's wrong in fuse. You must be able to deal with sync
until every task is frozen.

	Regards
		Oliver

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Miklos Szeredi]

> Am Dienstag, 3. Juli 2007 schrieb Miklos Szeredi:
> > > And a further question. The freezer is not atomic. What do you do
> > > if a task not yet frozen calls sys_sync(), but fuse is already frozen?
> > 
> > What do you do if a task not yet frozen writes to a pipe, on the other
> > end of which is a task already frozen?
> 
> The same as you do with a pipe when the reader is not ready.

Yes.  And if the fuse process is not ready, what will sys_sync() do?

(As a side note, sys_sync() actually does nothing in fuse, so these
examples are slightly silly, but the concept applies to any filesystem
operation).

> > It doesn't matter.  The only thing that should matter during suspend
> > (not hibernate) is saving the state of devices to ram, and putting the
> > devices to sleep.
> 
> Well, but you did remove sys_sync() from the freezer, which is
> and must be called in the hibernate path.

Since I don't know why sync() needs to be called from hibernate, I
can't argue about this.  But for _suspend_ it definitely should not be
needed.

> > I'm not sure why this can't be made atomic, but assuming, that it
> > can't, fuse should still not need to be implicated.  If it is, that's
> > an indication about something wrong in the suspend procedure.
> 
> Nope, something's wrong in fuse. You must be able to deal with sync
> until every task is frozen.

Why exactly?  That's like saying the pipe must be able to deal with
writes until every task is frozen.

As I've said, fuse is just special kind of IPC.  The task calling into
fuse can assume nothing about when it's request will be served.  Just
like a task writing into a pipe can assume nothing about when that
write will return.

Miklos

Re: Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Alan Stern]

On Tue, 3 Jul 2007, Oliver Neukum wrote:

> Well, but you did remove sys_sync() from the freezer, which is
> and must be called in the hibernate path.

That's not really true.  We _want_ to call sys_sync() in both the 
hibernate and suspend paths (in case the batteries run down), to help 
avoid filesystem problems if something goes wrong with the resume.  But 
it isn't a hard requirement.

> > I'm not sure why this can't be made atomic, but assuming, that it
> > can't, fuse should still not need to be implicated.  If it is, that's
> > an indication about something wrong in the suspend procedure.
> 
> Nope, something's wrong in fuse. You must be able to deal with sync
> until every task is frozen.

That's ridiculous.  FUSE itself runs partially as a user task.  How can
you expect it to carry out a sync or anything else when it is frozen?

I suppose you could "deal" with it by having the kernel portion return
an error if the userspace part is frozen.  If the hibernate/suspend 
code bothered to check the return value, it would immediately abort 
the suspend.

Alan Stern

Re: Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Miklos Szeredi]

> > Well, but you did remove sys_sync() from the freezer, which is
> > and must be called in the hibernate path.
> 
> That's not really true.  We _want_ to call sys_sync() in both the 
> hibernate and suspend paths (in case the batteries run down), to help 
> avoid filesystem problems if something goes wrong with the resume.  But 
> it isn't a hard requirement.
> 
> > > I'm not sure why this can't be made atomic, but assuming, that it
> > > can't, fuse should still not need to be implicated.  If it is, that's
> > > an indication about something wrong in the suspend procedure.
> > 
> > Nope, something's wrong in fuse. You must be able to deal with sync
> > until every task is frozen.
> 
> That's ridiculous.  FUSE itself runs partially as a user task.  How can
> you expect it to carry out a sync or anything else when it is frozen?
> 
> I suppose you could "deal" with it by having the kernel portion return
> an error if the userspace part is frozen.  If the hibernate/suspend 
> code bothered to check the return value, it would immediately abort 
> the suspend.

I strongly believe, that we don't want to deal with it.  If we want to
call sync(), do it while the system is fully operational.  It's a best
effort thing anyway, and you can loose data in other ways if resume
fails.

Miklos

Re: Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Oliver Neukum]

Am Dienstag, 3. Juli 2007 schrieben Sie:
> On Tue, 3 Jul 2007, Oliver Neukum wrote:
> 
> > Well, but you did remove sys_sync() from the freezer, which is
> > and must be called in the hibernate path.
> 
> That's not really true.  We _want_ to call sys_sync() in both the 
> hibernate and suspend paths (in case the batteries run down), to help 
> avoid filesystem problems if something goes wrong with the resume.  But 
> it isn't a hard requirement.

But the ability to launder pages is needed. During hibernation we need
to shrink memory. I don't see how this would be fundamentally different
from calling sync.

> > > I'm not sure why this can't be made atomic, but assuming, that it
> > > can't, fuse should still not need to be implicated.  If it is, that's
> > > an indication about something wrong in the suspend procedure.
> > 
> > Nope, something's wrong in fuse. You must be able to deal with sync
> > until every task is frozen.
> 
> That's ridiculous.  FUSE itself runs partially as a user task.  How can
> you expect it to carry out a sync or anything else when it is frozen?

I don't and it might point to a fundamental problem.
But I cannot help but notice that syscalls may happen while the system
is partially frozen. It must be dealt with.

> I suppose you could "deal" with it by having the kernel portion return
> an error if the userspace part is frozen.  If the hibernate/suspend 
> code bothered to check the return value, it would immediately abort 
> the suspend.

Where exactly would that code notice the errors?

	Regards
		Oliver

Re: Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Rafael J. Wysocki]

On Tuesday, 3 July 2007 22:19, Miklos Szeredi wrote:
> > > Well, but you did remove sys_sync() from the freezer, which is
> > > and must be called in the hibernate path.
> > 
> > That's not really true.  We _want_ to call sys_sync() in both the 
> > hibernate and suspend paths (in case the batteries run down), to help 
> > avoid filesystem problems if something goes wrong with the resume.  But 
> > it isn't a hard requirement.
> > 
> > > > I'm not sure why this can't be made atomic, but assuming, that it
> > > > can't, fuse should still not need to be implicated.  If it is, that's
> > > > an indication about something wrong in the suspend procedure.
> > > 
> > > Nope, something's wrong in fuse. You must be able to deal with sync
> > > until every task is frozen.
> > 
> > That's ridiculous.  FUSE itself runs partially as a user task.  How can
> > you expect it to carry out a sync or anything else when it is frozen?
> > 
> > I suppose you could "deal" with it by having the kernel portion return
> > an error if the userspace part is frozen.  If the hibernate/suspend 
> > code bothered to check the return value, it would immediately abort 
> > the suspend.

Er, do_sync() doesn't return a result.

> I strongly believe, that we don't want to deal with it.  If we want to
> call sync(), do it while the system is fully operational.  It's a best
> effort thing anyway, and you can loose data in other ways if resume
> fails.

The requirement of syncing when the system (including the user space) is fully
operational is FUSE-specific.  Thus we'd rather like to sync FUSE filesystems
before freezing the user space and freeze the other filesystems after freezing
the user space.

Greetings,
Rafael


-- 
"Premature optimization is the root of all evil." - Donald Knuth

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Benjamin Herrenschmidt]

On Tue, 2007-07-03 at 21:32 +0200, Oliver Neukum wrote:
> > I'm not sure why this can't be made atomic, but assuming, that it
> > can't, fuse should still not need to be implicated.  If it is,
> that's
> > an indication about something wrong in the suspend procedure.
> 
> Nope, something's wrong in fuse. You must be able to deal with sync
> until every task is frozen. 

Pipe dream

Ben.

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Oliver Neukum]

Am Dienstag, 3. Juli 2007 schrieb Benjamin Herrenschmidt:
> On Tue, 2007-07-03 at 21:32 +0200, Oliver Neukum wrote:
> > > I'm not sure why this can't be made atomic, but assuming, that it
> > > can't, fuse should still not need to be implicated.  If it is,
> > that's
> > > an indication about something wrong in the suspend procedure.
> > 
> > Nope, something's wrong in fuse. You must be able to deal with sync
> > until every task is frozen. 
> 
> Pipe dream

Then tell me how you want to avoid that condition.

	Oliver

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Benjamin Herrenschmidt]

On Tue, 2007-07-03 at 23:48 +0200, Oliver Neukum wrote:
> Am Dienstag, 3. Juli 2007 schrieb Benjamin Herrenschmidt:
> > On Tue, 2007-07-03 at 21:32 +0200, Oliver Neukum wrote:
> > > > I'm not sure why this can't be made atomic, but assuming, that it
> > > > can't, fuse should still not need to be implicated.  If it is,
> > > that's
> > > > an indication about something wrong in the suspend procedure.
> > > 
> > > Nope, something's wrong in fuse. You must be able to deal with sync
> > > until every task is frozen. 
> > 
> > Pipe dream
> 
> Then tell me how you want to avoid that condition.

Don't freeze :-)

Ben.

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Oliver Neukum]

Am Dienstag, 3. Juli 2007 schrieb Benjamin Herrenschmidt:
> On Tue, 2007-07-03 at 23:48 +0200, Oliver Neukum wrote:
> > Am Dienstag, 3. Juli 2007 schrieb Benjamin Herrenschmidt:
> > > On Tue, 2007-07-03 at 21:32 +0200, Oliver Neukum wrote:
> > > > > I'm not sure why this can't be made atomic, but assuming, that it
> > > > > can't, fuse should still not need to be implicated.  If it is,
> > > > that's
> > > > > an indication about something wrong in the suspend procedure.
> > > > 
> > > > Nope, something's wrong in fuse. You must be able to deal with sync
> > > > until every task is frozen. 
> > > 
> > > Pipe dream
> > 
> > Then tell me how you want to avoid that condition.
> 
> Don't freeze :-)

Then you will have to deal with all syscalls unfrozen tasks can make.

	Regards
		Oliver

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Benjamin Herrenschmidt]

On Wed, 2007-07-04 at 00:04 +0200, Oliver Neukum wrote:
> Am Dienstag, 3. Juli 2007 schrieb Benjamin Herrenschmidt:
> > On Tue, 2007-07-03 at 23:48 +0200, Oliver Neukum wrote:
> > > Am Dienstag, 3. Juli 2007 schrieb Benjamin Herrenschmidt:
> > > > On Tue, 2007-07-03 at 21:32 +0200, Oliver Neukum wrote:
> > > > > > I'm not sure why this can't be made atomic, but assuming, that it
> > > > > > can't, fuse should still not need to be implicated.  If it is,
> > > > > that's
> > > > > > an indication about something wrong in the suspend procedure.
> > > > > 
> > > > > Nope, something's wrong in fuse. You must be able to deal with sync
> > > > > until every task is frozen. 
> > > > 
> > > > Pipe dream
> > > 
> > > Then tell me how you want to avoid that condition.
> > 
> > Don't freeze :-)
> 
> Then you will have to deal with all syscalls unfrozen tasks can make.

Yup, and the majority of them is totally harmless. Looks like people
around here have a problem with the idea of writing robust drivers ...

Ben.

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Oliver Neukum]

Am Mittwoch, 4. Juli 2007 schrieb Benjamin Herrenschmidt:
> > > > > > Nope, something's wrong in fuse. You must be able to deal with sync
> > > > > > until every task is frozen. 
> > > > > 
> > > > > Pipe dream
> > > > 
> > > > Then tell me how you want to avoid that condition.
> > > 
> > > Don't freeze :-)
> > 
> > Then you will have to deal with all syscalls unfrozen tasks can make.
> 
> Yup, and the majority of them is totally harmless. Looks like people
> around here have a problem with the idea of writing robust drivers ...

The majority is meaningless here. The subsystem works or works not.
Reliably failing to work is better than working most of the time.

What I am having a problem with is the rest of the system changing its
behavior and expecting the drivers to cope with that out of the blue.

	Oliver

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Pavel Machek]

On Tue 2007-07-03 21:32:20, Oliver Neukum wrote:
> Am Dienstag, 3. Juli 2007 schrieb Miklos Szeredi:
> > > And a further question. The freezer is not atomic. What do you do
> > > if a task not yet frozen calls sys_sync(), but fuse is already frozen?
> > 
> > What do you do if a task not yet frozen writes to a pipe, on the other
> > end of which is a task already frozen?

There's some difference between uninterruptible and interruptible
sleep I'd say.

> > It doesn't matter.  The only thing that should matter during suspend
> > (not hibernate) is saving the state of devices to ram, and putting the
> > devices to sleep.
> 
> Well, but you did remove sys_sync() from the freezer, which is
> and must be called in the hibernate path.

Not "must". In fact, hibernation should be safe without sys_sync(). It
is just user un-friendly.

									Pavel
-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Rafael J. Wysocki]

On Thursday, 5 July 2007 01:45, Pavel Machek wrote:
> On Tue 2007-07-03 21:32:20, Oliver Neukum wrote:
> > Am Dienstag, 3. Juli 2007 schrieb Miklos Szeredi:
> > > > And a further question. The freezer is not atomic. What do you do
> > > > if a task not yet frozen calls sys_sync(), but fuse is already frozen?
> > > 
> > > What do you do if a task not yet frozen writes to a pipe, on the other
> > > end of which is a task already frozen?
> 
> There's some difference between uninterruptible and interruptible
> sleep I'd say.
> 
> > > It doesn't matter.  The only thing that should matter during suspend
> > > (not hibernate) is saving the state of devices to ram, and putting the
> > > devices to sleep.
> > 
> > Well, but you did remove sys_sync() from the freezer, which is
> > and must be called in the hibernate path.
> 
> Not "must". In fact, hibernation should be safe without sys_sync(). It
> is just user un-friendly.

In fact, I'd like to remove the sys_sync() from the freezer entirely, because
it just doesn't belong in there.

The only advantege of having sys_sync() in freeze_processes() is that we
have a chance to write out everything when applications cannot produce more
data to write, but there are filesystems which don't do that anyway (eg. XFS),
so generally there's no reason to bother.

Greetings,
Rafael


-- 
"Premature optimization is the root of all evil." - Donald Knuth

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Nigel Cunningham]

[-- Attachment #1.1: Type: text/plain, Size: 2199 bytes --]

Hi.

On Thursday 05 July 2007 22:25:06 Rafael J. Wysocki wrote:
> On Thursday, 5 July 2007 01:45, Pavel Machek wrote:
> > On Tue 2007-07-03 21:32:20, Oliver Neukum wrote:
> > > Am Dienstag, 3. Juli 2007 schrieb Miklos Szeredi:
> > > > > And a further question. The freezer is not atomic. What do you do
> > > > > if a task not yet frozen calls sys_sync(), but fuse is already 
frozen?
> > > > 
> > > > What do you do if a task not yet frozen writes to a pipe, on the other
> > > > end of which is a task already frozen?
> > 
> > There's some difference between uninterruptible and interruptible
> > sleep I'd say.
> > 
> > > > It doesn't matter.  The only thing that should matter during suspend
> > > > (not hibernate) is saving the state of devices to ram, and putting the
> > > > devices to sleep.
> > > 
> > > Well, but you did remove sys_sync() from the freezer, which is
> > > and must be called in the hibernate path.
> > 
> > Not "must". In fact, hibernation should be safe without sys_sync(). It
> > is just user un-friendly.
> 
> In fact, I'd like to remove the sys_sync() from the freezer entirely, 
because
> it just doesn't belong in there.
> 
> The only advantege of having sys_sync() in freeze_processes() is that we
> have a chance to write out everything when applications cannot produce more
> data to write, but there are filesystems which don't do that anyway (eg. 
XFS),
> so generally there's no reason to bother.

Shouldn't XFS - and fuse - be considered to be broken? Sync should sync data 
and if XFS isn't doing that, it's wrong.

In the case of fuse, we should have a mechanism by which fuse processes can be 
made to sync if they do have any pending I/O, and by which they can be frozen 
later than other userspace processes.

I'd like to see the sync stay, because it improves reliability and data 
integrity in the fail-to-resume case. Calling scripts would probably invoke 
sync themselves if they don't already, but that's racy. As it is at the 
moment, we know userspace is stopped, so syncing isn't racy.

Regards,

Nigel
-- 
See http://www.tuxonice.net for Howtos, FAQs, mailing
lists, wiki and bugzilla info.

[-- Attachment #1.2: Type: application/pgp-signature, Size: 189 bytes --]

[-- Attachment #2: Type: text/plain, Size: 0 bytes --]

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Rafael J. Wysocki]

On Thursday, 5 July 2007 14:38, Nigel Cunningham wrote:
> Hi.
> 
> On Thursday 05 July 2007 22:25:06 Rafael J. Wysocki wrote:
> > On Thursday, 5 July 2007 01:45, Pavel Machek wrote:
> > > On Tue 2007-07-03 21:32:20, Oliver Neukum wrote:
> > > > Am Dienstag, 3. Juli 2007 schrieb Miklos Szeredi:
> > > > > > And a further question. The freezer is not atomic. What do you do
> > > > > > if a task not yet frozen calls sys_sync(), but fuse is already 
> frozen?
> > > > > 
> > > > > What do you do if a task not yet frozen writes to a pipe, on the other
> > > > > end of which is a task already frozen?
> > > 
> > > There's some difference between uninterruptible and interruptible
> > > sleep I'd say.
> > > 
> > > > > It doesn't matter.  The only thing that should matter during suspend
> > > > > (not hibernate) is saving the state of devices to ram, and putting the
> > > > > devices to sleep.
> > > > 
> > > > Well, but you did remove sys_sync() from the freezer, which is
> > > > and must be called in the hibernate path.
> > > 
> > > Not "must". In fact, hibernation should be safe without sys_sync(). It
> > > is just user un-friendly.
> > 
> > In fact, I'd like to remove the sys_sync() from the freezer entirely, 
> because
> > it just doesn't belong in there.
> > 
> > The only advantege of having sys_sync() in freeze_processes() is that we
> > have a chance to write out everything when applications cannot produce more
> > data to write, but there are filesystems which don't do that anyway (eg. 
> XFS),
> > so generally there's no reason to bother.
> 
> Shouldn't XFS - and fuse - be considered to be broken? Sync should sync data 
> and if XFS isn't doing that, it's wrong.
> 
> In the case of fuse, we should have a mechanism by which fuse processes can be 
> made to sync if they do have any pending I/O, and by which they can be frozen 
> later than other userspace processes.
> 
> I'd like to see the sync stay, because it improves reliability and data 
> integrity in the fail-to-resume case. Calling scripts would probably invoke 
> sync themselves if they don't already, but that's racy. As it is at the 
> moment, we know userspace is stopped, so syncing isn't racy.

I'd like to move the sync out of the freezer, but to call it from the
suspend/hibernation code, so that we do

sys_sync();
error = freeze_processes();

etc.

Greetings,
Rafael


-- 
"Premature optimization is the root of all evil." - Donald Knuth

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Nigel Cunningham]

[-- Attachment #1.1: Type: text/plain, Size: 3253 bytes --]

Hi.

On Thursday 05 July 2007 23:35:45 Rafael J. Wysocki wrote:
> On Thursday, 5 July 2007 14:38, Nigel Cunningham wrote:
> > On Thursday 05 July 2007 22:25:06 Rafael J. Wysocki wrote:
> > > On Thursday, 5 July 2007 01:45, Pavel Machek wrote:
> > > > On Tue 2007-07-03 21:32:20, Oliver Neukum wrote:
> > > > > Am Dienstag, 3. Juli 2007 schrieb Miklos Szeredi:
> > > > > > > And a further question. The freezer is not atomic. What do you 
do
> > > > > > > if a task not yet frozen calls sys_sync(), but fuse is already 
> > frozen?
> > > > > > 
> > > > > > What do you do if a task not yet frozen writes to a pipe, on the 
other
> > > > > > end of which is a task already frozen?
> > > > 
> > > > There's some difference between uninterruptible and interruptible
> > > > sleep I'd say.
> > > > 
> > > > > > It doesn't matter.  The only thing that should matter during 
suspend
> > > > > > (not hibernate) is saving the state of devices to ram, and putting 
the
> > > > > > devices to sleep.
> > > > > 
> > > > > Well, but you did remove sys_sync() from the freezer, which is
> > > > > and must be called in the hibernate path.
> > > > 
> > > > Not "must". In fact, hibernation should be safe without sys_sync(). It
> > > > is just user un-friendly.
> > > 
> > > In fact, I'd like to remove the sys_sync() from the freezer entirely, 
> > because
> > > it just doesn't belong in there.
> > > 
> > > The only advantege of having sys_sync() in freeze_processes() is that we
> > > have a chance to write out everything when applications cannot produce 
more
> > > data to write, but there are filesystems which don't do that anyway (eg. 
> > XFS),
> > > so generally there's no reason to bother.
> > 
> > Shouldn't XFS - and fuse - be considered to be broken? Sync should sync 
data 
> > and if XFS isn't doing that, it's wrong.
> > 
> > In the case of fuse, we should have a mechanism by which fuse processes 
can be 
> > made to sync if they do have any pending I/O, and by which they can be 
frozen 
> > later than other userspace processes.
> > 
> > I'd like to see the sync stay, because it improves reliability and data 
> > integrity in the fail-to-resume case. Calling scripts would probably 
invoke 
> > sync themselves if they don't already, but that's racy. As it is at the 
> > moment, we know userspace is stopped, so syncing isn't racy.
> 
> I'd like to move the sync out of the freezer, but to call it from the
> suspend/hibernation code, so that we do
> 
> sys_sync();
> error = freeze_processes();

Yeah, I understand that. The problem then is that you're racing against 
userspace. That's not usually a problem, but that doesn't mean it's never a 
problem. Try running the stress suite while testing hibernating and you'll 
see what I mean. If something is submitting lots of I/O when you try to 
suspend, your sync call will race against that process if it's not yet 
frozen, and its continued activity will make your sync pointless (there'll be 
more unsynced data when you sys_sync call finishes). Stopping userspace 
before syncing removes that race.

Regards,

Nigel
-- 
See http://www.tuxonice.net for Howtos, FAQs, mailing
lists, wiki and bugzilla info.

[-- Attachment #1.2: Type: application/pgp-signature, Size: 189 bytes --]

[-- Attachment #2: Type: text/plain, Size: 0 bytes --]

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Rafael J. Wysocki]

Hi,

On Thursday, 5 July 2007 15:36, Nigel Cunningham wrote:
> Hi.
> 
> On Thursday 05 July 2007 23:35:45 Rafael J. Wysocki wrote:
> > On Thursday, 5 July 2007 14:38, Nigel Cunningham wrote:
> > > On Thursday 05 July 2007 22:25:06 Rafael J. Wysocki wrote:
> > > > On Thursday, 5 July 2007 01:45, Pavel Machek wrote:
> > > > > On Tue 2007-07-03 21:32:20, Oliver Neukum wrote:
> > > > > > Am Dienstag, 3. Juli 2007 schrieb Miklos Szeredi:
> > > > > > > > And a further question. The freezer is not atomic. What do you 
> do
> > > > > > > > if a task not yet frozen calls sys_sync(), but fuse is already 
> > > frozen?
> > > > > > > 
> > > > > > > What do you do if a task not yet frozen writes to a pipe, on the 
> other
> > > > > > > end of which is a task already frozen?
> > > > > 
> > > > > There's some difference between uninterruptible and interruptible
> > > > > sleep I'd say.
> > > > > 
> > > > > > > It doesn't matter.  The only thing that should matter during 
> suspend
> > > > > > > (not hibernate) is saving the state of devices to ram, and putting 
> the
> > > > > > > devices to sleep.
> > > > > > 
> > > > > > Well, but you did remove sys_sync() from the freezer, which is
> > > > > > and must be called in the hibernate path.
> > > > > 
> > > > > Not "must". In fact, hibernation should be safe without sys_sync(). It
> > > > > is just user un-friendly.
> > > > 
> > > > In fact, I'd like to remove the sys_sync() from the freezer entirely, 
> > > because
> > > > it just doesn't belong in there.
> > > > 
> > > > The only advantege of having sys_sync() in freeze_processes() is that we
> > > > have a chance to write out everything when applications cannot produce 
> more
> > > > data to write, but there are filesystems which don't do that anyway (eg. 
> > > XFS),
> > > > so generally there's no reason to bother.
> > > 
> > > Shouldn't XFS - and fuse - be considered to be broken? Sync should sync 
> data 
> > > and if XFS isn't doing that, it's wrong.
> > > 
> > > In the case of fuse, we should have a mechanism by which fuse processes 
> can be 
> > > made to sync if they do have any pending I/O, and by which they can be 
> frozen 
> > > later than other userspace processes.
> > > 
> > > I'd like to see the sync stay, because it improves reliability and data 
> > > integrity in the fail-to-resume case. Calling scripts would probably 
> invoke 
> > > sync themselves if they don't already, but that's racy. As it is at the 
> > > moment, we know userspace is stopped, so syncing isn't racy.
> > 
> > I'd like to move the sync out of the freezer, but to call it from the
> > suspend/hibernation code, so that we do
> > 
> > sys_sync();
> > error = freeze_processes();
> 
> Yeah, I understand that. The problem then is that you're racing against 
> userspace. That's not usually a problem, but that doesn't mean it's never a 
> problem. Try running the stress suite while testing hibernating and you'll 
> see what I mean. If something is submitting lots of I/O when you try to 
> suspend, your sync call will race against that process if it's not yet 
> frozen, and its continued activity will make your sync pointless (there'll be 
> more unsynced data when you sys_sync call finishes). Stopping userspace 
> before syncing removes that race.

Yes, that will make the suspend/hibernation less reliable in case the resume
fails (some data, written after the sync, may be lost).  However, the sync done
from within the freezer doesn't guarantee that there are no data lost anyway,
so we don't lose much by not doing it.

Now, there's a question how much data may be lost, potentially, if we do the
sync before the freezer and I don't think that's a lot.

Greetings,
Rafael


-- 
"Premature optimization is the root of all evil." - Donald Knuth

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Nigel Cunningham]

[-- Attachment #1.1: Type: text/plain, Size: 4382 bytes --]

Good morning!

On Thursday 05 July 2007 23:59:57 Rafael J. Wysocki wrote:
> On Thursday, 5 July 2007 15:36, Nigel Cunningham wrote:
> > On Thursday 05 July 2007 23:35:45 Rafael J. Wysocki wrote:
> > > On Thursday, 5 July 2007 14:38, Nigel Cunningham wrote:
> > > > On Thursday 05 July 2007 22:25:06 Rafael J. Wysocki wrote:
> > > > > On Thursday, 5 July 2007 01:45, Pavel Machek wrote:
> > > > > > On Tue 2007-07-03 21:32:20, Oliver Neukum wrote:
> > > > > > > Am Dienstag, 3. Juli 2007 schrieb Miklos Szeredi:
> > > > > > > > > And a further question. The freezer is not atomic. What do 
you 
> > do
> > > > > > > > > if a task not yet frozen calls sys_sync(), but fuse is 
already 
> > > > frozen?
> > > > > > > > 
> > > > > > > > What do you do if a task not yet frozen writes to a pipe, on 
the 
> > other
> > > > > > > > end of which is a task already frozen?
> > > > > > 
> > > > > > There's some difference between uninterruptible and interruptible
> > > > > > sleep I'd say.
> > > > > > 
> > > > > > > > It doesn't matter.  The only thing that should matter during 
> > suspend
> > > > > > > > (not hibernate) is saving the state of devices to ram, and 
putting 
> > the
> > > > > > > > devices to sleep.
> > > > > > > 
> > > > > > > Well, but you did remove sys_sync() from the freezer, which is
> > > > > > > and must be called in the hibernate path.
> > > > > > 
> > > > > > Not "must". In fact, hibernation should be safe without 
sys_sync(). It
> > > > > > is just user un-friendly.
> > > > > 
> > > > > In fact, I'd like to remove the sys_sync() from the freezer 
entirely, 
> > > > because
> > > > > it just doesn't belong in there.
> > > > > 
> > > > > The only advantege of having sys_sync() in freeze_processes() is 
that we
> > > > > have a chance to write out everything when applications cannot 
produce 
> > more
> > > > > data to write, but there are filesystems which don't do that anyway 
(eg. 
> > > > XFS),
> > > > > so generally there's no reason to bother.
> > > > 
> > > > Shouldn't XFS - and fuse - be considered to be broken? Sync should 
sync 
> > data 
> > > > and if XFS isn't doing that, it's wrong.
> > > > 
> > > > In the case of fuse, we should have a mechanism by which fuse 
processes 
> > can be 
> > > > made to sync if they do have any pending I/O, and by which they can be 
> > frozen 
> > > > later than other userspace processes.
> > > > 
> > > > I'd like to see the sync stay, because it improves reliability and 
data 
> > > > integrity in the fail-to-resume case. Calling scripts would probably 
> > invoke 
> > > > sync themselves if they don't already, but that's racy. As it is at 
the 
> > > > moment, we know userspace is stopped, so syncing isn't racy.
> > > 
> > > I'd like to move the sync out of the freezer, but to call it from the
> > > suspend/hibernation code, so that we do
> > > 
> > > sys_sync();
> > > error = freeze_processes();
> > 
> > Yeah, I understand that. The problem then is that you're racing against 
> > userspace. That's not usually a problem, but that doesn't mean it's never 
a 
> > problem. Try running the stress suite while testing hibernating and you'll 
> > see what I mean. If something is submitting lots of I/O when you try to 
> > suspend, your sync call will race against that process if it's not yet 
> > frozen, and its continued activity will make your sync pointless (there'll 
be 
> > more unsynced data when you sys_sync call finishes). Stopping userspace 
> > before syncing removes that race.
> 
> Yes, that will make the suspend/hibernation less reliable in case the resume
> fails (some data, written after the sync, may be lost).  However, the sync 
done
> from within the freezer doesn't guarantee that there are no data lost 
anyway,
> so we don't lose much by not doing it.
> 
> Now, there's a question how much data may be lost, potentially, if we do the
> sync before the freezer and I don't think that's a lot.

You're missing the point. I'm arguing that a sync from within the freezer 
should guarantee that there is no data loss. As I said about, XFS should be 
fixed to properly sync its data, and something should be done about fuse 
filesystems too.

Regards,

Nigel
-- 
See http://www.tuxonice.net for Howtos, FAQs, mailing
lists, wiki and bugzilla info.

[-- Attachment #1.2: Type: application/pgp-signature, Size: 189 bytes --]

[-- Attachment #2: Type: text/plain, Size: 0 bytes --]

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Miklos Szeredi]

> > You're missing the point. I'm arguing that a sync from within the freezer 
> > should guarantee that there is no data loss.
> 
> Well, it should, but it doesn't ...
> 
> Moreover, if FUSE implements syncing, then the sync from within the freezer
> will almost certainly deadlock.

Rafael, think positively: by the time fuse implements sync(), the
freezer will be long gone ;)

Miklos

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Oliver Neukum]

Am Freitag, 6. Juli 2007 schrieb Miklos Szeredi:
> > > You're missing the point. I'm arguing that a sync from within the freezer 
> > > should guarantee that there is no data loss.
> > 
> > Well, it should, but it doesn't ...
> > 
> > Moreover, if FUSE implements syncing, then the sync from within the freezer
> > will almost certainly deadlock.
> 
> Rafael, think positively: by the time fuse implements sync(), the
> freezer will be long gone ;)

Now you are entering really dangerous territory.
If you can implement a meaningfull sync method, you must have dirty
pages in the page cache. That means you are in the page freeing path
of the vm. Then we are in real trouble. Don't even think about it.

As far as suspend/hibernate is concerned, get yourself on the new
notifying chain and revert to synchronous operations when notified.

	Regards
		Oliver

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Miklos Szeredi]

> > > Moreover, if FUSE implements syncing, then the sync from within the freezer
> > > will almost certainly deadlock.
> > 
> > Rafael, think positively: by the time fuse implements sync(), the
> > freezer will be long gone ;)
> 
> Now you are entering really dangerous territory.
> If you can implement a meaningfull sync method, you must have dirty
> pages in the page cache. That means you are in the page freeing path
> of the vm. Then we are in real trouble. Don't even think about it.

VM induced deadlocks are real nasty.  I have thought about them a lot
already.  Suspend shouldn't introduce any big surprises.

Miklos

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Nigel Cunningham]

[-- Attachment #1.1: Type: text/plain, Size: 885 bytes --]

Hi.

On Friday 06 July 2007 19:09:43 Miklos Szeredi wrote:
> > > > Moreover, if FUSE implements syncing, then the sync from within the 
freezer
> > > > will almost certainly deadlock.
> > > 
> > > Rafael, think positively: by the time fuse implements sync(), the
> > > freezer will be long gone ;)
> > 
> > Now you are entering really dangerous territory.
> > If you can implement a meaningfull sync method, you must have dirty
> > pages in the page cache. That means you are in the page freeing path
> > of the vm. Then we are in real trouble. Don't even think about it.
> 
> VM induced deadlocks are real nasty.  I have thought about them a lot
> already.  Suspend shouldn't introduce any big surprises.

Suspend isn't introducing the surprises. Fuse is. It creates the potential 
deadlocks simply by existing (this isn't suspend or hibernate specific).

Nigel

[-- Attachment #1.2: Type: application/pgp-signature, Size: 189 bytes --]

[-- Attachment #2: Type: text/plain, Size: 0 bytes --]

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Miklos Szeredi]

> > VM induced deadlocks are real nasty.  I have thought about them a lot
> > already.  Suspend shouldn't introduce any big surprises.
> 
> Suspend isn't introducing the surprises. Fuse is. It creates the potential 
> deadlocks simply by existing (this isn't suspend or hibernate specific).

Yes.  And fuse is not going away any time soon.  Get over it ;)

Miklos

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Rafael J. Wysocki]

Hi,

On Thursday, 5 July 2007 23:49, Nigel Cunningham wrote:
> Good morning!
> 
> On Thursday 05 July 2007 23:59:57 Rafael J. Wysocki wrote:
> > On Thursday, 5 July 2007 15:36, Nigel Cunningham wrote:
> > > On Thursday 05 July 2007 23:35:45 Rafael J. Wysocki wrote:
> > > > On Thursday, 5 July 2007 14:38, Nigel Cunningham wrote:
> > > > > On Thursday 05 July 2007 22:25:06 Rafael J. Wysocki wrote:
> > > > > > On Thursday, 5 July 2007 01:45, Pavel Machek wrote:
> > > > > > > On Tue 2007-07-03 21:32:20, Oliver Neukum wrote:
> > > > > > > > Am Dienstag, 3. Juli 2007 schrieb Miklos Szeredi:
> > > > > > > > > > And a further question. The freezer is not atomic. What do 
> you 
> > > do
> > > > > > > > > > if a task not yet frozen calls sys_sync(), but fuse is 
> already 
> > > > > frozen?
> > > > > > > > > 
> > > > > > > > > What do you do if a task not yet frozen writes to a pipe, on 
> the 
> > > other
> > > > > > > > > end of which is a task already frozen?
> > > > > > > 
> > > > > > > There's some difference between uninterruptible and interruptible
> > > > > > > sleep I'd say.
> > > > > > > 
> > > > > > > > > It doesn't matter.  The only thing that should matter during 
> > > suspend
> > > > > > > > > (not hibernate) is saving the state of devices to ram, and 
> putting 
> > > the
> > > > > > > > > devices to sleep.
> > > > > > > > 
> > > > > > > > Well, but you did remove sys_sync() from the freezer, which is
> > > > > > > > and must be called in the hibernate path.
> > > > > > > 
> > > > > > > Not "must". In fact, hibernation should be safe without 
> sys_sync(). It
> > > > > > > is just user un-friendly.
> > > > > > 
> > > > > > In fact, I'd like to remove the sys_sync() from the freezer 
> entirely, 
> > > > > because
> > > > > > it just doesn't belong in there.
> > > > > > 
> > > > > > The only advantege of having sys_sync() in freeze_processes() is 
> that we
> > > > > > have a chance to write out everything when applications cannot 
> produce 
> > > more
> > > > > > data to write, but there are filesystems which don't do that anyway 
> (eg. 
> > > > > XFS),
> > > > > > so generally there's no reason to bother.
> > > > > 
> > > > > Shouldn't XFS - and fuse - be considered to be broken? Sync should 
> sync 
> > > data 
> > > > > and if XFS isn't doing that, it's wrong.
> > > > > 
> > > > > In the case of fuse, we should have a mechanism by which fuse 
> processes 
> > > can be 
> > > > > made to sync if they do have any pending I/O, and by which they can be 
> > > frozen 
> > > > > later than other userspace processes.
> > > > > 
> > > > > I'd like to see the sync stay, because it improves reliability and 
> data 
> > > > > integrity in the fail-to-resume case. Calling scripts would probably 
> > > invoke 
> > > > > sync themselves if they don't already, but that's racy. As it is at 
> the 
> > > > > moment, we know userspace is stopped, so syncing isn't racy.
> > > > 
> > > > I'd like to move the sync out of the freezer, but to call it from the
> > > > suspend/hibernation code, so that we do
> > > > 
> > > > sys_sync();
> > > > error = freeze_processes();
> > > 
> > > Yeah, I understand that. The problem then is that you're racing against 
> > > userspace. That's not usually a problem, but that doesn't mean it's never 
> a 
> > > problem. Try running the stress suite while testing hibernating and you'll 
> > > see what I mean. If something is submitting lots of I/O when you try to 
> > > suspend, your sync call will race against that process if it's not yet 
> > > frozen, and its continued activity will make your sync pointless (there'll 
> be 
> > > more unsynced data when you sys_sync call finishes). Stopping userspace 
> > > before syncing removes that race.
> > 
> > Yes, that will make the suspend/hibernation less reliable in case the resume
> > fails (some data, written after the sync, may be lost).  However, the sync 
> done
> > from within the freezer doesn't guarantee that there are no data lost 
> anyway,
> > so we don't lose much by not doing it.
> > 
> > Now, there's a question how much data may be lost, potentially, if we do the
> > sync before the freezer and I don't think that's a lot.
> 
> You're missing the point. I'm arguing that a sync from within the freezer 
> should guarantee that there is no data loss.

Well, it should, but it doesn't ...

Moreover, if FUSE implements syncing, then the sync from within the freezer
will almost certainly deadlock.

> As I said about, XFS should be fixed to properly sync its data, and
> something should be done about fuse filesystems too.

I think that we can't do anything about it, so we should just live with it.

Greetings,
Rafael


-- 
"Premature optimization is the root of all evil." - Donald Knuth

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Rafael J. Wysocki]

On Tuesday, 3 July 2007 20:26, Oliver Neukum wrote:
> Am Dienstag, 3. Juli 2007 schrieb Rafael J. Wysocki:
> > > > The main reason for deadlocks is because we do a sys_sync() after the
> > > > freeze, which we shouldn't do.
> > > 
> > > So why don't we remove the sys_sync() from freeze_processes() instead?
> > 
> > The patch follows (untested).
> 
> And a further question. The freezer is not atomic. What do you do
> if a task not yet frozen calls sys_sync(), but fuse is already frozen?

Hmm, if the sync is interruptible (I'm not sure), the task should be frozen while
waiting for it to complete.

Otherwise, the freezing of tasks will fail (no deadlock).

Greetings,
Rafael


-- 
"Premature optimization is the root of all evil." - Donald Knuth

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Pavel Machek]

Hi!

> > > The main reason for deadlocks is because we do a sys_sync() after the
> > > freeze, which we shouldn't do.
> > 
> > So why don't we remove the sys_sync() from freeze_processes() instead?
> 
> The patch follows (untested).
> 
> Greetings,
> Rafael
> 
> 
> ---
> From: Rafael J. Wysocki <rjw@sisk.pl>
> 
> We shouldn't sync filesystems from within the freezer, because it's not needed
> for suspend to RAM and leads to problems with FUSE.

Actually... It is not _needed_ for suspend to disk, either. Snapshot is
atomic, so it should be okay to suspend with filesystems dirty.

_But_, if anything goes wrong, we'd prefer to have at least
filesystems synced. Battery running out during s2ram is not quite
uncommon, so we perhaps should do sync somewhere there. (But we can do
it before freezer just fine).
								Pavel

-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Rafael J. Wysocki]

On Tuesday, 3 July 2007 21:27, Pavel Machek wrote:
> Hi!
> 
> > > > The main reason for deadlocks is because we do a sys_sync() after the
> > > > freeze, which we shouldn't do.
> > > 
> > > So why don't we remove the sys_sync() from freeze_processes() instead?
> > 
> > The patch follows (untested).
> > 
> > Greetings,
> > Rafael
> > 
> > 
> > ---
> > From: Rafael J. Wysocki <rjw@sisk.pl>
> > 
> > We shouldn't sync filesystems from within the freezer, because it's not needed
> > for suspend to RAM and leads to problems with FUSE.
> 
> Actually... It is not _needed_ for suspend to disk, either. Snapshot is
> atomic, so it should be okay to suspend with filesystems dirty.
> 
> _But_, if anything goes wrong, we'd prefer to have at least
> filesystems synced. Battery running out during s2ram is not quite
> uncommon, so we perhaps should do sync somewhere there. (But we can do
> it before freezer just fine).

OK

So, should I add the sync() to suspend_prepare(), before freeze_processes()
(in analogy with hibernate())?

Greetings
Rafael


-- 
"Premature optimization is the root of all evil." - Donald Knuth

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Benjamin Herrenschmidt]

On Tue, 2007-07-03 at 15:08 +0200, Rafael J. Wysocki wrote:
> On Tuesday, 3 July 2007 07:51, Benjamin Herrenschmidt wrote:
> > On Tue, 2007-07-03 at 05:29 +0100, Matthew Garrett wrote:
> > > Suspend to RAM on a machine with / on a fuse filesystem turns out to be 
> > > a screaming nightmare - either the suspend fails because syslog (for 
> > > instance) can't be frozen, or the machine deadlocks for some other 
> > > reason I haven't tracked down. We could "fix" fuse, or alternatively we 
> > > could do what we do for suspend to RAM on other platforms (PPC and APM) 
> > > and just not use the freezer.
> > 
> > The main reason for deadlocks is because we do a sys_sync() after the
> > freeze, which we shouldn't do.
> 
> So why don't we remove the sys_sync() from freeze_processes() instead?

Or rather, move it to before freeze_processes() so that you get at least
some level of sync'ing just in case you never wake up. Not perfect
(things may still get dirty) but it will help the immediate issue.

Ben.

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Alan Stern]

On Tue, 3 Jul 2007, Matthew Garrett wrote:

> Suspend to RAM on a machine with / on a fuse filesystem turns out to be 
> a screaming nightmare - either the suspend fails because syslog (for 
> instance) can't be frozen, or the machine deadlocks for some other 
> reason I haven't tracked down. We could "fix" fuse, or alternatively we 
> could do what we do for suspend to RAM on other platforms (PPC and APM) 
> and just not use the freezer.

Quite apart from the sync() matter, _any_ synchronous call to a FUSE 
filesystem during STR will cause trouble.  Even if the user task 
implementing the filesystem isn't frozen, when it tries to carry out 
some I/O to a suspended device it will either:

	block until the system wakes up, or

	cause the suspend to abort.

Neither outcome is desirable.

Alan Stern

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Matthew Garrett]

On Tue, Jul 03, 2007 at 12:03:33PM -0400, Alan Stern wrote:
> On Tue, 3 Jul 2007, Matthew Garrett wrote:
> 
> > Suspend to RAM on a machine with / on a fuse filesystem turns out to be 
> > a screaming nightmare - either the suspend fails because syslog (for 
> > instance) can't be frozen, or the machine deadlocks for some other 
> > reason I haven't tracked down. We could "fix" fuse, or alternatively we 
> > could do what we do for suspend to RAM on other platforms (PPC and APM) 
> > and just not use the freezer.
> 
> Quite apart from the sync() matter, _any_ synchronous call to a FUSE 
> filesystem during STR will cause trouble.  Even if the user task 
> implementing the filesystem isn't frozen, when it tries to carry out 
> some I/O to a suspended device it will either:
> 
> 	block until the system wakes up, or

For the suspend to RAM case, that sounds absolutely fine.

-- 
Matthew Garrett | mjg59@srcf.ucam.org

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Alan Stern]

On Tue, 3 Jul 2007, Matthew Garrett wrote:

> On Tue, Jul 03, 2007 at 12:03:33PM -0400, Alan Stern wrote:
> > On Tue, 3 Jul 2007, Matthew Garrett wrote:
> > 
> > > Suspend to RAM on a machine with / on a fuse filesystem turns out to be 
> > > a screaming nightmare - either the suspend fails because syslog (for 
> > > instance) can't be frozen, or the machine deadlocks for some other 
> > > reason I haven't tracked down. We could "fix" fuse, or alternatively we 
> > > could do what we do for suspend to RAM on other platforms (PPC and APM) 
> > > and just not use the freezer.
> > 
> > Quite apart from the sync() matter, _any_ synchronous call to a FUSE 
> > filesystem during STR will cause trouble.  Even if the user task 
> > implementing the filesystem isn't frozen, when it tries to carry out 
> > some I/O to a suspended device it will either:
> > 
> > 	block until the system wakes up, or
> 
> For the suspend to RAM case, that sounds absolutely fine.

It's not so good when your suspend process has to wait for the call to 
complete!

Alan Stern

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Matthew Garrett]

On Tue, Jul 03, 2007 at 12:57:17PM -0400, Alan Stern wrote:
> On Tue, 3 Jul 2007, Matthew Garrett wrote:
> > On Tue, Jul 03, 2007 at 12:03:33PM -0400, Alan Stern wrote:
> > > Quite apart from the sync() matter, _any_ synchronous call to a FUSE 
> > > filesystem during STR will cause trouble.  Even if the user task 
> > > implementing the filesystem isn't frozen, when it tries to carry out 
> > > some I/O to a suspended device it will either:
> > > 
> > > 	block until the system wakes up, or
> > 
> > For the suspend to RAM case, that sounds absolutely fine.
> 
> It's not so good when your suspend process has to wait for the call to 
> complete!

Why would it have to? Sorry, I suspect I'm missing something obvious 
here.

-- 
Matthew Garrett | mjg59@srcf.ucam.org

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Alan Stern]

On Tue, 3 Jul 2007, Matthew Garrett wrote:

> On Tue, Jul 03, 2007 at 12:57:17PM -0400, Alan Stern wrote:
> > On Tue, 3 Jul 2007, Matthew Garrett wrote:
> > > On Tue, Jul 03, 2007 at 12:03:33PM -0400, Alan Stern wrote:
> > > > Quite apart from the sync() matter, _any_ synchronous call to a FUSE 
> > > > filesystem during STR will cause trouble.  Even if the user task 
> > > > implementing the filesystem isn't frozen, when it tries to carry out 
> > > > some I/O to a suspended device it will either:
> > > > 
> > > > 	block until the system wakes up, or
> > > 
> > > For the suspend to RAM case, that sounds absolutely fine.
> > 
> > It's not so good when your suspend process has to wait for the call to 
> > complete!
> 
> Why would it have to? Sorry, I suspect I'm missing something obvious 
> here.

Well, the sys_sync() that caused your original problem did exactly 
that.  It's the reason you get deadlocks, right?

I agree that in general the suspend process should not have to wait for 
a userspace callback to complete.  Indeed, there's no particular 
reason that anything running during STR should have to wait for 
something in userspace to complete.  Given that fact, I don't see 
anything wrong with freezing userspace when doing STR.

Alan Stern

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Matthew Garrett]

On Tue, Jul 03, 2007 at 03:33:40PM -0400, Alan Stern wrote:
> On Tue, 3 Jul 2007, Matthew Garrett wrote:
> 
> > On Tue, Jul 03, 2007 at 12:57:17PM -0400, Alan Stern wrote:
> > > On Tue, 3 Jul 2007, Matthew Garrett wrote:
> > > > For the suspend to RAM case, that sounds absolutely fine.
> > > 
> > > It's not so good when your suspend process has to wait for the call to 
> > > complete!
> > 
> > Why would it have to? Sorry, I suspect I'm missing something obvious 
> > here.
> 
> Well, the sys_sync() that caused your original problem did exactly 
> that.  It's the reason you get deadlocks, right?

The sys_sync is unnecessary in the first case. There shouldn't be 
anything in the suspend path that's going to require userspace access to 
a device after that device has been suspended.

> I agree that in general the suspend process should not have to wait for 
> a userspace callback to complete.  Indeed, there's no particular 
> reason that anything running during STR should have to wait for 
> something in userspace to complete.  Given that fact, I don't see 
> anything wrong with freezing userspace when doing STR.

There's nothing wrong with it as such, it's just that our implementation 
appears to suck in a myriad of small ways that keep cropping up and 
biting people. Even without the sys_sync(), freezing processes results 
in the suspend failing because syslog is stuck in D state and won't go 
into the refrigerator.

-- 
Matthew Garrett | mjg59@srcf.ucam.org

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Alan Stern]

On Tue, 3 Jul 2007, Matthew Garrett wrote:

> > I agree that in general the suspend process should not have to wait for 
> > a userspace callback to complete.  Indeed, there's no particular 
> > reason that anything running during STR should have to wait for 
> > something in userspace to complete.  Given that fact, I don't see 
> > anything wrong with freezing userspace when doing STR.
> 
> There's nothing wrong with it as such, it's just that our implementation 
> appears to suck in a myriad of small ways that keep cropping up and 
> biting people. Even without the sys_sync(), freezing processes results 
> in the suspend failing because syslog is stuck in D state and won't go 
> into the refrigerator.

Okay, I can believe that.  The proper response then is to fix the 
freezer, not eliminate it.  Has the syslog problem been reported on 
linux-pm?  I don't recall hearing of it before.

Alan Stern

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Matthew Garrett]

On Tue, Jul 03, 2007 at 03:54:55PM -0400, Alan Stern wrote:
> On Tue, 3 Jul 2007, Matthew Garrett wrote:
> > There's nothing wrong with it as such, it's just that our implementation 
> > appears to suck in a myriad of small ways that keep cropping up and 
> > biting people. Even without the sys_sync(), freezing processes results 
> > in the suspend failing because syslog is stuck in D state and won't go 
> > into the refrigerator.
> 
> Okay, I can believe that.  The proper response then is to fix the 
> freezer, not eliminate it.  Has the syslog problem been reported on 
> linux-pm?  I don't recall hearing of it before.

See the start of this thread. It's just not clear what the freezer buys 
us - removing it gets rid of a load of subtle issues and complexity, and 
turns system suspend into something that looks more like runtime 
suspend (which might then encourage people to get runtime suspend 
right...)

-- 
Matthew Garrett | mjg59@srcf.ucam.org

Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Pavel Machek]

Hi!

> Suspend to RAM on a machine with / on a fuse filesystem turns out to be 
> a screaming nightmare - either the suspend fails because syslog (for 
> instance) can't be frozen, or the machine deadlocks for some other 
> reason I haven't tracked down. We could "fix" fuse, or alternatively we 
> could do what we do for suspend to RAM on other platforms (PPC and APM) 
> and just not use the freezer.
> 
> Signed-off-by: Matthew Garrett <mjg59@srcf.ucam.org>

Sorry, no.

* this needs audit of all drivers. Or we can just merge it and then
fix all the problems it causes. If you are willing to become suspend
maintainer and handle all that mess, perhaps we can do this.

* it does not solve FUSE vs. hibernation

* it does not solve FUSE vs. suspend-to-both

* userspace will now see CPUs going up and down at minimum

Now, we want to do something like this long-term, but I do not think
we can just remove the freezer like this.


								Pavel
-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

sysrq-t dumps of s2ram/fuse deadlock (was Re: [PATCH] Remove process freezer from suspend to RAM pathway)

[Pavel Machek]

Hi!

Can we get them? They are neccessary for debugging 'what in suspend
calls fuse' problem. And yes, that problem is there even when you
remove freezer.

Matthew, you seem to be the only one able to produce them...
							Pavel

-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

Re: sysrq-t dumps of s2ram/fuse deadlock (was Re: [PATCH] Remove process freezer from suspend to RAM pathway)

[Matthew Garrett]

On Mon, Jul 09, 2007 at 01:29:05PM +0000, Pavel Machek wrote:
> Hi!
> 
> Can we get them? They are neccessary for debugging 'what in suspend
> calls fuse' problem. And yes, that problem is there even when you
> remove freezer.

I can produce them, but haven't managed to do that in any way that lets 
me get them off the system yet.

> Matthew, you seem to be the only one able to produce them...

sys_sync may not do anything on a fuse filesystem, but if you've 
loopback-mounted an ext3 filesystem from that fuse filesystem then it's 
going to result in writes to it and deadlock.

-- 
Matthew Garrett | mjg59@srcf.ucam.org

Re: sysrq-t dumps of s2ram/fuse deadlock (was Re: [PATCH] Remove process freezer from suspend to RAM pathway)

[Pavel Machek]

On Wed 2007-07-11 14:28:20, Matthew Garrett wrote:
> On Mon, Jul 09, 2007 at 01:29:05PM +0000, Pavel Machek wrote:
> > Hi!
> > 
> > Can we get them? They are neccessary for debugging 'what in suspend
> > calls fuse' problem. And yes, that problem is there even when you
> > remove freezer.
> 
> I can produce them, but haven't managed to do that in any way that lets 
> me get them off the system yet.
> 
> > Matthew, you seem to be the only one able to produce them...
> 
> sys_sync may not do anything on a fuse filesystem, but if you've 
> loopback-mounted an ext3 filesystem from that fuse filesystem then it's 
> going to result in writes to it and deadlock.

Aha, yes, that explains the problem. Thanks!
									Pavel
-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

Re: Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Bodo Eggert]

Miklos Szeredi <miklos@szeredi.hu> wrote:

>> > So to summarize, the plan that makes things work with fuse is:
>> > 
>> > - For STR, don't do the freezer thing.
>> > 
>> > - For STD, don't sys_sync() after you froze
>> > 
>> > There might be -other- issues, but that should get you through some of
>> 
>> At the risk of repeating myself. Character device drivers are written
>> with the assumption that normal io and suspend/resume do not race
>> with each other due to the freezer.
>> What do you intend to do about that?
> 
> Oliver, can you please explain your worries in a bit more detail?

Try suspending while printing.
-- 
What's worse than a Male Chauvinist Pig?
A woman that won't do what she's told.

Friß, Spammer: tgyvaxz@.qcLz7l.7eggert.dyndns.org

Re: Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Alan Stern]

On Wed, 4 Jul 2007, Paul Mackerras wrote:

> Alan Stern writes:
> 
> > > Most drivers suspended their hardware in the second call.  If they are
> > > in the middle of a conversation with their device that *has* to be
> > > completed, they can do that by polling.
> > 
> > Ugh.  That will cause problems when you try to integrate runtime 
> > suspend.  In fact this whole approach is unsuitable for runtime PM and 
> > it obscures the similarities between runtime PM and STR.
> 
> Yes there are similarities, but it would be a big mistake to say that
> a requirement for STR is that all drivers do runtime PM.

That's not what I'm saying.  What I'm saying is that it would be a big 
mistake to force all drivers which implement runtime PM to do it using 
a separate code path from system PM.

> The main attraction of the late-suspend call is that it really does,
> reliably, guarantee that the driver's I/O request methods won't get
> called between the late-suspend call and the early-resume call.

For some drivers (like USB), carrying out an actual suspend requires a
delay.  Right now we implement those delays using wait_event(),
wait_for_completion(), and so on.  Would you have us check at runtime
whether or not a system suspend is underway and in each case use a
busy-loop instead if it is?

What happens if, in order to carry out the late-suspend, a driver needs
to acquire a mutex which happens to be held by some other task?  That
other task won't be able to run and release the mutex, so you will
deadlock.

Alan Stern

Re: Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Paul Mackerras]

Alan Stern writes:

> That's not what I'm saying.  What I'm saying is that it would be a big 
> mistake to force all drivers which implement runtime PM to do it using 
> a separate code path from system PM.

OK; I can accept that provided there is a way to change the "what to
do with an I/O request" policy from auto-resume to something else
while we're suspending the system (and presumably restore the old
policy on system resume if the device was runtime-suspended at the
point where the system was suspended).

> > The main attraction of the late-suspend call is that it really does,
> > reliably, guarantee that the driver's I/O request methods won't get
> > called between the late-suspend call and the early-resume call.
> 
> For some drivers (like USB), carrying out an actual suspend requires a
> delay.  Right now we implement those delays using wait_event(),
> wait_for_completion(), and so on.  Would you have us check at runtime
> whether or not a system suspend is underway and in each case use a
> busy-loop instead if it is?

No; the late suspend call isn't appropriate for all drivers.  It is a
simple and safe way to do the suspend for some drivers, mostly the
simpler ones.  Things that are complex enough to have a subsystem
(e.g. USB) would want to use the early suspend call.

> What happens if, in order to carry out the late-suspend, a driver needs
> to acquire a mutex which happens to be held by some other task?  That
> other task won't be able to run and release the mutex, so you will
> deadlock.

Then late-suspend is not appropriate for that driver, and it needs to
use the early-suspend call, and do something such as setting a flag
that the I/O request function tests.

Paul.

Re: Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Alan Stern]

On Wed, 4 Jul 2007, Paul Mackerras wrote:

> Whether or not to resume a suspended device when an I/O request comes
> in is a policy decision, and there could be cases where the user wants
> I/O requests to be blocked, or to fail, or to be dropped while the
> device is suspended, even for runtime power management.  For example,
> a sound card could be suspended due to a low-battery condition, and in
> that case you would want the driver to just drop any data that
> userspace tries to write to the soundcard.

We have provisions for that (my earlier description was somewhat 
incomplete).

> > Yes, the code could be changed to keep track of the reason for a device
> > suspend.  But that just raises the old problem of what to do when
> > there's an I/O request for a suspended device during STR.
> 
> Is this actually a real problem?  I would think the policy would be
> "block" for block devices (pun not intended :), "drop" for network
> devices, etc.

It is indeed a real problem, or at least, it can be.

> > Consider a particularly troublesome case: During STR, a non-frozen task
> > writes to /sys/bus/BBB/drivers/DDD/bind.  The sysfs core grabs the
> > device semaphore and calls the driver's probe routine.  If the driver
> > isn't PM-aware it simply tries to initialize the device and fails
> > because the device is already suspended.  That's no good; it isn't
> > transparent.
> 
> How did the device get suspended if it didn't have a driver?  If it
> did have a driver, why didn't the bind attempt fail?

Bus subsystems can suspend devices with no drivers.

> Suppose the device-model core code simply blocked all bind and unbind
> requests while suspend is under way, until resume is finished.
> Wouldn't that solve the problem?

It would help.  It would help even more if the sysfs core also blocked
all I/O while suspend is under way.  (Although this might be tricky, 
considering that the suspend is initiated by a sysfs write...)

The fact remains that lots of drivers would still need to be changed.  
In the read and write methods someone would have to add code amounting
to this:

	if (suspend_is_under_way()) {
		mutex_unlock(...);
		block_until_resume();
		goto restart;
	}

Freezing userspace is a small amount of code by comparison.

Alan Stern

Re: Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Paul Mackerras]

Alan Stern writes:

> > > Yes, the code could be changed to keep track of the reason for a device
> > > suspend.  But that just raises the old problem of what to do when
> > > there's an I/O request for a suspended device during STR.
> > 
> > Is this actually a real problem?  I would think the policy would be
> > "block" for block devices (pun not intended :), "drop" for network
> > devices, etc.
> 
> It is indeed a real problem, or at least, it can be.

How so?  Can you give me an example?

> > How did the device get suspended if it didn't have a driver?  If it
> > did have a driver, why didn't the bind attempt fail?
> 
> Bus subsystems can suspend devices with no drivers.

Interesting.  I assume this is for buses for which there is a
bus-specific but device-independent suspend procedure defined.

It would seem sensible to me that the PM core should get the bus to
resume such a device before calling a driver probe routine.  The
resume should be blocked or deferred while a system suspend is
underway.  In fact I think that all driver bind/unbind and probe
operations should be deferred while the system is suspending (i.e. put
on a list to be done after the system resumes).

> It would help.  It would help even more if the sysfs core also blocked
> all I/O while suspend is under way.  (Although this might be tricky, 
> considering that the suspend is initiated by a sysfs write...)

I didn't think sysfs got involved at all in normal read and write
requests, so I don't know how it would block them...

> The fact remains that lots of drivers would still need to be changed.  
> In the read and write methods someone would have to add code amounting
> to this:
> 
> 	if (suspend_is_under_way()) {
> 		mutex_unlock(...);
> 		block_until_resume();
> 		goto restart;
> 	}
> 
> Freezing userspace is a small amount of code by comparison.

Normally devices have some sort of queue of pending operations.  So
all that is required on suspend is to stop processing the queue and
wait for any currently-underway operations to complete.  The blocking
then happens naturally using the normal I/O wait mechanisms.

Paul.

Re: Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Alan Stern]

On Thu, 5 Jul 2007, Paul Mackerras wrote:

> Alan Stern writes:
> 
> > > > Yes, the code could be changed to keep track of the reason for a device
> > > > suspend.  But that just raises the old problem of what to do when
> > > > there's an I/O request for a suspended device during STR.
> > > 
> > > Is this actually a real problem?  I would think the policy would be
> > > "block" for block devices (pun not intended :), "drop" for network
> > > devices, etc.
> > 
> > It is indeed a real problem, or at least, it can be.
> 
> How so?  Can you give me an example?

The example I quoted earlier about binding during a suspend will do.  I 
agree that we can and should try to prevent it from ever occurring.

Read and write are a problem only in that fixing them would potentially
involve changing lots of drivers; I don't think they pose a serious
theoretical obstacle.  (Lord knows what will happen with async I/O!)

Any other entry points to drivers are also potential problems, but it's 
hard to say anything definite about them since they are so varied.

> > Bus subsystems can suspend devices with no drivers.
> 
> Interesting.  I assume this is for buses for which there is a
> bus-specific but device-independent suspend procedure defined.

Yes.

> It would seem sensible to me that the PM core should get the bus to
> resume such a device before calling a driver probe routine.  The
> resume should be blocked or deferred while a system suspend is
> underway.  In fact I think that all driver bind/unbind and probe
> operations should be deferred while the system is suspending (i.e. put
> on a list to be done after the system resumes).

Getting the PM core to resume a device before probing could be 
difficult; in general it doesn't know enough about specific device 
behaviors to do something like that.  But the subsystem certainly ought 
to take care of it.  USB does.

Yes, bind/unbind/etc. should be deferred during a system suspend.  But
it has to be done carefully, because these operations generally involve
locks that can't be released.  They need to be prevented at their
source, not in the driver core.  That's one reason why khubd needs to
be frozen (being part of the USB hub driver, it is the task responsible
for binding and unbinding drivers to USB devices).

Another thing to look out for is registration and unregistration of 
drivers.  These activities also cause bind/unbind operations.  Note 
that if userspace is frozen then neither insmod nor rmmod can run.  :-)

> > It would help.  It would help even more if the sysfs core also blocked
> > all I/O while suspend is under way.  (Although this might be tricky, 
> > considering that the suspend is initiated by a sysfs write...)
> 
> I didn't think sysfs got involved at all in normal read and write
> requests, so I don't know how it would block them...

All I/O to sysfs attributes passes through the routines in fs/sysfs/*.  
It could be blocked there.  (But if userspace is frozen it won't need 
to be.)

> Normally devices have some sort of queue of pending operations.

That's certainly true of block devices, whose drivers use the block 
subsystem.  It's not true for lots of other devices, though.

>  So
> all that is required on suspend is to stop processing the queue and
> wait for any currently-underway operations to complete.  The blocking
> then happens naturally using the normal I/O wait mechanisms.

In my experience, most non-block drivers do not have any queue of 
pending I/O operations.  They simply carry out requests as they arrive.

Alan Stern

Re: Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Alan Stern]

On Thu, 5 Jul 2007, Miklos Szeredi wrote:

> I fear, that your efforts to "save" the freezer are in vain.  It is
> already moderately hackish with that PF_FREEZER_SKIP and the kernel
> dotted randomly with try_to_freeze() calls, but adding bandaids to try
> to order freezing userspace processes in the right order would just
> make it a horrible mess.

I agree that bandaids won't work.  What's needed is something more 
radical.  Things like FUSE must be written so that the kernel parts 
_can_ freeze even while they are waiting for a response from a user 
thread.

Alan Stern

Re: Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Oliver Neukum]

Am Donnerstag, 5. Juli 2007 schrieb Alan Stern:
> On Thu, 5 Jul 2007, Oliver Neukum wrote:
> 
> > > Obviously.  But I wasn't about the server trying to acquire a lock
> > > held by a client.  I was talking about a client trying to acquire a
> > > lock held by _another_ client.
> > > 
> > > If this coincides with the server (or some other task which the server
> > > is depending on) being frozen before the clients, the freezer has a
> > > problem.
> > 
> > True, but that case can only happen if servers are frozen before clients.
> > You don't need a full dependency graph. A simple set sequence of two
> > classes of tasks will do.
> 
> Just to make things more complicated...  Since a server isn't
> restricted in what it can do, what happens when one server depends on
> another server?

The same principle applies. If you really want that you can solve this
by freezing servers in the reverse sequence they were started.

The main point remains. If you have a circular dependency anywhere
among the servers you can deadlock independent of the freezer.

	Regards
		Oliver

Re: Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Miklos Szeredi]

> > On Thu, 5 Jul 2007, Oliver Neukum wrote:
> > 
> > > > Obviously.  But I wasn't about the server trying to acquire a lock
> > > > held by a client.  I was talking about a client trying to acquire a
> > > > lock held by _another_ client.
> > > > 
> > > > If this coincides with the server (or some other task which the server
> > > > is depending on) being frozen before the clients, the freezer has a
> > > > problem.
> > > 
> > > True, but that case can only happen if servers are frozen before clients.
> > > You don't need a full dependency graph. A simple set sequence of two
> > > classes of tasks will do.
> > 
> > Just to make things more complicated...  Since a server isn't
> > restricted in what it can do, what happens when one server depends on
> > another server?
> 
> The same principle applies. If you really want that you can solve this
> by freezing servers in the reverse sequence they were started.

You just can't know what constitutes a "server", processes which read
from the fuse device are candidates, but all tasks which communicate
with these in some way are also.  And that basically makes every
userspace task in the system a candidate server and you are no further
than before.

> The main point remains. If you have a circular dependency anywhere
> among the servers you can deadlock independent of the freezer.

Well, Duh.

Miklos

Re: Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Miklos Szeredi]

> > I fear, that your efforts to "save" the freezer are in vain.  It is
> > already moderately hackish with that PF_FREEZER_SKIP and the kernel
> > dotted randomly with try_to_freeze() calls, but adding bandaids to try
> > to order freezing userspace processes in the right order would just
> > make it a horrible mess.
> 
> I agree that bandaids won't work.  What's needed is something more 
> radical.  Things like FUSE must be written so that the kernel parts 
> _can_ freeze even while they are waiting for a response from a user 
> thread.

This has already been discussed, with the conclusion, that it can't be
done without hacking VFS internals.

The basic problem is that the freezer tries to get every user process
out of the kernel even when those processes have _nothing_ to do with
drivers and could happily stay in kernel land across a suspend or even
hibernate.

If we could have a good grip on when a request is entering a driver,
it would be easy to take care of this.  I guess network and block
devices are easy.  For others there's no obvious common place where
such barriers could be placed so it's more work, but nothing
conceptually problematic.  Is this about right?

Miklos

Re: Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Oliver Neukum]

Am Donnerstag, 5. Juli 2007 schrieb Alan Stern:
> On Thu, 5 Jul 2007, Miklos Szeredi wrote:
> 
> > I fear, that your efforts to "save" the freezer are in vain.  It is
> > already moderately hackish with that PF_FREEZER_SKIP and the kernel
> > dotted randomly with try_to_freeze() calls, but adding bandaids to try
> > to order freezing userspace processes in the right order would just
> > make it a horrible mess.
> 
> I agree that bandaids won't work.  What's needed is something more 
> radical.  Things like FUSE must be written so that the kernel parts 
> _can_ freeze even while they are waiting for a response from a user 
> thread.

OK, some radical ideas.

In principle we want a deadlock here. Tasks that are frozen due to fuse
are as good as frozen, there's no need to formally freeze them.
The bad uninterruptible tasks are those waiting for hardware.

Can we detect the difference? Perhaps we can label those crucial locks
and when only tasks in the refrigerator and tasks waiting on these locks
are left we are content and go on to suspending the device tree.

	Flame away
		Oliver

Re: Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Rafael J. Wysocki]

On Thursday, 5 July 2007 23:37, Oliver Neukum wrote:
> Am Donnerstag, 5. Juli 2007 schrieb Alan Stern:
> > On Thu, 5 Jul 2007, Miklos Szeredi wrote:
> > 
> > > I fear, that your efforts to "save" the freezer are in vain.  It is
> > > already moderately hackish with that PF_FREEZER_SKIP and the kernel
> > > dotted randomly with try_to_freeze() calls, but adding bandaids to try
> > > to order freezing userspace processes in the right order would just
> > > make it a horrible mess.
> > 
> > I agree that bandaids won't work.  What's needed is something more 
> > radical.  Things like FUSE must be written so that the kernel parts 
> > _can_ freeze even while they are waiting for a response from a user 
> > thread.
> 
> OK, some radical ideas.
> 
> In principle we want a deadlock here. Tasks that are frozen due to fuse
> are as good as frozen, there's no need to formally freeze them.
> The bad uninterruptible tasks are those waiting for hardware.
> 
> Can we detect the difference? Perhaps we can label those crucial locks
> and when only tasks in the refrigerator and tasks waiting on these locks
> are left we are content and go on to suspending the device tree.

Well, I think we can just make the freezer handle uninterruptible tasks.

If an uninterruptible task is stuck holding a lock that's needed for suspend,
the freezerless suspend will deadlock anyway. :-)

The only reason (I know of) why we don't handle uninterruptible tasks in the
freezer is that we're afraid of the suspend process deadlocking with an
uninterruptible task holding a lock, but AFAICS the probability of such an
event is extremely small.

Also, the powermac suspend will deadlock in such cases, so the fact that
it doesn't deadlock means that they don't occur very often (if at all).

I have a patch for that, will post it in a separate thread in a while.

Greetings,
Rafael


-- 
"Premature optimization is the root of all evil." - Donald Knuth

Re: Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Benjamin Herrenschmidt]

On Fri, 2007-07-06 at 09:13 +0200, Rafael J. Wysocki wrote:
> 
> The only reason (I know of) why we don't handle uninterruptible tasks in the
> freezer is that we're afraid of the suspend process deadlocking with an
> uninterruptible task holding a lock, but AFAICS the probability of such an
> event is extremely small.

What would deadlock specifically ? One of the drivers trying to acquire
that lock ? It would be a driver bug then.

> Also, the powermac suspend will deadlock in such cases, so the fact that
> it doesn't deadlock means that they don't occur very often (if at all).

Ben.

Re: Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Oliver Neukum]

Am Freitag, 6. Juli 2007 schrieb Benjamin Herrenschmidt:
> On Fri, 2007-07-06 at 09:13 +0200, Rafael J. Wysocki wrote:
> > 
> > The only reason (I know of) why we don't handle uninterruptible tasks in the
> > freezer is that we're afraid of the suspend process deadlocking with an
> > uninterruptible task holding a lock, but AFAICS the probability of such an
> > event is extremely small.
> 
> What would deadlock specifically ? One of the drivers trying to acquire
> that lock ? It would be a driver bug then.

Your driver's write method looks like:

mutex_lock();
poke_some_hardware();
wait_event_uninterruptible(); //for result
res = evaluate_result();
mutex_unlock();
return res;

If you put a task into the refrigerator at wait_event_interruptible()
you will deadlock if you need this lock for the driver to go to suspend.
The suspend method then must not take the lock _and_ it must be
aware that there may be an ongoing operation.

	Regards
		Oliver

Re: Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Rafael J. Wysocki]

On Friday, 6 July 2007 11:31, Oliver Neukum wrote:
> Am Freitag, 6. Juli 2007 schrieb Benjamin Herrenschmidt:
> > On Fri, 2007-07-06 at 09:13 +0200, Rafael J. Wysocki wrote:
> > > 
> > > The only reason (I know of) why we don't handle uninterruptible tasks in the
> > > freezer is that we're afraid of the suspend process deadlocking with an
> > > uninterruptible task holding a lock, but AFAICS the probability of such an
> > > event is extremely small.
> > 
> > What would deadlock specifically ? One of the drivers trying to acquire
> > that lock ? It would be a driver bug then.
> 
> Your driver's write method looks like:
> 
> mutex_lock();
> poke_some_hardware();
> wait_event_uninterruptible(); //for result
> res = evaluate_result();
> mutex_unlock();
> return res;
> 
> If you put a task into the refrigerator at wait_event_interruptible()
> you will deadlock if you need this lock for the driver to go to suspend.
> The suspend method then must not take the lock _and_ it must be
> aware that there may be an ongoing operation.

s/interruptible/uninterruptible/

> you will deadlock if you need this lock for the driver to go to suspend.
> The suspend method then must not take the lock _and_ it must be
> aware that there may be an ongoing operation.

Well, is there any driver in the tree that works like that _and_ has a
.suspend() method requiring the same lock?

Besides, I'm not going to put the task into the refrigerator at that point.

Please read http://lkml.org/lkml/2007/7/6/71

Moreover, I claim that, in the context of your example, _if_ the task is stuck
at the wait_event_uninterruptible(), _then_ the freezerless suspend will
deadlock with the task.

Greetings,
Rafael


-- 
"Premature optimization is the root of all evil." - Donald Knuth

Re: Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Benjamin Herrenschmidt]

On Fri, 2007-07-06 at 11:31 +0200, Oliver Neukum wrote:
> Am Freitag, 6. Juli 2007 schrieb Benjamin Herrenschmidt:
> > On Fri, 2007-07-06 at 09:13 +0200, Rafael J. Wysocki wrote:
> > > 
> > > The only reason (I know of) why we don't handle uninterruptible tasks in the
> > > freezer is that we're afraid of the suspend process deadlocking with an
> > > uninterruptible task holding a lock, but AFAICS the probability of such an
> > > event is extremely small.
> > 
> > What would deadlock specifically ? One of the drivers trying to acquire
> > that lock ? It would be a driver bug then.
> 
> Your driver's write method looks like:
> 
> mutex_lock();
> poke_some_hardware();
> wait_event_uninterruptible(); //for result
> res = evaluate_result();
> mutex_unlock();
> return res;
> 
> If you put a task into the refrigerator at wait_event_interruptible()
> you will deadlock if you need this lock for the driver to go to suspend.
> The suspend method then must not take the lock _and_ it must be
> aware that there may be an ongoing operation.

Well... 2 things here. Either you have a freezer in which case the
chances of the above scenario are increased, or you don't, in which case
your suspend method will just sleep on the lock until outstanding HW
accesses that have that lock are completed, and everything is fine.

You need to be careful with one thing though, whether you have a freezer
or not. If you driver, in some code path, whatever it is (ioctl, kernel
thread, workqueue, ...) does something like:

mutex_lock
kmalloc(...,GFP_KERNEL);
mutex_unlock

And it's suspend callback then does:

mutex_lock

The problem here is that the disks might already have been suspended
prior to your driver being called. Thus, any attempt at pushing things
out to swap or dirty mmap'ings back to storage will hang, thus kmalloc
can potentially hang (afaik), and you will deadlock.

That's what I've been talking about earlier when I said that we should
have some security in SLAB/SLUB/Buddy allocators, to silently turn
GFP_KERNEL to at least GFP_NOIO or even ATOMIC before we start
suspending drivers.

Now, another way to deal with that would have to use
pre-suspend/post-resume notifications, and have drivers avoid doing the
above between those, but that's much harder. (Essentially, drivers would
have to either make sure they don't do things like blocking allocations,
even implicitely, or possibly fall back to a degraded synchronous mode
or that sort of thing).

I think it's much simpler to tweak slab/slub/buddy instead :-)

Note that the above issue is orthogonal to our freezer discussion, it's
just one of the potential deadlock cause we have with suspend that needs
to be fixed.

Cheers,
Ben.

Re: Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Rafael J. Wysocki]

On Saturday, 7 July 2007 04:44, Benjamin Herrenschmidt wrote:
> On Fri, 2007-07-06 at 11:31 +0200, Oliver Neukum wrote:
> > Am Freitag, 6. Juli 2007 schrieb Benjamin Herrenschmidt:
> > > On Fri, 2007-07-06 at 09:13 +0200, Rafael J. Wysocki wrote:
> > > > 
> > > > The only reason (I know of) why we don't handle uninterruptible tasks in the
> > > > freezer is that we're afraid of the suspend process deadlocking with an
> > > > uninterruptible task holding a lock, but AFAICS the probability of such an
> > > > event is extremely small.
> > > 
> > > What would deadlock specifically ? One of the drivers trying to acquire
> > > that lock ? It would be a driver bug then.
> > 
> > Your driver's write method looks like:
> > 
> > mutex_lock();
> > poke_some_hardware();
> > wait_event_uninterruptible(); //for result
> > res = evaluate_result();
> > mutex_unlock();
> > return res;
> > 
> > If you put a task into the refrigerator at wait_event_interruptible()
> > you will deadlock if you need this lock for the driver to go to suspend.
> > The suspend method then must not take the lock _and_ it must be
> > aware that there may be an ongoing operation.
> 
> Well... 2 things here. Either you have a freezer in which case the
> chances of the above scenario are increased,

How so? :-)

> or you don't, in which case 
> your suspend method will just sleep on the lock until outstanding HW
> accesses that have that lock are completed, and everything is fine.
> 
> You need to be careful with one thing though, whether you have a freezer
> or not. If you driver, in some code path, whatever it is (ioctl, kernel
> thread, workqueue, ...) does something like:
> 
> mutex_lock
> kmalloc(...,GFP_KERNEL);
> mutex_unlock
> 
> And it's suspend callback then does:
> 
> mutex_lock
> 
> The problem here is that the disks might already have been suspended
> prior to your driver being called. Thus, any attempt at pushing things
> out to swap or dirty mmap'ings back to storage will hang, thus kmalloc
> can potentially hang (afaik), and you will deadlock.
> 
> That's what I've been talking about earlier when I said that we should
> have some security in SLAB/SLUB/Buddy allocators, to silently turn
> GFP_KERNEL to at least GFP_NOIO or even ATOMIC before we start
> suspending drivers.
> 
> Now, another way to deal with that would have to use
> pre-suspend/post-resume notifications, and have drivers avoid doing the
> above between those, but that's much harder. (Essentially, drivers would
> have to either make sure they don't do things like blocking allocations,
> even implicitely, or possibly fall back to a degraded synchronous mode
> or that sort of thing).
> 
> I think it's much simpler to tweak slab/slub/buddy instead :-)
> 
> Note that the above issue is orthogonal to our freezer discussion, it's
> just one of the potential deadlock cause we have with suspend that needs
> to be fixed.

Agreed.

Greetings,
Rafael


-- 
"Premature optimization is the root of all evil." - Donald Knuth

Re: Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Benjamin Herrenschmidt]

> > Well... 2 things here. Either you have a freezer in which case the
> > chances of the above scenario are increased,
> 
> How so? :-)

I meant you have a freezer that freezes uninterruptible tasks.

Ben.

Re: Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Benjamin Herrenschmidt]

On Fri, 2007-07-06 at 11:53 +0200, Rafael J. Wysocki wrote:
> 
> Moreover, I claim that, in the context of your example, _if_ the task
> is stuck
> at the wait_event_uninterruptible(), _then_ the freezerless suspend
> will
> deadlock with the task.

Why would the task be stuck there if it's not becasue of a freezer ? The
only reason I see would be a dependency on something like kmalloc trying
to push things to disk, or a lock owned by another user process, but the
former is a generic issue I've discussed in a separate mail and the
later is a driver bug imho.

Ben.

Re: Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Benjamin Herrenschmidt]

On Thu, 2007-07-05 at 10:23 -0400, Alan Stern wrote:
> 
> How will that help?  Block the kernel thread in the freezer or block it 
> in the driver -- either way it is blocked.  So how do your deadlocks 
> get resolved?

Because nobody is waiting on that kernel thread anyway without a freezer
so there is no deadlock anymore.

> I disagree with your analysis -- not that it's completely wrong, but it 
> points out an existing basic problem in the kernel.  The kernel should 
> never depend on userspace!  More correctly, a task executing in the 
> kernel should never block with any sort of mutex or other lock held (in 
> a way that would preclude it from being frozen, let's say) while 
> waiting for a response from userspace.
> 
> Then the dependency graph would be easy to construct: User tasks can
> depend on whatever they want, and kernel threads never depend on a user
> task.

In an idea world, there would be no hunger...

> If this contradicts the existing implementations and APIs for userspace 
> filesystems, then so be it.  My conclusion would be that the 
> implementations and APIs should be changed.

Why are you guys working so hard and spending so much energy to try to
avoid doing the right thing is beyond my understanding...

> It _does_ apply to kernel threads.  That's exactly why I wrote above 
> that kernel threads which try to do I/O during a suspend will need 
> extra attention.

Ok none at all if you don't have a freezer.

Ben.

Re: Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Rafael J. Wysocki]

On Friday, 6 July 2007 00:59, Benjamin Herrenschmidt wrote:
> On Thu, 2007-07-05 at 10:23 -0400, Alan Stern wrote:
> > 
> > How will that help?  Block the kernel thread in the freezer or block it 
> > in the driver -- either way it is blocked.  So how do your deadlocks 
> > get resolved?
> 
> Because nobody is waiting on that kernel thread anyway without a freezer
> so there is no deadlock anymore.

I'm not sure what you mean.  The freezer doesn't wait for threads that are
already frozen ...

> > I disagree with your analysis -- not that it's completely wrong, but it 
> > points out an existing basic problem in the kernel.  The kernel should 
> > never depend on userspace!  More correctly, a task executing in the 
> > kernel should never block with any sort of mutex or other lock held (in 
> > a way that would preclude it from being frozen, let's say) while 
> > waiting for a response from userspace.
> > 
> > Then the dependency graph would be easy to construct: User tasks can
> > depend on whatever they want, and kernel threads never depend on a user
> > task.
> 
> In an idea world, there would be no hunger...
> 
> > If this contradicts the existing implementations and APIs for userspace 
> > filesystems, then so be it.  My conclusion would be that the 
> > implementations and APIs should be changed.
> 
> Why are you guys working so hard and spending so much energy to try to
> avoid doing the right thing is beyond my understanding...
> 
> > It _does_ apply to kernel threads.  That's exactly why I wrote above 
> > that kernel threads which try to do I/O during a suspend will need 
> > extra attention.
> 
> Ok none at all if you don't have a freezer.

Provided that drivers can handle that.

Greetings,
Rafael


-- 
"Premature optimization is the root of all evil." - Donald Knuth

Re: Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Alan Stern]

On Fri, 6 Jul 2007, Benjamin Herrenschmidt wrote:

> Why are you guys working so hard and spending so much energy to try to
> avoid doing the right thing is beyond my understanding...
> 
> > It _does_ apply to kernel threads.  That's exactly why I wrote above 
> > that kernel threads which try to do I/O during a suspend will need 
> > extra attention.
> 
> Ok none at all if you don't have a freezer.

In answer to both questions: We need the freezer in order to implement 
hibernate.  Even if we take your advice and stop using the freezer 
during suspend, these issues would still remain and would need to be 
solved.

Alan Stern

Re: Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Pavel Machek]

Hi!

> > How will that help?  Block the kernel thread in the freezer or block it 
> > in the driver -- either way it is blocked.  So how do your deadlocks 
> > get resolved?
> 
> Because nobody is waiting on that kernel thread anyway without a freezer
> so there is no deadlock anymore.

In the deadlock we are seeing, _someone_ is waiting on userspace
thread, that leads to deadlock with freezer. We don't know who,
because we have not seen the sysrq-t dumps.

The "unknown who" will deadlock on fused frozen by driver, too. We
really need to fix the "unknown who" here.
								Pavel
-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

Re: Re: [PATCH] Remove process freezer from suspend to RAM pathway

[Paul Mackerras]

Alan Stern writes:

> In answer to both questions: We need the freezer in order to implement 
> hibernate.  Even if we take your advice and stop using the freezer 
> during suspend, these issues would still remain and would need to be 
> solved.

Stepping back for a minute, let's think about what the freezer is
trying to achieve.  I think that currently there are three basic
design goals:

A. Ensure that device drivers don't get I/O requests after being
   suspended.

B. Ensure that driver suspend routines don't end up blocking forever
   on mutexes or semaphores held by frozen tasks.

C. Provide a way to get an atomic snapshot of memory for hibernation.

Now, it's easy enough to freeze all processes (or all except one), if
you don't have goal B.  Just offline non-boot cpus and disable
interrupts, or use stop_machine().  But goal B implies that you can't
necessarily just stop all tasks wherever they are.  In fact it means
there are points where it is safe to stop a given task, and there may
be points where it isn't safe to stop it - and there is no practical
way to determine those points reliably.[1]

That implies to me that we can have a freezer as long as we do nothing
that can sleep, while tasks are frozen.  In other words, I think the
freezer is a viable option for hibernation as long as we restrict
driver hibernate routines to doing only things which don't sleep.  I
_think_ that should be doable since hibernate routines only need to
wait for outstanding DMAs to complete, as I understand it, which can
be done by polling.

Paul.

[1] For a start, there's no way to determine which mutexes a task
holds.  Even if there were, we would then also have to know which
mutexes it is going to try to acquire before it releases the one we
want, which is pretty much unknowable.

One can say "we'll only freeze kernel tasks that ask to be frozen" but
then one has no way to guarantee A, and we still don't reliably
guarantee B if we have user-level filesystems or device drivers.