what can we do with frozen processes (or: calling sync with processes frozen)

what can we do with frozen processes (or: calling sync with processes frozen)

[Johannes Berg]

[-- Attachment #1.1: Type: text/plain, Size: 2585 bytes --]

Hi,

When on vacation I was doing a bit more stuff w/o battery and noticed
that I after my patches to implement proper pm_ops for PMU-based
machines the machine would hang during suspend. Now on the way back I'm
sitting in a train and it's working again. Couldn't really be the lower
atmospheric pressure causing it ;) So I looked again and realised that
when writing those patches I thought doing sys_sync() in the ->prepare
callback would be a good idea. It is, as it turns out, not such a great
idea at all, right now anyway.

When laptop mode is enabled [1], xfs will, upon getting a sync, try to
wake up one of it's tasks and then wait for something that task does (as
far as I can tell.) Even though I carry the patch to make xfs's
workqueues non-freezable this event doesn't seem to happen when
processes are frozen and so the system hangs. Not sure why that is but
it isn't really the point right now, read on.

I have now removed the call to sys_sync() to double-verify and that has
indeed cured the problem, but the bigger question really is whether it
should be safe to call it or not [2], especially given that it would
make sense to call it when processes can no longer submit IO.

The real thing we should be doing is think about a whole range of things
you can do in the kernel and then think about whether they should be
allowed to be done in the various stages. A lot of these things are at
least somewhat documented for early boot stages (and you get into
trouble when you do it wrong) but it seems that many developers never
suspend their machines so this needs to be documented in a more explicit
way rather than just breaking when you try (bootup is part of regular
testing routine in an obvious way that suspend isn't)

Yup, this is a lot of hard work, but it could be done gradually as we
run into these problems.

Back to the specific sys_sync() issue, what do people feel is
appropriate? Should XFS get into even more trouble? It seems not to be
able to guarantee full sync when processes are frozen right now, at
least not the way sync is currently implemented.

johannes


[1] which explains why I never really saw this at home because there I
boot with power plugged in and don't enable laptop mode manually while
on vacation I booted w/o power and laptop mode was enabled at boot

[2] actually, another question is why whatever condition xfs waits for
never happens, but something is apparently frozen that it is waiting for
even though I've made the workqueues we had problems with earlier
non-freezable

[-- Attachment #1.2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 190 bytes --]

[-- Attachment #2: Type: text/plain, Size: 0 bytes --]

Re: what can we do with frozen processes (or: calling sync with processes frozen)

[Rafael J. Wysocki]

Hi,

On Sunday, 18 March 2007 00:21, Johannes Berg wrote:
> Hi,
> 
> When on vacation I was doing a bit more stuff w/o battery and noticed
> that I after my patches to implement proper pm_ops for PMU-based
> machines the machine would hang during suspend. Now on the way back I'm
> sitting in a train and it's working again. Couldn't really be the lower
> atmospheric pressure causing it ;) So I looked again and realised that
> when writing those patches I thought doing sys_sync() in the ->prepare
> callback would be a good idea. It is, as it turns out, not such a great
> idea at all, right now anyway.

Well, it's not a good idea indeed.

> When laptop mode is enabled [1], xfs will, upon getting a sync, try to
> wake up one of it's tasks and then wait for something that task does (as
> far as I can tell.) Even though I carry the patch to make xfs's
> workqueues non-freezable this event doesn't seem to happen when
> processes are frozen and so the system hangs. Not sure why that is but
> it isn't really the point right now, read on.

XFS uses more kernel threads that are frozen (see
http://lxr.free-electrons.com/source/fs/xfs/linux-2.6/xfs_buf.c?a=x86_64#1700)

> I have now removed the call to sys_sync() to double-verify and that has
> indeed cured the problem, but the bigger question really is whether it
> should be safe to call it or not [2], especially given that it would
> make sense to call it when processes can no longer submit IO.

We call sys_sync() in freeze_processes() _before_ kernel threads are frozen.
I don't think it's safe to call it any time later before the kernel threads are
thawed.

> The real thing we should be doing is think about a whole range of things
> you can do in the kernel and then think about whether they should be
> allowed to be done in the various stages. A lot of these things are at
> least somewhat documented for early boot stages (and you get into
> trouble when you do it wrong) but it seems that many developers never
> suspend their machines

True, but they should do this.

> so this needs to be documented in a more explicit 
> way rather than just breaking when you try (bootup is part of regular
> testing routine in an obvious way that suspend isn't)
> 
> Yup, this is a lot of hard work, but it could be done gradually as we
> run into these problems.
> 
> Back to the specific sys_sync() issue, what do people feel is
> appropriate? Should XFS get into even more trouble? It seems not to be
> able to guarantee full sync when processes are frozen right now, at
> least not the way sync is currently implemented.

See above.

I think at least some filesystems use freezable kernel threads and will
block if you ask them to sync after these threads have been frozen.

Greetings,
Rafael

Re: what can we do with frozen processes (or: calling sync with processes frozen)

[Johannes Berg]

[-- Attachment #1.1: Type: text/plain, Size: 253 bytes --]

Hi,

> We call sys_sync() in freeze_processes() _before_ kernel threads are frozen.
> I don't think it's safe to call it any time later before the kernel threads are
> thawed.

Obviously isn't safe :)

Thanks for the clarification.

johannes

[-- Attachment #1.2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 190 bytes --]

[-- Attachment #2: Type: text/plain, Size: 0 bytes --]

Re: what can we do with frozen processes (or: calling sync with processes frozen)

[David Brownell]

On Saturday 17 March 2007 4:21 pm, Johannes Berg wrote:
> The real thing we should be doing is think about a whole range of things
> you can do in the kernel and then think about whether they should be
> allowed to be done in the various stages. 

ISTR making this point before.  Defining "stages" will be tricky too;
the early boot stuff (initramfs running programs while drivers init)
will be slippery, and the other key transitions -- suspend, resume,
shutdown, kexec -- aren't much better.

Plus, the "freezer" thing always seemed to me like a bit of a punt;
as in your XFS example, there are tasks that should be able to do
suspend related work before freezing random stuff, and likewise
afterward.  And I'm not sure that anyone pushing on this issue won't
find arch-specific differences...

Anyway, I agree with the point of yours which I quioted above.  :)

- dave