shrinking memory for suspend?

shrinking memory for suspend?

[Johannes Berg]

[-- Attachment #1.1: Type: text/plain, Size: 259 bytes --]

Hi,

Paul Mackerras noticed that even for suspend to ram we currently free
some memory in suspend_prepare. Is that necessary because we freeze
userspace so devices can allocate a bit of memory for suspend? Why do we
freeze userspace anyway?

johannes

[-- Attachment #1.2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 190 bytes --]

[-- Attachment #2: Type: text/plain, Size: 0 bytes --]

Re: shrinking memory for suspend?

[Johannes Berg]

[-- Attachment #1.1: Type: text/plain, Size: 923 bytes --]

Hi,

> I don't think it's related to the freezing of user space.  You can allocate
> memory with all of the user space frozen just fine.  I'd say it's related to
> the freezing of kernel threads, rather.

Ah, true, of course.

> > Why do we freeze userspace anyway?
> 
> Well, let me quote Linus:
> 
> "I _do_ realize the IO request queue issues, and that we cannot actually do 
> s2ram with some devices in the middle of a DMA. So we want to be able to 
> avoid *that*, there's no question about that. And I suspect that stopping 
> user threads and then waiting for a sync is practically one of the easier 
> ways to do so."

Hmm, yeah. Does this apply to kernel threads too? Actually, I guess
device drivers should wait for suspend and reject commands to devices
that are suspended already? Can it actually happen that we schedule to
userspace while calling all the suspend issues etc?

johannes

[-- Attachment #1.2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 190 bytes --]

[-- Attachment #2: Type: text/plain, Size: 0 bytes --]

Re: shrinking memory for suspend?

[Alan Stern]

On Mon, 30 Apr 2007, Johannes Berg wrote:

> Hi,
> 
> Paul Mackerras noticed that even for suspend to ram we currently free
> some memory in suspend_prepare. Is that necessary because we freeze
> userspace so devices can allocate a bit of memory for suspend? Why do we
> freeze userspace anyway?

So that user processes don't generate I/O requests while the devices are
suspended -- such requests would be bound to fail.

Alan Stern

Re: shrinking memory for suspend?

[Rafael J. Wysocki]

Hi,

On Monday, 30 April 2007 16:24, Johannes Berg wrote:
> Hi,
> 
> Paul Mackerras noticed that even for suspend to ram we currently free
> some memory in suspend_prepare. Is that necessary because we freeze
> userspace so devices can allocate a bit of memory for suspend?

I don't think it's related to the freezing of user space.  You can allocate
memory with all of the user space frozen just fine.  I'd say it's related to
the freezing of kernel threads, rather.

> Why do we freeze userspace anyway?

Well, let me quote Linus:

"I _do_ realize the IO request queue issues, and that we cannot actually do 
s2ram with some devices in the middle of a DMA. So we want to be able to 
avoid *that*, there's no question about that. And I suspect that stopping 
user threads and then waiting for a sync is practically one of the easier 
ways to do so."

Greetings,
Rafael

Re: shrinking memory for suspend?

[Rafael J. Wysocki]

On Monday, 30 April 2007 17:09, Johannes Berg wrote:
> Hi,
> 
> > I don't think it's related to the freezing of user space.  You can allocate
> > memory with all of the user space frozen just fine.  I'd say it's related to
> > the freezing of kernel threads, rather.
> 
> Ah, true, of course.
> 
> > > Why do we freeze userspace anyway?
> > 
> > Well, let me quote Linus:
> > 
> > "I _do_ realize the IO request queue issues, and that we cannot actually do 
> > s2ram with some devices in the middle of a DMA. So we want to be able to 
> > avoid *that*, there's no question about that. And I suspect that stopping 
> > user threads and then waiting for a sync is practically one of the easier 
> > ways to do so."
> 
> Hmm, yeah. Does this apply to kernel threads too? Actually, I guess
> device drivers should wait for suspend and reject commands to devices
> that are suspended already?

Kernel threads that belong to device drivers are frozen too (well, not exactly
all of them, but let's omit the ugly details for now), because they generally
need to be synchronized with the drivers' .suspend() and .resume() routines
and IMO it's easier to freeze them than handle that in any other way.

After .suspend() and before .resume() any commands should not be sent to the
device.

> Can it actually happen that we schedule to userspace while calling all the
> suspend issues etc? 

No, the user space is frozen (ie. in TASK_UNINTERRUPTIBLE) at that time.

Greetings,
Rafael

Re: shrinking memory for suspend?

[Pavel Machek]

Hi!

> Why do we
> freeze userspace anyway?

It makes the device driver's job easier. But I guess we should stop
doing the freeze for suspend-to-RAM. Some auditing required, but OTOH
we'll be one step closer to runtime power management on such drivers.

									Pavel
-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

Re: shrinking memory for suspend?

[Johannes Berg]

[-- Attachment #1.1: Type: text/plain, Size: 376 bytes --]

On Wed, 2007-05-02 at 13:29 +0200, Pavel Machek wrote:

> It makes the device driver's job easier. But I guess we should stop
> doing the freeze for suspend-to-RAM. Some auditing required, but OTOH
> we'll be one step closer to runtime power management on such drivers.

*shrug*. We've never done it on powerbooks and it always seemed to work
just fine...

johannes

[-- Attachment #1.2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 190 bytes --]

[-- Attachment #2: Type: text/plain, Size: 0 bytes --]

Re: shrinking memory for suspend?

[Rafael J. Wysocki]

On Wednesday, 2 May 2007 16:02, Johannes Berg wrote:
> On Wed, 2007-05-02 at 13:29 +0200, Pavel Machek wrote:
> 
> > It makes the device driver's job easier. But I guess we should stop
> > doing the freeze for suspend-to-RAM. Some auditing required, but OTOH
> > we'll be one step closer to runtime power management on such drivers.
> 
> *shrug*. We've never done it on powerbooks and it always seemed to work
> just fine...

I think that on a uniprocessor system it's quite safe, but on SMP it doesn't
seem so.  For example, imagine the situation in which one CPU is executing the
suspend code while another one is running userspace with system calls etc.
Pretty scary.

Greetings,
Rafael

Re: shrinking memory for suspend?

[Paul Mackerras]

Rafael J. Wysocki writes:

> I think that on a uniprocessor system it's quite safe, but on SMP it doesn't
> seem so.  For example, imagine the situation in which one CPU is executing the
> suspend code while another one is running userspace with system calls etc.
> Pretty scary.

Which is why the powermac/powerbook sleep code insists on there only
being one cpu active.  I have an SMP powermac which can sleep; I use a
little script to take the second cpu down before sleeping and bring it
back up after waking up.

Paul.

Re: shrinking memory for suspend?

[Rafael J. Wysocki]

On Thursday, 3 May 2007 12:34, Paul Mackerras wrote:
> Rafael J. Wysocki writes:
> 
> > I think that on a uniprocessor system it's quite safe, but on SMP it doesn't
> > seem so.  For example, imagine the situation in which one CPU is executing the
> > suspend code while another one is running userspace with system calls etc.
> > Pretty scary.
> 
> Which is why the powermac/powerbook sleep code insists on there only
> being one cpu active.  I have an SMP powermac which can sleep; I use a
> little script to take the second cpu down before sleeping and bring it
> back up after waking up.

That's quite intrusive.  Ideally, user space processes should not notice that
there have been a suspend at one point.

Greetings,
Rafael

Re: shrinking memory for suspend?

[Johannes Berg]

[-- Attachment #1.1: Type: text/plain, Size: 373 bytes --]

On Thu, 2007-05-03 at 12:17 +0200, Rafael J. Wysocki wrote:

> I think that on a uniprocessor system it's quite safe, but on SMP it doesn't
> seem so.  For example, imagine the situation in which one CPU is executing the
> suspend code while another one is running userspace with system calls etc.

We can still hot-unplug all the other CPUs though, no?

johannes

[-- Attachment #1.2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 190 bytes --]

[-- Attachment #2: Type: text/plain, Size: 0 bytes --]

Re: shrinking memory for suspend?

[Paul Mackerras]

Rafael J. Wysocki writes:

> > Which is why the powermac/powerbook sleep code insists on there only
> > being one cpu active.  I have an SMP powermac which can sleep; I use a
> > little script to take the second cpu down before sleeping and bring it
> > back up after waking up.
> 
> That's quite intrusive.  Ideally, user space processes should not notice that
> there have been a suspend at one point.

Certainly, *most* userspace processes should not notice, and they
don't.  But the process initiating the suspend most certainly knows
about the suspend happening.  And that's the one that arranges for
taking the second cpu down and bringing it up.  So your objection
seems silly to me, unless you meant something other than what your
words seem to say.

Paul.

Re: shrinking memory for suspend?

[Rafael J. Wysocki]

On Thursday, 3 May 2007 13:33, Johannes Berg wrote:
> On Thu, 2007-05-03 at 12:17 +0200, Rafael J. Wysocki wrote:
> 
> > I think that on a uniprocessor system it's quite safe, but on SMP it doesn't
> > seem so.  For example, imagine the situation in which one CPU is executing the
> > suspend code while another one is running userspace with system calls etc.
> 
> We can still hot-unplug all the other CPUs though, no?

We used to, but now we're "hot-unplugging" them after suspending devices.

Besides, preemptible kernels are not much different from SMP in that respect,
so ...

Greetings,
Rafael

Re: shrinking memory for suspend?

[Rafael J. Wysocki]

On Thursday, 3 May 2007 14:51, Paul Mackerras wrote:
> Rafael J. Wysocki writes:
> 
> > > Which is why the powermac/powerbook sleep code insists on there only
> > > being one cpu active.  I have an SMP powermac which can sleep; I use a
> > > little script to take the second cpu down before sleeping and bring it
> > > back up after waking up.
> > 
> > That's quite intrusive.  Ideally, user space processes should not notice that
> > there have been a suspend at one point.
> 
> Certainly, *most* userspace processes should not notice, and they
> don't.

This isn't quite true.  For example, if the suspend happens while one of the
'other' processes checks how many CPUs are online, it can get a wrong result
due to the suspend.

> But the process initiating the suspend most certainly knows 
> about the suspend happening.  And that's the one that arranges for
> taking the second cpu down and bringing it up.

Yes, obviously.

> So your objection seems silly to me, unless you meant something other than
> what your words seem to say.

Well, perhaps I should have used some other words, then. :-)

Besides, if your kernel is preemptible, the system behaves quite like an SMP
one.

Greetings,
Rafael

Re: shrinking memory for suspend?

[Alan Stern]

On Thu, 3 May 2007, Rafael J. Wysocki wrote:

> On Thursday, 3 May 2007 12:34, Paul Mackerras wrote:
> > Rafael J. Wysocki writes:
> > 
> > > I think that on a uniprocessor system it's quite safe, but on SMP it doesn't
> > > seem so.  For example, imagine the situation in which one CPU is executing the
> > > suspend code while another one is running userspace with system calls etc.
> > > Pretty scary.
> > 
> > Which is why the powermac/powerbook sleep code insists on there only
> > being one cpu active.  I have an SMP powermac which can sleep; I use a
> > little script to take the second cpu down before sleeping and bring it
> > back up after waking up.
> 
> That's quite intrusive.  Ideally, user space processes should not notice that
> there have been a suspend at one point.

What happens without preemption enabled when a device driver's suspend() 
method calls schedule()?

With or without preemption enabled, what happens when a user process tries
to do I/O to a suspended device?

Alan Stern

Re: shrinking memory for suspend?

[Pavel Machek]

Hi!

> > > Which is why the powermac/powerbook sleep code insists on there only
> > > being one cpu active.  I have an SMP powermac which can sleep; I use a
> > > little script to take the second cpu down before sleeping and bring it
> > > back up after waking up.
> > 
> > That's quite intrusive.  Ideally, user space processes should not notice that
> > there have been a suspend at one point.
> 
> Certainly, *most* userspace processes should not notice, and they
> don't.  But the process initiating the suspend most certainly knows
> about the suspend happening.  And that's the one that arranges for
> taking the second cpu down and bringing it up.  So your objection
> seems silly to me, unless you meant something other than what your
> words seem to say.

Not silly at all.

xterm #1						xterm #2

suspend-to-RAM					
script does echo 0 > CPU1/online as part of process	echo 0 > CPU1/online

If this happens, resume will most likely online CPU1, contrary to
user's commands.

Plus, if your script is bad and leaves other CPUs running, you
get rather nasty oops, or worse. Userspace should not be able to crash
kernel like _that_.
									Pavel
-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

Re: shrinking memory for suspend?

[Rafael J. Wysocki]

On Thursday, 3 May 2007 16:34, Alan Stern wrote:
> On Thu, 3 May 2007, Rafael J. Wysocki wrote:
> 
> > On Thursday, 3 May 2007 12:34, Paul Mackerras wrote:
> > > Rafael J. Wysocki writes:
> > > 
> > > > I think that on a uniprocessor system it's quite safe, but on SMP it doesn't
> > > > seem so.  For example, imagine the situation in which one CPU is executing the
> > > > suspend code while another one is running userspace with system calls etc.
> > > > Pretty scary.
> > > 
> > > Which is why the powermac/powerbook sleep code insists on there only
> > > being one cpu active.  I have an SMP powermac which can sleep; I use a
> > > little script to take the second cpu down before sleeping and bring it
> > > back up after waking up.
> > 
> > That's quite intrusive.  Ideally, user space processes should not notice that
> > there have been a suspend at one point.
> 
> What happens without preemption enabled when a device driver's suspend() 
> method calls schedule()?
> 
> With or without preemption enabled, what happens when a user process tries
> to do I/O to a suspended device?

Yeah, right.  So much for the unforzen user space during a suspend. :-)

Greetings,
Rafael

Re: shrinking memory for suspend?

[Paul Mackerras]

Alan Stern writes:

> > > Which is why the powermac/powerbook sleep code insists on there only
> > > being one cpu active.  I have an SMP powermac which can sleep; I use a
> > > little script to take the second cpu down before sleeping and bring it
> > > back up after waking up.
> > 
> > That's quite intrusive.  Ideally, user space processes should not notice that
> > there have been a suspend at one point.
> 
> What happens without preemption enabled when a device driver's suspend() 
> method calls schedule()?

That would be bad; don't do that. :)

In the context of the original powermac sleep code, drivers registered
a notifier for sleep which got called twice - the first time before
doing the sync, and the second time after it, with an argument to the
notifier to indicate which call this is.  Drivers are allowed to call
schedule on the first call but not on the second.

> With or without preemption enabled, what happens when a user process tries
> to do I/O to a suspended device?

If we have two calls to the driver, then the second one should happen
with only one cpu still active and preemption disabled, and that's the
call where the device hardware should actually be put into its suspend
state.

I'm not saying the original powermac code was perfect, but it does
work well for many users using powerbooks (which are all single-cpu
machines), without freezing processes.

I can see the attraction of the freezer, but there are still at least
three problems with it AFAICS:

* Working out which kernel threads should *not* be frozen is a manual,
  difficult and error-prone process
* Putting processes into an uninterruptible sleep stuffs up the load
  average
* It's intrusive in the sense that every kernel thread needs to have
  code in it to allow it to be frozen, and not all the kernel thread
  main loops have been "fixed".

Paul.

Re: shrinking memory for suspend?

[Rafael J. Wysocki]

On Friday, 4 May 2007 01:32, Paul Mackerras wrote:
> Alan Stern writes:
> 
> > > > Which is why the powermac/powerbook sleep code insists on there only
> > > > being one cpu active.  I have an SMP powermac which can sleep; I use a
> > > > little script to take the second cpu down before sleeping and bring it
> > > > back up after waking up.
> > > 
> > > That's quite intrusive.  Ideally, user space processes should not notice that
> > > there have been a suspend at one point.
> > 
> > What happens without preemption enabled when a device driver's suspend() 
> > method calls schedule()?
> 
> That would be bad; don't do that. :)
> 
> In the context of the original powermac sleep code, drivers registered
> a notifier for sleep which got called twice - the first time before
> doing the sync, and the second time after it, with an argument to the
> notifier to indicate which call this is.  Drivers are allowed to call
> schedule on the first call but not on the second.
> 
> > With or without preemption enabled, what happens when a user process tries
> > to do I/O to a suspended device?
> 
> If we have two calls to the driver, then the second one should happen
> with only one cpu still active and preemption disabled, and that's the
> call where the device hardware should actually be put into its suspend
> state.
> 
> I'm not saying the original powermac code was perfect, but it does
> work well for many users using powerbooks (which are all single-cpu
> machines), without freezing processes.
> 
> I can see the attraction of the freezer, but there are still at least
> three problems with it AFAICS:
> 
> * Working out which kernel threads should *not* be frozen is a manual,
>   difficult and error-prone process
> * Putting processes into an uninterruptible sleep stuffs up the load
>   average

This is fixable, I think.

> * It's intrusive in the sense that every kernel thread needs to have
>   code in it to allow it to be frozen, and not all the kernel thread
>   main loops have been "fixed".

IMO, the freezing of user space and the freezing of kernel threads are two
different issues.  While the freezing of user space is pretty straightforward,
the freezing of kernel threads is difficult.

OTOH, it turns out that the freezer is also useful for other purposes, like
kprobes and the CPU hotplug (yes, yes), so the infrastructure needed by it
is no longer suspend-specific.  Since we have the infrastructure (needed for
other purposes anyway), we can use it just fine.

Of course, the problem of working out which kernel threads should not be
frozen for the suspend is real.

Greetings,
Rafael

Re: shrinking memory for suspend?

[Johannes Berg]

[-- Attachment #1.1: Type: text/plain, Size: 245 bytes --]

On Thu, 2007-05-03 at 14:56 +0200, Rafael J. Wysocki wrote:

> Besides, preemptible kernels are not much different from SMP in that respect,
> so ...

Yeah, we just disable preempt in the interesting code paths on
powermac ;)

johannes

[-- Attachment #1.2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 190 bytes --]

[-- Attachment #2: Type: text/plain, Size: 0 bytes --]

Re: shrinking memory for suspend?

[Paul Mackerras]

Rafael J. Wysocki writes:

> IMO, the freezing of user space and the freezing of kernel threads are two
> different issues.  While the freezing of user space is pretty straightforward,
> the freezing of kernel threads is difficult.

In general I agree, but now that we have FUSE, plus the framework for
doing device drivers in userspace (I have forgotten its name), it gets
even more complicated because some userspace processes are now doing
the sorts of things that kernel threads do.

This is relevant for me because I use encfs regularly, and encfs is
based on FUSE.  What happens if there is an encfs filesystem mounted,
and we freeze all userspace processes and then do a sys_sync()?  Does
the system wait forever for the encfs filesystem to write out its
dirty data?

Paul.

Re: shrinking memory for suspend?

[Rafael J. Wysocki]

On Friday, 4 May 2007 12:49, Paul Mackerras wrote:
> Rafael J. Wysocki writes:
> 
> > IMO, the freezing of user space and the freezing of kernel threads are two
> > different issues.  While the freezing of user space is pretty straightforward,
> > the freezing of kernel threads is difficult.
> 
> In general I agree, but now that we have FUSE, plus the framework for
> doing device drivers in userspace (I have forgotten its name), it gets
> even more complicated because some userspace processes are now doing
> the sorts of things that kernel threads do.
> 
> This is relevant for me because I use encfs regularly, and encfs is
> based on FUSE.  What happens if there is an encfs filesystem mounted,
> and we freeze all userspace processes and then do a sys_sync()?  Does
> the system wait forever for the encfs filesystem to write out its
> dirty data?

Well, that's a good question and I'm afraid the answer is 'yes'.

Still, I think we can try to solve this particular problem instead of trying to
get rid of the freezing entirely, because it that case we'll have many other
problems to fix.

Greetings,
Rafael

Re: shrinking memory for suspend?

[Alan Stern]

On Fri, 4 May 2007, Paul Mackerras wrote:

> > What happens without preemption enabled when a device driver's suspend() 
> > method calls schedule()?
> 
> That would be bad; don't do that. :)
> 
> In the context of the original powermac sleep code, drivers registered
> a notifier for sleep which got called twice - the first time before
> doing the sync, and the second time after it, with an argument to the
> notifier to indicate which call this is.  Drivers are allowed to call
> schedule on the first call but not on the second.

But the generic non-PPC core merely calls a suspend() method with
interrupts enabled.  And drivers may use the same code for runtime power
management as well.  As such it cannot afford to spin during a delay; it
must sleep.

> > With or without preemption enabled, what happens when a user process tries
> > to do I/O to a suspended device?
> 
> If we have two calls to the driver, then the second one should happen
> with only one cpu still active and preemption disabled, and that's the
> call where the device hardware should actually be put into its suspend
> state.
> 
> I'm not saying the original powermac code was perfect, but it does
> work well for many users using powerbooks (which are all single-cpu
> machines), without freezing processes.

If you think in terms of suspend-to-RAM as just being one super-big
runtime suspend of every device, you may find it an appealing point of
view.  (From some of the things he has said, this seems to be how Linus
thinks of it.)  Since runtime PM doesn't involve freezing processes, you
can conclude that neither should STR.

On the other hand, runtime suspend generally involves wake-on-demand, 
including demands for I/O from userspace.  Should STR behave the same way?

Alan Stern