Re: suspend.c vs driver-model.txt

Re: suspend.c vs driver-model.txt

[Benjamin Herrenschmidt]

>What races can you see?

Well, existing PM callbacks aren't good enough :) they just don't
deal with dependencies properly. Also, they don't deal that well
with the need for drivers to trigger memory allocations and all
of the issues related to that we discussed during the BOF and
earlier.
driverfs bus-oriented ones would be better here in many regard,
especially since I beleive we found "correct" semantics for the
various steps of device save state & suspend.

>> The problem of saving to disk and of saving to memory (that is
>> machine sleep as I implement it on powerbooks today, RAM content
>> beeing preserved) is pretty similar.
>
>Actually it is quite different.
>
>Saving device state is common code, but suspend-to-ram can be done
>without scheduling, while you need to block for suspend-to-disk.

No. They end up beeing very similar. Suspend to RAM has to schedule
because some underlying device drivers will need to schedule to
properly block their queues as well.

I figured out in the Pmac implementation that I could actually let
the system schedule the whole time up to just before the very last
step of shutting down the CPU. Userland apps will simply block as
they rely on IOs for drivers that have been properly blocked, or
from swap while the swap device may be suspended, etc... CPU
intensive app would still work until it's very last timeslice is
used before suspend.

That's also how I got very fast wakeup times. Basically, processes
start again right away (well, just after a few really important things
like time are restored), and then drivers are kicked back into life,
asynchronously if possible, thus user processes that are blocked by
a given driver will come back to life normally.

>> So you really need to properly do the prepare/save/suspend steps
>> on all devices in proper bus ordering so that any device driver
>> has properly saved state information to memory (which may later
>> be saved to disk with suspend-to-disk) and has properly blocked
>> IO queues.
>> 
>> The specific case of the device which is used as a backstore
>> for the RAM save has to be dealt some specific way. 
>
>No it does not. I have half of RAM free, I just save-state, copy
>memory, continue devices, copy saved memory to swap.

So you resume devices from the "saved state" which isn't the
state the device was when the machine was really suspended,
right ? Which means that typically, on resume, the driver could
end up beeing out of sync with the device if some permanent state
information exist on the device, but I agree this is a rare case,
except for... storage. So I assume you have ways to prevent
filesystems to be touched at all ?

As I see it, for your scheme to work properly, you need to,
somewhat "atomically", save-state all devices so they are
in coherent state one to each other (devices can well be
inter-dependant), backup your RAM, then you can kick back
devices (well, some actually) into life.

This is really only a special case of the generic process I'm
suggesting then ;)

Basically, you still need to run the "block IOs then save state
and suspend" step on all devices in bus ordering. So that
part is common with suspend-to-ram. Actually, you don't need
to prevent scheduling before that point, except maybe for
keeping your "half of RAM free" watermark, but even then, I
don't see how you acheive that since the kernel itself may
allocate memory (see note below)

Then you need to use the explicit device model power
state functions to re-enable power state on the target device
of the backup. It should in turn re-enable parent devices up
to the host bus.

However, that would be inefficient as the net effect would be
to have your hard disk spin down, be eventually powered off,
then back up for suspend to RAM, then the machine powered off
(and so that hard disk as well).

Which is why I beleive it would make more sense to specifically
instruct the target device (and so it's parent) during the
device suspend loop to _not_ go to sleep, just suspend, block
IOs, then resume IOs, but _not_ do actual suspend.

If we stick to the 3 step model we discussed at OLS

 1) prepare for sleep (memory allocation, etc...), stop
    doing _any_ non-ATOMIC (or non-NOIO) memory allocation
    beyond the point, the driver has to pre-allocate what
    it will need from now on, eventually running with degraded
    perfs (serialized)
 2) block IOs & save state. Block drivers should stop their
    request queue, drivers impl. a direct /dev interface should
    block processes calling them until they are resumed, etc...
    typically done easily with a semaphore for most of them.
    then save state informations to pre-allocated memory
 3) suspend (IRQs off) Or optionally 4 steps with 3) suspend_irq_on
    and 4) suspend_irq_off.

Then suspend-to-disk would need to call steps 1 and 2 normally,
but not 3 for devices on the storage chain. Then, after the
RAM is copied, that device can be sent a resume request, though
in this case, you indeed need to make sure no user process or
journaling daemon or whatever will inject requests to your
target device queues that are not specifically your pages
beeing thrown to the backing store.
>
>> I understand (please correct me if I'm wrong) that your mecanism is to
>> implement that at a higher level, though I still fail to see the
>> "big picture" of it, especially how you can properly resume state
>> of all device drivers and other in-kernel state informations. Do you
>> store all pages including kernel pages to disk ? 
>
>Yes I store all pages including kernel ones to disk.
>
>> If not, what about
>> open inodes, sockets, etc... ? How do you resume kernel state information
>> for these ? 
>> How do you deal with device-drivers that are configured in
>> some specific way before suspend and has to come back up the same
>> way ?
>
>I need device support for suspend-to-disk, of course. But I need no
>special support on "suspend" device.

I still think it need to be handled slightly differently (see above),
I'm trying to see what has to be common and what not. Defining (and
then implementing) the proper device support is the biggest issue,
I think we have the semantics approximately right in mind Patrick
and I, it's time to write them down :)

(*note about device mem alloc): Some devices need to allocate memory
to be able to save sate. That can be a significant amount of memory
(some framebuffer may want to backup the fb content, huge !). 

So in your case, I beleive you should probably first send the notification
of step 1 (prepare for sleep) to drivers, then do your memory-crunching
thing, then call step 2 and step 3 for all but swap device.

Ben.




-------------------------------------------------------
This sf.net email is sponsored by: Dice - The leading online job board
for high-tech professionals. Search and apply for tech jobs today!
http://seeker.dice.com/seeker.epl?rel_code=31

Re: suspend.c vs driver-model.txt

[Pavel Machek]

Hi!

> >What races can you see?
> 
> Well, existing PM callbacks aren't good enough :) they just don't
> deal with dependencies properly. Also, they don't deal that well

Yep, I know. I'm using Patrick's devicefs callback.

> >> The problem of saving to disk and of saving to memory (that is
> >> machine sleep as I implement it on powerbooks today, RAM content
> >> beeing preserved) is pretty similar.
> >
> >Actually it is quite different.
> >
> >Saving device state is common code, but suspend-to-ram can be done
> >without scheduling, while you need to block for suspend-to-disk.
> 
> No. They end up beeing very similar. Suspend to RAM has to schedule
> because some underlying device drivers will need to schedule to
> properly block their queues as well.

Okay, true. Still they are different because suspend-to-disk needs
working disk driver to save pages.

> I figured out in the Pmac implementation that I could actually let
> the system schedule the whole time up to just before the very last
> step of shutting down the CPU. Userland apps will simply block as
> they rely on IOs for drivers that have been properly blocked, or
> from swap while the swap device may be suspended, etc... CPU
> intensive app would still work until it's very last timeslice is
> used before suspend.

Being extremely cpu-efficient is not 1-st priority goal of
swsusp. First make it work, then make it faster ;-).

> That's also how I got very fast wakeup times. Basically, processes
> start again right away (well, just after a few really important things
> like time are restored), and then drivers are kicked back into life,
> asynchronously if possible, thus user processes that are blocked by
> a given driver will come back to life normally.

I believe you give way too much responsibility to drivers...

> >> So you really need to properly do the prepare/save/suspend steps
> >> on all devices in proper bus ordering so that any device driver
> >> has properly saved state information to memory (which may later
> >> be saved to disk with suspend-to-disk) and has properly blocked
> >> IO queues.
> >> 
> >> The specific case of the device which is used as a backstore
> >> for the RAM save has to be dealt some specific way. 
> >
> >No it does not. I have half of RAM free, I just save-state, copy
> >memory, continue devices, copy saved memory to swap.
> 
> So you resume devices from the "saved state" which isn't the
> state the device was when the machine was really suspended,
> right ? Which means that typically, on resume, the driver could
> end up beeing out of sync with the device if some permanent state
> information exist on the device, but I agree this is a rare case,
> except for... storage. 

There's power cycle and whole kernel bootup. Device state is
completely lost during suspend to disk.

> So I assume you have ways to prevent
> filesystems to be touched at all ?

There's nothing alive that could touch filesystems. If user boots into
non-suspend-aware kernel and writes to disk disk, his fault.

>From Docs/swsusp.txt:

 * BIG FAT WARNING
*********************************************************
 *
 * If you have unsupported (*) devices using DMA...
 *                              ...say goodbye to your data.
 *
 * If you touch anything on disk between suspend and resume...
 *                              ...kiss your data goodbye.
 *
 * If your disk driver does not support suspend... (IDE does)
 *                              ...you'd better find out how to get along
 *                                 without your data.
 *
 * (*) pm interface support is needed to make it safe.

You need to append resume=/dev/your_swap_partition to kernel command
line. Then you suspend by echo 4 > /proc/acpi/sleep.

> As I see it, for your scheme to work properly, you need to,
> somewhat "atomically", save-state all devices so they are
> in coherent state one to each other (devices can well be
> inter-dependant), backup your RAM, then you can kick back
> devices (well, some actually) into life.

Yep, that's what I'm doing.

> This is really only a special case of the generic process I'm
> suggesting then ;)
> 
> Basically, you still need to run the "block IOs then save state
> and suspend" step on all devices in bus ordering. 

I don't need "block IO" step. I just stop everything that could
possibly ask drivers to do IO.

> So that
> part is common with suspend-to-ram. 

Yep, drivers part is identical with suspend-to-ram. 

> Actually, you don't need
> to prevent scheduling before that point, except maybe for
> keeping your "half of RAM free" watermark, but even then, I
> don't see how you acheive that since the kernel itself may
> allocate memory (see note below)

It is not 100% guaranteed that it is possible to get half of RAM
free. In such case suspend-to-disk fails.

> Then you need to use the explicit device model power
> state functions to re-enable power state on the target device
> of the backup. It should in turn re-enable parent devices up
> to the host bus.

No, I just re-enable everything.

> However, that would be inefficient as the net effect would be
> to have your hard disk spin down, be eventually powered off,
> then back up for suspend to RAM, then the machine powered off
			      ~~~
				\____ I believe you mean disk here.

> (and so that hard disk as well).

Yes, it is not effective (and disk will do ugly
spindown-spinup-powerdown cycle). It is still faster than bios
suspend-to-disk ;-).

> Which is why I beleive it would make more sense to specifically
> instruct the target device (and so it's parent) during the
> device suspend loop to _not_ go to sleep, just suspend, block
> IOs, then resume IOs, but _not_ do actual suspend.

In such case you can as well do suspend-but-don't-sleep for *all*
devices. They are going to be powered down, anyway, so who cares.

> If we stick to the 3 step model we discussed at OLS
> 
>  1) prepare for sleep (memory allocation, etc...), stop
>     doing _any_ non-ATOMIC (or non-NOIO) memory allocation
>     beyond the point, the driver has to pre-allocate what
>     it will need from now on, eventually running with degraded
>     perfs (serialized)
>  2) block IOs & save state. Block drivers should stop their
>     request queue, drivers impl. a direct /dev interface should
>     block processes calling them until they are resumed, etc...
>     typically done easily with a semaphore for most of them.
>     then save state informations to pre-allocated memory

I believe you are putting *way* too much responsibility to the
drivers. With your model each driver needs to be able to stop its
users. Ouch. Remember -- there's lot of drivers. You do not want to
add crap^Wcode to them. Better just stop all user programs so that
drivers don't have to care.

>  3) suspend (IRQs off) Or optionally 4 steps with 3) suspend_irq_on
>     and 4) suspend_irq_off.
> 
> Then suspend-to-disk would need to call steps 1 and 2 normally,
> but not 3 for devices on the storage chain. Then, after the

It is hard to tell which devices are on the storage chain. And it
should *not* be neccessary to treat them differently.

> >> If not, what about
> >> open inodes, sockets, etc... ? How do you resume kernel state information
> >> for these ? 
> >> How do you deal with device-drivers that are configured in
> >> some specific way before suspend and has to come back up the same
> >> way ?
> >
> >I need device support for suspend-to-disk, of course. But I need no
> >special support on "suspend" device.
> 
> I still think it need to be handled slightly differently (see above),
> I'm trying to see what has to be common and what not. Defining (and
> then implementing) the proper device support is the biggest issue,
> I think we have the semantics approximately right in mind Patrick
> and I, it's time to write them down :)

Hehe, I believe I have semantics right, too, and have written it down
in kernel/suspend.c ;-)))).

> (*note about device mem alloc): Some devices need to allocate memory
> to be able to save sate. That can be a significant amount of memory
> (some framebuffer may want to backup the fb content, huge !). 

After free_some_memory(), there's very likely *plenty* of memory
available. If framebuffer wants to backup the fb content, and it runs
out of memory, tough, and suspend fails.

[BTW you don't need/want to backup fb content; either its X or its
text console. Text console knows how to repaint itself. X knows how to
repaint itself.]

> So in your case, I beleive you should probably first send the notification
> of step 1 (prepare for sleep) to drivers, then do your memory-crunching
> thing, then call step 2 and step 3 for all but swap device.

If I do memory-freeing, step 1, step 2, step 3, memory-copy it should
be equivalent, AFAICS. That's what I'm doing (but for all devices).

								Pavel
-- 
Worst form of spam? Adding advertisment signatures ala sourceforge.net.
What goes next? Inserting advertisment *into* email?


-------------------------------------------------------
This sf.net email is sponsored by: Dice - The leading online job board
for high-tech professionals. Search and apply for tech jobs today!
http://seeker.dice.com/seeker.epl?rel_code=31

Re: suspend.c vs driver-model.txt

[Benjamin Herrenschmidt]

>I believe you are putting *way* too much responsibility to the
>drivers. With your model each driver needs to be able to stop its
>users. Ouch. Remember -- there's lot of drivers. You do not want to
>add crap^Wcode to them. Better just stop all user programs so that
>drivers don't have to care.

Well, you do have to deal with drivers. Users programs is not enough,
drivers can be kicked by things going on in the kernel as well.
 
Also, in most case, it's not that difficult to implement.

 - Network drivers just need to drop packets, pretty easily done
 - Block drivers just need to stop their queue
 - Things like serial drivers usually don't have to do anything
   terrific, just stop the chip interrupt and users will be blocked
   (or will timeout, but that is not a problem at this point)
 - For pmac sound driver, I choose to have a mutex that cause users
   to be blocked, but that was the "easy" way to quickly get that
   done, a more subtle mecanism could be done, dropping samples at
   approximately the output rate. The mutex makes it easy to make
   the whole driver SMP safe as well regarding mixer tweaks though.

I implemented this in a whole bunch of drivers used on Apple laptops
and in the end, the most code is _not_ doing the above properly in
drivers, but doing the actual state save/suspend & resume on the
chip (here you have to deal with broken HW, subtle timing issues, etc...)

So drivers _has_ to be done the right way. If a driver doesn't, then
it will be unsuitable for PM (be it suspend to RAM or to disk). No
problem with that ;)

>>  3) suspend (IRQs off) Or optionally 4 steps with 3) suspend_irq_on
>>     and 4) suspend_irq_off.
>> 
>> Then suspend-to-disk would need to call steps 1 and 2 normally,
>> but not 3 for devices on the storage chain. Then, after the
>
>It is hard to tell which devices are on the storage chain. And it
>should *not* be neccessary to treat them differently.

Well... I would prefer then broadcasting a "save_state/block" to
all of them, then no suspend, and a wakeup. That keeps the existing
semantics valid.

>Hehe, I believe I have semantics right, too, and have written it down
>in kernel/suspend.c ;-)))).

Hrm... not sure about those ;) Anyway, Patrick will probably write down
something precising what we came up with during OLS.

>> (*note about device mem alloc): Some devices need to allocate memory
>> to be able to save sate. That can be a significant amount of memory
>> (some framebuffer may want to backup the fb content, huge !). 
>
>After free_some_memory(), there's very likely *plenty* of memory
>available. If framebuffer wants to backup the fb content, and it runs
>out of memory, tough, and suspend fails.
>
>[BTW you don't need/want to backup fb content; either its X or its
>text console. Text console knows how to repaint itself. X knows how to
>repaint itself.]

Sure, though some fb apps don't. Though here, I agree we need those to
get notified as part of the process (which I do on pmac before doing the
device notification, as part of my /dev/apm_bios emulation) so they can
actually do that repainting properly.

>> So in your case, I beleive you should probably first send the notification
>> of step 1 (prepare for sleep) to drivers, then do your memory-crunching
>> thing, then call step 2 and step 3 for all but swap device.
>
>If I do memory-freeing, step 1, step 2, step 3, memory-copy it should
>be equivalent, AFAICS. That's what I'm doing (but for all devices).

Not sure. step 1 will affect memory allocation, swap etc... I'd rather
have that done before you start doing the memory freeing. Well, that's
mostly a matter of taste though.

Ben.




-------------------------------------------------------
This sf.net email is sponsored by: Dice - The leading online job board
for high-tech professionals. Search and apply for tech jobs today!
http://seeker.dice.com/seeker.epl?rel_code=31

Re: suspend.c vs driver-model.txt

[Benjamin Herrenschmidt]

>
>There's power cycle and whole kernel bootup. Device state is
>completely lost during suspend to disk.

As it is during suspend-to-ram in most cases (most devices except
a couple of them are actually powered down and the drivers have to
bring them back up, at least on pmac, though on x86, the APM BIOS
helps you a bit there).

So basically, drivers are responsible to bring back the device
in whatever state it was at the save_state step of the suspend,
in the same way for but suspend to disk and to RAM. What I want to
make sure is that the driver-side semantics remain the same for
swsusp and suspend-to-RAM.
That includes the fact that save state will actually block IOs, and
that is necessary to have a coherent state among devices.

Note that you don't need to run the last step of driver notification
though (the actual "suspend" state), this will save you the disk
powercycle effect on suspend.

Ben.





-------------------------------------------------------
This sf.net email is sponsored by: Dice - The leading online job board
for high-tech professionals. Search and apply for tech jobs today!
http://seeker.dice.com/seeker.epl?rel_code=31

Re: suspend.c vs driver-model.txt

[Benjamin Herrenschmidt]

>Why? You should modify network stack not to post any packets to them.

No, you should have the _driver_ ask the network stack not to get
packets on _this_ specific interface.

>>  - Block drivers just need to stop their queue
>
>Block layer should not attempt to pass anything to driver when
>suspended. (Theres one block layer, but 100 block drivers; which one
>would you like to modify?)

Again, it's a specific driver that has to inform the block layer.

Suspend hierarchy isn't per-subsystem. You may well need a given
subsystem to be alive for a separate driver to suspend itself
properly. It has to be bus-driven.

However, the bus-binding of a driver is completely orthogonal
to the subsystems to which that driver is attached (and gets
requests from).

>[Currently I'm stopping it at even higher level by stopping
>processes. I still think stopping everything is easiest solution, and
>probably only one doable for 2.6.]

Well, it could be done stopping process first, until all drivers are
safe enouh. That mean keeping the IO blocking semantics for drivers
while having the process stopped to help deal with drivers not
implementing that at first, but I'd prefer doing it the right way
first.

>>  - Things like serial drivers usually don't have to do anything
>>    terrific, just stop the chip interrupt and users will be blocked
>>    (or will timeout, but that is not a problem at this point)
>>  - For pmac sound driver, I choose to have a mutex that cause users
>>    to be blocked, but that was the "easy" way to quickly get that
>>    done, a more subtle mecanism could be done, dropping samples at
>>    approximately the output rate. The mutex makes it easy to make
>>    the whole driver SMP safe as well regarding mixer tweaks though.
>
>Why don't you just put that mutex into sys_read etc? Or why not system
>stopping? There's just too much drivers.

sys_read, sys_ioctl, etc... ? well, drivers can be kicked by other
means than just syscalls. And if you tear down subsystems, you take
the risk of stopping a subsystem before a driver that need it for
proper suspend had a chance to use it.

>Kernel tells them that console is being switched away; so it is their
>responsibility. Surprise, special console for suspend is not only
>pretty but also usefull ;-).

Yes, in this specific case, I agree, it's much better than make faked
fbcon ops I use in fbdev drivers ;)

Ben.




-------------------------------------------------------
This sf.net email is sponsored by: Dice - The leading online job board
for high-tech professionals. Search and apply for tech jobs today!
http://seeker.dice.com/seeker.epl?rel_code=31

Re: suspend.c vs driver-model.txt

[Benjamin Herrenschmidt]

>1) implies that new requests are not coming, so driver authors do not
>have to do if (suspended) sleep() in their hot paths.

What about, let's say, an USB device driver that needs the USB host
chip (and thus the whole USB subsystem) to be still alive to send
proper commands to the device for suspend ? (typically setup the
remote wakeup when supported, etc...)

There can be even more nasty dependencies when you start stacking
things.

There are ordering dependencies. Those dependencies are dictated
by the bus layout that is fortunately provided for us by the new
driver model (let's not mix it with driverfs). It's not a subsystem
that has to be suspended not to send requests to a driver, it's
a driver that has to stop accepting requests, though it's then
perfectly valid for the subsystem to provide a "helper" routine
for the driver to deal with that.
(Typically, those "helpers" already exist for other purposes anyway
like stopping your block queue, stopping your network queue, etc...)

Ben.




-------------------------------------------------------
This sf.net email is sponsored by: Dice - The leading online job board
for high-tech professionals. Search and apply for tech jobs today!
http://seeker.dice.com/seeker.epl?rel_code=31

Re: suspend.c vs driver-model.txt

[Pavel Machek]

Hi!

> >There's power cycle and whole kernel bootup. Device state is
> >completely lost during suspend to disk.
> 
> As it is during suspend-to-ram in most cases (most devices except
> a couple of them are actually powered down and the drivers have to
> bring them back up, at least on pmac, though on x86, the APM BIOS
> helps you a bit there).

APM BIOS is irrelevant here (when using APM BIOS, everything is done
by BIOS).

> So basically, drivers are responsible to bring back the device
> in whatever state it was at the save_state step of the suspend,
> in the same way for but suspend to disk and to RAM. What I want to
> make sure is that the driver-side semantics remain the same for
> swsusp and suspend-to-RAM.
> That includes the fact that save state will actually block IOs, and
> that is necessary to have a coherent state among devices.

I actually made suspend-to-RAM working as a side effect of making
suspend-to-disk working.

What I'm doing is "drivers don't have to care about stopping requests
because everything is stopped". If you are willing to go through all
drivers and add stopping in them, thats fine with me, but I believe
its post-2.6.

> Note that you don't need to run the last step of driver notification
> though (the actual "suspend" state), this will save you the disk
> powercycle effect on suspend.

Yep, I can get rid of spindown/spinup..
								Pavel
-- 
Casualities in World Trade Center: ~3k dead inside the building,
cryptography in U.S.A. and free speech in Czech Republic.


-------------------------------------------------------
This sf.net email is sponsored by: Dice - The leading online job board
for high-tech professionals. Search and apply for tech jobs today!
http://seeker.dice.com/seeker.epl?rel_code=31

Re: suspend.c vs driver-model.txt

[Benjamin Herrenschmidt]

>
>> I believe we should ease the pain of driver maintainers by
>> guaranteeing:
>> 
>> when driver_suspend() / driver_resume() is called
>> 
>> 1) userspace is stopped.
>
>I thought that was a given ;)

This isn't ;) I have userspace working up to the very latest
steps of suspend without trouble here ;) Userspace just stops
when blocked on an IO to a suspended driver, or just when the
CPU stops.

Though for swsusp, the problem of freeing the memory makes
you need to stop userspace, except if you can teach the VM to
not try to use those physical pages again until resume.

>> 2) we are running with just one active CPU.
>
>Sure. We can tie that into subsystem notification, like this:
>
>- The level at which we want to stop requests appears to be at the device 
>  class level. 

yes, _but_ the ordering is still dictated by the bus binding, not the
device class.

>- Device classes are coming RSN. 
>
>- CPUs belong to a device class.
>
>- We can integrate PM notication into device classes.

For the above reason, PM notification has to go via the bus hierarchy,
not the class hierarchy, though the device can then notify it's class
that it's no longer willing to do anything until resumed.

>- The CPU device class can shut off all APs (application processors [silly 
>  Intel term]).





-------------------------------------------------------
This sf.net email is sponsored by: Dice - The leading online job board
for high-tech professionals. Search and apply for tech jobs today!
http://seeker.dice.com/seeker.epl?rel_code=31

Re: suspend.c vs driver-model.txt

[Benjamin Herrenschmidt]

>Benjamin Herrenschmidt was arguing for doing driver_suspend() while
>userspace is still running. If you assumed stopped userspace, you
>don't have to add semaphore into device read() etc...

And stopping any kernel thread that may do that etc... I really
don't like it that much, but well, it can be a working solution for
2.6 for drivers having a direct userspace /dev interface.

>> > 2) we are running with just one active CPU.
>> 
>> Sure. We can tie that into subsystem notification, like this:
>> 
>> - The level at which we want to stop requests appears to be at the device 
>>   class level. 
>> 
>> - Device classes are coming RSN. 
>> 
>> - CPUs belong to a device class.
>> 
>> - We can integrate PM notication into device classes.
>> 
>> - The CPU device class can shut off all APs (application processors [silly 
>>   Intel term]).
>
>I believe shutting down CPUs should be done outside device_suspend()
>so all device_suspends() run on just one CPU...

That makes things easier, I tend to agree here.

Ben.




-------------------------------------------------------
This sf.net email is sponsored by: Dice - The leading online job board
for high-tech professionals. Search and apply for tech jobs today!
http://seeker.dice.com/seeker.epl?rel_code=31

Re: suspend.c vs driver-model.txt

[Pavel Machek]

Hi!

> >I believe you are putting *way* too much responsibility to the
> >drivers. With your model each driver needs to be able to stop its
> >users. Ouch. Remember -- there's lot of drivers. You do not want to
> >add crap^Wcode to them. Better just stop all user programs so that
> >drivers don't have to care.
> 
> Well, you do have to deal with drivers. Users programs is not enough,
> drivers can be kicked by things going on in the kernel as well.

Well, so that "thing" that could kick it has to be found and made
suspend-aware. Better than touching 1000 drivers.

> Also, in most case, it's not that difficult to implement.

Wrong level of abstraction.

>  - Network drivers just need to drop packets, pretty easily done

Why? You should modify network stack not to post any packets to them.

>  - Block drivers just need to stop their queue

Block layer should not attempt to pass anything to driver when
suspended. (Theres one block layer, but 100 block drivers; which one
would you like to modify?)

[Currently I'm stopping it at even higher level by stopping
processes. I still think stopping everything is easiest solution, and
probably only one doable for 2.6.]

>  - Things like serial drivers usually don't have to do anything
>    terrific, just stop the chip interrupt and users will be blocked
>    (or will timeout, but that is not a problem at this point)
>  - For pmac sound driver, I choose to have a mutex that cause users
>    to be blocked, but that was the "easy" way to quickly get that
>    done, a more subtle mecanism could be done, dropping samples at
>    approximately the output rate. The mutex makes it easy to make
>    the whole driver SMP safe as well regarding mixer tweaks though.

Why don't you just put that mutex into sys_read etc? Or why not system
stopping? There's just too much drivers.

> >>  3) suspend (IRQs off) Or optionally 4 steps with 3) suspend_irq_on
> >>     and 4) suspend_irq_off.
> >> 
> >> Then suspend-to-disk would need to call steps 1 and 2 normally,
> >> but not 3 for devices on the storage chain. Then, after the
> >
> >It is hard to tell which devices are on the storage chain. And it
> >should *not* be neccessary to treat them differently.
> 
> Well... I would prefer then broadcasting a "save_state/block" to
> all of them, then no suspend, and a wakeup. That keeps the existing
> semantics valid.

That's probably doable.

> >> (*note about device mem alloc): Some devices need to allocate memory
> >> to be able to save sate. That can be a significant amount of memory
> >> (some framebuffer may want to backup the fb content, huge !). 
> >
> >After free_some_memory(), there's very likely *plenty* of memory
> >available. If framebuffer wants to backup the fb content, and it runs
> >out of memory, tough, and suspend fails.
> >
> >[BTW you don't need/want to backup fb content; either its X or its
> >text console. Text console knows how to repaint itself. X knows how to
> >repaint itself.]
> 
> Sure, though some fb apps don't. Though here, I agree we need those to
> get notified as part of the process (which I do on pmac before doing the
> device notification, as part of my /dev/apm_bios emulation) so they can
> actually do that repainting properly.

Kernel tells them that console is being switched away; so it is their
responsibility. Surprise, special console for suspend is not only
pretty but also usefull ;-).
								Pavel
-- 
Casualities in World Trade Center: ~3k dead inside the building,
cryptography in U.S.A. and free speech in Czech Republic.


-------------------------------------------------------
This sf.net email is sponsored by: Dice - The leading online job board
for high-tech professionals. Search and apply for tech jobs today!
http://seeker.dice.com/seeker.epl?rel_code=31

Re: suspend.c vs driver-model.txt

[Patrick Mochel]

> > >There's power cycle and whole kernel bootup. Device state is
> > >completely lost during suspend to disk.
> > 
> > As it is during suspend-to-ram in most cases (most devices except
> > a couple of them are actually powered down and the drivers have to
> > bring them back up, at least on pmac, though on x86, the APM BIOS
> > helps you a bit there).
> 
> APM BIOS is irrelevant here (when using APM BIOS, everything is done
> by BIOS).

No, it's not. There is still a suspend sequence that must take place in 
the kernel. 

> What I'm doing is "drivers don't have to care about stopping requests
> because everything is stopped". If you are willing to go through all
> drivers and add stopping in them, thats fine with me, but I believe
> its post-2.6.

We have to deal with drivers. Someone has to modify the drivers to make 
them properly support power management. 

Now, it's not as bad as it sounds. Many drivers have pm_callback calls
that should be modified to the new style. So, we know what to do for many
devices. Other than those, the actively maintained drivers will be 
converted, as will be the most popular drivers. Others will gradually be 
taught to deal with it with PM properly. 

The burden is not necessarily on our shoulders. And, we can break 
everything. Trust me, I have permission ;)

	-pat



-------------------------------------------------------
This sf.net email is sponsored by: Dice - The leading online job board
for high-tech professionals. Search and apply for tech jobs today!
http://seeker.dice.com/seeker.epl?rel_code=31

Re: suspend.c vs driver-model.txt

[Benjamin Herrenschmidt]

>> Benjamin Herrenschmidt was arguing for doing driver_suspend() while
>> userspace is still running. If you assumed stopped userspace, you
>> don't have to add semaphore into device read() etc...
>
>Having been around this a few times for embedded devices I've found
>nothing that cleanly solves the problems except
>
>Call all the devices "suspend ready irq on" methods
>Disable interrupts
>Call all the devices "final suspend" method

Yes, I remember you telling me about it, which simply involves
breaking the 3rd (ie. "suspend") PM notification into a
"suspend_irq_on" and a "suspend_irq_off". Drivers that don't
care will use only one of them. But that's not a real issue,
the problems are really around the "save_state & block IOs"
step which is step 2.

 step 1: prepare (that is allocate any memory needed, make sure
         the driver can still operate without allocating more
         memory, or eventually only ATOMIC or NOIO).

 step 2: save_state & block IOs, that is make sure the saved state
         is consistent with the last IOs and that no more IO will
         be processed. The driver don't need to wait for completion
         of pending IOs at this stage I beleive if those don't
         impact the saved state, but it may be better to wait for
         them at first.

 step 3: suspend_irq_on, here we do the real suspend of the chip
         if possible (note that some arch may need to use D1 or D2
         instead of D3 for various reasons, but that will be dealt
         on a driver specific basis for now). That step don't need
         to be called by swsusp

 step 4: suspend_irq_off, followup of step 3) for drivers that
         need it.

Ben.




-------------------------------------------------------
This sf.net email is sponsored by: Dice - The leading online job board
for high-tech professionals. Search and apply for tech jobs today!
http://seeker.dice.com/seeker.epl?rel_code=31

Re: suspend.c vs driver-model.txt

[Benjamin Herrenschmidt]

>I'd not call it right way. 
>
>What you propose is adding test into *fast* path of every driver. I'm
>not touching fast path at all, I just stop all processes. That way
>suspend takes longer but system normally runs faster. 

Hrm... most of the time that test already exist. Network & block
drivers already have blocking semantics. Serial-like drivers will
only need to switch their interrupt source off if any or timer
or whatever. The only case on pmac where I had to touch the normal
path was some ioctl() path (which aren't really fast path anyway
like preventing some ioctl's to network drivers from banging sleeping
hardware), and the sound driver, but I needed that already for SMP
safety.

>> >>  - Things like serial drivers usually don't have to do anything
>> >>    terrific, just stop the chip interrupt and users will be blocked
>> >>    (or will timeout, but that is not a problem at this point)
>> >>  - For pmac sound driver, I choose to have a mutex that cause users
>> >>    to be blocked, but that was the "easy" way to quickly get that
>> >>    done, a more subtle mecanism could be done, dropping samples at
>> >>    approximately the output rate. The mutex makes it easy to make
>> >>    the whole driver SMP safe as well regarding mixer tweaks though.
>> >
>> >Why don't you just put that mutex into sys_read etc? Or why not system
>> >stopping? There's just too much drivers.
>> 
>> sys_read, sys_ioctl, etc... ? well, drivers can be kicked by other
>> means than just syscalls. And if you tear down subsystems, you take
>> the risk of stopping a subsystem before a driver that need it for
>> proper suspend had a chance to use it.
>
>Do you have example of such subsystem?

i2c, usb, ieee1394, then all the possibilities of stacking things
on top of each other...

>Certainly no driver depends on userspace so we may stop userspace.

Well, I know at least a DSL softmodem driver that relies on userspace
(or will soon) as the core softmodem code is moved out of the kernel ;)

>Hardware drivers do not depend on IP stack, etc...

_device_ driver include stacked devices. It includes a scsi-over-IP
device if any on which you have perfect right to want to send a command
before suspending.

Ben.





-------------------------------------------------------
This sf.net email is sponsored by: Dice - The leading online job board
for high-tech professionals. Search and apply for tech jobs today!
http://seeker.dice.com/seeker.epl?rel_code=31

Re: suspend.c vs driver-model.txt

[Benjamin Herrenschmidt]

>That's okay. We stop userspace. Then we attempt to suspend usb
>camera. Thanks to driver ordering, USB host chip is still alive. When
>all devices are done, USB host chips is suspended. Thanks to ordering,
>there's noone alive to ask USB host chip to do anything, so USB host
>chip driver does not need if (i_m_suspended) reject_request(), because
>it just can not happen.

In real life it did happen and that has been an issue for me in 2.4,
but well... that's because the USB device drivers don't deal with
suspend currently.

But the point here is that you don't suspend the _subsystem_ (USB,
block layer, network stack, whatever), but the individual drivers.

It do make sense to provide a way to stop requests. Requests are
not always originated from userspace. Several kernel services can
issue requests (VM, cache, whatever stuff on top of an i2c device,
etc...) and you can't shut down those services because you don't
have ordering rules for them, you only have ordering rules for
the drivers.

Ben.




-------------------------------------------------------
This sf.net email is sponsored by: Dice - The leading online job board
for high-tech professionals. Search and apply for tech jobs today!
http://seeker.dice.com/seeker.epl?rel_code=31

Re: suspend.c vs driver-model.txt

[Pavel Machek]

Hi!

> > > >There's power cycle and whole kernel bootup. Device state is
> > > >completely lost during suspend to disk.
> > > 
> > > As it is during suspend-to-ram in most cases (most devices except
> > > a couple of them are actually powered down and the drivers have to
> > > bring them back up, at least on pmac, though on x86, the APM BIOS
> > > helps you a bit there).
> > 
> > APM BIOS is irrelevant here (when using APM BIOS, everything is done
> > by BIOS).
> 
> No, it's not. There is still a suspend sequence that must take place in 
> the kernel. 

In ideal world of working APM biosen, such sequence would be
useless. Unfortunately, we live in real world.

> > What I'm doing is "drivers don't have to care about stopping requests
> > because everything is stopped". If you are willing to go through all
> > drivers and add stopping in them, thats fine with me, but I believe
> > its post-2.6.
> 
> We have to deal with drivers. Someone has to modify the drivers to make 
> them properly support power management. 
> 
> Now, it's not as bad as it sounds. Many drivers have pm_callback calls
> that should be modified to the new style. So, we know what to do for many
> devices. Other than those, the actively maintained drivers will be 
> converted, as will be the most popular drivers. Others will gradually be 
> taught to deal with it with PM properly. 

I believe we should ease the pain of driver maintainers by
guaranteeing:

when driver_suspend() / driver_resume() is called

1) userspace is stopped.

2) we are running with just one active CPU.

1) implies that new requests are not coming, so driver authors do not
have to do if (suspended) sleep() in their hot paths.

								Pavel
-- 
Casualities in World Trade Center: ~3k dead inside the building,
cryptography in U.S.A. and free speech in Czech Republic.


-------------------------------------------------------
This sf.net email is sponsored by: Dice - The leading online job board
for high-tech professionals. Search and apply for tech jobs today!
http://seeker.dice.com/seeker.epl?rel_code=31

Re: suspend.c vs driver-model.txt

[Patrick Mochel]

> > Well, you do have to deal with drivers. Users programs is not enough,
> > drivers can be kicked by things going on in the kernel as well.
> 
> Well, so that "thing" that could kick it has to be found and made
> suspend-aware. Better than touching 1000 drivers.

Ok, so the real prize is in finding exactly what the right level of 
abstraction is. I will give you that one. But in doing so, it is not a 
decision of "Touching Every Driver" vs. "Not Touching Any Drivers".

Drivers must be modified. 

Notifying subsystems of a suspend/resume transition is a good thing to do. 
We touched on this briefly in Ottawa, though nothing was written down, as 
we were more concerned with dealing with proper driver ordering. 

Even if we stop I/O requests to the devices, we still have to save and 
restore their state, as well as put the devices in their own low power 
state. 

There are about 6 pages in the ACPI 2.0 spec that I actually trust and am
willing to repeat. Section 9.1.6: "Transitioning from the Working to the 
Sleep State" (pg. 231), it says:

...
3) OSPM places all devices into their respective Dx state.
...

If you do that, you must save and restore all device context that will be 
lost during the transition. 

You don't want to re-init the device because you can assume information 
based on what you saved away. It wastes time, and it's unnecessary. 

Suspend states are simply an illusion to the user. They want the system to
come back to _exactly_ where they left off, as fast as possible.  
Powerbooks do this incredibly well. Windows even does a decent job. 

	-pat



-------------------------------------------------------
This sf.net email is sponsored by: Dice - The leading online job board
for high-tech professionals. Search and apply for tech jobs today!
http://seeker.dice.com/seeker.epl?rel_code=31

Re: suspend.c vs driver-model.txt

[Benjamin Herrenschmidt]

>Okay, if impact is minimal.... I can probably live with that. I still
>do not know if slightly faster suspend is worth the effort.

it's not only slightly faster, it "feels" safer as well not to
rely on userland & selected kernel threads to be the only possible
source of incoming IOs.

>> >Do you have example of such subsystem?
>> 
>> i2c, usb, ieee1394, then all the possibilities of stacking things
>> on top of each other...
>> 
>> >Certainly no driver depends on userspace so we may stop userspace.
>> 
>> Well, I know at least a DSL softmodem driver that relies on userspace
>> (or will soon) as the core softmodem code is moved out of the kernel ;)
>> 
>> >Hardware drivers do not depend on IP stack, etc...
>> 
>> _device_ driver include stacked devices. It includes a scsi-over-IP
>> device if any on which you have perfect right to want to send a command
>> before suspending.
>
>I believe device drivers do not need to know. If you are running
>scsi-over-IP you don't need to inform other side you are
>suspending. It is similar to your NFS server; you don't tell your NFS
>server that you are suspending. scsi-over-IP device should not need to
>know.

Well, SCSI-over-IP is probably not a good example, but you can have
very valid reasons to inform a device over whatever protocol chain
that you are suspending. 

Ben.





-------------------------------------------------------
This sf.net email is sponsored by: Dice - The leading online job board
for high-tech professionals. Search and apply for tech jobs today!
http://seeker.dice.com/seeker.epl?rel_code=31

Re: suspend.c vs driver-model.txt

[Patrick Mochel]

> I believe we should ease the pain of driver maintainers by
> guaranteeing:
> 
> when driver_suspend() / driver_resume() is called
> 
> 1) userspace is stopped.

I thought that was a given ;)

> 2) we are running with just one active CPU.

Sure. We can tie that into subsystem notification, like this:

- The level at which we want to stop requests appears to be at the device 
  class level. 

- Device classes are coming RSN. 

- CPUs belong to a device class.

- We can integrate PM notication into device classes.

- The CPU device class can shut off all APs (application processors [silly 
  Intel term]).

	-pat



-------------------------------------------------------
This sf.net email is sponsored by: Dice - The leading online job board
for high-tech professionals. Search and apply for tech jobs today!
http://seeker.dice.com/seeker.epl?rel_code=31

Re: suspend.c vs driver-model.txt

[Benjamin Herrenschmidt]

>I really need all DMAs to be off after step 2; so I need all drivers
>doing DMA to wait for completion. [I can not do atomic copy of memory
>with DMA writing over it behind my back.]

Ok, that is a valid point, so let's say that step 2) explicitely
requires IO operations to be complete and DMA stopped.

Ben.




-------------------------------------------------------
This sf.net email is sponsored by: Dice - The leading online job board
for high-tech professionals. Search and apply for tech jobs today!
http://seeker.dice.com/seeker.epl?rel_code=31

Re: suspend.c vs driver-model.txt

[Pavel Machek]

Hi!

> > > Well, you do have to deal with drivers. Users programs is not enough,
> > > drivers can be kicked by things going on in the kernel as well.
> > 
> > Well, so that "thing" that could kick it has to be found and made
> > suspend-aware. Better than touching 1000 drivers.
> 
> Ok, so the real prize is in finding exactly what the right level of 
> abstraction is. I will give you that one. But in doing so, it is not a 
> decision of "Touching Every Driver" vs. "Not Touching Any Drivers".

Agreed.

> Drivers must be modified. 

Yep.

> Even if we stop I/O requests to the devices, we still have to save and 
> restore their state, as well as put the devices in their own low power 
> state. 

We have to save/restore their state.

We do not need to put them in low power states for swsusp (we are
powered), but we should do that for S3.

> You don't want to re-init the device because you can assume information 
> based on what you saved away. It wastes time, and it's unnecessary. 

More time consuming etc but simpler in some cases ;-).

[ne2000 driver is so used to working with crap that you can live
without suspend/resume support. You go S3. You return. ne2000 driver
tries to communicate with the card, and it appears dead, so it kicks
it back alive after timeout. Hack hack ;-)].

> Suspend states are simply an illusion to the user. They want the system to
> come back to _exactly_ where they left off, as fast as possible.  
> Powerbooks do this incredibly well. Windows even does a decent job. 

Yes but it is not actually important if we can enter S3 in 100 msec or
120 msec, right? Same goes for resume. We want to avoid touching hot
paths.
								Pavel
-- 
Casualities in World Trade Center: ~3k dead inside the building,
cryptography in U.S.A. and free speech in Czech Republic.


-------------------------------------------------------
This sf.net email is sponsored by: Dice - The leading online job board
for high-tech professionals. Search and apply for tech jobs today!
http://seeker.dice.com/seeker.epl?rel_code=31

Re: suspend.c vs driver-model.txt

[Benjamin Herrenschmidt]

>All kernel services that can issue requests need kernel threads,
>right? My approach is to take kernel threads one by one and make them
>suspend-aware.

It wouldn' be bad to do that too. I just "feel" it safer and better
to define the driver semantics to be IO blocking. Some user processes
still want explicit notification, as will do some kernel threads I
beleive (example of user processes like this are those doing HW banging,
like XFree, there, console switch is really only a workaround, but
XFree is already equiped to deal with suspend requests from /dev/apm_bios).


Ben.




-------------------------------------------------------
This sf.net email is sponsored by: Dice - The leading online job board
for high-tech professionals. Search and apply for tech jobs today!
http://seeker.dice.com/seeker.epl?rel_code=31

Re: suspend.c vs driver-model.txt

[Pavel Machek]

Hi!

> > I believe we should ease the pain of driver maintainers by
> > guaranteeing:
> > 
> > when driver_suspend() / driver_resume() is called
> > 
> > 1) userspace is stopped.
> 
> I thought that was a given ;)

Benjamin Herrenschmidt was arguing for doing driver_suspend() while
userspace is still running. If you assumed stopped userspace, you
don't have to add semaphore into device read() etc...

> > 2) we are running with just one active CPU.
> 
> Sure. We can tie that into subsystem notification, like this:
> 
> - The level at which we want to stop requests appears to be at the device 
>   class level. 
> 
> - Device classes are coming RSN. 
> 
> - CPUs belong to a device class.
> 
> - We can integrate PM notication into device classes.
> 
> - The CPU device class can shut off all APs (application processors [silly 
>   Intel term]).

I believe shutting down CPUs should be done outside device_suspend()
so all device_suspends() run on just one CPU...
								Pavel
-- 
Casualities in World Trade Center: ~3k dead inside the building,
cryptography in U.S.A. and free speech in Czech Republic.


-------------------------------------------------------
This sf.net email is sponsored by: Dice - The leading online job board
for high-tech professionals. Search and apply for tech jobs today!
http://seeker.dice.com/seeker.epl?rel_code=31

Re: suspend.c vs driver-model.txt

[Patrick Mochel]

On Tue, 30 Jul 2002, Pavel Machek wrote:

> Hi!
> 
> > > I believe we should ease the pain of driver maintainers by
> > > guaranteeing:
> > > 
> > > when driver_suspend() / driver_resume() is called
> > > 
> > > 1) userspace is stopped.
> > 
> > I thought that was a given ;)
> 
> Benjamin Herrenschmidt was arguing for doing driver_suspend() while
> userspace is still running. If you assumed stopped userspace, you
> don't have to add semaphore into device read() etc...

Ok, let me go back and read again. :) I'm sure Ben will pipe up, too. 

> I believe shutting down CPUs should be done outside device_suspend()
> so all device_suspends() run on just one CPU...

Right. But, the subsystem notification would presumably happen outside of 
that.

	-pat



-------------------------------------------------------
This sf.net email is sponsored by: Dice - The leading online job board
for high-tech professionals. Search and apply for tech jobs today!
http://seeker.dice.com/seeker.epl?rel_code=31

Re: suspend.c vs driver-model.txt

[Pavel Machek]

Hi!

> > I believe shutting down CPUs should be done outside device_suspend()
> > so all device_suspends() run on just one CPU...
> 
> Right. But, the subsystem notification would presumably happen outside of 
> that.

Okay, I can live with that.

[So we are getting "struct subsystem { }" ?]
								Pavel
-- 
Casualities in World Trade Center: ~3k dead inside the building,
cryptography in U.S.A. and free speech in Czech Republic.


-------------------------------------------------------
This sf.net email is sponsored by: Dice - The leading online job board
for high-tech professionals. Search and apply for tech jobs today!
http://seeker.dice.com/seeker.epl?rel_code=31

Re: suspend.c vs driver-model.txt

[Patrick Mochel]

> [So we are getting "struct subsystem { }" ?]

We're getting at least 'struct device_class'.

	-pat



-------------------------------------------------------
This sf.net email is sponsored by: Dice - The leading online job board
for high-tech professionals. Search and apply for tech jobs today!
http://seeker.dice.com/seeker.epl?rel_code=31

Re: suspend.c vs driver-model.txt

[Pavel Machek]

Hi!


> >[Currently I'm stopping it at even higher level by stopping
> >processes. I still think stopping everything is easiest solution, and
> >probably only one doable for 2.6.]
> 
> Well, it could be done stopping process first, until all drivers are
> safe enouh. That mean keeping the IO blocking semantics for drivers
> while having the process stopped to help deal with drivers not
> implementing that at first, but I'd prefer doing it the right way
> first.

I'd not call it right way. 

What you propose is adding test into *fast* path of every driver. I'm
not touching fast path at all, I just stop all processes. That way
suspend takes longer but system normally runs faster.

> >>  - Things like serial drivers usually don't have to do anything
> >>    terrific, just stop the chip interrupt and users will be blocked
> >>    (or will timeout, but that is not a problem at this point)
> >>  - For pmac sound driver, I choose to have a mutex that cause users
> >>    to be blocked, but that was the "easy" way to quickly get that
> >>    done, a more subtle mecanism could be done, dropping samples at
> >>    approximately the output rate. The mutex makes it easy to make
> >>    the whole driver SMP safe as well regarding mixer tweaks though.
> >
> >Why don't you just put that mutex into sys_read etc? Or why not system
> >stopping? There's just too much drivers.
> 
> sys_read, sys_ioctl, etc... ? well, drivers can be kicked by other
> means than just syscalls. And if you tear down subsystems, you take
> the risk of stopping a subsystem before a driver that need it for
> proper suspend had a chance to use it.

Do you have example of such subsystem?

Certainly no driver depends on userspace so we may stop userspace.

Hardware drivers do not depend on IP stack, etc...
								Pavel
-- 
Casualities in World Trade Center: ~3k dead inside the building,
cryptography in U.S.A. and free speech in Czech Republic.


-------------------------------------------------------
This sf.net email is sponsored by: Dice - The leading online job board
for high-tech professionals. Search and apply for tech jobs today!
http://seeker.dice.com/seeker.epl?rel_code=31

Re: suspend.c vs driver-model.txt

[Pavel Machek]

Hi!

> >1) implies that new requests are not coming, so driver authors do not
> >have to do if (suspended) sleep() in their hot paths.
> 
> What about, let's say, an USB device driver that needs the USB host
> chip (and thus the whole USB subsystem) to be still alive to send
> proper commands to the device for suspend ? (typically setup the
> remote wakeup when supported, etc...)

That's okay. We stop userspace. Then we attempt to suspend usb
camera. Thanks to driver ordering, USB host chip is still alive. When
all devices are done, USB host chips is suspended. Thanks to ordering,
there's noone alive to ask USB host chip to do anything, so USB host
chip driver does not need if (i_m_suspended) reject_request(), because
it just can not happen.
								Pavel
-- 
Casualities in World Trade Center: ~3k dead inside the building,
cryptography in U.S.A. and free speech in Czech Republic.


-------------------------------------------------------
This sf.net email is sponsored by: Dice - The leading online job board
for high-tech professionals. Search and apply for tech jobs today!
http://seeker.dice.com/seeker.epl?rel_code=31

Re: suspend.c vs driver-model.txt

[Pavel Machek]

Hi!

> >> I believe we should ease the pain of driver maintainers by
> >> guaranteeing:
> >> 
> >> when driver_suspend() / driver_resume() is called
> >> 
> >> 1) userspace is stopped.
> >
> >I thought that was a given ;)
> 
> This isn't ;) I have userspace working up to the very latest
> steps of suspend without trouble here ;) Userspace just stops
> when blocked on an IO to a suspended driver, or just when the
> CPU stops.

Yep, but drivers need to be modified more in your model, and it may
have performance impact during "normal" use of system. I'd like to
avoid that.
								Pavel
-- 
Casualities in World Trade Center: ~3k dead inside the building,
cryptography in U.S.A. and free speech in Czech Republic.


-------------------------------------------------------
This sf.net email is sponsored by: Dice - The leading online job board
for high-tech professionals. Search and apply for tech jobs today!
http://seeker.dice.com/seeker.epl?rel_code=31

Re: suspend.c vs driver-model.txt

[Pavel Machek]

Hi!

> >I'd not call it right way. 
> >
> >What you propose is adding test into *fast* path of every driver. I'm
> >not touching fast path at all, I just stop all processes. That way
> >suspend takes longer but system normally runs faster. 
> 
> Hrm... most of the time that test already exist. Network & block
> drivers already have blocking semantics. Serial-like drivers will
> only need to switch their interrupt source off if any or timer
> or whatever. The only case on pmac where I had to touch the normal
> path was some ioctl() path (which aren't really fast path anyway
> like preventing some ioctl's to network drivers from banging sleeping
> hardware), and the sound driver, but I needed that already for SMP
> safety.

Okay, if impact is minimal.... I can probably live with that. I still
do not know if slightly faster suspend is worth the effort.

> >Do you have example of such subsystem?
> 
> i2c, usb, ieee1394, then all the possibilities of stacking things
> on top of each other...
> 
> >Certainly no driver depends on userspace so we may stop userspace.
> 
> Well, I know at least a DSL softmodem driver that relies on userspace
> (or will soon) as the core softmodem code is moved out of the kernel ;)
> 
> >Hardware drivers do not depend on IP stack, etc...
> 
> _device_ driver include stacked devices. It includes a scsi-over-IP
> device if any on which you have perfect right to want to send a command
> before suspending.

I believe device drivers do not need to know. If you are running
scsi-over-IP you don't need to inform other side you are
suspending. It is similar to your NFS server; you don't tell your NFS
server that you are suspending. scsi-over-IP device should not need to
know.

								Pavel
-- 
Casualities in World Trade Center: ~3k dead inside the building,
cryptography in U.S.A. and free speech in Czech Republic.


-------------------------------------------------------
This sf.net email is sponsored by: Dice - The leading online job board
for high-tech professionals. Search and apply for tech jobs today!
http://seeker.dice.com/seeker.epl?rel_code=31

Re: suspend.c vs driver-model.txt

[Pavel Machek]

Hi!

> >> Benjamin Herrenschmidt was arguing for doing driver_suspend() while
> >> userspace is still running. If you assumed stopped userspace, you
> >> don't have to add semaphore into device read() etc...
> >
> >Having been around this a few times for embedded devices I've found
> >nothing that cleanly solves the problems except
> >
> >Call all the devices "suspend ready irq on" methods
> >Disable interrupts
> >Call all the devices "final suspend" method
> 
> Yes, I remember you telling me about it, which simply involves
> breaking the 3rd (ie. "suspend") PM notification into a
> "suspend_irq_on" and a "suspend_irq_off". Drivers that don't
> care will use only one of them. But that's not a real issue,
> the problems are really around the "save_state & block IOs"
> step which is step 2.
> 
>  step 1: prepare (that is allocate any memory needed, make sure
>          the driver can still operate without allocating more
>          memory, or eventually only ATOMIC or NOIO).
> 
>  step 2: save_state & block IOs, that is make sure the saved state
>          is consistent with the last IOs and that no more IO will
>          be processed. The driver don't need to wait for completion
>          of pending IOs at this stage I beleive if those don't
>          impact the saved state, but it may be better to wait for
>          them at first.

I really need all DMAs to be off after step 2; so I need all drivers
doing DMA to wait for completion. [I can not do atomic copy of memory
with DMA writing over it behind my back.]
								Pavel
-- 
Casualities in World Trade Center: ~3k dead inside the building,
cryptography in U.S.A. and free speech in Czech Republic.


-------------------------------------------------------
This sf.net email is sponsored by: Dice - The leading online job board
for high-tech professionals. Search and apply for tech jobs today!
http://seeker.dice.com/seeker.epl?rel_code=31

Re: suspend.c vs driver-model.txt

[Pavel Machek]

Hi!

> But the point here is that you don't suspend the _subsystem_ (USB,
> block layer, network stack, whatever), but the individual drivers.

You are probably right at this.

> It do make sense to provide a way to stop requests. Requests are
> not always originated from userspace. Several kernel services can
> issue requests (VM, cache, whatever stuff on top of an i2c device,
> etc...) and you can't shut down those services because you don't
> have ordering rules for them, you only have ordering rules for
> the drivers.

All kernel services that can issue requests need kernel threads,
right? My approach is to take kernel threads one by one and make them
suspend-aware.
								Pavel
-- 
Casualities in World Trade Center: ~3k dead inside the building,
cryptography in U.S.A. and free speech in Czech Republic.


-------------------------------------------------------
This sf.net email is sponsored by: Dice - The leading online job board
for high-tech professionals. Search and apply for tech jobs today!
http://seeker.dice.com/seeker.epl?rel_code=31

Re: suspend.c vs driver-model.txt

[Pavel Machek]

Hi!

> >All kernel services that can issue requests need kernel threads,
> >right? My approach is to take kernel threads one by one and make them
> >suspend-aware.
> 
> It wouldn' be bad to do that too. I just "feel" it safer and better
> to define the driver semantics to be IO blocking. Some user
>processes

I do not know if such feelings are worth the effort of driver authors.

Certainly, it feels safer that way. But I believe that what I
currently have is bulletproof...
								Pavel
-- 
Casualities in World Trade Center: ~3k dead inside the building,
cryptography in U.S.A. and free speech in Czech Republic.


-------------------------------------------------------
This sf.net email is sponsored by: Dice - The leading online job board
for high-tech professionals. Search and apply for tech jobs today!
http://seeker.dice.com/seeker.epl?rel_code=31

Re: suspend.c vs driver-model.txt

[Alan Cox]

On Tue, 2002-07-30 at 19:32, Patrick Mochel wrote:
> 
> > > >There's power cycle and whole kernel bootup. Device state is
> > > >completely lost during suspend to disk.
> > > 
> > > As it is during suspend-to-ram in most cases (most devices except
> > > a couple of them are actually powered down and the drivers have to
> > > bring them back up, at least on pmac, though on x86, the APM BIOS
> > > helps you a bit there).
> > 
> > APM BIOS is irrelevant here (when using APM BIOS, everything is done
> > by BIOS).
> 
> No, it's not. There is still a suspend sequence that must take place in 
> the kernel. 

Also almost no APM bios gets AGP or MTRR restore right.



-------------------------------------------------------
This sf.net email is sponsored by: Dice - The leading online job board
for high-tech professionals. Search and apply for tech jobs today!
http://seeker.dice.com/seeker.epl?rel_code=31

Re: suspend.c vs driver-model.txt

[Alan Cox]

On Tue, 2002-07-30 at 20:00, Pavel Machek wrote:
> Hi!
> Benjamin Herrenschmidt was arguing for doing driver_suspend() while
> userspace is still running. If you assumed stopped userspace, you
> don't have to add semaphore into device read() etc...

Having been around this a few times for embedded devices I've found
nothing that cleanly solves the problems except

Call all the devices "suspend ready irq on" methods
Disable interrupts
Call all the devices "final suspend" method



-------------------------------------------------------
This sf.net email is sponsored by: Dice - The leading online job board
for high-tech professionals. Search and apply for tech jobs today!
http://seeker.dice.com/seeker.epl?rel_code=31

Re: suspend.c vs driver-model.txt

[Patrick Mochel]

On Tue, 30 Jul 2002, Pavel Machek wrote:

> Hi!
> 
> > >All kernel services that can issue requests need kernel threads,
> > >right? My approach is to take kernel threads one by one and make them
> > >suspend-aware.
> > 
> > It wouldn' be bad to do that too. I just "feel" it safer and better
> > to define the driver semantics to be IO blocking. Some user
> >processes
> 
> I do not know if such feelings are worth the effort of driver authors.
> 
> Certainly, it feels safer that way. But I believe that what I
> currently have is bulletproof...

Heh. Famous last words... ;)

I think we have a good start, thanks entirely to the two of you. However, 
we won't be able to completely validate that we have all the theory right 
until we have all the working pieces at least partially in place. 

Unfortunately, I can't dedicate much time, as I'm swamped with other 
infrastructural items. But, I can offer some assistance.

I'm cloning a BK tree right now. It will be at 

	bk://ldm.bkbits.net/linux-2.5-suspend/

Send me patches, and I'll integrate them. We can use this as a test bed
for getting the process right before anything actually gets integrated.
People can clone from this tree for the purpose of testing.

I know there are issues with using BitKeeper. Because of that, I've 
created a directory on kernel.org to export patches of the BK tree. It's 
at:

	http://kernel.org/pub/linux/kernel/people/mochel/suspend/

I'll try and update the BK tree and the patches every kernel version. If 
you notice that I haven't, and it's been a couple of days, kick me. 

Enjoy. 

	-pat



-------------------------------------------------------
This sf.net email is sponsored by: Dice - The leading online job board
for high-tech professionals. Search and apply for tech jobs today!
http://seeker.dice.com/seeker.epl?rel_code=31

ACPI Suspend Standby Sleep

[A.MAINS]

Hello

I just informed the list on how I finally got my Battery Information
working on my Presario. (Thank you everyone) I also mentioned I was
about to take the next endeavor into Standby / Suspend. 

I loaded ACPID. I also know there is a Kernel patch for software suspend
(swusp). However, I am wondering if there are any success stories using
the ACPI Daemon / Kernel for sleep.

If so, what do I have to do to utilize this feature?

Thanks Again

Andrew



-------------------------------------------------------
This sf.net email is sponsored by: Dice - The leading online job board
for high-tech professionals. Search and apply for tech jobs today!
http://seeker.dice.com/seeker.epl?rel_code=31

Re: suspend.c vs driver-model.txt

[Benjamin Herrenschmidt]

>I do not know if such feelings are worth the effort of driver authors.

My experience with drivers here is that it isn't very difficult
to get that done right, but well...

>Certainly, it feels safer that way. But I believe that what I
>currently have is bulletproof...

If you say so... I'm still afraid with relying on the kernel
not triggering any IO for whatever random reason, I'd rather
have things designed to be bulletproof from the ground up, but
that may well be due to my lack of intimate knowledge of various
parts of the kernel not directly related to arch code & drivers.

We can end the debate here anyway, there is nothing in the
driver semantics I propose incompatible with software suspend
anyway, I'd suggest to document those semantics with blocking
of IOs and time will show how drivers keep up.

Patrick: On my side, I'll start moving the pmac drivers to
the new model as soon as I can get a recent 2.5 to boot a pmac
again, then I'll move my PM stuff to the new model and see what
issues arise. I'll try to have both suspend-to-RAM and suspend-
to-disk using Pavel's stuff working so we can validate the whole
mecanism.

No promise on time frame though, I've been pretty busy lately
and I have to get 2.4 working on some new Apple HW as well...

Ben.




-------------------------------------------------------
This sf.net email is sponsored by: Dice - The leading online job board
for high-tech professionals. Search and apply for tech jobs today!
http://seeker.dice.com/seeker.epl?rel_code=31

Re: suspend.c vs driver-model.txt

[Pavel Machek]

Hi!

> > > >All kernel services that can issue requests need kernel threads,
> > > >right? My approach is to take kernel threads one by one and make them
> > > >suspend-aware.
> > > 
> > > It wouldn' be bad to do that too. I just "feel" it safer and better
> > > to define the driver semantics to be IO blocking. Some user
> > >processes
> > 
> > I do not know if such feelings are worth the effort of driver authors.
> > 
> > Certainly, it feels safer that way. But I believe that what I
> > currently have is bulletproof...
> 
> Heh. Famous last words... ;)

Yep.

> I think we have a good start, thanks entirely to the two of you. However, 
> we won't be able to completely validate that we have all the theory right 
> until we have all the working pieces at least partially in place. 
> 
> Unfortunately, I can't dedicate much time, as I'm swamped with other 
> infrastructural items. But, I can offer some assistance.
> 
> I'm cloning a BK tree right now. It will be at 
> 
> 	bk://ldm.bkbits.net/linux-2.5-suspend/
> 
> Send me patches, and I'll integrate them. We can use this as a test bed
> for getting the process right before anything actually gets integrated.
> People can clone from this tree for the purpose of testing.

Well, it *is* integrated just now. People are actually using swsusp/S3
in 2.5.X more-or-less successfully. Linus is taking swsusp patches
pretty easily, and we get more testing by working on mainline. In
short I believe there's no reason to keep suspend patches
separate. Its experimental feature, marked as experimental, and it
seems to work for people. 
								Pavel
-- 
Casualities in World Trade Center: ~3k dead inside the building,
cryptography in U.S.A. and free speech in Czech Republic.


-------------------------------------------------------
This sf.net email is sponsored by: Dice - The leading online job board
for high-tech professionals. Search and apply for tech jobs today!
http://seeker.dice.com/seeker.epl?rel_code=31

Re: ACPI Suspend Standby Sleep

[Pavel Machek]

Hi!
> 
> I just informed the list on how I finally got my Battery Information
> working on my Presario. (Thank you everyone) I also mentioned I was
> about to take the next endeavor into Standby / Suspend. 
> 
> I loaded ACPID. I also know there is a Kernel patch for software suspend
> (swusp). However, I am wondering if there are any success stories using
> the ACPI Daemon / Kernel for sleep.
> 
> If so, what do I have to do to utilize this feature?

You get 2.5.29, then echo 4 > /proc/acpi/sleep. Read Docs/swsusp.txt.
								Pavel
-- 
I'm pavel-7aPAsKgELzg@public.gmane.org "In my country we have almost anarchy and I don't care."
Panos Katsaloulis describing me w.r.t. patents at discuss-q33YXrgyAoMgsBAKwltoeQ@public.gmane.org


-------------------------------------------------------
This sf.net email is sponsored by:ThinkGeek
Welcome to geek heaven.
http://thinkgeek.com/sf