Re: suspend.c vs driver-model.txt

Re: suspend.c vs driver-model.txt

[Benjamin Herrenschmidt]

>What races can you see?

Well, existing PM callbacks aren't good enough :) they just don't
deal with dependencies properly. Also, they don't deal that well
with the need for drivers to trigger memory allocations and all
of the issues related to that we discussed during the BOF and
earlier.
driverfs bus-oriented ones would be better here in many regard,
especially since I beleive we found "correct" semantics for the
various steps of device save state & suspend.

>> The problem of saving to disk and of saving to memory (that is
>> machine sleep as I implement it on powerbooks today, RAM content
>> beeing preserved) is pretty similar.
>
>Actually it is quite different.
>
>Saving device state is common code, but suspend-to-ram can be done
>without scheduling, while you need to block for suspend-to-disk.

No. They end up beeing very similar. Suspend to RAM has to schedule
because some underlying device drivers will need to schedule to
properly block their queues as well.

I figured out in the Pmac implementation that I could actually let
the system schedule the whole time up to just before the very last
step of shutting down the CPU. Userland apps will simply block as
they rely on IOs for drivers that have been properly blocked, or
from swap while the swap device may be suspended, etc... CPU
intensive app would still work until it's very last timeslice is
used before suspend.

That's also how I got very fast wakeup times. Basically, processes
start again right away (well, just after a few really important things
like time are restored), and then drivers are kicked back into life,
asynchronously if possible, thus user processes that are blocked by
a given driver will come back to life normally.

>> So you really need to properly do the prepare/save/suspend steps
>> on all devices in proper bus ordering so that any device driver
>> has properly saved state information to memory (which may later
>> be saved to disk with suspend-to-disk) and has properly blocked
>> IO queues.
>> 
>> The specific case of the device which is used as a backstore
>> for the RAM save has to be dealt some specific way. 
>
>No it does not. I have half of RAM free, I just save-state, copy
>memory, continue devices, copy saved memory to swap.

So you resume devices from the "saved state" which isn't the
state the device was when the machine was really suspended,
right ? Which means that typically, on resume, the driver could
end up beeing out of sync with the device if some permanent state
information exist on the device, but I agree this is a rare case,
except for... storage. So I assume you have ways to prevent
filesystems to be touched at all ?

As I see it, for your scheme to work properly, you need to,
somewhat "atomically", save-state all devices so they are
in coherent state one to each other (devices can well be
inter-dependant), backup your RAM, then you can kick back
devices (well, some actually) into life.

This is really only a special case of the generic process I'm
suggesting then ;)

Basically, you still need to run the "block IOs then save state
and suspend" step on all devices in bus ordering. So that
part is common with suspend-to-ram. Actually, you don't need
to prevent scheduling before that point, except maybe for
keeping your "half of RAM free" watermark, but even then, I
don't see how you acheive that since the kernel itself may
allocate memory (see note below)

Then you need to use the explicit device model power
state functions to re-enable power state on the target device
of the backup. It should in turn re-enable parent devices up
to the host bus.

However, that would be inefficient as the net effect would be
to have your hard disk spin down, be eventually powered off,
then back up for suspend to RAM, then the machine powered off
(and so that hard disk as well).

Which is why I beleive it would make more sense to specifically
instruct the target device (and so it's parent) during the
device suspend loop to _not_ go to sleep, just suspend, block
IOs, then resume IOs, but _not_ do actual suspend.

If we stick to the 3 step model we discussed at OLS

 1) prepare for sleep (memory allocation, etc...), stop
    doing _any_ non-ATOMIC (or non-NOIO) memory allocation
    beyond the point, the driver has to pre-allocate what
    it will need from now on, eventually running with degraded
    perfs (serialized)
 2) block IOs & save state. Block drivers should stop their
    request queue, drivers impl. a direct /dev interface should
    block processes calling them until they are resumed, etc...
    typically done easily with a semaphore for most of them.
    then save state informations to pre-allocated memory
 3) suspend (IRQs off) Or optionally 4 steps with 3) suspend_irq_on
    and 4) suspend_irq_off.

Then suspend-to-disk would need to call steps 1 and 2 normally,
but not 3 for devices on the storage chain. Then, after the
RAM is copied, that device can be sent a resume request, though
in this case, you indeed need to make sure no user process or
journaling daemon or whatever will inject requests to your
target device queues that are not specifically your pages
beeing thrown to the backing store.
>
>> I understand (please correct me if I'm wrong) that your mecanism is to
>> implement that at a higher level, though I still fail to see the
>> "big picture" of it, especially how you can properly resume state
>> of all device drivers and other in-kernel state informations. Do you
>> store all pages including kernel pages to disk ? 
>
>Yes I store all pages including kernel ones to disk.
>
>> If not, what about
>> open inodes, sockets, etc... ? How do you resume kernel state information
>> for these ? 
>> How do you deal with device-drivers that are configured in
>> some specific way before suspend and has to come back up the same
>> way ?
>
>I need device support for suspend-to-disk, of course. But I need no
>special support on "suspend" device.

I still think it need to be handled slightly differently (see above),
I'm trying to see what has to be common and what not. Defining (and
then implementing) the proper device support is the biggest issue,
I think we have the semantics approximately right in mind Patrick
and I, it's time to write them down :)

(*note about device mem alloc): Some devices need to allocate memory
to be able to save sate. That can be a significant amount of memory
(some framebuffer may want to backup the fb content, huge !). 

So in your case, I beleive you should probably first send the notification
of step 1 (prepare for sleep) to drivers, then do your memory-crunching
thing, then call step 2 and step 3 for all but swap device.

Ben.




-------------------------------------------------------
This sf.net email is sponsored by: Dice - The leading online job board
for high-tech professionals. Search and apply for tech jobs today!
http://seeker.dice.com/seeker.epl?rel_code=31

Re: suspend.c vs driver-model.txt

[Pavel Machek]

Hi!

> >What races can you see?
> 
> Well, existing PM callbacks aren't good enough :) they just don't
> deal with dependencies properly. Also, they don't deal that well

Yep, I know. I'm using Patrick's devicefs callback.

> >> The problem of saving to disk and of saving to memory (that is
> >> machine sleep as I implement it on powerbooks today, RAM content
> >> beeing preserved) is pretty similar.
> >
> >Actually it is quite different.
> >
> >Saving device state is common code, but suspend-to-ram can be done
> >without scheduling, while you need to block for suspend-to-disk.
> 
> No. They end up beeing very similar. Suspend to RAM has to schedule
> because some underlying device drivers will need to schedule to
> properly block their queues as well.

Okay, true. Still they are different because suspend-to-disk needs
working disk driver to save pages.

> I figured out in the Pmac implementation that I could actually let
> the system schedule the whole time up to just before the very last
> step of shutting down the CPU. Userland apps will simply block as
> they rely on IOs for drivers that have been properly blocked, or
> from swap while the swap device may be suspended, etc... CPU
> intensive app would still work until it's very last timeslice is
> used before suspend.

Being extremely cpu-efficient is not 1-st priority goal of
swsusp. First make it work, then make it faster ;-).

> That's also how I got very fast wakeup times. Basically, processes
> start again right away (well, just after a few really important things
> like time are restored), and then drivers are kicked back into life,
> asynchronously if possible, thus user processes that are blocked by
> a given driver will come back to life normally.

I believe you give way too much responsibility to drivers...

> >> So you really need to properly do the prepare/save/suspend steps
> >> on all devices in proper bus ordering so that any device driver
> >> has properly saved state information to memory (which may later
> >> be saved to disk with suspend-to-disk) and has properly blocked
> >> IO queues.
> >> 
> >> The specific case of the device which is used as a backstore
> >> for the RAM save has to be dealt some specific way. 
> >
> >No it does not. I have half of RAM free, I just save-state, copy
> >memory, continue devices, copy saved memory to swap.
> 
> So you resume devices from the "saved state" which isn't the
> state the device was when the machine was really suspended,
> right ? Which means that typically, on resume, the driver could
> end up beeing out of sync with the device if some permanent state
> information exist on the device, but I agree this is a rare case,
> except for... storage. 

There's power cycle and whole kernel bootup. Device state is
completely lost during suspend to disk.

> So I assume you have ways to prevent
> filesystems to be touched at all ?

There's nothing alive that could touch filesystems. If user boots into
non-suspend-aware kernel and writes to disk disk, his fault.

>From Docs/swsusp.txt:

 * BIG FAT WARNING
*********************************************************
 *
 * If you have unsupported (*) devices using DMA...
 *                              ...say goodbye to your data.
 *
 * If you touch anything on disk between suspend and resume...
 *                              ...kiss your data goodbye.
 *
 * If your disk driver does not support suspend... (IDE does)
 *                              ...you'd better find out how to get along
 *                                 without your data.
 *
 * (*) pm interface support is needed to make it safe.

You need to append resume=/dev/your_swap_partition to kernel command
line. Then you suspend by echo 4 > /proc/acpi/sleep.

> As I see it, for your scheme to work properly, you need to,
> somewhat "atomically", save-state all devices so they are
> in coherent state one to each other (devices can well be
> inter-dependant), backup your RAM, then you can kick back
> devices (well, some actually) into life.

Yep, that's what I'm doing.

> This is really only a special case of the generic process I'm
> suggesting then ;)
> 
> Basically, you still need to run the "block IOs then save state
> and suspend" step on all devices in bus ordering. 

I don't need "block IO" step. I just stop everything that could
possibly ask drivers to do IO.

> So that
> part is common with suspend-to-ram. 

Yep, drivers part is identical with suspend-to-ram. 

> Actually, you don't need
> to prevent scheduling before that point, except maybe for
> keeping your "half of RAM free" watermark, but even then, I
> don't see how you acheive that since the kernel itself may
> allocate memory (see note below)

It is not 100% guaranteed that it is possible to get half of RAM
free. In such case suspend-to-disk fails.

> Then you need to use the explicit device model power
> state functions to re-enable power state on the target device
> of the backup. It should in turn re-enable parent devices up
> to the host bus.

No, I just re-enable everything.

> However, that would be inefficient as the net effect would be
> to have your hard disk spin down, be eventually powered off,
> then back up for suspend to RAM, then the machine powered off
			      ~~~
				\____ I believe you mean disk here.

> (and so that hard disk as well).

Yes, it is not effective (and disk will do ugly
spindown-spinup-powerdown cycle). It is still faster than bios
suspend-to-disk ;-).

> Which is why I beleive it would make more sense to specifically
> instruct the target device (and so it's parent) during the
> device suspend loop to _not_ go to sleep, just suspend, block
> IOs, then resume IOs, but _not_ do actual suspend.

In such case you can as well do suspend-but-don't-sleep for *all*
devices. They are going to be powered down, anyway, so who cares.

> If we stick to the 3 step model we discussed at OLS
> 
>  1) prepare for sleep (memory allocation, etc...), stop
>     doing _any_ non-ATOMIC (or non-NOIO) memory allocation
>     beyond the point, the driver has to pre-allocate what
>     it will need from now on, eventually running with degraded
>     perfs (serialized)
>  2) block IOs & save state. Block drivers should stop their
>     request queue, drivers impl. a direct /dev interface should
>     block processes calling them until they are resumed, etc...
>     typically done easily with a semaphore for most of them.
>     then save state informations to pre-allocated memory

I believe you are putting *way* too much responsibility to the
drivers. With your model each driver needs to be able to stop its
users. Ouch. Remember -- there's lot of drivers. You do not want to
add crap^Wcode to them. Better just stop all user programs so that
drivers don't have to care.

>  3) suspend (IRQs off) Or optionally 4 steps with 3) suspend_irq_on
>     and 4) suspend_irq_off.
> 
> Then suspend-to-disk would need to call steps 1 and 2 normally,
> but not 3 for devices on the storage chain. Then, after the

It is hard to tell which devices are on the storage chain. And it
should *not* be neccessary to treat them differently.

> >> If not, what about
> >> open inodes, sockets, etc... ? How do you resume kernel state information
> >> for these ? 
> >> How do you deal with device-drivers that are configured in
> >> some specific way before suspend and has to come back up the same
> >> way ?
> >
> >I need device support for suspend-to-disk, of course. But I need no
> >special support on "suspend" device.
> 
> I still think it need to be handled slightly differently (see above),
> I'm trying to see what has to be common and what not. Defining (and
> then implementing) the proper device support is the biggest issue,
> I think we have the semantics approximately right in mind Patrick
> and I, it's time to write them down :)

Hehe, I believe I have semantics right, too, and have written it down
in kernel/suspend.c ;-)))).

> (*note about device mem alloc): Some devices need to allocate memory
> to be able to save sate. That can be a significant amount of memory
> (some framebuffer may want to backup the fb content, huge !). 

After free_some_memory(), there's very likely *plenty* of memory
available. If framebuffer wants to backup the fb content, and it runs
out of memory, tough, and suspend fails.

[BTW you don't need/want to backup fb content; either its X or its
text console. Text console knows how to repaint itself. X knows how to
repaint itself.]

> So in your case, I beleive you should probably first send the notification
> of step 1 (prepare for sleep) to drivers, then do your memory-crunching
> thing, then call step 2 and step 3 for all but swap device.

If I do memory-freeing, step 1, step 2, step 3, memory-copy it should
be equivalent, AFAICS. That's what I'm doing (but for all devices).

								Pavel
-- 
Worst form of spam? Adding advertisment signatures ala sourceforge.net.
What goes next? Inserting advertisment *into* email?


-------------------------------------------------------
This sf.net email is sponsored by: Dice - The leading online job board
for high-tech professionals. Search and apply for tech jobs today!
http://seeker.dice.com/seeker.epl?rel_code=31

Re: suspend.c vs driver-model.txt

[Benjamin Herrenschmidt]

>I believe you are putting *way* too much responsibility to the
>drivers. With your model each driver needs to be able to stop its
>users. Ouch. Remember -- there's lot of drivers. You do not want to
>add crap^Wcode to them. Better just stop all user programs so that
>drivers don't have to care.

Well, you do have to deal with drivers. Users programs is not enough,
drivers can be kicked by things going on in the kernel as well.
 
Also, in most case, it's not that difficult to implement.

 - Network drivers just need to drop packets, pretty easily done
 - Block drivers just need to stop their queue
 - Things like serial drivers usually don't have to do anything
   terrific, just stop the chip interrupt and users will be blocked
   (or will timeout, but that is not a problem at this point)
 - For pmac sound driver, I choose to have a mutex that cause users
   to be blocked, but that was the "easy" way to quickly get that
   done, a more subtle mecanism could be done, dropping samples at
   approximately the output rate. The mutex makes it easy to make
   the whole driver SMP safe as well regarding mixer tweaks though.

I implemented this in a whole bunch of drivers used on Apple laptops
and in the end, the most code is _not_ doing the above properly in
drivers, but doing the actual state save/suspend & resume on the
chip (here you have to deal with broken HW, subtle timing issues, etc...)

So drivers _has_ to be done the right way. If a driver doesn't, then
it will be unsuitable for PM (be it suspend to RAM or to disk). No
problem with that ;)

>>  3) suspend (IRQs off) Or optionally 4 steps with 3) suspend_irq_on
>>     and 4) suspend_irq_off.
>> 
>> Then suspend-to-disk would need to call steps 1 and 2 normally,
>> but not 3 for devices on the storage chain. Then, after the
>
>It is hard to tell which devices are on the storage chain. And it
>should *not* be neccessary to treat them differently.

Well... I would prefer then broadcasting a "save_state/block" to
all of them, then no suspend, and a wakeup. That keeps the existing
semantics valid.

>Hehe, I believe I have semantics right, too, and have written it down
>in kernel/suspend.c ;-)))).

Hrm... not sure about those ;) Anyway, Patrick will probably write down
something precising what we came up with during OLS.

>> (*note about device mem alloc): Some devices need to allocate memory
>> to be able to save sate. That can be a significant amount of memory
>> (some framebuffer may want to backup the fb content, huge !). 
>
>After free_some_memory(), there's very likely *plenty* of memory
>available. If framebuffer wants to backup the fb content, and it runs
>out of memory, tough, and suspend fails.
>
>[BTW you don't need/want to backup fb content; either its X or its
>text console. Text console knows how to repaint itself. X knows how to
>repaint itself.]

Sure, though some fb apps don't. Though here, I agree we need those to
get notified as part of the process (which I do on pmac before doing the
device notification, as part of my /dev/apm_bios emulation) so they can
actually do that repainting properly.

>> So in your case, I beleive you should probably first send the notification
>> of step 1 (prepare for sleep) to drivers, then do your memory-crunching
>> thing, then call step 2 and step 3 for all but swap device.
>
>If I do memory-freeing, step 1, step 2, step 3, memory-copy it should
>be equivalent, AFAICS. That's what I'm doing (but for all devices).

Not sure. step 1 will affect memory allocation, swap etc... I'd rather
have that done before you start doing the memory freeing. Well, that's
mostly a matter of taste though.

Ben.




-------------------------------------------------------
This sf.net email is sponsored by: Dice - The leading online job board
for high-tech professionals. Search and apply for tech jobs today!
http://seeker.dice.com/seeker.epl?rel_code=31

Re: suspend.c vs driver-model.txt

[Pavel Machek]

Hi!

> >I believe you are putting *way* too much responsibility to the
> >drivers. With your model each driver needs to be able to stop its
> >users. Ouch. Remember -- there's lot of drivers. You do not want to
> >add crap^Wcode to them. Better just stop all user programs so that
> >drivers don't have to care.
> 
> Well, you do have to deal with drivers. Users programs is not enough,
> drivers can be kicked by things going on in the kernel as well.

Well, so that "thing" that could kick it has to be found and made
suspend-aware. Better than touching 1000 drivers.

> Also, in most case, it's not that difficult to implement.

Wrong level of abstraction.

>  - Network drivers just need to drop packets, pretty easily done

Why? You should modify network stack not to post any packets to them.

>  - Block drivers just need to stop their queue

Block layer should not attempt to pass anything to driver when
suspended. (Theres one block layer, but 100 block drivers; which one
would you like to modify?)

[Currently I'm stopping it at even higher level by stopping
processes. I still think stopping everything is easiest solution, and
probably only one doable for 2.6.]

>  - Things like serial drivers usually don't have to do anything
>    terrific, just stop the chip interrupt and users will be blocked
>    (or will timeout, but that is not a problem at this point)
>  - For pmac sound driver, I choose to have a mutex that cause users
>    to be blocked, but that was the "easy" way to quickly get that
>    done, a more subtle mecanism could be done, dropping samples at
>    approximately the output rate. The mutex makes it easy to make
>    the whole driver SMP safe as well regarding mixer tweaks though.

Why don't you just put that mutex into sys_read etc? Or why not system
stopping? There's just too much drivers.

> >>  3) suspend (IRQs off) Or optionally 4 steps with 3) suspend_irq_on
> >>     and 4) suspend_irq_off.
> >> 
> >> Then suspend-to-disk would need to call steps 1 and 2 normally,
> >> but not 3 for devices on the storage chain. Then, after the
> >
> >It is hard to tell which devices are on the storage chain. And it
> >should *not* be neccessary to treat them differently.
> 
> Well... I would prefer then broadcasting a "save_state/block" to
> all of them, then no suspend, and a wakeup. That keeps the existing
> semantics valid.

That's probably doable.

> >> (*note about device mem alloc): Some devices need to allocate memory
> >> to be able to save sate. That can be a significant amount of memory
> >> (some framebuffer may want to backup the fb content, huge !). 
> >
> >After free_some_memory(), there's very likely *plenty* of memory
> >available. If framebuffer wants to backup the fb content, and it runs
> >out of memory, tough, and suspend fails.
> >
> >[BTW you don't need/want to backup fb content; either its X or its
> >text console. Text console knows how to repaint itself. X knows how to
> >repaint itself.]
> 
> Sure, though some fb apps don't. Though here, I agree we need those to
> get notified as part of the process (which I do on pmac before doing the
> device notification, as part of my /dev/apm_bios emulation) so they can
> actually do that repainting properly.

Kernel tells them that console is being switched away; so it is their
responsibility. Surprise, special console for suspend is not only
pretty but also usefull ;-).
								Pavel
-- 
Casualities in World Trade Center: ~3k dead inside the building,
cryptography in U.S.A. and free speech in Czech Republic.


-------------------------------------------------------
This sf.net email is sponsored by: Dice - The leading online job board
for high-tech professionals. Search and apply for tech jobs today!
http://seeker.dice.com/seeker.epl?rel_code=31

Re: suspend.c vs driver-model.txt

[Benjamin Herrenschmidt]

>Why? You should modify network stack not to post any packets to them.

No, you should have the _driver_ ask the network stack not to get
packets on _this_ specific interface.

>>  - Block drivers just need to stop their queue
>
>Block layer should not attempt to pass anything to driver when
>suspended. (Theres one block layer, but 100 block drivers; which one
>would you like to modify?)

Again, it's a specific driver that has to inform the block layer.

Suspend hierarchy isn't per-subsystem. You may well need a given
subsystem to be alive for a separate driver to suspend itself
properly. It has to be bus-driven.

However, the bus-binding of a driver is completely orthogonal
to the subsystems to which that driver is attached (and gets
requests from).

>[Currently I'm stopping it at even higher level by stopping
>processes. I still think stopping everything is easiest solution, and
>probably only one doable for 2.6.]

Well, it could be done stopping process first, until all drivers are
safe enouh. That mean keeping the IO blocking semantics for drivers
while having the process stopped to help deal with drivers not
implementing that at first, but I'd prefer doing it the right way
first.

>>  - Things like serial drivers usually don't have to do anything
>>    terrific, just stop the chip interrupt and users will be blocked
>>    (or will timeout, but that is not a problem at this point)
>>  - For pmac sound driver, I choose to have a mutex that cause users
>>    to be blocked, but that was the "easy" way to quickly get that
>>    done, a more subtle mecanism could be done, dropping samples at
>>    approximately the output rate. The mutex makes it easy to make
>>    the whole driver SMP safe as well regarding mixer tweaks though.
>
>Why don't you just put that mutex into sys_read etc? Or why not system
>stopping? There's just too much drivers.

sys_read, sys_ioctl, etc... ? well, drivers can be kicked by other
means than just syscalls. And if you tear down subsystems, you take
the risk of stopping a subsystem before a driver that need it for
proper suspend had a chance to use it.

>Kernel tells them that console is being switched away; so it is their
>responsibility. Surprise, special console for suspend is not only
>pretty but also usefull ;-).

Yes, in this specific case, I agree, it's much better than make faked
fbcon ops I use in fbdev drivers ;)

Ben.




-------------------------------------------------------
This sf.net email is sponsored by: Dice - The leading online job board
for high-tech professionals. Search and apply for tech jobs today!
http://seeker.dice.com/seeker.epl?rel_code=31

Re: suspend.c vs driver-model.txt

[Pavel Machek]

Hi!


> >[Currently I'm stopping it at even higher level by stopping
> >processes. I still think stopping everything is easiest solution, and
> >probably only one doable for 2.6.]
> 
> Well, it could be done stopping process first, until all drivers are
> safe enouh. That mean keeping the IO blocking semantics for drivers
> while having the process stopped to help deal with drivers not
> implementing that at first, but I'd prefer doing it the right way
> first.

I'd not call it right way. 

What you propose is adding test into *fast* path of every driver. I'm
not touching fast path at all, I just stop all processes. That way
suspend takes longer but system normally runs faster.

> >>  - Things like serial drivers usually don't have to do anything
> >>    terrific, just stop the chip interrupt and users will be blocked
> >>    (or will timeout, but that is not a problem at this point)
> >>  - For pmac sound driver, I choose to have a mutex that cause users
> >>    to be blocked, but that was the "easy" way to quickly get that
> >>    done, a more subtle mecanism could be done, dropping samples at
> >>    approximately the output rate. The mutex makes it easy to make
> >>    the whole driver SMP safe as well regarding mixer tweaks though.
> >
> >Why don't you just put that mutex into sys_read etc? Or why not system
> >stopping? There's just too much drivers.
> 
> sys_read, sys_ioctl, etc... ? well, drivers can be kicked by other
> means than just syscalls. And if you tear down subsystems, you take
> the risk of stopping a subsystem before a driver that need it for
> proper suspend had a chance to use it.

Do you have example of such subsystem?

Certainly no driver depends on userspace so we may stop userspace.

Hardware drivers do not depend on IP stack, etc...
								Pavel
-- 
Casualities in World Trade Center: ~3k dead inside the building,
cryptography in U.S.A. and free speech in Czech Republic.


-------------------------------------------------------
This sf.net email is sponsored by: Dice - The leading online job board
for high-tech professionals. Search and apply for tech jobs today!
http://seeker.dice.com/seeker.epl?rel_code=31

Re: suspend.c vs driver-model.txt

[Benjamin Herrenschmidt]

>I'd not call it right way. 
>
>What you propose is adding test into *fast* path of every driver. I'm
>not touching fast path at all, I just stop all processes. That way
>suspend takes longer but system normally runs faster. 

Hrm... most of the time that test already exist. Network & block
drivers already have blocking semantics. Serial-like drivers will
only need to switch their interrupt source off if any or timer
or whatever. The only case on pmac where I had to touch the normal
path was some ioctl() path (which aren't really fast path anyway
like preventing some ioctl's to network drivers from banging sleeping
hardware), and the sound driver, but I needed that already for SMP
safety.

>> >>  - Things like serial drivers usually don't have to do anything
>> >>    terrific, just stop the chip interrupt and users will be blocked
>> >>    (or will timeout, but that is not a problem at this point)
>> >>  - For pmac sound driver, I choose to have a mutex that cause users
>> >>    to be blocked, but that was the "easy" way to quickly get that
>> >>    done, a more subtle mecanism could be done, dropping samples at
>> >>    approximately the output rate. The mutex makes it easy to make
>> >>    the whole driver SMP safe as well regarding mixer tweaks though.
>> >
>> >Why don't you just put that mutex into sys_read etc? Or why not system
>> >stopping? There's just too much drivers.
>> 
>> sys_read, sys_ioctl, etc... ? well, drivers can be kicked by other
>> means than just syscalls. And if you tear down subsystems, you take
>> the risk of stopping a subsystem before a driver that need it for
>> proper suspend had a chance to use it.
>
>Do you have example of such subsystem?

i2c, usb, ieee1394, then all the possibilities of stacking things
on top of each other...

>Certainly no driver depends on userspace so we may stop userspace.

Well, I know at least a DSL softmodem driver that relies on userspace
(or will soon) as the core softmodem code is moved out of the kernel ;)

>Hardware drivers do not depend on IP stack, etc...

_device_ driver include stacked devices. It includes a scsi-over-IP
device if any on which you have perfect right to want to send a command
before suspending.

Ben.





-------------------------------------------------------
This sf.net email is sponsored by: Dice - The leading online job board
for high-tech professionals. Search and apply for tech jobs today!
http://seeker.dice.com/seeker.epl?rel_code=31

Re: suspend.c vs driver-model.txt

[Pavel Machek]

Hi!

> >I'd not call it right way. 
> >
> >What you propose is adding test into *fast* path of every driver. I'm
> >not touching fast path at all, I just stop all processes. That way
> >suspend takes longer but system normally runs faster. 
> 
> Hrm... most of the time that test already exist. Network & block
> drivers already have blocking semantics. Serial-like drivers will
> only need to switch their interrupt source off if any or timer
> or whatever. The only case on pmac where I had to touch the normal
> path was some ioctl() path (which aren't really fast path anyway
> like preventing some ioctl's to network drivers from banging sleeping
> hardware), and the sound driver, but I needed that already for SMP
> safety.

Okay, if impact is minimal.... I can probably live with that. I still
do not know if slightly faster suspend is worth the effort.

> >Do you have example of such subsystem?
> 
> i2c, usb, ieee1394, then all the possibilities of stacking things
> on top of each other...
> 
> >Certainly no driver depends on userspace so we may stop userspace.
> 
> Well, I know at least a DSL softmodem driver that relies on userspace
> (or will soon) as the core softmodem code is moved out of the kernel ;)
> 
> >Hardware drivers do not depend on IP stack, etc...
> 
> _device_ driver include stacked devices. It includes a scsi-over-IP
> device if any on which you have perfect right to want to send a command
> before suspending.

I believe device drivers do not need to know. If you are running
scsi-over-IP you don't need to inform other side you are
suspending. It is similar to your NFS server; you don't tell your NFS
server that you are suspending. scsi-over-IP device should not need to
know.

								Pavel
-- 
Casualities in World Trade Center: ~3k dead inside the building,
cryptography in U.S.A. and free speech in Czech Republic.


-------------------------------------------------------
This sf.net email is sponsored by: Dice - The leading online job board
for high-tech professionals. Search and apply for tech jobs today!
http://seeker.dice.com/seeker.epl?rel_code=31

Re: suspend.c vs driver-model.txt

[Benjamin Herrenschmidt]

>Okay, if impact is minimal.... I can probably live with that. I still
>do not know if slightly faster suspend is worth the effort.

it's not only slightly faster, it "feels" safer as well not to
rely on userland & selected kernel threads to be the only possible
source of incoming IOs.

>> >Do you have example of such subsystem?
>> 
>> i2c, usb, ieee1394, then all the possibilities of stacking things
>> on top of each other...
>> 
>> >Certainly no driver depends on userspace so we may stop userspace.
>> 
>> Well, I know at least a DSL softmodem driver that relies on userspace
>> (or will soon) as the core softmodem code is moved out of the kernel ;)
>> 
>> >Hardware drivers do not depend on IP stack, etc...
>> 
>> _device_ driver include stacked devices. It includes a scsi-over-IP
>> device if any on which you have perfect right to want to send a command
>> before suspending.
>
>I believe device drivers do not need to know. If you are running
>scsi-over-IP you don't need to inform other side you are
>suspending. It is similar to your NFS server; you don't tell your NFS
>server that you are suspending. scsi-over-IP device should not need to
>know.

Well, SCSI-over-IP is probably not a good example, but you can have
very valid reasons to inform a device over whatever protocol chain
that you are suspending. 

Ben.





-------------------------------------------------------
This sf.net email is sponsored by: Dice - The leading online job board
for high-tech professionals. Search and apply for tech jobs today!
http://seeker.dice.com/seeker.epl?rel_code=31

Re: suspend.c vs driver-model.txt

[Patrick Mochel]

> > Well, you do have to deal with drivers. Users programs is not enough,
> > drivers can be kicked by things going on in the kernel as well.
> 
> Well, so that "thing" that could kick it has to be found and made
> suspend-aware. Better than touching 1000 drivers.

Ok, so the real prize is in finding exactly what the right level of 
abstraction is. I will give you that one. But in doing so, it is not a 
decision of "Touching Every Driver" vs. "Not Touching Any Drivers".

Drivers must be modified. 

Notifying subsystems of a suspend/resume transition is a good thing to do. 
We touched on this briefly in Ottawa, though nothing was written down, as 
we were more concerned with dealing with proper driver ordering. 

Even if we stop I/O requests to the devices, we still have to save and 
restore their state, as well as put the devices in their own low power 
state. 

There are about 6 pages in the ACPI 2.0 spec that I actually trust and am
willing to repeat. Section 9.1.6: "Transitioning from the Working to the 
Sleep State" (pg. 231), it says:

...
3) OSPM places all devices into their respective Dx state.
...

If you do that, you must save and restore all device context that will be 
lost during the transition. 

You don't want to re-init the device because you can assume information 
based on what you saved away. It wastes time, and it's unnecessary. 

Suspend states are simply an illusion to the user. They want the system to
come back to _exactly_ where they left off, as fast as possible.  
Powerbooks do this incredibly well. Windows even does a decent job. 

	-pat



-------------------------------------------------------
This sf.net email is sponsored by: Dice - The leading online job board
for high-tech professionals. Search and apply for tech jobs today!
http://seeker.dice.com/seeker.epl?rel_code=31

Re: suspend.c vs driver-model.txt

[Pavel Machek]

Hi!

> > > Well, you do have to deal with drivers. Users programs is not enough,
> > > drivers can be kicked by things going on in the kernel as well.
> > 
> > Well, so that "thing" that could kick it has to be found and made
> > suspend-aware. Better than touching 1000 drivers.
> 
> Ok, so the real prize is in finding exactly what the right level of 
> abstraction is. I will give you that one. But in doing so, it is not a 
> decision of "Touching Every Driver" vs. "Not Touching Any Drivers".

Agreed.

> Drivers must be modified. 

Yep.

> Even if we stop I/O requests to the devices, we still have to save and 
> restore their state, as well as put the devices in their own low power 
> state. 

We have to save/restore their state.

We do not need to put them in low power states for swsusp (we are
powered), but we should do that for S3.

> You don't want to re-init the device because you can assume information 
> based on what you saved away. It wastes time, and it's unnecessary. 

More time consuming etc but simpler in some cases ;-).

[ne2000 driver is so used to working with crap that you can live
without suspend/resume support. You go S3. You return. ne2000 driver
tries to communicate with the card, and it appears dead, so it kicks
it back alive after timeout. Hack hack ;-)].

> Suspend states are simply an illusion to the user. They want the system to
> come back to _exactly_ where they left off, as fast as possible.  
> Powerbooks do this incredibly well. Windows even does a decent job. 

Yes but it is not actually important if we can enter S3 in 100 msec or
120 msec, right? Same goes for resume. We want to avoid touching hot
paths.
								Pavel
-- 
Casualities in World Trade Center: ~3k dead inside the building,
cryptography in U.S.A. and free speech in Czech Republic.


-------------------------------------------------------
This sf.net email is sponsored by: Dice - The leading online job board
for high-tech professionals. Search and apply for tech jobs today!
http://seeker.dice.com/seeker.epl?rel_code=31

ACPI Suspend Standby Sleep

[A.MAINS]

Hello

I just informed the list on how I finally got my Battery Information
working on my Presario. (Thank you everyone) I also mentioned I was
about to take the next endeavor into Standby / Suspend. 

I loaded ACPID. I also know there is a Kernel patch for software suspend
(swusp). However, I am wondering if there are any success stories using
the ACPI Daemon / Kernel for sleep.

If so, what do I have to do to utilize this feature?

Thanks Again

Andrew



-------------------------------------------------------
This sf.net email is sponsored by: Dice - The leading online job board
for high-tech professionals. Search and apply for tech jobs today!
http://seeker.dice.com/seeker.epl?rel_code=31

Re: ACPI Suspend Standby Sleep

[Pavel Machek]

Hi!
> 
> I just informed the list on how I finally got my Battery Information
> working on my Presario. (Thank you everyone) I also mentioned I was
> about to take the next endeavor into Standby / Suspend. 
> 
> I loaded ACPID. I also know there is a Kernel patch for software suspend
> (swusp). However, I am wondering if there are any success stories using
> the ACPI Daemon / Kernel for sleep.
> 
> If so, what do I have to do to utilize this feature?

You get 2.5.29, then echo 4 > /proc/acpi/sleep. Read Docs/swsusp.txt.
								Pavel
-- 
I'm pavel-7aPAsKgELzg@public.gmane.org "In my country we have almost anarchy and I don't care."
Panos Katsaloulis describing me w.r.t. patents at discuss-q33YXrgyAoMgsBAKwltoeQ@public.gmane.org


-------------------------------------------------------
This sf.net email is sponsored by:ThinkGeek
Welcome to geek heaven.
http://thinkgeek.com/sf

Re: suspend.c vs driver-model.txt

[Benjamin Herrenschmidt]

>
>There's power cycle and whole kernel bootup. Device state is
>completely lost during suspend to disk.

As it is during suspend-to-ram in most cases (most devices except
a couple of them are actually powered down and the drivers have to
bring them back up, at least on pmac, though on x86, the APM BIOS
helps you a bit there).

So basically, drivers are responsible to bring back the device
in whatever state it was at the save_state step of the suspend,
in the same way for but suspend to disk and to RAM. What I want to
make sure is that the driver-side semantics remain the same for
swsusp and suspend-to-RAM.
That includes the fact that save state will actually block IOs, and
that is necessary to have a coherent state among devices.

Note that you don't need to run the last step of driver notification
though (the actual "suspend" state), this will save you the disk
powercycle effect on suspend.

Ben.





-------------------------------------------------------
This sf.net email is sponsored by: Dice - The leading online job board
for high-tech professionals. Search and apply for tech jobs today!
http://seeker.dice.com/seeker.epl?rel_code=31

Re: suspend.c vs driver-model.txt

[Pavel Machek]

Hi!

> >There's power cycle and whole kernel bootup. Device state is
> >completely lost during suspend to disk.
> 
> As it is during suspend-to-ram in most cases (most devices except
> a couple of them are actually powered down and the drivers have to
> bring them back up, at least on pmac, though on x86, the APM BIOS
> helps you a bit there).

APM BIOS is irrelevant here (when using APM BIOS, everything is done
by BIOS).

> So basically, drivers are responsible to bring back the device
> in whatever state it was at the save_state step of the suspend,
> in the same way for but suspend to disk and to RAM. What I want to
> make sure is that the driver-side semantics remain the same for
> swsusp and suspend-to-RAM.
> That includes the fact that save state will actually block IOs, and
> that is necessary to have a coherent state among devices.

I actually made suspend-to-RAM working as a side effect of making
suspend-to-disk working.

What I'm doing is "drivers don't have to care about stopping requests
because everything is stopped". If you are willing to go through all
drivers and add stopping in them, thats fine with me, but I believe
its post-2.6.

> Note that you don't need to run the last step of driver notification
> though (the actual "suspend" state), this will save you the disk
> powercycle effect on suspend.

Yep, I can get rid of spindown/spinup..
								Pavel
-- 
Casualities in World Trade Center: ~3k dead inside the building,
cryptography in U.S.A. and free speech in Czech Republic.


-------------------------------------------------------
This sf.net email is sponsored by: Dice - The leading online job board
for high-tech professionals. Search and apply for tech jobs today!
http://seeker.dice.com/seeker.epl?rel_code=31

Re: suspend.c vs driver-model.txt

[Patrick Mochel]

> > >There's power cycle and whole kernel bootup. Device state is
> > >completely lost during suspend to disk.
> > 
> > As it is during suspend-to-ram in most cases (most devices except
> > a couple of them are actually powered down and the drivers have to
> > bring them back up, at least on pmac, though on x86, the APM BIOS
> > helps you a bit there).
> 
> APM BIOS is irrelevant here (when using APM BIOS, everything is done
> by BIOS).

No, it's not. There is still a suspend sequence that must take place in 
the kernel. 

> What I'm doing is "drivers don't have to care about stopping requests
> because everything is stopped". If you are willing to go through all
> drivers and add stopping in them, thats fine with me, but I believe
> its post-2.6.

We have to deal with drivers. Someone has to modify the drivers to make 
them properly support power management. 

Now, it's not as bad as it sounds. Many drivers have pm_callback calls
that should be modified to the new style. So, we know what to do for many
devices. Other than those, the actively maintained drivers will be 
converted, as will be the most popular drivers. Others will gradually be 
taught to deal with it with PM properly. 

The burden is not necessarily on our shoulders. And, we can break 
everything. Trust me, I have permission ;)

	-pat



-------------------------------------------------------
This sf.net email is sponsored by: Dice - The leading online job board
for high-tech professionals. Search and apply for tech jobs today!
http://seeker.dice.com/seeker.epl?rel_code=31

Re: suspend.c vs driver-model.txt

[Pavel Machek]

Hi!

> > > >There's power cycle and whole kernel bootup. Device state is
> > > >completely lost during suspend to disk.
> > > 
> > > As it is during suspend-to-ram in most cases (most devices except
> > > a couple of them are actually powered down and the drivers have to
> > > bring them back up, at least on pmac, though on x86, the APM BIOS
> > > helps you a bit there).
> > 
> > APM BIOS is irrelevant here (when using APM BIOS, everything is done
> > by BIOS).
> 
> No, it's not. There is still a suspend sequence that must take place in 
> the kernel. 

In ideal world of working APM biosen, such sequence would be
useless. Unfortunately, we live in real world.

> > What I'm doing is "drivers don't have to care about stopping requests
> > because everything is stopped". If you are willing to go through all
> > drivers and add stopping in them, thats fine with me, but I believe
> > its post-2.6.
> 
> We have to deal with drivers. Someone has to modify the drivers to make 
> them properly support power management. 
> 
> Now, it's not as bad as it sounds. Many drivers have pm_callback calls
> that should be modified to the new style. So, we know what to do for many
> devices. Other than those, the actively maintained drivers will be 
> converted, as will be the most popular drivers. Others will gradually be 
> taught to deal with it with PM properly. 

I believe we should ease the pain of driver maintainers by
guaranteeing:

when driver_suspend() / driver_resume() is called

1) userspace is stopped.

2) we are running with just one active CPU.

1) implies that new requests are not coming, so driver authors do not
have to do if (suspended) sleep() in their hot paths.

								Pavel
-- 
Casualities in World Trade Center: ~3k dead inside the building,
cryptography in U.S.A. and free speech in Czech Republic.


-------------------------------------------------------
This sf.net email is sponsored by: Dice - The leading online job board
for high-tech professionals. Search and apply for tech jobs today!
http://seeker.dice.com/seeker.epl?rel_code=31

Re: suspend.c vs driver-model.txt

[Benjamin Herrenschmidt]

>1) implies that new requests are not coming, so driver authors do not
>have to do if (suspended) sleep() in their hot paths.

What about, let's say, an USB device driver that needs the USB host
chip (and thus the whole USB subsystem) to be still alive to send
proper commands to the device for suspend ? (typically setup the
remote wakeup when supported, etc...)

There can be even more nasty dependencies when you start stacking
things.

There are ordering dependencies. Those dependencies are dictated
by the bus layout that is fortunately provided for us by the new
driver model (let's not mix it with driverfs). It's not a subsystem
that has to be suspended not to send requests to a driver, it's
a driver that has to stop accepting requests, though it's then
perfectly valid for the subsystem to provide a "helper" routine
for the driver to deal with that.
(Typically, those "helpers" already exist for other purposes anyway
like stopping your block queue, stopping your network queue, etc...)

Ben.




-------------------------------------------------------
This sf.net email is sponsored by: Dice - The leading online job board
for high-tech professionals. Search and apply for tech jobs today!
http://seeker.dice.com/seeker.epl?rel_code=31

Re: suspend.c vs driver-model.txt

[Pavel Machek]

Hi!

> >1) implies that new requests are not coming, so driver authors do not
> >have to do if (suspended) sleep() in their hot paths.
> 
> What about, let's say, an USB device driver that needs the USB host
> chip (and thus the whole USB subsystem) to be still alive to send
> proper commands to the device for suspend ? (typically setup the
> remote wakeup when supported, etc...)

That's okay. We stop userspace. Then we attempt to suspend usb
camera. Thanks to driver ordering, USB host chip is still alive. When
all devices are done, USB host chips is suspended. Thanks to ordering,
there's noone alive to ask USB host chip to do anything, so USB host
chip driver does not need if (i_m_suspended) reject_request(), because
it just can not happen.
								Pavel
-- 
Casualities in World Trade Center: ~3k dead inside the building,
cryptography in U.S.A. and free speech in Czech Republic.


-------------------------------------------------------
This sf.net email is sponsored by: Dice - The leading online job board
for high-tech professionals. Search and apply for tech jobs today!
http://seeker.dice.com/seeker.epl?rel_code=31

Re: suspend.c vs driver-model.txt

[Benjamin Herrenschmidt]

>That's okay. We stop userspace. Then we attempt to suspend usb
>camera. Thanks to driver ordering, USB host chip is still alive. When
>all devices are done, USB host chips is suspended. Thanks to ordering,
>there's noone alive to ask USB host chip to do anything, so USB host
>chip driver does not need if (i_m_suspended) reject_request(), because
>it just can not happen.

In real life it did happen and that has been an issue for me in 2.4,
but well... that's because the USB device drivers don't deal with
suspend currently.

But the point here is that you don't suspend the _subsystem_ (USB,
block layer, network stack, whatever), but the individual drivers.

It do make sense to provide a way to stop requests. Requests are
not always originated from userspace. Several kernel services can
issue requests (VM, cache, whatever stuff on top of an i2c device,
etc...) and you can't shut down those services because you don't
have ordering rules for them, you only have ordering rules for
the drivers.

Ben.




-------------------------------------------------------
This sf.net email is sponsored by: Dice - The leading online job board
for high-tech professionals. Search and apply for tech jobs today!
http://seeker.dice.com/seeker.epl?rel_code=31

Re: suspend.c vs driver-model.txt

[Pavel Machek]

Hi!

> But the point here is that you don't suspend the _subsystem_ (USB,
> block layer, network stack, whatever), but the individual drivers.

You are probably right at this.

> It do make sense to provide a way to stop requests. Requests are
> not always originated from userspace. Several kernel services can
> issue requests (VM, cache, whatever stuff on top of an i2c device,
> etc...) and you can't shut down those services because you don't
> have ordering rules for them, you only have ordering rules for
> the drivers.

All kernel services that can issue requests need kernel threads,
right? My approach is to take kernel threads one by one and make them
suspend-aware.
								Pavel
-- 
Casualities in World Trade Center: ~3k dead inside the building,
cryptography in U.S.A. and free speech in Czech Republic.


-------------------------------------------------------
This sf.net email is sponsored by: Dice - The leading online job board
for high-tech professionals. Search and apply for tech jobs today!
http://seeker.dice.com/seeker.epl?rel_code=31

Re: suspend.c vs driver-model.txt

[Benjamin Herrenschmidt]

>All kernel services that can issue requests need kernel threads,
>right? My approach is to take kernel threads one by one and make them
>suspend-aware.

It wouldn' be bad to do that too. I just "feel" it safer and better
to define the driver semantics to be IO blocking. Some user processes
still want explicit notification, as will do some kernel threads I
beleive (example of user processes like this are those doing HW banging,
like XFree, there, console switch is really only a workaround, but
XFree is already equiped to deal with suspend requests from /dev/apm_bios).


Ben.




-------------------------------------------------------
This sf.net email is sponsored by: Dice - The leading online job board
for high-tech professionals. Search and apply for tech jobs today!
http://seeker.dice.com/seeker.epl?rel_code=31

Re: suspend.c vs driver-model.txt

[Pavel Machek]

Hi!

> >All kernel services that can issue requests need kernel threads,
> >right? My approach is to take kernel threads one by one and make them
> >suspend-aware.
> 
> It wouldn' be bad to do that too. I just "feel" it safer and better
> to define the driver semantics to be IO blocking. Some user
>processes

I do not know if such feelings are worth the effort of driver authors.

Certainly, it feels safer that way. But I believe that what I
currently have is bulletproof...
								Pavel
-- 
Casualities in World Trade Center: ~3k dead inside the building,
cryptography in U.S.A. and free speech in Czech Republic.


-------------------------------------------------------
This sf.net email is sponsored by: Dice - The leading online job board
for high-tech professionals. Search and apply for tech jobs today!
http://seeker.dice.com/seeker.epl?rel_code=31

Re: suspend.c vs driver-model.txt

[Patrick Mochel]

On Tue, 30 Jul 2002, Pavel Machek wrote:

> Hi!
> 
> > >All kernel services that can issue requests need kernel threads,
> > >right? My approach is to take kernel threads one by one and make them
> > >suspend-aware.
> > 
> > It wouldn' be bad to do that too. I just "feel" it safer and better
> > to define the driver semantics to be IO blocking. Some user
> >processes
> 
> I do not know if such feelings are worth the effort of driver authors.
> 
> Certainly, it feels safer that way. But I believe that what I
> currently have is bulletproof...

Heh. Famous last words... ;)

I think we have a good start, thanks entirely to the two of you. However, 
we won't be able to completely validate that we have all the theory right 
until we have all the working pieces at least partially in place. 

Unfortunately, I can't dedicate much time, as I'm swamped with other 
infrastructural items. But, I can offer some assistance.

I'm cloning a BK tree right now. It will be at 

	bk://ldm.bkbits.net/linux-2.5-suspend/

Send me patches, and I'll integrate them. We can use this as a test bed
for getting the process right before anything actually gets integrated.
People can clone from this tree for the purpose of testing.

I know there are issues with using BitKeeper. Because of that, I've 
created a directory on kernel.org to export patches of the BK tree. It's 
at:

	http://kernel.org/pub/linux/kernel/people/mochel/suspend/

I'll try and update the BK tree and the patches every kernel version. If 
you notice that I haven't, and it's been a couple of days, kick me. 

Enjoy. 

	-pat



-------------------------------------------------------
This sf.net email is sponsored by: Dice - The leading online job board
for high-tech professionals. Search and apply for tech jobs today!
http://seeker.dice.com/seeker.epl?rel_code=31

Re: suspend.c vs driver-model.txt

[Pavel Machek]

Hi!

> > > >All kernel services that can issue requests need kernel threads,
> > > >right? My approach is to take kernel threads one by one and make them
> > > >suspend-aware.
> > > 
> > > It wouldn' be bad to do that too. I just "feel" it safer and better
> > > to define the driver semantics to be IO blocking. Some user
> > >processes
> > 
> > I do not know if such feelings are worth the effort of driver authors.
> > 
> > Certainly, it feels safer that way. But I believe that what I
> > currently have is bulletproof...
> 
> Heh. Famous last words... ;)

Yep.

> I think we have a good start, thanks entirely to the two of you. However, 
> we won't be able to completely validate that we have all the theory right 
> until we have all the working pieces at least partially in place. 
> 
> Unfortunately, I can't dedicate much time, as I'm swamped with other 
> infrastructural items. But, I can offer some assistance.
> 
> I'm cloning a BK tree right now. It will be at 
> 
> 	bk://ldm.bkbits.net/linux-2.5-suspend/
> 
> Send me patches, and I'll integrate them. We can use this as a test bed
> for getting the process right before anything actually gets integrated.
> People can clone from this tree for the purpose of testing.

Well, it *is* integrated just now. People are actually using swsusp/S3
in 2.5.X more-or-less successfully. Linus is taking swsusp patches
pretty easily, and we get more testing by working on mainline. In
short I believe there's no reason to keep suspend patches
separate. Its experimental feature, marked as experimental, and it
seems to work for people. 
								Pavel
-- 
Casualities in World Trade Center: ~3k dead inside the building,
cryptography in U.S.A. and free speech in Czech Republic.


-------------------------------------------------------
This sf.net email is sponsored by: Dice - The leading online job board
for high-tech professionals. Search and apply for tech jobs today!
http://seeker.dice.com/seeker.epl?rel_code=31

Re: suspend.c vs driver-model.txt

[Benjamin Herrenschmidt]

>I do not know if such feelings are worth the effort of driver authors.

My experience with drivers here is that it isn't very difficult
to get that done right, but well...

>Certainly, it feels safer that way. But I believe that what I
>currently have is bulletproof...

If you say so... I'm still afraid with relying on the kernel
not triggering any IO for whatever random reason, I'd rather
have things designed to be bulletproof from the ground up, but
that may well be due to my lack of intimate knowledge of various
parts of the kernel not directly related to arch code & drivers.

We can end the debate here anyway, there is nothing in the
driver semantics I propose incompatible with software suspend
anyway, I'd suggest to document those semantics with blocking
of IOs and time will show how drivers keep up.

Patrick: On my side, I'll start moving the pmac drivers to
the new model as soon as I can get a recent 2.5 to boot a pmac
again, then I'll move my PM stuff to the new model and see what
issues arise. I'll try to have both suspend-to-RAM and suspend-
to-disk using Pavel's stuff working so we can validate the whole
mecanism.

No promise on time frame though, I've been pretty busy lately
and I have to get 2.4 working on some new Apple HW as well...

Ben.




-------------------------------------------------------
This sf.net email is sponsored by: Dice - The leading online job board
for high-tech professionals. Search and apply for tech jobs today!
http://seeker.dice.com/seeker.epl?rel_code=31

Re: suspend.c vs driver-model.txt

[Patrick Mochel]

> I believe we should ease the pain of driver maintainers by
> guaranteeing:
> 
> when driver_suspend() / driver_resume() is called
> 
> 1) userspace is stopped.

I thought that was a given ;)

> 2) we are running with just one active CPU.

Sure. We can tie that into subsystem notification, like this:

- The level at which we want to stop requests appears to be at the device 
  class level. 

- Device classes are coming RSN. 

- CPUs belong to a device class.

- We can integrate PM notication into device classes.

- The CPU device class can shut off all APs (application processors [silly 
  Intel term]).

	-pat



-------------------------------------------------------
This sf.net email is sponsored by: Dice - The leading online job board
for high-tech professionals. Search and apply for tech jobs today!
http://seeker.dice.com/seeker.epl?rel_code=31

Re: suspend.c vs driver-model.txt

[Benjamin Herrenschmidt]

>
>> I believe we should ease the pain of driver maintainers by
>> guaranteeing:
>> 
>> when driver_suspend() / driver_resume() is called
>> 
>> 1) userspace is stopped.
>
>I thought that was a given ;)

This isn't ;) I have userspace working up to the very latest
steps of suspend without trouble here ;) Userspace just stops
when blocked on an IO to a suspended driver, or just when the
CPU stops.

Though for swsusp, the problem of freeing the memory makes
you need to stop userspace, except if you can teach the VM to
not try to use those physical pages again until resume.

>> 2) we are running with just one active CPU.
>
>Sure. We can tie that into subsystem notification, like this:
>
>- The level at which we want to stop requests appears to be at the device 
>  class level. 

yes, _but_ the ordering is still dictated by the bus binding, not the
device class.

>- Device classes are coming RSN. 
>
>- CPUs belong to a device class.
>
>- We can integrate PM notication into device classes.

For the above reason, PM notification has to go via the bus hierarchy,
not the class hierarchy, though the device can then notify it's class
that it's no longer willing to do anything until resumed.

>- The CPU device class can shut off all APs (application processors [silly 
>  Intel term]).





-------------------------------------------------------
This sf.net email is sponsored by: Dice - The leading online job board
for high-tech professionals. Search and apply for tech jobs today!
http://seeker.dice.com/seeker.epl?rel_code=31

Re: suspend.c vs driver-model.txt

[Pavel Machek]

Hi!

> >> I believe we should ease the pain of driver maintainers by
> >> guaranteeing:
> >> 
> >> when driver_suspend() / driver_resume() is called
> >> 
> >> 1) userspace is stopped.
> >
> >I thought that was a given ;)
> 
> This isn't ;) I have userspace working up to the very latest
> steps of suspend without trouble here ;) Userspace just stops
> when blocked on an IO to a suspended driver, or just when the
> CPU stops.

Yep, but drivers need to be modified more in your model, and it may
have performance impact during "normal" use of system. I'd like to
avoid that.
								Pavel
-- 
Casualities in World Trade Center: ~3k dead inside the building,
cryptography in U.S.A. and free speech in Czech Republic.


-------------------------------------------------------
This sf.net email is sponsored by: Dice - The leading online job board
for high-tech professionals. Search and apply for tech jobs today!
http://seeker.dice.com/seeker.epl?rel_code=31

Re: suspend.c vs driver-model.txt

[Pavel Machek]

Hi!

> > I believe we should ease the pain of driver maintainers by
> > guaranteeing:
> > 
> > when driver_suspend() / driver_resume() is called
> > 
> > 1) userspace is stopped.
> 
> I thought that was a given ;)

Benjamin Herrenschmidt was arguing for doing driver_suspend() while
userspace is still running. If you assumed stopped userspace, you
don't have to add semaphore into device read() etc...

> > 2) we are running with just one active CPU.
> 
> Sure. We can tie that into subsystem notification, like this:
> 
> - The level at which we want to stop requests appears to be at the device 
>   class level. 
> 
> - Device classes are coming RSN. 
> 
> - CPUs belong to a device class.
> 
> - We can integrate PM notication into device classes.
> 
> - The CPU device class can shut off all APs (application processors [silly 
>   Intel term]).

I believe shutting down CPUs should be done outside device_suspend()
so all device_suspends() run on just one CPU...
								Pavel
-- 
Casualities in World Trade Center: ~3k dead inside the building,
cryptography in U.S.A. and free speech in Czech Republic.


-------------------------------------------------------
This sf.net email is sponsored by: Dice - The leading online job board
for high-tech professionals. Search and apply for tech jobs today!
http://seeker.dice.com/seeker.epl?rel_code=31

Re: suspend.c vs driver-model.txt

[Benjamin Herrenschmidt]

>Benjamin Herrenschmidt was arguing for doing driver_suspend() while
>userspace is still running. If you assumed stopped userspace, you
>don't have to add semaphore into device read() etc...

And stopping any kernel thread that may do that etc... I really
don't like it that much, but well, it can be a working solution for
2.6 for drivers having a direct userspace /dev interface.

>> > 2) we are running with just one active CPU.
>> 
>> Sure. We can tie that into subsystem notification, like this:
>> 
>> - The level at which we want to stop requests appears to be at the device 
>>   class level. 
>> 
>> - Device classes are coming RSN. 
>> 
>> - CPUs belong to a device class.
>> 
>> - We can integrate PM notication into device classes.
>> 
>> - The CPU device class can shut off all APs (application processors [silly 
>>   Intel term]).
>
>I believe shutting down CPUs should be done outside device_suspend()
>so all device_suspends() run on just one CPU...

That makes things easier, I tend to agree here.

Ben.




-------------------------------------------------------
This sf.net email is sponsored by: Dice - The leading online job board
for high-tech professionals. Search and apply for tech jobs today!
http://seeker.dice.com/seeker.epl?rel_code=31

Re: suspend.c vs driver-model.txt

[Patrick Mochel]

On Tue, 30 Jul 2002, Pavel Machek wrote:

> Hi!
> 
> > > I believe we should ease the pain of driver maintainers by
> > > guaranteeing:
> > > 
> > > when driver_suspend() / driver_resume() is called
> > > 
> > > 1) userspace is stopped.
> > 
> > I thought that was a given ;)
> 
> Benjamin Herrenschmidt was arguing for doing driver_suspend() while
> userspace is still running. If you assumed stopped userspace, you
> don't have to add semaphore into device read() etc...

Ok, let me go back and read again. :) I'm sure Ben will pipe up, too. 

> I believe shutting down CPUs should be done outside device_suspend()
> so all device_suspends() run on just one CPU...

Right. But, the subsystem notification would presumably happen outside of 
that.

	-pat



-------------------------------------------------------
This sf.net email is sponsored by: Dice - The leading online job board
for high-tech professionals. Search and apply for tech jobs today!
http://seeker.dice.com/seeker.epl?rel_code=31

Re: suspend.c vs driver-model.txt

[Pavel Machek]

Hi!

> > I believe shutting down CPUs should be done outside device_suspend()
> > so all device_suspends() run on just one CPU...
> 
> Right. But, the subsystem notification would presumably happen outside of 
> that.

Okay, I can live with that.

[So we are getting "struct subsystem { }" ?]
								Pavel
-- 
Casualities in World Trade Center: ~3k dead inside the building,
cryptography in U.S.A. and free speech in Czech Republic.


-------------------------------------------------------
This sf.net email is sponsored by: Dice - The leading online job board
for high-tech professionals. Search and apply for tech jobs today!
http://seeker.dice.com/seeker.epl?rel_code=31

Re: suspend.c vs driver-model.txt

[Patrick Mochel]

> [So we are getting "struct subsystem { }" ?]

We're getting at least 'struct device_class'.

	-pat



-------------------------------------------------------
This sf.net email is sponsored by: Dice - The leading online job board
for high-tech professionals. Search and apply for tech jobs today!
http://seeker.dice.com/seeker.epl?rel_code=31

Re: suspend.c vs driver-model.txt

[Alan Cox]

On Tue, 2002-07-30 at 20:00, Pavel Machek wrote:
> Hi!
> Benjamin Herrenschmidt was arguing for doing driver_suspend() while
> userspace is still running. If you assumed stopped userspace, you
> don't have to add semaphore into device read() etc...

Having been around this a few times for embedded devices I've found
nothing that cleanly solves the problems except

Call all the devices "suspend ready irq on" methods
Disable interrupts
Call all the devices "final suspend" method



-------------------------------------------------------
This sf.net email is sponsored by: Dice - The leading online job board
for high-tech professionals. Search and apply for tech jobs today!
http://seeker.dice.com/seeker.epl?rel_code=31

Re: suspend.c vs driver-model.txt

[Benjamin Herrenschmidt]

>> Benjamin Herrenschmidt was arguing for doing driver_suspend() while
>> userspace is still running. If you assumed stopped userspace, you
>> don't have to add semaphore into device read() etc...
>
>Having been around this a few times for embedded devices I've found
>nothing that cleanly solves the problems except
>
>Call all the devices "suspend ready irq on" methods
>Disable interrupts
>Call all the devices "final suspend" method

Yes, I remember you telling me about it, which simply involves
breaking the 3rd (ie. "suspend") PM notification into a
"suspend_irq_on" and a "suspend_irq_off". Drivers that don't
care will use only one of them. But that's not a real issue,
the problems are really around the "save_state & block IOs"
step which is step 2.

 step 1: prepare (that is allocate any memory needed, make sure
         the driver can still operate without allocating more
         memory, or eventually only ATOMIC or NOIO).

 step 2: save_state & block IOs, that is make sure the saved state
         is consistent with the last IOs and that no more IO will
         be processed. The driver don't need to wait for completion
         of pending IOs at this stage I beleive if those don't
         impact the saved state, but it may be better to wait for
         them at first.

 step 3: suspend_irq_on, here we do the real suspend of the chip
         if possible (note that some arch may need to use D1 or D2
         instead of D3 for various reasons, but that will be dealt
         on a driver specific basis for now). That step don't need
         to be called by swsusp

 step 4: suspend_irq_off, followup of step 3) for drivers that
         need it.

Ben.




-------------------------------------------------------
This sf.net email is sponsored by: Dice - The leading online job board
for high-tech professionals. Search and apply for tech jobs today!
http://seeker.dice.com/seeker.epl?rel_code=31

Re: suspend.c vs driver-model.txt

[Pavel Machek]

Hi!

> >> Benjamin Herrenschmidt was arguing for doing driver_suspend() while
> >> userspace is still running. If you assumed stopped userspace, you
> >> don't have to add semaphore into device read() etc...
> >
> >Having been around this a few times for embedded devices I've found
> >nothing that cleanly solves the problems except
> >
> >Call all the devices "suspend ready irq on" methods
> >Disable interrupts
> >Call all the devices "final suspend" method
> 
> Yes, I remember you telling me about it, which simply involves
> breaking the 3rd (ie. "suspend") PM notification into a
> "suspend_irq_on" and a "suspend_irq_off". Drivers that don't
> care will use only one of them. But that's not a real issue,
> the problems are really around the "save_state & block IOs"
> step which is step 2.
> 
>  step 1: prepare (that is allocate any memory needed, make sure
>          the driver can still operate without allocating more
>          memory, or eventually only ATOMIC or NOIO).
> 
>  step 2: save_state & block IOs, that is make sure the saved state
>          is consistent with the last IOs and that no more IO will
>          be processed. The driver don't need to wait for completion
>          of pending IOs at this stage I beleive if those don't
>          impact the saved state, but it may be better to wait for
>          them at first.

I really need all DMAs to be off after step 2; so I need all drivers
doing DMA to wait for completion. [I can not do atomic copy of memory
with DMA writing over it behind my back.]
								Pavel
-- 
Casualities in World Trade Center: ~3k dead inside the building,
cryptography in U.S.A. and free speech in Czech Republic.


-------------------------------------------------------
This sf.net email is sponsored by: Dice - The leading online job board
for high-tech professionals. Search and apply for tech jobs today!
http://seeker.dice.com/seeker.epl?rel_code=31

Re: suspend.c vs driver-model.txt

[Benjamin Herrenschmidt]

>I really need all DMAs to be off after step 2; so I need all drivers
>doing DMA to wait for completion. [I can not do atomic copy of memory
>with DMA writing over it behind my back.]

Ok, that is a valid point, so let's say that step 2) explicitely
requires IO operations to be complete and DMA stopped.

Ben.




-------------------------------------------------------
This sf.net email is sponsored by: Dice - The leading online job board
for high-tech professionals. Search and apply for tech jobs today!
http://seeker.dice.com/seeker.epl?rel_code=31

Re: suspend.c vs driver-model.txt

[Alan Cox]

On Tue, 2002-07-30 at 19:32, Patrick Mochel wrote:
> 
> > > >There's power cycle and whole kernel bootup. Device state is
> > > >completely lost during suspend to disk.
> > > 
> > > As it is during suspend-to-ram in most cases (most devices except
> > > a couple of them are actually powered down and the drivers have to
> > > bring them back up, at least on pmac, though on x86, the APM BIOS
> > > helps you a bit there).
> > 
> > APM BIOS is irrelevant here (when using APM BIOS, everything is done
> > by BIOS).
> 
> No, it's not. There is still a suspend sequence that must take place in 
> the kernel. 

Also almost no APM bios gets AGP or MTRR restore right.



-------------------------------------------------------
This sf.net email is sponsored by: Dice - The leading online job board
for high-tech professionals. Search and apply for tech jobs today!
http://seeker.dice.com/seeker.epl?rel_code=31

RE: suspend.c vs driver-model.txt

[Grover, Andrew]

> From: Patrick Mochel [mailto:mochel-3NddpPZAyC0@public.gmane.org] 
> > People interested in other arches need to step up and make 
> sure their
> > platform's requirements are met. Failing that, IA32 is the 
> HW that most
> > people have so that's what is going to be mde to work 
> first. Linux itself
> > was not a cross-platform OS from day one -- the 
> generalization came later.
> 
> I love these arguments of yours; I get a big kick out of 
> them. Like your 
> defense of "premature optimization is the root of all evil." 
> Sure it is, 
> but that's no excuse to write inefficient algorithms. 

It is an excuse to use simple algorithms until they are proven to be
inefficient, and only then replace them with something faster and more
complex.

But, I think you meant to target my *other* favorite software aphorism: "bad
code is better than no code"...yes? :) I still think that Linux's birth is a
great example of this, and how software can grow, evolve, and improve over
time.

> Just like it's no excuse to create a general scheme based on 
> one platform. 
> c.f. the original Linux Driver Model that you guys posted one 
> year ago. 
> That is the epitome of my argument. I'm not worked up about 
> it, but you 
> guys have done that kinda stuff more than you should, and I'm 
> glad you're 
> in the business of designing general APIs any more. 

(I assume there should be a "not" in there)

Well, the idea was that maybe it was not quite generic enough, but then it
would evolve into something that could work for all platforms, as people who
cared about other arches had their say. Didn't work out that way, I guess.

Considering that a year ago was pre-device tree, I think us taking a stab at
an interface was not unreasonable.

> driverfs was never about PM. It was about exposing kernel 
> infrastructure 
> to userspace. I gave you the device tree and power management 
> hooks first. 
> I apologize that I have not had a chance to follow through 
> and actually 
> implement the hooks for the drivers, but there turned out to 
> be a lot more 
> cool stuff to play with.. :)

OK, that's true. The device tree is the crucial thing for PM.

Regards -- Andy

http://www.joelonsoftware.com/articles/fog0000000017.html
http://www.joelonsoftware.com/articles/fog0000000339.html


-------------------------------------------------------
This sf.net email is sponsored by: Dice - The leading online job board
for high-tech professionals. Search and apply for tech jobs today!
http://seeker.dice.com/seeker.epl?rel_code=31

RE: suspend.c vs driver-model.txt

[Grover, Andrew]

> From: Patrick Mochel [mailto:mochel-3NddpPZAyC0@public.gmane.org] 

> The ACPI code that is in there is relatively close. I did 
> that on purpose, 
> thinking that this stuff will be generalized someday. 
> However, whatever 
> end solution that exists _must_ not be designed the behavior 
> of ACPI. In 
> fact, I am hesitant talking about this general model on this 
> list for fear 
> that someone that is already brainwashed by the Great 
> Firmware Abomination 
> will say either "We can already do this!" or "I'll do it (in 
> ACPI first 
> and generalize it)." 

linux-pm-devel-pyega4qmqnRoyOMFzWx49A@public.gmane.org was created to have a non-ACPI-specific PM
forum. Nobody has used that, but it's there, and I'm subscribed just in case
anyone does. ;-)

People interested in other arches need to step up and make sure their
platform's requirements are met. Failing that, IA32 is the HW that most
people have so that's what is going to be mde to work first. Linux itself
was not a cross-platform OS from day one -- the generalization came later.

> <harshness>
> The ACPI subsystem has a tendency to duplicate existing 
> interfaces, or 
> come up with completely new ones, intending to be applicable 
> to the rest 
> of the kernel. The are highly ACPI-centric, yet they expect 
> them to be 
> standard. They aren't publicized or discussed on any list but 
> this one. 
> But, they're left to be fixed up and cleaned up by the other 
> people in the 
> community. 

I agree that any time ACPI can use an existing interface over defining a new
one that's good, but for a lot of things existing interfaces didn't exist.
In 2.5 we now are starting to see std interfaces for device enum (thanks to
you) and processor performance control (the cpufreq project) but thermal,
battery, etc do not yet have arch-neutral interfaces for ACPI to plug into.
Are these the interface you are referring to, or different ones?

> I refuse to lay blame on anyone for my inability to work 
> faster or harder. 
> But, a lot of people complain that I've been working on this 
> for a year, 
> and they still can't suspend their laptop. My rebuttal is 
> that if I had 
> actually gotten cooperation and compromise at the source 
> level, in many 
> areas, we probably wouldn't still be dealing with this. 

If anyone complains about the lack of progress then they don't understand
the scope of the work. I think *that* is why we're not there yet, instead of
a lack of cooperation.

The only criticism I have is that driverfs etc. has grown to be much more
than PM. I know you've been working with Greg, but it's still too much,
IMHO. -EPLATEFULL, indeed.

> So, someone needs to do it. I don't have time. And, I don't 
> trust anyone 
> related to ACPI to do it right (except maybe Andy in a 
> different context, 
> but he's overworked as it is). 

Ben sounds willing. ;-)

Regards -- Andy


-------------------------------------------------------
This sf.net email is sponsored by:ThinkGeek
Welcome to geek heaven.
http://thinkgeek.com/sf

RE: suspend.c vs driver-model.txt

[Patrick Mochel]

> linux-pm-devel-pyega4qmqnRoyOMFzWx49A@public.gmane.org was created to have a non-ACPI-specific PM
> forum. Nobody has used that, but it's there, and I'm subscribed just in case
> anyone does. ;-)

Yeah, it's been kinda dead for a while. Splinter mail lists always fizzle 
out, probably because coming up with the design always gets preempted by 
doing real work. ;)

> People interested in other arches need to step up and make sure their
> platform's requirements are met. Failing that, IA32 is the HW that most
> people have so that's what is going to be mde to work first. Linux itself
> was not a cross-platform OS from day one -- the generalization came later.

I love these arguments of yours; I get a big kick out of them. Like your 
defense of "premature optimization is the root of all evil." Sure it is, 
but that's no excuse to write inefficient algorithms. 

Just like it's no excuse to create a general scheme based on one platform. 
c.f. the original Linux Driver Model that you guys posted one year ago. 
That is the epitome of my argument. I'm not worked up about it, but you 
guys have done that kinda stuff more than you should, and I'm glad you're 
in the business of designing general APIs any more. 

> The only criticism I have is that driverfs etc. has grown to be much more
> than PM. I know you've been working with Greg, but it's still too much,
> IMHO. -EPLATEFULL, indeed.

driverfs was never about PM. It was about exposing kernel infrastructure 
to userspace. I gave you the device tree and power management hooks first. 
I apologize that I have not had a chance to follow through and actually 
implement the hooks for the drivers, but there turned out to be a lot more 
cool stuff to play with.. :)

	-pat



-------------------------------------------------------
This sf.net email is sponsored by: Dice - The leading online job board
for high-tech professionals. Search and apply for tech jobs today!
http://seeker.dice.com/seeker.epl?rel_code=31

Re: suspend.c vs driver-model.txt

[Lyle]

Also, drivers/acpi/system.c seems to assume that SAVE_STATE and RESTORE_STATE 
will implicitly disable and enable I/O, but that's not the way it's specified, 
and that's not the way that any code wants to work -- that I can see.  Do we 
agree?

> The more I look at this, the more convinced I am that driver-model.txt 
> describes the proper approach, and suspend.c just has it backwards.  Even 
> though no drivers really implement the functionality properly yet, it makes 
> sense to get the framework right so that they *can*, eventually.
> 
> I don't know why driver.h has the enums listed in a different order from 
> driver-model.txt, or if it's really necessary to fix the order, unless 
> somebody decides to start doing ordered comparisons.
> 
> Just to make it clear what I mean



-------------------------------------------------------
This sf.net email is sponsored by:ThinkGeek
Welcome to geek heaven.
http://thinkgeek.com/sf

Re: suspend.c vs driver-model.txt

[Patrick Mochel]

On Wed, 24 Jul 2002, Lyle wrote:

> 
> Also, drivers/acpi/system.c seems to assume that SAVE_STATE and RESTORE_STATE 
> will implicitly disable and enable I/O, but that's not the way it's specified, 
> and that's not the way that any code wants to work -- that I can see.  Do we 
> agree?

You're right, there should be an explicit call to disable I/O for the 
devices. 

On a side, though related, note, while we were in Ottawa, several of us 
had a chance to sit down and hammer out exactly what needs to take place. 
The results exist in a notebook (a real, pencil-driven one), but have not 
had a chance to be expressed in ASCII text yet. (Frankly, it's a low 
priority right now: -EPLATEFULL). 

> > I don't know why driver.h has the enums listed in a different order from 
> > driver-model.txt, or if it's really necessary to fix the order, unless 
> > somebody decides to start doing ordered comparisons.

I assume you mean include/linux/device.h. Yes, the enums are in the wrong 
order. Thanks for pointing that out. 

	-pat



-------------------------------------------------------
This sf.net email is sponsored by:ThinkGeek
Welcome to geek heaven.
http://thinkgeek.com/sf

Re: suspend.c vs driver-model.txt

[Pavel Machek]

Hi!

> The more I look at this, the more convinced I am that driver-model.txt 
> describes the proper approach, and suspend.c just has it backwards.  Even 
> though no drivers really implement the functionality properly yet, it makes 
> sense to get the framework right so that they *can*, eventually.
> 
> I don't know why driver.h has the enums listed in a different order from 
> driver-model.txt, or if it's really necessary to fix the order, unless 
> somebody decides to start doing ordered comparisons.

The diff looks right, will apply...
									Pavel

> Just to make it clear what I mean:
> 			 
> *** kernel/suspend.c   Wed Jul 24 09:21:39 2002
> --- kernel/suspend.c.orig      Wed Jul 24 09:20:32 2002
> ***************
> *** 647,650 ****
>   {
> -       device_resume(RESUME_ENABLE);
>         device_resume(RESUME_RESTORE_STATE);
>   }
> --- 647,650 ----
>   {
>         device_resume(RESUME_RESTORE_STATE);
> +       device_resume(RESUME_ENABLE);
>   }
> ***************
> *** 655,658 ****
>         device_suspend(4, SUSPEND_NOTIFY);
> -       device_suspend(4, SUSPEND_DISABLE);
>         device_suspend(4, SUSPEND_SAVE_STATE);
>         if(!pm_suspend_state) {
> --- 655,658 ----
>         device_suspend(4, SUSPEND_NOTIFY);
>         device_suspend(4, SUSPEND_SAVE_STATE);
> +       device_suspend(4, SUSPEND_DISABLE);
>         if(!pm_suspend_state) {
> ***************
> *** 674,677 ****
>   {
> -       device_resume(RESUME_ENABLE);
>         device_resume(RESUME_RESTORE_STATE);
>         if(flags & RESUME_PHASE2) {
> --- 674,677 ----
>   {
>         device_resume(RESUME_RESTORE_STATE);
> +       device_resume(RESUME_ENABLE);
>         if(flags & RESUME_PHASE2) {
> 
> 
> 
> -------------------------------------------------------
> This sf.net email is sponsored by:ThinkGeek
> Welcome to geek heaven.
> http://thinkgeek.com/sf
> _______________________________________________
> Acpi-devel mailing list
> Acpi-devel-5NWGOfrQmneRv+LV9MX5uipxlwaOVQ5f@public.gmane.org
> https://lists.sourceforge.net/lists/listinfo/acpi-devel

-- 
Casualities in World Trade Center: ~3k dead inside the building,
cryptography in U.S.A. and free speech in Czech Republic.


-------------------------------------------------------
This sf.net email is sponsored by:ThinkGeek
Welcome to geek heaven.
http://thinkgeek.com/sf

Re: suspend.c vs driver-model.txt

[Lyle]

> You're right, there should be an explicit call to disable I/O for the 
> devices. 
> 
> On a side, though related, note, while we were in Ottawa, several of us 
> had a chance to sit down and hammer out exactly what needs to take place. 
> The results exist in a notebook (a real, pencil-driven one), but have not 
> had a chance to be expressed in ASCII text yet. (Frankly, it's a low 
> priority right now: -EPLATEFULL). 

Is there any chance that I can transcribe the notes into ASCII for you?  
Or are they too arcane?  Could you at least scan them?

> I assume you mean include/linux/device.h. Yes, the enums are in the wrong 

whoops, yes.




-------------------------------------------------------
This sf.net email is sponsored by:ThinkGeek
Welcome to geek heaven.
http://thinkgeek.com/sf

Re: suspend.c vs driver-model.txt

[Patrick Mochel]

On Wed, 24 Jul 2002, Lyle wrote:

> 
> > You're right, there should be an explicit call to disable I/O for the 
> > devices. 
> > 
> > On a side, though related, note, while we were in Ottawa, several of us 
> > had a chance to sit down and hammer out exactly what needs to take place. 
> > The results exist in a notebook (a real, pencil-driven one), but have not 
> > had a chance to be expressed in ASCII text yet. (Frankly, it's a low 
> > priority right now: -EPLATEFULL). 
> 
> Is there any chance that I can transcribe the notes into ASCII for you?  
> Or are they too arcane?  Could you at least scan them?

I don't do the scanning thing. :) 

I would love to pass it off on someone, but there's a lot of context 
involved. 

Basically, what we need is for someone to develop a general power 
management model for the kernel. The path to implementation is partially 
complete, but someone needs to fill in the blanks, and make sure that 
other platforms are accounted for. 

The mechanism for transition should be a driverfs file which a user or app 
can write to. There could also be a hook into the sys_reboot() call. 

The handler for this is generic. There will likely be generic power 
management transitions, like swsusp. But, there will always be 
platform-specific transitions, and platform-specific operations for even 
the generic transitions. So, the generic handler needs to pass the command 
to the "platform driver". 

So, there needs to be a platform driver. See include/linux/platform.h and 
kernel/platform.c for the start of this effort. There needs to be one of 
these for each platform type. They need to be registered on startup, and 
there needs to be prevention against multiple drivers being registered. 

The platform drivers need to then do whatever device transitions they 
need, using the device_suspend() helpers, then put the system to sleep 
using their platform-speciifc method.

The ACPI code that is in there is relatively close. I did that on purpose, 
thinking that this stuff will be generalized someday. However, whatever 
end solution that exists _must_ not be designed the behavior of ACPI. In 
fact, I am hesitant talking about this general model on this list for fear 
that someone that is already brainwashed by the Great Firmware Abomination 
will say either "We can already do this!" or "I'll do it (in ACPI first 
and generalize it)." 

<harshness>
The ACPI subsystem has a tendency to duplicate existing interfaces, or 
come up with completely new ones, intending to be applicable to the rest 
of the kernel. The are highly ACPI-centric, yet they expect them to be 
standard. They aren't publicized or discussed on any list but this one. 
But, they're left to be fixed up and cleaned up by the other people in the 
community. 

I refuse to lay blame on anyone for my inability to work faster or harder. 
But, a lot of people complain that I've been working on this for a year, 
and they still can't suspend their laptop. My rebuttal is that if I had 
actually gotten cooperation and compromise at the source level, in many 
areas, we probably wouldn't still be dealing with this. 
</harshness>

So, someone needs to do it. I don't have time. And, I don't trust anyone 
related to ACPI to do it right (except maybe Andy in a different context, 
but he's overworked as it is). 


	-pat



-------------------------------------------------------
This sf.net email is sponsored by:ThinkGeek
Welcome to geek heaven.
http://thinkgeek.com/sf

Re: suspend.c vs driver-model.txt

[Benjamin Herrenschmidt]

>
> .../...
>
>So, there needs to be a platform driver. See include/linux/platform.h and 
>kernel/platform.c for the start of this effort. There needs to be one of 
>these for each platform type. They need to be registered on startup, and 
>there needs to be prevention against multiple drivers being registered. 
>
>The platform drivers need to then do whatever device transitions they 
>need, using the device_suspend() helpers, then put the system to sleep 
>using their platform-speciifc method.

Good ! I've had my home-made bunch of stuff on pmac that looked just
like a platform driver dealing with all sort of things (basically
abtracting motherboard IOs to help drivers that need to toggle whatever
reset pin or power their device up/down physically).
(See include/asm-ppc/pmac_feature.c and arch/ppc/kernel/pmac_feature.c)

On PPC, we also have that _huge_ flat ppc_md structure that contains
pointers to functions like platform init, cpuinfo hooks, get/set_rtc_time,
irq hooks, etc... that get filled depending on the type of machine a
given kernel is booted (we support currently prep, chrp and pmac with
a single kernel binary).
(see include/asm-ppc/machdep.h)

What about generalizing the platform driver to do just those things ?

I'm not too sure about how the API would look like though. For the
ppc_md stuff, there is a fixed, well defined, but still arch specific
set of hooks to fill.

For the pmac_feature, I use a single ppc_md. hook with variable arg
list (though I do use it in a well defined format) to call a list
of "feature" functions that is usually exetended on every new kind
of motherboard I add support for. Not every machine support all
functions, etc... it would probably be a pain to define this as a
flat list of function pointers.

Anybody has better ideas on how to deal with those platform specific
things while still making them fit in a platform driver (their logical
place, especially if the platform driver ends up in driverfs, a nice
way to expose some motherboard specific settings/tweaks to userland ?

>The ACPI code that is in there is relatively close. I did that on purpose, 
>thinking that this stuff will be generalized someday. However, whatever 
>end solution that exists _must_ not be designed the behavior of ACPI. In 
>fact, I am hesitant talking about this general model on this list for fear 
>that someone that is already brainwashed by the Great Firmware Abomination 
>will say either "We can already do this!" or "I'll do it (in ACPI first 
>and generalize it)." 

I don't know ACPI well enough to talk, but the mecanism I use on PPC
is pretty basic today. The sleep procedure is dealt with by the driver
of the Power Management chip when it gets a sleep request from userland,
but that can (should be) moved to the platform driver.

Then, it does my own version of looping all devices sending them
suspend events etc... (did we ever write down what we defined during
OLS for those events semantics ?)

Then, calls the platform driver to deal with lower level motherboard
stuffs that haven't be done by any drivers and need to be done last

Finally it deals with the CPU (flushing caches, etc...).

I omitted intentionally the fact that it has to send a special command
to the power manager chip just before the low level mother board stuff,
that will cause that chip to shut the CPU down the next time it goes
to SLEEP state, which we do in the last step.

So most of this scenario is arch/platform specific cuisine. That is, all
but the first step has to be self contained somewhere in the platform
driver, those are arch/platform/cpu specific, nothing good we can do
about them.

This end up breaking the scenario into 2 main phases

 1) Looping around all drivers doing the suspend dance (suspend request,
save_state & suspend, ...) as we discussed. This is mostly (completely ?)
platform independant (except for the exact state a given device has to
be put to, oh well, let's deal that inside the device driver for now)

 2) Doing whatever additional arch specific code is needed to snooze
the machine.

1) is generic. 2) is arch/platform code.

The nice thing here, I beleive, is that suspend to disk could then just
be an additional generic step between 1) and 2) that takes care of
dumping memory content to disk.

As far as suspend to disk is concerned, we have all sort of nasty issues
we discussed that I don't feel like resuming right now, so I would rather
have the lowest level driver able to play the backing store export a
couple of "hooks" used to dump the pages to the medium, those would work
synchronously, without VM activity, etc... The driver that is target for
that would have pre-allocated what it needs, pre-mapped DMA mappings it
needs etc...
Though, thinking as I write, injecting requests via the BIO layer may
be a nicer solution, provided we make sure the driver won't get any
request _but_ the ones we feed it. I don't trust a mecanism that would
selectively stop given kernel threads, then just hope nobody will push
requests any more. That's way too fragile to my taste. I'd rather freeze
the request queue for real like any other device, then eventually provide
a special pass-through.

><harshness>
>The ACPI subsystem has a tendency to duplicate existing interfaces, or 
>come up with completely new ones, intending to be applicable to the rest 
>of the kernel. The are highly ACPI-centric, yet they expect them to be 
>standard. They aren't publicized or discussed on any list but this one. 
>But, they're left to be fixed up and cleaned up by the other people in the 
>community. 
>
>I refuse to lay blame on anyone for my inability to work faster or harder. 
>But, a lot of people complain that I've been working on this for a year, 
>and they still can't suspend their laptop. My rebuttal is that if I had 
>actually gotten cooperation and compromise at the source level, in many 
>areas, we probably wouldn't still be dealing with this. 
></harshness>

I'm probably to blame for not writing code here as I mostly only
contributed with speetch and ideas (and my old pmac code) so far.
I'm sorry, i'm not even been able to find time to convert the pmac
drivers to driverfs yet :( My time dedicated to linux is spare, and
so variable, and it tends to be quite small nowadays.

>So, someone needs to do it. I don't have time. And, I don't trust anyone 
>related to ACPI to do it right (except maybe Andy in a different context, 
>but he's overworked as it is). 

I've subscribed to this list, it may be a good oportunity to use it
for discussing & defining things at the API level now.

Regards,
Ben.




-------------------------------------------------------
This sf.net email is sponsored by:ThinkGeek
Welcome to geek heaven.
http://thinkgeek.com/sf

Re: suspend.c vs driver-model.txt

[Pavel Machek]

Hi!

> The nice thing here, I beleive, is that suspend to disk could then just
> be an additional generic step between 1) and 2) that takes care of
> dumping memory content to disk.

It can not, because after saving state you can not easily write to
swap partition.

> As far as suspend to disk is concerned, we have all sort of nasty issues
> we discussed that I don't feel like resuming right now, so I would rather
> have the lowest level driver able to play the backing store export a
> couple of "hooks" used to dump the pages to the medium, those would work
> synchronously, without VM activity, etc... The driver that is target for
> that would have pre-allocated what it needs, pre-mapped DMA mappings it
> needs etc...

Too complicated. Look at swsusp -- it is designed not to need
pre-allocated pages, pre-mapped DMA etc.

> Though, thinking as I write, injecting requests via the BIO layer may
> be a nicer solution, provided we make sure the driver won't get any
> request _but_ the ones we feed it. I don't trust a mecanism that would
> selectively stop given kernel threads, then just hope nobody will push
> requests any more. That's way too fragile to my taste. I'd rather
> freeze

Why is it fragile? You just have to be carefull. If you mark thread as
PF_IOTHREAD it must *not* generate requests unless told to.
								Pavel
-- 
Worst form of spam? Adding advertisment signatures ala sourceforge.net.
What goes next? Inserting advertisment *into* email?


-------------------------------------------------------
This sf.net email is sponsored by: Dice - The leading online job board
for high-tech professionals. Search and apply for tech jobs today!
http://seeker.dice.com/seeker.epl?rel_code=31

Re: suspend.c vs driver-model.txt

[Benjamin Herrenschmidt]

>Hi!
>
>> The nice thing here, I beleive, is that suspend to disk could then just
>> be an additional generic step between 1) and 2) that takes care of
>> dumping memory content to disk.
>
>It can not, because after saving state you can not easily write to
>swap partition.

Well, you have to. I mean, you can't write to swap before saving state
(or you'll miss some state information). Which means the device dealing
with the save has to be treated as a special case. At least, that's what
we figured out when discussing this during OLS. 

>> As far as suspend to disk is concerned, we have all sort of nasty issues
>> we discussed that I don't feel like resuming right now, so I would rather
>> have the lowest level driver able to play the backing store export a
>> couple of "hooks" used to dump the pages to the medium, those would work
>> synchronously, without VM activity, etc... The driver that is target for
>> that would have pre-allocated what it needs, pre-mapped DMA mappings it
>> needs etc...
>
>Too complicated. Look at swsusp -- it is designed not to need
>pre-allocated pages, pre-mapped DMA etc.

I didn't look at it recently, but does it work properly in a solid
way whatever state other devices are ?

>> Though, thinking as I write, injecting requests via the BIO layer may
>> be a nicer solution, provided we make sure the driver won't get any
>> request _but_ the ones we feed it. I don't trust a mecanism that would
>> selectively stop given kernel threads, then just hope nobody will push
>> requests any more. That's way too fragile to my taste. I'd rather
>> freeze
>
>Why is it fragile? You just have to be carefull. If you mark thread as
>PF_IOTHREAD it must *not* generate requests unless told to.

I still don't like that very much, but that could still be a solution.
Anyway, I need to look at your latest swsusp work.

Ben.




-------------------------------------------------------
This sf.net email is sponsored by: Dice - The leading online job board
for high-tech professionals. Search and apply for tech jobs today!
http://seeker.dice.com/seeker.epl?rel_code=31

Re: suspend.c vs driver-model.txt

[Pavel Machek]

Hi!

> >> The nice thing here, I beleive, is that suspend to disk could then just
> >> be an additional generic step between 1) and 2) that takes care of
> >> dumping memory content to disk.
> >
> >It can not, because after saving state you can not easily write to
> >swap partition.
> 
> Well, you have to. I mean, you can't write to swap before saving state
> (or you'll miss some state information). Which means the device dealing
> with the save has to be treated as a special case. At least, that's what
> we figured out when discussing this during OLS. 

Discussing with who?

Its rather simple: free half of memory, stop everything, save device
state, copy used memory into free memory, continue devices, save data
from "free" memory into swap.

> >Too complicated. Look at swsusp -- it is designed not to need
> >pre-allocated pages, pre-mapped DMA etc.
> 
> I didn't look at it recently, but does it work properly in a solid
> way whatever state other devices are ?

It is designed to be safe, yes.

								Pavel
-- 
Worst form of spam? Adding advertisment signatures ala sourceforge.net.
What goes next? Inserting advertisment *into* email?


-------------------------------------------------------
This sf.net email is sponsored by: Dice - The leading online job board
for high-tech professionals. Search and apply for tech jobs today!
http://seeker.dice.com/seeker.epl?rel_code=31

Re: suspend.c vs driver-model.txt

[Benjamin Herrenschmidt]

>Discussing with who?

We had a Power Management BOF there, among others, Patrick Mochel,
Greg KH, Andy, Paulus, Jeff Garzik, ...

>Its rather simple: free half of memory, stop everything, save device
>state, copy used memory into free memory, continue devices, save data
>from "free" memory into swap.

Ok, so you first push as much as you can to swap to free half
of memory. Then, when you say you copy used memory, I assume you copy
all of RAM including kernel memory (kernel data structures etc...)
right ?

What do you mean by "stop everything" ? Stop user processes ? Kernel
threads ? both ? Then, you only save device state (and then suspend
devices) after you have "stopped everything" ?

Ben.




-------------------------------------------------------
This sf.net email is sponsored by: Dice - The leading online job board
for high-tech professionals. Search and apply for tech jobs today!
http://seeker.dice.com/seeker.epl?rel_code=31

Re: suspend.c vs driver-model.txt

[Pavel Machek]

Hi!

> >Discussing with who?
> 
> We had a Power Management BOF there, among others, Patrick Mochel,
> Greg KH, Andy, Paulus, Jeff Garzik, ...

Oops... I guess I should have been there :-(. Sorry.

> >Its rather simple: free half of memory, stop everything, save device
> >state, copy used memory into free memory, continue devices, save data
> >from "free" memory into swap.
> 
> Ok, so you first push as much as you can to swap to free half
> of memory. Then, when you say you copy used memory, I assume you copy
> all of RAM including kernel memory (kernel data structures etc...)
> right ?

Yes.

> What do you mean by "stop everything" ? Stop user processes ? Kernel
> threads ? both ? 

Stop all user processes and all kernel processes that could cause
trouble. Don't stop kapm-idled (for example) but be sure to stop
ext3-commitd (or how is that beast called).

> Then, you only save device state (and then suspend
> devices) after you have "stopped everything" ?

Yes.

Code looks like this:

void do_software_suspend(void)
{
        arch_prepare_suspend();
        if (prepare_suspend_console())
                printk( "%sCan't allocate a console... proceeding\n", name_suspend);
        if (!prepare_suspend_processes()) {

                /* At this point, all user processes and "dangerous"
                   kernel threads are stopped. Free some memory, as we
                   need half of memory free. */

                free_some_memory();

                /* No need to invalidate any vfsmnt list -- they will be valid after resume, anyway.
                 *
                 * We sync here -- so you have consistent filesystem state when things go wrong.
                 * -- so that noone writes to disk after we do atomic copy of data.
                 */
                PRINTK("Syncing disks before copy\n");
                do_suspend_sync();

                /* Save state of all device drivers, and stop them. */
                if(drivers_suspend()==0)
                        /* If stopping device drivers worked, we proceed basically into
                         * suspend_save_image.
                         *
                         * do_magic(0) returns after system is resumed.
                         *
                         * do_magic() copies all "used" memory to "free" memory, then
                         * unsuspends all device drivers, and writes memory to disk
                         * using normal kernel mechanism.
                         */
                        do_magic(0);
                PRINTK("Restarting processes...\n");
                thaw_processes();
        }
        software_suspend_enabled = 1;
        MDELAY(1000);
        restore_console ();
}

(Most comments were added 5 minutes ago).

Does it look understendable now?
									Pavel
-- 
Worst form of spam? Adding advertisment signatures ala sourceforge.net.
What goes next? Inserting advertisment *into* email?


-------------------------------------------------------
This sf.net email is sponsored by: Dice - The leading online job board
for high-tech professionals. Search and apply for tech jobs today!
http://seeker.dice.com/seeker.epl?rel_code=31

Re: suspend.c vs driver-model.txt

[Benjamin Herrenschmidt]

>
>(Most comments were added 5 minutes ago).
>
>Does it look understendable now?

Yes, definitely more clear, there are probably interesting bits
in there to re-use for suspend-to-ram as well.
However, the mucking with kernel threads is definitely specific
to your need of keeping the target block device up without getting
nasty filesystem-originated requests.

Actually, I'm considering we could probably do both :) That is
suspend-to-disk is actually a superset of suspend-to-RAM. So
we could suspend-to-RAM-and-disk, basically allowing us to
wakeup fast from RAM, but still be able to wakeup from disk
if battery is exhausted.

Anyway, in your scheme, the main thing I'm currently concerned
about is the free_some_memory/do_suspend_sync/driver_suspend
thing which is nasty. How do you deal with memory beeing
re-allocated after free_some_memory ? (typically driver needing
RAM for saving state or a non-filesystem-related daemon doing
a kmalloc (or a driver from a tasklet). Also, do you push
as much as possible to swap ? Or may such an allocation still
cause more swap activity ? This can be nasty especially since
swap here can be an mmap'd file, or a swap file, thus possibly
racing with the fact you stopped filesystem specific daemons.

Then, finally, the drivers_suspend() state must be broken in
pieces, that is needed for both suspend-to-RAM and -to-disk,
(see my other email).

Ben.




-------------------------------------------------------
This sf.net email is sponsored by: Dice - The leading online job board
for high-tech professionals. Search and apply for tech jobs today!
http://seeker.dice.com/seeker.epl?rel_code=31

Re: suspend.c vs driver-model.txt

[Pavel Machek]

Hi!

> >(Most comments were added 5 minutes ago).
> >
> >Does it look understendable now?
> 
> Yes, definitely more clear, there are probably interesting bits
> in there to re-use for suspend-to-ram as well.
> However, the mucking with kernel threads is definitely specific
> to your need of keeping the target block device up without getting
> nasty filesystem-originated requests.

Stopping processes is needed (and being used) by suspend-to-ram,
because anything else is too hard for the drivers (I believe).

> Actually, I'm considering we could probably do both :) That is
> suspend-to-disk is actually a superset of suspend-to-RAM. So
> we could suspend-to-RAM-and-disk, basically allowing us to
> wakeup fast from RAM, but still be able to wakeup from disk
> if battery is exhausted.

Good idea. Thinkpads could do that, it was called RediSafe or
something like that. Disadvantage is that suspend then takes quite
long time.

> Anyway, in your scheme, the main thing I'm currently concerned
> about is the free_some_memory/do_suspend_sync/driver_suspend
> thing which is nasty. How do you deal with memory beeing
> re-allocated after free_some_memory ? (typically driver needing
> RAM for saving state or a non-filesystem-related daemon doing
> a kmalloc (or a driver from a tasklet). 

I free as much memory as possible. So, if driver allocation fails, it
would fail on running system, too. If there was anything that could be
swapped out, I'd already swapped it out.

> Also, do you push
> as much as possible to swap ? 

Yes.

> Or may such an allocation still
> cause more swap activity ? 

No.

> This can be nasty especially since
> swap here can be an mmap'd file, or a swap file, thus possibly
> racing with the fact you stopped filesystem specific daemons.

swsusp actually suspends right into swap *partition*. So I indeed need
swap partition and not swap file.

> Then, finally, the drivers_suspend() state must be broken in
> pieces, that is needed for both suspend-to-RAM and -to-disk,
> (see my other email).

Well, if you are willing to live with ugly-but-harmless
spindown-then-spinup and blank-then-unblank, its actually possible to
keep current system.
								Pavel
-- 
Worst form of spam? Adding advertisment signatures ala sourceforge.net.
What goes next? Inserting advertisment *into* email?


-------------------------------------------------------
This sf.net email is sponsored by: Dice - The leading online job board
for high-tech professionals. Search and apply for tech jobs today!
http://seeker.dice.com/seeker.epl?rel_code=31

Re: suspend.c vs driver-model.txt

[Benjamin Herrenschmidt]

>Stopping processes is needed (and being used) by suspend-to-ram,
>because anything else is too hard for the drivers (I believe).

I don't agree, and I have this working fairly well on pmac's today,
though that is a limited set of drivers used on these laptops I had
to fix.

>> Actually, I'm considering we could probably do both :) That is
>> suspend-to-disk is actually a superset of suspend-to-RAM. So
>> we could suspend-to-RAM-and-disk, basically allowing us to
>> wakeup fast from RAM, but still be able to wakeup from disk
>> if battery is exhausted.
>
>Good idea. Thinkpads could do that, it was called RediSafe or
>something like that. Disadvantage is that suspend then takes quite
>long time.

Well, if done to a swap partition, with the average throughput of
recent U/DMA disks on things like my laptop, that is about 14Mb/sec,
suspending would indeed take about 20 seconds, which is long, but
still realistic.

>> Anyway, in your scheme, the main thing I'm currently concerned
>> about is the free_some_memory/do_suspend_sync/driver_suspend
>> thing which is nasty. How do you deal with memory beeing
>> re-allocated after free_some_memory ? (typically driver needing
>> RAM for saving state or a non-filesystem-related daemon doing
>> a kmalloc (or a driver from a tasklet). 
>
>I free as much memory as possible. So, if driver allocation fails, it
>would fail on running system, too. If there was anything that could be
>swapped out, I'd already swapped it out.
>
>> Also, do you push
>> as much as possible to swap ? 
>
>Yes.
>
>> Or may such an allocation still
>> cause more swap activity ? 
>
>No.
>
>> This can be nasty especially since
>> swap here can be an mmap'd file, or a swap file, thus possibly
>> racing with the fact you stopped filesystem specific daemons.
>
>swsusp actually suspends right into swap *partition*. So I indeed need
>swap partition and not swap file.
>
>> Then, finally, the drivers_suspend() state must be broken in
>> pieces, that is needed for both suspend-to-RAM and -to-disk,
>> (see my other email).
>
>Well, if you are willing to live with ugly-but-harmless
>spindown-then-spinup and blank-then-unblank, its actually possible to
>keep current system.

Well... why not improve things then ? We are trying to get the proper
semantics of the various suspend steps thrown at drivers and then drivers
fixed, I'd rather see swsusp make use of this as well :)

Ben.




-------------------------------------------------------
This sf.net email is sponsored by: Dice - The leading online job board
for high-tech professionals. Search and apply for tech jobs today!
http://seeker.dice.com/seeker.epl?rel_code=31

Re: suspend.c vs driver-model.txt

[Patrick Mochel]

Sorry about the delay in responding...

> >The platform drivers need to then do whatever device transitions they 
> >need, using the device_suspend() helpers, then put the system to sleep 
> >using their platform-speciifc method.
> 
> Good ! I've had my home-made bunch of stuff on pmac that looked just
> like a platform driver dealing with all sort of things (basically
> abtracting motherboard IOs to help drivers that need to toggle whatever
> reset pin or power their device up/down physically).
> (See include/asm-ppc/pmac_feature.c and arch/ppc/kernel/pmac_feature.c)
> 
> On PPC, we also have that _huge_ flat ppc_md structure that contains
> pointers to functions like platform init, cpuinfo hooks, get/set_rtc_time,
> irq hooks, etc... that get filled depending on the type of machine a
> given kernel is booted (we support currently prep, chrp and pmac with
> a single kernel binary).
> (see include/asm-ppc/machdep.h)
> 
> What about generalizing the platform driver to do just those things ?
> 
> I'm not too sure about how the API would look like though. For the
> ppc_md stuff, there is a fixed, well defined, but still arch specific
> set of hooks to fill.
> 
> For the pmac_feature, I use a single ppc_md. hook with variable arg
> list (though I do use it in a well defined format) to call a list
> of "feature" functions that is usually exetended on every new kind
> of motherboard I add support for. Not every machine support all
> functions, etc... it would probably be a pain to define this as a
> flat list of function pointers.
> 
> Anybody has better ideas on how to deal with those platform specific
> things while still making them fit in a platform driver (their logical
> place, especially if the platform driver ends up in driverfs, a nice
> way to expose some motherboard specific settings/tweaks to userland ?

That's what the platform_notify() function pointer is for - to have the 
platform/firmware driver set device-specific information when it's 
registered. How you want to implement is up to you, and unfortunately 
there are no examples (yet). 

When you do get the call, you can simply create driverfs files with common 
handlers for the platform-specific attributes. How you map the device 
pointer that you get to the data is left as an exercise to the reader. ;)

> I don't know ACPI well enough to talk, but the mecanism I use on PPC
> is pretty basic today. The sleep procedure is dealt with by the driver
> of the Power Management chip when it gets a sleep request from userland,
> but that can (should be) moved to the platform driver.
> 
> Then, it does my own version of looping all devices sending them
> suspend events etc... (did we ever write down what we defined during
> OLS for those events semantics ?)

Yes. I'll try and get to them this week. (Operative word: try).

> This end up breaking the scenario into 2 main phases
> 
>  1) Looping around all drivers doing the suspend dance (suspend request,
> save_state & suspend, ...) as we discussed. This is mostly (completely ?)
> platform independant (except for the exact state a given device has to
> be put to, oh well, let's deal that inside the device driver for now)
> 
>  2) Doing whatever additional arch specific code is needed to snooze
> the machine.
> 
> 1) is generic. 2) is arch/platform code.

Yes, that's basically what we discussed, and the right way to go. We have
helper functions that walk the device list, doing what ever
platform-speciifc stuff in the meantime. (Similar to how ACPI does it 
today). 

There are just other details, like actually getting it right, and having 
the transition functions get called from a common place. And, making sure 
it looks the same to userspace. 

> As far as suspend to disk is concerned, we have all sort of nasty issues
> we discussed that I don't feel like resuming right now, so I would rather
> have the lowest level driver able to play the backing store export a
> couple of "hooks" used to dump the pages to the medium, those would work
> synchronously, without VM activity, etc... The driver that is target for
> that would have pre-allocated what it needs, pre-mapped DMA mappings it
> needs etc...
> Though, thinking as I write, injecting requests via the BIO layer may
> be a nicer solution, provided we make sure the driver won't get any
> request _but_ the ones we feed it. I don't trust a mecanism that would
> selectively stop given kernel threads, then just hope nobody will push
> requests any more. That's way too fragile to my taste. I'd rather freeze
> the request queue for real like any other device, then eventually provide
> a special pass-through.

I'm not going to comment, because I don't know the stuff well enough. 

> I'm probably to blame for not writing code here as I mostly only
> contributed with speetch and ideas (and my old pmac code) so far.
> I'm sorry, i'm not even been able to find time to convert the pmac
> drivers to driverfs yet :( My time dedicated to linux is spare, and
> so variable, and it tends to be quite small nowadays.

No need to apologize. I cc'ed you because you've always been incredibly 
helpful in keeping me honest. As much as I think you should quit your job 
and dedicate your life to getting Linux power management right, I realize 
that I'm being a bit unrealistic, and I have to life with the intermittent 
code. 

	-pat



-------------------------------------------------------
This sf.net email is sponsored by: Dice - The leading online job board
for high-tech professionals. Search and apply for tech jobs today!
http://seeker.dice.com/seeker.epl?rel_code=31

Re: suspend.c vs driver-model.txt

[Lyle]

> On a side, though related, note, while we were in Ottawa, several of us 
> had a chance to sit down and hammer out exactly what needs to take place. 
> The results exist in a notebook (a real, pencil-driven one), but have not 
> had a chance to be expressed in ASCII text yet. (Frankly, it's a low 
> priority right now: -EPLATEFULL). 

Did you talk at all about the fact that pci.h has its own set of power-management hooks that almost-but-not-quite map to the ACPI
ones, and how to resolve that?




-------------------------------------------------------
This sf.net email is sponsored by:ThinkGeek
Welcome to geek heaven.
http://thinkgeek.com/sf

suspend.c vs driver-model.txt

[Lyle]

The more I look at this, the more convinced I am that driver-model.txt 
describes the proper approach, and suspend.c just has it backwards.  Even 
though no drivers really implement the functionality properly yet, it makes 
sense to get the framework right so that they *can*, eventually.

I don't know why driver.h has the enums listed in a different order from 
driver-model.txt, or if it's really necessary to fix the order, unless 
somebody decides to start doing ordered comparisons.

Just to make it clear what I mean:
			 
*** kernel/suspend.c   Wed Jul 24 09:21:39 2002
--- kernel/suspend.c.orig      Wed Jul 24 09:20:32 2002
***************
*** 647,650 ****
  {
-       device_resume(RESUME_ENABLE);
        device_resume(RESUME_RESTORE_STATE);
  }
--- 647,650 ----
  {
        device_resume(RESUME_RESTORE_STATE);
+       device_resume(RESUME_ENABLE);
  }
***************
*** 655,658 ****
        device_suspend(4, SUSPEND_NOTIFY);
-       device_suspend(4, SUSPEND_DISABLE);
        device_suspend(4, SUSPEND_SAVE_STATE);
        if(!pm_suspend_state) {
--- 655,658 ----
        device_suspend(4, SUSPEND_NOTIFY);
        device_suspend(4, SUSPEND_SAVE_STATE);
+       device_suspend(4, SUSPEND_DISABLE);
        if(!pm_suspend_state) {
***************
*** 674,677 ****
  {
-       device_resume(RESUME_ENABLE);
        device_resume(RESUME_RESTORE_STATE);
        if(flags & RESUME_PHASE2) {
--- 674,677 ----
  {
        device_resume(RESUME_RESTORE_STATE);
+       device_resume(RESUME_ENABLE);
        if(flags & RESUME_PHASE2) {



-------------------------------------------------------
This sf.net email is sponsored by:ThinkGeek
Welcome to geek heaven.
http://thinkgeek.com/sf