Helping drivers to work without the freezer

Helping drivers to work without the freezer

[Alan Stern]

This idea just occurred to me.  It may or may not end up being useful, 
and I don't have any specific applications in mind yet.

System sleeps are supposed to be transparent to userspace.  An I/O 
request submitted just before the sleep starts should be blocked until 
after the system wakes up.

Without the freezer to do this blocking for them, drivers will have to 
do it themselves.  So when a driver is about to carry out an I/O 
operation, it has to insure that the operation is mutually exclusive 
with sleeping.

One way to do this is by a private mutex, which would protect both the
block of code doing the I/O and the suspend routine.  One for each
device; that's a lot of new mutexes.

My idea is instead to have the PM core provide a new pair of routines
for use by drivers.  Something like "thread_not_sleepable()" and 
"thread_sleepable()".  

The first routine would be called by a driver before starting to do
I/O, while no locks are held.  If a sleep transition had already
started, the routine would block until the sleep was over.  Otherwise,
the thread would be marked with a NOT_SLEEPABLE flag until the second
routine was called.  When the PM core wanted to start a system sleep
it would have to check whether any threads were marked NOT_SLEEPABLE,
and wait until none of them were.

This could make drivers a little simpler.  It would mean less code to
modify, and it would remove one entry from the messy I/O vs. unbind vs.
suspend synchronization problem.

Comments?

Alan Stern

Re: Helping drivers to work without the freezer

[Rafael J. Wysocki]

On Saturday, 8 of March 2008, Alan Stern wrote:
> This idea just occurred to me.  It may or may not end up being useful, 
> and I don't have any specific applications in mind yet.
> 
> System sleeps are supposed to be transparent to userspace.  An I/O 
> request submitted just before the sleep starts should be blocked until 
> after the system wakes up.
> 
> Without the freezer to do this blocking for them, drivers will have to 
> do it themselves.  So when a driver is about to carry out an I/O 
> operation, it has to insure that the operation is mutually exclusive 
> with sleeping.
> 
> One way to do this is by a private mutex, which would protect both the
> block of code doing the I/O and the suspend routine.  One for each
> device; that's a lot of new mutexes.
> 
> My idea is instead to have the PM core provide a new pair of routines
> for use by drivers.  Something like "thread_not_sleepable()" and 
> "thread_sleepable()".  
> 
> The first routine would be called by a driver before starting to do
> I/O, while no locks are held.  If a sleep transition had already
> started, the routine would block until the sleep was over.  Otherwise,
> the thread would be marked with a NOT_SLEEPABLE flag until the second
> routine was called.  When the PM core wanted to start a system sleep
> it would have to check whether any threads were marked NOT_SLEEPABLE,
> and wait until none of them were.
> 
> This could make drivers a little simpler.  It would mean less code to
> modify, and it would remove one entry from the messy I/O vs. unbind vs.
> suspend synchronization problem.
> 
> Comments?

Well, this is what the current freezer does with respect to kernel threads,
only the name of the flag is different. ;-)

You basically need something very similar to the current freezer in order
to implement the "PM core would have to check whether any threads were marked
NOT_SLEEPABLE, and wait until none of them were" functionality.

Thanks,
Rafael

Re: Helping drivers to work without the freezer

[Alan Stern]

On Sat, 8 Mar 2008, Rafael J. Wysocki wrote:

> > My idea is instead to have the PM core provide a new pair of routines
> > for use by drivers.  Something like "thread_not_sleepable()" and 
> > "thread_sleepable()".  
> > 
> > The first routine would be called by a driver before starting to do
> > I/O, while no locks are held.  If a sleep transition had already
> > started, the routine would block until the sleep was over.  Otherwise,
> > the thread would be marked with a NOT_SLEEPABLE flag until the second
> > routine was called.  When the PM core wanted to start a system sleep
> > it would have to check whether any threads were marked NOT_SLEEPABLE,
> > and wait until none of them were.
> > 
> > This could make drivers a little simpler.  It would mean less code to
> > modify, and it would remove one entry from the messy I/O vs. unbind vs.
> > suspend synchronization problem.
> > 
> > Comments?
> 
> Well, this is what the current freezer does with respect to kernel threads,
> only the name of the flag is different. ;-)

They aren't exactly the same, although they certainly are similar.  The 
difference lies in what happens when a task calls set_freezable() 
after a system sleep has begun; its TIF_FREEZE flag doesn't immediately 
get set.

Also, the current freezer doesn't offer a clear_freezable() routine.

> You basically need something very similar to the current freezer in order
> to implement the "PM core would have to check whether any threads were marked
> NOT_SLEEPABLE, and wait until none of them were" functionality.

Another approach would be to use something like an rwsem.  Hopefully 
without all the cache-line-bouncing overhead on SMP systems.

Alan Stern

Re: Helping drivers to work without the freezer

[Rafael J. Wysocki]

On Saturday, 8 of March 2008, Alan Stern wrote:
> On Sat, 8 Mar 2008, Rafael J. Wysocki wrote:
> 
> > > My idea is instead to have the PM core provide a new pair of routines
> > > for use by drivers.  Something like "thread_not_sleepable()" and 
> > > "thread_sleepable()".  
> > > 
> > > The first routine would be called by a driver before starting to do
> > > I/O, while no locks are held.  If a sleep transition had already
> > > started, the routine would block until the sleep was over.  Otherwise,
> > > the thread would be marked with a NOT_SLEEPABLE flag until the second
> > > routine was called.  When the PM core wanted to start a system sleep
> > > it would have to check whether any threads were marked NOT_SLEEPABLE,
> > > and wait until none of them were.
> > > 
> > > This could make drivers a little simpler.  It would mean less code to
> > > modify, and it would remove one entry from the messy I/O vs. unbind vs.
> > > suspend synchronization problem.
> > > 
> > > Comments?
> > 
> > Well, this is what the current freezer does with respect to kernel threads,
> > only the name of the flag is different. ;-)
> 
> They aren't exactly the same, although they certainly are similar.  The 
> difference lies in what happens when a task calls set_freezable() 
> after a system sleep has begun; its TIF_FREEZE flag doesn't immediately 
> get set.

We can only wait for them at one point, however.  Periodic checking if there
are no unsleepable tasks around wouldn't be very practical, IMHO.

> Also, the current freezer doesn't offer a clear_freezable() routine.

Oh, it would be easy to add one. :-)

> > You basically need something very similar to the current freezer in order
> > to implement the "PM core would have to check whether any threads were marked
> > NOT_SLEEPABLE, and wait until none of them were" functionality.
> 
> Another approach would be to use something like an rwsem.  Hopefully 
> without all the cache-line-bouncing overhead on SMP systems.

Well, to me, rwsem sounds definitely better.  Still, I think it's better to
avoid locking it for too long, so we could use a variable protected by the
rwsem such that if it's 'true', unsleepable tasks checking it will put
themselves into a wait queue which will be woken up by the PM core
during resume.

Thanks,
Rafael

Re: Helping drivers to work without the freezer

[Alan Stern]

On Sun, 9 Mar 2008, Rafael J. Wysocki wrote:

> > > Well, this is what the current freezer does with respect to kernel threads,
> > > only the name of the flag is different. ;-)
> > 
> > They aren't exactly the same, although they certainly are similar.  The 
> > difference lies in what happens when a task calls set_freezable() 
> > after a system sleep has begun; its TIF_FREEZE flag doesn't immediately 
> > get set.
> 
> We can only wait for them at one point, however.  Periodic checking if there
> are no unsleepable tasks around wouldn't be very practical, IMHO.

I'm not talking about periodic checking.  The waiting is done at only 
one point, the beginning of a system sleep.  Once that check succeeds, 
we know that all unsleepable tasks will block.

("Unsleepable" isn't such a great word.  What I mean is tasks in a 
critical section that must run exclusively WRT system sleeps.)

> > Another approach would be to use something like an rwsem.  Hopefully 
> > without all the cache-line-bouncing overhead on SMP systems.
> 
> Well, to me, rwsem sounds definitely better.  Still, I think it's better to
> avoid locking it for too long, so we could use a variable protected by the
> rwsem such that if it's 'true', unsleepable tasks checking it will put
> themselves into a wait queue which will be woken up by the PM core
> during resume.

Isn't that exactly what an rwsem does?  When it is locked for writing
(your flag is "true"), tasks attempting to get a read lock will put 
themselves into a wait queue which will be woken up when the write lock 
is released (when the PM core sets the flag to "false" during resume).

In this case we could use something a little simpler than a
general-purpose rwsem, since there will never be more than one task
getting a write lock at any time -- i.e., never more than one task
carrying out a system sleep.  It's not hard to design such a simplified
rwsem in a way that exerts virtually no overhead on readers whenever no
writer is present.

Alan Stern

Re: Helping drivers to work without the freezer

[Pavel Machek]

Hi!

> > Well, to me, rwsem sounds definitely better.  Still, I think it's better to
> > avoid locking it for too long, so we could use a variable protected by the
> > rwsem such that if it's 'true', unsleepable tasks checking it will put
> > themselves into a wait queue which will be woken up by the PM core
> > during resume.
> 
> Isn't that exactly what an rwsem does?  When it is locked for writing
> (your flag is "true"), tasks attempting to get a read lock will put 
> themselves into a wait queue which will be woken up when the write lock 
> is released (when the PM core sets the flag to "false" during resume).
> 
> In this case we could use something a little simpler than a
> general-purpose rwsem, since there will never be more than one task
> getting a write lock at any time -- i.e., never more than one task
> carrying out a system sleep.  It's not hard to design such a simplified
> rwsem in a way that exerts virtually no overhead on readers whenever no
> writer is present.

Just use generic rwsem... if we can see performance impact, we can
optimize it later.

-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html