How to unblock a thread from a hard isr (IRQF_NO_THREAD)

How to unblock a thread from a hard isr (IRQF_NO_THREAD)

[Till Straumann]

Hello list.

I have a driver which is designed to do most work in user-space.
The ISR is really simple. It just ACKs/clears the interrupt and
then should unblock a waiting thread.
It seems overkill (and costs me about 5-10us) to use a threaded
interrupt in such a case (hard isr unblocks a kthread whose only
'work' is unblocking another (user) thread).

I have found some references where users try to do a similar
thing via UIO
(e.g.,
http://article.gmane.org/gmane.linux.rt.user/7372/
http://article.gmane.org/gmane.linux.rt.user/7676)

However, the succinct answer was along the line 'that can't work with UIO'.

IMHO it would be very helpful to get a more detailed explanation
as to why it doesn't work (i.e., what things you are and are not
allowed to do from a hard-isr under RT_PREEMPT). I suspect it is
because UIO calls routines such as 'kill_fasync' which use
ordinary (as opposed to raw) spinlocks which means that the
caller could be preempted (under RT_PREEMPT), right?

What *is* the recommended mechanism to wake up a thread from
a hard-isr?

It would IMHO be desirable to modify the UIO driver so that it
is possible to support non-threaded handlers.

Thanks for any enlightenment (and the fantastic rt-preempt patch
in the first place)!

- Till

PS: Please CC me on any replies; I am currently not a subscriber.

Re: How to unblock a thread from a hard isr (IRQF_NO_THREAD)

[Tim Sander]

Hi Till
> What *is* the recommended mechanism to wake up a thread from
> a hard-isr?
I think its the waitqueue mechanism with the drawbacks you mentioned.

If you only want to wake up *one* usermode thread there is a hack
possible to save the context switch times to the bottom half interrupt thread?

Best regards
Tim

Re: How to unblock a thread from a hard isr (IRQF_NO_THREAD)

[Till Straumann]

On 11/08/2012 05:12 PM, Tim Sander wrote:
> Hi Till
>> What *is* the recommended mechanism to wake up a thread from
>> a hard-isr?
> I think its the waitqueue mechanism with the drawbacks you mentioned.
I thought I can't wake_up() from a hard-isr so that can hardly be
the recommended mechanism (unless you mean I *should* go
via the 'irq_thread' (kernel/irq/management.c) and wake my
user-land thread from there).

The IRQ management code must use something to let the
hard-isr unblock the irq-handler kthread - I'd like to use the same
(or a similar) method to let the hard-isr unblock a user-land
thread.
>
> If you only want to wake up *one* usermode thread there is a hack
> possible to save the context switch times to the bottom half interrupt thread?
What would such a hack be?

Thanks
- Till
>
> Best regards
> Tim

Re: How to unblock a thread from a hard isr (IRQF_NO_THREAD)

[Josh Cartwright]

On Thu, Nov 08, 2012 at 04:28:48PM -0600, Till Straumann wrote:
> Hello list.
>
> I have a driver which is designed to do most work in user-space.
> The ISR is really simple. It just ACKs/clears the interrupt and
> then should unblock a waiting thread.
> It seems overkill (and costs me about 5-10us) to use a threaded
> interrupt in such a case (hard isr unblocks a kthread whose only
> 'work' is unblocking another (user) thread).
>
> I have found some references where users try to do a similar
> thing via UIO
> (e.g.,
> http://article.gmane.org/gmane.linux.rt.user/7372/
> http://article.gmane.org/gmane.linux.rt.user/7676)
>
> However, the succinct answer was along the line 'that can't work with UIO'.
>
> IMHO it would be very helpful to get a more detailed explanation
> as to why it doesn't work (i.e., what things you are and are not
> allowed to do from a hard-isr under RT_PREEMPT). I suspect it is
> because UIO calls routines such as 'kill_fasync' which use
> ordinary (as opposed to raw) spinlocks which means that the
> caller could be preempted (under RT_PREEMPT), right?

Correct.  Not just kill_fasync(), but the use of
wake_up_interruptible() as well (and maybe others).

> What *is* the recommended mechanism to wake up a thread from
> a hard-isr?

I'd suggest you look at the hrtimer_sleeper code for an example of
process wakeup done in hardirq context.  (hint, the low-level answer to
your question is wake_up_process()).

   Josh

Re: How to unblock a thread from a hard isr (IRQF_NO_THREAD)

[Till Straumann]

Thanks, that's helpful.

Wouldn't it also be possible to make my own version of
prepare_to_wait()/finish_wait()/wake_up() etc. which internally
use a raw spinlock instead of a normal one?

- Till

On 11/08/2012 10:08 PM, Josh Cartwright wrote:
> On Thu, Nov 08, 2012 at 04:28:48PM -0600, Till Straumann wrote:
>> Hello list.
>>
>> I have a driver which is designed to do most work in user-space.
>> The ISR is really simple. It just ACKs/clears the interrupt and
>> then should unblock a waiting thread.
>> It seems overkill (and costs me about 5-10us) to use a threaded
>> interrupt in such a case (hard isr unblocks a kthread whose only
>> 'work' is unblocking another (user) thread).
>>
>> I have found some references where users try to do a similar
>> thing via UIO
>> (e.g.,
>> http://article.gmane.org/gmane.linux.rt.user/7372/
>> http://article.gmane.org/gmane.linux.rt.user/7676)
>>
>> However, the succinct answer was along the line 'that can't work with UIO'.
>>
>> IMHO it would be very helpful to get a more detailed explanation
>> as to why it doesn't work (i.e., what things you are and are not
>> allowed to do from a hard-isr under RT_PREEMPT). I suspect it is
>> because UIO calls routines such as 'kill_fasync' which use
>> ordinary (as opposed to raw) spinlocks which means that the
>> caller could be preempted (under RT_PREEMPT), right?
> Correct.  Not just kill_fasync(), but the use of
> wake_up_interruptible() as well (and maybe others).
>
>> What *is* the recommended mechanism to wake up a thread from
>> a hard-isr?
> I'd suggest you look at the hrtimer_sleeper code for an example of
> process wakeup done in hardirq context.  (hint, the low-level answer to
> your question is wake_up_process()).
>
>     Josh

Re: How to unblock a thread from a hard isr (IRQF_NO_THREAD)

[Josh Cartwright]

On Thu, Nov 08, 2012 at 10:28:53PM -0600, Till Straumann wrote:
> Thanks, that's helpful.
> 
> Wouldn't it also be possible to make my own version of
> prepare_to_wait()/finish_wait()/wake_up() etc. which internally
> use a raw spinlock instead of a normal one?

I believe that what you described used to exist in the form of a
'simple' waitqueue, but I don't recall what happened to it.  Maybe
Thomas can shed some light here.

In general though, if you care about determinism, you'd want to avoid
the use of such a mechanism unless you can somehow externally provide an
upper bound on the number of waiters.

  Josh

Re: How to unblock a thread from a hard isr (IRQF_NO_THREAD)

[Till Straumann]

On 11/08/2012 10:42 PM, Josh Cartwright wrote:
> On Thu, Nov 08, 2012 at 10:28:53PM -0600, Till Straumann wrote:
>> Thanks, that's helpful.
>>
>> Wouldn't it also be possible to make my own version of
>> prepare_to_wait()/finish_wait()/wake_up() etc. which internally
>> use a raw spinlock instead of a normal one?
> I believe that what you described used to exist in the form of a
> 'simple' waitqueue, but I don't recall what happened to it.  Maybe
> Thomas can shed some light here.
>
> In general though, if you care about determinism, you'd want to avoid
> the use of such a mechanism unless you can somehow externally provide an
> upper bound on the number of waiters.
>
>    Josh
Of course. However, for the type of synchronization device we are talking
about here (hardirq wakes up a user-land worker) a (hardirq-safe) 
completion-like
device supporting a single waiter only would be good enough.

- Till

Re: How to unblock a thread from a hard isr (IRQF_NO_THREAD)

[Till Straumann]

Hi Tim.

Thanks very much for the example. Being paranoid, it raises
one more question, though:

You say

     priv->wakeupTask=current;
     set_current_state(TASK_INTERRUPTIBLE);
     schedule();

but is it not necessary to take a raw_spin_lock_irqsave()
around storing the task pointer and changing the task
state? To be more precise: around starting the device
and setting the task state. I would have thought I would
have to code like this:

      priv->wakeupTask = current;
      raw_spin_lock_irqsave(...)
        start_device();
        /* What happens if a fast device would interrupt here
         * without the raw spinlock? Would I not sleep forever?
         */
        set_current_state(TASK_INTERRUPTIBLE);
      raw_spin_unlock_irqrestore(...)
      schedule();

Assuming that an IRQ could happen any time once the device
is started - don't I have to avoid giving the the hard-isr the
chance of executing wake_up_process() before
set_current_state(TASK_INTERRUPTIBLE) has finished?

Thanks again
- Till

On 11/09/2012 08:18 AM, Tim Sander wrote:
> Hi Till
>>>> What *is* the recommended mechanism to wake up a thread from
>>>> a hard-isr?
>>> I think its the waitqueue mechanism with the drawbacks you mentioned.
>> I thought I can't wake_up() from a hard-isr so that can hardly be
>> the recommended mechanism (unless you mean I *should* go
>> via the 'irq_thread' (kernel/irq/management.c) and wake my
>> user-land thread from there).
> For multiple waiters i don't know any other solution than the irq-thread
> mechanism with the additional delay.
>
>> The IRQ management code must use something to let the
>> hard-isr unblock the irq-handler kthread - I'd like to use the same
>> (or a similar) method to let the hard-isr unblock a user-land
>> thread.
>>
>>> If you only want to wake up *one* usermode thread there is a hack
>>> possible to save the context switch times to the bottom half interrupt
>>> thread?
>> What would such a hack be?
> Credits go to Thomas Gleixner for this hack (Keep in mind it only works
> for a *single* waiter, so your driver should probably only allow exclusive
> opens of this device.)
> Define a structure:
>
>      struct task_struct *wakeupTask;
>
> In the read routine which should wait for the interrupt pulse:
> 	priv->wakeupTask=current;
>      set_current_state(TASK_INTERRUPTIBLE);
>      schedule();
> 	//i'll be back :-)
> 	retval = copy_to_user(...);
> 	__set_current_state(TASK_RUNNING);
>
> And in isr context just do a:
> 	wake_up_process(current_instance->wakeupTask);
>
> Best regards
> Tim