Undesirable priority inversion for real-time threads that use timerfd

Undesirable priority inversion for real-time threads that use timerfd

[Joachim Achtzehnter]

We find that real-time threads running at priorities greater than 1, 
which use the timerfd abstraction in order to simultaneously wait for 
either a timeout or another event that is reported via a file descriptor 
experience undesirable delays for the timeouts when lower-priority 
real-time threads threads are running.

This happens because the notification about timeouts depends on kernel 
threads with names like ktimersoftd/N, where N is an index for a CPU 
core on the system. These threads are scheduled with real-time priority 1.

As an obvious fix for the priority inversion we can raise the priority 
of the ktimersoftd/N kernel threads to a priority that is at least as 
high as our highest-priority thread that uses timers.

The only downside of this fix is that all timeouts used by low-priority 
threads, presumably even non-real-time threads, will briefly preempt our 
high-priority threads, although hopefully not for long, hence there may 
be some impact on throughput.

A possibly more optimal approach would be to have a separate ktimersoftd 
thread for every priority at which threads wait for timer notifications. 
We could potentially build such a facility in user code, as long as 
there is some timer available that is not dependent on ktimersoftd. What 
about Posix timers? Would they have the same issue? Anything else we 
should be looking at?

Thanks,

Joachim

Re: Undesirable priority inversion for real-time threads that use timerfd

[Sebastian Andrzej Siewior]

On 2016-09-02 17:25:55 [-0700], Joachim Achtzehnter wrote:
> The only downside of this fix is that all timeouts used by low-priority
> threads, presumably even non-real-time threads, will briefly preempt our
> high-priority threads, although hopefully not for long, hence there may be
> some impact on throughput.

_all_ timers which are not marked as irq_safe are deferred here to the
timersoftirq. This is because they take (possible long standing)
sleeping locks.

> A possibly more optimal approach would be to have a separate ktimersoftd
> thread for every priority at which threads wait for timer notifications. We
you don't know this in advance. You are in hardirq context and a hrtimer
expires. You invoke the timerfd callback, need to take a sleeping lock
to figure out who is waiting on this. *That* is the point you learn
about the waiter.

> could potentially build such a facility in user code, as long as there is
> some timer available that is not dependent on ktimersoftd. What about Posix
> timers? Would they have the same issue? Anything else we should be looking
> at?

posix timers have the same issue. Basically all timers which are not
marked as irq_safe are completed in ktimersoftd. If you look at
clock_nanosleep() you will see that it is marked as irqsafe and the wake
up happens in hardirq.

Btw: the situation got better once we got ktimersoftd. Before that it
was a little worse.

> Thanks,
> 
> Joachim

Sebastian