ksoftirqd priority inversion wait_event_interruptible_timeout/schedule_timeout/del_timer_sync

ksoftirqd priority inversion wait_event_interruptible_timeout/schedule_timeout/del_timer_sync

[Kegl Rohit]

Hello!

Using 5.10.184-rt90 on arm (nxp imx6q).

A proprietary driver starts a `kthread` with `SCHED_FIFO` -98 priority.
This `kthread` waits for a completion using
`wait_event_interruptible_timeout(comp, 1000)`
and wakes up every second to perform its work. The completion itself
only fires very infrequently.

Tracing the kernel showed that `ksoftirqd` was periodically boosted to
the same `SCHED_FIFO` -98 priority.
This is caused by the `schedule_timeout` handler within
`wait_event_interruptible_timeout`.

del_timer_wait_running takes spin_lock_bh and causes the priority inversion.
This is very bad because e.g. network softirq processing happens with
very high priority for some time.

Was this issue addressed in a more recent kernel version?

----------------------------------

static long __sched
wait_for_common(struct completion *x, long timeout, int state)
{
return __wait_for_common(x, schedule_timeout, timeout, state);
}

----------------------------------

signed long __sched schedule_timeout(signed long timeout)
{
....

schedule();
del_singleshot_timer_sync(&timer.timer);
...
}

----------------------------------

#define del_singleshot_timer_sync(t) del_timer_sync(t)

----------------------------------

int del_timer_sync(struct timer_list *timer)
{
...
do {
ret = try_to_del_timer_sync(timer);

if (unlikely(ret < 0)) {
del_timer_wait_running(timer);
cpu_relax();
}
} while (ret < 0);
...
}

----------------------------------

/*
 * This function is called on PREEMPT_RT kernels when the fast path
 * deletion of a timer failed because the timer callback function was
 * running.
 *
 * This prevents priority inversion, if the softirq thread on a remote CPU
 * got preempted, and it prevents a life lock when the task which tries to
 * delete a timer preempted the softirq thread running the timer callback
 * function.
 */
static void del_timer_wait_running(struct timer_list *timer)
{
u32 tf;

tf = READ_ONCE(timer->flags);
if (!(tf & (TIMER_MIGRATING | TIMER_IRQSAFE))) {
struct timer_base *base = get_timer_base(tf);

/*
* Mark the base as contended and grab the expiry lock,
* which is held by the softirq across the timer
* callback. Drop the lock immediately so the softirq can
* expire the next timer. In theory the timer could already
* be running again, but that's more than unlikely and just
* causes another wait loop.
*/
atomic_inc(&base->timer_waiters);
spin_lock_bh(&base->expiry_lock);
atomic_dec(&base->timer_waiters);
spin_unlock_bh(&base->expiry_lock);
}
}

Thanks in advance.

Re: ksoftirqd priority inversion wait_event_interruptible_timeout/schedule_timeout/del_timer_sync

[Sebastian Andrzej Siewior]

On 2025-06-17 09:52:07 [+0200], Kegl Rohit wrote:
> Hello!
Hi,

> Using 5.10.184-rt90 on arm (nxp imx6q).
> 
> A proprietary driver starts a `kthread` with `SCHED_FIFO` -98 priority.
> This `kthread` waits for a completion using
> `wait_event_interruptible_timeout(comp, 1000)`
> and wakes up every second to perform its work. The completion itself
> only fires very infrequently.
> 
> Tracing the kernel showed that `ksoftirqd` was periodically boosted to
> the same `SCHED_FIFO` -98 priority.
> This is caused by the `schedule_timeout` handler within
> `wait_event_interruptible_timeout`.
> 
> del_timer_wait_running takes spin_lock_bh and causes the priority inversion.
> This is very bad because e.g. network softirq processing happens with
> very high priority for some time.
> 
> Was this issue addressed in a more recent kernel version?

The timer related bits moved to ktimersd thread (instead of ksoftirqd)
but the basic boosting problem remains.

There is work in progress to address this. The endgame is:
	https://lore.kernel.org/linux-rt-devel/20250613105653.1860729-1-bigeasy@linutronix.de/

Sebastian

Re: ksoftirqd priority inversion wait_event_interruptible_timeout/schedule_timeout/del_timer_sync

[Kegl Rohit]

thanks for your answer.
yes, I am aware of the ktimer thread split. In some kernel versions it
is split in and in some not.
In 5.10-rt the timers use ksoftirqd. Does a ktimer split backport
exist? Or do you think a backport would be more harmful than helpful?

On Mon, Jun 23, 2025 at 4:30 PM Sebastian Andrzej Siewior
<bigeasy@linutronix.de> wrote:
>
> On 2025-06-17 09:52:07 [+0200], Kegl Rohit wrote:
> > Hello!
> Hi,
>
> > Using 5.10.184-rt90 on arm (nxp imx6q).
> >
> > A proprietary driver starts a `kthread` with `SCHED_FIFO` -98 priority.
> > This `kthread` waits for a completion using
> > `wait_event_interruptible_timeout(comp, 1000)`
> > and wakes up every second to perform its work. The completion itself
> > only fires very infrequently.
> >
> > Tracing the kernel showed that `ksoftirqd` was periodically boosted to
> > the same `SCHED_FIFO` -98 priority.
> > This is caused by the `schedule_timeout` handler within
> > `wait_event_interruptible_timeout`.
> >
> > del_timer_wait_running takes spin_lock_bh and causes the priority inversion.
> > This is very bad because e.g. network softirq processing happens with
> > very high priority for some time.
> >
> > Was this issue addressed in a more recent kernel version?
>
> The timer related bits moved to ktimersd thread (instead of ksoftirqd)
> but the basic boosting problem remains.
>
> There is work in progress to address this. The endgame is:
>         https://lore.kernel.org/linux-rt-devel/20250613105653.1860729-1-bigeasy@linutronix.de/
>
> Sebastian