[PATCH 1/5] locking/rtmutex: Stop spinning on the owner when the waiter is woken

[PATCH 1/5] locking/rtmutex: Stop spinning on the owner when the waiter is woken

[Mathieu Desnoyers]

The top waiter of an rt_mutex spins while the owner runs,
rtmutex_spin_on_owner(), and does not stop when it is woken: its timeout
and a signal find it running, set it TASK_RUNNING, and the spin goes on
until the owner leaves its CPU or the waiter's own CPU has to
reschedule. A PI futex owner is a user task that may run as long as it
likes. On v7.4-rc1, in 8-CPU VMs, behind an owner spinning in userspace,
a 1ms pthread_mutex_timedlock() returns ETIMEDOUT 126ms to 631ms late on
average, 1.4s at most, a SIGUSR1 is handled up to 1.9s late, and a
waiter killed with SIGKILL is reaped 0.5s late on average. With
PREEMPT_RT the wound of a ww_mutex is missed the same way: two contexts
in an ABBA on two CPUs at SCHED_FIFO spin on each other, and the older
one gets its second lock after 96ms on average, 0.95s at most.

Stop spinning once the waiter is woken, where the sleep the spin stands
in for would end: rt_mutex_slowlock_block() then sees the timeout, the
signal or the wound, as after schedule(). An rtlock waiter is woken out
of TASK_RTLOCK_WAIT by the lock handover alone, and spins as before.

With this patch the 1ms waits end after 1.01ms on average, a signal is
handled within 0.3ms (1.4ms with PREEMPT_RT), and the ww_mutex ABBA
resolves in 12us to 33us; a contended PI mutex keeps its throughput.
The futex_lock_pi_inherit and test-ww_mutex cases added later in this
series catch both.

Fixes: 992caf7f1724 ("locking/rtmutex: Add adaptive spinwait mechanism")
Cc: stable@vger.kernel.org
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
---
 kernel/locking/rtmutex.c | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/kernel/locking/rtmutex.c b/kernel/locking/rtmutex.c
index 5a9534c715b8..323973be2074 100644
--- a/kernel/locking/rtmutex.c
+++ b/kernel/locking/rtmutex.c
@@ -1516,9 +1516,13 @@ static bool rtmutex_spin_on_owner(struct rt_mutex_base *lock,
 		 *  - current is not longer the top waiter
 		 *  - current is requested to reschedule (redundant
 		 *    for CONFIG_PREEMPT_RCU=y)
+		 *  - current is woken, by its timeout, a signal or
+		 *    whatever else would end the sleep the spin
+		 *    stands in for
 		 *  - the VCPU on which owner runs is preempted
 		 */
 		if (!owner_on_cpu(owner) || need_resched() ||
+		    task_is_running(current) ||
 		    !data_race(rt_mutex_waiter_is_top_waiter(lock, waiter))) {
 			res = false;
 			break;
-- 
2.47.3

Re: [PATCH 1/5] locking/rtmutex: Stop spinning on the owner when the waiter is woken

[Steven Rostedt]

On Fri,  6 Nov 2026 18:59:27 -0500
Mathieu Desnoyers <mathieu.desnoyers@efficios.com> wrote:

> The top waiter of an rt_mutex spins while the owner runs,
> rtmutex_spin_on_owner(), and does not stop when it is woken: its timeout
> and a signal find it running, set it TASK_RUNNING, and the spin goes on
> until the owner leaves its CPU or the waiter's own CPU has to
> reschedule. A PI futex owner is a user task that may run as long as it
> likes. On v7.4-rc1, in 8-CPU VMs, behind an owner spinning in userspace,
> a 1ms pthread_mutex_timedlock() returns ETIMEDOUT 126ms to 631ms late on
> average, 1.4s at most, a SIGUSR1 is handled up to 1.9s late, and a
> waiter killed with SIGKILL is reaped 0.5s late on average. With
> PREEMPT_RT the wound of a ww_mutex is missed the same way: two contexts
> in an ABBA on two CPUs at SCHED_FIFO spin on each other, and the older
> one gets its second lock after 96ms on average, 0.95s at most.
> 

The above is very verbose. Are you starting to use AI? ;-)

> Stop spinning once the waiter is woken, where the sleep the spin stands
> in for would end: rt_mutex_slowlock_block() then sees the timeout, the
> signal or the wound, as after schedule(). An rtlock waiter is woken out
> of TASK_RTLOCK_WAIT by the lock handover alone, and spins as before.

OK, because its state is set to TASK_RUNNING the schedule_rtlock() will
simply return.

> 
> With this patch the 1ms waits end after 1.01ms on average, a signal is
> handled within 0.3ms (1.4ms with PREEMPT_RT), and the ww_mutex ABBA
> resolves in 12us to 33us; a contended PI mutex keeps its throughput.
> The futex_lock_pi_inherit and test-ww_mutex cases added later in this
> series catch both.
> 
> Fixes: 992caf7f1724 ("locking/rtmutex: Add adaptive spinwait mechanism")
> Cc: stable@vger.kernel.org
> Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
> ---
>  kernel/locking/rtmutex.c | 4 ++++
>  1 file changed, 4 insertions(+)
> 
> diff --git a/kernel/locking/rtmutex.c b/kernel/locking/rtmutex.c
> index 5a9534c715b8..323973be2074 100644
> --- a/kernel/locking/rtmutex.c
> +++ b/kernel/locking/rtmutex.c
> @@ -1516,9 +1516,13 @@ static bool rtmutex_spin_on_owner(struct rt_mutex_base *lock,
>  		 *  - current is not longer the top waiter
>  		 *  - current is requested to reschedule (redundant
>  		 *    for CONFIG_PREEMPT_RCU=y)
> +		 *  - current is woken, by its timeout, a signal or
> +		 *    whatever else would end the sleep the spin
> +		 *    stands in for
>  		 *  - the VCPU on which owner runs is preempted
>  		 */
>  		if (!owner_on_cpu(owner) || need_resched() ||
> +		    task_is_running(current) ||

So the task enters here with its state non-running, even for
rt_spin_locks, and you break out of the spin if something woke it up.
Breaking out of the spin with the state as TASK_RUNNING just makes it
reanalyze the situation and for rt_spin_locks it will put itself back
to RTLOCK_WAIT and spin again, if the owner hasn't left the CPU.

Makes sense.

Reviewed-by: Steven Rostedt <rostedt@goodmis.org>

-- Steve


>  		    !data_race(rt_mutex_waiter_is_top_waiter(lock, waiter))) {
>  			res = false;
>  			break;

Re: [PATCH 1/5] locking/rtmutex: Stop spinning on the owner when the waiter is woken

[Peter Zijlstra]

On Sat, Nov 07, 2026 at 12:29:27AM -0500, Steven Rostedt wrote:
> On Fri,  6 Nov 2026 18:59:27 -0500
> Mathieu Desnoyers <mathieu.desnoyers@efficios.com> wrote:
> 
> > The top waiter of an rt_mutex spins while the owner runs,
> > rtmutex_spin_on_owner(), and does not stop when it is woken: its timeout
> > and a signal find it running, set it TASK_RUNNING, and the spin goes on
> > until the owner leaves its CPU or the waiter's own CPU has to
> > reschedule. A PI futex owner is a user task that may run as long as it
> > likes. On v7.4-rc1, in 8-CPU VMs, behind an owner spinning in userspace,
> > a 1ms pthread_mutex_timedlock() returns ETIMEDOUT 126ms to 631ms late on
> > average, 1.4s at most, a SIGUSR1 is handled up to 1.9s late, and a
> > waiter killed with SIGKILL is reaped 0.5s late on average. With
> > PREEMPT_RT the wound of a ww_mutex is missed the same way: two contexts
> > in an ABBA on two CPUs at SCHED_FIFO spin on each other, and the older
> > one gets its second lock after 96ms on average, 0.95s at most.
> > 
> 
> The above is very verbose. Are you starting to use AI? ;-)

He definitely is. I can't read the above, its word soup. Part of using
AI is making sure the Changelog is actually readable. This is not.

Re: [PATCH 1/5] locking/rtmutex: Stop spinning on the owner when the waiter is woken

[Mathieu Desnoyers]

On 2026-11-07 05:43, Peter Zijlstra wrote:
> On Sat, Nov 07, 2026 at 12:29:27AM -0500, Steven Rostedt wrote:
>> On Fri,  6 Nov 2026 18:59:27 -0500
>> Mathieu Desnoyers <mathieu.desnoyers@efficios.com> wrote:
>>
>>> The top waiter of an rt_mutex spins while the owner runs,
>>> rtmutex_spin_on_owner(), and does not stop when it is woken: its timeout
>>> and a signal find it running, set it TASK_RUNNING, and the spin goes on
>>> until the owner leaves its CPU or the waiter's own CPU has to
>>> reschedule. A PI futex owner is a user task that may run as long as it
>>> likes. On v7.4-rc1, in 8-CPU VMs, behind an owner spinning in userspace,
>>> a 1ms pthread_mutex_timedlock() returns ETIMEDOUT 126ms to 631ms late on
>>> average, 1.4s at most, a SIGUSR1 is handled up to 1.9s late, and a
>>> waiter killed with SIGKILL is reaped 0.5s late on average. With
>>> PREEMPT_RT the wound of a ww_mutex is missed the same way: two contexts
>>> in an ABBA on two CPUs at SCHED_FIFO spin on each other, and the older
>>> one gets its second lock after 96ms on average, 0.95s at most.
>>>
>>
>> The above is very verbose. Are you starting to use AI? ;-)

I definitely am, as stated in the cover letter. :-)

> 
> He definitely is. I can't read the above, its word soup. Part of using
> AI is making sure the Changelog is actually readable. This is not.

Feedback on the commit message wording is welcome, I'll try to improve
things for next iterations. Is the following easier to parse ? I fed
Peter's and my own criticism to Claude and got it to rewrite the entire
thing (and keep those learnings for next time):

locking/rtmutex: Stop spinning on the owner when the waiter is woken

The top waiter of an rt_mutex sets its sleep state. Then it spins in
rtmutex_spin_on_owner() as long as the owner is running. A wakeup sets
the waiter back to TASK_RUNNING. The spin loop does not check for this.
So a timeout or a signal is only handled when the owner blocks, or when
the waiter has to reschedule:

   CPU 0 (waiter)                            CPU 1 (owner)

                                             pthread_mutex_lock()
                                               takes the PI futex
                                             spins in userspace
   pthread_mutex_timedlock(1ms)
     futex_lock_pi()
       rt_mutex_wait_proxy_lock()
         set_current_state(TASK_INTERRUPTIBLE)
         rt_mutex_slowlock_block()
           rtmutex_spin_on_owner()
             <timeout> hrtimer_wakeup()
               wake_up_process(waiter)
                 __state = TASK_RUNNING
             owner_on_cpu(owner): true
             need_resched(): false
             keeps spinning                  still spins in userspace

The owner of a PI futex is a user task. It can run for a long time.
With an owner spinning in userspace, a 1ms pthread_mutex_timedlock()
returns up to 1.4s late. A signal to the waiter is delayed in the same
way.

With PREEMPT_RT, a ww_mutex wound is missed too. Two contexts take two
ww_mutexes in opposite order, on two CPUs. They spin on each other for
up to a second.

Fix this by also stopping the spin when the waiter is TASK_RUNNING. The
slowpath then handles the timeout, the signal or the wound, as it does
after schedule().

rtlock waiters are not affected. Only the lock handoff wakes them from
TASK_RTLOCK_WAIT.

With this patch, the 1ms timed lock returns after about 1ms. The
ww_mutex ABBA resolves in microseconds. PI mutex throughput is
unchanged. The futex and ww_mutex selftests added later in this series
cover both cases.

Fixes: 992caf7f1724 ("locking/rtmutex: Add adaptive spinwait mechanism")
Cc: stable@vger.kernel.org
Reviewed-by: Steven Rostedt <rostedt@goodmis.org>
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>

-- 
Mathieu Desnoyers
EfficiOS Inc.
https://www.efficios.com

Re: [PATCH 1/5] locking/rtmutex: Stop spinning on the owner when the waiter is woken

[Steven Rostedt]

On Sat, 7 Nov 2026 07:29:54 -0500
Mathieu Desnoyers <mathieu.desnoyers@efficios.com> wrote:

> >>
> >> The above is very verbose. Are you starting to use AI? ;-)  
> 
> I definitely am, as stated in the cover letter. :-)

I skipped that part of the cover letter. The first patch touches my
code (it has my commit as fixes) so I just jumped straight to that one.

> 
> > 
> > He definitely is. I can't read the above, its word soup. Part of using
> > AI is making sure the Changelog is actually readable. This is not.  
> 
> Feedback on the commit message wording is welcome, I'll try to improve
> things for next iterations. Is the following easier to parse ? I fed
> Peter's and my own criticism to Claude and got it to rewrite the entire
> thing (and keep those learnings for next time):
> 
> locking/rtmutex: Stop spinning on the owner when the waiter is woken
> 
> The top waiter of an rt_mutex sets its sleep state. Then it spins in
> rtmutex_spin_on_owner() as long as the owner is running. A wakeup sets
> the waiter back to TASK_RUNNING. The spin loop does not check for this.
> So a timeout or a signal is only handled when the owner blocks, or when
> the waiter has to reschedule:
> 
>    CPU 0 (waiter)                            CPU 1 (owner)
> 
>                                              pthread_mutex_lock()
>                                                takes the PI futex
>                                              spins in userspace
>    pthread_mutex_timedlock(1ms)
>      futex_lock_pi()
>        rt_mutex_wait_proxy_lock()
>          set_current_state(TASK_INTERRUPTIBLE)
>          rt_mutex_slowlock_block()
>            rtmutex_spin_on_owner()
>              <timeout> hrtimer_wakeup()
>                wake_up_process(waiter)
>                  __state = TASK_RUNNING
>              owner_on_cpu(owner): true
>              need_resched(): false
>              keeps spinning                  still spins in userspace
> 
> The owner of a PI futex is a user task. It can run for a long time.
> With an owner spinning in userspace, a 1ms pthread_mutex_timedlock()
> returns up to 1.4s late. A signal to the waiter is delayed in the same
> way.
> 
> With PREEMPT_RT, a ww_mutex wound is missed too. Two contexts take two
> ww_mutexes in opposite order, on two CPUs. They spin on each other for
> up to a second.
> 
> Fix this by also stopping the spin when the waiter is TASK_RUNNING. The
> slowpath then handles the timeout, the signal or the wound, as it does
> after schedule().
> 
> rtlock waiters are not affected. Only the lock handoff wakes them from
> TASK_RTLOCK_WAIT.
> 
> With this patch, the 1ms timed lock returns after about 1ms. The
> ww_mutex ABBA resolves in microseconds. PI mutex throughput is
> unchanged. The futex and ww_mutex selftests added later in this series
> cover both cases.

Still way too verbose for the fix. It makes it sound more complicated
than it is.

  locking/rtmutex: Stop spinning on the owner when the waiter is woken  

  The top waiter of an rt_mutex sets its sleep state. Then it spins in
  rtmutex_spin_on_owner() as long as the owner is running. A wakeup sets
  the waiter back to TASK_RUNNING. The spin loop does not check for this.
  So a timeout or a signal is only handled when the owner blocks, or when
  the waiter has to reschedule.

  The rtmutex_spin_on_owner() is just an optimization with the idea that
  if the owner is still running on a CPU, it will likely release it soon.
  If the owner gets preempted or schedules out on its own, the spinning
  will stop and the waiter will sleep. This came from PREEMPT_RT adaptive
  spinning for rt_spin_locks that allowed them to behave more like normal
  spin locks and speed up PREEMPT_RT performance.

  The problem is that when this was implemented for futex and normal
  mutexes it did not account for timeouts or other mechanisms where
  the owner releasing the lock wasn't the only way for the spinning
  task to exit the mutex acquire. The spinning currently does not
  account for anything else but the lock being released or the owner
  sleeping.

  Luckily, when the spinning task needs to exit the mutex lock code, it's
  state is set to TASK_RUNNING. By having the spin check the spinning
  task's state to see if it switches to running, it can exit the spin
  appropriately.

How about something like the above?

-- Steve


> 
> Fixes: 992caf7f1724 ("locking/rtmutex: Add adaptive spinwait mechanism")
> Cc: stable@vger.kernel.org
> Reviewed-by: Steven Rostedt <rostedt@goodmis.org>
> Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
> 

Re: [PATCH 1/5] locking/rtmutex: Stop spinning on the owner when the waiter is woken

[Mathieu Desnoyers]

On 2026-11-07 12:36, Steven Rostedt wrote:
[...]
> Still way too verbose for the fix. It makes it sound more complicated
> than it is.
> 
>    locking/rtmutex: Stop spinning on the owner when the waiter is woken
> 
>    The top waiter of an rt_mutex sets its sleep state. Then it spins in
>    rtmutex_spin_on_owner() as long as the owner is running. A wakeup sets
>    the waiter back to TASK_RUNNING. The spin loop does not check for this.
>    So a timeout or a signal is only handled when the owner blocks, or when
>    the waiter has to reschedule.
> 
>    The rtmutex_spin_on_owner() is just an optimization with the idea that
>    if the owner is still running on a CPU, it will likely release it soon.
>    If the owner gets preempted or schedules out on its own, the spinning
>    will stop and the waiter will sleep. This came from PREEMPT_RT adaptive
>    spinning for rt_spin_locks that allowed them to behave more like normal
>    spin locks and speed up PREEMPT_RT performance.
> 
>    The problem is that when this was implemented for futex and normal
>    mutexes it did not account for timeouts or other mechanisms where
>    the owner releasing the lock wasn't the only way for the spinning
>    task to exit the mutex acquire. The spinning currently does not
>    account for anything else but the lock being released or the owner
>    sleeping.
> 
>    Luckily, when the spinning task needs to exit the mutex lock code, it's
>    state is set to TASK_RUNNING. By having the spin check the spinning
>    task's state to see if it switches to running, it can exit the spin
>    appropriately.
> 
> How about something like the above?
It drops important information, such as the user visible impact of the
issue. How about this remix ?

locking/rtmutex: Stop spinning on the owner when the waiter is woken

The top waiter of an rt_mutex sets its sleep state. Then it spins in
rtmutex_spin_on_owner() as long as the owner is running. A wakeup sets
the waiter back to TASK_RUNNING. The spin loop does not check for this.
So a timeout or a signal is only handled when the owner blocks, or when
the waiter has to reschedule.

The spin is an optimization. A running owner will likely release the
lock soon, so the waiter spins instead of going to sleep. It comes
from the PREEMPT_RT adaptive spinning for rt_spin_locks. Only the
lock handoff wakes an rt_spin_lock waiter, so the spin did not need to
check for other wakeups there. The spin was then extended to all
rtmutex based locks. Their waiters can also be woken by a timeout, a
signal or a ww_mutex wound.

The owner of a PI futex is a user task. It can run for a long time.
With an owner spinning in userspace, a 1ms pthread_mutex_timedlock()
returned up to 1.4s late in testing. A signal to the waiter is delayed
too.

With PREEMPT_RT, two contexts that take two ww_mutexes in opposite
order spin on each other. Only something else that needs one of the two
CPUs stops them. By default, the fair server does it after 950ms, to
run SCHED_OTHER tasks.

Fix this by also stopping the spin when the waiter is TASK_RUNNING. The
slowpath then handles the wakeup, as it does after schedule().

PI mutex throughput is unchanged. The futex and ww_mutex selftests added
later in this series test both cases.

Fixes: 992caf7f1724 ("locking/rtmutex: Add adaptive spinwait mechanism")
Cc: stable@vger.kernel.org
Reviewed-by: Steven Rostedt <rostedt@goodmis.org>
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>

Thanks,

Mathieu

-- 
Mathieu Desnoyers
EfficiOS Inc.
https://www.efficios.com

Re: [PATCH 1/5] locking/rtmutex: Stop spinning on the owner when the waiter is woken

[Steven Rostedt]

On Sat, 7 Nov 2026 14:34:12 -0500
Mathieu Desnoyers <mathieu.desnoyers@efficios.com> wrote:

> > 
> > How about something like the above?  
> It drops important information, such as the user visible impact of the
> issue. How about this remix ?
> 
> locking/rtmutex: Stop spinning on the owner when the waiter is woken
> 
> The top waiter of an rt_mutex sets its sleep state. Then it spins in
> rtmutex_spin_on_owner() as long as the owner is running. A wakeup sets
> the waiter back to TASK_RUNNING. The spin loop does not check for this.
> So a timeout or a signal is only handled when the owner blocks, or when
> the waiter has to reschedule.
> 
> The spin is an optimization. A running owner will likely release the
> lock soon, so the waiter spins instead of going to sleep. It comes
> from the PREEMPT_RT adaptive spinning for rt_spin_locks. Only the
> lock handoff wakes an rt_spin_lock waiter, so the spin did not need to
> check for other wakeups there. The spin was then extended to all
> rtmutex based locks. Their waiters can also be woken by a timeout, a
> signal or a ww_mutex wound.
> 
> The owner of a PI futex is a user task. It can run for a long time.
> With an owner spinning in userspace, a 1ms pthread_mutex_timedlock()
> returned up to 1.4s late in testing. A signal to the waiter is delayed
> too.

The last sentence seems out of place. Perhaps be more specific?

 The timeout signal or any signal to the waiter will not stop it
 from spinning, and the signal will be delayed until it exits the spin.

> 
> With PREEMPT_RT, two contexts that take two ww_mutexes in opposite
> order spin on each other. Only something else that needs one of the two
> CPUs stops them. By default, the fair server does it after 950ms, to
> run SCHED_OTHER tasks.

Yeah, that is a big issue.

> 
> Fix this by also stopping the spin when the waiter is TASK_RUNNING. The
> slowpath then handles the wakeup, as it does after schedule().

The last sentence doesn't really make sense. I can figure it out
because I know what the slowpath does, but as you see in my version, I
go into a little more detail to why it's not an issue to break out of
that spin on TASK_RUNNING.


> PI mutex throughput is unchanged. The futex and ww_mutex selftests added
> later in this series test both cases.

I think you can drop the above. I don't see it adding anything for the
change log. That is more for cover letters.

Thanks,

-- Steve

Re: [PATCH 1/5] locking/rtmutex: Stop spinning on the owner when the waiter is woken

[Mathieu Desnoyers]

On 2026-11-07 17:18, Steven Rostedt wrote:
[...]
> The last sentence doesn't really make sense. I can figure it out
> because I know what the slowpath does, but as you see in my version, I
> go into a little more detail to why it's not an issue to break out of
> that spin on TASK_RUNNING.

Thanks for the feedback! I'll take your suggested wording as is and
add the two paragraphs about user-visible impact of the issue.

Mathieu


-- 
Mathieu Desnoyers
EfficiOS Inc.
https://www.efficios.com

[PATCH 2/5] locking/rtmutex: Initialize the ww_mutex context of a waiter

[Mathieu Desnoyers]

With PREEMPT_RT no deadlock on PI futexes is reported: two threads that
each block on a PI futex the other holds sleep for good, where
FUTEX_LOCK_PI returns EDEADLK without PREEMPT_RT, and a chain deeper
than max_lock_depth is admitted.

rt_mutex_init_waiter() leaves waiter->ww_ctx as it finds it, and only
__rt_mutex_slowlock_locked() sets it. The waiters of PI futexes and of
rtlocks keep the 0x11 poison of CONFIG_DEBUG_RT_MUTEXES, or what the
stack held unless the compiler zeroes it. Since
commit 6467822b8cc9 ("locking/rtmutex: Prevent spurious EDEADLK return
caused by ww_mutexes"), the PREEMPT_RT chain walk takes a waiter with a
context for a ww_mutex one: it turns deadlock detection off and drops
the -EDEADLK of a cycle. The ww_mutex build of the walk would also
dereference the poison, in __waiter_less(), on two such waiters of equal
priority on a lock it goes through.

Clear the context with the rest of the waiter. In 8-CPU VMs with
PREEMPT_RT and CONFIG_DEBUG_RT_MUTEXES, v7.4-rc1 lets the ABBA time out
and admits the deep chain, in 2 VMs of 2; with this patch both get
EDEADLK. The futex_lock_pi_inherit selftest added later in this series
has the two cases, deadlock and max_lock_depth, and found this.

Fixes: 6467822b8cc9 ("locking/rtmutex: Prevent spurious EDEADLK return caused by ww_mutexes")
Cc: stable@vger.kernel.org
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
---
 kernel/locking/rtmutex_common.h | 1 +
 1 file changed, 1 insertion(+)

diff --git a/kernel/locking/rtmutex_common.h b/kernel/locking/rtmutex_common.h
index c38b7bdea7b3..1f56d40c3e66 100644
--- a/kernel/locking/rtmutex_common.h
+++ b/kernel/locking/rtmutex_common.h
@@ -218,6 +218,7 @@ static inline void rt_mutex_init_waiter(struct rt_mutex_waiter *waiter)
 	RB_CLEAR_NODE(&waiter->tree.entry);
 	waiter->wake_state = TASK_NORMAL;
 	waiter->task = NULL;
+	waiter->ww_ctx = NULL;
 }
 
 static inline void rt_mutex_init_rtlock_waiter(struct rt_mutex_waiter *waiter)
-- 
2.47.3

[PATCH 3/5] locking/rtmutex: Clone a waiter's sort keys as it enters pi_waiters

[Mathieu Desnoyers]

An rt_mutex owner can inherit the priority of the wrong waiter. A waiter
has two copies of its sort keys: the lock tree's, updated when its
priority changes, and the pi_tree's, which orders its owner's
pi_waiters, cloned from the first when the waiter blocks and by the
chain walk while it is the top waiter of its lock. A waiter whose
priority changes while another is ahead of it, and that becomes the top
waiter later, as the one ahead times out (remove_waiter()) or the lock
changes hands (try_to_take_rt_mutex()), enters pi_waiters with keys it
no longer has. On v7.4-rc1, an owner with a SCHED_FIFO 50 waiter on one
mutex, and on another a SCHED_FIFO 70 waiter set to SCHED_FIFO 10 behind
one of 80 that times out, runs at SCHED_FIFO 10 while the SCHED_FIFO 50
task waits on it, in 3 runs of 3.

Clone the keys in rt_mutex_enqueue_pi(), under the wait_lock that the
lock tree copy is updated under and the owner's pi_lock that the
pi_tree copy is read under, and drop the clones its callers did. The
owner then runs at SCHED_FIFO 50, in 3 runs of 3. The
futex_lock_pi_inherit selftest added later in this series has the case,
promoted_waiter.

Found reviewing "sched/deadline: Take a deadline boost from the
waiter's rt_mutex node", posted in a series on top of this one, which
takes a deadline boost from these keys.

Fixes: f7853c342418 ("locking/rtmutex: Fix task->pi_waiters integrity")
Cc: stable@vger.kernel.org
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
---
 kernel/locking/rtmutex.c | 11 ++++++++---
 1 file changed, 8 insertions(+), 3 deletions(-)

diff --git a/kernel/locking/rtmutex.c b/kernel/locking/rtmutex.c
index 323973be2074..e89f0f0e604b 100644
--- a/kernel/locking/rtmutex.c
+++ b/kernel/locking/rtmutex.c
@@ -512,11 +512,19 @@ static __always_inline bool __pi_waiter_less(struct rb_node *a, const struct rb_
 	return rt_waiter_node_less(__node_2_rt_node(a), __node_2_rt_node(b));
 }
 
+/*
+ * Enqueue @waiter in the pi_waiters of @task, its lock's owner, with the
+ * sort keys of its lock tree copy, current under the wait_lock held here:
+ * the chain walk updates the tree copy of a waiter that is not the top
+ * waiter of its lock, and does not clone it, so a copy cloned before would
+ * enter @task's pi_waiters with the keys of a priority it no longer has.
+ */
 static __always_inline void
 rt_mutex_enqueue_pi(struct task_struct *task, struct rt_mutex_waiter *waiter)
 {
 	lockdep_assert_held(&task->pi_lock);
 
+	waiter_clone_prio(waiter, task);
 	rb_add_cached(&waiter->pi_tree.entry, &task->pi_waiters, __pi_waiter_less);
 }
 
@@ -1005,7 +1013,6 @@ static int __sched rt_mutex_adjust_prio_chain(struct task_struct *task,
 		 * and adjust the priority of the owner.
 		 */
 		rt_mutex_dequeue_pi(task, prerequeue_top_waiter);
-		waiter_clone_prio(waiter, task);
 		rt_mutex_enqueue_pi(task, waiter);
 		rt_mutex_adjust_prio(lock, task);
 
@@ -1022,7 +1029,6 @@ static int __sched rt_mutex_adjust_prio_chain(struct task_struct *task,
 		 */
 		rt_mutex_dequeue_pi(task, waiter);
 		waiter = rt_mutex_top_waiter(lock);
-		waiter_clone_prio(waiter, task);
 		rt_mutex_enqueue_pi(task, waiter);
 		rt_mutex_adjust_prio(lock, task);
 	} else {
@@ -1243,7 +1249,6 @@ static int __sched task_blocks_on_rt_mutex(struct rt_mutex_base *lock,
 	waiter->task = task;
 	waiter->lock = lock;
 	waiter_update_prio(waiter, task);
-	waiter_clone_prio(waiter, task);
 
 	/* Get the top priority waiter on the lock */
 	if (rt_mutex_has_waiters(lock))
-- 
2.47.3

Re: [PATCH 3/5] locking/rtmutex: Clone a waiter's sort keys as it enters pi_waiters

[Peter Zijlstra]

On Fri, Nov 06, 2026 at 06:59:29PM -0500, Mathieu Desnoyers wrote:

snip unreadable changelog

> Fixes: f7853c342418 ("locking/rtmutex: Fix task->pi_waiters integrity")
> Cc: stable@vger.kernel.org
> Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
> ---
>  kernel/locking/rtmutex.c | 11 ++++++++---
>  1 file changed, 8 insertions(+), 3 deletions(-)
> 
> diff --git a/kernel/locking/rtmutex.c b/kernel/locking/rtmutex.c
> index 323973be2074..e89f0f0e604b 100644
> --- a/kernel/locking/rtmutex.c
> +++ b/kernel/locking/rtmutex.c
> @@ -512,11 +512,19 @@ static __always_inline bool __pi_waiter_less(struct rb_node *a, const struct rb_
>  	return rt_waiter_node_less(__node_2_rt_node(a), __node_2_rt_node(b));
>  }
>  
> +/*
> + * Enqueue @waiter in the pi_waiters of @task, its lock's owner, with the
> + * sort keys of its lock tree copy, current under the wait_lock held here:
> + * the chain walk updates the tree copy of a waiter that is not the top
> + * waiter of its lock, and does not clone it, so a copy cloned before would
> + * enter @task's pi_waiters with the keys of a priority it no longer has.
> + */

Oh gawd, Mathieu, can you please tell your AI thing to never write a
Changelog or comment? This is again unreadable soup.

>  static __always_inline void
>  rt_mutex_enqueue_pi(struct task_struct *task, struct rt_mutex_waiter *waiter)
>  {
>  	lockdep_assert_held(&task->pi_lock);
>  
> +	waiter_clone_prio(waiter, task);
>  	rb_add_cached(&waiter->pi_tree.entry, &task->pi_waiters, __pi_waiter_less);
>  }
>  
> @@ -1005,7 +1013,6 @@ static int __sched rt_mutex_adjust_prio_chain(struct task_struct *task,
>  		 * and adjust the priority of the owner.
>  		 */
>  		rt_mutex_dequeue_pi(task, prerequeue_top_waiter);
> -		waiter_clone_prio(waiter, task);
>  		rt_mutex_enqueue_pi(task, waiter);
>  		rt_mutex_adjust_prio(lock, task);
>  
> @@ -1022,7 +1029,6 @@ static int __sched rt_mutex_adjust_prio_chain(struct task_struct *task,
>  		 */
>  		rt_mutex_dequeue_pi(task, waiter);
>  		waiter = rt_mutex_top_waiter(lock);
> -		waiter_clone_prio(waiter, task);
>  		rt_mutex_enqueue_pi(task, waiter);
>  		rt_mutex_adjust_prio(lock, task);
>  	} else {
> @@ -1243,7 +1249,6 @@ static int __sched task_blocks_on_rt_mutex(struct rt_mutex_base *lock,
>  	waiter->task = task;
>  	waiter->lock = lock;
>  	waiter_update_prio(waiter, task);
> -	waiter_clone_prio(waiter, task);

Reverse engineering the problem leads me to think this is the problem
spot. Because this is the one site that is not immediately followed by
rt_mutex_enqueue_pi().

I'm thinking there's race due to dropping pi_lock between this copy and
the subsequent enqueue?

>  
>  	/* Get the top priority waiter on the lock */
>  	if (rt_mutex_has_waiters(lock))
> -- 
> 2.47.3
> 

Re: [PATCH 3/5] locking/rtmutex: Clone a waiter's sort keys as it enters pi_waiters

[Mathieu Desnoyers]

On 2026-11-07 05:57, Peter Zijlstra wrote:
> On Fri, Nov 06, 2026 at 06:59:29PM -0500, Mathieu Desnoyers wrote:
[...]
>> @@ -1243,7 +1249,6 @@ static int __sched task_blocks_on_rt_mutex(struct rt_mutex_base *lock,
>>  	waiter->task = task;
>>  	waiter->lock = lock;
>>  	waiter_update_prio(waiter, task);
>> -	waiter_clone_prio(waiter, task);
>
> Reverse engineering the problem leads me to think this is the problem
> spot. Because this is the one site that is not immediately followed by
> rt_mutex_enqueue_pi().
>
> I'm thinking there's race due to dropping pi_lock between this copy and
> the subsequent enqueue?

I don't think there is a race there. task_blocks_on_rt_mutex() holds
the wait_lock from this copy to the enqueue. waiter->tree only changes
under the wait_lock. So at the enqueue, waiter->pi_tree still equals
waiter->tree.

The problem spots are the two enqueues with no copy before them, in
remove_waiter() and try_to_take_rt_mutex(). Both enqueue the new top
waiter of the lock. When a waiter's priority changes, the chain walk
updates its waiter->tree. It copies waiter->tree to waiter->pi_tree for
the top waiter only. So a waiter that was not the top waiter can enter
pi_waiters with old keys.

This copy goes away only because rt_mutex_enqueue_pi() now does it.

I rewrote the changelog and the comment. The new version of the patch
is below. Do they read better now? I would like your feedback on both
before I send v2.

Thanks,

Mathieu

-- >8 --
From: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Subject: locking/rtmutex: Clone the sort keys in rt_mutex_enqueue_pi()

An rt_mutex waiter has two copies of its sort keys. waiter->tree sorts
the waiters of its lock. waiter->pi_tree sorts the pi_waiters of the
lock owner. Only the top waiter of each lock is in pi_waiters.

When the priority of a waiter changes, the chain walk updates
waiter->tree. It copies waiter->tree to waiter->pi_tree for the top
waiter only. A waiter that is not the top waiter keeps its old pi_tree
keys.

Such a waiter can become the top waiter later. remove_waiter() does this
when the top waiter leaves. try_to_take_rt_mutex() does it when the lock
changes owner. Both put the new top waiter in pi_waiters without a new
copy. So pi_waiters sorts it by a priority it no longer has. The owner
then inherits from the wrong waiter, at that waiter's current priority.

In this example, O is the owner, a SCHED_OTHER task. L1 and L2 are two
rt_mutexes. W1, W2 and W3 are waiters. Their SCHED_FIFO priority is
under their name. T is any other task. It changes the priority of W2:

   T                         O                W1                            W2           W3
                                              FIFO 80                       FIFO 70      FIFO 50

                             locks L1, L2
                                                                                         blocks on L2
                                              blocks on L1 (timed)
                                                                            blocks on L1
   sched_setscheduler(W2)
     W2 now at FIFO 10
     rt_mutex_adjust_pi(W2)
       W2->tree: FIFO 10
       W2 not top: no copy
       W2->pi_tree: FIFO 70
                                              <timeout>
                                              remove_waiter()
                                                rt_mutex_enqueue_pi(O, W2)
                                                  W2 sorted as FIFO 70,
                                                  ahead of W3
                                                rt_mutex_adjust_prio(O)
                                                  O inherits W2's FIFO 10
                             runs at FIFO 10                                             still waits

The owner runs at SCHED_FIFO 10 while a SCHED_FIFO 50 task waits for
it. This needs no race. Each step can come long after the one before.

Fix this by copying the keys in rt_mutex_enqueue_pi(). All callers hold
the wait_lock and the owner's pi_lock, as waiter_clone_prio() requires.
Remove the three copies the callers made.

With this patch, the owner runs at SCHED_FIFO 50. The
futex_lock_pi_inherit selftest added later in this series tests this
case (promoted_waiter).

Fixes: f7853c342418 ("locking/rtmutex: Fix task->pi_waiters integrity")
Cc: stable@vger.kernel.org
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
---
  kernel/locking/rtmutex.c | 5 ++---
  1 file changed, 2 insertions(+), 3 deletions(-)

diff --git a/kernel/locking/rtmutex.c b/kernel/locking/rtmutex.c
index 323973be2074..975efa1d864c 100644
--- a/kernel/locking/rtmutex.c
+++ b/kernel/locking/rtmutex.c
@@ -517,6 +517,8 @@ rt_mutex_enqueue_pi(struct task_struct *task, struct rt_mutex_waiter *waiter)
  {
  	lockdep_assert_held(&task->pi_lock);
  
+	/* The tree keys can change while @waiter is not in pi_waiters. */
+	waiter_clone_prio(waiter, task);
  	rb_add_cached(&waiter->pi_tree.entry, &task->pi_waiters, __pi_waiter_less);
  }
  
@@ -1005,7 +1007,6 @@ static int __sched rt_mutex_adjust_prio_chain(struct task_struct *task,
  		 * and adjust the priority of the owner.
  		 */
  		rt_mutex_dequeue_pi(task, prerequeue_top_waiter);
-		waiter_clone_prio(waiter, task);
  		rt_mutex_enqueue_pi(task, waiter);
  		rt_mutex_adjust_prio(lock, task);
  
@@ -1022,7 +1023,6 @@ static int __sched rt_mutex_adjust_prio_chain(struct task_struct *task,
  		 */
  		rt_mutex_dequeue_pi(task, waiter);
  		waiter = rt_mutex_top_waiter(lock);
-		waiter_clone_prio(waiter, task);
  		rt_mutex_enqueue_pi(task, waiter);
  		rt_mutex_adjust_prio(lock, task);
  	} else {
@@ -1243,7 +1243,6 @@ static int __sched task_blocks_on_rt_mutex(struct rt_mutex_base *lock,
  	waiter->task = task;
  	waiter->lock = lock;
  	waiter_update_prio(waiter, task);
-	waiter_clone_prio(waiter, task);
  
  	/* Get the top priority waiter on the lock */
  	if (rt_mutex_has_waiters(lock))