[PATCH] rseq: defer time slice extension yield for sys_futex_wake

[PATCH] rseq: defer time slice extension yield for sys_futex_wake

[Alice Ryhl]

When a task is granted an rseq scheduler time slice extension, it is
expected to finish its critical section and relinquish the CPU via
rseq_slice_yield(2). If the task issues any other system call while a
grant is active, rseq_syscall_enter_work() forces an immediate
reschedule on syscall entry via cond_resched(). This may cause
significant latency penalty for userspace lock implementations that use
rseq time slice extensions when unlocking the futex.

In a userspace mutex unlock sequence:
1. The lock is released in userspace.
2. If there are waiters, the unlocking thread calls sys_futex_wake()
   to wake a sleeping waiter.

Because sys_futex_wake() is currently treated as an arbitrary syscall,
rseq_syscall_enter_work() schedules out the unlocking thread upon
syscall entry, which is before it has executed the wakeup. Consequently,
the lock is free in userspace, but the waiter remains blocked in the
kernel while the CPU switches to an unrelated task. The waiter is only
woken when the unlocking thread is eventually scheduled back in to
finish the syscall, causing lock handoff delays.

Thus, update rseq_syscall_enter_work() for sys_futex_wake() so that it
does not reschedule during syscall entry. The thread will yield the CPU
on the syscall exit path instead.

There is no need to apply this optimization to the multiplexed futex()
syscall since any userspace code that can invoke rseq_slice_yield() can
also invoke futex_wake().

This patch was verified via ftrace that when sys_futex_wake() is invoked
during an active slice grant, the wakeup (sched_waking) occurs before
any context switch:
1. sys_enter_futex_wake
2. sched_waking (wakes waiting thread)
3. sys_exit_futex_wake -> 0x1
4. sched_switch (on syscall exit due to TIF_NEED_RESCHED)

Assisted-by: LLM
Signed-off-by: Alice Ryhl <aliceryhl@google.com>
---
 Documentation/userspace-api/rseq.rst | 17 +++++++++++++----
 kernel/rseq.c                        | 16 +++++++++++++---
 2 files changed, 26 insertions(+), 7 deletions(-)

diff --git a/Documentation/userspace-api/rseq.rst b/Documentation/userspace-api/rseq.rst
index 8549a6c61531..30bdb1885f67 100644
--- a/Documentation/userspace-api/rseq.rst
+++ b/Documentation/userspace-api/rseq.rst
@@ -217,10 +217,10 @@ operation.
 
 If the thread issues a syscall other than rseq_slice_yield(2) within the
 granted timeslice extension, the grant is also revoked and the CPU is
-relinquished immediately when entering the kernel. This is required as
-syscalls might consume arbitrary CPU time until they reach a scheduling
-point when the preemption model is either NONE or VOLUNTARY and therefore
-might exceed the grant by far.
+relinquished. For most syscalls, this occurs immediately when entering the
+kernel. This is required as syscalls might consume arbitrary CPU time until
+they reach a scheduling point when the preemption model is either NONE or
+VOLUNTARY and therefore might exceed the grant by far.
 
 The preferred solution for user space is to use rseq_slice_yield(2) which
 is side effect free. The support for arbitrary syscalls is required to
@@ -228,5 +228,14 @@ support onion layer architectured applications, where the code handling the
 critical section and requesting the time slice extension has no control
 over the code within the critical section.
 
+For futex_wake(2), the CPU is instead relinquished when returning to userspace,
+so that it gets a chance to wake any waiting tasks before yielding the CPU.
+This makes it possible to terminate the critical region of a userspace mutex
+using rseq and futexes with futex_wake(2) instead of rseq_slice_yield(2).
+Currently, this is the only syscall that does not relinquish the CPU
+immediately on syscall entry. Note in particular that this applies only to the
+dedicated futex_wake(2) syscall, and not to the futex(2) syscall even when
+using op=FUTEX_WAKE.
+
 The kernel enforces flag consistency and terminates the thread with SIGSEGV
 if it detects a violation.
diff --git a/kernel/rseq.c b/kernel/rseq.c
index e75e3a5e312c..81bc2ed997f0 100644
--- a/kernel/rseq.c
+++ b/kernel/rseq.c
@@ -723,9 +723,18 @@ void rseq_syscall_enter_work(long syscall)
 		 * the task was already rescheduled before arriving here.
 		 */
 		if (!curr->rseq.event.sched_switch) {
-			rseq_slice_set_need_resched(curr);
+			if (syscall == __NR_futex_wake) {
+				/*
+				 * For this syscall, reschedule on syscall exit
+				 * instead of syscall entry to avoid delaying
+				 * the wakeup.
+				 */
+				set_tsk_need_resched(curr);
+			} else {
+				rseq_slice_set_need_resched(curr);
+			}
 
-			if (syscall == __NR_rseq_slice_yield) {
+			if (syscall == __NR_rseq_slice_yield || syscall == __NR_futex_wake) {
 				rseq_stat_inc(rseq_stats.s_yielded);
 				/* Update the yielded state for syscall return */
 				curr->rseq.slice.yielded = 1;
@@ -735,7 +744,8 @@ void rseq_syscall_enter_work(long syscall)
 		}
 	}
 	/* Reschedule on NONE/VOLUNTARY preemption models */
-	cond_resched();
+	if (syscall != __NR_futex_wake)
+		cond_resched();
 
 	/* Clear the grant in kernel state and user space */
 	curr->rseq.slice.state.granted = false;

---
base-commit: cee9395acd8043be0644b25c34bfa86623f2b935
change-id: 20260831-sys-futex-wake-time-slice-c36738bf7b94

Best regards,
-- 
Alice Ryhl <aliceryhl@google.com>

Re: [PATCH] rseq: defer time slice extension yield for sys_futex_wake

[Mathieu Desnoyers]

On 2026-08-31 08:57, Alice Ryhl wrote:
> When a task is granted an rseq scheduler time slice extension, it is
> expected to finish its critical section and relinquish the CPU via
> rseq_slice_yield(2). If the task issues any other system call while a
> grant is active, rseq_syscall_enter_work() forces an immediate
> reschedule on syscall entry via cond_resched(). This may cause
> significant latency penalty for userspace lock implementations that use
> rseq time slice extensions when unlocking the futex.
> 
> In a userspace mutex unlock sequence:
> 1. The lock is released in userspace.
> 2. If there are waiters, the unlocking thread calls sys_futex_wake()
>     to wake a sleeping waiter.
> 
> Because sys_futex_wake() is currently treated as an arbitrary syscall,
> rseq_syscall_enter_work() schedules out the unlocking thread upon
> syscall entry, which is before it has executed the wakeup. Consequently,
> the lock is free in userspace, but the waiter remains blocked in the
> kernel while the CPU switches to an unrelated task. The waiter is only
> woken when the unlocking thread is eventually scheduled back in to
> finish the syscall, causing lock handoff delays.
> 
> Thus, update rseq_syscall_enter_work() for sys_futex_wake() so that it
> does not reschedule during syscall entry. The thread will yield the CPU
> on the syscall exit path instead.
> 
> There is no need to apply this optimization to the multiplexed futex()
> syscall since any userspace code that can invoke rseq_slice_yield() can
> also invoke futex_wake().

Is the goal there to provide a single blessed way of doing futex wake,
or to allow the futex multiplexer to keep being used for that wake
scenario ?

The proposed change exposes two ABIs (multiplexer vs explicit futex
wake) with very different behaviors. I'm concerned that it would be
confusing to users.

Thoughts ?

Thanks,

Mathieu


-- 
Mathieu Desnoyers
EfficiOS Inc.
https://www.efficios.com

Re: [PATCH] rseq: defer time slice extension yield for sys_futex_wake

[Alice Ryhl]

On Mon, Aug 31, 2026 at 09:45:25AM -0400, Mathieu Desnoyers wrote:
> On 2026-08-31 08:57, Alice Ryhl wrote:
> > When a task is granted an rseq scheduler time slice extension, it is
> > expected to finish its critical section and relinquish the CPU via
> > rseq_slice_yield(2). If the task issues any other system call while a
> > grant is active, rseq_syscall_enter_work() forces an immediate
> > reschedule on syscall entry via cond_resched(). This may cause
> > significant latency penalty for userspace lock implementations that use
> > rseq time slice extensions when unlocking the futex.
> > 
> > In a userspace mutex unlock sequence:
> > 1. The lock is released in userspace.
> > 2. If there are waiters, the unlocking thread calls sys_futex_wake()
> >     to wake a sleeping waiter.
> > 
> > Because sys_futex_wake() is currently treated as an arbitrary syscall,
> > rseq_syscall_enter_work() schedules out the unlocking thread upon
> > syscall entry, which is before it has executed the wakeup. Consequently,
> > the lock is free in userspace, but the waiter remains blocked in the
> > kernel while the CPU switches to an unrelated task. The waiter is only
> > woken when the unlocking thread is eventually scheduled back in to
> > finish the syscall, causing lock handoff delays.
> > 
> > Thus, update rseq_syscall_enter_work() for sys_futex_wake() so that it
> > does not reschedule during syscall entry. The thread will yield the CPU
> > on the syscall exit path instead.
> > 
> > There is no need to apply this optimization to the multiplexed futex()
> > syscall since any userspace code that can invoke rseq_slice_yield() can
> > also invoke futex_wake().
> 
> Is the goal there to provide a single blessed way of doing futex wake,
> or to allow the futex multiplexer to keep being used for that wake
> scenario ?
> 
> The proposed change exposes two ABIs (multiplexer vs explicit futex
> wake) with very different behaviors. I'm concerned that it would be
> confusing to users.
> 
> Thoughts ?

My understanding is that the multiplexed futex syscall is soft
deprecated and the goal is that new code should use the dedicated
syscalls, so I didn't think it was needed to implement the perf
optimizations for the "old" API.

But I do agree it's confusing to do it that way, so I'm happy to also
support the multiplexed one if you think we should.

Note that we can't read the 'op' argument to the multiplexed futex
syscall inside rseq_syscall_enter_work(), so to implement it for that
one too, we would have to skip the cond_resched() for all calls to
sys_futex(), and then re-check inside of sys_futex() itself to call
cond_resched() there if `op != FUTEX_WAKE` and the rseq time slice
extension applies.

Though now that I think about it, maybe we want to skip the
cond_resched() for all futex ops? If you're invoking FUTEX_WAIT, then
there's not really much reason to call cond_resched() if you're calling
schedule() immediately afterwards.

Alice

Re: [PATCH] rseq: defer time slice extension yield for sys_futex_wake

[Dmitry Ilvokhin]

On Mon, Aug 31, 2026 at 12:57:26PM +0000, Alice Ryhl wrote:
> When a task is granted an rseq scheduler time slice extension, it is
> expected to finish its critical section and relinquish the CPU via
> rseq_slice_yield(2). If the task issues any other system call while a
> grant is active, rseq_syscall_enter_work() forces an immediate
> reschedule on syscall entry via cond_resched(). This may cause
> significant latency penalty for userspace lock implementations that use
> rseq time slice extensions when unlocking the futex.
> 
> In a userspace mutex unlock sequence:
> 1. The lock is released in userspace.
> 2. If there are waiters, the unlocking thread calls sys_futex_wake()
>    to wake a sleeping waiter.

Just out of curiosity, is there any publicly available implementation of
a mutex that combines rseq time-slice extensions and futexes?

I'm interested in learning more about how these two mechanisms interact
with each other.

Re: [PATCH] rseq: defer time slice extension yield for sys_futex_wake

[Alice Ryhl]

On Tue, Sep 01, 2026 at 11:49:43AM +0000, Dmitry Ilvokhin wrote:
> On Mon, Aug 31, 2026 at 12:57:26PM +0000, Alice Ryhl wrote:
> > When a task is granted an rseq scheduler time slice extension, it is
> > expected to finish its critical section and relinquish the CPU via
> > rseq_slice_yield(2). If the task issues any other system call while a
> > grant is active, rseq_syscall_enter_work() forces an immediate
> > reschedule on syscall entry via cond_resched(). This may cause
> > significant latency penalty for userspace lock implementations that use
> > rseq time slice extensions when unlocking the futex.
> > 
> > In a userspace mutex unlock sequence:
> > 1. The lock is released in userspace.
> > 2. If there are waiters, the unlocking thread calls sys_futex_wake()
> >    to wake a sleeping waiter.
> 
> Just out of curiosity, is there any publicly available implementation of
> a mutex that combines rseq time-slice extensions and futexes?
> 
> I'm interested in learning more about how these two mechanisms interact
> with each other.

This came up while I was looking into implementing one for use in Tokio.
There, we have quite a few cases where we have a doubly linked list
protected by a mutex, and I really really want to avoid preemption while
that lock is held. Some of those locks are known problems for contention
in Tokio.

But that implementation is still WIP.

Some pseudocode:

struct rseq_futex {
    // 0 = unlocked, 1 = locked, 2 = contended
    int futex;
};

void mutex_lock(struct rseq_futex *mutex) {
    for (;;) {
        __rseq_abi.slice_ctrl.request = 1;
        int expected = 0;
        // Change state from 0 -> 1 to lock.
        if (compare_exchange(&mutex->futex, &expected, 1))
            return;

        // lock is taken, use slow-path
        bool was_granted = __rseq_abi.slice_ctrl.granted;
        __rseq_abi.slice_ctrl.request = 0;

        // If the state is not already 2, then change it so that
        // mutex_unlock() knows to wake us up.
        if (expected == 2 || compare_exchange(&mutex->futex, &expected, 2)) {
            // State is 2, we can sleep until that changes.
            futex_wait(&mutex->futex, 2);
        } else if (was_granted) {
            rseq_slice_yield();
        }
    }
}

void mutex_unlock(struct rseq_futex *mutex) {
    // Unlock the mutex and return whether it was contended.
    int prev = atomic_swap(&mutex->futex, 0);

    // End the time slice extension for the critical region
    bool was_granted = __rseq_abi.slice_ctrl.granted;
    __rseq_abi.slice_ctrl.request = 0;

    if (prev == 2) {
        futex_wake(&mutex->futex);
    } else if (was_granted) {
        rseq_slice_yield();
    }
}

Though now that I think more about it, perhaps unlock should look like
this, to avoid a preemption point just immediately before futex_wake().

void mutex_unlock(struct rseq_futex *mutex) {
    // Unlock the mutex and return whether it was contended.
    int prev = atomic_exchange(&mutex->futex, 0);

    // Wake up contended waiters
    if (prev == 2)
        futex_wake(&mutex->futex);

    // End the time slice extension for the critical region
    bool was_granted = __rseq_abi.slice_ctrl.granted;
    __rseq_abi.slice_ctrl.request = 0;

    if (was_granted)
        rseq_slice_yield();
}

Re: [PATCH] rseq: defer time slice extension yield for sys_futex_wake

[Thomas Gleixner]

On Mon, Aug 31 2026 at 12:57, Alice Ryhl wrote:
> When a task is granted an rseq scheduler time slice extension, it is
> expected to finish its critical section and relinquish the CPU via
> rseq_slice_yield(2). If the task issues any other system call while a
> grant is active, rseq_syscall_enter_work() forces an immediate
> reschedule on syscall entry via cond_resched(). This may cause
> significant latency penalty for userspace lock implementations that use
> rseq time slice extensions when unlocking the futex.
>
> In a userspace mutex unlock sequence:
> 1. The lock is released in userspace.
> 2. If there are waiters, the unlocking thread calls sys_futex_wake()
>    to wake a sleeping waiter.
>
> Because sys_futex_wake() is currently treated as an arbitrary syscall,
> rseq_syscall_enter_work() schedules out the unlocking thread upon
> syscall entry, which is before it has executed the wakeup. Consequently,
> the lock is free in userspace, but the waiter remains blocked in the
> kernel while the CPU switches to an unrelated task. The waiter is only
> woken when the unlocking thread is eventually scheduled back in to
> finish the syscall, causing lock handoff delays.
>
> Thus, update rseq_syscall_enter_work() for sys_futex_wake() so that it
> does not reschedule during syscall entry. The thread will yield the CPU
> on the syscall exit path instead.

That's undermining the design and takes control away from the scheduler.

It granted a short extension with well defined semantics and then you
special case futex_wake() which can take arbitrary time to complete.

Thanks,

        tglx

Re: [PATCH] rseq: defer time slice extension yield for sys_futex_wakey

[Dmitry Ilvokhin]

On Fri, Sep 04, 2026 at 07:21:37AM +0200, Thomas Gleixner wrote:
> On Mon, Aug 31 2026 at 12:57, Alice Ryhl wrote:
> > When a task is granted an rseq scheduler time slice extension, it is
> > expected to finish its critical section and relinquish the CPU via
> > rseq_slice_yield(2). If the task issues any other system call while a
> > grant is active, rseq_syscall_enter_work() forces an immediate
> > reschedule on syscall entry via cond_resched(). This may cause
> > significant latency penalty for userspace lock implementations that use
> > rseq time slice extensions when unlocking the futex.
> >
> > In a userspace mutex unlock sequence:
> > 1. The lock is released in userspace.
> > 2. If there are waiters, the unlocking thread calls sys_futex_wake()
> >    to wake a sleeping waiter.
> >
> > Because sys_futex_wake() is currently treated as an arbitrary syscall,
> > rseq_syscall_enter_work() schedules out the unlocking thread upon
> > syscall entry, which is before it has executed the wakeup. Consequently,
> > the lock is free in userspace, but the waiter remains blocked in the
> > kernel while the CPU switches to an unrelated task. The waiter is only
> > woken when the unlocking thread is eventually scheduled back in to
> > finish the syscall, causing lock handoff delays.
> >
> > Thus, update rseq_syscall_enter_work() for sys_futex_wake() so that it
> > does not reschedule during syscall entry. The thread will yield the CPU
> > on the syscall exit path instead.
> 
> That's undermining the design and takes control away from the scheduler.
> 
> It granted a short extension with well defined semantics and then you
> special case futex_wake() which can take arbitrary time to complete.

Thomas, do you think the problem is worth solving, though?

Currently, it seems like the rseq time slice extension is a good fit for
userspace spinlocks implementation, but userspace adaptive mutexes don't
fit quite as well.

One can argue that adopting the rseq time slice extension for adaptive
mutexes can never make things worse. The extension allows the lock to be
released, so other threads are free to grab it. The only problem is a
potentially delayed waiter, but this can happen now anyway, even without
the time slice extension applied. That said, it doesn't mean we can't do
better here.

One option that I can think of is a best-effort
rseq_slice_yield_wake(uaddr) that is allowed to fail with a userspace
falling back to futex_wake() in case of a failure.

rseq_slice_yield_wake(uaddr) could look like this:

- Works only for private futexes with nr=1.

- Bails out early on a contended hb->lock.

- Limits the hb->chain walk time by the same time slice extension, that
  is already set.

This way the scheduler is still very much in control and in case of the
success, scheduler might pick a better task, since the waiter is now
available to run.