[PATCH] io_uring/futex: return a wait's data to the cache when it fails to arm

[PATCH] io_uring/futex: return a wait's data to the cache when it fails to arm

[Austin Schuh]

io_futex_wait() takes its io_futex_data from ctx->futex_cache, and a
completed wait gives it back in io_futex_complete().  A wait that fails
to arm kfree()s it instead.

The usual failure is -EAGAIN from futex_wait_setup(): the futex word no
longer holds the expected value.  That is routine for a waiter that
re-arms with the value it last read, since the word changes whenever
there is something to wake for.  Each such failure takes an object out
of the cache for good, so the cache drains and from then on waits
allocate.

The cache exists so that arming a wait doesn't have to go to the slab
allocator, and that matters most to the waiters that re-arm the most:
event loops, often real-time ones.  A kmalloc() or kfree() that misses
the per-CPU slab takes the node's list_lock, shared with every other
user of that kmalloc size on the system, and may go on to the page
allocator.

On PREEMPT_RT list_lock is a sleeping lock.  A real-time task that
finds it held waits for the holder, which may be any task on any CPU,
to get to run and release it.  Priority inheritance boosts the holder
only up to the waiter's priority, so the waiter still sits behind
whatever outranks that on the holder's CPU, such as threaded interrupt
handlers.  Re-arming a futex wait should not depend on any of that.

Put the data back in the cache on that path too, while the submit lock
still protects it.

In an event loop re-arming one IORING_OP_FUTEX_WAIT per message under
load, 0.39 arms per message failed with -EAGAIN, and io_futex_wait() did
0.39 kmalloc()s and kfree()s per message.  With this change, 100,000
waits that all fail with -EAGAIN do one kmalloc() in total.

Fixes: 194bb58c6090 ("io_uring: add support for futex wake and wait")
Assisted-by: LLM
Signed-off-by: Austin Schuh <austin.linux@gmail.com>
---

Notes:
    Based on for-7.4/io_uring (5b01d5955ac2).  Built for arm64 with
    PREEMPT_RT, W=1, no new warnings.  The numbers are from an equivalent
    change on 6.18.39-rt on a Jetson Orin Nano, counting kmem:kmalloc/kfree
    by call site with bpftrace while a test loops on IORING_OP_FUTEX_WAIT
    with a value the futex word doesn't hold.  Happy to turn that into
    a liburing test if it's useful.

 io_uring/futex.c | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)

diff --git a/io_uring/futex.c b/io_uring/futex.c
index 9203998654ff..9e4e63ee7bc1 100644
--- a/io_uring/futex.c
+++ b/io_uring/futex.c
@@ -353,12 +353,16 @@ int io_futex_wait(struct io_kiocb *req, unsigned int issue_flags)
 	}
 
 done_unlock:
+	if (ifd) {
+		/* As in io_futex_complete(), or failed arms would drain it. */
+		io_cache_free(&ctx->futex_cache, ifd);
+		io_req_async_data_clear(req, 0);
+	}
 	io_ring_submit_unlock(ctx, issue_flags);
 done:
 	if (ret < 0)
 		req_set_fail(req);
 	io_req_set_res(req, ret, 0);
-	io_req_async_data_free(req);
 	return IOU_COMPLETE;
 }
 

base-commit: 5b01d5955ac206e38097f777eff6ab2a8b948c41
-- 
2.47.3

Re: [PATCH] io_uring/futex: return a wait's data to the cache when it fails to arm

[Jens Axboe]

On 10/19/26 5:12 PM, Austin Schuh wrote:
> io_futex_wait() takes its io_futex_data from ctx->futex_cache, and a
> completed wait gives it back in io_futex_complete().  A wait that fails
> to arm kfree()s it instead.
> 
> The usual failure is -EAGAIN from futex_wait_setup(): the futex word no
> longer holds the expected value.  That is routine for a waiter that
> re-arms with the value it last read, since the word changes whenever
> there is something to wake for.  Each such failure takes an object out
> of the cache for good, so the cache drains and from then on waits
> allocate.
> 
> The cache exists so that arming a wait doesn't have to go to the slab
> allocator, and that matters most to the waiters that re-arm the most:
> event loops, often real-time ones.  A kmalloc() or kfree() that misses
> the per-CPU slab takes the node's list_lock, shared with every other
> user of that kmalloc size on the system, and may go on to the page
> allocator.
> 
> On PREEMPT_RT list_lock is a sleeping lock.  A real-time task that
> finds it held waits for the holder, which may be any task on any CPU,
> to get to run and release it.  Priority inheritance boosts the holder
> only up to the waiter's priority, so the waiter still sits behind
> whatever outranks that on the holder's CPU, such as threaded interrupt
> handlers.  Re-arming a futex wait should not depend on any of that.
> 
> Put the data back in the cache on that path too, while the submit lock
> still protects it.
> 
> In an event loop re-arming one IORING_OP_FUTEX_WAIT per message under
> load, 0.39 arms per message failed with -EAGAIN, and io_futex_wait() did
> 0.39 kmalloc()s and kfree()s per message.  With this change, 100,000
> waits that all fail with -EAGAIN do one kmalloc() in total.

Patches like this have been posted before, and my question is
always "is this a common case?"? Reading this:

> Notes:
>     Based on for-7.4/io_uring (5b01d5955ac2).  Built for arm64 with
>     PREEMPT_RT, W=1, no new warnings.  The numbers are from an equivalent
>     change on 6.18.39-rt on a Jetson Orin Nano, counting kmem:kmalloc/kfree
>     by call site with bpftrace while a test loops on IORING_OP_FUTEX_WAIT
>     with a value the futex word doesn't hold.  Happy to turn that into
>     a liburing test if it's useful.

makes me very much think "nope, not at all", as it's a deliberately
failing test case. And yes if you keep issuing failing waits,
then it'll drain the cache. The cache is for the hot FAST path,
which is not an error path.

Hmm?

-- 
Jens Axboe

Re: [PATCH] io_uring/futex: return a wait's data to the cache when it fails to arm

[Austin Schuh]

On Mon, Oct 19, 2026 at 5:54 PM Jens Axboe <axboe@kernel.dk> wrote:
> Patches like this have been posted before, and my question is
> always "is this a common case?"? Reading this:
>
> > Notes:
> >     Based on for-7.4/io_uring (5b01d5955ac2).  Built for arm64 with
> >     PREEMPT_RT, W=1, no new warnings.  The numbers are from an equivalent
> >     change on 6.18.39-rt on a Jetson Orin Nano, counting kmem:kmalloc/kfree
> >     by call site with bpftrace while a test loops on IORING_OP_FUTEX_WAIT
> >     with a value the futex word doesn't hold.  Happy to turn that into
> >     a liburing test if it's useful.
>
> makes me very much think "nope, not at all", as it's a deliberately
> failing test case. And yes if you keep issuing failing waits,
> then it'll drain the cache. The cache is for the hot FAST path,
> which is not an error path.

Fair on the test case, it only showed the mechanism.  But for an event
loop, -EAGAIN from a futex wait isn't an error path.  Right before
io_uring_enter() re-arms the wait that just fired, the event loop loads
its futex word.  If the word has already changed, it handles the
new message and re-arms with the new value instead.  A message published
between that load and futex_wait_setup() checking the word makes the
wait return -EAGAIN, and that is how the loop learns there is more to
do.  The window can be shrunk, but not removed; nearly all of what is
left is inside io_uring_enter().

Here's how often that happens, measured on a Jetson Orin Nano running
for-7.4/io_uring with PREEMPT_RT.  The event loops are AOS (a robotics
framework).  Each one keeps one IORING_OP_FUTEX_WAIT armed per channel,
on the channel's queue index, and re-arms it after every wake.  One
channel: a SCHED_FIFO 40 publisher at 2 kHz with a SCHED_FIFO 40
subscriber, then low-priority processes publishing and subscribing on
the same channel as well, then interrupt and system load on top.
bpftrace on fexit:io_futex_wait counted, per process, the waits armed
(IORING_OP_FUTEX_WAIT issued) and how many of them returned -EAGAIN
instead of sleeping.  Two minutes each:

                             real-time subscriber    low-priority procs
                 messages/s  arms/s  -EAGAIN  rate   arms/s  -EAGAIN  rate
  pair only           2,000   2,000        0    0%        -        -     -
  + low-priority     12,200  11,020    1,292   12%   14,760    1,229    8%
  + load             12,200   9,830    2,797   28%    4,120      377    9%

(messages/s is everything published on the channel, arms/s and
-EAGAIN are per second.  A subscriber arms fewer times than messages
arrive because one wake can cover several messages.  Under load the
low-priority processes arm less often because the load takes their CPUs.)

In all, 2,520 and 3,170 arms a second failed in the last two runs.
bpftrace also read the futex word as each failed arm returned.  Of the
683,303 failures across both runs, 88% found the word exactly one
message past the value the wait was armed with.  So the arm raced a
single publish, not a backlog.  Load makes it more likely because the
stretch between that load and the kernel's check gets longer.

Timed from syscall entry to io_futex_wait() returning, 74% of the
real-time subscriber's failures came within 6 us.  For the low-priority
processes, which can be preempted in there, a third took more than
40 us.

Without the patch, each of those failures is a kfree() and then a
kmalloc() on the next arm: about 3,000 a second here, from a slab cache
shared system-wide, and more the busier the system gets.  On PREEMPT_RT
the slab slow path's list_lock is a sleeping lock, which is where this
started for us.  The patch doesn't change the armed path at all; it
only stops the cache from draining.

I can send a v2 with these numbers and this explanation in the commit
message, in place of the synthetic test.

What I'm after is a futex wait an event loop can reliably re-arm without
latency spikes.  Is there a better way to avoid the allocation here?  If
you'd rather not have every failure refill the cache, I can limit it to
-EAGAIN, the one failure an event loop hits in normal operation, and
keep the kfree() for the rest.

Thanks for the prompt reply!
  Austin

Re: [PATCH] io_uring/futex: return a wait's data to the cache when it fails to arm

[Jens Axboe]

On 10/19/26 11:29 PM, Austin Schuh wrote:
> On Mon, Oct 19, 2026 at 5:54?PM Jens Axboe <axboe@kernel.dk> wrote:
>> Patches like this have been posted before, and my question is
>> always "is this a common case?"? Reading this:
>>
>>> Notes:
>>>     Based on for-7.4/io_uring (5b01d5955ac2).  Built for arm64 with
>>>     PREEMPT_RT, W=1, no new warnings.  The numbers are from an equivalent
>>>     change on 6.18.39-rt on a Jetson Orin Nano, counting kmem:kmalloc/kfree
>>>     by call site with bpftrace while a test loops on IORING_OP_FUTEX_WAIT
>>>     with a value the futex word doesn't hold.  Happy to turn that into
>>>     a liburing test if it's useful.
>>
>> makes me very much think "nope, not at all", as it's a deliberately
>> failing test case. And yes if you keep issuing failing waits,
>> then it'll drain the cache. The cache is for the hot FAST path,
>> which is not an error path.
> 
> Fair on the test case, it only showed the mechanism.  But for an event
> loop, -EAGAIN from a futex wait isn't an error path.  Right before
> io_uring_enter() re-arms the wait that just fired, the event loop loads
> its futex word.  If the word has already changed, it handles the
> new message and re-arms with the new value instead.  A message published
> between that load and futex_wait_setup() checking the word makes the
> wait return -EAGAIN, and that is how the loop learns there is more to
> do.  The window can be shrunk, but not removed; nearly all of what is
> left is inside io_uring_enter().
> 
> Here's how often that happens, measured on a Jetson Orin Nano running
> for-7.4/io_uring with PREEMPT_RT.  The event loops are AOS (a robotics
> framework).  Each one keeps one IORING_OP_FUTEX_WAIT armed per channel,
> on the channel's queue index, and re-arms it after every wake.  One
> channel: a SCHED_FIFO 40 publisher at 2 kHz with a SCHED_FIFO 40
> subscriber, then low-priority processes publishing and subscribing on
> the same channel as well, then interrupt and system load on top.
> bpftrace on fexit:io_futex_wait counted, per process, the waits armed
> (IORING_OP_FUTEX_WAIT issued) and how many of them returned -EAGAIN
> instead of sleeping.  Two minutes each:
> 
>                              real-time subscriber    low-priority procs
>                  messages/s  arms/s  -EAGAIN  rate   arms/s  -EAGAIN  rate
>   pair only           2,000   2,000        0    0%        -        -     -
>   + low-priority     12,200  11,020    1,292   12%   14,760    1,229    8%
>   + load             12,200   9,830    2,797   28%    4,120      377    9%
> 
> (messages/s is everything published on the channel, arms/s and
> -EAGAIN are per second.  A subscriber arms fewer times than messages
> arrive because one wake can cover several messages.  Under load the
> low-priority processes arm less often because the load takes their CPUs.)
> 
> In all, 2,520 and 3,170 arms a second failed in the last two runs.
> bpftrace also read the futex word as each failed arm returned.  Of the
> 683,303 failures across both runs, 88% found the word exactly one
> message past the value the wait was armed with.  So the arm raced a
> single publish, not a backlog.  Load makes it more likely because the
> stretch between that load and the kernel's check gets longer.
> 
> Timed from syscall entry to io_futex_wait() returning, 74% of the
> real-time subscriber's failures came within 6 us.  For the low-priority
> processes, which can be preempted in there, a third took more than
> 40 us.
> 
> Without the patch, each of those failures is a kfree() and then a
> kmalloc() on the next arm: about 3,000 a second here, from a slab cache
> shared system-wide, and more the busier the system gets.  On PREEMPT_RT
> the slab slow path's list_lock is a sleeping lock, which is where this
> started for us.  The patch doesn't change the armed path at all; it
> only stops the cache from draining.
> 
> I can send a v2 with these numbers and this explanation in the commit
> message, in place of the synthetic test.
> 
> What I'm after is a futex wait an event loop can reliably re-arm without
> latency spikes.  Is there a better way to avoid the allocation here?  If
> you'd rather not have every failure refill the cache, I can limit it to
> -EAGAIN, the one failure an event loop hits in normal operation, and
> keep the kfree() for the rest.

That's great info, and I agree it differs from the usual forced
error path. Futexes are inherently racy. Thanks, I'll get this
queued up.

-- 
Jens Axboe

Re: [PATCH] io_uring/futex: return a wait's data to the cache when it fails to arm

[Jens Axboe]

On Mon, 19 Oct 2026 16:12:43 -0700, Austin Schuh wrote:
> io_futex_wait() takes its io_futex_data from ctx->futex_cache, and a
> completed wait gives it back in io_futex_complete().  A wait that fails
> to arm kfree()s it instead.
> 
> The usual failure is -EAGAIN from futex_wait_setup(): the futex word no
> longer holds the expected value.  That is routine for a waiter that
> re-arms with the value it last read, since the word changes whenever
> there is something to wake for.  Each such failure takes an object out
> of the cache for good, so the cache drains and from then on waits
> allocate.
> 
> [...]

Applied, thanks!

[1/1] io_uring/futex: return a wait's data to the cache when it fails to arm
      commit: 02caa5922184e99da04b37fd3336066625ea8e2e

Best regards,
-- 
Jens Axboe