From: Jens Axboe <axboe@kernel.dk>
To: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Cc: stable <stable@vger.kernel.org>
Subject: Queued patches for 6.18-stable
Date: Wed, 28 Oct 2026 07:52:22 -0600 [thread overview]
Message-ID: <4bd1a1a7-d824-4393-89fc-4a4e8d7f7f30@kernel.dk> (raw)
[-- Attachment #1: Type: text/plain, Size: 79 bytes --]
Hi,
Like the 6.1 series, but for 6.18-stable.
Please apply!
--
Jens Axboe
[-- Attachment #2: 0048-io_uring-keep-the-drain-active-when-a-deferred-drain.patch --]
[-- Type: text/x-patch, Size: 3417 bytes --]
From 06fcc2219171a09c246db8eb8c6314186cb4d54d Mon Sep 17 00:00:00 2001
From: Jens Axboe <axboe@kernel.dk>
Date: Tue, 27 Oct 2026 12:02:04 -0600
Subject: [PATCH 48/48] io_uring: keep the drain active when a deferred drain
request is started
Commit 2295613cd3713beffc8665e956135a5d48dfc208 upstream.
ecb6817b769f ends the drain when the dispatched drain request has posted
its CQE and nothing else is deferred. But the deferred list is processed
in between, and if the request it starts is itself a drain request, the
drain is over before that one has even run. A request submitted after
that point then runs ahead of it:
1. P0 in flight, D1 and D2 (both IOSQE_IO_DRAIN) deferred behind it
2. P0 completes, D1 is started
3. D1 completes, drain_done is set, D2 is started, the list is empty
4. IO_RING_F_DRAIN_ACTIVE is cleared with D2 still in flight
Anything submitted now bypasses the drain. The same happens with D2 as
the head of a link, as those aren't tracked in ->drain_req at all.
Have io_queue_deferred() report whether it started a drain request, and
don't end the drain if it did. Starting non-drain requests and ending it
is still fine, they were only waiting on the drain request that just
completed.
[ 6.18: ->drain_active is a bool here ]
Fixes: ecb6817b769f ("io_uring: end the drain when the drain request completes")
Reported-by: Karl Mehltretter <kmehltretter@gmail.com>
Link: https://lore.kernel.org/io-uring/20261027043323.98916-1-kmehltretter@gmail.com/
Cc: stable@vger.kernel.org
Signed-off-by: Jens Axboe <axboe@kernel.dk>
---
io_uring/io_uring.c | 20 +++++++++++++-------
1 file changed, 13 insertions(+), 7 deletions(-)
diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c
index 2f76a68875da..04a90824c190 100644
--- a/io_uring/io_uring.c
+++ b/io_uring/io_uring.c
@@ -545,9 +545,9 @@ unsigned io_linked_nr(struct io_kiocb *req)
return nr;
}
-static __cold noinline void io_queue_deferred(struct io_ring_ctx *ctx)
+static __cold noinline bool io_queue_deferred(struct io_ring_ctx *ctx)
{
- bool drain_seen = false, first = true;
+ bool drain_seen = false, first = true, drain_queued = false;
lockdep_assert_held(&ctx->uring_lock);
__io_req_caches_free(ctx);
@@ -561,17 +561,21 @@ static __cold noinline void io_queue_deferred(struct io_ring_ctx *ctx)
drain_seen |= de->req->flags & REQ_F_IO_DRAIN;
if ((drain_seen || first) && ctx->nr_req_allocated != ctx->nr_drained)
- return;
+ return drain_queued;
list_del_init(&de->list);
ctx->nr_drained -= io_linked_nr(de->req);
- /* it runs alone, track its completion unless it's a link */
- if ((de->req->flags & REQ_F_IO_DRAIN) && !de->req->link)
- ctx->drain_req = de->req;
+ if (de->req->flags & REQ_F_IO_DRAIN) {
+ drain_queued = true;
+ /* it runs alone, track its completion unless it's a link */
+ if (!de->req->link)
+ ctx->drain_req = de->req;
+ }
io_req_task_queue(de->req);
kfree(de);
first = false;
}
+ return drain_queued;
}
void __io_commit_cqring_flush(struct io_ring_ctx *ctx)
@@ -1677,7 +1681,9 @@ void __io_submit_flush_completions(struct io_ring_ctx *ctx)
}
if (unlikely(ctx->drain_active)) {
- io_queue_deferred(ctx);
+ /* a drain request started from the list keeps the drain going */
+ if (io_queue_deferred(ctx))
+ drain_done = false;
if (drain_done && list_empty(&ctx->defer_list))
ctx->drain_active = false;
}
--
2.56.0
[-- Attachment #3: 0047-io_uring-end-the-drain-when-the-drain-request-comple.patch --]
[-- Type: text/x-patch, Size: 5166 bytes --]
From 5a0667d1f00af26ca85452de5d5f436b23d65812 Mon Sep 17 00:00:00 2001
From: Jens Axboe <axboe@kernel.dk>
Date: Mon, 19 Oct 2026 13:30:51 -0600
Subject: [PATCH 47/48] io_uring: end the drain when the drain request
completes
Commit ecb6817b769fd7d71af2c9f446188649020b2e53 upstream.
Since requests are drained based on allocated requests, the drain stays
active after the drain request has posted its CQE, until it is freed and
a new request finds the defer list empty. A drain request run by io-wq
is freed only after the worker drops its reference, so a request
submitted after the drain completed can still be deferred behind it,
and get issued much later than the application expects.
A drain request dispatched from the defer list runs alone. Track it,
and end the drain when its CQE is flushed and nothing else is deferred.
Have io-wq complete drain requests through task_work, so that this
happens under ->uring_lock before the CQE is visible. Linked drain
requests keep waiting until the link is freed.
Reported-by: Wei Gao <wegao@suse.com>
Link: https://lore.kernel.org/io-uring/20261018070154.16944-1-wegao@suse.com/
[ 6.18: ->drain_active and ->lockless_cq are bools here ]
Fixes: 8fb7aee05591 ("io_uring: drain based on allocates reqs")
Cc: stable@vger.kernel.org
Tested-by: Wei Gao <wegao@suse.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
---
include/linux/io_uring_types.h | 2 ++
io_uring/io_uring.c | 42 +++++++++++++++++++++++++++++++---
2 files changed, 41 insertions(+), 3 deletions(-)
diff --git a/include/linux/io_uring_types.h b/include/linux/io_uring_types.h
index 2576f72eada5..bdfc3cf7d989 100644
--- a/include/linux/io_uring_types.h
+++ b/include/linux/io_uring_types.h
@@ -450,6 +450,8 @@ struct io_ring_ctx {
atomic_t poll_wq_rcu_pending;
struct list_head defer_list;
unsigned nr_drained;
+ /* drain request dispatched from ->defer_list, until it completes */
+ struct io_kiocb *drain_req;
/* protected by ->completion_lock */
unsigned nr_req_allocated;
diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c
index c1a3e9162e85..2f76a68875da 100644
--- a/io_uring/io_uring.c
+++ b/io_uring/io_uring.c
@@ -551,6 +551,9 @@ static __cold noinline void io_queue_deferred(struct io_ring_ctx *ctx)
lockdep_assert_held(&ctx->uring_lock);
__io_req_caches_free(ctx);
+ /* nothing outside of ->defer_list is alive */
+ if (ctx->nr_req_allocated == ctx->nr_drained)
+ ctx->drain_req = NULL;
while (!list_empty(&ctx->defer_list)) {
struct io_defer_entry *de = list_first_entry(&ctx->defer_list,
@@ -562,6 +565,9 @@ static __cold noinline void io_queue_deferred(struct io_ring_ctx *ctx)
list_del_init(&de->list);
ctx->nr_drained -= io_linked_nr(de->req);
+ /* it runs alone, track its completion unless it's a link */
+ if ((de->req->flags & REQ_F_IO_DRAIN) && !de->req->link)
+ ctx->drain_req = de->req;
io_req_task_queue(de->req);
kfree(de);
first = false;
@@ -1011,9 +1017,11 @@ static void io_req_complete_post(struct io_kiocb *req, unsigned issue_flags)
/*
* Handle special CQ sync cases via task_work. DEFER_TASKRUN requires
- * the submitter task context, IOPOLL protects with uring_lock.
+ * the submitter task context, IOPOLL protects with uring_lock. A drain
+ * request ends the drain under uring_lock as its CQE is posted.
*/
- if (ctx->lockless_cq || (req->flags & REQ_F_REISSUE)) {
+ if (ctx->lockless_cq ||
+ (req->flags & (REQ_F_REISSUE | REQ_F_IO_DRAIN))) {
defer_complete:
req->io_task_work.func = io_req_task_complete;
io_req_task_work_add(req);
@@ -1612,11 +1620,33 @@ static void io_free_batch_list(struct io_ring_ctx *ctx,
} while (node);
}
+/*
+ * The dispatched drain request ran alone, and anything submitted after it was
+ * deferred. Once it completed, new requests may run if nothing is waiting.
+ */
+static __cold bool io_drain_req_completed(struct io_ring_ctx *ctx)
+{
+ struct io_wq_work_node *node;
+
+ __wq_list_for_each(node, &ctx->submit_state.compl_reqs) {
+ struct io_kiocb *req = container_of(node, struct io_kiocb,
+ comp_list);
+
+ /* a requeued request hasn't completed */
+ if (req == ctx->drain_req && !(req->flags & REQ_F_REISSUE)) {
+ ctx->drain_req = NULL;
+ return true;
+ }
+ }
+ return false;
+}
+
void __io_submit_flush_completions(struct io_ring_ctx *ctx)
__must_hold(&ctx->uring_lock)
{
struct io_submit_state *state = &ctx->submit_state;
struct io_wq_work_node *node;
+ bool drain_done = false;
__io_cq_lock(ctx);
__wq_list_for_each(node, &state->compl_reqs) {
@@ -1638,13 +1668,19 @@ void __io_submit_flush_completions(struct io_ring_ctx *ctx)
}
__io_cq_unlock_post(ctx);
+ if (unlikely(ctx->drain_active) && ctx->drain_req)
+ drain_done = io_drain_req_completed(ctx);
+
if (!wq_list_empty(&state->compl_reqs)) {
io_free_batch_list(ctx, state->compl_reqs.first);
INIT_WQ_LIST(&state->compl_reqs);
}
- if (unlikely(ctx->drain_active))
+ if (unlikely(ctx->drain_active)) {
io_queue_deferred(ctx);
+ if (drain_done && list_empty(&ctx->defer_list))
+ ctx->drain_active = false;
+ }
ctx->submit_state.cq_flush = false;
}
--
2.56.0
[-- Attachment #4: 0046-io_uring-waitid-prevent-cancel-reference-wrap.patch --]
[-- Type: text/x-patch, Size: 2403 bytes --]
From be0be0ef20bc5244f77ba8564ac7fea4d252a81d Mon Sep 17 00:00:00 2001
From: Junyeong Lee <jylee5498@gmail.com>
Date: Tue, 20 Oct 2026 00:21:00 +0900
Subject: [PATCH 46/48] io_uring/waitid: prevent cancel reference wrap
Commit 98787d3fa4e99564d2b2bc21255b21f670c450d0 upstream.
io_waitid packs the cancellation state into bit 31 of ->refs and uses
bits 30:0 as a 31-bit reference counter:
#define IO_WAITID_CANCEL_FLAG BIT(31)
#define IO_WAITID_REF_MASK GENMASK(30, 0)
__io_waitid_cancel() marks a request canceled, then increments the low
bits of ->refs to claim completion ownership. When another path owns the
request, the increment remains even though cancellation returns false.
The request remains discoverable by cancellation, so repeated failed
attempts can wrap the 31-bit reference field to zero. A later
cancellation can claim ownership and complete the request a second time.
Use atomic_fetch_or() when setting the cancel flag and inspect the old
flag value. The first cancel attempt continues to the existing ownership
increment, preserving its wake and issuer handoff semantics. Later
attempts see the cancel flag and return without incrementing the
reference field.
This makes repeated failed cancellation attempts idempotent while
leaving the first-cancel state transitions unchanged.
Fixes: f31ecf671ddc ("io_uring: add IORING_OP_WAITID support")
Cc: stable@vger.kernel.org
Assisted-by: LLM
Signed-off-by: Junyeong Lee <jylee5498@gmail.com>
Link: https://patch.msgid.link/20261019152100.63565-1-jylee5498@gmail.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
---
io_uring/waitid.c | 7 +++++--
1 file changed, 5 insertions(+), 2 deletions(-)
diff --git a/io_uring/waitid.c b/io_uring/waitid.c
index 2ba2369681d0..172cf770b29b 100644
--- a/io_uring/waitid.c
+++ b/io_uring/waitid.c
@@ -154,9 +154,12 @@ static bool __io_waitid_cancel(struct io_kiocb *req, bool copy_si)
/*
* Mark us canceled regardless of ownership. This will prevent a
- * potential retry from a spurious wakeup.
+ * potential retry from a spurious wakeup. Only the first cancel
+ * attempt should take a reference for the ownership handoff.
*/
- atomic_or(IO_WAITID_CANCEL_FLAG, &iw->refs);
+ if (atomic_fetch_or(IO_WAITID_CANCEL_FLAG, &iw->refs) &
+ IO_WAITID_CANCEL_FLAG)
+ return false;
/* claim ownership */
if (atomic_fetch_inc(&iw->refs) & IO_WAITID_REF_MASK)
--
2.56.0
[-- Attachment #5: 0045-io_uring-wait-wake-real-time-waiters-from-hard-inter.patch --]
[-- Type: text/x-patch, Size: 3854 bytes --]
From 316928bcb066b7e82b9e1a3504944599e21aaa98 Mon Sep 17 00:00:00 2001
From: Austin Schuh <austin.linux@gmail.com>
Date: Sat, 17 Oct 2026 10:17:26 -0700
Subject: [PATCH 45/48] io_uring/wait: wake real-time waiters from hard
interrupt context
Commit 5b01d5955ac206e38097f777eff6ab2a8b948c41 upstream.
Since commit cebf123c634a ("io_uring: implement our own schedule timeout
handling"), a CQ wait with a timeout sleeps on io_uring's own hrtimer
instead of an hrtimer sleeper. On PREEMPT_RT, an hrtimer that isn't
marked for hard interrupt expiry expires in the per-CPU ktimers thread,
which runs at SCHED_FIFO 1. A real-time task waiting in io_uring_enter()
is therefore not woken at its deadline while any other real-time task is
runnable on its CPU: the wakeup waits until every one of them has blocked.
hrtimer sleepers, which schedule_hrtimeout_range_clock() used to give
io_uring, don't have this problem: __hrtimer_setup_sleeper() marks the
timer hard when the sleeping task is in a real-time or deadline class.
Both of io_uring's callbacks are fine in hard interrupt context:
io_cqring_timer_wakeup() sets a flag and calls wake_up_process(), and
io_cqring_min_timer_wakeup() only reads ring state, under RCU, before
doing the same or re-arming the timer. Apply the sleeper rule to them.
Measured on a Jetson Orin Nano running 6.18.39-rt with this change
backported: a SCHED_FIFO 40 task waits in io_uring_enter() for 40,000
deadlines 500us apart, while a SCHED_FIFO 20 task runs bursts of up to
1ms on the same CPU. Before, 6,647 wakeups were more than 100us late,
the latest by 1,040us, and 2,758 deadlines passed without a wakeup at
all. After, none were more than 100us late and the latest was 18us.
This mirrors epoll: epoll_pwait2() waits on an hrtimer sleeper, so its
timeout is already hard for real-time tasks, and on the same setup it
was never more than 17us late.
[ 6.18: the wait code is still in io_uring.c ]
Fixes: cebf123c634a ("io_uring: implement our own schedule timeout handling")
Cc: stable@vger.kernel.org
Assisted-by: LLM
Signed-off-by: Austin Schuh <austin.linux@gmail.com>
Link: https://patch.msgid.link/20261017171726.601443-1-austin.linux@gmail.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
---
io_uring/io_uring.c | 16 +++++++++++++---
1 file changed, 13 insertions(+), 3 deletions(-)
diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c
index 8b16e4618506..c1a3e9162e85 100644
--- a/io_uring/io_uring.c
+++ b/io_uring/io_uring.c
@@ -48,6 +48,7 @@
#include <linux/uio.h>
#include <linux/bits.h>
+#include <linux/sched/rt.h>
#include <linux/sched/signal.h>
#include <linux/fs.h>
#include <linux/file.h>
@@ -2646,23 +2647,32 @@ static enum hrtimer_restart io_cqring_min_timer_wakeup(struct hrtimer *timer)
return io_cqring_timer_wakeup(timer);
}
+/* Same policy as for hrtimer sleepers, see __hrtimer_setup_sleeper(). */
+static enum hrtimer_mode io_cqring_timer_mode(void)
+{
+ if (IS_ENABLED(CONFIG_PREEMPT_RT) && rt_or_dl_task_policy(current))
+ return HRTIMER_MODE_ABS_HARD;
+ return HRTIMER_MODE_ABS;
+}
+
static int io_cqring_schedule_timeout(struct io_wait_queue *iowq,
clockid_t clock_id, ktime_t start_time)
{
+ enum hrtimer_mode mode = io_cqring_timer_mode();
ktime_t timeout;
if (iowq->min_timeout) {
timeout = ktime_add_ns(iowq->min_timeout, start_time);
hrtimer_setup_on_stack(&iowq->t, io_cqring_min_timer_wakeup, clock_id,
- HRTIMER_MODE_ABS);
+ mode);
} else {
timeout = iowq->timeout;
hrtimer_setup_on_stack(&iowq->t, io_cqring_timer_wakeup, clock_id,
- HRTIMER_MODE_ABS);
+ mode);
}
hrtimer_set_expires_range_ns(&iowq->t, timeout, 0);
- hrtimer_start_expires(&iowq->t, HRTIMER_MODE_ABS);
+ hrtimer_start_expires(&iowq->t, mode);
if (!READ_ONCE(iowq->hit_timeout))
schedule();
--
2.56.0
[-- Attachment #6: 0044-io_uring-net-keep-recv-buffer-select-length-within-M.patch --]
[-- Type: text/x-patch, Size: 1686 bytes --]
From 5ff7e8d5fe2ee9bf39cf9c4e8f68f4f7d76fe95a Mon Sep 17 00:00:00 2001
From: Tamaki Yanagawa <ty@000ty.net>
Date: Sun, 18 Oct 2026 04:22:32 +0000
Subject: [PATCH 44/48] io_uring/net: keep recv buffer select length within
MAX_RW_COUNT
Commit 96262cfab554cf11412de7a6247f0e82c94e130f upstream.
Since commit dcbd1c054848 ("io_uring/net: let io_recv_buf_select return
the length of the buffer region"), io_recv_buf_select() returns the
selected length as an int. For a non-bundle recv with sqe->len == 0 the
length comes from the ring provided buffer, which can be up to 4G, so a
buffer of 2G or more makes io_recv() fail the request with a bogus
negative result (e.g. -2147483648) instead of receiving into it. Clamp
the length to MAX_RW_COUNT, which import_ubuf() limits the transfer to
anyway.
Fixes: dcbd1c054848 ("io_uring/net: let io_recv_buf_select return the length of the buffer region")
Cc: stable@vger.kernel.org
Signed-off-by: Tamaki Yanagawa <ty@000ty.net>
Assisted-by: Claude:claude-opus-5-5
Link: https://patch.msgid.link/20261018042227.1105-1-ty@000ty.net
Signed-off-by: Jens Axboe <axboe@kernel.dk>
---
io_uring/net.c | 2 ++
1 file changed, 2 insertions(+)
diff --git a/io_uring/net.c b/io_uring/net.c
index 522be3edf51b..163ded47602d 100644
--- a/io_uring/net.c
+++ b/io_uring/net.c
@@ -1195,6 +1195,8 @@ static int io_recv_buf_select(struct io_kiocb *req, struct io_async_msghdr *kmsg
*sel = io_buffer_select(req, &len, sr->buf_group, issue_flags);
if (!sel->addr)
return -ENOBUFS;
+ /* ring provided buffers can be up to 4G, keep len an int */
+ len = min_t(size_t, len, MAX_RW_COUNT);
sr->buf = sel->addr;
sr->len = len;
map_ubuf:
--
2.56.0
[-- Attachment #7: 0043-io_uring-poll-don-t-loop-on-HUP-with-multiple-refere.patch --]
[-- Type: text/x-patch, Size: 2588 bytes --]
From d596270763c1254669248ae111c8759b86231308 Mon Sep 17 00:00:00 2001
From: Jens Axboe <axboe@kernel.dk>
Date: Fri, 16 Oct 2026 09:15:13 -0600
Subject: [PATCH 43/48] io_uring/poll: don't loop on HUP with multiple
references
Commit cf1d5c9d335e2c673811da641531da82cb01393a upstream.
io_poll_check_events() drops down to one reference when a multishot
request sees HUP with more than one, so that the loop goes around once
more. It did so by subtracting one less than it had. With the cmpxchg
that only takes the local count down: the exchange fails, the next pass
reads the unchanged count, and goes through the same motions again. The
loop then only ends once io_poll_issue() stops the request.
Drop the extra references from the counter instead, and loop once more
without trying the exchange. The next pass sees one reference, like it
did before, and ends unless a wake up came in.
Fixes: 937d38dcacd6 ("io_uring/poll: avoid retry wakeup loss when dropping ownership")
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
---
io_uring/poll.c | 16 ++++++++++++----
1 file changed, 12 insertions(+), 4 deletions(-)
diff --git a/io_uring/poll.c b/io_uring/poll.c
index 87f13b4ee06b..48381f0cbe32 100644
--- a/io_uring/poll.c
+++ b/io_uring/poll.c
@@ -223,12 +223,14 @@ static inline void io_poll_execute(struct io_kiocb *req, int res,
*/
static int io_poll_check_events(struct io_kiocb *req, io_tw_token_t tw)
{
+ bool again;
int v;
if (unlikely(tw.cancel))
return -ECANCELED;
do {
+ again = false;
v = atomic_read(&req->poll_refs);
if (unlikely(v != 1)) {
@@ -290,9 +292,15 @@ static int io_poll_check_events(struct io_kiocb *req, io_tw_token_t tw)
} else {
int ret;
- /* multiple refs and HUP, ensure we loop once more */
- if ((req->cqe.res & (POLLHUP | POLLRDHUP)) && v != 1)
- v--;
+ /*
+ * Multiple refs and HUP, ensure we loop once more. Drop
+ * down to one ref now, so the next pass ends unless a
+ * wake up came in.
+ */
+ if ((req->cqe.res & (POLLHUP | POLLRDHUP)) && v != 1) {
+ atomic_sub(v - 1, &req->poll_refs);
+ again = true;
+ }
ret = io_poll_issue(req, tw);
if (ret == IOU_COMPLETE)
@@ -311,7 +319,7 @@ static int io_poll_check_events(struct io_kiocb *req, io_tw_token_t tw)
* task_work while we were executing it.
*/
v &= IO_POLL_REF_MASK;
- } while (atomic_cmpxchg(&req->poll_refs, v, 0) != v);
+ } while (again || atomic_cmpxchg(&req->poll_refs, v, 0) != v);
io_napi_add(req);
return IOU_POLL_NO_ACTION;
--
2.56.0
[-- Attachment #8: 0042-io_uring-timeout-don-t-lose-a-cancel-as-io-wq-arms-a.patch --]
[-- Type: text/x-patch, Size: 3791 bytes --]
From f4711d5bfba12514d6f64180aaef0019aa4e7d63 Mon Sep 17 00:00:00 2001
From: Jens Axboe <axboe@kernel.dk>
Date: Thu, 15 Oct 2026 16:38:45 -0600
Subject: [PATCH 42/48] io_uring/timeout: don't lose a cancel as io-wq arms a
timeout
Commit 89f193dc3ef462059387e9691bf6af4b1808379a upstream.
An IORING_OP_TIMEOUT run by io-wq, because of IOSQE_ASYNC, has the same
problem as poll, futex and waitid. io_wq_submit_work() looks at
IO_WQ_WORK_CANCEL once, before it issues the request. A cancel in between
flags the work, gets -EALREADY, and searches ->timeout_list in vain,
while the worker then adds the request and starts the timer:
io-wq worker io_try_cancel()
io_wq_submit_work()
IO_WQ_WORK_CANCEL isn't set
io_async_cancel_one()
sets IO_WQ_WORK_CANCEL
= -EALREADY
io_timeout_cancel() = -ENOENT
io_timeout()
adds the request to ->timeout_list
hrtimer_start()
= IOU_ISSUE_SKIP_COMPLETE
Nobody looks at the flag again, and the timeout runs its course. At ring
exit, io_uring_try_cancel_requests() cancels io-wq work and then kills
the timeouts the same way, so an exiting task can get stuck for the full
duration of such a timeout.
Look at the flag under ->timeout_lock, which the search takes as well,
and fail the request if it is set. If the search came first, the flag is
visible. If it comes later, it finds the request on the list.
Fixes: ce35a47a3a02 ("io_uring: add IOSQE_ASYNC")
Signed-off-by: Jens Axboe <axboe@kernel.dk>
---
io_uring/io_uring.h | 14 ++++++++++----
io_uring/timeout.c | 7 +++++++
2 files changed, 17 insertions(+), 4 deletions(-)
diff --git a/io_uring/io_uring.h b/io_uring/io_uring.h
index 4b3acbb39436..922bdf68aec8 100644
--- a/io_uring/io_uring.h
+++ b/io_uring/io_uring.h
@@ -419,16 +419,22 @@ static inline void io_ring_submit_lock(struct io_ring_ctx *ctx,
* cancellations find it, rather than completing it. Whoever tried to cancel
* the request before that found it running. It has flagged the work item,
* has searched the list in vain, and won't come back. To be called with
- * ->uring_lock held, which the search needs as well, once the request is on
- * the list: if the search came first, the flag is visible.
+ * the lock the search takes held: if the search came first, the flag is
+ * visible. For most lists that is ->uring_lock, timeouts use ->timeout_lock.
*/
+static inline bool __io_wq_cancel_seen(struct io_kiocb *req,
+ unsigned int issue_flags)
+{
+ return (issue_flags & IO_URING_F_IOWQ) &&
+ (atomic_read(&req->work.flags) & IO_WQ_WORK_CANCEL);
+}
+
static inline bool io_wq_cancel_seen(struct io_kiocb *req,
unsigned int issue_flags)
{
lockdep_assert_held(&req->ctx->uring_lock);
- return (issue_flags & IO_URING_F_IOWQ) &&
- (atomic_read(&req->work.flags) & IO_WQ_WORK_CANCEL);
+ return __io_wq_cancel_seen(req, issue_flags);
}
static inline void io_commit_cqring(struct io_ring_ctx *ctx)
diff --git a/io_uring/timeout.c b/io_uring/timeout.c
index f1682a3f401b..dbbb9615ec9e 100644
--- a/io_uring/timeout.c
+++ b/io_uring/timeout.c
@@ -634,6 +634,13 @@ int io_timeout(struct io_kiocb *req, unsigned int issue_flags)
raw_spin_lock_irq(&ctx->timeout_lock);
+ /* the list is searched under ->timeout_lock, see io_wq_cancel_seen() */
+ if (__io_wq_cancel_seen(req, issue_flags)) {
+ raw_spin_unlock_irq(&ctx->timeout_lock);
+ io_req_task_queue_fail(req, -ECANCELED);
+ return IOU_ISSUE_SKIP_COMPLETE;
+ }
+
/*
* sqe->off holds how many events that need to occur for this
* timeout event to be satisfied. If it isn't set, then this is
--
2.56.0
[-- Attachment #9: 0041-io_uring-timeout-don-t-flush-with-the-sequence-behin.patch --]
[-- Type: text/x-patch, Size: 2193 bytes --]
From 6244d57e4d970eefc0ca4dd401cf7aadbabce889 Mon Sep 17 00:00:00 2001
From: Jens Axboe <axboe@kernel.dk>
Date: Thu, 15 Oct 2026 14:25:00 -0600
Subject: [PATCH 41/48] io_uring/timeout: don't flush with the sequence behind
the last flush
Commit 85f85d573ffa7772e1ad4504d0cec5113a5c5cf1 upstream.
io_flush_timeouts() measures the events since the last flush as the
distance of the current sequence from ->cq_last_tm_flush. A fired
timeout bumps ->cq_timeouts before its CQE is posted, so until it is
the sequence can be behind the last flush, and the distance wraps:
last flush: seq 100, cq_last_tm_flush = 100
Y: IORING_OP_TIMEOUT, off 50, target_seq 150
pure timeout A fires, cq_timeouts = 1
task_work posts A's CQE, cached_cq_tail = 101
pure timeout B fires before the flush, cq_timeouts = 2
io_flush_timeouts(): seq = 99, events_got = 0xffffffff
Y, and every sequence timeout behind it, is ended with -ETIME although
no event happened. Two timeouts expiring close together is all it takes,
a completion with IOSQE_CQE_SKIP_SUCCESS after a single one does it too.
Nothing but those in flight CQEs separates the two, so count from the
last flush when the sequence is behind it, as io_timeout() does.
Fixes: f010505b78a4 ("io_uring: flush timeouts that should already have expired")
Signed-off-by: Jens Axboe <axboe@kernel.dk>
---
io_uring/timeout.c | 8 ++++++++
1 file changed, 8 insertions(+)
diff --git a/io_uring/timeout.c b/io_uring/timeout.c
index 5a1b15420ef2..f1682a3f401b 100644
--- a/io_uring/timeout.c
+++ b/io_uring/timeout.c
@@ -132,6 +132,14 @@ __cold void io_flush_timeouts(struct io_ring_ctx *ctx)
raw_spin_lock_irq(&ctx->timeout_lock);
seq = READ_ONCE(ctx->cached_cq_tail) - atomic_read(&ctx->cq_timeouts);
+ /*
+ * A fired timeout bumps ->cq_timeouts before its CQE is posted, which
+ * leaves the sequence behind the last flush until it is. No event has
+ * happened then, see io_timeout().
+ */
+ if ((s32)(seq - ctx->cq_last_tm_flush) < 0)
+ seq = ctx->cq_last_tm_flush;
+
list_for_each_entry_safe(timeout, tmp, &ctx->timeout_list, list) {
struct io_kiocb *req = cmd_to_io_kiocb(timeout);
u32 events_needed, events_got;
--
2.56.0
[-- Attachment #10: 0040-io_uring-timeout-don-t-put-a-sequence-target-behind-.patch --]
[-- Type: text/x-patch, Size: 2205 bytes --]
From a2a0d36c6abacc05c508cbce2b052f1c8562f904 Mon Sep 17 00:00:00 2001
From: Jens Axboe <axboe@kernel.dk>
Date: Thu, 15 Oct 2026 14:18:22 -0600
Subject: [PATCH 40/48] io_uring/timeout: don't put a sequence target behind
the last flush
Commit fdfce905a58edea053ee7b68b5bd4348dd1b36e0 upstream.
io_timeout_fn() and io_kill_timeout() bump ->cq_timeouts when a timeout
fires, but its CQE is only posted later from task_work. Until then the
sequence io_timeout() computes is one below the last flush per fired
timeout. Now that ->cq_last_tm_flush is kept while sequence timeouts are
pending, a timeout armed in that window with a small offset gets a target
behind the base. Its distance from the base wraps, and it's never
completed by events, only by its timer. With DEFER_TASKRUN that window
is as wide as the application makes it, as submission runs before the
local task_work.
Nothing but those in flight CQEs separates the two, so use the last
flush as the base for the target when the sequence is behind it.
Fixes: c6c0b3eba88c ("io_uring/timeout: don't skip over reached sequences")
Signed-off-by: Jens Axboe <axboe@kernel.dk>
---
io_uring/timeout.c | 10 +++++++++-
1 file changed, 9 insertions(+), 1 deletion(-)
diff --git a/io_uring/timeout.c b/io_uring/timeout.c
index 7a5137379dca..5a1b15420ef2 100644
--- a/io_uring/timeout.c
+++ b/io_uring/timeout.c
@@ -637,7 +637,6 @@ int io_timeout(struct io_kiocb *req, unsigned int issue_flags)
}
tail = data_race(ctx->cached_cq_tail) - atomic_read(&ctx->cq_timeouts);
- timeout->target_seq = tail + off;
/*
* Update the last seq here in case io_flush_timeouts() hasn't, it only
@@ -651,6 +650,15 @@ int io_timeout(struct io_kiocb *req, unsigned int issue_flags)
ctx->cq_last_tm_flush = tail;
base = ctx->cq_last_tm_flush;
+ /*
+ * A fired timeout bumps ->cq_timeouts before its CQE is posted, which
+ * leaves the sequence behind the last flush until it is. Nothing else
+ * separates the two then, so count from the flush.
+ */
+ if ((s32)(tail - base) < 0)
+ tail = base;
+ timeout->target_seq = tail + off;
+
/*
* Insertion sort, ensuring the first entry in the list is always
* the one we need first.
--
2.56.0
[-- Attachment #11: 0039-io_uring-poll-avoid-retry-wakeup-loss-when-dropping-.patch --]
[-- Type: text/x-patch, Size: 1629 bytes --]
From ddfe414c96fc33b43719c8a00ae024f8c8e6c8b2 Mon Sep 17 00:00:00 2001
From: Juanlu Herrero <juanlu@fastmail.com>
Date: Tue, 13 Oct 2026 15:37:20 -0700
Subject: [PATCH 39/48] io_uring/poll: avoid retry wakeup loss when dropping
ownership
Commit 937d38dcacd675dc0572d6c9b6743960df3128ed upstream.
If a wakeup sets IO_POLL_RETRY_FLAG between the owning task consuming
the existing instance of IO_POLL_RETRY_FLAG and the loop check
of io_poll_check_events(), then we stall on processing the request
until an unrelated wakeup comes along.
Using atomic_cmpxchg instead of atomic_sub_return to guarantee both the
count v and poll_refs are 0 and no wake-ups are missed.
Cc: stable@vger.kernel.org
Fixes: a26a35e9019f ("io_uring: make poll refs more robust")
Signed-off-by: Juanlu Herrero <juanlu@fastmail.com>
Assisted-by: LLM
Suggested-by: Anand Natarajan <anand@meta.com>
Link: https://patch.msgid.link/20261013-pu-cmpxchg-v1-1-4ddc10fd8a03@fastmail.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
---
io_uring/poll.c | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/io_uring/poll.c b/io_uring/poll.c
index 00d9f6f35169..87f13b4ee06b 100644
--- a/io_uring/poll.c
+++ b/io_uring/poll.c
@@ -310,7 +310,8 @@ static int io_poll_check_events(struct io_kiocb *req, io_tw_token_t tw)
* Release all references, retry if someone tried to restart
* task_work while we were executing it.
*/
- } while (atomic_sub_return(v, &req->poll_refs) & IO_POLL_REF_MASK);
+ v &= IO_POLL_REF_MASK;
+ } while (atomic_cmpxchg(&req->poll_refs, v, 0) != v);
io_napi_add(req);
return IOU_POLL_NO_ACTION;
--
2.56.0
[-- Attachment #12: 0038-io_uring-look-up-the-file-before-queueing-to-io-wq.patch --]
[-- Type: text/x-patch, Size: 5486 bytes --]
From abb1e185c66c2d1600b539f449532e74c9f430dc Mon Sep 17 00:00:00 2001
From: Josef Bacik <josef@toxicpanda.com>
Date: Thu, 15 Oct 2026 19:31:07 +0000
Subject: [PATCH 38/48] io_uring: look up the file before queueing to io-wq
Commit 98bc044d5a5374f8d3605fe55a8e2df6d6fc3f82 upstream.
io_prep_async_work() decides by the file of a request what kind of work
it is for io-wq. I/O on regular files and block devices is bounded work,
limited to min(SQ entries, 4 * CPUs) workers, for anything else that may
block indefinitely the limit is RLIMIT_NPROC. Writes to a regular file
are hashed by the inode, so that they are run one after the other instead
of having a bunch of workers fight for the inode lock.
Since the file is only assigned when a request is issued, a request that
is queued without an attempt to issue it, REQ_F_FORCE_ASYNC, doesn't have
one at that point. io_prep_async_work() takes that for "not a regular
file or block device":
- Reads and writes end up as unbound work. 32 reads with IOSQE_ASYNC from
a slow block device on a ring with 4 SQ entries get 32 workers, not 4.
- Writes aren't hashed. Four buffered writes to one file get four
workers, three of which do nothing but wait for the lock.
The late assignment is for the benefit of links, where a request may use
a fixed file that one of its predecessors installs. By the time a request
gets to io_queue_iowq() these are done, it is as good as issued. Assign
the file there.
io_assign_file() moves up and is forced inline. With a third caller the
compiler may stop inlining it, and every request that is issued would
pay for a call.
IORING_OP_SPLICE is left as it is, unbound and not hashed. The file of
the request is where it writes to, which says nothing about how long it
takes: a splice from an empty pipe or a socket into a regular file waits
for its input. As bounded work, four of these on a ring with 4 SQ entries
keep all other bounded work of the task from running, and as hashed work
one of them blocks all writes to that file.
REQ_F_FORCE_ASYNC is not only set for IOSQE_ASYNC. What changes, and
brings back what was the case before the file assignment was deferred:
- Requests with IOSQE_ASYNC or IOSQE_IO_DRAIN, and all requests that are
submitted while a drain is active: reads and writes on regular files
and block devices count against the limit for bounded workers, including
the one set with IORING_REGISTER_IOWQ_MAX_WORKERS, so fewer of them run
at the same time.
- Of these, writes to one regular file are run one at a time, unless they
are O_DIRECT and the filesystem has FOP_DIO_PARALLEL_WRITE.
- IORING_OP_FALLOCATE and IORING_OP_FTRUNCATE, which always go to io-wq,
are run one at a time per file, and in turn with these writes.
- The descriptor or the fixed file slot is looked up when the request is
queued, not when a worker gets around to it, so closing or replacing it
in between no longer has an effect on the request.
Fixes: 6bf9c47a3989 ("io_uring: defer file assignment")
Assisted-by: LLM
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Link: https://patch.msgid.link/20261015-b4-io-uring-fixes-v1-43-c1556f5d7bf5@toxicpanda.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
---
io_uring/io_uring.c | 43 +++++++++++++++++++++++++++++--------------
1 file changed, 29 insertions(+), 14 deletions(-)
diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c
index 96e27af812d5..8b16e4618506 100644
--- a/io_uring/io_uring.c
+++ b/io_uring/io_uring.c
@@ -466,17 +466,46 @@ static void io_prep_async_link(struct io_kiocb *req)
}
}
+/* Called for every request that is issued: keep it inline */
+static __always_inline bool io_assign_file(struct io_kiocb *req,
+ const struct io_issue_def *def,
+ unsigned int issue_flags)
+{
+ if (req->file || !def->needs_file)
+ return true;
+
+ if (req->flags & REQ_F_FIXED_FILE)
+ req->file = io_file_get_fixed(req, req->cqe.fd, issue_flags);
+ else
+ req->file = io_file_get_normal(req, req->cqe.fd);
+
+ return !!req->file;
+}
+
static void io_queue_iowq(struct io_kiocb *req)
{
struct io_uring_task *tctx = req->tctx;
BUG_ON(!tctx);
+ lockdep_assert_held(&req->ctx->uring_lock);
if ((current->flags & PF_KTHREAD) || !tctx->io_wq) {
io_req_task_queue_fail(req, -ECANCELED);
return;
}
+ /*
+ * The file is looked up as late as possible, as it may be the result
+ * of an earlier request of the link. This one is up now, and what kind
+ * of work it is depends on the file. If that fails, leave the error
+ * handling to the worker.
+ *
+ * Not for a splice: whatever it writes to, it may wait for its input
+ * for ever. It has to remain unbound work that isn't hashed.
+ */
+ if (req->opcode != IORING_OP_SPLICE)
+ io_assign_file(req, &io_issue_defs[req->opcode], 0);
+
/* init ->work of the whole link before punting */
io_prep_async_link(req);
@@ -1830,20 +1859,6 @@ static __cold void io_drain_req(struct io_kiocb *req)
ctx->drain_active = false;
}
-static bool io_assign_file(struct io_kiocb *req, const struct io_issue_def *def,
- unsigned int issue_flags)
-{
- if (req->file || !def->needs_file)
- return true;
-
- if (req->flags & REQ_F_FIXED_FILE)
- req->file = io_file_get_fixed(req, req->cqe.fd, issue_flags);
- else
- req->file = io_file_get_normal(req, req->cqe.fd);
-
- return !!req->file;
-}
-
#define REQ_ISSUE_SLOW_FLAGS (REQ_F_CREDS | REQ_F_ARM_LTIMEOUT)
static inline int __io_issue_sqe(struct io_kiocb *req,
--
2.56.0
[-- Attachment #13: 0037-io_uring-timeout-don-t-end-a-linked-timeout-with-1.patch --]
[-- Type: text/x-patch, Size: 1858 bytes --]
From d6e52c8747e765606c3e50abfbd9a0d74fbe7614 Mon Sep 17 00:00:00 2001
From: Josef Bacik <josef@toxicpanda.com>
Date: Thu, 15 Oct 2026 19:31:06 +0000
Subject: [PATCH 37/48] io_uring/timeout: don't end a linked timeout with 1
Commit c74193430febdd120e35936dda3e5aeec3a54405 upstream.
A linked timeout that fires cancels the request it is linked to, and
completes with -ETIME if that worked out:
ret = io_try_cancel(req->tctx, &cd, 0);
...
io_req_set_res(req, ret ?: -ETIME, 0);
io_try_cancel() returns 0 or an error, except for requests that are parked
on ->futex_list or ->waitid_list. io_futex_cancel() and io_waitid_cancel()
return the number of requests that they have cancelled, and
io_try_cancel() hands it on.
So the linked timeout of an IORING_OP_FUTEX_WAIT, IORING_OP_FUTEX_WAITV or
IORING_OP_WAITID that it has cancelled completes with 1, each time.
The waitid test of liburing checks for that 1, and has to be changed to
expect -ETIME, which is what the linked timeout of any other request
completes with.
Fixes: f31ecf671ddc ("io_uring: add IORING_OP_WAITID support")
Fixes: 194bb58c6090 ("io_uring: add support for futex wake and wait")
Assisted-by: LLM
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Link: https://patch.msgid.link/20261015-b4-io-uring-fixes-v1-42-c1556f5d7bf5@toxicpanda.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
---
io_uring/timeout.c | 3 +++
1 file changed, 3 insertions(+)
diff --git a/io_uring/timeout.c b/io_uring/timeout.c
index f5fcda533425..7a5137379dca 100644
--- a/io_uring/timeout.c
+++ b/io_uring/timeout.c
@@ -342,6 +342,9 @@ static void io_req_task_link_timeout(struct io_tw_req tw_req, io_tw_token_t tw)
};
ret = io_try_cancel(req->tctx, &cd, 0);
+ /* a number of requests for futex and waitid */
+ if (ret > 0)
+ ret = 0;
} else {
ret = -ECANCELED;
}
--
2.56.0
[-- Attachment #14: 0036-io_uring-timeout-no-ABS-update-of-multishot-timeouts.patch --]
[-- Type: text/x-patch, Size: 3314 bytes --]
From 97b6125743e4fdb6b45de154de40c1cf1a581933 Mon Sep 17 00:00:00 2001
From: Josef Bacik <josef@toxicpanda.com>
Date: Thu, 15 Oct 2026 19:31:05 +0000
Subject: [PATCH 36/48] io_uring/timeout: no ABS update of multishot timeouts
Commit b088ae34d4da730dc647261b893a6eb622fe5e1f upstream.
IORING_TIMEOUT_MULTISHOT can't be combined with IORING_TIMEOUT_ABS, prep
rejects that, as the request is armed again with the same value each time
it fires. IORING_TIMEOUT_UPDATE doesn't check and takes an absolute time
for a multishot timeout. The first expiry is as asked for, but
io_timeout_complete() then arms the timer with ->mode of the original
request, i.e. it uses the absolute time as the interval. With
CLOCK_MONOTONIC the next expiry is an uptime away, with CLOCK_REALTIME
more than fifty years.
Fail such an update with -EINVAL, and do so before the timeout is
touched.
Fixes: ea97f6c8558e ("io_uring: add support for multishot timeouts")
Assisted-by: LLM
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Link: https://patch.msgid.link/20261015-b4-io-uring-fixes-v1-41-c1556f5d7bf5@toxicpanda.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
---
io_uring/timeout.c | 38 ++++++++++++++++++++++++++------------
1 file changed, 26 insertions(+), 12 deletions(-)
diff --git a/io_uring/timeout.c b/io_uring/timeout.c
index e0b454be8b42..f5fcda533425 100644
--- a/io_uring/timeout.c
+++ b/io_uring/timeout.c
@@ -278,22 +278,29 @@ static enum hrtimer_restart io_timeout_fn(struct hrtimer *timer)
return HRTIMER_NORESTART;
}
+static struct io_kiocb *io_timeout_find(struct io_ring_ctx *ctx,
+ struct io_cancel_data *cd)
+ __must_hold(&ctx->timeout_lock)
+{
+ struct io_timeout *timeout;
+
+ list_for_each_entry(timeout, &ctx->timeout_list, list) {
+ struct io_kiocb *req = cmd_to_io_kiocb(timeout);
+
+ if (io_cancel_req_match(req, cd))
+ return req;
+ }
+ return NULL;
+}
+
static struct io_kiocb *io_timeout_extract(struct io_ring_ctx *ctx,
struct io_cancel_data *cd)
__must_hold(&ctx->timeout_lock)
{
+ struct io_kiocb *req = io_timeout_find(ctx, cd);
struct io_timeout *timeout;
struct io_timeout_data *io;
- struct io_kiocb *req = NULL;
- list_for_each_entry(timeout, &ctx->timeout_list, list) {
- struct io_kiocb *tmp = cmd_to_io_kiocb(timeout);
-
- if (io_cancel_req_match(tmp, cd)) {
- req = tmp;
- break;
- }
- }
if (!req)
return ERR_PTR(-ENOENT);
@@ -426,14 +433,21 @@ static int io_timeout_update(struct io_ring_ctx *ctx, __u64 user_data,
__must_hold(&ctx->timeout_lock)
{
struct io_cancel_data cd = { .ctx = ctx, .data = user_data, };
- struct io_kiocb *req = io_timeout_extract(ctx, &cd);
+ struct io_kiocb *req = io_timeout_find(ctx, &cd);
struct io_timeout *timeout = io_kiocb_to_cmd(req, struct io_timeout);
struct io_timeout_data *data;
- if (IS_ERR(req))
- return PTR_ERR(req);
+ if (!req)
+ return -ENOENT;
data = req->async_data;
+ /* multishot requests only make sense with rel values */
+ if ((data->flags & IORING_TIMEOUT_MULTISHOT) && mode == HRTIMER_MODE_ABS)
+ return -EINVAL;
+
+ req = io_timeout_extract(ctx, &cd);
+ if (IS_ERR(req))
+ return PTR_ERR(req);
/* for a multishot request this isn't a sequence, keep the repeats */
if (!(data->flags & IORING_TIMEOUT_MULTISHOT))
timeout->off = 0; /* noseq */
--
2.56.0
[-- Attachment #15: 0035-io_uring-net-don-t-retry-recv-on-uncommitted-buffers.patch --]
[-- Type: text/x-patch, Size: 4190 bytes --]
From 3397127a10361a016e8806d2ac39a2d8292672b0 Mon Sep 17 00:00:00 2001
From: Josef Bacik <josef@toxicpanda.com>
Date: Thu, 15 Oct 2026 19:31:04 +0000
Subject: [PATCH 35/48] io_uring/net: don't retry recv on uncommitted buffers
Commit 8444de681a179e41a4eb92d5bbe770a0677074a6 upstream.
Commit 41b70df5b38b ("io_uring/net: commit partial buffers on retry")
commits ring provided buffers when a receive is going to be retried after
a partial transfer, so that another request using the same buffer group
can't be handed the memory the first one keeps receiving into. It commits
the number of bytes received so far. That does the job for a single
buffer which is consumed as a whole, but the request hangs on to all it
has mapped:
- For a buffer ring with IOU_PBUF_RING_INC only these bytes are consumed.
The rest of the buffer stays at the head of the ring.
- For a bundle only the buffers which hold these bytes are consumed, the
others that were peeked stay in the ring.
In both cases the next receive with that buffer group gets memory that
the retried request is still going to write to, and whoever comes last
overwrites the data of the other. What the retry fills isn't committed at
all, as the buffer list isn't known any more at that point. So the kernel
hands it out again later on, while the CQE tells the application that it
was used, and for an incremental buffer that it is done with.
The remainder can't be committed in advance: if the request ends up
receiving less, there is no way to tell the application that more was
consumed than the CQE accounts for. Complete the request with what it has
got instead of retrying it. MSG_WAITALL is no longer honoured for
receives into incremental buffers or multi buffer bundles, they can
return a short count, with the request marked as failed like for other
short MSG_WAITALL transfers.
Fixes: 41b70df5b38b ("io_uring/net: commit partial buffers on retry")
Assisted-by: LLM
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Link: https://patch.msgid.link/20261015-b4-io-uring-fixes-v1-40-c1556f5d7bf5@toxicpanda.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
---
io_uring/net.c | 24 ++++++++++++++++++++++--
1 file changed, 22 insertions(+), 2 deletions(-)
diff --git a/io_uring/net.c b/io_uring/net.c
index 455a191c9db4..522be3edf51b 100644
--- a/io_uring/net.c
+++ b/io_uring/net.c
@@ -498,6 +498,24 @@ static int io_bundle_nbufs(struct io_async_msghdr *kmsg, int ret)
return nbufs;
}
+/*
+ * A receive that is retried after a partial transfer keeps writing to what it
+ * has mapped. io_net_kbuf_recyle() only commits what was filled so far, which
+ * takes a buffer out of circulation if it's a single one that is consumed as a
+ * whole. The tail of an incrementally consumed buffer, or the yet unused
+ * buffers of a bundle, would be up for grabs. They can't be committed in
+ * advance either, the CQE has no way to tell if they end up not being used.
+ */
+static bool io_net_kbuf_can_retry(struct io_kiocb *req,
+ struct io_buffer_list *bl,
+ struct io_async_msghdr *kmsg)
+{
+ /* not ring provided, or committed at selection time */
+ if (!(req->flags & REQ_F_BUFFERS_COMMIT))
+ return true;
+ return !(bl->flags & IOBL_INC) && iter_is_ubuf(&kmsg->msg.msg_iter);
+}
+
static int io_net_kbuf_recyle(struct io_kiocb *req, struct io_buffer_list *bl,
struct io_async_msghdr *kmsg, int len)
{
@@ -1089,7 +1107,8 @@ int io_recvmsg(struct io_kiocb *req, unsigned int issue_flags)
io_kbuf_recycle(req, sel.buf_list, issue_flags);
return IOU_RETRY;
}
- if (ret > 0 && io_net_retry(sock, flags)) {
+ if (ret > 0 && io_net_retry(sock, flags) &&
+ io_net_kbuf_can_retry(req, sel.buf_list, kmsg)) {
sr->done_io += ret;
return io_net_kbuf_recyle(req, sel.buf_list, kmsg, ret);
}
@@ -1238,7 +1257,8 @@ int io_recv(struct io_kiocb *req, unsigned int issue_flags)
io_kbuf_recycle(req, sel.buf_list, issue_flags);
return IOU_RETRY;
}
- if (ret > 0 && io_net_retry(sock, flags)) {
+ if (ret > 0 && io_net_retry(sock, flags) &&
+ io_net_kbuf_can_retry(req, sel.buf_list, kmsg)) {
sr->len -= ret;
sr->buf += ret;
sr->done_io += ret;
--
2.56.0
[-- Attachment #16: 0034-io_uring-don-t-pass-restart-error-codes-to-userspace.patch --]
[-- Type: text/x-patch, Size: 4266 bytes --]
From d53569eb1f6db2da6d840cc28fda072b87a59560 Mon Sep 17 00:00:00 2001
From: Josef Bacik <josef@toxicpanda.com>
Date: Thu, 15 Oct 2026 19:31:03 +0000
Subject: [PATCH 34/48] io_uring: don't pass restart error codes to userspace
Commit e1a64bc64897f4c2fe460922c345f2ed8f889b5f upstream.
-ERESTARTSYS and friends are instructions for the system call exit code,
which either restarts the call or turns them into -EINTR. They are not
supposed to be seen by applications. A request isn't a system call,
nobody looks at what it returns before that is put into a CQE.
The read/write and the networking opcodes convert these codes to -EINTR
by hand. The others don't. An io-wq worker that is blocked in the open of
a FIFO, a tty or some other device, or in a splice from an empty pipe,
and that gets interrupted because the request is cancelled, completes it
with -512:
IORING_OP_OPENAT, IOSQE_ASYNC // FIFO without a writer
IORING_OP_ASYNC_CANCEL // -EALREADY, worker is signalled
CQE of the open: res == -512
There are more candidates, like whatever a file system or driver returns
from ->flush(), ->fsync(), ->fallocate() or ->uring_cmd(), or breaking a
lease or delegation.
Take care of it in io_issue_sqe(), which all of them go through. The
helper is moved from rw.c as is. The cost is a compare and a branch that
is not taken per request.
Fixes: 15b71abe7b52 ("io_uring: add support for IORING_OP_OPENAT")
Assisted-by: LLM
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Link: https://patch.msgid.link/20261015-b4-io-uring-fixes-v1-39-c1556f5d7bf5@toxicpanda.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
---
io_uring/io_uring.c | 5 +++++
io_uring/io_uring.h | 18 ++++++++++++++++++
io_uring/rw.c | 18 ------------------
3 files changed, 23 insertions(+), 18 deletions(-)
diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c
index b296db1dc217..96e27af812d5 100644
--- a/io_uring/io_uring.c
+++ b/io_uring/io_uring.c
@@ -1890,6 +1890,9 @@ static int io_issue_sqe(struct io_kiocb *req, unsigned int issue_flags)
ret = __io_issue_sqe(req, issue_flags, def);
if (ret == IOU_COMPLETE) {
+ /* not all handlers take care of the internal restart codes */
+ if (unlikely(req->cqe.res <= -ERESTARTSYS))
+ req->cqe.res = io_fixup_restart_res(req->cqe.res);
if (issue_flags & IO_URING_F_COMPLETE_DEFER)
io_req_complete_defer(req);
else
@@ -1905,6 +1908,8 @@ static int io_issue_sqe(struct io_kiocb *req, unsigned int issue_flags)
if ((req->ctx->flags & IORING_SETUP_IOPOLL) && def->iopoll_queue)
io_iopoll_req_issued(req, issue_flags);
}
+ if (unlikely(ret <= -ERESTARTSYS))
+ ret = io_fixup_restart_res(ret);
return ret;
}
diff --git a/io_uring/io_uring.h b/io_uring/io_uring.h
index 899c3a3bd6f4..4b3acbb39436 100644
--- a/io_uring/io_uring.h
+++ b/io_uring/io_uring.h
@@ -334,6 +334,24 @@ static inline void io_req_set_res32(struct io_kiocb *req, s32 res, u32 cflags,
req->big_cqe.extra2 = extra2;
}
+static inline ssize_t io_fixup_restart_res(ssize_t ret)
+{
+ switch (ret) {
+ case -ERESTARTSYS:
+ case -ERESTARTNOINTR:
+ case -ERESTARTNOHAND:
+ case -ERESTART_RESTARTBLOCK:
+ /*
+ * We can't just restart the syscall, since previously
+ * submitted sqes may already be in progress. Just fail
+ * this IO with EINTR.
+ */
+ return -EINTR;
+ default:
+ return ret;
+ }
+}
+
static inline void *io_uring_alloc_async_data(struct io_alloc_cache *cache,
struct io_kiocb *req)
{
diff --git a/io_uring/rw.c b/io_uring/rw.c
index 3666ff3d2e42..f46d08bbe9d8 100644
--- a/io_uring/rw.c
+++ b/io_uring/rw.c
@@ -625,24 +625,6 @@ static void io_complete_rw_iopoll(struct kiocb *kiocb, long res)
smp_store_release(&req->iopoll_completed, 1);
}
-static inline ssize_t io_fixup_restart_res(ssize_t ret)
-{
- switch (ret) {
- case -ERESTARTSYS:
- case -ERESTARTNOINTR:
- case -ERESTARTNOHAND:
- case -ERESTART_RESTARTBLOCK:
- /*
- * We can't just restart the syscall, since previously
- * submitted sqes may already be in progress. Just fail
- * this IO with EINTR.
- */
- return -EINTR;
- default:
- return ret;
- }
-}
-
static inline void io_rw_done(struct io_kiocb *req, ssize_t ret)
{
struct io_rw *rw = io_kiocb_to_cmd(req, struct io_rw);
--
2.56.0
[-- Attachment #17: 0033-io_uring-openclose-keep-the-lock-in-io_pipe_fixed.patch --]
[-- Type: text/x-patch, Size: 2131 bytes --]
From 03c2213fb2290e65e1c975fdfa8cc38f9ac385d0 Mon Sep 17 00:00:00 2001
From: Josef Bacik <josef@toxicpanda.com>
Date: Thu, 15 Oct 2026 19:31:02 +0000
Subject: [PATCH 33/48] io_uring/openclose: keep the lock in io_pipe_fixed()
Commit a3d7ee11289f657aa9af826d8f0844221292a27a upstream.
io_pipe_fixed() installs both ends of the pipe in the fixed file table,
drops ->uring_lock, and copies the slot numbers to the application. If
that faults, it takes the lock again and removes whatever is in these
slots.
If the request is run by io-wq the lock really is dropped in between, and
the slots are visible to everybody else. By the time the error path gets
to them, they may have been closed and reused, by IORING_OP_CLOSE and a
direct open or accept, or by IORING_REGISTER_FILES_UPDATE. Then a file is
removed from the table that has nothing to do with the failed request.
Do the copy with the lock held. Faulting in user memory under
->uring_lock is fine, the registration paths do it all the time.
Fixes: 53db8a71ecb4 ("io_uring: add support for IORING_OP_PIPE")
Assisted-by: LLM
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Link: https://patch.msgid.link/20261015-b4-io-uring-fixes-v1-38-c1556f5d7bf5@toxicpanda.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
---
io_uring/openclose.c | 11 +++++++----
1 file changed, 7 insertions(+), 4 deletions(-)
diff --git a/io_uring/openclose.c b/io_uring/openclose.c
index 606ce0664e6a..a7ad98952d0a 100644
--- a/io_uring/openclose.c
+++ b/io_uring/openclose.c
@@ -366,13 +366,16 @@ static int io_pipe_fixed(struct io_kiocb *req, struct file **files,
fds[1] = alloc_slot ? ret : slot - 1;
files[1] = NULL;
- io_ring_submit_unlock(ctx, issue_flags);
-
- if (!copy_to_user(p->fds, fds, sizeof(fds)))
+ /*
+ * Hang on to the lock. If this fails, the slots have to be cleared, and
+ * they must still hold what was put there.
+ */
+ if (!copy_to_user(p->fds, fds, sizeof(fds))) {
+ io_ring_submit_unlock(ctx, issue_flags);
return 0;
+ }
ret = -EFAULT;
- io_ring_submit_lock(ctx, issue_flags);
err:
if (fds[0] != -1)
io_fixed_fd_remove(ctx, fds[0]);
--
2.56.0
[-- Attachment #18: 0032-io_uring-sqpoll-let-the-thread-sleep-on-disabled-rin.patch --]
[-- Type: text/x-patch, Size: 1794 bytes --]
From 8f9ac20f4a024da911b7482012ce99559ac170f9 Mon Sep 17 00:00:00 2001
From: Josef Bacik <josef@toxicpanda.com>
Date: Thu, 15 Oct 2026 19:31:01 +0000
Subject: [PATCH 32/48] io_uring/sqpoll: let the thread sleep on disabled rings
Commit d492dcbb3b81c1c2d0aabef41fa1307093f3a556 upstream.
__io_sq_thread() doesn't submit anything as long as the ring has
IORING_SETUP_R_DISABLED set. The check that keeps the SQPOLL thread from
going to sleep while the SQ ring holds entries doesn't know about that.
If an application fills in SQEs first and enables the ring later, the
thread finds that there is work each time it is about to sleep, goes
around, doesn't submit it, and so on. It uses up a CPU until the ring
gets enabled.
Entries in the SQ ring of a disabled ring are no reason to stay awake.
io_register_enable_rings() clears the flag and then wakes the thread, and
the flag is looked at after prepare_to_wait(), so the thread can't miss
that.
Fixes: 0298ef969a11 ("io_uring: clean R_DISABLED startup mess")
Assisted-by: LLM
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Link: https://patch.msgid.link/20261015-b4-io-uring-fixes-v1-37-c1556f5d7bf5@toxicpanda.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
---
io_uring/sqpoll.c | 7 +++++++
1 file changed, 7 insertions(+)
diff --git a/io_uring/sqpoll.c b/io_uring/sqpoll.c
index 74c1a130cd87..448fa81ed87e 100644
--- a/io_uring/sqpoll.c
+++ b/io_uring/sqpoll.c
@@ -390,6 +390,13 @@ static int io_sq_thread(void *data)
*/
smp_mb__after_atomic();
+ /*
+ * Nothing gets submitted before the ring is
+ * enabled, which comes with a wake up.
+ */
+ if (READ_ONCE(ctx->flags) & IORING_SETUP_R_DISABLED)
+ continue;
+
if (io_sqring_entries(ctx)) {
needs_sched = false;
break;
--
2.56.0
[-- Attachment #19: 0031-io_uring-kbuf-handle-index-wrap-in-last-buffer-check.patch --]
[-- Type: text/x-patch, Size: 1646 bytes --]
From 82a157699647651cc4664c7cc53fe806bd0c038f Mon Sep 17 00:00:00 2001
From: Josef Bacik <josef@toxicpanda.com>
Date: Thu, 15 Oct 2026 19:31:00 +0000
Subject: [PATCH 31/48] io_uring/kbuf: handle index wrap in last buffer check
Commit 6e773b5161506a4d0e739eb3aa4667d28f20219c upstream.
io_ring_buffer_select() flags the request with REQ_F_BL_EMPTY if it takes
the last buffer of the ring. Head and tail are 16-bit counters, but
"head + 1" is computed as an int. If head is 65535 and tail has wrapped
to 0, that compares 65536 to 0 and the flag isn't set.
A bundle receive relies on the flag to tell that there is no point in
trying to get more data. Without it, it goes for another round, finds no
buffer, and the request, which has received data, is marked as failed on
account of the -ENOBUFS. That severs a link, and ends a multishot.
Fixes: c3f9109dbc9e ("io_uring/kbuf: flag request if buffer pool is empty after buffer pick")
Assisted-by: LLM
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Link: https://patch.msgid.link/20261015-b4-io-uring-fixes-v1-36-c1556f5d7bf5@toxicpanda.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
---
io_uring/kbuf.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/io_uring/kbuf.c b/io_uring/kbuf.c
index 9b9f0d369847..e1f4ee872740 100644
--- a/io_uring/kbuf.c
+++ b/io_uring/kbuf.c
@@ -203,7 +203,7 @@ static struct io_br_sel io_ring_buffer_select(struct io_kiocb *req, size_t *len,
if (unlikely(tail == head))
return sel;
- if (head + 1 == tail)
+ if ((__u16)(head + 1) == tail)
req->flags |= REQ_F_BL_EMPTY;
buf = io_ring_head_to_buf(br, head, bl->mask);
--
2.56.0
[-- Attachment #20: 0030-io_uring-zcrx-fix-buffer-offsets-in-areas-above-4GB.patch --]
[-- Type: text/x-patch, Size: 1705 bytes --]
From bdebb4b39aa79e6db1c81b18336da66733151550 Mon Sep 17 00:00:00 2001
From: Josef Bacik <josef@toxicpanda.com>
Date: Thu, 15 Oct 2026 19:30:59 +0000
Subject: [PATCH 30/48] io_uring/zcrx: fix buffer offsets in areas above 4GB
Commit ab4536bf2e5617b04f2d9396f06393c7bbfb071d upstream.
io_zcrx_queue_cqe() computes the offset of the buffer in its area from
net_iov_idx(), which is an unsigned int, so the shift is done in 32 bits
and only the result is widened to the u64 that goes into the CQE. An area
can be up to 1TB in size. For every niov beyond the first 4GB the CQE
points the application at a buffer 4GB, or a multiple of that, below the
one which has the data. It reads the wrong data, and then returns a
buffer it doesn't own via the refill queue while the real one is never
given back.
Fixes: 11ed914bbf94 ("io_uring/zcrx: add io_recvzc request")
Assisted-by: LLM
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Link: https://patch.msgid.link/20261015-b4-io-uring-fixes-v1-35-c1556f5d7bf5@toxicpanda.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
---
io_uring/zcrx.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/io_uring/zcrx.c b/io_uring/zcrx.c
index b44d8ba53e59..b725dab91732 100644
--- a/io_uring/zcrx.c
+++ b/io_uring/zcrx.c
@@ -962,7 +962,7 @@ static bool io_zcrx_queue_cqe(struct io_kiocb *req, struct net_iov *niov,
cqe->flags |= IORING_CQE_F_32;
area = io_zcrx_iov_to_area(niov);
- offset = off + (net_iov_idx(niov) << ifq->niov_shift);
+ offset = off + ((u64)net_iov_idx(niov) << ifq->niov_shift);
rcqe = (struct io_uring_zcrx_cqe *)(cqe + 1);
rcqe->off = offset + ((u64)area->area_id << IORING_ZCRX_AREA_SHIFT);
rcqe->__pad = 0;
--
2.56.0
[-- Attachment #21: 0029-io_uring-napi-don-t-busy-poll-past-an-expired-timeou.patch --]
[-- Type: text/x-patch, Size: 1557 bytes --]
From 80a2a7a9566a3d5ad533e36adc44ecde3cabe02e Mon Sep 17 00:00:00 2001
From: Josef Bacik <josef@toxicpanda.com>
Date: Thu, 15 Oct 2026 19:30:58 +0000
Subject: [PATCH 29/48] io_uring/napi: don't busy poll past an expired timeout
Commit 21ed976c12f6b77d7ea028a4fe9a08e69a56179f upstream.
__io_napi_busy_loop() caps the busy poll time to what is left until the
timeout of the wait. The difference is negative if the timeout has passed
already, which is what an application gets if it uses
IORING_ENTER_ABS_TIMER with a time that isn't in the future any more. The
min_t(u64, ...) turns that into a huge value, so the cap has no effect
and the wait, which should return right away, spins for the full busy
poll time first.
Fixes: d5cce407e4f5 ("io_uring/napi: postpone napi timeout adjustment")
Assisted-by: LLM
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Link: https://patch.msgid.link/20261015-b4-io-uring-fixes-v1-34-c1556f5d7bf5@toxicpanda.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
---
io_uring/napi.c | 2 ++
1 file changed, 2 insertions(+)
diff --git a/io_uring/napi.c b/io_uring/napi.c
index f26315e27f19..45180b172bec 100644
--- a/io_uring/napi.c
+++ b/io_uring/napi.c
@@ -365,6 +365,8 @@ void __io_napi_busy_loop(struct io_ring_ctx *ctx, struct io_wait_queue *iowq)
if (iowq->timeout != KTIME_MAX) {
ktime_t dt = ktime_sub(iowq->timeout, io_get_time(ctx));
+ /* an absolute timeout may have passed already */
+ dt = max_t(ktime_t, dt, 0);
iowq->napi_busy_poll_dt = min_t(u64, iowq->napi_busy_poll_dt, dt);
}
--
2.56.0
[-- Attachment #22: 0028-io_uring-napi-make-stale-entries-time-out-after-60s.patch --]
[-- Type: text/x-patch, Size: 1576 bytes --]
From ca566d5bd135a7714ed3a5a8e5d2f9b6d91d1aac Mon Sep 17 00:00:00 2001
From: Josef Bacik <josef@toxicpanda.com>
Date: Thu, 15 Oct 2026 19:30:57 +0000
Subject: [PATCH 28/48] io_uring/napi: make stale entries time out after 60s
Commit 9c151648780d89e5446e5f89f3fa2e416c3bb911 upstream.
With dynamic tracking a NAPI id that hasn't been seen for a minute is
meant to be dropped from the list the ring busy polls. The timeout is
added to jiffies, but it is defined as 60 * SEC_CONVERSION, and
SEC_CONVERSION isn't the number of jiffies in a second. It is the
internal factor of __timespec64_to_jiffies(), HZ scaled up by
SEC_JIFFIE_SC bits. For HZ=1000 that is 1000 << 22, which puts the
timeout at about eight years. On 32-bit the multiplication overflows.
So entries are never removed, and every id that a socket of the ring
ever had keeps being polled on each wait.
Fixes: 8d0c12a80cde ("io-uring: add napi busy poll support")
Assisted-by: LLM
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Link: https://patch.msgid.link/20261015-b4-io-uring-fixes-v1-33-c1556f5d7bf5@toxicpanda.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
---
io_uring/napi.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/io_uring/napi.c b/io_uring/napi.c
index 8d68366a4b90..f26315e27f19 100644
--- a/io_uring/napi.c
+++ b/io_uring/napi.c
@@ -6,7 +6,7 @@
#ifdef CONFIG_NET_RX_BUSY_POLL
/* Timeout for cleanout of stale entries. */
-#define NAPI_TIMEOUT (60 * SEC_CONVERSION)
+#define NAPI_TIMEOUT (60 * HZ)
struct io_napi_entry {
unsigned int napi_id;
--
2.56.0
[-- Attachment #23: 0027-io_uring-wait-no-early-ETIME-if-min_wait-finds-work.patch --]
[-- Type: text/x-patch, Size: 2672 bytes --]
From 6c5cf7ecd701f4868f5df1e20e3ef93094e69eed Mon Sep 17 00:00:00 2001
From: Josef Bacik <josef@toxicpanda.com>
Date: Thu, 15 Oct 2026 19:30:56 +0000
Subject: [PATCH 27/48] io_uring/wait: no early -ETIME if min_wait finds work
Commit 83cce67298ab1f4d2e9205a49f56975475cdb154 upstream.
When the min wait time is over and there is work to run, which for
DEFER_TASKRUN includes task_work that was queued lazily and didn't wake
the waiter, io_cqring_min_timer_wakeup() ends the wait via
io_cqring_timer_wakeup(). That flags it as timed out. io_cqring_wait()
runs the work, leaves the loop, and returns 0 if the CQ ring holds
something.
Running task_work doesn't have to result in a CQE, though. A poll wake up
can turn out to be spurious, for instance if somebody else has read the
data by the time the request is retried. The application then gets -ETIME
once the min wait time has passed, no matter how far away the timeout it
asked for is.
Wake the task without declaring a timeout. From now on a single CQE
satisfies the wait, like in the case where the timer is switched over to
the normal timeout. If running the work doesn't post one, the loop in
io_cqring_wait() goes back to sleep with the normal timeout.
[ 6.18: io_cqring_min_timer_wakeup() lives in io_uring.c, there is no wait.c yet ]
Fixes: 1100c4a2656d ("io_uring: add support for batch wait timeout")
Assisted-by: LLM
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Link: https://patch.msgid.link/20261015-b4-io-uring-fixes-v1-32-c1556f5d7bf5@toxicpanda.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
---
io_uring/io_uring.c | 13 ++++++++++---
1 file changed, 10 insertions(+), 3 deletions(-)
diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c
index c02dd8aa1559..b296db1dc217 100644
--- a/io_uring/io_uring.c
+++ b/io_uring/io_uring.c
@@ -2583,9 +2583,16 @@ static enum hrtimer_restart io_cqring_min_timer_wakeup(struct hrtimer *timer)
if (iowq->timeout == KTIME_MAX ||
ktime_compare(iowq->min_timeout, iowq->timeout) >= 0)
goto out_wake;
- /* work we may need to run, wake function will see if we need to wake */
- if (io_has_work(ctx))
- goto out_wake;
+ /*
+ * Work we need to run. It isn't known if that is going to post a CQE,
+ * so this isn't a timeout. Carry on with the normal one if it doesn't.
+ */
+ if (io_has_work(ctx)) {
+ iowq->min_timeout = 0;
+ iowq->cq_tail = iowq->cq_min_tail + 1;
+ wake_up_process(iowq->wq.private);
+ return HRTIMER_NORESTART;
+ }
/* got events since we started waiting, min timeout is done */
scoped_guard(rcu) {
struct io_rings *rings = io_get_rings(ctx);
--
2.56.0
[-- Attachment #24: 0026-io_uring-timeout-keep-repeat-count-on-multishot-upda.patch --]
[-- Type: text/x-patch, Size: 1714 bytes --]
From 0270e3f3b7d46d800e6a96cc574333ed594334b2 Mon Sep 17 00:00:00 2001
From: Josef Bacik <josef@toxicpanda.com>
Date: Thu, 15 Oct 2026 19:30:54 +0000
Subject: [PATCH 26/48] io_uring/timeout: keep repeat count on multishot update
Commit 3dc2a32cd356ee382aa4a8c7d844c53b2c0ad83b upstream.
IORING_TIMEOUT_UPDATE clears ->off of the timeout it updates, to turn a
timeout that waits for a number of completions into a pure one. For a
multishot timeout ->off isn't a sequence though, it tells whether the
number of repeats is limited: io_timeout_finish() only looks at
->repeats if ->off is set.
A multishot timeout that was set up to fire N times hence fires forever
once its interval has been updated. Leave ->off alone for those, they are
never treated as sequence timeouts anyway.
Fixes: ea97f6c8558e ("io_uring: add support for multishot timeouts")
Assisted-by: LLM
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Link: https://patch.msgid.link/20261015-b4-io-uring-fixes-v1-30-c1556f5d7bf5@toxicpanda.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
---
io_uring/timeout.c | 4 +++-
1 file changed, 3 insertions(+), 1 deletion(-)
diff --git a/io_uring/timeout.c b/io_uring/timeout.c
index df8894b7f71b..e0b454be8b42 100644
--- a/io_uring/timeout.c
+++ b/io_uring/timeout.c
@@ -433,8 +433,10 @@ static int io_timeout_update(struct io_ring_ctx *ctx, __u64 user_data,
if (IS_ERR(req))
return PTR_ERR(req);
- timeout->off = 0; /* noseq */
data = req->async_data;
+ /* for a multishot request this isn't a sequence, keep the repeats */
+ if (!(data->flags & IORING_TIMEOUT_MULTISHOT))
+ timeout->off = 0; /* noseq */
data->ts = *ts;
list_add_tail(&timeout->list, &ctx->timeout_list);
--
2.56.0
[-- Attachment #25: 0025-io_uring-timeout-don-t-skip-over-reached-sequences.patch --]
[-- Type: text/x-patch, Size: 4218 bytes --]
From 7c4bec25d65b1fba23b2257f80d90eff98a2ec07 Mon Sep 17 00:00:00 2001
From: Josef Bacik <josef@toxicpanda.com>
Date: Thu, 15 Oct 2026 19:30:53 +0000
Subject: [PATCH 25/48] io_uring/timeout: don't skip over reached sequences
Commit c6c0b3eba88c3b13ca11ea112404982e86d58f7a upstream.
io_flush_timeouts() copes with the wrap of the CQ sequence by looking at
the distance from ->cq_last_tm_flush. That relies on the target of each
pending timeout being at or ahead of the last flush.
io_timeout() breaks that. It sets ->cq_last_tm_flush to the current
sequence, on the assumption that nothing has been posted since the last
flush. That isn't true. A multishot request posts its CQEs right when it
is issued or retried, but they are committed, and the timeouts flushed,
at the end of the submission or task_work run:
X: IORING_OP_TIMEOUT, off == 1 target_seq == N + 1
io_submit_sqes()
multishot recv, posts two CQEs sequence == N + 2
Y: IORING_OP_TIMEOUT, off == 100 ->cq_last_tm_flush = N + 2
io_flush_timeouts()
X: events_needed = N + 1 - (N + 2) // 0xffffffff
events_got = 0 // not yet
X isn't completed although the completions it was waiting for have
happened, only its timer can end it now. And as the list is sorted and
the walk ends at the first timeout that isn't due, it gets in the way of
the ones behind it. The insertion sort has the same problem, it measures
the distance of the pending targets from the current sequence.
->cq_last_tm_flush only needs to be brought up to date if no timeout is
waiting for a sequence, as io_flush_timeouts() isn't called before the
first one is set up. Otherwise it is current enough. Timeouts with a
sequence come first in the list, so looking at the first entry will do.
Use the last flush as the base for the insertion sort as well.
Fixes: f010505b78a4 ("io_uring: flush timeouts that should already have expired")
Assisted-by: LLM
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Link: https://patch.msgid.link/20261015-b4-io-uring-fixes-v1-29-c1556f5d7bf5@toxicpanda.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
---
io_uring/timeout.c | 21 ++++++++++++++-------
1 file changed, 14 insertions(+), 7 deletions(-)
diff --git a/io_uring/timeout.c b/io_uring/timeout.c
index 3d03e178044b..df8894b7f71b 100644
--- a/io_uring/timeout.c
+++ b/io_uring/timeout.c
@@ -601,8 +601,9 @@ int io_timeout(struct io_kiocb *req, unsigned int issue_flags)
struct io_timeout *timeout = io_kiocb_to_cmd(req, struct io_timeout);
struct io_ring_ctx *ctx = req->ctx;
struct io_timeout_data *data = req->async_data;
+ struct io_timeout *first;
struct list_head *entry;
- u32 tail, off = timeout->off;
+ u32 tail, base, off = timeout->off;
raw_spin_lock_irq(&ctx->timeout_lock);
@@ -619,11 +620,17 @@ int io_timeout(struct io_kiocb *req, unsigned int issue_flags)
tail = data_race(ctx->cached_cq_tail) - atomic_read(&ctx->cq_timeouts);
timeout->target_seq = tail + off;
- /* Update the last seq here in case io_flush_timeouts() hasn't.
- * This is safe because ->completion_lock is held, and submissions
- * and completions are never mixed in the same ->completion_lock section.
+ /*
+ * Update the last seq here in case io_flush_timeouts() hasn't, it only
+ * gets called once sequences are in use. Leave it alone if there are
+ * timeouts waiting for one. CQEs may have been posted since the last
+ * flush, and a target that has been reached must not fall behind it.
*/
- ctx->cq_last_tm_flush = tail;
+ first = list_first_entry_or_null(&ctx->timeout_list, struct io_timeout,
+ list);
+ if (!first || io_is_timeout_noseq(cmd_to_io_kiocb(first)))
+ ctx->cq_last_tm_flush = tail;
+ base = ctx->cq_last_tm_flush;
/*
* Insertion sort, ensuring the first entry in the list is always
@@ -635,8 +642,8 @@ int io_timeout(struct io_kiocb *req, unsigned int issue_flags)
if (io_is_timeout_noseq(nxt))
continue;
- /* nxt.seq is behind @tail, otherwise would've been completed */
- if (off >= nextt->target_seq - tail)
+ /* no target is behind the last flush, see io_flush_timeouts() */
+ if (timeout->target_seq - base >= nextt->target_seq - base)
break;
}
add:
--
2.56.0
[-- Attachment #26: 0024-io_uring-fdinfo-fix-CQE-dump-of-CQE32-rings.patch --]
[-- Type: text/x-patch, Size: 2655 bytes --]
From eb42217a2ecb1ef3d3d6c764902b126ad5ded1b4 Mon Sep 17 00:00:00 2001
From: Josef Bacik <josef@toxicpanda.com>
Date: Thu, 15 Oct 2026 19:30:52 +0000
Subject: [PATCH 24/48] io_uring/fdinfo: fix CQE dump of CQE32 rings
Commit e4507a62c247081fffe299622c04c6f7c3d98e92 upstream.
Since the CQ ring dump was changed to deal with IORING_SETUP_CQE_MIXED, a
32b CQE is taken to occupy two entries of the ring. That is how it works
for mixed rings. With IORING_SETUP_CQE32 each entry has twice the size:
head and tail count 32b CQEs, and entry n is found at cqes[n << 1].
For such rings the dump reads cqes[n], and skips every other entry. It
shows half of the pending CQEs at best, and what it shows is taken from
the wrong place unless the head happens to be at the start of the ring.
The extra data of the last entry isn't shown at all, as the check that
keeps the dump of a mixed ring from reading beyond the array is applied to
these rings as well.
[ 6.18: without the last-slot check of f26f49600c9a, which is not in 6.18 ]
Fixes: 82ceb7fcc5ff ("io_uring/fdinfo: handle mixed sized CQEs")
Assisted-by: LLM
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Link: https://patch.msgid.link/20261015-b4-io-uring-fixes-v1-28-c1556f5d7bf5@toxicpanda.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
---
io_uring/fdinfo.c | 14 ++++++++++----
1 file changed, 10 insertions(+), 4 deletions(-)
diff --git a/io_uring/fdinfo.c b/io_uring/fdinfo.c
index 3585ad830850..23674b0be0d0 100644
--- a/io_uring/fdinfo.c
+++ b/io_uring/fdinfo.c
@@ -125,11 +125,17 @@ static void __io_uring_show_fdinfo(struct io_ring_ctx *ctx, struct seq_file *m)
cq_entries = min(cq_tail - cq_head, ctx->cq_entries);
for (i = 0; i < cq_entries; i++) {
struct io_uring_cqe *cqe;
- bool cqe32 = false;
+ bool cqe32 = false, two_entries = false;
- cqe = &r->cqes[(cq_head & cq_mask)];
- if (cqe->flags & IORING_CQE_F_32 || ctx->flags & IORING_SETUP_CQE32)
+ if (ctx->flags & IORING_SETUP_CQE32) {
+ /* all entries have twice the size */
+ cqe = &r->cqes[(cq_head & cq_mask) << 1];
cqe32 = true;
+ } else {
+ cqe = &r->cqes[(cq_head & cq_mask)];
+ two_entries = cqe->flags & IORING_CQE_F_32;
+ cqe32 = two_entries;
+ }
seq_printf(m, "%5u: user_data:%llu, res:%d, flag:%x",
cq_head & cq_mask, cqe->user_data, cqe->res,
cqe->flags);
@@ -138,7 +144,7 @@ static void __io_uring_show_fdinfo(struct io_ring_ctx *ctx, struct seq_file *m)
cqe->big_cqe[0], cqe->big_cqe[1]);
seq_printf(m, "\n");
cq_head++;
- if (cqe32) {
+ if (two_entries) {
cq_head++;
i++;
}
--
2.56.0
[-- Attachment #27: 0023-io_uring-register-handle-index-wrap-when-resizing.patch --]
[-- Type: text/x-patch, Size: 2165 bytes --]
From 09e194426a6287281fca13789881b4ae8f2f843d Mon Sep 17 00:00:00 2001
From: Josef Bacik <josef@toxicpanda.com>
Date: Thu, 15 Oct 2026 19:30:50 +0000
Subject: [PATCH 23/48] io_uring/register: handle index wrap when resizing
Commit e8dee8d5498c710a22a84076d8ecb661627032b4 upstream.
The ring heads and tails are free running 32-bit counters.
io_register_resize_rings() takes that into account when it checks whether
the pending entries fit into the new rings, but then copies them with
for (i = old_head; i < tail; i++)
If the tail has wrapped and the head hasn't, that copies nothing. The new
rings get the old head and tail, so the application finds whatever is in
the freshly allocated memory, zeroes, in place of the CQEs it hasn't
reaped yet, or the kernel in place of the SQEs that have not been
submitted yet. A ring that completes a million requests per second gets
there after little more than an hour.
Loop until the tail is reached, the number of iterations has been bounded
by the size of the new ring at that point.
Fixes: 79cfe9e59c2a ("io_uring/register: add IORING_REGISTER_RESIZE_RINGS")
Assisted-by: LLM
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Link: https://patch.msgid.link/20261015-b4-io-uring-fixes-v1-26-c1556f5d7bf5@toxicpanda.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
---
io_uring/register.c | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/io_uring/register.c b/io_uring/register.c
index 170043ea4a6d..de49874e8502 100644
--- a/io_uring/register.c
+++ b/io_uring/register.c
@@ -515,7 +515,7 @@ static int io_register_resize_rings(struct io_ring_ctx *ctx, void __user *arg)
old_head = READ_ONCE(o.rings->sq.head);
if (tail - old_head > p.sq_entries)
goto overflow;
- for (i = old_head; i < tail; i++) {
+ for (i = old_head; i != tail; i++) {
unsigned index, dst_mask, src_mask;
size_t sq_size;
@@ -545,7 +545,7 @@ static int io_register_resize_rings(struct io_ring_ctx *ctx, void __user *arg)
ret = -EOVERFLOW;
goto out;
}
- for (i = old_head; i < tail; i++) {
+ for (i = old_head; i != tail; i++) {
unsigned index, dst_mask, src_mask;
size_t cq_size;
--
2.56.0
[-- Attachment #28: 0022-io_uring-register-don-t-wrap-32b-CQEs-when-resizing.patch --]
[-- Type: text/x-patch, Size: 2966 bytes --]
From de56ab735ca7d15b8e70f8c34767d185b5d04739 Mon Sep 17 00:00:00 2001
From: Josef Bacik <josef@toxicpanda.com>
Date: Thu, 15 Oct 2026 19:30:49 +0000
Subject: [PATCH 22/48] io_uring/register: don't wrap 32b CQEs when resizing
Commit 0edf1930b8ed3c398d3cbdbafd8f72dbd1a3d9e4 upstream.
On an IORING_SETUP_CQE_MIXED ring, a 32b CQE uses two contiguous CQ
entries and is never allowed to wrap the end of the ring. When posting,
io_cqe_cache_refill() inserts a dummy IORING_CQE_F_SKIP entry if only
one entry is left before the ring wraps, and the application relies on
that to read a 32b CQE as one contiguous 32 byte slot.
io_register_resize_rings() copies pending CQEs to the new ring one 16b
entry at a time, placing entry i at i & (new_entries - 1). When the CQ
ring is shrunk, a pending 32b CQE that was contiguous in the old ring
can land with its first half on the last entry of the new ring and its
second half on entry 0. For example, with 8 old entries, 4 new entries
and a 32b CQE at index 3, the halves end up at entries 3 and 0. The
application then reads the extra data of that CQE from past the end of
the CQ array, and the CQE content it sees is wrong.
This was found by code inspection and has not been reproduced.
Fix by failing the resize with -EOVERFLOW if a pending 32b CQE would
wrap the new ring, the same way the resize fails when the pending CQEs
don't fit.
Fixes: e26dca67fde1 ("io_uring: add support for IORING_SETUP_CQE_MIXED")
Assisted-by: LLM
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Link: https://patch.msgid.link/20261015-b4-io-uring-fixes-v1-25-c1556f5d7bf5@toxicpanda.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
---
io_uring/register.c | 16 ++++++++++++++++
1 file changed, 16 insertions(+)
diff --git a/io_uring/register.c b/io_uring/register.c
index 741cd294773e..170043ea4a6d 100644
--- a/io_uring/register.c
+++ b/io_uring/register.c
@@ -406,6 +406,7 @@ static int io_register_resize_rings(struct io_ring_ctx *ctx, void __user *arg)
size_t size, sq_array_offset;
unsigned i, tail, old_head;
struct io_uring_params p;
+ bool cqe32_tail = false;
int ret;
/* limited to DEFER_TASKRUN for now */
@@ -558,6 +559,21 @@ static int io_register_resize_rings(struct io_ring_ctx *ctx, void __user *arg)
src_mask = (ctx->cq_entries << 1) - 1;
dst_mask = (p.cq_entries << 1) - 1;
}
+ /*
+ * A 32b CQE on a CQE_MIXED ring takes two contiguous entries
+ * and must not wrap. A smaller ring may place it on the last
+ * entry, fail the resize if that happens.
+ */
+ if (ctx->flags & IORING_SETUP_CQE_MIXED) {
+ if (cqe32_tail) {
+ cqe32_tail = false;
+ } else if (READ_ONCE(o.rings->cqes[index & src_mask].flags) &
+ IORING_CQE_F_32) {
+ if ((index & dst_mask) == dst_mask)
+ goto overflow;
+ cqe32_tail = true;
+ }
+ }
memcpy(&n.rings->cqes[index & dst_mask], &o.rings->cqes[index & src_mask], cq_size);
}
WRITE_ONCE(n.rings->cq.head, old_head);
--
2.56.0
[-- Attachment #29: 0021-io_uring-drop-the-CQE-cache-if-a-refill-fails.patch --]
[-- Type: text/x-patch, Size: 2296 bytes --]
From c1c14f78b3c63382689476de74621ac709cf720c Mon Sep 17 00:00:00 2001
From: Josef Bacik <josef@toxicpanda.com>
Date: Thu, 15 Oct 2026 19:30:47 +0000
Subject: [PATCH 21/48] io_uring: drop the CQE cache if a refill fails
Commit 78f9c9cffef02a290b6e97672aaa9c75df0d21da upstream.
->cqe_cached and ->cqe_sentinel delimit the CQ entries which can be
handed out without looking at the ring. io_get_cqe_overflow() only calls
io_cqe_cache_refill() if that range is too small. Without
IORING_SETUP_CQE_MIXED too small means empty. With it, a 32b CQE needs
two entries, and one may be left. If the refill then fails, the entry
stays in the cache, and the next 16b CQE gets it without any checks.
That CQE overtakes the 32b one, which went to the overflow list. The test
for IO_CHECK_CQ_OVERFLOW_BIT, which is there to prevent that, is part of
the refill:
CQ ring with 4 entries, head == 0
16b CQE, 32b CQE tail == 3, one entry cached
32b CQE no room, the CQE overflows
16b CQE written to entry 3, ahead of the 32b CQE
Forget about the rest of the cache when it is refilled.
Fixes: e26dca67fde1 ("io_uring: add support for IORING_SETUP_CQE_MIXED")
Reported-by: Leonid Ovchinnikov <teiny.work@gmail.com>
Closes: https://lore.kernel.org/io-uring/CAECZYD5OMD9vMTzjteFMMP40Dn+SWKj9SFj=43U6uO3XMnxJbw@mail.gmail.com/
Assisted-by: LLM
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Link: https://patch.msgid.link/20261015-b4-io-uring-fixes-v1-23-c1556f5d7bf5@toxicpanda.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
---
io_uring/io_uring.c | 7 +++++++
1 file changed, 7 insertions(+)
diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c
index bc258cd3198c..c02dd8aa1559 100644
--- a/io_uring/io_uring.c
+++ b/io_uring/io_uring.c
@@ -780,6 +780,13 @@ bool io_cqe_cache_refill(struct io_ring_ctx *ctx, bool overflow, bool cqe32)
if (!overflow && (ctx->check_cq & BIT(IO_CHECK_CQ_OVERFLOW_BIT)))
return false;
+ /*
+ * A single entry may be left if a 32b CQE is asked for. It must not be
+ * handed out later on if this fails, as the CQE is going to overflow.
+ */
+ ctx->cqe_cached = NULL;
+ ctx->cqe_sentinel = NULL;
+
/*
* Post dummy CQE if a 32b CQE is needed and there's only room for a
* 16b CQE before the ring wraps.
--
2.56.0
[-- Attachment #30: 0020-io_uring-futex-allocate-private-hash-before-waiting.patch --]
[-- Type: text/x-patch, Size: 3800 bytes --]
From f9a3f10da144c52dc2c725f8a0102b86704ee67e Mon Sep 17 00:00:00 2001
From: Josef Bacik <josef@toxicpanda.com>
Date: Thu, 15 Oct 2026 19:30:46 +0000
Subject: [PATCH 20/48] io_uring/futex: allocate private hash before waiting
Commit 3eff68bce25584cded3fb6ec033ff9a1c9913d04 upstream.
The futex code relies on there being no private futex waiters by the time
the first private hash of a mm is installed: __futex_pivot_hash() moves
the waiters from the previous private hash to the new one, but it does not
look for them in the global hash which is used as long as there is no
private hash. That is why every clone(CLONE_VM) allocates it up front: a
task which is alone in its mm cannot block in futex_wait() and create a
thread or invoke prctl(PR_FUTEX_HASH) at the same time.
IORING_OP_FUTEX_WAIT and IORING_OP_FUTEX_WAITV do not block the submitter
though:
io_uring_enter()
io_futex_wait()
futex_wait_setup() // no private hash: queued in the global hash
pthread_create()
futex_hash_allocate_default()
__futex_pivot_hash() // first private hash, nothing is moved
futex(FUTEX_WAKE_PRIVATE)
futex_hash() // private hash: no waiter
The wakeup is lost and the request only completes when it is cancelled.
prctl(PR_FUTEX_HASH, PR_FUTEX_HASH_SET_SLOTS, n) instead of
pthread_create() has the same effect.
Allocate the default private hash when a wait request on a private futex
is prepared, which happens in the context of the submitter. This is a RCU
protected pointer check once the hash exists. A failure to allocate it
fails the request with -ENOMEM.
[ 6.18: the futex vector is a local, not ifd->futexv, so free that on error ]
Fixes: 80367ad01d93 ("futex: Add basic infrastructure for local task local hash")
Assisted-by: LLM
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Link: https://patch.msgid.link/20261015-b4-io-uring-fixes-v1-22-c1556f5d7bf5@toxicpanda.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
---
io_uring/futex.c | 28 ++++++++++++++++++++++++----
1 file changed, 24 insertions(+), 4 deletions(-)
diff --git a/io_uring/futex.c b/io_uring/futex.c
index c052207a8baa..6f756de08705 100644
--- a/io_uring/futex.c
+++ b/io_uring/futex.c
@@ -149,6 +149,24 @@ int io_futex_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe)
return 0;
}
+/*
+ * A wait on a private futex does not block the submitter. Even a single
+ * threaded task can go on and have the mm private hash installed by clone()
+ * or prctl(), which does not pick up waiters from the global hash. Make
+ * sure the private hash exists before the first waiter is queued.
+ */
+static int io_futex_prep_private(struct io_kiocb *req)
+{
+ int ret = futex_hash_allocate_default();
+
+ if (unlikely(ret))
+ return ret;
+
+ /* inflight tracking only needed for mm private hash */
+ io_req_track_inflight(req);
+ return 0;
+}
+
int io_futex_wait_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe)
{
struct io_futex *iof = io_kiocb_to_cmd(req, struct io_futex);
@@ -158,9 +176,8 @@ int io_futex_wait_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe)
if (unlikely(ret))
return ret;
- /* inflight tracking only needed for mm private hash */
if (!(iof->futex_flags & FLAGS_SHARED))
- io_req_track_inflight(req);
+ return io_futex_prep_private(req);
return 0;
}
@@ -207,10 +224,13 @@ int io_futexv_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe)
return ret;
}
- /* inflight tracking only needed for mm private hash */
for (i = 0; i < iof->futex_nr; i++) {
if (!(futexv[i].w.flags & FLAGS_SHARED)) {
- io_req_track_inflight(req);
+ ret = io_futex_prep_private(req);
+ if (ret) {
+ kfree(futexv);
+ return ret;
+ }
break;
}
}
--
2.56.0
[-- Attachment #31: 0019-io_uring-poll-drop-stale-double-poll-entry-on-rearm.patch --]
[-- Type: text/x-patch, Size: 2425 bytes --]
From d94591c2f59e0285062285233654806ff56d8b80 Mon Sep 17 00:00:00 2001
From: Josef Bacik <josef@toxicpanda.com>
Date: Thu, 15 Oct 2026 19:30:45 +0000
Subject: [PATCH 19/48] io_uring/poll: drop stale double poll entry on rearm
Commit f2dbe87968a3aef74a11c025776d293b80964d3f upstream.
A file that uses two wait queues for poll, e.g. a tty, gets a second
io_poll entry, which IORING_OP_POLL_ADD keeps in ->async_data until the
request is freed. io_poll_add() can run more than once for a request
though: IORING_OP_POLL_REMOVE with one of the update flags disarms and
rearms it, and io_poll_task_func() reissues it if a wake up turns out to
be spurious. The second poll_wait() of the new round then finds
*poll_ptr set, with ->head cleared by the disarm, and __io_queue_proc()
fails what it takes for a third wait queue with -EINVAL.
For an update that means that the poll request is completed with
-ECANCELED, while the update itself reports success. For a spurious wake
up, e.g. tcsetattr() on the polled tty, the poll request fails with
-EINVAL.
Free the old entry and clear the poll flags before arming, which is what
io_arm_apoll() does for internal polling. All entries have been removed
from their wait queues at that point.
Fixes: b69de288e913 ("io_uring: allow events and user_data update of running poll requests")
Fixes: 6e5aedb9324a ("io_uring/poll: attempt request issue after racy poll wakeup")
Assisted-by: LLM
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Link: https://patch.msgid.link/20261015-b4-io-uring-fixes-v1-21-c1556f5d7bf5@toxicpanda.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
---
io_uring/poll.c | 9 +++++++++
1 file changed, 9 insertions(+)
diff --git a/io_uring/poll.c b/io_uring/poll.c
index f308eb793d72..00d9f6f35169 100644
--- a/io_uring/poll.c
+++ b/io_uring/poll.c
@@ -915,6 +915,15 @@ int io_poll_add(struct io_kiocb *req, unsigned int issue_flags)
ipt.pt._qproc = io_poll_queue_proc;
+ /*
+ * A poll update or a spurious wake up arms the request again. Forget
+ * the entries of the previous round like io_arm_apoll() does, a stale
+ * second entry would be taken for a third wait queue.
+ */
+ if (req_has_async_data(req))
+ io_req_async_data_free(req);
+ req->flags &= ~(REQ_F_SINGLE_POLL | REQ_F_DOUBLE_POLL);
+
ret = __io_arm_poll_handler(req, poll, &ipt, poll->events, issue_flags);
if (ret > 0) {
io_req_set_res(req, ipt.result_mask, 0);
--
2.56.0
[-- Attachment #32: 0018-io_uring-wait-look-at-all-CQ-waiters-on-wake-up.patch --]
[-- Type: text/x-patch, Size: 2039 bytes --]
From 60be3d9d9557c17883d0548a70e86eb15fd37736 Mon Sep 17 00:00:00 2001
From: Josef Bacik <josef@toxicpanda.com>
Date: Thu, 15 Oct 2026 19:30:44 +0000
Subject: [PATCH 18/48] io_uring/wait: look at all CQ waiters on wake up
Commit 356f92f2412686f822383b784fa102924407457a upstream.
io_wake_function() returns -1 if the waiter it is called for doesn't have
enough events yet, which makes __wake_up_common() stop walking the wait
queue. That assumes that the waiters behind it can't be satisfied either.
Each waiter has its own target though, computed from the min_complete it
passed in:
thread A: io_uring_enter(GETEVENTS, min_complete = 10) // queued first
thread B: io_uring_enter(GETEVENTS, min_complete = 1)
one CQE is posted: A isn't woken, and B isn't even considered
B keeps sleeping with the event it asked for sitting in the CQ ring, until
A's demand is met or a timeout or signal ends the wait.
Return 0 to carry on with the next waiter. Only waiters whose condition
is met are woken, like before.
[ 6.18: io_wake_function() lives in io_uring.c, there is no wait.c yet ]
Fixes: bda521624e75 ("io_uring: make CQ ring wakeups be more efficient")
Assisted-by: LLM
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Link: https://patch.msgid.link/20261015-b4-io-uring-fixes-v1-20-c1556f5d7bf5@toxicpanda.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
---
io_uring/io_uring.c | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c
index 44d9f54915ac..bc258cd3198c 100644
--- a/io_uring/io_uring.c
+++ b/io_uring/io_uring.c
@@ -2525,7 +2525,8 @@ static int io_wake_function(struct wait_queue_entry *curr, unsigned int mode,
*/
if (io_should_wake(iowq) || io_has_work(iowq->ctx))
return autoremove_wake_function(curr, mode, wake_flags, key);
- return -1;
+ /* keep going, a waiter further down may be content with less */
+ return 0;
}
int io_run_task_work_sig(struct io_ring_ctx *ctx)
--
2.56.0
[-- Attachment #33: 0017-io_uring-wake-ring-pollers-for-local-task_work.patch --]
[-- Type: text/x-patch, Size: 4369 bytes --]
From 529829c58622439ea0a246feefb032e38cc58240 Mon Sep 17 00:00:00 2001
From: Josef Bacik <josef@toxicpanda.com>
Date: Thu, 15 Oct 2026 19:30:43 +0000
Subject: [PATCH 17/48] io_uring: wake ring pollers for local task_work
Commit b88902d8745a701d24275d012f0830e3628e2a22 upstream.
With IORING_SETUP_DEFER_TASKRUN completions are queued as local work, and
only become CQEs when the submitter enters the kernel to wait for events.
For an application that finds out via poll() or epoll on the ring fd when
it is time to do that, io_uring_poll() reports EPOLLIN if local work is
pending.
That only helps if the work is there at the time of the call. When
DEFER_TASKRUN was added, io_req_local_work_add() did io_cqring_wake(),
which took care of tasks sleeping in poll() as well. Then polling got a
wait queue of its own, ->poll_wq, and the wake up in
io_req_local_work_add() was replaced by waking the submitter directly.
Since then ->poll_wq is only woken when CQEs are committed, which for
such a ring is done by the very task that is waiting to be told to:
submitter completion
poll(ring_fd)
io_uring_poll() // nothing yet
schedule()
io_req_local_work_add()
// ->cq_wait_nr: nobody waits
// sleeps on
Wake ->poll_wq when the first entry is put on the work list. Like for the
eventfd, that must not be done right away if the work is queued from a
wake up handler, as the chain of wake ups could get back to the wait
queue whose lock is held. Use an RCU callback in that case, with a
reference to the ring.
[ 6.18: io_req_local_work_add() lives in io_uring.c, and ->poll_activated is a bitfield rather than an int_flags bit ]
Fixes: 130bd686d9be ("io_uring: waitqueue-less cq waiting")
Assisted-by: LLM
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Link: https://patch.msgid.link/20261015-b4-io-uring-fixes-v1-19-c1556f5d7bf5@toxicpanda.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
---
include/linux/io_uring_types.h | 3 +++
io_uring/io_uring.c | 34 ++++++++++++++++++++++++++++++++++
2 files changed, 37 insertions(+)
diff --git a/include/linux/io_uring_types.h b/include/linux/io_uring_types.h
index 4f1607e00359..2576f72eada5 100644
--- a/include/linux/io_uring_types.h
+++ b/include/linux/io_uring_types.h
@@ -445,6 +445,9 @@ struct io_ring_ctx {
u32 iowq_limits[2];
struct callback_head poll_wq_task_work;
+ /* deferred ->poll_wq wake up for local task_work */
+ struct rcu_head poll_wq_rcu;
+ atomic_t poll_wq_rcu_pending;
struct list_head defer_list;
unsigned nr_drained;
diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c
index 68a86a2c5d7c..44d9f54915ac 100644
--- a/io_uring/io_uring.c
+++ b/io_uring/io_uring.c
@@ -1219,6 +1219,37 @@ static void io_ctx_mark_taskrun(struct io_ring_ctx *ctx)
}
}
+static void io_poll_wq_wake_rcu(struct rcu_head *rcu)
+{
+ struct io_ring_ctx *ctx = container_of(rcu, struct io_ring_ctx,
+ poll_wq_rcu);
+
+ atomic_set(&ctx->poll_wq_rcu_pending, 0);
+ io_poll_wq_wake(ctx);
+ percpu_ref_put(&ctx->refs);
+}
+
+/*
+ * io_uring_poll() reports pending local work as EPOLLIN, as it takes a trip to
+ * the kernel to turn it into CQEs. Let those know who are asleep already.
+ */
+static void io_local_work_wake_poll(struct io_ring_ctx *ctx, unsigned int flags)
+{
+ if (!(flags & IOU_F_TWQ_IN_WAKE)) {
+ io_poll_wq_wake(ctx);
+ return;
+ }
+ /*
+ * Called from a wake up handler with some wait queue lock held, this
+ * one could lead back to it. Do it from a clean context.
+ */
+ if (!wq_has_sleeper(&ctx->poll_wq) ||
+ atomic_xchg(&ctx->poll_wq_rcu_pending, 1))
+ return;
+ percpu_ref_get(&ctx->refs);
+ call_rcu_hurry(&ctx->poll_wq_rcu, io_poll_wq_wake_rcu);
+}
+
static void io_req_local_work_add(struct io_kiocb *req, unsigned flags)
{
struct io_ring_ctx *ctx = req->ctx;
@@ -1276,6 +1307,9 @@ static void io_req_local_work_add(struct io_kiocb *req, unsigned flags)
io_ctx_mark_taskrun(ctx);
if (ctx->has_evfd)
io_eventfd_signal(ctx, false, flags & IOU_F_TWQ_IN_WAKE);
+ /* whoever sets the flag wakes ->poll_wq afterwards */
+ if (ctx->poll_activated)
+ io_local_work_wake_poll(ctx, flags);
}
nr_wait = atomic_read(&ctx->cq_wait_nr);
--
2.56.0
[-- Attachment #34: 0016-io_uring-cancel-let-CANCEL_ALL-find-parked-requests.patch --]
[-- Type: text/x-patch, Size: 3827 bytes --]
From 7321cb380cf01cb2117e50d64d6636c25e25564d Mon Sep 17 00:00:00 2001
From: Josef Bacik <josef@toxicpanda.com>
Date: Thu, 15 Oct 2026 19:30:42 +0000
Subject: [PATCH 16/48] io_uring/cancel: let CANCEL_ALL find parked requests
Commit fe6ca13c98b1223f6005f32913ebd32e4df64b77 upstream.
A cancellation with IORING_ASYNC_CANCEL_ALL or _ANY goes on until it finds
nothing more. A request may stay where it was found for a while after it
has been cancelled, so each one that matches is stamped with the sequence
number of the cancellation, and is skipped from then on.
io_cancel_cb() does so for io-wq work as well. There is no need to:
pending work is taken off the list, and the workers are looked at once per
call. And it does harm. A request that is run by io-wq may be parked
while it still is the current work of the worker: poll gets armed for it,
or it is put on ->futex_list or ->waitid_list.
io-wq worker io_try_cancel()
io_wq_submit_work()
io_arm_poll_handler()
io_poll_add_hash()
io_async_cancel_one()
io_cancel_cb()
stamps the request
sets IO_WQ_WORK_CANCEL
= -EALREADY
io_poll_cancel()
skips the request
= -ENOENT
io_assign_current_work(NULL)
The flag comes too late for the worker to see it, and the search that
would have found the request takes it for done. The cancellation returns,
and the request stays.
With a multishot poll, IOSQE_ASYNC, and IORING_OP_ASYNC_CANCEL with
IORING_ASYNC_CANCEL_ALL up to 63us later, that happens to 4 of 1,000
requests (4 CPUs, KVM).
Take the stamp off again if the work matches. If the request is found
on one of the lists it gets a new one, and io_cancel_cb() doesn't match it
a second time.
__io_async_cancel() tries the io-wq of the caller twice. So far the stamp
has kept it from counting work that is running there the second time.
Skip it instead, so that the number that is returned stays the same.
Fixes: 8e29da69fead ("io_uring: add support for IORING_ASYNC_CANCEL_ALL")
Assisted-by: LLM
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Link: https://patch.msgid.link/20261015-b4-io-uring-fixes-v1-18-c1556f5d7bf5@toxicpanda.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
---
io_uring/cancel.c | 14 +++++++++++++-
1 file changed, 13 insertions(+), 1 deletion(-)
diff --git a/io_uring/cancel.c b/io_uring/cancel.c
index c33c556bcf43..50ec818aa19f 100644
--- a/io_uring/cancel.c
+++ b/io_uring/cancel.c
@@ -73,7 +73,16 @@ static bool io_cancel_cb(struct io_wq_work *work, void *data)
struct io_kiocb *req = container_of(work, struct io_kiocb, work);
struct io_cancel_data *cd = data;
- return io_cancel_req_match(req, cd);
+ if (!io_cancel_req_match(req, cd))
+ return false;
+ /*
+ * Pending work is gone once it has matched. Work that is running may
+ * park the request where the same cancellation looks next. Don't make
+ * it skip the request as one that it has dealt with.
+ */
+ if (cd->flags & (IORING_ASYNC_CANCEL_ALL | IORING_ASYNC_CANCEL_ANY))
+ req->cancel_seq_set = false;
+ return true;
}
static int io_async_cancel_one(struct io_uring_task *tctx,
@@ -190,6 +199,9 @@ static int __io_async_cancel(struct io_cancel_data *cd,
io_ring_submit_lock(ctx, issue_flags);
ret = -ENOENT;
list_for_each_entry(node, &ctx->tctx_list, ctx_node) {
+ /* has been done above, don't count what is running there */
+ if (all && node->task->io_uring == tctx)
+ continue;
ret = io_async_cancel_one(node->task->io_uring, cd);
if (ret != -ENOENT) {
if (!all)
--
2.56.0
[-- Attachment #35: 0015-io_uring-cancel-recognize-no-timeout-on-32-bit.patch --]
[-- Type: text/x-patch, Size: 1903 bytes --]
From 2014a9590fbee54ef7b4d8b9bdd3c49df879fa56 Mon Sep 17 00:00:00 2001
From: Josef Bacik <josef@toxicpanda.com>
Date: Thu, 15 Oct 2026 19:30:41 +0000
Subject: [PATCH 15/48] io_uring/cancel: recognize "no timeout" on 32-bit
Commit 3774e4d4c8533da2caae95556c4d635c7c4f45b8 upstream.
IORING_REGISTER_SYNC_CANCEL waits without a limit if both fields of
io_uring_sync_cancel_reg::timeout are -1. io_sync_cancel() compares them
with -1UL. They are of type long long. With a long of 32 bits -1UL is
4294967295, which is what it stays when it is converted for the
comparison, and the fields are never equal to it.
So { -1, -1 } is taken for a timeout. ktime_set() takes the nanoseconds as
an unsigned long, so what becomes of it is -1s + 4294967295ns = 3.29s.
If the request is being run by io-wq and can't be interrupted, the call
returns -ETIME after that time, rather than waiting for the request to be
gone. On i386, with a buffered read from a null_blk device that takes 4s
to complete a request, IOSQE_ASYNC, that is what each call does.
Fixes: 78a861b94959 ("io_uring: add sync cancelation API through io_uring_register()")
Assisted-by: LLM
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Link: https://patch.msgid.link/20261015-b4-io-uring-fixes-v1-17-c1556f5d7bf5@toxicpanda.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
---
io_uring/cancel.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/io_uring/cancel.c b/io_uring/cancel.c
index 2c83f5ab3001..c33c556bcf43 100644
--- a/io_uring/cancel.c
+++ b/io_uring/cancel.c
@@ -310,7 +310,7 @@ int io_sync_cancel(struct io_ring_ctx *ctx, void __user *arg)
if (ret != -EALREADY)
goto out;
- if (sc.timeout.tv_sec != -1UL || sc.timeout.tv_nsec != -1UL) {
+ if (sc.timeout.tv_sec != -1 || sc.timeout.tv_nsec != -1) {
struct timespec64 ts = {
.tv_sec = sc.timeout.tv_sec,
.tv_nsec = sc.timeout.tv_nsec
--
2.56.0
[-- Attachment #36: 0014-io_uring-cancel-don-t-forget-that-the-work-was-runni.patch --]
[-- Type: text/x-patch, Size: 3859 bytes --]
From 042064faa8cb9c47ac8eefbeaa53b9e37e34b96e Mon Sep 17 00:00:00 2001
From: Josef Bacik <josef@toxicpanda.com>
Date: Thu, 15 Oct 2026 19:30:40 +0000
Subject: [PATCH 14/48] io_uring/cancel: don't forget that the work was running
Commit 8aa0e5db4845fb5ac11abe40d7b1cac0656c3ee3 upstream.
If the request to be cancelled is being run by io-wq,
io_async_cancel_one() flags the work, signals the worker and returns
-EALREADY. Since commit ccbf726171b7 ("io_uring: perform poll removal even
if async work removal is successful") io_try_cancel() goes on regardless,
as poll may be armed for the request. If it isn't, and nothing else turns
up, the result is the -ENOENT of the last search.
__io_async_cancel() makes up for that. On -ENOENT it tries the io-wq of
every task that uses the ring, and that of the caller is one of them. So
the work is found a second time, and the result is -EALREADY after all.
Unless the worker is done with it by then. If the work is flagged before
io_wq_submit_work() looks at IO_WQ_WORK_CANCEL, all the worker does is to
queue the request to be failed by task work. Then it isn't anywhere to be
found until the task work has run, and
- IORING_REGISTER_SYNC_CANCEL returns -ENOENT,
- IORING_OP_ASYNC_CANCEL completes with -ENOENT,
for a request that they have just cancelled, and that completes with
-ECANCELED.
With a read of an empty pipe, IOSQE_ASYNC, and a sync cancel right behind
it, 1 to 3 of 1,000,000 calls return -ENOENT, and 50 to 110 with
IORING_SETUP_DEFER_TASKRUN (4 CPUs, KVM). Nothing is written to the pipe,
and all of these reads end with -ECANCELED.
Remember that the work was found. IORING_OP_ASYNC_CANCEL completes with
-EALREADY, as it does if the worker is a bit slower. For
IORING_REGISTER_SYNC_CANCEL it means to try again, which ends with 0.
The result of a linked timeout, which calls io_try_cancel() by itself,
and the number that is returned for IORING_ASYNC_CANCEL_ALL and _ANY don't
change.
Fixes: ccbf726171b7 ("io_uring: perform poll removal even if async work removal is successful")
Assisted-by: LLM
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Link: https://patch.msgid.link/20261015-b4-io-uring-fixes-v1-16-c1556f5d7bf5@toxicpanda.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
---
io_uring/cancel.c | 13 ++++++++++++-
io_uring/cancel.h | 2 ++
2 files changed, 14 insertions(+), 1 deletion(-)
diff --git a/io_uring/cancel.c b/io_uring/cancel.c
index 299071d28441..2c83f5ab3001 100644
--- a/io_uring/cancel.c
+++ b/io_uring/cancel.c
@@ -118,6 +118,8 @@ int io_try_cancel(struct io_uring_task *tctx, struct io_cancel_data *cd,
*/
if (!ret)
return 0;
+ if (ret == -EALREADY)
+ cd->running = true;
ret = io_poll_cancel(ctx, cd, issue_flags);
if (ret != -ENOENT)
@@ -174,6 +176,7 @@ static int __io_async_cancel(struct io_cancel_data *cd,
struct io_tctx_node *node;
int ret, nr = 0;
+ cd->running = false;
do {
ret = io_try_cancel(tctx, cd, issue_flags);
if (ret == -ENOENT)
@@ -195,7 +198,15 @@ static int __io_async_cancel(struct io_cancel_data *cd,
}
}
io_ring_submit_unlock(ctx, issue_flags);
- return all ? nr : ret;
+ if (all)
+ return nr;
+ /*
+ * The worker may be done with the work that was found running, and
+ * flagged, above. That isn't the same as not having found it.
+ */
+ if (ret == -ENOENT && cd->running)
+ ret = -EALREADY;
+ return ret;
}
int io_async_cancel(struct io_kiocb *req, unsigned int issue_flags)
diff --git a/io_uring/cancel.h b/io_uring/cancel.h
index 1b201a094303..e03a6fb0b7f9 100644
--- a/io_uring/cancel.h
+++ b/io_uring/cancel.h
@@ -11,6 +11,8 @@ struct io_cancel_data {
u8 opcode;
u32 flags;
int seq;
+ /* io_try_cancel() has found it being run by io-wq, and flagged it */
+ bool running;
};
int io_async_cancel_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe);
--
2.56.0
[-- Attachment #37: 0013-io_uring-don-t-lose-a-cancel-as-io-wq-parks-a-reques.patch --]
[-- Type: text/x-patch, Size: 4867 bytes --]
From 446d40f12752597094eec9461dd0f34881114853 Mon Sep 17 00:00:00 2001
From: Josef Bacik <josef@toxicpanda.com>
Date: Thu, 15 Oct 2026 19:30:39 +0000
Subject: [PATCH 13/48] io_uring: don't lose a cancel as io-wq parks a request
Commit 5cb049d2203d7a81a1c1a737c8df4575e71078e3 upstream.
IORING_OP_FUTEX_WAIT, IORING_OP_FUTEX_WAITV and IORING_OP_WAITID don't
block. They put the request on a list of the ring, where a wake up or a
cancellation finds it, and return IOU_ISSUE_SKIP_COMPLETE. They do so as
well if they are run by io-wq, e.g. because of IOSQE_ASYNC.
io_wq_submit_work() looks at IO_WQ_WORK_CANCEL once, before it issues the
request. Somebody who wants to cancel the request after that, and before
it is on the list, finds it running:
io-wq worker io_try_cancel()
io_wq_submit_work()
IO_WQ_WORK_CANCEL isn't set
io_async_cancel_one()
sets IO_WQ_WORK_CANCEL
signals the worker
= -EALREADY
...
io_futex_cancel() = -ENOENT
...
io_futex_wait()
adds the request to ->futex_list
= IOU_ISSUE_SKIP_COMPLETE
Nobody looks at the flag again. IORING_OP_ASYNC_CANCEL returns -EALREADY
and the request stays.
It is the problem that "io_uring/poll: don't lose a cancel as io-wq arms
poll" deals with for poll.
Look at the flag once the request is on the list, and cancel it if it is
set. The lists are searched and the requests are added with ->uring_lock
held, and the flag is set before the search. So either the search finds
the request, or it came first and the flag is seen.
What the canceller is told doesn't change, see the other patch, also for
IORING_ASYNC_CANCEL_ALL and _ANY.
Fixes: 194bb58c6090 ("io_uring: add support for futex wake and wait")
Fixes: 8f350194d5cf ("io_uring: add support for vectored futex waits")
Fixes: f31ecf671ddc ("io_uring: add IORING_OP_WAITID support")
Assisted-by: LLM
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Link: https://patch.msgid.link/20261015-b4-io-uring-fixes-v1-15-c1556f5d7bf5@toxicpanda.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
---
io_uring/futex.c | 4 ++++
io_uring/io_uring.h | 17 +++++++++++++++++
io_uring/waitid.c | 2 ++
3 files changed, 23 insertions(+)
diff --git a/io_uring/futex.c b/io_uring/futex.c
index 7f3b97c99692..c052207a8baa 100644
--- a/io_uring/futex.c
+++ b/io_uring/futex.c
@@ -278,6 +278,8 @@ int io_futexv_wait(struct io_kiocb *req, unsigned int issue_flags)
*/
__set_current_state(TASK_RUNNING);
hlist_add_head(&req->hash_node, &ctx->futex_list);
+ if (io_wq_cancel_seen(req, issue_flags))
+ __io_futex_cancel(req);
} else {
iof->futexv_unqueued = 1;
if (woken != -1)
@@ -318,6 +320,8 @@ int io_futex_wait(struct io_kiocb *req, unsigned int issue_flags)
&ifd->q, NULL, NULL);
if (!ret) {
hlist_add_head(&req->hash_node, &ctx->futex_list);
+ if (io_wq_cancel_seen(req, issue_flags))
+ __io_futex_cancel(req);
io_ring_submit_unlock(ctx, issue_flags);
return IOU_ISSUE_SKIP_COMPLETE;
diff --git a/io_uring/io_uring.h b/io_uring/io_uring.h
index ecaba6d95ae0..899c3a3bd6f4 100644
--- a/io_uring/io_uring.h
+++ b/io_uring/io_uring.h
@@ -396,6 +396,23 @@ static inline void io_ring_submit_lock(struct io_ring_ctx *ctx,
lockdep_assert_held(&ctx->uring_lock);
}
+/*
+ * For a handler that is run by io-wq and parks the request on a list where
+ * cancellations find it, rather than completing it. Whoever tried to cancel
+ * the request before that found it running. It has flagged the work item,
+ * has searched the list in vain, and won't come back. To be called with
+ * ->uring_lock held, which the search needs as well, once the request is on
+ * the list: if the search came first, the flag is visible.
+ */
+static inline bool io_wq_cancel_seen(struct io_kiocb *req,
+ unsigned int issue_flags)
+{
+ lockdep_assert_held(&req->ctx->uring_lock);
+
+ return (issue_flags & IO_URING_F_IOWQ) &&
+ (atomic_read(&req->work.flags) & IO_WQ_WORK_CANCEL);
+}
+
static inline void io_commit_cqring(struct io_ring_ctx *ctx)
{
/* order cqe stores with ring update */
diff --git a/io_uring/waitid.c b/io_uring/waitid.c
index a91491650440..2ba2369681d0 100644
--- a/io_uring/waitid.c
+++ b/io_uring/waitid.c
@@ -349,6 +349,8 @@ int io_waitid(struct io_kiocb *req, unsigned int issue_flags)
* a waitqueue callback, or if someone cancels it.
*/
if (!io_waitid_drop_issue_ref(req)) {
+ if (io_wq_cancel_seen(req, issue_flags))
+ __io_waitid_cancel(req, true);
io_ring_submit_unlock(ctx, issue_flags);
return IOU_ISSUE_SKIP_COMPLETE;
}
--
2.56.0
[-- Attachment #38: 0012-io_uring-poll-don-t-lose-a-cancel-as-io-wq-arms-poll.patch --]
[-- Type: text/x-patch, Size: 4605 bytes --]
From 97784836dbf384f2b8166446e3f562d74c99a39a Mon Sep 17 00:00:00 2001
From: Josef Bacik <josef@toxicpanda.com>
Date: Thu, 15 Oct 2026 19:30:38 +0000
Subject: [PATCH 12/48] io_uring/poll: don't lose a cancel as io-wq arms poll
Commit 6d2d4c2530e95ff6ddeb98195d49d99dbca14369 upstream.
A request on a pollable file that was forced to io-wq is issued without
blocking there. If that ends in -EAGAIN, io_wq_submit_work() arms poll
for it and returns. It looks at IO_WQ_WORK_CANCEL once, before it issues
the request.
Somebody who wants to cancel the request in between finds it running:
io-wq worker io_try_cancel()
io_wq_submit_work()
IO_WQ_WORK_CANCEL isn't set
io_issue_sqe() = -EAGAIN
io_async_cancel_one()
sets IO_WQ_WORK_CANCEL
signals the worker
= -EALREADY
io_poll_cancel() = -ENOENT
...
= -ENOENT
io_arm_poll_handler() = IO_APOLL_OK
return
Nobody looks at the flag again, and the signal doesn't matter to a
worker that doesn't sleep. The request is armed as if nothing had
happened:
- a linked timeout completes with -ENOENT, and the request it was
meant to limit waits for as long as it takes.
- IORING_OP_ASYNC_CANCEL returns -EALREADY, and the request stays.
- IORING_REGISTER_SYNC_CANCEL waits for the request to complete if it
gets -EALREADY. Since commit 5623eb1ed035 ("io_uring/tctx: add
separate lock for list of tctx's in ctx") it doesn't sleep but spins,
so its next attempt finds the armed poll.
The same goes for a multishot poll, recv or accept that is armed from
io-wq.
With a read of an empty pipe, IOSQE_ASYNC, and a linked timeout of 5ms
or a cancel in the same submission, in a guest that is slow enough (TCG,
KASAN), the request is lost in 21 of 24 rounds of 40 or 100 attempts.
While it arms poll from io-wq, __io_arm_poll_handler() owns the request.
Look at the flag once the request is in the hash table. The table is
searched and the request is added with ->uring_lock held, and the flag is
set before the search. So either the search finds the request, or it
came first and the flag is seen. With the request marked as cancelled, the
task work that is queued, right away or because ownership can't be
released, fails it with -ECANCELED.
What the canceller is told doesn't change. In this window a linked timeout
still completes with -ENOENT rather than -ETIME, a cancel request with
-EALREADY, and one with IORING_ASYNC_CANCEL_ALL or _ANY with a count that
doesn't include the request. But the request is cancelled, so what is new
is a request that completes with -ECANCELED after such a result.
This doesn't help in two cases:
- IORING_ASYNC_CANCEL_ALL and _ANY, if the work is flagged after the
test added here, while io-wq still has it as the current work of the
worker. io_cancel_cb() stamps it with the sequence of the cancel, so the
search of the hash table on behalf of the same cancel takes it for
done and skips it.
- requests that are parked elsewhere by io-wq, like IORING_OP_FUTEX_WAIT
and IORING_OP_WAITID. They have the same problem.
Fixes: 90fa02883f06 ("io_uring: implement async hybrid mode for pollable requests")
Assisted-by: LLM
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Link: https://patch.msgid.link/20261015-b4-io-uring-fixes-v1-14-c1556f5d7bf5@toxicpanda.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
---
io_uring/poll.c | 12 ++++++++++++
1 file changed, 12 insertions(+)
diff --git a/io_uring/poll.c b/io_uring/poll.c
index f122a0070155..f308eb793d72 100644
--- a/io_uring/poll.c
+++ b/io_uring/poll.c
@@ -614,6 +614,18 @@ static int __io_arm_poll_handler(struct io_kiocb *req,
io_poll_add_hash(req, issue_flags);
+ /*
+ * Whoever tried to cancel the request while io-wq was running it has
+ * flagged the work item. If it searched the hash table before the
+ * request got there it found nothing, and it won't come back. Both
+ * the search and the insertion are done with ->uring_lock held, and
+ * the flag is set before the search. If the search came first,
+ * the flag is visible by now.
+ */
+ if (ipt->owning &&
+ (atomic_read(&req->work.flags) & IO_WQ_WORK_CANCEL))
+ io_poll_mark_cancelled(req);
+
if (mask && (poll->events & EPOLLET) &&
io_poll_can_finish_inline(req, ipt)) {
__io_poll_execute(req, mask, 0);
--
2.56.0
[-- Attachment #39: 0011-io_uring-io-wq-let-cancel-find-picked-up-hashed-work.patch --]
[-- Type: text/x-patch, Size: 4402 bytes --]
From 24599c53a7a8df7feda07b7f0218d554b82aa621 Mon Sep 17 00:00:00 2001
From: Josef Bacik <josef@toxicpanda.com>
Date: Thu, 15 Oct 2026 19:30:37 +0000
Subject: [PATCH 11/48] io_uring/io-wq: let cancel find picked up hashed work
Commit e9efeb4fc97b0879512a3c3b995ed448f2487e12 upstream.
io_get_next_work() takes all the work items that share a hash off the
work list in one go, and the worker runs them one after the other. Only
the one it is working on is visible as ->cur_work. The others are neither
on the list nor current, so io_wq_cancel_cb() can't find them.
IORING_OP_ASYNC_CANCEL and IORING_REGISTER_SYNC_CANCEL return -ENOENT for
a request that has not even been started, e.g. a buffered write queued
behind another write to the same file that is blocked, and the request is
carried out. The cancellation done for a task or ring that goes away gets
there in the end, as it keeps trying, but it has to wait for each item to
become current.
Keep a pointer to the remainder of the chain next to ->cur_work, and have
io_wq_worker_cancel() look at it. The items on it belong to the worker,
which needs ->lock to move on to the next one. Matching work is flagged
with IO_WQ_WORK_CANCEL and fails with -ECANCELED when its turn comes. The
worker isn't signalled for it, that would interrupt the unrelated work it
is busy with. As the request isn't gone yet, it is reported like running
work, with -EALREADY.
Fixes: 86f3cd1b589a ("io-wq: handle hashed writes in chains")
Assisted-by: LLM
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Link: https://patch.msgid.link/20261015-b4-io-uring-fixes-v1-13-c1556f5d7bf5@toxicpanda.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
---
io_uring/io-wq.c | 35 +++++++++++++++++++++++++++++------
1 file changed, 29 insertions(+), 6 deletions(-)
diff --git a/io_uring/io-wq.c b/io_uring/io-wq.c
index d9da3761cb2b..2ed3eebd0a7b 100644
--- a/io_uring/io-wq.c
+++ b/io_uring/io-wq.c
@@ -54,6 +54,8 @@ struct io_worker {
struct io_wq_acct *acct;
struct io_wq_work *cur_work;
+ /* rest of the hashed chain ->cur_work was taken off the list with */
+ struct io_wq_work *next_work;
raw_spinlock_t lock;
struct completion ref_done;
@@ -579,6 +581,17 @@ static struct io_wq_work *io_get_next_work(struct io_wq_acct *acct,
return NULL;
}
+static void __io_assign_current_work(struct io_worker *worker,
+ struct io_wq_work *work)
+{
+ raw_spin_lock(&worker->lock);
+ worker->cur_work = work;
+ /* stays valid while in between two items of a chain */
+ if (work)
+ worker->next_work = wq_next_work(work);
+ raw_spin_unlock(&worker->lock);
+}
+
static void io_assign_current_work(struct io_worker *worker,
struct io_wq_work *work)
{
@@ -587,9 +600,7 @@ static void io_assign_current_work(struct io_worker *worker,
cond_resched();
}
- raw_spin_lock(&worker->lock);
- worker->cur_work = work;
- raw_spin_unlock(&worker->lock);
+ __io_assign_current_work(worker, work);
}
/*
@@ -620,9 +631,7 @@ static void io_worker_handle_work(struct io_wq_acct *acct,
* work list, but isn't yet discoverable as the
* current work item for this worker.
*/
- raw_spin_lock(&worker->lock);
- worker->cur_work = work;
- raw_spin_unlock(&worker->lock);
+ __io_assign_current_work(worker, work);
}
raw_spin_unlock(&acct->lock);
@@ -1171,6 +1180,7 @@ static bool __io_wq_worker_cancel(struct io_worker *worker,
static bool io_wq_worker_cancel(struct io_worker *worker, void *data)
{
struct io_cb_cancel_data *match = data;
+ struct io_wq_work *work;
/*
* Hold the lock to avoid ->cur_work going out of scope, caller
@@ -1179,6 +1189,19 @@ static bool io_wq_worker_cancel(struct io_worker *worker, void *data)
raw_spin_lock(&worker->lock);
if (__io_wq_worker_cancel(worker, match, worker->cur_work))
match->nr_running++;
+ /*
+ * Hashed work is taken off the work list a chain at a time, what the
+ * worker hasn't started yet can only be found here. It doesn't move on
+ * without the lock. Just mark the work, it fails once it is its turn.
+ */
+ for (work = worker->next_work; work; work = wq_next_work(work)) {
+ if (match->nr_running && !match->cancel_all)
+ break;
+ if (match->fn(work, match->data)) {
+ atomic_or(IO_WQ_WORK_CANCEL, &work->flags);
+ match->nr_running++;
+ }
+ }
raw_spin_unlock(&worker->lock);
return match->nr_running && !match->cancel_all;
--
2.56.0
[-- Attachment #40: 0010-io_uring-io-wq-don-t-exit-with-a-creation-pending.patch --]
[-- Type: text/x-patch, Size: 3276 bytes --]
From 2291eedb0df2a1f860ddfb3ecd5ed7c888421384 Mon Sep 17 00:00:00 2001
From: Josef Bacik <josef@toxicpanda.com>
Date: Thu, 15 Oct 2026 19:30:36 +0000
Subject: [PATCH 10/48] io_uring/io-wq: don't exit with a creation pending
Commit e231d2e8f80d3649ecd3851b7f2abdfbe606c04b upstream.
A worker that is the last one to go to sleep while work is pending asks
for another worker. io_wq_dec_running() bumps ->nr_running on behalf of
the new worker and queues create_worker_cb() as task_work for the owner
of the io-wq. Until that has run, everybody else takes it that a worker
is running which is going to look after the work list: io_wq_enqueue()
doesn't create one, and other workers that go to sleep don't find
themselves to be the last.
If the worker that made the request exits, io_worker_exit() cancels the
task_work, and io_worker_cancel_cb() drops ->nr_running without a look at
the work list:
worker A worker B io_wq_enqueue()
blocks, work is pending
nr_running++
queues create_worker_cb()
wakes, handles the work
idle timeout, decides to exit
picks up work 1
queues work 2
B is on the free list
still: wakes B
work 1 blocks
nr_running 2 -> 1
io_worker_exit()
io_worker_cancel_cb()
nr_running 1 -> 0
Work 2 has to wait for work 1, although there is room for more workers.
It takes an owner that doesn't get around to run its task_work for the
time of the idle timeout, less if the worker exits because its affinity
was changed or the io-wq is marked exit-on-idle.
Let the worker stay around until its request has been dealt with. Only
the worker itself sets ->create_state, so it can't be set behind its
back once it has decided to leave.
Fixes: 1d5f5ea7cb7d ("io-wq: remove worker to owner tw dependency")
Assisted-by: LLM
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Link: https://patch.msgid.link/20261015-b4-io-uring-fixes-v1-12-c1556f5d7bf5@toxicpanda.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
---
io_uring/io-wq.c | 8 ++++++--
1 file changed, 6 insertions(+), 2 deletions(-)
diff --git a/io_uring/io-wq.c b/io_uring/io-wq.c
index c4429b2fe486..d9da3761cb2b 100644
--- a/io_uring/io-wq.c
+++ b/io_uring/io-wq.c
@@ -755,9 +755,13 @@ static int io_wq_worker(void *data)
* idle-exit, drop the worker as well. This is used to avoid
* keeping io-wq workers around for tasks that no longer have
* any active io_uring instances.
+ * Not as long as a request of ours for another worker is
+ * pending, though. It is accounted as a running worker, which
+ * others rely on, and io_worker_exit() would cancel it.
*/
- if ((last_timeout && (exit_mask || acct->nr_workers > 1)) ||
- test_bit(IO_WQ_BIT_EXIT_ON_IDLE, &wq->state)) {
+ if (((last_timeout && (exit_mask || acct->nr_workers > 1)) ||
+ test_bit(IO_WQ_BIT_EXIT_ON_IDLE, &wq->state)) &&
+ !test_bit(0, &worker->create_state)) {
acct->nr_workers--;
if (test_and_clear_bit(IO_WORKER_F_FREE, &worker->flags))
hlist_nulls_del_init_rcu(&worker->nulls_node);
--
2.56.0
[-- Attachment #41: 0009-io_uring-io-wq-don-t-strand-work-when-a-worker-exits.patch --]
[-- Type: text/x-patch, Size: 4187 bytes --]
From 2abc25aa563ba54cf34319c1b0545b67edba8498 Mon Sep 17 00:00:00 2001
From: Josef Bacik <josef@toxicpanda.com>
Date: Thu, 15 Oct 2026 19:30:35 +0000
Subject: [PATCH 09/48] io_uring/io-wq: don't strand work when a worker exits
Commit ad41f325be1271b22d99353819c90f0d5e45acc8 upstream.
A worker exits if its idle timeout fired and it isn't the last one, if
its affinity was changed, or if the wq is marked exit-on-idle. It decides
so after it found the work list empty, and until io_worker_exit() is
through it stays on the free list and in ->nr_running:
worker io_wq_enqueue()
io_acct_run_queue() == false
queues work
io_acct_activate_free_worker()
wake_up_process(worker)
// found one, nothing to create
acct->nr_workers--
break
io_worker_exit()
Nobody is going to look at the work. If other workers exist, it has to
wait until one of them finishes what it is blocked on or hits its idle
timeout. If not, which is possible for the affinity and the exit-on-idle
case, it sits there until something else is queued. For exit-on-idle it
takes a task that closes its last ring and sets up a new one while the
workers are on their way out.
Make the worker take itself off the free list when it gives up its
->nr_workers slot, then leave ->nr_running, and only then take the final
look at the work list. That look and queueing work are both done under
acct->lock, so either the worker sees the work and stays, or the enqueue
side sees that the worker is gone and wakes or creates another one. If it
wants to stay and finds that its slot has been taken, a worker has been
created in its place, which is going to look at the work list, and it can
leave after all.
Fixes: 38aa434ab933 ("io_uring/io-wq: add exit-on-idle state")
Fixes: 05c5f4ee4da7 ("io-wq: get rid of FIXED worker flag")
Assisted-by: LLM
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Link: https://patch.msgid.link/20261015-b4-io-uring-fixes-v1-11-c1556f5d7bf5@toxicpanda.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
---
io_uring/io-wq.c | 36 +++++++++++++++++++++++++++++++++++-
1 file changed, 35 insertions(+), 1 deletion(-)
diff --git a/io_uring/io-wq.c b/io_uring/io-wq.c
index 5078bba7ac68..c4429b2fe486 100644
--- a/io_uring/io-wq.c
+++ b/io_uring/io-wq.c
@@ -693,6 +693,35 @@ static void io_worker_handle_work(struct io_wq_acct *acct,
} while (1);
}
+/*
+ * The worker has decided to exit, and has given up its ->nr_workers slot and
+ * its place on the free list. io_wq_enqueue() may have picked it from that
+ * list, or seen it running, after it last looked at the work list. Leave
+ * ->nr_running as well and check again. Returns false if the worker has to
+ * stay.
+ */
+static bool io_worker_retire(struct io_wq_acct *acct, struct io_worker *worker)
+{
+ /* turns io_wq_dec_running() and the scheduler hooks into no-ops */
+ clear_bit(IO_WORKER_F_UP, &worker->flags);
+ atomic_dec(&acct->nr_running);
+ if (!io_acct_run_queue(acct))
+ return true;
+ raw_spin_unlock(&acct->lock);
+
+ raw_spin_lock(&acct->workers_lock);
+ if (acct->nr_workers >= acct->max_workers) {
+ /* a worker was created in our place, the work is its business */
+ raw_spin_unlock(&acct->workers_lock);
+ return true;
+ }
+ acct->nr_workers++;
+ raw_spin_unlock(&acct->workers_lock);
+ atomic_inc(&acct->nr_running);
+ set_bit(IO_WORKER_F_UP, &worker->flags);
+ return false;
+}
+
static int io_wq_worker(void *data)
{
struct io_worker *worker = data;
@@ -730,9 +759,14 @@ static int io_wq_worker(void *data)
if ((last_timeout && (exit_mask || acct->nr_workers > 1)) ||
test_bit(IO_WQ_BIT_EXIT_ON_IDLE, &wq->state)) {
acct->nr_workers--;
+ if (test_and_clear_bit(IO_WORKER_F_FREE, &worker->flags))
+ hlist_nulls_del_init_rcu(&worker->nulls_node);
raw_spin_unlock(&acct->workers_lock);
__set_current_state(TASK_RUNNING);
- break;
+ if (io_worker_retire(acct, worker))
+ break;
+ last_timeout = false;
+ continue;
}
last_timeout = false;
__io_worker_idle(acct, worker);
--
2.56.0
[-- Attachment #42: 0008-io_uring-io-wq-don-t-go-to-sleep-on-pending-work.patch --]
[-- Type: text/x-patch, Size: 3557 bytes --]
From 5e26d7e70000079147bbcee972f1fc814bba9330 Mon Sep 17 00:00:00 2001
From: Josef Bacik <josef@toxicpanda.com>
Date: Thu, 15 Oct 2026 19:30:34 +0000
Subject: [PATCH 08/48] io_uring/io-wq: don't go to sleep on pending work
Commit f19c38feb96802ee53a99649d740d2bf630bcce7 upstream.
io_wq_enqueue() wakes a worker off the free list. If there is none it
creates one, unless ->nr_running says that a worker is running. That
relies on whoever takes ->nr_running to zero to check for pending work,
which io_wq_dec_running() no longer does for a worker without
->cur_work, on the grounds that an idle worker is either woken right
away or someone else has created a worker. Neither has to be the case:
worker A (idle) worker B (idle) io_wq_enqueue()
woken for work 1
woken (e.g. timeout) takes work 1
work list is empty
queues work 2
B is first on the free
list: wake_up_process(B)
leaves the free list
work 1 blocks
nr_running 2 -> 1
schedule_timeout()
nr_running 1 -> 0
->cur_work == NULL
Work 2 isn't looked at until work 1 is done or A's idle timeout fires,
five seconds later. ->cur_work is also NULL while a worker blocks on
->uring_lock in io_wq_put_work(), with the same result if it was the
only one running.
Let the idle worker leave ->nr_running by itself before it takes its last
look at the work list. Both that and queueing work are done under
acct->lock, so either the worker sees the work, or the enqueue side sees
that it can't count on this worker. This doesn't bring back the request
for a new worker which the check was meant to avoid, and as idle workers
don't get to io_wq_dec_running() any more, the check can go.
Fixes: 8343cae362e1 ("io_uring/io-wq: ignore non-busy worker going to sleep")
Assisted-by: LLM
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Link: https://patch.msgid.link/20261015-b4-io-uring-fixes-v1-10-c1556f5d7bf5@toxicpanda.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
---
io_uring/io-wq.c | 17 ++++++++++++++---
1 file changed, 14 insertions(+), 3 deletions(-)
diff --git a/io_uring/io-wq.c b/io_uring/io-wq.c
index 1094d3fac47f..5078bba7ac68 100644
--- a/io_uring/io-wq.c
+++ b/io_uring/io-wq.c
@@ -465,11 +465,9 @@ static void io_wq_dec_running(struct io_worker *worker)
if (!atomic_dec_and_test(&acct->nr_running))
return;
- if (!worker->cur_work)
- return;
if (!io_acct_run_queue(acct))
return;
- if (io_wq_hash_defer(worker->cur_work, acct)) {
+ if (worker->cur_work && io_wq_hash_defer(worker->cur_work, acct)) {
raw_spin_unlock(&acct->lock);
return;
}
@@ -741,6 +739,19 @@ static int io_wq_worker(void *data)
raw_spin_unlock(&acct->workers_lock);
if (io_run_task_work())
continue;
+ /*
+ * io_wq_enqueue() leaves new work to us for as long as we are
+ * accounted as running. Get out of ->nr_running before taking
+ * the last look at the work list, rather than when scheduling
+ * out. No need to ask for another worker then, we can do it.
+ */
+ clear_bit(IO_WORKER_F_RUNNING, &worker->flags);
+ atomic_dec(&acct->nr_running);
+ if (io_acct_run_queue(acct)) {
+ io_wq_worker_running(current);
+ io_worker_handle_work(acct, worker);
+ continue;
+ }
ret = schedule_timeout(WORKER_IDLE_TIMEOUT);
if (signal_pending(current)) {
struct ksignal ksig;
--
2.56.0
[-- Attachment #43: 0007-io_uring-don-t-flood-a-cancelling-task-with-task_wor.patch --]
[-- Type: text/x-patch, Size: 3560 bytes --]
From ff9e39cd140fed9452a68240a5276c9ea33ec914 Mon Sep 17 00:00:00 2001
From: Josef Bacik <josef@toxicpanda.com>
Date: Thu, 15 Oct 2026 19:30:30 +0000
Subject: [PATCH 07/48] io_uring: don't flood a cancelling task with task_work
Commit 22e643e61b9b6299b42ff7d88bab48bccada8a5d upstream.
io_ring_exit_work() asks each task which has used the ring to remove its
tctx node, by task_work, and repeats that until the list is empty.
io_tctx_exit_cb() doesn't touch the node while the task is in
io_uring_cancel_generic(), which removes all the nodes at its end. So the
work item queues the callback again as soon as it has completed, for
nothing.
That is more than a waste of time. task_work_run() only returns when it
finds the list empty. If the rings are released from inside the
cancellation, because requests of other rings held the last references to
their files, up to 64 work items keep sending callbacks to the one task
which is running them. It never finds the list empty, never gets back to
the cancellation, the nodes stay, and so it goes on: a task with 60 pairs
of rings which poll each other ran 280000 callbacks in 5 seconds, 20
seconds after exit_group(). It ends when all the senders happen to be late
at the same time. Until then the task keeps a CPU and the work items busy.
An exiting task can't be killed meanwhile. execve() cancels the requests in
the same way, so a live task can get stuck there as well, for minutes. If
it takes more than five, the work items run into the
WARN_ON_ONCE(time_after(jiffies, timeout)) of io_ring_exit_work(). No
privilege is needed for any of this.
If the callback has met a task in cancellation, wait a little before it is
sent again. Nobody has to wake the work item up for that: either the node
is gone by then, or it is the turn of the next round.
Fixes: d56d938b4bef ("io_uring: do ctx initiated file note removal")
Assisted-by: LLM
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Link: https://patch.msgid.link/20261015-b4-io-uring-fixes-v1-6-c1556f5d7bf5@toxicpanda.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
---
io_uring/io_uring.c | 12 +++++++++++-
1 file changed, 11 insertions(+), 1 deletion(-)
diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c
index 205a2db9d869..68a86a2c5d7c 100644
--- a/io_uring/io_uring.c
+++ b/io_uring/io_uring.c
@@ -3003,6 +3003,7 @@ struct io_tctx_exit {
struct callback_head task_work;
struct completion completion;
struct io_ring_ctx *ctx;
+ bool in_cancel;
};
static __cold void io_tctx_exit_cb(struct callback_head *cb)
@@ -3017,7 +3018,8 @@ static __cold void io_tctx_exit_cb(struct callback_head *cb)
* tctx can be NULL if the queueing of this task_work raced with
* work cancelation off the exec path.
*/
- if (tctx && !atomic_read(&tctx->in_cancel))
+ work->in_cancel = tctx && atomic_read(&tctx->in_cancel);
+ if (tctx && !work->in_cancel)
io_uring_del_tctx_node((unsigned long)work->ctx);
complete(&work->completion);
}
@@ -3110,6 +3112,14 @@ static __cold void io_ring_exit_work(struct work_struct *work)
* wait is marked as interruptible.
*/
wait_for_completion_interruptible(&exit.completion);
+ /*
+ * The task removes the node when it is done with its
+ * cancellation. Give it the time for that: it doesn't get back
+ * to it as long as it finds new task_work each time it has run
+ * the old one, and we may be one of many who send it.
+ */
+ if (exit.in_cancel)
+ schedule_timeout_idle(HZ / 20);
mutex_lock(&ctx->uring_lock);
}
mutex_unlock(&ctx->uring_lock);
--
2.56.0
[-- Attachment #44: 0006-io_uring-io-wq-don-t-hide-a-link-from-cancellations.patch --]
[-- Type: text/x-patch, Size: 6927 bytes --]
From 54dca1db46e44ae92771797f7da2d7fa51f5aec3 Mon Sep 17 00:00:00 2001
From: Josef Bacik <josef@toxicpanda.com>
Date: Thu, 15 Oct 2026 19:30:29 +0000
Subject: [PATCH 06/48] io_uring/io-wq: don't hide a link from cancellations
Commit 436d48a2ed13112d2d27085325838e5bce6b4507 upstream.
A task that exits cancels its tracked requests and waits until they are
gone. io_uring_cancel_generic() looks everywhere a request can be, and
goes to sleep if it found nothing to cancel. It is woken up each time
one of the requests of the task is completed by task work or freed.
When io-wq is done with a request that has a link, there is a time in which
the next request of the link can't be found:
io_worker_handle_work()
io_assign_current_work(worker, NULL)
io_wq_free_work(work)
io_req_find_next() // takes the link off the request
io_free_req() // task work for the task
io_assign_current_work(worker, linked)
io_run_task_work()
cond_resched()
worker->cur_work = linked
It is not pending, it is not the current work of a worker, and it is not
yet on the list that it is about to be parked on. If the pass that the
exiting task makes because of io_free_req() falls into that time, it finds
nothing and the task goes to sleep. The worker then runs the link, which
IO_WQ_BIT_EXIT doesn't prevent for bounded work. If that parks the request
nothing is completed or freed, and nobody wakes up the task. If it is the
process that exits it can't be killed any more either.
NOP, IOSQE_ASYNC | IOSQE_IO_LINK
FUTEX_WAIT, FUTEX2_PRIVATE
exit of the thread, or of the process, up to 63us later
With four of these per round 5 to 8 of 10,000 rounds hang (4 CPUs, KVM):
io_uring_cancel_generic
do_exit
Commit f8b632e89a10 ("io_uring: tighten task exit cancellations") is about
the same gap, and makes the task look again for whatever happens before
io_free_req().
Free the request once the link is the current work of the worker, or
pending. Whoever is woken up by that can find the link, until it is parked
where cancellations look as well. To do so split io_wq_free_work() in two.
[ 6.18: io_wq_free_work() has no REQ_F_CQE_SKIP link check yet (5530a5102e3c), kept its link logic in the split ]
Fixes: 768134d4f481 ("io_uring: don't do flush cancel under inflight_lock")
Assisted-by: LLM
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Link: https://patch.msgid.link/20261015-b4-io-uring-fixes-v1-5-c1556f5d7bf5@toxicpanda.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
---
io_uring/io-wq.c | 18 +++++++++++++++---
io_uring/io_uring.c | 29 +++++++++++++++++++++--------
io_uring/io_uring.h | 3 ++-
3 files changed, 38 insertions(+), 12 deletions(-)
diff --git a/io_uring/io-wq.c b/io_uring/io-wq.c
index ac7b820f0cf8..1094d3fac47f 100644
--- a/io_uring/io-wq.c
+++ b/io_uring/io-wq.c
@@ -640,11 +640,12 @@ static void io_worker_handle_work(struct io_wq_acct *acct,
/* handle a whole dependent link */
do {
bool do_kill = test_bit(IO_WQ_BIT_EXIT, &wq->state);
- struct io_wq_work *next_hashed, *linked;
+ struct io_wq_work *next_hashed, *linked, *done = work;
unsigned int work_flags = atomic_read(&work->flags);
unsigned int hash = __io_wq_is_hashed(work_flags)
? __io_get_work_hash(work_flags)
: -1U;
+ bool last;
next_hashed = wq_next_work(work);
@@ -660,7 +661,7 @@ static void io_worker_handle_work(struct io_wq_acct *acct,
io_wq_submit_work(work);
io_assign_current_work(worker, NULL);
- linked = io_wq_free_work(work);
+ last = io_wq_put_work(work, &linked);
work = next_hashed;
if (!work && linked && !io_wq_is_hashed(linked)) {
work = linked;
@@ -669,6 +670,13 @@ static void io_worker_handle_work(struct io_wq_acct *acct,
io_assign_current_work(worker, work);
if (linked)
io_wq_enqueue(wq, linked);
+ /*
+ * Not before the link is current or pending work again.
+ * This may be the last time that a task is woken up
+ * which exits and looks for requests to cancel.
+ */
+ if (last)
+ io_wq_free_work(done);
if (hash != -1U && !next_hashed) {
/* serialize hash clear with wake_up() */
@@ -1000,9 +1008,13 @@ void io_wq_set_exit_on_idle(struct io_wq *wq, bool enable)
static void io_run_cancel(struct io_wq_work *work, struct io_wq *wq)
{
do {
+ struct io_wq_work *linked;
+
atomic_or(IO_WQ_WORK_CANCEL, &work->flags);
io_wq_submit_work(work);
- work = io_wq_free_work(work);
+ if (io_wq_put_work(work, &linked))
+ io_wq_free_work(work);
+ work = linked;
} while (work);
}
diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c
index 8872facd86ac..205a2db9d869 100644
--- a/io_uring/io_uring.c
+++ b/io_uring/io_uring.c
@@ -1886,17 +1886,30 @@ int io_poll_issue(struct io_kiocb *req, io_tw_token_t tw)
return ret;
}
-struct io_wq_work *io_wq_free_work(struct io_wq_work *work)
+/*
+ * io-wq is done with @work. Returns true if it held the last reference, and
+ * hands out the next request of the link then. The caller has to make sure
+ * that cancellations can find the latter before it calls io_wq_free_work().
+ */
+bool io_wq_put_work(struct io_wq_work *work, struct io_wq_work **linked)
{
struct io_kiocb *req = container_of(work, struct io_kiocb, work);
struct io_kiocb *nxt = NULL;
- if (req_ref_put_and_test_atomic(req)) {
- if (req->flags & IO_REQ_LINK_FLAGS)
- nxt = io_req_find_next(req);
- io_free_req(req);
- }
- return nxt ? &nxt->work : NULL;
+ *linked = NULL;
+ if (!req_ref_put_and_test_atomic(req))
+ return false;
+
+ if (req->flags & IO_REQ_LINK_FLAGS)
+ nxt = io_req_find_next(req);
+ if (nxt)
+ *linked = &nxt->work;
+ return true;
+}
+
+void io_wq_free_work(struct io_wq_work *work)
+{
+ io_free_req(container_of(work, struct io_kiocb, work));
}
void io_wq_submit_work(struct io_wq_work *work)
@@ -1907,7 +1920,7 @@ void io_wq_submit_work(struct io_wq_work *work)
bool needs_poll = false;
int ret = 0, err = -ECANCELED;
- /* one will be dropped by io_wq_free_work() after returning to io-wq */
+ /* one will be dropped by io_wq_put_work() after returning to io-wq */
if (!(req->flags & REQ_F_REFCOUNT))
__io_req_set_refcount(req, 2);
else
diff --git a/io_uring/io_uring.h b/io_uring/io_uring.h
index 36918faff5af..ecaba6d95ae0 100644
--- a/io_uring/io_uring.h
+++ b/io_uring/io_uring.h
@@ -185,7 +185,8 @@ int io_do_iopoll(struct io_ring_ctx *ctx, bool force_nonspin);
__cold void io_iopoll_try_reap_events(struct io_ring_ctx *ctx);
void __io_submit_flush_completions(struct io_ring_ctx *ctx);
-struct io_wq_work *io_wq_free_work(struct io_wq_work *work);
+bool io_wq_put_work(struct io_wq_work *work, struct io_wq_work **linked);
+void io_wq_free_work(struct io_wq_work *work);
void io_wq_submit_work(struct io_wq_work *work);
void io_free_req(struct io_kiocb *req);
--
2.56.0
[-- Attachment #45: 0005-io_uring-io-wq-don-t-lose-the-signal-that-ends-work.patch --]
[-- Type: text/x-patch, Size: 3158 bytes --]
From 7b74a7886923455c9275a18d026ab56a0b542473 Mon Sep 17 00:00:00 2001
From: Josef Bacik <josef@toxicpanda.com>
Date: Thu, 15 Oct 2026 19:30:28 +0000
Subject: [PATCH 05/48] io_uring/io-wq: don't lose the signal that ends work
Commit 63c103279a95e82280092f2f4232e91e8117a252 upstream.
When a thread exits, only its tracked requests are cancelled. Then its
io-wq is torn down. io_wq_exit_workers() signals every worker, to get it
out of an interruptible sleep, and waits for all of them to exit. Before
they do, they go through the work that is left. Unbounded work is
cancelled. Bounded work is run, as it is expected to end by itself.
Not all of it does. An open has no file yet, so it counts as bounded, and
the open of a FIFO waits for the other end. That is what the signal is for.
But the workers are signalled once, and a worker that hasn't started the
work by then loses the signal:
io_wq_exit_workers() io_worker_handle_work()
io_wq_worker_wake()
set TIF_NOTIFY_SIGNAL
io_assign_current_work()
io_run_task_work()
clear TIF_NOTIFY_SIGNAL
io_wq_submit_work()
fifo_open()
wait_for_partner()
wait_for_completion_timeout()
The thread never gets out of do_exit():
IORING_OP_OPENAT of a FIFO without a writer, IOSQE_ASYNC
exit of the thread that has submitted it
hangs in more than 9 of 10 attempts (4 CPUs, KVM), until the process is
killed.
Set TIF_NOTIFY_SIGNAL for bounded work that is started with IO_WQ_BIT_EXIT
set. The bit is set before the workers are signalled, and it is tested
after the flag has been cleared, with clear_notify_signal() providing the
barrier. So either the flag stays, or the bit is seen.
Work that is started while the io-wq exits is treated like work that is
being run at that time: it isn't interrupted unless it waits
interruptibly.
Fixes: afcc4015d1bf ("io-wq: provide an io_wq_put_and_exit() helper")
Assisted-by: LLM
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Link: https://patch.msgid.link/20261015-b4-io-uring-fixes-v1-4-c1556f5d7bf5@toxicpanda.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
---
io_uring/io-wq.c | 10 ++++++++--
1 file changed, 8 insertions(+), 2 deletions(-)
diff --git a/io_uring/io-wq.c b/io_uring/io-wq.c
index 19514059ea3d..ac7b820f0cf8 100644
--- a/io_uring/io-wq.c
+++ b/io_uring/io-wq.c
@@ -648,9 +648,15 @@ static void io_worker_handle_work(struct io_wq_acct *acct,
next_hashed = wq_next_work(work);
- if (do_kill &&
- (work_flags & IO_WQ_WORK_UNBOUND))
+ /*
+ * io_wq_exit_workers() signals us once, so that we
+ * don't sleep for ever. If that was before this work
+ * became the current one, the signal is gone.
+ */
+ if (do_kill && (work_flags & IO_WQ_WORK_UNBOUND))
atomic_or(IO_WQ_WORK_CANCEL, &work->flags);
+ else if (do_kill)
+ set_notify_signal(current);
io_wq_submit_work(work);
io_assign_current_work(worker, NULL);
--
2.56.0
[-- Attachment #46: 0004-io_uring-wait-don-t-spin-on-a-stuck-overflow-list.patch --]
[-- Type: text/x-patch, Size: 2841 bytes --]
From e2656f64aa7318ca82194ec6d882908bdc89d784 Mon Sep 17 00:00:00 2001
From: Josef Bacik <josef@toxicpanda.com>
Date: Thu, 15 Oct 2026 19:30:27 +0000
Subject: [PATCH 04/48] io_uring/wait: don't spin on a stuck overflow list
Commit 34f5a664c34d3cb62f6bec7f4156754594a7d25b upstream.
As long as IO_CHECK_CQ_OVERFLOW_BIT is set, io_cqring_wait_schedule()
returns right away, before it gets to look at signals or arms the timeout,
for io_cqring_wait() to flush the overflow list. If that doesn't empty the
list, and the CQ ring doesn't hold the number of events the caller wants,
the loop starts over, with the same result. The task spins in the kernel,
and can't be killed.
The assumption is that this can't happen: if CQEs can't be flushed, the
ring is full, and min_events is capped to the size of the ring. But:
- With IORING_SETUP_CQE_MIXED a 32b CQE needs two contiguous entries. It
is stuck on the list if a single one is free, and then the ring holds
less than cq_entries events. A single threaded application that asks
for as many events as the ring has entries never gets out of
io_uring_enter().
- Whether the ring is full is up to the CQ head, which the application
can write to. While one thread waits for events on an empty ring,
another one can move the head back so that the ring looks full and
have a CQE overflow. The waiter can't be stopped any more, not even by
SIGKILL.
Only the application can make room in the CQ ring. Return to it if the
list could not be flushed.
[ 6.18: io_cqring_wait() lives in io_uring.c, there is no wait.c yet ]
Fixes: 3fcf19d592d5 ("io_uring: parse check_cq out of wq waiting")
Assisted-by: LLM
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Link: https://patch.msgid.link/20261015-b4-io-uring-fixes-v1-3-c1556f5d7bf5@toxicpanda.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
---
io_uring/io_uring.c | 13 +++++++++++--
1 file changed, 11 insertions(+), 2 deletions(-)
diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c
index 668662b74d73..8872facd86ac 100644
--- a/io_uring/io_uring.c
+++ b/io_uring/io_uring.c
@@ -2749,9 +2749,18 @@ static int io_cqring_wait(struct io_ring_ctx *ctx, int min_events, u32 flags,
check_cq = READ_ONCE(ctx->check_cq);
if (unlikely(check_cq)) {
- /* let the caller flush overflows, retry */
- if (check_cq & BIT(IO_CHECK_CQ_OVERFLOW_BIT))
+ if (check_cq & BIT(IO_CHECK_CQ_OVERFLOW_BIT)) {
io_cqring_do_overflow_flush(ctx);
+ /*
+ * No room for what has overflown. It takes the
+ * application to change that, so return to it.
+ */
+ if (test_bit(IO_CHECK_CQ_OVERFLOW_BIT,
+ &ctx->check_cq)) {
+ ret = 0;
+ break;
+ }
+ }
if (check_cq & BIT(IO_CHECK_CQ_DROPPED_BIT)) {
ret = -EBADR;
break;
--
2.56.0
[-- Attachment #47: 0003-io_uring-cmd-don-t-leave-list-poison-in-hash_node.patch --]
[-- Type: text/x-patch, Size: 2180 bytes --]
From 350f3ab86e068736e52c72d15caabff1ac286fc1 Mon Sep 17 00:00:00 2001
From: Josef Bacik <josef@toxicpanda.com>
Date: Thu, 15 Oct 2026 19:30:26 +0000
Subject: [PATCH 03/48] io_uring/cmd: don't leave list poison in ->hash_node
Commit 4c83e39f4d1944d7146fd06094f2c8053238d55c upstream.
io_uring_cmd_del_cancelable() uses hlist_del() to take a command off
->cancelable_uring_cmd, which leaves LIST_POISON1/2 behind in
->hash_node. Requests are recycled through the request cache without
being reinitialised, and not every user of ->hash_node sets it up before
it is used:
If a vectored futex wait is woken while futex_wait_multiple_setup() is
still queueing the futexes, io_futexv_wait() doesn't add the request to
->futex_list, and leaves the completion to the task_work queued by the
wake up. io_futexv_complete() ends in __io_futex_complete(), whose
hlist_del_init() is fine for a request that is unhashed, which is how all
other users of the node leave it. If the request was a cancelable
uring_cmd in its previous life, e.g. a ublk FETCH_REQ, it writes through
LIST_POISON2:
Oops: general protection fault, probably for non-canonical address
KASAN: maybe wild-memory-access in range [0xdead000000000120-...]
RIP: 0010:io_futexv_complete+0x219/0x420
Call Trace:
tctx_task_work_run+0x804/0x950
tctx_task_work+0x71/0xa0
task_work_run+0x118/0x200
Leave the node unhashed.
Fixes: 93b8cc60c37b ("io_uring: cancelable uring_cmd")
Assisted-by: LLM
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Link: https://patch.msgid.link/20261015-b4-io-uring-fixes-v1-2-c1556f5d7bf5@toxicpanda.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
---
io_uring/uring_cmd.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/io_uring/uring_cmd.c b/io_uring/uring_cmd.c
index d0fec7bb87d4..526fdf103499 100644
--- a/io_uring/uring_cmd.c
+++ b/io_uring/uring_cmd.c
@@ -87,7 +87,7 @@ static void io_uring_cmd_del_cancelable(struct io_uring_cmd *cmd,
cmd->flags &= ~IORING_URING_CMD_CANCELABLE;
io_ring_submit_lock(ctx, issue_flags);
- hlist_del(&req->hash_node);
+ hlist_del_init(&req->hash_node);
io_ring_submit_unlock(ctx, issue_flags);
}
--
2.56.0
[-- Attachment #48: 0002-io_uring-waitid-own-the-request-before-clearing-head.patch --]
[-- Type: text/x-patch, Size: 3288 bytes --]
From 5f1cdc99b92a2ea3d3dfb056e763f74e2e402e56 Mon Sep 17 00:00:00 2001
From: Josef Bacik <josef@toxicpanda.com>
Date: Thu, 15 Oct 2026 19:30:25 +0000
Subject: [PATCH 02/48] io_uring/waitid: own the request before clearing ->head
Commit c254eec4364b9f4b4cbdce43be3fb4df5a651a70 upstream.
io_waitid_remove_wq() only takes the wait queue lock if ->head is set.
io_waitid_wait() clears ->head first thing, before it tries to become the
owner of the request by bumping ->refs. If somebody else owns the request
at that point, nothing keeps it from completing the request while the
wake up handler is still going to touch it:
CPU0 (io_waitid_wait) CPU1 (io_waitid, holds a ref)
list_del_init(&wait->entry)
smp_store_release(&iw->head, NULL)
__do_wait() finds another child
io_waitid_remove_wq()
// ->head == NULL, no locking
io_waitid_finish(), IOU_COMPLETE
// request freed and reused
atomic_fetch_inc(&iw->refs)
The same goes for __io_waitid_cancel() if it claims the request in that
window. ->refs is part of the per opcode data of the request, so this
increments whatever the next user of the request keeps there. If that
happens to be zero, the handler concludes that it owns the request, and
queues io_waitid_cb() for it. With the window held open by a delay, and
the request reused for a receive that is waiting for data:
BUG: KASAN: slab-use-after-free in __wake_up_common+0x1ca/0x1f0
__wake_up_sync_key+0x36/0x50
sock_def_readable+0xf1/0x420
unix_stream_sendmsg+0x60b/0x1110
Allocated by task 87:
io_arm_apoll+0x1ed/0x940
Freed by task 87:
__io_submit_flush_completions+0x118d/0x1a10
tctx_task_work_run+0x327/0x950
Only clear ->head once the handler owns the request. If it doesn't, the
owner finds ->head set and has to wait for the lock, which is held by the
caller of the handler. Deleting the already deleted entry again is
harmless.
Fixes: ab673c1bcaf2 ("io_uring/waitid: use io_waitid_remove_wq() consistently")
Assisted-by: LLM
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Link: https://patch.msgid.link/20261015-b4-io-uring-fixes-v1-1-c1556f5d7bf5@toxicpanda.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
---
io_uring/waitid.c | 10 ++++++++--
1 file changed, 8 insertions(+), 2 deletions(-)
diff --git a/io_uring/waitid.c b/io_uring/waitid.c
index 2b756017423f..a91491650440 100644
--- a/io_uring/waitid.c
+++ b/io_uring/waitid.c
@@ -265,12 +265,18 @@ static int io_waitid_wait(struct wait_queue_entry *wait, unsigned mode,
return 0;
list_del_init(&wait->entry);
- smp_store_release(&iw->head, NULL);
- /* cancel is in progress */
+ /*
+ * Cancel or issue is in progress. Leave ->head alone, so that whoever
+ * owns the request has to take the wait queue lock, which we hold,
+ * before it can complete and free it.
+ */
if (atomic_fetch_inc(&iw->refs) & IO_WAITID_REF_MASK)
return 1;
+ /* pairs with the smp_load_acquire() in io_waitid_remove_wq() */
+ smp_store_release(&iw->head, NULL);
+
req->io_task_work.func = io_waitid_cb;
__io_req_task_work_add(req, IOU_F_TWQ_IN_WAKE);
return 1;
--
2.56.0
[-- Attachment #49: 0001-io-wq-order-the-exit-bit-against-worker-creation-tas.patch --]
[-- Type: text/x-patch, Size: 2127 bytes --]
From 715b22baa5413d2082f4ec56a5016ed0e7aa3bf4 Mon Sep 17 00:00:00 2001
From: Christian Brauner <brauner@kernel.org>
Date: Mon, 21 Sep 2026 15:44:53 +0200
Subject: [PATCH 01/48] io-wq: order the exit bit against worker creation task
work
Commit 0e5663856b69219ef2f02e04bac9f5c5be19718a upstream.
io_wq_exit_workers() cancels task work for queued worker creation with
task_work_cancel_match(). That has a plain load of task->task_works.
io_queue_worker_create() adds the new entry and tests IO_WQ_BIT_EXIT to
cancel it if the workqueue is already on its way out.
That must be ordered. task_work_add() has a full barrier via cmpxchg().
But io_wq_exit_start() sets the bit with set_bit() which doesn't have
any memory ordering.
So afaict, on weakly ordered architectures the exiting task may load
task_works before the store of the bit is visible. The other side tests
the bit before the store is visible as well. So the entry remains queued
with worker_refs and the exiting task waits on worker_done indefinitely.
If the task still has rings then io_uring_del_tctx_node() provides the
barrier via test_and_set_bit() in io_wq_set_exit_on_idle(). When it
has closed all rings though that barrier is gone.
Add the barrier after the set_bit().
Fixes: 71a85387546e ("io-wq: check for wq exit after adding new worker task_work")
Cc: stable@vger.kernel.org
Reviewed-by: Jens Axboe <axboe@kernel.dk>
Acked-by: Oleg Nesterov <oleg@redhat.com>
Link: https://patch.msgid.link/20260921-work-coredump-fixes-v3-4-8e4adb1619e6@kernel.org
Signed-off-by: Christian Brauner (Amutable) <brauner@kernel.org>
---
io_uring/io-wq.c | 2 ++
1 file changed, 2 insertions(+)
diff --git a/io_uring/io-wq.c b/io_uring/io-wq.c
index e8b8d2c1cf7f..19514059ea3d 100644
--- a/io_uring/io-wq.c
+++ b/io_uring/io-wq.c
@@ -1320,6 +1320,8 @@ static bool io_task_work_match(struct callback_head *cb, void *data)
void io_wq_exit_start(struct io_wq *wq)
{
set_bit(IO_WQ_BIT_EXIT, &wq->state);
+ /* Pairs with task_work_add() in io_queue_worker_create(). */
+ smp_mb__after_atomic();
}
static void io_wq_cancel_tw_create(struct io_wq *wq)
--
2.56.0
reply other threads:[~2026-10-28 13:52 UTC|newest]
Thread overview: [no followups] expand[flat|nested] mbox.gz Atom feed
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=4bd1a1a7-d824-4393-89fc-4a4e8d7f7f30@kernel.dk \
--to=axboe@kernel.dk \
--cc=gregkh@linuxfoundation.org \
--cc=stable@vger.kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox