From: Jens Axboe <axboe@kernel.dk>
To: stable@vger.kernel.org
Subject: [PATCH 6.18.y 8/8] io_uring/io-wq: apply the current cpuset to new workers
Date: Sat, 21 Nov 2026 21:31:37 +0000 [thread overview]
Message-ID: <20261121213137.1789687-9-axboe@kernel.dk> (raw)
In-Reply-To: <20261121213137.1789687-1-axboe@kernel.dk>
From: Mikhail Malyshev <mike.malyshev@gmail.com>
Commit 0659e44ce739ecd856164c75a5325b7959a2921d upstream.
Since commit 84eacf177faa ("io_uring/io-wq: inherit cpuset of cgroup in
io worker"), io_wq_create() copies the cpuset of the creating task into
wq->cpu_mask, and io_init_new_worker() gives each new worker that copy.
A cpuset change, or a move of the task to another cgroup, updates the
workers that exist but not the copy, so workers started later run on
the CPUs the cgroup had when the io-wq was created, which may since
have been given to other tasks. Workers come and go for the life of the
io-wq, and a worker that cpuset moved is also retired at its next idle
timeout and replaced on the old CPUs.
Keep wq->cpu_mask for the affinity of the io-wq itself, all CPUs or the
mask set with IORING_REGISTER_IOWQ_AFF, and intersect it with the cpuset
of the worker when the worker starts, falling back to the cpuset if the
two do not intersect, as __set_cpus_allowed_ptr() does. Use the same
mask in the idle exit test, and go back to all CPUs when the registered
affinity is removed.
Fixes: 84eacf177faa ("io_uring/io-wq: inherit cpuset of cgroup in io worker")
Cc: stable@vger.kernel.org
Signed-off-by: Mikhail Malyshev <mike.malyshev@gmail.com>
Link: https://patch.msgid.link/20261111162705.1150983-1-mike.malyshev@gmail.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
---
io_uring/io-wq.c | 44 +++++++++++++++++++++++++++++++++++++++-----
1 file changed, 39 insertions(+), 5 deletions(-)
diff --git a/io_uring/io-wq.c b/io_uring/io-wq.c
index fd53297af26e..0d74e365d54a 100644
--- a/io_uring/io-wq.c
+++ b/io_uring/io-wq.c
@@ -731,6 +731,34 @@ static bool io_worker_retire(struct io_wq_acct *acct, struct io_worker *worker)
return false;
}
+/*
+ * The CPUs a worker of @wq runs on: those of the io-wq affinity that the
+ * worker's cpuset allows, or the cpuset if they have none in common, as the
+ * scheduler does with a task's requested affinity when its cpuset changes.
+ * Read the cpuset each time: it can change while the io-wq exists.
+ */
+static void io_wq_worker_cpus(struct io_wq *wq, struct task_struct *tsk,
+ struct cpumask *cpus)
+{
+ cpuset_cpus_allowed(tsk, cpus);
+ if (cpumask_intersects(cpus, wq->cpu_mask))
+ cpumask_and(cpus, cpus, wq->cpu_mask);
+}
+
+/* Is the current worker on a CPU that a new worker would not get? */
+static bool io_wq_worker_off_cpus(struct io_wq *wq)
+{
+ cpumask_var_t cpus;
+ bool off;
+
+ if (!alloc_cpumask_var(&cpus, GFP_KERNEL))
+ return false;
+ io_wq_worker_cpus(wq, current, cpus);
+ off = !cpumask_test_cpu(raw_smp_processor_id(), cpus);
+ free_cpumask_var(cpus);
+ return off;
+}
+
static int io_wq_worker(void *data)
{
struct io_worker *worker = data;
@@ -809,8 +837,7 @@ static int io_wq_worker(void *data)
}
if (!ret) {
last_timeout = true;
- exit_mask = !cpumask_test_cpu(raw_smp_processor_id(),
- wq->cpu_mask);
+ exit_mask = io_wq_worker_off_cpus(wq);
}
}
@@ -860,9 +887,16 @@ void io_wq_worker_sleeping(struct task_struct *tsk)
static void io_init_new_worker(struct io_wq *wq, struct io_wq_acct *acct, struct io_worker *worker,
struct task_struct *tsk)
{
+ cpumask_var_t cpus;
+
tsk->worker_private = worker;
worker->task = tsk;
- set_cpus_allowed_ptr(tsk, wq->cpu_mask);
+ /* Without a mask, the worker keeps its creator's CPUs. */
+ if (alloc_cpumask_var(&cpus, GFP_KERNEL)) {
+ io_wq_worker_cpus(wq, tsk, cpus);
+ set_cpus_allowed_ptr(tsk, cpus);
+ free_cpumask_var(cpus);
+ }
raw_spin_lock(&acct->workers_lock);
hlist_nulls_add_head_rcu(&worker->nulls_node, &acct->free_list);
@@ -1361,7 +1395,7 @@ struct io_wq *io_wq_create(unsigned bounded, struct io_wq_data *data)
if (!alloc_cpumask_var(&wq->cpu_mask, GFP_KERNEL))
goto err;
- cpuset_cpus_allowed(data->task, wq->cpu_mask);
+ cpumask_copy(wq->cpu_mask, cpu_possible_mask);
wq->acct[IO_WQ_ACCT_BOUND].max_workers = bounded;
wq->acct[IO_WQ_ACCT_UNBOUND].max_workers =
task_rlimit(current, RLIMIT_NPROC);
@@ -1545,7 +1579,7 @@ int io_wq_cpu_affinity(struct io_uring_task *tctx, cpumask_var_t mask)
else
ret = -EINVAL;
} else {
- cpumask_copy(tctx->io_wq->cpu_mask, allowed_mask);
+ cpumask_copy(tctx->io_wq->cpu_mask, cpu_possible_mask);
}
rcu_read_unlock();
--
2.55.0
prev parent reply other threads:[~2026-11-21 21:31 UTC|newest]
Thread overview: 9+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-11-21 21:31 [PATCH 6.18.y 0/8] io_uring fixes for 6.18-stable Jens Axboe
2026-11-21 21:31 ` [PATCH 6.18.y 1/8] io_uring/napi: don't end the single NAPI busy poll early on need_resched() Jens Axboe
2026-11-21 21:31 ` [PATCH 6.18.y 2/8] io_uring: fail soft links on negative IOU_COMPLETE results Jens Axboe
2026-11-21 21:31 ` [PATCH 6.18.y 3/8] io_uring/futex: fail soft links when cancelling a wait Jens Axboe
2026-11-21 21:31 ` [PATCH 6.18.y 4/8] io_uring/rw: fail soft links on IOPOLL short or error IO Jens Axboe
2026-11-21 21:31 ` [PATCH 6.18.y 5/8] io_uring/timeout: don't rearm a multishot timeout on cancel Jens Axboe
2026-11-21 21:31 ` [PATCH 6.18.y 6/8] io_uring/rsrc: keep the old file when a file update fails Jens Axboe
2026-11-21 21:31 ` [PATCH 6.18.y 7/8] io_uring/kbuf: initialize arg->out_len from arg->max_len in io_buffers_select() Jens Axboe
2026-11-21 21:31 ` Jens Axboe [this message]
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20261121213137.1789687-9-axboe@kernel.dk \
--to=axboe@kernel.dk \
--cc=stable@vger.kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox