* Re: [PATCH] sched_ext: Fix deadlock with PSI trigger creation
2026-07-10 10:04 [PATCH] sched_ext: Fix deadlock with PSI trigger creation Matt Fleming
@ 2026-07-10 17:46 ` Andrea Righi
2026-07-10 18:24 ` Andrea Righi
2026-07-10 23:49 ` [PATCH cgroup/for-7.2-fixes] cgroup: Create the psimon kthread outside of cgroup_mutex Tejun Heo
1 sibling, 1 reply; 6+ messages in thread
From: Andrea Righi @ 2026-07-10 17:46 UTC (permalink / raw)
To: Matt Fleming
Cc: Tejun Heo, David Vernet, Changwoo Min, Johannes Weiner,
Suren Baghdasaryan, Peter Zijlstra, Edward Adam Davis,
Chen Ridong, sched-ext, cgroups, linux-kernel, stable,
kernel-team, Matt Fleming
Hi Matt,
On Fri, Jul 10, 2026 at 11:04:41AM +0100, Matt Fleming wrote:
> From: Matt Fleming <mfleming@cloudflare.com>
>
> scx_root_enable_workfn() currently takes scx_fork_rwsem for writing
> before acquiring cgroup_mutex. Since commit a5b98009f16d ("sched/psi:
> fix race between file release and pressure write"), pressure_write()
> holds cgroup_mutex across psi_trigger_create(), which may call
> kthread_create() for the psimon kthread. kthreadd's fork then enters
> scx_pre_fork() and waits for the read side of scx_fork_rwsem.
>
> This results in a deadlock. The enable worker holds scx_fork_rwsem and
> waits for cgroup_mutex, while the PSI writer holds cgroup_mutex and
> waits for psimon creation to complete. Any concurrent fork blocks on
> scx_pre_fork() behind the enable worker.
>
> The hung-task detector captured all three sides of the deadlock:
>
> scx_enable_help:
> __mutex_lock
> scx_enable_workfn
> kthread_worker_fn
>
> systemd:
> wait_for_completion_killable
> __kthread_create_on_node
> kthread_create_on_node
> psi_trigger_create
> pressure_write
> kernfs_fop_write_iter
>
> python3:
> percpu_rwsem_wait
> __percpu_down_read
> scx_pre_fork
> sched_fork
> copy_process
> kernel_clone
>
> It also identified systemd as the likely owner of the mutex on which
> scx_enable_help was blocked.
>
> We reproduced this on a 128-CPU AMD EPYC 7713 by enabling scx_lavd
> concurrently with writes to cgroup PSI trigger files. Unrelated tasks
> piled up in scx_pre_fork() and process creation on the box stopped.
>
> Fix the inversion by acquiring cgroup_mutex before scx_fork_rwsem in
> scx_root_enable_workfn() and releasing them in reverse order, while
> preserving the existing exclusion around cgroup and task initialisation.
>
> Fixes: a5b98009f16d ("sched/psi: fix race between file release and pressure write")
> Cc: stable@vger.kernel.org
> Signed-off-by: Matt Fleming <mfleming@cloudflare.com>
This seems to introduce the following (running the sched_ext kselftests):
[ 28.963575] ======================================================
[ 28.963670] WARNING: possible circular locking dependency detected
[ 28.963752] 7.1.0-virtme #1 Not tainted
[ 28.963804] ------------------------------------------------------
[ 28.963887] sched_ext_helpe/2619 is trying to acquire lock:
[ 28.963954] ffffffff83685240 (scx_cgroup_ops_rwsem){+.+.}-{0:0}, at: scx_root_disable+0x45d/0x840
[ 28.964071]
[ 28.964071] but task is already holding lock:
[ 28.964151] ffffffff83682910 (scx_fork_rwsem){++++}-{0:0}, at: scx_root_disable+0x160/0x840
[ 28.964260]
[ 28.964260] which lock already depends on the new lock.
[ 28.964260]
[ 28.964355]
[ 28.964355] the existing dependency chain (in reverse order) is:
[ 28.964455]
[ 28.964455] -> #2 (scx_fork_rwsem){++++}-{0:0}:
[ 28.964539] percpu_down_write+0x49/0x150
[ 28.964610] scx_root_enable_workfn+0x5d0/0xd30
[ 28.964679] kthread_worker_fn+0x121/0x360
[ 28.964749] kthread+0x10c/0x140
[ 28.964802] ret_from_fork+0x189/0x330
[ 28.964872] ret_from_fork_asm+0x1a/0x30
[ 28.964941]
[ 28.964941] -> #1 (cgroup_mutex){+.+.}-{4:4}:
[ 28.965025] __mutex_lock+0xbe/0xd80
[ 28.965070] scx_root_enable_workfn+0x5c4/0xd30
[ 28.965138] kthread_worker_fn+0x121/0x360
[ 28.965210] kthread+0x10c/0x140
[ 28.965263] ret_from_fork+0x189/0x330
[ 28.965331] ret_from_fork_asm+0x1a/0x30
[ 28.965402]
[ 28.965402] -> #0 (scx_cgroup_ops_rwsem){+.+.}-{0:0}:
[ 28.965485] __lock_acquire+0x14c5/0x2a40
[ 28.965554] lock_acquire+0xd3/0x280
[ 28.965608] percpu_down_write+0x49/0x150
[ 28.965677] scx_root_disable+0x45d/0x840
[ 28.965745] kthread_worker_fn+0x121/0x360
[ 28.965815] kthread+0x10c/0x140
[ 28.965867] ret_from_fork+0x189/0x330
[ 28.965935] ret_from_fork_asm+0x1a/0x30
[ 28.966002]
[ 28.966002] other info that might help us debug this:
[ 28.966002]
[ 28.966097] Chain exists of:
[ 28.966097] scx_cgroup_ops_rwsem --> cgroup_mutex --> scx_fork_rwsem
[ 28.966097]
[ 28.966233] Possible unsafe locking scenario:
[ 28.966233]
[ 28.966314] CPU0 CPU1
[ 28.966379] ---- ----
[ 28.966444] lock(scx_fork_rwsem);
[ 28.966496] lock(cgroup_mutex);
[ 28.966579] lock(scx_fork_rwsem);
[ 28.966661] lock(scx_cgroup_ops_rwsem);
[ 28.966713]
[ 28.966713] *** DEADLOCK ***
[ 28.966713]
[ 28.966794] 2 locks held by sched_ext_helpe/2619:
[ 28.966861] #0: ffffffff83682818 (scx_enable_mutex){+.+.}-{4:4}, at: scx_root_disable+0xba/0x840
[ 28.966974] #1: ffffffff83682910 (scx_fork_rwsem){++++}-{0:0}, at: scx_root_disable+0x160/0x840
[ 28.967090]
[ 28.967090] stack backtrace:
[ 28.967158] CPU: 8 UID: 0 PID: 2619 Comm: sched_ext_helpe Not tainted 7.1.0-virtme #1 PREEMPT(full)
[ 28.967164] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011
[ 28.967166] Call Trace:
[ 28.967169] <TASK>
[ 28.967172] dump_stack_lvl+0x6d/0xa0
[ 28.967175] print_circular_bug+0x2e1/0x300
[ 28.967179] check_noncircular+0x144/0x170
[ 28.967182] __lock_acquire+0x14c5/0x2a40
[ 28.967202] ? scx_root_disable+0x45d/0x840
[ 28.967204] lock_acquire+0xd3/0x280
[ 28.967207] ? scx_root_disable+0x45d/0x840
[ 28.967209] percpu_down_write+0x49/0x150
[ 28.967212] ? scx_root_disable+0x45d/0x840
[ 28.967213] scx_root_disable+0x45d/0x840
[ 28.967219] ? kthread_worker_fn+0x51/0x360
[ 28.967221] kthread_worker_fn+0x121/0x360
[ 28.967223] ? __pfx_scx_disable_workfn+0x10/0x10
[ 28.967224] ? __pfx_kthread_worker_fn+0x10/0x10
[ 28.967227] kthread+0x10c/0x140
[ 28.967228] ? __pfx_kthread+0x10/0x10
[ 28.967229] ret_from_fork+0x189/0x330
[ 28.967231] ? __pfx_kthread+0x10/0x10
[ 28.967232] ret_from_fork_asm+0x1a/0x30
[ 28.967235] </TASK>
Thanks,
-Andrea
> ---
> kernel/sched/ext/ext.c | 8 +++++---
> 1 file changed, 5 insertions(+), 3 deletions(-)
>
> diff --git a/kernel/sched/ext/ext.c b/kernel/sched/ext/ext.c
> index 691d53fe0f64..ba89eafe7964 100644
> --- a/kernel/sched/ext/ext.c
> +++ b/kernel/sched/ext/ext.c
> @@ -7193,7 +7193,10 @@ static void scx_root_enable_workfn(struct kthread_work *work)
> /*
> * Lock out forks, cgroup on/offlining and moves before opening the
> * floodgate so that they don't wander into the operations prematurely.
> + * cgroup_mutex must nest outside scx_fork_rwsem because cgroup file
> + * operations may create kthreads while holding cgroup_mutex.
> */
> + scx_cgroup_lock();
> percpu_down_write(&scx_fork_rwsem);
>
> WARN_ON_ONCE(scx_init_task_enabled);
> @@ -7216,7 +7219,6 @@ static void scx_root_enable_workfn(struct kthread_work *work)
> * while tasks are being initialized so that scx_cgroup_can_attach()
> * never sees uninitialized tasks.
> */
> - scx_cgroup_lock();
> set_cgroup_sched(sch_cgroup(sch), sch);
> ret = scx_cgroup_init(sch);
> if (ret)
> @@ -7283,8 +7285,8 @@ static void scx_root_enable_workfn(struct kthread_work *work)
> put_task_struct(p);
> }
> scx_task_iter_stop(&sti);
> - scx_cgroup_unlock();
> percpu_up_write(&scx_fork_rwsem);
> + scx_cgroup_unlock();
>
> /*
> * All tasks are READY. It's safe to turn on scx_enabled() and switch
> @@ -7369,8 +7371,8 @@ static void scx_root_enable_workfn(struct kthread_work *work)
> return;
>
> err_disable_unlock_all:
> - scx_cgroup_unlock();
> percpu_up_write(&scx_fork_rwsem);
> + scx_cgroup_unlock();
> /* we'll soon enter disable path, keep bypass on */
> err_disable:
> mutex_unlock(&scx_enable_mutex);
> --
> 2.43.0
>
^ permalink raw reply [flat|nested] 6+ messages in thread* [PATCH cgroup/for-7.2-fixes] cgroup: Create the psimon kthread outside of cgroup_mutex
2026-07-10 10:04 [PATCH] sched_ext: Fix deadlock with PSI trigger creation Matt Fleming
2026-07-10 17:46 ` Andrea Righi
@ 2026-07-10 23:49 ` Tejun Heo
2026-07-11 5:54 ` Matt Fleming
2026-07-12 17:49 ` Tejun Heo
1 sibling, 2 replies; 6+ messages in thread
From: Tejun Heo @ 2026-07-10 23:49 UTC (permalink / raw)
To: Matt Fleming
Cc: David Vernet, Andrea Righi, Changwoo Min, Johannes Weiner,
Suren Baghdasaryan, Peter Zijlstra, Edward Adam Davis,
Chen Ridong, Matt Fleming, sched-ext, cgroups, linux-kernel,
stable, kernel-team
a5b98009f16d ("sched/psi: fix race between file release and pressure write")
made pressure_write() hold cgroup_mutex across psi_trigger_create(), which
forks the psimon kthread for the first rtpoll trigger. As kthread creation
depends on the whole fork path, the commit inadvertently created a lot of
unwanted locking dependencies from cgroup_mutex.
sched_ext got hit by one: its enable path blocks forks and then grabs
cgroup_mutex, so a pressure write racing a scheduler enable deadlocks, with
every other fork piling up behind.
Fix it by splitting trigger creation so that the worker is forked with
cgroup_mutex dropped and the kernfs active reference left broken. The latter
matters because rmdir and cgroup.pressure writes drain active references
under cgroup_mutex. Publishing the trigger last keeps error reporting
synchronous and preserves the of->priv lifetime rules.
The trigger registered in the first stage pins the group's rtpoll machinery
across the unlocked window, leaving only creation races to resolve. The
catch-up poll on installation covers scheduling attempts dropped while there
was no worker.
Fixes: a5b98009f16d ("sched/psi: fix race between file release and pressure write")
Cc: stable@vger.kernel.org
Cc: Johannes Weiner <hannes@cmpxchg.org>
Cc: Edward Adam Davis <eadavis@qq.com>
Cc: Chen Ridong <chenridong@huaweicloud.com>
Reported-by: Matt Fleming <mfleming@cloudflare.com>
Closes: https://lore.kernel.org/all/20260710100441.2653477-1-matt@readmodwrite.com/
Signed-off-by: Tejun Heo <tj@kernel.org>
---
Matt, your reordering trades one deadlock for another: CLONE_INTO_CGROUP
forks grab cgroup_mutex inside the scx_fork_rwsem read section, so an
enable racing such a clone deadlocks the other way around. The fork has to
move out of the locked sections instead. Can you verify this fixes the
deadlock in your setup?
include/linux/psi.h | 4 ++
kernel/cgroup/cgroup.c | 23 +++++++++++++++-
kernel/sched/psi.c | 69 +++++++++++++++++++++++++++++++++++++------------
3 files changed, 78 insertions(+), 18 deletions(-)
--- a/include/linux/psi.h
+++ b/include/linux/psi.h
@@ -25,7 +25,9 @@ void psi_memstall_leave(unsigned long *f
int psi_show(struct seq_file *s, struct psi_group *group, enum psi_res res);
struct psi_trigger *psi_trigger_create(struct psi_group *group, char *buf,
enum psi_res res, struct file *file,
- struct kernfs_open_file *of);
+ struct kernfs_open_file *of,
+ bool *need_rtpoll_worker);
+int psi_trigger_create_rtpoll_worker(struct psi_group *group);
void psi_trigger_destroy(struct psi_trigger *t);
__poll_t psi_trigger_poll(void **trigger_ptr, struct file *file,
--- a/kernel/cgroup/cgroup.c
+++ b/kernel/cgroup/cgroup.c
@@ -3996,6 +3996,7 @@ static ssize_t pressure_write(struct ker
struct psi_trigger *new;
struct cgroup *cgrp;
struct psi_group *psi;
+ bool need_rtpoll_worker;
ssize_t ret = 0;
cgrp = cgroup_kn_lock_live(of->kn, false);
@@ -4015,12 +4016,32 @@ static ssize_t pressure_write(struct ker
}
psi = cgroup_psi(cgrp);
- new = psi_trigger_create(psi, buf, res, of->file, of);
+ new = psi_trigger_create(psi, buf, res, of->file, of,
+ &need_rtpoll_worker);
if (IS_ERR(new)) {
ret = PTR_ERR(new);
goto out_unlock;
}
+ /*
+ * The worker fork must run with neither cgroup_mutex nor the file's
+ * kernfs active reference held. The latter is broken since
+ * cgroup_kn_lock_live(). @of->priv may be released while unlocked, so
+ * recheck before publishing @new.
+ */
+ if (need_rtpoll_worker) {
+ cgroup_unlock();
+ ret = psi_trigger_create_rtpoll_worker(psi);
+ cgroup_lock();
+
+ if (!ret && !of->priv)
+ ret = -ENODEV;
+ if (ret) {
+ psi_trigger_destroy(new);
+ goto out_unlock;
+ }
+ }
+
smp_store_release(&ctx->psi.trigger, new);
out_unlock:
--- a/kernel/sched/psi.c
+++ b/kernel/sched/psi.c
@@ -1292,9 +1292,44 @@ int psi_show(struct seq_file *m, struct
return 0;
}
+/*
+ * Create @group's rtpoll worker after psi_trigger_create() reported the need
+ * for one. kthread creation depends on the whole fork path and we don't want
+ * all of that nested inside cgroup_mutex, so the caller must drop it and any
+ * other lock that forks can wait behind. If two callers race, the loser stops
+ * its never-woken kthread.
+ */
+int psi_trigger_create_rtpoll_worker(struct psi_group *group)
+{
+ struct task_struct *task;
+
+ task = kthread_create(psi_rtpoll_worker, group, "psimon");
+ if (IS_ERR(task))
+ return PTR_ERR(task);
+
+ scoped_guard(mutex, &group->rtpoll_trigger_lock) {
+ if (!rcu_access_pointer(group->rtpoll_task)) {
+ atomic_set(&group->rtpoll_wakeup, 0);
+ wake_up_process(task);
+ rcu_assign_pointer(group->rtpoll_task, task);
+
+ /*
+ * Poll once to catch up on scheduling attempts dropped
+ * while there was no rtpoll worker.
+ */
+ psi_schedule_rtpoll_work(group, 1, true);
+ return 0;
+ }
+ }
+
+ kthread_stop(task);
+ return 0;
+}
+
struct psi_trigger *psi_trigger_create(struct psi_group *group, char *buf,
enum psi_res res, struct file *file,
- struct kernfs_open_file *of)
+ struct kernfs_open_file *of,
+ bool *need_rtpoll_worker)
{
struct psi_trigger *t;
enum psi_states state;
@@ -1302,6 +1337,8 @@ struct psi_trigger *psi_trigger_create(s
bool privileged;
u32 window_us;
+ *need_rtpoll_worker = false;
+
if (static_branch_likely(&psi_disabled))
return ERR_PTR(-EOPNOTSUPP);
@@ -1362,26 +1399,14 @@ struct psi_trigger *psi_trigger_create(s
if (privileged) {
mutex_lock(&group->rtpoll_trigger_lock);
- if (!rcu_access_pointer(group->rtpoll_task)) {
- struct task_struct *task;
-
- task = kthread_create(psi_rtpoll_worker, group, "psimon");
- if (IS_ERR(task)) {
- kfree(t);
- mutex_unlock(&group->rtpoll_trigger_lock);
- return ERR_CAST(task);
- }
- atomic_set(&group->rtpoll_wakeup, 0);
- wake_up_process(task);
- rcu_assign_pointer(group->rtpoll_task, task);
- }
-
list_add(&t->node, &group->rtpoll_triggers);
group->rtpoll_min_period = min(group->rtpoll_min_period,
div_u64(t->win.size, UPDATES_PER_WINDOW));
group->rtpoll_nr_triggers[t->state]++;
group->rtpoll_states |= (1 << t->state);
+ *need_rtpoll_worker = !rcu_access_pointer(group->rtpoll_task);
+
mutex_unlock(&group->rtpoll_trigger_lock);
} else {
mutex_lock(&group->avgs_lock);
@@ -1541,6 +1566,8 @@ static ssize_t psi_write(struct file *fi
size_t buf_size;
struct seq_file *seq;
struct psi_trigger *new;
+ bool need_rtpoll_worker;
+ int ret;
if (static_branch_likely(&psi_disabled))
return -EOPNOTSUPP;
@@ -1565,12 +1592,22 @@ static ssize_t psi_write(struct file *fi
return -EBUSY;
}
- new = psi_trigger_create(&psi_system, buf, res, file, NULL);
+ new = psi_trigger_create(&psi_system, buf, res, file, NULL,
+ &need_rtpoll_worker);
if (IS_ERR(new)) {
mutex_unlock(&seq->lock);
return PTR_ERR(new);
}
+ if (need_rtpoll_worker) {
+ ret = psi_trigger_create_rtpoll_worker(&psi_system);
+ if (ret) {
+ psi_trigger_destroy(new);
+ mutex_unlock(&seq->lock);
+ return ret;
+ }
+ }
+
smp_store_release(&seq->private, new);
mutex_unlock(&seq->lock);
^ permalink raw reply [flat|nested] 6+ messages in thread