Re: [PATCH] sched_ext: Fix deadlock with PSI trigger creation

[Andrea Righi <arighi@nvidia.com>]

Hi Matt,

On Fri, Jul 10, 2026 at 11:04:41AM +0100, Matt Fleming wrote:
> From: Matt Fleming <mfleming@cloudflare.com>
> 
> scx_root_enable_workfn() currently takes scx_fork_rwsem for writing
> before acquiring cgroup_mutex. Since commit a5b98009f16d ("sched/psi:
> fix race between file release and pressure write"), pressure_write()
> holds cgroup_mutex across psi_trigger_create(), which may call
> kthread_create() for the psimon kthread. kthreadd's fork then enters
> scx_pre_fork() and waits for the read side of scx_fork_rwsem.
> 
> This results in a deadlock. The enable worker holds scx_fork_rwsem and
> waits for cgroup_mutex, while the PSI writer holds cgroup_mutex and
> waits for psimon creation to complete. Any concurrent fork blocks on
> scx_pre_fork() behind the enable worker.
> 
> The hung-task detector captured all three sides of the deadlock:
> 
>   scx_enable_help:
>     __mutex_lock
>     scx_enable_workfn
>     kthread_worker_fn
> 
>   systemd:
>     wait_for_completion_killable
>     __kthread_create_on_node
>     kthread_create_on_node
>     psi_trigger_create
>     pressure_write
>     kernfs_fop_write_iter
> 
>   python3:
>     percpu_rwsem_wait
>     __percpu_down_read
>     scx_pre_fork
>     sched_fork
>     copy_process
>     kernel_clone
> 
> It also identified systemd as the likely owner of the mutex on which
> scx_enable_help was blocked.
> 
> We reproduced this on a 128-CPU AMD EPYC 7713 by enabling scx_lavd
> concurrently with writes to cgroup PSI trigger files. Unrelated tasks
> piled up in scx_pre_fork() and process creation on the box stopped.
> 
> Fix the inversion by acquiring cgroup_mutex before scx_fork_rwsem in
> scx_root_enable_workfn() and releasing them in reverse order, while
> preserving the existing exclusion around cgroup and task initialisation.
> 
> Fixes: a5b98009f16d ("sched/psi: fix race between file release and pressure write")
> Cc: stable@vger.kernel.org
> Signed-off-by: Matt Fleming <mfleming@cloudflare.com>

This seems to introduce the following (running the sched_ext kselftests):

[   28.963575] ======================================================
[   28.963670] WARNING: possible circular locking dependency detected
[   28.963752] 7.1.0-virtme #1 Not tainted
[   28.963804] ------------------------------------------------------
[   28.963887] sched_ext_helpe/2619 is trying to acquire lock:
[   28.963954] ffffffff83685240 (scx_cgroup_ops_rwsem){+.+.}-{0:0}, at: scx_root_disable+0x45d/0x840
[   28.964071]
[   28.964071] but task is already holding lock:
[   28.964151] ffffffff83682910 (scx_fork_rwsem){++++}-{0:0}, at: scx_root_disable+0x160/0x840
[   28.964260]
[   28.964260] which lock already depends on the new lock.
[   28.964260]
[   28.964355]
[   28.964355] the existing dependency chain (in reverse order) is:
[   28.964455]
[   28.964455] -> #2 (scx_fork_rwsem){++++}-{0:0}:
[   28.964539]        percpu_down_write+0x49/0x150
[   28.964610]        scx_root_enable_workfn+0x5d0/0xd30
[   28.964679]        kthread_worker_fn+0x121/0x360
[   28.964749]        kthread+0x10c/0x140
[   28.964802]        ret_from_fork+0x189/0x330
[   28.964872]        ret_from_fork_asm+0x1a/0x30
[   28.964941]
[   28.964941] -> #1 (cgroup_mutex){+.+.}-{4:4}:
[   28.965025]        __mutex_lock+0xbe/0xd80
[   28.965070]        scx_root_enable_workfn+0x5c4/0xd30
[   28.965138]        kthread_worker_fn+0x121/0x360
[   28.965210]        kthread+0x10c/0x140
[   28.965263]        ret_from_fork+0x189/0x330
[   28.965331]        ret_from_fork_asm+0x1a/0x30
[   28.965402]
[   28.965402] -> #0 (scx_cgroup_ops_rwsem){+.+.}-{0:0}:
[   28.965485]        __lock_acquire+0x14c5/0x2a40
[   28.965554]        lock_acquire+0xd3/0x280
[   28.965608]        percpu_down_write+0x49/0x150
[   28.965677]        scx_root_disable+0x45d/0x840
[   28.965745]        kthread_worker_fn+0x121/0x360
[   28.965815]        kthread+0x10c/0x140
[   28.965867]        ret_from_fork+0x189/0x330
[   28.965935]        ret_from_fork_asm+0x1a/0x30
[   28.966002]
[   28.966002] other info that might help us debug this:
[   28.966002]
[   28.966097] Chain exists of:
[   28.966097]   scx_cgroup_ops_rwsem --> cgroup_mutex --> scx_fork_rwsem
[   28.966097]
[   28.966233]  Possible unsafe locking scenario:
[   28.966233]
[   28.966314]        CPU0                    CPU1
[   28.966379]        ----                    ----
[   28.966444]   lock(scx_fork_rwsem);
[   28.966496]                                lock(cgroup_mutex);
[   28.966579]                                lock(scx_fork_rwsem);
[   28.966661]   lock(scx_cgroup_ops_rwsem);
[   28.966713]
[   28.966713]  *** DEADLOCK ***
[   28.966713]
[   28.966794] 2 locks held by sched_ext_helpe/2619:
[   28.966861]  #0: ffffffff83682818 (scx_enable_mutex){+.+.}-{4:4}, at: scx_root_disable+0xba/0x840
[   28.966974]  #1: ffffffff83682910 (scx_fork_rwsem){++++}-{0:0}, at: scx_root_disable+0x160/0x840
[   28.967090]
[   28.967090] stack backtrace:
[   28.967158] CPU: 8 UID: 0 PID: 2619 Comm: sched_ext_helpe Not tainted 7.1.0-virtme #1 PREEMPT(full)
[   28.967164] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011
[   28.967166] Call Trace:
[   28.967169]  <TASK>
[   28.967172]  dump_stack_lvl+0x6d/0xa0
[   28.967175]  print_circular_bug+0x2e1/0x300
[   28.967179]  check_noncircular+0x144/0x170
[   28.967182]  __lock_acquire+0x14c5/0x2a40
[   28.967202]  ? scx_root_disable+0x45d/0x840
[   28.967204]  lock_acquire+0xd3/0x280
[   28.967207]  ? scx_root_disable+0x45d/0x840
[   28.967209]  percpu_down_write+0x49/0x150
[   28.967212]  ? scx_root_disable+0x45d/0x840
[   28.967213]  scx_root_disable+0x45d/0x840
[   28.967219]  ? kthread_worker_fn+0x51/0x360
[   28.967221]  kthread_worker_fn+0x121/0x360
[   28.967223]  ? __pfx_scx_disable_workfn+0x10/0x10
[   28.967224]  ? __pfx_kthread_worker_fn+0x10/0x10
[   28.967227]  kthread+0x10c/0x140
[   28.967228]  ? __pfx_kthread+0x10/0x10
[   28.967229]  ret_from_fork+0x189/0x330
[   28.967231]  ? __pfx_kthread+0x10/0x10
[   28.967232]  ret_from_fork_asm+0x1a/0x30
[   28.967235]  </TASK>

Thanks,
-Andrea

> ---
>  kernel/sched/ext/ext.c | 8 +++++---
>  1 file changed, 5 insertions(+), 3 deletions(-)
> 
> diff --git a/kernel/sched/ext/ext.c b/kernel/sched/ext/ext.c
> index 691d53fe0f64..ba89eafe7964 100644
> --- a/kernel/sched/ext/ext.c
> +++ b/kernel/sched/ext/ext.c
> @@ -7193,7 +7193,10 @@ static void scx_root_enable_workfn(struct kthread_work *work)
>  	/*
>  	 * Lock out forks, cgroup on/offlining and moves before opening the
>  	 * floodgate so that they don't wander into the operations prematurely.
> +	 * cgroup_mutex must nest outside scx_fork_rwsem because cgroup file
> +	 * operations may create kthreads while holding cgroup_mutex.
>  	 */
> +	scx_cgroup_lock();
>  	percpu_down_write(&scx_fork_rwsem);
>  
>  	WARN_ON_ONCE(scx_init_task_enabled);
> @@ -7216,7 +7219,6 @@ static void scx_root_enable_workfn(struct kthread_work *work)
>  	 * while tasks are being initialized so that scx_cgroup_can_attach()
>  	 * never sees uninitialized tasks.
>  	 */
> -	scx_cgroup_lock();
>  	set_cgroup_sched(sch_cgroup(sch), sch);
>  	ret = scx_cgroup_init(sch);
>  	if (ret)
> @@ -7283,8 +7285,8 @@ static void scx_root_enable_workfn(struct kthread_work *work)
>  		put_task_struct(p);
>  	}
>  	scx_task_iter_stop(&sti);
> -	scx_cgroup_unlock();
>  	percpu_up_write(&scx_fork_rwsem);
> +	scx_cgroup_unlock();
>  
>  	/*
>  	 * All tasks are READY. It's safe to turn on scx_enabled() and switch
> @@ -7369,8 +7371,8 @@ static void scx_root_enable_workfn(struct kthread_work *work)
>  	return;
>  
>  err_disable_unlock_all:
> -	scx_cgroup_unlock();
>  	percpu_up_write(&scx_fork_rwsem);
> +	scx_cgroup_unlock();
>  	/* we'll soon enter disable path, keep bypass on */
>  err_disable:
>  	mutex_unlock(&scx_enable_mutex);
> -- 
> 2.43.0
>