All of lore.kernel.org
 help / color / mirror / Atom feed
From: Tejun Heo <tj@kernel.org>
To: void@manifault.com
Cc: kernel-team@meta.com, linux-kernel@vger.kernel.org,
	sched-ext@meta.com, aboorvad@linux.ibm.com
Subject: [PATCHSET sched_ext/for-6.12-fixes] sched_ext: Fix locking enable/disable path bugs includling locking order one
Date: Mon, 23 Sep 2024 08:59:27 -1000	[thread overview]
Message-ID: <20240923190020.1446325-1-tj@kernel.org> (raw)

Aboorva Devarajan reported an issue where sched_ext init code can
occasionally deadlock when scheduler loading races CPU hotplug. The deadlock
scenario is as follows:

       scx_ops_enable()                               hotplug

                                          percpu_down_write(&cpu_hotplug_lock)
   percpu_down_write(&scx_fork_rwsem)
   block on cpu_hotplug_lock
                                          kthread_create() waits for kthreadd
					  kthreadd blocks on scx_fork_rwsem

Note that this doesn't trigger lockdep because the hotplug side dependency
bounces through kthreadd.

This is primarily caused by SCX enable/disable paths grabbing big locks
together. This patchset updates the enable/disable paths to decouple the
locks. In the process, it also fixes several subtle bugs in the enable path.

This patchset contains the following patches:

 0001-sched_ext-Relocate-check_hotplug_seq-call-in-scx_ops.patch
 0002-sched_ext-Remove-SCX_OPS_PREPPING.patch
 0003-sched_ext-Initialize-in-bypass-mode.patch
 0004-sched_ext-Fix-SCX_TASK_INIT-SCX_TASK_READY-transitio.patch
 0005-sched_ext-Enable-scx_ops_init_task-separately.patch
 0006-sched_ext-Add-scx_cgroup_enabled-to-gate-cgroup-oper.patch
 0007-sched_ext-Decouple-locks-in-scx_ops_disable_workfn.patch
 0008-sched_ext-Decouple-locks-in-scx_ops_enable.patch

 0001-0002 are prep patches.

 0003 removes a race window in the enable path that can cause stalls and
 prepares for further locking updates.

 0004-0005 remove race windows in the enable path that can cause invalid task
 state transitions.

 0006 fixes a bug in cgroup enable path which can skip invocation of
 ops.cgroup_exit() and prepares for further locking updates.

 0007-0008 decouple the big locks and fix the deadlock.

This patchset can also be found in the following git branch:

 git://git.kernel.org/pub/scm/linux/kernel/git/tj/sched_ext.git scx-enable-locking-fix

diffstat follows. Thanks.

 kernel/sched/ext.c |  199 ++++++++++++++++++++++++++++++++------------------------------------
 1 file changed, 94 insertions(+), 105 deletions(-)

--
tejun

             reply	other threads:[~2024-09-23 19:00 UTC|newest]

Thread overview: 10+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2024-09-23 18:59 Tejun Heo [this message]
2024-09-23 18:59 ` [PATCH 1/8] sched_ext: Relocate check_hotplug_seq() call in scx_ops_enable() Tejun Heo
2024-09-23 18:59 ` [PATCH 2/8] sched_ext: Remove SCX_OPS_PREPPING Tejun Heo
2024-09-23 18:59 ` [PATCH 3/8] sched_ext: Initialize in bypass mode Tejun Heo
2024-09-23 18:59 ` [PATCH 4/8] sched_ext: Fix SCX_TASK_INIT -> SCX_TASK_READY transitions in scx_ops_enable() Tejun Heo
2024-09-23 18:59 ` [PATCH 5/8] sched_ext: Enable scx_ops_init_task() separately Tejun Heo
2024-09-23 18:59 ` [PATCH 6/8] sched_ext: Add scx_cgroup_enabled to gate cgroup operations and fix scx_tg_online() Tejun Heo
2024-09-23 18:59 ` [PATCH 7/8] sched_ext: Decouple locks in scx_ops_disable_workfn() Tejun Heo
2024-09-23 18:59 ` [PATCH 8/8] sched_ext: Decouple locks in scx_ops_enable() Tejun Heo
2024-09-27 20:03 ` [PATCHSET sched_ext/for-6.12-fixes] sched_ext: Fix locking enable/disable path bugs includling locking order one Tejun Heo

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20240923190020.1446325-1-tj@kernel.org \
    --to=tj@kernel.org \
    --cc=aboorvad@linux.ibm.com \
    --cc=kernel-team@meta.com \
    --cc=linux-kernel@vger.kernel.org \
    --cc=sched-ext@meta.com \
    --cc=void@manifault.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.