* [PATCH v8 00/33] sched/deadline: bandwidth accounting fixes and selftests
@ 2026-11-19 20:15 Mathieu Desnoyers
2026-11-19 20:15 ` [PATCH v8 01/33] sched/deadline: Keep departing bandwidth across a root domain rebuild Mathieu Desnoyers
` (32 more replies)
0 siblings, 33 replies; 35+ messages in thread
From: Mathieu Desnoyers @ 2026-11-19 20:15 UTC (permalink / raw)
To: juri.lelli, peterz, mingo, vincent.guittot
Cc: Mathieu Desnoyers, tglx, will, boqun, dietmar.eggemann, rostedt,
bsegall, mgorman, vschneid, kprateek.nayak, luca.abeni,
mathieu.poirier, longman, ridong.chen, tj, hannes, mkoutny, shuah,
bigeasy, clrkwllms, parri.andrea, m.szyprowski, kayracizmeci,
weihengqiuu, linux-kernel, cgroups, linux-kselftest,
linux-rt-devel
Hi Ingo, Juri,
This is v8 of this series. As Juri proposed, its first 14 patches now
form their own series [1]. That series also takes the 0-lag overflow
fix posted alone [3]. It adds deadline_switch_test, and the cases of
deadline_bw_test that pass with its fixes. This series applies on top
of it, and holds the rest of v7.
This series fixes 26 bugs of SCHED_DEADLINE. Most of them are in its
bandwidth accounting. As Juri asked, the patches come in groups, one
per theme. Each group ends with the cases of deadline_bw_test that test
its fixes. The fixes carry Cc: stable. The test patches do not.
One of the bugs takes the kernel down (patch 13). Two let an owner in
the rtmutex chain walk keep a donor that can exit. The owner would then
read freed memory (patches 20 and 21). These two also close the known
limit that patch 11 of the first series describes. One leaves a task
that never runs again (patch 24). Most of the others leave total_bw of
a root domain, or this_bw of a runqueue, wrong. The root domain then
admits deadline tasks beyond its capacity, or refuses every one.
Juri, your fix 1/4 that Ingo dropped from sched/urgent is patch 3. The
other three are in the first series.
The groups
==========
The rebuild of the root domains' accounting (1-6):
1 Keep the bandwidth of departing tasks across a root domain rebuild.
2 Rebuild the root domains' accounting from the task list.
3 Record the root domain that holds a task's bandwidth.
4 Leave out of a rebuild a dying task that released its bandwidth.
5 Add deadline_rebuild_test.
6 Add the 7 cases of deadline_bw_test for this group.
cpuset attach (7-12):
7 Move a task's bandwidth only where an attach made room for it.
8 Wait for an attach before rebuilding the sched domains.
9 Make a policy or affinity change of a deadline task wait for a
cpuset attach in progress.
10 Give back the bandwidth an attach did not move.
11 Reserve an attach's bandwidth in the root domain each task moves to.
12 Add the 22 cases of deadline_bw_test for this group.
The root domain of a task, with partitions and isolcpus=domain (13-18):
13 Make a rebuild count a task in the root domain it runs in.
14 Move a task's bandwidth to the root domain it runs in.
15 Pick a CPU for a task in the root domain that counts its
bandwidth, when that root domain has one.
16 Refuse to take offline the last CPU a task has in the root domain
that counts its bandwidth. Also refuse an offline that moves a
deadline task of a child partition to another root domain.
17 Refuse a cpuset change that leaves a deadline task no CPU in the
root domain that counts its bandwidth.
18 Add the 24 cases of deadline_bw_test for this group.
extra_bw (19):
19 Count only the active CPUs when a change of total_bw is spread over
extra_bw, at an offline migration and at a rebuild.
The rtmutex chain walk (20-22):
20 Continue the chain walk when the deadline donor of a waiter
changes, so that no owner keeps a donor that can exit.
21 Continue the chain walk when the top pi waiter of an owner moves to
another lock. The next owner then gets the new priority or donor.
22 Add the 2 cases of deadline_bw_test for this group.
Deadline boosts (23-28):
23 Start a new period at a deboost in the class, if the boost
refilled the task's runtime from its donor.
24 Replenish a throttled task that a waiter boosts while it sleeps, so
that it runs again.
25 Count in this_bw the bandwidth a boosted task is given.
26 Give a deadline to a task that enters the class while it waits on
an rt_mutex.
27 Start a new period at a deboost out of the class, if the boost
refilled the task's runtime from its donor. Its bandwidth is then
released at once.
28 Add the 7 cases of deadline_bw_test for this group.
Parameter changes (29-33):
29 Cancel the replenishment timer with the parameters.
30 Cap the runtime left of a constrained task at dl_runtime when its
parameters change.
31 Check the runtime left of a task that comes back to the class
runnable, as a wakeup does.
32 Check the runtime left of a runnable task whose parameters change.
33 Add the 6 cases of deadline_bw_test for this group.
Patches 3 to 17 go together. 3 records the root domain that counts a
task's bandwidth. The fixes from 4 to 14 fix the rebuild, attach and
migration paths that the record exposes. 15 to 17 keep a task in the
root domain that counts it, as long as that root domain has a CPU for
it. Patch 19 changes code that these fixes rework, so it comes after
them. Patch 25 completes "sched/deadline: Fix this_bw when a task is
changed during a remote wakeup", of the first series.
Peter, patches 20 and 21 change the rtmutex chain walk. In 20, a waiter
records its deadline donor with its sort keys. 21 removes a test that
stopped the walk too early. The step before can move the top pi waiter
of an owner to another lock. The owner then changed, but the walk
stopped at it. The test dates from the rtmutex core, and the early stop
hits RT priorities too.
Three patches also touch the core scheduler. Patch 9 touches
__sched_setscheduler() and sched_setaffinity(). A change of a deadline
task waits there for a cpuset attach in progress. Patch 11 touches
__set_cpus_allowed_ptr_locked(). A cpuset attach can reserve a deadline
task's bandwidth in one root domain. If the CPU picked for the task is
in the other one, the patch steers the task to the reserved CPU. Patch
15 touches select_fallback_rq() and __set_cpus_allowed_ptr_locked() for
deadline tasks. When select_fallback_rq() widens the mask of a deadline
task, the wider mask can leave its root domain. Patch 15 then gives the
task the CPUs of its root domain instead. At a CPU offline, the cpuset
update that comes next sets the real mask.
Waiman, patches 16 and 17 add refusals. They apply when a change would
leave a deadline task no CPU in the root domain that counts it. That
needs a cpuset with CPUs of two root domains: with isolcpus="domain", or
on cgroup v1 with sched_load_balance off. A CPU offline or a write to
cpuset.cpus then fails with EBUSY. cpuset makes a partition root
invalid, with a new reason. Nothing is refused with admission control
off (sched_rt_runtime_us set to -1). Patch 16 now works out the CPUs a
cpuset gets from the top down, as cgroup v2 does. It also refuses an
offline that makes a child partition invalid, when a deadline task in it
would change root domain.
Known limits
============
The fixes leave these cases open. Each patch named below also states
its limit in its message.
- Patches 20 and 21: when the donor of a chain changes, the owners
further down keep the old donor until the walk reaches them. The donor
pointer holds no reference. If the old donor exits first, they would
read freed memory. Its exit starts no walk when it is not the top
waiter of its lock. This was found by reading.
- Patch 1: a task that left the class keeps its bandwidth until its
0-lag time. A rebuild still drops it earlier if the CPU of the task
is inactive, or if the task dies during the rebuild.
- Patch 14: a task can run in a root domain that does not count its
bandwidth. The bandwidth then moves there without a check for room,
as dl_task_offline_migration() does upstream.
- Patch 15: a CPU picked while the sched domains are rebuilt has no
preference for a root domain, as upstream. No bandwidth moves
meanwhile, nor before the walk of the task list that follows reaches
the task. But a wakeup can pick its CPU in that window, and be queued
after the walk counted the task. Its bandwidth can then move to the
other root domain with no check for room. Push and pull between the
two root domains during the rebuild are part of the same limit. This
was found by reading, not reproduced.
- Patches 15 to 17: a task with no CPU left in the root domain that
counts it still moves, and its bandwidth with it. With admission
control on, patches 16 and 17 refuse the CPU offline and the cpuset
changes that I found to lead there. Nothing is refused during a
suspend, or with admission control off.
- A cpuset that holds isolated and housekeeping CPUs still admits a
deadline task. Juri pointed at the XXX in dl_bw_capacity() for this
in the v5 thread. It is left for later.
Testing
=======
On top of the first series, the 123 cases pass wherever they run.
They ran with lockdep, and logged no kernel warning. The 20 QEMU guests
are of seven kinds:
- 5 guests of 8 vCPUs, one with proxy execution off, and 1 of 4 vCPUs.
- 3 guests with PREEMPT_RT, of 8, 8 and 4 vCPUs.
- 3 guests without CONFIG_CPUSETS, of 4, 4 and 5 vCPUs.
- 4 guests of 5 vCPUs booted with isolcpus=domain,1,2 or
isolcpus=domain,0,1, 2 of them with PREEMPT_RT.
- 1 guest of 7 vCPUs booted with isolcpus=domain,0,1.
- 1 guest of 8 vCPUs booted with isolcpus=domain,6,7.
- 2 guests with CONFIG_RT_GROUP_SCHED, of 8 and 4 vCPUs.
The 25 isolcpus cases run in the guests of the fourth and fifth kinds.
In the fourth, the cases that need 4 CPUs that are not isolated skip.
rebuild_waking_keeps also skips with isolcpus=domain,1,2: it needs the
first CPU isolated. Some of them also run in the guest of the sixth
kind. move_cancelled runs only in the guests of the last kind.
At each test patch, the cases added so far pass in 12 of these guests,
with no kernel warning:
- 3 of 8 vCPUs, one with proxy execution off, and 1 with PREEMPT_RT.
- 2 without CONFIG_CPUSETS.
- 5 booted with isolcpus=domain, one of them with PREEMPT_RT.
- 1 with CONFIG_RT_GROUP_SCHED.
No guest combines isolcpus=domain with a kernel without CONFIG_CPUSETS.
Such a kernel hits a BUG at boot on v7.4-rc2 with isolcpus=domain,0,1
[2].
On v7.4-rc2, the cases that the test patches add fail as they list. The
three cases of deadline_rebuild_test fail there too.
Each case added for a fix fails without that fix. Each fix names the
cases that test it, and the test patches list the rest. These cases are
exceptions:
- deboost_blocked_keeps_order passes on v7.4-rc2, and fails on v5.
- return_asleep_new_params passes everywhere. It covers the wakeup
path that patches 31 and 32 copy.
- deboost_keeps_own_runtime passes with and without the flag of patch
23. It checks that a task keeps the runtime of its own reservation.
- pi_chain_donor_leaves passes on v7.4-rc2 and with the first series.
It fails on v6.
- deboost_new_params_reboost and rebuild_record_taken pass on
v7.4-rc2. They test bugs of v7 of this series, and fail there.
- hotplug_child_partition_kept tests a refusal that is not needed.
v7 refuses it too, by reading. On v7.4-rc2, it fails in 2 of 8 runs.
Each of the 33 patches builds alone, on top of the first series. At
each test patch, the series also builds without CONFIG_CPUSETS, with
PREEMPT_RT and with CONFIG_RT_GROUP_SCHED. The selftests build there
with no warning. kernel/sched/ also builds without CONFIG_RT_MUTEXES. The
cpuset selftest test_cpuset_prs.sh passes, with and without PREEMPT_RT.
Not tested: cgroup v1 cpusets, threaded cgroups (every move is a write
to cgroup.procs), the sched_ext server, and CPUs of unequal capacity.
How this was developed
======================
An LLM coding assistant wrote these patches, their messages and the
tests, under my direction and review. I know the concepts behind
SCHED_DEADLINE, but not its implementation well. So I ask those who
know this code for a careful reading of the fixes. A reproducer that
stops failing shows that the symptom goes away. It does not show that
the fix is the one you would have written.
The bugs came from:
- flaky cgroup selftests of a feature I am developing;
- the review of Juri's four fixes, and his review of v5;
- building deadline_bw_test, and reading the code;
- the Sashiko AI reviews of v1 to v7, and of the first series. Each
finding was checked against the code first.
- a second LLM read of the fixes proposed for the Sashiko findings. It
found the gaps fixed in patches 15 and 26.
Changes since v7
================
- The first 14 patches of v7 are in the first series [1]: v7 1-8, 10,
13, 14, 16, 20 and 37.
- The patches come in groups, one per theme, as Juri asked. Each group
ends with the cases of deadline_bw_test that test it. The test of v7
is split along the groups. deadline_rebuild_test comes with the
rebuild group.
- The messages credit the Sashiko AI review without the number of the
posting, as Juri suggested.
- New patch 21, from the Sashiko review of v7. The rtmutex chain walk
stopped at an owner whose top pi waiter had moved to another lock.
The next owner kept the old priority, or a deadline donor that can
exit. The test dates from the rtmutex core. pi_chain_top_moves and
pi_chain_top_moves_rt test it.
- Patch 20 cites pi_chain_top_moves. In v7, it cited
pi_chain_donor_leaves. On top of the first series, that case passes
without patch 20. pi_chain_top_moves needs both patches 20 and 21.
- New patch 19: "sched/deadline: Count only active CPUs when spreading
extra_bw", posted alone before [3].
- New patch 27, from the Sashiko review of the first series. A task
deboosted out of the class after a refill from its donor got a wrong
0-lag time. It mixed the donor's runtime and deadline with its own
parameters. Its bandwidth was then released too early or too late.
deboost_out_after_donor_refill tests it.
- Fixes to v7 patches, from the same review:
- 9: the restore of the old mask reads the cpuset mask again after
the wait.
- 13: with no active CPU in its mask, a task is counted in its
cpuset. The CPU recorded is kept only if the cpuset holds it. The
record is read once. It reads the CPUs isolated at boot.
HK_TYPE_DOMAIN is updated only after the rebuild.
rebuild_record_taken tests it.
- 14: once the walk of the task list after a rebuild has counted a
task, its bandwidth can move again. v7 blocked moves until the end
of the walk. A task that the walk had counted could then run in the
other root domain. It stayed counted in the old one until its next
wakeup.
- 14: the proxy execution check added in v7 is removed. It also
skipped ordinary migrations of a task blocked on a mutex. It stays
in the proxy execution RFC.
- 15: select_fallback_rq() keeps a deadline task in its root domain
when it widens the mask. On v7, a deadline task with a period of
2ms that spun on the CPU going offline left its root domain in
every test run. hotplug_fallback_running tests it. The offline
migration of a throttled task moves no bandwidth during a rebuild,
nor before the walk has counted the task. A sleeping task whose new
mask keeps its CPU in the root domain wakes up there.
- 16: the CPUs a nested cpuset gets after an offline are computed from
the top down. hotplug_nested_refused tests it. A partition root left
with no CPU of its own takes those of a child partition, as cpuset
does. The offline is refused if that moves a deadline task to
another root domain. hotplug_child_partition_kept and
hotplug_partition_invalid_refused test it. These were found by
reading, and reproduced before the fix.
- 26: a blocked task starts a new period only when its donor changes.
v7 started one at each boost, also with the same donor. Each step
of a chain walk then changed the deadline. With patch 21, a walk in
a ww_mutex cycle on PREEMPT_RT would go on until max_lock_depth.
This was found by reading.
- 32: the new parameters of a boosted task are checked at its
deboost, before the test that boosts it again.
deboost_new_params_reboost tests it.
Not changed:
- sched_setattr() still fails with EPERM for a task moved asleep to
another partition, or asleep on a CPU that went offline. The task is
still on a CPU of the old root domain, and the affinity check of
__sched_setscheduler() reads that one. The check is upstream code,
and it refuses but never admits wrongly.
- An attach whose mask keeps a CPU of the task's root domain reserves
nothing. The task then stays in that root domain.
- A write to cpuset.cpus that makes a partition root invalid is not
refused. A deadline task can then run in a root domain that did not
admit it, as upstream. A write of "member" does the same and cannot
fail. The next rebuild counts the task where it runs.
Posted separately, as v7 announced: the report of the boot BUG with
isolcpus=domain and no CPUSETS [2].
Based on the first series [1]. That series is based on
tip sched/urgent, ec6f6731538e (v7.4-rc2).
[1] https://lore.kernel.org/all/cover.1795115075.git.mathieu.desnoyers@efficios.com/
[2] https://lore.kernel.org/lkml/20261117144817.265109-1-mathieu.desnoyers@efficios.com/
[3] https://lore.kernel.org/all/cover.1794926626.git.mathieu.desnoyers@efficios.com/
v7: https://lore.kernel.org/all/cover.1794922146.git.mathieu.desnoyers@efficios.com/
v6: https://lore.kernel.org/all/20261116190549.230119-1-mathieu.desnoyers@efficios.com/
v5: https://lore.kernel.org/all/20261108185406.1549770-1-mathieu.desnoyers@efficios.com/
v4: https://lore.kernel.org/all/20261107014451.1487825-1-mathieu.desnoyers@efficios.com/
Mathieu Desnoyers (33):
sched/deadline: Keep departing bandwidth across a root domain rebuild
sched/deadline: Rebuild the root domains' accounting from the task
list
sched/deadline: Record the root domain holding a task's bandwidth
sched/deadline: Leave a dying task out of a rebuild of the accounting
selftests/sched: Test SCHED_DEADLINE bandwidth across a root domain
rebuild
selftests/sched: Test the root domain rebuild in deadline_bw_test
sched/deadline: Move a task's bandwidth only where room was made
cgroup/cpuset: Wait for an attach before rebuilding the sched domains
sched: Make deadline policy and affinity changes wait for a cpuset
attach
cgroup/cpuset: Give back the bandwidth an attach did not move
cgroup/cpuset: Reserve the bandwidth in the root domain a task moves
to
selftests/sched: Test cpuset moves in deadline_bw_test
sched/deadline: Make a rebuild count a task in the root domain it runs
in
sched/deadline: Move a task's bandwidth to the root domain it runs in
sched/deadline: Keep a task in the root domain that counts its
bandwidth
sched/deadline: Keep the last CPU a task has in its root domain
cgroup/cpuset: Refuse to take a deadline task out of its root domain
selftests/sched: Test the root domain of a task in deadline_bw_test
sched/deadline: Count only active CPUs when spreading extra_bw
rtmutex: Continue the chain walk when a waiter's deadline donor
changes
rtmutex: Continue the walk when the owner's top waiter changes lock
selftests/sched: Test the rtmutex chain walk in deadline_bw_test
sched/deadline: Reset the donor's runtime at a deboost in the class
sched/deadline: Replenish a throttled task boosted while asleep
sched/deadline: Count the bandwidth a boosted task is given
sched/deadline: Start a period for a task that enters the class
blocked
sched/deadline: Start a period at a deboost out of the class
selftests/sched: Test deadline boosts in deadline_bw_test
sched/deadline: Cancel the replenishment timer with the parameters
sched/deadline: Cap the runtime left of a constrained task at
dl_runtime
sched/deadline: Check the runtime of a task that comes back runnable
sched/deadline: Check the runtime of a task whose parameters change
selftests/sched: Test parameter changes in deadline_bw_test
include/linux/cpuset.h | 28 +-
include/linux/sched.h | 38 +
include/linux/sched/deadline.h | 39 +-
kernel/cgroup/cgroup.c | 3 -
kernel/cgroup/cpuset-internal.h | 15 +-
kernel/cgroup/cpuset.c | 751 +-
kernel/locking/rtmutex.c | 46 +-
kernel/locking/rtmutex_api.c | 2 +-
kernel/locking/rtmutex_common.h | 3 +
kernel/sched/core.c | 134 +-
kernel/sched/deadline.c | 1337 +++-
kernel/sched/sched.h | 20 +
kernel/sched/syscalls.c | 87 +-
kernel/sched/topology.c | 11 +
.../selftests/sched/deadline/.gitignore | 1 +
.../testing/selftests/sched/deadline/Makefile | 2 +-
.../sched/deadline/deadline_bw_test.c | 6191 ++++++++++++++++-
.../sched/deadline/deadline_rebuild_test.c | 497 ++
18 files changed, 8815 insertions(+), 390 deletions(-)
create mode 100644 tools/testing/selftests/sched/deadline/deadline_rebuild_test.c
base-commit: ec6f6731538e0d45872eb279fdeb477bed152d6f
prerequisite-patch-id: 5e2478ffdfb7583751bf44bdf87fb930bc23551b
prerequisite-patch-id: d5e5ef986ff2d9bc39a87b56e96a2acbd3158687
prerequisite-patch-id: 99e47f80396008ee2b2576157d37842481ce52d2
prerequisite-patch-id: a10824f98c11a4c902dad55cab3ec03b3dbe2c8a
prerequisite-patch-id: 8959e5a408641b4fa81bc79737cd17d5f1b03389
prerequisite-patch-id: 16acf1d18f01d20310d6db767dfbec2f072e0372
prerequisite-patch-id: e1946bb015a7b14ed4e2cd0421e5e30b0c7afdda
prerequisite-patch-id: 4daf48a89779fbd8c8bc5fc1d7a7d6df9bca2d9d
prerequisite-patch-id: 980d8045f9fe8c3757bb3d82c61d68e321cf74bb
prerequisite-patch-id: 4e79615a88d73e3018b5633b6e5abfc1e5d2c06d
prerequisite-patch-id: fb69a0665dc38ca9022cc2bd75234a90e2035b7f
prerequisite-patch-id: 7a115ab83ad96ffaf8e833e0eaeb75f859af2e79
prerequisite-patch-id: 8574b50d81a09ec6091e42d65338adcab2986d71
prerequisite-patch-id: 1aa25df96d60d30f549c71fdfba5589cb239928f
prerequisite-patch-id: d38384e0b7d043a3556ee02fb09316a6227038da
prerequisite-patch-id: 83ded04a76ecf973e2b8c6e04ff83c2b7278915a
prerequisite-patch-id: b8f5d9f9146e887050bbd8ad30463fd81cad9411
--
2.47.3
^ permalink raw reply [flat|nested] 35+ messages in thread
* [PATCH v8 01/33] sched/deadline: Keep departing bandwidth across a root domain rebuild
2026-11-19 20:15 [PATCH v8 00/33] sched/deadline: bandwidth accounting fixes and selftests Mathieu Desnoyers
@ 2026-11-19 20:15 ` Mathieu Desnoyers
2026-11-19 20:15 ` [PATCH v8 02/33] sched/deadline: Rebuild the root domains' accounting from the task list Mathieu Desnoyers
` (31 subsequent siblings)
32 siblings, 0 replies; 35+ messages in thread
From: Mathieu Desnoyers @ 2026-11-19 20:15 UTC (permalink / raw)
To: juri.lelli, peterz, mingo, vincent.guittot
Cc: Mathieu Desnoyers, tglx, will, boqun, dietmar.eggemann, rostedt,
bsegall, mgorman, vschneid, kprateek.nayak, luca.abeni,
mathieu.poirier, longman, ridong.chen, tj, hannes, mkoutny, shuah,
bigeasy, clrkwllms, parri.andrea, m.szyprowski, kayracizmeci,
weihengqiuu, linux-kernel, cgroups, linux-kselftest,
linux-rt-devel, stable
A task that leaves SCHED_DEADLINE, or dies, keeps its bandwidth in
total_bw until its 0-lag time. This keeps admission control on the safe
side. Then inactive_task_timer() removes it from the root domain of the
task's CPU. Two things break this:
- A rebuild of the accounting clears every root domain, then adds back
the deadline tasks of the cpusets. It misses a task that left
SCHED_DEADLINE, or a dead one. Their bandwidth is dropped, and the
timer removes it again.
- A task that left SCHED_DEADLINE can also move to another root
domain. The timer then takes total_bw from that one, which never
had it.
Either way, a root domain admits deadline tasks beyond its capacity
until the next rebuild.
Fix this with dl_departed_list. A departing task goes on it with the
CPU it is on. A rebuild puts its bandwidth back in that CPU's root
domain. The timer removes it from there, and so does
sched_dl_overflow() for a task that returns to SCHED_DEADLINE.
Priority inheritance can boost a departed task back into the class,
after it moved to another root domain. The boost cancels the timer and
takes the task off the list. Keep the CPU recorded then. When the boost
ends, the bandwidth is taken from the root domain that counts it, not
from the one the task runs in. The pi_departed_moved case, added to
deadline_bw_test later in this series, tests this.
A rebuild still drops the bandwidth of a task whose CPU is inactive, or
that dies during the rebuild. Each task records a count of the rebuilds,
dl_bw_gen, with its bandwidth. The timer removes nothing if a rebuild
dropped the bandwidth since.
Make dl_cookie and dl_bw_visited() static: only deadline.c uses them.
deadline_rebuild_test, added later in this series, fails every time on
v7.3 and v7.4-rc2 kernels. A one-CPU partition root admits two tasks of
600ms every second there, where it has room for one. It passes with
this patch.
Fixes: 387e31300b57 ("sched/deadline: Fix the update of the total -deadline utilization")
Fixes: f9a25f776d78 ("cpusets: Rebuild root domain deadline accounting information")
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Cc: stable@vger.kernel.org
---
include/linux/sched.h | 16 +++
include/linux/sched/deadline.h | 5 +-
kernel/cgroup/cpuset.c | 11 +-
kernel/sched/deadline.c | 254 +++++++++++++++++++++++++++++----
4 files changed, 249 insertions(+), 37 deletions(-)
diff --git a/include/linux/sched.h b/include/linux/sched.h
index 88cda9c10cb8..b6d4e365afee 100644
--- a/include/linux/sched.h
+++ b/include/linux/sched.h
@@ -739,6 +739,22 @@ struct sched_dl_entity {
*/
struct hrtimer inactive_timer;
+ /*
+ * A task that leaves SCHED_DEADLINE, or dies, before its "0-lag
+ * time" keeps its bandwidth in its root domain until the inactive
+ * timer removes it. Meanwhile, @dl_departed_node links the task on
+ * the list that a rebuild of the root domains' accounting reads.
+ * @dl_departed_cpu is a CPU of the root domain that counts the
+ * bandwidth. It stays recorded while priority inheritance boosts the
+ * task back into the class. Otherwise @dl_departed_cpu is -1.
+ *
+ * @dl_bw_gen is the value of dl_bw_gen when the bandwidth was last
+ * counted. If a rebuild dropped it since, there is nothing to remove.
+ */
+ struct list_head dl_departed_node;
+ int dl_departed_cpu;
+ unsigned int dl_bw_gen;
+
/*
* Bits for DL-server functionality. Also see the comment near
* dl_server_update().
diff --git a/include/linux/sched/deadline.h b/include/linux/sched/deadline.h
index 273538200a44..bdf39ac1d43d 100644
--- a/include/linux/sched/deadline.h
+++ b/include/linux/sched/deadline.h
@@ -32,7 +32,7 @@ static inline bool dl_time_before(u64 a, u64 b)
struct root_domain;
extern void dl_add_task_root_domain(struct task_struct *p);
extern void dl_clear_root_domain(struct root_domain *rd);
-extern void dl_clear_root_domain_cpu(int cpu);
+extern void dl_clear_root_domains(void);
/*
* Return whether moving DL task @p to @new_mask requires moving DL
* bandwidth accounting between root domains. This helper is specific to
@@ -43,9 +43,6 @@ extern void dl_clear_root_domain_cpu(int cpu);
extern bool dl_task_needs_bw_move(struct task_struct *p,
const struct cpumask *new_mask);
-extern u64 dl_cookie;
-extern bool dl_bw_visited(int cpu, u64 cookie);
-
static inline bool dl_server(struct sched_dl_entity *dl_se)
{
return dl_se->dl_server;
diff --git a/kernel/cgroup/cpuset.c b/kernel/cgroup/cpuset.c
index 3219f927c35e..4159abfbbbb5 100644
--- a/kernel/cgroup/cpuset.c
+++ b/kernel/cgroup/cpuset.c
@@ -985,22 +985,15 @@ void dl_rebuild_rd_accounting(void)
{
struct cpuset *cs = NULL;
struct cgroup_subsys_state *pos_css;
- int cpu;
- u64 cookie = ++dl_cookie;
lockdep_assert_cpuset_lock_held();
lockdep_assert_cpus_held();
lockdep_assert_held(&sched_domains_mutex);
+ dl_clear_root_domains();
+
rcu_read_lock();
- for_each_possible_cpu(cpu) {
- if (dl_bw_visited(cpu, cookie))
- continue;
-
- dl_clear_root_domain_cpu(cpu);
- }
-
cpuset_for_each_descendant_pre(cs, pos_css, &top_cpuset) {
if (cpumask_empty(cs->effective_cpus)) {
diff --git a/kernel/sched/deadline.c b/kernel/sched/deadline.c
index 63890866dd0c..30dce8cb1209 100644
--- a/kernel/sched/deadline.c
+++ b/kernel/sched/deadline.c
@@ -163,7 +163,13 @@ static inline unsigned long dl_bw_capacity(int i)
}
}
-bool dl_bw_visited(int cpu, u64 cookie)
+/*
+ * Used for dl_bw check and update, used under sched_rt_handler()::mutex and
+ * sched_domains_mutex.
+ */
+static u64 dl_cookie;
+
+static bool dl_bw_visited(int cpu, u64 cookie)
{
struct root_domain *rd = cpu_rq(cpu)->rd;
@@ -217,6 +223,112 @@ __dl_overflow(struct dl_bw *dl_b, unsigned long cap, u64 old_bw, u64 new_bw)
cap_scale(dl_b->bw, cap) < dl_b->total_bw - old_bw + new_bw;
}
+/*
+ * A task that leaves SCHED_DEADLINE, or dies, before its 0-lag time keeps
+ * its bandwidth in total_bw until that time. Then the inactive timer
+ * removes it. Meanwhile the task may run anywhere, under another policy.
+ * The timer holds a reference to it.
+ *
+ * A rebuild of the root domains' accounting (dl_clear_root_domains() and
+ * dl_add_task_root_domain()) does not find such a task among the deadline
+ * tasks of the cpusets. So dl_departed_list holds these tasks, from their
+ * departure to their timer. The rebuild puts their bandwidth back in the
+ * root domain of the CPU each one was on when it left.
+ *
+ * dl_bw_gen numbers the rebuilds of the accounting: how many times the
+ * root domains were wiped and reconstructed (dl_clear_root_domains()
+ * increments it). A task records the number of its last definite
+ * state, counted or settled. That number dates the state:
+ *
+ * - Current: the bandwidth is counted, or a release settled it since
+ * the rebuild began. A release subtracts; the walk of the cpusets
+ * skips.
+ * - One behind: the clear that just ran wiped it. The rebuild puts
+ * it back.
+ * - Older: an earlier rebuild dropped it. A task on the departed list
+ * stays out, and its release removes nothing. A task that is in the
+ * class is counted again.
+ *
+ * Only equalities are compared, so the wrap of the counter is
+ * harmless.
+ *
+ * A rebuild drops the bandwidth when the CPU recorded is no longer
+ * active. It also drops the bandwidth of a task that dies while the
+ * rebuild runs. The cpuset walk skips that task, and it was not dead yet
+ * when the list was put back.
+ *
+ * dl_departed_lock protects the list and both counts. Take it with
+ * interrupts disabled, under the task's rq lock or pi_lock or in the
+ * rebuild, and before any dl_bw lock.
+ */
+static LIST_HEAD(dl_departed_list);
+static DEFINE_RAW_SPINLOCK(dl_departed_lock);
+static unsigned int dl_bw_gen;
+
+/*
+ * Put @p on dl_departed_list with @cpu, a CPU of the root domain that
+ * keeps its bandwidth. @p leaves SCHED_DEADLINE, or dies, before its
+ * 0-lag time. Do nothing if @p is on the list already. Keep the CPU
+ * recorded if @p has one: priority inheritance boosted @p back into the
+ * class and its boost ends. The caller holds the rq lock of @p.
+ */
+static void dl_task_depart(struct task_struct *p, int cpu)
+{
+ struct sched_dl_entity *dl_se = &p->dl;
+
+ lockdep_assert_irqs_disabled();
+ guard(raw_spinlock)(&dl_departed_lock);
+ if (!list_empty(&dl_se->dl_departed_node))
+ return;
+ if (dl_se->dl_departed_cpu < 0)
+ dl_se->dl_departed_cpu = cpu;
+ list_add(&dl_se->dl_departed_node, &dl_departed_list);
+}
+
+/*
+ * Take @p off dl_departed_list. Priority inheritance boosts @p back into
+ * the deadline class. Keep the CPU recorded: its root domain still counts
+ * the bandwidth, wherever @p runs now. The caller holds the rq lock of
+ * @p.
+ */
+static void dl_task_undepart(struct task_struct *p)
+{
+ struct sched_dl_entity *dl_se = &p->dl;
+
+ lockdep_assert_irqs_disabled();
+ guard(raw_spinlock)(&dl_departed_lock);
+ list_del_init(&dl_se->dl_departed_node);
+}
+
+/*
+ * Remove the bandwidth of @p from the root domain that counts it. That is
+ * the root domain of the CPU recorded when @p departed, or else of @cpu.
+ * Clear the record. Take @p off dl_departed_list. Remove nothing if a
+ * rebuild dropped the bandwidth since it was counted. Then record the
+ * current dl_bw_gen in @p, so that a rebuild in progress does not count it
+ * again. The caller holds the rq lock of @p.
+ */
+static void dl_task_release_bw(struct task_struct *p, int cpu)
+{
+ struct sched_dl_entity *dl_se = &p->dl;
+ struct dl_bw *dl_b;
+
+ lockdep_assert_irqs_disabled();
+ guard(raw_spinlock)(&dl_departed_lock);
+ if (dl_se->dl_departed_cpu >= 0) {
+ cpu = dl_se->dl_departed_cpu;
+ dl_se->dl_departed_cpu = -1;
+ list_del_init(&dl_se->dl_departed_node);
+ }
+ if (dl_se->dl_bw_gen == dl_bw_gen) {
+ dl_b = dl_bw_of(cpu);
+ raw_spin_lock(&dl_b->lock);
+ __dl_sub(dl_b, dl_se->dl_bw, dl_bw_cpus(cpu));
+ raw_spin_unlock(&dl_b->lock);
+ }
+ dl_se->dl_bw_gen = dl_bw_gen;
+}
+
static inline
void __add_running_bw(u64 dl_bw, struct dl_rq *dl_rq)
{
@@ -509,13 +621,9 @@ static void task_non_contending(struct sched_dl_entity *dl_se, bool dl_task)
sub_running_bw(dl_se, dl_rq);
if (!dl_task || READ_ONCE(p->__state) == TASK_DEAD) {
- struct dl_bw *dl_b = dl_bw_of(task_cpu(p));
-
if (dl_task && READ_ONCE(p->__state) == TASK_DEAD)
sub_rq_bw(dl_se, &rq->dl);
- raw_spin_lock(&dl_b->lock);
- __dl_sub(dl_b, dl_se->dl_bw, dl_bw_cpus(task_cpu(p)));
- raw_spin_unlock(&dl_b->lock);
+ dl_task_release_bw(p, task_cpu(p));
__dl_clear_params(dl_se);
}
}
@@ -524,8 +632,18 @@ static void task_non_contending(struct sched_dl_entity *dl_se, bool dl_task)
}
dl_se->dl_non_contending = 1;
- if (!dl_server(dl_se))
- get_task_struct(dl_task_of(dl_se));
+ if (!dl_server(dl_se)) {
+ struct task_struct *p = dl_task_of(dl_se);
+
+ get_task_struct(p);
+ /*
+ * A dying task keeps its bandwidth in the root domain of the
+ * CPU it runs on. Record that CPU. switched_from_dl() records
+ * a task that leaves SCHED_DEADLINE.
+ */
+ if (READ_ONCE(p->__state) == TASK_DEAD)
+ dl_task_depart(p, task_cpu(p));
+ }
hrtimer_start(timer, ns_to_ktime(zerolag_time), HRTIMER_MODE_REL_HARD);
}
@@ -2277,17 +2395,13 @@ static enum hrtimer_restart inactive_task_timer(struct hrtimer *timer)
goto no_task;
if (!dl_task(p) || READ_ONCE(p->__state) == TASK_DEAD) {
- struct dl_bw *dl_b = dl_bw_of(task_cpu(p));
-
if (READ_ONCE(p->__state) == TASK_DEAD && dl_se->dl_non_contending) {
sub_running_bw(&p->dl, dl_rq_of_se(&p->dl));
sub_rq_bw(&p->dl, dl_rq_of_se(&p->dl));
dl_se->dl_non_contending = 0;
}
- raw_spin_lock(&dl_b->lock);
- __dl_sub(dl_b, p->dl.dl_bw, dl_bw_cpus(task_cpu(p)));
- raw_spin_unlock(&dl_b->lock);
+ dl_task_release_bw(p, task_cpu(p));
__dl_clear_params(dl_se);
goto unlock;
@@ -3553,12 +3667,33 @@ void dl_add_task_root_domain(struct task_struct *p)
dl_get_task_effective_cpus(p, msk);
cpu = cpumask_first_and(cpu_active_mask, msk);
BUG_ON(cpu >= nr_cpu_ids);
+
+ raw_spin_lock(&dl_departed_lock);
+ /*
+ * Skip a task whose dl_bw_gen is current. Either
+ * dl_clear_root_domains() put it back as a departed task, and
+ * priority inheritance boosted it into the class since. Or it died
+ * since the rebuild started, and dl_task_release_bw() released its
+ * bandwidth. A task that died since with its 0-lag time ahead is on
+ * the departed list. Count it in the root domain of the CPU
+ * recorded, if that CPU is active.
+ */
+ if (p->dl.dl_bw_gen == dl_bw_gen)
+ goto unlock;
+ if (p->dl.dl_departed_cpu >= 0) {
+ cpu = p->dl.dl_departed_cpu;
+ if (!cpu_active(cpu))
+ goto unlock;
+ }
rq = cpu_rq(cpu);
dl_b = &rq->rd->dl_bw;
raw_spin_lock(&dl_b->lock);
__dl_add(dl_b, p->dl.dl_bw, cpumask_weight(rq->rd->span));
raw_spin_unlock(&dl_b->lock);
+ p->dl.dl_bw_gen = dl_bw_gen;
+unlock:
+ raw_spin_unlock(&dl_departed_lock);
raw_spin_unlock_irqrestore(&p->pi_lock, rf.flags);
}
@@ -3616,9 +3751,51 @@ void dl_clear_root_domain(struct root_domain *rd)
dl_server_add_bw(rd, i);
}
-void dl_clear_root_domain_cpu(int cpu)
+/*
+ * Clear every root domain before a rebuild of their accounting, and count
+ * the rebuild in dl_bw_gen. Put back the bandwidth of their servers. Put
+ * back the bandwidth of the tasks on dl_departed_list, which no cpuset
+ * holds as deadline tasks. Put each in the root domain of the CPU
+ * recorded for it, or drop it if that CPU is not active.
+ * dl_add_task_root_domain() then adds the deadline tasks of the cpusets.
+ */
+void dl_clear_root_domains(void)
{
- dl_clear_root_domain(cpu_rq(cpu)->rd);
+ struct sched_dl_entity *dl_se;
+ u64 cookie = ++dl_cookie;
+ struct dl_bw *dl_b;
+ int cpu;
+
+ guard(raw_spinlock_irqsave)(&dl_departed_lock);
+ dl_bw_gen++;
+ for_each_possible_cpu(cpu) {
+ if (!dl_bw_visited(cpu, cookie))
+ dl_clear_root_domain(cpu_rq(cpu)->rd);
+ }
+ list_for_each_entry(dl_se, &dl_departed_list, dl_departed_node) {
+ cpu = dl_se->dl_departed_cpu;
+ /*
+ * Skip bandwidth that an earlier rebuild dropped. It is not
+ * counted now, and its release removes nothing
+ * (dl_task_release_bw()).
+ */
+ if (dl_se->dl_bw_gen != dl_bw_gen - 1)
+ continue;
+ /*
+ * Drop the bandwidth of a task whose CPU is inactive. A CPU
+ * that goes offline is in the default root domain by then.
+ * The root domain that counted the task cannot be found
+ * from the recorded number. The release then removes
+ * nothing, and no later rebuild restores it.
+ */
+ if (!cpu_active(cpu))
+ continue;
+ dl_b = dl_bw_of(cpu);
+ raw_spin_lock(&dl_b->lock);
+ __dl_add(dl_b, dl_se->dl_bw, dl_bw_cpus(cpu));
+ raw_spin_unlock(&dl_b->lock);
+ dl_se->dl_bw_gen = dl_bw_gen;
+ }
}
static void switched_from_dl(struct rq *rq, struct task_struct *p)
@@ -3674,9 +3851,20 @@ static void switched_from_dl(struct rq *rq, struct task_struct *p)
* We cannot use inactive_task_timer() to invoke sub_running_bw()
* at the 0-lag time, because the task could have been migrated
* while SCHED_OTHER in the meanwhile.
+ *
+ * For the same reason, the timer cannot take the task's bandwidth
+ * from the root domain of the CPU the task is on by then. And a
+ * rebuild of the root domains meanwhile would not find the task
+ * among the deadline tasks. So record a CPU of the root domain that
+ * keeps the bandwidth until the timer removes it (dl_task_depart()).
+ * Test dl_non_contending: it tells that the timer is armed and its
+ * handler has not run. hrtimer_active() stays true for a while after
+ * the handler returns.
*/
- if (p->dl.dl_non_contending)
+ if (p->dl.dl_non_contending) {
p->dl.dl_non_contending = 0;
+ dl_task_depart(p, task_cpu(p));
+ }
/*
* Since this might be the only -deadline task on the rq,
@@ -3696,6 +3884,17 @@ static void switched_from_dl(struct rq *rq, struct task_struct *p)
static void switched_to_dl(struct rq *rq, struct task_struct *p)
{
cancel_inactive_timer(&p->dl);
+ /*
+ * A task on the departed list gets here when priority inheritance
+ * boosts it back into the class before its 0-lag time. (A policy change
+ * took it off the list in sched_dl_overflow().) The inactive timer no
+ * longer removes its bandwidth: the task does when it leaves the class
+ * again. So take it off the list. The task keeps the CPU recorded. It
+ * takes the bandwidth from the root domain of that CPU, even if it
+ * moved to another root domain meanwhile.
+ */
+ if (!list_empty(&p->dl.dl_departed_node))
+ dl_task_undepart(p);
/*
* In case a task is setscheduled to SCHED_DEADLINE we need to keep
@@ -3810,12 +4009,6 @@ DEFINE_SCHED_CLASS(dl) = {
#endif
};
-/*
- * Used for dl_bw check and update, used under sched_rt_handler()::mutex and
- * sched_domains_mutex.
- */
-u64 dl_cookie;
-
int sched_dl_global_validate(void)
{
u64 runtime = global_rt_runtime();
@@ -3936,8 +4129,6 @@ int sched_dl_overflow(struct task_struct *p, int policy,
if (dl_policy(policy) && !task_has_dl_policy(p) &&
!__dl_overflow(dl_b, cap, 0, new_bw)) {
- if (hrtimer_active(&p->dl.inactive_timer))
- __dl_sub(dl_b, p->dl.dl_bw, cpus);
__dl_add(dl_b, new_bw, cpus);
err = 0;
} else if (dl_policy(policy) && task_has_dl_policy(p) &&
@@ -3963,6 +4154,19 @@ int sched_dl_overflow(struct task_struct *p, int policy,
}
raw_spin_unlock(&dl_b->lock);
+ /*
+ * A task that comes back before its 0-lag time still has the
+ * bandwidth it left with. The new bandwidth replaces it, so remove the
+ * old one from the root domain that keeps it (dl_task_release_bw()).
+ * No rebuild can run meanwhile: the caller holds cpuset_mutex.
+ */
+ if (!err && dl_policy(policy) && !task_has_dl_policy(p)) {
+ lockdep_assert_cpuset_lock_held();
+ if (p->dl.dl_departed_cpu >= 0)
+ dl_task_release_bw(p, -1);
+ p->dl.dl_bw_gen = dl_bw_gen;
+ }
+
return err;
}
@@ -4103,6 +4307,8 @@ void init_dl_entity(struct sched_dl_entity *dl_se)
init_dl_task_timer(dl_se);
init_dl_inactive_task_timer(dl_se);
__dl_clear_params(dl_se);
+ INIT_LIST_HEAD(&dl_se->dl_departed_node);
+ dl_se->dl_departed_cpu = -1;
}
bool dl_param_changed(struct task_struct *p, const struct sched_attr *attr)
--
2.47.3
^ permalink raw reply related [flat|nested] 35+ messages in thread
* [PATCH v8 02/33] sched/deadline: Rebuild the root domains' accounting from the task list
2026-11-19 20:15 [PATCH v8 00/33] sched/deadline: bandwidth accounting fixes and selftests Mathieu Desnoyers
2026-11-19 20:15 ` [PATCH v8 01/33] sched/deadline: Keep departing bandwidth across a root domain rebuild Mathieu Desnoyers
@ 2026-11-19 20:15 ` Mathieu Desnoyers
2026-11-19 20:15 ` [PATCH v8 03/33] sched/deadline: Record the root domain holding a task's bandwidth Mathieu Desnoyers
` (30 subsequent siblings)
32 siblings, 0 replies; 35+ messages in thread
From: Mathieu Desnoyers @ 2026-11-19 20:15 UTC (permalink / raw)
To: juri.lelli, peterz, mingo, vincent.guittot
Cc: Mathieu Desnoyers, tglx, will, boqun, dietmar.eggemann, rostedt,
bsegall, mgorman, vschneid, kprateek.nayak, luca.abeni,
mathieu.poirier, longman, ridong.chen, tj, hannes, mkoutny, shuah,
bigeasy, clrkwllms, parri.andrea, m.szyprowski, kayracizmeci,
weihengqiuu, linux-kernel, cgroups, linux-kselftest,
linux-rt-devel, stable
A rebuild of the root domains, as on CPU hotplug, clears their deadline
accounting. dl_rebuild_rd_accounting() then adds back the deadline tasks
of each cpuset. Without CONFIG_CPUSETS, it is a stub. So the first
hotplug drops every deadline task's bandwidth, and their releases wrap
total_bw around. In testing, a root domain admitted 9 tasks of 10ms
every 100ms before a hotplug with deadline sleepers, and none after.
With cpusets, the rebuild skips a cpuset whose nr_deadline_tasks is 0.
This count spares suspend and resume a costly walk of all tasks. An
attach moves it by a tally taken in cpuset_can_attach(). A task that
leaves the class between the tally and the move is taken off the old
cpuset twice. Its count can then reach 0 with a deadline task in it.
Every rebuild would then drop that task's bandwidth.
Fix both by walking the task list, with or without cpusets. The walk
finds every deadline task, so the count and its hooks go. It holds
tasklist_lock for reading, so fork and exit wait meanwhile. It takes
the pi_lock of deadline tasks only. The walk that the count avoided
took the pi_lock of every task. It leaves out an exited task with a
dl_bw of 0. It also leaves out a zombie whose bandwidth an earlier
rebuild dropped, as dl_clear_root_domains() does.
The walk visits every thread, and a suspend or a resume runs one
rebuild for each CPU. So keep one count for the whole system,
dl_nr_bw_tasks: the number of tasks whose dl_bw is not 0.
__setparam_dl() and __dl_clear_params() are the only places that change
the dl_bw of a task, and they update the count. With a count of 0, the
rebuild does not walk the task list, as Juri Lelli suggested.
In testing, a 64-vCPU guest held 10000 threads and no deadline task. A
rebuild took 0.7 to 1.2 ms without the count, and 0.03 ms with it.
Without cpusets, a new dl_accounting_mutex keeps admissions out of a
rebuild, as cpuset_mutex does with them.
Three hotplug cases, added to deadline_bw_test later in this series,
fail without cpusets. With this patch, its five hotplug cases pass, in 4
VMs. Its death_zombie_hotplug case tests the zombie. The Sashiko AI
reviews of this series found the wrong count, the exited tasks and the
zombie.
Fixes: f9a25f776d78 ("cpusets: Rebuild root domain deadline accounting information")
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Cc: stable@vger.kernel.org
---
include/linux/cpuset.h | 9 --
include/linux/sched/deadline.h | 16 ++++
kernel/cgroup/cgroup.c | 3 -
kernel/cgroup/cpuset-internal.h | 6 +-
kernel/cgroup/cpuset.c | 67 +-------------
kernel/sched/deadline.c | 154 ++++++++++++++++++++++++++------
kernel/sched/syscalls.c | 14 +--
kernel/sched/topology.c | 8 ++
8 files changed, 163 insertions(+), 114 deletions(-)
diff --git a/include/linux/cpuset.h b/include/linux/cpuset.h
index 914983387e95..db1cc2fc0324 100644
--- a/include/linux/cpuset.h
+++ b/include/linux/cpuset.h
@@ -72,8 +72,6 @@ extern int cpuset_init(void);
extern void cpuset_init_smp(void);
extern void cpuset_force_rebuild(void);
extern void cpuset_update_active_cpus(void);
-extern void inc_dl_tasks_cs(struct task_struct *task);
-extern void dec_dl_tasks_cs(struct task_struct *task);
extern void cpuset_lock(void);
extern void cpuset_unlock(void);
extern void lockdep_assert_cpuset_lock_held(void);
@@ -129,7 +127,6 @@ static inline int cpuset_do_page_mem_spread(void)
extern bool current_cpuset_is_being_rebound(void);
-extern void dl_rebuild_rd_accounting(void);
extern void rebuild_sched_domains(void);
extern void cpuset_print_current_mems_allowed(void);
@@ -194,8 +191,6 @@ static inline void cpuset_update_active_cpus(void)
partition_sched_domains(1, NULL, NULL);
}
-static inline void inc_dl_tasks_cs(struct task_struct *task) { }
-static inline void dec_dl_tasks_cs(struct task_struct *task) { }
static inline void cpuset_lock(void) { }
static inline void cpuset_unlock(void) { }
static inline void lockdep_assert_cpuset_lock_held(void) { }
@@ -273,10 +268,6 @@ static inline bool current_cpuset_is_being_rebound(void)
return false;
}
-static inline void dl_rebuild_rd_accounting(void)
-{
-}
-
static inline void rebuild_sched_domains(void)
{
guard(cpus_read_lock)();
diff --git a/include/linux/sched/deadline.h b/include/linux/sched/deadline.h
index bdf39ac1d43d..60be226699bb 100644
--- a/include/linux/sched/deadline.h
+++ b/include/linux/sched/deadline.h
@@ -33,6 +33,22 @@ struct root_domain;
extern void dl_add_task_root_domain(struct task_struct *p);
extern void dl_clear_root_domain(struct root_domain *rd);
extern void dl_clear_root_domains(void);
+extern void dl_rebuild_rd_accounting(void);
+/*
+ * Keep a rebuild of the root domains' accounting and an admission of
+ * bandwidth from running at the same time. With cpusets, cpuset_mutex
+ * does it, since every rebuild comes from the cpuset code. Without
+ * cpusets, dl_accounting_lock() takes a mutex of its own.
+ */
+#ifdef CONFIG_CPUSETS
+static inline void dl_accounting_lock(void) { }
+static inline void dl_accounting_unlock(void) { }
+static inline void lockdep_assert_dl_accounting_held(void) { }
+#else
+extern void dl_accounting_lock(void);
+extern void dl_accounting_unlock(void);
+extern void lockdep_assert_dl_accounting_held(void);
+#endif
/*
* Return whether moving DL task @p to @new_mask requires moving DL
* bandwidth accounting between root domains. This helper is specific to
diff --git a/kernel/cgroup/cgroup.c b/kernel/cgroup/cgroup.c
index a284225f8dc9..2cfa770c2dcb 100644
--- a/kernel/cgroup/cgroup.c
+++ b/kernel/cgroup/cgroup.c
@@ -7240,9 +7240,6 @@ static void do_cgroup_task_dead(struct task_struct *tsk)
if (thread_group_leader(tsk) && atomic_read(&tsk->signal->live))
list_add_tail(&tsk->cg_list, &cset->dying_tasks);
- if (dl_task(tsk))
- dec_dl_tasks_cs(tsk);
-
WARN_ON_ONCE(cgroup_task_frozen(tsk));
if (unlikely(!(tsk->flags & PF_KTHREAD) &&
test_bit(CGRP_FREEZE, &task_dfl_cgroup(tsk)->flags)))
diff --git a/kernel/cgroup/cpuset-internal.h b/kernel/cgroup/cpuset-internal.h
index 1ba94ffaff9d..bf847953c78d 100644
--- a/kernel/cgroup/cpuset-internal.h
+++ b/kernel/cgroup/cpuset-internal.h
@@ -166,11 +166,7 @@ struct cpuset {
*/
bool remote_partition_updating;
- /*
- * number of SCHED_DEADLINE tasks attached to this cpuset, so that we
- * know when to rebuild associated root domain bandwidth information.
- */
- atomic_t nr_deadline_tasks;
+ /* Number of SCHED_DEADLINE tasks the attach in progress moves here. */
int nr_migrate_dl_tasks;
/* DL bandwidth that needs destination reservation for this attach. */
u64 sum_migrate_dl_bw;
diff --git a/kernel/cgroup/cpuset.c b/kernel/cgroup/cpuset.c
index 4159abfbbbb5..7628cf77bdfd 100644
--- a/kernel/cgroup/cpuset.c
+++ b/kernel/cgroup/cpuset.c
@@ -225,20 +225,6 @@ struct tmpmasks {
cpumask_var_t new_cpus; /* For update_cpumasks_hier() */
};
-void inc_dl_tasks_cs(struct task_struct *p)
-{
- struct cpuset *cs = task_cs(p);
-
- atomic_inc(&cs->nr_deadline_tasks);
-}
-
-void dec_dl_tasks_cs(struct task_struct *p)
-{
- struct cpuset *cs = task_cs(p);
-
- atomic_dec(&cs->nr_deadline_tasks);
-}
-
static inline bool is_partition_valid(const struct cpuset *cs)
{
return cs->partition_root_state > 0;
@@ -965,54 +951,6 @@ static int generate_sched_domains(cpumask_var_t **domains,
return ndoms;
}
-static void dl_update_tasks_root_domain(struct cpuset *cs)
-{
- struct css_task_iter it;
- struct task_struct *task;
-
- if (atomic_read(&cs->nr_deadline_tasks) == 0)
- return;
-
- css_task_iter_start(&cs->css, 0, &it);
-
- while ((task = css_task_iter_next(&it)))
- dl_add_task_root_domain(task);
-
- css_task_iter_end(&it);
-}
-
-void dl_rebuild_rd_accounting(void)
-{
- struct cpuset *cs = NULL;
- struct cgroup_subsys_state *pos_css;
-
- lockdep_assert_cpuset_lock_held();
- lockdep_assert_cpus_held();
- lockdep_assert_held(&sched_domains_mutex);
-
- dl_clear_root_domains();
-
- rcu_read_lock();
-
- cpuset_for_each_descendant_pre(cs, pos_css, &top_cpuset) {
-
- if (cpumask_empty(cs->effective_cpus)) {
- pos_css = css_rightmost_descendant(pos_css);
- continue;
- }
-
- css_get(&cs->css);
-
- rcu_read_unlock();
-
- dl_update_tasks_root_domain(cs);
-
- rcu_read_lock();
- css_put(&cs->css);
- }
- rcu_read_unlock();
-}
-
/*
* Rebuild scheduler domains.
*
@@ -3181,9 +3119,7 @@ static void clear_attach_data(struct llist_head *head, bool cancel)
llist_for_each_entry_safe(cs, next, lnode, attach_node) {
init_llist_node(&cs->attach_node);
if (cs->nr_migrate_dl_tasks) {
- if (!cancel)
- atomic_add(cs->nr_migrate_dl_tasks, &cs->nr_deadline_tasks);
- else if (cs->dl_bw_cpu >= 0) /* && cancel */
+ if (cancel && cs->dl_bw_cpu >= 0)
dl_bw_free(cs->dl_bw_cpu, cs->sum_migrate_dl_bw);
cs->nr_migrate_dl_tasks = 0;
cs->sum_migrate_dl_bw = 0;
@@ -3272,7 +3208,6 @@ static int cpuset_can_attach(struct cgroup_taskset *tset)
* contribute to sum_migrate_dl_bw.
*/
cs->nr_migrate_dl_tasks++;
- oldcs->nr_migrate_dl_tasks--;
if (dl_task_needs_bw_move(task, cs->effective_cpus))
cs->sum_migrate_dl_bw += task->dl.dl_bw;
}
diff --git a/kernel/sched/deadline.c b/kernel/sched/deadline.c
index 30dce8cb1209..585070a9d317 100644
--- a/kernel/sched/deadline.c
+++ b/kernel/sched/deadline.c
@@ -230,10 +230,10 @@ __dl_overflow(struct dl_bw *dl_b, unsigned long cap, u64 old_bw, u64 new_bw)
* The timer holds a reference to it.
*
* A rebuild of the root domains' accounting (dl_clear_root_domains() and
- * dl_add_task_root_domain()) does not find such a task among the deadline
- * tasks of the cpusets. So dl_departed_list holds these tasks, from their
- * departure to their timer. The rebuild puts their bandwidth back in the
- * root domain of the CPU each one was on when it left.
+ * dl_add_task_root_domain()) does not count such a task as a deadline
+ * task. So dl_departed_list holds these tasks, from their departure to
+ * their timer. The rebuild puts their bandwidth back in the root domain
+ * of the CPU each one was on when it left.
*
* dl_bw_gen numbers the rebuilds of the accounting: how many times the
* root domains were wiped and reconstructed (dl_clear_root_domains()
@@ -241,7 +241,7 @@ __dl_overflow(struct dl_bw *dl_b, unsigned long cap, u64 old_bw, u64 new_bw)
* state, counted or settled. That number dates the state:
*
* - Current: the bandwidth is counted, or a release settled it since
- * the rebuild began. A release subtracts; the walk of the cpusets
+ * the rebuild began. A release subtracts; the walk of the task list
* skips.
* - One behind: the clear that just ran wiped it. The rebuild puts
* it back.
@@ -254,8 +254,8 @@ __dl_overflow(struct dl_bw *dl_b, unsigned long cap, u64 old_bw, u64 new_bw)
*
* A rebuild drops the bandwidth when the CPU recorded is no longer
* active. It also drops the bandwidth of a task that dies while the
- * rebuild runs. The cpuset walk skips that task, and it was not dead yet
- * when the list was put back.
+ * rebuild runs. The walk of the task list does not find that task, and
+ * it was not dead yet when the list was put back.
*
* dl_departed_lock protects the list and both counts. Take it with
* interrupts disabled, under the task's rq lock or pi_lock or in the
@@ -265,6 +265,22 @@ static LIST_HEAD(dl_departed_list);
static DEFINE_RAW_SPINLOCK(dl_departed_lock);
static unsigned int dl_bw_gen;
+/*
+ * The number of tasks with a bandwidth: those whose dl_bw is not 0. A
+ * special entity and a server are not counted. __setparam_dl() and
+ * __dl_clear_params() are the only places that change the dl_bw of a
+ * task, and they update the count. With a count of 0, a rebuild of the
+ * accounting has no task to count, and does not walk the task list.
+ */
+static atomic_t dl_nr_bw_tasks;
+
+/* Return true if @dl_se is a task that holds a bandwidth. */
+static inline bool dl_se_holds_bw(struct sched_dl_entity *dl_se)
+{
+ return dl_se->dl_bw && !dl_se->dl_server &&
+ !(dl_se->flags & SCHED_FLAG_SUGOV);
+}
+
/*
* Put @p on dl_departed_list with @cpu, a CPU of the root domain that
* keeps its bandwidth. @p leaves SCHED_DEADLINE, or dies, before its
@@ -3648,7 +3664,7 @@ static void dl_get_task_effective_cpus(struct task_struct *p, struct cpumask *cp
cpuset_cpus_allowed_locked(p, cpus);
}
-/* The caller should hold cpuset_mutex */
+/* From dl_rebuild_rd_accounting(), under the locks it asserts. */
void dl_add_task_root_domain(struct task_struct *p)
{
struct rq_flags rf;
@@ -3658,7 +3674,13 @@ void dl_add_task_root_domain(struct task_struct *p)
struct cpumask *msk;
raw_spin_lock_irqsave(&p->pi_lock, rf.flags);
- if (!dl_task(p) || dl_entity_is_special(&p->dl)) {
+ /*
+ * Count nothing for a special entity, or for a task with no
+ * bandwidth. The walk of the task list also finds exited tasks. An
+ * exited task whose bandwidth went with its parameters has a dl_bw
+ * of 0.
+ */
+ if (!dl_task(p) || dl_entity_is_special(&p->dl) || !p->dl.dl_bw) {
raw_spin_unlock_irqrestore(&p->pi_lock, rf.flags);
return;
}
@@ -3681,6 +3703,14 @@ void dl_add_task_root_domain(struct task_struct *p)
if (p->dl.dl_bw_gen == dl_bw_gen)
goto unlock;
if (p->dl.dl_departed_cpu >= 0) {
+ /*
+ * The walk also finds a dead task that departed before an
+ * earlier rebuild, if it is not reaped yet. If that rebuild
+ * dropped its bandwidth, dl_clear_root_domains() skipped it,
+ * and its release removes nothing. Skip it here too.
+ */
+ if (p->dl.dl_bw_gen != dl_bw_gen - 1)
+ goto unlock;
cpu = p->dl.dl_departed_cpu;
if (!cpu_active(cpu))
goto unlock;
@@ -3751,13 +3781,80 @@ void dl_clear_root_domain(struct root_domain *rd)
dl_server_add_bw(rd, i);
}
+#ifndef CONFIG_CPUSETS
+/*
+ * With cpusets, cpuset_mutex keeps an admission of bandwidth from running
+ * against a rebuild of the accounting. __sched_setscheduler() takes it for
+ * every change of or into SCHED_DEADLINE. The callers of
+ * partition_sched_domains() hold it from the attachment of the new root
+ * domains to the rebuild. Without cpusets, cpuset_lock() is a stub, and
+ * this mutex takes its place in both.
+ */
+static DEFINE_MUTEX(dl_accounting_mutex);
+
+void dl_accounting_lock(void)
+{
+ mutex_lock(&dl_accounting_mutex);
+}
+
+void dl_accounting_unlock(void)
+{
+ mutex_unlock(&dl_accounting_mutex);
+}
+
+void lockdep_assert_dl_accounting_held(void)
+{
+ lockdep_assert_held(&dl_accounting_mutex);
+}
+#endif /* !CONFIG_CPUSETS */
+
+/*
+ * Rebuild the root domains' accounting. Clear it, then walk all tasks and
+ * count each deadline task again. The caller holds the locks that this
+ * function asserts.
+ */
+void dl_rebuild_rd_accounting(void)
+{
+ struct task_struct *g, *p;
+
+ lockdep_assert_cpus_held();
+ lockdep_assert_held(&sched_domains_mutex);
+ lockdep_assert_cpuset_lock_held();
+ lockdep_assert_dl_accounting_held();
+
+ dl_clear_root_domains();
+
+ /*
+ * Without a task that holds a bandwidth, there is nothing to count.
+ * The walk would still visit every thread of the system. A CPU
+ * that goes offline or comes back runs a rebuild, so a suspend and
+ * a resume run one for each CPU. A task gets a bandwidth under the
+ * locks this rebuild holds, so the count cannot leave 0 meanwhile.
+ */
+ if (!atomic_read(&dl_nr_bw_tasks))
+ return;
+
+ read_lock(&tasklist_lock);
+ for_each_process_thread(g, p) {
+ /*
+ * Test the class without the task's locks, and take them only
+ * for a deadline task. A task enters the class under the locks
+ * this rebuild holds, or by a boost. A boost brings no
+ * bandwidth of its own to count.
+ */
+ if (dl_task(p))
+ dl_add_task_root_domain(p);
+ }
+ read_unlock(&tasklist_lock);
+}
+
/*
* Clear every root domain before a rebuild of their accounting, and count
* the rebuild in dl_bw_gen. Put back the bandwidth of their servers. Put
- * back the bandwidth of the tasks on dl_departed_list, which no cpuset
- * holds as deadline tasks. Put each in the root domain of the CPU
- * recorded for it, or drop it if that CPU is not active.
- * dl_add_task_root_domain() then adds the deadline tasks of the cpusets.
+ * back the bandwidth of the tasks on dl_departed_list, which the walk of
+ * the deadline tasks does not count. Put each in the root domain of the
+ * CPU recorded for it, or drop it if that CPU is not active.
+ * dl_add_task_root_domain() then adds the deadline tasks.
*/
void dl_clear_root_domains(void)
{
@@ -3821,12 +3918,6 @@ static void switched_from_dl(struct rq *rq, struct task_struct *p)
p->dl.runtime = 0;
}
- /*
- * In case a task is setscheduled out from SCHED_DEADLINE we need to
- * keep track of that on its cpuset (for correct bandwidth tracking).
- */
- dec_dl_tasks_cs(p);
-
if (!task_on_rq_queued(p)) {
/*
* Inactive timer is armed. However, p is leaving DEADLINE and
@@ -3896,12 +3987,6 @@ static void switched_to_dl(struct rq *rq, struct task_struct *p)
if (!list_empty(&p->dl.dl_departed_node))
dl_task_undepart(p);
- /*
- * In case a task is setscheduled to SCHED_DEADLINE we need to keep
- * track of that on its cpuset (for correct bandwidth tracking).
- */
- inc_dl_tasks_cs(p);
-
/* If p is not queued we will update its parameters at next wakeup. */
if (!task_on_rq_queued(p)) {
add_rq_bw(&p->dl, &rq->dl);
@@ -4158,10 +4243,12 @@ int sched_dl_overflow(struct task_struct *p, int policy,
* A task that comes back before its 0-lag time still has the
* bandwidth it left with. The new bandwidth replaces it, so remove the
* old one from the root domain that keeps it (dl_task_release_bw()).
- * No rebuild can run meanwhile: the caller holds cpuset_mutex.
+ * No rebuild can run meanwhile: the caller holds cpuset_mutex, or
+ * dl_accounting_mutex without cpusets.
*/
if (!err && dl_policy(policy) && !task_has_dl_policy(p)) {
lockdep_assert_cpuset_lock_held();
+ lockdep_assert_dl_accounting_held();
if (p->dl.dl_departed_cpu >= 0)
dl_task_release_bw(p, -1);
p->dl.dl_bw_gen = dl_bw_gen;
@@ -4181,12 +4268,18 @@ int sched_dl_overflow(struct task_struct *p, int policy,
void __setparam_dl(struct task_struct *p, const struct sched_attr *attr)
{
struct sched_dl_entity *dl_se = &p->dl;
+ bool held;
+ held = dl_se_holds_bw(dl_se);
dl_se->dl_runtime = attr->sched_runtime;
dl_se->dl_deadline = attr->sched_deadline;
dl_se->dl_period = attr->sched_period ?: dl_se->dl_deadline;
dl_se->flags = attr->sched_flags & SCHED_DL_FLAGS;
dl_se->dl_bw = to_ratio(dl_se->dl_period, dl_se->dl_runtime);
+ if (dl_se_holds_bw(dl_se) && !held)
+ atomic_inc(&dl_nr_bw_tasks);
+ else if (!dl_se_holds_bw(dl_se) && held)
+ atomic_dec(&dl_nr_bw_tasks);
dl_se->dl_density = to_ratio(dl_se->dl_deadline, dl_se->dl_runtime);
}
@@ -4276,6 +4369,9 @@ bool __checkparam_dl(const struct sched_attr *attr)
*/
static void __dl_clear_params(struct sched_dl_entity *dl_se)
{
+ if (dl_se_holds_bw(dl_se))
+ WARN_ON_ONCE(atomic_dec_return(&dl_nr_bw_tasks) < 0);
+
dl_se->dl_runtime = 0;
dl_se->dl_deadline = 0;
dl_se->dl_period = 0;
@@ -4306,6 +4402,12 @@ void init_dl_entity(struct sched_dl_entity *dl_se)
RB_CLEAR_NODE(&dl_se->rb_node);
init_dl_task_timer(dl_se);
init_dl_inactive_task_timer(dl_se);
+ /*
+ * A new task has a copy of the entity of its parent. That copy
+ * holds no bandwidth, and dl_nr_bw_tasks does not count it. Clear
+ * dl_bw first, so that __dl_clear_params() leaves the count alone.
+ */
+ dl_se->dl_bw = 0;
__dl_clear_params(dl_se);
INIT_LIST_HEAD(&dl_se->dl_departed_node);
dl_se->dl_departed_cpu = -1;
diff --git a/kernel/sched/syscalls.c b/kernel/sched/syscalls.c
index 5983232476a9..96a5f0a7adbc 100644
--- a/kernel/sched/syscalls.c
+++ b/kernel/sched/syscalls.c
@@ -568,6 +568,7 @@ int __sched_setscheduler(struct task_struct *p,
if (pi && (dl_policy(policy) || dl_policy(p->policy))) {
cpuset_locked = true;
cpuset_lock();
+ dl_accounting_lock();
}
/*
@@ -596,9 +597,7 @@ int __sched_setscheduler(struct task_struct *p,
* Reject SCHED_DEADLINE on exited tasks. A zombie will never be
* scheduled again, so sched_dl_overflow() would charge total_bw
* that nothing will ever release. And reject a change of policy of
- * one in the class: cgroup_task_dead() took it off the count of
- * deadline tasks of its cpuset, which switched_from_dl() would
- * decrement a second time.
+ * one in the class: its bandwidth went at its last schedule.
*/
if ((dl_policy(policy) || dl_task(p)) && p->exit_state) {
retval = -ESRCH;
@@ -669,6 +668,7 @@ int __sched_setscheduler(struct task_struct *p,
policy = oldpolicy = -1;
task_rq_unlock(rq, p, &rf);
if (cpuset_locked) {
+ dl_accounting_unlock();
cpuset_unlock();
cpuset_locked = false;
}
@@ -779,8 +779,10 @@ int __sched_setscheduler(struct task_struct *p,
task_rq_unlock(rq, p, &rf);
if (pi) {
- if (cpuset_locked)
+ if (cpuset_locked) {
+ dl_accounting_unlock();
cpuset_unlock();
+ }
rt_mutex_adjust_pi(p);
}
@@ -792,8 +794,10 @@ int __sched_setscheduler(struct task_struct *p,
unlock:
task_rq_unlock(rq, p, &rf);
- if (cpuset_locked)
+ if (cpuset_locked) {
+ dl_accounting_unlock();
cpuset_unlock();
+ }
return retval;
}
diff --git a/kernel/sched/topology.c b/kernel/sched/topology.c
index 9d3a1b8dd393..35e56a45c238 100644
--- a/kernel/sched/topology.c
+++ b/kernel/sched/topology.c
@@ -3528,7 +3528,15 @@ static void partition_sched_domains_locked(int ndoms_new, cpumask_var_t doms_new
void partition_sched_domains(int ndoms_new, cpumask_var_t doms_new[],
struct sched_domain_attr *dattr_new)
{
+ /*
+ * The root domains attached here count no task's bandwidth until
+ * dl_rebuild_rd_accounting() at the end. No admission of bandwidth may
+ * run meanwhile. With cpusets, the callers hold cpuset_mutex, which
+ * keeps admissions out. Without cpusets, dl_accounting_lock() does it.
+ */
+ dl_accounting_lock();
sched_domains_mutex_lock();
partition_sched_domains_locked(ndoms_new, doms_new, dattr_new);
sched_domains_mutex_unlock();
+ dl_accounting_unlock();
}
--
2.47.3
^ permalink raw reply related [flat|nested] 35+ messages in thread
* [PATCH v8 03/33] sched/deadline: Record the root domain holding a task's bandwidth
2026-11-19 20:15 [PATCH v8 00/33] sched/deadline: bandwidth accounting fixes and selftests Mathieu Desnoyers
2026-11-19 20:15 ` [PATCH v8 01/33] sched/deadline: Keep departing bandwidth across a root domain rebuild Mathieu Desnoyers
2026-11-19 20:15 ` [PATCH v8 02/33] sched/deadline: Rebuild the root domains' accounting from the task list Mathieu Desnoyers
@ 2026-11-19 20:15 ` Mathieu Desnoyers
2026-11-19 20:15 ` [PATCH v8 04/33] sched/deadline: Leave a dying task out of a rebuild of the accounting Mathieu Desnoyers
` (29 subsequent siblings)
32 siblings, 0 replies; 35+ messages in thread
From: Mathieu Desnoyers @ 2026-11-19 20:15 UTC (permalink / raw)
To: juri.lelli, peterz, mingo, vincent.guittot
Cc: Mathieu Desnoyers, tglx, will, boqun, dietmar.eggemann, rostedt,
bsegall, mgorman, vschneid, kprateek.nayak, luca.abeni,
mathieu.poirier, longman, ridong.chen, tj, hannes, mkoutny, shuah,
bigeasy, clrkwllms, parri.andrea, m.szyprowski, kayracizmeci,
weihengqiuu, linux-kernel, cgroups, linux-kselftest,
linux-rt-devel, stable
Admission, moves and releases of a deadline task's bandwidth find the
root domain that counts it from the CPU the task is on. That CPU can be
in another root domain. Two cases show it.
Commit 7f51412a415d ("sched/deadline: Fix bandwidth check/update when
migrating tasks between exclusive cpusets") moves a deadline task's
bandwidth in two halves. cpuset_can_attach() reserves it in the
destination. set_cpus_allowed_dl() frees it in the root domain of the
task's runqueue. Since commit 5dd74441cbf4 ("cgroup/cpuset: Reserve DL
bandwidth only for root-domain moves"), both halves move nothing when
that root domain overlaps the destination. But a sleeping task stays on
the CPU it last ran on. So a sleeping task moved to a partition and back
leaves its bandwidth in the partition:
admin task T
makes P a partition root of CPU 3
CPU 3 is now in root domain B
CPUs 0-2 are in root domain A
sched_setattr(SCHED_DEADLINE)
A: total_bw += bw of T
runs on CPU 0, then sleeps
echo T > P/cgroup.procs
cpuset_can_attach()
dl_task_needs_bw_move(): true,
A does not overlap P
B: total_bw += bw of T
set_cpus_allowed_dl()
dl_task_needs_bw_move(): true
A: total_bw -= bw of T
still asleep on CPU 0
echo T > cgroup.procs of the root cgroup
cpuset_can_attach()
dl_task_needs_bw_move(): false,
A overlaps the root cpuset
set_cpus_allowed_dl()
dl_task_needs_bw_move(): false
kill -9 T
wakes up on a CPU of A
exits
A: total_bw -= bw of T
B keeps the bandwidth of T. A does not count T while T is in A, so A
admits beyond its capacity. After T exits, the total_bw of A is short
by the bandwidth of T, and can underflow.
The second case needs no move:
1. A deadline task T sleeps on a CPU of root domain A.
2. That CPU goes offline, and joins the default root domain.
3. The rebuild counts T in A, the root domain of its cpuset.
4. sched_setattr() takes the sleeping T out of SCHED_DEADLINE.
5. The release takes the bandwidth from the root domain of the CPU T
is on: the default one.
A keeps the bandwidth of T until the next rebuild, and refuses deadline
tasks that would fit.
The first case was reproduced on v7.4-rc2. The second needs the earlier
patches of this series: without "sched/deadline: Release the bandwidth
of a task that leaves the class asleep", nothing releases the bandwidth
of that sleeping task. The hotplug_sleeper_switched case, added to
deadline_bw_test later in this series, tests it. Without this patch and
with the CPU still offline, A admitted 63 tasks of 10% where it had room
for 66, in 3 guests of 3.
Fix this by recording, in dl_bw_cpu, a CPU of the root domain that
counts the bandwidth. The record replaces dl_departed_cpu. It is used
as follows:
- Each site that counts the bandwidth sets the record.
- Each release or move reads it. The release clears it.
- If dl_bw_gen shows that a rebuild dropped the bandwidth, nothing is
taken back. set_cpus_allowed_dl() makes that test under
dl_departed_lock, so that a rebuild cannot change dl_bw_gen
meanwhile.
- A move skips a bandwidth of 0, and does not update its record. So
sched_dl_overflow() uses the CPU the task is on for it. The
move_zero_bw case of deadline_bw_test tests this.
- A rebuild drops departed bandwidth whose CPU is inactive. So a
departing task records the CPU it departs from, if that CPU is
active and in the same root domain.
The affinity check of sched_setaffinity() reads the record too. A task
moved asleep is still on a CPU of the root domain it left. Testing that
root domain let a new mask take the task out of the one that counts it.
The affinity_moved_asleep case tests this.
A move now leaves out a sugov kthread and a task boosted into the class
with no bandwidth of its own. The rebuild leaves them out too: no root
domain counts theirs. A move would take a sugov kthread's bandwidth
from a root domain that never held it.
Weiheng Qiu reported the moved sleeper, and Juri Lelli analysed it. His
fix, linked below, took the first active CPU of the task's mask. A mask
can have none during CPU hotplug. Tip dropped the fix after a sugov
kthread hit its BUG_ON() there. The Sashiko AI reviews of this series
found the other cases.
Fixes: 7f51412a415d ("sched/deadline: Fix bandwidth check/update when migrating tasks between exclusive cpusets")
Reported-by: Weiheng Qiu <weihengqiuu@gmail.com>
Closes: https://lore.kernel.org/all/14508cb2-805f-41ac-9f6b-bfd7c3d7cb1c@gmail.com/
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Cc: stable@vger.kernel.org
Link: https://lore.kernel.org/all/20261103-b4-sched-dl-bw-fixes-v1-1-dd761fe005b4@redhat.com/
---
include/linux/sched.h | 10 +-
kernel/sched/deadline.c | 245 ++++++++++++++++++++++++++++------------
kernel/sched/syscalls.c | 11 +-
3 files changed, 186 insertions(+), 80 deletions(-)
diff --git a/include/linux/sched.h b/include/linux/sched.h
index b6d4e365afee..4098b4e0f67e 100644
--- a/include/linux/sched.h
+++ b/include/linux/sched.h
@@ -744,15 +744,17 @@ struct sched_dl_entity {
* time" keeps its bandwidth in its root domain until the inactive
* timer removes it. Meanwhile, @dl_departed_node links the task on
* the list that a rebuild of the root domains' accounting reads.
- * @dl_departed_cpu is a CPU of the root domain that counts the
- * bandwidth. It stays recorded while priority inheritance boosts the
- * task back into the class. Otherwise @dl_departed_cpu is -1.
+ * Otherwise that node is empty.
+ *
+ * @dl_bw_cpu is a CPU of the root domain that counts the bandwidth,
+ * or -1. Each site that counts the bandwidth sets it. Each site that
+ * removes or moves the bandwidth reads it.
*
* @dl_bw_gen is the value of dl_bw_gen when the bandwidth was last
* counted. If a rebuild dropped it since, there is nothing to remove.
*/
struct list_head dl_departed_node;
- int dl_departed_cpu;
+ int dl_bw_cpu;
unsigned int dl_bw_gen;
/*
diff --git a/kernel/sched/deadline.c b/kernel/sched/deadline.c
index 585070a9d317..5a8f38632e54 100644
--- a/kernel/sched/deadline.c
+++ b/kernel/sched/deadline.c
@@ -233,7 +233,7 @@ __dl_overflow(struct dl_bw *dl_b, unsigned long cap, u64 old_bw, u64 new_bw)
* dl_add_task_root_domain()) does not count such a task as a deadline
* task. So dl_departed_list holds these tasks, from their departure to
* their timer. The rebuild puts their bandwidth back in the root domain
- * of the CPU each one was on when it left.
+ * of the CPU recorded with each task's bandwidth.
*
* dl_bw_gen numbers the rebuilds of the accounting: how many times the
* root domains were wiped and reconstructed (dl_clear_root_domains()
@@ -255,7 +255,10 @@ __dl_overflow(struct dl_bw *dl_b, unsigned long cap, u64 old_bw, u64 new_bw)
* A rebuild drops the bandwidth when the CPU recorded is no longer
* active. It also drops the bandwidth of a task that dies while the
* rebuild runs. The walk of the task list does not find that task, and
- * it was not dead yet when the list was put back.
+ * it was not dead yet when the list was put back. The same happens to a
+ * task that priority inheritance deboosts while the rebuild runs. It was
+ * in the class when the list was put back, and out of it when the walk
+ * reaches it.
*
* dl_departed_lock protects the list and both counts. Take it with
* interrupts disabled, under the task's rq lock or pi_lock or in the
@@ -282,66 +285,75 @@ static inline bool dl_se_holds_bw(struct sched_dl_entity *dl_se)
}
/*
- * Put @p on dl_departed_list with @cpu, a CPU of the root domain that
+ * Put @p on dl_departed_list. The root domain of the CPU recorded with it
* keeps its bandwidth. @p leaves SCHED_DEADLINE, or dies, before its
- * 0-lag time. Do nothing if @p is on the list already. Keep the CPU
- * recorded if @p has one: priority inheritance boosted @p back into the
- * class and its boost ends. The caller holds the rq lock of @p.
+ * 0-lag time. Do nothing if @p is on the list already. The caller holds
+ * the rq lock of @p.
*/
-static void dl_task_depart(struct task_struct *p, int cpu)
+static void dl_task_depart(struct task_struct *p)
{
struct sched_dl_entity *dl_se = &p->dl;
+ int cpu = task_cpu(p);
lockdep_assert_irqs_disabled();
guard(raw_spinlock)(&dl_departed_lock);
- if (!list_empty(&dl_se->dl_departed_node))
+ if (!list_empty(&dl_se->dl_departed_node) || dl_se->dl_bw_cpu < 0)
return;
- if (dl_se->dl_departed_cpu < 0)
- dl_se->dl_departed_cpu = cpu;
+ /*
+ * A rebuild drops the bandwidth if the CPU recorded is not active.
+ * That CPU can be one that @p left long ago. Record the CPU @p
+ * departs from instead, if it is active and the same root domain
+ * counts the bandwidth.
+ */
+ if (dl_se->dl_bw_gen == dl_bw_gen && cpu_active(cpu) &&
+ cpu_rq(cpu)->rd == cpu_rq(dl_se->dl_bw_cpu)->rd)
+ dl_se->dl_bw_cpu = cpu;
list_add(&dl_se->dl_departed_node, &dl_departed_list);
}
/*
- * Take @p off dl_departed_list. Priority inheritance boosts @p back into
- * the deadline class. Keep the CPU recorded: its root domain still counts
- * the bandwidth, wherever @p runs now. The caller holds the rq lock of
- * @p.
+ * Take @p off dl_departed_list. @p comes back to the deadline class, and
+ * its bandwidth is that of a deadline task again. The caller holds the rq
+ * lock of @p.
*/
static void dl_task_undepart(struct task_struct *p)
{
struct sched_dl_entity *dl_se = &p->dl;
lockdep_assert_irqs_disabled();
+ if (list_empty(&dl_se->dl_departed_node))
+ return;
guard(raw_spinlock)(&dl_departed_lock);
list_del_init(&dl_se->dl_departed_node);
}
/*
* Remove the bandwidth of @p from the root domain that counts it. That is
- * the root domain of the CPU recorded when @p departed, or else of @cpu.
- * Clear the record. Take @p off dl_departed_list. Remove nothing if a
- * rebuild dropped the bandwidth since it was counted. Then record the
- * current dl_bw_gen in @p, so that a rebuild in progress does not count it
- * again. The caller holds the rq lock of @p.
+ * the root domain of the CPU recorded with @p. Clear the record.
+ * Take @p off dl_departed_list. Remove nothing if a rebuild dropped the
+ * bandwidth since it was counted. Then record the current dl_bw_gen in
+ * @p, so that a rebuild in progress does not count it again. The caller
+ * holds the rq lock of @p.
*/
-static void dl_task_release_bw(struct task_struct *p, int cpu)
+static void dl_task_release_bw(struct task_struct *p)
{
struct sched_dl_entity *dl_se = &p->dl;
struct dl_bw *dl_b;
+ int cpu;
lockdep_assert_irqs_disabled();
guard(raw_spinlock)(&dl_departed_lock);
- if (dl_se->dl_departed_cpu >= 0) {
- cpu = dl_se->dl_departed_cpu;
- dl_se->dl_departed_cpu = -1;
+ if (!list_empty(&dl_se->dl_departed_node))
list_del_init(&dl_se->dl_departed_node);
- }
- if (dl_se->dl_bw_gen == dl_bw_gen) {
+ cpu = dl_se->dl_bw_cpu;
+ WARN_ON_ONCE(dl_se->dl_bw && cpu < 0);
+ if (cpu >= 0 && dl_se->dl_bw_gen == dl_bw_gen) {
dl_b = dl_bw_of(cpu);
raw_spin_lock(&dl_b->lock);
__dl_sub(dl_b, dl_se->dl_bw, dl_bw_cpus(cpu));
raw_spin_unlock(&dl_b->lock);
}
+ dl_se->dl_bw_cpu = -1;
dl_se->dl_bw_gen = dl_bw_gen;
}
@@ -504,6 +516,7 @@ static void dl_change_utilization(struct task_struct *p, u64 new_bw)
}
static void __dl_clear_params(struct sched_dl_entity *dl_se);
+static unsigned int dl_task_root_domain_cpu(struct task_struct *p);
/*
* The utilization of a task cannot be immediately removed from
@@ -639,7 +652,7 @@ static void task_non_contending(struct sched_dl_entity *dl_se, bool dl_task)
if (!dl_task || READ_ONCE(p->__state) == TASK_DEAD) {
if (dl_task && READ_ONCE(p->__state) == TASK_DEAD)
sub_rq_bw(dl_se, &rq->dl);
- dl_task_release_bw(p, task_cpu(p));
+ dl_task_release_bw(p);
__dl_clear_params(dl_se);
}
}
@@ -653,12 +666,13 @@ static void task_non_contending(struct sched_dl_entity *dl_se, bool dl_task)
get_task_struct(p);
/*
- * A dying task keeps its bandwidth in the root domain of the
- * CPU it runs on. Record that CPU. switched_from_dl() records
- * a task that leaves SCHED_DEADLINE.
+ * A dying task keeps its bandwidth until its 0-lag time. Put
+ * it on the departed list. The CPU recorded with its bandwidth
+ * tells the root domain that counts it. switched_from_dl() does
+ * the same for a task that leaves SCHED_DEADLINE.
*/
if (READ_ONCE(p->__state) == TASK_DEAD)
- dl_task_depart(p, task_cpu(p));
+ dl_task_depart(p);
}
hrtimer_start(timer, ns_to_ktime(zerolag_time), HRTIMER_MODE_REL_HARD);
@@ -896,19 +910,31 @@ static struct rq *dl_task_offline_migration(struct rq *rq, struct task_struct *p
}
/*
- * And we finally need to fix up root_domain(s) bandwidth accounting,
- * since p is still hanging out in the old (now moved to default) root
- * domain.
+ * And we finally need to fix up root_domain(s) bandwidth accounting.
+ * The bandwidth of @p is counted in the root domain of the CPU
+ * recorded with it. That is not the root domain of @rq if @p was
+ * moved to another cpuset while it was not running. If a rebuild
+ * dropped the bandwidth, there is nothing to move: the next rebuild
+ * counts it.
*/
- dl_b = &rq->rd->dl_bw;
- raw_spin_lock(&dl_b->lock);
- __dl_sub(dl_b, p->dl.dl_bw, cpumask_weight(rq->rd->span));
- raw_spin_unlock(&dl_b->lock);
+ scoped_guard (raw_spinlock, &dl_departed_lock) {
+ if (p->dl.dl_bw_gen == dl_bw_gen) {
+ struct rq *src_rq = cpu_rq(dl_task_root_domain_cpu(p));
- dl_b = &later_rq->rd->dl_bw;
- raw_spin_lock(&dl_b->lock);
- __dl_add(dl_b, p->dl.dl_bw, cpumask_weight(later_rq->rd->span));
- raw_spin_unlock(&dl_b->lock);
+ dl_b = &src_rq->rd->dl_bw;
+ raw_spin_lock(&dl_b->lock);
+ __dl_sub(dl_b, p->dl.dl_bw,
+ cpumask_weight(src_rq->rd->span));
+ raw_spin_unlock(&dl_b->lock);
+
+ dl_b = &later_rq->rd->dl_bw;
+ raw_spin_lock(&dl_b->lock);
+ __dl_add(dl_b, p->dl.dl_bw,
+ cpumask_weight(later_rq->rd->span));
+ raw_spin_unlock(&dl_b->lock);
+ }
+ p->dl.dl_bw_cpu = later_rq->cpu;
+ }
set_task_cpu(p, later_rq->cpu);
double_unlock_balance(later_rq, rq);
@@ -2417,7 +2443,7 @@ static enum hrtimer_restart inactive_task_timer(struct hrtimer *timer)
dl_se->dl_non_contending = 0;
}
- dl_task_release_bw(p, task_cpu(p));
+ dl_task_release_bw(p);
__dl_clear_params(dl_se);
goto unlock;
@@ -3565,14 +3591,26 @@ static void task_woken_dl(struct rq *rq, struct task_struct *p)
}
}
+/*
+ * Return a CPU of the root domain that counts the bandwidth of @p: the
+ * CPU recorded with it. Without a record, no root domain counts that
+ * bandwidth. Return the CPU @p is on then. Read the record once:
+ * dl_task_check_affinity() holds no lock.
+ */
+static unsigned int dl_task_root_domain_cpu(struct task_struct *p)
+{
+ int cpu = READ_ONCE(p->dl.dl_bw_cpu);
+
+ if (cpu < 0)
+ return task_cpu(p);
+ return cpu;
+}
+
static void set_cpus_allowed_dl(struct task_struct *p,
struct affinity_context *ctx)
{
- struct rq *rq;
-
WARN_ON_ONCE(!dl_task(p));
- rq = task_rq(p);
/*
* Migrating a SCHED_DEADLINE task between exclusive
* cpusets (different root_domains) entails a bandwidth
@@ -3580,17 +3618,31 @@ static void set_cpus_allowed_dl(struct task_struct *p,
* domain (see cpuset_can_attach()).
*/
if (dl_task_needs_bw_move(p, ctx->new_mask)) {
- struct dl_bw *src_dl_b;
+ unsigned int src_cpu = dl_task_root_domain_cpu(p);
+ struct dl_bw *src_dl_b = &cpu_rq(src_cpu)->rd->dl_bw;
+ unsigned int cpu;
- src_dl_b = dl_bw_of(cpu_of(rq));
/*
* We now free resources of the root_domain we are migrating
* off. In the worst case, sched_setattr() may temporary fail
- * until we complete the update.
+ * until we complete the update. A rebuild that dropped the
+ * bandwidth of @p leaves nothing to free. dl_departed_lock
+ * keeps a rebuild from changing dl_bw_gen meanwhile.
*/
- raw_spin_lock(&src_dl_b->lock);
- __dl_sub(src_dl_b, p->dl.dl_bw, dl_bw_cpus(task_cpu(p)));
- raw_spin_unlock(&src_dl_b->lock);
+ guard(raw_spinlock)(&dl_departed_lock);
+ if (p->dl.dl_bw_gen == dl_bw_gen) {
+ raw_spin_lock(&src_dl_b->lock);
+ __dl_sub(src_dl_b, p->dl.dl_bw, dl_bw_cpus(src_cpu));
+ raw_spin_unlock(&src_dl_b->lock);
+ }
+
+ /*
+ * cpuset_can_attach() made room in the root domain of the new
+ * mask. That root domain counts the bandwidth from now on.
+ */
+ cpu = cpumask_any_and(ctx->new_mask, cpu_active_mask);
+ p->dl.dl_bw_cpu = cpu < nr_cpu_ids ? (int)cpu : -1;
+ p->dl.dl_bw_gen = dl_bw_gen;
}
set_cpus_allowed_common(p, ctx);
@@ -3599,10 +3651,18 @@ static void set_cpus_allowed_dl(struct task_struct *p,
bool dl_task_needs_bw_move(struct task_struct *p,
const struct cpumask *new_mask)
{
- if (!dl_task(p))
+ unsigned int cpu;
+
+ /*
+ * Nothing to move for a special entity, or for a task boosted into
+ * the class with no bandwidth of its own. No root domain counts
+ * their bandwidth: dl_add_task_root_domain() leaves them out.
+ */
+ if (!dl_task(p) || dl_entity_is_special(&p->dl) || !p->dl.dl_bw)
return false;
- return !cpumask_intersects(task_rq(p)->rd->span, new_mask);
+ cpu = dl_task_root_domain_cpu(p);
+ return !cpumask_intersects(cpu_rq(cpu)->rd->span, new_mask);
}
/* Assumes rq->lock is held */
@@ -3678,7 +3738,8 @@ void dl_add_task_root_domain(struct task_struct *p)
* Count nothing for a special entity, or for a task with no
* bandwidth. The walk of the task list also finds exited tasks. An
* exited task whose bandwidth went with its parameters has a dl_bw
- * of 0.
+ * of 0. So has a task boosted into the class with no bandwidth of
+ * its own, which gets no record.
*/
if (!dl_task(p) || dl_entity_is_special(&p->dl) || !p->dl.dl_bw) {
raw_spin_unlock_irqrestore(&p->pi_lock, rf.flags);
@@ -3702,7 +3763,9 @@ void dl_add_task_root_domain(struct task_struct *p)
*/
if (p->dl.dl_bw_gen == dl_bw_gen)
goto unlock;
- if (p->dl.dl_departed_cpu >= 0) {
+ if (!list_empty(&p->dl.dl_departed_node)) {
+ int recorded = p->dl.dl_bw_cpu;
+
/*
* The walk also finds a dead task that departed before an
* earlier rebuild, if it is not reaped yet. If that rebuild
@@ -3711,9 +3774,9 @@ void dl_add_task_root_domain(struct task_struct *p)
*/
if (p->dl.dl_bw_gen != dl_bw_gen - 1)
goto unlock;
- cpu = p->dl.dl_departed_cpu;
- if (!cpu_active(cpu))
+ if (recorded < 0 || !cpu_active(recorded))
goto unlock;
+ cpu = recorded;
}
rq = cpu_rq(cpu);
dl_b = &rq->rd->dl_bw;
@@ -3721,6 +3784,7 @@ void dl_add_task_root_domain(struct task_struct *p)
raw_spin_lock(&dl_b->lock);
__dl_add(dl_b, p->dl.dl_bw, cpumask_weight(rq->rd->span));
raw_spin_unlock(&dl_b->lock);
+ p->dl.dl_bw_cpu = cpu;
p->dl.dl_bw_gen = dl_bw_gen;
unlock:
raw_spin_unlock(&dl_departed_lock);
@@ -3870,7 +3934,7 @@ void dl_clear_root_domains(void)
dl_clear_root_domain(cpu_rq(cpu)->rd);
}
list_for_each_entry(dl_se, &dl_departed_list, dl_departed_node) {
- cpu = dl_se->dl_departed_cpu;
+ cpu = dl_se->dl_bw_cpu;
/*
* Skip bandwidth that an earlier rebuild dropped. It is not
* counted now, and its release removes nothing
@@ -3885,7 +3949,7 @@ void dl_clear_root_domains(void)
* from the recorded number. The release then removes
* nothing, and no later rebuild restores it.
*/
- if (!cpu_active(cpu))
+ if (cpu < 0 || !cpu_active(cpu))
continue;
dl_b = dl_bw_of(cpu);
raw_spin_lock(&dl_b->lock);
@@ -3946,15 +4010,16 @@ static void switched_from_dl(struct rq *rq, struct task_struct *p)
* For the same reason, the timer cannot take the task's bandwidth
* from the root domain of the CPU the task is on by then. And a
* rebuild of the root domains meanwhile would not find the task
- * among the deadline tasks. So record a CPU of the root domain that
- * keeps the bandwidth until the timer removes it (dl_task_depart()).
+ * among the deadline tasks. So put the task on the departed list
+ * until the timer removes the bandwidth (dl_task_depart()). The CPU
+ * recorded with the bandwidth tells the root domain that keeps it.
* Test dl_non_contending: it tells that the timer is armed and its
* handler has not run. hrtimer_active() stays true for a while after
* the handler returns.
*/
if (p->dl.dl_non_contending) {
p->dl.dl_non_contending = 0;
- dl_task_depart(p, task_cpu(p));
+ dl_task_depart(p);
}
/*
@@ -3982,10 +4047,11 @@ static void switched_to_dl(struct rq *rq, struct task_struct *p)
* longer removes its bandwidth: the task does when it leaves the class
* again. So take it off the list. The task keeps the CPU recorded. It
* takes the bandwidth from the root domain of that CPU, even if it
- * moved to another root domain meanwhile.
+ * moved to another root domain meanwhile. A rebuild while it is
+ * boosted counts it like any deadline task, and records the CPU it
+ * picks.
*/
- if (!list_empty(&p->dl.dl_departed_node))
- dl_task_undepart(p);
+ dl_task_undepart(p);
/* If p is not queued we will update its parameters at next wakeup. */
if (!task_on_rq_queued(p)) {
@@ -4192,9 +4258,10 @@ int sched_dl_overflow(struct task_struct *p, int policy,
u64 period = attr->sched_period ?: attr->sched_deadline;
u64 runtime = attr->sched_runtime;
u64 new_bw = dl_policy(policy) ? to_ratio(period, runtime) : 0;
- int cpus, err = -1, cpu = task_cpu(p);
- struct dl_bw *dl_b = dl_bw_of(cpu);
+ int cpus, err = -1, cpu;
+ struct dl_bw *dl_b;
unsigned long cap;
+ bool counted;
if (attr->sched_flags & SCHED_FLAG_SUGOV)
return 0;
@@ -4203,6 +4270,23 @@ int sched_dl_overflow(struct task_struct *p, int policy,
if (new_bw == p->dl.dl_bw && task_has_dl_policy(p))
return 0;
+ /*
+ * Change the bandwidth of a deadline task where it is counted: in
+ * the root domain of the CPU recorded with it. A task moved asleep
+ * has not run there yet, so the CPU it is on can be elsewhere. No
+ * bandwidth is counted if a rebuild dropped it, or if the task
+ * enters the class. A runtime below about a millionth of the period
+ * gives a bandwidth of 0. It counts for nothing, and no move updates
+ * its record. In these three cases, the root domain of the CPU the
+ * task is on takes the new bandwidth. The caller holds cpuset_mutex,
+ * or dl_accounting_mutex without cpusets, so no rebuild runs
+ * meanwhile.
+ */
+ counted = task_has_dl_policy(p) && p->dl.dl_bw_cpu >= 0 &&
+ p->dl.dl_bw_gen == dl_bw_gen;
+ cpu = counted && p->dl.dl_bw ? p->dl.dl_bw_cpu : task_cpu(p);
+ dl_b = dl_bw_of(cpu);
+
/*
* Either if a task, enters, leave, or stays -deadline but changes
* its parameters, we may need to update accordingly the total
@@ -4217,7 +4301,8 @@ int sched_dl_overflow(struct task_struct *p, int policy,
__dl_add(dl_b, new_bw, cpus);
err = 0;
} else if (dl_policy(policy) && task_has_dl_policy(p) &&
- !__dl_overflow(dl_b, cap, p->dl.dl_bw, new_bw)) {
+ !__dl_overflow(dl_b, cap, counted ? p->dl.dl_bw : 0,
+ new_bw)) {
/*
* XXX this is slightly incorrect: when the task
* utilization decreases, we should delay the total
@@ -4225,7 +4310,8 @@ int sched_dl_overflow(struct task_struct *p, int policy,
* But this would require to set the task's "inactive
* timer" when the task is not inactive.
*/
- __dl_sub(dl_b, p->dl.dl_bw, cpus);
+ if (counted)
+ __dl_sub(dl_b, p->dl.dl_bw, cpus);
__dl_add(dl_b, new_bw, cpus);
dl_change_utilization(p, new_bw);
err = 0;
@@ -4246,11 +4332,20 @@ int sched_dl_overflow(struct task_struct *p, int policy,
* No rebuild can run meanwhile: the caller holds cpuset_mutex, or
* dl_accounting_mutex without cpusets.
*/
- if (!err && dl_policy(policy) && !task_has_dl_policy(p)) {
- lockdep_assert_cpuset_lock_held();
- lockdep_assert_dl_accounting_held();
- if (p->dl.dl_departed_cpu >= 0)
- dl_task_release_bw(p, -1);
+ if (!err && dl_policy(policy)) {
+ if (!task_has_dl_policy(p)) {
+ lockdep_assert_cpuset_lock_held();
+ lockdep_assert_dl_accounting_held();
+ /*
+ * The new bandwidth replaces any the task still has
+ * counted. That one may be on the departed list, or
+ * taken off it by a boost into the class
+ * (switched_to_dl()). The record says where it is.
+ */
+ if (p->dl.dl_bw_cpu >= 0)
+ dl_task_release_bw(p);
+ }
+ p->dl.dl_bw_cpu = cpu;
p->dl.dl_bw_gen = dl_bw_gen;
}
@@ -4409,8 +4504,8 @@ void init_dl_entity(struct sched_dl_entity *dl_se)
*/
dl_se->dl_bw = 0;
__dl_clear_params(dl_se);
+ dl_se->dl_bw_cpu = -1;
INIT_LIST_HEAD(&dl_se->dl_departed_node);
- dl_se->dl_departed_cpu = -1;
}
bool dl_param_changed(struct task_struct *p, const struct sched_attr *attr)
diff --git a/kernel/sched/syscalls.c b/kernel/sched/syscalls.c
index 96a5f0a7adbc..6562581721ed 100644
--- a/kernel/sched/syscalls.c
+++ b/kernel/sched/syscalls.c
@@ -1185,6 +1185,8 @@ SYSCALL_DEFINE4(sched_getattr, pid_t, pid, struct sched_attr __user *, uattr,
int dl_task_check_affinity(struct task_struct *p, const struct cpumask *mask)
{
+ unsigned int cpu;
+
/*
* If the task isn't a deadline task or admission control is
* disabled then we don't care about affinity changes.
@@ -1204,9 +1206,16 @@ int dl_task_check_affinity(struct task_struct *p, const struct cpumask *mask)
* if admission test is enabled, we only admit -deadline
* tasks allowed to run on all the CPUs in the task's
* root_domain.
+ *
+ * That is the root domain that counts the bandwidth of the task:
+ * the one of the CPU recorded with it. A task moved asleep to
+ * another cpuset is still on a CPU of the root domain it left. A
+ * bandwidth of 0 is counted nowhere, and no move updates its
+ * record: use the CPU the task is on.
*/
+ cpu = READ_ONCE(p->dl.dl_bw) ? dl_task_root_domain_cpu(p) : task_cpu(p);
guard(rcu)();
- if (!cpumask_subset(task_rq(p)->rd->span, mask))
+ if (!cpumask_subset(cpu_rq(cpu)->rd->span, mask))
return -EBUSY;
return 0;
--
2.47.3
^ permalink raw reply related [flat|nested] 35+ messages in thread
* [PATCH v8 04/33] sched/deadline: Leave a dying task out of a rebuild of the accounting
2026-11-19 20:15 [PATCH v8 00/33] sched/deadline: bandwidth accounting fixes and selftests Mathieu Desnoyers
` (2 preceding siblings ...)
2026-11-19 20:15 ` [PATCH v8 03/33] sched/deadline: Record the root domain holding a task's bandwidth Mathieu Desnoyers
@ 2026-11-19 20:15 ` Mathieu Desnoyers
2026-11-19 20:15 ` [PATCH v8 05/33] selftests/sched: Test SCHED_DEADLINE bandwidth across a root domain rebuild Mathieu Desnoyers
` (28 subsequent siblings)
32 siblings, 0 replies; 35+ messages in thread
From: Mathieu Desnoyers @ 2026-11-19 20:15 UTC (permalink / raw)
To: juri.lelli, peterz, mingo, vincent.guittot
Cc: Mathieu Desnoyers, tglx, will, boqun, dietmar.eggemann, rostedt,
bsegall, mgorman, vschneid, kprateek.nayak, luca.abeni,
mathieu.poirier, longman, ridong.chen, tj, hannes, mkoutny, shuah,
bigeasy, clrkwllms, parri.andrea, m.szyprowski, kayracizmeci,
weihengqiuu, linux-kernel, cgroups, linux-kselftest,
linux-rt-devel, stable
A rebuild of the root domains' accounting takes the pi_lock of each
deadline task, not its rq lock. A dying task past its 0-lag time
releases its bandwidth at its last schedule. It holds its rq lock there,
not its pi_lock. Then it clears dl_bw. The rebuild can find the task
between the two steps.
In this diagram, T is a deadline task that exits on CPU 0 past its 0-lag
time. bw is its dl_bw. CPU 1 rebuilds the accounting:
CPU 0 (T) CPU 1 (rebuild)
do_task_dead(): TASK_DEAD, under pi_lock
__schedule(): T's rq lock taken
task_non_contending(T)
dl_task_release_bw(T)
total_bw -= bw
dl_bw_gen recorded
record cleared
dl_rebuild_rd_accounting()
dl_clear_root_domains()
dl_bw_gen++
total_bw reset
dl_add_task_root_domain(T)
T's pi_lock taken
dl_bw != 0, dl_bw_gen old
total_bw += bw
__dl_clear_params(T): dl_bw = 0
dl_bw_gen does not help, since the release came before the rebuild. The
rebuild would count a bandwidth that nothing releases again. The root
domain would then refuse deadline tasks it has room for, until the next
rebuild.
Fix this with the record of the bandwidth, dl_bw_cpu. The release
clears it under dl_departed_lock, which the walk holds. So leave out a
dead task with no record. A dead task with a record has not reached its
last schedule. The walk counts it as a live task. That schedule then
releases the bandwidth, or puts the task on the departed list. A dead
task on the departed list keeps its bandwidth until its timer, and the
rebuild counts it as before.
This patch completes "sched/deadline: Rebuild the root domains'
accounting from the task list". The walk of that patch finds exiting
tasks, which the cpuset iterator skipped. The fix needs the record
that "sched/deadline: Record the root domain holding a task's
bandwidth" adds. The three go to stable together.
The Sashiko AI review of this series found this window. It also found a
gap in the first version of this fix. That version also left out a task
between TASK_DEAD and its last schedule, with its 0-lag time ahead.
Fixes: f9a25f776d78 ("cpusets: Rebuild root domain deadline accounting information")
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Reviewed-by: Juri Lelli <juri.lelli@redhat.com>
Cc: stable@vger.kernel.org
---
kernel/sched/deadline.c | 11 +++++++++++
1 file changed, 11 insertions(+)
diff --git a/kernel/sched/deadline.c b/kernel/sched/deadline.c
index 5a8f38632e54..40d87cd31d33 100644
--- a/kernel/sched/deadline.c
+++ b/kernel/sched/deadline.c
@@ -3777,6 +3777,17 @@ void dl_add_task_root_domain(struct task_struct *p)
if (recorded < 0 || !cpu_active(recorded))
goto unlock;
cpu = recorded;
+ } else if (READ_ONCE(p->__state) == TASK_DEAD && p->dl.dl_bw_cpu < 0) {
+ /*
+ * Count nothing for a dead task with no record. Its last
+ * schedule released the bandwidth under its rq lock, and has
+ * not cleared dl_bw yet. The release clears the record under
+ * dl_departed_lock, which this walk holds. A dead task with a
+ * record has not reached its last schedule. Count it as a
+ * live task. That schedule then releases what the walk
+ * counted, or puts the task on the departed list.
+ */
+ goto unlock;
}
rq = cpu_rq(cpu);
dl_b = &rq->rd->dl_bw;
--
2.47.3
^ permalink raw reply related [flat|nested] 35+ messages in thread
* [PATCH v8 05/33] selftests/sched: Test SCHED_DEADLINE bandwidth across a root domain rebuild
2026-11-19 20:15 [PATCH v8 00/33] sched/deadline: bandwidth accounting fixes and selftests Mathieu Desnoyers
` (3 preceding siblings ...)
2026-11-19 20:15 ` [PATCH v8 04/33] sched/deadline: Leave a dying task out of a rebuild of the accounting Mathieu Desnoyers
@ 2026-11-19 20:15 ` Mathieu Desnoyers
2026-11-19 20:15 ` [PATCH v8 06/33] selftests/sched: Test the root domain rebuild in deadline_bw_test Mathieu Desnoyers
` (27 subsequent siblings)
32 siblings, 0 replies; 35+ messages in thread
From: Mathieu Desnoyers @ 2026-11-19 20:15 UTC (permalink / raw)
To: juri.lelli, peterz, mingo, vincent.guittot
Cc: Mathieu Desnoyers, tglx, will, boqun, dietmar.eggemann, rostedt,
bsegall, mgorman, vschneid, kprateek.nayak, luca.abeni,
mathieu.poirier, longman, ridong.chen, tj, hannes, mkoutny, shuah,
bigeasy, clrkwllms, parri.andrea, m.szyprowski, kayracizmeci,
weihengqiuu, linux-kernel, cgroups, linux-kselftest,
linux-rt-devel
A task that leaves SCHED_DEADLINE, or dies, keeps its bandwidth until
its 0-lag time. No selftest checks what happens to this bandwidth
meanwhile.
Add deadline_rebuild_test, with three cases. In each case, a task of
400ms every second runs 380ms, then leaves SCHED_DEADLINE. Its 0-lag
time is then about 570ms ahead. P is a partition root of one CPU.
Before the 0-lag time:
- dies_before_rebuild: the task exits in P. Then a second partition
root is made, which rebuilds the accounting of every root domain.
- switched_out_before_rebuild: the same, but the task switches to
SCHED_OTHER and sleeps in P.
- moved_after_switch_out: the task switches to SCHED_OTHER outside P.
Then it moves into P and dies there.
After the 0-lag time, P must admit one task of 600ms every second, and
refuse a second one with EBUSY. A kernel with the bug removes the
bandwidth from P's root domain at the 0-lag time. But the rebuild
dropped it already, or P never had it. total_bw then wraps around, and
P admits both tasks.
The test skips without root, 3 CPUs, the cgroup v2 cpuset controller or
deadline admission control. Admission control is off when
sched_rt_runtime_us is -1.
On v7.4-rc2, in 8-CPU VMs, the three cases fail. "sched/deadline: Keep
departing bandwidth across a root domain rebuild" fixes them. With the
whole series, the three cases pass, with and without PREEMPT_RT.
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
---
.../selftests/sched/deadline/.gitignore | 1 +
.../testing/selftests/sched/deadline/Makefile | 2 +-
.../sched/deadline/deadline_rebuild_test.c | 497 ++++++++++++++++++
3 files changed, 499 insertions(+), 1 deletion(-)
create mode 100644 tools/testing/selftests/sched/deadline/deadline_rebuild_test.c
diff --git a/tools/testing/selftests/sched/deadline/.gitignore b/tools/testing/selftests/sched/deadline/.gitignore
index 5a572318e1ba..42f6b2945e96 100644
--- a/tools/testing/selftests/sched/deadline/.gitignore
+++ b/tools/testing/selftests/sched/deadline/.gitignore
@@ -1,2 +1,3 @@
deadline_switch_test
+deadline_rebuild_test
deadline_bw_test
diff --git a/tools/testing/selftests/sched/deadline/Makefile b/tools/testing/selftests/sched/deadline/Makefile
index c3156059bdcf..15f1e96a21e6 100644
--- a/tools/testing/selftests/sched/deadline/Makefile
+++ b/tools/testing/selftests/sched/deadline/Makefile
@@ -3,6 +3,6 @@
CFLAGS += -O2 -Wall -g -I./ $(KHDR_INCLUDES)
LDLIBS += -lpthread
-TEST_GEN_PROGS := deadline_switch_test deadline_bw_test
+TEST_GEN_PROGS := deadline_switch_test deadline_rebuild_test deadline_bw_test
include ../../lib.mk
diff --git a/tools/testing/selftests/sched/deadline/deadline_rebuild_test.c b/tools/testing/selftests/sched/deadline/deadline_rebuild_test.c
new file mode 100644
index 000000000000..714a0ff03f1b
--- /dev/null
+++ b/tools/testing/selftests/sched/deadline/deadline_rebuild_test.c
@@ -0,0 +1,497 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Test the bandwidth of a SCHED_DEADLINE task that leaves the policy, or
+ * dies, before its 0-lag time. The bandwidth stays in the task's root
+ * domain until the inactive timer removes it. It must be removed once,
+ * from the root domain that holds it.
+ *
+ * P is a partition root of one CPU. In each case the task is a deadline
+ * task of 400ms every second, which first runs 380ms. The cases are:
+ * - dies_before_rebuild: the task is in P and exits. Before its 0-lag
+ * time, the test makes a second partition root, Q. This rebuilds the
+ * accounting of every root domain.
+ * - switched_out_before_rebuild: the same, but the task switches to
+ * SCHED_OTHER and sleeps instead of exiting.
+ * - moved_after_switch_out: the task is outside P. It switches to
+ * SCHED_OTHER and sleeps. Before its 0-lag time, the test moves it into
+ * P and kills it there.
+ *
+ * Once the inactive timer has fired, P must admit one task of 600ms
+ * every second, and refuse a second one with EBUSY. A kernel with the
+ * bug removes the bandwidth from P's root domain after the rebuild
+ * dropped it, or when P never had it. total_bw of that root domain then
+ * wraps around, and P admits both tasks.
+ *
+ * The test needs root, 3 CPUs, the cgroup v2 cpuset controller and
+ * deadline admission control.
+ */
+#define _GNU_SOURCE
+#include <errno.h>
+#include <fcntl.h>
+#include <limits.h>
+#include <sched.h>
+#include <signal.h>
+#include <stdint.h>
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+#include <sys/stat.h>
+#include <sys/syscall.h>
+#include <sys/sysinfo.h>
+#include <sys/wait.h>
+#include <time.h>
+#include <unistd.h>
+
+#include "../../kselftest.h"
+
+#ifndef SCHED_DEADLINE
+#define SCHED_DEADLINE 6
+#endif
+
+#define NSEC_PER_MSEC 1000000ULL
+#define NSEC_PER_SEC 1000000000ULL
+
+struct dl_attr {
+ uint32_t size;
+ uint32_t sched_policy;
+ uint64_t sched_flags;
+ int32_t sched_nice;
+ uint32_t sched_priority;
+ uint64_t sched_runtime;
+ uint64_t sched_deadline;
+ uint64_t sched_period;
+};
+
+static char root[PATH_MAX], cg_p[PATH_MAX], cg_q[PATH_MAX];
+static int ncpus;
+
+static int set_deadline(pid_t pid, uint64_t runtime)
+{
+ struct dl_attr attr = {
+ .size = sizeof(attr),
+ .sched_policy = SCHED_DEADLINE,
+ .sched_runtime = runtime,
+ .sched_deadline = NSEC_PER_SEC,
+ .sched_period = NSEC_PER_SEC,
+ };
+
+ return syscall(__NR_sched_setattr, pid, &attr, 0) ? -errno : 0;
+}
+
+static int set_other(pid_t pid)
+{
+ struct sched_param sp = { .sched_priority = 0 };
+
+ return sched_setscheduler(pid, SCHED_OTHER, &sp) ? -errno : 0;
+}
+
+/*
+ * Return the CLOCK_MONOTONIC time in ns. clock_gettime() cannot fail: the
+ * clock is always supported, and the pointer is valid.
+ */
+static uint64_t now_ns(void)
+{
+ struct timespec ts;
+
+ clock_gettime(CLOCK_MONOTONIC, &ts);
+ return ts.tv_sec * NSEC_PER_SEC + ts.tv_nsec;
+}
+
+static void sleep_ns(uint64_t ns)
+{
+ struct timespec ts = { .tv_sec = ns / NSEC_PER_SEC, .tv_nsec = ns % NSEC_PER_SEC };
+
+ while (nanosleep(&ts, &ts) && errno == EINTR)
+ ;
+}
+
+static void reap(pid_t pid)
+{
+ int status;
+
+ if (pid <= 0)
+ return;
+ if (kill(pid, SIGKILL) && errno != ESRCH)
+ ksft_print_msg("kill %d: %s\n", pid, strerror(errno));
+ /* Ignore the result: the task may be dead already. */
+ set_other(pid);
+ while (waitpid(pid, &status, 0) != pid) {
+ if (errno == EINTR)
+ continue;
+ ksft_print_msg("waitpid %d: %s\n", pid, strerror(errno));
+ break;
+ }
+}
+
+static int write_file(const char *dir, const char *name, const char *val)
+{
+ char path[PATH_MAX];
+ int fd, err = 0;
+ ssize_t n;
+
+ if (snprintf(path, sizeof(path), "%s/%s", dir, name) >= (int)sizeof(path))
+ return -ENAMETOOLONG;
+ fd = open(path, O_WRONLY);
+ if (fd < 0)
+ return -errno;
+ n = write(fd, val, strlen(val));
+ if (n < 0)
+ err = -errno;
+ else if (n != (ssize_t)strlen(val))
+ err = -EIO;
+ if (close(fd) && !err)
+ err = -errno;
+ return err;
+}
+
+static int read_file(const char *path, char *buf, size_t len)
+{
+ int fd = open(path, O_RDONLY);
+ ssize_t n;
+
+ if (fd < 0)
+ return -1;
+ do {
+ n = read(fd, buf, len - 1);
+ } while (n < 0 && errno == EINTR);
+ close(fd);
+ if (n < 0)
+ return -1;
+ buf[n] = 0;
+ return 0;
+}
+
+static int enter(const char *cg, pid_t pid)
+{
+ char buf[16];
+
+ snprintf(buf, sizeof(buf), "%d", pid);
+ return write_file(cg, "cgroup.procs", buf);
+}
+
+/* Make @cg a partition root of CPU @cpu alone. Return 0, or -1 on error. */
+static int make_partition(const char *cg, int cpu)
+{
+ char path[PATH_MAX], buf[32];
+
+ if (mkdir(cg, 0755) && errno != EEXIST)
+ return -1;
+ snprintf(buf, sizeof(buf), "%d", cpu);
+ if (write_file(cg, "cpuset.cpus", buf) || write_file(cg, "cpuset.cpus.partition", "root"))
+ return -1;
+ snprintf(path, sizeof(path), "%s/cpuset.cpus.partition", cg);
+ if (read_file(path, buf, sizeof(buf)) || strcmp(buf, "root\n"))
+ return -1;
+ return 0;
+}
+
+static void remove_partition(const char *cg)
+{
+ int err = write_file(cg, "cpuset.cpus.partition", "member");
+ uint64_t end;
+
+ if (err && err != -ENOENT)
+ ksft_print_msg("%s back to a member: %s\n", cg, strerror(-err));
+ /* A cgroup whose last task just died takes a moment to empty. */
+ end = now_ns() + NSEC_PER_SEC;
+ while (rmdir(cg)) {
+ if (errno == ENOENT)
+ return;
+ if (errno != EBUSY || now_ns() > end) {
+ ksft_print_msg("rmdir %s: %s\n", cg, strerror(errno));
+ return;
+ }
+ sleep_ns(10 * NSEC_PER_MSEC);
+ }
+}
+
+/*
+ * Return the CPU time of the calling thread in ns. clock_gettime() cannot
+ * fail: the clock is always supported, and the pointer is valid.
+ */
+static uint64_t cpu_time_ns(void)
+{
+ struct timespec ts;
+
+ clock_gettime(CLOCK_THREAD_CPUTIME_ID, &ts);
+ return ts.tv_sec * NSEC_PER_SEC + ts.tv_nsec;
+}
+
+enum depart { DIES, SWITCHED_OUT };
+
+/*
+ * Fork a child that enters @cg, if given, then SCHED_DEADLINE at 400ms
+ * every second. The child runs 380ms of its runtime. Then it dies if @how
+ * is DIES. Otherwise it switches to SCHED_OTHER, writes a byte to @fd and
+ * sleeps. Its 0-lag time is then about 570ms ahead. Return the pid of the
+ * child, or -1. Only the read end of @fd stays open in the caller.
+ */
+static pid_t departing(const char *cg, enum depart how, int fd[2])
+{
+ pid_t pid = fork();
+
+ if (pid < 0)
+ ksft_print_msg("fork: %s\n", strerror(errno));
+ if (!pid) {
+ uint64_t t0;
+
+ close(fd[0]);
+ if (cg && enter(cg, getpid()))
+ _exit(2);
+ if (set_deadline(0, 400 * NSEC_PER_MSEC))
+ _exit(3);
+ t0 = cpu_time_ns();
+ while (cpu_time_ns() - t0 < 380 * NSEC_PER_MSEC)
+ ;
+ if (how == DIES)
+ _exit(0);
+ if (set_other(0) || write(fd[1], "x", 1) != 1)
+ _exit(4);
+ for (;;)
+ pause();
+ }
+ close(fd[1]);
+ fd[1] = -1;
+ return pid;
+}
+
+/*
+ * Wait until the child is out of SCHED_DEADLINE: dead or switched out.
+ * Return 0 on success. If the child failed, return its exit status, or
+ * -1. Set *@pid to -1 once the child is reaped.
+ */
+static int departed(pid_t *pid, enum depart how, int fd)
+{
+ int status;
+ char c;
+
+ if (*pid < 0)
+ return -1;
+ /*
+ * A child that switches out sends a byte. Without the byte, the
+ * child died or failed, and its exit status tells which.
+ */
+ if (how == SWITCHED_OUT) {
+ ssize_t n;
+
+ do {
+ n = read(fd, &c, 1);
+ } while (n < 0 && errno == EINTR);
+ if (n == 1)
+ return 0;
+ }
+ if (waitpid(*pid, &status, 0) != *pid)
+ return -1;
+ *pid = -1;
+ return WIFEXITED(status) ? WEXITSTATUS(status) : -1;
+}
+
+/*
+ * Fork a child that enters @cg and sleeps there. The child moves itself,
+ * so that it last ran on a CPU of @cg. Wait until it is there. Return its
+ * pid, or -1.
+ */
+static pid_t sleeper(const char *cg)
+{
+ int fd[2];
+ pid_t pid;
+ char c;
+
+ if (pipe(fd))
+ return -1;
+ pid = fork();
+ if (pid < 0)
+ ksft_print_msg("fork: %s\n", strerror(errno));
+ if (!pid) {
+ close(fd[0]);
+ if (enter(cg, getpid()) || write(fd[1], "x", 1) != 1)
+ _exit(1);
+ for (;;)
+ pause();
+ }
+ close(fd[1]);
+ if (pid > 0 && read(fd[0], &c, 1) != 1) {
+ reap(pid);
+ pid = -1;
+ }
+ close(fd[0]);
+ return pid;
+}
+
+/*
+ * Check that P admits one task of 600ms every second, but not two. The
+ * caller must wait until every inactive timer has fired.
+ */
+static int check_p(void)
+{
+ pid_t a = sleeper(cg_p), b = sleeper(cg_p);
+ int ret = KSFT_FAIL, e1, e2;
+
+ if (a < 0 || b < 0) {
+ ksft_print_msg("no task could enter P\n");
+ goto out;
+ }
+ e1 = set_deadline(a, 600 * NSEC_PER_MSEC);
+ e2 = set_deadline(b, 600 * NSEC_PER_MSEC);
+ if (e1)
+ ksft_print_msg("a first task refused in P: %s\n", strerror(-e1));
+ else if (!e2)
+ ksft_print_msg("1.2 CPUs of deadline tasks admitted in a partition of one CPU\n");
+ else if (e2 != -EBUSY)
+ ksft_print_msg("the second task refused with %s, not EBUSY\n", strerror(-e2));
+ else
+ ret = KSFT_PASS;
+out:
+ reap(a);
+ reap(b);
+ return ret;
+}
+
+/*
+ * Have a task of P die or switch out, as @how says. Make Q before its
+ * 0-lag time, then check P.
+ */
+static int test_rebuild(enum depart how)
+{
+ int ret = KSFT_FAIL, fd[2], err;
+ pid_t pid = -1;
+
+ if (pipe(fd))
+ return KSFT_FAIL;
+ if (make_partition(cg_p, ncpus - 1)) {
+ ksft_print_msg("no partition root on CPU %d\n", ncpus - 1);
+ ret = KSFT_SKIP;
+ goto out;
+ }
+ pid = departing(cg_p, how, fd);
+ err = departed(&pid, how, fd[0]);
+ if (err) {
+ ksft_print_msg("the deadline task failed (%d)\n", err);
+ goto out;
+ }
+ if (make_partition(cg_q, ncpus - 2)) {
+ ksft_print_msg("no partition root on CPU %d\n", ncpus - 2);
+ ret = KSFT_SKIP;
+ goto out;
+ }
+ /*
+ * Wait past the 0-lag time. In the SWITCHED_OUT case the task is still
+ * alive then: it sleeps in P as a SCHED_OTHER task.
+ */
+ sleep_ns(1500 * NSEC_PER_MSEC);
+ ret = check_p();
+out:
+ reap(pid);
+ remove_partition(cg_q);
+ remove_partition(cg_p);
+ close(fd[0]);
+ if (fd[1] >= 0)
+ close(fd[1]);
+ return ret;
+}
+
+/*
+ * Have a task outside P switch to SCHED_OTHER and sleep. Before its 0-lag
+ * time, move it into P and kill it. Then check P.
+ */
+static int test_move(void)
+{
+ int ret = KSFT_FAIL, fd[2], err;
+ pid_t pid = -1;
+
+ if (pipe(fd))
+ return KSFT_FAIL;
+ if (make_partition(cg_p, ncpus - 1)) {
+ ksft_print_msg("no partition root on CPU %d\n", ncpus - 1);
+ ret = KSFT_SKIP;
+ goto out;
+ }
+ pid = departing(NULL, SWITCHED_OUT, fd);
+ err = departed(&pid, SWITCHED_OUT, fd[0]);
+ if (err) {
+ ksft_print_msg("the deadline task failed (%d)\n", err);
+ goto out;
+ }
+ if (enter(cg_p, pid)) {
+ ksft_print_msg("the task cannot move into P\n");
+ goto out;
+ }
+ /* The task wakes up and dies on P's CPU, before its 0-lag time. */
+ reap(pid);
+ pid = -1;
+ sleep_ns(1500 * NSEC_PER_MSEC);
+ ret = check_p();
+out:
+ reap(pid);
+ remove_partition(cg_p);
+ close(fd[0]);
+ if (fd[1] >= 0)
+ close(fd[1]);
+ return ret;
+}
+
+static int find_cgroup2(void)
+{
+ char line[PATH_MAX + 64], dev[64], dir[PATH_MAX], type[64];
+ FILE *f = fopen("/proc/self/mounts", "r");
+ int found = 0;
+
+ while (f && !found && fgets(line, sizeof(line), f)) {
+ if (sscanf(line, "%63s %4095s %63s", dev, dir, type) == 3 &&
+ !strcmp(type, "cgroup2")) {
+ snprintf(root, sizeof(root), "%s", dir);
+ found = 1;
+ }
+ }
+ if (f)
+ fclose(f);
+ return found ? 0 : -1;
+}
+
+static void report(const char *name, int ret)
+{
+ if (ret == KSFT_PASS)
+ ksft_test_result_pass("%s\n", name);
+ else if (ret == KSFT_SKIP)
+ ksft_test_result_skip("%s\n", name);
+ else
+ ksft_test_result_fail("%s\n", name);
+}
+
+int main(void)
+{
+ char buf[256], path[PATH_MAX + 32];
+ int added;
+
+ ksft_print_header();
+ if (geteuid())
+ ksft_exit_skip("needs root\n");
+ ncpus = get_nprocs();
+ if (ncpus < 3)
+ ksft_exit_skip("needs 3 CPUs\n");
+ if (read_file("/proc/sys/kernel/sched_rt_runtime_us", buf, sizeof(buf)) || atol(buf) < 0)
+ ksft_exit_skip("no deadline admission control\n");
+ if (find_cgroup2())
+ ksft_exit_skip("cgroup v2 is not mounted\n");
+ /*
+ * Enable the cpuset controller for the root's children, if it is
+ * off. It is turned off again at the end.
+ */
+ snprintf(path, sizeof(path), "%s/cgroup.subtree_control", root);
+ if (read_file(path, buf, sizeof(buf)))
+ ksft_exit_skip("no cgroup.subtree_control\n");
+ added = !strstr(buf, "cpuset");
+ if (added && write_file(root, "cgroup.subtree_control", "+cpuset"))
+ ksft_exit_skip("no cpuset controller\n");
+ if (snprintf(cg_p, sizeof(cg_p), "%s/deadline_rebuild_p", root) >= (int)sizeof(cg_p) ||
+ snprintf(cg_q, sizeof(cg_q), "%s/deadline_rebuild_q", root) >= (int)sizeof(cg_q))
+ ksft_exit_skip("cgroup v2 mount path too long\n");
+
+ ksft_set_plan(3);
+ report("dies_before_rebuild", test_rebuild(DIES));
+ report("switched_out_before_rebuild", test_rebuild(SWITCHED_OUT));
+ report("moved_after_switch_out", test_move());
+ if (added && write_file(root, "cgroup.subtree_control", "-cpuset"))
+ ksft_print_msg("the cpuset controller stays enabled\n");
+ ksft_finished();
+}
--
2.47.3
^ permalink raw reply related [flat|nested] 35+ messages in thread
* [PATCH v8 06/33] selftests/sched: Test the root domain rebuild in deadline_bw_test
2026-11-19 20:15 [PATCH v8 00/33] sched/deadline: bandwidth accounting fixes and selftests Mathieu Desnoyers
` (4 preceding siblings ...)
2026-11-19 20:15 ` [PATCH v8 05/33] selftests/sched: Test SCHED_DEADLINE bandwidth across a root domain rebuild Mathieu Desnoyers
@ 2026-11-19 20:15 ` Mathieu Desnoyers
2026-11-19 20:15 ` [PATCH v8 07/33] sched/deadline: Move a task's bandwidth only where room was made Mathieu Desnoyers
` (26 subsequent siblings)
32 siblings, 0 replies; 35+ messages in thread
From: Mathieu Desnoyers @ 2026-11-19 20:15 UTC (permalink / raw)
To: juri.lelli, peterz, mingo, vincent.guittot
Cc: Mathieu Desnoyers, tglx, will, boqun, dietmar.eggemann, rostedt,
bsegall, mgorman, vschneid, kprateek.nayak, luca.abeni,
mathieu.poirier, longman, ridong.chen, tj, hannes, mkoutny, shuah,
bigeasy, clrkwllms, parri.andrea, m.szyprowski, kayracizmeci,
weihengqiuu, linux-kernel, cgroups, linux-kselftest,
linux-rt-devel
Add seven cases to deadline_bw_test. Most test the rebuild of the root
domains' accounting, earlier in this series. They cover:
- CPU hotplug with sleeping deadline tasks, and with tasks that left the
class before their 0-lag time
- a zombie deadline task during a CPU hotplug (death_zombie_hotplug)
- a priority inheritance owner that left the class, then moves
(pi_departed_moved)
- sched_setaffinity() racing with sched_setattr() (affinity_vs_setattr)
The comment above each case gives its scenario. Two cases follow
reproducers of Juri Lelli's sched-deadline-tests:
hotplug_sleeper_switched and hotplug_no_admission_pinned.
The message of each fix names the cases that test it. These cases are
not named there:
- hotplug_departed and hotplug_no_admission_pinned: "sched/deadline: Keep
departing bandwidth across a root domain rebuild"
- affinity_vs_setattr tests "sched/deadline: Check SCHED_DEADLINE
affinity under the rq lock", of the first series. It comes here
because it fails without CONFIG_CPUSETS before these fixes.
On v7.4-rc2, in 8 VMs of 8 CPUs, six of these cases fail in every VM:
- affinity_vs_setattr
- death_zombie_hotplug
- hotplug_departed
- hotplug_no_admission_pinned
- hotplug_sleeper_switched
- pi_departed_moved
hotplug_sleepers passes there. It fails on v7.4-rc2 and with the first
series without CONFIG_CPUSETS. It also fails there when
isolcpus=domain,0,1 isolates the boot CPU. The rebuild fixes make it
pass.
With the fixes up to here, every case passes wherever it runs.
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Link: https://github.com/jlelli/sched-deadline-tests
---
.../sched/deadline/deadline_bw_test.c | 654 +++++++++++++++++-
1 file changed, 650 insertions(+), 4 deletions(-)
diff --git a/tools/testing/selftests/sched/deadline/deadline_bw_test.c b/tools/testing/selftests/sched/deadline/deadline_bw_test.c
index 6ff9da964403..ba2a18f354d4 100644
--- a/tools/testing/selftests/sched/deadline/deadline_bw_test.c
+++ b/tools/testing/selftests/sched/deadline/deadline_bw_test.c
@@ -31,8 +31,11 @@
* housekeeping CPU. P is made anew for each case. Making it rebuilds the
* accounting, which clears what an earlier case left.
*
- * The cases of hp_cases[] run after them, without P. They take CPUs
- * offline. The test skips them if no CPU can go offline.
+ * The cases of hp_cases[] run after them, without P. These are the
+ * hotplug cases, and affinity_vs_setattr. The test skips the hotplug
+ * cases if no CPU can go offline. It still runs affinity_vs_setattr.
+ * death_zombie_hotplug takes CPUs offline too, but it is in p_cases[]: it
+ * needs the cpuset setup of those cases.
*
* The server cases change the fair server of a CPU through debugfs, and
* the test then puts it back.
@@ -4554,7 +4557,10 @@ static int params_waking(void)
return ret;
}
-/* CPU hotplug, from the main process, with no P */
+/*
+ * CPU hotplug, from the main process, with no P. Two cases below run with
+ * P instead: death_zombie_hotplug and pi_departed_moved.
+ */
static int hp_cpu = -1;
@@ -4584,6 +4590,213 @@ static void cpu_up(int cpu)
ksft_print_msg("CPU %d cannot come back online: %s\n", cpu, strerror(-err));
}
+/*
+ * Four deadline tasks of 30% sleep. Take hp_cpu offline and back. Once
+ * the sleepers are gone, the housekeeping root domain must admit what it
+ * did before. Skip unless 22 probe tasks fit: the sleepers take the room
+ * of 12, and the offline CPU that of about 10.
+ */
+static int hotplug_sleepers(void)
+{
+ int base = probe(NULL), after, i, ret = KSFT_PASS;
+ pid_t pid[4];
+
+ if (base < 22) {
+ ksft_print_msg("needs room for 22 probe tasks\n");
+ return KSFT_SKIP;
+ }
+ for (i = 0; i < 4; i++) {
+ mv_state[0] = 0;
+ pid[i] = mover(MV_ASLEEP, 30 * NSEC_PER_MSEC);
+ if (pid[i] < 0 || child_up("a deadline sleeper", 0)) {
+ while (i >= 0)
+ reap(pid[i--]);
+ return KSFT_FAIL;
+ }
+ }
+ if (cpu_online(hp_cpu, 0) || cpu_online(hp_cpu, 1)) {
+ ksft_print_msg("offline and online of CPU %d with deadline sleepers failed\n",
+ hp_cpu);
+ ret = KSFT_FAIL;
+ cpu_up(hp_cpu);
+ }
+ for (i = 0; i < 4; i++)
+ reap(pid[i]);
+ sleep_ns(3 * PERIOD);
+ after = probe(NULL);
+ if (ret == KSFT_PASS && after != base) {
+ ksft_print_msg("hotplug with sleepers: %d probe tasks admitted before, %d after\n",
+ base, after);
+ ret = KSFT_FAIL;
+ }
+ return ret;
+}
+
+/*
+ * Fork a task that runs on @cpu, then restores its affinity, becomes a
+ * deadline task of 30% and sleeps. A deadline task cannot have an
+ * affinity of one CPU, so the task goes to @cpu first. It sets
+ * mv_state[0] as mover() does. Return its pid, or -1.
+ */
+static pid_t sleeper_on(int cpu)
+{
+ pid_t pid = fork_child("a deadline sleeper");
+
+ if (!pid) {
+ cpu_set_t all, one;
+
+ CPU_ZERO(&one);
+ CPU_SET(cpu, &one);
+ if (sched_getaffinity(0, sizeof(all), &all) ||
+ sched_setaffinity(0, sizeof(one), &one))
+ _exit(1);
+ spin_ns(NSEC_PER_MSEC);
+ if (sched_setaffinity(0, sizeof(all), &all) ||
+ set_deadline(0, 30 * NSEC_PER_MSEC, PERIOD)) {
+ mv_state[0] = -1;
+ _exit(1);
+ }
+ mv_state[0] = 1;
+ for (;;)
+ pause();
+ }
+ return pid;
+}
+
+/*
+ * A deadline task of 30% sleeps on hp_cpu. Take hp_cpu offline, then
+ * switch the task to SCHED_OTHER. hp_cpu has left the housekeeping root
+ * domain by then, but that root domain still counts the task. The switch
+ * must release the bandwidth there.
+ *
+ * Check this with hp_cpu still offline: bringing it back rebuilds the
+ * accounting, which would hide a bandwidth left behind. The reference is
+ * what the housekeeping root domain admits with hp_cpu offline and no
+ * task. Once the CPU is back and the task is gone, that root domain must
+ * also admit what it did at first.
+ */
+static int hotplug_sleeper_switched(void)
+{
+ int base = probe(NULL), base_off, after, cpu, ret = KSFT_PASS;
+ pid_t pid;
+
+ if (cpu_online(hp_cpu, 0)) {
+ ksft_print_msg("CPU %d cannot go offline\n", hp_cpu);
+ cpu_up(hp_cpu);
+ return KSFT_SKIP;
+ }
+ base_off = probe(NULL);
+ cpu_up(hp_cpu);
+ mv_state[0] = 0;
+ pid = sleeper_on(hp_cpu);
+ if (pid < 0 || child_up("a deadline sleeper", 0)) {
+ reap(pid);
+ return KSFT_FAIL;
+ }
+ sleep_ns(PERIOD);
+ cpu = task_cpu_of(pid);
+ if (cpu != hp_cpu) {
+ ksft_print_msg("the sleeper is on CPU %d, not on CPU %d\n", cpu, hp_cpu);
+ reap(pid);
+ return KSFT_SKIP;
+ }
+ if (cpu_online(hp_cpu, 0)) {
+ ksft_print_msg("CPU %d cannot go offline under a deadline sleeper\n", hp_cpu);
+ ret = KSFT_FAIL;
+ } else if (set_other(pid)) {
+ ksft_print_msg("the sleeper cannot leave SCHED_DEADLINE\n");
+ ret = KSFT_FAIL;
+ } else {
+ sleep_ns(3 * PERIOD);
+ after = probe(NULL);
+ if (after != base_off) {
+ ksft_print_msg("a sleeper switched out, its CPU still offline: %d probe tasks admitted, %d expected\n",
+ after, base_off);
+ ret = KSFT_FAIL;
+ }
+ }
+ cpu_up(hp_cpu);
+ reap(pid);
+ sleep_ns(3 * PERIOD);
+ after = probe(NULL);
+ if (ret == KSFT_PASS && after != base) {
+ ksft_print_msg("a sleeper switched out on an offline CPU: %d probe tasks admitted before, %d after\n",
+ base, after);
+ ret = KSFT_FAIL;
+ }
+ return ret;
+}
+
+/*
+ * Turn admission control off (sched_rt_runtime_us = -1). A task pins
+ * itself to hp_cpu, becomes a deadline task of 50% and spins. Without
+ * admission control, a deadline task can have an affinity of one CPU.
+ * Take hp_cpu offline: the mask of the task holds no active CPU. The
+ * task must go on running on another CPU. Once the CPU is back, the task
+ * is gone and admission control is on again, the housekeeping root domain
+ * must admit what it did before.
+ */
+static int hotplug_no_admission_pinned(void)
+{
+ int base = probe(NULL), after, ret = KSFT_PASS, last;
+ char saved[64];
+ pid_t pid;
+
+ if (read_file("/proc/sys/kernel/sched_rt_runtime_us", saved, sizeof(saved)) ||
+ write_file("/proc/sys/kernel", "sched_rt_runtime_us", "-1")) {
+ ksft_print_msg("admission control cannot be turned off\n");
+ return KSFT_SKIP;
+ }
+ mv_state[0] = 0;
+ mv_state[1] = 0;
+ pid = fork_child("a pinned deadline task");
+ if (!pid) {
+ cpu_set_t one;
+
+ CPU_ZERO(&one);
+ CPU_SET(hp_cpu, &one);
+ if (sched_setaffinity(0, sizeof(one), &one) ||
+ set_deadline(0, 50 * NSEC_PER_MSEC, PERIOD)) {
+ mv_state[0] = -1;
+ _exit(1);
+ }
+ mv_state[0] = 1;
+ for (;;)
+ mv_state[1]++;
+ }
+ if (pid < 0 || child_up("a pinned deadline task", 0)) {
+ ret = KSFT_FAIL;
+ goto out;
+ }
+ sleep_ns(PERIOD);
+ if (cpu_online(hp_cpu, 0)) {
+ ksft_print_msg("CPU %d cannot go offline under a pinned deadline task\n", hp_cpu);
+ ret = KSFT_FAIL;
+ goto out;
+ }
+ last = mv_state[1];
+ sleep_ns(3 * PERIOD);
+ if (mv_state[1] == last) {
+ ksft_print_msg("the pinned task does not run after its CPU went offline\n");
+ ret = KSFT_FAIL;
+ }
+out:
+ cpu_up(hp_cpu);
+ reap(pid);
+ if (write_file("/proc/sys/kernel", "sched_rt_runtime_us", saved)) {
+ ksft_print_msg("admission control cannot be turned on again\n");
+ return KSFT_FAIL;
+ }
+ sleep_ns(3 * PERIOD);
+ after = probe(NULL);
+ if (ret == KSFT_PASS && after != base) {
+ ksft_print_msg("a pinned task without admission control: %d probe tasks admitted before, %d after\n",
+ base, after);
+ ret = KSFT_FAIL;
+ }
+ return ret;
+}
+
/*
* Deadline sleepers take most of the free bandwidth. Taking hp_cpu
* offline must then fail with EBUSY. Once the sleepers are gone, the
@@ -4638,6 +4851,303 @@ static int hotplug_refused(void)
return ret;
}
+/*
+ * A deadline task of 40% spins. The case switches it to SCHED_OTHER
+ * before its 0-lag time. Then it takes the CPU that the task is on
+ * offline, before that time too. Past the 0-lag time, with the CPU still
+ * offline, the housekeeping root domain must admit what it admits with a
+ * CPU offline and no task. Once the CPU is back and the task is gone, it
+ * must admit what it did at the start.
+ */
+static int hotplug_departed(void)
+{
+ int base = probe(NULL), base_off, after, cpu, ret = KSFT_PASS;
+ pid_t pid;
+
+ /*
+ * Take the reference with hp_cpu offline and no task of ours. The
+ * task may be on another CPU: the case assumes that any one CPU
+ * offline leaves the same room. The case compares while that CPU
+ * is still offline. Bringing it back rebuilds the accounting,
+ * which would hide an error.
+ */
+ if (cpu_online(hp_cpu, 0)) {
+ ksft_print_msg("offline of CPU %d refused\n", hp_cpu);
+ return KSFT_FAIL;
+ }
+ base_off = probe(NULL);
+ cpu_up(hp_cpu);
+ if (base <= 0 || base_off <= 0) {
+ ksft_print_msg("no room to probe with the CPU online or offline\n");
+ return KSFT_SKIP;
+ }
+ mv_state[0] = 0;
+ mv_state[1] = 0;
+ pid = mover(MV_THROTTLED, 40 * NSEC_PER_MSEC);
+ if (pid < 0 || child_up("a departing deadline task", 0)) {
+ reap(pid);
+ return KSFT_FAIL;
+ }
+ sleep_ns(PERIOD / 2);
+ /* Not asleep: its bandwidth stays until its 0-lag time. */
+ if (set_other(pid))
+ ret = KSFT_FAIL;
+ cpu = task_cpu_of(pid);
+ if (cpu < 0 || !cpu_hotpluggable(cpu)) {
+ cpu_set_t set;
+
+ /* Out of the class now, it may be pinned to hp_cpu. */
+ cpu = hp_cpu;
+ CPU_ZERO(&set);
+ CPU_SET(cpu, &set);
+ if (sched_setaffinity(pid, sizeof(set), &set))
+ ret = KSFT_FAIL;
+ }
+ if (ret == KSFT_PASS && cpu_online(cpu, 0)) {
+ ksft_print_msg("offline of CPU %d refused\n", cpu);
+ ret = KSFT_FAIL;
+ }
+ sleep_ns(3 * PERIOD); /* its 0-lag time, the CPU offline */
+ if (ret == KSFT_PASS) {
+ after = probe(NULL);
+ if (after != base_off) {
+ ksft_print_msg("departed, CPU %d offline: %d admitted, %d expected\n",
+ cpu, after, base_off);
+ ret = KSFT_FAIL;
+ }
+ }
+ cpu_up(cpu);
+ reap(pid);
+ sleep_ns(3 * PERIOD);
+ after = probe(NULL);
+ if (ret == KSFT_PASS && after != base) {
+ ksft_print_msg("departed, CPU back online: %d admitted before, %d after\n",
+ base, after);
+ ret = KSFT_FAIL;
+ }
+ return ret;
+}
+
+#define ZOMBIE_RUNTIME (200 * NSEC_PER_MSEC)
+#define ZOMBIE_PERIOD (2 * NSEC_PER_SEC)
+
+/*
+ * Q is a partition root of two CPUs. The kernel records, with the
+ * bandwidth of a task, a CPU of the root domain that counts it. The steps:
+ * 1. A task enters Q and becomes a deadline task of 200ms every 2s.
+ * 2. It runs 190ms and exits, its 0-lag time about 1.7s ahead. The case
+ * does not reap it yet. A rebuild of the accounting still finds a
+ * zombie in the task list.
+ * 3. total_bw of Q must count the zombie.
+ * 4. The case takes each CPU of Q offline and back, one after the other.
+ * One of them is the CPU recorded with the bandwidth of the zombie.
+ * The rebuild at its offline drops that bandwidth.
+ * 5. Before the 0-lag time, total_bw of Q must be what it was without
+ * the task. No later rebuild may count the zombie again: the release
+ * at the 0-lag time removes nothing by then.
+ * 6. The case reaps the zombie. Past the 0-lag time, total_bw of Q must
+ * still be that value.
+ *
+ * The case runs with the cases in P, from the main process: Q needs the
+ * cpuset controller. Q has two CPUs because the last CPU of a root domain
+ * that holds deadline bandwidth cannot go offline.
+ */
+static int death_zombie_hotplug(void)
+{
+ int cpu[2] = { top_cpu(3), top_cpu(2) }, i, ret = KSFT_PASS;
+ long long base, bw, held, after;
+ char cpus[32];
+ uint64_t t;
+ pid_t pid;
+
+ if (ncpus < 4 || !cpu_hotpluggable(cpu[0]) || !cpu_hotpluggable(cpu[1])) {
+ ksft_print_msg("needs 4 non-isolated CPUs, 2 hotpluggable beside P\n");
+ return KSFT_SKIP;
+ }
+ snprintf(cpus, sizeof(cpus), "%d,%d", cpu[0], cpu[1]);
+ if (make_partition_cpus(cg_q, cpus)) {
+ ksft_print_msg("no partition root Q of CPUs %s\n", cpus);
+ remove_partition(cg_q);
+ return KSFT_SKIP;
+ }
+ base = total_bw_of(cpu[0]);
+ if (base < 0) {
+ ksft_print_msg("no total_bw in /sys/kernel/debug/sched/debug\n");
+ remove_partition(cg_q);
+ return KSFT_SKIP;
+ }
+ bw = (ZOMBIE_RUNTIME << 20) / ZOMBIE_PERIOD; /* as to_ratio() */
+ mv_state[0] = 0;
+ pid = fork_child("a deadline task that exits");
+ if (!pid) {
+ if (enter(cg_q, getpid()) || set_deadline(0, ZOMBIE_RUNTIME, ZOMBIE_PERIOD))
+ _exit(1);
+ /* 10ms left: the 0-lag time is 100ms before the deadline. */
+ spin_ns(ZOMBIE_RUNTIME - 10 * NSEC_PER_MSEC);
+ mv_state[0] = 1;
+ _exit(0);
+ }
+ if (pid < 0 || child_up("a task of Q that exits", 0) ||
+ wait_for(1000, proc_state_of(pid) == 'Z')) {
+ ksft_print_msg("no zombie deadline task\n");
+ reap(pid);
+ remove_partition(cg_q);
+ return KSFT_FAIL;
+ }
+ /* Its 0-lag time is about 1.7s ahead. Check total_bw within 1s. */
+ t = now_ns();
+ held = total_bw_of(cpu[0]);
+ if (held != base + bw) {
+ ksft_print_msg("total_bw of Q: %lld with the zombie, %lld expected\n",
+ held, base + bw);
+ ret = KSFT_FAIL;
+ }
+ for (i = 0; i < 2 && ret == KSFT_PASS; i++) {
+ if (cpu_online(cpu[i], 0) || cpu_online(cpu[i], 1)) {
+ ksft_print_msg("offline and online of CPU %d failed\n", cpu[i]);
+ cpu_up(cpu[i]);
+ ret = KSFT_FAIL;
+ }
+ }
+ after = total_bw_of(cpu[0]);
+ if (ret == KSFT_PASS && now_ns() - t > NSEC_PER_SEC) {
+ ksft_print_msg("the hotplugs took over 1s: the 0-lag time may be past\n");
+ ret = KSFT_SKIP;
+ } else if (ret == KSFT_PASS && after != base) {
+ ksft_print_msg("total_bw of Q after the hotplugs: %lld, %lld expected%s\n",
+ after, base,
+ after == base + bw ? ": the zombie counted again" : "");
+ ret = KSFT_FAIL;
+ }
+ reap(pid);
+ sleep_ns(ZOMBIE_PERIOD); /* past its 0-lag time */
+ after = total_bw_of(cpu[0]);
+ if (ret == KSFT_PASS && after != base) {
+ ksft_print_msg("total_bw of Q past the 0-lag time: %lld, %lld expected\n",
+ after, base);
+ ret = KSFT_FAIL;
+ }
+ remove_partition(cg_q);
+ return ret;
+}
+
+/*
+ * Fork the owner of pi_departed_moved(). It takes shm_lock, becomes a
+ * deadline task and runs most of its runtime. It sets mv_state[0] to 1
+ * and sleeps 300ms. Then it sets mv_state[1] to 1 and sleeps with the
+ * lock. Return its pid, or -1.
+ */
+static pid_t departed_owner(void)
+{
+ pid_t pid = fork_child("the owner of the lock");
+
+ if (!pid) {
+ if (pthread_mutex_lock(shm_lock) ||
+ set_deadline(0, ZOMBIE_RUNTIME, ZOMBIE_PERIOD)) {
+ mv_state[0] = -1;
+ _exit(1);
+ }
+ /* 20ms left: the 0-lag time is 200ms before the deadline. */
+ spin_ns(ZOMBIE_RUNTIME - 20 * NSEC_PER_MSEC);
+ mv_state[0] = 1;
+ sleep_ns(300 * NSEC_PER_MSEC);
+ mv_state[1] = 1;
+ for (;;)
+ pause();
+ }
+ return pid;
+}
+
+/*
+ * The owner is a deadline task of 200ms every 2s, outside P. It holds
+ * shm_lock, runs 180ms and sleeps, its 0-lag time about 1.6s ahead. The
+ * steps:
+ * 1. The case switches the sleeping owner to SCHED_OTHER. The
+ * housekeeping root domain keeps its bandwidth until the 0-lag time.
+ * 2. The case moves the owner into P. The owner wakes up and runs there.
+ * 3. A deadline waiter of 5ms every 50ms, outside P, blocks on shm_lock
+ * and boosts the owner.
+ * 4. The case kills the waiter, which ends the boost.
+ *
+ * After step 4, total_bw of the housekeeping root domain must still count
+ * the bandwidth of the owner. total_bw of P never counted it and must be
+ * unchanged: the end of the boost must not subtract it there. Past the
+ * 0-lag time, both must be what they were at the start.
+ */
+static int pi_departed_moved(void)
+{
+ int p_cpu = top_cpu(1), r_cpu = online[0], ret = KSFT_PASS;
+ long long base_p, base_r, bw, in_p, in_r;
+ pid_t owner, waiter = -1;
+
+ base_p = total_bw_of(p_cpu);
+ base_r = total_bw_of(r_cpu);
+ if (base_p < 0 || base_r < 0) {
+ ksft_print_msg("no total_bw in /sys/kernel/debug/sched/debug\n");
+ return KSFT_SKIP;
+ }
+ if (shm_lock_init())
+ return KSFT_SKIP;
+ bw = (ZOMBIE_RUNTIME << 20) / ZOMBIE_PERIOD; /* as to_ratio() */
+ mv_state[1] = 0;
+ owner = departed_owner();
+ if (owner < 0)
+ return KSFT_FAIL;
+ if (child_up("the owner of the lock", 0)) {
+ reap(owner);
+ return KSFT_FAIL;
+ }
+ /* The owner is asleep, about 1.6s before its 0-lag time. */
+ sleep_ns(20 * NSEC_PER_MSEC);
+ if (set_other(owner) || enter(cg_p, owner)) {
+ ksft_print_msg("the owner cannot leave the class and enter P\n");
+ ret = KSFT_FAIL;
+ goto out;
+ }
+ if (wait_state(&mv_state[1], 1) || task_cpu_of(owner) != p_cpu) {
+ ksft_print_msg("the owner did not run in P\n");
+ ret = KSFT_FAIL;
+ goto out;
+ }
+ waiter = lock_waiter();
+ if (waiter < 0) {
+ ret = KSFT_FAIL;
+ goto out;
+ }
+ if (wait_state(&mv_state[2], 1)) {
+ ksft_print_msg("no waiter\n");
+ ret = KSFT_FAIL;
+ goto out;
+ }
+ sleep_ns(50 * NSEC_PER_MSEC); /* blocked, boosting the owner */
+ reap(waiter);
+ waiter = -1;
+ sleep_ns(150 * NSEC_PER_MSEC); /* the waiter is past its 0-lag time */
+ in_p = total_bw_of(p_cpu);
+ in_r = total_bw_of(r_cpu);
+ if (in_p != base_p || in_r != base_r + bw) {
+ ksft_print_msg("after the boost: total_bw of P %lld, %lld expected\n",
+ in_p, base_p);
+ ksft_print_msg("after the boost: total_bw outside %lld, %lld expected\n",
+ in_r, base_r + bw);
+ ret = KSFT_FAIL;
+ }
+out:
+ reap(waiter);
+ reap(owner);
+ sleep_ns(ZOMBIE_PERIOD); /* past the owner's 0-lag time */
+ in_p = total_bw_of(p_cpu);
+ in_r = total_bw_of(r_cpu);
+ if (ret == KSFT_PASS && (in_p != base_p || in_r != base_r)) {
+ ksft_print_msg("past the 0-lag time: total_bw of P %lld, %lld expected\n",
+ in_p, base_p);
+ ksft_print_msg("past the 0-lag time: total_bw outside %lld, %lld expected\n",
+ in_r, base_r);
+ ret = KSFT_FAIL;
+ }
+ return ret;
+}
+
/*
* The bandwidth of a fair server must leave the root domain with its CPU
* and come back with it, whatever its size. Probe the housekeeping root
@@ -4743,6 +5253,134 @@ static void report(const char *name, int ret)
ksft_test_result_fail("%s\n", name);
}
+/* sched_setaffinity() against sched_setattr(), with no P */
+
+/*
+ * The state that affinity_racer() shares with the case. pid and one are
+ * its target and its mask. go and done count the rounds. ret is the
+ * result of its last call.
+ */
+static struct {
+ pid_t pid;
+ cpu_set_t one;
+ atomic_int go;
+ atomic_int done;
+ atomic_int stop;
+ int ret;
+} aff;
+
+/*
+ * Give aff.pid the affinity aff.one, once for each new value of aff.go.
+ * Return when aff.stop is set.
+ */
+static void *affinity_racer(void *arg)
+{
+ int round;
+
+ for (round = 1; ; round++) {
+ while (aff.go < round && !aff.stop)
+ ;
+ if (aff.stop)
+ break;
+ aff.ret = sched_setaffinity(aff.pid, sizeof(aff.one), &aff.one) ? -errno : 0;
+ aff.done = round;
+ }
+ return arg;
+}
+
+/*
+ * Race sched_setaffinity() against sched_setattr() on a sleeping task.
+ * One thread gives the task an affinity of one CPU. Another one makes it
+ * a deadline task. A deadline task must be allowed on its whole root
+ * domain, so at most one of the two calls may succeed:
+ * - sched_setattr() fails with EPERM if the affinity came first;
+ * - sched_setaffinity() fails with EBUSY if the policy came first.
+ *
+ * Do it 1000 times. Each round starts with the task back in SCHED_OTHER
+ * with its full affinity. Once the task is gone, the free bandwidth must
+ * be what it was. The case needs no cpuset: it runs without P, with the
+ * hotplug cases.
+ */
+static int affinity_vs_setattr(void)
+{
+ const char *name = "affinity_vs_setattr";
+ int base, i, cpu, both = 0, ret = KSFT_PASS;
+ cpu_set_t all;
+ pthread_t t;
+ pid_t pid;
+
+ base = probe(NULL);
+ pid = fork_child(name);
+ if (!pid) {
+ for (;;)
+ pause();
+ }
+ if (pid < 0)
+ return KSFT_FAIL;
+ /*
+ * Wait until the task sleeps in pause(). A task that still runs
+ * when it becomes a deadline task can overrun its 2us. Its
+ * bandwidth then stays counted until its 0-lag time, seconds after
+ * it is gone, and the next cases would see it.
+ */
+ if (wait_for(1000, proc_state_of(pid) == 'S') ||
+ sched_getaffinity(pid, sizeof(all), &all)) {
+ reap(pid);
+ return KSFT_FAIL;
+ }
+ CPU_ZERO(&aff.one);
+ for (cpu = 0; cpu < CPU_SETSIZE; cpu++) {
+ if (CPU_ISSET(cpu, &all)) {
+ CPU_SET(cpu, &aff.one);
+ break;
+ }
+ }
+ aff.pid = pid;
+ aff.go = 0;
+ aff.done = 0;
+ aff.stop = 0;
+ if (pthread_create(&t, NULL, affinity_racer, NULL)) {
+ reap(pid);
+ return KSFT_FAIL;
+ }
+
+ for (i = 1; i <= 1000 && ret == KSFT_PASS; i++) {
+ int err;
+
+ if (set_other(pid) || sched_setaffinity(pid, sizeof(all), &all)) {
+ ksft_print_msg("%s: the task cannot be reset: %s\n", name, strerror(errno));
+ ret = KSFT_FAIL;
+ break;
+ }
+ aff.go = i;
+ err = set_deadline(pid, 2 * NSEC_PER_USEC, PERIOD);
+ while (aff.done < i)
+ ;
+ if (!err && !aff.ret) {
+ both++;
+ } else if ((err && err != -EPERM) || (aff.ret && aff.ret != -EBUSY)) {
+ ksft_print_msg("%s: round %d: sched_setattr: %s, sched_setaffinity: %s\n",
+ name, i, strerror(-err), strerror(-aff.ret));
+ ret = KSFT_FAIL;
+ }
+ }
+ aff.stop = 1;
+ join(t);
+ if (both) {
+ ksft_print_msg("%s: %d rounds of %d: a deadline task with an affinity of one CPU\n",
+ name, both, i - 1);
+ ret = KSFT_FAIL;
+ }
+ /* A deadline task of 2us every 100ms would take seconds to exit. */
+ if (set_other(pid))
+ ksft_print_msg("%s: the task stays a deadline task\n", name);
+ reap(pid);
+ sleep_ns(3 * PERIOD);
+ if (ret == KSFT_PASS && admits(base, "after the task is gone"))
+ ret = KSFT_FAIL;
+ return ret;
+}
+
/* The tables of the cases, and main() */
enum where {
@@ -4760,6 +5398,7 @@ static const struct p_case {
{ "pi_other_owner_asleep", pi_other_owner_asleep, MEASURED_IN_P },
{ "pi_unlock_running", pi_unlock_running, MEASURED_IN_P },
{ "pi_chain_two_owners", pi_chain_two_owners, MEASURED_IN_P },
+ { "pi_departed_moved", pi_departed_moved, FROM_OUTSIDE },
{ "params_asleep", params_asleep, IN_P },
{ "params_non_contending", params_non_contending, MEASURED_IN_P },
{ "params_switched_asleep", params_switched_asleep, MEASURED_IN_P },
@@ -4790,6 +5429,7 @@ static const struct p_case {
{ "death_pi_deadline_owner", death_pi_deadline_owner, MEASURED_IN_P },
{ "death_group", death_group, MEASURED_IN_P },
{ "death_zombie_switched", death_zombie_switched, FROM_OUTSIDE },
+ { "death_zombie_hotplug", death_zombie_hotplug, FROM_OUTSIDE },
{ "yield_switched_out", yield_switched_out, IN_P },
{ "yield_killed", yield_killed, MEASURED_IN_P },
{ "overrun_caught", overrun_caught, IN_P },
@@ -4808,9 +5448,15 @@ static const struct p_case {
static const struct hp_case {
const char *name;
int (*fn)(void);
+ int no_hp; /* takes no CPU offline */
} hp_cases[] = {
+ { "hotplug_sleepers", hotplug_sleepers },
+ { "hotplug_sleeper_switched", hotplug_sleeper_switched },
+ { "hotplug_no_admission_pinned", hotplug_no_admission_pinned },
{ "hotplug_refused", hotplug_refused },
+ { "hotplug_departed", hotplug_departed },
{ "hotplug_server", hotplug_server },
+ { "affinity_vs_setattr", affinity_vs_setattr, 1 },
};
#define N_HP_CASES ((int)ARRAY_SIZE(hp_cases))
@@ -4949,7 +5595,7 @@ int main(void)
if (hp_cpu < 0)
ksft_print_msg("no CPU hotplug: hotplug cases skipped\n");
for (i = 0; i < N_HP_CASES; i++) {
- if (hp_cpu < 0) {
+ if (hp_cpu < 0 && !hp_cases[i].no_hp) {
ksft_test_result_skip("%s\n", hp_cases[i].name);
continue;
}
--
2.47.3
^ permalink raw reply related [flat|nested] 35+ messages in thread
* [PATCH v8 07/33] sched/deadline: Move a task's bandwidth only where room was made
2026-11-19 20:15 [PATCH v8 00/33] sched/deadline: bandwidth accounting fixes and selftests Mathieu Desnoyers
` (5 preceding siblings ...)
2026-11-19 20:15 ` [PATCH v8 06/33] selftests/sched: Test the root domain rebuild in deadline_bw_test Mathieu Desnoyers
@ 2026-11-19 20:15 ` Mathieu Desnoyers
2026-11-19 20:15 ` [PATCH v8 08/33] cgroup/cpuset: Wait for an attach before rebuilding the sched domains Mathieu Desnoyers
` (25 subsequent siblings)
32 siblings, 0 replies; 35+ messages in thread
From: Mathieu Desnoyers @ 2026-11-19 20:15 UTC (permalink / raw)
To: juri.lelli, peterz, mingo, vincent.guittot
Cc: Mathieu Desnoyers, tglx, will, boqun, dietmar.eggemann, rostedt,
bsegall, mgorman, vschneid, kprateek.nayak, luca.abeni,
mathieu.poirier, longman, ridong.chen, tj, hannes, mkoutny, shuah,
bigeasy, clrkwllms, parri.andrea, m.szyprowski, kayracizmeci,
weihengqiuu, linux-kernel, cgroups, linux-kselftest,
linux-rt-devel, stable
Commit 7f51412a415d ("sched/deadline: Fix bandwidth check/update when
migrating tasks between exclusive cpusets") moves a deadline task's
bandwidth in two halves. cpuset_can_attach() reserves it in the
destination root domain. set_cpus_allowed_dl() then frees it in the
source, when the task's affinity changes. Its comment relies on the
first half: "We already made space for us in the destination domain".
But set_cpus_allowed_dl() runs on every change of affinity. It frees the
bandwidth whenever the new mask leaves the task's root domain, with or
without a reservation. sched_setaffinity() gets there too, if the task's
cpuset spans two root domains. With isolcpus="domain", any cpuset that
is not a partition root can hold isolated CPUs and other CPUs, which are
in two root domains. With admission control on, __sched_setaffinity()
refuses such a mask. With sched_rt_runtime_us at -1, nothing refuses it.
The bandwidth is then freed in the source and added nowhere: no root
domain counts it any more. The destination would underflow when the task
leaves.
Fix this by flagging the affinity changes that come with a reservation,
as SCA_DL_BW_RESERVED. cpuset_attach_task() sets it, through the new
set_cpus_allowed_bw_reserved(). set_cpus_allowed_dl() moves the
bandwidth only with the flag. Without it, the bandwidth stays where it
is counted, until the next rebuild of the accounting counts it anew.
The flag also skips the early exit of __set_cpus_allowed_ptr_locked()
for a mask that does not change. The task may already have that mask
from a change that moved no bandwidth, such as a sched_setaffinity()
without admission control.
The attach still flags a task that sched_setattr() made a deadline task
after cpuset_can_attach(), with nothing reserved for it. "sched: Make
deadline policy and affinity changes wait for a cpuset attach" fixes
that.
No case of deadline_bw_test fails before this patch and passes with it.
no_admission_affinity_moves, added later in this series, covers this
path together with "sched/deadline: Move a task's bandwidth to the root
domain it runs in".
Fixes: 7f51412a415d ("sched/deadline: Fix bandwidth check/update when migrating tasks between exclusive cpusets")
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Cc: stable@vger.kernel.org
---
include/linux/sched.h | 2 ++
kernel/cgroup/cpuset.c | 13 ++++++++++++-
kernel/sched/core.c | 26 +++++++++++++++++++++++++-
kernel/sched/deadline.c | 14 +++++++++-----
kernel/sched/sched.h | 2 ++
5 files changed, 50 insertions(+), 7 deletions(-)
diff --git a/include/linux/sched.h b/include/linux/sched.h
index 4098b4e0f67e..e88616448b85 100644
--- a/include/linux/sched.h
+++ b/include/linux/sched.h
@@ -1966,6 +1966,8 @@ extern void set_cpus_allowed_force(struct task_struct *p, const struct cpumask *
* Return: zero if successful, or a negative error code
*/
extern int set_cpus_allowed_ptr(struct task_struct *p, const struct cpumask *new_mask);
+extern int set_cpus_allowed_bw_reserved(struct task_struct *p,
+ const struct cpumask *new_mask);
extern int dup_user_cpus_ptr(struct task_struct *dst, struct task_struct *src, int node);
extern void release_user_cpus_ptr(struct task_struct *p);
extern int dl_task_check_affinity(struct task_struct *p, const struct cpumask *mask);
diff --git a/kernel/cgroup/cpuset.c b/kernel/cgroup/cpuset.c
index 7628cf77bdfd..194a438e25c3 100644
--- a/kernel/cgroup/cpuset.c
+++ b/kernel/cgroup/cpuset.c
@@ -3261,6 +3261,7 @@ static cpumask_var_t cpus_attach;
static void cpuset_attach_task(struct cpuset *cs, struct task_struct *task)
{
+ bool reserved = false;
struct mm_struct *mm;
lockdep_assert_cpuset_lock_held();
@@ -3270,11 +3271,21 @@ static void cpuset_attach_task(struct cpuset *cs, struct task_struct *task)
else
cpumask_andnot(cpus_attach, task_cpu_possible_mask(task),
subpartitions_cpus);
+ /*
+ * cpuset_reserve_dl_bw() made room in @cs for the bandwidth of the
+ * deadline tasks that cpuset_can_attach() counted. Flag the move, so
+ * that the bandwidth of this task moves with it.
+ */
+ if (dl_task(task) && dl_task_needs_bw_move(task, cs->effective_cpus))
+ reserved = true;
/*
* can_attach beforehand should guarantee that this doesn't
* fail. TODO: have a better way to handle failure here
*/
- WARN_ON_ONCE(set_cpus_allowed_ptr(task, cpus_attach));
+ if (reserved)
+ WARN_ON_ONCE(set_cpus_allowed_bw_reserved(task, cpus_attach));
+ else
+ WARN_ON_ONCE(set_cpus_allowed_ptr(task, cpus_attach));
if (cpuset_v2() && !attach_ctx.mems_updated)
return;
diff --git a/kernel/sched/core.c b/kernel/sched/core.c
index 32fe889e02fb..bd5f3dfa50ac 100644
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -3207,7 +3207,15 @@ static int __set_cpus_allowed_ptr_locked(struct task_struct *p,
}
if (!(ctx->flags & SCA_MIGRATE_ENABLE)) {
- if (cpumask_equal(&p->cpus_mask, ctx->new_mask)) {
+ /*
+ * A move with SCA_DL_BW_RESERVED must reach
+ * set_cpus_allowed_dl(), even if the mask does not change.
+ * @p may have this mask from a change that moved no
+ * bandwidth, such as sched_setaffinity() without admission
+ * control.
+ */
+ if (cpumask_equal(&p->cpus_mask, ctx->new_mask) &&
+ !(ctx->flags & SCA_DL_BW_RESERVED)) {
if (ctx->flags & SCA_USER)
swap(p->user_cpus_ptr, ctx->user_mask);
goto out;
@@ -3280,6 +3288,22 @@ int set_cpus_allowed_ptr(struct task_struct *p, const struct cpumask *new_mask)
}
EXPORT_SYMBOL_GPL(set_cpus_allowed_ptr);
+/*
+ * Like set_cpus_allowed_ptr(), for a caller that has made room for the
+ * deadline bandwidth of @p in the root domain of @new_mask. Move that
+ * bandwidth there with @p. Return 0 on success, or a negative error
+ * code.
+ */
+int set_cpus_allowed_bw_reserved(struct task_struct *p, const struct cpumask *new_mask)
+{
+ struct affinity_context ac = {
+ .new_mask = new_mask,
+ .flags = SCA_DL_BW_RESERVED,
+ };
+
+ return __set_cpus_allowed_ptr(p, &ac);
+}
+
/*
* Change a given task's CPU affinity to the intersection of its current
* affinity mask and @subset_mask, writing the resulting mask to @new_mask.
diff --git a/kernel/sched/deadline.c b/kernel/sched/deadline.c
index 40d87cd31d33..56d1f6636edc 100644
--- a/kernel/sched/deadline.c
+++ b/kernel/sched/deadline.c
@@ -3612,12 +3612,16 @@ static void set_cpus_allowed_dl(struct task_struct *p,
WARN_ON_ONCE(!dl_task(p));
/*
- * Migrating a SCHED_DEADLINE task between exclusive
- * cpusets (different root_domains) entails a bandwidth
- * update. We already made space for us in the destination
- * domain (see cpuset_can_attach()).
+ * Migrating a SCHED_DEADLINE task between exclusive cpusets
+ * (different root_domains) entails a bandwidth update. Move the
+ * bandwidth only if the caller made room for it in the destination,
+ * as SCA_DL_BW_RESERVED says. Only cpuset_attach_task() sets it.
+ * Otherwise, the bandwidth stays where it is counted, until the next
+ * rebuild of the accounting. Moving it without room would leave no
+ * root domain counting it.
*/
- if (dl_task_needs_bw_move(p, ctx->new_mask)) {
+ if ((ctx->flags & SCA_DL_BW_RESERVED) &&
+ dl_task_needs_bw_move(p, ctx->new_mask)) {
unsigned int src_cpu = dl_task_root_domain_cpu(p);
struct dl_bw *src_dl_b = &cpu_rq(src_cpu)->rd->dl_bw;
unsigned int cpu;
diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h
index 20ad13f388cb..61c73c7d3f27 100644
--- a/kernel/sched/sched.h
+++ b/kernel/sched/sched.h
@@ -2973,6 +2973,8 @@ extern struct task_struct *pick_task_idle(struct rq *rq, struct rq_flags *rf);
#define SCA_MIGRATE_ENABLE 0x04
#define SCA_USER 0x08
#define SCA_DL_CHECK 0x10
+/* The caller made room for the deadline bandwidth in the new root domain. */
+#define SCA_DL_BW_RESERVED 0x20
extern void update_group_capacity(struct sched_domain *sd, int cpu);
--
2.47.3
^ permalink raw reply related [flat|nested] 35+ messages in thread
* [PATCH v8 08/33] cgroup/cpuset: Wait for an attach before rebuilding the sched domains
2026-11-19 20:15 [PATCH v8 00/33] sched/deadline: bandwidth accounting fixes and selftests Mathieu Desnoyers
` (6 preceding siblings ...)
2026-11-19 20:15 ` [PATCH v8 07/33] sched/deadline: Move a task's bandwidth only where room was made Mathieu Desnoyers
@ 2026-11-19 20:15 ` Mathieu Desnoyers
2026-11-19 20:15 ` [PATCH v8 09/33] sched: Make deadline policy and affinity changes wait for a cpuset attach Mathieu Desnoyers
` (24 subsequent siblings)
32 siblings, 0 replies; 35+ messages in thread
From: Mathieu Desnoyers @ 2026-11-19 20:15 UTC (permalink / raw)
To: juri.lelli, peterz, mingo, vincent.guittot
Cc: Mathieu Desnoyers, tglx, will, boqun, dietmar.eggemann, rostedt,
bsegall, mgorman, vschneid, kprateek.nayak, luca.abeni,
mathieu.poirier, longman, ridong.chen, tj, hannes, mkoutny, shuah,
bigeasy, clrkwllms, parri.andrea, m.szyprowski, kayracizmeci,
weihengqiuu, linux-kernel, cgroups, linux-kselftest,
linux-rt-devel, stable
cpuset_can_attach() reserves the bandwidth of the deadline tasks it
moves in the destination root domain. cpuset_attach() moves their
bandwidth later, on the strength of that reservation.
Commit 75f7a25ec6bb ("cgroup/cpuset: Prevent race between task attach
and cpuset state change") lets cpuset_attach() rely on what
cpuset_can_attach() set up. The writes of cpus, mems and cpus.partition
wait for an attach in progress, in wait_attach_done_lock(). CPU hotplug
cannot run in between either: it takes cpus_write_lock, and the attach
holds cpus_read_lock throughout.
But a rebuild of the sched domains does not wait. A write of
sched_rt_runtime_us and the energy model rebuild them through
rebuild_sched_domains(). The cgroup v1 writes of sched_load_balance and
sched_relax_domain_level, and hk_sd_workfn(), use cpuset_full_lock().
In this example, a rebuild drops the reservation. T is a deadline task
asleep in root domain R. Thread A moves T to a partition root P, which
has a root domain of its own. Thread B writes sched_rt_runtime_us:
A B
cpuset_can_attach()
takes cpuset_mutex
cpuset_reserve_dl_bw()
P's total_bw += T's bandwidth
drops cpuset_mutex
rebuild_sched_domains()
takes cpuset_mutex
dl_rebuild_rd_accounting()
clears R's and P's total_bw
counts T in R (not moved yet)
drops cpuset_mutex
cpuset_attach()
takes cpuset_mutex
cpuset_attach_task(T)
set_cpus_allowed_dl(T)
R's total_bw -= T's bandwidth
records a CPU of P
P now lacks the bandwidth of T, and admits deadline tasks beyond its
capacity. When T leaves P, P's total_bw underflows. In testing, the
race left a destination admitting 3 tasks of 10% too many, or P
admitting no task at all.
Fix this by waiting for an attach in progress in
rebuild_sched_domains_cpuslocked() and in cpuset_full_lock(). The
latter then takes its locks in the order of the writes of cpus and
cpus.partition. A rebuild from CPU hotplug finds no attach in progress.
The online, offline and kill of a cpuset never wait: they hold
cgroup_mutex, which the attach holds throughout.
The move_vs_rebuild case, added to deadline_bw_test later in this
series, tests this race. Without this patch, it fails in 5 of 8 VMs with
8 CPUs. With it, the case passes in 8 of 8. The next patch needs this
wait too: without it, it would give back a reservation that a rebuild
dropped already.
The Sashiko AI reviews of this series found the race and the cgroup v1
writes.
Fixes: 2ef269ef1ac0 ("cgroup/cpuset: Free DL BW in case can_attach() fails")
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Reviewed-by: Juri Lelli <juri.lelli@redhat.com>
Cc: stable@vger.kernel.org
---
kernel/cgroup/cpuset.c | 16 +++++++++++++---
1 file changed, 13 insertions(+), 3 deletions(-)
diff --git a/kernel/cgroup/cpuset.c b/kernel/cgroup/cpuset.c
index 194a438e25c3..51b6be460794 100644
--- a/kernel/cgroup/cpuset.c
+++ b/kernel/cgroup/cpuset.c
@@ -296,17 +296,20 @@ void lockdep_assert_cpuset_lock_held(void)
lockdep_assert_held(&cpuset_mutex);
}
+static void wait_attach_done_lock(void) __acquires(&cpuset_mutex);
+
/**
* cpuset_full_lock - Acquire full protection for cpuset modification
*
* Takes both CPU hotplug read lock (cpus_read_lock()) and cpuset mutex
- * to safely modify cpuset data.
+ * to safely modify cpuset data. Waits for an attach in progress before
+ * it takes cpuset mutex.
*/
void cpuset_full_lock(void)
{
mutex_lock(&cpuset_top_mutex);
cpus_read_lock();
- mutex_lock(&cpuset_mutex);
+ wait_attach_done_lock();
}
void cpuset_full_unlock(void)
@@ -1005,7 +1008,14 @@ void rebuild_sched_domains_locked(void)
static void rebuild_sched_domains_cpuslocked(void)
{
- mutex_lock(&cpuset_mutex);
+ /*
+ * Wait for an attach in progress. cpuset_can_attach() has reserved
+ * the deadline bandwidth of the moving tasks in their new root
+ * domain, and cpuset_attach() moves it on the strength of that
+ * reservation. A rebuild of the accounting in between would drop
+ * the reservation.
+ */
+ wait_attach_done_lock();
rebuild_sched_domains_locked();
mutex_unlock(&cpuset_mutex);
}
--
2.47.3
^ permalink raw reply related [flat|nested] 35+ messages in thread
* [PATCH v8 09/33] sched: Make deadline policy and affinity changes wait for a cpuset attach
2026-11-19 20:15 [PATCH v8 00/33] sched/deadline: bandwidth accounting fixes and selftests Mathieu Desnoyers
` (7 preceding siblings ...)
2026-11-19 20:15 ` [PATCH v8 08/33] cgroup/cpuset: Wait for an attach before rebuilding the sched domains Mathieu Desnoyers
@ 2026-11-19 20:15 ` Mathieu Desnoyers
2026-11-19 20:15 ` [PATCH v8 10/33] cgroup/cpuset: Give back the bandwidth an attach did not move Mathieu Desnoyers
` (23 subsequent siblings)
32 siblings, 0 replies; 35+ messages in thread
From: Mathieu Desnoyers @ 2026-11-19 20:15 UTC (permalink / raw)
To: juri.lelli, peterz, mingo, vincent.guittot
Cc: Mathieu Desnoyers, tglx, will, boqun, dietmar.eggemann, rostedt,
bsegall, mgorman, vschneid, kprateek.nayak, luca.abeni,
mathieu.poirier, longman, ridong.chen, tj, hannes, mkoutny, shuah,
bigeasy, clrkwllms, parri.andrea, m.szyprowski, kayracizmeci,
weihengqiuu, linux-kernel, cgroups, linux-kselftest,
linux-rt-devel, stable
__sched_setscheduler() takes cpuset_mutex for a change that involves
SCHED_DEADLINE. Commit 710da3c8ea7d ("sched/core: Prevent race condition
between cpuset and __sched_setscheduler()") added that lock. It keeps
the cpuset code from changing the root domains while the deadline
admission test runs.
But a cpuset attach drops cpuset_mutex between cpuset_can_attach() and
cpuset_attach(). A sched_setattr() can run in between. Take a process
in the root cgroup. It has deadline threads, and a thread T that is not
a deadline task. The process is moved into a partition root P:
echo PID > P/cgroup.procs sched_setattr(T, SCHED_DEADLINE)
cpuset_can_attach()
lock cpuset_mutex
reserve room in P for the deadline
threads: T is not one of them
unlock cpuset_mutex
lock cpuset_mutex
admit T in the root domain it
is still in
unlock cpuset_mutex
cpuset_attach()
lock cpuset_mutex
move T to P, where nothing is
reserved for it
unlock cpuset_mutex
The attach still moves the bandwidth of T. It is freed in the root
domain T left, and P has no room reserved to count it. No root domain
counts it any more, and P can admit deadline tasks beyond its capacity.
In testing, P was a one-CPU partition root, which admits 95%. The
process had two sleeping deadline threads of 30% each, and
sched_setattr() made T a deadline task of 40%. The move and the
sched_setattr() both succeeded, with no room in P for the three. With
threads of 20% and a T of 30%, P then admitted five more tasks of 10%,
where it had room for two.
Fix this by taking cpuset_mutex in __sched_setscheduler() only when no
attach is in progress. cpuset_partition_write() and
cpuset_hotplug_update_tasks() already wait for an attach that way. A
deadline policy change then comes before the attach, and is part of its
reservation, or after it.
With this patch, and no room in P for the three, the two never both
succeed. If the sched_setattr() comes first, the move is refused with
EBUSY. If it comes after, T has its new CPUs and has not run on them.
The call is then refused with EPERM, as for any sleeping task moved to
another partition.
sched_setaffinity() on a deadline task can run in between as well. A
later patch of this series reserves the bandwidth of the task on one CPU
of its new affinity, and records that CPU with the task. It is
"cgroup/cpuset: Reserve the bandwidth in the root domain a task moves
to". A new affinity in between could leave that CPU out. cpuset_attach()
would then move no bandwidth, and the task would stay counted in the
root domain it left. So sched_setaffinity() waits the same way when the
task is a deadline task.
sched_setaffinity() tests the policy with no lock, before it decides to
wait. A sched_setattr() can make the task a deadline task right after
that test. So test the policy again under the runqueue lock, where the
mask changes. If the task is a deadline task there and the caller did
not wait, the call starts over, and waits. The new SCA_DL_NOWAIT flag
asks for that test. After a race with a cpuset update,
__sched_setaffinity() sets the mask a second time, within the cpuset.
That second change waits for an attach too, so that the test cannot
skip it. It reads the cpuset mask again after the wait, since the
attach can move the task.
For user-space, these calls now wait until a cpuset attach in progress
ends:
- sched_setattr() and sched_setscheduler() on a deadline task
- sched_setattr() that makes a task a deadline task
- sched_setaffinity() on a deadline task
The attach can be the move of any process. The wait is not
interruptible: a signal does not end it. Outside that window, the
results of these calls are unchanged.
Three cases, added to deadline_bw_test later in this series, test this.
move_became_deadline and move_became_deadline_no_room fail without this
patch. move_affinity_changed fails once the bandwidth is reserved on one
CPU, if sched_setaffinity() does not wait.
A fourth case, move_affinity_became_deadline, covers the second test of
the policy. Its window is a few instructions wide. A delay added after
the first test, for testing only, widens it. The case then fails without
the second test in 3 guests of 3. It passes with it.
Fixes: 710da3c8ea7d ("sched/core: Prevent race condition between cpuset and __sched_setscheduler()")
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Cc: stable@vger.kernel.org
---
include/linux/cpuset.h | 2 ++
kernel/cgroup/cpuset.c | 11 ++++++++
kernel/sched/core.c | 10 +++++++
kernel/sched/sched.h | 2 ++
kernel/sched/syscalls.c | 59 +++++++++++++++++++++++++++++++++++------
5 files changed, 76 insertions(+), 8 deletions(-)
diff --git a/include/linux/cpuset.h b/include/linux/cpuset.h
index db1cc2fc0324..5d0c01a25354 100644
--- a/include/linux/cpuset.h
+++ b/include/linux/cpuset.h
@@ -73,6 +73,7 @@ extern void cpuset_init_smp(void);
extern void cpuset_force_rebuild(void);
extern void cpuset_update_active_cpus(void);
extern void cpuset_lock(void);
+extern void cpuset_lock_no_attach(void);
extern void cpuset_unlock(void);
extern void lockdep_assert_cpuset_lock_held(void);
extern void cpuset_cpus_allowed_locked(struct task_struct *p, struct cpumask *mask);
@@ -192,6 +193,7 @@ static inline void cpuset_update_active_cpus(void)
}
static inline void cpuset_lock(void) { }
+static inline void cpuset_lock_no_attach(void) { }
static inline void cpuset_unlock(void) { }
static inline void lockdep_assert_cpuset_lock_held(void) { }
diff --git a/kernel/cgroup/cpuset.c b/kernel/cgroup/cpuset.c
index 51b6be460794..ce9c57617f49 100644
--- a/kernel/cgroup/cpuset.c
+++ b/kernel/cgroup/cpuset.c
@@ -378,6 +378,17 @@ static void wait_attach_done_lock(void)
}
}
+/**
+ * cpuset_lock_no_attach - Acquire cpuset_mutex with no task attach in progress
+ *
+ * A task attach drops cpuset_mutex between cpuset_can_attach() and
+ * cpuset_attach(). Use this to keep a change out of that window.
+ */
+void cpuset_lock_no_attach(void)
+{
+ wait_attach_done_lock();
+}
+
static inline void check_insane_mems_config(nodemask_t *nodes)
{
if (!cpusets_insane_config() &&
diff --git a/kernel/sched/core.c b/kernel/sched/core.c
index bd5f3dfa50ac..c0a714dde425 100644
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -3195,6 +3195,16 @@ static int __set_cpus_allowed_ptr_locked(struct task_struct *p,
goto out;
}
+ /*
+ * sched_setaffinity() tested the policy with no lock, and did not
+ * wait for a cpuset attach in progress. @p became a deadline task
+ * since: let the caller wait and try again.
+ */
+ if ((ctx->flags & SCA_DL_NOWAIT) && task_has_dl_policy(p)) {
+ ret = -EAGAIN;
+ goto out;
+ }
+
/*
* Check under the same lock that __sched_setscheduler() holds for
* its admission test, so that p cannot become SCHED_DEADLINE
diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h
index 61c73c7d3f27..f4cd35811b96 100644
--- a/kernel/sched/sched.h
+++ b/kernel/sched/sched.h
@@ -2975,6 +2975,8 @@ extern struct task_struct *pick_task_idle(struct rq *rq, struct rq_flags *rf);
#define SCA_DL_CHECK 0x10
/* The caller made room for the deadline bandwidth in the new root domain. */
#define SCA_DL_BW_RESERVED 0x20
+/* The caller did not wait for a cpuset attach: -EAGAIN for a deadline task. */
+#define SCA_DL_NOWAIT 0x40
extern void update_group_capacity(struct sched_domain *sd, int cpu);
diff --git a/kernel/sched/syscalls.c b/kernel/sched/syscalls.c
index 6562581721ed..ae595a4e7a8f 100644
--- a/kernel/sched/syscalls.c
+++ b/kernel/sched/syscalls.c
@@ -564,10 +564,15 @@ int __sched_setscheduler(struct task_struct *p,
* information. Do not take cpuset_mutex when pi is unset: the
* caller is normalize_rt_tasks(), which holds tasklist_lock and
* cannot sleep.
+ *
+ * Wait for a task attach in progress. cpuset_can_attach() reserved
+ * the bandwidth of the deadline tasks it saw, and cpuset_attach()
+ * moves them. A task that enters or leaves the class in between
+ * would not match that reservation.
*/
if (pi && (dl_policy(policy) || dl_policy(p->policy))) {
cpuset_locked = true;
- cpuset_lock();
+ cpuset_lock_no_attach();
dl_accounting_lock();
}
@@ -1246,6 +1251,23 @@ int __sched_setaffinity(struct task_struct *p, struct affinity_context *ctx)
cpuset_cpus_allowed(p, cpus_allowed);
if (!cpumask_subset(new_mask, cpus_allowed)) {
+ bool locked = false;
+
+ /*
+ * A sched_setattr() may have made @p a deadline task since
+ * the first call. SCA_DL_NOWAIT would then skip the second
+ * call. Wait for a cpuset attach in progress instead, as
+ * sched_setaffinity() does for a deadline task. That attach
+ * can move @p to another cpuset: read its mask again after
+ * the wait.
+ */
+ if (ctx->flags & SCA_DL_NOWAIT) {
+ ctx->flags &= ~SCA_DL_NOWAIT;
+ cpuset_lock_no_attach();
+ locked = true;
+ cpuset_cpus_allowed(p, cpus_allowed);
+ }
+
/*
* We must have raced with a concurrent cpuset update.
* Just reset the cpumask to the cpuset's cpus_allowed.
@@ -1274,6 +1296,8 @@ int __sched_setaffinity(struct task_struct *p, struct affinity_context *ctx)
*/
ctx->flags &= ~SCA_DL_CHECK;
__set_cpus_allowed_ptr(p, ctx);
+ if (locked)
+ cpuset_unlock();
retval = -EINVAL;
}
@@ -1289,6 +1313,7 @@ long sched_setaffinity(pid_t pid, const struct cpumask *in_mask)
struct affinity_context ac;
struct cpumask *user_mask;
int retval;
+ bool dl;
CLASS(find_get_task, p)(pid);
if (!p)
@@ -1318,13 +1343,31 @@ long sched_setaffinity(pid_t pid, const struct cpumask *in_mask)
return -ENOMEM;
}
- ac = (struct affinity_context){
- .new_mask = in_mask,
- .user_mask = user_mask,
- .flags = SCA_USER,
- };
-
- retval = __sched_setaffinity(p, &ac);
+ /*
+ * Wait for a cpuset attach in progress before changing the affinity
+ * of a deadline task. cpuset_can_attach() reserved the bandwidth of
+ * the task for the CPUs it moves to, and cpuset_attach() moves it
+ * there. A new affinity in between could change those CPUs.
+ *
+ * The policy is tested here with no lock. A task that becomes a
+ * deadline task after this test is found under its runqueue lock,
+ * with SCA_DL_NOWAIT. Wait and try again then.
+ */
+ for (;;) {
+ dl = task_has_dl_policy(p);
+ ac = (struct affinity_context){
+ .new_mask = in_mask,
+ .user_mask = user_mask,
+ .flags = SCA_USER | (dl ? 0 : SCA_DL_NOWAIT),
+ };
+ if (dl)
+ cpuset_lock_no_attach();
+ retval = __sched_setaffinity(p, &ac);
+ if (dl)
+ cpuset_unlock();
+ if (dl || retval != -EAGAIN)
+ break;
+ }
kfree(ac.user_mask);
return retval;
--
2.47.3
^ permalink raw reply related [flat|nested] 35+ messages in thread
* [PATCH v8 10/33] cgroup/cpuset: Give back the bandwidth an attach did not move
2026-11-19 20:15 [PATCH v8 00/33] sched/deadline: bandwidth accounting fixes and selftests Mathieu Desnoyers
` (8 preceding siblings ...)
2026-11-19 20:15 ` [PATCH v8 09/33] sched: Make deadline policy and affinity changes wait for a cpuset attach Mathieu Desnoyers
@ 2026-11-19 20:15 ` Mathieu Desnoyers
2026-11-19 20:15 ` [PATCH v8 11/33] cgroup/cpuset: Reserve the bandwidth in the root domain a task moves to Mathieu Desnoyers
` (22 subsequent siblings)
32 siblings, 0 replies; 35+ messages in thread
From: Mathieu Desnoyers @ 2026-11-19 20:15 UTC (permalink / raw)
To: juri.lelli, peterz, mingo, vincent.guittot
Cc: Mathieu Desnoyers, tglx, will, boqun, dietmar.eggemann, rostedt,
bsegall, mgorman, vschneid, kprateek.nayak, luca.abeni,
mathieu.poirier, longman, ridong.chen, tj, hannes, mkoutny, shuah,
bigeasy, clrkwllms, parri.andrea, m.szyprowski, kayracizmeci,
weihengqiuu, linux-kernel, cgroups, linux-kselftest,
linux-rt-devel, stable
Commit 2ef269ef1ac0 ("cgroup/cpuset: Free DL BW in case can_attach()
fails") reserves the bandwidth of an attach as one sum.
cpuset_can_attach() adds up the bandwidth of the deadline tasks it
moves. Once all tasks pass their checks, it reserves the sum in the
destination root domain. If another controller refuses the attach,
cpuset_cancel_attach() frees the sum. If the attach goes through, the
move of each task uses its part of the sum.
But a task can leave SCHED_DEADLINE between cpuset_can_attach() and
cpuset_attach(). The previous patch makes sched_setattr() wait for the
attach. sysrq-n does not wait: it cannot sleep. The task then moves no
bandwidth. Its part of the reservation stays in the destination's
total_bw, until the next rebuild of the accounting. The destination can
then refuse deadline tasks that would fit.
Fix this by keeping the tally of what the attach moves, in
sum_moved_dl_bw. Once the attach is through, give back what it did not
move.
set_cpus_allowed_dl() decides the move under the runqueue lock.
cpuset_attach_task() does not hold it. So the tally counts what
set_cpus_allowed_dl() reports in the affinity context.
The random_mix case, added to deadline_bw_test later in this series,
found the bug. It could no longer lower a fair server in a partition
root with no room left.
The move_normalized case, added with that test, fires sysrq-n while a
process with a sleeping deadline thread of 30% moves into a one-CPU
partition root. Without this patch, the partition kept the 30% in 3
guests of 3. It admitted 6 tasks of 10% where it had room for 9.
Fixes: 2ef269ef1ac0 ("cgroup/cpuset: Free DL BW in case can_attach() fails")
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Cc: stable@vger.kernel.org
---
include/linux/sched.h | 3 ++-
kernel/cgroup/cpuset-internal.h | 2 ++
kernel/cgroup/cpuset.c | 30 +++++++++++++++++++++++++-----
kernel/sched/core.c | 12 ++++++++----
kernel/sched/deadline.c | 4 ++++
kernel/sched/sched.h | 2 ++
6 files changed, 43 insertions(+), 10 deletions(-)
diff --git a/include/linux/sched.h b/include/linux/sched.h
index e88616448b85..81a8d80a1aee 100644
--- a/include/linux/sched.h
+++ b/include/linux/sched.h
@@ -1967,7 +1967,8 @@ extern void set_cpus_allowed_force(struct task_struct *p, const struct cpumask *
*/
extern int set_cpus_allowed_ptr(struct task_struct *p, const struct cpumask *new_mask);
extern int set_cpus_allowed_bw_reserved(struct task_struct *p,
- const struct cpumask *new_mask);
+ const struct cpumask *new_mask,
+ u64 *moved);
extern int dup_user_cpus_ptr(struct task_struct *dst, struct task_struct *src, int node);
extern void release_user_cpus_ptr(struct task_struct *p);
extern int dl_task_check_affinity(struct task_struct *p, const struct cpumask *mask);
diff --git a/kernel/cgroup/cpuset-internal.h b/kernel/cgroup/cpuset-internal.h
index bf847953c78d..93aa46d76b04 100644
--- a/kernel/cgroup/cpuset-internal.h
+++ b/kernel/cgroup/cpuset-internal.h
@@ -170,6 +170,8 @@ struct cpuset {
int nr_migrate_dl_tasks;
/* DL bandwidth that needs destination reservation for this attach. */
u64 sum_migrate_dl_bw;
+ /* The part of sum_migrate_dl_bw moved. The rest is given back. */
+ u64 sum_moved_dl_bw;
/*
* CPU used for temporary DL bandwidth allocation during attach;
* -1 if no DL bandwidth was allocated in the current attach.
diff --git a/kernel/cgroup/cpuset.c b/kernel/cgroup/cpuset.c
index ce9c57617f49..2f39c3d12550 100644
--- a/kernel/cgroup/cpuset.c
+++ b/kernel/cgroup/cpuset.c
@@ -3140,10 +3140,24 @@ static void clear_attach_data(struct llist_head *head, bool cancel)
llist_for_each_entry_safe(cs, next, lnode, attach_node) {
init_llist_node(&cs->attach_node);
if (cs->nr_migrate_dl_tasks) {
- if (cancel && cs->dl_bw_cpu >= 0)
- dl_bw_free(cs->dl_bw_cpu, cs->sum_migrate_dl_bw);
+ /*
+ * Give back the bandwidth the attach did not move.
+ * If the attach is cancelled, that is all of the
+ * reservation. Otherwise, it is what no move used,
+ * such as the part for a task that left
+ * SCHED_DEADLINE. The class it left releases its own
+ * bandwidth, from the root domain that counts it.
+ */
+ if (cs->dl_bw_cpu >= 0) {
+ u64 bw = cancel ? cs->sum_migrate_dl_bw :
+ cs->sum_migrate_dl_bw - cs->sum_moved_dl_bw;
+
+ if (bw)
+ dl_bw_free(cs->dl_bw_cpu, bw);
+ }
cs->nr_migrate_dl_tasks = 0;
cs->sum_migrate_dl_bw = 0;
+ cs->sum_moved_dl_bw = 0;
cs->dl_bw_cpu = -1;
}
}
@@ -3303,10 +3317,16 @@ static void cpuset_attach_task(struct cpuset *cs, struct task_struct *task)
* can_attach beforehand should guarantee that this doesn't
* fail. TODO: have a better way to handle failure here
*/
- if (reserved)
- WARN_ON_ONCE(set_cpus_allowed_bw_reserved(task, cpus_attach));
- else
+ if (reserved) {
+ u64 moved;
+
+ /* Count what the move reports it moved. */
+ WARN_ON_ONCE(set_cpus_allowed_bw_reserved(task, cpus_attach,
+ &moved));
+ cs->sum_moved_dl_bw += moved;
+ } else {
WARN_ON_ONCE(set_cpus_allowed_ptr(task, cpus_attach));
+ }
if (cpuset_v2() && !attach_ctx.mems_updated)
return;
diff --git a/kernel/sched/core.c b/kernel/sched/core.c
index c0a714dde425..fff2725a4cd5 100644
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -3301,17 +3301,21 @@ EXPORT_SYMBOL_GPL(set_cpus_allowed_ptr);
/*
* Like set_cpus_allowed_ptr(), for a caller that has made room for the
* deadline bandwidth of @p in the root domain of @new_mask. Move that
- * bandwidth there with @p. Return 0 on success, or a negative error
- * code.
+ * bandwidth there with @p, and set @moved to the bandwidth moved, 0
+ * if none. Return 0 on success, or a negative error code.
*/
-int set_cpus_allowed_bw_reserved(struct task_struct *p, const struct cpumask *new_mask)
+int set_cpus_allowed_bw_reserved(struct task_struct *p,
+ const struct cpumask *new_mask, u64 *moved)
{
struct affinity_context ac = {
.new_mask = new_mask,
.flags = SCA_DL_BW_RESERVED,
};
+ int ret;
- return __set_cpus_allowed_ptr(p, &ac);
+ ret = __set_cpus_allowed_ptr(p, &ac);
+ *moved = ac.dl_bw_moved;
+ return ret;
}
/*
diff --git a/kernel/sched/deadline.c b/kernel/sched/deadline.c
index 56d1f6636edc..317367daa87d 100644
--- a/kernel/sched/deadline.c
+++ b/kernel/sched/deadline.c
@@ -3619,6 +3619,9 @@ static void set_cpus_allowed_dl(struct task_struct *p,
* Otherwise, the bandwidth stays where it is counted, until the next
* rebuild of the accounting. Moving it without room would leave no
* root domain counting it.
+ *
+ * Report in @ctx the bandwidth moved. The caller made its own test
+ * without the runqueue lock, so it must count only what this reports.
*/
if ((ctx->flags & SCA_DL_BW_RESERVED) &&
dl_task_needs_bw_move(p, ctx->new_mask)) {
@@ -3647,6 +3650,7 @@ static void set_cpus_allowed_dl(struct task_struct *p,
cpu = cpumask_any_and(ctx->new_mask, cpu_active_mask);
p->dl.dl_bw_cpu = cpu < nr_cpu_ids ? (int)cpu : -1;
p->dl.dl_bw_gen = dl_bw_gen;
+ ctx->dl_bw_moved = p->dl.dl_bw;
}
set_cpus_allowed_common(p, ctx);
diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h
index f4cd35811b96..65fc17909d56 100644
--- a/kernel/sched/sched.h
+++ b/kernel/sched/sched.h
@@ -2678,6 +2678,8 @@ extern const u32 sched_prio_to_wmult[40];
struct affinity_context {
const struct cpumask *new_mask;
struct cpumask *user_mask;
+ /* Out: the deadline bandwidth moved to the new root domain. */
+ u64 dl_bw_moved;
unsigned int flags;
};
--
2.47.3
^ permalink raw reply related [flat|nested] 35+ messages in thread
* [PATCH v8 11/33] cgroup/cpuset: Reserve the bandwidth in the root domain a task moves to
2026-11-19 20:15 [PATCH v8 00/33] sched/deadline: bandwidth accounting fixes and selftests Mathieu Desnoyers
` (9 preceding siblings ...)
2026-11-19 20:15 ` [PATCH v8 10/33] cgroup/cpuset: Give back the bandwidth an attach did not move Mathieu Desnoyers
@ 2026-11-19 20:15 ` Mathieu Desnoyers
2026-11-19 20:15 ` [PATCH v8 12/33] selftests/sched: Test cpuset moves in deadline_bw_test Mathieu Desnoyers
` (21 subsequent siblings)
32 siblings, 0 replies; 35+ messages in thread
From: Mathieu Desnoyers @ 2026-11-19 20:15 UTC (permalink / raw)
To: juri.lelli, peterz, mingo, vincent.guittot
Cc: Mathieu Desnoyers, tglx, will, boqun, dietmar.eggemann, rostedt,
bsegall, mgorman, vschneid, kprateek.nayak, luca.abeni,
mathieu.poirier, longman, ridong.chen, tj, hannes, mkoutny, shuah,
bigeasy, clrkwllms, parri.andrea, m.szyprowski, kayracizmeci,
weihengqiuu, linux-kernel, cgroups, linux-kselftest,
linux-rt-devel, stable
Commit 7f51412a415d ("sched/deadline: Fix bandwidth check/update when
migrating tasks between exclusive cpusets") moves a deadline task's
bandwidth in two halves. cpuset_can_attach() reserves it in the
destination. set_cpus_allowed_dl() frees it in the source when the
task's affinity changes. The reservation goes to the root domain of the
first active CPU of the destination cpuset. So cpuset_can_attach()
treats the cpuset as one root domain.
But a cpuset can span more than one root domain. With isolcpus=domain,
the root cpuset holds the isolated CPUs, which are in the default root
domain. A task also keeps the affinity set by sched_setaffinity(). Its
new CPUs can then all be in a root domain that got no reservation. The
task runs there, and its bandwidth is reserved in the other root domain.
Take a machine with CPUs 0-4, booted with isolcpus=domain,1,2. P is a
partition root of CPU 4. A deadline task T, pinned to CPUs 1-2, is moved
into P and back to the root cgroup:
total_bw of the root domain of
admin, task T CPUs 0, 3 CPUs 1, 2 CPU 4 (P)
sched_setaffinity(T, CPUs 1-2)
sched_setattr(T, SCHED_DEADLINE) + bw
echo T > P/cgroup.procs
cpuset_can_attach()
reserves on CPU 4 + bw
set_cpus_allowed_dl()
frees where recorded, CPU 1 - bw
records CPU 4
T runs on CPU 4
echo T > cgroup.procs of the root
cpuset_can_attach()
reserves on CPU 0 + bw
__set_cpus_allowed_ptr()
keeps T's affinity: CPUs 1-2
set_cpus_allowed_dl()
frees where recorded, CPU 4 - bw
records CPU 1
T runs on CPU 1
T exits
releases where recorded, CPU 1 - bw
--------- --------- ---------
left after T exits + bw - bw 0
T ran in the root domain of CPUs 1 and 2 without being counted there.
In testing, the total_bw of that root domain went below zero when T
died. It then refused every deadline task, until the next rebuild of
the accounting.
Fix this by reserving the bandwidth on the CPU that the move records
with the task. The new dl_task_bw_reserve_cpu() works out that CPU and
records it with the task. It is the CPU the task is on, if the task can
stay there. Else it is the first active CPU of the new mask, a
housekeeping one if any. The rebuild of the accounting makes the same
choice, later in this series. cpuset_can_attach() reserves the bandwidth
per such CPU, and refuses with EBUSY as before. cpuset_attach_task()
moves the bandwidth to the CPU recorded, and does not work it out again:
the task may have run on another CPU since. cpuset_attach() gives back
the rest.
The record is a new field of the task, dl_bw_resv_cpu.
cpuset_attach_task() moves the bandwidth of a task only if the task has
that record. Juri Lelli suggested to track the tasks that
cpuset_can_attach() counts. A task it did not count cannot use the room
made for another task.
A new mask can also span both root domains. A running task would then go
to the CPU that cpumask_any_and_distribute() picks. A sleeping one would
wake up where select_fallback_rq() picks. Either CPU can be in the root
domain without the reservation. So __set_cpus_allowed_ptr_locked() takes
the CPU with the reservation, if cpumask_any_and_distribute() picked a
CPU of the other root domain. A sleeping task gets the CPU chosen as its
wakeup CPU.
A new mask that still holds an active CPU of the root domain counting
the task moves no bandwidth at the attach. The task can still land in
the other root domain, if it must leave its CPU. A later patch of this
series moves the bandwidth there, without a check for room. It is
"sched/deadline: Move a task's bandwidth to the root domain it runs in".
The sums are kept per CPU, not per cpuset: one cpuset can need room in
two root domains. One set of sums is enough, since cgroup_mutex lets
one attach run at a time. guarantee_active_cpus() now takes the cpuset
to start from.
Later in this series, deadline_bw_test gets four cases for this:
move_isolated, move_isolated_refused, move_isolated_asleep and
move_spanning. Without this patch, all four fail with
isolcpus=domain,1,2. With isolcpus=domain,0,1, move_isolated,
move_spanning and six other move cases fail. The case
move_back_cpu_offline tests a mask that holds an offline CPU.
Fixes: 7f51412a415d ("sched/deadline: Fix bandwidth check/update when migrating tasks between exclusive cpusets")
Suggested-by: Juri Lelli <juri.lelli@redhat.com>
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Cc: stable@vger.kernel.org
---
include/linux/sched.h | 8 +-
include/linux/sched/deadline.h | 15 ++-
kernel/cgroup/cpuset-internal.h | 12 --
kernel/cgroup/cpuset.c | 216 +++++++++++++++++++-------------
kernel/sched/core.c | 27 +++-
kernel/sched/deadline.c | 133 ++++++++++++++++----
kernel/sched/sched.h | 2 +
kernel/sched/syscalls.c | 5 +-
8 files changed, 282 insertions(+), 136 deletions(-)
diff --git a/include/linux/sched.h b/include/linux/sched.h
index 81a8d80a1aee..8e7aa758d863 100644
--- a/include/linux/sched.h
+++ b/include/linux/sched.h
@@ -752,10 +752,16 @@ struct sched_dl_entity {
*
* @dl_bw_gen is the value of dl_bw_gen when the bandwidth was last
* counted. If a rebuild dropped it since, there is nothing to remove.
+ *
+ * @dl_bw_resv_cpu is a CPU of the root domain where a cpuset attach
+ * in progress reserved room for the bandwidth, or -1.
+ * cpuset_can_attach() sets it, and cpuset_attach() moves the
+ * bandwidth there.
*/
struct list_head dl_departed_node;
int dl_bw_cpu;
unsigned int dl_bw_gen;
+ int dl_bw_resv_cpu;
/*
* Bits for DL-server functionality. Also see the comment near
@@ -1968,7 +1974,7 @@ extern void set_cpus_allowed_force(struct task_struct *p, const struct cpumask *
extern int set_cpus_allowed_ptr(struct task_struct *p, const struct cpumask *new_mask);
extern int set_cpus_allowed_bw_reserved(struct task_struct *p,
const struct cpumask *new_mask,
- u64 *moved);
+ int cpu, u64 *moved);
extern int dup_user_cpus_ptr(struct task_struct *dst, struct task_struct *src, int node);
extern void release_user_cpus_ptr(struct task_struct *p);
extern int dl_task_check_affinity(struct task_struct *p, const struct cpumask *mask);
diff --git a/include/linux/sched/deadline.h b/include/linux/sched/deadline.h
index 60be226699bb..cbb7cd71db58 100644
--- a/include/linux/sched/deadline.h
+++ b/include/linux/sched/deadline.h
@@ -50,14 +50,17 @@ extern void dl_accounting_unlock(void);
extern void lockdep_assert_dl_accounting_held(void);
#endif
/*
- * Return whether moving DL task @p to @new_mask requires moving DL
- * bandwidth accounting between root domains. This helper is specific to
- * DL bandwidth move accounting semantics and is shared by
- * cpuset_can_attach() and set_cpus_allowed_dl() so both paths use the
- * same source root-domain test.
+ * Return the CPU that a move of DL task @p to the CPUs of @new_mask
+ * records with its DL bandwidth. Return -1 if the bandwidth stays in the
+ * root domain that counts it. Like the move, keep the CPUs within the
+ * affinity set by the user. cpuset_can_attach() reserves the bandwidth in
+ * the root domain of that CPU. cpuset_attach_task() then moves it there,
+ * and set_cpus_allowed_dl() makes the same test.
*/
-extern bool dl_task_needs_bw_move(struct task_struct *p,
+extern int dl_task_bw_reserve_cpu(struct task_struct *p,
const struct cpumask *new_mask);
+extern int dl_task_bw_reserved_cpu(struct task_struct *p);
+extern void dl_task_bw_unreserve(struct task_struct *p);
static inline bool dl_server(struct sched_dl_entity *dl_se)
{
diff --git a/kernel/cgroup/cpuset-internal.h b/kernel/cgroup/cpuset-internal.h
index 93aa46d76b04..a83443dfb788 100644
--- a/kernel/cgroup/cpuset-internal.h
+++ b/kernel/cgroup/cpuset-internal.h
@@ -166,18 +166,6 @@ struct cpuset {
*/
bool remote_partition_updating;
- /* Number of SCHED_DEADLINE tasks the attach in progress moves here. */
- int nr_migrate_dl_tasks;
- /* DL bandwidth that needs destination reservation for this attach. */
- u64 sum_migrate_dl_bw;
- /* The part of sum_migrate_dl_bw moved. The rest is given back. */
- u64 sum_moved_dl_bw;
- /*
- * CPU used for temporary DL bandwidth allocation during attach;
- * -1 if no DL bandwidth was allocated in the current attach.
- */
- int dl_bw_cpu;
-
/* Invalid partition error code, not lock protected */
enum prs_errcode prs_err;
diff --git a/kernel/cgroup/cpuset.c b/kernel/cgroup/cpuset.c
index 2f39c3d12550..7bdb9e21a7ed 100644
--- a/kernel/cgroup/cpuset.c
+++ b/kernel/cgroup/cpuset.c
@@ -272,7 +272,6 @@ struct cpuset top_cpuset = {
.flags = BIT(CS_CPU_EXCLUSIVE) |
BIT(CS_MEM_EXCLUSIVE) | BIT(CS_SCHED_LOAD_BALANCE),
.partition_root_state = PRS_ROOT,
- .dl_bw_cpu = -1,
};
/**
@@ -487,28 +486,24 @@ static inline bool partition_is_populated(struct cpuset *cs,
}
/*
- * Return in pmask the portion of a task's cpusets's cpus_allowed that
- * are online and are capable of running the task. If none are found,
- * walk up the cpuset hierarchy until we find one that does have some
- * appropriate cpus.
+ * Return in pmask the portion of the cpus_allowed of @cs that are online
+ * and are capable of running @tsk. If none are found, walk up the cpuset
+ * hierarchy until we find one that does have some appropriate cpus.
*
* One way or another, we guarantee to return some non-empty subset
* of cpu_active_mask.
*
* Call with callback_lock or cpuset_mutex held.
*/
-static void guarantee_active_cpus(struct task_struct *tsk,
+static void guarantee_active_cpus(struct cpuset *cs, struct task_struct *tsk,
struct cpumask *pmask)
{
const struct cpumask *possible_mask = task_cpu_possible_mask(tsk);
- struct cpuset *cs;
if (WARN_ON(!cpumask_and(pmask, possible_mask, cpu_active_mask)))
cpumask_copy(pmask, cpu_active_mask);
rcu_read_lock();
- cs = task_cs(tsk);
-
while (!cpumask_intersects(cs->effective_cpus, pmask)) {
cs = parent_cs(cs);
if (unlikely(!cs)) {
@@ -630,7 +625,6 @@ struct cpuset *dup_or_alloc_cpuset(struct cpuset *cs)
if (!trial)
return NULL;
- trial->dl_bw_cpu = -1;
init_llist_node(&trial->attach_node);
/* Setup cpumask pointer array */
@@ -3106,61 +3100,89 @@ static int cpuset_can_attach_check(struct cpuset *cs, struct cpuset *oldcs,
return 0;
}
+/*
+ * The deadline bandwidth of the attach in progress, per CPU. When the
+ * bandwidth of a deadline task moves to another root domain,
+ * dl_task_bw_reserve_cpu() returns the CPU that the move records with it.
+ * It also records that CPU with the task. cpuset_can_attach() sums the
+ * bandwidth of these tasks per such CPU. It reserves each sum in the root
+ * domain of its CPU. cpuset_attach_task() moves the bandwidth of each task
+ * to the CPU recorded with it. It counts what it moves, and cpuset_attach()
+ * gives back the rest.
+ * dl_attach_cpus holds the CPUs with a sum. cgroup_mutex lets one attach
+ * run at a time, and cpuset_mutex protects these.
+ */
+struct cpuset_dl_bw {
+ u64 sum;
+ u64 moved; /* the part of sum moved */
+ bool reserved; /* dl_bw_alloc() took sum */
+};
+
+static DEFINE_PER_CPU(struct cpuset_dl_bw, cpuset_dl_bw);
+static cpumask_var_t dl_attach_cpus;
+
static int cpuset_reserve_dl_bw(void)
{
- struct cpuset *cs;
int cpu, ret;
- llist_for_each_entry(cs, dst_cs_head.first, attach_node) {
- if (!cs->sum_migrate_dl_bw)
- continue;
+ for_each_cpu(cpu, dl_attach_cpus) {
+ struct cpuset_dl_bw *b = per_cpu_ptr(&cpuset_dl_bw, cpu);
- cpu = cpumask_any_and(cpu_active_mask, cs->effective_cpus);
- if (unlikely(cpu >= nr_cpu_ids))
- return -EINVAL;
-
- ret = dl_bw_alloc(cpu, cs->sum_migrate_dl_bw);
+ ret = dl_bw_alloc(cpu, b->sum);
if (ret)
return ret;
-
- cs->dl_bw_cpu = cpu;
+ b->reserved = true;
}
return 0;
}
/*
- * Clear and optionally apply (@cancel is false) the attach related data in the
- * source or destination cpuset.
+ * Give back the bandwidth the attach reserved and did not move. If the
+ * attach is cancelled, that is all of the reservation. Otherwise, it is
+ * what no move used, such as the part for a task that left
+ * SCHED_DEADLINE. The class it left releases its own bandwidth, from the
+ * root domain that counts it.
*/
-static void clear_attach_data(struct llist_head *head, bool cancel)
+static void cpuset_free_dl_bw(void)
+{
+ int cpu;
+
+ for_each_cpu(cpu, dl_attach_cpus) {
+ struct cpuset_dl_bw *b = per_cpu_ptr(&cpuset_dl_bw, cpu);
+
+ if (b->reserved && b->sum != b->moved)
+ dl_bw_free(cpu, b->sum - b->moved);
+ *b = (struct cpuset_dl_bw){ };
+ }
+ cpumask_clear(dl_attach_cpus);
+}
+
+/* Empty @head, the list of the source or destination cpusets. */
+static void clear_attach_data(struct llist_head *head)
{
struct cpuset *cs, *next;
struct llist_node *lnode = __llist_del_all(head);
- llist_for_each_entry_safe(cs, next, lnode, attach_node) {
+ llist_for_each_entry_safe(cs, next, lnode, attach_node)
init_llist_node(&cs->attach_node);
- if (cs->nr_migrate_dl_tasks) {
- /*
- * Give back the bandwidth the attach did not move.
- * If the attach is cancelled, that is all of the
- * reservation. Otherwise, it is what no move used,
- * such as the part for a task that left
- * SCHED_DEADLINE. The class it left releases its own
- * bandwidth, from the root domain that counts it.
- */
- if (cs->dl_bw_cpu >= 0) {
- u64 bw = cancel ? cs->sum_migrate_dl_bw :
- cs->sum_migrate_dl_bw - cs->sum_moved_dl_bw;
+}
- if (bw)
- dl_bw_free(cs->dl_bw_cpu, bw);
- }
- cs->nr_migrate_dl_tasks = 0;
- cs->sum_migrate_dl_bw = 0;
- cs->sum_moved_dl_bw = 0;
- cs->dl_bw_cpu = -1;
- }
- }
+/*
+ * Protected by cpuset_mutex. cpus_attach is used by cpuset_can_attach() and
+ * cpuset_attach_task(), but we can't allocate it dynamically there. Define
+ * it global and allocate from cpuset_init().
+ */
+static cpumask_var_t cpus_attach;
+
+/* Set @mask to the CPUs that an attach to @cs gives @task. */
+static void cpuset_attach_cpus(struct cpuset *cs, struct task_struct *task,
+ struct cpumask *mask)
+{
+ if (cs != &top_cpuset)
+ guarantee_active_cpus(cs, task, mask);
+ else
+ cpumask_andnot(mask, task_cpu_possible_mask(task),
+ subpartitions_cpus);
}
/* Called by cgroups to determine if a cpuset is usable; cpuset_mutex held */
@@ -3170,7 +3192,7 @@ static int cpuset_can_attach(struct cgroup_taskset *tset)
struct cpuset *cs, *oldcs;
struct task_struct *task;
bool setsched_check;
- int ret;
+ int cpu, ret;
cs = oldcs = NULL;
mutex_lock(&cpuset_mutex);
@@ -3236,15 +3258,20 @@ static int cpuset_can_attach(struct cgroup_taskset *tset)
goto out_unlock;
}
- if (dl_task(task)) {
- /*
- * Count all migrating DL tasks for cpuset task accounting.
- * Only tasks that need a root-domain bandwidth move
- * contribute to sum_migrate_dl_bw.
- */
- cs->nr_migrate_dl_tasks++;
- if (dl_task_needs_bw_move(task, cs->effective_cpus))
- cs->sum_migrate_dl_bw += task->dl.dl_bw;
+ if (!dl_task(task))
+ continue;
+ /*
+ * Sum the bandwidth of the deadline tasks that move to another
+ * root domain, per CPU that the move records. That CPU depends
+ * on the CPUs the task gets. A cpuset can span more than one
+ * root domain, and the affinity of a task can keep it in any
+ * of them.
+ */
+ cpuset_attach_cpus(cs, task, cpus_attach);
+ cpu = dl_task_bw_reserve_cpu(task, cpus_attach);
+ if (cpu >= 0) {
+ per_cpu_ptr(&cpuset_dl_bw, cpu)->sum += task->dl.dl_bw;
+ cpumask_set_cpu(cpu, dl_attach_cpus);
}
}
@@ -3268,8 +3295,11 @@ static int cpuset_can_attach(struct cgroup_taskset *tset)
out_unlock:
if (ret) {
- clear_attach_data(&src_cs_head, true);
- clear_attach_data(&dst_cs_head, true);
+ clear_attach_data(&src_cs_head);
+ clear_attach_data(&dst_cs_head);
+ cpuset_free_dl_bw();
+ cgroup_taskset_for_each(task, css, tset)
+ dl_task_bw_unreserve(task);
} else {
attach_ctx.in_progress++;
}
@@ -3280,50 +3310,59 @@ static int cpuset_can_attach(struct cgroup_taskset *tset)
static void cpuset_cancel_attach(struct cgroup_taskset *tset)
{
+ struct cgroup_subsys_state *css;
+ struct task_struct *task;
+
mutex_lock(&cpuset_mutex);
dec_attach_in_progress_locked();
- clear_attach_data(&src_cs_head, true);
- clear_attach_data(&dst_cs_head, true);
+ clear_attach_data(&src_cs_head);
+ clear_attach_data(&dst_cs_head);
+ cpuset_free_dl_bw();
+ cgroup_taskset_for_each(task, css, tset)
+ dl_task_bw_unreserve(task);
mutex_unlock(&cpuset_mutex);
}
-/*
- * Protected by cpuset_mutex. cpus_attach is used only by cpuset_attach_task()
- * but we can't allocate it dynamically there. Define it global and
- * allocate from cpuset_init().
- */
-static cpumask_var_t cpus_attach;
-
static void cpuset_attach_task(struct cpuset *cs, struct task_struct *task)
{
- bool reserved = false;
+ struct cpuset_dl_bw *b = NULL;
struct mm_struct *mm;
+ int cpu = -1;
lockdep_assert_cpuset_lock_held();
- if (cs != &top_cpuset)
- guarantee_active_cpus(task, cpus_attach);
- else
- cpumask_andnot(cpus_attach, task_cpu_possible_mask(task),
- subpartitions_cpus);
+ cpuset_attach_cpus(cs, task, cpus_attach);
/*
- * cpuset_reserve_dl_bw() made room in @cs for the bandwidth of the
- * deadline tasks that cpuset_can_attach() counted. Flag the move, so
- * that the bandwidth of this task moves with it.
+ * cpuset_can_attach() made room for the bandwidth of each deadline
+ * task it moves, in the root domain of the CPU it recorded with the
+ * task. Flag the move with that CPU, so that the bandwidth goes
+ * there. The task may have run on another CPU since: the CPU is not
+ * worked out again.
+ *
+ * A task with no CPU recorded needs no move, or was not a deadline
+ * task in cpuset_can_attach(). Its move is not flagged, so that it
+ * cannot use the room made for another task. Its bandwidth stays
+ * where it is counted.
*/
- if (dl_task(task) && dl_task_needs_bw_move(task, cs->effective_cpus))
- reserved = true;
+ cpu = dl_task_bw_reserved_cpu(task);
+ if (cpu >= 0)
+ b = per_cpu_ptr(&cpuset_dl_bw, cpu);
/*
* can_attach beforehand should guarantee that this doesn't
* fail. TODO: have a better way to handle failure here
*/
- if (reserved) {
+ if (b) {
u64 moved;
/* Count what the move reports it moved. */
WARN_ON_ONCE(set_cpus_allowed_bw_reserved(task, cpus_attach,
- &moved));
- cs->sum_moved_dl_bw += moved;
+ cpu, &moved));
+ b->moved += moved;
+ /*
+ * set_cpus_allowed_dl() cleared the record, unless the task
+ * left SCHED_DEADLINE. Clear it then too.
+ */
+ dl_task_bw_unreserve(task);
} else {
WARN_ON_ONCE(set_cpus_allowed_ptr(task, cpus_attach));
}
@@ -3397,6 +3436,13 @@ static void cpuset_attach(struct cgroup_taskset *tset)
if (cpuset_v2() && !attach_ctx.cpus_updated && !attach_ctx.mems_updated) {
llist_for_each_entry(cs, dst_cs_head.first, attach_node)
cs->old_mems_allowed = attach_ctx.nodemask_to;
+ /*
+ * No mask changes, so no bandwidth moves. Forget the CPU
+ * that cpuset_can_attach() recorded with a deadline task, as
+ * cpuset_attach_task() does. A later attach must not find it.
+ */
+ cgroup_taskset_for_each(task, css, tset)
+ dl_task_bw_unreserve(task);
goto out;
}
@@ -3407,8 +3453,9 @@ static void cpuset_attach(struct cgroup_taskset *tset)
schedule_flush_migrate_mm();
cs->old_mems_allowed = attach_ctx.nodemask_to;
out:
- clear_attach_data(&src_cs_head, false);
- clear_attach_data(&dst_cs_head, false);
+ clear_attach_data(&src_cs_head);
+ clear_attach_data(&dst_cs_head);
+ cpuset_free_dl_bw();
dec_attach_in_progress_locked();
mutex_unlock(&cpuset_mutex);
@@ -3913,6 +3960,7 @@ int __init cpuset_init(void)
cpuset1_init(&top_cpuset);
BUG_ON(!alloc_cpumask_var(&cpus_attach, GFP_KERNEL));
+ BUG_ON(!zalloc_cpumask_var(&dl_attach_cpus, GFP_KERNEL));
if (housekeeping_enabled(HK_TYPE_DOMAIN_BOOT))
cpumask_andnot(isolated_cpus, cpu_possible_mask,
@@ -4211,7 +4259,7 @@ static void __cpuset_cpus_allowed_locked(struct task_struct *tsk, struct cpumask
cs = task_cs(tsk);
if (cs != &top_cpuset)
- guarantee_active_cpus(tsk, pmask);
+ guarantee_active_cpus(cs, tsk, pmask);
/*
* Tasks in the top cpuset won't get update to their cpumasks
* when a hotplug online/offline event happens. So we include all
diff --git a/kernel/sched/core.c b/kernel/sched/core.c
index fff2725a4cd5..7615ba02d52b 100644
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -3250,6 +3250,22 @@ static int __set_cpus_allowed_ptr_locked(struct task_struct *p,
goto out;
}
+ /*
+ * The caller made room for the deadline bandwidth of @p in the root
+ * domain of ctx->dl_bw_cpu. The new mask can span two root domains:
+ * keep @p in that one. Also wake a sleeping @p up on dest_cpu, since
+ * select_fallback_rq() could pick a CPU of the other one.
+ */
+ if ((ctx->flags & SCA_DL_BW_RESERVED) &&
+ cpumask_test_cpu(ctx->dl_bw_cpu, cpu_valid_mask) &&
+ cpumask_test_cpu(ctx->dl_bw_cpu, ctx->new_mask)) {
+ if (!cpumask_test_cpu(dest_cpu,
+ cpu_rq(ctx->dl_bw_cpu)->rd->span))
+ dest_cpu = ctx->dl_bw_cpu;
+ if (!task_on_rq_queued(p))
+ p->wake_cpu = dest_cpu;
+ }
+
do_set_cpus_allowed(p, ctx);
return affine_move_task(rq, p, rf, dest_cpu, ctx->flags);
@@ -3300,15 +3316,18 @@ EXPORT_SYMBOL_GPL(set_cpus_allowed_ptr);
/*
* Like set_cpus_allowed_ptr(), for a caller that has made room for the
- * deadline bandwidth of @p in the root domain of @new_mask. Move that
- * bandwidth there with @p, and set @moved to the bandwidth moved, 0
- * if none. Return 0 on success, or a negative error code.
+ * deadline bandwidth of @p in the root domain of @cpu. If the move records
+ * @cpu with that bandwidth, move the bandwidth there with @p. Set @moved
+ * to the bandwidth moved, 0 if none. Return 0 on success, or a negative
+ * error code.
*/
int set_cpus_allowed_bw_reserved(struct task_struct *p,
- const struct cpumask *new_mask, u64 *moved)
+ const struct cpumask *new_mask, int cpu,
+ u64 *moved)
{
struct affinity_context ac = {
.new_mask = new_mask,
+ .dl_bw_cpu = cpu,
.flags = SCA_DL_BW_RESERVED,
};
int ret;
diff --git a/kernel/sched/deadline.c b/kernel/sched/deadline.c
index 317367daa87d..915c02bc6432 100644
--- a/kernel/sched/deadline.c
+++ b/kernel/sched/deadline.c
@@ -3620,44 +3620,63 @@ static void set_cpus_allowed_dl(struct task_struct *p,
* rebuild of the accounting. Moving it without room would leave no
* root domain counting it.
*
- * Report in @ctx the bandwidth moved. The caller made its own test
- * without the runqueue lock, so it must count only what this reports.
+ * cpuset_can_attach() made room for @p in the root domain of
+ * ctx->dl_bw_cpu, and recorded that CPU with @p. Move the bandwidth
+ * there. Do not work out the CPU again: @p may have run on another
+ * CPU since. __set_cpus_allowed_ptr_locked() keeps @p in that root
+ * domain.
+ *
+ * Report in @ctx the bandwidth moved. The caller gives back the room
+ * that no move used.
*/
if ((ctx->flags & SCA_DL_BW_RESERVED) &&
- dl_task_needs_bw_move(p, ctx->new_mask)) {
+ p->dl.dl_bw_resv_cpu == ctx->dl_bw_cpu) {
unsigned int src_cpu = dl_task_root_domain_cpu(p);
struct dl_bw *src_dl_b = &cpu_rq(src_cpu)->rd->dl_bw;
- unsigned int cpu;
+ p->dl.dl_bw_resv_cpu = -1;
/*
- * We now free resources of the root_domain we are migrating
- * off. In the worst case, sched_setattr() may temporary fail
- * until we complete the update. A rebuild that dropped the
- * bandwidth of @p leaves nothing to free. dl_departed_lock
- * keeps a rebuild from changing dl_bw_gen meanwhile.
+ * Nothing moves if @p left the CPUs it was to move to, or if
+ * its root domain already counts it: @p ran there since.
*/
- guard(raw_spinlock)(&dl_departed_lock);
- if (p->dl.dl_bw_gen == dl_bw_gen) {
- raw_spin_lock(&src_dl_b->lock);
- __dl_sub(src_dl_b, p->dl.dl_bw, dl_bw_cpus(src_cpu));
- raw_spin_unlock(&src_dl_b->lock);
+ if (p->dl.dl_bw && !dl_entity_is_special(&p->dl) &&
+ cpu_active(ctx->dl_bw_cpu) &&
+ cpumask_test_cpu(ctx->dl_bw_cpu, ctx->new_mask) &&
+ cpu_rq(src_cpu)->rd != cpu_rq(ctx->dl_bw_cpu)->rd) {
+ /*
+ * We now free resources of the root_domain we are
+ * migrating off. In the worst case, sched_setattr()
+ * may temporary fail until we complete the update. A
+ * rebuild that dropped the bandwidth of @p leaves
+ * nothing to free. dl_departed_lock keeps a rebuild
+ * from changing dl_bw_gen meanwhile.
+ */
+ guard(raw_spinlock)(&dl_departed_lock);
+ if (p->dl.dl_bw_gen == dl_bw_gen) {
+ raw_spin_lock(&src_dl_b->lock);
+ __dl_sub(src_dl_b, p->dl.dl_bw, dl_bw_cpus(src_cpu));
+ raw_spin_unlock(&src_dl_b->lock);
+ }
+
+ /* Record the CPU the caller made room on. */
+ p->dl.dl_bw_cpu = ctx->dl_bw_cpu;
+ p->dl.dl_bw_gen = dl_bw_gen;
+ ctx->dl_bw_moved = p->dl.dl_bw;
}
-
- /*
- * cpuset_can_attach() made room in the root domain of the new
- * mask. That root domain counts the bandwidth from now on.
- */
- cpu = cpumask_any_and(ctx->new_mask, cpu_active_mask);
- p->dl.dl_bw_cpu = cpu < nr_cpu_ids ? (int)cpu : -1;
- p->dl.dl_bw_gen = dl_bw_gen;
- ctx->dl_bw_moved = p->dl.dl_bw;
}
set_cpus_allowed_common(p, ctx);
}
-bool dl_task_needs_bw_move(struct task_struct *p,
- const struct cpumask *new_mask)
+/*
+ * Return the CPU that a move of @p to the CPUs of @new_mask records with
+ * its bandwidth. That is the CPU @p is on, if @p can stay there. Else it
+ * is the first active CPU of @new_mask, a housekeeping one if there is
+ * one. Return -1 if the bandwidth stays in the root domain that counts
+ * it. The caller holds the rq lock of @p.
+ */
+static int __dl_task_bw_move_cpu(struct task_struct *p,
+ const struct cpumask *new_mask)
{
unsigned int cpu;
@@ -3667,10 +3686,69 @@ bool dl_task_needs_bw_move(struct task_struct *p,
* their bandwidth: dl_add_task_root_domain() leaves them out.
*/
if (!dl_task(p) || dl_entity_is_special(&p->dl) || !p->dl.dl_bw)
- return false;
+ return -1;
+ /*
+ * Test the active CPUs only. The default root domain also holds the
+ * offline CPUs, and the mask of the top cpuset holds them too.
+ */
cpu = dl_task_root_domain_cpu(p);
- return !cpumask_intersects(cpu_rq(cpu)->rd->span, new_mask);
+ if (cpumask_intersects_and(cpu_rq(cpu)->rd->span, new_mask,
+ cpu_active_mask))
+ return -1;
+ /*
+ * A task moved asleep is still on a CPU of the root domain it left.
+ * If the new mask holds that CPU, the task wakes up there.
+ */
+ cpu = task_cpu(p);
+ if (cpu_active(cpu) && cpumask_test_cpu(cpu, new_mask))
+ return cpu;
+ cpu = cpumask_first_and_and(new_mask, cpu_active_mask,
+ housekeeping_cpumask(HK_TYPE_DOMAIN_BOOT));
+ if (cpu >= nr_cpu_ids)
+ cpu = cpumask_any_and(new_mask, cpu_active_mask);
+ return cpu < nr_cpu_ids ? (int)cpu : -1;
+}
+
+/*
+ * Work out the CPU that a move of @p to the CPUs of @new_mask records
+ * with its bandwidth. Record it with @p: a cpuset attach reserves room
+ * on it. Return it, or -1 if the bandwidth stays where it is.
+ */
+int dl_task_bw_reserve_cpu(struct task_struct *p, const struct cpumask *new_mask)
+{
+ struct rq_flags rf;
+ struct rq *rq;
+ int cpu;
+
+ /* Like __set_cpus_allowed_ptr(), keep within the user affinity. */
+ rq = task_rq_lock(p, &rf);
+ if (p->user_cpus_ptr &&
+ cpumask_and(rq->scratch_mask, new_mask, p->user_cpus_ptr))
+ new_mask = rq->scratch_mask;
+ cpu = __dl_task_bw_move_cpu(p, new_mask);
+ p->dl.dl_bw_resv_cpu = cpu;
+ task_rq_unlock(rq, p, &rf);
+ return cpu;
+}
+
+/* Return the CPU that dl_task_bw_reserve_cpu() recorded with @p, or -1. */
+int dl_task_bw_reserved_cpu(struct task_struct *p)
+{
+ return READ_ONCE(p->dl.dl_bw_resv_cpu);
+}
+
+/* Forget the CPU that dl_task_bw_reserve_cpu() recorded with @p. */
+void dl_task_bw_unreserve(struct task_struct *p)
+{
+ struct rq_flags rf;
+ struct rq *rq;
+
+ if (READ_ONCE(p->dl.dl_bw_resv_cpu) < 0)
+ return;
+ rq = task_rq_lock(p, &rf);
+ p->dl.dl_bw_resv_cpu = -1;
+ task_rq_unlock(rq, p, &rf);
}
/* Assumes rq->lock is held */
@@ -4524,6 +4602,7 @@ void init_dl_entity(struct sched_dl_entity *dl_se)
dl_se->dl_bw = 0;
__dl_clear_params(dl_se);
dl_se->dl_bw_cpu = -1;
+ dl_se->dl_bw_resv_cpu = -1;
INIT_LIST_HEAD(&dl_se->dl_departed_node);
}
diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h
index 65fc17909d56..a719b62dc114 100644
--- a/kernel/sched/sched.h
+++ b/kernel/sched/sched.h
@@ -2680,6 +2680,8 @@ struct affinity_context {
struct cpumask *user_mask;
/* Out: the deadline bandwidth moved to the new root domain. */
u64 dl_bw_moved;
+ /* In, with SCA_DL_BW_RESERVED: the CPU the caller made room on. */
+ int dl_bw_cpu;
unsigned int flags;
};
diff --git a/kernel/sched/syscalls.c b/kernel/sched/syscalls.c
index ae595a4e7a8f..80de69646766 100644
--- a/kernel/sched/syscalls.c
+++ b/kernel/sched/syscalls.c
@@ -1346,8 +1346,9 @@ long sched_setaffinity(pid_t pid, const struct cpumask *in_mask)
/*
* Wait for a cpuset attach in progress before changing the affinity
* of a deadline task. cpuset_can_attach() reserved the bandwidth of
- * the task for the CPUs it moves to, and cpuset_attach() moves it
- * there. A new affinity in between could change those CPUs.
+ * the task on one CPU, and recorded that CPU with the task.
+ * cpuset_attach() moves the bandwidth there. A new affinity in
+ * between could leave that CPU out, and the bandwidth would not move.
*
* The policy is tested here with no lock. A task that becomes a
* deadline task after this test is found under its runqueue lock,
--
2.47.3
^ permalink raw reply related [flat|nested] 35+ messages in thread
* [PATCH v8 12/33] selftests/sched: Test cpuset moves in deadline_bw_test
2026-11-19 20:15 [PATCH v8 00/33] sched/deadline: bandwidth accounting fixes and selftests Mathieu Desnoyers
` (10 preceding siblings ...)
2026-11-19 20:15 ` [PATCH v8 11/33] cgroup/cpuset: Reserve the bandwidth in the root domain a task moves to Mathieu Desnoyers
@ 2026-11-19 20:15 ` Mathieu Desnoyers
2026-11-19 20:15 ` [PATCH v8 13/33] sched/deadline: Make a rebuild count a task in the root domain it runs in Mathieu Desnoyers
` (20 subsequent siblings)
32 siblings, 0 replies; 35+ messages in thread
From: Mathieu Desnoyers @ 2026-11-19 20:15 UTC (permalink / raw)
To: juri.lelli, peterz, mingo, vincent.guittot
Cc: Mathieu Desnoyers, tglx, will, boqun, dietmar.eggemann, rostedt,
bsegall, mgorman, vschneid, kprateek.nayak, luca.abeni,
mathieu.poirier, longman, ridong.chen, tj, hannes, mkoutny, shuah,
bigeasy, clrkwllms, parri.andrea, m.szyprowski, kayracizmeci,
weihengqiuu, linux-kernel, cgroups, linux-kselftest,
linux-rt-devel
Add the 22 cases of deadline_bw_test that test the fixes of cpuset
attach, earlier in this series. They cover cpuset moves, and what can
happen during them:
- a rebuild of the sched domains, a policy change, an affinity change, a
wakeup, the death of a thread, a refusal by another controller
- a task asleep, throttled, yielded, frozen or reclaiming
- moves between the housekeeping CPUs and CPUs isolated by
isolcpus=domain
- sysrq-n during a move (move_normalized)
- a random mix of all of the above, with affinity changes (random_mix)
The comment above each case gives its scenario. random_mix also
compares the this_bw of its CPUs. Five cases need CPUs isolated by
isolcpus=domain. move_normalized fires sysrq-n, which changes the whole
system, so it needs DEADLINE_TEST_SYSRQ=1. A case skips without what it
needs.
The message of each fix names the cases that test it. These cases are
not named there:
- move_asleep, move_non_contending, move_died, move_woken and
move_woken_full test "sched/deadline: Record the root domain holding a
task's bandwidth". They also need the attach fixes, so they come here.
- frozen_moved_switched, move_throttled, reclaim_moved and yield_moved
fail with isolcpus=domain,0,1 before the attach fixes.
On v7.4-rc2, in 8 VMs of 8 CPUs, twelve of these cases fail in every VM:
move_affinity_became_deadline move_died
move_affinity_changed move_non_contending
move_asleep move_vs_rebuild
move_back_cpu_offline move_woken
move_became_deadline move_woken_full
move_became_deadline_no_room random_mix
The cases that need isolcpus=domain skip there. move_normalized did not
run there.
With the fixes up to here, every case passes wherever it runs.
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
---
.../sched/deadline/deadline_bw_test.c | 2384 ++++++++++++++++-
1 file changed, 2379 insertions(+), 5 deletions(-)
diff --git a/tools/testing/selftests/sched/deadline/deadline_bw_test.c b/tools/testing/selftests/sched/deadline/deadline_bw_test.c
index ba2a18f354d4..fa1ff62d6f42 100644
--- a/tools/testing/selftests/sched/deadline/deadline_bw_test.c
+++ b/tools/testing/selftests/sched/deadline/deadline_bw_test.c
@@ -5,9 +5,11 @@
* - priority inheritance
* - parameter changes
* - cpuset moves
+ * - affinity changes
* - deaths, yields and runtime overruns
* - bandwidth reclaiming and fair server changes
* - CPU hotplug
+ * - a random mix of most of these
*
* Most cases measure the free bandwidth of a root domain before and after
* they run. probe() counts the tasks of 10ms every 100ms that the root
@@ -25,7 +27,8 @@
* The CPUs that isolcpus="domain" isolates are in the default root
* domain: the root domain of the isolated CPUs. The other CPUs are the
* housekeeping CPUs, in the housekeeping root domain. The test leaves the
- * isolated CPUs out, and keeps its tasks off them.
+ * isolated CPUs out, and keeps its tasks off them. Only the cases that
+ * need them use them.
*
* The cases of p_cases[] run with P, a partition root of the last
* housekeeping CPU. P is made anew for each case. Making it rebuilds the
@@ -42,7 +45,8 @@
*
* The test needs root, deadline admission control and 3 housekeeping
* CPUs. Some cases need 4. Most also need the cgroup v2 cpuset
- * controller. Some need debugfs or CPU hotplug.
+ * controller. Some need debugfs or CPU hotplug. Five need
+ * isolcpus="domain" at boot.
*/
#define _GNU_SOURCE
#include <errno.h>
@@ -106,6 +110,7 @@ struct dl_attr {
static char root[PATH_MAX], cg_p[PATH_MAX], cg_q[PATH_MAX];
/* The online CPUs at the start, ascending, except the isolated ones */
static int ncpus, *online;
+static int nisol, *isol; /* the online CPUs that isolcpus="domain" isolates */
static char isolated[4096]; /* the CPU list of /sys/devices/system/cpu/isolated */
/* Return the @n-th highest online CPU, counting from 1. */
@@ -2539,7 +2544,7 @@ static int child_up(const char *what, int fast)
return 0;
}
-enum mv_mode { MV_ASLEEP, MV_THROTTLED, MV_NON_CONTENDING };
+enum mv_mode { MV_ASLEEP, MV_THROTTLED, MV_YIELDED, MV_NON_CONTENDING };
/*
* Fork a process in the caller's cgroup. The process becomes a deadline
@@ -2547,6 +2552,8 @@ enum mv_mode { MV_ASLEEP, MV_THROTTLED, MV_NON_CONTENDING };
* - MV_ASLEEP: run 1ms, set mv_state[0] to 1, then sleep.
* - MV_THROTTLED: set mv_state[0] to 1, then spin and count in
* mv_state[1].
+ * - MV_YIELDED: set mv_state[0] to 1. Then spin 0.5ms, count in
+ * mv_state[1] and yield, in a loop.
* - MV_NON_CONTENDING: sleep a period, run a quarter of its runtime, set
* mv_state[0] to 1, then sleep. Its 0-lag time is still ahead.
* If it is refused SCHED_DEADLINE, the process sets mv_state[0] to -1 and
@@ -2574,6 +2581,14 @@ static pid_t mover(enum mv_mode mode, uint64_t runtime)
for (;;)
pause();
}
+ if (mode == MV_YIELDED) {
+ *mv_state = 1;
+ for (;;) {
+ spin_ns(NSEC_PER_MSEC / 2);
+ mv_state[1]++;
+ sched_yield();
+ }
+ }
*mv_state = 1;
for (;;)
mv_state[1]++;
@@ -2581,6 +2596,69 @@ static pid_t mover(enum mv_mode mode, uint64_t runtime)
return pid;
}
+static int wait_spinner(int moving);
+
+/*
+ * Move a deadline task into P and back, from outside P. The task is a
+ * mover() process of 30ms every 100ms, forked in the cgroup of the main
+ * process. Wait until it is in the state of @mode:
+ * - MV_ASLEEP: asleep for two periods, past its 0-lag time.
+ * - MV_NON_CONTENDING: asleep for 1ms, before its 0-lag time.
+ * - MV_THROTTLED, MV_YIELDED: its counter has stopped for 5ms.
+ * Then write its pid to cgroup.procs of P, then of the root cgroup. Both
+ * moves must succeed. A throttled or yielded task must run again within
+ * three periods. Then kill the task. P and the root cgroup must each
+ * admit as many probe tasks as before the case.
+ */
+static int moves(enum mv_mode mode, const char *name)
+{
+ int base_p, base_r, after_p, after_r, ret = KSFT_PASS;
+ pid_t pid;
+
+ base_p = probe(cg_p);
+ base_r = probe(root);
+ mv_state[0] = 0;
+ mv_state[1] = 0;
+ pid = mover(mode, 30 * NSEC_PER_MSEC);
+ if (pid < 0 || child_up(name, mode == MV_NON_CONTENDING)) {
+ reap(pid);
+ return KSFT_FAIL;
+ }
+ if (mode == MV_NON_CONTENDING) {
+ sleep_ns(NSEC_PER_MSEC);
+ } else if (mode == MV_ASLEEP) {
+ sleep_ns(2 * PERIOD);
+ } else if (wait_spinner(0)) {
+ /* Throttled, or yielded: its counter stops. */
+ ksft_print_msg("%s: never stopped\n", name);
+ reap(pid);
+ return KSFT_FAIL;
+ }
+ if (enter(cg_p, pid) || enter(root, pid)) {
+ ksft_print_msg("%s: a move refused\n", name);
+ ret = KSFT_FAIL;
+ }
+ if (mode == MV_THROTTLED || mode == MV_YIELDED) {
+ int last = mv_state[1];
+
+ sleep_ns(3 * PERIOD);
+ if (mv_state[1] == last) {
+ ksft_print_msg("%s: moved, never ran again\n", name);
+ ret = KSFT_FAIL;
+ }
+ }
+ reap(pid);
+ sleep_ns(3 * PERIOD);
+ after_p = probe(cg_p);
+ after_r = probe(root);
+ if (ret == KSFT_PASS && (after_p != base_p || after_r != base_r)) {
+ ksft_print_msg("%s: probe tasks admitted, P %d -> %d, the other CPUs %d -> %d\n",
+ name, base_p, after_p, base_r, after_r);
+ ret = KSFT_FAIL;
+ }
+ return ret;
+}
+
/*
* A deadline task of 60ms every 100ms sleeps in P. Move a second sleeping
* task of 60ms every 100ms into P from outside. The move must fail with
@@ -2722,6 +2800,44 @@ static int move_zero_bw(void)
return ret;
}
+static int move_asleep(void)
+{
+ return moves(MV_ASLEEP, "move_asleep");
+}
+
+static int move_throttled(void)
+{
+ return moves(MV_THROTTLED, "move_throttled");
+}
+
+static int move_non_contending(void)
+{
+ return moves(MV_NON_CONTENDING, "move_non_contending");
+}
+
+/*
+ * Fork a process that reads sched_rt_runtime_us and writes the same value
+ * back @n times, 5ms apart. Then it exits. Each write rebuilds the sched
+ * domains. Return its pid, or -1.
+ */
+static pid_t rebuilder(int n)
+{
+ pid_t pid = fork_child("a sysctl writer");
+
+ if (!pid) {
+ char buf[64];
+
+ if (read_file("/proc/sys/kernel/sched_rt_runtime_us", buf, sizeof(buf)))
+ _exit(1);
+ while (n-- > 0) {
+ write_file("/proc/sys/kernel", "sched_rt_runtime_us", buf);
+ sleep_ns(5 * NSEC_PER_MSEC);
+ }
+ _exit(0);
+ }
+ return pid;
+}
+
static void *pauser(void *arg)
{
for (;;)
@@ -2729,9 +2845,200 @@ static void *pauser(void *arg)
return arg;
}
-/* What late_setter() does, and the result. */
+/*
+ * Run as the deadline thread of wide_mover(): a deadline task of 30% that
+ * runs 1ms, sets mv_state[0] to 1 and sleeps.
+ */
+static void *dl_pauser(void *arg)
+{
+ if (set_deadline(0, 30 * NSEC_PER_MSEC, PERIOD)) {
+ mv_state[0] = -1;
+ return arg;
+ }
+ spin_ns(NSEC_PER_MSEC);
+ *mv_state = 1;
+ for (;;)
+ pause();
+ return arg;
+}
+
+/*
+ * Fork a process of @threads sleeping threads, then one deadline thread,
+ * dl_pauser(). A cgroup move of the process moves its threads one by one,
+ * between cpuset_can_attach() and cpuset_attach(). The deadline thread
+ * moves last, so that a rebuild in that window can still find it in the
+ * cgroup it leaves. Return the pid, or -1.
+ */
+static pid_t wide_mover(int threads)
+{
+ pid_t pid = fork_child("a wide deadline process");
+
+ if (!pid) {
+ pthread_attr_t a;
+ pthread_t t;
+ int i;
+
+ if (pthread_attr_init(&a) || pthread_attr_setstacksize(&a, THREAD_STACK))
+ goto no_threads;
+ for (i = 0; i < threads; i++) {
+ if (pthread_create(&t, &a, pauser, NULL))
+ goto no_threads;
+ }
+ if (pthread_create(&t, NULL, dl_pauser, NULL))
+ goto no_threads;
+ for (;;)
+ pause();
+no_threads:
+ mv_state[0] = -2;
+ _exit(1);
+ }
+ return pid;
+}
+
+/*
+ * Move a process in and out of P while rebuilder() rebuilds the sched
+ * domains. The process is wide_mover(): 2048 sleeping threads, then one
+ * sleeping deadline thread of 30%. rebuilder() writes the
+ * sched_rt_runtime_us sysctl 20 times, 5ms apart. Each write is a
+ * rebuild. Without the 5ms gaps, the rebuilds would be over before the
+ * first move reaches the window below.
+ *
+ * A rebuild between cpuset_can_attach() and cpuset_attach() would drop
+ * the bandwidth that cpuset_can_attach() reserved in the destination.
+ * cpuset_attach() would then move the bandwidth of the thread all the
+ * same. So a rebuild must wait for the end of a move in progress.
+ *
+ * A later rebuild corrects the accounting. So move until rebuilder()
+ * exits: no rebuild follows the last move. The free bandwidth of P and
+ * of the other CPUs must then be what it was with the process asleep in
+ * the root cgroup. Run six rounds.
+ */
+static int move_vs_rebuild(void)
+{
+ int base_p, base_r, with_p, with_r, after_p, after_r, round, ret = KSFT_PASS;
+ const char *name = "move_vs_rebuild";
+ pid_t pid, writer;
+
+ base_p = probe(cg_p);
+ base_r = probe(root);
+ mv_state[0] = 0;
+ pid = wide_mover(2048);
+ if (pid < 0 || child_up(name, 0)) {
+ reap(pid);
+ return mv_state[0] == -2 ? KSFT_SKIP : KSFT_FAIL;
+ }
+ /* The reference for the rounds: the task asleep in the root cgroup. */
+ sleep_ns(2 * PERIOD);
+ with_p = probe(cg_p);
+ with_r = probe(root);
+ for (round = 0; round < 6 && ret == KSFT_PASS; round++) {
+ int status, err = 0;
+
+ writer = rebuilder(20);
+ if (writer < 0) {
+ ret = KSFT_FAIL;
+ break;
+ }
+ while (!err && waitpid(writer, &status, WNOHANG) == 0) {
+ err = enter(cg_p, pid);
+ if (!err)
+ err = enter(root, pid);
+ }
+ if (err) {
+ ksft_print_msg("%s: a move refused: %s\n", name, strerror(-err));
+ reap(writer);
+ ret = KSFT_FAIL;
+ break;
+ }
+ sleep_ns(3 * PERIOD);
+ after_p = probe(cg_p);
+ after_r = probe(root);
+ if (after_p != with_p || after_r != with_r) {
+ ksft_print_msg("%s: round %d: P %d -> %d probe tasks, the rest %d -> %d\n",
+ name, round, with_p, after_p, with_r, after_r);
+ ret = KSFT_FAIL;
+ }
+ }
+ reap(pid);
+ sleep_ns(3 * PERIOD);
+ after_p = probe(cg_p);
+ after_r = probe(root);
+ if (ret == KSFT_PASS && (after_p != base_p || after_r != base_r)) {
+ ksft_print_msg("%s: task gone: P %d -> %d probe tasks, the rest %d -> %d\n",
+ name, base_p, after_p, base_r, after_r);
+ ret = KSFT_FAIL;
+ }
+ return ret;
+}
+
+/* The runtime every 100ms of the deadline threads of late_mover() */
+static uint64_t late_thread_runtime;
+
+/*
+ * Run as a deadline thread of late_mover(): a deadline task of
+ * late_thread_runtime that counts itself in mv_state[1] and sleeps.
+ */
+static void *dl_sleeper(void *arg)
+{
+ if (set_deadline(0, late_thread_runtime, PERIOD)) {
+ mv_state[0] = -1;
+ return arg;
+ }
+ mv_state[1]++;
+ for (;;)
+ pause();
+ return arg;
+}
+
+/*
+ * Fork a process of two deadline threads, dl_sleeper(), then @threads
+ * sleeping threads. Its main thread, the thread group leader, sleeps and
+ * is not a deadline task. A cgroup move of the process handles the
+ * leader first. The sleeping threads make the move long. Return the pid,
+ * or -1.
+ */
+static pid_t late_mover(int threads)
+{
+ pid_t pid = fork_child("a wide process");
+
+ if (!pid) {
+ pthread_attr_t a;
+ pthread_t t;
+ int i;
+
+ if (pthread_attr_init(&a) || pthread_attr_setstacksize(&a, THREAD_STACK))
+ goto no_threads;
+ for (i = 0; i < 2; i++) {
+ if (pthread_create(&t, &a, dl_sleeper, NULL))
+ goto no_threads;
+ }
+ for (i = 0; i < threads; i++) {
+ if (pthread_create(&t, &a, pauser, NULL))
+ goto no_threads;
+ }
+ if (wait_for_step(5000, NSEC_PER_MSEC, mv_state[1] == 2 || mv_state[0]))
+ goto no_threads;
+ if (!mv_state[0])
+ mv_state[0] = 1;
+ for (;;)
+ pause();
+no_threads:
+ mv_state[0] = -2;
+ _exit(1);
+ }
+ return pid;
+}
+
+/*
+ * What late_setter(), late_pinner() and late_killer() do, and the result.
+ * Each acts on thread @pid, @delay ns after @go is set. It stores its
+ * result in @ret and the time it returned in @done. late_setter() uses
+ * @runtime. late_killer() uses @tgid and @kill.
+ */
static struct {
pid_t pid;
+ pid_t tgid;
+ int kill;
uint64_t runtime;
uint64_t delay;
atomic_int go;
@@ -2758,18 +3065,167 @@ static void *late_setter(void *arg)
return arg;
}
+/*
+ * Make the main thread of a process a deadline task while the process
+ * moves from the root cgroup into P.
+ *
+ * The process is late_mover(). Its main thread sleeps and is not a
+ * deadline task. Two of its threads are sleeping deadline tasks of
+ * @thread percent each. 2048 more sleeping threads make the move long.
+ * A thread of the test, late_setter(), calls sched_setattr() on the main
+ * thread to make it a deadline task of @leader percent.
+ *
+ * cpuset_can_attach() reserves room in P for the two deadline threads.
+ * If the main thread became a deadline task after that, no room would be
+ * reserved for it. cpuset_attach() handles the main thread first.
+ *
+ * sched_setattr() waits for the end of a move in progress. So it runs
+ * before the move or after it:
+ * - Before: the move counts three deadline tasks. With room in P for
+ * the three, the move must succeed. P must then count the three, and
+ * the root domain the process left must count none. Without room,
+ * the move must be refused with EBUSY.
+ * - After: sched_setattr() is refused with EPERM. The main thread is
+ * then allowed on P's CPU only. It is still asleep on a CPU of the
+ * root domain it left.
+ * Without room in P for the three, the case fails if the move and
+ * sched_setattr() both succeed.
+ *
+ * Try 30 delays of sched_setattr(), spread over the time a move takes.
+ */
+static int became_deadline(int thread, int leader, const char *name)
+{
+ int base_p, base_r, ret = KSFT_PASS, i, raced = 0;
+ int need = (2 * thread + leader) / 10, fits;
+ uint64_t start, len;
+ pid_t pid;
+
+ base_p = probe(cg_p);
+ base_r = probe(root);
+ fits = base_p >= need;
+ if (base_p < 2 * thread / 10) {
+ ksft_print_msg("%s: P admits %d probe tasks, %d needed\n", name, base_p,
+ 2 * thread / 10);
+ return KSFT_SKIP;
+ }
+ late_thread_runtime = thread * NSEC_PER_MSEC;
+ late.runtime = leader * NSEC_PER_MSEC;
+ mv_state[0] = 0;
+ mv_state[1] = 0;
+ pid = late_mover(2048);
+ if (pid < 0 || child_up(name, 0)) {
+ reap(pid);
+ return mv_state[0] == -2 ? KSFT_SKIP : KSFT_FAIL;
+ }
+ /* Time one move, to spread the delays over it. */
+ start = now_ns();
+ if (enter(cg_p, pid) || enter(root, pid)) {
+ ksft_print_msg("%s: a move refused\n", name);
+ reap(pid);
+ return KSFT_FAIL;
+ }
+ len = (now_ns() - start) / 2;
+ sleep_ns(2 * PERIOD);
+
+ late.pid = pid;
+ for (i = 0; i < 30 && ret == KSFT_PASS; i++) {
+ pthread_t t;
+ int err;
+
+ late.go = 0;
+ late.delay = len * i / 30;
+ if (pthread_create(&t, NULL, late_setter, NULL)) {
+ ret = KSFT_FAIL;
+ break;
+ }
+ sleep_ns(NSEC_PER_MSEC);
+ late.go = 1;
+ err = enter(cg_p, pid);
+ join(t);
+ if (err && (fits || err != -EBUSY || late.ret)) {
+ ksft_print_msg("%s: the move refused: %s\n", name, strerror(-err));
+ ret = KSFT_FAIL;
+ } else if (!err && !late.ret && !fits) {
+ ksft_print_msg("%s: the move and the change both succeeded, with no room in P\n",
+ name);
+ ret = KSFT_FAIL;
+ } else if (!err && !late.ret) {
+ raced++;
+ if (admits_in(cg_p, base_p - need, "P, with the three tasks") ||
+ admits_in(root, base_r, "the root domain the tasks left"))
+ ret = KSFT_FAIL;
+ } else if (late.ret && late.ret != -EPERM) {
+ ksft_print_msg("%s: sched_setattr: %s\n", name, strerror(-late.ret));
+ ret = KSFT_FAIL;
+ }
+ if (ret == KSFT_FAIL)
+ ksft_print_msg("%s: try %d, the change %llu us into a move of %llu us\n",
+ name, i, (unsigned long long)(late.delay / NSEC_PER_USEC),
+ (unsigned long long)(len / NSEC_PER_USEC));
+ if ((!late.ret && set_other(pid)) || enter(root, pid)) {
+ ksft_print_msg("%s: the process cannot go back\n", name);
+ ret = KSFT_FAIL;
+ }
+ sleep_ns(2 * PERIOD);
+ }
+ if (ret == KSFT_PASS && fits && !raced)
+ ksft_print_msg("%s: no change succeeded with a move\n", name);
+ reap(pid);
+ sleep_ns(3 * PERIOD);
+ if (ret == KSFT_PASS && (probe(cg_p) != base_p || probe(root) != base_r)) {
+ ksft_print_msg("%s: the free bandwidth changed\n", name);
+ ret = KSFT_FAIL;
+ }
+ return ret;
+}
+
+/* Run it with room in P: two threads of 20% and a main thread of 30%. */
+static int move_became_deadline(void)
+{
+ return became_deadline(20, 30, "move_became_deadline");
+}
+
+/* Run it without room in P: two threads of 30% and a main thread of 40%. */
+static int move_became_deadline_no_room(void)
+{
+ return became_deadline(30, 40, "move_became_deadline_no_room");
+}
+
static void no_op(int sig)
{
}
+/* The CPU that dl_sleeper_tid() first runs on, or -1 for any CPU */
+static int sleeper_cpu = -1;
+
/*
* Run as the deadline thread of pinned_mover(): a deadline task of 30%.
* Store the tid in mv_state[3], set mv_state[0] to 1 and sleep. A signal
* wakes the thread up. It then sleeps again. If mv_state[4] is set, this
* thread exits instead, and the process stays.
+ *
+ * With sleeper_cpu set, first run 1ms pinned on that CPU, then restore
+ * the previous affinity. Do this before sched_setattr(): the affinity of
+ * a deadline task must include every CPU of its root domain.
*/
static void *dl_sleeper_tid(void *arg)
{
+ if (sleeper_cpu >= 0) {
+ cpu_set_t all, one;
+
+ CPU_ZERO(&one);
+ CPU_SET(sleeper_cpu, &one);
+ if (sched_getaffinity(0, sizeof(all), &all) ||
+ sched_setaffinity(0, sizeof(one), &one)) {
+ mv_state[0] = -2;
+ return arg;
+ }
+ spin_ns(NSEC_PER_MSEC);
+ if (sched_setaffinity(0, sizeof(all), &all)) {
+ mv_state[0] = -2;
+ return arg;
+ }
+ }
if (set_deadline(0, 30 * NSEC_PER_MSEC, PERIOD)) {
mv_state[0] = -1;
return arg;
@@ -2834,6 +3290,618 @@ static pid_t pinned_mover(int threads, int fifo)
return pid;
}
+/*
+ * Restrict late.pid to the CPU of P and the CPU in @arg, late.delay ns
+ * after late.go.
+ */
+static void *late_pinner(void *arg)
+{
+ uint64_t start;
+ cpu_set_t set;
+
+ CPU_ZERO(&set);
+ CPU_SET(top_cpu(1), &set);
+ CPU_SET((long)arg, &set);
+ while (!late.go)
+ ;
+ start = now_ns();
+ while (now_ns() - start < late.delay)
+ ;
+ late.ret = sched_setaffinity(late.pid, sizeof(set), &set) ? -errno : 0;
+ late.done = now_ns();
+ return NULL;
+}
+
+/*
+ * Change the affinity of a sleeping deadline thread while its process
+ * moves from P to the root cgroup.
+ *
+ * The process is pinned_mover(): 2048 sleeping threads, then a deadline
+ * thread of 30%. That thread went to sleep on a CPU of the root cgroup,
+ * and was moved into P asleep. So P counts it, and it is still on that
+ * CPU. A thread of the test, late_pinner(), calls sched_setaffinity()
+ * on it.
+ *
+ * cpuset_can_attach() reserves the bandwidth of the thread on one CPU of
+ * the root cgroup. The move records that CPU. It is the CPU the thread
+ * is on, if its affinity allows that CPU. If not, it is the first CPU of
+ * the root cgroup that its affinity allows.
+ *
+ * sched_setaffinity() restricts the thread to two CPUs: the CPU of P,
+ * and a CPU of the root cgroup that the thread is not on. The affinity
+ * of a deadline task must include every CPU of the root domain that
+ * counts it, so the mask needs the CPU of P. The move then records the
+ * second CPU.
+ *
+ * sched_setaffinity() waits for the end of a move in progress. So it
+ * runs before the move or after it:
+ * - Before: the bandwidth is reserved on the CPU the move records. The
+ * root domain of that CPU must count the thread once the move is
+ * done, and P must no longer count it.
+ * - After: the thread is counted in the root domain of the root
+ * cgroup. The mask does not include every CPU of that root domain.
+ * sched_setaffinity() is refused with EBUSY.
+ *
+ * Try 20 delays of sched_setaffinity(), spread over the time a move
+ * takes.
+ */
+static int move_affinity_changed(void)
+{
+ int with_p, with_r, ret = KSFT_PASS, i, raced = 0;
+ const char *name = "move_affinity_changed";
+ uint64_t start, len;
+ cpu_set_t all;
+ pid_t pid;
+
+ CPU_ZERO(&all);
+ for (i = 0; i < ncpus; i++)
+ CPU_SET(online[i], &all);
+ mv_state[0] = 0;
+ mv_state[3] = 0;
+ mv_state[4] = 0;
+ pid = pinned_mover(2048, 0);
+ if (pid < 0 || child_up(name, 0)) {
+ reap(pid);
+ return mv_state[0] == -2 ? KSFT_SKIP : KSFT_FAIL;
+ }
+ /* The reference: the thread asleep in the root cgroup. */
+ sleep_ns(2 * PERIOD);
+ with_p = probe(cg_p);
+ with_r = probe(root);
+ /* Time one move, to spread the delays over it. */
+ start = now_ns();
+ if (enter(cg_p, pid) || enter(root, pid)) {
+ ksft_print_msg("%s: a move refused\n", name);
+ reap(pid);
+ return KSFT_FAIL;
+ }
+ len = (now_ns() - start) / 2;
+
+ late.pid = mv_state[3];
+ for (i = 0; i < 20 && ret == KSFT_PASS; i++) {
+ pthread_t t;
+ long cpu;
+ int err;
+
+ if (sched_setaffinity(late.pid, sizeof(all), &all) || enter(cg_p, pid)) {
+ ksft_print_msg("%s: the process cannot go into P\n", name);
+ ret = KSFT_FAIL;
+ break;
+ }
+ /* Not the CPU the thread is on: the move would record that one. */
+ cpu = task_cpu_of(late.pid) == online[1] ? online[0] : online[1];
+ late.go = 0;
+ late.delay = len * i / 20;
+ if (pthread_create(&t, NULL, late_pinner, (void *)cpu)) {
+ ret = KSFT_FAIL;
+ break;
+ }
+ sleep_ns(NSEC_PER_MSEC);
+ late.go = 1;
+ err = enter(root, pid);
+ join(t);
+ if (err) {
+ ksft_print_msg("%s: the move refused: %s\n", name, strerror(-err));
+ ret = KSFT_FAIL;
+ } else if (!late.ret) {
+ raced++;
+ sleep_ns(2 * PERIOD);
+ if (admits_in(cg_p, with_p, "P, which the thread left") ||
+ admits_in(root, with_r, "the root domain of the thread")) {
+ ksft_print_msg("%s: try %d, the change %llu us into a move of %llu us\n",
+ name, i, (unsigned long long)(late.delay / NSEC_PER_USEC),
+ (unsigned long long)(len / NSEC_PER_USEC));
+ ret = KSFT_FAIL;
+ }
+ } else if (late.ret && late.ret != -EBUSY) {
+ ksft_print_msg("%s: sched_setaffinity: %s\n", name, strerror(-late.ret));
+ ret = KSFT_FAIL;
+ }
+ }
+ if (ret == KSFT_PASS && !raced)
+ ksft_print_msg("%s: no change succeeded\n", name);
+ reap(pid);
+ return ret;
+}
+
+/*
+ * Make a thread a deadline task while sched_setaffinity() runs on it, and
+ * move its process from P to the root cgroup meanwhile.
+ *
+ * sched_setaffinity() waits for the end of a move in progress when the
+ * task is a deadline task. It tests the policy first, with no lock. Here
+ * the thread T is SCHED_OTHER at that test:
+ *
+ * sched_setaffinity(T) sched_setattr(T) echo PID > cgroup.procs
+ *
+ * T is SCHED_OTHER: no wait
+ * T is a deadline task
+ * cpuset_can_attach()
+ * reserves room for T
+ * changes the affinity of T
+ * cpuset_attach()
+ * moves T
+ *
+ * The new affinity is the CPU of P and one CPU of the root cgroup, which
+ * is not the CPU the room was reserved on. Once the move is done, the
+ * root domain of the root cgroup must count T, and P must not.
+ *
+ * Delay injection: the window after the test of the policy is a few
+ * instructions wide. The case only reaches it on a kernel that can make
+ * sched_setaffinity() sleep there. It passes without reaching it
+ * otherwise.
+ */
+static int move_affinity_became_deadline(void)
+{
+ const char *name = "move_affinity_became_deadline";
+ int with_p, with_r, ret = KSFT_PASS, i;
+ uint64_t start, len, delay_ms;
+ cpu_set_t all;
+ pid_t pid;
+
+ CPU_ZERO(&all);
+ for (i = 0; i < ncpus; i++)
+ CPU_SET(online[i], &all);
+ mv_state[0] = 0;
+ mv_state[3] = 0;
+ mv_state[4] = 0;
+ pid = pinned_mover(2048, 0);
+ if (pid < 0 || child_up(name, 0)) {
+ reap(pid);
+ return mv_state[0] == -2 ? KSFT_SKIP : KSFT_FAIL;
+ }
+ /* The reference: the thread a deadline task, asleep in the root cgroup. */
+ sleep_ns(2 * PERIOD);
+ with_p = probe(cg_p);
+ with_r = probe(root);
+ /* Time one move, to place the change inside it. */
+ start = now_ns();
+ if (enter(cg_p, pid) || enter(root, pid)) {
+ ksft_print_msg("%s: a move refused\n", name);
+ reap(pid);
+ return KSFT_FAIL;
+ }
+ len = (now_ns() - start) / 2;
+ delay_ms = 3 * len / NSEC_PER_MSEC + 30;
+
+ late.pid = mv_state[3];
+ for (i = 0; i < 10 && ret == KSFT_PASS; i++) {
+ uint64_t t0, at;
+ pthread_t t;
+ int err;
+
+ if (sched_setaffinity(late.pid, sizeof(all), &all) || enter(cg_p, pid) ||
+ set_other(late.pid)) {
+ ksft_print_msg("%s: the thread cannot go into P as SCHED_OTHER\n", name);
+ ret = KSFT_FAIL;
+ break;
+ }
+ /*
+ * Wake the thread once, so that it runs on the CPU of P. A
+ * thread moved asleep is refused SCHED_DEADLINE with EPERM.
+ */
+ if (syscall(SYS_tgkill, pid, late.pid, SIGUSR1)) {
+ ret = KSFT_FAIL;
+ break;
+ }
+ sleep_ns(5 * NSEC_PER_MSEC);
+ /*
+ * Delay injection: sched_setaffinity() would be asked here to
+ * sleep delay_ms after its test of the policy.
+ */
+ late.go = 0;
+ late.delay = 0;
+ if (pthread_create(&t, NULL, late_pinner, (void *)(long)online[1])) {
+ ret = KSFT_FAIL;
+ break;
+ }
+ sleep_ns(NSEC_PER_MSEC);
+ t0 = now_ns();
+ late.go = 1;
+ /* With the delay, the change has tested the policy and sleeps. */
+ sleep_ns(5 * NSEC_PER_MSEC);
+ err = set_deadline(late.pid, 30 * NSEC_PER_MSEC, PERIOD);
+ if (err) {
+ ksft_print_msg("%s: sched_setattr: %s\n", name, strerror(-err));
+ ret = KSFT_FAIL;
+ }
+ /* Start the move so that the change wakes up inside it. */
+ at = t0 + delay_ms * NSEC_PER_MSEC - len * (i + 1) / 11;
+ while (now_ns() < at)
+ ;
+ err = enter(root, pid);
+ join(t);
+ if (err) {
+ ksft_print_msg("%s: the move refused: %s\n", name, strerror(-err));
+ ret = KSFT_FAIL;
+ } else if (!late.ret) {
+ sleep_ns(2 * PERIOD);
+ if (admits_in(cg_p, with_p, "P, which the thread left") ||
+ admits_in(root, with_r, "the root domain of the thread")) {
+ ksft_print_msg("%s: try %d, the change landed in a move of %llu us\n",
+ name, i, (unsigned long long)(len / NSEC_PER_USEC));
+ ret = KSFT_FAIL;
+ }
+ } else if (late.ret && late.ret != -EBUSY) {
+ ksft_print_msg("%s: sched_setaffinity: %s\n", name, strerror(-late.ret));
+ ret = KSFT_FAIL;
+ }
+ }
+ reap(pid);
+ return ret;
+}
+
+/*
+ * Wake thread late.pid of process late.tgid up with a signal, late.delay
+ * ns after late.go. With late.kill, the thread then exits.
+ */
+static void *late_killer(void *arg)
+{
+ uint64_t start;
+
+ while (!late.go)
+ ;
+ start = now_ns();
+ while (now_ns() - start < late.delay)
+ ;
+ mv_state[4] = late.kill;
+ late.ret = syscall(SYS_tgkill, late.tgid, late.pid, SIGUSR1) ? -errno : 0;
+ late.done = now_ns();
+ return arg;
+}
+
+/*
+ * Kill a sleeping deadline thread of 30% while its process moves into
+ * P. A thread that is part of a move cannot exit before the move ends:
+ * its exit waits for it. So a thread killed during the move dies right
+ * after it, and must release its bandwidth where the move left it. Once the process
+ * is gone, the free bandwidth of P and of the root domain the process
+ * left must be what it was.
+ *
+ * Try 12 delays spread over the time a move takes, each with a new
+ * process.
+ */
+static int move_died(void)
+{
+ int base_p, base_r, ret = KSFT_PASS, i;
+ const char *name = "move_died";
+ uint64_t len = 0;
+
+ base_p = probe(cg_p);
+ base_r = probe(root);
+ for (i = 0; i < 12 && ret == KSFT_PASS; i++) {
+ pthread_t t;
+ pid_t pid;
+ int err;
+
+ mv_state[0] = 0;
+ mv_state[3] = 0;
+ mv_state[4] = 0;
+ pid = pinned_mover(2048, 0);
+ if (pid < 0 || child_up(name, 0)) {
+ reap(pid);
+ return mv_state[0] == -2 ? KSFT_SKIP : KSFT_FAIL;
+ }
+ if (!i) {
+ /* Time one move, to spread the delays over it. */
+ uint64_t start = now_ns();
+
+ if (enter(cg_p, pid) || enter(root, pid)) {
+ ksft_print_msg("%s: a move refused\n", name);
+ reap(pid);
+ return KSFT_FAIL;
+ }
+ len = (now_ns() - start) / 2;
+ }
+ late.pid = mv_state[3];
+ late.tgid = pid;
+ late.kill = 1;
+ late.go = 0;
+ late.delay = len * i / 12;
+ if (pthread_create(&t, NULL, late_killer, NULL)) {
+ reap(pid);
+ return KSFT_FAIL;
+ }
+ sleep_ns(NSEC_PER_MSEC);
+ late.go = 1;
+ err = enter(cg_p, pid);
+ join(t);
+ if (err) {
+ ksft_print_msg("%s: the move refused: %s\n", name, strerror(-err));
+ ret = KSFT_FAIL;
+ }
+ reap(pid);
+ sleep_ns(3 * PERIOD);
+ if (ret == KSFT_PASS &&
+ (admits_in(cg_p, base_p, "P, after the thread died") ||
+ admits_in(root, base_r, "the root domain the process left"))) {
+ ksft_print_msg("%s: try %d, the death %llu us into a move of %llu us\n",
+ name, i, (unsigned long long)(late.delay / NSEC_PER_USEC),
+ (unsigned long long)(len / NSEC_PER_USEC));
+ ret = KSFT_FAIL;
+ }
+ }
+ return ret;
+}
+
+static int late_sysrq_fd;
+
+/*
+ * Write 'n' to /proc/sysrq-trigger, late.delay ns after late.go. sysrq-n
+ * makes every real-time and deadline task SCHED_OTHER.
+ */
+static void *late_sysrq(void *arg)
+{
+ uint64_t start;
+
+ while (!late.go)
+ ;
+ start = now_ns();
+ while (now_ns() - start < late.delay)
+ ;
+ late.ret = write(late_sysrq_fd, "n", 1) == 1 ? 0 : -errno;
+ late.done = now_ns();
+ return arg;
+}
+
+/*
+ * Make sysrq-n take a sleeping deadline thread of 30% out of the class
+ * while its process moves into P. cpuset_can_attach() reserves room in P
+ * for the thread. sysrq-n does not wait for the move. If it comes after
+ * the reservation, the thread moves no bandwidth, and cpuset_attach()
+ * must give the room back. Once the process is gone, the free bandwidth of P and of
+ * the root domain the process left must be what it was.
+ *
+ * sysrq-n changes the whole system. So the case runs last of the cases
+ * with P, and only with DEADLINE_TEST_SYSRQ set.
+ *
+ * Try 12 delays spread over the time a move takes, each with a new
+ * process.
+ */
+static int move_normalized(void)
+{
+ int base_p, base_r, ret = KSFT_PASS, i;
+ const char *name = "move_normalized";
+ uint64_t len = 0;
+
+ if (!getenv("DEADLINE_TEST_SYSRQ")) {
+ ksft_print_msg("sysrq-n renices every real-time task: set DEADLINE_TEST_SYSRQ=1\n");
+ return KSFT_SKIP;
+ }
+ late_sysrq_fd = open("/proc/sysrq-trigger", O_WRONLY);
+ if (late_sysrq_fd < 0) {
+ ksft_print_msg("/proc/sysrq-trigger: %s\n", strerror(errno));
+ return KSFT_SKIP;
+ }
+ base_p = probe(cg_p);
+ base_r = probe(root);
+ for (i = 0; i < 12 && ret == KSFT_PASS; i++) {
+ pthread_t t;
+ pid_t pid;
+ int err;
+
+ mv_state[0] = 0;
+ mv_state[3] = 0;
+ mv_state[4] = 0;
+ pid = pinned_mover(2048, 0);
+ if (pid < 0 || child_up(name, 0)) {
+ reap(pid);
+ close(late_sysrq_fd);
+ return mv_state[0] == -2 ? KSFT_SKIP : KSFT_FAIL;
+ }
+ if (!i) {
+ /* Time one move, to spread the delays over it. */
+ uint64_t start = now_ns();
+
+ if (enter(cg_p, pid) || enter(root, pid)) {
+ ksft_print_msg("%s: a move refused\n", name);
+ reap(pid);
+ close(late_sysrq_fd);
+ return KSFT_FAIL;
+ }
+ len = (now_ns() - start) / 2;
+ }
+ late.go = 0;
+ late.delay = len * i / 12;
+ if (pthread_create(&t, NULL, late_sysrq, NULL)) {
+ reap(pid);
+ close(late_sysrq_fd);
+ return KSFT_FAIL;
+ }
+ sleep_ns(NSEC_PER_MSEC);
+ late.go = 1;
+ err = enter(cg_p, pid);
+ join(t);
+ if (err) {
+ ksft_print_msg("%s: the move refused: %s\n", name, strerror(-err));
+ ret = KSFT_FAIL;
+ } else if (late.ret) {
+ ksft_print_msg("%s: sysrq-n: %s\n", name, strerror(-late.ret));
+ ret = KSFT_SKIP;
+ }
+ reap(pid);
+ sleep_ns(3 * PERIOD);
+ if (ret == KSFT_PASS &&
+ (admits_in(cg_p, base_p, "P, after sysrq-n") ||
+ admits_in(root, base_r, "the root domain the process left"))) {
+ ksft_print_msg("%s: try %d, sysrq-n %llu us into a move of %llu us\n",
+ name, i, (unsigned long long)(late.delay / NSEC_PER_USEC),
+ (unsigned long long)(len / NSEC_PER_USEC));
+ ret = KSFT_FAIL;
+ }
+ }
+ close(late_sysrq_fd);
+ return ret;
+}
+
+static int hold(int n);
+static void release_held(void);
+
+/*
+ * Wake a sleeping deadline thread of 30% up while its process moves from
+ * P to the root cgroup.
+ *
+ * The process is pinned_mover(): 2048 sleeping threads, then the deadline
+ * thread. That thread first ran on the second CPU of the root cgroup,
+ * went to sleep there, and was moved into P asleep. So P counts it, and
+ * it is still on that CPU. A thread of the test, late_killer(), wakes it
+ * up with a signal.
+ *
+ * cpuset_can_attach() reserves room for the thread on the CPU that the
+ * move back records: the CPU the thread is on. If the thread wakes up
+ * after that, it runs on the CPU of P. cpuset_attach() must still move
+ * the bandwidth to the CPU recorded. Working the CPU out again would
+ * give the first CPU of the root cgroup, where nothing is reserved.
+ *
+ * Both CPUs are in the same root domain. So that root domain must count
+ * the thread once the move is done, and P must no longer count it.
+ *
+ * With @full, hold() first fills that root domain with deadline threads
+ * of the test, until 40% is free. The thread of 30% then fits once, and
+ * not twice.
+ *
+ * Try 12 delays spread over the time a move takes, each with a new
+ * process.
+ */
+static int woken(int full, const char *name)
+{
+ int base_p, base_r, free_r, ret = KSFT_PASS, i;
+ uint64_t len = 0;
+
+ base_p = probe(cg_p);
+ base_r = probe(root);
+ if (base_r < 5) {
+ ksft_print_msg("%s: no room for the thread\n", name);
+ return KSFT_SKIP;
+ }
+ free_r = base_r;
+ if (full) {
+ if (hold(base_r - 4)) {
+ release_held();
+ return KSFT_FAIL;
+ }
+ free_r = 4;
+ }
+ for (i = 0; i < 12 && ret == KSFT_PASS; i++) {
+ pid_t pid, tid;
+ pthread_t t;
+ int err;
+
+ mv_state[0] = 0;
+ mv_state[3] = 0;
+ mv_state[4] = 0;
+ /* Not the first CPU: the move falls back to that one. */
+ sleeper_cpu = online[1];
+ pid = pinned_mover(2048, 0);
+ sleeper_cpu = -1;
+ if (pid < 0 || child_up(name, 0)) {
+ reap(pid);
+ release_held();
+ return mv_state[0] == -2 ? KSFT_SKIP : KSFT_FAIL;
+ }
+ tid = mv_state[3];
+ sleep_ns(PERIOD);
+ if (!i) {
+ /*
+ * Time the move into P, to spread the delays over
+ * a move.
+ */
+ uint64_t start = now_ns();
+
+ err = enter(cg_p, pid);
+ if (!err) {
+ len = now_ns() - start;
+ err = enter(root, pid);
+ }
+ if (err) {
+ ksft_print_msg("%s: a move refused\n", name);
+ reap(pid);
+ release_held();
+ return KSFT_FAIL;
+ }
+ }
+ if (enter(cg_p, pid)) {
+ ksft_print_msg("%s: the process cannot go into P\n", name);
+ reap(pid);
+ release_held();
+ return KSFT_FAIL;
+ }
+ if (task_cpu_of(tid) != online[1]) {
+ ksft_print_msg("%s: the thread is on CPU %d, not on CPU %d\n", name,
+ task_cpu_of(tid), online[1]);
+ reap(pid);
+ release_held();
+ return KSFT_SKIP;
+ }
+ late.pid = tid;
+ late.tgid = pid;
+ late.kill = 0;
+ late.go = 0;
+ late.delay = len * i / 12;
+ if (pthread_create(&t, NULL, late_killer, NULL)) {
+ reap(pid);
+ release_held();
+ return KSFT_FAIL;
+ }
+ sleep_ns(NSEC_PER_MSEC);
+ late.go = 1;
+ err = enter(root, pid);
+ join(t);
+ if (err) {
+ ksft_print_msg("%s: the move refused: %s\n", name, strerror(-err));
+ ret = KSFT_FAIL;
+ }
+ sleep_ns(2 * PERIOD);
+ if (ret == KSFT_PASS &&
+ (admits_in(cg_p, base_p, "P, which the thread left") ||
+ admits_in(root, free_r - 3, "the root domain of the thread"))) {
+ ksft_print_msg("%s: try %d, the wakeup %llu us into a move of %llu us\n",
+ name, i, (unsigned long long)(late.delay / NSEC_PER_USEC),
+ (unsigned long long)(len / NSEC_PER_USEC));
+ ret = KSFT_FAIL;
+ }
+ reap(pid);
+ sleep_ns(3 * PERIOD);
+ }
+ release_held();
+ sleep_ns(3 * PERIOD);
+ if (ret == KSFT_PASS &&
+ (admits_in(cg_p, base_p, "P, at the end") ||
+ admits_in(root, base_r, "the root domain of the rest, at the end")))
+ ret = KSFT_FAIL;
+ return ret;
+}
+
+static int move_woken(void)
+{
+ return woken(0, "move_woken");
+}
+
+static int move_woken_full(void)
+{
+ return woken(1, "move_woken_full");
+}
+
/*
* Have the cpu controller refuse a move after cpuset_can_attach()
* reserved room. cpuset_cancel_attach() must then give the room back.
@@ -2959,6 +4027,78 @@ static int move_cancelled(void)
return ret;
}
+/* Return whether every task of @cg is frozen. */
+static int cg_frozen(const char *cg)
+{
+ char path[PATH_MAX + 32], buf[256];
+
+ snprintf(path, sizeof(path), "%s/cgroup.events", cg);
+ return !read_file(path, buf, sizeof(buf)) && strstr(buf, "frozen 1");
+}
+
+/*
+ * Freeze P with a running deadline task of 30% in it. Move the frozen
+ * task to the root cgroup, switch it to SCHED_OTHER, thaw P and kill the
+ * task. Do it three times. The free bandwidth of P and of the root
+ * domain of the other CPUs must then be what it was.
+ *
+ * A frozen task sleeps in its signal handling. The root cgroup is not
+ * frozen, so the move thaws the task. It is woken on P's CPU, before
+ * cpuset_attach() gives it the CPUs of the root cgroup.
+ */
+static int frozen_moved_switched(void)
+{
+ const char *what = "a deadline task to freeze";
+ int base_p, base_r, ret = KSFT_PASS, i;
+
+ base_p = probe(cg_p);
+ base_r = probe(root);
+ for (i = 0; i < 3 && ret == KSFT_PASS; i++) {
+ pid_t pid;
+
+ mv_state[0] = 0;
+ pid = fork_child(what);
+ if (!pid) {
+ if (enter(cg_p, getpid()) ||
+ set_deadline(0, 30 * NSEC_PER_MSEC, PERIOD)) {
+ mv_state[0] = -1;
+ _exit(1);
+ }
+ mv_state[0] = 1;
+ for (;;)
+ spin_ns(NSEC_PER_MSEC);
+ }
+ if (pid < 0 || child_up(what, 0)) {
+ reap(pid);
+ return KSFT_FAIL;
+ }
+ if (write_file(cg_p, "cgroup.freeze", "1")) {
+ ksft_print_msg("P cannot be frozen\n");
+ reap(pid);
+ return KSFT_SKIP;
+ }
+ if (wait_for(2000, cg_frozen(cg_p))) {
+ ksft_print_msg("P does not freeze\n");
+ ret = KSFT_FAIL;
+ } else if (enter(root, pid)) {
+ ksft_print_msg("the frozen task cannot leave P\n");
+ ret = KSFT_FAIL;
+ } else if (set_other(pid)) {
+ ksft_print_msg("the task cannot leave SCHED_DEADLINE\n");
+ ret = KSFT_FAIL;
+ }
+ if (write_file(cg_p, "cgroup.freeze", "0"))
+ ksft_print_msg("P cannot be thawed\n");
+ reap(pid);
+ }
+ sleep_ns(3 * PERIOD);
+ if (ret == KSFT_PASS &&
+ (admits_in(cg_p, base_p, "P, after the frozen tasks") ||
+ admits_in(root, base_r, "the root domain of the rest")))
+ ret = KSFT_FAIL;
+ return ret;
+}
+
/* Deaths, in P */
enum die_mode { DIE_RUNNING, DIE_SWITCHED_OUT, DIE_SHRUNK };
@@ -3558,6 +4698,11 @@ static int yield_switched_out(void)
return ret;
}
+static int yield_moved(void)
+{
+ return moves(MV_YIELDED, "yield_moved");
+}
+
/* Kill a deadline task while sched_yield() throttles it. */
static int yield_killed(void)
{
@@ -3833,6 +4978,109 @@ static int reclaim_switched(void)
return ret;
}
+/*
+ * Fork a reclaiming deadline task of 10%. It spins, and publishes in
+ * mv_state[1] the CPU time it has used, in milliseconds. It idles while
+ * mv_state[2] is set, and the moves happen then. A cpuset move waits for
+ * an RCU grace period. Where RCU runs in kthreads (PREEMPT_RT), a task
+ * that reclaims its whole CPU keeps that grace period from ending. Return
+ * the pid, or -1.
+ */
+static pid_t reclaim_mover(void)
+{
+ pid_t pid = fork_child("a reclaiming task");
+
+ if (!pid) {
+ if (set_deadline_flags(0, 10 * NSEC_PER_MSEC, PERIOD,
+ SCHED_FLAG_RECLAIM)) {
+ mv_state[0] = -1;
+ _exit(1);
+ }
+ mv_state[0] = 1;
+ for (;;) {
+ if (mv_state[2]) {
+ sleep_ns(10 * NSEC_PER_MSEC);
+ continue;
+ }
+ spin_ns(NSEC_PER_MSEC);
+ mv_state[1] = (int)(thread_time_ns() / NSEC_PER_MSEC);
+ }
+ }
+ return pid;
+}
+
+/* Idle the reclaiming task, then move it to @cg. Return 0, or -errno. */
+static int move_idle(const char *cg, pid_t pid)
+{
+ int err;
+
+ mv_state[2] = 1;
+ sleep_ns(20 * NSEC_PER_MSEC); /* it stops spinning */
+ err = enter(cg, pid);
+ mv_state[2] = 0;
+ return err;
+}
+
+/* Return the CPU share, in percent, of the reclaiming task over @ns. */
+static int mover_share(uint64_t ns)
+{
+ uint64_t wall = now_ns();
+ int cpu = mv_state[1];
+
+ sleep_ns(ns);
+ cpu = mv_state[1] - cpu;
+ wall = now_ns() - wall;
+ return (int)(100 * (uint64_t)cpu * NSEC_PER_MSEC / wall);
+}
+
+/*
+ * Move a reclaiming task into P, back out, and into P again. It must
+ * reclaim as much of P's CPU as it did the first time. Once it is gone,
+ * the free bandwidth of both root domains must be what it was.
+ */
+static int reclaim_moved(void)
+{
+ int base_p, base_r, in_p, again = 0, ret = KSFT_PASS;
+ pid_t pid;
+
+ base_p = probe(cg_p);
+ base_r = probe(root);
+ mv_state[0] = 0;
+ mv_state[1] = 0;
+ mv_state[2] = 1; /* idle until it is in P */
+ pid = reclaim_mover();
+ if (pid < 0)
+ return KSFT_FAIL;
+ if (child_up("a reclaiming task", 0) || move_idle(cg_p, pid)) {
+ mv_state[2] = 1;
+ reap(pid);
+ return KSFT_FAIL;
+ }
+ sleep_ns(2 * PERIOD);
+ in_p = mover_share(SHARE_NS);
+ if (move_idle(root, pid) || move_idle(cg_p, pid)) {
+ ksft_print_msg("a move of a reclaiming task refused\n");
+ ret = KSFT_FAIL;
+ } else {
+ sleep_ns(2 * PERIOD);
+ again = mover_share(SHARE_NS);
+ if (in_p < RECLAIM_MIN || again < in_p - SHARE_SLACK ||
+ again > in_p + SHARE_SLACK) {
+ ksft_print_msg("reclaiming: %d%% of P's CPU, %d%% after a move\n",
+ in_p, again);
+ ret = KSFT_FAIL;
+ }
+ }
+ mv_state[2] = 1;
+ reap(pid);
+ sleep_ns(3 * PERIOD);
+ if (ret == KSFT_PASS && (probe(cg_p) != base_p || probe(root) != base_r)) {
+ ksft_print_msg("a reclaiming task moved: the free bandwidth changed\n");
+ ret = KSFT_FAIL;
+ }
+ return ret;
+}
+
/* Deadline servers */
#define SERVER_RUNTIME (100 * NSEC_PER_MSEC) /* the cases start at 10% */
@@ -4557,6 +5805,281 @@ static int params_waking(void)
return ret;
}
+/* A random mix of the transitions above */
+
+#define MIX_WORKERS 6
+#define MIX_NS (3 * NSEC_PER_SEC)
+#define MIX_SEED 20261103 /* fixed: a rerun starts with the same draws */
+
+/*
+ * random_mix: six workers change their own state at random for 3s, and
+ * the case changes them from outside too. Q is a partition root of two
+ * CPUs, next to P. The case needs 4 housekeeping CPUs.
+ *
+ * Each worker does one of these, over and over:
+ * - become a deadline task, with random parameters and flags
+ * - switch to SCHED_OTHER
+ * - sleep
+ * - spin, at times past its runtime
+ * - call sched_yield()
+ * - wait up to 5ms for shm_lock and hold it 1ms. It is the mutex of the
+ * priority inheritance cases. A deadline waiter boosts its owner.
+ * - exit
+ *
+ * The case does one of these, over and over:
+ * - set the deadline parameters of a random worker
+ * - switch it to SCHED_OTHER
+ * - change its affinity
+ * - move it into P, into Q or into the root cgroup
+ * - kill it
+ * - change the fair server of P's CPU or of one CPU of Q
+ * It starts anew every worker it finds dead.
+ *
+ * The case does not check what a single transition does: the other cases
+ * do that. It checks the accounting afterwards. Once every worker is gone
+ * and its 0-lag time is past:
+ * - P, Q and the housekeeping root domain must each admit what they
+ * admitted before;
+ * - this_bw of P's CPU and of the two CPUs of Q must be what it was.
+ * An over- or underflow of running_bw or this_bw also shows as a warning
+ * in the kernel log, which the test does not read.
+ */
+static atomic_int *mix_state; /* 0: stop, 1 + i: worker i's rounds */
+
+static pid_t mix_worker(int idx, unsigned int seed)
+{
+ static const uint64_t runtime[] = { 5, 10, 20, 30 };
+ static const uint64_t period[] = { 50, 100 };
+ static const uint64_t flag[] = { 0, SCHED_FLAG_RECLAIM, SCHED_FLAG_DL_OVERRUN };
+ pid_t pid = fork_child("a worker of the mix");
+
+ if (!pid) {
+ struct sigaction sa = { .sa_handler = catch_xcpu };
+ struct timespec ts;
+ int err;
+
+ if (sigaction(SIGXCPU, &sa, NULL))
+ _exit(1);
+ seed += idx;
+ /*
+ * Ignore the errors. Each call below is made in whatever
+ * state the worker is in, and the kernel may refuse it.
+ */
+ while (!mix_state[0]) {
+ unsigned int r = rand_r(&seed), v = r / 8;
+
+ switch (r % 8) {
+ case 0:
+ set_deadline_flags(0, runtime[v % 4] * NSEC_PER_MSEC,
+ period[(v / 4) % 2] * NSEC_PER_MSEC,
+ flag[(v / 8) % 3]);
+ break;
+ case 1:
+ set_other(0);
+ break;
+ case 2:
+ sleep_ns((v % 20 + 1) * NSEC_PER_MSEC);
+ break;
+ case 3:
+ spin_ns((v % 15 + 1) * NSEC_PER_MSEC);
+ break;
+ case 4:
+ sched_yield();
+ break;
+ case 5:
+ clock_gettime(CLOCK_REALTIME, &ts);
+ ts.tv_nsec += 5 * NSEC_PER_MSEC;
+ if (ts.tv_nsec >= NSEC_PER_SEC) {
+ ts.tv_nsec -= NSEC_PER_SEC;
+ ts.tv_sec++;
+ }
+ err = pthread_mutex_timedlock(shm_lock, &ts);
+ if (err == EOWNERDEAD) /* its owner died */
+ err = pthread_mutex_consistent(shm_lock);
+ if (!err) {
+ spin_ns(NSEC_PER_MSEC);
+ if (pthread_mutex_unlock(shm_lock))
+ _exit(1);
+ }
+ break;
+ case 6:
+ _exit(0); /* the case starts another */
+ default:
+ sleep_ns(NSEC_PER_MSEC / 2);
+ break;
+ }
+ mix_state[1 + idx]++;
+ }
+ _exit(0);
+ }
+ return pid;
+}
+
+static cpu_set_t mix_mask[4];
+
+static int random_mix(void)
+{
+ static const uint64_t runtime[] = { 5, 10, 20, 30 };
+ int cpu_a = top_cpu(3), cpu_b = top_cpu(2), cpu_p = top_cpu(1);
+ int base_p, base_q, base_r, i, ret = KSFT_PASS, rounds = 0, sums;
+ long ta = 0, ra, tb = 0, rb, tp = 0, rp, ta2, ra2, tb2, rb2, tp2, rp2;
+ pid_t pid[MIX_WORKERS] = { -1, -1, -1, -1, -1, -1 };
+ unsigned int seed = MIX_SEED;
+ uint64_t end;
+ char cpus[32];
+
+ if (ncpus < 4) {
+ ksft_print_msg("needs 4 non-isolated CPUs\n");
+ return KSFT_SKIP;
+ }
+ snprintf(cpus, sizeof(cpus), "%d,%d", cpu_a, cpu_b);
+ if (make_partition_cpus(cg_q, cpus)) {
+ remove_partition(cg_q);
+ ksft_print_msg("no partition root of two CPUs\n");
+ return KSFT_SKIP;
+ }
+ /*
+ * The affinities of the mix: every CPU of the test, those of Q and
+ * P, those of P and the rest, and all but the first.
+ */
+ for (i = 0; i < 4; i++)
+ CPU_ZERO(&mix_mask[i]);
+ for (i = 0; i < ncpus; i++) {
+ int c = online[i];
+
+ CPU_SET(c, &mix_mask[0]);
+ if (c == cpu_a || c == cpu_b || c == cpu_p)
+ CPU_SET(c, &mix_mask[1]);
+ if (c != cpu_a && c != cpu_b)
+ CPU_SET(c, &mix_mask[2]);
+ if (i)
+ CPU_SET(c, &mix_mask[3]);
+ }
+ if (shm_lock_init()) {
+ remove_partition(cg_q);
+ return KSFT_SKIP; /* no shared mutex for the workers */
+ }
+ /* Without this_bw in sched_debug, check the admissions only. */
+ sums = !rq_bw_of(cpu_a, &ta, &ra) && !rq_bw_of(cpu_b, &tb, &rb) &&
+ !rq_bw_of(cpu_p, &tp, &rp);
+ base_p = probe(cg_p);
+ base_q = probe(cg_q);
+ base_r = probe(root);
+ if (base_p <= 0 || base_q <= 0 || base_r <= 0) {
+ remove_partition(cg_q);
+ ksft_print_msg("no room to probe P, Q or the rest\n");
+ return KSFT_SKIP;
+ }
+ mix_state = mv_state + 16;
+ for (i = 0; i <= MIX_WORKERS; i++)
+ mix_state[i] = 0;
+ for (i = 0; i < MIX_WORKERS; i++) {
+ pid[i] = mix_worker(i, seed);
+ if (pid[i] < 0) {
+ ret = KSFT_FAIL;
+ goto out;
+ }
+ }
+
+ end = now_ns() + MIX_NS;
+ while (now_ns() < end) {
+ unsigned int r = rand_r(&seed);
+ int w = r % MIX_WORKERS;
+
+ switch ((r / MIX_WORKERS) % 8) {
+ case 0:
+ set_deadline(pid[w], runtime[r % 4] * NSEC_PER_MSEC, PERIOD);
+ break;
+ case 7:
+ /*
+ * Refused with EBUSY if the mask leaves out a CPU
+ * of the root domain that counts a deadline task.
+ */
+ sched_setaffinity(pid[w], sizeof(mix_mask[0]), &mix_mask[r % 4]);
+ break;
+ case 1:
+ set_other(pid[w]);
+ break;
+ case 2:
+ enter(cg_p, pid[w]);
+ break;
+ case 3:
+ enter(cg_q, pid[w]);
+ break;
+ case 4:
+ enter(root, pid[w]);
+ break;
+ case 5:
+ /* An exited worker is a zombie: the kill succeeds. */
+ if (kill(pid[w], SIGKILL) && errno != ESRCH)
+ ksft_print_msg("kill %d: %s\n", pid[w], strerror(errno));
+ break;
+ default:
+ server_set(r % 2 ? cpu_p : cpu_a,
+ (r % 3 + 1) * SERVER_RUNTIME, SERVER_PERIOD);
+ break;
+ }
+ sleep_ns((r % 2000 + 100) * NSEC_PER_USEC);
+ /* Start anew each worker that exited or was killed. */
+ for (i = 0; i < MIX_WORKERS; i++) {
+ pid_t done = waitpid(pid[i], NULL, WNOHANG);
+
+ if (done < 0 && errno != EINTR) {
+ ksft_print_msg("waitpid %d: %s\n", pid[i], strerror(errno));
+ ret = KSFT_FAIL;
+ goto out;
+ }
+ if (done == pid[i]) {
+ pid[i] = mix_worker(i, seed + rounds);
+ if (pid[i] < 0) {
+ ret = KSFT_FAIL;
+ goto out;
+ }
+ rounds++;
+ }
+ }
+ }
+out:
+ mix_state[0] = 1;
+ for (i = 0; i < MIX_WORKERS; i++)
+ reap(pid[i]);
+ /*
+ * Wait for the 0-lag times of the workers. Putting the fair servers
+ * back needs the room they held.
+ */
+ sleep_ns(5 * PERIOD);
+ server_restore();
+ if (ret != KSFT_PASS) {
+ remove_partition(cg_q);
+ return ret;
+ }
+ for (i = 0, rounds = 0; i < MIX_WORKERS; i++)
+ rounds += mix_state[1 + i];
+ if (!rounds) {
+ ksft_print_msg("the workers of the mix took no transition\n");
+ ret = KSFT_FAIL;
+ }
+ if (ret == KSFT_PASS && sums) {
+ if (rq_bw_of(cpu_a, &ta2, &ra2) || rq_bw_of(cpu_b, &tb2, &rb2) ||
+ rq_bw_of(cpu_p, &tp2, &rp2)) {
+ ret = KSFT_FAIL;
+ } else if (ta2 != ta || tb2 != tb || tp2 != tp) {
+ ksft_print_msg("this_bw %ld %ld %ld, was %ld %ld %ld\n",
+ ta2, tb2, tp2, ta, tb, tp);
+ ret = KSFT_FAIL;
+ }
+ }
+ if (ret == KSFT_PASS &&
+ (admits_in(cg_p, base_p, "P after the mix") ||
+ admits_in(cg_q, base_q, "Q after the mix") ||
+ admits_in(root, base_r, "the root domain of the rest after the mix")))
+ ret = KSFT_FAIL;
+ if (ret != KSFT_PASS)
+ ksft_print_msg("seed %u, %d transitions\n", MIX_SEED, rounds);
+ remove_partition(cg_q);
+ return ret;
+}
+
/*
* CPU hotplug, from the main process, with no P. Two cases below run with
* P instead: death_zombie_hotplug and pi_departed_moved.
@@ -5223,6 +6746,834 @@ static int in_cpu_list(const char *list, int cpu)
return 0;
}
+/*
+ * Wake @pid, a task that sleeps in read() on a pipe: write a byte to @fd.
+ * The task counts its wakeups in mv_state[1]. Wait until it has run and
+ * sleeps again. Return 0, or -1.
+ */
+static int fallback_wake(int fd, pid_t pid)
+{
+ int n = mv_state[1];
+
+ if (write(fd, "w", 1) != 1) {
+ ksft_print_msg("write to the deadline task: %s\n", strerror(errno));
+ return -1;
+ }
+ if (wait_for(1000, mv_state[1] != n) ||
+ wait_for(1000, proc_state_of(pid) == 'S')) {
+ ksft_print_msg("the deadline task did not wake up and sleep again\n");
+ return -1;
+ }
+ return 0;
+}
+
+/*
+ * The cpusets of the cases below: B and Q in the root cgroup, C in B. A
+ * case makes those it needs. cs_setup() names them and enables the cpuset
+ * controller of the root cgroup if it is off. cs_cleanup() removes them,
+ * and turns the controller off again if cs_setup() turned it on.
+ */
+static char cs_b[PATH_MAX], cs_c[PATH_MAX], cs_q[PATH_MAX];
+static int cs_added;
+
+static int cs_setup(void)
+{
+ char path[PATH_MAX + 32], buf[256];
+
+ cs_added = 0;
+ if (!root[0] ||
+ snprintf(cs_b, sizeof(cs_b), "%s/deadline_bw_b", root) >= (int)sizeof(cs_b) ||
+ snprintf(cs_c, sizeof(cs_c), "%s/c", cs_b) >= (int)sizeof(cs_c) ||
+ snprintf(cs_q, sizeof(cs_q), "%s/deadline_bw_q", root) >= (int)sizeof(cs_q))
+ return -1;
+ snprintf(path, sizeof(path), "%s/cgroup.subtree_control", root);
+ if (read_file(path, buf, sizeof(buf)))
+ return -1;
+ if (!strstr(buf, "cpuset")) {
+ if (write_file(root, "cgroup.subtree_control", "+cpuset"))
+ return -1;
+ cs_added = 1;
+ }
+ return 0;
+}
+
+/* Make the cpuset @cg, and give it the CPUs of @cpus. Return 0, or -1. */
+static int cs_make(const char *cg, const char *cpus)
+{
+ int err = 0;
+
+ if (mkdir(cg, 0755) && errno != EEXIST)
+ err = -errno;
+ if (!err)
+ err = write_file(cg, "cpuset.cpus", cpus);
+ if (err)
+ ksft_print_msg("cpuset %s of CPUs %s: %s\n", cg, cpus, strerror(-err));
+ return err ? -1 : 0;
+}
+
+static void cs_cleanup(void)
+{
+ remove_partition(cs_c);
+ remove_partition(cs_b);
+ remove_partition(cs_q);
+ if (cs_added && write_file(root, "cgroup.subtree_control", "-cpuset"))
+ ksft_print_msg("the cpuset controller stays enabled\n");
+}
+
+/*
+ * Return whether the partition file of @cg holds @what. If not, and
+ * @tell, print what it holds.
+ */
+static int partition_has(const char *cg, const char *what, int tell)
+{
+ char path[PATH_MAX + 32], buf[256];
+
+ snprintf(path, sizeof(path), "%s/cpuset.cpus.partition", cg);
+ if (read_file(path, buf, sizeof(buf)))
+ return 0;
+ if (strstr(buf, what))
+ return 1;
+ if (tell)
+ ksft_print_msg("%s/cpuset.cpus.partition: %s", cg, buf);
+ return 0;
+}
+
+/*
+ * Read total_bw of the root domain of @d into @def, and of the root
+ * domain of @h into @hk. Return 0, or -1.
+ */
+static int totals_of(int d, int h, long long *def, long long *hk)
+{
+ *def = total_bw_of(d);
+ *hk = total_bw_of(h);
+ if (*def == -1 || *hk == -1) {
+ ksft_print_msg("no total_bw in /sys/kernel/debug/sched/debug\n");
+ return -1;
+ }
+ return 0;
+}
+
+/*
+ * Return KSFT_PASS if total_bw of the root domains of @d and @h is
+ * @def and @hk, else KSFT_FAIL. @what names the step in the message.
+ */
+static int totals_check(const char *what, int d, int h, long long def, long long hk)
+{
+ long long now_def, now_hk;
+
+ if (totals_of(d, h, &now_def, &now_hk))
+ return KSFT_FAIL;
+ if (now_def == def && now_hk == hk)
+ return KSFT_PASS;
+ ksft_print_msg("%s: total_bw %lld, %lld, not %lld, %lld\n",
+ what, now_def, now_hk, def, hk);
+ return KSFT_FAIL;
+}
+
+/*
+ * Fork a task that runs on @cpu. Then it allows every CPU if @all. If
+ * not, it allows the housekeeping CPUs that were online at the start,
+ * even those offline now. It becomes a deadline task of @runtime every
+ * @period. If @fd is -1, it spins and counts in mv_state[1]. Else it
+ * sleeps in read() on @fd and counts in mv_state[1] each byte that wakes
+ * it. On the byte 'x', it spins 90ms and exits. It sets mv_state[0] as
+ * mover() does. Return its pid, or -1.
+ */
+static pid_t task_from(int cpu, int all, uint64_t runtime, uint64_t period, int fd)
+{
+ pid_t pid = fork_child("a deadline task");
+
+ if (!pid) {
+ cpu_set_t set, one;
+ int i;
+ char c;
+
+ CPU_ZERO(&one);
+ CPU_SET(cpu, &one);
+ if (sched_setaffinity(0, sizeof(one), &one))
+ _exit(1);
+ spin_ns(NSEC_PER_MSEC);
+ CPU_ZERO(&set);
+ for (i = 0; i < (all ? CPU_SETSIZE : ncpus); i++)
+ CPU_SET(all ? i : online[i], &set);
+ if (sched_setaffinity(0, sizeof(set), &set) ||
+ set_deadline(0, runtime, period)) {
+ mv_state[0] = -1;
+ _exit(1);
+ }
+ mv_state[0] = 1;
+ while (fd < 0)
+ mv_state[1]++;
+ while (read(fd, &c, 1) == 1) {
+ if (c == 'x') {
+ spin_ns(90 * NSEC_PER_MSEC);
+ _exit(0);
+ }
+ mv_state[1]++;
+ }
+ _exit(0);
+ }
+ return pid;
+}
+
+/*
+ * The default root domain spans every CPU that is offline. It also spans
+ * the CPU of an isolated partition. The steps:
+ * 1. Take hp_cpu offline. Make Q an isolated partition of Y, another
+ * housekeeping CPU.
+ * 2. A task in the root cgroup becomes a deadline task of 10% and sleeps.
+ * The housekeeping root domain counts it. Its affinity holds the
+ * housekeeping CPUs, hp_cpu too.
+ * 3. Move it asleep into Q. Wake it: it runs on Y. The default root
+ * domain counts it.
+ * 4. Move it asleep back to the root cgroup. Its mask holds hp_cpu, but
+ * no active CPU of the default root domain.
+ * 5. Wake it. It runs on a housekeeping CPU.
+ * Check after step 4 and again after step 5. The housekeeping root
+ * domain must count the task. The default root domain must not. An
+ * offline CPU of the default root domain is no CPU where the task can
+ * stay.
+ *
+ * The references of total_bw are taken after step 1. Once the task is
+ * gone, total_bw must be back to them.
+ */
+static int move_back_cpu_offline(void)
+{
+ int y = hp_cpu == top_cpu(1) ? top_cpu(2) : top_cpu(1), h = online[0];
+ int fd[2] = { -1, -1 }, cpu, err, ret = KSFT_SKIP;
+ long long def0, hk0, def, hk, bw;
+ char buf[16];
+ pid_t pid = -1;
+
+ if (cs_setup())
+ return KSFT_SKIP;
+ if (cpu_online(hp_cpu, 0)) {
+ ksft_print_msg("CPU %d cannot go offline\n", hp_cpu);
+ goto out;
+ }
+ snprintf(buf, sizeof(buf), "%d", y);
+ if (cs_make(cs_q, buf) || write_file(cs_q, "cpuset.cpus.partition", "isolated") ||
+ !partition_has(cs_q, "isolated", 1) || partition_has(cs_q, "invalid", 0)) {
+ ksft_print_msg("no isolated partition Q of CPU %d\n", y);
+ goto out;
+ }
+ if (totals_of(y, h, &def0, &hk0) || pipe(fd))
+ goto out;
+ ret = KSFT_FAIL;
+ mv_state[0] = 0;
+ mv_state[1] = 0;
+ pid = task_from(h, 0, 10 * NSEC_PER_MSEC, PERIOD, fd[0]);
+ if (pid < 0 || child_up("move_back_cpu_offline", 0) ||
+ wait_for(1000, proc_state_of(pid) == 'S') || totals_of(y, h, &def, &hk))
+ goto out;
+ bw = hk - hk0;
+ if (bw <= 0 || def != def0) {
+ ksft_print_msg("the deadline task adds %lld, %lld to total_bw\n",
+ def - def0, bw);
+ goto out;
+ }
+ err = enter(cs_q, pid);
+ if (err) {
+ ksft_print_msg("a move into Q refused: %s\n", strerror(-err));
+ goto out;
+ }
+ if (fallback_wake(fd[1], pid) ||
+ totals_check("in Q", y, h, def0 + bw, hk0))
+ goto out;
+ cpu = task_cpu_of(pid);
+ if (cpu != y) {
+ ksft_print_msg("woken in Q on CPU %d, not %d\n", cpu, y);
+ goto out;
+ }
+ err = enter(root, pid);
+ if (err) {
+ ksft_print_msg("the move back refused: %s\n", strerror(-err));
+ goto out;
+ }
+ if (totals_check("moved back asleep", y, h, def0, hk0 + bw) ||
+ fallback_wake(fd[1], pid) ||
+ totals_check("woken", y, h, def0, hk0 + bw))
+ goto out;
+ cpu = task_cpu_of(pid);
+ if (cpu == y || in_cpu_list(isolated, cpu)) {
+ ksft_print_msg("woken on CPU %d\n", cpu);
+ goto out;
+ }
+ reap(pid);
+ pid = -1;
+ sleep_ns(3 * PERIOD);
+ ret = totals_check("task gone", y, h, def0, hk0);
+out:
+ reap(pid);
+ if (fd[0] >= 0) {
+ close(fd[0]);
+ close(fd[1]);
+ }
+ cs_cleanup();
+ cpu_up(hp_cpu);
+ return ret;
+}
+
+/* Moves between two root domains, from the main process, with P */
+
+/*
+ * Fork a deadline task of 30ms every 100ms, pinned to the @n CPUs of
+ * @cpus. It runs 1ms, then sleeps. If @fd is not -1, each byte read from
+ * @fd wakes it up, and it counts the wakeups in mv_state[1]. It sets
+ * mv_state[0] as mover() does. Return its pid, or -1.
+ */
+static pid_t pinned_sleeper(const int *cpus, int n, int fd)
+{
+ pid_t pid = fork_child("a pinned deadline task");
+
+ if (!pid) {
+ cpu_set_t set;
+ char c;
+ int i;
+
+ CPU_ZERO(&set);
+ for (i = 0; i < n; i++)
+ CPU_SET(cpus[i], &set);
+ if (sched_setaffinity(0, sizeof(set), &set) ||
+ set_deadline(0, 30 * NSEC_PER_MSEC, PERIOD)) {
+ mv_state[0] = -1;
+ _exit(1);
+ }
+ spin_ns(NSEC_PER_MSEC);
+ mv_state[0] = 1;
+ while (fd >= 0 && read(fd, &c, 1) == 1)
+ mv_state[1]++;
+ for (;;)
+ pause();
+ }
+ return pid;
+}
+
+/*
+ * Read total_bw of three root domains into @bw: the root domain of the
+ * isolated CPUs, the housekeeping root domain, and P. Return 0, or -1.
+ */
+static int split_totals(long long bw[3])
+{
+ bw[0] = total_bw_of(isol[0]);
+ bw[1] = total_bw_of(online[0]);
+ bw[2] = total_bw_of(top_cpu(1));
+ if (bw[0] == -1 || bw[1] == -1 || bw[2] == -1) {
+ ksft_print_msg("no total_bw in /sys/kernel/debug/sched/debug\n");
+ return -1;
+ }
+ return 0;
+}
+
+/*
+ * Return KSFT_PASS if total_bw is now @want, else KSFT_FAIL. The message
+ * gives total_bw of the root domain of the isolated CPUs, then of the
+ * housekeeping root domain, then of P.
+ */
+static int split_check(const char *what, const long long want[3])
+{
+ long long bw[3];
+
+ if (split_totals(bw))
+ return KSFT_FAIL;
+ if (bw[0] == want[0] && bw[1] == want[1] && bw[2] == want[2])
+ return KSFT_PASS;
+ ksft_print_msg("%s: total_bw %lld, %lld, %lld, not %lld, %lld, %lld\n",
+ what, bw[0], bw[1], bw[2], want[0], want[1], want[2]);
+ return KSFT_FAIL;
+}
+
+/*
+ * Read total_bw into @base, for the cases below. Return 0, or nonzero if
+ * the case must skip. It needs CPUs isolated by isolcpus="domain".
+ */
+static int split_skip(long long base[3])
+{
+ if (!nisol) {
+ ksft_print_msg("needs CPUs isolated by isolcpus=domain\n");
+ return 1;
+ }
+ return split_totals(base);
+}
+
+/*
+ * The root cpuset holds the CPUs that isolcpus="domain" isolates, so it
+ * spans two root domains. The affinity of a task can keep it in either.
+ * A move into the root cpuset must reserve the bandwidth of the task in
+ * the root domain that the task gets.
+ *
+ * Run twice, with a deadline sleeper of 30% pinned to the isolated CPUs,
+ * then with another one pinned to the housekeeping CPUs except P's. Move
+ * the sleeper into P and back to the root cgroup, then kill it. Once it
+ * is gone, total_bw of the three root domains must be what it was.
+ */
+static int move_isolated(void)
+{
+ long long base[3];
+ int round, err;
+ pid_t pid;
+
+ if (split_skip(base))
+ return KSFT_SKIP;
+ for (round = 0; round < 2; round++) {
+ const char *what = round ? "pinned to the other CPUs" :
+ "pinned to the isolated CPUs";
+
+ mv_state[0] = 0;
+ pid = round ? pinned_sleeper(online, ncpus - 1, -1) :
+ pinned_sleeper(isol, nisol, -1);
+ if (pid < 0 || child_up(what, 0)) {
+ reap(pid);
+ return KSFT_FAIL;
+ }
+ err = enter(cg_p, pid);
+ if (!err)
+ err = enter(root, pid);
+ reap(pid);
+ if (err) {
+ ksft_print_msg("%s: a move refused: %s\n", what, strerror(-err));
+ return KSFT_FAIL;
+ }
+ sleep_ns(3 * PERIOD);
+ if (split_check(what, base))
+ return KSFT_FAIL;
+ }
+ return KSFT_PASS;
+}
+
+/*
+ * Fork a process pinned to the isolated CPUs. Its threads become deadline
+ * tasks of 10ms every 100ms, as in probe(), until one is refused. Then it
+ * sets mv_state[1] to 1, or to -1 on an error. The threads sleep until the
+ * process is killed.
+ */
+static pid_t fill_isolated(void)
+{
+ pid_t pid = fork_child("the tasks that fill the isolated CPUs");
+
+ if (!pid) {
+ int i, max = 16 * nisol + 16;
+ struct probe_slot *s = calloc(max, sizeof(*s));
+ pthread_attr_t a;
+ cpu_set_t set;
+
+ CPU_ZERO(&set);
+ for (i = 0; i < nisol; i++)
+ CPU_SET(isol[i], &set);
+ if (!s || sched_setaffinity(0, sizeof(set), &set) ||
+ sem_init(&probe_done, 0, 0) || pipe(probe_hold) ||
+ pthread_attr_init(&a) || pthread_attr_setstacksize(&a, THREAD_STACK)) {
+ mv_state[1] = -1;
+ _exit(1);
+ }
+ for (i = 0; i < max; i++) {
+ if (pthread_create(&s[i].t, &a, probe_thread, &s[i]) ||
+ sem_wait_for(&probe_done, 10000)) {
+ mv_state[1] = -1;
+ _exit(1);
+ }
+ if (s[i].err)
+ break;
+ }
+ mv_state[1] = i < max && s[i].err == -EBUSY ? 1 : -1;
+ for (;;)
+ pause();
+ }
+ return pid;
+}
+
+/*
+ * Move a deadline sleeper pinned to the isolated CPUs into P. Fill the
+ * isolated CPUs with deadline tasks of 10%, until one is refused. Moving
+ * the sleeper back must then be refused with EBUSY.
+ */
+static int move_isolated_refused(void)
+{
+ pid_t pid, fill = -1;
+ long long base[3];
+ int err, ret = KSFT_FAIL;
+
+ if (split_skip(base))
+ return KSFT_SKIP;
+ mv_state[0] = 0;
+ pid = pinned_sleeper(isol, nisol, -1);
+ if (pid < 0 || child_up("a sleeper pinned to the isolated CPUs", 0))
+ goto out;
+ err = enter(cg_p, pid);
+ if (err) {
+ ksft_print_msg("a move into P refused: %s\n", strerror(-err));
+ goto out;
+ }
+ mv_state[1] = 0;
+ fill = fill_isolated();
+ if (fill < 0 || wait_for(10000, mv_state[1]) || mv_state[1] != 1) {
+ ksft_print_msg("the isolated CPUs cannot be filled\n");
+ goto out;
+ }
+ err = enter(root, pid);
+ if (err == -EBUSY) {
+ ret = KSFT_PASS;
+ } else {
+ ksft_print_msg("a move back to the full isolated CPUs: %s, not EBUSY\n",
+ err ? strerror(-err) : "accepted");
+ }
+out:
+ reap(fill);
+ reap(pid);
+ sleep_ns(3 * PERIOD);
+ if (ret == KSFT_PASS)
+ ret = split_check("the isolated CPUs filled", base);
+ return ret;
+}
+
+/*
+ * A deadline sleeper is pinned to the isolated CPUs. The case allows it
+ * all CPUs, then moves it into P and back, without waking it. So it stays
+ * on an isolated CPU, which its mask holds again. The root domain of the
+ * isolated CPUs must still count its bandwidth. Then wake it: it must run
+ * on an isolated CPU, and that root domain must still count it.
+ */
+static int move_isolated_asleep(void)
+{
+ long long base[3], want[3];
+ int fd[2], err, cpu, i, ret = KSFT_FAIL;
+ cpu_set_t set;
+ pid_t pid;
+
+ if (split_skip(base))
+ return KSFT_SKIP;
+ if (pipe(fd)) {
+ ksft_print_msg("pipe: %s\n", strerror(errno));
+ return KSFT_FAIL;
+ }
+ mv_state[0] = 0;
+ mv_state[1] = 0;
+ pid = pinned_sleeper(isol, nisol, fd[0]);
+ if (pid < 0 || child_up("a sleeper pinned to the isolated CPUs", 0) ||
+ wait_for(1000, proc_state_of(pid) == 'S') || split_totals(want))
+ goto out;
+ CPU_ZERO(&set);
+ for (i = 0; i < CPU_SETSIZE; i++)
+ CPU_SET(i, &set);
+ err = sched_setaffinity(pid, sizeof(set), &set) ? -errno : 0;
+ if (!err)
+ err = enter(cg_p, pid);
+ if (!err)
+ err = enter(root, pid);
+ if (err) {
+ ksft_print_msg("the affinity of all CPUs, or a move: %s\n", strerror(-err));
+ goto out;
+ }
+ for (i = 0; i < 2; i++) {
+ const char *what = i ? "woken" : "moved back asleep";
+
+ cpu = task_cpu_of(pid);
+ if (!in_cpu_list(isolated, cpu)) {
+ ksft_print_msg("%s on CPU %d, not an isolated one\n", what, cpu);
+ goto out;
+ }
+ if (split_check(what, want))
+ goto out;
+ if (!i && fallback_wake(fd[1], pid))
+ goto out;
+ }
+ ret = KSFT_PASS;
+out:
+ reap(pid);
+ close(fd[0]);
+ close(fd[1]);
+ sleep_ns(3 * PERIOD);
+ if (ret == KSFT_PASS)
+ ret = split_check("task gone", base);
+ return ret;
+}
+
+/* The CPUs of move_spanning(): two root domains, an isolated CPU first */
+static cpu_set_t span_set;
+
+/*
+ * Fork a task that pins itself to the isolated CPUs, becomes a deadline
+ * task of 30ms every 100ms, then takes the CPUs of span_set. If @fd is
+ * -1, it spins and counts in mv_state[1]. Else it sleeps, each byte read
+ * from @fd wakes it up, and it counts the wakeups there. It sets
+ * mv_state[0] as mover() does. Return its pid, or -1.
+ */
+static pid_t spanning_task(int fd)
+{
+ pid_t pid = fork_child("a deadline task on two root domains");
+
+ if (!pid) {
+ cpu_set_t set;
+ char c;
+ int i;
+
+ CPU_ZERO(&set);
+ for (i = 0; i < nisol; i++)
+ CPU_SET(isol[i], &set);
+ if (sched_setaffinity(0, sizeof(set), &set) ||
+ set_deadline(0, 30 * NSEC_PER_MSEC, PERIOD) ||
+ sched_setaffinity(0, sizeof(span_set), &span_set)) {
+ mv_state[0] = -1;
+ _exit(1);
+ }
+ mv_state[0] = 1;
+ while (fd < 0)
+ mv_state[1]++;
+ while (read(fd, &c, 1) == 1)
+ mv_state[1]++;
+ for (;;)
+ pause();
+ }
+ return pid;
+}
+
+/*
+ * A task becomes a deadline task on the isolated CPUs, where it is
+ * counted. Then it takes a wider affinity: the isolated CPUs, and the
+ * housekeeping CPUs above the first isolated one except P's. In the root
+ * cpuset, its CPUs span two root domains, and the lowest one is isolated.
+ * The case moves it into P, where it runs, and back to the root cgroup.
+ * The task must then run on a housekeeping CPU, and the housekeeping root
+ * domain must count its bandwidth. The root domain of the isolated CPUs
+ * must count it no more. Run twice: with a running task, then with a
+ * sleeper woken in P and again after the move back.
+ *
+ * The kernel counts a task that has a housekeeping CPU in the
+ * housekeeping root domain. The move reserves there, and a rebuild of the
+ * accounting makes the same choice. The lowest CPU of the task is
+ * isolated, so a move that takes the first CPU of the mask reserves in
+ * the wrong root domain.
+ */
+static int move_spanning(void)
+{
+ int fd[2] = { -1, -1 }, round, cpu, err, i, n = 0, ret = KSFT_PASS;
+ long long base[3], want[3];
+ pid_t pid;
+
+ if (split_skip(base))
+ return KSFT_SKIP;
+ CPU_ZERO(&span_set);
+ for (i = 0; i < nisol; i++)
+ CPU_SET(isol[i], &span_set);
+ for (i = 0; i < ncpus - 1; i++) {
+ if (online[i] > isol[0]) {
+ CPU_SET(online[i], &span_set);
+ n++;
+ }
+ }
+ if (!n) {
+ ksft_print_msg("needs a CPU above the first isolated one, besides P\n");
+ return KSFT_SKIP;
+ }
+ for (round = 0; round < 2 && ret == KSFT_PASS; round++) {
+ const char *what = round ? "a sleeper" : "a running task";
+
+ ret = KSFT_FAIL;
+ if (round && pipe(fd)) {
+ ksft_print_msg("pipe: %s\n", strerror(errno));
+ break;
+ }
+ mv_state[0] = 0;
+ mv_state[1] = 0;
+ pid = spanning_task(fd[0]);
+ if (pid < 0 || child_up(what, 0) || split_totals(want))
+ goto next;
+ /*
+ * Moved back, its bandwidth goes to the housekeeping root
+ * domain, not to the root domain of the isolated CPUs.
+ */
+ want[1] += want[0] - base[0];
+ want[0] = base[0];
+ err = enter(cg_p, pid);
+ if (err) {
+ ksft_print_msg("%s: a move into P refused: %s\n", what, strerror(-err));
+ goto next;
+ }
+ if ((round && fallback_wake(fd[1], pid)) ||
+ wait_for(1000, task_cpu_of(pid) == top_cpu(1))) {
+ ksft_print_msg("%s: not on P's CPU\n", what);
+ goto next;
+ }
+ err = enter(root, pid);
+ if (err) {
+ ksft_print_msg("%s: the move back refused: %s\n", what, strerror(-err));
+ goto next;
+ }
+ if (round ? fallback_wake(fd[1], pid) :
+ wait_for(1000, task_cpu_of(pid) != top_cpu(1)))
+ goto next;
+ sleep_ns(PERIOD);
+ cpu = task_cpu_of(pid);
+ if (in_cpu_list(isolated, cpu)) {
+ ksft_print_msg("%s: moved back to isolated CPU %d\n", what, cpu);
+ goto next;
+ }
+ ret = split_check(what, want);
+next:
+ reap(pid);
+ if (round) {
+ close(fd[0]);
+ close(fd[1]);
+ }
+ sleep_ns(3 * PERIOD);
+ if (ret == KSFT_PASS)
+ ret = split_check("task gone", base);
+ }
+ return ret;
+}
+
+/*
+ * A deadline task moved asleep into another cpuset still sleeps on a CPU
+ * of the root domain it left. The root domain it moved to counts its
+ * bandwidth. sched_setaffinity() must check a new mask against the root
+ * domain that counts the task, not the one it sleeps in.
+ *
+ * H is a cpuset here, not the CPU of fallback_case(). It holds the
+ * housekeeping CPUs except P's. The steps:
+ * 1. Pin a fair task to the isolated CPUs.
+ * 2. Move it into H. Its affinity has no CPU of H, so it gets all the
+ * CPUs of H.
+ * 3. Make it a deadline task there, and let it sleep.
+ * 4. Move it asleep into the root cgroup. Its affinity gives it the
+ * isolated CPUs again, and their root domain counts it.
+ * 5. Set its affinity to the CPUs of H. That would take it out of the
+ * root domain that counts it, so it must fail with EBUSY.
+ *
+ * If the change is accepted, wake the task and report where it runs.
+ */
+static int affinity_moved_asleep(void)
+{
+ char cg_h[PATH_MAX], cpus[4096], what[64];
+ int fd[2], i, n = 0, cpu, err, made = 0, ret = KSFT_SKIP;
+ long long base[3], want[3], bw;
+ cpu_set_t set;
+ pid_t pid = -1;
+
+ if (split_skip(base))
+ return KSFT_SKIP;
+ if (pipe(fd)) {
+ ksft_print_msg("pipe: %s\n", strerror(errno));
+ return KSFT_FAIL;
+ }
+ /*
+ * H holds the housekeeping CPUs except P's. A task there can become
+ * a deadline task: its mask holds its whole root domain.
+ */
+ CPU_ZERO(&set);
+ for (i = 0; i < ncpus - 1 && n < (int)sizeof(cpus); i++) {
+ n += snprintf(cpus + n, sizeof(cpus) - n, "%s%d", i ? "," : "", online[i]);
+ CPU_SET(online[i], &set);
+ }
+ if (snprintf(cg_h, sizeof(cg_h), "%s/deadline_bw_h", root) >= (int)sizeof(cg_h))
+ goto out;
+ if (mkdir(cg_h, 0755) && errno != EEXIST) {
+ ksft_print_msg("mkdir %s: %s\n", cg_h, strerror(errno));
+ goto out;
+ }
+ made = 1;
+ if (n >= (int)sizeof(cpus) || write_file(cg_h, "cpuset.cpus", cpus)) {
+ ksft_print_msg("cpuset H cannot be made\n");
+ goto out;
+ }
+ ret = KSFT_FAIL;
+ mv_state[0] = 0;
+ mv_state[1] = 0;
+ pid = fork_child("a task moved asleep");
+ if (!pid) {
+ cpu_set_t iso;
+ char c;
+
+ CPU_ZERO(&iso);
+ for (i = 0; i < nisol; i++)
+ CPU_SET(isol[i], &iso);
+ if (sched_setaffinity(0, sizeof(iso), &iso)) {
+ mv_state[1] = -1;
+ _exit(1);
+ }
+ mv_state[1] = 1;
+ /* Wait for the move into H. Become a deadline task there. */
+ if (read(fd[0], &c, 1) != 1 || set_deadline(0, 30 * NSEC_PER_MSEC, PERIOD)) {
+ mv_state[0] = -1;
+ _exit(1);
+ }
+ spin_ns(NSEC_PER_MSEC);
+ mv_state[0] = 1;
+ while (read(fd[0], &c, 1) == 1)
+ mv_state[1]++;
+ for (;;)
+ pause();
+ }
+ if (pid < 0)
+ goto out;
+ if (wait_for(1000, mv_state[1]) || mv_state[1] != 1) {
+ ksft_print_msg("the task cannot take the isolated CPUs\n");
+ goto out;
+ }
+ err = enter(cg_h, pid);
+ if (err) {
+ ksft_print_msg("a move into H refused: %s\n", strerror(-err));
+ goto out;
+ }
+ if (write(fd[1], "d", 1) != 1) {
+ ksft_print_msg("write to the task: %s\n", strerror(errno));
+ goto out;
+ }
+ if (child_up("a task moved into H", 0) || wait_for(1000, proc_state_of(pid) == 'S'))
+ goto out;
+ bw = (30 * NSEC_PER_MSEC << 20) / PERIOD; /* as to_ratio() */
+ memcpy(want, base, sizeof(want));
+ want[1] += bw;
+ if (split_check("a deadline task in H", want))
+ goto out;
+
+ /*
+ * Moved asleep, it stays on its CPU. The root domain of the
+ * isolated CPUs counts it.
+ */
+ err = enter(root, pid);
+ if (err) {
+ ksft_print_msg("the move into the root cgroup refused: %s\n", strerror(-err));
+ goto out;
+ }
+ cpu = task_cpu_of(pid);
+ if (in_cpu_list(isolated, cpu)) {
+ ksft_print_msg("moved asleep on CPU %d, an isolated one\n", cpu);
+ goto out;
+ }
+ want[0] += bw;
+ want[1] -= bw;
+ if (split_check("moved asleep", want))
+ goto out;
+
+ /* The CPUs of H leave out the root domain that counts the task. */
+ err = sched_setaffinity(pid, sizeof(set), &set) ? -errno : 0;
+ if (err == -EBUSY) {
+ ret = KSFT_PASS;
+ goto out;
+ }
+ ksft_print_msg("the affinity of the CPUs of H: %s, not EBUSY\n",
+ err ? strerror(-err) : "accepted");
+ if (err || fallback_wake(fd[1], pid))
+ goto out;
+ /* Report its CPU, and whether the root domain of that CPU counts it. */
+ cpu = task_cpu_of(pid);
+ memcpy(want, base, sizeof(want));
+ want[in_cpu_list(isolated, cpu) ? 0 : 1] += bw;
+ snprintf(what, sizeof(what), "woken on CPU %d", cpu);
+ if (split_check(what, want) == KSFT_PASS)
+ ksft_print_msg("%s: its root domain counts it\n", what);
+out:
+ reap(pid);
+ close(fd[0]);
+ close(fd[1]);
+ sleep_ns(3 * PERIOD);
+ if (ret == KSFT_PASS)
+ ret = split_check("task gone", base);
+ if (made && wait_for(1000, !rmdir(cg_h) || errno == ENOENT))
+ ksft_print_msg("rmdir %s: %s\n", cg_h, strerror(errno));
+ return ret;
+}
+
/* Setup */
static int find_cgroup2(void)
@@ -5413,9 +7764,26 @@ static const struct p_case {
{ "return_asleep_new_params", return_asleep_new_params, MEASURED_IN_P },
{ "deboost_blocked_keeps_order", deboost_blocked_keeps_order, MEASURED_IN_P },
{ "pi_chain_donor_leaves", pi_chain_donor_leaves, MEASURED_IN_P },
+ { "move_asleep", move_asleep, FROM_OUTSIDE },
+ { "move_throttled", move_throttled, FROM_OUTSIDE },
+ { "move_non_contending", move_non_contending, FROM_OUTSIDE },
{ "move_refused", move_refused, FROM_OUTSIDE },
{ "move_zero_bw", move_zero_bw, FROM_OUTSIDE },
+ { "move_vs_rebuild", move_vs_rebuild, FROM_OUTSIDE },
+ { "move_became_deadline", move_became_deadline, FROM_OUTSIDE },
+ { "move_became_deadline_no_room", move_became_deadline_no_room, FROM_OUTSIDE },
+ { "move_affinity_changed", move_affinity_changed, FROM_OUTSIDE },
+ { "move_affinity_became_deadline", move_affinity_became_deadline, FROM_OUTSIDE },
+ { "move_died", move_died, FROM_OUTSIDE },
+ { "move_woken", move_woken, FROM_OUTSIDE },
+ { "move_woken_full", move_woken_full, FROM_OUTSIDE },
{ "move_cancelled", move_cancelled, FROM_OUTSIDE },
+ { "frozen_moved_switched", frozen_moved_switched, FROM_OUTSIDE },
+ { "move_isolated", move_isolated, FROM_OUTSIDE },
+ { "move_isolated_refused", move_isolated_refused, FROM_OUTSIDE },
+ { "move_isolated_asleep", move_isolated_asleep, FROM_OUTSIDE },
+ { "move_spanning", move_spanning, FROM_OUTSIDE },
+ { "affinity_moved_asleep", affinity_moved_asleep, FROM_OUTSIDE },
{ "death_running", death_running, MEASURED_IN_P },
{ "death_switched_out", death_switched_out, MEASURED_IN_P },
{ "death_shrunk", death_shrunk, MEASURED_IN_P },
@@ -5431,16 +7799,20 @@ static const struct p_case {
{ "death_zombie_switched", death_zombie_switched, FROM_OUTSIDE },
{ "death_zombie_hotplug", death_zombie_hotplug, FROM_OUTSIDE },
{ "yield_switched_out", yield_switched_out, IN_P },
+ { "yield_moved", yield_moved, FROM_OUTSIDE },
{ "yield_killed", yield_killed, MEASURED_IN_P },
{ "overrun_caught", overrun_caught, IN_P },
{ "overrun_group_death", overrun_group_death, MEASURED_IN_P },
{ "reclaim_runs", reclaim_runs, MEASURED_IN_P },
{ "reclaim_switched", reclaim_switched, MEASURED_IN_P },
+ { "reclaim_moved", reclaim_moved, FROM_OUTSIDE },
{ "server_params", server_params, IN_P },
{ "server_rebuilds", server_rebuilds, FROM_OUTSIDE },
{ "server_states", server_states, FROM_OUTSIDE },
{ "server_reclaim", server_reclaim, MEASURED_IN_P },
{ "params_waking", params_waking, FROM_OUTSIDE },
+ { "random_mix", random_mix, FROM_OUTSIDE },
+ { "move_normalized", move_normalized, FROM_OUTSIDE },
};
#define N_P_CASES ((int)ARRAY_SIZE(p_cases))
@@ -5457,6 +7829,7 @@ static const struct hp_case {
{ "hotplug_departed", hotplug_departed },
{ "hotplug_server", hotplug_server },
{ "affinity_vs_setattr", affinity_vs_setattr, 1 },
+ { "move_back_cpu_offline", move_back_cpu_offline },
};
#define N_HP_CASES ((int)ARRAY_SIZE(hp_cases))
@@ -5560,7 +7933,8 @@ int main(void)
if (read_file("/sys/devices/system/cpu/isolated", isolated, sizeof(isolated)))
isolated[0] = 0;
ncpus = read_online(&online, 0);
- if (ncpus < 0)
+ nisol = read_online(&isol, 1);
+ if (ncpus < 0 || nisol < 0)
ksft_exit_fail_msg("the online CPUs cannot be read\n");
if (ncpus < 3)
ksft_exit_skip("needs 3 CPUs not isolated by isolcpus=domain\n");
--
2.47.3
^ permalink raw reply related [flat|nested] 35+ messages in thread
* [PATCH v8 13/33] sched/deadline: Make a rebuild count a task in the root domain it runs in
2026-11-19 20:15 [PATCH v8 00/33] sched/deadline: bandwidth accounting fixes and selftests Mathieu Desnoyers
` (11 preceding siblings ...)
2026-11-19 20:15 ` [PATCH v8 12/33] selftests/sched: Test cpuset moves in deadline_bw_test Mathieu Desnoyers
@ 2026-11-19 20:15 ` Mathieu Desnoyers
2026-11-19 20:15 ` [PATCH v8 14/33] sched/deadline: Move a task's bandwidth to " Mathieu Desnoyers
` (19 subsequent siblings)
32 siblings, 0 replies; 35+ messages in thread
From: Mathieu Desnoyers @ 2026-11-19 20:15 UTC (permalink / raw)
To: juri.lelli, peterz, mingo, vincent.guittot
Cc: Mathieu Desnoyers, tglx, will, boqun, dietmar.eggemann, rostedt,
bsegall, mgorman, vschneid, kprateek.nayak, luca.abeni,
mathieu.poirier, longman, ridong.chen, tj, hannes, mkoutny, shuah,
bigeasy, clrkwllms, parri.andrea, m.szyprowski, kayracizmeci,
weihengqiuu, linux-kernel, cgroups, linux-kselftest,
linux-rt-devel, stable
A rebuild of the accounting counts each deadline task on the first
active CPU of its cpuset. Commit 318e18ed22e8 ("sched/deadline: Walk up
cpuset hierarchy to decide root domain when hot-unplug") made this
choice. It relies on the bandwidth check to keep such a CPU active.
This guess would go wrong in three ways:
- A cpuset, like the root one, can hold CPUs isolated by
isolcpus="domain" beside housekeeping CPUs. If an isolated CPU comes
first, the task would be counted in the default root domain. Its own
root domain would then be short of that bandwidth.
- A task confined to isolated CPUs can see them all go offline. The
rebuild would then find no active CPU, and hit its BUG_ON().
- dl_task_offline_migration() moves a throttled task, with its
bandwidth, off CPUs that went offline. The rebuild would count it in
its cpuset, not where it moved.
In the first and third cases, a root domain would admit deadline tasks
beyond its capacity.
Fix this by choosing the CPU in this order:
1. The CPU the task is on, if it is active and in the task's mask. A
mask can hold CPUs of two root domains, for instance after
select_fallback_rq() widened it. Also the CPU the task is queued on
out of its mask, if it is active and recorded with the bandwidth.
Without admission control, dl_task_offline_migration() can put a
throttled task there. It runs there until it blocks.
2. dl_bw_cpu, the CPU recorded with the bandwidth, if it is active and
in the mask. A task moved asleep is not on its new CPUs yet.
3. The first active CPU of the mask, a housekeeping one if any.
4. With no active CPU in the mask: the task falls back to its cpuset.
Count it there. Keep dl_bw_cpu if the cpuset holds it.
The mask of the task comes before its cpuset. A partition change sets
the task masks before its rebuild. dl_get_task_effective_cpus() reads
the CPUs isolated at boot, HK_TYPE_DOMAIN_BOOT. HK_TYPE_DOMAIN also
leaves out the CPUs of isolated partitions, and is updated after the
rebuild.
Leave out a task with no active CPU, instead of the BUG_ON(). dl_bw_gen
then keeps its release from removing anything.
dl_task_offline_migration() counts it again when it moves it.
A task that select_fallback_rq() moves to another root domain still runs
where its bandwidth is not counted. The next patch moves the bandwidth
with it.
The Sashiko AI reviews of this series found these cases. The case
rebuild_record_taken, added to deadline_bw_test later in this series,
tests the rule for a task with no active CPU in its mask.
Fixes: f9a25f776d78 ("cpusets: Rebuild root domain deadline accounting information")
Fixes: 318e18ed22e8 ("sched/deadline: Walk up cpuset hierarchy to decide root domain when hot-unplug")
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Cc: stable@vger.kernel.org
---
kernel/sched/deadline.c | 100 ++++++++++++++++++++++++++++++++++------
1 file changed, 87 insertions(+), 13 deletions(-)
diff --git a/kernel/sched/deadline.c b/kernel/sched/deadline.c
index 915c02bc6432..df55aba27ddc 100644
--- a/kernel/sched/deadline.c
+++ b/kernel/sched/deadline.c
@@ -926,14 +926,20 @@ static struct rq *dl_task_offline_migration(struct rq *rq, struct task_struct *p
__dl_sub(dl_b, p->dl.dl_bw,
cpumask_weight(src_rq->rd->span));
raw_spin_unlock(&dl_b->lock);
-
- dl_b = &later_rq->rd->dl_bw;
- raw_spin_lock(&dl_b->lock);
- __dl_add(dl_b, p->dl.dl_bw,
- cpumask_weight(later_rq->rd->span));
- raw_spin_unlock(&dl_b->lock);
}
+ /*
+ * Count the bandwidth in the root domain the task moves to.
+ * Do it also when a rebuild left the task out, for want of an
+ * active CPU. A rebuild that reaches the task after this finds
+ * its dl_bw_gen current, and leaves it alone.
+ */
+ dl_b = &later_rq->rd->dl_bw;
+ raw_spin_lock(&dl_b->lock);
+ __dl_add(dl_b, p->dl.dl_bw,
+ cpumask_weight(later_rq->rd->span));
+ raw_spin_unlock(&dl_b->lock);
p->dl.dl_bw_cpu = later_rq->cpu;
+ p->dl.dl_bw_gen = dl_bw_gen;
}
set_task_cpu(p, later_rq->cpu);
@@ -3790,8 +3796,14 @@ static void dl_get_task_effective_cpus(struct task_struct *p, struct cpumask *cp
{
const struct cpumask *hk_msk;
- hk_msk = housekeeping_cpumask(HK_TYPE_DOMAIN);
- if (housekeeping_enabled(HK_TYPE_DOMAIN)) {
+ /*
+ * Use the CPUs isolated at boot. HK_TYPE_DOMAIN also leaves out the
+ * CPUs of isolated partitions, and is updated after the rebuild. A
+ * CPU that just left such a partition is then in a new root domain,
+ * but HK_TYPE_DOMAIN still says it is isolated.
+ */
+ hk_msk = housekeeping_cpumask(HK_TYPE_DOMAIN_BOOT);
+ if (housekeeping_enabled(HK_TYPE_DOMAIN_BOOT)) {
if (!cpumask_intersects(p->cpus_ptr, hk_msk)) {
/*
* CPUs isolated by isolcpu="domain" always belong to
@@ -3810,6 +3822,62 @@ static void dl_get_task_effective_cpus(struct task_struct *p, struct cpumask *cp
cpuset_cpus_allowed_locked(p, cpus);
}
+/*
+ * Return a CPU of the root domain where a rebuild of the accounting should
+ * count the bandwidth of @p, or nr_cpu_ids if there is none.
+ *
+ * Prefer the CPU @p is on, if it is active and in the mask of @p. The
+ * task runs, or wakes up, in the root domain of that CPU. A mask can hold
+ * CPUs of two root domains: housekeeping CPUs and CPUs isolated by
+ * isolcpus="domain". When every CPU of a mask goes offline,
+ * select_fallback_rq() can widen it to both. It moves no bandwidth.
+ * Prefer that CPU also if @p is queued there, out of its mask, and the
+ * CPU is the one recorded. Without admission control,
+ * dl_task_offline_migration() can put a throttled task and its bandwidth
+ * on any active CPU. The task runs there until it blocks.
+ *
+ * Else prefer the CPU recorded with the bandwidth, if it is active and in
+ * the mask. It is in the root domain that admitted the bandwidth, or that
+ * its last move put it in. A task moved asleep is still on a CPU of the
+ * root domain it left, out of its mask. Else take the first active CPU of
+ * the mask, a housekeeping one if there is one. If no CPU of the mask is
+ * active, use the CPUs of the cpuset instead of the mask. A task on
+ * isolated CPUs that all went offline can have no active CPU at all.
+ */
+static unsigned int dl_task_rebuild_cpu(struct task_struct *p)
+{
+ struct cpumask *msk = this_cpu_cpumask_var_ptr(local_cpu_mask_dl);
+ /*
+ * Read the record once. The last schedule of a dead task clears it
+ * under its rq lock, and the caller holds only the pi_lock.
+ */
+ int recorded = READ_ONCE(p->dl.dl_bw_cpu);
+ unsigned int cpu;
+
+ /*
+ * Look at the mask of the task first. A change of the cpusets sets
+ * the mask of each task before its rebuild. If no CPU of the mask
+ * is active, ask the cpuset. The task wakes up on one of its CPUs.
+ * Take the CPU recorded only if it is one of them: a partition root
+ * can have taken it since it was recorded.
+ */
+ cpumask_and(msk, &p->cpus_mask, cpu_active_mask);
+ cpu = task_cpu(p);
+ if (cpumask_test_cpu(cpu, msk) ||
+ (cpu == recorded && cpu_active(cpu) && task_on_rq_queued(p)))
+ return cpu;
+ if (cpumask_empty(msk)) {
+ dl_get_task_effective_cpus(p, msk);
+ cpumask_and(msk, msk, cpu_active_mask);
+ }
+ if (recorded >= 0 && cpumask_test_cpu(recorded, msk))
+ return recorded;
+ cpu = cpumask_first_and(msk, housekeeping_cpumask(HK_TYPE_DOMAIN_BOOT));
+ if (cpu < nr_cpu_ids)
+ return cpu;
+ return cpumask_first(msk);
+}
+
/* From dl_rebuild_rd_accounting(), under the locks it asserts. */
void dl_add_task_root_domain(struct task_struct *p)
{
@@ -3817,7 +3885,6 @@ void dl_add_task_root_domain(struct task_struct *p)
struct rq *rq;
struct dl_bw *dl_b;
unsigned int cpu;
- struct cpumask *msk;
raw_spin_lock_irqsave(&p->pi_lock, rf.flags);
/*
@@ -3832,10 +3899,17 @@ void dl_add_task_root_domain(struct task_struct *p)
return;
}
- msk = this_cpu_cpumask_var_ptr(local_cpu_mask_dl);
- dl_get_task_effective_cpus(p, msk);
- cpu = cpumask_first_and(cpu_active_mask, msk);
- BUG_ON(cpu >= nr_cpu_ids);
+ cpu = dl_task_rebuild_cpu(p);
+ if (cpu >= nr_cpu_ids) {
+ /*
+ * No active CPU can count the bandwidth of this task. Leave it
+ * out of this rebuild. dl_bw_gen then keeps its release from
+ * removing anything. A later rebuild, or
+ * dl_task_offline_migration(), counts it again.
+ */
+ raw_spin_unlock_irqrestore(&p->pi_lock, rf.flags);
+ return;
+ }
raw_spin_lock(&dl_departed_lock);
/*
--
2.47.3
^ permalink raw reply related [flat|nested] 35+ messages in thread
* [PATCH v8 14/33] sched/deadline: Move a task's bandwidth to the root domain it runs in
2026-11-19 20:15 [PATCH v8 00/33] sched/deadline: bandwidth accounting fixes and selftests Mathieu Desnoyers
` (12 preceding siblings ...)
2026-11-19 20:15 ` [PATCH v8 13/33] sched/deadline: Make a rebuild count a task in the root domain it runs in Mathieu Desnoyers
@ 2026-11-19 20:15 ` Mathieu Desnoyers
2026-11-19 20:15 ` [PATCH v8 15/33] sched/deadline: Keep a task in the root domain that counts its bandwidth Mathieu Desnoyers
` (18 subsequent siblings)
32 siblings, 0 replies; 35+ messages in thread
From: Mathieu Desnoyers @ 2026-11-19 20:15 UTC (permalink / raw)
To: juri.lelli, peterz, mingo, vincent.guittot
Cc: Mathieu Desnoyers, tglx, will, boqun, dietmar.eggemann, rostedt,
bsegall, mgorman, vschneid, kprateek.nayak, luca.abeni,
mathieu.poirier, longman, ridong.chen, tj, hannes, mkoutny, shuah,
bigeasy, clrkwllms, parri.andrea, m.szyprowski, kayracizmeci,
weihengqiuu, linux-kernel, cgroups, linux-kselftest,
linux-rt-devel, stable
When a CPU goes offline, select_fallback_rq() picks a new CPU for a
deadline task asleep on it. balance_push() moves a queued task the same
way. The new CPU can be in another root domain. With isolcpus="domain",
the isolated CPUs are in the default root domain. A cpuset that is not
a partition can hold CPUs of both root domains.
Commit 59d06cea1198 ("sched/deadline: Fix bandwidth accounting at all
levels after offline migration") fixed this for a throttled task.
dl_task_offline_migration() moves its bandwidth with it. A task that
select_fallback_rq() places still moves no bandwidth. So it runs in a
root domain that does not count its bandwidth, until the next rebuild.
That root domain can admit deadline tasks beyond its capacity.
Fix this in migrate_task_rq_dl(). The CPU recorded with the bandwidth
gives the root domain that counts it. If the new CPU is in another root
domain, move the bandwidth there. Record the new CPU. As
dl_task_offline_migration() does, add the bandwidth even without room,
since the task runs there. Add it also if a rebuild left the task out.
Do not move it to a CPU that is not active. A migration-disabled task
can wake up on a CPU that goes offline. That CPU is in the default root
domain by then. The task leaves it before it runs anywhere else.
Compare the root domains, not the CPU the task leaves. That CPU can
still be in the mask when select_fallback_rq() picks a CPU of the other
root domain. A task can also wake up on its CPU with no migration while
another root domain counts its bandwidth. A cpuset move or a rebuild can
move the bandwidth while the task sleeps. A change of the mask can then
give that CPU back. So task_woken_dl() makes the same check.
Both checks read the record without dl_departed_lock, so a move inside
a root domain takes no lock. dl_task_move_bw() checks again under
dl_departed_lock. Its caller holds the pi_lock or the rq lock of the
task. Together, the two locks keep the record stable.
Move no bandwidth while the sched domains are rebuilt. The CPUs then
change root domains one by one. Two CPUs can be in different root
domains only because one of them was attached to its new root domain
first. dl_bw_rebuild_begin() opens this window in
partition_sched_domains_locked(). dl_clear_root_domains() closes it.
Then the walk of the task list counts each task. Until it reaches a
task, no move of that task is made: the walk counts it where it is.
Once the walk counted the task, its bandwidth moves again.
These cases, added to deadline_bw_test later in this series, fail
without this patch: - hotplug_fallback, hotplug_fallback_moved and
hotplug_fallback_in_mask: a task woken on an isolated CPU stays counted
in the housekeeping root domain. - no_admission_affinity_moves: without
admission control, a task given an affinity of one isolated CPU stays
counted in the housekeeping root domain.
This follows the Sashiko AI reviews of this series.
Fixes: 59d06cea1198 ("sched/deadline: Fix bandwidth accounting at all levels after offline migration")
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Cc: stable@vger.kernel.org
---
include/linux/sched/deadline.h | 1 +
kernel/sched/deadline.c | 184 ++++++++++++++++++++++++++++++---
kernel/sched/topology.c | 3 +
3 files changed, 176 insertions(+), 12 deletions(-)
diff --git a/include/linux/sched/deadline.h b/include/linux/sched/deadline.h
index cbb7cd71db58..bdabd54af729 100644
--- a/include/linux/sched/deadline.h
+++ b/include/linux/sched/deadline.h
@@ -33,6 +33,7 @@ struct root_domain;
extern void dl_add_task_root_domain(struct task_struct *p);
extern void dl_clear_root_domain(struct root_domain *rd);
extern void dl_clear_root_domains(void);
+extern void dl_bw_rebuild_begin(void);
extern void dl_rebuild_rd_accounting(void);
/*
* Keep a rebuild of the root domains' accounting and an admission of
diff --git a/kernel/sched/deadline.c b/kernel/sched/deadline.c
index df55aba27ddc..fb8f32a31cf9 100644
--- a/kernel/sched/deadline.c
+++ b/kernel/sched/deadline.c
@@ -268,6 +268,71 @@ static LIST_HEAD(dl_departed_list);
static DEFINE_RAW_SPINLOCK(dl_departed_lock);
static unsigned int dl_bw_gen;
+/*
+ * Odd while the sched domains are rebuilt: from before the first CPU
+ * leaves its root domain until dl_clear_root_domains(). Meanwhile the
+ * CPUs change root domains one by one. Housekeeping CPUs pass through
+ * def_root_domain, next to the CPUs that isolcpus="domain" isolates. So
+ * the root domain of the CPU recorded with a task says nothing about
+ * where its bandwidth is counted. The new root domains count no task
+ * yet.
+ *
+ * Changed under dl_departed_lock. dl_task_bw_elsewhere() reads it
+ * without the lock.
+ */
+static unsigned int dl_bw_rebuild_seq;
+
+/*
+ * True while dl_rebuild_rd_accounting() walks the task list: from
+ * dl_clear_root_domains() until the walk ends. The root domains are final
+ * by then. A task the walk has not reached yet has a dl_bw_gen that is
+ * not current. Changed under dl_departed_lock. dl_task_bw_elsewhere()
+ * reads it without the lock.
+ */
+static bool dl_bw_walking;
+
+/* Return true while the sched domains are rebuilt. */
+static inline bool dl_bw_rebuilding(void)
+{
+ return READ_ONCE(dl_bw_rebuild_seq) & 1;
+}
+
+/*
+ * Return true if the bandwidth of @p must not move now. Either the sched
+ * domains are rebuilt, or the walk of the task list that follows has not
+ * reached @p yet. The walk then counts @p. A move must not count it
+ * first.
+ *
+ * The answer is exact under dl_departed_lock. Without it, the answer is a
+ * hint: dl_task_move_bw() tests again under the lock.
+ */
+static inline bool dl_bw_move_blocked(struct task_struct *p)
+{
+ if (dl_bw_rebuilding())
+ return true;
+ return READ_ONCE(dl_bw_walking) &&
+ READ_ONCE(p->dl.dl_bw_gen) != READ_ONCE(dl_bw_gen);
+}
+
+/*
+ * Start a rebuild of the sched domains. Call it before the first CPU
+ * leaves its root domain. dl_clear_root_domains() ends the rebuild.
+ */
+void dl_bw_rebuild_begin(void)
+{
+ lockdep_assert_held(&sched_domains_mutex);
+ guard(raw_spinlock_irqsave)(&dl_departed_lock);
+ WRITE_ONCE(dl_bw_rebuild_seq, dl_bw_rebuild_seq + 1);
+}
+
+/* End the walk of the task list that dl_clear_root_domains() started. */
+static void dl_bw_walk_end(void)
+{
+ guard(raw_spinlock_irqsave)(&dl_departed_lock);
+ WARN_ON_ONCE(!dl_bw_walking);
+ WRITE_ONCE(dl_bw_walking, false);
+}
+
/*
* The number of tasks with a bandwidth: those whose dl_bw is not 0. A
* special entity and a server are not counted. __setparam_dl() and
@@ -2920,11 +2985,82 @@ select_task_rq_dl(struct task_struct *p, int cpu, int flags)
return cpu;
}
-static void migrate_task_rq_dl(struct task_struct *p, int new_cpu __maybe_unused)
+/*
+ * Return true if the root domain of @cpu may not count the bandwidth of
+ * @p. Either another root domain counts it, or a rebuild left @p out.
+ * Test without dl_departed_lock, so that a move inside a root domain
+ * takes no lock. dl_task_move_bw() tests again under it.
+ *
+ * Return false if @cpu is not active. A migration-disabled task can wake
+ * up on a CPU that goes offline, after it joined the default root domain.
+ * Keep the bandwidth where it is counted. balance_push() moves the task
+ * off that CPU once it enables migration, or its next wakeup does once
+ * the CPU is offline. That migration moves the bandwidth if needed.
+ *
+ * Return false while the sched domains are rebuilt. The CPUs then move
+ * to their new root domains one by one. That alone can make the two root
+ * domains differ. The rebuild of the accounting that follows counts @p.
+ * Return false also while that rebuild has not reached @p. Once it has
+ * counted @p, the root domains are final, and a move is right again.
+ */
+static bool dl_task_bw_elsewhere(struct task_struct *p, int cpu)
+{
+ int bw_cpu = READ_ONCE(p->dl.dl_bw_cpu);
+
+ if (bw_cpu < 0 || !p->dl.dl_bw || !cpu_active(cpu) ||
+ dl_bw_move_blocked(p))
+ return false;
+ return READ_ONCE(p->dl.dl_bw_gen) != READ_ONCE(dl_bw_gen) ||
+ cpu_rq(bw_cpu)->rd != cpu_rq(cpu)->rd;
+}
+
+/*
+ * Move the bandwidth of @p to the root domain of @new_cpu. Count it there
+ * also if a rebuild left @p out. Add it even without room, since @p runs
+ * there. The caller holds the rq lock of @p, or its pi_lock for a wakeup.
+ */
+static void dl_task_move_bw(struct task_struct *p, int new_cpu)
+{
+ struct dl_bw *dl_b;
+ int cpu;
+
+ if (dl_entity_is_special(&p->dl) || !p->dl.dl_bw)
+ return;
+
+ guard(raw_spinlock)(&dl_departed_lock);
+ cpu = p->dl.dl_bw_cpu;
+ if (cpu < 0 || dl_bw_move_blocked(p))
+ return;
+ if (p->dl.dl_bw_gen == dl_bw_gen) {
+ if (cpu_rq(cpu)->rd == cpu_rq(new_cpu)->rd)
+ return;
+ dl_b = dl_bw_of(cpu);
+ raw_spin_lock(&dl_b->lock);
+ __dl_sub(dl_b, p->dl.dl_bw, dl_bw_cpus(cpu));
+ raw_spin_unlock(&dl_b->lock);
+ }
+ dl_b = dl_bw_of(new_cpu);
+ raw_spin_lock(&dl_b->lock);
+ __dl_add(dl_b, p->dl.dl_bw, dl_bw_cpus(new_cpu));
+ raw_spin_unlock(&dl_b->lock);
+ p->dl.dl_bw_cpu = new_cpu;
+ p->dl.dl_bw_gen = dl_bw_gen;
+}
+
+static void migrate_task_rq_dl(struct task_struct *p, int new_cpu)
{
struct rq_flags rf;
struct rq *rq;
+ /*
+ * The new CPU can be in another root domain than the one that counts
+ * the bandwidth. A mask can hold CPUs of two root domains, and
+ * select_fallback_rq() can widen it. Move the bandwidth there, as
+ * dl_task_offline_migration() does for a throttled task.
+ */
+ if (unlikely(dl_task_bw_elsewhere(p, new_cpu)))
+ dl_task_move_bw(p, new_cpu);
+
if (READ_ONCE(p->__state) != TASK_WAKING)
return;
@@ -3587,6 +3723,15 @@ static void pull_dl_task(struct rq *this_rq)
*/
static void task_woken_dl(struct rq *rq, struct task_struct *p)
{
+ /*
+ * A task can wake up on the CPU it slept on, with no migration,
+ * while another root domain counts its bandwidth. The bandwidth
+ * moved while the task slept out of its mask, and the mask then got
+ * that CPU back. Move the bandwidth to this root domain.
+ */
+ if (unlikely(dl_task_bw_elsewhere(p, cpu_of(rq))))
+ dl_task_move_bw(p, cpu_of(rq));
+
if (!task_on_cpu(rq, p) &&
!test_tsk_need_resched(rq->curr) &&
p->nr_cpus_allowed > 1 &&
@@ -3622,9 +3767,10 @@ static void set_cpus_allowed_dl(struct task_struct *p,
* (different root_domains) entails a bandwidth update. Move the
* bandwidth only if the caller made room for it in the destination,
* as SCA_DL_BW_RESERVED says. Only cpuset_attach_task() sets it.
- * Otherwise, the bandwidth stays where it is counted, until the next
- * rebuild of the accounting. Moving it without room would leave no
- * root domain counting it.
+ * Otherwise, the bandwidth stays where it is counted, until the task
+ * runs in another root domain (migrate_task_rq_dl(), task_woken_dl()),
+ * or the next rebuild of the accounting. Freeing it here without room
+ * would leave no root domain counting it.
*
* cpuset_can_attach() made room for @p in the root domain of
* ctx->dl_bw_cpu, and recorded that CPU with @p. Move the bandwidth
@@ -3827,10 +3973,11 @@ static void dl_get_task_effective_cpus(struct task_struct *p, struct cpumask *cp
* count the bandwidth of @p, or nr_cpu_ids if there is none.
*
* Prefer the CPU @p is on, if it is active and in the mask of @p. The
- * task runs, or wakes up, in the root domain of that CPU. A mask can hold
- * CPUs of two root domains: housekeeping CPUs and CPUs isolated by
- * isolcpus="domain". When every CPU of a mask goes offline,
- * select_fallback_rq() can widen it to both. It moves no bandwidth.
+ * task runs, or most often wakes up, in the root domain of that CPU. A
+ * mask can hold CPUs of two root domains: housekeeping CPUs and CPUs
+ * isolated by isolcpus="domain". When every CPU of a mask goes offline,
+ * select_fallback_rq() can widen it to both. A task that then runs in
+ * the other root domain takes its bandwidth along.
* Prefer that CPU also if @p is queued there, out of its mask, and the
* CPU is the one recorded. Without admission control,
* dl_task_offline_migration() can put a throttled task and its bandwidth
@@ -3904,8 +4051,8 @@ void dl_add_task_root_domain(struct task_struct *p)
/*
* No active CPU can count the bandwidth of this task. Leave it
* out of this rebuild. dl_bw_gen then keeps its release from
- * removing anything. A later rebuild, or
- * dl_task_offline_migration(), counts it again.
+ * removing anything. A later rebuild counts it again. So does
+ * the move of the task to an active CPU.
*/
raw_spin_unlock_irqrestore(&p->pi_lock, rf.flags);
return;
@@ -4066,8 +4213,10 @@ void dl_rebuild_rd_accounting(void)
* a resume run one for each CPU. A task gets a bandwidth under the
* locks this rebuild holds, so the count cannot leave 0 meanwhile.
*/
- if (!atomic_read(&dl_nr_bw_tasks))
+ if (!atomic_read(&dl_nr_bw_tasks)) {
+ dl_bw_walk_end();
return;
+ }
read_lock(&tasklist_lock);
for_each_process_thread(g, p) {
@@ -4081,6 +4230,7 @@ void dl_rebuild_rd_accounting(void)
dl_add_task_root_domain(p);
}
read_unlock(&tasklist_lock);
+ dl_bw_walk_end();
}
/*
@@ -4099,7 +4249,17 @@ void dl_clear_root_domains(void)
int cpu;
guard(raw_spinlock_irqsave)(&dl_departed_lock);
- dl_bw_gen++;
+ /* dl_task_bw_elsewhere() reads it without the lock. */
+ WRITE_ONCE(dl_bw_gen, dl_bw_gen + 1);
+ /*
+ * The root domains are final. End the rebuild of the sched domains,
+ * and start the walk of the task list. From now on, a task that the
+ * walk has counted moves its bandwidth again. A task that the walk
+ * has not reached still waits for it: see dl_bw_move_blocked().
+ */
+ WARN_ON_ONCE(!(dl_bw_rebuild_seq & 1));
+ WRITE_ONCE(dl_bw_rebuild_seq, dl_bw_rebuild_seq + 1);
+ WRITE_ONCE(dl_bw_walking, true);
for_each_possible_cpu(cpu) {
if (!dl_bw_visited(cpu, cookie))
dl_clear_root_domain(cpu_rq(cpu)->rd);
diff --git a/kernel/sched/topology.c b/kernel/sched/topology.c
index 35e56a45c238..754756d2db00 100644
--- a/kernel/sched/topology.c
+++ b/kernel/sched/topology.c
@@ -3430,6 +3430,9 @@ static void partition_sched_domains_locked(int ndoms_new, cpumask_var_t doms_new
n = ndoms_new;
}
+ /* dl_clear_root_domains() ends it, in dl_rebuild_rd_accounting(). */
+ dl_bw_rebuild_begin();
+
/* Destroy deleted domains: */
for (i = 0; i < ndoms_cur; i++) {
for (j = 0; j < n && !new_topology; j++) {
--
2.47.3
^ permalink raw reply related [flat|nested] 35+ messages in thread
* [PATCH v8 15/33] sched/deadline: Keep a task in the root domain that counts its bandwidth
2026-11-19 20:15 [PATCH v8 00/33] sched/deadline: bandwidth accounting fixes and selftests Mathieu Desnoyers
` (13 preceding siblings ...)
2026-11-19 20:15 ` [PATCH v8 14/33] sched/deadline: Move a task's bandwidth to " Mathieu Desnoyers
@ 2026-11-19 20:15 ` Mathieu Desnoyers
2026-11-19 20:15 ` [PATCH v8 16/33] sched/deadline: Keep the last CPU a task has in its root domain Mathieu Desnoyers
` (17 subsequent siblings)
32 siblings, 0 replies; 35+ messages in thread
From: Mathieu Desnoyers @ 2026-11-19 20:15 UTC (permalink / raw)
To: juri.lelli, peterz, mingo, vincent.guittot
Cc: Mathieu Desnoyers, tglx, will, boqun, dietmar.eggemann, rostedt,
bsegall, mgorman, vschneid, kprateek.nayak, luca.abeni,
mathieu.poirier, longman, ridong.chen, tj, hannes, mkoutny, shuah,
bigeasy, clrkwllms, parri.andrea, m.szyprowski, kayracizmeci,
weihengqiuu, linux-kernel, cgroups, linux-kselftest,
linux-rt-devel, stable
The root domain that admits a deadline task counts its bandwidth. The
mask of the task can hold CPUs of another root domain. With
isolcpus="domain", the isolated CPUs are in the default root domain. A
cpuset that is not a partition root can hold isolated and housekeeping
CPUs.
The code that picks a CPU for a task does not look at the root domain
that counts it:
- select_fallback_rq() and dl_task_offline_migration() pick a CPU when
the CPU of the task goes offline.
- __set_cpus_allowed_ptr_locked() picks one when the mask changes.
- select_task_rq_dl() can leave a task on the CPU it slept on, which
can be in the other root domain.
- dl_task_rebuild_cpu() picks the CPU that a rebuild counts a task
with. It prefers the CPU the task is on, then a housekeeping CPU.
Each can take the task, or its bandwidth, to the other root domain
while its own root domain still has a CPU for it. The bandwidth then
moves with no admission test, and the root domain it arrives in can
exceed its limit.
Add dl_task_bw_span(), which returns the CPUs of the root domain that
counts a task. Make each of these places try one of those CPUs first.
After a rebuild the root domains are new, and the CPU recorded with the
bandwidth can be offline. So dl_task_rebuild_cpu() cannot use the root
domain. It looks at the kind of the CPU recorded, housekeeping or
isolated. It prefers a CPU of the mask of the same kind.
The CPU recorded can be the one going offline. Its root domain still
counts the task until the rebuild attaches that CPU to the default root
domain. So dl_task_bw_span() still returns that root domain until then.
This covers the common case: a task that runs on the CPU when it goes
offline, which balance_push() moves.
select_fallback_rq() can also widen the mask of the task. It first gives
the task the CPUs of its cpuset, then the housekeeping CPUs. At an
offline, this happens before the cpuset update. The wider mask can then
hold no active CPU of the root domain that counts the task. The task
then leaves that root domain, and its bandwidth moves with no admission
test. It stays out after the cpuset update. In testing without the rule
below, a deadline task with a period of 2ms spun on the CPU going
offline. It left this way in every run. So when the wider mask has no
active CPU of that root domain, allow the CPUs of the root domain
instead. The cpuset update that comes next sets the real mask.
dl_task_bw_span() returns NULL while the sched domains are rebuilt. The
CPUs then change root domains one by one. A housekeeping CPU passes
through the default root domain, which holds the isolated CPUs. Its
span would send a housekeeping task to an isolated CPU. So inside a
rebuild, the CPU is picked as it was before this patch. If that CPU is
in the other root domain, the next wakeup brings the task back.
dl_task_offline_migration() moves no bandwidth then either, nor
before the walk that follows counts the task. The walk counts it on
the side of the CPU recorded.
A known limit remains. A wakeup can pick its CPU inside a rebuild,
before the walk of the task list that follows reaches the task. If it
is queued after the walk counted the task, its bandwidth can still
move to the other root domain, with no admission test. This was found
by reading, not reproduced.
A task with no CPU left in its root domain still moves, and its
bandwidth with it. The next two patches refuse the changes that lead
to that.
The case hotplug_keeps_root_domain, added to deadline_bw_test later in
this series, tests this. A cpuset holds two isolated CPUs and two
housekeeping CPUs, H and K. A deadline task counted in the housekeeping
root domain sleeps on H. Then H goes offline. The task must wake up on
K. Without this patch it woke up on an isolated CPU: CPU 0, in a guest
with isolcpus=domain,0,1.
These cases test the other paths:
- hotplug_spinner_keeps: a running task whose CPU goes offline;
- hotplug_fallback_running: the same, when the cpuset of the task has
no CPU left;
- hotplug_waking_keeps: a task that wakes up often, at the offline;
- hotplug_isolated_keeps: the same, for a task counted on the isolated
side;
- rebuild_waking_keeps: a wakeup inside a rebuild. It fails in most
runs if dl_task_bw_span() does not return NULL there. That needs
isolcpus=domain,0,1.
Fixes: 332ac17ef5bf ("sched/deadline: Add bandwidth management for SCHED_DEADLINE tasks")
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Cc: stable@vger.kernel.org
---
kernel/sched/core.c | 55 +++++++++++++
kernel/sched/deadline.c | 173 +++++++++++++++++++++++++++++++++-------
kernel/sched/sched.h | 1 +
3 files changed, 201 insertions(+), 28 deletions(-)
diff --git a/kernel/sched/core.c b/kernel/sched/core.c
index 7615ba02d52b..fe2919be3775 100644
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -3264,6 +3264,27 @@ static int __set_cpus_allowed_ptr_locked(struct task_struct *p,
dest_cpu = ctx->dl_bw_cpu;
if (!task_on_rq_queued(p))
p->wake_cpu = dest_cpu;
+ } else {
+ const struct cpumask *span = dl_task_bw_span(p);
+ unsigned int cpu = nr_cpu_ids;
+
+ /*
+ * No room was made for @p in another root domain. Keep @p
+ * in the root domain that counts its bandwidth, if the new
+ * mask holds one of its CPUs. Also wake a sleeping @p up
+ * there. A wakeup during a rebuild of the sched domains has
+ * no span to prefer.
+ */
+ if (span && cpumask_test_cpu(dest_cpu, span))
+ cpu = dest_cpu;
+ else if (span)
+ cpu = cpumask_first_and_and(cpu_valid_mask,
+ ctx->new_mask, span);
+ if (cpu < nr_cpu_ids) {
+ dest_cpu = cpu;
+ if (!task_on_rq_queued(p))
+ p->wake_cpu = dest_cpu;
+ }
}
do_set_cpus_allowed(p, ctx);
@@ -3668,8 +3689,22 @@ static int select_fallback_rq(int cpu, struct task_struct *p)
int nid = cpu_to_node(cpu);
const struct cpumask *nodemask = NULL;
enum { cpuset, possible, fail } state = cpuset;
+ const struct cpumask *span;
+ bool span_forced = false;
int dest_cpu;
+ /*
+ * Keep a deadline task in the root domain that counts its
+ * bandwidth, if one of its CPUs is allowed and online.
+ */
+ span = dl_task_bw_span(p);
+ if (span) {
+ for_each_cpu_and(dest_cpu, span, p->cpus_ptr) {
+ if (is_cpu_allowed(p, dest_cpu))
+ return dest_cpu;
+ }
+ }
+
/*
* If the node that the CPU is on has been offlined, cpu_to_node()
* will return -1. There is no CPU on the node, and we should
@@ -3686,6 +3721,26 @@ static int select_fallback_rq(int cpu, struct task_struct *p)
}
for (;;) {
+ /* The mask is wider now: look again in that root domain. */
+ if (span && state != cpuset) {
+ for_each_cpu_and(dest_cpu, span, p->cpus_ptr) {
+ if (is_cpu_allowed(p, dest_cpu))
+ goto out;
+ }
+ /*
+ * The wider mask has no active CPU in that root
+ * domain. Allow the CPUs of the root domain instead.
+ * At an offline, the cpuset update comes next and sets
+ * the real mask.
+ */
+ if (!span_forced && cpumask_intersects(span, cpu_active_mask) &&
+ cpumask_subset(span, task_cpu_possible_mask(p))) {
+ span_forced = true;
+ set_cpus_allowed_force(p, span);
+ continue;
+ }
+ }
+
/* Any allowed, online CPU? */
for_each_cpu(dest_cpu, p->cpus_ptr) {
if (!is_cpu_allowed(p, dest_cpu))
diff --git a/kernel/sched/deadline.c b/kernel/sched/deadline.c
index fb8f32a31cf9..c33e73e920eb 100644
--- a/kernel/sched/deadline.c
+++ b/kernel/sched/deadline.c
@@ -277,8 +277,8 @@ static unsigned int dl_bw_gen;
* where its bandwidth is counted. The new root domains count no task
* yet.
*
- * Changed under dl_departed_lock. dl_task_bw_elsewhere() reads it
- * without the lock.
+ * Changed under dl_departed_lock. dl_task_bw_span() and
+ * dl_task_bw_elsewhere() read it without the lock.
*/
static unsigned int dl_bw_rebuild_seq;
@@ -323,6 +323,8 @@ void dl_bw_rebuild_begin(void)
lockdep_assert_held(&sched_domains_mutex);
guard(raw_spinlock_irqsave)(&dl_departed_lock);
WRITE_ONCE(dl_bw_rebuild_seq, dl_bw_rebuild_seq + 1);
+ /* Before the first write of rq->rd: see dl_task_bw_span(). */
+ smp_wmb();
}
/* End the walk of the task list that dl_clear_root_domains() started. */
@@ -927,18 +929,36 @@ static struct rq *find_lock_later_rq(struct task_struct *task, struct rq *rq);
static struct rq *dl_task_offline_migration(struct rq *rq, struct task_struct *p)
{
+ const struct cpumask *span;
struct rq *later_rq = NULL;
struct dl_bw *dl_b;
later_rq = find_lock_later_rq(p, rq);
+ /*
+ * find_lock_later_rq() looks in the root domain of @rq. Once @rq is
+ * offline, a rebuild attaches it to def_root_domain. That root domain
+ * may not count @p. Do not take @p out of the root domain that counts
+ * it while @p has an active CPU there.
+ */
+ span = dl_task_bw_span(p);
+ if (later_rq && span && !cpumask_test_cpu(later_rq->cpu, span) &&
+ cpumask_first_and_and(cpu_active_mask, p->cpus_ptr, span) < nr_cpu_ids) {
+ double_unlock_balance(rq, later_rq);
+ later_rq = NULL;
+ }
if (!later_rq) {
int cpu;
/*
* If we cannot preempt any rq, fall back to pick any
- * online CPU:
+ * online CPU. Try the root domain that counts @p first.
*/
- cpu = cpumask_any_and(cpu_active_mask, p->cpus_ptr);
+ cpu = nr_cpu_ids;
+ if (span)
+ cpu = cpumask_first_and_and(cpu_active_mask,
+ p->cpus_ptr, span);
+ if (cpu >= nr_cpu_ids)
+ cpu = cpumask_any_and(cpu_active_mask, p->cpus_ptr);
if (cpu >= nr_cpu_ids) {
/*
* Failed to find any suitable CPU.
@@ -981,8 +1001,15 @@ static struct rq *dl_task_offline_migration(struct rq *rq, struct task_struct *p
* moved to another cpuset while it was not running. If a rebuild
* dropped the bandwidth, there is nothing to move: the next rebuild
* counts it.
+ *
+ * Move nothing while the sched domains are rebuilt, or before the
+ * walk that follows has counted @p, as dl_task_move_bw() does. The
+ * CPU picked above then had no span to prefer. The walk counts @p
+ * on the side of the CPU recorded.
*/
scoped_guard (raw_spinlock, &dl_departed_lock) {
+ if (dl_bw_move_blocked(p))
+ break;
if (p->dl.dl_bw_gen == dl_bw_gen) {
struct rq *src_rq = cpu_rq(dl_task_root_domain_cpu(p));
@@ -2940,6 +2967,7 @@ static int
select_task_rq_dl(struct task_struct *p, int cpu, int flags)
{
struct task_struct *curr, *donor;
+ const struct cpumask *span;
bool select_rq;
struct rq *rq;
@@ -2980,6 +3008,20 @@ select_task_rq_dl(struct task_struct *p, int cpu, int flags)
dl_task_is_earliest_deadline(p, cpu_rq(target)))
cpu = target;
}
+
+ /*
+ * Keep @p in the root domain that counts its bandwidth. Its mask
+ * can hold CPUs of another root domain, with isolcpus="domain",
+ * and @p can sleep on one of them.
+ */
+ span = dl_task_bw_span(p);
+ if (span && !cpumask_test_cpu(cpu, span)) {
+ unsigned int in_span;
+
+ in_span = cpumask_first_and_and(span, p->cpus_ptr, cpu_active_mask);
+ if (in_span < nr_cpu_ids)
+ cpu = in_span;
+ }
rcu_read_unlock();
return cpu;
@@ -3757,6 +3799,61 @@ static unsigned int dl_task_root_domain_cpu(struct task_struct *p)
return cpu;
}
+/*
+ * Return the CPUs of the root domain that counts the bandwidth of @p, or
+ * NULL if none does. The mask of @p can hold CPUs of another root domain,
+ * with isolcpus="domain". Where a CPU is picked for @p, a CPU of this
+ * root domain comes first: no other root domain admitted @p.
+ *
+ * Return NULL while the sched domains are rebuilt. The CPU recorded can
+ * then sit in def_root_domain on its way to a new root domain. The span
+ * of def_root_domain holds the isolated CPUs, and no longer the
+ * housekeeping CPUs that already left it. A caller would send a
+ * housekeeping task to an isolated CPU.
+ *
+ * Return NULL also for a task that the walk of the task list after the
+ * rebuild has not reached yet. Its dl_bw_gen is not current.
+ *
+ * The caller holds the pi_lock or the rq lock of @p. With interrupts
+ * off, the root domain is not freed.
+ */
+const struct cpumask *dl_task_bw_span(struct task_struct *p)
+{
+ unsigned int seq = READ_ONCE(dl_bw_rebuild_seq);
+ struct root_domain *rd;
+ int cpu;
+
+ if (!task_has_dl_policy(p) || dl_entity_is_special(&p->dl) ||
+ !p->dl.dl_bw || (seq & 1))
+ return NULL;
+ cpu = READ_ONCE(p->dl.dl_bw_cpu);
+ if (cpu < 0 || READ_ONCE(p->dl.dl_bw_gen) != READ_ONCE(dl_bw_gen))
+ return NULL;
+ /*
+ * The CPU recorded can be going offline. It keeps its root domain
+ * until the rebuild attaches it to def_root_domain. The rebuild then
+ * records an active CPU with @p, or leaves @p out. Until then, the
+ * root domain of the CPU recorded still counts @p.
+ *
+ * A CPU that isolcpus="domain" isolates is always in
+ * def_root_domain, which then counts @p. A housekeeping CPU that is
+ * not active and in def_root_domain was attached there by the
+ * rebuild. def_root_domain does not count @p then.
+ */
+ rd = cpu_rq(cpu)->rd;
+ /*
+ * A rebuild that started since writes rq->rd after it made
+ * dl_bw_rebuild_seq odd. Use this root domain only if none did.
+ */
+ smp_rmb();
+ if (READ_ONCE(dl_bw_rebuild_seq) != seq)
+ return NULL;
+ if (!cpu_active(cpu) && rd == &def_root_domain &&
+ housekeeping_test_cpu(cpu, HK_TYPE_DOMAIN_BOOT))
+ return NULL;
+ return rd->span;
+}
+
static void set_cpus_allowed_dl(struct task_struct *p,
struct affinity_context *ctx)
{
@@ -3972,24 +4069,24 @@ static void dl_get_task_effective_cpus(struct task_struct *p, struct cpumask *cp
* Return a CPU of the root domain where a rebuild of the accounting should
* count the bandwidth of @p, or nr_cpu_ids if there is none.
*
- * Prefer the CPU @p is on, if it is active and in the mask of @p. The
- * task runs, or most often wakes up, in the root domain of that CPU. A
- * mask can hold CPUs of two root domains: housekeeping CPUs and CPUs
- * isolated by isolcpus="domain". When every CPU of a mask goes offline,
- * select_fallback_rq() can widen it to both. A task that then runs in
- * the other root domain takes its bandwidth along.
- * Prefer that CPU also if @p is queued there, out of its mask, and the
- * CPU is the one recorded. Without admission control,
- * dl_task_offline_migration() can put a throttled task and its bandwidth
- * on any active CPU. The task runs there until it blocks.
+ * A mask can hold CPUs of two root domains: housekeeping CPUs and CPUs
+ * isolated by isolcpus="domain". Keep the bandwidth on the side of the
+ * CPU recorded with it. That CPU is in the root domain that admitted the
+ * bandwidth, or that its last move put it in. Take the CPU @p is on if it
+ * is active, in the mask of @p and on that side. Else take the CPU
+ * recorded, if it is active and in the mask. Else take a CPU of the mask
+ * on that side.
*
- * Else prefer the CPU recorded with the bandwidth, if it is active and in
- * the mask. It is in the root domain that admitted the bandwidth, or that
- * its last move put it in. A task moved asleep is still on a CPU of the
- * root domain it left, out of its mask. Else take the first active CPU of
- * the mask, a housekeeping one if there is one. If no CPU of the mask is
- * active, use the CPUs of the cpuset instead of the mask. A task on
- * isolated CPUs that all went offline can have no active CPU at all.
+ * With no CPU on that side, or no CPU recorded, prefer the CPU @p is on.
+ * Else take the first active CPU of the mask, a housekeeping one if there
+ * is one. A task on isolated CPUs that all went offline can have no
+ * active CPU at all.
+ *
+ * One case comes before all of these. Take the CPU @p is on if @p is
+ * queued there, out of its mask, and the CPU is the one recorded. Without
+ * admission control, dl_task_offline_migration() can put a throttled task
+ * and its bandwidth on any active CPU. The task runs there until it
+ * blocks.
*/
static unsigned int dl_task_rebuild_cpu(struct task_struct *p)
{
@@ -4004,21 +4101,41 @@ static unsigned int dl_task_rebuild_cpu(struct task_struct *p)
/*
* Look at the mask of the task first. A change of the cpusets sets
* the mask of each task before its rebuild. If no CPU of the mask
- * is active, ask the cpuset. The task wakes up on one of its CPUs.
- * Take the CPU recorded only if it is one of them: a partition root
- * can have taken it since it was recorded.
+ * is active, ask the cpuset. The CPU recorded is taken below only if
+ * the cpuset holds it: a partition root can have taken it since it
+ * was recorded.
*/
cpumask_and(msk, &p->cpus_mask, cpu_active_mask);
cpu = task_cpu(p);
- if (cpumask_test_cpu(cpu, msk) ||
- (cpu == recorded && cpu_active(cpu) && task_on_rq_queued(p)))
+ if (cpu == recorded && cpu_active(cpu) && task_on_rq_queued(p))
return cpu;
if (cpumask_empty(msk)) {
dl_get_task_effective_cpus(p, msk);
cpumask_and(msk, msk, cpu_active_mask);
}
- if (recorded >= 0 && cpumask_test_cpu(recorded, msk))
- return recorded;
+ /*
+ * Keep the bandwidth where it was counted: with the housekeeping
+ * CPUs, or with the CPUs that isolcpus="domain" isolates. The root
+ * domains are new, and the CPU recorded can be offline. So prefer a
+ * CPU of the mask of the same kind as the CPU recorded: housekeeping
+ * or not.
+ */
+ if (recorded >= 0) {
+ const struct cpumask *hk = housekeeping_cpumask(HK_TYPE_DOMAIN_BOOT);
+ bool rec_hk = cpumask_test_cpu(recorded, hk);
+ unsigned int same;
+
+ if (cpumask_test_cpu(cpu, msk) && cpumask_test_cpu(cpu, hk) == rec_hk)
+ return cpu;
+ if (cpumask_test_cpu(recorded, msk))
+ return recorded;
+ for_each_cpu(same, msk) {
+ if (cpumask_test_cpu(same, hk) == rec_hk)
+ return same;
+ }
+ }
+ if (cpumask_test_cpu(cpu, msk))
+ return cpu;
cpu = cpumask_first_and(msk, housekeeping_cpumask(HK_TYPE_DOMAIN_BOOT));
if (cpu < nr_cpu_ids)
return cpu;
diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h
index a719b62dc114..4f1e7ad94f80 100644
--- a/kernel/sched/sched.h
+++ b/kernel/sched/sched.h
@@ -361,6 +361,7 @@ extern bool __checkparam_dl(const struct sched_attr *attr);
extern bool dl_param_changed(struct task_struct *p, const struct sched_attr *attr);
extern int dl_cpuset_cpumask_can_shrink(const struct cpumask *cur, const struct cpumask *trial);
extern int dl_bw_deactivate(int cpu);
+extern const struct cpumask *dl_task_bw_span(struct task_struct *p);
extern s64 dl_scaled_delta_exec(struct rq *rq, struct sched_dl_entity *dl_se, s64 delta_exec);
/*
* SCHED_DEADLINE supports servers (nested scheduling) with the following
--
2.47.3
^ permalink raw reply related [flat|nested] 35+ messages in thread
* [PATCH v8 16/33] sched/deadline: Keep the last CPU a task has in its root domain
2026-11-19 20:15 [PATCH v8 00/33] sched/deadline: bandwidth accounting fixes and selftests Mathieu Desnoyers
` (14 preceding siblings ...)
2026-11-19 20:15 ` [PATCH v8 15/33] sched/deadline: Keep a task in the root domain that counts its bandwidth Mathieu Desnoyers
@ 2026-11-19 20:15 ` Mathieu Desnoyers
2026-11-19 20:22 ` Mathieu Desnoyers
2026-11-19 20:15 ` [PATCH v8 17/33] cgroup/cpuset: Refuse to take a deadline task out of " Mathieu Desnoyers
` (16 subsequent siblings)
32 siblings, 1 reply; 35+ messages in thread
From: Mathieu Desnoyers @ 2026-11-19 20:15 UTC (permalink / raw)
To: juri.lelli, peterz, mingo, vincent.guittot
Cc: Mathieu Desnoyers, tglx, will, boqun, dietmar.eggemann, rostedt,
bsegall, mgorman, vschneid, kprateek.nayak, luca.abeni,
mathieu.poirier, longman, ridong.chen, tj, hannes, mkoutny, shuah,
bigeasy, clrkwllms, parri.andrea, m.szyprowski, kayracizmeci,
weihengqiuu, linux-kernel, cgroups, linux-kselftest,
linux-rt-devel, stable
dl_bw_deactivate() refuses to take a CPU offline when the rest of its
root domain cannot hold the deadline bandwidth counted there. It also
keeps the last CPU of a root domain that counts a deadline task. That
test comes from commit d4742f6ed7ea ("sched/deadline: Correctly account
for allocated bandwidth during hotplug").
That test looks at the root domain, not at the mask of each task. With
isolcpus="domain", a cpuset that is not a partition root can hold one
housekeeping CPU and some isolated CPUs. A deadline task in it can be
counted in the housekeeping root domain. When that housekeeping CPU
goes offline, the root domain keeps its other CPUs, so the offline is
accepted. But the task has only isolated CPUs left. It runs in the
default root domain, which did not admit it. Its bandwidth follows with
no admission test, and the default root domain can exceed its limit.
Refuse the offline with EBUSY in that case. Walk the deadline tasks
counted in the root domain of the CPU. Refuse if one of them has CPUs
left to run on, but none in that root domain. Skip the walk if no task
holds a bandwidth. A task whose mask holds no other CPU gets the CPUs
its cpuset has after the offline. With no CPU left in a cpuset, cgroup
v2 gives it the CPUs of its parent, worked out from the top down. A
partition root that holds a task and has no CPU of its own left takes
the CPUs of a child partition instead. cgroup v1 moves its tasks to the
nearest ancestor with CPUs. Add cpuset_cpus_allowed_without(), which
works out the CPUs the same way. Like __set_cpus_allowed_ptr(), it keeps
them within the affinity set by the user, if they intersect. If the
cpuset gives no CPU the task can run on, select_fallback_rq() picks one.
The previous patch makes it keep a deadline task in its root domain when
it can. Skip a dead task: it runs no more.
When a partition root takes the CPUs of a child partition, cpuset makes
that child invalid, with its own child partitions. Their CPUs then join
the root domain of the parent, with their tasks. So a deadline task can
change root domain even if another root domain than that of the CPU
counts it. The walk above does not see it. In testing without this
check, an offline put three deadline tasks of 40% in a root domain of
one CPU. That is over its limit. Refuse the offline if one of these
cpusets holds a deadline task. Add cpuset_offline_moves_dl() for this.
The CPUs keep their root domain in two cases, and the offline is then
accepted. Both partitions are isolated: the CPUs stay in the default
root domain. Or neither is isolated, and the parent takes all the CPUs
of its child: its root domain then spans the same CPUs. When the last
CPU left to the top cpuset goes offline, cpuset makes every partition
invalid. Refuse that offline if a partition holds a deadline task.
Do not refuse during a suspend. It takes the CPUs offline with the
tasks frozen, and the resume puts them back. Do not refuse without
admission control either: there is no limit to exceed.
An administrator who needs the CPU offline can first give the cpuset of
the task another CPU of that root domain. The offline is then accepted,
and the task wakes up on that CPU: the hotplug_keeps_root_domain case
tests this. The administrator can also turn admission control off, by
writing -1 to /proc/sys/kernel/sched_rt_runtime_us.
The walk cannot miss a task. sched_setattr() only makes a deadline task
of a task whose mask holds its whole root domain. sched_setaffinity()
only gives a deadline task such a mask. A mask also gets smaller by a
cpuset change or a cpuset move, and both exclude CPU hotplug.
For user-space, the refused offline is a write of 0 to
/sys/devices/system/cpu/cpuN/online that fails with EBUSY. An offline
that leaves each deadline task a CPU in its root domain is accepted as
before.
The case hotplug_last_refused, added to deadline_bw_test later in this
series, tests this. Without this patch the offline is accepted. The
cases hotplug_user_mask_refused, hotplug_nested_refused and
hotplug_dead_accepted test the affinity set by the user, nested cpusets
and a dead task. hotplug_child_partition_kept and
hotplug_partition_invalid_refused test a child partition made invalid.
Fixes: d4742f6ed7ea ("sched/deadline: Correctly account for allocated bandwidth during hotplug")
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Cc: stable@vger.kernel.org
---
include/linux/cpuset.h | 17 +++
kernel/cgroup/cpuset.c | 250 ++++++++++++++++++++++++++++++++++++++++
kernel/sched/deadline.c | 72 +++++++++++-
3 files changed, 338 insertions(+), 1 deletion(-)
diff --git a/include/linux/cpuset.h b/include/linux/cpuset.h
index 5d0c01a25354..d3fceb919c46 100644
--- a/include/linux/cpuset.h
+++ b/include/linux/cpuset.h
@@ -16,6 +16,7 @@
#include <linux/nodemask.h>
#include <linux/mm.h>
#include <linux/mmu_context.h>
+#include <linux/sched/isolation.h>
#include <linux/jump_label.h>
extern bool lockdep_is_cpuset_held(void);
@@ -79,6 +80,9 @@ extern void lockdep_assert_cpuset_lock_held(void);
extern void cpuset_cpus_allowed_locked(struct task_struct *p, struct cpumask *mask);
extern void cpuset_cpus_allowed(struct task_struct *p, struct cpumask *mask);
extern bool cpuset_cpus_allowed_fallback(struct task_struct *p);
+extern bool cpuset_cpus_allowed_without(struct task_struct *p, int cpu,
+ struct cpumask *mask);
+extern bool cpuset_offline_moves_dl(int cpu);
extern int cpuset_num_cpus(struct cgroup *cgroup);
extern nodemask_t cpuset_mems_allowed(struct task_struct *p);
#define cpuset_current_mems_allowed (current->mems_allowed)
@@ -214,6 +218,19 @@ static inline bool cpuset_cpus_allowed_fallback(struct task_struct *p)
return false;
}
+static inline bool cpuset_cpus_allowed_without(struct task_struct *p, int cpu,
+ struct cpumask *mask)
+{
+ cpumask_and(mask, task_cpu_fallback_mask(p), cpu_active_mask);
+ cpumask_clear_cpu(cpu, mask);
+ return false;
+}
+
+static inline bool cpuset_offline_moves_dl(int cpu)
+{
+ return false;
+}
+
static inline int cpuset_num_cpus(struct cgroup *cgroup)
{
return num_online_cpus();
diff --git a/kernel/cgroup/cpuset.c b/kernel/cgroup/cpuset.c
index 7bdb9e21a7ed..5299652d47b6 100644
--- a/kernel/cgroup/cpuset.c
+++ b/kernel/cgroup/cpuset.c
@@ -1126,6 +1126,28 @@ static void update_partition_sd_lb(struct cpuset *cs, int old_prs)
cpuset_force_rebuild();
}
+/*
+ * Set @new_cpus to the effective CPUs that @cp gets. On entry, @new_cpus
+ * holds those that @from gets. @from is @cp or one of its ancestors. Work
+ * down from the child of @from to @cp, as compute_effective_cpumask() does
+ * for a cpuset that is not a valid partition root.
+ */
+static void effective_cpus_below(struct cpuset *cp, struct cpuset *from,
+ struct cpumask *new_cpus)
+{
+ struct cpuset *a;
+ int depth = 0, i;
+
+ for (a = cp; a != from; a = parent_cs(a))
+ depth++;
+ while (depth--) {
+ for (a = cp, i = 0; i < depth; i++)
+ a = parent_cs(a);
+ if (cpumask_intersects(new_cpus, a->cpus_allowed) || !is_in_v2_mode())
+ cpumask_and(new_cpus, new_cpus, a->cpus_allowed);
+ }
+}
+
/*
* tasks_nocpu_error - Return true if tasks will have no effective_cpus
*/
@@ -4312,6 +4334,234 @@ void cpuset_cpus_allowed(struct task_struct *tsk, struct cpumask *pmask)
spin_unlock_irqrestore(&callback_lock, flags);
}
+/*
+ * Return true if @cs holds a task, or one of its cpusets that is not a
+ * valid partition root does. This is partition_is_populated() for the
+ * offline path, which holds the CPU hotplug lock for writing instead of
+ * cpuset_mutex. That lock keeps tasks from moving to another cgroup:
+ * cgroup_attach_lock() takes it. The caller holds rcu_read_lock().
+ */
+static bool partition_populated_offline(struct cpuset *cs)
+{
+ struct cgroup_subsys_state *pos_css;
+ struct cpuset *cp;
+
+ if (cgroup_has_tasks(cs->css.cgroup))
+ return true;
+ cpuset_for_each_descendant_pre(cp, pos_css, cs) {
+ if (cp == cs)
+ continue;
+ if (is_partition_valid(cp)) {
+ pos_css = css_rightmost_descendant(pos_css);
+ continue;
+ }
+ if (cgroup_is_populated(cp->css.cgroup))
+ return true;
+ }
+ return false;
+}
+
+/*
+ * Return the child partition of the valid partition root @p that the
+ * offline of @cpu makes invalid, or NULL. Set @new_cpus to the active CPUs
+ * that @p then has. This follows compute_partition_effective_cpumask().
+ * The offline can leave a populated @p no CPU of its own. The child
+ * partition that holds the last CPUs left then becomes invalid, and @p
+ * takes its CPUs. The caller holds rcu_read_lock().
+ */
+static struct cpuset *partition_child_lost(struct cpuset *p, int cpu,
+ struct cpumask *new_cpus)
+{
+ bool populated = partition_populated_offline(p);
+ struct cgroup_subsys_state *css;
+ struct cpuset *child, *lost = NULL;
+
+ cpumask_and(new_cpus, p->effective_xcpus, cpu_active_mask);
+ cpumask_clear_cpu(cpu, new_cpus);
+ if (cpumask_empty(new_cpus))
+ return NULL;
+ cpuset_for_each_child(child, css, p) {
+ if (!is_partition_valid(child))
+ continue;
+ if (populated && cpumask_subset(new_cpus, child->effective_xcpus)) {
+ lost = child;
+ continue;
+ }
+ cpumask_andnot(new_cpus, new_cpus, child->effective_xcpus);
+ }
+ return lost;
+}
+
+/*
+ * Set @pmask to the active CPUs that the cpuset of @tsk gives it once @cpu
+ * is offline. Keep the CPUs possible for @tsk, as
+ * cpuset_update_tasks_cpumask() does. As __set_cpus_allowed_ptr() does,
+ * keep those of the affinity the user set, if one is left. Return true.
+ *
+ * If no CPU possible for @tsk is left, the cpuset cannot set the mask of
+ * @tsk. select_fallback_rq() then picks the CPU. It widens the mask to
+ * cpuset.cpus first, on cgroup v2, then to task_cpu_fallback_mask(): the
+ * housekeeping CPUs. It drops the affinity the user set. Give the active
+ * CPUs of that wider mask, and return false. For a deadline task,
+ * select_fallback_rq() may then pick a CPU of its root domain instead.
+ * The caller checks for that.
+ *
+ * The caller holds the CPU hotplug lock for writing, which keeps the
+ * cpusets and their tasks as they are. It does not keep the housekeeping
+ * CPUs: housekeeping_update() does not take that lock. The caller holds
+ * the pi_lock of @tsk, which keeps its user affinity.
+ */
+bool cpuset_cpus_allowed_without(struct task_struct *tsk, int cpu,
+ struct cpumask *pmask)
+{
+ const struct cpumask *possible = task_cpu_possible_mask(tsk);
+ struct cpuset *cs = task_cs(tsk), *from;
+
+ guard(rcu)();
+ if (!is_in_v2_mode()) {
+ /*
+ * cgroup v1 moves the tasks of a cpuset left with no CPU to
+ * the nearest ancestor that keeps one.
+ */
+ for (from = cs; from; from = parent_cs(from)) {
+ cpumask_and(pmask, from->effective_cpus, cpu_active_mask);
+ cpumask_clear_cpu(cpu, pmask);
+ if (cpumask_intersects(pmask, possible))
+ break;
+ }
+ } else {
+ /*
+ * cgroup v2 works out the CPUs of each cpuset from the top
+ * down. A cpuset keeps those of its parent that cpuset.cpus
+ * holds, or takes all of them if there are none. Start from
+ * the nearest partition root that keeps an active CPU, or from
+ * the top cpuset. A partition root left with no CPU of its own
+ * can take those of a child partition.
+ */
+ for (from = cs; ; from = parent_cs(from)) {
+ if (from != &top_cpuset && !is_partition_valid(from))
+ continue;
+ cpumask_and(pmask, from->effective_cpus, cpu_active_mask);
+ cpumask_clear_cpu(cpu, pmask);
+ if (cpumask_empty(pmask) && from != &top_cpuset &&
+ !partition_child_lost(from, cpu, pmask))
+ cpumask_clear(pmask);
+ if (!cpumask_empty(pmask) || from == &top_cpuset)
+ break;
+ }
+ effective_cpus_below(cs, from, pmask);
+ }
+ if (!cpumask_and(pmask, pmask, possible)) {
+ if (is_in_v2_mode() && cpumask_subset(cs->cpus_allowed, possible)) {
+ cpumask_and(pmask, cs->cpus_allowed, cpu_active_mask);
+ cpumask_clear_cpu(cpu, pmask);
+ if (!cpumask_empty(pmask))
+ return false;
+ }
+ cpumask_and(pmask, task_cpu_fallback_mask(tsk), cpu_active_mask);
+ cpumask_clear_cpu(cpu, pmask);
+ return false;
+ }
+ if (tsk->user_cpus_ptr && cpumask_intersects(pmask, tsk->user_cpus_ptr))
+ cpumask_and(pmask, pmask, tsk->user_cpus_ptr);
+ return true;
+}
+
+/*
+ * Scratch mask of cpuset_offline_moves_dl(). The CPU hotplug lock
+ * protects it.
+ */
+static cpumask_t dl_offline_cpus;
+
+/*
+ * Return true if a task of @cs has the deadline policy. A dead task does
+ * not count: it runs no more.
+ */
+static bool dl_task_in_cpuset(struct cpuset *cs)
+{
+ struct css_task_iter it;
+ struct task_struct *task;
+ bool found = false;
+
+ css_task_iter_start(&cs->css, 0, &it);
+ while (!found && (task = css_task_iter_next(&it)))
+ found = task->policy == SCHED_DEADLINE &&
+ READ_ONCE(task->__state) != TASK_DEAD;
+ css_task_iter_end(&it);
+ return found;
+}
+
+/*
+ * Return true if the offline of @cpu makes a partition invalid, and a
+ * deadline task of its cpusets then runs in another root domain. That root
+ * domain did not admit the task.
+ *
+ * The partition root @p that holds @cpu can be left with no CPU of its
+ * own. It then takes the CPUs of a child partition, which becomes invalid
+ * with its own child partitions. A task of these cpusets keeps its root
+ * domain in two cases only. Its partition is isolated, and so is @p: the
+ * CPUs stay in the default root domain. Or its partition is the child
+ * whose CPUs @p takes, and neither is isolated. The root domain of @p
+ * then spans the same CPUs. If @cpu is the last CPU left to the top
+ * cpuset, every partition becomes invalid, and every task of a partition
+ * changes root domain.
+ *
+ * The caller holds the CPU hotplug lock for writing, which keeps the
+ * cpusets and their tasks as they are.
+ */
+bool cpuset_offline_moves_dl(int cpu)
+{
+ struct cgroup_subsys_state *pos_css;
+ struct cpuset *cp, *a, *p = &top_cpuset, *lost = NULL;
+ bool found = false;
+
+ if (!is_in_v2_mode())
+ return false;
+ rcu_read_lock();
+ cpuset_for_each_descendant_pre(cp, pos_css, &top_cpuset) {
+ if (cp != &top_cpuset && is_partition_valid(cp) &&
+ cpumask_test_cpu(cpu, cp->effective_cpus)) {
+ p = cp;
+ break;
+ }
+ }
+ if (p != &top_cpuset) {
+ lost = partition_child_lost(p, cpu, &dl_offline_cpus);
+ } else {
+ cpumask_and(&dl_offline_cpus, top_cpuset.effective_cpus, cpu_active_mask);
+ cpumask_clear_cpu(cpu, &dl_offline_cpus);
+ if (cpumask_empty(&dl_offline_cpus) && !cpumask_empty(subpartitions_cpus))
+ lost = &top_cpuset;
+ }
+ if (!lost) {
+ rcu_read_unlock();
+ return false;
+ }
+ cpuset_for_each_descendant_pre(cp, pos_css, lost) {
+ if (cp == &top_cpuset)
+ continue;
+ /* The partition that gives @cp its CPUs. */
+ for (a = cp; a != lost && !is_partition_valid(a); a = parent_cs(a))
+ ;
+ if (a == &top_cpuset ||
+ (a->partition_root_state == PRS_ISOLATED &&
+ p->partition_root_state == PRS_ISOLATED) ||
+ (a == lost && a->partition_root_state != PRS_ISOLATED &&
+ p->partition_root_state != PRS_ISOLATED))
+ continue;
+ if (!css_tryget_online(&cp->css))
+ continue;
+ rcu_read_unlock();
+ found = dl_task_in_cpuset(cp);
+ rcu_read_lock();
+ css_put(&cp->css);
+ if (found)
+ break;
+ }
+ rcu_read_unlock();
+ return found;
+}
+
/**
* cpuset_cpus_allowed_fallback - final fallback before complete catastrophe.
* @tsk: pointer to task_struct with which the scheduler is struggling
diff --git a/kernel/sched/deadline.c b/kernel/sched/deadline.c
index c33e73e920eb..eead45a0c119 100644
--- a/kernel/sched/deadline.c
+++ b/kernel/sched/deadline.c
@@ -5066,9 +5066,79 @@ static int dl_bw_manage(enum dl_bw_request req, int cpu, u64 dl_bw)
return overflow ? -EBUSY : 0;
}
+/*
+ * Return true if @p has CPUs to run on once @cpu is offline, but none in
+ * the root domain of @cpu, which counts its bandwidth. A dead task runs no
+ * more. It can wait for its 0-lag time, or for its parent to reap it.
+ * Return false for it.
+ */
+static bool dl_task_leaves_span(struct task_struct *p, int cpu,
+ const struct cpumask *span)
+{
+ struct cpumask *msk;
+
+ guard(raw_spinlock_irqsave)(&p->pi_lock);
+ if (READ_ONCE(p->__state) == TASK_DEAD || dl_task_bw_span(p) != span)
+ return false;
+ msk = this_cpu_cpumask_var_ptr(local_cpu_mask_dl);
+ /* A task in a migrate_disable() section has a cpus_ptr of one CPU. */
+ cpumask_and(msk, &p->cpus_mask, cpu_active_mask);
+ cpumask_clear_cpu(cpu, msk);
+ /* With no CPU left, ask which CPUs the cpuset gives @p. */
+ if (cpumask_empty(msk) && !cpuset_cpus_allowed_without(p, cpu, msk)) {
+ /*
+ * select_fallback_rq() picks the CPU. It gives @p the CPUs of
+ * @span if one of them is active and @p can run on all of them.
+ */
+ if (cpumask_any_and_but(span, cpu_active_mask, cpu) < nr_cpu_ids &&
+ cpumask_subset(span, task_cpu_possible_mask(p)))
+ return false;
+ }
+ return !cpumask_empty(msk) && !cpumask_intersects(msk, span);
+}
+
+/*
+ * Return true if @cpu is the last CPU that a deadline task can run on in
+ * the root domain that counts its bandwidth. With @cpu offline, the task
+ * would run in another root domain, which did not admit it. The mask of
+ * a task can hold CPUs of two root domains: with isolcpus="domain", or on
+ * cgroup v1 with sched_load_balance off.
+ *
+ * Also return true if the offline makes a cpuset partition invalid, and a
+ * deadline task then runs in another root domain.
+ *
+ * A suspend takes the CPUs offline with the tasks frozen, and the resume
+ * puts them back. Without admission control, there is no limit to exceed.
+ * Return false in both cases.
+ */
+static bool dl_cpu_last_of_task(int cpu)
+{
+ const struct cpumask *span;
+ struct task_struct *g, *p;
+
+ if (cpuhp_tasks_frozen || !dl_bandwidth_enabled())
+ return false;
+ /* No task holds a bandwidth: do not walk the task list. */
+ if (!atomic_read(&dl_nr_bw_tasks))
+ return false;
+ if (cpuset_offline_moves_dl(cpu))
+ return true;
+ guard(rcu)();
+ span = cpu_rq(cpu)->rd->span;
+ for_each_process_thread(g, p) {
+ if (task_has_dl_policy(p) && dl_task_leaves_span(p, cpu, span))
+ return true;
+ }
+ return false;
+}
+
int dl_bw_deactivate(int cpu)
{
- return dl_bw_manage(dl_bw_req_deactivate, cpu, 0);
+ int ret = dl_bw_manage(dl_bw_req_deactivate, cpu, 0);
+
+ if (!ret && dl_cpu_last_of_task(cpu))
+ ret = -EBUSY;
+ return ret;
}
int dl_bw_alloc(int cpu, u64 dl_bw)
--
2.47.3
^ permalink raw reply related [flat|nested] 35+ messages in thread
* [PATCH v8 17/33] cgroup/cpuset: Refuse to take a deadline task out of its root domain
2026-11-19 20:15 [PATCH v8 00/33] sched/deadline: bandwidth accounting fixes and selftests Mathieu Desnoyers
` (15 preceding siblings ...)
2026-11-19 20:15 ` [PATCH v8 16/33] sched/deadline: Keep the last CPU a task has in its root domain Mathieu Desnoyers
@ 2026-11-19 20:15 ` Mathieu Desnoyers
2026-11-19 20:15 ` [PATCH v8 18/33] selftests/sched: Test the root domain of a task in deadline_bw_test Mathieu Desnoyers
` (15 subsequent siblings)
32 siblings, 0 replies; 35+ messages in thread
From: Mathieu Desnoyers @ 2026-11-19 20:15 UTC (permalink / raw)
To: juri.lelli, peterz, mingo, vincent.guittot
Cc: Mathieu Desnoyers, tglx, will, boqun, dietmar.eggemann, rostedt,
bsegall, mgorman, vschneid, kprateek.nayak, luca.abeni,
mathieu.poirier, longman, ridong.chen, tj, hannes, mkoutny, shuah,
bigeasy, clrkwllms, parri.andrea, m.szyprowski, kayracizmeci,
weihengqiuu, linux-kernel, cgroups, linux-kselftest,
linux-rt-devel, stable
A cpuset attach asks the destination root domain for room before it
moves a deadline task there. A change of the CPUs of a cpuset asks
nothing. validate_change() only checks that a partition root keeps
enough capacity for its deadline bandwidth. That check comes from
commit f82f80426f7a ("sched/deadline: Ensure that updates to exclusive
cpusets don't break AC").
With isolcpus="domain", a cpuset that is not a partition root can hold
housekeeping CPUs and isolated CPUs, which are in two root domains. Two
changes can take every CPU of one root domain away from that cpuset:
- a write to cpuset.cpus, of that cpuset or of an ancestor;
- a partition root that takes those CPUs, when it is created or when
it grows.
A deadline task counted in that root domain then runs in the other one,
which did not admit it. Its bandwidth follows with no admission test,
and that root domain can exceed its limit.
Refuse both:
- A write to cpuset.cpus fails with EBUSY, and changes nothing. The
cpuset.cpus file of cgroup v1 gets the same check.
- The write that makes the partition root, or makes it grow, still
succeeds. cpuset makes the partition root invalid. For a "root"
partition, cpuset.cpus.partition then reads "root invalid (Deadline
task would leave its root domain)".
A task goes to another root domain by a move to another cpuset, which
asks for room.
A change that leaves each deadline task a CPU in its root domain is
accepted as before.
The check follows __set_cpus_allowed_ptr(). That keeps the new mask
within the affinity set by the user, if they intersect. So a mask that
grows can also take a task out of its root domain, and it is checked
too. Changes that cannot fail are not checked: a partition that is
disabled or becomes invalid, and a CPU that comes online.
dl_task_keeps_root_domain() tells whether a new mask keeps a task in
the root domain that counts it. Without admission control it accepts
any mask: there is no limit to exceed. An administrator who needs the
change can turn admission control off, by writing -1 to
/proc/sys/kernel/sched_rt_runtime_us.
The cases mask_out_refused and partition_takes_refused, added to
deadline_bw_test later in this series, test this. Without this patch the
write is accepted and the partition root is valid. These cases test the
other paths: - partition_takes_fallback_refused: the CPUs of an
ancestor; - mask_grows_refused: a mask that grows; -
partition_swap_refused: a partition root that takes and gives CPUs in
one change; - partition_refused_no_warn: a refusal with no warning.
Fixes: f82f80426f7a ("sched/deadline: Ensure that updates to exclusive cpusets don't break AC")
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Cc: stable@vger.kernel.org
---
include/linux/sched/deadline.h | 2 +
kernel/cgroup/cpuset-internal.h | 1 +
kernel/cgroup/cpuset.c | 197 +++++++++++++++++++++++++++++++-
kernel/sched/deadline.c | 25 ++++
4 files changed, 222 insertions(+), 3 deletions(-)
diff --git a/include/linux/sched/deadline.h b/include/linux/sched/deadline.h
index bdabd54af729..67426834a9a4 100644
--- a/include/linux/sched/deadline.h
+++ b/include/linux/sched/deadline.h
@@ -62,6 +62,8 @@ extern int dl_task_bw_reserve_cpu(struct task_struct *p,
const struct cpumask *new_mask);
extern int dl_task_bw_reserved_cpu(struct task_struct *p);
extern void dl_task_bw_unreserve(struct task_struct *p);
+extern bool dl_task_keeps_root_domain(struct task_struct *p,
+ const struct cpumask *new_mask);
static inline bool dl_server(struct sched_dl_entity *dl_se)
{
diff --git a/kernel/cgroup/cpuset-internal.h b/kernel/cgroup/cpuset-internal.h
index a83443dfb788..8fe0225704eb 100644
--- a/kernel/cgroup/cpuset-internal.h
+++ b/kernel/cgroup/cpuset-internal.h
@@ -36,6 +36,7 @@ enum prs_errcode {
PERR_ACCESS,
PERR_REMOTE,
PERR_ISOPARENT,
+ PERR_DLTASK,
};
/* bits in struct cpuset flags field */
diff --git a/kernel/cgroup/cpuset.c b/kernel/cgroup/cpuset.c
index 5299652d47b6..4350977601eb 100644
--- a/kernel/cgroup/cpuset.c
+++ b/kernel/cgroup/cpuset.c
@@ -61,6 +61,7 @@ static const char * const perr_strings[] = {
[PERR_ACCESS] = "Enable partition not permitted",
[PERR_REMOTE] = "Have remote partition underneath",
[PERR_ISOPARENT] = "Root partition is not allowed with isolated parent",
+ [PERR_DLTASK] = "Deadline task would leave its root domain",
};
/*
@@ -1126,6 +1127,36 @@ static void update_partition_sd_lb(struct cpuset *cs, int old_prs)
cpuset_force_rebuild();
}
+/*
+ * Return false if the CPUs of @new_cpus make a deadline task of @cp leave
+ * the root domain that counts its bandwidth. Called under rcu_read_lock(),
+ * which it drops and takes again.
+ */
+static bool dl_tasks_keep_cpus(struct cpuset *cp, const struct cpumask *new_cpus)
+{
+ struct css_task_iter it;
+ struct task_struct *task;
+ bool keep = true;
+
+ if (!css_tryget_online(&cp->css))
+ return true;
+ rcu_read_unlock();
+
+ css_task_iter_start(&cp->css, 0, &it);
+ while (keep && (task = css_task_iter_next(&it))) {
+ /* The top cpuset leaves the mask of these tasks alone. */
+ if (cp == &top_cpuset &&
+ (task->flags & (PF_KTHREAD | PF_NO_SETAFFINITY)))
+ continue;
+ keep = dl_task_keeps_root_domain(task, new_cpus);
+ }
+ css_task_iter_end(&it);
+
+ rcu_read_lock();
+ css_put(&cp->css);
+ return keep;
+}
+
/*
* Set @new_cpus to the effective CPUs that @cp gets. On entry, @new_cpus
* holds those that @from gets. @from is @cp or one of its ancestors. Work
@@ -1148,6 +1179,78 @@ static void effective_cpus_below(struct cpuset *cp, struct cpuset *from,
}
}
+/*
+ * A scratch mask for dl_tasks_keep_without(). Its callers still need each
+ * mask of their struct tmpmasks after the call. Protected by
+ * cpuset_mutex.
+ */
+static cpumask_var_t dl_keep_cpus;
+
+/*
+ * Return false if a partition that takes the CPUs of @xcpus makes a
+ * deadline task leave the root domain that counts its bandwidth. A cpuset
+ * can hold CPUs of two root domains: with isolcpus="domain", or on cgroup
+ * v1 with sched_load_balance off. Such a task would run in a root domain
+ * that did not admit it.
+ *
+ * @root is the partition root that gives the CPUs. @part is the cpuset
+ * that takes them. In the same change, @part can give the CPUs of @gives
+ * back to @root. @gives is NULL if it gives none back.
+ *
+ * Work out the new CPUs of each cpuset of @root from the top down, as
+ * update_cpumasks_hier() does, down to the next partition roots. If
+ * @enable, @part becomes a partition root. Then do the same for its own
+ * cpusets, starting from the active CPUs of @xcpus.
+ */
+static bool dl_tasks_keep_without(struct cpuset *root, struct cpuset *part,
+ const struct cpumask *xcpus,
+ const struct cpumask *gives, bool enable)
+{
+ struct cpumask *new_cpus = dl_keep_cpus;
+ struct cgroup_subsys_state *pos_css;
+ struct cpuset *cp;
+ bool keep = true;
+
+ lockdep_assert_cpuset_lock_held();
+ rcu_read_lock();
+ cpuset_for_each_descendant_pre(cp, pos_css, root) {
+ if (cp == part || (cp != root && is_partition_valid(cp))) {
+ pos_css = css_rightmost_descendant(pos_css);
+ continue;
+ }
+ /* The CPUs that @root has after the change. */
+ cpumask_copy(new_cpus, root->effective_cpus);
+ if (gives) {
+ cpumask_or(new_cpus, new_cpus, gives);
+ cpumask_and(new_cpus, new_cpus, cpu_active_mask);
+ }
+ cpumask_andnot(new_cpus, new_cpus, xcpus);
+ effective_cpus_below(cp, root, new_cpus);
+ if (cpumask_equal(cp->effective_cpus, new_cpus))
+ continue;
+ keep = dl_tasks_keep_cpus(cp, new_cpus);
+ if (!keep)
+ break;
+ }
+ if (keep && enable) {
+ cpuset_for_each_descendant_pre(cp, pos_css, part) {
+ if (cp != part && is_partition_valid(cp)) {
+ pos_css = css_rightmost_descendant(pos_css);
+ continue;
+ }
+ cpumask_and(new_cpus, xcpus, cpu_active_mask);
+ effective_cpus_below(cp, part, new_cpus);
+ if (cpumask_equal(cp->effective_cpus, new_cpus))
+ continue;
+ keep = dl_tasks_keep_cpus(cp, new_cpus);
+ if (!keep)
+ break;
+ }
+ }
+ rcu_read_unlock();
+ return keep;
+}
+
/*
* tasks_nocpu_error - Return true if tasks will have no effective_cpus
*/
@@ -1698,6 +1801,8 @@ static int remote_partition_enable(struct cpuset *cs, int new_prs,
return PERR_NOCPUS;
if (prstate_housekeeping_conflict(new_prs, PRS_ROOT, tmp->new_cpus, NULL))
return PERR_HKEEPING;
+ if (!dl_tasks_keep_without(&top_cpuset, cs, tmp->new_cpus, NULL, true))
+ return PERR_DLTASK;
spin_lock_irq(&callback_lock);
partition_xcpus_add(new_prs, NULL, tmp->new_cpus);
@@ -1801,6 +1906,10 @@ static void remote_cpus_update(struct cpuset *cs, struct cpumask *xcpus,
else if (prstate_housekeeping_conflict(prs, PRS_ROOT,
excpus, tmp->delmask))
WRITE_ONCE(cs->prs_err, PERR_HKEEPING);
+ else if (!dl_tasks_keep_without(&top_cpuset, cs, tmp->addmask,
+ deleting ? tmp->delmask : NULL,
+ false))
+ WRITE_ONCE(cs->prs_err, PERR_DLTASK);
if (cs->prs_err)
goto invalidate;
}
@@ -1948,6 +2057,9 @@ static int update_parent_effective_cpumask(struct cpuset *cs, int cmd,
if (nocpu)
return PERR_NOCPUS;
+ if (!dl_tasks_keep_without(parent, cs, tmp->delmask, NULL, true))
+ return PERR_DLTASK;
+
/*
* This function will only be called when all the preliminary
* checks have passed. At this point, the following condition
@@ -2035,6 +2147,20 @@ static int update_parent_effective_cpumask(struct cpuset *cs, int cmd,
deleting = false;
adding = cpumask_and(tmp->addmask,
cs->effective_xcpus, parent->effective_xcpus);
+ } else if (deleting &&
+ !dl_tasks_keep_without(parent, cs, tmp->delmask,
+ adding ? tmp->addmask : NULL,
+ is_partition_invalid(cs))) {
+ /*
+ * As above: the partition is or becomes invalid. Only
+ * a valid one holds CPUs of its parent to give back.
+ */
+ part_error = PERR_DLTASK;
+ deleting = false;
+ if (is_partition_valid(cs))
+ adding = cpumask_and(tmp->addmask,
+ cs->effective_xcpus,
+ parent->effective_xcpus);
}
} else {
/*
@@ -2106,10 +2232,13 @@ static int update_parent_effective_cpumask(struct cpuset *cs, int cmd,
}
rcu_read_unlock();
}
- if (exclusive)
- deleting = !cpumask_empty(tmp->delmask);
- else
+ if (!exclusive)
part_error = PERR_NOTEXCL;
+ else if (!dl_tasks_keep_without(parent, cs, tmp->delmask,
+ NULL, true))
+ part_error = PERR_DLTASK;
+ else
+ deleting = !cpumask_empty(tmp->delmask);
}
/*
* When both cs and parent remain valid, parent's
@@ -2518,6 +2647,62 @@ static void partition_cpus_change(struct cpuset *cs, struct cpuset *trialcs,
}
}
+/*
+ * Set @new_cpus to the effective CPUs that @cp gets if @cs takes the CPUs
+ * of @trialcs. @cp is @cs or one of its descendants.
+ */
+static void trial_effective_cpus(struct cpuset *cp, struct cpuset *cs,
+ struct cpuset *trialcs, struct cpumask *new_cpus)
+{
+ struct cpuset *parent = parent_cs(cs);
+
+ if (!cpumask_and(new_cpus, trialcs->cpus_allowed, parent->effective_cpus) &&
+ is_in_v2_mode())
+ cpumask_copy(new_cpus, parent->effective_cpus);
+ effective_cpus_below(cp, cs, new_cpus);
+}
+
+/*
+ * Return false if the CPUs of @trialcs make a deadline task of @cs, or of
+ * a descendant, leave the root domain that counts its bandwidth. A cpuset
+ * can hold CPUs of two root domains: with isolcpus="domain", or on cgroup
+ * v1 with sched_load_balance off. The task would run in a root domain that
+ * did not admit it. A move of the task to another cpuset is the way there:
+ * it asks the other root domain for room.
+ *
+ * A partition root takes its CPUs from its parent, which checks the same
+ * for its own tasks: see dl_tasks_keep_without().
+ */
+static bool dl_tasks_keep_root_domain(struct cpuset *cs, struct cpuset *trialcs,
+ struct cpumask *new_cpus)
+{
+ struct cgroup_subsys_state *pos_css;
+ struct cpuset *cp;
+ bool keep = true;
+
+ if (is_partition_valid(cs))
+ return true;
+ rcu_read_lock();
+ cpuset_for_each_descendant_pre(cp, pos_css, cs) {
+ if (is_partition_valid(cp)) {
+ pos_css = css_rightmost_descendant(pos_css);
+ continue;
+ }
+ trial_effective_cpus(cp, cs, trialcs, new_cpus);
+ /*
+ * Test a mask that grows too. A task whose user affinity
+ * holds none of the old CPUs gets only the new ones.
+ */
+ if (cpumask_equal(cp->effective_cpus, new_cpus))
+ continue;
+ keep = dl_tasks_keep_cpus(cp, new_cpus);
+ if (!keep)
+ break;
+ }
+ rcu_read_unlock();
+ return keep;
+}
+
/**
* update_cpumask - update the cpus_allowed mask of a cpuset and all tasks in it
* @cs: the cpuset to consider
@@ -2550,6 +2735,11 @@ static int update_cpumask(struct cpuset *cs, struct cpuset *trialcs,
if (alloc_tmpmasks(&tmp))
return -ENOMEM;
+ if (!dl_tasks_keep_root_domain(cs, trialcs, tmp.new_cpus)) {
+ free_tmpmasks(&tmp);
+ return -EBUSY;
+ }
+
/*
* Check all the descendants in update_cpumasks_hier() if
* effective_xcpus is to be changed.
@@ -3983,6 +4173,7 @@ int __init cpuset_init(void)
BUG_ON(!alloc_cpumask_var(&cpus_attach, GFP_KERNEL));
BUG_ON(!zalloc_cpumask_var(&dl_attach_cpus, GFP_KERNEL));
+ BUG_ON(!alloc_cpumask_var(&dl_keep_cpus, GFP_KERNEL));
if (housekeeping_enabled(HK_TYPE_DOMAIN_BOOT))
cpumask_andnot(isolated_cpus, cpu_possible_mask,
diff --git a/kernel/sched/deadline.c b/kernel/sched/deadline.c
index eead45a0c119..641674f65931 100644
--- a/kernel/sched/deadline.c
+++ b/kernel/sched/deadline.c
@@ -5141,6 +5141,31 @@ int dl_bw_deactivate(int cpu)
return ret;
}
+/*
+ * Return false if @p has CPUs in the root domain that counts its
+ * bandwidth and @new_mask holds none of them. A cpuset refuses such a
+ * mask: only a move to another cpuset makes room in another root domain.
+ * Like __set_cpus_allowed_ptr(), keep @new_mask within the affinity set by
+ * the user, if they intersect.
+ */
+bool dl_task_keeps_root_domain(struct task_struct *p,
+ const struct cpumask *new_mask)
+{
+ const struct cpumask *span;
+
+ if (!dl_bandwidth_enabled() || !task_has_dl_policy(p))
+ return true;
+ /* pi_lock also protects p->user_cpus_ptr. */
+ guard(raw_spinlock_irqsave)(&p->pi_lock);
+ guard(rcu)();
+ span = dl_task_bw_span(p);
+ if (!span || !cpumask_intersects(span, p->cpus_ptr))
+ return true;
+ if (p->user_cpus_ptr && cpumask_intersects(new_mask, p->user_cpus_ptr))
+ return cpumask_intersects_and(span, new_mask, p->user_cpus_ptr);
+ return cpumask_intersects(span, new_mask);
+}
+
int dl_bw_alloc(int cpu, u64 dl_bw)
{
return dl_bw_manage(dl_bw_req_alloc, cpu, dl_bw);
--
2.47.3
^ permalink raw reply related [flat|nested] 35+ messages in thread
* [PATCH v8 18/33] selftests/sched: Test the root domain of a task in deadline_bw_test
2026-11-19 20:15 [PATCH v8 00/33] sched/deadline: bandwidth accounting fixes and selftests Mathieu Desnoyers
` (16 preceding siblings ...)
2026-11-19 20:15 ` [PATCH v8 17/33] cgroup/cpuset: Refuse to take a deadline task out of " Mathieu Desnoyers
@ 2026-11-19 20:15 ` Mathieu Desnoyers
2026-11-19 20:15 ` [PATCH v8 19/33] sched/deadline: Count only active CPUs when spreading extra_bw Mathieu Desnoyers
` (14 subsequent siblings)
32 siblings, 0 replies; 35+ messages in thread
From: Mathieu Desnoyers @ 2026-11-19 20:15 UTC (permalink / raw)
To: juri.lelli, peterz, mingo, vincent.guittot
Cc: Mathieu Desnoyers, tglx, will, boqun, dietmar.eggemann, rostedt,
bsegall, mgorman, vschneid, kprateek.nayak, luca.abeni,
mathieu.poirier, longman, ridong.chen, tj, hannes, mkoutny, shuah,
bigeasy, clrkwllms, parri.andrea, m.szyprowski, kayracizmeci,
weihengqiuu, linux-kernel, cgroups, linux-kselftest,
linux-rt-devel
Earlier in this series, fixes keep a deadline task in the root domain
that counts its bandwidth. Add the 24 cases of deadline_bw_test that
test them. They cover CPU hotplug and cpuset changes with partition
roots, and with CPUs isolated by isolcpus=domain:
- the CPU a task falls back to when its CPU goes offline
- a task that runs, sleeps or wakes while its CPU goes offline
- the refusals of a CPU offline or a cpuset change that would leave a
deadline task no CPU in its root domain
- a rebuild that counts a task in the root domain it runs in
- a CPU offline that would take more bandwidth than the root domain has
free, with the CPU's fair server (hotplug_server_refused)
The comment above each case gives its scenario. Twenty of these cases
need CPUs isolated by isolcpus=domain, and skip without them.
The message of each fix names the cases that test it.
hotplug_server_refused is not named there. It fails with
isolcpus=domain,0,1 before these fixes.
On v7.4-rc2, in 8 VMs of 8 CPUs, hotplug_partition_invalid_refused fails
in every VM. hotplug_child_partition_kept fails in 2 of 8. The cases
that need isolcpus=domain skip there. rebuild_record_taken fails with the
first series and isolcpus=domain,0,1 on 7 CPUs.
With the fixes up to here, every case passes wherever it runs.
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
---
.../sched/deadline/deadline_bw_test.c | 1940 ++++++++++++++++-
1 file changed, 1937 insertions(+), 3 deletions(-)
diff --git a/tools/testing/selftests/sched/deadline/deadline_bw_test.c b/tools/testing/selftests/sched/deadline/deadline_bw_test.c
index fa1ff62d6f42..a290e332831d 100644
--- a/tools/testing/selftests/sched/deadline/deadline_bw_test.c
+++ b/tools/testing/selftests/sched/deadline/deadline_bw_test.c
@@ -35,8 +35,12 @@
* accounting, which clears what an earlier case left.
*
* The cases of hp_cases[] run after them, without P. These are the
- * hotplug cases, and affinity_vs_setattr. The test skips the hotplug
- * cases if no CPU can go offline. It still runs affinity_vs_setattr.
+ * hotplug cases, and nine other cases: mask_out_refused,
+ * partition_takes_refused, no_admission_affinity_moves,
+ * affinity_vs_setattr, rebuild_waking_keeps,
+ * partition_takes_fallback_refused, mask_grows_refused,
+ * partition_swap_refused and partition_refused_no_warn. The test skips
+ * the hotplug cases if no CPU can go offline. It still runs these nine.
* death_zombie_hotplug takes CPUs offline too, but it is in p_cases[]: it
* needs the cpuset setup of those cases.
*
@@ -45,7 +49,7 @@
*
* The test needs root, deadline admission control and 3 housekeeping
* CPUs. Some cases need 4. Most also need the cgroup v2 cpuset
- * controller. Some need debugfs or CPU hotplug. Five need
+ * controller. Some need debugfs or CPU hotplug. Twenty-five need
* isolcpus="domain" at boot.
*/
#define _GNU_SOURCE
@@ -6728,6 +6732,73 @@ static int hotplug_server(void)
return KSFT_PASS;
}
+/*
+ * A CPU that goes offline takes its capacity and the bandwidth of its
+ * fair server out of the root domain. Set the server of hp_cpu to the
+ * admission limit less 10%. The offline then costs the root domain the
+ * room of one probe task. Hold every probe task that the root domain
+ * admits but one: the CPU can still go offline. Hold one more: the
+ * offline must fail with EBUSY.
+ */
+static int hotplug_server_refused(void)
+{
+ uint64_t rt_runtime, rt_period, runtime;
+ int n, err, ret = KSFT_PASS;
+ char buf[32];
+
+ if (ext_switched_all()) {
+ ksft_print_msg("a hotplug can end the sched_ext scheduler\n");
+ return KSFT_SKIP;
+ }
+ if (read_file("/proc/sys/kernel/sched_rt_runtime_us", buf, sizeof(buf)) ||
+ read_file("/proc/sys/kernel/sched_rt_period_us", buf, sizeof(buf))) {
+ ksft_print_msg("no sched_rt_runtime_us or sched_rt_period_us\n");
+ return KSFT_SKIP;
+ }
+ read_file("/proc/sys/kernel/sched_rt_runtime_us", buf, sizeof(buf));
+ rt_runtime = atoll(buf);
+ read_file("/proc/sys/kernel/sched_rt_period_us", buf, sizeof(buf));
+ rt_period = atoll(buf);
+ if (!rt_period || 5 * rt_runtime < rt_period) {
+ ksft_print_msg("the admission limit is below a fifth of a CPU\n");
+ return KSFT_SKIP;
+ }
+ runtime = SERVER_PERIOD * rt_runtime / rt_period - SERVER_PERIOD / 10;
+ if (server_set(hp_cpu, runtime, SERVER_PERIOD)) {
+ ksft_print_msg("the fair server of CPU %d cannot be set\n", hp_cpu);
+ return KSFT_SKIP;
+ }
+ n = probe(NULL);
+ if (n < 2) {
+ ksft_print_msg("no room for two probe tasks\n");
+ return KSFT_SKIP;
+ }
+ if (hold(n - 1)) {
+ release_held();
+ return KSFT_FAIL;
+ }
+ err = cpu_online(hp_cpu, 0);
+ if (err) {
+ ksft_print_msg("CPU %d, room for one probe task left: offline refused (%s)\n",
+ hp_cpu, strerror(-err));
+ ret = KSFT_FAIL;
+ } else if (cpu_online(hp_cpu, 1) || hold(1)) {
+ cpu_up(hp_cpu);
+ ret = KSFT_FAIL;
+ } else {
+ err = cpu_online(hp_cpu, 0);
+ if (err != -EBUSY) {
+ ksft_print_msg("CPU %d, no room left: offline %s, not EBUSY\n",
+ hp_cpu, err ? strerror(-err) : "accepted");
+ if (!err)
+ cpu_up(hp_cpu);
+ ret = KSFT_FAIL;
+ }
+ }
+ release_held();
+ return ret;
+}
+
/* Return whether @cpu is in @list, a CPU list as sysfs prints it. */
static int in_cpu_list(const char *list, int cpu)
{
@@ -6767,6 +6838,483 @@ static int fallback_wake(int fd, pid_t pid)
return 0;
}
+/*
+ * The CPUs of fallback_case() and no_admission_affinity_moves(). I1 and I2
+ * are isolated by isolcpus="domain". H and K are housekeeping CPUs: CPUs
+ * that isolcpus does not isolate. H is above I2. K is another one.
+ *
+ * H is above I2 because of select_fallback_rq(). It gives a task that
+ * must leave its CPU the first CPU of its mask. A task allowed on I2 and
+ * H then goes to I2, unless the kernel keeps it in its root domain.
+ *
+ * fb_i1_hp says whether I1 can go offline.
+ */
+static int fb_i1 = -1, fb_i2 = -1, fb_h = -1, fb_k = -1, fb_i1_hp;
+
+/*
+ * Read total_bw of the root domain of the isolated CPUs into @def, and of
+ * the housekeeping root domain into @hk. Return 0, or -1.
+ */
+static int fallback_totals(long long *def, long long *hk)
+{
+ *def = total_bw_of(fb_i2);
+ *hk = total_bw_of(fb_h);
+ if (*def == -1 || *hk == -1) {
+ ksft_print_msg("no total_bw in /sys/kernel/debug/sched/debug\n");
+ return -1;
+ }
+ return 0;
+}
+
+/*
+ * Find I1, I2, H and K. Take a hotpluggable CPU as I1 if there is one.
+ * With @offline, I1 must be hotpluggable. Return 0, or -1 if the CPUs do
+ * not allow the case. fb_k is set only once all four are found.
+ */
+static int fallback_cpus(int offline)
+{
+ int i;
+
+ if (fb_k >= 0)
+ return offline && !fb_i1_hp ? -1 : 0;
+ /* A failed scan can leave I1, I2 or H set. Scan from scratch. */
+ fb_i1 = -1;
+ fb_i2 = -1;
+ fb_h = -1;
+ for (i = 0; i < nisol; i++) {
+ if (fb_i1 < 0 && cpu_hotpluggable(isol[i]))
+ fb_i1 = isol[i];
+ else if (fb_i2 < 0)
+ fb_i2 = isol[i];
+ }
+ fb_i1_hp = fb_i1 >= 0;
+ if (fb_i1 < 0 && nisol >= 2) {
+ fb_i1 = isol[0];
+ fb_i2 = isol[1];
+ }
+ for (i = 0; fb_i2 >= 0 && i < ncpus && fb_h < 0; i++) {
+ if (online[i] > fb_i2)
+ fb_h = online[i];
+ }
+ for (i = 0; fb_h >= 0 && i < ncpus && fb_k < 0; i++) {
+ if (online[i] != fb_h)
+ fb_k = online[i];
+ }
+ if (fb_i1 >= 0 && fb_i2 >= 0 && fb_k >= 0)
+ return offline && !fb_i1_hp ? -1 : 0;
+ fb_k = -1;
+ return -1;
+}
+
+enum fb_mode { FB_RUN_I1, FB_ON_I1, FB_ON_K, FB_ON_H, FB_MASK_H, FB_LAST_H, FB_PART_H, FB_STAY_K };
+
+/*
+ * B is a cpuset of I1, I2 and H. C is a cpuset of I1, in B. Neither is a
+ * partition. A deadline task of 10% is allowed on all CPUs. It sleeps on
+ * K or on H, and the housekeeping root domain counts it. The case moves
+ * it, asleep, into C or into B.
+ *
+ * In C, the task has I1 only. The move asks the root domain of the
+ * isolated CPUs for room, and that root domain counts the task. In B, the
+ * task still has H, a CPU of the housekeeping root domain. That root
+ * domain goes on counting it.
+ *
+ * The case then does one thing, and wakes the task. The table gives the
+ * cpuset the task is in, the CPU it sleeps on, what the case does, and
+ * the CPU the task must wake up on:
+ *
+ * case in asleep on then wakes on
+ * hotplug_fallback C I1 I1 goes offline I2
+ * hotplug_fallback_running C I1, runs I1 goes offline I2
+ * hotplug_fallback_moved C K I1 goes offline I2
+ * hotplug_fallback_in_mask C H I1 goes offline I2
+ * mask_out_refused C H C is given H only: I1
+ * EBUSY
+ * hotplug_last_refused B H H offline: EBUSY H
+ * hotplug_keeps_root_domain B H H goes offline K
+ * partition_takes_refused B H Q takes H: invalid H
+ *
+ * hotplug_fallback wakes the task once after the move, so that it sleeps
+ * on I1. In hotplug_fallback_running, the task then spins on I1 with a
+ * period of 2ms. It is moved off I1 before the cpuset update of the
+ * offline, by select_fallback_rq(). The case checks where it runs after
+ * the offline. hotplug_keeps_root_domain gives B the CPU K as well. In
+ * partition_takes_refused, Q is a cpuset of H next to B. The case makes
+ * it a partition root, which must be invalid.
+ *
+ * With I1 offline, C has no CPU left, and the task gets the CPUs of B:
+ * I2 and H. It must wake up on I2, in the root domain that counts it. On
+ * H, it would run in a root domain that did not admit it. The three
+ * refusals have one reason: each change would leave the task no CPU in
+ * the root domain that counts it.
+ *
+ * In every case, the root domain of the CPU the task wakes on must count
+ * it, and the other one must not. The case checks the root domain of that
+ * CPU, not its number. The reference values of total_bw are taken with no
+ * task: with the CPU of the case offline if the case takes one offline,
+ * else with every CPU online. Once the task is gone, total_bw must be
+ * back to them. The messages give total_bw of the root domain of the
+ * isolated CPUs first, then of the housekeeping one.
+ */
+static int fallback_case(enum fb_mode mode)
+{
+ int in_b = mode == FB_LAST_H || mode == FB_PART_H || mode == FB_STAY_K;
+ int off_cpu;
+ int on_h = mode == FB_ON_H || mode == FB_MASK_H || in_b;
+ int no_offline = mode == FB_MASK_H || mode == FB_LAST_H || mode == FB_PART_H;
+ char cg_q[PATH_MAX];
+ int running = mode == FB_RUN_I1;
+ int on_i1 = mode == FB_ON_I1 || running;
+ char cg_b[PATH_MAX], cg_c[PATH_MAX], path[PATH_MAX + 32], buf[256], one[16];
+ long long def0, hk0, def_off = 0, hk_off = 0, def_ref, hk_ref, def, hk, bw;
+ int fd[2], ret = KSFT_SKIP, added = 0, made = 0, w, at, err;
+ pid_t pid = -1;
+
+ /* hotplug_keeps_root_domain takes H offline, not I1. */
+ if (!root[0] || fallback_cpus(!no_offline && mode != FB_STAY_K)) {
+ ksft_print_msg("needs isolcpus=domain: 2 isolated, 2 housekeeping CPUs%s\n",
+ no_offline ? "" : ", CPU hotplug");
+ return KSFT_SKIP;
+ }
+ off_cpu = mode == FB_STAY_K ? fb_h : fb_i1;
+ if ((mode == FB_LAST_H || mode == FB_STAY_K) && !cpu_hotpluggable(fb_h)) {
+ ksft_print_msg("CPU %d cannot go offline\n", fb_h);
+ return KSFT_SKIP;
+ }
+ snprintf(path, sizeof(path), "%s/cgroup.subtree_control", root);
+ if (read_file(path, buf, sizeof(buf)))
+ return KSFT_SKIP;
+ if (!strstr(buf, "cpuset")) {
+ if (write_file(root, "cgroup.subtree_control", "+cpuset"))
+ return KSFT_SKIP;
+ added = 1;
+ }
+ if (snprintf(cg_q, sizeof(cg_q), "%s/deadline_bw_q", root) >= (int)sizeof(cg_q) ||
+ snprintf(cg_b, sizeof(cg_b), "%s/deadline_bw_b", root) >= (int)sizeof(cg_b) ||
+ snprintf(cg_c, sizeof(cg_c), "%s/c", cg_b) >= (int)sizeof(cg_c))
+ goto out;
+ if (mkdir(cg_b, 0755) && errno != EEXIST) {
+ ksft_print_msg("mkdir %s: %s\n", cg_b, strerror(errno));
+ goto out;
+ }
+ made = 1;
+ if (mode == FB_STAY_K)
+ snprintf(buf, sizeof(buf), "%d,%d,%d,%d", fb_i1, fb_i2, fb_h, fb_k);
+ else
+ snprintf(buf, sizeof(buf), "%d,%d,%d", fb_i1, fb_i2, fb_h);
+ snprintf(one, sizeof(one), "%d", fb_i1);
+ if (write_file(cg_b, "cpuset.cpus", buf) ||
+ write_file(cg_b, "cgroup.subtree_control", "+cpuset") ||
+ (mkdir(cg_c, 0755) && errno != EEXIST) ||
+ write_file(cg_c, "cpuset.cpus", one)) {
+ ksft_print_msg("cpusets B and C cannot be made\n");
+ goto out;
+ }
+ if (fallback_totals(&def0, &hk0))
+ goto out;
+ ret = KSFT_FAIL;
+ if (!no_offline) {
+ if (cpu_online(off_cpu, 0)) {
+ ksft_print_msg("offline of CPU %d refused\n", off_cpu);
+ goto out;
+ }
+ /* H may be the offline CPU: read the housekeeping total_bw on K. */
+ def_off = total_bw_of(fb_i2);
+ hk_off = total_bw_of(fb_k);
+ if (def_off == -1 || hk_off == -1)
+ ret = KSFT_SKIP;
+ cpu_up(off_cpu);
+ if (ret == KSFT_SKIP)
+ goto out;
+ }
+
+ if (pipe(fd)) {
+ ksft_print_msg("pipe: %s\n", strerror(errno));
+ goto out;
+ }
+ mv_state[0] = 0;
+ mv_state[1] = 0;
+ pid = fork_child("a deadline task");
+ if (!pid) {
+ cpu_set_t set;
+ int i, err;
+ char c;
+
+ /* Run on K or H, then allow every CPU again, and stay there. */
+ close(fd[1]);
+ CPU_ZERO(&set);
+ CPU_SET(on_h ? fb_h : fb_k, &set);
+ err = sched_setaffinity(0, sizeof(set), &set);
+ for (i = 0; i < CPU_SETSIZE; i++)
+ CPU_SET(i, &set);
+ if (err || sched_setaffinity(0, sizeof(set), &set) ||
+ (running ? set_deadline(0, 1 * NSEC_PER_MSEC, 2 * NSEC_PER_MSEC) :
+ set_deadline(0, 10 * NSEC_PER_MSEC, PERIOD))) {
+ mv_state[0] = -1;
+ _exit(1);
+ }
+ mv_state[0] = 1;
+ /* hotplug_fallback_running: spin from the first wakeup on. */
+ if (running) {
+ if (read(fd[0], &c, 1) == 1)
+ mv_state[1]++;
+ for (;;)
+ ;
+ }
+ while (read(fd[0], &c, 1) == 1)
+ mv_state[1]++;
+ _exit(0);
+ }
+ close(fd[0]);
+ if (pid < 0 || child_up("hotplug_fallback", 0))
+ goto out_fd;
+ if (wait_for(1000, proc_state_of(pid) == 'S') || fallback_totals(&def, &hk))
+ goto out_fd;
+ bw = def + hk - def0 - hk0;
+ if (bw <= 0) {
+ ksft_print_msg("the deadline task adds %lld to total_bw\n", bw);
+ goto out_fd;
+ }
+
+ /*
+ * Move it asleep into C, or into B. For hotplug_fallback, wake it
+ * once so that it runs on I1.
+ */
+ if (enter(in_b ? cg_b : cg_c, pid) ||
+ (on_i1 && !running && fallback_wake(fd[1], pid)))
+ goto out_fd;
+ if (running) {
+ int n = mv_state[1];
+
+ if (write(fd[1], "w", 1) != 1 || wait_for(1000, mv_state[1] != n) ||
+ wait_for(1000, task_cpu_of(pid) == fb_i1)) {
+ ksft_print_msg("the deadline task does not run on CPU %d\n", fb_i1);
+ goto out_fd;
+ }
+ }
+ at = task_cpu_of(pid);
+ if (on_i1 ? at != fb_i1 : on_h ? at != fb_h : at == fb_i1) {
+ ksft_print_msg("the deadline task sleeps on CPU %d\n", at);
+ ret = on_h ? KSFT_SKIP : KSFT_FAIL;
+ goto out_fd;
+ }
+ /* With no CPU offline, the reference is total_bw with no task. */
+ def_ref = no_offline ? def0 : def_off;
+ hk_ref = no_offline ? hk0 : hk_off;
+ if (mode == FB_MASK_H) {
+ snprintf(one, sizeof(one), "%d", fb_h);
+ err = write_file(cg_c, "cpuset.cpus", one);
+ if (err != -EBUSY) {
+ ksft_print_msg("C given CPU %d only: %s, not EBUSY\n", fb_h,
+ err ? strerror(-err) : "accepted");
+ goto out_fd;
+ }
+ } else if (mode == FB_LAST_H) {
+ err = cpu_online(fb_h, 0);
+ if (err != -EBUSY) {
+ ksft_print_msg("offline of CPU %d: %s, not EBUSY\n", fb_h,
+ err ? strerror(-err) : "accepted");
+ cpu_up(fb_h);
+ goto out_fd;
+ }
+ } else if (mode == FB_PART_H) {
+ snprintf(one, sizeof(one), "%d", fb_h);
+ if ((mkdir(cg_q, 0755) && errno != EEXIST) ||
+ write_file(cg_q, "cpuset.cpus", one) ||
+ write_file(cg_q, "cpuset.cpus.partition", "root")) {
+ ksft_print_msg("cpuset Q cannot be made\n");
+ goto out_fd;
+ }
+ snprintf(path, sizeof(path), "%s/cpuset.cpus.partition", cg_q);
+ if (read_file(path, buf, sizeof(buf)) || !strstr(buf, "Deadline task")) {
+ ksft_print_msg("Q takes CPU %d from B: %s", fb_h, buf);
+ goto out_fd;
+ }
+ } else if (cpu_online(off_cpu, 0)) {
+ ksft_print_msg("offline of CPU %d with the task asleep refused\n", off_cpu);
+ goto out_fd;
+ }
+ if (running)
+ sleep_ns(PERIOD);
+ else if (fallback_wake(fd[1], pid))
+ goto out_up;
+ def = total_bw_of(fb_i2);
+ hk = total_bw_of(fb_k);
+ if (def == -1 || hk == -1)
+ goto out_up;
+ w = task_cpu_of(pid);
+ ret = KSFT_PASS;
+ if (in_cpu_list(isolated, w) ? def != def_ref + bw || hk != hk_ref :
+ hk != hk_ref + bw || def != def_ref) {
+ ksft_print_msg("woken on CPU %d: total_bw %lld, %lld, not %lld, %lld\n",
+ w, def, hk,
+ in_cpu_list(isolated, w) ? def_ref + bw : def_ref,
+ in_cpu_list(isolated, w) ? hk_ref : hk_ref + bw);
+ ret = KSFT_FAIL;
+ } else if (in_cpu_list(isolated, w) == in_b) {
+ ksft_print_msg("woken on CPU %d: out of the root domain that counted it\n",
+ w);
+ ret = KSFT_FAIL;
+ }
+ reap(pid);
+ pid = -1;
+ sleep_ns(3 * PERIOD);
+ def = total_bw_of(fb_i2);
+ hk = total_bw_of(fb_k);
+ if (def != def_ref || hk != hk_ref) {
+ ksft_print_msg("task gone: total_bw %lld, %lld, not %lld, %lld\n",
+ def, hk, def_ref, hk_ref);
+ ret = KSFT_FAIL;
+ }
+out_up:
+ reap(pid);
+ if (!no_offline)
+ cpu_up(off_cpu);
+ pid = -1;
+out_fd:
+ reap(pid);
+ close(fd[1]);
+ sleep_ns(3 * PERIOD);
+ if (ret == KSFT_PASS && (fallback_totals(&def, &hk) || def != def0 || hk != hk0)) {
+ ksft_print_msg("task gone, CPU %d online: total_bw %lld, %lld, not %lld, %lld\n",
+ off_cpu, def, hk, def0, hk0);
+ ret = KSFT_FAIL;
+ }
+out:
+ if (made) {
+ if (rmdir(cg_q) && errno != ENOENT)
+ ksft_print_msg("rmdir %s: %s\n", cg_q, strerror(errno));
+ if (wait_for(1000, !rmdir(cg_c) || errno == ENOENT) ||
+ wait_for(1000, !rmdir(cg_b) || errno == ENOENT))
+ ksft_print_msg("rmdir %s: %s\n", cg_b, strerror(errno));
+ }
+ if (added && write_file(root, "cgroup.subtree_control", "-cpuset"))
+ ksft_print_msg("the cpuset controller stays enabled\n");
+ return ret;
+}
+
+static int hotplug_fallback_running(void)
+{
+ return fallback_case(FB_RUN_I1);
+}
+
+static int hotplug_fallback(void)
+{
+ return fallback_case(FB_ON_I1);
+}
+
+static int hotplug_fallback_moved(void)
+{
+ return fallback_case(FB_ON_K);
+}
+
+static int hotplug_fallback_in_mask(void)
+{
+ return fallback_case(FB_ON_H);
+}
+
+static int mask_out_refused(void)
+{
+ return fallback_case(FB_MASK_H);
+}
+
+static int hotplug_last_refused(void)
+{
+ return fallback_case(FB_LAST_H);
+}
+
+static int hotplug_keeps_root_domain(void)
+{
+ return fallback_case(FB_STAY_K);
+}
+
+static int partition_takes_refused(void)
+{
+ return fallback_case(FB_PART_H);
+}
+
+/*
+ * Turn admission control off. A task pins itself to H, becomes a deadline
+ * task of 10% and spins. The housekeeping root domain counts it. The case
+ * then sets its affinity to I2 with sched_setaffinity(). Nothing reserves
+ * room for it: its bandwidth must follow it to the root domain of the
+ * isolated CPUs. Once the task is gone and admission control is on again,
+ * total_bw must be what it was.
+ */
+static int no_admission_affinity_moves(void)
+{
+ long long def0, hk0, def, hk, bw;
+ int ret = KSFT_FAIL, at;
+ char saved[64];
+ cpu_set_t set;
+ pid_t pid;
+
+ if (fallback_cpus(0)) {
+ ksft_print_msg("needs isolcpus=domain: 2 isolated, 2 housekeeping CPUs\n");
+ return KSFT_SKIP;
+ }
+ if (fallback_totals(&def0, &hk0))
+ return KSFT_SKIP;
+ if (read_file("/proc/sys/kernel/sched_rt_runtime_us", saved, sizeof(saved)) ||
+ write_file("/proc/sys/kernel", "sched_rt_runtime_us", "-1")) {
+ ksft_print_msg("admission control cannot be turned off\n");
+ return KSFT_SKIP;
+ }
+ mv_state[0] = 0;
+ pid = fork_child("a deadline task");
+ if (!pid) {
+ CPU_ZERO(&set);
+ CPU_SET(fb_h, &set);
+ if (sched_setaffinity(0, sizeof(set), &set) ||
+ set_deadline(0, 10 * NSEC_PER_MSEC, PERIOD)) {
+ mv_state[0] = -1;
+ _exit(1);
+ }
+ mv_state[0] = 1;
+ for (;;)
+ spin_ns(NSEC_PER_MSEC);
+ }
+ if (pid < 0 || child_up("no_admission_affinity_moves", 0))
+ goto out;
+ sleep_ns(PERIOD);
+ if (fallback_totals(&def, &hk))
+ goto out;
+ bw = hk - hk0;
+ if (bw <= 0 || def != def0) {
+ ksft_print_msg("the deadline task adds %lld to total_bw of its root domain\n", bw);
+ goto out;
+ }
+ CPU_ZERO(&set);
+ CPU_SET(fb_i2, &set);
+ if (sched_setaffinity(pid, sizeof(set), &set)) {
+ ksft_print_msg("sched_setaffinity to CPU %d: %s\n", fb_i2, strerror(errno));
+ goto out;
+ }
+ sleep_ns(2 * PERIOD);
+ at = task_cpu_of(pid);
+ if (at != fb_i2 || fallback_totals(&def, &hk))
+ goto out;
+ ret = KSFT_PASS;
+ if (def != def0 + bw || hk != hk0) {
+ ksft_print_msg("on CPU %d: total_bw %lld, %lld, not %lld, %lld\n",
+ at, def, hk, def0 + bw, hk0);
+ ret = KSFT_FAIL;
+ }
+out:
+ reap(pid);
+ if (write_file("/proc/sys/kernel", "sched_rt_runtime_us", saved)) {
+ ksft_print_msg("admission control cannot be turned on again\n");
+ return KSFT_FAIL;
+ }
+ sleep_ns(3 * PERIOD);
+ if (ret == KSFT_PASS && (fallback_totals(&def, &hk) || def != def0 || hk != hk0)) {
+ ksft_print_msg("task gone: total_bw %lld, %lld, not %lld, %lld\n",
+ def, hk, def0, hk0);
+ ret = KSFT_FAIL;
+ }
+ return ret;
+}
+
/*
* The cpusets of the cases below: B and Q in the root cgroup, C in B. A
* case makes those it needs. cs_setup() names them and enables the cpuset
@@ -7014,6 +7562,1368 @@ static int move_back_cpu_offline(void)
return ret;
}
+enum sp_mode { SP_HK_10, SP_HK_WAKING, SP_ISO_WAKING };
+
+/*
+ * Return a CPU other than @not: an isolated one if @side, else a
+ * housekeeping one. Return -1 if there is none.
+ */
+static int other_cpu(int side, int not)
+{
+ int n = side ? nisol : ncpus, *cpus = side ? isol : online, i;
+
+ for (i = n - 1; i >= 0; i--) {
+ if (cpus[i] != not)
+ return cpus[i];
+ }
+ return -1;
+}
+
+/*
+ * A deadline task loses its CPU to an offline. It is in the root cgroup,
+ * and allowed on every CPU: its mask holds CPUs of the two root domains.
+ * It must stay in the root domain that counts it. The steps:
+ * 1. A task runs on the last housekeeping CPU that can go offline. Then
+ * it allows every CPU and becomes a deadline task, of 10% for
+ * SP_HK_10, of 90% for SP_HK_WAKING. The housekeeping root domain
+ * counts it.
+ * For SP_ISO_WAKING, the task runs on the last isolated CPU that can
+ * go offline. Then it allows the isolated CPUs only, and becomes a
+ * deadline task of 90%. The root domain of the isolated CPUs counts
+ * it. Then it allows every CPU.
+ * The task of SP_HK_10 spins. The others spin 50us and sleep 50us, in
+ * a loop.
+ * 2. Wait a period. X is the CPU the task runs on.
+ * 3. Take X offline.
+ * 4. Wait two periods. The task must run on a CPU of its root domain.
+ * That root domain must still count it.
+ * The task of SP_HK_10 is throttled most of the time. Its replenishment
+ * timer then finds its CPU offline, and picks a CPU for it. The other
+ * tasks wake up while the offline goes on, and the wakeup picks a CPU
+ * for them.
+ *
+ * A kernel that does not prefer the root domain of the task at a wakeup
+ * gives it the first CPU of its mask. The waking cases fail on such a
+ * kernel only if that CPU is of the other kind. SP_HK_WAKING needs an
+ * isolated CPU first, SP_ISO_WAKING a housekeeping CPU first.
+ *
+ * The kernel records with the task the CPU where it became a deadline
+ * task. The waking cases need X to be that CPU: a wakeup in the offline
+ * looks at the root domain of that CPU. SP_HK_10 takes any housekeeping
+ * CPU as X.
+ *
+ * Do it in rounds, each with a new task. Stop once 3 rounds check
+ * something, or after 10 rounds. A round checks nothing if X cannot go
+ * offline, or for a waking case, if X is not the CPU recorded.
+ *
+ * The references of total_bw are taken with no task: with X offline
+ * after the task is gone, and with every CPU online. The messages give
+ * total_bw of the root domain of the isolated CPUs first, then of the
+ * housekeeping one.
+ */
+static int spin_offline(enum sp_mode mode)
+{
+ uint64_t runtime = (mode == SP_HK_10 ? 10 : 90) * NSEC_PER_MSEC;
+ int iso = mode == SP_ISO_WAKING, start = -1, rounds = 0, round;
+ long long def0, hk0, def, hk, ref_def, ref_hk, bw;
+ int x, d, h, cpu, i, down, ret = KSFT_SKIP;
+ pid_t pid;
+
+ for (i = 0; i < (iso ? nisol : ncpus); i++) {
+ if (cpu_hotpluggable(iso ? isol[i] : online[i]))
+ start = iso ? isol[i] : online[i];
+ }
+ if (!nisol || (iso && nisol < 2) || start < 0) {
+ ksft_print_msg("needs isolcpus=domain: %d isolated CPUs, one hotpluggable\n",
+ iso ? 2 : 1);
+ return KSFT_SKIP;
+ }
+ if (totals_of(isol[0], online[0], &def0, &hk0))
+ return KSFT_SKIP;
+ ret = KSFT_PASS;
+ for (round = 0; round < 10 && rounds < 3 && ret == KSFT_PASS; round++) {
+ mv_state[0] = 0;
+ mv_state[1] = 0;
+ pid = fork_child("a spinning deadline task");
+ if (!pid) {
+ cpu_set_t set;
+
+ CPU_ZERO(&set);
+ CPU_SET(start, &set);
+ if (sched_setaffinity(0, sizeof(set), &set))
+ _exit(1);
+ spin_ns(NSEC_PER_MSEC);
+ for (i = 0; i < (iso ? nisol : CPU_SETSIZE); i++)
+ CPU_SET(iso ? isol[i] : i, &set);
+ if (sched_setaffinity(0, sizeof(set), &set) ||
+ set_deadline(0, runtime, PERIOD)) {
+ mv_state[0] = -1;
+ _exit(1);
+ }
+ for (i = 0; i < CPU_SETSIZE; i++)
+ CPU_SET(i, &set);
+ if (sched_setaffinity(0, sizeof(set), &set)) {
+ mv_state[0] = -1;
+ _exit(1);
+ }
+ mv_state[0] = 1;
+ for (;;) {
+ mv_state[1]++;
+ if (mode == SP_HK_10)
+ continue;
+ spin_ns(50 * NSEC_PER_USEC);
+ sleep_ns(50 * NSEC_PER_USEC);
+ }
+ }
+ ret = KSFT_FAIL;
+ down = 0;
+ if (pid < 0 || child_up("spin_offline", 0))
+ goto next;
+ sleep_ns(PERIOD);
+ x = task_cpu_of(pid);
+ if (mode == SP_HK_10 ? in_cpu_list(isolated, x) || !cpu_hotpluggable(x) :
+ x != start) {
+ ret = KSFT_PASS; /* check nothing */
+ goto next;
+ }
+ d = other_cpu(1, x);
+ h = other_cpu(0, x);
+ if (totals_of(d, h, &def, &hk))
+ goto next;
+ bw = iso ? def - def0 : hk - hk0;
+ if (bw <= 0 || (iso ? hk != hk0 : def != def0)) {
+ ksft_print_msg("on CPU %d, the deadline task adds %lld, %lld to total_bw\n",
+ task_cpu_of(pid), def - def0, hk - hk0);
+ goto next;
+ }
+ if (task_cpu_of(pid) != x) {
+ ret = KSFT_PASS; /* moved off X: check nothing */
+ goto next;
+ }
+ rounds++;
+ if (cpu_online(x, 0)) {
+ ksft_print_msg("offline of CPU %d with the task on it refused\n", x);
+ goto next;
+ }
+ down = 1;
+ sleep_ns(2 * PERIOD);
+ cpu = task_cpu_of(pid);
+ if (totals_of(d, h, &def, &hk))
+ goto next;
+ reap(pid);
+ pid = -1;
+ sleep_ns(3 * PERIOD);
+ if (totals_of(d, h, &ref_def, &ref_hk))
+ goto next;
+ if (in_cpu_list(isolated, cpu) != iso) {
+ ksft_print_msg("round %d: CPU %d offline, the task runs on CPU %d\n",
+ round, x, cpu);
+ goto next;
+ }
+ if (iso ? def != ref_def + bw || hk != ref_hk :
+ hk != ref_hk + bw || def != ref_def) {
+ ksft_print_msg("round %d: on CPU %d: total_bw %lld, %lld, not %lld, %lld\n",
+ round, cpu, def, hk, iso ? ref_def + bw : ref_def,
+ iso ? ref_hk : ref_hk + bw);
+ goto next;
+ }
+ ret = KSFT_PASS;
+next:
+ reap(pid);
+ if (down)
+ cpu_up(x);
+ sleep_ns(3 * PERIOD);
+ if (ret == KSFT_PASS)
+ ret = totals_check("task gone", isol[0], online[0], def0, hk0);
+ }
+ if (ret == KSFT_PASS && !rounds) {
+ ksft_print_msg("the task never ran on a CPU that can go offline\n");
+ ret = KSFT_SKIP;
+ }
+ return ret;
+}
+
+static int hotplug_spinner_keeps(void)
+{
+ return spin_offline(SP_HK_10);
+}
+
+static int hotplug_waking_keeps(void)
+{
+ return spin_offline(SP_HK_WAKING);
+}
+
+static int hotplug_isolated_keeps(void)
+{
+ return spin_offline(SP_ISO_WAKING);
+}
+
+/*
+ * A rebuild of the sched domains must keep a deadline task in the root
+ * domain that counts it. The task wakes up often, so that a wakeup can
+ * land inside a rebuild. The steps:
+ * 1. A task runs on H, the highest housekeeping CPU. Then it allows every
+ * CPU and becomes a deadline task of 90%. The housekeeping root domain
+ * counts it. It spins 50us and sleeps 50us, in a loop.
+ * 2. Wait a period. The task must run on a housekeeping CPU.
+ * 3. Make Q a partition root of another housekeeping CPU, and remove it.
+ * Each of the two steps rebuilds the sched domains.
+ * 4. The task must run on a housekeeping CPU. The housekeeping root
+ * domain must still count it, and the other one must not.
+ * A kernel that gets this wrong gives the task the first CPU of the root
+ * domain of the isolated CPUs. That root domain then counts it, with no
+ * admission test. This shows only if that first CPU is isolated, as with
+ * isolcpus=domain,0,1. The case skips otherwise.
+ *
+ * Do it in 20 rounds, each with a new task. The first rebuild of a new
+ * task is the one most likely to go wrong. The case is not deterministic.
+ * The messages give total_bw of the root domain of the isolated CPUs
+ * first, then of the housekeeping one.
+ */
+static int rebuild_waking_keeps(void)
+{
+ int h = top_cpu(1), q = other_cpu(0, top_cpu(1)), d = nisol ? isol[0] : -1;
+ long long def0, hk0, def, hk;
+ int round, cpu, ret;
+ pid_t pid;
+
+ if (d < 0 || d > online[0] || q < 0) {
+ ksft_print_msg("needs isolcpus=domain with the first CPU isolated, as with 0,1\n");
+ return KSFT_SKIP;
+ }
+ if (cs_setup())
+ return KSFT_SKIP;
+ ret = KSFT_SKIP;
+ if (totals_of(d, h, &def0, &hk0))
+ goto out;
+ ret = KSFT_PASS;
+ for (round = 0; round < 20 && ret == KSFT_PASS; round++) {
+ mv_state[0] = 0;
+ pid = fork_child("a waking deadline task");
+ if (!pid) {
+ cpu_set_t set;
+ int i;
+
+ CPU_ZERO(&set);
+ CPU_SET(h, &set);
+ if (sched_setaffinity(0, sizeof(set), &set))
+ _exit(1);
+ spin_ns(NSEC_PER_MSEC);
+ for (i = 0; i < CPU_SETSIZE; i++)
+ CPU_SET(i, &set);
+ if (sched_setaffinity(0, sizeof(set), &set) ||
+ set_deadline(0, 90 * NSEC_PER_MSEC, PERIOD)) {
+ mv_state[0] = -1;
+ _exit(1);
+ }
+ mv_state[0] = 1;
+ for (;;) {
+ spin_ns(50 * NSEC_PER_USEC);
+ sleep_ns(50 * NSEC_PER_USEC);
+ }
+ }
+ ret = KSFT_FAIL;
+ if (pid < 0 || child_up("rebuild_waking_keeps", 0))
+ goto next;
+ sleep_ns(PERIOD);
+ cpu = task_cpu_of(pid);
+ if (totals_of(d, h, &def, &hk))
+ goto next;
+ if (in_cpu_list(isolated, cpu) || def != def0 || hk <= hk0) {
+ ksft_print_msg("round %d: on CPU %d before the rebuild: total_bw %lld, %lld\n",
+ round, cpu, def, hk);
+ goto next;
+ }
+ if (make_partition(cs_q, q)) {
+ ksft_print_msg("no partition root Q of CPU %d\n", q);
+ goto next;
+ }
+ remove_partition(cs_q);
+ cpu = task_cpu_of(pid);
+ if (totals_check("after the rebuild", d, h, def, hk) != KSFT_PASS)
+ ksft_print_msg("round %d: the task runs on CPU %d\n", round, cpu);
+ else if (in_cpu_list(isolated, cpu))
+ ksft_print_msg("round %d: after the rebuild, the task runs on CPU %d\n",
+ round, cpu);
+ else
+ ret = KSFT_PASS;
+next:
+ remove_partition(cs_q);
+ reap(pid);
+ sleep_ns(3 * PERIOD);
+ if (ret == KSFT_PASS)
+ ret = totals_check("task gone", d, h, def0, hk0);
+ }
+out:
+ cs_cleanup();
+ return ret;
+}
+
+/*
+ * A user affinity can keep a task out of the root domain that counts
+ * it. The CPUs are those of fallback_case(). B is a cpuset of I1, I2, H
+ * and K. C, in B, is a cpuset of H. Neither is a partition. The steps:
+ * 1. A task in the root cgroup allows I1 and I2 only. It becomes a
+ * deadline task of 10%. The root domain of the isolated CPUs counts
+ * it.
+ * 2. It sets its affinity to I1, I2 and H. That holds the root domain
+ * that counts it, so it is accepted. It sleeps.
+ * 3. Move it asleep into C. It has H only. The housekeeping root domain
+ * counts it now. Wake it: it runs on H.
+ * 4. Take H offline. C has no CPU left, and the task would get the CPUs
+ * of B: I1, I2 and K. Its affinity keeps I1 and I2 of them. That holds
+ * no CPU of the root domain that counts it.
+ * The offline must fail with EBUSY. The task must still sleep on H, and
+ * total_bw must not change. Wake it: it must run on H.
+ */
+static int hotplug_user_mask_refused(void)
+{
+ int fd[2] = { -1, -1 }, err, cpu, ret = KSFT_SKIP;
+ long long def0, hk0, def, hk, bw;
+ char buf[64];
+ pid_t pid = -1;
+
+ if (fallback_cpus(0) || !cpu_hotpluggable(fb_h)) {
+ ksft_print_msg("needs isolcpus=domain: 2 isolated, 2 housekeeping CPUs, hotplug\n");
+ return KSFT_SKIP;
+ }
+ if (cs_setup())
+ return KSFT_SKIP;
+ snprintf(buf, sizeof(buf), "%d,%d,%d,%d", fb_i1, fb_i2, fb_h, fb_k);
+ if (cs_make(cs_b, buf) || write_file(cs_b, "cgroup.subtree_control", "+cpuset"))
+ goto out;
+ snprintf(buf, sizeof(buf), "%d", fb_h);
+ if (cs_make(cs_c, buf) || fallback_totals(&def0, &hk0) || pipe(fd))
+ goto out;
+ ret = KSFT_FAIL;
+ mv_state[0] = 0;
+ mv_state[1] = 0;
+ pid = fork_child("a deadline task");
+ if (!pid) {
+ cpu_set_t set;
+ char c;
+
+ CPU_ZERO(&set);
+ CPU_SET(fb_i1, &set);
+ CPU_SET(fb_i2, &set);
+ if (sched_setaffinity(0, sizeof(set), &set) ||
+ set_deadline(0, 10 * NSEC_PER_MSEC, PERIOD)) {
+ mv_state[0] = -1;
+ _exit(1);
+ }
+ CPU_SET(fb_h, &set);
+ if (sched_setaffinity(0, sizeof(set), &set)) {
+ mv_state[0] = -1;
+ _exit(1);
+ }
+ mv_state[0] = 1;
+ while (read(fd[0], &c, 1) == 1)
+ mv_state[1]++;
+ _exit(0);
+ }
+ if (pid < 0 || child_up("hotplug_user_mask_refused", 0) ||
+ wait_for(1000, proc_state_of(pid) == 'S') || fallback_totals(&def, &hk))
+ goto out;
+ bw = def - def0;
+ if (bw <= 0 || hk != hk0) {
+ ksft_print_msg("the deadline task adds %lld, %lld to total_bw\n", bw, hk - hk0);
+ goto out;
+ }
+ err = enter(cs_c, pid);
+ if (err) {
+ ksft_print_msg("a move into C refused: %s\n", strerror(-err));
+ goto out;
+ }
+ if (fallback_wake(fd[1], pid) ||
+ totals_check("in C", fb_i2, fb_h, def0, hk0 + bw))
+ goto out;
+ cpu = task_cpu_of(pid);
+ if (cpu != fb_h) {
+ ksft_print_msg("woken in C on CPU %d, not %d\n", cpu, fb_h);
+ goto out;
+ }
+ err = cpu_online(fb_h, 0);
+ if (err != -EBUSY) {
+ ksft_print_msg("offline of CPU %d: %s, not EBUSY\n", fb_h,
+ err ? strerror(-err) : "accepted");
+ if (!err) {
+ if (!fallback_wake(fd[1], pid))
+ ksft_print_msg("woken on CPU %d\n", task_cpu_of(pid));
+ cpu_up(fb_h);
+ }
+ goto out;
+ }
+ cpu = task_cpu_of(pid);
+ if (cpu != fb_h || proc_state_of(pid) != 'S') {
+ ksft_print_msg("offline refused: the task is on CPU %d, state %c\n",
+ cpu, proc_state_of(pid));
+ goto out;
+ }
+ if (totals_check("offline refused", fb_i2, fb_h, def0, hk0 + bw) ||
+ fallback_wake(fd[1], pid))
+ goto out;
+ cpu = task_cpu_of(pid);
+ if (cpu != fb_h) {
+ ksft_print_msg("woken on CPU %d, not %d\n", cpu, fb_h);
+ goto out;
+ }
+ ret = totals_check("woken", fb_i2, fb_h, def0, hk0 + bw);
+out:
+ reap(pid);
+ if (fd[0] >= 0) {
+ close(fd[0]);
+ close(fd[1]);
+ }
+ sleep_ns(3 * PERIOD);
+ if (ret == KSFT_PASS)
+ ret = totals_check("task gone", fb_i2, fb_h, def0, hk0);
+ cs_cleanup();
+ return ret;
+}
+
+/*
+ * cgroup v2 works out the CPUs of each cpuset from the top down, at an
+ * offline too. A cpuset left with no CPU takes all those of its parent.
+ * Its child then keeps those that its own cpuset.cpus holds. The CPUs are
+ * those of fallback_case(). B, in the root cgroup, has cpuset.cpus H. C,
+ * in B, has cpuset.cpus I1, I2 and H. Neither is a partition. C gets H.
+ * The steps:
+ * 1. A task in the root cgroup runs on H. Then it allows every CPU and
+ * becomes a deadline task of 10%. It sleeps. The housekeeping root
+ * domain counts it.
+ * 2. Move it asleep into C. It keeps H.
+ * 3. Take H offline. B has no CPU left, and gets all the CPUs of the root
+ * cgroup. C keeps I1 and I2 of them: no CPU of the root domain that
+ * counts the task. The root cgroup itself still has K. The kernel must
+ * not take its CPUs for those of C.
+ * The offline must fail with EBUSY. The task must still sleep on H, and
+ * total_bw must not change. Wake it: it must run on H.
+ */
+static int hotplug_nested_refused(void)
+{
+ int fd[2] = { -1, -1 }, err, cpu, ret = KSFT_SKIP;
+ long long def0, hk0, def, hk, bw;
+ char buf[64];
+ pid_t pid = -1;
+
+ if (fallback_cpus(0) || !cpu_hotpluggable(fb_h)) {
+ ksft_print_msg("needs isolcpus=domain: 2 isolated, 2 housekeeping CPUs, hotplug\n");
+ return KSFT_SKIP;
+ }
+ if (cs_setup())
+ return KSFT_SKIP;
+ snprintf(buf, sizeof(buf), "%d", fb_h);
+ if (cs_make(cs_b, buf) ||
+ write_file(cs_b, "cgroup.subtree_control", "+cpuset"))
+ goto out;
+ snprintf(buf, sizeof(buf), "%d,%d,%d", fb_i1, fb_i2, fb_h);
+ if (cs_make(cs_c, buf) || fallback_totals(&def0, &hk0) || pipe(fd))
+ goto out;
+ ret = KSFT_FAIL;
+ mv_state[0] = 0;
+ mv_state[1] = 0;
+ pid = task_from(fb_h, 1, 10 * NSEC_PER_MSEC, PERIOD, fd[0]);
+ if (pid < 0 || child_up("hotplug_nested_refused", 0) ||
+ wait_for(1000, proc_state_of(pid) == 'S') ||
+ fallback_totals(&def, &hk))
+ goto out;
+ bw = hk - hk0;
+ err = enter(cs_c, pid);
+ cpu = task_cpu_of(pid);
+ if (bw <= 0 || def != def0 || err || cpu != fb_h) {
+ ksft_print_msg("a deadline task adds %lld, %lld to total_bw, in C: %s, on CPU %d\n",
+ def - def0, bw, strerror(-err), cpu);
+ goto out;
+ }
+ if (totals_check("in C", fb_i2, fb_h, def0, hk0 + bw))
+ goto out;
+ err = cpu_online(fb_h, 0);
+ if (err != -EBUSY) {
+ ksft_print_msg("offline of CPU %d: %s, not EBUSY\n", fb_h,
+ err ? strerror(-err) : "accepted");
+ if (!err) {
+ if (!fallback_wake(fd[1], pid))
+ ksft_print_msg("woken on CPU %d\n",
+ task_cpu_of(pid));
+ cpu_up(fb_h);
+ }
+ goto out;
+ }
+ cpu = task_cpu_of(pid);
+ if (cpu != fb_h || proc_state_of(pid) != 'S') {
+ ksft_print_msg("offline refused: the task is on CPU %d, state %c\n",
+ cpu, proc_state_of(pid));
+ goto out;
+ }
+ if (totals_check("offline refused", fb_i2, fb_h, def0, hk0 + bw) ||
+ fallback_wake(fd[1], pid))
+ goto out;
+ cpu = task_cpu_of(pid);
+ if (cpu != fb_h) {
+ ksft_print_msg("woken on CPU %d, not %d\n", cpu, fb_h);
+ goto out;
+ }
+ ret = totals_check("woken", fb_i2, fb_h, def0, hk0 + bw);
+out:
+ reap(pid);
+ if (fd[0] >= 0) {
+ close(fd[0]);
+ close(fd[1]);
+ }
+ sleep_ns(3 * PERIOD);
+ if (ret == KSFT_PASS)
+ ret = totals_check("task gone", fb_i2, fb_h, def0, hk0);
+ cs_cleanup();
+ return ret;
+}
+
+/*
+ * The CPUs of the two cases below. X, Y and Z are the three highest
+ * online CPUs: CPUs that isolcpus="domain" does not isolate. W, the
+ * lowest one, stays with the root cgroup. Return 0, or -1 if they are
+ * missing or X cannot go offline.
+ */
+static int pc_x, pc_y, pc_z, pc_w;
+
+static int partition_cpus(void)
+{
+ if (ncpus < 4 || !cpu_hotpluggable(top_cpu(3))) {
+ ksft_print_msg("needs 4 CPUs, CPU hotplug\n");
+ return -1;
+ }
+ pc_x = top_cpu(3);
+ pc_y = top_cpu(2);
+ pc_z = top_cpu(1);
+ pc_w = online[0];
+ return 0;
+}
+
+/*
+ * A partition root left with no CPU of its own takes the CPUs of a child
+ * partition, if it holds a task. The child partition becomes invalid. B
+ * is an isolated partition of X, Y and Z. C, in B, is an isolated
+ * partition of Y and Z. So B keeps X only. The steps:
+ * 1. Take X offline and back with no task, to read the total_bw of each
+ * root domain without X.
+ * 2. A task in the root cgroup runs on W. Then it allows every CPU and
+ * becomes a deadline task of 10%. It sleeps. The housekeeping root
+ * domain counts it.
+ * 3. Move it asleep into B. It has X only. The default root domain, which
+ * spans the CPUs of isolated partitions, counts it.
+ * 4. Take X offline. B takes Y and Z, and C becomes invalid. The task gets
+ * Y and Z: CPUs of the same root domain.
+ * The offline must be accepted. The default root domain must still count
+ * the task. Wake it: it must run on Y or Z.
+ */
+static int hotplug_child_partition_kept(void)
+{
+ int fd[2] = { -1, -1 }, err, cpu, ret = KSFT_SKIP, x_off = 0;
+ long long def0, hk0, def_off, hk_off, def, hk, bw;
+ char buf[64];
+ pid_t pid = -1;
+
+ if (partition_cpus() || cs_setup())
+ return KSFT_SKIP;
+ snprintf(buf, sizeof(buf), "%d,%d,%d", pc_x, pc_y, pc_z);
+ if (cs_make(cs_b, buf) ||
+ write_file(cs_b, "cpuset.cpus.partition", "isolated") ||
+ write_file(cs_b, "cgroup.subtree_control", "+cpuset"))
+ goto out;
+ snprintf(buf, sizeof(buf), "%d,%d", pc_y, pc_z);
+ if (cs_make(cs_c, buf) ||
+ write_file(cs_c, "cpuset.cpus.partition", "isolated") ||
+ !partition_has(cs_b, "isolated", 1) || partition_has(cs_b, "invalid", 0) ||
+ !partition_has(cs_c, "isolated", 1) || partition_has(cs_c, "invalid", 0))
+ goto out;
+ if (totals_of(pc_y, pc_w, &def0, &hk0))
+ goto out;
+ err = cpu_online(pc_x, 0);
+ if (err) {
+ ksft_print_msg("offline of CPU %d with no task: %s\n", pc_x, strerror(-err));
+ goto out;
+ }
+ err = totals_of(pc_y, pc_w, &def_off, &hk_off);
+ cpu_up(pc_x);
+ if (err || pipe(fd))
+ goto out;
+ ret = KSFT_FAIL;
+ mv_state[0] = 0;
+ mv_state[1] = 0;
+ pid = task_from(pc_w, 1, 10 * NSEC_PER_MSEC, PERIOD, fd[0]);
+ if (pid < 0 || child_up("hotplug_child_partition_kept", 0) ||
+ wait_for(1000, proc_state_of(pid) == 'S') || totals_of(pc_y, pc_w, &def, &hk))
+ goto out;
+ bw = hk - hk0;
+ err = enter(cs_b, pid);
+ if (bw <= 0 || def != def0 || err) {
+ ksft_print_msg("a deadline task adds %lld, %lld to total_bw, in B: %s\n",
+ def - def0, bw, strerror(-err));
+ goto out;
+ }
+ if (totals_check("in B", pc_y, pc_w, def0 + bw, hk0) || fallback_wake(fd[1], pid))
+ goto out;
+ cpu = task_cpu_of(pid);
+ if (cpu != pc_x) {
+ ksft_print_msg("woken in B on CPU %d, not %d\n", cpu, pc_x);
+ goto out;
+ }
+ err = cpu_online(pc_x, 0);
+ if (err) {
+ ksft_print_msg("offline of CPU %d: %s, not accepted\n", pc_x, strerror(-err));
+ goto out;
+ }
+ x_off = 1;
+ if (!partition_has(cs_c, "invalid", 1) ||
+ totals_check("X offline", pc_y, pc_w, def_off + bw, hk_off) ||
+ fallback_wake(fd[1], pid))
+ goto out;
+ cpu = task_cpu_of(pid);
+ if (cpu != pc_y && cpu != pc_z) {
+ ksft_print_msg("woken on CPU %d, not %d or %d\n", cpu, pc_y, pc_z);
+ goto out;
+ }
+ ret = KSFT_PASS;
+out:
+ reap(pid);
+ if (fd[0] >= 0) {
+ close(fd[0]);
+ close(fd[1]);
+ }
+ if (x_off)
+ cpu_up(pc_x);
+ sleep_ns(3 * PERIOD);
+ if (ret == KSFT_PASS)
+ ret = totals_check("task gone", pc_y, pc_w, def0, hk0);
+ cs_cleanup();
+ return ret;
+}
+
+/*
+ * The same rule, with a child partition of another kind. B is a root
+ * partition of X and Y. C, in B, is an isolated partition of Y. So B keeps
+ * X only, in a root domain of its own. N, in B, is a cpuset that is not a
+ * partition. The steps:
+ * 1. A task that is not a deadline task sleeps in N.
+ * 2. A task in the root cgroup runs on W. Then it allows every CPU and
+ * becomes a deadline task of 10%. It sleeps. The housekeeping root
+ * domain counts it.
+ * 3. Move it asleep into C. The default root domain counts it.
+ * 4. Take X offline. B would take Y, and C would become invalid. Y would
+ * leave the default root domain for that of B, with the deadline task.
+ * That root domain did not admit it.
+ * The offline must fail with EBUSY. C must stay valid, and total_bw must
+ * not change. Wake the task: it must run on Y. Once the task is gone, the
+ * offline must be accepted.
+ */
+static int hotplug_partition_invalid_refused(void)
+{
+ int fd[2] = { -1, -1 }, err, cpu, ret = KSFT_SKIP, x_off = 0;
+ long long def0, hk0, def, hk, bw;
+ char buf[64], cs_n[PATH_MAX];
+ pid_t pid = -1, other = -1;
+
+ if (partition_cpus() || cs_setup())
+ return KSFT_SKIP;
+ if (snprintf(cs_n, sizeof(cs_n), "%s/n", cs_b) >= (int)sizeof(cs_n))
+ goto out;
+ snprintf(buf, sizeof(buf), "%d,%d", pc_x, pc_y);
+ if (cs_make(cs_b, buf) ||
+ write_file(cs_b, "cpuset.cpus.partition", "root") ||
+ write_file(cs_b, "cgroup.subtree_control", "+cpuset"))
+ goto out;
+ snprintf(buf, sizeof(buf), "%d", pc_y);
+ if (cs_make(cs_c, buf) ||
+ write_file(cs_c, "cpuset.cpus.partition", "isolated") ||
+ !partition_has(cs_b, "root", 1) || partition_has(cs_b, "invalid", 0) ||
+ !partition_has(cs_c, "isolated", 1) || partition_has(cs_c, "invalid", 0))
+ goto out;
+ if (mkdir(cs_n, 0755) && errno != EEXIST) {
+ ksft_print_msg("mkdir %s: %s\n", cs_n, strerror(errno));
+ goto out;
+ }
+ other = fork_child("a task in N");
+ if (!other) {
+ for (;;)
+ pause();
+ }
+ if (other < 0 || enter(cs_n, other) || totals_of(pc_y, pc_w, &def0, &hk0) ||
+ pipe(fd))
+ goto out;
+ ret = KSFT_FAIL;
+ mv_state[0] = 0;
+ mv_state[1] = 0;
+ pid = task_from(pc_w, 1, 10 * NSEC_PER_MSEC, PERIOD, fd[0]);
+ if (pid < 0 || child_up("hotplug_partition_invalid_refused", 0) ||
+ wait_for(1000, proc_state_of(pid) == 'S') || totals_of(pc_y, pc_w, &def, &hk))
+ goto out;
+ bw = hk - hk0;
+ err = enter(cs_c, pid);
+ if (bw <= 0 || def != def0 || err) {
+ ksft_print_msg("a deadline task adds %lld, %lld to total_bw, in C: %s\n",
+ def - def0, bw, strerror(-err));
+ goto out;
+ }
+ if (totals_check("in C", pc_y, pc_w, def0 + bw, hk0))
+ goto out;
+ err = cpu_online(pc_x, 0);
+ if (err != -EBUSY) {
+ ksft_print_msg("offline of CPU %d: %s, not EBUSY\n", pc_x,
+ err ? strerror(-err) : "accepted");
+ x_off = !err;
+ goto out;
+ }
+ if (partition_has(cs_c, "invalid", 0) ||
+ totals_check("offline refused", pc_y, pc_w, def0 + bw, hk0) ||
+ fallback_wake(fd[1], pid))
+ goto out;
+ cpu = task_cpu_of(pid);
+ if (cpu != pc_y) {
+ ksft_print_msg("woken on CPU %d, not %d\n", cpu, pc_y);
+ goto out;
+ }
+ reap(pid);
+ pid = -1;
+ sleep_ns(3 * PERIOD);
+ err = cpu_online(pc_x, 0);
+ if (err) {
+ ksft_print_msg("offline of CPU %d with the task gone: %s\n", pc_x,
+ strerror(-err));
+ goto out;
+ }
+ x_off = 1;
+ ret = KSFT_PASS;
+out:
+ reap(pid);
+ reap(other);
+ if (fd[0] >= 0) {
+ close(fd[0]);
+ close(fd[1]);
+ }
+ if (x_off)
+ cpu_up(pc_x);
+ sleep_ns(3 * PERIOD);
+ if (ret == KSFT_PASS)
+ ret = totals_check("task gone", pc_y, pc_w, def0, hk0);
+ if (cs_n[0] && wait_for(1000, !rmdir(cs_n) || errno == ENOENT))
+ ksft_print_msg("rmdir %s: %s\n", cs_n, strerror(errno));
+ cs_cleanup();
+ return ret;
+}
+
+/*
+ * A dead task runs no more, so it cannot leave its root domain. The CPUs
+ * are those of fallback_case(). B is a cpuset of I1, I2 and H, not a
+ * partition. The steps:
+ * 1. A task in the root cgroup runs on H. Then it allows every CPU and
+ * becomes a deadline task of 100ms every 1s. It sleeps.
+ * 2. Move it asleep into B. It keeps H. The housekeeping root domain
+ * counts it.
+ * 3. Wake it. It spins 90ms and exits. The case does not reap it. Its
+ * 0-lag time is about 800ms ahead.
+ * 4. Take H offline. Once H is offline, the dead task would have no CPU
+ * in the root domain that counts it. That does not matter: it never
+ * runs again.
+ * The offline must be accepted. The case then reaps the task and brings
+ * H back. Past the 0-lag time, total_bw must be what it was.
+ *
+ * The case depends on time: it skips if the task takes more than 500ms
+ * to show as a zombie.
+ */
+static int hotplug_dead_accepted(void)
+{
+ int fd[2] = { -1, -1 }, err, cpu, ret = KSFT_SKIP;
+ long long def0, hk0, def, hk;
+ uint64_t t;
+ char buf[64];
+ pid_t pid = -1;
+
+ if (fallback_cpus(0) || !cpu_hotpluggable(fb_h)) {
+ ksft_print_msg("needs isolcpus=domain: 2 isolated, 2 housekeeping CPUs, hotplug\n");
+ return KSFT_SKIP;
+ }
+ if (cs_setup())
+ return KSFT_SKIP;
+ snprintf(buf, sizeof(buf), "%d,%d,%d", fb_i1, fb_i2, fb_h);
+ if (cs_make(cs_b, buf) || fallback_totals(&def0, &hk0) || pipe(fd))
+ goto out;
+ ret = KSFT_FAIL;
+ mv_state[0] = 0;
+ mv_state[1] = 0;
+ pid = task_from(fb_h, 1, 100 * NSEC_PER_MSEC, NSEC_PER_SEC, fd[0]);
+ if (pid < 0 || child_up("hotplug_dead_accepted", 0) ||
+ wait_for(1000, proc_state_of(pid) == 'S'))
+ goto out;
+ err = enter(cs_b, pid);
+ if (err) {
+ ksft_print_msg("a move into B refused: %s\n", strerror(-err));
+ goto out;
+ }
+ if (fallback_totals(&def, &hk))
+ goto out;
+ cpu = task_cpu_of(pid);
+ if (cpu != fb_h || def != def0 || hk <= hk0) {
+ ksft_print_msg("in B on CPU %d: total_bw %lld, %lld\n", cpu, def, hk);
+ goto out;
+ }
+ if (write(fd[1], "x", 1) != 1) {
+ ksft_print_msg("write to the deadline task: %s\n", strerror(errno));
+ goto out;
+ }
+ t = now_ns();
+ if (wait_for(1000, proc_state_of(pid) == 'Z'))
+ goto out;
+ if (now_ns() - t > 500 * NSEC_PER_MSEC) {
+ ksft_print_msg("the task took over 500ms to exit\n");
+ ret = KSFT_SKIP;
+ goto out;
+ }
+ sleep_ns(10 * NSEC_PER_MSEC); /* past do_task_dead() */
+ err = cpu_online(fb_h, 0);
+ if (err) {
+ ksft_print_msg("offline of CPU %d under a dead task: %s\n", fb_h,
+ strerror(-err));
+ goto out;
+ }
+ reap(pid);
+ pid = -1;
+ cpu_up(fb_h);
+ sleep_ns(NSEC_PER_SEC); /* past the 0-lag time */
+ ret = totals_check("task gone", fb_i2, fb_h, def0, hk0);
+out:
+ reap(pid);
+ cpu_up(fb_h);
+ if (fd[0] >= 0) {
+ close(fd[0]);
+ close(fd[1]);
+ }
+ cs_cleanup();
+ return ret;
+}
+
+/*
+ * A cpuset that loses all its CPUs takes them from its parent, through
+ * its own cpuset.cpus. The CPUs are those of fallback_case(). The steps:
+ * 1. B, in the root cgroup, has cpuset.cpus H. C, in B, has cpuset.cpus
+ * I2. I2 is not in B, so C gets H.
+ * 2. A task in the root cgroup runs on H. Then it allows every CPU and
+ * becomes a deadline task of 10%. It sleeps. The housekeeping root
+ * domain counts it.
+ * 3. Move it asleep into C. It keeps H.
+ * 4. Q, in the root cgroup, has cpuset.cpus H. Make it a partition root.
+ * B would then get the CPUs of the root cgroup, and C would get I2
+ * of them. The task would have no CPU in the root domain that counts
+ * it.
+ * Q must be an invalid partition root, for the deadline task. Wake the
+ * task: it must run on H, and the housekeeping root domain must count it.
+ */
+static int partition_takes_fallback_refused(void)
+{
+ int fd[2] = { -1, -1 }, cpu, err, ret = KSFT_SKIP;
+ long long def0, hk0, def, hk, bw;
+ char buf[64];
+ pid_t pid = -1;
+
+ if (fallback_cpus(0)) {
+ ksft_print_msg("needs isolcpus=domain: 2 isolated, 2 housekeeping CPUs\n");
+ return KSFT_SKIP;
+ }
+ if (cs_setup())
+ return KSFT_SKIP;
+ snprintf(buf, sizeof(buf), "%d", fb_h);
+ if (cs_make(cs_b, buf) || write_file(cs_b, "cgroup.subtree_control", "+cpuset"))
+ goto out;
+ snprintf(buf, sizeof(buf), "%d", fb_i2);
+ if (cs_make(cs_c, buf) || fallback_totals(&def0, &hk0) || pipe(fd))
+ goto out;
+ ret = KSFT_FAIL;
+ mv_state[0] = 0;
+ mv_state[1] = 0;
+ pid = task_from(fb_h, 1, 10 * NSEC_PER_MSEC, PERIOD, fd[0]);
+ if (pid < 0 || child_up("partition_takes_fallback_refused", 0) ||
+ wait_for(1000, proc_state_of(pid) == 'S') || fallback_totals(&def, &hk))
+ goto out;
+ bw = hk - hk0;
+ err = enter(cs_c, pid);
+ cpu = task_cpu_of(pid);
+ if (bw <= 0 || def != def0 || err || cpu != fb_h) {
+ ksft_print_msg("a deadline task adds %lld, %lld to total_bw, in C: %s, on CPU %d\n",
+ def - def0, bw, strerror(-err), cpu);
+ goto out;
+ }
+ snprintf(buf, sizeof(buf), "%d", fb_h);
+ if (cs_make(cs_q, buf))
+ goto out;
+ write_file(cs_q, "cpuset.cpus.partition", "root");
+ if (!partition_has(cs_q, "Deadline task", 1) ||
+ fallback_wake(fd[1], pid))
+ goto out;
+ cpu = task_cpu_of(pid);
+ if (cpu != fb_h) {
+ ksft_print_msg("woken on CPU %d, not %d\n", cpu, fb_h);
+ goto out;
+ }
+ ret = totals_check("woken", fb_i2, fb_h, def0, hk0 + bw);
+out:
+ reap(pid);
+ if (fd[0] >= 0) {
+ close(fd[0]);
+ close(fd[1]);
+ }
+ sleep_ns(3 * PERIOD);
+ if (ret == KSFT_PASS)
+ ret = totals_check("task gone", fb_i2, fb_h, def0, hk0);
+ cs_cleanup();
+ return ret;
+}
+
+/*
+ * A mask that grows can still take a task out of its root domain. That
+ * happens if the user affinity of the task holds only the new CPUs.
+ * The CPUs: I1 is isolated by isolcpus="domain", H is the first
+ * housekeeping CPU. The steps:
+ * 1. Q is a partition root of the other housekeeping CPUs. The
+ * housekeeping root domain of the root cgroup then has H only.
+ * 2. B, in the root cgroup, is a cpuset of H and I1. A task enters B. It
+ * sets its affinity to I1, and runs there. It is not a deadline task.
+ * 3. Give B the CPU H only. The task gets H: its affinity holds no CPU of
+ * B. Wake it: it runs on H.
+ * 4. The task becomes a deadline task of 10%. Its mask holds the whole
+ * housekeeping root domain of the root cgroup, H. That root domain
+ * counts it. It sleeps.
+ * 5. Give B the CPUs H and I1 again. The task would then get I1 only, its
+ * affinity, in another root domain.
+ * The write of step 5 must fail with EBUSY. Wake the task: it must run on
+ * H, and the housekeeping root domain must count it.
+ */
+static int mask_grows_refused(void)
+{
+ int h = online[0], i1 = nisol ? isol[0] : -1, fd[2] = { -1, -1 };
+ int i, n = 0, err, cpu, ret = KSFT_SKIP;
+ long long def0, hk0, def, hk, bw;
+ char buf[4096];
+ pid_t pid = -1;
+
+ if (i1 < 0) {
+ ksft_print_msg("needs CPUs isolated by isolcpus=domain\n");
+ return KSFT_SKIP;
+ }
+ if (cs_setup())
+ return KSFT_SKIP;
+ for (i = 1; i < ncpus && n < (int)sizeof(buf); i++)
+ n += snprintf(buf + n, sizeof(buf) - n, "%s%d", i > 1 ? "," : "", online[i]);
+ if (n >= (int)sizeof(buf) || make_partition_cpus(cs_q, buf)) {
+ ksft_print_msg("no partition root Q of CPUs %s\n", buf);
+ goto out;
+ }
+ snprintf(buf, sizeof(buf), "%d,%d", h, i1);
+ if (cs_make(cs_b, buf) || totals_of(i1, h, &def0, &hk0) || pipe(fd))
+ goto out;
+ ret = KSFT_FAIL;
+ mv_state[0] = 0;
+ mv_state[1] = 0;
+ pid = fork_child("a task with an affinity");
+ if (!pid) {
+ cpu_set_t set;
+ char c;
+
+ CPU_ZERO(&set);
+ CPU_SET(i1, &set);
+ if (enter(cs_b, getpid()) || sched_setaffinity(0, sizeof(set), &set)) {
+ mv_state[1] = -1;
+ _exit(1);
+ }
+ spin_ns(NSEC_PER_MSEC);
+ mv_state[1] = 1;
+ /* Wait for B of H only. Run there, and become a deadline task. */
+ if (read(fd[0], &c, 1) != 1)
+ _exit(1);
+ spin_ns(NSEC_PER_MSEC);
+ if (set_deadline(0, 10 * NSEC_PER_MSEC, PERIOD)) {
+ mv_state[0] = -1;
+ _exit(1);
+ }
+ mv_state[0] = 1;
+ while (read(fd[0], &c, 1) == 1)
+ mv_state[1]++;
+ _exit(0);
+ }
+ if (pid < 0 || wait_for(1000, mv_state[1]) || mv_state[1] != 1 ||
+ wait_for(1000, proc_state_of(pid) == 'S')) {
+ ksft_print_msg("the task cannot take CPU %d\n", i1);
+ goto out;
+ }
+ snprintf(buf, sizeof(buf), "%d", h);
+ err = write_file(cs_b, "cpuset.cpus", buf);
+ if (err) {
+ ksft_print_msg("B given CPU %d only: %s\n", h, strerror(-err));
+ goto out;
+ }
+ mv_state[1] = 0;
+ if (write(fd[1], "d", 1) != 1 || child_up("mask_grows_refused", 0) ||
+ wait_for(1000, proc_state_of(pid) == 'S') || totals_of(i1, h, &def, &hk))
+ goto out;
+ bw = hk - hk0;
+ cpu = task_cpu_of(pid);
+ if (bw <= 0 || def != def0 || cpu != h) {
+ ksft_print_msg("a deadline task on CPU %d adds %lld, %lld to total_bw\n",
+ cpu, def - def0, bw);
+ goto out;
+ }
+ snprintf(buf, sizeof(buf), "%d,%d", h, i1);
+ err = write_file(cs_b, "cpuset.cpus", buf);
+ if (err != -EBUSY) {
+ ksft_print_msg("B given CPUs %s: %s, not EBUSY\n", buf,
+ err ? strerror(-err) : "accepted");
+ if (!err && !fallback_wake(fd[1], pid))
+ ksft_print_msg("woken on CPU %d\n", task_cpu_of(pid));
+ goto out;
+ }
+ if (fallback_wake(fd[1], pid))
+ goto out;
+ cpu = task_cpu_of(pid);
+ if (cpu != h) {
+ ksft_print_msg("woken on CPU %d, not %d\n", cpu, h);
+ goto out;
+ }
+ ret = totals_check("woken", i1, h, def0, hk0 + bw);
+out:
+ reap(pid);
+ if (fd[0] >= 0) {
+ close(fd[0]);
+ close(fd[1]);
+ }
+ sleep_ns(3 * PERIOD);
+ if (ret == KSFT_PASS)
+ ret = totals_check("task gone", i1, h, def0, hk0);
+ cs_cleanup();
+ return ret;
+}
+
+/*
+ * One write to cpuset.cpus can make a partition root take CPUs and give
+ * others back. The CPUs: I1 is isolated by isolcpus="domain". H is a
+ * housekeeping CPU. The steps:
+ * 1. Q is an isolated partition root of I1.
+ * 2. B, in the root cgroup, has cpuset.cpus H and I1. Q holds I1, so B
+ * gets H.
+ * 3. A task in the root cgroup runs on H. Then it allows every CPU and
+ * becomes a deadline task of 10%. It sleeps. The housekeeping root
+ * domain counts it.
+ * 4. Move it asleep into B. It keeps H.
+ * 5. Write H to the cpuset.cpus of Q. Q would take H and give I1 back. B
+ * would then get I1 only, in another root domain.
+ * Q must become an invalid partition root, for the deadline task. Wake
+ * the task: it must run on H, and the housekeeping root domain must count
+ * it.
+ */
+static int partition_swap_refused(void)
+{
+ int h = online[ncpus - 1], i1 = nisol ? isol[0] : -1, fd[2] = { -1, -1 };
+ int cpu, err, ret = KSFT_SKIP;
+ long long def0, hk0, def, hk, bw;
+ char buf[64];
+ pid_t pid = -1;
+
+ if (i1 < 0) {
+ ksft_print_msg("needs CPUs isolated by isolcpus=domain\n");
+ return KSFT_SKIP;
+ }
+ if (cs_setup())
+ return KSFT_SKIP;
+ snprintf(buf, sizeof(buf), "%d", i1);
+ if (cs_make(cs_q, buf) || write_file(cs_q, "cpuset.cpus.partition", "isolated") ||
+ !partition_has(cs_q, "isolated", 1) || partition_has(cs_q, "invalid", 0)) {
+ ksft_print_msg("no isolated partition Q of CPU %d\n", i1);
+ goto out;
+ }
+ snprintf(buf, sizeof(buf), "%d,%d", h, i1);
+ if (cs_make(cs_b, buf) || totals_of(i1, h, &def0, &hk0) || pipe(fd))
+ goto out;
+ ret = KSFT_FAIL;
+ mv_state[0] = 0;
+ mv_state[1] = 0;
+ pid = task_from(h, 1, 10 * NSEC_PER_MSEC, PERIOD, fd[0]);
+ if (pid < 0 || child_up("partition_swap_refused", 0) ||
+ wait_for(1000, proc_state_of(pid) == 'S') || totals_of(i1, h, &def, &hk))
+ goto out;
+ bw = hk - hk0;
+ err = enter(cs_b, pid);
+ cpu = task_cpu_of(pid);
+ if (bw <= 0 || def != def0 || err || cpu != h) {
+ ksft_print_msg("a deadline task adds %lld, %lld to total_bw, in B: %s, on CPU %d\n",
+ def - def0, bw, strerror(-err), cpu);
+ goto out;
+ }
+ snprintf(buf, sizeof(buf), "%d", h);
+ write_file(cs_q, "cpuset.cpus", buf);
+ if (!partition_has(cs_q, "Deadline task", 1) || fallback_wake(fd[1], pid))
+ goto out;
+ cpu = task_cpu_of(pid);
+ if (cpu != h) {
+ ksft_print_msg("woken on CPU %d, not %d\n", cpu, h);
+ goto out;
+ }
+ ret = totals_check("woken", i1, h, def0, hk0 + bw);
+out:
+ reap(pid);
+ if (fd[0] >= 0) {
+ close(fd[0]);
+ close(fd[1]);
+ }
+ sleep_ns(3 * PERIOD);
+ if (ret == KSFT_PASS)
+ ret = totals_check("task gone", i1, h, def0, hk0);
+ cs_cleanup();
+ return ret;
+}
+
+/* Return the taint flags of the kernel, or -1. */
+static long taint_flags(void)
+{
+ char buf[64];
+
+ if (read_file("/proc/sys/kernel/tainted", buf, sizeof(buf)))
+ return -1;
+ return atol(buf);
+}
+
+#define TAINT_WARN (1L << 9)
+
+/*
+ * An invalid partition root holds no CPU of its parent. A refusal must
+ * not give any back. The CPUs are those of fallback_case(). B is a cpuset
+ * of I1, I2 and H, not a partition. The steps:
+ * 1. A task in the root cgroup runs on H. Then it allows every CPU and
+ * becomes a deadline task of 10%. It sleeps. The housekeeping root
+ * domain counts it.
+ * 2. Move it asleep into B. It keeps H.
+ * 3. Q, in the root cgroup, has cpuset.cpus H and K. Its
+ * cpuset.cpus.exclusive is H. Make it a partition root. It would take H,
+ * the only housekeeping CPU of B. Q must be invalid, for the deadline
+ * task.
+ * 4. Write H to the cpuset.cpus of Q. The same check refuses again.
+ * The kernel must not warn: the taint flag W must stay clear. Q must
+ * still be invalid for the deadline task. The case skips if the flag
+ * was set before.
+ */
+static int partition_refused_no_warn(void)
+{
+ int fd[2] = { -1, -1 }, cpu, err, ret = KSFT_SKIP;
+ long long def0, hk0, def, hk, bw;
+ char buf[64];
+ pid_t pid = -1;
+
+ if (fallback_cpus(0)) {
+ ksft_print_msg("needs isolcpus=domain: 2 isolated, 2 housekeeping CPUs\n");
+ return KSFT_SKIP;
+ }
+ if (taint_flags() < 0 || (taint_flags() & TAINT_WARN)) {
+ ksft_print_msg("the kernel has warned already\n");
+ return KSFT_SKIP;
+ }
+ if (cs_setup())
+ return KSFT_SKIP;
+ snprintf(buf, sizeof(buf), "%d,%d,%d", fb_i1, fb_i2, fb_h);
+ if (cs_make(cs_b, buf) || fallback_totals(&def0, &hk0) || pipe(fd))
+ goto out;
+ ret = KSFT_FAIL;
+ mv_state[0] = 0;
+ mv_state[1] = 0;
+ pid = task_from(fb_h, 1, 10 * NSEC_PER_MSEC, PERIOD, fd[0]);
+ if (pid < 0 || child_up("partition_refused_no_warn", 0) ||
+ wait_for(1000, proc_state_of(pid) == 'S') || fallback_totals(&def, &hk))
+ goto out;
+ bw = hk - hk0;
+ err = enter(cs_b, pid);
+ cpu = task_cpu_of(pid);
+ if (bw <= 0 || def != def0 || err || cpu != fb_h) {
+ ksft_print_msg("a deadline task adds %lld, %lld to total_bw, in B: %s, on CPU %d\n",
+ def - def0, bw, strerror(-err), cpu);
+ goto out;
+ }
+ snprintf(buf, sizeof(buf), "%d,%d", fb_h, fb_k);
+ if (cs_make(cs_q, buf))
+ goto out;
+ snprintf(buf, sizeof(buf), "%d", fb_h);
+ if (write_file(cs_q, "cpuset.cpus.exclusive", buf)) {
+ ksft_print_msg("Q cannot have CPU %d exclusive\n", fb_h);
+ goto out;
+ }
+ write_file(cs_q, "cpuset.cpus.partition", "root");
+ if (!partition_has(cs_q, "Deadline task", 1))
+ goto out;
+ write_file(cs_q, "cpuset.cpus", buf);
+ if (taint_flags() & TAINT_WARN) {
+ ksft_print_msg("Q given CPU %d: the kernel warned\n", fb_h);
+ goto out;
+ }
+ if (!partition_has(cs_q, "Deadline task", 1) || fallback_wake(fd[1], pid))
+ goto out;
+ cpu = task_cpu_of(pid);
+ if (cpu != fb_h) {
+ ksft_print_msg("woken on CPU %d, not %d\n", cpu, fb_h);
+ goto out;
+ }
+ ret = totals_check("woken", fb_i2, fb_h, def0, hk0 + bw);
+out:
+ reap(pid);
+ if (fd[0] >= 0) {
+ close(fd[0]);
+ close(fd[1]);
+ }
+ sleep_ns(3 * PERIOD);
+ if (ret == KSFT_PASS)
+ ret = totals_check("task gone", fb_i2, fb_h, def0, hk0);
+ cs_cleanup();
+ return ret;
+}
+
+/*
+ * A rebuild must count a task where its cpuset lets it run. The kernel
+ * records, with the bandwidth of a task, a CPU of the root domain that
+ * counts it. A partition root can take that CPU later. A, B and D are the
+ * three highest housekeeping CPUs. C is the first one. Q is a partition
+ * root of the other housekeeping CPUs, C among them. The steps:
+ * 1. Take D offline. The housekeeping root domain then has A and B only.
+ * 2. A task in the root cgroup sets its affinity to A and B. It becomes a
+ * deadline task of 10% and sleeps. Bring D back online. The task keeps
+ * its affinity.
+ * 3. Take A and B offline. No CPU of the task's mask is active. Its
+ * cpuset gives it D, so the rebuild counts it on D. D is recorded.
+ * 4. Make Q a member, then a partition root of D instead of C. The
+ * cpuset of the task now gives it C. Its mask is still A and B.
+ * Q must not count the task. The root domain of C must count it. Wake
+ * the task: it must run on C, and total_bw must not change.
+ *
+ * The references of total_bw are taken once the task is gone, with the
+ * CPUs as in step 4.
+ */
+static int rebuild_record_taken(void)
+{
+ int a = top_cpu(3), b = top_cpu(2), c = online[0], d = top_cpu(1);
+ int fd[2] = { -1, -1 }, i, n, err, cpu, ret = KSFT_SKIP;
+ long long q, h, q0, h0, bw;
+ char buf[4096];
+ pid_t pid = -1;
+
+ if (ncpus < 4 || !cpu_hotpluggable(a) || !cpu_hotpluggable(b) ||
+ !cpu_hotpluggable(d)) {
+ ksft_print_msg("needs 4 CPUs, 3 of them hotpluggable\n");
+ return KSFT_SKIP;
+ }
+ if (cs_setup())
+ return KSFT_SKIP;
+ /* Q has every housekeeping CPU but A, B and D. */
+ for (i = n = 0; i < ncpus - 3; i++)
+ n += snprintf(buf + n, sizeof(buf) - n, "%s%d", n ? "," : "",
+ online[i]);
+ if (make_partition_cpus(cs_q, buf)) {
+ ksft_print_msg("Q cannot be a partition root of CPUs %s\n", buf);
+ goto out;
+ }
+ ret = KSFT_FAIL;
+ err = cpu_online(d, 0);
+ if (err) {
+ ksft_print_msg("offline of CPU %d: %s\n", d, strerror(-err));
+ goto out;
+ }
+ bw = total_bw_of(a);
+ if (bw == -1) {
+ ksft_print_msg("no total_bw in /sys/kernel/debug/sched/debug\n");
+ ret = KSFT_SKIP;
+ goto out;
+ }
+ if (pipe(fd)) {
+ ksft_print_msg("pipe: %s\n", strerror(errno));
+ goto out;
+ }
+ mv_state[0] = 0;
+ mv_state[1] = 0;
+ pid = fork_child("a deadline task");
+ if (!pid) {
+ cpu_set_t set;
+ char ch;
+
+ CPU_ZERO(&set);
+ CPU_SET(a, &set);
+ CPU_SET(b, &set);
+ if (sched_setaffinity(0, sizeof(set), &set) ||
+ set_deadline(0, 10 * NSEC_PER_MSEC, PERIOD)) {
+ mv_state[0] = -1;
+ _exit(1);
+ }
+ mv_state[0] = 1;
+ while (read(fd[0], &ch, 1) == 1)
+ mv_state[1]++;
+ _exit(0);
+ }
+ if (pid < 0 || child_up("rebuild_record_taken", 0) ||
+ wait_for(1000, proc_state_of(pid) == 'S'))
+ goto out;
+ bw = total_bw_of(a) - bw;
+ if (bw <= 0) {
+ ksft_print_msg("the deadline task adds %lld to total_bw\n", bw);
+ goto out;
+ }
+ err = cpu_online(d, 1);
+ if (!err)
+ err = cpu_online(a, 0);
+ if (!err)
+ err = cpu_online(b, 0);
+ if (err) {
+ ksft_print_msg("hotplug of CPUs %d, %d, %d: %s\n", d, a, b,
+ strerror(-err));
+ goto out;
+ }
+ /*
+ * Q has D and every CPU it had but C. One write of cpuset.cpus
+ * would make Q an invalid partition root: the check sees the root
+ * cgroup lose D, and get C back too late.
+ */
+ for (i = 1, n = 0; i < ncpus - 3; i++)
+ n += snprintf(buf + n, sizeof(buf) - n, "%d,", online[i]);
+ snprintf(buf + n, sizeof(buf) - n, "%d", d);
+ err = write_file(cs_q, "cpuset.cpus.partition", "member");
+ if (err || make_partition_cpus(cs_q, buf)) {
+ ksft_print_msg("Q cannot be a partition root of CPUs %s\n", buf);
+ partition_has(cs_q, "root\n", 1);
+ goto out;
+ }
+ q = total_bw_of(d);
+ h = total_bw_of(c);
+ if (fallback_wake(fd[1], pid))
+ goto out;
+ /* Go on after a failure: the totals tell more. */
+ cpu = task_cpu_of(pid);
+ if (cpu != c)
+ ksft_print_msg("woken on CPU %d, not %d\n", cpu, c);
+ else if (!totals_check("woken", d, c, q, h))
+ ret = KSFT_PASS;
+ reap(pid);
+ pid = -1;
+ sleep_ns(3 * PERIOD);
+ if (totals_of(d, c, &q0, &h0)) {
+ ret = KSFT_FAIL;
+ goto out;
+ }
+ if (q != q0 || h != h0 + bw) {
+ ksft_print_msg("Q counts %lld of %lld, the root cgroup %lld\n",
+ q - q0, bw, h - h0);
+ ret = KSFT_FAIL;
+ }
+out:
+ reap(pid);
+ if (fd[0] >= 0) {
+ close(fd[0]);
+ close(fd[1]);
+ }
+ cpu_up(a);
+ cpu_up(b);
+ cpu_up(d);
+ cs_cleanup();
+ return ret;
+}
+
/* Moves between two root domains, from the main process, with P */
/*
@@ -7828,8 +9738,32 @@ static const struct hp_case {
{ "hotplug_refused", hotplug_refused },
{ "hotplug_departed", hotplug_departed },
{ "hotplug_server", hotplug_server },
+ { "hotplug_server_refused", hotplug_server_refused },
+ { "hotplug_fallback", hotplug_fallback },
+ { "hotplug_fallback_running", hotplug_fallback_running },
+ { "hotplug_fallback_moved", hotplug_fallback_moved },
+ { "hotplug_fallback_in_mask", hotplug_fallback_in_mask },
+ { "mask_out_refused", mask_out_refused, 1 },
+ { "hotplug_last_refused", hotplug_last_refused },
+ { "hotplug_keeps_root_domain", hotplug_keeps_root_domain },
+ { "partition_takes_refused", partition_takes_refused, 1 },
+ { "no_admission_affinity_moves", no_admission_affinity_moves, 1 },
{ "affinity_vs_setattr", affinity_vs_setattr, 1 },
{ "move_back_cpu_offline", move_back_cpu_offline },
+ { "hotplug_spinner_keeps", hotplug_spinner_keeps },
+ { "hotplug_waking_keeps", hotplug_waking_keeps },
+ { "hotplug_isolated_keeps", hotplug_isolated_keeps },
+ { "rebuild_waking_keeps", rebuild_waking_keeps, 1 },
+ { "hotplug_user_mask_refused", hotplug_user_mask_refused },
+ { "hotplug_nested_refused", hotplug_nested_refused },
+ { "hotplug_dead_accepted", hotplug_dead_accepted },
+ { "hotplug_child_partition_kept", hotplug_child_partition_kept },
+ { "hotplug_partition_invalid_refused", hotplug_partition_invalid_refused },
+ { "partition_takes_fallback_refused", partition_takes_fallback_refused, 1 },
+ { "mask_grows_refused", mask_grows_refused, 1 },
+ { "partition_swap_refused", partition_swap_refused, 1 },
+ { "partition_refused_no_warn", partition_refused_no_warn, 1 },
+ { "rebuild_record_taken", rebuild_record_taken },
};
#define N_HP_CASES ((int)ARRAY_SIZE(hp_cases))
--
2.47.3
^ permalink raw reply related [flat|nested] 35+ messages in thread
* [PATCH v8 19/33] sched/deadline: Count only active CPUs when spreading extra_bw
2026-11-19 20:15 [PATCH v8 00/33] sched/deadline: bandwidth accounting fixes and selftests Mathieu Desnoyers
` (17 preceding siblings ...)
2026-11-19 20:15 ` [PATCH v8 18/33] selftests/sched: Test the root domain of a task in deadline_bw_test Mathieu Desnoyers
@ 2026-11-19 20:15 ` Mathieu Desnoyers
2026-11-19 20:15 ` [PATCH v8 20/33] rtmutex: Continue the chain walk when a waiter's deadline donor changes Mathieu Desnoyers
` (13 subsequent siblings)
32 siblings, 0 replies; 35+ messages in thread
From: Mathieu Desnoyers @ 2026-11-19 20:15 UTC (permalink / raw)
To: juri.lelli, peterz, mingo, vincent.guittot
Cc: Mathieu Desnoyers, tglx, will, boqun, dietmar.eggemann, rostedt,
bsegall, mgorman, vschneid, kprateek.nayak, luca.abeni,
mathieu.poirier, longman, ridong.chen, tj, hannes, mkoutny, shuah,
bigeasy, clrkwllms, parri.andrea, m.szyprowski, kayracizmeci,
weihengqiuu, linux-kernel, cgroups, linux-kselftest,
linux-rt-devel
__dl_add() and __dl_sub() change the total_bw of a root domain. They
also spread the change over the extra_bw of its CPUs, which GRUB uses
for SCHED_FLAG_RECLAIM tasks. __dl_update() changes only the active
CPUs, so the cpus argument must count those. Most callers pass
dl_bw_cpus(), which does.
dl_task_offline_migration() and dl_add_task_root_domain() pass
cpumask_weight(rd->span) instead. That also counts the inactive CPUs of
the span. A span holds one while that CPU goes offline, which is when
dl_task_offline_migration() runs. Each active CPU then gets a smaller
share of the change than it should. A later change made with
dl_bw_cpus() does not undo it exactly. extra_bw stays off until the
next rebuild of the root domains resets it. SCHED_FLAG_RECLAIM tasks
would reclaim a little more or less than their share meanwhile. This
was found by reading.
Fix this by passing dl_bw_cpus() of the CPU whose root domain changes.
Fixes: 59d06cea1198 ("sched/deadline: Fix bandwidth accounting at all levels after offline migration")
Fixes: f9a25f776d78 ("cpusets: Rebuild root domain deadline accounting information")
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
---
kernel/sched/deadline.c | 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)
diff --git a/kernel/sched/deadline.c b/kernel/sched/deadline.c
index 641674f65931..914915abb912 100644
--- a/kernel/sched/deadline.c
+++ b/kernel/sched/deadline.c
@@ -1016,7 +1016,7 @@ static struct rq *dl_task_offline_migration(struct rq *rq, struct task_struct *p
dl_b = &src_rq->rd->dl_bw;
raw_spin_lock(&dl_b->lock);
__dl_sub(dl_b, p->dl.dl_bw,
- cpumask_weight(src_rq->rd->span));
+ dl_bw_cpus(cpu_of(src_rq)));
raw_spin_unlock(&dl_b->lock);
}
/*
@@ -1028,7 +1028,7 @@ static struct rq *dl_task_offline_migration(struct rq *rq, struct task_struct *p
dl_b = &later_rq->rd->dl_bw;
raw_spin_lock(&dl_b->lock);
__dl_add(dl_b, p->dl.dl_bw,
- cpumask_weight(later_rq->rd->span));
+ dl_bw_cpus(cpu_of(later_rq)));
raw_spin_unlock(&dl_b->lock);
p->dl.dl_bw_cpu = later_rq->cpu;
p->dl.dl_bw_gen = dl_bw_gen;
@@ -4217,7 +4217,7 @@ void dl_add_task_root_domain(struct task_struct *p)
dl_b = &rq->rd->dl_bw;
raw_spin_lock(&dl_b->lock);
- __dl_add(dl_b, p->dl.dl_bw, cpumask_weight(rq->rd->span));
+ __dl_add(dl_b, p->dl.dl_bw, dl_bw_cpus(cpu));
raw_spin_unlock(&dl_b->lock);
p->dl.dl_bw_cpu = cpu;
p->dl.dl_bw_gen = dl_bw_gen;
--
2.47.3
^ permalink raw reply related [flat|nested] 35+ messages in thread
* [PATCH v8 20/33] rtmutex: Continue the chain walk when a waiter's deadline donor changes
2026-11-19 20:15 [PATCH v8 00/33] sched/deadline: bandwidth accounting fixes and selftests Mathieu Desnoyers
` (18 preceding siblings ...)
2026-11-19 20:15 ` [PATCH v8 19/33] sched/deadline: Count only active CPUs when spreading extra_bw Mathieu Desnoyers
@ 2026-11-19 20:15 ` Mathieu Desnoyers
2026-11-19 20:16 ` [PATCH v8 21/33] rtmutex: Continue the walk when the owner's top waiter changes lock Mathieu Desnoyers
` (12 subsequent siblings)
32 siblings, 0 replies; 35+ messages in thread
From: Mathieu Desnoyers @ 2026-11-19 20:15 UTC (permalink / raw)
To: juri.lelli, peterz, mingo, vincent.guittot
Cc: Mathieu Desnoyers, tglx, will, boqun, dietmar.eggemann, rostedt,
bsegall, mgorman, vschneid, kprateek.nayak, luca.abeni,
mathieu.poirier, longman, ridong.chen, tj, hannes, mkoutny, shuah,
bigeasy, clrkwllms, parri.andrea, m.szyprowski, kayracizmeci,
weihengqiuu, linux-kernel, cgroups, linux-kselftest,
linux-rt-devel, stable
A task boosted by a SCHED_DEADLINE waiter runs on the parameters of a
donor, dl.pi_se. Commit 2279f540ea7d ("sched/deadline: Fix priority
inheritance with multiple scheduling classes") made the donor follow the
chain. rt_mutex_setprio() copies the donor of the top waiter to the
owner. So the owner at the end of a chain points at the deadline task at
its head. That task waits on no lock the owner holds.
The chain walk stops at a task whose waiter has the same sort keys as
the task: the priority and the deadline. Then "there is no further
priority adjustment necessary". Before 2279f540ea7d, this was enough.
The refill read the parameters of the owner's own top waiter, which is
blocked and so alive.
But a boost can change the donor of a blocked task and leave its keys
as they were. The task keeps a deadline priority. It sleeps, so no
refill moves its deadline. The walk then stops one lock too early.
M and O are SCHED_OTHER tasks. W and Z are SCHED_DEADLINE tasks. Z has
the earlier deadline. L1, L1' and L2 are rt_mutexes:
O M W Z
locks L2 locks L1 and L1'
blocks on L1'
rt_mutex_setprio(M, W)
M->dl.pi_se = &W->dl
blocks on L2
rt_mutex_setprio(O, M)
O->dl.pi_se = &W->dl
blocks on L1
rt_mutex_setprio(M, Z)
M->dl.pi_se = &Z->dl
chain walk at M:
keys equal, stops
times out on L1'
exits
O runs on W's parameters, with a later deadline than Z's, while Z waits
for it. After W exits, O's next refill would read W's freed task_struct.
A deboost has the same effect. Take a deadline task that a waiter with
an earlier deadline boosts, and that blocks on a second lock. It keeps
its own deadline. When that waiter leaves, rt_mutex_setprio() makes the
task its own donor again. Its keys stay equal. A policy change in
__setscheduler_dl_pi() can also change the donor of a blocked task with
no change of keys. rt_mutex_adjust_pi() then skips the walk.
Setting up the deadline tasks needs CAP_SYS_NICE. The timeout of a
FUTEX_LOCK_PI needs no privilege. The Sashiko AI review of this series
found the deboost case.
Fix this by recording the donor in the waiter with the sort keys, in
waiter_update_prio(). rt_waiter_up_to_date() compares both. The walk and
rt_mutex_adjust_pi() use it. A new donor then takes the walk on to the
next owner, and rt_mutex_setprio() gives that owner the new donor. The
waiter's copy is only compared, never dereferenced. A task without a
deadline priority is always its own donor, so its walks do not change.
A known limit remains. Two walks can overlap on a chain of three locks
or more. The owners further down then keep the old donor until the
first walk reaches them.
The case pi_chain_donor_leaves of deadline_bw_test tests the missed
boost. W leaves SCHED_DEADLINE there, in place of exiting. It passes on
v7.4-rc2, and without this patch. The case pi_chain_top_moves, added to
deadline_bw_test later in this series, needs this patch and the next
one. If one of the two is missing, another deadline task gets 30ms of
200ms in every run. With both, it passes.
Fixes: 2279f540ea7d ("sched/deadline: Fix priority inheritance with multiple scheduling classes")
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Cc: stable@vger.kernel.org # v5.15+
---
kernel/locking/rtmutex.c | 22 ++++++++++++++++++++--
kernel/locking/rtmutex_api.c | 2 +-
kernel/locking/rtmutex_common.h | 3 +++
3 files changed, 24 insertions(+), 3 deletions(-)
diff --git a/kernel/locking/rtmutex.c b/kernel/locking/rtmutex.c
index 5a9534c715b8..c78dfe9c93fc 100644
--- a/kernel/locking/rtmutex.c
+++ b/kernel/locking/rtmutex.c
@@ -369,10 +369,12 @@ static __always_inline void
waiter_update_prio(struct rt_mutex_waiter *waiter, struct task_struct *task)
{
lockdep_assert_held(&waiter->lock->wait_lock);
+ lockdep_assert_held(&task->pi_lock);
lockdep_assert(RB_EMPTY_NODE(&waiter->tree.entry));
waiter->tree.prio = __waiter_prio(task);
waiter->tree.deadline = task->dl.deadline;
+ waiter->pi_se = task->dl.pi_se;
}
/*
@@ -433,6 +435,21 @@ static __always_inline int rt_waiter_node_equal(struct rt_waiter_node *left,
return 1;
}
+/*
+ * Return true if @waiter is up to date with its task @task. Compare the
+ * sort keys and the deadline donor, which rt_mutex_setprio() copies from
+ * @task to the lock owner. A boost or a deboost of @task can change its
+ * donor and leave its keys as they were. The owner must then get the new
+ * donor: the old one can exit. The caller holds @task->pi_lock, which
+ * protects @task->dl.pi_se. @waiter->pi_se is never dereferenced.
+ */
+static __always_inline bool
+rt_waiter_up_to_date(struct rt_mutex_waiter *waiter, struct task_struct *task)
+{
+ return rt_waiter_node_equal(&waiter->tree, task_to_waiter_node(task)) &&
+ waiter->pi_se == task->dl.pi_se;
+}
+
static inline bool rt_mutex_steal(struct rt_mutex_waiter *waiter,
struct rt_mutex_waiter *top_waiter)
{
@@ -824,12 +841,13 @@ static int __sched rt_mutex_adjust_prio_chain(struct task_struct *task,
/*
* If the waiter priority is the same as the task priority
- * then there is no further priority adjustment necessary. If
+ * then there is no further priority adjustment necessary. The
+ * task must also pass on the same deadline donor. If
* deadlock detection is off, we stop the chain walk. If its
* enabled we continue, but stop the requeueing in the chain
* walk.
*/
- if (rt_waiter_node_equal(&waiter->tree, task_to_waiter_node(task))) {
+ if (rt_waiter_up_to_date(waiter, task)) {
if (!detect_deadlock)
goto out_unlock_pi;
else
diff --git a/kernel/locking/rtmutex_api.c b/kernel/locking/rtmutex_api.c
index eb18b094473c..6bdbd36bac94 100644
--- a/kernel/locking/rtmutex_api.c
+++ b/kernel/locking/rtmutex_api.c
@@ -510,7 +510,7 @@ void __sched rt_mutex_adjust_pi(struct task_struct *task)
raw_spin_lock_irqsave(&task->pi_lock, flags);
waiter = task->pi_blocked_on;
- if (!waiter || rt_waiter_node_equal(&waiter->tree, task_to_waiter_node(task))) {
+ if (!waiter || rt_waiter_up_to_date(waiter, task)) {
raw_spin_unlock_irqrestore(&task->pi_lock, flags);
return;
}
diff --git a/kernel/locking/rtmutex_common.h b/kernel/locking/rtmutex_common.h
index c38b7bdea7b3..c0f391c352e3 100644
--- a/kernel/locking/rtmutex_common.h
+++ b/kernel/locking/rtmutex_common.h
@@ -43,6 +43,8 @@ struct rt_waiter_node {
* @pi_tree: node to enqueue into the mutex owner waiters tree
* @task: task reference to the blocked task
* @lock: Pointer to the rt_mutex on which the waiter blocks
+ * @pi_se: The deadline donor of @task when @tree was last updated.
+ * Only compared, never dereferenced.
* @wake_state: Wakeup state to use (TASK_NORMAL or TASK_RTLOCK_WAIT)
* @ww_ctx: WW context pointer
*
@@ -54,6 +56,7 @@ struct rt_mutex_waiter {
struct rt_waiter_node pi_tree;
struct task_struct *task;
struct rt_mutex_base *lock;
+ struct sched_dl_entity *pi_se;
unsigned int wake_state;
struct ww_acquire_ctx *ww_ctx;
};
--
2.47.3
^ permalink raw reply related [flat|nested] 35+ messages in thread
* [PATCH v8 21/33] rtmutex: Continue the walk when the owner's top waiter changes lock
2026-11-19 20:15 [PATCH v8 00/33] sched/deadline: bandwidth accounting fixes and selftests Mathieu Desnoyers
` (19 preceding siblings ...)
2026-11-19 20:15 ` [PATCH v8 20/33] rtmutex: Continue the chain walk when a waiter's deadline donor changes Mathieu Desnoyers
@ 2026-11-19 20:16 ` Mathieu Desnoyers
2026-11-19 20:16 ` [PATCH v8 22/33] selftests/sched: Test the rtmutex chain walk in deadline_bw_test Mathieu Desnoyers
` (11 subsequent siblings)
32 siblings, 0 replies; 35+ messages in thread
From: Mathieu Desnoyers @ 2026-11-19 20:16 UTC (permalink / raw)
To: juri.lelli, peterz, mingo, vincent.guittot
Cc: Mathieu Desnoyers, tglx, will, boqun, dietmar.eggemann, rostedt,
bsegall, mgorman, vschneid, kprateek.nayak, luca.abeni,
mathieu.poirier, longman, ridong.chen, tj, hannes, mkoutny, shuah,
bigeasy, clrkwllms, parri.andrea, m.szyprowski, kayracizmeci,
weihengqiuu, linux-kernel, cgroups, linux-kselftest,
linux-rt-devel, stable
The chain walk stops at an owner whose top pi waiter is not the top
waiter of the lock just walked. That lock then does not set the
owner's priority, so there is nothing to pass on.
But the step before can change the owner's top pi waiter. It
requeues a waiter whose priority dropped. The top pi waiter of the
owner can then be a waiter on another lock. The owner's priority, or
its deadline donor, changed. The walk still stops at it, and the next
owner keeps the old ones.
O and M are SCHED_OTHER tasks. W and Z are SCHED_DEADLINE tasks. W
has the earlier deadline. X is a SCHED_FIFO task:
O M W Z X
locks L2 locks L1 and L1'
blocks on L1'
M->dl.pi_se = &W->dl
blocks on L2
O->dl.pi_se = &W->dl
blocks on L1
sched_setscheduler(SCHED_OTHER)
rt_mutex_adjust_pi(W)
M->dl.pi_se = &Z->dl
walk at M: L1' is not M's top
stops
blocks on L1'
walk stops at M
times out on L1'
not the top waiter: no walk
exits
O's next refill reads W's freed task_struct. With RT priorities
only, O stays boosted to W's old priority until it unlocks L2.
Remove this test. The next test already stops the walk when the
owner's waiter is up to date. An out of date waiter is now requeued,
and the walk goes on to the next owner.
The cases pi_chain_top_moves and pi_chain_top_moves_rt, added to
deadline_bw_test later in this series, test this. pi_chain_top_moves_rt
fails without this patch and passes with it. pi_chain_top_moves also
needs the previous patch.
The Sashiko AI review of this series found this.
Fixes: 23f78d4a03c5 ("[PATCH] pi-futex: rt mutex core")
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Cc: stable@vger.kernel.org
---
kernel/locking/rtmutex.c | 24 ++++++++----------------
1 file changed, 8 insertions(+), 16 deletions(-)
diff --git a/kernel/locking/rtmutex.c b/kernel/locking/rtmutex.c
index c78dfe9c93fc..c06261259a56 100644
--- a/kernel/locking/rtmutex.c
+++ b/kernel/locking/rtmutex.c
@@ -821,23 +821,15 @@ static int __sched rt_mutex_adjust_prio_chain(struct task_struct *task,
* Drop out, when the task has no waiters. Note,
* top_waiter can be NULL, when we are in the deboosting
* mode!
+ *
+ * Do not compare top_waiter with the top pi waiter of the
+ * task here. The previous step can have made a waiter on
+ * another lock the top pi waiter of the task. The priority of
+ * the task changed then, and the next owner must follow. The
+ * test below tells whether there is something to pass on.
*/
- if (top_waiter) {
- if (!task_has_pi_waiters(task))
- goto out_unlock_pi;
- /*
- * If deadlock detection is off, we stop here if we
- * are not the top pi waiter of the task. If deadlock
- * detection is enabled we continue, but stop the
- * requeueing in the chain walk.
- */
- if (top_waiter != task_top_pi_waiter(task)) {
- if (!detect_deadlock)
- goto out_unlock_pi;
- else
- requeue = false;
- }
- }
+ if (top_waiter && !task_has_pi_waiters(task))
+ goto out_unlock_pi;
/*
* If the waiter priority is the same as the task priority
--
2.47.3
^ permalink raw reply related [flat|nested] 35+ messages in thread
* [PATCH v8 22/33] selftests/sched: Test the rtmutex chain walk in deadline_bw_test
2026-11-19 20:15 [PATCH v8 00/33] sched/deadline: bandwidth accounting fixes and selftests Mathieu Desnoyers
` (20 preceding siblings ...)
2026-11-19 20:16 ` [PATCH v8 21/33] rtmutex: Continue the walk when the owner's top waiter changes lock Mathieu Desnoyers
@ 2026-11-19 20:16 ` Mathieu Desnoyers
2026-11-19 20:16 ` [PATCH v8 23/33] sched/deadline: Reset the donor's runtime at a deboost in the class Mathieu Desnoyers
` (10 subsequent siblings)
32 siblings, 0 replies; 35+ messages in thread
From: Mathieu Desnoyers @ 2026-11-19 20:16 UTC (permalink / raw)
To: juri.lelli, peterz, mingo, vincent.guittot
Cc: Mathieu Desnoyers, tglx, will, boqun, dietmar.eggemann, rostedt,
bsegall, mgorman, vschneid, kprateek.nayak, luca.abeni,
mathieu.poirier, longman, ridong.chen, tj, hannes, mkoutny, shuah,
bigeasy, clrkwllms, parri.andrea, m.szyprowski, kayracizmeci,
weihengqiuu, linux-kernel, cgroups, linux-kselftest,
linux-rt-devel
Add pi_chain_top_moves and pi_chain_top_moves_rt to deadline_bw_test.
They test the two rtmutex fixes earlier in this series. In both, a
chain walk changes the top waiter of an owner to a waiter of another
lock it holds. The walk must go on to the next owner. That owner must
then get the new donor, or the new priority. The first case uses
deadline tasks, the second RT tasks. The comment above each
case gives its scenario.
On v7.4-rc2, in 8 VMs of 8 CPUs, pi_chain_top_moves_rt fails in every
VM.
pi_chain_top_moves passes there. It fails with the first series, until
"rtmutex: Continue the chain walk when a waiter's deadline donor
changes" and "rtmutex: Continue the walk when the owner's top waiter
changes lock". It needs both.
With the fixes up to here, both cases pass wherever they run.
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
---
.../sched/deadline/deadline_bw_test.c | 293 ++++++++++++++++++
1 file changed, 293 insertions(+)
diff --git a/tools/testing/selftests/sched/deadline/deadline_bw_test.c b/tools/testing/selftests/sched/deadline/deadline_bw_test.c
index a290e332831d..c83e0272771f 100644
--- a/tools/testing/selftests/sched/deadline/deadline_bw_test.c
+++ b/tools/testing/selftests/sched/deadline/deadline_bw_test.c
@@ -2521,6 +2521,297 @@ static int pi_chain_donor_leaves(void)
return ret;
}
+/*
+ * The threads of top_moves(): O, M, W, Z and Y. top_tid[] holds their
+ * tids. top_lock is L1'. top_y_ns is the CPU time Y gets in O's window,
+ * or -1.
+ */
+enum { TM_O, TM_M, TM_W, TM_Z, TM_Y, TM_N };
+static _Atomic pid_t top_tid[TM_N];
+static pthread_mutex_t top_lock;
+static atomic_int top_err, top_rt, top_m_locking, top_w_locking, top_y_stop;
+static _Atomic long long top_y_ns;
+static sem_t top_o_go, top_m_go;
+
+/* The priority /proc shows for a SCHED_FIFO task of priority @p. */
+#define FIFO_TASK_PRIO(p) (-1 - (p))
+
+/*
+ * O: take pi_lock2 (L2), and sleep until top_o_go. With Y (@arg), spin
+ * 200ms and measure the CPU time Y gets meanwhile. Then unlock L2.
+ */
+static void *top_o(void *arg)
+{
+ uint64_t t0, y0;
+
+ if (pthread_mutex_lock(&pi_lock2)) {
+ top_err = 1;
+ return arg;
+ }
+ top_tid[TM_O] = gettid_(); /* O holds L2 */
+ sem_wait_sig(&top_o_go);
+ if (arg) {
+ y0 = cpu_time_of(*(pthread_t *)arg);
+ t0 = now_ns();
+ while (now_ns() - t0 < 2 * PERIOD)
+ ;
+ top_y_ns = cpu_time_of(*(pthread_t *)arg) - y0;
+ }
+ pthread_mutex_unlock(&pi_lock2);
+ return NULL;
+}
+
+/*
+ * M: take pi_lock (L1) and top_lock (L1'). At top_m_go, block on L2,
+ * which O holds. Unlock L2, L1 and L1' once it gets L2.
+ */
+static void *top_m(void *arg)
+{
+ if (pthread_mutex_lock(&pi_lock)) {
+ top_err = 1;
+ return arg;
+ }
+ if (pthread_mutex_lock(&top_lock)) {
+ pthread_mutex_unlock(&pi_lock);
+ top_err = 1;
+ return arg;
+ }
+ top_tid[TM_M] = gettid_(); /* M holds L1 and L1' */
+ sem_wait_sig(&top_m_go);
+ top_m_locking = 1;
+ if (pthread_mutex_lock(&pi_lock2))
+ top_err = 1;
+ else
+ pthread_mutex_unlock(&pi_lock2);
+ pthread_mutex_unlock(&pi_lock);
+ pthread_mutex_unlock(&top_lock);
+ return arg;
+}
+
+/*
+ * W and Z: W blocks on L1', Z on L1. @arg is 1 for W. With top_rt, W
+ * becomes SCHED_FIFO 50 and Z SCHED_FIFO 20. Otherwise W becomes a
+ * deadline task of 2ms every 80ms and Z one of 5ms every 200ms. The main
+ * thread switches W to SCHED_OTHER while it waits.
+ */
+static void *top_waiter(void *arg)
+{
+ int w = (long)arg;
+ struct sched_param sp = { .sched_priority = w ? 50 : 20 };
+ uint64_t runtime = (w ? 2 : 5) * NSEC_PER_MSEC;
+ pthread_mutex_t *l = w ? &top_lock : &pi_lock;
+
+ if (top_rt ? sched_setscheduler(0, SCHED_FIFO, &sp) :
+ set_deadline(0, runtime, 40 * runtime)) {
+ top_err = 1;
+ return arg;
+ }
+ top_tid[w ? TM_W : TM_Z] = gettid_();
+ if (w)
+ top_w_locking = 1;
+ if (pthread_mutex_lock(l))
+ top_err = 1;
+ else
+ pthread_mutex_unlock(l);
+ if (set_other(0))
+ top_err = 1;
+ return arg;
+}
+
+/* Y: become a deadline task of 30ms every 100ms, and spin. */
+static void *top_y(void *arg)
+{
+ if (set_deadline(0, 30 * NSEC_PER_MSEC, PERIOD)) {
+ top_err = 1;
+ return arg;
+ }
+ top_tid[TM_Y] = gettid_();
+ while (!top_y_stop)
+ ;
+ return arg;
+}
+
+/*
+ * A chain walk must go on past a task whose top waiter moves to another
+ * lock. The priority of that task changes, and the next owner must
+ * follow.
+ *
+ * L1 is pi_lock, L1' is top_lock and L2 is pi_lock2. All threads run on
+ * the CPU of P. W and Z set their own policy when they start. With @rt,
+ * W is SCHED_FIFO 50 and Z SCHED_FIFO 20. Without @rt, W is a deadline
+ * task of 2ms every 80ms and Z one of 5ms every 200ms. W has the earlier
+ * deadline. O and M are SCHED_OTHER. Without @rt, Y is a deadline task
+ * of 30ms every 100ms. It spins all along.
+ *
+ * 1. O takes L2, and sleeps.
+ * 2. M takes L1 and L1', and sleeps.
+ * 3. W blocks on L1'. W boosts M.
+ * 4. M blocks on L2. M passes W on to O.
+ * 5. Z blocks on L1. W stays the top waiter of M.
+ * 6. The main thread switches W to SCHED_OTHER. The chain walk requeues
+ * W on L1'. Z on L1 becomes the top waiter of M, and Z boosts M. The
+ * walk must go on to O. Z must boost O too.
+ *
+ * With @rt, the case reads O's priority. It must be the one of Z. With
+ * the walk stopped at M, O keeps the priority of W.
+ *
+ * Without @rt, W sleeps and has run almost nothing. So its parameters
+ * are cleared at once. 250ms later, the main thread wakes O up. O
+ * measures the CPU time Y gets over the next 200ms, and unlocks L2.
+ *
+ * With Z as its donor, O gets 5ms every 200ms. Y runs at the start of
+ * each of its periods, and gets its 60ms. With W as its donor, O runs on
+ * cleared parameters. Each refill gives it no runtime and a deadline of
+ * about now. O then runs before Y until Y's deadline is past. Y runs
+ * only then, one period late, and gets about 30ms. The case fails if Y
+ * gets less than 45ms.
+ *
+ * W does not exit before O unlocks L2. It waits for L1', which M
+ * releases after it gets L2.
+ */
+static int top_moves(int rt)
+{
+ int ret = KSFT_FAIL, i, have[TM_N] = { 0 };
+ int boost = rt ? FIFO_TASK_PRIO(50) : DL_TASK_PRIO;
+ pthread_mutexattr_t a;
+ pthread_t t[TM_N];
+ long long y_ns;
+ int err, o_prio = 0;
+
+ if (pi_init())
+ return KSFT_SKIP;
+ if (pi_init_lock2()) {
+ pi_fini();
+ return KSFT_SKIP;
+ }
+ err = pthread_mutexattr_init(&a);
+ if (!err) {
+ err = pthread_mutexattr_setprotocol(&a, PTHREAD_PRIO_INHERIT);
+ if (!err)
+ err = pthread_mutex_init(&top_lock, &a);
+ pthread_mutexattr_destroy(&a);
+ }
+ if (err) {
+ ksft_print_msg("no third priority inheritance mutex: %s\n",
+ strerror(err));
+ ret = KSFT_SKIP;
+ goto out_lock2;
+ }
+ if (sem_init(&top_o_go, 0, 0) || sem_init(&top_m_go, 0, 0)) {
+ ksft_print_msg("sem_init: %s\n", strerror(errno));
+ goto out_lock3;
+ }
+ for (i = 0; i < TM_N; i++)
+ top_tid[i] = 0;
+ top_err = 0;
+ top_rt = rt;
+ top_m_locking = 0;
+ top_w_locking = 0;
+ top_y_stop = 0;
+ top_y_ns = -1;
+ if (!rt) {
+ have[TM_Y] = !pthread_create(&t[TM_Y], NULL, top_y, NULL);
+ if (!have[TM_Y] || wait_for(1000, top_tid[TM_Y] || top_err)) {
+ ksft_print_msg("Y never became a deadline task\n");
+ goto out;
+ }
+ }
+ have[TM_O] = !pthread_create(&t[TM_O], NULL, top_o,
+ rt ? NULL : &t[TM_Y]);
+ if (!have[TM_O] || wait_for(1000, top_tid[TM_O] || top_err)) {
+ ksft_print_msg("O never took L2\n");
+ goto out;
+ }
+ have[TM_M] = !pthread_create(&t[TM_M], NULL, top_m, NULL);
+ if (!have[TM_M] || wait_for(1000, top_tid[TM_M] || top_err)) {
+ ksft_print_msg("M never took L1 and L1'\n");
+ goto out;
+ }
+ have[TM_W] = !pthread_create(&t[TM_W], NULL, top_waiter, (void *)1L);
+ if (!have[TM_W] ||
+ wait_for(1000, (top_w_locking &&
+ proc_state_of(top_tid[TM_W]) == 'S') || top_err)) {
+ ksft_print_msg("W never blocked on L1'\n");
+ goto out;
+ }
+ sem_post(&top_m_go);
+ if (wait_for(1000, (top_m_locking && prio_of(top_tid[TM_O]) == boost) ||
+ top_err)) {
+ ksft_print_msg("O was not boosted through M\n");
+ goto out;
+ }
+ have[TM_Z] = !pthread_create(&t[TM_Z], NULL, top_waiter, (void *)0L);
+ if (!have[TM_Z] ||
+ wait_for(1000, (top_tid[TM_Z] &&
+ proc_state_of(top_tid[TM_Z]) == 'S') || top_err)) {
+ ksft_print_msg("Z never blocked on L1\n");
+ goto out;
+ }
+ sleep_ns(10 * NSEC_PER_MSEC); /* the chain walk of Z is done */
+ err = top_err ? -EINVAL : set_other(top_tid[TM_W]);
+ if (err) {
+ ksft_print_msg("W cannot leave its class: %s\n",
+ strerror(-err));
+ goto out;
+ }
+ if (rt) {
+ sleep_ns(10 * NSEC_PER_MSEC);
+ if (prio_of(top_tid[TM_M]) != FIFO_TASK_PRIO(20)) {
+ ksft_print_msg("Z does not boost M\n");
+ goto out;
+ }
+ o_prio = prio_of(top_tid[TM_O]);
+ } else {
+ sleep_ns(250 * NSEC_PER_MSEC);
+ }
+ ret = KSFT_PASS;
+out:
+ /* Let every thread end: O unlocks L2, then M unlocks L1 and L1'. */
+ sem_post(&top_m_go);
+ sem_post(&top_o_go);
+ for (i = TM_O; i < TM_Y; i++)
+ if (have[i])
+ join(t[i]);
+ top_y_stop = 1;
+ if (have[TM_Y])
+ join(t[TM_Y]);
+ y_ns = top_y_ns;
+ if (top_err) {
+ ksft_print_msg("a thread of the chain failed\n");
+ ret = KSFT_FAIL;
+ } else if (ret == KSFT_PASS && rt && o_prio != FIFO_TASK_PRIO(20)) {
+ ksft_print_msg("O runs at priority %d, not %d: W still boosts it\n",
+ o_prio, FIFO_TASK_PRIO(20));
+ ret = KSFT_FAIL;
+ } else if (ret == KSFT_PASS && !rt &&
+ y_ns < 45 * (long long)NSEC_PER_MSEC) {
+ ksft_print_msg("Y ran %lld ms of 200: W is still the donor of O\n",
+ y_ns / (long long)NSEC_PER_MSEC);
+ ret = KSFT_FAIL;
+ }
+ sem_destroy(&top_o_go);
+ sem_destroy(&top_m_go);
+out_lock3:
+ pthread_mutex_destroy(&top_lock);
+out_lock2:
+ pthread_mutex_destroy(&pi_lock2);
+ sem_destroy(&holder_go);
+ pi_fini();
+ return ret;
+}
+
+/* W and Z are deadline tasks. */
+static int pi_chain_top_moves(void)
+{
+ return top_moves(0);
+}
+
+/* W and Z are SCHED_FIFO tasks. */
+static int pi_chain_top_moves_rt(void)
+{
+ return top_moves(1);
+}
+
/* cpuset moves, from the main process, outside P */
/* Shared with the processes of a case: atomic in shared memory. */
@@ -9674,6 +9965,8 @@ static const struct p_case {
{ "return_asleep_new_params", return_asleep_new_params, MEASURED_IN_P },
{ "deboost_blocked_keeps_order", deboost_blocked_keeps_order, MEASURED_IN_P },
{ "pi_chain_donor_leaves", pi_chain_donor_leaves, MEASURED_IN_P },
+ { "pi_chain_top_moves", pi_chain_top_moves, MEASURED_IN_P },
+ { "pi_chain_top_moves_rt", pi_chain_top_moves_rt, MEASURED_IN_P },
{ "move_asleep", move_asleep, FROM_OUTSIDE },
{ "move_throttled", move_throttled, FROM_OUTSIDE },
{ "move_non_contending", move_non_contending, FROM_OUTSIDE },
--
2.47.3
^ permalink raw reply related [flat|nested] 35+ messages in thread
* [PATCH v8 23/33] sched/deadline: Reset the donor's runtime at a deboost in the class
2026-11-19 20:15 [PATCH v8 00/33] sched/deadline: bandwidth accounting fixes and selftests Mathieu Desnoyers
` (21 preceding siblings ...)
2026-11-19 20:16 ` [PATCH v8 22/33] selftests/sched: Test the rtmutex chain walk in deadline_bw_test Mathieu Desnoyers
@ 2026-11-19 20:16 ` Mathieu Desnoyers
2026-11-19 20:16 ` [PATCH v8 24/33] sched/deadline: Replenish a throttled task boosted while asleep Mathieu Desnoyers
` (9 subsequent siblings)
32 siblings, 0 replies; 35+ messages in thread
From: Mathieu Desnoyers @ 2026-11-19 20:16 UTC (permalink / raw)
To: juri.lelli, peterz, mingo, vincent.guittot
Cc: Mathieu Desnoyers, tglx, will, boqun, dietmar.eggemann, rostedt,
bsegall, mgorman, vschneid, kprateek.nayak, luca.abeni,
mathieu.poirier, longman, ridong.chen, tj, hannes, mkoutny, shuah,
bigeasy, clrkwllms, parri.andrea, m.szyprowski, kayracizmeci,
weihengqiuu, linux-kernel, cgroups, linux-kselftest,
linux-rt-devel, stable
A task boosted by a deadline waiter, its donor, runs on the donor's
reservation. Commit 2d3d891d3344 ("sched/deadline: Add SCHED_DEADLINE
inheritance logic") uses the waiter's parameters "during the whole
period it holds the pi-lock".
A boosted task that runs out of runtime is not throttled.
replenish_dl_entity() refills it from the waiter's parameters: it adds
the waiter's runtime, and moves the deadline by the waiter's period.
From then on, the runtime and the deadline of the task are not those of
its own reservation.
A task that leaves the class at its deboost does not keep them.
"sched/deadline: Reset the deadline and the runtime with the parameters"
resets them for a task with no parameters. For a task with parameters,
they are cleared with the parameters at its 0-lag time.
But the boosted task can be a SCHED_DEADLINE task itself, when the
waiter has an earlier deadline. It stays in the class at its deboost.
rt_mutex_setprio() only resets dl.pi_se. enqueue_dl_entity() keeps a
deadline that is ahead. So a task that was refilled keeps the waiter's
runtime and deadline after the unlock. A task that sched_setattr() makes
SCHED_DEADLINE while boosted ends its boost the same way.
In one period, the task can run what the waiter left on top of its own
runtime. Other deadline tasks can miss deadlines.
Fix this by recording that refill in a new flag, dl_donor_refill. If
the boost ends in the class with the flag set, start a new period at
the deboost. The next enqueue is too late for a task blocked on another
rt_mutex: the chain walk copies its deadline into its waiter first.
The task can still hold a lock that another deadline task waits on. The
new deadline can be later than that waiter's. rt_mutex_setprio() then
boosts the task again.
A task that was not refilled keeps its runtime and its deadline. They
are those of its own reservation. A new period there would give it a
full runtime at every contended unlock.
The Sashiko AI review of this series found the sched_setattr() case.
Four cases of deadline_bw_test test this fix:
deboost_blocked_keeps_order, deboost_keeps_own_runtime, and two added
later in this series, deboost_in_class_resets_period and
deboost_after_donor_refill.
In deboost_after_donor_refill, an owner of 5ms every 100ms runs 8ms with
the lock. Its donor is a task of 2ms every 10ms. A deboost that tests
the runtime against dl_runtime, in place of the flag, misses this
refill: it leaves less than dl_runtime. The owner then held less than
1ms after the unlock, in 3 guests of 3. With the flag, it holds its own
5ms.
Fixes: 2d3d891d3344 ("sched/deadline: Add SCHED_DEADLINE inheritance logic")
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Cc: stable@vger.kernel.org
---
include/linux/sched.h | 6 ++++++
kernel/sched/core.c | 16 ++++++++++++---
kernel/sched/deadline.c | 43 +++++++++++++++++++++++++++++++++++++++++
kernel/sched/sched.h | 10 ++++++++++
kernel/sched/syscalls.c | 2 +-
5 files changed, 73 insertions(+), 4 deletions(-)
diff --git a/include/linux/sched.h b/include/linux/sched.h
index 8e7aa758d863..2f2f59402f09 100644
--- a/include/linux/sched.h
+++ b/include/linux/sched.h
@@ -683,6 +683,11 @@ struct sched_dl_entity {
* @dl_overrun tells if the task asked to be informed about runtime
* overruns.
*
+ * @dl_donor_refill tells if a boosted task had its runtime refilled
+ * from the parameters of its donor. Its runtime and deadline are
+ * then not those of its own reservation. The end of the boost
+ * clears it.
+ *
* @dl_server tells if this is a server entity.
*
* @dl_server_active tells if the dlserver is active(started).
@@ -715,6 +720,7 @@ struct sched_dl_entity {
unsigned int dl_yielded : 1;
unsigned int dl_non_contending : 1;
unsigned int dl_overrun : 1;
+ unsigned int dl_donor_refill : 1;
unsigned int dl_server : 1;
unsigned int dl_server_active : 1;
unsigned int dl_defer : 1;
diff --git a/kernel/sched/core.c b/kernel/sched/core.c
index fe2919be3775..781eff645525 100644
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -7957,16 +7957,26 @@ void rt_mutex_setprio(struct task_struct *p, struct task_struct *pi_task)
p->dl.pi_se = pi_task->dl.pi_se;
scope->flags |= ENQUEUE_REPLENISH;
} else {
- p->dl.pi_se = &p->dl;
+ dl_deboost(&p->dl);
+ /*
+ * dl_deboost() can start a new period. Its
+ * deadline can then be after the top waiter's.
+ * Boost the task again then.
+ */
+ if (pi_task && dl_prio(pi_task->prio) &&
+ dl_entity_preempt(&pi_task->dl, &p->dl)) {
+ p->dl.pi_se = pi_task->dl.pi_se;
+ scope->flags |= ENQUEUE_REPLENISH;
+ }
}
} else if (rt_prio(prio)) {
if (dl_prio(oldprio))
- p->dl.pi_se = &p->dl;
+ dl_drop_boost(&p->dl);
if (oldprio < prio)
scope->flags |= ENQUEUE_HEAD;
} else {
if (dl_prio(oldprio))
- p->dl.pi_se = &p->dl;
+ dl_drop_boost(&p->dl);
if (rt_prio(oldprio))
p->rt.timeout = 0;
}
diff --git a/kernel/sched/deadline.c b/kernel/sched/deadline.c
index 914915abb912..66a1150c4811 100644
--- a/kernel/sched/deadline.c
+++ b/kernel/sched/deadline.c
@@ -93,12 +93,48 @@ static inline bool is_dl_boosted(struct sched_dl_entity *dl_se)
{
return pi_of(dl_se) != dl_se;
}
+
+/*
+ * Record that @dl_se had its runtime refilled from the parameters of its
+ * donor, if it is boosted.
+ */
+static inline void dl_note_donor_refill(struct sched_dl_entity *dl_se)
+{
+ if (is_dl_boosted(dl_se))
+ dl_se->dl_donor_refill = 1;
+}
+
+/*
+ * End the boost of @dl_se, whose task stays in the class. If the boost
+ * refilled its runtime from the donor's parameters, start a new period
+ * now. The next enqueue is too late for a task blocked on another
+ * rt_mutex: the chain walk copies its deadline into its waiter node
+ * first. Without a refill, the task keeps the runtime and the deadline
+ * of its own reservation.
+ *
+ * The caller holds the rq lock and the pi_lock of the task, and has
+ * updated the rq clock.
+ */
+void dl_deboost(struct sched_dl_entity *dl_se)
+{
+ struct rq *rq = rq_of_dl_se(dl_se);
+
+ if (dl_se->dl_donor_refill) {
+ dl_se->deadline = rq_clock(rq) + dl_se->dl_deadline;
+ dl_se->runtime = dl_se->dl_runtime;
+ }
+ dl_drop_boost(dl_se);
+}
#else /* !CONFIG_RT_MUTEXES: */
static inline struct sched_dl_entity *pi_of(struct sched_dl_entity *dl_se)
{
return dl_se;
}
+static inline void dl_note_donor_refill(struct sched_dl_entity *dl_se)
+{
+}
+
static inline bool is_dl_boosted(struct sched_dl_entity *dl_se)
{
return false;
@@ -1052,6 +1088,7 @@ static inline void replenish_dl_new_period(struct sched_dl_entity *dl_se,
/* for non-boosted task, pi_of(dl_se) == dl_se */
dl_se->deadline = rq_clock(rq) + pi_of(dl_se)->dl_deadline;
dl_se->runtime = pi_of(dl_se)->dl_runtime;
+ dl_note_donor_refill(dl_se);
/*
* If it is a deferred reservation, and the server
@@ -1150,6 +1187,7 @@ static void replenish_dl_entity(struct sched_dl_entity *dl_se)
if (!dl_se->dl_deadline) {
dl_se->deadline = rq_clock(rq);
dl_se->runtime = 0;
+ dl_note_donor_refill(dl_se);
}
if (dl_se->runtime < 0 || dl_se->dl_yielded)
dl_se->runtime = 0;
@@ -1172,11 +1210,15 @@ static void replenish_dl_entity(struct sched_dl_entity *dl_se)
(dl_se->dl_defer_armed && dl_entity_overflow(dl_se, rq_clock(rq)))) {
dl_se->deadline = rq_clock(rq) + deadline;
dl_se->runtime = runtime;
+ dl_note_donor_refill(dl_se);
}
if (dl_se->dl_yielded && dl_se->runtime > 0)
dl_se->runtime = 0;
+ if (dl_se->runtime <= 0)
+ dl_note_donor_refill(dl_se);
+
/*
* We keep moving the deadline away until we get some
* available runtime for the entity. This ensures correct
@@ -4929,6 +4971,7 @@ static void __dl_clear_params(struct sched_dl_entity *dl_se)
dl_se->dl_yielded = 0;
dl_se->dl_non_contending = 0;
dl_se->dl_overrun = 0;
+ dl_se->dl_donor_refill = 0;
dl_se->dl_server = 0;
dl_se->dl_defer = 0;
dl_se->dl_defer_running = 0;
diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h
index 4f1e7ad94f80..f80ea5541b3f 100644
--- a/kernel/sched/sched.h
+++ b/kernel/sched/sched.h
@@ -356,6 +356,16 @@ extern int sched_dl_global_validate(void);
extern void sched_dl_do_global(void);
extern int sched_dl_overflow(struct task_struct *p, int policy, const struct sched_attr *attr);
extern void __setparam_dl(struct task_struct *p, const struct sched_attr *attr);
+#ifdef CONFIG_RT_MUTEXES
+extern void dl_deboost(struct sched_dl_entity *dl_se);
+
+/* End the boost of @dl_se. The caller holds the rq lock of its task. */
+static inline void dl_drop_boost(struct sched_dl_entity *dl_se)
+{
+ dl_se->pi_se = dl_se;
+ dl_se->dl_donor_refill = 0;
+}
+#endif
extern void __getparam_dl(struct task_struct *p, struct sched_attr *attr, unsigned int flags);
extern bool __checkparam_dl(const struct sched_attr *attr);
extern bool dl_param_changed(struct task_struct *p, const struct sched_attr *attr);
diff --git a/kernel/sched/syscalls.c b/kernel/sched/syscalls.c
index 80de69646766..a27868ff16fb 100644
--- a/kernel/sched/syscalls.c
+++ b/kernel/sched/syscalls.c
@@ -310,7 +310,7 @@ static inline void __setscheduler_dl_pi(int newprio, int policy,
* the class. Drop its boost then, as rt_mutex_setprio() does.
*/
if (!dl_prio(newprio))
- p->dl.pi_se = &p->dl;
+ dl_drop_boost(&p->dl);
}
#else /* !CONFIG_RT_MUTEXES */
static inline void __setscheduler_dl_pi(int newprio, int policy,
--
2.47.3
^ permalink raw reply related [flat|nested] 35+ messages in thread
* [PATCH v8 24/33] sched/deadline: Replenish a throttled task boosted while asleep
2026-11-19 20:15 [PATCH v8 00/33] sched/deadline: bandwidth accounting fixes and selftests Mathieu Desnoyers
` (22 preceding siblings ...)
2026-11-19 20:16 ` [PATCH v8 23/33] sched/deadline: Reset the donor's runtime at a deboost in the class Mathieu Desnoyers
@ 2026-11-19 20:16 ` Mathieu Desnoyers
2026-11-19 20:16 ` [PATCH v8 25/33] sched/deadline: Count the bandwidth a boosted task is given Mathieu Desnoyers
` (8 subsequent siblings)
32 siblings, 0 replies; 35+ messages in thread
From: Mathieu Desnoyers @ 2026-11-19 20:16 UTC (permalink / raw)
To: juri.lelli, peterz, mingo, vincent.guittot
Cc: Mathieu Desnoyers, tglx, will, boqun, dietmar.eggemann, rostedt,
bsegall, mgorman, vschneid, kprateek.nayak, luca.abeni,
mathieu.poirier, longman, ridong.chen, tj, hannes, mkoutny, shuah,
bigeasy, clrkwllms, parri.andrea, m.szyprowski, kayracizmeci,
weihengqiuu, linux-kernel, cgroups, linux-kselftest,
linux-rt-devel, stable
A deadline task can be throttled as it blocks, with an rt_mutex held.
Its replenishment timer is then armed. A deadline waiter can boost the
task while it sleeps.
Commit feff2e65efd8 ("sched/deadline: Unthrottle PI boosted threads
while enqueuing") lets a boost override the throttle. enqueue_task_dl()
lifts the throttle of a boosted task, and dl_task_timer() ignores a
boosted task. This covers a task that is enqueued while boosted. But a
boost of a sleeping task does not enqueue it.
T is a SCHED_DEADLINE task that holds the rt_mutex L. W is a
SCHED_DEADLINE task with an earlier deadline:
T W T's timer
uses up its runtime, sleeps
dl_throttled = 1
start_dl_timer()
pthread_mutex_timedlock(L)
rt_mutex_setprio(T, W)
T sleeps: no enqueue
dl_task_timer()
T is boosted: returns
times out
rt_mutex_setprio(T, NULL)
wakes up
enqueue_dl_entity()
dl_throttled: waits for the timer
The timer has already fired. T would never run again, not even to die
of SIGKILL. Every later waiter on L would hang too. This was found by
reading.
Fix this by returning early in dl_task_timer() only for a boosted task
that is queued. A boosted task that sleeps then takes the existing path
for a throttled task that is not queued: it is replenished.
The refill reads the parameters of the donor, from the timer. So this
relies on two earlier patches. "sched/deadline: Do not replenish a boost
from cleared parameters" keeps the refill from looping on cleared
parameters. "rtmutex: Continue the chain walk when a waiter's deadline
donor changes" keeps the donor pointer from going stale. After a deboost in
the class, "sched/deadline: Reset the donor's runtime at a deboost in
the class" starts a period of the task's own reservation.
The Sashiko AI review of this series found this.
Fixes: feff2e65efd8 ("sched/deadline: Unthrottle PI boosted threads while enqueuing")
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Cc: stable@vger.kernel.org
---
kernel/sched/deadline.c | 9 ++++++---
1 file changed, 6 insertions(+), 3 deletions(-)
diff --git a/kernel/sched/deadline.c b/kernel/sched/deadline.c
index 66a1150c4811..f4b670fee4a0 100644
--- a/kernel/sched/deadline.c
+++ b/kernel/sched/deadline.c
@@ -1637,10 +1637,13 @@ static enum hrtimer_restart dl_task_timer(struct hrtimer *timer)
}
/*
- * The task might have been boosted by someone else and might be in the
- * boosting/deboosting path, its not throttled.
+ * A boost overrides the throttle. enqueue_task_dl() lifts it when the
+ * boost enqueues the task, or when the task wakes up boosted. So a
+ * queued task needs nothing here. A boost of a sleeping task does not
+ * enqueue it, and it can end before the wakeup. Nothing would lift the
+ * throttle then. So let a sleeping task reach the !queued case below.
*/
- if (is_dl_boosted(dl_se))
+ if (is_dl_boosted(dl_se) && task_on_rq_queued(p))
goto unlock;
/*
--
2.47.3
^ permalink raw reply related [flat|nested] 35+ messages in thread
* [PATCH v8 25/33] sched/deadline: Count the bandwidth a boosted task is given
2026-11-19 20:15 [PATCH v8 00/33] sched/deadline: bandwidth accounting fixes and selftests Mathieu Desnoyers
` (23 preceding siblings ...)
2026-11-19 20:16 ` [PATCH v8 24/33] sched/deadline: Replenish a throttled task boosted while asleep Mathieu Desnoyers
@ 2026-11-19 20:16 ` Mathieu Desnoyers
2026-11-19 20:16 ` [PATCH v8 26/33] sched/deadline: Start a period for a task that enters the class blocked Mathieu Desnoyers
` (7 subsequent siblings)
32 siblings, 0 replies; 35+ messages in thread
From: Mathieu Desnoyers @ 2026-11-19 20:16 UTC (permalink / raw)
To: juri.lelli, peterz, mingo, vincent.guittot
Cc: Mathieu Desnoyers, tglx, will, boqun, dietmar.eggemann, rostedt,
bsegall, mgorman, vschneid, kprateek.nayak, luca.abeni,
mathieu.poirier, longman, ridong.chen, tj, hannes, mkoutny, shuah,
bigeasy, clrkwllms, parri.andrea, m.szyprowski, kayracizmeci,
weihengqiuu, linux-kernel, cgroups, linux-kselftest,
linux-rt-devel, stable
sched_setattr(SCHED_DEADLINE) on a sleeping task that priority
inheritance boosted into the class leaves the new bandwidth out of
this_bw. switched_to_dl() does not run, since the task is in the class
already. sched_dl_overflow() changes this_bw only for a task of the
deadline policy. When the task wakes up, __add_running_bw() warns:
WARN_ON_ONCE(dl_rq->running_bw > dl_rq->this_bw);
When the task leaves the runqueue, this_bw would stay short by that
bandwidth, and skew GRUB reclaiming on that CPU.
Fix this by changing this_bw in __setparam_dl(), which every task that
gains deadline parameters goes through. If no runqueue counts the
bandwidth of the task, dl_change_utilization() now adds the new one. It
now also runs when the bandwidth does not change. It then returns early,
so that a sleeping task keeps its running_bw and its inactive timer.
A sugov kthread given a reservation of its own now has its bandwidth
counted in this_bw. It reaches dl_change_utilization(), so the
WARN_ON_ONCE() for SCHED_FLAG_SUGOV there goes. sched_dl_overflow()
returned early for an unchanged bandwidth, even if no root domain counts
it. That is the case for such a kthread, and for a task that a rebuild
left out. Admit such a bandwidth as new. Without room, that fails with
EBUSY. The Sashiko AI review of this series found this early return.
A reproducer shows the warning, and this_bw unchanged by the change.
With this patch, this_bw takes the new bandwidth, and nothing warns.
Fixes: 8fd27231c330 ("sched/deadline: Track the "total rq utilization" too")
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Reviewed-by: Juri Lelli <juri.lelli@redhat.com>
Cc: stable@vger.kernel.org
---
kernel/sched/deadline.c | 54 ++++++++++++++++++++++++++++++++---------
1 file changed, 43 insertions(+), 11 deletions(-)
diff --git a/kernel/sched/deadline.c b/kernel/sched/deadline.c
index f4b670fee4a0..8ac78958a17a 100644
--- a/kernel/sched/deadline.c
+++ b/kernel/sched/deadline.c
@@ -600,22 +600,37 @@ void cancel_inactive_timer(struct sched_dl_entity *dl_se)
cancel_dl_timer(dl_se, &dl_se->inactive_timer);
}
+/*
+ * Make the runqueue of a sleeping task count @new_bw as the bandwidth of
+ * the task. Do nothing for a queued task. The dequeue and the enqueue
+ * around the change of its parameters update this_bw.
+ */
static void dl_change_utilization(struct task_struct *p, u64 new_bw)
{
- WARN_ON_ONCE(p->dl.flags & SCHED_FLAG_SUGOV);
+ struct sched_dl_entity *dl_se = &p->dl;
+ struct rq *rq = task_rq(p);
if (task_on_rq_queued(p))
return;
/*
- * The task is waking up on another CPU, and no runqueue counts its
- * bandwidth. The enqueue adds it, with the new parameters that
- * __setparam_dl() sets right after this.
+ * No runqueue counts the bandwidth of the task. Add the new one to
+ * this runqueue, so that add_rq_bw() adds nothing later. A task that
+ * enters the class asleep comes here, and so does a special entity
+ * given a reservation of its own. So does a task waking up on
+ * another CPU: task_cpu() already points at the CPU it wakes on.
*/
- if (!p->dl.dl_bw_oncpu)
+ if (!dl_se->dl_bw_oncpu) {
+ __add_rq_bw(new_bw, &rq->dl);
+ dl_se->dl_bw_oncpu = 1;
+ return;
+ }
+
+ /* Same bandwidth: leave running_bw and the inactive timer alone. */
+ if (new_bw == dl_se->dl_bw)
return;
- dl_rq_change_utilization(task_rq(p), &p->dl, new_bw);
+ dl_rq_change_utilization(rq, dl_se, new_bw);
}
static void __dl_clear_params(struct sched_dl_entity *dl_se);
@@ -4759,10 +4774,6 @@ int sched_dl_overflow(struct task_struct *p, int policy,
if (attr->sched_flags & SCHED_FLAG_SUGOV)
return 0;
- /* !deadline task may carry old deadline bandwidth */
- if (new_bw == p->dl.dl_bw && task_has_dl_policy(p))
- return 0;
-
/*
* Change the bandwidth of a deadline task where it is counted: in
* the root domain of the CPU recorded with it. A task moved asleep
@@ -4777,6 +4788,16 @@ int sched_dl_overflow(struct task_struct *p, int policy,
*/
counted = task_has_dl_policy(p) && p->dl.dl_bw_cpu >= 0 &&
p->dl.dl_bw_gen == dl_bw_gen;
+
+ /*
+ * !deadline task may carry old deadline bandwidth. The same
+ * bandwidth needs no admission if it is counted. If it is not, admit
+ * it as new below. A special entity given a reservation of its own
+ * has no bandwidth counted, nor has a task that a rebuild left out.
+ */
+ if (new_bw == p->dl.dl_bw && counted)
+ return 0;
+
cpu = counted && p->dl.dl_bw ? p->dl.dl_bw_cpu : task_cpu(p);
dl_b = dl_bw_of(cpu);
@@ -4806,7 +4827,6 @@ int sched_dl_overflow(struct task_struct *p, int policy,
if (counted)
__dl_sub(dl_b, p->dl.dl_bw, cpus);
__dl_add(dl_b, new_bw, cpus);
- dl_change_utilization(p, new_bw);
err = 0;
} else if (!dl_policy(policy) && task_has_dl_policy(p)) {
/*
@@ -4856,8 +4876,20 @@ int sched_dl_overflow(struct task_struct *p, int policy,
void __setparam_dl(struct task_struct *p, const struct sched_attr *attr)
{
struct sched_dl_entity *dl_se = &p->dl;
+ u64 period = attr->sched_period ?: attr->sched_deadline;
bool held;
+ /*
+ * Change the bandwidth that the runqueue counts here, with the
+ * parameters. A task that priority inheritance boosted into the
+ * class is in it already, so switched_to_dl() does not run for it.
+ *
+ * A special entity has no bandwidth on a runqueue. Test the flags
+ * of the request, since those of the entity are still the old ones.
+ */
+ if (!(attr->sched_flags & SCHED_FLAG_SUGOV))
+ dl_change_utilization(p, to_ratio(period, attr->sched_runtime));
+
held = dl_se_holds_bw(dl_se);
dl_se->dl_runtime = attr->sched_runtime;
dl_se->dl_deadline = attr->sched_deadline;
--
2.47.3
^ permalink raw reply related [flat|nested] 35+ messages in thread
* [PATCH v8 26/33] sched/deadline: Start a period for a task that enters the class blocked
2026-11-19 20:15 [PATCH v8 00/33] sched/deadline: bandwidth accounting fixes and selftests Mathieu Desnoyers
` (24 preceding siblings ...)
2026-11-19 20:16 ` [PATCH v8 25/33] sched/deadline: Count the bandwidth a boosted task is given Mathieu Desnoyers
@ 2026-11-19 20:16 ` Mathieu Desnoyers
2026-11-19 20:16 ` [PATCH v8 27/33] sched/deadline: Start a period at a deboost out of the class Mathieu Desnoyers
` (6 subsequent siblings)
32 siblings, 0 replies; 35+ messages in thread
From: Mathieu Desnoyers @ 2026-11-19 20:16 UTC (permalink / raw)
To: juri.lelli, peterz, mingo, vincent.guittot
Cc: Mathieu Desnoyers, tglx, will, boqun, dietmar.eggemann, rostedt,
bsegall, mgorman, vschneid, kprateek.nayak, luca.abeni,
mathieu.poirier, longman, ridong.chen, tj, hannes, mkoutny, shuah,
bigeasy, clrkwllms, parri.andrea, m.szyprowski, kayracizmeci,
weihengqiuu, linux-kernel, cgroups, linux-kselftest,
linux-rt-devel, stable
Commit fb00aca47440 ("rtmutex: Turn the plist into an rb-tree") sorts
the waiters of an rt_mutex by priority. For deadline tasks, "the one
with the earliest deadline wins". A waiter takes the deadline of its
task.
A task can enter the class while it waits on an rt_mutex.
sched_setattr() can make it SCHED_DEADLINE. A priority inheritance
boost can also put it in the class, if it owns another rt_mutex.
switched_to_dl() leaves a task that is not queued to its wakeup. But
the chain walk reads the deadline first, and copies it into the waiter.
A task new to the class has a deadline of 0.
So the waiter goes before every other deadline waiter. It takes the
rt_mutex first, whatever their deadlines. Those waiters can then miss
their deadlines. It also boosts a deadline owner whose own deadline is
earlier.
Fix this by starting a new period in switched_to_dl() for a task that
waits on an rt_mutex, if its deadline has passed. The period uses the
reservation the task runs on: its own, or the copy a boost made. This
is what setup_new_dl_entity() does at an enqueue. At the wakeup,
update_dl_entity() then checks that period as for any wakeup.
A new donor of a task with no parameters of its own needs the same. For
a queued task, the enqueue of the boost starts a period from the new
donor. For a blocked task, dl_boost_blocked() starts it at the boost.
The setattr_blocked_keeps_order and boost_blocked_keeps_order cases,
added to deadline_bw_test later in this series, test this. Both fail on
v7.4-rc2 and on the series without this patch. The triage of the Sashiko
AI review of this series found the bug.
Fixes: fb00aca47440 ("rtmutex: Turn the plist into an rb-tree")
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Cc: stable@vger.kernel.org
---
kernel/sched/core.c | 4 ++++
kernel/sched/deadline.c | 46 +++++++++++++++++++++++++++++++++++++++++
kernel/sched/sched.h | 1 +
3 files changed, 51 insertions(+)
diff --git a/kernel/sched/core.c b/kernel/sched/core.c
index 781eff645525..d30b28bfd919 100644
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -7954,8 +7954,12 @@ void rt_mutex_setprio(struct task_struct *p, struct task_struct *pi_task)
if (!dl_prio(p->normal_prio) ||
(pi_task && dl_prio(pi_task->prio) &&
dl_entity_preempt(&pi_task->dl, &p->dl))) {
+ bool new_donor = p->dl.pi_se != pi_task->dl.pi_se;
+
p->dl.pi_se = pi_task->dl.pi_se;
scope->flags |= ENQUEUE_REPLENISH;
+ if (!scope->queued && new_donor)
+ dl_boost_blocked(p);
} else {
dl_deboost(&p->dl);
/*
diff --git a/kernel/sched/deadline.c b/kernel/sched/deadline.c
index 8ac78958a17a..77a1d1f463f6 100644
--- a/kernel/sched/deadline.c
+++ b/kernel/sched/deadline.c
@@ -125,6 +125,12 @@ void dl_deboost(struct sched_dl_entity *dl_se)
}
dl_drop_boost(dl_se);
}
+
+/* Return true if @p waits on an rt_mutex. The caller holds its pi_lock. */
+static inline bool dl_task_pi_blocked(struct task_struct *p)
+{
+ return p->pi_blocked_on;
+}
#else /* !CONFIG_RT_MUTEXES: */
static inline struct sched_dl_entity *pi_of(struct sched_dl_entity *dl_se)
{
@@ -139,6 +145,11 @@ static inline bool is_dl_boosted(struct sched_dl_entity *dl_se)
{
return false;
}
+
+static inline bool dl_task_pi_blocked(struct task_struct *p)
+{
+ return false;
+}
#endif /* !CONFIG_RT_MUTEXES */
static inline u8 dl_get_type(struct sched_dl_entity *dl_se, struct rq *rq)
@@ -1116,6 +1127,30 @@ static inline void replenish_dl_new_period(struct sched_dl_entity *dl_se,
trace_sched_dl_replenish_tp(dl_se, cpu_of(rq), dl_get_type(dl_se, rq));
}
+#ifdef CONFIG_RT_MUTEXES
+/*
+ * Start a period from the new donor of @p, if @p has no reservation of its
+ * own and waits on an rt_mutex. The enqueue does this for a queued task, in
+ * replenish_dl_entity(). A blocked task is not enqueued, but the chain walk
+ * copies its deadline into its waiter.
+ *
+ * The caller calls it only when the donor changes. A new period changes the
+ * deadline, so the chain walk finds the waiter out of date and goes on. In a
+ * ww_mutex cycle, a new period at each step would never let the walk stop.
+ *
+ * The caller holds the rq lock and the pi_lock of @p, and has updated the
+ * rq clock.
+ */
+void dl_boost_blocked(struct task_struct *p)
+{
+ struct sched_dl_entity *dl_se = &p->dl;
+
+ if (!dl_se->dl_deadline && !dl_se->dl_throttled &&
+ dl_task_pi_blocked(p))
+ replenish_dl_new_period(dl_se, task_rq(p));
+}
+#endif
+
/*
* We are being explicitly informed that a new instance is starting,
* and this means that:
@@ -4565,6 +4600,17 @@ static void switched_to_dl(struct rq *rq, struct task_struct *p)
if (!task_on_rq_queued(p)) {
add_rq_bw(&p->dl, &rq->dl);
+ /*
+ * A task that waits on an rt_mutex cannot wait for its wakeup.
+ * The chain walk sorts its waiter by its deadline first. A task
+ * new to the class has a deadline of 0, which goes before every
+ * real one. Start a new period now, as setup_new_dl_entity()
+ * does at an enqueue.
+ */
+ if (dl_task_pi_blocked(p) && !p->dl.dl_throttled &&
+ dl_time_before(p->dl.deadline, rq_clock(rq)))
+ replenish_dl_new_period(&p->dl, rq);
+
return;
}
diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h
index f80ea5541b3f..717ced2bb487 100644
--- a/kernel/sched/sched.h
+++ b/kernel/sched/sched.h
@@ -358,6 +358,7 @@ extern int sched_dl_overflow(struct task_struct *p, int policy, const struct sc
extern void __setparam_dl(struct task_struct *p, const struct sched_attr *attr);
#ifdef CONFIG_RT_MUTEXES
extern void dl_deboost(struct sched_dl_entity *dl_se);
+extern void dl_boost_blocked(struct task_struct *p);
/* End the boost of @dl_se. The caller holds the rq lock of its task. */
static inline void dl_drop_boost(struct sched_dl_entity *dl_se)
--
2.47.3
^ permalink raw reply related [flat|nested] 35+ messages in thread
* [PATCH v8 27/33] sched/deadline: Start a period at a deboost out of the class
2026-11-19 20:15 [PATCH v8 00/33] sched/deadline: bandwidth accounting fixes and selftests Mathieu Desnoyers
` (25 preceding siblings ...)
2026-11-19 20:16 ` [PATCH v8 26/33] sched/deadline: Start a period for a task that enters the class blocked Mathieu Desnoyers
@ 2026-11-19 20:16 ` Mathieu Desnoyers
2026-11-19 20:16 ` [PATCH v8 28/33] selftests/sched: Test deadline boosts in deadline_bw_test Mathieu Desnoyers
` (5 subsequent siblings)
32 siblings, 0 replies; 35+ messages in thread
From: Mathieu Desnoyers @ 2026-11-19 20:16 UTC (permalink / raw)
To: juri.lelli, peterz, mingo, vincent.guittot
Cc: Mathieu Desnoyers, tglx, will, boqun, dietmar.eggemann, rostedt,
bsegall, mgorman, vschneid, kprateek.nayak, luca.abeni,
mathieu.poirier, longman, ridong.chen, tj, hannes, mkoutny, shuah,
bigeasy, clrkwllms, parri.andrea, m.szyprowski, kayracizmeci,
weihengqiuu, linux-kernel, cgroups, linux-kselftest,
linux-rt-devel, stable
A task that leaves SCHED_DEADLINE keeps its parameters, runtime and
deadline until its 0-lag time. Commit a649f237db18 ("sched,dl: Fix sched
class hopping CBS hole") keeps them. So leaving and coming back gains
nothing. A priority inheritance boost before then keeps them too.
replenish_dl_entity() takes the donor's runtime and deadline only for a
task with no parameters of its own.
A boosted task is not throttled. When it runs out of runtime, it is
refilled from the donor's parameters. If the task stays in the class at
its deboost, "sched/deadline: Reset the donor's runtime at a deboost in
the class" starts a new period.
A task that leaves the class at its deboost is missed. switched_from_dl()
calls task_non_contending(), which computes the 0-lag time from the
runtime and deadline of the donor's refills, with the task's own
dl_runtime and dl_period. The root domain then keeps the task's
bandwidth in total_bw for too long, or releases it too early. Until it
is released, deadline tasks that would fit are refused.
Fix this by starting a period of its own in switched_from_dl(), as
dl_deboost() does. Share that code in dl_start_own_period(). The 0-lag
time of the new period is now, or has passed, so the bandwidth is
released at once.
The Sashiko AI review found this, on "sched/deadline: Reset the
deadline and the runtime with the parameters" of the first series. The
deboost_out_after_donor_refill case, added to deadline_bw_test later in
this series, tests it. An owner of 5ms every 100ms leaves the class
while it holds the lock, and is refilled by a donor of 2ms every 10ms.
Without this patch, its bandwidth was still in total_bw right after the
unlock, in 40 runs of 40. With it, the bandwidth was gone in 60 runs of
60.
Fixes: 2d3d891d3344 ("sched/deadline: Add SCHED_DEADLINE inheritance logic")
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Cc: stable@vger.kernel.org
---
kernel/sched/deadline.c | 29 +++++++++++++++++++++++------
1 file changed, 23 insertions(+), 6 deletions(-)
diff --git a/kernel/sched/deadline.c b/kernel/sched/deadline.c
index 77a1d1f463f6..d284036ecb8a 100644
--- a/kernel/sched/deadline.c
+++ b/kernel/sched/deadline.c
@@ -83,6 +83,16 @@ static inline int on_dl_rq(struct sched_dl_entity *dl_se)
return !RB_EMPTY_NODE(&dl_se->rb_node);
}
+/*
+ * Start a period of the reservation of @dl_se now. The caller holds the rq
+ * lock and has updated the rq clock.
+ */
+static inline void dl_start_own_period(struct sched_dl_entity *dl_se)
+{
+ dl_se->deadline = rq_clock(rq_of_dl_se(dl_se)) + dl_se->dl_deadline;
+ dl_se->runtime = dl_se->dl_runtime;
+}
+
#ifdef CONFIG_RT_MUTEXES
static inline struct sched_dl_entity *pi_of(struct sched_dl_entity *dl_se)
{
@@ -117,12 +127,8 @@ static inline void dl_note_donor_refill(struct sched_dl_entity *dl_se)
*/
void dl_deboost(struct sched_dl_entity *dl_se)
{
- struct rq *rq = rq_of_dl_se(dl_se);
-
- if (dl_se->dl_donor_refill) {
- dl_se->deadline = rq_clock(rq) + dl_se->dl_deadline;
- dl_se->runtime = dl_se->dl_runtime;
- }
+ if (dl_se->dl_donor_refill)
+ dl_start_own_period(dl_se);
dl_drop_boost(dl_se);
}
@@ -4504,6 +4510,17 @@ void dl_clear_root_domains(void)
static void switched_from_dl(struct rq *rq, struct task_struct *p)
{
+ /*
+ * The boost of a task with parameters of its own ends, and the task
+ * leaves the class. If the boost refilled its runtime from the
+ * donor's parameters, its runtime and deadline are not those of its
+ * reservation. task_non_contending() would mix them with its own
+ * dl_runtime and dl_period. Start a period of its own instead, as
+ * dl_deboost() does. Its 0-lag time is then now, or has passed.
+ */
+ if (p->dl.dl_runtime && p->dl.dl_donor_refill)
+ dl_start_own_period(&p->dl);
+
/*
* task_non_contending() can start the "inactive timer" (if the 0-lag
* time is in the future). If the task switches back to dl before
--
2.47.3
^ permalink raw reply related [flat|nested] 35+ messages in thread
* [PATCH v8 28/33] selftests/sched: Test deadline boosts in deadline_bw_test
2026-11-19 20:15 [PATCH v8 00/33] sched/deadline: bandwidth accounting fixes and selftests Mathieu Desnoyers
` (26 preceding siblings ...)
2026-11-19 20:16 ` [PATCH v8 27/33] sched/deadline: Start a period at a deboost out of the class Mathieu Desnoyers
@ 2026-11-19 20:16 ` Mathieu Desnoyers
2026-11-19 20:16 ` [PATCH v8 29/33] sched/deadline: Cancel the replenishment timer with the parameters Mathieu Desnoyers
` (4 subsequent siblings)
32 siblings, 0 replies; 35+ messages in thread
From: Mathieu Desnoyers @ 2026-11-19 20:16 UTC (permalink / raw)
To: juri.lelli, peterz, mingo, vincent.guittot
Cc: Mathieu Desnoyers, tglx, will, boqun, dietmar.eggemann, rostedt,
bsegall, mgorman, vschneid, kprateek.nayak, luca.abeni,
mathieu.poirier, longman, ridong.chen, tj, hannes, mkoutny, shuah,
bigeasy, clrkwllms, parri.andrea, m.szyprowski, kayracizmeci,
weihengqiuu, linux-kernel, cgroups, linux-kselftest,
linux-rt-devel
Add the seven cases of deadline_bw_test that test the fixes of priority
inheritance boosts and deboosts, earlier in this series. They cover:
- a deboost after the donor refilled the owner's runtime, in the class
and out of it
- a deboost of an owner that left the class asleep, or that enters the
class while boosted
- a former deadline owner that sleeps with the lock
- the order of the waiters of a lock, when sched_setattr() or a boost
puts a waiter in the class while it waits
The comment above each case gives its scenario.
The message of each fix names the cases that test it.
deboost_released_resets_period and pi_former_owner_asleep are not named
there. They fail on v7.4-rc2, and pass with the first series alone.
They test deboosts, so they come here.
On v7.4-rc2, in 8 VMs of 8 CPUs, six of these cases fail in every VM:
- boost_blocked_keeps_order
- deboost_after_donor_refill
- deboost_in_class_resets_period
- deboost_released_resets_period
- pi_former_owner_asleep
- setattr_blocked_keeps_order
deboost_out_after_donor_refill is new, and was not run there. It fails
without "sched/deadline: Start a period at a deboost out of the class".
With the fixes up to here, every case passes wherever it runs.
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
---
.../sched/deadline/deadline_bw_test.c | 479 ++++++++++++++++--
1 file changed, 444 insertions(+), 35 deletions(-)
diff --git a/tools/testing/selftests/sched/deadline/deadline_bw_test.c b/tools/testing/selftests/sched/deadline/deadline_bw_test.c
index c83e0272771f..fe63c8f093e7 100644
--- a/tools/testing/selftests/sched/deadline/deadline_bw_test.c
+++ b/tools/testing/selftests/sched/deadline/deadline_bw_test.c
@@ -880,12 +880,31 @@ static int pi_chain_two_owners(void)
}
/*
- * Run as the owner thread: take pi_lock, and sleep 400ms while holding
- * it. Return NULL, or 1 on an error.
+ * Run as the owner thread. With @arg (was_dl) set, first become a
+ * deadline task of 30ms every 100ms, run 1ms, and sleep on owner_go. The
+ * main thread switches the owner to SCHED_OTHER during that sleep. Then
+ * take pi_lock, and sleep 400ms while holding it. Return NULL, or 1 on an
+ * error.
*/
static void *sleeping_owner(void *arg)
{
+ int was_dl = (long)arg;
+
owner_tid = gettid_();
+ if (was_dl) {
+ if (set_deadline(0, 30 * NSEC_PER_MSEC, PERIOD)) {
+ owner_holds = -1;
+ sem_post(&owner_ready);
+ return (void *)1;
+ }
+ spin_ns(NSEC_PER_MSEC);
+ /*
+ * Sleep. The main thread switches this thread to SCHED_OTHER
+ * 200ms later, past its 0-lag time.
+ */
+ sem_post(&owner_ready);
+ sem_wait_sig(&owner_go);
+ }
if (take_pi_lock())
return (void *)1;
owner_holds = 2;
@@ -896,30 +915,47 @@ static void *sleeping_owner(void *arg)
/*
* A SCHED_OTHER owner takes pi_lock and sleeps 400ms while holding it.
- * The main thread then waits 20ms for the lock, 5 times. For each wait it
- * becomes a deadline task of 5ms every 50ms. Each wait boosts the
+ * The main thread then waits 20ms for the lock, @rounds times. For each
+ * wait it becomes a deadline task of 5ms every 50ms. Each wait boosts the
* sleeping owner, and its timeout deboosts it. Every wait must time out.
+ *
+ * With @was_dl, the owner was a deadline task of 30ms every 100ms before.
+ * It ran 1ms and slept. The main thread switched it to SCHED_OTHER 200ms
+ * later, past its 0-lag time.
*/
-static int pi_other_owner_asleep(void)
+static int pi_asleep(int was_dl, int rounds)
{
pthread_t o;
int r, ret = KSFT_PASS;
if (pi_init())
return KSFT_SKIP;
- if (pthread_create(&o, NULL, sleeping_owner, NULL)) {
+ if (pthread_create(&o, NULL, sleeping_owner, (void *)(long)was_dl)) {
ksft_print_msg("the owner thread could not start\n");
return KSFT_FAIL;
}
- if (wait_for(5000, owner_holds) || owner_holds != 2) {
+ if (was_dl) {
+ if (sem_wait_for(&owner_ready, 5000)) {
+ ksft_print_msg("the owner did not become a deadline task\n");
+ ret = KSFT_FAIL;
+ } else {
+ sleep_ns(200 * NSEC_PER_MSEC);
+ if (set_other(owner_tid)) {
+ ksft_print_msg("the owner cannot leave SCHED_DEADLINE\n");
+ ret = KSFT_FAIL;
+ }
+ }
+ sem_post(&owner_go);
+ }
+ if (ret == KSFT_PASS && (wait_for(5000, owner_holds) || owner_holds != 2)) {
ksft_print_msg("the owner never took the lock\n");
ret = KSFT_FAIL;
}
if (ret == KSFT_PASS)
sleep_ns(20 * NSEC_PER_MSEC);
- for (r = 0; r < 5 && ret == KSFT_PASS; r++) {
+ for (r = 0; r < rounds && ret == KSFT_PASS; r++) {
if (wait_lock_for(20, 1)) {
- ksft_print_msg("round %d of 5 failed\n", r + 1);
+ ksft_print_msg("round %d of %d failed\n", r + 1, rounds);
ret = KSFT_FAIL;
}
}
@@ -927,6 +963,16 @@ static int pi_other_owner_asleep(void)
return ret;
}
+static int pi_other_owner_asleep(void)
+{
+ return pi_asleep(0, 5);
+}
+
+static int pi_former_owner_asleep(void)
+{
+ return pi_asleep(1, 5);
+}
+
/*
* Run as the owner thread. Become a deadline task of 30ms every 100ms,
* and take pi_lock. Hold it for 300ms, spinning 2ms and sleeping 5ms in
@@ -1635,9 +1681,9 @@ static atomic_int owner_first_period_ms;
static _Atomic pid_t donor_tid;
/*
- * Run as the owner O of deboost_resets_period(). Store the CPU time of
- * step 4, in ms, in owner_first_period_ms. Return NULL, 1 on an error, or
- * 2 if no waiter boosted it within 1s.
+ * Run as the owner O of deboost_case(). @arg is its @mode. Store the CPU
+ * time of step 4, in ms, in owner_first_period_ms. Return NULL, 1 on an
+ * error, or 2 if no waiter boosted it within 1s.
*/
static void *deboosted_owner(void *arg)
{
@@ -1645,6 +1691,17 @@ static void *deboosted_owner(void *arg)
int err;
owner_tid = gettid_();
+ if (arg == (void *)1L) {
+ err = set_deadline(0, 60 * NSEC_PER_MSEC, PERIOD);
+ if (err) {
+ ksft_print_msg("the owner refused as a deadline task: %s\n",
+ strerror(-err));
+ return (void *)1;
+ }
+ spin_ns(NSEC_PER_MSEC);
+ sem_post(&owner_ready);
+ sem_wait_sig(&owner_go);
+ }
if (take_pi_lock())
return (void *)1;
owner_holds = 2;
@@ -1655,9 +1712,18 @@ static void *deboosted_owner(void *arg)
return (void *)2; /* never boosted */
}
}
+ if (arg == (void *)2L) {
+ err = set_deadline(0, 10 * NSEC_PER_MSEC, PERIOD);
+ if (err) {
+ ksft_print_msg("the boosted owner refused as a deadline task: %s\n",
+ strerror(-err));
+ drop_pi_lock();
+ return (void *)1;
+ }
+ }
spin_ns(5 * NSEC_PER_MSEC);
drop_pi_lock();
- err = set_deadline(0, 10 * NSEC_PER_MSEC, PERIOD);
+ err = arg == (void *)2L ? 0 : set_deadline(0, 10 * NSEC_PER_MSEC, PERIOD);
if (err) {
ksft_print_msg("the owner refused as a deadline task: %s\n", strerror(-err));
return (void *)1;
@@ -1694,8 +1760,8 @@ static void *donor_waiter(void *arg)
}
/*
- * The owner O starts as SCHED_OTHER. The donor D is the waiter that
- * boosts it: a deadline task of 40ms every 100ms.
+ * Run a deboost case. The owner O starts as SCHED_OTHER. The donor D is
+ * the waiter that boosts it: a deadline task of 40ms every 100ms.
* 1. O takes pi_lock and spins.
* 2. D blocks on pi_lock. This boosts O, which then runs on D's runtime.
* 3. O spins 5ms more and unlocks. The unlock deboosts O.
@@ -1704,8 +1770,16 @@ static void *donor_waiter(void *arg)
*
* O must get about 10ms in step 4. The case fails above 20ms: O then
* still runs on the runtime that the boost left.
+ *
+ * @mode changes one step:
+ * - 0: nothing.
+ * - 1: before step 1, O runs 1ms as a deadline task of 60ms every 100ms,
+ * and sleeps. The main thread switches it to SCHED_OTHER 300ms later,
+ * past its 0-lag time.
+ * - 2: O becomes the deadline task of step 4 in step 3 instead, before
+ * the 5ms. It stays in the deadline class at the deboost.
*/
-static int deboost_resets_period(void)
+static int deboost_case(long mode)
{
int err, ret = KSFT_PASS;
void *o_ret, *w_ret;
@@ -1714,10 +1788,27 @@ static int deboost_resets_period(void)
if (pi_init())
return KSFT_SKIP;
owner_first_period_ms = -1;
- if (pthread_create(&o, NULL, deboosted_owner, NULL)) {
+ if (pthread_create(&o, NULL, deboosted_owner, (void *)mode)) {
ksft_print_msg("the owner thread could not start\n");
return KSFT_FAIL;
}
+ if (mode == 1) {
+ if (sem_wait_for(&owner_ready, 5000)) {
+ /* The owner may get there late. Let it end. */
+ sem_post(&owner_go);
+ join(o);
+ return KSFT_FAIL; /* the owner said why */
+ }
+ sleep_ns(3 * PERIOD);
+ err = proc_state_of(owner_tid) == 'S' ? set_other(owner_tid) : -EAGAIN;
+ sem_post(&owner_go);
+ if (err) {
+ ksft_print_msg("the owner could not leave the class asleep: %s\n",
+ strerror(-err));
+ join(o);
+ return KSFT_FAIL;
+ }
+ }
if (wait_for(5000, owner_holds) || owner_holds != 2) {
ksft_print_msg("the owner never took the lock\n");
join(o);
@@ -1752,6 +1843,27 @@ static int deboost_resets_period(void)
return ret;
}
+/* Run deboost_case() with a plain SCHED_OTHER owner. */
+static int deboost_resets_period(void)
+{
+ return deboost_case(0);
+}
+
+/*
+ * Run deboost_case() with an owner that left the deadline class asleep.
+ * Its bandwidth was released then.
+ */
+static int deboost_released_resets_period(void)
+{
+ return deboost_case(1);
+}
+
+/* Run deboost_case() with an owner that enters the class while boosted. */
+static int deboost_in_class_resets_period(void)
+{
+ return deboost_case(2);
+}
+
/*
* Return the runtime that the deadline task @tid has left, in ns, from
* dl.runtime in /proc/<tid>/sched. Return -1 if it cannot be read.
@@ -1782,14 +1894,16 @@ static long long dl_runtime_left(pid_t tid)
#define REFILL_DONOR_RUNTIME (2 * NSEC_PER_MSEC)
#define REFILL_DONOR_PERIOD (10 * NSEC_PER_MSEC)
-static sem_t refill_donor_ready;
-static atomic_int refill_donor_locking;
+static sem_t refill_donor_ready, refill_measured;
+static atomic_int refill_donor_locking, refill_hold_donor;
static _Atomic long long refill_owner_left;
/*
- * Run as the donor D of deboost_keeps_own_runtime(). Become a deadline
- * task of 2ms every 10ms and sleep until the owner holds pi_lock. Then
- * block on pi_lock. Return NULL once it got the lock, or 1 on an error.
+ * Run as the donor D of refill_case(). Become a deadline task of 2ms
+ * every 10ms and sleep until the owner holds pi_lock. Then block on
+ * pi_lock. With refill_hold_donor, stay a deadline task after the unlock
+ * until refill_measured is posted. Return NULL once it got the lock, or 1
+ * on an error.
*/
static void *refill_donor(void *arg)
{
@@ -1809,16 +1923,18 @@ static void *refill_donor(void *arg)
err = take_pi_lock();
if (!err)
drop_pi_lock();
+ if (refill_hold_donor)
+ sem_wait_sig(&refill_measured);
if (set_other(0))
ksft_print_msg("the donor cannot leave SCHED_DEADLINE\n");
return (void *)(long)err;
}
/*
- * Run as the owner O of deboost_keeps_own_runtime(). @arg is the CPU
- * time, in ms, that O spins with the lock once it is boosted. Store the
- * runtime O has left after the unlock in refill_owner_left. Return NULL,
- * 1 on an error, or 2 if the donor did not block on the lock within 1s.
+ * Run as the owner O of refill_case(). @arg is the CPU time, in ms, that
+ * O spins with the lock once it is boosted. Store the runtime O has left
+ * after the unlock in refill_owner_left. Return NULL, 1 on an error, or 2
+ * if the donor did not block on the lock within 1s.
*/
static void *refill_owner(void *arg)
{
@@ -1865,18 +1981,25 @@ static void *refill_owner(void *arg)
* preempts O: its deadline is
* earlier
* blocks on pi_lock, boosts O
- * runs 1ms more
+ * runs @spin_ms more
* unlocks: the boost ends
* takes pi_lock, unlocks
* reads its dl.runtime
*
* A boosted task that runs out of runtime is not throttled. Its runtime
- * is refilled from the parameters of its donor. Here O has used about 3ms
- * of its own 5ms at the unlock, and was never refilled. It must keep what
- * it has left. A new period would hand it a full runtime at every
- * contended unlock. The case fails if O has 4ms or more.
+ * is refilled from the parameters of its donor: 2ms here.
+ *
+ * - @spin_ms is 8: O runs out of its own 5ms, and is refilled from D at
+ * least once. At the unlock it holds what the last refill left, 2ms at
+ * most. That is not the runtime of its own reservation. O must start a
+ * new period at the unlock, with 5ms. The case fails if O has 3.5ms or
+ * less.
+ * - @spin_ms is 1: O has used about 3ms of its own 5ms at the unlock, and
+ * was never refilled. It must keep what it has left. A new period would
+ * hand it a full runtime at every contended unlock. The case fails if O
+ * has 4ms or more.
*/
-static int deboost_keeps_own_runtime(void)
+static int refill_case(long spin_ms)
{
void *o_ret = NULL, *d_ret = NULL;
int err, ret = KSFT_PASS;
@@ -1903,7 +2026,7 @@ static int deboost_keeps_own_runtime(void)
join(d);
return KSFT_FAIL;
}
- if (pthread_create(&o, NULL, refill_owner, (void *)1L)) {
+ if (pthread_create(&o, NULL, refill_owner, (void *)spin_ms)) {
ksft_print_msg("the owner thread could not start\n");
sem_post(&owner_go);
join(d);
@@ -1930,7 +2053,11 @@ static int deboost_keeps_own_runtime(void)
} else if (left < 0) {
ksft_print_msg("the runtime of the owner cannot be read\n");
ret = KSFT_SKIP;
- } else if (left >= 4 * NSEC_PER_MSEC) {
+ } else if (spin_ms == 8 && left <= 3500 * NSEC_PER_USEC) {
+ ksft_print_msg("the owner has %lld us after the unlock: what the donor's refill left\n",
+ left / NSEC_PER_USEC);
+ ret = KSFT_FAIL;
+ } else if (spin_ms == 1 && left >= 4 * NSEC_PER_MSEC) {
ksft_print_msg("the owner has %lld us after the unlock: a new runtime, with no refill from the donor\n",
left / NSEC_PER_USEC);
ret = KSFT_FAIL;
@@ -1938,6 +2065,157 @@ static int deboost_keeps_own_runtime(void)
return ret;
}
+/* Run refill_case() with an owner that is refilled from its donor. */
+static int deboost_after_donor_refill(void)
+{
+ return refill_case(8);
+}
+
+/* Run refill_case() with an owner that stays on its own runtime. */
+static int deboost_keeps_own_runtime(void)
+{
+ return refill_case(1);
+}
+
+/*
+ * Run as the owner O of deboost_out_after_donor_refill(). Store total_bw
+ * of O's root domain right after the unlock in refill_owner_left. Return
+ * NULL, 1 on an error, or 2 if the donor did not block on the lock within
+ * 1s.
+ */
+static void *former_owner(void *arg)
+{
+ uint64_t t0;
+ int err;
+
+ owner_tid = gettid_();
+ err = set_deadline(0, REFILL_OWN_RUNTIME, PERIOD);
+ if (err) {
+ ksft_print_msg("the owner refused as a deadline task: %s\n", strerror(-err));
+ return (void *)1;
+ }
+ spin_ns(2 * NSEC_PER_MSEC);
+ if (take_pi_lock())
+ return (void *)1;
+ if (set_other(0)) {
+ ksft_print_msg("the owner cannot leave SCHED_DEADLINE\n");
+ drop_pi_lock();
+ return (void *)1;
+ }
+ owner_holds = 2;
+ sem_post(&owner_go);
+ t0 = now_ns();
+ while (!refill_donor_locking || proc_state_of(donor_tid) != 'S') {
+ if (now_ns() - t0 > NSEC_PER_SEC) {
+ drop_pi_lock();
+ return (void *)2;
+ }
+ }
+ spin_ns(8 * NSEC_PER_MSEC);
+ drop_pi_lock();
+ refill_owner_left = total_bw_of(top_cpu(1));
+ sem_post(&refill_measured);
+ return NULL;
+}
+
+/*
+ * As refill_case() with @spin_ms 8, but O leaves SCHED_DEADLINE once it
+ * holds pi_lock. O and D share the CPU of P.
+ *
+ * owner O: 5ms every 100ms donor D: 2ms every 10ms
+ *
+ * sleeps
+ * starts its period, runs 2ms
+ * takes pi_lock
+ * switches to SCHED_OTHER: its 0-lag
+ * time is about 40ms ahead
+ * wakes D up
+ * preempts O
+ * blocks on pi_lock, boosts O
+ * runs 8ms: runs out of its own
+ * runtime, is refilled from D
+ * unlocks: leaves the class
+ * takes pi_lock, unlocks, waits
+ * reads total_bw
+ *
+ * At the unlock, O's runtime and deadline come from D's refills. O must
+ * start a period of its own there. Its 0-lag time is then now, and its
+ * bandwidth must be gone from total_bw at once. Mixed with O's own
+ * parameters, the runtime and deadline of the refills put the 0-lag time
+ * about 100ms ahead.
+ */
+static int deboost_out_after_donor_refill(void)
+{
+ void *o_ret = NULL, *d_ret = NULL;
+ int err, ret = KSFT_PASS;
+ long long base;
+ pthread_t o, d;
+
+ if (total_bw_of(top_cpu(1)) < 0) {
+ ksft_print_msg("sched_debug tells of no total_bw\n");
+ return KSFT_SKIP;
+ }
+ if (pi_init())
+ return KSFT_SKIP;
+ if (sem_init(&refill_donor_ready, 0, 0) || sem_init(&refill_measured, 0, 0)) {
+ ksft_print_msg("sem_init: %s\n", strerror(errno));
+ pi_fini();
+ return KSFT_SKIP;
+ }
+ refill_donor_locking = 0;
+ refill_hold_donor = 1;
+ refill_owner_left = -1;
+ donor_tid = 0;
+ if (pthread_create(&d, NULL, refill_donor, NULL)) {
+ ksft_print_msg("the donor thread could not start\n");
+ refill_hold_donor = 0;
+ return KSFT_FAIL;
+ }
+ if (sem_wait_for(&refill_donor_ready, 5000) || donor_tid <= 0) {
+ ksft_print_msg("the donor is not ready\n");
+ sem_post(&owner_go);
+ sem_post(&refill_measured);
+ join(d);
+ refill_hold_donor = 0;
+ return KSFT_FAIL;
+ }
+ base = total_bw_of(top_cpu(1)); /* with D, without O */
+ if (pthread_create(&o, NULL, former_owner, NULL)) {
+ ksft_print_msg("the owner thread could not start\n");
+ sem_post(&owner_go);
+ sem_post(&refill_measured);
+ join(d);
+ refill_hold_donor = 0;
+ return KSFT_FAIL;
+ }
+ err = pthread_join(o, &o_ret);
+ if (err) {
+ ksft_print_msg("pthread_join: %s\n", strerror(err));
+ ret = KSFT_FAIL;
+ }
+ if (o_ret == (void *)1)
+ sem_post(&owner_go); /* the owner never woke the donor up */
+ if (o_ret)
+ sem_post(&refill_measured);
+ err = pthread_join(d, &d_ret);
+ if (err) {
+ ksft_print_msg("pthread_join: %s\n", strerror(err));
+ ret = KSFT_FAIL;
+ }
+ refill_hold_donor = 0;
+ if (o_ret == (void *)2) {
+ ksft_print_msg("the donor never blocked on the lock\n");
+ ret = KSFT_FAIL;
+ } else if (o_ret || d_ret) {
+ ret = KSFT_FAIL; /* the thread said why */
+ } else if (refill_owner_left != base) {
+ ksft_print_msg("total_bw %lld right after the unlock, %lld expected\n",
+ (long long)refill_owner_left, base);
+ ret = KSFT_FAIL;
+ }
+ return ret;
+}
+
static sem_t ret_ready, ret_go;
static _Atomic pid_t ret_tid;
static atomic_int ret_first_period_ms;
@@ -2032,7 +2310,7 @@ static pthread_mutex_t pi_lock2;
static sem_t holder_go;
static atomic_int holder_holds, chain_waits, donor_done, x_waits, l2_takes;
static atomic_int chain_rank, x_rank;
-static _Atomic pid_t x_tid;
+static _Atomic pid_t x_tid, fair_tid;
/* The holder thread: take pi_lock2, and hold it until holder_go. */
static void *l2_holder(void *arg)
@@ -2190,6 +2468,7 @@ static int pi_init_lock2(void)
x_rank = -1;
donor_tid = 0;
x_tid = 0;
+ fair_tid = 0;
return 0;
}
@@ -2288,6 +2567,129 @@ static int deboost_blocked_keeps_order(void)
return ret;
}
+/*
+ * The fair waiter: wait on pi_lock2 as a fair task. With @arg set, hold
+ * pi_lock meanwhile, for a donor to boost it. Otherwise the main thread
+ * makes it a deadline task while it waits. It leaves the class at the end.
+ */
+static void *fair_waiter(void *arg)
+{
+ int err;
+
+ fair_tid = gettid_();
+ if (arg && take_pi_lock())
+ return (void *)1;
+ chain_waits = 1;
+ err = take_l2_ranked(&chain_rank);
+ if (arg)
+ drop_pi_lock();
+ else if (set_other(0))
+ ksft_print_msg("a waiter cannot leave SCHED_DEADLINE\n");
+ return err ? (void *)1 : NULL;
+}
+
+/*
+ * Check the order of the waiters of pi_lock2 when one of them enters the
+ * class while it waits. The holder has pi_lock2:
+ * 1. The fair waiter waits on pi_lock2. With @boost, it holds pi_lock.
+ * 2. The other waiter, a deadline task, waits on pi_lock2.
+ * 3. The fair waiter enters the class:
+ * - Without @boost, the main thread makes it a deadline task of 10ms
+ * every 200ms.
+ * - With @boost, a donor of 40ms every 100ms waits on pi_lock, and
+ * boosts it.
+ * 4. The holder drops pi_lock2.
+ *
+ * The other waiter must take pi_lock2 first: it started its period
+ * earlier, with a shorter deadline. At step 3 the fair waiter is not
+ * queued, and the chain walk copies its deadline into its waiter node on
+ * pi_lock2. A deadline of 0 in the node would put the fair waiter first.
+ */
+static int blocked_keeps_order(long boost)
+{
+ void *f_ret = NULL, *x_ret = NULL, *w_ret = NULL;
+ int ret = KSFT_FAIL, have_f = 0, have_x = 0, have_w = 0, err;
+ pthread_t h, f, x, w;
+
+ if (pi_init())
+ return KSFT_SKIP;
+ if (pi_init_lock2()) {
+ pi_fini();
+ return KSFT_SKIP;
+ }
+ if (pthread_create(&h, NULL, l2_holder, NULL)) {
+ ksft_print_msg("the holder thread could not start\n");
+ goto out_fini;
+ }
+ if (wait_for(5000, holder_holds) || holder_holds != 1) {
+ ksft_print_msg("the holder never took pi_lock2\n");
+ goto out;
+ }
+ have_f = !pthread_create(&f, NULL, fair_waiter, (void *)boost);
+ if (!have_f || wait_for(1000, chain_waits && proc_state_of(fair_tid) == 'S')) {
+ ksft_print_msg("the fair waiter never waited on pi_lock2\n");
+ goto out;
+ }
+ have_x = !pthread_create(&x, NULL, early_waiter, NULL);
+ if (!have_x || wait_for(1000, x_waits && proc_state_of(x_tid) == 'S')) {
+ ksft_print_msg("the other waiter never waited on pi_lock2\n");
+ goto out;
+ }
+ if (boost) {
+ have_w = !pthread_create(&w, NULL, donor_waiter, NULL);
+ if (!have_w || wait_for(1000, donor_tid && proc_state_of(donor_tid) == 'S')) {
+ ksft_print_msg("the donor never waited on pi_lock\n");
+ goto out;
+ }
+ if (prio_of(fair_tid) != DL_TASK_PRIO) {
+ ksft_print_msg("the donor never boosted the fair waiter\n");
+ goto out;
+ }
+ } else {
+ err = set_deadline(fair_tid, 10 * NSEC_PER_MSEC, 2 * PERIOD);
+ if (err) {
+ ksft_print_msg("the fair waiter refused as a deadline task: %s\n",
+ strerror(-err));
+ if (err == -EBUSY)
+ ret = KSFT_SKIP; /* no room in P */
+ goto out;
+ }
+ }
+ ret = KSFT_PASS;
+out:
+ sem_post(&holder_go); /* pi_lock2 goes to its top waiter */
+ if (have_w && pthread_join(w, &w_ret))
+ ret = KSFT_FAIL;
+ if (have_x && pthread_join(x, &x_ret))
+ ret = KSFT_FAIL;
+ if (have_f && pthread_join(f, &f_ret))
+ ret = KSFT_FAIL;
+ join(h);
+ if (f_ret || x_ret || w_ret) {
+ ret = KSFT_FAIL; /* the thread said why */
+ } else if (ret == KSFT_PASS && chain_rank < x_rank) {
+ ksft_print_msg("the fair waiter took pi_lock2 before the earlier waiter\n");
+ ret = KSFT_FAIL;
+ }
+out_fini:
+ pthread_mutex_destroy(&pi_lock2);
+ sem_destroy(&holder_go);
+ pi_fini();
+ return ret;
+}
+
+/* sched_setattr() puts the fair waiter in the class. */
+static int setattr_blocked_keeps_order(void)
+{
+ return blocked_keeps_order(0);
+}
+
+/* A boost puts the fair waiter in the class. */
+static int boost_blocked_keeps_order(void)
+{
+ return blocked_keeps_order(1);
+}
+
/*
* The threads of pi_chain_donor_leaves(): O, M, W, Z and Y. leave_tid[]
* holds their tids. leave_y_ns is the CPU time Y gets in O's window, or
@@ -9948,6 +10350,7 @@ static const struct p_case {
} p_cases[] = {
{ "pi_deadline_owner", pi_deadline_owner, MEASURED_IN_P },
{ "pi_other_owner_asleep", pi_other_owner_asleep, MEASURED_IN_P },
+ { "pi_former_owner_asleep", pi_former_owner_asleep, MEASURED_IN_P },
{ "pi_unlock_running", pi_unlock_running, MEASURED_IN_P },
{ "pi_chain_two_owners", pi_chain_two_owners, MEASURED_IN_P },
{ "pi_departed_moved", pi_departed_moved, FROM_OUTSIDE },
@@ -9961,9 +10364,15 @@ static const struct p_case {
{ "params_keep_boosted", params_keep_boosted, MEASURED_IN_P },
{ "boosted_owner_policy", boosted_owner_policy, MEASURED_IN_P },
{ "deboost_resets_period", deboost_resets_period, MEASURED_IN_P },
+ { "deboost_released_resets_period", deboost_released_resets_period, MEASURED_IN_P },
+ { "deboost_in_class_resets_period", deboost_in_class_resets_period, MEASURED_IN_P },
+ { "deboost_after_donor_refill", deboost_after_donor_refill, MEASURED_IN_P },
+ { "deboost_out_after_donor_refill", deboost_out_after_donor_refill, MEASURED_IN_P },
{ "deboost_keeps_own_runtime", deboost_keeps_own_runtime, MEASURED_IN_P },
{ "return_asleep_new_params", return_asleep_new_params, MEASURED_IN_P },
{ "deboost_blocked_keeps_order", deboost_blocked_keeps_order, MEASURED_IN_P },
+ { "setattr_blocked_keeps_order", setattr_blocked_keeps_order, MEASURED_IN_P },
+ { "boost_blocked_keeps_order", boost_blocked_keeps_order, MEASURED_IN_P },
{ "pi_chain_donor_leaves", pi_chain_donor_leaves, MEASURED_IN_P },
{ "pi_chain_top_moves", pi_chain_top_moves, MEASURED_IN_P },
{ "pi_chain_top_moves_rt", pi_chain_top_moves_rt, MEASURED_IN_P },
--
2.47.3
^ permalink raw reply related [flat|nested] 35+ messages in thread
* [PATCH v8 29/33] sched/deadline: Cancel the replenishment timer with the parameters
2026-11-19 20:15 [PATCH v8 00/33] sched/deadline: bandwidth accounting fixes and selftests Mathieu Desnoyers
` (27 preceding siblings ...)
2026-11-19 20:16 ` [PATCH v8 28/33] selftests/sched: Test deadline boosts in deadline_bw_test Mathieu Desnoyers
@ 2026-11-19 20:16 ` Mathieu Desnoyers
2026-11-19 20:16 ` [PATCH v8 30/33] sched/deadline: Cap the runtime left of a constrained task at dl_runtime Mathieu Desnoyers
` (3 subsequent siblings)
32 siblings, 0 replies; 35+ messages in thread
From: Mathieu Desnoyers @ 2026-11-19 20:16 UTC (permalink / raw)
To: juri.lelli, peterz, mingo, vincent.guittot
Cc: Mathieu Desnoyers, tglx, will, boqun, dietmar.eggemann, rostedt,
bsegall, mgorman, vschneid, kprateek.nayak, luca.abeni,
mathieu.poirier, longman, ridong.chen, tj, hannes, mkoutny, shuah,
bigeasy, clrkwllms, parri.andrea, m.szyprowski, kayracizmeci,
weihengqiuu, linux-kernel, cgroups, linux-kselftest,
linux-rt-devel, stable
A task that leaves SCHED_DEADLINE has its parameters cleared at its
0-lag time. Its replenishment timer can still be armed then. If the
task comes back and throttles before that timer fires, start_dl_timer()
does not restart it, since it is queued. The replenishment would then
come at the period of the reservation the task left. Too early, it would
give the task runtime it has not earned. Too late, it would delay the
task by up to a period.
Commit a649f237db18 ("sched,dl: Fix sched class hopping CBS hole") armed
the replenishment timer when a task leaves the class. The timer then
reset the task if it was still out. Commit 209a0cbda7a0
("sched/deadline: Improve the tracking of active utilization") moved
that reset to the inactive timer, at the 0-lag time. Nothing cancels
the replenishment timer there.
The timer of a throttled task fires at its next period, which can come
after the 0-lag time:
- A task that yielded is throttled with runtime left. Its 0-lag time
comes before its deadline.
- A constrained-deadline task that overran by little reaches its 0-lag
time before the end of its period.
Fix this by cancelling the timer where the parameters are cleared. This
includes task_non_contending(). switched_from_dl() calls it to release
the bandwidth of a task that leaves asleep past its 0-lag time. A
constrained task that overran as it blocked has the timer armed there
too.
The Sashiko AI review of this series found this. It has no reproducer.
Fixes: 209a0cbda7a0 ("sched/deadline: Improve the tracking of active utilization")
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Reviewed-by: Juri Lelli <juri.lelli@redhat.com>
Cc: stable@vger.kernel.org
---
kernel/sched/deadline.c | 2 ++
1 file changed, 2 insertions(+)
diff --git a/kernel/sched/deadline.c b/kernel/sched/deadline.c
index d284036ecb8a..95db8afd8353 100644
--- a/kernel/sched/deadline.c
+++ b/kernel/sched/deadline.c
@@ -788,6 +788,7 @@ static void task_non_contending(struct sched_dl_entity *dl_se, bool dl_task)
if (dl_task && READ_ONCE(p->__state) == TASK_DEAD)
sub_rq_bw(dl_se, &rq->dl);
dl_task_release_bw(p);
+ cancel_replenish_timer(dl_se);
__dl_clear_params(dl_se);
}
}
@@ -2643,6 +2644,7 @@ static enum hrtimer_restart inactive_task_timer(struct hrtimer *timer)
}
dl_task_release_bw(p);
+ cancel_replenish_timer(dl_se);
__dl_clear_params(dl_se);
goto unlock;
--
2.47.3
^ permalink raw reply related [flat|nested] 35+ messages in thread
* [PATCH v8 30/33] sched/deadline: Cap the runtime left of a constrained task at dl_runtime
2026-11-19 20:15 [PATCH v8 00/33] sched/deadline: bandwidth accounting fixes and selftests Mathieu Desnoyers
` (28 preceding siblings ...)
2026-11-19 20:16 ` [PATCH v8 29/33] sched/deadline: Cancel the replenishment timer with the parameters Mathieu Desnoyers
@ 2026-11-19 20:16 ` Mathieu Desnoyers
2026-11-19 20:16 ` [PATCH v8 31/33] sched/deadline: Check the runtime of a task that comes back runnable Mathieu Desnoyers
` (2 subsequent siblings)
32 siblings, 0 replies; 35+ messages in thread
From: Mathieu Desnoyers @ 2026-11-19 20:16 UTC (permalink / raw)
To: juri.lelli, peterz, mingo, vincent.guittot
Cc: Mathieu Desnoyers, tglx, will, boqun, dietmar.eggemann, rostedt,
bsegall, mgorman, vschneid, kprateek.nayak, luca.abeni,
mathieu.poirier, longman, ridong.chen, tj, hannes, mkoutny, shuah,
bigeasy, clrkwllms, parri.andrea, m.szyprowski, kayracizmeci,
weihengqiuu, linux-kernel, cgroups, linux-kselftest,
linux-rt-devel, stable
sched_setattr() can change the parameters of a SCHED_DEADLINE task. The
task keeps its runtime and its absolute deadline: the new parameters
take effect at the next period. At the next wakeup, update_dl_entity()
checks the runtime left against the new parameters.
For a constrained task, that check uses the density, dl_runtime /
dl_deadline. Commit 2317d5f1c349 ("sched/deadline: Use deadline instead
of period when calculating overflow") made dl_entity_overflow() compare
with it: "We care about if the runtime can make its deadline, not its
period." Commit 3effcb4247e7 ("sched/deadline: Use the revised wakeup
rule for suspending constrained dl tasks") then cut the runtime of a
constrained task that overflows to density * laxity. Both bound the
runtime by dl_runtime only while the deadline is at most dl_deadline
ahead.
A change to a shorter dl_deadline breaks that. The old deadline can be
farther ahead. The check then keeps more runtime than the new
dl_runtime, or computes more. For example, take a task of 50ms every
100ms with 49ms left. Changed to 8ms in 20ms every 100ms, it would wake
with 39.6ms. It would then run far more than its new runtime in this
period. Admission control already counts the new, smaller bandwidth. So
other deadline tasks could miss their deadlines.
An implicit task is not affected. Its check compares with its
bandwidth, which bounds the runtime for any deadline.
Fix this by keeping no more than dl_runtime before the check, for a
constrained task. A boosted task runs on the parameters of its donor,
and is left alone. The runtime is signed and can be negative there, so
compare it as signed.
The cases change_constrained_new_params and wake_constrained_new_params,
added to deadline_bw_test later in this series, test this. The first
changes a running task, the second a sleeping one. Both fail without
this patch.
The Sashiko AI review of this series found this.
Fixes: 2317d5f1c349 ("sched/deadline: Use deadline instead of period when calculating overflow")
Fixes: 3effcb4247e7 ("sched/deadline: Use the revised wakeup rule for suspending constrained dl tasks")
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Cc: stable@vger.kernel.org
---
kernel/sched/deadline.c | 19 +++++++++++++++++++
1 file changed, 19 insertions(+)
diff --git a/kernel/sched/deadline.c b/kernel/sched/deadline.c
index 95db8afd8353..01497171919d 100644
--- a/kernel/sched/deadline.c
+++ b/kernel/sched/deadline.c
@@ -1468,6 +1468,25 @@ static void update_dl_entity(struct sched_dl_entity *dl_se)
{
struct rq *rq = rq_of_dl_se(dl_se);
+ /*
+ * The checks below compare the runtime left with the density,
+ * dl_runtime / dl_deadline. That bounds the runtime by dl_runtime
+ * only while the deadline is at most dl_deadline ahead. After a
+ * change to a shorter dl_deadline, the deadline can be farther
+ * ahead, and the runtime left larger than dl_runtime. A constrained
+ * task would then run more than dl_runtime in a period. Keep no
+ * more than dl_runtime.
+ *
+ * An implicit task is checked against its bandwidth, which holds
+ * for any deadline. A boosted task runs on the parameters of its
+ * donor. Leave both alone.
+ *
+ * The runtime is signed, and can be negative here.
+ */
+ if (!dl_is_implicit(dl_se) && !is_dl_boosted(dl_se) &&
+ dl_se->runtime > (s64)dl_se->dl_runtime)
+ dl_se->runtime = dl_se->dl_runtime;
+
if (dl_time_before(dl_se->deadline, rq_clock(rq)) ||
dl_entity_overflow(dl_se, rq_clock(rq))) {
--
2.47.3
^ permalink raw reply related [flat|nested] 35+ messages in thread
* [PATCH v8 31/33] sched/deadline: Check the runtime of a task that comes back runnable
2026-11-19 20:15 [PATCH v8 00/33] sched/deadline: bandwidth accounting fixes and selftests Mathieu Desnoyers
` (29 preceding siblings ...)
2026-11-19 20:16 ` [PATCH v8 30/33] sched/deadline: Cap the runtime left of a constrained task at dl_runtime Mathieu Desnoyers
@ 2026-11-19 20:16 ` Mathieu Desnoyers
2026-11-19 20:16 ` [PATCH v8 32/33] sched/deadline: Check the runtime of a task whose parameters change Mathieu Desnoyers
2026-11-19 20:16 ` [PATCH v8 33/33] selftests/sched: Test parameter changes in deadline_bw_test Mathieu Desnoyers
32 siblings, 0 replies; 35+ messages in thread
From: Mathieu Desnoyers @ 2026-11-19 20:16 UTC (permalink / raw)
To: juri.lelli, peterz, mingo, vincent.guittot
Cc: Mathieu Desnoyers, tglx, will, boqun, dietmar.eggemann, rostedt,
bsegall, mgorman, vschneid, kprateek.nayak, luca.abeni,
mathieu.poirier, longman, ridong.chen, tj, hannes, mkoutny, shuah,
bigeasy, clrkwllms, parri.andrea, m.szyprowski, kayracizmeci,
weihengqiuu, linux-kernel, cgroups, linux-kselftest,
linux-rt-devel, stable
Commit a649f237db18 ("sched,dl: Fix sched class hopping CBS hole")
keeps the runtime and the deadline of a task that leaves
SCHED_DEADLINE. A task that comes back before its deadline goes on with
them. So leaving the class and coming back gives no new runtime.
But the task can come back with new parameters. If it is asleep, its
wakeup checks the runtime left against them, with dl_entity_overflow().
If it is runnable, enqueue_dl_entity() starts a new period only if the
deadline has passed. The runtime left is not checked.
So a task can come back with a smaller runtime, and still run on the
old one. Admission control counts only the new one. Juri Lelli found
this case. On v7.4-rc2, a task of 50ms every 100ms came back through
SCHED_OTHER with 10ms every 100ms. It ran 39ms of the next 100ms.
Fix this by applying the wakeup rule to a task that enters the class
runnable, with update_dl_entity(). It starts a new period, or reduces
the runtime of a constrained task, if the runtime left does not fit the
new parameters. The task gets no more than a wakeup would give it.
The return_running_new_params case, added to deadline_bw_test later in
this series, tests this. It fails on v7.4-rc2 and passes with this
patch. The return_asleep_new_params case tests the wakeup, and passes on
both.
Fixes: a649f237db18 ("sched,dl: Fix sched class hopping CBS hole")
Reported-by: Juri Lelli <juri.lelli@redhat.com>
Closes: https://lore.kernel.org/all/avHvPjz4dL0zNyGb@jlelli-thinkpadt14gen4.remote.csb/
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Cc: stable@vger.kernel.org
---
kernel/sched/deadline.c | 8 ++++++++
1 file changed, 8 insertions(+)
diff --git a/kernel/sched/deadline.c b/kernel/sched/deadline.c
index 01497171919d..7ed2f525b883 100644
--- a/kernel/sched/deadline.c
+++ b/kernel/sched/deadline.c
@@ -2913,6 +2913,14 @@ enqueue_dl_entity(struct sched_dl_entity *dl_se, int flags)
update_dl_entity(dl_se);
} else if (flags & ENQUEUE_REPLENISH) {
replenish_dl_entity(dl_se);
+ } else if ((flags & ENQUEUE_CLASS) && !is_dl_boosted(dl_se)) {
+ /*
+ * The task enters the class runnable. It may come back before
+ * its deadline, with the runtime left from older parameters.
+ * Check that runtime against the new parameters, as a wakeup
+ * does.
+ */
+ update_dl_entity(dl_se);
} else if ((flags & ENQUEUE_MOVE) &&
!is_dl_boosted(dl_se) &&
dl_time_before(dl_se->deadline, rq_clock(rq_of_dl_se(dl_se)))) {
--
2.47.3
^ permalink raw reply related [flat|nested] 35+ messages in thread
* [PATCH v8 32/33] sched/deadline: Check the runtime of a task whose parameters change
2026-11-19 20:15 [PATCH v8 00/33] sched/deadline: bandwidth accounting fixes and selftests Mathieu Desnoyers
` (30 preceding siblings ...)
2026-11-19 20:16 ` [PATCH v8 31/33] sched/deadline: Check the runtime of a task that comes back runnable Mathieu Desnoyers
@ 2026-11-19 20:16 ` Mathieu Desnoyers
2026-11-19 20:16 ` [PATCH v8 33/33] selftests/sched: Test parameter changes in deadline_bw_test Mathieu Desnoyers
32 siblings, 0 replies; 35+ messages in thread
From: Mathieu Desnoyers @ 2026-11-19 20:16 UTC (permalink / raw)
To: juri.lelli, peterz, mingo, vincent.guittot
Cc: Mathieu Desnoyers, tglx, will, boqun, dietmar.eggemann, rostedt,
bsegall, mgorman, vschneid, kprateek.nayak, luca.abeni,
mathieu.poirier, longman, ridong.chen, tj, hannes, mkoutny, shuah,
bigeasy, clrkwllms, parri.andrea, m.szyprowski, kayracizmeci,
weihengqiuu, linux-kernel, cgroups, linux-kselftest,
linux-rt-devel, stable
Commit 40767b0dc768 ("sched/deadline: Fix deadline parameter
modification handling") lets new parameters take effect at the next
period. So sched_setattr() cannot give a task a fresh runtime.
But admission control counts the new bandwidth at once. When a task
shrinks its reservation, the runtime left from the old one is not
checked against the new one. A comment in sched_dl_overflow() names
the gap: the old bandwidth should stay counted until the 0-lag time.
So a running task can shrink its reservation, and still run on the old
one. Other tasks can meanwhile be admitted into the bandwidth it gave
back. On v7.4-rc2, a task of 50ms every 100ms changed to 10ms every
100ms. It ran 39ms of the next 100ms.
Fix this by applying the wakeup rule after a change of parameters, as
the previous patch does for a task that enters the class.
__setparam_dl() marks a change of the runtime, the deadline or the
period. The next enqueue then calls update_dl_entity(). It starts a new
period, or reduces the runtime of a constrained task, if the runtime
left does not fit. The task gets no more than a wakeup would give it. A
boosted task runs on the parameters of its donor. So it keeps the mark
until its deboost. dl_deboost() checks it there, before
rt_mutex_setprio() compares the task with its top waiter.
A larger reservation still takes effect at the next period.
The enqueue flags cannot tell a change of parameters: a cgroup move
also uses ENQUEUE_MOVE. Without the mark, a task that waited behind
others could fail the check, and get a later deadline for no reason.
The change_running_new_params case, added to deadline_bw_test later in
this series, tests this. change_boosted_new_params tests a task that
changes its parameters while boosted. Both fail on v7.4-rc2 and pass
with this patch. deboost_new_params_reboost tests the check at the
deboost.
Fixes: 40767b0dc768 ("sched/deadline: Fix deadline parameter modification handling")
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Cc: stable@vger.kernel.org
---
include/linux/sched.h | 5 +++++
kernel/sched/deadline.c | 38 +++++++++++++++++++++++++++++++++-----
2 files changed, 38 insertions(+), 5 deletions(-)
diff --git a/include/linux/sched.h b/include/linux/sched.h
index 2f2f59402f09..86585c663e4b 100644
--- a/include/linux/sched.h
+++ b/include/linux/sched.h
@@ -715,6 +715,10 @@ struct sched_dl_entity {
* @dl_bw_oncpu tells if the this_bw of a runqueue counts the task's
* @dl_bw: add_rq_bw() and sub_rq_bw() set and clear it, under that
* runqueue's lock. Not for servers, which keep @dl_bw_attached.
+ *
+ * @dl_new_params tells if sched_setattr() changed the runtime, the
+ * deadline or the period. The next enqueue that is not boosted, or
+ * the deboost, checks the runtime left against them, and clears it.
*/
unsigned int dl_throttled : 1;
unsigned int dl_yielded : 1;
@@ -729,6 +733,7 @@ struct sched_dl_entity {
unsigned int dl_defer_idle : 1;
unsigned int dl_bw_attached : 1;
unsigned int dl_bw_oncpu : 1;
+ unsigned int dl_new_params : 1;
/*
* Bandwidth enforcement timer. Each -deadline task has its
diff --git a/kernel/sched/deadline.c b/kernel/sched/deadline.c
index 7ed2f525b883..2849484428c6 100644
--- a/kernel/sched/deadline.c
+++ b/kernel/sched/deadline.c
@@ -114,6 +114,8 @@ static inline void dl_note_donor_refill(struct sched_dl_entity *dl_se)
dl_se->dl_donor_refill = 1;
}
+static void update_dl_entity(struct sched_dl_entity *dl_se);
+
/*
* End the boost of @dl_se, whose task stays in the class. If the boost
* refilled its runtime from the donor's parameters, start a new period
@@ -122,6 +124,10 @@ static inline void dl_note_donor_refill(struct sched_dl_entity *dl_se)
* first. Without a refill, the task keeps the runtime and the deadline
* of its own reservation.
*
+ * If sched_setattr() changed the parameters during the boost, check the
+ * runtime left against them now, for the same reason. rt_mutex_setprio()
+ * then compares the result with the top waiter.
+ *
* The caller holds the rq lock and the pi_lock of the task, and has
* updated the rq clock.
*/
@@ -130,6 +136,12 @@ void dl_deboost(struct sched_dl_entity *dl_se)
if (dl_se->dl_donor_refill)
dl_start_own_period(dl_se);
dl_drop_boost(dl_se);
+
+ if (dl_se->dl_new_params) {
+ dl_se->dl_new_params = 0;
+ if (!dl_se->dl_throttled)
+ update_dl_entity(dl_se);
+ }
}
/* Return true if @p waits on an rt_mutex. The caller holds its pi_lock. */
@@ -2864,7 +2876,16 @@ static void __dequeue_dl_entity(struct sched_dl_entity *dl_se)
static void
enqueue_dl_entity(struct sched_dl_entity *dl_se, int flags)
{
+ bool new_params = dl_se->dl_new_params;
+
WARN_ON_ONCE(on_dl_rq(dl_se));
+ /*
+ * A boosted task is refilled from its donor. Keep the mark until the
+ * runtime is checked against its own parameters. dl_deboost() checks
+ * it at the deboost, and an enqueue that is not boosted checks it.
+ */
+ if (!is_dl_boosted(dl_se))
+ dl_se->dl_new_params = 0;
update_stats_enqueue_dl(dl_rq_of_se(dl_se), dl_se, flags);
@@ -2913,12 +2934,13 @@ enqueue_dl_entity(struct sched_dl_entity *dl_se, int flags)
update_dl_entity(dl_se);
} else if (flags & ENQUEUE_REPLENISH) {
replenish_dl_entity(dl_se);
- } else if ((flags & ENQUEUE_CLASS) && !is_dl_boosted(dl_se)) {
+ } else if (((flags & ENQUEUE_CLASS) || new_params) &&
+ !is_dl_boosted(dl_se)) {
/*
- * The task enters the class runnable. It may come back before
- * its deadline, with the runtime left from older parameters.
- * Check that runtime against the new parameters, as a wakeup
- * does.
+ * The task enters the class, or its parameters change, while
+ * it is runnable. It may have runtime left from older
+ * parameters. Check that runtime against the new parameters,
+ * as a wakeup does.
*/
update_dl_entity(dl_se);
} else if ((flags & ENQUEUE_MOVE) &&
@@ -4971,6 +4993,11 @@ void __setparam_dl(struct task_struct *p, const struct sched_attr *attr)
u64 period = attr->sched_period ?: attr->sched_deadline;
bool held;
+ if (attr->sched_runtime != dl_se->dl_runtime ||
+ attr->sched_deadline != dl_se->dl_deadline ||
+ period != dl_se->dl_period)
+ dl_se->dl_new_params = 1;
+
/*
* Change the bandwidth that the runqueue counts here, with the
* parameters. A task that priority inheritance boosted into the
@@ -5104,6 +5131,7 @@ static void __dl_clear_params(struct sched_dl_entity *dl_se)
dl_se->dl_defer_running = 0;
dl_se->dl_defer_armed = 0;
dl_se->dl_bw_oncpu = 0;
+ dl_se->dl_new_params = 0;
#ifdef CONFIG_RT_MUTEXES
dl_se->pi_se = dl_se;
--
2.47.3
^ permalink raw reply related [flat|nested] 35+ messages in thread
* [PATCH v8 33/33] selftests/sched: Test parameter changes in deadline_bw_test
2026-11-19 20:15 [PATCH v8 00/33] sched/deadline: bandwidth accounting fixes and selftests Mathieu Desnoyers
` (31 preceding siblings ...)
2026-11-19 20:16 ` [PATCH v8 32/33] sched/deadline: Check the runtime of a task whose parameters change Mathieu Desnoyers
@ 2026-11-19 20:16 ` Mathieu Desnoyers
32 siblings, 0 replies; 35+ messages in thread
From: Mathieu Desnoyers @ 2026-11-19 20:16 UTC (permalink / raw)
To: juri.lelli, peterz, mingo, vincent.guittot
Cc: Mathieu Desnoyers, tglx, will, boqun, dietmar.eggemann, rostedt,
bsegall, mgorman, vschneid, kprateek.nayak, luca.abeni,
mathieu.poirier, longman, ridong.chen, tj, hannes, mkoutny, shuah,
bigeasy, clrkwllms, parri.andrea, m.szyprowski, kayracizmeci,
weihengqiuu, linux-kernel, cgroups, linux-kselftest,
linux-rt-devel
Add the six cases of deadline_bw_test that test the fixes of parameter
changes, earlier in this series. In each, a task with runtime left
changes its parameters, or comes back to the class with new ones:
- a running task that leaves the class and comes back at once
- a running or sleeping task that shrinks its reservation, to an
implicit or a constrained deadline
- a boosted task that changes its parameters, which the deboost then
checks
The comment above each case gives its scenario.
On v7.4-rc2, in 8 VMs of 8 CPUs, five of these cases fail in every VM:
- change_boosted_new_params
- change_constrained_new_params
- change_running_new_params
- return_running_new_params
- wake_constrained_new_params
deboost_new_params_reboost passes there and with the first series.
There, a task made SCHED_DEADLINE while it waits on a PI mutex keeps a
deadline of 0. So the case passes without the deboost it was written
for, until "sched/deadline: Start a period for a task that enters the
class blocked".
With the whole series, every case passes wherever it runs, PREEMPT_RT
included. The isolcpus cases ran in guests booted with
isolcpus=domain,1,2 or isolcpus=domain,0,1. No kernel warning is
logged.
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
---
.../sched/deadline/deadline_bw_test.c | 447 ++++++++++++++++--
1 file changed, 413 insertions(+), 34 deletions(-)
diff --git a/tools/testing/selftests/sched/deadline/deadline_bw_test.c b/tools/testing/selftests/sched/deadline/deadline_bw_test.c
index fe63c8f093e7..1e9d2a9c90f6 100644
--- a/tools/testing/selftests/sched/deadline/deadline_bw_test.c
+++ b/tools/testing/selftests/sched/deadline/deadline_bw_test.c
@@ -283,6 +283,20 @@ static int set_deadline(pid_t tid, uint64_t runtime, uint64_t period)
return set_deadline_flags(tid, runtime, period, 0);
}
+/* Make @tid a deadline task with a relative deadline shorter than @period. */
+static int set_constrained(pid_t tid, uint64_t runtime, uint64_t deadline, uint64_t period)
+{
+ struct dl_attr attr = {
+ .size = sizeof(attr),
+ .sched_policy = SCHED_DEADLINE,
+ .sched_runtime = runtime,
+ .sched_deadline = deadline,
+ .sched_period = period,
+ };
+
+ return syscall(SYS_sched_setattr, tid, &attr, 0) ? -errno : 0;
+}
+
static int set_other(pid_t tid)
{
struct dl_attr attr = { .size = sizeof(attr), .sched_policy = SCHED_OTHER };
@@ -2221,17 +2235,30 @@ static _Atomic pid_t ret_tid;
static atomic_int ret_first_period_ms;
/*
- * Run as the thread of return_asleep_new_params(). Become a deadline task
- * of 50ms every 100ms and run 20ms. 30ms of runtime are left, and the
- * 0-lag time is about 20ms ahead. Then sleep. The main thread switches
- * the thread to SCHED_OTHER and back, with 10ms every 100ms. Then spin
- * for 100ms, and store the CPU time used, in ms, in ret_first_period_ms.
- * Return NULL, or 1 on an error.
+ * Run as the thread of the return cases. Become a deadline task of 50ms
+ * every 100ms and run 20ms. 30ms of runtime are left, and the 0-lag time
+ * is about 20ms ahead. Then change to new parameters, as @arg says:
+ * - 0: the thread switches itself to SCHED_OTHER and back to
+ * SCHED_DEADLINE, while it runs.
+ * - 1: the thread sleeps. The main thread switches it to SCHED_OTHER and
+ * back.
+ * - 2: the thread changes its own parameters while it runs. It stays in
+ * SCHED_DEADLINE.
+ * - 3: as 2, while a donor boosts the thread. The thread takes pi_lock and
+ * wakes the donor up. The donor blocks on pi_lock. The thread then
+ * changes its parameters, and unlocks.
+ * - 4: as 2, to a constrained deadline: 8ms in 20ms, every 100ms.
+ * - 5: the thread sleeps. The main thread changes it to 8ms in 20ms,
+ * every 100ms.
+ * The new parameters are 10ms every 100ms, except in modes 4 and 5.
+ * Then spin for 100ms, and store the CPU time used, in ms, in
+ * ret_first_period_ms. Return NULL, or 1 on an error.
*/
static void *returner(void *arg)
{
+ long mode = (long)arg;
uint64_t t0, cpu0;
- int err;
+ int err = 0;
ret_tid = gettid_();
err = set_deadline(0, 50 * NSEC_PER_MSEC, PERIOD);
@@ -2240,8 +2267,34 @@ static void *returner(void *arg)
return (void *)1;
}
spin_ns(20 * NSEC_PER_MSEC);
- sem_post(&ret_ready);
- sem_wait_sig(&ret_go);
+ if (mode == 1 || mode == 5) {
+ sem_post(&ret_ready);
+ sem_wait_sig(&ret_go);
+ } else if (mode == 4) {
+ err = set_constrained(0, 8 * NSEC_PER_MSEC, 20 * NSEC_PER_MSEC, PERIOD);
+ } else if (mode == 3) {
+ if (take_pi_lock())
+ return (void *)1;
+ sem_post(&owner_go);
+ t0 = now_ns();
+ while (!refill_donor_locking || proc_state_of(donor_tid) != 'S') {
+ if (now_ns() - t0 > NSEC_PER_SEC) {
+ ksft_print_msg("the donor did not block on the lock\n");
+ drop_pi_lock();
+ return (void *)1;
+ }
+ }
+ err = set_deadline(0, 10 * NSEC_PER_MSEC, PERIOD);
+ drop_pi_lock();
+ } else {
+ err = mode ? 0 : set_other(0);
+ if (!err)
+ err = set_deadline(0, 10 * NSEC_PER_MSEC, PERIOD);
+ }
+ if (err) {
+ ksft_print_msg("the task could not change: %s\n", strerror(-err));
+ return (void *)1;
+ }
t0 = now_ns();
cpu0 = thread_time_ns();
while (now_ns() - t0 < PERIOD)
@@ -2251,46 +2304,80 @@ static void *returner(void *arg)
}
/*
- * A task leaves SCHED_DEADLINE while it sleeps, and comes back at once
- * with a smaller runtime. Its 0-lag time is ahead, so it keeps its
- * deadline and the runtime it has left. returner() runs the task. It must
- * get about 10ms of CPU time in the 100ms after the change: its new
- * runtime, not the 30ms left from the old one. The case fails above 20ms.
+ * Run a return case: returner() with @mode as its argument. The thread
+ * must get about 10ms of CPU time in the 100ms after the change, 8ms in
+ * modes 4 and 5. That is its new runtime, not the 30ms left from the old
+ * one. The case fails above 20ms.
*/
-static int return_asleep_new_params(void)
+static int return_case(long mode)
{
- void *t_ret;
+ void *t_ret, *d_ret = NULL;
int err, ret = KSFT_PASS;
- pthread_t t;
+ pthread_t t, d;
if (sem_init(&ret_ready, 0, 0) || sem_init(&ret_go, 0, 0)) {
ksft_print_msg("sem_init: %s\n", strerror(errno));
return KSFT_FAIL;
}
ret_first_period_ms = -1;
- if (pthread_create(&t, NULL, returner, NULL)) {
+ if (mode == 3) {
+ /* The donor: a deadline task of 2ms every 10ms */
+ if (pi_init() || sem_init(&refill_donor_ready, 0, 0))
+ return KSFT_SKIP;
+ refill_donor_locking = 0;
+ donor_tid = 0;
+ if (pthread_create(&d, NULL, refill_donor, NULL)) {
+ ksft_print_msg("the donor thread could not start\n");
+ return KSFT_FAIL;
+ }
+ if (sem_wait_for(&refill_donor_ready, 5000) || donor_tid <= 0) {
+ ksft_print_msg("the donor is not ready\n");
+ sem_post(&owner_go);
+ join(d);
+ return KSFT_FAIL;
+ }
+ }
+ if (pthread_create(&t, NULL, returner, (void *)mode)) {
ksft_print_msg("the task thread could not start\n");
+ if (mode == 3) {
+ sem_post(&owner_go);
+ join(d);
+ }
return KSFT_FAIL;
}
- if (sem_wait_for(&ret_ready, 5000)) {
- /* The thread may get there late. Let it end. */
+ if (mode == 1 || mode == 5) {
+ if (sem_wait_for(&ret_ready, 5000)) {
+ /* The thread may get there late. Let it end. */
+ sem_post(&ret_go);
+ join(t);
+ return KSFT_FAIL; /* the task said why */
+ }
+ /* Before its 0-lag time, about 20ms ahead */
+ err = wait_for(10, proc_state_of(ret_tid) == 'S') ? -EAGAIN : 0;
+ if (!err && mode == 1)
+ err = set_other(ret_tid);
+ if (!err && mode == 1)
+ err = set_deadline(ret_tid, 10 * NSEC_PER_MSEC, PERIOD);
+ if (!err && mode == 5)
+ err = set_constrained(ret_tid, 8 * NSEC_PER_MSEC, 20 * NSEC_PER_MSEC,
+ PERIOD);
sem_post(&ret_go);
- join(t);
- return KSFT_FAIL; /* the task said why */
- }
- /* Before its 0-lag time, about 20ms ahead */
- err = wait_for(10, proc_state_of(ret_tid) == 'S') ? -EAGAIN : 0;
- if (!err)
- err = set_other(ret_tid);
- if (!err)
- err = set_deadline(ret_tid, 10 * NSEC_PER_MSEC, PERIOD);
- sem_post(&ret_go);
- if (err) {
- ksft_print_msg("the task could not change asleep: %s\n", strerror(-err));
- join(t);
- return KSFT_FAIL;
+ if (err) {
+ ksft_print_msg("the task could not change asleep: %s\n",
+ strerror(-err));
+ join(t);
+ return KSFT_FAIL;
+ }
}
err = pthread_join(t, &t_ret);
+ if (mode == 3) {
+ if (t_ret)
+ sem_post(&owner_go); /* the task never woke the donor up */
+ if (pthread_join(d, &d_ret) || d_ret)
+ ret = KSFT_FAIL;
+ pi_fini();
+ sem_destroy(&refill_donor_ready);
+ }
if (err) {
ksft_print_msg("pthread_join: %s\n", strerror(err));
ret = KSFT_FAIL;
@@ -2306,6 +2393,292 @@ static int return_asleep_new_params(void)
return ret;
}
+/*
+ * A running task leaves SCHED_DEADLINE and comes back at once with a
+ * smaller runtime. Its 0-lag time is ahead, so it keeps its deadline and
+ * the runtime it has left. It must not run on more than the new runtime.
+ */
+static int return_running_new_params(void)
+{
+ return return_case(0);
+}
+
+/* The same, for a task that leaves and comes back while it sleeps. */
+static int return_asleep_new_params(void)
+{
+ return return_case(1);
+}
+
+/* The same, for a task that changes its parameters within the class. */
+static int change_running_new_params(void)
+{
+ return return_case(2);
+}
+
+/*
+ * The same, for a task that changes its parameters while a donor boosts
+ * it. The donor is a deadline task of 2ms every 10ms. Its deadline is the
+ * earlier one, so the task runs on its own runtime while boosted. The
+ * unlock ends the boost. The runtime left must then be checked against
+ * the new parameters.
+ */
+static int change_boosted_new_params(void)
+{
+ return return_case(3);
+}
+
+/*
+ * A running task of 50ms every 100ms has 30ms left. It changes itself to
+ * 8ms in 20ms, every 100ms. Its deadline stays 80ms ahead, further than
+ * the new 20ms. The check of the runtime left uses the new density over
+ * those 80ms. That allows 32ms, so the check keeps the 30ms. The task
+ * must keep no more than its new 8ms.
+ */
+static int change_constrained_new_params(void)
+{
+ return return_case(4);
+}
+
+/*
+ * The same, for a task that the main thread changes while it sleeps. The
+ * check is at its wakeup.
+ */
+static int wake_constrained_new_params(void)
+{
+ return return_case(5);
+}
+
+static sem_t rb_donor_go;
+static atomic_int rb_abort, rb_donor_locking, rb_donor_done, rb_waiter_locking;
+static _Atomic pid_t rb_waiter_tid;
+static _Atomic long long rb_spin_ns;
+
+/*
+ * Run as the owner O of deboost_new_params_reboost(). Its comment has the
+ * steps. Store in rb_spin_ns the time that O's last spin of 20ms takes.
+ * Return NULL, or 1 on an error.
+ */
+static void *reboost_owner(void *arg)
+{
+ uint64_t t0;
+ int err;
+
+ owner_tid = gettid_();
+ if (take_pi_lock())
+ return (void *)1;
+ owner_holds = 2;
+ sem_wait_sig(&owner_go);
+ if (rb_abort) {
+ drop_pi_lock();
+ return NULL;
+ }
+ err = set_deadline(0, 50 * NSEC_PER_MSEC, PERIOD);
+ if (err) {
+ ksft_print_msg("the owner refused as a deadline task: %s\n",
+ strerror(-err));
+ goto out;
+ }
+ sem_post(&rb_donor_go);
+ t0 = now_ns();
+ while (!rb_donor_locking || proc_state_of(donor_tid) != 'S') {
+ if (now_ns() - t0 > NSEC_PER_SEC) {
+ ksft_print_msg("the donor did not block on the lock\n");
+ err = -1;
+ goto out;
+ }
+ }
+ err = set_deadline(rb_waiter_tid, 2 * NSEC_PER_MSEC, PERIOD);
+ if (err) {
+ ksft_print_msg("the waiter refused as a deadline task: %s\n",
+ strerror(-err));
+ goto out;
+ }
+ err = set_deadline(0, 5 * NSEC_PER_MSEC, PERIOD);
+ if (err) {
+ ksft_print_msg("the owner could not change: %s\n",
+ strerror(-err));
+ goto out;
+ }
+ t0 = now_ns();
+ while (!rb_donor_done) {
+ if (now_ns() - t0 > NSEC_PER_SEC) {
+ ksft_print_msg("the donor did not time out\n");
+ err = -1;
+ goto out;
+ }
+ }
+ t0 = now_ns();
+ spin_ns(20 * NSEC_PER_MSEC);
+ rb_spin_ns = now_ns() - t0;
+out:
+ drop_pi_lock();
+ if (set_other(0))
+ ksft_print_msg("the owner cannot leave SCHED_DEADLINE\n");
+ return err ? (void *)1 : NULL;
+}
+
+/*
+ * Run as the donor D of deboost_new_params_reboost(). Become a deadline
+ * task of 5ms every 50ms, and sleep until rb_donor_go. Then wait 20ms for
+ * pi_lock. Return NULL if the wait timed out or the case gave up, or 1
+ * otherwise.
+ */
+static void *reboost_donor(void *arg)
+{
+ struct timespec ts;
+ int err;
+
+ err = set_deadline(0, 5 * NSEC_PER_MSEC, PERIOD / 2);
+ if (err) {
+ ksft_print_msg("the donor refused as a deadline task: %s\n",
+ strerror(-err));
+ donor_tid = -1;
+ return (void *)1;
+ }
+ donor_tid = gettid_();
+ sem_wait_sig(&rb_donor_go);
+ if (rb_abort) {
+ set_other(0);
+ return NULL;
+ }
+ clock_gettime(CLOCK_REALTIME, &ts);
+ ts.tv_nsec += 20 * 1000000L;
+ ts.tv_sec += ts.tv_nsec / 1000000000L;
+ ts.tv_nsec %= 1000000000L;
+ rb_donor_locking = 1;
+ err = pthread_mutex_timedlock(&pi_lock, &ts);
+ rb_donor_done = 1;
+ if (!err)
+ pthread_mutex_unlock(&pi_lock);
+ if (set_other(0))
+ ksft_print_msg("the donor cannot leave SCHED_DEADLINE\n");
+ if (err != ETIMEDOUT) {
+ ksft_print_msg("the donor: %s, not a timeout\n",
+ err ? strerror(err) : "got pi_lock");
+ return (void *)1;
+ }
+ return NULL;
+}
+
+/*
+ * Run as the waiter W of deboost_new_params_reboost(). Wait for pi_lock
+ * as a fair task. The owner makes W a deadline task during the wait.
+ * Return NULL, or 1 on an error.
+ */
+static void *reboost_waiter(void *arg)
+{
+ rb_waiter_tid = gettid_();
+ rb_waiter_locking = 1;
+ if (take_pi_lock())
+ return (void *)1;
+ drop_pi_lock();
+ if (set_other(0))
+ ksft_print_msg("the waiter cannot leave SCHED_DEADLINE\n");
+ return NULL;
+}
+
+/*
+ * Check that a deboost boosts the owner again when the check of its new
+ * parameters moves its deadline after the one of its top waiter. The
+ * threads share the CPU of P:
+ *
+ * owner O donor D waiter W
+ * fair 5ms every 50ms fair
+ *
+ * takes pi_lock sleeps
+ * blocks on pi_lock
+ * becomes 50ms every 100ms
+ * wakes D up
+ * preempts O
+ * waits 20ms for
+ * pi_lock, boosts O
+ * makes W 2ms every 100ms
+ * changes itself to 5ms
+ * every 100ms
+ * spins
+ * times out: W is now
+ * the top waiter
+ * spins 20ms
+ * unlocks
+ * takes pi_lock
+ *
+ * W gets its deadline after O does, so W's deadline is the later one.
+ * W blocks as a fair task: a deadline W could not preempt O to block.
+ *
+ * O has run about 20ms of its 50ms when D times out. The deboost checks
+ * the runtime left against the new parameters. It is too much, so O
+ * starts a new period: 5ms and a deadline 100ms ahead. W's deadline is
+ * now the earlier one. W must boost O again. A boosted O is refilled
+ * from W, so its last spin takes about 20ms. Without the boost, O is
+ * throttled once its 5ms run out, until its new deadline. The case fails
+ * if the spin takes more than 50ms.
+ */
+static int deboost_new_params_reboost(void)
+{
+ void *o_ret = NULL, *d_ret = NULL, *w_ret = NULL;
+ int ret = KSFT_FAIL, have_d = 0, have_w = 0;
+ pthread_t o, d, w;
+
+ if (pi_init())
+ return KSFT_SKIP;
+ if (sem_init(&rb_donor_go, 0, 0)) {
+ ksft_print_msg("sem_init: %s\n", strerror(errno));
+ pi_fini();
+ return KSFT_FAIL;
+ }
+ donor_tid = 0;
+ rb_abort = 0;
+ rb_donor_locking = 0;
+ rb_donor_done = 0;
+ rb_waiter_locking = 0;
+ rb_waiter_tid = 0;
+ rb_spin_ns = -1;
+ if (pthread_create(&o, NULL, reboost_owner, NULL)) {
+ ksft_print_msg("the owner thread could not start\n");
+ goto out_fini;
+ }
+ if (wait_for(5000, owner_holds) || owner_holds != 2) {
+ ksft_print_msg("the owner never took pi_lock\n");
+ goto out;
+ }
+ have_d = !pthread_create(&d, NULL, reboost_donor, NULL);
+ if (!have_d || wait_for(1000, donor_tid &&
+ proc_state_of(donor_tid) == 'S') ||
+ donor_tid < 0) {
+ ksft_print_msg("the donor never slept\n");
+ goto out;
+ }
+ have_w = !pthread_create(&w, NULL, reboost_waiter, NULL);
+ if (!have_w || wait_for(1000, rb_waiter_locking &&
+ proc_state_of(rb_waiter_tid) == 'S')) {
+ ksft_print_msg("the waiter never waited on pi_lock\n");
+ goto out;
+ }
+ ret = KSFT_PASS;
+out:
+ /* On a failure, the owner and the donor end without the steps. */
+ rb_abort = ret != KSFT_PASS;
+ sem_post(&owner_go);
+ if (pthread_join(o, &o_ret))
+ ret = KSFT_FAIL;
+ sem_post(&rb_donor_go);
+ if (have_d && pthread_join(d, &d_ret))
+ ret = KSFT_FAIL;
+ if (have_w && pthread_join(w, &w_ret))
+ ret = KSFT_FAIL;
+ if (o_ret || d_ret || w_ret) {
+ ret = KSFT_FAIL; /* the thread said why */
+ } else if (ret == KSFT_PASS && rb_spin_ns > PERIOD / 2) {
+ ksft_print_msg("the spin took %lld ms: not boosted again\n",
+ rb_spin_ns / (long long)NSEC_PER_MSEC);
+ ret = KSFT_FAIL;
+ }
+out_fini:
+ sem_destroy(&rb_donor_go);
+ pi_fini();
+ return ret;
+}
+
static pthread_mutex_t pi_lock2;
static sem_t holder_go;
static atomic_int holder_holds, chain_waits, donor_done, x_waits, l2_takes;
@@ -10369,7 +10742,13 @@ static const struct p_case {
{ "deboost_after_donor_refill", deboost_after_donor_refill, MEASURED_IN_P },
{ "deboost_out_after_donor_refill", deboost_out_after_donor_refill, MEASURED_IN_P },
{ "deboost_keeps_own_runtime", deboost_keeps_own_runtime, MEASURED_IN_P },
+ { "return_running_new_params", return_running_new_params, MEASURED_IN_P },
{ "return_asleep_new_params", return_asleep_new_params, MEASURED_IN_P },
+ { "change_running_new_params", change_running_new_params, MEASURED_IN_P },
+ { "change_boosted_new_params", change_boosted_new_params, MEASURED_IN_P },
+ { "change_constrained_new_params", change_constrained_new_params, MEASURED_IN_P },
+ { "wake_constrained_new_params", wake_constrained_new_params, MEASURED_IN_P },
+ { "deboost_new_params_reboost", deboost_new_params_reboost, MEASURED_IN_P },
{ "deboost_blocked_keeps_order", deboost_blocked_keeps_order, MEASURED_IN_P },
{ "setattr_blocked_keeps_order", setattr_blocked_keeps_order, MEASURED_IN_P },
{ "boost_blocked_keeps_order", boost_blocked_keeps_order, MEASURED_IN_P },
--
2.47.3
^ permalink raw reply related [flat|nested] 35+ messages in thread
* Re: [PATCH v8 16/33] sched/deadline: Keep the last CPU a task has in its root domain
2026-11-19 20:15 ` [PATCH v8 16/33] sched/deadline: Keep the last CPU a task has in its root domain Mathieu Desnoyers
@ 2026-11-19 20:22 ` Mathieu Desnoyers
0 siblings, 0 replies; 35+ messages in thread
From: Mathieu Desnoyers @ 2026-11-19 20:22 UTC (permalink / raw)
To: Waiman Long
Cc: Mathieu Desnoyers, juri.lelli, peterz, mingo, vincent.guittot,
tglx, will, boqun, dietmar.eggemann, rostedt, bsegall, mgorman,
vschneid, kprateek.nayak, luca.abeni, mathieu.poirier,
ridong.chen, tj, hannes, mkoutny, shuah, bigeasy, clrkwllms,
parri.andrea, m.szyprowski, kayracizmeci, weihengqiuu,
linux-kernel, cgroups, linux-kselftest, linux-rt-devel
Hi Waiman,
This patch refuses a CPU offline when a deadline task would leave the
root domain that counts its bandwidth. It also refuses an offline that
makes a partition invalid, when a deadline task in it would change root
domain. To decide, it predicts the new effective CPUs of each cpuset
after the offline.
We checked the prediction against the cpuset code, cgroup-v2.rst and
test_cpuset_prs.sh. This is what we rely on. Please tell us if a point
is wrong:
1. A cpuset that is not a valid partition root keeps the CPUs of its
parent that its cpuset.cpus holds. If there are none, it takes all
of the parent's CPUs. cpuset.cpus.exclusive plays no part.
2. A valid partition root that keeps a CPU of its own only loses the
offline CPU. A child partition is made invalid only when its
populated parent has no other CPU left. The parent then takes the
CPUs of that child.
3. A populated partition that loses its last CPU is made invalid. It
then follows rule 1 against its parent. This holds for a local and
for a remote partition.
4. When the last CPU left to the top cpuset goes offline, every
partition is made invalid.
5. Any hotplug event tries each invalid partition again, an offline
too. So an offline can make a partition valid, when what blocked it
went away earlier.
6. housekeeping_update() can run during an offline, as the changelog of
a84097e625f2 says. We see no lock we could hold in the offline path.
So we read HK_TYPE_DOMAIN under RCU and accept an old value.
We have one question.
7. At an offline, select_fallback_rq() runs before the cpuset update.
cpuset_cpus_allowed_fallback() gives the task its cpuset.cpus, then
the housekeeping CPUs follow. A deadline task could land in another
root domain. The previous patch makes select_fallback_rq() give it
the CPUs of its root domain instead. The task can then run outside
its cpuset until the cpuset update sets its mask. The comment in
cpuset_cpus_allowed_fallback() already allows a temporary mask of
this kind. Is our case acceptable too? On cgroup v1, a task of the
top cpuset keeps that mask, as it keeps the housekeeping mask today.
One note on cgroup v1. When a cpuset is left with no CPU, a work item
moves its tasks to an ancestor. We refuse the offline when that move
would take a deadline task to another root domain. The attach has its
own admission test. But if it fails, cgroup_transfer_tasks() stops, and
the other tasks stay in the empty cpuset. Tell us if you prefer to let
the attach decide.
Thanks,
Mathieu
^ permalink raw reply [flat|nested] 35+ messages in thread
end of thread, other threads:[~2026-11-19 20:23 UTC | newest]
Thread overview: 35+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2026-11-19 20:15 [PATCH v8 00/33] sched/deadline: bandwidth accounting fixes and selftests Mathieu Desnoyers
2026-11-19 20:15 ` [PATCH v8 01/33] sched/deadline: Keep departing bandwidth across a root domain rebuild Mathieu Desnoyers
2026-11-19 20:15 ` [PATCH v8 02/33] sched/deadline: Rebuild the root domains' accounting from the task list Mathieu Desnoyers
2026-11-19 20:15 ` [PATCH v8 03/33] sched/deadline: Record the root domain holding a task's bandwidth Mathieu Desnoyers
2026-11-19 20:15 ` [PATCH v8 04/33] sched/deadline: Leave a dying task out of a rebuild of the accounting Mathieu Desnoyers
2026-11-19 20:15 ` [PATCH v8 05/33] selftests/sched: Test SCHED_DEADLINE bandwidth across a root domain rebuild Mathieu Desnoyers
2026-11-19 20:15 ` [PATCH v8 06/33] selftests/sched: Test the root domain rebuild in deadline_bw_test Mathieu Desnoyers
2026-11-19 20:15 ` [PATCH v8 07/33] sched/deadline: Move a task's bandwidth only where room was made Mathieu Desnoyers
2026-11-19 20:15 ` [PATCH v8 08/33] cgroup/cpuset: Wait for an attach before rebuilding the sched domains Mathieu Desnoyers
2026-11-19 20:15 ` [PATCH v8 09/33] sched: Make deadline policy and affinity changes wait for a cpuset attach Mathieu Desnoyers
2026-11-19 20:15 ` [PATCH v8 10/33] cgroup/cpuset: Give back the bandwidth an attach did not move Mathieu Desnoyers
2026-11-19 20:15 ` [PATCH v8 11/33] cgroup/cpuset: Reserve the bandwidth in the root domain a task moves to Mathieu Desnoyers
2026-11-19 20:15 ` [PATCH v8 12/33] selftests/sched: Test cpuset moves in deadline_bw_test Mathieu Desnoyers
2026-11-19 20:15 ` [PATCH v8 13/33] sched/deadline: Make a rebuild count a task in the root domain it runs in Mathieu Desnoyers
2026-11-19 20:15 ` [PATCH v8 14/33] sched/deadline: Move a task's bandwidth to " Mathieu Desnoyers
2026-11-19 20:15 ` [PATCH v8 15/33] sched/deadline: Keep a task in the root domain that counts its bandwidth Mathieu Desnoyers
2026-11-19 20:15 ` [PATCH v8 16/33] sched/deadline: Keep the last CPU a task has in its root domain Mathieu Desnoyers
2026-11-19 20:22 ` Mathieu Desnoyers
2026-11-19 20:15 ` [PATCH v8 17/33] cgroup/cpuset: Refuse to take a deadline task out of " Mathieu Desnoyers
2026-11-19 20:15 ` [PATCH v8 18/33] selftests/sched: Test the root domain of a task in deadline_bw_test Mathieu Desnoyers
2026-11-19 20:15 ` [PATCH v8 19/33] sched/deadline: Count only active CPUs when spreading extra_bw Mathieu Desnoyers
2026-11-19 20:15 ` [PATCH v8 20/33] rtmutex: Continue the chain walk when a waiter's deadline donor changes Mathieu Desnoyers
2026-11-19 20:16 ` [PATCH v8 21/33] rtmutex: Continue the walk when the owner's top waiter changes lock Mathieu Desnoyers
2026-11-19 20:16 ` [PATCH v8 22/33] selftests/sched: Test the rtmutex chain walk in deadline_bw_test Mathieu Desnoyers
2026-11-19 20:16 ` [PATCH v8 23/33] sched/deadline: Reset the donor's runtime at a deboost in the class Mathieu Desnoyers
2026-11-19 20:16 ` [PATCH v8 24/33] sched/deadline: Replenish a throttled task boosted while asleep Mathieu Desnoyers
2026-11-19 20:16 ` [PATCH v8 25/33] sched/deadline: Count the bandwidth a boosted task is given Mathieu Desnoyers
2026-11-19 20:16 ` [PATCH v8 26/33] sched/deadline: Start a period for a task that enters the class blocked Mathieu Desnoyers
2026-11-19 20:16 ` [PATCH v8 27/33] sched/deadline: Start a period at a deboost out of the class Mathieu Desnoyers
2026-11-19 20:16 ` [PATCH v8 28/33] selftests/sched: Test deadline boosts in deadline_bw_test Mathieu Desnoyers
2026-11-19 20:16 ` [PATCH v8 29/33] sched/deadline: Cancel the replenishment timer with the parameters Mathieu Desnoyers
2026-11-19 20:16 ` [PATCH v8 30/33] sched/deadline: Cap the runtime left of a constrained task at dl_runtime Mathieu Desnoyers
2026-11-19 20:16 ` [PATCH v8 31/33] sched/deadline: Check the runtime of a task that comes back runnable Mathieu Desnoyers
2026-11-19 20:16 ` [PATCH v8 32/33] sched/deadline: Check the runtime of a task whose parameters change Mathieu Desnoyers
2026-11-19 20:16 ` [PATCH v8 33/33] selftests/sched: Test parameter changes in deadline_bw_test Mathieu Desnoyers
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox