* [PATCH v2 1/4] sched/fair: expose cpu.max.runtime to set bandwidth runtime directly
2026-11-10 11:08 [PATCH v2 0/4] sched/fair: cpu.max.runtime and idle refill for CPU credits Fernand Sieber
@ 2026-11-10 11:08 ` Fernand Sieber
2026-11-11 8:28 ` bot+bpf-ci
2026-11-10 11:08 ` [PATCH v2 2/4] sched/fair: refill bandwidth for the periods the timer slept through Fernand Sieber
` (4 subsequent siblings)
5 siblings, 1 reply; 14+ messages in thread
From: Fernand Sieber @ 2026-11-10 11:08 UTC (permalink / raw)
To: mingo, peterz, juri.lelli, vincent.guittot, tj
Cc: bsegall, dietmar.eggemann, rostedt, mgorman, vschneid,
kprateek.nayak, hannes, mkoutny, void, arighi, changwoo, skhan,
corbet, rdunlap, cgroups, sched-ext, linux-doc, linux-kselftest,
bpf, linux-kernel, nh-open-source, hborghor, fmubeen, dwmw,
karataev, sieberf
Add cpu.max.runtime, a cgroup v2 file that reads and sets the CFS
bandwidth runtime, cfs_b->runtime, in microseconds. Userspace can then
grant a cgroup runtime beyond its quota. Refills still add only the
quota, capped at quota + burst, so the excess is used up over time and
the group returns to its quota per period.
A write is rejected without a quota or above quota + burst, the most
the group can accumulate. A read is capped at quota + burst so it can
be written back; uncapped slack from __return_cfs_rq_runtime() can
exceed that until the next refill. Runtime the CPUs already took, at
most a slice each, is not included and a write leaves it in place.
A write distributes the new runtime to throttled cfs_rqs at once, as
the slack timer does, rather than waiting for the unsynchronized
period timer. It also shifts runtime_snap by the change, so cpu.stat
burst accounting neither counts the write nor hides what the group ran
since the last refill.
Also drop the burst <= quota limit and require only quota + burst <=
max_bw_runtime_us (about 203 days), the quota's own limit, so a group
can accumulate many periods of quota. This applies to cgroup v1
cpu.cfs_burst_us too, which shares tg_set_bandwidth().
The file exists with CONFIG_CFS_BANDWIDTH only.
Signed-off-by: Fernand Sieber <sieberf@amazon.com>
---
Documentation/admin-guide/cgroup-v2.rst | 22 +-
Documentation/scheduler/sched-bwc.rst | 13 +-
kernel/sched/core.c | 34 ++-
kernel/sched/fair.c | 61 ++++
kernel/sched/sched.h | 2 +
tools/testing/selftests/cgroup/test_cpu.c | 336 ++++++++++++++++++++++
6 files changed, 459 insertions(+), 9 deletions(-)
diff --git a/Documentation/admin-guide/cgroup-v2.rst b/Documentation/admin-guide/cgroup-v2.rst
index 9e3c67b02ae2e..c5a02057f0242 100644
--- a/Documentation/admin-guide/cgroup-v2.rst
+++ b/Documentation/admin-guide/cgroup-v2.rst
@@ -1231,12 +1231,32 @@ will be referred to. All time durations are in microseconds.
A read-write single value file which exists on non-root
cgroups. The default is "0".
- The burst in the range [0, $MAX].
+ The burst. The cgroup can accumulate the runtime it does not
+ use, up to $MAX + cpu.max.burst. The burst may exceed $MAX, as
+ long as $MAX + cpu.max.burst is at most 2^44 - 1 (about 203
+ days).
This file affects only processes under the fair-class scheduler and a BPF
scheduler with the ``cgroup_set_bandwidth`` callback depending on what
the callback actually does.
+ cpu.max.runtime
+ A read-write single value file which exists on non-root
+ cgroups if the kernel is built with CONFIG_CFS_BANDWIDTH.
+
+ The runtime the cgroup has accumulated and not used yet, in
+ microseconds. Each period refills it by $MAX, up to $MAX +
+ cpu.max.burst, and a read returns at most $MAX + cpu.max.burst.
+ A write sets it, up to $MAX + cpu.max.burst, and lets a cgroup
+ that ran out of runtime run again at once. A write fails with
+ EINVAL while $MAX is "max".
+
+ The runtime does not include what the cgroup's CPUs already took
+ from it, at most a slice each (sched_cfs_bandwidth_slice_us).
+ A write leaves that with the CPUs.
+
+ This file affects only processes under the fair-class scheduler.
+
cpu.pressure
A read-write nested-keyed file.
diff --git a/Documentation/scheduler/sched-bwc.rst b/Documentation/scheduler/sched-bwc.rst
index e881a945c1888..c22b3c8c14e29 100644
--- a/Documentation/scheduler/sched-bwc.rst
+++ b/Documentation/scheduler/sched-bwc.rst
@@ -90,8 +90,8 @@ bandwidth restriction in place, such a group is described as an unconstrained
bandwidth group. This represents the traditional work-conserving behavior for
CFS.
-Writing any (valid) positive value(s) no smaller than cpu.cfs_burst_us will
-enact the specified bandwidth limit. The minimum quota allowed for the quota or
+Writing any (valid) positive value(s) will enact the specified bandwidth
+limit. The minimum quota allowed for the quota or
period is 1ms. There is also an upper bound on the period length of 1s.
Additional restrictions exist when bandwidth limits are used in a hierarchical
fashion, these are explained in more detail below.
@@ -101,9 +101,10 @@ and return the group to an unconstrained state once more.
A value of 0 for cpu.cfs_burst_us indicates that the group can not accumulate
any unused bandwidth. It makes the traditional bandwidth control behavior for
-CFS unchanged. Writing any (valid) positive value(s) no larger than
-cpu.cfs_quota_us into cpu.cfs_burst_us will enact the cap on unused bandwidth
-accumulation.
+CFS unchanged. Writing any (valid) positive value(s) into cpu.cfs_burst_us
+will enact the cap on unused bandwidth accumulation: the group can accumulate
+up to cpu.cfs_quota_us + cpu.cfs_burst_us. The burst may be larger than the
+quota, as long as their sum is at most 2^44 - 1 microseconds (about 203 days).
Any updates to a group's bandwidth specification will result in it becoming
unthrottled if it is in a constrained state.
@@ -243,4 +244,4 @@ Examples
# echo 50000 > cpu.cfs_period_us /* period = 50ms */
# echo 10000 > cpu.cfs_burst_us /* burst = 10ms */
- Larger buffer setting (no larger than quota) allows greater burst capacity.
+ Larger buffer setting allows greater burst capacity.
diff --git a/kernel/sched/core.c b/kernel/sched/core.c
index 175b52a7ea793..76fcd70ff207a 100644
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -10131,8 +10131,7 @@ static int tg_set_bandwidth(struct task_group *tg,
if (quota_us != RUNTIME_INF && quota_us > max_bw_runtime_us)
return -EINVAL;
- if (quota_us != RUNTIME_INF && (burst_us > quota_us ||
- burst_us + quota_us > max_bw_runtime_us))
+ if (quota_us != RUNTIME_INF && burst_us > max_bw_runtime_us - quota_us)
return -EINVAL;
#ifdef CONFIG_CFS_BANDWIDTH
@@ -10193,6 +10192,29 @@ static int cpu_burst_write_u64(struct cgroup_subsys_state *css,
tg_bandwidth(tg, &period_us, "a_us, NULL);
return tg_set_bandwidth(tg, period_us, quota_us, burst_us);
}
+
+#ifdef CONFIG_CFS_BANDWIDTH
+static int cpu_runtime_write_u64(struct cgroup_subsys_state *css,
+ struct cftype *cftype, u64 runtime_us)
+{
+ struct task_group *tg = css_tg(css);
+
+ if (runtime_us > max_bw_runtime_us)
+ return -EINVAL;
+
+ return cfs_bandwidth_set_runtime(&tg->cfs_bandwidth,
+ runtime_us * NSEC_PER_USEC);
+}
+
+static u64 cpu_runtime_read_u64(struct cgroup_subsys_state *css,
+ struct cftype *cftype)
+{
+ struct task_group *tg = css_tg(css);
+
+ return div_u64(cfs_bandwidth_get_runtime(&tg->cfs_bandwidth),
+ NSEC_PER_USEC);
+}
+#endif /* CONFIG_CFS_BANDWIDTH */
#endif /* CONFIG_GROUP_SCHED_BANDWIDTH */
#ifdef CONFIG_RT_GROUP_SCHED
@@ -10544,7 +10566,15 @@ static struct cftype cpu_files[] = {
.read_u64 = cpu_burst_read_u64,
.write_u64 = cpu_burst_write_u64,
},
+#ifdef CONFIG_CFS_BANDWIDTH
+ {
+ .name = "max.runtime",
+ .flags = CFTYPE_NOT_ON_ROOT,
+ .read_u64 = cpu_runtime_read_u64,
+ .write_u64 = cpu_runtime_write_u64,
+ },
#endif /* CONFIG_CFS_BANDWIDTH */
+#endif /* CONFIG_GROUP_SCHED_BANDWIDTH */
#ifdef CONFIG_UCLAMP_TASK_GROUP
{
.name = "uclamp.min",
diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index d4cc0caee6265..58e948ed1792f 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -7421,6 +7421,8 @@ static bool distribute_cfs_runtime(struct cfs_bandwidth *cfs_b)
struct cfs_rq *cfs_rq;
struct rq *rq;
+ lockdep_assert_irqs_disabled();
+
guard(rcu)();
list_for_each_entry_rcu(cfs_rq, &cfs_b->throttled_cfs_rq,
@@ -7811,6 +7813,65 @@ static void init_cfs_rq_runtime(struct cfs_rq *cfs_rq)
INIT_LIST_HEAD(&cfs_rq->throttled_limbo_list);
}
+/*
+ * The runtime of @cfs_b, as cpu.max.runtime reads it.
+ *
+ * __return_cfs_rq_runtime() returns slack without capping it, so the runtime
+ * can exceed quota + burst until the next refill caps it. Report it capped,
+ * so that a value read can always be written back.
+ */
+u64 cfs_bandwidth_get_runtime(struct cfs_bandwidth *cfs_b)
+{
+ u64 runtime;
+
+ guard(raw_spinlock_irq)(&cfs_b->lock);
+
+ runtime = cfs_b->runtime;
+ if (cfs_b->quota != RUNTIME_INF)
+ runtime = min(runtime, cfs_b->quota + cfs_b->burst);
+
+ return runtime;
+}
+
+/*
+ * Set the runtime of @cfs_b, as a cpu.max.runtime write does, and hand it to
+ * the throttled cfs_rqs at once rather than at the next period.
+ */
+int cfs_bandwidth_set_runtime(struct cfs_bandwidth *cfs_b, u64 runtime)
+{
+ bool throttled;
+ s64 snap;
+
+ guard(irq)();
+
+ scoped_guard (raw_spinlock, &cfs_b->lock) {
+ if (cfs_b->quota == RUNTIME_INF ||
+ runtime > cfs_b->quota + cfs_b->burst)
+ return -EINVAL;
+
+ /*
+ * The next refill counts runtime_snap - runtime as burst usage.
+ * Move runtime_snap by what the write adds or takes away, so
+ * that the write itself is not counted as burst usage but what
+ * the group ran since the last refill still is.
+ */
+ snap = cfs_b->runtime_snap + runtime - cfs_b->runtime;
+ cfs_b->runtime_snap = max(snap, 0);
+ cfs_b->runtime = runtime;
+
+ throttled = !list_empty(&cfs_b->throttled_cfs_rq) && runtime;
+ }
+
+ /*
+ * distribute_cfs_runtime() takes the rq locks, so it runs without
+ * cfs_b->lock and with interrupts disabled, as from the slack timer.
+ */
+ if (throttled)
+ distribute_cfs_runtime(cfs_b);
+
+ return 0;
+}
+
void start_cfs_bandwidth(struct cfs_bandwidth *cfs_b)
{
lockdep_assert_held(&cfs_b->lock);
diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h
index a4df59c9c9b60..1f60c84e0dc38 100644
--- a/kernel/sched/sched.h
+++ b/kernel/sched/sched.h
@@ -578,6 +578,8 @@ extern void init_cfs_bandwidth(struct cfs_bandwidth *cfs_b, struct cfs_bandwidth
extern void __refill_cfs_bandwidth_runtime(struct cfs_bandwidth *cfs_b);
extern void start_cfs_bandwidth(struct cfs_bandwidth *cfs_b);
+u64 cfs_bandwidth_get_runtime(struct cfs_bandwidth *cfs_b);
+int cfs_bandwidth_set_runtime(struct cfs_bandwidth *cfs_b, u64 runtime);
extern void unthrottle_cfs_rq(struct cfs_rq *cfs_rq);
extern bool cfs_task_bw_constrained(struct task_struct *p);
diff --git a/tools/testing/selftests/cgroup/test_cpu.c b/tools/testing/selftests/cgroup/test_cpu.c
index 3456e3e988c68..278d7e16eeb2c 100644
--- a/tools/testing/selftests/cgroup/test_cpu.c
+++ b/tools/testing/selftests/cgroup/test_cpu.c
@@ -809,6 +809,337 @@ static int test_cpucg_max_nested(const char *root)
return ret;
}
+/* Wait until the period timer of @cgroup starts a new period. */
+static int wait_period_start(const char *cgroup)
+{
+ long nr_periods = cg_read_key_long(cgroup, "cpu.stat", "nr_periods");
+ int i;
+
+ for (i = 0; i < 3000; i++) {
+ usleep(1000);
+ if (cg_read_key_long(cgroup, "cpu.stat", "nr_periods") != nr_periods)
+ return 0;
+ }
+ return -1;
+}
+
+static long long monotonic_ns(void)
+{
+ struct timespec ts;
+
+ clock_gettime(CLOCK_MONOTONIC, &ts);
+ return ts.tv_sec * 1000000000LL + ts.tv_nsec;
+}
+
+/* For (long)arg seconds, run for 1ms and then sleep for 10ms. */
+static int run_and_sleep(const char *cgroup, void *arg)
+{
+ long long end = monotonic_ns() + (long)arg * 1000000000LL;
+ long long start;
+
+ while ((start = monotonic_ns()) < end) {
+ while (monotonic_ns() - start < 1000000)
+ ;
+ usleep(10000);
+ }
+ return 0;
+}
+
+static int test_cpucg_max_runtime(const char *root)
+{
+ int ret = KSFT_FAIL;
+ long quota_usec = 1000; /* 1ms (minimum) */
+ long period_usec = 100000; /* 100ms */
+ long burst_usec = 5000000; /* 5s, so cap = 5001ms */
+ long runtime_usec = 2500000; /* 2500ms, half of the 5s run */
+ long duration_sec = 5;
+ /* The injected runtime plus the quota refilled each period of the run */
+ long expected_usec = runtime_usec +
+ duration_sec * USEC_PER_SEC / period_usec * quota_usec;
+ long usage_usec;
+ char *cpucg;
+ char buf[64];
+ int pid;
+
+ cpucg = cg_name(root, "cpucg_runtime_test");
+ if (!cpucg)
+ goto cleanup;
+
+ if (cg_create(cpucg))
+ goto cleanup;
+
+ /* Without a quota there is no runtime to set. */
+ if (!cg_write_numeric(cpucg, "cpu.max.runtime", runtime_usec))
+ goto cleanup;
+
+ snprintf(buf, sizeof(buf), "%ld %ld", quota_usec, period_usec);
+ if (cg_write(cpucg, "cpu.max", buf))
+ goto cleanup;
+ if (cg_write_numeric(cpucg, "cpu.max.burst", burst_usec))
+ goto cleanup;
+
+ /* Start burner, let it settle, then inject credits */
+ struct cpu_hog_func_param param = {
+ .nprocs = 1,
+ .ts = { .tv_sec = duration_sec, .tv_nsec = 0 },
+ .clock_type = CPU_HOG_CLOCK_WALL,
+ };
+ pid = cg_run_nowait(cpucg, hog_cpus_timed, (void *)¶m);
+ if (pid < 0)
+ goto cleanup;
+
+ usleep(100000);
+ if (cg_write_numeric(cpucg, "cpu.max.runtime", runtime_usec)) {
+ kill(pid, SIGKILL);
+ waitpid(pid, NULL, 0);
+ goto cleanup;
+ }
+
+ waitpid(pid, NULL, 0);
+
+ usage_usec = cg_read_key_long(cpucg, "cpu.stat", "usage_usec");
+ if (usage_usec <= 0)
+ goto cleanup;
+
+ /* At least the injected runtime, at most 5% above the expected usage. */
+ if (usage_usec < runtime_usec ||
+ usage_usec > expected_usec + expected_usec / 20)
+ goto cleanup;
+
+ ret = KSFT_PASS;
+
+cleanup:
+ cg_destroy(cpucg);
+ free(cpucg);
+ return ret;
+}
+
+/*
+ * A cpu.max.runtime write is not burst usage: writing a runtime below the
+ * current one leaves cpu.stat's nr_bursts and burst_usec where they were.
+ */
+static int test_cpucg_max_runtime_burst_stats(const char *root)
+{
+ int ret = KSFT_FAIL;
+ long nr_bursts, burst_usec;
+ char *cpucg;
+
+ cpucg = cg_name(root, "cpucg_runtime_burst_stats_test");
+ if (!cpucg)
+ goto cleanup;
+
+ if (cg_create(cpucg))
+ goto cleanup;
+
+ /* A long period, so the write lands before the timer's first refill. */
+ if (cg_write(cpucg, "cpu.max", "10000 1000000"))
+ goto cleanup;
+ if (cg_write_numeric(cpucg, "cpu.max.burst", 190000))
+ goto cleanup;
+ if (cg_write_numeric(cpucg, "cpu.max.runtime", 0))
+ goto cleanup;
+
+ /* The idle group refills for a few periods. */
+ sleep(3);
+ nr_bursts = cg_read_key_long(cpucg, "cpu.stat", "nr_bursts");
+ burst_usec = cg_read_key_long(cpucg, "cpu.stat", "burst_usec");
+ if (nr_bursts != 0 || burst_usec != 0)
+ goto cleanup;
+
+ ret = KSFT_PASS;
+
+cleanup:
+ cg_destroy(cpucg);
+ free(cpucg);
+ return ret;
+}
+
+/*
+ * A cpu.max.runtime write hands the runtime to a throttled group at once,
+ * rather than at the next period: with a 1s period, the group runs again
+ * within 100ms of the write, before the next period starts.
+ */
+static int test_cpucg_max_runtime_unthrottle(const char *root)
+{
+ int ret = KSFT_FAIL;
+ long usage_before, usage_after, nr_periods;
+ char *cpucg;
+ int pid;
+
+ cpucg = cg_name(root, "cpucg_runtime_unthrottle_test");
+ if (!cpucg)
+ goto cleanup;
+
+ if (cg_create(cpucg))
+ goto cleanup;
+
+ /* 1ms per 1s period, with room for a runtime of up to 1s. */
+ if (cg_write(cpucg, "cpu.max", "1000 1000000"))
+ goto cleanup;
+ if (cg_write_numeric(cpucg, "cpu.max.burst", 999000))
+ goto cleanup;
+
+ struct cpu_hog_func_param param = {
+ .nprocs = 1,
+ .ts = { .tv_sec = 2, .tv_nsec = 0 },
+ .clock_type = CPU_HOG_CLOCK_WALL,
+ };
+ pid = cg_run_nowait(cpucg, hog_cpus_timed, (void *)¶m);
+ if (pid < 0)
+ goto cleanup;
+
+ /*
+ * The period timer's phase is random: start right after a boundary,
+ * so that the next one is some 900ms away. The hog uses up its runtime
+ * and is throttled for the period.
+ */
+ if (wait_period_start(cpucg))
+ goto cleanup_hog;
+ nr_periods = cg_read_key_long(cpucg, "cpu.stat", "nr_periods");
+ usleep(50000);
+ usage_before = cg_read_key_long(cpucg, "cpu.stat", "usage_usec");
+ if (usage_before < 0 || usage_before > 20000)
+ goto cleanup_hog;
+
+ if (cg_write_numeric(cpucg, "cpu.max.runtime", 500000))
+ goto cleanup_hog;
+
+ /* No period started meanwhile: only the write lets the hog run. */
+ usleep(100000);
+ usage_after = cg_read_key_long(cpucg, "cpu.stat", "usage_usec");
+ if (cg_read_key_long(cpucg, "cpu.stat", "nr_periods") != nr_periods)
+ goto cleanup_hog;
+ if (usage_after - usage_before < 50000)
+ goto cleanup_hog;
+
+ ret = KSFT_PASS;
+
+cleanup_hog:
+ kill(pid, SIGKILL);
+ waitpid(pid, NULL, 0);
+cleanup:
+ cg_destroy(cpucg);
+ free(cpucg);
+ return ret;
+}
+
+/*
+ * What a group ran beyond its quota before a cpu.max.runtime write is still
+ * burst usage: a write of the runtime the group has left does not hide it.
+ */
+static int test_cpucg_max_runtime_burst_kept(const char *root)
+{
+ int ret = KSFT_FAIL;
+ long burst_before, burst_after, runtime_usec;
+ char *cpucg;
+ int pid;
+
+ cpucg = cg_name(root, "cpucg_runtime_burst_kept_test");
+ if (!cpucg)
+ goto cleanup;
+
+ if (cg_create(cpucg))
+ goto cleanup;
+
+ /* 10ms per 1s period, and a runtime of 1s to run on. */
+ if (cg_write(cpucg, "cpu.max", "10000 1000000"))
+ goto cleanup;
+ if (cg_write_numeric(cpucg, "cpu.max.burst", 990000))
+ goto cleanup;
+ if (cg_write_numeric(cpucg, "cpu.max.runtime", 1000000))
+ goto cleanup;
+
+ struct cpu_hog_func_param param = {
+ .nprocs = 1,
+ .ts = { .tv_sec = 10, .tv_nsec = 0 },
+ .clock_type = CPU_HOG_CLOCK_WALL,
+ };
+ pid = cg_run_nowait(cpucg, hog_cpus_timed, (void *)¶m);
+ if (pid < 0)
+ goto cleanup;
+
+ /* Run some 300ms, all in one period, and stop. */
+ if (wait_period_start(cpucg)) {
+ kill(pid, SIGKILL);
+ waitpid(pid, NULL, 0);
+ goto cleanup;
+ }
+ burst_before = cg_read_key_long(cpucg, "cpu.stat", "burst_usec");
+ usleep(300000);
+ kill(pid, SIGKILL);
+ waitpid(pid, NULL, 0);
+
+ /* Write back what is left: the next refill counts the 300ms run. */
+ runtime_usec = cg_read_long(cpucg, "cpu.max.runtime");
+ if (cg_write_numeric(cpucg, "cpu.max.runtime", runtime_usec))
+ goto cleanup;
+ sleep(1);
+ burst_after = cg_read_key_long(cpucg, "cpu.stat", "burst_usec");
+ if (burst_before < 0 || burst_after - burst_before < 250000)
+ goto cleanup;
+
+ ret = KSFT_PASS;
+
+cleanup:
+ cg_destroy(cpucg);
+ free(cpucg);
+ return ret;
+}
+
+/*
+ * cpu.max.runtime reads at most quota + burst, so that a value read can be
+ * written back. A cfs_rq that dequeues returns its unused runtime, which can
+ * take the runtime above quota + burst until the next refill caps it: run
+ * tasks that keep doing that, and read and write back the runtime meanwhile.
+ */
+static int test_cpucg_max_runtime_read_capped(const char *root)
+{
+ int ret = KSFT_FAIL;
+ long cap_usec = 100000; /* quota 100ms, no burst */
+ long long end;
+ long runtime_usec;
+ char *cpucg;
+ int pids[2];
+ int i, n = 0;
+
+ cpucg = cg_name(root, "cpucg_runtime_read_capped_test");
+ if (!cpucg)
+ goto cleanup;
+
+ if (cg_create(cpucg))
+ goto cleanup;
+
+ if (cg_write(cpucg, "cpu.max", "100000 100000"))
+ goto cleanup;
+
+ for (n = 0; n < 2; n++) {
+ pids[n] = cg_run_nowait(cpucg, run_and_sleep, (void *)4L);
+ if (pids[n] < 0)
+ goto cleanup_tasks;
+ }
+
+ end = monotonic_ns() + 3000000000LL;
+ while (monotonic_ns() < end) {
+ runtime_usec = cg_read_long(cpucg, "cpu.max.runtime");
+ if (runtime_usec < 0 || runtime_usec > cap_usec)
+ goto cleanup_tasks;
+ if (cg_write_numeric(cpucg, "cpu.max.runtime", runtime_usec))
+ goto cleanup_tasks;
+ }
+
+ ret = KSFT_PASS;
+
+cleanup_tasks:
+ for (i = 0; i < n; i++) {
+ kill(pids[i], SIGKILL);
+ waitpid(pids[i], NULL, 0);
+ }
+cleanup:
+ cg_destroy(cpucg);
+ free(cpucg);
+ return ret;
+}
+
#define T(x) { x, #x }
struct cpucg_test {
int (*fn)(const char *root);
@@ -823,6 +1154,11 @@ struct cpucg_test {
T(test_cpucg_nested_weight_underprovisioned),
T(test_cpucg_max),
T(test_cpucg_max_nested),
+ T(test_cpucg_max_runtime),
+ T(test_cpucg_max_runtime_burst_stats),
+ T(test_cpucg_max_runtime_unthrottle),
+ T(test_cpucg_max_runtime_burst_kept),
+ T(test_cpucg_max_runtime_read_capped),
};
#undef T
--
2.47.3
Amazon Development Centre (South Africa) (Proprietary) Limited
29 Gogosoa Street, Observatory, Cape Town, Western Cape, 7925, South Africa
Registration Number: 2004 / 034463 / 07
^ permalink raw reply related [flat|nested] 14+ messages in thread* Re: [PATCH v2 1/4] sched/fair: expose cpu.max.runtime to set bandwidth runtime directly
2026-11-10 11:08 ` [PATCH v2 1/4] sched/fair: expose cpu.max.runtime to set bandwidth runtime directly Fernand Sieber
@ 2026-11-11 8:28 ` bot+bpf-ci
0 siblings, 0 replies; 14+ messages in thread
From: bot+bpf-ci @ 2026-11-11 8:28 UTC (permalink / raw)
To: sieberf, mingo, peterz, juri.lelli, vincent.guittot, tj
Cc: bsegall, dietmar.eggemann, rostedt, mgorman, vschneid,
kprateek.nayak, hannes, mkoutny, void, arighi, changwoo, skhan,
corbet, rdunlap, cgroups, sched-ext, linux-doc, linux-kselftest,
bpf, linux-kernel, nh-open-source, hborghor, fmubeen, dwmw,
karataev, sieberf, ast, andrii, daniel, martin.lau, eddyz87,
yonghong.song, mason, ihor.solodrai
[-- Attachment #1: Type: text/plain, Size: 1901 bytes --]
> sched/fair: expose cpu.max.runtime to set bandwidth runtime directly
TL;DR: In fair.c, cpu.max.runtime may return stale leftover runtime (e.g.
50000) after a group is switched to "max" quota, while writes fail; a
behaviour question about returning 0 or failing the read.
> diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
> --- a/kernel/sched/fair.c
> +++ b/kernel/sched/fair.c
[ ... ]
> +u64 cfs_bandwidth_get_runtime(struct cfs_bandwidth *cfs_b)
> +{
> + u64 runtime;
> +
> + guard(raw_spinlock_irq)(&cfs_b->lock);
> +
> + runtime = cfs_b->runtime;
> + if (cfs_b->quota != RUNTIME_INF)
> + runtime = min(runtime, cfs_b->quota + cfs_b->burst);
> +
> + return runtime;
> +}
When the group is switched to "max", does cpu.max.runtime return a stale
value?
tg_set_cfs_bandwidth() calls __refill_cfs_bandwidth_runtime(), which
returns early when quota is RUNTIME_INF:
if (unlikely(cfs_b->quota == RUNTIME_INF))
return;
After that nothing updates cfs_b->runtime, so the value left over from
the last finite quota is returned uncapped here.
For example, write "50000 100000" to cpu.max, leave the group idle, then
write "max 100000". cpu.max.runtime keeps reading 50000 for as long as
the group is unlimited, while a group that never had a quota reads 0.
Writes fail with EINVAL in that state, but the read reports runtime the
group does not have and that is not being tracked.
Should this return a defined value (for example 0) when quota is
RUNTIME_INF, or fail the read the way the write fails?
This is documented as "The runtime the cgroup has accumulated and not
used yet", which has no meaning for an unconstrained group.
---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md
CI run summary: https://github.com/kernel-patches/bpf/actions/runs/42982707042
^ permalink raw reply [flat|nested] 14+ messages in thread
* [PATCH v2 2/4] sched/fair: refill bandwidth for the periods the timer slept through
2026-11-10 11:08 [PATCH v2 0/4] sched/fair: cpu.max.runtime and idle refill for CPU credits Fernand Sieber
2026-11-10 11:08 ` [PATCH v2 1/4] sched/fair: expose cpu.max.runtime to set bandwidth runtime directly Fernand Sieber
@ 2026-11-10 11:08 ` Fernand Sieber
2026-11-10 11:08 ` [PATCH v2 3/4] sched_ext: add cgroup_set_runtime ops callback Fernand Sieber
` (3 subsequent siblings)
5 siblings, 0 replies; 14+ messages in thread
From: Fernand Sieber @ 2026-11-10 11:08 UTC (permalink / raw)
To: mingo, peterz, juri.lelli, vincent.guittot, tj
Cc: bsegall, dietmar.eggemann, rostedt, mgorman, vschneid,
kprateek.nayak, hannes, mkoutny, void, arighi, changwoo, skhan,
corbet, rdunlap, cgroups, sched-ext, linux-doc, linux-kselftest,
bpf, linux-kernel, nh-open-source, hborghor, fmubeen, dwmw,
karataev, sieberf
cpu.max.burst lets a group accumulate unused quota up to quota + burst.
But the period timer stops after a period in which the group took no
runtime, and start_cfs_bandwidth() restarts it with hrtimer_forward_now()
and discards the periods that passed. A group that sits idle stops
accumulating, and one that wakes after idling for a long time has only
what it had when the timer stopped.
Count the missed periods instead. While the timer is stopped, its expiry
is the first period boundary it missed, so the overrun at restart is the
number of refills owed. Add quota for each, capped at quota + burst as
__refill_cfs_bandwidth_runtime() caps it.
Without a burst the cap is quota. The timer stops right after a refill
that nothing drew from, so the runtime is at quota then, and groups
without a burst see no change. Groups with a burst do see a change. A
group that has been idle now has up to quota + burst when it runs again.
The documentation of cpu.max.burst, cpu.max.runtime and cpu.cfs_burst_us
says so.
A change of the bandwidth (cpu.max, cpu.max.burst) applies from that
moment, and the group keeps what it earned before it. The periods owed
are first refilled at the bandwidth they were earned under, capped at
its quota + burst, and then the new bandwidth applies, with its own cap.
A group that had no quota earned nothing, so one given a quota after a
long time unconstrained does not start full. A cpu.max.runtime write
also refills the periods owed first, so that it replaces them, and only
the periods after the write refill.
cpu.max.runtime reports the runtime the group has now, owed refills
included, without starting the timer.
The idle periods are not added to cpu.stat's nr_periods, which counts the
periods in which the timer ran. A timer that runs late by several
periods still refills only once, unlike a stopped one; this patch leaves
that as it is.
Signed-off-by: Fernand Sieber <sieberf@amazon.com>
---
Documentation/admin-guide/cgroup-v2.rst | 15 +-
Documentation/scheduler/sched-bwc.rst | 5 +-
kernel/sched/core.c | 6 +
kernel/sched/fair.c | 79 +++++++++-
kernel/sched/sched.h | 1 +
tools/testing/selftests/cgroup/test_cpu.c | 183 ++++++++++++++++++++++
6 files changed, 278 insertions(+), 11 deletions(-)
diff --git a/Documentation/admin-guide/cgroup-v2.rst b/Documentation/admin-guide/cgroup-v2.rst
index c5a02057f0242..2c220e5998639 100644
--- a/Documentation/admin-guide/cgroup-v2.rst
+++ b/Documentation/admin-guide/cgroup-v2.rst
@@ -1232,9 +1232,9 @@ will be referred to. All time durations are in microseconds.
cgroups. The default is "0".
The burst. The cgroup can accumulate the runtime it does not
- use, up to $MAX + cpu.max.burst. The burst may exceed $MAX, as
- long as $MAX + cpu.max.burst is at most 2^44 - 1 (about 203
- days).
+ use, up to $MAX + cpu.max.burst, also in the periods in which it
+ does not run. The burst may exceed $MAX, as long as $MAX +
+ cpu.max.burst is at most 2^44 - 1 (about 203 days).
This file affects only processes under the fair-class scheduler and a BPF
scheduler with the ``cgroup_set_bandwidth`` callback depending on what
@@ -1246,10 +1246,11 @@ will be referred to. All time durations are in microseconds.
The runtime the cgroup has accumulated and not used yet, in
microseconds. Each period refills it by $MAX, up to $MAX +
- cpu.max.burst, and a read returns at most $MAX + cpu.max.burst.
- A write sets it, up to $MAX + cpu.max.burst, and lets a cgroup
- that ran out of runtime run again at once. A write fails with
- EINVAL while $MAX is "max".
+ cpu.max.burst, whether the cgroup runs in it or not. A read
+ returns at most $MAX + cpu.max.burst. A write sets it, up to
+ $MAX + cpu.max.burst, and lets a cgroup that ran out of runtime
+ run again at once. A write fails with EINVAL while $MAX is
+ "max".
The runtime does not include what the cgroup's CPUs already took
from it, at most a slice each (sched_cfs_bandwidth_slice_us).
diff --git a/Documentation/scheduler/sched-bwc.rst b/Documentation/scheduler/sched-bwc.rst
index c22b3c8c14e29..ccf7434ca73f0 100644
--- a/Documentation/scheduler/sched-bwc.rst
+++ b/Documentation/scheduler/sched-bwc.rst
@@ -103,8 +103,9 @@ A value of 0 for cpu.cfs_burst_us indicates that the group can not accumulate
any unused bandwidth. It makes the traditional bandwidth control behavior for
CFS unchanged. Writing any (valid) positive value(s) into cpu.cfs_burst_us
will enact the cap on unused bandwidth accumulation: the group can accumulate
-up to cpu.cfs_quota_us + cpu.cfs_burst_us. The burst may be larger than the
-quota, as long as their sum is at most 2^44 - 1 microseconds (about 203 days).
+up to cpu.cfs_quota_us + cpu.cfs_burst_us, also over periods in which it does
+not run. The burst may be larger than the quota, as long as their sum is at
+most 2^44 - 1 microseconds (about 203 days).
Any updates to a group's bandwidth specification will result in it becoming
unthrottled if it is in a constrained state.
diff --git a/kernel/sched/core.c b/kernel/sched/core.c
index 76fcd70ff207a..75a5f5d1b8f54 100644
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -9856,6 +9856,12 @@ static int tg_set_cfs_bandwidth(struct task_group *tg,
cfs_bandwidth_usage_inc();
scoped_guard (raw_spinlock_irq, &cfs_b->lock) {
+ /*
+ * The new bandwidth applies from now on. The group keeps what
+ * it earned while idle under the old one.
+ */
+ cfs_bandwidth_refill_idle_periods(cfs_b);
+
cfs_b->period = ns_to_ktime(period);
cfs_b->quota = quota;
cfs_b->burst = burst;
diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index 58e948ed1792f..bc068b3fa4ed1 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -7813,6 +7813,75 @@ static void init_cfs_rq_runtime(struct cfs_rq *cfs_rq)
INIT_LIST_HEAD(&cfs_rq->throttled_limbo_list);
}
+/*
+ * Runtime after @periods refills of quota, capped at quota + burst as each
+ * refill in __refill_cfs_bandwidth_runtime() caps it. The runtime can be above
+ * the cap (__return_cfs_rq_runtime() does not cap it), and a refill then cuts
+ * it to the cap.
+ */
+static u64 cfs_bandwidth_refilled(struct cfs_bandwidth *cfs_b, u64 periods)
+{
+ u64 cap = cfs_b->quota + cfs_b->burst;
+ u64 runtime = cfs_b->runtime;
+
+ if (!periods || cfs_b->quota == RUNTIME_INF)
+ return runtime;
+ if (runtime >= cap ||
+ periods >= div64_u64(cap - runtime, cfs_b->quota) + 1)
+ return cap;
+ return runtime + periods * cfs_b->quota;
+}
+
+/*
+ * The period timer stops after a period in which the group took no runtime
+ * (do_sched_cfs_period_timer()), and start_cfs_bandwidth() restarts it on
+ * the next demand. The periods it slept through refill quota too: unused
+ * quota accumulates up to quota + burst whether or not the group ran. While
+ * the timer is stopped its expiry is the first period boundary it missed,
+ * so the boundaries passed since then are the refills still owed.
+ *
+ * Without burst the cap is quota. The timer stops right after a refill that
+ * nothing drew from, so the runtime is at quota then, and this changes nothing
+ * for groups without a burst.
+ *
+ * This is the overrun that hrtimer_forward_now() returns, without moving the
+ * expiry: one for the expiry itself, plus one for each full period since.
+ */
+static u64 cfs_bandwidth_idle_periods(struct cfs_bandwidth *cfs_b)
+{
+ ktime_t now = ktime_get();
+ s64 delta;
+
+ if (cfs_b->period_active)
+ return 0;
+
+ delta = ktime_to_ns(ktime_sub(now, hrtimer_get_expires(&cfs_b->period_timer)));
+ if (delta < 0)
+ return 0;
+ return div64_u64(delta, ktime_to_ns(cfs_b->period)) + 1;
+}
+
+/*
+ * Refill the periods a stopped timer slept through, at the bandwidth they
+ * were earned under, and move its expiry past them so that they are not
+ * counted again. A change of the bandwidth or a cpu.max.runtime write calls
+ * this first, so that the change applies from then on and the group keeps
+ * what it earned before. A group without a quota earned nothing.
+ */
+void cfs_bandwidth_refill_idle_periods(struct cfs_bandwidth *cfs_b)
+{
+ u64 idle_periods;
+
+ lockdep_assert_held(&cfs_b->lock);
+
+ if (cfs_b->period_active)
+ return;
+
+ idle_periods = hrtimer_forward_now(&cfs_b->period_timer, cfs_b->period);
+ cfs_b->runtime = cfs_bandwidth_refilled(cfs_b, idle_periods);
+ cfs_b->runtime_snap = cfs_b->runtime;
+}
+
/*
* The runtime of @cfs_b, as cpu.max.runtime reads it.
*
@@ -7826,7 +7895,7 @@ u64 cfs_bandwidth_get_runtime(struct cfs_bandwidth *cfs_b)
guard(raw_spinlock_irq)(&cfs_b->lock);
- runtime = cfs_b->runtime;
+ runtime = cfs_bandwidth_refilled(cfs_b, cfs_bandwidth_idle_periods(cfs_b));
if (cfs_b->quota != RUNTIME_INF)
runtime = min(runtime, cfs_b->quota + cfs_b->burst);
@@ -7849,6 +7918,12 @@ int cfs_bandwidth_set_runtime(struct cfs_bandwidth *cfs_b, u64 runtime)
runtime > cfs_b->quota + cfs_b->burst)
return -EINVAL;
+ /*
+ * Refill the periods that passed before the write, so that the
+ * write replaces them and only the periods after it refill.
+ */
+ cfs_bandwidth_refill_idle_periods(cfs_b);
+
/*
* The next refill counts runtime_snap - runtime as burst usage.
* Move runtime_snap by what the write adds or takes away, so
@@ -7879,8 +7954,8 @@ void start_cfs_bandwidth(struct cfs_bandwidth *cfs_b)
if (cfs_b->period_active)
return;
+ cfs_bandwidth_refill_idle_periods(cfs_b);
cfs_b->period_active = 1;
- hrtimer_forward_now(&cfs_b->period_timer, cfs_b->period);
hrtimer_start_expires(&cfs_b->period_timer, HRTIMER_MODE_ABS_PINNED);
}
diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h
index 1f60c84e0dc38..676e6371d0265 100644
--- a/kernel/sched/sched.h
+++ b/kernel/sched/sched.h
@@ -580,6 +580,7 @@ extern void __refill_cfs_bandwidth_runtime(struct cfs_bandwidth *cfs_b);
extern void start_cfs_bandwidth(struct cfs_bandwidth *cfs_b);
u64 cfs_bandwidth_get_runtime(struct cfs_bandwidth *cfs_b);
int cfs_bandwidth_set_runtime(struct cfs_bandwidth *cfs_b, u64 runtime);
+void cfs_bandwidth_refill_idle_periods(struct cfs_bandwidth *cfs_b);
extern void unthrottle_cfs_rq(struct cfs_rq *cfs_rq);
extern bool cfs_task_bw_constrained(struct task_struct *p);
diff --git a/tools/testing/selftests/cgroup/test_cpu.c b/tools/testing/selftests/cgroup/test_cpu.c
index 278d7e16eeb2c..04a061da514ce 100644
--- a/tools/testing/selftests/cgroup/test_cpu.c
+++ b/tools/testing/selftests/cgroup/test_cpu.c
@@ -1140,6 +1140,186 @@ static int test_cpucg_max_runtime_read_capped(const char *root)
return ret;
}
+/*
+ * A group with a burst keeps earning quota while it is idle, up to
+ * quota + burst, and cpu.max.runtime reads what it has earned. Without a
+ * burst the runtime stays at quota.
+ */
+static int test_cpucg_max_idle_refill(const char *root)
+{
+ int ret = KSFT_FAIL;
+ long quota_usec = 10000; /* 10ms */
+ long period_usec = 100000; /* 100ms */
+ long burst_usec = 990000; /* cap 1s */
+ long runtime_usec;
+ char *cpucg;
+ char buf[64];
+
+ cpucg = cg_name(root, "cpucg_idle_refill_test");
+ if (!cpucg)
+ goto cleanup;
+
+ if (cg_create(cpucg))
+ goto cleanup;
+
+ snprintf(buf, sizeof(buf), "%ld %ld", quota_usec, period_usec);
+ if (cg_write(cpucg, "cpu.max", buf))
+ goto cleanup;
+
+ /* No burst: the runtime stays at quota while idle. */
+ sleep(1);
+ runtime_usec = cg_read_long(cpucg, "cpu.max.runtime");
+ if (runtime_usec != quota_usec)
+ goto cleanup;
+
+ /* Burst: the write refills once (20ms), then 1s idle earns 10 more. */
+ if (cg_write_numeric(cpucg, "cpu.max.burst", burst_usec))
+ goto cleanup;
+ sleep(1);
+ runtime_usec = cg_read_long(cpucg, "cpu.max.runtime");
+ if (runtime_usec < 110000 || runtime_usec > 130000)
+ goto cleanup;
+
+ /* Capped at quota + burst. */
+ sleep(10);
+ runtime_usec = cg_read_long(cpucg, "cpu.max.runtime");
+ if (runtime_usec != quota_usec + burst_usec)
+ goto cleanup;
+
+ /*
+ * A write sets the runtime from now on: 500ms later, five refills, or
+ * six if the sleep overruns into the next period.
+ */
+ if (cg_write_numeric(cpucg, "cpu.max.runtime", 0))
+ goto cleanup;
+ usleep(500000);
+ runtime_usec = cg_read_long(cpucg, "cpu.max.runtime");
+ if (runtime_usec < 50000 || runtime_usec > 60000)
+ goto cleanup;
+
+ ret = KSFT_PASS;
+
+cleanup:
+ cg_destroy(cpucg);
+ free(cpucg);
+ return ret;
+}
+
+/*
+ * A cpu.max or cpu.max.burst write applies from then on: what a group with
+ * a quota earned while idle is kept, capped at the new quota + burst, and a
+ * group that had no quota has earned nothing while it had none.
+ */
+static int test_cpucg_max_idle_refill_write(const char *root)
+{
+ int ret = KSFT_FAIL;
+ long quota_usec = 10000; /* 10ms */
+ long period_usec = 100000; /* 100ms */
+ long runtime_usec;
+ char *cpucg = NULL, *fresh = NULL;
+ char buf[64];
+
+ cpucg = cg_name(root, "cpucg_idle_refill_write_test");
+ fresh = cg_name(root, "cpucg_idle_refill_fresh_test");
+ if (!cpucg || !fresh)
+ goto cleanup;
+
+ if (cg_create(cpucg) || cg_create(fresh))
+ goto cleanup;
+
+ snprintf(buf, sizeof(buf), "%ld %ld", quota_usec, period_usec);
+ if (cg_write(cpucg, "cpu.max", buf))
+ goto cleanup;
+ if (cg_write_numeric(cpucg, "cpu.max.burst", 190000))
+ goto cleanup;
+
+ /* 3s idle fills the group to quota + burst. */
+ sleep(3);
+ runtime_usec = cg_read_long(cpucg, "cpu.max.runtime");
+ if (runtime_usec != 200000)
+ goto cleanup;
+
+ /* Writing the same cpu.max keeps what the group earned. */
+ if (cg_write(cpucg, "cpu.max", buf))
+ goto cleanup;
+ runtime_usec = cg_read_long(cpucg, "cpu.max.runtime");
+ if (runtime_usec != 200000)
+ goto cleanup;
+
+ /* A lower burst keeps it too, capped at the new quota + burst. */
+ sleep(2);
+ if (cg_write_numeric(cpucg, "cpu.max.burst", 90000))
+ goto cleanup;
+ runtime_usec = cg_read_long(cpucg, "cpu.max.runtime");
+ if (runtime_usec != 100000)
+ goto cleanup;
+
+ /* A group given a quota after idling without one starts with a refill. */
+ if (cg_write_numeric(fresh, "cpu.max.burst", 190000))
+ goto cleanup;
+ sleep(2);
+ if (cg_write(fresh, "cpu.max", buf))
+ goto cleanup;
+ runtime_usec = cg_read_long(fresh, "cpu.max.runtime");
+ if (runtime_usec != quota_usec)
+ goto cleanup;
+
+ ret = KSFT_PASS;
+
+cleanup:
+ if (fresh)
+ cg_destroy(fresh);
+ if (cpucg)
+ cg_destroy(cpucg);
+ free(fresh);
+ free(cpucg);
+ return ret;
+}
+
+/*
+ * A cpu.max.runtime write sets the runtime exactly, and only the periods
+ * after it refill.
+ */
+static int test_cpucg_max_idle_refill_runtime_write(const char *root)
+{
+ int ret = KSFT_FAIL;
+ long runtime_usec;
+ char *cpucg;
+
+ cpucg = cg_name(root, "cpucg_idle_refill_runtime_test");
+ if (!cpucg)
+ goto cleanup;
+
+ if (cg_create(cpucg))
+ goto cleanup;
+
+ if (cg_write(cpucg, "cpu.max", "10000 100000"))
+ goto cleanup;
+ if (cg_write_numeric(cpucg, "cpu.max.burst", 190000))
+ goto cleanup;
+
+ sleep(3);
+ if (cg_write_numeric(cpucg, "cpu.max.runtime", 30000))
+ goto cleanup;
+ /* One refill if a period boundary passes before the read. */
+ runtime_usec = cg_read_long(cpucg, "cpu.max.runtime");
+ if (runtime_usec != 30000 && runtime_usec != 40000)
+ goto cleanup;
+
+ /* 250ms after the write: two or three refills, by the boundaries. */
+ usleep(250000);
+ runtime_usec = cg_read_long(cpucg, "cpu.max.runtime");
+ if (runtime_usec < 50000 || runtime_usec > 60000)
+ goto cleanup;
+
+ ret = KSFT_PASS;
+
+cleanup:
+ cg_destroy(cpucg);
+ free(cpucg);
+ return ret;
+}
+
#define T(x) { x, #x }
struct cpucg_test {
int (*fn)(const char *root);
@@ -1159,6 +1339,9 @@ struct cpucg_test {
T(test_cpucg_max_runtime_unthrottle),
T(test_cpucg_max_runtime_burst_kept),
T(test_cpucg_max_runtime_read_capped),
+ T(test_cpucg_max_idle_refill),
+ T(test_cpucg_max_idle_refill_write),
+ T(test_cpucg_max_idle_refill_runtime_write),
};
#undef T
--
2.47.3
Amazon Development Centre (South Africa) (Proprietary) Limited
29 Gogosoa Street, Observatory, Cape Town, Western Cape, 7925, South Africa
Registration Number: 2004 / 034463 / 07
^ permalink raw reply related [flat|nested] 14+ messages in thread* [PATCH v2 3/4] sched_ext: add cgroup_set_runtime ops callback
2026-11-10 11:08 [PATCH v2 0/4] sched/fair: cpu.max.runtime and idle refill for CPU credits Fernand Sieber
2026-11-10 11:08 ` [PATCH v2 1/4] sched/fair: expose cpu.max.runtime to set bandwidth runtime directly Fernand Sieber
2026-11-10 11:08 ` [PATCH v2 2/4] sched/fair: refill bandwidth for the periods the timer slept through Fernand Sieber
@ 2026-11-10 11:08 ` Fernand Sieber
2026-11-11 8:08 ` bot+bpf-ci
2026-11-10 11:08 ` [PATCH v2 4/4] sched_ext: charge BPF scheduler tasks to cpu.max bandwidth Fernand Sieber
` (2 subsequent siblings)
5 siblings, 1 reply; 14+ messages in thread
From: Fernand Sieber @ 2026-11-10 11:08 UTC (permalink / raw)
To: mingo, peterz, juri.lelli, vincent.guittot, tj
Cc: bsegall, dietmar.eggemann, rostedt, mgorman, vschneid,
kprateek.nayak, hannes, mkoutny, void, arighi, changwoo, skhan,
corbet, rdunlap, cgroups, sched-ext, linux-doc, linux-kselftest,
bpf, linux-kernel, nh-open-source, hborghor, fmubeen, dwmw,
karataev, sieberf
Add a sched_ext_ops callback, and its cid-form variant, that is invoked
when userspace writes cpu.max.runtime. A BPF scheduler that implements
ops.cgroup_set_bandwidth() to enforce a cgroup's CPU bandwidth also
needs the written runtime, so that its budget for the cgroup matches the
fair class's, for example after userspace restores a saved runtime or
adds to it.
The callback is invoked on every write, after the fair class has applied
it. Unlike cgroup_set_bandwidth(), it does not skip a write of the value
last written. The runtime goes down as the cgroup runs, so writing the
same value again is a top-up.
A new tg_bandwidth_mutex is held across the fair class's update and the
callback, by the cpu.max.runtime write and by tg_set_bandwidth(), which
the cpu.max and cgroup v1 cpu.cfs_* writes go through and which calls
cgroup_set_bandwidth(). The runtime passed is then at most the quota +
burst the scheduler was last given, and cpu.max and cpu.max.runtime
writes reach it in the order the fair class applied them. The mutex is
taken before, and the callbacks run after, the fair class's
cpus_read_lock() and cfs_constraints_mutex sections, so
cgroup_set_bandwidth() runs as it did before.
Like cgroup_set_bandwidth(), the callback runs in process context and
may be sleepable.
Signed-off-by: Fernand Sieber <sieberf@amazon.com>
---
Documentation/admin-guide/cgroup-v2.rst | 4 +++-
kernel/sched/core.c | 25 +++++++++++++++++++++++--
kernel/sched/ext/ext.c | 19 +++++++++++++++++++
kernel/sched/ext/ext.h | 2 ++
kernel/sched/ext/internal.h | 21 +++++++++++++++++++++
5 files changed, 68 insertions(+), 3 deletions(-)
diff --git a/Documentation/admin-guide/cgroup-v2.rst b/Documentation/admin-guide/cgroup-v2.rst
index 2c220e5998639..b77ca6f8ca548 100644
--- a/Documentation/admin-guide/cgroup-v2.rst
+++ b/Documentation/admin-guide/cgroup-v2.rst
@@ -1256,7 +1256,9 @@ will be referred to. All time durations are in microseconds.
from it, at most a slice each (sched_cfs_bandwidth_slice_us).
A write leaves that with the CPUs.
- This file affects only processes under the fair-class scheduler.
+ This file affects only processes under the fair-class scheduler and a BPF
+ scheduler with the ``cgroup_set_runtime`` callback depending on what
+ the callback actually does.
cpu.pressure
A read-write nested-keyed file.
diff --git a/kernel/sched/core.c b/kernel/sched/core.c
index 75a5f5d1b8f54..25a4acb72106c 100644
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -10099,6 +10099,14 @@ static u64 cpu_period_read_u64(struct cgroup_subsys_state *css,
return period_us;
}
+/*
+ * Held across a bandwidth or runtime write and the sched_ext callback that
+ * follows it, so that the BPF scheduler gets the writes in the order the fair
+ * class applied them. The callback runs after the fair class's update has
+ * dropped cpus_read_lock() and cfs_constraints_mutex.
+ */
+static DEFINE_MUTEX(tg_bandwidth_mutex);
+
static int tg_set_bandwidth(struct task_group *tg,
u64 period_us, u64 quota_us, u64 burst_us)
{
@@ -10140,6 +10148,8 @@ static int tg_set_bandwidth(struct task_group *tg,
if (quota_us != RUNTIME_INF && burst_us > max_bw_runtime_us - quota_us)
return -EINVAL;
+ guard(mutex)(&tg_bandwidth_mutex);
+
#ifdef CONFIG_CFS_BANDWIDTH
ret = tg_set_cfs_bandwidth(tg, period_us, quota_us, burst_us);
#endif /* CONFIG_CFS_BANDWIDTH */
@@ -10204,12 +10214,23 @@ static int cpu_runtime_write_u64(struct cgroup_subsys_state *css,
struct cftype *cftype, u64 runtime_us)
{
struct task_group *tg = css_tg(css);
+ int ret;
if (runtime_us > max_bw_runtime_us)
return -EINVAL;
- return cfs_bandwidth_set_runtime(&tg->cfs_bandwidth,
- runtime_us * NSEC_PER_USEC);
+ /*
+ * Serialize against tg_set_bandwidth(), so that the BPF scheduler gets
+ * the runtime checked against the current quota + burst, and this and
+ * cpu.max writes reach it in the order the fair class took them.
+ */
+ guard(mutex)(&tg_bandwidth_mutex);
+
+ ret = cfs_bandwidth_set_runtime(&tg->cfs_bandwidth,
+ runtime_us * NSEC_PER_USEC);
+ if (!ret)
+ scx_group_set_runtime(tg, runtime_us);
+ return ret;
}
static u64 cpu_runtime_read_u64(struct cgroup_subsys_state *css,
diff --git a/kernel/sched/ext/ext.c b/kernel/sched/ext/ext.c
index ebdf7b91f9d37..265a80dd93282 100644
--- a/kernel/sched/ext/ext.c
+++ b/kernel/sched/ext/ext.c
@@ -5817,6 +5817,19 @@ void scx_group_set_bandwidth(struct task_group *tg,
percpu_up_read(&scx_cgroup_ops_rwsem);
}
+
+void scx_group_set_runtime(struct task_group *tg, u64 runtime_us)
+{
+ struct scx_sched *sch;
+
+ percpu_down_read(&scx_cgroup_ops_rwsem);
+ sch = scx_tg_knob_sched(tg);
+
+ if (scx_cgroup_enabled && sch && SCX_HAS_OP(sch, cgroup_set_runtime))
+ SCX_CALL_OP(sch, cgroup_set_runtime, NULL, tg_cgrp(tg), runtime_us);
+
+ percpu_up_read(&scx_cgroup_ops_rwsem);
+}
#endif /* CONFIG_EXT_GROUP_SCHED */
#if defined(CONFIG_EXT_GROUP_SCHED) || defined(CONFIG_EXT_SUB_SCHED)
@@ -9230,6 +9243,7 @@ static int bpf_scx_check_member(const struct btf_type *t,
case offsetof(struct sched_ext_ops, cgroup_exit):
case offsetof(struct sched_ext_ops, cgroup_prep_move):
case offsetof(struct sched_ext_ops, cgroup_set_bandwidth):
+ case offsetof(struct sched_ext_ops, cgroup_set_runtime):
#endif
case offsetof(struct sched_ext_ops, cpu_online):
case offsetof(struct sched_ext_ops, cpu_offline):
@@ -9392,6 +9406,7 @@ static void sched_ext_ops__cgroup_cancel_move(struct task_struct *p, struct cgro
static void sched_ext_ops__cgroup_set_weight(struct cgroup *cgrp, u32 weight) {}
static void sched_ext_ops__cgroup_set_bandwidth(struct cgroup *cgrp, u64 period_us, u64 quota_us, u64 burst_us) {}
static void sched_ext_ops__cgroup_set_idle(struct cgroup *cgrp, bool idle) {}
+static void sched_ext_ops__cgroup_set_runtime(struct cgroup *cgrp, u64 runtime_us) {}
#endif /* CONFIG_EXT_GROUP_SCHED */
static s32 sched_ext_ops__sub_attach(struct scx_sub_attach_args *args) { return -EINVAL; }
static void sched_ext_ops__sub_detach(struct scx_sub_detach_args *args) {}
@@ -9434,6 +9449,7 @@ static struct sched_ext_ops __bpf_ops_sched_ext_ops = {
.cgroup_set_weight = sched_ext_ops__cgroup_set_weight,
.cgroup_set_bandwidth = sched_ext_ops__cgroup_set_bandwidth,
.cgroup_set_idle = sched_ext_ops__cgroup_set_idle,
+ .cgroup_set_runtime = sched_ext_ops__cgroup_set_runtime,
#endif
.sub_attach = sched_ext_ops__sub_attach,
.sub_detach = sched_ext_ops__sub_detach,
@@ -9502,6 +9518,7 @@ static struct sched_ext_ops_cid __bpf_ops_sched_ext_ops_cid = {
.cpuctl_set_weight = sched_ext_ops__cgroup_set_weight,
.cpuctl_set_bandwidth = sched_ext_ops__cgroup_set_bandwidth,
.cpuctl_set_idle = sched_ext_ops__cgroup_set_idle,
+ .cpuctl_set_runtime = sched_ext_ops__cgroup_set_runtime,
#endif
.sub_attach = sched_ext_ops__sub_attach,
.sub_detach = sched_ext_ops__sub_detach,
@@ -12226,6 +12243,7 @@ static const u32 scx_kf_allow_flags[] = {
[SCX_OP_IDX(cgroup_set_weight)] = SCX_KF_ALLOW_UNLOCKED,
[SCX_OP_IDX(cgroup_set_bandwidth)] = SCX_KF_ALLOW_UNLOCKED,
[SCX_OP_IDX(cgroup_set_idle)] = SCX_KF_ALLOW_UNLOCKED,
+ [SCX_OP_IDX(cgroup_set_runtime)] = SCX_KF_ALLOW_UNLOCKED,
#endif /* CONFIG_EXT_GROUP_SCHED */
[SCX_OP_IDX(sub_attach)] = SCX_KF_ALLOW_UNLOCKED,
[SCX_OP_IDX(sub_detach)] = SCX_KF_ALLOW_UNLOCKED,
@@ -12395,6 +12413,7 @@ static int __init scx_init(void)
CID_OFFSET_MATCH(cgroup_set_weight, cpuctl_set_weight);
CID_OFFSET_MATCH(cgroup_set_bandwidth, cpuctl_set_bandwidth);
CID_OFFSET_MATCH(cgroup_set_idle, cpuctl_set_idle);
+ CID_OFFSET_MATCH(cgroup_set_runtime, cpuctl_set_runtime);
#endif
/* @priv tail must align since both share the same data block */
CID_OFFSET_MATCH(priv, priv);
diff --git a/kernel/sched/ext/ext.h b/kernel/sched/ext/ext.h
index d32bc01071b01..6d1ce582bf724 100644
--- a/kernel/sched/ext/ext.h
+++ b/kernel/sched/ext/ext.h
@@ -94,6 +94,7 @@ void scx_cgroup_cancel_attach(struct cgroup_taskset *tset);
void scx_group_set_weight(struct task_group *tg, unsigned long cgrp_weight);
void scx_group_set_idle(struct task_group *tg, bool idle);
void scx_group_set_bandwidth(struct task_group *tg, u64 period_us, u64 quota_us, u64 burst_us);
+void scx_group_set_runtime(struct task_group *tg, u64 runtime_us);
#else /* CONFIG_EXT_GROUP_SCHED */
static inline void scx_tg_init(struct task_group *tg) {}
static inline int scx_tg_online(struct task_group *tg) { return 0; }
@@ -104,5 +105,6 @@ static inline void scx_cgroup_cancel_attach(struct cgroup_taskset *tset) {}
static inline void scx_group_set_weight(struct task_group *tg, unsigned long cgrp_weight) {}
static inline void scx_group_set_idle(struct task_group *tg, bool idle) {}
static inline void scx_group_set_bandwidth(struct task_group *tg, u64 period_us, u64 quota_us, u64 burst_us) {}
+static inline void scx_group_set_runtime(struct task_group *tg, u64 runtime_us) {}
#endif /* CONFIG_EXT_GROUP_SCHED */
#endif /* CONFIG_CGROUP_SCHED */
diff --git a/kernel/sched/ext/internal.h b/kernel/sched/ext/internal.h
index ff0a3d508533f..e794507596d6f 100644
--- a/kernel/sched/ext/internal.h
+++ b/kernel/sched/ext/internal.h
@@ -863,6 +863,26 @@ struct sched_ext_ops {
*/
void (*cgroup_set_idle)(struct cgroup *cgrp, bool idle);
+ /**
+ * @cgroup_set_runtime: A cgroup's runtime is being set
+ * @cgrp: cgroup whose runtime is being set
+ * @runtime_us: runtime in microseconds
+ *
+ * Set the runtime @cgrp has accumulated and not used yet to
+ * @runtime_us. This is from the cpu.max.runtime cgroup interface,
+ * which exists with CONFIG_CFS_BANDWIDTH only. It is called on every
+ * write, also of an unchanged value. The runtime goes down as @cgrp
+ * runs, so a write of the same value is a top-up.
+ *
+ * @cgrp has a quota, and @runtime_us is at most the quota + burst the
+ * scheduler was last given, by @cgroup_init or @cgroup_set_bandwidth:
+ * cpu.max and cpu.max.runtime writes reach the scheduler in the order
+ * the fair class applied them.
+ *
+ * Delivery follows the same rule as cgroup_set_weight().
+ */
+ void (*cgroup_set_runtime)(struct cgroup *cgrp, u64 runtime_us);
+
#endif /* CONFIG_EXT_GROUP_SCHED */
/**
@@ -1239,6 +1259,7 @@ struct sched_ext_ops_cid {
void (*cpuctl_set_bandwidth)(struct cgroup *cgrp, u64 period_us, u64 quota_us,
u64 burst_us);
void (*cpuctl_set_idle)(struct cgroup *cgrp, bool idle);
+ void (*cpuctl_set_runtime)(struct cgroup *cgrp, u64 runtime_us);
#endif /* CONFIG_EXT_GROUP_SCHED */
s32 (*sub_attach)(struct scx_sub_attach_args *args);
void (*sub_detach)(struct scx_sub_detach_args *args);
--
2.47.3
Amazon Development Centre (South Africa) (Proprietary) Limited
29 Gogosoa Street, Observatory, Cape Town, Western Cape, 7925, South Africa
Registration Number: 2004 / 034463 / 07
^ permalink raw reply related [flat|nested] 14+ messages in thread* Re: [PATCH v2 3/4] sched_ext: add cgroup_set_runtime ops callback
2026-11-10 11:08 ` [PATCH v2 3/4] sched_ext: add cgroup_set_runtime ops callback Fernand Sieber
@ 2026-11-11 8:08 ` bot+bpf-ci
0 siblings, 0 replies; 14+ messages in thread
From: bot+bpf-ci @ 2026-11-11 8:08 UTC (permalink / raw)
To: sieberf, mingo, peterz, juri.lelli, vincent.guittot, tj
Cc: bsegall, dietmar.eggemann, rostedt, mgorman, vschneid,
kprateek.nayak, hannes, mkoutny, void, arighi, changwoo, skhan,
corbet, rdunlap, cgroups, sched-ext, linux-doc, linux-kselftest,
bpf, linux-kernel, nh-open-source, hborghor, fmubeen, dwmw,
karataev, sieberf, ast, andrii, daniel, martin.lau, eddyz87,
yonghong.song, mason, ihor.solodrai
[-- Attachment #1: Type: text/plain, Size: 3194 bytes --]
> sched_ext: add cgroup_set_runtime ops callback
TL;DR: The documented guarantee that cgroup_set_runtime never exceeds the
quota the scheduler was last given may not hold for schedulers lacking
cgroup_init or cgroup_set_bandwidth; comment and commit message may need
qualifying, or validation tightened.
> diff --git a/kernel/sched/ext/internal.h b/kernel/sched/ext/internal.h
> --- a/kernel/sched/ext/internal.h
> +++ b/kernel/sched/ext/internal.h
[ ... ]
> @@ -863,6 +863,26 @@ struct sched_ext_ops {
> */
> void (*cgroup_set_idle)(struct cgroup *cgrp, bool idle);
>
> + /**
> + * @cgroup_set_runtime: A cgroup's runtime is being set
> + * @cgrp: cgroup whose runtime is being set
> + * @runtime_us: runtime in microseconds
[ ... ]
> + * @cgrp has a quota, and @runtime_us is at most the quota + burst the
> + * scheduler was last given, by @cgroup_init or @cgroup_set_bandwidth:
> + * cpu.max and cpu.max.runtime writes reach the scheduler in the order
> + * the fair class applied them.
> + *
> + * Delivery follows the same rule as cgroup_set_weight().
> + */
> + void (*cgroup_set_runtime)(struct cgroup *cgrp, u64 runtime_us);
Does this guarantee hold for a scheduler that implements
cgroup_set_runtime but not cgroup_init or cgroup_set_bandwidth? The
commit message makes the same claim: "The runtime passed is then at most
the quota + burst the scheduler was last given".
scx_group_set_bandwidth() still updates tg->scx.bw_* when the op is
missing:
if (scx_cgroup_enabled && sch && SCX_HAS_OP(sch, cgroup_set_bandwidth) && ...)
SCX_CALL_OP(sch, cgroup_set_bandwidth, ...);
tg->scx.bw_period_us = period_us;
tg->scx.bw_quota_us = quota_us;
tg->scx.bw_burst_us = burst_us;
while scx_group_set_runtime() calls the op whenever
SCX_HAS_OP(sch, cgroup_set_runtime) is true, and the fair class only
checks the runtime against its own cfs_b->quota + cfs_b->burst in
cfs_bandwidth_set_runtime().
For example, take a scheduler with cgroup_init and cgroup_set_runtime but
no cgroup_set_bandwidth. Cgroup C has cpu.max "10000 100000" at load, so
ops.cgroup_init() sees bw_quota_us=10000. Userspace then writes cpu.max
"50000 100000", which the fair class applies but the scheduler is never
told about. A following write of 40000 to cpu.max.runtime passes
cfs_bandwidth_set_runtime() since 40000 <= 50000, and
ops.cgroup_set_runtime(C, 40000) runs, four times the quota the scheduler
was last given.
Similarly, a scheduler without cgroup_init that loads after C's quota was
set is never given a quota, yet still receives cgroup_set_runtime for that
cgroup.
Should the comment (and commit message) make the guarantee conditional on
the scheduler implementing @cgroup_init and @cgroup_set_bandwidth, or
should validation reject cgroup_set_runtime without cgroup_set_bandwidth?
The later commit eb360e17b06e ("sched_ext: charge BPF scheduler tasks to
cpu.max bandwidth") does not change this text.
---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md
CI run summary: https://github.com/kernel-patches/bpf/actions/runs/42982707042
^ permalink raw reply [flat|nested] 14+ messages in thread
* [PATCH v2 4/4] sched_ext: charge BPF scheduler tasks to cpu.max bandwidth
2026-11-10 11:08 [PATCH v2 0/4] sched/fair: cpu.max.runtime and idle refill for CPU credits Fernand Sieber
` (2 preceding siblings ...)
2026-11-10 11:08 ` [PATCH v2 3/4] sched_ext: add cgroup_set_runtime ops callback Fernand Sieber
@ 2026-11-10 11:08 ` Fernand Sieber
2026-11-11 8:28 ` bot+bpf-ci
2026-11-10 20:10 ` [PATCH v2 0/4] sched/fair: cpu.max.runtime and idle refill for CPU credits Tejun Heo
2026-11-11 12:29 ` [PATCH v3 0/2] sched/fair: cpu.max.runtime and idle refill for a burstable budget Fernand Sieber
5 siblings, 1 reply; 14+ messages in thread
From: Fernand Sieber @ 2026-11-10 11:08 UTC (permalink / raw)
To: mingo, peterz, juri.lelli, vincent.guittot, tj
Cc: bsegall, dietmar.eggemann, rostedt, mgorman, vschneid,
kprateek.nayak, hannes, mkoutny, void, arighi, changwoo, skhan,
corbet, rdunlap, cgroups, sched-ext, linux-doc, linux-kselftest,
bpf, linux-kernel, nh-open-source, hborghor, fmubeen, dwmw,
karataev, sieberf
A BPF scheduler that enforces cpu.max keeps each cgroup's budget
itself, and no runtime moves between it and the fair class. One loaded
after a cpu.max.runtime write starts the cgroup on a fresh quota,
losing the unspent grant.
Conversely, a read reports the fair class's runtime, which refills to
quota + burst meanwhile. On disable, the fair class takes the tasks
back with that full runtime, so a cgroup that used up its budget runs
over its quota until the period accounting catches up.
Under SCX_OPS_SWITCH_PARTIAL, both budgets apply at once: a cgroup with
tasks in both classes runs up to its quota in each, which no handover
at load or unload can fix. Keep one budget, the fair class's, and let
a BPF scheduler opt in to drawing from it with the new
SCX_OPS_CHARGE_CPU_MAX flag.
With SCX_OPS_CHARGE_CPU_MAX, update_curr_scx() charges a task's runtime
to its cgroups' cfs_rqs as a fair task's would, without throttling it.
A task the scheduler stops late leaves its CPU in debt, which the next
periods pay, also once the task has moved; debt past minus a slice
moves to cfs_b, whose next refill pays up to a quota of it. The new
kfunc scx_bpf_task_cpu_max_runtime() returns the runtime left less that
debt, and the scheduler runs the task only while it is positive.
Cgroups whose period timer is stopped are skipped, as only a charge
restarts it.
Both classes then draw from one budget: under SCX_OPS_SWITCH_PARTIAL a
cgroup gets its quota per period however its tasks are split, and
nothing is handed over at load or unload. A task returning to the fair
class leaves its CPU at most a slice in debt; schedulers without the
flag behave as before.
cfs_b->lock is taken at most twice per millisecond of run time per CPU.
The cgroup_runtime selftest covers these cases.
Signed-off-by: Fernand Sieber <sieberf@amazon.com>
---
Documentation/admin-guide/cgroup-v2.rst | 4 +-
kernel/sched/ext/ext.c | 34 +
kernel/sched/ext/internal.h | 24 +-
kernel/sched/fair.c | 122 ++++
kernel/sched/sched.h | 12 +
tools/sched_ext/include/scx/common.bpf.h | 1 +
tools/sched_ext/include/scx/compat.h | 1 +
tools/testing/selftests/sched_ext/Makefile | 1 +
.../selftests/sched_ext/cgroup_runtime.bpf.c | 195 ++++++
.../selftests/sched_ext/cgroup_runtime.c | 652 ++++++++++++++++++
tools/testing/selftests/sched_ext/config | 1 +
11 files changed, 1045 insertions(+), 2 deletions(-)
create mode 100644 tools/testing/selftests/sched_ext/cgroup_runtime.bpf.c
create mode 100644 tools/testing/selftests/sched_ext/cgroup_runtime.c
diff --git a/Documentation/admin-guide/cgroup-v2.rst b/Documentation/admin-guide/cgroup-v2.rst
index b77ca6f8ca548..db9fa6e6dbd49 100644
--- a/Documentation/admin-guide/cgroup-v2.rst
+++ b/Documentation/admin-guide/cgroup-v2.rst
@@ -1258,7 +1258,9 @@ will be referred to. All time durations are in microseconds.
This file affects only processes under the fair-class scheduler and a BPF
scheduler with the ``cgroup_set_runtime`` callback depending on what
- the callback actually does.
+ the callback actually does. A BPF scheduler with the
+ ``SCX_OPS_CHARGE_CPU_MAX`` flag draws what its processes run from
+ the same runtime, so the cgroup's processes in both share it.
cpu.pressure
A read-write nested-keyed file.
diff --git a/kernel/sched/ext/ext.c b/kernel/sched/ext/ext.c
index 265a80dd93282..6ffc9ca2c5c9f 100644
--- a/kernel/sched/ext/ext.c
+++ b/kernel/sched/ext/ext.c
@@ -1625,6 +1625,11 @@ static void expire_task_slice(struct task_struct *p)
scx_set_task_slice(p, 0);
}
+static bool scx_task_charges_cpu_max(const struct task_struct *p)
+{
+ return scx_task_sched(p)->ops.flags & SCX_OPS_CHARGE_CPU_MAX;
+}
+
static void update_curr_scx(struct rq *rq)
{
struct task_struct *donor;
@@ -1650,6 +1655,9 @@ static void update_curr_scx(struct rq *rq)
if (unlikely(donor == scx_rescuee(rq)))
scx_rescue_charge(rq, delta_exec);
+ if (scx_task_charges_cpu_max(donor))
+ cfs_bandwidth_charge_task(rq, donor, delta_exec);
+
dl_server_update(&rq->ext_server, delta_exec);
}
@@ -2605,6 +2613,13 @@ static bool dequeue_task_scx(struct rq *rq, struct task_struct *p, int core_deq_
ops_dequeue(rq, p, deq_flags);
+ /* charge what @p ran, then return what @rq holds for its cgroups */
+ if (scx_task_charges_cpu_max(p)) {
+ if (task_current_donor(rq, p))
+ update_curr_scx(rq);
+ cfs_bandwidth_return_task(rq, p);
+ }
+
/*
* A current scheduling context which is going off @rq first gets
* dequeued and then stops running. As we want running <-> stopping
@@ -11793,6 +11808,24 @@ __bpf_kfunc s32 scx_bpf_task_cpu(const struct task_struct *p)
return task_cpu(p);
}
+/**
+ * scx_bpf_task_cpu_max_runtime - Runtime a task has left under cpu.max
+ * @p: task of interest
+ *
+ * Return the runtime in nsecs that @p's cgroup and its ancestors have left
+ * under their cpu.max quotas for @p on its current CPU, the least over those
+ * with a quota: the cgroup's cpu.max.runtime plus what the CPU holds for
+ * the cgroup, less what it owes. U64_MAX if none limits it. Read without
+ * locks.
+ *
+ * With %SCX_OPS_CHARGE_CPU_MAX, the scheduler should run @p only while this
+ * returns more than 0. The runtime grows when a period refills it.
+ */
+__bpf_kfunc u64 scx_bpf_task_cpu_max_runtime(struct task_struct *p)
+{
+ return cfs_bandwidth_task_runtime(p);
+}
+
/**
* scx_bpf_task_cid - cid a task is currently associated with
* @p: task of interest
@@ -12119,6 +12152,7 @@ BTF_ID_FLAGS(func, scx_bpf_get_online_cpumask, KF_ACQUIRE)
BTF_ID_FLAGS(func, scx_bpf_put_cpumask, KF_RELEASE)
BTF_ID_FLAGS(func, scx_bpf_task_running, KF_RCU)
BTF_ID_FLAGS(func, scx_bpf_task_cpu, KF_RCU)
+BTF_ID_FLAGS(func, scx_bpf_task_cpu_max_runtime, KF_RCU)
BTF_ID_FLAGS(func, scx_bpf_task_cid, KF_RCU)
BTF_ID_FLAGS(func, scx_bpf_locked_rq, KF_IMPLICIT_ARGS | KF_RET_NULL)
BTF_ID_FLAGS(func, scx_bpf_cpu_curr, KF_IMPLICIT_ARGS | KF_RET_NULL | KF_RCU_PROTECTED)
diff --git a/kernel/sched/ext/internal.h b/kernel/sched/ext/internal.h
index e794507596d6f..e8c0b573d0a43 100644
--- a/kernel/sched/ext/internal.h
+++ b/kernel/sched/ext/internal.h
@@ -243,6 +243,27 @@ enum scx_ops_flags {
*/
SCX_OPS_ENQ_BLOCKED = 1LLU << 10,
+ /*
+ * If set, what the scheduler's tasks run is charged to the cpu.max
+ * bandwidth of their cgroups, the runtime the fair class keeps and
+ * cpu.max.runtime reads, as what fair tasks run is. The tasks of a
+ * cgroup in both classes then share one quota, also with
+ * %SCX_OPS_SWITCH_PARTIAL, and nothing is handed over when tasks
+ * switch class.
+ *
+ * The kernel never throttles the scheduler's tasks. The scheduler
+ * enforces the quota by running a task only while
+ * scx_bpf_task_cpu_max_runtime() returns more than 0. What a task runs
+ * beyond that is charged: up to a slice (sched_cfs_bandwidth_slice_us)
+ * per CPU as a debt the cgroup's fair tasks on that CPU pay back, and up
+ * to a quota more from the cgroup's next period. The scheduler does not
+ * need ops.cgroup_set_runtime(), as a write of cpu.max.runtime sets that
+ * runtime.
+ *
+ * Without CONFIG_CFS_BANDWIDTH, this flag has no effect.
+ */
+ SCX_OPS_CHARGE_CPU_MAX = 1LLU << 11,
+
SCX_OPS_ALL_FLAGS = SCX_OPS_KEEP_BUILTIN_IDLE |
SCX_OPS_ENQ_LAST |
SCX_OPS_ENQ_EXITING |
@@ -253,7 +274,8 @@ enum scx_ops_flags {
SCX_OPS_ALWAYS_ENQ_IMMED |
SCX_OPS_TID_TO_TASK |
SCX_OPS_LAZY_RESCHED |
- SCX_OPS_ENQ_BLOCKED,
+ SCX_OPS_ENQ_BLOCKED |
+ SCX_OPS_CHARGE_CPU_MAX,
/* high 8 bits are internal, don't include in SCX_OPS_ALL_FLAGS */
__SCX_OPS_INTERNAL_MASK = 0xffLLU << 56,
diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index bc068b3fa4ed1..9ff239f122bda 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -6875,6 +6875,11 @@ void __refill_cfs_bandwidth_runtime(struct cfs_bandwidth *cfs_b)
}
cfs_b->runtime = min(cfs_b->runtime, cfs_b->quota + cfs_b->burst);
+
+ /* pay the debt cfs_bandwidth_charge_task() left */
+ runtime = min(cfs_b->runtime, cfs_b->debt);
+ cfs_b->runtime -= runtime;
+ cfs_b->debt -= runtime;
cfs_b->runtime_snap = cfs_b->runtime;
}
@@ -8076,6 +8081,123 @@ bool cfs_task_bw_constrained(struct task_struct *p)
return false;
}
+/*
+ * Charge @delta_exec, which @p ran on @rq outside the fair class, to the
+ * bandwidth of its task group and of each ancestor with a quota, as
+ * account_cfs_rq_runtime() charges a fair task. A cfs_rq that runs out takes
+ * another slice from cfs_b, but @p is never throttled: the BPF scheduler that
+ * runs it stops it, and the group's fair tasks are throttled when they run.
+ *
+ * Debt below -slice that cfs_b cannot cover moves to cfs_b, up to a quota,
+ * and the next refill pays it, so that a scheduler which keeps running @p
+ * cannot put the group's fair tasks on @rq in more debt than a slice.
+ */
+void cfs_bandwidth_charge_task(struct rq *rq, struct task_struct *p,
+ u64 delta_exec)
+{
+ s64 floor = -(s64)sched_cfs_bandwidth_slice();
+ struct task_group *tg;
+
+ lockdep_assert_rq_held(rq);
+
+ if (!cfs_bandwidth_used())
+ return;
+
+ for (tg = task_group(p); tg; tg = tg->parent) {
+ struct cfs_bandwidth *cfs_b = tg_cfs_bandwidth(tg);
+ struct cfs_rq *cfs_rq = tg_cfs_rq(tg, cpu_of(rq));
+
+ if (!cfs_rq->runtime_enabled)
+ continue;
+
+ cfs_rq->runtime_remaining -= delta_exec;
+
+ /*
+ * A throttled cfs_rq gets runtime from distribute_cfs_runtime().
+ * Without runtime, an active period timer refills it, so skip
+ * the lock. A stopped one is restarted by the assign.
+ */
+ if (cfs_rq->runtime_remaining <= 0 && !cfs_rq->throttled &&
+ (READ_ONCE(cfs_b->runtime) || !READ_ONCE(cfs_b->period_active))) {
+ scoped_guard (raw_spinlock, &cfs_b->lock)
+ __assign_cfs_rq_runtime(cfs_b, cfs_rq,
+ sched_cfs_bandwidth_slice());
+ }
+
+ if (cfs_rq->runtime_remaining < floor) {
+ scoped_guard (raw_spinlock, &cfs_b->lock)
+ cfs_b->debt = min_t(u64, cfs_b->debt + floor -
+ cfs_rq->runtime_remaining, cfs_b->quota);
+ cfs_rq->runtime_remaining = floor;
+ }
+ }
+}
+
+/*
+ * @p leaves @rq: give the runtime @rq holds for its groups back, less
+ * min_cfs_rq_runtime, as a fair dequeue does when a cfs_rq empties. Pay
+ * what @rq owes from cfs_b, as @p may not run on @rq again.
+ */
+void cfs_bandwidth_return_task(struct rq *rq, struct task_struct *p)
+{
+ struct task_group *tg;
+
+ lockdep_assert_rq_held(rq);
+
+ if (!cfs_bandwidth_used())
+ return;
+
+ for (tg = task_group(p); tg; tg = tg->parent) {
+ struct cfs_bandwidth *cfs_b = tg_cfs_bandwidth(tg);
+ struct cfs_rq *cfs_rq = tg_cfs_rq(tg, cpu_of(rq));
+
+ return_cfs_rq_runtime(cfs_rq);
+
+ if (cfs_rq->runtime_enabled && cfs_rq->runtime_remaining < 0 &&
+ !cfs_rq->throttled && READ_ONCE(cfs_b->runtime)) {
+ scoped_guard (raw_spinlock, &cfs_b->lock)
+ __assign_cfs_rq_runtime(cfs_b, cfs_rq, 0);
+ }
+ }
+}
+
+/*
+ * The runtime @p's groups have left for it on its CPU, without locks: the
+ * least, over the groups with a quota, of what the CPU holds or owes and
+ * cfs_b's runtime. Debt counts up to a quota less 1ns, so that a full period
+ * lets @p run and pay the rest. A group whose period timer is stopped does
+ * not count, as the next charge restarts the timer and refills what it owes.
+ * RUNTIME_INF if no group counts.
+ */
+u64 cfs_bandwidth_task_runtime(struct task_struct *p)
+{
+ u64 runtime = RUNTIME_INF;
+ struct task_group *tg;
+ int cpu = task_cpu(p);
+
+ if (!cfs_bandwidth_used())
+ return runtime;
+
+ guard(rcu)();
+
+ for (tg = task_group(p); tg; tg = tg->parent) {
+ struct cfs_bandwidth *cfs_b = tg_cfs_bandwidth(tg);
+ struct cfs_rq *cfs_rq = tg_cfs_rq(tg, cpu);
+ s64 held;
+
+ if (!READ_ONCE(cfs_rq->runtime_enabled) ||
+ !READ_ONCE(cfs_b->period_active))
+ continue;
+
+ held = max_t(s64, READ_ONCE(cfs_rq->runtime_remaining),
+ 1 - (s64)READ_ONCE(cfs_b->quota));
+ held += READ_ONCE(cfs_b->runtime);
+ runtime = min_t(u64, runtime, max_t(s64, held, 0));
+ }
+
+ return runtime;
+}
+
#ifdef CONFIG_NO_HZ_FULL
/* called from pick_next_task_fair() */
static void sched_fair_update_stop_tick(struct rq *rq, struct task_struct *p)
diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h
index 676e6371d0265..fd98000ffd744 100644
--- a/kernel/sched/sched.h
+++ b/kernel/sched/sched.h
@@ -456,6 +456,7 @@ struct cfs_bandwidth {
u64 runtime;
u64 burst;
u64 runtime_snap;
+ u64 debt;
s64 hierarchical_quota;
u8 idle;
@@ -621,6 +622,17 @@ static inline bool cfs_task_bw_constrained(struct task_struct *p) { return false
#endif /* !CONFIG_CGROUP_SCHED */
+#ifdef CONFIG_CFS_BANDWIDTH
+void cfs_bandwidth_charge_task(struct rq *rq, struct task_struct *p, u64 delta_exec);
+void cfs_bandwidth_return_task(struct rq *rq, struct task_struct *p);
+u64 cfs_bandwidth_task_runtime(struct task_struct *p);
+#else
+static inline void cfs_bandwidth_charge_task(struct rq *rq, struct task_struct *p,
+ u64 delta_exec) { }
+static inline void cfs_bandwidth_return_task(struct rq *rq, struct task_struct *p) { }
+static inline u64 cfs_bandwidth_task_runtime(struct task_struct *p) { return RUNTIME_INF; }
+#endif
+
/*
* A weight of 0 or 1 can cause arithmetics problems.
* A weight of a cfs_rq is the sum of weights of which entities
diff --git a/tools/sched_ext/include/scx/common.bpf.h b/tools/sched_ext/include/scx/common.bpf.h
index 6194484d4d9a9..cfb7f1b70665f 100644
--- a/tools/sched_ext/include/scx/common.bpf.h
+++ b/tools/sched_ext/include/scx/common.bpf.h
@@ -98,6 +98,7 @@ s32 scx_bpf_pick_any_cpu_node(const cpumask_t *cpus_allowed, int node, u64 flags
s32 scx_bpf_pick_any_cpu(const cpumask_t *cpus_allowed, u64 flags) __ksym;
bool scx_bpf_task_running(const struct task_struct *p) __ksym;
s32 scx_bpf_task_cpu(const struct task_struct *p) __ksym;
+u64 scx_bpf_task_cpu_max_runtime(struct task_struct *p) __ksym __weak;
struct rq *scx_bpf_cpu_rq(s32 cpu) __ksym __weak;
struct rq *scx_bpf_locked_rq(void) __ksym;
struct task_struct *scx_bpf_cpu_curr(s32 cpu) __ksym __weak;
diff --git a/tools/sched_ext/include/scx/compat.h b/tools/sched_ext/include/scx/compat.h
index 07fb55e63c57c..60b9972f1a5dc 100644
--- a/tools/sched_ext/include/scx/compat.h
+++ b/tools/sched_ext/include/scx/compat.h
@@ -215,6 +215,7 @@ static inline bool __COMPAT_struct_has_field(const char *type, const char *field
#define SCX_OPS_BUILTIN_IDLE_PER_NODE SCX_OPS_FLAG(SCX_OPS_BUILTIN_IDLE_PER_NODE)
#define SCX_OPS_ALWAYS_ENQ_IMMED SCX_OPS_FLAG(SCX_OPS_ALWAYS_ENQ_IMMED)
#define SCX_OPS_ENQ_BLOCKED SCX_OPS_FLAG(SCX_OPS_ENQ_BLOCKED)
+#define SCX_OPS_CHARGE_CPU_MAX SCX_OPS_FLAG(SCX_OPS_CHARGE_CPU_MAX)
#define SCX_PICK_IDLE_FLAG(name) __COMPAT_ENUM_OR_ZERO("scx_pick_idle_cpu_flags", #name)
diff --git a/tools/testing/selftests/sched_ext/Makefile b/tools/testing/selftests/sched_ext/Makefile
index c5d3a2eaea7da..4a727b8dc7aab 100644
--- a/tools/testing/selftests/sched_ext/Makefile
+++ b/tools/testing/selftests/sched_ext/Makefile
@@ -165,6 +165,7 @@ all_test_bpfprogs := $(foreach prog,$(wildcard *.bpf.c),$(INCLUDE_DIR)/$(patsubs
auto-test-targets := \
cgroup_nr_cpus \
+ cgroup_runtime \
create_dsq \
dequeue \
dequeue_iter \
diff --git a/tools/testing/selftests/sched_ext/cgroup_runtime.bpf.c b/tools/testing/selftests/sched_ext/cgroup_runtime.bpf.c
new file mode 100644
index 0000000000000..ec98a25d19c29
--- /dev/null
+++ b/tools/testing/selftests/sched_ext/cgroup_runtime.bpf.c
@@ -0,0 +1,195 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * A scheduler with SCX_OPS_CHARGE_CPU_MAX, for the cgroup_runtime tests. It
+ * runs a task only while scx_bpf_task_cpu_max_runtime() says it has runtime
+ * left, unless @ignore_runtime is set. A task that has none waits in HELD_DSQ,
+ * and a timer kicks the idle CPUs every millisecond, so that they take it
+ * once a period refills the runtime.
+ *
+ * partial_ops sets SCX_OPS_SWITCH_PARTIAL and full_ops does not. Both pass
+ * the cpu.max and cpu.max.runtime writes of @cgid on to the test.
+ */
+
+#include <scx/common.bpf.h>
+
+char _license[] SEC("license") = "GPL";
+
+#define HELD_DSQ 0
+#define KICK_NS 1000000
+
+UEI_DEFINE(uei);
+
+/* the cgroup under test, set before attaching */
+u64 cgid;
+
+/* run tasks whether or not they have runtime left */
+bool ignore_runtime;
+
+/* tasks held for lack of runtime */
+u64 nr_held;
+
+/* the cpu.max and cpu.max.runtime writes of @cgid */
+u64 nr_set_bw;
+u64 quota_us;
+u64 burst_us;
+u64 nr_set_runtime;
+/* cgroup_set_runtime() calls above the quota + burst last given */
+u64 nr_bad_order;
+
+struct kick_timer {
+ struct bpf_timer timer;
+};
+
+struct {
+ __uint(type, BPF_MAP_TYPE_ARRAY);
+ __uint(max_entries, 1);
+ __type(key, u32);
+ __type(value, struct kick_timer);
+} kick_timer SEC(".maps");
+
+static bool runnable(struct task_struct *p)
+{
+ return ignore_runtime || scx_bpf_task_cpu_max_runtime(p);
+}
+
+s32 BPF_STRUCT_OPS(cgroup_runtime_select_cpu, struct task_struct *p,
+ s32 prev_cpu, u64 wake_flags)
+{
+ return prev_cpu;
+}
+
+void BPF_STRUCT_OPS(cgroup_runtime_enqueue, struct task_struct *p,
+ u64 enq_flags)
+{
+ if (!runnable(p)) {
+ __sync_fetch_and_add(&nr_held, 1);
+ scx_bpf_dsq_insert(p, HELD_DSQ, SCX_SLICE_DFL, enq_flags);
+ return;
+ }
+ /* the last task of a CPU going idle must go to its local DSQ */
+ if (enq_flags & SCX_ENQ_LAST)
+ scx_bpf_dsq_insert(p, SCX_DSQ_LOCAL, SCX_SLICE_DFL, enq_flags);
+ else
+ scx_bpf_dsq_insert(p, SCX_DSQ_GLOBAL, SCX_SLICE_DFL, enq_flags);
+}
+
+void BPF_STRUCT_OPS(cgroup_runtime_dispatch, s32 cpu, struct task_struct *prev)
+{
+ struct task_struct *p = __COMPAT_scx_bpf_dsq_peek(HELD_DSQ);
+
+ if (p && runnable(p))
+ scx_bpf_dsq_move_to_local(HELD_DSQ, 0);
+}
+
+/*
+ * Stop a task that ran out, so that it goes through enqueue again.
+ * SCX_OPS_ENQ_LAST makes that so also when it is alone on its CPU.
+ */
+void BPF_STRUCT_OPS(cgroup_runtime_tick, struct task_struct *p)
+{
+ if (!runnable(p))
+ scx_bpf_task_set_slice(p, 0);
+}
+
+static int kick_timerfn(void *map, int *key, struct bpf_timer *timer)
+{
+ s32 cpu;
+
+ if (scx_bpf_dsq_nr_queued(HELD_DSQ))
+ bpf_for(cpu, 0, scx_bpf_nr_cpu_ids())
+ scx_bpf_kick_cpu(cpu, SCX_KICK_IDLE);
+
+ bpf_timer_start(timer, KICK_NS, 0);
+ return 0;
+}
+
+s32 BPF_STRUCT_OPS_SLEEPABLE(cgroup_runtime_init)
+{
+ struct bpf_timer *timer;
+ u32 key = 0;
+ s32 ret;
+
+ ret = scx_bpf_create_dsq(HELD_DSQ, -1);
+ if (ret)
+ return ret;
+
+ timer = bpf_map_lookup_elem(&kick_timer, &key);
+ if (!timer)
+ return -ESRCH;
+ bpf_timer_init(timer, &kick_timer, CLOCK_MONOTONIC);
+ bpf_timer_set_callback(timer, kick_timerfn);
+ return bpf_timer_start(timer, KICK_NS, 0);
+}
+
+static bool is_target(struct cgroup *cgrp)
+{
+ return cgid && cgrp->kn->id == cgid;
+}
+
+/* sleepable: cpu.max writes call it in process context */
+void BPF_STRUCT_OPS_SLEEPABLE(cgroup_runtime_cgroup_set_bandwidth,
+ struct cgroup *cgrp, u64 period_us, u64 quota,
+ u64 burst)
+{
+ if (!is_target(cgrp))
+ return;
+
+ __sync_fetch_and_add(&nr_set_bw, 1);
+ quota_us = quota;
+ burst_us = burst;
+}
+
+void BPF_STRUCT_OPS_SLEEPABLE(cgroup_runtime_cgroup_set_runtime,
+ struct cgroup *cgrp, u64 runtime_us)
+{
+ if (!is_target(cgrp))
+ return;
+
+ __sync_fetch_and_add(&nr_set_runtime, 1);
+ if (runtime_us > quota_us + burst_us)
+ __sync_fetch_and_add(&nr_bad_order, 1);
+}
+
+s32 BPF_STRUCT_OPS(cgroup_runtime_cgroup_init, struct cgroup *cgrp,
+ struct scx_cgroup_init_args *args)
+{
+ if (is_target(cgrp)) {
+ quota_us = args->bw_quota_us;
+ burst_us = args->bw_burst_us;
+ }
+ return 0;
+}
+
+void BPF_STRUCT_OPS(cgroup_runtime_exit, struct scx_exit_info *ei)
+{
+ UEI_RECORD(uei, ei);
+}
+
+SCX_OPS_DEFINE(partial_ops,
+ .select_cpu = (void *)cgroup_runtime_select_cpu,
+ .enqueue = (void *)cgroup_runtime_enqueue,
+ .dispatch = (void *)cgroup_runtime_dispatch,
+ .tick = (void *)cgroup_runtime_tick,
+ .init = (void *)cgroup_runtime_init,
+ .cgroup_init = (void *)cgroup_runtime_cgroup_init,
+ .cgroup_set_bandwidth = (void *)cgroup_runtime_cgroup_set_bandwidth,
+ .cgroup_set_runtime = (void *)cgroup_runtime_cgroup_set_runtime,
+ .exit = (void *)cgroup_runtime_exit,
+ .flags = SCX_OPS_SWITCH_PARTIAL |
+ SCX_OPS_ENQ_LAST |
+ SCX_OPS_CHARGE_CPU_MAX,
+ .name = "cgroup_runtime");
+
+SCX_OPS_DEFINE(full_ops,
+ .select_cpu = (void *)cgroup_runtime_select_cpu,
+ .enqueue = (void *)cgroup_runtime_enqueue,
+ .dispatch = (void *)cgroup_runtime_dispatch,
+ .tick = (void *)cgroup_runtime_tick,
+ .init = (void *)cgroup_runtime_init,
+ .cgroup_init = (void *)cgroup_runtime_cgroup_init,
+ .cgroup_set_bandwidth = (void *)cgroup_runtime_cgroup_set_bandwidth,
+ .cgroup_set_runtime = (void *)cgroup_runtime_cgroup_set_runtime,
+ .exit = (void *)cgroup_runtime_exit,
+ .flags = SCX_OPS_ENQ_LAST |
+ SCX_OPS_CHARGE_CPU_MAX,
+ .name = "cgroup_runtime_full");
diff --git a/tools/testing/selftests/sched_ext/cgroup_runtime.c b/tools/testing/selftests/sched_ext/cgroup_runtime.c
new file mode 100644
index 0000000000000..86d50a63a6e73
--- /dev/null
+++ b/tools/testing/selftests/sched_ext/cgroup_runtime.c
@@ -0,0 +1,652 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Verify that a BPF scheduler with SCX_OPS_CHARGE_CPU_MAX shares a cgroup's
+ * cpu.max quota with the fair class: what the cgroup's tasks run in both
+ * classes, with and without SCX_OPS_SWITCH_PARTIAL and while tasks switch
+ * class, stays within the quota; nothing is gained or lost when the
+ * scheduler loads and unloads; and a scheduler that ignores
+ * scx_bpf_task_cpu_max_runtime() leaves the fair tasks at most a quota of
+ * debt and a slice per CPU, while one that stops tasks late pays the overrun
+ * back.
+ * Also verify that cpu.max and cpu.max.runtime writes reach the scheduler
+ * in the order the fair class applied them.
+ *
+ * Usage is cpu.stat's usage_usec. A window of n periods may hold n + 1
+ * refills and start or end with up to a quota, and each CPU may hold up to
+ * a slice (sched_cfs_bandwidth_slice_us), so the checks allow that.
+ */
+
+#define _GNU_SOURCE
+#include <bpf/bpf.h>
+#include <errno.h>
+#include <fcntl.h>
+#include <limits.h>
+#include <linux/limits.h>
+#include <pthread.h>
+#include <sched.h>
+#include <scx/common.h>
+#include <signal.h>
+#include <stdarg.h>
+#include <stdbool.h>
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+#include <sys/wait.h>
+#include <time.h>
+#include <unistd.h>
+
+#include "cgroup_runtime.bpf.skel.h"
+#include "cgroup_util.h"
+#include "scx_test.h"
+
+#ifndef SCHED_EXT
+#define SCHED_EXT 7
+#endif
+
+#define PERIOD_US 100000LL
+#define QUOTA_US 20000LL
+#define NR_PERIODS 10
+#define NR_BURNERS 2
+
+struct ctx {
+ struct cgroup_runtime *skel;
+ struct bpf_link *link;
+ char root[PATH_MAX];
+ char parent[PATH_MAX];
+ char cg[PATH_MAX];
+ bool parent_created;
+ bool cg_created;
+ pid_t burner[NR_BURNERS];
+ long long slice_us;
+ long long saved_slice_us;
+};
+
+static int join_path(char *dst, size_t dst_size, const char *parent, const char *name)
+{
+ int ret;
+
+ ret = snprintf(dst, dst_size, "%s/%s", parent, name);
+ if (ret < 0 || (size_t)ret >= dst_size)
+ return -ENAMETOOLONG;
+ return 0;
+}
+
+static u64 now_us(void)
+{
+ struct timespec ts;
+
+ clock_gettime(CLOCK_MONOTONIC, &ts);
+ return ts.tv_sec * 1000000ULL + ts.tv_nsec / 1000;
+}
+
+static u64 cgroup_id(const char *path)
+{
+ union {
+ u64 id;
+ unsigned char bytes[8];
+ } id = {};
+ struct file_handle *handle;
+ int mount_id, ret;
+
+ handle = calloc(1, sizeof(*handle) + sizeof(id));
+ if (!handle)
+ return 0;
+ handle->handle_bytes = sizeof(id);
+ ret = name_to_handle_at(AT_FDCWD, path, handle, &mount_id, 0);
+ if (!ret && handle->handle_bytes == sizeof(id))
+ memcpy(id.bytes, handle->f_handle, sizeof(id));
+ free(handle);
+
+ return ret ? 0 : id.id;
+}
+
+static long long read_runtime(struct ctx *ctx)
+{
+ return cg_read_long(ctx->cg, "cpu.max.runtime");
+}
+
+static long long read_usage(struct ctx *ctx)
+{
+ return cg_read_key_long(ctx->cg, "cpu.stat", "usage_usec");
+}
+
+static int write_str(const char *cgroup, const char *file, const char *fmt, ...)
+{
+ char buf[64];
+ va_list ap;
+
+ va_start(ap, fmt);
+ vsnprintf(buf, sizeof(buf), fmt, ap);
+ va_end(ap);
+ return cg_write(cgroup, file, buf);
+}
+
+static bool controller_enabled(const char *cgroup, const char *file, const char *controller)
+{
+ char buf[4096], *saveptr, *token;
+
+ if (cg_read(cgroup, file, buf, sizeof(buf)))
+ return false;
+ for (token = strtok_r(buf, "\n ", &saveptr); token;
+ token = strtok_r(NULL, "\n ", &saveptr))
+ if (!strcmp(token, controller))
+ return true;
+ return false;
+}
+
+static int set_policy(pid_t pid, int policy)
+{
+ struct sched_param param = {};
+
+ return sched_setscheduler(pid, policy, ¶m);
+}
+
+static void stop_burners(struct ctx *ctx)
+{
+ int i;
+
+ for (i = 0; i < NR_BURNERS; i++) {
+ if (ctx->burner[i] <= 0)
+ continue;
+ kill(ctx->burner[i], SIGKILL);
+ waitpid(ctx->burner[i], NULL, 0);
+ ctx->burner[i] = 0;
+ }
+}
+
+/* a task in @ctx->cg that spins on CPU @i + 1 under @policy */
+static int start_burner(struct ctx *ctx, int i, int policy)
+{
+ cpu_set_t set;
+ pid_t pid;
+
+ pid = fork();
+ if (pid < 0)
+ return -errno;
+ if (!pid)
+ for (;;)
+ ;
+
+ ctx->burner[i] = pid;
+ CPU_ZERO(&set);
+ CPU_SET((i + 1) % libbpf_num_possible_cpus(), &set);
+ if (sched_setaffinity(pid, sizeof(set), &set) ||
+ cg_enter(ctx->cg, pid) || set_policy(pid, policy))
+ return -1;
+ return 0;
+}
+
+static void cleanup(void *arg)
+{
+ struct ctx *ctx = arg;
+
+ if (!ctx)
+ return;
+ stop_burners(ctx);
+ bpf_link__destroy(ctx->link);
+ cgroup_runtime__destroy(ctx->skel);
+ if (ctx->saved_slice_us)
+ write_str("/proc/sys/kernel", "sched_cfs_bandwidth_slice_us", "%lld",
+ ctx->saved_slice_us);
+ if (ctx->cg_created)
+ cg_destroy(ctx->cg);
+ if (ctx->parent_created)
+ cg_destroy(ctx->parent);
+ free(ctx);
+}
+
+/*
+ * Open the skeleton and make parent/cg under the cgroup2 root, with a quota
+ * of @quota us per @period us, a burst of @burst us and a runtime of
+ * @runtime us on cg.
+ */
+static enum scx_test_status setup_ctx(void **ctxp, long long quota,
+ long long period, long long burst,
+ long long runtime)
+{
+ struct ctx *ctx;
+ char name[64];
+
+ ctx = calloc(1, sizeof(*ctx));
+ if (!ctx)
+ return SCX_TEST_FAIL;
+ *ctxp = ctx;
+
+ /* SCX_ENUM_INIT() exits if vmlinux BTF can't be loaded: before cgroups */
+ ctx->skel = cgroup_runtime__open();
+ if (!ctx->skel) {
+ SCX_ERR("Failed to open skel");
+ return SCX_TEST_FAIL;
+ }
+ SCX_ENUM_INIT(ctx->skel);
+
+ ctx->slice_us = 5000;
+ if (!access("/proc/sys/kernel/sched_cfs_bandwidth_slice_us", R_OK))
+ ctx->slice_us = cg_read_long("/proc/sys/kernel",
+ "sched_cfs_bandwidth_slice_us");
+
+ if (cg_find_unified_root(ctx->root, sizeof(ctx->root), NULL))
+ return SCX_TEST_SKIP;
+ if (!controller_enabled(ctx->root, "cgroup.subtree_control", "cpu") &&
+ cg_write(ctx->root, "cgroup.subtree_control", "+cpu"))
+ return SCX_TEST_SKIP;
+
+ snprintf(name, sizeof(name), "scx_cgroup_runtime_%d", getpid());
+ if (join_path(ctx->parent, sizeof(ctx->parent), ctx->root, name) ||
+ join_path(ctx->cg, sizeof(ctx->cg), ctx->parent, "cg")) {
+ SCX_ERR("Cgroup path is too long");
+ return SCX_TEST_FAIL;
+ }
+
+ if (cg_create(ctx->parent)) {
+ SCX_ERR("Failed to create %s", ctx->parent);
+ return SCX_TEST_FAIL;
+ }
+ ctx->parent_created = true;
+ if (cg_write(ctx->parent, "cgroup.subtree_control", "+cpu")) {
+ SCX_ERR("Failed to enable cpu in %s", ctx->parent);
+ return SCX_TEST_FAIL;
+ }
+ if (cg_create(ctx->cg)) {
+ SCX_ERR("Failed to create %s", ctx->cg);
+ return SCX_TEST_FAIL;
+ }
+ ctx->cg_created = true;
+
+ if (access(cg_control(ctx->cg, "cpu.max.runtime"), F_OK))
+ return SCX_TEST_SKIP;
+
+ if (write_str(ctx->cg, "cpu.max", "%lld %lld", quota, period) ||
+ write_str(ctx->cg, "cpu.max.burst", "%lld", burst) ||
+ write_str(ctx->cg, "cpu.max.runtime", "%lld", runtime)) {
+ SCX_ERR("Failed to configure %s", ctx->cg);
+ return SCX_TEST_FAIL;
+ }
+
+ ctx->skel->bss->cgid = cgroup_id(ctx->cg);
+ if (!ctx->skel->bss->cgid) {
+ SCX_ERR("Failed to read the cgroup ID of %s", ctx->cg);
+ return SCX_TEST_FAIL;
+ }
+ if (libbpf_num_possible_cpus() < NR_BURNERS + 1)
+ return SCX_TEST_SKIP;
+ return SCX_TEST_PASS;
+}
+
+static enum scx_test_status setup_rate(void **ctxp)
+{
+ return setup_ctx(ctxp, QUOTA_US, PERIOD_US, 0, QUOTA_US);
+}
+
+static enum scx_test_status attach_ops(struct ctx *ctx, bool partial)
+{
+ ctx->link = bpf_map__attach_struct_ops(partial ? ctx->skel->maps.partial_ops :
+ ctx->skel->maps.full_ops);
+ SCX_FAIL_IF(!ctx->link, "Failed to attach scheduler");
+ return SCX_TEST_PASS;
+}
+
+static enum scx_test_status load_ops(struct ctx *ctx, bool partial)
+{
+ bpf_map__set_autoattach(ctx->skel->maps.partial_ops, false);
+ bpf_map__set_autoattach(ctx->skel->maps.full_ops, false);
+ SCX_FAIL_IF(cgroup_runtime__load(ctx->skel), "Failed to load skel");
+ return attach_ops(ctx, partial);
+}
+
+static void detach(struct ctx *ctx)
+{
+ bpf_link__destroy(ctx->link);
+ ctx->link = NULL;
+}
+
+/*
+ * Run two burners in cg under @policy[], the first switching class every
+ * 7 ms if @toggle, and check that cg runs its quota per period, no more and
+ * not much less.
+ */
+static enum scx_test_status run_rate(struct ctx *ctx, bool partial,
+ const int *policy, bool toggle)
+{
+ long long tol = NR_BURNERS * ctx->slice_us + QUOTA_US;
+ long long used, u0;
+ bool ext = false;
+ u64 end;
+ int i;
+
+ if (load_ops(ctx, partial))
+ return SCX_TEST_FAIL;
+ for (i = 0; i < NR_BURNERS; i++)
+ SCX_EQ(start_burner(ctx, i, policy[i]), 0);
+
+ usleep(3 * PERIOD_US);
+ u0 = read_usage(ctx);
+ end = now_us() + NR_PERIODS * PERIOD_US;
+ while (now_us() < end) {
+ if (toggle) {
+ ext = !ext;
+ SCX_EQ(set_policy(ctx->burner[0], ext ? SCHED_EXT : SCHED_OTHER), 0);
+ }
+ usleep(7000);
+ }
+ used = read_usage(ctx) - u0;
+ stop_burners(ctx);
+
+ printf("ran %lld us in %d periods of quota %lld us, %lu held\n",
+ used, NR_PERIODS, QUOTA_US, (unsigned long)ctx->skel->bss->nr_held);
+ SCX_GE(used, QUOTA_US * NR_PERIODS - tol);
+ SCX_LE(used, QUOTA_US * NR_PERIODS + tol);
+ SCX_ASSERT_ALIVE(ctx->skel, uei);
+ return SCX_TEST_PASS;
+}
+
+static enum scx_test_status run_full(void *arg)
+{
+ static const int policy[] = { SCHED_OTHER, SCHED_OTHER };
+
+ return run_rate(arg, false, policy, false);
+}
+
+static enum scx_test_status run_partial_ext(void *arg)
+{
+ static const int policy[] = { SCHED_EXT, SCHED_EXT };
+
+ return run_rate(arg, true, policy, false);
+}
+
+static enum scx_test_status run_partial_fair(void *arg)
+{
+ static const int policy[] = { SCHED_OTHER, SCHED_OTHER };
+
+ return run_rate(arg, true, policy, false);
+}
+
+static enum scx_test_status run_partial_mixed(void *arg)
+{
+ static const int policy[] = { SCHED_EXT, SCHED_OTHER };
+
+ return run_rate(arg, true, policy, false);
+}
+
+static enum scx_test_status run_switch(void *arg)
+{
+ static const int policy[] = { SCHED_OTHER, SCHED_EXT };
+
+ return run_rate(arg, true, policy, true);
+}
+
+/*
+ * With a quota of 1 ms per second and a runtime of 3 s, what cg runs while
+ * the scheduler loads, runs and unloads is what its runtime loses, up to a
+ * slice per CPU and the few refills.
+ */
+#define LU_QUOTA_US 1000LL
+#define LU_PERIOD_US 1000000LL
+
+static enum scx_test_status setup_load_unload(void **ctxp)
+{
+ return setup_ctx(ctxp, LU_QUOTA_US, LU_PERIOD_US, 4000000, 3000000);
+}
+
+static enum scx_test_status run_load_unload(void *arg)
+{
+ struct ctx *ctx = arg;
+ long long r0, u0, drawn, used, refills, tol;
+ u64 t0, t1;
+ int i;
+
+ bpf_map__set_autoattach(ctx->skel->maps.partial_ops, false);
+ bpf_map__set_autoattach(ctx->skel->maps.full_ops, false);
+ SCX_FAIL_IF(cgroup_runtime__load(ctx->skel), "Failed to load skel");
+
+ t0 = now_us();
+ r0 = read_runtime(ctx);
+ u0 = read_usage(ctx);
+ for (i = 0; i < NR_BURNERS; i++)
+ SCX_EQ(start_burner(ctx, i, SCHED_OTHER), 0);
+ usleep(100000);
+ if (attach_ops(ctx, false))
+ return SCX_TEST_FAIL;
+ usleep(200000);
+ detach(ctx);
+ usleep(100000);
+ stop_burners(ctx);
+ drawn = r0 - read_runtime(ctx);
+ used = read_usage(ctx) - u0;
+ t1 = now_us();
+
+ refills = LU_QUOTA_US * ((t1 - t0) / LU_PERIOD_US + 2);
+ tol = NR_BURNERS * ctx->slice_us;
+ printf("ran %lld us, runtime down %lld us\n", used, drawn);
+ SCX_GE(used, 300000);
+ SCX_GE(used, drawn - tol);
+ SCX_LE(used, drawn + refills + tol);
+ return SCX_TEST_PASS;
+}
+
+/*
+ * A scheduler that ignores scx_bpf_task_cpu_max_runtime() runs a task far
+ * past the quota. Once the task is back on the fair class, it runs its quota
+ * again, less at most a quota and a slice of debt.
+ */
+static enum scx_test_status run_clamp(void *arg)
+{
+ struct ctx *ctx = arg;
+ long long u0, ignored, used;
+
+ ctx->skel->bss->ignore_runtime = true;
+ if (load_ops(ctx, true))
+ return SCX_TEST_FAIL;
+ SCX_EQ(start_burner(ctx, 0, SCHED_EXT), 0);
+
+ u0 = read_usage(ctx);
+ usleep(5 * PERIOD_US);
+ ignored = read_usage(ctx) - u0;
+
+ SCX_EQ(set_policy(ctx->burner[0], SCHED_OTHER), 0);
+ u0 = read_usage(ctx);
+ usleep(5 * PERIOD_US);
+ used = read_usage(ctx) - u0;
+ stop_burners(ctx);
+
+ printf("ran %lld us ignoring the quota, then %lld us on the fair class\n",
+ ignored, used);
+ SCX_GE(ignored, 4 * PERIOD_US);
+ SCX_GE(used, 4 * QUOTA_US - 2 * ctx->slice_us);
+ SCX_LE(used, 6 * QUOTA_US + ctx->slice_us);
+ SCX_ASSERT_ALIVE(ctx->skel, uei);
+ return SCX_TEST_PASS;
+}
+
+/*
+ * The scheduler stops a task only at a tick, so each period it overruns a
+ * quota of 2 ms by up to a tick. With a slice of 1 ms, the next tick's
+ * charge takes the CPU past the -slice floor. The debt must still be paid
+ * from the next periods, so over 100 periods cg runs its quota, with the
+ * second burner moving between CPUs 2 and 3 every 7 ms.
+ */
+#define DEBT_QUOTA_US 2000LL
+#define DEBT_PERIOD_US 10000LL
+#define DEBT_PERIODS 100
+
+static enum scx_test_status setup_debt(void **ctxp)
+{
+ enum scx_test_status ret;
+ struct ctx *ctx;
+
+ ret = setup_ctx(ctxp, DEBT_QUOTA_US, DEBT_PERIOD_US, 0, DEBT_QUOTA_US);
+ if (ret != SCX_TEST_PASS)
+ return ret;
+ ctx = *ctxp;
+ if (write_str("/proc/sys/kernel", "sched_cfs_bandwidth_slice_us", "1000"))
+ return SCX_TEST_SKIP;
+ ctx->saved_slice_us = ctx->slice_us;
+ ctx->slice_us = 1000;
+ return SCX_TEST_PASS;
+}
+
+static enum scx_test_status run_debt(void *arg)
+{
+ struct ctx *ctx = arg;
+ long long tol = (NR_BURNERS + 1) * ctx->slice_us + DEBT_QUOTA_US;
+ long long want = DEBT_QUOTA_US * DEBT_PERIODS;
+ long long used, u0;
+ bool moved = false;
+ cpu_set_t set;
+ u64 end;
+ int i;
+
+ if (libbpf_num_possible_cpus() < 4)
+ return SCX_TEST_SKIP;
+ if (load_ops(ctx, true))
+ return SCX_TEST_FAIL;
+ for (i = 0; i < NR_BURNERS; i++)
+ SCX_EQ(start_burner(ctx, i, SCHED_EXT), 0);
+
+ usleep(3 * DEBT_PERIOD_US);
+ u0 = read_usage(ctx);
+ end = now_us() + DEBT_PERIODS * DEBT_PERIOD_US;
+ while (now_us() < end) {
+ moved = !moved;
+ CPU_ZERO(&set);
+ CPU_SET(moved ? 3 : 2, &set);
+ SCX_EQ(sched_setaffinity(ctx->burner[1], sizeof(set), &set), 0);
+ usleep(7000);
+ }
+ used = read_usage(ctx) - u0;
+ stop_burners(ctx);
+
+ printf("ran %lld us in %d periods of quota %lld us\n", used,
+ DEBT_PERIODS, DEBT_QUOTA_US);
+ SCX_GE(used, want - tol);
+ SCX_LE(used, want + tol);
+ SCX_ASSERT_ALIVE(ctx->skel, uei);
+ return SCX_TEST_PASS;
+}
+
+/*
+ * One thread alternates cpu.max between quotas of 100ms and 10ms, without a
+ * burst, while another writes a runtime of 90ms, which only the first
+ * allows. The scheduler must never get a runtime above the quota it was last
+ * given, and must end with the quota cpu.max has.
+ */
+#define INTERLEAVE_US 2000000ULL
+
+struct interleave {
+ struct ctx *ctx;
+ long nr_ok;
+ long nr_einval;
+ long nr_err;
+};
+
+static void *interleave_quota(void *arg)
+{
+ struct interleave *il = arg;
+ u64 end = now_us() + INTERLEAVE_US;
+ bool high = false;
+
+ while (now_us() < end) {
+ if (write_str(il->ctx->cg, "cpu.max", "%lld %lld",
+ high ? 100000LL : 10000LL, 1000000LL))
+ il->nr_err++;
+ else
+ il->nr_ok++;
+ high = !high;
+ }
+ return NULL;
+}
+
+static void *interleave_runtime(void *arg)
+{
+ struct interleave *il = arg;
+ u64 end = now_us() + INTERLEAVE_US;
+
+ while (now_us() < end) {
+ int fd = cg_open(il->ctx->cg, "cpu.max.runtime", O_WRONLY);
+
+ if (fd < 0) {
+ il->nr_err++;
+ continue;
+ }
+ if (write(fd, "90000", 5) == 5)
+ il->nr_ok++;
+ else if (errno == EINVAL)
+ il->nr_einval++;
+ else
+ il->nr_err++;
+ close(fd);
+ }
+ return NULL;
+}
+
+static enum scx_test_status setup_interleave(void **ctxp)
+{
+ return setup_ctx(ctxp, 100000, 1000000, 0, 100000);
+}
+
+static enum scx_test_status run_interleave(void *arg)
+{
+ struct ctx *ctx = arg;
+ struct interleave quota = { .ctx = ctx }, runtime = { .ctx = ctx };
+ pthread_t tq, tr;
+ char buf[64];
+
+ if (load_ops(ctx, true))
+ return SCX_TEST_FAIL;
+ SCX_EQ(ctx->skel->bss->quota_us, 100000);
+
+ SCX_EQ(pthread_create(&tq, NULL, interleave_quota, "a), 0);
+ SCX_EQ(pthread_create(&tr, NULL, interleave_runtime, &runtime), 0);
+ pthread_join(tq, NULL);
+ pthread_join(tr, NULL);
+
+ SCX_EQ(quota.nr_err, 0);
+ SCX_EQ(runtime.nr_err, 0);
+ SCX_GT(runtime.nr_ok, 0);
+ SCX_EQ(ctx->skel->bss->nr_bad_order, 0);
+ SCX_EQ(ctx->skel->bss->nr_set_bw, quota.nr_ok);
+ SCX_EQ(ctx->skel->bss->nr_set_runtime, runtime.nr_ok);
+
+ SCX_EQ(cg_read(ctx->cg, "cpu.max", buf, sizeof(buf)), 0);
+ SCX_EQ(ctx->skel->bss->quota_us, atoll(buf));
+ SCX_EQ(ctx->skel->bss->burst_us, 0);
+
+ printf("cpu.max writes %ld, cpu.max.runtime writes %ld (%ld EINVAL)\n",
+ quota.nr_ok, runtime.nr_ok, runtime.nr_einval);
+ SCX_ASSERT_ALIVE(ctx->skel, uei);
+ return SCX_TEST_PASS;
+}
+
+#define RUNTIME_TEST(__name, __setup, __desc) \
+ struct scx_test cgroup_runtime_##__name = { \
+ .name = "cgroup_runtime_" #__name, \
+ .description = __desc, \
+ .setup = __setup, \
+ .run = run_##__name, \
+ .cleanup = cleanup, \
+ }
+
+RUNTIME_TEST(full, setup_rate, "Without partial switch, the BPF scheduler's tasks run their quota");
+RUNTIME_TEST(partial_ext, setup_rate, "SCHED_EXT tasks of a cgroup run its quota");
+RUNTIME_TEST(partial_fair, setup_rate, "Fair tasks of a cgroup run its quota");
+RUNTIME_TEST(partial_mixed, setup_rate, "Fair and SCHED_EXT tasks of a cgroup share its quota");
+RUNTIME_TEST(switch, setup_rate, "A task switching class every 7 ms stays within the quota");
+RUNTIME_TEST(load_unload, setup_load_unload, "Load and unload neither gain nor lose runtime");
+RUNTIME_TEST(clamp, setup_rate, "A scheduler ignoring the quota leaves a bounded debt");
+RUNTIME_TEST(debt, setup_debt, "Debt past a quota is paid in the next periods");
+RUNTIME_TEST(interleave, setup_interleave, "cpu.max writes reach the scheduler in order");
+
+/* REGISTER_SCX_TEST() can be used once per file */
+__attribute__((constructor))
+static void register_cgroup_runtime_tests(void)
+{
+ scx_test_register(&cgroup_runtime_full);
+ scx_test_register(&cgroup_runtime_partial_ext);
+ scx_test_register(&cgroup_runtime_partial_fair);
+ scx_test_register(&cgroup_runtime_partial_mixed);
+ scx_test_register(&cgroup_runtime_switch);
+ scx_test_register(&cgroup_runtime_load_unload);
+ scx_test_register(&cgroup_runtime_clamp);
+ scx_test_register(&cgroup_runtime_debt);
+ scx_test_register(&cgroup_runtime_interleave);
+}
diff --git a/tools/testing/selftests/sched_ext/config b/tools/testing/selftests/sched_ext/config
index 8173f9170ffe8..495b140403a18 100644
--- a/tools/testing/selftests/sched_ext/config
+++ b/tools/testing/selftests/sched_ext/config
@@ -9,3 +9,4 @@ CONFIG_DEBUG_INFO=y
CONFIG_DEBUG_INFO_BTF=y
CONFIG_EXPERT=y
CONFIG_SCHED_PROXY_EXEC=y
+CONFIG_CFS_BANDWIDTH=y
--
2.47.3
Amazon Development Centre (South Africa) (Proprietary) Limited
29 Gogosoa Street, Observatory, Cape Town, Western Cape, 7925, South Africa
Registration Number: 2004 / 034463 / 07
^ permalink raw reply related [flat|nested] 14+ messages in thread* Re: [PATCH v2 4/4] sched_ext: charge BPF scheduler tasks to cpu.max bandwidth
2026-11-10 11:08 ` [PATCH v2 4/4] sched_ext: charge BPF scheduler tasks to cpu.max bandwidth Fernand Sieber
@ 2026-11-11 8:28 ` bot+bpf-ci
0 siblings, 0 replies; 14+ messages in thread
From: bot+bpf-ci @ 2026-11-11 8:28 UTC (permalink / raw)
To: sieberf, mingo, peterz, juri.lelli, vincent.guittot, tj
Cc: bsegall, dietmar.eggemann, rostedt, mgorman, vschneid,
kprateek.nayak, hannes, mkoutny, void, arighi, changwoo, skhan,
corbet, rdunlap, cgroups, sched-ext, linux-doc, linux-kselftest,
bpf, linux-kernel, nh-open-source, hborghor, fmubeen, dwmw,
karataev, sieberf, ast, andrii, daniel, martin.lau, eddyz87,
yonghong.song, mason, ihor.solodrai
[-- Attachment #1: Type: text/plain, Size: 1222 bytes --]
> sched_ext: charge BPF scheduler tasks to cpu.max bandwidth
TL;DR: Nit only, in the selftest: join_path(), cgroup_id() and
controller_enabled() in cgroup_runtime.c duplicate helpers in
cgroup_nr_cpus.c; consider moving them into shared util.c/util.h.
> diff --git a/tools/testing/selftests/sched_ext/cgroup_runtime.c b/tools/testing/selftests/sched_ext/cgroup_runtime.c
> new file mode 100644
> --- /dev/null
> +++ b/tools/testing/selftests/sched_ext/cgroup_runtime.c
[ ... ]
> +static int join_path(char *dst, size_t dst_size, const char *parent, const char *name)
> +{
> + int ret;
> +
> + ret = snprintf(dst, dst_size, "%s/%s", parent, name);
> + if (ret < 0 || (size_t)ret >= dst_size)
> + return -ENAMETOOLONG;
> + return 0;
> +}
[ ... ]
this isn't a bug, but join_path(), cgroup_id() and controller_enabled() look
identical to the static helpers of the same names in cgroup_nr_cpus.c.
Could they move into the shared util.c/util.h so both cgroup tests use one
copy?
---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md
CI run summary: https://github.com/kernel-patches/bpf/actions/runs/42982707042
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [PATCH v2 0/4] sched/fair: cpu.max.runtime and idle refill for CPU credits
2026-11-10 11:08 [PATCH v2 0/4] sched/fair: cpu.max.runtime and idle refill for CPU credits Fernand Sieber
` (3 preceding siblings ...)
2026-11-10 11:08 ` [PATCH v2 4/4] sched_ext: charge BPF scheduler tasks to cpu.max bandwidth Fernand Sieber
@ 2026-11-10 20:10 ` Tejun Heo
2026-11-11 12:20 ` Fernand Sieber
2026-11-11 12:29 ` [PATCH v3 0/2] sched/fair: cpu.max.runtime and idle refill for a burstable budget Fernand Sieber
5 siblings, 1 reply; 14+ messages in thread
From: Tejun Heo @ 2026-11-10 20:10 UTC (permalink / raw)
To: Fernand Sieber
Cc: mingo, peterz, juri.lelli, vincent.guittot, bsegall,
dietmar.eggemann, rostedt, mgorman, vschneid, kprateek.nayak,
hannes, mkoutny, void, arighi, changwoo, skhan, corbet, rdunlap,
cgroups, sched-ext, linux-doc, linux-kselftest, bpf, linux-kernel,
nh-open-source, hborghor, fmubeen, dwmw, karataev
Hello, Fernand.
On Tue, Nov 10, 2026 at 01:08:47PM +0200, Fernand Sieber wrote:
> This series adds cpu.max.runtime, a cgroup v2 file to read and set the
> runtime the CFS bandwidth controller has for a cgroup, lets
> cpu.max.burst exceed the quota, and makes an idle group keep earning
> quota up to quota + burst. Patch 3 passes the writes on to BPF
> schedulers, and patch 4 lets them charge their tasks to the same
> runtime.
I think this patchset is too maximalist. As described, the use case is
specific to selling burstable CPU budget. Nobody else has asked for
this. There may be other use cases which haven't been stated yet but the
description doesn't surface any, so as it stands this is pretty niche.
On top of that, the series integrates into both fair and sched_ext and
then makes the two work together so that the handoff is seamless. That's
too much for something which doesn't have a strong use case or an
established generic need. As with scx_bpf_cgroup_report_throttled(),
patch 4 modifies fair's internal state from sched_ext, which I don't
think is a good idea as the cgroup piping is split at the interface
level:
https://lore.kernel.org/all/ahdFAZVnb3DTv8Jo@slm.duckdns.org/
Seamlessly switching tasks between the classes in partial mode has a lot
of other integration problems beyond this one and at least I'm not
aiming to achieve it. It'd cost a lot more than it'd be useful.
The sched_ext part can just be a feature of your own BPF scheduler,
which you seem to be using already. There, the credit level is just
state in your BPF maps which your userspace can read and set, so the
maximalist version can wait until the need is proven. If you want a
handover between your scheduler and fair, that can be done from
userspace.
If a burstable budget is something useful to add to fair, can you do
that first and separately? It should be able to stand on its own.
Thanks.
--
tejun
^ permalink raw reply [flat|nested] 14+ messages in thread* Re: [PATCH v2 0/4] sched/fair: cpu.max.runtime and idle refill for CPU credits
2026-11-10 20:10 ` [PATCH v2 0/4] sched/fair: cpu.max.runtime and idle refill for CPU credits Tejun Heo
@ 2026-11-11 12:20 ` Fernand Sieber
2026-11-11 17:26 ` Tejun Heo
0 siblings, 1 reply; 14+ messages in thread
From: Fernand Sieber @ 2026-11-11 12:20 UTC (permalink / raw)
To: tj
Cc: mingo, peterz, juri.lelli, vincent.guittot, bsegall,
dietmar.eggemann, rostedt, mgorman, vschneid, kprateek.nayak,
hannes, mkoutny, void, arighi, changwoo, skhan, corbet, rdunlap,
cgroups, sched-ext, linux-doc, linux-kselftest, bpf, linux-kernel,
nh-open-source, hborghor, fmubeen, dwmw, karataev, sieberf
Hi Tejun,
Thank you for your prompt feedback.
On Tue, 10 Nov 2026 10:10:41 -1000, Tejun Heo wrote:
> If a burstable budget is something useful to add to fair, can you do
> that first and separately? It should be able to stand on its own.
That works for me as a starting point. I will resend just patches 1
and 2 as v3.
> Seamlessly switching tasks between the classes in partial mode has a lot
> of other integration problems beyond this one
> [...]
> The sched_ext part can just be a feature of your own BPF scheduler,
> which you seem to be using already. There, the credit level is just
> state in your BPF maps which your userspace can read and set, so the
> maximalist version can wait until the need is proven. If you want a
> handover between your scheduler and fair, that can be done from
> userspace.
The minimal viable bar for us is to never lose track of CPU credits.
With patches 1 and 2, the userspace handover you suggest covers the
planned transitions: on roll out and roll back, userspace reads
cpu.max.runtime before the switch and writes it back after. The one
case it cannot cover is an unplanned exit, a BPF scheduler erroring out
or killed by the watchdog and falling back to fair, where there is no
window for userspace to hand the level back.
Thanks,
Fernand
Amazon Development Centre (South Africa) (Proprietary) Limited
29 Gogosoa Street, Observatory, Cape Town, Western Cape, 7925, South Africa
Registration Number: 2004 / 034463 / 07
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [PATCH v2 0/4] sched/fair: cpu.max.runtime and idle refill for CPU credits
2026-11-11 12:20 ` Fernand Sieber
@ 2026-11-11 17:26 ` Tejun Heo
0 siblings, 0 replies; 14+ messages in thread
From: Tejun Heo @ 2026-11-11 17:26 UTC (permalink / raw)
To: Fernand Sieber
Cc: mingo, peterz, juri.lelli, vincent.guittot, bsegall,
dietmar.eggemann, rostedt, mgorman, vschneid, kprateek.nayak,
hannes, mkoutny, void, arighi, changwoo, skhan, corbet, rdunlap,
cgroups, sched-ext, linux-doc, linux-kselftest, bpf, linux-kernel,
nh-open-source, hborghor, fmubeen, dwmw, karataev
Hello, Fernand.
On Wed, Nov 11, 2026 at 02:20:39PM +0200, Fernand Sieber wrote:
> The minimal viable bar for us is to never lose track of CPU credits.
...
> The one case it cannot cover is an unplanned exit, a BPF scheduler
> erroring out or killed by the watchdog and falling back to fair, where
> there is no window for userspace to hand the level back.
Where you set your minimum viable bar isn't an argument for a design
decision. Stating that this is the bar doesn't make it so. It has to be
justifiable in a generic manner.
This is a crashing scheduler. The kernel is going to be scanning all the
CPUs and tasks to abort the in-flight scheduler while pushing all tasks to
temporary dumb FIFO scheduling to make forward progress. Accounting wise,
the credits may not be lost, but how meaningful is that across such a
transition? If a machine is doing this regularly to the point where
accurate accounting on that particular transition matters, credit
accounting is the least of its problems.
Thanks.
--
tejun
^ permalink raw reply [flat|nested] 14+ messages in thread
* [PATCH v3 0/2] sched/fair: cpu.max.runtime and idle refill for a burstable budget
2026-11-10 11:08 [PATCH v2 0/4] sched/fair: cpu.max.runtime and idle refill for CPU credits Fernand Sieber
` (4 preceding siblings ...)
2026-11-10 20:10 ` [PATCH v2 0/4] sched/fair: cpu.max.runtime and idle refill for CPU credits Tejun Heo
@ 2026-11-11 12:29 ` Fernand Sieber
2026-11-11 12:29 ` [PATCH v3 1/2] sched/fair: expose cpu.max.runtime to set bandwidth runtime directly Fernand Sieber
2026-11-11 12:29 ` [PATCH v3 2/2] sched/fair: refill bandwidth for the periods the timer slept through Fernand Sieber
5 siblings, 2 replies; 14+ messages in thread
From: Fernand Sieber @ 2026-11-11 12:29 UTC (permalink / raw)
To: mingo, peterz, juri.lelli, vincent.guittot
Cc: tj, bsegall, dietmar.eggemann, rostedt, mgorman, vschneid,
kprateek.nayak, hannes, mkoutny, skhan, corbet, rdunlap, cgroups,
linux-doc, linux-kselftest, linux-kernel, nh-open-source,
hborghor, fmubeen, dwmw, sieberf
This series makes the CFS bandwidth controller's bucket a burstable CPU
budget that userspace can save and restore. It lets cpu.max.burst exceed
the quota, so the bucket can hold more than one quota, and adds
cpu.max.runtime to read and set what it holds (patch 1). It makes a
group that is not running keep earning quota up to quota + burst
(patch 2).
v2 had two further patches that passed the level on to a BPF scheduler.
Tejun asked for the fair side first and on its own, so they are dropped
here and will be proposed separately if this is accepted.
Use case
========
We run virtual machines whose CPU time is governed by CPU credits, as
burstable instances are. An instance earns credits while it runs below
its baseline, spends them above it, and holds at most a capped amount;
with none left it is held to the baseline. The starting credits, the
baseline and the cap are part of the instance type's guarantee and must
hold whatever the host does to the instance.
Each instance's vCPU threads run in their own cgroup, and the controller
already is that bucket of credits: the quota per period is the rate
credits are earned at, quota + burst is the cap, and the runtime is the
level. It does the per-CPU charging and the throttling in the scheduler,
at a granularity userspace accounting cannot match. What is missing is a
cap above one quota, a way to read and write the level, and earning
while idle. An instance that idles for a minute with a 100ms period
earns one quota instead of up to 600.
The level has to be saved and restored by the host's control software:
an instance starts with the credits its type grants, not with one quota;
a host kernel live update (kexec) or a reboot creates the cgroups anew,
so every level has to be read before and written back after, or every
instance loses or gains credits; a live migration moves the level with
the instance; and the control plane raises or lowers it when credits
move between instances. A container checkpoint and restore has the same
problem.
Ben pointed out that a cpu.max or cpu.max.burst write already adds a
quota to the runtime, and that cpuacct gives the usage. That falls
short. A write can only add, and only one quota, so setting a level
means writing a temporary quota equal to the difference and then the
real quota back, and lowering one means shrinking the burst until the
cap cuts the runtime and widening it again; each leaves the cgroup
running with the wrong bandwidth in between, and each write visits every
CPU's cfs_rq. And usage is not the level: deriving it means replaying
every period's refill in userspace, including the periods the timer
slept through, which duplicates the controller's accounting.
Patch 2 changes behavior for existing cpu.max.burst users. A group with
a burst that becomes runnable after idling has up to quota + burst, as
if it had been running below its quota throughout, instead of the
runtime it had when the timer stopped. Groups without a burst see no
change.
Testing
=======
On tip/sched/core in qemu with lockdep, the cgroup test_cpu selftests
pass, 18 of 18, including the nine new ones for these two patches, with
no lockdep report in dmesg. Both patches build with W=1 and no warning
in kernel/sched, with and without CONFIG_CFS_BANDWIDTH and on 32-bit,
and kernel-doc reports nothing on the files they touch.
An AI coding assistant (Claude Code) helped write the selftests and
changelogs and ran the build and test matrix; I reviewed all of it.
Changes in v3
=============
- Patches 3 and 4, the sched_ext side, dropped (Tejun).
- Patch 1: cpu.max.runtime reads 0 while cpu.max is "max" instead of a
level the group no longer has, and taking a group's quota away drops
its runtime instead of keeping it for the next quota (kernel-patches
CI); the lockdep_assert_irqs_disabled() in distribute_cfs_runtime()
is gone, as both bandwidth timers run with hardirqs enabled on
PREEMPT_RT, and a comment states the real precondition; the
burst_kept selftest grants the runtime after a period has started,
so a hog cannot burn it while the test waits; CONFIG_CFS_BANDWIDTH
added to tools/testing/selftests/cgroup/config.
- Patch 2: a cpu.max write starts the next period from the write, so
the first refill under a new period is one new period away and not up
to one old period away; selftest for it.
v2: https://lore.kernel.org/all/cover.1794303670.git.sieberf@amazon.com/
v1: https://lore.kernel.org/all/20260525193622.70282-1-sieberf@amazon.com/
Fernand Sieber (2):
sched/fair: expose cpu.max.runtime to set bandwidth runtime directly
sched/fair: refill bandwidth for the periods the timer slept through
Documentation/admin-guide/cgroup-v2.rst | 24 +-
Documentation/scheduler/sched-bwc.rst | 14 +-
kernel/sched/core.c | 56 ++-
kernel/sched/fair.c | 163 +++++-
kernel/sched/sched.h | 4 +
tools/testing/selftests/cgroup/config | 1 +
tools/testing/selftests/cgroup/test_cpu.c | 587 ++++++++++++++++++++++
7 files changed, 839 insertions(+), 10 deletions(-)
base-commit: d7e670601af6c564ccf6d3b2ddf70ee4f41a6d1c
--
2.47.3
Amazon Development Centre (South Africa) (Proprietary) Limited
29 Gogosoa Street, Observatory, Cape Town, Western Cape, 7925, South Africa
Registration Number: 2004 / 034463 / 07
^ permalink raw reply [flat|nested] 14+ messages in thread* [PATCH v3 1/2] sched/fair: expose cpu.max.runtime to set bandwidth runtime directly
2026-11-11 12:29 ` [PATCH v3 0/2] sched/fair: cpu.max.runtime and idle refill for a burstable budget Fernand Sieber
@ 2026-11-11 12:29 ` Fernand Sieber
2026-11-11 12:29 ` [PATCH v3 2/2] sched/fair: refill bandwidth for the periods the timer slept through Fernand Sieber
1 sibling, 0 replies; 14+ messages in thread
From: Fernand Sieber @ 2026-11-11 12:29 UTC (permalink / raw)
To: mingo, peterz, juri.lelli, vincent.guittot
Cc: tj, bsegall, dietmar.eggemann, rostedt, mgorman, vschneid,
kprateek.nayak, hannes, mkoutny, skhan, corbet, rdunlap, cgroups,
linux-doc, linux-kselftest, linux-kernel, nh-open-source,
hborghor, fmubeen, dwmw, sieberf
Add cpu.max.runtime, a cgroup v2 file that reads and sets the CFS
bandwidth runtime, cfs_b->runtime, in microseconds. Userspace can then
grant a cgroup runtime beyond its quota. Refills still add only the
quota, capped at quota + burst, so the excess is used up over time and
the group returns to its quota per period.
A write is rejected without a quota or above quota + burst, the most
the group can accumulate. A read is capped at quota + burst so it can
be written back; uncapped slack from __return_cfs_rq_runtime() can
exceed that until the next refill. Runtime the CPUs already took, at
most a slice each, is not included and a write leaves it in place.
Without a quota a read reports 0, and taking a group's quota away
drops the runtime it had, rather than keeping it for the next time
the group is given a quota.
A write distributes the new runtime to throttled cfs_rqs at once, as
the slack timer does, rather than waiting for the unsynchronized
period timer. It also shifts runtime_snap by the change, so cpu.stat
burst accounting neither counts the write nor hides what the group ran
since the last refill.
Also drop the burst <= quota limit and require only quota + burst <=
max_bw_runtime_us (about 203 days), the quota's own limit, so a group
can accumulate many periods of quota. This applies to cgroup v1
cpu.cfs_burst_us too, which shares tg_set_bandwidth().
The file exists with CONFIG_CFS_BANDWIDTH only.
Signed-off-by: Fernand Sieber <sieberf@amazon.com>
---
Documentation/admin-guide/cgroup-v2.rst | 23 +-
Documentation/scheduler/sched-bwc.rst | 13 +-
kernel/sched/core.c | 44 ++-
kernel/sched/fair.c | 65 ++++
kernel/sched/sched.h | 2 +
tools/testing/selftests/cgroup/config | 1 +
tools/testing/selftests/cgroup/test_cpu.c | 356 ++++++++++++++++++++++
7 files changed, 495 insertions(+), 9 deletions(-)
diff --git a/Documentation/admin-guide/cgroup-v2.rst b/Documentation/admin-guide/cgroup-v2.rst
index 9e3c67b02ae2e..eb802f8ba877f 100644
--- a/Documentation/admin-guide/cgroup-v2.rst
+++ b/Documentation/admin-guide/cgroup-v2.rst
@@ -1231,12 +1231,33 @@ will be referred to. All time durations are in microseconds.
A read-write single value file which exists on non-root
cgroups. The default is "0".
- The burst in the range [0, $MAX].
+ The burst. The cgroup can accumulate the runtime it does not
+ use, up to $MAX + cpu.max.burst. The burst may exceed $MAX, as
+ long as $MAX + cpu.max.burst is at most 2^44 - 1 (about 203
+ days).
This file affects only processes under the fair-class scheduler and a BPF
scheduler with the ``cgroup_set_bandwidth`` callback depending on what
the callback actually does.
+ cpu.max.runtime
+ A read-write single value file which exists on non-root
+ cgroups if the kernel is built with CONFIG_CFS_BANDWIDTH.
+
+ The runtime the cgroup has accumulated and not used yet, in
+ microseconds. Each period refills it by $MAX, up to $MAX +
+ cpu.max.burst, and a read returns at most $MAX + cpu.max.burst.
+ A write sets it, up to $MAX + cpu.max.burst, and lets a cgroup
+ that ran out of runtime run again at once. A write fails with
+ EINVAL while $MAX is "max", a read then returns 0, and setting
+ $MAX to "max" drops the runtime the cgroup had.
+
+ The runtime does not include what the cgroup's CPUs already took
+ from it, at most a slice each (sched_cfs_bandwidth_slice_us).
+ A write leaves that with the CPUs.
+
+ This file affects only processes under the fair-class scheduler.
+
cpu.pressure
A read-write nested-keyed file.
diff --git a/Documentation/scheduler/sched-bwc.rst b/Documentation/scheduler/sched-bwc.rst
index e881a945c1888..c22b3c8c14e29 100644
--- a/Documentation/scheduler/sched-bwc.rst
+++ b/Documentation/scheduler/sched-bwc.rst
@@ -90,8 +90,8 @@ bandwidth restriction in place, such a group is described as an unconstrained
bandwidth group. This represents the traditional work-conserving behavior for
CFS.
-Writing any (valid) positive value(s) no smaller than cpu.cfs_burst_us will
-enact the specified bandwidth limit. The minimum quota allowed for the quota or
+Writing any (valid) positive value(s) will enact the specified bandwidth
+limit. The minimum quota allowed for the quota or
period is 1ms. There is also an upper bound on the period length of 1s.
Additional restrictions exist when bandwidth limits are used in a hierarchical
fashion, these are explained in more detail below.
@@ -101,9 +101,10 @@ and return the group to an unconstrained state once more.
A value of 0 for cpu.cfs_burst_us indicates that the group can not accumulate
any unused bandwidth. It makes the traditional bandwidth control behavior for
-CFS unchanged. Writing any (valid) positive value(s) no larger than
-cpu.cfs_quota_us into cpu.cfs_burst_us will enact the cap on unused bandwidth
-accumulation.
+CFS unchanged. Writing any (valid) positive value(s) into cpu.cfs_burst_us
+will enact the cap on unused bandwidth accumulation: the group can accumulate
+up to cpu.cfs_quota_us + cpu.cfs_burst_us. The burst may be larger than the
+quota, as long as their sum is at most 2^44 - 1 microseconds (about 203 days).
Any updates to a group's bandwidth specification will result in it becoming
unthrottled if it is in a constrained state.
@@ -243,4 +244,4 @@ Examples
# echo 50000 > cpu.cfs_period_us /* period = 50ms */
# echo 10000 > cpu.cfs_burst_us /* burst = 10ms */
- Larger buffer setting (no larger than quota) allows greater burst capacity.
+ Larger buffer setting allows greater burst capacity.
diff --git a/kernel/sched/core.c b/kernel/sched/core.c
index 175b52a7ea793..e068d81e1fc5f 100644
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -9860,6 +9860,16 @@ static int tg_set_cfs_bandwidth(struct task_group *tg,
cfs_b->quota = quota;
cfs_b->burst = burst;
+ /*
+ * A group without a quota accumulates no runtime, so drop what
+ * it had rather than report it and add to it once it has a
+ * quota again.
+ */
+ if (!runtime_enabled) {
+ cfs_b->runtime = 0;
+ cfs_b->runtime_snap = 0;
+ }
+
__refill_cfs_bandwidth_runtime(cfs_b);
/*
@@ -10131,8 +10141,7 @@ static int tg_set_bandwidth(struct task_group *tg,
if (quota_us != RUNTIME_INF && quota_us > max_bw_runtime_us)
return -EINVAL;
- if (quota_us != RUNTIME_INF && (burst_us > quota_us ||
- burst_us + quota_us > max_bw_runtime_us))
+ if (quota_us != RUNTIME_INF && burst_us > max_bw_runtime_us - quota_us)
return -EINVAL;
#ifdef CONFIG_CFS_BANDWIDTH
@@ -10193,6 +10202,29 @@ static int cpu_burst_write_u64(struct cgroup_subsys_state *css,
tg_bandwidth(tg, &period_us, "a_us, NULL);
return tg_set_bandwidth(tg, period_us, quota_us, burst_us);
}
+
+#ifdef CONFIG_CFS_BANDWIDTH
+static int cpu_runtime_write_u64(struct cgroup_subsys_state *css,
+ struct cftype *cftype, u64 runtime_us)
+{
+ struct task_group *tg = css_tg(css);
+
+ if (runtime_us > max_bw_runtime_us)
+ return -EINVAL;
+
+ return cfs_bandwidth_set_runtime(&tg->cfs_bandwidth,
+ runtime_us * NSEC_PER_USEC);
+}
+
+static u64 cpu_runtime_read_u64(struct cgroup_subsys_state *css,
+ struct cftype *cftype)
+{
+ struct task_group *tg = css_tg(css);
+
+ return div_u64(cfs_bandwidth_get_runtime(&tg->cfs_bandwidth),
+ NSEC_PER_USEC);
+}
+#endif /* CONFIG_CFS_BANDWIDTH */
#endif /* CONFIG_GROUP_SCHED_BANDWIDTH */
#ifdef CONFIG_RT_GROUP_SCHED
@@ -10544,7 +10576,15 @@ static struct cftype cpu_files[] = {
.read_u64 = cpu_burst_read_u64,
.write_u64 = cpu_burst_write_u64,
},
+#ifdef CONFIG_CFS_BANDWIDTH
+ {
+ .name = "max.runtime",
+ .flags = CFTYPE_NOT_ON_ROOT,
+ .read_u64 = cpu_runtime_read_u64,
+ .write_u64 = cpu_runtime_write_u64,
+ },
#endif /* CONFIG_CFS_BANDWIDTH */
+#endif /* CONFIG_GROUP_SCHED_BANDWIDTH */
#ifdef CONFIG_UCLAMP_TASK_GROUP
{
.name = "uclamp.min",
diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index d4cc0caee6265..8fee8d27faa5a 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -7413,6 +7413,12 @@ static void unthrottle_cfs_rq_async(struct cfs_rq *cfs_rq)
__unthrottle_cfs_rq_async(cfs_rq);
}
+/*
+ * Hand the runtime of @cfs_b to its throttled cfs_rqs. The caller must not
+ * migrate, as the cfs_rqs of this CPU are unthrottled from it here. The period
+ * and slack timers run from a timer callback, and cfs_bandwidth_set_runtime()
+ * disables interrupts.
+ */
static bool distribute_cfs_runtime(struct cfs_bandwidth *cfs_b)
{
bool throttled = false, unthrottle_local = false;
@@ -7811,6 +7817,65 @@ static void init_cfs_rq_runtime(struct cfs_rq *cfs_rq)
INIT_LIST_HEAD(&cfs_rq->throttled_limbo_list);
}
+/*
+ * The runtime of @cfs_b, as cpu.max.runtime reads it.
+ *
+ * A group without a quota has no runtime: nothing refills it and a write is
+ * rejected, so report 0 as for a group that never had one.
+ *
+ * __return_cfs_rq_runtime() returns slack without capping it, so the runtime
+ * can exceed quota + burst until the next refill caps it. Report it capped,
+ * so that a value read can always be written back.
+ */
+u64 cfs_bandwidth_get_runtime(struct cfs_bandwidth *cfs_b)
+{
+ guard(raw_spinlock_irq)(&cfs_b->lock);
+
+ if (cfs_b->quota == RUNTIME_INF)
+ return 0;
+
+ return min(cfs_b->runtime, cfs_b->quota + cfs_b->burst);
+}
+
+/*
+ * Set the runtime of @cfs_b, as a cpu.max.runtime write does, and hand it to
+ * the throttled cfs_rqs at once rather than at the next period.
+ */
+int cfs_bandwidth_set_runtime(struct cfs_bandwidth *cfs_b, u64 runtime)
+{
+ bool throttled;
+ s64 snap;
+
+ guard(irq)();
+
+ scoped_guard (raw_spinlock, &cfs_b->lock) {
+ if (cfs_b->quota == RUNTIME_INF ||
+ runtime > cfs_b->quota + cfs_b->burst)
+ return -EINVAL;
+
+ /*
+ * The next refill counts runtime_snap - runtime as burst usage.
+ * Move runtime_snap by what the write adds or takes away, so
+ * that the write itself is not counted as burst usage but what
+ * the group ran since the last refill still is.
+ */
+ snap = cfs_b->runtime_snap + runtime - cfs_b->runtime;
+ cfs_b->runtime_snap = max(snap, 0);
+ cfs_b->runtime = runtime;
+
+ throttled = !list_empty(&cfs_b->throttled_cfs_rq) && runtime;
+ }
+
+ /*
+ * distribute_cfs_runtime() takes the rq locks, so it runs without
+ * cfs_b->lock and with interrupts disabled, as from the slack timer.
+ */
+ if (throttled)
+ distribute_cfs_runtime(cfs_b);
+
+ return 0;
+}
+
void start_cfs_bandwidth(struct cfs_bandwidth *cfs_b)
{
lockdep_assert_held(&cfs_b->lock);
diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h
index a4df59c9c9b60..1f60c84e0dc38 100644
--- a/kernel/sched/sched.h
+++ b/kernel/sched/sched.h
@@ -578,6 +578,8 @@ extern void init_cfs_bandwidth(struct cfs_bandwidth *cfs_b, struct cfs_bandwidth
extern void __refill_cfs_bandwidth_runtime(struct cfs_bandwidth *cfs_b);
extern void start_cfs_bandwidth(struct cfs_bandwidth *cfs_b);
+u64 cfs_bandwidth_get_runtime(struct cfs_bandwidth *cfs_b);
+int cfs_bandwidth_set_runtime(struct cfs_bandwidth *cfs_b, u64 runtime);
extern void unthrottle_cfs_rq(struct cfs_rq *cfs_rq);
extern bool cfs_task_bw_constrained(struct task_struct *p);
diff --git a/tools/testing/selftests/cgroup/config b/tools/testing/selftests/cgroup/config
index 39f979690dd3b..aaf5872c71410 100644
--- a/tools/testing/selftests/cgroup/config
+++ b/tools/testing/selftests/cgroup/config
@@ -1,3 +1,4 @@
+CONFIG_CFS_BANDWIDTH=y
CONFIG_CGROUPS=y
CONFIG_CGROUP_CPUACCT=y
CONFIG_CGROUP_FREEZER=y
diff --git a/tools/testing/selftests/cgroup/test_cpu.c b/tools/testing/selftests/cgroup/test_cpu.c
index 3456e3e988c68..b5a9fbcf722f6 100644
--- a/tools/testing/selftests/cgroup/test_cpu.c
+++ b/tools/testing/selftests/cgroup/test_cpu.c
@@ -809,6 +809,357 @@ static int test_cpucg_max_nested(const char *root)
return ret;
}
+/* Wait until the period timer of @cgroup starts a new period. */
+static int wait_period_start(const char *cgroup)
+{
+ long nr_periods = cg_read_key_long(cgroup, "cpu.stat", "nr_periods");
+ int i;
+
+ for (i = 0; i < 3000; i++) {
+ usleep(1000);
+ if (cg_read_key_long(cgroup, "cpu.stat", "nr_periods") != nr_periods)
+ return 0;
+ }
+ return -1;
+}
+
+static long long monotonic_ns(void)
+{
+ struct timespec ts;
+
+ clock_gettime(CLOCK_MONOTONIC, &ts);
+ return ts.tv_sec * 1000000000LL + ts.tv_nsec;
+}
+
+/* For (long)arg seconds, run for 1ms and then sleep for 10ms. */
+static int run_and_sleep(const char *cgroup, void *arg)
+{
+ long long end = monotonic_ns() + (long)arg * 1000000000LL;
+ long long start;
+
+ while ((start = monotonic_ns()) < end) {
+ while (monotonic_ns() - start < 1000000)
+ ;
+ usleep(10000);
+ }
+ return 0;
+}
+
+static int test_cpucg_max_runtime(const char *root)
+{
+ int ret = KSFT_FAIL;
+ long quota_usec = 1000; /* 1ms (minimum) */
+ long period_usec = 100000; /* 100ms */
+ long burst_usec = 5000000; /* 5s, so cap = 5001ms */
+ long runtime_usec = 2500000; /* 2500ms, half of the 5s run */
+ long duration_sec = 5;
+ /* The injected runtime plus the quota refilled each period of the run */
+ long expected_usec = runtime_usec +
+ duration_sec * USEC_PER_SEC / period_usec * quota_usec;
+ long usage_usec;
+ char *cpucg;
+ char buf[64];
+ int pid;
+
+ cpucg = cg_name(root, "cpucg_runtime_test");
+ if (!cpucg)
+ goto cleanup;
+
+ if (cg_create(cpucg))
+ goto cleanup;
+
+ /* Without a quota there is no runtime to set, and none to read. */
+ if (!cg_write_numeric(cpucg, "cpu.max.runtime", runtime_usec))
+ goto cleanup;
+ if (cg_read_long(cpucg, "cpu.max.runtime") != 0)
+ goto cleanup;
+
+ snprintf(buf, sizeof(buf), "%ld %ld", quota_usec, period_usec);
+ if (cg_write(cpucg, "cpu.max", buf))
+ goto cleanup;
+ if (cg_write_numeric(cpucg, "cpu.max.burst", burst_usec))
+ goto cleanup;
+
+ /* Start burner, let it settle, then inject credits */
+ struct cpu_hog_func_param param = {
+ .nprocs = 1,
+ .ts = { .tv_sec = duration_sec, .tv_nsec = 0 },
+ .clock_type = CPU_HOG_CLOCK_WALL,
+ };
+ pid = cg_run_nowait(cpucg, hog_cpus_timed, (void *)¶m);
+ if (pid < 0)
+ goto cleanup;
+
+ usleep(100000);
+ if (cg_write_numeric(cpucg, "cpu.max.runtime", runtime_usec)) {
+ kill(pid, SIGKILL);
+ waitpid(pid, NULL, 0);
+ goto cleanup;
+ }
+
+ waitpid(pid, NULL, 0);
+
+ usage_usec = cg_read_key_long(cpucg, "cpu.stat", "usage_usec");
+ if (usage_usec <= 0)
+ goto cleanup;
+
+ /* At least the injected runtime, at most 5% above the expected usage. */
+ if (usage_usec < runtime_usec ||
+ usage_usec > expected_usec + expected_usec / 20)
+ goto cleanup;
+
+ /* Taking the quota away drops the runtime rather than storing it. */
+ if (cg_write(cpucg, "cpu.max", "max"))
+ goto cleanup;
+ if (cg_read_long(cpucg, "cpu.max.runtime") != 0)
+ goto cleanup;
+ if (cg_write(cpucg, "cpu.max", buf))
+ goto cleanup;
+ if (cg_read_long(cpucg, "cpu.max.runtime") > quota_usec)
+ goto cleanup;
+
+ ret = KSFT_PASS;
+
+cleanup:
+ cg_destroy(cpucg);
+ free(cpucg);
+ return ret;
+}
+
+/*
+ * A cpu.max.runtime write is not burst usage: writing a runtime below the
+ * current one leaves cpu.stat's nr_bursts and burst_usec where they were.
+ */
+static int test_cpucg_max_runtime_burst_stats(const char *root)
+{
+ int ret = KSFT_FAIL;
+ long nr_bursts, burst_usec;
+ char *cpucg;
+
+ cpucg = cg_name(root, "cpucg_runtime_burst_stats_test");
+ if (!cpucg)
+ goto cleanup;
+
+ if (cg_create(cpucg))
+ goto cleanup;
+
+ /* A long period, so the write lands before the timer's first refill. */
+ if (cg_write(cpucg, "cpu.max", "10000 1000000"))
+ goto cleanup;
+ if (cg_write_numeric(cpucg, "cpu.max.burst", 190000))
+ goto cleanup;
+ if (cg_write_numeric(cpucg, "cpu.max.runtime", 0))
+ goto cleanup;
+
+ /* The idle group refills for a few periods. */
+ sleep(3);
+ nr_bursts = cg_read_key_long(cpucg, "cpu.stat", "nr_bursts");
+ burst_usec = cg_read_key_long(cpucg, "cpu.stat", "burst_usec");
+ if (nr_bursts != 0 || burst_usec != 0)
+ goto cleanup;
+
+ ret = KSFT_PASS;
+
+cleanup:
+ cg_destroy(cpucg);
+ free(cpucg);
+ return ret;
+}
+
+/*
+ * A cpu.max.runtime write hands the runtime to a throttled group at once,
+ * rather than at the next period: with a 1s period, the group runs again
+ * within 100ms of the write, before the next period starts.
+ */
+static int test_cpucg_max_runtime_unthrottle(const char *root)
+{
+ int ret = KSFT_FAIL;
+ long usage_before, usage_after, nr_periods;
+ char *cpucg;
+ int pid;
+
+ cpucg = cg_name(root, "cpucg_runtime_unthrottle_test");
+ if (!cpucg)
+ goto cleanup;
+
+ if (cg_create(cpucg))
+ goto cleanup;
+
+ /* 1ms per 1s period, with room for a runtime of up to 1s. */
+ if (cg_write(cpucg, "cpu.max", "1000 1000000"))
+ goto cleanup;
+ if (cg_write_numeric(cpucg, "cpu.max.burst", 999000))
+ goto cleanup;
+
+ struct cpu_hog_func_param param = {
+ .nprocs = 1,
+ .ts = { .tv_sec = 2, .tv_nsec = 0 },
+ .clock_type = CPU_HOG_CLOCK_WALL,
+ };
+ pid = cg_run_nowait(cpucg, hog_cpus_timed, (void *)¶m);
+ if (pid < 0)
+ goto cleanup;
+
+ /*
+ * The period timer's phase is random: start right after a boundary,
+ * so that the next one is some 900ms away. The hog uses up its runtime
+ * and is throttled for the period.
+ */
+ if (wait_period_start(cpucg))
+ goto cleanup_hog;
+ nr_periods = cg_read_key_long(cpucg, "cpu.stat", "nr_periods");
+ usleep(50000);
+ usage_before = cg_read_key_long(cpucg, "cpu.stat", "usage_usec");
+ if (usage_before < 0 || usage_before > 20000)
+ goto cleanup_hog;
+
+ if (cg_write_numeric(cpucg, "cpu.max.runtime", 500000))
+ goto cleanup_hog;
+
+ /* No period started meanwhile: only the write lets the hog run. */
+ usleep(100000);
+ usage_after = cg_read_key_long(cpucg, "cpu.stat", "usage_usec");
+ if (cg_read_key_long(cpucg, "cpu.stat", "nr_periods") != nr_periods)
+ goto cleanup_hog;
+ if (usage_after - usage_before < 50000)
+ goto cleanup_hog;
+
+ ret = KSFT_PASS;
+
+cleanup_hog:
+ kill(pid, SIGKILL);
+ waitpid(pid, NULL, 0);
+cleanup:
+ cg_destroy(cpucg);
+ free(cpucg);
+ return ret;
+}
+
+/*
+ * What a group ran beyond its quota before a cpu.max.runtime write is still
+ * burst usage: a write of the runtime the group has left does not hide it.
+ */
+static int test_cpucg_max_runtime_burst_kept(const char *root)
+{
+ int ret = KSFT_FAIL;
+ long burst_before, burst_after, runtime_usec;
+ char *cpucg;
+ int pid;
+
+ cpucg = cg_name(root, "cpucg_runtime_burst_kept_test");
+ if (!cpucg)
+ goto cleanup;
+
+ if (cg_create(cpucg))
+ goto cleanup;
+
+ /* 10ms per 1s period, and a runtime of 1s to run on. */
+ if (cg_write(cpucg, "cpu.max", "10000 1000000"))
+ goto cleanup;
+ if (cg_write_numeric(cpucg, "cpu.max.burst", 990000))
+ goto cleanup;
+
+ struct cpu_hog_func_param param = {
+ .nprocs = 1,
+ .ts = { .tv_sec = 10, .tv_nsec = 0 },
+ .clock_type = CPU_HOG_CLOCK_WALL,
+ };
+ pid = cg_run_nowait(cpucg, hog_cpus_timed, (void *)¶m);
+ if (pid < 0)
+ goto cleanup;
+
+ /*
+ * The period timer's phase is random, so grant the runtime only once
+ * a period has started: the hog spends it all in that one period.
+ */
+ if (wait_period_start(cpucg)) {
+ kill(pid, SIGKILL);
+ waitpid(pid, NULL, 0);
+ goto cleanup;
+ }
+ if (cg_write_numeric(cpucg, "cpu.max.runtime", 1000000)) {
+ kill(pid, SIGKILL);
+ waitpid(pid, NULL, 0);
+ goto cleanup;
+ }
+ burst_before = cg_read_key_long(cpucg, "cpu.stat", "burst_usec");
+
+ /* Run some 300ms, all in one period, and stop. */
+ usleep(300000);
+ kill(pid, SIGKILL);
+ waitpid(pid, NULL, 0);
+
+ /* Write back what is left: the next refill counts the 300ms run. */
+ runtime_usec = cg_read_long(cpucg, "cpu.max.runtime");
+ if (cg_write_numeric(cpucg, "cpu.max.runtime", runtime_usec))
+ goto cleanup;
+ sleep(1);
+ burst_after = cg_read_key_long(cpucg, "cpu.stat", "burst_usec");
+ if (burst_before < 0 || burst_after - burst_before < 250000)
+ goto cleanup;
+
+ ret = KSFT_PASS;
+
+cleanup:
+ cg_destroy(cpucg);
+ free(cpucg);
+ return ret;
+}
+
+/*
+ * cpu.max.runtime reads at most quota + burst, so that a value read can be
+ * written back. A cfs_rq that dequeues returns its unused runtime, which can
+ * take the runtime above quota + burst until the next refill caps it: run
+ * tasks that keep doing that, and read and write back the runtime meanwhile.
+ */
+static int test_cpucg_max_runtime_read_capped(const char *root)
+{
+ int ret = KSFT_FAIL;
+ long cap_usec = 100000; /* quota 100ms, no burst */
+ long long end;
+ long runtime_usec;
+ char *cpucg;
+ int pids[2];
+ int i, n = 0;
+
+ cpucg = cg_name(root, "cpucg_runtime_read_capped_test");
+ if (!cpucg)
+ goto cleanup;
+
+ if (cg_create(cpucg))
+ goto cleanup;
+
+ if (cg_write(cpucg, "cpu.max", "100000 100000"))
+ goto cleanup;
+
+ for (n = 0; n < 2; n++) {
+ pids[n] = cg_run_nowait(cpucg, run_and_sleep, (void *)4L);
+ if (pids[n] < 0)
+ goto cleanup_tasks;
+ }
+
+ end = monotonic_ns() + 3000000000LL;
+ while (monotonic_ns() < end) {
+ runtime_usec = cg_read_long(cpucg, "cpu.max.runtime");
+ if (runtime_usec < 0 || runtime_usec > cap_usec)
+ goto cleanup_tasks;
+ if (cg_write_numeric(cpucg, "cpu.max.runtime", runtime_usec))
+ goto cleanup_tasks;
+ }
+
+ ret = KSFT_PASS;
+
+cleanup_tasks:
+ for (i = 0; i < n; i++) {
+ kill(pids[i], SIGKILL);
+ waitpid(pids[i], NULL, 0);
+ }
+cleanup:
+ cg_destroy(cpucg);
+ free(cpucg);
+ return ret;
+}
+
#define T(x) { x, #x }
struct cpucg_test {
int (*fn)(const char *root);
@@ -823,6 +1174,11 @@ struct cpucg_test {
T(test_cpucg_nested_weight_underprovisioned),
T(test_cpucg_max),
T(test_cpucg_max_nested),
+ T(test_cpucg_max_runtime),
+ T(test_cpucg_max_runtime_burst_stats),
+ T(test_cpucg_max_runtime_unthrottle),
+ T(test_cpucg_max_runtime_burst_kept),
+ T(test_cpucg_max_runtime_read_capped),
};
#undef T
--
2.47.3
Amazon Development Centre (South Africa) (Proprietary) Limited
29 Gogosoa Street, Observatory, Cape Town, Western Cape, 7925, South Africa
Registration Number: 2004 / 034463 / 07
^ permalink raw reply related [flat|nested] 14+ messages in thread* [PATCH v3 2/2] sched/fair: refill bandwidth for the periods the timer slept through
2026-11-11 12:29 ` [PATCH v3 0/2] sched/fair: cpu.max.runtime and idle refill for a burstable budget Fernand Sieber
2026-11-11 12:29 ` [PATCH v3 1/2] sched/fair: expose cpu.max.runtime to set bandwidth runtime directly Fernand Sieber
@ 2026-11-11 12:29 ` Fernand Sieber
1 sibling, 0 replies; 14+ messages in thread
From: Fernand Sieber @ 2026-11-11 12:29 UTC (permalink / raw)
To: mingo, peterz, juri.lelli, vincent.guittot
Cc: tj, bsegall, dietmar.eggemann, rostedt, mgorman, vschneid,
kprateek.nayak, hannes, mkoutny, skhan, corbet, rdunlap, cgroups,
linux-doc, linux-kselftest, linux-kernel, nh-open-source,
hborghor, fmubeen, dwmw, sieberf
cpu.max.burst lets a group accumulate unused quota up to quota + burst.
But the period timer stops after a period in which the group took no
runtime, and start_cfs_bandwidth() restarts it with hrtimer_forward_now()
and discards the periods that passed. A group that sits idle stops
accumulating, and one that wakes after idling for a long time has only
what it had when the timer stopped.
Count the missed periods instead. While the timer is stopped, its expiry
is the first period boundary it missed, so the overrun at restart is the
number of refills owed. Add quota for each, capped at quota + burst as
__refill_cfs_bandwidth_runtime() caps it.
Without a burst the cap is quota. The timer stops right after a refill
that nothing drew from, so the runtime is at quota then, and groups
without a burst see no change. Groups with a burst do see a change. A
group that has been idle now has up to quota + burst when it runs again.
The documentation of cpu.max.burst, cpu.max.runtime and cpu.cfs_burst_us
says so.
A change of the bandwidth (cpu.max, cpu.max.burst) applies from that
moment, and the group keeps what it earned before it. The periods owed
are first refilled at the bandwidth they were earned under, capped at
its quota + burst, and then the new bandwidth applies, with its own cap.
A group that had no quota earned nothing, so one given a quota after a
long time unconstrained does not start full. The next period starts from
the change too: a stopped timer's expiry is a boundary of the period the
group had, so leaving it would put the first refill under the new
bandwidth up to one old period away. A cpu.max.runtime write also
refills the periods owed first, so that it replaces them, and only the
periods after the write refill.
cpu.max.runtime reports the runtime the group has now, owed refills
included, without starting the timer.
The idle periods are not added to cpu.stat's nr_periods, which counts the
periods in which the timer ran. A timer that runs late by several
periods still refills only once, unlike a stopped one; this patch leaves
that as it is.
Signed-off-by: Fernand Sieber <sieberf@amazon.com>
---
Documentation/admin-guide/cgroup-v2.rst | 17 +-
Documentation/scheduler/sched-bwc.rst | 5 +-
kernel/sched/core.c | 12 ++
kernel/sched/fair.c | 100 +++++++++-
kernel/sched/sched.h | 2 +
tools/testing/selftests/cgroup/test_cpu.c | 231 ++++++++++++++++++++++
6 files changed, 355 insertions(+), 12 deletions(-)
diff --git a/Documentation/admin-guide/cgroup-v2.rst b/Documentation/admin-guide/cgroup-v2.rst
index eb802f8ba877f..12d40b55f2dbb 100644
--- a/Documentation/admin-guide/cgroup-v2.rst
+++ b/Documentation/admin-guide/cgroup-v2.rst
@@ -1232,9 +1232,9 @@ will be referred to. All time durations are in microseconds.
cgroups. The default is "0".
The burst. The cgroup can accumulate the runtime it does not
- use, up to $MAX + cpu.max.burst. The burst may exceed $MAX, as
- long as $MAX + cpu.max.burst is at most 2^44 - 1 (about 203
- days).
+ use, up to $MAX + cpu.max.burst, also in the periods in which it
+ does not run. The burst may exceed $MAX, as long as $MAX +
+ cpu.max.burst is at most 2^44 - 1 (about 203 days).
This file affects only processes under the fair-class scheduler and a BPF
scheduler with the ``cgroup_set_bandwidth`` callback depending on what
@@ -1246,11 +1246,12 @@ will be referred to. All time durations are in microseconds.
The runtime the cgroup has accumulated and not used yet, in
microseconds. Each period refills it by $MAX, up to $MAX +
- cpu.max.burst, and a read returns at most $MAX + cpu.max.burst.
- A write sets it, up to $MAX + cpu.max.burst, and lets a cgroup
- that ran out of runtime run again at once. A write fails with
- EINVAL while $MAX is "max", a read then returns 0, and setting
- $MAX to "max" drops the runtime the cgroup had.
+ cpu.max.burst, whether the cgroup runs in it or not. A read
+ returns at most $MAX + cpu.max.burst. A write sets it, up to
+ $MAX + cpu.max.burst, and lets a cgroup that ran out of runtime
+ run again at once. A write fails with EINVAL while $MAX is
+ "max", a read then returns 0, and setting $MAX to "max" drops
+ the runtime the cgroup had.
The runtime does not include what the cgroup's CPUs already took
from it, at most a slice each (sched_cfs_bandwidth_slice_us).
diff --git a/Documentation/scheduler/sched-bwc.rst b/Documentation/scheduler/sched-bwc.rst
index c22b3c8c14e29..ccf7434ca73f0 100644
--- a/Documentation/scheduler/sched-bwc.rst
+++ b/Documentation/scheduler/sched-bwc.rst
@@ -103,8 +103,9 @@ A value of 0 for cpu.cfs_burst_us indicates that the group can not accumulate
any unused bandwidth. It makes the traditional bandwidth control behavior for
CFS unchanged. Writing any (valid) positive value(s) into cpu.cfs_burst_us
will enact the cap on unused bandwidth accumulation: the group can accumulate
-up to cpu.cfs_quota_us + cpu.cfs_burst_us. The burst may be larger than the
-quota, as long as their sum is at most 2^44 - 1 microseconds (about 203 days).
+up to cpu.cfs_quota_us + cpu.cfs_burst_us, also over periods in which it does
+not run. The burst may be larger than the quota, as long as their sum is at
+most 2^44 - 1 microseconds (about 203 days).
Any updates to a group's bandwidth specification will result in it becoming
unthrottled if it is in a constrained state.
diff --git a/kernel/sched/core.c b/kernel/sched/core.c
index e068d81e1fc5f..265088af6a561 100644
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -9856,10 +9856,22 @@ static int tg_set_cfs_bandwidth(struct task_group *tg,
cfs_bandwidth_usage_inc();
scoped_guard (raw_spinlock_irq, &cfs_b->lock) {
+ /*
+ * The new bandwidth applies from now on. The group keeps what
+ * it earned while idle under the old one.
+ */
+ cfs_bandwidth_refill_idle_periods(cfs_b);
+
cfs_b->period = ns_to_ktime(period);
cfs_b->quota = quota;
cfs_b->burst = burst;
+ /*
+ * The next period starts from here too, rather than at the
+ * boundary left over from the old one.
+ */
+ cfs_bandwidth_reanchor_period(cfs_b);
+
/*
* A group without a quota accumulates no runtime, so drop what
* it had rather than report it and add to it once it has a
diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index 8fee8d27faa5a..0a65a8e232c96 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -7817,6 +7817,92 @@ static void init_cfs_rq_runtime(struct cfs_rq *cfs_rq)
INIT_LIST_HEAD(&cfs_rq->throttled_limbo_list);
}
+/*
+ * Runtime after @periods refills of quota, capped at quota + burst as each
+ * refill in __refill_cfs_bandwidth_runtime() caps it. The runtime can be above
+ * the cap (__return_cfs_rq_runtime() does not cap it), and a refill then cuts
+ * it to the cap.
+ */
+static u64 cfs_bandwidth_refilled(struct cfs_bandwidth *cfs_b, u64 periods)
+{
+ u64 cap = cfs_b->quota + cfs_b->burst;
+ u64 runtime = cfs_b->runtime;
+
+ if (!periods || cfs_b->quota == RUNTIME_INF)
+ return runtime;
+ if (runtime >= cap ||
+ periods >= div64_u64(cap - runtime, cfs_b->quota) + 1)
+ return cap;
+ return runtime + periods * cfs_b->quota;
+}
+
+/*
+ * The period timer stops after a period in which the group took no runtime
+ * (do_sched_cfs_period_timer()), and start_cfs_bandwidth() restarts it on
+ * the next demand. The periods it slept through refill quota too: unused
+ * quota accumulates up to quota + burst whether or not the group ran. While
+ * the timer is stopped its expiry is the first period boundary it missed,
+ * so the boundaries passed since then are the refills still owed.
+ *
+ * Without burst the cap is quota. The timer stops right after a refill that
+ * nothing drew from, so the runtime is at quota then, and this changes nothing
+ * for groups without a burst.
+ *
+ * This is the overrun that hrtimer_forward_now() returns, without moving the
+ * expiry: one for the expiry itself, plus one for each full period since.
+ */
+static u64 cfs_bandwidth_idle_periods(struct cfs_bandwidth *cfs_b)
+{
+ ktime_t now = ktime_get();
+ s64 delta;
+
+ if (cfs_b->period_active)
+ return 0;
+
+ delta = ktime_to_ns(ktime_sub(now, hrtimer_get_expires(&cfs_b->period_timer)));
+ if (delta < 0)
+ return 0;
+ return div64_u64(delta, ktime_to_ns(cfs_b->period)) + 1;
+}
+
+/*
+ * Refill the periods a stopped timer slept through, at the bandwidth they
+ * were earned under, and move its expiry past them so that they are not
+ * counted again. A change of the bandwidth or a cpu.max.runtime write calls
+ * this first, so that the change applies from then on and the group keeps
+ * what it earned before. A group without a quota earned nothing.
+ */
+void cfs_bandwidth_refill_idle_periods(struct cfs_bandwidth *cfs_b)
+{
+ u64 idle_periods;
+
+ lockdep_assert_held(&cfs_b->lock);
+
+ if (cfs_b->period_active)
+ return;
+
+ idle_periods = hrtimer_forward_now(&cfs_b->period_timer, cfs_b->period);
+ cfs_b->runtime = cfs_bandwidth_refilled(cfs_b, idle_periods);
+ cfs_b->runtime_snap = cfs_b->runtime;
+}
+
+/*
+ * Start the next period of a stopped timer from now, as a bandwidth change
+ * does. Its expiry is a boundary of the period the group had; move it one new
+ * period past now, so that cfs_bandwidth_refill_idle_periods() owes nothing
+ * for it when the timer restarts.
+ */
+void cfs_bandwidth_reanchor_period(struct cfs_bandwidth *cfs_b)
+{
+ lockdep_assert_held(&cfs_b->lock);
+
+ if (cfs_b->period_active)
+ return;
+
+ hrtimer_set_expires(&cfs_b->period_timer,
+ ktime_add(ktime_get(), cfs_b->period));
+}
+
/*
* The runtime of @cfs_b, as cpu.max.runtime reads it.
*
@@ -7829,12 +7915,16 @@ static void init_cfs_rq_runtime(struct cfs_rq *cfs_rq)
*/
u64 cfs_bandwidth_get_runtime(struct cfs_bandwidth *cfs_b)
{
+ u64 runtime;
+
guard(raw_spinlock_irq)(&cfs_b->lock);
if (cfs_b->quota == RUNTIME_INF)
return 0;
- return min(cfs_b->runtime, cfs_b->quota + cfs_b->burst);
+ runtime = cfs_bandwidth_refilled(cfs_b, cfs_bandwidth_idle_periods(cfs_b));
+
+ return min(runtime, cfs_b->quota + cfs_b->burst);
}
/*
@@ -7853,6 +7943,12 @@ int cfs_bandwidth_set_runtime(struct cfs_bandwidth *cfs_b, u64 runtime)
runtime > cfs_b->quota + cfs_b->burst)
return -EINVAL;
+ /*
+ * Refill the periods that passed before the write, so that the
+ * write replaces them and only the periods after it refill.
+ */
+ cfs_bandwidth_refill_idle_periods(cfs_b);
+
/*
* The next refill counts runtime_snap - runtime as burst usage.
* Move runtime_snap by what the write adds or takes away, so
@@ -7883,8 +7979,8 @@ void start_cfs_bandwidth(struct cfs_bandwidth *cfs_b)
if (cfs_b->period_active)
return;
+ cfs_bandwidth_refill_idle_periods(cfs_b);
cfs_b->period_active = 1;
- hrtimer_forward_now(&cfs_b->period_timer, cfs_b->period);
hrtimer_start_expires(&cfs_b->period_timer, HRTIMER_MODE_ABS_PINNED);
}
diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h
index 1f60c84e0dc38..a03ac077a8f49 100644
--- a/kernel/sched/sched.h
+++ b/kernel/sched/sched.h
@@ -580,6 +580,8 @@ extern void __refill_cfs_bandwidth_runtime(struct cfs_bandwidth *cfs_b);
extern void start_cfs_bandwidth(struct cfs_bandwidth *cfs_b);
u64 cfs_bandwidth_get_runtime(struct cfs_bandwidth *cfs_b);
int cfs_bandwidth_set_runtime(struct cfs_bandwidth *cfs_b, u64 runtime);
+void cfs_bandwidth_refill_idle_periods(struct cfs_bandwidth *cfs_b);
+void cfs_bandwidth_reanchor_period(struct cfs_bandwidth *cfs_b);
extern void unthrottle_cfs_rq(struct cfs_rq *cfs_rq);
extern bool cfs_task_bw_constrained(struct task_struct *p);
diff --git a/tools/testing/selftests/cgroup/test_cpu.c b/tools/testing/selftests/cgroup/test_cpu.c
index b5a9fbcf722f6..2b7c3bf89a1c9 100644
--- a/tools/testing/selftests/cgroup/test_cpu.c
+++ b/tools/testing/selftests/cgroup/test_cpu.c
@@ -1160,6 +1160,233 @@ static int test_cpucg_max_runtime_read_capped(const char *root)
return ret;
}
+/*
+ * A group with a burst keeps earning quota while it is idle, up to
+ * quota + burst, and cpu.max.runtime reads what it has earned. Without a
+ * burst the runtime stays at quota.
+ */
+static int test_cpucg_max_idle_refill(const char *root)
+{
+ int ret = KSFT_FAIL;
+ long quota_usec = 10000; /* 10ms */
+ long period_usec = 100000; /* 100ms */
+ long burst_usec = 990000; /* cap 1s */
+ long runtime_usec;
+ char *cpucg;
+ char buf[64];
+
+ cpucg = cg_name(root, "cpucg_idle_refill_test");
+ if (!cpucg)
+ goto cleanup;
+
+ if (cg_create(cpucg))
+ goto cleanup;
+
+ snprintf(buf, sizeof(buf), "%ld %ld", quota_usec, period_usec);
+ if (cg_write(cpucg, "cpu.max", buf))
+ goto cleanup;
+
+ /* No burst: the runtime stays at quota while idle. */
+ sleep(1);
+ runtime_usec = cg_read_long(cpucg, "cpu.max.runtime");
+ if (runtime_usec != quota_usec)
+ goto cleanup;
+
+ /* Burst: the write refills once (20ms), then 1s idle earns 10 more. */
+ if (cg_write_numeric(cpucg, "cpu.max.burst", burst_usec))
+ goto cleanup;
+ sleep(1);
+ runtime_usec = cg_read_long(cpucg, "cpu.max.runtime");
+ if (runtime_usec < 110000 || runtime_usec > 130000)
+ goto cleanup;
+
+ /* Capped at quota + burst. */
+ sleep(10);
+ runtime_usec = cg_read_long(cpucg, "cpu.max.runtime");
+ if (runtime_usec != quota_usec + burst_usec)
+ goto cleanup;
+
+ /*
+ * A write sets the runtime from now on: 500ms later, five refills, or
+ * six if the sleep overruns into the next period.
+ */
+ if (cg_write_numeric(cpucg, "cpu.max.runtime", 0))
+ goto cleanup;
+ usleep(500000);
+ runtime_usec = cg_read_long(cpucg, "cpu.max.runtime");
+ if (runtime_usec < 50000 || runtime_usec > 60000)
+ goto cleanup;
+
+ ret = KSFT_PASS;
+
+cleanup:
+ cg_destroy(cpucg);
+ free(cpucg);
+ return ret;
+}
+
+/*
+ * A cpu.max or cpu.max.burst write applies from then on: what a group with
+ * a quota earned while idle is kept, capped at the new quota + burst, and a
+ * group that had no quota has earned nothing while it had none.
+ */
+static int test_cpucg_max_idle_refill_write(const char *root)
+{
+ int ret = KSFT_FAIL;
+ long quota_usec = 10000; /* 10ms */
+ long period_usec = 100000; /* 100ms */
+ long runtime_usec;
+ char *cpucg = NULL, *fresh = NULL;
+ char buf[64];
+
+ cpucg = cg_name(root, "cpucg_idle_refill_write_test");
+ fresh = cg_name(root, "cpucg_idle_refill_fresh_test");
+ if (!cpucg || !fresh)
+ goto cleanup;
+
+ if (cg_create(cpucg) || cg_create(fresh))
+ goto cleanup;
+
+ snprintf(buf, sizeof(buf), "%ld %ld", quota_usec, period_usec);
+ if (cg_write(cpucg, "cpu.max", buf))
+ goto cleanup;
+ if (cg_write_numeric(cpucg, "cpu.max.burst", 190000))
+ goto cleanup;
+
+ /* 3s idle fills the group to quota + burst. */
+ sleep(3);
+ runtime_usec = cg_read_long(cpucg, "cpu.max.runtime");
+ if (runtime_usec != 200000)
+ goto cleanup;
+
+ /* Writing the same cpu.max keeps what the group earned. */
+ if (cg_write(cpucg, "cpu.max", buf))
+ goto cleanup;
+ runtime_usec = cg_read_long(cpucg, "cpu.max.runtime");
+ if (runtime_usec != 200000)
+ goto cleanup;
+
+ /* A lower burst keeps it too, capped at the new quota + burst. */
+ sleep(2);
+ if (cg_write_numeric(cpucg, "cpu.max.burst", 90000))
+ goto cleanup;
+ runtime_usec = cg_read_long(cpucg, "cpu.max.runtime");
+ if (runtime_usec != 100000)
+ goto cleanup;
+
+ /* A group given a quota after idling without one starts with a refill. */
+ if (cg_write_numeric(fresh, "cpu.max.burst", 190000))
+ goto cleanup;
+ sleep(2);
+ if (cg_write(fresh, "cpu.max", buf))
+ goto cleanup;
+ runtime_usec = cg_read_long(fresh, "cpu.max.runtime");
+ if (runtime_usec != quota_usec)
+ goto cleanup;
+
+ ret = KSFT_PASS;
+
+cleanup:
+ if (fresh)
+ cg_destroy(fresh);
+ if (cpucg)
+ cg_destroy(cpucg);
+ free(fresh);
+ free(cpucg);
+ return ret;
+}
+
+/*
+ * A cpu.max.runtime write sets the runtime exactly, and only the periods
+ * after it refill.
+ */
+static int test_cpucg_max_idle_refill_runtime_write(const char *root)
+{
+ int ret = KSFT_FAIL;
+ long runtime_usec;
+ char *cpucg;
+
+ cpucg = cg_name(root, "cpucg_idle_refill_runtime_test");
+ if (!cpucg)
+ goto cleanup;
+
+ if (cg_create(cpucg))
+ goto cleanup;
+
+ if (cg_write(cpucg, "cpu.max", "10000 100000"))
+ goto cleanup;
+ if (cg_write_numeric(cpucg, "cpu.max.burst", 190000))
+ goto cleanup;
+
+ sleep(3);
+ if (cg_write_numeric(cpucg, "cpu.max.runtime", 30000))
+ goto cleanup;
+ /* One refill if a period boundary passes before the read. */
+ runtime_usec = cg_read_long(cpucg, "cpu.max.runtime");
+ if (runtime_usec != 30000 && runtime_usec != 40000)
+ goto cleanup;
+
+ /* 250ms after the write: two or three refills, by the boundaries. */
+ usleep(250000);
+ runtime_usec = cg_read_long(cpucg, "cpu.max.runtime");
+ if (runtime_usec < 50000 || runtime_usec > 60000)
+ goto cleanup;
+
+ ret = KSFT_PASS;
+
+cleanup:
+ cg_destroy(cpucg);
+ free(cpucg);
+ return ret;
+}
+
+/*
+ * A cpu.max write starts the next period from the write, so the first refill
+ * under a new period comes one new period later, not up to one old period
+ * later.
+ */
+static int test_cpucg_max_idle_refill_period_change(const char *root)
+{
+ int ret = KSFT_FAIL;
+ long runtime_usec;
+ char *cpucg;
+
+ cpucg = cg_name(root, "cpucg_idle_refill_period_test");
+ if (!cpucg)
+ goto cleanup;
+
+ if (cg_create(cpucg))
+ goto cleanup;
+
+ /* 10ms of quota every second, the longest period, and a 1s cap. */
+ if (cg_write(cpucg, "cpu.max", "10000 1000000"))
+ goto cleanup;
+ if (cg_write_numeric(cpucg, "cpu.max.burst", 990000))
+ goto cleanup;
+
+ /* Idle until the period timer stops, then start from nothing. */
+ sleep(2);
+ if (cg_write_numeric(cpucg, "cpu.max.runtime", 0))
+ goto cleanup;
+
+ /* 1ms of quota every 10ms from the write on. */
+ if (cg_write(cpucg, "cpu.max", "1000 10000"))
+ goto cleanup;
+
+ /* 200ms is twenty refills of the new quota, plus the write's own. */
+ usleep(200000);
+ runtime_usec = cg_read_long(cpucg, "cpu.max.runtime");
+ if (runtime_usec < 10000)
+ goto cleanup;
+
+ ret = KSFT_PASS;
+
+cleanup:
+ cg_destroy(cpucg);
+ free(cpucg);
+ return ret;
+}
+
#define T(x) { x, #x }
struct cpucg_test {
int (*fn)(const char *root);
@@ -1179,6 +1406,10 @@ struct cpucg_test {
T(test_cpucg_max_runtime_unthrottle),
T(test_cpucg_max_runtime_burst_kept),
T(test_cpucg_max_runtime_read_capped),
+ T(test_cpucg_max_idle_refill),
+ T(test_cpucg_max_idle_refill_write),
+ T(test_cpucg_max_idle_refill_runtime_write),
+ T(test_cpucg_max_idle_refill_period_change),
};
#undef T
--
2.47.3
Amazon Development Centre (South Africa) (Proprietary) Limited
29 Gogosoa Street, Observatory, Cape Town, Western Cape, 7925, South Africa
Registration Number: 2004 / 034463 / 07
^ permalink raw reply related [flat|nested] 14+ messages in thread