* [RFC PATCH v6 21/25] sched/rt: Hook HCBS migration functions
From: Yuri Andriaccio @ 2026-06-08 12:15 UTC (permalink / raw)
To: Ingo Molnar, Peter Zijlstra, Juri Lelli, Vincent Guittot,
Dietmar Eggemann, Steven Rostedt, Ben Segall, Mel Gorman,
Valentin Schneider, Tejun Heo, Johannes Weiner,
Michal Koutný
Cc: cgroups, linux-kernel, Luca Abeni, Yuri Andriaccio
In-Reply-To: <20260608121546.69910-1-yurand2000@gmail.com>
Hook rt-cgroup migration functions:
- select_task_rt_rq
Always return the cpu where the task is scheduled.
- balance_rt
- put_prev_task_rt
If a server is throttled, put_prev_task_rt is invoked and a push is
necessary so that the task can keep running on another server if possible.
- switched_to_rt
Keep track of the deadline server that is assigned to the task switching
to FIFO/RR priority.
Co-developed-by: Alessio Balsini <a.balsini@sssup.it>
Signed-off-by: Alessio Balsini <a.balsini@sssup.it>
Co-developed-by: Andrea Parri <parri.andrea@gmail.com>
Signed-off-by: Andrea Parri <parri.andrea@gmail.com>
Co-developed-by: luca abeni <luca.abeni@santannapisa.it>
Signed-off-by: luca abeni <luca.abeni@santannapisa.it>
Signed-off-by: Yuri Andriaccio <yurand2000@gmail.com>
---
kernel/sched/rt.c | 38 +++++++++++++++++++++++++++++++-------
1 file changed, 31 insertions(+), 7 deletions(-)
diff --git a/kernel/sched/rt.c b/kernel/sched/rt.c
index 276eebe8d0a9..964704d88ba1 100644
--- a/kernel/sched/rt.c
+++ b/kernel/sched/rt.c
@@ -976,6 +976,10 @@ select_task_rq_rt(struct task_struct *p, int cpu, int flags)
struct rq *rq;
bool test;
+ /* Just return the task_cpu for processes inside task groups */
+ if (is_dl_group(rt_rq_of_se(&p->rt)))
+ goto out;
+
/* For anything but wake ups, just return the task_cpu */
if (!(flags & (WF_TTWU | WF_FORK)))
goto out;
@@ -1065,21 +1069,25 @@ static void check_preempt_equal_prio(struct rq *rq, struct task_struct *p)
resched_curr(rq);
}
-static int balance_rt(struct rq *rq, struct task_struct *p, struct rq_flags *rf)
+static int balance_rt(struct rq *global_rq, struct task_struct *p, struct rq_flags *rf)
{
- if (!on_rt_rq(&p->rt) && need_pull_rt_task(rq, p)) {
+ struct rt_rq *rt_rq = rt_rq_of_se(&p->rt);
+
+ if (!on_rt_rq(&p->rt) && need_pull_rt_task(rq_of_rt_rq(rt_rq), p)) {
/*
* This is OK, because current is on_cpu, which avoids it being
* picked for load-balance and preemption/IRQs are still
* disabled avoiding further scheduler activity on it and we've
* not yet started the picking loop.
*/
- rq_unpin_lock(rq, rf);
- pull_rt_rq_task(&rq->rt);
- rq_repin_lock(rq, rf);
+ rq_unpin_lock(global_rq, rf);
+ pull_rt_rq_task(rt_rq);
+ rq_repin_lock(global_rq, rf);
}
- return sched_stop_runnable(rq) || sched_dl_runnable(rq) || sched_rt_runnable(rq);
+ return sched_stop_runnable(global_rq) ||
+ sched_dl_runnable(global_rq) ||
+ sched_rt_runnable(global_rq);
}
/*
@@ -1241,6 +1249,13 @@ static void put_prev_task_rt(struct rq *rq, struct task_struct *p, struct task_s
*/
if (on_rt_rq(&p->rt) && p->nr_cpus_allowed > 1)
enqueue_pushable_task(rt_rq, p);
+
+ if (is_dl_group(rt_rq)) {
+ struct sched_dl_entity *dl_se = dl_group_of(rt_rq);
+
+ if (dl_se->dl_throttled)
+ rt_queue_push_tasks(rt_rq);
+ }
}
/* Only try algorithms three times */
@@ -2050,12 +2065,21 @@ static void switching_to_rt(struct rq *rq, struct task_struct *p) {}
*/
static void switched_to_rt(struct rq *rq, struct task_struct *p)
{
+ struct rt_rq *rt_rq = rt_rq_of_se(&p->rt);
+
/*
* If we are running, update the avg_rt tracking, as the running time
* will now on be accounted into the latter.
*/
if (task_current(rq, p)) {
update_rt_rq_load_avg(rq_clock_pelt(rq), rq, 0);
+
+ if (is_dl_group(rt_rq)) {
+ struct sched_dl_entity *dl_se = dl_group_of(rt_rq);
+
+ p->dl_server = dl_se;
+ }
+
return;
}
@@ -2066,7 +2090,7 @@ static void switched_to_rt(struct rq *rq, struct task_struct *p)
*/
if (task_on_rq_queued(p)) {
if (p->nr_cpus_allowed > 1 && rq->rt.overloaded)
- rt_queue_push_tasks(rt_rq_of_se(&p->rt));
+ rt_queue_push_tasks(rt_rq);
if (p->prio < rq->donor->prio && cpu_online(cpu_of(rq)))
resched_curr(rq);
--
2.54.0
^ permalink raw reply related
* [RFC PATCH v6 20/25] sched/rt: Add HCBS migration code to related functions
From: Yuri Andriaccio @ 2026-06-08 12:15 UTC (permalink / raw)
To: Ingo Molnar, Peter Zijlstra, Juri Lelli, Vincent Guittot,
Dietmar Eggemann, Steven Rostedt, Ben Segall, Mel Gorman,
Valentin Schneider, Tejun Heo, Johannes Weiner,
Michal Koutný
Cc: cgroups, linux-kernel, Luca Abeni, Yuri Andriaccio
In-Reply-To: <20260608121546.69910-1-yurand2000@gmail.com>
Update rt_queue_{push/pull}_task{s} to differentiate between cgroup and
global runqueue balancing.
Introduce new balance callbacks for cgroup migration.
Add rq_to_push_from and rq_to_push_to fields for cgroup related migration.
Balance callbacks are only called on the root runqueues, thus it is
necessary to store which non-root runqueues need to be balanced.
Update migration functions to specialize for cgroup migration:
- find_lowest_rt_rq():
Scan all the cpus to get the cgroup specific lowest_mask.
- find_lock_lowest_rt_rq():
Use appropriate rt_rqs to differentiate the cgroup being checked.
Prevent migration for throttled cgroups.
- push_rt_rq_task():
Allow pushing away for migration disabled tasks only if the tasks
belong to the same cgroup.
- pull_rt_rq_task():
Use appropriate rt_rqs and push away for migration disabled only
if the task to pull and curr are in the same runqueue.
Add tg_of_se to get the task group a scheduling entity is assigned to.
This is different from the active context of the group.
Add new macros for field access and non-CONFIG_RT_GROUP_SCHED code.
Co-developed-by: Alessio Balsini <a.balsini@sssup.it>
Signed-off-by: Alessio Balsini <a.balsini@sssup.it>
Co-developed-by: Andrea Parri <parri.andrea@gmail.com>
Signed-off-by: Andrea Parri <parri.andrea@gmail.com>
Co-developed-by: luca abeni <luca.abeni@santannapisa.it>
Signed-off-by: luca abeni <luca.abeni@santannapisa.it>
Signed-off-by: Yuri Andriaccio <yurand2000@gmail.com>
---
kernel/sched/rt.c | 263 ++++++++++++++++++++++++++++++++-----------
kernel/sched/sched.h | 26 +++++
2 files changed, 221 insertions(+), 68 deletions(-)
diff --git a/kernel/sched/rt.c b/kernel/sched/rt.c
index fc7af6bda3f8..276eebe8d0a9 100644
--- a/kernel/sched/rt.c
+++ b/kernel/sched/rt.c
@@ -364,31 +364,61 @@ static inline int has_pushable_tasks(struct rt_rq *rt_rq)
static DEFINE_PER_CPU(struct balance_callback, rt_push_head);
static DEFINE_PER_CPU(struct balance_callback, rt_pull_head);
+static DEFINE_PER_CPU(struct balance_callback, rt_group_push_head);
+static DEFINE_PER_CPU(struct balance_callback, rt_group_pull_head);
static void push_rt_tasks(struct rq *);
static void pull_rt_task(struct rq *);
+static void push_group_rt_tasks(struct rq *);
+static void pull_group_rt_task(struct rq *);
static inline void rt_queue_push_tasks(struct rt_rq *rt_rq)
{
- struct rq *rq = global_rq_of_rt_rq(rt_rq);
-
- if (is_dl_group(rt_rq))
- return;
+ struct rq *rq = rq_of_rt_rq(rt_rq);
+ struct rq *global_rq = global_rq_of_rt_rq(rt_rq);
if (!has_pushable_tasks(rt_rq))
return;
- queue_balance_callback(rq, &per_cpu(rt_push_head, rq->cpu), push_rt_tasks);
+ if (!rt_group_sched_enabled() || !is_dl_group(rt_rq)) {
+
+ queue_balance_callback(global_rq,
+ &per_cpu(rt_push_head, rq->cpu),
+ push_rt_tasks);
+ } else {
+
+ if (rq_to_push_from(global_rq))
+ return;
+
+ rq_to_push_from(global_rq) = rq;
+ queue_balance_callback(global_rq,
+ &per_cpu(rt_group_push_head, global_rq->cpu),
+ push_group_rt_tasks);
+ }
}
static inline void rt_queue_pull_task(struct rt_rq *rt_rq)
{
- struct rq *rq = global_rq_of_rt_rq(rt_rq);
+ struct rq *rq = rq_of_rt_rq(rt_rq);
+ struct rq *global_rq = global_rq_of_rt_rq(rt_rq);
+ struct sched_dl_entity *dl_se;
- if (is_dl_group(rt_rq))
- return;
+ if (!rt_group_sched_enabled() || !is_dl_group(rt_rq)) {
- queue_balance_callback(rq, &per_cpu(rt_pull_head, rq->cpu), pull_rt_task);
+ queue_balance_callback(global_rq,
+ &per_cpu(rt_pull_head, rq->cpu),
+ pull_rt_task);
+ } else {
+
+ dl_se = dl_group_of(rt_rq);
+ if (dl_se->dl_throttled || rq_to_pull_to(global_rq))
+ return;
+
+ rq_to_pull_to(global_rq) = rq;
+ queue_balance_callback(global_rq,
+ &per_cpu(rt_group_pull_head, global_rq->cpu),
+ pull_group_rt_task);
+ }
}
static void push_rt_rq_tasks(struct rt_rq *rt_rq);
@@ -403,6 +433,27 @@ static void pull_rt_task(struct rq *global_rq) {
pull_rt_rq_task(&global_rq->rt);
}
+static void push_group_rt_tasks(struct rq *global_rq)
+{
+ struct rq *rq = rq_to_push_from(global_rq);
+ struct rt_rq *rt_rq = &rq->rt;
+
+ if (rt_rq->rt_nr_running <= 1 && !dl_group_of(rt_rq)->dl_throttled)
+ return;
+
+ push_rt_rq_tasks(rt_rq);
+ rq_to_push_from(global_rq) = NULL;
+}
+
+static void pull_group_rt_task(struct rq *global_rq)
+{
+ struct rq *rq = rq_to_pull_to(global_rq);
+ struct rt_rq *rt_rq = &rq->rt;
+
+ pull_rt_rq_task(rt_rq);
+ rq_to_pull_to(global_rq) = NULL;
+}
+
static void enqueue_pushable_task(struct rt_rq *rt_rq, struct task_struct *p)
{
plist_del(&p->pushable_tasks, &rt_rq->pushable_tasks);
@@ -1220,35 +1271,71 @@ static DEFINE_PER_CPU(cpumask_var_t, local_cpu_mask);
static int find_lowest_rt_rq(struct task_struct *task)
{
struct sched_domain *sd;
- struct cpumask *lowest_mask = this_cpu_cpumask_var_ptr(local_cpu_mask);
- int this_cpu = smp_processor_id();
- int cpu = task_cpu(task);
- int ret;
-
- /* Make sure the mask is initialized first */
- if (unlikely(!lowest_mask))
- return -1;
+ struct cpumask mask, *lowest_mask;
+ struct sched_dl_entity *dl_se;
+ struct rt_rq *rt_rq, *task_rt_rq = rt_rq_of_se(&task->rt);
+ int cpu, this_cpu = smp_processor_id();
+ int ret, prio, lowest_prio;
if (task->nr_cpus_allowed == 1)
return -1; /* No other targets possible */
- /*
- * If we're on asym system ensure we consider the different capacities
- * of the CPUs when searching for the lowest_mask.
- */
- if (sched_asym_cpucap_active()) {
+ if (!rt_group_sched_enabled() || !is_dl_group(task_rt_rq)) {
+
+ lowest_mask = this_cpu_cpumask_var_ptr(local_cpu_mask);
- ret = cpupri_find_fitness(&task_rq(task)->rd->cpupri,
- task, lowest_mask,
- rt_task_fits_capacity);
+ /* Make sure the mask is initialized first */
+ if (unlikely(!lowest_mask))
+ return -1;
+
+ /*
+ * If we're on asym system ensure we consider the different
+ * capacities of the CPUs when searching for the lowest_mask.
+ */
+ if (sched_asym_cpucap_active()) {
+
+ ret = cpupri_find_fitness(&task_rq(task)->rd->cpupri,
+ task, lowest_mask,
+ rt_task_fits_capacity);
+ } else {
+
+ ret = cpupri_find(&task_rq(task)->rd->cpupri,
+ task, lowest_mask);
+ }
+
+ if (!ret)
+ return -1; /* No targets found */
} else {
- ret = cpupri_find(&task_rq(task)->rd->cpupri,
- task, lowest_mask);
+ lowest_prio = task->prio - 1;
+ lowest_mask = &mask;
+ cpumask_clear(lowest_mask);
+ for_each_cpu_and(cpu, cpu_online_mask, task->cpus_ptr) {
+ dl_se = dl_se_of_tg(task_rt_rq->tg, cpu);
+ rt_rq = &dl_se->my_q->rt;
+ prio = rt_rq->highest_prio.curr;
+
+ /*
+ * If we're on asym system ensure we consider the
+ * different capacities of the CPUs when searching for
+ * the lowest_mask.
+ */
+ if (dl_se->dl_throttled || !rt_task_fits_capacity(task, cpu))
+ continue;
+
+ if (prio >= lowest_prio) {
+ if (prio > lowest_prio) {
+ cpumask_clear(lowest_mask);
+ lowest_prio = prio;
+ }
+
+ cpumask_set_cpu(cpu, lowest_mask);
+ }
+ }
}
- if (!ret)
- return -1; /* No targets found */
+ if (cpumask_empty(lowest_mask))
+ return -1;
/*
* At this point we have built a mask of CPUs representing the
@@ -1258,6 +1345,7 @@ static int find_lowest_rt_rq(struct task_struct *task)
* We prioritize the last CPU that the task executed on since
* it is most likely cache-hot in that location.
*/
+ cpu = task_cpu(task);
if (cpumask_test_cpu(cpu, lowest_mask))
return cpu;
@@ -1268,30 +1356,27 @@ static int find_lowest_rt_rq(struct task_struct *task)
if (!cpumask_test_cpu(this_cpu, lowest_mask))
this_cpu = -1; /* Skip this_cpu opt if not among lowest */
- rcu_read_lock();
- for_each_domain(cpu, sd) {
- if (sd->flags & SD_WAKE_AFFINE) {
+ scoped_guard(rcu) {
+ for_each_domain(cpu, sd) {
int best_cpu;
+ if (!(sd->flags & SD_WAKE_AFFINE))
+ continue;
+
/*
* "this_cpu" is cheaper to preempt than a
* remote processor.
*/
if (this_cpu != -1 &&
- cpumask_test_cpu(this_cpu, sched_domain_span(sd))) {
- rcu_read_unlock();
+ cpumask_test_cpu(this_cpu, sched_domain_span(sd)))
return this_cpu;
- }
best_cpu = cpumask_any_and_distribute(lowest_mask,
sched_domain_span(sd));
- if (best_cpu < nr_cpu_ids) {
- rcu_read_unlock();
+ if (best_cpu < nr_cpu_ids)
return best_cpu;
- }
}
}
- rcu_read_unlock();
/*
* And finally, if there were no matches within the domains
@@ -1342,27 +1427,35 @@ static struct task_struct *pick_next_pushable_task(struct rt_rq *rt_rq)
/* Will lock the rq it finds */
static struct rt_rq *find_lock_lowest_rt_rq(struct task_struct *task, struct rt_rq *rt_rq)
{
- struct rq *rq = rq_of_rt_rq(rt_rq);
- struct rq *lowest_rq = NULL;
- int tries;
- int cpu;
+ struct rq *lowest_rq, *rq = global_rq_of_rt_rq(rt_rq);
+ struct rt_rq *lowest_rt_rq;
+ struct sched_dl_entity *lowest_dl_se;
+ int tries, cpu;
+ bool dl_group;
+
+ dl_group = rt_group_sched_enabled() && is_dl_group(rt_rq);
for (tries = 0; tries < RT_MAX_TRIES; tries++) {
cpu = find_lowest_rt_rq(task);
if ((cpu == -1) || (cpu == rq->cpu))
- break;
+ return NULL;
lowest_rq = cpu_rq(cpu);
+ if (dl_group) {
+ lowest_dl_se = dl_se_of_tg(rt_rq->tg, cpu);
+ lowest_rt_rq = &lowest_dl_se->my_q->rt;
+ } else {
+ lowest_rt_rq = &lowest_rq->rt;
+ }
- if (lowest_rq->rt.highest_prio.curr <= task->prio) {
+ if (lowest_rt_rq->highest_prio.curr <= task->prio) {
/*
* Target rq has tasks of equal or higher priority,
* retrying does not release any lock and is unlikely
* to yield a different result.
*/
- lowest_rq = NULL;
- break;
+ return NULL;
}
/* if the prio of this runqueue changed, try again */
@@ -1378,25 +1471,24 @@ static struct rt_rq *find_lock_lowest_rt_rq(struct task_struct *task, struct rt_
* check the task migration disable flag here too.
*/
if (unlikely(is_migration_disabled(task) ||
+ (dl_group && lowest_dl_se->dl_throttled) ||
!cpumask_test_cpu(lowest_rq->cpu, &task->cpus_mask) ||
task != pick_next_pushable_task(rt_rq))) {
double_unlock_balance(rq, lowest_rq);
- lowest_rq = NULL;
- break;
+ return NULL;
}
}
/* If this rq is still suitable use it. */
- if (lowest_rq->rt.highest_prio.curr > task->prio)
- break;
+ if (lowest_rt_rq->highest_prio.curr > task->prio)
+ return lowest_rt_rq;
/* try again */
double_unlock_balance(rq, lowest_rq);
- lowest_rq = NULL;
}
- return &lowest_rq->rt;
+ return NULL;
}
static struct rq *find_lock_lowest_rq(struct task_struct *task, struct rq *rq) {
@@ -1413,12 +1505,10 @@ static struct rq *find_lock_lowest_rq(struct task_struct *task, struct rq *rq) {
static int push_rt_rq_task(struct rt_rq *rt_rq, bool pull)
{
struct task_struct *next_task;
- struct rq *lowest_rq, *rq = rq_of_rt_rq(rt_rq);
+ struct rq *lowest_rq, *rq = global_rq_of_rt_rq(rt_rq);
struct rt_rq *lowest_rt_rq;
int ret = 0;
-
- if (is_dl_group(rt_rq))
- return 0;
+ bool dl_group;
if (!rt_rq->overloaded)
return 0;
@@ -1433,7 +1523,8 @@ static int push_rt_rq_task(struct rt_rq *rt_rq, bool pull)
* higher priority than current. If that's the case
* just reschedule current.
*/
- if (unlikely(next_task->prio < rq->donor->prio)) {
+ dl_group = rt_group_sched_enabled() && is_dl_group(rt_rq);
+ if (!dl_group && unlikely(next_task->prio < rq->donor->prio)) {
resched_curr(rq);
return 0;
}
@@ -1445,6 +1536,13 @@ static int push_rt_rq_task(struct rt_rq *rt_rq, bool pull)
if (!pull || rq->push_busy)
return 0;
+ /*
+ * If the current task does not belong to the same task group
+ * we cannot push it away.
+ */
+ if (dl_group && rq->donor->sched_task_group != rt_rq->tg)
+ return 0;
+
/*
* Invoking find_lowest_rt_rq() on anything but an RT task doesn't
* make sense. Per the above priority check, curr has to
@@ -1521,7 +1619,7 @@ static int push_rt_rq_task(struct rt_rq *rt_rq, bool pull)
goto retry;
}
- lowest_rq = rq_of_rt_rq(lowest_rt_rq);
+ lowest_rq = global_rq_of_rt_rq(lowest_rt_rq);
move_queued_task_locked(rq, lowest_rq, next_task);
resched_curr(lowest_rq);
ret = 1;
@@ -1718,16 +1816,22 @@ void rto_push_irq_work_func(struct irq_work *work)
static void pull_rt_rq_task(struct rt_rq *this_rt_rq)
{
- struct rq *this_rq = rq_of_rt_rq(this_rt_rq);
+ struct rq* this_rq = global_rq_of_rt_rq(this_rt_rq);
int this_cpu = this_rq->cpu, cpu;
bool resched = false;
- struct task_struct *p, *push_task;
+ struct task_struct *p, *push_task = NULL;
+ struct sched_dl_entity *src_dl_se;
struct rt_rq *src_rt_rq;
struct rq *src_rq;
- int rt_overload_count = rt_overloaded(this_rq);
+ int rt_overload_count;
+ const struct cpumask *cpu_mask;
+ bool dl_group;
- if (is_dl_group(&this_rq->rt))
- return;
+ dl_group = rt_group_sched_enabled() && is_dl_group(this_rt_rq);
+ if (dl_group)
+ goto group_sched;
+
+ rt_overload_count = rt_overloaded(this_rq);
if (likely(!rt_overload_count))
return;
@@ -1750,12 +1854,26 @@ static void pull_rt_rq_task(struct rt_rq *this_rt_rq)
}
#endif
- for_each_cpu(cpu, this_rq->rd->rto_mask) {
+group_sched:
+ if (!dl_group)
+ cpu_mask = this_rq->rd->rto_mask;
+ else
+ cpu_mask = cpu_online_mask;
+
+ for_each_cpu(cpu, cpu_mask) {
if (this_cpu == cpu)
continue;
src_rq = cpu_rq(cpu);
- src_rt_rq = &src_rq->rt;
+ if (!dl_group) {
+ src_rt_rq = &src_rq->rt;
+ } else {
+ src_dl_se = dl_se_of_tg(this_rt_rq->tg, cpu);
+ src_rt_rq = &src_dl_se->my_q->rt;
+
+ if (src_rt_rq->rt_nr_running <= 1 && !src_dl_se->dl_throttled)
+ continue;
+ }
/*
* Don't bother taking the src_rq->lock if the next highest
@@ -1796,12 +1914,21 @@ static void pull_rt_rq_task(struct rt_rq *this_rt_rq)
* This is just that p is waking up and hasn't
* had a chance to schedule. We only pull
* p if it is lower in priority than the
- * current task on the run queue
+ * current task on the run queue and p is
+ * in the same runqueue as donor.
*/
- if (p->prio < src_rq->donor->prio)
+ if (tg_of_se(&src_rq->donor->rt) == this_rt_rq->tg &&
+ p->prio < src_rq->donor->prio)
goto skip;
if (is_migration_disabled(p)) {
+ /*
+ * If the current task does not belong to the
+ * same task group we cannot push it away.
+ */
+ if (tg_of_se(&src_rq->donor->rt) != this_rt_rq->tg)
+ goto skip;
+
push_task = get_push_task(src_rq);
} else {
move_queued_task_locked(src_rq, this_rq, p);
diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h
index 53248cbbeaf8..3acc88a035a5 100644
--- a/kernel/sched/sched.h
+++ b/kernel/sched/sched.h
@@ -1341,6 +1341,16 @@ struct rq {
struct list_head cfsb_csd_list;
#endif
+#ifdef CONFIG_RT_GROUP_SCHED
+ /*
+ * Balance callbacks operate only on global runqueues.
+ * These pointers allow referencing cgroup specific runqueues
+ * for balancing operations.
+ */
+ struct rq *rq_to_push_from;
+ struct rq *rq_to_pull_to;
+#endif
+
atomic_t nr_iowait;
} __no_randomize_layout;
@@ -3366,6 +3376,11 @@ static inline struct rt_rq *rt_rq_of_se(struct sched_rt_entity *rt_se)
return rt_se->rt_rq;
}
+static inline struct task_group *tg_of_se(struct sched_rt_entity *rt_se)
+{
+ return rt_rq_of_se(rt_se)->tg;
+}
+
static inline int is_dl_group(struct rt_rq *rt_rq)
{
return rt_rq->tg != &root_task_group;
@@ -3382,6 +3397,9 @@ static inline struct sched_dl_entity *dl_group_of(struct rt_rq *rt_rq)
return rt_rq->tg->dl_se[rq_of_rt_rq(rt_rq)->cpu];
}
+#define rq_to_push_from(rq) ((rq)->rq_to_push_from)
+#define rq_to_pull_to(rq) ((rq)->rq_to_pull_to)
+#define dl_se_of_tg(tg, cpu) ((tg)->dl_se[(cpu)])
#define dl_bw_lock_of_tg(tg) (&(tg)->dl_bandwidth.dl_runtime_lock)
#else
static inline struct task_struct *rt_task_of(struct sched_rt_entity *rt_se)
@@ -3406,6 +3424,11 @@ static inline struct rt_rq *rt_rq_of_se(struct sched_rt_entity *rt_se)
return &rq->rt;
}
+static inline struct task_group *tg_of_se(struct sched_rt_entity *rt_se)
+{
+ return &root_task_group;
+}
+
static inline int is_dl_group(struct rt_rq *rt_rq)
{
return 0;
@@ -3416,6 +3439,9 @@ static inline struct sched_dl_entity *dl_group_of(struct rt_rq *rt_rq)
return NULL;
}
+#define rq_to_push_from(rq) (rq)
+#define rq_to_pull_to(rq) (rq)
+#define dl_se_of_tg(tg, cpu) ((struct sched_dl_entity*)NULL)
#define dl_bw_lock_of_tg(tg) ((raw_spinlock_t*)NULL)
#endif
--
2.54.0
^ permalink raw reply related
* [RFC PATCH v6 19/25] sched/rt: Remove old RT_GROUP_SCHED data structures
From: Yuri Andriaccio @ 2026-06-08 12:15 UTC (permalink / raw)
To: Ingo Molnar, Peter Zijlstra, Juri Lelli, Vincent Guittot,
Dietmar Eggemann, Steven Rostedt, Ben Segall, Mel Gorman,
Valentin Schneider, Tejun Heo, Johannes Weiner,
Michal Koutný
Cc: cgroups, linux-kernel, Luca Abeni, Yuri Andriaccio
In-Reply-To: <20260608121546.69910-1-yurand2000@gmail.com>
From: luca abeni <luca.abeni@santannapisa.it>
Completely remove the old RT_GROUP_SCHED's functions and data structures:
- Remove the fields back and my_q from sched_rt_entity.
- Remove the rt_bandwidth data structure.
- Remove the field rt_bandwidth from task_group.
- Remove the rt_bandwidth_enabled function.
- Remove the fields rt_queued, rt_throttled, rt_time, rt_runtime,
rt_runtime_lock and rt_nr_boosted from rt_rq.
All of the removed fields and data are similarly represented in previously
added fields in rq, rt_rq, dl_bandwidth and in the dl server themselves.
Co-developed-by: Yuri Andriaccio <yurand2000@gmail.com>
Signed-off-by: Yuri Andriaccio <yurand2000@gmail.com>
Signed-off-by: luca abeni <luca.abeni@santannapisa.it>
---
include/linux/sched.h | 3 ---
kernel/sched/sched.h | 33 ---------------------------------
2 files changed, 36 deletions(-)
diff --git a/include/linux/sched.h b/include/linux/sched.h
index 0021069581c2..e934ec9fc3a9 100644
--- a/include/linux/sched.h
+++ b/include/linux/sched.h
@@ -628,12 +628,9 @@ struct sched_rt_entity {
unsigned short on_rq;
unsigned short on_list;
- struct sched_rt_entity *back;
#ifdef CONFIG_RT_GROUP_SCHED
/* rq on which this entity is (to be) queued: */
struct rt_rq *rt_rq;
- /* rq "owned" by this entity/group: */
- struct rt_rq *my_q;
#endif
} __randomize_layout;
diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h
index 394f40dc26db..53248cbbeaf8 100644
--- a/kernel/sched/sched.h
+++ b/kernel/sched/sched.h
@@ -313,15 +313,6 @@ struct rt_prio_array {
struct list_head queue[MAX_RT_PRIO];
};
-struct rt_bandwidth {
- /* nests inside the rq lock: */
- raw_spinlock_t rt_runtime_lock;
- ktime_t rt_period;
- u64 rt_runtime;
- struct hrtimer rt_period_timer;
- unsigned int rt_period_active;
-};
-
struct dl_bandwidth {
raw_spinlock_t dl_runtime_lock;
u64 dl_runtime;
@@ -343,12 +334,6 @@ static inline int dl_bandwidth_enabled(void)
* - cache the fraction of bandwidth that is currently allocated in
* each root domain;
*
- * This is all done in the data structure below. It is similar to the
- * one used for RT-throttling (rt_bandwidth), with the main difference
- * that, since here we are only interested in admission control, we
- * do not decrease any runtime while the group "executes", neither we
- * need a timer to replenish it.
- *
* With respect to SMP, bandwidth is given on a per root domain basis,
* meaning that:
* - bw (< 100%) is the deadline bandwidth of each CPU;
@@ -511,11 +496,9 @@ struct task_group {
* different deadline server, and a runqueue per CPU. All the dl-servers
* share the same dl_bandwidth object.
*/
- struct sched_rt_entity **rt_se;
struct sched_dl_entity **dl_se;
struct rt_rq **rt_rq;
- struct rt_bandwidth rt_bandwidth;
struct dl_bandwidth dl_bandwidth;
#endif
@@ -842,11 +825,6 @@ struct scx_rq {
};
#endif /* CONFIG_SCHED_CLASS_EXT */
-static inline int rt_bandwidth_enabled(void)
-{
- return 0;
-}
-
/* RT IPI pull logic requires IRQ_WORK */
#if defined(CONFIG_IRQ_WORK) && defined(CONFIG_SMP)
# define HAVE_RT_PUSH_IPI
@@ -864,17 +842,6 @@ struct rt_rq {
bool overloaded;
struct plist_head pushable_tasks;
- int rt_queued;
-
-#ifdef CONFIG_RT_GROUP_SCHED
- int rt_throttled;
- u64 rt_time; /* consumed RT time, goes up in update_curr_rt */
- u64 rt_runtime; /* allotted RT time, "slice" from rt_bandwidth, RT sharing/balancing */
- /* Nests inside the rq lock: */
- raw_spinlock_t rt_runtime_lock;
-
- unsigned int rt_nr_boosted;
-#endif
#ifdef CONFIG_CGROUP_SCHED
struct task_group *tg; /* this tg has "this" rt_rq on given CPU for runnable entities */
#endif
--
2.54.0
^ permalink raw reply related
* [RFC PATCH v6 17/25] sched/rt: Update rt-cgroup schedulability checks
From: Yuri Andriaccio @ 2026-06-08 12:15 UTC (permalink / raw)
To: Ingo Molnar, Peter Zijlstra, Juri Lelli, Vincent Guittot,
Dietmar Eggemann, Steven Rostedt, Ben Segall, Mel Gorman,
Valentin Schneider, Tejun Heo, Johannes Weiner,
Michal Koutný
Cc: cgroups, linux-kernel, Luca Abeni, Yuri Andriaccio
In-Reply-To: <20260608121546.69910-1-yurand2000@gmail.com>
From: luca abeni <luca.abeni@santannapisa.it>
Introduce cgroup-v2 control files:
- cpu.rt.max:
Get/set the bandwidth of the given cgroup, or inherith from parent.
- cpu.rt.internal:
Get the actual remaning bandwidth for the group, removing the bw of the
group's children.
Introduce a number of functions to update the cgroup settings across the
whole hierarchy:
- tg_subtree_has_rt_tasks()
Checks if the active context rooted at tg is running rt workload.
Child groups which do not share the same active context are ignored.
- tg_compute_children_bw()
Computes the total bandwidth of the active context rooted at tg minux
the root of the context itself.
- tg_rt_schedulable()
Runs admission tests for the current cgroup tree and the given
bandwidth update.
- tg_update_active_context()
Updates the active context of a given subtree with a new one.
- tg_rt_bandwidth() / tg_rt_internal_bandwidth()
Read the max (internal) bandwidth set to the cgroup.
- tg_set_rt_bandwidth()
Set the bandwidth of the group.
Update sched_rt_can_attach to run only tasks in the root cgroup or HCBS
cgroups which have non-zero runtime.
Update and reuse __checkparam_dl to check for numerical issues regarding
the dl_server's parameters.
Add from_ratio function to convert from period and bw to runtime, inverse
of the to_ratio function.
Add dl_check_tg(), which performs an admission control test similar to
__dl_overflow, but this time we are updating the cgroup's total bandwidth
rather than scheduling a new DEADLINE task or updating a non-cgroup
deadline server.
Add rcu_sched lock guard for rcu_read_{lock/unlock}_sched.
Add sched_domains lock guard for sched_domains_mutex_{lock/unlock}.
Add lock/unlock methods for sched_rt_handler_mutex and its lock guard.
Add asserts for held sched_domains_mutex and sched_rt_handler_mutex.
Co-developed-by: Alessio Balsini <a.balsini@sssup.it>
Signed-off-by: Alessio Balsini <a.balsini@sssup.it>
Co-developed-by: Andrea Parri <parri.andrea@gmail.com>
Signed-off-by: Andrea Parri <parri.andrea@gmail.com>
Co-developed-by: luca abeni <luca.abeni@santannapisa.it>
Signed-off-by: luca abeni <luca.abeni@santannapisa.it>
Signed-off-by: Yuri Andriaccio <yurand2000@gmail.com>
---
include/linux/rcupdate.h | 1 +
include/linux/sched.h | 2 +
kernel/sched/core.c | 55 ++++++
kernel/sched/deadline.c | 60 ++++--
kernel/sched/rt.c | 393 +++++++++++++++++++++++++++++++--------
kernel/sched/sched.h | 18 +-
kernel/sched/syscalls.c | 2 +-
7 files changed, 445 insertions(+), 86 deletions(-)
diff --git a/include/linux/rcupdate.h b/include/linux/rcupdate.h
index bfa765132de8..70432ca3dbb9 100644
--- a/include/linux/rcupdate.h
+++ b/include/linux/rcupdate.h
@@ -1179,6 +1179,7 @@ extern int rcu_expedited;
extern int rcu_normal;
DEFINE_LOCK_GUARD_0(rcu, rcu_read_lock(), rcu_read_unlock())
+DEFINE_LOCK_GUARD_0(rcu_sched, rcu_read_lock_sched(), rcu_read_unlock_sched())
DECLARE_LOCK_GUARD_0_ATTRS(rcu, __acquires_shared(RCU), __releases_shared(RCU))
#endif /* __LINUX_RCUPDATE_H */
diff --git a/include/linux/sched.h b/include/linux/sched.h
index b20451fcda55..0021069581c2 100644
--- a/include/linux/sched.h
+++ b/include/linux/sched.h
@@ -2522,4 +2522,6 @@ extern void migrate_enable(void);
DEFINE_LOCK_GUARD_0(migrate, migrate_disable(), migrate_enable())
+DEFINE_LOCK_GUARD_0(sched_domains, sched_domains_mutex_lock(), sched_domains_mutex_unlock())
+
#endif
diff --git a/kernel/sched/core.c b/kernel/sched/core.c
index a8a81c69b3d3..1ad1efe1dca7 100644
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -4815,6 +4815,14 @@ u64 to_ratio(u64 period, u64 runtime)
return div64_u64(runtime << BW_SHIFT, period);
}
+u64 from_ratio(u64 period, u64 bw)
+{
+ if (bw == BW_UNIT)
+ return RUNTIME_INF;
+
+ return (bw * period) >> BW_SHIFT;
+}
+
/*
* wake_up_new_task - wake up a newly created task for the first time.
*
@@ -10415,6 +10423,41 @@ static ssize_t cpu_max_write(struct kernfs_open_file *of,
}
#endif /* CONFIG_CFS_BANDWIDTH */
+#ifdef CONFIG_RT_GROUP_SCHED
+static int cpu_rt_max_show(struct seq_file *sf, void *v)
+{
+ struct task_group *tg = css_tg(seq_css(sf));
+ long period_us, runtime_us;
+
+ tg_rt_bandwidth(tg, &period_us, &runtime_us);
+ cpu_period_quota_print(sf, period_us, runtime_us);
+ return 0;
+}
+
+static int cpu_rt_internal_show(struct seq_file *sf, void *v)
+{
+ struct task_group *tg = css_tg(seq_css(sf));
+ long period_us, runtime_us;
+
+ tg_rt_internal_bandwidth(tg, &period_us, &runtime_us);
+ cpu_period_quota_print(sf, period_us, runtime_us);
+ return 0;
+}
+
+static ssize_t cpu_rt_max_write(struct kernfs_open_file *of,
+ char *buf, size_t nbytes, loff_t off)
+{
+ struct task_group *tg = css_tg(of_css(of));
+ u64 period_us, runtime_us;
+ int ret;
+
+ ret = cpu_period_quota_parse(buf, &period_us, &runtime_us);
+ if (!ret)
+ ret = tg_set_rt_bandwidth(tg, period_us, runtime_us);
+ return ret ?: nbytes;
+}
+#endif /* CONFIG_RT_GROUP_SCHED */
+
static struct cftype cpu_files[] = {
#ifdef CONFIG_GROUP_SCHED_WEIGHT
{
@@ -10450,6 +10493,18 @@ static struct cftype cpu_files[] = {
.write_u64 = cpu_burst_write_u64,
},
#endif /* CONFIG_CFS_BANDWIDTH */
+#ifdef CONFIG_RT_GROUP_SCHED
+ {
+ .name = "rt.max",
+ .seq_show = cpu_rt_max_show,
+ .write = cpu_rt_max_write,
+ },
+ {
+ .name = "rt.internal",
+ .flags = CFTYPE_NOT_ON_ROOT,
+ .seq_show = cpu_rt_internal_show,
+ },
+#endif /* CONFIG_RT_GROUP_SCHED */
#ifdef CONFIG_UCLAMP_TASK_GROUP
{
.name = "uclamp.min",
diff --git a/kernel/sched/deadline.c b/kernel/sched/deadline.c
index a63253ec6441..b7102f643171 100644
--- a/kernel/sched/deadline.c
+++ b/kernel/sched/deadline.c
@@ -346,10 +346,45 @@ void cancel_inactive_timer(struct sched_dl_entity *dl_se)
cancel_dl_timer(dl_se, &dl_se->inactive_timer);
}
+/*
+ * Used for dl_bw check and update, used under sched_rt_handler()::mutex and
+ * sched_domains_mutex.
+ */
+u64 dl_cookie;
+
#ifdef CONFIG_RT_GROUP_SCHED
+int dl_check_tg(unsigned long total)
+{
+ int which_cpu;
+ int cap;
+ struct dl_bw *dl_b;
+ u64 gen = ++dl_cookie;
+
+ lockdep_assert_held(&sched_domains_mutex);
+ lockdep_assert_held(&sched_rt_handler_mutex);
+
+ for_each_possible_cpu(which_cpu) {
+ guard(rcu_sched)();
+
+ if (!dl_bw_visited(which_cpu, gen)) {
+ cap = dl_bw_capacity(which_cpu);
+ dl_b = dl_bw_of(which_cpu);
+
+ guard(raw_spinlock_irqsave)(&dl_b->lock);
+
+ if (dl_b->bw != -1 &&
+ cap_scale(dl_b->bw, cap) < dl_b->total_bw + cap_scale(total, cap))
+ return 0;
+ }
+
+ }
+
+ return 1;
+}
+
void dl_init_tg(struct sched_dl_entity *dl_se, u64 rt_runtime, u64 rt_period)
{
- struct rq *rq = container_of_const(dl_se->dl_rq, struct rq, dl);
+ struct rq *rq = rq_of_dl_se(dl_se);
int is_active;
u64 new_bw;
@@ -3497,12 +3532,6 @@ DEFINE_SCHED_CLASS(dl) = {
#endif
};
-/*
- * Used for dl_bw check and update, used under sched_rt_handler()::mutex and
- * sched_domains_mutex.
- */
-u64 dl_cookie;
-
int sched_dl_global_validate(void)
{
u64 runtime = global_rt_runtime();
@@ -3514,6 +3543,9 @@ int sched_dl_global_validate(void)
int cpu, cap, cpus, ret = 0;
unsigned long flags;
+ lockdep_assert_held(&sched_domains_mutex);
+ lockdep_assert_held(&sched_rt_handler_mutex);
+
/*
* Here we want to check the bandwidth not being set to some
* value smaller than the currently allocated bandwidth in
@@ -3566,6 +3598,9 @@ void sched_dl_do_global(void)
int cpu;
unsigned long flags;
+ lockdep_assert_held(&sched_domains_mutex);
+ lockdep_assert_held(&sched_rt_handler_mutex);
+
if (global_rt_runtime() != RUNTIME_INF)
new_bw = to_ratio(global_rt_period(), global_rt_runtime());
@@ -3711,7 +3746,7 @@ void __getparam_dl(struct task_struct *p, struct sched_attr *attr, unsigned int
* below 2^63 ns (we have to check both sched_deadline and
* sched_period, as the latter can be zero).
*/
-bool __checkparam_dl(const struct sched_attr *attr)
+bool __checkparam_dl(const struct sched_attr *attr, bool allow_zero_runtime)
{
u64 period, max, min;
@@ -3720,14 +3755,16 @@ bool __checkparam_dl(const struct sched_attr *attr)
return true;
/* deadline != 0 */
- if (attr->sched_deadline == 0)
+ if ((!allow_zero_runtime || attr->sched_runtime != 0) &&
+ attr->sched_deadline == 0)
return false;
/*
* Since we truncate DL_SCALE bits, make sure we're at least
* that big.
*/
- if (attr->sched_runtime < (1ULL << DL_SCALE))
+ if ((!allow_zero_runtime || attr->sched_runtime != 0) &&
+ attr->sched_runtime < (1ULL << DL_SCALE))
return false;
/*
@@ -3750,7 +3787,8 @@ bool __checkparam_dl(const struct sched_attr *attr)
max = (u64)READ_ONCE(sysctl_sched_dl_period_max) * NSEC_PER_USEC;
min = (u64)READ_ONCE(sysctl_sched_dl_period_min) * NSEC_PER_USEC;
- if (period < min || period > max)
+ if ((!allow_zero_runtime || period != 0) &&
+ (period < min || period > max))
return false;
return true;
diff --git a/kernel/sched/rt.c b/kernel/sched/rt.c
index 4f1e7af2e88d..a32b1f68e645 100644
--- a/kernel/sched/rt.c
+++ b/kernel/sched/rt.c
@@ -1,4 +1,3 @@
-#pragma GCC diagnostic ignored "-Wunused-function"
// SPDX-License-Identifier: GPL-2.0
/*
* Real-Time Scheduling Class (mapped to the SCHED_FIFO and SCHED_RR
@@ -2111,9 +2110,6 @@ DEFINE_SCHED_CLASS(rt) = {
};
#ifdef CONFIG_RT_GROUP_SCHED
-/*
- * Ensure that the real time constraints are schedulable.
- */
static inline int tg_has_rt_tasks(struct task_group *tg)
{
struct task_struct *task;
@@ -2134,38 +2130,114 @@ static inline int tg_has_rt_tasks(struct task_group *tg)
return ret;
}
-struct rt_schedulable_data {
+static int __tg_subtree_has_rt_tasks(struct task_group *tg, void *data) {
+ struct task_group *ctx = data;
+
+ if (dl_bandwidth_read(tg)->active_context == ctx && tg_has_rt_tasks(tg))
+ return 1;
+ else
+ return 0;
+}
+
+static int tg_subtree_has_rt_tasks(struct task_group *tg) {
+ lockdep_assert(rcu_read_lock_held());
+ return walk_tg_tree_from(tg, __tg_subtree_has_rt_tasks, tg_nop,
+ dl_bandwidth_read(tg)->active_context);
+}
+
+struct tg_update_data {
struct task_group *tg;
u64 rt_period;
u64 rt_runtime;
};
-static int tg_rt_schedulable(struct task_group *tg, void *data)
+struct tg_compute_children_bw_data {
+ struct tg_update_data update;
+ struct task_group *active_context;
+ u64 bw_sum;
+};
+
+static int __tg_compute_children_bw(struct task_group *tg, void *data) {
+ struct tg_compute_children_bw_data *d = data;
+ const struct dl_bandwidth *dl_b = dl_bandwidth_read(tg);
+ u64 period, runtime;
+
+ /* Skip the current task group from the sum. */
+ if (tg == d->active_context)
+ return 0;
+
+ period = dl_b->dl_period;
+ runtime = dl_b->dl_runtime;
+ if (tg == d->update.tg) {
+ period = d->update.rt_period;
+ runtime = d->update.rt_runtime;
+ }
+
+ if (runtime == RUNTIME_INF ||
+ dl_bandwidth_read(tg->parent)->active_context != d->active_context)
+ return 0;
+
+ d->bw_sum += to_ratio(period, runtime);
+ return 0;
+}
+
+static unsigned long tg_compute_children_bw(struct task_group *tg,
+ struct tg_update_data *data)
+{
+ struct tg_compute_children_bw_data sum_data = {
+ .active_context = tg,
+ .bw_sum = 0,
+ .update = (struct tg_update_data) {
+ .tg = data->tg,
+ .rt_period = data->rt_period,
+ .rt_runtime = data->rt_runtime,
+ }
+ };
+
+ lockdep_assert(rcu_read_lock_held());
+ walk_tg_tree_from(tg, __tg_compute_children_bw, tg_nop, &sum_data);
+ return sum_data.bw_sum;
+}
+
+struct rt_schedulable_data {
+ struct tg_update_data update;
+ u64 rt_runtime_remainder;
+};
+
+static int __tg_rt_schedulable(struct task_group *tg, void *data)
{
struct rt_schedulable_data *d = data;
- struct task_group *child;
+ const struct dl_bandwidth *dl_b;
u64 total, sum = 0;
u64 period, runtime;
- period = ktime_to_ns(tg->rt_bandwidth.rt_period);
- runtime = tg->rt_bandwidth.rt_runtime;
+ dl_b = dl_bandwidth_read(tg);
+ period = dl_b->dl_period;
+ runtime = dl_b->dl_runtime;
- if (tg == d->tg) {
- period = d->rt_period;
- runtime = d->rt_runtime;
+ if (tg == d->update.tg) {
+ period = d->update.rt_period;
+ runtime = d->update.rt_runtime;
}
+ /*
+ * "max" groups are always schedulable, as they defer their access
+ * control to their first non-max parent.
+ */
+ if (runtime == RUNTIME_INF)
+ return 0;
+
/*
* Cannot have more runtime than the period.
*/
- if (runtime > period && runtime != RUNTIME_INF)
+ if (runtime > period)
return -EINVAL;
/*
* Ensure we don't starve existing RT tasks if runtime turns zero.
*/
- if (rt_bandwidth_enabled() && !runtime &&
- tg->rt_bandwidth.rt_runtime && tg_has_rt_tasks(tg))
+ if (dl_bandwidth_enabled() && !runtime && tg != &root_task_group &&
+ tg_subtree_has_rt_tasks(tg))
return -EBUSY;
total = to_ratio(period, runtime);
@@ -2176,58 +2248,146 @@ static int tg_rt_schedulable(struct task_group *tg, void *data)
if (total > to_ratio(global_rt_period(), global_rt_runtime()))
return -EINVAL;
+ if (tg == &root_task_group) {
+ if (!dl_check_tg(total))
+ return -EBUSY;
+ }
+
/*
- * The sum of our children's runtime should not exceed our own.
+ * The sum of our children's runtime, plus our own bw, should not
+ * exceed our own max.
*/
- list_for_each_entry_rcu(child, &tg->children, siblings) {
- period = ktime_to_ns(child->rt_bandwidth.rt_period);
- runtime = child->rt_bandwidth.rt_runtime;
+ sum = tg_compute_children_bw(tg, &d->update);
+ if (sum > total)
+ return -EINVAL;
- if (child == d->tg) {
- period = d->rt_period;
- runtime = d->rt_runtime;
- }
+ /*
+ * Compute remaining runtime
+ */
+ if (tg == d->update.tg)
+ d->rt_runtime_remainder = from_ratio(period, total - sum);
+
+ return 0;
+}
- sum += to_ratio(period, runtime);
+static int tg_rt_schedulable(struct tg_update_data *data, u64 *remainder_runtime)
+{
+ int err;
+ struct rt_schedulable_data d = {
+ .update = (struct tg_update_data) {
+ .tg = data->tg,
+ .rt_period = data->rt_period,
+ .rt_runtime = data->rt_runtime,
+ },
+ .rt_runtime_remainder = 0,
+ };
+
+ /*
+ * Walk the cgroup tree and check schedulability constraints.
+ */
+ lockdep_assert(rcu_read_lock_held());
+ err = walk_tg_tree(__tg_rt_schedulable, tg_nop, &d);
+ if (err)
+ return err;
+
+ *remainder_runtime = d.rt_runtime_remainder;
+ return 0;
+}
+
+struct tg_update_active_context_data {
+ struct task_group *new_active_context;
+ struct task_group *old_active_context;
+};
+
+static int __tg_update_active_context(struct task_group *tg, void *data) {
+ struct tg_update_active_context_data *d = data;
+
+ if (dl_bandwidth_read(tg)->active_context == d->old_active_context) {
+ guard(raw_spinlock_irq)(dl_bw_lock_of_tg(tg));
+ dl_bandwidth_write(tg)->active_context = d->new_active_context;
}
- if (sum > total)
- return -EINVAL;
+ return 0;
+}
+
+static void tg_update_active_context(struct task_group *tg,
+ struct task_group *old_context,
+ struct task_group *new_context)
+{
+ struct tg_update_active_context_data data = {
+ .new_active_context = new_context,
+ .old_active_context = old_context,
+ };
+ lockdep_assert(rcu_read_lock_held());
+ walk_tg_tree_from(tg, __tg_update_active_context, tg_nop, &data);
+}
+
+int tg_rt_bandwidth(struct task_group *tg,
+ long *rt_period_us, long *rt_runtime_us)
+{
+ const struct dl_bandwidth *dl_b;
+
+ guard(raw_spinlock_irq)(dl_bw_lock_of_tg(tg));
+ dl_b = dl_bandwidth_read(tg);
+
+ *rt_runtime_us = -1;
+ if (dl_b->dl_runtime != RUNTIME_INF) {
+ *rt_runtime_us = dl_b->dl_runtime;
+ do_div(*rt_runtime_us, NSEC_PER_USEC);
+ }
+
+ *rt_period_us = dl_b->dl_period;
+ do_div(*rt_period_us, NSEC_PER_USEC);
return 0;
}
-static int __rt_schedulable(struct task_group *tg, u64 period, u64 runtime)
+int tg_rt_internal_bandwidth(struct task_group *tg,
+ long *rt_period_us, long *rt_runtime_us)
{
- int ret;
+ const struct dl_bandwidth *dl_b;
- struct rt_schedulable_data data = {
- .tg = tg,
- .rt_period = period,
- .rt_runtime = runtime,
- };
+ guard(raw_spinlock_irq)(dl_bw_lock_of_tg(tg));
+ dl_b = dl_bandwidth_read(tg);
- rcu_read_lock();
- ret = walk_tg_tree(tg_rt_schedulable, tg_nop, &data);
- rcu_read_unlock();
+ *rt_runtime_us = dl_b->dl_internal_runtime;
+ do_div(*rt_runtime_us, NSEC_PER_USEC);
- return ret;
+ *rt_period_us = dl_b->dl_period;
+ do_div(*rt_period_us, NSEC_PER_USEC);
+
+ return 0;
}
-static int tg_set_rt_bandwidth(struct task_group *tg,
- u64 rt_period, u64 rt_runtime)
+int tg_set_rt_bandwidth(struct task_group *tg,
+ u64 rt_period_us, u64 rt_runtime_us)
{
- int i, err = 0;
+ struct tg_update_data update;
+ struct task_group *parent_ctx;
+ struct dl_bandwidth *dl_b;
+ u64 rt_period, rt_runtime, old_rt_runtime;
+ u64 rt_actual_runtime = 0;
+ u64 bw, children_bw;
+ struct sched_attr attr;
+ int err, i;
- /*
- * Disallowing the root group RT runtime is BAD, it would disallow the
- * kernel creating (and or operating) RT threads.
- */
- if (tg == &root_task_group && rt_runtime == 0)
+ if (rt_runtime_us == RUNTIME_INF)
+ rt_runtime = RUNTIME_INF;
+ else if ((u64)rt_runtime_us > U64_MAX / NSEC_PER_USEC)
return -EINVAL;
+ else
+ rt_runtime = (u64)rt_runtime_us * NSEC_PER_USEC;
- /* No period doesn't make any sense. */
- if (rt_period == 0)
+ if ((u64)rt_period_us > U64_MAX / NSEC_PER_USEC)
+ return -EINVAL;
+ else
+ rt_period = (u64)rt_period_us * NSEC_PER_USEC;
+
+ /*
+ * The root_task_group bandwidth settings are only used to reserve bw
+ * for HCBS cgroups; runtime == "max" has no meaning there.
+ */
+ if (rt_runtime == RUNTIME_INF && tg == &root_task_group)
return -EINVAL;
/*
@@ -2236,34 +2396,119 @@ static int tg_set_rt_bandwidth(struct task_group *tg,
if (rt_runtime != RUNTIME_INF && rt_runtime > max_rt_runtime)
return -EINVAL;
- mutex_lock(&rt_constraints_mutex);
- err = __rt_schedulable(tg, rt_period, rt_runtime);
+ /*
+ * Check if the runtime and period min and max values are admissible.
+ */
+ attr = (struct sched_attr){
+ .sched_flags = 0,
+ .sched_runtime = rt_runtime,
+ .sched_deadline = rt_period,
+ .sched_period = rt_period,
+ };
+
+ if (rt_runtime != RUNTIME_INF && !__checkparam_dl(&attr, true))
+ return -EINVAL;
+
+ update = (struct tg_update_data) {
+ .tg = tg,
+ .rt_period = rt_period,
+ .rt_runtime = rt_runtime,
+ };
+
+ guard(mutex)(&rt_constraints_mutex);
+ old_rt_runtime = dl_bandwidth_read(tg)->dl_runtime;
+
+ /*
+ * Disallow changing from/to "max" and a HCBS reservation if the group
+ * and all of its "max" children have active tasks.
+ */
+ guard(sched_rt_handler)();
+ guard(sched_domains)();
+ guard(rcu)();
+ if (((rt_runtime == RUNTIME_INF && old_rt_runtime != RUNTIME_INF) ||
+ (rt_runtime != RUNTIME_INF && old_rt_runtime == RUNTIME_INF)) &&
+ tg_subtree_has_rt_tasks(tg))
+ return -EINVAL;
+
+ err = tg_rt_schedulable(&update, &rt_actual_runtime);
if (err)
- goto unlock;
+ return err;
- raw_spin_lock_irq(&tg->rt_bandwidth.rt_runtime_lock);
- tg->rt_bandwidth.rt_period = ns_to_ktime(rt_period);
- tg->rt_bandwidth.rt_runtime = rt_runtime;
+ scoped_guard(raw_spinlock_irq, dl_bw_lock_of_tg(tg)) {
+ dl_b = dl_bandwidth_write(tg);
+ dl_b->dl_period = rt_period;
+ dl_b->dl_runtime = rt_runtime;
+ dl_b->dl_internal_runtime = rt_actual_runtime;
+ }
+
+ if (tg == &root_task_group)
+ return 0;
+ parent_ctx = dl_bandwidth_read(tg->parent)->active_context;
+
+ /*
+ * If changing from/to "max" and a HCBS reservation, must update the
+ * active_context of self and all of its subtree.
+ */
+ if ((rt_runtime == RUNTIME_INF && old_rt_runtime != RUNTIME_INF) ||
+ (rt_runtime != RUNTIME_INF && old_rt_runtime == RUNTIME_INF))
+ {
+ if (rt_runtime == RUNTIME_INF)
+ tg_update_active_context(tg, dl_b->active_context, parent_ctx);
+ else
+ tg_update_active_context(tg, dl_b->active_context, tg);
+
+ }
+
+ WARN_ON(rt_runtime == RUNTIME_INF && rt_actual_runtime != 0);
for_each_possible_cpu(i) {
- struct rt_rq *rt_rq = tg->rt_rq[i];
+ dl_init_tg(tg->dl_se[i], rt_actual_runtime, rt_period);
+ }
+
+ /*
+ * Update the dl_servers of the parent's active context
+ */
+ if (parent_ctx == &root_task_group)
+ return 0;
+
+ scoped_guard(raw_spinlock_irq, dl_bw_lock_of_tg(parent_ctx)) {
+ dl_b = dl_bandwidth_write(parent_ctx);
- raw_spin_lock(&rt_rq->rt_runtime_lock);
- rt_rq->rt_runtime = rt_runtime;
- raw_spin_unlock(&rt_rq->rt_runtime_lock);
+ bw = to_ratio(dl_b->dl_period, dl_b->dl_runtime);
+ children_bw = tg_compute_children_bw(parent_ctx, &update);
+
+ rt_period = dl_b->dl_period;
+ rt_actual_runtime = from_ratio(rt_period, bw - children_bw);
+ dl_b->dl_internal_runtime = rt_actual_runtime;
}
- raw_spin_unlock_irq(&tg->rt_bandwidth.rt_runtime_lock);
-unlock:
- mutex_unlock(&rt_constraints_mutex);
- return err;
+ for_each_possible_cpu(i) {
+ dl_init_tg(parent_ctx->dl_se[i], rt_actual_runtime, rt_period);
+ }
+
+ return 0;
}
int sched_rt_can_attach(struct task_group *tg)
{
+ struct task_group *ctx;
+
+ /* If rt group sched is disabled, tasks are always run in the root rq */
+ if (!rt_group_sched_enabled())
+ return 1;
+
+ /* Can always run on the root task group */
+ scoped_guard(raw_spinlock_irqsave, dl_bw_lock_of_tg(tg)) {
+ ctx = dl_bandwidth_read(tg)->active_context;
+ if (ctx == &root_task_group)
+ return 1;
+ }
+
/* Don't accept real-time tasks when there is no way for them to run */
- if (rt_group_sched_enabled() && tg->dl_bandwidth.dl_runtime == 0)
- return 0;
+ scoped_guard(raw_spinlock_irqsave, dl_bw_lock_of_tg(ctx)) {
+ if (dl_bandwidth_read(ctx)->dl_runtime == 0)
+ return 0;
+ }
return 1;
}
@@ -2279,24 +2524,26 @@ static int sched_rt_global_validate(void)
NSEC_PER_USEC > max_rt_runtime)))
return -EINVAL;
-#ifdef CONFIG_RT_GROUP_SCHED
- if (!rt_group_sched_enabled())
- return 0;
-
- scoped_guard(mutex, &rt_constraints_mutex)
- return __rt_schedulable(NULL, 0, 0);
-#endif
return 0;
}
+DEFINE_MUTEX(sched_rt_handler_mutex);
+
+void sched_rt_handler_mutex_lock() {
+ mutex_lock(&sched_rt_handler_mutex);
+}
+
+void sched_rt_handler_mutex_unlock() {
+ mutex_unlock(&sched_rt_handler_mutex);
+}
+
static int sched_rt_handler(const struct ctl_table *table, int write, void *buffer,
size_t *lenp, loff_t *ppos)
{
int old_period, old_runtime;
- static DEFINE_MUTEX(mutex);
int ret;
- mutex_lock(&mutex);
+ sched_rt_handler_mutex_lock();
sched_domains_mutex_lock();
old_period = sysctl_sched_rt_period;
old_runtime = sysctl_sched_rt_runtime;
@@ -2320,7 +2567,7 @@ static int sched_rt_handler(const struct ctl_table *table, int write, void *buff
sysctl_sched_rt_runtime = old_runtime;
}
sched_domains_mutex_unlock();
- mutex_unlock(&mutex);
+ sched_rt_handler_mutex_unlock();
/*
* After changing maximum available bandwidth for DEADLINE, we need to
diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h
index efe52e162ba5..394f40dc26db 100644
--- a/kernel/sched/sched.h
+++ b/kernel/sched/sched.h
@@ -366,7 +366,7 @@ extern void sched_dl_do_global(void);
extern int sched_dl_overflow(struct task_struct *p, int policy, const struct sched_attr *attr);
extern void __setparam_dl(struct task_struct *p, const struct sched_attr *attr);
extern void __getparam_dl(struct task_struct *p, struct sched_attr *attr, unsigned int flags);
-extern bool __checkparam_dl(const struct sched_attr *attr);
+extern bool __checkparam_dl(const struct sched_attr *attr, bool allow_zero_runtime);
extern bool dl_param_changed(struct task_struct *p, const struct sched_attr *attr);
extern int dl_cpuset_cpumask_can_shrink(const struct cpumask *cur, const struct cpumask *trial);
extern int dl_bw_deactivate(int cpu);
@@ -425,6 +425,7 @@ extern void dl_server_init(struct sched_dl_entity *dl_se, struct dl_rq *dl_rq,
struct rq *served_rq,
dl_server_pick_f pick_task);
extern void sched_init_dl_servers(void);
+extern int dl_check_tg(unsigned long total);
extern void dl_init_tg(struct sched_dl_entity *dl_se, u64 rt_runtime, u64 rt_period);
extern void fair_server_init(struct rq *rq);
@@ -607,6 +608,12 @@ extern void start_cfs_bandwidth(struct cfs_bandwidth *cfs_b);
extern void unthrottle_cfs_rq(struct cfs_rq *cfs_rq);
extern bool cfs_task_bw_constrained(struct task_struct *p);
+extern int tg_rt_bandwidth(struct task_group *tg,
+ long *rt_period_us, long *rt_runtime_us);
+extern int tg_rt_internal_bandwidth(struct task_group *tg,
+ long *rt_period_us, long *rt_runtime_us);
+extern int tg_set_rt_bandwidth(struct task_group *tg,
+ u64 rt_period_us, u64 rt_runtime_us);
extern int sched_rt_can_attach(struct task_group *tg);
extern struct task_group *sched_create_group(struct task_group *parent);
@@ -2045,6 +2052,14 @@ DEFINE_LOCK_GUARD_1(raw_spin_rq_lock_irq, struct rq,
raw_spin_rq_lock_irq(_T->lock),
raw_spin_rq_unlock_irq(_T->lock))
+extern struct mutex sched_rt_handler_mutex;
+extern void sched_rt_handler_mutex_lock(void);
+extern void sched_rt_handler_mutex_unlock(void);
+
+DEFINE_LOCK_GUARD_0(sched_rt_handler,
+ sched_rt_handler_mutex_lock(),
+ sched_rt_handler_mutex_unlock())
+
#ifdef CONFIG_NUMA
enum numa_topology_type {
@@ -2938,6 +2953,7 @@ extern void init_cfs_throttle_work(struct task_struct *p);
#define MAX_BW ((1ULL << MAX_BW_BITS) - 1)
extern u64 to_ratio(u64 period, u64 runtime);
+extern u64 from_ratio(u64 period, u64 bw);
extern void init_entity_runnable_average(struct sched_entity *se);
extern void post_init_entity_util_avg(struct task_struct *p);
diff --git a/kernel/sched/syscalls.c b/kernel/sched/syscalls.c
index 773f744c0460..e5b8d2f42ea8 100644
--- a/kernel/sched/syscalls.c
+++ b/kernel/sched/syscalls.c
@@ -528,7 +528,7 @@ int __sched_setscheduler(struct task_struct *p,
*/
if (attr->sched_priority > MAX_RT_PRIO-1)
return -EINVAL;
- if ((dl_policy(policy) && !__checkparam_dl(attr)) ||
+ if ((dl_policy(policy) && !__checkparam_dl(attr, false)) ||
(rt_policy(policy) != (attr->sched_priority != 0)))
return -EINVAL;
--
2.54.0
^ permalink raw reply related
* [RFC PATCH v6 18/25] sched/rt: Update task's RT runqueue when switching scheduling class
From: Yuri Andriaccio @ 2026-06-08 12:15 UTC (permalink / raw)
To: Ingo Molnar, Peter Zijlstra, Juri Lelli, Vincent Guittot,
Dietmar Eggemann, Steven Rostedt, Ben Segall, Mel Gorman,
Valentin Schneider, Tejun Heo, Johannes Weiner,
Michal Koutný
Cc: cgroups, linux-kernel, Luca Abeni, Yuri Andriaccio
In-Reply-To: <20260608121546.69910-1-yurand2000@gmail.com>
Signed-off-by: Yuri Andriaccio <yurand2000@gmail.com>
---
kernel/sched/rt.c | 20 ++++++++++++++++++++
1 file changed, 20 insertions(+)
diff --git a/kernel/sched/rt.c b/kernel/sched/rt.c
index a32b1f68e645..fc7af6bda3f8 100644
--- a/kernel/sched/rt.c
+++ b/kernel/sched/rt.c
@@ -1897,6 +1897,25 @@ void __init init_sched_rt_class(void)
}
}
+#ifdef CONFIG_RT_GROUP_SCHED
+static void switching_to_rt(struct rq *rq, struct task_struct *p)
+{
+ struct task_group *tg = p->sched_task_group;
+ int cpu = rq->cpu;
+
+ if (tg == &root_task_group)
+ return;
+
+ guard(raw_spinlock_irqsave)(dl_bw_lock_of_tg(tg));
+ if (!rt_group_sched_enabled())
+ tg = &root_task_group;
+
+ p->rt.rt_rq = dl_bandwidth_read(tg)->active_context->rt_rq[cpu];
+}
+#else
+static void switching_to_rt(struct rq *rq, struct task_struct *p) {}
+#endif
+
/*
* When switching a task to RT, we may overload the runqueue
* with RT tasks. In this case we try to push them off to
@@ -2095,6 +2114,7 @@ DEFINE_SCHED_CLASS(rt) = {
.get_rr_interval = get_rr_interval_rt,
+ .switching_to = switching_to_rt,
.switched_to = switched_to_rt,
.prio_changed = prio_changed_rt,
--
2.54.0
^ permalink raw reply related
* [RFC PATCH v6 16/25] sched/rt: Remove support for cgroups-v1
From: Yuri Andriaccio @ 2026-06-08 12:15 UTC (permalink / raw)
To: Ingo Molnar, Peter Zijlstra, Juri Lelli, Vincent Guittot,
Dietmar Eggemann, Steven Rostedt, Ben Segall, Mel Gorman,
Valentin Schneider, Tejun Heo, Johannes Weiner,
Michal Koutný
Cc: cgroups, linux-kernel, Luca Abeni, Yuri Andriaccio
In-Reply-To: <20260608121546.69910-1-yurand2000@gmail.com>
Disable control files for cgroups-v1.
Remove cpu_rt_group_init function and functions related to the
cgroup-v1 control files 'rt_runtime_us' and 'rt_period_us'.
Signed-off-by: Yuri Andriaccio <yurand2000@gmail.com>
---
kernel/sched/core.c | 50 --------------------------------------------
kernel/sched/rt.c | 49 +------------------------------------------
kernel/sched/sched.h | 4 ----
3 files changed, 1 insertion(+), 102 deletions(-)
diff --git a/kernel/sched/core.c b/kernel/sched/core.c
index 1252f45feda0..a8a81c69b3d3 100644
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -10150,32 +10150,6 @@ static int cpu_burst_write_u64(struct cgroup_subsys_state *css,
}
#endif /* CONFIG_GROUP_SCHED_BANDWIDTH */
-#ifdef CONFIG_RT_GROUP_SCHED
-static int cpu_rt_runtime_write(struct cgroup_subsys_state *css,
- struct cftype *cft, s64 val)
-{
- return sched_group_set_rt_runtime(css_tg(css), val);
-}
-
-static s64 cpu_rt_runtime_read(struct cgroup_subsys_state *css,
- struct cftype *cft)
-{
- return sched_group_rt_runtime(css_tg(css));
-}
-
-static int cpu_rt_period_write_uint(struct cgroup_subsys_state *css,
- struct cftype *cftype, u64 rt_period_us)
-{
- return sched_group_set_rt_period(css_tg(css), rt_period_us);
-}
-
-static u64 cpu_rt_period_read_uint(struct cgroup_subsys_state *css,
- struct cftype *cft)
-{
- return sched_group_rt_period(css_tg(css));
-}
-#endif /* CONFIG_RT_GROUP_SCHED */
-
#ifdef CONFIG_GROUP_SCHED_WEIGHT
static s64 cpu_idle_read_s64(struct cgroup_subsys_state *css,
struct cftype *cft)
@@ -10253,20 +10227,6 @@ static struct cftype cpu_legacy_files[] = {
};
#ifdef CONFIG_RT_GROUP_SCHED
-static struct cftype rt_group_files[] = {
- {
- .name = "rt_runtime_us",
- .read_s64 = cpu_rt_runtime_read,
- .write_s64 = cpu_rt_runtime_write,
- },
- {
- .name = "rt_period_us",
- .read_u64 = cpu_rt_period_read_uint,
- .write_u64 = cpu_rt_period_write_uint,
- },
- { } /* Terminate */
-};
-
# ifdef CONFIG_RT_GROUP_SCHED_DEFAULT_DISABLED
DEFINE_STATIC_KEY_FALSE(rt_group_sched);
# else
@@ -10289,16 +10249,6 @@ static int __init setup_rt_group_sched(char *str)
return 1;
}
__setup("rt_group_sched=", setup_rt_group_sched);
-
-static int __init cpu_rt_group_init(void)
-{
- if (!rt_group_sched_enabled())
- return 0;
-
- WARN_ON(cgroup_add_legacy_cftypes(&cpu_cgrp_subsys, rt_group_files));
- return 0;
-}
-subsys_initcall(cpu_rt_group_init);
#endif /* CONFIG_RT_GROUP_SCHED */
static int cpu_extra_stat_show(struct seq_file *sf,
diff --git a/kernel/sched/rt.c b/kernel/sched/rt.c
index 168a92945b4a..4f1e7af2e88d 100644
--- a/kernel/sched/rt.c
+++ b/kernel/sched/rt.c
@@ -1,3 +1,4 @@
+#pragma GCC diagnostic ignored "-Wunused-function"
// SPDX-License-Identifier: GPL-2.0
/*
* Real-Time Scheduling Class (mapped to the SCHED_FIFO and SCHED_RR
@@ -2258,54 +2259,6 @@ static int tg_set_rt_bandwidth(struct task_group *tg,
return err;
}
-int sched_group_set_rt_runtime(struct task_group *tg, long rt_runtime_us)
-{
- u64 rt_runtime, rt_period;
-
- rt_period = ktime_to_ns(tg->rt_bandwidth.rt_period);
- rt_runtime = (u64)rt_runtime_us * NSEC_PER_USEC;
- if (rt_runtime_us < 0)
- rt_runtime = RUNTIME_INF;
- else if ((u64)rt_runtime_us > U64_MAX / NSEC_PER_USEC)
- return -EINVAL;
-
- return tg_set_rt_bandwidth(tg, rt_period, rt_runtime);
-}
-
-long sched_group_rt_runtime(struct task_group *tg)
-{
- u64 rt_runtime_us;
-
- if (tg->rt_bandwidth.rt_runtime == RUNTIME_INF)
- return -1;
-
- rt_runtime_us = tg->rt_bandwidth.rt_runtime;
- do_div(rt_runtime_us, NSEC_PER_USEC);
- return rt_runtime_us;
-}
-
-int sched_group_set_rt_period(struct task_group *tg, u64 rt_period_us)
-{
- u64 rt_runtime, rt_period;
-
- if (rt_period_us > U64_MAX / NSEC_PER_USEC)
- return -EINVAL;
-
- rt_period = rt_period_us * NSEC_PER_USEC;
- rt_runtime = tg->rt_bandwidth.rt_runtime;
-
- return tg_set_rt_bandwidth(tg, rt_period, rt_runtime);
-}
-
-long sched_group_rt_period(struct task_group *tg)
-{
- u64 rt_period_us;
-
- rt_period_us = ktime_to_ns(tg->rt_bandwidth.rt_period);
- do_div(rt_period_us, NSEC_PER_USEC);
- return rt_period_us;
-}
-
int sched_rt_can_attach(struct task_group *tg)
{
/* Don't accept real-time tasks when there is no way for them to run */
diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h
index bde49f216081..efe52e162ba5 100644
--- a/kernel/sched/sched.h
+++ b/kernel/sched/sched.h
@@ -607,10 +607,6 @@ extern void start_cfs_bandwidth(struct cfs_bandwidth *cfs_b);
extern void unthrottle_cfs_rq(struct cfs_rq *cfs_rq);
extern bool cfs_task_bw_constrained(struct task_struct *p);
-extern int sched_group_set_rt_runtime(struct task_group *tg, long rt_runtime_us);
-extern int sched_group_set_rt_period(struct task_group *tg, u64 rt_period_us);
-extern long sched_group_rt_runtime(struct task_group *tg);
-extern long sched_group_rt_period(struct task_group *tg);
extern int sched_rt_can_attach(struct task_group *tg);
extern struct task_group *sched_create_group(struct task_group *parent);
--
2.54.0
^ permalink raw reply related
* [RFC PATCH v6 15/25] sched/rt: Update task event callbacks for HCBS scheduling
From: Yuri Andriaccio @ 2026-06-08 12:15 UTC (permalink / raw)
To: Ingo Molnar, Peter Zijlstra, Juri Lelli, Vincent Guittot,
Dietmar Eggemann, Steven Rostedt, Ben Segall, Mel Gorman,
Valentin Schneider, Tejun Heo, Johannes Weiner,
Michal Koutný
Cc: cgroups, linux-kernel, Luca Abeni, Yuri Andriaccio
In-Reply-To: <20260608121546.69910-1-yurand2000@gmail.com>
Update wakeup_preempt_rt, switched_{from/to}_rt and prio_changed_rt with
rt-cgroup's specific preemption rules:
- In wakeup_preempt_rt(), whenever a task wakes up, it must be checked if
it is served by a deadline server or it lives on the global runqueue.
Preemption rules (as documented in the function), change based on the
current task's donor and woken task runqueue:
- If both tasks are FIFO/RR tasks on the global runqueue, or the same
cgroup, run as normal.
- If woken is inside a cgroup, but donor is a FIFO task on the global
runqueue, always preempt. If donor is a DEADLINE task, check if the dl
server preempts donor.
- If both tasks are FIFO/RR tasks in served but different groups, check
whether the woken server preempts the donor server.
- In prio_changed_rt(), if the task is not running, only run preemption
checks if the running task resides on the same task group of the task
that changed priority.
Update sched_rt_can_attach() to check if a task can be attached to a given
cgroup. For now the check only consists in checking if the group has
non-zero bandwidth. Remove the tsk argument from sched_rt_can_attach, as
it is unused.
Change cpu_cgroup_can_attach() to check if the attachee is a FIFO/RR
task before attaching it to a cgroup.
Update __sched_setscheduler() to perform checks when trying to switch
to FIFO/RR for a task inside a cgroup, as the group needs to have
runtime allocated.
Update task_is_throttled_rt() for SCHED_CORE, returning the is_throttled
value of the server if present, while global rt-tasks are never throttled.
Update migration functions to ignore cgroups migration, to be implemented
in later patches.
Co-developed-by: Alessio Balsini <a.balsini@sssup.it>
Signed-off-by: Alessio Balsini <a.balsini@sssup.it>
Co-developed-by: Andrea Parri <parri.andrea@gmail.com>
Signed-off-by: Andrea Parri <parri.andrea@gmail.com>
Co-developed-by: luca abeni <luca.abeni@santannapisa.it>
Signed-off-by: luca abeni <luca.abeni@santannapisa.it>
Signed-off-by: Yuri Andriaccio <yurand2000@gmail.com>
---
kernel/sched/core.c | 2 +-
kernel/sched/rt.c | 98 ++++++++++++++++++++++++++++++++++++++---
kernel/sched/sched.h | 2 +-
kernel/sched/syscalls.c | 12 +++++
4 files changed, 105 insertions(+), 9 deletions(-)
diff --git a/kernel/sched/core.c b/kernel/sched/core.c
index 9e47a02cfaf7..1252f45feda0 100644
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -9545,7 +9545,7 @@ static int cpu_cgroup_can_attach(struct cgroup_taskset *tset)
goto scx_check;
cgroup_taskset_for_each(task, css, tset) {
- if (!sched_rt_can_attach(css_tg(css), task))
+ if (rt_task(task) && !sched_rt_can_attach(css_tg(css)))
return -EINVAL;
}
scx_check:
diff --git a/kernel/sched/rt.c b/kernel/sched/rt.c
index 61e9dab894d1..168a92945b4a 100644
--- a/kernel/sched/rt.c
+++ b/kernel/sched/rt.c
@@ -372,6 +372,9 @@ static inline void rt_queue_push_tasks(struct rt_rq *rt_rq)
{
struct rq *rq = global_rq_of_rt_rq(rt_rq);
+ if (is_dl_group(rt_rq))
+ return;
+
if (!has_pushable_tasks(rt_rq))
return;
@@ -382,6 +385,9 @@ static inline void rt_queue_pull_task(struct rt_rq *rt_rq)
{
struct rq *rq = global_rq_of_rt_rq(rt_rq);
+ if (is_dl_group(rt_rq))
+ return;
+
queue_balance_callback(rq, &per_cpu(rt_pull_head, rq->cpu), pull_rt_task);
}
@@ -1031,7 +1037,55 @@ static int balance_rt(struct rq *rq, struct task_struct *p, struct rq_flags *rf)
static void wakeup_preempt_rt(struct rq *rq, struct task_struct *p, int flags)
{
struct task_struct *donor = rq->donor;
+ struct sched_dl_entity *woken_dl_se = NULL;
+ struct sched_dl_entity *donor_dl_se = NULL;
+
+ if (!rt_group_sched_enabled())
+ goto same_group_sched;
+
+ /*
+ * Preemption checks are different if the waking task and the current donor
+ * are running on the global runqueue or in a cgroup. The following rules
+ * apply:
+ * - dl-tasks (and equally dl_servers) always preempt FIFO/RR tasks.
+ * - if donor is a FIFO/RR task inside a cgroup (i.e. run by a
+ * dl_server), or donor is a DEADLINE task and waking is a FIFO/RR
+ * task on the root cgroup, do nothing.
+ * - if waking is inside a cgroup but donor is a FIFO/RR task in the
+ * root cgroup, always reschedule.
+ * - if they are both on the global runqueue or in the same cgroup, run
+ * the standard code.
+ * - if they are both in a cgroup, but not the same one, check whether the
+ * woken task's dl_server preempts the donor's dl_server.
+ * - if donor is a DEADLINE task and waking is in a cgroup, check whether
+ * the woken task's server preempts donor.
+ */
+ if (is_dl_group(rt_rq_of_se(&p->rt)))
+ woken_dl_se = dl_group_of(rt_rq_of_se(&p->rt));
+ if (is_dl_group(rt_rq_of_se(&donor->rt)))
+ donor_dl_se = dl_group_of(rt_rq_of_se(&donor->rt));
+ else if (task_has_dl_policy(donor))
+ donor_dl_se = &donor->dl;
+
+ if (woken_dl_se != NULL && donor_dl_se != NULL) {
+ if (woken_dl_se == donor_dl_se) {
+ goto same_group_sched;
+ }
+
+ if (dl_entity_preempt(woken_dl_se, donor_dl_se))
+ resched_curr(rq);
+
+ return;
+
+ } else if (woken_dl_se != NULL) {
+ resched_curr(rq);
+ return;
+
+ } else if (donor_dl_se != NULL) {
+ return;
+ }
+same_group_sched:
/*
* XXX If we're preempted by DL, queue a push?
*/
@@ -1055,7 +1109,8 @@ static void wakeup_preempt_rt(struct rq *rq, struct task_struct *p, int flags)
* to move current somewhere else, making room for our non-migratable
* task.
*/
- if (p->prio == donor->prio && !test_tsk_need_resched(rq->curr))
+ if (!is_dl_group(rt_rq_of_se(&p->rt)) &&
+ p->prio == donor->prio && !test_tsk_need_resched(rq->curr))
check_preempt_equal_prio(rq, p);
}
@@ -1362,6 +1417,9 @@ static int push_rt_rq_task(struct rt_rq *rt_rq, bool pull)
struct rt_rq *lowest_rt_rq;
int ret = 0;
+ if (is_dl_group(rt_rq))
+ return 0;
+
if (!rt_rq->overloaded)
return 0;
@@ -1668,6 +1726,9 @@ static void pull_rt_rq_task(struct rt_rq *this_rt_rq)
struct rq *src_rq;
int rt_overload_count = rt_overloaded(this_rq);
+ if (is_dl_group(&this_rq->rt))
+ return;
+
if (likely(!rt_overload_count))
return;
@@ -1811,6 +1872,8 @@ static void rq_offline_rt(struct rq *rq)
*/
static void switched_from_rt(struct rq *rq, struct task_struct *p)
{
+ struct rt_rq *rt_rq = rt_rq_of_se(&p->rt);
+
/*
* If there are other RT tasks then we will reschedule
* and the scheduling of the other RT tasks will handle
@@ -1818,10 +1881,10 @@ static void switched_from_rt(struct rq *rq, struct task_struct *p)
* we may need to handle the pulling of RT tasks
* now.
*/
- if (!task_on_rq_queued(p) || rq->rt.rt_nr_running)
+ if (!task_on_rq_queued(p) || rt_rq->rt_nr_running)
return;
- rt_queue_pull_task(rt_rq_of_se(&p->rt));
+ rt_queue_pull_task(rt_rq);
}
void __init init_sched_rt_class(void)
@@ -1858,6 +1921,7 @@ static void switched_to_rt(struct rq *rq, struct task_struct *p)
if (task_on_rq_queued(p)) {
if (p->nr_cpus_allowed > 1 && rq->rt.overloaded)
rt_queue_push_tasks(rt_rq_of_se(&p->rt));
+
if (p->prio < rq->donor->prio && cpu_online(cpu_of(rq)))
resched_curr(rq);
}
@@ -1870,6 +1934,8 @@ static void switched_to_rt(struct rq *rq, struct task_struct *p)
static void
prio_changed_rt(struct rq *rq, struct task_struct *p, u64 oldprio)
{
+ struct rt_rq *rt_rq = rt_rq_of_se(&p->rt);
+
if (!task_on_rq_queued(p))
return;
@@ -1882,15 +1948,24 @@ prio_changed_rt(struct rq *rq, struct task_struct *p, u64 oldprio)
* may need to pull tasks to this runqueue.
*/
if (oldprio < p->prio)
- rt_queue_pull_task(rt_rq_of_se(&p->rt));
+ rt_queue_pull_task(rt_rq);
/*
* If there's a higher priority task waiting to run
* then reschedule.
*/
- if (p->prio > rq->rt.highest_prio.curr)
+ if (p->prio > rt_rq->highest_prio.curr)
resched_curr(rq);
} else {
+ /*
+ * This task is not running, thus we check against the currently
+ * running task for preemption. We can preempt only if both tasks are
+ * in the same cgroup or on the global runqueue.
+ */
+ if (rt_group_sched_enabled() &&
+ rt_rq->tg != rt_rq_of_se(&rq->curr->rt)->tg)
+ return;
+
/*
* This task is not running, but if it is
* greater than the current running task
@@ -1983,7 +2058,16 @@ static unsigned int get_rr_interval_rt(struct rq *rq, struct task_struct *task)
#ifdef CONFIG_SCHED_CORE
static int task_is_throttled_rt(struct task_struct *p, int cpu)
{
+#ifdef CONFIG_RT_GROUP_SCHED
+ struct rt_rq *rt_rq;
+
+ rt_rq = task_group(p)->rt_rq[cpu];
+ WARN_ON(!rt_group_sched_enabled() && rt_rq->tg != &root_task_group);
+
+ return dl_group_of(rt_rq)->dl_throttled;
+#else
return 0;
+#endif
}
#endif /* CONFIG_SCHED_CORE */
@@ -2222,10 +2306,10 @@ long sched_group_rt_period(struct task_group *tg)
return rt_period_us;
}
-int sched_rt_can_attach(struct task_group *tg, struct task_struct *tsk)
+int sched_rt_can_attach(struct task_group *tg)
{
/* Don't accept real-time tasks when there is no way for them to run */
- if (rt_group_sched_enabled() && rt_task(tsk) && tg->rt_bandwidth.rt_runtime == 0)
+ if (rt_group_sched_enabled() && tg->dl_bandwidth.dl_runtime == 0)
return 0;
return 1;
diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h
index 66d5bd1aa4f1..bde49f216081 100644
--- a/kernel/sched/sched.h
+++ b/kernel/sched/sched.h
@@ -611,7 +611,7 @@ extern int sched_group_set_rt_runtime(struct task_group *tg, long rt_runtime_us)
extern int sched_group_set_rt_period(struct task_group *tg, u64 rt_period_us);
extern long sched_group_rt_runtime(struct task_group *tg);
extern long sched_group_rt_period(struct task_group *tg);
-extern int sched_rt_can_attach(struct task_group *tg, struct task_struct *tsk);
+extern int sched_rt_can_attach(struct task_group *tg);
extern struct task_group *sched_create_group(struct task_group *parent);
extern void sched_online_group(struct task_group *tg,
diff --git a/kernel/sched/syscalls.c b/kernel/sched/syscalls.c
index 9c1ba10ea5a7..773f744c0460 100644
--- a/kernel/sched/syscalls.c
+++ b/kernel/sched/syscalls.c
@@ -606,6 +606,18 @@ int __sched_setscheduler(struct task_struct *p,
change:
if (user) {
+ /*
+ * Do not allow real-time tasks into groups that have no runtime
+ * assigned.
+ */
+ if (rt_group_sched_enabled() &&
+ dl_bandwidth_enabled() && rt_policy(policy) &&
+ !sched_rt_can_attach(task_group(p)) &&
+ !task_group_is_autogroup(task_group(p))) {
+ retval = -EPERM;
+ goto unlock;
+ }
+
if (dl_bandwidth_enabled() && dl_policy(policy) &&
!(attr->sched_flags & SCHED_FLAG_SUGOV)) {
cpumask_t *span = rq->rd->span;
--
2.54.0
^ permalink raw reply related
* [RFC PATCH v6 14/25] sched/rt: Implement dl-server operations for rt-cgroups.
From: Yuri Andriaccio @ 2026-06-08 12:15 UTC (permalink / raw)
To: Ingo Molnar, Peter Zijlstra, Juri Lelli, Vincent Guittot,
Dietmar Eggemann, Steven Rostedt, Ben Segall, Mel Gorman,
Valentin Schneider, Tejun Heo, Johannes Weiner,
Michal Koutný
Cc: cgroups, linux-kernel, Luca Abeni, Yuri Andriaccio
In-Reply-To: <20260608121546.69910-1-yurand2000@gmail.com>
Implement rt_server_pick, the callback that deadline servers use to
pick a task to schedule.
rt_server_pick(): pick the next runnable rt task and tell the
scheduler that it is going to be scheduled next.
Let enqueue_task_rt function start the attached deadline server when the
first task is enqueued on a specific rq/server.
The server is not symmetrically stopped in dequeue_task_rt as it is
stopped when server_pick_task returns NULL (see deadline.c).
Change update_curr_rt to perform a deadline server update if the
updated task is served by non-root group.
Update {enqueue/dequeue}_pushable_task and rt_{set/clear}_overload to
only set the CPU-wise overload flag only if the root runqueues are
overloaded, but not for HCBS runqueues.
Update inc/dec_dl_tasks to account the number of active tasks in the
local runqueue for rt-cgroups servers, as their local runqueue is
different from the global runqueue, and thus when a rt-group server is
activated/deactivated, the number of served tasks' must be
added/removed. This uses nr_running to be compatible with future
dl-server interfaces. Account also the deadline server so that it is
picked for shutdown when its runqueue is empty (future patches will
try to pull tasks before stopping).
Update inc/dec_rt_prio_smp to change a rq's cpupri only if the rt_rq
is the global runqueue, since cgroups are scheduled via their
dl-server priority.
Update inc/dec_rt_tasks to account for waking/sleeping tasks on the
global runqueue, when the task runs on the root cgroup, or its local
dl server is active. The accounting is not done when servers are
throttled, as they will add/sub the number of tasks running when they
get enqueued/dequeued. For rt cgroups, account for the number of active
tasks in the nr_running field of the local runqueue (add/sub_nr_running),
as this number is used when a dl server is enqueued/dequeued.
Update set_task_rq to record the rt_rq of the cgroup's active_context,
tracking where to schedule the given task.
Update set_task_rq to record the dl_rq, tracking which deadline
server manages a task.
Update set_task_rq to not use the parent field anymore, as it is
unused by this patchset's code. Remove the unused parent field from
sched_rt_entity.
Co-developed-by: Alessio Balsini <a.balsini@sssup.it>
Signed-off-by: Alessio Balsini <a.balsini@sssup.it>
Co-developed-by: Andrea Parri <parri.andrea@gmail.com>
Signed-off-by: Andrea Parri <parri.andrea@gmail.com>
Co-developed-by: luca abeni <luca.abeni@santannapisa.it>
Signed-off-by: luca abeni <luca.abeni@santannapisa.it>
Signed-off-by: Yuri Andriaccio <yurand2000@gmail.com>
---
include/linux/sched.h | 1 -
kernel/sched/deadline.c | 8 +++++
kernel/sched/rt.c | 70 ++++++++++++++++++++++++++++++++++++-----
kernel/sched/sched.h | 11 +++++--
4 files changed, 79 insertions(+), 11 deletions(-)
diff --git a/include/linux/sched.h b/include/linux/sched.h
index 411ffe9b34b3..b20451fcda55 100644
--- a/include/linux/sched.h
+++ b/include/linux/sched.h
@@ -630,7 +630,6 @@ struct sched_rt_entity {
struct sched_rt_entity *back;
#ifdef CONFIG_RT_GROUP_SCHED
- struct sched_rt_entity *parent;
/* rq on which this entity is (to be) queued: */
struct rt_rq *rt_rq;
/* rq "owned" by this entity/group: */
diff --git a/kernel/sched/deadline.c b/kernel/sched/deadline.c
index 166d23f45cab..a63253ec6441 100644
--- a/kernel/sched/deadline.c
+++ b/kernel/sched/deadline.c
@@ -2096,6 +2096,10 @@ void inc_dl_tasks(struct sched_dl_entity *dl_se, struct dl_rq *dl_rq)
if (!dl_server(dl_se))
add_nr_running(rq_of_dl_rq(dl_rq), 1);
+ else if (rq_of_dl_se(dl_se) != dl_se->my_q) {
+ WARN_ON(dl_se->my_q->rt.rt_nr_running != dl_se->my_q->nr_running);
+ add_nr_running(rq_of_dl_rq(dl_rq), dl_se->my_q->nr_running + 1);
+ }
inc_dl_deadline(dl_rq, deadline);
}
@@ -2108,6 +2112,10 @@ void dec_dl_tasks(struct sched_dl_entity *dl_se, struct dl_rq *dl_rq)
if (!dl_server(dl_se))
sub_nr_running(rq_of_dl_rq(dl_rq), 1);
+ else if (rq_of_dl_se(dl_se) != dl_se->my_q) {
+ WARN_ON(dl_se->my_q->rt.rt_nr_running != dl_se->my_q->nr_running);
+ sub_nr_running(rq_of_dl_rq(dl_rq), dl_se->my_q->nr_running - 1);
+ }
dec_dl_deadline(dl_rq, dl_se->deadline);
}
diff --git a/kernel/sched/rt.c b/kernel/sched/rt.c
index a6adf21772a6..61e9dab894d1 100644
--- a/kernel/sched/rt.c
+++ b/kernel/sched/rt.c
@@ -284,9 +284,19 @@ int alloc_rt_sched_group(struct task_group *tg, struct task_group *parent)
return 1;
}
+static struct sched_rt_entity *pick_next_rt_entity(struct rt_rq *rt_rq);
+
static struct task_struct *rt_server_pick(struct sched_dl_entity *dl_se, struct rq_flags *rf)
{
- return NULL;
+ struct rt_rq *rt_rq = &dl_se->my_q->rt;
+ struct task_struct *p;
+
+ if (!sched_rt_runnable(dl_se->my_q))
+ return NULL;
+
+ p = rt_task_of(pick_next_rt_entity(rt_rq));
+
+ return p;
}
#else /* !CONFIG_RT_GROUP_SCHED */
@@ -314,6 +324,9 @@ static inline int rt_overloaded(struct rq *rq)
static inline void rt_set_overload(struct rq *rq)
{
+ if (is_dl_group(&rq->rt))
+ return;
+
if (!rq->online)
return;
@@ -333,6 +346,9 @@ static inline void rt_set_overload(struct rq *rq)
static inline void rt_clear_overload(struct rq *rq)
{
+ if (is_dl_group(&rq->rt))
+ return;
+
if (!rq->online)
return;
@@ -392,7 +408,7 @@ static void enqueue_pushable_task(struct rt_rq *rt_rq, struct task_struct *p)
rt_rq->highest_prio.next = p->prio;
if (!rt_rq->overloaded) {
- rt_set_overload(global_rq_of_rt_rq(rt_rq));
+ rt_set_overload(rq_of_rt_rq(rt_rq));
rt_rq->overloaded = 1;
}
}
@@ -410,7 +426,7 @@ static void dequeue_pushable_task(struct rt_rq *rt_rq, struct task_struct *p)
rt_rq->highest_prio.next = MAX_RT_PRIO-1;
if (rt_rq->overloaded) {
- rt_clear_overload(global_rq_of_rt_rq(rt_rq));
+ rt_clear_overload(rq_of_rt_rq(rt_rq));
rt_rq->overloaded = 0;
}
}
@@ -511,6 +527,7 @@ static inline int rt_se_prio(struct sched_rt_entity *rt_se)
static void update_curr_rt(struct rq *rq)
{
struct task_struct *donor = rq->donor;
+ struct rt_rq *rt_rq;
s64 delta_exec;
if (donor->sched_class != &rt_sched_class)
@@ -520,21 +537,32 @@ static void update_curr_rt(struct rq *rq)
if (unlikely(delta_exec <= 0))
return;
- if (!rt_bandwidth_enabled())
+ if (!rt_group_sched_enabled())
+ return;
+
+ if (!dl_bandwidth_enabled())
return;
+
+ rt_rq = rt_rq_of_se(&donor->rt);
+ if (is_dl_group(rt_rq)) {
+ struct sched_dl_entity *dl_se = dl_group_of(rt_rq);
+
+ dl_server_update(dl_se, delta_exec);
+ }
}
static void
inc_rt_prio_smp(struct rt_rq *rt_rq, int prio, int prev_prio)
{
- struct rq *rq = rq_of_rt_rq(rt_rq);
+ struct rq *rq;
/*
* Change rq's cpupri only if rt_rq is the top queue.
*/
- if (IS_ENABLED(CONFIG_RT_GROUP_SCHED) && &rq->rt != rt_rq)
+ if (is_dl_group(rt_rq))
return;
+ rq = rq_of_rt_rq(rt_rq);
if (rq->online && prio < prev_prio)
cpupri_set(&rq->rd->cpupri, rq->cpu, prio);
}
@@ -542,14 +570,15 @@ inc_rt_prio_smp(struct rt_rq *rt_rq, int prio, int prev_prio)
static void
dec_rt_prio_smp(struct rt_rq *rt_rq, int prio, int prev_prio)
{
- struct rq *rq = rq_of_rt_rq(rt_rq);
+ struct rq *rq;
/*
* Change rq's cpupri only if rt_rq is the top queue.
*/
- if (IS_ENABLED(CONFIG_RT_GROUP_SCHED) && &rq->rt != rt_rq)
+ if (is_dl_group(rt_rq))
return;
+ rq = rq_of_rt_rq(rt_rq);
if (rq->online && rt_rq->highest_prio.curr != prev_prio)
cpupri_set(&rq->rd->cpupri, rq->cpu, rt_rq->highest_prio.curr);
}
@@ -610,6 +639,15 @@ void inc_rt_tasks(struct sched_rt_entity *rt_se, struct rt_rq *rt_rq)
rt_rq->rr_nr_running += is_rr_task(rt_se);
inc_rt_prio(rt_rq, rt_se_prio(rt_se));
+
+ if (is_dl_group(rt_rq)) {
+ struct sched_dl_entity *dl_se = dl_group_of(rt_rq);
+
+ if (!dl_se->dl_throttled)
+ add_nr_running(global_rq_of_rt_rq(rt_rq), 1);
+ }
+
+ add_nr_running(rq_of_rt_rq(rt_rq), 1);
}
static inline
@@ -620,6 +658,15 @@ void dec_rt_tasks(struct sched_rt_entity *rt_se, struct rt_rq *rt_rq)
rt_rq->rr_nr_running -= is_rr_task(rt_se);
dec_rt_prio(rt_rq, rt_se_prio(rt_se));
+
+ if (is_dl_group(rt_rq)) {
+ struct sched_dl_entity *dl_se = dl_group_of(rt_rq);
+
+ if (!dl_se->dl_throttled)
+ sub_nr_running(global_rq_of_rt_rq(rt_rq), 1);
+ }
+
+ sub_nr_running(rq_of_rt_rq(rt_rq), 1);
}
/*
@@ -806,6 +853,13 @@ enqueue_task_rt(struct rq *rq, struct task_struct *p, int flags)
check_schedstat_required();
update_stats_wait_start_rt(rt_rq_of_se(rt_se), rt_se);
+ /* Task arriving in an idle group of tasks. */
+ if (is_dl_group(rt_rq) && rt_rq->rt_nr_running == 0) {
+ struct sched_dl_entity *dl_se = dl_group_of(rt_rq);
+
+ dl_server_start(dl_se);
+ }
+
enqueue_rt_entity(rt_se, flags);
if (task_is_blocked(p))
diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h
index 58f67093145e..66d5bd1aa4f1 100644
--- a/kernel/sched/sched.h
+++ b/kernel/sched/sched.h
@@ -2310,10 +2310,11 @@ static inline void set_task_rq(struct task_struct *p, unsigned int cpu)
* root_task_group's rt_rq than switching in rt_rq_of_se()
* Clobbers tg(!)
*/
+ guard(raw_spinlock_irqsave)(&tg->dl_bandwidth.dl_runtime_lock);
if (!rt_group_sched_enabled())
tg = &root_task_group;
- p->rt.rt_rq = tg->rt_rq[cpu];
- p->rt.parent = tg->rt_se[cpu];
+ p->rt.rt_rq = tg->dl_bandwidth.active_context->rt_rq[cpu];
+ p->dl.dl_rq = &cpu_rq(cpu)->dl;
#endif /* CONFIG_RT_GROUP_SCHED */
}
@@ -2976,6 +2977,9 @@ static inline void add_nr_running(struct rq *rq, unsigned count)
unsigned prev_nr = rq->nr_running;
rq->nr_running = prev_nr + count;
+ if (rq != cpu_rq(rq->cpu))
+ return;
+
if (trace_sched_update_nr_running_tp_enabled()) {
call_trace_sched_update_nr_running(rq, count);
}
@@ -2989,6 +2993,9 @@ static inline void add_nr_running(struct rq *rq, unsigned count)
static inline void sub_nr_running(struct rq *rq, unsigned count)
{
rq->nr_running -= count;
+ if (rq != cpu_rq(rq->cpu))
+ return;
+
if (trace_sched_update_nr_running_tp_enabled()) {
call_trace_sched_update_nr_running(rq, -count);
}
--
2.54.0
^ permalink raw reply related
* [RFC PATCH v6 12/25] sched/rt: Add {alloc/unregister/free}_rt_sched_group
From: Yuri Andriaccio @ 2026-06-08 12:15 UTC (permalink / raw)
To: Ingo Molnar, Peter Zijlstra, Juri Lelli, Vincent Guittot,
Dietmar Eggemann, Steven Rostedt, Ben Segall, Mel Gorman,
Valentin Schneider, Tejun Heo, Johannes Weiner,
Michal Koutný
Cc: cgroups, linux-kernel, Luca Abeni, Yuri Andriaccio
In-Reply-To: <20260608121546.69910-1-yurand2000@gmail.com>
Add allocation and deallocation code for rt-cgroups.
Declare dl_server specific functions (only skeleton, but no
implementation yet), needed by the deadline servers to be called when
trying to schedule.
Initialize a cgroup's active context to that of its parent.
Co-developed-by: Alessio Balsini <a.balsini@sssup.it>
Signed-off-by: Alessio Balsini <a.balsini@sssup.it>
Co-developed-by: Andrea Parri <parri.andrea@gmail.com>
Signed-off-by: Andrea Parri <parri.andrea@gmail.com>
Co-developed-by: luca abeni <luca.abeni@santannapisa.it>
Signed-off-by: luca abeni <luca.abeni@santannapisa.it>
Signed-off-by: Yuri Andriaccio <yurand2000@gmail.com>
---
kernel/sched/rt.c | 156 +++++++++++++++++++++++++++++++++++++++++++++-
1 file changed, 154 insertions(+), 2 deletions(-)
diff --git a/kernel/sched/rt.c b/kernel/sched/rt.c
index dbba7a57d6f1..a6adf21772a6 100644
--- a/kernel/sched/rt.c
+++ b/kernel/sched/rt.c
@@ -120,24 +120,176 @@ struct dl_bandwidth *dl_bandwidth_write(struct task_group *tg)
void unregister_rt_sched_group(struct task_group *tg)
{
+ int i;
+
+ if (!rt_group_sched_enabled())
+ return;
+
+ if (!tg->dl_se || !tg->rt_rq)
+ return;
+ for_each_possible_cpu(i) {
+ if (!tg->dl_se[i] || !tg->rt_rq[i])
+ continue;
+
+ if (tg->dl_se[i]->dl_runtime)
+ dl_init_tg(tg->dl_se[i], 0, tg->dl_se[i]->dl_period);
+ }
}
void free_rt_sched_group(struct task_group *tg)
{
+ int i;
+ unsigned long flags;
+
if (!rt_group_sched_enabled())
return;
+
+ if (!tg->dl_se || !tg->rt_rq)
+ return;
+
+ for_each_possible_cpu(i) {
+ if (!tg->dl_se[i] || !tg->rt_rq[i])
+ continue;
+
+ /*
+ * Shutdown the dl_server and free it
+ *
+ * Since the dl timer is going to be cancelled,
+ * we risk to never decrease the running bw...
+ * Fix this issue by changing the group runtime
+ * to 0 immediately before freeing it.
+ */
+ if (tg->dl_se[i]->dl_runtime)
+ dl_init_tg(tg->dl_se[i], 0, tg->dl_se[i]->dl_period);
+
+ raw_spin_rq_lock_irqsave(cpu_rq(i), flags);
+ hrtimer_cancel(&tg->dl_se[i]->dl_timer);
+ raw_spin_rq_unlock_irqrestore(cpu_rq(i), flags);
+ kfree(tg->dl_se[i]);
+
+ /* Free the local per-cpu runqueue */
+ kfree(rq_of_rt_rq(tg->rt_rq[i]));
+ }
+
+ kfree(tg->rt_rq);
+ kfree(tg->dl_se);
}
+static inline void __rt_rq_free(struct rt_rq **rt_rq)
+{
+ int i;
+
+ for_each_possible_cpu(i) {
+ kfree(rq_of_rt_rq(rt_rq[i]));
+ }
+
+ kfree(rt_rq);
+}
+
+DEFINE_FREE(rt_rq_free, struct rt_rq **, if (_T) __rt_rq_free(_T))
+
+static inline void __dl_se_free(struct sched_dl_entity **dl_se)
+{
+ int i;
+
+ for_each_possible_cpu(i) {
+ kfree(dl_se[i]);
+ }
+
+ kfree(dl_se);
+}
+
+DEFINE_FREE(dl_se_free, struct sched_dl_entity **, if (_T) __dl_se_free(_T))
+
+static int __alloc_rt_sched_group_data(struct task_group *tg) {
+ /* Instantiate automatic cleanup in event of kalloc fail */
+ struct rt_rq **tg_rt_rq __free(rt_rq_free) = NULL;
+ struct sched_dl_entity **tg_dl_se __free(dl_se_free) = NULL;
+ struct sched_dl_entity *dl_se __free(kfree) = NULL;
+ struct rq *s_rq __free(kfree) = NULL;
+ int i;
+
+ tg_rt_rq = kcalloc(nr_cpu_ids, sizeof(struct rt_rq *), GFP_KERNEL);
+ if (!tg_rt_rq)
+ return 0;
+
+ tg_dl_se = kcalloc(nr_cpu_ids,
+ sizeof(struct sched_dl_entity *), GFP_KERNEL);
+ if (!tg_dl_se)
+ return 0;
+
+ for_each_possible_cpu(i) {
+ s_rq = kzalloc_node(sizeof(struct rq),
+ GFP_KERNEL, cpu_to_node(i));
+ if (!s_rq)
+ return 0;
+
+ dl_se = kzalloc_node(sizeof(struct sched_dl_entity),
+ GFP_KERNEL, cpu_to_node(i));
+ if (!dl_se)
+ return 0;
+
+ tg_rt_rq[i] = &no_free_ptr(s_rq)->rt;
+ tg_dl_se[i] = no_free_ptr(dl_se);
+ }
+
+ tg->rt_rq = no_free_ptr(tg_rt_rq);
+ tg->dl_se = no_free_ptr(tg_dl_se);
+
+ return 1;
+}
+
+static struct task_struct *rt_server_pick(struct sched_dl_entity *dl_se, struct rq_flags *rf);
+
int alloc_rt_sched_group(struct task_group *tg, struct task_group *parent)
{
+ struct sched_dl_entity *dl_se;
+ struct rq *s_rq;
+ int i;
+
if (!rt_group_sched_enabled())
return 1;
+ /* Allocate all necessary resources beforehand */
+ if (!__alloc_rt_sched_group_data(tg))
+ return 0;
+
+ /* Initialize the allocated resources now. */
+ scoped_guard(raw_spinlock_irq, dl_bw_lock_of_tg(parent)) {
+ init_dl_bandwidth(&tg->dl_bandwidth, 0, RUNTIME_INF,
+ dl_bandwidth_read(parent)->active_context);
+ }
+
+ for_each_possible_cpu(i) {
+ s_rq = rq_of_rt_rq(tg->rt_rq[i]);
+ dl_se = tg->dl_se[i];
+
+ init_rt_rq(&s_rq->rt);
+ s_rq->cpu = i;
+ s_rq->rt.tg = tg;
+
+ init_dl_entity(dl_se);
+ dl_se->dl_runtime = 0;
+ dl_se->dl_deadline = 0;
+ dl_se->dl_period = 0;
+ dl_se->runtime = 0;
+ dl_se->deadline = 0;
+ dl_se->dl_bw = to_ratio(dl_se->dl_period, dl_se->dl_runtime);
+ dl_se->dl_density = to_ratio(dl_se->dl_deadline, dl_se->dl_runtime);
+ dl_se->dl_server = 1;
+ dl_server_init(dl_se, &cpu_rq(i)->dl, s_rq, rt_server_pick);
+ }
+
return 1;
}
-#else /* !CONFIG_RT_GROUP_SCHED: */
+static struct task_struct *rt_server_pick(struct sched_dl_entity *dl_se, struct rq_flags *rf)
+{
+ return NULL;
+}
+
+#else /* !CONFIG_RT_GROUP_SCHED */
void unregister_rt_sched_group(struct task_group *tg) { }
@@ -147,7 +299,7 @@ int alloc_rt_sched_group(struct task_group *tg, struct task_group *parent)
{
return 1;
}
-#endif /* !CONFIG_RT_GROUP_SCHED */
+#endif /* CONFIG_RT_GROUP_SCHED */
static inline bool need_pull_rt_task(struct rq *rq, struct task_struct *prev)
{
--
2.54.0
^ permalink raw reply related
* [RFC PATCH v6 13/25] sched/deadline: Account rt-cgroups bandwidth in deadline tasks schedulability tests.
From: Yuri Andriaccio @ 2026-06-08 12:15 UTC (permalink / raw)
To: Ingo Molnar, Peter Zijlstra, Juri Lelli, Vincent Guittot,
Dietmar Eggemann, Steven Rostedt, Ben Segall, Mel Gorman,
Valentin Schneider, Tejun Heo, Johannes Weiner,
Michal Koutný
Cc: cgroups, linux-kernel, Luca Abeni, Yuri Andriaccio
In-Reply-To: <20260608121546.69910-1-yurand2000@gmail.com>
From: luca abeni <luca.abeni@santannapisa.it>
Account the rt-cgroups hierarchy's reserved bandwidth in the
schedulability test of deadline entities. This mechanism allows to
completely reserve portion of the rt-bandwidth to rt-cgroups even if
they do not use all of it.
Account for the rt-cgroups' reserved bandwidth also when changing the
total dedicated bandwidth for real time tasks.
Co-developed-by: Alessio Balsini <a.balsini@sssup.it>
Signed-off-by: Alessio Balsini <a.balsini@sssup.it>
Co-developed-by: Andrea Parri <parri.andrea@gmail.com>
Signed-off-by: Andrea Parri <parri.andrea@gmail.com>
Co-developed-by: Yuri Andriaccio <yurand2000@gmail.com>
Signed-off-by: Yuri Andriaccio <yurand2000@gmail.com>
Signed-off-by: luca abeni <luca.abeni@santannapisa.it>
---
kernel/sched/deadline.c | 20 +++++++++++++++++---
1 file changed, 17 insertions(+), 3 deletions(-)
diff --git a/kernel/sched/deadline.c b/kernel/sched/deadline.c
index afadc3521bc0..166d23f45cab 100644
--- a/kernel/sched/deadline.c
+++ b/kernel/sched/deadline.c
@@ -205,11 +205,22 @@ void __dl_add(struct dl_bw *dl_b, u64 tsk_bw, int cpus)
__dl_update(dl_b, -((s32)tsk_bw / cpus));
}
+static inline u64 get_dl_groups_bw(void)
+{
+#ifdef CONFIG_RT_GROUP_SCHED
+ return to_ratio(root_task_group.dl_bandwidth.dl_period,
+ root_task_group.dl_bandwidth.dl_runtime);
+#else
+ return 0;
+#endif
+}
+
static inline bool
__dl_overflow(struct dl_bw *dl_b, unsigned long cap, u64 old_bw, u64 new_bw)
{
return dl_b->bw != -1 &&
- cap_scale(dl_b->bw, cap) < dl_b->total_bw - old_bw + new_bw;
+ cap_scale(dl_b->bw, cap) < dl_b->total_bw - old_bw + new_bw
+ + cap_scale(get_dl_groups_bw(), cap);
}
static inline
@@ -3490,8 +3501,9 @@ int sched_dl_global_validate(void)
u64 period = global_rt_period();
u64 new_bw = to_ratio(period, runtime);
u64 cookie = ++dl_cookie;
+ u64 dl_groups_root = get_dl_groups_bw();
struct dl_bw *dl_b;
- int cpu, cpus, ret = 0;
+ int cpu, cap, cpus, ret = 0;
unsigned long flags;
/*
@@ -3506,10 +3518,12 @@ int sched_dl_global_validate(void)
goto next;
dl_b = dl_bw_of(cpu);
+ cap = dl_bw_capacity(cpu);
cpus = dl_bw_cpus(cpu);
raw_spin_lock_irqsave(&dl_b->lock, flags);
- if (new_bw * cpus < dl_b->total_bw)
+ if (new_bw * cpus < dl_b->total_bw +
+ cap_scale(dl_groups_root, cap))
ret = -EBUSY;
raw_spin_unlock_irqrestore(&dl_b->lock, flags);
--
2.54.0
^ permalink raw reply related
* [RFC PATCH v6 11/25] sched/deadline: Add dl_init_tg
From: Yuri Andriaccio @ 2026-06-08 12:15 UTC (permalink / raw)
To: Ingo Molnar, Peter Zijlstra, Juri Lelli, Vincent Guittot,
Dietmar Eggemann, Steven Rostedt, Ben Segall, Mel Gorman,
Valentin Schneider, Tejun Heo, Johannes Weiner,
Michal Koutný
Cc: cgroups, linux-kernel, Luca Abeni, Yuri Andriaccio
In-Reply-To: <20260608121546.69910-1-yurand2000@gmail.com>
From: luca abeni <luca.abeni@santannapisa.it>
Add dl_init_tg to initialize and/or update a rt-cgroup dl_server and to
also account the allocated bandwidth. This function is currently unhooked
and will be later used to allocate bandwidth to rt-cgroups.
Add lock guard for raw_spin_rq_lock_irq for cleaner code.
Co-developed-by: Alessio Balsini <a.balsini@sssup.it>
Signed-off-by: Alessio Balsini <a.balsini@sssup.it>
Co-developed-by: Andrea Parri <parri.andrea@gmail.com>
Signed-off-by: Andrea Parri <parri.andrea@gmail.com>
Co-developed-by: Yuri Andriaccio <yurand2000@gmail.com>
Signed-off-by: Yuri Andriaccio <yurand2000@gmail.com>
Signed-off-by: luca abeni <luca.abeni@santannapisa.it>
---
kernel/sched/deadline.c | 31 +++++++++++++++++++++++++++++++
kernel/sched/sched.h | 5 +++++
2 files changed, 36 insertions(+)
diff --git a/kernel/sched/deadline.c b/kernel/sched/deadline.c
index 673c6f2b5ece..afadc3521bc0 100644
--- a/kernel/sched/deadline.c
+++ b/kernel/sched/deadline.c
@@ -335,6 +335,37 @@ void cancel_inactive_timer(struct sched_dl_entity *dl_se)
cancel_dl_timer(dl_se, &dl_se->inactive_timer);
}
+#ifdef CONFIG_RT_GROUP_SCHED
+void dl_init_tg(struct sched_dl_entity *dl_se, u64 rt_runtime, u64 rt_period)
+{
+ struct rq *rq = container_of_const(dl_se->dl_rq, struct rq, dl);
+ int is_active;
+ u64 new_bw;
+
+ guard(raw_spin_rq_lock_irq)(rq);
+ is_active = dl_se->my_q->rt.rt_nr_running > 0;
+
+ update_rq_clock(rq);
+ dl_server_stop(dl_se);
+
+ new_bw = to_ratio(rt_period, rt_runtime);
+ dl_rq_change_utilization(rq, dl_se, new_bw);
+
+ dl_se->dl_runtime = rt_runtime;
+ dl_se->dl_deadline = rt_period;
+ dl_se->dl_period = rt_period;
+
+ dl_se->runtime = 0;
+ dl_se->deadline = 0;
+
+ dl_se->dl_bw = new_bw;
+ dl_se->dl_density = new_bw;
+
+ if (is_active)
+ dl_server_start(dl_se);
+}
+#endif
+
static void dl_change_utilization(struct task_struct *p, u64 new_bw)
{
WARN_ON_ONCE(p->dl.flags & SCHED_FLAG_SUGOV);
diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h
index 0ba87be1c98f..58f67093145e 100644
--- a/kernel/sched/sched.h
+++ b/kernel/sched/sched.h
@@ -425,6 +425,7 @@ extern void dl_server_init(struct sched_dl_entity *dl_se, struct dl_rq *dl_rq,
struct rq *served_rq,
dl_server_pick_f pick_task);
extern void sched_init_dl_servers(void);
+extern void dl_init_tg(struct sched_dl_entity *dl_se, u64 rt_runtime, u64 rt_period);
extern void fair_server_init(struct rq *rq);
extern void ext_server_init(struct rq *rq);
@@ -2044,6 +2045,10 @@ static inline struct rq *_this_rq_lock_irq(struct rq_flags *rf) __acquires_ret
return rq;
}
+DEFINE_LOCK_GUARD_1(raw_spin_rq_lock_irq, struct rq,
+ raw_spin_rq_lock_irq(_T->lock),
+ raw_spin_rq_unlock_irq(_T->lock))
+
#ifdef CONFIG_NUMA
enum numa_topology_type {
--
2.54.0
^ permalink raw reply related
* [RFC PATCH v6 10/25] sched/core: Initialize HCBS specific structures.
From: Yuri Andriaccio @ 2026-06-08 12:15 UTC (permalink / raw)
To: Ingo Molnar, Peter Zijlstra, Juri Lelli, Vincent Guittot,
Dietmar Eggemann, Steven Rostedt, Ben Segall, Mel Gorman,
Valentin Schneider, Tejun Heo, Johannes Weiner,
Michal Koutný
Cc: cgroups, linux-kernel, Luca Abeni, Yuri Andriaccio
In-Reply-To: <20260608121546.69910-1-yurand2000@gmail.com>
From: luca abeni <luca.abeni@santannapisa.it>
Update autogroups' creation/destruction to use the new data structures.
Initialize the default bandwidth for rt-cgroups (sched_init).
Initialize rt-scheduler's specific data structures for the root control
group (sched_init).
Remove init_tg_rt_entry in favour of manual setup of the necessary data
structures in sched_init.
Add utility functions to check (and get) if a rt_rq entity is connected
to a rt-cgroup.
Add read/write accessors for dl_bandwidth.
Add dl_bw_lock_of_tg macro to reference the a task group dl_bandwidth's
spinlock.
Co-developed-by: Alessio Balsini <a.balsini@sssup.it>
Signed-off-by: Alessio Balsini <a.balsini@sssup.it>
Co-developed-by: Andrea Parri <parri.andrea@gmail.com>
Signed-off-by: Andrea Parri <parri.andrea@gmail.com>
Co-developed-by: Yuri Andriaccio <yurand2000@gmail.com>
Signed-off-by: Yuri Andriaccio <yurand2000@gmail.com>
Signed-off-by: luca abeni <luca.abeni@santannapisa.it>
---
kernel/sched/autogroup.c | 4 ++--
kernel/sched/core.c | 11 ++++++++--
kernel/sched/deadline.c | 11 ++++++++++
kernel/sched/rt.c | 45 ++++++++++++++++++++++++++++------------
kernel/sched/sched.h | 38 ++++++++++++++++++++++++++++++---
5 files changed, 89 insertions(+), 20 deletions(-)
diff --git a/kernel/sched/autogroup.c b/kernel/sched/autogroup.c
index e380cf9372bb..2122a0740a19 100644
--- a/kernel/sched/autogroup.c
+++ b/kernel/sched/autogroup.c
@@ -52,7 +52,7 @@ static inline void autogroup_destroy(struct kref *kref)
#ifdef CONFIG_RT_GROUP_SCHED
/* We've redirected RT tasks to the root task group... */
- ag->tg->rt_se = NULL;
+ ag->tg->dl_se = NULL;
ag->tg->rt_rq = NULL;
#endif
sched_release_group(ag->tg);
@@ -109,7 +109,7 @@ static inline struct autogroup *autogroup_create(void)
* the policy change to proceed.
*/
free_rt_sched_group(tg);
- tg->rt_se = root_task_group.rt_se;
+ tg->dl_se = root_task_group.dl_se;
tg->rt_rq = root_task_group.rt_rq;
#endif /* CONFIG_RT_GROUP_SCHED */
tg->autogroup = ag;
diff --git a/kernel/sched/core.c b/kernel/sched/core.c
index e38ca8192d2d..9e47a02cfaf7 100644
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -8911,7 +8911,7 @@ void __init sched_init(void)
scx_tg_init(&root_task_group);
#endif /* CONFIG_EXT_GROUP_SCHED */
#ifdef CONFIG_RT_GROUP_SCHED
- root_task_group.rt_se = (struct sched_rt_entity **)ptr;
+ root_task_group.dl_se = (struct sched_dl_entity **)ptr;
ptr += nr_cpu_ids * sizeof(void **);
root_task_group.rt_rq = (struct rt_rq **)ptr;
@@ -8922,6 +8922,11 @@ void __init sched_init(void)
init_defrootdomain();
+#ifdef CONFIG_RT_GROUP_SCHED
+ init_dl_bandwidth(&root_task_group.dl_bandwidth,
+ global_rt_period(), 0, &root_task_group);
+#endif /* CONFIG_RT_GROUP_SCHED */
+
#ifdef CONFIG_CGROUP_SCHED
task_group_cache = KMEM_CACHE(task_group, 0);
@@ -8973,7 +8978,9 @@ void __init sched_init(void)
* starts working after scheduler_running, which is not the case
* yet.
*/
- init_tg_rt_entry(&root_task_group, &rq->rt, NULL, i, NULL);
+ rq->rt.tg = &root_task_group;
+ root_task_group.rt_rq[i] = &rq->rt;
+ root_task_group.dl_se[i] = NULL;
#endif
rq->next_class = &idle_sched_class;
diff --git a/kernel/sched/deadline.c b/kernel/sched/deadline.c
index c12882348a03..673c6f2b5ece 100644
--- a/kernel/sched/deadline.c
+++ b/kernel/sched/deadline.c
@@ -508,6 +508,17 @@ static inline int is_leftmost(struct sched_dl_entity *dl_se, struct dl_rq *dl_rq
static void init_dl_rq_bw_ratio(struct dl_rq *dl_rq);
+void init_dl_bandwidth(struct dl_bandwidth *dl_b, u64 period, u64 runtime,
+ struct task_group *active_context)
+{
+ raw_spin_lock_init(&dl_b->dl_runtime_lock);
+ dl_b->dl_period = period;
+ dl_b->dl_runtime = runtime;
+ dl_b->dl_internal_runtime = 0;
+ dl_b->active_context = active_context;
+}
+
+
void init_dl_bw(struct dl_bw *dl_b)
{
raw_spin_lock_init(&dl_b->lock);
diff --git a/kernel/sched/rt.c b/kernel/sched/rt.c
index 4575c234ae46..dbba7a57d6f1 100644
--- a/kernel/sched/rt.c
+++ b/kernel/sched/rt.c
@@ -86,26 +86,47 @@ void init_rt_rq(struct rt_rq *rt_rq)
#ifdef CONFIG_RT_GROUP_SCHED
-void unregister_rt_sched_group(struct task_group *tg)
+DEFINE_MUTEX(rt_constraints_mutex);
+
+const struct dl_bandwidth *dl_bandwidth_read(struct task_group *tg)
{
+ int held;
+
+ if (IS_ENABLED(CONFIG_LOCKDEP) && debug_locks) {
+ held = 0;
+ if (lockdep_is_held(&rt_constraints_mutex)) {
+ __assume_ctx_lock(&rt_constraints_mutex);
+ held = 1;
+ }
+
+ if (lockdep_is_held(dl_bw_lock_of_tg(tg))) {
+ __assume_ctx_lock(dl_bw_lock_of_tg(tg));
+ held = 1;
+ }
+ lockdep_assert(held);
+ }
+
+ return (const struct dl_bandwidth *)&tg->dl_bandwidth;
}
-void free_rt_sched_group(struct task_group *tg)
+struct dl_bandwidth *dl_bandwidth_write(struct task_group *tg)
{
- if (!rt_group_sched_enabled())
- return;
+ lockdep_assert_held(&rt_constraints_mutex);
+ lockdep_assert_held(dl_bw_lock_of_tg(tg));
+
+ return &tg->dl_bandwidth;
}
-void init_tg_rt_entry(struct task_group *tg, struct rt_rq *rt_rq,
- struct sched_rt_entity *rt_se, int cpu,
- struct sched_rt_entity *parent)
+void unregister_rt_sched_group(struct task_group *tg)
{
- rt_rq->highest_prio.curr = MAX_RT_PRIO-1;
- rt_rq->tg = tg;
- tg->rt_rq[cpu] = rt_rq;
- tg->rt_se[cpu] = rt_se;
+}
+
+void free_rt_sched_group(struct task_group *tg)
+{
+ if (!rt_group_sched_enabled())
+ return;
}
int alloc_rt_sched_group(struct task_group *tg, struct task_group *parent)
@@ -1802,8 +1823,6 @@ DEFINE_SCHED_CLASS(rt) = {
/*
* Ensure that the real time constraints are schedulable.
*/
-static DEFINE_MUTEX(rt_constraints_mutex);
-
static inline int tg_has_rt_tasks(struct task_group *tg)
{
struct task_struct *task;
diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h
index f3c259ab9344..0ba87be1c98f 100644
--- a/kernel/sched/sched.h
+++ b/kernel/sched/sched.h
@@ -606,9 +606,6 @@ extern void start_cfs_bandwidth(struct cfs_bandwidth *cfs_b);
extern void unthrottle_cfs_rq(struct cfs_rq *cfs_rq);
extern bool cfs_task_bw_constrained(struct task_struct *p);
-extern void init_tg_rt_entry(struct task_group *tg, struct rt_rq *rt_rq,
- struct sched_rt_entity *rt_se, int cpu,
- struct sched_rt_entity *parent);
extern int sched_group_set_rt_runtime(struct task_group *tg, long rt_runtime_us);
extern int sched_group_set_rt_period(struct task_group *tg, u64 rt_period_us);
extern long sched_group_rt_runtime(struct task_group *tg);
@@ -2926,6 +2923,8 @@ extern void resched_curr(struct rq *rq);
extern void resched_curr_lazy(struct rq *rq);
extern void resched_cpu(int cpu);
+extern void init_dl_bandwidth(struct dl_bandwidth *dl_b, u64 period, u64 runtime,
+ struct task_group *active_context);
extern void init_dl_entity(struct sched_dl_entity *dl_se);
extern void init_cfs_throttle_work(struct task_struct *p);
@@ -3349,6 +3348,9 @@ extern void set_rq_offline(struct rq *rq);
extern bool sched_smp_initialized;
+extern const struct dl_bandwidth *dl_bandwidth_read(struct task_group *tg);
+extern struct dl_bandwidth *dl_bandwidth_write(struct task_group *tg);
+
#ifdef CONFIG_RT_GROUP_SCHED
static inline struct task_struct *rt_task_of(struct sched_rt_entity *rt_se)
{
@@ -3372,6 +3374,24 @@ static inline struct rt_rq *rt_rq_of_se(struct sched_rt_entity *rt_se)
WARN_ON(!rt_group_sched_enabled() && rt_se->rt_rq->tg != &root_task_group);
return rt_se->rt_rq;
}
+
+static inline int is_dl_group(struct rt_rq *rt_rq)
+{
+ return rt_rq->tg != &root_task_group;
+}
+
+/*
+ * Return the scheduling entity of this group of tasks.
+ */
+static inline struct sched_dl_entity *dl_group_of(struct rt_rq *rt_rq)
+{
+ if (WARN_ON_ONCE(!is_dl_group(rt_rq)))
+ return NULL;
+
+ return rt_rq->tg->dl_se[rq_of_rt_rq(rt_rq)->cpu];
+}
+
+#define dl_bw_lock_of_tg(tg) (&(tg)->dl_bandwidth.dl_runtime_lock)
#else
static inline struct task_struct *rt_task_of(struct sched_rt_entity *rt_se)
{
@@ -3394,6 +3414,18 @@ static inline struct rt_rq *rt_rq_of_se(struct sched_rt_entity *rt_se)
return &rq->rt;
}
+
+static inline int is_dl_group(struct rt_rq *rt_rq)
+{
+ return 0;
+}
+
+static inline struct sched_dl_entity *dl_group_of(struct rt_rq *rt_rq)
+{
+ return NULL;
+}
+
+#define dl_bw_lock_of_tg(tg) ((raw_spinlock_t*)NULL)
#endif
DEFINE_LOCK_GUARD_2(double_rq_lock, struct rq,
--
2.54.0
^ permalink raw reply related
* [RFC PATCH v6 09/25] sched/rt: Introduce HCBS specific structs in task_group
From: Yuri Andriaccio @ 2026-06-08 12:15 UTC (permalink / raw)
To: Ingo Molnar, Peter Zijlstra, Juri Lelli, Vincent Guittot,
Dietmar Eggemann, Steven Rostedt, Ben Segall, Mel Gorman,
Valentin Schneider, Tejun Heo, Johannes Weiner,
Michal Koutný
Cc: cgroups, linux-kernel, Luca Abeni, Yuri Andriaccio
In-Reply-To: <20260608121546.69910-1-yurand2000@gmail.com>
From: luca abeni <luca.abeni@santannapisa.it>
Add an array of sched_dl_entity objects in task_group.
Create the dl_bandwidth struct and add a field for it in task_group.
Add a rq pointer field in struct rt_rq.
---
For each CPU on the host system, the task_group manages a sched_dl_entity and
a rt_rq object, which in turn keeps a pointer to its locally managed runqueue.
The sched_dl_entity object manages the deadline server which will be scheduled
for execution on the CPU, while the rt_rq object is instead used to reference
the local runqueue's specific data and entities and it is used when an actual
task must be scheduled when the CPU is given to the dl_server.
The dl_bandwidth object keeps track of the currently allocated bandwidth for
the cgroup and the currently active context. RT-cgroups can either run tasks
themselves or can delegate the scheduling of their tasks to their parent, the
active_context field keeps track of which cgroup is serving the tasks.
Co-developed-by: Alessio Balsini <a.balsini@sssup.it>
Signed-off-by: Alessio Balsini <a.balsini@sssup.it>
Co-developed-by: Andrea Parri <parri.andrea@gmail.com>
Signed-off-by: Andrea Parri <parri.andrea@gmail.com>
Co-developed-by: Yuri Andriaccio <yurand2000@gmail.com>
Signed-off-by: Yuri Andriaccio <yurand2000@gmail.com>
Signed-off-by: luca abeni <luca.abeni@santannapisa.it>
---
kernel/sched/sched.h | 22 ++++++++++++++++++++++
1 file changed, 22 insertions(+)
diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h
index 3aa29fe932fc..f3c259ab9344 100644
--- a/kernel/sched/sched.h
+++ b/kernel/sched/sched.h
@@ -322,6 +322,15 @@ struct rt_bandwidth {
unsigned int rt_period_active;
};
+struct dl_bandwidth {
+ raw_spinlock_t dl_runtime_lock;
+ u64 dl_runtime;
+ u64 dl_internal_runtime;
+ u64 dl_period;
+ struct task_group *active_context;
+};
+
+
static inline int dl_bandwidth_enabled(void)
{
return sysctl_sched_rt_runtime >= 0;
@@ -495,10 +504,17 @@ struct task_group {
#endif /* CONFIG_FAIR_GROUP_SCHED */
#ifdef CONFIG_RT_GROUP_SCHED
+ /*
+ * Each task group manages a different scheduling entity per CPU, i.e. a
+ * different deadline server, and a runqueue per CPU. All the dl-servers
+ * share the same dl_bandwidth object.
+ */
struct sched_rt_entity **rt_se;
+ struct sched_dl_entity **dl_se;
struct rt_rq **rt_rq;
struct rt_bandwidth rt_bandwidth;
+ struct dl_bandwidth dl_bandwidth;
#endif
struct scx_task_group scx;
@@ -861,6 +877,12 @@ struct rt_rq {
#ifdef CONFIG_CGROUP_SCHED
struct task_group *tg; /* this tg has "this" rt_rq on given CPU for runnable entities */
#endif
+
+ /*
+ * The cgroup's served runqueue if the rt_rq entity belongs to a cgroup,
+ * otherwise the top-level global runqueue.
+ */
+ struct rq *rq;
};
static inline bool rt_rq_is_runnable(struct rt_rq *rt_rq)
--
2.54.0
^ permalink raw reply related
* [RFC PATCH v6 07/25] sched/rt: Disable RT_GROUP_SCHED
From: Yuri Andriaccio @ 2026-06-08 12:15 UTC (permalink / raw)
To: Ingo Molnar, Peter Zijlstra, Juri Lelli, Vincent Guittot,
Dietmar Eggemann, Steven Rostedt, Ben Segall, Mel Gorman,
Valentin Schneider, Tejun Heo, Johannes Weiner,
Michal Koutný
Cc: cgroups, linux-kernel, Luca Abeni, Yuri Andriaccio
In-Reply-To: <20260608121546.69910-1-yurand2000@gmail.com>
Disable the old RT_GROUP_SCHED scheduler. Note that this does not
completely remove all the RT_GROUP_SCHED functionality, just unhooks it
and removes most of the relevant functions. Some of the RT_GROUP_SCHED
functions are kept because they will be adapted for the HCBS scheduling.
Most notably:
- Disable the initialization of the rt_bandwidth for group scheduling.
- Unhook any functionality for RT_GROUP_SCHED in normal rt.c code, leaving
only non-group functionality.
- Remove group related field initialization in init_rt_rq().
- Remove all the unhooked (and so unused) functions from RT_GROUP_SCHED.
- Remove all allocation/deallocation code for rt-groups, always returning
failure on allocation.
- Update inc/dec_rt_tasks active tasks' counters, as rt scheduling
entities now only represent a single task, and not a group of tasks
anymore.
- Remove unused rq_of_rt_se function.
Signed-off-by: Yuri Andriaccio <yurand2000@gmail.com>
---
kernel/sched/core.c | 6 -
kernel/sched/deadline.c | 34 --
kernel/sched/debug.c | 6 -
kernel/sched/rt.c | 852 ++--------------------------------------
kernel/sched/sched.h | 32 +-
kernel/sched/syscalls.c | 13 -
6 files changed, 28 insertions(+), 915 deletions(-)
diff --git a/kernel/sched/core.c b/kernel/sched/core.c
index b8871449d3c6..e38ca8192d2d 100644
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -8922,11 +8922,6 @@ void __init sched_init(void)
init_defrootdomain();
-#ifdef CONFIG_RT_GROUP_SCHED
- init_rt_bandwidth(&root_task_group.rt_bandwidth,
- global_rt_period(), global_rt_runtime());
-#endif /* CONFIG_RT_GROUP_SCHED */
-
#ifdef CONFIG_CGROUP_SCHED
task_group_cache = KMEM_CACHE(task_group, 0);
@@ -8978,7 +8973,6 @@ void __init sched_init(void)
* starts working after scheduler_running, which is not the case
* yet.
*/
- rq->rt.rt_runtime = global_rt_runtime();
init_tg_rt_entry(&root_task_group, &rq->rt, NULL, i, NULL);
#endif
rq->next_class = &idle_sched_class;
diff --git a/kernel/sched/deadline.c b/kernel/sched/deadline.c
index b3059658a74a..c12882348a03 100644
--- a/kernel/sched/deadline.c
+++ b/kernel/sched/deadline.c
@@ -1533,40 +1533,6 @@ static void update_curr_dl_se(struct rq *rq, struct sched_dl_entity *dl_se, s64
} else {
trace_sched_dl_update_tp(dl_se, cpu_of(rq), dl_get_type(dl_se, rq));
}
-
- /*
- * The dl_server does not account for real-time workload because it
- * is running fair work.
- */
- if (dl_se->dl_server)
- return;
-
-#ifdef CONFIG_RT_GROUP_SCHED
- /*
- * Because -- for now -- we share the rt bandwidth, we need to
- * account our runtime there too, otherwise actual rt tasks
- * would be able to exceed the shared quota.
- *
- * Account to the root rt group for now.
- *
- * The solution we're working towards is having the RT groups scheduled
- * using deadline servers -- however there's a few nasties to figure
- * out before that can happen.
- */
- if (rt_bandwidth_enabled()) {
- struct rt_rq *rt_rq = &rq->rt;
-
- raw_spin_lock(&rt_rq->rt_runtime_lock);
- /*
- * We'll let actual RT tasks worry about the overflow here, we
- * have our own CBS to keep us inline; only account when RT
- * bandwidth is relevant.
- */
- if (sched_rt_bandwidth_account(rt_rq))
- rt_rq->rt_time += delta_exec;
- raw_spin_unlock(&rt_rq->rt_runtime_lock);
- }
-#endif /* CONFIG_RT_GROUP_SCHED */
}
/*
diff --git a/kernel/sched/debug.c b/kernel/sched/debug.c
index 74c1617cf652..40cc905a65b7 100644
--- a/kernel/sched/debug.c
+++ b/kernel/sched/debug.c
@@ -1009,12 +1009,6 @@ void print_rt_rq(struct seq_file *m, int cpu, struct rt_rq *rt_rq)
PU(rt_nr_running);
-#ifdef CONFIG_RT_GROUP_SCHED
- P(rt_throttled);
- PN(rt_time);
- PN(rt_runtime);
-#endif
-
#undef PN
#undef PU
#undef P
diff --git a/kernel/sched/rt.c b/kernel/sched/rt.c
index fe5b58f8fc69..7b526a86083c 100644
--- a/kernel/sched/rt.c
+++ b/kernel/sched/rt.c
@@ -82,115 +82,19 @@ void init_rt_rq(struct rt_rq *rt_rq)
rt_rq->highest_prio.next = MAX_RT_PRIO-1;
rt_rq->overloaded = 0;
plist_head_init(&rt_rq->pushable_tasks);
- /* We start is dequeued state, because no RT tasks are queued */
- rt_rq->rt_queued = 0;
-
-#ifdef CONFIG_RT_GROUP_SCHED
- rt_rq->rt_time = 0;
- rt_rq->rt_throttled = 0;
- rt_rq->rt_runtime = 0;
- raw_spin_lock_init(&rt_rq->rt_runtime_lock);
- rt_rq->tg = &root_task_group;
-#endif
}
#ifdef CONFIG_RT_GROUP_SCHED
-static int do_sched_rt_period_timer(struct rt_bandwidth *rt_b, int overrun);
-
-static enum hrtimer_restart sched_rt_period_timer(struct hrtimer *timer)
-{
- struct rt_bandwidth *rt_b =
- container_of(timer, struct rt_bandwidth, rt_period_timer);
- int idle = 0;
- int overrun;
-
- raw_spin_lock(&rt_b->rt_runtime_lock);
- for (;;) {
- overrun = hrtimer_forward_now(timer, rt_b->rt_period);
- if (!overrun)
- break;
-
- raw_spin_unlock(&rt_b->rt_runtime_lock);
- idle = do_sched_rt_period_timer(rt_b, overrun);
- raw_spin_lock(&rt_b->rt_runtime_lock);
- }
- if (idle)
- rt_b->rt_period_active = 0;
- raw_spin_unlock(&rt_b->rt_runtime_lock);
-
- return idle ? HRTIMER_NORESTART : HRTIMER_RESTART;
-}
-
-void init_rt_bandwidth(struct rt_bandwidth *rt_b, u64 period, u64 runtime)
-{
- rt_b->rt_period = ns_to_ktime(period);
- rt_b->rt_runtime = runtime;
-
- raw_spin_lock_init(&rt_b->rt_runtime_lock);
-
- hrtimer_setup(&rt_b->rt_period_timer, sched_rt_period_timer, CLOCK_MONOTONIC,
- HRTIMER_MODE_REL_HARD);
-}
-
-static inline void do_start_rt_bandwidth(struct rt_bandwidth *rt_b)
-{
- raw_spin_lock(&rt_b->rt_runtime_lock);
- if (!rt_b->rt_period_active) {
- rt_b->rt_period_active = 1;
- /*
- * SCHED_DEADLINE updates the bandwidth, as a run away
- * RT task with a DL task could hog a CPU. But DL does
- * not reset the period. If a deadline task was running
- * without an RT task running, it can cause RT tasks to
- * throttle when they start up. Kick the timer right away
- * to update the period.
- */
- hrtimer_forward_now(&rt_b->rt_period_timer, ns_to_ktime(0));
- hrtimer_start_expires(&rt_b->rt_period_timer,
- HRTIMER_MODE_ABS_PINNED_HARD);
- }
- raw_spin_unlock(&rt_b->rt_runtime_lock);
-}
-
-static void start_rt_bandwidth(struct rt_bandwidth *rt_b)
-{
- if (!rt_bandwidth_enabled() || rt_b->rt_runtime == RUNTIME_INF)
- return;
-
- do_start_rt_bandwidth(rt_b);
-}
-
-static void destroy_rt_bandwidth(struct rt_bandwidth *rt_b)
-{
- hrtimer_cancel(&rt_b->rt_period_timer);
-}
-
void unregister_rt_sched_group(struct task_group *tg)
{
- if (!rt_group_sched_enabled())
- return;
- if (tg->rt_se)
- destroy_rt_bandwidth(&tg->rt_bandwidth);
}
void free_rt_sched_group(struct task_group *tg)
{
- int i;
-
if (!rt_group_sched_enabled())
return;
-
- for_each_possible_cpu(i) {
- if (tg->rt_rq)
- kfree(tg->rt_rq[i]);
- if (tg->rt_se)
- kfree(tg->rt_se[i]);
- }
-
- kfree(tg->rt_rq);
- kfree(tg->rt_se);
}
void init_tg_rt_entry(struct task_group *tg, struct rt_rq *rt_rq,
@@ -200,66 +104,19 @@ void init_tg_rt_entry(struct task_group *tg, struct rt_rq *rt_rq,
struct rq *rq = cpu_rq(cpu);
rt_rq->highest_prio.curr = MAX_RT_PRIO-1;
- rt_rq->rt_nr_boosted = 0;
rt_rq->rq = rq;
rt_rq->tg = tg;
tg->rt_rq[cpu] = rt_rq;
tg->rt_se[cpu] = rt_se;
-
- if (!rt_se)
- return;
-
- if (!parent)
- rt_se->rt_rq = &rq->rt;
- else
- rt_se->rt_rq = parent->my_q;
-
- rt_se->my_q = rt_rq;
- rt_se->parent = parent;
- INIT_LIST_HEAD(&rt_se->run_list);
}
int alloc_rt_sched_group(struct task_group *tg, struct task_group *parent)
{
- struct rt_rq *rt_rq;
- struct sched_rt_entity *rt_se;
- int i;
-
if (!rt_group_sched_enabled())
return 1;
- tg->rt_rq = kzalloc_objs(rt_rq, nr_cpu_ids);
- if (!tg->rt_rq)
- goto err;
- tg->rt_se = kzalloc_objs(rt_se, nr_cpu_ids);
- if (!tg->rt_se)
- goto err;
-
- init_rt_bandwidth(&tg->rt_bandwidth, ktime_to_ns(global_rt_period()), 0);
-
- for_each_possible_cpu(i) {
- rt_rq = kzalloc_node(sizeof(struct rt_rq),
- GFP_KERNEL, cpu_to_node(i));
- if (!rt_rq)
- goto err;
-
- rt_se = kzalloc_node(sizeof(struct sched_rt_entity),
- GFP_KERNEL, cpu_to_node(i));
- if (!rt_se)
- goto err_free_rq;
-
- init_rt_rq(rt_rq);
- rt_rq->rt_runtime = tg->rt_bandwidth.rt_runtime;
- init_tg_rt_entry(tg, rt_rq, rt_se, i, parent->rt_se[i]);
- }
-
return 1;
-
-err_free_rq:
- kfree(rt_rq);
-err:
- return 0;
}
#else /* !CONFIG_RT_GROUP_SCHED: */
@@ -389,9 +246,6 @@ static void dequeue_pushable_task(struct rt_rq *rt_rq, struct task_struct *p)
}
}
-static void enqueue_top_rt_rq(struct rt_rq *rt_rq);
-static void dequeue_top_rt_rq(struct rt_rq *rt_rq, unsigned int count);
-
static inline int on_rt_rq(struct sched_rt_entity *rt_se)
{
return rt_se->on_rq;
@@ -438,16 +292,6 @@ static inline bool rt_task_fits_capacity(struct task_struct *p, int cpu)
#ifdef CONFIG_RT_GROUP_SCHED
-static inline u64 sched_rt_runtime(struct rt_rq *rt_rq)
-{
- return rt_rq->rt_runtime;
-}
-
-static inline u64 sched_rt_period(struct rt_rq *rt_rq)
-{
- return ktime_to_ns(rt_rq->tg->rt_bandwidth.rt_period);
-}
-
typedef struct task_group *rt_rq_iter_t;
static inline struct task_group *next_task_group(struct task_group *tg)
@@ -473,457 +317,20 @@ static inline struct task_group *next_task_group(struct task_group *tg)
iter && (rt_rq = iter->rt_rq[cpu_of(rq)]); \
iter = next_task_group(iter))
-#define for_each_sched_rt_entity(rt_se) \
- for (; rt_se; rt_se = rt_se->parent)
-
-static inline struct rt_rq *group_rt_rq(struct sched_rt_entity *rt_se)
-{
- return rt_se->my_q;
-}
-
static void enqueue_rt_entity(struct sched_rt_entity *rt_se, unsigned int flags);
static void dequeue_rt_entity(struct sched_rt_entity *rt_se, unsigned int flags);
-static void sched_rt_rq_enqueue(struct rt_rq *rt_rq)
-{
- struct task_struct *donor = rq_of_rt_rq(rt_rq)->donor;
- struct rq *rq = rq_of_rt_rq(rt_rq);
- struct sched_rt_entity *rt_se;
-
- int cpu = cpu_of(rq);
-
- rt_se = rt_rq->tg->rt_se[cpu];
-
- if (rt_rq->rt_nr_running) {
- if (!rt_se)
- enqueue_top_rt_rq(rt_rq);
- else if (!on_rt_rq(rt_se))
- enqueue_rt_entity(rt_se, 0);
-
- if (rt_rq->highest_prio.curr < donor->prio)
- resched_curr(rq);
- }
-}
-
-static void sched_rt_rq_dequeue(struct rt_rq *rt_rq)
-{
- struct sched_rt_entity *rt_se;
- int cpu = cpu_of(rq_of_rt_rq(rt_rq));
-
- rt_se = rt_rq->tg->rt_se[cpu];
-
- if (!rt_se) {
- dequeue_top_rt_rq(rt_rq, rt_rq->rt_nr_running);
- /* Kick cpufreq (see the comment in kernel/sched/sched.h). */
- cpufreq_update_util(rq_of_rt_rq(rt_rq), 0);
- }
- else if (on_rt_rq(rt_se))
- dequeue_rt_entity(rt_se, 0);
-}
-
-static inline int rt_rq_throttled(struct rt_rq *rt_rq)
-{
- return rt_rq->rt_throttled && !rt_rq->rt_nr_boosted;
-}
-
-static int rt_se_boosted(struct sched_rt_entity *rt_se)
-{
- struct rt_rq *rt_rq = group_rt_rq(rt_se);
- struct task_struct *p;
-
- if (rt_rq)
- return !!rt_rq->rt_nr_boosted;
-
- p = rt_task_of(rt_se);
- return p->prio != p->normal_prio;
-}
-
-static inline const struct cpumask *sched_rt_period_mask(void)
-{
- return this_rq()->rd->span;
-}
-
-static inline
-struct rt_rq *sched_rt_period_rt_rq(struct rt_bandwidth *rt_b, int cpu)
-{
- return container_of(rt_b, struct task_group, rt_bandwidth)->rt_rq[cpu];
-}
-
-static inline struct rt_bandwidth *sched_rt_bandwidth(struct rt_rq *rt_rq)
-{
- return &rt_rq->tg->rt_bandwidth;
-}
-
-bool sched_rt_bandwidth_account(struct rt_rq *rt_rq)
-{
- struct rt_bandwidth *rt_b = sched_rt_bandwidth(rt_rq);
-
- return (hrtimer_active(&rt_b->rt_period_timer) ||
- rt_rq->rt_time < rt_b->rt_runtime);
-}
-
-/*
- * We ran out of runtime, see if we can borrow some from our neighbours.
- */
-static void do_balance_runtime(struct rt_rq *rt_rq)
-{
- struct rt_bandwidth *rt_b = sched_rt_bandwidth(rt_rq);
- struct root_domain *rd = rq_of_rt_rq(rt_rq)->rd;
- int i, weight;
- u64 rt_period;
-
- weight = cpumask_weight(rd->span);
-
- raw_spin_lock(&rt_b->rt_runtime_lock);
- rt_period = ktime_to_ns(rt_b->rt_period);
- for_each_cpu(i, rd->span) {
- struct rt_rq *iter = sched_rt_period_rt_rq(rt_b, i);
- s64 diff;
-
- if (iter == rt_rq)
- continue;
-
- raw_spin_lock(&iter->rt_runtime_lock);
- /*
- * Either all rqs have inf runtime and there's nothing to steal
- * or __disable_runtime() below sets a specific rq to inf to
- * indicate its been disabled and disallow stealing.
- */
- if (iter->rt_runtime == RUNTIME_INF)
- goto next;
-
- /*
- * From runqueues with spare time, take 1/n part of their
- * spare time, but no more than our period.
- */
- diff = iter->rt_runtime - iter->rt_time;
- if (diff > 0) {
- diff = div_u64((u64)diff, weight);
- if (rt_rq->rt_runtime + diff > rt_period)
- diff = rt_period - rt_rq->rt_runtime;
- iter->rt_runtime -= diff;
- rt_rq->rt_runtime += diff;
- if (rt_rq->rt_runtime == rt_period) {
- raw_spin_unlock(&iter->rt_runtime_lock);
- break;
- }
- }
-next:
- raw_spin_unlock(&iter->rt_runtime_lock);
- }
- raw_spin_unlock(&rt_b->rt_runtime_lock);
-}
-
-/*
- * Ensure this RQ takes back all the runtime it lend to its neighbours.
- */
-static void __disable_runtime(struct rq *rq)
-{
- struct root_domain *rd = rq->rd;
- rt_rq_iter_t iter;
- struct rt_rq *rt_rq;
-
- if (unlikely(!scheduler_running))
- return;
-
- for_each_rt_rq(rt_rq, iter, rq) {
- struct rt_bandwidth *rt_b = sched_rt_bandwidth(rt_rq);
- s64 want;
- int i;
-
- raw_spin_lock(&rt_b->rt_runtime_lock);
- raw_spin_lock(&rt_rq->rt_runtime_lock);
- /*
- * Either we're all inf and nobody needs to borrow, or we're
- * already disabled and thus have nothing to do, or we have
- * exactly the right amount of runtime to take out.
- */
- if (rt_rq->rt_runtime == RUNTIME_INF ||
- rt_rq->rt_runtime == rt_b->rt_runtime)
- goto balanced;
- raw_spin_unlock(&rt_rq->rt_runtime_lock);
-
- /*
- * Calculate the difference between what we started out with
- * and what we current have, that's the amount of runtime
- * we lend and now have to reclaim.
- */
- want = rt_b->rt_runtime - rt_rq->rt_runtime;
-
- /*
- * Greedy reclaim, take back as much as we can.
- */
- for_each_cpu(i, rd->span) {
- struct rt_rq *iter = sched_rt_period_rt_rq(rt_b, i);
- s64 diff;
-
- /*
- * Can't reclaim from ourselves or disabled runqueues.
- */
- if (iter == rt_rq || iter->rt_runtime == RUNTIME_INF)
- continue;
-
- raw_spin_lock(&iter->rt_runtime_lock);
- if (want > 0) {
- diff = min_t(s64, iter->rt_runtime, want);
- iter->rt_runtime -= diff;
- want -= diff;
- } else {
- iter->rt_runtime -= want;
- want -= want;
- }
- raw_spin_unlock(&iter->rt_runtime_lock);
-
- if (!want)
- break;
- }
-
- raw_spin_lock(&rt_rq->rt_runtime_lock);
- /*
- * We cannot be left wanting - that would mean some runtime
- * leaked out of the system.
- */
- WARN_ON_ONCE(want);
-balanced:
- /*
- * Disable all the borrow logic by pretending we have inf
- * runtime - in which case borrowing doesn't make sense.
- */
- rt_rq->rt_runtime = RUNTIME_INF;
- rt_rq->rt_throttled = 0;
- raw_spin_unlock(&rt_rq->rt_runtime_lock);
- raw_spin_unlock(&rt_b->rt_runtime_lock);
-
- /* Make rt_rq available for pick_next_task() */
- sched_rt_rq_enqueue(rt_rq);
- }
-}
-
-static void __enable_runtime(struct rq *rq)
-{
- rt_rq_iter_t iter;
- struct rt_rq *rt_rq;
-
- if (unlikely(!scheduler_running))
- return;
-
- /*
- * Reset each runqueue's bandwidth settings
- */
- for_each_rt_rq(rt_rq, iter, rq) {
- struct rt_bandwidth *rt_b = sched_rt_bandwidth(rt_rq);
-
- raw_spin_lock(&rt_b->rt_runtime_lock);
- raw_spin_lock(&rt_rq->rt_runtime_lock);
- rt_rq->rt_runtime = rt_b->rt_runtime;
- rt_rq->rt_time = 0;
- rt_rq->rt_throttled = 0;
- raw_spin_unlock(&rt_rq->rt_runtime_lock);
- raw_spin_unlock(&rt_b->rt_runtime_lock);
- }
-}
-
-static void balance_runtime(struct rt_rq *rt_rq)
-{
- if (!sched_feat(RT_RUNTIME_SHARE))
- return;
-
- if (rt_rq->rt_time > rt_rq->rt_runtime) {
- raw_spin_unlock(&rt_rq->rt_runtime_lock);
- do_balance_runtime(rt_rq);
- raw_spin_lock(&rt_rq->rt_runtime_lock);
- }
-}
-
-static int do_sched_rt_period_timer(struct rt_bandwidth *rt_b, int overrun)
-{
- int i, idle = 1, throttled = 0;
- const struct cpumask *span;
-
- span = sched_rt_period_mask();
-
- /*
- * FIXME: isolated CPUs should really leave the root task group,
- * whether they are isolcpus or were isolated via cpusets, lest
- * the timer run on a CPU which does not service all runqueues,
- * potentially leaving other CPUs indefinitely throttled. If
- * isolation is really required, the user will turn the throttle
- * off to kill the perturbations it causes anyway. Meanwhile,
- * this maintains functionality for boot and/or troubleshooting.
- */
- if (rt_b == &root_task_group.rt_bandwidth)
- span = cpu_online_mask;
-
- for_each_cpu(i, span) {
- int enqueue = 0;
- struct rt_rq *rt_rq = sched_rt_period_rt_rq(rt_b, i);
- struct rq *rq = rq_of_rt_rq(rt_rq);
- struct rq_flags rf;
- int skip;
-
- /*
- * When span == cpu_online_mask, taking each rq->lock
- * can be time-consuming. Try to avoid it when possible.
- */
- raw_spin_lock(&rt_rq->rt_runtime_lock);
- if (!sched_feat(RT_RUNTIME_SHARE) && rt_rq->rt_runtime != RUNTIME_INF)
- rt_rq->rt_runtime = rt_b->rt_runtime;
- skip = !rt_rq->rt_time && !rt_rq->rt_nr_running;
- raw_spin_unlock(&rt_rq->rt_runtime_lock);
- if (skip)
- continue;
-
- rq_lock(rq, &rf);
- update_rq_clock(rq);
-
- if (rt_rq->rt_time) {
- u64 runtime;
-
- raw_spin_lock(&rt_rq->rt_runtime_lock);
- if (rt_rq->rt_throttled)
- balance_runtime(rt_rq);
- runtime = rt_rq->rt_runtime;
- rt_rq->rt_time -= min(rt_rq->rt_time, overrun*runtime);
- if (rt_rq->rt_throttled && rt_rq->rt_time < runtime) {
- rt_rq->rt_throttled = 0;
- enqueue = 1;
-
- /*
- * When we're idle and a woken (rt) task is
- * throttled wakeup_preempt() will set
- * skip_update and the time between the wakeup
- * and this unthrottle will get accounted as
- * 'runtime'.
- */
- if (rt_rq->rt_nr_running && rq->curr == rq->idle)
- rq_clock_cancel_skipupdate(rq);
- }
- if (rt_rq->rt_time || rt_rq->rt_nr_running)
- idle = 0;
- raw_spin_unlock(&rt_rq->rt_runtime_lock);
- } else if (rt_rq->rt_nr_running) {
- idle = 0;
- if (!rt_rq_throttled(rt_rq))
- enqueue = 1;
- }
- if (rt_rq->rt_throttled)
- throttled = 1;
-
- if (enqueue)
- sched_rt_rq_enqueue(rt_rq);
- rq_unlock(rq, &rf);
- }
-
- if (!throttled && (!rt_bandwidth_enabled() || rt_b->rt_runtime == RUNTIME_INF))
- return 1;
-
- return idle;
-}
-
-static int sched_rt_runtime_exceeded(struct rt_rq *rt_rq)
-{
- u64 runtime = sched_rt_runtime(rt_rq);
-
- if (rt_rq->rt_throttled)
- return rt_rq_throttled(rt_rq);
-
- if (runtime >= sched_rt_period(rt_rq))
- return 0;
-
- balance_runtime(rt_rq);
- runtime = sched_rt_runtime(rt_rq);
- if (runtime == RUNTIME_INF)
- return 0;
-
- if (rt_rq->rt_time > runtime) {
- struct rt_bandwidth *rt_b = sched_rt_bandwidth(rt_rq);
-
- /*
- * Don't actually throttle groups that have no runtime assigned
- * but accrue some time due to boosting.
- */
- if (likely(rt_b->rt_runtime)) {
- rt_rq->rt_throttled = 1;
- printk_deferred_once("sched: RT throttling activated\n");
- } else {
- /*
- * In case we did anyway, make it go away,
- * replenishment is a joke, since it will replenish us
- * with exactly 0 ns.
- */
- rt_rq->rt_time = 0;
- }
-
- if (rt_rq_throttled(rt_rq)) {
- sched_rt_rq_dequeue(rt_rq);
- return 1;
- }
- }
-
- return 0;
-}
-
-#else /* !CONFIG_RT_GROUP_SCHED: */
+#else /* !CONFIG_RT_GROUP_SCHED */
typedef struct rt_rq *rt_rq_iter_t;
#define for_each_rt_rq(rt_rq, iter, rq) \
for ((void) iter, rt_rq = &rq->rt; rt_rq; rt_rq = NULL)
-#define for_each_sched_rt_entity(rt_se) \
- for (; rt_se; rt_se = NULL)
-
-static inline struct rt_rq *group_rt_rq(struct sched_rt_entity *rt_se)
-{
- return NULL;
-}
-
-static inline void sched_rt_rq_enqueue(struct rt_rq *rt_rq)
-{
- struct rq *rq = rq_of_rt_rq(rt_rq);
-
- if (!rt_rq->rt_nr_running)
- return;
-
- enqueue_top_rt_rq(rt_rq);
- resched_curr(rq);
-}
-
-static inline void sched_rt_rq_dequeue(struct rt_rq *rt_rq)
-{
- dequeue_top_rt_rq(rt_rq, rt_rq->rt_nr_running);
-}
-
-static inline int rt_rq_throttled(struct rt_rq *rt_rq)
-{
- return false;
-}
-
-static inline const struct cpumask *sched_rt_period_mask(void)
-{
- return cpu_online_mask;
-}
-
-static inline
-struct rt_rq *sched_rt_period_rt_rq(struct rt_bandwidth *rt_b, int cpu)
-{
- return &cpu_rq(cpu)->rt;
-}
-
-static void __enable_runtime(struct rq *rq) { }
-static void __disable_runtime(struct rq *rq) { }
-
-#endif /* !CONFIG_RT_GROUP_SCHED */
+#endif /* CONFIG_RT_GROUP_SCHED */
static inline int rt_se_prio(struct sched_rt_entity *rt_se)
{
-#ifdef CONFIG_RT_GROUP_SCHED
- struct rt_rq *rt_rq = group_rt_rq(rt_se);
-
- if (rt_rq)
- return rt_rq->highest_prio.curr;
-#endif
-
return rt_task_of(rt_se)->prio;
}
@@ -943,67 +350,8 @@ static void update_curr_rt(struct rq *rq)
if (unlikely(delta_exec <= 0))
return;
-#ifdef CONFIG_RT_GROUP_SCHED
- struct sched_rt_entity *rt_se = &donor->rt;
-
if (!rt_bandwidth_enabled())
return;
-
- for_each_sched_rt_entity(rt_se) {
- struct rt_rq *rt_rq = rt_rq_of_se(rt_se);
- int exceeded;
-
- if (sched_rt_runtime(rt_rq) != RUNTIME_INF) {
- raw_spin_lock(&rt_rq->rt_runtime_lock);
- rt_rq->rt_time += delta_exec;
- exceeded = sched_rt_runtime_exceeded(rt_rq);
- if (exceeded)
- resched_curr(rq);
- raw_spin_unlock(&rt_rq->rt_runtime_lock);
- if (exceeded)
- do_start_rt_bandwidth(sched_rt_bandwidth(rt_rq));
- }
- }
-#endif /* CONFIG_RT_GROUP_SCHED */
-}
-
-static void
-dequeue_top_rt_rq(struct rt_rq *rt_rq, unsigned int count)
-{
- struct rq *rq = rq_of_rt_rq(rt_rq);
-
- BUG_ON(&rq->rt != rt_rq);
-
- if (!rt_rq->rt_queued)
- return;
-
- BUG_ON(!rq->nr_running);
-
- sub_nr_running(rq, count);
- rt_rq->rt_queued = 0;
-
-}
-
-static void
-enqueue_top_rt_rq(struct rt_rq *rt_rq)
-{
- struct rq *rq = rq_of_rt_rq(rt_rq);
-
- BUG_ON(&rq->rt != rt_rq);
-
- if (rt_rq->rt_queued)
- return;
-
- if (rt_rq_throttled(rt_rq))
- return;
-
- if (rt_rq->rt_nr_running) {
- add_nr_running(rq, rt_rq->rt_nr_running);
- rt_rq->rt_queued = 1;
- }
-
- /* Kick cpufreq (see the comment in kernel/sched/sched.h). */
- cpufreq_update_util(rq, 0);
}
static void
@@ -1074,58 +422,11 @@ dec_rt_prio(struct rt_rq *rt_rq, int prio)
dec_rt_prio_smp(rt_rq, prio, prev_prio);
}
-#ifdef CONFIG_RT_GROUP_SCHED
-
-static void
-inc_rt_group(struct sched_rt_entity *rt_se, struct rt_rq *rt_rq)
-{
- if (rt_se_boosted(rt_se))
- rt_rq->rt_nr_boosted++;
-
- start_rt_bandwidth(&rt_rq->tg->rt_bandwidth);
-}
-
-static void
-dec_rt_group(struct sched_rt_entity *rt_se, struct rt_rq *rt_rq)
-{
- if (rt_se_boosted(rt_se))
- rt_rq->rt_nr_boosted--;
-
- WARN_ON(!rt_rq->rt_nr_running && rt_rq->rt_nr_boosted);
-}
-
-#else /* !CONFIG_RT_GROUP_SCHED: */
-
-static void
-inc_rt_group(struct sched_rt_entity *rt_se, struct rt_rq *rt_rq)
-{
-}
-
-static inline
-void dec_rt_group(struct sched_rt_entity *rt_se, struct rt_rq *rt_rq) {}
-
-#endif /* !CONFIG_RT_GROUP_SCHED */
-
static inline
-unsigned int rt_se_nr_running(struct sched_rt_entity *rt_se)
+unsigned int is_rr_task(struct sched_rt_entity *rt_se)
{
- struct rt_rq *group_rq = group_rt_rq(rt_se);
-
- if (group_rq)
- return group_rq->rt_nr_running;
- else
- return 1;
-}
-
-static inline
-unsigned int rt_se_rr_nr_running(struct sched_rt_entity *rt_se)
-{
- struct rt_rq *group_rq = group_rt_rq(rt_se);
struct task_struct *tsk;
- if (group_rq)
- return group_rq->rr_nr_running;
-
tsk = rt_task_of(rt_se);
return (tsk->policy == SCHED_RR) ? 1 : 0;
@@ -1134,26 +435,21 @@ unsigned int rt_se_rr_nr_running(struct sched_rt_entity *rt_se)
static inline
void inc_rt_tasks(struct sched_rt_entity *rt_se, struct rt_rq *rt_rq)
{
- int prio = rt_se_prio(rt_se);
-
- WARN_ON(!rt_prio(prio));
- rt_rq->rt_nr_running += rt_se_nr_running(rt_se);
- rt_rq->rr_nr_running += rt_se_rr_nr_running(rt_se);
+ WARN_ON(!rt_prio(rt_se_prio(rt_se)));
+ rt_rq->rt_nr_running += 1;
+ rt_rq->rr_nr_running += is_rr_task(rt_se);
- inc_rt_prio(rt_rq, prio);
- inc_rt_group(rt_se, rt_rq);
+ inc_rt_prio(rt_rq, rt_se_prio(rt_se));
}
static inline
void dec_rt_tasks(struct sched_rt_entity *rt_se, struct rt_rq *rt_rq)
{
WARN_ON(!rt_prio(rt_se_prio(rt_se)));
- WARN_ON(!rt_rq->rt_nr_running);
- rt_rq->rt_nr_running -= rt_se_nr_running(rt_se);
- rt_rq->rr_nr_running -= rt_se_rr_nr_running(rt_se);
+ rt_rq->rt_nr_running -= 1;
+ rt_rq->rr_nr_running -= is_rr_task(rt_se);
dec_rt_prio(rt_rq, rt_se_prio(rt_se));
- dec_rt_group(rt_se, rt_rq);
}
/*
@@ -1182,10 +478,6 @@ static void __delist_rt_entity(struct sched_rt_entity *rt_se, struct rt_prio_arr
static inline struct sched_statistics *
__schedstats_from_rt_se(struct sched_rt_entity *rt_se)
{
- /* schedstats is not supported for rt group. */
- if (!rt_entity_is_task(rt_se))
- return NULL;
-
return &rt_task_of(rt_se)->stats;
}
@@ -1198,9 +490,7 @@ update_stats_wait_start_rt(struct rt_rq *rt_rq, struct sched_rt_entity *rt_se)
if (!schedstat_enabled())
return;
- if (rt_entity_is_task(rt_se))
- p = rt_task_of(rt_se);
-
+ p = rt_task_of(rt_se);
stats = __schedstats_from_rt_se(rt_se);
if (!stats)
return;
@@ -1217,9 +507,7 @@ update_stats_enqueue_sleeper_rt(struct rt_rq *rt_rq, struct sched_rt_entity *rt_
if (!schedstat_enabled())
return;
- if (rt_entity_is_task(rt_se))
- p = rt_task_of(rt_se);
-
+ p = rt_task_of(rt_se);
stats = __schedstats_from_rt_se(rt_se);
if (!stats)
return;
@@ -1247,9 +535,7 @@ update_stats_wait_end_rt(struct rt_rq *rt_rq, struct sched_rt_entity *rt_se)
if (!schedstat_enabled())
return;
- if (rt_entity_is_task(rt_se))
- p = rt_task_of(rt_se);
-
+ p = rt_task_of(rt_se);
stats = __schedstats_from_rt_se(rt_se);
if (!stats)
return;
@@ -1267,12 +553,10 @@ update_stats_dequeue_rt(struct rt_rq *rt_rq, struct sched_rt_entity *rt_se,
if (!schedstat_enabled())
return;
- if (rt_entity_is_task(rt_se)) {
- p = rt_task_of(rt_se);
+ p = rt_task_of(rt_se);
- if (p != rq->curr)
- update_stats_wait_end_rt(rt_rq, rt_se);
- }
+ if (p != rq->curr)
+ update_stats_wait_end_rt(rt_rq, rt_se);
if ((flags & DEQUEUE_SLEEP) && p) {
unsigned int state;
@@ -1292,21 +576,8 @@ static void __enqueue_rt_entity(struct sched_rt_entity *rt_se, unsigned int flag
{
struct rt_rq *rt_rq = rt_rq_of_se(rt_se);
struct rt_prio_array *array = &rt_rq->active;
- struct rt_rq *group_rq = group_rt_rq(rt_se);
struct list_head *queue = array->queue + rt_se_prio(rt_se);
- /*
- * Don't enqueue the group if its throttled, or when empty.
- * The latter is a consequence of the former when a child group
- * get throttled and the current group doesn't have any other
- * active members.
- */
- if (group_rq && (rt_rq_throttled(group_rq) || !group_rq->rt_nr_running)) {
- if (rt_se->on_list)
- __delist_rt_entity(rt_se, array);
- return;
- }
-
if (move_entity(flags)) {
WARN_ON_ONCE(rt_se->on_list);
if (flags & ENQUEUE_HEAD)
@@ -1336,57 +607,18 @@ static void __dequeue_rt_entity(struct sched_rt_entity *rt_se, unsigned int flag
dec_rt_tasks(rt_se, rt_rq);
}
-/*
- * Because the prio of an upper entry depends on the lower
- * entries, we must remove entries top - down.
- */
-static void dequeue_rt_stack(struct sched_rt_entity *rt_se, unsigned int flags)
-{
- struct sched_rt_entity *back = NULL;
- unsigned int rt_nr_running;
-
- for_each_sched_rt_entity(rt_se) {
- rt_se->back = back;
- back = rt_se;
- }
-
- rt_nr_running = rt_rq_of_se(back)->rt_nr_running;
-
- for (rt_se = back; rt_se; rt_se = rt_se->back) {
- if (on_rt_rq(rt_se))
- __dequeue_rt_entity(rt_se, flags);
- }
-
- dequeue_top_rt_rq(rt_rq_of_se(back), rt_nr_running);
-}
-
static void enqueue_rt_entity(struct sched_rt_entity *rt_se, unsigned int flags)
{
- struct rq *rq = rq_of_rt_se(rt_se);
-
update_stats_enqueue_rt(rt_rq_of_se(rt_se), rt_se, flags);
- dequeue_rt_stack(rt_se, flags);
- for_each_sched_rt_entity(rt_se)
- __enqueue_rt_entity(rt_se, flags);
- enqueue_top_rt_rq(&rq->rt);
+ __enqueue_rt_entity(rt_se, flags);
}
static void dequeue_rt_entity(struct sched_rt_entity *rt_se, unsigned int flags)
{
- struct rq *rq = rq_of_rt_se(rt_se);
-
update_stats_dequeue_rt(rt_rq_of_se(rt_se), rt_se, flags);
- dequeue_rt_stack(rt_se, flags);
-
- for_each_sched_rt_entity(rt_se) {
- struct rt_rq *rt_rq = group_rt_rq(rt_se);
-
- if (rt_rq && rt_rq->rt_nr_running)
- __enqueue_rt_entity(rt_se, flags);
- }
- enqueue_top_rt_rq(&rq->rt);
+ __dequeue_rt_entity(rt_se, flags);
}
/*
@@ -1446,13 +678,7 @@ requeue_rt_entity(struct rt_rq *rt_rq, struct sched_rt_entity *rt_se, int head)
static void requeue_task_rt(struct rq *rq, struct task_struct *p, int head)
{
- struct sched_rt_entity *rt_se = &p->rt;
- struct rt_rq *rt_rq;
-
- for_each_sched_rt_entity(rt_se) {
- rt_rq = rt_rq_of_se(rt_se);
- requeue_rt_entity(rt_rq, rt_se, head);
- }
+ requeue_rt_entity(rt_rq_of_se(&p->rt), &p->rt, head);
}
static void yield_task_rt(struct rq *rq)
@@ -1653,21 +879,6 @@ static struct sched_rt_entity *pick_next_rt_entity(struct rt_rq *rt_rq)
return next;
}
-static struct task_struct *_pick_next_task_rt(struct rq *rq)
-{
- struct sched_rt_entity *rt_se;
- struct rt_rq *rt_rq = &rq->rt;
-
- do {
- rt_se = pick_next_rt_entity(rt_rq);
- if (unlikely(!rt_se))
- return NULL;
- rt_rq = group_rt_rq(rt_se);
- } while (rt_rq);
-
- return rt_task_of(rt_se);
-}
-
static struct task_struct *pick_task_rt(struct rq *rq, struct rq_flags *rf)
{
struct task_struct *p;
@@ -1675,7 +886,7 @@ static struct task_struct *pick_task_rt(struct rq *rq, struct rq_flags *rf)
if (!sched_rt_runnable(rq))
return NULL;
- p = _pick_next_task_rt(rq);
+ p = rt_task_of(pick_next_rt_entity(&rq->rt));
return p;
}
@@ -2358,8 +1569,6 @@ static void rq_online_rt(struct rq *rq)
if (rq->rt.overloaded)
rt_set_overload(rq);
- __enable_runtime(rq);
-
cpupri_set(&rq->rd->cpupri, rq->cpu, rq->rt.highest_prio.curr);
}
@@ -2369,8 +1578,6 @@ static void rq_offline_rt(struct rq *rq)
if (rq->rt.overloaded)
rt_clear_overload(rq);
- __disable_runtime(rq);
-
cpupri_set(&rq->rd->cpupri, rq->cpu, CPUPRI_INVALID);
}
@@ -2531,12 +1738,10 @@ static void task_tick_rt(struct rq *rq, struct task_struct *p, int queued)
* Requeue to the end of queue if we (and all of our ancestors) are not
* the only element on the queue
*/
- for_each_sched_rt_entity(rt_se) {
- if (rt_se->run_list.prev != rt_se->run_list.next) {
- requeue_task_rt(rq, p, 0);
- resched_curr(rq);
- return;
- }
+ if (rt_se->run_list.prev != rt_se->run_list.next) {
+ requeue_task_rt(rq, p, 0);
+ resched_curr(rq);
+ return;
}
}
@@ -2554,16 +1759,7 @@ static unsigned int get_rr_interval_rt(struct rq *rq, struct task_struct *task)
#ifdef CONFIG_SCHED_CORE
static int task_is_throttled_rt(struct task_struct *p, int cpu)
{
- struct rt_rq *rt_rq;
-
-#ifdef CONFIG_RT_GROUP_SCHED // XXX maybe add task_rt_rq(), see also sched_rt_period_rt_rq
- rt_rq = task_group(p)->rt_rq[cpu];
- WARN_ON(!rt_group_sched_enabled() && rt_rq->tg != &root_task_group);
-#else
- rt_rq = &cpu_rq(cpu)->rt;
-#endif
-
- return rt_rq_throttled(rt_rq);
+ return 0;
}
#endif /* CONFIG_SCHED_CORE */
diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h
index a03866f68a3b..a217c4ab6660 100644
--- a/kernel/sched/sched.h
+++ b/kernel/sched/sched.h
@@ -827,7 +827,7 @@ struct scx_rq {
static inline int rt_bandwidth_enabled(void)
{
- return sysctl_sched_rt_runtime >= 0;
+ return 0;
}
/* RT IPI pull logic requires IRQ_WORK */
@@ -867,7 +867,7 @@ struct rt_rq {
static inline bool rt_rq_is_runnable(struct rt_rq *rt_rq)
{
- return rt_rq->rt_queued && rt_rq->rt_nr_running;
+ return rt_rq->rt_nr_running;
}
/* Deadline class' related fields in a runqueue */
@@ -2794,7 +2794,7 @@ static inline bool sched_dl_runnable(struct rq *rq)
static inline bool sched_rt_runnable(struct rq *rq)
{
- return rq->rt.rt_queued > 0;
+ return rq->rt.rt_nr_running > 0;
}
static inline bool sched_fair_runnable(struct rq *rq)
@@ -2906,9 +2906,6 @@ extern void resched_curr(struct rq *rq);
extern void resched_curr_lazy(struct rq *rq);
extern void resched_cpu(int cpu);
-extern void init_rt_bandwidth(struct rt_bandwidth *rt_b, u64 period, u64 runtime);
-extern bool sched_rt_bandwidth_account(struct rt_rq *rt_rq);
-
extern void init_dl_entity(struct sched_dl_entity *dl_se);
extern void init_cfs_throttle_work(struct task_struct *p);
@@ -3333,12 +3330,8 @@ extern void set_rq_offline(struct rq *rq);
extern bool sched_smp_initialized;
#ifdef CONFIG_RT_GROUP_SCHED
-#define rt_entity_is_task(rt_se) (!(rt_se)->my_q)
-
static inline struct task_struct *rt_task_of(struct sched_rt_entity *rt_se)
{
- WARN_ON_ONCE(!rt_entity_is_task(rt_se));
-
return container_of_const(rt_se, struct task_struct, rt);
}
@@ -3354,17 +3347,7 @@ static inline struct rt_rq *rt_rq_of_se(struct sched_rt_entity *rt_se)
WARN_ON(!rt_group_sched_enabled() && rt_se->rt_rq->tg != &root_task_group);
return rt_se->rt_rq;
}
-
-static inline struct rq *rq_of_rt_se(struct sched_rt_entity *rt_se)
-{
- struct rt_rq *rt_rq = rt_se->rt_rq;
-
- WARN_ON(!rt_group_sched_enabled() && rt_rq->tg != &root_task_group);
- return rt_rq->rq;
-}
#else
-#define rt_entity_is_task(rt_se) (1)
-
static inline struct task_struct *rt_task_of(struct sched_rt_entity *rt_se)
{
return container_of_const(rt_se, struct task_struct, rt);
@@ -3375,16 +3358,9 @@ static inline struct rq *rq_of_rt_rq(struct rt_rq *rt_rq)
return container_of_const(rt_rq, struct rq, rt);
}
-static inline struct rq *rq_of_rt_se(struct sched_rt_entity *rt_se)
-{
- struct task_struct *p = rt_task_of(rt_se);
-
- return task_rq(p);
-}
-
static inline struct rt_rq *rt_rq_of_se(struct sched_rt_entity *rt_se)
{
- struct rq *rq = rq_of_rt_se(rt_se);
+ struct rq *rq = task_rq(rt_task_of(rt_se));
return &rq->rt;
}
diff --git a/kernel/sched/syscalls.c b/kernel/sched/syscalls.c
index b215b0ead9a6..9c1ba10ea5a7 100644
--- a/kernel/sched/syscalls.c
+++ b/kernel/sched/syscalls.c
@@ -606,19 +606,6 @@ int __sched_setscheduler(struct task_struct *p,
change:
if (user) {
-#ifdef CONFIG_RT_GROUP_SCHED
- /*
- * Do not allow real-time tasks into groups that have no runtime
- * assigned.
- */
- if (rt_group_sched_enabled() &&
- rt_bandwidth_enabled() && rt_policy(policy) &&
- task_group(p)->rt_bandwidth.rt_runtime == 0 &&
- !task_group_is_autogroup(task_group(p))) {
- retval = -EPERM;
- goto unlock;
- }
-#endif /* CONFIG_RT_GROUP_SCHED */
if (dl_bandwidth_enabled() && dl_policy(policy) &&
!(attr->sched_flags & SCHED_FLAG_SUGOV)) {
cpumask_t *span = rq->rd->span;
--
2.54.0
^ permalink raw reply related
* [RFC PATCH v6 08/25] sched/rt: Remove unnecessary runqueue pointer in struct rt_rq
From: Yuri Andriaccio @ 2026-06-08 12:15 UTC (permalink / raw)
To: Ingo Molnar, Peter Zijlstra, Juri Lelli, Vincent Guittot,
Dietmar Eggemann, Steven Rostedt, Ben Segall, Mel Gorman,
Valentin Schneider, Tejun Heo, Johannes Weiner,
Michal Koutný
Cc: cgroups, linux-kernel, Luca Abeni, Yuri Andriaccio
In-Reply-To: <20260608121546.69910-1-yurand2000@gmail.com>
Remove the rq field in struct rt_rq.
The rq field now is just caching the pointer to the global runqueue of the
given rt_rq, so it is unnecessary as the global runqueue can be retrieved
in other ways.
Introduce global_rq_of_rt_rq to retrieve the global runqueue which serves a
rt_rq's dl_server.
Rework rq_of_rt_rq to retrieve the runqueue a rt_rq is serving.
Signed-off-by: Yuri Andriaccio <yurand2000@gmail.com>
---
kernel/sched/rt.c | 21 +++++++++------------
kernel/sched/sched.h | 16 ++++++++++++----
2 files changed, 21 insertions(+), 16 deletions(-)
diff --git a/kernel/sched/rt.c b/kernel/sched/rt.c
index 7b526a86083c..4575c234ae46 100644
--- a/kernel/sched/rt.c
+++ b/kernel/sched/rt.c
@@ -101,10 +101,7 @@ void init_tg_rt_entry(struct task_group *tg, struct rt_rq *rt_rq,
struct sched_rt_entity *rt_se, int cpu,
struct sched_rt_entity *parent)
{
- struct rq *rq = cpu_rq(cpu);
-
rt_rq->highest_prio.curr = MAX_RT_PRIO-1;
- rt_rq->rq = rq;
rt_rq->tg = tg;
tg->rt_rq[cpu] = rt_rq;
@@ -184,7 +181,7 @@ static void pull_rt_task(struct rq *);
static inline void rt_queue_push_tasks(struct rt_rq *rt_rq)
{
- struct rq *rq = container_of_const(rt_rq, struct rq, rt);
+ struct rq *rq = global_rq_of_rt_rq(rt_rq);
if (!has_pushable_tasks(rt_rq))
return;
@@ -194,7 +191,7 @@ static inline void rt_queue_push_tasks(struct rt_rq *rt_rq)
static inline void rt_queue_pull_task(struct rt_rq *rt_rq)
{
- struct rq *rq = container_of_const(rt_rq, struct rq, rt);
+ struct rq *rq = global_rq_of_rt_rq(rt_rq);
queue_balance_callback(rq, &per_cpu(rt_pull_head, rq->cpu), pull_rt_task);
}
@@ -222,7 +219,7 @@ static void enqueue_pushable_task(struct rt_rq *rt_rq, struct task_struct *p)
rt_rq->highest_prio.next = p->prio;
if (!rt_rq->overloaded) {
- rt_set_overload(rq_of_rt_rq(rt_rq));
+ rt_set_overload(global_rq_of_rt_rq(rt_rq));
rt_rq->overloaded = 1;
}
}
@@ -240,7 +237,7 @@ static void dequeue_pushable_task(struct rt_rq *rt_rq, struct task_struct *p)
rt_rq->highest_prio.next = MAX_RT_PRIO-1;
if (rt_rq->overloaded) {
- rt_clear_overload(rq_of_rt_rq(rt_rq));
+ rt_clear_overload(global_rq_of_rt_rq(rt_rq));
rt_rq->overloaded = 0;
}
}
@@ -495,7 +492,7 @@ update_stats_wait_start_rt(struct rt_rq *rt_rq, struct sched_rt_entity *rt_se)
if (!stats)
return;
- __update_stats_wait_start(rq_of_rt_rq(rt_rq), p, stats);
+ __update_stats_wait_start(global_rq_of_rt_rq(rt_rq), p, stats);
}
static inline void
@@ -512,7 +509,7 @@ update_stats_enqueue_sleeper_rt(struct rt_rq *rt_rq, struct sched_rt_entity *rt_
if (!stats)
return;
- __update_stats_enqueue_sleeper(rq_of_rt_rq(rt_rq), p, stats);
+ __update_stats_enqueue_sleeper(global_rq_of_rt_rq(rt_rq), p, stats);
}
static inline void
@@ -540,7 +537,7 @@ update_stats_wait_end_rt(struct rt_rq *rt_rq, struct sched_rt_entity *rt_se)
if (!stats)
return;
- __update_stats_wait_end(rq_of_rt_rq(rt_rq), p, stats);
+ __update_stats_wait_end(global_rq_of_rt_rq(rt_rq), p, stats);
}
static inline void
@@ -564,11 +561,11 @@ update_stats_dequeue_rt(struct rt_rq *rt_rq, struct sched_rt_entity *rt_se,
state = READ_ONCE(p->__state);
if (state & TASK_INTERRUPTIBLE)
__schedstat_set(p->stats.sleep_start,
- rq_clock(rq_of_rt_rq(rt_rq)));
+ rq_clock(global_rq_of_rt_rq(rt_rq)));
if (state & TASK_UNINTERRUPTIBLE)
__schedstat_set(p->stats.block_start,
- rq_clock(rq_of_rt_rq(rt_rq)));
+ rq_clock(global_rq_of_rt_rq(rt_rq)));
}
}
diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h
index a217c4ab6660..3aa29fe932fc 100644
--- a/kernel/sched/sched.h
+++ b/kernel/sched/sched.h
@@ -857,8 +857,6 @@ struct rt_rq {
raw_spinlock_t rt_runtime_lock;
unsigned int rt_nr_boosted;
-
- struct rq *rq; /* this is always top-level rq, cache? */
#endif
#ifdef CONFIG_CGROUP_SCHED
struct task_group *tg; /* this tg has "this" rt_rq on given CPU for runnable entities */
@@ -3337,9 +3335,14 @@ static inline struct task_struct *rt_task_of(struct sched_rt_entity *rt_se)
static inline struct rq *rq_of_rt_rq(struct rt_rq *rt_rq)
{
- /* Cannot fold with non-CONFIG_RT_GROUP_SCHED version, layout */
WARN_ON(!rt_group_sched_enabled() && rt_rq->tg != &root_task_group);
- return rt_rq->rq;
+ return container_of_const(rt_rq, struct rq, rt);
+}
+
+static inline struct rq *global_rq_of_rt_rq(struct rt_rq *rt_rq)
+{
+ /* Cannot fold with non-CONFIG_RT_GROUP_SCHED version, layout */
+ return cpu_rq(rq_of_rt_rq(rt_rq)->cpu);
}
static inline struct rt_rq *rt_rq_of_se(struct sched_rt_entity *rt_se)
@@ -3358,6 +3361,11 @@ static inline struct rq *rq_of_rt_rq(struct rt_rq *rt_rq)
return container_of_const(rt_rq, struct rq, rt);
}
+static inline struct rq *global_rq_of_rt_rq(struct rt_rq *rt_rq)
+{
+ return container_of_const(rt_rq, struct rq, rt);
+}
+
static inline struct rt_rq *rt_rq_of_se(struct sched_rt_entity *rt_se)
{
struct rq *rq = task_rq(rt_task_of(rt_se));
--
2.54.0
^ permalink raw reply related
* [RFC PATCH v6 06/25] sched/rt: Move functions from rt.c to sched.h
From: Yuri Andriaccio @ 2026-06-08 12:15 UTC (permalink / raw)
To: Ingo Molnar, Peter Zijlstra, Juri Lelli, Vincent Guittot,
Dietmar Eggemann, Steven Rostedt, Ben Segall, Mel Gorman,
Valentin Schneider, Tejun Heo, Johannes Weiner,
Michal Koutný
Cc: cgroups, linux-kernel, Luca Abeni, Yuri Andriaccio
In-Reply-To: <20260608121546.69910-1-yurand2000@gmail.com>
From: luca abeni <luca.abeni@santannapisa.it>
Make the following functions/macros be non-static and move them in
sched.h, so that they can be also used in other source files:
- rt_entity_is_task()
- rt_task_of()
- rq_of_rt_rq()
- rt_rq_of_se()
- rq_of_rt_se()
There are no functional changes, apart from the use of container_of_const()
instead of container_of() where applicable. This is needed by future patches.
Signed-off-by: luca abeni <luca.abeni@santannapisa.it>
Signed-off-by: Yuri Andriaccio <yurand2000@gmail.com>
---
kernel/sched/rt.c | 56 ------------------------------------------
kernel/sched/sched.h | 58 ++++++++++++++++++++++++++++++++++++++++++++
2 files changed, 58 insertions(+), 56 deletions(-)
diff --git a/kernel/sched/rt.c b/kernel/sched/rt.c
index 0f0d9c283bd4..fe5b58f8fc69 100644
--- a/kernel/sched/rt.c
+++ b/kernel/sched/rt.c
@@ -166,36 +166,6 @@ static void destroy_rt_bandwidth(struct rt_bandwidth *rt_b)
hrtimer_cancel(&rt_b->rt_period_timer);
}
-#define rt_entity_is_task(rt_se) (!(rt_se)->my_q)
-
-static inline struct task_struct *rt_task_of(struct sched_rt_entity *rt_se)
-{
- WARN_ON_ONCE(!rt_entity_is_task(rt_se));
-
- return container_of(rt_se, struct task_struct, rt);
-}
-
-static inline struct rq *rq_of_rt_rq(struct rt_rq *rt_rq)
-{
- /* Cannot fold with non-CONFIG_RT_GROUP_SCHED version, layout */
- WARN_ON(!rt_group_sched_enabled() && rt_rq->tg != &root_task_group);
- return rt_rq->rq;
-}
-
-static inline struct rt_rq *rt_rq_of_se(struct sched_rt_entity *rt_se)
-{
- WARN_ON(!rt_group_sched_enabled() && rt_se->rt_rq->tg != &root_task_group);
- return rt_se->rt_rq;
-}
-
-static inline struct rq *rq_of_rt_se(struct sched_rt_entity *rt_se)
-{
- struct rt_rq *rt_rq = rt_se->rt_rq;
-
- WARN_ON(!rt_group_sched_enabled() && rt_rq->tg != &root_task_group);
- return rt_rq->rq;
-}
-
void unregister_rt_sched_group(struct task_group *tg)
{
if (!rt_group_sched_enabled())
@@ -294,32 +264,6 @@ int alloc_rt_sched_group(struct task_group *tg, struct task_group *parent)
#else /* !CONFIG_RT_GROUP_SCHED: */
-#define rt_entity_is_task(rt_se) (1)
-
-static inline struct task_struct *rt_task_of(struct sched_rt_entity *rt_se)
-{
- return container_of(rt_se, struct task_struct, rt);
-}
-
-static inline struct rq *rq_of_rt_rq(struct rt_rq *rt_rq)
-{
- return container_of(rt_rq, struct rq, rt);
-}
-
-static inline struct rq *rq_of_rt_se(struct sched_rt_entity *rt_se)
-{
- struct task_struct *p = rt_task_of(rt_se);
-
- return task_rq(p);
-}
-
-static inline struct rt_rq *rt_rq_of_se(struct sched_rt_entity *rt_se)
-{
- struct rq *rq = rq_of_rt_se(rt_se);
-
- return &rq->rt;
-}
-
void unregister_rt_sched_group(struct task_group *tg) { }
void free_rt_sched_group(struct task_group *tg) { }
diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h
index 970386ce4dbf..a03866f68a3b 100644
--- a/kernel/sched/sched.h
+++ b/kernel/sched/sched.h
@@ -3332,6 +3332,64 @@ extern void set_rq_offline(struct rq *rq);
extern bool sched_smp_initialized;
+#ifdef CONFIG_RT_GROUP_SCHED
+#define rt_entity_is_task(rt_se) (!(rt_se)->my_q)
+
+static inline struct task_struct *rt_task_of(struct sched_rt_entity *rt_se)
+{
+ WARN_ON_ONCE(!rt_entity_is_task(rt_se));
+
+ return container_of_const(rt_se, struct task_struct, rt);
+}
+
+static inline struct rq *rq_of_rt_rq(struct rt_rq *rt_rq)
+{
+ /* Cannot fold with non-CONFIG_RT_GROUP_SCHED version, layout */
+ WARN_ON(!rt_group_sched_enabled() && rt_rq->tg != &root_task_group);
+ return rt_rq->rq;
+}
+
+static inline struct rt_rq *rt_rq_of_se(struct sched_rt_entity *rt_se)
+{
+ WARN_ON(!rt_group_sched_enabled() && rt_se->rt_rq->tg != &root_task_group);
+ return rt_se->rt_rq;
+}
+
+static inline struct rq *rq_of_rt_se(struct sched_rt_entity *rt_se)
+{
+ struct rt_rq *rt_rq = rt_se->rt_rq;
+
+ WARN_ON(!rt_group_sched_enabled() && rt_rq->tg != &root_task_group);
+ return rt_rq->rq;
+}
+#else
+#define rt_entity_is_task(rt_se) (1)
+
+static inline struct task_struct *rt_task_of(struct sched_rt_entity *rt_se)
+{
+ return container_of_const(rt_se, struct task_struct, rt);
+}
+
+static inline struct rq *rq_of_rt_rq(struct rt_rq *rt_rq)
+{
+ return container_of_const(rt_rq, struct rq, rt);
+}
+
+static inline struct rq *rq_of_rt_se(struct sched_rt_entity *rt_se)
+{
+ struct task_struct *p = rt_task_of(rt_se);
+
+ return task_rq(p);
+}
+
+static inline struct rt_rq *rt_rq_of_se(struct sched_rt_entity *rt_se)
+{
+ struct rq *rq = rq_of_rt_se(rt_se);
+
+ return &rq->rt;
+}
+#endif
+
DEFINE_LOCK_GUARD_2(double_rq_lock, struct rq,
double_rq_lock(_T->lock, _T->lock2),
double_rq_unlock(_T->lock, _T->lock2))
--
2.54.0
^ permalink raw reply related
* [RFC PATCH v6 05/25] sched/rt: Pass an rt_rq instead of an rq where needed
From: Yuri Andriaccio @ 2026-06-08 12:15 UTC (permalink / raw)
To: Ingo Molnar, Peter Zijlstra, Juri Lelli, Vincent Guittot,
Dietmar Eggemann, Steven Rostedt, Ben Segall, Mel Gorman,
Valentin Schneider, Tejun Heo, Johannes Weiner,
Michal Koutný
Cc: cgroups, linux-kernel, Luca Abeni, Yuri Andriaccio
In-Reply-To: <20260608121546.69910-1-yurand2000@gmail.com>
From: luca abeni <luca.abeni@santannapisa.it>
Make rt.c code access the runqueue through the rt_rq data structure rather
than passing an rq pointer directly. This allows future patches to define
rt_rq data structures which do not refer only to the global runqueue, but
also to local cgroup runqueues (as rt_rq will not be always equal to
&rq->rt).
Co-developed-by: Yuri Andriaccio <yurand2000@gmail.com>
Signed-off-by: Yuri Andriaccio <yurand2000@gmail.com>
Signed-off-by: luca abeni <luca.abeni@santannapisa.it>
---
kernel/sched/rt.c | 169 +++++++++++++++++++++++++++-------------------
1 file changed, 100 insertions(+), 69 deletions(-)
diff --git a/kernel/sched/rt.c b/kernel/sched/rt.c
index e6ea728f519e..0f0d9c283bd4 100644
--- a/kernel/sched/rt.c
+++ b/kernel/sched/rt.c
@@ -370,9 +370,9 @@ static inline void rt_clear_overload(struct rq *rq)
cpumask_clear_cpu(rq->cpu, rq->rd->rto_mask);
}
-static inline int has_pushable_tasks(struct rq *rq)
+static inline int has_pushable_tasks(struct rt_rq *rt_rq)
{
- return !plist_head_empty(&rq->rt.pushable_tasks);
+ return !plist_head_empty(&rt_rq->pushable_tasks);
}
static DEFINE_PER_CPU(struct balance_callback, rt_push_head);
@@ -381,50 +381,66 @@ static DEFINE_PER_CPU(struct balance_callback, rt_pull_head);
static void push_rt_tasks(struct rq *);
static void pull_rt_task(struct rq *);
-static inline void rt_queue_push_tasks(struct rq *rq)
+static inline void rt_queue_push_tasks(struct rt_rq *rt_rq)
{
- if (!has_pushable_tasks(rq))
+ struct rq *rq = container_of_const(rt_rq, struct rq, rt);
+
+ if (!has_pushable_tasks(rt_rq))
return;
queue_balance_callback(rq, &per_cpu(rt_push_head, rq->cpu), push_rt_tasks);
}
-static inline void rt_queue_pull_task(struct rq *rq)
+static inline void rt_queue_pull_task(struct rt_rq *rt_rq)
{
+ struct rq *rq = container_of_const(rt_rq, struct rq, rt);
+
queue_balance_callback(rq, &per_cpu(rt_pull_head, rq->cpu), pull_rt_task);
}
-static void enqueue_pushable_task(struct rq *rq, struct task_struct *p)
+static void push_rt_rq_tasks(struct rt_rq *rt_rq);
+
+static void push_rt_tasks(struct rq *global_rq) {
+ push_rt_rq_tasks(&global_rq->rt);
+}
+
+static void pull_rt_rq_task(struct rt_rq *this_rt_rq);
+
+static void pull_rt_task(struct rq *global_rq) {
+ pull_rt_rq_task(&global_rq->rt);
+}
+
+static void enqueue_pushable_task(struct rt_rq *rt_rq, struct task_struct *p)
{
- plist_del(&p->pushable_tasks, &rq->rt.pushable_tasks);
+ plist_del(&p->pushable_tasks, &rt_rq->pushable_tasks);
plist_node_init(&p->pushable_tasks, p->prio);
- plist_add(&p->pushable_tasks, &rq->rt.pushable_tasks);
+ plist_add(&p->pushable_tasks, &rt_rq->pushable_tasks);
/* Update the highest prio pushable task */
- if (p->prio < rq->rt.highest_prio.next)
- rq->rt.highest_prio.next = p->prio;
+ if (p->prio < rt_rq->highest_prio.next)
+ rt_rq->highest_prio.next = p->prio;
- if (!rq->rt.overloaded) {
- rt_set_overload(rq);
- rq->rt.overloaded = 1;
+ if (!rt_rq->overloaded) {
+ rt_set_overload(rq_of_rt_rq(rt_rq));
+ rt_rq->overloaded = 1;
}
}
-static void dequeue_pushable_task(struct rq *rq, struct task_struct *p)
+static void dequeue_pushable_task(struct rt_rq *rt_rq, struct task_struct *p)
{
- plist_del(&p->pushable_tasks, &rq->rt.pushable_tasks);
+ plist_del(&p->pushable_tasks, &rt_rq->pushable_tasks);
/* Update the new highest prio pushable task */
- if (has_pushable_tasks(rq)) {
- p = plist_first_entry(&rq->rt.pushable_tasks,
+ if (has_pushable_tasks(rt_rq)) {
+ p = plist_first_entry(&rt_rq->pushable_tasks,
struct task_struct, pushable_tasks);
- rq->rt.highest_prio.next = p->prio;
+ rt_rq->highest_prio.next = p->prio;
} else {
- rq->rt.highest_prio.next = MAX_RT_PRIO-1;
+ rt_rq->highest_prio.next = MAX_RT_PRIO-1;
- if (rq->rt.overloaded) {
- rt_clear_overload(rq);
- rq->rt.overloaded = 0;
+ if (rt_rq->overloaded) {
+ rt_clear_overload(rq_of_rt_rq(rt_rq));
+ rt_rq->overloaded = 0;
}
}
}
@@ -1436,6 +1452,7 @@ static void
enqueue_task_rt(struct rq *rq, struct task_struct *p, int flags)
{
struct sched_rt_entity *rt_se = &p->rt;
+ struct rt_rq *rt_rq = rt_rq_of_se(rt_se);
if (flags & ENQUEUE_WAKEUP)
rt_se->timeout = 0;
@@ -1449,17 +1466,18 @@ enqueue_task_rt(struct rq *rq, struct task_struct *p, int flags)
return;
if (!task_current(rq, p) && p->nr_cpus_allowed > 1)
- enqueue_pushable_task(rq, p);
+ enqueue_pushable_task(rt_rq, p);
}
static bool dequeue_task_rt(struct rq *rq, struct task_struct *p, int flags)
{
struct sched_rt_entity *rt_se = &p->rt;
+ struct rt_rq *rt_rq = rt_rq_of_se(rt_se);
update_curr_rt(rq);
dequeue_rt_entity(rt_se, flags);
- dequeue_pushable_task(rq, p);
+ dequeue_pushable_task(rt_rq, p);
return true;
}
@@ -1498,7 +1516,7 @@ static void yield_task_rt(struct rq *rq)
requeue_task_rt(rq, rq->donor, 0);
}
-static int find_lowest_rq(struct task_struct *task);
+static int find_lowest_rt_rq(struct task_struct *task);
static int
select_task_rq_rt(struct task_struct *p, int cpu, int flags)
@@ -1548,7 +1566,7 @@ select_task_rq_rt(struct task_struct *p, int cpu, int flags)
(curr->nr_cpus_allowed < 2 || donor->prio <= p->prio);
if (test || !rt_task_fits_capacity(p, cpu)) {
- int target = find_lowest_rq(p);
+ int target = find_lowest_rt_rq(p);
/*
* Bail out if we were forcing a migration to find a better
@@ -1606,7 +1624,7 @@ static int balance_rt(struct rq *rq, struct task_struct *p, struct rq_flags *rf)
* not yet started the picking loop.
*/
rq_unpin_lock(rq, rf);
- pull_rt_task(rq);
+ pull_rt_rq_task(&rq->rt);
rq_repin_lock(rq, rf);
}
@@ -1650,14 +1668,14 @@ static void wakeup_preempt_rt(struct rq *rq, struct task_struct *p, int flags)
static inline void set_next_task_rt(struct rq *rq, struct task_struct *p, bool first)
{
struct sched_rt_entity *rt_se = &p->rt;
- struct rt_rq *rt_rq = &rq->rt;
+ struct rt_rq *rt_rq = rt_rq_of_se(&p->rt);
p->se.exec_start = rq_clock_task(rq);
if (on_rt_rq(&p->rt))
update_stats_wait_end_rt(rt_rq, rt_se);
/* The running task is never eligible for pushing */
- dequeue_pushable_task(rq, p);
+ dequeue_pushable_task(rt_rq, p);
if (!first)
return;
@@ -1670,7 +1688,7 @@ static inline void set_next_task_rt(struct rq *rq, struct task_struct *p, bool f
if (rq->donor->sched_class != &rt_sched_class)
update_rt_rq_load_avg(rq_clock_pelt(rq), rq, 0);
- rt_queue_push_tasks(rq);
+ rt_queue_push_tasks(rt_rq);
}
static struct sched_rt_entity *pick_next_rt_entity(struct rt_rq *rt_rq)
@@ -1721,7 +1739,7 @@ static struct task_struct *pick_task_rt(struct rq *rq, struct rq_flags *rf)
static void put_prev_task_rt(struct rq *rq, struct task_struct *p, struct task_struct *next)
{
struct sched_rt_entity *rt_se = &p->rt;
- struct rt_rq *rt_rq = &rq->rt;
+ struct rt_rq *rt_rq = rt_rq_of_se(&p->rt);
if (on_rt_rq(&p->rt))
update_stats_wait_start_rt(rt_rq, rt_se);
@@ -1737,7 +1755,7 @@ static void put_prev_task_rt(struct rq *rq, struct task_struct *p, struct task_s
* if it is still active
*/
if (on_rt_rq(&p->rt) && p->nr_cpus_allowed > 1)
- enqueue_pushable_task(rq, p);
+ enqueue_pushable_task(rt_rq, p);
}
/* Only try algorithms three times */
@@ -1747,16 +1765,16 @@ static void put_prev_task_rt(struct rq *rq, struct task_struct *p, struct task_s
* Return the highest pushable rq's task, which is suitable to be executed
* on the CPU, NULL otherwise
*/
-static struct task_struct *pick_highest_pushable_task(struct rq *rq, int cpu)
+static struct task_struct *pick_highest_pushable_task(struct rt_rq *rt_rq, int cpu)
{
- struct plist_head *head = &rq->rt.pushable_tasks;
+ struct plist_head *head = &rt_rq->pushable_tasks;
struct task_struct *p;
- if (!has_pushable_tasks(rq))
+ if (!has_pushable_tasks(rt_rq))
return NULL;
plist_for_each_entry(p, head, pushable_tasks) {
- if (task_is_pushable(rq, p, cpu))
+ if (task_is_pushable(rq_of_rt_rq(rt_rq), p, cpu))
return p;
}
@@ -1765,7 +1783,7 @@ static struct task_struct *pick_highest_pushable_task(struct rq *rq, int cpu)
static DEFINE_PER_CPU(cpumask_var_t, local_cpu_mask);
-static int find_lowest_rq(struct task_struct *task)
+static int find_lowest_rt_rq(struct task_struct *task)
{
struct sched_domain *sd;
struct cpumask *lowest_mask = this_cpu_cpumask_var_ptr(local_cpu_mask);
@@ -1856,12 +1874,13 @@ static int find_lowest_rq(struct task_struct *task)
return -1;
}
-static struct task_struct *pick_next_pushable_task(struct rq *rq)
+static struct task_struct *pick_next_pushable_task(struct rt_rq *rt_rq)
{
- struct plist_head *head = &rq->rt.pushable_tasks;
+ struct rq *rq = rq_of_rt_rq(rt_rq);
+ struct plist_head *head = &rt_rq->pushable_tasks;
struct task_struct *i, *p = NULL;
- if (!has_pushable_tasks(rq))
+ if (!has_pushable_tasks(rt_rq))
return NULL;
plist_for_each_entry(i, head, pushable_tasks) {
@@ -1887,14 +1906,15 @@ static struct task_struct *pick_next_pushable_task(struct rq *rq)
}
/* Will lock the rq it finds */
-static struct rq *find_lock_lowest_rq(struct task_struct *task, struct rq *rq)
+static struct rt_rq *find_lock_lowest_rt_rq(struct task_struct *task, struct rt_rq *rt_rq)
{
+ struct rq *rq = rq_of_rt_rq(rt_rq);
struct rq *lowest_rq = NULL;
int tries;
int cpu;
for (tries = 0; tries < RT_MAX_TRIES; tries++) {
- cpu = find_lowest_rq(task);
+ cpu = find_lowest_rt_rq(task);
if ((cpu == -1) || (cpu == rq->cpu))
break;
@@ -1925,7 +1945,7 @@ static struct rq *find_lock_lowest_rq(struct task_struct *task, struct rq *rq)
*/
if (unlikely(is_migration_disabled(task) ||
!cpumask_test_cpu(lowest_rq->cpu, &task->cpus_mask) ||
- task != pick_next_pushable_task(rq))) {
+ task != pick_next_pushable_task(rt_rq))) {
double_unlock_balance(rq, lowest_rq);
lowest_rq = NULL;
@@ -1942,7 +1962,13 @@ static struct rq *find_lock_lowest_rq(struct task_struct *task, struct rq *rq)
lowest_rq = NULL;
}
- return lowest_rq;
+ return &lowest_rq->rt;
+}
+
+static struct rq *find_lock_lowest_rq(struct task_struct *task, struct rq *rq) {
+ struct rt_rq *rt_rq = find_lock_lowest_rt_rq(task, &rq->rt);
+
+ return rq_of_rt_rq(rt_rq);
}
/*
@@ -1950,16 +1976,17 @@ static struct rq *find_lock_lowest_rq(struct task_struct *task, struct rq *rq)
* running task can migrate over to a CPU that is running a task
* of lesser priority.
*/
-static int push_rt_task(struct rq *rq, bool pull)
+static int push_rt_rq_task(struct rt_rq *rt_rq, bool pull)
{
struct task_struct *next_task;
- struct rq *lowest_rq;
+ struct rq *lowest_rq, *rq = rq_of_rt_rq(rt_rq);
+ struct rt_rq *lowest_rt_rq;
int ret = 0;
- if (!rq->rt.overloaded)
+ if (!rt_rq->overloaded)
return 0;
- next_task = pick_next_pushable_task(rq);
+ next_task = pick_next_pushable_task(rt_rq);
if (!next_task)
return 0;
@@ -1982,7 +2009,7 @@ static int push_rt_task(struct rq *rq, bool pull)
return 0;
/*
- * Invoking find_lowest_rq() on anything but an RT task doesn't
+ * Invoking find_lowest_rt_rq() on anything but an RT task doesn't
* make sense. Per the above priority check, curr has to
* be of higher priority than next_task, so no need to
* reschedule when bailing out.
@@ -1993,7 +2020,7 @@ static int push_rt_task(struct rq *rq, bool pull)
if (rq->donor->sched_class != &rt_sched_class)
return 0;
- cpu = find_lowest_rq(rq->curr);
+ cpu = find_lowest_rt_rq(rq->curr);
if (cpu == -1 || cpu == rq->cpu)
return 0;
@@ -2022,19 +2049,19 @@ static int push_rt_task(struct rq *rq, bool pull)
/* We might release rq lock */
get_task_struct(next_task);
- /* find_lock_lowest_rq locks the rq if found */
- lowest_rq = find_lock_lowest_rq(next_task, rq);
- if (!lowest_rq) {
+ /* find_lock_lowest_rt_rq locks the rq if found */
+ lowest_rt_rq = find_lock_lowest_rt_rq(next_task, rt_rq);
+ if (!lowest_rt_rq) {
struct task_struct *task;
/*
- * find_lock_lowest_rq releases rq->lock
+ * find_lock_lowest_rt_rq releases rq->lock
* so it is possible that next_task has migrated.
*
* We need to make sure that the task is still on the same
* run-queue and is also still the next task eligible for
* pushing.
*/
- task = pick_next_pushable_task(rq);
+ task = pick_next_pushable_task(rt_rq);
if (task == next_task) {
/*
* The task hasn't migrated, and is still the next
@@ -2057,6 +2084,7 @@ static int push_rt_task(struct rq *rq, bool pull)
goto retry;
}
+ lowest_rq = rq_of_rt_rq(lowest_rt_rq);
move_queued_task_locked(rq, lowest_rq, next_task);
resched_curr(lowest_rq);
ret = 1;
@@ -2068,10 +2096,10 @@ static int push_rt_task(struct rq *rq, bool pull)
return ret;
}
-static void push_rt_tasks(struct rq *rq)
+static void push_rt_rq_tasks(struct rt_rq *rt_rq)
{
- /* push_rt_task will return true if it moved an RT */
- while (push_rt_task(rq, false))
+ /* push_rt_rq_task will return true if it moved an RT */
+ while (push_rt_rq_task(rt_rq, false))
;
}
@@ -2227,9 +2255,9 @@ void rto_push_irq_work_func(struct irq_work *work)
* We do not need to grab the lock to check for has_pushable_tasks.
* When it gets updated, a check is made if a push is possible.
*/
- if (has_pushable_tasks(rq)) {
+ if (has_pushable_tasks(&rq->rt)) {
raw_spin_rq_lock(rq);
- while (push_rt_task(rq, true))
+ while (push_rt_rq_task(&rq->rt, true))
;
raw_spin_rq_unlock(rq);
}
@@ -2251,11 +2279,13 @@ void rto_push_irq_work_func(struct irq_work *work)
}
#endif /* HAVE_RT_PUSH_IPI */
-static void pull_rt_task(struct rq *this_rq)
+static void pull_rt_rq_task(struct rt_rq *this_rt_rq)
{
+ struct rq *this_rq = rq_of_rt_rq(this_rt_rq);
int this_cpu = this_rq->cpu, cpu;
bool resched = false;
struct task_struct *p, *push_task;
+ struct rt_rq *src_rt_rq;
struct rq *src_rq;
int rt_overload_count = rt_overloaded(this_rq);
@@ -2285,6 +2315,7 @@ static void pull_rt_task(struct rq *this_rq)
continue;
src_rq = cpu_rq(cpu);
+ src_rt_rq = &src_rq->rt;
/*
* Don't bother taking the src_rq->lock if the next highest
@@ -2293,8 +2324,8 @@ static void pull_rt_task(struct rq *this_rq)
* logically higher, the src_rq will push this task away.
* And if its going logically lower, we do not care
*/
- if (src_rq->rt.highest_prio.next >=
- this_rq->rt.highest_prio.curr)
+ if (src_rt_rq->highest_prio.next >=
+ this_rt_rq->highest_prio.curr)
continue;
/*
@@ -2309,13 +2340,13 @@ static void pull_rt_task(struct rq *this_rq)
* We can pull only a task, which is pushable
* on its rq, and no others.
*/
- p = pick_highest_pushable_task(src_rq, this_cpu);
+ p = pick_highest_pushable_task(src_rt_rq, this_cpu);
/*
* Do we have an RT task that preempts
* the to-be-scheduled task?
*/
- if (p && (p->prio < this_rq->rt.highest_prio.curr)) {
+ if (p && (p->prio < this_rt_rq->highest_prio.curr)) {
WARN_ON(p == src_rq->curr);
WARN_ON(!task_on_rq_queued(p));
@@ -2374,7 +2405,7 @@ static void task_woken_rt(struct rq *rq, struct task_struct *p)
rq->donor->prio <= p->prio);
if (need_to_push)
- push_rt_tasks(rq);
+ push_rt_rq_tasks(rt_rq_of_se(&p->rt));
}
/* Assumes rq->lock is held */
@@ -2415,7 +2446,7 @@ static void switched_from_rt(struct rq *rq, struct task_struct *p)
if (!task_on_rq_queued(p) || rq->rt.rt_nr_running)
return;
- rt_queue_pull_task(rq);
+ rt_queue_pull_task(rt_rq_of_se(&p->rt));
}
void __init init_sched_rt_class(void)
@@ -2451,7 +2482,7 @@ static void switched_to_rt(struct rq *rq, struct task_struct *p)
*/
if (task_on_rq_queued(p)) {
if (p->nr_cpus_allowed > 1 && rq->rt.overloaded)
- rt_queue_push_tasks(rq);
+ rt_queue_push_tasks(rt_rq_of_se(&p->rt));
if (p->prio < rq->donor->prio && cpu_online(cpu_of(rq)))
resched_curr(rq);
}
@@ -2476,7 +2507,7 @@ prio_changed_rt(struct rq *rq, struct task_struct *p, u64 oldprio)
* may need to pull tasks to this runqueue.
*/
if (oldprio < p->prio)
- rt_queue_pull_task(rq);
+ rt_queue_pull_task(rt_rq_of_se(&p->rt));
/*
* If there's a higher priority task waiting to run
--
2.54.0
^ permalink raw reply related
* [RFC PATCH v6 04/25] sched/deadline: Distinguish between dl_rq and my_q
From: Yuri Andriaccio @ 2026-06-08 12:15 UTC (permalink / raw)
To: Ingo Molnar, Peter Zijlstra, Juri Lelli, Vincent Guittot,
Dietmar Eggemann, Steven Rostedt, Ben Segall, Mel Gorman,
Valentin Schneider, Tejun Heo, Johannes Weiner,
Michal Koutný
Cc: cgroups, linux-kernel, Luca Abeni, Yuri Andriaccio
In-Reply-To: <20260608121546.69910-1-yurand2000@gmail.com>
From: luca abeni <luca.abeni@santannapisa.it>
Split the single runqueue pointer in sched_dl_entity into two separate
pointers, following the existing pattern used by sched_rt_entity:
- dl_rq: Points to the deadline runqueue where this entity is queued
(global runqueue).
- my_q: Points to the runqueue that this entity serves (for servers).
This distinction is currently redundant for the fair_server and ext_servers
(both point to the same CPU's structures), but is essential for future RT
cgroup support where deadline servers will be queued on the global dl_rq while
serving tasks from cgroup-specific runqueues.
Update rq_of_dl_se() to use container_of_const() to recover the global rq from
dl_rq, and update fair.c and ext.c to explicitly use my_q (local rq) when
accessing the served runqueue.
Update dl_server_init() to take a dl_rq pointer (use to retrieve the
global runqueue where the dl_server is scheduled) and a rq pointer (for
the local runqueue served by the server).
Signed-off-by: luca abeni <luca.abeni@santannapisa.it>
Signed-off-by: Yuri Andriaccio <yurand2000@gmail.com>
---
include/linux/sched.h | 6 ++++--
kernel/sched/deadline.c | 17 ++++++++++++-----
kernel/sched/ext.c | 4 ++--
kernel/sched/fair.c | 4 ++--
kernel/sched/sched.h | 3 ++-
5 files changed, 22 insertions(+), 12 deletions(-)
diff --git a/include/linux/sched.h b/include/linux/sched.h
index ee06cba5c6f5..411ffe9b34b3 100644
--- a/include/linux/sched.h
+++ b/include/linux/sched.h
@@ -733,9 +733,11 @@ struct sched_dl_entity {
* Bits for DL-server functionality. Also see the comment near
* dl_server_update().
*
- * @rq the runqueue this server is for
+ * @dl_rq the runqueue on which this entity is (to be) queued
+ * @my_q the runqueue "owned" by this entity
*/
- struct rq *rq;
+ struct dl_rq *dl_rq;
+ struct rq *my_q;
dl_server_pick_f server_pick_task;
#ifdef CONFIG_RT_MUTEXES
diff --git a/kernel/sched/deadline.c b/kernel/sched/deadline.c
index 63e88ecdd5ed..b3059658a74a 100644
--- a/kernel/sched/deadline.c
+++ b/kernel/sched/deadline.c
@@ -65,10 +65,12 @@ static inline struct rq *rq_of_dl_rq(struct dl_rq *dl_rq)
static inline struct rq *rq_of_dl_se(struct sched_dl_entity *dl_se)
{
- struct rq *rq = dl_se->rq;
+ struct rq *rq;
if (!dl_server(dl_se))
rq = task_rq(dl_task_of(dl_se));
+ else
+ rq = container_of_const(dl_se->dl_rq, struct rq, dl);
return rq;
}
@@ -1817,11 +1819,14 @@ void dl_server_start(struct sched_dl_entity *dl_se)
void dl_server_stop(struct sched_dl_entity *dl_se)
{
+ struct rq *rq;
+
if (!dl_server(dl_se) || !dl_server_active(dl_se))
return;
- trace_sched_dl_server_stop_tp(dl_se, cpu_of(dl_se->rq),
- dl_get_type(dl_se, dl_se->rq));
+ rq = rq_of_dl_se(dl_se);
+ trace_sched_dl_server_stop_tp(dl_se, cpu_of(rq),
+ dl_get_type(dl_se, rq));
dequeue_dl_entity(dl_se, DEQUEUE_SLEEP);
hrtimer_try_to_cancel(&dl_se->dl_timer);
dl_se->dl_defer_armed = 0;
@@ -1830,10 +1835,12 @@ void dl_server_stop(struct sched_dl_entity *dl_se)
dl_se->dl_server_active = 0;
}
-void dl_server_init(struct sched_dl_entity *dl_se, struct rq *rq,
+void dl_server_init(struct sched_dl_entity *dl_se, struct dl_rq *dl_rq,
+ struct rq *served_rq,
dl_server_pick_f pick_task)
{
- dl_se->rq = rq;
+ dl_se->dl_rq = dl_rq;
+ dl_se->my_q = served_rq;
dl_se->server_pick_task = pick_task;
}
diff --git a/kernel/sched/ext.c b/kernel/sched/ext.c
index 65631e577ee9..306bd22a4731 100644
--- a/kernel/sched/ext.c
+++ b/kernel/sched/ext.c
@@ -3252,7 +3252,7 @@ ext_server_pick_task(struct sched_dl_entity *dl_se, struct rq_flags *rf)
if (!scx_enabled())
return NULL;
- return do_pick_task_scx(dl_se->rq, rf, true);
+ return do_pick_task_scx(dl_se->my_q, rf, true);
}
/*
@@ -3264,7 +3264,7 @@ void ext_server_init(struct rq *rq)
init_dl_entity(dl_se);
- dl_server_init(dl_se, rq, ext_server_pick_task);
+ dl_server_init(dl_se, &rq->dl, rq, ext_server_pick_task);
}
#ifdef CONFIG_SCHED_CORE
diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index 3ebec186f982..2bc749ae9203 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -9315,7 +9315,7 @@ pick_next_task_fair(struct rq *rq, struct task_struct *prev, struct rq_flags *rf
static struct task_struct *
fair_server_pick_task(struct sched_dl_entity *dl_se, struct rq_flags *rf)
{
- return pick_task_fair(dl_se->rq, rf);
+ return pick_task_fair(dl_se->my_q, rf);
}
void fair_server_init(struct rq *rq)
@@ -9324,7 +9324,7 @@ void fair_server_init(struct rq *rq)
init_dl_entity(dl_se);
- dl_server_init(dl_se, rq, fair_server_pick_task);
+ dl_server_init(dl_se, &rq->dl, rq, fair_server_pick_task);
}
/*
diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h
index 9f63b15d309d..970386ce4dbf 100644
--- a/kernel/sched/sched.h
+++ b/kernel/sched/sched.h
@@ -412,7 +412,8 @@ extern void dl_server_update_idle(struct sched_dl_entity *dl_se, s64 delta_exec)
extern void dl_server_update(struct sched_dl_entity *dl_se, s64 delta_exec);
extern void dl_server_start(struct sched_dl_entity *dl_se);
extern void dl_server_stop(struct sched_dl_entity *dl_se);
-extern void dl_server_init(struct sched_dl_entity *dl_se, struct rq *rq,
+extern void dl_server_init(struct sched_dl_entity *dl_se, struct dl_rq *dl_rq,
+ struct rq *served_rq,
dl_server_pick_f pick_task);
extern void sched_init_dl_servers(void);
--
2.54.0
^ permalink raw reply related
* [RFC PATCH v6 03/25] sched/deadline: Do not access dl_se->rq directly
From: Yuri Andriaccio @ 2026-06-08 12:15 UTC (permalink / raw)
To: Ingo Molnar, Peter Zijlstra, Juri Lelli, Vincent Guittot,
Dietmar Eggemann, Steven Rostedt, Ben Segall, Mel Gorman,
Valentin Schneider, Tejun Heo, Johannes Weiner,
Michal Koutný
Cc: cgroups, linux-kernel, Luca Abeni, Yuri Andriaccio
In-Reply-To: <20260608121546.69910-1-yurand2000@gmail.com>
From: luca abeni <luca.abeni@santannapisa.it>
Make deadline.c code access the runqueue of a scheduling entity saved in
the sched_dl_entity data structure. This allows future patches to save
different runqueues in sched_dl_entity other than the global runqueues.
Move dl_server_apply_params call in sched_init_dl_servers as the rq_of_dl_se
function will return the correct deadline entity only if the dl_server flag
is set.
Add a WARN_ON on the return value of dl_server_apply_params in
sched_init_dl_servers as this function may fail if the kernel is not
configured correctly.
Signed-off-by: luca abeni <luca.abeni@santannapisa.it>
Signed-off-by: Yuri Andriaccio <yurand2000@gmail.com>
---
kernel/sched/deadline.c | 34 +++++++++++++++++-----------------
1 file changed, 17 insertions(+), 17 deletions(-)
diff --git a/kernel/sched/deadline.c b/kernel/sched/deadline.c
index ddfd6bc63ab1..63e88ecdd5ed 100644
--- a/kernel/sched/deadline.c
+++ b/kernel/sched/deadline.c
@@ -869,7 +869,7 @@ static void replenish_dl_entity(struct sched_dl_entity *dl_se)
* and arm the defer timer.
*/
if (dl_se->dl_defer && !dl_se->dl_defer_running &&
- dl_time_before(rq_clock(dl_se->rq), dl_se->deadline - dl_se->runtime)) {
+ dl_time_before(rq_clock(rq), dl_se->deadline - dl_se->runtime)) {
if (!is_dl_boosted(dl_se)) {
/*
@@ -1170,11 +1170,11 @@ static enum hrtimer_restart dl_server_timer(struct hrtimer *timer, struct sched_
* of time. The dl_server_min_res serves as a limit to avoid
* forwarding the timer for a too small amount of time.
*/
- if (dl_time_before(rq_clock(dl_se->rq),
+ if (dl_time_before(rq_clock(rq),
(dl_se->deadline - dl_se->runtime - dl_server_min_res))) {
/* reset the defer timer */
- fw = dl_se->deadline - rq_clock(dl_se->rq) - dl_se->runtime;
+ fw = dl_se->deadline - rq_clock(rq) - dl_se->runtime;
hrtimer_forward_now(timer, ns_to_ktime(fw));
return HRTIMER_RESTART;
@@ -1185,7 +1185,7 @@ static enum hrtimer_restart dl_server_timer(struct hrtimer *timer, struct sched_
enqueue_dl_entity(dl_se, ENQUEUE_REPLENISH);
- if (!dl_task(dl_se->rq->curr) || dl_entity_preempt(dl_se, &dl_se->rq->curr->dl))
+ if (!dl_task(rq->curr) || dl_entity_preempt(dl_se, &rq->curr->dl))
resched_curr(rq);
__push_dl_task(rq, rf);
@@ -1481,7 +1481,7 @@ static void update_curr_dl_se(struct rq *rq, struct sched_dl_entity *dl_se, s64
hrtimer_try_to_cancel(&dl_se->dl_timer);
- replenish_dl_new_period(dl_se, dl_se->rq);
+ replenish_dl_new_period(dl_se, rq);
if (idle)
dl_se->dl_defer_idle = 1;
@@ -1578,14 +1578,14 @@ static void update_curr_dl_se(struct rq *rq, struct sched_dl_entity *dl_se, s64
void dl_server_update_idle(struct sched_dl_entity *dl_se, s64 delta_exec)
{
if (dl_se->dl_server_active && dl_se->dl_runtime && dl_se->dl_defer)
- update_curr_dl_se(dl_se->rq, dl_se, delta_exec);
+ update_curr_dl_se(rq_of_dl_se(dl_se), dl_se, delta_exec);
}
void dl_server_update(struct sched_dl_entity *dl_se, s64 delta_exec)
{
/* 0 runtime = fair server disabled */
if (dl_se->dl_server_active && dl_se->dl_runtime)
- update_curr_dl_se(dl_se->rq, dl_se, delta_exec);
+ update_curr_dl_se(rq_of_dl_se(dl_se), dl_se, delta_exec);
}
/*
@@ -1794,7 +1794,7 @@ void dl_server_update(struct sched_dl_entity *dl_se, s64 delta_exec)
*/
void dl_server_start(struct sched_dl_entity *dl_se)
{
- struct rq *rq = dl_se->rq;
+ struct rq *rq;
dl_se->dl_defer_idle = 0;
if (!dl_server(dl_se) || dl_se->dl_server_active || !dl_se->dl_runtime)
@@ -1803,16 +1803,16 @@ void dl_server_start(struct sched_dl_entity *dl_se)
/*
* Update the current task to 'now'.
*/
+ rq = rq_of_dl_se(dl_se);
rq->donor->sched_class->update_curr(rq);
-
if (WARN_ON_ONCE(!cpu_online(cpu_of(rq))))
return;
trace_sched_dl_server_start_tp(dl_se, cpu_of(rq), dl_get_type(dl_se, rq));
dl_se->dl_server_active = 1;
enqueue_dl_entity(dl_se, ENQUEUE_WAKEUP);
- if (!dl_task(dl_se->rq->curr) || dl_entity_preempt(dl_se, &rq->curr->dl))
- resched_curr(dl_se->rq);
+ if (!dl_task(rq->curr) || dl_entity_preempt(dl_se, &rq->curr->dl))
+ resched_curr(rq);
}
void dl_server_stop(struct sched_dl_entity *dl_se)
@@ -1856,9 +1856,9 @@ void sched_init_dl_servers(void)
WARN_ON(dl_server(dl_se));
- dl_server_apply_params(dl_se, runtime, period, 1);
-
dl_se->dl_server = 1;
+ WARN_ON(dl_server_apply_params(dl_se, runtime, period, 1));
+
dl_se->dl_defer = 1;
setup_new_dl_entity(dl_se);
@@ -1867,9 +1867,9 @@ void sched_init_dl_servers(void)
WARN_ON(dl_server(dl_se));
- dl_server_apply_params(dl_se, runtime, period, 1);
-
dl_se->dl_server = 1;
+ WARN_ON(dl_server_apply_params(dl_se, runtime, period, 1));
+
dl_se->dl_defer = 1;
setup_new_dl_entity(dl_se);
#endif
@@ -1895,7 +1895,7 @@ int dl_server_apply_params(struct sched_dl_entity *dl_se, u64 runtime, u64 perio
{
u64 old_bw = init ? 0 : to_ratio(dl_se->dl_period, dl_se->dl_runtime);
u64 new_bw = to_ratio(period, runtime);
- struct rq *rq = dl_se->rq;
+ struct rq *rq = rq_of_dl_se(dl_se);
int cpu = cpu_of(rq);
struct dl_bw *dl_b;
unsigned long cap;
@@ -1971,7 +1971,7 @@ static enum hrtimer_restart inactive_task_timer(struct hrtimer *timer)
p = dl_task_of(dl_se);
rq = task_rq_lock(p, &rf);
} else {
- rq = dl_se->rq;
+ rq = rq_of_dl_se(dl_se);
rq_lock(rq, &rf);
}
--
2.54.0
^ permalink raw reply related
* [RFC PATCH v6 02/25] sched/rt: Update default bandwidth for real-time tasks to ONE
From: Yuri Andriaccio @ 2026-06-08 12:15 UTC (permalink / raw)
To: Ingo Molnar, Peter Zijlstra, Juri Lelli, Vincent Guittot,
Dietmar Eggemann, Steven Rostedt, Ben Segall, Mel Gorman,
Valentin Schneider, Tejun Heo, Johannes Weiner,
Michal Koutný
Cc: cgroups, linux-kernel, Luca Abeni, Yuri Andriaccio
In-Reply-To: <20260608121546.69910-1-yurand2000@gmail.com>
Set the default total bandwidth for SCHED_DEADLINE tasks and servers to ONE.
FIFO/RR tasks are already throttled by fair-servers and ext-servers, and
the sysctl_sched_rt_runtime parameter now only defines the total bw that
is allowed to deadline entities.
---
Already merged in sched/tip:
https://git.kernel.org/tip/c2e390197ad1360db6686a8c89abaafaf83adf72
Signed-off-by: Yuri Andriaccio <yurand2000@gmail.com>
---
kernel/sched/rt.c | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/kernel/sched/rt.c b/kernel/sched/rt.c
index 4ee8faf01441..e6ea728f519e 100644
--- a/kernel/sched/rt.c
+++ b/kernel/sched/rt.c
@@ -19,9 +19,9 @@ int sysctl_sched_rt_period = 1000000;
/*
* part of the period that we allow rt tasks to run in us.
- * default: 0.95s
+ * default: 1s
*/
-int sysctl_sched_rt_runtime = 950000;
+int sysctl_sched_rt_runtime = 1000000;
#ifdef CONFIG_SYSCTL
static int sysctl_sched_rr_timeslice = (MSEC_PER_SEC * RR_TIMESLICE) / HZ;
--
2.54.0
^ permalink raw reply related
* [RFC PATCH v6 01/25] sched/deadline: Fix replenishment logic for non-deferred servers
From: Yuri Andriaccio @ 2026-06-08 12:15 UTC (permalink / raw)
To: Ingo Molnar, Peter Zijlstra, Juri Lelli, Vincent Guittot,
Dietmar Eggemann, Steven Rostedt, Ben Segall, Mel Gorman,
Valentin Schneider, Tejun Heo, Johannes Weiner,
Michal Koutný
Cc: cgroups, linux-kernel, Luca Abeni, Yuri Andriaccio
In-Reply-To: <20260608121546.69910-1-yurand2000@gmail.com>
Enqueue and replenish non-deferred deadline servers when their runtime is
exhausted and the replenishment timer could not be started because it is
too close to the wake-up instant.
---
Already merged in sched/tip:
https://git.kernel.org/tip/eecd5e117cfa63a353f4c69fdcea5d9b14af698e
Signed-off-by: Yuri Andriaccio <yurand2000@gmail.com>
---
kernel/sched/deadline.c | 8 ++++++--
1 file changed, 6 insertions(+), 2 deletions(-)
diff --git a/kernel/sched/deadline.c b/kernel/sched/deadline.c
index 7db4c87df83b..ddfd6bc63ab1 100644
--- a/kernel/sched/deadline.c
+++ b/kernel/sched/deadline.c
@@ -1515,8 +1515,12 @@ static void update_curr_dl_se(struct rq *rq, struct sched_dl_entity *dl_se, s64
if (unlikely(is_dl_boosted(dl_se) || !start_dl_timer(dl_se))) {
if (dl_server(dl_se)) {
- replenish_dl_new_period(dl_se, rq);
- start_dl_timer(dl_se);
+ if (dl_se->dl_defer) {
+ replenish_dl_new_period(dl_se, rq);
+ start_dl_timer(dl_se);
+ } else {
+ enqueue_dl_entity(dl_se, ENQUEUE_REPLENISH);
+ }
} else {
enqueue_task_dl(rq, dl_task_of(dl_se), ENQUEUE_REPLENISH);
}
--
2.54.0
^ permalink raw reply related
* [RFC PATCH v6 00/25] Hierarchical Constant Bandwidth Server
From: Yuri Andriaccio @ 2026-06-08 12:15 UTC (permalink / raw)
To: Ingo Molnar, Peter Zijlstra, Juri Lelli, Vincent Guittot,
Dietmar Eggemann, Steven Rostedt, Ben Segall, Mel Gorman,
Valentin Schneider, Tejun Heo, Johannes Weiner,
Michal Koutný
Cc: cgroups, linux-kernel, Luca Abeni, Yuri Andriaccio
Hello,
This is the v6 for Hierarchical Constant Bandwidth Server, aiming at replacing
the current RT_GROUP_SCHED mechanism with something more robust and
theoretically sound. The patchset has been presented at OSPM25 and OSPM26
(https://retis.sssup.it/ospm-summit/), and a summary of its inner workings can
be found at https://lwn.net/Articles/1021332/ . You can find the previous
versions of this patchset at the bottom of the page, in particular version 1
which talks in more detail what this patchset is all about and how it is
implemented.
This v6 version works on the comments by the reviewers and introduces the
following meaningful changes:
- Update to kernel version 7.1.
- Refactorings and general cleanups.
- Removal of substantial duplicated code.
- Express more locking constraints in code.
- New cpu.rt.max interface.
- Refactoring of migration code to reduce code duplication.
The new migration code now reuses the existing push/pull and similar functions
and specializes where needed, substantially reducing the footprint of group
migration code from previous versions.
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
New cgroup-v2 interface:
After extensive discussions with the kernel's maintainers, we have built a new
interface to support HCBS scheduling. Since this will be a cgroup-v2 only
feature (the fate of cgroup-v1 old RT_GROUP_SCHED has yet to be decided), it was
possible to drop the original v1 interface entirely and create a completely new
one that is similar to those that are already existing.
Every cgroup has now two new files:
- cpu.rt.max (similar to the cpu.max file)
- cpu.rt.internal (read-only, not available in the root cgroup, it may be
removed if deemed unnecessary, see later for details)
In this new interface, HCBS cgroups may either be set to use deadline servers,
and thus reserving a specified amount of bandwidth, very similarly to the
previous system, or can delegate their FIFO/RR tasks' scheduling to the nearest
ancestor that it is configured (default on group creation). If the nearest
configured ancestor is the root cgroup, tasks will be effectively run on the
root runqueue even if their cgroup is not the root task group.
This means that subtrees are allowed to retain the original non-RT_GROUP_SCHED
behaviour, scheduling on root, while the feature is nonetheless active. In the
meantime other subtrees may use HCBS, and the whole hierarchy can coexist
without issues.
This behaviour is specified in the cpu.rt.max file, which accepts the string
"<runtime | 'max'> <period>". A zero runtime disables FIFO/RR scheduling for
tasks in that group, a non-zero runtime creates a reservation and uses HCBS, a
runtime of 'max' instead tells the scheduler to use the nearest configured
ancestor for the FIFO/RR task scheduling.
The admission test now does not only check the immediate children of a cgroup
for schedulability (recall that a group's bandwidth must be always greater than
or equal to its children total bandwidth), but it has to check its whole
subtree: if a child delegates its tasks to its parents (runtime = 'max'), then
this child's own children (the grandchildrens) are effectively viewed as
immediate children that compete for the same bandwidth of their grandparent, and
so on down the hierarchy.
To support both threaded and domain cgroups, the original test that allowed only
to run tasks in leaf cgroups has been removed: this is already enforced for
domain cgroups by existing code, while this must not be the case for threaded
cgroups.
Since groups in the middle of the hierarchy can now also run tasks, their
dl_servers must be configured properly: a parent cgroup dl_servers can only use
their assigned bandwidth minus the total of their children. The cpu.rt.internal
file reads exactly what is this "remainder" bandwidth. Since dl_servers must
have a runtime and period values assigned, the period is taken from the user
configured cpu.rt.max file and the runtime is computed from the remainder bw.
This runtime and the period are the values shown by cpu.rt.internal.
Supporting both threaded and domain cgroups also dropped all the extra code
related to active and 'live' cgroups as mentioned in previous RFCs.
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Summary of the patches:
1-2) Commits already included in sched/tip (not yet in mainline).
3-8) Preparation patches, so that the RT classes' code can be used both
for normal and cgroup scheduling.
9-19) Implementation of HCBS, no migration.
The old RT_GROUP_SCHED code is removed.
16) Remove support for cgroup-v1.
17) Implement cgroup-v2 cpu.rt.max interface.
20-24) Add support for tasks migration.
25) Documentation for HCBS.
Updates from v5:
- Rebase to latest master.
- General rebasing/cleanup.
- More locking contraints expressed in code.
- New cpu.rt.max interface.
- Refactoring of migration code.
Updates from v4:
- Rebase to latest tip/master.
- General rebasing/cleanup.
- Update default sysctl_sched_rt_runtime to 1s, same as the period.
- Fix non-deferred deadline server replenishment logic.
- Add missing RCU read sections.
- Account HCBS servers along with their tasks when the servers are active.
- Release bandwidth resources early in unregister_rt_sched_group.
- Drop server_try_pull_task as it is now redundant.
- Remove dl_server_stop call in dequeue_task_rt.
- Update to reuse __checkparam_dl for deadline servers.
Updates from v3:
- Rebase to latest tip/master.
- General rebasing/cleanup.
- Add Documentation.
- Define **live** and **active** groups.
- Introduce server_try_pull_task in place of the removed server_has_task.
- Introduce RELEASE_LOCK helper macro for guard-based locking.
- Update inc/dec_dl_tasks to account for served runqueues regardless of the
server type.
- Fix computing of new bandwidth values in dl_init_tg.
- Fix check in dl_check_tg to use capacity scaling.
- Fix wakeup_preempt_rt to check if curr is a DEADLINE task.
Updates from v2:
- Rebase to latest tip/master.
- Remove fair-servers' bw reclaiming.
- Fix a check which prevented execution of wakeup_preempt code.
- Fix a priority check in group_pull_rt_task between tasks of different groups.
- Rework allocation/deallocation code for rt-cgroups.
- Update signatures for some group related migration functions.
- Add documentation for wakeup_preempt preemption rules.
Updates from v1:
- Rebase to latest tip/master.
- Add migration code.
- Split big patches for more readability.
- Refactor code to use guarded locks where applicable.
- Remove unnecessary patches from v1 which have been addressed differently by
mainline updates.
- Remove unnecessary checks and general code cleanup.
Notes:
Patches 1-2 have already been merged in sched/tip, but not yet merged in master.
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Testing v6:
The patchset has been tested with a suite of tests tailored to stress all the
implemented functionalities.
The tests are available at https://github.com/Yurand2000/HCBS-Test-Suite .
Refer to the README of the repository for more details.
Follow these steps to test HCBS v6:
- Get the HCBS patch up and running. Any kernel/disto should work effortlessly.
- Get, compile and _install_ the tests.
- Run the `go_rt.sh` script to set the frequency of the CPUs to a fixed value
and disable hyperthreading and power saving features.
- Run the `run_tests.sh full` script, to run the whole test suite.
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Future Work:
We think the current patchset is stable enough. Our current test suite
demonstrates, on our limited hardware, that the kernel does not throw warnings
and that it is actually possible to guarantee time reservations and isolation
among tenants.
Comments on the new cpu.rt.max interface are to be expected, but hopefully with
this new ideas we have solved some of the issues mentioned in the past, such as
not being able to use the cpu controller because standard FIFO/RR tasks had to
be migrated to the root cgroup first. For the future it needs to be investigated
how to integrate this interface with the cpuset controller and with the multiCPU
feature which was presented at OSPM26.
Additional future work:
- unprivileged FIFO/RR in cgroups.
- capacity aware bandwidth reservation.
- hotplug/hotunplug management.
Have a nice day,
Yuri
v1: https://lore.kernel.org/all/20250605071412.139240-1-yurand2000@gmail.com/
v2: https://lore.kernel.org/all/20250731105543.40832-1-yurand2000@gmail.com/
v3: https://lore.kernel.org/all/20250929092221.10947-1-yurand2000@gmail.com/
v4: https://lore.kernel.org/all/20251201124205.11169-1-yurand2000@gmail.com/
v5: https://lore.kernel.org/all/20260430213835.62217-1-yurand2000@gmail.com/
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Yuri Andriaccio (14):
sched/deadline: Fix replenishment logic for non-deferred servers
sched/rt: Update default bandwidth for real-time tasks to ONE
sched/rt: Disable RT_GROUP_SCHED
sched/rt: Remove unnecessary runqueue pointer in struct rt_rq
sched/rt: Add {alloc/unregister/free}_rt_sched_group
sched/rt: Implement dl-server operations for rt-cgroups.
sched/rt: Update task event callbacks for HCBS scheduling
sched/rt: Remove support for cgroups-v1
sched/rt: Update task's RT runqueue when switching scheduling class
sched/rt: Add HCBS migration code to related functions
sched/rt: Hook HCBS migration functions
sched/rt: Try pull task on empty server pick.
sched/core: Execute enqueued balance callbacks after
migrate_disable_switch
Documentation: Update documentation for real-time cgroups
luca abeni (11):
sched/deadline: Do not access dl_se->rq directly
sched/deadline: Distinguish between dl_rq and my_q
sched/rt: Pass an rt_rq instead of an rq where needed
sched/rt: Move functions from rt.c to sched.h
sched/rt: Introduce HCBS specific structs in task_group
sched/core: Initialize HCBS specific structures.
sched/deadline: Add dl_init_tg
sched/deadline: Account rt-cgroups bandwidth in deadline tasks
schedulability tests.
sched/rt: Update rt-cgroup schedulability checks
sched/rt: Remove old RT_GROUP_SCHED data structures
sched/core: Execute enqueued balance callbacks when changing allowed
CPUs
Documentation/scheduler/sched-rt-group.rst | 470 ++++-
include/linux/rcupdate.h | 1 +
include/linux/sched.h | 12 +-
kernel/sched/autogroup.c | 4 +-
kernel/sched/core.c | 143 +-
kernel/sched/deadline.c | 221 ++-
kernel/sched/debug.c | 6 -
kernel/sched/ext.c | 4 +-
kernel/sched/fair.c | 4 +-
kernel/sched/rt.c | 2046 +++++++++-----------
kernel/sched/sched.h | 214 +-
kernel/sched/syscalls.c | 11 +-
12 files changed, 1761 insertions(+), 1375 deletions(-)
base-commit: e43ffb69e0438cddd72aaa30898b4dc446f664f8
--
2.54.0
^ permalink raw reply
* Re: [PATCH] mm: memcontrol-v1: use nofail allocations for soft limit trees
From: Michal Hocko @ 2026-06-08 8:02 UTC (permalink / raw)
To: Ruoyu Wang
Cc: Johannes Weiner, Roman Gushchin, Shakeel Butt, Muchun Song,
Andrew Morton, cgroups, linux-mm, linux-kernel
In-Reply-To: <20260608063644.39-1-ruoyuw560@gmail.com>
On Mon 08-06-26 14:36:44, Ruoyu Wang wrote:
> memcg1_init() allocates one soft-limit tree node per NUMA node and
> then initializes the returned object. If kzalloc_node() fails, the rb_root
> and lock initialization dereference NULL.
>
> The per-node soft-limit tree is required by memcg v1. Use nofail
> GFP_KERNEL allocations for these init-time objects so the init path does
> not continue without the required tree nodes.
This is an early init code executing in during boot. Have you really
seen this allocation failing?
> Signed-off-by: Ruoyu Wang <ruoyuw560@gmail.com>
> ---
> mm/memcontrol-v1.c | 3 ++-
> 1 file changed, 2 insertions(+), 1 deletion(-)
>
> diff --git a/mm/memcontrol-v1.c b/mm/memcontrol-v1.c
> index 433bba9dfe715..3f41a15d8a8cf 100644
> --- a/mm/memcontrol-v1.c
> +++ b/mm/memcontrol-v1.c
> @@ -2246,7 +2246,8 @@ static int __init memcg1_init(void)
> for_each_node(node) {
> struct mem_cgroup_tree_per_node *rtpn;
>
> - rtpn = kzalloc_node(sizeof(*rtpn), GFP_KERNEL, node);
> + rtpn = kzalloc_node(sizeof(*rtpn), GFP_KERNEL | __GFP_NOFAIL,
> + node);
>
> rtpn->rb_root = RB_ROOT;
> rtpn->rb_rightmost = NULL;
> --
> 2.51.0
--
Michal Hocko
SUSE Labs
^ permalink raw reply
* [PATCH] mm: memcontrol-v1: use nofail allocations for soft limit trees
From: Ruoyu Wang @ 2026-06-08 6:36 UTC (permalink / raw)
To: Johannes Weiner
Cc: Michal Hocko, Roman Gushchin, Shakeel Butt, Muchun Song,
Andrew Morton, cgroups, linux-mm, linux-kernel, Ruoyu Wang
memcg1_init() allocates one soft-limit tree node per NUMA node and
then initializes the returned object. If kzalloc_node() fails, the rb_root
and lock initialization dereference NULL.
The per-node soft-limit tree is required by memcg v1. Use nofail
GFP_KERNEL allocations for these init-time objects so the init path does
not continue without the required tree nodes.
Signed-off-by: Ruoyu Wang <ruoyuw560@gmail.com>
---
mm/memcontrol-v1.c | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/mm/memcontrol-v1.c b/mm/memcontrol-v1.c
index 433bba9dfe715..3f41a15d8a8cf 100644
--- a/mm/memcontrol-v1.c
+++ b/mm/memcontrol-v1.c
@@ -2246,7 +2246,8 @@ static int __init memcg1_init(void)
for_each_node(node) {
struct mem_cgroup_tree_per_node *rtpn;
- rtpn = kzalloc_node(sizeof(*rtpn), GFP_KERNEL, node);
+ rtpn = kzalloc_node(sizeof(*rtpn), GFP_KERNEL | __GFP_NOFAIL,
+ node);
rtpn->rb_root = RB_ROOT;
rtpn->rb_rightmost = NULL;
--
2.51.0
^ permalink raw reply related
* [PATCH 8/8] block, bfq: don't grab queue_lock to initialize bfq
From: Yu Kuai @ 2026-06-08 3:42 UTC (permalink / raw)
To: nilay, tom.leiming, bvanassche, tj, josef, axboe, yukuai
Cc: akpm, chrisl, kasong, shikemeng, nphamcs, bhe, baohua,
youngjun.park, cgroups, linux-block, linux-kernel, linux-mm
In-Reply-To: <cover.1780621988.git.yukuai@fygo.io>
From: Yu Kuai <yukuai@fygo.io>
The request_queue is frozen and quiesced while the elevator init_sched()
method runs, so queue_lock is not needed for BFQ cgroup initialization.
Signed-off-by: Yu Kuai <yukuai@fygo.io>
---
block/bfq-iosched.c | 5 -----
1 file changed, 5 deletions(-)
diff --git a/block/bfq-iosched.c b/block/bfq-iosched.c
index 42ccfd0c6140..5cabee2d4e7c 100644
--- a/block/bfq-iosched.c
+++ b/block/bfq-iosched.c
@@ -7207,14 +7207,11 @@ static int bfq_init_queue(struct request_queue *q, struct elevator_queue *eq)
bfqd = kzalloc_node(sizeof(*bfqd), GFP_KERNEL, q->node);
if (!bfqd)
return -ENOMEM;
eq->elevator_data = bfqd;
-
- spin_lock_irq(&q->queue_lock);
q->elevator = eq;
- spin_unlock_irq(&q->queue_lock);
/*
* Our fallback bfqq if bfq_find_alloc_queue() runs into OOM issues.
* Grab a permanent reference to it, so that the normal code flow
* will not attempt to free it.
@@ -7243,11 +7240,10 @@ static int bfq_init_queue(struct request_queue *q, struct elevator_queue *eq)
bfqd->num_actuators = 1;
/*
* If the disk supports multiple actuators, copy independent
* access ranges from the request queue structure.
*/
- spin_lock_irq(&q->queue_lock);
if (ia_ranges) {
/*
* Check if the disk ia_ranges size exceeds the current bfq
* actuator limit.
*/
@@ -7269,11 +7265,10 @@ static int bfq_init_queue(struct request_queue *q, struct elevator_queue *eq)
/* Otherwise use single-actuator dev info */
if (bfqd->num_actuators == 1) {
bfqd->sector[0] = 0;
bfqd->nr_sectors[0] = get_capacity(q->disk);
}
- spin_unlock_irq(&q->queue_lock);
INIT_LIST_HEAD(&bfqd->dispatch);
hrtimer_setup(&bfqd->idle_slice_timer, bfq_idle_slice_timer, CLOCK_MONOTONIC,
HRTIMER_MODE_REL);
--
2.51.0
^ permalink raw reply related
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox