[RFC PATCH v5 00/29] Hierarchical Constant Bandwidth Server

[Yuri Andriaccio <yurand2000@gmail.com>]

Hello,

This is the v5 for Hierarchical Constant Bandwidth Server, aiming at replacing
the current RT_GROUP_SCHED mechanism with something more robust and
theoretically sound. The patchset has been presented at OSPM25 and OSPM26
(https://retis.sssup.it/ospm-summit/), and a summary of its inner workings can
be found at https://lwn.net/Articles/1021332/ . You can find the previous
versions of this patchset at the bottom of the page, in particular version 1
which talks in more detail what this patchset is all about and how it is
implemented.

This v5 version works on the comments by the reviewers and introduces the
following meaningful changes:
- Update to kernel version 7.0.
- General refactorings, cleanups, extensive use of lock guard for cleaner code.
- Add missing rcu read sections in deadline.c and rt.c code.
- Include fix for non-deferred deadline server logic (Patch 1).
- Account HCBS deadline servers along with all the active tasks when the servers
  are active. This ensures correct behaviour for servers that are just
  replenished but have no tasks to run.
- Update and reuse __checkparam_dl to also check for HCBS servers' parameters.
- Update default sysctl_sched_rt_runtime to 1s, as sysctl_sched_rt_period. These
  parameters only manage the deadline tasks' and servers' bandwidth, not the
  actual parameters of the fair (and ext) servers.
- Add early release of cgroup resources in unregister_rt_sched_group, reducing
  from two to one the number of RCU grace periods to wait for the release of
  reserved deadline bandwidth.
- Remove rt_server_try_pull, as it is now possible to pull tasks directly in
  rt_server_pick on server replenish.
- Remove dl_server_stop call when emptying a cgroup's runqueue, as the server is
  nonetheless stopped on the next server pick (if the pull operation fails).

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Summary of the patches:
     1) Replenishment logic fix for non-deferred deadline servers
   2-5) Preparation patches, so that the RT classes' code can be used both
        for normal and cgroup scheduling.
  6-17) Implementation of HCBS, no migration and only one level hierarchy.
        The old RT_GROUP_SCHED code is removed.
 18-19) Remove cgroups v1 in favour of v2.
    20) Add support for deeper hierarchies.
    21) Update default bandwidth for deadline entities.
 22-26) Add support for tasks migration.
    27) Documentation for HCBS.
 28-29) Debug BUG_ONs optional patches.

Updates from v4:
- Rebase to latest tip/master.
- General rebasing/cleanup.
- Update default sysctl_sched_rt_runtime to 1s, same as the period.
- Fix non-deferred deadline server replenishment logic.
- Add missing RCU read sections.
- Account HCBS servers along with their tasks when the servers are active.
- Release bandwidth resources early in unregister_rt_sched_group.
- Drop server_try_pull_task as it is now redundant.
- Remove dl_server_stop call in dequeue_task_rt.
- Update to reuse __checkparam_dl for deadline servers.

Updates from v3:
- Rebase to latest tip/master.
- General rebasing/cleanup.
- Add Documentation.
- Define **live** and **active** groups.
- Introduce server_try_pull_task in place of the removed server_has_task.
- Introduce RELEASE_LOCK helper macro for guard-based locking.
- Update inc/dec_dl_tasks to account for served runqueues regardless of the
  server type.
- Fix computing of new bandwidth values in dl_init_tg.
- Fix check in dl_check_tg to use capacity scaling.
- Fix wakeup_preempt_rt to check if curr is a DEADLINE task.

Updates from v2:
- Rebase to latest tip/master.
- Remove fair-servers' bw reclaiming.
- Fix a check which prevented execution of wakeup_preempt code.
- Fix a priority check in group_pull_rt_task between tasks of different groups.
- Rework allocation/deallocation code for rt-cgroups.
- Update signatures for some group related migration functions.
- Add documentation for wakeup_preempt preemption rules.

Updates from v1:
- Rebase to latest tip/master.
- Add migration code.
- Split big patches for more readability.
- Refactor code to use guarded locks where applicable.
- Remove unnecessary patches from v1 which have been addressed differently by
  mainline updates.
- Remove unnecessary checks and general code cleanup.

Notes:

Patch 1 has already been submitted for review at:
https://lore.kernel.org/all/20260420163410.20808-1-yurand2000@gmail.com/

Patches 28-29 are completely optional and are not meant to be included in the
final patchset: they just add some invasive BUG_ONs that assert some
preconditions expected on some function calls.

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Testing v5:

The patchset has been tested with a suite of tests tailored to stress all the
implemented functionalities.
The tests are available at https://github.com/Yurand2000/HCBS-Test-Suite .
Refer to the README of the repository for more details.

Follow these steps to test HCBS v5:
- Get the HCBS patch up and running. Any kernel/disto should work effortlessly.
- Get, compile and _install_ the tests.
- Run the `go_rt.sh` script to set the frequency of the CPUs to a fixed value
  and disable hyperthreading and power saving features.
- Run the `run_tests.sh full` script, to run the whole test suite.

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Future Work:

We think the current patchset is stable enough. Our current test suite
demonstrates, on our limited hardware, that the kernel does not throw warnings
and that it is actually possible to guarantee time reservations and isolation
among tenants.

In the hope that the pre-migration patches (2-19) have reached a decent final
form, we of course expect comments on the migration related code (22-26) and the
other patches (1,20-21).

Since the updates on the latest comments were already worked onto, we've decided
to release v5 without the multiCPU feature, presented at OSPM26, as the code is
not yet fully tested and cleaned, in the hope to release it in a future v6 RFC.

Additional future work:
 - capacity aware bandwidth reservation.
 - hotplug/hotunplug management.

Have a nice day,
Yuri

v1: https://lore.kernel.org/all/20250605071412.139240-1-yurand2000@gmail.com/
v2: https://lore.kernel.org/all/20250731105543.40832-1-yurand2000@gmail.com/
v3: https://lore.kernel.org/all/20250929092221.10947-1-yurand2000@gmail.com/
v4: https://lore.kernel.org/all/20251201124205.11169-1-yurand2000@gmail.com/

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

Yuri Andriaccio (13):
  sched/deadline: Fix replenishment logic for non-deferred servers
  sched/rt: Disable RT_GROUP_SCHED
  sched/rt: Remove unnecessary runqueue pointer in struct rt_rq
  sched/rt: Implement dl-server operations for rt-cgroups
  sched/rt: Update task event callbacks for HCBS scheduling
  sched/rt: Allow zeroing the runtime of the root control group
  sched/rt: Remove support for cgroups-v1
  sched/rt: Update default bandwidth for real-time tasks to ONE
  sched/rt: Try pull task on empty server pick.
  sched/core: Execute enqueued balance callbacks after
    migrate_disable_switch
  Documentation: Update documentation for real-time cgroups
  sched/rt: Add debug BUG_ONs for pre-migration code
  sched/rt: Add debug BUG_ONs in migration code

luca abeni (16):
  sched/deadline: Do not access dl_se->rq directly
  sched/deadline: Distinguish between dl_rq and my_q
  sched/rt: Pass an rt_rq instead of an rq where needed
  sched/rt: Move functions from rt.c to sched.h
  sched/rt: Introduce HCBS specific structs in task_group
  sched/core: Initialize HCBS specific structures
  sched/deadline: Add dl_init_tg
  sched/rt: Add {alloc/unregister/free}_rt_sched_group
  sched/deadline: Account rt-cgroups bandwidth in deadline tasks
    schedulability tests.
  sched/rt: Update rt-cgroup schedulability checks
  sched/rt: Remove old RT_GROUP_SCHED data structures
  sched/core: Cgroup v2 support
  sched/deadline: Allow deeper hierarchies of RT cgroups
  sched/rt: Add rt-cgroup migration functions
  sched/rt: Hook HCBS migration functions
  sched/core: Execute enqueued balance callbacks when changing allowed
    CPUs

 Documentation/scheduler/sched-rt-group.rst |  504 ++-
 include/linux/rcupdate.h                   |    1 +
 include/linux/sched.h                      |   10 +-
 kernel/sched/autogroup.c                   |    4 +-
 kernel/sched/core.c                        |   74 +-
 kernel/sched/deadline.c                    |  251 +-
 kernel/sched/debug.c                       |    6 -
 kernel/sched/ext.c                         |    4 +-
 kernel/sched/fair.c                        |    4 +-
 kernel/sched/rt.c                          | 3240 ++++++++++----------
 kernel/sched/sched.h                       |  178 +-
 kernel/sched/syscalls.c                    |    9 +-
 12 files changed, 2393 insertions(+), 1892 deletions(-)


base-commit: 028ef9c96e96197026887c0f092424679298aae8
--
2.53.0