[PATCH 0/2] sched: expose RT throttling info to facilitate priority reversal processing

[wen.yang@linux.dev]

From: Wen Yang <wen.yang@linux.dev>

This series helps to solve a priority reversal issue, where a CFS task
waits for an rtmutex lock, an RT task holding the lock stops due to
RT throttling, and higher priority RT tasks frequently trigger RT
throttling due to long-term CPU consumption.

Details of it:

A priority inversion scenario can occur when a CFS task is starved
due to RT throttling. The scenario is as follows:

0. An rtmutex (e.g., softirq_ctrl.lock) is contended by both CFS
   tasks (e.g., ksoftirqd) and RT tasks (e.g., ktimer).
1. An RT task 'A' (e.g., ktimer) acquired the rtmutex.
2. A CFS task 'B' (e.g., ksoftirqd) attempts to acquire the same
   rtmutex and blocks.
3. A higher-priority RT task 'C' (e.g., stress-ng) runs for an
   extended period, preempting task 'A' and causing the RT runqueue
   to be throttled.
4. Once throttled, CFS task 'B' should run, but it remains blocked
   because the lock is still held by the non-running RT task 'A'. This
   can even lead to the CPU going idle.
5. When the throttle period ends, the high-priority RT task 'C'
   resumes execution, and the cycle repeats, leading to indefinite
   starvation of CFS task 'B'.

A typical stack trace for the blocked ksoftirqd shows it in a 'D'
(TASK_RTLOCK_WAIT) state, waiting on the lock:
     ksoftirqd/5-61      [005] d...211 58212.064160: sched_switch: prev_comm=ksoftirqd/5 prev_pid=61 prev_prio=120 prev_state=D ==> next_comm=swapper/5 next_pid=0 next_prio=120
     ksoftirqd/5-61      [005] d...211 58212.064161: <stack trace>
 => __schedule
 => schedule_rtlock
 => rtlock_slowlock_locked
 => rt_spin_lock
 => __local_bh_disable_ip
 => run_ksoftirqd
 => smpboot_thread_fn
 => kthread
 => ret_from_fork

These two patches expose the TASK_RTLOCK_WAIT state and add
throttle_count to rt_rq for monitoring in /proc/sched_debug.

User-space tools like stalld can use this info to detect and resolve
the inversion, for example, by boosting the lock holder or adjusting 
the priority of the blocked CFS task in TASK_RTLOCK_WAIT state.

Wen Yang (2):
  sched/debug: add explicit TASK_RTLOCK_WAIT printing
  sched/rt: add RT throttle statistics

 fs/proc/array.c              |  3 ++-
 include/linux/sched.h        | 21 +++++++++------------
 include/trace/events/sched.h |  1 +
 kernel/sched/debug.c         |  1 +
 kernel/sched/rt.c            |  1 +
 kernel/sched/sched.h         |  1 +
 6 files changed, 15 insertions(+), 13 deletions(-)

Cc: linux-kernel@vger.kernel.org
-- 
2.25.1