Linux kernel -stable discussions
 help / color / mirror / Atom feed
* [REGRESSION] 6.12.y, 6.18.y: fair server runs SCHED_OTHER ahead of SCHED_FIFO on an idle CPU (fixed in 6.19)
@ 2026-11-20 15:09 Florent Carli
  2026-11-20 17:09 ` Juri Lelli
  0 siblings, 1 reply; 10+ messages in thread
From: Florent Carli @ 2026-11-20 15:09 UTC (permalink / raw)
  To: stable; +Cc: Peter Zijlstra, Juri Lelli, Vincent Guittot, linux-rt-users

Hello,

As I understand it, the fair server is deferred: it should run
SCHED_OTHER tasks ahead of RT tasks only when RT tasks have kept the
CPU busy for most of a period (about 950 ms out of 1 s), so that fair
tasks do not starve.

On 6.12.y and 6.18.y, I see it run SCHED_OTHER tasks ahead of RT tasks
on a CPU where nothing starves. My setup is a PREEMPT_RT system (Debian
trixie, 6.12.117 rt) with a time critical SCHED_FIFO 98 thread on an
isolated CPU. That CPU is idle most of the time. The thread wakes up
every 300 us and runs for a few us. From time to time, a SCHED_OTHER
task that wakes up on that CPU runs ahead of the FIFO thread, for as
long as it wants to run: a container health check (bash) delayed the
FIFO thread by 3.4 ms this way.

6.19.y and later do not show this.

My reading of the 6.12.y code: about one second after the last fair
activity on the CPU, the zero-laxity timer fires and replenishes the
server with dl_defer_running = 1. Idle time then drains its runtime,
and dl_server_update_idle_time() clears dl_defer_running after about
50 ms of idle. A fair task that wakes up inside those ~50 ms runs under
the server, ahead of any FIFO task, until it sleeps.

Reproducer, as root, on an isolated CPU with nothing else on it:

  CPU=22
  python3 - "$CPU" <<'EOF' &
  import os, sys, time
  os.sched_setaffinity(0, {int(sys.argv[1])})
  while True:
      time.sleep(1.02)
      end = time.monotonic() + 0.003
      while time.monotonic() < end:
          pass
  EOF
  cyclictest -m -p 98 -a $CPU --mainaffinity=0 -t 1 -i 200 -D 30 -q

--mainaffinity matters: cyclictest's main thread is SCHED_OTHER, and
when it stays on $CPU its activity every ~60 ms keeps pushing the
server's period out and hides the problem. Any other fair activity on
the CPU more often than every ~950 ms does the same.

Results, sleep between 1.00 and 1.04 s, 3 ms spin. Max FIFO latency,
and wake-ups later than 500 us:

  bare metal, 6.12.117 Debian rt         3001 us, 10 per 20 s
  (10 ms spin instead of 3 ms)           9986 us
  KVM guest on dedicated host CPUs:
  6.12.117 Debian rt                     3000 us, 10 per 20 s
  6.12.118, vanilla PREEMPT_RT           2900 us, 8 in 270 s
  6.12.118 + v6.12-rt queue              3000 us, 14-15 per 30 s
  6.18.15 Debian rt                      3000 us, 10 per 20 s
  6.19.14, 7.3.4 Debian rt               < 25 us, 0
  all of the above with
  fair_server/cpuN/runtime = 0           < 25 us, 0

The lazy preemption patches of the -rt queue make the window much
easier to hit, but vanilla 6.12.118 is affected too.

The fix went into 6.19:

  e636ffb9e31b ("sched/deadline: Fix dl_server time accounting")
  f5a538c07df2 ("sched/deadline: Fix dl_server stop condition")
  1e0a2ba7afb1 ("sched: Provide idle_rq() helper")
  ca1e8eede4fc ("sched/deadline: Fix server stopping with runnable tasks")

None of them carries a stable tag, and only 1e0a2ba7afb1 applies
cleanly to 6.12.y or 6.18.y.

I have a backport of the four to 6.12.118, with small adaptations
(rq->donor does not exist in 6.12, so rq->curr is used). With it, the
reproducer shows no late wake-up, both on vanilla 6.12.118 and with the
-rt queue, and the starvation guard still works: under a SCHED_FIFO
busy loop, a SCHED_OTHER task on the same CPU still gets 4.7 to 4.9%
of it, as without the backport (0% with the server runtime at 0).

Would you take this series for 6.18.y and 6.12.y? I can send the 6.12.y
backport as patches and prepare the 6.18.y one, or leave it to the
authors if they prefer.

Until then, writing 0 to /sys/kernel/debug/sched/fair_server/cpuN/runtime
on the isolated CPUs avoids the problem.

Thanks,
Florent.

^ permalink raw reply	[flat|nested] 10+ messages in thread

end of thread, other threads:[~2026-11-21 15:27 UTC | newest]

Thread overview: 10+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2026-11-20 15:09 [REGRESSION] 6.12.y, 6.18.y: fair server runs SCHED_OTHER ahead of SCHED_FIFO on an idle CPU (fixed in 6.19) Florent Carli
2026-11-20 17:09 ` Juri Lelli
2026-11-20 18:19   ` Florent Carli
2026-11-21 15:27     ` Sasha Levin
2026-11-20 18:19   ` [PATCH 6.12.y 0/4] sched/deadline: dl_server fixes from 6.19 Florent Carli
2026-11-20 18:19     ` [PATCH 6.12.y 1/4] sched/deadline: Fix dl_server time accounting Florent Carli
2026-11-20 18:19     ` [PATCH 6.12.y 2/4] sched/deadline: Fix dl_server stop condition Florent Carli
2026-11-20 18:19     ` [PATCH 6.12.y 3/4] sched: Provide idle_rq() helper Florent Carli
2026-11-20 18:19     ` [PATCH 6.12.y 4/4] sched/deadline: Fix server stopping with runnable tasks Florent Carli
2026-11-21 15:27     ` [PATCH 6.12.y 0/4] sched/deadline: dl_server fixes from 6.19 Sasha Levin

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox