Re: sched/deadline: Use revised wakeup rule for dl_server

[Andreas Ziegler <br025@umbiko.net>]

On 2026-05-11 09:47, Christian Loehle wrote:
> On 5/9/26 12:42, Andreas Ziegler wrote:
>> Hi Christian, Everyone,
>> 
>> On 2026-05-08 14:13, Christian Loehle wrote:
>>> On 5/8/26 13:06, Andreas Ziegler wrote:
>>>> Hi Christian,
>>>> 
>>>> On 2026-05-08 09:20, Christian Loehle wrote:
>>>>> On 5/8/26 09:09, Andreas Ziegler wrote:
>>>>>> Linux kernel version: 6.12
>>>>>>   CONFIG_PREEMPT_RT (w/ PREEMPT_RT patch applied)
>>>>>> Architecture: aarch64
>>>>>> Platform: Raspberry Pi 4
>>>>>> 
>>>>>> Hi everyone,
>>>>>> 
>>>>>> Commit d66792919d4f (sched/deadline: Use revised wakeup rule for 
>>>>>> dl_server) [1] introduced a marked degradation in scheduling 
>>>>>> latency for real-time tasks in the presence of heavy I/O load.
>>>>>> 
>>>>>> --- a/kernel/sched/deadline.c
>>>>>> +++ b/kernel/sched/deadline.c
>>>>>> @@ -1079,7 +1079,7 @@ static void update_dl_entity(struct 
>>>>>> sched_dl_entity *dl_se)
>>>>>>      if (dl_time_before(dl_se->deadline, rq_clock(rq)) ||
>>>>>>          dl_entity_overflow(dl_se, rq_clock(rq))) {
>>>>>> 
>>>>>> -        if (unlikely(!dl_is_implicit(dl_se) &&
>>>>>> +        if (unlikely((!dl_is_implicit(dl_se) || dl_se->dl_defer) 
>>>>>> &&
>>>>>>                   !dl_time_before(dl_se->deadline, rq_clock(rq)) 
>>>>>> &&
>>>>>>                   !is_dl_boosted(dl_se))) {
>>>>>>              update_dl_revised_wakeup(dl_se, rq);
>>>>>> 
>>>>>> This was observed using a modified version of Con Kolivas' 
>>>>>> interactivity benchmark [2]; kernel bisection eventually pointed 
>>>>>> to the above mentioned commit.
>>>>>> 
>>>>>> Benchmark results before d66792919d4f:
>>>>>> 
>>>>>> --- Benchmarking simulated cpu of Audio real time in the presence 
>>>>>> of simulated ---
>>>>>> Load    Latency +/- SD   median  max [100n]    Desired CPU  
>>>>>> Deadlines met [%]
>>>>>> None      76.6 +/- 8.3654    76  166
>>>>>> Video      78.5 +/- 3.9433    78  107
>>>>>> X      76.4 +/- 8.123     75  157
>>>>>> Burn      72.0 +/- 6.4733    71  127
>>>>>> Write     255.3 +/- 26.627   252  331
>>>>>> Read     226.6 +/- 12.38    227  262
>>>>>> Ring      84.2 +/- 6.6207    83  125
>>>>>> Compile     225.3 +/- 23.949   222  328
>>>>>> 
>>>>>>      136.8 +/- 78.462        331
>>>>>> 
>>>>>> Benchmark results after d66792919d4f:
>>>>>> 
>>>>>> --- Benchmarking simulated cpu of Audio real time in the presence 
>>>>>> of simulated ---
>>>>>> Load    Latency +/- SD   median  max [100n]    Desired CPU  
>>>>>> Deadlines met [%]
>>>>>> None      68.4 +/- 9.7864    67  169
>>>>>> Video      74.4 +/- 3.724     74   97
>>>>>> X      72.0 +/- 6.5681    71  129
>>>>>> Burn      66.9 +/- 5.9059    66  117
>>>>>> Write    9576.9 +/- 67639    250500418        98.1         98.1
>>>>>> Read     209.3 +/- 11.018   209  267
>>>>>> Ring      80.5 +/- 8.0993    78  125
>>>>>> Compile     239.0 +/- 29.447   234  372
>>>>>> 
>>>>>>     1298.4 +/- 24118       500418
>>>>>> 
>>>>>> Reverting this commit obviously solves the issue for me. I have no 
>>>>>> idea why this issue appears exclusively with heavy write loads in 
>>>>>> the background.
>>>>>> 
>>>>>> Is this a scheduler issue, or rather something in the background?
>>>>>> 
>>>>> 
>>>>> Hi Andreas,
>>>>> You're using cpufreq schedutil for your tests I'm assuming?
>>>>> Is there a difference in cpufreq behavior (avg cpufreq or OPP 
>>>>> residencies?)
>>>>> Does the regression also happen on powersave/performance governor?
>>>> 
>>>> Actually this is a very stripped-down system. The 'performance' 
>>>> cpufreq governor is the only one compiled in, the processor cores 
>>>> run on a fixed frequency. CONFIG_PM_OPP is not set.
>>> 
>>> That certainly makes the analysis easier.
>>> I couldn't reproduce the issue so far on my system but it does seem 
>>> like the dl server
>>> would get potentially unbounded running time with very frequent
>>> starting and stopping of the dlserver (which presumably happens 
>>> because of
>>> the writeback) reset the runtime, which then leads to your 25s 
>>> observed latency.
>>> Peter, how is the revised wakeup rule supposed to behave here?
>>> 
>>>> [snip]
>> 
>> This seems to be a case of runtime starvation. If I change 
>> sched_rt_runtime_us to a smaller value, the benchmark returns 
>> reasonable latency values.
>> 
>> # echo "980000" > /proc/sys/kernel/sched_rt_runtime_us
>> 
>> I could live with this workaround, since it seems not to impact 
>> overall latency values in a noticeable way.
>> 
> 
> Not a very stable workaround unfortunately :/
> While I try to reproduce this, what you're observing should imply that 
> the
> background SCHED_NORMAL work is enough to fully utilize the system, 
> right?
> interbench Write does 4k (buffered) writes of a 1GB file and then 
> close+open
> and repeat, nothing fancy really. Does this actually produce 
> significant CPU
> utilization for you? Can you just run the background work and see what 
> that
> looks like?
> (What you're seeing looks like a bug in any case, just so I'm not going 
> down
> a wrong path when trying to reproduce here).

You are right, and this was a false positive; the problem seems to be 
intermittent (maybe 1/20) and I just got lucky for one session.

Some background information about the current state of the system:
   /* CONFIG_CPU_FREQ is not set */
   Root filesystem in RAM (initrd)
   Cpu 3 is isolated: boot parameters: console=tty1 
console=ttyAMA0,115200 isolcpus=nohz,domain,managed_irq,3 nohz_full=3 
rcu_nocbs=3

Background load is normally near 100% idle; this is from top after 
reboot:

Mem: 95724K used, 853524K free, 42408K shrd, 72K buff, 43352K cached
CPU:  0.0% usr  0.0% sys  0.0% nic  100% idle  0.0% io  0.0% irq  0.0% 
sirq
Load average: 0.21 0.17 0.07 3/126 702

The file size used by interbench is even less than 1GB, due to the 
limits of the rootfs. Typical values are around 100-200 MiB. It is 
written in an infinite loop until receiving the stop message (via pipe) 
from the controlling process. The check for the abort signal occurs 
after a completed write, not on block level.

I just noticed that interbench seems to have a bug itself: it uses only 
one processor - looks like a mangled cpu mask. Top output during the 
write benchmark:

Mem: 358024K used, 591224K free, 298516K shrd, 2504K buff, 299464K 
cached
CPU:  1.8% usr 23.1% sys  0.0% nic 74.9% idle  0.0% io  0.0% irq  0.0% 
sirq
Load average: 1.21 0.46 0.29 5/129 2116
   PID  PPID USER     STAT   VSZ %VSZ CPU %CPU COMMAND
  2106  2105 root     S     1228  0.1   0 23.6 interbench -r -t 60 -u -w 
Write -W
  2109  2105 root     S     1228  0.1   0  1.2 interbench -r -t 60 -u -w 
Write -W
  1829  1274 root     R     1600  0.1   2  0.0 top -d 5
    22     2 root     SW       0  0.0   0  0.0 [rcuc/0]
  1270     2 root     IW       0  0.0   0  0.0 [kworker/0:0-eve]
   652     1 mpd      S    27632  2.9   0  0.0 /usr/bin/mpd
  2023  2021 root     S     4476  0.4   0  0.0 sshd-session: root@notty
   675   673 root     S     4448  0.4   1  0.0 sshd-session: root@pts/0
   673   601 root     S     4140  0.4   0  0.0 sshd-session: root [priv]
  2021   601 root     S     4140  0.4   0  0.0 sshd-session: root [priv]
   601     1 root     S     3736  0.3   1  0.0 sshd: /usr/sbin/sshd 
[listener] 0
  2024  2023 root     S     3224  0.3   1  0.0 /usr/libexec/sftp-server
  2025  2023 root     S     3188  0.3   2  0.0 /usr/libexec/sftp-server
   501     1 root     S     1884  0.2   1  0.0 /usr/sbin/wpa_supplicant 
-B -P /va
   131     1 root     S     1672  0.1   0  0.0 /sbin/mdev -df
   676   675 root     S     1636  0.1   1  0.0 -sh
  1274   605 root     S     1636  0.1   1  0.0 -sh
   605     1 root     S     1592  0.1   1  0.0 /usr/sbin/telnetd -F
   527     1 root     S     1576  0.1   2  0.0 udhcpc -t1 -A2 -b -R -O 
search -O
     1     0 root     S     1576  0.1   0  0.0 init

I tried limiting interbench's rather excessive SCHED_FIFO priorities to 
values normal for the system, but without success.