Re: [PATCH v2 3/2] sched/deadline: Check bandwidth overflow earlier for hotplug

[Dietmar Eggemann <dietmar.eggemann@arm.com>]

On 10/02/2025 18:09, Juri Lelli wrote:
> Hi Christian,
> 
> Thanks for taking a look as well.
> 
> On 07/02/25 15:55, Christian Loehle wrote:
>> On 2/7/25 14:04, Jon Hunter wrote:
>>>
>>>
>>> On 07/02/2025 13:38, Dietmar Eggemann wrote:
>>>> On 07/02/2025 11:38, Jon Hunter wrote:
>>>>>
>>>>> On 06/02/2025 09:29, Juri Lelli wrote:
>>>>>> On 05/02/25 16:56, Jon Hunter wrote:
>>>>>>
>>>>>> ...
>>>>>>
>>>>>>> Thanks! That did make it easier :-)
>>>>>>>
>>>>>>> Here is what I see ...
>>>>>>
>>>>>> Thanks!
>>>>>>
>>>>>> Still different from what I can repro over here, so, unfortunately, I
>>>>>> had to add additional debug printks. Pushed to the same branch/repo.
>>>>>>
>>>>>> Could I ask for another run with it? Please also share the complete
>>>>>> dmesg from boot, as I would need to check debug output when CPUs are
>>>>>> first onlined.
>>>>
>>>> So you have a system with 2 big and 4 LITTLE CPUs (Denver0 Denver1 A57_0
>>>> A57_1 A57_2 A57_3) in one MC sched domain and (Denver1 and A57_0) are
>>>> isol CPUs?
>>>
>>> I believe that 1-2 are the denvers (even thought they are listed as 0-1 in device-tree).
>>
>> Interesting, I have yet to reproduce this with equal capacities in isolcpus.
>> Maybe I didn't try hard enough yet.
>>
>>>
>>>> This should be easy to set up for me on my Juno-r0 [A53 A57 A57 A53 A53 A53]
>>>
>>> Yes I think it is similar to this.
>>>
>>> Thanks!
>>> Jon
>>>
>>
>> I could reproduce that on a different LLLLbb with isolcpus=3,4 (Lb) and
>> the offlining order:
>> echo 0 > /sys/devices/system/cpu/cpu5/online
>> echo 0 > /sys/devices/system/cpu/cpu1/online
>> echo 0 > /sys/devices/system/cpu/cpu3/online
>> echo 0 > /sys/devices/system/cpu/cpu2/online
>> echo 0 > /sys/devices/system/cpu/cpu4/online
>>
>> while the following offlining order succeeds:
>> echo 0 > /sys/devices/system/cpu/cpu5/online
>> echo 0 > /sys/devices/system/cpu/cpu4/online
>> echo 0 > /sys/devices/system/cpu/cpu1/online
>> echo 0 > /sys/devices/system/cpu/cpu2/online
>> echo 0 > /sys/devices/system/cpu/cpu3/online
>> (Both offline an isolcpus last, both have CPU0 online)
>>

Could reproduce on Juno-r0:

0 1 2 3 4 5

L b b L L L

      ^^^
      isol = [3-4] so both L

echo 0 > /sys/devices/system/cpu/cpu1/online
echo 0 > /sys/devices/system/cpu/cpu4/online
echo 0 > /sys/devices/system/cpu/cpu5/online
echo 0 > /sys/devices/system/cpu/cpu2/online - isol
echo 0 > /sys/devices/system/cpu/cpu3/online - isol

>> The issue only triggers with sugov DL threads (I guess that's obvious, but
>> just to mention it).

IMHO, it doesn't have to be a sugov DL task. Any DL task will do.

// on a 2. shell:
# chrt -d -T 5000000 -D 10000000 -P 16666666 -p 0 $$

# ps -eTo comm,pid,class | grep DLN
bash             1243 DLN

5000000/16666666 = 0.3, 0.3 << 10 = 307 (task util, bandwidth requirement)

> It wasn't obvious to me at first :). So thanks for confirming.
> 
>> I'll investigate some more later but wanted to share for now.
> 
> So, problem actually is that I am not yet sure what we should do with
> sugovs' bandwidth wrt root domain accounting. W/o isolation it's all
> good, as it gets accounted for correctly on the dynamic domains sugov
> tasks can run on. But with isolation and sugov affected_cpus that cross
> isolation domains (e.g., one BIG one little), we can get into troubles
> not knowing if sugov contribution should fall on the DEF or DYN domain.

# echo 0 > /sys/devices/system/cpu/cpu1/online
[   87.402722] __dl_bw_capacity() mask=0-2,5 cap=2940
[   87.407551] dl_bw_cpus() cpu=1 rd->span=0-2,5 cpu_active_mask=0-5 cpumask_weight(rd->span)=4
[   87.416019] dl_bw_manage: cpu=1 cap=1916 fair_server_bw=52428 total_bw=524284 dl_bw_cpus=4 type=DYN span=0-2,5

# echo 0 > /sys/devices/system/cpu/cpu2/online
[   95.562270] __dl_bw_capacity() mask=0,2,5 cap=1916
[   95.567091] dl_bw_cpus() cpu=2 rd->span=0,2,5 cpu_active_mask=0,2-5 cpumask_weight(rd->span)=3
[   95.575735] dl_bw_manage: cpu=2 cap=892 fair_server_bw=52428 total_bw=157284 dl_bw_cpus=3 type=DYN span=0,2,5

# echo 0 > /sys/devices/system/cpu/cpu5/online
[  100.573131] __dl_bw_capacity() mask=0,5 cap=892
[  100.577713] dl_bw_cpus() cpu=5 rd->span=0,5 cpu_active_mask=0,3-5 cpumask_weight(rd->span)=2
[  100.586186] dl_bw_manage: cpu=5 cap=446 fair_server_bw=52428 total_bw=104856 dl_bw_cpus=2 type=DYN span=0,5

# echo 0 > /sys/devices/system/cpu/cpu3/online
[  110.232755] __dl_bw_capacity() mask=1-5 cap=892
[  110.237333] dl_bw_cpus() cpu=6 rd->span=1-5 cpu_active_mask=0,3-4 cpus=2
[  110.244064] dl_bw_manage: cpu=3 cap=446 fair_server_bw=52428 total_bw=419428 dl_bw_cpus=2 type=DEF span=1-5


# echo 0 > /sys/devices/system/cpu/cpu4/online
[  175.870273] __dl_bw_capacity() mask=1-5 cap=446
[  175.874850] dl_bw_cpus() cpu=6 rd->span=1-5 cpu_active_mask=0,4 cpus=1
[  175.881407] dl_bw_manage: cpu=4 cap=0 fair_server_bw=52428 total_bw=367000 dl_bw_cpus=1 type=DEF span=1-5
                                   ^^^^^                                                            ^^^^^^^^
                                   w/o/ cpu4 cap is 0!                                              cpu0 is not part of it                                                                                                     
...
[  175.897600] dl_bw_manage() cpu=4 cap=0 overflow=1 return=-16
                                          ^^^^^^^^^^ -EBUSY
                                          
-bash: echo: write error: Device or resource busy

sched_cpu_deactivate()

  dl_bw_deactivate(cpu)

    dl_bw_manage(dl_bw_req_deactivate, cpu, 0);

      return overflow ? -EBUSY : 0;

Looks like in DEF there is no CPU capacity left but we still have 1 DLN
task with a bandwidth requirement of 307.