Re: [PATCH v2 3/2] sched/deadline: Check bandwidth overflow earlier for hotplug

[Dietmar Eggemann <dietmar.eggemann@arm.com>]

On 11/02/2025 10:21, Juri Lelli wrote:
> On 11/02/25 09:36, Dietmar Eggemann wrote:
>> On 10/02/2025 18:09, Juri Lelli wrote:
>>> Hi Christian,
>>>
>>> Thanks for taking a look as well.
>>>
>>> On 07/02/25 15:55, Christian Loehle wrote:
>>>> On 2/7/25 14:04, Jon Hunter wrote:
>>>>>
>>>>>
>>>>> On 07/02/2025 13:38, Dietmar Eggemann wrote:
>>>>>> On 07/02/2025 11:38, Jon Hunter wrote:
>>>>>>>
>>>>>>> On 06/02/2025 09:29, Juri Lelli wrote:
>>>>>>>> On 05/02/25 16:56, Jon Hunter wrote:
>>>>>>>>
>>>>>>>> ...
>>>>>>>>
>>>>>>>>> Thanks! That did make it easier :-)
>>>>>>>>>
>>>>>>>>> Here is what I see ...
>>>>>>>>
>>>>>>>> Thanks!
>>>>>>>>
>>>>>>>> Still different from what I can repro over here, so, unfortunately, I
>>>>>>>> had to add additional debug printks. Pushed to the same branch/repo.
>>>>>>>>
>>>>>>>> Could I ask for another run with it? Please also share the complete
>>>>>>>> dmesg from boot, as I would need to check debug output when CPUs are
>>>>>>>> first onlined.
>>>>>>
>>>>>> So you have a system with 2 big and 4 LITTLE CPUs (Denver0 Denver1 A57_0
>>>>>> A57_1 A57_2 A57_3) in one MC sched domain and (Denver1 and A57_0) are
>>>>>> isol CPUs?
>>>>>
>>>>> I believe that 1-2 are the denvers (even thought they are listed as 0-1 in device-tree).
>>>>
>>>> Interesting, I have yet to reproduce this with equal capacities in isolcpus.
>>>> Maybe I didn't try hard enough yet.
>>>>
>>>>>
>>>>>> This should be easy to set up for me on my Juno-r0 [A53 A57 A57 A53 A53 A53]
>>>>>
>>>>> Yes I think it is similar to this.
>>>>>
>>>>> Thanks!
>>>>> Jon
>>>>>
>>>>
>>>> I could reproduce that on a different LLLLbb with isolcpus=3,4 (Lb) and
>>>> the offlining order:
>>>> echo 0 > /sys/devices/system/cpu/cpu5/online
>>>> echo 0 > /sys/devices/system/cpu/cpu1/online
>>>> echo 0 > /sys/devices/system/cpu/cpu3/online
>>>> echo 0 > /sys/devices/system/cpu/cpu2/online
>>>> echo 0 > /sys/devices/system/cpu/cpu4/online
>>>>
>>>> while the following offlining order succeeds:
>>>> echo 0 > /sys/devices/system/cpu/cpu5/online
>>>> echo 0 > /sys/devices/system/cpu/cpu4/online
>>>> echo 0 > /sys/devices/system/cpu/cpu1/online
>>>> echo 0 > /sys/devices/system/cpu/cpu2/online
>>>> echo 0 > /sys/devices/system/cpu/cpu3/online
>>>> (Both offline an isolcpus last, both have CPU0 online)
>>>>
>>
>> Could reproduce on Juno-r0:
>>
>> 0 1 2 3 4 5
>>
>> L b b L L L
>>
>>       ^^^
>>       isol = [3-4] so both L
>>
>> echo 0 > /sys/devices/system/cpu/cpu1/online
>> echo 0 > /sys/devices/system/cpu/cpu4/online
>> echo 0 > /sys/devices/system/cpu/cpu5/online
>> echo 0 > /sys/devices/system/cpu/cpu2/online - isol
>> echo 0 > /sys/devices/system/cpu/cpu3/online - isol
>>
>>>> The issue only triggers with sugov DL threads (I guess that's obvious, but
>>>> just to mention it).
>>
>> IMHO, it doesn't have to be a sugov DL task. Any DL task will do.
> 
> OK, but in this case we actually want to fail. If we have allocated
> bandwidth for an actual DL task (not a dl server or a 'fake' sugov), we
> don't want to inadvertently leave it w/o bandwidth by turning CPUs off.

Obviously ... ;-)

Same platform w/ isol = [2-3] with slow switching CPUfreq driver to
force having 'sugov' tasks.

# ps2 | grep DLN
   95    95 S 140      0   - DLN sugov:0
   96    96 S 140      0   - DLN sugov:1

# taskset -p 95; taskset -p 96
pid 95's current affinity mask: 39
pid 96's current affinity mask: 6

offline order: CPU1 -> 4 -> 5 -> 3 -> 2

...

pid 95's current affinity mask: 1
pid 96's current affinity mask: 4

root@juno:~# echo 0 > /sys/devices/system/cpu/cpu2/online
[  227.673757] dl_bw_cpus() cpu=6 rd->span=1-5 cpu_active_mask=0,2 cpus=1
[  227.680329] dl_bw_cpus() cpu=6 rd->span=1-5 cpu_active_mask=0,2 cpus=1
[  227.686882] dl_bw_manage: cpu=2 cap=0 fair_server_bw=52428 total_bw=157285 dl_bw_cpus=1 type=DEF span=1-5
[  227.686900] dl_bw_cpus() cpu=6 rd->span=1-5 cpu_active_mask=0,2 cpus=1
[  227.703066] dl_bw_manage() cpu=2 cap=0 overflow=1 return=-16
-bash: echo: write error: Device or resource busy

So it seems 'sugov:1' getting in the way here.

pid 95's current affinity mask: 1
pid 96's current affinity mask: 5

Looks like it's not a 'bL' issue but rather one with '>=2 CPU frequency
policies' and slow-switching CPUfreq drivers.