Re: [PATCH v2 3/2] sched/deadline: Check bandwidth overflow earlier for hotplug

Jon Hunter
2025-02-05T16:56:43Z

Hi Juri,

On 05/02/2025 10:12, Juri Lelli wrote:
> On 05/02/25 07:53, Juri Lelli wrote:
>> On 03/02/25 11:01, Jon Hunter wrote:
>>> Hi Juri,
>>>
>>> On 16/01/2025 15:55, Juri Lelli wrote:
>>>> On 16/01/25 13:14, Jon Hunter wrote:
>>
>> ...
>>
>>>>> [ 210.595431] dl_bw_manage: cpu=5 cap=3072 fair_server_bw=52428 total_bw=209712 dl_bw_cpus=4
>>>>> [ 210.606269] dl_bw_manage: cpu=4 cap=2048 fair_server_bw=52428 total_bw=157284 dl_bw_cpus=3
>>>>> [ 210.617281] dl_bw_manage: cpu=3 cap=1024 fair_server_bw=52428 total_bw=104856 dl_bw_cpus=2
>>>>> [ 210.627205] dl_bw_manage: cpu=2 cap=1024 fair_server_bw=52428 total_bw=262140 dl_bw_cpus=2
>>>>> [ 210.637752] dl_bw_manage: cpu=1 cap=0 fair_server_bw=52428 total_bw=262140 dl_bw_cpus=1
>>>> ^
>>>> Different than before but still not what I expected. Looks like there
>>>> are conditions/path I currently cannot replicate on my setup, so more
>>>> thinking. Unfortunately I will be out traveling next week, so this
>>>> might required a bit of time.
>>>
>>>
>>> I see that this is now in the mainline and our board is still failing to
>>> suspend. Let me know if there is anything else you need me to test.
>>
>> Ah, can you actually add 'sched_verbose' and to your kernel cmdline? It
>> should print our additional debug info on the console when domains get
>> reconfigured by hotplug/suspends, e.g.
>>
>> dl_bw_manage: cpu=3 cap=3072 fair_server_bw=52428 total_bw=209712 dl_bw_cpus=4
>> CPU0 attaching NULL sched-domain.
>> CPU3 attaching NULL sched-domain.
>> CPU4 attaching NULL sched-domain.
>> CPU5 attaching NULL sched-domain.
>> CPU0 attaching sched-domain(s):
>> domain-0: span=0,4-5 level=MC
>> groups: 0:{ span=0 cap=766 }, 4:{ span=4 cap=908 }, 5:{ span=5 cap=989 }
>> CPU4 attaching sched-domain(s):
>> domain-0: span=0,4-5 level=MC
>> groups: 4:{ span=4 cap=908 }, 5:{ span=5 cap=989 }, 0:{ span=0 cap=766 }
>> CPU5 attaching sched-domain(s):
>> domain-0: span=0,4-5 level=MC
>> groups: 5:{ span=5 cap=989 }, 0:{ span=0 cap=766 }, 4:{ span=4 cap=908 }
>> root domain span: 0,4-5
>> rd 0,4-5: Checking EAS, CPUs do not have asymmetric capacities
>> psci: CPU3 killed (polled 0 ms)
>>
>> Can you please share this information as well if you are able to collect
>> it (while still running with my last proposed fix)?
>
> Also, if you don't mind, add the following on top of the existing
> changes.
>
> Just to be sure we don't get out of sync, I pushed current set to
>
> https://github.com/jlelli/linux.git experimental/dl-debug

Thanks! That did make it easier :-)

Here is what I see ...

[ 53.823979] PM: suspend entry (deep)
[ 53.827715] Filesystems sync: 0.000 seconds
[ 53.832859] Freezing user space processes
[ 53.838132] Freezing user space processes completed (elapsed 0.001 seconds)
[ 53.845118] OOM killer disabled.
[ 53.848348] Freezing remaining freezable tasks
[ 53.853884] Freezing remaining freezable tasks completed (elapsed 0.001 seconds)
[ 53.900686] tegra-xusb 3530000.usb: Firmware timestamp: 2020-07-06 13:39:28 UTC
[ 53.918492] dwc-eth-dwmac 2490000.ethernet eth0: Link is Down
[ 53.962316] Disabling non-boot CPUs ...
[ 53.966192] dl_bw_manage: cpu=5 cap=3072 fair_server_bw=52428 total_bw=209712 dl_bw_cpus=4 type=DYN span=0,3-5
[ 53.966231] CPU0 attaching NULL sched-domain.
[ 53.980574] span=1-2
[ 53.982767] CPU3 attaching NULL sched-domain.
[ 53.987119] span=0-2
[ 53.989309] CPU4 attaching NULL sched-domain.
[ 53.993662] span=0-3
[ 53.995853] CPU5 attaching NULL sched-domain.
[ 54.000206] span=0-4
[ 54.002433] CPU0 attaching sched-domain(s):
[ 54.006614] domain-0: span=0,3-4 level=MC
[ 54.010711] groups: 0:{ span=0 cap=1022 }, 3:{ span=3 cap=1022 }, 4:{ span=4 }
[ 54.018126] CPU3 attaching sched-domain(s):
[ 54.022307] domain-0: span=0,3-4 level=MC
[ 54.026404] groups: 3:{ span=3 cap=1022 }, 4:{ span=4 }, 0:{ span=0 cap=1023 }
[ 54.033821] CPU4 attaching sched-domain(s):
[ 54.038001] domain-0: span=0,3-4 level=MC
[ 54.042098] groups: 4:{ span=4 }, 0:{ span=0 cap=1023 }, 3:{ span=3 cap=1022 }
[ 54.049508] root domain span: 0,3-4
[ 54.052997] default domain span: 1-2,5
[ 54.056756] rd 0,3-4: Checking EAS, CPUs do not have asymmetric capacities
[ 54.064688] psci: CPU5 killed (polled 0 ms)
[ 54.069495] dl_bw_manage: cpu=4 cap=2048 fair_server_bw=52428 total_bw=157284 dl_bw_cpus=3 type=DYN span=0,3-4
[ 54.069547] CPU0 attaching NULL sched-domain.
[ 54.083910] span=1-2,5
[ 54.086277] CPU3 attaching NULL sched-domain.
[ 54.090633] span=0-2,5
[ 54.092999] CPU4 attaching NULL sched-domain.
[ 54.097351] span=0-3,5
[ 54.099756] CPU0 attaching sched-domain(s):
[ 54.103941] domain-0: span=0,3 level=MC
[ 54.107865] groups: 0:{ span=0 }, 3:{ span=3 cap=1023 }
[ 54.113279] CPU3 attaching sched-domain(s):
[ 54.117459] domain-0: span=0,3 level=MC
[ 54.121382] groups: 3:{ span=3 cap=1023 }, 0:{ span=0 }
[ 54.126793] root domain span: 0,3
[ 54.130109] default domain span: 1-2,4-5
[ 54.134040] rd 0,3: Checking EAS, CPUs do not have asymmetric capacities
[ 54.141597] psci: CPU4 killed (polled 0 ms)
[ 54.146819] dl_bw_manage: cpu=3 cap=1024 fair_server_bw=52428 total_bw=104856 dl_bw_cpus=2 type=DYN span=0,3
[ 54.156727] CPU0 attaching NULL sched-domain.
[ 54.161086] span=1-2,4-5
[ 54.163632] CPU3 attaching NULL sched-domain.
[ 54.167988] span=0-2,4-5
[ 54.170553] CPU0 attaching NULL sched-domain.
[ 54.174909] span=0-5
[ 54.177096] root domain span: 0
[ 54.180239] default domain span: 1-5
[ 54.183821] rd 0: Checking EAS, CPUs do not have asymmetric capacities
[ 54.191728] psci: CPU3 killed (polled 4 ms)
[ 54.196389] dl_bw_manage: cpu=2 cap=1024 fair_server_bw=52428 total_bw=262140 dl_bw_cpus=2 type=DEF span=1-5
[ 54.196518] rd 0: Checking EAS, CPUs do not have asymmetric capacities
[ 54.214816] psci: CPU2 killed (polled 0 ms)
[ 54.219411] dl_bw_manage: cpu=1 cap=0 fair_server_bw=52428 total_bw=262140 dl_bw_cpus=1 type=DEF span=1-5
[ 54.219493] Error taking CPU1 down: -16
[ 54.232948] Non-boot CPUs are not disabled
[ 54.237046] Enabling non-boot CPUs ...
[ 54.241216] Detected PIPT I-cache on CPU2
[ 54.245258] CPU features: SANITY CHECK: Unexpected variation in SYS_CTR_EL0. Boot CPU: 0x0000008444c004, CPU2: 0x0000009444c004
[ 54.256744] CPU features: SANITY CHECK: Unexpected variation in SYS_ID_AA64DFR0_EL1. Boot CPU: 0x00000010305106, CPU2: 0x00000010305116
[ 54.268954] CPU features: SANITY CHECK: Unexpected variation in SYS_ID_DFR0_EL1. Boot CPU: 0x00000003010066, CPU2: 0x00000003001066
[ 54.280865] CPU2: Booted secondary processor 0x0000000001 [0x4e0f0030]
[ 54.288270] rd 0: Checking EAS, CPUs do not have asymmetric capacities
[ 54.295061] CPU2 is up
[ 54.297599] Detected PIPT I-cache on CPU3
[ 54.301642] CPU3: Booted secondary processor 0x0000000101 [0x411fd073]
[ 54.308419] CPU0 attaching NULL sched-domain.
[ 54.312786] span=1-5
[ 54.315031] CPU0 attaching sched-domain(s):
[ 54.319220] domain-0: span=0,3 level=MC
[ 54.323145] groups: 0:{ span=0 }, 3:{ span=3 cap=1016 }
[ 54.328564] CPU3 attaching sched-domain(s):
[ 54.332746] domain-0: span=0,3 level=MC
[ 54.336671] groups: 3:{ span=3 cap=1016 }, 0:{ span=0 }
[ 54.342080] root domain span: 0,3
[ 54.345405] default domain span: 1-2,4-5
[ 54.349338] rd 0,3: Checking EAS, CPUs do not have asymmetric capacities
[ 54.356122] CPU3 is up
[ 54.358649] Detected PIPT I-cache on CPU4
[ 54.362677] CPU4: Booted secondary processor 0x0000000102 [0x411fd073]
[ 54.369399] CPU0 attaching NULL sched-domain.
[ 54.373767] span=1-2,4-5
[ 54.376310] CPU3 attaching NULL sched-domain.
[ 54.380667] span=0-2,4-5
[ 54.383251] CPU0 attaching sched-domain(s):
[ 54.387439] domain-0: span=0,3-4 level=MC
[ 54.391538] groups: 0:{ span=0 }, 3:{ span=3 cap=1021 }, 4:{ span=4 }
[ 54.398173] CPU3 attaching sched-domain(s):
[ 54.402356] domain-0: span=0,3-4 level=MC
[ 54.406456] groups: 3:{ span=3 cap=1021 }, 4:{ span=4 }, 0:{ span=0 }
[ 54.413090] CPU4 attaching sched-domain(s):
[ 54.417271] domain-0: span=0,3-4 level=MC
[ 54.421373] groups: 4:{ span=4 }, 0:{ span=0 }, 3:{ span=3 cap=1021 }
[ 54.428005] root domain span: 0,3-4
[ 54.431503] default domain span: 1-2,5
[ 54.435259] rd 0,3-4: Checking EAS, CPUs do not have asymmetric capacities
[ 54.442287] CPU4 is up
[ 54.444821] Detected PIPT I-cache on CPU5
[ 54.448848] CPU5: Booted secondary processor 0x0000000103 [0x411fd073]
[ 54.455574] CPU0 attaching NULL sched-domain.
[ 54.459950] span=1-2,5
[ 54.462315] CPU3 attaching NULL sched-domain.
[ 54.466674] span=0-2,5
[ 54.469042] CPU4 attaching NULL sched-domain.
[ 54.473401] span=0-3,5
[ 54.475812] CPU0 attaching sched-domain(s):
[ 54.480000] domain-0: span=0,3-5 level=MC
[ 54.484099] groups: 0:{ span=0 }, 3:{ span=3 }, 4:{ span=4 }, 5:{ span=5 }
[ 54.491171] CPU3 attaching sched-domain(s):
[ 54.495352] domain-0: span=0,3-5 level=MC
[ 54.499452] groups: 3:{ span=3 }, 4:{ span=4 }, 5:{ span=5 }, 0:{ span=0 }
[ 54.506519] CPU4 attaching sched-domain(s):
[ 54.510703] domain-0: span=0,3-5 level=MC
[ 54.514800] groups: 4:{ span=4 }, 5:{ span=5 }, 0:{ span=0 }, 3:{ span=3 }
[ 54.521869] CPU5 attaching sched-domain(s):
[ 54.526050] domain-0: span=0,3-5 level=MC
[ 54.530150] groups: 5:{ span=5 }, 0:{ span=0 }, 3:{ span=3 }, 4:{ span=4 }
[ 54.537217] root domain span: 0,3-5
[ 54.540716] default domain span: 1-2
[ 54.544303] rd 0,3-5: Checking EAS, CPUs do not have asymmetric capacities
[ 54.551393] rd 0,3-5: Checking EAS, CPUs do not have asymmetric capacities
[ 54.558281] CPU5 is up
[ 54.568000] dwc-eth-dwmac 2490000.ethernet eth0: configuring for phy/rgmii link mode
[ 55.585391] dwc-eth-dwmac 2490000.ethernet: Failed to reset the dma
[ 55.591664] dwc-eth-dwmac 2490000.ethernet eth0: stmmac_hw_setup: DMA engine initialization failed
[ 55.600905] dwc-eth-dwmac 2490000.ethernet eth0: Link is Up - 1Gbps/Full - flow control rx/tx
[ 55.615015] usb-conn-gpio 3520000.padctl:ports:usb2-0:connector: repeated role: device
[ 55.617967] tegra-xusb 3530000.usb: Firmware timestamp: 2020-07-06 13:39:28 UTC
[ 55.655665] OOM killer enabled.
[ 55.658813] Restarting tasks ... done.
[ 55.664082] random: crng reseeded on system resumption
[ 55.664403] VDDIO_SDMMC3_AP: voltage operation not allowed
[ 55.674862] PM: suspend exit

--
nvpublic