Re: [PATCH v3 00/14] thermal: Rework binding cooling devices to trip points

Rafael J. Wysocki
2024-08-26T09:58:26Z

On Sat, Aug 24, 2024 at 8:45 PM Nícolas F. R. A. Prado
<nfraprado@collabora.com> wrote:
>
> On Mon, Aug 19, 2024 at 05:49:07PM +0200, Rafael J. Wysocki wrote:
> > Hi Everyone,
> >
> > This is one more update of
> >
> > https://lore.kernel.org/linux-pm/3134863.CbtlEUcBR6@rjwysocki.net/#r
> >
> > the cover letter of which was sent separately by mistake:
> >
> > https://lore.kernel.org/linux-pm/CAJZ5v0jo5vh2uD5t4GqBnN0qukMBG_ty33PB=NiEqigqxzBcsw@mail.gmail.com/
> >
> > and it has been updated once already:
> >
> > https://lore.kernel.org/linux-pm/114901234.nniJfEyVGO@rjwysocki.net/
> >
> > Relative to the v2 above it drops 3 patches, one because it was broken ([04/17
> > in the v2), and two more that would need to be rebased significantly, either
> > because of dropping the other broken patch or because of the recent Bang-bang
> > governor fixes:
> >
> > https://lore.kernel.org/linux-pm/1903691.tdWV9SEqCh@rjwysocki.net/
> >
> > The remaining 14 patches, 2 of which have been slightly rebased and the rest
> > is mostly unchanged (except for some very minor subject and changelog fixes),
> > is not expected to be controversial and are targeting 6.12, on top of the
> > current linux-next material.
> >
> > The original motivation for this series quoted below has not changed:
> >
> > The code for binding cooling devices to trip points (and unbinding them from
> > trip point) is one of the murkiest pieces of the thermal subsystem. It is
> > convoluted, bloated with unnecessary code doing questionable things, and it
> > works backwards.
> >
> > The idea is to bind cooling devices to trip points in accordance with some
> > information known to the thermal zone owner (thermal driver). This information
> > is not known to the thermal core when the thermal zone is registered, so the
> > driver needs to be involved, but instead of just asking the driver whether
> > or not the given cooling device should be bound to a given trip point, the
> > thermal core expects the driver to carry out all of the binding process
> > including calling functions specifically provided by the core for this
> > purpose which is cumbersome and counter-intuitive.
> >
> > Because the driver has no information regarding the representation of the trip
> > points at the core level, it is forced to walk them (and it has to avoid some
> > locking traps while doing this), or it needs to make questionable assumptions
> > regarding the ordering of the trips in the core. There are drivers doing both
> > these things.
> >
> > The first 5 patches in the series are preliminary.
> >
> > Patch [06/14] introduces a new .should_bind() callback for thermal zones and
> > patches [07,09-12/14] modifies drivers to use it instead of the .bind() and
> > .unbind() callbacks which allows them to be simplified quite a bit.
> >
> > The other patches [08,13-14/14] get rid of code that becomes unused after the
> > previous changes and do some cleanups on top of that.
> >
> > The entire series along with 2 patches on top of it (that were present in the
> > v2 of this set of patches) is available in the thermal-core-testing git branch:
> >
> > https://git.kernel.org/pub/scm/linux/kernel/git/rafael/linux-pm.git/log/?h=thermal-core-testing
> >
> > (note that this branch is going to be rebased shortly on top of 6.11-rc4
> > and the thermal control material in linux-next).
> >
> > Thanks!
>
> Hi,
>
> KernelCI has identified a boot regression originating from this series. I've
> verified that reverting the series fixes the issue.

Thanks for the report!

There was a bug in the original patch [12/14] that would cause
symptoms like what you are observing to appear, which was reported on
Friday and has since been fixed in the tree. Please see:

https://lore.kernel.org/linux-pm/CAJZ5v0iw7uXE_cfU5VXOjFDg9GM8Hu0+hKxqfzU3v0OM5KK9oQ@mail.gmail.com/

You probably have not tested the fixed tree yet, so please let
kernelci run again on it and if the issue is still there, please let
me know.

> Affected platforms:
> * mt8195-cherry-tomato-r2
> * mt8192-asurada-spherion-r0
> * mt8183-kukui-jacuzzi-juniper-sku16
> * mt8186-corsola-steelix-sku131072
> * sc7180-trogdor-kingoftown
> * sc7180-trogdor-lazor-limozeen
>
> Relevant log from mt8195-cherry-tomato-r2 (with additional debug configs
> enabled):
>
> [ 11.326726] BUG: sleeping function called from invalid context at kernel/locking/rwsem.c:1578
> [ 11.335294] in_atomic(): 1, irqs_disabled(): 1, non_block: 0, pid: 165, name: udevd
> [ 11.342944] preempt_count: 1, expected: 0
> [ 11.346943] RCU nest depth: 0, expected: 0
> [ 11.351028] 4 locks held by udevd/165:
> [ 11.354766] #0: ffff4dc8825db0f8 (&dev->mutex){....}-{3:3}, at: __driver_attach+0x130/0x4a8
> [ 11.363207] #1: ffffc208f386c3c8 (thermal_list_lock){+.+.}-{3:3}, at: thermal_zone_device_register_with_trips+0x85c/0xcd8
> [ 11.374248] #2: ffff4dc7dc3586f0 (&tz->lock){+.+.}-{3:3}, at: thermal_zone_cdev_binding.part.0+0x98/0x280
> [ 11.383896] #3: ffffc208f39b7b78 (devtree_lock){....}-{2:2}, at: of_get_next_child+0x2c/0xc4
> [ 11.392418] irq event stamp: 173740
> [ 11.395895] hardirqs last enabled at (173739): [<ffffc208ecde804c>] _raw_spin_unlock_irqrestore+0x84/0x90
> [ 11.405537] hardirqs last disabled at (173740): [<ffffc208ecde6f7c>] _raw_spin_lock_irqsave+0xe0/0xf4
> [ 11.414742] softirqs last enabled at (172404): [<ffffc208e978bb20>] handle_softirqs+0x534/0x874
> [ 11.423517] softirqs last disabled at (172393): [<ffffc208e961097c>] __do_softirq+0x14/0x20
> [ 11.431857] CPU: 5 UID: 0 PID: 165 Comm: udevd Not tainted 6.11.0-rc4-next-20240822-00002-gfbbbf9faa56a #628
> [ 11.441670] Hardware name: Acer Tomato (rev2) board (DT)
> [ 11.446970] Call trace:
> [ 11.449407] dump_backtrace+0x98/0xf0
> [ 11.453059] show_stack+0x18/0x24
> [ 11.456364] dump_stack_lvl+0x90/0xd0
> [ 11.460018] dump_stack+0x1c/0x28
> [ 11.463322] __might_resched+0x358/0x570
> [ 11.467234] __might_sleep+0xa4/0x16c
> [ 11.470885] down_write+0x8c/0x21c
> [ 11.474277] kernfs_remove+0x64/0x98
> [ 11.477844] sysfs_remove_dir+0xa8/0xe8
> [ 11.481669] __kobject_del+0xb0/0x27c
> [ 11.485321] kobject_release+0xfc/0x134
> [ 11.489146] kobject_put+0xb0/0x130
> [ 11.492624] of_node_put+0x18/0x28
> [ 11.496016] of_get_next_child+0x64/0xc4
> [ 11.499929] thermal_of_should_bind+0x154/0x390
> [ 11.504449] thermal_zone_cdev_binding.part.0+0x174/0x280
> [ 11.509836] thermal_zone_device_register_with_trips+0x914/0xcd8
> [ 11.515831] thermal_of_zone_register+0x284/0x464
> [ 11.520523] devm_thermal_of_zone_register+0x80/0xf4
> [ 11.525476] lvts_domain_init+0x500/0x760 [lvts_thermal]
> [ 11.530785] lvts_probe+0x1b4/0x3ac [lvts_thermal]
> [ 11.535565] platform_probe+0xc4/0x214
> [ 11.539303] really_probe+0x188/0x5d0
> [ 11.542954] __driver_probe_device+0x160/0x2e8
> [ 11.547386] driver_probe_device+0x5c/0x298
> [ 11.551558] __driver_attach+0x13c/0x4a8
> [ 11.555470] bus_for_each_dev+0xf8/0x180
> [ 11.559383] driver_attach+0x3c/0x58
> [ 11.562947] bus_add_driver+0x1c4/0x458
> [ 11.566772] driver_register+0xf4/0x3c0
> [ 11.570598] __platform_driver_register+0x60/0x88
> [ 11.575291] lvts_driver_init+0x20/0x1000 [lvts_thermal]
> [ 11.580593] do_one_initcall+0xcc/0x284
> [ 11.584418] do_init_module+0x278/0x740
> [ 11.588244] load_module+0xed8/0x1434
> [ 11.591897] init_module_from_file+0xdc/0x1fc
> [ 11.596243] idempotent_init_module+0x2bc/0x604
> [ 11.600762] __arm64_sys_finit_module+0xac/0x100
> [ 11.605368] invoke_syscall+0x6c/0x258
> [ 11.609107] el0_svc_common.constprop.0+0xac/0x230
> [ 11.613886] do_el0_svc+0x40/0x58
> [ 11.617190] el0_svc+0x48/0xb8
> [ 11.620234] el0t_64_sync_handler+0x100/0x12c
> [ 11.624580] el0t_64_sync+0x190/0x194
> [ 11.628233]
> [ 11.629713] =============================
> [ 11.633708] [ BUG: Invalid wait context ]
> [ 11.637705] 6.11.0-rc4-next-20240822-00002-gfbbbf9faa56a #628 Tainted: G W
> [ 11.645953] -----------------------------
> [ 11.649950] udevd/165 is trying to lock:
> [ 11.653859] ffff4dc880881148 (&root->kernfs_rwsem){++++}-{3:3}, at: kernfs_remove+0x64/0x98
> [ 11.662200] other info that might help us debug this:
> [ 11.667238] context-{4:4}
> [ 11.669846] 4 locks held by udevd/165:
> [ 11.673582] #0: ffff4dc8825db0f8 (&dev->mutex){....}-{3:3}, at: __driver_attach+0x130/0x4a8
> [ 11.682009] #1: ffffc208f386c3c8 (thermal_list_lock){+.+.}-{3:3}, at: thermal_zone_device_register_with_trips+0x85c/0xcd8
> [ 11.693041] #2: ffff4dc7dc3586f0 (&tz->lock){+.+.}-{3:3}, at: thermal_zone_cdev_binding.part.0+0x98/0x280
> [ 11.702684] #3: ffffc208f39b7b78 (devtree_lock){....}-{2:2}, at: of_get_next_child+0x2c/0xc4
> [ 11.711199] stack backtrace:
> [ 11.714067] CPU: 5 UID: 0 PID: 165 Comm: udevd Tainted: G W 6.11.0-rc4-next-20240822-00002-gfbbbf9faa56a #628
> [ 11.725355] Tainted: [W]=WARN
> [ 11.728310] Hardware name: Acer Tomato (rev2) board (DT)
> [ 11.733608] Call trace:
> [ 11.736041] dump_backtrace+0x98/0xf0
> [ 11.739692] show_stack+0x18/0x24
> [ 11.742994] dump_stack_lvl+0x90/0xd0
> [ 11.746645] dump_stack+0x1c/0x28
> [ 11.749948] __lock_acquire+0x10f8/0x2710
> [ 11.753948] lock_acquire.part.0+0x218/0x518
> [ 11.758206] lock_acquire+0x90/0xb4
> [ 11.761683] down_write+0xb4/0x21c
> [ 11.765074] kernfs_remove+0x64/0x98
> [ 11.768637] sysfs_remove_dir+0xa8/0xe8
> [ 11.772461] __kobject_del+0xb0/0x27c
> [ 11.776111] kobject_release+0xfc/0x134
> [ 11.779935] kobject_put+0xb0/0x130
> [ 11.783413] of_node_put+0x18/0x28
> [ 11.786803] of_get_next_child+0x64/0xc4
> [ 11.790714] thermal_of_should_bind+0x154/0x390
> [ 11.795231] thermal_zone_cdev_binding.part.0+0x174/0x280
> [ 11.800617] thermal_zone_device_register_with_trips+0x914/0xcd8
> [ 11.806609] thermal_of_zone_register+0x284/0x464
> [ 11.811301] devm_thermal_of_zone_register+0x80/0xf4
> [ 11.816253] lvts_domain_init+0x500/0x760 [lvts_thermal]
> [ 11.821553] lvts_probe+0x1b4/0x3ac [lvts_thermal]
> [ 11.826332] platform_probe+0xc4/0x214
> [ 11.830069] really_probe+0x188/0x5d0
> [ 11.833719] __driver_probe_device+0x160/0x2e8
> [ 11.838150] driver_probe_device+0x5c/0x298
> [ 11.842320] __driver_attach+0x13c/0x4a8
> [ 11.846230] bus_for_each_dev+0xf8/0x180
> [ 11.850141] driver_attach+0x3c/0x58
> [ 11.853704] bus_add_driver+0x1c4/0x458
> [ 11.857529] driver_register+0xf4/0x3c0
> [ 11.861352] __platform_driver_register+0x60/0x88
> [ 11.866043] lvts_driver_init+0x20/0x1000 [lvts_thermal]
> [ 11.871342] do_one_initcall+0xcc/0x284
> [ 11.875166] do_init_module+0x278/0x740
> [ 11.878990] load_module+0xed8/0x1434
> [ 11.882641] init_module_from_file+0xdc/0x1fc
> [ 11.886986] idempotent_init_module+0x2bc/0x604
> [ 11.891504] __arm64_sys_finit_module+0xac/0x100
> [ 11.896109] invoke_syscall+0x6c/0x258
> [ 11.899846] el0_svc_common.constprop.0+0xac/0x230
> [ 11.904624] do_el0_svc+0x40/0x58
> [ 11.907927] el0_svc+0x48/0xb8
> [ 11.910969] el0t_64_sync_handler+0x100/0x12c
> [ 11.915314] el0t_64_sync+0x190/0x194
> [ 36.261761] watchdog: Watchdog detected hard LOCKUP on cpu 0
> [ 36.267414] Modules linked in: cbmem cros_ec_lid_angle cros_ec_sensors(+) cros_ec_sensors_core pcie_mediatek_gen3 sbs_battery cros_kbd_led_backlight industrialio_triggered_buffer kfifo_buf cros_ec_chardev cros_ec_rpmsg lvts_thermal(+) cros_ec_typec leds_cros_ec mtk_svs snd_sof_mt8195 mtk_adsp_common snd_sof_xtensa_dsp snd_sof_of mt6577_auxadc snd_soc_mt8195_afe snd_sof snd_sof_utils mtk_scp mtk_rpmsg mtk_scp_ipi pwm_bl mtk_wdt coreboot_table backlight mt8195_mt6359 ramoops reed_solomon
> [ 36.310414] irq event stamp: 197347
> [ 36.313890] hardirqs last enabled at (197347): [<ffffc208ecdc994c>] exit_to_kernel_mode+0x38/0x118
> [ 36.322923] hardirqs last disabled at (197346): [<ffffc208ecdcac44>] el1_interrupt+0x24/0x54
> [ 36.331347] softirqs last enabled at (197268): [<ffffc208e978bb20>] handle_softirqs+0x534/0x874
> [ 36.340117] softirqs last disabled at (197263): [<ffffc208e961097c>] __do_softirq+0x14/0x20
>
> Full log at http://0x0.st/XyID.txt
>
> Let me know if you need any more information.
>
> #regzbot introduced: next-20240821..next-20240822
> #regzbot title: Hang during boot in sysfs_remove_dir() called by thermal_of_zone_register()
>
> Thanks,
> Nícolas
>