[PATCH v1] thermal: core: Address thermal zone removal races with resume

[PATCH v1] thermal: core: Address thermal zone removal races with resume

[Rafael J. Wysocki]

From: "Rafael J. Wysocki" <rafael.j.wysocki@intel.com>

Since thermal_zone_pm_complete() and thermal_zone_device_resume()
reinitialize the poll_queue delayed work for the given thermal zone,
the cancel_delayed_work_sync() in thermal_zone_device_unregister()
may miss some already running work items and the thermal zone may
be freed prematurely [1].

There are two failing scenarios that both start with
running thermal_pm_notify_complete() right before invoking
thermal_zone_device_unregister() for one of the thermal zones.

In the first scenario, there is a work item already running for
the given thermal zone when thermal_pm_notify_complete() calls
thermal_zone_pm_complete() for that thermal zone and it continues to
run when thermal_zone_device_unregister() starts.  Since the poll_queue
delayed work has been reinitialized by thermal_pm_notify_complete(), the
running work item will be missed by the cancel_delayed_work_sync() in
thermal_zone_device_unregister() and if it continues to run past the
freeing of the thermal zone object, a use-after-free will occur.

In the second scenario, thermal_zone_device_resume() queued up by
thermal_pm_notify_complete() runs right after the thermal_zone_exit()
called by thermal_zone_device_unregister() has returned.  The poll_queue
delayed work is reinitialized by it before cancel_delayed_work_sync() is
called by thermal_zone_device_unregister(), so it may continue to run
after the freeing of the thermal zone object, which also leads to a
use-after-free.

Address the first failing scenario by ensuring that no thermal work
items will be running when thermal_pm_notify_complete() is called.
For this purpose, first move the cancel_delayed_work() call from
thermal_zone_pm_complete() to thermal_zone_pm_prepare() to prevent
new work from entering the workqueue going forward.  Next, switch
over to using a dedicated workqueue for thermal events and update
the code in thermal_pm_notify() to flush that workqueue after
thermal_pm_notify_prepare() has returned which will take care of
all leftover thermal work already on the workqueue (that leftover
work would do nothing useful anyway because all of the thermal zones
have been flagged as suspended).

The second failing scenario is addressed by adding a tz->state check
to thermal_zone_device_resume() to prevent it from reinitializing
the poll_queue delayed work if the thermal zone is going away.

Note that the above changes will also facilitate relocating the suspend
and resume of thermal zones closer to the suspend and resume of devices,
respectively.

Fixes: 5a5efdaffda5 ("thermal: core: Resume thermal zones asynchronously")
Reported-by: Mauricio Faria de Oliveira <mfo@igalia.com>
Closes: https://lore.kernel.org/linux-pm/20260324-thermal-core-uaf-init_delayed_work-v1-1-6611ae76a8a1@igalia.com/ [1]
Signed-off-by: Rafael J. Wysocki <rafael.j.wysocki@intel.com>
---
 drivers/thermal/thermal_core.c | 29 ++++++++++++++++++++++++-----
 1 file changed, 24 insertions(+), 5 deletions(-)

diff --git a/drivers/thermal/thermal_core.c b/drivers/thermal/thermal_core.c
index b7d706ed7ed9..248bd0a82a08 100644
--- a/drivers/thermal/thermal_core.c
+++ b/drivers/thermal/thermal_core.c
@@ -41,6 +41,8 @@ static struct thermal_governor *def_governor;
 
 static bool thermal_pm_suspended;
 
+static struct workqueue_struct *thermal_wq __ro_after_init;
+
 /*
  * Governor section: set of functions to handle thermal governors
  *
@@ -313,7 +315,7 @@ static void thermal_zone_device_set_polling(struct thermal_zone_device *tz,
 	if (delay > HZ)
 		delay = round_jiffies_relative(delay);
 
-	mod_delayed_work(system_freezable_power_efficient_wq, &tz->poll_queue, delay);
+	mod_delayed_work(thermal_wq, &tz->poll_queue, delay);
 }
 
 static void thermal_zone_recheck(struct thermal_zone_device *tz, int error)
@@ -1785,6 +1787,10 @@ static void thermal_zone_device_resume(struct work_struct *work)
 
 	guard(thermal_zone)(tz);
 
+	/* If the thermal zone is going away, there's nothing to do. */
+	if (tz->state & TZ_STATE_FLAG_EXIT)
+		return;
+
 	tz->state &= ~(TZ_STATE_FLAG_SUSPENDED | TZ_STATE_FLAG_RESUMING);
 
 	thermal_debug_tz_resume(tz);
@@ -1811,6 +1817,9 @@ static void thermal_zone_pm_prepare(struct thermal_zone_device *tz)
 	}
 
 	tz->state |= TZ_STATE_FLAG_SUSPENDED;
+
+	/* Prevent new work from getting to the workqueue subsequently. */
+	cancel_delayed_work(&tz->poll_queue);
 }
 
 static void thermal_pm_notify_prepare(void)
@@ -1829,8 +1838,6 @@ static void thermal_zone_pm_complete(struct thermal_zone_device *tz)
 {
 	guard(thermal_zone)(tz);
 
-	cancel_delayed_work(&tz->poll_queue);
-
 	reinit_completion(&tz->resume);
 	tz->state |= TZ_STATE_FLAG_RESUMING;
 
@@ -1840,7 +1847,7 @@ static void thermal_zone_pm_complete(struct thermal_zone_device *tz)
 	 */
 	INIT_DELAYED_WORK(&tz->poll_queue, thermal_zone_device_resume);
 	/* Queue up the work without a delay. */
-	mod_delayed_work(system_freezable_power_efficient_wq, &tz->poll_queue, 0);
+	mod_delayed_work(thermal_wq, &tz->poll_queue, 0);
 }
 
 static void thermal_pm_notify_complete(void)
@@ -1863,6 +1870,11 @@ static int thermal_pm_notify(struct notifier_block *nb,
 	case PM_RESTORE_PREPARE:
 	case PM_SUSPEND_PREPARE:
 		thermal_pm_notify_prepare();
+		/*
+		 * Allow any leftover thermal work items already on the
+		 * worqueue to complete so they don't get in the way later.
+		 */
+		flush_workqueue(thermal_wq);
 		break;
 	case PM_POST_HIBERNATION:
 	case PM_POST_RESTORE:
@@ -1895,9 +1907,14 @@ static int __init thermal_init(void)
 	if (result)
 		goto error;
 
+	thermal_wq = alloc_workqueue("thermal_events",
+				      WQ_FREEZABLE | WQ_POWER_EFFICIENT | WQ_PERCPU, 0);
+	if (!thermal_wq)
+		goto unregister_netlink;
+
 	result = thermal_register_governors();
 	if (result)
-		goto unregister_netlink;
+		goto destroy_workqueue;
 
 	thermal_class = kzalloc_obj(*thermal_class);
 	if (!thermal_class) {
@@ -1924,6 +1941,8 @@ static int __init thermal_init(void)
 
 unregister_governors:
 	thermal_unregister_governors();
+destroy_workqueue:
+	destroy_workqueue(thermal_wq);
 unregister_netlink:
 	thermal_netlink_exit();
 error:
-- 
2.51.0

Re: [PATCH v1] thermal: core: Address thermal zone removal races with resume

[Mauricio Faria de Oliveira]

On 2026-03-26 08:45, Rafael J. Wysocki wrote:
> Address the first failing scenario by ensuring that no thermal work
> items will be running when thermal_pm_notify_complete() is called.
> For this purpose, first move the cancel_delayed_work() call from
> thermal_zone_pm_complete() to thermal_zone_pm_prepare() to prevent
> new work from entering the workqueue going forward.  Next, switch
> over to using a dedicated workqueue for thermal events and update
> the code in thermal_pm_notify() to flush that workqueue after
> thermal_pm_notify_prepare() has returned which will take care of
> all leftover thermal work already on the workqueue (that leftover
> work would do nothing useful anyway because all of the thermal zones
> have been flagged as suspended).

Thanks for coming up with this alternative. I spent some time earlier
today thinking of corner cases in that it might fail, and it held OK.

However, slightly unrelated: apparently, flushing the workqueue in
thermal_pm_notify() reintroduces the issue addressed by the Fixes:
commit, but moving it from PM_POST_* to PM_*_PREPARE?

IIIUC, that issue is __thermal_zone_device_update() might take long
thus block other thermal zones and other PM notifiers after thermal.

Apparently, at least the latter also applies to PM_*_PREPARE?

Say, a currently running work item (i.e., that cancel_delayed_work() 
cannot cancel) wins the race for tz->lock and doesn't see tz->state
TZ_STATE_FLAG_SUSPENDED set, so it runs, and say it might take long.

Now, the workqueue flush blocks on it, also taking long, which thus
blocks other PM notifiers.
 
> The second failing scenario is addressed by adding a tz->state check
> to thermal_zone_device_resume() to prevent it from reinitializing
> the poll_queue delayed work if the thermal zone is going away.

This also held OK in the thinking of corner cases.

Thanks,

-- 
Mauricio

Re: [PATCH v1] thermal: core: Address thermal zone removal races with resume

[Rafael J. Wysocki]

On Thu, Mar 26, 2026 at 7:35 PM Mauricio Faria de Oliveira
<mfo@igalia.com> wrote:
>
> On 2026-03-26 08:45, Rafael J. Wysocki wrote:
> > Address the first failing scenario by ensuring that no thermal work
> > items will be running when thermal_pm_notify_complete() is called.
> > For this purpose, first move the cancel_delayed_work() call from
> > thermal_zone_pm_complete() to thermal_zone_pm_prepare() to prevent
> > new work from entering the workqueue going forward.  Next, switch
> > over to using a dedicated workqueue for thermal events and update
> > the code in thermal_pm_notify() to flush that workqueue after
> > thermal_pm_notify_prepare() has returned which will take care of
> > all leftover thermal work already on the workqueue (that leftover
> > work would do nothing useful anyway because all of the thermal zones
> > have been flagged as suspended).
>
> Thanks for coming up with this alternative. I spent some time earlier
> today thinking of corner cases in that it might fail, and it held OK.
>
> However, slightly unrelated: apparently, flushing the workqueue in
> thermal_pm_notify() reintroduces the issue addressed by the Fixes:
> commit, but moving it from PM_POST_* to PM_*_PREPARE?

Note that the work in question will be thermal_zone_device_check(),
which simply calls thermal_zone_device_update() that essentially
invokes __thermal_zone_device_update() under tz->lock.

Thus thermal_zone_device_update() can only run as a whole before or
after thermal_zone_pm_complete() for the given zone because the latter
also acquires tz->lock and releases it at the end.  If it runs before
the latter, it will be waited for because the latter will block on the
lock, but that happens without the changes in the $subject patch.  If
it runs after the latter, __thermal_zone_device_update() will see that
tz->state is not TZ_STATE_READY (because TZ_STATE_FLAG_SUSPENDED is
set) and it will bail out immediately.

So I don't see the problem here.

PM_POST_* is different because thermal_zone_device_resume() calls
__thermal_zone_device_update() when tz->state is TZ_STATE_READY and
that may take time.

> IIIUC, that issue is __thermal_zone_device_update() might take long
> thus block other thermal zones and other PM notifiers after thermal.
>
> Apparently, at least the latter also applies to PM_*_PREPARE?

Not at the point when the flush_workqueue() is called.

> Say, a currently running work item (i.e., that cancel_delayed_work()
> cannot cancel) wins the race for tz->lock and doesn't see tz->state
> TZ_STATE_FLAG_SUSPENDED set, so it runs, and say it might take long.
>
> Now, the workqueue flush blocks on it, also taking long, which thus
> blocks other PM notifiers.
>
> > The second failing scenario is addressed by adding a tz->state check
> > to thermal_zone_device_resume() to prevent it from reinitializing
> > the poll_queue delayed work if the thermal zone is going away.
>
> This also held OK in the thinking of corner cases.

Thanks for the feedback!

Re: [PATCH v1] thermal: core: Address thermal zone removal races with resume

[Mauricio Faria de Oliveira]

On 2026-03-26 16:03, Rafael J. Wysocki wrote:
> On Thu, Mar 26, 2026 at 7:35 PM Mauricio Faria de Oliveira
> <mfo@igalia.com> wrote:
>>
>> On 2026-03-26 08:45, Rafael J. Wysocki wrote:
>> > Address the first failing scenario by ensuring that no thermal work
>> > items will be running when thermal_pm_notify_complete() is called.
>> > For this purpose, first move the cancel_delayed_work() call from
>> > thermal_zone_pm_complete() to thermal_zone_pm_prepare() to prevent
>> > new work from entering the workqueue going forward.  Next, switch
>> > over to using a dedicated workqueue for thermal events and update
>> > the code in thermal_pm_notify() to flush that workqueue after
>> > thermal_pm_notify_prepare() has returned which will take care of
>> > all leftover thermal work already on the workqueue (that leftover
>> > work would do nothing useful anyway because all of the thermal zones
>> > have been flagged as suspended).
>>
>> Thanks for coming up with this alternative. I spent some time earlier
>> today thinking of corner cases in that it might fail, and it held OK.
>>
>> However, slightly unrelated: apparently, flushing the workqueue in
>> thermal_pm_notify() reintroduces the issue addressed by the Fixes:
>> commit, but moving it from PM_POST_* to PM_*_PREPARE?
> 
> Note that the work in question will be thermal_zone_device_check(),
> which simply calls thermal_zone_device_update() that essentially
> invokes __thermal_zone_device_update() under tz->lock.
> 
> Thus thermal_zone_device_update() can only run as a whole before or
> after thermal_zone_pm_complete() for the given zone because the latter

      ^ thermal_zone_pm_prepare(), you mean? 
        (in PM_*_PREPARE path; sets TZ_STATE_FLAG_SUSPENDED mentioned
below.)

> also acquires tz->lock and releases it at the end.  If it runs before
> the latter, it will be waited for because the latter will block on the
> lock, but that happens without the changes in the $subject patch.  If

Ok, right; that already happens.

> it runs after the latter, __thermal_zone_device_update() will see that
> tz->state is not TZ_STATE_READY (because TZ_STATE_FLAG_SUSPENDED is
> set) and it will bail out immediately.
> 
> So I don't see the problem here.

Well, __thermal_zone_device_update() taking long might indeed impact
the PM_*_PREPARE path too (the former case, "it will be waited for"),
however, as you said, it happens without this patch, and it is not
fixed with the patch I proposed either. :)

> PM_POST_* is different because thermal_zone_device_resume() calls
> __thermal_zone_device_update() when tz->state is TZ_STATE_READY and
> that may take time.
> 
>> IIIUC, that issue is __thermal_zone_device_update() might take long
>> thus block other thermal zones and other PM notifiers after thermal.
>>
>> Apparently, at least the latter also applies to PM_*_PREPARE?
> 
> Not at the point when the flush_workqueue() is called.

Thanks for clarifying.

>> Say, a currently running work item (i.e., that cancel_delayed_work()
>> cannot cancel) wins the race for tz->lock and doesn't see tz->state
>> TZ_STATE_FLAG_SUSPENDED set, so it runs, and say it might take long.
>>
>> Now, the workqueue flush blocks on it, also taking long, which thus
>> blocks other PM notifiers.
>>
>> > The second failing scenario is addressed by adding a tz->state check
>> > to thermal_zone_device_resume() to prevent it from reinitializing
>> > the poll_queue delayed work if the thermal zone is going away.
>>
>> This also held OK in the thinking of corner cases.
> 
> Thanks for the feedback!

Glad to help!

-- 
Mauricio

Re: [PATCH v1] thermal: core: Address thermal zone removal races with resume

[Dan Carpenter]

Hi Rafael,

kernel test robot noticed the following build warnings:

https://git-scm.com/docs/git-format-patch#_base_tree_information]

url:    https://github.com/intel-lab-lkp/linux/commits/Rafael-J-Wysocki/thermal-core-Address-thermal-zone-removal-races-with-resume/20260327-222101
base:   https://git.kernel.org/pub/scm/linux/kernel/git/rafael/linux-pm.git thermal
patch link:    https://lore.kernel.org/r/12876512.O9o76ZdvQC%40rafael.j.wysocki
patch subject: [PATCH v1] thermal: core: Address thermal zone removal races with resume
config: i386-randconfig-141-20260328 (https://download.01.org/0day-ci/archive/20260328/202603280307.P9324SuS-lkp@intel.com/config)
compiler: clang version 20.1.8 (https://github.com/llvm/llvm-project 87f0227cb60147a26a1eeb4fb06e3b505e9c7261)
smatch: v0.5.0-9004-gb810ac53

If you fix the issue in a separate patch/commit (i.e. not just a new version of
the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <lkp@intel.com>
| Reported-by: Dan Carpenter <error27@gmail.com>
| Closes: https://lore.kernel.org/r/202603280307.P9324SuS-lkp@intel.com/

smatch warnings:
drivers/thermal/thermal_core.c:1913 thermal_init() warn: missing error code 'result'

vim +/result +1913 drivers/thermal/thermal_core.c

203d3d4aa48233 drivers/thermal/thermal.c      Zhang Rui         2008-01-17  1900  static int __init thermal_init(void)
203d3d4aa48233 drivers/thermal/thermal.c      Zhang Rui         2008-01-17  1901  {
80a26a5c22b90a drivers/thermal/thermal_core.c Zhang Rui         2013-03-26  1902  	int result;
80a26a5c22b90a drivers/thermal/thermal_core.c Zhang Rui         2013-03-26  1903  
755113d7678681 drivers/thermal/thermal_core.c Daniel Lezcano    2024-01-09  1904  	thermal_debug_init();
755113d7678681 drivers/thermal/thermal_core.c Daniel Lezcano    2024-01-09  1905  
d2a89b52839597 drivers/thermal/thermal_core.c Daniel Lezcano    2020-07-17  1906  	result = thermal_netlink_init();
d2a89b52839597 drivers/thermal/thermal_core.c Daniel Lezcano    2020-07-17  1907  	if (result)
d2a89b52839597 drivers/thermal/thermal_core.c Daniel Lezcano    2020-07-17  1908  		goto error;
d2a89b52839597 drivers/thermal/thermal_core.c Daniel Lezcano    2020-07-17  1909  
f777132b791874 drivers/thermal/thermal_core.c Rafael J. Wysocki 2026-03-26  1910  	thermal_wq = alloc_workqueue("thermal_events",
f777132b791874 drivers/thermal/thermal_core.c Rafael J. Wysocki 2026-03-26  1911  				      WQ_FREEZABLE | WQ_POWER_EFFICIENT | WQ_PERCPU, 0);
f777132b791874 drivers/thermal/thermal_core.c Rafael J. Wysocki 2026-03-26  1912  	if (!thermal_wq)
f777132b791874 drivers/thermal/thermal_core.c Rafael J. Wysocki 2026-03-26 @1913  		goto unregister_netlink;

result = -ENOMEM;


-- 
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki

Re: [PATCH v1] thermal: core: Address thermal zone removal races with resume

[Rafael J. Wysocki]

Hi,

On Sat, Mar 28, 2026 at 9:03 AM Dan Carpenter <error27@gmail.com> wrote:
>
> Hi Rafael,
>
> kernel test robot noticed the following build warnings:
>
> https://git-scm.com/docs/git-format-patch#_base_tree_information]
>
> url:    https://github.com/intel-lab-lkp/linux/commits/Rafael-J-Wysocki/thermal-core-Address-thermal-zone-removal-races-with-resume/20260327-222101
> base:   https://git.kernel.org/pub/scm/linux/kernel/git/rafael/linux-pm.git thermal
> patch link:    https://lore.kernel.org/r/12876512.O9o76ZdvQC%40rafael.j.wysocki
> patch subject: [PATCH v1] thermal: core: Address thermal zone removal races with resume
> config: i386-randconfig-141-20260328 (https://download.01.org/0day-ci/archive/20260328/202603280307.P9324SuS-lkp@intel.com/config)
> compiler: clang version 20.1.8 (https://github.com/llvm/llvm-project 87f0227cb60147a26a1eeb4fb06e3b505e9c7261)
> smatch: v0.5.0-9004-gb810ac53
>
> If you fix the issue in a separate patch/commit (i.e. not just a new version of
> the same patch/commit), kindly add following tags
> | Reported-by: kernel test robot <lkp@intel.com>
> | Reported-by: Dan Carpenter <error27@gmail.com>
> | Closes: https://lore.kernel.org/r/202603280307.P9324SuS-lkp@intel.com/
>
> smatch warnings:
> drivers/thermal/thermal_core.c:1913 thermal_init() warn: missing error code 'result'
>
> vim +/result +1913 drivers/thermal/thermal_core.c
>
> 203d3d4aa48233 drivers/thermal/thermal.c      Zhang Rui         2008-01-17  1900  static int __init thermal_init(void)
> 203d3d4aa48233 drivers/thermal/thermal.c      Zhang Rui         2008-01-17  1901  {
> 80a26a5c22b90a drivers/thermal/thermal_core.c Zhang Rui         2013-03-26  1902        int result;
> 80a26a5c22b90a drivers/thermal/thermal_core.c Zhang Rui         2013-03-26  1903
> 755113d7678681 drivers/thermal/thermal_core.c Daniel Lezcano    2024-01-09  1904        thermal_debug_init();
> 755113d7678681 drivers/thermal/thermal_core.c Daniel Lezcano    2024-01-09  1905
> d2a89b52839597 drivers/thermal/thermal_core.c Daniel Lezcano    2020-07-17  1906        result = thermal_netlink_init();
> d2a89b52839597 drivers/thermal/thermal_core.c Daniel Lezcano    2020-07-17  1907        if (result)
> d2a89b52839597 drivers/thermal/thermal_core.c Daniel Lezcano    2020-07-17  1908                goto error;
> d2a89b52839597 drivers/thermal/thermal_core.c Daniel Lezcano    2020-07-17  1909
> f777132b791874 drivers/thermal/thermal_core.c Rafael J. Wysocki 2026-03-26  1910        thermal_wq = alloc_workqueue("thermal_events",
> f777132b791874 drivers/thermal/thermal_core.c Rafael J. Wysocki 2026-03-26  1911                                      WQ_FREEZABLE | WQ_POWER_EFFICIENT | WQ_PERCPU, 0);
> f777132b791874 drivers/thermal/thermal_core.c Rafael J. Wysocki 2026-03-26  1912        if (!thermal_wq)
> f777132b791874 drivers/thermal/thermal_core.c Rafael J. Wysocki 2026-03-26 @1913                goto unregister_netlink;
>
> result = -ENOMEM;
>
>
> --

This has been fixed in the v2:
https://lore.kernel.org/linux-pm/6267615.lOV4Wx5bFT@rafael.j.wysocki/