OPP's mutex locking issue

OPP's mutex locking issue

[Chanwoo Choi]

Dear all,

The commit 052c6f19141dd ("PM / OPP: Move away from RCU locking")
used the mutex instead of RCU locking. After that, I get a deadlock issue
between Devfreq framework and Devfreq-Cooling device of thermal framework.


[Description]
Originally, Devfreq framework used the opp_notifier (dev_pm_opp_register_notifier)
in order to catch the OPP_EVENT_*. When Devfreq receives the notification,
it called the 'update_devfreq()' which updates the frequency of devfreq device
according to the available status of OPP. Because the dev_pm_opp_disable/enable()
could affect the minimum/maximum frequency of devfreq device.

The commit a76caf55e5b35 ("thermal: Add devfreq cooling") allows
the devfreq device to be used the cooling device. When the cooling down
are required, the devfreq_cooling.c calls dev_pm_opp_disable()
in order to disable the specific OPP.

In this case, the deadlock occurs.

In order to resolve this issue on the device driver side,
the OPP functions should not be used in the .notifier_call function.
Or, the .notifier_call function have to use the workqueue.


[Deadlock Sequence]
- base commit: v4.14-rc1
- I tried to change the 'cur_state' value of devfreq-cooling device directly.

root@localhost:~# cat /sys/class/thermal/cooling_device0/type 
thermal-devfreq-0
root@localhost:~# echo 2 > /sys/class/thermal/cooling_device0/cur_state
devfreq_cooling_set_cur_state() in drivers/thermal/devfreq_cooling.c
	partition_enable_opps()
		dev_pm_opp_disable();
	
dev_pm_opp_disable()
	mutex_lock(&opp_table->lock); : Firstly, lock mutex
	blocking_notifier_call_chain(&opp_table->head, OPP_EVNET_ENABLE/DISABLE, opp);
		--> devfreq_notifier_call() (.notifiear_call callback function of opp_notifier)				
			update_devfreq(devfreq); 
				devfreq->profile->target()
					devfreq_recommended_opp() (in drivers/devfreq/exynos-bus.c)
						dev_pm_opp_find_freq_floor/ceil()
							mutex_lock(&opp_table->lock); : Second, lock mutex : deadlock


[Kernel Log]
root@localhost:~# echo 2 > /sys/class/thermal/cooling_device0/cur_state                                                      
[   71.144017] INFO: task kworker/u16:1:108 blocked for more than 2 seconds.
[   71.144292]       Not tainted 4.14.0-rc1+ #48
[   71.144471] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[   71.144780] kworker/u16:1   D    0   108      2 0x00000000
[   71.146134] Workqueue: devfreq_wq devfreq_monitor
[   71.151437] Call trace:
[   71.153645] [<ffffff8008086360>] __switch_to+0xa0/0xb4
[   71.158671] [<ffffff8008aba76c>] __schedule+0x1f4/0x6c8
[   71.163958] [<ffffff8008abac74>] schedule+0x34/0x94
[   71.168782] [<ffffff8008abb158>] schedule_preempt_disabled+0x14/0x24
[   71.175024] [<ffffff8008abbe74>] __mutex_lock.isra.6+0x1a8/0x6c0
[   71.181041] [<ffffff8008abc39c>] __mutex_lock_slowpath+0x10/0x18
[   71.186782] [<ffffff8008abc3d4>] mutex_lock+0x30/0x38
[   71.191889] [<ffffff800875f77c>] devfreq_monitor+0x24/0x88
[   71.197309] [<ffffff80080b6680>] process_one_work+0x148/0x3f4
[   71.203070] [<ffffff80080b6984>] worker_thread+0x58/0x3c8
[   71.208416] [<ffffff80080bc484>] kthread+0x100/0x12c
[   71.213365] [<ffffff8008085380>] ret_from_fork+0x10/0x20
[   71.218715] INFO: task bash:795 blocked for more than 2 seconds.
[   71.224657]       Not tainted 4.14.0-rc1+ #48
[   71.228995] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[   71.236794] bash            D    0   795    340 0x00400200
[   71.242274] Call trace:
[   71.244697] [<ffffff8008086360>] __switch_to+0xa0/0xb4
[   71.249831] [<ffffff8008aba76c>] __schedule+0x1f4/0x6c8
[   71.255197] [<ffffff8008abac74>] schedule+0x34/0x94
[   71.259875] [<ffffff8008abb158>] schedule_preempt_disabled+0x14/0x24
[   71.266199] [<ffffff8008abbe74>] __mutex_lock.isra.6+0x1a8/0x6c0
[   71.272203] [<ffffff8008abc39c>] __mutex_lock_slowpath+0x10/0x18
[   71.278175] [<ffffff8008abc3d4>] mutex_lock+0x30/0x38
[   71.283225] [<ffffff8008547ed4>] _find_freq_ceil+0x24/0xb4
[   71.288697] [<ffffff80085489c4>] dev_pm_opp_find_freq_ceil+0x34/0x80
[   71.295030] [<ffffff800875f588>] devfreq_recommended_opp+0x34/0x5c
[   71.301179] [<ffffff8008760b4c>] exynos_bus_target+0x28/0x1f4
[   71.306919] [<ffffff800875f678>] update_devfreq+0xc8/0x1a8
[   71.312378] [<ffffff800875f804>] devfreq_notifier_call+0x24/0x40
[   71.318378] [<ffffff80080bdb70>] notifier_call_chain+0x4c/0x88
[   71.324196] [<ffffff80080bdf20>] __blocking_notifier_call_chain+0x4c/0x8c
[   71.331421] [<ffffff80080bdf74>] blocking_notifier_call_chain+0x14/0x1c
[   71.337573] [<ffffff80085490f0>] _opp_set_availability+0xcc/0x114
[   71.343642] [<ffffff8008549160>] dev_pm_opp_disable+0x10/0x18
[   71.349372] [<ffffff80086d9840>] devfreq_cooling_set_cur_state+0xe0/0x12c
[   71.356163] [<ffffff80086d6c1c>] thermal_cooling_device_cur_state_store+0x4c/0x74
[   71.363593] [<ffffff800852ecf4>] dev_attr_store+0x18/0x28
[   71.368993] [<ffffff800825d9c8>] sysfs_kf_write+0x40/0x50
[   71.374351] [<ffffff800825ccd4>] kernfs_fop_write+0xc0/0x1d0
[   71.380016] [<ffffff80081e9bec>] __vfs_write+0x28/0x124
[   71.385203] [<ffffff80081e9e9c>] vfs_write+0xa0/0x170
[   71.390517] [<ffffff80081ea0fc>] SyS_write+0x44/0xa0
[   71.395314] Exception stack(0xffffff801259bec0 to 0xffffff801259c000)
[   71.401639] bec0: 0000000000000001 00000000004d78f8 0000000000000002 0000000000000000
[   71.409434] bee0: 0000000000000002 00000000004d78f8 00000000f7b29d50 0000000000000004
[   71.417590] bf00: 0000000000000002 0000000000000002 0000000000000001 0000000000000000
[   71.425072] bf20: 0000000000000000 00000000ffc1d8dc 00000000f7a5a0cc 0000000000000000
[   71.433290] bf40: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
[   71.440672] bf60: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
[   71.448493] bf80: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
[   71.456324] bfa0: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
[   71.464115] bfc0: 00000000f7ab20a0 0000000060060010 0000000000000001 0000000000000004
[   71.471920] bfe0: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
[   71.479745] [<ffffff8008083808>] __sys_trace_return+0x0/0x4


[Register exynos-bus devfreq device as a cooling device]
diff --git a/drivers/devfreq/exynos-bus.c b/drivers/devfreq/exynos-bus.c
index 49f68929e024..e299f2ed6a83 100644
--- a/drivers/devfreq/exynos-bus.c
+++ b/drivers/devfreq/exynos-bus.c
@@ -15,6 +15,7 @@
 #include <linux/clk.h>
 #include <linux/devfreq.h>
 #include <linux/devfreq-event.h>
+#include <linux/devfreq_cooling.h>
 #include <linux/device.h>
 #include <linux/export.h>
 #include <linux/module.h>
@@ -41,6 +42,8 @@ struct exynos_bus {
        struct clk *clk;
        unsigned int voltage_tolerance;
        unsigned int ratio;
+
+       struct thermal_cooling_device *cdev;
 };

 /*
@@ -467,6 +470,14 @@ static int exynos_bus_probe(struct platform_device *pdev)
                goto err;
        }

+       /* Register devfreq cooling device */
+       bus->cdev = of_devfreq_cooling_register(np, bus->devfreq);
+       if (IS_ERR(bus->cdev) < 0) {
+               dev_err(dev, "failed to register cooling device\n");
+               ret = PTR_ERR(bus->cdev);
+               goto err;
+       }
+
        goto out;
 passive:
        /* Initialize the struct profile and governor data for passive device */

-- 
Best Regards,
Chanwoo Choi
Samsung Electronics

[PATCH] PM / OPP: Call notifier without holding opp_table->lock

[Viresh Kumar]

The notifier callbacks may want to call some OPP helper routines which
may try to take the same opp_table->lock again and cause a deadlock. One
such usecase was reported by Chanwoo Choi, where calling
dev_pm_opp_disable() leads us to the devfreq's OPP notifier handler,
which further calls dev_pm_opp_find_freq_floor() and it deadlocks.

We don't really need the opp_table->lock to be held across the notifier
call though, all we want to make sure is that the 'opp' doesn't get
freed while being used from within the notifier chain. We can do it with
help of dev_pm_opp_get/put() as well. Lets do it.

Reported-by: Chanwoo Choi <cw00.choi@samsung.com>
Signed-off-by: Viresh Kumar <viresh.kumar@linaro.org>
---
 drivers/base/power/opp/core.c | 7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/drivers/base/power/opp/core.c b/drivers/base/power/opp/core.c
index 4360b4efcd4c..668fd940d362 100644
--- a/drivers/base/power/opp/core.c
+++ b/drivers/base/power/opp/core.c
@@ -1627,6 +1627,9 @@ static int _opp_set_availability(struct device *dev, unsigned long freq,
 
 	opp->available = availability_req;
 
+	dev_pm_opp_get(opp);
+	mutex_unlock(&opp_table->lock);
+
 	/* Notify the change of the OPP availability */
 	if (availability_req)
 		blocking_notifier_call_chain(&opp_table->head, OPP_EVENT_ENABLE,
@@ -1635,8 +1638,12 @@ static int _opp_set_availability(struct device *dev, unsigned long freq,
 		blocking_notifier_call_chain(&opp_table->head,
 					     OPP_EVENT_DISABLE, opp);
 
+	dev_pm_opp_put(opp);
+	goto put_table;
+
 unlock:
 	mutex_unlock(&opp_table->lock);
+put_table:
 	dev_pm_opp_put_opp_table(opp_table);
 	return r;
 }
-- 
2.7.4

Re: [PATCH] PM / OPP: Call notifier without holding opp_table->lock

[Stephen Boyd]

On 09/20, Viresh Kumar wrote:
> The notifier callbacks may want to call some OPP helper routines which
> may try to take the same opp_table->lock again and cause a deadlock. One
> such usecase was reported by Chanwoo Choi, where calling
> dev_pm_opp_disable() leads us to the devfreq's OPP notifier handler,
> which further calls dev_pm_opp_find_freq_floor() and it deadlocks.
> 
> We don't really need the opp_table->lock to be held across the notifier
> call though, all we want to make sure is that the 'opp' doesn't get
> freed while being used from within the notifier chain. We can do it with
> help of dev_pm_opp_get/put() as well. Lets do it.

s/Lets/Let's/

> 
> Reported-by: Chanwoo Choi <cw00.choi@samsung.com>
> Signed-off-by: Viresh Kumar <viresh.kumar@linaro.org>
> ---
>  drivers/base/power/opp/core.c | 7 +++++++
>  1 file changed, 7 insertions(+)
> 
> diff --git a/drivers/base/power/opp/core.c b/drivers/base/power/opp/core.c
> index 4360b4efcd4c..668fd940d362 100644
> --- a/drivers/base/power/opp/core.c
> +++ b/drivers/base/power/opp/core.c
> @@ -1627,6 +1627,9 @@ static int _opp_set_availability(struct device *dev, unsigned long freq,
>  
>  	opp->available = availability_req;
>  
> +	dev_pm_opp_get(opp);
> +	mutex_unlock(&opp_table->lock);

Does this prevent the OPP from changing while the lock is
released? That would be the only difference from before. It's
possible that nobody cares about this situation though.

-- 
Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum,
a Linux Foundation Collaborative Project

Re: [PATCH] PM / OPP: Call notifier without holding opp_table->lock

[Viresh Kumar]

On 20-09-17, 10:00, Stephen Boyd wrote:
> On 09/20, Viresh Kumar wrote:

> > diff --git a/drivers/base/power/opp/core.c b/drivers/base/power/opp/core.c
> > index 4360b4efcd4c..668fd940d362 100644
> > --- a/drivers/base/power/opp/core.c
> > +++ b/drivers/base/power/opp/core.c
> > @@ -1627,6 +1627,9 @@ static int _opp_set_availability(struct device *dev, unsigned long freq,
> >  
> >  	opp->available = availability_req;
> >  
> > +	dev_pm_opp_get(opp);
> > +	mutex_unlock(&opp_table->lock);
> 
> Does this prevent the OPP from changing while the lock is
> released?

No, its just ref counting and will only prevent it from getting freed.

There is only one thing that can change for an OPP though after it is
created, its availability.

> That would be the only difference from before. It's
> possible that nobody cares about this situation though.

I am not sure if its worth caring for right now :)

Also the notifier chain will not start again until the previous call
chain is finished. So we are kind of synchronized here.

-- 
viresh

Re: [PATCH] PM / OPP: Call notifier without holding opp_table->lock

[Stephen Boyd]

On 09/20, Viresh Kumar wrote:
> On 20-09-17, 10:00, Stephen Boyd wrote:
> > On 09/20, Viresh Kumar wrote:
> 
> > > diff --git a/drivers/base/power/opp/core.c b/drivers/base/power/opp/core.c
> > > index 4360b4efcd4c..668fd940d362 100644
> > > --- a/drivers/base/power/opp/core.c
> > > +++ b/drivers/base/power/opp/core.c
> > > @@ -1627,6 +1627,9 @@ static int _opp_set_availability(struct device *dev, unsigned long freq,
> > >  
> > >  	opp->available = availability_req;
> > >  
> > > +	dev_pm_opp_get(opp);
> > > +	mutex_unlock(&opp_table->lock);
> > 
> > Does this prevent the OPP from changing while the lock is
> > released?
> 
> No, its just ref counting and will only prevent it from getting freed.
> 
> There is only one thing that can change for an OPP though after it is
> created, its availability.

Ok.

> 
> > That would be the only difference from before. It's
> > possible that nobody cares about this situation though.
> 
> I am not sure if its worth caring for right now :)
> 
> Also the notifier chain will not start again until the previous call
> chain is finished. So we are kind of synchronized here.
> 

Yep. Just me worrying out loud. You can add my

Reviewed-by: Stephen Boyd <sboyd@codeaurora.org>

-- 
Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum,
a Linux Foundation Collaborative Project

[PATCH V2] PM / OPP: Call notifier without holding opp_table->lock

[Viresh Kumar]

The notifier callbacks may want to call some OPP helper routines which
may try to take the same opp_table->lock again and cause a deadlock. One
such usecase was reported by Chanwoo Choi, where calling
dev_pm_opp_disable() leads us to the devfreq's OPP notifier handler,
which further calls dev_pm_opp_find_freq_floor() and it deadlocks.

We don't really need the opp_table->lock to be held across the notifier
call though, all we want to make sure is that the 'opp' doesn't get
freed while being used from within the notifier chain. We can do it with
help of dev_pm_opp_get/put() as well. Let's do it.

Reported-by: Chanwoo Choi <cw00.choi@samsung.com>
Reviewed-by: Stephen Boyd <sboyd@codeaurora.org>
Signed-off-by: Viresh Kumar <viresh.kumar@linaro.org>
---
V1->V2:
- s/Lets/Let's/ in commit log and added Stephen's tag.

 drivers/base/power/opp/core.c | 7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/drivers/base/power/opp/core.c b/drivers/base/power/opp/core.c
index 4360b4efcd4c..668fd940d362 100644
--- a/drivers/base/power/opp/core.c
+++ b/drivers/base/power/opp/core.c
@@ -1627,6 +1627,9 @@ static int _opp_set_availability(struct device *dev, unsigned long freq,
 
 	opp->available = availability_req;
 
+	dev_pm_opp_get(opp);
+	mutex_unlock(&opp_table->lock);
+
 	/* Notify the change of the OPP availability */
 	if (availability_req)
 		blocking_notifier_call_chain(&opp_table->head, OPP_EVENT_ENABLE,
@@ -1635,8 +1638,12 @@ static int _opp_set_availability(struct device *dev, unsigned long freq,
 		blocking_notifier_call_chain(&opp_table->head,
 					     OPP_EVENT_DISABLE, opp);
 
+	dev_pm_opp_put(opp);
+	goto put_table;
+
 unlock:
 	mutex_unlock(&opp_table->lock);
+put_table:
 	dev_pm_opp_put_opp_table(opp_table);
 	return r;
 }
-- 
2.7.4

Re: [PATCH V2] PM / OPP: Call notifier without holding opp_table->lock

[Chanwoo Choi]

Hi Viresh,

On 2017년 09월 21일 05:25, Viresh Kumar wrote:
> The notifier callbacks may want to call some OPP helper routines which
> may try to take the same opp_table->lock again and cause a deadlock. One
> such usecase was reported by Chanwoo Choi, where calling
> dev_pm_opp_disable() leads us to the devfreq's OPP notifier handler,
> which further calls dev_pm_opp_find_freq_floor() and it deadlocks.
> 
> We don't really need the opp_table->lock to be held across the notifier
> call though, all we want to make sure is that the 'opp' doesn't get
> freed while being used from within the notifier chain. We can do it with
> help of dev_pm_opp_get/put() as well. Let's do it.
> 
> Reported-by: Chanwoo Choi <cw00.choi@samsung.com>
> Reviewed-by: Stephen Boyd <sboyd@codeaurora.org>
> Signed-off-by: Viresh Kumar <viresh.kumar@linaro.org>
> ---
> V1->V2:
> - s/Lets/Let's/ in commit log and added Stephen's tag.
> 

Thanks for your fixup. Looks good to me.
IMHO, this patch should be posted to stable@vger.kernel.org.

Tested-by: Chanwoo Choi <cw00.choi@samsung.com>
Reviewed-by: Chanwoo Choi <cw00.choi@samsung.com>

[snip] 

-- 
Best Regards,
Chanwoo Choi
Samsung Electronics

[PATCH V3] PM / OPP: Call notifier without holding opp_table->lock

[Viresh Kumar]

The notifier callbacks may want to call some OPP helper routines which
may try to take the same opp_table->lock again and cause a deadlock. One
such usecase was reported by Chanwoo Choi, where calling
dev_pm_opp_disable() leads us to the devfreq's OPP notifier handler,
which further calls dev_pm_opp_find_freq_floor() and it deadlocks.

We don't really need the opp_table->lock to be held across the notifier
call though, all we want to make sure is that the 'opp' doesn't get
freed while being used from within the notifier chain. We can do it with
help of dev_pm_opp_get/put() as well. Let's do it.

Cc: 4.11+ <stable@vger.kernel.org> # 4.11+
Fixes: 5b650b388844 ("PM / OPP: Take kref from _find_opp_table()")
Reported-by: Chanwoo Choi <cw00.choi@samsung.com>
Reviewed-by: Stephen Boyd <sboyd@codeaurora.org>
Reviewed-by: Chanwoo Choi <cw00.choi@samsung.com>
Tested-by: Chanwoo Choi <cw00.choi@samsung.com>
Signed-off-by: Viresh Kumar <viresh.kumar@linaro.org>
---
V2->V3:
- Added fixes, Cc and tags from Chanwoo.

 drivers/base/power/opp/core.c | 7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/drivers/base/power/opp/core.c b/drivers/base/power/opp/core.c
index 4360b4efcd4c..668fd940d362 100644
--- a/drivers/base/power/opp/core.c
+++ b/drivers/base/power/opp/core.c
@@ -1627,6 +1627,9 @@ static int _opp_set_availability(struct device *dev, unsigned long freq,
 
 	opp->available = availability_req;
 
+	dev_pm_opp_get(opp);
+	mutex_unlock(&opp_table->lock);
+
 	/* Notify the change of the OPP availability */
 	if (availability_req)
 		blocking_notifier_call_chain(&opp_table->head, OPP_EVENT_ENABLE,
@@ -1635,8 +1638,12 @@ static int _opp_set_availability(struct device *dev, unsigned long freq,
 		blocking_notifier_call_chain(&opp_table->head,
 					     OPP_EVENT_DISABLE, opp);
 
+	dev_pm_opp_put(opp);
+	goto put_table;
+
 unlock:
 	mutex_unlock(&opp_table->lock);
+put_table:
 	dev_pm_opp_put_opp_table(opp_table);
 	return r;
 }
-- 
2.7.4