Re: [PATCH 1/2] cpufreq: unlock correct rwsem while updating policy->cpu

["Jon Medhurst (Tixy)" <tixy@linaro.org>]

On Mon, 2013-09-16 at 20:40 +0530, Viresh Kumar wrote:
> Current code looks like this:
> 
>         WARN_ON(lock_policy_rwsem_write(cpu));
>         update_policy_cpu(policy, new_cpu);
>         unlock_policy_rwsem_write(cpu);
> 
> {lock|unlock}_policy_rwsem_write(cpu) takes/releases policy->cpu's rwsem.
> Because cpu is changing with the call to update_policy_cpu(), the
> unlock_policy_rwsem_write() will release the incorrect lock.
> 
> The right solution would be to take rwsem lock/unlock for both old and new cpu.

I thought possibly something like that, then wondered if threads could
take the locks in different orders at the same time, leading to
deadlock? So as I wasn't familiar with cpufreq I thought I'd leave it to
the experts to worry about :-)

This patch contains a bug, see inline comment below. Even with that
fixed it still doesn't work for me. I get a lockdep warning (below). I
have verified the cpu and locks are different locks, and it's not a
harmless false positive because I later get the message 'cpufreq:
__cpufreq_remove_dev_prepare: Failed to stop governor'.

=============================================
[ INFO: possible recursive locking detected ]
3.11.0+ #4 Not tainted
---------------------------------------------
swapper/0/1 is trying to acquire lock:
  (&per_cpu(cpu_policy_rwsem, cpu)){+.+...}, at: [<c0293c03>] update_policy_cpu+0x53/0xa4

but task is already holding lock:
  (&per_cpu(cpu_policy_rwsem, cpu)){+.+...}, at: [<c0293bef>] update_policy_cpu+0x3f/0xa4
other info that might help us debug this:
  Possible unsafe locking scenario:
        CPU0
        ----
   lock(&per_cpu(cpu_policy_rwsem, cpu));
   lock(&per_cpu(cpu_policy_rwsem, cpu));

*** DEADLOCK ***

  May be due to missing lock nesting notation
4 locks held by swapper/0/1:
 #0:  (bL_switcher_activation_lock){+.+.+.}, at: [<c0019b03>] bL_switcher_enable+0x17/0x2d8
 #1:  ((bL_activation_notifier).rwsem){.+.+.+}, at: [<c00370bd>] __blocking_notifier_call_chain+0x1d/0x40
 #2:  (subsys mutex#6){+.+.+.}, at: [<c023279d>] subsys_interface_unregister+0x1d/0x68
 #3:  (&per_cpu(cpu_policy_rwsem, cpu)){+.+...}, at: [<c0293bef>] update_policy_cpu+0x3f/0xa4

> This patch fixes this bug by taking both locks directly instead of calling
> lock_policy_rwsem_write().
> 
> Reported-by: Jon Medhurst<tixy@linaro.org>
> Signed-off-by: Viresh Kumar <viresh.kumar@linaro.org>
> ---
> Hi Rafael,
> 
> Probably we can get this patch in for 3.12? The second one can go in 3.13.
> 
> These are compile tested only at my end. Tixy reported these and probably can
> give his tested-by once he is done testing them.
> 
>  drivers/cpufreq/cpufreq.c | 9 +++++++--
>  1 file changed, 7 insertions(+), 2 deletions(-)
> 
If I take mainline code and just change the line above to:
   up_write(&per_cpu(cpu_policy_rwsem, (per_cpu(cpufreq_cpu_data,
cpu))->last_cpu));
then the big_little cpufreq driver works for me.

> diff --git a/drivers/cpufreq/cpufreq.c b/drivers/cpufreq/cpufreq.c
> index 43c24aa..c18bf7b 100644
> --- a/drivers/cpufreq/cpufreq.c
> +++ b/drivers/cpufreq/cpufreq.c
> @@ -952,9 +952,16 @@ static void update_policy_cpu(struct cpufreq_policy *policy, unsigned int cpu)
>  	if (cpu == policy->cpu)
>  		return;
>  
> +	/* take direct locks as lock_policy_rwsem_write wouldn't work here */
> +	down_write(&per_cpu(cpu_policy_rwsem, policy->cpu));
> +	down_write(&per_cpu(cpu_policy_rwsem, cpu));
> +
>  	policy->last_cpu = policy->cpu;
>  	policy->cpu = cpu;
>  
> +	up_write(&per_cpu(cpu_policy_rwsem, cpu));
> +	up_write(&per_cpu(cpu_policy_rwsem, policy->cpu));

You've just overwritten policy->cpu with cpu. I tried using
policy->last_cpu to fix that, but it still doesn't work for me (giving
the lockdep warning I mentioned.) If I change the code to just lock the
original policy->cpu lock only, then all is fine.

Also, this locking is now just happening around policy->cpu update,
whereas before this change, it was locked for the whole of
update_policy_cpu, i.e. cpufreq_frequency_table_update_policy_cpu and
the notifier callbacks. Is that change of lock coverage OK?

> +
>  #ifdef CONFIG_CPU_FREQ_TABLE
>  	cpufreq_frequency_table_update_policy_cpu(policy);
>  #endif
> @@ -1203,9 +1210,7 @@ static int __cpufreq_remove_dev_prepare(struct device *dev,
>  
>  		new_cpu = cpufreq_nominate_new_policy_cpu(policy, cpu, frozen);
>  		if (new_cpu >= 0) {
> -			WARN_ON(lock_policy_rwsem_write(cpu));
>  			update_policy_cpu(policy, new_cpu);
> -			unlock_policy_rwsem_write(cpu);
>  
>  			if (!frozen) {
>  				pr_debug("%s: policy Kobject moved to cpu: %d "

-- 
Tixy