[BUG -tip/sched] INFO: suspicious RCU usage

[BUG -tip/sched] INFO: suspicious RCU usage

[Yong Zhang]

===============================
[ INFO: suspicious RCU usage. ]
-------------------------------
/build/linux/kernel/sched/fair.c:5091 suspicious rcu_dereference_check() usage!

other info that might help us debug this:


rcu_scheduler_active = 1, debug_locks = 1
1 lock held by swapper/0/1:
 #0:  (&sb->s_type->i_mutex_key#2){+.+.+.}, at: [<ffffffff812a0e0c>] debugfs_create_file+0x8c/0x270

stack backtrace:
Pid: 1, comm: swapper/0 Not tainted 3.2.0-rc1-00339-gcd490c5 #14
Call Trace:
 <IRQ>  [<ffffffff815f9968>] ? _raw_spin_unlock_irqrestore+0x38/0x80
 [<ffffffff8109407f>] lockdep_rcu_suspicious+0xbf/0xe0
 [<ffffffff8107dd48>] trigger_load_balance+0x5b8/0x5c0
 [<ffffffff81070544>] scheduler_tick+0x104/0x160
 [<ffffffff8104a1ce>] update_process_times+0x6e/0x90
 [<ffffffff8108e32c>] tick_periodic+0x2c/0x70
 [<ffffffff8108e394>] tick_handle_periodic+0x24/0x80
 [<ffffffff812a07a7>] ? T.562+0xf7/0x170
 [<ffffffff815fce56>] smp_apic_timer_interrupt+0x66/0x98
 [<ffffffff815fb2f0>] apic_timer_interrupt+0x70/0x80
 <EOI>  [<ffffffff810984a4>] ? lock_release+0xe4/0x2b0
 [<ffffffff815f98f3>] _raw_spin_unlock+0x23/0x60
 [<ffffffff812a07a7>] T.562+0xf7/0x170
 [<ffffffff812a0e71>] debugfs_create_file+0xf1/0x270
 [<ffffffff810e96c9>] trace_create_file+0x19/0x50
 [<ffffffff81af7b3c>] tracer_init_debugfs+0x2bd/0x3c0
 [<ffffffff810e96c9>] ? trace_create_file+0x19/0x50
 [<ffffffff81af787f>] ? clear_boot_tracer+0x2d/0x2d
 [<ffffffff810001d3>] do_one_initcall+0x43/0x180
 [<ffffffff81adf637>] kernel_init+0xd0/0x156
 [<ffffffff815fc574>] kernel_thread_helper+0x4/0x10
 [<ffffffff8106fdb8>] ? finish_task_switch+0x88/0xf0
 [<ffffffff815f99eb>] ? _raw_spin_unlock_irq+0x3b/0x70
 [<ffffffff815f9f61>] ? retint_restore_args+0xe/0xe
 [<ffffffff81adf567>] ? parse_early_options+0x20/0x20
 [<ffffffff815fc570>] ? gs_change+0xb/0xb

Seems due to commit [0b005cf5: sched, nohz: Implement sched group, domain
aware nohz idle load balancing]

Thanks,
Yong

Re: [BUG -tip/sched] INFO: suspicious RCU usage

[Peter Zijlstra]

On Wed, 2011-12-07 at 15:39 +0800, Yong Zhang wrote:
> ===============================
> [ INFO: suspicious RCU usage. ]
> -------------------------------
> /build/linux/kernel/sched/fair.c:5091 suspicious rcu_dereference_check() usage!
> 
> other info that might help us debug this:
> 
> 
> rcu_scheduler_active = 1, debug_locks = 1
> 1 lock held by swapper/0/1:
>  #0:  (&sb->s_type->i_mutex_key#2){+.+.+.}, at: [<ffffffff812a0e0c>] debugfs_create_file+0x8c/0x270
> 
> stack backtrace:
> Pid: 1, comm: swapper/0 Not tainted 3.2.0-rc1-00339-gcd490c5 #14
> Call Trace:
>  <IRQ>  [<ffffffff815f9968>] ? _raw_spin_unlock_irqrestore+0x38/0x80
>  [<ffffffff8109407f>] lockdep_rcu_suspicious+0xbf/0xe0
>  [<ffffffff8107dd48>] trigger_load_balance+0x5b8/0x5c0
>  [<ffffffff81070544>] scheduler_tick+0x104/0x160
>  [<ffffffff8104a1ce>] update_process_times+0x6e/0x90
>  [<ffffffff8108e32c>] tick_periodic+0x2c/0x70
>  [<ffffffff8108e394>] tick_handle_periodic+0x24/0x80
>  [<ffffffff812a07a7>] ? T.562+0xf7/0x170
>  [<ffffffff815fce56>] smp_apic_timer_interrupt+0x66/0x98
>  [<ffffffff815fb2f0>] apic_timer_interrupt+0x70/0x80
>  <EOI>  [<ffffffff810984a4>] ? lock_release+0xe4/0x2b0
>  [<ffffffff815f98f3>] _raw_spin_unlock+0x23/0x60
>  [<ffffffff812a07a7>] T.562+0xf7/0x170
>  [<ffffffff812a0e71>] debugfs_create_file+0xf1/0x270
>  [<ffffffff810e96c9>] trace_create_file+0x19/0x50
>  [<ffffffff81af7b3c>] tracer_init_debugfs+0x2bd/0x3c0
>  [<ffffffff810e96c9>] ? trace_create_file+0x19/0x50
>  [<ffffffff81af787f>] ? clear_boot_tracer+0x2d/0x2d
>  [<ffffffff810001d3>] do_one_initcall+0x43/0x180
>  [<ffffffff81adf637>] kernel_init+0xd0/0x156
>  [<ffffffff815fc574>] kernel_thread_helper+0x4/0x10
>  [<ffffffff8106fdb8>] ? finish_task_switch+0x88/0xf0
>  [<ffffffff815f99eb>] ? _raw_spin_unlock_irq+0x3b/0x70
>  [<ffffffff815f9f61>] ? retint_restore_args+0xe/0xe
>  [<ffffffff81adf567>] ? parse_early_options+0x20/0x20
>  [<ffffffff815fc570>] ? gs_change+0xb/0xb
> 
> Seems due to commit [0b005cf5: sched, nohz: Implement sched group, domain
> aware nohz idle load balancing]

I think something like the below should suffice, the sd argument only
lives inside the for_each_domain() so wrapping the whole block should
suffice.

---
 kernel/sched/fair.c |    9 +++++++--
 1 files changed, 7 insertions(+), 2 deletions(-)

diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index 6482136..a4d2b7a 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -5088,23 +5088,28 @@ static inline int nohz_kick_needed(struct rq *rq, int cpu)
 	if (rq->nr_running >= 2)
 		goto need_kick;
 
+	rcu_read_lock();
 	for_each_domain(cpu, sd) {
 		struct sched_group *sg = sd->groups;
 		struct sched_group_power *sgp = sg->sgp;
 		int nr_busy = atomic_read(&sgp->nr_busy_cpus);
 
 		if (sd->flags & SD_SHARE_PKG_RESOURCES && nr_busy > 1)
-			goto need_kick;
+			goto need_kick_unlock;
 
 		if (sd->flags & SD_ASYM_PACKING && nr_busy != sg->group_weight
 		    && (cpumask_first_and(nohz.idle_cpus_mask,
 					  sched_domain_span(sd)) < cpu))
-			goto need_kick;
+			goto need_kick_unlock;
 
 		if (!(sd->flags & (SD_SHARE_PKG_RESOURCES | SD_ASYM_PACKING)))
 			break;
 	}
+	rcu_read_unlock();
 	return 0;
+
+need_kick_unlock:
+	rcu_read_unlock();
 need_kick:
 	return 1;
 }

Re: [BUG -tip/sched] INFO: suspicious RCU usage

[Peter Zijlstra]

I made that

---
Subject: sched, nohz: Fix missing RCU read lock
From: Peter Zijlstra <a.p.zijlstra@chello.nl>
Date: Wed Dec 07 14:28:44 CET 2011

Yong Zhang reported:

> [ INFO: suspicious RCU usage. ]
> kernel/sched/fair.c:5091 suspicious rcu_dereference_check() usage!

This is due to the sched_domain stuff being RCU protected and commit
0b005cf5 ("sched, nohz: Implement sched group, domain
aware nohz idle load balancing") overlooking this fact.

The sd variable only lives inside the for_each_domain() block, so we
only need to wrap that.

Cc: Suresh Siddha <suresh.b.siddha@intel.com>
Reported-by: Yong Zhang <yong.zhang0@gmail.com>
Signed-off-by: Peter Zijlstra <a.p.zijlstra@chello.nl>
Link: http://lkml.kernel.org/n/tip-gt616u9aetdzgb1oc4m6uydr@git.kernel.org
---
Index: linux-2.6/kernel/sched/fair.c
===================================================================
--- linux-2.6.orig/kernel/sched/fair.c
+++ linux-2.6/kernel/sched/fair.c
@@ -5088,23 +5088,28 @@ static inline int nohz_kick_needed(struc
 	if (rq->nr_running >= 2)
 		goto need_kick;
 
+	rcu_read_lock();
 	for_each_domain(cpu, sd) {
 		struct sched_group *sg = sd->groups;
 		struct sched_group_power *sgp = sg->sgp;
 		int nr_busy = atomic_read(&sgp->nr_busy_cpus);
 
 		if (sd->flags & SD_SHARE_PKG_RESOURCES && nr_busy > 1)
-			goto need_kick;
+			goto need_kick_unlock;
 
 		if (sd->flags & SD_ASYM_PACKING && nr_busy != sg->group_weight
 		    && (cpumask_first_and(nohz.idle_cpus_mask,
 					  sched_domain_span(sd)) < cpu))
-			goto need_kick;
+			goto need_kick_unlock;
 
 		if (!(sd->flags & (SD_SHARE_PKG_RESOURCES | SD_ASYM_PACKING)))
 			break;
 	}
+	rcu_read_unlock();
 	return 0;
+
+need_kick_unlock:
+	rcu_read_unlock();
 need_kick:
 	return 1;
 }

Re: [BUG -tip/sched] INFO: suspicious RCU usage

[Yong Zhang]

On Wed, Dec 07, 2011 at 02:32:08PM +0100, Peter Zijlstra wrote:
> I made that
> 
> ---
> Subject: sched, nohz: Fix missing RCU read lock
> From: Peter Zijlstra <a.p.zijlstra@chello.nl>
> Date: Wed Dec 07 14:28:44 CET 2011
> 
> Yong Zhang reported:
> 
> > [ INFO: suspicious RCU usage. ]
> > kernel/sched/fair.c:5091 suspicious rcu_dereference_check() usage!
> 
> This is due to the sched_domain stuff being RCU protected and commit
> 0b005cf5 ("sched, nohz: Implement sched group, domain
> aware nohz idle load balancing") overlooking this fact.
> 
> The sd variable only lives inside the for_each_domain() block, so we
> only need to wrap that.
> 

Works for me.
Tested-by: Yong Zhang <yong.zhang0@gmail.com>

Thanks,
Yong

> Cc: Suresh Siddha <suresh.b.siddha@intel.com>
> Reported-by: Yong Zhang <yong.zhang0@gmail.com>
> Signed-off-by: Peter Zijlstra <a.p.zijlstra@chello.nl>
> Link: http://lkml.kernel.org/n/tip-gt616u9aetdzgb1oc4m6uydr@git.kernel.org
> ---
> Index: linux-2.6/kernel/sched/fair.c
> ===================================================================
> --- linux-2.6.orig/kernel/sched/fair.c
> +++ linux-2.6/kernel/sched/fair.c
> @@ -5088,23 +5088,28 @@ static inline int nohz_kick_needed(struc
>  	if (rq->nr_running >= 2)
>  		goto need_kick;
>  
> +	rcu_read_lock();
>  	for_each_domain(cpu, sd) {
>  		struct sched_group *sg = sd->groups;
>  		struct sched_group_power *sgp = sg->sgp;
>  		int nr_busy = atomic_read(&sgp->nr_busy_cpus);
>  
>  		if (sd->flags & SD_SHARE_PKG_RESOURCES && nr_busy > 1)
> -			goto need_kick;
> +			goto need_kick_unlock;
>  
>  		if (sd->flags & SD_ASYM_PACKING && nr_busy != sg->group_weight
>  		    && (cpumask_first_and(nohz.idle_cpus_mask,
>  					  sched_domain_span(sd)) < cpu))
> -			goto need_kick;
> +			goto need_kick_unlock;
>  
>  		if (!(sd->flags & (SD_SHARE_PKG_RESOURCES | SD_ASYM_PACKING)))
>  			break;
>  	}
> +	rcu_read_unlock();
>  	return 0;
> +
> +need_kick_unlock:
> +	rcu_read_unlock();
>  need_kick:
>  	return 1;
>  }
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/

-- 
Only stand for myself

[tip:sched/core] sched, nohz: Fix missing RCU read lock

[tip-bot for Peter Zijlstra]

Commit-ID:  067491b7313c41f49607fce782d29344d1472587
Gitweb:     http://git.kernel.org/tip/067491b7313c41f49607fce782d29344d1472587
Author:     Peter Zijlstra <peterz@infradead.org>
AuthorDate: Wed, 7 Dec 2011 14:32:08 +0100
Committer:  Ingo Molnar <mingo@elte.hu>
CommitDate: Thu, 8 Dec 2011 05:45:48 +0100

sched, nohz: Fix missing RCU read lock

Yong Zhang reported:

 > [ INFO: suspicious RCU usage. ]
 > kernel/sched/fair.c:5091 suspicious rcu_dereference_check() usage!

This is due to the sched_domain stuff being RCU protected and
commit 0b005cf5 ("sched, nohz: Implement sched group, domain
aware nohz idle load balancing") overlooking this fact.

The sd variable only lives inside the for_each_domain() block,
so we only need to wrap that.

Reported-by: Yong Zhang <yong.zhang0@gmail.com>
Tested-by: Yong Zhang <yong.zhang0@gmail.com>
Signed-off-by: Peter Zijlstra <a.p.zijlstra@chello.nl>
Cc: Suresh Siddha <suresh.b.siddha@intel.com>
Link: http://lkml.kernel.org/r/1323264728.32012.107.camel@twins
Signed-off-by: Ingo Molnar <mingo@elte.hu>
---
 kernel/sched/fair.c |    9 +++++++--
 1 files changed, 7 insertions(+), 2 deletions(-)

diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index 6482136..a4d2b7a 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -5088,23 +5088,28 @@ static inline int nohz_kick_needed(struct rq *rq, int cpu)
 	if (rq->nr_running >= 2)
 		goto need_kick;
 
+	rcu_read_lock();
 	for_each_domain(cpu, sd) {
 		struct sched_group *sg = sd->groups;
 		struct sched_group_power *sgp = sg->sgp;
 		int nr_busy = atomic_read(&sgp->nr_busy_cpus);
 
 		if (sd->flags & SD_SHARE_PKG_RESOURCES && nr_busy > 1)
-			goto need_kick;
+			goto need_kick_unlock;
 
 		if (sd->flags & SD_ASYM_PACKING && nr_busy != sg->group_weight
 		    && (cpumask_first_and(nohz.idle_cpus_mask,
 					  sched_domain_span(sd)) < cpu))
-			goto need_kick;
+			goto need_kick_unlock;
 
 		if (!(sd->flags & (SD_SHARE_PKG_RESOURCES | SD_ASYM_PACKING)))
 			break;
 	}
+	rcu_read_unlock();
 	return 0;
+
+need_kick_unlock:
+	rcu_read_unlock();
 need_kick:
 	return 1;
 }