per_cpu_counter_sum lockdep warning

per_cpu_counter_sum lockdep warning

[Balbir Singh]

Saw this warning on an x86_64 box, while booting up 2.6.26-rc4. Has anybody else
seen it? Working on it?

=================================
[ INFO: inconsistent lock state ]
2.6.26-rc4 #2
---------------------------------
inconsistent {in-hardirq-W} -> {hardirq-on-W} usage.
init/1 [HC0[0]:SC0[0]:HE1:SE1] takes:
 (&fbc->lock){+...}, at: [<ffffffff80386382>] __percpu_counter_sum+0xf/0x5a
{in-hardirq-W} state was registered at:
  [<ffffffffffffffff>] 0xffffffffffffffff
irq event stamp: 694563
hardirqs last  enabled at (694563): [<ffffffff80297533>] kmem_cache_free+0x1a9/0x1bb
hardirqs last disabled at (694562): [<ffffffff802973aa>] kmem_cache_free+0x20/0x1bb
softirqs last  enabled at (692960): [<ffffffff8023a27b>] __do_softirq+0xda/0xe3
softirqs last disabled at (692945): [<ffffffff8020c4ac>] call_softirq+0x1c/0x28

other info that might help us debug this:
no locks held by init/1.

stack backtrace:
Pid: 1, comm: init Not tainted 2.6.26-rc4 #2

Call Trace:
 [<ffffffff802518e6>] print_usage_bug+0x15e/0x16f
 [<ffffffff8025281f>] mark_lock+0x22f/0x416
 [<ffffffff80386382>] ? __percpu_counter_sum+0xf/0x5a
 [<ffffffff80253576>] __lock_acquire+0x4e7/0xc8a
 [<ffffffff80386382>] ? __percpu_counter_sum+0xf/0x5a
 [<ffffffff80253da7>] lock_acquire+0x8e/0xb2
 [<ffffffff80386382>] ? __percpu_counter_sum+0xf/0x5a
 [<ffffffff805990d7>] _spin_lock+0x26/0x53
 [<ffffffff80386382>] __percpu_counter_sum+0xf/0x5a
 [<ffffffff803139e2>] ext3_statfs+0xd6/0x160
 [<ffffffff8029c6be>] vfs_statfs+0x4e/0x6d
 [<ffffffff8029c85f>] vfs_statfs_native+0x22/0x48
 [<ffffffff8029c945>] sys_statfs+0x4d/0x89
 [<ffffffff80252bfe>] ? trace_hardirqs_on+0xf9/0x124
 [<ffffffff8037da3d>] ? __up_read+0x8f/0x97
 [<ffffffff8024ad5a>] ? up_read+0x26/0x2a
 [<ffffffff80598cf0>] ? trace_hardirqs_on_thunk+0x35/0x3a
 [<ffffffff80252bfe>] ? trace_hardirqs_on+0xf9/0x124
 [<ffffffff80598cf0>] ? trace_hardirqs_on_thunk+0x35/0x3a
 [<ffffffff8020b1fb>] system_call_after_swapgs+0x7b/0x80

-- 
	Warm Regards,
	Balbir Singh
	Linux Technology Center
	IBM, ISTL

Re: per_cpu_counter_sum lockdep warning

[Matthew Wilcox]

On Wed, Jun 04, 2008 at 08:57:16AM +0530, Balbir Singh wrote:
> Saw this warning on an x86_64 box, while booting up 2.6.26-rc4. Has anybody else
> seen it? Working on it?

I've neither seen it, nor am I working on it, but I can decode it.

> inconsistent {in-hardirq-W} -> {hardirq-on-W} usage.

Translation: "This lock was previously grabbed in hardirq context.  Now
someone's taking it in process context without interrupts disabled.
That could lead to a deadlock."

> init/1 [HC0[0]:SC0[0]:HE1:SE1] takes:
>  (&fbc->lock){+...}, at: [<ffffffff80386382>] __percpu_counter_sum+0xf/0x5a

That's the name of the lock -- &fbc->lock and the function where it
happens.

> {in-hardirq-W} state was registered at:
>   [<ffffffffffffffff>] 0xffffffffffffffff

Drat, no backtrace for the guy who took the lock in hardirq context.

> Call Trace:
>  [<ffffffff802518e6>] print_usage_bug+0x15e/0x16f
>  [<ffffffff8025281f>] mark_lock+0x22f/0x416
>  [<ffffffff80386382>] ? __percpu_counter_sum+0xf/0x5a
>  [<ffffffff80253576>] __lock_acquire+0x4e7/0xc8a
>  [<ffffffff80386382>] ? __percpu_counter_sum+0xf/0x5a
>  [<ffffffff80253da7>] lock_acquire+0x8e/0xb2
>  [<ffffffff80386382>] ? __percpu_counter_sum+0xf/0x5a
>  [<ffffffff805990d7>] _spin_lock+0x26/0x53
>  [<ffffffff80386382>] __percpu_counter_sum+0xf/0x5a
>  [<ffffffff803139e2>] ext3_statfs+0xd6/0x160

ext3_statfs was the one who asked for the lock to be taken without
disabling interrupts.


Some percpu counters are supposed to be used from interrupt context.
These are created with percpu_counter_init_irq.  Others are not and
should be created with percpu_counter_init.  It seems like someone's
made a mess of that rule.  This is likely to be a driver, IMO.  Perhaps
you could work on tracking this down?

-- 
Intel are signing my paycheques ... these opinions are still mine
"Bill, look, we understand that you're interested in selling us this
operating system, but compare it to ours.  We can't possibly take such
a retrograde step."

Re: per_cpu_counter_sum lockdep warning

[Balbir Singh]

Matthew Wilcox wrote:
> On Wed, Jun 04, 2008 at 08:57:16AM +0530, Balbir Singh wrote:
>> Saw this warning on an x86_64 box, while booting up 2.6.26-rc4. Has anybody else
>> seen it? Working on it?
> 
> I've neither seen it, nor am I working on it, but I can decode it.
> 
>> inconsistent {in-hardirq-W} -> {hardirq-on-W} usage.
> 
> Translation: "This lock was previously grabbed in hardirq context.  Now
> someone's taking it in process context without interrupts disabled.
> That could lead to a deadlock."
> 

I understand this part. I did not want to interpret the data, but I think that's
a better way of reporting problems.

>> init/1 [HC0[0]:SC0[0]:HE1:SE1] takes:
>>  (&fbc->lock){+...}, at: [<ffffffff80386382>] __percpu_counter_sum+0xf/0x5a
> 
> That's the name of the lock -- &fbc->lock and the function where it
> happens.
> 
>> {in-hardirq-W} state was registered at:
>>   [<ffffffffffffffff>] 0xffffffffffffffff
> 
> Drat, no backtrace for the guy who took the lock in hardirq context.
> 
>> Call Trace:
>>  [<ffffffff802518e6>] print_usage_bug+0x15e/0x16f
>>  [<ffffffff8025281f>] mark_lock+0x22f/0x416
>>  [<ffffffff80386382>] ? __percpu_counter_sum+0xf/0x5a
>>  [<ffffffff80253576>] __lock_acquire+0x4e7/0xc8a
>>  [<ffffffff80386382>] ? __percpu_counter_sum+0xf/0x5a
>>  [<ffffffff80253da7>] lock_acquire+0x8e/0xb2
>>  [<ffffffff80386382>] ? __percpu_counter_sum+0xf/0x5a
>>  [<ffffffff805990d7>] _spin_lock+0x26/0x53
>>  [<ffffffff80386382>] __percpu_counter_sum+0xf/0x5a
>>  [<ffffffff803139e2>] ext3_statfs+0xd6/0x160
> 
> ext3_statfs was the one who asked for the lock to be taken without
> disabling interrupts.
> 
> 
> Some percpu counters are supposed to be used from interrupt context.
> These are created with percpu_counter_init_irq.  Others are not and
> should be created with percpu_counter_init.  It seems like someone's
> made a mess of that rule.  This is likely to be a driver, IMO.  Perhaps
> you could work on tracking this down?
> 

Sure, I will. Let me poke harder, I'll recheck all patches I have applied (if
any) on my current tree.


-- 
	Warm Regards,
	Balbir Singh
	Linux Technology Center
	IBM, ISTL

Re: per_cpu_counter_sum lockdep warning

[Balbir Singh]

> 
> Some percpu counters are supposed to be used from interrupt context.
> These are created with percpu_counter_init_irq.  Others are not and
> should be created with percpu_counter_init.  It seems like someone's
> made a mess of that rule.  This is likely to be a driver, IMO.  Perhaps
> you could work on tracking this down?
> 

Sorry for the noise. It was patch I had applied on my tree, not yet in mainline.
The explanation though did help me find and fix the problem.


-- 
	Warm Regards,
	Balbir Singh
	Linux Technology Center
	IBM, ISTL