lockdep and debug objects together are broken?

lockdep and debug objects together are broken?

[Nick Piggin]

Hi,

I've had a problem frustrating my testing because lockdep was silently turning
itself off... I patched out the code to disable lockdep after the first error,
and it started showing up weird errors. kernel/fork.c:990 seemed to be the
first to trigger (hard irqs disabled) from a call_usermodehelper call. Later,
migration thread was reported to try to unlock rq->lock although it was
holding no locks. Then init was reported to return to userspace without
releasing an objectdebug hash lock.

All that went away and everything seemed to work properly with debug objects
configured out.

I didn't get too far in trying to debug the problem. But it should be easy
enough to reproduce (if not, I can post traces or test patches).

Thanks,
Nick

Re: lockdep and debug objects together are broken?

[Vegard Nossum]

On Tue, Jan 20, 2009 at 9:55 AM, Nick Piggin <npiggin@suse.de> wrote:
> Hi,
>
> I've had a problem frustrating my testing because lockdep was silently turning
> itself off... I patched out the code to disable lockdep after the first error,
> and it started showing up weird errors. kernel/fork.c:990 seemed to be the
> first to trigger (hard irqs disabled) from a call_usermodehelper call. Later,
> migration thread was reported to try to unlock rq->lock although it was
> holding no locks. Then init was reported to return to userspace without
> releasing an objectdebug hash lock.
>
> All that went away and everything seemed to work properly with debug objects
> configured out.
>
> I didn't get too far in trying to debug the problem. But it should be easy
> enough to reproduce (if not, I can post traces or test patches).

I just built a kernel with lockdep and debugobjects enabled, and
everything seemed fine. I think you should post your kernel version,
config, and the lockdep patch (if needed -- it didn't seem to turn
itself off here).


Vegard

-- 
"The animistic metaphor of the bug that maliciously sneaked in while
the programmer was not looking is intellectually dishonest as it
disguises that the error is the programmer's own creation."
	-- E. W. Dijkstra, EWD1036

Re: lockdep and debug objects together are broken?

[Nick Piggin]

On Tue, Jan 20, 2009 at 10:11:47PM +0100, Vegard Nossum wrote:
> On Tue, Jan 20, 2009 at 9:55 AM, Nick Piggin <npiggin@suse.de> wrote:
> > Hi,
> >
> > I've had a problem frustrating my testing because lockdep was silently turning
> > itself off... I patched out the code to disable lockdep after the first error,
> > and it started showing up weird errors. kernel/fork.c:990 seemed to be the
> > first to trigger (hard irqs disabled) from a call_usermodehelper call. Later,
> > migration thread was reported to try to unlock rq->lock although it was
> > holding no locks. Then init was reported to return to userspace without
> > releasing an objectdebug hash lock.
> >
> > All that went away and everything seemed to work properly with debug objects
> > configured out.
> >
> > I didn't get too far in trying to debug the problem. But it should be easy
> > enough to reproduce (if not, I can post traces or test patches).
> 
> I just built a kernel with lockdep and debugobjects enabled, and
> everything seemed fine. I think you should post your kernel version,
> config, and the lockdep patch (if needed -- it didn't seem to turn
> itself off here).

Are you sure? Ie. sysrq+D a still works properly? In that case, you
wouldn't need the lockdep patch because it just prevents the feature from being
switched off.

I'll have to dig a bit further, then. The annoying thing is that
lockdep turns itself off at the drop of a hat (and this particular
problem seems to happen without any backtraces), so it invalidates
all your lockdep testing if you don't realise it has turned itself
off.

Is there a way to re-arm lockdep? That would be neat.

Re: lockdep and debug objects together are broken?

[Ingo Molnar]

* Nick Piggin <npiggin@suse.de> wrote:

> On Tue, Jan 20, 2009 at 10:11:47PM +0100, Vegard Nossum wrote:
> > On Tue, Jan 20, 2009 at 9:55 AM, Nick Piggin <npiggin@suse.de> wrote:
> > > Hi,
> > >
> > > I've had a problem frustrating my testing because lockdep was silently turning
> > > itself off... I patched out the code to disable lockdep after the first error,
> > > and it started showing up weird errors. kernel/fork.c:990 seemed to be the
> > > first to trigger (hard irqs disabled) from a call_usermodehelper call. Later,
> > > migration thread was reported to try to unlock rq->lock although it was
> > > holding no locks. Then init was reported to return to userspace without
> > > releasing an objectdebug hash lock.
> > >
> > > All that went away and everything seemed to work properly with debug objects
> > > configured out.
> > >
> > > I didn't get too far in trying to debug the problem. But it should be easy
> > > enough to reproduce (if not, I can post traces or test patches).
> > 
> > I just built a kernel with lockdep and debugobjects enabled, and
> > everything seemed fine. I think you should post your kernel version,
> > config, and the lockdep patch (if needed -- it didn't seem to turn
> > itself off here).
> 
> Are you sure? Ie. sysrq+D a still works properly? In that case, you
> wouldn't need the lockdep patch because it just prevents the feature from being
> switched off.
> 
> I'll have to dig a bit further, then. The annoying thing is that
> lockdep turns itself off at the drop of a hat (and this particular
> problem seems to happen without any backtraces), so it invalidates
> all your lockdep testing if you don't realise it has turned itself
> off.
> 
> Is there a way to re-arm lockdep? That would be neat.

Not at the moment, and it looks somewhat complicated. All lock state 
freezes the moment lockdep disarms itself. That's very much a key design 
element: rarely will you see any real lockdep-inflicted crash - even if it 
has a bug it is self-disabling itself and running for the door very 
efficiently.

So by the time you'd rearm, there's a lot of tasks with no proper locking 
state built up. We might be able to re-arm via stop_machine_run perhaps.

	Ingo

Re: lockdep and debug objects together are broken?

[Peter Zijlstra]

On Wed, 2009-01-21 at 12:42 +0100, Ingo Molnar wrote:

> So by the time you'd rearm, there's a lot of tasks with no proper locking 
> state built up. We might be able to re-arm via stop_machine_run perhaps.

Won't work either, kstopmachine only preempts everybody. We'd require
something stronger.

What we need is a point where there's guaranteed no locks held, for
regular tasks that would be a trip to userspace and back, but for kernel
tasks that's a bit harder -- does the freezer stuff guarantee this?

Supposing we have such a point for all tasks, what you then do is wipe
all lock state and rig a trigger to start tracking lock state once you
passed through the point.

Re: lockdep and debug objects together are broken?

[Ingo Molnar]

* Peter Zijlstra <a.p.zijlstra@chello.nl> wrote:

> On Wed, 2009-01-21 at 12:42 +0100, Ingo Molnar wrote:
> 
> > So by the time you'd rearm, there's a lot of tasks with no proper 
> > locking state built up. We might be able to re-arm via 
> > stop_machine_run perhaps.
> 
> Won't work either, kstopmachine only preempts everybody. We'd require 
> something stronger.

indeed - mutexes wont be covered.

> What we need is a point where there's guaranteed no locks held, for 
> regular tasks that would be a trip to userspace and back, but for kernel 
> tasks that's a bit harder -- does the freezer stuff guarantee this?

yes, that might be doable.

Sounds very fragile though.

	Ingo

Re: lockdep and debug objects together are broken?

[Nick Piggin]

On Wed, Jan 21, 2009 at 12:42:29PM +0100, Ingo Molnar wrote:
> 
> * Nick Piggin <npiggin@suse.de> wrote:
> 
> > On Tue, Jan 20, 2009 at 10:11:47PM +0100, Vegard Nossum wrote:
> > > On Tue, Jan 20, 2009 at 9:55 AM, Nick Piggin <npiggin@suse.de> wrote:
> > > > Hi,
> > > >
> > > > I've had a problem frustrating my testing because lockdep was silently turning
> > > > itself off... I patched out the code to disable lockdep after the first error,
> > > > and it started showing up weird errors. kernel/fork.c:990 seemed to be the
> > > > first to trigger (hard irqs disabled) from a call_usermodehelper call. Later,
> > > > migration thread was reported to try to unlock rq->lock although it was
> > > > holding no locks. Then init was reported to return to userspace without
> > > > releasing an objectdebug hash lock.
> > > >
> > > > All that went away and everything seemed to work properly with debug objects
> > > > configured out.
> > > >
> > > > I didn't get too far in trying to debug the problem. But it should be easy
> > > > enough to reproduce (if not, I can post traces or test patches).
> > > 
> > > I just built a kernel with lockdep and debugobjects enabled, and
> > > everything seemed fine. I think you should post your kernel version,
> > > config, and the lockdep patch (if needed -- it didn't seem to turn
> > > itself off here).
> > 
> > Are you sure? Ie. sysrq+D a still works properly? In that case, you
> > wouldn't need the lockdep patch because it just prevents the feature from being
> > switched off.
> > 
> > I'll have to dig a bit further, then. The annoying thing is that
> > lockdep turns itself off at the drop of a hat (and this particular
> > problem seems to happen without any backtraces), so it invalidates
> > all your lockdep testing if you don't realise it has turned itself
> > off.
> > 
> > Is there a way to re-arm lockdep? That would be neat.
> 
> Not at the moment, and it looks somewhat complicated. All lock state 
> freezes the moment lockdep disarms itself. That's very much a key design 
> element: rarely will you see any real lockdep-inflicted crash - even if it 
> has a bug it is self-disabling itself and running for the door very 
> efficiently.

Lockdep isn't exactly for production systems though, is it? If you
want to debug some problem but you have other code (that you don't
have knowledge to debug) is switching it off...

Also, I'd guess that most bugs in lockdep would probably fall pretty
neatly into either the "pretty harmless" or "completely take down the
system" categories ;)

Re: lockdep and debug objects together are broken?

[Ingo Molnar]

* Nick Piggin <npiggin@suse.de> wrote:

> On Wed, Jan 21, 2009 at 12:42:29PM +0100, Ingo Molnar wrote:
> > 
> > * Nick Piggin <npiggin@suse.de> wrote:
> > 
> > > On Tue, Jan 20, 2009 at 10:11:47PM +0100, Vegard Nossum wrote:
> > > > On Tue, Jan 20, 2009 at 9:55 AM, Nick Piggin <npiggin@suse.de> wrote:
> > > > > Hi,
> > > > >
> > > > > I've had a problem frustrating my testing because lockdep was silently turning
> > > > > itself off... I patched out the code to disable lockdep after the first error,
> > > > > and it started showing up weird errors. kernel/fork.c:990 seemed to be the
> > > > > first to trigger (hard irqs disabled) from a call_usermodehelper call. Later,
> > > > > migration thread was reported to try to unlock rq->lock although it was
> > > > > holding no locks. Then init was reported to return to userspace without
> > > > > releasing an objectdebug hash lock.
> > > > >
> > > > > All that went away and everything seemed to work properly with debug objects
> > > > > configured out.
> > > > >
> > > > > I didn't get too far in trying to debug the problem. But it should be easy
> > > > > enough to reproduce (if not, I can post traces or test patches).
> > > > 
> > > > I just built a kernel with lockdep and debugobjects enabled, and
> > > > everything seemed fine. I think you should post your kernel version,
> > > > config, and the lockdep patch (if needed -- it didn't seem to turn
> > > > itself off here).
> > > 
> > > Are you sure? Ie. sysrq+D a still works properly? In that case, you
> > > wouldn't need the lockdep patch because it just prevents the feature from being
> > > switched off.
> > > 
> > > I'll have to dig a bit further, then. The annoying thing is that
> > > lockdep turns itself off at the drop of a hat (and this particular
> > > problem seems to happen without any backtraces), so it invalidates
> > > all your lockdep testing if you don't realise it has turned itself
> > > off.
> > > 
> > > Is there a way to re-arm lockdep? That would be neat.
> > 
> > Not at the moment, and it looks somewhat complicated. All lock state 
> > freezes the moment lockdep disarms itself. That's very much a key design 
> > element: rarely will you see any real lockdep-inflicted crash - even if it 
> > has a bug it is self-disabling itself and running for the door very 
> > efficiently.
> 
> Lockdep isn't exactly for production systems though, is it? If you want 
> to debug some problem but you have other code (that you don't have 
> knowledge to debug) is switching it off...
> 
> Also, I'd guess that most bugs in lockdep would probably fall pretty 
> neatly into either the "pretty harmless" or "completely take down the 
> system" categories ;)

i think lockdep could be expanded into production use via code patching 
techniques.

So in that sense the rearm bit could be useful - it would give us a 
lockdep variant that would run for the first 5 minutes of uptime (where 
90% of all lockdep reports trigger: lockdep maps the dependencies very 
quickly) - and could turn itself off after that, and patch out / disable 
its callbacks.

The memory footprint would still remain, but that is not nearly as much of 
a problem for production systems as runtime overhead.

	Ingo