Real-life kernel debugging scenario

Real-life kernel debugging scenario

[walt]

This describes a real problem I've had twice in the last two
years while tracking Linus's kernel tree:

I update my local kernel sources every morning using cg-update
(formerly bk-pull) and compile and install and reboot the new
kernel.

Okay.  On rare occasions I get a kernel panic on reboot.  So...
I know that something Linus committed in the last 24 hours is
responsible for the problem.

The last two times this happened I was able to guess which
commit caused the problem and I emailed the developer off-
list and got the problem fixed very quickly. (This is why
I love open-source software!)

My worry:  what happens when I'm not smart enough to guess
which developer to email?  My first instinct is to back out
the most recent commits one-by-one until the bug goes away.

First:  is this an optimal tactic?

Second:  how to back out individual commits using git or
cogito?  I suppose this is already spelled out in the docs,
but I invite everyone to point me to the relevant places
in the docs that have escaped my attention so far.

Thanks!

Re: Real-life kernel debugging scenario

[David Lang]

On Mon, 7 Nov 2005, walt wrote:

> My worry:  what happens when I'm not smart enough to guess
> which developer to email?  My first instinct is to back out
> the most recent commits one-by-one until the bug goes away.
>
> First:  is this an optimal tactic?

it will work, but it's not optimal. this is exactly what git bisect is 
designed for.

you tell it that the prior night's version was good and the current 
version is bad. It picks a version 'halfway' in between the two and you 
test that. tell git if the test failed or not and it will then give 
you the next one to try. repeat until you identify exactly which commit 
triggers the problem.

unfortunantly I can't trivially point you at the right place in the docs.

David Lang

-- 
There are two ways of constructing a software design. One way is to make it so simple that there are obviously no deficiencies. And the other way is to make it so complicated that there are no obvious deficiencies.
  -- C.A.R. Hoare

Re: Real-life kernel debugging scenario

[Junio C Hamano]

walt <wa1ter@myrealbox.com> writes:

> First:  is this an optimal tactic?

Not if you are a git user.

> Second:  how to back out individual commits using git or
> cogito?  I suppose this is already spelled out in the docs,
> but I invite everyone to point me to the relevant places
> in the docs that have escaped my attention so far.

git-bisect(1).

Re: Real-life kernel debugging scenario

[Linus Torvalds]

On Mon, 7 Nov 2005, walt wrote:
> 
> Okay.  On rare occasions I get a kernel panic on reboot.  So...
> I know that something Linus committed in the last 24 hours is
> responsible for the problem.
> 
> The last two times this happened I was able to guess which
> commit caused the problem and I emailed the developer off-
> list and got the problem fixed very quickly. (This is why
> I love open-source software!)
> 
> My worry:  what happens when I'm not smart enough to guess
> which developer to email?  My first instinct is to back out
> the most recent commits one-by-one until the bug goes away.

No. This is what "git bisect" is there for.

It works wonderfully well. To the point where I don't even try to be smart 
about things: I've had two cases in the last month of merges that caused 
problems for me, and instead of even trying to guess which patch it was, I 
just bisected it.

> First:  is this an optimal tactic?
> 
> Second:  how to back out individual commits using git or
> cogito?  I suppose this is already spelled out in the docs,
> but I invite everyone to point me to the relevant places
> in the docs that have escaped my attention so far.

First, try "git bisect". It doesn't depend on backing out individual 
commits, instead it starts a special "bisect" branch, and (as the name 
implies) does a binary search within that branch to figure out what is 
wrong.

The way to use "git bisect" couldn't be easier.

Figure out what the oldest bad state you know about is (that's usually the 
head of "master", since that's what you just tried to boot and failed at). 
Also, figure out the most recent known-good commit (usually the _previous_ 
kernel you ran: and if you've only done a single "pull" in between, it 
will be ORIG_HEAD).

Then do

	git bisect start
	git bisect bad master		<- mark "master" as the bad state
	git bisect good ORIG_HEAD	<- mark ORIG_HEAD as good (or
					   whatever other known-good 
					   thing you booted laste)

and at this point "git bisect" will churn for a while, and tell you what 
the mid-point between those two commits are, and check that state out as 
the head of the bew "bisect" branch.

Compile and reboot.

If it's good, just do

	git bisect good		<- mark current head as good

otherwise, reboot into a good kernel instead, and do (surprise surprise, 
git really is very intuitive):

	git bisect bad		<- mark current head as bad

and whatever you do, git will select a new half-way point. Do this for a 
while, until git tells you exactly which commit was the first bad commit. 
That's your culprit.

It really works wonderfully well, except for the case where there was 
_another_ commit that broke something in between, like introduced some 
stupid compile error. In that case you should not mark that commit good or 
bad: you should try to find another commit close-by, and do a "git reset 
--hard <newcommit>" to try out _that_ commit instead, and then test that 
instead (and mark it good or bad).

You can do "git bisect visualize" while you do all this to see what's 
going on by starting up gitk on the bisection range.

Finally, once you've figured out exactly which commit was bad, you can 
then go back to the master branch, and try reverting just that commit:

	git checkout master
	git revert <bad-commit-id>

to verify that the top-of-kernel works with that single commit reverted.

		Linus

Re: Real-life kernel debugging scenario

[wa1ter]

On Mon, 7 Nov 2005, Linus Torvalds wrote:

[...]
> The way to use "git bisect" couldn't be easier.

I see that Junio just added your entire response to Documentation/howto/
isolate-bugs-with-bisect.txt where even I could manage to find it :o)

> Also, figure out the most recent known-good commit (usually the _previous_
> kernel you ran: and if you've only done a single "pull" in between, it
> will be ORIG_HEAD).

The built-in variable ORIG_HEAD isn't explained anywhere AFAICT, or at
least it wasn't until today.  Are there other such magic variables which
might be useful for us to know about?

Re: Real-life kernel debugging scenario

[Jon Loeliger]

On Wed, 2005-11-09 at 11:40, wa1ter@myrealbox.com wrote:
> 
> On Mon, 7 Nov 2005, Linus Torvalds wrote:
> 
> [...]
> > The way to use "git bisect" couldn't be easier.
> 
> I see that Junio just added your entire response to
> Documentation/howto/
> isolate-bugs-with-bisect.txt where even I could manage to find it :o)

Why, thank you. :-)


> The built-in variable ORIG_HEAD isn't explained anywhere AFAICT, or at
> least it wasn't until today.  Are there other such magic variables
> which
> might be useful for us to know about?

Hey Junio,

I'm going to call "I told you so!" on this one! :-)

Ref:
    Message-ID:  <E1EXTw5-00063o-Gt@jdl.com>
    From: Jon Loeliger <jdl@freescale.com>
    Subject: Now What?
    Date: Wed, 2 Nov 2005 18:30:37 -0700

Where in I mumbled:

    I feel that an explanation of all of the behind-the-scripts-
    in-.git communication files is needed.  In particular these:

        FETCH_HEAD
        MERGE_HEAD
        LAST_MERGE
        MERGE_MSG

    These need to be mentioned and explained because they
    frequently form exactly the critical missing link or
    starting point after a failed fetch or merge.

You know.

jdl

Re: Real-life kernel debugging scenario

[Linus Torvalds]

On Wed, 9 Nov 2005, wa1ter@myrealbox.com wrote:
> 
> The built-in variable ORIG_HEAD isn't explained anywhere AFAICT, or at
> least it wasn't until today.  Are there other such magic variables which
> might be useful for us to know about?

No, ORIG_HEAD is pretty much it.

There's a few special files that a non-committed merge leaves around 
(either because you asked it to not be committed, or because it had 
clashes and requires manual fixing), but they are not normally useful to 
any regular people. You can poke around in the ".git" directory after such 
a merge if you want (MERGE_HEAD, MERGE_MSG).

ORIG_HEAD is _very_ useful, though, and I use it all the time. Any time 
you've pulled something from somebody else, and you wonder what you 
pulled, just go

	gitk ORIG_HEAD..

and you'll see exactly what new stuff you got in your branch. I often do 
that command line several times a day as I merge stuff. Otherwise the new 
stuff tends to be hidden in the noise.

(There's also FETCH_HEAD, which is the head of the last fetch, so if you 
aren't interested in any potential merge, you could instead use the range 
ORIG_HEAD..FETCH_HEAD, but quite frankly, I doubt anybody really cares 
except for the internal git fetching/pulling logic).

		Linus