Re: Linux 2.6.27-rc3: kernel BUG at mm/vmalloc.c - bisected

Re: Linux 2.6.27-rc3: kernel BUG at mm/vmalloc.c - bisected

[Andrew Morton]

On Fri, 22 Aug 2008 11:37:33 -0400 "Alan D. Brunelle" <Alan.Brunelle@hp.com> wrote:

> Andrew Morton wrote:
> 
> > 
> > You could confirm/debug it with something along the lines of this:
> > 
> > --- a/mm/vmalloc.c~a
> > +++ a/mm/vmalloc.c
> > @@ -214,7 +214,9 @@ __get_vm_area_node(unsigned long size, u
> >  	unsigned long align = 1;
> >  	unsigned long addr;
> >  
> > -	BUG_ON(in_interrupt());
> > +	if (preempt_count() > 10)
> > +		printk("%s: preempt_count()=%d\n", __func__, preempt_count());
> > +	WARN_ON(in_interrupt());
> >  	if (flags & VM_IOREMAP) {
> >  		int bit = fls(size);
> >  
> > _
> > 
> > 
> > But this bug could be in practically anywhere in the kernel and
> > bisection is by far the best way to find it.  It's sad and odd that
> > bisection landed you on a merge commit.  I'd suggest that you persist
> > with the bisection (please). 
> > http://www.kernel.org/doc/local/git-quick.html#example might be useful.
> 
> Sorry, was off on other things for the last couple of days:
> 
> I /did/ bisect it down to the aforementioned merge, the question is: How
> to crack open that merge into it's composite pieces? (Where do I go to
> bisect within that?)

urgh, it's irritating when git-bisect directs you to a merge commit - it
hasn't done it for me for ages.

One (probably wrong) approach is to run

	gitk 1c89ac55017f982355c7761e1c912c88c941483d

then peer at the output, work out which real commits were in that
merge.

It looks like the merge ended with
b1b135c8d619cb2c7045d6ee4e48375882518bb5 and started with
40c42076ebd362dc69210cccea101ac80b6d4bd4, so perhaps you can do

	git bisect bad b1b135c8d619cb2c7045d6ee4e48375882518bb5
	git bisect good 40c42076ebd362dc69210cccea101ac80b6d4bd4

That has an off-by-one error, in that it assumes that
40c42076ebd362dc69210cccea101ac80b6d4bd4 is good, which we don't know. 
Perhaps one can pick the preceding commit in the `git log' output to
fix that.

Anyway, I've sent sufficient wrongness to the right list for us to work
out how to do this for real ;)

Re: Linux 2.6.27-rc3: kernel BUG at mm/vmalloc.c - bisected

[Petr Baudis]

On Fri, Aug 22, 2008 at 09:25:49AM -0700, Andrew Morton wrote:
> On Fri, 22 Aug 2008 11:37:33 -0400 "Alan D. Brunelle" <Alan.Brunelle@hp.com> wrote:
> 
> > I /did/ bisect it down to the aforementioned merge, the question is: How
> > to crack open that merge into it's composite pieces? (Where do I go to
> > bisect within that?)
> 
> urgh, it's irritating when git-bisect directs you to a merge commit - it
> hasn't done it for me for ages.

Hmm, but doesn't that happen only when it's actually really the merge
commit that introduces the bug? Both parents of the merge commit were
marked as good by the user, so...

> One (probably wrong) approach is to run
> 
> 	gitk 1c89ac55017f982355c7761e1c912c88c941483d
> 
> then peer at the output, work out which real commits were in that
> merge.
> 
> It looks like the merge ended with
> b1b135c8d619cb2c7045d6ee4e48375882518bb5 and started with
> 40c42076ebd362dc69210cccea101ac80b6d4bd4, so perhaps you can do
> 
> 	git bisect bad b1b135c8d619cb2c7045d6ee4e48375882518bb5
> 	git bisect good 40c42076ebd362dc69210cccea101ac80b6d4bd4

...I don't quite get this - according to the bisection log,

	# good: [b1b135c8d619cb2c7045d6ee4e48375882518bb5] fix spinlock recursion in hvc_console

and now you want to mark it as bad?

You could try to revert some of the merged commits at the point of the
merge, though.

-- 
				Petr "Pasky" Baudis
The next generation of interesting software will be done
on the Macintosh, not the IBM PC.  -- Bill Gates

Re: Linux 2.6.27-rc3: kernel BUG at mm/vmalloc.c - bisected

[Andrew Morton]

On Fri, 22 Aug 2008 19:16:51 +0200
Petr Baudis <pasky@suse.cz> wrote:

> On Fri, Aug 22, 2008 at 09:25:49AM -0700, Andrew Morton wrote:
> > On Fri, 22 Aug 2008 11:37:33 -0400 "Alan D. Brunelle" <Alan.Brunelle@hp.com> wrote:
> > 
> > > I /did/ bisect it down to the aforementioned merge, the question is: How
> > > to crack open that merge into it's composite pieces? (Where do I go to
> > > bisect within that?)
> > 
> > urgh, it's irritating when git-bisect directs you to a merge commit - it
> > hasn't done it for me for ages.
> 
> Hmm, but doesn't that happen only when it's actually really the merge
> commit that introduces the bug? Both parents of the merge commit were
> marked as good by the user, so...

A merge commit doesn't contain any kernel changes?  It's the individual
commits (aka "patches") which were in that merge which broke stuff. 
Confused.

We're trying to dive inside that merge commit to find out which of the
real commits caused the regression.

> > One (probably wrong) approach is to run
> > 
> > 	gitk 1c89ac55017f982355c7761e1c912c88c941483d
> > 
> > then peer at the output, work out which real commits were in that
> > merge.
> > 
> > It looks like the merge ended with
> > b1b135c8d619cb2c7045d6ee4e48375882518bb5 and started with
> > 40c42076ebd362dc69210cccea101ac80b6d4bd4, so perhaps you can do
> > 
> > 	git bisect bad b1b135c8d619cb2c7045d6ee4e48375882518bb5
> > 	git bisect good 40c42076ebd362dc69210cccea101ac80b6d4bd4
> 
> ...I don't quite get this - according to the bisection log,
> 
> 	# good: [b1b135c8d619cb2c7045d6ee4e48375882518bb5] fix spinlock recursion in hvc_console
> 
> and now you want to mark it as bad?

<what bisection log?>

I assume that Alan's bisection search ended up saying that the merge
commit (1c89ac55017f982355c7761e1c912c88c941483d) was the first bad
commit.

Now I don't know what's going on.

> You could try to revert some of the merged commits at the point of the
> merge, though.

Re: Linux 2.6.27-rc3: kernel BUG at mm/vmalloc.c - bisected

[Alan D. Brunelle]

Andrew Morton wrote:
> On Fri, 22 Aug 2008 19:16:51 +0200
> Petr Baudis <pasky@suse.cz> wrote:
> 
>> On Fri, Aug 22, 2008 at 09:25:49AM -0700, Andrew Morton wrote:
>>> On Fri, 22 Aug 2008 11:37:33 -0400 "Alan D. Brunelle" <Alan.Brunelle@hp.com> wrote:
>>>
>>>> I /did/ bisect it down to the aforementioned merge, the question is: How
>>>> to crack open that merge into it's composite pieces? (Where do I go to
>>>> bisect within that?)
>>> urgh, it's irritating when git-bisect directs you to a merge commit - it
>>> hasn't done it for me for ages.
>> Hmm, but doesn't that happen only when it's actually really the merge
>> commit that introduces the bug? Both parents of the merge commit were
>> marked as good by the user, so...
> 
> A merge commit doesn't contain any kernel changes?  It's the individual
> commits (aka "patches") which were in that merge which broke stuff. 
> Confused.
> 
> We're trying to dive inside that merge commit to find out which of the
> real commits caused the regression.
> 
>>> One (probably wrong) approach is to run
>>>
>>> 	gitk 1c89ac55017f982355c7761e1c912c88c941483d
>>>
>>> then peer at the output, work out which real commits were in that
>>> merge.
>>>
>>> It looks like the merge ended with
>>> b1b135c8d619cb2c7045d6ee4e48375882518bb5 and started with
>>> 40c42076ebd362dc69210cccea101ac80b6d4bd4, so perhaps you can do
>>>
>>> 	git bisect bad b1b135c8d619cb2c7045d6ee4e48375882518bb5
>>> 	git bisect good 40c42076ebd362dc69210cccea101ac80b6d4bd4
>> ...I don't quite get this - according to the bisection log,
>>
>> 	# good: [b1b135c8d619cb2c7045d6ee4e48375882518bb5] fix spinlock recursion in hvc_console
>>
>> and now you want to mark it as bad?
> 
> <what bisection log?>
> 
> I assume that Alan's bisection search ended up saying that the merge
> commit (1c89ac55017f982355c7761e1c912c88c941483d) was the first bad
> commit.
> 
> Now I don't know what's going on.
> 
>> You could try to revert some of the merged commits at the point of the
>> merge, though.

You're right - this is where the bisection ended up (in fact, just
removing this merge commit allowed later kernels to boot just fine).

I've patched the kernel w/ your suggestion, and now I'm getting
something a bit different - and I'm looking into this...

Begin: Loading essential drivers... ...

[    6.594525] fuse init (API version 7.9)

[    6.618686] ACPI: SSDT CFFD0D0A, 08C4 (r1 HPQOEM  CPU_TM2        1
MSFT  100000E)
[    6.621481] BUG: unable to handle kernel NULL pointer dereference at
0000000000000858
[    6.625017] IP: [<ffffffff8025e282>] debug_mutex_add_waiter+0x32/0x80

[    6.625017] PGD 21a456067 PUD 21a45f067 PMD 0

[    6.625017] Oops: 0002 [1] SMP

[    6.625017] CPU 1

[    6.625017] Modules linked in: processor(+) fan thermal_sys fuse

[    6.625017] Pid: 1259, comm: modprobe Not tainted 2.6.27-rc3 #24

[    6.625017] RIP: 0010:[<ffffffff8025e282>]  [<ffffffff8025e282>]
debug_mutex_add_waiter+0x32/0x80

[    6.625017] RSP: 0018:ffff88021a073998  EFLAGS: 00010002

[    6.625017] RAX: 0000000000000000 RBX: ffff88021a0739d8 RCX:
0000000000000000

[    6.625017] RDX: 0000000000000001 RSI: ffff88021a0739d8 RDI:
ffffffffa0091a60

[    6.625017] RBP: ffff88021a0739b8 R08: ffffffff811deea0 R09:
ffff8800a6fdb000

[    6.625017] R10: ffffffffa008f524 R11: 0000000000000000 R12:
ffffffffa0091a60

[    6.625017] R13: ffff88021a072000 R14: ffff88021a9840a0 R15:
ffffffffa0091a98

[    6.625017] FS:  00007f40ce2de6e0(0000) GS:ffff88022fc02a00(0000)
knlGS:0000000000000000

[    6.625017] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b

[    6.625017] CR2: 0000000000000858 CR3: 000000022d86b000 CR4:
00000000000006e0

[    6.625017] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
0000000000000000

[    6.625017] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7:
0000000000000400

[    6.625017] Process modprobe (pid: 1259, threadinfo ffff88021a072000,
task ffff88021a9840a0)
[    6.625017] Stack:  0000000000000000 ffffffffa0091a60
0000000000000246 ffffffffa008f524

[    6.625017]  ffff88021a073a38 ffffffff8049f556 ffffffffa008f524
ffffffffa0091a18

[    6.625017]  ffff88021a0739d8 ffff88021a0739d8 1111111111111111
1111111111111111

[    6.625017] Call Trace:

[    6.625017]  [<ffffffffa008f524>] ? get_idr+0x44/0xa0 [thermal_sys]

[    6.625017]  [<ffffffff8049f556>] mutex_lock_nested+0xa6/0x250

[    6.625017]  [<ffffffffa008f524>] ? get_idr+0x44/0xa0 [thermal_sys]

[    6.625017]  [<ffffffff80363584>] ? idr_pre_get+0x44/0x90

[    6.625017]  [<ffffffffa008f524>] get_idr+0x44/0xa0 [thermal_sys]

[    6.625017]  [<ffffffffa008fe43>]
thermal_cooling_device_register+0x83/0x250 [thermal_sys]

[    6.625017]  [<ffffffffa019b2a3>] acpi_processor_start+0x64b/0x774
[processor]
[    6.625017]  [<ffffffff8031a90b>] ? __sysfs_add_one+0x6b/0xa0

[    6.625017]  [<ffffffff8031b9fc>] ? sysfs_do_create_link+0xbc/0x150

[    6.625017]  [<ffffffff803a7f1e>] acpi_start_single_object+0x2d/0x52

[    6.625017]  [<ffffffff803a9516>] acpi_device_probe+0x7e/0x92

[    6.625017]  [<ffffffff803dd3ab>] driver_probe_device+0x9b/0x1a0

[    6.625017]  [<ffffffff803dd536>] __driver_attach+0x86/0x90

[    6.625017]  [<ffffffff803dd4b0>] ? __driver_attach+0x0/0x90

[    6.625017]  [<ffffffff803dc8fd>] bus_for_each_dev+0x5d/0x90

[    6.625017]  [<ffffffff803dd1ec>] driver_attach+0x1c/0x20

[    6.625017]  [<ffffffff803dcf39>] bus_add_driver+0x1e9/0x260

[    6.625017]  [<ffffffffa0222000>] ? acpi_processor_init+0x0/0x107
[processor]

[    6.625017]  [<ffffffff803dd70f>] driver_register+0x5f/0x140

[    6.625017]  [<ffffffffa0222000>] ? acpi_processor_init+0x0/0x107
[processor]

[    6.625017]  [<ffffffff803a9826>] acpi_bus_register_driver+0x3e/0x40

[    6.625017]  [<ffffffffa0222094>] acpi_processor_init+0x94/0x107
[processor]

[    6.625017]  [<ffffffff80209040>] _stext+0x40/0x180

[    6.625017]  [<ffffffff802a88d1>] ? __vunmap+0xa1/0x110

[    6.625017]  [<ffffffff80267642>] sys_init_module+0x142/0x1dc0

[    6.625017]  [<ffffffff80367ad6>] ? __up_read+0x46/0xb0

[    6.625017]  [<ffffffff8048e530>] ? cpu_down+0x0/0x70

[    6.625017]  [<ffffffff8020c34b>] system_call_fastpath+0x16/0x1b

[    6.625017]

Re: Linux 2.6.27-rc3: kernel BUG at mm/vmalloc.c - bisected

[Björn Steinbrink]

On 2008.08.22 10:51:36 -0700, Andrew Morton wrote:
> On Fri, 22 Aug 2008 19:16:51 +0200 Petr Baudis <pasky@suse.cz> wrote:
> > On Fri, Aug 22, 2008 at 09:25:49AM -0700, Andrew Morton wrote:
> > > One (probably wrong) approach is to run
> > > 
> > > 	gitk 1c89ac55017f982355c7761e1c912c88c941483d
> > > 
> > > then peer at the output, work out which real commits were in that
> > > merge.
> > > 
> > > It looks like the merge ended with
> > > b1b135c8d619cb2c7045d6ee4e48375882518bb5 and started with
> > > 40c42076ebd362dc69210cccea101ac80b6d4bd4, so perhaps you can do
> > > 
> > > 	git bisect bad b1b135c8d619cb2c7045d6ee4e48375882518bb5
> > > 	git bisect good 40c42076ebd362dc69210cccea101ac80b6d4bd4
> > 
> > ...I don't quite get this - according to the bisection log,
> > 
> > 	# good: [b1b135c8d619cb2c7045d6ee4e48375882518bb5] fix spinlock recursion in hvc_console
> > 
> > and now you want to mark it as bad?
> 
> <what bisection log?>

Alan provided his bisection log as an attachment to the original bug
report.

> I assume that Alan's bisection search ended up saying that the merge
> commit (1c89ac55017f982355c7761e1c912c88c941483d) was the first bad
> commit.

Yep, and that's totally correct as far as bisect is concerned. The
parents of that merge commit are:
88fa08f67bee1a0c765237bdac106a32872f57d2
b1b135c8d619cb2c7045d6ee4e48375882518bb5

And Alan marked both of them as good.

So, unless Alan made a mistake during his bisection, each of the
branches is correct, but the merge did not lead to a correct result. So
while there were no textual conflicts, there were still incompatible
changes regarding the code semantics and compatibility was not restored
during the merge.

To get an overview over what got merged together you can can use
something like:
gitk --left-right 1c89ac55017^1...1c89ac55017^2

Which shows all commits that were on only one side of the merge, with
nice "arrows" that indicate from which side the commit is coming. The
conflict should be between one commit from the left and one commit from
the right side, obviously.

Björn

Re: Linux 2.6.27-rc3: kernel BUG at mm/vmalloc.c - bisected

[Alan D. Brunelle]

[-- Attachment #1: Type: text/plain, Size: 2223 bytes --]

Björn Steinbrink wrote:
> On 2008.08.22 10:51:36 -0700, Andrew Morton wrote:
>> On Fri, 22 Aug 2008 19:16:51 +0200 Petr Baudis <pasky@suse.cz> wrote:
>>> On Fri, Aug 22, 2008 at 09:25:49AM -0700, Andrew Morton wrote:
>>>> One (probably wrong) approach is to run
>>>>
>>>> 	gitk 1c89ac55017f982355c7761e1c912c88c941483d
>>>>
>>>> then peer at the output, work out which real commits were in that
>>>> merge.
>>>>
>>>> It looks like the merge ended with
>>>> b1b135c8d619cb2c7045d6ee4e48375882518bb5 and started with
>>>> 40c42076ebd362dc69210cccea101ac80b6d4bd4, so perhaps you can do
>>>>
>>>> 	git bisect bad b1b135c8d619cb2c7045d6ee4e48375882518bb5
>>>> 	git bisect good 40c42076ebd362dc69210cccea101ac80b6d4bd4
>>> ...I don't quite get this - according to the bisection log,
>>>
>>> 	# good: [b1b135c8d619cb2c7045d6ee4e48375882518bb5] fix spinlock recursion in hvc_console
>>>
>>> and now you want to mark it as bad?
>> <what bisection log?>
> 
> Alan provided his bisection log as an attachment to the original bug
> report.
> 
>> I assume that Alan's bisection search ended up saying that the merge
>> commit (1c89ac55017f982355c7761e1c912c88c941483d) was the first bad
>> commit.
> 
> Yep, and that's totally correct as far as bisect is concerned. The
> parents of that merge commit are:
> 88fa08f67bee1a0c765237bdac106a32872f57d2
> b1b135c8d619cb2c7045d6ee4e48375882518bb5
> 
> And Alan marked both of them as good.
> 
> So, unless Alan made a mistake during his bisection, each of the
> branches is correct, but the merge did not lead to a correct result. So
> while there were no textual conflicts, there were still incompatible
> changes regarding the code semantics and compatibility was not restored
> during the merge.

It's important to note that even if I did make a mistake during the
bisection process (and I certainly wouldn't discount that), recent
kernels still fail: but when I take out that commit from a recent
kernel, it fails.

I put in Andrew's suggested patch (to help find things), and now I
repeatedly get the problems in the attached log.

Not being an x86 knowledgeable person, I'm a bit concerned about the RSP
value?! (I enabled stack overflow checking, but that didn't stop things.)

Alan

[-- Attachment #2: new_prob.txt --]
[-- Type: text/plain, Size: 16450 bytes --]

Loading, please wait...
Begin: Loading essential drivers... ...
[    6.604300] fuse init (API version 7.9)
[    6.625529] ACPI: SSDT CFFD0D0A, 08C4 (r1 HPQOEM  CPU_TM2        1 MSFT  100000E)
[    6.631954] BUG: unable to handle kernel NULL pointer dereference at 0000000000000858
[    6.640583] IP: [<ffffffff8025e282>] debug_mutex_add_waiter+0x32/0x80
[    6.640583] PGD 21a84f067 PUD 21ad55067 PMD 0 
[    6.642421] Oops: 0002 [1] SMP 
[    6.642421] CPU 2 
[    6.642421] Modules linked in: processor(+) fan thermal_sys fuse
[    6.642421] Pid: 1259, comm: modprobe Not tainted 2.6.27-rc3 #26
[    6.642421] RIP: 0010:[<ffffffff8025e282>]  [<ffffffff8025e282>] debug_mutex_add_waiter+0x32/0x80
[    6.642421] RSP: 0018:ffff88021a1a5998  EFLAGS: 00010002
[    6.642421] RAX: 0000000000000000 RBX: ffff88021a1a59d8 RCX: 0000000000000000
[    6.642421] RDX: 0000000000000001 RSI: ffff88021a1a59d8 RDI: ffffffffa0091a60
[    6.642421] RBP: ffff88021a1a59b8 R08: ffffffff811deea0 R09: ffff8800a6fed000
[    6.642421] R10: ffffffffa008f524 R11: 0000000000000002 R12: ffffffffa0091a60
[    6.642421] R13: ffff88021a1a4000 R14: ffff88021a34c0a0 R15: ffffffffa0091a98
[    6.642421] FS:  00007f2c513606e0(0000) GS:ffff88022fc02e00(0000) knlGS:0000000000000000
[    6.642421] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[    6.642421] CR2: 0000000000000858 CR3: 000000021a8f7000 CR4: 00000000000006e0
[    6.642421] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[    6.642421] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[    6.642421] Process modprobe (pid: 1259, threadinfo ffff88021a1a4000, task ffff88021a34c0a0)
[    6.642421] Stack:  0000000000000000 ffffffffa0091a60 0000000000000246 ffffffffa008f524
[    6.642421]  ffff88021a1a5a38 ffffffff8049f556 ffffffffa008f524 ffffffffa0091a18
[    6.642421]  ffff88021a1a59d8 ffff88021a1a59d8 1111111111111111 1111111111111111
[    6.642421] Call Trace:
[    6.642421]  [<ffffffffa008f524>] ? get_idr+0x44/0xa0 [thermal_sys]
[    6.642421]  [<ffffffff8049f556>] mutex_lock_nested+0xa6/0x250
[    6.642421]  [<ffffffffa008f524>] ? get_idr+0x44/0xa0 [thermal_sys]
[    6.642421]  [<ffffffff80363584>] ? idr_pre_get+0x44/0x90
[    6.642421]  [<ffffffffa008f524>] get_idr+0x44/0xa0 [thermal_sys]
[    6.642421]  [<ffffffffa008fe43>] thermal_cooling_device_register+0x83/0x250 [thermal_sys]
[    6.642421]  [<ffffffffa019b2a3>] acpi_processor_start+0x64b/0x774 [processor]
[    6.642421]  [<ffffffff8031a90b>] ? __sysfs_add_one+0x6b/0xa0
[    6.642421]  [<ffffffff8031b9fc>] ? sysfs_do_create_link+0xbc/0x150
[    6.642421]  [<ffffffff803a7f1e>] acpi_start_single_object+0x2d/0x52
[    6.642421]  [<ffffffff803a9516>] acpi_device_probe+0x7e/0x92
[    6.642421]  [<ffffffff803dd3ab>] driver_probe_device+0x9b/0x1a0
[    6.642421]  [<ffffffff803dd536>] __driver_attach+0x86/0x90
[    6.642421]  [<ffffffff803dd4b0>] ? __driver_attach+0x0/0x90
[    6.642421]  [<ffffffff803dc8fd>] bus_for_each_dev+0x5d/0x90
[    6.642421]  [<ffffffff803dd1ec>] driver_attach+0x1c/0x20
[    6.642421]  [<ffffffff803dcf39>] bus_add_driver+0x1e9/0x260
[    6.642421]  [<ffffffffa0222000>] ? acpi_processor_init+0x0/0x107 [processor]
[    6.642421]  [<ffffffff803dd70f>] driver_register+0x5f/0x140
[    6.642421]  [<ffffffffa0222000>] ? acpi_processor_init+0x0/0x107 [processor]
[    6.642421]  [<ffffffff803a9826>] acpi_bus_register_driver+0x3e/0x40
[    6.642421]  [<ffffffffa0222094>] acpi_processor_init+0x94/0x107 [processor]
[    6.642421]  [<ffffffff80209040>] _stext+0x40/0x180
[    6.642421]  [<ffffffff802a88d1>] ? __vunmap+0xa1/0x110
[    6.642421]  [<ffffffff80267642>] sys_init_module+0x142/0x1dc0
[    6.642421]  [<ffffffff80367ad6>] ? __up_read+0x46/0xb0
[    6.642421]  [<ffffffff8048e530>] ? cpu_down+0x0/0x70
[    6.642421]  [<ffffffff8020c34b>] system_call_fastpath+0x16/0x1b
[    6.642421] 
[    6.642421] 
[    6.642421] Code: 20 48 89 5d e8 4c 89 65 f0 48 89 f3 4c 89 6d f8 8b 47 08 49 89 d5 49 89 fc 89 c2 25 ff ff 00 00 c1 ea 10 39 c2 74 1d 49 8b 4 
[    6.652421] RIP  [<ffffffff8025e282>] debug_mutex_add_waiter+0x32/0x80
[    6.652421]  RSP <ffff88021a1a5998>
[    6.652421] CR2: 0000000000000858
[    6.652421] ---[ end trace 53d26a9a6cd6aa38 ]---
[    6.991546] ------------[ cut here ]------------
[    6.996188] kernel BUG at kernel/sched.c:1155!
[    7.005292] invalid opcode: 0000 [2] SMP 
[    7.005292] CPU 2 
[    7.005292] Modules linked in: processor(+) fan thermal_sys fuse
[    7.011541] Pid: 1259, comm: modprobe Tainted: G      D   2.6.27-rc3 #26
[    7.011541] RIP: 0010:[<ffffffff8022cc2b>]  [<ffffffff8022cc2b>] resched_task+0x6b/0x70
[    7.011541] RSP: 0018:ffff88022f0efe98  EFLAGS: 00010046
[    7.011541] RAX: 00000000000006ee RBX: ffff880028092e00 RCX: ffffffff81046000
[    7.011541] RDX: 00000000000006ee RSI: ffff88021a34c0a0 RDI: ffffffff805ee660
[    7.011541] RBP: ffff88022f0efe98 R08: 0000000000000000 R09: 0000000000000001
[    7.011541] R10: 0000000000000000 R11: 0000000000000000 R12: ffff88021a34c0a0
[    7.011541] R13: 0000000000000001 R14: ffff880028092e98 R15: ffff88021a34c0d8
[    7.011541] FS:  00007f2c513606e0(0000) GS:ffff88022fc02e00(0000) knlGS:0000000000000000
[    7.011541] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[    7.011541] CR2: 0000000000000858 CR3: 000000021a8f7000 CR4: 00000000000006e0
[    7.011541] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[    7.011541] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[    7.011541] Process modprobe (pid: 1259, threadinfo ffff88021a1a4000, task ffff88021a34c0a0)
[    7.011541] Stack:  ffff88022f0efed8 ffffffff8023774f ffff88022f0efed8 ffff8800280b6e00
[    7.011541]  ffffffff80238510 ffff8800280b2b20 ffff8800280b2aa0 7fffffffffffffff
[    7.011541]  ffff88022f0efef8 ffffffff8023855d ffff8800280b2aa0 ffff8800280b7660
[    7.011541] Call Trace:
[    7.011541]  <IRQ>  [<ffffffff8023774f>] task_tick_fair+0xbf/0xd0
[    7.011541]  [<ffffffff80238510>] ? hrtick+0x0/0x90
[    7.011541]  [<ffffffff8023855d>] hrtick+0x4d/0x90
[    7.011541]  [<ffffffff80254f65>] __run_hrtimer+0x95/0xb0
[    7.011541]  [<ffffffff80255c4b>] hrtimer_interrupt+0x11b/0x190
[    7.011541]  [<ffffffff8021d643>] smp_apic_timer_interrupt+0x83/0xc0
[    7.011541]  [<ffffffff8020d0e6>] apic_timer_interrupt+0x66/0x70
[    7.011541]  <EOI>  [<ffffffff804a12eb>] ? _spin_unlock_irq+0x2b/0x30
[    7.011541]  [<ffffffff804a0b75>] ? __down_read+0xa5/0xb7
[    7.011541]  [<ffffffff8026fad5>] ? acct_collect+0x45/0x1d0
[    7.011541]  [<ffffffff8049fb57>] ? down_read+0x37/0x40
[    7.011541]  [<ffffffff8026fad5>] ? acct_collect+0x45/0x1d0
[    7.011541]  [<ffffffff8026fad5>] ? acct_collect+0x45/0x1d0
[    7.011541]  [<ffffffff8023f27d>] ? do_exit+0x18d/0x9b0
[    7.011541]  [<ffffffff803c7d19>] ? do_unblank_screen+0x19/0x130
[    7.011541]  [<ffffffff804a1a17>] ? oops_end+0x87/0x90
[    7.011541]  [<ffffffff804a3cd3>] ? do_page_fault+0x663/0x800
[    7.011541]  [<ffffffff804a15ed>] ? error_exit+0x0/0x9a
[    7.011541]  [<ffffffffa008f524>] ? get_idr+0x44/0xa0 [thermal_sys]
[    7.011541]  [<ffffffff8025e282>] ? debug_mutex_add_waiter+0x32/0x80
[    7.011541]  [<ffffffffa008f524>] ? get_idr+0x44/0xa0 [thermal_sys]
[    7.011541]  [<ffffffff8049f556>] ? mutex_lock_nested+0xa6/0x250
[    7.011541]  [<ffffffffa008f524>] ? get_idr+0x44/0xa0 [thermal_sys]
[    7.011541]  [<ffffffff80363584>] ? idr_pre_get+0x44/0x90
[    7.011541]  [<ffffffffa008f524>] ? get_idr+0x44/0xa0 [thermal_sys]
[    7.011541]  [<ffffffffa008fe43>] ? thermal_cooling_device_register+0x83/0x250 [thermal_sys]
[    7.011541]  [<ffffffffa019b2a3>] ? acpi_processor_start+0x64b/0x774 [processor]
[    7.011541]  [<ffffffff8031a90b>] ? __sysfs_add_one+0x6b/0xa0
[    7.011541]  [<ffffffff8031b9fc>] ? sysfs_do_create_link+0xbc/0x150
[    7.011541]  [<ffffffff803a7f1e>] ? acpi_start_single_object+0x2d/0x52
[    7.011541]  [<ffffffff803a9516>] ? acpi_device_probe+0x7e/0x92
[    7.011541]  [<ffffffff803dd3ab>] ? driver_probe_device+0x9b/0x1a0
[    7.011541]  [<ffffffff803dd536>] ? __driver_attach+0x86/0x90
[    7.011541]  [<ffffffff803dd4b0>] ? __driver_attach+0x0/0x90
[    7.011541]  [<ffffffff803dc8fd>] ? bus_for_each_dev+0x5d/0x90
[    7.011541]  [<ffffffff803dd1ec>] ? driver_attach+0x1c/0x20
[    7.011541]  [<ffffffff803dcf39>] ? bus_add_driver+0x1e9/0x260
[    7.011541]  [<ffffffffa0222000>] ? acpi_processor_init+0x0/0x107 [processor]
[    7.011541]  [<ffffffff803dd70f>] ? driver_register+0x5f/0x140
[    7.011541]  [<ffffffffa0222000>] ? acpi_processor_init+0x0/0x107 [processor]
[    7.011541]  [<ffffffff803a9826>] ? acpi_bus_register_driver+0x3e/0x40
[    7.011541]  [<ffffffffa0222094>] ? acpi_processor_init+0x94/0x107 [processor]
[    7.011541]  [<ffffffff80209040>] ? _stext+0x40/0x180
[    7.011541]  [<ffffffff802a88d1>] ? __vunmap+0xa1/0x110
[    7.011541]  [<ffffffff80267642>] ? sys_init_module+0x142/0x1dc0
[    7.011541]  [<ffffffff80367ad6>] ? __up_read+0x46/0xb0
[    7.011541]  [<ffffffff8048e530>] ? cpu_down+0x0/0x70
[    7.011541]  [<ffffffff8020c34b>] ? system_call_fastpath+0x16/0x1b
[    7.011541] 
[    7.011541] 
[    7.011541] Code: 8b 47 08 8b 50 1c 65 8b 04 25 24 00 00 00 39 c2 74 0d 0f ae f0 48 8b 47 08 f6 40 18 04 74 02 c9 c3 89 d7 ff 15 1f 7b 3c 00 c 
[    7.021541] RIP  [<ffffffff8022cc2b>] resched_task+0x6b/0x70
[    7.021541]  RSP <ffff88022f0efe98>
[    7.021541] ---[ end trace 53d26a9a6cd6aa38 ]---
[    7.021541] Kernel panic - not syncing: Aiee, killing interrupt handler!
[    7.021541] ------------[ cut here ]------------
[    7.021541] WARNING: at kernel/smp.c:328 smp_call_function_mask+0x25a/0x260()
[    7.021541] Modules linked in: processor(+) fan thermal_sys fuse
[    7.021541] Pid: 1259, comm: modprobe Tainted: G      D   2.6.27-rc3 #26
[    7.021541] 
[    7.021541] Call Trace:
[    7.021541]  <IRQ>  [<ffffffff8023bacf>] warn_on_slowpath+0x5f/0x80
[    7.021541]  [<ffffffff802650ca>] smp_call_function_mask+0x25a/0x260
[    7.021541]  [<ffffffff8036957d>] ? string+0x3d/0xd0
[    7.021541]  [<ffffffff80369a4b>] ? vsnprintf+0x43b/0x720
[    7.021541]  [<ffffffff8036957d>] ? string+0x3d/0xd0
[    7.021541]  [<ffffffff8036957d>] ? string+0x3d/0xd0
[    7.021541]  [<ffffffff80369a4b>] ? vsnprintf+0x43b/0x720
[    7.021541]  [<ffffffff80368d1e>] ? number+0x2ae/0x2d0
[    7.021541]  [<ffffffff80368d1e>] ? number+0x2ae/0x2d0
[    7.021541]  [<ffffffff80269e4d>] ? kallsyms_lookup+0x5d/0xa0
[    7.021541]  [<ffffffff80368d1e>] ? number+0x2ae/0x2d0
[    7.021541]  [<ffffffff80369a4b>] ? vsnprintf+0x43b/0x720
[    7.021541]  [<ffffffff80369d98>] ? sprintf+0x68/0x70
[    7.021541]  [<ffffffff8036957d>] ? string+0x3d/0xd0
[    7.021541]  [<ffffffff804a3f63>] ? __atomic_notifier_call_chain+0x83/0xa0
[    7.021541]  [<ffffffff804a3ee0>] ? __atomic_notifier_call_chain+0x0/0xa0
[    7.021541]  [<ffffffff804a0eb6>] ? _spin_unlock+0x26/0x30
[    7.021541]  [<ffffffff8021c470>] ? stop_this_cpu+0x0/0x30
[    7.021541]  [<ffffffff80265110>] smp_call_function+0x40/0x50
[    7.021541]  [<ffffffff8021c4f3>] native_smp_send_stop+0x23/0x40
[    7.021541]  [<ffffffff8023be1f>] panic+0xaf/0x190
[    7.021541]  [<ffffffff8023cc77>] ? printk+0x67/0x70
[    7.021541]  [<ffffffff8049f4a9>] ? mutex_unlock+0x9/0x10
[    7.021541]  [<ffffffff80256c91>] ? blocking_notifier_call_chain+0x11/0x20
[    7.021541]  [<ffffffff8023f9f9>] do_exit+0x909/0x9b0
[    7.021541]  [<ffffffff803c7d19>] ? do_unblank_screen+0x19/0x130
[    7.021541]  [<ffffffff804a1a17>] oops_end+0x87/0x90
[    7.021541]  [<ffffffff8020e08e>] die+0x5e/0x90
[    7.021541]  [<ffffffff804a1f20>] do_trap+0x130/0x150
[    7.021541]  [<ffffffff8020e662>] do_invalid_op+0x92/0xb0
[    7.021541]  [<ffffffff8022cc2b>] ? resched_task+0x6b/0x70
[    7.021541]  [<ffffffff8025788e>] ? sched_clock_cpu+0x14e/0x190
[    7.021541]  [<ffffffff804a15ed>] error_exit+0x0/0x9a
[    7.021541]  [<ffffffff8022cc2b>] ? resched_task+0x6b/0x70
[    7.021541]  [<ffffffff8023774f>] task_tick_fair+0xbf/0xd0
[    7.021541]  [<ffffffff80238510>] ? hrtick+0x0/0x90
[    7.021541]  [<ffffffff8023855d>] hrtick+0x4d/0x90
[    7.021541]  [<ffffffff80254f65>] __run_hrtimer+0x95/0xb0
[    7.021541]  [<ffffffff80255c4b>] hrtimer_interrupt+0x11b/0x190
[    7.021541]  [<ffffffff8021d643>] smp_apic_timer_interrupt+0x83/0xc0
[    7.021541]  [<ffffffff8020d0e6>] apic_timer_interrupt+0x66/0x70
[    7.021541]  <EOI>  [<ffffffff804a12eb>] ? _spin_unlock_irq+0x2b/0x30
[    7.021541]  [<ffffffff804a0b75>] ? __down_read+0xa5/0xb7
[    7.021541]  [<ffffffff8026fad5>] ? acct_collect+0x45/0x1d0
[    7.021541]  [<ffffffff8049fb57>] ? down_read+0x37/0x40
[    7.021541]  [<ffffffff8026fad5>] ? acct_collect+0x45/0x1d0
[    7.021541]  [<ffffffff8026fad5>] ? acct_collect+0x45/0x1d0
[    7.021541]  [<ffffffff8023f27d>] ? do_exit+0x18d/0x9b0
[    7.021541]  [<ffffffff803c7d19>] ? do_unblank_screen+0x19/0x130
[    7.021541]  [<ffffffff804a1a17>] ? oops_end+0x87/0x90
[    7.021541]  [<ffffffff804a3cd3>] ? do_page_fault+0x663/0x800
[    7.021541]  [<ffffffff804a15ed>] ? error_exit+0x0/0x9a
[    7.021541]  [<ffffffffa008f524>] ? get_idr+0x44/0xa0 [thermal_sys]
[    7.021541]  [<ffffffff8025e282>] ? debug_mutex_add_waiter+0x32/0x80
[    7.021541]  [<ffffffffa008f524>] ? get_idr+0x44/0xa0 [thermal_sys]
[    7.021541]  [<ffffffff8049f556>] ? mutex_lock_nested+0xa6/0x250
[    7.021541]  [<ffffffffa008f524>] ? get_idr+0x44/0xa0 [thermal_sys]
[    7.021541]  [<ffffffff80363584>] ? idr_pre_get+0x44/0x90
[    7.021541]  [<ffffffffa008f524>] ? get_idr+0x44/0xa0 [thermal_sys]
[    7.021541]  [<ffffffffa008fe43>] ? thermal_cooling_device_register+0x83/0x250 [thermal_sys]
[    7.021541]  [<ffffffffa019b2a3>] ? acpi_processor_start+0x64b/0x774 [processor]
[    7.021541]  [<ffffffff8031a90b>] ? __sysfs_add_one+0x6b/0xa0
[    7.021541]  [<ffffffff8031b9fc>] ? sysfs_do_create_link+0xbc/0x150
[    7.021541]  [<ffffffff803a7f1e>] ? acpi_start_single_object+0x2d/0x52
[    7.021541]  [<ffffffff803a9516>] ? acpi_device_probe+0x7e/0x92
[    7.021541]  [<ffffffff803dd3ab>] ? driver_probe_device+0x9b/0x1a0
[    7.021541]  [<ffffffff803dd536>] ? __driver_attach+0x86/0x90
[    7.021541]  [<ffffffff803dd4b0>] ? __driver_attach+0x0/0x90
[    7.021541]  [<ffffffff803dc8fd>] ? bus_for_each_dev+0x5d/0x90
[    7.021541]  [<ffffffff803dd1ec>] ? driver_attach+0x1c/0x20
[    7.021541]  [<ffffffff803dcf39>] ? bus_add_driver+0x1e9/0x260
[    7.021541]  [<ffffffffa0222000>] ? acpi_processor_init+0x0/0x107 [processor]
[    7.021541]  [<ffffffff803dd70f>] ? driver_register+0x5f/0x140
[    7.021541]  [<ffffffffa0222000>] ? acpi_processor_init+0x0/0x107 [processor]
[    7.021541]  [<ffffffff803a9826>] ? acpi_bus_register_driver+0x3e/0x40
[    7.021541]  [<ffffffffa0222094>] ? acpi_processor_init+0x94/0x107 [processor]
[    7.021541]  [<ffffffff80209040>] ? _stext+0x40/0x180
[    7.021541]  [<ffffffff802a88d1>] ? __vunmap+0xa1/0x110
[    7.021541]  [<ffffffff80267642>] ? sys_init_module+0x142/0x1dc0
[    7.021541]  [<ffffffff80367ad6>] ? __up_read+0x46/0xb0
[    7.021541]  [<ffffffff8048e530>] ? cpu_down+0x0/0x70
[    7.021541]  [<ffffffff8020c34b>] ? system_call_fastpath+0x16/0x1b
[    7.021541] 
[    7.021541] ---[ end trace 53d26a9a6cd6aa38 ]---
[    7.990001] BUG: spinlock lockup on CPU#0, swapper/0, ffff8800280b6e00
[    7.990001] Pid: 0, comm: swapper Tainted: G      D W 2.6.27-rc3 #26
[    7.990001] 
[    7.990001] Call Trace:
[    7.990001]  <IRQ>  [<ffffffff8036ee4e>] _raw_spin_lock+0x18e/0x1a0
[    7.990001]  [<ffffffff804a0fc4>] _spin_lock+0x34/0x40
[    7.990001]  [<ffffffff80230ed6>] ? sched_rt_period_timer+0x126/0x1e0
[    7.990001]  [<ffffffff80230ed6>] sched_rt_period_timer+0x126/0x1e0
[    7.990001]  [<ffffffff80230db0>] ? sched_rt_period_timer+0x0/0x1e0
[    7.990001]  [<ffffffff80254f65>] __run_hrtimer+0x95/0xb0
[    7.990001]  [<ffffffff80255c4b>] hrtimer_interrupt+0x11b/0x190
[    7.990001]  [<ffffffff8021d643>] smp_apic_timer_interrupt+0x83/0xc0
[    7.990001]  [<ffffffff8020d0e6>] apic_timer_interrupt+0x66/0x70
[    7.990001]  <EOI>  [<ffffffff802138a0>] ? mwait_idle+0x40/0x50
[    7.990001]  [<ffffffff8020a8d2>] ? enter_idle+0x22/0x30
[    7.990001]  [<ffffffff8020ae5c>] ? cpu_idle+0x4c/0xc0
[    7.990001]  [<ffffffff8048d7d1>] ? rest_init+0x61/0x70
[    7.990001] 

 CTRL-A Z for help |115200 8N1 | NOR | Minicom 2.3-rc | VT102 | Online 00:03                                                                     

Re: Linux 2.6.27-rc3: kernel BUG at mm/vmalloc.c - bisected

[Jeff King]

On Fri, Aug 22, 2008 at 09:37:30PM +0200, Björn Steinbrink wrote:

> Yep, and that's totally correct as far as bisect is concerned. The
> parents of that merge commit are:
> 88fa08f67bee1a0c765237bdac106a32872f57d2
> b1b135c8d619cb2c7045d6ee4e48375882518bb5
> 
> And Alan marked both of them as good.
> 
> So, unless Alan made a mistake during his bisection, each of the
> branches is correct, but the merge did not lead to a correct result. So
> while there were no textual conflicts, there were still incompatible
> changes regarding the code semantics and compatibility was not restored
> during the merge.

One thing that I have seen proposed (but never tried myself) is that you
can linearize the changes using "rebase -i" (or cherry-picking), and
then bisect that result. That is, given a history

A-B-C-D
 \   /
  E-F

where the merge "D" introduces the bug, you could try creating:

  A-B-C-E'-F'

and bisecting that. And you should know that C is good from your
previous bisection, but that F' probably is not, since it should be
textually the same as D (unless, of course, you had textual conflicts
during the rebase that you fixed up differently).

So in essence you are testing each of E and F, but based on the other
work. So you should be able to find the one patch that causes the
conflict. And depending on the conflict, you may get more information by
doing it the other way. I.e.,:

  A-E-F-B'-C'

-Peff

Re: Linux 2.6.27-rc3: kernel BUG at mm/vmalloc.c - bisected

[Alan D. Brunelle]

Alan D. Brunelle wrote:
> Björn Steinbrink wrote:
>> On 2008.08.22 10:51:36 -0700, Andrew Morton wrote:
>>> On Fri, 22 Aug 2008 19:16:51 +0200 Petr Baudis <pasky@suse.cz> wrote:
>>>> On Fri, Aug 22, 2008 at 09:25:49AM -0700, Andrew Morton wrote:
>>>>> One (probably wrong) approach is to run
>>>>>
>>>>> 	gitk 1c89ac55017f982355c7761e1c912c88c941483d
>>>>>
>>>>> then peer at the output, work out which real commits were in that
>>>>> merge.
>>>>>
>>>>> It looks like the merge ended with
>>>>> b1b135c8d619cb2c7045d6ee4e48375882518bb5 and started with
>>>>> 40c42076ebd362dc69210cccea101ac80b6d4bd4, so perhaps you can do
>>>>>
>>>>> 	git bisect bad b1b135c8d619cb2c7045d6ee4e48375882518bb5
>>>>> 	git bisect good 40c42076ebd362dc69210cccea101ac80b6d4bd4
>>>> ...I don't quite get this - according to the bisection log,
>>>>
>>>> 	# good: [b1b135c8d619cb2c7045d6ee4e48375882518bb5] fix spinlock recursion in hvc_console
>>>>
>>>> and now you want to mark it as bad?
>>> <what bisection log?>
>> Alan provided his bisection log as an attachment to the original bug
>> report.
>>
>>> I assume that Alan's bisection search ended up saying that the merge
>>> commit (1c89ac55017f982355c7761e1c912c88c941483d) was the first bad
>>> commit.
>> Yep, and that's totally correct as far as bisect is concerned. The
>> parents of that merge commit are:
>> 88fa08f67bee1a0c765237bdac106a32872f57d2
>> b1b135c8d619cb2c7045d6ee4e48375882518bb5
>>
>> And Alan marked both of them as good.
>>
>> So, unless Alan made a mistake during his bisection, each of the
>> branches is correct, but the merge did not lead to a correct result. So
>> while there were no textual conflicts, there were still incompatible
>> changes regarding the code semantics and compatibility was not restored
>> during the merge.
> 
> It's important to note that even if I did make a mistake during the
> bisection process (and I certainly wouldn't discount that), recent
> kernels still fail: but when I take out that commit from a recent
> kernel, it fails.

s/fails/succeeds/

> 
> I put in Andrew's suggested patch (to help find things), and now I
> repeatedly get the problems in the attached log.
> 
> Not being an x86 knowledgeable person, I'm a bit concerned about the RSP
> value?! (I enabled stack overflow checking, but that didn't stop things.)
> 
> Alan
> 

Re: Linux 2.6.27-rc3: kernel BUG at mm/vmalloc.c - bisected

[Junio C Hamano]

Andrew Morton <akpm@linux-foundation.org> writes:

> On Fri, 22 Aug 2008 19:16:51 +0200
> Petr Baudis <pasky@suse.cz> wrote:
>
>> On Fri, Aug 22, 2008 at 09:25:49AM -0700, Andrew Morton wrote:
> ...
>> > urgh, it's irritating when git-bisect directs you to a merge commit - it
>> > hasn't done it for me for ages.
>> 
>> Hmm, but doesn't that happen only when it's actually really the merge
>> commit that introduces the bug? Both parents of the merge commit were
>> marked as good by the user, so...
>
> A merge commit doesn't contain any kernel changes?  It's the individual
> commits (aka "patches") which were in that merge which broke stuff. 
> Confused.
>
> We're trying to dive inside that merge commit to find out which of the
> real commits caused the regression.

You may find neither parents were buggy, but the result of the merge is.

A trivial example is when one branch changes the semantics of an existing
function and converts all the call sites to the updated semantics, while
the other branch adds a new call site that still relies on the old
behaviour of that function.  The merge most likely won't textually
conflict, and neither git merge nor quilt patch would report conflicts,
but the end result is that the new call site added by the latter branch
now gets an unexpected outcome from the function and can misbehave.  You
cannot blame the breakage to either branch for such a breakage.

Re: Linux 2.6.27-rc3: kernel BUG at mm/vmalloc.c - bisected

[Andrew Morton]

On Fri, 22 Aug 2008 14:05:21 -0700
Junio C Hamano <gitster@pobox.com> wrote:

> Andrew Morton <akpm@linux-foundation.org> writes:
> 
> > On Fri, 22 Aug 2008 19:16:51 +0200
> > Petr Baudis <pasky@suse.cz> wrote:
> >
> >> On Fri, Aug 22, 2008 at 09:25:49AM -0700, Andrew Morton wrote:
> > ...
> >> > urgh, it's irritating when git-bisect directs you to a merge commit - it
> >> > hasn't done it for me for ages.
> >> 
> >> Hmm, but doesn't that happen only when it's actually really the merge
> >> commit that introduces the bug? Both parents of the merge commit were
> >> marked as good by the user, so...
> >
> > A merge commit doesn't contain any kernel changes?  It's the individual
> > commits (aka "patches") which were in that merge which broke stuff. 
> > Confused.
> >
> > We're trying to dive inside that merge commit to find out which of the
> > real commits caused the regression.
> 
> You may find neither parents were buggy, but the result of the merge is.
> 
> A trivial example is when one branch changes the semantics of an existing
> function and converts all the call sites to the updated semantics, while
> the other branch adds a new call site that still relies on the old
> behaviour of that function.  The merge most likely won't textually
> conflict, and neither git merge nor quilt patch would report conflicts,
> but the end result is that the new call site added by the latter branch
> now gets an unexpected outcome from the function and can misbehave.  You
> cannot blame the breakage to either branch for such a breakage.

Sure, but

a) whoever added a change like that without also causing the build to
   break is slappable and

b) there are now two commits (one in each branch) either one of
   which, when applied on top of the other branch will introduce the
   regression.

   That's useful infomation, but we don't know how to get
   git-bisect to give it to us.


It's pretty simple.  If git-bisect tells us that the regression was
introduced by a merge commit, we want to perform a bisection within
that merge's individual commits.

Re: Linux 2.6.27-rc3: kernel BUG at mm/vmalloc.c - bisected

[Jeff King]

On Fri, Aug 22, 2008 at 02:16:51PM -0700, Andrew Morton wrote:

> Sure, but
> 
> a) whoever added a change like that without also causing the build to
>    break is slappable and

Well, the point is to find out _who_ to slap. ;)

> b) there are now two commits (one in each branch) either one of
>    which, when applied on top of the other branch will introduce the
>    regression.
> 
>    That's useful infomation, but we don't know how to get
>    git-bisect to give it to us.
>
> It's pretty simple.  If git-bisect tells us that the regression was
> introduced by a merge commit, we want to perform a bisection within
> that merge's individual commits.

I mentioned linearizing the two lines of development in another
message in this thread. But there are, of course, two linearizations,
neither of which might be possible to generate automatically, and one
might give you a better answer than the other.

-Peff

Re: Linux 2.6.27-rc3: kernel BUG at mm/vmalloc.c - bisected

[Björn Steinbrink]

On 2008.08.22 14:16:51 -0700, Andrew Morton wrote:
> It's pretty simple.  If git-bisect tells us that the regression was
> introduced by a merge commit, we want to perform a bisection within
> that merge's individual commits.

bisect already did that. It asked for the left side and the right side,
both were good. What you can still do is creating a _new_ history where
the commits are not in parallel but linearized, like Jeff described. But
that's (in general) not a trivial task as you need to reapply individual
commit patches which can cause conflicts that were already solved in the
existing merges to show up again. Or, it can even produce new conflicts,
for example when a commit on one side was reverted before the merge
happened. In that case, you get into a state with changes that were
never visible before.

Björn