Re: [BUG] no boot with 2.6.21-rc3 and later

Re: [BUG] no boot with 2.6.21-rc3 and later

[Bob Tracy]

I originally wrote:
> Platform is a Dell CPxJ 650GT notebook.  Attempts to boot 2.6.21-rc3
> and -rc4 produce the following console output:
> 
> (apparently normal boot)
> ...
> Uniform Multi-Platform E-IDE driver Revision: 7.00alpha2
> ide: Assuming 33MHz system bus speed for PIO modes; override with idebus=xx
> PIIX4: IDE controller at PCI slot 0000:00:07.1
> PIIX4: chipset revision 1
> PIIX4: not 100% native mode: will probe irqs later
>     ide0: BM-DMA at 0x0860-0x0867, BIOS settings: hda:DMA, hdb:pio
>     ide1: BM-DMA at 0x0868-0x086f, BIOS settings: hdc:DMA, hdd:pio
> (hangs here)
> 
> The next two lines of output for 2.6.21-rc2 and earlier are:
> Probing IDE interface ide0...
> hda: TOSHIBA MK3017GAP, ATA DISK drive

After nine iterations of git-bisect --> compile --> reboot over the past
two days, the commit that caused this problem has been identified.

6bb74df481223731af6c7e0ff3adb31f6442cfcd is first bad commit
commit 6bb74df481223731af6c7e0ff3adb31f6442cfcd
Author: john stultz <johnstul@us.ibm.com>
Date:   Mon Mar 5 00:30:50 2007 -0800

    [PATCH] clocksource init adjustments (fix bug #7426)

    This patch resolves the issue found here:
    http://bugme.osdl.org/show_bug.cgi?id=7426

    The basic summary is:
    (omitted for brevity)

    Signed-off-by: John Stultz <johnstul@us.ibm.com>
    Cc: Thomas Gleixner <tglx@linutronix.de>
    Cc: Ingo Molnar <mingo@elte.hu>
    Cc: "David S. Miller" <davem@davemloft.net>
    Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
    Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>

:040000 040000 b259cf01f6ad791a0f58c9956e6e0111cb9aa310 0e8fbf31c0bae9ca6039fc7781b53c12f7a67592 M      arch
:040000 040000 19bd5cf3c55c6b7947ec66b68eab643c99f15bb8 ef8fda0445287e34eeb466985901ca64191b6539 M      drivers
:040000 040000 4f84ad7bdf509ed83efd95d174796f1a6a948b0e 51624663be2fdce8aaeb3b49e188520aa3735d90 M      include
:040000 040000 ef82986d4590fed84b9a128da90e65f7e43f43a9 778d0cf7273d7514b436ca4e91a6948f34694972 M      kernel

-- 
-----------------------------------------------------------------------
Bob Tracy                   WTO + WIPO = DMCA? http://www.anti-dmca.org
rct@frus.com
-----------------------------------------------------------------------

Re: [BUG] no boot with 2.6.21-rc3 and later

[Bartlomiej Zolnierkiewicz]

Thanks for narrowing it down, I'm adding John to cc:.

On Wednesday 21 March 2007, Bob Tracy wrote:
> I originally wrote:
> > Platform is a Dell CPxJ 650GT notebook.  Attempts to boot 2.6.21-rc3
> > and -rc4 produce the following console output:
> > 
> > (apparently normal boot)
> > ...
> > Uniform Multi-Platform E-IDE driver Revision: 7.00alpha2
> > ide: Assuming 33MHz system bus speed for PIO modes; override with idebus=xx
> > PIIX4: IDE controller at PCI slot 0000:00:07.1
> > PIIX4: chipset revision 1
> > PIIX4: not 100% native mode: will probe irqs later
> >     ide0: BM-DMA at 0x0860-0x0867, BIOS settings: hda:DMA, hdb:pio
> >     ide1: BM-DMA at 0x0868-0x086f, BIOS settings: hdc:DMA, hdd:pio
> > (hangs here)
> > 
> > The next two lines of output for 2.6.21-rc2 and earlier are:
> > Probing IDE interface ide0...
> > hda: TOSHIBA MK3017GAP, ATA DISK drive
> 
> After nine iterations of git-bisect --> compile --> reboot over the past
> two days, the commit that caused this problem has been identified.
> 
> 6bb74df481223731af6c7e0ff3adb31f6442cfcd is first bad commit
> commit 6bb74df481223731af6c7e0ff3adb31f6442cfcd
> Author: john stultz <johnstul@us.ibm.com>
> Date:   Mon Mar 5 00:30:50 2007 -0800
> 
>     [PATCH] clocksource init adjustments (fix bug #7426)
> 
>     This patch resolves the issue found here:
>     http://bugme.osdl.org/show_bug.cgi?id=7426
> 
>     The basic summary is:
>     (omitted for brevity)
> 
>     Signed-off-by: John Stultz <johnstul@us.ibm.com>
>     Cc: Thomas Gleixner <tglx@linutronix.de>
>     Cc: Ingo Molnar <mingo@elte.hu>
>     Cc: "David S. Miller" <davem@davemloft.net>
>     Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
>     Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
> 
> :040000 040000 b259cf01f6ad791a0f58c9956e6e0111cb9aa310 0e8fbf31c0bae9ca6039fc7781b53c12f7a67592 M      arch
> :040000 040000 19bd5cf3c55c6b7947ec66b68eab643c99f15bb8 ef8fda0445287e34eeb466985901ca64191b6539 M      drivers
> :040000 040000 4f84ad7bdf509ed83efd95d174796f1a6a948b0e 51624663be2fdce8aaeb3b49e188520aa3735d90 M      include
> :040000 040000 ef82986d4590fed84b9a128da90e65f7e43f43a9 778d0cf7273d7514b436ca4e91a6948f34694972 M      kernel
> 
> -- 
> -----------------------------------------------------------------------
> Bob Tracy                   WTO + WIPO = DMCA? http://www.anti-dmca.org
> rct@frus.com
> -----------------------------------------------------------------------

Re: [BUG] no boot with 2.6.21-rc3 and later

[john stultz]

On Wed, 2007-03-21 at 21:18 +0100, Bartlomiej Zolnierkiewicz wrote:
> Thanks for narrowing it down, I'm adding John to cc:.
> 
> On Wednesday 21 March 2007, Bob Tracy wrote:
> > I originally wrote:
> > > Platform is a Dell CPxJ 650GT notebook.  Attempts to boot 2.6.21-rc3
> > > and -rc4 produce the following console output:
> > > 
> > > (apparently normal boot)
> > > ...
> > > Uniform Multi-Platform E-IDE driver Revision: 7.00alpha2
> > > ide: Assuming 33MHz system bus speed for PIO modes; override with idebus=xx
> > > PIIX4: IDE controller at PCI slot 0000:00:07.1
> > > PIIX4: chipset revision 1
> > > PIIX4: not 100% native mode: will probe irqs later
> > >     ide0: BM-DMA at 0x0860-0x0867, BIOS settings: hda:DMA, hdb:pio
> > >     ide1: BM-DMA at 0x0868-0x086f, BIOS settings: hdc:DMA, hdd:pio
> > > (hangs here)
> > > 
> > > The next two lines of output for 2.6.21-rc2 and earlier are:
> > > Probing IDE interface ide0...
> > > hda: TOSHIBA MK3017GAP, ATA DISK drive


Hmmm. Are you able to capture a full dmesg? If so could you send it to
me? Also, does booting w/ "clocksource=jiffies" change the behavior?

thanks
-john

Re: [BUG] no boot with 2.6.21-rc3 and later

[Bob Tracy]

john stultz wrote:
> Hmmm. Are you able to capture a full dmesg?

For the "it hangs" case, short of either trying to boot with the
console I/O redirected to a serial port or hooking up a printer to
the console and scanning in the resulting output, I don't think so.
There's too much output prior to the hang point to make manual
transcription feasible.

> Also, does booting w/ "clocksource=jiffies" change the behavior?

I'll build a post-"bad"-commit kernel later this evening and let you
know.  Unfortunately, real life has to intrude for a few hours :-).
If specifying the clocksource allows for a normal bootup, would you
like to see the corresponding dmesg output?

-- 
-----------------------------------------------------------------------
Bob Tracy                   WTO + WIPO = DMCA? http://www.anti-dmca.org
rct@frus.com
-----------------------------------------------------------------------

Re: [BUG] no boot with 2.6.21-rc3 and later

[john stultz]

On Wed, 2007-03-21 at 16:00 -0600, Bob Tracy wrote:
> john stultz wrote:
> > Hmmm. Are you able to capture a full dmesg?
> 
> For the "it hangs" case, short of either trying to boot with the
> console I/O redirected to a serial port or hooking up a printer to
> the console and scanning in the resulting output, I don't think so.
> There's too much output prior to the hang point to make manual
> transcription feasible.

Hmm. You could give netconsole a try(see
Documentation/networking/netconsole.txt), although if the hang is before
that inits you might be out of luck.

> > Also, does booting w/ "clocksource=jiffies" change the behavior?
> 
> I'll build a post-"bad"-commit kernel later this evening and let you
> know.  Unfortunately, real life has to intrude for a few hours :-).

No problem. When you can get to it. :)

Also trying booting w/ "notsc" would be a useful data point.

> If specifying the clocksource allows for a normal bootup, would you
> like to see the corresponding dmesg output?

No, although a pre-bad-commit dmesg would help.

thanks
-john

Re: [BUG] no boot with 2.6.21-rc3 and later

[Bob Tracy]

john stultz wrote:
> > > Also, does booting w/ "clocksource=jiffies" change the behavior?

Works fine with 2.6.21-rc4.  I'm running on that kernel as I type this.

> Also trying booting w/ "notsc" would be a useful data point.

Boot hangs at the point indicated in my original message.  I *did*
notice the blurb in the console messages about the pit clocksource
being selected/used.  There was also a complaint about it being
unstable, with a negative delta.  I think these messages are consistent
with the pre-bad-commit case other than where they appear in the boot
messages.

> (...) a pre-bad-commit dmesg would help.

Sent under separate cover to John.

-- 
-----------------------------------------------------------------------
Bob Tracy                   WTO + WIPO = DMCA? http://www.anti-dmca.org
rct@frus.com
-----------------------------------------------------------------------

Re: [BUG] no boot with 2.6.21-rc3 and later

[john stultz]

On Wed, 2007-03-21 at 21:45 -0600, Bob Tracy wrote:
> john stultz wrote:
> > > > Also, does booting w/ "clocksource=jiffies" change the behavior?
> 
> Works fine with 2.6.21-rc4.  I'm running on that kernel as I type this.
> 
> > Also trying booting w/ "notsc" would be a useful data point.
> 
> Boot hangs at the point indicated in my original message.  I *did*
> notice the blurb in the console messages about the pit clocksource
> being selected/used.  There was also a complaint about it being
> unstable, with a negative delta.  I think these messages are consistent
> with the pre-bad-commit case other than where they appear in the boot
> messages.
> 
> > (...) a pre-bad-commit dmesg would help.
> 
> Sent under separate cover to John.

Yea. From looking at your boot log, it does look like the PIT
clocksource is being used. 

This does point to an issue w/ that clocksource, however I noticed its
getting selected over the ACPI PM timer, because you have the PIIX4 bug.
That bug drops the ACPI PM timers rating to the same as the PIT, so
that's why you're getting the PIT.

Realistically, the PIT is much slower then even the triple read ACPI PM,
so the de-ranking code is probably dropping it too far.

Try this patch and let me know if it does the right thing. Although I do
still need to dig a bit on the PIT hang issue.

-john


Don't drop ACPI PM quite so low if we see the PIIX4 bug.

Signed-off-by: John Stultz <johnstul@us.ibm.com>

diff --git a/drivers/clocksource/acpi_pm.c b/drivers/clocksource/acpi_pm.c
index d42060e..5ac309e 100644
--- a/drivers/clocksource/acpi_pm.c
+++ b/drivers/clocksource/acpi_pm.c
@@ -90,7 +90,7 @@ __setup("acpi_pm_good", acpi_pm_good_set
 static inline void acpi_pm_need_workaround(void)
 {
 	clocksource_acpi_pm.read = acpi_pm_read_slow;
-	clocksource_acpi_pm.rating = 110;
+	clocksource_acpi_pm.rating = 120;
 }
 
 /*

Re: [BUG] no boot with 2.6.21-rc3 and later

[Bob Tracy]

john stultz wrote:
> Try this patch and let me know if it does the right thing.

Will do.  I'll report back in a few hours.

> Although I do
> still need to dig a bit on the PIT hang issue.

Any chance this might be related to the APIC issues currently being
discussed in other threads?

-- 
-----------------------------------------------------------------------
Bob Tracy                   WTO + WIPO = DMCA? http://www.anti-dmca.org
rct@frus.com
-----------------------------------------------------------------------

Re: [BUG] no boot with 2.6.21-rc3 and later

[john stultz]

On Thu, 2007-03-22 at 13:14 -0600, Bob Tracy wrote:
> john stultz wrote:
> > Try this patch and let me know if it does the right thing.
> 
> Will do.  I'll report back in a few hours.
> 
> > Although I do
> > still need to dig a bit on the PIT hang issue.
> 
> Any chance this might be related to the APIC issues currently being
> discussed in other threads?

Hmmm. Good thought, I'll have to look into it. It could be that if the
PIT is disabled in favor of the local apic, we'll have to make sure its
not being used as a clocksource.

-john

Re: [BUG] no boot with 2.6.21-rc3 and later

[Bob Tracy]

john stultz wrote:
> On Thu, 2007-03-22 at 13:14 -0600, Bob Tracy wrote:
> > john stultz wrote:
> > > Try this patch and let me know if it does the right thing.
> > 
> > Will do.  I'll report back in a few hours.
> > 
> > > Although I do
> > > still need to dig a bit on the PIT hang issue.
> > 
> > Any chance this might be related to the APIC issues currently being
> > discussed in other threads?
> 
> Hmmm. Good thought, I'll have to look into it. It could be that if the
> PIT is disabled in favor of the local apic, we'll have to make sure its
> not being used as a clocksource.

According to the dmesg output, the local apic is disabled in the BIOS on
this system, and I'm not booting with "lapic" to enable it.

Anyway, the "few hours" turned out to be more like a few minutes :-).
The patch worked as far as selecting/installing the acpi_pm clocksource
instead of the pit clocksource, and there was no hang during boot.  If
you would like to see the dmesg output, let me know and I'll forward it
to you in a separate message.

-- 
-----------------------------------------------------------------------
Bob Tracy                   WTO + WIPO = DMCA? http://www.anti-dmca.org
rct@frus.com
-----------------------------------------------------------------------

Re: [BUG] no boot with 2.6.21-rc3 and later

[john stultz]

On Thu, 2007-03-22 at 13:39 -0600, Bob Tracy wrote:
> john stultz wrote:
> > On Thu, 2007-03-22 at 13:14 -0600, Bob Tracy wrote:
> > > john stultz wrote:
> > > > Try this patch and let me know if it does the right thing.
> > > 
> > > Will do.  I'll report back in a few hours.
> > > 
> > > > Although I do
> > > > still need to dig a bit on the PIT hang issue.
> > > 
> > > Any chance this might be related to the APIC issues currently being
> > > discussed in other threads?
> > 
> > Hmmm. Good thought, I'll have to look into it. It could be that if the
> > PIT is disabled in favor of the local apic, we'll have to make sure its
> > not being used as a clocksource.
> 
> According to the dmesg output, the local apic is disabled in the BIOS on
> this system, and I'm not booting with "lapic" to enable it.
> 
> Anyway, the "few hours" turned out to be more like a few minutes :-).
> The patch worked as far as selecting/installing the acpi_pm clocksource
> instead of the pit clocksource, and there was no hang during boot.  If
> you would like to see the dmesg output, let me know and I'll forward it
> to you in a separate message.

Great to hear! Please do send me the dmesg output so I can look to
narrow down what's happening w/ the PIT.

I'll also go ahead and push that acpi_pm change to akpm.

thanks
-john

Re: [BUG] no boot with 2.6.21-rc3 and later

[Bob Tracy]

john stultz wrote:
> Great to hear! Please do send me the dmesg output so I can look to
> narrow down what's happening w/ the PIT.

Sent under separate cover...

-- 
-----------------------------------------------------------------------
Bob Tracy                   WTO + WIPO = DMCA? http://www.anti-dmca.org
rct@frus.com
-----------------------------------------------------------------------

Re: [BUG] no boot with 2.6.21-rc3 and later

[Thomas Gleixner]

On Thu, 2007-03-22 at 12:25 -0700, john stultz wrote:
> On Thu, 2007-03-22 at 13:14 -0600, Bob Tracy wrote:
> > john stultz wrote:
> > > Try this patch and let me know if it does the right thing.
> > 
> > Will do.  I'll report back in a few hours.
> > 
> > > Although I do
> > > still need to dig a bit on the PIT hang issue.
> > 
> > Any chance this might be related to the APIC issues currently being
> > discussed in other threads?
> 
> Hmmm. Good thought, I'll have to look into it. It could be that if the
> PIT is disabled in favor of the local apic, we'll have to make sure its
> not being used as a clocksource.

Ouch, yes. That's fatal and can happen. Not sure, what to do about that.

	tglx

[PATCH] correct slow acpi_pm rating (fixes no boot with 2.6.21-rc3 and later)

[john stultz]

Tested by Bob Tracy.

On Wed, 2007-03-21 at 21:45 -0600, Bob Tracy wrote:
> john stultz wrote:
> > > > Also, does booting w/ "clocksource=jiffies" change the behavior?
> 
> Works fine with 2.6.21-rc4.  I'm running on that kernel as I type this.
> 
> > Also trying booting w/ "notsc" would be a useful data point.
> 
> Boot hangs at the point indicated in my original message.  I *did*
> notice the blurb in the console messages about the pit clocksource
> being selected/used.  There was also a complaint about it being
> unstable, with a negative delta.  I think these messages are consistent
> with the pre-bad-commit case other than where they appear in the boot
> messages.
> 
> > (...) a pre-bad-commit dmesg would help.
> 
> Sent under separate cover to John.

Yea. From looking at your boot log, it does look like the PIT
clocksource is being used. 

This does point to an issue w/ that clocksource, however I noticed its
getting selected over the ACPI PM timer, because you have the PIIX4 bug.
That bug drops the ACPI PM timers rating to the same as the PIT, so
that's why you're getting the PIT.

Realistically, the PIT is much slower then even the triple read ACPI PM,
so the de-ranking code is probably dropping it too far.

Try this patch and let me know if it does the right thing. Although I do
still need to dig a bit on the PIT hang issue.

-john


Don't drop ACPI PM quite so low if we see the PIIX4 bug.

Signed-off-by: John Stultz <johnstul@us.ibm.com>

diff --git a/drivers/clocksource/acpi_pm.c b/drivers/clocksource/acpi_pm.c
index d42060e..5ac309e 100644
--- a/drivers/clocksource/acpi_pm.c
+++ b/drivers/clocksource/acpi_pm.c
@@ -90,7 +90,7 @@ __setup("acpi_pm_good", acpi_pm_good_set
 static inline void acpi_pm_need_workaround(void)
 {
 	clocksource_acpi_pm.read = acpi_pm_read_slow;
-	clocksource_acpi_pm.rating = 110;
+	clocksource_acpi_pm.rating = 120;
 }
 
 /*

Re: [BUG] no boot with 2.6.21-rc3 and later

[Bob Tracy]

Jiri wrote:
> Looks like it's related to some change in drivers/ide. As there have been 
> only 13 patches in this area between rc2 and rc3, it should take only 3 or 
> 4 reboots to figure the offending patch using git-bisect - could you 
> please give it a try?

I applied all of the 2.6.21-rc2-rc3 incremental patch except for the
portion applicable to "drivers/ide" files.  The problem seems to be
elsewhere: 2.6.21-rc3 minus the drivers/ide changes still hangs at the
same spot during the boot process.

Any ideas where to look next?  Thanks!

-- 
-----------------------------------------------------------------------
Bob Tracy                   WTO + WIPO = DMCA? http://www.anti-dmca.org
rct@frus.com
-----------------------------------------------------------------------

Re: [BUG] no boot with 2.6.21-rc3 and later

[Jiri Kosina]

On Mon, 19 Mar 2007, Bob Tracy wrote:

> I applied all of the 2.6.21-rc2-rc3 incremental patch except for the 
> portion applicable to "drivers/ide" files.  The problem seems to be 
> elsewhere: 2.6.21-rc3 minus the drivers/ide changes still hangs at the 
> same spot during the boot process. Any ideas where to look next?  
> Thanks!

This really is an excellent opportunity for git-bisect, as you are 
trivially and immediately able to reproduce the problem. There are not 
that many patches between rc2 and rc3, so it would require only a few 
reboots to identify offending patch, and git-bisect is really trivial to 
use - see the manpage for the example on the process of bisecting.

-- 
Jiri Kosina

[BUG] no boot with 2.6.21-rc3 and later

[Bob Tracy]

Platform is a Dell CPxJ 650GT notebook.  Attempts to boot 2.6.21-rc3
and -rc4 produce the following console output:

(apparently normal boot)
...
Uniform Multi-Platform E-IDE driver Revision: 7.00alpha2
ide: Assuming 33MHz system bus speed for PIO modes; override with idebus=xx
PIIX4: IDE controller at PCI slot 0000:00:07.1
PIIX4: chipset revision 1
PIIX4: not 100% native mode: will probe irqs later
    ide0: BM-DMA at 0x0860-0x0867, BIOS settings: hda:DMA, hdb:pio
    ide1: BM-DMA at 0x0868-0x086f, BIOS settings: hdc:DMA, hdd:pio
(hangs here)

The next two lines of output for 2.6.21-rc2 and earlier are:
Probing IDE interface ide0...
hda: TOSHIBA MK3017GAP, ATA DISK drive

Any idea what's going on here?

-- 
-----------------------------------------------------------------------
Bob Tracy                   WTO + WIPO = DMCA? http://www.anti-dmca.org
rct@frus.com
-----------------------------------------------------------------------

Re: [BUG] no boot with 2.6.21-rc3 and later

[Jiri Kosina]

(added Bartlomiej to CC)

On Sat, 17 Mar 2007, Bob Tracy wrote:

> Uniform Multi-Platform E-IDE driver Revision: 7.00alpha2
> ide: Assuming 33MHz system bus speed for PIO modes; override with idebus=xx
> PIIX4: IDE controller at PCI slot 0000:00:07.1
> PIIX4: chipset revision 1
> PIIX4: not 100% native mode: will probe irqs later
>     ide0: BM-DMA at 0x0860-0x0867, BIOS settings: hda:DMA, hdb:pio
>     ide1: BM-DMA at 0x0868-0x086f, BIOS settings: hdc:DMA, hdd:pio
> (hangs here)
> The next two lines of output for 2.6.21-rc2 and earlier are:
> Probing IDE interface ide0...
> hda: TOSHIBA MK3017GAP, ATA DISK drive

Looks like it's related to some change in drivers/ide. As there have been 
only 13 patches in this area between rc2 and rc3, it should take only 3 or 
4 reboots to figure the offending patch using git-bisect - could you 
please give it a try?

Thanks,

-- 
Jiri Kosina