Linus' tree broken?

Linus' tree broken?

[Jes Sorensen]

Hi,

Have anyone else had seen issues with the Linus git tree failing to
boot? I'm trying a pull as of this morning and it seems to hang when
it's brining up the ethernet devices.

It's not locking up, just the boot scripts never continues - happens
both on SLES10 and FC7 roots :(

Just wanted to check before spending real time on it.

Cheers,
Jes

RE: Linus' tree broken?

[Luck, Tony]

> Have anyone else had seen issues with the Linus git tree failing to
> boot? I'm trying a pull as of this morning and it seems to hang when
> it's brining up the ethernet devices.
>
> It's not locking up, just the boot scripts never continues - happens
> both on SLES10 and FC7 roots :(
>
> Just wanted to check before spending real time on it.

Pulled Linus' tree (the -rc1 commit is at the head) and it built
and booted on my tiger.  Only new strange stuff in the dmesg is a
lot of

ACPI Exception (processor_throttling-0084): AE_NOT_FOUND, Evaluating _PTC [20070126]
ACPI Exception (processor_throttling-0147): AE_NOT_FOUND, Evaluating _TSS [20070126]

messages (this pair of message repeats 16 times)

-Tony

Re: Linus' tree broken?

[Jes Sorensen]

>>>>> "Tony" = Luck, Tony <tony.luck@intel.com> writes:

>> Have anyone else had seen issues with the Linus git tree failing to
>> boot? I'm trying a pull as of this morning and it seems to hang
>> when it's brining up the ethernet devices.
>> 
>> It's not locking up, just the boot scripts never continues -
>> happens both on SLES10 and FC7 roots :(
>> 
>> Just wanted to check before spending real time on it.

Tony> Pulled Linus' tree (the -rc1 commit is at the head) and it built
Tony> and booted on my tiger.  Only new strange stuff in the dmesg is
Tony> a lot of

Strange!

I heard from John Keller that he had problems too. What kind of
userland are you running? I've seen hanging around udev.

Cheers,
Jes

RE: Linus' tree broken?

[Luck, Tony]

> I heard from John Keller that he had problems too. What kind of
> userland are you running? I've seen hanging around udev.

That box has a RHEL4.5 userland.  I'll try it on some others.

-Tony

Re: Linus' tree broken?

[Doug Chapman]

On Mon, 2007-07-23 at 15:41 +0200, Jes Sorensen wrote:
> Hi,
> 
> Have anyone else had seen issues with the Linus git tree failing to
> boot? I'm trying a pull as of this morning and it seems to hang when
> it's brining up the ethernet devices.
> 
> It's not locking up, just the boot scripts never continues - happens
> both on SLES10 and FC7 roots :(
> 
> Just wanted to check before spending real time on it.
> 
> Cheers,
> Jes
> -

Jes,

Doesn't boot for me either but I think I am seeing a different issue.  I
hit a panic very early in bootup:

16 CPUs available, 16 CPUs total
MCA related initialization done
kernel unaligned access to 0xffffffffffffffff, ip=0xa0000001007f1511
swapper[0]: error during unaligned kernel access
 -1 [1]
Modules linked in:


About to dig into this more deeply now.

- Doug

RE: Linus' tree broken?

[Luck, Tony]

> I heard from John Keller that he had problems too. What kind of
> userland are you running? I've seen hanging around udev.

I took 2.6.23-rc1 over to a Hitachi box running FC7 userspace.
It booted just fine there.

Maybe we broke SN2?

-Tony

Re: Linus' tree broken?

[Len Brown]

On Monday 23 July 2007 12:05, Luck, Tony wrote:
> > Have anyone else had seen issues with the Linus git tree failing to
> > boot? I'm trying a pull as of this morning and it seems to hang when
> > it's brining up the ethernet devices.
> >
> > It's not locking up, just the boot scripts never continues - happens
> > both on SLES10 and FC7 roots :(
> >
> > Just wanted to check before spending real time on it.
> 
> Pulled Linus' tree (the -rc1 commit is at the head) and it built
> and booted on my tiger.  Only new strange stuff in the dmesg is a
> lot of
> 
> ACPI Exception (processor_throttling-0084): AE_NOT_FOUND, Evaluating _PTC [20070126]
> ACPI Exception (processor_throttling-0147): AE_NOT_FOUND, Evaluating _TSS [20070126]
> 
> messages (this pair of message repeats 16 times)

ignore those, they're harmless and going away shortly.

-Len

Re: Linus' tree broken?

[Jes Sorensen]

Luck, Tony wrote:
>> I heard from John Keller that he had problems too. What kind of
>> userland are you running? I've seen hanging around udev.
> 
> I took 2.6.23-rc1 over to a Hitachi box running FC7 userspace.
> It booted just fine there.
> 
> Maybe we broke SN2?
> 
> -Tony

Could be, but it sounds like there's something fishy in the current
tree - Doug Chapman is having problems as well, presumably on zx1 boxes.

I'm out of the office through the rest of the week, so I will look into
it when I get back if it doesn't magically disappear before then :-)

Cheers,
Jes

Re: Linus' tree broken?

[Doug Chapman]

On Mon, 2007-07-23 at 22:22 +0200, Jes Sorensen wrote:
> Luck, Tony wrote:
> >> I heard from John Keller that he had problems too. What kind of
> >> userland are you running? I've seen hanging around udev.
> > 
> > I took 2.6.23-rc1 over to a Hitachi box running FC7 userspace.
> > It booted just fine there.
> > 
> > Maybe we broke SN2?
> > 
> > -Tony
> 
> Could be, but it sounds like there's something fishy in the current
> tree - Doug Chapman is having problems as well, presumably on zx1 boxes.

I am seeing the panic on HP zx1 and zx2 systems.  Currently doing a git
bisect to narrow it down.

- Doug

RE: Linus' tree broken?

[Luck, Tony]

> > Could be, but it sounds like there's something fishy in the current
> > tree - Doug Chapman is having problems as well, presumably on zx1 boxes.
>
> I am seeing the panic on HP zx1 and zx2 systems.  Currently doing a git
> bisect to narrow it down.

2.6.23-rc1 booted OK on my rx2620.  On the zx2000 too (though X is currently
behaving weirdly, I think that might be some other issue though).

-Tony

Re: Linus' tree broken?

[Russ Anderson]

Tony Luck wrote:
> 
> > I heard from John Keller that he had problems too. What kind of
> > userland are you running? I've seen hanging around udev.
> 
> I took 2.6.23-rc1 over to a Hitachi box running FC7 userspace.
> It booted just fine there.
> 
> Maybe we broke SN2?

Neither Linus' tree (2.6.23-rc1) nor Tony's test tree (2.6.22) boot on the 
three SN2 systems I tried booting on (SLES 10 roots).  Tony's test tree
from ~3 weeks ago does boot.

The last boot message is "Starting udevd".  

The cpu LEDs are toggling between 0 & 1 "Kernel: CPU idle".
It doesn't panic.

-- 
Russ Anderson, OS RAS/Partitioning Project Lead  
SGI - Silicon Graphics Inc          rja@sgi.com

Re: Linus' tree broken?

[Peter Chubb]

>>>>> "Doug" = Doug Chapman <doug.chapman@hp.com> writes:

Doug> On Mon, 2007-07-23 at 15:41 +0200, Jes Sorensen wrote:
>> Hi,
>> 
>> Have anyone else had seen issues with the Linus git tree failing to
>> boot? I'm trying a pull as of this morning and it seems to hang
>> when it's brining up the ethernet devices.

Bringing up on Ski I see, `Out of interrupt vectors' --- see
http://www.gelato.unsw.edu.au/kerncomp/results//2007-07-23-17-00/sim_defconfig-boot.html

RE: Linus' tree broken?

[Doug Chapman]

On Mon, 2007-07-23 at 13:42 -0700, Luck, Tony wrote:
> > > Could be, but it sounds like there's something fishy in the current
> > > tree - Doug Chapman is having problems as well, presumably on zx1 boxes.
> >
> > I am seeing the panic on HP zx1 and zx2 systems.  Currently doing a git
> > bisect to narrow it down.
> 
> 2.6.23-rc1 booted OK on my rx2620.  On the zx2000 too (though X is currently
> behaving weirdly, I think that might be some other issue though).
> 
> -Tony

Odd,

I am hitting this on my rx2620, rx6600 and rx2600 (i.e. everywhere I
have tried it) and the bisection is showing that it appears to have been
broken before -rc1.


It appears to have something to do with grabbing the kernel args.  The
panic happens here:

0xa0000001007f1511 is in setup_arch (arch/ia64/kernel/setup.c:570).
565
566             /* enable IA-64 Machine Check Abort Handling unless disabled
*/
567             if (!nomca)
568                     ia64_mca_init();
569
570             platform_setup(cmdline_p);
571             paging_init();
572     }


If it is working for others I am wondering if the version of elilo makes
a difference.  I am using RHEL5 as my base system.  I might try grabbing
elilo from upstream or from another release and see if that makes a
difference.

Bisection almost done, should know what commit triggered this soon.  I
tried reverting a couple seemingly obvious patches but didn't find it
that way.

- Doug

RE: Linus' tree broken?

[Luck, Tony]

> I am hitting this on my rx2620, rx6600 and rx2600 (i.e. everywhere I
> have tried it) and the bisection is showing that it appears to have been
> broken before -rc1.

Are you booting a generic kernel, or a zx1_defconfig based one?  Look
back in the linux-ia64 archives a few days for a post titled:

 "Regression in serial console on ia64 after 2.6.22"

Perhaps you are hitting this?

-Tony

RE: Linus' tree broken?

[Doug Chapman]

On Mon, 2007-07-23 at 15:42 -0700, Luck, Tony wrote:
> > I am hitting this on my rx2620, rx6600 and rx2600 (i.e. everywhere I
> > have tried it) and the bisection is showing that it appears to have been
> > broken before -rc1.
> 
> Are you booting a generic kernel, or a zx1_defconfig based one?  Look

My config is a custom config based on the RHEL/Fedora builds.

> back in the linux-ia64 archives a few days for a post titled:
> 
>  "Regression in serial console on ia64 after 2.6.22"
> 
> Perhaps you are hitting this?

I don't think I am seeing the same thing.  My console is OK, it just
panics.  I had suspected that same patch mentioned in the post you point
to as a possible culprit but backing it out did not appear to fix my
issue.

Also, I am not passing any console= arg to the kernel and letting it
automagically determine the right thing (which is still working).

In fact I am not passing _any_ args to the kernel. perhaps that is the
difference?

- Doug

Re: Linus' tree broken?

[Jes Sorensen]

Russ Anderson wrote:
> Neither Linus' tree (2.6.23-rc1) nor Tony's test tree (2.6.22) boot on the 
> three SN2 systems I tried booting on (SLES 10 roots).  Tony's test tree
> from ~3 weeks ago does boot.
> 
> The last boot message is "Starting udevd".  
> 
> The cpu LEDs are toggling between 0 & 1 "Kernel: CPU idle".
> It doesn't panic.
> 

Yeah I see the same, I can hit return on the console and it reacts but
doesn't get any further.

git commit 7dcca30a32aadb0520417521b0c44f42d09fe05c works for me - thats
the 2.6.22 Sun Jul 8 marker from Linus. And I know git
f695baf2df9e0413d3521661070103711545207a is hosed - Sun Jul 22.

Sorry just sitting waiting for a plane, don't have much time to look at
it right now.

Wonder if Tony's .config is much different from everyone else's? Mine is
an SN2 specific kernel, haven't tried GENERIC yet.

Cheers,
Jes

Re: Linus' tree broken?

[Jes Sorensen]

Doug Chapman wrote:
> 
> It appears to have something to do with grabbing the kernel args.  The
> panic happens here:
> 
> 0xa0000001007f1511 is in setup_arch (arch/ia64/kernel/setup.c:570).
> 565
> 566             /* enable IA-64 Machine Check Abort Handling unless disabled
> */
> 567             if (!nomca)
> 568                     ia64_mca_init();
> 569
> 570             platform_setup(cmdline_p);
> 571             paging_init();
> 572     }

Doug,

I see the exact same OOPS if I compile a generic kernel instead of an
SN2 specific one.

Cheers,
Jes

Re: Linus' tree broken?

[Russ Anderson]

Russ Anderson wrote:
> Tony Luck wrote:
> > 
> > Maybe we broke SN2?
> 
> Neither Linus' tree (2.6.23-rc1) nor Tony's test tree (2.6.22) boot on the 
> three SN2 systems I tried booting on (SLES 10 roots).  Tony's test tree
> from ~3 weeks ago does boot.
> 
> The last boot message is "Starting udevd".  

More specificly, it hangs in /sbin/udevsettle .


-- 
Russ Anderson, OS RAS/Partitioning Project Lead  
SGI - Silicon Graphics Inc          rja@sgi.com