xen-unstable HV fails to boot on AMD quad core

xen-unstable HV fails to boot on AMD quad core

[Daniel P. Berrange]

I'm attempting to do some testing with xen-unstable but am unable to get
the hypervisor to boot. With both xen-unstable 16125, and staging 16142
the HV appears to immediately hang at boot time. No console output is
displayed at all.

If I go back to 16000, then the console stays black for approx 10 seconds
before Xen HV boots. This is on a single CPU, Quad Core AMD host.

I'm about to try a binary search to narrow down where the problem might
be, but first was wondering if anyone else has seen this, or if there's
any recent changesets I should pay particular attention to.... ?

Regards,
Dan.
-- 
|=- Red Hat, Engineering, Emerging Technologies, Boston.  +1 978 392 2496 -=|
|=-           Perl modules: http://search.cpan.org/~danberr/              -=|
|=-               Projects: http://freshmeat.net/~danielpb/               -=|
|=-  GnuPG: 7D3B9505   F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505  -=| 

Re: xen-unstable HV fails to boot on AMD quad core

[Keir Fraser]

Are you using the new SVGA text console support? That would explain a few
seconds delay to seeing any output on 16000, although 10s is longer than I'd
expect. Also the late initialisation of SVGA text console means there's
quite a bit of scope for things to go wrong during bootup before you have a
chance of seeing anything on screen. Can you connect a serial line?

As for possible changesets, plenty of exciting stuff has been going in over
the last few days. Particularly relevant might be changesets 16093 and 16094
which are specifically regarding support for the new quad cores.

 -- Keir

On 19/10/07 16:31, "Daniel P. Berrange" <berrange@redhat.com> wrote:

> I'm attempting to do some testing with xen-unstable but am unable to get
> the hypervisor to boot. With both xen-unstable 16125, and staging 16142
> the HV appears to immediately hang at boot time. No console output is
> displayed at all.
> 
> If I go back to 16000, then the console stays black for approx 10 seconds
> before Xen HV boots. This is on a single CPU, Quad Core AMD host.
> 
> I'm about to try a binary search to narrow down where the problem might
> be, but first was wondering if anyone else has seen this, or if there's
> any recent changesets I should pay particular attention to.... ?
> 
> Regards,
> Dan.

Re: xen-unstable HV fails to boot on AMD quad core

[Daniel P. Berrange]

On Fri, Oct 19, 2007 at 04:40:14PM +0100, Keir Fraser wrote:
> Are you using the new SVGA text console support? That would explain a few
> seconds delay to seeing any output on 16000, although 10s is longer than I'd
> expect. Also the late initialisation of SVGA text console means there's
> quite a bit of scope for things to go wrong during bootup before you have a
> chance of seeing anything on screen. Can you connect a serial line?

I don't know if I'm using SVGA console support. This is basically a 100%
clean checkout of xen-unstable.hg and I just did 'make' in the 'xen'
directory without changing any build config options, and I've not given any
special boot parameters to the HV. So if SVGA stuff is enabled by default, 
then I'm using it....   Is there a grub boot option I can use to disable
it ?

Unfortunately this box is one of the 'nice' new legacy free machines
and thus no serial :-(

> As for possible changesets, plenty of exciting stuff has been going in over
> the last few days. Particularly relevant might be changesets 16093 and 16094
> which are specifically regarding support for the new quad cores.

Thanks, I'll try some builds before & after these

Regards,
Dan.
-- 
|=- Red Hat, Engineering, Emerging Technologies, Boston.  +1 978 392 2496 -=|
|=-           Perl modules: http://search.cpan.org/~danberr/              -=|
|=-               Projects: http://freshmeat.net/~danielpb/               -=|
|=-  GnuPG: 7D3B9505   F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505  -=| 

Re: xen-unstable HV fails to boot on AMD quad core

[Keir Fraser]

On 19/10/07 16:51, "Daniel P. Berrange" <berrange@redhat.com> wrote:

> I don't know if I'm using SVGA console support. This is basically a 100%
> clean checkout of xen-unstable.hg and I just did 'make' in the 'xen'
> directory without changing any build config options, and I've not given any
> special boot parameters to the HV. So if SVGA stuff is enabled by default,
> then I'm using it....   Is there a grub boot option I can use to disable
> it ?

No it's not enabled by default. So the 10s delay is rather weird! Is this a
property of any version of Xen on this machine (e.g., 3.1.0, 3.1.1)? Or is
it a new property of recent xen-unstable?

> Unfortunately this box is one of the 'nice' new legacy free machines
> and thus no serial :-(

Oh dear. :-( For that kind of machine I tend to stick in a PCI serial card,
work out its BIOS-assigned resources, and then manually specify the I/O
ports and IRQ on the Xen command line. Working with no serial at all is way
too painful.

 -- Keir

>> As for possible changesets, plenty of exciting stuff has been going in over
>> the last few days. Particularly relevant might be changesets 16093 and 16094
>> which are specifically regarding support for the new quad cores.
> 
> Thanks, I'll try some builds before & after these

Re: xen-unstable HV fails to boot on AMD quad core

[Daniel P. Berrange]

On Fri, Oct 19, 2007 at 05:12:04PM +0100, Keir Fraser wrote:
> On 19/10/07 16:51, "Daniel P. Berrange" <berrange@redhat.com> wrote:
> 
> > I don't know if I'm using SVGA console support. This is basically a 100%
> > clean checkout of xen-unstable.hg and I just did 'make' in the 'xen'
> > directory without changing any build config options, and I've not given any
> > special boot parameters to the HV. So if SVGA stuff is enabled by default,
> > then I'm using it....   Is there a grub boot option I can use to disable
> > it ?
> 
> No it's not enabled by default. So the 10s delay is rather weird! Is this a
> property of any version of Xen on this machine (e.g., 3.1.0, 3.1.1)? Or is
> it a new property of recent xen-unstable?

3.1.0 works without any issue. The 3.1.1 rc's did too, though I've not
tested the final release. Will do some builds to try & find  where it
came in...

I'm also going to update some other machines to xen-unstable

> > Unfortunately this box is one of the 'nice' new legacy free machines
> > and thus no serial :-(
> 
> Oh dear. :-( For that kind of machine I tend to stick in a PCI serial card,
> work out its BIOS-assigned resources, and then manually specify the I/O
> ports and IRQ on the Xen command line. Working with no serial at all is way
> too painful.

Yes, its seriously painful.

Dan.
-- 
|=- Red Hat, Engineering, Emerging Technologies, Boston.  +1 978 392 2496 -=|
|=-           Perl modules: http://search.cpan.org/~danberr/              -=|
|=-               Projects: http://freshmeat.net/~danielpb/               -=|
|=-  GnuPG: 7D3B9505   F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505  -=| 

Re: xen-unstable HV fails to boot on AMD quad core

[Daniel P. Berrange]

On Fri, Oct 19, 2007 at 04:31:09PM +0100, Daniel P. Berrange wrote:
> I'm attempting to do some testing with xen-unstable but am unable to get
> the hypervisor to boot. With both xen-unstable 16125, and staging 16142
> the HV appears to immediately hang at boot time. No console output is
> displayed at all.
> 
> If I go back to 16000, then the console stays black for approx 10 seconds
> before Xen HV boots. This is on a single CPU, Quad Core AMD host.

So, I found that the problem hits in changeset 16104.  Prior to that changeset
it will always boot, post that changeset it hangs immediately.  NB, when I
tested with 16103 I had pulled in the compile fix from 16105 only.

Further, it only appears to have trouble on this one machine. Another AMD
dual core host works fine, as does an Intel box I tried. 

As for the 10 second pause, I'm putting that down to a BIOS bug, since other
machines don't exhibit that.

Regards,
Dan.
-- 
|=- Red Hat, Engineering, Emerging Technologies, Boston.  +1 978 392 2496 -=|
|=-           Perl modules: http://search.cpan.org/~danberr/              -=|
|=-               Projects: http://freshmeat.net/~danielpb/               -=|
|=-  GnuPG: 7D3B9505   F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505  -=| 

Re: xen-unstable HV fails to boot on AMD quad core

[Keir Fraser]

On 19/10/07 18:51, "Daniel P. Berrange" <berrange@redhat.com> wrote:

> So, I found that the problem hits in changeset 16104.  Prior to that changeset
> it will always boot, post that changeset it hangs immediately.  NB, when I
> tested with 16103 I had pulled in the compile fix from 16105 only.
> 
> Further, it only appears to have trouble on this one machine. Another AMD
> dual core host works fine, as does an Intel box I tried.

"xentrace/x86: PV guest tracing extensions"??

I'm surprised as that one appears pretty harmless. Perhaps it's a latent bug
revealed?

 -- Keir

Re: xen-unstable HV fails to boot on AMD quad core

[Daniel P. Berrange]

On Fri, Oct 19, 2007 at 07:02:44PM +0100, Keir Fraser wrote:
> On 19/10/07 18:51, "Daniel P. Berrange" <berrange@redhat.com> wrote:
> 
> > So, I found that the problem hits in changeset 16104.  Prior to that changeset
> > it will always boot, post that changeset it hangs immediately.  NB, when I
> > tested with 16103 I had pulled in the compile fix from 16105 only.
> > 
> > Further, it only appears to have trouble on this one machine. Another AMD
> > dual core host works fine, as does an Intel box I tried.
> 
> "xentrace/x86: PV guest tracing extensions"??
> 
> I'm surprised as that one appears pretty harmless. Perhaps it's a latent bug
> revealed?

I'm inclined to think all the problems are a hardware issue. I've managed to 
get access to another AMD quad-core & that also works fine. So I'd ignore 
this issue unless I can find a 2nd box to reproduce on.

Regards,
Dan.
-- 
|=- Red Hat, Engineering, Emerging Technologies, Boston.  +1 978 392 2496 -=|
|=-           Perl modules: http://search.cpan.org/~danberr/              -=|
|=-               Projects: http://freshmeat.net/~danielpb/               -=|
|=-  GnuPG: 7D3B9505   F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505  -=| 

Re: xen-unstable HV fails to boot on AMD quad core

[Keir Fraser]

On 19/10/07 18:51, "Daniel P. Berrange" <berrange@redhat.com> wrote:

>> If I go back to 16000, then the console stays black for approx 10 seconds
>> before Xen HV boots. This is on a single CPU, Quad Core AMD host.
> 
> So, I found that the problem hits in changeset 16104.  Prior to that changeset
> it will always boot, post that changeset it hangs immediately.  NB, when I
> tested with 16103 I had pulled in the compile fix from 16105 only.
> 
> Further, it only appears to have trouble on this one machine. Another AMD
> dual core host works fine, as does an Intel box I tried.

I've seen a non-debug build just crash very early on my home Intel box, but
cannot repro on machines in the office. So there's something odd going on
and it's fairly likely this is the same bug that you saw. At least I have a
serial line connected, so I'll take a look this evening.

 -- Keir