RHEL (CentOS, Oracle Liux) kernel 2.6.32-358 does not start at XCP

RHEL (CentOS, Oracle Liux) kernel 2.6.32-358 does not start at XCP

[Constantin Dunayev]

[-- Attachment #1.1: Type: text/plain, Size: 657 bytes --]

 Hi, everyone,
Two days ago I have filed a bug to Red Hat   https://bugzilla.redhat.com/show_bug.cgi?id=918239
The problem is, that new RHEL-6.4's kernel 2.6.32-358 does not start at HVM on Xen Cloud Platform 1.6 (maybe, also affected XenServer 6.1)
I have tested kernels -358 from CentOS, Oracle and -356 from Red Hat both i686 and x86_64 at 2 different server hardware
The sympthoms: kernel does not boot at all, does not produce any log; previous (all 2.6.32-279 and 2.6.32-220), and Oracle's kernel-uek work perfectly.

Andrew Jones from Red Hat advised me to ask, does anybody else have this issue?

Best regards,
-- 
Constantin Dunayev,
Kharkov, UA 

[-- Attachment #1.2: Type: text/html, Size: 872 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: RHEL (CentOS, Oracle Liux) kernel 2.6.32-358 does not start at XCP

[Pasi Kärkkäinen]

On Thu, Mar 07, 2013 at 02:26:10PM +0400, Constantin Dunayev wrote:
>    Hi, everyone,
>    Two days ago I have filed a bug to Red Hat
>     [1]https://bugzilla.redhat.com/show_bug.cgi?id=918239
>

That bug is marked as private, so it's not visible to other people.

>    The problem is, that new RHEL-6.4's kernel 2.6.32-358 does not start at
>    HVM on Xen Cloud Platform 1.6 (maybe, also affected XenServer 6.1)
>    I have tested kernels -358 from CentOS, Oracle and -356 from Red Hat both
>    i686 and x86_64 at 2 different server hardware
>    The sympthoms: kernel does not boot at all, does not produce any log;
>    previous (all 2.6.32-279 and 2.6.32-220), and Oracle's kernel-uek work
>    perfectly.
> 
>    Andrew Jones from Red Hat advised me to ask, does anybody else have this
>    issue?
> 

Did you try booting the domU kernel with "earlyprintk=xen console=hvc0" cmdline options? 

-- Pasi

Re: RHEL (CentOS, Oracle Liux) kernel 2.6.32-358 does not start at XCP

[Andrew Cooper]

On 07/03/13 14:13, Pasi Kärkkäinen wrote:
> On Thu, Mar 07, 2013 at 02:26:10PM +0400, Constantin Dunayev wrote:
>>    Hi, everyone,
>>    Two days ago I have filed a bug to Red Hat
>>     [1]https://bugzilla.redhat.com/show_bug.cgi?id=918239
>>
> That bug is marked as private, so it's not visible to other people.
>
>>    The problem is, that new RHEL-6.4's kernel 2.6.32-358 does not start at
>>    HVM on Xen Cloud Platform 1.6 (maybe, also affected XenServer 6.1)
>>    I have tested kernels -358 from CentOS, Oracle and -356 from Red Hat both
>>    i686 and x86_64 at 2 different server hardware
>>    The sympthoms: kernel does not boot at all, does not produce any log;
>>    previous (all 2.6.32-279 and 2.6.32-220), and Oracle's kernel-uek work
>>    perfectly.
>>
>>    Andrew Jones from Red Hat advised me to ask, does anybody else have this
>>    issue?
>>
> Did you try booting the domU kernel with "earlyprintk=xen console=hvc0" cmdline options? 
>
> -- Pasi

Or alternatively, start with pause-on-crash, and use xenctx with the
appropriate symbol table.

~Andrew (who also unable to view the bug ticket)

>
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel

Re: RHEL (CentOS, Oracle Liux) kernel 2.6.32-358 does not start at XCP

[Pasi Kärkkäinen]

On Thu, Mar 07, 2013 at 02:39:24PM +0000, Andrew Cooper wrote:
> On 07/03/13 14:13, Pasi Kärkkäinen wrote:
> > On Thu, Mar 07, 2013 at 02:26:10PM +0400, Constantin Dunayev wrote:
> >>    Hi, everyone,
> >>    Two days ago I have filed a bug to Red Hat
> >>     [1]https://bugzilla.redhat.com/show_bug.cgi?id=918239
> >>
> > That bug is marked as private, so it's not visible to other people.
> >
> >>    The problem is, that new RHEL-6.4's kernel 2.6.32-358 does not start at
> >>    HVM on Xen Cloud Platform 1.6 (maybe, also affected XenServer 6.1)
> >>    I have tested kernels -358 from CentOS, Oracle and -356 from Red Hat both
> >>    i686 and x86_64 at 2 different server hardware
> >>    The sympthoms: kernel does not boot at all, does not produce any log;
> >>    previous (all 2.6.32-279 and 2.6.32-220), and Oracle's kernel-uek work
> >>    perfectly.
> >>
> >>    Andrew Jones from Red Hat advised me to ask, does anybody else have this
> >>    issue?
> >>
> > Did you try booting the domU kernel with "earlyprintk=xen console=hvc0" cmdline options? 
> >
> > -- Pasi
> 
> Or alternatively, start with pause-on-crash, and use xenctx with the
> appropriate symbol table.
> 
> ~Andrew (who also unable to view the bug ticket)
> 

Yep, instructions for using xenctx here:
http://wiki.xen.org/wiki/Xen_Common_Problems#My_Xen_PV_guest_kernel_crashes.2C_how_can_I_debug_it_and_get_a_stack.2Fcall_trace.3F

-- Pasi

Re: RHEL (CentOS, Oracle Liux) kernel 2.6.32-358 does not start at XCP

[Andrew Jones]

----- Original Message -----
> On Thu, Mar 07, 2013 at 02:26:10PM +0400, Constantin Dunayev wrote:
> >    Hi, everyone,
> >    Two days ago I have filed a bug to Red Hat
> >     [1]https://bugzilla.redhat.com/show_bug.cgi?id=918239
> >
> 
> That bug is marked as private, so it's not visible to other people.
> 

Strange. There's no reason for the BZ to be private. It was opened
externally, and there's nothing in the history indicating its visibility
was changed. I also don't see any way to make it any more public than
it already is (of course it doesn't help that the bugzilla interface
keeps changing, so maybe there's a new series of clicks that aren't
obvious to me...)

> >    The problem is, that new RHEL-6.4's kernel 2.6.32-358 does not
> >    start at
> >    HVM on Xen Cloud Platform 1.6 (maybe, also affected XenServer
> >    6.1)
> >    I have tested kernels -358 from CentOS, Oracle and -356 from Red
> >    Hat both
> >    i686 and x86_64 at 2 different server hardware
> >    The sympthoms: kernel does not boot at all, does not produce any
> >    log;
> >    previous (all 2.6.32-279 and 2.6.32-220), and Oracle's
> >    kernel-uek work
> >    perfectly.
> > 
> >    Andrew Jones from Red Hat advised me to ask, does anybody else
> >    have this
> >    issue?
> > 
> 
> Did you try booting the domU kernel with "earlyprintk=xen
> console=hvc0" cmdline options?
> 
> -- Pasi
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel
> 

Re: RHEL (CentOS, Oracle Liux) kernel 2.6.32-358 does not start at XCP

[Andrew Jones]

----- Original Message -----
> 
> 
> ----- Original Message -----
> > On Thu, Mar 07, 2013 at 02:26:10PM +0400, Constantin Dunayev wrote:
> > >    Hi, everyone,
> > >    Two days ago I have filed a bug to Red Hat
> > >     [1]https://bugzilla.redhat.com/show_bug.cgi?id=918239
> > >
> > 
> > That bug is marked as private, so it's not visible to other people.
> > 
> 
> Strange. There's no reason for the BZ to be private. It was opened
> externally, and there's nothing in the history indicating its
> visibility
> was changed. I also don't see any way to make it any more public than
> it already is (of course it doesn't help that the bugzilla interface
> keeps changing, so maybe there's a new series of clicks that aren't
> obvious to me...)

Ok, so there were a serious of clicks (which are obvious now in hind-
sight). I've opened the BZ to the public.

> 
> > >    The problem is, that new RHEL-6.4's kernel 2.6.32-358 does not
> > >    start at
> > >    HVM on Xen Cloud Platform 1.6 (maybe, also affected XenServer
> > >    6.1)
> > >    I have tested kernels -358 from CentOS, Oracle and -356 from
> > >    Red
> > >    Hat both
> > >    i686 and x86_64 at 2 different server hardware
> > >    The sympthoms: kernel does not boot at all, does not produce
> > >    any
> > >    log;
> > >    previous (all 2.6.32-279 and 2.6.32-220), and Oracle's
> > >    kernel-uek work
> > >    perfectly.
> > > 
> > >    Andrew Jones from Red Hat advised me to ask, does anybody else
> > >    have this
> > >    issue?
> > > 
> > 
> > Did you try booting the domU kernel with "earlyprintk=xen
> > console=hvc0" cmdline options?
> > 
> > -- Pasi
> > 
> > 
> > _______________________________________________
> > Xen-devel mailing list
> > Xen-devel@lists.xen.org
> > http://lists.xen.org/xen-devel
> > 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel
> 

Re: RHEL (CentOS, Oracle Liux) kernel 2.6.32-358 does not start at XCP

[Pasi Kärkkäinen]

On Fri, Mar 08, 2013 at 07:28:31AM -0500, Andrew Jones wrote:
> 
> 
> ----- Original Message -----
> > 
> > 
> > ----- Original Message -----
> > > On Thu, Mar 07, 2013 at 02:26:10PM +0400, Constantin Dunayev wrote:
> > > >    Hi, everyone,
> > > >    Two days ago I have filed a bug to Red Hat
> > > >     [1]https://bugzilla.redhat.com/show_bug.cgi?id=918239
> > > >
> > > 
> > > That bug is marked as private, so it's not visible to other people.
> > > 
> > 
> > Strange. There's no reason for the BZ to be private. It was opened
> > externally, and there's nothing in the history indicating its
> > visibility
> > was changed. I also don't see any way to make it any more public than
> > it already is (of course it doesn't help that the bugzilla interface
> > keeps changing, so maybe there's a new series of clicks that aren't
> > obvious to me...)
> 
> Ok, so there were a serious of clicks (which are obvious now in hind-
> sight). I've opened the BZ to the public.
> 

Thanks!

-- Pasi

Re: RHEL (CentOS, Oracle Liux) kernel 2.6.32-358 does not start at XCP

[Andrew Cooper]

On 07/03/13 10:26, Constantin Dunayev wrote:
> Hi, everyone,
> Two days ago I have filed a bug to Red Hat
>  https://bugzilla.redhat.com/show_bug.cgi?id=918239
> The problem is, that new RHEL-6.4's kernel 2.6.32-358 does not start
> at HVM on Xen Cloud Platform 1.6 (maybe, also affected XenServer 6.1)
> I have tested kernels -358 from CentOS, Oracle and -356 from Red Hat
> both i686 and x86_64 at 2 different server hardware
> The sympthoms: kernel does not boot at all, does not produce any log;
> previous (all 2.6.32-279 and 2.6.32-220), and Oracle's kernel-uek work
> perfectly.
>
> Andrew Jones from Red Hat advised me to ask, does anybody else have
> this issue?
>
> Best regards,
> -- 
> Constantin Dunayev,
> Kharkov, UA  

To follow up on this issue.  My findings are presented in full on the BZ
ticket, but in summary:

RHEL 6.4 appears to be somewhat viridian aware

The final message on the console before the VCPU spins is

"Switching to clocksource hyperv_clocksource"

and is presumably waiting for an interrupt from a clocksource we do not
implement.

Disabling viridian for the domain appears to work around the issue:

For XenServer,XCP use:

xe vm-param-set vm=$VM platform:viridian=false

For xl/xm, ensure viridian=false in the VM configuration file

~Andrew