HVM direct boot broken in xen-unstable

HVM direct boot broken in xen-unstable

[Byrne, John (HP Labs)]

Hi!

On x86_64, changeset 18081, running/building on rhel5, trying to use the HVM direct boot causes the domain to reboot immediately and then the log says the domain is restarting too rapidly. No obvious hints as to where the problem is. If I build the in-tree ioemu code, things work. Any clues as to where to look or how to get some useful debugging output?

Thanks,

John Byrne

Re: HVM direct boot broken in xen-unstable

[Daniel P. Berrange]

On Sat, Jul 19, 2008 at 12:52:07AM +0000, Byrne, John (HP Labs) wrote:
> Hi!
> 
> On x86_64, changeset 18081, running/building on rhel5, trying to use
> the HVM direct boot causes the domain to reboot immediately and then
> the log says the domain is restarting too rapidly. No obvious hints 
> as to where the problem is. If I build the in-tree ioemu code, things
> work. Any clues as to where to look or how to get some useful debugging
> output?

It appears the code is missing from the external ioemu code. The original
changesets for HVM kernel boot were

changeset:   16959:17cce0554151
user:        Keir Fraser <keir.fraser@citrix.com>
date:        Fri Feb 01 11:14:53 2008 +0000
summary:     Direct Linux boot: Support booting relocatable Linux kernels.

changeset:   16960:67ca9c37ef02
user:        Keir Fraser <keir.fraser@citrix.com>
date:        Fri Feb 01 11:16:37 2008 +0000
summary:     Direct Linux boot: Support booting non-relocatable Linux kernels.


Part of the first changeset is missing, and all of the second. So the
kernel will be getting overwritten by the HVM firmware hence the crashes
of the guest you see

Daniel
-- 
|: Red Hat, Engineering, London   -o-   http://people.redhat.com/berrange/ :|
|: http://libvirt.org  -o-  http://virt-manager.org  -o-  http://ovirt.org :|
|: http://autobuild.org       -o-         http://search.cpan.org/~danberr/ :|
|: GnuPG: 7D3B9505  -o-  F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|

Re: HVM direct boot broken in xen-unstable

[Ian Jackson]

Daniel P. Berrange writes ("Re: [Xen-devel] HVM direct boot broken in xen-unstable"):
> It appears the code is missing from the external ioemu code. The original
> changesets for HVM kernel boot were

Thanks for tracking that down.  I will port those across.

Ian.

Re: HVM direct boot broken in xen-unstable

[Ian Jackson]

Daniel P. Berrange writes ("Re: [Xen-devel] HVM direct boot broken in xen-unstable"):
> On Sat, Jul 19, 2008 at 12:52:07AM +0000, Byrne, John (HP Labs) wrote:
> > Hi!
> > 
> > On x86_64, changeset 18081, running/building on rhel5, trying to use
> > the HVM direct boot causes the domain to reboot immediately and then
> > the log says the domain is restarting too rapidly. No obvious hints 
> > as to where the problem is. If I build the in-tree ioemu code, things
> > work. Any clues as to where to look or how to get some useful debugging
> > output?

Byrne, could you send me a copy of your config file so I can reproduce
this ?  Also, some idea of how you made (or how you obtained) the
kernel image and initramfs.  I don't think I've ever done a direct
kernel boot using this code (which is in patches supplied by Daniel)
and offhand I can't seem to find the documentation.

> It appears the code is missing from the external ioemu code. The original
> changesets for HVM kernel boot were

I've looked at the code and these changes are ones I did attempt to
port across to the new qemu.  You'll see for example that the `evil
helper' is still there, just slightly further down the file in pc.c.

I haven't ever executed the resulting codepaths and evidently I broke
something, which is not all that surprising.

Thanks for reporting the bug.  I look forward to fixing it :-).

Ian.

Re: HVM direct boot broken in xen-unstable

[Daniel P. Berrange]

On Mon, Jul 21, 2008 at 04:46:17PM +0100, Ian Jackson wrote:
> Daniel P. Berrange writes ("Re: [Xen-devel] HVM direct boot broken in xen-unstable"):
> > On Sat, Jul 19, 2008 at 12:52:07AM +0000, Byrne, John (HP Labs) wrote:
> > > Hi!
> > > 
> > > On x86_64, changeset 18081, running/building on rhel5, trying to use
> > > the HVM direct boot causes the domain to reboot immediately and then
> > > the log says the domain is restarting too rapidly. No obvious hints 
> > > as to where the problem is. If I build the in-tree ioemu code, things
> > > work. Any clues as to where to look or how to get some useful debugging
> > > output?
> 
> Byrne, could you send me a copy of your config file so I can reproduce
> this ?  Also, some idea of how you made (or how you obtained) the
> kernel image and initramfs.  I don't think I've ever done a direct
> kernel boot using this code (which is in patches supplied by Daniel)
> and offhand I can't seem to find the documentation.

It works in the same way as booting PV off kernel+initrd - basically 
take a HVM guest config file and add

  loader="/usr/lib/xen/bin/hvmloader"
  kernel="/path/to/vmlinuz"
  ramdisk="/path/to/initrd"

The vmlinuz & initrd  would typically be the ones for installation
of your distro, eg The $URL/images/pxeboot/ files for Fedora/RHEL

http://download.fedora.redhat.com/pub/fedora/linux/releases/9/Fedora/i386/os/images/pxeboot/

It should boot your kernel & initrd and not pay attention to hardisk
or CDROM devices.

As mentioned in the original commit logs, older kernels need to use the
evil helper, newer kernels should 'just work' (thanks to the relocatable
kernel support).

Daniel
-- 
|: Red Hat, Engineering, London   -o-   http://people.redhat.com/berrange/ :|
|: http://libvirt.org  -o-  http://virt-manager.org  -o-  http://ovirt.org :|
|: http://autobuild.org       -o-         http://search.cpan.org/~danberr/ :|
|: GnuPG: 7D3B9505  -o-  F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|

RE: HVM direct boot broken in xen-unstable

[Byrne, John (HP Labs)]

Daniel, Ian:

The old ioemu version functions without the "loader" line which I was missing in my config. I'm testing to make sure that isn't the issue I'm seeing with the git version.

John


> -----Original Message-----
> From: Daniel P. Berrange [mailto:berrange@redhat.com]
> Sent: Monday, July 21, 2008 8:52 AM
> To: Ian Jackson
> Cc: Byrne, John (HP Labs); xen-devel@lists.xensource.com
> Subject: Re: [Xen-devel] HVM direct boot broken in xen-unstable
>
> On Mon, Jul 21, 2008 at 04:46:17PM +0100, Ian Jackson wrote:
> > Daniel P. Berrange writes ("Re: [Xen-devel] HVM direct boot broken in
> xen-unstable"):
> > > On Sat, Jul 19, 2008 at 12:52:07AM +0000, Byrne, John (HP Labs)
> wrote:
> > > > Hi!
> > > >
> > > > On x86_64, changeset 18081, running/building on rhel5, trying to
> use
> > > > the HVM direct boot causes the domain to reboot immediately and
> then
> > > > the log says the domain is restarting too rapidly. No obvious
> hints
> > > > as to where the problem is. If I build the in-tree ioemu code,
> things
> > > > work. Any clues as to where to look or how to get some useful
> debugging
> > > > output?
> >
> > Byrne, could you send me a copy of your config file so I can
> reproduce
> > this ?  Also, some idea of how you made (or how you obtained) the
> > kernel image and initramfs.  I don't think I've ever done a direct
> > kernel boot using this code (which is in patches supplied by Daniel)
> > and offhand I can't seem to find the documentation.
>
> It works in the same way as booting PV off kernel+initrd - basically
> take a HVM guest config file and add
>
>   loader="/usr/lib/xen/bin/hvmloader"
>   kernel="/path/to/vmlinuz"
>   ramdisk="/path/to/initrd"
>
> The vmlinuz & initrd  would typically be the ones for installation
> of your distro, eg The $URL/images/pxeboot/ files for Fedora/RHEL
>
> http://download.fedora.redhat.com/pub/fedora/linux/releases/9/Fedora/i3
> 86/os/images/pxeboot/
>
> It should boot your kernel & initrd and not pay attention to hardisk
> or CDROM devices.
>
> As mentioned in the original commit logs, older kernels need to use the
> evil helper, newer kernels should 'just work' (thanks to the
> relocatable
> kernel support).
>
> Daniel
> --
> |: Red Hat, Engineering, London   -o-
> http://people.redhat.com/berrange/ :|
> |: http://libvirt.org  -o-  http://virt-manager.org  -o-
> http://ovirt.org :|
> |: http://autobuild.org       -o-
> http://search.cpan.org/~danberr/ :|
> |: GnuPG: 7D3B9505  -o-  F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B
> 9505 :|

RE: HVM direct boot broken in xen-unstable

[Byrne, John (HP Labs)]

Daniel, Ian:

No difference with "loader". The domain still disappears almost immediately.

I'll try to understand the original code (now that I know where to look) and see if I can understand what is happening.

Thanks for your help,

John Byrne


> -----Original Message-----
> From: Byrne, John (HP Labs)
> Sent: Monday, July 21, 2008 8:59 AM
> To: 'Daniel P. Berrange'; Ian Jackson
> Cc: xen-devel@lists.xensource.com
> Subject: RE: [Xen-devel] HVM direct boot broken in xen-unstable
>
> Daniel, Ian:
>
> The old ioemu version functions without the "loader" line which I was
> missing in my config. I'm testing to make sure that isn't the issue I'm
> seeing with the git version.
>
> John
>
>
> > -----Original Message-----
> > From: Daniel P. Berrange [mailto:berrange@redhat.com]
> > Sent: Monday, July 21, 2008 8:52 AM
> > To: Ian Jackson
> > Cc: Byrne, John (HP Labs); xen-devel@lists.xensource.com
> > Subject: Re: [Xen-devel] HVM direct boot broken in xen-unstable
> >
> > On Mon, Jul 21, 2008 at 04:46:17PM +0100, Ian Jackson wrote:
> > > Daniel P. Berrange writes ("Re: [Xen-devel] HVM direct boot broken
> in
> > xen-unstable"):
> > > > On Sat, Jul 19, 2008 at 12:52:07AM +0000, Byrne, John (HP Labs)
> > wrote:
> > > > > Hi!
> > > > >
> > > > > On x86_64, changeset 18081, running/building on rhel5, trying
> to
> > use
> > > > > the HVM direct boot causes the domain to reboot immediately and
> > then
> > > > > the log says the domain is restarting too rapidly. No obvious
> > hints
> > > > > as to where the problem is. If I build the in-tree ioemu code,
> > things
> > > > > work. Any clues as to where to look or how to get some useful
> > debugging
> > > > > output?
> > >
> > > Byrne, could you send me a copy of your config file so I can
> > reproduce
> > > this ?  Also, some idea of how you made (or how you obtained) the
> > > kernel image and initramfs.  I don't think I've ever done a direct
> > > kernel boot using this code (which is in patches supplied by
> Daniel)
> > > and offhand I can't seem to find the documentation.
> >
> > It works in the same way as booting PV off kernel+initrd - basically
> > take a HVM guest config file and add
> >
> >   loader="/usr/lib/xen/bin/hvmloader"
> >   kernel="/path/to/vmlinuz"
> >   ramdisk="/path/to/initrd"
> >
> > The vmlinuz & initrd  would typically be the ones for installation
> > of your distro, eg The $URL/images/pxeboot/ files for Fedora/RHEL
> >
> >
> http://download.fedora.redhat.com/pub/fedora/linux/releases/9/Fedora/i3
> > 86/os/images/pxeboot/
> >
> > It should boot your kernel & initrd and not pay attention to hardisk
> > or CDROM devices.
> >
> > As mentioned in the original commit logs, older kernels need to use
> the
> > evil helper, newer kernels should 'just work' (thanks to the
> > relocatable
> > kernel support).
> >
> > Daniel
> > --
> > |: Red Hat, Engineering, London   -o-
> > http://people.redhat.com/berrange/ :|
> > |: http://libvirt.org  -o-  http://virt-manager.org  -o-
> > http://ovirt.org :|
> > |: http://autobuild.org       -o-
> > http://search.cpan.org/~danberr/ :|
> > |: GnuPG: 7D3B9505  -o-  F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B
> > 9505 :|

Re: HVM direct boot broken in xen-unstable

[Daniel P. Berrange]

On Mon, Jul 21, 2008 at 04:46:17PM +0100, Ian Jackson wrote:
> Daniel P. Berrange writes ("Re: [Xen-devel] HVM direct boot broken in xen-unstable"):
> > On Sat, Jul 19, 2008 at 12:52:07AM +0000, Byrne, John (HP Labs) wrote:
> > > Hi!
> > > 
> > > On x86_64, changeset 18081, running/building on rhel5, trying to use
> > > the HVM direct boot causes the domain to reboot immediately and then
> > > the log says the domain is restarting too rapidly. No obvious hints 
> > > as to where the problem is. If I build the in-tree ioemu code, things
> > > work. Any clues as to where to look or how to get some useful debugging
> > > output?
> 
> Byrne, could you send me a copy of your config file so I can reproduce
> this ?  Also, some idea of how you made (or how you obtained) the
> kernel image and initramfs.  I don't think I've ever done a direct
> kernel boot using this code (which is in patches supplied by Daniel)
> and offhand I can't seem to find the documentation.
> 
> > It appears the code is missing from the external ioemu code. The original
> > changesets for HVM kernel boot were
> 
> I've looked at the code and these changes are ones I did attempt to
> port across to the new qemu.  You'll see for example that the `evil
> helper' is still there, just slightly further down the file in pc.c.
> 
> I haven't ever executed the resulting codepaths and evidently I broke
> something, which is not all that surprising.

I think I've identified the problem. In the load_linux() method, the
line where we copy the header into guest memory:

    /* store the finalized header and load the rest of the kernel */
    cpu_physical_memory_write(real_addr, header, 1024);


Is done before the Xen hook runs:

    xen_relocator_hook(prot_addr, protocol, header, kernel_size,
                       real_addr, setup_size-1024);

The xen_relocator_hook modifies byte 0x214 in the header to point to
its hook, and so this change isn't getting written into memory.

The     cpu_physical_memory_write(real_addr, header, 1024) call just
needs to be moved down to be immediately after the call to the xen
hook.

Regards,
Daniel
-- 
|: Red Hat, Engineering, London   -o-   http://people.redhat.com/berrange/ :|
|: http://libvirt.org  -o-  http://virt-manager.org  -o-  http://ovirt.org :|
|: http://autobuild.org       -o-         http://search.cpan.org/~danberr/ :|
|: GnuPG: 7D3B9505  -o-  F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|

Re: HVM direct boot broken in xen-unstable

[Ian Jackson]

Daniel P. Berrange writes ("Re: [Xen-devel] HVM direct boot broken in xen-unstable"):
> It works in the same way as booting PV off kernel+initrd - basically 
> take a HVM guest config file and add [...]

Thanks.  With that I was able to reproduce the problem and I think I
have now fixed it in this changeset:

    commit a7104be2d6a521dfdcf1ce9f82ae40525e223775
    Author: Ian Jackson <ian.jackson@eu.citrix.com>
    Date:   Tue Jul 22 15:49:52 2008 +0100

	Fix HVM direct kernel booting.

This is available now (untested other than ad-hoc by me) at
   http://www.chiark.greenend.org.uk/~ijackson/qemu-xen.git
and will make it to the usual locations on xenbits in due course.

> I think I've identified the problem. In the load_linux() method, the
> line where we copy the header into guest memory:
> [etc]

This was indeed a problem but not the only one.

Ian.

RE: HVM direct boot broken in xen-unstable

[Byrne, John (HP Labs)]

I will test this sometime today. (As soon as I get my test system back.)

Thanks,

John

> -----Original Message-----
> From: Ian Jackson [mailto:Ian.Jackson@eu.citrix.com]
> Sent: Tuesday, July 22, 2008 7:52 AM
> To: Daniel P. Berrange
> Cc: Byrne, John (HP Labs); xen-devel@lists.xensource.com
> Subject: Re: [Xen-devel] HVM direct boot broken in xen-unstable
>
> Daniel P. Berrange writes ("Re: [Xen-devel] HVM direct boot broken in
> xen-unstable"):
> > It works in the same way as booting PV off kernel+initrd - basically
> > take a HVM guest config file and add [...]
>
> Thanks.  With that I was able to reproduce the problem and I think I
> have now fixed it in this changeset:
>
>     commit a7104be2d6a521dfdcf1ce9f82ae40525e223775
>     Author: Ian Jackson <ian.jackson@eu.citrix.com>
>     Date:   Tue Jul 22 15:49:52 2008 +0100
>
>         Fix HVM direct kernel booting.
>
> This is available now (untested other than ad-hoc by me) at
>    http://www.chiark.greenend.org.uk/~ijackson/qemu-xen.git
> and will make it to the usual locations on xenbits in due course.
>
> > I think I've identified the problem. In the load_linux() method, the
> > line where we copy the header into guest memory:
> > [etc]
>
> This was indeed a problem but not the only one.
>
> Ian.

RE: HVM direct boot broken in xen-unstable

[Byrne, John (HP Labs)]

There still seems to be an issue somewhere.

If I boot a "router" kernel with a combined kernel/initramfs (no ramdisk line), it successfully boots.
When I boot a rhel5 kernel, the kernel hangs at "checking if image is initramfs" saying it has bad gzip magic numbers.

John

> -----Original Message-----
> From: Ian Jackson [mailto:Ian.Jackson@eu.citrix.com]
> Sent: Tuesday, July 22, 2008 7:52 AM
> To: Daniel P. Berrange
> Cc: Byrne, John (HP Labs); xen-devel@lists.xensource.com
> Subject: Re: [Xen-devel] HVM direct boot broken in xen-unstable
>
> Daniel P. Berrange writes ("Re: [Xen-devel] HVM direct boot broken in
> xen-unstable"):
> > It works in the same way as booting PV off kernel+initrd - basically
> > take a HVM guest config file and add [...]
>
> Thanks.  With that I was able to reproduce the problem and I think I
> have now fixed it in this changeset:
>
>     commit a7104be2d6a521dfdcf1ce9f82ae40525e223775
>     Author: Ian Jackson <ian.jackson@eu.citrix.com>
>     Date:   Tue Jul 22 15:49:52 2008 +0100
>
>         Fix HVM direct kernel booting.
>
> This is available now (untested other than ad-hoc by me) at
>    http://www.chiark.greenend.org.uk/~ijackson/qemu-xen.git
> and will make it to the usual locations on xenbits in due course.
>
> > I think I've identified the problem. In the load_linux() method, the
> > line where we copy the header into guest memory:
> > [etc]
>
> This was indeed a problem but not the only one.
>
> Ian.

RE: HVM direct boot broken in xen-unstable

[Ian Jackson]

Byrne, John (HP Labs) writes ("RE: [Xen-devel] HVM direct boot broken in xen-unstable"):
> There still seems to be an issue somewhere.
> 
> If I boot a "router" kernel with a combined kernel/initramfs (no ramdisk line), it successfully boots.
> When I boot a rhel5 kernel, the kernel hangs at "checking if image is initramfs" saying it has bad gzip magic numbers.

I've tested Debian sarge 2.6.18-5-686 and Centos 5 2.6.23.1-42.fc8 and
both found the initramfs without any difficulty.

Can you perhaps email me your acutal kernel image and initramfs, and
of course your domain config file, and I could try to reproduce your
problem here ?

Thanks,
Ian.