Xen PXE solution

Xen PXE solution

[Andrew D. Ball]

Greetings.

I've been asked to work on a PXE solution for booting Xen domU's from a
network, and would like some feedback and any help anyone is willing to
provide.

I've been limiting my work to HVM domU's at the moment, because I don't
think the NBI's (as far as I know, these are bootloaders) that I have
available work in paravirtualized domU's.  Something build around the
Linux kernel and kexec would be really cool for paravirtualized domU's.

So, I'd like to know if something like the PXE ROM's that come on most
servers nowadays would be more useful than doing something like booting
from PXE client ISO image.  Ideally, I think the configuration may look
something like this:

(device 
  (vif
    (bridge xenbr0) (mac 00:16:3e:00:00:11) (type ioemu) 
    (pxe true)
  )
)

Then, I'm thinking the behavior would be to boot PXE on that NIC by
default, then try other boot devices.  Of course, the 'boot' option that
currently accepts 'a','b', or 'c' could include pxe as an option, but
it's unclear to me which order the NICs would try to boot in.  Maybe all
of the NICs with PXE ROMs in configuration order.

It looks like Etherboot 5.4.2 has at least the beginnings of full PXE
support for some drivers.  AMD pcnet32 looked okay, but I don't have a
really demanding PXE environment for testing.

Many thanks for your help!
Andrew
=====
Andrew D. Ball <aball@us dot ibm dot com>

Re: Xen PXE solution

[Ben Thomas]

[-- Attachment #1.1: Type: text/plain, Size: 2140 bytes --]

Hi Andrew,

For what it's worth, we've been using a PXE solution based upon etherboot.
It's effectively a floppy-like boot image that works quite well.  My memory
is that there were some instruction decode/emulation issues on VMX, but that
we've submitted all of those patches to this list. I believe that they were
incorporated into xen-unstable.

This approach has been working quite well for us for a while now, but may
not completely address all of your needs.

fwiw,
-b


On 7/31/06, Andrew D. Ball <aball@us.ibm.com> wrote:
>
> Greetings.
>
> I've been asked to work on a PXE solution for booting Xen domU's from a
> network, and would like some feedback and any help anyone is willing to
> provide.
>
> I've been limiting my work to HVM domU's at the moment, because I don't
> think the NBI's (as far as I know, these are bootloaders) that I have
> available work in paravirtualized domU's.  Something build around the
> Linux kernel and kexec would be really cool for paravirtualized domU's.
>
> So, I'd like to know if something like the PXE ROM's that come on most
> servers nowadays would be more useful than doing something like booting
> from PXE client ISO image.  Ideally, I think the configuration may look
> something like this:
>
> (device
>   (vif
>     (bridge xenbr0) (mac 00:16:3e:00:00:11) (type ioemu)
>     (pxe true)
>   )
> )
>
> Then, I'm thinking the behavior would be to boot PXE on that NIC by
> default, then try other boot devices.  Of course, the 'boot' option that
> currently accepts 'a','b', or 'c' could include pxe as an option, but
> it's unclear to me which order the NICs would try to boot in.  Maybe all
> of the NICs with PXE ROMs in configuration order.
>
> It looks like Etherboot 5.4.2 has at least the beginnings of full PXE
> support for some drivers.  AMD pcnet32 looked okay, but I don't have a
> really demanding PXE environment for testing.
>
> Many thanks for your help!
> Andrew
> =====
> Andrew D. Ball <aball@us dot ibm dot com>
>
>
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
>

[-- Attachment #1.2: Type: text/html, Size: 2731 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: Xen PXE solution

[Henning Sprang]

Hi  Ben,

On 8/1/06, Ben Thomas <ben@virtualiron.com> wrote:
> [...]
>  For what it's worth, we've been using a PXE solution based upon etherboot.
> It's effectively a floppy-like boot image that works quite well.  My memory
> is that there were some instruction decode/emulation issues on VMX, but that
> we've submitted all of those patches to this list. I believe that they were
> incorporated into xen-unstable.

This is probably more for the xen-users list, but here is the only
post I found about somebody doing etherboot with xen hvm, which I am
trying currently without luck.
Does your mail mean that with xen-testing at the moment my efforts to
try booting xen hvm with an etherboot image and pxe cannot succeed?

When booting from the etherboot iso, I get
"Loading pcnet32.zli....Ready"

but no dhcp requests or anything happens.  xm dmesg's last lines are:

(XEN) (GUEST: 60) Booting from CD-Rom...
(XEN) (GUEST: 60) Unknown opcode at 0F60:0E8A=0x1048A
(XEN) (GUEST: 60) Halt called from %eip 0xD39D1

No interesting info xend-debug.log

I'd be a candidate for testing xen pxe solutions, because I really
want to use it.

Henning

Re: Xen PXE solution

[Ben Thomas]

Hi Henning,

It's been a little while, and my recollection is rusty.  I do
know that we ended up fixing some opcode emulation when
getting the PXE booting to work.  I was pretty sure that we
had posted that to the xen-devel list, but again I plead
a faulty memory on the specifics.

I do know that we are constantly (hundreds to thousands of
times per day) booting many, many xen-based systems via PXE
for both VMX and SVM systems. Currently, we're using the
pcnet32 support.  As Ian will (or probably already did) point
out, it's probably worthwhile to move forward.

I'd be happy to give you more specifics on what we're doing
to get PXE booted.  I doubt that the entire list cares, so
please send me an email if you're interested.

-b


Henning Sprang wrote:
> Hi  Ben,
> 
> On 8/1/06, Ben Thomas <ben@virtualiron.com> wrote:
> 
>> [...]
>>  For what it's worth, we've been using a PXE solution based upon 
>> etherboot.
>> It's effectively a floppy-like boot image that works quite well.  My 
>> memory
>> is that there were some instruction decode/emulation issues on VMX, 
>> but that
>> we've submitted all of those patches to this list. I believe that they 
>> were
>> incorporated into xen-unstable.
> 
> 
> This is probably more for the xen-users list, but here is the only
> post I found about somebody doing etherboot with xen hvm, which I am
> trying currently without luck.
> Does your mail mean that with xen-testing at the moment my efforts to
> try booting xen hvm with an etherboot image and pxe cannot succeed?
> 
> When booting from the etherboot iso, I get
> "Loading pcnet32.zli....Ready"
> 
> but no dhcp requests or anything happens.  xm dmesg's last lines are:
> 
> (XEN) (GUEST: 60) Booting from CD-Rom...
> (XEN) (GUEST: 60) Unknown opcode at 0F60:0E8A=0x1048A
> (XEN) (GUEST: 60) Halt called from %eip 0xD39D1
> 
> No interesting info xend-debug.log
> 
> I'd be a candidate for testing xen pxe solutions, because I really
> want to use it.
> 
> Henning


-- 
------------------------------------------------------------------------
Ben Thomas                                         Virtual Iron Software
bthomas@virtualiron.com                            Tower 1, Floor 2
978-849-1214                                       900 Chelmsford Street
                                                    Lowell, MA 01851

Re: Xen PXE solution

[Jeremy Katz]

On Mon, 2006-07-31 at 10:57 -0400, Andrew D. Ball wrote:
> I've been asked to work on a PXE solution for booting Xen domU's from a
> network, and would like some feedback and any help anyone is willing to
> provide.

Cool

> I've been limiting my work to HVM domU's at the moment, because I don't
> think the NBI's (as far as I know, these are bootloaders) that I have
> available work in paravirtualized domU's.  Something build around the
> Linux kernel and kexec would be really cool for paravirtualized domU's.

HVM is definitely the first step here.  

> So, I'd like to know if something like the PXE ROM's that come on most
> servers nowadays would be more useful than doing something like booting
> from PXE client ISO image.  Ideally, I think the configuration may look
> something like this:

*nod*  So are you looking at integrating into the rombios?

> (device 
>   (vif
>     (bridge xenbr0) (mac 00:16:3e:00:00:11) (type ioemu) 
>     (pxe true)
>   )
> )

Quite possibly PXE would even default to true.

> Then, I'm thinking the behavior would be to boot PXE on that NIC by
> default, then try other boot devices.  Of course, the 'boot' option that
> currently accepts 'a','b', or 'c' could include pxe as an option, but
> it's unclear to me which order the NICs would try to boot in.  Maybe all
> of the NICs with PXE ROMs in configuration order.

I think it's definitely better to go with pxe as an option to boot and
then doing the NICs in config order.  This matches up with how real
hardware works[1]

Jeremy

[1] Also -- does anyone know if the boot option currently take multiple
arguments?  I'll have to try when I get to the office.  It should be a
list and fall back through them as things aren't available so that a
default order of say PXE, CD, HD can be set up and be nice and useful :)

Re: Xen PXE solution

[Andrew D. Ball]

As far as I know, the boot order option only accepts one of
{'a','b','c'}.  I more descriptive list like you suggested would be
much, much better.

Peace.
Andrew


On Wed, 2006-08-02 at 08:42 -0400, Jeremy Katz wrote:
> On Mon, 2006-07-31 at 10:57 -0400, Andrew D. Ball wrote:
> > I've been asked to work on a PXE solution for booting Xen domU's from a
> > network, and would like some feedback and any help anyone is willing to
> > provide.
> 
> Cool
> 
> > I've been limiting my work to HVM domU's at the moment, because I don't
> > think the NBI's (as far as I know, these are bootloaders) that I have
> > available work in paravirtualized domU's.  Something build around the
> > Linux kernel and kexec would be really cool for paravirtualized domU's.
> 
> HVM is definitely the first step here.  
> 
> > So, I'd like to know if something like the PXE ROM's that come on most
> > servers nowadays would be more useful than doing something like booting
> > from PXE client ISO image.  Ideally, I think the configuration may look
> > something like this:
> 
> *nod*  So are you looking at integrating into the rombios?
> 
> > (device 
> >   (vif
> >     (bridge xenbr0) (mac 00:16:3e:00:00:11) (type ioemu) 
> >     (pxe true)
> >   )
> > )
> 
> Quite possibly PXE would even default to true.
> 
> > Then, I'm thinking the behavior would be to boot PXE on that NIC by
> > default, then try other boot devices.  Of course, the 'boot' option that
> > currently accepts 'a','b', or 'c' could include pxe as an option, but
> > it's unclear to me which order the NICs would try to boot in.  Maybe all
> > of the NICs with PXE ROMs in configuration order.
> 
> I think it's definitely better to go with pxe as an option to boot and
> then doing the NICs in config order.  This matches up with how real
> hardware works[1]
> 
> Jeremy
> 
> [1] Also -- does anyone know if the boot option currently take multiple
> arguments?  I'll have to try when I get to the office.  It should be a
> list and fall back through them as things aren't available so that a
> default order of say PXE, CD, HD can be set up and be nice and useful :)
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
> 

Re: Xen PXE solution

[Andrew D. Ball]

Good morning.

Here's how far I got before I was swamped by other work:

-- generated a ROM image for the pcnet32 driver from etherboot. (was
before the realtek device)
-- loaded the image in different parts of memory that is scanned by the 
ROMBIOS for expansion ROMS
-- Noticed what was probably memory corruption errors preventing the PXE
ROM from loading and running

This dropped down to the bottom of my priorities, as it's possible to
get the same functionality with etherboot as a virtual ISO image instead
of an expansion ROM.  This would still be useful as an expansion ROM
though, IMHO.

The most important work to be done that I can think of before this can
work is to clearly document which parts of memory are occupied by
what firmware for HVM domU's and proceed with troubleshooting the
memory corruption.

If you'd like, we can set up a wiki page to work on this.

Hope this helps.
Andrew

On Mon, 2006-10-09 at 11:57 -0400, Gregory Haskins wrote:
> >>> On Mon, Jul 31, 2006 at 10:57 AM, in message
> <1154357859.9876.14.camel@localhost.localdomain>, "Andrew D. Ball"
> <aball@us.ibm.com> wrote: 
> > Greetings.
> > 
> > I've been asked to work on a PXE solution for booting Xen domU's from a
> > network, and would like some feedback and any help anyone is willing to
> > provide.
> 
> Hi Andrew,
>   I am just wondering what the status of this project is?   I have the need for this type of functionality as well.  We could possibly collaborate on this effort if work still remains to be done.
> 
> -Greg Haskins
> Novell
> 

Re: Xen PXE solution

[Andrew D. Ball]

On Mon, 2006-10-16 at 13:46 -0400, Ben Thomas wrote:
> 
>         The most important work to be done that I can think of before
>         this can
>         work is to clearly document which parts of memory are occupied
>         by
>         what firmware for HVM domU's and proceed with troubleshooting
>         the
>         memory corruption.
> 
>  
> 
> 
> Is this along the lines of what you are looking for ?
> 

Sure is!  Didn't know this was logged -- it's been awhile since I worked
on this part.

I think it would be useful to document this layout somewhere, probably
on the wiki.  I saw an SMBIOS fix that scared me, because it used
0xc0000 as a scrap area.  Shouldn't hurt anything, since the VGA BIOS is
loaded after the SMBIOS tables, but scares me nonetheless because I
don't know of any documentation for the HVM virtual firmware other than
the code and the logs.

Peace.
Andrew

> (XEN) (GUEST: 6) HVM Loader
> (XEN) (GUEST: 6) Detected Xen v3.0-unstable
> (XEN) (GUEST: 6) Writing SMBIOS tables [0009f000 - 0009f140] ...
> (XEN) (GUEST: 6) Loading ROMBIOS [000f0000 - 00100000] ...
> (XEN) (GUEST: 6) Creating MP tables ...
> (XEN) (GUEST: 6) Loading Cirrus VGABIOS [000c0000 - 000c8000] ...
> (XEN) (GUEST: 6) Loading ACPI [000ea000 - 000eb000] ...
> (XEN) (GUEST: 6) Loading VMXAssist [000d0000 - 000d5200] ...
> (XEN) (GUEST: 6) VMX go ...
> 
> I have some questions/concerns about the layout of memory, which is
> why I have this info.
> 
> -b

RE: Xen PXE solution

[Ben Thomas]

[-- Attachment #1.1: Type: text/plain, Size: 2432 bytes --]

Hi Andrew,

I added the address logging to check out a few questions I had.
The use of 0C0000 is interesting, but as you note, it's safe as
it's used as a scratch area before it's filled in with the
vga bios.

My main concern is with the use of memory below 0A0000.  I'm
chasing a problem in an OS, and it sure looks like a problem
I submitted a fix for (to the OS, not this project) about a
year back. It was related to too little free memory in the
first 1MB. Hence, my interest in the layout, especially the
use of memory below 0A0000. It's likely that my suspicions are
incorrect, but curiousity usually wins.

Anyhow, I saw your note and thought I'd pass on what little I
had about it.

-b


-----Original Message-----
From: Andrew D. Ball [mailto:aball@us.ibm.com]
Sent: Mon 10/16/2006 3:33 PM
To: Ben Thomas; xen-devel@lists.xensource.com
Subject: Re: [Xen-devel] Xen PXE solution
 
On Mon, 2006-10-16 at 13:46 -0400, Ben Thomas wrote:
> 
>         The most important work to be done that I can think of before
>         this can
>         work is to clearly document which parts of memory are occupied
>         by
>         what firmware for HVM domU's and proceed with troubleshooting
>         the
>         memory corruption.
> 
>  
> 
> 
> Is this along the lines of what you are looking for ?
> 

Sure is!  Didn't know this was logged -- it's been awhile since I worked
on this part.

I think it would be useful to document this layout somewhere, probably
on the wiki.  I saw an SMBIOS fix that scared me, because it used
0xc0000 as a scrap area.  Shouldn't hurt anything, since the VGA BIOS is
loaded after the SMBIOS tables, but scares me nonetheless because I
don't know of any documentation for the HVM virtual firmware other than
the code and the logs.

Peace.
Andrew

> (XEN) (GUEST: 6) HVM Loader
> (XEN) (GUEST: 6) Detected Xen v3.0-unstable
> (XEN) (GUEST: 6) Writing SMBIOS tables [0009f000 - 0009f140] ...
> (XEN) (GUEST: 6) Loading ROMBIOS [000f0000 - 00100000] ...
> (XEN) (GUEST: 6) Creating MP tables ...
> (XEN) (GUEST: 6) Loading Cirrus VGABIOS [000c0000 - 000c8000] ...
> (XEN) (GUEST: 6) Loading ACPI [000ea000 - 000eb000] ...
> (XEN) (GUEST: 6) Loading VMXAssist [000d0000 - 000d5200] ...
> (XEN) (GUEST: 6) VMX go ...
> 
> I have some questions/concerns about the layout of memory, which is
> why I have this info.
> 
> -b



[-- Attachment #1.2: Type: text/html, Size: 3513 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

RE: Xen PXE solution

[Ian Pratt]

> It looks like Etherboot 5.4.2 has at least the beginnings of 
> full PXE support for some drivers.  AMD pcnet32 looked okay, 
> but I don't have a really demanding PXE environment for testing.

We'll have to see whether vmxassist is up to the job of running the 16b
etherboot code on VT hosts. Also, focusing on etherboot rlt8139 support
rather than ne2000 or pcnet32 would be better.

Best,
Ian

RE: Xen PXE solution

[Ian Pratt]

> This is probably more for the xen-users list, but here is the only
> post I found about somebody doing etherboot with xen hvm, which I am
> trying currently without luck.
> Does your mail mean that with xen-testing at the moment my efforts to
> try booting xen hvm with an etherboot image and pxe cannot succeed?

Tim Deegan this working with the pcnet32 driver some time back, but
right now I'd be inclined to try it with an etherboot with an rtl8139
driver (now the default).  

Ian
 
> When booting from the etherboot iso, I get
> "Loading pcnet32.zli....Ready"
> 
> but no dhcp requests or anything happens.  xm dmesg's last lines are:
> 
> (XEN) (GUEST: 60) Booting from CD-Rom...
> (XEN) (GUEST: 60) Unknown opcode at 0F60:0E8A=0x1048A
> (XEN) (GUEST: 60) Halt called from %eip 0xD39D1
> 
> No interesting info xend-debug.log
> 
> I'd be a candidate for testing xen pxe solutions, because I really
> want to use it.
> 
> Henning
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel