Xen1.2 NetBSD port snapshot available and set_gdt patch for Xen1.2

Xen1.2 NetBSD port snapshot available and set_gdt patch for Xen1.2

[Christian Limpach]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 593 bytes --]

Hi!

I've updated the NetBSD port to Xen1.2.  Updated files are available at
http://lola.pin.lu/netbsd/xen/

See http://lola.pin.lu/netbsd/xen/readme-040203 for details.  Domain
creation is now possible using the xc_dom_*.py tools.

I've also found a bug in Xen1.2 and to run NetBSD you'll have to apply a
patch until this is fixed in the repository.  set_gdt() copies Xen's GDT
entries to the wrong page, this instantly kills the hypervisor when the
new GDT is activated.  Also the check for new GDT pages does always check
the first GDT page, which allows installing arbitrary GDT entries.


[-- Attachment #2: xen1.2_gdt.diff --]
[-- Type: TEXT/PLAIN, Size: 924 bytes --]

--- ../../xeno-1.2.bk/xen/arch/i386/mm.c	2004-01-21 14:57:32.000000000 +0100
+++ xen/arch/i386/mm.c	2004-02-02 21:35:35.000000000 +0100
@@ -238,7 +238,7 @@ long set_gdt(struct task_struct *p, 
                 goto out;
 
             /* Check all potential GDT entries in the page. */
-            gdt_page = map_domain_mem(frames[0] << PAGE_SHIFT);
+            gdt_page = map_domain_mem(frames[i] << PAGE_SHIFT);
             for ( j = 0; j < 512; j++ )
                 if ( !check_descriptor(gdt_page[j*2], gdt_page[j*2+1]) )
                     goto out;
@@ -274,7 +274,7 @@ long set_gdt(struct task_struct *p, 
     }
 
     /* Copy reserved GDT entries to the new GDT. */
-    vgdt = map_domain_mem(frames[i] << PAGE_SHIFT);
+    vgdt = map_domain_mem(frames[0] << PAGE_SHIFT);
     memcpy(vgdt + FIRST_RESERVED_GDT_ENTRY, 
            gdt_table + FIRST_RESERVED_GDT_ENTRY, 
            NR_RESERVED_GDT_ENTRIES*8);

[-- Attachment #3: Type: TEXT/PLAIN, Size: 38 bytes --]


-- 
Christian Limpach <chris@pin.lu>

Re: Xen1.2 NetBSD port snapshot available and set_gdt patch for Xen1.2

[Ian Pratt]

> I've updated the NetBSD port to Xen1.2.  Updated files are available at

Cool -- that was quick work.

I see that the linux domain building routine was close enough to
what you needed, so you didn't need a NetBSD specific routine.

How do you think it best to distribute this stuff?  We'd be
happy to put whatever you think appropriate in to the Xen tree.

> I've also found a bug in Xen1.2 and to run NetBSD you'll have to apply a
> patch until this is fixed in the repository.  set_gdt() copies Xen's GDT
> entries to the wrong page, this instantly kills the hypervisor when the
> new GDT is activated.  Also the check for new GDT pages does always check
> the first GDT page, which allows installing arbitrary GDT entries.

Oops. A couple of embarrassing ones there... 
I've applied the fix to the 1.2 tree and pushed it to
bk://xen.bkbits.net

Fortunately, the bug had already been expunged in the 1.3 tree
thanks to a complete rewrite of that bit of code.

Cheers,
Ian


-------------------------------------------------------
The SF.Net email is sponsored by EclipseCon 2004
Premiere Conference on Open Tools Development and Integration
See the breadth of Eclipse activity. February 3-5 in Anaheim, CA.
http://www.eclipsecon.org/osdn

Re: Xen1.2 NetBSD port snapshot available and set_gdt patch for Xen1.2

[Christian Limpach]

> I see that the linux domain building routine was close enough to
> what you needed, so you didn't need a NetBSD specific routine.

yes, that was neat.  I'm not happy with the hack of passing the symbol table
in as an initrd but I can fix that by embedding the symbol table into the
kernel image's data section.  That would also remove the dumpsym program
from the build step and then I can include the mangling of the kernel into a
XenoGues image into the regular build.  That would also make NetBSD/xen
cross-buildable, i.e. you can build it on a Linux host.

> How do you think it best to distribute this stuff?  We'd be
> happy to put whatever you think appropriate in to the Xen tree.

I think there should be a netbsd config file for the domain creation tools,
once I've made the above changes.
I don't think you'd want the whole kernel tree in the Xen tree?
You could put the sys/arch/xen files into the Xen tree and also the 5
additional files which need changes, but it's not really stable/useable yet.
Also, I'm waiting for the OK to commit all this to the NetBSD tree and then
it won't make much sense to have it in the Xen tree anymore.

    christian



-------------------------------------------------------
The SF.Net email is sponsored by EclipseCon 2004
Premiere Conference on Open Tools Development and Integration
See the breadth of Eclipse activity. February 3-5 in Anaheim, CA.
http://www.eclipsecon.org/osdn

Re: Xen1.2 NetBSD port snapshot available and set_gdt patch for Xen1.2

[Ian Pratt]

> I don't think you'd want the whole kernel tree in the Xen tree?
> You could put the sys/arch/xen files into the Xen tree and also the 5
> additional files which need changes, but it's not really stable/useable yet.
> Also, I'm waiting for the OK to commit all this to the NetBSD tree and then
> it won't make much sense to have it in the Xen tree anymore.

If you can get the changes into the main stream NetBSD tree that
would be great. It would would set a useful precedent for
persuading Linus to do the same in 2.6...
 
> I think there should be a netbsd config file for the domain creation tools,
> once I've made the above changes.

Having the netbsd config file and any extra build tools in the
Xen tree would seem sensible.

If it looked like there was a problem getting the kernel mods
into the netbsd tree we could either put a patch or 'sparse tree'
in the Xen tree in the meantime.

Cheers,
Ian


-------------------------------------------------------
The SF.Net email is sponsored by EclipseCon 2004
Premiere Conference on Open Tools Development and Integration
See the breadth of Eclipse activity. February 3-5 in Anaheim, CA.
http://www.eclipsecon.org/osdn

Re: Xen1.2 NetBSD port snapshot available and set_gdt patch for Xen1.2

[Rolf Neugebauer]

On Tue, 2004-02-03 at 08:42, Ian Pratt wrote:
> > I don't think you'd want the whole kernel tree in the Xen tree?
> > You could put the sys/arch/xen files into the Xen tree and also the 5
> > additional files which need changes, but it's not really stable/useable yet.
> > Also, I'm waiting for the OK to commit all this to the NetBSD tree and then
> > it won't make much sense to have it in the Xen tree anymore.
> 
> If you can get the changes into the main stream NetBSD tree that
> would be great. It would would set a useful precedent for
> persuading Linus to do the same in 2.6...

I agree it would be great to get this into the NetBSD tree. Christian,
what is the maintenance model for ports in NetBSD? Are the NetBSD folks
fairly easy to convince (I'm more familiar with the FreeBSD model)? Is
it easy for a maintainer to become a commiter to the CVS etc.? 

It leaves the slight problem on how to deal with the shared files
(hypervisor-if.h and friends). If they are not in the main NetBSD tree
the port won't compile. If the NetbSD source contains a copy it is more
difficult to maintain consistency. I'm more in favour of having a copy
in the NetBSD tree as it allows compilation directly from the CVS. To
address the consistency issue maybe we should add a version number to
hypervisor-if.h and friends and pass that down either as a separate
hypercall (i.e., a new domain has to 'register' with Xen) or as part of
a infrequently use hypercall (like set_trap_table). we can do the same
for the 'device driver' interface.

 
> > I think there should be a netbsd config file for the domain creation tools,
> > once I've made the above changes.
> 
> Having the netbsd config file and any extra build tools in the
> Xen tree would seem sensible.
> 
> If it looked like there was a problem getting the kernel mods
> into the netbsd tree we could either put a patch or 'sparse tree'
> in the Xen tree in the meantime.

that's certainly the way to go. However, if I understand Christian
correct, the port is against the -current (i.e. the development tree)
which, if it is similar to the FreeBSD model, may not have
snapshots/version numbers we can base the sparse tree against. 

Christian, do you have an idea how far -current has diverged from
-stable in the parts of the tree which matter? or, i fact how often
relavant files in -current change wrt to your changes/patches.

I'm going to give the 1.2 port a spin later today or tomorrow and let
you know how it works out.

Cheers
Rolf



> Cheers,
> Ian
> 
> 
> -------------------------------------------------------
> The SF.Net email is sponsored by EclipseCon 2004
> Premiere Conference on Open Tools Development and Integration
> See the breadth of Eclipse activity. February 3-5 in Anaheim, CA.
> http://www.eclipsecon.org/osdn
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/xen-devel



-------------------------------------------------------
The SF.Net email is sponsored by EclipseCon 2004
Premiere Conference on Open Tools Development and Integration
See the breadth of Eclipse activity. February 3-5 in Anaheim, CA.
http://www.eclipsecon.org/osdn

Re: Xen1.2 NetBSD port snapshot available and set_gdtpatch for Xen1.2

[Christian Limpach]

> I agree it would be great to get this into the NetBSD tree. Christian,
> what is the maintenance model for ports in NetBSD? Are the NetBSD folks
> fairly easy to convince (I'm more familiar with the FreeBSD model)? Is
> it easy for a maintainer to become a commiter to the CVS etc.?

I have commit access to the NetBSD CVS and I can sponsor new people.

> It leaves the slight problem on how to deal with the shared files
> (hypervisor-if.h and friends). If they are not in the main NetBSD tree
> the port won't compile. If the NetbSD source contains a copy it is more
> difficult to maintain consistency. I'm more in favour of having a copy
> in the NetBSD tree as it allows compilation directly from the CVS.

yes, I've chosen to include the interface header files.  We have
autobuilders which regularly build all ports and those only work if
everything is included.

> To
> address the consistency issue maybe we should add a version number to
> hypervisor-if.h and friends and pass that down either as a separate
> hypercall (i.e., a new domain has to 'register' with Xen) or as part of
> a infrequently use hypercall (like set_trap_table). we can do the same
> for the 'device driver' interface.

Or try to keep the interface changes to a minimum and keep backwards
compatibility.  I know that's sometimes a pain but I've also found that it
leads to better interfaces if you have to be careful when adding/changing
interfaces...

> that's certainly the way to go. However, if I understand Christian
> correct, the port is against the -current (i.e. the development tree)
> which, if it is similar to the FreeBSD model, may not have
> snapshots/version numbers we can base the sparse tree against.

yes, it's against -current.  We have a kernel version number (bumped when
there are changes to exported kernel structures) but there are no snapshots
based on that version number.  I think there are weekly unversioned
snapshots on ftp.netbsd.org.

> Christian, do you have an idea how far -current has diverged from
> -stable in the parts of the tree which matter? or, i fact how often
> relavant files in -current change wrt to your changes/patches.

It's diverged quite a bit since there hasn't been a release in a long time.
I think the relevant files don't change often or rather the interfaces to
the rest of the kernel don't change often because such changes would require
changes in all ports.  Once the port is in the tree it won't get any more
out of sync than any other port.

There should be a release soon (NetBSD 2.0) but there's no momentum yet to
make it happen...

> I'm going to give the 1.2 port a spin later today or tomorrow and let
> you know how it works out.

just be sure to have an updated 1.2 with the gdt fix...

    christian



-------------------------------------------------------
The SF.Net email is sponsored by EclipseCon 2004
Premiere Conference on Open Tools Development and Integration
See the breadth of Eclipse activity. February 3-5 in Anaheim, CA.
http://www.eclipsecon.org/osdn

A query..

[I RATTAN]

I am not clear about the capabilities of Xen, hence the question:

Is it possible to run multiple instances of Linux on the same
box? If so, is there a howto available?

Thanks in advance.

-ishwar




-------------------------------------------------------
The SF.Net email is sponsored by EclipseCon 2004
Premiere Conference on Open Tools Development and Integration
See the breadth of Eclipse activity. February 3-5 in Anaheim, CA.
http://www.eclipsecon.org/osdn

Re: A query..

[Yan-Ching CHU]

> I am not clear about the capabilities of Xen, hence the question:

> Is it possible to run multiple instances of Linux on the same
> box? If so, is there a howto available?
>

Yes. Take look at the following links.

Xen General:
http://www.cl.cam.ac.uk/netos/xen

Bin Ren's Howto for Xen 1.1. For 1.2 version go to BK unstable repository,
instructions for downloading from BK can be found here as well (Hi Bin do
you think you can make the 1.2 version available other than in BK only?
Thanks):
http://www.cl.cam.ac.uk/~br260/doc/xeno-1.1-howto.txt

Jans' Xen FAQ:
http://xen.epiuse.com/xen-faq.txt

My howto:
http://www.csc.liv.ac.uk/~u2ycc/xen/xen-install-domain-fs-howto.txt

Xen for Grid by Greg Newby:
http://www.arsc.edu/~newby/grid/xen-for-grid.txt

Cheers,
Yan-Ching CHU



-------------------------------------------------------
The SF.Net email is sponsored by EclipseCon 2004
Premiere Conference on Open Tools Development and Integration
See the breadth of Eclipse activity. February 3-5 in Anaheim, CA.
http://www.eclipsecon.org/osdn

Re: A query..

[Bin Ren]

On 5 Feb 2004, at 13:24, Yan-Ching CHU wrote:

> Bin Ren's Howto for Xen 1.1. For 1.2 version go to BK unstable 
> repository,
> instructions for downloading from BK can be found here as well (Hi Bin 
> do
> you think you can make the 1.2 version available other than in BK only?
> Thanks):
> http://www.cl.cam.ac.uk/~br260/doc/xeno-1.1-howto.txt

1.2 now also available at:

http://www.cl.cam.ac.uk/~br260/doc/xeno-1.2-howto.txt

Again, the credit of updating the howto to version 1.2 goes to Mark 
Williamson.

-- Bin



-------------------------------------------------------
The SF.Net email is sponsored by EclipseCon 2004
Premiere Conference on Open Tools Development and Integration
See the breadth of Eclipse activity. February 3-5 in Anaheim, CA.
http://www.eclipsecon.org/osdn

Re: Xen1.2 NetBSD port snapshot available and set_gdtpatch for Xen1.2

[Keir Fraser]

> > It leaves the slight problem on how to deal with the shared files
> > (hypervisor-if.h and friends). If they are not in the main NetBSD tree
> > the port won't compile. If the NetbSD source contains a copy it is more
> > difficult to maintain consistency. I'm more in favour of having a copy
> > in the NetBSD tree as it allows compilation directly from the CVS.
> 
> yes, I've chosen to include the interface header files.  We have
> autobuilders which regularly build all ports and those only work if
> everything is included.

Having the interface headers copied into the guest source tree makes
sense when building completely separately from Xen. For safety we
should add a version number to the interface header files and use that
to check the guest-OS build against the running Xen at guest boot
time. We've avoided this kind of issue up to now since we've only
had Linux fully ported to Xen so building the two together made
sense. 

> > address the consistency issue maybe we should add a version number to
> > hypervisor-if.h and friends and pass that down either as a separate
> > hypercall (i.e., a new domain has to 'register' with Xen) or as part of
> > a infrequently use hypercall (like set_trap_table). we can do the same
> > for the 'device driver' interface.
> 
> Or try to keep the interface changes to a minimum and keep backwards
> compatibility.  I know that's sometimes a pain but I've also found that it
> leads to better interfaces if you have to be careful when adding/changing
> interfaces...

Although some parts are now pretty stable, others are going to be in
flux for a while yet. For example -- the I/O interface is likely to
change somewhat when device drivers are pulled out into isolated
domains. Also new bidirectional console support; the CPU-management
interface (when we add support for SMP guests); ... In general we're
treating the existing interface as a rough prototype and improving it
as we address the weaknesses in each Xen subsystem (e.g., I/O, memory
management, CPU abstraction). This work is still ongoing. However, we
will try harder to minimise the effect on guest OSes now that others
are using the Xen interfaces. :-)

 -- Keir


-------------------------------------------------------
The SF.Net email is sponsored by EclipseCon 2004
Premiere Conference on Open Tools Development and Integration
See the breadth of Eclipse activity. February 3-5 in Anaheim, CA.
http://www.eclipsecon.org/osdn

Re: Xen1.2 NetBSD port snapshot available and set_gdt patch for Xen1.2

[Keir Fraser]

> > I see that the linux domain building routine was close enough to
> > what you needed, so you didn't need a NetBSD specific routine.
> 
> yes, that was neat.  I'm not happy with the hack of passing the symbol table
> in as an initrd but I can fix that by embedding the symbol table into the
> kernel image's data section.  That would also remove the dumpsym program
> from the build step and then I can include the mangling of the kernel into a
> XenoGues image into the regular build.  That would also make NetBSD/xen
> cross-buildable, i.e. you can build it on a Linux host.

Would it be useful to have simnple ELF loading support in the domain
builder? This would get rid of teh need for symtab munging and the
cheesy XenoGues stuff when building NetBSD.

> > How do you think it best to distribute this stuff?  We'd be
> > happy to put whatever you think appropriate in to the Xen tree.
> 
> I think there should be a netbsd config file for the domain creation tools,
> once I've made the above changes.
> I don't think you'd want the whole kernel tree in the Xen tree?
> You could put the sys/arch/xen files into the Xen tree and also the 5
> additional files which need changes, but it's not really stable/useable yet.
> Also, I'm waiting for the OK to commit all this to the NetBSD tree and then
> it won't make much sense to have it in the Xen tree anymore.

If/when upgrading to 1.3, please be aware of a couple of interface
changes (there will also likely be more, as we add device-driver
isolation and proper bidirectional console support for example):
 1. The MMU_update interface takes a physical pointer to a PTE, not a
 virtual address.
 2. The block-device and network I/O rings are now indexed using
 non-wrapping counters. e.g. rather than
   i = (i + 1) % RING_SIZE;
   ent = ring[i];
 you do
   ent = ring[++i];
 There are predefined index types NET_RING_IDX and BLK_RING_IDX.

The former change was required because I stripped locking out of the
page-management code and it's hard to ensure that the virtual-address
space doesn't change under your feet without using locks.

The latter change makes I/O easier to debug, and gets rid of nasty "is
this ring empty or full?" corner cases.

 -- Keir


-------------------------------------------------------
The SF.Net email is sponsored by EclipseCon 2004
Premiere Conference on Open Tools Development and Integration
See the breadth of Eclipse activity. February 3-5 in Anaheim, CA.
http://www.eclipsecon.org/osdn

Re: Xen1.2 NetBSD port snapshot available and set_gdt patch for Xen1.2

[Christian Limpach]

> > yes, that was neat.  I'm not happy with the hack of passing the symbol
table
> > in as an initrd but I can fix that by embedding the symbol table into
the
> > kernel image's data section.  That would also remove the dumpsym program
> > from the build step and then I can include the mangling of the kernel
into a
> > XenoGues image into the regular build.  That would also make NetBSD/xen
> > cross-buildable, i.e. you can build it on a Linux host.
>
> Would it be useful to have simnple ELF loading support in the domain
> builder? This would get rid of teh need for symtab munging and the
> cheesy XenoGues stuff when building NetBSD.

yes, although the loader will have to get the setup right:  we need an ELF
header in front of the symbol table, i.e. the layout after loading should
be:
    ...
__bss_start:
    bss space
_end:
    somewhat mangled ELF header
    STRTAB/SYMTAB sections (only the STRTAB sections which are referenced by
a SYMTAB section)
esym:

Right now the dumpsym program extracts the mangled header and the required
sections, we pass this in as an initrd and the kernel then copies the initrd
to _end and initializes esym from MOD_LEN.  This is fine except that it
needs 2 files and that's annoying.

I would add our loader to the domain builder except that it has a 4 clause
BSD licence and I don't know if you want code in Xen which has the
advertising clause.  The loader is in the NetBSD tree at
sys/lib/libsa/loadfile_elf32.c.  grub also includes a loader which sets
things up correctly...

> If/when upgrading to 1.3, please be aware of a couple of interface
> changes (there will also likely be more, as we add device-driver
> isolation and proper bidirectional console support for example):
>  1. The MMU_update interface takes a physical pointer to a PTE, not a
>  virtual address.

I think I have all/most of these in macros/functions.

>  2. The block-device and network I/O rings are now indexed using
>  non-wrapping counters. e.g. rather than
>    i = (i + 1) % RING_SIZE;
>    ent = ring[i];
>  you do
>    ent = ring[++i];
>  There are predefined index types NET_RING_IDX and BLK_RING_IDX.

That's ent = ring[MASK_NET_{R,T}X_IDX(++i)], right?
I think I'll use non-wrapping counters when writing the block-device driver
and I'll apply the mask when accessing the ring counters...

    christian



-------------------------------------------------------
The SF.Net email is sponsored by EclipseCon 2004
Premiere Conference on Open Tools Development and Integration
See the breadth of Eclipse activity. February 3-5 in Anaheim, CA.
http://www.eclipsecon.org/osdn

Re: Xen1.2 NetBSD port snapshot available and set_gdt patch for Xen1.2

[Keir Fraser]

> yes, although the loader will have to get the setup right:  we need an ELF
> header in front of the symbol table, i.e. the layout after loading should
> be:
>     ...
> __bss_start:
>     bss space
> _end:
>     somewhat mangled ELF header
>     STRTAB/SYMTAB sections (only the STRTAB sections which are referenced by
> a SYMTAB section)
> esym:
> 
> Right now the dumpsym program extracts the mangled header and the required
> sections, we pass this in as an initrd and the kernel then copies the initrd
> to _end and initializes esym from MOD_LEN.  This is fine except that it
> needs 2 files and that's annoying.
> 
> I would add our loader to the domain builder except that it has a 4 clause
> BSD licence and I don't know if you want code in Xen which has the
> advertising clause.  The loader is in the NetBSD tree at
> sys/lib/libsa/loadfile_elf32.c.  grub also includes a loader which sets
> things up correctly...

Yes, I took a look at the GRUB code. It wouldn't be hard to add an
xc_netbsd_build() function I think. Most of the code would be in
common with the Linux builder.

> >  2. The block-device and network I/O rings are now indexed using
> >  non-wrapping counters. e.g. rather than
> >    i = (i + 1) % RING_SIZE;
> >    ent = ring[i];
> >  you do
> >    ent = ring[++i];
> >  There are predefined index types NET_RING_IDX and BLK_RING_IDX.
> 
> That's ent = ring[MASK_NET_{R,T}X_IDX(++i)], right?
> I think I'll use non-wrapping counters when writing the block-device driver
> and I'll apply the mask when accessing the ring counters...

Oops, yes!

 -- Keir


-------------------------------------------------------
The SF.Net email is sponsored by EclipseCon 2004
Premiere Conference on Open Tools Development and Integration
See the breadth of Eclipse activity. February 3-5 in Anaheim, CA.
http://www.eclipsecon.org/osdn