[bug] pv-grub doesn't run on rhel5

[bug] pv-grub doesn't run on rhel5

[Gerd Hoffmann]

  Hi,

Trying to boot pv-grub on RHEL-5 makes it fail with

  [ ... ]
  gnttab_table mapped at 0x10001000.
  Initialising scheduler
  (XEN) mm.c:694:d6 Bad L3 flags 6
  ERROR: mmu_update failed
  Do_exit called!

Guess more recent xen versions are less picky here and just silenly
fixup this?  This is with 32bit pae.  64bit pvgrub runs fine.

Another issue:  As 64bit pv-grub can't boot 32bit kernels, the 64bit xen
build should create both 32bit and 64bit builds of pv-grub, so you can
boot 32-on-64 guests with it?

cheers,
  Gerd

-- 
http://kraxel.fedorapeople.org/xenner/

Re: [bug] pv-grub doesn't run on rhel5

[Samuel Thibault]

Gerd Hoffmann, le Thu 14 Aug 2008 12:38:15 +0200, a écrit :
> Trying to boot pv-grub on RHEL-5 makes it fail with
> 
>   [ ... ]
>   gnttab_table mapped at 0x10001000.
>   Initialising scheduler
>   (XEN) mm.c:694:d6 Bad L3 flags 6
>   ERROR: mmu_update failed
>   Do_exit called!

It looks like it's mini-os itself that has troubles very early.

It works in my environment.

> Guess more recent xen versions are less picky here and just silenly
> fixup this?

No, the check is still there, and I wonder how flags 6 could make it to
L3 flags.  Could you send me your mini-os.gz image?

> This is with 32bit pae.

Is it a 64 bit or 32 bit hypervisor?  Which version of the hypervisor?

> Another issue:  As 64bit pv-grub can't boot 32bit kernels, the 64bit xen
> build should create both 32bit and 64bit builds of pv-grub, so you can
> boot 32-on-64 guests with it?

I'm afraid Keir will consider it is too late in the 3.3 release to make
such a change in the build process.

Actually I am currently working on how we could switch a PV domain
between 32 and 64.  That won't ever go into 3.3 either, however.

Samuel

Re: [bug] pv-grub doesn't run on rhel5

[Samuel Thibault]

Gerd Hoffmann, le Thu 14 Aug 2008 14:20:59 +0200, a écrit :
> Samuel Thibault wrote:
> >> Guess more recent xen versions are less picky here and just silenly
> >> fixup this?
> > 
> > No, the check is still there, and I wonder how flags 6 could make it to
> > L3 flags.  Could you send me your mini-os.gz image?
> 
> Attached.

I have no problem running it under xen 3.3 indeed.  Are you able to run
the 3.3 mini-os itself for instance?

Samuel

Re: [bug] pv-grub doesn't run on rhel5

[Gerd Hoffmann]

Samuel Thibault wrote:
> Gerd Hoffmann, le Thu 14 Aug 2008 14:20:59 +0200, a écrit :
>> Samuel Thibault wrote:
>>>> Guess more recent xen versions are less picky here and just silenly
>>>> fixup this?
>>> No, the check is still there, and I wonder how flags 6 could make it to
>>> L3 flags.  Could you send me your mini-os.gz image?
>> Attached.
> 
> I have no problem running it under xen 3.3 indeed.  Are you able to run
> the 3.3 mini-os itself for instance?

That runs fine.  It doesn't touch address space above 1gb though, thus
it has no need to touch the pae pgd.

cheers,
  Gerd

-- 
http://kraxel.fedorapeople.org/xenner/

Re: [bug] pv-grub doesn't run on rhel5

[Samuel Thibault]

Gerd Hoffmann, le Thu 14 Aug 2008 16:18:21 +0200, a écrit :
> Samuel Thibault wrote:
> > Gerd Hoffmann, le Thu 14 Aug 2008 14:20:59 +0200, a écrit :
> >> Samuel Thibault wrote:
> >>>> Guess more recent xen versions are less picky here and just silenly
> >>>> fixup this?
> >>> No, the check is still there, and I wonder how flags 6 could make it to
> >>> L3 flags.  Could you send me your mini-os.gz image?
> >> Attached.
> > 
> > I have no problem running it under xen 3.3 indeed.  Are you able to run
> > the 3.3 mini-os itself for instance?
> 
> That runs fine.  It doesn't touch address space above 1gb though, thus
> it has no need to touch the pae pgd.

That's possible indeed.

The problem is a bug in the hypervisor's mod_l3_entry which calls
adjust_guest_l3e before calling get_page_from_l3e, see c/s 17061.  Is it
possible for you to backport the patch?

Samuel

Re: [bug] pv-grub doesn't run on rhel5

[Gerd Hoffmann]

Samuel Thibault wrote:
> The problem is a bug in the hypervisor's mod_l3_entry which calls
> adjust_guest_l3e before calling get_page_from_l3e, see c/s 17061.  Is it
> possible for you to backport the patch?

So due to the wrong ordering xen first sets the bits, then notices they
shouldn't be set, then complains the guest passed invalid bits?  Uhm,
well, no way to fix minios to run on older xen versions then I guess
(which I'd prefer).  Except maybe limiting the 32bit version to 1GB
address space.  Which should be fine at least for pvgrub ...

Another pvgrub issue:  Is booting guests with the framebuffer enabled
supposed to work?  I suspect that needs some work on the qemu side.
Right now I see the pvgrub screen just fine, after booting the guest
kernel I see funny, scrambled pictures though.  I suspect that needs
some work on the qemu side, so it remaps the framebuffer correctly.

cheers,
  Gerd

-- 
http://kraxel.fedorapeople.org/xenner/

Re: [bug] pv-grub doesn't run on rhel5

[Samuel Thibault]

Gerd Hoffmann, le Thu 14 Aug 2008 17:04:07 +0200, a écrit :
> Samuel Thibault wrote:
> > The problem is a bug in the hypervisor's mod_l3_entry which calls
> > adjust_guest_l3e before calling get_page_from_l3e, see c/s 17061.  Is it
> > possible for you to backport the patch?
> 
> So due to the wrong ordering xen first sets the bits, then notices they
> shouldn't be set, then complains the guest passed invalid bits?

Yes.

> Uhm, well, no way to fix minios to run on older xen versions then I
> guess (which I'd prefer).  Except maybe limiting the 32bit version to
> 1GB address space.  Which should be fine at least for pvgrub ...

That should be feasible indeed.

> Another pvgrub issue:  Is booting guests with the framebuffer enabled
> supposed to work?

Yes, but you have to not enable the framebuffer in the grub
configuration, because the framebuffer is currently not able to restart.

> I suspect that needs some work on the qemu side.

Exactly.

Samuel

Re: [bug] pv-grub doesn't run on rhel5

[Samuel Thibault]

Samuel Thibault, le Thu 14 Aug 2008 16:14:00 +0100, a écrit :
> Gerd Hoffmann, le Thu 14 Aug 2008 17:04:07 +0200, a écrit :
> > Uhm, well, no way to fix minios to run on older xen versions then I
> > guess (which I'd prefer).  Except maybe limiting the 32bit version to
> > 1GB address space.  Which should be fine at least for pvgrub ...
> 
> That should be feasible indeed.

As seen below.  I'm not sure whether we want that into mainline Xen 3.3
however?

BTW, Keir commited the 32on64 pv-grub build, it should appear after the
regression tests.

Samuel



pv-grub: limit address space usage to less than a GB so as to be
runnable on xen hypervisor revisions earlier than c/s 17061.

Signed-off-by: Samuel Thibault <samuel.thibault@eu.citrix.com>

diff -r 38783464a671 extras/mini-os/arch/x86/mm.c
--- a/extras/mini-os/arch/x86/mm.c	Thu Aug 14 16:16:44 2008 +0100
+++ b/extras/mini-os/arch/x86/mm.c	Thu Aug 14 16:30:59 2008 +0100
@@ -414,18 +414,22 @@ pgentry_t *need_pgt(unsigned long addr)
 }
 
 static unsigned long demand_map_area_start;
+#ifndef DEMAND_MAP_PAGES
 #ifdef __x86_64__
 #define DEMAND_MAP_PAGES ((128ULL << 30) / PAGE_SIZE)
 #else
 #define DEMAND_MAP_PAGES ((2ULL << 30) / PAGE_SIZE)
 #endif
+#endif
 
 #ifdef HAVE_LIBC
 unsigned long heap, brk, heap_mapped, heap_end;
+#ifndef HEAP_PAGES
 #ifdef __x86_64__
 #define HEAP_PAGES ((128ULL << 30) / PAGE_SIZE)
 #else
 #define HEAP_PAGES ((1ULL << 30) / PAGE_SIZE)
+#endif
 #endif
 #endif
 
diff -r 38783464a671 stubdom/Makefile
--- a/stubdom/Makefile	Thu Aug 14 16:16:44 2008 +0100
+++ b/stubdom/Makefile	Thu Aug 14 16:30:59 2008 +0100
@@ -33,6 +33,7 @@ TARGET_CFLAGS=
 TARGET_CFLAGS=
 NEWLIB_CFLAGS+=-D_I386MACH_ALLOW_HW_INTERRUPTS
 STUBDOM_SUPPORTED=1
+PVGRUB_CPPFLAGS=-DDEMAND_MAP_PAGES='((256ULL << 20) / PAGE_SIZE)' -DHEAP_PAGES='((256ULL << 20) / PAGE_SIZE)'
 endif
 ifeq ($(GNU_TARGET_ARCH), x86_64)
 TARGET_CFLAGS=-mno-red-zone
@@ -296,7 +297,7 @@ c-stubdom: mini-os-$(XEN_TARGET_ARCH)-c 
 
 .PHONY: pv-grub
 pv-grub: mini-os-$(XEN_TARGET_ARCH)-grub libxc grub
-	DEF_CPPFLAGS="$(TARGET_CPPFLAGS)" DEF_CFLAGS="-DCONFIG_GRUB $(TARGET_CFLAGS)" DEF_LDFLAGS="$(TARGET_LDFLAGS)" $(MAKE) -C $(MINI_OS) OBJ_DIR=$(CURDIR)/$< APP_OBJS=$(CURDIR)/grub-$(XEN_TARGET_ARCH)/main.a
+	DEF_CPPFLAGS="$(TARGET_CPPFLAGS) $(PVGRUB_CPPFLAGS)" DEF_CFLAGS="-DCONFIG_GRUB $(TARGET_CFLAGS)" DEF_LDFLAGS="$(TARGET_LDFLAGS)" $(MAKE) -C $(MINI_OS) OBJ_DIR=$(CURDIR)/$< APP_OBJS=$(CURDIR)/grub-$(XEN_TARGET_ARCH)/main.a
 
 #########
 # install

Re: [bug] pv-grub doesn't run on rhel5

[Gerd Hoffmann]

  Hi,

>> Another pvgrub issue:  Is booting guests with the framebuffer enabled
>> supposed to work?
> 
> Yes, but you have to not enable the framebuffer in the grub
> configuration, because the framebuffer is currently not able to restart.
> 
>> I suspect that needs some work on the qemu side.
> 
> Exactly.

I'll have a look I guess as I'm busy with qemu coding anyway ...

cheers,
  Gerd

-- 
http://kraxel.fedorapeople.org/xenner/

[PATCH] xenfb: make restartable [Was: pv-grub doesn't run on rhel5]

[Samuel Thibault]

Samuel Thibault, le Thu 14 Aug 2008 16:14:00 +0100, a écrit :
> > Another pvgrub issue:  Is booting guests with the framebuffer enabled
> > supposed to work?
> 
> Yes, but you have to not enable the framebuffer in the grub
> configuration, because the framebuffer is currently not able to restart.

Oh actually it was a lot easier than I thought, see patch below (which
is actually partly bug fixes).



xenfb: make restartable

- Fix the pixel unmapping, which should happen during the Closing state.
- Fix qemu segfault when a guest shuts its fb down.
- Once connected, if frontend state goes from Closed to Initialized,
restart the connection loop.

Signed-off-by: Samuel Thibault <samuel.thibault@eu.citrix.com>

diff -r 38783464a671 extras/mini-os/fbfront.c
--- a/extras/mini-os/fbfront.c	Thu Aug 14 16:16:44 2008 +0100
+++ b/extras/mini-os/fbfront.c	Thu Aug 14 18:59:27 2008 +0100
@@ -226,8 +226,7 @@ void shutdown_kbdfront(struct kbdfront_d
     xenbus_wait_for_value(path, "6", &dev->events);
 
     err = xenbus_printf(XBT_NIL, nodename, "state", "%u", 1);
-    // does not work yet.
-    //xenbus_wait_for_value(path, "2", &dev->events);
+    xenbus_wait_for_value(path, "2", &dev->events);
 
     xenbus_unwatch_path(XBT_NIL, path);
 
@@ -566,8 +565,7 @@ void shutdown_fbfront(struct fbfront_dev
     xenbus_wait_for_value(path, "6", &dev->events);
 
     err = xenbus_printf(XBT_NIL, nodename, "state", "%u", 1);
-    // does not work yet
-    //xenbus_wait_for_value(path, "2", &dev->events);
+    xenbus_wait_for_value(path, "2", &dev->events);
 
     xenbus_unwatch_path(XBT_NIL, path);
 
diff -r 38783464a671 stubdom/README
--- a/stubdom/README	Thu Aug 14 16:16:44 2008 +0100
+++ b/stubdom/README	Thu Aug 14 18:59:27 2008 +0100
@@ -126,9 +126,4 @@ Limitations
 ===========
 
 - You can not boot a 64bit kernel with a 32bit-compiled PV-GRUB and vice-versa.
-To cross-compile a 32bit PV-GRUB,
-
-export XEN_TARGET_ARCH=x86_32
-
-- bootsplash is supported, but the ioemu backend does not yet support restart
-for use by the booted kernel.
+You need to choose between pv-grub-x86_64.gz and pv-grub-x86_32.gz
diff -r 38783464a671 tools/ioemu/hw/xenfb.c
--- a/tools/ioemu/hw/xenfb.c	Thu Aug 14 16:16:44 2008 +0100
+++ b/tools/ioemu/hw/xenfb.c	Thu Aug 14 18:59:27 2008 +0100
@@ -445,6 +445,13 @@ static void xenfb_unbind(struct xenfb_de
 		xc_evtchn_unbind(dev->xenfb->evt_xch, dev->port);
 		dev->port = -1;
 	}
+
+	if (!strcmp(dev->devicetype, "vfb")) {
+	    if (dev->xenfb->pixels) {
+		    munmap(dev->xenfb->pixels, dev->xenfb->fb_len);
+		    dev->xenfb->pixels = NULL;
+	    }
+	}
 }
 
 
@@ -452,10 +459,6 @@ static void xenfb_detach_dom(struct xenf
 {
 	xenfb_unbind(&xenfb->fb);
 	xenfb_unbind(&xenfb->kbd);
-	if (xenfb->pixels) {
-		munmap(xenfb->pixels, xenfb->fb_len);
-		xenfb->pixels = NULL;
-	}
 }
 
 /* Remove the backend area in xenbus since the framebuffer really is
@@ -653,6 +656,16 @@ static int xenfb_on_state_change(struct 
 		   not much point in us continuing. */
 		return -1;
 	case XenbusStateInitialising:
+		if (dev->state != XenbusStateClosed)
+		    /* Do not let a domain make us skip the closing state */
+		    return 0;
+		xenfb_switch_state(dev, XenbusStateInitWait);
+		xs_unwatch(dev->xenfb->xsh, dev->otherend, "");
+		if (!strcmp(dev->devicetype, "vkbd")) {
+		    fprintf(stderr, "FB: Waiting for KBD backend creation\n");
+		    xenfb_wait_for_frontend(&dev->xenfb->kbd, xenfb_frontend_initialized_kbd);
+		}
+		break;
 	case XenbusStateInitWait:
 	case XenbusStateInitialised:
 	case XenbusStateConnected:
@@ -1274,6 +1287,9 @@ static void xenfb_update(void *opaque)
     struct xenfb *xenfb = opaque;
     int period;
 
+    if (!xenfb->fb.page)
+        return;
+
     if (xenfb_queue_full(xenfb))
         return;
 

Re: [PATCH] xenfb: make restartable [Was: pv-grub doesn't run on rhel5]

[Gerd Hoffmann]

Samuel Thibault wrote:
> Samuel Thibault, le Thu 14 Aug 2008 16:14:00 +0100, a écrit :
>>> Another pvgrub issue:  Is booting guests with the framebuffer enabled
>>> supposed to work?
>> Yes, but you have to not enable the framebuffer in the grub
>> configuration, because the framebuffer is currently not able to restart.
> 
> Oh actually it was a lot easier than I thought, see patch below (which
> is actually partly bug fixes).

Yea, looks reasonable, I've ended up with something quite simliar
(against the new framebuffer code).

cheers,
  Gerd

-- 
http://kraxel.fedorapeople.org/xenner/

Re: [PATCH] xenfb: make restartable

[Markus Armbruster]

Samuel Thibault <samuel.thibault@eu.citrix.com> writes:

> Samuel Thibault, le Thu 14 Aug 2008 16:14:00 +0100, a écrit :
>> > Another pvgrub issue:  Is booting guests with the framebuffer enabled
>> > supposed to work?
>> 
>> Yes, but you have to not enable the framebuffer in the grub
>> configuration, because the framebuffer is currently not able to restart.
>
> Oh actually it was a lot easier than I thought, see patch below (which
> is actually partly bug fixes).
>
>
>
> xenfb: make restartable
>
> - Fix the pixel unmapping, which should happen during the Closing state.
> - Fix qemu segfault when a guest shuts its fb down.
> - Once connected, if frontend state goes from Closed to Initialized,
> restart the connection loop.
>
> Signed-off-by: Samuel Thibault <samuel.thibault@eu.citrix.com>
>
> diff -r 38783464a671 extras/mini-os/fbfront.c
> --- a/extras/mini-os/fbfront.c	Thu Aug 14 16:16:44 2008 +0100
> +++ b/extras/mini-os/fbfront.c	Thu Aug 14 18:59:27 2008 +0100
[Not familiar with this code...]
> diff -r 38783464a671 tools/ioemu/hw/xenfb.c
> --- a/tools/ioemu/hw/xenfb.c	Thu Aug 14 16:16:44 2008 +0100
> +++ b/tools/ioemu/hw/xenfb.c	Thu Aug 14 18:59:27 2008 +0100
> @@ -445,6 +445,13 @@ static void xenfb_unbind(struct xenfb_de
>  		xc_evtchn_unbind(dev->xenfb->evt_xch, dev->port);
>  		dev->port = -1;
>  	}
> +
> +	if (!strcmp(dev->devicetype, "vfb")) {
> +	    if (dev->xenfb->pixels) {
> +		    munmap(dev->xenfb->pixels, dev->xenfb->fb_len);
> +		    dev->xenfb->pixels = NULL;
> +	    }
> +	}
>  }
>  
>  

The check for vfb is ugly.  The only alternative I can see is fb ==
dev->xenfb->fb.  Matter of taste.

Also somewhat ugly is the fact that xenfb_unbind() is no longer the
exact reverse of xenfb_bind(), but reverses xenfb_map_fb() as well.
Could be avoided by creating xenfb_unmap_fb() and calling it after
xenfb_unbind() from xenfb_detach_dom() unconditionally, and from
xenfb_on_state_change() when dev is fb.

> @@ -452,10 +459,6 @@ static void xenfb_detach_dom(struct xenf
>  {
>  	xenfb_unbind(&xenfb->fb);
>  	xenfb_unbind(&xenfb->kbd);
> -	if (xenfb->pixels) {
> -		munmap(xenfb->pixels, xenfb->fb_len);
> -		xenfb->pixels = NULL;
> -	}
>  }
>  
>  /* Remove the backend area in xenbus since the framebuffer really is
> @@ -653,6 +656,16 @@ static int xenfb_on_state_change(struct 
>  		   not much point in us continuing. */
>  		return -1;
>  	case XenbusStateInitialising:
> +		if (dev->state != XenbusStateClosed)
> +		    /* Do not let a domain make us skip the closing state */
> +		    return 0;

Why does this work?

> +		xenfb_switch_state(dev, XenbusStateInitWait);
> +		xs_unwatch(dev->xenfb->xsh, dev->otherend, "");
> +		if (!strcmp(dev->devicetype, "vkbd")) {
> +		    fprintf(stderr, "FB: Waiting for KBD backend creation\n");
> +		    xenfb_wait_for_frontend(&dev->xenfb->kbd, xenfb_frontend_initialized_kbd);
> +		}
> +		break;

Man, I hate this state machine...  not your fault.

Gerd's new one is much clearer (I asked for that).

>  	case XenbusStateInitWait:
>  	case XenbusStateInitialised:
>  	case XenbusStateConnected:
> @@ -1274,6 +1287,9 @@ static void xenfb_update(void *opaque)
>      struct xenfb *xenfb = opaque;
>      int period;
>  
> +    if (!xenfb->fb.page)
> +        return;
> +
>      if (xenfb_queue_full(xenfb))
>          return;
>  
>

Re: [PATCH] xenfb: make restartable

[Samuel Thibault]

Markus Armbruster, le Fri 15 Aug 2008 08:39:06 -0400, a écrit :
> > +	if (!strcmp(dev->devicetype, "vfb")) {
> > +	    if (dev->xenfb->pixels) {
> > +		    munmap(dev->xenfb->pixels, dev->xenfb->fb_len);
> > +		    dev->xenfb->pixels = NULL;
> > +	    }
> > +	}
> >  }
> >  
> >  
> 
> The check for vfb is ugly.  The only alternative I can see is fb ==
> dev->xenfb->fb.  Matter of taste.

Hum, that was what I had written previously actually.  I don't see why
it is "ugly", in that pixels is a device resource.  It should probably
be in xenfb->fb actually.

> > @@ -653,6 +656,16 @@ static int xenfb_on_state_change(struct 
> >  		   not much point in us continuing. */
> >  		return -1;
> >  	case XenbusStateInitialising:
> > +		if (dev->state != XenbusStateClosed)
> > +		    /* Do not let a domain make us skip the closing state */
> > +		    return 0;
> 
> Why does this work?

It's meant to not work.  If the frontend is not playing well (going
through the closing state), then ignore what it's doing until it plays
well. An alternative could be to just shut down.  In any case, we don't
want to let it drive us to remapping something else without unmapping
the previous fb.

> > +		xenfb_switch_state(dev, XenbusStateInitWait);
> > +		xs_unwatch(dev->xenfb->xsh, dev->otherend, "");
> > +		if (!strcmp(dev->devicetype, "vkbd")) {
> > +		    fprintf(stderr, "FB: Waiting for KBD backend creation\n");
> > +		    xenfb_wait_for_frontend(&dev->xenfb->kbd, xenfb_frontend_initialized_kbd);
> > +		}
> > +		break;
> 
> Man, I hate this state machine... 

I agree :)

> Gerd's new one is much clearer (I asked for that).

Then since 3.3 is about to be released, we'd probably have this change
with it.

Samuel

Re: [PATCH] xenfb: make restartable

[Markus Armbruster]

Samuel Thibault <samuel.thibault@eu.citrix.com> writes:

> Markus Armbruster, le Fri 15 Aug 2008 08:39:06 -0400, a écrit :
>> > +	if (!strcmp(dev->devicetype, "vfb")) {
>> > +	    if (dev->xenfb->pixels) {
>> > +		    munmap(dev->xenfb->pixels, dev->xenfb->fb_len);
>> > +		    dev->xenfb->pixels = NULL;
>> > +	    }
>> > +	}
>> >  }
>> >  
>> >  
>> 
>> The check for vfb is ugly.  The only alternative I can see is fb ==
>> dev->xenfb->fb.  Matter of taste.
>
> Hum, that was what I had written previously actually.  I don't see why
> it is "ugly", in that pixels is a device resource.  It should probably
> be in xenfb->fb actually.

struct xenfb_device captures stuff that both devices have.

By "is ugly" I meant to say that checking for vfb by testing its
devicetype string looks ugly to me.  But fb == dev->xenfb->fb isn't
exactly the acme of elegance either.

>> > @@ -653,6 +656,16 @@ static int xenfb_on_state_change(struct 
>> >  		   not much point in us continuing. */
>> >  		return -1;
>> >  	case XenbusStateInitialising:
>> > +		if (dev->state != XenbusStateClosed)
>> > +		    /* Do not let a domain make us skip the closing state */
>> > +		    return 0;
>> 
>> Why does this work?
>
> It's meant to not work.  If the frontend is not playing well (going
> through the closing state), then ignore what it's doing until it plays
> well. An alternative could be to just shut down.  In any case, we don't
> want to let it drive us to remapping something else without unmapping
> the previous fb.

We need to make up our mind how to handle invalid state transitions.

Is silently ignoring guest misbehavior smart?  I'd rather log it.

[...]

Re: [PATCH] xenfb: make restartable

[Samuel Thibault]

Markus Armbruster, le Fri 15 Aug 2008 18:53:01 -0400, a écrit :
> We need to make up our mind how to handle invalid state transitions.
> 
> Is silently ignoring guest misbehavior smart?  I'd rather log it.

Why not indeed, but beware of logging too much.

Samuel

Re: [bug] pv-grub doesn't run on rhel5

[Jeremy Fitzhardinge]

Samuel Thibault wrote:
> The problem is a bug in the hypervisor's mod_l3_entry which calls
> adjust_guest_l3e before calling get_page_from_l3e, see c/s 17061.  Is it
> possible for you to backport the patch?
>   

Fedora 8 just updated their Xen package because the pvops kernels also 
trigger this bug.

    J

Re: [bug] pv-grub doesn't run on rhel5

[Gerd Hoffmann]

Jeremy Fitzhardinge wrote:
> Samuel Thibault wrote:
>> The problem is a bug in the hypervisor's mod_l3_entry which calls
>> adjust_guest_l3e before calling get_page_from_l3e, see c/s 17061.  Is it
>> possible for you to backport the patch?
>>   
> 
> Fedora 8 just updated their Xen package because the pvops kernels also
> trigger this bug.

Then rhel-5 will probably fixed too.  Not being able to run pvops
kernels isn't going to fly.  Having the 1GB limit for pvgrub would be
nice nevertheless.

cheers,
  Gerd

-- 
http://kraxel.fedorapeople.org/xenner/

[PATCH] compiled a 32bit pv-grub on x86_64 xen target [Was: pv-grub doesn't run on rhel5]

[Samuel Thibault]

Samuel Thibault, le Thu 14 Aug 2008 12:15:20 +0100, a écrit :
> Gerd Hoffmann, le Thu 14 Aug 2008 12:38:15 +0200, a écrit :
> > Another issue:  As 64bit pv-grub can't boot 32bit kernels, the 64bit xen
> > build should create both 32bit and 64bit builds of pv-grub, so you can
> > boot 32-on-64 guests with it?
> 
> I'm afraid Keir will consider it is too late in the 3.3 release to make
> such a change in the build process.

Here is a patch anyway, it doesn't touch any code so at least shouldn't
break anything.



pv-grub: On x86_64, also build an x86_32 pv-grub

This requires suffixing obj directories and having grub compiled outside
sources.

Signed-off-by: Samuel Thibault <samuel.thibault@eu.citrix.com>

diff -r 37f3bdce0394 .hgignore
--- a/.hgignore	Mon Aug 11 12:36:26 2008 +0100
+++ b/.hgignore	Thu Aug 14 13:30:23 2008 +0100
@@ -90,20 +90,16 @@
 ^stubdom/gcc-.*$
 ^stubdom/include$
 ^stubdom/ioemu$
-^stubdom/libxc$
+^stubdom/libxc-.*$
 ^stubdom/lwip-.*$
 ^stubdom/mini-os-.*$
-^stubdom/mk-headers$
+^stubdom/mk-headers-.*$
 ^stubdom/newlib-.*$
 ^stubdom/pciutils-.*$
 ^stubdom/zlib-.*$
-^stubdom/grub-cvs$
-^stubdom/grub/stage2$
-^stubdom/grub/netboot$
-^stubdom/grub/dirs$
+^stubdom/grub-.*$
 ^stubdom/lwip/
 ^stubdom/ioemu/
-^stubdom/grub-upstream/
 ^tools/.*/build/lib.*/.*\.py$
 ^tools/blktap/Makefile\.smh$
 ^tools/blktap/drivers/blktapctrl$
diff -r 37f3bdce0394 Makefile
--- a/Makefile	Mon Aug 11 12:36:26 2008 +0100
+++ b/Makefile	Thu Aug 14 13:30:23 2008 +0100
@@ -22,6 +22,9 @@ build: kernels
 	$(MAKE) -C xen build
 	$(MAKE) -C tools build
 	$(MAKE) -C stubdom build
+ifeq (x86_64,$(XEN_TARGET_ARCH))
+	XEN_TARGET_ARCH=x86_32 $(MAKE) -C stubdom pv-grub
+endif
 	$(MAKE) -C docs build
 
 # The test target is for unit tests that can run without an installation.  Of
@@ -71,6 +74,9 @@ install-kernels:
 .PHONY: install-stubdom
 install-stubdom:
 	$(MAKE) -C stubdom install
+ifeq (x86_64,$(XEN_TARGET_ARCH))
+	XEN_TARGET_ARCH=x86_32 $(MAKE) -C stubdom install-grub
+endif
 
 .PHONY: install-docs
 install-docs:
@@ -109,6 +115,9 @@ clean::
 	$(MAKE) -C xen clean
 	$(MAKE) -C tools clean
 	$(MAKE) -C stubdom crossclean
+ifeq (x86_64,$(XEN_TARGET_ARCH))
+	XEN_TARGET_ARCH=x86_32 $(MAKE) -C stubdom crossclean
+endif
 	$(MAKE) -C docs clean
 
 # clean, but blow away kernel build tree plus tarballs
@@ -117,6 +126,9 @@ distclean:
 	$(MAKE) -C xen distclean
 	$(MAKE) -C tools distclean
 	$(MAKE) -C stubdom distclean
+ifeq (x86_64,$(XEN_TARGET_ARCH))
+	XEN_TARGET_ARCH=x86_32 $(MAKE) -C stubdom distclean
+endif
 	$(MAKE) -C docs distclean
 	rm -rf dist patches/tmp
 	for i in $(ALLKERNELS) ; do $(MAKE) $$i-delete ; done
diff -r 37f3bdce0394 extras/mini-os/Makefile
--- a/extras/mini-os/Makefile	Mon Aug 11 12:36:26 2008 +0100
+++ b/extras/mini-os/Makefile	Thu Aug 14 13:30:23 2008 +0100
@@ -66,8 +66,8 @@ ifeq ($(lwip),y)
 # lwIP library
 LWC	:= $(shell find $(LWIPDIR)/ -type f -name '*.c')
 LWC	:= $(filter-out %6.c %ip6_addr.c %ethernetif.c, $(LWC))
-LWC	+= lwip-arch.c lwip-net.c
 LWO	:= $(patsubst %.c,%.o,$(LWC))
+LWO	+= $(addprefix $(OBJ_DIR)/,lwip-arch.o lwip-net.o)
 
 $(OBJ_DIR)/lwip.a: $(LWO)
 	$(RM) $@
@@ -79,7 +79,7 @@ OBJS := $(filter-out $(OBJ_DIR)/lwip%.o 
 OBJS := $(filter-out $(OBJ_DIR)/lwip%.o $(LWO), $(OBJS))
 
 ifeq ($(libc),y)
-APP_LDLIBS += -L$(XEN_ROOT)/stubdom/libxc -whole-archive -lxenguest -lxenctrl -no-whole-archive
+APP_LDLIBS += -L$(XEN_ROOT)/stubdom/libxc-$(XEN_TARGET_ARCH) -whole-archive -lxenguest -lxenctrl -no-whole-archive
 APP_LDLIBS += -lpci
 APP_LDLIBS += -lz
 APP_LDLIBS += -lm
diff -r 37f3bdce0394 stubdom/Makefile
--- a/stubdom/Makefile	Mon Aug 11 12:36:26 2008 +0100
+++ b/stubdom/Makefile	Thu Aug 14 13:30:23 2008 +0100
@@ -91,9 +91,9 @@ NEWLIB_STAMPFILE=$(CROSS_ROOT)/$(GNU_TAR
 NEWLIB_STAMPFILE=$(CROSS_ROOT)/$(GNU_TARGET_ARCH)-xen-elf/lib/libc.a
 .PHONY: cross-newlib
 cross-newlib: $(NEWLIB_STAMPFILE)
-$(NEWLIB_STAMPFILE): mk-headers newlib-$(NEWLIB_VERSION)
-	mkdir -p newlib-build
-	( cd newlib-build && \
+$(NEWLIB_STAMPFILE): mk-headers-$(XEN_TARGET_ARCH) newlib-$(NEWLIB_VERSION)
+	mkdir -p newlib-$(XEN_TARGET_ARCH)
+	( cd newlib-$(XEN_TARGET_ARCH) && \
 	  CC_FOR_TARGET="$(CC) $(TARGET_CPPFLAGS) $(TARGET_CFLAGS) $(NEWLIB_CFLAGS)" AR_FOR_TARGET=$(AR) LD_FOR_TARGET=$(LD) RANLIB_FOR_TARGET=$(RANLIB) ../newlib-$(NEWLIB_VERSION)/configure --prefix=$(CROSS_PREFIX) --verbose --target=$(GNU_TARGET_ARCH)-xen-elf --enable-newlib-io-long-long --disable-multilib && \
 	  $(MAKE) && \
 	  DESTDIR= $(MAKE) install )
@@ -105,12 +105,15 @@ zlib-$(ZLIB_VERSION).tar.gz:
 zlib-$(ZLIB_VERSION).tar.gz:
 	$(WGET) $(ZLIB_URL)/$@
 
+zlib-$(XEN_TARGET_ARCH): zlib-$(ZLIB_VERSION).tar.gz 
+	tar xzf $<
+	mv zlib-$(ZLIB_VERSION) $@
+
 ZLIB_STAMPFILE=$(CROSS_ROOT)/$(GNU_TARGET_ARCH)-xen-elf/lib/libz.a
 .PHONY: cross-zlib
 cross-zlib: $(ZLIB_STAMPFILE)
-$(ZLIB_STAMPFILE): zlib-$(ZLIB_VERSION).tar.gz $(NEWLIB_STAMPFILE)
-	tar xzf $<
-	( cd zlib-$(ZLIB_VERSION) && \
+$(ZLIB_STAMPFILE): zlib-$(XEN_TARGET_ARCH) $(NEWLIB_STAMPFILE)
+	( cd $< && \
 	  CFLAGS="$(TARGET_CPPFLAGS) $(TARGET_CFLAGS)" CC=$(CC) ./configure --prefix=$(CROSS_PREFIX)/$(GNU_TARGET_ARCH)-xen-elf && \
 	  $(MAKE) libz.a && \
 	  $(MAKE) install )
@@ -122,16 +125,17 @@ pciutils-$(LIBPCI_VERSION).tar.bz2:
 pciutils-$(LIBPCI_VERSION).tar.bz2:
 	$(WGET) $(LIBPCI_URL)/$@
 
-pciutils-$(LIBPCI_VERSION): pciutils-$(LIBPCI_VERSION).tar.bz2
+pciutils-$(XEN_TARGET_ARCH): pciutils-$(LIBPCI_VERSION).tar.bz2
 	tar xjf $<
+	mv pciutils-$(LIBPCI_VERSION) $@
 	patch -d $@ -p1 < pciutils.patch
 	touch $@
 
 LIBPCI_STAMPFILE=$(CROSS_ROOT)/$(GNU_TARGET_ARCH)-xen-elf/lib/libpci.a
 .PHONY: cross-libpci
 cross-libpci: $(LIBPCI_STAMPFILE)
-$(LIBPCI_STAMPFILE): pciutils-$(LIBPCI_VERSION) $(NEWLIB_STAMPFILE) $(ZLIB_STAMPFILE)
-	( cd pciutils-$(LIBPCI_VERSION) && \
+$(LIBPCI_STAMPFILE): pciutils-$(XEN_TARGET_ARCH) $(NEWLIB_STAMPFILE) $(ZLIB_STAMPFILE)
+	( cd $< && \
 	  cp ../libpci.config.h lib/config.h && \
 	  echo '#define PCILIB_VERSION "$(LIBPCI_VERSION)"' >> lib/config.h && \
 	  cp ../libpci.config.mak lib/config.mk && \
@@ -148,8 +152,9 @@ lwip-$(LWIP_VERSION).tar.gz:
 lwip-$(LWIP_VERSION).tar.gz:
 	$(WGET) $(LWIP_URL)/$@
 
-lwip: lwip-$(LWIP_VERSION).tar.gz
+lwip-$(XEN_TARGET_ARCH): lwip-$(LWIP_VERSION).tar.gz
 	tar xzf $<
+	mv lwip $@
 	patch -d $@ -p0 < lwip.patch-cvs
 	touch $@
 
@@ -160,7 +165,7 @@ lwip: lwip-$(LWIP_VERSION).tar.gz
 .PHONY: $(CROSS_ROOT)
 $(CROSS_ROOT): cross-newlib cross-zlib cross-libpci
 
-mk-headers:
+mk-headers-$(XEN_TARGET_ARCH):
 	mkdir -p include/xen && \
           ln -sf $(addprefix ../../,$(wildcard $(XEN_ROOT)/xen/include/public/*.h)) include/xen && \
           ln -sf $(addprefix ../../$(XEN_ROOT)/xen/include/public/,arch-ia64 arch-x86 hvm io xsm) include/xen && \
@@ -169,13 +174,13 @@ mk-headers:
 	  ln -sf $(addprefix ../../,$(wildcard $(XEN_ROOT)/tools/include/xen-foreign/*)) include/xen-foreign/ && \
 	  $(MAKE) -C include/xen-foreign/ && \
 	  ( [ -h include/xen/foreign ] || ln -sf ../xen-foreign include/xen/foreign )
-	mkdir -p libxc
-	[ -h libxc/Makefile ] || ( cd libxc && \
+	mkdir -p libxc-$(XEN_TARGET_ARCH)
+	[ -h libxc-$(XEN_TARGET_ARCH)/Makefile ] || ( cd libxc-$(XEN_TARGET_ARCH) && \
 	  ln -sf ../$(XEN_ROOT)/tools/libxc/*.h . && \
 	  ln -sf ../$(XEN_ROOT)/tools/libxc/*.c . && \
 	  ln -sf ../$(XEN_ROOT)/tools/libxc/Makefile . )
-	mkdir -p libxc/$(XEN_TARGET_ARCH)
-	[ -h libxc/$(XEN_TARGET_ARCH) ] || ( cd libxc/$(XEN_TARGET_ARCH) && \
+	mkdir -p libxc-$(XEN_TARGET_ARCH)/$(XEN_TARGET_ARCH)
+	[ -h libxc-$(XEN_TARGET_ARCH)/$(XEN_TARGET_ARCH) ] || ( cd libxc-$(XEN_TARGET_ARCH)/$(XEN_TARGET_ARCH) && \
 	  ln -sf ../$(XEN_ROOT)/tools/libxc/$(XEN_TARGET_ARCH)/*.c . && \
 	  ln -sf ../$(XEN_ROOT)/tools/libxc/$(XEN_TARGET_ARCH)/*.h . && \
 	  ln -sf ../$(XEN_ROOT)/tools/libxc/$(XEN_TARGET_ARCH)/Makefile . )
@@ -196,9 +201,9 @@ endif
 	[ ! -h ioemu/config-host.h ] || rm -f ioemu/config-host.h
 	[ ! -h ioemu/config-host.mak ] || rm -f ioemu/config-host.mak
 	$(MAKE) -C $(MINI_OS) links
-	touch mk-headers
+	touch mk-headers-$(XEN_TARGET_ARCH)
 
-TARGETS_MINIOS=$(addprefix mini-os-,$(TARGETS))
+TARGETS_MINIOS=$(addprefix mini-os-$(XEN_TARGET_ARCH)-,$(TARGETS))
 $(TARGETS_MINIOS): mini-os-%:
 	[ -d $@ ] || \
 	for i in $$(cd $(MINI_OS) ; find . -type d) ; do \
@@ -210,9 +215,9 @@ TARGETS_MINIOS=$(addprefix mini-os-,$(TA
 #######
 
 .PHONY: libxc
-libxc: libxc/libxenctrl.a libxc/libxenguest.a
-libxc/libxenctrl.a libxc/libxenguest.a:: cross-zlib
-	CPPFLAGS="$(TARGET_CPPFLAGS)" CFLAGS="$(TARGET_CFLAGS)" $(MAKE) -C libxc
+libxc: libxc-$(XEN_TARGET_ARCH)/libxenctrl.a libxc-$(XEN_TARGET_ARCH)/libxenguest.a
+libxc-$(XEN_TARGET_ARCH)/libxenctrl.a libxc-$(XEN_TARGET_ARCH)/libxenguest.a:: cross-zlib
+	CPPFLAGS="$(TARGET_CPPFLAGS)" CFLAGS="$(TARGET_CFLAGS)" $(MAKE) -C libxc-$(XEN_TARGET_ARCH)
 
 #######
 # ioemu
@@ -224,12 +229,12 @@ ifeq ($(CONFIG_QEMU),ioemu)
 	[ -f ioemu/config-host.mak ] || \
 	  ( cd ioemu ; \
           XEN_TARGET_ARCH=$(XEN_TARGET_ARCH) CFLAGS="$(TARGET_CFLAGS)" sh configure --prefix=/usr --enable-stubdom $(IOEMU_OPTIONS))
-	CPPFLAGS="$(TARGET_CPPFLAGS)" $(MAKE) -C ioemu LWIPDIR=$(CURDIR)/lwip TOOLS=
+	CPPFLAGS="$(TARGET_CPPFLAGS)" $(MAKE) -C ioemu LWIPDIR=$(CURDIR)/lwip-$(XEN_TARGET_ARCH) TOOLS=
 else
 	[ -f ioemu/config-host.mak ] || \
 	  ( cd ioemu ; \
           CONFIG_STUBDOM=yes XEN_ROOT=$(abspath $(XEN_ROOT)) XEN_TARGET_ARCH=$(XEN_TARGET_ARCH) CFLAGS="$(TARGET_CFLAGS)" sh ./xen-setup --cc=$(CC) --disable-gcc-check $(IOEMU_OPTIONS))
-	CPPFLAGS= TARGET_CPPFLAGS="$(TARGET_CPPFLAGS)" $(MAKE) -C ioemu LWIPDIR=$(CURDIR)/lwip TOOLS= CONFIG_STUBDOM=yes
+	CPPFLAGS= TARGET_CPPFLAGS="$(TARGET_CPPFLAGS)" $(MAKE) -C ioemu LWIPDIR=$(CURDIR)/lwip-$(XEN_TARGET_ARCH) TOOLS= CONFIG_STUBDOM=yes
 endif
 
 ######
@@ -238,7 +243,7 @@ endif
 
 .PHONY: caml
 caml: $(CROSS_ROOT)
-	CPPFLAGS="$(TARGET_CPPFLAGS)" CFLAGS="$(TARGET_CFLAGS)" $(MAKE) -C $@ LWIPDIR=$(CURDIR)/lwip 
+	CPPFLAGS="$(TARGET_CPPFLAGS)" CFLAGS="$(TARGET_CFLAGS)" $(MAKE) -C $@ LWIPDIR=$(CURDIR)/lwip-$(XEN_TARGET_ARCH) 
 
 ###
 # C
@@ -246,7 +251,7 @@ caml: $(CROSS_ROOT)
 
 .PHONY: c
 c: $(CROSS_ROOT)
-	CPPFLAGS="$(TARGET_CPPFLAGS)" CFLAGS="$(TARGET_CFLAGS)" $(MAKE) -C $@ LWIPDIR=$(CURDIR)/lwip 
+	CPPFLAGS="$(TARGET_CPPFLAGS)" CFLAGS="$(TARGET_CFLAGS)" $(MAKE) -C $@ LWIPDIR=$(CURDIR)/lwip-$(XEN_TARGET_ARCH) 
 
 ######
 # Grub
@@ -264,7 +269,8 @@ grub-upstream: grub-$(GRUB_VERSION).tar.
 
 .PHONY: grub
 grub: grub-upstream $(CROSS_ROOT)
-	CPPFLAGS="$(TARGET_CPPFLAGS)" CFLAGS="$(TARGET_CFLAGS)" $(MAKE) -C $@
+	mkdir -p grub-$(XEN_TARGET_ARCH)
+	CPPFLAGS="$(TARGET_CPPFLAGS)" CFLAGS="$(TARGET_CFLAGS)" $(MAKE) -C $@ OBJ_DIR=$(CURDIR)/grub-$(XEN_TARGET_ARCH)
 
 ########
 # minios
@@ -276,21 +282,21 @@ else
 else
 ioemu-stubdom: APP_OBJS=$(CURDIR)/ioemu/i386-stubdom/qemu.a $(CURDIR)/ioemu/i386-stubdom/libqemu.a $(CURDIR)/ioemu/libqemu_common.a
 endif
-ioemu-stubdom: mini-os-ioemu lwip libxc ioemu
-	DEF_CPPFLAGS="$(TARGET_CPPFLAGS)" DEF_CFLAGS="-DCONFIG_QEMU $(TARGET_CFLAGS)" DEF_LDFLAGS="$(TARGET_LDFLAGS)" $(MAKE) -C $(MINI_OS) OBJ_DIR=$(CURDIR)/$< LWIPDIR=$(CURDIR)/lwip APP_OBJS="$(APP_OBJS)"
+ioemu-stubdom: mini-os-$(XEN_TARGET_ARCH)-ioemu lwip-$(XEN_TARGET_ARCH) libxc ioemu
+	DEF_CPPFLAGS="$(TARGET_CPPFLAGS)" DEF_CFLAGS="-DCONFIG_QEMU $(TARGET_CFLAGS)" DEF_LDFLAGS="$(TARGET_LDFLAGS)" $(MAKE) -C $(MINI_OS) OBJ_DIR=$(CURDIR)/$< LWIPDIR=$(CURDIR)/lwip-$(XEN_TARGET_ARCH) APP_OBJS="$(APP_OBJS)"
 
 CAMLLIB = $(shell ocamlc -where)
 .PHONY: caml-stubdom
-caml-stubdom: mini-os-caml lwip libxc caml
-	DEF_CPPFLAGS="$(TARGET_CPPFLAGS)" DEF_CFLAGS="-DCONFIG_CAML $(TARGET_CFLAGS)" DEF_LDFLAGS="$(TARGET_LDFLAGS)" $(MAKE) -C $(MINI_OS) OBJ_DIR=$(CURDIR)/$< LWIPDIR=$(CURDIR)/lwip APP_OBJS="$(CURDIR)/caml/main-caml.o $(CURDIR)/caml/caml.o $(CAMLLIB)/libasmrun.a"
+caml-stubdom: mini-os-$(XEN_TARGET_ARCH)-caml lwip-$(XEN_TARGET_ARCH) libxc caml
+	DEF_CPPFLAGS="$(TARGET_CPPFLAGS)" DEF_CFLAGS="-DCONFIG_CAML $(TARGET_CFLAGS)" DEF_LDFLAGS="$(TARGET_LDFLAGS)" $(MAKE) -C $(MINI_OS) OBJ_DIR=$(CURDIR)/$< LWIPDIR=$(CURDIR)/lwip-$(XEN_TARGET_ARCH) APP_OBJS="$(CURDIR)/caml/main-caml.o $(CURDIR)/caml/caml.o $(CAMLLIB)/libasmrun.a"
 
 .PHONY: c-stubdom
-c-stubdom: mini-os-c lwip libxc c
-	DEF_CPPFLAGS="$(TARGET_CPPFLAGS)" DEF_CFLAGS="-DCONFIG_C $(TARGET_CFLAGS)" DEF_LDFLAGS="$(TARGET_LDFLAGS)" $(MAKE) -C $(MINI_OS) OBJ_DIR=$(CURDIR)/$< LWIPDIR=$(CURDIR)/lwip APP_OBJS=$(CURDIR)/c/main.a
+c-stubdom: mini-os-$(XEN_TARGET_ARCH)-c lwip-$(XEN_TARGET_ARCH) libxc c
+	DEF_CPPFLAGS="$(TARGET_CPPFLAGS)" DEF_CFLAGS="-DCONFIG_C $(TARGET_CFLAGS)" DEF_LDFLAGS="$(TARGET_LDFLAGS)" $(MAKE) -C $(MINI_OS) OBJ_DIR=$(CURDIR)/$< LWIPDIR=$(CURDIR)/lwip-$(XEN_TARGET_ARCH) APP_OBJS=$(CURDIR)/c/main.a
 
 .PHONY: pv-grub
-pv-grub: mini-os-grub libxc grub
-	DEF_CPPFLAGS="$(TARGET_CPPFLAGS)" DEF_CFLAGS="-DCONFIG_GRUB $(TARGET_CFLAGS)" DEF_LDFLAGS="$(TARGET_LDFLAGS)" $(MAKE) -C $(MINI_OS) OBJ_DIR=$(CURDIR)/$< APP_OBJS=$(CURDIR)/grub/main.a
+pv-grub: mini-os-$(XEN_TARGET_ARCH)-grub libxc grub
+	DEF_CPPFLAGS="$(TARGET_CPPFLAGS)" DEF_CFLAGS="-DCONFIG_GRUB $(TARGET_CFLAGS)" DEF_LDFLAGS="$(TARGET_LDFLAGS)" $(MAKE) -C $(MINI_OS) OBJ_DIR=$(CURDIR)/$< APP_OBJS=$(CURDIR)/grub-$(XEN_TARGET_ARCH)/main.a
 
 #########
 # install
@@ -310,11 +316,11 @@ install-ioemu: ioemu-stubdom
 	$(INSTALL_DIR) "$(DESTDIR)/usr/lib/xen/bin"
 	$(INSTALL_PROG) stubdom-dm "$(DESTDIR)/usr/lib/xen/bin"
 	$(INSTALL_DIR) "$(DESTDIR)/usr/lib/xen/boot"
-	$(INSTALL_DATA) mini-os-ioemu/mini-os.gz "$(DESTDIR)/usr/lib/xen/boot/ioemu-stubdom.gz"
+	$(INSTALL_DATA) mini-os-$(XEN_TARGET_ARCH)-ioemu/mini-os.gz "$(DESTDIR)/usr/lib/xen/boot/ioemu-stubdom.gz"
 
 install-grub: pv-grub
 	$(INSTALL_DIR) "$(DESTDIR)/usr/lib/xen/boot"
-	$(INSTALL_DATA) mini-os-grub/mini-os.gz "$(DESTDIR)/usr/lib/xen/boot/pv-grub.gz"
+	$(INSTALL_DATA) mini-os-$(XEN_TARGET_ARCH)-grub/mini-os.gz "$(DESTDIR)/usr/lib/xen/boot/pv-grub-$(XEN_TARGET_ARCH).gz"
 
 #######
 # clean
@@ -323,30 +329,30 @@ install-grub: pv-grub
 # Only clean the libxc/ioemu/mini-os part
 .PHONY: clean
 clean:
-	rm -fr mini-os-ioemu
-	rm -fr mini-os-c
-	rm -fr mini-os-caml
-	rm -fr mini-os-grub
+	rm -fr mini-os-$(XEN_TARGET_ARCH)-ioemu
+	rm -fr mini-os-$(XEN_TARGET_ARCH)-c
+	rm -fr mini-os-$(XEN_TARGET_ARCH)-caml
+	rm -fr mini-os-$(XEN_TARGET_ARCH)-grub
 	$(MAKE) -C caml clean
 	$(MAKE) -C c clean
-	$(MAKE) -C grub clean
-	[ ! -d libxc ] || $(MAKE) -C libxc clean
+	rm -fr grub-$(XEN_TARGET_ARCH)
+	[ ! -d libxc-$(XEN_TARGET_ARCH) ] || $(MAKE) -C libxc-$(XEN_TARGET_ARCH) clean
 	[ ! -d ioemu ] || $(MAKE) -C ioemu clean
 
 # clean the cross-compilation result
 .PHONY: crossclean
 crossclean: clean
 	rm -fr $(CROSS_ROOT)
-	rm -fr newlib-build
-	rm -fr zlib-$(ZLIB_VERSION) pciutils-$(LIBPCI_VERSION)
-	rm -fr libxc ioemu
-	rm -f mk-headers
+	rm -fr newlib-$(XEN_TARGET_ARCH)
+	rm -fr zlib-$(XEN_TARGET_ARCH) pciutils-$(XEN_TARGET_ARCH)
+	rm -fr libxc-$(XEN_TARGET_ARCH) ioemu
+	rm -f mk-headers-$(XEN_TARGET_ARCH)
 
 # clean patched sources
 .PHONY: patchclean
 patchclean: crossclean
 	rm -fr newlib-$(NEWLIB_VERSION)
-	rm -fr lwip
+	rm -fr lwip-$(XEN_TARGET_ARCH)
 	rm -fr grub-upstream
 
 # clean downloads
diff -r 37f3bdce0394 stubdom/grub/Makefile
--- a/stubdom/grub/Makefile	Mon Aug 11 12:36:26 2008 +0100
+++ b/stubdom/grub/Makefile	Thu Aug 14 13:30:23 2008 +0100
@@ -3,7 +3,7 @@ include $(XEN_ROOT)/Config.mk
 include $(XEN_ROOT)/Config.mk
 vpath %.c ../grub-upstream
 
-BOOT=boot-$(XEN_TARGET_ARCH).o
+BOOT=$(OBJ_DIR)/boot-$(XEN_TARGET_ARCH).o
 
 DEF_CPPFLAGS += -I$(XEN_ROOT)/tools/libxc -I$(XEN_ROOT)/tools/include -I.
 DEF_CPPFLAGS += -I../grub-upstream/stage1
@@ -17,7 +17,7 @@ DEF_CPPFLAGS += -DPRESET_MENU_STRING='""
 DEF_CPPFLAGS += -DPRESET_MENU_STRING='""'
 DEF_CPPFLAGS += -DPACKAGE='"grubdom"' -DVERSION='"0.97"'
 
-all: main.a
+all: $(OBJ_DIR)/main.a
 
 STAGE2_SOURCES=builtins.c char_io.c cmdline.c common.c console.c disk_io.c graphics.c gunzip.c md5.c serial.c stage2.c terminfo.c tparm.c
 
@@ -59,16 +59,26 @@ NETBOOT_SOURCES:=$(addprefix netboot/,$(
 
 $(BOOT): DEF_CPPFLAGS+=-D__ASSEMBLY__
 
-OBJS = $(NETBOOT_SOURCES:.c=.o) $(STAGE2_SOURCES:.c=.o) kexec.o mini-os.o
+PV_GRUB_SOURCES = kexec.c mini-os.c
 
-dirs:
-	mkdir -p netboot stage2
+SOURCES = $(NETBOOT_SOURCES) $(STAGE2_SOURCES) $(PV_GRUB_SOURCES)
+
+OBJS = $(addprefix $(OBJ_DIR)/,$(SOURCES:.c=.o))
+
+$(OBJ_DIR)/dirs:
+	mkdir -p $(OBJ_DIR)/netboot $(OBJ_DIR)/stage2
 	touch $@
 
-$(OBJS): dirs
+$(OBJS): $(OBJ_DIR)/dirs
 
-main.a: $(BOOT) $(OBJS)
+$(OBJ_DIR)/main.a: $(BOOT) $(OBJS)
 	$(AR) cr $@ $^
+
+$(OBJ_DIR)/%.o: %.c
+	$(CC) -c $(CPPFLAGS) $(CFLAGS) $< -o $@
+
+$(OBJ_DIR)/%.o: %.S
+	$(CC) -c $(CPPFLAGS) $(CFLAGS) $< -o $@
 
 clean:
 	rm -fr dirs *.a *.o stage2 netboot