Re: Bug#759018: [PATCH RFC] Provide prebuilt grub-xen binaries for host (dom0) use

Re: Bug#759018: [PATCH RFC] Provide prebuilt grub-xen binaries for host (dom0) use

[Ian Campbell]

On Sat, 2014-08-23 at 21:38 +0100, Ian Campbell wrote:

(Context for new CC: I'd like to provide a Debian package for dom0 which
includes pvgrub2 binaries which can be referenced from your domain cfg
files as the bootloader/kernel. The associated Debian wishlist bug is
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=759018)

> diff --git a/debian/grub-xen-host_grub.cfg
> b/debian/grub-xen-host_grub.cfg
> new file mode 100644
> index 0000000..1ad895c
> --- /dev/null
> +++ b/debian/grub-xen-host_grub.cfg
> @@ -0,0 +1,25 @@
> +# First search for a suitable grub to chainload
> +search -s -f /boot/grub/@@TARGET@@/core.elf
> +if multiboot /boot/grub/@@TARGET@@/core.elf ; then
> +       echo "Chainloading (${root})/boot/grub/@@TARGET@@/core.elf"
> +       boot
> +fi
> +
> +search -s -f /grub/@@TARGET@@/core.elf
> +if multiboot /grub/@@TARGET@@/core.elf ; then
> +       echo "Chainloading (${root})/grub/@@TARGET@@/core.elf"
> +       boot
> +fi
[...]
> diff --git a/debian/rules b/debian/rules
> index f478e94..18f4776 100755
> --- a/debian/rules
> +++ b/debian/rules
> [...]
> +debian/stamps/build-grub-xen-host-i386: GRUB_TARGET := i386-xen
> +debian/stamps/build-grub-xen-host-amd64: GRUB_TARGET := x86_64-xen
> [...]
> +       sed -e "s/@@TARGET@@/$(GRUB_TARGET)/g" \

The net result of the above is that the dom0 core.img built by the
package will search /boot/grub/{i386,x86_64}-xen/core.elf (for the arch
matching itself only) and then the version without the /boot prefix (for
systems with separate /boot).

Colin mentioned at debconf that it would be good if these paths were
standard across all distros -- probably this is best treated as
something which upstream Xen specifies a standard on, so CCing xen-devel
and Ian J who has previously been involved with these conversations. I'm
not sure what the best way to promulgate the spec is -- I think a patch
to add xen.git/docs/misc/pvgrub2.markdown would be sufficient (it would
end up under http://xenbits.xen.org/docs/).

Other alternatives for the paths might be things in the /boot/xen
"namespace" rather than /boot/grub, I don't mind which so long as it is
decided. Unless someone has a strong preference I'm inclined to go with
the paths as they happen to be in the patch above.

Ian.

Re: Bug#759018: [PATCH RFC] Provide prebuilt grub-xen binaries for host (dom0) use

[Colin Watson]

On Sun, Aug 24, 2014 at 10:42:37PM +0100, Ian Campbell wrote:
> The net result of the above is that the dom0 core.img built by the
> package will search /boot/grub/{i386,x86_64}-xen/core.elf (for the arch
> matching itself only) and then the version without the /boot prefix (for
> systems with separate /boot).
> 
> Colin mentioned at debconf that it would be good if these paths were
> standard across all distros -- probably this is best treated as
> something which upstream Xen specifies a standard on, so CCing xen-devel
> and Ian J who has previously been involved with these conversations. I'm
> not sure what the best way to promulgate the spec is -- I think a patch
> to add xen.git/docs/misc/pvgrub2.markdown would be sufficient (it would
> end up under http://xenbits.xen.org/docs/).

That was I think roughly what Ian J suggested late last year.

> Other alternatives for the paths might be things in the /boot/xen
> "namespace" rather than /boot/grub, I don't mind which so long as it is
> decided. Unless someone has a strong preference I'm inclined to go with
> the paths as they happen to be in the patch above.

My preference would be to have it be under /boot/xen/, if you're OK with
that.  The rationale is that some distributions configure GRUB with
"--program-transform-name=s/grub/grub2/", resulting in /boot/grub2/, and
I think life will be better if dom0 code doesn't have to deal with that.

As for the basename, "core.elf" is probably a bit generic if it's under
/boot/xen/.  We just need something vaguely sensible there, though, and
then the Debian grub-xen package and equivalents in other distributions
can create it.

There is also the question of whether the guest-side name should mention
GRUB.  One might argue that it shouldn't because all that matters is
that it uses the Multiboot protocol.  Then there is the question of who
gets to own the architecture names ...

As a strawman, how about:

  /boot/xen/pvboot-$arch.elf

... where $arch is the first bit of a GNU config triplet, modified such
that we always use i386 for that family?

-- 
Colin Watson                                       [cjwatson@debian.org]

Re: Bug#759018: [PATCH RFC] Provide prebuilt grub-xen binaries for host (dom0) use

[Ian Jackson]

Colin Watson writes ("Re: Bug#759018: [PATCH RFC] Provide prebuilt grub-xen binaries for host (dom0) use"):
...
> There is also the question of whether the guest-side name should mention
> GRUB.  One might argue that it shouldn't because all that matters is
> that it uses the Multiboot protocol.  Then there is the question of who
> gets to own the architecture names ...

The general scheme seems sound.


Ian Campbell:

> > I'm not sure what the best way to promulgate the spec is -- I
> > think a patch to add xen.git/docs/misc/pvgrub2.markdown would be
> > sufficient (it would end up under http://xenbits.xen.org/docs/).

Since this path is in /boot/xen and contains an executable which
expects to run in the Xen PV environment, it could also use Xen names
for the architectures.  I don't know whether a GNU config triplet arch
name as you suggest is easier or harder than that.


I have a question about the spec's wording about bitness:

+It is not in general possible under Xen for a bootloader to boot a
+kernel of a different width from itself, and this extends to
+chainloading from a stage one. Therefore it is permissible to have
+both `/boot/xen/pvboot-i386.elf` and `/boot/xen/pvboot-x86\_64.elf`
+present in a guest to be used by the appropriate stage 1 (e.g. for
+systems with 32-bit userspace and an optional 64-bit kernel).

Is it therefore expected that the host admin will be told out of band
what bitness the guest would prefer ?  And that then the host
toolstack will set up that bitness of guest, load its pvgrub for that
bitness, and hope that the guest has an appropriate-bitness core image
load in the canonical place ?

I wonder if we might, in the future, want this to be more automatic.

I guess we could have a feature in the host's 64-bit pvgrub which
would look for and load 64-bit guest pvgrub if it exists, and
alternatively check for 32-bit guest pvgrub and if found exit
signalling somehow to the host toolstack to restart the domain with
the other bitness.

But what if the host has both 64- and 32-bit pvgrub but in fact only
has one bitness of kernel ?  Signalling this back to the host by
somehow hiding or renaming one of the bitnesses of guest pvgrub seems
unpleasant.

I mention this just in case there's a better way of organising this
which might depend on a refinement to the host/guest interface.


Thanks,
Ian.

Re: Bug#759018: [PATCH RFC] Provide prebuilt grub-xen binaries for host (dom0) use

[Ian Campbell]

On Thu, 2014-09-04 at 15:15 +0100, Ian Jackson wrote:
> Colin Watson writes ("Re: Bug#759018: [PATCH RFC] Provide prebuilt grub-xen binaries for host (dom0) use"):
> ...
> > There is also the question of whether the guest-side name should mention
> > GRUB.  One might argue that it shouldn't because all that matters is
> > that it uses the Multiboot protocol.  Then there is the question of who
> > gets to own the architecture names ...
> 
> The general scheme seems sound.
> 
> 
> Ian Campbell:
> 
> > > I'm not sure what the best way to promulgate the spec is -- I
> > > think a patch to add xen.git/docs/misc/pvgrub2.markdown would be
> > > sufficient (it would end up under http://xenbits.xen.org/docs/).
> 
> Since this path is in /boot/xen and contains an executable which
> expects to run in the Xen PV environment, it could also use Xen names
> for the architectures.  I don't know whether a GNU config triplet arch
> name as you suggest is easier or harder than that.

About the same. Colin suggested the GNU config triplet names in his
strawman and I couldn't think of a reason to change.

> I have a question about the spec's wording about bitness:
> 
> +It is not in general possible under Xen for a bootloader to boot a
> +kernel of a different width from itself, and this extends to
> +chainloading from a stage one. Therefore it is permissible to have
> +both `/boot/xen/pvboot-i386.elf` and `/boot/xen/pvboot-x86\_64.elf`
> +present in a guest to be used by the appropriate stage 1 (e.g. for
> +systems with 32-bit userspace and an optional 64-bit kernel).
> 
> Is it therefore expected that the host admin will be told out of band
> what bitness the guest would prefer ?  And that then the host
> toolstack will set up that bitness of guest, load its pvgrub for that
> bitness, and hope that the guest has an appropriate-bitness core image
> load in the canonical place ?

Yes. Essentially you write kernel = /path/too/pvgrub-<32bit-name> or
-<64bit-name> in your guest config. Yes, this sucks.

AIUI in cloud interfaces etc you generally have to ask for a 32- or
64-bit domain explicitly too.

> I wonder if we might, in the future, want this to be more automatic.

I suppose that Would Be Nice(tm).

I'm not sure but it might be that for a PVH guest (once grub is ported
to that) we might be able to switch bitness at runtime and avoid this
whole mess (which comes largely from the size of the p2m entries and the
difficulties in switching, which is hidden from pvh guests)

> I guess we could have a feature in the host's 64-bit pvgrub which
> would look for and load 64-bit guest pvgrub if it exists, and
> alternatively check for 32-bit guest pvgrub and if found exit
> signalling somehow to the host toolstack to restart the domain with
> the other bitness.
> 
> But what if the host has both 64- and 32-bit pvgrub but in fact only
> has one bitness of kernel ?  Signalling this back to the host by
> somehow hiding or renaming one of the bitnesses of guest pvgrub seems
> unpleasant.

You mean if all guests happen to only use one bitness? You waste a
roundtrip through a stunt domain which will do the exiting trick and
restart, or you supply the right dom0 grub to start with I guess.

> I mention this just in case there's a better way of organising this
> which might depend on a refinement to the host/guest interface.

I can't think of anything which would need to be changed in the spec now
preemptively, FWIW.

Ian.

Re: Bug#759018: [PATCH RFC] Provide prebuilt grub-xen binaries for host (dom0) use

[Andrew Cooper]

On 04/09/14 15:25, Ian Campbell wrote:
> On Thu, 2014-09-04 at 15:15 +0100, Ian Jackson wrote:
>> Colin Watson writes ("Re: Bug#759018: [PATCH RFC] Provide prebuilt grub-xen binaries for host (dom0) use"):
>> ...
>>> There is also the question of whether the guest-side name should mention
>>> GRUB.  One might argue that it shouldn't because all that matters is
>>> that it uses the Multiboot protocol.  Then there is the question of who
>>> gets to own the architecture names ...
>> The general scheme seems sound.
>>
>>
>> Ian Campbell:
>>
>>>> I'm not sure what the best way to promulgate the spec is -- I
>>>> think a patch to add xen.git/docs/misc/pvgrub2.markdown would be
>>>> sufficient (it would end up under http://xenbits.xen.org/docs/).
>> Since this path is in /boot/xen and contains an executable which
>> expects to run in the Xen PV environment, it could also use Xen names
>> for the architectures.  I don't know whether a GNU config triplet arch
>> name as you suggest is easier or harder than that.
> About the same. Colin suggested the GNU config triplet names in his
> strawman and I couldn't think of a reason to change.
>
>> I have a question about the spec's wording about bitness:
>>
>> +It is not in general possible under Xen for a bootloader to boot a
>> +kernel of a different width from itself, and this extends to
>> +chainloading from a stage one. Therefore it is permissible to have
>> +both `/boot/xen/pvboot-i386.elf` and `/boot/xen/pvboot-x86\_64.elf`
>> +present in a guest to be used by the appropriate stage 1 (e.g. for
>> +systems with 32-bit userspace and an optional 64-bit kernel).
>>
>> Is it therefore expected that the host admin will be told out of band
>> what bitness the guest would prefer ?  And that then the host
>> toolstack will set up that bitness of guest, load its pvgrub for that
>> bitness, and hope that the guest has an appropriate-bitness core image
>> load in the canonical place ?
> Yes. Essentially you write kernel = /path/too/pvgrub-<32bit-name> or
> -<64bit-name> in your guest config. Yes, this sucks.
>
> AIUI in cloud interfaces etc you generally have to ask for a 32- or
> 64-bit domain explicitly too.
>
>> I wonder if we might, in the future, want this to be more automatic.
> I suppose that Would Be Nice(tm).
>
> I'm not sure but it might be that for a PVH guest (once grub is ported
> to that) we might be able to switch bitness at runtime and avoid this
> whole mess (which comes largely from the size of the p2m entries and the
> difficulties in switching, which is hidden from pvh guests)
>
>> I guess we could have a feature in the host's 64-bit pvgrub which
>> would look for and load 64-bit guest pvgrub if it exists, and
>> alternatively check for 32-bit guest pvgrub and if found exit
>> signalling somehow to the host toolstack to restart the domain with
>> the other bitness.
>>
>> But what if the host has both 64- and 32-bit pvgrub but in fact only
>> has one bitness of kernel ?  Signalling this back to the host by
>> somehow hiding or renaming one of the bitnesses of guest pvgrub seems
>> unpleasant.
> You mean if all guests happen to only use one bitness? You waste a
> roundtrip through a stunt domain which will do the exiting trick and
> restart, or you supply the right dom0 grub to start with I guess.

There is no real technical obstacle to prevent PV domains switching
width after starting.

In the past, the toolstack has directly loaded the target running
kernel, but that behaviour/assumption is no longer correct given these
chainloading plans.

As we no longer support 2-level PV guests, allowing a PV domain to
switch between 3 and 4 levels becomes easier to manage from Xens point
of view.

>From the top of my head, it would involve relaxing the restriction that
3 level PV guests can't pin L4 tables (but still enforce that a 3 level
PV guest can't load an L4 pagetable), and provide a new hypercall which
takes a desired with, new cr3 (referring to a pinned pagetable of the
appropriate new width) and a new eip to jump to.

~Andrew

Re: Bug#759018: [PATCH RFC] Provide prebuilt grub-xen binaries for host (dom0) use

[Ian Campbell]

On Thu, 2014-09-04 at 15:38 +0100, Andrew Cooper wrote:
> There is no real technical obstacle to prevent PV domains switching
> width after starting.

All I know is that when this stuff was first invented lots of people
thought quite hard about this and it is trickier than you might think.

> 
> In the past, the toolstack has directly loaded the target running
> kernel, but that behaviour/assumption is no longer correct given these
> chainloading plans.
> 
> As we no longer support 2-level PV guests, allowing a PV domain to
> switch between 3 and 4 levels becomes easier to manage from Xens point
> of view.
> 
> From the top of my head, it would involve relaxing the restriction that
> 3 level PV guests can't pin L4 tables (but still enforce that a 3 level
> PV guest can't load an L4 pagetable), and provide a new hypercall which
> takes a desired with, new cr3 (referring to a pinned pagetable of the
> appropriate new width) and a new eip to jump to.

and rewrite/remap the p2m and m2p tables on the fly. and arrange somehow
that all the memory used by the guest is actually visible in the m2p
mapping for that guest (since 32-bit guests don't see the whole host
m2p). and a couple of other things which I can't quite remember right
now.

Anyway, you are welcome to try ;-)

Ian.

Re: Bug#759018: [PATCH RFC] Provide prebuilt grub-xen binaries for host (dom0) use

[Ian Campbell]

On Thu, 2014-09-04 at 15:15 +0100, Ian Jackson wrote:
> I mention this just in case there's a better way of organising this
> which might depend on a refinement to the host/guest interface.

We discussed this IRL (nearer to then than now, so I hope I recall the
outcome correctly) and decided that we couldn't think of any change we
could proactively make to the spec which would help the person who
eventually comes to implement some sort of scheme for dynamically
selecting the bitness of pvgrub to use.

v2 of the spec was posted at
http://lists.xen.org/archives/html/xen-devel/2014-08/msg02964.html does
it look ok to you?

Ian.