Xen DomU Bootloader Experiences

Xen DomU Bootloader Experiences

[Elliott Mitchell]

A few times there have been mentions of a need to choose between boot
methods for DomUs.  There is a need to decide on ones to recommend and
put effort into supportting.  I may not have tried that many nor done
particularly great amounts of experimentation, but I do have some
experience with multiple User Domain bootloaders.

PyGRUB
Xen's bootloader.  PyGRUB is quite functional within its limits.  In
particular it simulates the domain's environment in Domain 0.  This means
the security exposure is problematic.  Another big concern is that it
only does GRUB v1 syntax.  For a long while Debian had a package for
generating those files on a modern system, but that package was dropped.

Yet PyGRUB does avoid needing to use external tools to retrieve the
kernel.  If the kernel is updated inside the domain, this does get the
new kernel.  Further being architecture-independent this works on x86,
ARM*, RISC-V and PowerPC.

As it is the only GRUB-flavor loader available on ARM*, that is the only
place where I've used PyGRUB.


PvGRUB
I'm sure nearly everyone knows about PvGRUB.  By being a proper port of
GRUB to run directly on Xen, it overcomes PyGRUB's disadvantages.  The
one disadvantage is needing to get patches into an external project for
changes in Xen.

Two changes to Xen urgently need propogation to PvGRUB.  I'm unsure
whether PvGRUB unmaps its mapping of vcpu_info data.  The second is
needing to work on ARM*, RISC-V and PowerPC.  The latter is the one and
only way in which PvGRUB is inferior to PyGRUB.

As PvGRUB is only available for x86, that is the only place I've used
PvGRUB.


EDK2/Tianocore
Quite well-known for being the basis of most x86 firmwares, plus being
part of a typical Qemu setup.  Not nearly as well known for being a Xen
DomU bootloader.

When it was working you would build their ArmVirtXen.dsc file and get
XEN_EFI.fd as output.  You would then use XEN_EFI.fd for the domain's
kernel.  If you looked at the console you saw something which looked and
acted pretty similar to a UEFI firmware on x86 machines.  This was
extremely functional for OSes which didn't particularly like GRUB.
Notably I've read of it being able to load a Redmond OS and it was quite
functional for booting an ARM64 port of FreeBSD.

Sometime after November 16th, 2022 or commit fff6d81270.  The built
images stopped functioning.  This is actually rather concerning since it
may also effects firmwares built for x86 HVM domains.  I don't presently
know whether there are multiple bugs, or a single one effecting all Xen
builds.

There is also an urgent need to get EDK2/Tianocore updated to match
Xen/ARM's disallowing mapping the shared information page multiple times.
As I did not wish to become deeply involved with EDK2/Tianocore I sent a
patch to xen-devel close to 1.5 years ago.  Lack of action suggests there
is an urgent need for a liason.



Recommendations:
PyGRUB is functional within its limits.  Problems are GRUBv1 syntax and
running within Domain 0.  Given this I feel the Xen Project should be
heading towards deprecating PyGRUB.  Since PvGRUB works for x86 now, I
would default to neither building nor installing PyGRUB on x86.  For
other architectures PyGRUB is still useful.

The Xen Project should formally ask the GRUB Project to port PvGRUB to
ARM, RISC-V and PowerPC.  The need for PvGRUB on ARM seems rather urgent.
Without a proper bootloader VMs aren't too useful.


The Xen Project needs people to work with EDK2/Tianocore.  The oldest
report I've seen of the EDK2/Tianocore issue dates to mid-2023.  Now two
years later the bug is still present.

The ability to configure XEN_EFI.fd as a domain kernel is a feature
highly worthy of being ported to x86.  For OSes which don't particularly
like GRUB, but do have PV drivers this is an ideal boot method.


-- 
(\___(\___(\______          --=> 8-) EHM <=--          ______/)___/)___/)
 \BS (    |         ehem+sigmsg@m5p.com  PGP 87145445         |    )   /
  \_CS\   |  _____  -O #include <stddisclaimer.h> O-   _____  |   /  _/
8A19\___\_|_/58D2 7E3D DDF4 7BA6 <-PGP-> 41D1 B375 37D0 8714\_|_/___/5445

Re: Xen DomU Bootloader Experiences

[Juergen Gross]

[-- Attachment #1.1.1: Type: text/plain, Size: 4863 bytes --]

On 12.11.25 22:13, Elliott Mitchell wrote:
> A few times there have been mentions of a need to choose between boot
> methods for DomUs.  There is a need to decide on ones to recommend and
> put effort into supportting.  I may not have tried that many nor done
> particularly great amounts of experimentation, but I do have some
> experience with multiple User Domain bootloaders.
> 
> PyGRUB
> Xen's bootloader.  PyGRUB is quite functional within its limits.  In
> particular it simulates the domain's environment in Domain 0.  This means
> the security exposure is problematic.  Another big concern is that it
> only does GRUB v1 syntax.  For a long while Debian had a package for
> generating those files on a modern system, but that package was dropped.
> 
> Yet PyGRUB does avoid needing to use external tools to retrieve the
> kernel.  If the kernel is updated inside the domain, this does get the
> new kernel.  Further being architecture-independent this works on x86,
> ARM*, RISC-V and PowerPC.
> 
> As it is the only GRUB-flavor loader available on ARM*, that is the only
> place where I've used PyGRUB.

There is one further advantage for PyGRUB: it can look into the kernel
_before_ the domU is being created, so it can tell Xen tools whether a
32- or 64-bit domU is needed based on the selected kernel.

This is the main reason why PyGRUB is still existing.

> PvGRUB
> I'm sure nearly everyone knows about PvGRUB.  By being a proper port of
> GRUB to run directly on Xen, it overcomes PyGRUB's disadvantages.  The
> one disadvantage is needing to get patches into an external project for
> changes in Xen.
> 
> Two changes to Xen urgently need propogation to PvGRUB.  I'm unsure
> whether PvGRUB unmaps its mapping of vcpu_info data.  The second is
> needing to work on ARM*, RISC-V and PowerPC.  The latter is the one and
> only way in which PvGRUB is inferior to PyGRUB.
> 
> As PvGRUB is only available for x86, that is the only place I've used
> PvGRUB.

Naming is difficult. :-)

You are talking about grub-pv. pv-grub is the Xen-internal variant based
on Mini-OS and legacy grub 0.97, supporting grub for PV-domUs.

grub-pv comes basically in three flavors, all x86-only:

- for 32-bit PV-guests
- for 64-bit PV-guests
- for PVH-guests (32- or 64-bit)

Adding PVH support to upstream Grub for e.g. Arm should be rather easy.

> 
> 
> EDK2/Tianocore
> Quite well-known for being the basis of most x86 firmwares, plus being
> part of a typical Qemu setup.  Not nearly as well known for being a Xen
> DomU bootloader.
> 
> When it was working you would build their ArmVirtXen.dsc file and get
> XEN_EFI.fd as output.  You would then use XEN_EFI.fd for the domain's
> kernel.  If you looked at the console you saw something which looked and
> acted pretty similar to a UEFI firmware on x86 machines.  This was
> extremely functional for OSes which didn't particularly like GRUB.
> Notably I've read of it being able to load a Redmond OS and it was quite
> functional for booting an ARM64 port of FreeBSD.
> 
> Sometime after November 16th, 2022 or commit fff6d81270.  The built
> images stopped functioning.  This is actually rather concerning since it
> may also effects firmwares built for x86 HVM domains.  I don't presently
> know whether there are multiple bugs, or a single one effecting all Xen
> builds.
> 
> There is also an urgent need to get EDK2/Tianocore updated to match
> Xen/ARM's disallowing mapping the shared information page multiple times.
> As I did not wish to become deeply involved with EDK2/Tianocore I sent a
> patch to xen-devel close to 1.5 years ago.  Lack of action suggests there
> is an urgent need for a liason.
> 
> 
> 
> Recommendations:
> PyGRUB is functional within its limits.  Problems are GRUBv1 syntax and
> running within Domain 0.  Given this I feel the Xen Project should be
> heading towards deprecating PyGRUB.  Since PvGRUB works for x86 now, I
> would default to neither building nor installing PyGRUB on x86.  For
> other architectures PyGRUB is still useful.
> 
> The Xen Project should formally ask the GRUB Project to port PvGRUB to
> ARM, RISC-V and PowerPC.  The need for PvGRUB on ARM seems rather urgent.
> Without a proper bootloader VMs aren't too useful.

Well, I did the grub-pvh implementation.

Doing that for other architectures shouldn't be rocket science. :-)

> The Xen Project needs people to work with EDK2/Tianocore.  The oldest
> report I've seen of the EDK2/Tianocore issue dates to mid-2023.  Now two
> years later the bug is still present.
> 
> The ability to configure XEN_EFI.fd as a domain kernel is a feature
> highly worthy of being ported to x86.  For OSes which don't particularly
> like GRUB, but do have PV drivers this is an ideal boot method.


Juergen

[-- Attachment #1.1.2: OpenPGP public key --]
[-- Type: application/pgp-keys, Size: 3743 bytes --]

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 495 bytes --]

Re: Xen DomU Bootloader Experiences

[Elliott Mitchell]

On Thu, Nov 13, 2025 at 07:46:25AM +0100, Juergen Gross wrote:
> On 12.11.25 22:13, Elliott Mitchell wrote:
> > A few times there have been mentions of a need to choose between boot
> > methods for DomUs.  There is a need to decide on ones to recommend and
> > put effort into supportting.  I may not have tried that many nor done
> > particularly great amounts of experimentation, but I do have some
> > experience with multiple User Domain bootloaders.
> > 
> > PyGRUB
> > Xen's bootloader.  PyGRUB is quite functional within its limits.  In
> > particular it simulates the domain's environment in Domain 0.  This means
> > the security exposure is problematic.  Another big concern is that it
> > only does GRUB v1 syntax.  For a long while Debian had a package for
> > generating those files on a modern system, but that package was dropped.
> > 
> > Yet PyGRUB does avoid needing to use external tools to retrieve the
> > kernel.  If the kernel is updated inside the domain, this does get the
> > new kernel.  Further being architecture-independent this works on x86,
> > ARM*, RISC-V and PowerPC.
> > 
> > As it is the only GRUB-flavor loader available on ARM*, that is the only
> > place where I've used PyGRUB.
> 
> There is one further advantage for PyGRUB: it can look into the kernel
> _before_ the domU is being created, so it can tell Xen tools whether a
> 32- or 64-bit domU is needed based on the selected kernel.
> 
> This is the main reason why PyGRUB is still existing.

Okay.

While I can believe this is a big advantage in some environments, I
suspect this is rarely used.  Whereas the security profile of PyGRUB is
troublesome,  GRUBv1 syntax nearly overwhelms all advantages by itself.
Do you think changing the default to not build/install on x86 is
inappropriate?

I suspect Tianocore/EDK2's 32/64-bit support might be a better choice for
an environment where this matters.


> > PvGRUB
> > I'm sure nearly everyone knows about PvGRUB.  By being a proper port of
> > GRUB to run directly on Xen, it overcomes PyGRUB's disadvantages.  The
> > one disadvantage is needing to get patches into an external project for
> > changes in Xen.
> > 
> > Two changes to Xen urgently need propogation to PvGRUB.  I'm unsure
> > whether PvGRUB unmaps its mapping of vcpu_info data.  The second is
> > needing to work on ARM*, RISC-V and PowerPC.  The latter is the one and
> > only way in which PvGRUB is inferior to PyGRUB.
> > 
> > As PvGRUB is only available for x86, that is the only place I've used
> > PvGRUB.
> 
> Naming is difficult. :-)
> 
> You are talking about grub-pv. pv-grub is the Xen-internal variant based
> on Mini-OS and legacy grub 0.97, supporting grub for PV-domUs.
> 
> grub-pv comes basically in three flavors, all x86-only:
> 
> - for 32-bit PV-guests
> - for 64-bit PV-guests
> - for PVH-guests (32- or 64-bit)
> 
> Adding PVH support to upstream Grub for e.g. Arm should be rather easy.

I figured as much, yet this has not yet been done.  The other two
implementations using GRUBv1 syntax is a severe weakness.  I rate this as
high priority since GRUBv2 syntax is rather valuable.  I would try to
lay groundwork for RISC-V and PowerPC too.


> > EDK2/Tianocore
> > Quite well-known for being the basis of most x86 firmwares, plus being
> > part of a typical Qemu setup.  Not nearly as well known for being a Xen
> > DomU bootloader.
> > 
> > When it was working you would build their ArmVirtXen.dsc file and get
> > XEN_EFI.fd as output.  You would then use XEN_EFI.fd for the domain's
> > kernel.  If you looked at the console you saw something which looked and
> > acted pretty similar to a UEFI firmware on x86 machines.  This was
> > extremely functional for OSes which didn't particularly like GRUB.
> > Notably I've read of it being able to load a Redmond OS and it was quite
> > functional for booting an ARM64 port of FreeBSD.
> > 
> > Sometime after November 16th, 2022 or commit fff6d81270.  The built
> > images stopped functioning.  This is actually rather concerning since it
> > may also effects firmwares built for x86 HVM domains.  I don't presently
> > know whether there are multiple bugs, or a single one effecting all Xen
> > builds.
> > 
> > There is also an urgent need to get EDK2/Tianocore updated to match
> > Xen/ARM's disallowing mapping the shared information page multiple times.
> > As I did not wish to become deeply involved with EDK2/Tianocore I sent a
> > patch to xen-devel close to 1.5 years ago.  Lack of action suggests there
> > is an urgent need for a liason.
> > 
> > 
> > 
> > Recommendations:
> > PyGRUB is functional within its limits.  Problems are GRUBv1 syntax and
> > running within Domain 0.  Given this I feel the Xen Project should be
> > heading towards deprecating PyGRUB.  Since PvGRUB works for x86 now, I
> > would default to neither building nor installing PyGRUB on x86.  For
> > other architectures PyGRUB is still useful.
> > 
> > The Xen Project should formally ask the GRUB Project to port PvGRUB to
> > ARM, RISC-V and PowerPC.  The need for PvGRUB on ARM seems rather urgent.
> > Without a proper bootloader VMs aren't too useful.
> 
> Well, I did the grub-pvh implementation.
> 
> Doing that for other architectures shouldn't be rocket science. :-)

Okay.  I've seen some it written about a number of times and no action.


My reading had been the people most knowledgeable about adapting the
tools were unsure of how to direct effort.  Proper GRUBv2 is high-value
for Linux.  Tianocore/EDK2 is valuable for everyone, high value for non-
Linux.  Worst case Tianocore/EDK2 can boot GRUB-UEFI, I doubt GRUB can
boot Tianocore/EDK2.

Is there any scenario not adaquately addressed by trying for that pair?


-- 
(\___(\___(\______          --=> 8-) EHM <=--          ______/)___/)___/)
 \BS (    |         ehem+sigmsg@m5p.com  PGP 87145445         |    )   /
  \_CS\   |  _____  -O #include <stddisclaimer.h> O-   _____  |   /  _/
8A19\___\_|_/58D2 7E3D DDF4 7BA6 <-PGP-> 41D1 B375 37D0 8714\_|_/___/5445

Re: Xen DomU Bootloader Experiences

[Grygorii Strashko]

On 12.11.25 23:13, Elliott Mitchell wrote:
> A few times there have been mentions of a need to choose between boot
> methods for DomUs.  There is a need to decide on ones to recommend and
> put effort into supportting.  I may not have tried that many nor done
> particularly great amounts of experimentation, but I do have some
> experience with multiple User Domain bootloaders.
> 
> PyGRUB
> Xen's bootloader.  PyGRUB is quite functional within its limits.  In
> particular it simulates the domain's environment in Domain 0.  This means
> the security exposure is problematic.  Another big concern is that it
> only does GRUB v1 syntax.  For a long while Debian had a package for
> generating those files on a modern system, but that package was dropped.
> 
> Yet PyGRUB does avoid needing to use external tools to retrieve the
> kernel.  If the kernel is updated inside the domain, this does get the
> new kernel.  Further being architecture-independent this works on x86,
> ARM*, RISC-V and PowerPC.
> 
> As it is the only GRUB-flavor loader available on ARM*, that is the only
> place where I've used PyGRUB.
> 
> 
> PvGRUB
> I'm sure nearly everyone knows about PvGRUB.  By being a proper port of
> GRUB to run directly on Xen, it overcomes PyGRUB's disadvantages.  The
> one disadvantage is needing to get patches into an external project for
> changes in Xen.
> 
> Two changes to Xen urgently need propogation to PvGRUB.  I'm unsure
> whether PvGRUB unmaps its mapping of vcpu_info data.  The second is
> needing to work on ARM*, RISC-V and PowerPC.  The latter is the one and
> only way in which PvGRUB is inferior to PyGRUB.
> 
> As PvGRUB is only available for x86, that is the only place I've used
> PvGRUB.
> 
> 
> EDK2/Tianocore
> Quite well-known for being the basis of most x86 firmwares, plus being
> part of a typical Qemu setup.  Not nearly as well known for being a Xen
> DomU bootloader.
> 
> When it was working you would build their ArmVirtXen.dsc file and get
> XEN_EFI.fd as output.  You would then use XEN_EFI.fd for the domain's
> kernel.  If you looked at the console you saw something which looked and
> acted pretty similar to a UEFI firmware on x86 machines.  This was
> extremely functional for OSes which didn't particularly like GRUB.
> Notably I've read of it being able to load a Redmond OS and it was quite
> functional for booting an ARM64 port of FreeBSD.
> 
> Sometime after November 16th, 2022 or commit fff6d81270.  The built
> images stopped functioning.  This is actually rather concerning since it
> may also effects firmwares built for x86 HVM domains.  I don't presently
> know whether there are multiple bugs, or a single one effecting all Xen
> builds.
> 
> There is also an urgent need to get EDK2/Tianocore updated to match
> Xen/ARM's disallowing mapping the shared information page multiple times.
> As I did not wish to become deeply involved with EDK2/Tianocore I sent a
> patch to xen-devel close to 1.5 years ago.  Lack of action suggests there
> is an urgent need for a liason.
> 
> 
> 
> Recommendations:
> PyGRUB is functional within its limits.  Problems are GRUBv1 syntax and
> running within Domain 0.  Given this I feel the Xen Project should be
> heading towards deprecating PyGRUB.  Since PvGRUB works for x86 now, I
> would default to neither building nor installing PyGRUB on x86.  For
> other architectures PyGRUB is still useful.
> 
> The Xen Project should formally ask the GRUB Project to port PvGRUB to
> ARM, RISC-V and PowerPC.  The need for PvGRUB on ARM seems rather urgent.
> Without a proper bootloader VMs aren't too useful.
> 
> 
> The Xen Project needs people to work with EDK2/Tianocore.  The oldest
> report I've seen of the EDK2/Tianocore issue dates to mid-2023.  Now two
> years later the bug is still present.
> 
> The ability to configure XEN_EFI.fd as a domain kernel is a feature
> highly worthy of being ported to x86.  For OSes which don't particularly
> like GRUB, but do have PV drivers this is an ideal boot method.
> 

Just curious, you did not mention u-boot which is widely used on ARM,
for example Android guests boot using it.

-- 
Best regards,
-grygorii

Re: Xen DomU Bootloader Experiences

[Teddy Astie]

Le 12/11/2025 à 22:16, Elliott Mitchell a écrit :
> A few times there have been mentions of a need to choose between boot
> methods for DomUs.  There is a need to decide on ones to recommend and
> put effort into supportting.  I may not have tried that many nor done
> particularly great amounts of experimentation, but I do have some
> experience with multiple User Domain bootloaders.
>
> PyGRUB
> Xen's bootloader.  PyGRUB is quite functional within its limits.  In
> particular it simulates the domain's environment in Domain 0.  This means
> the security exposure is problematic.  Another big concern is that it
> only does GRUB v1 syntax.  For a long while Debian had a package for
> generating those files on a modern system, but that package was dropped.
>
> Yet PyGRUB does avoid needing to use external tools to retrieve the
> kernel.  If the kernel is updated inside the domain, this does get the
> new kernel.  Further being architecture-independent this works on x86,
> ARM*, RISC-V and PowerPC.
>
> As it is the only GRUB-flavor loader available on ARM*, that is the only
> place where I've used PyGRUB.
>

To be fair, I don't really like PyGRUB, it hasn't really worked well for
me (perhaps I'm trying too modern distros ?). It's still what's being
used for XAPI, which doesn't make things practical and even sometimes
brittle.

I'm more tending to the other alternatives.

>
> PvGRUB
> I'm sure nearly everyone knows about PvGRUB.  By being a proper port of
> GRUB to run directly on Xen, it overcomes PyGRUB's disadvantages.  The
> one disadvantage is needing to get patches into an external project for
> changes in Xen.
>
> Two changes to Xen urgently need propogation to PvGRUB.  I'm unsure
> whether PvGRUB unmaps its mapping of vcpu_info data.  The second is

I don't think it maps vcpu_info, otherwise, most (all ?) Linux kernels
will immediately break.

> needing to work on ARM*, RISC-V and PowerPC.  The latter is the one and
> only way in which PvGRUB is inferior to PyGRUB.
>

Perhaps it could be a good idea; although I'm more tending to EDK2
instead which works well with it and cover more cases (unless EDK2 stops
working for some obscure reason). For a more long-term solution, I would
prefer something different to GRUB but basing on "Boot Loader
Specification", maybe Edera's Sprout [1] or rust-hypervisor-firmware [2]
as a PVH firmware.

> As PvGRUB is only available for x86, that is the only place I've used
> PvGRUB.
>

>
> EDK2/Tianocore
> Quite well-known for being the basis of most x86 firmwares, plus being
> part of a typical Qemu setup.  Not nearly as well known for being a Xen
> DomU bootloader.
>
> When it was working you would build their ArmVirtXen.dsc file and get
> XEN_EFI.fd as output.  You would then use XEN_EFI.fd for the domain's
> kernel.  If you looked at the console you saw something which looked and
> acted pretty similar to a UEFI firmware on x86 machines.  This was
> extremely functional for OSes which didn't particularly like GRUB.
> Notably I've read of it being able to load a Redmond OS and it was quite
> functional for booting an ARM64 port of FreeBSD.
>
> Sometime after November 16th, 2022 or commit fff6d81270.  The built
> images stopped functioning.  This is actually rather concerning since it
> may also effects firmwares built for x86 HVM domains.  I don't presently
> know whether there are multiple bugs, or a single one effecting all Xen
> builds.
>

Actually, you can use it for x86 like you do for ARM, with PVH (using
the same binary as for HVM). But it broke at some point, and I'm mostly
living with a older but working custom built version of it.

We definitely want to sort-out this, as it is going to be a blocker for
providing better foundations for PVH guests on XAPI/XCP-ng.

> There is also an urgent need to get EDK2/Tianocore updated to match
> Xen/ARM's disallowing mapping the shared information page multiple times.
> As I did not wish to become deeply involved with EDK2/Tianocore I sent a
> patch to xen-devel close to 1.5 years ago.  Lack of action suggests there
> is an urgent need for a liason.
>
>
>
> Recommendations:
> PyGRUB is functional within its limits.  Problems are GRUBv1 syntax and
> running within Domain 0.  Given this I feel the Xen Project should be
> heading towards deprecating PyGRUB.  Since PvGRUB works for x86 now, I
> would default to neither building nor installing PyGRUB on x86.  For
> other architectures PyGRUB is still useful.
>

+1. At least making it deprecated, and at the same time improving the
ergonomics of alternatives (i.e PVH+EDK2, legacy grub-pv and/or "PV-GRUB").

> The Xen Project should formally ask the GRUB Project to port PvGRUB to
> ARM, RISC-V and PowerPC.  The need for PvGRUB on ARM seems rather urgent.
> Without a proper bootloader VMs aren't too useful.
>

EDK2 / U-Boot should be able to fill this niche at least for ARM; not
sure about RISC-V though, but I guess it is going to be a similar story.

Unsure about PowerPC through.

>
> The Xen Project needs people to work with EDK2/Tianocore.  The oldest
> report I've seen of the EDK2/Tianocore issue dates to mid-2023.  Now two
> years later the bug is still present.
>

> The ability to configure XEN_EFI.fd as a domain kernel is a feature
> highly worthy of being ported to x86.  For OSes which don't particularly
> like GRUB, but do have PV drivers this is an ideal boot method.
>

AFAICT it is already there :)

>

[1] https://github.com/edera-dev/sprout
[2] https://github.com/cloud-hypervisor/rust-hypervisor-firmware


--
Teddy Astie | Vates XCP-ng Developer

XCP-ng & Xen Orchestra - Vates solutions

web: https://vates.tech

Re: Xen DomU Bootloader Experiences

[Elliott Mitchell]

On Thu, Nov 13, 2025 at 10:13:07PM +0200, Grygorii Strashko wrote:
> 
> On 12.11.25 23:13, Elliott Mitchell wrote:
> > 
> > Recommendations:
> > PyGRUB is functional within its limits.  Problems are GRUBv1 syntax and
> > running within Domain 0.  Given this I feel the Xen Project should be
> > heading towards deprecating PyGRUB.  Since PvGRUB works for x86 now, I
> > would default to neither building nor installing PyGRUB on x86.  For
> > other architectures PyGRUB is still useful.
> > 
> > The Xen Project should formally ask the GRUB Project to port PvGRUB to
> > ARM, RISC-V and PowerPC.  The need for PvGRUB on ARM seems rather urgent.
> > Without a proper bootloader VMs aren't too useful.
> > 
> > 
> > The Xen Project needs people to work with EDK2/Tianocore.  The oldest
> > report I've seen of the EDK2/Tianocore issue dates to mid-2023.  Now two
> > years later the bug is still present.
> > 
> > The ability to configure XEN_EFI.fd as a domain kernel is a feature
> > highly worthy of being ported to x86.  For OSes which don't particularly
> > like GRUB, but do have PV drivers this is an ideal boot method.
> 
> Just curious, you did not mention u-boot which is widely used on ARM,
> for example Android guests boot using it.

As I have not used U-Boot to load a DomU I did not write about something
I have not done.  Have you used U-Boot to load a DomU?  Was it a pleasant
experience?  Was it easy to setup?  Was it difficult to setup?

PyGRUB still has a bit of value since it can still load some Linux
guests.  GRUBv1 syntax is a major weakness, so I think the Xen Project
should be heading towards deprecating PyGRUB.  Mostly this needs GRUB's
Xen support to be ported to ARM and RISC-V.

GRUB's Xen support works great for x86.  Pleasant experience with minimal
trouble.  This is great for loading Linux in a DomU.

Tianocore/EDK2 support worked (past tense) great for ARM64.  Pleasant
experience with minimal trouble.  I haven't done full installation, but I
expect full installation will be a breeze (sigh getting patches into some
projects is extremely difficult).



Grygorii Strashko, do you think U-Boot is likely to cover some scenario
not covered by either GRUB or Tianocore/EDK2?  Do you think U-Boot is
likely to do a better job with some scenario than either GRUB or
Tianocore/EDK2?

You mentioned using U-Boot to load Android DomUs on ARM.  Is that likely
to be difficult to switch to Tianocore/EDK2?


-- 
(\___(\___(\______          --=> 8-) EHM <=--          ______/)___/)___/)
 \BS (    |         ehem+sigmsg@m5p.com  PGP 87145445         |    )   /
  \_CS\   |  _____  -O #include <stddisclaimer.h> O-   _____  |   /  _/
8A19\___\_|_/58D2 7E3D DDF4 7BA6 <-PGP-> 41D1 B375 37D0 8714\_|_/___/5445

Re: Xen DomU Bootloader Experiences

[Elliott Mitchell]

On Thu, Nov 13, 2025 at 10:23:20PM +0000, Teddy Astie wrote:
> Le 12/11/2025 à 22:16, Elliott Mitchell a écrit :
> > A few times there have been mentions of a need to choose between boot
> > methods for DomUs.  There is a need to decide on ones to recommend and
> > put effort into supportting.  I may not have tried that many nor done
> > particularly great amounts of experimentation, but I do have some
> > experience with multiple User Domain bootloaders.
> > 
> > PyGRUB
> > Xen's bootloader.  PyGRUB is quite functional within its limits.  In
> > particular it simulates the domain's environment in Domain 0.  This means
> > the security exposure is problematic.  Another big concern is that it
> > only does GRUB v1 syntax.  For a long while Debian had a package for
> > generating those files on a modern system, but that package was dropped.
> > 
> > Yet PyGRUB does avoid needing to use external tools to retrieve the
> > kernel.  If the kernel is updated inside the domain, this does get the
> > new kernel.  Further being architecture-independent this works on x86,
> > ARM*, RISC-V and PowerPC.
> > 
> > As it is the only GRUB-flavor loader available on ARM*, that is the only
> > place where I've used PyGRUB.
> > 
> 
> To be fair, I don't really like PyGRUB, it hasn't really worked well for 
> me (perhaps I'm trying too modern distros ?). It's still what's being 
> used for XAPI, which doesn't make things practical and even sometimes 
> brittle.
> 
> I'm more tending to the other alternatives.
> 
> > 
> > PvGRUB
> > I'm sure nearly everyone knows about PvGRUB.  By being a proper port of
> > GRUB to run directly on Xen, it overcomes PyGRUB's disadvantages.  The
> > one disadvantage is needing to get patches into an external project for
> > changes in Xen.
> > 
> > Two changes to Xen urgently need propogation to PvGRUB.  I'm unsure
> > whether PvGRUB unmaps its mapping of vcpu_info data.  The second is
> 
> I don't think it maps vcpu_info, otherwise, most (all ?) Linux kernels 
> will immediately break.

I thought multiply mapping the vcpu_info was deprecated, but still
allowed on x86.  Only on ARM{,64} was it actually disabled.  As such it
might still have a private mapping, but it is possible it never had one
or was cleaned up at some point.

I keep mentioning this as Tianocore/EDK2 does a private mapping, but
fails to remove it when bootloading finishes.  Since support for multiply
mapped vcpu_info was removed, Tianocore/EDK2 got broken by Xen.  This is
still unfixed.

> > needing to work on ARM*, RISC-V and PowerPC.  The latter is the one and
> > only way in which PvGRUB is inferior to PyGRUB.
> 
> Perhaps it could be a good idea; although I'm more tending to EDK2 
> instead which works well with it and cover more cases (unless EDK2 stops 
> working for some obscure reason). For a more long-term solution, I would 
> prefer something different to GRUB but basing on "Boot Loader 
> Specification", maybe Edera's Sprout [1] or rust-hypervisor-firmware [2] 
> as a PVH firmware.

I concur with Tianocore/EDK2 covering more cases.  I do suspect keeping
it functional may be more difficult so I'm slightly wary of it.  Whereas
PvGRUB is lower overhead and faster for *Linux*.


> > EDK2/Tianocore
> > Quite well-known for being the basis of most x86 firmwares, plus being
> > part of a typical Qemu setup.  Not nearly as well known for being a Xen
> > DomU bootloader.
> > 
> > When it was working you would build their ArmVirtXen.dsc file and get
> > XEN_EFI.fd as output.  You would then use XEN_EFI.fd for the domain's
> > kernel.  If you looked at the console you saw something which looked and
> > acted pretty similar to a UEFI firmware on x86 machines.  This was
> > extremely functional for OSes which didn't particularly like GRUB.
> > Notably I've read of it being able to load a Redmond OS and it was quite
> > functional for booting an ARM64 port of FreeBSD.
> > 
> > Sometime after November 16th, 2022 or commit fff6d81270.  The built
> > images stopped functioning.  This is actually rather concerning since it
> > may also effects firmwares built for x86 HVM domains.  I don't presently
> > know whether there are multiple bugs, or a single one effecting all Xen
> > builds.
> 
> Actually, you can use it for x86 like you do for ARM, with PVH (using 
> the same binary as for HVM). But it broke at some point, and I'm mostly 
> living with a older but working custom built version of it.
> 
> We definitely want to sort-out this, as it is going to be a blocker for 
> providing better foundations for PVH guests on XAPI/XCP-ng.

Well, Tianocore/EDK2's Xen support was broken sometime mid-late 2022.
The only working firmware images I have are 5 years old and fail to load
as PVH.  Being used as DomU kernel may have worked on x86 at some point,
but I've never seen it.

Meanwhile Tianocore/EDK2 ARM was broken on *Xen*'s side.  Usually that
would mean the Xen developers are supposed to get updates into
Tianocore/EDK2.


> > Recommendations:
> > PyGRUB is functional within its limits.  Problems are GRUBv1 syntax and
> > running within Domain 0.  Given this I feel the Xen Project should be
> > heading towards deprecating PyGRUB.  Since PvGRUB works for x86 now, I
> > would default to neither building nor installing PyGRUB on x86.  For
> > other architectures PyGRUB is still useful.
> 
> +1. At least making it deprecated, and at the same time improving the 
> ergonomics of alternatives (i.e PVH+EDK2, legacy grub-pv and/or "PV-GRUB").

Trying to used legacy GRUB syntax is hopeless.  Most distribution scripts
will have been updated to be GRUBv2 syntax-only.  As such PV GRUB is
pretty clearly the way to go.


> > The ability to configure XEN_EFI.fd as a domain kernel is a feature
> > highly worthy of being ported to x86.  For OSes which don't particularly
> > like GRUB, but do have PV drivers this is an ideal boot method.
> > 
> 
> AFAICT it is already there :)

I've never seen it working.  It may have worked at one time, but it has
remained broken for rather a long time.

Perhaps time to add CI on Xen's side?


-- 
(\___(\___(\______          --=> 8-) EHM <=--          ______/)___/)___/)
 \BS (    |         ehem+sigmsg@m5p.com  PGP 87145445         |    )   /
  \_CS\   |  _____  -O #include <stddisclaimer.h> O-   _____  |   /  _/
8A19\___\_|_/58D2 7E3D DDF4 7BA6 <-PGP-> 41D1 B375 37D0 8714\_|_/___/5445