Re: We are not able to virtualize FreeBSD using xen 4.17 on Arm 32 bit

Re: We are not able to virtualize FreeBSD using xen 4.17 on Arm 32 bit

[Henry Wang]

+(xen-devel and Arm maintainers, including Julien)

> On Nov 27, 2023, at 18:03, Mario Marietto <marietto2008@gmail.com> wrote:
> 
> Hello.
> We have just virtualized Debian 12 on our arm (32 bit) Chromebook model xe303c12 . As host / dom0 we have chosen Devuan 5,and for guest / domU,Debian 12. It works great. But our goal is different. We want to virtualize FreeBSD as domU. Can we have a working Xen PV network driver for a FreeBSD arm guest ?. I found that Julien Grall has ported the Xen drivers to FreeBSD on arm. I would like to know if Julien's work was accepted upstream by FreeBSD, in which case FreeBSD as a Xen guest on arm should work if we enable the Xen PV drivers in the FreeBSD on arm kernel. If Julien's work was not accepted upstream by FreeBSD, we will have to find his patches and apply them ourselves to the FreeBSD on arm kernel.
> We found these slides :
> 
> https://events.static.linuxfound.org/sites/events/files/slides/Porting%20FreeBSD%20on%20Xen%20on%20ARM%20.pdf
> 
> Slide 13 refers to a XENHVM FreeBSD on arm kernel config - that is what we want to find.
> It looks like when that slide presentation was written, there were some limitations on FreeBSD Xen guests. For example, for our debian bookworm guest, I am using vcpus = '2' to match the number of real cpus on our Chromebook, but slide 13 mentions support for only 1 VCPU with a FreeBSD guest, so I will need to change that vcpus = '1' in the FreeBSD guest config unless support for 2 or more vcpus was added later, which is possible because that slide presentation is 9 years old.
> Here is where I would expect to find the XENHVM FreeBSD on arm kernel config file:
> 
> https://cgit.freebsd.org/src/tree/sys/arm/conf
> 
> But it is not there unless I am not understanding something correctly. For now, unfortunately conclude that the support for Xen on arm that Julien Grall mentioned in that slide presentation 9 years ago was never added to the official FreeBSD source code. I am searching the web now to see if the patches that Julien Grall wrote are still posted somewhere online. If we cannot find them, we can ask here and on the xen-users mailing list. Julien regularly reads that list and responds to questions about Xen on arm, so I think he will tell us how to find the patches if we cannot find them online.
> According to this page from the FreeBSD wiki:
> 
> https://wiki.freebsd.org/Xen
> 
> I think FreeBSD only supports Xen on x86, not arm. So this is going to be a bit of a challenge to get a Xen FreeBSD guest on arm working. We know Julien Grall has some patches that made it work in the past !
> I found a slightly newer slide presentation by Julien here:
> 
> https://www.slideshare.net/xen_com_mgr/bsdcan-2015-how-to-port-your-bsd
> 
> It is about the same, but it mentions the GENERIC FreeBSD kernel supports Xen on arm64, but still says we need the XENHVM FreeBSD config for Xen on arm 32 bit, which I haven't found online yet.
> Please,take a look at this output of the linux kernel that can boot on Xen, and the FreeBSD kernel that cannot :
> 
> % file zImage-6.1.59-stb-xen-cbe+
> zImage-6.1.59-stb-xen-cbe+: Linux kernel ARM boot executable zImage (little-endian)
> 
> % file FREEBSD-XENVIRT 
> FREEBSD-XENVIRT: ELF 32-bit LSB executable, ARM, EABI5 version 1 (SYSV), dynamically linked, interpreter /red/herring, for FreeBSD 11.0 (1100048), not stripped
> 
> 
> The FreeBSD kernel that won't boot is in ELF format but the Linux kernel that does boot is in zImage format. This is the error we get :
> # xl create freebsd.cfg 
> Parsing config from freebsd.cfg xc: error: panic: xg_dom_elfloader.c:63: xc_dom_guest_type: image not capable of booting inside a HV
> M container: Invalid kernel 
> libxl: error: libxl_dom.c:571:libxl__build_dom: xc_dom_parse_image failed 
> libxl: error: libxl_create.c:1640:domcreate_rebuild_done: Domain 4:cannot (re-)build domain: -3 
> libxl: error: libxl_domain.c:1183:libxl__destroy_domid: Domain 4:Non-existant domain 
> libxl: error: libxl_domain.c:1137:domain_destroy_callback: Domain 4:Unable to destroy guest 
> libxl: error: libxl_domain.c:1064:domain_destroy_cb: Domain 4:Destruction of domain failed
> 
> We are spending time reading the docs on xenbits.xenproject.org, and according to those docs Xen on arm only knows how to boot a kernel in the zImage format, so the FreeBSD kernel is in a format that modern Xen incorrectly detects as an x86 kernel.
> I also watched Julien Grall's 30 minute video presentation of his work to boot FreeBSD/arm on Xen at FOSDEM 2014 here :
> 
> https://archive.fosdem.org/2014/schedule/event/freebsd_xen_arm/
> 
> In that video, and in other places, Julien mentions that the boot ABI for FreeBSD/arm on Xen was not yet developed and he was getting occasional crashes and needed to investigate the problem. He mentioned the zImage ABI that Linux uses, but pointed out FreeBSD does not use that format, and back then it was an open question which format to use to boot FreeBSD/arm on Xen. Unfortunately, nine years later, the only supported format is still the zImage format that Linux uses.
> It looks like Julien's work back then was using an ELF binary to boot FreeBSD/arm on Xen instead of the supported zImage format that Linux uses and the modern Xen toolstack exits with an error when trying to boot the FreeBSD ELF formatted binary that Julien's patch creates. So the best solution would be to try to port the rules to build a FreeBSD kernel in the zImage format instead of the ELF format. I have been studying the Makefiles in Linux to see how Linux builds the Linux arm kernel in the zImage format, but it is not trivial to understand.
> -- 
> Mario.

Re: We are not able to virtualize FreeBSD using xen 4.17 on Arm 32 bit

[Roger Pau Monné]

On Mon, Nov 27, 2023 at 10:28:13AM +0000, Henry Wang wrote:
> +(xen-devel and Arm maintainers, including Julien)
> 
> > On Nov 27, 2023, at 18:03, Mario Marietto <marietto2008@gmail.com>
> > wrote:
> > 
> > Hello.  We have just virtualized Debian 12 on our arm (32 bit)
> > Chromebook model xe303c12 . As host / dom0 we have chosen Devuan
> > 5,and for guest / domU,Debian 12. It works great. But our goal is
> > different. We want to virtualize FreeBSD as domU. Can we have a
> > working Xen PV network driver for a FreeBSD arm guest ?. I found
> > that Julien Grall has ported the Xen drivers to FreeBSD on arm. I
> > would like to know if Julien's work was accepted upstream by
> > FreeBSD, in which case FreeBSD as a Xen guest on arm should work
> > if we enable the Xen PV drivers in the FreeBSD on arm kernel. If
> > Julien's work was not accepted upstream by FreeBSD, we will have
> > to find his patches and apply them ourselves to the FreeBSD on arm
> > kernel.

I've added Elliot on Cc as he is working on upstreaming the patches to
FreeBSD.  He will be able to provide a better update than myself.

Regards, Roger.

Re: We are not able to virtualize FreeBSD using xen 4.17 on Arm 32 bit

[Mario Marietto]

[-- Attachment #1: Type: text/plain, Size: 1582 bytes --]

@Chuck Zmudzinski <brchuckz@netscape.net> : Stay tuned. They want to help
us. The xen developers are great. Very good support for us. I'm sure that
you can give a good contribution to understand what's our problem and how
to implement a fix with the help of all those good guys.

On Mon, Nov 27, 2023 at 11:56 AM Roger Pau Monné <roger.pau@citrix.com>
wrote:

> On Mon, Nov 27, 2023 at 10:28:13AM +0000, Henry Wang wrote:
> > +(xen-devel and Arm maintainers, including Julien)
> >
> > > On Nov 27, 2023, at 18:03, Mario Marietto <marietto2008@gmail.com>
> > > wrote:
> > >
> > > Hello.  We have just virtualized Debian 12 on our arm (32 bit)
> > > Chromebook model xe303c12 . As host / dom0 we have chosen Devuan
> > > 5,and for guest / domU,Debian 12. It works great. But our goal is
> > > different. We want to virtualize FreeBSD as domU. Can we have a
> > > working Xen PV network driver for a FreeBSD arm guest ?. I found
> > > that Julien Grall has ported the Xen drivers to FreeBSD on arm. I
> > > would like to know if Julien's work was accepted upstream by
> > > FreeBSD, in which case FreeBSD as a Xen guest on arm should work
> > > if we enable the Xen PV drivers in the FreeBSD on arm kernel. If
> > > Julien's work was not accepted upstream by FreeBSD, we will have
> > > to find his patches and apply them ourselves to the FreeBSD on arm
> > > kernel.
>
> I've added Elliot on Cc as he is working on upstreaming the patches to
> FreeBSD.  He will be able to provide a better update than myself.
>
> Regards, Roger.
>


-- 
Mario.

[-- Attachment #2: Type: text/html, Size: 2277 bytes --]

Re: We are not able to virtualize FreeBSD using xen 4.17 on Arm 32 bit

[Chuck Zmudzinski]

On 11/27/2023 7:45 AM, Mario Marietto wrote:
> @Chuck Zmudzinski <mailto:brchuckz@netscape.net> : Stay tuned. They want to help us. The xen developers are great. Very good support for us. I'm sure that you can give a good contribution to understand what's our problem and how to implement a fix with the help of all those good guys.
> 
> On Mon, Nov 27, 2023 at 11:56 AM Roger Pau Monné <roger.pau@citrix.com <mailto:roger.pau@citrix.com>> wrote:
> 
>     On Mon, Nov 27, 2023 at 10:28:13AM +0000, Henry Wang wrote:
>     > +(xen-devel and Arm maintainers, including Julien)
>     >
>     > > On Nov 27, 2023, at 18:03, Mario Marietto <marietto2008@gmail.com <mailto:marietto2008@gmail.com>>
>     > > wrote:
>     > >
>     > > Hello.  We have just virtualized Debian 12 on our arm (32 bit)
>     > > Chromebook model xe303c12 . As host / dom0 we have chosen Devuan
>     > > 5,and for guest / domU,Debian 12. It works great. But our goal is
>     > > different. We want to virtualize FreeBSD as domU. Can we have a
>     > > working Xen PV network driver for a FreeBSD arm guest ?. I found
>     > > that Julien Grall has ported the Xen drivers to FreeBSD on arm. I
>     > > would like to know if Julien's work was accepted upstream by
>     > > FreeBSD, in which case FreeBSD as a Xen guest on arm should work
>     > > if we enable the Xen PV drivers in the FreeBSD on arm kernel. If
>     > > Julien's work was not accepted upstream by FreeBSD, we will have
>     > > to find his patches and apply them ourselves to the FreeBSD on arm
>     > > kernel.
> 
>     I've added Elliot on Cc as he is working on upstreaming the patches to
>     FreeBSD.  He will be able to provide a better update than myself.
> 
>     Regards, Roger.

I have been collaborating with Mario, and I can explain what we have done so far :

We are using Julien's patch set against an old development version of FreeBSD 11
from 2014-12-03 :

https://xenbits.xen.org/gitweb/?p=people/julieng/freebsd.git;a=shortlog;h=refs/heads/xen-arm-v2.2

We successfully built the XENVIRT kernel and FreeBSD world, and created the
FreeBSD rootfs according to Julien's instructions here :

https://lists.freebsd.org/pipermail/freebsd-xen/2014-November/002202.html

There were some adjustments to the instructions :

To build the kernel, we used :

$ sudo make TARGET_ARCH=armv6 KERNCONF=XENVIRT buildkernel

instead of

$ sudo make TARGET_ARCH=armv6 KERNCONF=XENHVM buildkernel

The FreeBSD 'kernel' file is in ELF format and did not work, and we spent
some time trying to convert it to the zImage format without realizing the
build of the FreeBSD kernel creates the 'kernel.bin' file in the zImage format.
So when booting with the 'kernel.bin' file instead, it actually boots :

user@devuan-bunsen ~ % sudo xl create freebsd.cfg
Parsing config from freebsd.cfg
user@devuan-bunsen ~ % sudo xl li
Name                                        ID   Mem VCPUs	State	Time(s)
Domain-0                                     0   768     2     r-----    1439.4
freebsd                                      1  1152     1     r-----       3.0
user@devuan-bunsen ~ %

However, the guest is still not working correctly :

1. Attaching the console with the -c option at creation or with
   'xl console freebsd' results in no output to the console.

2. The timestamp on the virtual disk image file shows that the filesystem
   was at best mounted read-only, if it was mounted at all by the guest
   FreeBSD kernel.

3. The 'xl shutdown freebsd' command does not work, it just hangs. To stop
   the guest, you need to do 'xl destroy freebsd'.

However, I think we can get the console to work and the rootfs to mount because I
just realized I forgot to do the steps from Julien's instructions of editing the
/etc/fstab and /etc/ttys files in the FreeBSD rootfs :

$ echo "/dev/xbd0       /       ufs     rw      1       1" > /mnt/etc/fstab
$ vi /mnt/etc/ttys (add the line 'xc0 "/usr/libexec/getty Pc" xterm on secure")

I will add those and see if the console and disk are working.

But it would be much better if we could have this working with FreeBSD 13 / 14
instead of the old FreeBSD 11. Also, Julien's patch set only supports one vcpu
and it would be great to get the smp support for FreeBSD also.

Thanks,

Chuck

> 
> 
> 
> -- 
> Mario.

Re: We are not able to virtualize FreeBSD using xen 4.17 on Arm 32 bit

[Chuck Zmudzinski]

On 11/27/2023 10:22 AM, Chuck Zmudzinski wrote:
> On 11/27/2023 7:45 AM, Mario Marietto wrote:
>> @Chuck Zmudzinski <mailto:brchuckz@netscape.net> : Stay tuned. They want to help us. The xen developers are great. Very good support for us. I'm sure that you can give a good contribution to understand what's our problem and how to implement a fix with the help of all those good guys.
>> 
>> On Mon, Nov 27, 2023 at 11:56 AM Roger Pau Monné <roger.pau@citrix.com <mailto:roger.pau@citrix.com>> wrote:
>> 
>>     On Mon, Nov 27, 2023 at 10:28:13AM +0000, Henry Wang wrote:
>>     > +(xen-devel and Arm maintainers, including Julien)
>>     >
>>     > > On Nov 27, 2023, at 18:03, Mario Marietto <marietto2008@gmail.com <mailto:marietto2008@gmail.com>>
>>     > > wrote:
>>     > >
>>     > > Hello.  We have just virtualized Debian 12 on our arm (32 bit)
>>     > > Chromebook model xe303c12 . As host / dom0 we have chosen Devuan
>>     > > 5,and for guest / domU,Debian 12. It works great. But our goal is
>>     > > different. We want to virtualize FreeBSD as domU. Can we have a
>>     > > working Xen PV network driver for a FreeBSD arm guest ?. I found
>>     > > that Julien Grall has ported the Xen drivers to FreeBSD on arm. I
>>     > > would like to know if Julien's work was accepted upstream by
>>     > > FreeBSD, in which case FreeBSD as a Xen guest on arm should work
>>     > > if we enable the Xen PV drivers in the FreeBSD on arm kernel. If
>>     > > Julien's work was not accepted upstream by FreeBSD, we will have
>>     > > to find his patches and apply them ourselves to the FreeBSD on arm
>>     > > kernel.
>> 
>>     I've added Elliot on Cc as he is working on upstreaming the patches to
>>     FreeBSD.  He will be able to provide a better update than myself.
>> 
>>     Regards, Roger.
> 
> I have been collaborating with Mario, and I can explain what we have done so far :
> 
> We are using Julien's patch set against an old development version of FreeBSD 11
> from 2014-12-03 :
> 
> https://xenbits.xen.org/gitweb/?p=people/julieng/freebsd.git;a=shortlog;h=refs/heads/xen-arm-v2.2
> 
> We successfully built the XENVIRT kernel and FreeBSD world, and created the
> FreeBSD rootfs according to Julien's instructions here :
> 
> https://lists.freebsd.org/pipermail/freebsd-xen/2014-November/002202.html
> 
> There were some adjustments to the instructions :
> 
> To build the kernel, we used :
> 
> $ sudo make TARGET_ARCH=armv6 KERNCONF=XENVIRT buildkernel
> 
> instead of
> 
> $ sudo make TARGET_ARCH=armv6 KERNCONF=XENHVM buildkernel
> 
> The FreeBSD 'kernel' file is in ELF format and did not work, and we spent
> some time trying to convert it to the zImage format without realizing the
> build of the FreeBSD kernel creates the 'kernel.bin' file in the zImage format.
> So when booting with the 'kernel.bin' file instead, it actually boots :
> 
> user@devuan-bunsen ~ % sudo xl create freebsd.cfg
> Parsing config from freebsd.cfg
> user@devuan-bunsen ~ % sudo xl li
> Name                                        ID   Mem VCPUs	State	Time(s)
> Domain-0                                     0   768     2     r-----    1439.4
> freebsd                                      1  1152     1     r-----       3.0
> user@devuan-bunsen ~ %
> 
> However, the guest is still not working correctly :
> 
> 1. Attaching the console with the -c option at creation or with
>    'xl console freebsd' results in no output to the console.
> 
> 2. The timestamp on the virtual disk image file shows that the filesystem
>    was at best mounted read-only, if it was mounted at all by the guest
>    FreeBSD kernel.
> 
> 3. The 'xl shutdown freebsd' command does not work, it just hangs. To stop
>    the guest, you need to do 'xl destroy freebsd'.
> 
> However, I think we can get the console to work and the rootfs to mount because I
> just realized I forgot to do the steps from Julien's instructions of editing the
> /etc/fstab and /etc/ttys files in the FreeBSD rootfs :
> 
> $ echo "/dev/xbd0       /       ufs     rw      1       1" > /mnt/etc/fstab
> $ vi /mnt/etc/ttys (add the line 'xc0 "/usr/libexec/getty Pc" xterm on secure")
> 
> I will add those and see if the console and disk are working.

Unfortunately, adding xc0 to /etc/ttys and /dev/xbd0 as the root device in
/etc/fstab did not make the console or disk work. Still no output on the
xen console from the guest kernel, and the timestamp on the rootfs image
file did not change so it did not mount read-write.

We could use some advice for troubleshooting this. Now, we are blind because
we are not getting any xen console output But I am pleased we were able to
demonstrate that Julien's old patch set for FreeBSD 11 allows us to boot
FreeBSD 11 on a modern version of Xen on arm - we are using the Debian
stable Xen 4.17 packages.

Cheers

> 
> But it would be much better if we could have this working with FreeBSD 13 / 14
> instead of the old FreeBSD 11. Also, Julien's patch set only supports one vcpu
> and it would be great to get the smp support for FreeBSD also.
> 
> Thanks,
> 
> Chuck
> 
>> 
>> 
>> 
>> -- 
>> Mario.
> 

Re: We are not able to virtualize FreeBSD using xen 4.17 on Arm 32 bit

[Elliott Mitchell]

On Mon, Nov 27, 2023 at 10:57:42AM -0500, Chuck Zmudzinski wrote:
> On 11/27/2023 10:22 AM, Chuck Zmudzinski wrote:
> > 
> > I have been collaborating with Mario, and I can explain what we have done so far :
> > 
> > We are using Julien's patch set against an old development version of FreeBSD 11
> > from 2014-12-03 :
> > 
> > https://xenbits.xen.org/gitweb/?p=people/julieng/freebsd.git;a=shortlog;h=refs/heads/xen-arm-v2.2
> > 
> > We successfully built the XENVIRT kernel and FreeBSD world, and created the
> > FreeBSD rootfs according to Julien's instructions here :
> > 
> > https://lists.freebsd.org/pipermail/freebsd-xen/2014-November/002202.html
> > 
> > There were some adjustments to the instructions :
> > 
> > To build the kernel, we used :
> > 
> > $ sudo make TARGET_ARCH=armv6 KERNCONF=XENVIRT buildkernel
> > 
> > instead of
> > 
> > $ sudo make TARGET_ARCH=armv6 KERNCONF=XENHVM buildkernel
> > 
> > The FreeBSD 'kernel' file is in ELF format and did not work, and we spent
> > some time trying to convert it to the zImage format without realizing the
> > build of the FreeBSD kernel creates the 'kernel.bin' file in the zImage format.
> > So when booting with the 'kernel.bin' file instead, it actually boots :
> > 
> > user@devuan-bunsen ~ % sudo xl create freebsd.cfg
> > Parsing config from freebsd.cfg
> > user@devuan-bunsen ~ % sudo xl li
> > Name                                        ID   Mem VCPUs	State	Time(s)
> > Domain-0                                     0   768     2     r-----    1439.4
> > freebsd                                      1  1152     1     r-----       3.0
> > user@devuan-bunsen ~ %
> > 
> > However, the guest is still not working correctly :
> > 
> > 1. Attaching the console with the -c option at creation or with
> >    'xl console freebsd' results in no output to the console.
> > 
> > 2. The timestamp on the virtual disk image file shows that the filesystem
> >    was at best mounted read-only, if it was mounted at all by the guest
> >    FreeBSD kernel.
> > 
> > 3. The 'xl shutdown freebsd' command does not work, it just hangs. To stop
> >    the guest, you need to do 'xl destroy freebsd'.
> > 
> > However, I think we can get the console to work and the rootfs to mount because I
> > just realized I forgot to do the steps from Julien's instructions of editing the
> > /etc/fstab and /etc/ttys files in the FreeBSD rootfs :
> > 
> > $ echo "/dev/xbd0       /       ufs     rw      1       1" > /mnt/etc/fstab
> > $ vi /mnt/etc/ttys (add the line 'xc0 "/usr/libexec/getty Pc" xterm on secure")
> > 
> > I will add those and see if the console and disk are working.
> 
> Unfortunately, adding xc0 to /etc/ttys and /dev/xbd0 as the root device in
> /etc/fstab did not make the console or disk work. Still no output on the
> xen console from the guest kernel, and the timestamp on the rootfs image
> file did not change so it did not mount read-write.
> 
> We could use some advice for troubleshooting this. Now, we are blind because
> we are not getting any xen console output But I am pleased we were able to
> demonstrate that Julien's old patch set for FreeBSD 11 allows us to boot
> FreeBSD 11 on a modern version of Xen on arm - we are using the Debian
> stable Xen 4.17 packages.

The only method for booting I ever tried was Tianocore/EDK2.  As brought
up previously, that needs an update for Xen 4.17 though:
https://lists.xenproject.org/archives/html/xen-devel/2023-10/msg00278.html
Essentially they had two uses of the shared info page, one was unmapped
and the other never got unmapped.  That is good enough for FreeBSD's
bootloader to function.

FreeBSD's handling of patches is unfortunately very friendly to people
who have permission to commit, or know someone responsible for a
particular area.  If you're trying to get patches for something which
has shared responsibility and you're unfunded you're SoL.  As such my
work has basically stalled.  Everything works, but without anyone with
an interest it is simply sitting here mouldering.  Combined with the
annoyance curse it is simply sitting.

Julien Grall's patches are very much PoC.  As such I've done a lot of
updating.  Take a look at branch "submit":
https://gitlab.com/ehem/freebsd-src.git

Problem is FreeBSD's interrupt situation is troublesome.  Rather than 1
interrupt framework, there are 4.  Each has different built-in
assumptions.  "INTRNG" was designed for ARM and deliberately threw away
the x86 assumptions, but then added other assumptions.

The tip of the "submit" branch has a mix of different potential
approaches to working with "INTRNG".  Several of these might be rejected,
but hopefully one might be accepted.  Really I need someone in the
FreeBSD organization with the authority to override Michal Meloun's
proselytizing about INTRNG being perfect.

The other problem is no one in the FreeBSD organization wants to deal
with it's interrupt disaster.  By the time anyone is allowed to commit,
they've been thoroughly burned by the situation and are unwilling to
touch the interrupt system.  Thus leading to the antipattern of it being
just barely maintained.

I've got it working, just I'm stuck.


BTW Roger Pau Monné, now that Xen 4.18 is out, take a look at the
"royger" branch?


-- 
(\___(\___(\______          --=> 8-) EHM <=--          ______/)___/)___/)
 \BS (    |         ehem+sigmsg@m5p.com  PGP 87145445         |    )   /
  \_CS\   |  _____  -O #include <stddisclaimer.h> O-   _____  |   /  _/
8A19\___\_|_/58D2 7E3D DDF4 7BA6 <-PGP-> 41D1 B375 37D0 8714\_|_/___/5445

Re: We are not able to virtualize FreeBSD using xen 4.17 on Arm 32 bit

[Mario Marietto]

[-- Attachment #1: Type: text/plain, Size: 6439 bytes --]

Hello.

I know two FreeBSD developers that can commit your patches,if they think
that they are good. I could drop an email to them. I've hardly helped one
of them with the testing when we have implemented the passthrough of a
recent nvidia gpu on Linux. The second one is experienced with arm and
virtualization. Plus I know the main developer of bhyve. I'm happy to ask
them to review your work.

On Tue, Nov 28, 2023 at 12:04 AM Elliott Mitchell <ehem+freebsd@m5p.com>
wrote:

> On Mon, Nov 27, 2023 at 10:57:42AM -0500, Chuck Zmudzinski wrote:
> > On 11/27/2023 10:22 AM, Chuck Zmudzinski wrote:
> > >
> > > I have been collaborating with Mario, and I can explain what we have
> done so far :
> > >
> > > We are using Julien's patch set against an old development version of
> FreeBSD 11
> > > from 2014-12-03 :
> > >
> > >
> https://xenbits.xen.org/gitweb/?p=people/julieng/freebsd.git;a=shortlog;h=refs/heads/xen-arm-v2.2
> > >
> > > We successfully built the XENVIRT kernel and FreeBSD world, and
> created the
> > > FreeBSD rootfs according to Julien's instructions here :
> > >
> > >
> https://lists.freebsd.org/pipermail/freebsd-xen/2014-November/002202.html
> > >
> > > There were some adjustments to the instructions :
> > >
> > > To build the kernel, we used :
> > >
> > > $ sudo make TARGET_ARCH=armv6 KERNCONF=XENVIRT buildkernel
> > >
> > > instead of
> > >
> > > $ sudo make TARGET_ARCH=armv6 KERNCONF=XENHVM buildkernel
> > >
> > > The FreeBSD 'kernel' file is in ELF format and did not work, and we
> spent
> > > some time trying to convert it to the zImage format without realizing
> the
> > > build of the FreeBSD kernel creates the 'kernel.bin' file in the
> zImage format.
> > > So when booting with the 'kernel.bin' file instead, it actually boots :
> > >
> > > user@devuan-bunsen ~ % sudo xl create freebsd.cfg
> > > Parsing config from freebsd.cfg
> > > user@devuan-bunsen ~ % sudo xl li
> > > Name                                        ID   Mem VCPUs  State
>  Time(s)
> > > Domain-0                                     0   768     2     r-----
>   1439.4
> > > freebsd                                      1  1152     1     r-----
>      3.0
> > > user@devuan-bunsen ~ %
> > >
> > > However, the guest is still not working correctly :
> > >
> > > 1. Attaching the console with the -c option at creation or with
> > >    'xl console freebsd' results in no output to the console.
> > >
> > > 2. The timestamp on the virtual disk image file shows that the
> filesystem
> > >    was at best mounted read-only, if it was mounted at all by the guest
> > >    FreeBSD kernel.
> > >
> > > 3. The 'xl shutdown freebsd' command does not work, it just hangs. To
> stop
> > >    the guest, you need to do 'xl destroy freebsd'.
> > >
> > > However, I think we can get the console to work and the rootfs to
> mount because I
> > > just realized I forgot to do the steps from Julien's instructions of
> editing the
> > > /etc/fstab and /etc/ttys files in the FreeBSD rootfs :
> > >
> > > $ echo "/dev/xbd0       /       ufs     rw      1       1" >
> /mnt/etc/fstab
> > > $ vi /mnt/etc/ttys (add the line 'xc0 "/usr/libexec/getty Pc" xterm on
> secure")
> > >
> > > I will add those and see if the console and disk are working.
> >
> > Unfortunately, adding xc0 to /etc/ttys and /dev/xbd0 as the root device
> in
> > /etc/fstab did not make the console or disk work. Still no output on the
> > xen console from the guest kernel, and the timestamp on the rootfs image
> > file did not change so it did not mount read-write.
> >
> > We could use some advice for troubleshooting this. Now, we are blind
> because
> > we are not getting any xen console output But I am pleased we were able
> to
> > demonstrate that Julien's old patch set for FreeBSD 11 allows us to boot
> > FreeBSD 11 on a modern version of Xen on arm - we are using the Debian
> > stable Xen 4.17 packages.
>
> The only method for booting I ever tried was Tianocore/EDK2.  As brought
> up previously, that needs an update for Xen 4.17 though:
> https://lists.xenproject.org/archives/html/xen-devel/2023-10/msg00278.html
> Essentially they had two uses of the shared info page, one was unmapped
> and the other never got unmapped.  That is good enough for FreeBSD's
> bootloader to function.
>
> FreeBSD's handling of patches is unfortunately very friendly to people
> who have permission to commit, or know someone responsible for a
> particular area.  If you're trying to get patches for something which
> has shared responsibility and you're unfunded you're SoL.  As such my
> work has basically stalled.  Everything works, but without anyone with
> an interest it is simply sitting here mouldering.  Combined with the
> annoyance curse it is simply sitting.
>
> Julien Grall's patches are very much PoC.  As such I've done a lot of
> updating.  Take a look at branch "submit":
> https://gitlab.com/ehem/freebsd-src.git
>
> Problem is FreeBSD's interrupt situation is troublesome.  Rather than 1
> interrupt framework, there are 4.  Each has different built-in
> assumptions.  "INTRNG" was designed for ARM and deliberately threw away
> the x86 assumptions, but then added other assumptions.
>
> The tip of the "submit" branch has a mix of different potential
> approaches to working with "INTRNG".  Several of these might be rejected,
> but hopefully one might be accepted.  Really I need someone in the
> FreeBSD organization with the authority to override Michal Meloun's
> proselytizing about INTRNG being perfect.
>
> The other problem is no one in the FreeBSD organization wants to deal
> with it's interrupt disaster.  By the time anyone is allowed to commit,
> they've been thoroughly burned by the situation and are unwilling to
> touch the interrupt system.  Thus leading to the antipattern of it being
> just barely maintained.
>
> I've got it working, just I'm stuck.
>
>
> BTW Roger Pau Monné, now that Xen 4.18 is out, take a look at the
> "royger" branch?
>
>
> --
> (\___(\___(\______          --=> 8-) EHM <=--          ______/)___/)___/)
>  \BS (    |         ehem+sigmsg@m5p.com  PGP 87145445         |    )   /
>   \_CS\   |  _____  -O #include <stddisclaimer.h> O-   _____  |   /  _/
> 8A19\___\_|_/58D2 7E3D DDF4 7BA6 <-PGP-> 41D1 B375 37D0 8714\_|_/___/5445
>
>
>

-- 
Mario.

[-- Attachment #2: Type: text/html, Size: 8371 bytes --]

Re: We are not able to virtualize FreeBSD using xen 4.17 on Arm 32 bit

[Roger Pau Monné]

On Mon, Nov 27, 2023 at 03:04:30PM -0800, Elliott Mitchell wrote:
> BTW Roger Pau Monné, now that Xen 4.18 is out, take a look at the
> "royger" branch?

I've pushed a bunch of those, there are still some, I've made comments
on the branch.

I think there isn't much left after the swept I've done.

If you can rebase and reply to the comments I will take a look at
what's remaining.

Regards, Roger.

Re: We are not able to virtualize FreeBSD using xen 4.17 on Arm 32 bit

[Mario Marietto]

[-- Attachment #1: Type: text/plain, Size: 636 bytes --]

For booting a FreeBSD kernel as a guest OS on XEN,should we install xen
4.18 from source ?

On Tue, Nov 28, 2023 at 2:45 PM Roger Pau Monné <roger.pau@citrix.com>
wrote:

> On Mon, Nov 27, 2023 at 03:04:30PM -0800, Elliott Mitchell wrote:
> > BTW Roger Pau Monné, now that Xen 4.18 is out, take a look at the
> > "royger" branch?
>
> I've pushed a bunch of those, there are still some, I've made comments
> on the branch.
>
> I think there isn't much left after the swept I've done.
>
> If you can rebase and reply to the comments I will take a look at
> what's remaining.
>
> Regards, Roger.
>


-- 
Mario.

[-- Attachment #2: Type: text/html, Size: 1087 bytes --]

Re: We are not able to virtualize FreeBSD using xen 4.17 on Arm 32 bit

[Roger Pau Monné]

On Tue, Nov 28, 2023 at 03:09:14PM +0100, Mario Marietto wrote:
> For booting a FreeBSD kernel as a guest OS on XEN,should we install xen
> 4.18 from source ?

Please avoid top-posting.

I don't think so, I'm not aware of the FreeBSD port requiring a
specific version of Xen.  I do think the work is limited to aarch64
however, so there's no support in sight for arm32 FreeBSD guests as
far as I'm aware.

Regards, Roger.

Re: We are not able to virtualize FreeBSD using xen 4.17 on Arm 32 bit

[Elliott Mitchell]

On Tue, Nov 28, 2023 at 04:10:50PM +0100, Roger Pau Monné wrote:
> On Tue, Nov 28, 2023 at 03:09:14PM +0100, Mario Marietto wrote:
> > For booting a FreeBSD kernel as a guest OS on XEN,should we install xen
> > 4.18 from source ?

> I don't think so, I'm not aware of the FreeBSD port requiring a
> specific version of Xen.  I do think the work is limited to aarch64
> however, so there's no support in sight for arm32 FreeBSD guests as
> far as I'm aware.

I've only ever tried arm64, but since arm32 didn't appear to need much
to have operational I tried to make it possible.  In theory it /should/
work on arm32, but I've never tried it.  What was missing was I had never
added it to the configuration and one link was needed.  Updated "submit"
branch has the tiny adjustment.

(the only difference is the hypercall wrappers, register naming and where
the op code goes, very simple compatibility)


On Tue, Nov 28, 2023 at 02:45:40PM +0100, Roger Pau Monné wrote:
> On Mon, Nov 27, 2023 at 03:04:30PM -0800, Elliott Mitchell wrote:
> > BTW Roger Pau Monné, now that Xen 4.18 is out, take a look at the
> > "royger" branch?
> 
> I've pushed a bunch of those, there are still some, I've made comments
> on the branch.
> 
> I think there isn't much left after the swept I've done.
> 
> If you can rebase and reply to the comments I will take a look at
> what's remaining.

Done.  I'm unsure you'll like the xs_attach_children() approach.  Thing
is that really is appropriate given the situation.  #2 is the urgent one
as that is the handy approach to the hypercall declarations.


-- 
(\___(\___(\______          --=> 8-) EHM <=--          ______/)___/)___/)
 \BS (    |         ehem+sigmsg@m5p.com  PGP 87145445         |    )   /
  \_CS\   |  _____  -O #include <stddisclaimer.h> O-   _____  |   /  _/
8A19\___\_|_/58D2 7E3D DDF4 7BA6 <-PGP-> 41D1 B375 37D0 8714\_|_/___/5445

Re: We are not able to virtualize FreeBSD using xen 4.17 on Arm 32 bit

[Mario Marietto]

[-- Attachment #1: Type: text/plain, Size: 6935 bytes --]

Hello to everyone.

I tried to use xen as a hypervisor instead of kvm + libvirt + virt-manager
to boot FreeBSD on my ARM Chromebook where I have installed Devuan 5,since
Stefano said :

"That might work. libvirt + virt-manager with the xen accelerator might
work on the ARM Chromebook. That's because as far as I know Xen integration
in libvirt is done via linking to libxl directly and libxl is supported and
working on ARM"

Unfortunately something is not working properly. What I did has been to
reboot the machine in xen,enable libvirtd & and virtlogd & and virt-manager
&,but this is what happened :

Traceback (most recent call last):
  File "/usr/lib/xen-4.17/bin/pygrub", line 884, in <module>
    raise RuntimeError("Unable to find partition containing kernel")
RuntimeError: Unable to find partition containing kernel

I think it does not recognize the FreeBSD file system structure and its
kernel. Libvirt seems to have been programmed to boot Linux,not FreeBSD.

In Fact,I did the counterproof and it seems to be like this :


According with this post :

https://blog.roberthallam.org/2020/05/solving-unable-to-find-partition-containing-kernel/

I've created a file called menu.lst inside the boot directory of the image
file called "debian.img",adding the following content inside :

default 0
timeout 10
title Debian
root (hd0,1)
kernel /boot/vmlinux-6.1.59-stb-xen-cbe+ root=/dev/xvda
initrd /boot/initrd.img-6.1.59-stb-xen-cbe+

and I tried again to boot the image using virt-manager. It gave this error
again :

root@devuan-bunsen:/mnt/zroot2/zroot2/OS/Chromebook/FreeBSD-guestOS/linux-xen/debian2/boot#
2023-11-29 15:21:09.266+0000: 2467: error : libxlDomainStartPerform:1256 :
internal error: libxenlight failed to create new domain 'debian12'

but giving a look inside the log file and I found this interesting
situation ;

Using <class 'grub.GrubConf.GrubConfigFile'> to parse /boot/grub/menu.lst
(B )0 [1;24r [m [?7h [?1h = [H [J [?1h = [1B    pyGRUB  version 0.6
[1B [0m
lqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqk
[1B [0m x [0;7m Debian 12
          [m [0m x
[1B [0m x [72C [0m x
[1B [0m x [72C [0m x
[1B [0m x [72C [0m x
[1B [0m x [72C [0m x
[1B [0m x [72C [0m x
[1B [0m x [72C [0m x
[1B [0m x [72C [0m x
[1B [0m
mqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqj
[1B [70D [0m Use the ↑ and ↓ keys to select which entry is highlighted. [1B
[58DPress enter to boot the selected OS, 'e' to edit the [1B [52Dcommands
before booting, 'a' to modify the kernel arguments [1B [59Dbefore booting,
or 'c' for a command line. [12A [26C [17B [68DWill boot selected entry in
10 seconds [?1h =
[J [17A [73C [17B [68DWill boot selected entry in  9 seconds [?1h =
[J [17A [73C [17B [68DWill boot selected entry in  8 seconds [?1h =
[J [17A [73C [17B [68DWill boot selected entry in  7 seconds [?1h =
[J [17A [73C [17B [68DWill boot selected entry in  6 seconds [?1h =
[J [17A [73C [17B [68DWill boot selected entry in  5 seconds [?1h =
[J [17A [73C [17B [68DWill boot selected entry in  4 seconds [?1h =
[J [17A [73C [17B [68DWill boot selected entry in  3 seconds [?1h =
[J [17A [73C [17B [68DWill boot selected entry in  2 seconds [?1h =
[J [17A [73C [17B [68DWill boot selected entry in  1 seconds [?1l > [24;1H
[?1l >

so,it seems that it tried to boot,but for an unknown reason,it still gives
the error.

Anyway : My xen setup is not broken anymore ; Using Libvirt Linux seems to
be able to boot,FreeBSD does not. Using the "raw" method of booting FreeBSD
as domU could be another story that I will try soon.

But before trying to compile the correct freebsd kernel that's recognized
by xen using the Elliott Michell code,I need to understand well what's the
procedure that will work. So below you can read what I will try to do :

$ truncate -s 100G xenvm.img
$ mdconfig -f xenvm.img -u 0
$ newfs /dev/md0
$ mount /dev/md0 /mnt
$ git clone https://gitlab.com/ehem/freebsd-src.git
$ cd freebsd-src

$ make  -DNO_MODULES   KERNCONF=GENERIC TARGET=arm TARGET_ARCH=armv7
DESTDIR=/build buildkernel

$ echo "/dev/xbd0       /       ufs     rw      1
1" > /mnt/etc/fstab

$ nano /build/etc/ttys (add the line 'xc0 "/usr/libexec/getty Pc"
xterm on secure")
$ umount /build
$ mdconfig -d -u 0

Do you see errors ? some missing ?


---> I've only ever tried arm64, but since arm32 didn't appear to need
much to be operational I tried to
make it possible.  In theory it /should/ work on arm32, but I've never
tried it.  What was missing was
I had never added it to the configuration and one link was needed.
Updated "submit" branch has a tiny
adjustment.

I didn't understand how to apply the patch that's inside the "submit" branch.



On Wed, Nov 29, 2023 at 12:05 AM Elliott Mitchell <ehem+freebsd@m5p.com>
wrote:

> On Tue, Nov 28, 2023 at 04:10:50PM +0100, Roger Pau Monné wrote:
> > On Tue, Nov 28, 2023 at 03:09:14PM +0100, Mario Marietto wrote:
> > > For booting a FreeBSD kernel as a guest OS on XEN,should we install xen
> > > 4.18 from source ?
>
> > I don't think so, I'm not aware of the FreeBSD port requiring a
> > specific version of Xen.  I do think the work is limited to aarch64
> > however, so there's no support in sight for arm32 FreeBSD guests as
> > far as I'm aware.
>
> I've only ever tried arm64, but since arm32 didn't appear to need much
> to have operational I tried to make it possible.  In theory it /should/
> work on arm32, but I've never tried it.  What was missing was I had never
> added it to the configuration and one link was needed.  Updated "submit"
> branch has the tiny adjustment.
>
> (the only difference is the hypercall wrappers, register naming and where
> the op code goes, very simple compatibility)
>
>
> On Tue, Nov 28, 2023 at 02:45:40PM +0100, Roger Pau Monné wrote:
> > On Mon, Nov 27, 2023 at 03:04:30PM -0800, Elliott Mitchell wrote:
> > > BTW Roger Pau Monné, now that Xen 4.18 is out, take a look at the
> > > "royger" branch?
> >
> > I've pushed a bunch of those, there are still some, I've made comments
> > on the branch.
> >
> > I think there isn't much left after the swept I've done.
> >
> > If you can rebase and reply to the comments I will take a look at
> > what's remaining.
>
> Done.  I'm unsure you'll like the xs_attach_children() approach.  Thing
> is that really is appropriate given the situation.  #2 is the urgent one
> as that is the handy approach to the hypercall declarations.
>
>
> --
> (\___(\___(\______          --=> 8-) EHM <=--          ______/)___/)___/)
>  \BS (    |         ehem+sigmsg@m5p.com  PGP 87145445         |    )   /
>   \_CS\   |  _____  -O #include <stddisclaimer.h> O-   _____  |   /  _/
> 8A19\___\_|_/58D2 7E3D DDF4 7BA6 <-PGP-> 41D1 B375 37D0 8714\_|_/___/5445
>
>
>

-- 
Mario.

[-- Attachment #2: Type: text/html, Size: 11562 bytes --]

Re: We are not able to virtualize FreeBSD using xen 4.17 on Arm 32 bit

[Stefano Stabellini]

[-- Attachment #1: Type: text/plain, Size: 7782 bytes --]

Hi Mario,

I suggest to do one step at a time and first to get it fully working
using "xl" to create the FreeBSD guest. And afterwords to try to get
libvirt to work. Both xl and libvirt call the same library underneath
(libxl) to create guests, but xl is tested a lot more with Xen on ARM.

I am not going to comment on how to create a FreeBSD xenvm.img as I
don't have any knowledge on FreeBSD.



On Wed, 29 Nov 2023, Mario Marietto wrote:
> Hello to everyone.
> 
> I tried to use xen as a hypervisor instead of kvm + libvirt + virt-manager to boot FreeBSD on my ARM Chromebook where I have installed
> Devuan 5,since Stefano said : 
> 
> "That might work. libvirt + virt-manager with the xen accelerator might work on the ARM Chromebook. That's because as far as I know Xen
> integration in libvirt is done via linking to libxl directly and libxl is supported and working on ARM" 
> 
> Unfortunately something is not working properly. What I did has been to reboot the machine in xen,enable libvirtd & and virtlogd & and
> virt-manager &,but this is what happened :
> 
> Traceback (most recent call last):
>   File "/usr/lib/xen-4.17/bin/pygrub", line 884, in <module>
>     raise RuntimeError("Unable to find partition containing kernel")
> RuntimeError: Unable to find partition containing kernel
> 
> I think it does not recognize the FreeBSD file system structure and its kernel. Libvirt seems to have been programmed to boot Linux,not
> FreeBSD. 
> 
> In Fact,I did the counterproof and it seems to be like this :
> 
> 
> According with this post :
> 
> https://blog.roberthallam.org/2020/05/solving-unable-to-find-partition-containing-kernel/
> 
> I've created a file called menu.lst inside the boot directory of the image file called "debian.img",adding the following content inside :
> 
> default 0
> timeout 10
> title Debian
> root (hd0,1)
> kernel /boot/vmlinux-6.1.59-stb-xen-cbe+ root=/dev/xvda
> initrd /boot/initrd.img-6.1.59-stb-xen-cbe+
> 
> and I tried again to boot the image using virt-manager. It gave this error again :
> 
> root@devuan-bunsen:/mnt/zroot2/zroot2/OS/Chromebook/FreeBSD-guestOS/linux-xen/debian2/boot# 2023-11-29 15:21:09.266+0000: 2467: error :
> libxlDomainStartPerform:1256 : internal error: libxenlight failed to create new domain 'debian12'
> 
> but giving a look inside the log file and I found this interesting situation ;
> 
> Using <class 'grub.GrubConf.GrubConfigFile'> to parse /boot/grub/menu.lst
> (B )0 [1;24r [m [?7h [?1h = [H [J [?1h = [1B    pyGRUB  version 0.6
> [1B [0m lqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqk
> [1B [0m x [0;7m Debian 12                                                             [m [0m x
> [1B [0m x [72C [0m x
> [1B [0m x [72C [0m x
> [1B [0m x [72C [0m x
> [1B [0m x [72C [0m x
> [1B [0m x [72C [0m x
> [1B [0m x [72C [0m x
> [1B [0m x [72C [0m x
> [1B [0m mqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqj [1B [70D [0m Use the ↑ and ↓ keys to select which
> entry is highlighted. [1B [58DPress enter to boot the selected OS, 'e' to edit the [1B [52Dcommands before booting, 'a' to modify the
> kernel arguments [1B [59Dbefore booting, or 'c' for a command line. [12A [26C [17B [68DWill boot selected entry in 10 seconds [?1h =
> [J [17A [73C [17B [68DWill boot selected entry in  9 seconds [?1h =
> [J [17A [73C [17B [68DWill boot selected entry in  8 seconds [?1h =
> [J [17A [73C [17B [68DWill boot selected entry in  7 seconds [?1h =
> [J [17A [73C [17B [68DWill boot selected entry in  6 seconds [?1h =
> [J [17A [73C [17B [68DWill boot selected entry in  5 seconds [?1h =
> [J [17A [73C [17B [68DWill boot selected entry in  4 seconds [?1h =
> [J [17A [73C [17B [68DWill boot selected entry in  3 seconds [?1h =
> [J [17A [73C [17B [68DWill boot selected entry in  2 seconds [?1h =
> [J [17A [73C [17B [68DWill boot selected entry in  1 seconds [?1l > [24;1H
> [?1l >
> 
> so,it seems that it tried to boot,but for an unknown reason,it still gives the error.
>  
> Anyway : My xen setup is not broken anymore ; Using Libvirt Linux seems to be able to boot,FreeBSD does not. Using the "raw" method of
> booting FreeBSD as domU could be another story that I will try soon.
> 
> But before trying to compile the correct freebsd kernel that's recognized by xen using the Elliott Michell code,I need to understand well
> what's the procedure that will work. So below you can read what I will try to do :
> 
> $ truncate -s 100G xenvm.img
> $ mdconfig -f xenvm.img -u 0 
> $ newfs /dev/md0 
> $ mount /dev/md0 /mnt
> $ git clone https://gitlab.com/ehem/freebsd-src.git
> $ cd freebsd-src
> $ make  -DNO_MODULES   KERNCONF=GENERIC TARGET=arm TARGET_ARCH=armv7 DESTDIR=/build buildkernel
> $ echo "/dev/xbd0       /       ufs     rw      1 
> 1" > /mnt/etc/fstab  
> 
> $ nano /build/etc/ttys (add the line 'xc0 "/usr/libexec/getty Pc" xterm on secure")
> $ umount /build
> $ mdconfig -d -u 0
> 
> Do you see errors ? some missing ?
> ---> I've only ever tried arm64, but since arm32 didn't appear to need much to be operational I tried to
> make it possible.  In theory it /should/ work on arm32, but I've never tried it.  What was missing was 
> I had never added it to the configuration and one link was needed. Updated "submit" branch has a tiny 
> adjustment.
> 
> I didn't understand how to apply the patch that's inside the "submit" branch. 
> 
> 
> On Wed, Nov 29, 2023 at 12:05 AM Elliott Mitchell <ehem+freebsd@m5p.com> wrote:
>       On Tue, Nov 28, 2023 at 04:10:50PM +0100, Roger Pau Monné wrote:
>       > On Tue, Nov 28, 2023 at 03:09:14PM +0100, Mario Marietto wrote:
>       > > For booting a FreeBSD kernel as a guest OS on XEN,should we install xen
>       > > 4.18 from source ?
> 
>       > I don't think so, I'm not aware of the FreeBSD port requiring a
>       > specific version of Xen.  I do think the work is limited to aarch64
>       > however, so there's no support in sight for arm32 FreeBSD guests as
>       > far as I'm aware.
> 
>       I've only ever tried arm64, but since arm32 didn't appear to need much
>       to have operational I tried to make it possible.  In theory it /should/
>       work on arm32, but I've never tried it.  What was missing was I had never
>       added it to the configuration and one link was needed.  Updated "submit"
>       branch has the tiny adjustment.
> 
>       (the only difference is the hypercall wrappers, register naming and where
>       the op code goes, very simple compatibility)
> 
> 
>       On Tue, Nov 28, 2023 at 02:45:40PM +0100, Roger Pau Monné wrote:
>       > On Mon, Nov 27, 2023 at 03:04:30PM -0800, Elliott Mitchell wrote:
>       > > BTW Roger Pau Monné, now that Xen 4.18 is out, take a look at the
>       > > "royger" branch?
>       >
>       > I've pushed a bunch of those, there are still some, I've made comments
>       > on the branch.
>       >
>       > I think there isn't much left after the swept I've done.
>       >
>       > If you can rebase and reply to the comments I will take a look at
>       > what's remaining.
> 
>       Done.  I'm unsure you'll like the xs_attach_children() approach.  Thing
>       is that really is appropriate given the situation.  #2 is the urgent one
>       as that is the handy approach to the hypercall declarations.
> 
> 
>       --
>       (\___(\___(\______          --=> 8-) EHM <=--          ______/)___/)___/)
>        \BS (    |         ehem+sigmsg@m5p.com  PGP 87145445         |    )   /
>         \_CS\   |  _____  -O #include <stddisclaimer.h> O-   _____  |   /  _/
>       8A19\___\_|_/58D2 7E3D DDF4 7BA6 <-PGP-> 41D1 B375 37D0 8714\_|_/___/5445
> 
> 
> 
> 
> --
> Mario.
> 
> 

Re: We are not able to virtualize FreeBSD using xen 4.17 on Arm 32 bit

[Elliott Mitchell]

On Tue, Nov 28, 2023 at 03:04:50PM -0800, Elliott Mitchell wrote:
> On Tue, Nov 28, 2023 at 04:10:50PM +0100, Roger Pau Monné wrote:
> > On Tue, Nov 28, 2023 at 03:09:14PM +0100, Mario Marietto wrote:
> > > For booting a FreeBSD kernel as a guest OS on XEN,should we install xen
> > > 4.18 from source ?
> 
> > I don't think so, I'm not aware of the FreeBSD port requiring a
> > specific version of Xen.  I do think the work is limited to aarch64
> > however, so there's no support in sight for arm32 FreeBSD guests as
> > far as I'm aware.
> 
> I've only ever tried arm64, but since arm32 didn't appear to need much
> to have operational I tried to make it possible.  In theory it /should/
> work on arm32, but I've never tried it.  What was missing was I had never
> added it to the configuration and one link was needed.  Updated "submit"
> branch has the tiny adjustment.
> 
> (the only difference is the hypercall wrappers, register naming and where
> the op code goes, very simple compatibility)

Ugh, goof and send this before the final check completes...   Appears a
bit more work is needed to get this operational.  Mainly need to figure
out the Clang option to enable the key op code.

Issue is the arm hypercall.h header and the HYPERVISOR_*() wrappers.  I
fear the traditional approach might be easier to get working.


-- 
(\___(\___(\______          --=> 8-) EHM <=--          ______/)___/)___/)
 \BS (    |         ehem+sigmsg@m5p.com  PGP 87145445         |    )   /
  \_CS\   |  _____  -O #include <stddisclaimer.h> O-   _____  |   /  _/
8A19\___\_|_/58D2 7E3D DDF4 7BA6 <-PGP-> 41D1 B375 37D0 8714\_|_/___/5445

Re: We are not able to virtualize FreeBSD using xen 4.17 on Arm 32 bit

[Stefano Stabellini]

[-- Attachment #1: Type: text/plain, Size: 5276 bytes --]

On Mon, 27 Nov 2023, Chuck Zmudzinski wrote:
> On 11/27/2023 10:22 AM, Chuck Zmudzinski wrote:
> > On 11/27/2023 7:45 AM, Mario Marietto wrote:
> >> @Chuck Zmudzinski <mailto:brchuckz@netscape.net> : Stay tuned. They want to help us. The xen developers are great. Very good support for us. I'm sure that you can give a good contribution to understand what's our problem and how to implement a fix with the help of all those good guys.
> >> 
> >> On Mon, Nov 27, 2023 at 11:56 AM Roger Pau Monné <roger.pau@citrix.com <mailto:roger.pau@citrix.com>> wrote:
> >> 
> >>     On Mon, Nov 27, 2023 at 10:28:13AM +0000, Henry Wang wrote:
> >>     > +(xen-devel and Arm maintainers, including Julien)
> >>     >
> >>     > > On Nov 27, 2023, at 18:03, Mario Marietto <marietto2008@gmail.com <mailto:marietto2008@gmail.com>>
> >>     > > wrote:
> >>     > >
> >>     > > Hello.  We have just virtualized Debian 12 on our arm (32 bit)
> >>     > > Chromebook model xe303c12 . As host / dom0 we have chosen Devuan
> >>     > > 5,and for guest / domU,Debian 12. It works great. But our goal is
> >>     > > different. We want to virtualize FreeBSD as domU. Can we have a
> >>     > > working Xen PV network driver for a FreeBSD arm guest ?. I found
> >>     > > that Julien Grall has ported the Xen drivers to FreeBSD on arm. I
> >>     > > would like to know if Julien's work was accepted upstream by
> >>     > > FreeBSD, in which case FreeBSD as a Xen guest on arm should work
> >>     > > if we enable the Xen PV drivers in the FreeBSD on arm kernel. If
> >>     > > Julien's work was not accepted upstream by FreeBSD, we will have
> >>     > > to find his patches and apply them ourselves to the FreeBSD on arm
> >>     > > kernel.
> >> 
> >>     I've added Elliot on Cc as he is working on upstreaming the patches to
> >>     FreeBSD.  He will be able to provide a better update than myself.
> >> 
> >>     Regards, Roger.
> > 
> > I have been collaborating with Mario, and I can explain what we have done so far :
> > 
> > We are using Julien's patch set against an old development version of FreeBSD 11
> > from 2014-12-03 :
> > 
> > https://xenbits.xen.org/gitweb/?p=people/julieng/freebsd.git;a=shortlog;h=refs/heads/xen-arm-v2.2
> > 
> > We successfully built the XENVIRT kernel and FreeBSD world, and created the
> > FreeBSD rootfs according to Julien's instructions here :
> > 
> > https://lists.freebsd.org/pipermail/freebsd-xen/2014-November/002202.html
> > 
> > There were some adjustments to the instructions :
> > 
> > To build the kernel, we used :
> > 
> > $ sudo make TARGET_ARCH=armv6 KERNCONF=XENVIRT buildkernel
> > 
> > instead of
> > 
> > $ sudo make TARGET_ARCH=armv6 KERNCONF=XENHVM buildkernel
> > 
> > The FreeBSD 'kernel' file is in ELF format and did not work, and we spent
> > some time trying to convert it to the zImage format without realizing the
> > build of the FreeBSD kernel creates the 'kernel.bin' file in the zImage format.
> > So when booting with the 'kernel.bin' file instead, it actually boots :
> > 
> > user@devuan-bunsen ~ % sudo xl create freebsd.cfg
> > Parsing config from freebsd.cfg
> > user@devuan-bunsen ~ % sudo xl li
> > Name                                        ID   Mem VCPUs	State	Time(s)
> > Domain-0                                     0   768     2     r-----    1439.4
> > freebsd                                      1  1152     1     r-----       3.0
> > user@devuan-bunsen ~ %
> > 
> > However, the guest is still not working correctly :
> > 
> > 1. Attaching the console with the -c option at creation or with
> >    'xl console freebsd' results in no output to the console.
> > 
> > 2. The timestamp on the virtual disk image file shows that the filesystem
> >    was at best mounted read-only, if it was mounted at all by the guest
> >    FreeBSD kernel.
> > 
> > 3. The 'xl shutdown freebsd' command does not work, it just hangs. To stop
> >    the guest, you need to do 'xl destroy freebsd'.
> > 
> > However, I think we can get the console to work and the rootfs to mount because I
> > just realized I forgot to do the steps from Julien's instructions of editing the
> > /etc/fstab and /etc/ttys files in the FreeBSD rootfs :
> > 
> > $ echo "/dev/xbd0       /       ufs     rw      1       1" > /mnt/etc/fstab
> > $ vi /mnt/etc/ttys (add the line 'xc0 "/usr/libexec/getty Pc" xterm on secure")
> > 
> > I will add those and see if the console and disk are working.
> 
> Unfortunately, adding xc0 to /etc/ttys and /dev/xbd0 as the root device in
> /etc/fstab did not make the console or disk work. Still no output on the
> xen console from the guest kernel, and the timestamp on the rootfs image
> file did not change so it did not mount read-write.
> 
> We could use some advice for troubleshooting this. Now, we are blind because
> we are not getting any xen console output But I am pleased we were able to
> demonstrate that Julien's old patch set for FreeBSD 11 allows us to boot
> FreeBSD 11 on a modern version of Xen on arm - we are using the Debian
> stable Xen 4.17 packages.

You can use the DEBUG hypercalls to check how far we got into the
booting process:
https://wiki.xenproject.org/wiki/Xen_ARM_DEBUG_hypercalls

For instance add the following to FreeBSD code:

asm volatile("hvc 0xfffd");

Re: We are not able to virtualize FreeBSD using xen 4.17 on Arm 32 bit

[Mario Marietto]

[-- Attachment #1: Type: text/plain, Size: 6445 bytes --]

Hello.

to try to compile the correct freebsd kernel that's recognized by xen using
the Elliott Michell code, are the commands below the correct ones ?

$ truncate -s 100G xenvm.img
$ mdconfig -f xenvm.img -u 0
$ newfs /dev/md0
$ mount /dev/md0 /mnt
$ git clone https://gitlab.com/ehem/freebsd-src.git
$ cd freebsd-src

$ make  -DNO_MODULES   KERNCONF=GENERIC TARGET=arm TARGET_ARCH=armv7 buildkernel

$ echo "/dev/xbd0       /       ufs     rw      1
1" > /mnt/etc/fstab

$ nano /build/etc/ttys (add the line 'xc0 "/usr/libexec/getty Pc"
xterm on secure")
$ umount /build
$ mdconfig -d -u 0

Do you see errors ? some missing ? very thanks.


On Tue, Nov 28, 2023 at 3:28 AM Stefano Stabellini <sstabellini@kernel.org>
wrote:

> On Mon, 27 Nov 2023, Chuck Zmudzinski wrote:
> > On 11/27/2023 10:22 AM, Chuck Zmudzinski wrote:
> > > On 11/27/2023 7:45 AM, Mario Marietto wrote:
> > >> @Chuck Zmudzinski <mailto:brchuckz@netscape.net> : Stay tuned. They
> want to help us. The xen developers are great. Very good support for us.
> I'm sure that you can give a good contribution to understand what's our
> problem and how to implement a fix with the help of all those good guys.
> > >>
> > >> On Mon, Nov 27, 2023 at 11:56 AM Roger Pau Monné <
> roger.pau@citrix.com <mailto:roger.pau@citrix.com>> wrote:
> > >>
> > >>     On Mon, Nov 27, 2023 at 10:28:13AM +0000, Henry Wang wrote:
> > >>     > +(xen-devel and Arm maintainers, including Julien)
> > >>     >
> > >>     > > On Nov 27, 2023, at 18:03, Mario Marietto <
> marietto2008@gmail.com <mailto:marietto2008@gmail.com>>
> > >>     > > wrote:
> > >>     > >
> > >>     > > Hello.  We have just virtualized Debian 12 on our arm (32 bit)
> > >>     > > Chromebook model xe303c12 . As host / dom0 we have chosen
> Devuan
> > >>     > > 5,and for guest / domU,Debian 12. It works great. But our
> goal is
> > >>     > > different. We want to virtualize FreeBSD as domU. Can we have
> a
> > >>     > > working Xen PV network driver for a FreeBSD arm guest ?. I
> found
> > >>     > > that Julien Grall has ported the Xen drivers to FreeBSD on
> arm. I
> > >>     > > would like to know if Julien's work was accepted upstream by
> > >>     > > FreeBSD, in which case FreeBSD as a Xen guest on arm should
> work
> > >>     > > if we enable the Xen PV drivers in the FreeBSD on arm kernel.
> If
> > >>     > > Julien's work was not accepted upstream by FreeBSD, we will
> have
> > >>     > > to find his patches and apply them ourselves to the FreeBSD
> on arm
> > >>     > > kernel.
> > >>
> > >>     I've added Elliot on Cc as he is working on upstreaming the
> patches to
> > >>     FreeBSD.  He will be able to provide a better update than myself.
> > >>
> > >>     Regards, Roger.
> > >
> > > I have been collaborating with Mario, and I can explain what we have
> done so far :
> > >
> > > We are using Julien's patch set against an old development version of
> FreeBSD 11
> > > from 2014-12-03 :
> > >
> > >
> https://xenbits.xen.org/gitweb/?p=people/julieng/freebsd.git;a=shortlog;h=refs/heads/xen-arm-v2.2
> > >
> > > We successfully built the XENVIRT kernel and FreeBSD world, and
> created the
> > > FreeBSD rootfs according to Julien's instructions here :
> > >
> > >
> https://lists.freebsd.org/pipermail/freebsd-xen/2014-November/002202.html
> > >
> > > There were some adjustments to the instructions :
> > >
> > > To build the kernel, we used :
> > >
> > > $ sudo make TARGET_ARCH=armv6 KERNCONF=XENVIRT buildkernel
> > >
> > > instead of
> > >
> > > $ sudo make TARGET_ARCH=armv6 KERNCONF=XENHVM buildkernel
> > >
> > > The FreeBSD 'kernel' file is in ELF format and did not work, and we
> spent
> > > some time trying to convert it to the zImage format without realizing
> the
> > > build of the FreeBSD kernel creates the 'kernel.bin' file in the
> zImage format.
> > > So when booting with the 'kernel.bin' file instead, it actually boots :
> > >
> > > user@devuan-bunsen ~ % sudo xl create freebsd.cfg
> > > Parsing config from freebsd.cfg
> > > user@devuan-bunsen ~ % sudo xl li
> > > Name                                        ID   Mem VCPUs  State
>  Time(s)
> > > Domain-0                                     0   768     2     r-----
>   1439.4
> > > freebsd                                      1  1152     1     r-----
>      3.0
> > > user@devuan-bunsen ~ %
> > >
> > > However, the guest is still not working correctly :
> > >
> > > 1. Attaching the console with the -c option at creation or with
> > >    'xl console freebsd' results in no output to the console.
> > >
> > > 2. The timestamp on the virtual disk image file shows that the
> filesystem
> > >    was at best mounted read-only, if it was mounted at all by the guest
> > >    FreeBSD kernel.
> > >
> > > 3. The 'xl shutdown freebsd' command does not work, it just hangs. To
> stop
> > >    the guest, you need to do 'xl destroy freebsd'.
> > >
> > > However, I think we can get the console to work and the rootfs to
> mount because I
> > > just realized I forgot to do the steps from Julien's instructions of
> editing the
> > > /etc/fstab and /etc/ttys files in the FreeBSD rootfs :
> > >
> > > $ echo "/dev/xbd0       /       ufs     rw      1       1" >
> /mnt/etc/fstab
> > > $ vi /mnt/etc/ttys (add the line 'xc0 "/usr/libexec/getty Pc" xterm on
> secure")
> > >
> > > I will add those and see if the console and disk are working.
> >
> > Unfortunately, adding xc0 to /etc/ttys and /dev/xbd0 as the root device
> in
> > /etc/fstab did not make the console or disk work. Still no output on the
> > xen console from the guest kernel, and the timestamp on the rootfs image
> > file did not change so it did not mount read-write.
> >
> > We could use some advice for troubleshooting this. Now, we are blind
> because
> > we are not getting any xen console output But I am pleased we were able
> to
> > demonstrate that Julien's old patch set for FreeBSD 11 allows us to boot
> > FreeBSD 11 on a modern version of Xen on arm - we are using the Debian
> > stable Xen 4.17 packages.
>
> You can use the DEBUG hypercalls to check how far we got into the
> booting process:
> https://wiki.xenproject.org/wiki/Xen_ARM_DEBUG_hypercalls
>
> For instance add the following to FreeBSD code:
>
> asm volatile("hvc 0xfffd");
>
>

-- 
Mario.

[-- Attachment #2: Type: text/html, Size: 9217 bytes --]

Re: We are not able to virtualize FreeBSD using xen 4.17 on Arm 32 bit

[Mario Marietto]

[-- Attachment #1: Type: text/plain, Size: 6901 bytes --]

oh sorry I'm missing the DESTDIR parameter :

$ make  -DNO_MODULES   KERNCONF=GENERIC TARGET=arm TARGET_ARCH=armv7
DESTDIR=/build buildkernel



On Tue, Nov 28, 2023 at 11:41 AM Mario Marietto <marietto2008@gmail.com>
wrote:

> Hello.
>
> to try to compile the correct freebsd kernel that's recognized by xen
> using the Elliott Michell code, are the commands below the correct ones ?
>
> $ truncate -s 100G xenvm.img
> $ mdconfig -f xenvm.img -u 0
> $ newfs /dev/md0
> $ mount /dev/md0 /mnt
> $ git clone https://gitlab.com/ehem/freebsd-src.git
> $ cd freebsd-src
>
> $ make  -DNO_MODULES   KERNCONF=GENERIC TARGET=arm TARGET_ARCH=armv7 buildkernel
>
> $ echo "/dev/xbd0       /       ufs     rw      1
> 1" > /mnt/etc/fstab
>
> $ nano /build/etc/ttys (add the line 'xc0 "/usr/libexec/getty Pc" xterm on secure")
> $ umount /build
> $ mdconfig -d -u 0
>
> Do you see errors ? some missing ? very thanks.
>
>
> On Tue, Nov 28, 2023 at 3:28 AM Stefano Stabellini <sstabellini@kernel.org>
> wrote:
>
>> On Mon, 27 Nov 2023, Chuck Zmudzinski wrote:
>> > On 11/27/2023 10:22 AM, Chuck Zmudzinski wrote:
>> > > On 11/27/2023 7:45 AM, Mario Marietto wrote:
>> > >> @Chuck Zmudzinski <mailto:brchuckz@netscape.net> : Stay tuned. They
>> want to help us. The xen developers are great. Very good support for us.
>> I'm sure that you can give a good contribution to understand what's our
>> problem and how to implement a fix with the help of all those good guys.
>> > >>
>> > >> On Mon, Nov 27, 2023 at 11:56 AM Roger Pau Monné <
>> roger.pau@citrix.com <mailto:roger.pau@citrix.com>> wrote:
>> > >>
>> > >>     On Mon, Nov 27, 2023 at 10:28:13AM +0000, Henry Wang wrote:
>> > >>     > +(xen-devel and Arm maintainers, including Julien)
>> > >>     >
>> > >>     > > On Nov 27, 2023, at 18:03, Mario Marietto <
>> marietto2008@gmail.com <mailto:marietto2008@gmail.com>>
>> > >>     > > wrote:
>> > >>     > >
>> > >>     > > Hello.  We have just virtualized Debian 12 on our arm (32
>> bit)
>> > >>     > > Chromebook model xe303c12 . As host / dom0 we have chosen
>> Devuan
>> > >>     > > 5,and for guest / domU,Debian 12. It works great. But our
>> goal is
>> > >>     > > different. We want to virtualize FreeBSD as domU. Can we
>> have a
>> > >>     > > working Xen PV network driver for a FreeBSD arm guest ?. I
>> found
>> > >>     > > that Julien Grall has ported the Xen drivers to FreeBSD on
>> arm. I
>> > >>     > > would like to know if Julien's work was accepted upstream by
>> > >>     > > FreeBSD, in which case FreeBSD as a Xen guest on arm should
>> work
>> > >>     > > if we enable the Xen PV drivers in the FreeBSD on arm
>> kernel. If
>> > >>     > > Julien's work was not accepted upstream by FreeBSD, we will
>> have
>> > >>     > > to find his patches and apply them ourselves to the FreeBSD
>> on arm
>> > >>     > > kernel.
>> > >>
>> > >>     I've added Elliot on Cc as he is working on upstreaming the
>> patches to
>> > >>     FreeBSD.  He will be able to provide a better update than myself.
>> > >>
>> > >>     Regards, Roger.
>> > >
>> > > I have been collaborating with Mario, and I can explain what we have
>> done so far :
>> > >
>> > > We are using Julien's patch set against an old development version of
>> FreeBSD 11
>> > > from 2014-12-03 :
>> > >
>> > >
>> https://xenbits.xen.org/gitweb/?p=people/julieng/freebsd.git;a=shortlog;h=refs/heads/xen-arm-v2.2
>> > >
>> > > We successfully built the XENVIRT kernel and FreeBSD world, and
>> created the
>> > > FreeBSD rootfs according to Julien's instructions here :
>> > >
>> > >
>> https://lists.freebsd.org/pipermail/freebsd-xen/2014-November/002202.html
>> > >
>> > > There were some adjustments to the instructions :
>> > >
>> > > To build the kernel, we used :
>> > >
>> > > $ sudo make TARGET_ARCH=armv6 KERNCONF=XENVIRT buildkernel
>> > >
>> > > instead of
>> > >
>> > > $ sudo make TARGET_ARCH=armv6 KERNCONF=XENHVM buildkernel
>> > >
>> > > The FreeBSD 'kernel' file is in ELF format and did not work, and we
>> spent
>> > > some time trying to convert it to the zImage format without realizing
>> the
>> > > build of the FreeBSD kernel creates the 'kernel.bin' file in the
>> zImage format.
>> > > So when booting with the 'kernel.bin' file instead, it actually boots
>> :
>> > >
>> > > user@devuan-bunsen ~ % sudo xl create freebsd.cfg
>> > > Parsing config from freebsd.cfg
>> > > user@devuan-bunsen ~ % sudo xl li
>> > > Name                                        ID   Mem VCPUs  State
>>  Time(s)
>> > > Domain-0                                     0   768     2
>>  r-----    1439.4
>> > > freebsd                                      1  1152     1
>>  r-----       3.0
>> > > user@devuan-bunsen ~ %
>> > >
>> > > However, the guest is still not working correctly :
>> > >
>> > > 1. Attaching the console with the -c option at creation or with
>> > >    'xl console freebsd' results in no output to the console.
>> > >
>> > > 2. The timestamp on the virtual disk image file shows that the
>> filesystem
>> > >    was at best mounted read-only, if it was mounted at all by the
>> guest
>> > >    FreeBSD kernel.
>> > >
>> > > 3. The 'xl shutdown freebsd' command does not work, it just hangs. To
>> stop
>> > >    the guest, you need to do 'xl destroy freebsd'.
>> > >
>> > > However, I think we can get the console to work and the rootfs to
>> mount because I
>> > > just realized I forgot to do the steps from Julien's instructions of
>> editing the
>> > > /etc/fstab and /etc/ttys files in the FreeBSD rootfs :
>> > >
>> > > $ echo "/dev/xbd0       /       ufs     rw      1       1" >
>> /mnt/etc/fstab
>> > > $ vi /mnt/etc/ttys (add the line 'xc0 "/usr/libexec/getty Pc" xterm
>> on secure")
>> > >
>> > > I will add those and see if the console and disk are working.
>> >
>> > Unfortunately, adding xc0 to /etc/ttys and /dev/xbd0 as the root device
>> in
>> > /etc/fstab did not make the console or disk work. Still no output on the
>> > xen console from the guest kernel, and the timestamp on the rootfs image
>> > file did not change so it did not mount read-write.
>> >
>> > We could use some advice for troubleshooting this. Now, we are blind
>> because
>> > we are not getting any xen console output But I am pleased we were able
>> to
>> > demonstrate that Julien's old patch set for FreeBSD 11 allows us to boot
>> > FreeBSD 11 on a modern version of Xen on arm - we are using the Debian
>> > stable Xen 4.17 packages.
>>
>> You can use the DEBUG hypercalls to check how far we got into the
>> booting process:
>> https://wiki.xenproject.org/wiki/Xen_ARM_DEBUG_hypercalls
>>
>> For instance add the following to FreeBSD code:
>>
>> asm volatile("hvc 0xfffd");
>>
>>
>
> --
> Mario.
>


-- 
Mario.

[-- Attachment #2: Type: text/html, Size: 9828 bytes --]

Re: We are not able to virtualize FreeBSD using xen 4.17 on Arm 32 bit

[Chuck Zmudzinski]

On 11/27/2023 9:28 PM, Stefano Stabellini wrote:
> On Mon, 27 Nov 2023, Chuck Zmudzinski wrote:
>> On 11/27/2023 10:22 AM, Chuck Zmudzinski wrote:
>> > On 11/27/2023 7:45 AM, Mario Marietto wrote:
>> >> @Chuck Zmudzinski <mailto:brchuckz@netscape.net> : Stay tuned. They want to help us. The xen developers are great. Very good support for us. I'm sure that you can give a good contribution to understand what's our problem and how to implement a fix with the help of all those good guys.
>> >> 
>> >> On Mon, Nov 27, 2023 at 11:56 AM Roger Pau Monné <roger.pau@citrix.com <mailto:roger.pau@citrix.com>> wrote:
>> >> 
>> >>     On Mon, Nov 27, 2023 at 10:28:13AM +0000, Henry Wang wrote:
>> >>     > +(xen-devel and Arm maintainers, including Julien)
>> >>     >
>> >>     > > On Nov 27, 2023, at 18:03, Mario Marietto <marietto2008@gmail.com <mailto:marietto2008@gmail.com>>
>> >>     > > wrote:
>> >>     > >
>> >>     > > Hello.  We have just virtualized Debian 12 on our arm (32 bit)
>> >>     > > Chromebook model xe303c12 . As host / dom0 we have chosen Devuan
>> >>     > > 5,and for guest / domU,Debian 12. It works great. But our goal is
>> >>     > > different. We want to virtualize FreeBSD as domU. Can we have a
>> >>     > > working Xen PV network driver for a FreeBSD arm guest ?. I found
>> >>     > > that Julien Grall has ported the Xen drivers to FreeBSD on arm. I
>> >>     > > would like to know if Julien's work was accepted upstream by
>> >>     > > FreeBSD, in which case FreeBSD as a Xen guest on arm should work
>> >>     > > if we enable the Xen PV drivers in the FreeBSD on arm kernel. If
>> >>     > > Julien's work was not accepted upstream by FreeBSD, we will have
>> >>     > > to find his patches and apply them ourselves to the FreeBSD on arm
>> >>     > > kernel.
>> >> 
>> >>     I've added Elliot on Cc as he is working on upstreaming the patches to
>> >>     FreeBSD.  He will be able to provide a better update than myself.
>> >> 
>> >>     Regards, Roger.
>> > 
>> > I have been collaborating with Mario, and I can explain what we have done so far :
>> > 
>> > We are using Julien's patch set against an old development version of FreeBSD 11
>> > from 2014-12-03 :
>> > 
>> > https://xenbits.xen.org/gitweb/?p=people/julieng/freebsd.git;a=shortlog;h=refs/heads/xen-arm-v2.2
>> > 
>> > We successfully built the XENVIRT kernel and FreeBSD world, and created the
>> > FreeBSD rootfs according to Julien's instructions here :
>> > 
>> > https://lists.freebsd.org/pipermail/freebsd-xen/2014-November/002202.html
>> > 
>> > There were some adjustments to the instructions :
>> > 
>> > To build the kernel, we used :
>> > 
>> > $ sudo make TARGET_ARCH=armv6 KERNCONF=XENVIRT buildkernel
>> > 
>> > instead of
>> > 
>> > $ sudo make TARGET_ARCH=armv6 KERNCONF=XENHVM buildkernel
>> > 
>> > The FreeBSD 'kernel' file is in ELF format and did not work, and we spent
>> > some time trying to convert it to the zImage format without realizing the
>> > build of the FreeBSD kernel creates the 'kernel.bin' file in the zImage format.
>> > So when booting with the 'kernel.bin' file instead, it actually boots :
>> > 
>> > user@devuan-bunsen ~ % sudo xl create freebsd.cfg
>> > Parsing config from freebsd.cfg
>> > user@devuan-bunsen ~ % sudo xl li
>> > Name                                        ID   Mem VCPUs	State	Time(s)
>> > Domain-0                                     0   768     2     r-----    1439.4
>> > freebsd                                      1  1152     1     r-----       3.0
>> > user@devuan-bunsen ~ %
>> > 
>> > However, the guest is still not working correctly :
>> > 
>> > 1. Attaching the console with the -c option at creation or with
>> >    'xl console freebsd' results in no output to the console.
>> > 
>> > 2. The timestamp on the virtual disk image file shows that the filesystem
>> >    was at best mounted read-only, if it was mounted at all by the guest
>> >    FreeBSD kernel.
>> > 
>> > 3. The 'xl shutdown freebsd' command does not work, it just hangs. To stop
>> >    the guest, you need to do 'xl destroy freebsd'.
>> > 
>> > However, I think we can get the console to work and the rootfs to mount because I
>> > just realized I forgot to do the steps from Julien's instructions of editing the
>> > /etc/fstab and /etc/ttys files in the FreeBSD rootfs :
>> > 
>> > $ echo "/dev/xbd0       /       ufs     rw      1       1" > /mnt/etc/fstab
>> > $ vi /mnt/etc/ttys (add the line 'xc0 "/usr/libexec/getty Pc" xterm on secure")
>> > 
>> > I will add those and see if the console and disk are working.
>> 
>> Unfortunately, adding xc0 to /etc/ttys and /dev/xbd0 as the root device in
>> /etc/fstab did not make the console or disk work. Still no output on the
>> xen console from the guest kernel, and the timestamp on the rootfs image
>> file did not change so it did not mount read-write.
>> 
>> We could use some advice for troubleshooting this. Now, we are blind because
>> we are not getting any xen console output But I am pleased we were able to
>> demonstrate that Julien's old patch set for FreeBSD 11 allows us to boot
>> FreeBSD 11 on a modern version of Xen on arm - we are using the Debian
>> stable Xen 4.17 packages.
> 
> You can use the DEBUG hypercalls to check how far we got into the
> booting process:
> https://wiki.xenproject.org/wiki/Xen_ARM_DEBUG_hypercalls
> 
> For instance add the following to FreeBSD code:
> 
> asm volatile("hvc 0xfffd");
> 

It took me a while, but I finally got this approach to work to debug the FreeBSD
kernel. Thanks!

The problem was the compiler was reporting hvc is an invalid instruction. To
get the compiler to accept the hvc instruction as valid, I first spent quite a
bit of time porting the patches from the old development version of FreeBSD 11 on
which Julien's patches were based to FreeBSD 12.4, because that old development
version of FreeBSD did not support armv7 but only armv6, and I thought maybe
the compiler is rejecting the hvc instruction because the kernel build was
targeting armv6 and I was not sure hypervisor extensions were available for
armv6. But FreeBSD 12 and higher has support to target armv7 for the kernel.
There were quite a few changes to account for between FreeBSD 11 and FreeBSD 12.4,
I had to add about 12 more patches, but I also removed some of Julien's patches
that were either applied in FreeBSD 12.4 or no longer applicable to FreeBSD 12.4.

So when I was able to build a FreeBSD 12.4 kernel + Julien's arm/xenvirt patches
targeting armv7 instead of armv6, I got the same behavior: the guest started but
no output on the console, and the compiler at first still did not accept the
hvc instruction (FreeBSD uses the clang compiler by default to build the kernel).
After some searches on the Internet I discovered that adding the -mthumb CFLAG
when compiling the objects with an hvc instruction enabled the compiler to accept
the hvc instruction.

So I was able to get output like this in the dom0 Xen console log from the hvc
instruction in the guest :

(XEN) arch/arm/traps.c:1983:d2v0 HSR=0x80000005 pc=0xffff000c gva=0xffff000c gpa=0x000000ffff000c
(XEN) arch/arm/traps.c:1983:d2v0 HSR=0x80000005 pc=0xffff000c gva=0xffff000c gpa=0x000000ffff000c
...

For now, I only put one hvc instruction in the FreeBSD code - it is where the
kernel prints the copyright and version information to the console. So I don't
understand why the message from the hvc instruction is appearing multiple times
in the Xen logs...

In any case, this provides a way to debug the boot of FreeBSD / arm on Xen, so
thanks, Stefano, for this suggestion!

Cheers,

Chuck

Re: We are not able to virtualize FreeBSD using xen 4.17 on Arm 32 bit

[Stefano Stabellini]

[-- Attachment #1: Type: text/plain, Size: 8351 bytes --]

On Mon, 11 Dec 2023, Chuck Zmudzinski wrote:
> On 11/27/2023 9:28 PM, Stefano Stabellini wrote:
> > On Mon, 27 Nov 2023, Chuck Zmudzinski wrote:
> >> On 11/27/2023 10:22 AM, Chuck Zmudzinski wrote:
> >> > On 11/27/2023 7:45 AM, Mario Marietto wrote:
> >> >> @Chuck Zmudzinski <mailto:brchuckz@netscape.net> : Stay tuned. They want to help us. The xen developers are great. Very good support for us. I'm sure that you can give a good contribution to understand what's our problem and how to implement a fix with the help of all those good guys.
> >> >> 
> >> >> On Mon, Nov 27, 2023 at 11:56 AM Roger Pau Monné <roger.pau@citrix.com <mailto:roger.pau@citrix.com>> wrote:
> >> >> 
> >> >>     On Mon, Nov 27, 2023 at 10:28:13AM +0000, Henry Wang wrote:
> >> >>     > +(xen-devel and Arm maintainers, including Julien)
> >> >>     >
> >> >>     > > On Nov 27, 2023, at 18:03, Mario Marietto <marietto2008@gmail.com <mailto:marietto2008@gmail.com>>
> >> >>     > > wrote:
> >> >>     > >
> >> >>     > > Hello.  We have just virtualized Debian 12 on our arm (32 bit)
> >> >>     > > Chromebook model xe303c12 . As host / dom0 we have chosen Devuan
> >> >>     > > 5,and for guest / domU,Debian 12. It works great. But our goal is
> >> >>     > > different. We want to virtualize FreeBSD as domU. Can we have a
> >> >>     > > working Xen PV network driver for a FreeBSD arm guest ?. I found
> >> >>     > > that Julien Grall has ported the Xen drivers to FreeBSD on arm. I
> >> >>     > > would like to know if Julien's work was accepted upstream by
> >> >>     > > FreeBSD, in which case FreeBSD as a Xen guest on arm should work
> >> >>     > > if we enable the Xen PV drivers in the FreeBSD on arm kernel. If
> >> >>     > > Julien's work was not accepted upstream by FreeBSD, we will have
> >> >>     > > to find his patches and apply them ourselves to the FreeBSD on arm
> >> >>     > > kernel.
> >> >> 
> >> >>     I've added Elliot on Cc as he is working on upstreaming the patches to
> >> >>     FreeBSD.  He will be able to provide a better update than myself.
> >> >> 
> >> >>     Regards, Roger.
> >> > 
> >> > I have been collaborating with Mario, and I can explain what we have done so far :
> >> > 
> >> > We are using Julien's patch set against an old development version of FreeBSD 11
> >> > from 2014-12-03 :
> >> > 
> >> > https://xenbits.xen.org/gitweb/?p=people/julieng/freebsd.git;a=shortlog;h=refs/heads/xen-arm-v2.2
> >> > 
> >> > We successfully built the XENVIRT kernel and FreeBSD world, and created the
> >> > FreeBSD rootfs according to Julien's instructions here :
> >> > 
> >> > https://lists.freebsd.org/pipermail/freebsd-xen/2014-November/002202.html
> >> > 
> >> > There were some adjustments to the instructions :
> >> > 
> >> > To build the kernel, we used :
> >> > 
> >> > $ sudo make TARGET_ARCH=armv6 KERNCONF=XENVIRT buildkernel
> >> > 
> >> > instead of
> >> > 
> >> > $ sudo make TARGET_ARCH=armv6 KERNCONF=XENHVM buildkernel
> >> > 
> >> > The FreeBSD 'kernel' file is in ELF format and did not work, and we spent
> >> > some time trying to convert it to the zImage format without realizing the
> >> > build of the FreeBSD kernel creates the 'kernel.bin' file in the zImage format.
> >> > So when booting with the 'kernel.bin' file instead, it actually boots :
> >> > 
> >> > user@devuan-bunsen ~ % sudo xl create freebsd.cfg
> >> > Parsing config from freebsd.cfg
> >> > user@devuan-bunsen ~ % sudo xl li
> >> > Name                                        ID   Mem VCPUs	State	Time(s)
> >> > Domain-0                                     0   768     2     r-----    1439.4
> >> > freebsd                                      1  1152     1     r-----       3.0
> >> > user@devuan-bunsen ~ %
> >> > 
> >> > However, the guest is still not working correctly :
> >> > 
> >> > 1. Attaching the console with the -c option at creation or with
> >> >    'xl console freebsd' results in no output to the console.
> >> > 
> >> > 2. The timestamp on the virtual disk image file shows that the filesystem
> >> >    was at best mounted read-only, if it was mounted at all by the guest
> >> >    FreeBSD kernel.
> >> > 
> >> > 3. The 'xl shutdown freebsd' command does not work, it just hangs. To stop
> >> >    the guest, you need to do 'xl destroy freebsd'.
> >> > 
> >> > However, I think we can get the console to work and the rootfs to mount because I
> >> > just realized I forgot to do the steps from Julien's instructions of editing the
> >> > /etc/fstab and /etc/ttys files in the FreeBSD rootfs :
> >> > 
> >> > $ echo "/dev/xbd0       /       ufs     rw      1       1" > /mnt/etc/fstab
> >> > $ vi /mnt/etc/ttys (add the line 'xc0 "/usr/libexec/getty Pc" xterm on secure")
> >> > 
> >> > I will add those and see if the console and disk are working.
> >> 
> >> Unfortunately, adding xc0 to /etc/ttys and /dev/xbd0 as the root device in
> >> /etc/fstab did not make the console or disk work. Still no output on the
> >> xen console from the guest kernel, and the timestamp on the rootfs image
> >> file did not change so it did not mount read-write.
> >> 
> >> We could use some advice for troubleshooting this. Now, we are blind because
> >> we are not getting any xen console output But I am pleased we were able to
> >> demonstrate that Julien's old patch set for FreeBSD 11 allows us to boot
> >> FreeBSD 11 on a modern version of Xen on arm - we are using the Debian
> >> stable Xen 4.17 packages.
> > 
> > You can use the DEBUG hypercalls to check how far we got into the
> > booting process:
> > https://wiki.xenproject.org/wiki/Xen_ARM_DEBUG_hypercalls
> > 
> > For instance add the following to FreeBSD code:
> > 
> > asm volatile("hvc 0xfffd");
> > 
> 
> It took me a while, but I finally got this approach to work to debug the FreeBSD
> kernel. Thanks!
> 
> The problem was the compiler was reporting hvc is an invalid instruction. To
> get the compiler to accept the hvc instruction as valid, I first spent quite a
> bit of time porting the patches from the old development version of FreeBSD 11 on
> which Julien's patches were based to FreeBSD 12.4, because that old development
> version of FreeBSD did not support armv7 but only armv6, and I thought maybe
> the compiler is rejecting the hvc instruction because the kernel build was
> targeting armv6 and I was not sure hypervisor extensions were available for
> armv6. But FreeBSD 12 and higher has support to target armv7 for the kernel.
> There were quite a few changes to account for between FreeBSD 11 and FreeBSD 12.4,
> I had to add about 12 more patches, but I also removed some of Julien's patches
> that were either applied in FreeBSD 12.4 or no longer applicable to FreeBSD 12.4.
> 
> So when I was able to build a FreeBSD 12.4 kernel + Julien's arm/xenvirt patches
> targeting armv7 instead of armv6, I got the same behavior: the guest started but
> no output on the console, and the compiler at first still did not accept the
> hvc instruction (FreeBSD uses the clang compiler by default to build the kernel).
> After some searches on the Internet I discovered that adding the -mthumb CFLAG
> when compiling the objects with an hvc instruction enabled the compiler to accept
> the hvc instruction.
> 
> So I was able to get output like this in the dom0 Xen console log from the hvc
> instruction in the guest :
> 
> (XEN) arch/arm/traps.c:1983:d2v0 HSR=0x80000005 pc=0xffff000c gva=0xffff000c gpa=0x000000ffff000c
> (XEN) arch/arm/traps.c:1983:d2v0 HSR=0x80000005 pc=0xffff000c gva=0xffff000c gpa=0x000000ffff000c
> ...
> 
> For now, I only put one hvc instruction in the FreeBSD code - it is where the
> kernel prints the copyright and version information to the console. So I don't
> understand why the message from the hvc instruction is appearing multiple times
> in the Xen logs...
> 
> In any case, this provides a way to debug the boot of FreeBSD / arm on Xen, so
> thanks, Stefano, for this suggestion!

You are welcome! :-)

I am glad you could make progress. You'll find that the hvc 0xfffd and
other debug hypercalls are really useful. Also it is easy to write using
just hvc and assembly a very simple early printk implementation. See the
__HYPERVISOR_console_io hypercall:

- immediate: 0xEA1
- r12: hypercall number 18
- r0: CONSOLEIO_write 0x0
- r1: number of chars
- r2: guest address of the string

Cheers,

Stefano

Re: We are not able to virtualize FreeBSD using xen 4.17 on Arm 32 bit

[Julien Grall]

Hi Stefano,

On 13/12/2023 00:18, Stefano Stabellini wrote:
> On Mon, 11 Dec 2023, Chuck Zmudzinski wrote:
>> On 11/27/2023 9:28 PM, Stefano Stabellini wrote:
>>> On Mon, 27 Nov 2023, Chuck Zmudzinski wrote:
>>>> On 11/27/2023 10:22 AM, Chuck Zmudzinski wrote:
>>>>> On 11/27/2023 7:45 AM, Mario Marietto wrote:
>>>>>> @Chuck Zmudzinski <mailto:brchuckz@netscape.net> : Stay tuned. They want to help us. The xen developers are great. Very good support for us. I'm sure that you can give a good contribution to understand what's our problem and how to implement a fix with the help of all those good guys.
>>>>>>
>>>>>> On Mon, Nov 27, 2023 at 11:56 AM Roger Pau Monné <roger.pau@citrix.com <mailto:roger.pau@citrix.com>> wrote:
>>>>>>
>>>>>>      On Mon, Nov 27, 2023 at 10:28:13AM +0000, Henry Wang wrote:
>>>>>>      > +(xen-devel and Arm maintainers, including Julien)
>>>>>>      >
>>>>>>      > > On Nov 27, 2023, at 18:03, Mario Marietto <marietto2008@gmail.com <mailto:marietto2008@gmail.com>>
>>>>>>      > > wrote:
>>>>>>      > >
>>>>>>      > > Hello.  We have just virtualized Debian 12 on our arm (32 bit)
>>>>>>      > > Chromebook model xe303c12 . As host / dom0 we have chosen Devuan
>>>>>>      > > 5,and for guest / domU,Debian 12. It works great. But our goal is
>>>>>>      > > different. We want to virtualize FreeBSD as domU. Can we have a
>>>>>>      > > working Xen PV network driver for a FreeBSD arm guest ?. I found
>>>>>>      > > that Julien Grall has ported the Xen drivers to FreeBSD on arm. I
>>>>>>      > > would like to know if Julien's work was accepted upstream by
>>>>>>      > > FreeBSD, in which case FreeBSD as a Xen guest on arm should work
>>>>>>      > > if we enable the Xen PV drivers in the FreeBSD on arm kernel. If
>>>>>>      > > Julien's work was not accepted upstream by FreeBSD, we will have
>>>>>>      > > to find his patches and apply them ourselves to the FreeBSD on arm
>>>>>>      > > kernel.
>>>>>>
>>>>>>      I've added Elliot on Cc as he is working on upstreaming the patches to
>>>>>>      FreeBSD.  He will be able to provide a better update than myself.
>>>>>>
>>>>>>      Regards, Roger.
>>>>>
>>>>> I have been collaborating with Mario, and I can explain what we have done so far :
>>>>>
>>>>> We are using Julien's patch set against an old development version of FreeBSD 11
>>>>> from 2014-12-03 :
>>>>>
>>>>> https://xenbits.xen.org/gitweb/?p=people/julieng/freebsd.git;a=shortlog;h=refs/heads/xen-arm-v2.2
>>>>>
>>>>> We successfully built the XENVIRT kernel and FreeBSD world, and created the
>>>>> FreeBSD rootfs according to Julien's instructions here :
>>>>>
>>>>> https://lists.freebsd.org/pipermail/freebsd-xen/2014-November/002202.html
>>>>>
>>>>> There were some adjustments to the instructions :
>>>>>
>>>>> To build the kernel, we used :
>>>>>
>>>>> $ sudo make TARGET_ARCH=armv6 KERNCONF=XENVIRT buildkernel
>>>>>
>>>>> instead of
>>>>>
>>>>> $ sudo make TARGET_ARCH=armv6 KERNCONF=XENHVM buildkernel
>>>>>
>>>>> The FreeBSD 'kernel' file is in ELF format and did not work, and we spent
>>>>> some time trying to convert it to the zImage format without realizing the
>>>>> build of the FreeBSD kernel creates the 'kernel.bin' file in the zImage format.
>>>>> So when booting with the 'kernel.bin' file instead, it actually boots :
>>>>>
>>>>> user@devuan-bunsen ~ % sudo xl create freebsd.cfg
>>>>> Parsing config from freebsd.cfg
>>>>> user@devuan-bunsen ~ % sudo xl li
>>>>> Name                                        ID   Mem VCPUs	State	Time(s)
>>>>> Domain-0                                     0   768     2     r-----    1439.4
>>>>> freebsd                                      1  1152     1     r-----       3.0
>>>>> user@devuan-bunsen ~ %
>>>>>
>>>>> However, the guest is still not working correctly :
>>>>>
>>>>> 1. Attaching the console with the -c option at creation or with
>>>>>     'xl console freebsd' results in no output to the console.
>>>>>
>>>>> 2. The timestamp on the virtual disk image file shows that the filesystem
>>>>>     was at best mounted read-only, if it was mounted at all by the guest
>>>>>     FreeBSD kernel.
>>>>>
>>>>> 3. The 'xl shutdown freebsd' command does not work, it just hangs. To stop
>>>>>     the guest, you need to do 'xl destroy freebsd'.
>>>>>
>>>>> However, I think we can get the console to work and the rootfs to mount because I
>>>>> just realized I forgot to do the steps from Julien's instructions of editing the
>>>>> /etc/fstab and /etc/ttys files in the FreeBSD rootfs :
>>>>>
>>>>> $ echo "/dev/xbd0       /       ufs     rw      1       1" > /mnt/etc/fstab
>>>>> $ vi /mnt/etc/ttys (add the line 'xc0 "/usr/libexec/getty Pc" xterm on secure")
>>>>>
>>>>> I will add those and see if the console and disk are working.
>>>>
>>>> Unfortunately, adding xc0 to /etc/ttys and /dev/xbd0 as the root device in
>>>> /etc/fstab did not make the console or disk work. Still no output on the
>>>> xen console from the guest kernel, and the timestamp on the rootfs image
>>>> file did not change so it did not mount read-write.
>>>>
>>>> We could use some advice for troubleshooting this. Now, we are blind because
>>>> we are not getting any xen console output But I am pleased we were able to
>>>> demonstrate that Julien's old patch set for FreeBSD 11 allows us to boot
>>>> FreeBSD 11 on a modern version of Xen on arm - we are using the Debian
>>>> stable Xen 4.17 packages.
>>>
>>> You can use the DEBUG hypercalls to check how far we got into the
>>> booting process:
>>> https://wiki.xenproject.org/wiki/Xen_ARM_DEBUG_hypercalls
>>>
>>> For instance add the following to FreeBSD code:
>>>
>>> asm volatile("hvc 0xfffd");
>>>
>>
>> It took me a while, but I finally got this approach to work to debug the FreeBSD
>> kernel. Thanks!
>>
>> The problem was the compiler was reporting hvc is an invalid instruction. To
>> get the compiler to accept the hvc instruction as valid, I first spent quite a
>> bit of time porting the patches from the old development version of FreeBSD 11 on
>> which Julien's patches were based to FreeBSD 12.4, because that old development
>> version of FreeBSD did not support armv7 but only armv6, and I thought maybe
>> the compiler is rejecting the hvc instruction because the kernel build was
>> targeting armv6 and I was not sure hypervisor extensions were available for
>> armv6. But FreeBSD 12 and higher has support to target armv7 for the kernel.
>> There were quite a few changes to account for between FreeBSD 11 and FreeBSD 12.4,
>> I had to add about 12 more patches, but I also removed some of Julien's patches
>> that were either applied in FreeBSD 12.4 or no longer applicable to FreeBSD 12.4.
>>
>> So when I was able to build a FreeBSD 12.4 kernel + Julien's arm/xenvirt patches
>> targeting armv7 instead of armv6, I got the same behavior: the guest started but
>> no output on the console, and the compiler at first still did not accept the
>> hvc instruction (FreeBSD uses the clang compiler by default to build the kernel).
>> After some searches on the Internet I discovered that adding the -mthumb CFLAG
>> when compiling the objects with an hvc instruction enabled the compiler to accept
>> the hvc instruction.
>>
>> So I was able to get output like this in the dom0 Xen console log from the hvc
>> instruction in the guest :
>>
>> (XEN) arch/arm/traps.c:1983:d2v0 HSR=0x80000005 pc=0xffff000c gva=0xffff000c gpa=0x000000ffff000c
>> (XEN) arch/arm/traps.c:1983:d2v0 HSR=0x80000005 pc=0xffff000c gva=0xffff000c gpa=0x000000ffff000c
>> ...
>>
>> For now, I only put one hvc instruction in the FreeBSD code - it is where the
>> kernel prints the copyright and version information to the console. So I don't
>> understand why the message from the hvc instruction is appearing multiple times
>> in the Xen logs...
>>
>> In any case, this provides a way to debug the boot of FreeBSD / arm on Xen, so
>> thanks, Stefano, for this suggestion!
> 
> You are welcome! :-)
> 
> I am glad you could make progress. You'll find that the hvc 0xfffd and
> other debug hypercalls are really useful. Also it is easy to write using
> just hvc and assembly a very simple early printk implementation. See the
> __HYPERVISOR_console_io hypercall:
> 
> - immediate: 0xEA1
> - r12: hypercall number 18
> - r0: CONSOLEIO_write 0x0
> - r1: number of chars
> - r2: guest address of the string

A word of caution, this may not work properly if you have the MMU turned 
off. The string would have to be static to prevent any caching.

Cheers,

-- 
Julien Grall