Xen for Apple Silicon (M1 and beyond)

Xen for Apple Silicon (M1 and beyond)

[Techguru]

[-- Attachment #1: Type: text/plain, Size: 750 bytes --]

Hello,

Stefano, on IRC, suggested that I start a discussion on this mailing
list regarding my intention to bring up a fully function XEN on Apple
Silicon (M1 and beyond).

I am in the process of getting up to speed on your governance
policies, applied for Coverity access to use some of the known issues
there as training wheels, and putting my gitlab fork into good working
order with CI.

I would rather not duplicate effort which has already been proposed,
so definitely open to any pointers, any requests for how to chunk
things into smaller patches, any requests for particular
ordering/sequencing of patch sets.

Some obvious areas:   16k pages; device trees; interrupts; power
management; any essential device drivers.



[-- Attachment #2: Type: text/html, Size: 1753 bytes --]

Re: Xen for Apple Silicon (M1 and beyond)

[Jan Beulich]

On 24.07.2024 03:14, Techguru wrote:
> I am in the process of getting up to speed on your governance
> policies, applied for Coverity access to use some of the known issues
> there as training wheels, and putting my gitlab fork into good working
> order with CI.
> 
> I would rather not duplicate effort which has already been proposed,
> so definitely open to any pointers, any requests for how to chunk
> things into smaller patches, any requests for particular
> ordering/sequencing of patch sets.
> 
> Some obvious areas:   16k pages;

For this I'd suggest that you start with a conceptual model rather than
thinking of patches straight away. It has been the topic of past
discussions, without - iirc - conclusive results. I'm sure you're aware
that 4k page size is, sadly, baked into certain parts of the public
interface (see e.g. io/ring.h). Otoh it may well be that you're goal is
explicitly not to be able to run any guests working with 4k pages. If
so, what would need ensuring is that such guests would cleanly fail to
start, rather than crash early in a cryptic way.

Jan

> device trees; interrupts; power
> management; any essential device drivers.
> 
> 
> 

Re: Xen for Apple Silicon (M1 and beyond)

[Stefano Stabellini]

On Wed, 24 Jul 2024, Techguru wrote:
> Hello,
> Stefano, on IRC, suggested that I start a discussion on this mailing
> list regarding my intention to bring up a fully function XEN on Apple
> Silicon (M1 and beyond).  I am in the process of getting up to speed
> on your governance policies, applied for Coverity access to use some
> of the known issues there as training wheels, and putting my gitlab
> fork into good working order with CI.
> 
> I would rather not duplicate effort which has already been proposed,
> so definitely open to any pointers, any requests for how to chunk
> things into smaller patches, any requests for particular
> ordering/sequencing of patch sets.
> 
> Some obvious areas:
> 16k pages;

For this, do you know if 4K pages are supported (in addition to 16K)? It
would be a lot easier to keep running Xen using 4K pages and run guests
on top of Xen which use 16K pages. In the past, we had Linux using 64K
pages running on top of Xen using 4K pages for example.


> device trees;

You might be able to use the device tree which is used to boot Linux on
Apple silicon


> interrupts;

If the interrupt controller is not an ARM GIC, then we need a driver for
it in Xen


> power management;
> any essential device drivers.

In addition to the interrupt controller, we might need drivers for the
UART or framebuffer to see Xen output. Also the timer. Xen has a driver
for the ARM Generic Timer but if Apple has a different timer, that is
something else to consider.

Re: Xen for Apple Silicon (M1 and beyond)

[Techguru]

[-- Attachment #1: Type: text/plain, Size: 2664 bytes --]

A bit about Apple Silicon M1(2,3,4) architecture, SoC and motherboard.
(AFAIK)

There are no EFI assumptions nor contracts, and no EFI firmware.

Between those 4 generations of silicon, there are about 80 different
roles "SoC cores/components" can play, each with its own signed Apple
Firmware.

Most of those SoC cores/functions have nothing to do with the
traditional ARM compute cores.

At least one of those SoC cores is actually an x86 processor core (on
models which support HDMI), running x86 firmware.

Many of the other non-compute cores (Networking, USC (A-C),
Thunderbolt ... yada yada) are various size Arm cores running an Apple
RTOS Variant of L4 microkernel.

Each of those, again, has its own Apple Signed firmware which is
paired with a particular Darwin kernel release.

One point of friction is that there is no stable ABI spec for each of
the firmware modules.   Apple reserves the right to completely change
the firmware's

interface with the paired Darwin kernel.

It is perhaps best to think of the SoC and related motherboard
hardware as a "compute cluster" rather than a single von-neuman
machine.

I could be wrong about a lot of the above, and invite corrections with
citations.

> For this, do you know if 4K pages are supported (in addition to 16K)? It
> 
> would be a lot easier to keep running Xen using 4K pages and run guests
> 
> on top of Xen which use 16K pages. In the past, we had Linux using 64K
> 
> pages running on top of Xen using 4K pages for example.

Given the state of all of the _other_ SoC cores running L4 RTOS, I
have reservations about running with anything other than 16k native
pages.  Will have to

investigate what options, if any, are available for keeping 16k pages
for existing firmware, while presenting 4k pages to guest OS.

However, both Rosetta and Docker appear to run 4k guests just fine, so
not sure how many pushups they are doing in the process.

TBD

>>  device trees;
> 
> You might be able to use the device tree which is used to boot Linux
> on
> 
> Apple silicon

Yes, the Asahi team has done some wizard level heavy lifting in that
area.  If I recall correctly, their model is let apple boot loader
load the apple signed firmware, pass off to the Asahi boot kernel,
 then present an EFI compatible DT and driver surface on top of that
for boot-essential drivers.  They also have a fairly nifty
lightweight hypervisor that assists with debug and ultimately does
boot via u-boot -> grub -> linux in the current incarnation

> 



[-- Attachment #2: Type: text/html, Size: 5470 bytes --]

Re: Xen for Apple Silicon (M1 and beyond)

[Stefano Stabellini]

[-- Attachment #1: Type: text/plain, Size: 2155 bytes --]

On Thu, 25 Jul 2024, Techguru wrote:
> A bit about Apple Silicon M1(2,3,4) architecture, SoC and motherboard. (AFAIK)
> 
> There are no EFI assumptions nor contracts, and no EFI firmware.
> Between those 4 generations of silicon, there are about 80 different roles "SoC cores/components" can play, each with its own signed Apple
> Firmware.
> Most of those SoC cores/functions have nothing to do with the traditional ARM compute cores.
> At least one of those SoC cores is actually an x86 processor core (on models which support HDMI), running x86 firmware.
> Many of the other non-compute cores (Networking, USC (A-C), Thunderbolt ... yada yada) are various size Arm cores running an Apple RTOS
> Variant of L4 microkernel.
> Each of those, again, has its own Apple Signed firmware which is paired with a particular Darwin kernel release.
> One point of friction is that there is no stable ABI spec for each of the firmware modules.   Apple reserves the right to completely change
> the firmware's
> interface with the paired Darwin kernel.
> 
> It is perhaps best to think of the SoC and related motherboard hardware as a "compute cluster" rather than a single von-neuman machine.
> 
> I could be wrong about a lot of the above, and invite corrections with citations.
> 
> For this, do you know if 4K pages are supported (in addition to 16K)? It
> would be a lot easier to keep running Xen using 4K pages and run guests
> on top of Xen which use 16K pages. In the past, we had Linux using 64K
> pages running on top of Xen using 4K pages for example.
> 
> Given the state of all of the _other_ SoC cores running L4 RTOS, I have reservations about running with anything other than 16k native
> pages.  Will have to
> investigate what options, if any, are available for keeping 16k pages for existing firmware, while presenting 4k pages to guest OS.
> 
> However, both Rosetta and Docker appear to run 4k guests just fine, so not sure how many pushups they are doing in the process.

I just wanted to clarify that I am suggesting to run Xen with 4K pages
and the guests with 16K pages. I am *not* suggesting to change the page
granularity of the guests.