HVF EL2 support in QEMU (aka FEAT_NV/FEAT_NV2) for MacOS

HVF EL2 support in QEMU (aka FEAT_NV/FEAT_NV2) for MacOS

[Alex Bennée]

Hi all,

Has anyone had a chance to play with HVF's EL2 support?

Looking at Apple's API docs they provide the following Nested
virtualization APIs since MacOS 15.0+:

  hv_vm_config_get_el2_supported
  hv_vm_config_get_el2_enabled
  hv_vm_config_set_el2_enabled

although they don't give any indication to which silicon supports this.
I'm making the assumption that this is available on M4 silicon but it
would be nice to know if anyone has verified this or has access to the
silicon to check.

Wider context:

We are interested in this for the SplitAccel patches that Philippe is
working on:

  https://gitlab.com/philmd/qemu/-/tags/split-accel-rfc-v8

(preparatory patches are already on list).

The SplitAccel approach works by dynamically switch guests between HVF
and TCG so we can emulate things that HVF can't (for example EL3/EL2)
but still run the main EL1/EL0 workload at full speed.

Currently when we detect the guest enables stage 2 page tables we have
to switch to TCG to run because it would be impractical to manage the
fragmented layer 1 guest memory to present a useful memory to the layer
2 guest. This works but obviously means that any layer 2 guests run a
lot slower. With a fully virtualised EL2 we can just worry about
emulating the less frequently called EL3 services.

Marc mentioned UTM already has NV support although I couldn't find any
references to the above APIs in the utm-edition branch. 

So to recap:

  - do we know which Apple silicon supports FEAT_NV2?
  - has anyone played with enabling it in QEMU or its forks?
  - what is missing to get this enabled upstream?

And I'd follow that with:

  - is anyone interested in sponsoring some Mac hardware so we could
    enable proper HVF testing in our CI?

Thanks,

-- 
Alex Bennée
Virtualisation Tech Lead @ Linaro

Re: HVF EL2 support in QEMU (aka FEAT_NV/FEAT_NV2) for MacOS

[Marc Zyngier]

On Tue, 15 Jul 2025 12:15:52 +0100,
Alex Bennée <alex.bennee@linaro.org> wrote:

>   - do we know which Apple silicon supports FEAT_NV2?

M2 and latter definitely support FEAT_NV2. That's how KVM NV support
has been developed for two years until I was given better HW.

Whether Apple supports NV on M2 in HVF, I have no idea. The rumour
mill says "no", but I don't have a way to check. The M3 I use at $WORK
is definitely able to give me EL2 without VHE with UTM. I haven't
played with M4, but I have it on the record that it behaves like M3
with UTM.

	M.

-- 
Without deviation from the norm, progress is not possible.

Re: HVF EL2 support in QEMU (aka FEAT_NV/FEAT_NV2) for MacOS

[Joelle van Dyne]

UTM currently supports NV only with the Apple Virtualization backend,
not QEMU HVF. While M2 supports NV, it is not enabled by XNU kernel
and `hv_vm_config_get_el2_supported` returns false. I heard there was
some compatibility issue in the hardware. M3 and newer generations
fully support NV in hardware and by XNU.

On Tue, Jul 15, 2025 at 4:51 AM Marc Zyngier <maz@kernel.org> wrote:
>
> On Tue, 15 Jul 2025 12:15:52 +0100,
> Alex Bennée <alex.bennee@linaro.org> wrote:
>
> >   - do we know which Apple silicon supports FEAT_NV2?
>
> M2 and latter definitely support FEAT_NV2. That's how KVM NV support
> has been developed for two years until I was given better HW.
>
> Whether Apple supports NV on M2 in HVF, I have no idea. The rumour
> mill says "no", but I don't have a way to check. The M3 I use at $WORK
> is definitely able to give me EL2 without VHE with UTM. I haven't
> played with M4, but I have it on the record that it behaves like M3
> with UTM.
>
>         M.
>
> --
> Without deviation from the norm, progress is not possible.

Re: HVF EL2 support in QEMU (aka FEAT_NV/FEAT_NV2) for MacOS

[Danny Canter]

Joelle is correct, M3 and newer SoCs have support for the EL2 APIs.

-Danny

> On Jul 15, 2025, at 8:53 AM, Joelle van Dyne <j@getutm.app> wrote:
> 
> UTM currently supports NV only with the Apple Virtualization backend,
> not QEMU HVF. While M2 supports NV, it is not enabled by XNU kernel
> and `hv_vm_config_get_el2_supported` returns false. I heard there was
> some compatibility issue in the hardware. M3 and newer generations
> fully support NV in hardware and by XNU.
> 
> On Tue, Jul 15, 2025 at 4:51 AM Marc Zyngier <maz@kernel.org> wrote:
>> 
>> On Tue, 15 Jul 2025 12:15:52 +0100,
>> Alex Bennée <alex.bennee@linaro.org> wrote:
>> 
>>>  - do we know which Apple silicon supports FEAT_NV2?
>> 
>> M2 and latter definitely support FEAT_NV2. That's how KVM NV support
>> has been developed for two years until I was given better HW.
>> 
>> Whether Apple supports NV on M2 in HVF, I have no idea. The rumour
>> mill says "no", but I don't have a way to check. The M3 I use at $WORK
>> is definitely able to give me EL2 without VHE with UTM. I haven't
>> played with M4, but I have it on the record that it behaves like M3
>> with UTM.
>> 
>>        M.
>> 
>> --
>> Without deviation from the norm, progress is not possible.

Re: HVF EL2 support in QEMU (aka FEAT_NV/FEAT_NV2) for MacOS

[Alex Bennée]

Danny Canter <danny_canter@apple.com> writes:

> Joelle is correct, M3 and newer SoCs have support for the EL2 APIs.

Thanks for the confirmation. I'm looking at getting a M4 Mini for my
team so we can help review and test patches for HVF going forward.

Are you aware of any work that needs doing in the wider QEMU to support
nested virt or should it just be a case of doing the plumbing in
accel/hvf to turn it on?

>
> -Danny
>
>> On Jul 15, 2025, at 8:53 AM, Joelle van Dyne <j@getutm.app> wrote:
>> 
>> UTM currently supports NV only with the Apple Virtualization backend,
>> not QEMU HVF. While M2 supports NV, it is not enabled by XNU kernel
>> and `hv_vm_config_get_el2_supported` returns false. I heard there was
>> some compatibility issue in the hardware. M3 and newer generations
>> fully support NV in hardware and by XNU.
>> 
>> On Tue, Jul 15, 2025 at 4:51 AM Marc Zyngier <maz@kernel.org> wrote:
>>> 
>>> On Tue, 15 Jul 2025 12:15:52 +0100,
>>> Alex Bennée <alex.bennee@linaro.org> wrote:
>>> 
>>>>  - do we know which Apple silicon supports FEAT_NV2?
>>> 
>>> M2 and latter definitely support FEAT_NV2. That's how KVM NV support
>>> has been developed for two years until I was given better HW.
>>> 
>>> Whether Apple supports NV on M2 in HVF, I have no idea. The rumour
>>> mill says "no", but I don't have a way to check. The M3 I use at $WORK
>>> is definitely able to give me EL2 without VHE with UTM. I haven't
>>> played with M4, but I have it on the record that it behaves like M3
>>> with UTM.
>>> 
>>>        M.
>>> 
>>> --
>>> Without deviation from the norm, progress is not possible.

-- 
Alex Bennée
Virtualisation Tech Lead @ Linaro

Re: HVF EL2 support in QEMU (aka FEAT_NV/FEAT_NV2) for MacOS

[Danny Canter]

[-- Attachment #1: Type: text/plain, Size: 2699 bytes --]

> Are you aware of any work that needs doing in the wider QEMU to support nested virt or should it just be a case of doing the plumbing inaccel/hvf to turn it on?

I’m not the greatest person to chime in unfortunately as I’m not an expert on QEMU by any means.

What I can offer that may be of assistance:

- Fundamentally, the APIs expose the EL2 states. The state shadowing is managed by Hypervisor.framework directly.
- The new GIC APIs take care of interrupt injection, and would need to be used in concert with the EL2 APIs if you need interrupt support (highly likely 😬). This would
be where I’d investigate on the QEMU end first to see if this would fit within the architecture without a bunch of reworks. https://developer.apple.com/documentation/hypervisor/hv_gic_create(_:)


> On Jul 16, 2025, at 5:17 AM, Alex Bennée <alex.bennee@linaro.org> wrote:
> 
> Danny Canter <danny_canter@apple.com <mailto:danny_canter@apple.com>> writes:
> 
>> Joelle is correct, M3 and newer SoCs have support for the EL2 APIs.
> 
> Thanks for the confirmation. I'm looking at getting a M4 Mini for my
> team so we can help review and test patches for HVF going forward.
> 
> Are you aware of any work that needs doing in the wider QEMU to support
> nested virt or should it just be a case of doing the plumbing in
> accel/hvf to turn it on?
> 
>> 
>> -Danny
>> 
>>> On Jul 15, 2025, at 8:53 AM, Joelle van Dyne <j@getutm.app> wrote:
>>> 
>>> UTM currently supports NV only with the Apple Virtualization backend,
>>> not QEMU HVF. While M2 supports NV, it is not enabled by XNU kernel
>>> and `hv_vm_config_get_el2_supported` returns false. I heard there was
>>> some compatibility issue in the hardware. M3 and newer generations
>>> fully support NV in hardware and by XNU.
>>> 
>>> On Tue, Jul 15, 2025 at 4:51 AM Marc Zyngier <maz@kernel.org> wrote:
>>>> 
>>>> On Tue, 15 Jul 2025 12:15:52 +0100,
>>>> Alex Bennée <alex.bennee@linaro.org> wrote:
>>>> 
>>>>> - do we know which Apple silicon supports FEAT_NV2?
>>>> 
>>>> M2 and latter definitely support FEAT_NV2. That's how KVM NV support
>>>> has been developed for two years until I was given better HW.
>>>> 
>>>> Whether Apple supports NV on M2 in HVF, I have no idea. The rumour
>>>> mill says "no", but I don't have a way to check. The M3 I use at $WORK
>>>> is definitely able to give me EL2 without VHE with UTM. I haven't
>>>> played with M4, but I have it on the record that it behaves like M3
>>>> with UTM.
>>>> 
>>>>       M.
>>>> 
>>>> --
>>>> Without deviation from the norm, progress is not possible.
> 
> -- 
> Alex Bennée
> Virtualisation Tech Lead @ Linaro


[-- Attachment #2: Type: text/html, Size: 12848 bytes --]

Re: HVF EL2 support in QEMU (aka FEAT_NV/FEAT_NV2) for MacOS

[Peter Maydell]

On Thu, 17 Jul 2025 at 16:16, Danny Canter <danny_canter@apple.com> wrote:
> - The new GIC APIs take care of interrupt injection, and would need to be used in concert with the EL2 APIs if you need interrupt support (highly likely 😬). This would
> be where I’d investigate on the QEMU end first to see if this would fit within the architecture without a bunch of reworks. https://developer.apple.com/documentation/hypervisor/hv_gic_create(_:)

Ah, Hypervisor.framework supports providing a GIC now? We
should definitely switch to using that instead of our
userspace fully emulated GICv3 whenever the host supports it.

thanks
-- PMM

Re: HVF EL2 support in QEMU (aka FEAT_NV/FEAT_NV2) for MacOS

[Mark Burton]

[-- Attachment #1: Type: text/plain, Size: 1642 bytes --]

So long as we can also switch to the emulated GIC when we want/need to :-) (looking at you KVM :-))

Cheers
Mark

Sent from Outlook for Android<https://aka.ms/AAb9ysg>
________________________________
From: Peter Maydell <peter.maydell@linaro.org>
Sent: Thursday, July 17, 2025 5:36:18 PM
To: Danny Canter <danny_canter@apple.com>
Cc: Alex Bennée <alex.bennee@linaro.org>; Joelle van Dyne <j@getutm.app>; Marc Zyngier <maz@kernel.org>; qemu-devel <qemu-devel@nongnu.org>; Alexander Graf <agraf@csgraf.de>; Ynddal <mads@ynddal.dk>; Cameron Esfahani <dirty@apple.com>; Roman Bolshakov <rbolshakov@ddn.com>; Phil Dennis-Jordan <phil@philjordan.eu>; Philippe Mathieu-Daudé <philmd@linaro.org>; Richard Henderson <richard.henderson@linaro.org>; Mark Burton <mburton@qti.qualcomm.com>
Subject: Re: HVF EL2 support in QEMU (aka FEAT_NV/FEAT_NV2) for MacOS

WARNING: This email originated from outside of Qualcomm. Please be wary of any links or attachments, and do not enable macros.

On Thu, 17 Jul 2025 at 16:16, Danny Canter <danny_canter@apple.com> wrote:
> - The new GIC APIs take care of interrupt injection, and would need to be used in concert with the EL2 APIs if you need interrupt support (highly likely 😬). This would
> be where I’d investigate on the QEMU end first to see if this would fit within the architecture without a bunch of reworks. https://developer.apple.com/documentation/hypervisor/hv_gic_create(_:)

Ah, Hypervisor.framework supports providing a GIC now? We
should definitely switch to using that instead of our
userspace fully emulated GICv3 whenever the host supports it.

thanks
-- PMM

[-- Attachment #2: Type: text/html, Size: 3180 bytes --]

Re: HVF EL2 support in QEMU (aka FEAT_NV/FEAT_NV2) for MacOS

[Marc Zyngier]

On Thu, 17 Jul 2025 16:45:37 +0100,
Mark Burton <mburton@qti.qualcomm.com> wrote:
> 
> So long as we can also switch to the emulated GIC when we want/need
> to :-) (looking at you KVM :-))

KVM really doesn't need such a non-feature. Specially as it cannot be
implemented without an actual GICv3 in HW, for obvious reasons.

My not-so-secret plan is to eventually deprecate the userspace
interface anyway, as it has been bitrotting for quite some time, and
can be replaced with an in-kernel GICv2 emulation.

But this is getting majorly off-topic in this thread...

	M.

-- 
Without deviation from the norm, progress is not possible.

Re: HVF EL2 support in QEMU (aka FEAT_NV/FEAT_NV2) for MacOS

[Mark Burton]

> On 17 Jul 2025, at 18:13, Marc Zyngier <maz@kernel.org> wrote:
> 
> WARNING: This email originated from outside of Qualcomm. Please be wary of any links or attachments, and do not enable macros.
> 
> On Thu, 17 Jul 2025 16:45:37 +0100,
> Mark Burton <mburton@qti.qualcomm.com> wrote:
>> 
>> So long as we can also switch to the emulated GIC when we want/need
>> to :-) (looking at you KVM :-))
> 
> KVM really doesn't need such a non-feature. Specially as it cannot be
> implemented without an actual GICv3 in HW, for obvious reasons.
> 
> My not-so-secret plan is to eventually deprecate the userspace
> interface anyway, as it has been bitrotting for quite some time, and
> can be replaced with an in-kernel GICv2 emulation.
> 
> But this is getting majorly off-topic in this thread...


Different use case, but yes, I agree the user space interface interface is an issue….

Cheers
Mark.

> 
>        M.
> 
> --
> Without deviation from the norm, progress is not possible.