[PATCH] system: Try hardware accelerators (KVM, HVF) before software one (TCG)

[PATCH] system: Try hardware accelerators (KVM, HVF) before software one (TCG)

[Philippe Mathieu-Daudé]

As Daniel suggested [*]:

> We should consider to rank HVF above TCG, on the basis
> that HW acceleration is faster and should provide a
> host<->guest security boundary that we don't claim for TCG

[*] https://lore.kernel.org/qemu-devel/Z07YASl2Pd3CPtjE@redhat.com/
Suggested-by: Daniel P. Berrangé <berrange@redhat.com>
Signed-off-by: Philippe Mathieu-Daudé <philmd@linaro.org>
---
 system/vl.c | 5 ++---
 1 file changed, 2 insertions(+), 3 deletions(-)

diff --git a/system/vl.c b/system/vl.c
index 0843b7ab49b..e20391281f8 100644
--- a/system/vl.c
+++ b/system/vl.c
@@ -2379,7 +2379,6 @@ static void configure_accelerators(const char *progname)
             /* Select the default accelerator */
             bool have_tcg = accel_find("tcg");
             bool have_kvm = accel_find("kvm");
-            bool have_hvf = accel_find("hvf");
 
             if (have_tcg && have_kvm) {
                 if (g_str_has_suffix(progname, "kvm")) {
@@ -2390,10 +2389,10 @@ static void configure_accelerators(const char *progname)
                 }
             } else if (have_kvm) {
                 accelerators = "kvm";
+            } else if (accel_find("hvf")) {
+                accelerators = "hvf";
             } else if (have_tcg) {
                 accelerators = "tcg";
-            } else if (have_hvf) {
-                accelerators = "hvf";
             } else {
                 error_report("No accelerator selected and"
                              " no default accelerator available");
-- 
2.47.1

Re: [PATCH] system: Try hardware accelerators (KVM, HVF) before software one (TCG)

[Daniel P. Berrangé]

On Fri, Jan 03, 2025 at 04:05:58PM +0100, Philippe Mathieu-Daudé wrote:
> As Daniel suggested [*]:
> 
> > We should consider to rank HVF above TCG, on the basis
> > that HW acceleration is faster and should provide a
> > host<->guest security boundary that we don't claim for TCG
> 
> [*] https://lore.kernel.org/qemu-devel/Z07YASl2Pd3CPtjE@redhat.com/

Note, my statement above was on the basis that HVF passes all our
functional tests, thus indicating a decent level of confidence
in the correctness of the HVF impl.

If anyone knows any show stopper problems with HVF that would
justify blocking its promotion ahead of TCG.... say now.

> Suggested-by: Daniel P. Berrangé <berrange@redhat.com>
> Signed-off-by: Philippe Mathieu-Daudé <philmd@linaro.org>
> ---
>  system/vl.c | 5 ++---
>  1 file changed, 2 insertions(+), 3 deletions(-)

Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>


With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [PATCH] system: Try hardware accelerators (KVM, HVF) before software one (TCG)

[Philippe Mathieu-Daudé]

On 3/1/25 16:15, Daniel P. Berrangé wrote:
> On Fri, Jan 03, 2025 at 04:05:58PM +0100, Philippe Mathieu-Daudé wrote:
>> As Daniel suggested [*]:
>>
>>> We should consider to rank HVF above TCG, on the basis
>>> that HW acceleration is faster and should provide a
>>> host<->guest security boundary that we don't claim for TCG
>>
>> [*] https://lore.kernel.org/qemu-devel/Z07YASl2Pd3CPtjE@redhat.com/
> 
> Note, my statement above was on the basis that HVF passes all our
> functional tests, thus indicating a decent level of confidence
> in the correctness of the HVF impl.

Indeed, I forgot about that, and only tested in my 'HVF-only'
directory before posting, but ...

> If anyone knows any show stopper problems with HVF that would
> justify blocking its promotion ahead of TCG.... say now.

... here we go:

  3/15 qemu:qtest+qtest-aarch64 / qtest-aarch64/migration-test 
       ERROR            0.88s   killed by signal 11 SIGSEGV

>> Suggested-by: Daniel P. Berrangé <berrange@redhat.com>
>> Signed-off-by: Philippe Mathieu-Daudé <philmd@linaro.org>
>> ---
>>   system/vl.c | 5 ++---
>>   1 file changed, 2 insertions(+), 3 deletions(-)
> 
> Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
> 
> 
> With regards,
> Daniel

Re: [PATCH] system: Try hardware accelerators (KVM, HVF) before software one (TCG)

[Peter Xu]

On Fri, Jan 03, 2025 at 06:16:38PM +0100, Philippe Mathieu-Daudé wrote:
> On 3/1/25 16:15, Daniel P. Berrangé wrote:
> > On Fri, Jan 03, 2025 at 04:05:58PM +0100, Philippe Mathieu-Daudé wrote:
> > > As Daniel suggested [*]:
> > > 
> > > > We should consider to rank HVF above TCG, on the basis
> > > > that HW acceleration is faster and should provide a
> > > > host<->guest security boundary that we don't claim for TCG
> > > 
> > > [*] https://lore.kernel.org/qemu-devel/Z07YASl2Pd3CPtjE@redhat.com/
> > 
> > Note, my statement above was on the basis that HVF passes all our
> > functional tests, thus indicating a decent level of confidence
> > in the correctness of the HVF impl.
> 
> Indeed, I forgot about that, and only tested in my 'HVF-only'
> directory before posting, but ...
> 
> > If anyone knows any show stopper problems with HVF that would
> > justify blocking its promotion ahead of TCG.... say now.
> 
> ... here we go:
> 
>  3/15 qemu:qtest+qtest-aarch64 / qtest-aarch64/migration-test       ERROR
> 0.88s   killed by signal 11 SIGSEGV

Hmm.. I think migration-test specifies either kvm or tcg in all its tests,
so I don't yet know why this patch can affect it..

migrate_start():
    cmd_source = g_strdup_printf("-accel kvm%s -accel tcg "
                                 "-machine %s,%s "
                                 ...);

May need a closer look on the crashed stack.

-- 
Peter Xu

Re: [PATCH] system: Try hardware accelerators (KVM, HVF) before software one (TCG)

[Phil Dennis-Jordan]

[-- Attachment #1: Type: text/plain, Size: 2761 bytes --]

On Fri, 3 Jan 2025 at 16:16, Daniel P. Berrangé <berrange@redhat.com> wrote:

> On Fri, Jan 03, 2025 at 04:05:58PM +0100, Philippe Mathieu-Daudé wrote:
> > As Daniel suggested [*]:
> >
> > > We should consider to rank HVF above TCG, on the basis
> > > that HW acceleration is faster and should provide a
> > > host<->guest security boundary that we don't claim for TCG
> >
> > [*] https://lore.kernel.org/qemu-devel/Z07YASl2Pd3CPtjE@redhat.com/
>
> Note, my statement above was on the basis that HVF passes all our
> functional tests, thus indicating a decent level of confidence
> in the correctness of the HVF impl.
>
> If anyone knows any show stopper problems with HVF that would
> justify blocking its promotion ahead of TCG.... say now.
>

I don't know about showstoppers, but:

1. As far as I'm aware there are/were problems with the virtual IOMMU
devices in HVF. It's been a while (~half a year?) since I tried them, but I
had problems getting guests booted with intel_iommu etc.

2. I think there might also be a few remaining edge cases where the x86
instruction emulation on fault/trap is incomplete. Most notably, MMIO using
SSE/AVX/etc. instructions will, I think, fail. In practice this is a fairly
obscure use case - I'm not aware of any guest OS that actually performs
MMIO using these instructions. I have a patch kicking around that adds
support for missing 64-bit variants of common scalar arithmetic
instructions with memory operands. I can dig that up and post it - do we
have a good way of adding tests for this kind of thing?

3. As far as I'm aware, there's no CI happening on HVF? Or has that
changed? macOS is notoriously a pain in the rear in terms of CI thanks to
its licensing, and the big cloud CI platforms tend to run it in a VM which
in turn typically doesn't support nested HVF. I've been working on an
on-prem solution to provisioning bare-metal Macs to run clean-slate OS
images for CI. This has been a side project though and I haven't had the
resources to focus on that project to see it through. It might be possible
to do this in the cloud on Amazon's EC2 Mac Minis as well, but those aren't
exactly cheap.



> > Suggested-by: Daniel P. Berrangé <berrange@redhat.com>
> > Signed-off-by: Philippe Mathieu-Daudé <philmd@linaro.org>
> > ---
> >  system/vl.c | 5 ++---
> >  1 file changed, 2 insertions(+), 3 deletions(-)
>
> Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
>
>
> With regards,
> Daniel
> --
> |: https://berrange.com      -o-
> https://www.flickr.com/photos/dberrange :|
> |: https://libvirt.org         -o-
> https://fstop138.berrange.com :|
> |: https://entangle-photo.org    -o-
> https://www.instagram.com/dberrange :|
>
>
>

[-- Attachment #2: Type: text/html, Size: 4274 bytes --]

Re: [PATCH] system: Try hardware accelerators (KVM, HVF) before software one (TCG)

[Daniel P. Berrangé]

On Sat, Jan 04, 2025 at 12:28:14PM +0100, Phil Dennis-Jordan wrote:
> On Fri, 3 Jan 2025 at 16:16, Daniel P. Berrangé <berrange@redhat.com> wrote:
> 
> > On Fri, Jan 03, 2025 at 04:05:58PM +0100, Philippe Mathieu-Daudé wrote:
> > > As Daniel suggested [*]:
> > >
> > > > We should consider to rank HVF above TCG, on the basis
> > > > that HW acceleration is faster and should provide a
> > > > host<->guest security boundary that we don't claim for TCG
> > >
> > > [*] https://lore.kernel.org/qemu-devel/Z07YASl2Pd3CPtjE@redhat.com/
> >
> > Note, my statement above was on the basis that HVF passes all our
> > functional tests, thus indicating a decent level of confidence
> > in the correctness of the HVF impl.
> >
> > If anyone knows any show stopper problems with HVF that would
> > justify blocking its promotion ahead of TCG.... say now.
> >
> 
> I don't know about showstoppers, but:
> 
> 1. As far as I'm aware there are/were problems with the virtual IOMMU
> devices in HVF. It's been a while (~half a year?) since I tried them, but I
> had problems getting guests booted with intel_iommu etc.

I think that vIOMMU is niche enough that we can merely consider it
a nice-to-fix bug, and not block promoting HVF.

> 2. I think there might also be a few remaining edge cases where the x86
> instruction emulation on fault/trap is incomplete. Most notably, MMIO using
> SSE/AVX/etc. instructions will, I think, fail. In practice this is a fairly
> obscure use case - I'm not aware of any guest OS that actually performs
> MMIO using these instructions. I have a patch kicking around that adds
> support for missing 64-bit variants of common scalar arithmetic
> instructions with memory operands. I can dig that up and post it - do we
> have a good way of adding tests for this kind of thing?

Not sure how best to test this, other than finding a guest OS that
exhibits this ? Others probably have better suggestions...

> 3. As far as I'm aware, there's no CI happening on HVF? Or has that
> changed? macOS is notoriously a pain in the rear in terms of CI thanks to
> its licensing, and the big cloud CI platforms tend to run it in a VM which
> in turn typically doesn't support nested HVF. I've been working on an
> on-prem solution to provisioning bare-metal Macs to run clean-slate OS
> images for CI. This has been a side project though and I haven't had the
> resources to focus on that project to see it through. It might be possible
> to do this in the cloud on Amazon's EC2 Mac Minis as well, but those aren't
> exactly cheap.

The only CI we have is running under Cirrus CI which uses VMs on
real mac aarch64 hardware, but I don't think we can test HVF there.

Mostly we rely on regular contributors periodically running tests
and reporting on problems. This is not ideal, but also not a blocker
for enabling HVF - it just means macOS isn't a full tier 1 platform
for us.

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [PATCH] system: Try hardware accelerators (KVM, HVF) before software one (TCG)

[Phil Dennis-Jordan]

[-- Attachment #1: Type: text/plain, Size: 4218 bytes --]

On Mon, 6 Jan 2025 at 19:43, Daniel P. Berrangé <berrange@redhat.com> wrote:

> On Sat, Jan 04, 2025 at 12:28:14PM +0100, Phil Dennis-Jordan wrote:
> > On Fri, 3 Jan 2025 at 16:16, Daniel P. Berrangé <berrange@redhat.com>
> wrote:
> >
> > > On Fri, Jan 03, 2025 at 04:05:58PM +0100, Philippe Mathieu-Daudé wrote:
> > > > As Daniel suggested [*]:
> > > >
> > > > > We should consider to rank HVF above TCG, on the basis
> > > > > that HW acceleration is faster and should provide a
> > > > > host<->guest security boundary that we don't claim for TCG
> > > >
> > > > [*] https://lore.kernel.org/qemu-devel/Z07YASl2Pd3CPtjE@redhat.com/
> > >
> > > Note, my statement above was on the basis that HVF passes all our
> > > functional tests, thus indicating a decent level of confidence
> > > in the correctness of the HVF impl.
> > >
> > > If anyone knows any show stopper problems with HVF that would
> > > justify blocking its promotion ahead of TCG.... say now.
> > >
> >
> > I don't know about showstoppers, but:
> >
> > 1. As far as I'm aware there are/were problems with the virtual IOMMU
> > devices in HVF. It's been a while (~half a year?) since I tried them,
> but I
> > had problems getting guests booted with intel_iommu etc.
>
> I think that vIOMMU is niche enough that we can merely consider it
> a nice-to-fix bug, and not block promoting HVF.
>

Yes, I assume those are mainly useful with very large numbers of vCPUs
and/or PCI device passthrough? Neither of those are relevant on macOS
hosts. (One other thing they're useful with is device driver
development/testing/debugging - real Macs come with IOMMUs so
PCI/Thunderbolt device drivers MUST translate addresses for DMA correctly -
but that's very niche indeed.)

> 2. I think there might also be a few remaining edge cases where the x86
> > instruction emulation on fault/trap is incomplete. Most notably, MMIO
> using
> > SSE/AVX/etc. instructions will, I think, fail. In practice this is a
> fairly
> > obscure use case - I'm not aware of any guest OS that actually performs
> > MMIO using these instructions. I have a patch kicking around that adds
> > support for missing 64-bit variants of common scalar arithmetic
> > instructions with memory operands. I can dig that up and post it - do we
> > have a good way of adding tests for this kind of thing?
>
> Not sure how best to test this, other than finding a guest OS that
> exhibits this ? Others probably have better suggestions...
>

I encountered it while trying the aforementioned vIOMMU; macOS guests
seemed to use 64-bit instructions for accessing some of the registers.

I've just checked, and there actually seem to be some instruction emulation
tests in the kvm-unit-tests project; those are designed for KVM, but I
expect it should be possible to tweak some of them so they're useful for
testing HVF. kvm-unit-tests don't seem to be part of QEMU's regular test
suite at the moment though, as far as I can tell?


> > 3. As far as I'm aware, there's no CI happening on HVF? Or has that
> > changed? macOS is notoriously a pain in the rear in terms of CI thanks to
> > its licensing, and the big cloud CI platforms tend to run it in a VM
> which
> > in turn typically doesn't support nested HVF. I've been working on an
> > on-prem solution to provisioning bare-metal Macs to run clean-slate OS
> > images for CI. This has been a side project though and I haven't had the
> > resources to focus on that project to see it through. It might be
> possible
> > to do this in the cloud on Amazon's EC2 Mac Minis as well, but those
> aren't
> > exactly cheap.
>
> The only CI we have is running under Cirrus CI which uses VMs on
> real mac aarch64 hardware, but I don't think we can test HVF there.
>

Thanks for confirming.


> Mostly we rely on regular contributors periodically running tests
> and reporting on problems. This is not ideal, but also not a blocker
> for enabling HVF - it just means macOS isn't a full tier 1 platform
> for us.
>

OK, that makes sense, thanks. At least this gives me a better idea of what
will be covered by CI and what won't when I'm reviewing patches.

Phil

[-- Attachment #2: Type: text/html, Size: 5644 bytes --]

Re: [PATCH] system: Try hardware accelerators (KVM, HVF) before software one (TCG)

[Richard Henderson]

On 1/3/25 07:05, Philippe Mathieu-Daudé wrote:
> As Daniel suggested [*]:
> 
>> We should consider to rank HVF above TCG, on the basis
>> that HW acceleration is faster and should provide a
>> host<->guest security boundary that we don't claim for TCG
> 
> [*] https://lore.kernel.org/qemu-devel/Z07YASl2Pd3CPtjE@redhat.com/
> Suggested-by: Daniel P. Berrangé <berrange@redhat.com>
> Signed-off-by: Philippe Mathieu-Daudé <philmd@linaro.org>
> ---
>   system/vl.c | 5 ++---
>   1 file changed, 2 insertions(+), 3 deletions(-)

Reviewed-by: Richard Henderson <richard.henderson@linaro.org>

Do nvmm and/or whpx deserve the same consideration?


r~