[PATCH 0/6] Adding the Android Emulator hypervisor driver accelerator

[PATCH 0/6] Adding the Android Emulator hypervisor driver accelerator

[Haitao Shan]

Hi, qemu maintainers and community members,

The following 6 patches implemented a new x86_64 CPU accelerator called
the Android Emulator hypervisor driver (AEHD).

The Android Emulator hypervisor driver is a hypervisor for Windows (7
or later), made by porting the KVM from the linux kernel 4.9-rc7. Its
initial purpose was to support the Android Emulator on the AMD
platforms as the old name "Android Emulator Hypervisor Driver for AMD
Processors" suggested. Despite the name, Intel processors have been
supported ever since its first release. Since Intel dropped HAXM support,
the android emulator is switching from HAXM to AEHD.
Refer to:
https://github.com/google/android-emulator-hypervisor-driver.

This patchset implements the user space support for the new
hypervisor. It was initially made by cloning and modifying the KVM
source codes. In order to support users who wanted to use the new
hypervisor to run generic OSes such as Windows and Ubuntu,  it was
periodically rebased to latest QEMU releases ever since QEMU v4.2.0.
Refer to: https://github.com/qemu-gvm/qemu-gvm.

Given that both the Windows driver and patched QEMU were out and
tested for a few years, we would like to see if we could submit our
work to the QEMU community. Users can have another choice of
hypervisor on Windows.

The patchset is tested on Windows 11 (64bit) with the AEHD 2.1 by
installing Ubuntu 22.04 and Windows 10 guests and playing with
savevm/loadvm. It is also tested on Linux (Ubuntu 22.04) and MacOS
(Intel and Apple Silicon), making sure it does not break others.

Any comments are welcome. Thanks!

Note:
The Android Emulator hypervisor driver will be maintained by
Google. However, there is no plan to rebase it to the latest KVM.

-- 
Haitao @Google

Re: [PATCH 0/6] Adding the Android Emulator hypervisor driver accelerator

[Daniel P. Berrangé]

On Thu, Mar 02, 2023 at 06:25:59PM -0800, Haitao Shan wrote:
> The Android Emulator hypervisor driver is a hypervisor for Windows (7
> or later), made by porting the KVM from the linux kernel 4.9-rc7. Its
> initial purpose was to support the Android Emulator on the AMD
> platforms as the old name "Android Emulator Hypervisor Driver for AMD
> Processors" suggested. Despite the name, Intel processors have been
> supported ever since its first release. Since Intel dropped HAXM support,
> the android emulator is switching from HAXM to AEHD.

When HAXM was proposed for deprecation & removal from QEMU, the suggestion
was that users should switch to Windows' native replacement WHPX, which
QEMU already has support for.

What is the rationale for wanting to introduce a 3rd party hypervisor
solution like AEHD, for the Android emulator, rather than just sticking
with the standard WHPX hypervisor available for Windows ? IIUC, the
Android emulator can already support WHPX according to these pages:

  https://developer.android.com/studio/run/emulator-acceleration
  https://learn.microsoft.com/en-us/dotnet/maui/android/emulator/hardware-acceleration?view=net-maui-7.0

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [PATCH 0/6] Adding the Android Emulator hypervisor driver accelerator

[Haitao Shan]

On Fri, Mar 3, 2023 at 2:34 AM Daniel P. Berrangé <berrange@redhat.com> wrote:
>
> On Thu, Mar 02, 2023 at 06:25:59PM -0800, Haitao Shan wrote:
> > The Android Emulator hypervisor driver is a hypervisor for Windows (7
> > or later), made by porting the KVM from the linux kernel 4.9-rc7. Its
> > initial purpose was to support the Android Emulator on the AMD
> > platforms as the old name "Android Emulator Hypervisor Driver for AMD
> > Processors" suggested. Despite the name, Intel processors have been
> > supported ever since its first release. Since Intel dropped HAXM support,
> > the android emulator is switching from HAXM to AEHD.
>
> When HAXM was proposed for deprecation & removal from QEMU, the suggestion
> was that users should switch to Windows' native replacement WHPX, which
> QEMU already has support for.
Sorry I was not aware of Intel's suggestion when HAXM was deprecated.
Is this a decision already, which shuts the door for any other 3rd party
hypervisors?

>
> What is the rationale for wanting to introduce a 3rd party hypervisor
> solution like AEHD, for the Android emulator, rather than just sticking
> with the standard WHPX hypervisor available for Windows ? IIUC, the
> Android emulator can already support WHPX according to these pages:
>
>   https://developer.android.com/studio/run/emulator-acceleration
>   https://learn.microsoft.com/en-us/dotnet/maui/android/emulator/hardware-acceleration?view=net-maui-7.0
You are right. WHPX is supported by the android emulator. The reason for
supporting AEHD:

1. HyperV is a type-1 hypervisor, which does not coexist with other hypervisors.
According to our data, there are both users who have to live with HyperV on or
with HyperV off. Those users depend on certain Windows features (or 3rd party
programs) that have to turn on or turn off HyperV. Offering AEHD allows us to
serve both types of users. This is the major benefit. I think this
argument is true
for upstream QEMU users as well.

2. Actually, AEHD started development before WHPX was added to the android
emulator. But porting/writing a new hypervisor just took lots of time.
In the middle,
Microsoft offered us WHPX. It could be the case that AEHD was never started if
WHPX had been offered one or two years earlier. However, we decided to continue.
First, see reason 1. Second, at least at that time, WHPX was noticeably slower
than both HAXM and AEHD. Third, Microsoft clearly stated there would not be
any backporting of WHPX to older versions of Windows. And those using old
versions of Windows need a solution in addition to HAXM.

3. Compared with HAXM, which looks like the default solution when HyperV
must be off, AEHD supports both Intel and AMD. And according to user feedback
and our own tests, AEHD can support Windows 10. This was the reason
why I maintained a patched QEMU ever since 4.2.0 specifically for them.

4. Although it is called Android Emulator hypervisor driver, it has nothing that
is android specific. And I've seen the upstream QEMU successfully refactored the
accelerator logic. This made adding a new hypervisor support less intrusive to
the main code base. I have a good will and intention to maintain what I
submitted if it could be approved by the community. I hope this does
not place (or
at least place very minimum) burden on the maintainers, should it be accepted.

Hope this can answer your questions. Thanks!

>
> With regards,
> Daniel
> --
> |: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
> |: https://libvirt.org         -o-            https://fstop138.berrange.com :|
> |: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
>


-- 
Haitao @Google

Re: [PATCH 0/6] Adding the Android Emulator hypervisor driver accelerator

[Daniel P. Berrangé]

On Fri, Mar 03, 2023 at 08:50:18AM -0800, Haitao Shan wrote:
> On Fri, Mar 3, 2023 at 2:34 AM Daniel P. Berrangé <berrange@redhat.com> wrote:
> >
> > On Thu, Mar 02, 2023 at 06:25:59PM -0800, Haitao Shan wrote:
> > > The Android Emulator hypervisor driver is a hypervisor for Windows (7
> > > or later), made by porting the KVM from the linux kernel 4.9-rc7. Its
> > > initial purpose was to support the Android Emulator on the AMD
> > > platforms as the old name "Android Emulator Hypervisor Driver for AMD
> > > Processors" suggested. Despite the name, Intel processors have been
> > > supported ever since its first release. Since Intel dropped HAXM support,
> > > the android emulator is switching from HAXM to AEHD.
> >
> > When HAXM was proposed for deprecation & removal from QEMU, the suggestion
> > was that users should switch to Windows' native replacement WHPX, which
> > QEMU already has support for.
>
> Sorry I was not aware of Intel's suggestion when HAXM was deprecated.
> Is this a decision already, which shuts the door for any other 3rd party
> hypervisors?

No, we're always open to new proposals. It merely means that it
might be harder to justify why the new hypervisor is a net benefit
for QEMU, when there is a competing solution supported by the OS
vendor.

> > What is the rationale for wanting to introduce a 3rd party hypervisor
> > solution like AEHD, for the Android emulator, rather than just sticking
> > with the standard WHPX hypervisor available for Windows ? IIUC, the
> > Android emulator can already support WHPX according to these pages:
> >
> >   https://developer.android.com/studio/run/emulator-acceleration
> >   https://learn.microsoft.com/en-us/dotnet/maui/android/emulator/hardware-acceleration?view=net-maui-7.0
> You are right. WHPX is supported by the android emulator. The reason for
> supporting AEHD:
> 
> 1. HyperV is a type-1 hypervisor, which does not coexist with other hypervisors.
> According to our data, there are both users who have to live with HyperV on or
> with HyperV off. Those users depend on certain Windows features (or 3rd party
> programs) that have to turn on or turn off HyperV. Offering AEHD allows us to
> serve both types of users. This is the major benefit. I think this
> argument is true
> for upstream QEMU users as well.

Yes, it is an akward situation if users cannot enable use of WHPX.
I don't have any better suggestion in that scenario.

> 2. Actually, AEHD started development before WHPX was added to the android
> emulator. But porting/writing a new hypervisor just took lots of time.
> In the middle,
> Microsoft offered us WHPX. It could be the case that AEHD was never started if
> WHPX had been offered one or two years earlier. However, we decided to continue.
> First, see reason 1. Second, at least at that time, WHPX was noticeably slower
> than both HAXM and AEHD. Third, Microsoft clearly stated there would not be
> any backporting of WHPX to older versions of Windows. And those using old
> versions of Windows need a solution in addition to HAXM.

FYI, from the QEMU POV, we only aim to support the two most recent major
releases of Windows.

At a technical level we've set the compilation baseline to Windows 8, but
per the support policy we only really support Windows 10 and 11 right now.

> 3. Compared with HAXM, which looks like the default solution when HyperV
> must be off, AEHD supports both Intel and AMD. And according to user feedback
> and our own tests, AEHD can support Windows 10. This was the reason
> why I maintained a patched QEMU ever since 4.2.0 specifically for them.

Yep, no disagreement that AEHD looks more useful for HAXM given the
cross vendor CPU support.

> 4. Although it is called Android Emulator hypervisor driver, it has nothing that
> is android specific. And I've seen the upstream QEMU successfully refactored the
> accelerator logic. This made adding a new hypervisor support less intrusive to
> the main code base. I have a good will and intention to maintain what I
> submitted if it could be approved by the community. I hope this does
> not place (or
> at least place very minimum) burden on the maintainers, should it be accepted.

Ok, thank you.

I don't have any specific requests / suggestion myself, but I see others
replied wondering whether there's any sense is sharing code with the
KVM driver in QEMU. Hopefully the QEMU KVM maintainers will comment with
their views on the matter.

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [PATCH 0/6] Adding the Android Emulator hypervisor driver accelerator

[Haitao Shan]

On Fri, Mar 3, 2023 at 9:12 AM Daniel P. Berrangé <berrange@redhat.com> wrote:
>
> On Fri, Mar 03, 2023 at 08:50:18AM -0800, Haitao Shan wrote:
> > On Fri, Mar 3, 2023 at 2:34 AM Daniel P. Berrangé <berrange@redhat.com> wrote:
> > >
> > > On Thu, Mar 02, 2023 at 06:25:59PM -0800, Haitao Shan wrote:
> > > > The Android Emulator hypervisor driver is a hypervisor for Windows (7
> > > > or later), made by porting the KVM from the linux kernel 4.9-rc7. Its
> > > > initial purpose was to support the Android Emulator on the AMD
> > > > platforms as the old name "Android Emulator Hypervisor Driver for AMD
> > > > Processors" suggested. Despite the name, Intel processors have been
> > > > supported ever since its first release. Since Intel dropped HAXM support,
> > > > the android emulator is switching from HAXM to AEHD.
> > >
> > > When HAXM was proposed for deprecation & removal from QEMU, the suggestion
> > > was that users should switch to Windows' native replacement WHPX, which
> > > QEMU already has support for.
> >
> > Sorry I was not aware of Intel's suggestion when HAXM was deprecated.
> > Is this a decision already, which shuts the door for any other 3rd party
> > hypervisors?
>
> No, we're always open to new proposals. It merely means that it
> might be harder to justify why the new hypervisor is a net benefit
> for QEMU, when there is a competing solution supported by the OS
> vendor.
Thanks for the clarification. It is great that the door is not shut completely.

>
> > > What is the rationale for wanting to introduce a 3rd party hypervisor
> > > solution like AEHD, for the Android emulator, rather than just sticking
> > > with the standard WHPX hypervisor available for Windows ? IIUC, the
> > > Android emulator can already support WHPX according to these pages:
> > >
> > >   https://developer.android.com/studio/run/emulator-acceleration
> > >   https://learn.microsoft.com/en-us/dotnet/maui/android/emulator/hardware-acceleration?view=net-maui-7.0
> > You are right. WHPX is supported by the android emulator. The reason for
> > supporting AEHD:
> >
> > 1. HyperV is a type-1 hypervisor, which does not coexist with other hypervisors.
> > According to our data, there are both users who have to live with HyperV on or
> > with HyperV off. Those users depend on certain Windows features (or 3rd party
> > programs) that have to turn on or turn off HyperV. Offering AEHD allows us to
> > serve both types of users. This is the major benefit. I think this
> > argument is true
> > for upstream QEMU users as well.
>
> Yes, it is an akward situation if users cannot enable use of WHPX.
> I don't have any better suggestion in that scenario.
>
> > 2. Actually, AEHD started development before WHPX was added to the android
> > emulator. But porting/writing a new hypervisor just took lots of time.
> > In the middle,
> > Microsoft offered us WHPX. It could be the case that AEHD was never started if
> > WHPX had been offered one or two years earlier. However, we decided to continue.
> > First, see reason 1. Second, at least at that time, WHPX was noticeably slower
> > than both HAXM and AEHD. Third, Microsoft clearly stated there would not be
> > any backporting of WHPX to older versions of Windows. And those using old
> > versions of Windows need a solution in addition to HAXM.
>
> FYI, from the QEMU POV, we only aim to support the two most recent major
> releases of Windows.
Just want to point out that only Windows 10 1809+ (or
perhaps 1803+) supports WHPX. And how to enable WHPX varies
among versions of Windows 11/10. For example, on Windows 11,
there is no need to explicitly enable it using the Windows Features
dialog, which is good according to my taste.

But I do agree that earlier Windows 10 versions should not be considered
as recent major releases any more nowadays.

>
> At a technical level we've set the compilation baseline to Windows 8, but
> per the support policy we only really support Windows 10 and 11 right now.
>
> > 3. Compared with HAXM, which looks like the default solution when HyperV
> > must be off, AEHD supports both Intel and AMD. And according to user feedback
> > and our own tests, AEHD can support Windows 10. This was the reason
> > why I maintained a patched QEMU ever since 4.2.0 specifically for them.
>
> Yep, no disagreement that AEHD looks more useful for HAXM given the
> cross vendor CPU support.
Oops. By saying supporting Windows 10, I really mean Windows 10 guests.

>
> > 4. Although it is called Android Emulator hypervisor driver, it has nothing that
> > is android specific. And I've seen the upstream QEMU successfully refactored the
> > accelerator logic. This made adding a new hypervisor support less intrusive to
> > the main code base. I have a good will and intention to maintain what I
> > submitted if it could be approved by the community. I hope this does
> > not place (or
> > at least place very minimum) burden on the maintainers, should it be accepted.
>
> Ok, thank you.
>
> I don't have any specific requests / suggestion myself, but I see others
> replied wondering whether there's any sense is sharing code with the
> KVM driver in QEMU. Hopefully the QEMU KVM maintainers will comment with
> their views on the matter.
>
> With regards,
> Daniel
> --
> |: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
> |: https://libvirt.org         -o-            https://fstop138.berrange.com :|
> |: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
>


-- 
Haitao @Google

Re: [PATCH 0/6] Adding the Android Emulator hypervisor driver accelerator

[Paolo Bonzini]

On 3/3/23 18:39, Haitao Shan wrote:
>> No, we're always open to new proposals. It merely means that it
>> might be harder to justify why the new hypervisor is a net benefit
>> for QEMU, when there is a competing solution supported by the OS
>> vendor.
>
> Thanks for the clarification. It is great that the door is not shut completely.

Hi,

sorry for not answering before.

I think in general QEMU should be open to merging work from the Android 
Emulator.  If AEHD is useful to the Android emulator, I would consider 
it interesting for QEMU as well.

However, I would rather have it as an extension to KVM if possible 
rather than a completely new emulator.  One possibility is to introduce 
a new file that encapsulates all KVM ioctls, with a struct that 
encapsulates the Unix file descriptor/Windows HANDLE.  For example

int kvm_ioctl_get_supported_cpuid(KVMState *s, struct kvm_cpuid *cpuid,
                                   int max)
{
     cpuid->nent = max;
#ifdef CONFIG_POSIX
     return ioctl(s, KVM_GET_SUPPORTED_VCPUID, cpuid);
#else
     size_t size = sizeof(*cpuid) + max * sizeof(*cpuid->entries);
     return aehd_ioctl(s, AEHD_GET_SUPPORTED_CPUID, cpuid, size, cpuid, 
size);
#endif
}

int kvm_ioctl_create_vcpu(KVMState *s, int vcpu_id, CPUState *out)
{
#ifdef CONFIG_POSIX
     out.kvm_fd = kvm_vm_ioctl(KVM_CREATE_VCPU, vcpu_id);
     return out.kvm_fd;
#else
     return aehd_vm_ioctl(s, AEHD_CREATE_VCPU, &vcpu_id, sizeof(vcpu_id),
                          &out.kvm_fd, sizeof(out.kvm_fd));
#endif
}

etc.

These are just general examples, the actual level of abstraction is up 
to you.

Paolo

Re: [PATCH 0/6] Adding the Android Emulator hypervisor driver accelerator

[Haitao Shan]

On Tue, Apr 4, 2023 at 4:55 AM Paolo Bonzini <pbonzini@redhat.com> wrote:
>
> On 3/3/23 18:39, Haitao Shan wrote:
> >> No, we're always open to new proposals. It merely means that it
> >> might be harder to justify why the new hypervisor is a net benefit
> >> for QEMU, when there is a competing solution supported by the OS
> >> vendor.
> >
> > Thanks for the clarification. It is great that the door is not shut completely.
>
> Hi,
>
> sorry for not answering before.
Thanks for your reply. I was taking a long vacation and did not see
your reply earlier.

>
> I think in general QEMU should be open to merging work from the Android
> Emulator.  If AEHD is useful to the Android emulator, I would consider
> it interesting for QEMU as well.
Thanks for being open to us. For patchset V1, the most important
feedback we can have is that our work can be useful to the community
(not just the android emulator).

>
> However, I would rather have it as an extension to KVM if possible
> rather than a completely new emulator.  One possibility is to introduce
> a new file that encapsulates all KVM ioctls, with a struct that
> encapsulates the Unix file descriptor/Windows HANDLE.  For example
>
> int kvm_ioctl_get_supported_cpuid(KVMState *s, struct kvm_cpuid *cpuid,
>                                    int max)
> {
>      cpuid->nent = max;
> #ifdef CONFIG_POSIX
>      return ioctl(s, KVM_GET_SUPPORTED_VCPUID, cpuid);
> #else
>      size_t size = sizeof(*cpuid) + max * sizeof(*cpuid->entries);
>      return aehd_ioctl(s, AEHD_GET_SUPPORTED_CPUID, cpuid, size, cpuid,
> size);
> #endif
> }
>
> int kvm_ioctl_create_vcpu(KVMState *s, int vcpu_id, CPUState *out)
> {
> #ifdef CONFIG_POSIX
>      out.kvm_fd = kvm_vm_ioctl(KVM_CREATE_VCPU, vcpu_id);
>      return out.kvm_fd;
> #else
>      return aehd_vm_ioctl(s, AEHD_CREATE_VCPU, &vcpu_id, sizeof(vcpu_id),
>                           &out.kvm_fd, sizeof(out.kvm_fd));
> #endif
> }
>
> etc.
>
> These are just general examples, the actual level of abstraction is up
> to you.
I will work on the new patchset. And most likely it will take some time.

>
> Paolo
>


-- 
Haitao @Google

Re: [PATCH 0/6] Adding the Android Emulator hypervisor driver accelerator

[startergo via]

[-- Attachment #1: Type: text/plain, Size: 271 bytes --]

I have tried the qemu_staging_virgl_aehd version on Win11 with macOS guest,
It looks like the AVX2 cpu feature is not passed correctly as it causes KP. See here for more details:

[MateuszKrawczuk/qemu at master_wip_aehd](https://github.com/MateuszKrawczuk/qemu/issues/2)

[-- Attachment #2: Type: text/html, Size: 960 bytes --]