[RFC/PATCH v0 00/12] Gunyah hypervisor support

[RFC/PATCH v0 00/12] Gunyah hypervisor support

[Srivatsa Vaddagiri]

Gunyah is an open-source Type-1 hypervisor, that is currently supported on ARM64
architecture. Source code for it can be obtained from:

https://github.com/quic/gunyah-hypervisor.

This patch series adds support for Gunyah hypervisor via a new
accelerator option, 'gunyah'. This patch series is based on the Linux kernel's
Gunyah driver, which is being actively developed and not yet merged upstream
[1].

This patch series is thus *NOT YET READY* for merge. Early version of this patch
is being published to solicit comments from Qemu community.

This patch has been tested with the open-source version of Gunyah hypervisor.
Instructions to build hypervisor and test this patch are provided in this
patch series.

Limitations:

1) SMP is not yet supported. This restriction will be removed in the next version
of this patch series.

2) virtio-pci is not yet supported. hw/virtio/virtio-pci.c seems to support only
KVM and would need changes to support other hypervisors. If there is any ongoing
work in this regard, would like to build upon it and add support for Gunyah
hypervisor.


Ref:

1. https://lore.kernel.org/lkml/20230613172054.3959700-1-quic_eberman@quicinc.com/

Base commit on which this series was tested:

d77b7b28a8 target/arm: Fix 64-bit SSRA


Srivatsa Vaddagiri (12):
  hw/arm/virt: Avoid NULL pointer de-reference
  update-linux-headers: Include gunyah.h
  gunyah: Basic support
  gunyah: Add VM properties
  gunyah: Support memory assignment
  gunyah: Add IRQFD and IOEVENTFD functions
  gunyah: Add gicv3 interrupt controller
  gunyah: Specific device-tree location
  gunyah: Customize device-tree
  gunyah: CPU execution loop
  gunyah: Workarounds (NOT FOR MERGE)
  gunyah: Documentation

 MAINTAINERS                     |  10 +
 accel/Kconfig                   |   3 +
 accel/gunyah/gunyah-accel-ops.c | 168 +++++++++
 accel/gunyah/gunyah-all.c       | 629 ++++++++++++++++++++++++++++++++
 accel/gunyah/meson.build        |   7 +
 accel/meson.build               |   1 +
 accel/stubs/gunyah-stub.c       |  13 +
 accel/stubs/meson.build         |   1 +
 docs/about/build-platforms.rst  |   2 +-
 docs/system/arm/gunyah.rst      | 214 +++++++++++
 hw/arm/boot.c                   |   3 +-
 hw/arm/virt.c                   |  27 +-
 hw/intc/arm_gicv3_common.c      |   3 +
 hw/intc/arm_gicv3_gunyah.c      | 106 ++++++
 hw/intc/arm_gicv3_its_common.c  |   4 +
 hw/intc/meson.build             |   1 +
 include/hw/core/cpu.h           |   6 +
 include/sysemu/gunyah.h         |  58 +++
 include/sysemu/gunyah_int.h     |  61 ++++
 meson.build                     |   9 +
 meson_options.txt               |   2 +
 scripts/meson-buildoptions.sh   |   3 +
 scripts/update-linux-headers.sh |   3 +-
 target/arm/cpu64.c              |   5 +-
 target/arm/gunyah.c             | 144 ++++++++
 target/arm/meson.build          |   4 +
 26 files changed, 1480 insertions(+), 7 deletions(-)
 create mode 100644 accel/gunyah/gunyah-accel-ops.c
 create mode 100644 accel/gunyah/gunyah-all.c
 create mode 100644 accel/gunyah/meson.build
 create mode 100644 accel/stubs/gunyah-stub.c
 create mode 100644 docs/system/arm/gunyah.rst
 create mode 100644 hw/intc/arm_gicv3_gunyah.c
 create mode 100644 include/sysemu/gunyah.h
 create mode 100644 include/sysemu/gunyah_int.h
 create mode 100644 target/arm/gunyah.c

-- 
2.25.1

[RFC/PATCH v0 01/12] hw/arm/virt: Avoid NULL pointer de-reference

[Srivatsa Vaddagiri]

Avoid dereferencing a NULL pointer that its_class_name() could return.

Signed-off-by: Srivatsa Vaddagiri <quic_svaddagi@quicinc.com>
---
 hw/arm/virt.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/hw/arm/virt.c b/hw/arm/virt.c
index a13c658bbf..b55d5c7282 100644
--- a/hw/arm/virt.c
+++ b/hw/arm/virt.c
@@ -661,7 +661,7 @@ static void create_its(VirtMachineState *vms)
     const char *itsclass = its_class_name();
     DeviceState *dev;
 
-    if (!strcmp(itsclass, "arm-gicv3-its")) {
+    if (itsclass && !strcmp(itsclass, "arm-gicv3-its")) {
         if (!vms->tcg_its) {
             itsclass = NULL;
         }
-- 
2.25.1

Re: [RFC/PATCH v0 01/12] hw/arm/virt: Avoid NULL pointer de-reference

[Philippe Mathieu-Daudé]

Hi Srivatsa,

(+Markus/Peter for QOM fu)

On 11/10/23 18:52, Srivatsa Vaddagiri wrote:
> Avoid dereferencing a NULL pointer that its_class_name() could return.

While your patch is correct, there is some code smell
around its_class_name(). IMHO a foo_class_name() handler
should never return NULL.

I'm trying to rework apic_get_class() similarly, see:
https://lore.kernel.org/qemu-devel/20231003082728.83496-1-philmd@linaro.org/

> Signed-off-by: Srivatsa Vaddagiri <quic_svaddagi@quicinc.com>
> ---
>   hw/arm/virt.c | 2 +-
>   1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/hw/arm/virt.c b/hw/arm/virt.c
> index a13c658bbf..b55d5c7282 100644
> --- a/hw/arm/virt.c
> +++ b/hw/arm/virt.c
> @@ -661,7 +661,7 @@ static void create_its(VirtMachineState *vms)
>       const char *itsclass = its_class_name();
>       DeviceState *dev;
>   
> -    if (!strcmp(itsclass, "arm-gicv3-its")) {
> +    if (itsclass && !strcmp(itsclass, "arm-gicv3-its")) {
>           if (!vms->tcg_its) {
>               itsclass = NULL;
>           }

Re: [RFC/PATCH v0 01/12] hw/arm/virt: Avoid NULL pointer de-reference

[Markus Armbruster]

Philippe Mathieu-Daudé <philmd@linaro.org> writes:

> Hi Srivatsa,
>
> (+Markus/Peter for QOM fu)

QOM fu needs Paolo; cc'ed.

I'm not sure how much this is about QOM, though.  Perhaps it's more
about good taste.

> On 11/10/23 18:52, Srivatsa Vaddagiri wrote:
>> Avoid dereferencing a NULL pointer that its_class_name() could return.
>
> While your patch is correct, there is some code smell
> around its_class_name(). IMHO a foo_class_name() handler
> should never return NULL.
>
> I'm trying to rework apic_get_class() similarly, see:
> https://lore.kernel.org/qemu-devel/20231003082728.83496-1-philmd@linaro.org/

In both cases, we have a function to find the device model to use with
current QEMU configuration and system state.  The fact that one of them
returns a class name and the other a class is detail.

Observe: this usable device model exists for any QEMU configuration.  It
may not be usable in certain system states, though.

Since the function deals with both, it can fail.

We can separate the two concerns: first map configuration to device
model (can't fail), then check the system state (can fail).

Feels like a matter of taste to me.  Does it result in simpler function
contracts and more readable code?  Can we do it the same way everywhere?

Re: [RFC/PATCH v0 01/12] hw/arm/virt: Avoid NULL pointer de-reference

[Srivatsa Vaddagiri]

* Philippe Mathieu-Daud? <philmd@linaro.org> [2023-10-12 06:30:24]:

> Hi Srivatsa,
> 
> (+Markus/Peter for QOM fu)
> 
> On 11/10/23 18:52, Srivatsa Vaddagiri wrote:
> > Avoid dereferencing a NULL pointer that its_class_name() could return.
> 
> While your patch is correct, there is some code smell
> around its_class_name(). IMHO a foo_class_name() handler
> should never return NULL.

I saw its_class_name() returning NULL for KVM in some case and took the
same route for Gunyah (as ITS is not supported atm):

const char *its_class_name(void)
{
    if (kvm_irqchip_in_kernel()) {
        /* KVM implementation requires this capability */
        return kvm_direct_msi_enabled() ? "arm-its-kvm" : NULL;
    } else if (gunyah_enabled()) { 
        /* ITS is not yet supported */
        return NULL; 
}

I guess I could have its_class_name() return "arm-gicv3-its" (for Gunyah case)
and set vms->tcg_its to false, which will avoid the NULL-pointer deref I was
hitting.

I will drop this patch in next version in that case.

- vatsa

[RFC/PATCH v0 02/12] update-linux-headers: Include gunyah.h

[Srivatsa Vaddagiri]

gunyah.h containts UAPI definitions exported by Gunyah Linux kernel
driver. This file will be referenced by Gunyah accelerator driver in
Qemu.

Note: Gunyah Linux kernel driver is not yet merged in Linux kernel.
v14 of the patch series has been posted on mailing lists.

https://lore.kernel.org/lkml/20230613172054.3959700-1-quic_eberman@quicinc.com/

One of the subsequent versions of the patch is expected to be merged
upstream soon, after this this change to 'update-linux-headers' can be
run against the main Linux kernel source repository from kernel.org.

In the meantime, a version of Gunyah kernel driver is available in
Android kernel repository, against which this change can be run.

https://android.googlesource.com/kernel/common
Branch: android14-6.1

Signed-off-by: Srivatsa Vaddagiri <quic_svaddagi@quicinc.com>
---
 scripts/update-linux-headers.sh | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/scripts/update-linux-headers.sh b/scripts/update-linux-headers.sh
index 35a64bb501..3f3608bcbe 100755
--- a/scripts/update-linux-headers.sh
+++ b/scripts/update-linux-headers.sh
@@ -161,7 +161,8 @@ done
 rm -rf "$output/linux-headers/linux"
 mkdir -p "$output/linux-headers/linux"
 for header in const.h stddef.h kvm.h vfio.h vfio_ccw.h vfio_zdev.h vhost.h \
-              psci.h psp-sev.h userfaultfd.h memfd.h mman.h nvme_ioctl.h vduse.h; do
+              psci.h psp-sev.h userfaultfd.h memfd.h mman.h nvme_ioctl.h \
+              vduse.h gunyah.h; do
     cp "$tmpdir/include/linux/$header" "$output/linux-headers/linux"
 done
 
-- 
2.25.1

Re: [RFC/PATCH v0 02/12] update-linux-headers: Include gunyah.h

[Srivatsa Vaddagiri]

* Srivatsa Vaddagiri <quic_svaddagi@quicinc.com> [2023-10-11 16:52:24]:

> One of the subsequent versions of the patch is expected to be merged
> upstream soon, after this this change to 'update-linux-headers' can be

Sorry about the typo, will be fixed in next version!

s/this this/which, this

> run against the main Linux kernel source repository from kernel.org.

Re: [RFC/PATCH v0 02/12] update-linux-headers: Include gunyah.h

[Alex Bennée]

Srivatsa Vaddagiri <quic_svaddagi@quicinc.com> writes:

> gunyah.h containts UAPI definitions exported by Gunyah Linux kernel
> driver. This file will be referenced by Gunyah accelerator driver in
> Qemu.
>
> Note: Gunyah Linux kernel driver is not yet merged in Linux kernel.
> v14 of the patch series has been posted on mailing lists.
>
> https://lore.kernel.org/lkml/20230613172054.3959700-1-quic_eberman@quicinc.com/
>
> One of the subsequent versions of the patch is expected to be merged
> upstream soon, after this this change to 'update-linux-headers' can be
> run against the main Linux kernel source repository from kernel.org.
>
> In the meantime, a version of Gunyah kernel driver is available in
> Android kernel repository, against which this change can be run.
>
> https://android.googlesource.com/kernel/common
> Branch: android14-6.1
>
> Signed-off-by: Srivatsa Vaddagiri <quic_svaddagi@quicinc.com>

with fix:

Reviewed-by: Alex Bennée <alex.bennee@linaro.org>

although it will need another fixup on rebase due to other headers being
added. Maybe now is the time to sort the header list to lessen the
chance of clashes in future?


> ---
>  scripts/update-linux-headers.sh | 3 ++-
>  1 file changed, 2 insertions(+), 1 deletion(-)
>
> diff --git a/scripts/update-linux-headers.sh b/scripts/update-linux-headers.sh
> index 35a64bb501..3f3608bcbe 100755
> --- a/scripts/update-linux-headers.sh
> +++ b/scripts/update-linux-headers.sh
> @@ -161,7 +161,8 @@ done
>  rm -rf "$output/linux-headers/linux"
>  mkdir -p "$output/linux-headers/linux"
>  for header in const.h stddef.h kvm.h vfio.h vfio_ccw.h vfio_zdev.h vhost.h \
> -              psci.h psp-sev.h userfaultfd.h memfd.h mman.h nvme_ioctl.h vduse.h; do
> +              psci.h psp-sev.h userfaultfd.h memfd.h mman.h nvme_ioctl.h \
> +              vduse.h gunyah.h; do
>      cp "$tmpdir/include/linux/$header" "$output/linux-headers/linux"
>  done

-- 
Alex Bennée
Virtualisation Tech Lead @ Linaro

Re: [RFC/PATCH v0 02/12] update-linux-headers: Include gunyah.h

[Srivatsa Vaddagiri]

* Alex Benn?e <alex.bennee@linaro.org> [2023-11-29 16:44:06]:

> Srivatsa Vaddagiri <quic_svaddagi@quicinc.com> writes:
> 
> > gunyah.h containts UAPI definitions exported by Gunyah Linux kernel
> > driver. This file will be referenced by Gunyah accelerator driver in
> > Qemu.
> >
> > Note: Gunyah Linux kernel driver is not yet merged in Linux kernel.
> > v14 of the patch series has been posted on mailing lists.
> >
> > https://lore.kernel.org/lkml/20230613172054.3959700-1-quic_eberman@quicinc.com/
> >
> > One of the subsequent versions of the patch is expected to be merged
> > upstream soon, after this this change to 'update-linux-headers' can be
> > run against the main Linux kernel source repository from kernel.org.
> >
> > In the meantime, a version of Gunyah kernel driver is available in
> > Android kernel repository, against which this change can be run.
> >
> > https://android.googlesource.com/kernel/common
> > Branch: android14-6.1
> >
> > Signed-off-by: Srivatsa Vaddagiri <quic_svaddagi@quicinc.com>
> 
> with fix:
> 
> Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
> 
> although it will need another fixup on rebase due to other headers being
> added. Maybe now is the time to sort the header list to lessen the
> chance of clashes in future?

Sure will suggest sorted header list in my next revision!

- vatsa

 
> > ---
> >  scripts/update-linux-headers.sh | 3 ++-
> >  1 file changed, 2 insertions(+), 1 deletion(-)
> >
> > diff --git a/scripts/update-linux-headers.sh b/scripts/update-linux-headers.sh
> > index 35a64bb501..3f3608bcbe 100755
> > --- a/scripts/update-linux-headers.sh
> > +++ b/scripts/update-linux-headers.sh
> > @@ -161,7 +161,8 @@ done
> >  rm -rf "$output/linux-headers/linux"
> >  mkdir -p "$output/linux-headers/linux"
> >  for header in const.h stddef.h kvm.h vfio.h vfio_ccw.h vfio_zdev.h vhost.h \
> > -              psci.h psp-sev.h userfaultfd.h memfd.h mman.h nvme_ioctl.h vduse.h; do
> > +              psci.h psp-sev.h userfaultfd.h memfd.h mman.h nvme_ioctl.h \
> > +              vduse.h gunyah.h; do
> >      cp "$tmpdir/include/linux/$header" "$output/linux-headers/linux"
> >  done
> 
> -- 
> Alex Bennée
> Virtualisation Tech Lead @ Linaro

[RFC/PATCH v0 03/12] gunyah: Basic support

[Srivatsa Vaddagiri]

Add a new accelerator, gunyah, with basic functionality of creating a
VM. Subsequent patches will add support for other functions required to
run a VM.

Signed-off-by: Srivatsa Vaddagiri <quic_svaddagi@quicinc.com>
---
 MAINTAINERS                     |   7 +++
 accel/Kconfig                   |   3 +
 accel/gunyah/gunyah-accel-ops.c | 102 ++++++++++++++++++++++++++++++++
 accel/gunyah/gunyah-all.c       |  70 ++++++++++++++++++++++
 accel/gunyah/meson.build        |   7 +++
 accel/meson.build               |   1 +
 accel/stubs/gunyah-stub.c       |  13 ++++
 accel/stubs/meson.build         |   1 +
 docs/about/build-platforms.rst  |   2 +-
 hw/arm/virt.c                   |   3 +
 include/sysemu/gunyah.h         |  43 ++++++++++++++
 include/sysemu/gunyah_int.h     |  27 +++++++++
 meson.build                     |   9 +++
 meson_options.txt               |   2 +
 scripts/meson-buildoptions.sh   |   3 +
 target/arm/cpu64.c              |   5 +-
 16 files changed, 295 insertions(+), 3 deletions(-)
 create mode 100644 accel/gunyah/gunyah-accel-ops.c
 create mode 100644 accel/gunyah/gunyah-all.c
 create mode 100644 accel/gunyah/meson.build
 create mode 100644 accel/stubs/gunyah-stub.c
 create mode 100644 include/sysemu/gunyah.h
 create mode 100644 include/sysemu/gunyah_int.h

diff --git a/MAINTAINERS b/MAINTAINERS
index 6111b6b4d9..0e76dda91d 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -510,6 +510,13 @@ F: accel/hvf/
 F: include/sysemu/hvf.h
 F: include/sysemu/hvf_int.h
 
+GUNYAH
+M: Srivatsa Vaddagiri <quic_svaddagi@quicinc.com>
+S: Maintained
+F: accel/gunyah
+F: include/sysemu/gunyah.h
+F: include/sysemu/gunyah_int.h
+
 WHPX CPUs
 M: Sunil Muthuswamy <sunilmut@microsoft.com>
 S: Supported
diff --git a/accel/Kconfig b/accel/Kconfig
index 8bdedb7d15..bab1edebf4 100644
--- a/accel/Kconfig
+++ b/accel/Kconfig
@@ -19,3 +19,6 @@ config KVM
 config XEN
     bool
     select FSDEV_9P if VIRTFS
+
+config GUNYAH
+    bool
diff --git a/accel/gunyah/gunyah-accel-ops.c b/accel/gunyah/gunyah-accel-ops.c
new file mode 100644
index 0000000000..1c1c20abf1
--- /dev/null
+++ b/accel/gunyah/gunyah-accel-ops.c
@@ -0,0 +1,102 @@
+/*
+ * QEMU Gunyah hypervisor support
+ *
+ * (based on KVM accelerator code structure)
+ *
+ * Copyright 2008 IBM Corporation
+ *           2008 Red Hat, Inc.
+ *
+ * Copyright(c) 2023 Qualcomm Innovation Center, Inc. All Rights Reserved.
+ *
+ * SPDX-License-Identifier: GPL-2.0-or-later
+ */
+
+#include "qemu/osdep.h"
+#include "qemu/thread.h"
+#include "hw/boards.h"
+#include "hw/core/cpu.h"
+#include "sysemu/accel-ops.h"
+#include "sysemu/cpus.h"
+#include "sysemu/gunyah.h"
+#include "sysemu/gunyah_int.h"
+#include "qapi/visitor.h"
+#include "qapi/error.h"
+
+bool gunyah_allowed;
+
+static int gunyah_init(MachineState *ms)
+{
+    return gunyah_create_vm();
+}
+
+static void gunyah_accel_instance_init(Object *obj)
+{
+    GUNYAHState *s = GUNYAH_STATE(obj);
+
+    s->fd = -1;
+    s->vmfd = -1;
+}
+
+static void gunyah_accel_class_init(ObjectClass *oc, void *data)
+{
+    AccelClass *ac = ACCEL_CLASS(oc);
+
+    ac->name = "GUNYAH";
+    ac->init_machine = gunyah_init;
+    ac->allowed = &gunyah_allowed;
+}
+
+static const TypeInfo gunyah_accel_type = {
+    .name = TYPE_GUNYAH_ACCEL,
+    .parent = TYPE_ACCEL,
+    .instance_init = gunyah_accel_instance_init,
+    .class_init = gunyah_accel_class_init,
+    .instance_size = sizeof(GUNYAHState),
+};
+
+static void gunyah_type_init(void)
+{
+    type_register_static(&gunyah_accel_type);
+}
+type_init(gunyah_type_init);
+
+static void gunyah_start_vcpu_thread(CPUState *cpu)
+{
+    char thread_name[VCPU_THREAD_NAME_SIZE];
+
+    cpu->thread = g_malloc0(sizeof(QemuThread));
+    cpu->halt_cond = g_malloc0(sizeof(QemuCond));
+    qemu_cond_init(cpu->halt_cond);
+
+    snprintf(thread_name, VCPU_THREAD_NAME_SIZE, "CPU %d/Gunyah",
+             cpu->cpu_index);
+    qemu_thread_create(cpu->thread, thread_name, gunyah_cpu_thread_fn,
+                       cpu, QEMU_THREAD_JOINABLE);
+}
+
+static void gunyah_kick_vcpu_thread(CPUState *cpu)
+{
+    cpus_kick_thread(cpu);
+}
+
+static void gunyah_accel_ops_class_init(ObjectClass *oc, void *data)
+{
+    AccelOpsClass *ops = ACCEL_OPS_CLASS(oc);
+
+    ops->create_vcpu_thread = gunyah_start_vcpu_thread;
+    ops->kick_vcpu_thread = gunyah_kick_vcpu_thread;
+};
+
+static const TypeInfo gunyah_accel_ops_type = {
+    .name = ACCEL_OPS_NAME("gunyah"),
+    .parent = TYPE_ACCEL_OPS,
+    .class_init = gunyah_accel_ops_class_init,
+    .abstract = true,
+};
+
+static void gunyah_accel_ops_register_types(void)
+{
+    type_register_static(&gunyah_accel_ops_type);
+}
+
+type_init(gunyah_accel_ops_register_types);
diff --git a/accel/gunyah/gunyah-all.c b/accel/gunyah/gunyah-all.c
new file mode 100644
index 0000000000..370add75f9
--- /dev/null
+++ b/accel/gunyah/gunyah-all.c
@@ -0,0 +1,70 @@
+/*
+ * QEMU Gunyah hypervisor support
+ *
+ * (based on KVM accelerator code structure)
+ *
+ * Copyright 2008 IBM Corporation
+ *           2008 Red Hat, Inc.
+ *
+ * Copyright(c) 2023 Qualcomm Innovation Center, Inc. All Rights Reserved.
+ *
+ * SPDX-License-Identifier: GPL-2.0-or-later
+ */
+
+#include <sys/ioctl.h>
+#include "qemu/osdep.h"
+#include "qemu/typedefs.h"
+#include "hw/core/cpu.h"
+#include "sysemu/cpus.h"
+#include "sysemu/gunyah.h"
+#include "sysemu/gunyah_int.h"
+#include "linux-headers/linux/gunyah.h"
+#include "qemu/error-report.h"
+
+static int gunyah_ioctl(int type, ...)
+{
+    void *arg;
+    va_list ap;
+    GUNYAHState *s = GUNYAH_STATE(current_accel());
+
+    assert(s->fd);
+
+    va_start(ap, type);
+    arg = va_arg(ap, void *);
+    va_end(ap);
+
+    return ioctl(s->fd, type, arg);
+}
+
+int gunyah_create_vm(void)
+{
+    GUNYAHState *s;
+
+    s = GUNYAH_STATE(current_accel());
+
+    s->fd = qemu_open_old("/dev/gunyah", O_RDWR);
+    if (s->fd == -1) {
+        error_report("Could not access Gunyah kernel module at /dev/gunyah: %s",
+                                strerror(errno));
+        exit(1);
+    }
+
+    s->vmfd = gunyah_ioctl(GH_CREATE_VM, 0);
+    if (s->vmfd < 0) {
+        error_report("Could not create VM: %s", strerror(errno));
+        exit(1);
+    }
+
+    return 0;
+}
+
+void *gunyah_cpu_thread_fn(void *arg)
+{
+    CPUState *cpu = arg;
+
+    do {
+        /* Do nothing */
+    } while (!cpu->unplug || cpu_can_run(cpu));
+
+    return NULL;
+}
diff --git a/accel/gunyah/meson.build b/accel/gunyah/meson.build
new file mode 100644
index 0000000000..226eea0f2c
--- /dev/null
+++ b/accel/gunyah/meson.build
@@ -0,0 +1,7 @@
+gunyah_ss = ss.source_set()
+gunyah_ss.add(files(
+  'gunyah-all.c',
+  'gunyah-accel-ops.c',
+))
+
+specific_ss.add_all(when: 'CONFIG_GUNYAH', if_true: gunyah_ss)
diff --git a/accel/meson.build b/accel/meson.build
index 638a9a03ba..e2b87f74b0 100644
--- a/accel/meson.build
+++ b/accel/meson.build
@@ -9,6 +9,7 @@ if have_system
   subdir('kvm')
   subdir('xen')
   subdir('stubs')
+  subdir('gunyah')
 endif
 
 # qtest
diff --git a/accel/stubs/gunyah-stub.c b/accel/stubs/gunyah-stub.c
new file mode 100644
index 0000000000..d515469c01
--- /dev/null
+++ b/accel/stubs/gunyah-stub.c
@@ -0,0 +1,13 @@
+/*
+ * QEMU Gunyah stub
+ *
+ * Copyright(c) 2023 Qualcomm Innovation Center, Inc. All Rights Reserved.
+ *
+ * SPDX-License-Identifier: GPL-2.0-or-later
+ *
+ */
+
+#include "sysemu/gunyah.h"
+#include "sysemu/gunyah_int.h"
+
+bool gunyah_allowed;
diff --git a/accel/stubs/meson.build b/accel/stubs/meson.build
index f7a9486e06..4f594e09de 100644
--- a/accel/stubs/meson.build
+++ b/accel/stubs/meson.build
@@ -3,5 +3,6 @@ sysemu_stubs_ss.add(when: 'CONFIG_HAX', if_false: files('hax-stub.c'))
 sysemu_stubs_ss.add(when: 'CONFIG_XEN', if_false: files('xen-stub.c'))
 sysemu_stubs_ss.add(when: 'CONFIG_KVM', if_false: files('kvm-stub.c'))
 sysemu_stubs_ss.add(when: 'CONFIG_TCG', if_false: files('tcg-stub.c'))
+sysemu_stubs_ss.add(when: 'CONFIG_GUNYAH', if_false: files('gunyah-stub.c'))
 
 specific_ss.add_all(when: ['CONFIG_SYSTEM_ONLY'], if_true: sysemu_stubs_ss)
diff --git a/docs/about/build-platforms.rst b/docs/about/build-platforms.rst
index 0e2cb9e770..5a3e241f1a 100644
--- a/docs/about/build-platforms.rst
+++ b/docs/about/build-platforms.rst
@@ -40,7 +40,7 @@ Those hosts are officially supported, with various accelerators:
    * - CPU Architecture
      - Accelerators
    * - Arm
-     - kvm (64 bit only), tcg, xen
+     - kvm (64 bit only), tcg, xen, gunyah (64 bit only)
    * - MIPS (little endian only)
      - kvm, tcg
    * - PPC
diff --git a/hw/arm/virt.c b/hw/arm/virt.c
index b55d5c7282..4222183eb5 100644
--- a/hw/arm/virt.c
+++ b/hw/arm/virt.c
@@ -80,6 +80,7 @@
 #include "hw/virtio/virtio-iommu.h"
 #include "hw/char/pl011.h"
 #include "qemu/guest-random.h"
+#include "sysemu/gunyah.h"
 
 #define DEFINE_VIRT_MACHINE_LATEST(major, minor, latest) \
     static void virt_##major##_##minor##_class_init(ObjectClass *oc, \
@@ -1623,6 +1624,8 @@ static void virt_build_smbios(VirtMachineState *vms)
 
     if (kvm_enabled()) {
         product = "KVM Virtual Machine";
+    } else if (gunyah_enabled()) {
+        product = "Gunyah Virtual Machine";
     }
 
     smbios_set_defaults("QEMU", product,
diff --git a/include/sysemu/gunyah.h b/include/sysemu/gunyah.h
new file mode 100644
index 0000000000..aded49cdf6
--- /dev/null
+++ b/include/sysemu/gunyah.h
@@ -0,0 +1,43 @@
+/*
+ * QEMU Gunyah hypervisor support
+ *
+ * Copyright(c) 2023 Qualcomm Innovation Center, Inc. All Rights Reserved.
+ *
+ * SPDX-License-Identifier: GPL-2.0-or-later
+ */
+
+/* header to be included in non-Gunyah-specific code */
+
+#ifndef QEMU_GUNYAH_H
+#define QEMU_GUNYAH_H
+
+#include "qemu/accel.h"
+#include "qom/object.h"
+
+#ifdef NEED_CPU_H
+#include "cpu.h"
+#ifdef CONFIG_GUNYAH
+# define CONFIG_GUNYAH_IS_POSSIBLE
+#endif
+#else
+# define CONFIG_GUNYAH_IS_POSSIBLE
+#endif
+
+#ifdef CONFIG_GUNYAH_IS_POSSIBLE
+
+extern bool gunyah_allowed;
+
+#define gunyah_enabled() (gunyah_allowed)
+
+#define TYPE_GUNYAH_ACCEL ACCEL_CLASS_NAME("gunyah")
+typedef struct GUNYAHState GUNYAHState;
+DECLARE_INSTANCE_CHECKER(GUNYAHState, GUNYAH_STATE,
+                         TYPE_GUNYAH_ACCEL)
+
+#else   /* CONFIG_GUNYAH_IS_POSSIBLE */
+
+#define gunyah_enabled()    0
+
+#endif  /* CONFIG_GUNYAH_IS_POSSIBLE */
+
+#endif  /* QEMU_GUNYAH_H */
diff --git a/include/sysemu/gunyah_int.h b/include/sysemu/gunyah_int.h
new file mode 100644
index 0000000000..37de628b37
--- /dev/null
+++ b/include/sysemu/gunyah_int.h
@@ -0,0 +1,27 @@
+/*
+ * QEMU Gunyah hypervisor support
+ *
+ * Copyright(c) 2023 Qualcomm Innovation Center, Inc. All Rights Reserved.
+ *
+ * SPDX-License-Identifier: GPL-2.0-or-later
+ */
+
+/* header to be included in Gunyah-specific code */
+
+#ifndef GUNYAH_INT_H
+#define GUNYAH_INT_H
+
+#include "qemu/accel.h"
+#include "qemu/typedefs.h"
+
+struct GUNYAHState {
+    AccelState parent_obj;
+
+    int fd;
+    int vmfd;
+};
+
+int gunyah_create_vm(void);
+void *gunyah_cpu_thread_fn(void *arg);
+
+#endif    /* GUNYAH_INT_H */
diff --git a/meson.build b/meson.build
index 98e68ef0b1..ca46b51c55 100644
--- a/meson.build
+++ b/meson.build
@@ -136,6 +136,7 @@ if cpu in ['x86', 'x86_64', 'arm', 'aarch64']
   # i386 emulator provides xenpv machine type for multiple architectures
   accelerator_targets += {
     'CONFIG_XEN': ['i386-softmmu', 'x86_64-softmmu', 'aarch64-softmmu'],
+    'CONFIG_GUNYAH': ['aarch64-softmmu']
   }
 endif
 if cpu in ['x86', 'x86_64']
@@ -608,6 +609,7 @@ iokit = []
 emulator_link_args = []
 nvmm =not_found
 hvf = not_found
+gunyah = not_found
 midl = not_found
 widl = not_found
 pathcch = not_found
@@ -668,6 +670,9 @@ if get_option('hax').allowed()
     accelerators += 'CONFIG_HAX'
   endif
 endif
+if get_option('gunyah').allowed()
+    accelerators += 'CONFIG_GUNYAH'
+endif
 if targetos == 'netbsd'
   nvmm = cc.find_library('nvmm', required: get_option('nvmm'))
   if nvmm.found()
@@ -718,6 +723,9 @@ endif
 if 'CONFIG_WHPX' not in accelerators and get_option('whpx').enabled()
   error('WHPX not available on this platform')
 endif
+if 'CONFIG_GUNYAH' not in accelerators and get_option('gunyah').enabled()
+  error('Gunyah not available on this platform')
+endif
 
 ################
 # Dependencies #
@@ -4149,6 +4157,7 @@ if have_system
     summary_info += {'xen ctrl version':  xen.version()}
   endif
   summary_info += {'Xen emulation':     config_all.has_key('CONFIG_XEN_EMU')}
+  summary_info += {'Gunyah support':    config_all.has_key('CONFIG_GUNYAH')}
 endif
 summary_info += {'TCG support':       config_all.has_key('CONFIG_TCG')}
 if config_all.has_key('CONFIG_TCG')
diff --git a/meson_options.txt b/meson_options.txt
index aaea5ddd77..b914f0e853 100644
--- a/meson_options.txt
+++ b/meson_options.txt
@@ -105,6 +105,8 @@ option('dbus_display', type: 'feature', value: 'auto',
        description: '-display dbus support')
 option('tpm', type : 'feature', value : 'auto',
        description: 'TPM support')
+option('gunyah', type: 'feature', value: 'auto',
+       description: 'Gunyah acceleration support')
 
 # Do not enable it by default even for Mingw32, because it doesn't
 # work on Wine.
diff --git a/scripts/meson-buildoptions.sh b/scripts/meson-buildoptions.sh
index 9da3fe299b..0c95f79645 100644
--- a/scripts/meson-buildoptions.sh
+++ b/scripts/meson-buildoptions.sh
@@ -112,6 +112,7 @@ meson_options_help() {
   printf "%s\n" '  guest-agent-msi Build MSI package for the QEMU Guest Agent'
   printf "%s\n" '  hax             HAX acceleration support'
   printf "%s\n" '  hvf             HVF acceleration support'
+  printf "%s\n" '  gunyah          Gunyah acceleration support'
   printf "%s\n" '  iconv           Font glyph conversion support'
   printf "%s\n" '  jack            JACK sound support'
   printf "%s\n" '  keyring         Linux keyring support'
@@ -312,6 +313,8 @@ _meson_option_parse() {
     --disable-guest-agent) printf "%s" -Dguest_agent=disabled ;;
     --enable-guest-agent-msi) printf "%s" -Dguest_agent_msi=enabled ;;
     --disable-guest-agent-msi) printf "%s" -Dguest_agent_msi=disabled ;;
+    --enable-gunyah) printf "%s" -Dgunyah=enabled ;;
+    --disable-gunyah) printf "%s" -Dgunyah=disabled ;;
     --enable-hax) printf "%s" -Dhax=enabled ;;
     --disable-hax) printf "%s" -Dhax=disabled ;;
     --enable-hexagon-idef-parser) printf "%s" -Dhexagon_idef_parser=true ;;
diff --git a/target/arm/cpu64.c b/target/arm/cpu64.c
index 96158093cc..a712f1a3b3 100644
--- a/target/arm/cpu64.c
+++ b/target/arm/cpu64.c
@@ -33,6 +33,7 @@
 #include "hw/qdev-properties.h"
 #include "internals.h"
 #include "cpregs.h"
+#include "sysemu/gunyah.h"
 
 void arm_cpu_sve_finalize(ARMCPU *cpu, Error **errp)
 {
@@ -686,7 +687,7 @@ static void aarch64_host_initfn(Object *obj)
 
 static void aarch64_max_initfn(Object *obj)
 {
-    if (kvm_enabled() || hvf_enabled()) {
+    if (kvm_enabled() || hvf_enabled() || gunyah_enabled()) {
         /* With KVM or HVF, '-cpu max' is identical to '-cpu host' */
         aarch64_host_initfn(obj);
         return;
@@ -706,7 +707,7 @@ static const ARMCPUInfo aarch64_cpus[] = {
     { .name = "cortex-a57",         .initfn = aarch64_a57_initfn },
     { .name = "cortex-a53",         .initfn = aarch64_a53_initfn },
     { .name = "max",                .initfn = aarch64_max_initfn },
-#if defined(CONFIG_KVM) || defined(CONFIG_HVF)
+#if defined(CONFIG_KVM) || defined(CONFIG_HVF) || defined(CONFIG_GUNYAH)
     { .name = "host",               .initfn = aarch64_host_initfn },
 #endif
 };
-- 
2.25.1

Re: [RFC/PATCH v0 03/12] gunyah: Basic support

[Philippe Mathieu-Daudé]

Hi Srivatsa,

On 11/10/23 18:52, Srivatsa Vaddagiri wrote:
> Add a new accelerator, gunyah, with basic functionality of creating a
> VM. Subsequent patches will add support for other functions required to
> run a VM.
> 
> Signed-off-by: Srivatsa Vaddagiri <quic_svaddagi@quicinc.com>
> ---
>   MAINTAINERS                     |   7 +++
>   accel/Kconfig                   |   3 +
>   accel/gunyah/gunyah-accel-ops.c | 102 ++++++++++++++++++++++++++++++++
>   accel/gunyah/gunyah-all.c       |  70 ++++++++++++++++++++++
>   accel/gunyah/meson.build        |   7 +++
>   accel/meson.build               |   1 +
>   accel/stubs/gunyah-stub.c       |  13 ++++
>   accel/stubs/meson.build         |   1 +
>   docs/about/build-platforms.rst  |   2 +-
>   hw/arm/virt.c                   |   3 +
>   include/sysemu/gunyah.h         |  43 ++++++++++++++
>   include/sysemu/gunyah_int.h     |  27 +++++++++
>   meson.build                     |   9 +++
>   meson_options.txt               |   2 +
>   scripts/meson-buildoptions.sh   |   3 +
>   target/arm/cpu64.c              |   5 +-
>   16 files changed, 295 insertions(+), 3 deletions(-)
>   create mode 100644 accel/gunyah/gunyah-accel-ops.c
>   create mode 100644 accel/gunyah/gunyah-all.c
>   create mode 100644 accel/gunyah/meson.build
>   create mode 100644 accel/stubs/gunyah-stub.c
>   create mode 100644 include/sysemu/gunyah.h
>   create mode 100644 include/sysemu/gunyah_int.h

Can we move gunyah_int.h to accel/gunyah/?

Re: [RFC/PATCH v0 03/12] gunyah: Basic support

[Alex Bennée]

Philippe Mathieu-Daudé <philmd@linaro.org> writes:

> Hi Srivatsa,
>
> On 11/10/23 18:52, Srivatsa Vaddagiri wrote:
>> Add a new accelerator, gunyah, with basic functionality of creating a
>> VM. Subsequent patches will add support for other functions required to
>> run a VM.
>> Signed-off-by: Srivatsa Vaddagiri <quic_svaddagi@quicinc.com>
>> ---
>>   MAINTAINERS                     |   7 +++
>>   accel/Kconfig                   |   3 +
>>   accel/gunyah/gunyah-accel-ops.c | 102 ++++++++++++++++++++++++++++++++
>>   accel/gunyah/gunyah-all.c       |  70 ++++++++++++++++++++++
>>   accel/gunyah/meson.build        |   7 +++
>>   accel/meson.build               |   1 +
>>   accel/stubs/gunyah-stub.c       |  13 ++++
>>   accel/stubs/meson.build         |   1 +
>>   docs/about/build-platforms.rst  |   2 +-
>>   hw/arm/virt.c                   |   3 +
>>   include/sysemu/gunyah.h         |  43 ++++++++++++++
>>   include/sysemu/gunyah_int.h     |  27 +++++++++
>>   meson.build                     |   9 +++
>>   meson_options.txt               |   2 +
>>   scripts/meson-buildoptions.sh   |   3 +
>>   target/arm/cpu64.c              |   5 +-
>>   16 files changed, 295 insertions(+), 3 deletions(-)
>>   create mode 100644 accel/gunyah/gunyah-accel-ops.c
>>   create mode 100644 accel/gunyah/gunyah-all.c
>>   create mode 100644 accel/gunyah/meson.build
>>   create mode 100644 accel/stubs/gunyah-stub.c
>>   create mode 100644 include/sysemu/gunyah.h
>>   create mode 100644 include/sysemu/gunyah_int.h
>
> Can we move gunyah_int.h to accel/gunyah/?

If it's all internal to gunyah itself you could rename it to internal.h
(although we do have various forms of foo-internal.h across the code
base).

However I see the reason is the other accelerators have done so:

       accel/hvf/hvf-accel-ops.c
  58:#include "sysemu/hvf_int.h"
       accel/hvf/hvf-all.c
  14:#include "sysemu/hvf_int.h"
       accel/kvm/kvm-accel-ops.c
  20:#include "sysemu/kvm_int.h"
       accel/kvm/kvm-all.c
  31:#include "sysemu/kvm_int.h"

-- 
Alex Bennée
Virtualisation Tech Lead @ Linaro

Re: [RFC/PATCH v0 03/12] gunyah: Basic support

[Srivatsa Vaddagiri]

* Alex Benn?e <alex.bennee@linaro.org> [2023-10-12 12:32:34]:

> >>   create mode 100644 include/sysemu/gunyah_int.h
> >
> > Can we move gunyah_int.h to accel/gunyah/?
> 
> If it's all internal to gunyah itself you could rename it to internal.h
> (although we do have various forms of foo-internal.h across the code
> base).

Sure!

- vatsa

Re: [RFC/PATCH v0 03/12] gunyah: Basic support

[Alex Bennée]

Srivatsa Vaddagiri <quic_svaddagi@quicinc.com> writes:

> Add a new accelerator, gunyah, with basic functionality of creating a
> VM. Subsequent patches will add support for other functions required to
> run a VM.
>
> Signed-off-by: Srivatsa Vaddagiri <quic_svaddagi@quicinc.com>

Hmm this failed to build:

  FAILED: libqemu-aarch64-softmmu.fa.p/accel_gunyah_gunyah-all.c.o 
  cc -m64 -mcx16 -Ilibqemu-aarch64-softmmu.fa.p -I. -I../.. -Itarget/arm -I../../target/arm -Iqapi -Itrace -Iui -Iui/shader -I/usr/include/pixman-1 -I/usr/include/capstone -I/usr/include/spice-server -I/usr/include/spice-1 -I/usr/include/glib-2.0 -I/usr/lib/x86_64-linux-gnu/glib-2.0/include -fdiagnostics-color=auto -Wall -Winvalid-pch -Werror -std=gnu11 -O2 -g -fstack-protector-strong -Wundef -Wwrite-strings -Wmissing-prototypes -Wstrict-prototypes -Wredundant-decls -Wold-style-declaration -Wold-style-definition -Wtype-limits -Wformat-security -Wformat-y2k -Winit-self -Wignored-qualifiers -Wempty-body -Wnested-externs -Wendif-labels -Wexpansion-to-defined -Wimplicit-fallthrough=2 -Wmissing-format-attribute -Wno-missing-include-dirs -Wno-shift-negative-value -Wno-psabi -Wshadow=local -isystem /home/alex/lsrc/qemu.git/linux-headers -isystem linux-headers -iquote . -iquote /home/alex/lsrc/qemu.git -iquote /home/alex/lsrc/qemu.git/include -iquote /home/alex/lsrc/qemu.git/host/include/x86_64 -iquote /home/alex/lsrc/qemu.git/host/include/generic -iquote /home/alex/lsrc/qemu.git/tcg/i386 -pthread -D_GNU_SOURCE -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -fno-strict-aliasing -fno-common -fwrapv -fPIE -isystem../../linux-headers -isystemlinux-headers -DNEED_CPU_H '-DCONFIG_TARGET="aarch64-softmmu-config-target.h"' '-DCONFIG_DEVICES="aarch64-softmmu-config-devices.h"' -MD -MQ libqemu-aarch64-softmmu.fa.p/accel_gunyah_gunyah-all.c.o -MF libqemu-aarch64-softmmu.fa.p/accel_gunyah_gunyah-all.c.o.d -o libqemu-aarch64-softmmu.fa.p/accel_gunyah_gunyah-all.c.o -c ../../accel/gunyah/gunyah-all.c
  ../../accel/gunyah/gunyah-all.c:21:10: fatal error: linux-headers/linux/gunyah.h: No such file or directory
     21 | #include "linux-headers/linux/gunyah.h"
        |          ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  compilation terminated.

Easiest solution:

  add PATCH after 02 importing headers

I usually add a !MERGE to the summary just to remind myself that we'll
need to properly update the headers before we merge. e.g: linux-headers: update to v6.6-with-gunyah (!MERGE)

There are some other merge failures bellow so perhaps its time to do a
re-base and send v1?

Whats the current state of the kernel merge?

<snip>
> diff --git a/scripts/meson-buildoptions.sh b/scripts/meson-buildoptions.sh
> index 9da3fe299b..0c95f79645 100644
> --- a/scripts/meson-buildoptions.sh
> +++ b/scripts/meson-buildoptions.sh
> @@ -112,6 +112,7 @@ meson_options_help() {
>    printf "%s\n" '  guest-agent-msi Build MSI package for the QEMU Guest Agent'
>    printf "%s\n" '  hax             HAX acceleration support'
>    printf "%s\n" '  hvf             HVF acceleration support'
> +  printf "%s\n" '  gunyah          Gunyah acceleration support'

need rebasing due to hax drop

>    printf "%s\n" '  iconv           Font glyph conversion support'
>    printf "%s\n" '  jack            JACK sound support'
>    printf "%s\n" '  keyring         Linux keyring support'
> @@ -312,6 +313,8 @@ _meson_option_parse() {
>      --disable-guest-agent) printf "%s" -Dguest_agent=disabled ;;
>      --enable-guest-agent-msi) printf "%s" -Dguest_agent_msi=enabled ;;
>      --disable-guest-agent-msi) printf "%s" -Dguest_agent_msi=disabled ;;
> +    --enable-gunyah) printf "%s" -Dgunyah=enabled ;;
> +    --disable-gunyah) printf "%s" -Dgunyah=disabled ;;

ditto.

>      --enable-hax) printf "%s" -Dhax=enabled ;;
>      --disable-hax) printf "%s" -Dhax=disabled ;;
>      --enable-hexagon-idef-parser) printf "%s" -Dhexagon_idef_parser=true ;;
> diff --git a/target/arm/cpu64.c b/target/arm/cpu64.c
> index 96158093cc..a712f1a3b3 100644
> --- a/target/arm/cpu64.c
> +++ b/target/arm/cpu64.c
> @@ -33,6 +33,7 @@
>  #include "hw/qdev-properties.h"
>  #include "internals.h"
>  #include "cpregs.h"
> +#include "sysemu/gunyah.h"

This should be moved up next to the other accelerator includes.

>  
>  void arm_cpu_sve_finalize(ARMCPU *cpu, Error **errp)
>  {
> @@ -686,7 +687,7 @@ static void aarch64_host_initfn(Object *obj)
>  
>  static void aarch64_max_initfn(Object *obj)
>  {
> -    if (kvm_enabled() || hvf_enabled()) {
> +    if (kvm_enabled() || hvf_enabled() || gunyah_enabled()) {
>          /* With KVM or HVF, '-cpu max' is identical to '-cpu host' */
>          aarch64_host_initfn(obj);
>          return;
> @@ -706,7 +707,7 @@ static const ARMCPUInfo aarch64_cpus[] = {
>      { .name = "cortex-a57",         .initfn = aarch64_a57_initfn },
>      { .name = "cortex-a53",         .initfn = aarch64_a53_initfn },
>      { .name = "max",                .initfn = aarch64_max_initfn },
> -#if defined(CONFIG_KVM) || defined(CONFIG_HVF)
> +#if defined(CONFIG_KVM) || defined(CONFIG_HVF) || defined(CONFIG_GUNYAH)
>      { .name = "host",               .initfn = aarch64_host_initfn },
>  #endif
>  };

-- 
Alex Bennée
Virtualisation Tech Lead @ Linaro

Re: [RFC/PATCH v0 03/12] gunyah: Basic support

[Srivatsa Vaddagiri]

* Alex Benn?e <alex.bennee@linaro.org> [2023-11-29 16:56:38]:

> Srivatsa Vaddagiri <quic_svaddagi@quicinc.com> writes:
> 
> > Add a new accelerator, gunyah, with basic functionality of creating a
> > VM. Subsequent patches will add support for other functions required to
> > run a VM.
> >
> > Signed-off-by: Srivatsa Vaddagiri <quic_svaddagi@quicinc.com>
> 
> Hmm this failed to build:
> 
>   FAILED: libqemu-aarch64-softmmu.fa.p/accel_gunyah_gunyah-all.c.o 
>   cc -m64 -mcx16 -Ilibqemu-aarch64-softmmu.fa.p -I. -I../.. -Itarget/arm -I../../target/arm -Iqapi -Itrace -Iui -Iui/shader -I/usr/include/pixman-1 -I/usr/include/capstone -I/usr/include/spice-server -I/usr/include/spice-1 -I/usr/include/glib-2.0 -I/usr/lib/x86_64-linux-gnu/glib-2.0/include -fdiagnostics-color=auto -Wall -Winvalid-pch -Werror -std=gnu11 -O2 -g -fstack-protector-strong -Wundef -Wwrite-strings -Wmissing-prototypes -Wstrict-prototypes -Wredundant-decls -Wold-style-declaration -Wold-style-definition -Wtype-limits -Wformat-security -Wformat-y2k -Winit-self -Wignored-qualifiers -Wempty-body -Wnested-externs -Wendif-labels -Wexpansion-to-defined -Wimplicit-fallthrough=2 -Wmissing-format-attribute -Wno-missing-include-dirs -Wno-shift-negative-value -Wno-psabi -Wshadow=local -isystem /home/alex/lsrc/qemu.git/linux-headers -isystem linux-headers -iquote . -iquote /home/alex/lsrc/qemu.git -iquote /home/alex/lsrc/qemu.git/include -iquote /home/alex/lsrc/qemu.git/host/include/x86_64 -iquote /home/alex/lsrc/qemu.git/host/include/generic -iquote /home/alex/lsrc/qemu.git/tcg/i386 -pthread -D_GNU_SOURCE -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -fno-strict-aliasing -fno-common -fwrapv -fPIE -isystem../../linux-headers -isystemlinux-headers -DNEED_CPU_H '-DCONFIG_TARGET="aarch64-softmmu-config-target.h"' '-DCONFIG_DEVICES="aarch64-softmmu-config-devices.h"' -MD -MQ libqemu-aarch64-softmmu.fa.p/accel_gunyah_gunyah-all.c.o -MF libqemu-aarch64-softmmu.fa.p/accel_gunyah_gunyah-all.c.o.d -o libqemu-aarch64-softmmu.fa.p/accel_gunyah_gunyah-all.c.o -c ../../accel/gunyah/gunyah-all.c
>   ../../accel/gunyah/gunyah-all.c:21:10: fatal error: linux-headers/linux/gunyah.h: No such file or directory
>      21 | #include "linux-headers/linux/gunyah.h"
>         |          ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>   compilation terminated.
> 
> Easiest solution:
> 
>   add PATCH after 02 importing headers
> 
> I usually add a !MERGE to the summary just to remind myself that we'll
> need to properly update the headers before we merge. e.g: linux-headers: update to v6.6-with-gunyah (!MERGE)

I had run update-linux-headers.sh after Patch 02 before trying to compile rest of
the patch series. It was run against a Linux kernel tree having the Gunyah
patches. If it helps, I can include the actual header itself as a Patch (with
!MERGE noted), so that it saves some trouble.

> There are some other merge failures bellow so perhaps its time to do a
> re-base and send v1?

Yes working on v1. I have tested virtio-pci working I think, testing SMP now after
which will send out v1 (hopefully early next week).

> Whats the current state of the kernel merge?

The kernel patches are being reworked to avoid pinning of memory and to allow
on-demand allocation of VM memory. That should address the major feedback
received. We expect the next version of kernel patches to be published later
this month, and a potential merge in kernel.org hopefully before April (6.9)?!

It will also require some rework in Qemu support to use the new guest
memfd APIs for allocating VM memory (for confidential VMs). I will publish
support for it in v2 (could be a month away I think).

> <snip>
> > diff --git a/scripts/meson-buildoptions.sh b/scripts/meson-buildoptions.sh
> > index 9da3fe299b..0c95f79645 100644
> > --- a/scripts/meson-buildoptions.sh
> > +++ b/scripts/meson-buildoptions.sh
> > @@ -112,6 +112,7 @@ meson_options_help() {
> >    printf "%s\n" '  guest-agent-msi Build MSI package for the QEMU Guest Agent'
> >    printf "%s\n" '  hax             HAX acceleration support'
> >    printf "%s\n" '  hvf             HVF acceleration support'
> > +  printf "%s\n" '  gunyah          Gunyah acceleration support'
> 
> need rebasing due to hax drop
> 
> >    printf "%s\n" '  iconv           Font glyph conversion support'
> >    printf "%s\n" '  jack            JACK sound support'
> >    printf "%s\n" '  keyring         Linux keyring support'
> > @@ -312,6 +313,8 @@ _meson_option_parse() {
> >      --disable-guest-agent) printf "%s" -Dguest_agent=disabled ;;
> >      --enable-guest-agent-msi) printf "%s" -Dguest_agent_msi=enabled ;;
> >      --disable-guest-agent-msi) printf "%s" -Dguest_agent_msi=disabled ;;
> > +    --enable-gunyah) printf "%s" -Dgunyah=enabled ;;
> > +    --disable-gunyah) printf "%s" -Dgunyah=disabled ;;
> 
> ditto.
> 
> >      --enable-hax) printf "%s" -Dhax=enabled ;;
> >      --disable-hax) printf "%s" -Dhax=disabled ;;
> >      --enable-hexagon-idef-parser) printf "%s" -Dhexagon_idef_parser=true ;;
> > diff --git a/target/arm/cpu64.c b/target/arm/cpu64.c
> > index 96158093cc..a712f1a3b3 100644
> > --- a/target/arm/cpu64.c
> > +++ b/target/arm/cpu64.c
> > @@ -33,6 +33,7 @@
> >  #include "hw/qdev-properties.h"
> >  #include "internals.h"
> >  #include "cpregs.h"
> > +#include "sysemu/gunyah.h"
> 
> This should be moved up next to the other accelerator includes.

Sure thanks for this feedback. Will incorporate them in v1.

- vatsa

[RFC/PATCH v0 04/12] gunyah: Add VM properties

[Srivatsa Vaddagiri]

Add 'protected-vm' and 'preshmem-size' properties that can be specified
for a VM.

Protected VMs are those that have 'protected-vm' property set. Their
memory cannot be accessed by their (potentially untrusted) host. They
are useful to run secure applications whose data should remain private
to the VM.

Since a protected VM may need some memory shared with its host, for
exchange of information, 'preshmem-size' specifies what portion of a
protected VM's memory should be shared with its host, with the remaining
portion being private to it.

Signed-off-by: Srivatsa Vaddagiri <quic_svaddagi@quicinc.com>
---
 accel/gunyah/gunyah-accel-ops.c | 65 +++++++++++++++++++++++++++++++++
 include/sysemu/gunyah_int.h     |  2 +
 2 files changed, 67 insertions(+)

diff --git a/accel/gunyah/gunyah-accel-ops.c b/accel/gunyah/gunyah-accel-ops.c
index 1c1c20abf1..77089c4f36 100644
--- a/accel/gunyah/gunyah-accel-ops.c
+++ b/accel/gunyah/gunyah-accel-ops.c
@@ -37,6 +37,60 @@ static void gunyah_accel_instance_init(Object *obj)
     s->vmfd = -1;
 }
 
+static void gunyah_get_preshmem_size(Object *obj, Visitor *v,
+                                    const char *name, void *opaque,
+                                    Error **errp)
+{
+    GUNYAHState *s = GUNYAH_STATE(obj);
+    uint32_t value = s->preshmem_size;
+
+    visit_type_uint32(v, name, &value, errp);
+}
+
+static void gunyah_set_preshmem_size(Object *obj, Visitor *v,
+                                    const char *name, void *opaque,
+                                    Error **errp)
+{
+    GUNYAHState *s = GUNYAH_STATE(obj);
+    uint32_t value;
+
+    if (s->fd != -1) {
+        error_setg(errp, "Cannot set properties after VM is created");
+        return;
+    }
+
+    if (!visit_type_uint32(v, name, &value, errp)) {
+        error_setg(errp, "preshmem-size must be an unsigned integer");
+        return;
+    }
+
+    if (value & (value - 1)) {
+        error_setg(errp, "preshmem-size must be a power of two");
+        return;
+    }
+
+    if (!s->is_protected_vm) {
+        error_setg(errp, "preshmem-size is applicable only for protected VMs");
+        return;
+    }
+
+    s->preshmem_size = value;
+}
+
+static bool gunyah_get_protected_vm(Object *obj, Error **errp)
+{
+    GUNYAHState *s = GUNYAH_STATE(obj);
+
+    return s->is_protected_vm;
+}
+
+static void gunyah_set_protected_vm(Object *obj, bool value, Error **errp)
+{
+    GUNYAHState *s = GUNYAH_STATE(obj);
+
+    s->is_protected_vm = value;
+}
+
 static void gunyah_accel_class_init(ObjectClass *oc, void *data)
 {
     AccelClass *ac = ACCEL_CLASS(oc);
@@ -44,6 +98,17 @@ static void gunyah_accel_class_init(ObjectClass *oc, void *data)
     ac->name = "GUNYAH";
     ac->init_machine = gunyah_init;
     ac->allowed = &gunyah_allowed;
+
+    object_class_property_add_bool(oc, "protected-vm",
+                    gunyah_get_protected_vm, gunyah_set_protected_vm);
+    object_class_property_set_description(oc, "protected-vm",
+            "Launch a VM of protected type");
+
+    object_class_property_add(oc, "preshmem-size", "uint32",
+                gunyah_get_preshmem_size, gunyah_set_preshmem_size, NULL, NULL);
+    object_class_property_set_description(oc, "preshmem-size",
+        "This property is applicable for protected VMs and indicates "
+        "the portion of VM's memory that should be shared with its host");
 }
 
 static const TypeInfo gunyah_accel_type = {
diff --git a/include/sysemu/gunyah_int.h b/include/sysemu/gunyah_int.h
index 37de628b37..b1fd7f9ea2 100644
--- a/include/sysemu/gunyah_int.h
+++ b/include/sysemu/gunyah_int.h
@@ -19,6 +19,8 @@ struct GUNYAHState {
 
     int fd;
     int vmfd;
+    bool is_protected_vm;
+    uint32_t preshmem_size;
 };
 
 int gunyah_create_vm(void);
-- 
2.25.1

[RFC/PATCH v0 05/12] gunyah: Support memory assignment

[Srivatsa Vaddagiri]

Gunyah hypervisor supports several APIs for a host VM to assign some of
its memory to the VM being created.

    Lend - assigned memory is made private to VM (host loses access)
    Share - assigned memory is shared between host and guest VM

No APIs exist however, at this time, for a protected VM to share some of its
(private) memory with host at runtime.

Since, in practice, even a protected VM may need some shared memory to
exchange information with its host, we split VM's memory into two
portions - one that is kept private (via Lend API) and other that is
shared with host (via Share API). Shared portion size is specified via
'preshmem-size' property of VM.

Note: Gunyah kernel driver from Android common kernel repository exposes
two separate ioctls for lend and share operations [1]. A subsequent
version of kernel driver that will be merged in Linux kernel repository
will combine the two ioctls into one, when this patch will be updated.

1. Refer GH_VM_ANDROID_LEND_USER_MEM and GH_VM_SET_USER_MEM_REGION in
https://android.googlesource.com/kernel/common/+/refs/heads/android14-6.1/drivers/virt/gunyah/vm_mgr.c

Signed-off-by: Srivatsa Vaddagiri <quic_svaddagi@quicinc.com>
---
 accel/gunyah/gunyah-all.c   | 248 ++++++++++++++++++++++++++++++++++++
 include/sysemu/gunyah_int.h |  25 ++++
 2 files changed, 273 insertions(+)

diff --git a/accel/gunyah/gunyah-all.c b/accel/gunyah/gunyah-all.c
index 370add75f9..38d0a52b7f 100644
--- a/accel/gunyah/gunyah-all.c
+++ b/accel/gunyah/gunyah-all.c
@@ -14,12 +14,20 @@
 #include <sys/ioctl.h>
 #include "qemu/osdep.h"
 #include "qemu/typedefs.h"
+#include "qemu/units.h"
 #include "hw/core/cpu.h"
 #include "sysemu/cpus.h"
 #include "sysemu/gunyah.h"
 #include "sysemu/gunyah_int.h"
 #include "linux-headers/linux/gunyah.h"
+#include "exec/memory.h"
 #include "qemu/error-report.h"
+#include "exec/address-spaces.h"
+
+static void gunyah_region_add(MemoryListener *listener,
+                           MemoryRegionSection *section);
+static void gunyah_region_del(MemoryListener *listener,
+                           MemoryRegionSection *section);
 
 static int gunyah_ioctl(int type, ...)
 {
@@ -36,9 +44,32 @@ static int gunyah_ioctl(int type, ...)
     return ioctl(s->fd, type, arg);
 }
 
+int gunyah_vm_ioctl(int type, ...)
+{
+    void *arg;
+    va_list ap;
+    GUNYAHState *s = GUNYAH_STATE(current_accel());
+
+    assert(s->vmfd);
+
+    va_start(ap, type);
+    arg = va_arg(ap, void *);
+    va_end(ap);
+
+    return ioctl(s->vmfd, type, arg);
+}
+
+static MemoryListener gunyah_memory_listener = {
+    .name = "gunyah",
+    .priority = MEMORY_LISTENER_PRIORITY_ACCEL,
+    .region_add = gunyah_region_add,
+    .region_del = gunyah_region_del,
+};
+
 int gunyah_create_vm(void)
 {
     GUNYAHState *s;
+    int i;
 
     s = GUNYAH_STATE(current_accel());
 
@@ -55,6 +86,15 @@ int gunyah_create_vm(void)
         exit(1);
     }
 
+    qemu_mutex_init(&s->slots_lock);
+    s->nr_slots = GUNYAH_MAX_MEM_SLOTS;
+    for (i = 0; i < s->nr_slots; ++i) {
+        s->slots[i].start = 0;
+        s->slots[i].size = 0;
+        s->slots[i].id = i;
+    }
+
+    memory_listener_register(&gunyah_memory_listener, &address_space_memory);
     return 0;
 }
 
@@ -68,3 +108,211 @@ void *gunyah_cpu_thread_fn(void *arg)
 
     return NULL;
 }
+
+#define gunyah_slots_lock(s)    qemu_mutex_lock(&s->slots_lock)
+#define gunyah_slots_unlock(s)  qemu_mutex_unlock(&s->slots_lock)
+
+static gunyah_slot *gunyah_find_overlap_slot(GUNYAHState *s,
+                uint64_t start, uint64_t size)
+{
+    gunyah_slot *slot;
+    int i;
+
+    for (i = 0; i < s->nr_slots; ++i) {
+        slot = &s->slots[i];
+        if (slot->size && start < (slot->start + slot->size) &&
+            (start + size) > slot->start) {
+            return slot;
+        }
+    }
+
+    return NULL;
+}
+
+/* Called with s->slots_lock held */
+static gunyah_slot *gunyah_get_free_slot(GUNYAHState *s)
+{
+    int i;
+
+    for (i = 0; i < s->nr_slots; i++) {
+        if (s->slots[i].size == 0) {
+            return &s->slots[i];
+        }
+    }
+
+    return NULL;
+}
+
+static void gunyah_add_mem(GUNYAHState *s, MemoryRegionSection *section,
+        bool lend, enum gh_mem_flags flags)
+{
+    gunyah_slot *slot;
+    MemoryRegion *area = section->mr;
+    struct gh_userspace_memory_region gumr;
+    int ret;
+
+    slot = gunyah_get_free_slot(s);
+    if (!slot) {
+        error_report("No free slots to add memory!");
+        exit(1);
+    }
+
+    slot->size = int128_get64(section->size);
+    slot->mem = memory_region_get_ram_ptr(area) + section->offset_within_region;
+    slot->start = section->offset_within_address_space;
+    slot->lend = lend;
+
+    gumr.label = slot->id;
+    gumr.flags = flags;
+    gumr.guest_phys_addr = slot->start;
+    gumr.memory_size = slot->size;
+    gumr.userspace_addr = (__u64) slot->mem;
+
+    /*
+     * GH_VM_ANDROID_LEND_USER_MEM is temporary, until
+     * GH_VM_SET_USER_MEM_REGION is enhanced to support lend option also.
+     */
+    if (lend) {
+        ret = gunyah_vm_ioctl(GH_VM_ANDROID_LEND_USER_MEM, &gumr);
+    } else {
+        ret = gunyah_vm_ioctl(GH_VM_SET_USER_MEM_REGION, &gumr);
+    }
+
+    if (ret) {
+        error_report("failed to add mem (%s)", strerror(errno));
+        exit(1);
+    }
+}
+
+/*
+ * Check if memory of a protected VM needs to be split into two portions - one
+ * private to it and other shared with host.
+ */
+static bool split_mem(GUNYAHState *s,
+        MemoryRegion *area, MemoryRegionSection *section)
+{
+    bool writeable = !area->readonly && !area->rom_device;
+
+    /*
+     * Do not split if its not a protected VM OR if the shared mem size is not
+     * specified.
+     */
+    if (!s->is_protected_vm || !s->preshmem_size) {
+        return false;
+    }
+
+    /* Split only memory that can be written to by guest */
+    if (!memory_region_is_ram(area) || !writeable) {
+        return false;
+    }
+
+    /* Have we reserved already? */
+    if (qatomic_read(&s->preshmem_reserved)) {
+        return false;
+    }
+
+    /* Do we have enough available memory? */
+    if (section->size <= s->preshmem_size) {
+        return false;
+    }
+
+    return true;
+}
+
+static void gunyah_set_phys_mem(GUNYAHState *s,
+        MemoryRegionSection *section, bool add)
+{
+    MemoryRegion *area = section->mr;
+    bool writable = !area->readonly && !area->rom_device;
+    enum gh_mem_flags flags = 0;
+    uint64_t page_size = qemu_real_host_page_size();
+    MemoryRegionSection mrs = *section;
+    bool lend = s->is_protected_vm, split = false;
+    struct gunyah_slot *slot;
+
+    /*
+     * Gunyah hypervisor, at this time, does not support mapping memory
+     * at low address (< 1GiB). Below code will be updated once
+     * that limitation is addressed.
+     */
+    if (section->offset_within_address_space < GiB) {
+        return;
+    }
+
+    if (!memory_region_is_ram(area)) {
+        if (writable) {
+            return;
+        } else if (!memory_region_is_romd(area)) {
+            /*
+             * If the memory device is not in romd_mode, then we actually want
+             * to remove the gunyah memory slot so all accesses will trap.
+             */
+             add = false;
+        }
+    }
+
+    if (!QEMU_IS_ALIGNED(int128_get64(section->size), page_size) ||
+        !QEMU_IS_ALIGNED(section->offset_within_address_space, page_size)) {
+        error_report("Not page aligned");
+        add = false;
+    }
+
+    gunyah_slots_lock(s);
+
+    slot = gunyah_find_overlap_slot(s,
+            section->offset_within_address_space,
+            int128_get64(section->size));
+
+    if (!add) {
+        if (slot) {
+            error_report("Memory slot removal not yet supported!");
+            exit(1);
+        }
+        /* Nothing to be done as address range was not previously registered */
+        goto done;
+    } else {
+        if (slot) {
+            error_report("Overlapping slot registration not supported!");
+            exit(1);
+        }
+    }
+
+    if (area->readonly ||
+        (!memory_region_is_ram(area) && memory_region_is_romd(area))) {
+        flags = GH_MEM_ALLOW_READ | GH_MEM_ALLOW_EXEC;
+    } else {
+        flags = GH_MEM_ALLOW_READ | GH_MEM_ALLOW_WRITE | GH_MEM_ALLOW_EXEC;
+    }
+
+    split = split_mem(s, area, &mrs);
+    if (split) {
+        mrs.size -= s->preshmem_size;
+        gunyah_add_mem(s, &mrs, true, flags);
+        lend = false;
+        mrs.offset_within_region += mrs.size;
+        mrs.offset_within_address_space += mrs.size;
+        mrs.size = s->preshmem_size;
+        qatomic_set(&s->preshmem_reserved, true);
+    }
+
+    gunyah_add_mem(s, &mrs, lend, flags);
+
+done:
+    gunyah_slots_unlock(s);
+}
+
+static void gunyah_region_add(MemoryListener *listener,
+                           MemoryRegionSection *section)
+{
+    GUNYAHState *s = GUNYAH_STATE(current_accel());
+
+    gunyah_set_phys_mem(s, section, true);
+}
+
+static void gunyah_region_del(MemoryListener *listener,
+                           MemoryRegionSection *section)
+{
+    GUNYAHState *s = GUNYAH_STATE(current_accel());
+
+    gunyah_set_phys_mem(s, section, false);
+}
diff --git a/include/sysemu/gunyah_int.h b/include/sysemu/gunyah_int.h
index b1fd7f9ea2..17b4ef9920 100644
--- a/include/sysemu/gunyah_int.h
+++ b/include/sysemu/gunyah_int.h
@@ -13,17 +13,42 @@
 
 #include "qemu/accel.h"
 #include "qemu/typedefs.h"
+#include "qemu/thread.h"
+
+typedef struct gunyah_slot {
+    uint64_t start;
+    uint64_t size;
+    uint8_t *mem;
+    uint32_t id;
+    uint32_t flags;
+
+    /*
+     * @lend indicates if memory was lent.
+     *
+     * This flag is temporarily used until the upstream Gunyah kernel driver
+     * patches are updated to support indication of lend vs share via flags
+     * field of GH_SET_USER_MEM_API interface.
+     */
+    bool lend;
+} gunyah_slot;
+
+#define GUNYAH_MAX_MEM_SLOTS    32
 
 struct GUNYAHState {
     AccelState parent_obj;
 
+    QemuMutex slots_lock;
+    gunyah_slot slots[GUNYAH_MAX_MEM_SLOTS];
+    uint32_t nr_slots;
     int fd;
     int vmfd;
     bool is_protected_vm;
+    bool preshmem_reserved;
     uint32_t preshmem_size;
 };
 
 int gunyah_create_vm(void);
+int gunyah_vm_ioctl(int type, ...);
 void *gunyah_cpu_thread_fn(void *arg);
 
 #endif    /* GUNYAH_INT_H */
-- 
2.25.1

[RFC/PATCH v0 06/12] gunyah: Add IRQFD and IOEVENTFD functions

[Srivatsa Vaddagiri]

IRQFD function allows registering of an @eventfd and @irq. @irq will be
injected inside guest when @eventfd is written into.

IOEVENTFD function allows registering an @eventfd and a guest physical
address, @addr, along with optional data. A poll() on @eventfd  will be
woken up when guest attempts to access @addr.

Signed-off-by: Srivatsa Vaddagiri <quic_svaddagi@quicinc.com>
---
 accel/gunyah/gunyah-all.c   | 94 +++++++++++++++++++++++++++++++++++++
 include/sysemu/gunyah_int.h |  1 +
 2 files changed, 95 insertions(+)

diff --git a/accel/gunyah/gunyah-all.c b/accel/gunyah/gunyah-all.c
index 38d0a52b7f..6ec60aa8e8 100644
--- a/accel/gunyah/gunyah-all.c
+++ b/accel/gunyah/gunyah-all.c
@@ -23,11 +23,21 @@
 #include "exec/memory.h"
 #include "qemu/error-report.h"
 #include "exec/address-spaces.h"
+#include "qapi/error.h"
+#include "qemu/event_notifier.h"
 
 static void gunyah_region_add(MemoryListener *listener,
                            MemoryRegionSection *section);
 static void gunyah_region_del(MemoryListener *listener,
                            MemoryRegionSection *section);
+static void gunyah_mem_ioeventfd_add(MemoryListener *listener,
+                                  MemoryRegionSection *section,
+                                  bool match_data, uint64_t data,
+                                  EventNotifier *e);
+static void gunyah_mem_ioeventfd_del(MemoryListener *listener,
+                                  MemoryRegionSection *section,
+                                  bool match_data, uint64_t data,
+                                  EventNotifier *e);
 
 static int gunyah_ioctl(int type, ...)
 {
@@ -64,6 +74,8 @@ static MemoryListener gunyah_memory_listener = {
     .priority = MEMORY_LISTENER_PRIORITY_ACCEL,
     .region_add = gunyah_region_add,
     .region_del = gunyah_region_del,
+    .eventfd_add = gunyah_mem_ioeventfd_add,
+    .eventfd_del = gunyah_mem_ioeventfd_del,
 };
 
 int gunyah_create_vm(void)
@@ -316,3 +328,85 @@ static void gunyah_region_del(MemoryListener *listener,
 
     gunyah_set_phys_mem(s, section, false);
 }
+
+int gunyah_add_irqfd(int irqfd, int label, Error **errp)
+{
+    int ret;
+    struct gh_fn_desc fdesc;
+    struct gh_fn_irqfd_arg ghirqfd;
+
+    fdesc.type = GH_FN_IRQFD;
+    fdesc.arg_size = sizeof(struct gh_fn_irqfd_arg);
+    fdesc.arg = (__u64)(&ghirqfd);
+
+    ghirqfd.fd = irqfd;
+    ghirqfd.label = label;
+    ghirqfd.flags = GH_IRQFD_FLAGS_LEVEL;
+
+    ret = gunyah_vm_ioctl(GH_VM_ADD_FUNCTION, &fdesc);
+    if (ret) {
+        error_setg_errno(errp, errno, "GH_FN_IRQFD failed");
+    }
+
+    return ret;
+}
+
+static int gunyah_set_ioeventfd_mmio(int fd, hwaddr addr,
+        uint32_t size, uint32_t data, bool datamatch, bool assign)
+{
+    int ret;
+    struct gh_fn_ioeventfd_arg io;
+    struct gh_fn_desc fdesc;
+
+    io.fd = fd;
+    io.datamatch = datamatch ? data : 0;
+    io.len = size;
+    io.addr = addr;
+    io.flags = 0;
+
+    fdesc.type = GH_FN_IOEVENTFD;
+    fdesc.arg_size = sizeof(struct gh_fn_ioeventfd_arg);
+    fdesc.arg = (__u64)(&io);
+
+    if (assign) {
+        ret = gunyah_vm_ioctl(GH_VM_ADD_FUNCTION, &fdesc);
+    } else {
+        ret = gunyah_vm_ioctl(GH_VM_REMOVE_FUNCTION, &fdesc);
+    }
+
+    return ret;
+}
+
+static void gunyah_mem_ioeventfd_add(MemoryListener *listener,
+                                  MemoryRegionSection *section,
+                                  bool match_data, uint64_t data,
+                                  EventNotifier *e)
+{
+    int fd = event_notifier_get_fd(e);
+    int r;
+
+    r = gunyah_set_ioeventfd_mmio(fd, section->offset_within_address_space,
+                               int128_get64(section->size), data, match_data,
+                               true);
+    if (r < 0) {
+        error_report("error adding ioeventfd: %s", strerror(errno));
+        exit(1);
+    }
+}
+
+static void gunyah_mem_ioeventfd_del(MemoryListener *listener,
+                                  MemoryRegionSection *section,
+                                  bool match_data, uint64_t data,
+                                  EventNotifier *e)
+{
+    int fd = event_notifier_get_fd(e);
+    int r;
+
+    r = gunyah_set_ioeventfd_mmio(fd, section->offset_within_address_space,
+                               int128_get64(section->size), data, match_data,
+                               false);
+    if (r < 0) {
+        error_report("error deleting ioeventfd: %s", strerror(errno));
+        exit(1);
+    }
+}
diff --git a/include/sysemu/gunyah_int.h b/include/sysemu/gunyah_int.h
index 17b4ef9920..011b5a072c 100644
--- a/include/sysemu/gunyah_int.h
+++ b/include/sysemu/gunyah_int.h
@@ -50,5 +50,6 @@ struct GUNYAHState {
 int gunyah_create_vm(void);
 int gunyah_vm_ioctl(int type, ...);
 void *gunyah_cpu_thread_fn(void *arg);
+int gunyah_add_irqfd(int irqfd, int label, Error **errp);
 
 #endif    /* GUNYAH_INT_H */
-- 
2.25.1

[RFC/PATCH v0 07/12] gunyah: Add gicv3 interrupt controller

[Srivatsa Vaddagiri]

Gunyah hypervisor supports emulation of a GICv3 compatible interrupt
controller. Emulation is handled by hypervisor itself, with Qemu being
allowed to specify some of the properties such as IO address at which
GICv3 should be mapped in guest address space. These properties are
conveyed to hypervisor via the device-tree, which is parsed by
hypervisor (or more specifically Resource Manager VM, which is the
trusted agent of hypervisor), before VM begins execution.

Injection of interrupts inside guest is supported by doorbell API of
Gunyah hypervisor. Each doorbell is associated with a specific
interrupt. An eventfd is created and associated with each doorbell/irq.
Injection of a specific irq is accomplished by writing to the eventfd
associated with that irq.

Signed-off-by: Srivatsa Vaddagiri <quic_svaddagi@quicinc.com>
---
 MAINTAINERS                    |   1 +
 accel/gunyah/gunyah-all.c      |   5 ++
 hw/arm/virt.c                  |   2 +
 hw/intc/arm_gicv3_common.c     |   3 +
 hw/intc/arm_gicv3_gunyah.c     | 106 +++++++++++++++++++++++++++++++++
 hw/intc/arm_gicv3_its_common.c |   4 ++
 hw/intc/meson.build            |   1 +
 include/sysemu/gunyah_int.h    |   2 +
 8 files changed, 124 insertions(+)
 create mode 100644 hw/intc/arm_gicv3_gunyah.c

diff --git a/MAINTAINERS b/MAINTAINERS
index 0e76dda91d..368ba02dce 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -515,6 +515,7 @@ M: Srivatsa Vaddagiri <quic_svaddagi@quicinc.com>
 S: Maintained
 F: accel/gunyah
 F: include/sysemu/gunyah.h
+F: target/arm/arm_gicv3_gunyah.c
 F: include/sysemu/gunyah_int.h
 
 WHPX CPUs
diff --git a/accel/gunyah/gunyah-all.c b/accel/gunyah/gunyah-all.c
index 6ec60aa8e8..ea6c74fdc6 100644
--- a/accel/gunyah/gunyah-all.c
+++ b/accel/gunyah/gunyah-all.c
@@ -410,3 +410,8 @@ static void gunyah_mem_ioeventfd_del(MemoryListener *listener,
         exit(1);
     }
 }
+
+GUNYAHState *get_gunyah_state(void)
+{
+    return GUNYAH_STATE(current_accel());
+}
diff --git a/hw/arm/virt.c b/hw/arm/virt.c
index 4222183eb5..0d4c5aa819 100644
--- a/hw/arm/virt.c
+++ b/hw/arm/virt.c
@@ -1937,6 +1937,8 @@ static void finalize_gic_version(VirtMachineState *vms)
                 gics_supported |= VIRT_GIC_VERSION_4_MASK;
             }
         }
+    } else if (gunyah_enabled()) {
+        gics_supported |= VIRT_GIC_VERSION_3_MASK;
     } else {
         error_report("Unsupported accelerator, can not determine GIC support");
         exit(1);
diff --git a/hw/intc/arm_gicv3_common.c b/hw/intc/arm_gicv3_common.c
index 2ebf880ead..74c7117b89 100644
--- a/hw/intc/arm_gicv3_common.c
+++ b/hw/intc/arm_gicv3_common.c
@@ -32,6 +32,7 @@
 #include "gicv3_internal.h"
 #include "hw/arm/linux-boot-if.h"
 #include "sysemu/kvm.h"
+#include "sysemu/gunyah.h"
 
 
 static void gicv3_gicd_no_migration_shift_bug_post_load(GICv3State *cs)
@@ -614,6 +615,8 @@ const char *gicv3_class_name(void)
 {
     if (kvm_irqchip_in_kernel()) {
         return "kvm-arm-gicv3";
+    } else if (gunyah_enabled()) {
+        return "gunyah-arm-gicv3";
     } else {
         if (kvm_enabled()) {
             error_report("Userspace GICv3 is not supported with KVM");
diff --git a/hw/intc/arm_gicv3_gunyah.c b/hw/intc/arm_gicv3_gunyah.c
new file mode 100644
index 0000000000..f52e82bf9a
--- /dev/null
+++ b/hw/intc/arm_gicv3_gunyah.c
@@ -0,0 +1,106 @@
+/*
+ * QEMU Gunyah hypervisor support
+ *
+ * Copyright(c) 2023 Qualcomm Innovation Center, Inc. All Rights Reserved.
+ *
+ * SPDX-License-Identifier: GPL-2.0-or-later
+ */
+
+#include "qemu/osdep.h"
+#include "qapi/error.h"
+#include "cpu.h"
+#include "hw/intc/arm_gicv3_common.h"
+#include "qemu/error-report.h"
+#include "qemu/module.h"
+#include "sysemu/gunyah.h"
+#include "sysemu/gunyah_int.h"
+#include "sysemu/runstate.h"
+#include "gicv3_internal.h"
+#include "vgic_common.h"
+#include "migration/blocker.h"
+#include "qom/object.h"
+#include "target/arm/cpregs.h"
+#include "qemu/event_notifier.h"
+
+struct GUNYAHARMGICv3Class {
+    ARMGICv3CommonClass parent_class;
+    DeviceRealize parent_realize;
+    ResettablePhases parent_phases;
+};
+
+#define TYPE_GUNYAH_ARM_GICV3 "gunyah-arm-gicv3"
+typedef struct GUNYAHARMGICv3Class GUNYAHARMGICv3Class;
+
+/* This is reusing the GICv3State typedef from ARM_GICV3_ITS_COMMON */
+DECLARE_OBJ_CHECKERS(GICv3State, GUNYAHARMGICv3Class,
+                     GUNYAH_ARM_GICV3, TYPE_GUNYAH_ARM_GICV3)
+
+static EventNotifier *irq_notify;
+
+static void gunyah_arm_gicv3_set_irq(void *opaque, int irq, int level)
+{
+    GICv3State *s = (GICv3State *)opaque;
+
+    if (irq < s->num_irq - GIC_INTERNAL) {
+        event_notifier_set(&irq_notify[irq]);
+    }
+}
+
+static void gunyah_arm_gicv3_realize(DeviceState *dev, Error **errp)
+{
+    GICv3State *s = GUNYAH_ARM_GICV3(dev);
+    GUNYAHARMGICv3Class *ggc = GUNYAH_ARM_GICV3_GET_CLASS(s);
+    Error *local_err = NULL;
+    int i;
+    GUNYAHState *state = get_gunyah_state();
+
+    ggc->parent_realize(dev, &local_err);
+    if (local_err) {
+        error_propagate(errp, local_err);
+        return;
+    }
+
+    if (s->revision != 3) {
+        error_setg(errp, "unsupported GIC revision %d for in-kernel GIC",
+                   s->revision);
+        return;
+    }
+
+    gicv3_init_irqs_and_mmio(s, gunyah_arm_gicv3_set_irq, NULL);
+
+    irq_notify = g_malloc_n(s->num_irq - GIC_INTERNAL, sizeof(EventNotifier));
+
+    for (i = 0; i < s->num_irq - GIC_INTERNAL; ++i) {
+        event_notifier_init(&irq_notify[i], 0);
+        gunyah_add_irqfd(irq_notify[i].wfd, i, errp);
+    }
+
+    state->nr_irqs = s->num_irq - GIC_INTERNAL;
+}
+
+static void gunyah_arm_gicv3_class_init(ObjectClass *klass, void *data)
+{
+    DeviceClass *dc = DEVICE_CLASS(klass);
+    ResettableClass *rc = RESETTABLE_CLASS(klass);
+    GUNYAHARMGICv3Class *ggc = GUNYAH_ARM_GICV3_CLASS(klass);
+
+    device_class_set_parent_realize(dc, gunyah_arm_gicv3_realize,
+                                    &ggc->parent_realize);
+    resettable_class_set_parent_phases(rc, NULL, NULL, NULL,
+                                       &ggc->parent_phases);
+}
+
+static const TypeInfo gunyah_arm_gicv3_info = {
+    .name = TYPE_GUNYAH_ARM_GICV3,
+    .parent = TYPE_ARM_GICV3_COMMON,
+    .instance_size = sizeof(GICv3State),
+    .class_init = gunyah_arm_gicv3_class_init,
+    .class_size = sizeof(GUNYAHARMGICv3Class),
+};
+
+static void gunyah_arm_gicv3_register_types(void)
+{
+    type_register_static(&gunyah_arm_gicv3_info);
+}
+
+type_init(gunyah_arm_gicv3_register_types)
diff --git a/hw/intc/arm_gicv3_its_common.c b/hw/intc/arm_gicv3_its_common.c
index abaf77057e..c225c601ab 100644
--- a/hw/intc/arm_gicv3_its_common.c
+++ b/hw/intc/arm_gicv3_its_common.c
@@ -25,6 +25,7 @@
 #include "qemu/log.h"
 #include "qemu/module.h"
 #include "sysemu/kvm.h"
+#include "sysemu/gunyah.h"
 
 static int gicv3_its_pre_save(void *opaque)
 {
@@ -165,6 +166,9 @@ const char *its_class_name(void)
     if (kvm_irqchip_in_kernel()) {
         /* KVM implementation requires this capability */
         return kvm_direct_msi_enabled() ? "arm-its-kvm" : NULL;
+    } else if (gunyah_enabled()) {
+        /* ITS is not yet supported */
+        return NULL;
     } else {
         /* Software emulation based model */
         return "arm-gicv3-its";
diff --git a/hw/intc/meson.build b/hw/intc/meson.build
index ed355941d1..6e680cf0f2 100644
--- a/hw/intc/meson.build
+++ b/hw/intc/meson.build
@@ -73,3 +73,4 @@ specific_ss.add(when: 'CONFIG_LOONGARCH_IPI', if_true: files('loongarch_ipi.c'))
 specific_ss.add(when: 'CONFIG_LOONGARCH_PCH_PIC', if_true: files('loongarch_pch_pic.c'))
 specific_ss.add(when: 'CONFIG_LOONGARCH_PCH_MSI', if_true: files('loongarch_pch_msi.c'))
 specific_ss.add(when: 'CONFIG_LOONGARCH_EXTIOI', if_true: files('loongarch_extioi.c'))
+specific_ss.add(when: ['CONFIG_ARM_GIC', 'CONFIG_GUNYAH', 'TARGET_AARCH64'], if_true: files('arm_gicv3_gunyah.c'))
diff --git a/include/sysemu/gunyah_int.h b/include/sysemu/gunyah_int.h
index 011b5a072c..dc5b4847a9 100644
--- a/include/sysemu/gunyah_int.h
+++ b/include/sysemu/gunyah_int.h
@@ -45,11 +45,13 @@ struct GUNYAHState {
     bool is_protected_vm;
     bool preshmem_reserved;
     uint32_t preshmem_size;
+    uint32_t nr_irqs;
 };
 
 int gunyah_create_vm(void);
 int gunyah_vm_ioctl(int type, ...);
 void *gunyah_cpu_thread_fn(void *arg);
 int gunyah_add_irqfd(int irqfd, int label, Error **errp);
+GUNYAHState *get_gunyah_state(void);
 
 #endif    /* GUNYAH_INT_H */
-- 
2.25.1

[RFC/PATCH v0 08/12] gunyah: Specific device-tree location

[Srivatsa Vaddagiri]

Specify the location of device-tree and its size, as Gunyah requires the
device-tree to be parsed before VM can begin its execution.

Signed-off-by: Srivatsa Vaddagiri <quic_svaddagi@quicinc.com>
---
 MAINTAINERS             |  1 +
 hw/arm/virt.c           |  6 ++++++
 include/sysemu/gunyah.h |  7 +++++++
 target/arm/gunyah.c     | 45 +++++++++++++++++++++++++++++++++++++++++
 target/arm/meson.build  |  4 ++++
 5 files changed, 63 insertions(+)
 create mode 100644 target/arm/gunyah.c

diff --git a/MAINTAINERS b/MAINTAINERS
index 368ba02dce..5a51633d11 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -514,6 +514,7 @@ GUNYAH
 M: Srivatsa Vaddagiri <quic_svaddagi@quicinc.com>
 S: Maintained
 F: accel/gunyah
+F: target/arm/gunyah.c
 F: include/sysemu/gunyah.h
 F: target/arm/arm_gicv3_gunyah.c
 F: include/sysemu/gunyah_int.h
diff --git a/hw/arm/virt.c b/hw/arm/virt.c
index 0d4c5aa819..4293e01383 100644
--- a/hw/arm/virt.c
+++ b/hw/arm/virt.c
@@ -1676,6 +1676,12 @@ void virt_machine_done(Notifier *notifier, void *data)
         exit(1);
     }
 
+    if (gunyah_enabled()) {
+        if (gunyah_arm_set_dtb(info->dtb_start, vms->fdt_size)) {
+            exit(1);
+        }
+    }
+
     fw_cfg_add_extra_pci_roots(vms->bus, vms->fw_cfg);
 
     virt_acpi_setup(vms);
diff --git a/include/sysemu/gunyah.h b/include/sysemu/gunyah.h
index aded49cdf6..101e190619 100644
--- a/include/sysemu/gunyah.h
+++ b/include/sysemu/gunyah.h
@@ -34,10 +34,17 @@ typedef struct GUNYAHState GUNYAHState;
 DECLARE_INSTANCE_CHECKER(GUNYAHState, GUNYAH_STATE,
                          TYPE_GUNYAH_ACCEL)
 
+int gunyah_arm_set_dtb(__u64 dtb_start, __u64 dtb_size);
+
 #else   /* CONFIG_GUNYAH_IS_POSSIBLE */
 
 #define gunyah_enabled()    0
 
+static inline int gunyah_arm_set_dtb(__u64 dtb_start, __u64 dtb_size)
+{
+    return -1;
+}
+
 #endif  /* CONFIG_GUNYAH_IS_POSSIBLE */
 
 #endif  /* QEMU_GUNYAH_H */
diff --git a/target/arm/gunyah.c b/target/arm/gunyah.c
new file mode 100644
index 0000000000..73c1c2a88a
--- /dev/null
+++ b/target/arm/gunyah.c
@@ -0,0 +1,45 @@
+/*
+ * QEMU Gunyah hypervisor support
+ *
+ * Copyright(c) 2023 Qualcomm Innovation Center, Inc. All Rights Reserved.
+ *
+ * SPDX-License-Identifier: GPL-2.0-or-later
+ */
+
+#include "qemu/osdep.h"
+#include "qemu/error-report.h"
+#include "sysemu/gunyah.h"
+#include "sysemu/gunyah_int.h"
+#include "linux-headers/linux/gunyah.h"
+
+/*
+ * Specify location of device-tree in guest address space.
+ *
+ * @dtb_start - Guest physical address where VM's device-tree is found
+ * @dtb_size - Size of device-tree (and any free space after it).
+ *
+ * RM or Resource Manager VM is a trusted and privileged VM that works in
+ * collaboration with Gunyah hypevisor to setup resources for a VM before it can
+ * begin execution. One of its functions includes inspection/modification of a
+ * VM's device-tree before VM begins its execution. Modification can
+ * include specification of runtime resources allocated by hypervisor,
+ * details of which needs to be visible to VM.  VM's device-tree is modified
+ * "inline" making use of "free" space that could exist at the end of device
+ * tree.
+ */
+int gunyah_arm_set_dtb(__u64 dtb_start, __u64 dtb_size)
+{
+    int ret;
+    struct gh_vm_dtb_config dtb;
+
+    dtb.guest_phys_addr = dtb_start;
+    dtb.size = dtb_size;
+
+    ret = gunyah_vm_ioctl(GH_VM_SET_DTB_CONFIG, &dtb);
+    if (ret != 0) {
+        error_report("GH_VM_SET_DTB_CONFIG failed: %s", strerror(errno));
+        exit(1);
+    }
+
+    return 0;
+}
diff --git a/target/arm/meson.build b/target/arm/meson.build
index e645e456da..c7da15ca89 100644
--- a/target/arm/meson.build
+++ b/target/arm/meson.build
@@ -36,3 +36,7 @@ endif
 
 target_arch += {'arm': arm_ss}
 target_softmmu_arch += {'arm': arm_system_ss}
+
+arm_system_ss.add(when: 'CONFIG_GUNYAH', if_true: files(
+  'gunyah.c',
+))
-- 
2.25.1

Re: [RFC/PATCH v0 08/12] gunyah: Specific device-tree location

[Philippe Mathieu-Daudé]

On 11/10/23 18:52, Srivatsa Vaddagiri wrote:
> Specify the location of device-tree and its size, as Gunyah requires the
> device-tree to be parsed before VM can begin its execution.
> 
> Signed-off-by: Srivatsa Vaddagiri <quic_svaddagi@quicinc.com>
> ---
>   MAINTAINERS             |  1 +
>   hw/arm/virt.c           |  6 ++++++
>   include/sysemu/gunyah.h |  7 +++++++
>   target/arm/gunyah.c     | 45 +++++++++++++++++++++++++++++++++++++++++
>   target/arm/meson.build  |  4 ++++
>   5 files changed, 63 insertions(+)
>   create mode 100644 target/arm/gunyah.c


> diff --git a/include/sysemu/gunyah.h b/include/sysemu/gunyah.h
> index aded49cdf6..101e190619 100644
> --- a/include/sysemu/gunyah.h
> +++ b/include/sysemu/gunyah.h
> @@ -34,10 +34,17 @@ typedef struct GUNYAHState GUNYAHState;
>   DECLARE_INSTANCE_CHECKER(GUNYAHState, GUNYAH_STATE,
>                            TYPE_GUNYAH_ACCEL)
>   
> +int gunyah_arm_set_dtb(__u64 dtb_start, __u64 dtb_size);

By moving this declaration before the #ifdef ...

> +
>   #else   /* CONFIG_GUNYAH_IS_POSSIBLE */
>   
>   #define gunyah_enabled()    0
>   
> +static inline int gunyah_arm_set_dtb(__u64 dtb_start, __u64 dtb_size)
> +{
> +    return -1;
> +}

... you don't need this stub.

> +
>   #endif  /* CONFIG_GUNYAH_IS_POSSIBLE */
>   
>   #endif  /* QEMU_GUNYAH_H */

[RFC/PATCH v0 09/12] gunyah: Customize device-tree

[Srivatsa Vaddagiri]

Customize device-tree with Gunyah specific properties. Some of these
properties include specification of doorbells that need to be created
and associated with various interrupts.

Signed-off-by: Srivatsa Vaddagiri <quic_svaddagi@quicinc.com>
---
 hw/arm/virt.c           | 11 ++++++
 include/sysemu/gunyah.h |  7 ++++
 target/arm/gunyah.c     | 79 +++++++++++++++++++++++++++++++++++++++++
 3 files changed, 97 insertions(+)

diff --git a/hw/arm/virt.c b/hw/arm/virt.c
index 4293e01383..15fdf7be05 100644
--- a/hw/arm/virt.c
+++ b/hw/arm/virt.c
@@ -2024,6 +2024,14 @@ static void virt_cpu_post_init(VirtMachineState *vms, MemoryRegion *sysmem)
     }
 }
 
+static void virt_modify_dtb(const struct arm_boot_info *binfo, void *fdt)
+{
+    const VirtMachineState *vms = container_of(binfo, VirtMachineState,
+                                                 bootinfo);
+
+    gunyah_arm_fdt_customize(fdt, vms->memmap[VIRT_MEM].base, vms->gic_phandle);
+}
+
 static void machvirt_init(MachineState *machine)
 {
     VirtMachineState *vms = VIRT_MACHINE(machine);
@@ -2335,6 +2343,9 @@ static void machvirt_init(MachineState *machine)
     vms->bootinfo.skip_dtb_autoload = true;
     vms->bootinfo.firmware_loaded = firmware_loaded;
     vms->bootinfo.psci_conduit = vms->psci_conduit;
+    if (gunyah_enabled()) {
+        vms->bootinfo.modify_dtb = virt_modify_dtb;
+    }
     arm_load_kernel(ARM_CPU(first_cpu), machine, &vms->bootinfo);
 
     vms->machine_done.notify = virt_machine_done;
diff --git a/include/sysemu/gunyah.h b/include/sysemu/gunyah.h
index 101e190619..f077837437 100644
--- a/include/sysemu/gunyah.h
+++ b/include/sysemu/gunyah.h
@@ -35,6 +35,8 @@ DECLARE_INSTANCE_CHECKER(GUNYAHState, GUNYAH_STATE,
                          TYPE_GUNYAH_ACCEL)
 
 int gunyah_arm_set_dtb(__u64 dtb_start, __u64 dtb_size);
+void gunyah_arm_fdt_customize(void *fdt, uint64_t mem_base,
+                uint32_t gic_phandle);
 
 #else   /* CONFIG_GUNYAH_IS_POSSIBLE */
 
@@ -45,6 +47,11 @@ static inline int gunyah_arm_set_dtb(__u64 dtb_start, __u64 dtb_size)
     return -1;
 }
 
+static inline void gunyah_arm_fdt_customize(void *fdt, uint64_t mem_base,
+                uint32_t gic_phandle)
+{
+}
+
 #endif  /* CONFIG_GUNYAH_IS_POSSIBLE */
 
 #endif  /* QEMU_GUNYAH_H */
diff --git a/target/arm/gunyah.c b/target/arm/gunyah.c
index 73c1c2a88a..1521f2d414 100644
--- a/target/arm/gunyah.c
+++ b/target/arm/gunyah.c
@@ -11,6 +11,9 @@
 #include "sysemu/gunyah.h"
 #include "sysemu/gunyah_int.h"
 #include "linux-headers/linux/gunyah.h"
+#include "exec/memory.h"
+#include "sysemu/device_tree.h"
+#include "hw/arm/fdt.h"
 
 /*
  * Specify location of device-tree in guest address space.
@@ -43,3 +46,79 @@ int gunyah_arm_set_dtb(__u64 dtb_start, __u64 dtb_size)
 
     return 0;
 }
+
+void gunyah_arm_fdt_customize(void *fdt, uint64_t mem_base,
+            uint32_t gic_phandle)
+{
+    char *nodename;
+    int i;
+    GUNYAHState *state = get_gunyah_state();
+
+    qemu_fdt_add_subnode(fdt, "/gunyah-vm-config");
+    qemu_fdt_setprop_string(fdt, "/gunyah-vm-config",
+                                "image-name", "qemu-vm");
+    qemu_fdt_setprop_string(fdt, "/gunyah-vm-config", "os-type", "linux");
+
+    nodename = g_strdup_printf("/gunyah-vm-config/memory");
+    qemu_fdt_add_subnode(fdt, nodename);
+    qemu_fdt_setprop_cell(fdt, nodename, "#address-cells", 2);
+    qemu_fdt_setprop_cell(fdt, nodename, "#size-cells", 2);
+    qemu_fdt_setprop_u64(fdt, nodename, "base-address", mem_base);
+
+    g_free(nodename);
+
+    nodename = g_strdup_printf("/gunyah-vm-config/interrupts");
+    qemu_fdt_add_subnode(fdt, nodename);
+    qemu_fdt_setprop_cell(fdt, nodename, "config", gic_phandle);
+    g_free(nodename);
+
+    nodename = g_strdup_printf("/gunyah-vm-config/vcpus");
+    qemu_fdt_add_subnode(fdt, nodename);
+    qemu_fdt_setprop_string(fdt, nodename, "affinity", "proxy");
+    g_free(nodename);
+
+    nodename = g_strdup_printf("/gunyah-vm-config/vdevices");
+    qemu_fdt_add_subnode(fdt, nodename);
+    qemu_fdt_setprop_string(fdt, nodename, "generate", "/hypervisor");
+    g_free(nodename);
+
+    for (i = 0; i < state->nr_slots; ++i) {
+        if (!state->slots[i].start || state->slots[i].lend ||
+                state->slots[i].start == mem_base) {
+            continue;
+        }
+
+        nodename = g_strdup_printf("/gunyah-vm-config/vdevices/shm-%x", i);
+        qemu_fdt_add_subnode(fdt, nodename);
+        qemu_fdt_setprop_string(fdt, nodename, "vdevice-type", "shm");
+        qemu_fdt_setprop_string(fdt, nodename, "push-compatible", "dma");
+        qemu_fdt_setprop(fdt, nodename, "peer-default", NULL, 0);
+        qemu_fdt_setprop_u64(fdt, nodename, "dma_base", 0);
+        g_free(nodename);
+
+        nodename = g_strdup_printf("/gunyah-vm-config/vdevices/shm-%x/memory",
+                                                                        i);
+        qemu_fdt_add_subnode(fdt, nodename);
+        qemu_fdt_setprop_cell(fdt, nodename, "label", i);
+        qemu_fdt_setprop_cell(fdt, nodename, "#address-cells", 2);
+        qemu_fdt_setprop_u64(fdt, nodename, "base", state->slots[i].start);
+        g_free(nodename);
+    }
+
+    for (i = 0; i < state->nr_irqs; ++i) {
+        nodename = g_strdup_printf("/gunyah-vm-config/vdevices/bell-%x", i);
+        qemu_fdt_add_subnode(fdt, nodename);
+        qemu_fdt_setprop_string(fdt, nodename, "vdevice-type", "doorbell");
+        char *p = g_strdup_printf("/hypervisor/bell-%x", i);
+        qemu_fdt_setprop_string(fdt, nodename, "generate", p);
+        g_free(p);
+        qemu_fdt_setprop_cell(fdt, nodename, "label", i);
+        qemu_fdt_setprop(fdt, nodename, "peer-default", NULL, 0);
+        qemu_fdt_setprop(fdt, nodename, "source-can-clear", NULL, 0);
+
+        qemu_fdt_setprop_cells(fdt, nodename, "interrupts",
+                GIC_FDT_IRQ_TYPE_SPI, i, GIC_FDT_IRQ_FLAGS_LEVEL_HI);
+
+        g_free(nodename);
+    }
+}
-- 
2.25.1

Re: [RFC/PATCH v0 09/12] gunyah: Customize device-tree

[Philippe Mathieu-Daudé]

On 11/10/23 18:52, Srivatsa Vaddagiri wrote:
> Customize device-tree with Gunyah specific properties. Some of these
> properties include specification of doorbells that need to be created
> and associated with various interrupts.
> 
> Signed-off-by: Srivatsa Vaddagiri <quic_svaddagi@quicinc.com>
> ---
>   hw/arm/virt.c           | 11 ++++++
>   include/sysemu/gunyah.h |  7 ++++
>   target/arm/gunyah.c     | 79 +++++++++++++++++++++++++++++++++++++++++
>   3 files changed, 97 insertions(+)


> diff --git a/include/sysemu/gunyah.h b/include/sysemu/gunyah.h
> index 101e190619..f077837437 100644
> --- a/include/sysemu/gunyah.h
> +++ b/include/sysemu/gunyah.h
> @@ -35,6 +35,8 @@ DECLARE_INSTANCE_CHECKER(GUNYAHState, GUNYAH_STATE,
>                            TYPE_GUNYAH_ACCEL)
>   
>   int gunyah_arm_set_dtb(__u64 dtb_start, __u64 dtb_size);
> +void gunyah_arm_fdt_customize(void *fdt, uint64_t mem_base,
> +                uint32_t gic_phandle);
>   
>   #else   /* CONFIG_GUNYAH_IS_POSSIBLE */
>   
> @@ -45,6 +47,11 @@ static inline int gunyah_arm_set_dtb(__u64 dtb_start, __u64 dtb_size)
>       return -1;
>   }
>   
> +static inline void gunyah_arm_fdt_customize(void *fdt, uint64_t mem_base,
> +                uint32_t gic_phandle)
> +{

Similar comment than previous patch: stub can be avoided.

> +}
> +
>   #endif  /* CONFIG_GUNYAH_IS_POSSIBLE */
>   
>   #endif  /* QEMU_GUNYAH_H */

[RFC/PATCH v0 10/12] gunyah: CPU execution loop

[Srivatsa Vaddagiri]

Complete the cpu execution loop. At this time, we recognize exits
associated with only MMIO access. Future patches will add support for
recognizing other exit reasons, such as PSCI calls made by guest.

Signed-off-by: Srivatsa Vaddagiri <quic_svaddagi@quicinc.com>
---
 accel/gunyah/gunyah-accel-ops.c |   1 +
 accel/gunyah/gunyah-all.c       | 216 ++++++++++++++++++++++++++++++--
 include/hw/core/cpu.h           |   6 +
 include/sysemu/gunyah.h         |   1 +
 include/sysemu/gunyah_int.h     |   3 +
 target/arm/gunyah.c             |  13 ++
 6 files changed, 229 insertions(+), 11 deletions(-)

diff --git a/accel/gunyah/gunyah-accel-ops.c b/accel/gunyah/gunyah-accel-ops.c
index 77089c4f36..32727e7bf2 100644
--- a/accel/gunyah/gunyah-accel-ops.c
+++ b/accel/gunyah/gunyah-accel-ops.c
@@ -150,6 +150,7 @@ static void gunyah_accel_ops_class_init(ObjectClass *oc, void *data)
 
     ops->create_vcpu_thread = gunyah_start_vcpu_thread;
     ops->kick_vcpu_thread = gunyah_kick_vcpu_thread;
+    ops->synchronize_post_reset = gunyah_cpu_synchronize_post_reset;
 };
 
 static const TypeInfo gunyah_accel_ops_type = {
diff --git a/accel/gunyah/gunyah-all.c b/accel/gunyah/gunyah-all.c
index ea6c74fdc6..edc61e542b 100644
--- a/accel/gunyah/gunyah-all.c
+++ b/accel/gunyah/gunyah-all.c
@@ -25,6 +25,9 @@
 #include "exec/address-spaces.h"
 #include "qapi/error.h"
 #include "qemu/event_notifier.h"
+#include "qemu/main-loop.h"
+#include "sysemu/runstate.h"
+#include "qemu/guest-random.h"
 
 static void gunyah_region_add(MemoryListener *listener,
                            MemoryRegionSection *section);
@@ -69,6 +72,18 @@ int gunyah_vm_ioctl(int type, ...)
     return ioctl(s->vmfd, type, arg);
 }
 
+static int gunyah_vcpu_ioctl(CPUState *cpu, int type, ...)
+{
+    void *arg;
+    va_list ap;
+
+    va_start(ap, type);
+    arg = va_arg(ap, void *);
+    va_end(ap);
+
+    return ioctl(cpu->gunyah.fd, type, arg);
+}
+
 static MemoryListener gunyah_memory_listener = {
     .name = "gunyah",
     .priority = MEMORY_LISTENER_PRIORITY_ACCEL,
@@ -110,17 +125,6 @@ int gunyah_create_vm(void)
     return 0;
 }
 
-void *gunyah_cpu_thread_fn(void *arg)
-{
-    CPUState *cpu = arg;
-
-    do {
-        /* Do nothing */
-    } while (!cpu->unplug || cpu_can_run(cpu));
-
-    return NULL;
-}
-
 #define gunyah_slots_lock(s)    qemu_mutex_lock(&s->slots_lock)
 #define gunyah_slots_unlock(s)  qemu_mutex_unlock(&s->slots_lock)
 
@@ -287,6 +291,11 @@ static void gunyah_set_phys_mem(GUNYAHState *s,
             error_report("Overlapping slot registration not supported!");
             exit(1);
         }
+
+        if (qatomic_read(&s->vm_started)) {
+            error_report("Memory map changes after VM start not supported!");
+            exit(1);
+        }
     }
 
     if (area->readonly ||
@@ -415,3 +424,188 @@ GUNYAHState *get_gunyah_state(void)
 {
     return GUNYAH_STATE(current_accel());
 }
+
+static void gunyah_ipi_signal(int sig)
+{
+    if (current_cpu) {
+        qatomic_set(&current_cpu->gunyah.run->immediate_exit, 1);
+    }
+}
+
+static void gunyah_cpu_kick_self(void)
+{
+    qatomic_set(&current_cpu->gunyah.run->immediate_exit, 1);
+}
+
+static int gunyah_init_vcpu(CPUState *cpu, Error **errp)
+{
+    int ret;
+    struct gh_fn_desc fdesc;
+    struct gh_fn_vcpu_arg vcpu;
+    struct sigaction sigact;
+    sigset_t set;
+
+    /* init cpu signals */
+    memset(&sigact, 0, sizeof(sigact));
+    sigact.sa_handler = gunyah_ipi_signal;
+    sigaction(SIG_IPI, &sigact, NULL);
+
+    pthread_sigmask(SIG_BLOCK, NULL, &set);
+    sigdelset(&set, SIG_IPI);
+
+    ret = pthread_sigmask(SIG_SETMASK, &set, NULL);
+    if (ret) {
+        error_report("pthread_sigmask: %s", strerror(ret));
+        exit(1);
+    }
+
+    vcpu.id = cpu->cpu_index;
+    fdesc.type = GH_FN_VCPU;
+    fdesc.arg_size = sizeof(struct gh_fn_vcpu_arg);
+    fdesc.arg = (__u64)(&vcpu);
+
+    ret = gunyah_vm_ioctl(GH_VM_ADD_FUNCTION, &fdesc);
+    if (ret < 0) {
+        error_report("could not create VCPU %d: %s", vcpu.id, strerror(errno));
+        exit(1);
+    }
+
+    cpu->gunyah.fd = ret;
+    cpu->gunyah.run = mmap(0, 4096, PROT_READ | PROT_WRITE, MAP_SHARED, ret, 0);
+    if (cpu->gunyah.run == MAP_FAILED) {
+        error_report("mmap of vcpu run structure failed : %s", strerror(errno));
+        exit(1);
+    }
+
+    return 0;
+}
+
+static void gunyah_vcpu_destroy(CPUState *cpu)
+{
+    int ret;
+
+    ret = munmap(cpu->gunyah.run, 4096);
+    if (ret < 0) {
+        error_report("munmap of vcpu run structure failed: %s",
+                strerror(errno));
+        exit(1);
+    }
+
+    close(cpu->gunyah.fd);
+}
+
+static int gunyah_vcpu_exec(CPUState *cpu)
+{
+    int ret;
+    GUNYAHState *s = GUNYAH_STATE(current_accel());
+
+    ret = qatomic_cmpxchg(&s->vm_started, 0, 1);
+    if (!ret) {
+        ret = gunyah_vm_ioctl(GH_VM_START);
+        if (ret != 0) {
+            error_report("Failed to start VM: %s", strerror(errno));
+            exit(1);
+        }
+        qatomic_set(&s->vm_started, 1);
+    }
+
+    qemu_mutex_unlock_iothread();
+    cpu_exec_start(cpu);
+
+    do {
+        struct gh_vcpu_run *run = cpu->gunyah.run;
+        int exit_reason, ret;
+
+        if (qatomic_read(&cpu->exit_request)) {
+            gunyah_cpu_kick_self();
+        }
+
+        /* Todo: Check need for smp_rmb() here */
+
+        ret = gunyah_vcpu_ioctl(cpu, GH_VCPU_RUN);
+        if (ret < 0) {
+            if (errno == EINTR || errno == EAGAIN) {
+                qatomic_set(&run->immediate_exit, 0);
+                /* Todo: Check need for smp_wmb() here */
+                ret = EXCP_INTERRUPT;
+                break;
+            }
+
+            error_report("GH_VCPU_RUN: %s", strerror(errno));
+            ret = -1;
+            break;
+        }
+
+        exit_reason = run->exit_reason;
+        switch (exit_reason) {
+        case GH_VCPU_EXIT_MMIO:
+            address_space_rw(&address_space_memory,
+                run->mmio.phys_addr, MEMTXATTRS_UNSPECIFIED,
+                run->mmio.data,
+                run->mmio.len,
+                run->mmio.is_write);
+            break;
+
+        /* Todo: Handle VM shutdown */
+        default:
+            error_report("unhandled exit %d", exit_reason);
+        }
+    } while (ret == 0);
+
+    cpu_exec_end(cpu);
+    qemu_mutex_lock_iothread();
+
+    if (ret < 0) {
+        cpu_dump_state(cpu, stderr, CPU_DUMP_CODE);
+        vm_stop(RUN_STATE_INTERNAL_ERROR);
+    }
+
+    qatomic_set(&cpu->exit_request, 0);
+
+    return ret;
+}
+
+void *gunyah_cpu_thread_fn(void *arg)
+{
+    CPUState *cpu = arg;
+
+    rcu_register_thread();
+
+    qemu_mutex_lock_iothread();
+    qemu_thread_get_self(cpu->thread);
+
+    cpu->thread_id = qemu_get_thread_id();
+    cpu->can_do_io = 1;
+    current_cpu = cpu;
+
+    gunyah_init_vcpu(cpu, &error_fatal);
+
+    /* signal CPU creation */
+    cpu_thread_signal_created(cpu);
+    qemu_guest_random_seed_thread_part2(cpu->random_seed);
+
+    do {
+        if (cpu_can_run(cpu)) {
+            gunyah_vcpu_exec(cpu);
+        }
+        qemu_wait_io_event(cpu);
+    } while (!cpu->unplug || cpu_can_run(cpu));
+
+    gunyah_vcpu_destroy(cpu);
+    cpu_thread_signal_destroyed(cpu);
+    qemu_mutex_unlock_iothread();
+    rcu_unregister_thread();
+    return NULL;
+}
+
+static void do_gunyah_cpu_synchronize_post_reset(CPUState *cpu,
+                                run_on_cpu_data arg)
+{
+    gunyah_arch_put_registers(cpu, 0);
+    cpu->vcpu_dirty = false;
+}
+
+void gunyah_cpu_synchronize_post_reset(CPUState *cpu)
+{
+    run_on_cpu(cpu, do_gunyah_cpu_synchronize_post_reset, RUN_ON_CPU_NULL);
+}
diff --git a/include/hw/core/cpu.h b/include/hw/core/cpu.h
index fdcbe87352..b3901e134d 100644
--- a/include/hw/core/cpu.h
+++ b/include/hw/core/cpu.h
@@ -240,6 +240,7 @@ typedef struct SavedIOTLB {
 
 struct KVMState;
 struct kvm_run;
+struct gh_vcpu_run;
 
 /* work queue */
 
@@ -443,6 +444,11 @@ struct CPUState {
 
     /* track IOMMUs whose translations we've cached in the TCG TLB */
     GArray *iommu_notifiers;
+
+    struct {
+        int fd;
+        struct gh_vcpu_run *run;
+    } gunyah;
 };
 
 typedef QTAILQ_HEAD(CPUTailQ, CPUState) CPUTailQ;
diff --git a/include/sysemu/gunyah.h b/include/sysemu/gunyah.h
index f077837437..c96334502b 100644
--- a/include/sysemu/gunyah.h
+++ b/include/sysemu/gunyah.h
@@ -16,6 +16,7 @@
 
 #ifdef NEED_CPU_H
 #include "cpu.h"
+
 #ifdef CONFIG_GUNYAH
 # define CONFIG_GUNYAH_IS_POSSIBLE
 #endif
diff --git a/include/sysemu/gunyah_int.h b/include/sysemu/gunyah_int.h
index dc5b4847a9..92ba0bc9f0 100644
--- a/include/sysemu/gunyah_int.h
+++ b/include/sysemu/gunyah_int.h
@@ -46,6 +46,7 @@ struct GUNYAHState {
     bool preshmem_reserved;
     uint32_t preshmem_size;
     uint32_t nr_irqs;
+    uint32_t vm_started;
 };
 
 int gunyah_create_vm(void);
@@ -53,5 +54,7 @@ int gunyah_vm_ioctl(int type, ...);
 void *gunyah_cpu_thread_fn(void *arg);
 int gunyah_add_irqfd(int irqfd, int label, Error **errp);
 GUNYAHState *get_gunyah_state(void);
+int gunyah_arch_put_registers(CPUState *cs, int level);
+void gunyah_cpu_synchronize_post_reset(CPUState *cpu);
 
 #endif    /* GUNYAH_INT_H */
diff --git a/target/arm/gunyah.c b/target/arm/gunyah.c
index 1521f2d414..06600dbdb7 100644
--- a/target/arm/gunyah.c
+++ b/target/arm/gunyah.c
@@ -122,3 +122,16 @@ void gunyah_arm_fdt_customize(void *fdt, uint64_t mem_base,
         g_free(nodename);
     }
 }
+
+int gunyah_arch_put_registers(CPUState *cs, int level)
+{
+    /*
+     * No support (yet) to set/get vCPU registers. We specify device-tree
+     * location, which is passed on to VM via X0. Image entry point is assumed
+     * to be the beginning of slot containing device-tree, which seems to be
+     * true currently. In future, Gunyah could add API to set boot CPU's
+     * register context.
+     */
+
+    return 0;
+}
-- 
2.25.1

Re: [RFC/PATCH v0 10/12] gunyah: CPU execution loop

[Philippe Mathieu-Daudé]

On 11/10/23 18:52, Srivatsa Vaddagiri wrote:
> Complete the cpu execution loop. At this time, we recognize exits
> associated with only MMIO access. Future patches will add support for
> recognizing other exit reasons, such as PSCI calls made by guest.
> 
> Signed-off-by: Srivatsa Vaddagiri <quic_svaddagi@quicinc.com>
> ---
>   accel/gunyah/gunyah-accel-ops.c |   1 +
>   accel/gunyah/gunyah-all.c       | 216 ++++++++++++++++++++++++++++++--
>   include/hw/core/cpu.h           |   6 +
>   include/sysemu/gunyah.h         |   1 +
>   include/sysemu/gunyah_int.h     |   3 +
>   target/arm/gunyah.c             |  13 ++
>   6 files changed, 229 insertions(+), 11 deletions(-)


> @@ -110,17 +125,6 @@ int gunyah_create_vm(void)
>       return 0;
>   }
>   
> -void *gunyah_cpu_thread_fn(void *arg)
> -{
> -    CPUState *cpu = arg;
> -
> -    do {
> -        /* Do nothing */
> -    } while (!cpu->unplug || cpu_can_run(cpu));
> -
> -    return NULL;
> -}

This diff could be nicer if you define gunyah_cpu_thread_fn()
in the final place in previous patches, so you don't need to
move it here.

>   #define gunyah_slots_lock(s)    qemu_mutex_lock(&s->slots_lock)
>   #define gunyah_slots_unlock(s)  qemu_mutex_unlock(&s->slots_lock)

[...]

> diff --git a/include/hw/core/cpu.h b/include/hw/core/cpu.h
> index fdcbe87352..b3901e134d 100644
> --- a/include/hw/core/cpu.h
> +++ b/include/hw/core/cpu.h
> @@ -240,6 +240,7 @@ typedef struct SavedIOTLB {
>   
>   struct KVMState;
>   struct kvm_run;
> +struct gh_vcpu_run;
>   
>   /* work queue */
>   
> @@ -443,6 +444,11 @@ struct CPUState {
>   
>       /* track IOMMUs whose translations we've cached in the TCG TLB */
>       GArray *iommu_notifiers;
> +
> +    struct {
> +        int fd;
> +        struct gh_vcpu_run *run;
> +    } gunyah;

NACK. Please declare as struct AccelCPUState in gunyah_int.h.

>   };

Re: [RFC/PATCH v0 10/12] gunyah: CPU execution loop

[Srivatsa Vaddagiri]

* Philippe Mathieu-Daud? <philmd@linaro.org> [2023-10-12 06:43:38]:

> > -void *gunyah_cpu_thread_fn(void *arg)
> > -{
> > -    CPUState *cpu = arg;
> > -
> > -    do {
> > -        /* Do nothing */
> > -    } while (!cpu->unplug || cpu_can_run(cpu));
> > -
> > -    return NULL;
> > -}
> 
> This diff could be nicer if you define gunyah_cpu_thread_fn()
> in the final place in previous patches, so you don't need to
> move it here.

Sure.

> > diff --git a/include/hw/core/cpu.h b/include/hw/core/cpu.h
> > index fdcbe87352..b3901e134d 100644
> > --- a/include/hw/core/cpu.h
> > +++ b/include/hw/core/cpu.h
> > @@ -240,6 +240,7 @@ typedef struct SavedIOTLB {
> >   struct KVMState;
> >   struct kvm_run;
> > +struct gh_vcpu_run;
> >   /* work queue */
> > @@ -443,6 +444,11 @@ struct CPUState {
> >       /* track IOMMUs whose translations we've cached in the TCG TLB */
> >       GArray *iommu_notifiers;
> > +
> > +    struct {
> > +        int fd;
> > +        struct gh_vcpu_run *run;
> > +    } gunyah;
> 
> NACK. Please declare as struct AccelCPUState in gunyah_int.h.

Ok thanks for that input. Will address it next version.

- vatsa

[RFC/PATCH v0 11/12] gunyah: Workarounds (NOT FOR MERGE)

[Srivatsa Vaddagiri]

These are some work-arounds required temporarily until some limitations
with Gunyah hypervisor are addressed.

Signed-off-by: Srivatsa Vaddagiri <quic_svaddagi@quicinc.com>
---
 accel/gunyah/gunyah-all.c   | 18 ++++++++++++++++++
 hw/arm/boot.c               |  3 ++-
 hw/arm/virt.c               |  3 ++-
 include/sysemu/gunyah_int.h |  1 +
 target/arm/gunyah.c         |  7 +++++++
 5 files changed, 30 insertions(+), 2 deletions(-)

diff --git a/accel/gunyah/gunyah-all.c b/accel/gunyah/gunyah-all.c
index edc61e542b..67372a3a82 100644
--- a/accel/gunyah/gunyah-all.c
+++ b/accel/gunyah/gunyah-all.c
@@ -145,6 +145,24 @@ static gunyah_slot *gunyah_find_overlap_slot(GUNYAHState *s,
     return NULL;
 }
 
+gunyah_slot *gunyah_find_slot_by_addr(uint64_t addr)
+{
+    GUNYAHState *s = GUNYAH_STATE(current_accel());
+    int i;
+    gunyah_slot *slot = NULL;
+
+    gunyah_slots_lock(s);
+    for (i = 0; i < s->nr_slots; ++i) {
+        slot = &s->slots[i];
+        if (slot->size &&
+            (addr >= slot->start && addr <= slot->start + slot->size))
+                break;
+    }
+    gunyah_slots_unlock(s);
+
+    return slot;
+}
+
 /* Called with s->slots_lock held */
 static gunyah_slot *gunyah_get_free_slot(GUNYAHState *s)
 {
diff --git a/hw/arm/boot.c b/hw/arm/boot.c
index 720f22531a..72a22c191f 100644
--- a/hw/arm/boot.c
+++ b/hw/arm/boot.c
@@ -413,7 +413,8 @@ static int fdt_add_memory_node(void *fdt, uint32_t acells, hwaddr mem_base,
     char *nodename;
     int ret;
 
-    nodename = g_strdup_printf("/memory@%" PRIx64, mem_base);
+    /* Workaround until RM can parse memory nodes of type memory@XYZ. */
+    nodename = g_strdup_printf("/memory");
     qemu_fdt_add_subnode(fdt, nodename);
     qemu_fdt_setprop_string(fdt, nodename, "device_type", "memory");
     ret = qemu_fdt_setprop_sized_cells(fdt, nodename, "reg", acells, mem_base,
diff --git a/hw/arm/virt.c b/hw/arm/virt.c
index 15fdf7be05..ab4811a785 100644
--- a/hw/arm/virt.c
+++ b/hw/arm/virt.c
@@ -164,7 +164,8 @@ static const MemMapEntry base_memmap[] = {
     [VIRT_PCIE_PIO] =           { 0x3eff0000, 0x00010000 },
     [VIRT_PCIE_ECAM] =          { 0x3f000000, 0x01000000 },
     /* Actual RAM size depends on initial RAM and device memory settings */
-    [VIRT_MEM] =                { GiB, LEGACY_RAMLIMIT_BYTES },
+    /* Workaround until Gunyah can accept mapping that starts from GiB */
+    [VIRT_MEM] =                { 2 * GiB, LEGACY_RAMLIMIT_BYTES },
 };
 
 /*
diff --git a/include/sysemu/gunyah_int.h b/include/sysemu/gunyah_int.h
index 92ba0bc9f0..1880a578b3 100644
--- a/include/sysemu/gunyah_int.h
+++ b/include/sysemu/gunyah_int.h
@@ -56,5 +56,6 @@ int gunyah_add_irqfd(int irqfd, int label, Error **errp);
 GUNYAHState *get_gunyah_state(void);
 int gunyah_arch_put_registers(CPUState *cs, int level);
 void gunyah_cpu_synchronize_post_reset(CPUState *cpu);
+gunyah_slot *gunyah_find_slot_by_addr(uint64_t addr);
 
 #endif    /* GUNYAH_INT_H */
diff --git a/target/arm/gunyah.c b/target/arm/gunyah.c
index 06600dbdb7..55d5ccc2a4 100644
--- a/target/arm/gunyah.c
+++ b/target/arm/gunyah.c
@@ -34,6 +34,13 @@ int gunyah_arm_set_dtb(__u64 dtb_start, __u64 dtb_size)
 {
     int ret;
     struct gh_vm_dtb_config dtb;
+    gunyah_slot *slot = gunyah_find_slot_by_addr(dtb_start);
+
+    /*
+     * RM should consider 'totalsize' field to be inclusive of free space. Use
+     * this workaround until RM is fixed.
+     */
+    dtb_size = slot->start + slot->size - dtb_start;
 
     dtb.guest_phys_addr = dtb_start;
     dtb.size = dtb_size;
-- 
2.25.1

[RFC/PATCH v0 12/12] gunyah: Documentation

[Srivatsa Vaddagiri]

Add gunyah.rst that provide some informaiton on how to build and test
'gunyah' accelerator with open-source Gunyah hypervisor.

Signed-off-by: Srivatsa Vaddagiri <quic_svaddagi@quicinc.com>
---
 MAINTAINERS                |   1 +
 docs/system/arm/gunyah.rst | 214 +++++++++++++++++++++++++++++++++++++
 2 files changed, 215 insertions(+)
 create mode 100644 docs/system/arm/gunyah.rst

diff --git a/MAINTAINERS b/MAINTAINERS
index 5a51633d11..f2933a3683 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -518,6 +518,7 @@ F: target/arm/gunyah.c
 F: include/sysemu/gunyah.h
 F: target/arm/arm_gicv3_gunyah.c
 F: include/sysemu/gunyah_int.h
+F: docs/system/arm/gunyah.rst
 
 WHPX CPUs
 M: Sunil Muthuswamy <sunilmut@microsoft.com>
diff --git a/docs/system/arm/gunyah.rst b/docs/system/arm/gunyah.rst
new file mode 100644
index 0000000000..4b3a792310
--- /dev/null
+++ b/docs/system/arm/gunyah.rst
@@ -0,0 +1,214 @@
+'gunyah' accelerator (``gunyah``)
+=================================
+
+Gunyah is a high performance, scalable and flexible hypervisor built for
+demanding battery powered, real-time, safety and security use cases.
+
+The Gunyah Hypervisor open source project provides a reference Type-1 hypervisor
+configuration suitable for general purpose hosting of multiple trusted and
+dependent VMs. Further information on open-source version of Gunyah Hypervisor
+can be obtained from:
+
+https://github.com/quic/gunyah-hypervisor
+
+To get started with open-source version of Gunyah Hypervisor, refer to the
+instructions available at:
+
+https://github.com/quic/gunyah-support-scripts
+
+Build and testing
+-----------------
+
+Patch and build Gunyah hypervisor
+`````````````````````````````````
+
+Refer to the information present in **Quick Start** guide to get sources for
+Gunyah open-source hypervisor:
+
+https://github.com/quic/gunyah-support-scripts/blob/develop/quickstart.md
+
+Resource Manager VM (RM) is a privileged VM that acts as an extension of Gunyah
+hypervisor. It assists the hypervisor in various tasks related to creation and
+management of VMs. More information on RM is provided at:
+
+https://github.com/quic/gunyah-resource-manager
+
+Instructions provided in **Quick Start** guide will include cloning the sources
+for RM (under 'resource-manager' directory) and hypervisor (under 'hyp'
+directory). These directories will be found inside the docker image created as
+part of steps provided under **Building the Docker Image**.
+
+Before building hypervisor, as described under **Building a Gunyah Hypervisor
+image** section of **Quick Start** guide, apply below changes to hypervisor and
+RM on which 'gunyah' accelerator currently depends. These changes are being
+discussed with maintainers and if accepted this document will be modified
+appropriately.
+
+RM patch (in 'resource-manager' directory):
+
+.. code-block:: bash
+
+	diff --git a/src/vm_creation/vm_creation.c b/src/vm_creation/vm_creation.c
+	index df8edfb..b73b37e 100644
+	--- a/src/vm_creation/vm_creation.c
+	+++ b/src/vm_creation/vm_creation.c
+	@@ -510,7 +510,10 @@ process_dtb(vm_t *vm)
+	        // Estimate a final dtb size after applying the overlay.
+	        size_t original_dtb_size =
+	                util_balign_up(fdt_totalsize(temp_addr), sizeof(uint32_t));
+	-       size_t final_dtb_size = original_dtb_size + dtbo_ret.size;
+	+       size_t final_dtb_size = util_balign_up(original_dtb_size + dtbo_ret.size, 8);
+
+
+Hypervisor patch (in 'hyp' directory):
+
+.. code-block:: bash
+
+	diff --git a/config/platform/qemu.conf b/config/platform/qemu.conf
+	index bc612f2..9a292a4 100644
+	--- a/config/platform/qemu.conf
+	+++ b/config/platform/qemu.conf
+	@@ -35,7 +35,7 @@ configs HLOS_RAM_FS_BASE=0x40800000
+	 configs PLATFORM_HEAP_PRIVATE_SIZE=0x200000
+	 configs PLATFORM_RW_DATA_SIZE=0x200000
+	 configs PLATFORM_ROOTVM_LMA_BASE=0x80480000U
+	-configs PLATFORM_ROOTVM_LMA_SIZE=0xa0000U
+	+configs PLATFORM_ROOTVM_LMA_SIZE=0x100000U
+	 configs PLATFORM_PHYS_ADDRESS_BITS=36
+	 configs PLATFORM_VM_ADDRESS_SPACE_BITS=36
+	 configs PLATFORM_PGTABLE_4K_GRANULE=1
+
+Build Gunyah hypervisor (including RM) as per the instructions provided in
+**Quick Start** guide, viz:
+
+.. code-block:: bash
+
+        cd ~/share
+        build-gunyah.sh qemu
+
+
+Patch and build Linux kernel
+````````````````````````````
+
+**Building the Docker Image** step of **Quick Start** guide would have cloned
+the sources for Linux kernel (under '~/linux/src/linux' directory in docker
+image) and also applied below patches.
+
+https://lore.kernel.org/lkml/20230613172054.3959700-1-quic_eberman@quicinc.com/
+
+Above patches provide a kernel driver that interacts with Gunyah hypervisor and
+assists a VMM (like Qemu) in creation and management of a VM. Apply below patch
+to Linux kernel, as the 'gunyah' accelerator in Qemu depends on it.
+
+.. code-block:: bash
+
+	diff --git a/drivers/virt/gunyah/gunyah_ioeventfd.c b/drivers/virt/gunyah/gunyah_ioeventfd.c
+	index 5b1b9fd9a..7bfb67800 100644
+	--- a/drivers/virt/gunyah/gunyah_ioeventfd.c
+	+++ b/drivers/virt/gunyah/gunyah_ioeventfd.c
+	@@ -88,6 +88,7 @@ static long gh_ioeventfd_bind(struct gh_vm_function_instance *f)
+	        }
+	        iofd->io_handler.addr = args->addr;
+	        iofd->io_handler.ops = &io_ops;
+	+       iofd->io_handler.fd = args->fd;
+
+	        ret = gh_vm_add_io_handler(f->ghvm, &iofd->io_handler);
+	        if (ret)
+	@@ -120,7 +121,7 @@ static bool gh_ioevent_compare(const struct gh_vm_function_instance *f,
+	        if (sizeof(*other) != size)
+	                return false;
+
+	-       return instance->addr == other->addr;
+	+       return instance->addr == other->addr && instance->fd == other->fd;
+	 }
+
+	 DECLARE_GH_VM_FUNCTION_INIT(ioeventfd, GH_FN_IOEVENTFD, 3,
+	diff --git a/drivers/virt/gunyah/vm_mgr.c b/drivers/virt/gunyah/vm_mgr.c
+	index afea49238..c4604d4e9 100644
+	--- a/drivers/virt/gunyah/vm_mgr.c
+	+++ b/drivers/virt/gunyah/vm_mgr.c
+	@@ -291,6 +291,10 @@ static int _gh_vm_io_handler_compare(const struct rb_node *node, const struct rb
+	                return -1;
+	        if (n->len > p->len)
+	                return 1;
+	+       if (n->fd > p->fd)
+	+               return 1;
+	+       if (n->fd < p->fd)
+	+               return -1;
+	        /* one of the io handlers doesn't have datamatch and the other does.
+	         * For purposes of comparison, that makes them identical since the
+	         * one that doesn't have datamatch will cover the same handler that
+	diff --git a/include/linux/gunyah_vm_mgr.h b/include/linux/gunyah_vm_mgr.h
+	index 527e94624..b2e95d02c 100644
+	--- a/include/linux/gunyah_vm_mgr.h
+	+++ b/include/linux/gunyah_vm_mgr.h
+	@@ -136,6 +136,7 @@ struct gh_vm_io_handler {
+	        bool datamatch;
+	        u8 len;
+	        u64 data;
+	+       int fd;
+	        struct gh_vm_io_handler_ops *ops;
+	 };
+
+Run 'build-linux.sh' inside the docker container to build Linux kernel, which
+provides the image for host kernel.
+
+.. code-block:: bash
+
+        $ cd ~/
+        $ build-linux.sh
+
+
+This will result in Linux kernel binary (Image) and an initrd image (initrd.img)
+being created and copied to '~/tools/qemu/imgs' directory. Those images can be
+used for both host kernel as well as for a VM.
+
+
+Configure and build Qemu
+````````````````````````
+
+Apply the proposed patches for 'gunyah' accelerator support in Qemu and build
+it.
+
+.. code-block:: bash
+
+        $ ./configure --target-list=aarch64-softmmu --enable-debug --enable-gunyah --static
+        $ make -j4
+
+Incorporate Qemu image in root disk
+```````````````````````````````````
+
+Copy over the resulting Qemu static binary ('qemu-system-aarch64') into the
+docker image. In addition, copy 'efi-virtio.rom' and 'en-us' (from Qemu
+repository) into the docker image.
+
+Follow the instructions under **SVM booting Linux** to prepare a virtual disk
+that can host the required images. Since Qemu is going to be used in place of
+CrosVM, ignore the CrosVM related steps, and instead copy these files to
+'~/share/docker-share/rootfs/usr/gunyah' folder before running
+'~/utils/bldextfs.sh' script to build root image.
+
+qemu-system-aarch64, efi-virtio.rom, en-us, Image, initrd.img
+
+The root disk created ('rootfs-extfs-disk.img') will have above files under
+/usr/gunyah folder.
+
+Running VM
+``````````
+
+.. code-block:: bash
+
+        $ cd /usr/gunyah
+        $ ./qemu-system-aarch64 -cpu cortex-a57 -nographic -m 256M --accel gunyah \
+        -machine virt,highmem=off -initrd ./initrd.img -append \
+        "rw root=/dev/ram rdinit=/sbin/init earlyprintk=serial panic=0"
+        -kernel ./Image
+
+Limitations
+-----------
+
+Below features are not yet supported.
+
+* SMP (PSCI support needs to be added)
+* virtio-pci (support for which in Qemu seems to rely heavily on KVM, which
+  needs to be made multi-hypervisor friendly).
-- 
2.25.1

Re: [RFC/PATCH v0 12/12] gunyah: Documentation

[Philippe Mathieu-Daudé]

Hi Srivatsa,

On 11/10/23 18:52, Srivatsa Vaddagiri wrote:
> Add gunyah.rst that provide some informaiton on how to build and test
> 'gunyah' accelerator with open-source Gunyah hypervisor.
> 
> Signed-off-by: Srivatsa Vaddagiri <quic_svaddagi@quicinc.com>
> ---
>   MAINTAINERS                |   1 +
>   docs/system/arm/gunyah.rst | 214 +++++++++++++++++++++++++++++++++++++
>   2 files changed, 215 insertions(+)
>   create mode 100644 docs/system/arm/gunyah.rst


> +Limitations
> +-----------
> +
> +Below features are not yet supported.
> +
> +* SMP (PSCI support needs to be added)
> +* virtio-pci (support for which in Qemu seems to rely heavily on KVM, which
> +  needs to be made multi-hypervisor friendly).

Is QUIC interested in working on that? My work introducing
AccelCPUState to make CPUState accel-agnostic was done in that
direction, but I didn't notice much interest in the community
so I unprioritized it.

Regards,

Phil.

Re: [RFC/PATCH v0 12/12] gunyah: Documentation

[Srivatsa Vaddagiri]

* Philippe Mathieu-Daud? <philmd@linaro.org> [2023-10-12 06:52:04]:

> > +Limitations
> > +-----------
> > +
> > +Below features are not yet supported.
> > +
> > +* virtio-pci (support for which in Qemu seems to rely heavily on KVM, which
> > +  needs to be made multi-hypervisor friendly).
> 
> Is QUIC interested in working on that? My work introducing
> AccelCPUState to make CPUState accel-agnostic was done in that
> direction, but I didn't notice much interest in the community
> so I unprioritized it.

Hi Phil,
	We do want to see Gunyah support merged in Qemu at the earliest (as soon
as the kernel driver is merged upstream that is), so any dependent change in
Qemu for Gunyah would be of much interest to us! I am not sure though if Quic
can sign up for the entire "make cpustate accel agnostic" work. Can you point
to your ongoing work that I could take a look at? Would that address virtio-pci
becoming accelerator agnostic? 

Thanks!
vatsa

Re: [RFC/PATCH v0 12/12] gunyah: Documentation

[Alex Bennée]

Srivatsa Vaddagiri <quic_svaddagi@quicinc.com> writes:

> * Philippe Mathieu-Daud? <philmd@linaro.org> [2023-10-12 06:52:04]:
>
>> > +Limitations
>> > +-----------
>> > +
>> > +Below features are not yet supported.
>> > +
>> > +* virtio-pci (support for which in Qemu seems to rely heavily on KVM, which
>> > +  needs to be made multi-hypervisor friendly).
>> 
>> Is QUIC interested in working on that? My work introducing
>> AccelCPUState to make CPUState accel-agnostic was done in that
>> direction, but I didn't notice much interest in the community
>> so I unprioritized it.
>
> Hi Phil,
> 	We do want to see Gunyah support merged in Qemu at the earliest (as soon
> as the kernel driver is merged upstream that is), so any dependent change in
> Qemu for Gunyah would be of much interest to us! I am not sure though if Quic
> can sign up for the entire "make cpustate accel agnostic" work. Can you point
> to your ongoing work that I could take a look at? Would that address virtio-pci
> becoming accelerator agnostic?

Why wouldn't virtio-pci be accelerator agnostic?

It works with KVM and TCG for example. There are I suppose complications
if you want to share emulated devices with a real PCI bus rather than
emulating it all within QEMU.

>
> Thanks!
> vatsa


-- 
Alex Bennée
Virtualisation Tech Lead @ Linaro

Re: [RFC/PATCH v0 12/12] gunyah: Documentation

[Srivatsa Vaddagiri]

* Alex Benn?e <alex.bennee@linaro.org> [2023-10-12 15:55:59]:

> > Hi Phil,
> > 	We do want to see Gunyah support merged in Qemu at the earliest (as soon
> > as the kernel driver is merged upstream that is), so any dependent change in
> > Qemu for Gunyah would be of much interest to us! I am not sure though if Quic
> > can sign up for the entire "make cpustate accel agnostic" work. Can you point
> > to your ongoing work that I could take a look at? Would that address virtio-pci
> > becoming accelerator agnostic?
> 
> Why wouldn't virtio-pci be accelerator agnostic?

I checked usage of few KVM APIs in virtio-pci.c. I think most of them are to do
with use of MSI and IRQFD. If lets say we are not supporting MSI, then I *think*
current virtio-pci should work just fine. It would use virtio_pci_notify ->
pci_set_irq -> .. -> qemu_set_irq, which should land in
gunyah_arm_gicv3_set_irq [Patch 7/12] AFAICT. Let me try getting virtio-pci
working and then I can update this thread again!

- vatsa

Re: [RFC/PATCH v0 12/12] gunyah: Documentation

[Alex Bennée]

Srivatsa Vaddagiri <quic_svaddagi@quicinc.com> writes:

(add VirtIO maintainer MST to CC)

> * Alex Benn?e <alex.bennee@linaro.org> [2023-10-12 15:55:59]:
>
>> > Hi Phil,
>> > 	We do want to see Gunyah support merged in Qemu at the earliest (as soon
>> > as the kernel driver is merged upstream that is), so any dependent change in
>> > Qemu for Gunyah would be of much interest to us! I am not sure though if Quic
>> > can sign up for the entire "make cpustate accel agnostic" work. Can you point
>> > to your ongoing work that I could take a look at? Would that address virtio-pci
>> > becoming accelerator agnostic?
>> 
>> Why wouldn't virtio-pci be accelerator agnostic?
>
> I checked usage of few KVM APIs in virtio-pci.c. I think most of them are to do
> with use of MSI and IRQFD. If lets say we are not supporting MSI, then I *think*
> current virtio-pci should work just fine. It would use virtio_pci_notify ->
> pci_set_irq -> .. -> qemu_set_irq, which should land in
> gunyah_arm_gicv3_set_irq [Patch 7/12] AFAICT. Let me try getting virtio-pci
> working and then I can update this thread again!

Hmm yeah looking at the file the relationship between KVM and virtio-pci
is a bit tangled up. Fundamentally the reason I say virtio-pci should be
accelerator agnostic is that we use virtio-pci under TCG emulation with
no KVM at all. IOW if all of the PCI emulation is done within QEMU there
should be no limit to which accelerators support it.

(warning! potentially incomplete understanding ahead)

However as you have seen there is an optimisation where KVM can take
over some of the PCI bus emulation and instead of synchronous
trap-and-exit to QEMU simply queues MSIs on irqfd which QEMU can then
consume the events and behave appropriately. I'm not sure what happens
with real PCI buses and pass-through sitting alongside virtual PCI
devices.

I suspect other hypervisors might want to support a similar thing and
even might end up re-using the irqfd mechanism for signalling events
from the hypervisor to the VMM. So Philippe's suggestion to fix this
would be to properly abstract whats needed to set this up and then clean
up the hardwired case with a generalised class representing the
in-[kernel|hypervisor] interface. Gunyah could then provide an
implementation of that interface if it ever supports injecting MSIs
directly.

-- 
Alex Bennée
Virtualisation Tech Lead @ Linaro

Re: [RFC/PATCH v0 00/12] Gunyah hypervisor support

[Paolo Bonzini]

On 10/11/23 18:52, Srivatsa Vaddagiri wrote:
> Gunyah is an open-source Type-1 hypervisor, that is currently supported on ARM64
> architecture. Source code for it can be obtained from:
> 
> https://github.com/quic/gunyah-hypervisor.
> 
> This patch series adds support for Gunyah hypervisor via a new
> accelerator option, 'gunyah'. This patch series is based on the Linux kernel's
> Gunyah driver, which is being actively developed and not yet merged upstream
> [1].
> 
> This patch series is thus *NOT YET READY* for merge. Early version of this patch
> is being published to solicit comments from Qemu community.
> 
> This patch has been tested with the open-source version of Gunyah hypervisor.
> Instructions to build hypervisor and test this patch are provided in this
> patch series.
> 
> Limitations:
> 
> 1) SMP is not yet supported. This restriction will be removed in the next version
> of this patch series.
> 
> 2) virtio-pci is not yet supported. hw/virtio/virtio-pci.c seems to support only
> KVM and would need changes to support other hypervisors. If there is any ongoing
> work in this regard, would like to build upon it and add support for Gunyah
> hypervisor.

virtio-pci does support other hypervisors; what is not yet supported is 
cross-hypervisor support for IRQFD[1].  This is more of a QEMU issue 
than something specific to virtio-pci.  The way to fix it is to add 
wrappers for kvm_irqchip_add_irqfd_notifier_gsi and 
kvm_irqchip_remove_irqfd_notifier_gsi to AccelOpsClass, and move the 
stub implementations in accel/stubs/kvm-stub.c to the abstract 
superclass.  Then you can switch the users of these KVM functions to the 
AccelOpsClass wrapper.

By the way I've just posted a series to remove support for really really 
old kernels, and this should remove more KVM-specific code from device 
models.

Paolo

[1] a Linux eventfd (or QEMU EventNotifier) that triggers a guest 
interrupt when it becomes readable.  QEMU writes to the eventfd and the 
kernel responds by injecting the interrupt