[RFC/PATCH v1 00/11] Gunyah hypervisor support

[RFC/PATCH v1 00/11] Gunyah hypervisor support

[Srivatsa Vaddagiri]

Gunyah is an open-source Type-1 hypervisor, that is currently supported on ARM64
architecture. Source code for it can be obtained from:

https://github.com/quic/gunyah-hypervisor

This patch series adds support for Gunyah hypervisor via a new
accelerator option, 'gunyah'. This patch series is based on the Linux kernel's
Gunyah driver, which is being actively developed and not yet merged upstream
[1].

This patch series is thus *NOT YET READY* for merge. Early version of this patch
is being published to solicit comments from Qemu community.

This patch has been tested with the open-source version of Gunyah hypervisor.
Instructions to build hypervisor and test this patch are provided in this
patch series.

Changes in v1:

* Fixed SMP boot issues
* Addressed comments received for previous version (v0) of the patches series

Limitations:

Confidential guests (or protected VMs) are not yet supported.

Prior version, v0, of this patch can be referenced at:

	https://lists.nongnu.org/archive/html/qemu-devel/2023-10/msg03202.html

Ref:

1. https://lore.kernel.org/lkml/20230613172054.3959700-1-quic_eberman@quicinc.com/

Base commit on which this series was tested:

0c1eccd368 Merge tag 'hw-cpus-20240105' of https://github.com/philmd/qemu into staging


Srivatsa Vaddagiri (11):
  gunyah: UAPI header (NOT FOR MERGE)
  gunyah: Basic support
  gunyah: Add VM properties
  gunyah: Support memory assignment
  gunyah: Add IRQFD and IOEVENTFD functions
  gunyah: Add gicv3 interrupt controller
  gunyah: Specific device-tree location
  gunyah: Customize device-tree
  gunyah: CPU execution loop
  gunyah: Workarounds (NOT FOR MERGE)
  gunyah: Documentation

 MAINTAINERS                     |  11 +
 accel/Kconfig                   |   3 +
 accel/gunyah/gunyah-accel-ops.c | 180 +++++++++
 accel/gunyah/gunyah-all.c       | 634 ++++++++++++++++++++++++++++++++
 accel/gunyah/meson.build        |   7 +
 accel/meson.build               |   1 +
 accel/stubs/gunyah-stub.c       |  23 ++
 accel/stubs/meson.build         |   1 +
 docs/about/build-platforms.rst  |   2 +-
 docs/system/arm/gunyah.rst      | 358 ++++++++++++++++++
 hw/arm/boot.c                   |   3 +-
 hw/arm/virt.c                   |  28 +-
 hw/intc/arm_gicv3_common.c      |   3 +
 hw/intc/arm_gicv3_gunyah.c      | 106 ++++++
 hw/intc/arm_gicv3_its_common.c  |   3 +
 hw/intc/meson.build             |   1 +
 include/hw/core/cpu.h           |   1 +
 include/sysemu/gunyah.h         |  34 ++
 include/sysemu/gunyah_int.h     |  67 ++++
 linux-headers/linux/gunyah.h    | 311 ++++++++++++++++
 meson.build                     |  12 +-
 meson_options.txt               |   2 +
 scripts/meson-buildoptions.sh   |   3 +
 target/arm/cpu.c                |   3 +-
 target/arm/cpu64.c              |   5 +-
 target/arm/gunyah.c             | 144 ++++++++
 target/arm/meson.build          |   3 +
 27 files changed, 1942 insertions(+), 7 deletions(-)
 create mode 100644 accel/gunyah/gunyah-accel-ops.c
 create mode 100644 accel/gunyah/gunyah-all.c
 create mode 100644 accel/gunyah/meson.build
 create mode 100644 accel/stubs/gunyah-stub.c
 create mode 100644 docs/system/arm/gunyah.rst
 create mode 100644 hw/intc/arm_gicv3_gunyah.c
 create mode 100644 include/sysemu/gunyah.h
 create mode 100644 include/sysemu/gunyah_int.h
 create mode 100644 linux-headers/linux/gunyah.h
 create mode 100644 target/arm/gunyah.c

-- 
2.25.1

[RFC/PATCH v1 01/11] gunyah: UAPI header (NOT FOR MERGE)

[Srivatsa Vaddagiri]

Gunyah UAPI header file is provided for ease of use, until Gunyah kernel
driver is merged upstream. scripts/update-linux-headers.sh will be
updated via a separate patch after that for Qemu to pick up gunyah.h
UAPI header from Linux kernel sources.

This header file is based on the Gunyah driver present in Android-14.

https://android.googlesource.com/kernel/common
Branch: android14-6.1

Signed-off-by: Srivatsa Vaddagiri <quic_svaddagi@quicinc.com>
---
 linux-headers/linux/gunyah.h | 311 +++++++++++++++++++++++++++++++++++
 1 file changed, 311 insertions(+)
 create mode 100644 linux-headers/linux/gunyah.h

diff --git a/linux-headers/linux/gunyah.h b/linux-headers/linux/gunyah.h
new file mode 100644
index 0000000000..c8d250a834
--- /dev/null
+++ b/linux-headers/linux/gunyah.h
@@ -0,0 +1,311 @@
+/* SPDX-License-Identifier: GPL-2.0-only WITH Linux-syscall-note */
+/*
+ * Copyright (c) 2022-2023 Qualcomm Innovation Center, Inc. All rights reserved.
+ */
+
+#ifndef _LINUX_GUNYAH_H
+#define _LINUX_GUNYAH_H
+
+/*
+ * Userspace interface for /dev/gunyah - gunyah based virtual machine
+ */
+
+#include <linux/types.h>
+#include <linux/ioctl.h>
+
+#define GH_IOCTL_TYPE			'G'
+
+/*
+ * ioctls for /dev/gunyah fds:
+ */
+#define GH_CREATE_VM			_IO(GH_IOCTL_TYPE, 0x0) /* Returns a Gunyah VM fd */
+
+/*
+ * ioctls for VM fds
+ */
+
+/**
+ * enum gh_mem_flags - Possible flags on &struct gh_userspace_memory_region
+ * @GH_MEM_ALLOW_READ: Allow guest to read the memory
+ * @GH_MEM_ALLOW_WRITE: Allow guest to write to the memory
+ * @GH_MEM_ALLOW_EXEC: Allow guest to execute instructions in the memory
+ */
+enum gh_mem_flags {
+	GH_MEM_ALLOW_READ	= 1UL << 0,
+	GH_MEM_ALLOW_WRITE	= 1UL << 1,
+	GH_MEM_ALLOW_EXEC	= 1UL << 2,
+};
+
+/**
+ * struct gh_userspace_memory_region - Userspace memory descripion for GH_VM_SET_USER_MEM_REGION
+ * @label: Identifer to the region which is unique to the VM.
+ * @flags: Flags for memory parcel behavior. See &enum gh_mem_flags.
+ * @guest_phys_addr: Location of the memory region in guest's memory space (page-aligned)
+ * @memory_size: Size of the region (page-aligned)
+ * @userspace_addr: Location of the memory region in caller (userspace)'s memory
+ *
+ * See Documentation/virt/gunyah/vm-manager.rst for further details.
+ */
+struct gh_userspace_memory_region {
+	__u32 label;
+	__u32 flags;
+	__u64 guest_phys_addr;
+	__u64 memory_size;
+	__u64 userspace_addr;
+};
+
+#define GH_VM_SET_USER_MEM_REGION	_IOW(GH_IOCTL_TYPE, 0x1, \
+						struct gh_userspace_memory_region)
+
+/**
+ * struct gh_vm_dtb_config - Set the location of the VM's devicetree blob
+ * @guest_phys_addr: Address of the VM's devicetree in guest memory.
+ * @size: Maximum size of the devicetree including space for overlays.
+ *        Resource manager applies an overlay to the DTB and dtb_size should
+ *        include room for the overlay. A page of memory is typicaly plenty.
+ */
+struct gh_vm_dtb_config {
+	__u64 guest_phys_addr;
+	__u64 size;
+};
+#define GH_VM_SET_DTB_CONFIG	_IOW(GH_IOCTL_TYPE, 0x2, struct gh_vm_dtb_config)
+
+#define GH_VM_START		_IO(GH_IOCTL_TYPE, 0x3)
+
+/**
+ * enum gh_fn_type - Valid types of Gunyah VM functions
+ * @GH_FN_VCPU: create a vCPU instance to control a vCPU
+ *              &struct gh_fn_desc.arg is a pointer to &struct gh_fn_vcpu_arg
+ *              Return: file descriptor to manipulate the vcpu.
+ * @GH_FN_IRQFD: register eventfd to assert a Gunyah doorbell
+ *               &struct gh_fn_desc.arg is a pointer to &struct gh_fn_irqfd_arg
+ * @GH_FN_IOEVENTFD: register ioeventfd to trigger when VM faults on parameter
+ *                   &struct gh_fn_desc.arg is a pointer to &struct gh_fn_ioeventfd_arg
+ */
+enum gh_fn_type {
+	GH_FN_VCPU = 1,
+	GH_FN_IRQFD,
+	GH_FN_IOEVENTFD,
+};
+
+#define GH_FN_MAX_ARG_SIZE		256
+
+/**
+ * struct gh_fn_vcpu_arg - Arguments to create a vCPU.
+ * @id: vcpu id
+ *
+ * Create this function with &GH_VM_ADD_FUNCTION using type &GH_FN_VCPU.
+ *
+ * The vcpu type will register with the VM Manager to expect to control
+ * vCPU number `vcpu_id`. It returns a file descriptor allowing interaction with
+ * the vCPU. See the Gunyah vCPU API description sections for interacting with
+ * the Gunyah vCPU file descriptors.
+ */
+struct gh_fn_vcpu_arg {
+	__u32 id;
+};
+
+/**
+ * enum gh_irqfd_flags - flags for use in gh_fn_irqfd_arg
+ * @GH_IRQFD_FLAGS_LEVEL: make the interrupt operate like a level triggered
+ *                        interrupt on guest side. Triggering IRQFD before
+ *                        guest handles the interrupt causes interrupt to
+ *                        stay asserted.
+ */
+enum gh_irqfd_flags {
+	GH_IRQFD_FLAGS_LEVEL		= 1UL << 0,
+};
+
+/**
+ * struct gh_fn_irqfd_arg - Arguments to create an irqfd function.
+ *
+ * Create this function with &GH_VM_ADD_FUNCTION using type &GH_FN_IRQFD.
+ *
+ * Allows setting an eventfd to directly trigger a guest interrupt.
+ * irqfd.fd specifies the file descriptor to use as the eventfd.
+ * irqfd.label corresponds to the doorbell label used in the guest VM's devicetree.
+ *
+ * @fd: an eventfd which when written to will raise a doorbell
+ * @label: Label of the doorbell created on the guest VM
+ * @flags: see &enum gh_irqfd_flags
+ * @padding: padding bytes
+ */
+struct gh_fn_irqfd_arg {
+	__u32 fd;
+	__u32 label;
+	__u32 flags;
+	__u32 padding;
+};
+
+/**
+ * enum gh_ioeventfd_flags - flags for use in gh_fn_ioeventfd_arg
+ * @GH_IOEVENTFD_FLAGS_DATAMATCH: the event will be signaled only if the
+ *                                written value to the registered address is
+ *                                equal to &struct gh_fn_ioeventfd_arg.datamatch
+ */
+enum gh_ioeventfd_flags {
+	GH_IOEVENTFD_FLAGS_DATAMATCH	= 1UL << 0,
+};
+
+/**
+ * struct gh_fn_ioeventfd_arg - Arguments to create an ioeventfd function
+ * @datamatch: data used when GH_IOEVENTFD_DATAMATCH is set
+ * @addr: Address in guest memory
+ * @len: Length of access
+ * @fd: When ioeventfd is matched, this eventfd is written
+ * @flags: See &enum gh_ioeventfd_flags
+ * @padding: padding bytes
+ *
+ * Create this function with &GH_VM_ADD_FUNCTION using type &GH_FN_IOEVENTFD.
+ *
+ * Attaches an ioeventfd to a legal mmio address within the guest. A guest write
+ * in the registered address will signal the provided event instead of triggering
+ * an exit on the GH_VCPU_RUN ioctl.
+ */
+struct gh_fn_ioeventfd_arg {
+	__u64 datamatch;
+	__u64 addr;        /* legal mmio address */
+	__u32 len;         /* 1, 2, 4, or 8 bytes; or 0 to ignore length */
+	__s32 fd;
+	__u32 flags;
+	__u32 padding;
+};
+
+/**
+ * struct gh_fn_desc - Arguments to create a VM function
+ * @type: Type of the function. See &enum gh_fn_type.
+ * @arg_size: Size of argument to pass to the function. arg_size <= GH_FN_MAX_ARG_SIZE
+ * @arg: Pointer to argument given to the function. See &enum gh_fn_type for expected
+ *       arguments for a function type.
+ */
+struct gh_fn_desc {
+	__u32 type;
+	__u32 arg_size;
+	__u64 arg;
+};
+
+#define GH_VM_ADD_FUNCTION	_IOW(GH_IOCTL_TYPE, 0x4, struct gh_fn_desc)
+#define GH_VM_REMOVE_FUNCTION	_IOW(GH_IOCTL_TYPE, 0x7, struct gh_fn_desc)
+
+/*
+ * ioctls for vCPU fds
+ */
+
+/**
+ * enum gh_vm_status - Stores status reason why VM is not runnable (exited).
+ * @GH_VM_STATUS_LOAD_FAILED: VM didn't start because it couldn't be loaded.
+ * @GH_VM_STATUS_EXITED: VM requested shutdown/reboot.
+ *                       Use &struct gh_vm_exit_info.reason for further details.
+ * @GH_VM_STATUS_CRASHED: VM state is unknown and has crashed.
+ */
+enum gh_vm_status {
+	GH_VM_STATUS_LOAD_FAILED	= 1,
+	GH_VM_STATUS_EXITED		= 2,
+	GH_VM_STATUS_CRASHED		= 3,
+};
+
+/*
+ * Gunyah presently sends max 4 bytes of exit_reason.
+ * If that changes, this macro can be safely increased without breaking
+ * userspace so long as struct gh_vcpu_run < PAGE_SIZE.
+ */
+#define GH_VM_MAX_EXIT_REASON_SIZE	8u
+
+/**
+ * struct gh_vm_exit_info - Reason for VM exit as reported by Gunyah
+ * See Gunyah documentation for values.
+ * @type: Describes how VM exited
+ * @padding: padding bytes
+ * @reason_size: Number of bytes valid for `reason`
+ * @reason: See Gunyah documentation for interpretation. Note: these values are
+ *          not interpreted by Linux and need to be converted from little-endian
+ *          as applicable.
+ */
+struct gh_vm_exit_info {
+	__u16 type;
+	__u16 padding;
+	__u32 reason_size;
+	__u8 reason[GH_VM_MAX_EXIT_REASON_SIZE];
+};
+
+/**
+ * enum gh_vcpu_exit - Stores reason why &GH_VCPU_RUN ioctl recently exited with status 0
+ * @GH_VCPU_EXIT_UNKNOWN: Not used, status != 0
+ * @GH_VCPU_EXIT_MMIO: vCPU performed a read or write that could not be handled
+ *                     by hypervisor or Linux. Use @struct gh_vcpu_run.mmio for
+ *                     details of the read/write.
+ * @GH_VCPU_EXIT_STATUS: vCPU not able to run because the VM has exited.
+ *                       Use @struct gh_vcpu_run.status for why VM has exited.
+ */
+enum gh_vcpu_exit {
+	GH_VCPU_EXIT_UNKNOWN,
+	GH_VCPU_EXIT_MMIO,
+	GH_VCPU_EXIT_STATUS,
+};
+
+/**
+ * struct gh_vcpu_run - Application code obtains a pointer to the gh_vcpu_run
+ *                      structure by mmap()ing a vcpu fd.
+ * @immediate_exit: polled when scheduling the vcpu. If set, immediately returns -EINTR.
+ * @padding: padding bytes
+ * @exit_reason: Set when GH_VCPU_RUN returns successfully and gives reason why
+ *               GH_VCPU_RUN has stopped running the vCPU. See &enum gh_vcpu_exit.
+ * @mmio: Used when exit_reason == GH_VCPU_EXIT_MMIO
+ *        The guest has faulted on an memory-mapped I/O instruction that
+ *        couldn't be satisfied by gunyah.
+ * @mmio.phys_addr: Address guest tried to access
+ * @mmio.data: the value that was written if `is_write == 1`. Filled by
+ *        user for reads (`is_write == 0`).
+ * @mmio.len: Length of write. Only the first `len` bytes of `data`
+ *       are considered by Gunyah.
+ * @mmio.is_write: 1 if VM tried to perform a write, 0 for a read
+ * @status: Used when exit_reason == GH_VCPU_EXIT_STATUS.
+ *          The guest VM is no longer runnable. This struct informs why.
+ * @status.status: See &enum gh_vm_status for possible values
+ * @status.exit_info: Used when status == GH_VM_STATUS_EXITED
+ */
+struct gh_vcpu_run {
+	/* in */
+	__u8 immediate_exit;
+	__u8 padding[7];
+
+	/* out */
+	__u32 exit_reason;
+
+	union {
+		struct {
+			__u64 phys_addr;
+			__u8  data[8];
+			__u32 len;
+			__u8  is_write;
+		} mmio;
+
+		struct {
+			enum gh_vm_status status;
+			struct gh_vm_exit_info exit_info;
+		} status;
+	};
+};
+
+#define GH_VCPU_RUN		_IO(GH_IOCTL_TYPE, 0x5)
+#define GH_VCPU_MMAP_SIZE	_IO(GH_IOCTL_TYPE, 0x6)
+
+/**
+ * ANDROID: android14-6.1 unfortunately contains UAPI that won't be carried
+ * in kernel.org. Expose orthogonal ioctls that will never conflict with
+ * kernel.org for these UAPIs. See b/268234781.
+ */
+#define GH_ANDROID_IOCTL_TYPE		'A'
+
+#define GH_VM_ANDROID_LEND_USER_MEM	_IOW(GH_ANDROID_IOCTL_TYPE, 0x11, \
+						struct gh_userspace_memory_region)
+
+struct gh_vm_firmware_config {
+	__u64 guest_phys_addr;
+	__u64 size;
+};
+
+#define GH_VM_ANDROID_SET_FW_CONFIG	_IOW(GH_ANDROID_IOCTL_TYPE, 0x12, \
+						struct gh_vm_firmware_config)
+
+#endif
-- 
2.25.1

[RFC/PATCH v1 02/11] gunyah: Basic support

[Srivatsa Vaddagiri]

Add a new accelerator, gunyah, with basic functionality of creating a
VM. Subsequent patches will add support for other functions required to
run a VM.

Signed-off-by: Srivatsa Vaddagiri <quic_svaddagi@quicinc.com>
---
 MAINTAINERS                     |   7 +++
 accel/Kconfig                   |   3 +
 accel/gunyah/gunyah-accel-ops.c | 108 ++++++++++++++++++++++++++++++++
 accel/gunyah/gunyah-all.c       |  70 +++++++++++++++++++++
 accel/gunyah/meson.build        |   7 +++
 accel/meson.build               |   1 +
 accel/stubs/gunyah-stub.c       |  13 ++++
 accel/stubs/meson.build         |   1 +
 docs/about/build-platforms.rst  |   2 +-
 hw/arm/virt.c                   |   3 +
 include/sysemu/gunyah.h         |  30 +++++++++
 include/sysemu/gunyah_int.h     |  27 ++++++++
 meson.build                     |  12 +++-
 meson_options.txt               |   2 +
 scripts/meson-buildoptions.sh   |   3 +
 target/arm/cpu.c                |   3 +-
 target/arm/cpu64.c              |   5 +-
 17 files changed, 292 insertions(+), 5 deletions(-)
 create mode 100644 accel/gunyah/gunyah-accel-ops.c
 create mode 100644 accel/gunyah/gunyah-all.c
 create mode 100644 accel/gunyah/meson.build
 create mode 100644 accel/stubs/gunyah-stub.c
 create mode 100644 include/sysemu/gunyah.h
 create mode 100644 include/sysemu/gunyah_int.h

diff --git a/MAINTAINERS b/MAINTAINERS
index 00ec1f7eca..cdda9208e2 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -537,6 +537,13 @@ F: accel/hvf/
 F: include/sysemu/hvf.h
 F: include/sysemu/hvf_int.h
 
+GUNYAH
+M: Srivatsa Vaddagiri <quic_svaddagi@quicinc.com>
+S: Maintained
+F: accel/gunyah
+F: include/sysemu/gunyah.h
+F: include/sysemu/gunyah_int.h
+
 WHPX CPUs
 M: Sunil Muthuswamy <sunilmut@microsoft.com>
 S: Supported
diff --git a/accel/Kconfig b/accel/Kconfig
index a30cf2eb48..60d86426e2 100644
--- a/accel/Kconfig
+++ b/accel/Kconfig
@@ -16,3 +16,6 @@ config KVM
 config XEN
     bool
     select FSDEV_9P if VIRTFS
+
+config GUNYAH
+    bool
diff --git a/accel/gunyah/gunyah-accel-ops.c b/accel/gunyah/gunyah-accel-ops.c
new file mode 100644
index 0000000000..1419de57e8
--- /dev/null
+++ b/accel/gunyah/gunyah-accel-ops.c
@@ -0,0 +1,108 @@
+/*
+ * QEMU Gunyah hypervisor support
+ *
+ * (based on KVM accelerator code structure)
+ *
+ * Copyright 2008 IBM Corporation
+ *           2008 Red Hat, Inc.
+ *
+ * Copyright(c) 2023 Qualcomm Innovation Center, Inc. All Rights Reserved.
+ *
+ * SPDX-License-Identifier: GPL-2.0-or-later
+ */
+
+#include "qemu/osdep.h"
+#include "qemu/thread.h"
+#include "hw/boards.h"
+#include "hw/core/cpu.h"
+#include "sysemu/accel-ops.h"
+#include "sysemu/cpus.h"
+#include "sysemu/gunyah.h"
+#include "sysemu/gunyah_int.h"
+#include "qapi/visitor.h"
+#include "qapi/error.h"
+
+bool gunyah_allowed;
+
+static int gunyah_init(MachineState *ms)
+{
+    return gunyah_create_vm();
+}
+
+static void gunyah_accel_instance_init(Object *obj)
+{
+    GUNYAHState *s = GUNYAH_STATE(obj);
+
+    s->fd = -1;
+    s->vmfd = -1;
+}
+
+static void gunyah_accel_class_init(ObjectClass *oc, void *data)
+{
+    AccelClass *ac = ACCEL_CLASS(oc);
+
+    ac->name = "GUNYAH";
+    ac->init_machine = gunyah_init;
+    ac->allowed = &gunyah_allowed;
+}
+
+static const TypeInfo gunyah_accel_type = {
+    .name = TYPE_GUNYAH_ACCEL,
+    .parent = TYPE_ACCEL,
+    .instance_init = gunyah_accel_instance_init,
+    .class_init = gunyah_accel_class_init,
+    .instance_size = sizeof(GUNYAHState),
+};
+
+static void gunyah_type_init(void)
+{
+    type_register_static(&gunyah_accel_type);
+}
+type_init(gunyah_type_init);
+
+static void gunyah_start_vcpu_thread(CPUState *cpu)
+{
+    char thread_name[VCPU_THREAD_NAME_SIZE];
+
+    cpu->thread = g_malloc0(sizeof(QemuThread));
+    cpu->halt_cond = g_malloc0(sizeof(QemuCond));
+    qemu_cond_init(cpu->halt_cond);
+
+    snprintf(thread_name, VCPU_THREAD_NAME_SIZE, "CPU %d/Gunyah",
+             cpu->cpu_index);
+    qemu_thread_create(cpu->thread, thread_name, gunyah_cpu_thread_fn,
+                       cpu, QEMU_THREAD_JOINABLE);
+}
+
+static void gunyah_kick_vcpu_thread(CPUState *cpu)
+{
+    cpus_kick_thread(cpu);
+}
+
+static bool gunyah_vcpu_thread_is_idle(CPUState *cpu)
+{
+    return false;
+}
+
+static void gunyah_accel_ops_class_init(ObjectClass *oc, void *data)
+{
+    AccelOpsClass *ops = ACCEL_OPS_CLASS(oc);
+
+    ops->create_vcpu_thread = gunyah_start_vcpu_thread;
+    ops->kick_vcpu_thread = gunyah_kick_vcpu_thread;
+    ops->cpu_thread_is_idle = gunyah_vcpu_thread_is_idle;
+};
+
+static const TypeInfo gunyah_accel_ops_type = {
+    .name = ACCEL_OPS_NAME("gunyah"),
+    .parent = TYPE_ACCEL_OPS,
+    .class_init = gunyah_accel_ops_class_init,
+    .abstract = true,
+};
+
+static void gunyah_accel_ops_register_types(void)
+{
+    type_register_static(&gunyah_accel_ops_type);
+}
+
+type_init(gunyah_accel_ops_register_types);
diff --git a/accel/gunyah/gunyah-all.c b/accel/gunyah/gunyah-all.c
new file mode 100644
index 0000000000..370add75f9
--- /dev/null
+++ b/accel/gunyah/gunyah-all.c
@@ -0,0 +1,70 @@
+/*
+ * QEMU Gunyah hypervisor support
+ *
+ * (based on KVM accelerator code structure)
+ *
+ * Copyright 2008 IBM Corporation
+ *           2008 Red Hat, Inc.
+ *
+ * Copyright(c) 2023 Qualcomm Innovation Center, Inc. All Rights Reserved.
+ *
+ * SPDX-License-Identifier: GPL-2.0-or-later
+ */
+
+#include <sys/ioctl.h>
+#include "qemu/osdep.h"
+#include "qemu/typedefs.h"
+#include "hw/core/cpu.h"
+#include "sysemu/cpus.h"
+#include "sysemu/gunyah.h"
+#include "sysemu/gunyah_int.h"
+#include "linux-headers/linux/gunyah.h"
+#include "qemu/error-report.h"
+
+static int gunyah_ioctl(int type, ...)
+{
+    void *arg;
+    va_list ap;
+    GUNYAHState *s = GUNYAH_STATE(current_accel());
+
+    assert(s->fd);
+
+    va_start(ap, type);
+    arg = va_arg(ap, void *);
+    va_end(ap);
+
+    return ioctl(s->fd, type, arg);
+}
+
+int gunyah_create_vm(void)
+{
+    GUNYAHState *s;
+
+    s = GUNYAH_STATE(current_accel());
+
+    s->fd = qemu_open_old("/dev/gunyah", O_RDWR);
+    if (s->fd == -1) {
+        error_report("Could not access Gunyah kernel module at /dev/gunyah: %s",
+                                strerror(errno));
+        exit(1);
+    }
+
+    s->vmfd = gunyah_ioctl(GH_CREATE_VM, 0);
+    if (s->vmfd < 0) {
+        error_report("Could not create VM: %s", strerror(errno));
+        exit(1);
+    }
+
+    return 0;
+}
+
+void *gunyah_cpu_thread_fn(void *arg)
+{
+    CPUState *cpu = arg;
+
+    do {
+        /* Do nothing */
+    } while (!cpu->unplug || cpu_can_run(cpu));
+
+    return NULL;
+}
diff --git a/accel/gunyah/meson.build b/accel/gunyah/meson.build
new file mode 100644
index 0000000000..226eea0f2c
--- /dev/null
+++ b/accel/gunyah/meson.build
@@ -0,0 +1,7 @@
+gunyah_ss = ss.source_set()
+gunyah_ss.add(files(
+  'gunyah-all.c',
+  'gunyah-accel-ops.c',
+))
+
+specific_ss.add_all(when: 'CONFIG_GUNYAH', if_true: gunyah_ss)
diff --git a/accel/meson.build b/accel/meson.build
index 5eaeb68338..970bad6c5a 100644
--- a/accel/meson.build
+++ b/accel/meson.build
@@ -9,6 +9,7 @@ if have_system
   subdir('kvm')
   subdir('xen')
   subdir('stubs')
+  subdir('gunyah')
 endif
 
 # qtest
diff --git a/accel/stubs/gunyah-stub.c b/accel/stubs/gunyah-stub.c
new file mode 100644
index 0000000000..1edbe1433e
--- /dev/null
+++ b/accel/stubs/gunyah-stub.c
@@ -0,0 +1,13 @@
+/*
+ * QEMU Gunyah stub
+ *
+ * Copyright(c) 2023 Qualcomm Innovation Center, Inc. All Rights Reserved.
+ *
+ * SPDX-License-Identifier: GPL-2.0-or-later
+ *
+ */
+
+#include "qemu/osdep.h"
+#include "sysemu/gunyah.h"
+
+bool gunyah_allowed;
diff --git a/accel/stubs/meson.build b/accel/stubs/meson.build
index 91a2d21925..459a8acd5c 100644
--- a/accel/stubs/meson.build
+++ b/accel/stubs/meson.build
@@ -2,5 +2,6 @@ system_stubs_ss = ss.source_set()
 system_stubs_ss.add(when: 'CONFIG_XEN', if_false: files('xen-stub.c'))
 system_stubs_ss.add(when: 'CONFIG_KVM', if_false: files('kvm-stub.c'))
 system_stubs_ss.add(when: 'CONFIG_TCG', if_false: files('tcg-stub.c'))
+system_stubs_ss.add(when: 'CONFIG_GUNYAH', if_false: files('gunyah-stub.c'))
 
 specific_ss.add_all(when: ['CONFIG_SYSTEM_ONLY'], if_true: system_stubs_ss)
diff --git a/docs/about/build-platforms.rst b/docs/about/build-platforms.rst
index f2a7aec56f..081d792d1b 100644
--- a/docs/about/build-platforms.rst
+++ b/docs/about/build-platforms.rst
@@ -40,7 +40,7 @@ Those hosts are officially supported, with various accelerators:
    * - CPU Architecture
      - Accelerators
    * - Arm
-     - kvm (64 bit only), tcg, xen
+     - kvm (64 bit only), tcg, xen, gunyah (64 bit only)
    * - MIPS (little endian only)
      - kvm, tcg
    * - PPC
diff --git a/hw/arm/virt.c b/hw/arm/virt.c
index 2793121cb4..6a62fc29ad 100644
--- a/hw/arm/virt.c
+++ b/hw/arm/virt.c
@@ -81,6 +81,7 @@
 #include "hw/virtio/virtio-iommu.h"
 #include "hw/char/pl011.h"
 #include "qemu/guest-random.h"
+#include "sysemu/gunyah.h"
 
 #define DEFINE_VIRT_MACHINE_LATEST(major, minor, latest) \
     static void virt_##major##_##minor##_class_init(ObjectClass *oc, \
@@ -1600,6 +1601,8 @@ static void virt_build_smbios(VirtMachineState *vms)
 
     if (kvm_enabled()) {
         product = "KVM Virtual Machine";
+    } else if (gunyah_enabled()) {
+        product = "Gunyah Virtual Machine";
     }
 
     smbios_set_defaults("QEMU", product,
diff --git a/include/sysemu/gunyah.h b/include/sysemu/gunyah.h
new file mode 100644
index 0000000000..4f26938521
--- /dev/null
+++ b/include/sysemu/gunyah.h
@@ -0,0 +1,30 @@
+/*
+ * QEMU Gunyah hypervisor support
+ *
+ * Copyright(c) 2023 Qualcomm Innovation Center, Inc. All Rights Reserved.
+ *
+ * SPDX-License-Identifier: GPL-2.0-or-later
+ */
+
+/* header to be included in non-Gunyah-specific code */
+
+#ifndef QEMU_GUNYAH_H
+#define QEMU_GUNYAH_H
+
+#include "qemu/accel.h"
+#include "qom/object.h"
+
+#ifdef NEED_CPU_H
+#include "cpu.h"
+#endif
+
+extern bool gunyah_allowed;
+
+#define gunyah_enabled() (gunyah_allowed)
+
+#define TYPE_GUNYAH_ACCEL ACCEL_CLASS_NAME("gunyah")
+typedef struct GUNYAHState GUNYAHState;
+DECLARE_INSTANCE_CHECKER(GUNYAHState, GUNYAH_STATE,
+                         TYPE_GUNYAH_ACCEL)
+
+#endif  /* QEMU_GUNYAH_H */
diff --git a/include/sysemu/gunyah_int.h b/include/sysemu/gunyah_int.h
new file mode 100644
index 0000000000..37de628b37
--- /dev/null
+++ b/include/sysemu/gunyah_int.h
@@ -0,0 +1,27 @@
+/*
+ * QEMU Gunyah hypervisor support
+ *
+ * Copyright(c) 2023 Qualcomm Innovation Center, Inc. All Rights Reserved.
+ *
+ * SPDX-License-Identifier: GPL-2.0-or-later
+ */
+
+/* header to be included in Gunyah-specific code */
+
+#ifndef GUNYAH_INT_H
+#define GUNYAH_INT_H
+
+#include "qemu/accel.h"
+#include "qemu/typedefs.h"
+
+struct GUNYAHState {
+    AccelState parent_obj;
+
+    int fd;
+    int vmfd;
+};
+
+int gunyah_create_vm(void);
+void *gunyah_cpu_thread_fn(void *arg);
+
+#endif    /* GUNYAH_INT_H */
diff --git a/meson.build b/meson.build
index 371edafae6..6ded60063e 100644
--- a/meson.build
+++ b/meson.build
@@ -246,7 +246,8 @@ accelerator_targets += { 'CONFIG_XEN': xen_targets }
 
 if cpu in ['aarch64']
   accelerator_targets += {
-    'CONFIG_HVF': ['aarch64-softmmu']
+    'CONFIG_HVF': ['aarch64-softmmu'],
+    'CONFIG_GUNYAH': ['aarch64-softmmu']
   }
 endif
 
@@ -709,6 +710,11 @@ if get_option('hvf').allowed()
   endif
 endif
 
+gunyah = not_found
+if get_option('gunyah').allowed()
+    accelerators += 'CONFIG_GUNYAH'
+endif
+
 nvmm = not_found
 if host_os == 'netbsd'
   nvmm = cc.find_library('nvmm', required: get_option('nvmm'))
@@ -855,6 +861,9 @@ elif get_option('plugins')
 else
   gmodule = not_found
 endif
+if 'CONFIG_GUNYAH' not in accelerators and get_option('gunyah').enabled()
+  error('Gunyah not available on this platform')
+endif
 
 # This workaround is required due to a bug in pkg-config file for glib as it
 # doesn't define GLIB_STATIC_COMPILATION for pkg-config --static
@@ -4215,6 +4224,7 @@ if have_system
     summary_info += {'xen ctrl version':  xen.version()}
   endif
   summary_info += {'Xen emulation':     config_all_devices.has_key('CONFIG_XEN_EMU')}
+  summary_info += {'Gunyah support':    config_all_accel.has_key('CONFIG_GUNYAH')}
 endif
 summary_info += {'TCG support':       config_all_accel.has_key('CONFIG_TCG')}
 if config_all_accel.has_key('CONFIG_TCG')
diff --git a/meson_options.txt b/meson_options.txt
index 0a99a059ec..48c39e8dc0 100644
--- a/meson_options.txt
+++ b/meson_options.txt
@@ -109,6 +109,8 @@ option('dbus_display', type: 'feature', value: 'auto',
        description: '-display dbus support')
 option('tpm', type : 'feature', value : 'auto',
        description: 'TPM support')
+option('gunyah', type: 'feature', value: 'auto',
+       description: 'Gunyah acceleration support')
 
 # Do not enable it by default even for Mingw32, because it doesn't
 # work on Wine.
diff --git a/scripts/meson-buildoptions.sh b/scripts/meson-buildoptions.sh
index 680fa3f581..24e89564b8 100644
--- a/scripts/meson-buildoptions.sh
+++ b/scripts/meson-buildoptions.sh
@@ -125,6 +125,7 @@ meson_options_help() {
   printf "%s\n" '  guest-agent-msi Build MSI package for the QEMU Guest Agent'
   printf "%s\n" '  hv-balloon      hv-balloon driver (requires Glib 2.68+ GTree API)'
   printf "%s\n" '  hvf             HVF acceleration support'
+  printf "%s\n" '  gunyah          Gunyah acceleration support'
   printf "%s\n" '  iconv           Font glyph conversion support'
   printf "%s\n" '  jack            JACK sound support'
   printf "%s\n" '  keyring         Linux keyring support'
@@ -333,6 +334,8 @@ _meson_option_parse() {
     --disable-guest-agent) printf "%s" -Dguest_agent=disabled ;;
     --enable-guest-agent-msi) printf "%s" -Dguest_agent_msi=enabled ;;
     --disable-guest-agent-msi) printf "%s" -Dguest_agent_msi=disabled ;;
+    --enable-gunyah) printf "%s" -Dgunyah=enabled ;;
+    --disable-gunyah) printf "%s" -Dgunyah=disabled ;;
     --enable-hexagon-idef-parser) printf "%s" -Dhexagon_idef_parser=true ;;
     --disable-hexagon-idef-parser) printf "%s" -Dhexagon_idef_parser=false ;;
     --enable-hv-balloon) printf "%s" -Dhv_balloon=enabled ;;
diff --git a/target/arm/cpu.c b/target/arm/cpu.c
index 1c8b787482..e48310fba2 100644
--- a/target/arm/cpu.c
+++ b/target/arm/cpu.c
@@ -44,6 +44,7 @@
 #include "sysemu/tcg.h"
 #include "sysemu/qtest.h"
 #include "sysemu/hw_accel.h"
+#include "sysemu/gunyah.h"
 #include "kvm_arm.h"
 #include "disas/capstone.h"
 #include "fpu/softfloat.h"
@@ -1358,7 +1359,7 @@ static void arm_cpu_initfn(Object *obj)
     cpu->psci_version = QEMU_PSCI_VERSION_0_1; /* By default assume PSCI v0.1 */
     cpu->kvm_target = QEMU_KVM_ARM_TARGET_NONE;
 
-    if (tcg_enabled() || hvf_enabled()) {
+    if (tcg_enabled() || hvf_enabled() || gunyah_enabled()) {
         /* TCG and HVF implement PSCI 1.1 */
         cpu->psci_version = QEMU_PSCI_VERSION_1_1;
     }
diff --git a/target/arm/cpu64.c b/target/arm/cpu64.c
index 8e30a7993e..67691b3555 100644
--- a/target/arm/cpu64.c
+++ b/target/arm/cpu64.c
@@ -34,6 +34,7 @@
 #include "internals.h"
 #include "cpu-features.h"
 #include "cpregs.h"
+#include "sysemu/gunyah.h"
 
 void arm_cpu_sve_finalize(ARMCPU *cpu, Error **errp)
 {
@@ -725,7 +726,7 @@ static void aarch64_host_initfn(Object *obj)
 
 static void aarch64_max_initfn(Object *obj)
 {
-    if (kvm_enabled() || hvf_enabled()) {
+    if (kvm_enabled() || hvf_enabled() || gunyah_enabled()) {
         /* With KVM or HVF, '-cpu max' is identical to '-cpu host' */
         aarch64_host_initfn(obj);
         return;
@@ -745,7 +746,7 @@ static const ARMCPUInfo aarch64_cpus[] = {
     { .name = "cortex-a57",         .initfn = aarch64_a57_initfn },
     { .name = "cortex-a53",         .initfn = aarch64_a53_initfn },
     { .name = "max",                .initfn = aarch64_max_initfn },
-#if defined(CONFIG_KVM) || defined(CONFIG_HVF)
+#if defined(CONFIG_KVM) || defined(CONFIG_HVF) || defined(CONFIG_GUNYAH)
     { .name = "host",               .initfn = aarch64_host_initfn },
 #endif
 };
-- 
2.25.1

[RFC/PATCH v1 03/11] gunyah: Add VM properties

[Srivatsa Vaddagiri]

Add 'protected-vm' and 'preshmem-size' properties that can be specified
for a VM.

Protected VMs are those that have 'protected-vm' property set. Their
memory cannot be accessed by their (potentially untrusted) host. They
are useful to run secure applications whose data should remain private
to the VM.

Since a protected VM may need some memory shared with its host, for
exchange of information, 'preshmem-size' specifies what portion of a
protected VM's memory should be shared with its host, with the remaining
portion being private to it.

Signed-off-by: Srivatsa Vaddagiri <quic_svaddagi@quicinc.com>
---
 accel/gunyah/gunyah-accel-ops.c | 65 +++++++++++++++++++++++++++++++++
 include/sysemu/gunyah_int.h     |  2 +
 2 files changed, 67 insertions(+)

diff --git a/accel/gunyah/gunyah-accel-ops.c b/accel/gunyah/gunyah-accel-ops.c
index 1419de57e8..fe732a1d47 100644
--- a/accel/gunyah/gunyah-accel-ops.c
+++ b/accel/gunyah/gunyah-accel-ops.c
@@ -37,6 +37,60 @@ static void gunyah_accel_instance_init(Object *obj)
     s->vmfd = -1;
 }
 
+static void gunyah_get_preshmem_size(Object *obj, Visitor *v,
+                                    const char *name, void *opaque,
+                                    Error **errp)
+{
+    GUNYAHState *s = GUNYAH_STATE(obj);
+    uint32_t value = s->preshmem_size;
+
+    visit_type_uint32(v, name, &value, errp);
+}
+
+static void gunyah_set_preshmem_size(Object *obj, Visitor *v,
+                                    const char *name, void *opaque,
+                                    Error **errp)
+{
+    GUNYAHState *s = GUNYAH_STATE(obj);
+    uint32_t value;
+
+    if (s->fd != -1) {
+        error_setg(errp, "Cannot set properties after VM is created");
+        return;
+    }
+
+    if (!visit_type_uint32(v, name, &value, errp)) {
+        error_setg(errp, "preshmem-size must be an unsigned integer");
+        return;
+    }
+
+    if (value & (value - 1)) {
+        error_setg(errp, "preshmem-size must be a power of two");
+        return;
+    }
+
+    if (!s->is_protected_vm) {
+        error_setg(errp, "preshmem-size is applicable only for protected VMs");
+        return;
+    }
+
+    s->preshmem_size = value;
+}
+
+static bool gunyah_get_protected_vm(Object *obj, Error **errp)
+{
+    GUNYAHState *s = GUNYAH_STATE(obj);
+
+    return s->is_protected_vm;
+}
+
+static void gunyah_set_protected_vm(Object *obj, bool value, Error **errp)
+{
+    GUNYAHState *s = GUNYAH_STATE(obj);
+
+    s->is_protected_vm = value;
+}
+
 static void gunyah_accel_class_init(ObjectClass *oc, void *data)
 {
     AccelClass *ac = ACCEL_CLASS(oc);
@@ -44,6 +98,17 @@ static void gunyah_accel_class_init(ObjectClass *oc, void *data)
     ac->name = "GUNYAH";
     ac->init_machine = gunyah_init;
     ac->allowed = &gunyah_allowed;
+
+    object_class_property_add_bool(oc, "protected-vm",
+                    gunyah_get_protected_vm, gunyah_set_protected_vm);
+    object_class_property_set_description(oc, "protected-vm",
+            "Launch a VM of protected type");
+
+    object_class_property_add(oc, "preshmem-size", "uint32",
+                gunyah_get_preshmem_size, gunyah_set_preshmem_size, NULL, NULL);
+    object_class_property_set_description(oc, "preshmem-size",
+        "This property is applicable for protected VMs and indicates "
+        "the portion of VM's memory that should be shared with its host");
 }
 
 static const TypeInfo gunyah_accel_type = {
diff --git a/include/sysemu/gunyah_int.h b/include/sysemu/gunyah_int.h
index 37de628b37..b1fd7f9ea2 100644
--- a/include/sysemu/gunyah_int.h
+++ b/include/sysemu/gunyah_int.h
@@ -19,6 +19,8 @@ struct GUNYAHState {
 
     int fd;
     int vmfd;
+    bool is_protected_vm;
+    uint32_t preshmem_size;
 };
 
 int gunyah_create_vm(void);
-- 
2.25.1

[RFC/PATCH v1 04/11] gunyah: Support memory assignment

[Srivatsa Vaddagiri]

Gunyah hypervisor supports several APIs for a host VM to assign some of
its memory to the VM being created.

    Lend - assigned memory is made private to VM (host loses access)
    Share - assigned memory is shared between host and guest VM

No APIs exist however, at this time, for a protected VM to share some of its
(private) memory with host at runtime.

Since, in practice, even a protected VM may need some shared memory to
exchange information with its host, we split VM's memory into two
portions - one that is kept private (via Lend API) and other that is
shared with host (via Share API). Shared portion size is specified via
'preshmem-size' property of VM.

Note: Gunyah kernel driver from Android common kernel repository exposes
two separate ioctls for lend and share operations [1]. A subsequent
version of kernel driver that will be merged in Linux kernel repository
will combine the two ioctls into one, when this patch will be updated.

1. Refer GH_VM_ANDROID_LEND_USER_MEM and GH_VM_SET_USER_MEM_REGION in
https://android.googlesource.com/kernel/common/+/refs/heads/android14-6.1/drivers/virt/gunyah/vm_mgr.c

Signed-off-by: Srivatsa Vaddagiri <quic_svaddagi@quicinc.com>
---
 accel/gunyah/gunyah-all.c   | 248 ++++++++++++++++++++++++++++++++++++
 include/sysemu/gunyah_int.h |  25 ++++
 2 files changed, 273 insertions(+)

diff --git a/accel/gunyah/gunyah-all.c b/accel/gunyah/gunyah-all.c
index 370add75f9..8b6b2d2017 100644
--- a/accel/gunyah/gunyah-all.c
+++ b/accel/gunyah/gunyah-all.c
@@ -14,12 +14,20 @@
 #include <sys/ioctl.h>
 #include "qemu/osdep.h"
 #include "qemu/typedefs.h"
+#include "qemu/units.h"
 #include "hw/core/cpu.h"
 #include "sysemu/cpus.h"
 #include "sysemu/gunyah.h"
 #include "sysemu/gunyah_int.h"
 #include "linux-headers/linux/gunyah.h"
+#include "exec/memory.h"
 #include "qemu/error-report.h"
+#include "exec/address-spaces.h"
+
+static void gunyah_region_add(MemoryListener *listener,
+                           MemoryRegionSection *section);
+static void gunyah_region_del(MemoryListener *listener,
+                           MemoryRegionSection *section);
 
 static int gunyah_ioctl(int type, ...)
 {
@@ -36,9 +44,32 @@ static int gunyah_ioctl(int type, ...)
     return ioctl(s->fd, type, arg);
 }
 
+int gunyah_vm_ioctl(int type, ...)
+{
+    void *arg;
+    va_list ap;
+    GUNYAHState *s = GUNYAH_STATE(current_accel());
+
+    assert(s->vmfd);
+
+    va_start(ap, type);
+    arg = va_arg(ap, void *);
+    va_end(ap);
+
+    return ioctl(s->vmfd, type, arg);
+}
+
+static MemoryListener gunyah_memory_listener = {
+    .name = "gunyah",
+    .priority = MEMORY_LISTENER_PRIORITY_ACCEL,
+    .region_add = gunyah_region_add,
+    .region_del = gunyah_region_del,
+};
+
 int gunyah_create_vm(void)
 {
     GUNYAHState *s;
+    int i;
 
     s = GUNYAH_STATE(current_accel());
 
@@ -55,9 +86,226 @@ int gunyah_create_vm(void)
         exit(1);
     }
 
+    qemu_mutex_init(&s->slots_lock);
+    s->nr_slots = GUNYAH_MAX_MEM_SLOTS;
+    for (i = 0; i < s->nr_slots; ++i) {
+        s->slots[i].start = 0;
+        s->slots[i].size = 0;
+        s->slots[i].id = i;
+    }
+
+    memory_listener_register(&gunyah_memory_listener, &address_space_memory);
     return 0;
 }
 
+#define gunyah_slots_lock(s)    qemu_mutex_lock(&s->slots_lock)
+#define gunyah_slots_unlock(s)  qemu_mutex_unlock(&s->slots_lock)
+
+static gunyah_slot *gunyah_find_overlap_slot(GUNYAHState *s,
+                uint64_t start, uint64_t size)
+{
+    gunyah_slot *slot;
+    int i;
+
+    for (i = 0; i < s->nr_slots; ++i) {
+        slot = &s->slots[i];
+        if (slot->size && start < (slot->start + slot->size) &&
+            (start + size) > slot->start) {
+            return slot;
+        }
+    }
+
+    return NULL;
+}
+
+/* Called with s->slots_lock held */
+static gunyah_slot *gunyah_get_free_slot(GUNYAHState *s)
+{
+    int i;
+
+    for (i = 0; i < s->nr_slots; i++) {
+        if (s->slots[i].size == 0) {
+            return &s->slots[i];
+        }
+    }
+
+    return NULL;
+}
+
+static void gunyah_add_mem(GUNYAHState *s, MemoryRegionSection *section,
+        bool lend, enum gh_mem_flags flags)
+{
+    gunyah_slot *slot;
+    MemoryRegion *area = section->mr;
+    struct gh_userspace_memory_region gumr;
+    int ret;
+
+    slot = gunyah_get_free_slot(s);
+    if (!slot) {
+        error_report("No free slots to add memory!");
+        exit(1);
+    }
+
+    slot->size = int128_get64(section->size);
+    slot->mem = memory_region_get_ram_ptr(area) + section->offset_within_region;
+    slot->start = section->offset_within_address_space;
+    slot->lend = lend;
+
+    gumr.label = slot->id;
+    gumr.flags = flags;
+    gumr.guest_phys_addr = slot->start;
+    gumr.memory_size = slot->size;
+    gumr.userspace_addr = (__u64) slot->mem;
+
+    /*
+     * GH_VM_ANDROID_LEND_USER_MEM is temporary, until
+     * GH_VM_SET_USER_MEM_REGION is enhanced to support lend option also.
+     */
+    if (lend) {
+        ret = gunyah_vm_ioctl(GH_VM_ANDROID_LEND_USER_MEM, &gumr);
+    } else {
+        ret = gunyah_vm_ioctl(GH_VM_SET_USER_MEM_REGION, &gumr);
+    }
+
+    if (ret) {
+        error_report("failed to add mem (%s)", strerror(errno));
+        exit(1);
+    }
+}
+
+/*
+ * Check if memory of a protected VM needs to be split into two portions - one
+ * private to it and other shared with host.
+ */
+static bool split_mem(GUNYAHState *s,
+        MemoryRegion *area, MemoryRegionSection *section)
+{
+    bool writeable = !area->readonly && !area->rom_device;
+
+    /*
+     * Do not split if its not a protected VM OR if the shared mem size is not
+     * specified.
+     */
+    if (!s->is_protected_vm || !s->preshmem_size) {
+        return false;
+    }
+
+    /* Split only memory that can be written to by guest */
+    if (!memory_region_is_ram(area) || !writeable) {
+        return false;
+    }
+
+    /* Have we reserved already? */
+    if (qatomic_read(&s->preshmem_reserved)) {
+        return false;
+    }
+
+    /* Do we have enough available memory? */
+    if (section->size <= s->preshmem_size) {
+        return false;
+    }
+
+    return true;
+}
+
+static void gunyah_set_phys_mem(GUNYAHState *s,
+        MemoryRegionSection *section, bool add)
+{
+    MemoryRegion *area = section->mr;
+    bool writable = !area->readonly && !area->rom_device;
+    enum gh_mem_flags flags = 0;
+    uint64_t page_size = qemu_real_host_page_size();
+    MemoryRegionSection mrs = *section;
+    bool lend = s->is_protected_vm, split = false;
+    struct gunyah_slot *slot;
+
+    /*
+     * Gunyah hypervisor, at this time, does not support mapping memory
+     * at low address (< 1GiB). Below code will be updated once
+     * that limitation is addressed.
+     */
+    if (section->offset_within_address_space < GiB) {
+        return;
+    }
+
+    if (!memory_region_is_ram(area)) {
+        if (writable) {
+            return;
+        } else if (!memory_region_is_romd(area)) {
+            /*
+             * If the memory device is not in romd_mode, then we actually want
+             * to remove the gunyah memory slot so all accesses will trap.
+             */
+             add = false;
+        }
+    }
+
+    if (!QEMU_IS_ALIGNED(int128_get64(section->size), page_size) ||
+        !QEMU_IS_ALIGNED(section->offset_within_address_space, page_size)) {
+        error_report("Not page aligned");
+        add = false;
+    }
+
+    gunyah_slots_lock(s);
+
+    slot = gunyah_find_overlap_slot(s,
+            section->offset_within_address_space,
+            int128_get64(section->size));
+
+    if (!add) {
+        if (slot) {
+            error_report("Memory slot removal not yet supported!");
+            exit(1);
+        }
+        /* Nothing to be done as address range was not previously registered */
+        goto done;
+    } else {
+        if (slot) {
+            error_report("Overlapping slot registration not supported!");
+            exit(1);
+        }
+    }
+
+    if (area->readonly ||
+        (!memory_region_is_ram(area) && memory_region_is_romd(area))) {
+        flags = GH_MEM_ALLOW_READ | GH_MEM_ALLOW_EXEC;
+    } else {
+        flags = GH_MEM_ALLOW_READ | GH_MEM_ALLOW_WRITE | GH_MEM_ALLOW_EXEC;
+    }
+
+    split = split_mem(s, area, &mrs);
+    if (split) {
+        mrs.size -= s->preshmem_size;
+        gunyah_add_mem(s, &mrs, true, flags);
+        lend = false;
+        mrs.offset_within_region += mrs.size;
+        mrs.offset_within_address_space += mrs.size;
+        mrs.size = s->preshmem_size;
+        qatomic_set(&s->preshmem_reserved, true);
+    }
+
+    gunyah_add_mem(s, &mrs, lend, flags);
+
+done:
+    gunyah_slots_unlock(s);
+}
+
+static void gunyah_region_add(MemoryListener *listener,
+                           MemoryRegionSection *section)
+{
+    GUNYAHState *s = GUNYAH_STATE(current_accel());
+
+    gunyah_set_phys_mem(s, section, true);
+}
+
+static void gunyah_region_del(MemoryListener *listener,
+                           MemoryRegionSection *section)
+{
+    GUNYAHState *s = GUNYAH_STATE(current_accel());
+
+    gunyah_set_phys_mem(s, section, false);
+}
+
 void *gunyah_cpu_thread_fn(void *arg)
 {
     CPUState *cpu = arg;
diff --git a/include/sysemu/gunyah_int.h b/include/sysemu/gunyah_int.h
index b1fd7f9ea2..17b4ef9920 100644
--- a/include/sysemu/gunyah_int.h
+++ b/include/sysemu/gunyah_int.h
@@ -13,17 +13,42 @@
 
 #include "qemu/accel.h"
 #include "qemu/typedefs.h"
+#include "qemu/thread.h"
+
+typedef struct gunyah_slot {
+    uint64_t start;
+    uint64_t size;
+    uint8_t *mem;
+    uint32_t id;
+    uint32_t flags;
+
+    /*
+     * @lend indicates if memory was lent.
+     *
+     * This flag is temporarily used until the upstream Gunyah kernel driver
+     * patches are updated to support indication of lend vs share via flags
+     * field of GH_SET_USER_MEM_API interface.
+     */
+    bool lend;
+} gunyah_slot;
+
+#define GUNYAH_MAX_MEM_SLOTS    32
 
 struct GUNYAHState {
     AccelState parent_obj;
 
+    QemuMutex slots_lock;
+    gunyah_slot slots[GUNYAH_MAX_MEM_SLOTS];
+    uint32_t nr_slots;
     int fd;
     int vmfd;
     bool is_protected_vm;
+    bool preshmem_reserved;
     uint32_t preshmem_size;
 };
 
 int gunyah_create_vm(void);
+int gunyah_vm_ioctl(int type, ...);
 void *gunyah_cpu_thread_fn(void *arg);
 
 #endif    /* GUNYAH_INT_H */
-- 
2.25.1

[RFC/PATCH v1 05/11] gunyah: Add IRQFD and IOEVENTFD functions

[Srivatsa Vaddagiri]

IRQFD function allows registering of an @eventfd and @irq. @irq will be
injected inside guest when @eventfd is written into.

IOEVENTFD function allows registering an @eventfd and a guest physical
address, @addr, along with optional data. A poll() on @eventfd  will be
woken up when guest attempts to access @addr.

Signed-off-by: Srivatsa Vaddagiri <quic_svaddagi@quicinc.com>
---
 accel/gunyah/gunyah-all.c   | 94 +++++++++++++++++++++++++++++++++++++
 include/sysemu/gunyah_int.h |  1 +
 2 files changed, 95 insertions(+)

diff --git a/accel/gunyah/gunyah-all.c b/accel/gunyah/gunyah-all.c
index 8b6b2d2017..8aaf5b7354 100644
--- a/accel/gunyah/gunyah-all.c
+++ b/accel/gunyah/gunyah-all.c
@@ -23,11 +23,21 @@
 #include "exec/memory.h"
 #include "qemu/error-report.h"
 #include "exec/address-spaces.h"
+#include "qapi/error.h"
+#include "qemu/event_notifier.h"
 
 static void gunyah_region_add(MemoryListener *listener,
                            MemoryRegionSection *section);
 static void gunyah_region_del(MemoryListener *listener,
                            MemoryRegionSection *section);
+static void gunyah_mem_ioeventfd_add(MemoryListener *listener,
+                                  MemoryRegionSection *section,
+                                  bool match_data, uint64_t data,
+                                  EventNotifier *e);
+static void gunyah_mem_ioeventfd_del(MemoryListener *listener,
+                                  MemoryRegionSection *section,
+                                  bool match_data, uint64_t data,
+                                  EventNotifier *e);
 
 static int gunyah_ioctl(int type, ...)
 {
@@ -64,6 +74,8 @@ static MemoryListener gunyah_memory_listener = {
     .priority = MEMORY_LISTENER_PRIORITY_ACCEL,
     .region_add = gunyah_region_add,
     .region_del = gunyah_region_del,
+    .eventfd_add = gunyah_mem_ioeventfd_add,
+    .eventfd_del = gunyah_mem_ioeventfd_del,
 };
 
 int gunyah_create_vm(void)
@@ -306,6 +318,88 @@ static void gunyah_region_del(MemoryListener *listener,
     gunyah_set_phys_mem(s, section, false);
 }
 
+int gunyah_add_irqfd(int irqfd, int label, Error **errp)
+{
+    int ret;
+    struct gh_fn_desc fdesc;
+    struct gh_fn_irqfd_arg ghirqfd;
+
+    fdesc.type = GH_FN_IRQFD;
+    fdesc.arg_size = sizeof(struct gh_fn_irqfd_arg);
+    fdesc.arg = (__u64)(&ghirqfd);
+
+    ghirqfd.fd = irqfd;
+    ghirqfd.label = label;
+    ghirqfd.flags = GH_IRQFD_FLAGS_LEVEL;
+
+    ret = gunyah_vm_ioctl(GH_VM_ADD_FUNCTION, &fdesc);
+    if (ret) {
+        error_setg_errno(errp, errno, "GH_FN_IRQFD failed");
+    }
+
+    return ret;
+}
+
+static int gunyah_set_ioeventfd_mmio(int fd, hwaddr addr,
+        uint32_t size, uint32_t data, bool datamatch, bool assign)
+{
+    int ret;
+    struct gh_fn_ioeventfd_arg io;
+    struct gh_fn_desc fdesc;
+
+    io.fd = fd;
+    io.datamatch = datamatch ? data : 0;
+    io.len = size;
+    io.addr = addr;
+    io.flags = datamatch ? GH_IOEVENTFD_FLAGS_DATAMATCH : 0;
+
+    fdesc.type = GH_FN_IOEVENTFD;
+    fdesc.arg_size = sizeof(struct gh_fn_ioeventfd_arg);
+    fdesc.arg = (__u64)(&io);
+
+    if (assign) {
+        ret = gunyah_vm_ioctl(GH_VM_ADD_FUNCTION, &fdesc);
+    } else {
+        ret = gunyah_vm_ioctl(GH_VM_REMOVE_FUNCTION, &fdesc);
+    }
+
+    return ret;
+}
+
+static void gunyah_mem_ioeventfd_add(MemoryListener *listener,
+                                  MemoryRegionSection *section,
+                                  bool match_data, uint64_t data,
+                                  EventNotifier *e)
+{
+    int fd = event_notifier_get_fd(e);
+    int r;
+
+    r = gunyah_set_ioeventfd_mmio(fd, section->offset_within_address_space,
+                               int128_get64(section->size), data, match_data,
+                               true);
+    if (r < 0) {
+        error_report("error adding ioeventfd: %s", strerror(errno));
+        exit(1);
+    }
+}
+
+static void gunyah_mem_ioeventfd_del(MemoryListener *listener,
+                                  MemoryRegionSection *section,
+                                  bool match_data, uint64_t data,
+                                  EventNotifier *e)
+{
+    int fd = event_notifier_get_fd(e);
+    int r;
+
+    r = gunyah_set_ioeventfd_mmio(fd, section->offset_within_address_space,
+                               int128_get64(section->size), data, match_data,
+                               false);
+    if (r < 0) {
+        error_report("error deleting ioeventfd: %s", strerror(errno));
+        exit(1);
+    }
+}
+
 void *gunyah_cpu_thread_fn(void *arg)
 {
     CPUState *cpu = arg;
diff --git a/include/sysemu/gunyah_int.h b/include/sysemu/gunyah_int.h
index 17b4ef9920..011b5a072c 100644
--- a/include/sysemu/gunyah_int.h
+++ b/include/sysemu/gunyah_int.h
@@ -50,5 +50,6 @@ struct GUNYAHState {
 int gunyah_create_vm(void);
 int gunyah_vm_ioctl(int type, ...);
 void *gunyah_cpu_thread_fn(void *arg);
+int gunyah_add_irqfd(int irqfd, int label, Error **errp);
 
 #endif    /* GUNYAH_INT_H */
-- 
2.25.1

[RFC/PATCH v1 06/11] gunyah: Add gicv3 interrupt controller

[Srivatsa Vaddagiri]

Gunyah hypervisor supports emulation of a GICv3 compatible interrupt
controller. Emulation is handled by hypervisor itself, with Qemu being
allowed to specify some of the properties such as IO address at which
GICv3 should be mapped in guest address space. These properties are
conveyed to hypervisor via the device-tree, which is parsed by
hypervisor (or more specifically Resource Manager VM, which is the
trusted agent of hypervisor), before VM begins execution.

Injection of interrupts inside guest is supported by doorbell API of
Gunyah hypervisor. Each doorbell is associated with a specific
interrupt. An eventfd is created and associated with each doorbell/irq.
Injection of a specific irq is accomplished by writing to the eventfd
associated with that irq.

Signed-off-by: Srivatsa Vaddagiri <quic_svaddagi@quicinc.com>
---
 MAINTAINERS                    |   2 +
 accel/gunyah/gunyah-all.c      |   5 ++
 hw/arm/virt.c                  |   5 ++
 hw/intc/arm_gicv3_common.c     |   3 +
 hw/intc/arm_gicv3_gunyah.c     | 106 +++++++++++++++++++++++++++++++++
 hw/intc/arm_gicv3_its_common.c |   3 +
 hw/intc/meson.build            |   1 +
 include/sysemu/gunyah_int.h    |   2 +
 8 files changed, 127 insertions(+)
 create mode 100644 hw/intc/arm_gicv3_gunyah.c

diff --git a/MAINTAINERS b/MAINTAINERS
index cdda9208e2..b90a4558a1 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -543,6 +543,8 @@ S: Maintained
 F: accel/gunyah
 F: include/sysemu/gunyah.h
 F: include/sysemu/gunyah_int.h
+F: target/arm/arm_gicv3_gunyah.c
+F: hw/intc/arm_gicv3_gunyah.c
 
 WHPX CPUs
 M: Sunil Muthuswamy <sunilmut@microsoft.com>
diff --git a/accel/gunyah/gunyah-all.c b/accel/gunyah/gunyah-all.c
index 8aaf5b7354..4e4a2b89db 100644
--- a/accel/gunyah/gunyah-all.c
+++ b/accel/gunyah/gunyah-all.c
@@ -400,6 +400,11 @@ static void gunyah_mem_ioeventfd_del(MemoryListener *listener,
     }
 }
 
+GUNYAHState *get_gunyah_state(void)
+{
+    return GUNYAH_STATE(current_accel());
+}
+
 void *gunyah_cpu_thread_fn(void *arg)
 {
     CPUState *cpu = arg;
diff --git a/hw/arm/virt.c b/hw/arm/virt.c
index 6a62fc29ad..4f4e10c234 100644
--- a/hw/arm/virt.c
+++ b/hw/arm/virt.c
@@ -643,6 +643,9 @@ static void create_its(VirtMachineState *vms)
         if (!vms->tcg_its) {
             itsclass = NULL;
         }
+    } else if (!strcmp(itsclass, "arm-its-gunyah")) {
+        /* ITS is not yet supported */
+        itsclass = NULL;
     }
 
     if (!itsclass) {
@@ -1914,6 +1917,8 @@ static void finalize_gic_version(VirtMachineState *vms)
                 gics_supported |= VIRT_GIC_VERSION_4_MASK;
             }
         }
+    } else if (gunyah_enabled()) {
+        gics_supported |= VIRT_GIC_VERSION_3_MASK;
     } else {
         error_report("Unsupported accelerator, can not determine GIC support");
         exit(1);
diff --git a/hw/intc/arm_gicv3_common.c b/hw/intc/arm_gicv3_common.c
index cb55c72681..9d6bc52e45 100644
--- a/hw/intc/arm_gicv3_common.c
+++ b/hw/intc/arm_gicv3_common.c
@@ -32,6 +32,7 @@
 #include "gicv3_internal.h"
 #include "hw/arm/linux-boot-if.h"
 #include "sysemu/kvm.h"
+#include "sysemu/gunyah.h"
 
 
 static void gicv3_gicd_no_migration_shift_bug_post_load(GICv3State *cs)
@@ -614,6 +615,8 @@ const char *gicv3_class_name(void)
 {
     if (kvm_irqchip_in_kernel()) {
         return "kvm-arm-gicv3";
+    } else if (gunyah_enabled()) {
+        return "gunyah-arm-gicv3";
     } else {
         if (kvm_enabled()) {
             error_report("Userspace GICv3 is not supported with KVM");
diff --git a/hw/intc/arm_gicv3_gunyah.c b/hw/intc/arm_gicv3_gunyah.c
new file mode 100644
index 0000000000..f52e82bf9a
--- /dev/null
+++ b/hw/intc/arm_gicv3_gunyah.c
@@ -0,0 +1,106 @@
+/*
+ * QEMU Gunyah hypervisor support
+ *
+ * Copyright(c) 2023 Qualcomm Innovation Center, Inc. All Rights Reserved.
+ *
+ * SPDX-License-Identifier: GPL-2.0-or-later
+ */
+
+#include "qemu/osdep.h"
+#include "qapi/error.h"
+#include "cpu.h"
+#include "hw/intc/arm_gicv3_common.h"
+#include "qemu/error-report.h"
+#include "qemu/module.h"
+#include "sysemu/gunyah.h"
+#include "sysemu/gunyah_int.h"
+#include "sysemu/runstate.h"
+#include "gicv3_internal.h"
+#include "vgic_common.h"
+#include "migration/blocker.h"
+#include "qom/object.h"
+#include "target/arm/cpregs.h"
+#include "qemu/event_notifier.h"
+
+struct GUNYAHARMGICv3Class {
+    ARMGICv3CommonClass parent_class;
+    DeviceRealize parent_realize;
+    ResettablePhases parent_phases;
+};
+
+#define TYPE_GUNYAH_ARM_GICV3 "gunyah-arm-gicv3"
+typedef struct GUNYAHARMGICv3Class GUNYAHARMGICv3Class;
+
+/* This is reusing the GICv3State typedef from ARM_GICV3_ITS_COMMON */
+DECLARE_OBJ_CHECKERS(GICv3State, GUNYAHARMGICv3Class,
+                     GUNYAH_ARM_GICV3, TYPE_GUNYAH_ARM_GICV3)
+
+static EventNotifier *irq_notify;
+
+static void gunyah_arm_gicv3_set_irq(void *opaque, int irq, int level)
+{
+    GICv3State *s = (GICv3State *)opaque;
+
+    if (irq < s->num_irq - GIC_INTERNAL) {
+        event_notifier_set(&irq_notify[irq]);
+    }
+}
+
+static void gunyah_arm_gicv3_realize(DeviceState *dev, Error **errp)
+{
+    GICv3State *s = GUNYAH_ARM_GICV3(dev);
+    GUNYAHARMGICv3Class *ggc = GUNYAH_ARM_GICV3_GET_CLASS(s);
+    Error *local_err = NULL;
+    int i;
+    GUNYAHState *state = get_gunyah_state();
+
+    ggc->parent_realize(dev, &local_err);
+    if (local_err) {
+        error_propagate(errp, local_err);
+        return;
+    }
+
+    if (s->revision != 3) {
+        error_setg(errp, "unsupported GIC revision %d for in-kernel GIC",
+                   s->revision);
+        return;
+    }
+
+    gicv3_init_irqs_and_mmio(s, gunyah_arm_gicv3_set_irq, NULL);
+
+    irq_notify = g_malloc_n(s->num_irq - GIC_INTERNAL, sizeof(EventNotifier));
+
+    for (i = 0; i < s->num_irq - GIC_INTERNAL; ++i) {
+        event_notifier_init(&irq_notify[i], 0);
+        gunyah_add_irqfd(irq_notify[i].wfd, i, errp);
+    }
+
+    state->nr_irqs = s->num_irq - GIC_INTERNAL;
+}
+
+static void gunyah_arm_gicv3_class_init(ObjectClass *klass, void *data)
+{
+    DeviceClass *dc = DEVICE_CLASS(klass);
+    ResettableClass *rc = RESETTABLE_CLASS(klass);
+    GUNYAHARMGICv3Class *ggc = GUNYAH_ARM_GICV3_CLASS(klass);
+
+    device_class_set_parent_realize(dc, gunyah_arm_gicv3_realize,
+                                    &ggc->parent_realize);
+    resettable_class_set_parent_phases(rc, NULL, NULL, NULL,
+                                       &ggc->parent_phases);
+}
+
+static const TypeInfo gunyah_arm_gicv3_info = {
+    .name = TYPE_GUNYAH_ARM_GICV3,
+    .parent = TYPE_ARM_GICV3_COMMON,
+    .instance_size = sizeof(GICv3State),
+    .class_init = gunyah_arm_gicv3_class_init,
+    .class_size = sizeof(GUNYAHARMGICv3Class),
+};
+
+static void gunyah_arm_gicv3_register_types(void)
+{
+    type_register_static(&gunyah_arm_gicv3_info);
+}
+
+type_init(gunyah_arm_gicv3_register_types)
diff --git a/hw/intc/arm_gicv3_its_common.c b/hw/intc/arm_gicv3_its_common.c
index 331d6b93cc..fe3752abc3 100644
--- a/hw/intc/arm_gicv3_its_common.c
+++ b/hw/intc/arm_gicv3_its_common.c
@@ -25,6 +25,7 @@
 #include "qemu/log.h"
 #include "qemu/module.h"
 #include "sysemu/kvm.h"
+#include "sysemu/gunyah.h"
 
 static int gicv3_its_pre_save(void *opaque)
 {
@@ -164,6 +165,8 @@ const char *its_class_name(void)
 {
     if (kvm_irqchip_in_kernel()) {
         return "arm-its-kvm";
+    } else if (gunyah_enabled()) {
+        return "arm-its-gunyah";
     } else {
         /* Software emulation based model */
         return "arm-gicv3-its";
diff --git a/hw/intc/meson.build b/hw/intc/meson.build
index ed355941d1..6e680cf0f2 100644
--- a/hw/intc/meson.build
+++ b/hw/intc/meson.build
@@ -73,3 +73,4 @@ specific_ss.add(when: 'CONFIG_LOONGARCH_IPI', if_true: files('loongarch_ipi.c'))
 specific_ss.add(when: 'CONFIG_LOONGARCH_PCH_PIC', if_true: files('loongarch_pch_pic.c'))
 specific_ss.add(when: 'CONFIG_LOONGARCH_PCH_MSI', if_true: files('loongarch_pch_msi.c'))
 specific_ss.add(when: 'CONFIG_LOONGARCH_EXTIOI', if_true: files('loongarch_extioi.c'))
+specific_ss.add(when: ['CONFIG_ARM_GIC', 'CONFIG_GUNYAH', 'TARGET_AARCH64'], if_true: files('arm_gicv3_gunyah.c'))
diff --git a/include/sysemu/gunyah_int.h b/include/sysemu/gunyah_int.h
index 011b5a072c..dc5b4847a9 100644
--- a/include/sysemu/gunyah_int.h
+++ b/include/sysemu/gunyah_int.h
@@ -45,11 +45,13 @@ struct GUNYAHState {
     bool is_protected_vm;
     bool preshmem_reserved;
     uint32_t preshmem_size;
+    uint32_t nr_irqs;
 };
 
 int gunyah_create_vm(void);
 int gunyah_vm_ioctl(int type, ...);
 void *gunyah_cpu_thread_fn(void *arg);
 int gunyah_add_irqfd(int irqfd, int label, Error **errp);
+GUNYAHState *get_gunyah_state(void);
 
 #endif    /* GUNYAH_INT_H */
-- 
2.25.1

[RFC/PATCH v1 07/11] gunyah: Specify device-tree location

[Srivatsa Vaddagiri]

Specify the location of device-tree and its size, as Gunyah requires the
device-tree to be parsed before VM can begin its execution.

Signed-off-by: Srivatsa Vaddagiri <quic_svaddagi@quicinc.com>
---
 MAINTAINERS               |  1 +
 accel/stubs/gunyah-stub.c |  5 +++++
 hw/arm/virt.c             |  6 ++++++
 include/sysemu/gunyah.h   |  2 ++
 target/arm/gunyah.c       | 45 +++++++++++++++++++++++++++++++++++++++
 target/arm/meson.build    |  3 +++
 6 files changed, 62 insertions(+)
 create mode 100644 target/arm/gunyah.c

diff --git a/MAINTAINERS b/MAINTAINERS
index b90a4558a1..171713bedc 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -541,6 +541,7 @@ GUNYAH
 M: Srivatsa Vaddagiri <quic_svaddagi@quicinc.com>
 S: Maintained
 F: accel/gunyah
+F: target/arm/gunyah.c
 F: include/sysemu/gunyah.h
 F: include/sysemu/gunyah_int.h
 F: target/arm/arm_gicv3_gunyah.c
diff --git a/accel/stubs/gunyah-stub.c b/accel/stubs/gunyah-stub.c
index 1edbe1433e..faeb2af915 100644
--- a/accel/stubs/gunyah-stub.c
+++ b/accel/stubs/gunyah-stub.c
@@ -11,3 +11,8 @@
 #include "sysemu/gunyah.h"
 
 bool gunyah_allowed;
+
+int gunyah_arm_set_dtb(__u64 dtb_start, __u64 dtb_size)
+{
+    return -1;
+}
diff --git a/hw/arm/virt.c b/hw/arm/virt.c
index 4f4e10c234..60fbe2f7c4 100644
--- a/hw/arm/virt.c
+++ b/hw/arm/virt.c
@@ -1656,6 +1656,12 @@ void virt_machine_done(Notifier *notifier, void *data)
         exit(1);
     }
 
+    if (gunyah_enabled()) {
+        if (gunyah_arm_set_dtb(info->dtb_start, vms->fdt_size)) {
+            exit(1);
+        }
+    }
+
     fw_cfg_add_extra_pci_roots(vms->bus, vms->fw_cfg);
 
     virt_acpi_setup(vms);
diff --git a/include/sysemu/gunyah.h b/include/sysemu/gunyah.h
index 4f26938521..a73d17bfb9 100644
--- a/include/sysemu/gunyah.h
+++ b/include/sysemu/gunyah.h
@@ -27,4 +27,6 @@ typedef struct GUNYAHState GUNYAHState;
 DECLARE_INSTANCE_CHECKER(GUNYAHState, GUNYAH_STATE,
                          TYPE_GUNYAH_ACCEL)
 
+int gunyah_arm_set_dtb(__u64 dtb_start, __u64 dtb_size);
+
 #endif  /* QEMU_GUNYAH_H */
diff --git a/target/arm/gunyah.c b/target/arm/gunyah.c
new file mode 100644
index 0000000000..73c1c2a88a
--- /dev/null
+++ b/target/arm/gunyah.c
@@ -0,0 +1,45 @@
+/*
+ * QEMU Gunyah hypervisor support
+ *
+ * Copyright(c) 2023 Qualcomm Innovation Center, Inc. All Rights Reserved.
+ *
+ * SPDX-License-Identifier: GPL-2.0-or-later
+ */
+
+#include "qemu/osdep.h"
+#include "qemu/error-report.h"
+#include "sysemu/gunyah.h"
+#include "sysemu/gunyah_int.h"
+#include "linux-headers/linux/gunyah.h"
+
+/*
+ * Specify location of device-tree in guest address space.
+ *
+ * @dtb_start - Guest physical address where VM's device-tree is found
+ * @dtb_size - Size of device-tree (and any free space after it).
+ *
+ * RM or Resource Manager VM is a trusted and privileged VM that works in
+ * collaboration with Gunyah hypevisor to setup resources for a VM before it can
+ * begin execution. One of its functions includes inspection/modification of a
+ * VM's device-tree before VM begins its execution. Modification can
+ * include specification of runtime resources allocated by hypervisor,
+ * details of which needs to be visible to VM.  VM's device-tree is modified
+ * "inline" making use of "free" space that could exist at the end of device
+ * tree.
+ */
+int gunyah_arm_set_dtb(__u64 dtb_start, __u64 dtb_size)
+{
+    int ret;
+    struct gh_vm_dtb_config dtb;
+
+    dtb.guest_phys_addr = dtb_start;
+    dtb.size = dtb_size;
+
+    ret = gunyah_vm_ioctl(GH_VM_SET_DTB_CONFIG, &dtb);
+    if (ret != 0) {
+        error_report("GH_VM_SET_DTB_CONFIG failed: %s", strerror(errno));
+        exit(1);
+    }
+
+    return 0;
+}
diff --git a/target/arm/meson.build b/target/arm/meson.build
index 46b5a21eb3..797df04717 100644
--- a/target/arm/meson.build
+++ b/target/arm/meson.build
@@ -36,3 +36,6 @@ endif
 
 target_arch += {'arm': arm_ss}
 target_system_arch += {'arm': arm_system_ss}
+arm_system_ss.add(when: 'CONFIG_GUNYAH', if_true: files(
+  'gunyah.c',
+))
-- 
2.25.1

[RFC/PATCH v1 08/11] gunyah: Customize device-tree

[Srivatsa Vaddagiri]

Customize device-tree with Gunyah specific properties. Some of these
properties include specification of doorbells that need to be created
and associated with various interrupts.

Signed-off-by: Srivatsa Vaddagiri <quic_svaddagi@quicinc.com>
---
 accel/stubs/gunyah-stub.c |  5 +++
 hw/arm/virt.c             | 11 ++++++
 include/sysemu/gunyah.h   |  2 +
 target/arm/gunyah.c       | 79 +++++++++++++++++++++++++++++++++++++++
 4 files changed, 97 insertions(+)

diff --git a/accel/stubs/gunyah-stub.c b/accel/stubs/gunyah-stub.c
index faeb2af915..e9a8752bb4 100644
--- a/accel/stubs/gunyah-stub.c
+++ b/accel/stubs/gunyah-stub.c
@@ -16,3 +16,8 @@ int gunyah_arm_set_dtb(__u64 dtb_start, __u64 dtb_size)
 {
     return -1;
 }
+
+void gunyah_arm_fdt_customize(void *fdt, uint64_t mem_base,
+                uint32_t gic_phandle) {
+    return;
+}
diff --git a/hw/arm/virt.c b/hw/arm/virt.c
index 60fbe2f7c4..1aaadc1e1d 100644
--- a/hw/arm/virt.c
+++ b/hw/arm/virt.c
@@ -2005,6 +2005,14 @@ static void virt_cpu_post_init(VirtMachineState *vms, MemoryRegion *sysmem)
     }
 }
 
+static void virt_modify_dtb(const struct arm_boot_info *binfo, void *fdt)
+{
+    const VirtMachineState *vms = container_of(binfo, VirtMachineState,
+                                                 bootinfo);
+
+    gunyah_arm_fdt_customize(fdt, vms->memmap[VIRT_MEM].base, vms->gic_phandle);
+}
+
 static void machvirt_init(MachineState *machine)
 {
     VirtMachineState *vms = VIRT_MACHINE(machine);
@@ -2311,6 +2319,9 @@ static void machvirt_init(MachineState *machine)
     vms->bootinfo.skip_dtb_autoload = true;
     vms->bootinfo.firmware_loaded = firmware_loaded;
     vms->bootinfo.psci_conduit = vms->psci_conduit;
+    if (gunyah_enabled()) {
+        vms->bootinfo.modify_dtb = virt_modify_dtb;
+    }
     arm_load_kernel(ARM_CPU(first_cpu), machine, &vms->bootinfo);
 
     vms->machine_done.notify = virt_machine_done;
diff --git a/include/sysemu/gunyah.h b/include/sysemu/gunyah.h
index a73d17bfb9..d0c1d8564c 100644
--- a/include/sysemu/gunyah.h
+++ b/include/sysemu/gunyah.h
@@ -28,5 +28,7 @@ DECLARE_INSTANCE_CHECKER(GUNYAHState, GUNYAH_STATE,
                          TYPE_GUNYAH_ACCEL)
 
 int gunyah_arm_set_dtb(__u64 dtb_start, __u64 dtb_size);
+void gunyah_arm_fdt_customize(void *fdt, uint64_t mem_base,
+                uint32_t gic_phandle);
 
 #endif  /* QEMU_GUNYAH_H */
diff --git a/target/arm/gunyah.c b/target/arm/gunyah.c
index 73c1c2a88a..1521f2d414 100644
--- a/target/arm/gunyah.c
+++ b/target/arm/gunyah.c
@@ -11,6 +11,9 @@
 #include "sysemu/gunyah.h"
 #include "sysemu/gunyah_int.h"
 #include "linux-headers/linux/gunyah.h"
+#include "exec/memory.h"
+#include "sysemu/device_tree.h"
+#include "hw/arm/fdt.h"
 
 /*
  * Specify location of device-tree in guest address space.
@@ -43,3 +46,79 @@ int gunyah_arm_set_dtb(__u64 dtb_start, __u64 dtb_size)
 
     return 0;
 }
+
+void gunyah_arm_fdt_customize(void *fdt, uint64_t mem_base,
+            uint32_t gic_phandle)
+{
+    char *nodename;
+    int i;
+    GUNYAHState *state = get_gunyah_state();
+
+    qemu_fdt_add_subnode(fdt, "/gunyah-vm-config");
+    qemu_fdt_setprop_string(fdt, "/gunyah-vm-config",
+                                "image-name", "qemu-vm");
+    qemu_fdt_setprop_string(fdt, "/gunyah-vm-config", "os-type", "linux");
+
+    nodename = g_strdup_printf("/gunyah-vm-config/memory");
+    qemu_fdt_add_subnode(fdt, nodename);
+    qemu_fdt_setprop_cell(fdt, nodename, "#address-cells", 2);
+    qemu_fdt_setprop_cell(fdt, nodename, "#size-cells", 2);
+    qemu_fdt_setprop_u64(fdt, nodename, "base-address", mem_base);
+
+    g_free(nodename);
+
+    nodename = g_strdup_printf("/gunyah-vm-config/interrupts");
+    qemu_fdt_add_subnode(fdt, nodename);
+    qemu_fdt_setprop_cell(fdt, nodename, "config", gic_phandle);
+    g_free(nodename);
+
+    nodename = g_strdup_printf("/gunyah-vm-config/vcpus");
+    qemu_fdt_add_subnode(fdt, nodename);
+    qemu_fdt_setprop_string(fdt, nodename, "affinity", "proxy");
+    g_free(nodename);
+
+    nodename = g_strdup_printf("/gunyah-vm-config/vdevices");
+    qemu_fdt_add_subnode(fdt, nodename);
+    qemu_fdt_setprop_string(fdt, nodename, "generate", "/hypervisor");
+    g_free(nodename);
+
+    for (i = 0; i < state->nr_slots; ++i) {
+        if (!state->slots[i].start || state->slots[i].lend ||
+                state->slots[i].start == mem_base) {
+            continue;
+        }
+
+        nodename = g_strdup_printf("/gunyah-vm-config/vdevices/shm-%x", i);
+        qemu_fdt_add_subnode(fdt, nodename);
+        qemu_fdt_setprop_string(fdt, nodename, "vdevice-type", "shm");
+        qemu_fdt_setprop_string(fdt, nodename, "push-compatible", "dma");
+        qemu_fdt_setprop(fdt, nodename, "peer-default", NULL, 0);
+        qemu_fdt_setprop_u64(fdt, nodename, "dma_base", 0);
+        g_free(nodename);
+
+        nodename = g_strdup_printf("/gunyah-vm-config/vdevices/shm-%x/memory",
+                                                                        i);
+        qemu_fdt_add_subnode(fdt, nodename);
+        qemu_fdt_setprop_cell(fdt, nodename, "label", i);
+        qemu_fdt_setprop_cell(fdt, nodename, "#address-cells", 2);
+        qemu_fdt_setprop_u64(fdt, nodename, "base", state->slots[i].start);
+        g_free(nodename);
+    }
+
+    for (i = 0; i < state->nr_irqs; ++i) {
+        nodename = g_strdup_printf("/gunyah-vm-config/vdevices/bell-%x", i);
+        qemu_fdt_add_subnode(fdt, nodename);
+        qemu_fdt_setprop_string(fdt, nodename, "vdevice-type", "doorbell");
+        char *p = g_strdup_printf("/hypervisor/bell-%x", i);
+        qemu_fdt_setprop_string(fdt, nodename, "generate", p);
+        g_free(p);
+        qemu_fdt_setprop_cell(fdt, nodename, "label", i);
+        qemu_fdt_setprop(fdt, nodename, "peer-default", NULL, 0);
+        qemu_fdt_setprop(fdt, nodename, "source-can-clear", NULL, 0);
+
+        qemu_fdt_setprop_cells(fdt, nodename, "interrupts",
+                GIC_FDT_IRQ_TYPE_SPI, i, GIC_FDT_IRQ_FLAGS_LEVEL_HI);
+
+        g_free(nodename);
+    }
+}
-- 
2.25.1

[RFC/PATCH v1 09/11] gunyah: CPU execution loop

[Srivatsa Vaddagiri]

Complete the cpu execution loop. At this time, we recognize exits
associated with only MMIO access. Future patches will add support for
recognizing other exit reasons, such as PSCI calls made by guest.

Signed-off-by: Srivatsa Vaddagiri <quic_svaddagi@quicinc.com>
---
 accel/gunyah/gunyah-accel-ops.c |   7 ++
 accel/gunyah/gunyah-all.c       | 201 +++++++++++++++++++++++++++++++-
 include/hw/core/cpu.h           |   1 +
 include/sysemu/gunyah_int.h     |   9 ++
 target/arm/gunyah.c             |  13 +++
 5 files changed, 230 insertions(+), 1 deletion(-)

diff --git a/accel/gunyah/gunyah-accel-ops.c b/accel/gunyah/gunyah-accel-ops.c
index fe732a1d47..a13b1c8cb5 100644
--- a/accel/gunyah/gunyah-accel-ops.c
+++ b/accel/gunyah/gunyah-accel-ops.c
@@ -91,6 +91,11 @@ static void gunyah_set_protected_vm(Object *obj, bool value, Error **errp)
     s->is_protected_vm = value;
 }
 
+static void gunyah_setup_post(MachineState *ms, AccelState *accel)
+{
+    gunyah_start_vm();
+}
+
 static void gunyah_accel_class_init(ObjectClass *oc, void *data)
 {
     AccelClass *ac = ACCEL_CLASS(oc);
@@ -98,6 +103,7 @@ static void gunyah_accel_class_init(ObjectClass *oc, void *data)
     ac->name = "GUNYAH";
     ac->init_machine = gunyah_init;
     ac->allowed = &gunyah_allowed;
+    ac->setup_post = gunyah_setup_post;
 
     object_class_property_add_bool(oc, "protected-vm",
                     gunyah_get_protected_vm, gunyah_set_protected_vm);
@@ -156,6 +162,7 @@ static void gunyah_accel_ops_class_init(ObjectClass *oc, void *data)
     ops->create_vcpu_thread = gunyah_start_vcpu_thread;
     ops->kick_vcpu_thread = gunyah_kick_vcpu_thread;
     ops->cpu_thread_is_idle = gunyah_vcpu_thread_is_idle;
+    ops->synchronize_post_reset = gunyah_cpu_synchronize_post_reset;
 };
 
 static const TypeInfo gunyah_accel_ops_type = {
diff --git a/accel/gunyah/gunyah-all.c b/accel/gunyah/gunyah-all.c
index 4e4a2b89db..3eeecfd286 100644
--- a/accel/gunyah/gunyah-all.c
+++ b/accel/gunyah/gunyah-all.c
@@ -25,6 +25,9 @@
 #include "exec/address-spaces.h"
 #include "qapi/error.h"
 #include "qemu/event_notifier.h"
+#include "qemu/main-loop.h"
+#include "sysemu/runstate.h"
+#include "qemu/guest-random.h"
 
 static void gunyah_region_add(MemoryListener *listener,
                            MemoryRegionSection *section);
@@ -69,6 +72,18 @@ int gunyah_vm_ioctl(int type, ...)
     return ioctl(s->vmfd, type, arg);
 }
 
+static int gunyah_vcpu_ioctl(CPUState *cpu, int type, ...)
+{
+    void *arg;
+    va_list ap;
+
+    va_start(ap, type);
+    arg = va_arg(ap, void *);
+    va_end(ap);
+
+    return ioctl(cpu->accel->fd, type, arg);
+}
+
 static MemoryListener gunyah_memory_listener = {
     .name = "gunyah",
     .priority = MEMORY_LISTENER_PRIORITY_ACCEL,
@@ -276,6 +291,11 @@ static void gunyah_set_phys_mem(GUNYAHState *s,
             error_report("Overlapping slot registration not supported!");
             exit(1);
         }
+
+        if (qatomic_read(&s->vm_started)) {
+            error_report("Memory map changes after VM start not supported!");
+            exit(1);
+        }
     }
 
     if (area->readonly ||
@@ -405,13 +425,192 @@ GUNYAHState *get_gunyah_state(void)
     return GUNYAH_STATE(current_accel());
 }
 
+static void gunyah_ipi_signal(int sig)
+{
+    if (current_cpu) {
+        qatomic_set(&current_cpu->accel->run->immediate_exit, 1);
+    }
+}
+
+static void gunyah_cpu_kick_self(void)
+{
+    qatomic_set(&current_cpu->accel->run->immediate_exit, 1);
+}
+
+static int gunyah_init_vcpu(CPUState *cpu, Error **errp)
+{
+    int ret;
+    struct gh_fn_desc fdesc;
+    struct gh_fn_vcpu_arg vcpu;
+    struct sigaction sigact;
+    sigset_t set;
+
+    cpu->accel = g_new0(AccelCPUState, 1);
+
+    /* init cpu signals */
+    memset(&sigact, 0, sizeof(sigact));
+    sigact.sa_handler = gunyah_ipi_signal;
+    sigaction(SIG_IPI, &sigact, NULL);
+
+    pthread_sigmask(SIG_BLOCK, NULL, &set);
+    sigdelset(&set, SIG_IPI);
+
+    ret = pthread_sigmask(SIG_SETMASK, &set, NULL);
+    if (ret) {
+        error_report("pthread_sigmask: %s", strerror(ret));
+        exit(1);
+    }
+
+    vcpu.id = cpu->cpu_index;
+    fdesc.type = GH_FN_VCPU;
+    fdesc.arg_size = sizeof(struct gh_fn_vcpu_arg);
+    fdesc.arg = (__u64)(&vcpu);
+
+    ret = gunyah_vm_ioctl(GH_VM_ADD_FUNCTION, &fdesc);
+    if (ret < 0) {
+        error_report("could not create VCPU %d: %s", vcpu.id, strerror(errno));
+        exit(1);
+    }
+
+    cpu->accel->fd = ret;
+    cpu->accel->run = mmap(0, 4096, PROT_READ | PROT_WRITE, MAP_SHARED, ret, 0);
+    if (cpu->accel->run == MAP_FAILED) {
+        error_report("mmap of vcpu run structure failed : %s", strerror(errno));
+        exit(1);
+    }
+
+    return 0;
+}
+
+static void gunyah_vcpu_destroy(CPUState *cpu)
+{
+    int ret;
+
+    ret = munmap(cpu->accel->run, 4096);
+    if (ret < 0) {
+        error_report("munmap of vcpu run structure failed: %s",
+                strerror(errno));
+        exit(1);
+    }
+
+    close(cpu->accel->fd);
+    g_free(cpu->accel);
+}
+
+void gunyah_start_vm(void)
+{
+    int ret;
+    GUNYAHState *s = GUNYAH_STATE(current_accel());
+
+    ret = gunyah_vm_ioctl(GH_VM_START);
+    if (ret != 0) {
+        error_report("Failed to start VM: %s", strerror(errno));
+        exit(1);
+    }
+    qatomic_set(&s->vm_started, 1);
+}
+
+static int gunyah_vcpu_exec(CPUState *cpu)
+{
+    int ret;
+
+    qemu_mutex_unlock_iothread();
+    cpu_exec_start(cpu);
+
+    do {
+        struct gh_vcpu_run *run = cpu->accel->run;
+        int exit_reason;
+
+        if (qatomic_read(&cpu->exit_request)) {
+            gunyah_cpu_kick_self();
+        }
+
+        /* Todo: Check need for smp_rmb() here */
+
+        ret = gunyah_vcpu_ioctl(cpu, GH_VCPU_RUN);
+        if (ret < 0) {
+            if (errno == EINTR || errno == EAGAIN) {
+                qatomic_set(&run->immediate_exit, 0);
+                /* Todo: Check need for smp_wmb() here */
+                ret = EXCP_INTERRUPT;
+                break;
+            }
+
+            error_report("GH_VCPU_RUN: %s", strerror(errno));
+            ret = -1;
+            break;
+        }
+
+        exit_reason = run->exit_reason;
+        switch (exit_reason) {
+        case GH_VCPU_EXIT_MMIO:
+            address_space_rw(&address_space_memory,
+                run->mmio.phys_addr, MEMTXATTRS_UNSPECIFIED,
+                run->mmio.data,
+                run->mmio.len,
+                run->mmio.is_write);
+            break;
+
+        /* Todo: Handle VM shutdown */
+        default:
+            error_report("unhandled exit %d", exit_reason);
+        }
+    } while (ret == 0);
+
+    cpu_exec_end(cpu);
+    qemu_mutex_lock_iothread();
+
+    if (ret < 0) {
+        cpu_dump_state(cpu, stderr, CPU_DUMP_CODE);
+        vm_stop(RUN_STATE_INTERNAL_ERROR);
+    }
+
+    qatomic_set(&cpu->exit_request, 0);
+
+    return ret;
+}
+
 void *gunyah_cpu_thread_fn(void *arg)
 {
     CPUState *cpu = arg;
 
+    rcu_register_thread();
+
+    qemu_mutex_lock_iothread();
+    qemu_thread_get_self(cpu->thread);
+
+    cpu->thread_id = qemu_get_thread_id();
+    cpu->neg.can_do_io = true;
+    current_cpu = cpu;
+
+    gunyah_init_vcpu(cpu, &error_fatal);
+
+    /* signal CPU creation */
+    cpu_thread_signal_created(cpu);
+    qemu_guest_random_seed_thread_part2(cpu->random_seed);
+
     do {
-        /* Do nothing */
+        if (cpu_can_run(cpu)) {
+            gunyah_vcpu_exec(cpu);
+        }
+        qemu_wait_io_event(cpu);
     } while (!cpu->unplug || cpu_can_run(cpu));
 
+    gunyah_vcpu_destroy(cpu);
+    cpu_thread_signal_destroyed(cpu);
+    qemu_mutex_unlock_iothread();
+    rcu_unregister_thread();
     return NULL;
 }
+
+static void do_gunyah_cpu_synchronize_post_reset(CPUState *cpu,
+                                run_on_cpu_data arg)
+{
+    gunyah_arch_put_registers(cpu, 0);
+    cpu->vcpu_dirty = false;
+}
+
+void gunyah_cpu_synchronize_post_reset(CPUState *cpu)
+{
+    run_on_cpu(cpu, do_gunyah_cpu_synchronize_post_reset, RUN_ON_CPU_NULL);
+}
diff --git a/include/hw/core/cpu.h b/include/hw/core/cpu.h
index 238c02c05e..ef248a658b 100644
--- a/include/hw/core/cpu.h
+++ b/include/hw/core/cpu.h
@@ -371,6 +371,7 @@ struct CPUWatchpoint {
 
 struct KVMState;
 struct kvm_run;
+struct gh_vcpu_run;
 
 /* work queue */
 
diff --git a/include/sysemu/gunyah_int.h b/include/sysemu/gunyah_int.h
index dc5b4847a9..72b3027e4e 100644
--- a/include/sysemu/gunyah_int.h
+++ b/include/sysemu/gunyah_int.h
@@ -46,12 +46,21 @@ struct GUNYAHState {
     bool preshmem_reserved;
     uint32_t preshmem_size;
     uint32_t nr_irqs;
+    uint32_t vm_started;
+};
+
+struct AccelCPUState {
+    int fd;
+    struct gh_vcpu_run *run;
 };
 
 int gunyah_create_vm(void);
+void gunyah_start_vm(void);
 int gunyah_vm_ioctl(int type, ...);
 void *gunyah_cpu_thread_fn(void *arg);
 int gunyah_add_irqfd(int irqfd, int label, Error **errp);
 GUNYAHState *get_gunyah_state(void);
+int gunyah_arch_put_registers(CPUState *cs, int level);
+void gunyah_cpu_synchronize_post_reset(CPUState *cpu);
 
 #endif    /* GUNYAH_INT_H */
diff --git a/target/arm/gunyah.c b/target/arm/gunyah.c
index 1521f2d414..06600dbdb7 100644
--- a/target/arm/gunyah.c
+++ b/target/arm/gunyah.c
@@ -122,3 +122,16 @@ void gunyah_arm_fdt_customize(void *fdt, uint64_t mem_base,
         g_free(nodename);
     }
 }
+
+int gunyah_arch_put_registers(CPUState *cs, int level)
+{
+    /*
+     * No support (yet) to set/get vCPU registers. We specify device-tree
+     * location, which is passed on to VM via X0. Image entry point is assumed
+     * to be the beginning of slot containing device-tree, which seems to be
+     * true currently. In future, Gunyah could add API to set boot CPU's
+     * register context.
+     */
+
+    return 0;
+}
-- 
2.25.1

[RFC/PATCH v1 10/11] gunyah: Workarounds (NOT FOR MERGE)

[Srivatsa Vaddagiri]

These are some work-arounds required temporarily until some limitations
with Gunyah hypervisor are addressed.

Signed-off-by: Srivatsa Vaddagiri <quic_svaddagi@quicinc.com>
---
 accel/gunyah/gunyah-all.c   | 18 ++++++++++++++++++
 hw/arm/boot.c               |  3 ++-
 hw/arm/virt.c               |  3 ++-
 include/sysemu/gunyah_int.h |  1 +
 target/arm/gunyah.c         |  7 +++++++
 5 files changed, 30 insertions(+), 2 deletions(-)

diff --git a/accel/gunyah/gunyah-all.c b/accel/gunyah/gunyah-all.c
index 3eeecfd286..7d300a16bd 100644
--- a/accel/gunyah/gunyah-all.c
+++ b/accel/gunyah/gunyah-all.c
@@ -145,6 +145,24 @@ static gunyah_slot *gunyah_find_overlap_slot(GUNYAHState *s,
     return NULL;
 }
 
+gunyah_slot *gunyah_find_slot_by_addr(uint64_t addr)
+{
+    GUNYAHState *s = GUNYAH_STATE(current_accel());
+    int i;
+    gunyah_slot *slot = NULL;
+
+    gunyah_slots_lock(s);
+    for (i = 0; i < s->nr_slots; ++i) {
+        slot = &s->slots[i];
+        if (slot->size &&
+            (addr >= slot->start && addr <= slot->start + slot->size))
+                break;
+    }
+    gunyah_slots_unlock(s);
+
+    return slot;
+}
+
 /* Called with s->slots_lock held */
 static gunyah_slot *gunyah_get_free_slot(GUNYAHState *s)
 {
diff --git a/hw/arm/boot.c b/hw/arm/boot.c
index 84ea6a807a..77aa16ee8c 100644
--- a/hw/arm/boot.c
+++ b/hw/arm/boot.c
@@ -413,7 +413,8 @@ static int fdt_add_memory_node(void *fdt, uint32_t acells, hwaddr mem_base,
     char *nodename;
     int ret;
 
-    nodename = g_strdup_printf("/memory@%" PRIx64, mem_base);
+    /* Workaround until RM can parse memory nodes of type memory@XYZ. */
+    nodename = g_strdup_printf("/memory");
     qemu_fdt_add_subnode(fdt, nodename);
     qemu_fdt_setprop_string(fdt, nodename, "device_type", "memory");
     ret = qemu_fdt_setprop_sized_cells(fdt, nodename, "reg", acells, mem_base,
diff --git a/hw/arm/virt.c b/hw/arm/virt.c
index 1aaadc1e1d..798eb70af2 100644
--- a/hw/arm/virt.c
+++ b/hw/arm/virt.c
@@ -165,7 +165,8 @@ static const MemMapEntry base_memmap[] = {
     [VIRT_PCIE_PIO] =           { 0x3eff0000, 0x00010000 },
     [VIRT_PCIE_ECAM] =          { 0x3f000000, 0x01000000 },
     /* Actual RAM size depends on initial RAM and device memory settings */
-    [VIRT_MEM] =                { GiB, LEGACY_RAMLIMIT_BYTES },
+    /* Workaround until Gunyah can accept mapping that starts from GiB */
+    [VIRT_MEM] =                { 2 * GiB, LEGACY_RAMLIMIT_BYTES },
 };
 
 /*
diff --git a/include/sysemu/gunyah_int.h b/include/sysemu/gunyah_int.h
index 72b3027e4e..2534c8883f 100644
--- a/include/sysemu/gunyah_int.h
+++ b/include/sysemu/gunyah_int.h
@@ -62,5 +62,6 @@ int gunyah_add_irqfd(int irqfd, int label, Error **errp);
 GUNYAHState *get_gunyah_state(void);
 int gunyah_arch_put_registers(CPUState *cs, int level);
 void gunyah_cpu_synchronize_post_reset(CPUState *cpu);
+gunyah_slot *gunyah_find_slot_by_addr(uint64_t addr);
 
 #endif    /* GUNYAH_INT_H */
diff --git a/target/arm/gunyah.c b/target/arm/gunyah.c
index 06600dbdb7..55d5ccc2a4 100644
--- a/target/arm/gunyah.c
+++ b/target/arm/gunyah.c
@@ -34,6 +34,13 @@ int gunyah_arm_set_dtb(__u64 dtb_start, __u64 dtb_size)
 {
     int ret;
     struct gh_vm_dtb_config dtb;
+    gunyah_slot *slot = gunyah_find_slot_by_addr(dtb_start);
+
+    /*
+     * RM should consider 'totalsize' field to be inclusive of free space. Use
+     * this workaround until RM is fixed.
+     */
+    dtb_size = slot->start + slot->size - dtb_start;
 
     dtb.guest_phys_addr = dtb_start;
     dtb.size = dtb_size;
-- 
2.25.1

[RFC/PATCH v1 11/11] gunyah: Documentation

[Srivatsa Vaddagiri]

Add gunyah.rst that provide some informaiton on how to build and test
'gunyah' accelerator with open-source Gunyah hypervisor.

Signed-off-by: Srivatsa Vaddagiri <quic_svaddagi@quicinc.com>
---
 MAINTAINERS                |   1 +
 docs/system/arm/gunyah.rst | 358 +++++++++++++++++++++++++++++++++++++
 2 files changed, 359 insertions(+)
 create mode 100644 docs/system/arm/gunyah.rst

diff --git a/MAINTAINERS b/MAINTAINERS
index 171713bedc..cb5969fc2b 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -546,6 +546,7 @@ F: include/sysemu/gunyah.h
 F: include/sysemu/gunyah_int.h
 F: target/arm/arm_gicv3_gunyah.c
 F: hw/intc/arm_gicv3_gunyah.c
+F: docs/system/arm/gunyah.rst
 
 WHPX CPUs
 M: Sunil Muthuswamy <sunilmut@microsoft.com>
diff --git a/docs/system/arm/gunyah.rst b/docs/system/arm/gunyah.rst
new file mode 100644
index 0000000000..5970c5190d
--- /dev/null
+++ b/docs/system/arm/gunyah.rst
@@ -0,0 +1,358 @@
+'gunyah' accelerator (``gunyah``)
+=================================
+
+Gunyah is a high performance, scalable and flexible hypervisor built for
+demanding battery powered, real-time, safety and security use cases.
+
+The Gunyah Hypervisor open source project provides a reference Type-1 hypervisor
+configuration suitable for general purpose hosting of multiple trusted and
+dependent VMs. Further information on open-source version of Gunyah Hypervisor
+can be obtained from:
+
+https://github.com/quic/gunyah-hypervisor
+
+To get started with open-source version of Gunyah Hypervisor, refer to the
+instructions available at:
+
+https://github.com/quic/gunyah-support-scripts
+
+Build and testing
+-----------------
+
+Configure and build Qemu
+````````````````````````
+
+Apply the proposed patches for 'gunyah' accelerator support in Qemu and build
+it.
+
+.. code-block:: bash
+
+        $ ./configure --target-list=aarch64-softmmu --enable-debug --enable-gunyah --static
+        $ make -j4
+        $ mv build/qemu-system-aarch64 build/qemu-gunyah
+
+
+Clone gunyah-support scripts
+````````````````````````````
+
+.. code-block:: bash
+
+        $ git clone https://github.com/quic/gunyah-support-scripts
+
+Instructions in this document to build and test Gunyah hypervisor was validated
+with the latest commit in gunyah-support-scripts being:
+
+b8e3862 tools: Faster upgrades to latest versions
+
+
+Patch gunyah-support scripts
+````````````````````````````
+Apply below patch to gunyah-support scripts. This is required **temporarily** until
+the scripts can be updated to support Qemu as VMM (in addition to CrosVM) and
+also to fix some issues.
+
+.. code-block:: bash
+
+	diff --git a/scripts/build-docker-img.sh b/scripts/build-docker-img.sh
+	index 98e7881..a6aa774 100755
+	--- a/scripts/build-docker-img.sh
+	+++ b/scripts/build-docker-img.sh
+	@@ -38,7 +38,7 @@ DOCKER_OPTIONS=" build . "
+	 #DOCKER_OPTIONS+=" --progress=plain "
+	 
+	 #  no-cache alleviates some install errors for not finding some packages
+	-#DOCKER_OPTIONS+=" --no-cache "
+	+DOCKER_OPTIONS+=" --no-cache "
+	 
+	 # user environment related so the permissions will same as the host machine
+	 DOCKER_OPTIONS+=" --build-arg UID=$(id -u) "
+	diff --git a/scripts/core-utils/clone-linux.sh b/scripts/core-utils/clone-linux.sh
+	index 714162e..2b79bc7 100755
+	--- a/scripts/core-utils/clone-linux.sh
+	+++ b/scripts/core-utils/clone-linux.sh
+	@@ -26,8 +26,7 @@ cd ${LINUX_DIR}/src
+	 LINUX_VER="v6.5"
+	 echo -e "\nCloning Linux ${LINUX_VER}:"
+	 git clone \
+	-    --depth=1 --progress -c advice.detachedHead=false \
+	-    -b ${LINUX_VER}  \
+	+    --progress -c advice.detachedHead=false \
+	     https://github.com/torvalds/linux.git   || {
+	 	echo "Unable to clone Linux"
+	 	return
+	@@ -58,7 +57,11 @@ echo "Installed b4 to ${LINUX_DIR}/tools/b4"
+	 
+	 cd ${LINUX_DIR}/src/linux
+	 
+	-${LINUX_DIR}/tools/b4/b4.sh shazam https://lore.kernel.org/all/20230613172054.3959700-1-quic_eberman@quicinc.com/
+	+
+	+${LINUX_DIR}/tools/b4/b4.sh am https://lore.kernel.org/all/20230613172054.3959700-1-quic_eberman@quicinc.com/
+	+git checkout -b v14_20230613_quic_eberman_quicinc_com 858fd168a95c5b9669aac8db6c14a9aeab446375
+	+git am ./v14_20230613_quic_eberman_drivers_for_gunyah_hypervisor.mbx
+	+
+	 echo "Applied gunyah drivers patch successfully"
+	 
+	 echo "Generate gunyah.config"
+	diff --git a/scripts/dockerfile-hyp b/scripts/dockerfile-hyp
+	index f117290..29533b3 100644
+	--- a/scripts/dockerfile-hyp
+	+++ b/scripts/dockerfile-hyp
+	@@ -124,7 +124,7 @@ RUN echo ""                                           >> "${HOME}/.bashrc" && \
+	     echo                                           "" >> "${HOME}/.bashrc"
+	 
+	 ## Add any .bashrc user customizations to .bashrc
+	-COPY bashrc-extn .
+	+COPY --chown=$USER:$USER bashrc-extn .
+	 RUN cat bashrc-extn >> "${HOME}/.bashrc"  &&  rm -f bashrc-extn
+	 
+	 CMD /bin/bash
+	diff --git a/scripts/install-core-tools.sh b/scripts/install-core-tools.sh
+	index dc7a846..95c9d54 100755
+	--- a/scripts/install-core-tools.sh
+	+++ b/scripts/install-core-tools.sh
+	@@ -23,7 +23,8 @@ fi
+	 
+	 if [[ ! -d ${TOOLS_DIR} ]]; then
+	     echo "Creating tools mount folder"
+	-    mkdir -p ${TOOLS_DIR}
+	+    sudo mkdir -p ${TOOLS_DIR}
+	+    sudo chown $USER:$USER ${TOOLS_DIR}
+	 
+	     touch ${TOOLS_DIR}/.tools-env
+	     chmod 0775 ${TOOLS_DIR}/.tools-env
+	diff --git a/scripts/install-wsp-imgs.sh b/scripts/install-wsp-imgs.sh
+	index 12150f3..32107e0 100755
+	--- a/scripts/install-wsp-imgs.sh
+	+++ b/scripts/install-wsp-imgs.sh
+	@@ -100,15 +100,23 @@ if [[ ! -f ${WORKSPACE}/run-qemu.sh ]]; then
+	     cp ${BASE_DIR}/utils/run-qemu.sh ${WORKSPACE}/run-qemu.sh
+	 fi
+	 
+	-if [[ ! -f ${WORKSPACE}/crosvm/crosvm ]]; then
+	-    mkdir -p ${WORKSPACE}/crosvm
+	-    cd ${WORKSPACE}/crosvm
+	-    . clone-crosvm.sh
+	-    . build-crosvm.sh
+	-
+	-    echo -e 'export CROSVM_FILE_PATH=${WORKSPACE}/crosvm/crosvm' >> ${WORKSPACE}/.wsp-env
+	-    . ${WORKSPACE}/.wsp-env
+	-fi
+	+cp ${BASE_DIR}/utils/qemu-gunyah ${WORKSPACE}/
+	+cp ${BASE_DIR}/utils/efi-virtio.rom ${WORKSPACE}/
+	+cp ${BASE_DIR}/utils/en-us ${WORKSPACE}/
+	+cp ${BASE_DIR}/utils/svm_disk.img ${WORKSPACE}/
+	+
+	+#if [[ ! -f ${WORKSPACE}/crosvm/crosvm ]]; then
+	+#    mkdir -p ${WORKSPACE}/crosvm
+	+#    cd ${WORKSPACE}/crosvm
+	+#    . clone-crosvm.sh
+	+#    . build-crosvm.sh
+	+
+	+#    echo -e 'export CROSVM_FILE_PATH=${WORKSPACE}/crosvm/crosvm' >> ${WORKSPACE}/.wsp-env
+	+#    . ${WORKSPACE}/.wsp-env
+	+#fi
+	+
+	+echo -e 'export CROSVM_FILE_PATH=${WORKSPACE}/qemu-gunyah' >> ${WORKSPACE}/.wsp-env
+	+. ${WORKSPACE}/.wsp-env
+	 
+	 if [[ ! -f ${WORKSPACE}/rootfs/rootfs-extfs-disk.img ]]; then
+	     echo -e "\nrootfs image not found, creating new one"
+	diff --git a/scripts/migrate-tools-to-vol.sh b/scripts/migrate-tools-to-vol.sh
+	index e5240c6..330f807 100755
+	--- a/scripts/migrate-tools-to-vol.sh
+	+++ b/scripts/migrate-tools-to-vol.sh
+	@@ -76,14 +76,14 @@ if [[ ! -d ${WORKSPACE}/linux ]]; then
+	     echo "Done copying linux files"
+	 fi
+	 
+	-if [[ -d ~/share/docker-share/crosvm ]]; then
+	-    mv ~/share/docker-share/crosvm ${WORKSPACE}/
+	-    echo "Found crosvm, moved into workspace folder"
+	-    mv ${WORKSPACE}/crosvm/crosvm ${WORKSPACE}/crosvm/crosvm-src
+	-    cp ${WORKSPACE}/crosvm/crosvm-src/crosvm  ${WORKSPACE}/crosvm/crosvm
+	-    rm -rf ${WORKSPACE}/crosvm/crosvm-src
+	-    echo -e 'export CROSVM_FILE_PATH=${WORKSPACE}/crosvm/crosvm' >> ${WORKSPACE}/.wsp-env
+	-fi
+	+#if [[ -d ~/share/docker-share/crosvm ]]; then
+	+#    mv ~/share/docker-share/crosvm ${WORKSPACE}/
+	+#    echo "Found crosvm, moved into workspace folder"
+	+#    mv ${WORKSPACE}/crosvm/crosvm ${WORKSPACE}/crosvm/crosvm-src
+	+#    cp ${WORKSPACE}/crosvm/crosvm-src/crosvm  ${WORKSPACE}/crosvm/crosvm
+	+#    rm -rf ${WORKSPACE}/crosvm/crosvm-src
+	+#    echo -e 'export CROSVM_FILE_PATH=${WORKSPACE}/crosvm/crosvm' >> ${WORKSPACE}/.wsp-env
+	+#fi
+	 
+	 if [[ -d ~/share/docker-share/rootfs ]]; then
+	     mv ~/share/docker-share/rootfs ${WORKSPACE}/
+	diff --git a/scripts/utils/build-rootfs-img.sh b/scripts/utils/build-rootfs-img.sh
+	index d110965..9ffe530 100755
+	--- a/scripts/utils/build-rootfs-img.sh
+	+++ b/scripts/utils/build-rootfs-img.sh
+	@@ -177,6 +177,9 @@ if [[ ! -f ${SVM_DESTINATION}/svm.sh ]]; then
+	 	echo -e '--params "rw root=/dev/ram rdinit=/sbin/init earlyprintk=serial panic=0" \\' >> ./svm.sh
+	 	echo -e ' /usr/gunyah/Image $@\n' >> ./svm.sh
+	 
+	+	sudo cp ${WORKSPACE}/svm_disk.img ${SVM_DESTINATION}
+	+	sudo cp ${WORKSPACE}/efi-virtio.rom ${SVM_DESTINATION}
+	+	sudo cp ${WORKSPACE}/en-us ${SVM_DESTINATION}
+	 	sudo cp ./svm.sh ${SVM_DESTINATION}
+	 	rm -f ./svm.sh
+	 	sudo chmod 0775 ${SVM_DESTINATION}/svm.sh
+	@@ -216,13 +219,15 @@ if [[ ! -f ${ROOTFS_REFERENCE_DIR}/lib/libgcc_s.so.1 ]]; then
+	 	export MACHINE=qemuarm64
+	 	export DISTRO=rpb
+	 
+	-	mkdir ${ROOTFS_BASE}/oe-rpb
+	+	mkdir -p ${ROOTFS_BASE}/oe-rpb
+	 	cd ${ROOTFS_BASE}/oe-rpb
+	 
+	 	# fetch
+	 	~/bin/repo init -u https://github.com/96boards/oe-rpb-manifest.git -b qcom/master
+	 	~/bin/repo sync
+	 
+	+	rm layers/meta-qcom/recipes-kernel/linux/linux-yocto_6.5.bbappend
+	+
+	 	# add config for libgcc and other virtualization options
+	 	echo -e "\n" > ./extra_local.conf
+	 	echo "INHERIT += 'buildstats buildstats-summary'" >> ./extra_local.conf
+	@@ -269,5 +274,5 @@ if [[ -f ${WORKSPACE}/rootfs/rootfs-extfs-disk.img ]]; then
+	 else
+	 	echo "Creating rootfs image file from reference : `pwd`"
+	 	cd ${WORKSPACE}/rootfs
+	-	. ~/utils/bldextfs.sh -f ${WORKSPACE}/rootfs/reference -o ${WORKSPACE}/rootfs/rootfs-extfs-disk.img -s 800M
+	+	. ~/utils/bldextfs.sh -f ${WORKSPACE}/rootfs/reference -o ${WORKSPACE}/rootfs/rootfs-extfs-disk.img -s 2G
+	 fi
+
+Copy Qemu files
+```````````````
+
+Copy Qemu and related files to `utils` directory of gunyah-support scripts.
+
+.. code-block:: bash
+
+        # qemu-gunyah is nothing but qemu-system-aarch64 binary that supports gunyah accelerator
+        cp qemu-gunyah scripts/utils
+
+        # efi-virtio.rom is found under `pc-bios` directory of Qemu
+        cp efi-virtio.rom scripts/utils
+
+        # en-us is found under `pc-bios/keymaps` directory of Qemu
+        cp en-us scripts/utils
+
+        # svm_disk.img will serve as the root disk for VM. It will have init and
+        # other programs that are required to boot VM. It can be prepared from
+        # any aarch64-based distro such as Ubuntu.
+        cp svm_disk.img scripts/utils
+
+Build docker image
+``````````````````
+
+.. code-block:: bash
+
+        cd scripts
+        ./build-docker-img.sh
+
+Rest of steps below need to be run inside docker. Launch the docker as:
+
+.. code-block:: bash
+
+        # SOME_FOLDER is any directory on host. This will be accessible from
+        # inside docker and is useful to share files between host and docker
+        # environments.
+        export HOST_TO_DOCKER_SHARED_DIR=SOME_FOLDER
+        cd scripts
+        ./run-docker.sh
+
+
+Clone and build a Gunyah Hypervisor image
+`````````````````````````````````````````
+
+.. code-block:: bash
+
+        cd ~/share/gunyah
+        clone-gunyah.sh
+
+Cloned sources includes that for Resource Manager (RM) under `resource-manager`
+directory. RM is a privileged VM that acts as an extension of Gunyah
+hypervisor and assists the hypervisor in various tasks related to creation and
+management of VMs. More information on RM is provided at:
+
+https://github.com/quic/gunyah-resource-manager
+
+Gunyah hypervisor source is available under `hyp` directory.
+
+
+Patch Gunyah hypervisor and Resource Manager
+````````````````````````````````````````````
+
+Apply below changes to hypervisor and RM on which 'gunyah' Qemu accelerator
+currently depends. These changes are being discussed with maintainers and if
+accepted this document will be modified appropriately.
+
+RM patch (in 'resource-manager' directory):
+
+.. code-block:: bash
+
+	diff --git a/src/vm_creation/vm_creation.c b/src/vm_creation/vm_creation.c
+	index df8edfb..b73b37e 100644
+	--- a/src/vm_creation/vm_creation.c
+	+++ b/src/vm_creation/vm_creation.c
+	@@ -510,7 +510,10 @@ process_dtb(vm_t *vm)
+	        // Estimate a final dtb size after applying the overlay.
+	        size_t original_dtb_size =
+	                util_balign_up(fdt_totalsize(temp_addr), sizeof(uint32_t));
+	-       size_t final_dtb_size = original_dtb_size + dtbo_ret.size;
+	+       size_t final_dtb_size = util_balign_up(original_dtb_size + dtbo_ret.size, 8);
+
+
+Hypervisor patch (in 'hyp' directory):
+
+.. code-block:: bash
+
+	diff --git a/config/platform/qemu.conf b/config/platform/qemu.conf
+	index bc612f2..9a292a4 100644
+	--- a/config/platform/qemu.conf
+	+++ b/config/platform/qemu.conf
+	@@ -35,7 +35,7 @@ configs HLOS_RAM_FS_BASE=0x40800000
+	 configs PLATFORM_HEAP_PRIVATE_SIZE=0x200000
+	 configs PLATFORM_RW_DATA_SIZE=0x200000
+	 configs PLATFORM_ROOTVM_LMA_BASE=0x80480000U
+	-configs PLATFORM_ROOTVM_LMA_SIZE=0xa0000U
+	+configs PLATFORM_ROOTVM_LMA_SIZE=0x100000U
+	 configs PLATFORM_PHYS_ADDRESS_BITS=36
+	 configs PLATFORM_VM_ADDRESS_SPACE_BITS=36
+	 configs PLATFORM_PGTABLE_4K_GRANULE=1
+
+Build Gunyah hypervisor
+```````````````````````
+
+.. code-block:: bash
+
+        cd ~/share
+        build-gunyah.sh qemu
+
+Launch host-VM under Gunyah hypervisor
+``````````````````````````````````````
+
+.. code-block:: bash
+
+        cd ~/mnt/workspace
+        run-qemu.sh dtb
+        run-qemu.sh
+
+
+Running a secondary VM with Qemu as VMM
+```````````````````````````````````````
+
+.. code-block:: bash
+
+        $ cd /usr/gunyah
+        $ ./qemu-gunyah -cpu cortex-a57 -nographic -hda svm_disk.img -m 256M -smp cpus=8 --accel gunyah -machine virt,highmem=off -append "rw root=/dev/vda   rdinit=/sbin/init earlyprintk=serial panic=0" -kernel ./Image
+
+
+Limitations
+-----------
+
+Below features are not yet supported.
+
+* Protected VM (or confidential guests)
-- 
2.25.1

Re: [RFC/PATCH v1 07/11] gunyah: Specify device-tree location

[Philippe Mathieu-Daudé]

Hi Srivatsa,

On 9/1/24 10:00, Srivatsa Vaddagiri wrote:
> Specify the location of device-tree and its size, as Gunyah requires the
> device-tree to be parsed before VM can begin its execution.
> 
> Signed-off-by: Srivatsa Vaddagiri <quic_svaddagi@quicinc.com>
> ---
>   MAINTAINERS               |  1 +
>   accel/stubs/gunyah-stub.c |  5 +++++
>   hw/arm/virt.c             |  6 ++++++
>   include/sysemu/gunyah.h   |  2 ++
>   target/arm/gunyah.c       | 45 +++++++++++++++++++++++++++++++++++++++
>   target/arm/meson.build    |  3 +++
>   6 files changed, 62 insertions(+)
>   create mode 100644 target/arm/gunyah.c

(Please enable scripts/git.orderfile)

> diff --git a/include/sysemu/gunyah.h b/include/sysemu/gunyah.h
> index 4f26938521..a73d17bfb9 100644
> --- a/include/sysemu/gunyah.h
> +++ b/include/sysemu/gunyah.h
> @@ -27,4 +27,6 @@ typedef struct GUNYAHState GUNYAHState;
>   DECLARE_INSTANCE_CHECKER(GUNYAHState, GUNYAH_STATE,
>                            TYPE_GUNYAH_ACCEL)
>   
> +int gunyah_arm_set_dtb(__u64 dtb_start, __u64 dtb_size);
I'm getting:

In file included from hw/intc/arm_gicv3_common.c:35:
include/sysemu/gunyah.h:30:24: error: unknown type name '__u64'
int gunyah_arm_set_dtb(__u64 dtb_start, __u64 dtb_size);
                        ^
include/sysemu/gunyah.h:30:41: error: unknown type name '__u64'
int gunyah_arm_set_dtb(__u64 dtb_start, __u64 dtb_size);
                                         ^
2 errors generated.

Re: [RFC/PATCH v1 07/11] gunyah: Specify device-tree location

[Philippe Mathieu-Daudé]

On 9/1/24 10:00, Srivatsa Vaddagiri wrote:
> Specify the location of device-tree and its size, as Gunyah requires the
> device-tree to be parsed before VM can begin its execution.
> 
> Signed-off-by: Srivatsa Vaddagiri <quic_svaddagi@quicinc.com>
> ---
>   MAINTAINERS               |  1 +
>   accel/stubs/gunyah-stub.c |  5 +++++
>   hw/arm/virt.c             |  6 ++++++
>   include/sysemu/gunyah.h   |  2 ++
>   target/arm/gunyah.c       | 45 +++++++++++++++++++++++++++++++++++++++
>   target/arm/meson.build    |  3 +++
>   6 files changed, 62 insertions(+)
>   create mode 100644 target/arm/gunyah.c


> diff --git a/target/arm/gunyah.c b/target/arm/gunyah.c
> new file mode 100644
> index 0000000000..73c1c2a88a
> --- /dev/null
> +++ b/target/arm/gunyah.c
> @@ -0,0 +1,45 @@
> +/*
> + * QEMU Gunyah hypervisor support
> + *
> + * Copyright(c) 2023 Qualcomm Innovation Center, Inc. All Rights Reserved.
> + *
> + * SPDX-License-Identifier: GPL-2.0-or-later
> + */
> +
> +#include "qemu/osdep.h"
> +#include "qemu/error-report.h"
> +#include "sysemu/gunyah.h"
> +#include "sysemu/gunyah_int.h"
> +#include "linux-headers/linux/gunyah.h"
I'm getting on macOS:

In file included from ../../target/arm/gunyah.c:13:
linux-headers/linux/gunyah.h:13:10: fatal error: 'linux/types.h' file 
not found
#include <linux/types.h>
          ^~~~~~~~~~~~~~~
1 error generated.

Maybe we need the following change?

-- >8 --
diff --git a/meson.build b/meson.build
index 6ded60063e..fb20ca04d1 100644
--- a/meson.build
+++ b/meson.build
@@ -712,5 +712,5 @@ endif

  gunyah = not_found
-if get_option('gunyah').allowed()
+if get_option('gunyah').allowed() and host_os == 'linux'
      accelerators += 'CONFIG_GUNYAH'
  endif
---

Re: [RFC/PATCH v1 07/11] gunyah: Specify device-tree location

[Srivatsa Vaddagiri]

* Philippe Mathieu-Daud? <philmd@linaro.org> [2024-01-09 14:31:03]:

> Hi Srivatsa,
> 
> On 9/1/24 10:00, Srivatsa Vaddagiri wrote:
> > Specify the location of device-tree and its size, as Gunyah requires the
> > device-tree to be parsed before VM can begin its execution.
> > 
> > Signed-off-by: Srivatsa Vaddagiri <quic_svaddagi@quicinc.com>
> > ---
> >   MAINTAINERS               |  1 +
> >   accel/stubs/gunyah-stub.c |  5 +++++
> >   hw/arm/virt.c             |  6 ++++++
> >   include/sysemu/gunyah.h   |  2 ++
> >   target/arm/gunyah.c       | 45 +++++++++++++++++++++++++++++++++++++++
> >   target/arm/meson.build    |  3 +++
> >   6 files changed, 62 insertions(+)
> >   create mode 100644 target/arm/gunyah.c
> 
> (Please enable scripts/git.orderfile)

Sure will do so from the next version!

> 
> > diff --git a/include/sysemu/gunyah.h b/include/sysemu/gunyah.h
> > index 4f26938521..a73d17bfb9 100644
> > --- a/include/sysemu/gunyah.h
> > +++ b/include/sysemu/gunyah.h
> > @@ -27,4 +27,6 @@ typedef struct GUNYAHState GUNYAHState;
> >   DECLARE_INSTANCE_CHECKER(GUNYAHState, GUNYAH_STATE,
> >                            TYPE_GUNYAH_ACCEL)
> > +int gunyah_arm_set_dtb(__u64 dtb_start, __u64 dtb_size);
> I'm getting:
> 
> In file included from hw/intc/arm_gicv3_common.c:35:
> include/sysemu/gunyah.h:30:24: error: unknown type name '__u64'
> int gunyah_arm_set_dtb(__u64 dtb_start, __u64 dtb_size);
>                        ^
> include/sysemu/gunyah.h:30:41: error: unknown type name '__u64'
> int gunyah_arm_set_dtb(__u64 dtb_start, __u64 dtb_size);
>                                         ^
> 2 errors generated.

Hmm I don't get that error when compiling on Linux. I think uint64_t will work
better for all platforms where Qemu can get compiled?

- vatsa

Re: [RFC/PATCH v1 07/11] gunyah: Specify device-tree location

[Srivatsa Vaddagiri]

* Philippe Mathieu-Daud? <philmd@linaro.org> [2024-01-09 14:36:12]:

> > +#include "qemu/osdep.h"
> > +#include "qemu/error-report.h"
> > +#include "sysemu/gunyah.h"
> > +#include "sysemu/gunyah_int.h"
> > +#include "linux-headers/linux/gunyah.h"
> I'm getting on macOS:
> 
> In file included from ../../target/arm/gunyah.c:13:
> linux-headers/linux/gunyah.h:13:10: fatal error: 'linux/types.h' file not
> found
> #include <linux/types.h>
>          ^~~~~~~~~~~~~~~
> 1 error generated.
> 
> Maybe we need the following change?
> 
> -- >8 --
> diff --git a/meson.build b/meson.build
> index 6ded60063e..fb20ca04d1 100644
> --- a/meson.build
> +++ b/meson.build
> @@ -712,5 +712,5 @@ endif
> 
>  gunyah = not_found
> -if get_option('gunyah').allowed()
> +if get_option('gunyah').allowed() and host_os == 'linux'
>      accelerators += 'CONFIG_GUNYAH'
>  endif

Ok sure will include that change in my next version.

- vatsa

Re: [RFC/PATCH v1 09/11] gunyah: CPU execution loop

[Philippe Mathieu-Daudé]

Hi Srivatsa,

On 9/1/24 10:00, Srivatsa Vaddagiri wrote:
> Complete the cpu execution loop. At this time, we recognize exits
> associated with only MMIO access. Future patches will add support for
> recognizing other exit reasons, such as PSCI calls made by guest.
> 
> Signed-off-by: Srivatsa Vaddagiri <quic_svaddagi@quicinc.com>
> ---
>   accel/gunyah/gunyah-accel-ops.c |   7 ++
>   accel/gunyah/gunyah-all.c       | 201 +++++++++++++++++++++++++++++++-
>   include/hw/core/cpu.h           |   1 +
>   include/sysemu/gunyah_int.h     |   9 ++
>   target/arm/gunyah.c             |  13 +++
>   5 files changed, 230 insertions(+), 1 deletion(-)


> diff --git a/include/hw/core/cpu.h b/include/hw/core/cpu.h
> index 238c02c05e..ef248a658b 100644
> --- a/include/hw/core/cpu.h
> +++ b/include/hw/core/cpu.h
> @@ -371,6 +371,7 @@ struct CPUWatchpoint {
>   
>   struct KVMState;
>   struct kvm_run;
> +struct gh_vcpu_run;

Please do not forward-declare this structure here, it is not required.

>   
>   /* work queue */
>   
> diff --git a/include/sysemu/gunyah_int.h b/include/sysemu/gunyah_int.h
> index dc5b4847a9..72b3027e4e 100644
> --- a/include/sysemu/gunyah_int.h
> +++ b/include/sysemu/gunyah_int.h
> @@ -46,12 +46,21 @@ struct GUNYAHState {
>       bool preshmem_reserved;
>       uint32_t preshmem_size;
>       uint32_t nr_irqs;
> +    uint32_t vm_started;
> +};
> +
> +struct AccelCPUState {
> +    int fd;
> +    struct gh_vcpu_run *run;
>   };
>   
>   int gunyah_create_vm(void);
> +void gunyah_start_vm(void);
>   int gunyah_vm_ioctl(int type, ...);
>   void *gunyah_cpu_thread_fn(void *arg);
>   int gunyah_add_irqfd(int irqfd, int label, Error **errp);
>   GUNYAHState *get_gunyah_state(void);
> +int gunyah_arch_put_registers(CPUState *cs, int level);
> +void gunyah_cpu_synchronize_post_reset(CPUState *cpu);
>   
>   #endif    /* GUNYAH_INT_H */

Re: [RFC/PATCH v1 09/11] gunyah: CPU execution loop

[Philippe Mathieu-Daudé]

Hi Srivatsa,

On 9/1/24 10:00, Srivatsa Vaddagiri wrote:
> Complete the cpu execution loop. At this time, we recognize exits
> associated with only MMIO access. Future patches will add support for
> recognizing other exit reasons, such as PSCI calls made by guest.
> 
> Signed-off-by: Srivatsa Vaddagiri <quic_svaddagi@quicinc.com>
> ---
>   accel/gunyah/gunyah-accel-ops.c |   7 ++
>   accel/gunyah/gunyah-all.c       | 201 +++++++++++++++++++++++++++++++-
>   include/hw/core/cpu.h           |   1 +
>   include/sysemu/gunyah_int.h     |   9 ++
>   target/arm/gunyah.c             |  13 +++
>   5 files changed, 230 insertions(+), 1 deletion(-)


> diff --git a/include/hw/core/cpu.h b/include/hw/core/cpu.h
> index 238c02c05e..ef248a658b 100644
> --- a/include/hw/core/cpu.h
> +++ b/include/hw/core/cpu.h
> @@ -371,6 +371,7 @@ struct CPUWatchpoint {
>   
>   struct KVMState;
>   struct kvm_run;
> +struct gh_vcpu_run;

Please do not forward-declare this structure here, it is not required.

>   
>   /* work queue */
>   
> diff --git a/include/sysemu/gunyah_int.h b/include/sysemu/gunyah_int.h
> index dc5b4847a9..72b3027e4e 100644
> --- a/include/sysemu/gunyah_int.h
> +++ b/include/sysemu/gunyah_int.h
> @@ -46,12 +46,21 @@ struct GUNYAHState {
>       bool preshmem_reserved;
>       uint32_t preshmem_size;
>       uint32_t nr_irqs;
> +    uint32_t vm_started;
> +};
> +
> +struct AccelCPUState {
> +    int fd;
> +    struct gh_vcpu_run *run;
>   };
>   
>   int gunyah_create_vm(void);
> +void gunyah_start_vm(void);
>   int gunyah_vm_ioctl(int type, ...);
>   void *gunyah_cpu_thread_fn(void *arg);
>   int gunyah_add_irqfd(int irqfd, int label, Error **errp);
>   GUNYAHState *get_gunyah_state(void);
> +int gunyah_arch_put_registers(CPUState *cs, int level);
> +void gunyah_cpu_synchronize_post_reset(CPUState *cpu);
>   
>   #endif    /* GUNYAH_INT_H */

Re: [RFC/PATCH v1 07/11] gunyah: Specify device-tree location

[Alex Bennée]

Srivatsa Vaddagiri <quic_svaddagi@quicinc.com> writes:

> * Philippe Mathieu-Daud? <philmd@linaro.org> [2024-01-09 14:31:03]:
>
>> Hi Srivatsa,
>> 
>> On 9/1/24 10:00, Srivatsa Vaddagiri wrote:
>> > Specify the location of device-tree and its size, as Gunyah requires the
>> > device-tree to be parsed before VM can begin its execution.
>> > 
>> > Signed-off-by: Srivatsa Vaddagiri <quic_svaddagi@quicinc.com>
>> > ---
>> >   MAINTAINERS               |  1 +
>> >   accel/stubs/gunyah-stub.c |  5 +++++
>> >   hw/arm/virt.c             |  6 ++++++
>> >   include/sysemu/gunyah.h   |  2 ++
>> >   target/arm/gunyah.c       | 45 +++++++++++++++++++++++++++++++++++++++
>> >   target/arm/meson.build    |  3 +++
>> >   6 files changed, 62 insertions(+)
>> >   create mode 100644 target/arm/gunyah.c
>> 
>> (Please enable scripts/git.orderfile)
>
> Sure will do so from the next version!
>
>> 
>> > diff --git a/include/sysemu/gunyah.h b/include/sysemu/gunyah.h
>> > index 4f26938521..a73d17bfb9 100644
>> > --- a/include/sysemu/gunyah.h
>> > +++ b/include/sysemu/gunyah.h
>> > @@ -27,4 +27,6 @@ typedef struct GUNYAHState GUNYAHState;
>> >   DECLARE_INSTANCE_CHECKER(GUNYAHState, GUNYAH_STATE,
>> >                            TYPE_GUNYAH_ACCEL)
>> > +int gunyah_arm_set_dtb(__u64 dtb_start, __u64 dtb_size);
>> I'm getting:
>> 
>> In file included from hw/intc/arm_gicv3_common.c:35:
>> include/sysemu/gunyah.h:30:24: error: unknown type name '__u64'
>> int gunyah_arm_set_dtb(__u64 dtb_start, __u64 dtb_size);
>>                        ^
>> include/sysemu/gunyah.h:30:41: error: unknown type name '__u64'
>> int gunyah_arm_set_dtb(__u64 dtb_start, __u64 dtb_size);
>>                                         ^
>> 2 errors generated.
>
> Hmm I don't get that error when compiling on Linux. I think uint64_t will work
> better for all platforms where Qemu can get compiled?

Yes, aside from imported headers we state:

  In the event that you require a specific width, use a standard type
  like int32_t, uint32_t, uint64_t, etc.  The specific types are
  mandatory for VMState fields.

  Don't use Linux kernel internal types like u32, __u32 or __le32.

in style.rst

>
> - vatsa

-- 
Alex Bennée
Virtualisation Tech Lead @ Linaro