[PATCH v3 00/14] macOS PV Graphics and new vmapple machine type

[PATCH v3 00/14] macOS PV Graphics and new vmapple machine type

[Phil Dennis-Jordan]

(Apologies to anyone who has received more than one version of this
series of emails; my git-send-email was misconfigured and this is
a new attempt.)

This patch set introduces a new ARM and macOS HVF specific machine type
called "vmapple", as well as a family of display devices based on the
ParavirtualizedGraphics.framework in macOS. One of the display adapter
variants, apple-gfx-vmapple, is required for the new machine type, while
apple-gfx-pci can be used to enable 3D graphics acceleration with x86-64
macOS guest OSes.

Previous versions of this patch set were submitted semi-separately:
the original vmapple patch set by Alexander Graf included a monolithic
implementation of apple-gfx-vmapple. I subsequently reviewed and reworked
the latter to support the PCI variant of the device as well and submitted
the result in isolation. As requested in subsequent review, I have now
recombined this with the original vmapple patch set, which I have updated
and improved in a few ways as well.

The vmapple machine type approximates the configuration in macOS's own
Virtualization.framework when running arm64 macOS guests. In addition to
generic components such as a GICv3 and an XHCI USB controller, it
includes nonstandard extensions to the virtio block device, a special
"hardware" aes engine, a configuration device, a pvpanic variant, a
"backdoor" interface, and of course the apple-gfx paravirtualised display
adapter.

There are currently a few limitations to this which aren't intrinsic,
just imperfect emulation of the VZF, but it's good enough to be just
about usable for some purposes:

 * macOS 12 guests only. Versions 13+ currently fail during early boot.
 * macOS 11+ arm64 hosts only, with hvf accel. (Perhaps some differences
   between Apple M series CPUs and TCG's aarch64 implementation? macOS
   hosts only because ParavirtualizedGraphics.framework is a black box
   implementing most of the logic behind the apple-gfx device.)
 * PCI devices use legacy IRQs, not MSI/MSI-X. As far as I can tell,
   we'd need to include the GICv3 ITS, but it's unclear to me what
   exactly needs wiring up.
 * Due to lack of MSI(-X), event delivery from USB devices to the guest
   macOS isn't working correctly. My current conclusion is that the
   OS's XHCI driver simply was never designed to work with legacy IRQs.
   The upshot is that keyboard and mouse/tablet input is very laggy.
   The solution would be to implement MSI(-X) support or figure out how
   to make hcd-xhci-sysbus work with the macOS guest, if at all possible.
   (EHCI and UHCI/OHCI controllers are not an option as the VMAPPLE
   guest kernel does not include drivers for these.)
 * The guest OS must first be provisioned using Virtualization.framework;
   the disk images can subsequently be used in Qemu. (See docs.)

The apple-gfx device can be used independently from the vmapple machine
type, at least in the PCI variant. It mainly targets x86-64 macOS guests
from version 11 on, but also includes a UEFI bootrom for basic
framebuffer mode. macOS 11 is also required on the host side, as well
as a GPU that supports the Metal API. On the guest side, this provides
3D acceleration/GPGPU support with a baseline Metal feature set,
irrespective of the host GPU's feature set. A few limitations in the
current integration:

 * Although it works fine with TCG, it does not work correctly
   cross-architecture: x86-64 guests on arm64 hosts appear to make
   some boot progress, but rendering is corrupted. I suspect
   incompatible texture memory layouts; I have no idea if this is
   fixable.
 * ParavirtualizedGraphics.framework and the guest driver support
   multi-headed configurations. The current Qemu integration always
   connects precisely 1 display.
 * State serialisation and deserialisation is currently not
   implemented, though supported in principle by the framework.
   Both apple-gfx variants thus set up a migration blocker.
 * Rendering efficiency could be better. The GPU-rendered guest
   framebuffer is copied to system memory and uses Qemu's usual
   CPU-based drawing. For maximum efficiency, the Metal texture
   containing the guest framebuffer could be drawn directly to
   a Metal view in the host window, staying on the GPU. (Similar
   to the OpenGL/virgl render path on other platforms.)

---

v2 -> v3:

 * Merged the apple-gfx and vmapple patchsets.
 * Squashed a bunch of later apple-gfx patches into the main one.
   (dGPU support, queried MMIO area size, host GPU picking logic.)
 * Rebased on latest upstream, fixing any breakages due to internal
   Qemu API changes.
 * apple-gfx: Switched to re-entrant MMIO. This is supported by the
   underlying framework and simplifies the MMIO forwarding code which
   was previously different on x86-64 vs aarch64.
 * vmapple: Fixes for minor bugs and comments from the last round of
   review.
 * vmapple aes, conf, apple-gfx: Switched reset methods to implement
   the ResettableClass base's interface.
 * vmapple: switched from virtio-hid to an XHCI USB controller and
   USB mouse and tablet devices. macOS does not provide drivers for
   virtio HID devices, at least not in version 12's vmapple kernel.
   So input now sort of works (interrupt issues) rather than not
   at all. Use network-based remote access to the guest OS as a
   work-around.


Alexander Graf (9):
  hw: Add vmapple subdir
  hw/misc/pvpanic: Add MMIO interface
  hvf: arm: Ignore writes to CNTP_CTL_EL0
  gpex: Allow more than 4 legacy IRQs
  hw/vmapple/aes: Introduce aes engine
  hw/vmapple/bdif: Introduce vmapple backdoor interface
  hw/vmapple/cfg: Introduce vmapple cfg region
  hw/vmapple/virtio-blk: Add support for apple virtio-blk
  hw/vmapple/vmapple: Add vmapple machine type

Phil Dennis-Jordan (5):
  hw/display/apple-gfx: Introduce ParavirtualizedGraphics.Framework
    support
  hw/display/apple-gfx: Adds PCI implementation
  ui/cocoa: Adds non-app runloop on main thread mode
  hw/display/apple-gfx: Adds configurable mode list
  MAINTAINERS: Add myself as maintainer for apple-gfx, reviewer for HVF

 MAINTAINERS                     |  15 +
 docs/system/arm/vmapple.rst     |  63 +++
 docs/system/target-arm.rst      |   1 +
 hw/Kconfig                      |   1 +
 hw/arm/sbsa-ref.c               |   2 +-
 hw/arm/virt.c                   |   2 +-
 hw/block/virtio-blk.c           |  19 +-
 hw/display/Kconfig              |  14 +
 hw/display/apple-gfx-pci.m      | 179 +++++++++
 hw/display/apple-gfx-vmapple.m  | 215 ++++++++++
 hw/display/apple-gfx.h          |  72 ++++
 hw/display/apple-gfx.m          | 668 ++++++++++++++++++++++++++++++++
 hw/display/meson.build          |   3 +
 hw/display/trace-events         |  26 ++
 hw/i386/microvm.c               |   2 +-
 hw/loongarch/virt.c             |   2 +-
 hw/meson.build                  |   1 +
 hw/mips/loongson3_virt.c        |   2 +-
 hw/misc/Kconfig                 |   4 +
 hw/misc/meson.build             |   1 +
 hw/misc/pvpanic-mmio.c          |  61 +++
 hw/openrisc/virt.c              |  12 +-
 hw/pci-host/gpex.c              |  36 +-
 hw/riscv/virt.c                 |  12 +-
 hw/vmapple/Kconfig              |  32 ++
 hw/vmapple/aes.c                | 584 ++++++++++++++++++++++++++++
 hw/vmapple/bdif.c               | 245 ++++++++++++
 hw/vmapple/cfg.c                | 106 +++++
 hw/vmapple/meson.build          |   5 +
 hw/vmapple/trace-events         |  26 ++
 hw/vmapple/trace.h              |   1 +
 hw/vmapple/virtio-blk.c         | 212 ++++++++++
 hw/vmapple/vmapple.c            | 661 +++++++++++++++++++++++++++++++
 hw/xtensa/virt.c                |   2 +-
 include/hw/misc/pvpanic.h       |   1 +
 include/hw/pci-host/gpex.h      |   7 +-
 include/hw/pci/pci_ids.h        |   1 +
 include/hw/virtio/virtio-blk.h  |  12 +-
 include/hw/vmapple/bdif.h       |  31 ++
 include/hw/vmapple/cfg.h        |  68 ++++
 include/hw/vmapple/virtio-blk.h |  39 ++
 include/qemu-main.h             |   2 +
 meson.build                     |   5 +
 target/arm/hvf/hvf.c            |   9 +
 ui/cocoa.m                      |  15 +-
 45 files changed, 3443 insertions(+), 34 deletions(-)
 create mode 100644 docs/system/arm/vmapple.rst
 create mode 100644 hw/display/apple-gfx-pci.m
 create mode 100644 hw/display/apple-gfx-vmapple.m
 create mode 100644 hw/display/apple-gfx.h
 create mode 100644 hw/display/apple-gfx.m
 create mode 100644 hw/misc/pvpanic-mmio.c
 create mode 100644 hw/vmapple/Kconfig
 create mode 100644 hw/vmapple/aes.c
 create mode 100644 hw/vmapple/bdif.c
 create mode 100644 hw/vmapple/cfg.c
 create mode 100644 hw/vmapple/meson.build
 create mode 100644 hw/vmapple/trace-events
 create mode 100644 hw/vmapple/trace.h
 create mode 100644 hw/vmapple/virtio-blk.c
 create mode 100644 hw/vmapple/vmapple.c
 create mode 100644 include/hw/vmapple/bdif.h
 create mode 100644 include/hw/vmapple/cfg.h
 create mode 100644 include/hw/vmapple/virtio-blk.h

-- 
2.39.3 (Apple Git-145)

[PATCH v3 01/14] hw/display/apple-gfx: Introduce ParavirtualizedGraphics.Framework support

[Phil Dennis-Jordan]

MacOS provides a framework (library) that allows any vmm to implement a
paravirtualized 3d graphics passthrough to the host metal stack called
ParavirtualizedGraphics.Framework (PVG). The library abstracts away
almost every aspect of the paravirtualized device model and only provides
and receives callbacks on MMIO access as well as to share memory address
space between the VM and PVG.

This patch implements a QEMU device that drives PVG for the VMApple
variant of it.

Signed-off-by: Alexander Graf <graf@amazon.com>
Co-authored-by: Alexander Graf <graf@amazon.com>

Subsequent changes:

 * Cherry-pick/rebase conflict fixes
 * BQL function renaming
 * Moved from hw/vmapple/ (useful outside that machine type)
 * Code review comments: Switched to DEFINE_TYPES macro & little endian
   MMIO.
 * Removed some dead/superfluous code
 * Mad set_mode thread & memory safe
 * Added migration blocker due to lack of (de-)serialisation.
 * Fixes to ObjC refcounting and autorelease pool usage.
 * Fixed ObjC new/init misuse
 * Switched to ObjC category extension for private property.
 * Simplified task memory mapping and made it thread safe.
 * Refactoring to split generic and vmapple MMIO variant specific
   code.
 * Switched to asynchronous MMIO writes on x86-64
 * Rendering and graphics update are now done asynchronously
 * Fixed cursor handling
 * Coding convention fixes
 * Removed software cursor compositing

Signed-off-by: Phil Dennis-Jordan <phil@philjordan.eu>

---

v3:

 * Rebased on latest upstream, fixed breakages including switching to Resettable methods.
 * Squashed patches dealing with dGPUs, MMIO area size, and GPU picking.
 * Allow re-entrant MMIO; this simplifies the code and solves the divergence
   between x86-64 and arm64 variants.

 hw/display/Kconfig             |   9 +
 hw/display/apple-gfx-vmapple.m | 215 +++++++++++++
 hw/display/apple-gfx.h         |  57 ++++
 hw/display/apple-gfx.m         | 536 +++++++++++++++++++++++++++++++++
 hw/display/meson.build         |   2 +
 hw/display/trace-events        |  26 ++
 meson.build                    |   4 +
 7 files changed, 849 insertions(+)
 create mode 100644 hw/display/apple-gfx-vmapple.m
 create mode 100644 hw/display/apple-gfx.h
 create mode 100644 hw/display/apple-gfx.m

diff --git a/hw/display/Kconfig b/hw/display/Kconfig
index a4552c8ed78..179a479d220 100644
--- a/hw/display/Kconfig
+++ b/hw/display/Kconfig
@@ -143,3 +143,12 @@ config XLNX_DISPLAYPORT
 
 config DM163
     bool
+
+config MAC_PVG
+    bool
+    default y
+
+config MAC_PVG_VMAPPLE
+    bool
+    depends on MAC_PVG
+    depends on ARM
diff --git a/hw/display/apple-gfx-vmapple.m b/hw/display/apple-gfx-vmapple.m
new file mode 100644
index 00000000000..d8fc7651dde
--- /dev/null
+++ b/hw/display/apple-gfx-vmapple.m
@@ -0,0 +1,215 @@
+/*
+ * QEMU Apple ParavirtualizedGraphics.framework device, vmapple (arm64) variant
+ *
+ * Copyright © 2023 Amazon.com, Inc. or its affiliates. All Rights Reserved.
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or later.
+ * See the COPYING file in the top-level directory.
+ *
+ * ParavirtualizedGraphics.framework is a set of libraries that macOS provides
+ * which implements 3d graphics passthrough to the host as well as a
+ * proprietary guest communication channel to drive it. This device model
+ * implements support to drive that library from within QEMU as an MMIO-based
+ * system device for macOS on arm64 VMs.
+ */
+
+#include "apple-gfx.h"
+#include "monitor/monitor.h"
+#include "hw/sysbus.h"
+#include "hw/irq.h"
+#include "trace.h"
+#import <ParavirtualizedGraphics/ParavirtualizedGraphics.h>
+
+_Static_assert(__aarch64__, "");
+
+/*
+ * ParavirtualizedGraphics.Framework only ships header files for the PCI
+ * variant which does not include IOSFC descriptors and host devices. We add
+ * their definitions here so that we can also work with the ARM version.
+ */
+typedef bool(^IOSFCRaiseInterrupt)(uint32_t vector);
+typedef bool(^IOSFCUnmapMemory)(
+    void *a, void *b, void *c, void *d, void *e, void *f);
+typedef bool(^IOSFCMapMemory)(
+    uint64_t phys, uint64_t len, bool ro, void **va, void *e, void *f);
+
+@interface PGDeviceDescriptor (IOSurfaceMapper)
+@property (readwrite, nonatomic) bool usingIOSurfaceMapper;
+@end
+
+@interface PGIOSurfaceHostDeviceDescriptor : NSObject
+-(PGIOSurfaceHostDeviceDescriptor *)init;
+@property (readwrite, nonatomic, copy, nullable) IOSFCMapMemory mapMemory;
+@property (readwrite, nonatomic, copy, nullable) IOSFCUnmapMemory unmapMemory;
+@property (readwrite, nonatomic, copy, nullable) IOSFCRaiseInterrupt raiseInterrupt;
+@end
+
+@interface PGIOSurfaceHostDevice : NSObject
+-(instancetype)initWithDescriptor:(PGIOSurfaceHostDeviceDescriptor *) desc;
+-(uint32_t)mmioReadAtOffset:(size_t) offset;
+-(void)mmioWriteAtOffset:(size_t) offset value:(uint32_t)value;
+@end
+
+typedef struct AppleGFXVmappleState {
+    SysBusDevice parent_obj;
+
+    AppleGFXState common;
+
+    qemu_irq irq_gfx;
+    qemu_irq irq_iosfc;
+    MemoryRegion iomem_iosfc;
+    PGIOSurfaceHostDevice *pgiosfc;
+} AppleGFXVmappleState;
+
+OBJECT_DECLARE_SIMPLE_TYPE(AppleGFXVmappleState, APPLE_GFX_VMAPPLE)
+
+
+static uint64_t apple_iosfc_read(void *opaque, hwaddr offset, unsigned size)
+{
+    AppleGFXVmappleState *s = opaque;
+    uint64_t res = 0;
+
+    bql_unlock();
+    res = [s->pgiosfc mmioReadAtOffset:offset];
+    bql_lock();
+
+    trace_apple_iosfc_read(offset, res);
+
+    return res;
+}
+
+static void apple_iosfc_write(
+    void *opaque, hwaddr offset, uint64_t val, unsigned size)
+{
+    AppleGFXVmappleState *s = opaque;
+
+    trace_apple_iosfc_write(offset, val);
+
+    [s->pgiosfc mmioWriteAtOffset:offset value:val];
+}
+
+static const MemoryRegionOps apple_iosfc_ops = {
+    .read = apple_iosfc_read,
+    .write = apple_iosfc_write,
+    .endianness = DEVICE_LITTLE_ENDIAN,
+    .valid = {
+        .min_access_size = 4,
+        .max_access_size = 8,
+    },
+    .impl = {
+        .min_access_size = 4,
+        .max_access_size = 8,
+    },
+};
+
+static PGIOSurfaceHostDevice *apple_gfx_prepare_iosurface_host_device(
+    AppleGFXVmappleState *s)
+{
+    PGIOSurfaceHostDeviceDescriptor *iosfc_desc =
+        [PGIOSurfaceHostDeviceDescriptor new];
+    PGIOSurfaceHostDevice *iosfc_host_dev = nil;
+
+    iosfc_desc.mapMemory =
+        ^(uint64_t phys, uint64_t len, bool ro, void **va, void *e, void *f) {
+            trace_apple_iosfc_map_memory(phys, len, ro, va, e, f);
+            MemoryRegion *tmp_mr;
+            *va = gpa2hva(&tmp_mr, phys, len, NULL);
+            return (bool)true;
+        };
+
+    iosfc_desc.unmapMemory =
+        ^(void *a, void *b, void *c, void *d, void *e, void *f) {
+            trace_apple_iosfc_unmap_memory(a, b, c, d, e, f);
+            return (bool)true;
+        };
+
+    iosfc_desc.raiseInterrupt = ^(uint32_t vector) {
+        trace_apple_iosfc_raise_irq(vector);
+        bool locked = bql_locked();
+        if (!locked) {
+            bql_lock();
+        }
+        qemu_irq_pulse(s->irq_iosfc);
+        if (!locked) {
+            bql_unlock();
+        }
+        return (bool)true;
+    };
+
+    iosfc_host_dev =
+        [[PGIOSurfaceHostDevice alloc] initWithDescriptor:iosfc_desc];
+    [iosfc_desc release];
+    return iosfc_host_dev;
+}
+
+static void apple_gfx_vmapple_realize(DeviceState *dev, Error **errp)
+{
+    @autoreleasepool {
+        AppleGFXVmappleState *s = APPLE_GFX_VMAPPLE(dev);
+
+        PGDeviceDescriptor *desc = [PGDeviceDescriptor new];
+        desc.usingIOSurfaceMapper = true;
+        desc.raiseInterrupt = ^(uint32_t vector) {
+            bool locked;
+
+            trace_apple_gfx_raise_irq(vector);
+            locked = bql_locked();
+            if (!locked) {
+                bql_lock();
+            }
+            qemu_irq_pulse(s->irq_gfx);
+            if (!locked) {
+                bql_unlock();
+            }
+        };
+
+        s->pgiosfc = apple_gfx_prepare_iosurface_host_device(s);
+
+        apple_gfx_common_realize(&s->common, desc);
+        [desc release];
+        desc = nil;
+    }
+}
+
+static void apple_gfx_vmapple_init(Object *obj)
+{
+    AppleGFXVmappleState *s = APPLE_GFX_VMAPPLE(obj);
+
+    apple_gfx_common_init(obj, &s->common, TYPE_APPLE_GFX_VMAPPLE);
+
+    memory_region_init_io(&s->iomem_iosfc, obj, &apple_iosfc_ops, s,
+                          TYPE_APPLE_GFX_VMAPPLE, 0x10000);
+    sysbus_init_mmio(SYS_BUS_DEVICE(s), &s->common.iomem_gfx);
+    sysbus_init_mmio(SYS_BUS_DEVICE(s), &s->iomem_iosfc);
+    sysbus_init_irq(SYS_BUS_DEVICE(s), &s->irq_gfx);
+    sysbus_init_irq(SYS_BUS_DEVICE(s), &s->irq_iosfc);
+}
+
+static void apple_gfx_vmapple_reset(Object *obj, ResetType type)
+{
+    AppleGFXVmappleState *s = APPLE_GFX_VMAPPLE(obj);
+    [s->common.pgdev reset];
+}
+
+
+static void apple_gfx_vmapple_class_init(ObjectClass *klass, void *data)
+{
+    DeviceClass *dc = DEVICE_CLASS(klass);
+    ResettableClass *rc = RESETTABLE_CLASS(klass);
+
+    assert(rc->phases.hold == NULL);
+    rc->phases.hold = apple_gfx_vmapple_reset;
+
+    dc->realize = apple_gfx_vmapple_realize;
+}
+
+static TypeInfo apple_gfx_vmapple_types[] = {
+    {
+        .name          = TYPE_APPLE_GFX_VMAPPLE,
+        .parent        = TYPE_SYS_BUS_DEVICE,
+        .instance_size = sizeof(AppleGFXVmappleState),
+        .class_init    = apple_gfx_vmapple_class_init,
+        .instance_init = apple_gfx_vmapple_init,
+    }
+};
+DEFINE_TYPES(apple_gfx_vmapple_types)
diff --git a/hw/display/apple-gfx.h b/hw/display/apple-gfx.h
new file mode 100644
index 00000000000..995ecf7f4a7
--- /dev/null
+++ b/hw/display/apple-gfx.h
@@ -0,0 +1,57 @@
+#ifndef QEMU_APPLE_GFX_H
+#define QEMU_APPLE_GFX_H
+
+#define TYPE_APPLE_GFX_VMAPPLE      "apple-gfx-vmapple"
+#define TYPE_APPLE_GFX_PCI          "apple-gfx-pci"
+
+#include "qemu/typedefs.h"
+
+typedef struct AppleGFXState AppleGFXState;
+
+void apple_gfx_common_init(Object *obj, AppleGFXState *s, const char* obj_name);
+
+#ifdef __OBJC__
+
+#include "qemu/osdep.h"
+#include "exec/memory.h"
+#include "ui/surface.h"
+#include <dispatch/dispatch.h>
+
+@class PGDeviceDescriptor;
+@protocol PGDevice;
+@protocol PGDisplay;
+@protocol MTLDevice;
+@protocol MTLTexture;
+@protocol MTLCommandQueue;
+
+typedef QTAILQ_HEAD(, PGTask_s) AppleGFXTaskList;
+
+struct AppleGFXState {
+    MemoryRegion iomem_gfx;
+    id<PGDevice> pgdev;
+    id<PGDisplay> pgdisp;
+    AppleGFXTaskList tasks;
+    QemuConsole *con;
+    id<MTLDevice> mtl;
+    id<MTLCommandQueue> mtl_queue;
+    bool handles_frames;
+    bool new_frame;
+    bool cursor_show;
+    QEMUCursor *cursor;
+
+    dispatch_queue_t render_queue;
+    /* The following fields should only be accessed from render_queue: */
+    bool gfx_update_requested;
+    bool new_frame_ready;
+    bool using_managed_texture_storage;
+    int32_t pending_frames;
+    void *vram;
+    DisplaySurface *surface;
+    id<MTLTexture> texture;
+};
+
+void apple_gfx_common_realize(AppleGFXState *s, PGDeviceDescriptor *desc);
+
+#endif /* __OBJC__ */
+
+#endif
diff --git a/hw/display/apple-gfx.m b/hw/display/apple-gfx.m
new file mode 100644
index 00000000000..837300f9cd4
--- /dev/null
+++ b/hw/display/apple-gfx.m
@@ -0,0 +1,536 @@
+/*
+ * QEMU Apple ParavirtualizedGraphics.framework device
+ *
+ * Copyright © 2023 Amazon.com, Inc. or its affiliates. All Rights Reserved.
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or later.
+ * See the COPYING file in the top-level directory.
+ *
+ * ParavirtualizedGraphics.framework is a set of libraries that macOS provides
+ * which implements 3d graphics passthrough to the host as well as a
+ * proprietary guest communication channel to drive it. This device model
+ * implements support to drive that library from within QEMU.
+ */
+
+#include "apple-gfx.h"
+#include "trace.h"
+#include "qemu/main-loop.h"
+#include "ui/console.h"
+#include "monitor/monitor.h"
+#include "qapi/error.h"
+#include "migration/blocker.h"
+#include <mach/mach_vm.h>
+#import <ParavirtualizedGraphics/ParavirtualizedGraphics.h>
+
+static const PGDisplayCoord_t apple_gfx_modes[] = {
+    { .x = 1440, .y = 1080 },
+    { .x = 1280, .y = 1024 },
+};
+
+typedef struct PGTask_s { // Name matches forward declaration in PG header
+    QTAILQ_ENTRY(PGTask_s) node;
+    mach_vm_address_t address;
+    uint64_t len;
+} AppleGFXTask;
+
+static Error *apple_gfx_mig_blocker;
+
+static void apple_gfx_render_frame_completed(AppleGFXState *s, void *vram,
+                                             id<MTLTexture> texture);
+
+static AppleGFXTask *apple_gfx_new_task(AppleGFXState *s, uint64_t len)
+{
+    mach_vm_address_t task_mem;
+    AppleGFXTask *task;
+    kern_return_t r;
+
+    r = mach_vm_allocate(mach_task_self(), &task_mem, len, VM_FLAGS_ANYWHERE);
+    if (r != KERN_SUCCESS || task_mem == 0) {
+        return NULL;
+    }
+
+    task = g_new0(AppleGFXTask, 1);
+
+    task->address = task_mem;
+    task->len = len;
+    QTAILQ_INSERT_TAIL(&s->tasks, task, node);
+
+    return task;
+}
+
+static uint64_t apple_gfx_read(void *opaque, hwaddr offset, unsigned size)
+{
+    AppleGFXState *s = opaque;
+    uint64_t res = 0;
+
+    bql_unlock();
+    res = [s->pgdev mmioReadAtOffset:offset];
+    bql_lock();
+
+    trace_apple_gfx_read(offset, res);
+
+    return res;
+}
+
+static void apple_gfx_write(void *opaque, hwaddr offset, uint64_t val,
+                            unsigned size)
+{
+    AppleGFXState *s = opaque;
+
+    trace_apple_gfx_write(offset, val);
+
+    bql_unlock();
+    [s->pgdev mmioWriteAtOffset:offset value:val];
+    bql_lock();
+}
+
+static const MemoryRegionOps apple_gfx_ops = {
+    .read = apple_gfx_read,
+    .write = apple_gfx_write,
+    .endianness = DEVICE_LITTLE_ENDIAN,
+    .valid = {
+        .min_access_size = 4,
+        .max_access_size = 8,
+    },
+    .impl = {
+        .min_access_size = 4,
+        .max_access_size = 4,
+    },
+};
+
+static void apple_gfx_render_new_frame(AppleGFXState *s)
+{
+    BOOL r;
+    void *vram = s->vram;
+    uint32_t width = surface_width(s->surface);
+    uint32_t height = surface_height(s->surface);
+    MTLRegion region = MTLRegionMake2D(0, 0, width, height);
+    id<MTLCommandBuffer> command_buffer = [s->mtl_queue commandBuffer];
+    id<MTLTexture> texture = s->texture;
+    r = [s->pgdisp encodeCurrentFrameToCommandBuffer:command_buffer
+                                             texture:texture
+                                              region:region];
+    if (!r) {
+        return;
+    }
+    [texture retain];
+    if (s->using_managed_texture_storage) {
+        /* "Managed" textures exist in both VRAM and RAM and must be synced. */
+        id<MTLBlitCommandEncoder> blit = [command_buffer blitCommandEncoder];
+        [blit synchronizeResource:texture];
+        [blit endEncoding];
+    }
+    [command_buffer retain];
+    [command_buffer addCompletedHandler:
+        ^(id<MTLCommandBuffer> cb)
+        {
+            dispatch_async(s->render_queue, ^{
+                apple_gfx_render_frame_completed(s, vram, texture);
+                [texture release];
+            });
+            [command_buffer release];
+        }];
+    [command_buffer commit];
+}
+
+static void copy_mtl_texture_to_surface_mem(id<MTLTexture> texture, void *vram)
+{
+    /* TODO: Skip this entirely on a pure Metal or headless/guest-only
+     * rendering path, else use a blit command encoder? Needs careful
+     * (double?) buffering design. */
+    size_t width = texture.width, height = texture.height;
+    MTLRegion region = MTLRegionMake2D(0, 0, width, height);
+    [texture getBytes:vram
+          bytesPerRow:(width * 4)
+        bytesPerImage:(width * height * 4)
+           fromRegion:region
+          mipmapLevel:0
+                slice:0];
+}
+
+static void apple_gfx_render_frame_completed(AppleGFXState *s, void *vram,
+                                             id<MTLTexture> texture)
+{
+    --s->pending_frames;
+    assert(s->pending_frames >= 0);
+
+    if (vram != s->vram) {
+        /* Display mode has changed, drop this old frame. */
+        assert(texture != s->texture);
+        g_free(vram);
+    } else {
+        copy_mtl_texture_to_surface_mem(texture, vram);
+        if (s->gfx_update_requested) {
+            s->gfx_update_requested = false;
+            dpy_gfx_update_full(s->con);
+            graphic_hw_update_done(s->con);
+            s->new_frame_ready = false;
+        } else {
+            s->new_frame_ready = true;
+        }
+    }
+    if (s->pending_frames > 0) {
+        apple_gfx_render_new_frame(s);
+    }
+}
+
+static void apple_gfx_fb_update_display(void *opaque)
+{
+    AppleGFXState *s = opaque;
+
+    dispatch_async(s->render_queue, ^{
+        if (s->pending_frames > 0) {
+            s->gfx_update_requested = true;
+        } else {
+            if (s->new_frame_ready) {
+                dpy_gfx_update_full(s->con);
+                s->new_frame_ready = false;
+            }
+            graphic_hw_update_done(s->con);
+        }
+    });
+}
+
+static const GraphicHwOps apple_gfx_fb_ops = {
+    .gfx_update = apple_gfx_fb_update_display,
+    .gfx_update_async = true,
+};
+
+static void update_cursor(AppleGFXState *s)
+{
+    dpy_mouse_set(s->con, s->pgdisp.cursorPosition.x,
+                  s->pgdisp.cursorPosition.y, s->cursor_show);
+}
+
+static void set_mode(AppleGFXState *s, uint32_t width, uint32_t height)
+{
+    void *vram = NULL;
+    DisplaySurface *surface;
+    MTLTextureDescriptor *textureDescriptor;
+    id<MTLTexture> texture = nil;
+    __block bool no_change = false;
+
+    dispatch_sync(s->render_queue,
+        ^{
+            if (s->surface &&
+                width == surface_width(s->surface) &&
+                height == surface_height(s->surface)) {
+                no_change = true;
+            }
+        });
+
+    if (no_change) {
+        return;
+    }
+
+    vram = g_malloc0(width * height * 4);
+    surface = qemu_create_displaysurface_from(width, height, PIXMAN_LE_a8r8g8b8,
+                                              width * 4, vram);
+
+    @autoreleasepool {
+        textureDescriptor =
+            [MTLTextureDescriptor
+                texture2DDescriptorWithPixelFormat:MTLPixelFormatBGRA8Unorm
+                                             width:width
+                                            height:height
+                                         mipmapped:NO];
+        textureDescriptor.usage = s->pgdisp.minimumTextureUsage;
+        texture = [s->mtl newTextureWithDescriptor:textureDescriptor];
+    }
+
+    s->using_managed_texture_storage =
+        (texture.storageMode == MTLStorageModeManaged);
+
+    dispatch_sync(s->render_queue,
+        ^{
+            id<MTLTexture> old_texture = nil;
+            void *old_vram = s->vram;
+            s->vram = vram;
+            s->surface = surface;
+
+            dpy_gfx_replace_surface(s->con, surface);
+
+            old_texture = s->texture;
+            s->texture = texture;
+            [old_texture release];
+
+            if (s->pending_frames == 0) {
+                g_free(old_vram);
+            }
+        });
+}
+
+static void create_fb(AppleGFXState *s)
+{
+    s->con = graphic_console_init(NULL, 0, &apple_gfx_fb_ops, s);
+    set_mode(s, 1440, 1080);
+
+    s->cursor_show = true;
+}
+
+static size_t apple_gfx_get_default_mmio_range_size(void)
+{
+    size_t mmio_range_size;
+    @autoreleasepool {
+        PGDeviceDescriptor *desc = [PGDeviceDescriptor new];
+        mmio_range_size = desc.mmioLength;
+        [desc release];
+    }
+    return mmio_range_size;
+}
+
+void apple_gfx_common_init(Object *obj, AppleGFXState *s, const char* obj_name)
+{
+    Error *local_err = NULL;
+    int r;
+    size_t mmio_range_size = apple_gfx_get_default_mmio_range_size();
+
+    trace_apple_gfx_common_init(obj_name, mmio_range_size);
+    memory_region_init_io(&s->iomem_gfx, obj, &apple_gfx_ops, s, obj_name,
+                          mmio_range_size);
+    s->iomem_gfx.disable_reentrancy_guard = true;
+
+    /* TODO: PVG framework supports serialising device state: integrate it! */
+    if (apple_gfx_mig_blocker == NULL) {
+        error_setg(&apple_gfx_mig_blocker,
+                  "Migration state blocked by apple-gfx display device");
+        r = migrate_add_blocker(&apple_gfx_mig_blocker, &local_err);
+        if (r < 0) {
+            error_report_err(local_err);
+        }
+    }
+}
+
+static void apple_gfx_register_task_mapping_handlers(AppleGFXState *s,
+                                                     PGDeviceDescriptor *desc)
+{
+    desc.createTask = ^(uint64_t vmSize, void * _Nullable * _Nonnull baseAddress) {
+        AppleGFXTask *task = apple_gfx_new_task(s, vmSize);
+        *baseAddress = (void*)task->address;
+        trace_apple_gfx_create_task(vmSize, *baseAddress);
+        return task;
+    };
+
+    desc.destroyTask = ^(AppleGFXTask * _Nonnull task) {
+        trace_apple_gfx_destroy_task(task);
+        QTAILQ_REMOVE(&s->tasks, task, node);
+        mach_vm_deallocate(mach_task_self(), task->address, task->len);
+        g_free(task);
+    };
+
+    desc.mapMemory = ^(AppleGFXTask * _Nonnull task, uint32_t rangeCount,
+                       uint64_t virtualOffset, bool readOnly,
+                       PGPhysicalMemoryRange_t * _Nonnull ranges) {
+        kern_return_t r;
+        mach_vm_address_t target, source;
+        trace_apple_gfx_map_memory(task, rangeCount, virtualOffset, readOnly);
+        for (int i = 0; i < rangeCount; i++) {
+            PGPhysicalMemoryRange_t *range = &ranges[i];
+            MemoryRegion *tmp_mr;
+            /* TODO: Bounds checks? r/o? */
+            bql_lock();
+
+            trace_apple_gfx_map_memory_range(i, range->physicalAddress,
+                                             range->physicalLength);
+
+            target = task->address + virtualOffset;
+            source = (mach_vm_address_t)gpa2hva(&tmp_mr,
+                                                range->physicalAddress,
+                                                range->physicalLength, NULL);
+            vm_prot_t cur_protection = 0;
+            vm_prot_t max_protection = 0;
+            // Map guest RAM at range->physicalAddress into PG task memory range
+            r = mach_vm_remap(mach_task_self(),
+                              &target, range->physicalLength, vm_page_size - 1,
+                              VM_FLAGS_FIXED | VM_FLAGS_OVERWRITE,
+                              mach_task_self(),
+                              source, false /* shared mapping, no copy */,
+                              &cur_protection, &max_protection,
+                              VM_INHERIT_COPY);
+            trace_apple_gfx_remap(r, source, target);
+            g_assert(r == KERN_SUCCESS);
+
+            bql_unlock();
+
+            virtualOffset += range->physicalLength;
+        }
+        return (bool)true;
+    };
+
+    desc.unmapMemory = ^(AppleGFXTask * _Nonnull task, uint64_t virtualOffset,
+                         uint64_t length) {
+        kern_return_t r;
+        mach_vm_address_t range_address;
+
+        trace_apple_gfx_unmap_memory(task, virtualOffset, length);
+
+        /* Replace task memory range with fresh pages, undoing the mapping
+         * from guest RAM. */
+        range_address = task->address + virtualOffset;
+        r = mach_vm_allocate(mach_task_self(), &range_address, length,
+                             VM_FLAGS_FIXED | VM_FLAGS_OVERWRITE);
+        g_assert(r == KERN_SUCCESS);
+
+        return (bool)true;
+    };
+
+    desc.readMemory = ^(uint64_t physicalAddress, uint64_t length,
+                        void * _Nonnull dst) {
+        trace_apple_gfx_read_memory(physicalAddress, length, dst);
+        cpu_physical_memory_read(physicalAddress, dst, length);
+        return (bool)true;
+    };
+
+}
+
+static PGDisplayDescriptor *apple_gfx_prepare_display_handlers(AppleGFXState *s)
+{
+    PGDisplayDescriptor *disp_desc = [PGDisplayDescriptor new];
+
+    disp_desc.name = @"QEMU display";
+    disp_desc.sizeInMillimeters = NSMakeSize(400., 300.); /* A 20" display */
+    disp_desc.queue = dispatch_get_main_queue();
+    disp_desc.newFrameEventHandler = ^(void) {
+        trace_apple_gfx_new_frame();
+        dispatch_async(s->render_queue, ^{
+            /* Drop frames if we get too far ahead. */
+            if (s->pending_frames >= 2)
+                return;
+            ++s->pending_frames;
+            if (s->pending_frames > 1) {
+                return;
+            }
+            @autoreleasepool {
+                apple_gfx_render_new_frame(s);
+            }
+        });
+    };
+    disp_desc.modeChangeHandler = ^(PGDisplayCoord_t sizeInPixels,
+                                    OSType pixelFormat) {
+        trace_apple_gfx_mode_change(sizeInPixels.x, sizeInPixels.y);
+        set_mode(s, sizeInPixels.x, sizeInPixels.y);
+    };
+    disp_desc.cursorGlyphHandler = ^(NSBitmapImageRep *glyph,
+                                     PGDisplayCoord_t hotSpot) {
+        uint32_t bpp = glyph.bitsPerPixel;
+        size_t width = glyph.pixelsWide;
+        size_t height = glyph.pixelsHigh;
+        size_t padding_bytes_per_row = glyph.bytesPerRow - width * 4;
+        const uint8_t* px_data = glyph.bitmapData;
+
+        trace_apple_gfx_cursor_set(bpp, width, height);
+
+        if (s->cursor) {
+            cursor_unref(s->cursor);
+            s->cursor = NULL;
+        }
+
+        if (bpp == 32) { /* Shouldn't be anything else, but just to be safe...*/
+            s->cursor = cursor_alloc(width, height);
+            s->cursor->hot_x = hotSpot.x;
+            s->cursor->hot_y = hotSpot.y;
+
+            uint32_t *dest_px = s->cursor->data;
+
+            for (size_t y = 0; y < height; ++y) {
+                for (size_t x = 0; x < width; ++x) {
+                    /* NSBitmapImageRep's red & blue channels are swapped
+                     * compared to QEMUCursor's. */
+                    *dest_px =
+                        (px_data[0] << 16u) |
+                        (px_data[1] <<  8u) |
+                        (px_data[2] <<  0u) |
+                        (px_data[3] << 24u);
+                    ++dest_px;
+                    px_data += 4;
+                }
+                px_data += padding_bytes_per_row;
+            }
+            dpy_cursor_define(s->con, s->cursor);
+            update_cursor(s);
+        }
+    };
+    disp_desc.cursorShowHandler = ^(BOOL show) {
+        trace_apple_gfx_cursor_show(show);
+        s->cursor_show = show;
+        update_cursor(s);
+    };
+    disp_desc.cursorMoveHandler = ^(void) {
+        trace_apple_gfx_cursor_move();
+        update_cursor(s);
+    };
+
+    return disp_desc;
+}
+
+static NSArray<PGDisplayMode*>* apple_gfx_prepare_display_mode_array(void)
+{
+    PGDisplayMode *modes[ARRAY_SIZE(apple_gfx_modes)];
+    NSArray<PGDisplayMode*>* mode_array = nil;
+    int i;
+
+    for (i = 0; i < ARRAY_SIZE(apple_gfx_modes); i++) {
+        modes[i] =
+            [[PGDisplayMode alloc] initWithSizeInPixels:apple_gfx_modes[i] refreshRateInHz:60.];
+    }
+
+    mode_array = [NSArray arrayWithObjects:modes count:ARRAY_SIZE(apple_gfx_modes)];
+
+    for (i = 0; i < ARRAY_SIZE(apple_gfx_modes); i++) {
+        [modes[i] release];
+        modes[i] = nil;
+    }
+
+    return mode_array;
+}
+
+static id<MTLDevice> copy_suitable_metal_device(void)
+{
+    id<MTLDevice> dev = nil;
+    NSArray<id<MTLDevice>> *devs = MTLCopyAllDevices();
+
+    /* Prefer a unified memory GPU. Failing that, pick a non-removable GPU. */
+    for (size_t i = 0; i < devs.count; ++i) {
+        if (devs[i].hasUnifiedMemory) {
+            dev = devs[i];
+            break;
+        }
+        if (!devs[i].removable) {
+            dev = devs[i];
+        }
+    }
+
+    if (dev != nil) {
+        [dev retain];
+    } else {
+        dev = MTLCreateSystemDefaultDevice();
+    }
+    [devs release];
+
+    return dev;
+}
+
+void apple_gfx_common_realize(AppleGFXState *s, PGDeviceDescriptor *desc)
+{
+    PGDisplayDescriptor *disp_desc = nil;
+
+    QTAILQ_INIT(&s->tasks);
+    s->render_queue = dispatch_queue_create("apple-gfx.render",
+                                            DISPATCH_QUEUE_SERIAL);
+    s->mtl = copy_suitable_metal_device();
+    s->mtl_queue = [s->mtl newCommandQueue];
+
+    desc.device = s->mtl;
+
+    apple_gfx_register_task_mapping_handlers(s, desc);
+
+    s->pgdev = PGNewDeviceWithDescriptor(desc);
+
+    disp_desc = apple_gfx_prepare_display_handlers(s);
+    s->pgdisp = [s->pgdev newDisplayWithDescriptor:disp_desc
+                                              port:0 serialNum:1234];
+    [disp_desc release];
+    s->pgdisp.modeList = apple_gfx_prepare_display_mode_array();
+
+    create_fb(s);
+}
diff --git a/hw/display/meson.build b/hw/display/meson.build
index 7db05eace97..70d855749c0 100644
--- a/hw/display/meson.build
+++ b/hw/display/meson.build
@@ -65,6 +65,8 @@ system_ss.add(when: 'CONFIG_ARTIST', if_true: files('artist.c'))
 
 system_ss.add(when: 'CONFIG_ATI_VGA', if_true: [files('ati.c', 'ati_2d.c', 'ati_dbg.c'), pixman])
 
+system_ss.add(when: 'CONFIG_MAC_PVG',         if_true: [files('apple-gfx.m'), pvg, metal])
+system_ss.add(when: 'CONFIG_MAC_PVG_VMAPPLE', if_true: [files('apple-gfx-vmapple.m'), pvg, metal])
 
 if config_all_devices.has_key('CONFIG_VIRTIO_GPU')
   virtio_gpu_ss = ss.source_set()
diff --git a/hw/display/trace-events b/hw/display/trace-events
index 781f8a33203..1809a358e36 100644
--- a/hw/display/trace-events
+++ b/hw/display/trace-events
@@ -191,3 +191,29 @@ dm163_bits_ppi(unsigned dest_width) "dest_width : %u"
 dm163_leds(int led, uint32_t value) "led %d: 0x%x"
 dm163_channels(int channel, uint8_t value) "channel %d: 0x%x"
 dm163_refresh_rate(uint32_t rr) "refresh rate %d"
+
+# apple-gfx.m
+apple_gfx_read(uint64_t offset, uint64_t res) "offset=0x%"PRIx64" res=0x%"PRIx64
+apple_gfx_write(uint64_t offset, uint64_t val) "offset=0x%"PRIx64" val=0x%"PRIx64
+apple_gfx_create_task(uint32_t vm_size, void *va) "vm_size=0x%x base_addr=%p"
+apple_gfx_destroy_task(void *task) "task=%p"
+apple_gfx_map_memory(void *task, uint32_t range_count, uint64_t virtual_offset, uint32_t read_only) "task=%p range_count=0x%x virtual_offset=0x%"PRIx64" read_only=%d"
+apple_gfx_map_memory_range(uint32_t i, uint64_t phys_addr, uint64_t phys_len) "[%d] phys_addr=0x%"PRIx64" phys_len=0x%"PRIx64
+apple_gfx_remap(uint64_t retval, uint64_t source, uint64_t target) "retval=%"PRId64" source=0x%"PRIx64" target=0x%"PRIx64
+apple_gfx_unmap_memory(void *task, uint64_t virtual_offset, uint64_t length) "task=%p virtual_offset=0x%"PRIx64" length=0x%"PRIx64
+apple_gfx_read_memory(uint64_t phys_address, uint64_t length, void *dst) "phys_addr=0x%"PRIx64" length=0x%"PRIx64" dest=%p"
+apple_gfx_raise_irq(uint32_t vector) "vector=0x%x"
+apple_gfx_new_frame(void) ""
+apple_gfx_mode_change(uint64_t x, uint64_t y) "x=%"PRId64" y=%"PRId64
+apple_gfx_cursor_set(uint32_t bpp, uint64_t width, uint64_t height) "bpp=%d width=%"PRId64" height=0x%"PRId64
+apple_gfx_cursor_show(uint32_t show) "show=%d"
+apple_gfx_cursor_move(void) ""
+apple_gfx_common_init(const char *device_name, size_t mmio_size) "device: %s; MMIO size: %zu bytes"
+
+# apple-gfx-vmapple.m
+apple_iosfc_read(uint64_t offset, uint64_t res) "offset=0x%"PRIx64" res=0x%"PRIx64
+apple_iosfc_write(uint64_t offset, uint64_t val) "offset=0x%"PRIx64" val=0x%"PRIx64
+apple_iosfc_map_memory(uint64_t phys, uint64_t len, uint32_t ro, void *va, void *e, void *f) "phys=0x%"PRIx64" len=0x%"PRIx64" ro=%d va=%p e=%p f=%p"
+apple_iosfc_unmap_memory(void *a, void *b, void *c, void *d, void *e, void *f) "a=%p b=%p c=%p d=%p e=%p f=%p"
+apple_iosfc_raise_irq(uint32_t vector) "vector=0x%x"
+
diff --git a/meson.build b/meson.build
index 10464466ff3..f09df3f09d5 100644
--- a/meson.build
+++ b/meson.build
@@ -741,6 +741,8 @@ socket = []
 version_res = []
 coref = []
 iokit = []
+pvg = []
+metal = []
 emulator_link_args = []
 midl = not_found
 widl = not_found
@@ -762,6 +764,8 @@ elif host_os == 'darwin'
   coref = dependency('appleframeworks', modules: 'CoreFoundation')
   iokit = dependency('appleframeworks', modules: 'IOKit', required: false)
   host_dsosuf = '.dylib'
+  pvg = dependency('appleframeworks', modules: 'ParavirtualizedGraphics')
+  metal = dependency('appleframeworks', modules: 'Metal')
 elif host_os == 'sunos'
   socket = [cc.find_library('socket'),
             cc.find_library('nsl'),
-- 
2.39.3 (Apple Git-145)

[PATCH v3 02/14] hw/display/apple-gfx: Adds PCI implementation

[Phil Dennis-Jordan]

This change wires up the PCI variant of the paravirtualised
graphics device, mainly useful for x86-64 macOS guests, implemented
by macOS's ParavirtualizedGraphics.framework. It builds on code
shared with the vmapple/mmio variant of the PVG device.

Signed-off-by: Phil Dennis-Jordan <phil@philjordan.eu>
---
 hw/display/Kconfig         |   5 ++
 hw/display/apple-gfx-pci.m | 138 +++++++++++++++++++++++++++++++++++++
 hw/display/meson.build     |   1 +
 3 files changed, 144 insertions(+)
 create mode 100644 hw/display/apple-gfx-pci.m

diff --git a/hw/display/Kconfig b/hw/display/Kconfig
index 179a479d220..c2ec268f8e9 100644
--- a/hw/display/Kconfig
+++ b/hw/display/Kconfig
@@ -152,3 +152,8 @@ config MAC_PVG_VMAPPLE
     bool
     depends on MAC_PVG
     depends on ARM
+
+config MAC_PVG_PCI
+    bool
+    depends on MAC_PVG && PCI
+    default y if PCI_DEVICES
diff --git a/hw/display/apple-gfx-pci.m b/hw/display/apple-gfx-pci.m
new file mode 100644
index 00000000000..9370258ee46
--- /dev/null
+++ b/hw/display/apple-gfx-pci.m
@@ -0,0 +1,138 @@
+/*
+ * QEMU Apple ParavirtualizedGraphics.framework device, PCI variant
+ *
+ * Copyright © 2023-2024 Phil Dennis-Jordan
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or later.
+ * See the COPYING file in the top-level directory.
+ *
+ * ParavirtualizedGraphics.framework is a set of libraries that macOS provides
+ * which implements 3d graphics passthrough to the host as well as a
+ * proprietary guest communication channel to drive it. This device model
+ * implements support to drive that library from within QEMU as a PCI device
+ * aimed primarily at x86-64 macOS VMs.
+ */
+
+#include "apple-gfx.h"
+#include "hw/pci/pci_device.h"
+#include "hw/pci/msi.h"
+#include "qapi/error.h"
+#include "trace.h"
+#import <ParavirtualizedGraphics/ParavirtualizedGraphics.h>
+
+typedef struct AppleGFXPCIState {
+    PCIDevice parent_obj;
+
+    AppleGFXState common;
+} AppleGFXPCIState;
+
+OBJECT_DECLARE_SIMPLE_TYPE(AppleGFXPCIState, APPLE_GFX_PCI)
+
+static const char* apple_gfx_pci_option_rom_path = NULL;
+
+static void apple_gfx_init_option_rom_path(void)
+{
+    NSURL *option_rom_url = PGCopyOptionROMURL();
+    const char *option_rom_path = option_rom_url.fileSystemRepresentation;
+    if (option_rom_url.fileURL && option_rom_path != NULL) {
+        apple_gfx_pci_option_rom_path = g_strdup(option_rom_path);
+    }
+    [option_rom_url release];
+}
+
+static void apple_gfx_pci_init(Object *obj)
+{
+    AppleGFXPCIState *s = APPLE_GFX_PCI(obj);
+
+    if (!apple_gfx_pci_option_rom_path) {
+        /* Done on device not class init to avoid -daemonize ObjC fork crash */
+        PCIDeviceClass *pci = PCI_DEVICE_CLASS(object_get_class(obj));
+        apple_gfx_init_option_rom_path();
+        pci->romfile = apple_gfx_pci_option_rom_path;
+    }
+
+    apple_gfx_common_init(obj, &s->common, TYPE_APPLE_GFX_PCI);
+}
+
+static void apple_gfx_pci_interrupt(PCIDevice *dev, AppleGFXPCIState *s,
+                                    uint32_t vector)
+{
+    bool msi_ok;
+    trace_apple_gfx_raise_irq(vector);
+
+    msi_ok = msi_enabled(dev);
+    if (msi_ok) {
+        msi_notify(dev, vector);
+    }
+}
+
+static void apple_gfx_pci_realize(PCIDevice *dev, Error **errp)
+{
+    AppleGFXPCIState *s = APPLE_GFX_PCI(dev);
+    Error *err = NULL;
+    int ret;
+
+    pci_register_bar(dev, PG_PCI_BAR_MMIO,
+                     PCI_BASE_ADDRESS_SPACE_MEMORY, &s->common.iomem_gfx);
+
+    ret = msi_init(dev, 0x0 /* config offset; 0 = find space */,
+                   PG_PCI_MAX_MSI_VECTORS, true /* msi64bit */,
+                   false /*msi_per_vector_mask*/, &err);
+    if (ret != 0) {
+        error_propagate(errp, err);
+        return;
+    }
+
+    @autoreleasepool {
+        PGDeviceDescriptor *desc = [PGDeviceDescriptor new];
+        desc.raiseInterrupt = ^(uint32_t vector) {
+            apple_gfx_pci_interrupt(dev, s, vector);
+        };
+
+        apple_gfx_common_realize(&s->common, desc);
+        [desc release];
+        desc = nil;
+    }
+}
+
+static void apple_gfx_pci_reset(Object *obj, ResetType type)
+{
+    AppleGFXPCIState *s = APPLE_GFX_PCI(obj);
+    [s->common.pgdev reset];
+}
+
+static void apple_gfx_pci_class_init(ObjectClass *klass, void *data)
+{
+    DeviceClass *dc = DEVICE_CLASS(klass);
+    PCIDeviceClass *pci = PCI_DEVICE_CLASS(klass);
+    ResettableClass *rc = RESETTABLE_CLASS(klass);
+
+    assert(rc->phases.hold == NULL);
+    rc->phases.hold = apple_gfx_pci_reset;
+    dc->desc = "macOS Paravirtualized Graphics PCI Display Controller";
+    dc->hotpluggable = false;
+    set_bit(DEVICE_CATEGORY_DISPLAY, dc->categories);
+
+    pci->vendor_id = PG_PCI_VENDOR_ID;
+    pci->device_id = PG_PCI_DEVICE_ID;
+    pci->class_id = PCI_CLASS_DISPLAY_OTHER;
+    pci->realize = apple_gfx_pci_realize;
+
+    // TODO: Property for setting mode list
+}
+
+static TypeInfo apple_gfx_pci_types[] = {
+    {
+        .name          = TYPE_APPLE_GFX_PCI,
+        .parent        = TYPE_PCI_DEVICE,
+        .instance_size = sizeof(AppleGFXPCIState),
+        .class_init    = apple_gfx_pci_class_init,
+        .instance_init = apple_gfx_pci_init,
+        .interfaces = (InterfaceInfo[]) {
+            { INTERFACE_PCIE_DEVICE },
+            { },
+        },
+    }
+};
+DEFINE_TYPES(apple_gfx_pci_types)
+
diff --git a/hw/display/meson.build b/hw/display/meson.build
index 70d855749c0..ceb7bb07612 100644
--- a/hw/display/meson.build
+++ b/hw/display/meson.build
@@ -67,6 +67,7 @@ system_ss.add(when: 'CONFIG_ATI_VGA', if_true: [files('ati.c', 'ati_2d.c', 'ati_
 
 system_ss.add(when: 'CONFIG_MAC_PVG',         if_true: [files('apple-gfx.m'), pvg, metal])
 system_ss.add(when: 'CONFIG_MAC_PVG_VMAPPLE', if_true: [files('apple-gfx-vmapple.m'), pvg, metal])
+system_ss.add(when: 'CONFIG_MAC_PVG_PCI',     if_true: [files('apple-gfx-pci.m'), pvg, metal])
 
 if config_all_devices.has_key('CONFIG_VIRTIO_GPU')
   virtio_gpu_ss = ss.source_set()
-- 
2.39.3 (Apple Git-145)

[PATCH v3 03/14] ui/cocoa: Adds non-app runloop on main thread mode

[Phil Dennis-Jordan]

Various system frameworks on macOS and other Apple platforms
require a main runloop to be processing events on the process’s
main thread. The Cocoa UI’s requirement to run the process as a
Cocoa application automatically enables this runloop, but it
can be useful to have the runloop handling events even without
the Cocoa UI active.

This change adds a non-app runloop mode to the cocoa_main
function. This can be requested by other code, while the Cocoa UI
additionally enables app mode. This arrangement ensures there is
only one qemu_main function switcheroo, and the Cocoa UI’s app
mode requirement and other subsystems’ runloop requests don’t
conflict with each other.

The main runloop is required for the AppleGFX PV graphics device,
so the runloop request call has been added to its initialisation.

Signed-off-by: Phil Dennis-Jordan <phil@philjordan.eu>
---
 hw/display/apple-gfx.m |  3 +++
 include/qemu-main.h    |  2 ++
 ui/cocoa.m             | 15 +++++++++++++--
 3 files changed, 18 insertions(+), 2 deletions(-)

diff --git a/hw/display/apple-gfx.m b/hw/display/apple-gfx.m
index 837300f9cd4..6ef1048d93d 100644
--- a/hw/display/apple-gfx.m
+++ b/hw/display/apple-gfx.m
@@ -14,6 +14,7 @@
 
 #include "apple-gfx.h"
 #include "trace.h"
+#include "qemu-main.h"
 #include "qemu/main-loop.h"
 #include "ui/console.h"
 #include "monitor/monitor.h"
@@ -299,6 +300,8 @@ void apple_gfx_common_init(Object *obj, AppleGFXState *s, const char* obj_name)
             error_report_err(local_err);
         }
     }
+
+    cocoa_enable_runloop_on_main_thread();
 }
 
 static void apple_gfx_register_task_mapping_handlers(AppleGFXState *s,
diff --git a/include/qemu-main.h b/include/qemu-main.h
index 940960a7dbc..da4516e69eb 100644
--- a/include/qemu-main.h
+++ b/include/qemu-main.h
@@ -8,4 +8,6 @@
 int qemu_default_main(void);
 extern int (*qemu_main)(void);
 
+void cocoa_enable_runloop_on_main_thread(void);
+
 #endif /* QEMU_MAIN_H */
diff --git a/ui/cocoa.m b/ui/cocoa.m
index 4c2dd335323..40f65d7a45d 100644
--- a/ui/cocoa.m
+++ b/ui/cocoa.m
@@ -2028,6 +2028,7 @@ static void cocoa_clipboard_request(QemuClipboardInfo *info,
     exit(status);
 }
 
+static bool run_as_cocoa_app = false;
 static int cocoa_main(void)
 {
     QemuThread thread;
@@ -2040,7 +2041,11 @@ static int cocoa_main(void)
 
     // Start the main event loop
     COCOA_DEBUG("Main thread: entering OSX run loop\n");
-    [NSApp run];
+    if (run_as_cocoa_app) {
+        [NSApp run];
+    } else {
+        CFRunLoopRun();
+    }
     COCOA_DEBUG("Main thread: left OSX run loop, which should never happen\n");
 
     abort();
@@ -2114,13 +2119,19 @@ static void cocoa_cursor_define(DisplayChangeListener *dcl, QEMUCursor *cursor)
     });
 }
 
+void cocoa_enable_runloop_on_main_thread(void)
+{
+    qemu_main = cocoa_main;
+}
+
 static void cocoa_display_init(DisplayState *ds, DisplayOptions *opts)
 {
     NSAutoreleasePool * pool = [[NSAutoreleasePool alloc] init];
 
     COCOA_DEBUG("qemu_cocoa: cocoa_display_init\n");
 
-    qemu_main = cocoa_main;
+    run_as_cocoa_app = true;
+    cocoa_enable_runloop_on_main_thread();
 
     // Pull this console process up to being a fully-fledged graphical
     // app with a menubar and Dock icon
-- 
2.39.3 (Apple Git-145)

[PATCH v3 04/14] hw/display/apple-gfx: Adds configurable mode list

[Phil Dennis-Jordan]

This change adds a property 'display_modes' on the graphics device
which permits specifying a list of display modes. (screen resolution
and refresh rate)

PCI variant of apple-gfx only for the moment.

Signed-off-by: Phil Dennis-Jordan <phil@philjordan.eu>
---
 hw/display/apple-gfx-pci.m |  43 ++++++++++-
 hw/display/apple-gfx.h     |  17 ++++-
 hw/display/apple-gfx.m     | 151 ++++++++++++++++++++++++++++++++++---
 3 files changed, 198 insertions(+), 13 deletions(-)

diff --git a/hw/display/apple-gfx-pci.m b/hw/display/apple-gfx-pci.m
index 9370258ee46..ea86a1f4a21 100644
--- a/hw/display/apple-gfx-pci.m
+++ b/hw/display/apple-gfx-pci.m
@@ -16,6 +16,7 @@
 #include "apple-gfx.h"
 #include "hw/pci/pci_device.h"
 #include "hw/pci/msi.h"
+#include "hw/qdev-properties.h"
 #include "qapi/error.h"
 #include "trace.h"
 #import <ParavirtualizedGraphics/ParavirtualizedGraphics.h>
@@ -101,6 +102,46 @@ static void apple_gfx_pci_reset(Object *obj, ResetType type)
     [s->common.pgdev reset];
 }
 
+static void apple_gfx_pci_get_display_modes(Object *obj, Visitor *v,
+                                            const char *name, void *opaque,
+                                            Error **errp)
+{
+    Property *prop = opaque;
+    AppleGFXDisplayModeList *mode_list = object_field_prop_ptr(obj, prop);
+
+    apple_gfx_get_display_modes(mode_list, v, name, errp);
+}
+
+static void apple_gfx_pci_set_display_modes(Object *obj, Visitor *v,
+                                            const char *name, void *opaque,
+                                            Error **errp)
+{
+    Property *prop = opaque;
+    AppleGFXDisplayModeList *mode_list = object_field_prop_ptr(obj, prop);
+
+    apple_gfx_set_display_modes(mode_list, v, name, errp);
+}
+
+const PropertyInfo apple_gfx_pci_prop_display_modes = {
+    .name  = "display_modes",
+    .description =
+        "Colon-separated list of display modes; "
+        "<width>x<height>@<refresh-rate>; the first mode is considered "
+        "'native'. Example: 3840x2160@60:2560x1440@60:1920x1080@60",
+    .get   = apple_gfx_pci_get_display_modes,
+    .set   = apple_gfx_pci_set_display_modes,
+};
+
+#define DEFINE_PROP_DISPLAY_MODES(_name, _state, _field) \
+    DEFINE_PROP(_name, _state, _field, apple_gfx_pci_prop_display_modes, \
+                AppleGFXDisplayModeList)
+
+static Property apple_gfx_pci_properties[] = {
+    DEFINE_PROP_DISPLAY_MODES("display-modes", AppleGFXPCIState,
+                              common.display_modes),
+    DEFINE_PROP_END_OF_LIST(),
+};
+
 static void apple_gfx_pci_class_init(ObjectClass *klass, void *data)
 {
     DeviceClass *dc = DEVICE_CLASS(klass);
@@ -118,7 +159,7 @@ static void apple_gfx_pci_class_init(ObjectClass *klass, void *data)
     pci->class_id = PCI_CLASS_DISPLAY_OTHER;
     pci->realize = apple_gfx_pci_realize;
 
-    // TODO: Property for setting mode list
+    device_class_set_props(dc, apple_gfx_pci_properties);
 }
 
 static TypeInfo apple_gfx_pci_types[] = {
diff --git a/hw/display/apple-gfx.h b/hw/display/apple-gfx.h
index 995ecf7f4a7..baad4a98652 100644
--- a/hw/display/apple-gfx.h
+++ b/hw/display/apple-gfx.h
@@ -5,14 +5,28 @@
 #define TYPE_APPLE_GFX_PCI          "apple-gfx-pci"
 
 #include "qemu/typedefs.h"
+#include "qemu/osdep.h"
 
 typedef struct AppleGFXState AppleGFXState;
 
+typedef struct AppleGFXDisplayMode {
+    uint16_t width_px;
+    uint16_t height_px;
+    uint16_t refresh_rate_hz;
+} AppleGFXDisplayMode;
+
+typedef struct AppleGFXDisplayModeList {
+    GArray *modes;
+} AppleGFXDisplayModeList;
+
 void apple_gfx_common_init(Object *obj, AppleGFXState *s, const char* obj_name);
+void apple_gfx_get_display_modes(AppleGFXDisplayModeList *mode_list, Visitor *v,
+                                 const char *name, Error **errp);
+void apple_gfx_set_display_modes(AppleGFXDisplayModeList *mode_list, Visitor *v,
+                                 const char *name, Error **errp);
 
 #ifdef __OBJC__
 
-#include "qemu/osdep.h"
 #include "exec/memory.h"
 #include "ui/surface.h"
 #include <dispatch/dispatch.h>
@@ -38,6 +52,7 @@ struct AppleGFXState {
     bool new_frame;
     bool cursor_show;
     QEMUCursor *cursor;
+    AppleGFXDisplayModeList display_modes;
 
     dispatch_queue_t render_queue;
     /* The following fields should only be accessed from render_queue: */
diff --git a/hw/display/apple-gfx.m b/hw/display/apple-gfx.m
index 6ef1048d93d..358192db6a0 100644
--- a/hw/display/apple-gfx.m
+++ b/hw/display/apple-gfx.m
@@ -16,6 +16,9 @@
 #include "trace.h"
 #include "qemu-main.h"
 #include "qemu/main-loop.h"
+#include "qemu/cutils.h"
+#include "qapi/visitor.h"
+#include "qapi/error.h"
 #include "ui/console.h"
 #include "monitor/monitor.h"
 #include "qapi/error.h"
@@ -23,9 +26,10 @@
 #include <mach/mach_vm.h>
 #import <ParavirtualizedGraphics/ParavirtualizedGraphics.h>
 
-static const PGDisplayCoord_t apple_gfx_modes[] = {
-    { .x = 1440, .y = 1080 },
-    { .x = 1280, .y = 1024 },
+static const AppleGFXDisplayMode apple_gfx_default_modes[] = {
+    { 1920, 1080, 60 },
+    { 1440, 1080, 60 },
+    { 1280, 1024, 60 },
 };
 
 typedef struct PGTask_s { // Name matches forward declaration in PG header
@@ -264,7 +268,6 @@ static void set_mode(AppleGFXState *s, uint32_t width, uint32_t height)
 static void create_fb(AppleGFXState *s)
 {
     s->con = graphic_console_init(NULL, 0, &apple_gfx_fb_ops, s);
-    set_mode(s, 1440, 1080);
 
     s->cursor_show = true;
 }
@@ -466,20 +469,24 @@ static void apple_gfx_register_task_mapping_handlers(AppleGFXState *s,
     return disp_desc;
 }
 
-static NSArray<PGDisplayMode*>* apple_gfx_prepare_display_mode_array(void)
+static NSArray<PGDisplayMode*>* apple_gfx_create_display_mode_array(
+    const AppleGFXDisplayMode display_modes[], int display_mode_count)
 {
-    PGDisplayMode *modes[ARRAY_SIZE(apple_gfx_modes)];
+    PGDisplayMode **modes = alloca(sizeof(modes[0]) * display_mode_count);
     NSArray<PGDisplayMode*>* mode_array = nil;
     int i;
 
-    for (i = 0; i < ARRAY_SIZE(apple_gfx_modes); i++) {
+    for (i = 0; i < display_mode_count; i++) {
+        const AppleGFXDisplayMode *mode = &display_modes[i];
+        PGDisplayCoord_t mode_size = { mode->width_px, mode->height_px };
         modes[i] =
-            [[PGDisplayMode alloc] initWithSizeInPixels:apple_gfx_modes[i] refreshRateInHz:60.];
+            [[PGDisplayMode alloc] initWithSizeInPixels:mode_size
+                                        refreshRateInHz:mode->refresh_rate_hz];
     }
 
-    mode_array = [NSArray arrayWithObjects:modes count:ARRAY_SIZE(apple_gfx_modes)];
+    mode_array = [NSArray arrayWithObjects:modes count:display_mode_count];
 
-    for (i = 0; i < ARRAY_SIZE(apple_gfx_modes); i++) {
+    for (i = 0; i < display_mode_count; i++) {
         [modes[i] release];
         modes[i] = nil;
     }
@@ -516,6 +523,8 @@ static void apple_gfx_register_task_mapping_handlers(AppleGFXState *s,
 void apple_gfx_common_realize(AppleGFXState *s, PGDeviceDescriptor *desc)
 {
     PGDisplayDescriptor *disp_desc = nil;
+    const AppleGFXDisplayMode *display_modes = apple_gfx_default_modes;
+    int num_display_modes = ARRAY_SIZE(apple_gfx_default_modes);
 
     QTAILQ_INIT(&s->tasks);
     s->render_queue = dispatch_queue_create("apple-gfx.render",
@@ -533,7 +542,127 @@ void apple_gfx_common_realize(AppleGFXState *s, PGDeviceDescriptor *desc)
     s->pgdisp = [s->pgdev newDisplayWithDescriptor:disp_desc
                                               port:0 serialNum:1234];
     [disp_desc release];
-    s->pgdisp.modeList = apple_gfx_prepare_display_mode_array();
+
+    if (s->display_modes.modes != NULL && s->display_modes.modes->len > 0) {
+        display_modes =
+            &g_array_index(s->display_modes.modes, AppleGFXDisplayMode, 0);
+        num_display_modes = s->display_modes.modes->len;
+    }
+    s->pgdisp.modeList =
+        apple_gfx_create_display_mode_array(display_modes, num_display_modes);
 
     create_fb(s);
 }
+
+void apple_gfx_get_display_modes(AppleGFXDisplayModeList *mode_list, Visitor *v,
+                                 const char *name, Error **errp)
+{
+    GArray *modes = mode_list->modes;
+    /* 3 uint16s (max 5 digits) and 3 separator characters per mode + nul. */
+    size_t buffer_size = (5 + 1) * 3 * modes->len + 1;
+
+    char *buffer = alloca(buffer_size);
+    char *pos = buffer;
+
+    unsigned used = 0;
+    buffer[0] = '\0';
+    for (guint i = 0; i < modes->len; ++i)
+    {
+        AppleGFXDisplayMode *mode =
+            &g_array_index(modes, AppleGFXDisplayMode, i);
+        int rc = snprintf(pos, buffer_size - used,
+                          "%s%"PRIu16"x%"PRIu16"@%"PRIu16,
+                          i > 0 ? ":" : "",
+                          mode->width_px, mode->height_px,
+                          mode->refresh_rate_hz);
+        used += rc;
+        pos += rc;
+        assert(used < buffer_size);
+    }
+
+    pos = buffer;
+    visit_type_str(v, name, &pos, errp);
+}
+
+void apple_gfx_set_display_modes(AppleGFXDisplayModeList *mode_list, Visitor *v,
+                                 const char *name, Error **errp)
+{
+    Error *local_err = NULL;
+    const char *endptr;
+    char *str;
+    int ret;
+    unsigned int val;
+    uint32_t num_modes;
+    GArray *modes;
+    uint32_t mode_idx;
+
+    visit_type_str(v, name, &str, &local_err);
+    if (local_err) {
+        error_propagate(errp, local_err);
+        return;
+    }
+
+    // Count colons to estimate modes. No leading/trailing colons so start at 1.
+    num_modes = 1;
+    for (size_t i = 0; str[i] != '\0'; ++i)
+    {
+        if (str[i] == ':') {
+            ++num_modes;
+        }
+    }
+
+    modes = g_array_sized_new(false, true, sizeof(AppleGFXDisplayMode), num_modes);
+
+    endptr = str;
+    for (mode_idx = 0; mode_idx < num_modes; ++mode_idx)
+    {
+        AppleGFXDisplayMode mode = {};
+        if (mode_idx > 0)
+        {
+            if (*endptr != ':') {
+                goto separator_error;
+            }
+            ++endptr;
+        }
+
+        ret = qemu_strtoui(endptr, &endptr, 10, &val);
+        if (ret || val > UINT16_MAX || val == 0) {
+            error_setg(errp, "width of '%s' must be a decimal integer number "
+                       "of pixels in the range 1..65535", name);
+            goto out;
+        }
+        mode.width_px = val;
+        if (*endptr != 'x') {
+            goto separator_error;
+        }
+
+        ret = qemu_strtoui(endptr + 1, &endptr, 10, &val);
+        if (ret || val > UINT16_MAX || val == 0) {
+            error_setg(errp, "height of '%s' must be a decimal integer number "
+                       "of pixels in the range 1..65535", name);
+            goto out;
+        }
+        mode.height_px = val;
+        if (*endptr != '@') {
+            goto separator_error;
+        }
+
+        ret = qemu_strtoui(endptr + 1, &endptr, 10, &val);
+        if (ret) {
+            error_setg(errp, "refresh rate of '%s'"
+                       " must be a non-negative decimal integer (Hertz)", name);
+        }
+        mode.refresh_rate_hz = val;
+        g_array_append_val(modes, mode);
+    }
+
+    mode_list->modes = modes;
+    goto out;
+
+separator_error:
+    error_setg(errp, "Each display mode takes the format "
+               "'<width>x<height>@<rate>', modes are separated by colons. (:)");
+out:
+    g_free(str);
+    return;
+}
-- 
2.39.3 (Apple Git-145)

[PATCH v3 05/14] MAINTAINERS: Add myself as maintainer for apple-gfx, reviewer for HVF

[Phil Dennis-Jordan]

I'm happy to take responsibility for the macOS PV graphics code. As
HVF patches don't seem to get much attention at the moment, I'm also
adding myself as designated reviewer for HVF and x86 HVF to try and
improve that.

I anticipate that the resulting workload should be covered by the
funding I'm receiving for improving Qemu in combination with macOS. As
of right now this runs out at the end of 2024; I expect the workload on
apple-gfx should be relatively minor and manageable in my spare time
beyond that. I may have to remove myself from more general HVF duties
once the contract runs out if it's more than I can manage.

Signed-off-by: Phil Dennis-Jordan <phil@philjordan.eu>
---
 MAINTAINERS | 7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/MAINTAINERS b/MAINTAINERS
index ffacd60f407..0133b874847 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -510,6 +510,7 @@ F: target/arm/hvf/
 X86 HVF CPUs
 M: Cameron Esfahani <dirty@apple.com>
 M: Roman Bolshakov <rbolshakov@ddn.com>
+R: Phil Dennis-Jordan <phil@philjordan.eu>
 W: https://wiki.qemu.org/Features/HVF
 S: Maintained
 F: target/i386/hvf/
@@ -517,6 +518,7 @@ F: target/i386/hvf/
 HVF
 M: Cameron Esfahani <dirty@apple.com>
 M: Roman Bolshakov <rbolshakov@ddn.com>
+R: Phil Dennis-Jordan <phil@philjordan.eu>
 W: https://wiki.qemu.org/Features/HVF
 S: Maintained
 F: accel/hvf/
@@ -2624,6 +2626,11 @@ F: hw/display/edid*
 F: include/hw/display/edid.h
 F: qemu-edid.c
 
+macOS PV Graphics (apple-gfx)
+M: Phil Dennis-Jordan <phil@philjordan.eu>
+S: Maintained
+F: hw/display/apple-gfx*
+
 PIIX4 South Bridge (i82371AB)
 M: Hervé Poussineau <hpoussin@reactos.org>
 M: Philippe Mathieu-Daudé <philmd@linaro.org>
-- 
2.39.3 (Apple Git-145)

[PATCH v3 06/14] hw: Add vmapple subdir

[Phil Dennis-Jordan]

From: Alexander Graf <graf@amazon.com>

We will introduce a number of devices that are specific to the vmapple
target machine. To keep them all tidily together, let's put them into
a single target directory.

Signed-off-by: Alexander Graf <graf@amazon.com>
Signed-off-by: Phil Dennis-Jordan <phil@philjordan.eu>
---
 MAINTAINERS             | 7 +++++++
 hw/Kconfig              | 1 +
 hw/meson.build          | 1 +
 hw/vmapple/Kconfig      | 1 +
 hw/vmapple/meson.build  | 0
 hw/vmapple/trace-events | 2 ++
 hw/vmapple/trace.h      | 1 +
 meson.build             | 1 +
 8 files changed, 14 insertions(+)
 create mode 100644 hw/vmapple/Kconfig
 create mode 100644 hw/vmapple/meson.build
 create mode 100644 hw/vmapple/trace-events
 create mode 100644 hw/vmapple/trace.h

diff --git a/MAINTAINERS b/MAINTAINERS
index 0133b874847..4e7f25e5299 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -2777,6 +2777,13 @@ F: hw/hyperv/hv-balloon*.h
 F: include/hw/hyperv/dynmem-proto.h
 F: include/hw/hyperv/hv-balloon.h
 
+VMapple
+M: Alexander Graf <agraf@csgraf.de>
+R: Phil Dennis-Jordan <phil@philjordan.eu>
+S: Maintained
+F: hw/vmapple/*
+F: include/hw/vmapple/*
+
 Subsystems
 ----------
 Overall Audio backends
diff --git a/hw/Kconfig b/hw/Kconfig
index 6fdaff1b1be..4f0ad88a310 100644
--- a/hw/Kconfig
+++ b/hw/Kconfig
@@ -42,6 +42,7 @@ source ufs/Kconfig
 source usb/Kconfig
 source virtio/Kconfig
 source vfio/Kconfig
+source vmapple/Kconfig
 source xen/Kconfig
 source watchdog/Kconfig
 
diff --git a/hw/meson.build b/hw/meson.build
index e86badc5417..0b7151f948c 100644
--- a/hw/meson.build
+++ b/hw/meson.build
@@ -40,6 +40,7 @@ subdir('ufs')
 subdir('usb')
 subdir('vfio')
 subdir('virtio')
+subdir('vmapple')
 subdir('watchdog')
 subdir('xen')
 subdir('xenpv')
diff --git a/hw/vmapple/Kconfig b/hw/vmapple/Kconfig
new file mode 100644
index 00000000000..8b137891791
--- /dev/null
+++ b/hw/vmapple/Kconfig
@@ -0,0 +1 @@
+
diff --git a/hw/vmapple/meson.build b/hw/vmapple/meson.build
new file mode 100644
index 00000000000..e69de29bb2d
diff --git a/hw/vmapple/trace-events b/hw/vmapple/trace-events
new file mode 100644
index 00000000000..9ccc5790487
--- /dev/null
+++ b/hw/vmapple/trace-events
@@ -0,0 +1,2 @@
+# See docs/devel/tracing.rst for syntax documentation.
+
diff --git a/hw/vmapple/trace.h b/hw/vmapple/trace.h
new file mode 100644
index 00000000000..572adbefe04
--- /dev/null
+++ b/hw/vmapple/trace.h
@@ -0,0 +1 @@
+#include "trace/trace-hw_vmapple.h"
diff --git a/meson.build b/meson.build
index f09df3f09d5..092db201b95 100644
--- a/meson.build
+++ b/meson.build
@@ -3456,6 +3456,7 @@ if have_system
     'hw/usb',
     'hw/vfio',
     'hw/virtio',
+    'hw/vmapple',
     'hw/watchdog',
     'hw/xen',
     'hw/gpio',
-- 
2.39.3 (Apple Git-145)

[PATCH v3 07/14] hw/misc/pvpanic: Add MMIO interface

[Phil Dennis-Jordan]

From: Alexander Graf <graf@amazon.com>

In addition to the ISA and PCI variants of pvpanic, let's add an MMIO
platform device that we can use in embedded arm environments.

Signed-off-by: Alexander Graf <graf@amazon.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org>
Tested-by: Philippe Mathieu-Daudé <philmd@linaro.org>
Signed-off-by: Phil Dennis-Jordan <phil@philjordan.eu>

---
v3:
 * Rebased on upstream, updated a header path
 
 hw/misc/Kconfig           |  4 +++
 hw/misc/meson.build       |  1 +
 hw/misc/pvpanic-mmio.c    | 61 +++++++++++++++++++++++++++++++++++++++
 include/hw/misc/pvpanic.h |  1 +
 4 files changed, 67 insertions(+)
 create mode 100644 hw/misc/pvpanic-mmio.c

diff --git a/hw/misc/Kconfig b/hw/misc/Kconfig
index 1e08785b832..891329971a9 100644
--- a/hw/misc/Kconfig
+++ b/hw/misc/Kconfig
@@ -143,6 +143,10 @@ config PVPANIC_ISA
     depends on ISA_BUS
     select PVPANIC_COMMON
 
+config PVPANIC_MMIO
+    bool
+    select PVPANIC_COMMON
+
 config AUX
     bool
     select I2C
diff --git a/hw/misc/meson.build b/hw/misc/meson.build
index 2ca8717be28..19021700192 100644
--- a/hw/misc/meson.build
+++ b/hw/misc/meson.build
@@ -127,6 +127,7 @@ system_ss.add(when: 'CONFIG_ARMSSE_MHU', if_true: files('armsse-mhu.c'))
 
 system_ss.add(when: 'CONFIG_PVPANIC_ISA', if_true: files('pvpanic-isa.c'))
 system_ss.add(when: 'CONFIG_PVPANIC_PCI', if_true: files('pvpanic-pci.c'))
+system_ss.add(when: 'CONFIG_PVPANIC_MMIO', if_true: files('pvpanic-mmio.c'))
 system_ss.add(when: 'CONFIG_AUX', if_true: files('auxbus.c'))
 system_ss.add(when: 'CONFIG_ASPEED_SOC', if_true: files(
   'aspeed_hace.c',
diff --git a/hw/misc/pvpanic-mmio.c b/hw/misc/pvpanic-mmio.c
new file mode 100644
index 00000000000..56738efee53
--- /dev/null
+++ b/hw/misc/pvpanic-mmio.c
@@ -0,0 +1,61 @@
+/*
+ * QEMU simulated pvpanic device (MMIO frontend)
+ *
+ * Copyright © 2023 Amazon.com, Inc. or its affiliates. All Rights Reserved.
+ *
+ * SPDX-License-Identifier: GPL-2.0-or-later
+ */
+
+#include "qemu/osdep.h"
+
+#include "hw/qdev-properties.h"
+#include "hw/misc/pvpanic.h"
+#include "hw/sysbus.h"
+#include "standard-headers/misc/pvpanic.h"
+
+OBJECT_DECLARE_SIMPLE_TYPE(PVPanicMMIOState, PVPANIC_MMIO_DEVICE)
+
+#define PVPANIC_MMIO_SIZE 0x2
+
+struct PVPanicMMIOState {
+    SysBusDevice parent_obj;
+
+    PVPanicState pvpanic;
+};
+
+static void pvpanic_mmio_initfn(Object *obj)
+{
+    PVPanicMMIOState *s = PVPANIC_MMIO_DEVICE(obj);
+
+    pvpanic_setup_io(&s->pvpanic, DEVICE(s), PVPANIC_MMIO_SIZE);
+    sysbus_init_mmio(SYS_BUS_DEVICE(obj), &s->pvpanic.mr);
+}
+
+static Property pvpanic_mmio_properties[] = {
+    DEFINE_PROP_UINT8("events", PVPanicMMIOState, pvpanic.events,
+                      PVPANIC_PANICKED | PVPANIC_CRASH_LOADED),
+    DEFINE_PROP_END_OF_LIST(),
+};
+
+static void pvpanic_mmio_class_init(ObjectClass *klass, void *data)
+{
+    DeviceClass *dc = DEVICE_CLASS(klass);
+
+    device_class_set_props(dc, pvpanic_mmio_properties);
+    set_bit(DEVICE_CATEGORY_MISC, dc->categories);
+}
+
+static const TypeInfo pvpanic_mmio_info = {
+    .name          = TYPE_PVPANIC_MMIO_DEVICE,
+    .parent        = TYPE_SYS_BUS_DEVICE,
+    .instance_size = sizeof(PVPanicMMIOState),
+    .instance_init = pvpanic_mmio_initfn,
+    .class_init    = pvpanic_mmio_class_init,
+};
+
+static void pvpanic_register_types(void)
+{
+    type_register_static(&pvpanic_mmio_info);
+}
+
+type_init(pvpanic_register_types)
diff --git a/include/hw/misc/pvpanic.h b/include/hw/misc/pvpanic.h
index 9a71a5ad0d7..049a94c1125 100644
--- a/include/hw/misc/pvpanic.h
+++ b/include/hw/misc/pvpanic.h
@@ -26,6 +26,7 @@
 
 #define TYPE_PVPANIC_ISA_DEVICE "pvpanic"
 #define TYPE_PVPANIC_PCI_DEVICE "pvpanic-pci"
+#define TYPE_PVPANIC_MMIO_DEVICE "pvpanic-mmio"
 
 #define PVPANIC_IOPORT_PROP "ioport"
 
-- 
2.39.3 (Apple Git-145)

[PATCH v3 08/14] hvf: arm: Ignore writes to CNTP_CTL_EL0

[Phil Dennis-Jordan]

From: Alexander Graf <graf@amazon.com>

MacOS unconditionally disables interrupts of the physical timer on boot
and then continues to use the virtual one. We don't really want to support
a full physical timer emulation, so let's just ignore those writes.

Signed-off-by: Alexander Graf <graf@amazon.com>
Signed-off-by: Phil Dennis-Jordan <phil@philjordan.eu>
---
 target/arm/hvf/hvf.c | 9 +++++++++
 1 file changed, 9 insertions(+)

diff --git a/target/arm/hvf/hvf.c b/target/arm/hvf/hvf.c
index 6cea483d422..b45b764dfd0 100644
--- a/target/arm/hvf/hvf.c
+++ b/target/arm/hvf/hvf.c
@@ -11,6 +11,7 @@
 
 #include "qemu/osdep.h"
 #include "qemu/error-report.h"
+#include "qemu/log.h"
 
 #include "sysemu/runstate.h"
 #include "sysemu/hvf.h"
@@ -184,6 +185,7 @@ void hvf_arm_init_debug(void)
 #define SYSREG_OSLSR_EL1      SYSREG(2, 0, 1, 1, 4)
 #define SYSREG_OSDLR_EL1      SYSREG(2, 0, 1, 3, 4)
 #define SYSREG_CNTPCT_EL0     SYSREG(3, 3, 14, 0, 1)
+#define SYSREG_CNTP_CTL_EL0   SYSREG(3, 3, 14, 2, 1)
 #define SYSREG_PMCR_EL0       SYSREG(3, 3, 9, 12, 0)
 #define SYSREG_PMUSERENR_EL0  SYSREG(3, 3, 9, 14, 0)
 #define SYSREG_PMCNTENSET_EL0 SYSREG(3, 3, 9, 12, 1)
@@ -1620,6 +1622,13 @@ static int hvf_sysreg_write(CPUState *cpu, uint32_t reg, uint64_t val)
     case SYSREG_OSLAR_EL1:
         env->cp15.oslsr_el1 = val & 1;
         return 0;
+    case SYSREG_CNTP_CTL_EL0:
+        /*
+         * Guests should not rely on the physical counter, but macOS emits
+         * disable writes to it. Let it do so, but ignore the requests.
+         */
+        qemu_log_mask(LOG_UNIMP, "Unsupported write to CNTP_CTL_EL0\n");
+        return 0;
     case SYSREG_OSDLR_EL1:
         /* Dummy register */
         return 0;
-- 
2.39.3 (Apple Git-145)

[PATCH v3 09/14] gpex: Allow more than 4 legacy IRQs

[Phil Dennis-Jordan]

From: Alexander Graf <graf@amazon.com>

Some boards such as vmapple don't do real legacy PCI IRQ swizzling.
Instead, they just keep allocating more board IRQ lines for each new
legacy IRQ. Let's support that mode by giving instantiators a new
"nr_irqs" property they can use to support more than 4 legacy IRQ lines.
In this mode, GPEX will export more IRQ lines, one for each device.

Signed-off-by: Alexander Graf <graf@amazon.com>
Signed-off-by: Phil Dennis-Jordan <phil@philjordan.eu>
---
 hw/arm/sbsa-ref.c          |  2 +-
 hw/arm/virt.c              |  2 +-
 hw/i386/microvm.c          |  2 +-
 hw/loongarch/virt.c        |  2 +-
 hw/mips/loongson3_virt.c   |  2 +-
 hw/openrisc/virt.c         | 12 ++++++------
 hw/pci-host/gpex.c         | 36 +++++++++++++++++++++++++++++++-----
 hw/riscv/virt.c            | 12 ++++++------
 hw/xtensa/virt.c           |  2 +-
 include/hw/pci-host/gpex.h |  7 +++----
 10 files changed, 52 insertions(+), 27 deletions(-)

diff --git a/hw/arm/sbsa-ref.c b/hw/arm/sbsa-ref.c
index e3195d54497..7e7322486c2 100644
--- a/hw/arm/sbsa-ref.c
+++ b/hw/arm/sbsa-ref.c
@@ -673,7 +673,7 @@ static void create_pcie(SBSAMachineState *sms)
     /* Map IO port space */
     sysbus_mmio_map(SYS_BUS_DEVICE(dev), 2, base_pio);
 
-    for (i = 0; i < GPEX_NUM_IRQS; i++) {
+    for (i = 0; i < PCI_NUM_PINS; i++) {
         sysbus_connect_irq(SYS_BUS_DEVICE(dev), i,
                            qdev_get_gpio_in(sms->gic, irq + i));
         gpex_set_irq_num(GPEX_HOST(dev), i, irq + i);
diff --git a/hw/arm/virt.c b/hw/arm/virt.c
index 8b2b991d978..bd3b17be2ea 100644
--- a/hw/arm/virt.c
+++ b/hw/arm/virt.c
@@ -1547,7 +1547,7 @@ static void create_pcie(VirtMachineState *vms)
     /* Map IO port space */
     sysbus_mmio_map(SYS_BUS_DEVICE(dev), 2, base_pio);
 
-    for (i = 0; i < GPEX_NUM_IRQS; i++) {
+    for (i = 0; i < PCI_NUM_PINS; i++) {
         sysbus_connect_irq(SYS_BUS_DEVICE(dev), i,
                            qdev_get_gpio_in(vms->gic, irq + i));
         gpex_set_irq_num(GPEX_HOST(dev), i, irq + i);
diff --git a/hw/i386/microvm.c b/hw/i386/microvm.c
index 40edcee7af2..216a169c413 100644
--- a/hw/i386/microvm.c
+++ b/hw/i386/microvm.c
@@ -139,7 +139,7 @@ static void create_gpex(MicrovmMachineState *mms)
                                     mms->gpex.mmio64.base, mmio64_alias);
     }
 
-    for (i = 0; i < GPEX_NUM_IRQS; i++) {
+    for (i = 0; i < PCI_NUM_PINS; i++) {
         sysbus_connect_irq(SYS_BUS_DEVICE(dev), i,
                            x86ms->gsi[mms->gpex.irq + i]);
     }
diff --git a/hw/loongarch/virt.c b/hw/loongarch/virt.c
index 75980b6e3c7..8eb7277a8d1 100644
--- a/hw/loongarch/virt.c
+++ b/hw/loongarch/virt.c
@@ -703,7 +703,7 @@ static void virt_devices_init(DeviceState *pch_pic,
     memory_region_add_subregion(get_system_memory(), VIRT_PCI_IO_BASE,
                                 pio_alias);
 
-    for (i = 0; i < GPEX_NUM_IRQS; i++) {
+    for (i = 0; i < PCI_NUM_PINS; i++) {
         sysbus_connect_irq(d, i,
                            qdev_get_gpio_in(pch_pic, 16 + i));
         gpex_set_irq_num(GPEX_HOST(gpex_dev), i, 16 + i);
diff --git a/hw/mips/loongson3_virt.c b/hw/mips/loongson3_virt.c
index 2067b4fecb5..acafd73129d 100644
--- a/hw/mips/loongson3_virt.c
+++ b/hw/mips/loongson3_virt.c
@@ -458,7 +458,7 @@ static inline void loongson3_virt_devices_init(MachineState *machine,
                                 virt_memmap[VIRT_PCIE_PIO].base, s->pio_alias);
     sysbus_mmio_map(SYS_BUS_DEVICE(dev), 2, virt_memmap[VIRT_PCIE_PIO].base);
 
-    for (i = 0; i < GPEX_NUM_IRQS; i++) {
+    for (i = 0; i < PCI_NUM_PINS; i++) {
         irq = qdev_get_gpio_in(pic, PCIE_IRQ_BASE + i);
         sysbus_connect_irq(SYS_BUS_DEVICE(dev), i, irq);
         gpex_set_irq_num(GPEX_HOST(dev), i, PCIE_IRQ_BASE + i);
diff --git a/hw/openrisc/virt.c b/hw/openrisc/virt.c
index f8a68a6a6b1..16a5676c4bb 100644
--- a/hw/openrisc/virt.c
+++ b/hw/openrisc/virt.c
@@ -318,7 +318,7 @@ static void create_pcie_irq_map(void *fdt, char *nodename, int irq_base,
 {
     int pin, dev;
     uint32_t irq_map_stride = 0;
-    uint32_t full_irq_map[GPEX_NUM_IRQS * GPEX_NUM_IRQS * 6] = {};
+    uint32_t full_irq_map[PCI_NUM_PINS * PCI_NUM_PINS * 6] = {};
     uint32_t *irq_map = full_irq_map;
 
     /*
@@ -330,11 +330,11 @@ static void create_pcie_irq_map(void *fdt, char *nodename, int irq_base,
      * possible slot) seeing the interrupt-map-mask will allow the table
      * to wrap to any number of devices.
      */
-    for (dev = 0; dev < GPEX_NUM_IRQS; dev++) {
+    for (dev = 0; dev < PCI_NUM_PINS; dev++) {
         int devfn = dev << 3;
 
-        for (pin = 0; pin < GPEX_NUM_IRQS; pin++) {
-            int irq_nr = irq_base + ((pin + PCI_SLOT(devfn)) % GPEX_NUM_IRQS);
+        for (pin = 0; pin < PCI_NUM_PINS; pin++) {
+            int irq_nr = irq_base + ((pin + PCI_SLOT(devfn)) % PCI_NUM_PINS);
             int i = 0;
 
             /* Fill PCI address cells */
@@ -357,7 +357,7 @@ static void create_pcie_irq_map(void *fdt, char *nodename, int irq_base,
     }
 
     qemu_fdt_setprop(fdt, nodename, "interrupt-map", full_irq_map,
-                     GPEX_NUM_IRQS * GPEX_NUM_IRQS *
+                     PCI_NUM_PINS * PCI_NUM_PINS *
                      irq_map_stride * sizeof(uint32_t));
 
     qemu_fdt_setprop_cells(fdt, nodename, "interrupt-map-mask",
@@ -409,7 +409,7 @@ static void openrisc_virt_pcie_init(OR1KVirtState *state,
     memory_region_add_subregion(get_system_memory(), pio_base, alias);
 
     /* Connect IRQ lines. */
-    for (i = 0; i < GPEX_NUM_IRQS; i++) {
+    for (i = 0; i < PCI_NUM_PINS; i++) {
         pcie_irq = get_per_cpu_irq(cpus, num_cpus, irq_base + i);
 
         sysbus_connect_irq(SYS_BUS_DEVICE(dev), i, pcie_irq);
diff --git a/hw/pci-host/gpex.c b/hw/pci-host/gpex.c
index e9cf455bf52..4aca0a95771 100644
--- a/hw/pci-host/gpex.c
+++ b/hw/pci-host/gpex.c
@@ -32,6 +32,7 @@
 #include "qemu/osdep.h"
 #include "qapi/error.h"
 #include "hw/irq.h"
+#include "hw/pci/pci_bus.h"
 #include "hw/pci-host/gpex.h"
 #include "hw/qdev-properties.h"
 #include "migration/vmstate.h"
@@ -50,7 +51,7 @@ static void gpex_set_irq(void *opaque, int irq_num, int level)
 
 int gpex_set_irq_num(GPEXHost *s, int index, int gsi)
 {
-    if (index >= GPEX_NUM_IRQS) {
+    if (index >= s->nr_irqs) {
         return -EINVAL;
     }
 
@@ -74,14 +75,29 @@ static PCIINTxRoute gpex_route_intx_pin_to_irq(void *opaque, int pin)
     return route;
 }
 
+static int gpex_swizzle_map_irq_fn(PCIDevice *pci_dev, int pin)
+{
+    PCIBus *bus = pci_device_root_bus(pci_dev);
+
+    return (PCI_SLOT(pci_dev->devfn) + pin) % bus->nirq;
+}
+
 static void gpex_host_realize(DeviceState *dev, Error **errp)
 {
     PCIHostState *pci = PCI_HOST_BRIDGE(dev);
     GPEXHost *s = GPEX_HOST(dev);
     SysBusDevice *sbd = SYS_BUS_DEVICE(dev);
     PCIExpressHost *pex = PCIE_HOST_BRIDGE(dev);
+    pci_map_irq_fn map_irq_fn = pci_swizzle_map_irq_fn;
     int i;
 
+    s->irq = g_malloc0(s->nr_irqs * sizeof(*s->irq));
+    s->irq_num = g_malloc0(s->nr_irqs * sizeof(*s->irq_num));
+
+    if (s->nr_irqs != PCI_NUM_PINS) {
+        map_irq_fn = gpex_swizzle_map_irq_fn;
+    }
+
     pcie_host_mmcfg_init(pex, PCIE_MMCFG_SIZE_MAX);
     sysbus_init_mmio(sbd, &pex->mmio);
 
@@ -128,19 +144,27 @@ static void gpex_host_realize(DeviceState *dev, Error **errp)
         sysbus_init_mmio(sbd, &s->io_ioport);
     }
 
-    for (i = 0; i < GPEX_NUM_IRQS; i++) {
+    for (i = 0; i < s->nr_irqs; i++) {
         sysbus_init_irq(sbd, &s->irq[i]);
         s->irq_num[i] = -1;
     }
 
-    pci->bus = pci_register_root_bus(dev, "pcie.0", gpex_set_irq,
-                                     pci_swizzle_map_irq_fn, s, &s->io_mmio,
-                                     &s->io_ioport, 0, 4, TYPE_PCIE_BUS);
+    pci->bus = pci_register_root_bus(dev, "pcie.0", gpex_set_irq, map_irq_fn,
+                                     s, &s->io_mmio, &s->io_ioport, 0,
+                                     s->nr_irqs, TYPE_PCIE_BUS);
 
     pci_bus_set_route_irq_fn(pci->bus, gpex_route_intx_pin_to_irq);
     qdev_realize(DEVICE(&s->gpex_root), BUS(pci->bus), &error_fatal);
 }
 
+static void gpex_host_unrealize(DeviceState *dev)
+{
+    GPEXHost *s = GPEX_HOST(dev);
+
+    g_free(s->irq);
+    g_free(s->irq_num);
+}
+
 static const char *gpex_host_root_bus_path(PCIHostState *host_bridge,
                                           PCIBus *rootbus)
 {
@@ -166,6 +190,7 @@ static Property gpex_host_properties[] = {
                        gpex_cfg.mmio64.base, 0),
     DEFINE_PROP_SIZE(PCI_HOST_ABOVE_4G_MMIO_SIZE, GPEXHost,
                      gpex_cfg.mmio64.size, 0),
+    DEFINE_PROP_UINT32("nr-irqs", GPEXHost, nr_irqs, PCI_NUM_PINS),
     DEFINE_PROP_END_OF_LIST(),
 };
 
@@ -176,6 +201,7 @@ static void gpex_host_class_init(ObjectClass *klass, void *data)
 
     hc->root_bus_path = gpex_host_root_bus_path;
     dc->realize = gpex_host_realize;
+    dc->unrealize = gpex_host_unrealize;
     set_bit(DEVICE_CATEGORY_BRIDGE, dc->categories);
     dc->fw_name = "pci";
     device_class_set_props(dc, gpex_host_properties);
diff --git a/hw/riscv/virt.c b/hw/riscv/virt.c
index cef41c150aa..6c3ed4b3d9c 100644
--- a/hw/riscv/virt.c
+++ b/hw/riscv/virt.c
@@ -167,7 +167,7 @@ static void create_pcie_irq_map(RISCVVirtState *s, void *fdt, char *nodename,
 {
     int pin, dev;
     uint32_t irq_map_stride = 0;
-    uint32_t full_irq_map[GPEX_NUM_IRQS * GPEX_NUM_IRQS *
+    uint32_t full_irq_map[PCI_NUM_PINS * PCI_NUM_PINS *
                           FDT_MAX_INT_MAP_WIDTH] = {};
     uint32_t *irq_map = full_irq_map;
 
@@ -179,11 +179,11 @@ static void create_pcie_irq_map(RISCVVirtState *s, void *fdt, char *nodename,
      * possible slot) seeing the interrupt-map-mask will allow the table
      * to wrap to any number of devices.
      */
-    for (dev = 0; dev < GPEX_NUM_IRQS; dev++) {
+    for (dev = 0; dev < PCI_NUM_PINS; dev++) {
         int devfn = dev * 0x8;
 
-        for (pin = 0; pin < GPEX_NUM_IRQS; pin++) {
-            int irq_nr = PCIE_IRQ + ((pin + PCI_SLOT(devfn)) % GPEX_NUM_IRQS);
+        for (pin = 0; pin < PCI_NUM_PINS; pin++) {
+            int irq_nr = PCIE_IRQ + ((pin + PCI_SLOT(devfn)) % PCI_NUM_PINS);
             int i = 0;
 
             /* Fill PCI address cells */
@@ -209,7 +209,7 @@ static void create_pcie_irq_map(RISCVVirtState *s, void *fdt, char *nodename,
     }
 
     qemu_fdt_setprop(fdt, nodename, "interrupt-map", full_irq_map,
-                     GPEX_NUM_IRQS * GPEX_NUM_IRQS *
+                     PCI_NUM_PINS * PCI_NUM_PINS *
                      irq_map_stride * sizeof(uint32_t));
 
     qemu_fdt_setprop_cells(fdt, nodename, "interrupt-map-mask",
@@ -1157,7 +1157,7 @@ static inline DeviceState *gpex_pcie_init(MemoryRegion *sys_mem,
 
     sysbus_mmio_map(SYS_BUS_DEVICE(dev), 2, pio_base);
 
-    for (i = 0; i < GPEX_NUM_IRQS; i++) {
+    for (i = 0; i < PCI_NUM_PINS; i++) {
         irq = qdev_get_gpio_in(irqchip, PCIE_IRQ + i);
 
         sysbus_connect_irq(SYS_BUS_DEVICE(dev), i, irq);
diff --git a/hw/xtensa/virt.c b/hw/xtensa/virt.c
index 5310a888613..8f5c2009d29 100644
--- a/hw/xtensa/virt.c
+++ b/hw/xtensa/virt.c
@@ -93,7 +93,7 @@ static void create_pcie(MachineState *ms, CPUXtensaState *env, int irq_base,
     /* Connect IRQ lines. */
     extints = xtensa_get_extints(env);
 
-    for (i = 0; i < GPEX_NUM_IRQS; i++) {
+    for (i = 0; i < PCI_NUM_PINS; i++) {
         void *q = extints[irq_base + i];
 
         sysbus_connect_irq(SYS_BUS_DEVICE(dev), i, q);
diff --git a/include/hw/pci-host/gpex.h b/include/hw/pci-host/gpex.h
index dce883573ba..8f87a3872cb 100644
--- a/include/hw/pci-host/gpex.h
+++ b/include/hw/pci-host/gpex.h
@@ -32,8 +32,6 @@ OBJECT_DECLARE_SIMPLE_TYPE(GPEXHost, GPEX_HOST)
 #define TYPE_GPEX_ROOT_DEVICE "gpex-root"
 OBJECT_DECLARE_SIMPLE_TYPE(GPEXRootState, GPEX_ROOT_DEVICE)
 
-#define GPEX_NUM_IRQS 4
-
 struct GPEXRootState {
     /*< private >*/
     PCIDevice parent_obj;
@@ -60,8 +58,9 @@ struct GPEXHost {
     MemoryRegion io_mmio;
     MemoryRegion io_ioport_window;
     MemoryRegion io_mmio_window;
-    qemu_irq irq[GPEX_NUM_IRQS];
-    int irq_num[GPEX_NUM_IRQS];
+    uint32_t nr_irqs;
+    qemu_irq *irq;
+    int *irq_num;
 
     bool allow_unmapped_accesses;
 
-- 
2.39.3 (Apple Git-145)

[PATCH v3 10/14] hw/vmapple/aes: Introduce aes engine

[Phil Dennis-Jordan]

From: Alexander Graf <graf@amazon.com>

VMApple contains an "aes" engine device that it uses to encrypt and
decrypt its nvram. It has trivial hard coded keys it uses for that
purpose.

Add device emulation for this device model.

Signed-off-by: Alexander Graf <graf@amazon.com>
Co-authored-by: Phil Dennis-Jordan <phil@philjordan.eu>
Signed-off-by: Phil Dennis-Jordan <phil@philjordan.eu>

---
v3:

 * Rebased on latest upstream and fixed minor breakages.
 * Replaced legacy device reset method with Resettable method

 hw/vmapple/Kconfig      |   2 +
 hw/vmapple/aes.c        | 584 ++++++++++++++++++++++++++++++++++++++++
 hw/vmapple/meson.build  |   1 +
 hw/vmapple/trace-events |  19 ++
 4 files changed, 606 insertions(+)
 create mode 100644 hw/vmapple/aes.c

diff --git a/hw/vmapple/Kconfig b/hw/vmapple/Kconfig
index 8b137891791..a73504d5999 100644
--- a/hw/vmapple/Kconfig
+++ b/hw/vmapple/Kconfig
@@ -1 +1,3 @@
+config VMAPPLE_AES
+    bool
 
diff --git a/hw/vmapple/aes.c b/hw/vmapple/aes.c
new file mode 100644
index 00000000000..074fbdd9c36
--- /dev/null
+++ b/hw/vmapple/aes.c
@@ -0,0 +1,584 @@
+/*
+ * QEMU Apple AES device emulation
+ *
+ * Copyright © 2023 Amazon.com, Inc. or its affiliates. All Rights Reserved.
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or later.
+ * See the COPYING file in the top-level directory.
+ */
+
+#include "qemu/osdep.h"
+#include "hw/irq.h"
+#include "migration/vmstate.h"
+#include "qemu/log.h"
+#include "qemu/module.h"
+#include "trace.h"
+#include "hw/sysbus.h"
+#include "crypto/hash.h"
+#include "crypto/aes.h"
+#include "crypto/cipher.h"
+
+#define TYPE_AES          "apple-aes"
+#define MAX_FIFO_SIZE     9
+
+#define CMD_KEY           0x1
+#define CMD_KEY_CONTEXT_SHIFT    27
+#define CMD_KEY_CONTEXT_MASK     (0x1 << CMD_KEY_CONTEXT_SHIFT)
+#define CMD_KEY_SELECT_SHIFT     24
+#define CMD_KEY_SELECT_MASK      (0x7 << CMD_KEY_SELECT_SHIFT)
+#define CMD_KEY_KEY_LEN_SHIFT    22
+#define CMD_KEY_KEY_LEN_MASK     (0x3 << CMD_KEY_KEY_LEN_SHIFT)
+#define CMD_KEY_ENCRYPT_SHIFT    20
+#define CMD_KEY_ENCRYPT_MASK     (0x1 << CMD_KEY_ENCRYPT_SHIFT)
+#define CMD_KEY_BLOCK_MODE_SHIFT 16
+#define CMD_KEY_BLOCK_MODE_MASK  (0x3 << CMD_KEY_BLOCK_MODE_SHIFT)
+#define CMD_IV            0x2
+#define CMD_IV_CONTEXT_SHIFT     26
+#define CMD_IV_CONTEXT_MASK      (0x3 << CMD_KEY_CONTEXT_SHIFT)
+#define CMD_DSB           0x3
+#define CMD_SKG           0x4
+#define CMD_DATA          0x5
+#define CMD_DATA_KEY_CTX_SHIFT   27
+#define CMD_DATA_KEY_CTX_MASK    (0x1 << CMD_DATA_KEY_CTX_SHIFT)
+#define CMD_DATA_IV_CTX_SHIFT    25
+#define CMD_DATA_IV_CTX_MASK     (0x3 << CMD_DATA_IV_CTX_SHIFT)
+#define CMD_DATA_LEN_MASK        0xffffff
+#define CMD_STORE_IV      0x6
+#define CMD_STORE_IV_ADDR_MASK   0xffffff
+#define CMD_WRITE_REG     0x7
+#define CMD_FLAG          0x8
+#define CMD_FLAG_STOP_MASK       BIT(26)
+#define CMD_FLAG_RAISE_IRQ_MASK  BIT(27)
+#define CMD_FLAG_INFO_MASK       0xff
+#define CMD_MAX           0x10
+
+#define CMD_SHIFT         28
+
+#define REG_STATUS            0xc
+#define REG_STATUS_DMA_READ_RUNNING     BIT(0)
+#define REG_STATUS_DMA_READ_PENDING     BIT(1)
+#define REG_STATUS_DMA_WRITE_RUNNING    BIT(2)
+#define REG_STATUS_DMA_WRITE_PENDING    BIT(3)
+#define REG_STATUS_BUSY                 BIT(4)
+#define REG_STATUS_EXECUTING            BIT(5)
+#define REG_STATUS_READY                BIT(6)
+#define REG_STATUS_TEXT_DPA_SEEDED      BIT(7)
+#define REG_STATUS_UNWRAP_DPA_SEEDED    BIT(8)
+
+#define REG_IRQ_STATUS        0x18
+#define REG_IRQ_STATUS_INVALID_CMD      BIT(2)
+#define REG_IRQ_STATUS_FLAG             BIT(5)
+#define REG_IRQ_ENABLE        0x1c
+#define REG_WATERMARK         0x20
+#define REG_Q_STATUS          0x24
+#define REG_FLAG_INFO         0x30
+#define REG_FIFO              0x200
+
+static const uint32_t key_lens[4] = {
+    [0] = 16,
+    [1] = 24,
+    [2] = 32,
+    [3] = 64,
+};
+
+struct key {
+    uint32_t key_len;
+    uint32_t key[8];
+};
+
+struct iv {
+    uint32_t iv[4];
+};
+
+struct context {
+    struct key key;
+    struct iv iv;
+};
+
+static struct key builtin_keys[7] = {
+    [1] = {
+        .key_len = 32,
+        .key = { 0x1 },
+    },
+    [2] = {
+        .key_len = 32,
+        .key = { 0x2 },
+    },
+    [3] = {
+        .key_len = 32,
+        .key = { 0x3 },
+    }
+};
+
+typedef struct AESState {
+    /* Private */
+    SysBusDevice parent_obj;
+
+    /* Public */
+    qemu_irq irq;
+    MemoryRegion iomem1;
+    MemoryRegion iomem2;
+
+    uint32_t status;
+    uint32_t q_status;
+    uint32_t irq_status;
+    uint32_t irq_enable;
+    uint32_t watermark;
+    uint32_t flag_info;
+    uint32_t fifo[MAX_FIFO_SIZE];
+    uint32_t fifo_idx;
+    struct key key[2];
+    struct iv iv[4];
+    bool is_encrypt;
+    QCryptoCipherMode block_mode;
+} AESState;
+
+OBJECT_DECLARE_SIMPLE_TYPE(AESState, AES)
+
+static void aes_update_irq(AESState *s)
+{
+    qemu_set_irq(s->irq, !!(s->irq_status & s->irq_enable));
+}
+
+static uint64_t aes1_read(void *opaque, hwaddr offset, unsigned size)
+{
+    AESState *s = opaque;
+    uint64_t res = 0;
+
+    switch (offset) {
+    case REG_STATUS:
+        res = s->status;
+        break;
+    case REG_IRQ_STATUS:
+        res = s->irq_status;
+        break;
+    case REG_IRQ_ENABLE:
+        res = s->irq_enable;
+        break;
+    case REG_WATERMARK:
+        res = s->watermark;
+        break;
+    case REG_Q_STATUS:
+        res = s->q_status;
+        break;
+    case REG_FLAG_INFO:
+        res = s->flag_info;
+        break;
+
+    default:
+        trace_aes_read_unknown(offset);
+        break;
+    }
+
+    trace_aes_read(offset, res);
+
+    return res;
+}
+
+static void fifo_append(AESState *s, uint64_t val)
+{
+    if (s->fifo_idx == MAX_FIFO_SIZE) {
+        /* Exceeded the FIFO. Bail out */
+        return;
+    }
+
+    s->fifo[s->fifo_idx++] = val;
+}
+
+static bool has_payload(AESState *s, uint32_t elems)
+{
+    return s->fifo_idx >= (elems + 1);
+}
+
+static bool cmd_key(AESState *s)
+{
+    uint32_t cmd = s->fifo[0];
+    uint32_t key_select = (cmd & CMD_KEY_SELECT_MASK) >> CMD_KEY_SELECT_SHIFT;
+    uint32_t ctxt = (cmd & CMD_KEY_CONTEXT_MASK) >> CMD_KEY_CONTEXT_SHIFT;
+    uint32_t key_len;
+
+    switch ((cmd & CMD_KEY_BLOCK_MODE_MASK) >> CMD_KEY_BLOCK_MODE_SHIFT) {
+    case 0:
+        s->block_mode = QCRYPTO_CIPHER_MODE_ECB;
+        break;
+    case 1:
+        s->block_mode = QCRYPTO_CIPHER_MODE_CBC;
+        break;
+    default:
+        return false;
+    }
+
+    s->is_encrypt = !!((cmd & CMD_KEY_ENCRYPT_MASK) >> CMD_KEY_ENCRYPT_SHIFT);
+    key_len = key_lens[((cmd & CMD_KEY_KEY_LEN_MASK) >> CMD_KEY_KEY_LEN_SHIFT)];
+
+    if (key_select) {
+        trace_aes_cmd_key_select_builtin(ctxt, key_select,
+                                         s->is_encrypt ? "en" : "de",
+                                         QCryptoCipherMode_str(s->block_mode));
+        s->key[ctxt] = builtin_keys[key_select];
+    } else {
+        trace_aes_cmd_key_select_new(ctxt, key_len,
+                                     s->is_encrypt ? "en" : "de",
+                                     QCryptoCipherMode_str(s->block_mode));
+        if (key_len > sizeof(s->key[ctxt].key)) {
+            return false;
+        }
+        if (!has_payload(s, key_len / sizeof(uint32_t))) {
+            /* wait for payload */
+            return false;
+        }
+        memcpy(&s->key[ctxt].key, &s->fifo[1], key_len);
+        s->key[ctxt].key_len = key_len;
+    }
+
+    return true;
+}
+
+static bool cmd_iv(AESState *s)
+{
+    uint32_t cmd = s->fifo[0];
+    uint32_t ctxt = (cmd & CMD_IV_CONTEXT_MASK) >> CMD_IV_CONTEXT_SHIFT;
+
+    if (!has_payload(s, 4)) {
+        /* wait for payload */
+        return false;
+    }
+    memcpy(&s->iv[ctxt].iv, &s->fifo[1], sizeof(s->iv[ctxt].iv));
+    trace_aes_cmd_iv(ctxt, s->fifo[1], s->fifo[2], s->fifo[3], s->fifo[4]);
+
+    return true;
+}
+
+static char hexdigit2str(uint8_t val)
+{
+    g_assert(val < 0x10);
+    if (val >= 0xa) {
+        return 'a' + (val - 0xa);
+    } else {
+        return '0' + val;
+    }
+}
+
+static void dump_data(const char *desc, const void *p, size_t len)
+{
+    char *hex = alloca((len * 2) + 1);
+    const uint8_t *data = p;
+    char *hexp = hex;
+    size_t i;
+
+    if (len > 0x1000) {
+        /* Too large buffer, let's bail out */
+        return;
+    }
+
+    for (i = 0; i < len; i++) {
+        uint8_t val = data[i];
+        *(hexp++) = hexdigit2str(val >> 4);
+        *(hexp++) = hexdigit2str(val & 0xf);
+    }
+    *hexp = '\0';
+
+    trace_aes_dump_data(desc, hex);
+}
+
+static bool cmd_data(AESState *s)
+{
+    uint32_t cmd = s->fifo[0];
+    uint32_t ctxt_iv = 0;
+    uint32_t ctxt_key = (cmd & CMD_DATA_KEY_CTX_MASK) >> CMD_DATA_KEY_CTX_SHIFT;
+    uint32_t len = cmd & CMD_DATA_LEN_MASK;
+    uint64_t src_addr = s->fifo[2];
+    uint64_t dst_addr = s->fifo[3];
+    QCryptoCipherAlgo alg;
+    QCryptoCipher *cipher;
+    char *src;
+    char *dst;
+
+    src_addr |= ((uint64_t)s->fifo[1] << 16) & 0xffff00000000ULL;
+    dst_addr |= ((uint64_t)s->fifo[1] << 32) & 0xffff00000000ULL;
+
+    trace_aes_cmd_data(ctxt_key, ctxt_iv, src_addr, dst_addr, len);
+
+    if (!has_payload(s, 3)) {
+        /* wait for payload */
+        trace_aes_cmd_data_error("No payload");
+        return false;
+    }
+
+    if (ctxt_key >= ARRAY_SIZE(s->key) ||
+        ctxt_iv >= ARRAY_SIZE(s->iv)) {
+        /* Invalid input */
+        trace_aes_cmd_data_error("Invalid key or iv");
+        return false;
+    }
+
+    src = g_malloc0(len);
+    dst = g_malloc0(len);
+
+    cpu_physical_memory_read(src_addr, src, len);
+
+    dump_data("cmd_data(): src_data=", src, len);
+
+    switch (s->key[ctxt_key].key_len) {
+    case 128 / 8:
+        alg = QCRYPTO_CIPHER_ALGO_AES_128;
+        break;
+    case 192 / 8:
+        alg = QCRYPTO_CIPHER_ALGO_AES_192;
+        break;
+    case 256 / 8:
+        alg = QCRYPTO_CIPHER_ALGO_AES_256;
+        break;
+    default:
+        trace_aes_cmd_data_error("Invalid key len");
+        goto err_free;
+    }
+    cipher = qcrypto_cipher_new(alg, s->block_mode,
+                                (void *)s->key[ctxt_key].key,
+                                s->key[ctxt_key].key_len, NULL);
+    g_assert(cipher != NULL);
+    if (s->block_mode != QCRYPTO_CIPHER_MODE_ECB) {
+        if (qcrypto_cipher_setiv(cipher, (void *)s->iv[ctxt_iv].iv,
+                                 sizeof(s->iv[ctxt_iv].iv), NULL) != 0) {
+            trace_aes_cmd_data_error("Failed to set IV");
+            goto err_free_cipher;
+        }
+    }
+    if (s->is_encrypt) {
+        if (qcrypto_cipher_encrypt(cipher, src, dst, len, NULL) != 0) {
+            trace_aes_cmd_data_error("Encrypt failed");
+            goto err_free_cipher;
+        }
+    } else {
+        if (qcrypto_cipher_decrypt(cipher, src, dst, len, NULL) != 0) {
+            trace_aes_cmd_data_error("Decrypt failed");
+            goto err_free_cipher;
+        }
+    }
+    qcrypto_cipher_free(cipher);
+
+    dump_data("cmd_data(): dst_data=", dst, len);
+    cpu_physical_memory_write(dst_addr, dst, len);
+    g_free(src);
+    g_free(dst);
+
+    return true;
+
+err_free_cipher:
+    qcrypto_cipher_free(cipher);
+err_free:
+    g_free(src);
+    g_free(dst);
+    return false;
+}
+
+static bool cmd_store_iv(AESState *s)
+{
+    uint32_t cmd = s->fifo[0];
+    uint32_t ctxt = (cmd & CMD_IV_CONTEXT_MASK) >> CMD_IV_CONTEXT_SHIFT;
+    uint64_t addr = s->fifo[1];
+
+    if (!has_payload(s, 1)) {
+        /* wait for payload */
+        return false;
+    }
+
+    if (ctxt >= ARRAY_SIZE(s->iv)) {
+        /* Invalid context selected */
+        return false;
+    }
+
+    addr |= ((uint64_t)cmd << 32) & 0xff00000000ULL;
+    cpu_physical_memory_write(addr, &s->iv[ctxt].iv, sizeof(s->iv[ctxt].iv));
+
+    trace_aes_cmd_store_iv(ctxt, addr, s->iv[ctxt].iv[0], s->iv[ctxt].iv[1],
+                           s->iv[ctxt].iv[2], s->iv[ctxt].iv[3]);
+
+    return true;
+}
+
+static bool cmd_flag(AESState *s)
+{
+    uint32_t cmd = s->fifo[0];
+    uint32_t raise_irq = cmd & CMD_FLAG_RAISE_IRQ_MASK;
+
+    /* We always process data when it's coming in, so fire an IRQ immediately */
+    if (raise_irq) {
+        s->irq_status |= REG_IRQ_STATUS_FLAG;
+    }
+
+    s->flag_info = cmd & CMD_FLAG_INFO_MASK;
+
+    trace_aes_cmd_flag(!!raise_irq, s->flag_info);
+
+    return true;
+}
+
+static void fifo_process(AESState *s)
+{
+    uint32_t cmd = s->fifo[0] >> CMD_SHIFT;
+    bool success = false;
+
+    if (!s->fifo_idx) {
+        return;
+    }
+
+    switch (cmd) {
+    case CMD_KEY:
+        success = cmd_key(s);
+        break;
+    case CMD_IV:
+        success = cmd_iv(s);
+        break;
+    case CMD_DATA:
+        success = cmd_data(s);
+        break;
+    case CMD_STORE_IV:
+        success = cmd_store_iv(s);
+        break;
+    case CMD_FLAG:
+        success = cmd_flag(s);
+        break;
+    default:
+        s->irq_status |= REG_IRQ_STATUS_INVALID_CMD;
+        break;
+    }
+
+    if (success) {
+        s->fifo_idx = 0;
+    }
+
+    trace_aes_fifo_process(cmd, success ? 1 : 0);
+}
+
+static void aes1_write(void *opaque, hwaddr offset, uint64_t val, unsigned size)
+{
+    AESState *s = opaque;
+
+    trace_aes_write(offset, val);
+
+    switch (offset) {
+    case REG_IRQ_STATUS:
+        s->irq_status &= ~val;
+        break;
+    case REG_IRQ_ENABLE:
+        s->irq_enable = val;
+        break;
+    case REG_FIFO:
+        fifo_append(s, val);
+        fifo_process(s);
+        break;
+    default:
+        trace_aes_write_unknown(offset);
+        return;
+    }
+
+    aes_update_irq(s);
+}
+
+static const MemoryRegionOps aes1_ops = {
+    .read = aes1_read,
+    .write = aes1_write,
+    .endianness = DEVICE_NATIVE_ENDIAN,
+    .valid = {
+        .min_access_size = 4,
+        .max_access_size = 8,
+    },
+    .impl = {
+        .min_access_size = 4,
+        .max_access_size = 4,
+    },
+};
+
+static uint64_t aes2_read(void *opaque, hwaddr offset, unsigned size)
+{
+    uint64_t res = 0;
+
+    switch (offset) {
+    case 0:
+        res = 0;
+        break;
+    default:
+        trace_aes_2_read_unknown(offset);
+        break;
+    }
+
+    trace_aes_2_read(offset, res);
+
+    return res;
+}
+
+static void aes2_write(void *opaque, hwaddr offset, uint64_t val, unsigned size)
+{
+    trace_aes_2_write(offset, val);
+
+    switch (offset) {
+    default:
+        trace_aes_2_write_unknown(offset);
+        return;
+    }
+}
+
+static const MemoryRegionOps aes2_ops = {
+    .read = aes2_read,
+    .write = aes2_write,
+    .endianness = DEVICE_NATIVE_ENDIAN,
+    .valid = {
+        .min_access_size = 4,
+        .max_access_size = 8,
+    },
+    .impl = {
+        .min_access_size = 4,
+        .max_access_size = 4,
+    },
+};
+
+static void aes_reset(Object *obj, ResetType type)
+{
+    AESState *s = AES(obj);
+
+    s->status = 0x3f80;
+    s->q_status = 2;
+    s->irq_status = 0;
+    s->irq_enable = 0;
+    s->watermark = 0;
+}
+
+static void aes_init(Object *obj)
+{
+    AESState *s = AES(obj);
+
+    memory_region_init_io(&s->iomem1, obj, &aes1_ops, s, TYPE_AES, 0x4000);
+    memory_region_init_io(&s->iomem2, obj, &aes2_ops, s, TYPE_AES, 0x4000);
+    sysbus_init_mmio(SYS_BUS_DEVICE(s), &s->iomem1);
+    sysbus_init_mmio(SYS_BUS_DEVICE(s), &s->iomem2);
+    sysbus_init_irq(SYS_BUS_DEVICE(s), &s->irq);
+}
+
+static void aes_realize(DeviceState *dev, Error **errp)
+{
+}
+
+static void aes_class_init(ObjectClass *klass, void *data)
+{
+    DeviceClass *dc = DEVICE_CLASS(klass);
+    ResettableClass *rc = RESETTABLE_CLASS(klass);
+
+    rc->phases.hold = aes_reset;
+    dc->realize = aes_realize;
+}
+
+static const TypeInfo aes_info = {
+    .name          = TYPE_AES,
+    .parent        = TYPE_SYS_BUS_DEVICE,
+    .instance_size = sizeof(AESState),
+    .class_init    = aes_class_init,
+    .instance_init = aes_init,
+};
+
+static void aes_register_types(void)
+{
+    type_register_static(&aes_info);
+}
+
+type_init(aes_register_types)
diff --git a/hw/vmapple/meson.build b/hw/vmapple/meson.build
index e69de29bb2d..bcd4dcb28d2 100644
--- a/hw/vmapple/meson.build
+++ b/hw/vmapple/meson.build
@@ -0,0 +1 @@
+system_ss.add(when: 'CONFIG_VMAPPLE_AES',  if_true: files('aes.c'))
diff --git a/hw/vmapple/trace-events b/hw/vmapple/trace-events
index 9ccc5790487..1c9a3326eb4 100644
--- a/hw/vmapple/trace-events
+++ b/hw/vmapple/trace-events
@@ -1,2 +1,21 @@
 # See docs/devel/tracing.rst for syntax documentation.
 
+# aes.c
+aes_read_unknown(uint64_t offset) "offset=0x%"PRIx64
+aes_read(uint64_t offset, uint64_t res) "offset=0x%"PRIx64" res=0x%"PRIx64
+aes_cmd_key_select_builtin(uint32_t ctx, uint32_t key_id, const char *direction, const char *cipher) "[%d] Selecting builtin key %d to %scrypt with %s"
+aes_cmd_key_select_new(uint32_t ctx, uint32_t key_len, const char *direction, const char *cipher) "[%d] Selecting new key size=%d to %scrypt with %s"
+aes_cmd_iv(uint32_t ctx, uint32_t iv0, uint32_t iv1, uint32_t iv2, uint32_t iv3) "[%d] 0x%08x 0x%08x 0x%08x 0x%08x"
+aes_cmd_data(uint32_t key, uint32_t iv, uint64_t src, uint64_t dst, uint32_t len) "[key=%d iv=%d] src=0x%"PRIx64" dst=0x%"PRIx64" len=0x%x"
+aes_cmd_data_error(const char *reason) "reason=%s"
+aes_cmd_store_iv(uint32_t ctx, uint64_t addr, uint32_t iv0, uint32_t iv1, uint32_t iv2, uint32_t iv3) "[%d] addr=0x%"PRIx64"x -> 0x%08x 0x%08x 0x%08x 0x%08x"
+aes_cmd_flag(uint32_t raise, uint32_t flag_info) "raise=%d flag_info=0x%x"
+aes_fifo_process(uint32_t cmd, uint32_t success) "cmd=%d success=%d"
+aes_write_unknown(uint64_t offset) "offset=0x%"PRIx64
+aes_write(uint64_t offset, uint64_t val) "offset=0x%"PRIx64" val=0x%"PRIx64
+aes_2_read_unknown(uint64_t offset) "offset=0x%"PRIx64
+aes_2_read(uint64_t offset, uint64_t res) "offset=0x%"PRIx64" res=0x%"PRIx64
+aes_2_write_unknown(uint64_t offset) "offset=0x%"PRIx64
+aes_2_write(uint64_t offset, uint64_t val) "offset=0x%"PRIx64" val=0x%"PRIx64
+aes_dump_data(const char *desc, const char *hex) "%s%s"
+
-- 
2.39.3 (Apple Git-145)

[PATCH v3 11/14] hw/vmapple/bdif: Introduce vmapple backdoor interface

[Phil Dennis-Jordan]

From: Alexander Graf <graf@amazon.com>

The VMApple machine exposes AUX and ROOT block devices (as well as USB OTG
emulation) via virtio-pci as well as a special, simple backdoor platform
device.

This patch implements this backdoor platform device to the best of my
understanding. I left out any USB OTG parts; they're only needed for
guest recovery and I don't understand the protocol yet.

Signed-off-by: Alexander Graf <graf@amazon.com>
Signed-off-by: Phil Dennis-Jordan <phil@philjordan.eu>
---
 hw/vmapple/Kconfig        |   3 +
 hw/vmapple/bdif.c         | 245 ++++++++++++++++++++++++++++++++++++++
 hw/vmapple/meson.build    |   1 +
 hw/vmapple/trace-events   |   5 +
 include/hw/vmapple/bdif.h |  31 +++++
 5 files changed, 285 insertions(+)
 create mode 100644 hw/vmapple/bdif.c
 create mode 100644 include/hw/vmapple/bdif.h

diff --git a/hw/vmapple/Kconfig b/hw/vmapple/Kconfig
index a73504d5999..68f88876eb9 100644
--- a/hw/vmapple/Kconfig
+++ b/hw/vmapple/Kconfig
@@ -1,3 +1,6 @@
 config VMAPPLE_AES
     bool
 
+config VMAPPLE_BDIF
+    bool
+
diff --git a/hw/vmapple/bdif.c b/hw/vmapple/bdif.c
new file mode 100644
index 00000000000..36b5915ff30
--- /dev/null
+++ b/hw/vmapple/bdif.c
@@ -0,0 +1,245 @@
+/*
+ * VMApple Backdoor Interface
+ *
+ * Copyright © 2023 Amazon.com, Inc. or its affiliates. All Rights Reserved.
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or later.
+ * See the COPYING file in the top-level directory.
+ */
+
+#include "qemu/osdep.h"
+#include "hw/vmapple/bdif.h"
+#include "qemu/log.h"
+#include "qemu/module.h"
+#include "qapi/error.h"
+#include "trace.h"
+#include "hw/block/block.h"
+#include "sysemu/block-backend.h"
+
+#define REG_DEVID_MASK      0xffff0000
+#define DEVID_ROOT          0x00000000
+#define DEVID_AUX           0x00010000
+#define DEVID_USB           0x00100000
+
+#define REG_STATUS          0x0
+#define REG_STATUS_ACTIVE     BIT(0)
+#define REG_CFG             0x4
+#define REG_CFG_ACTIVE        BIT(1)
+#define REG_UNK1            0x8
+#define REG_BUSY            0x10
+#define REG_BUSY_READY        BIT(0)
+#define REG_UNK2            0x400
+#define REG_CMD             0x408
+#define REG_NEXT_DEVICE     0x420
+#define REG_UNK3            0x434
+
+typedef struct vblk_sector {
+    uint32_t pad;
+    uint32_t pad2;
+    uint32_t sector;
+    uint32_t pad3;
+} VblkSector;
+
+typedef struct vblk_req_cmd {
+    uint64_t addr;
+    uint32_t len;
+    uint32_t flags;
+} VblkReqCmd;
+
+typedef struct vblk_req {
+    VblkReqCmd sector;
+    VblkReqCmd data;
+    VblkReqCmd retval;
+} VblkReq;
+
+#define VBLK_DATA_FLAGS_READ  0x00030001
+#define VBLK_DATA_FLAGS_WRITE 0x00010001
+
+#define VBLK_RET_SUCCESS  0
+#define VBLK_RET_FAILED   1
+
+static uint64_t bdif_read(void *opaque, hwaddr offset, unsigned size)
+{
+    uint64_t ret = -1;
+    uint64_t devid = (offset & REG_DEVID_MASK);
+
+    switch (offset & ~REG_DEVID_MASK) {
+    case REG_STATUS:
+        ret = REG_STATUS_ACTIVE;
+        break;
+    case REG_CFG:
+        ret = REG_CFG_ACTIVE;
+        break;
+    case REG_UNK1:
+        ret = 0x420;
+        break;
+    case REG_BUSY:
+        ret = REG_BUSY_READY;
+        break;
+    case REG_UNK2:
+        ret = 0x1;
+        break;
+    case REG_UNK3:
+        ret = 0x0;
+        break;
+    case REG_NEXT_DEVICE:
+        switch (devid) {
+        case DEVID_ROOT:
+            ret = 0x8000000;
+            break;
+        case DEVID_AUX:
+            ret = 0x10000;
+            break;
+        }
+        break;
+    }
+
+    trace_bdif_read(offset, size, ret);
+    return ret;
+}
+
+static void le2cpu_sector(VblkSector *sector)
+{
+    sector->sector = le32_to_cpu(sector->sector);
+}
+
+static void le2cpu_reqcmd(VblkReqCmd *cmd)
+{
+    cmd->addr = le64_to_cpu(cmd->addr);
+    cmd->len = le32_to_cpu(cmd->len);
+    cmd->flags = le32_to_cpu(cmd->flags);
+}
+
+static void le2cpu_req(VblkReq *req)
+{
+    le2cpu_reqcmd(&req->sector);
+    le2cpu_reqcmd(&req->data);
+    le2cpu_reqcmd(&req->retval);
+}
+
+static void vblk_cmd(uint64_t devid, BlockBackend *blk, uint64_t value,
+                     uint64_t static_off)
+{
+    VblkReq req;
+    VblkSector sector;
+    uint64_t off = 0;
+    char *buf = NULL;
+    uint8_t ret = VBLK_RET_FAILED;
+    int r;
+
+    cpu_physical_memory_read(value, &req, sizeof(req));
+    le2cpu_req(&req);
+
+    if (req.sector.len != sizeof(sector)) {
+        ret = VBLK_RET_FAILED;
+        goto out;
+    }
+
+    /* Read the vblk command */
+    cpu_physical_memory_read(req.sector.addr, &sector, sizeof(sector));
+    le2cpu_sector(&sector);
+
+    off = sector.sector * 512ULL + static_off;
+
+    /* Sanity check that we're not allocating bogus sizes */
+    if (req.data.len > (128 * 1024 * 1024)) {
+        goto out;
+    }
+
+    buf = g_malloc0(req.data.len);
+    switch (req.data.flags) {
+    case VBLK_DATA_FLAGS_READ:
+        r = blk_pread(blk, off, req.data.len, buf, 0);
+        trace_bdif_vblk_read(devid == DEVID_AUX ? "aux" : "root",
+                             req.data.addr, off, req.data.len, r);
+        if (r < 0) {
+            goto out;
+        }
+        cpu_physical_memory_write(req.data.addr, buf, req.data.len);
+        ret = VBLK_RET_SUCCESS;
+        break;
+    case VBLK_DATA_FLAGS_WRITE:
+        /* Not needed, iBoot only reads */
+        break;
+    default:
+        break;
+    }
+
+out:
+    g_free(buf);
+    cpu_physical_memory_write(req.retval.addr, &ret, 1);
+}
+
+static void bdif_write(void *opaque, hwaddr offset,
+                       uint64_t value, unsigned size)
+{
+    VMAppleBdifState *s = opaque;
+    uint64_t devid = (offset & REG_DEVID_MASK);
+
+    trace_bdif_write(offset, size, value);
+
+    switch (offset & ~REG_DEVID_MASK) {
+    case REG_CMD:
+        switch (devid) {
+        case DEVID_ROOT:
+            vblk_cmd(devid, s->root, value, 0x0);
+            break;
+        case DEVID_AUX:
+            vblk_cmd(devid, s->aux, value, 0x0);
+            break;
+        }
+        break;
+    }
+}
+
+static const MemoryRegionOps bdif_ops = {
+    .read = bdif_read,
+    .write = bdif_write,
+    .endianness = DEVICE_NATIVE_ENDIAN,
+    .valid = {
+        .min_access_size = 1,
+        .max_access_size = 8,
+    },
+    .impl = {
+        .min_access_size = 1,
+        .max_access_size = 8,
+    },
+};
+
+static void bdif_init(Object *obj)
+{
+    VMAppleBdifState *s = VMAPPLE_BDIF(obj);
+
+    memory_region_init_io(&s->mmio, obj, &bdif_ops, obj,
+                         "VMApple Backdoor Interface", VMAPPLE_BDIF_SIZE);
+    sysbus_init_mmio(SYS_BUS_DEVICE(obj), &s->mmio);
+}
+
+static Property bdif_properties[] = {
+    DEFINE_PROP_DRIVE("aux", VMAppleBdifState, aux),
+    DEFINE_PROP_DRIVE("root", VMAppleBdifState, root),
+    DEFINE_PROP_END_OF_LIST(),
+};
+
+static void bdif_class_init(ObjectClass *klass, void *data)
+{
+    DeviceClass *dc = DEVICE_CLASS(klass);
+
+    dc->desc = "VMApple Backdoor Interface";
+    device_class_set_props(dc, bdif_properties);
+}
+
+static const TypeInfo bdif_info = {
+    .name          = TYPE_VMAPPLE_BDIF,
+    .parent        = TYPE_SYS_BUS_DEVICE,
+    .instance_size = sizeof(VMAppleBdifState),
+    .instance_init = bdif_init,
+    .class_init    = bdif_class_init,
+};
+
+static void bdif_register_types(void)
+{
+    type_register_static(&bdif_info);
+}
+
+type_init(bdif_register_types)
diff --git a/hw/vmapple/meson.build b/hw/vmapple/meson.build
index bcd4dcb28d2..d4624713deb 100644
--- a/hw/vmapple/meson.build
+++ b/hw/vmapple/meson.build
@@ -1 +1,2 @@
 system_ss.add(when: 'CONFIG_VMAPPLE_AES',  if_true: files('aes.c'))
+system_ss.add(when: 'CONFIG_VMAPPLE_BDIF', if_true: files('bdif.c'))
diff --git a/hw/vmapple/trace-events b/hw/vmapple/trace-events
index 1c9a3326eb4..fc8e9cc5897 100644
--- a/hw/vmapple/trace-events
+++ b/hw/vmapple/trace-events
@@ -19,3 +19,8 @@ aes_2_write_unknown(uint64_t offset) "offset=0x%"PRIx64
 aes_2_write(uint64_t offset, uint64_t val) "offset=0x%"PRIx64" val=0x%"PRIx64
 aes_dump_data(const char *desc, const char *hex) "%s%s"
 
+# bdif.c
+bdif_read(uint64_t offset, uint32_t size, uint64_t value) "offset=0x%"PRIx64" size=0x%x value=0x%"PRIx64
+bdif_write(uint64_t offset, uint32_t size, uint64_t value) "offset=0x%"PRIx64" size=0x%x value=0x%"PRIx64
+bdif_vblk_read(const char *dev, uint64_t addr, uint64_t offset, uint32_t len, int r) "dev=%s addr=0x%"PRIx64" off=0x%"PRIx64" size=0x%x r=%d"
+
diff --git a/include/hw/vmapple/bdif.h b/include/hw/vmapple/bdif.h
new file mode 100644
index 00000000000..65ee43457b9
--- /dev/null
+++ b/include/hw/vmapple/bdif.h
@@ -0,0 +1,31 @@
+/*
+ * VMApple Backdoor Interface
+ *
+ * Copyright © 2023 Amazon.com, Inc. or its affiliates. All Rights Reserved.
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or later.
+ * See the COPYING file in the top-level directory.
+ */
+
+#ifndef HW_VMAPPLE_BDIF_H
+#define HW_VMAPPLE_BDIF_H
+
+#include "hw/sysbus.h"
+#include "qom/object.h"
+
+#define TYPE_VMAPPLE_BDIF "vmapple-bdif"
+OBJECT_DECLARE_SIMPLE_TYPE(VMAppleBdifState, VMAPPLE_BDIF)
+
+struct VMAppleBdifState {
+    /* <private> */
+    SysBusDevice parent_obj;
+
+    /* <public> */
+    BlockBackend *aux;
+    BlockBackend *root;
+    MemoryRegion mmio;
+};
+
+#define VMAPPLE_BDIF_SIZE 0x00200000
+
+#endif /* HW_VMAPPLE_BDIF_H */
-- 
2.39.3 (Apple Git-145)

[PATCH v3 12/14] hw/vmapple/cfg: Introduce vmapple cfg region

[Phil Dennis-Jordan]

From: Alexander Graf <graf@amazon.com>

Instead of device tree or other more standardized means, VMApple passes
platform configuration to the first stage boot loader in a binary encoded
format that resides at a dedicated RAM region in physical address space.

This patch models this configuration space as a qdev device which we can
then map at the fixed location in the address space. That way, we can
influence and annotate all configuration fields easily.

Signed-off-by: Alexander Graf <graf@amazon.com>
Signed-off-by: Phil Dennis-Jordan <phil@philjordan.eu>

---
v3:

 * Replaced legacy device reset method with Resettable method

 hw/vmapple/Kconfig       |   3 ++
 hw/vmapple/cfg.c         | 106 +++++++++++++++++++++++++++++++++++++++
 hw/vmapple/meson.build   |   1 +
 include/hw/vmapple/cfg.h |  68 +++++++++++++++++++++++++
 4 files changed, 178 insertions(+)
 create mode 100644 hw/vmapple/cfg.c
 create mode 100644 include/hw/vmapple/cfg.h

diff --git a/hw/vmapple/Kconfig b/hw/vmapple/Kconfig
index 68f88876eb9..8bbeb9a9237 100644
--- a/hw/vmapple/Kconfig
+++ b/hw/vmapple/Kconfig
@@ -4,3 +4,6 @@ config VMAPPLE_AES
 config VMAPPLE_BDIF
     bool
 
+config VMAPPLE_CFG
+    bool
+
diff --git a/hw/vmapple/cfg.c b/hw/vmapple/cfg.c
new file mode 100644
index 00000000000..a5e5c62f59f
--- /dev/null
+++ b/hw/vmapple/cfg.c
@@ -0,0 +1,106 @@
+/*
+ * VMApple Configuration Region
+ *
+ * Copyright © 2023 Amazon.com, Inc. or its affiliates. All Rights Reserved.
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or later.
+ * See the COPYING file in the top-level directory.
+ */
+
+#include "qemu/osdep.h"
+#include "hw/vmapple/cfg.h"
+#include "qemu/log.h"
+#include "qemu/module.h"
+#include "qapi/error.h"
+
+static void vmapple_cfg_reset(Object *obj, ResetType type)
+{
+    VMAppleCfgState *s = VMAPPLE_CFG(obj);
+    VMAppleCfg *cfg;
+
+    cfg = memory_region_get_ram_ptr(&s->mem);
+    memset((void *)cfg, 0, VMAPPLE_CFG_SIZE);
+    *cfg = s->cfg;
+}
+
+static void vmapple_cfg_realize(DeviceState *dev, Error **errp)
+{
+    VMAppleCfgState *s = VMAPPLE_CFG(dev);
+    uint32_t i;
+
+    strncpy(s->cfg.serial, s->serial, sizeof(s->cfg.serial));
+    strncpy(s->cfg.model, s->model, sizeof(s->cfg.model));
+    strncpy(s->cfg.soc_name, s->soc_name, sizeof(s->cfg.soc_name));
+    strncpy(s->cfg.unk8, "D/A", sizeof(s->cfg.soc_name));
+    s->cfg.ecid = cpu_to_be64(s->cfg.ecid);
+    s->cfg.version = 2;
+    s->cfg.unk1 = 1;
+    s->cfg.unk2 = 1;
+    s->cfg.unk3 = 0x20;
+    s->cfg.unk4 = 0;
+    s->cfg.unk5 = 1;
+    s->cfg.unk6 = 1;
+    s->cfg.unk7 = 0;
+    s->cfg.unk10 = 1;
+
+    g_assert(s->cfg.nr_cpus < ARRAY_SIZE(s->cfg.cpu_ids));
+    for (i = 0; i < s->cfg.nr_cpus; i++) {
+        s->cfg.cpu_ids[i] = i;
+    }
+}
+
+static void vmapple_cfg_init(Object *obj)
+{
+    VMAppleCfgState *s = VMAPPLE_CFG(obj);
+
+    memory_region_init_ram(&s->mem, obj, "VMApple Config", VMAPPLE_CFG_SIZE,
+                           &error_fatal);
+    sysbus_init_mmio(SYS_BUS_DEVICE(obj), &s->mem);
+
+    s->serial = (char *)"1234";
+    s->model = (char *)"VM0001";
+    s->soc_name = (char *)"Apple M1 (Virtual)";
+}
+
+static Property vmapple_cfg_properties[] = {
+    DEFINE_PROP_UINT32("nr-cpus", VMAppleCfgState, cfg.nr_cpus, 1),
+    DEFINE_PROP_UINT64("ecid", VMAppleCfgState, cfg.ecid, 0),
+    DEFINE_PROP_UINT64("ram-size", VMAppleCfgState, cfg.ram_size, 0),
+    DEFINE_PROP_UINT32("run_installer1", VMAppleCfgState, cfg.run_installer1, 0),
+    DEFINE_PROP_UINT32("run_installer2", VMAppleCfgState, cfg.run_installer2, 0),
+    DEFINE_PROP_UINT32("rnd", VMAppleCfgState, cfg.rnd, 0),
+    DEFINE_PROP_MACADDR("mac-en0", VMAppleCfgState, cfg.mac_en0),
+    DEFINE_PROP_MACADDR("mac-en1", VMAppleCfgState, cfg.mac_en1),
+    DEFINE_PROP_MACADDR("mac-wifi0", VMAppleCfgState, cfg.mac_wifi0),
+    DEFINE_PROP_MACADDR("mac-bt0", VMAppleCfgState, cfg.mac_bt0),
+    DEFINE_PROP_STRING("serial", VMAppleCfgState, serial),
+    DEFINE_PROP_STRING("model", VMAppleCfgState, model),
+    DEFINE_PROP_STRING("soc_name", VMAppleCfgState, soc_name),
+    DEFINE_PROP_END_OF_LIST(),
+};
+
+static void vmapple_cfg_class_init(ObjectClass *klass, void *data)
+{
+    DeviceClass *dc = DEVICE_CLASS(klass);
+    ResettableClass *rc = RESETTABLE_CLASS(klass);
+
+    dc->realize = vmapple_cfg_realize;
+    dc->desc = "VMApple Configuration Region";
+    device_class_set_props(dc, vmapple_cfg_properties);
+    rc->phases.hold = vmapple_cfg_reset;
+}
+
+static const TypeInfo vmapple_cfg_info = {
+    .name          = TYPE_VMAPPLE_CFG,
+    .parent        = TYPE_SYS_BUS_DEVICE,
+    .instance_size = sizeof(VMAppleCfgState),
+    .instance_init = vmapple_cfg_init,
+    .class_init    = vmapple_cfg_class_init,
+};
+
+static void vmapple_cfg_register_types(void)
+{
+    type_register_static(&vmapple_cfg_info);
+}
+
+type_init(vmapple_cfg_register_types)
diff --git a/hw/vmapple/meson.build b/hw/vmapple/meson.build
index d4624713deb..64b78693a31 100644
--- a/hw/vmapple/meson.build
+++ b/hw/vmapple/meson.build
@@ -1,2 +1,3 @@
 system_ss.add(when: 'CONFIG_VMAPPLE_AES',  if_true: files('aes.c'))
 system_ss.add(when: 'CONFIG_VMAPPLE_BDIF', if_true: files('bdif.c'))
+system_ss.add(when: 'CONFIG_VMAPPLE_CFG',  if_true: files('cfg.c'))
diff --git a/include/hw/vmapple/cfg.h b/include/hw/vmapple/cfg.h
new file mode 100644
index 00000000000..3337064e447
--- /dev/null
+++ b/include/hw/vmapple/cfg.h
@@ -0,0 +1,68 @@
+/*
+ * VMApple Configuration Region
+ *
+ * Copyright © 2023 Amazon.com, Inc. or its affiliates. All Rights Reserved.
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or later.
+ * See the COPYING file in the top-level directory.
+ */
+
+#ifndef HW_VMAPPLE_CFG_H
+#define HW_VMAPPLE_CFG_H
+
+#include "hw/sysbus.h"
+#include "qom/object.h"
+#include "net/net.h"
+
+typedef struct VMAppleCfg {
+    uint32_t version;         /* 0x000 */
+    uint32_t nr_cpus;         /* 0x004 */
+    uint32_t unk1;            /* 0x008 */
+    uint32_t unk2;            /* 0x00c */
+    uint32_t unk3;            /* 0x010 */
+    uint32_t unk4;            /* 0x014 */
+    uint64_t ecid;            /* 0x018 */
+    uint64_t ram_size;        /* 0x020 */
+    uint32_t run_installer1;  /* 0x028 */
+    uint32_t unk5;            /* 0x02c */
+    uint32_t unk6;            /* 0x030 */
+    uint32_t run_installer2;  /* 0x034 */
+    uint32_t rnd;             /* 0x038 */
+    uint32_t unk7;            /* 0x03c */
+    MACAddr mac_en0;          /* 0x040 */
+    uint8_t pad1[2];
+    MACAddr mac_en1;          /* 0x048 */
+    uint8_t pad2[2];
+    MACAddr mac_wifi0;        /* 0x050 */
+    uint8_t pad3[2];
+    MACAddr mac_bt0;          /* 0x058 */
+    uint8_t pad4[2];
+    uint8_t reserved[0xa0];   /* 0x060 */
+    uint32_t cpu_ids[0x80];   /* 0x100 */
+    uint8_t scratch[0x200];   /* 0x180 */
+    char serial[32];          /* 0x380 */
+    char unk8[32];            /* 0x3a0 */
+    char model[32];           /* 0x3c0 */
+    uint8_t unk9[32];         /* 0x3e0 */
+    uint32_t unk10;           /* 0x400 */
+    char soc_name[32];        /* 0x404 */
+} VMAppleCfg;
+
+#define TYPE_VMAPPLE_CFG "vmapple-cfg"
+OBJECT_DECLARE_SIMPLE_TYPE(VMAppleCfgState, VMAPPLE_CFG)
+
+struct VMAppleCfgState {
+    /* <private> */
+    SysBusDevice parent_obj;
+    VMAppleCfg cfg;
+
+    /* <public> */
+    MemoryRegion mem;
+    char *serial;
+    char *model;
+    char *soc_name;
+};
+
+#define VMAPPLE_CFG_SIZE 0x00010000
+
+#endif /* HW_VMAPPLE_CFG_H */
-- 
2.39.3 (Apple Git-145)

[PATCH v3 13/14] hw/vmapple/virtio-blk: Add support for apple virtio-blk

[Phil Dennis-Jordan]

From: Alexander Graf <graf@amazon.com>

Apple has its own virtio-blk PCI device ID where it deviates from the
official virtio-pci spec slightly: It puts a new "apple type"
field at a static offset in config space and introduces a new barrier
command.

This patch first creates a mechanism for virtio-blk downstream classes to
handle unknown commands. It then creates such a downstream class and a new
vmapple-virtio-blk-pci class which support the additional apple type config
identifier as well as the barrier command.

It then exposes 2 subclasses from that that we can use to expose root and
aux virtio-blk devices: "vmapple-virtio-root" and "vmapple-virtio-aux".

Signed-off-by: Alexander Graf <graf@amazon.com>
Signed-off-by: Phil Dennis-Jordan <phil@philjordan.eu>
---
 hw/block/virtio-blk.c           |  19 ++-
 hw/vmapple/Kconfig              |   3 +
 hw/vmapple/meson.build          |   1 +
 hw/vmapple/virtio-blk.c         | 212 ++++++++++++++++++++++++++++++++
 include/hw/pci/pci_ids.h        |   1 +
 include/hw/virtio/virtio-blk.h  |  12 +-
 include/hw/vmapple/virtio-blk.h |  39 ++++++
 7 files changed, 282 insertions(+), 5 deletions(-)
 create mode 100644 hw/vmapple/virtio-blk.c
 create mode 100644 include/hw/vmapple/virtio-blk.h

diff --git a/hw/block/virtio-blk.c b/hw/block/virtio-blk.c
index 115795392c4..cecc4cef9e4 100644
--- a/hw/block/virtio-blk.c
+++ b/hw/block/virtio-blk.c
@@ -50,12 +50,12 @@ static void virtio_blk_init_request(VirtIOBlock *s, VirtQueue *vq,
     req->mr_next = NULL;
 }
 
-static void virtio_blk_free_request(VirtIOBlockReq *req)
+void virtio_blk_free_request(VirtIOBlockReq *req)
 {
     g_free(req);
 }
 
-static void virtio_blk_req_complete(VirtIOBlockReq *req, unsigned char status)
+void virtio_blk_req_complete(VirtIOBlockReq *req, unsigned char status)
 {
     VirtIOBlock *s = req->dev;
     VirtIODevice *vdev = VIRTIO_DEVICE(s);
@@ -966,8 +966,18 @@ static int virtio_blk_handle_request(VirtIOBlockReq *req, MultiReqBuffer *mrb)
         break;
     }
     default:
-        virtio_blk_req_complete(req, VIRTIO_BLK_S_UNSUPP);
-        virtio_blk_free_request(req);
+    {
+        /*
+         * Give subclasses a chance to handle unknown requests. This way the
+         * class lookup is not in the hot path.
+         */
+        VirtIOBlkClass *vbk = VIRTIO_BLK_GET_CLASS(s);
+        if (!vbk->handle_unknown_request ||
+            !vbk->handle_unknown_request(req, mrb, type)) {
+            virtio_blk_req_complete(req, VIRTIO_BLK_S_UNSUPP);
+            virtio_blk_free_request(req);
+        }
+    }
     }
     return 0;
 }
@@ -2044,6 +2054,7 @@ static const TypeInfo virtio_blk_info = {
     .instance_size = sizeof(VirtIOBlock),
     .instance_init = virtio_blk_instance_init,
     .class_init = virtio_blk_class_init,
+    .class_size = sizeof(VirtIOBlkClass),
 };
 
 static void virtio_register_types(void)
diff --git a/hw/vmapple/Kconfig b/hw/vmapple/Kconfig
index 8bbeb9a9237..bcd1be63e3c 100644
--- a/hw/vmapple/Kconfig
+++ b/hw/vmapple/Kconfig
@@ -7,3 +7,6 @@ config VMAPPLE_BDIF
 config VMAPPLE_CFG
     bool
 
+config VMAPPLE_VIRTIO_BLK
+    bool
+
diff --git a/hw/vmapple/meson.build b/hw/vmapple/meson.build
index 64b78693a31..bf17cf906c9 100644
--- a/hw/vmapple/meson.build
+++ b/hw/vmapple/meson.build
@@ -1,3 +1,4 @@
 system_ss.add(when: 'CONFIG_VMAPPLE_AES',  if_true: files('aes.c'))
 system_ss.add(when: 'CONFIG_VMAPPLE_BDIF', if_true: files('bdif.c'))
 system_ss.add(when: 'CONFIG_VMAPPLE_CFG',  if_true: files('cfg.c'))
+system_ss.add(when: 'CONFIG_VMAPPLE_VIRTIO_BLK',  if_true: files('virtio-blk.c'))
diff --git a/hw/vmapple/virtio-blk.c b/hw/vmapple/virtio-blk.c
new file mode 100644
index 00000000000..720eaa61a86
--- /dev/null
+++ b/hw/vmapple/virtio-blk.c
@@ -0,0 +1,212 @@
+/*
+ * VMApple specific VirtIO Block implementation
+ *
+ * Copyright © 2023 Amazon.com, Inc. or its affiliates. All Rights Reserved.
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or later.
+ * See the COPYING file in the top-level directory.
+ *
+ * VMApple uses almost standard VirtIO Block, but with a few key differences:
+ *
+ *  - Different PCI device/vendor ID
+ *  - An additional "type" identifier to differentiate AUX and Root volumes
+ *  - An additional BARRIER command
+ */
+
+#include "qemu/osdep.h"
+#include "hw/vmapple/virtio-blk.h"
+#include "qemu/log.h"
+#include "qemu/module.h"
+#include "qapi/error.h"
+
+#define VIRTIO_BLK_T_APPLE_BARRIER     0x10000
+
+#define VIRTIO_APPLE_TYPE_ROOT 1
+#define VIRTIO_APPLE_TYPE_AUX  2
+
+static bool vmapple_virtio_blk_handle_unknown_request(VirtIOBlockReq *req,
+                                                      MultiReqBuffer *mrb,
+                                                      uint32_t type)
+{
+    switch (type) {
+    case VIRTIO_BLK_T_APPLE_BARRIER:
+        /* We ignore barriers for now. YOLO. */
+        virtio_blk_req_complete(req, VIRTIO_BLK_S_OK);
+        virtio_blk_free_request(req);
+        return true;
+    default:
+        return false;
+    }
+}
+
+/*
+ * VMApple virtio-blk uses the same config format as normal virtio, with one
+ * exception: It adds an "apple type" specififer at the same location that
+ * the spec reserves for max_secure_erase_sectors. Let's hook into the
+ * get_config code path here, run it as usual and then patch in the apple type.
+ */
+static void vmapple_virtio_blk_get_config(VirtIODevice *vdev, uint8_t *config)
+{
+    VMAppleVirtIOBlk *dev = VMAPPLE_VIRTIO_BLK(vdev);
+    VMAppleVirtIOBlkClass *vvbk = VMAPPLE_VIRTIO_BLK_GET_CLASS(dev);
+    struct virtio_blk_config *blkcfg = (struct virtio_blk_config *)config;
+
+    vvbk->get_config(vdev, config);
+
+    g_assert(dev->parent_obj.config_size >= endof(struct virtio_blk_config, zoned));
+
+    /* Apple abuses the field for max_secure_erase_sectors as type id */
+    blkcfg->max_secure_erase_sectors = dev->apple_type;
+}
+
+static Property vmapple_virtio_blk_properties[] = {
+    DEFINE_PROP_UINT32("apple-type", VMAppleVirtIOBlk, apple_type, 0),
+    DEFINE_PROP_END_OF_LIST(),
+};
+
+static void vmapple_virtio_blk_class_init(ObjectClass *klass, void *data)
+{
+    DeviceClass *dc = DEVICE_CLASS(klass);
+    VirtIOBlkClass *vbk = VIRTIO_BLK_CLASS(klass);
+    VirtioDeviceClass *vdc = VIRTIO_DEVICE_CLASS(klass);
+    VMAppleVirtIOBlkClass *vvbk = VMAPPLE_VIRTIO_BLK_CLASS(klass);
+
+    vbk->handle_unknown_request = vmapple_virtio_blk_handle_unknown_request;
+    vvbk->get_config = vdc->get_config;
+    vdc->get_config = vmapple_virtio_blk_get_config;
+    device_class_set_props(dc, vmapple_virtio_blk_properties);
+}
+
+static const TypeInfo vmapple_virtio_blk_info = {
+    .name          = TYPE_VMAPPLE_VIRTIO_BLK,
+    .parent        = TYPE_VIRTIO_BLK,
+    .instance_size = sizeof(VMAppleVirtIOBlk),
+    .class_init    = vmapple_virtio_blk_class_init,
+};
+
+/* PCI Devices */
+
+typedef struct VMAppleVirtIOBlkPCI {
+    VirtIOPCIProxy parent_obj;
+    VMAppleVirtIOBlk vdev;
+    uint32_t apple_type;
+} VMAppleVirtIOBlkPCI;
+
+/*
+ * vmapple-virtio-blk-pci: This extends VirtioPCIProxy.
+ */
+#define TYPE_VMAPPLE_VIRTIO_BLK_PCI "vmapple-virtio-blk-pci-base"
+DECLARE_INSTANCE_CHECKER(VMAppleVirtIOBlkPCI, VMAPPLE_VIRTIO_BLK_PCI,
+                         TYPE_VMAPPLE_VIRTIO_BLK_PCI)
+
+static Property vmapple_virtio_blk_pci_properties[] = {
+    DEFINE_PROP_UINT32("class", VirtIOPCIProxy, class_code, 0),
+    DEFINE_PROP_BIT("ioeventfd", VirtIOPCIProxy, flags,
+                    VIRTIO_PCI_FLAG_USE_IOEVENTFD_BIT, true),
+    DEFINE_PROP_UINT32("vectors", VirtIOPCIProxy, nvectors,
+                       DEV_NVECTORS_UNSPECIFIED),
+    DEFINE_PROP_END_OF_LIST(),
+};
+
+static void vmapple_virtio_blk_pci_realize(VirtIOPCIProxy *vpci_dev, Error **errp)
+{
+    VMAppleVirtIOBlkPCI *dev = VMAPPLE_VIRTIO_BLK_PCI(vpci_dev);
+    DeviceState *vdev = DEVICE(&dev->vdev);
+    VirtIOBlkConf *conf = &dev->vdev.parent_obj.conf;
+
+    if (conf->num_queues == VIRTIO_BLK_AUTO_NUM_QUEUES) {
+        conf->num_queues = virtio_pci_optimal_num_queues(0);
+    }
+
+    if (vpci_dev->nvectors == DEV_NVECTORS_UNSPECIFIED) {
+        vpci_dev->nvectors = conf->num_queues + 1;
+    }
+
+    /*
+     * We don't support zones, but we need the additional config space size.
+     * Let's just expose the feature so the rest of the virtio-blk logic
+     * allocates enough space for us. The guest will ignore zones anyway.
+     */
+    virtio_add_feature(&dev->vdev.parent_obj.host_features, VIRTIO_BLK_F_ZONED);
+    /* Propagate the apple type down to the virtio-blk device */
+    qdev_prop_set_uint32(DEVICE(&dev->vdev), "apple-type", dev->apple_type);
+    /* and spawn the virtio-blk device */
+    qdev_realize(vdev, BUS(&vpci_dev->bus), errp);
+
+    /*
+     * The virtio-pci machinery adjusts its vendor/device ID based on whether
+     * we support modern or legacy virtio. Let's patch it back to the Apple
+     * identifiers here.
+     */
+    pci_config_set_vendor_id(vpci_dev->pci_dev.config, PCI_VENDOR_ID_APPLE);
+    pci_config_set_device_id(vpci_dev->pci_dev.config, PCI_DEVICE_ID_APPLE_VIRTIO_BLK);
+}
+
+static void vmapple_virtio_blk_pci_class_init(ObjectClass *klass, void *data)
+{
+    DeviceClass *dc = DEVICE_CLASS(klass);
+    VirtioPCIClass *k = VIRTIO_PCI_CLASS(klass);
+    PCIDeviceClass *pcidev_k = PCI_DEVICE_CLASS(klass);
+
+    set_bit(DEVICE_CATEGORY_STORAGE, dc->categories);
+    device_class_set_props(dc, vmapple_virtio_blk_pci_properties);
+    k->realize = vmapple_virtio_blk_pci_realize;
+    pcidev_k->vendor_id = PCI_VENDOR_ID_APPLE;
+    pcidev_k->device_id = PCI_DEVICE_ID_APPLE_VIRTIO_BLK;
+    pcidev_k->revision = VIRTIO_PCI_ABI_VERSION;
+    pcidev_k->class_id = PCI_CLASS_STORAGE_SCSI;
+}
+
+static void vmapple_virtio_blk_pci_instance_init(Object *obj)
+{
+    VMAppleVirtIOBlkPCI *dev = VMAPPLE_VIRTIO_BLK_PCI(obj);
+
+    virtio_instance_init_common(obj, &dev->vdev, sizeof(dev->vdev),
+                                TYPE_VMAPPLE_VIRTIO_BLK);
+}
+
+static const VirtioPCIDeviceTypeInfo vmapple_virtio_blk_pci_info = {
+    .base_name     = TYPE_VMAPPLE_VIRTIO_BLK_PCI,
+    .generic_name  = "vmapple-virtio-blk-pci",
+    .instance_size = sizeof(VMAppleVirtIOBlkPCI),
+    .instance_init = vmapple_virtio_blk_pci_instance_init,
+    .class_init    = vmapple_virtio_blk_pci_class_init,
+};
+
+static void vmapple_virtio_root_instance_init(Object *obj)
+{
+    VMAppleVirtIOBlkPCI *dev = VMAPPLE_VIRTIO_BLK_PCI(obj);
+
+    dev->apple_type = VIRTIO_APPLE_TYPE_ROOT;
+}
+
+static const TypeInfo vmapple_virtio_root_info = {
+    .name          = TYPE_VMAPPLE_VIRTIO_ROOT,
+    .parent        = "vmapple-virtio-blk-pci",
+    .instance_size = sizeof(VMAppleVirtIOBlkPCI),
+    .instance_init = vmapple_virtio_root_instance_init,
+};
+
+static void vmapple_virtio_aux_instance_init(Object *obj)
+{
+    VMAppleVirtIOBlkPCI *dev = VMAPPLE_VIRTIO_BLK_PCI(obj);
+
+    dev->apple_type = VIRTIO_APPLE_TYPE_AUX;
+}
+
+static const TypeInfo vmapple_virtio_aux_info = {
+    .name          = TYPE_VMAPPLE_VIRTIO_AUX,
+    .parent        = "vmapple-virtio-blk-pci",
+    .instance_size = sizeof(VMAppleVirtIOBlkPCI),
+    .instance_init = vmapple_virtio_aux_instance_init,
+};
+
+static void vmapple_virtio_blk_register_types(void)
+{
+    type_register_static(&vmapple_virtio_blk_info);
+    virtio_pci_types_register(&vmapple_virtio_blk_pci_info);
+    type_register_static(&vmapple_virtio_root_info);
+    type_register_static(&vmapple_virtio_aux_info);
+}
+
+type_init(vmapple_virtio_blk_register_types)
diff --git a/include/hw/pci/pci_ids.h b/include/hw/pci/pci_ids.h
index f1a53fea8d6..33e2898be95 100644
--- a/include/hw/pci/pci_ids.h
+++ b/include/hw/pci/pci_ids.h
@@ -191,6 +191,7 @@
 #define PCI_DEVICE_ID_APPLE_UNI_N_AGP    0x0020
 #define PCI_DEVICE_ID_APPLE_U3_AGP       0x004b
 #define PCI_DEVICE_ID_APPLE_UNI_N_GMAC   0x0021
+#define PCI_DEVICE_ID_APPLE_VIRTIO_BLK   0x1a00
 
 #define PCI_VENDOR_ID_SUN                0x108e
 #define PCI_DEVICE_ID_SUN_EBUS           0x1000
diff --git a/include/hw/virtio/virtio-blk.h b/include/hw/virtio/virtio-blk.h
index 5c14110c4b1..28d5046ea6c 100644
--- a/include/hw/virtio/virtio-blk.h
+++ b/include/hw/virtio/virtio-blk.h
@@ -24,7 +24,7 @@
 #include "qapi/qapi-types-virtio.h"
 
 #define TYPE_VIRTIO_BLK "virtio-blk-device"
-OBJECT_DECLARE_SIMPLE_TYPE(VirtIOBlock, VIRTIO_BLK)
+OBJECT_DECLARE_TYPE(VirtIOBlock, VirtIOBlkClass, VIRTIO_BLK)
 
 /* This is the last element of the write scatter-gather list */
 struct virtio_blk_inhdr
@@ -100,6 +100,16 @@ typedef struct MultiReqBuffer {
     bool is_write;
 } MultiReqBuffer;
 
+typedef struct VirtIOBlkClass {
+    /*< private >*/
+    VirtioDeviceClass parent;
+    /*< public >*/
+    bool (*handle_unknown_request)(VirtIOBlockReq *req, MultiReqBuffer *mrb,
+                                   uint32_t type);
+} VirtIOBlkClass;
+
 void virtio_blk_handle_vq(VirtIOBlock *s, VirtQueue *vq);
+void virtio_blk_free_request(VirtIOBlockReq *req);
+void virtio_blk_req_complete(VirtIOBlockReq *req, unsigned char status);
 
 #endif
diff --git a/include/hw/vmapple/virtio-blk.h b/include/hw/vmapple/virtio-blk.h
new file mode 100644
index 00000000000..b23106a3dfb
--- /dev/null
+++ b/include/hw/vmapple/virtio-blk.h
@@ -0,0 +1,39 @@
+/*
+ * VMApple specific VirtIO Block implementation
+ *
+ * Copyright © 2023 Amazon.com, Inc. or its affiliates. All Rights Reserved.
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or later.
+ * See the COPYING file in the top-level directory.
+ */
+
+#ifndef HW_VMAPPLE_CFG_H
+#define HW_VMAPPLE_CFG_H
+
+#include "hw/sysbus.h"
+#include "qom/object.h"
+#include "hw/virtio/virtio-pci.h"
+#include "hw/virtio/virtio-blk.h"
+
+#define TYPE_VMAPPLE_VIRTIO_BLK "vmapple-virtio-blk"
+#define TYPE_VMAPPLE_VIRTIO_ROOT "vmapple-virtio-root"
+#define TYPE_VMAPPLE_VIRTIO_AUX "vmapple-virtio-aux"
+
+OBJECT_DECLARE_TYPE(VMAppleVirtIOBlk, VMAppleVirtIOBlkClass, VMAPPLE_VIRTIO_BLK)
+
+typedef struct VMAppleVirtIOBlkClass {
+    /*< private >*/
+    VirtIOBlkClass parent;
+    /*< public >*/
+    void (*get_config)(VirtIODevice *vdev, uint8_t *config);
+} VMAppleVirtIOBlkClass;
+
+typedef struct VMAppleVirtIOBlk {
+    /* <private> */
+    VirtIOBlock parent_obj;
+
+    /* <public> */
+    uint32_t apple_type;
+} VMAppleVirtIOBlk;
+
+#endif /* HW_VMAPPLE_CFG_H */
-- 
2.39.3 (Apple Git-145)

[PATCH v3 14/14] hw/vmapple/vmapple: Add vmapple machine type

[Phil Dennis-Jordan]

From: Alexander Graf <graf@amazon.com>

Apple defines a new "vmapple" machine type as part of its proprietary
macOS Virtualization.Framework vmm. This machine type is similar to the
virt one, but with subtle differences in base devices, a few special
vmapple device additions and a vastly different boot chain.

This patch reimplements this machine type in QEMU. To use it, you
have to have a readily installed version of macOS for VMApple,
run on macOS with -accel hvf, pass the Virtualization.Framework
boot rom (AVPBooter) in via -bios, pass the aux and root volume as pflash
and pass aux and root volume as virtio drives. In addition, you also
need to find the machine UUID and pass that as -M vmapple,uuid= parameter:

$ qemu-system-aarch64 -accel hvf -M vmapple,uuid=0x1234 -m 4G \
    -bios /System/Library/Frameworks/Virtualization.framework/Versions/A/Resources/AVPBooter.vmapple2.bin
    -drive file=aux,if=pflash,format=raw \
    -drive file=root,if=pflash,format=raw \
    -drive file=aux,if=none,id=aux,format=raw \
    -device vmapple-virtio-aux,drive=aux \
    -drive file=root,if=none,id=root,format=raw \
    -device vmapple-virtio-root,drive=root

With all these in place, you should be able to see macOS booting
successfully.

Known issues:
 - Keyboard and mouse/tablet input is laggy. The reason for this is
   either that macOS's XHCI driver is broken when the device/platform
   does not support MSI/MSI-X, or there's some unfortunate interplay
   with Qemu's XHCI implementation in this scenario.
 - Currently only macOS 12 guests are supported. The boot process for
   13+ will need further investigation and adjustment.

Signed-off-by: Alexander Graf <graf@amazon.com>
Co-authored-by: Phil Dennis-Jordan <phil@philjordan.eu>
Signed-off-by: Phil Dennis-Jordan <phil@philjordan.eu>

---
v3:
 * Rebased on latest upstream, updated affinity and NIC creation
API usage
 * Included Apple-variant virtio-blk in build dependency
 * Updated API usage for setting 'redist-region-count' array-typed property on GIC.
 * Switched from virtio HID devices (for which macOS 12 does not contain drivers) to an XHCI USB controller and USB HID devices.

 MAINTAINERS                 |   1 +
 docs/system/arm/vmapple.rst |  63 ++++
 docs/system/target-arm.rst  |   1 +
 hw/vmapple/Kconfig          |  20 ++
 hw/vmapple/meson.build      |   1 +
 hw/vmapple/vmapple.c        | 661 ++++++++++++++++++++++++++++++++++++
 6 files changed, 747 insertions(+)
 create mode 100644 docs/system/arm/vmapple.rst
 create mode 100644 hw/vmapple/vmapple.c

diff --git a/MAINTAINERS b/MAINTAINERS
index 4e7f25e5299..89ef071a01a 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -2783,6 +2783,7 @@ R: Phil Dennis-Jordan <phil@philjordan.eu>
 S: Maintained
 F: hw/vmapple/*
 F: include/hw/vmapple/*
+F: docs/system/arm/vmapple.rst
 
 Subsystems
 ----------
diff --git a/docs/system/arm/vmapple.rst b/docs/system/arm/vmapple.rst
new file mode 100644
index 00000000000..acb921ffb35
--- /dev/null
+++ b/docs/system/arm/vmapple.rst
@@ -0,0 +1,63 @@
+VMApple machine emulation
+========================================================================================
+
+VMApple is the device model that the macOS built-in hypervisor called "Virtualization.framework"
+exposes to Apple Silicon macOS guests. The "vmapple" machine model in QEMU implements the same
+device model, but does not use any code from Virtualization.Framework.
+
+Prerequisites
+-------------
+
+To run the vmapple machine model, you need to
+
+ * Run on Apple Silicon
+ * Run on macOS 12.0 or above
+ * Have an already installed copy of a Virtualization.Framework macOS 12 virtual machine. I will
+   assume that you installed it using the macosvm CLI.
+
+First, we need to extract the UUID from the virtual machine that you installed. You can do this
+by running the following shell script:
+
+.. code-block:: bash
+  :caption: uuid.sh script to extract the UUID from a macosvm.json file
+
+  #!/bin/bash
+
+  MID=$(cat "$1" | python3 -c 'import json,sys;obj=json.load(sys.stdin);print(obj["machineId"]);')
+  echo "$MID" | base64 -d | plutil -extract ECID raw -
+
+Now we also need to trim the aux partition. It contains metadata that we can just discard:
+
+.. code-block:: bash
+  :caption: Command to trim the aux file
+
+  $ dd if="aux.img" of="aux.img.trimmed" bs=$(( 0x4000 )) skip=1
+
+How to run
+----------
+
+Then, we can launch QEMU with the Virtualization.Framework pre-boot environment and the readily
+installed target disk images. I recommend to port forward the VM's ssh and vnc ports to the host
+to get better interactive access into the target system:
+
+.. code-block:: bash
+  :caption: Example execution command line
+
+  $ UUID=$(uuid.sh macosvm.json)
+  $ AVPBOOTER=/System/Library/Frameworks/Virtualization.framework/Resources/AVPBooter.vmapple2.bin
+  $ AUX=aux.img.trimmed
+  $ DISK=disk.img
+  $ qemu-system-aarch64 \
+       -serial mon:stdio \
+       -m 4G \
+       -accel hvf \
+       -M vmapple,uuid=$UUID \
+       -bios $AVPBOOTER \
+        -drive file="$AUX",if=pflash,format=raw \
+        -drive file="$DISK",if=pflash,format=raw \
+       -drive file="$AUX",if=none,id=aux,format=raw \
+       -drive file="$DISK",if=none,id=root,format=raw \
+       -device vmapple-virtio-aux,drive=aux \
+       -device vmapple-virtio-root,drive=root \
+       -net user,ipv6=off,hostfwd=tcp::2222-:22,hostfwd=tcp::5901-:5900 \
+       -net nic,model=virtio-net-pci \
diff --git a/docs/system/target-arm.rst b/docs/system/target-arm.rst
index 7b992722846..f1948abb545 100644
--- a/docs/system/target-arm.rst
+++ b/docs/system/target-arm.rst
@@ -107,6 +107,7 @@ undocumented; you can get a complete list by running
    arm/stellaris
    arm/stm32
    arm/virt
+   arm/vmapple
    arm/xenpvh
    arm/xlnx-versal-virt
    arm/xlnx-zynq
diff --git a/hw/vmapple/Kconfig b/hw/vmapple/Kconfig
index bcd1be63e3c..0f83d4259fc 100644
--- a/hw/vmapple/Kconfig
+++ b/hw/vmapple/Kconfig
@@ -10,3 +10,23 @@ config VMAPPLE_CFG
 config VMAPPLE_VIRTIO_BLK
     bool
 
+config VMAPPLE
+    bool
+    depends on ARM
+    depends on HVF
+    default y if ARM
+    imply PCI_DEVICES
+    select ARM_GIC
+    select PLATFORM_BUS
+    select PCI_EXPRESS
+    select PCI_EXPRESS_GENERIC_BRIDGE
+    select PL011 # UART
+    select PL031 # RTC
+    select PL061 # GPIO
+    select GPIO_PWR
+    select PVPANIC_MMIO
+    select VMAPPLE_AES
+    select VMAPPLE_BDIF
+    select VMAPPLE_CFG
+    select MAC_PVG_VMAPPLE
+    select VMAPPLE_VIRTIO_BLK
diff --git a/hw/vmapple/meson.build b/hw/vmapple/meson.build
index bf17cf906c9..e572f7d5602 100644
--- a/hw/vmapple/meson.build
+++ b/hw/vmapple/meson.build
@@ -2,3 +2,4 @@ system_ss.add(when: 'CONFIG_VMAPPLE_AES',  if_true: files('aes.c'))
 system_ss.add(when: 'CONFIG_VMAPPLE_BDIF', if_true: files('bdif.c'))
 system_ss.add(when: 'CONFIG_VMAPPLE_CFG',  if_true: files('cfg.c'))
 system_ss.add(when: 'CONFIG_VMAPPLE_VIRTIO_BLK',  if_true: files('virtio-blk.c'))
+specific_ss.add(when: 'CONFIG_VMAPPLE',     if_true: files('vmapple.c'))
diff --git a/hw/vmapple/vmapple.c b/hw/vmapple/vmapple.c
new file mode 100644
index 00000000000..f0060a6f7ee
--- /dev/null
+++ b/hw/vmapple/vmapple.c
@@ -0,0 +1,661 @@
+/*
+ * VMApple machine emulation
+ *
+ * Copyright © 2023 Amazon.com, Inc. or its affiliates. All Rights Reserved.
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or later.
+ * See the COPYING file in the top-level directory.
+ *
+ * VMApple is the device model that the macOS built-in hypervisor called
+ * "Virtualization.framework" exposes to Apple Silicon macOS guests. The
+ * machine model in this file implements the same device model in QEMU, but
+ * does not use any code from Virtualization.Framework.
+ */
+
+#include "qemu/osdep.h"
+#include "qemu/help-texts.h"
+#include "qemu/datadir.h"
+#include "qemu/units.h"
+#include "qemu/option.h"
+#include "monitor/qdev.h"
+#include "hw/sysbus.h"
+#include "hw/arm/boot.h"
+#include "hw/arm/primecell.h"
+#include "hw/boards.h"
+#include "hw/usb.h"
+#include "net/net.h"
+#include "sysemu/sysemu.h"
+#include "sysemu/runstate.h"
+#include "sysemu/kvm.h"
+#include "sysemu/hvf.h"
+#include "hw/loader.h"
+#include "qapi/error.h"
+#include "qapi/qmp/qlist.h"
+#include "qemu/bitops.h"
+#include "qemu/error-report.h"
+#include "qemu/module.h"
+#include "hw/pci-host/gpex.h"
+#include "hw/virtio/virtio-pci.h"
+#include "hw/qdev-properties.h"
+#include "hw/intc/arm_gic.h"
+#include "hw/intc/arm_gicv3_common.h"
+#include "hw/irq.h"
+#include "hw/usb/xhci.h"
+#include "qapi/visitor.h"
+#include "qapi/qapi-visit-common.h"
+#include "standard-headers/linux/input.h"
+#include "target/arm/internals.h"
+#include "target/arm/kvm_arm.h"
+#include "hw/char/pl011.h"
+#include "qemu/guest-random.h"
+#include "sysemu/reset.h"
+#include "qemu/log.h"
+#include "hw/vmapple/cfg.h"
+#include "hw/misc/pvpanic.h"
+#include "hw/vmapple/bdif.h"
+
+struct VMAppleMachineClass {
+    MachineClass parent;
+};
+
+struct VMAppleMachineState {
+    MachineState parent;
+    
+    Notifier machine_done;
+    struct arm_boot_info bootinfo;
+    MemMapEntry *memmap;
+    const int *irqmap;
+    DeviceState *gic;
+    DeviceState *cfg;
+    Notifier powerdown_notifier;
+    PCIBus *bus;
+    MemoryRegion fw_mr;
+    uint64_t uuid;
+};
+
+#define DEFINE_VMAPPLE_MACHINE_LATEST(major, minor, latest) \
+    static void vmapple##major##_##minor##_class_init(ObjectClass *oc, \
+                                                    void *data) \
+    { \
+        MachineClass *mc = MACHINE_CLASS(oc); \
+        vmapple_machine_##major##_##minor##_options(mc); \
+        mc->desc = "QEMU " # major "." # minor " Apple Virtual Machine"; \
+        if (latest) { \
+            mc->alias = "vmapple"; \
+        } \
+    } \
+    static const TypeInfo machvmapple##major##_##minor##_info = { \
+        .name = MACHINE_TYPE_NAME("vmapple-" # major "." # minor), \
+        .parent = TYPE_VMAPPLE_MACHINE, \
+        .class_init = vmapple##major##_##minor##_class_init, \
+    }; \
+    static void machvmapple_machine_##major##_##minor##_init(void) \
+    { \
+        type_register_static(&machvmapple##major##_##minor##_info); \
+    } \
+    type_init(machvmapple_machine_##major##_##minor##_init);
+
+#define DEFINE_VMAPPLE_MACHINE_AS_LATEST(major, minor) \
+    DEFINE_VMAPPLE_MACHINE_LATEST(major, minor, true)
+#define DEFINE_VMAPPLE_MACHINE(major, minor) \
+    DEFINE_VMAPPLE_MACHINE_LATEST(major, minor, false)
+
+#define TYPE_VMAPPLE_MACHINE   MACHINE_TYPE_NAME("vmapple")
+OBJECT_DECLARE_TYPE(VMAppleMachineState, VMAppleMachineClass, VMAPPLE_MACHINE)
+
+/* Number of external interrupt lines to configure the GIC with */
+#define NUM_IRQS 256
+
+enum {
+    VMAPPLE_FIRMWARE,
+    VMAPPLE_CONFIG,
+    VMAPPLE_MEM,
+    VMAPPLE_GIC_DIST,
+    VMAPPLE_GIC_REDIST,
+    VMAPPLE_UART,
+    VMAPPLE_RTC,
+    VMAPPLE_PCIE,
+    VMAPPLE_PCIE_MMIO,
+    VMAPPLE_PCIE_ECAM,
+    VMAPPLE_GPIO,
+    VMAPPLE_PVPANIC,
+    VMAPPLE_APV_GFX,
+    VMAPPLE_APV_IOSFC,
+    VMAPPLE_AES_1,
+    VMAPPLE_AES_2,
+    VMAPPLE_BDOOR,
+    VMAPPLE_MEMMAP_LAST,
+};
+
+static MemMapEntry memmap[] = {
+    [VMAPPLE_FIRMWARE] =           { 0x00100000, 0x00100000 },
+    [VMAPPLE_CONFIG] =             { 0x00400000, 0x00010000 },
+
+    [VMAPPLE_GIC_DIST] =           { 0x10000000, 0x00010000 },
+    [VMAPPLE_GIC_REDIST] =         { 0x10010000, 0x00400000 },
+
+    [VMAPPLE_UART] =               { 0x20010000, 0x00010000 },
+    [VMAPPLE_RTC] =                { 0x20050000, 0x00001000 },
+    [VMAPPLE_GPIO] =               { 0x20060000, 0x00001000 },
+    [VMAPPLE_PVPANIC] =            { 0x20070000, 0x00000002 },
+    [VMAPPLE_BDOOR] =              { 0x30000000, 0x00200000 },
+    [VMAPPLE_APV_GFX] =            { 0x30200000, 0x00010000 },
+    [VMAPPLE_APV_IOSFC] =          { 0x30210000, 0x00010000 },
+    [VMAPPLE_AES_1] =              { 0x30220000, 0x00004000 },
+    [VMAPPLE_AES_2] =              { 0x30230000, 0x00004000 },
+    [VMAPPLE_PCIE_ECAM] =          { 0x40000000, 0x10000000 },
+    [VMAPPLE_PCIE_MMIO] =          { 0x50000000, 0x1fff0000 },
+
+    /* Actual RAM size depends on configuration */
+    [VMAPPLE_MEM] =                { 0x70000000ULL, GiB},
+};
+
+static const int irqmap[] = {
+    [VMAPPLE_UART] = 1,
+    [VMAPPLE_RTC] = 2,
+    [VMAPPLE_GPIO] = 0x5,
+    [VMAPPLE_APV_IOSFC] = 0x10,
+    [VMAPPLE_APV_GFX] = 0x11,
+    [VMAPPLE_AES_1] = 0x12,
+    [VMAPPLE_PCIE] = 0x20,
+};
+
+#define GPEX_NUM_IRQS 16
+
+static void create_bdif(VMAppleMachineState *vms, MemoryRegion *mem)
+{
+    DeviceState *bdif;
+    SysBusDevice *bdif_sb;
+    DriveInfo *di_aux = drive_get(IF_PFLASH, 0, 0);
+    DriveInfo *di_root = drive_get(IF_PFLASH, 0, 1);
+
+    if (!di_aux) {
+        error_report("No AUX device found. Please specify one as pflash drive");
+        exit(1);
+    }
+
+    if (!di_root) {
+        /* Fall back to the first IF_VIRTIO device as root device */
+        di_root = drive_get(IF_VIRTIO, 0, 0);
+    }
+
+    if (!di_root) {
+        error_report("No root device found. Please specify one as virtio drive");
+        exit(1);
+    }
+
+    /* PV backdoor device */
+    bdif = qdev_new(TYPE_VMAPPLE_BDIF);
+    bdif_sb = SYS_BUS_DEVICE(bdif);
+    sysbus_mmio_map(bdif_sb, 0, vms->memmap[VMAPPLE_BDOOR].base);
+
+    qdev_prop_set_drive(DEVICE(bdif), "aux", blk_by_legacy_dinfo(di_aux));
+    qdev_prop_set_drive(DEVICE(bdif), "root", blk_by_legacy_dinfo(di_root));
+
+    sysbus_realize_and_unref(bdif_sb, &error_fatal);
+}
+
+static void create_pvpanic(VMAppleMachineState *vms, MemoryRegion *mem)
+{
+    SysBusDevice *cfg;
+
+    vms->cfg = qdev_new(TYPE_PVPANIC_MMIO_DEVICE);
+    cfg = SYS_BUS_DEVICE(vms->cfg);
+    sysbus_mmio_map(cfg, 0, vms->memmap[VMAPPLE_PVPANIC].base);
+
+    sysbus_realize_and_unref(cfg, &error_fatal);
+}
+
+static void create_cfg(VMAppleMachineState *vms, MemoryRegion *mem)
+{
+    SysBusDevice *cfg;
+    MachineState *machine = MACHINE(vms);
+    uint32_t rnd = 1;
+
+    vms->cfg = qdev_new(TYPE_VMAPPLE_CFG);
+    cfg = SYS_BUS_DEVICE(vms->cfg);
+    sysbus_mmio_map(cfg, 0, vms->memmap[VMAPPLE_CONFIG].base);
+
+    qemu_guest_getrandom_nofail(&rnd, sizeof(rnd));
+
+    qdev_prop_set_uint32(vms->cfg, "nr-cpus", machine->smp.cpus);
+    qdev_prop_set_uint64(vms->cfg, "ecid", vms->uuid);
+    qdev_prop_set_uint64(vms->cfg, "ram-size", machine->ram_size);
+    qdev_prop_set_uint32(vms->cfg, "rnd", rnd);
+
+    sysbus_realize_and_unref(cfg, &error_fatal);
+}
+
+static void create_gfx(VMAppleMachineState *vms, MemoryRegion *mem)
+{
+    int irq_gfx = vms->irqmap[VMAPPLE_APV_GFX];
+    int irq_iosfc = vms->irqmap[VMAPPLE_APV_IOSFC];
+    SysBusDevice *aes;
+
+    aes = SYS_BUS_DEVICE(qdev_new("apple-gfx-vmapple"));
+    sysbus_mmio_map(aes, 0, vms->memmap[VMAPPLE_APV_GFX].base);
+    sysbus_mmio_map(aes, 1, vms->memmap[VMAPPLE_APV_IOSFC].base);
+    sysbus_connect_irq(aes, 0, qdev_get_gpio_in(vms->gic, irq_gfx));
+    sysbus_connect_irq(aes, 1, qdev_get_gpio_in(vms->gic, irq_iosfc));
+    sysbus_realize_and_unref(aes, &error_fatal);
+}
+
+static void create_aes(VMAppleMachineState *vms, MemoryRegion *mem)
+{
+    int irq = vms->irqmap[VMAPPLE_AES_1];
+    SysBusDevice *aes;
+
+    aes = SYS_BUS_DEVICE(qdev_new("apple-aes"));
+    sysbus_mmio_map(aes, 0, vms->memmap[VMAPPLE_AES_1].base);
+    sysbus_mmio_map(aes, 1, vms->memmap[VMAPPLE_AES_2].base);
+    sysbus_connect_irq(aes, 0, qdev_get_gpio_in(vms->gic, irq));
+    sysbus_realize_and_unref(aes, &error_fatal);
+}
+
+static inline int arm_gic_ppi_index(int cpu_nr, int ppi_index)
+{
+    return NUM_IRQS + cpu_nr * GIC_INTERNAL + ppi_index;
+}
+
+static void create_gic(VMAppleMachineState *vms, MemoryRegion *mem)
+{
+    MachineState *ms = MACHINE(vms);
+    /* We create a standalone GIC */
+    SysBusDevice *gicbusdev;
+    QList *redist_region_count;
+    int i;
+    unsigned int smp_cpus = ms->smp.cpus;
+
+    vms->gic = qdev_new(gicv3_class_name());
+    qdev_prop_set_uint32(vms->gic, "revision", 3);
+    qdev_prop_set_uint32(vms->gic, "num-cpu", smp_cpus);
+    /*
+     * Note that the num-irq property counts both internal and external
+     * interrupts; there are always 32 of the former (mandated by GIC spec).
+     */
+    qdev_prop_set_uint32(vms->gic, "num-irq", NUM_IRQS + 32);
+
+    uint32_t redist0_capacity =
+                vms->memmap[VMAPPLE_GIC_REDIST].size / GICV3_REDIST_SIZE;
+    uint32_t redist0_count = MIN(smp_cpus, redist0_capacity);
+
+    redist_region_count = qlist_new();
+    qlist_append_int(redist_region_count, redist0_count);
+    qdev_prop_set_array(vms->gic, "redist-region-count", redist_region_count);
+
+    gicbusdev = SYS_BUS_DEVICE(vms->gic);
+    sysbus_realize_and_unref(gicbusdev, &error_fatal);
+    sysbus_mmio_map(gicbusdev, 0, vms->memmap[VMAPPLE_GIC_DIST].base);
+    sysbus_mmio_map(gicbusdev, 1, vms->memmap[VMAPPLE_GIC_REDIST].base);
+
+    /*
+     * Wire the outputs from each CPU's generic timer and the GICv3
+     * maintenance interrupt signal to the appropriate GIC PPI inputs,
+     * and the GIC's IRQ/FIQ/VIRQ/VFIQ interrupt outputs to the CPU's inputs.
+     */
+    for (i = 0; i < smp_cpus; i++) {
+        DeviceState *cpudev = DEVICE(qemu_get_cpu(i));
+
+        /* Map the virt timer to PPI 27 */
+        qdev_connect_gpio_out(cpudev, GTIMER_VIRT,
+                              qdev_get_gpio_in(vms->gic,
+                                               arm_gic_ppi_index(i, 27)));
+
+        /* Map the GIC IRQ and FIQ lines to CPU */
+        sysbus_connect_irq(gicbusdev, i, qdev_get_gpio_in(cpudev, ARM_CPU_IRQ));
+        sysbus_connect_irq(gicbusdev, i + smp_cpus,
+                           qdev_get_gpio_in(cpudev, ARM_CPU_FIQ));
+    }
+}
+
+static void create_uart(const VMAppleMachineState *vms, int uart,
+                        MemoryRegion *mem, Chardev *chr)
+{
+    hwaddr base = vms->memmap[uart].base;
+    int irq = vms->irqmap[uart];
+    DeviceState *dev = qdev_new(TYPE_PL011);
+    SysBusDevice *s = SYS_BUS_DEVICE(dev);
+
+    qdev_prop_set_chr(dev, "chardev", chr);
+    sysbus_realize_and_unref(SYS_BUS_DEVICE(dev), &error_fatal);
+    memory_region_add_subregion(mem, base,
+                                sysbus_mmio_get_region(s, 0));
+    sysbus_connect_irq(s, 0, qdev_get_gpio_in(vms->gic, irq));
+}
+
+static void create_rtc(const VMAppleMachineState *vms)
+{
+    hwaddr base = vms->memmap[VMAPPLE_RTC].base;
+    int irq = vms->irqmap[VMAPPLE_RTC];
+
+    sysbus_create_simple("pl031", base, qdev_get_gpio_in(vms->gic, irq));
+}
+
+static DeviceState *gpio_key_dev;
+static void vmapple_powerdown_req(Notifier *n, void *opaque)
+{
+    /* use gpio Pin 3 for power button event */
+    qemu_set_irq(qdev_get_gpio_in(gpio_key_dev, 0), 1);
+}
+
+static void create_gpio_devices(const VMAppleMachineState *vms, int gpio,
+                                MemoryRegion *mem)
+{
+    DeviceState *pl061_dev;
+    hwaddr base = vms->memmap[gpio].base;
+    int irq = vms->irqmap[gpio];
+    SysBusDevice *s;
+
+    pl061_dev = qdev_new("pl061");
+    /* Pull lines down to 0 if not driven by the PL061 */
+    qdev_prop_set_uint32(pl061_dev, "pullups", 0);
+    qdev_prop_set_uint32(pl061_dev, "pulldowns", 0xff);
+    s = SYS_BUS_DEVICE(pl061_dev);
+    sysbus_realize_and_unref(s, &error_fatal);
+    memory_region_add_subregion(mem, base, sysbus_mmio_get_region(s, 0));
+    sysbus_connect_irq(s, 0, qdev_get_gpio_in(vms->gic, irq));
+    gpio_key_dev = sysbus_create_simple("gpio-key", -1,
+                                        qdev_get_gpio_in(pl061_dev, 3));
+}
+
+static void vmapple_firmware_init(VMAppleMachineState *vms,
+                                  MemoryRegion *sysmem)
+{
+    hwaddr size = vms->memmap[VMAPPLE_FIRMWARE].size;
+    hwaddr base = vms->memmap[VMAPPLE_FIRMWARE].base;
+    const char *bios_name;
+    int image_size;
+    char *fname;
+
+    bios_name = MACHINE(vms)->firmware;
+    if (!bios_name) {
+        error_report("No firmware specified");
+        exit(1);
+    }
+
+    fname = qemu_find_file(QEMU_FILE_TYPE_BIOS, bios_name);
+    if (!fname) {
+        error_report("Could not find ROM image '%s'", bios_name);
+        exit(1);
+    }
+
+    memory_region_init_ram(&vms->fw_mr, NULL, "firmware", size, NULL);
+    image_size = load_image_mr(fname, &vms->fw_mr);
+
+    g_free(fname);
+    if (image_size < 0) {
+        error_report("Could not load ROM image '%s'", bios_name);
+        exit(1);
+    }
+
+    memory_region_add_subregion(get_system_memory(), base, &vms->fw_mr);
+}
+
+static void create_pcie(VMAppleMachineState *vms)
+{
+    hwaddr base_mmio = vms->memmap[VMAPPLE_PCIE_MMIO].base;
+    hwaddr size_mmio = vms->memmap[VMAPPLE_PCIE_MMIO].size;
+    hwaddr base_ecam = vms->memmap[VMAPPLE_PCIE_ECAM].base;
+    hwaddr size_ecam = vms->memmap[VMAPPLE_PCIE_ECAM].size;
+    int irq = vms->irqmap[VMAPPLE_PCIE];
+    MemoryRegion *mmio_alias;
+    MemoryRegion *mmio_reg;
+    MemoryRegion *ecam_alias;
+    MemoryRegion *ecam_reg;
+    DeviceState *dev;
+    int i;
+    PCIHostState *pci;
+    DeviceState *usb_controller;
+    USBBus *usb_bus;
+
+    dev = qdev_new(TYPE_GPEX_HOST);
+    qdev_prop_set_uint32(dev, "nr-irqs", GPEX_NUM_IRQS);
+    sysbus_realize_and_unref(SYS_BUS_DEVICE(dev), &error_fatal);
+
+    /* Map only the first size_ecam bytes of ECAM space */
+    ecam_alias = g_new0(MemoryRegion, 1);
+    ecam_reg = sysbus_mmio_get_region(SYS_BUS_DEVICE(dev), 0);
+    memory_region_init_alias(ecam_alias, OBJECT(dev), "pcie-ecam",
+                             ecam_reg, 0, size_ecam);
+    memory_region_add_subregion(get_system_memory(), base_ecam, ecam_alias);
+
+    /*
+     * Map the MMIO window from [0x50000000-0x7fff0000] in PCI space into
+     * system address space at [0x50000000-0x7fff0000].
+     */
+    mmio_alias = g_new0(MemoryRegion, 1);
+    mmio_reg = sysbus_mmio_get_region(SYS_BUS_DEVICE(dev), 1);
+    memory_region_init_alias(mmio_alias, OBJECT(dev), "pcie-mmio",
+                             mmio_reg, base_mmio, size_mmio);
+    memory_region_add_subregion(get_system_memory(), base_mmio, mmio_alias);
+
+    for (i = 0; i < GPEX_NUM_IRQS; i++) {
+        sysbus_connect_irq(SYS_BUS_DEVICE(dev), i,
+                           qdev_get_gpio_in(vms->gic, irq + i));
+        gpex_set_irq_num(GPEX_HOST(dev), i, irq + i);
+    }
+
+    pci = PCI_HOST_BRIDGE(dev);
+    vms->bus = pci->bus;
+    g_assert_nonnull(vms->bus);
+
+    while ((dev = qemu_create_nic_device("virtio-net-pci", true, NULL))) {
+        qdev_realize_and_unref(dev, BUS(vms->bus), &error_fatal);
+    }
+
+    usb_controller = qdev_new(TYPE_QEMU_XHCI);
+    qdev_realize_and_unref(usb_controller, BUS(pci->bus), &error_fatal);
+
+    usb_bus = USB_BUS(object_resolve_type_unambiguous(TYPE_USB_BUS,
+                                                      &error_fatal));
+    usb_create_simple(usb_bus, "usb-kbd");
+    usb_create_simple(usb_bus, "usb-tablet");
+}
+
+static void vmapple_reset(void *opaque)
+{
+    VMAppleMachineState *vms = opaque;
+    hwaddr base = vms->memmap[VMAPPLE_FIRMWARE].base;
+
+    cpu_set_pc(first_cpu, base);
+}
+
+static void mach_vmapple_init(MachineState *machine)
+{
+    VMAppleMachineState *vms = VMAPPLE_MACHINE(machine);
+    MachineClass *mc = MACHINE_GET_CLASS(machine);
+    const CPUArchIdList *possible_cpus;
+    MemoryRegion *sysmem = get_system_memory();
+    int n;
+    unsigned int smp_cpus = machine->smp.cpus;
+    unsigned int max_cpus = machine->smp.max_cpus;
+
+    vms->memmap = memmap;
+    machine->usb = true;
+
+    possible_cpus = mc->possible_cpu_arch_ids(machine);
+    assert(possible_cpus->len == max_cpus);
+    for (n = 0; n < possible_cpus->len; n++) {
+        Object *cpu;
+        CPUState *cs;
+
+        if (n >= smp_cpus) {
+            break;
+        }
+
+        cpu = object_new(possible_cpus->cpus[n].type);
+        object_property_set_int(cpu, "mp-affinity",
+                                possible_cpus->cpus[n].arch_id, NULL);
+
+        cs = CPU(cpu);
+        cs->cpu_index = n;
+
+        numa_cpu_pre_plug(&possible_cpus->cpus[cs->cpu_index], DEVICE(cpu),
+                          &error_fatal);
+
+        object_property_set_bool(cpu, "has_el3", false, NULL);
+        object_property_set_bool(cpu, "has_el2", false, NULL);
+        object_property_set_int(cpu, "psci-conduit", QEMU_PSCI_CONDUIT_HVC,
+                                NULL);
+
+        /* Secondary CPUs start in PSCI powered-down state */
+        if (n > 0) {
+            object_property_set_bool(cpu, "start-powered-off", true, NULL);
+        }
+
+        object_property_set_link(cpu, "memory", OBJECT(sysmem), &error_abort);
+        qdev_realize(DEVICE(cpu), NULL, &error_fatal);
+        object_unref(cpu);
+    }
+
+    memory_region_add_subregion(sysmem, vms->memmap[VMAPPLE_MEM].base,
+                                machine->ram);
+
+    create_gic(vms, sysmem);
+    create_bdif(vms, sysmem);
+    create_pvpanic(vms, sysmem);
+    create_aes(vms, sysmem);
+    create_gfx(vms, sysmem);
+    create_uart(vms, VMAPPLE_UART, sysmem, serial_hd(0));
+    create_rtc(vms);
+    create_pcie(vms);
+
+    create_gpio_devices(vms, VMAPPLE_GPIO, sysmem);
+
+    vmapple_firmware_init(vms, sysmem);
+    create_cfg(vms, sysmem);
+
+    /* connect powerdown request */
+    vms->powerdown_notifier.notify = vmapple_powerdown_req;
+    qemu_register_powerdown_notifier(&vms->powerdown_notifier);
+
+    vms->bootinfo.ram_size = machine->ram_size;
+    vms->bootinfo.board_id = -1;
+    vms->bootinfo.loader_start = vms->memmap[VMAPPLE_MEM].base;
+    vms->bootinfo.skip_dtb_autoload = true;
+    vms->bootinfo.firmware_loaded = true;
+    arm_load_kernel(ARM_CPU(first_cpu), machine, &vms->bootinfo);
+
+    qemu_register_reset(vmapple_reset, vms);
+}
+
+static CpuInstanceProperties
+vmapple_cpu_index_to_props(MachineState *ms, unsigned cpu_index)
+{
+    MachineClass *mc = MACHINE_GET_CLASS(ms);
+    const CPUArchIdList *possible_cpus = mc->possible_cpu_arch_ids(ms);
+
+    assert(cpu_index < possible_cpus->len);
+    return possible_cpus->cpus[cpu_index].props;
+}
+
+
+static int64_t vmapple_get_default_cpu_node_id(const MachineState *ms, int idx)
+{
+    return idx % ms->numa_state->num_nodes;
+}
+
+static const CPUArchIdList *vmapple_possible_cpu_arch_ids(MachineState *ms)
+{
+    int n;
+    unsigned int max_cpus = ms->smp.max_cpus;
+
+    if (ms->possible_cpus) {
+        assert(ms->possible_cpus->len == max_cpus);
+        return ms->possible_cpus;
+    }
+
+    ms->possible_cpus = g_malloc0(sizeof(CPUArchIdList) +
+                                  sizeof(CPUArchId) * max_cpus);
+    ms->possible_cpus->len = max_cpus;
+    for (n = 0; n < ms->possible_cpus->len; n++) {
+        ms->possible_cpus->cpus[n].type = ms->cpu_type;
+        ms->possible_cpus->cpus[n].arch_id =
+            arm_build_mp_affinity(n, GICV3_TARGETLIST_BITS);
+        ms->possible_cpus->cpus[n].props.has_thread_id = true;
+        ms->possible_cpus->cpus[n].props.thread_id = n;
+    }
+    return ms->possible_cpus;
+}
+
+static void vmapple_get_uuid(Object *obj, Visitor *v, const char *name,
+                             void *opaque, Error **errp)
+{
+    VMAppleMachineState *vms = VMAPPLE_MACHINE(obj);
+    uint64_t value = be64_to_cpu(vms->uuid);
+
+    visit_type_uint64(v, name, &value, errp);
+}
+
+static void vmapple_set_uuid(Object *obj, Visitor *v, const char *name,
+                             void *opaque, Error **errp)
+{
+    VMAppleMachineState *vms = VMAPPLE_MACHINE(obj);
+    Error *error = NULL;
+    uint64_t value;
+
+    visit_type_uint64(v, name, &value, &error);
+    if (error) {
+        error_propagate(errp, error);
+        return;
+    }
+
+    vms->uuid = cpu_to_be64(value);
+}
+
+static void vmapple_machine_class_init(ObjectClass *oc, void *data)
+{
+    MachineClass *mc = MACHINE_CLASS(oc);
+
+    mc->init = mach_vmapple_init;
+    mc->max_cpus = 32;
+    mc->block_default_type = IF_VIRTIO;
+    mc->no_cdrom = 1;
+    mc->pci_allow_0_address = true;
+    mc->minimum_page_bits = 12;
+    mc->possible_cpu_arch_ids = vmapple_possible_cpu_arch_ids;
+    mc->cpu_index_to_instance_props = vmapple_cpu_index_to_props;
+    if (hvf_enabled()) {
+        mc->default_cpu_type = ARM_CPU_TYPE_NAME("host");
+    } else {
+        mc->default_cpu_type = ARM_CPU_TYPE_NAME("max");
+    }
+    mc->get_default_cpu_node_id = vmapple_get_default_cpu_node_id;
+    mc->default_ram_id = "mach-vmapple.ram";
+
+    object_register_sugar_prop(TYPE_VIRTIO_PCI, "disable-legacy",
+                               "on", true);
+
+    object_class_property_add(oc, "uuid", "uint64", vmapple_get_uuid,
+                              vmapple_set_uuid, NULL, NULL);
+    object_class_property_set_description(oc, "uuid", "Machine UUID (SDOM)");
+}
+
+static void vmapple_instance_init(Object *obj)
+{
+    VMAppleMachineState *vms = VMAPPLE_MACHINE(obj);
+
+    vms->irqmap = irqmap;
+}
+
+static const TypeInfo vmapple_machine_info = {
+    .name          = TYPE_VMAPPLE_MACHINE,
+    .parent        = TYPE_MACHINE,
+    .abstract      = true,
+    .instance_size = sizeof(VMAppleMachineState),
+    .class_size    = sizeof(VMAppleMachineClass),
+    .class_init    = vmapple_machine_class_init,
+    .instance_init = vmapple_instance_init,
+};
+
+static void machvmapple_machine_init(void)
+{
+    type_register_static(&vmapple_machine_info);
+}
+type_init(machvmapple_machine_init);
+
+static void vmapple_machine_8_1_options(MachineClass *mc)
+{
+}
+DEFINE_VMAPPLE_MACHINE_AS_LATEST(8, 1)
+
-- 
2.39.3 (Apple Git-145)

Re: [PATCH v3 02/14] hw/display/apple-gfx: Adds PCI implementation

[BALATON Zoltan]

[-- Attachment #1: Type: text/plain, Size: 6623 bytes --]

On Sat, 28 Sep 2024, Phil Dennis-Jordan wrote:
> This change wires up the PCI variant of the paravirtualised
> graphics device, mainly useful for x86-64 macOS guests, implemented
> by macOS's ParavirtualizedGraphics.framework. It builds on code
> shared with the vmapple/mmio variant of the PVG device.
>
> Signed-off-by: Phil Dennis-Jordan <phil@philjordan.eu>
> ---
> hw/display/Kconfig         |   5 ++
> hw/display/apple-gfx-pci.m | 138 +++++++++++++++++++++++++++++++++++++
> hw/display/meson.build     |   1 +
> 3 files changed, 144 insertions(+)
> create mode 100644 hw/display/apple-gfx-pci.m
>
> diff --git a/hw/display/Kconfig b/hw/display/Kconfig
> index 179a479d220..c2ec268f8e9 100644
> --- a/hw/display/Kconfig
> +++ b/hw/display/Kconfig
> @@ -152,3 +152,8 @@ config MAC_PVG_VMAPPLE
>     bool
>     depends on MAC_PVG
>     depends on ARM
> +
> +config MAC_PVG_PCI
> +    bool
> +    depends on MAC_PVG && PCI
> +    default y if PCI_DEVICES
> diff --git a/hw/display/apple-gfx-pci.m b/hw/display/apple-gfx-pci.m
> new file mode 100644
> index 00000000000..9370258ee46
> --- /dev/null
> +++ b/hw/display/apple-gfx-pci.m
> @@ -0,0 +1,138 @@
> +/*
> + * QEMU Apple ParavirtualizedGraphics.framework device, PCI variant
> + *
> + * Copyright © 2023-2024 Phil Dennis-Jordan
> + *
> + * This work is licensed under the terms of the GNU GPL, version 2 or later.
> + * See the COPYING file in the top-level directory.
> + *
> + * ParavirtualizedGraphics.framework is a set of libraries that macOS provides
> + * which implements 3d graphics passthrough to the host as well as a
> + * proprietary guest communication channel to drive it. This device model
> + * implements support to drive that library from within QEMU as a PCI device
> + * aimed primarily at x86-64 macOS VMs.
> + */
> +
> +#include "apple-gfx.h"
> +#include "hw/pci/pci_device.h"
> +#include "hw/pci/msi.h"
> +#include "qapi/error.h"
> +#include "trace.h"
> +#import <ParavirtualizedGraphics/ParavirtualizedGraphics.h>
> +
> +typedef struct AppleGFXPCIState {
> +    PCIDevice parent_obj;
> +
> +    AppleGFXState common;
> +} AppleGFXPCIState;

You don't need typedef here because OBJECT_DECLARE_SIMPLE_TYPE will add 
that. You can also put the struct AppleGFXPCIState definition after the 
OBJECT_DECLARE_SIMPLE_TYPE line. (See other devices for example.)

Regards,
BALATON Zoltan

> +
> +OBJECT_DECLARE_SIMPLE_TYPE(AppleGFXPCIState, APPLE_GFX_PCI)
> +
> +static const char* apple_gfx_pci_option_rom_path = NULL;
> +
> +static void apple_gfx_init_option_rom_path(void)
> +{
> +    NSURL *option_rom_url = PGCopyOptionROMURL();
> +    const char *option_rom_path = option_rom_url.fileSystemRepresentation;
> +    if (option_rom_url.fileURL && option_rom_path != NULL) {
> +        apple_gfx_pci_option_rom_path = g_strdup(option_rom_path);
> +    }
> +    [option_rom_url release];
> +}
> +
> +static void apple_gfx_pci_init(Object *obj)
> +{
> +    AppleGFXPCIState *s = APPLE_GFX_PCI(obj);
> +
> +    if (!apple_gfx_pci_option_rom_path) {
> +        /* Done on device not class init to avoid -daemonize ObjC fork crash */
> +        PCIDeviceClass *pci = PCI_DEVICE_CLASS(object_get_class(obj));
> +        apple_gfx_init_option_rom_path();
> +        pci->romfile = apple_gfx_pci_option_rom_path;
> +    }
> +
> +    apple_gfx_common_init(obj, &s->common, TYPE_APPLE_GFX_PCI);
> +}
> +
> +static void apple_gfx_pci_interrupt(PCIDevice *dev, AppleGFXPCIState *s,
> +                                    uint32_t vector)
> +{
> +    bool msi_ok;
> +    trace_apple_gfx_raise_irq(vector);
> +
> +    msi_ok = msi_enabled(dev);
> +    if (msi_ok) {
> +        msi_notify(dev, vector);
> +    }
> +}
> +
> +static void apple_gfx_pci_realize(PCIDevice *dev, Error **errp)
> +{
> +    AppleGFXPCIState *s = APPLE_GFX_PCI(dev);
> +    Error *err = NULL;
> +    int ret;
> +
> +    pci_register_bar(dev, PG_PCI_BAR_MMIO,
> +                     PCI_BASE_ADDRESS_SPACE_MEMORY, &s->common.iomem_gfx);
> +
> +    ret = msi_init(dev, 0x0 /* config offset; 0 = find space */,
> +                   PG_PCI_MAX_MSI_VECTORS, true /* msi64bit */,
> +                   false /*msi_per_vector_mask*/, &err);
> +    if (ret != 0) {
> +        error_propagate(errp, err);
> +        return;
> +    }
> +
> +    @autoreleasepool {
> +        PGDeviceDescriptor *desc = [PGDeviceDescriptor new];
> +        desc.raiseInterrupt = ^(uint32_t vector) {
> +            apple_gfx_pci_interrupt(dev, s, vector);
> +        };
> +
> +        apple_gfx_common_realize(&s->common, desc);
> +        [desc release];
> +        desc = nil;
> +    }
> +}
> +
> +static void apple_gfx_pci_reset(Object *obj, ResetType type)
> +{
> +    AppleGFXPCIState *s = APPLE_GFX_PCI(obj);
> +    [s->common.pgdev reset];
> +}
> +
> +static void apple_gfx_pci_class_init(ObjectClass *klass, void *data)
> +{
> +    DeviceClass *dc = DEVICE_CLASS(klass);
> +    PCIDeviceClass *pci = PCI_DEVICE_CLASS(klass);
> +    ResettableClass *rc = RESETTABLE_CLASS(klass);
> +
> +    assert(rc->phases.hold == NULL);
> +    rc->phases.hold = apple_gfx_pci_reset;
> +    dc->desc = "macOS Paravirtualized Graphics PCI Display Controller";
> +    dc->hotpluggable = false;
> +    set_bit(DEVICE_CATEGORY_DISPLAY, dc->categories);
> +
> +    pci->vendor_id = PG_PCI_VENDOR_ID;
> +    pci->device_id = PG_PCI_DEVICE_ID;
> +    pci->class_id = PCI_CLASS_DISPLAY_OTHER;
> +    pci->realize = apple_gfx_pci_realize;
> +
> +    // TODO: Property for setting mode list
> +}
> +
> +static TypeInfo apple_gfx_pci_types[] = {
> +    {
> +        .name          = TYPE_APPLE_GFX_PCI,
> +        .parent        = TYPE_PCI_DEVICE,
> +        .instance_size = sizeof(AppleGFXPCIState),
> +        .class_init    = apple_gfx_pci_class_init,
> +        .instance_init = apple_gfx_pci_init,
> +        .interfaces = (InterfaceInfo[]) {
> +            { INTERFACE_PCIE_DEVICE },
> +            { },
> +        },
> +    }
> +};
> +DEFINE_TYPES(apple_gfx_pci_types)
> +
> diff --git a/hw/display/meson.build b/hw/display/meson.build
> index 70d855749c0..ceb7bb07612 100644
> --- a/hw/display/meson.build
> +++ b/hw/display/meson.build
> @@ -67,6 +67,7 @@ system_ss.add(when: 'CONFIG_ATI_VGA', if_true: [files('ati.c', 'ati_2d.c', 'ati_
>
> system_ss.add(when: 'CONFIG_MAC_PVG',         if_true: [files('apple-gfx.m'), pvg, metal])
> system_ss.add(when: 'CONFIG_MAC_PVG_VMAPPLE', if_true: [files('apple-gfx-vmapple.m'), pvg, metal])
> +system_ss.add(when: 'CONFIG_MAC_PVG_PCI',     if_true: [files('apple-gfx-pci.m'), pvg, metal])
>
> if config_all_devices.has_key('CONFIG_VIRTIO_GPU')
>   virtio_gpu_ss = ss.source_set()
>

Re: [PATCH v3 02/14] hw/display/apple-gfx: Adds PCI implementation

[Phil Dennis-Jordan]

[-- Attachment #1: Type: text/plain, Size: 578 bytes --]

On Sat, 28 Sept 2024 at 12:39, BALATON Zoltan <balaton@eik.bme.hu> wrote:

> > +typedef struct AppleGFXPCIState {
> > +    PCIDevice parent_obj;
> > +
> > +    AppleGFXState common;
> > +} AppleGFXPCIState;
>
> You don't need typedef here because OBJECT_DECLARE_SIMPLE_TYPE will add
> that. You can also put the struct AppleGFXPCIState definition after the
> OBJECT_DECLARE_SIMPLE_TYPE line. (See other devices for example.)
>
>
Thanks for pointing that out. I've locally applied that advice to this and
the other relevant type declarations in the series, it'll be in v4.

Phil

[-- Attachment #2: Type: text/html, Size: 1008 bytes --]

Re: [PATCH v3 01/14] hw/display/apple-gfx: Introduce ParavirtualizedGraphics.Framework support

[Akihiko Odaki]

On 2024/09/28 17:57, Phil Dennis-Jordan wrote:
> MacOS provides a framework (library) that allows any vmm to implement a
> paravirtualized 3d graphics passthrough to the host metal stack called
> ParavirtualizedGraphics.Framework (PVG). The library abstracts away
> almost every aspect of the paravirtualized device model and only provides
> and receives callbacks on MMIO access as well as to share memory address
> space between the VM and PVG.
> 
> This patch implements a QEMU device that drives PVG for the VMApple
> variant of it.

I think it is better to name it MMIO variant instead of VMApple. There 
is nothing specific to VMApple in: hw/display/apple-gfx-vmapple.m

> 
> Signed-off-by: Alexander Graf <graf@amazon.com>
> Co-authored-by: Alexander Graf <graf@amazon.com>
> 
> Subsequent changes:
> 
>   * Cherry-pick/rebase conflict fixes
>   * BQL function renaming
>   * Moved from hw/vmapple/ (useful outside that machine type)
>   * Code review comments: Switched to DEFINE_TYPES macro & little endian
>     MMIO.
>   * Removed some dead/superfluous code
>   * Mad set_mode thread & memory safe
>   * Added migration blocker due to lack of (de-)serialisation.
>   * Fixes to ObjC refcounting and autorelease pool usage.
>   * Fixed ObjC new/init misuse
>   * Switched to ObjC category extension for private property.
>   * Simplified task memory mapping and made it thread safe.
>   * Refactoring to split generic and vmapple MMIO variant specific
>     code.
>   * Switched to asynchronous MMIO writes on x86-64
>   * Rendering and graphics update are now done asynchronously
>   * Fixed cursor handling
>   * Coding convention fixes
>   * Removed software cursor compositing
> 
> Signed-off-by: Phil Dennis-Jordan <phil@philjordan.eu>
> 
> ---
> 
> v3:
> 
>   * Rebased on latest upstream, fixed breakages including switching to Resettable methods.
>   * Squashed patches dealing with dGPUs, MMIO area size, and GPU picking.
>   * Allow re-entrant MMIO; this simplifies the code and solves the divergence
>     between x86-64 and arm64 variants.
> 
>   hw/display/Kconfig             |   9 +
>   hw/display/apple-gfx-vmapple.m | 215 +++++++++++++
>   hw/display/apple-gfx.h         |  57 ++++
>   hw/display/apple-gfx.m         | 536 +++++++++++++++++++++++++++++++++
>   hw/display/meson.build         |   2 +
>   hw/display/trace-events        |  26 ++
>   meson.build                    |   4 +
>   7 files changed, 849 insertions(+)
>   create mode 100644 hw/display/apple-gfx-vmapple.m
>   create mode 100644 hw/display/apple-gfx.h
>   create mode 100644 hw/display/apple-gfx.m
> 
> diff --git a/hw/display/Kconfig b/hw/display/Kconfig
> index a4552c8ed78..179a479d220 100644
> --- a/hw/display/Kconfig
> +++ b/hw/display/Kconfig
> @@ -143,3 +143,12 @@ config XLNX_DISPLAYPORT
>   
>   config DM163
>       bool
> +
> +config MAC_PVG
> +    bool
> +    default y
> +
> +config MAC_PVG_VMAPPLE
> +    bool
> +    depends on MAC_PVG
> +    depends on ARM

Use AARCH64 instead.

> diff --git a/hw/display/apple-gfx-vmapple.m b/hw/display/apple-gfx-vmapple.m
> new file mode 100644
> index 00000000000..d8fc7651dde
> --- /dev/null
> +++ b/hw/display/apple-gfx-vmapple.m
> @@ -0,0 +1,215 @@
> +/*
> + * QEMU Apple ParavirtualizedGraphics.framework device, vmapple (arm64) variant
> + *
> + * Copyright © 2023 Amazon.com, Inc. or its affiliates. All Rights Reserved.
> + *
> + * This work is licensed under the terms of the GNU GPL, version 2 or later.
> + * See the COPYING file in the top-level directory.
> + *
> + * ParavirtualizedGraphics.framework is a set of libraries that macOS provides
> + * which implements 3d graphics passthrough to the host as well as a
> + * proprietary guest communication channel to drive it. This device model
> + * implements support to drive that library from within QEMU as an MMIO-based
> + * system device for macOS on arm64 VMs.
> + */
> +
> +#include "apple-gfx.h"
> +#include "monitor/monitor.h"
> +#include "hw/sysbus.h"
> +#include "hw/irq.h"
> +#include "trace.h"
> +#import <ParavirtualizedGraphics/ParavirtualizedGraphics.h>
> +
> +_Static_assert(__aarch64__, "");

I don't think this assertion is worthwhile. This assertion will trigger 
if you accidentally remove depends on AARCH64 from Kconfig, but I don't 
think such code change happens by accident, and there is no reason to 
believe that this assertion won't be removed in such a case.

> +
> +/*
> + * ParavirtualizedGraphics.Framework only ships header files for the PCI
> + * variant which does not include IOSFC descriptors and host devices. We add
> + * their definitions here so that we can also work with the ARM version.
> + */
> +typedef bool(^IOSFCRaiseInterrupt)(uint32_t vector);
> +typedef bool(^IOSFCUnmapMemory)(
> +    void *a, void *b, void *c, void *d, void *e, void *f);

Omit dummy parameter names.

> +typedef bool(^IOSFCMapMemory)(
> +    uint64_t phys, uint64_t len, bool ro, void **va, void *e, void *f);
> +
> +@interface PGDeviceDescriptor (IOSurfaceMapper)
> +@property (readwrite, nonatomic) bool usingIOSurfaceMapper;
> +@end
> +
> +@interface PGIOSurfaceHostDeviceDescriptor : NSObject
> +-(PGIOSurfaceHostDeviceDescriptor *)init;
> +@property (readwrite, nonatomic, copy, nullable) IOSFCMapMemory mapMemory;
> +@property (readwrite, nonatomic, copy, nullable) IOSFCUnmapMemory unmapMemory;
> +@property (readwrite, nonatomic, copy, nullable) IOSFCRaiseInterrupt raiseInterrupt;
> +@end
> +
> +@interface PGIOSurfaceHostDevice : NSObject
> +-(instancetype)initWithDescriptor:(PGIOSurfaceHostDeviceDescriptor *) desc;
> +-(uint32_t)mmioReadAtOffset:(size_t) offset;
> +-(void)mmioWriteAtOffset:(size_t) offset value:(uint32_t)value;

It has inconsistent spacing between types and parameter names. It is 
conventional to omit spaces between them as far as I remember.

> +@end
> +
> +typedef struct AppleGFXVmappleState {
> +    SysBusDevice parent_obj;
> +
> +    AppleGFXState common;
> +
> +    qemu_irq irq_gfx;
> +    qemu_irq irq_iosfc;
> +    MemoryRegion iomem_iosfc;
> +    PGIOSurfaceHostDevice *pgiosfc;
> +} AppleGFXVmappleState;
> +
> +OBJECT_DECLARE_SIMPLE_TYPE(AppleGFXVmappleState, APPLE_GFX_VMAPPLE)
> +
> +
> +static uint64_t apple_iosfc_read(void *opaque, hwaddr offset, unsigned size)
> +{
> +    AppleGFXVmappleState *s = opaque;
> +    uint64_t res = 0;
> +
> +    bql_unlock();

It is dangerous to unlock BQL at an arbitrary place. Instead of 
unlocking, I suggest:
- running [s->pgiosfc mmioReadAtOffset:offset] on another thread
- using a bottom half to request operations that require BQL from the 
thread running [s->pgiosfc mmioReadAtOffset:offset]
- calling AIO_WAIT_WHILE() to process the bottom half and to wait for 
the completion of [s->pgiosfc mmioReadAtOffset:offset]

> +    res = [s->pgiosfc mmioReadAtOffset:offset];
> +    bql_lock();
> +
> +    trace_apple_iosfc_read(offset, res);
> +
> +    return res;
> +}
> +
> +static void apple_iosfc_write(
> +    void *opaque, hwaddr offset, uint64_t val, unsigned size)
> +{
> +    AppleGFXVmappleState *s = opaque;
> +
> +    trace_apple_iosfc_write(offset, val);
> +
> +    [s->pgiosfc mmioWriteAtOffset:offset value:val];
> +}
> +
> +static const MemoryRegionOps apple_iosfc_ops = {
> +    .read = apple_iosfc_read,
> +    .write = apple_iosfc_write,
> +    .endianness = DEVICE_LITTLE_ENDIAN,
> +    .valid = {
> +        .min_access_size = 4,
> +        .max_access_size = 8,
> +    },
> +    .impl = {
> +        .min_access_size = 4,
> +        .max_access_size = 8,
> +    },
> +};
> +
> +static PGIOSurfaceHostDevice *apple_gfx_prepare_iosurface_host_device(
> +    AppleGFXVmappleState *s)
> +{
> +    PGIOSurfaceHostDeviceDescriptor *iosfc_desc =
> +        [PGIOSurfaceHostDeviceDescriptor new];
> +    PGIOSurfaceHostDevice *iosfc_host_dev = nil;
> +
> +    iosfc_desc.mapMemory =
> +        ^(uint64_t phys, uint64_t len, bool ro, void **va, void *e, void *f) {
> +            trace_apple_iosfc_map_memory(phys, len, ro, va, e, f);
> +            MemoryRegion *tmp_mr;
> +            *va = gpa2hva(&tmp_mr, phys, len, NULL);

Use: dma_memory_map()

> +            return (bool)true;

Why cast?

> +        };
> +
> +    iosfc_desc.unmapMemory =
> +        ^(void *a, void *b, void *c, void *d, void *e, void *f) {
> +            trace_apple_iosfc_unmap_memory(a, b, c, d, e, f);
> +            return (bool)true;
> +        };
> +
> +    iosfc_desc.raiseInterrupt = ^(uint32_t vector) {
> +        trace_apple_iosfc_raise_irq(vector);
> +        bool locked = bql_locked();
> +        if (!locked) {
> +            bql_lock();
> +        }
 > +        qemu_irq_pulse(s->irq_iosfc);> +        if (!locked) {
> +            bql_unlock();
> +        }
> +        return (bool)true;
> +    };
> +
> +    iosfc_host_dev =
> +        [[PGIOSurfaceHostDevice alloc] initWithDescriptor:iosfc_desc];
> +    [iosfc_desc release];
> +    return iosfc_host_dev;
> +}
> +
> +static void apple_gfx_vmapple_realize(DeviceState *dev, Error **errp)
> +{
> +    @autoreleasepool {

This autoreleasepool is not used.

> +        AppleGFXVmappleState *s = APPLE_GFX_VMAPPLE(dev);
> +
> +        PGDeviceDescriptor *desc = [PGDeviceDescriptor new];
> +        desc.usingIOSurfaceMapper = true;
> +        desc.raiseInterrupt = ^(uint32_t vector) {
> +            bool locked;
> +
> +            trace_apple_gfx_raise_irq(vector);
> +            locked = bql_locked();
> +            if (!locked) {
> +                bql_lock();
> +            }
> +            qemu_irq_pulse(s->irq_gfx);
> +            if (!locked) {
> +                bql_unlock();
> +            }
> +        };
> +
> +        s->pgiosfc = apple_gfx_prepare_iosurface_host_device(s);
> +
> +        apple_gfx_common_realize(&s->common, desc);
> +        [desc release];
> +        desc = nil;
> +    }
> +}
> +
> +static void apple_gfx_vmapple_init(Object *obj)
> +{
> +    AppleGFXVmappleState *s = APPLE_GFX_VMAPPLE(obj);
> +
> +    apple_gfx_common_init(obj, &s->common, TYPE_APPLE_GFX_VMAPPLE);
> +
> +    memory_region_init_io(&s->iomem_iosfc, obj, &apple_iosfc_ops, s,
> +                          TYPE_APPLE_GFX_VMAPPLE, 0x10000);
> +    sysbus_init_mmio(SYS_BUS_DEVICE(s), &s->common.iomem_gfx);
> +    sysbus_init_mmio(SYS_BUS_DEVICE(s), &s->iomem_iosfc);
> +    sysbus_init_irq(SYS_BUS_DEVICE(s), &s->irq_gfx);
> +    sysbus_init_irq(SYS_BUS_DEVICE(s), &s->irq_iosfc);
> +}
> +
> +static void apple_gfx_vmapple_reset(Object *obj, ResetType type)
> +{
> +    AppleGFXVmappleState *s = APPLE_GFX_VMAPPLE(obj);
> +    [s->common.pgdev reset];
> +}
> +
> +
> +static void apple_gfx_vmapple_class_init(ObjectClass *klass, void *data)
> +{
> +    DeviceClass *dc = DEVICE_CLASS(klass);
> +    ResettableClass *rc = RESETTABLE_CLASS(klass);
> +
> +    assert(rc->phases.hold == NULL);

Please remove this assertion. Having such an assertion for all types 
that inherit TYPE_SYS_BUS_DEVICE will make the code base too cluttered, 
and there is nothing special with TYPE_APPLE_GFX_VMAPPLE.

> +    rc->phases.hold = apple_gfx_vmapple_reset;
> +
> +    dc->realize = apple_gfx_vmapple_realize;
> +}
> +
> +static TypeInfo apple_gfx_vmapple_types[] = {
> +    {
> +        .name          = TYPE_APPLE_GFX_VMAPPLE,
> +        .parent        = TYPE_SYS_BUS_DEVICE,
> +        .instance_size = sizeof(AppleGFXVmappleState),
> +        .class_init    = apple_gfx_vmapple_class_init,
> +        .instance_init = apple_gfx_vmapple_init,
> +    }
> +};
> +DEFINE_TYPES(apple_gfx_vmapple_types)
> diff --git a/hw/display/apple-gfx.h b/hw/display/apple-gfx.h
> new file mode 100644
> index 00000000000..995ecf7f4a7
> --- /dev/null
> +++ b/hw/display/apple-gfx.h
> @@ -0,0 +1,57 @@
> +#ifndef QEMU_APPLE_GFX_H
> +#define QEMU_APPLE_GFX_H
> +
> +#define TYPE_APPLE_GFX_VMAPPLE      "apple-gfx-vmapple"
> +#define TYPE_APPLE_GFX_PCI          "apple-gfx-pci"
> +
> +#include "qemu/typedefs.h"
> +
> +typedef struct AppleGFXState AppleGFXState;
> +
> +void apple_gfx_common_init(Object *obj, AppleGFXState *s, const char* obj_name);
> +
> +#ifdef __OBJC__

This ifdef is unnecessary.

> +
> +#include "qemu/osdep.h"
> +#include "exec/memory.h"
> +#include "ui/surface.h"
> +#include <dispatch/dispatch.h>
> +
> +@class PGDeviceDescriptor;
> +@protocol PGDevice;
> +@protocol PGDisplay;
> +@protocol MTLDevice;
> +@protocol MTLTexture;
> +@protocol MTLCommandQueue;
> +
> +typedef QTAILQ_HEAD(, PGTask_s) AppleGFXTaskList;
> +
> +struct AppleGFXState {
> +    MemoryRegion iomem_gfx;
> +    id<PGDevice> pgdev;
> +    id<PGDisplay> pgdisp;
> +    AppleGFXTaskList tasks;
> +    QemuConsole *con;
> +    id<MTLDevice> mtl;
> +    id<MTLCommandQueue> mtl_queue;
> +    bool handles_frames;
> +    bool new_frame;
> +    bool cursor_show;
> +    QEMUCursor *cursor;
> +
> +    dispatch_queue_t render_queue;
> +    /* The following fields should only be accessed from render_queue: */
> +    bool gfx_update_requested;
> +    bool new_frame_ready;
> +    bool using_managed_texture_storage;
> +    int32_t pending_frames;
> +    void *vram;
> +    DisplaySurface *surface;
> +    id<MTLTexture> texture;
> +};
> +
> +void apple_gfx_common_realize(AppleGFXState *s, PGDeviceDescriptor *desc);
> +
> +#endif /* __OBJC__ */
> +
> +#endif
> diff --git a/hw/display/apple-gfx.m b/hw/display/apple-gfx.m
> new file mode 100644
> index 00000000000..837300f9cd4
> --- /dev/null
> +++ b/hw/display/apple-gfx.m
> @@ -0,0 +1,536 @@
> +/*
> + * QEMU Apple ParavirtualizedGraphics.framework device
> + *
> + * Copyright © 2023 Amazon.com, Inc. or its affiliates. All Rights Reserved.
> + *
> + * This work is licensed under the terms of the GNU GPL, version 2 or later.
> + * See the COPYING file in the top-level directory.
> + *
> + * ParavirtualizedGraphics.framework is a set of libraries that macOS provides
> + * which implements 3d graphics passthrough to the host as well as a
> + * proprietary guest communication channel to drive it. This device model
> + * implements support to drive that library from within QEMU.
> + */
> +
> +#include "apple-gfx.h"
> +#include "trace.h"
> +#include "qemu/main-loop.h"
> +#include "ui/console.h"
> +#include "monitor/monitor.h"
> +#include "qapi/error.h"
> +#include "migration/blocker.h"
> +#include <mach/mach_vm.h>
> +#import <ParavirtualizedGraphics/ParavirtualizedGraphics.h>
> +
> +static const PGDisplayCoord_t apple_gfx_modes[] = {
> +    { .x = 1440, .y = 1080 },
> +    { .x = 1280, .y = 1024 },
> +};
> +
> +typedef struct PGTask_s { // Name matches forward declaration in PG header

Let's name it AppleGFXTask. It is a common practice to have the same tag 
name and typedef in QEMU.

Please also avoid using // comment as suggested in: docs/devel/style.rst
You can just remove the comment in this case as the usage of struct tag 
names in QTAILQ_HEAD() is very common.

> +    QTAILQ_ENTRY(PGTask_s) node;
> +    mach_vm_address_t address;
> +    uint64_t len;
> +} AppleGFXTask;
> +
> +static Error *apple_gfx_mig_blocker;
> +
> +static void apple_gfx_render_frame_completed(AppleGFXState *s, void *vram,
> +                                             id<MTLTexture> texture);
> +
> +static AppleGFXTask *apple_gfx_new_task(AppleGFXState *s, uint64_t len)
> +{
> +    mach_vm_address_t task_mem;
> +    AppleGFXTask *task;
> +    kern_return_t r;
> +
> +    r = mach_vm_allocate(mach_task_self(), &task_mem, len, VM_FLAGS_ANYWHERE);
> +    if (r != KERN_SUCCESS || task_mem == 0) {

I think you can just check r. There is no reason to reject if the kernel 
insists it successfully allocated memory at 0 though that should never 
happen.

> +        return NULL;
> +    }
> +
> +    task = g_new0(AppleGFXTask, 1);
> +
> +    task->address = task_mem;
> +    task->len = len;
> +    QTAILQ_INSERT_TAIL(&s->tasks, task, node);
> +
> +    return task;
> +}
> +
> +static uint64_t apple_gfx_read(void *opaque, hwaddr offset, unsigned size)
> +{
> +    AppleGFXState *s = opaque;
> +    uint64_t res = 0;

Don't initialize it with an arbitrary value as such an initialization 
suppresses an uninitialized variable warning.

Please also keep naming consistent; you have used "r" for a similar 
variable previously.

> +
> +    bql_unlock();
> +    res = [s->pgdev mmioReadAtOffset:offset];
> +    bql_lock();
> +
> +    trace_apple_gfx_read(offset, res);
> +
> +    return res;
> +}
> +
> +static void apple_gfx_write(void *opaque, hwaddr offset, uint64_t val,
> +                            unsigned size)
> +{
> +    AppleGFXState *s = opaque;
> +
> +    trace_apple_gfx_write(offset, val);
> +
> +    bql_unlock();
> +    [s->pgdev mmioWriteAtOffset:offset value:val];
> +    bql_lock();
> +}
> +
> +static const MemoryRegionOps apple_gfx_ops = {
> +    .read = apple_gfx_read,
> +    .write = apple_gfx_write,
> +    .endianness = DEVICE_LITTLE_ENDIAN,
> +    .valid = {
> +        .min_access_size = 4,
> +        .max_access_size = 8,
> +    },
> +    .impl = {
> +        .min_access_size = 4,
> +        .max_access_size = 4,
> +    },
> +};
> +
> +static void apple_gfx_render_new_frame(AppleGFXState *s)
> +{
> +    BOOL r;
> +    void *vram = s->vram;
> +    uint32_t width = surface_width(s->surface);
> +    uint32_t height = surface_height(s->surface);
> +    MTLRegion region = MTLRegionMake2D(0, 0, width, height);
> +    id<MTLCommandBuffer> command_buffer = [s->mtl_queue commandBuffer];
> +    id<MTLTexture> texture = s->texture;
> +    r = [s->pgdisp encodeCurrentFrameToCommandBuffer:command_buffer
> +                                             texture:texture
> +                                              region:region];
> +    if (!r) {
> +        return;
> +    }
> +    [texture retain];
> +    if (s->using_managed_texture_storage) {
> +        /* "Managed" textures exist in both VRAM and RAM and must be synced. */
> +        id<MTLBlitCommandEncoder> blit = [command_buffer blitCommandEncoder];
> +        [blit synchronizeResource:texture];
> +        [blit endEncoding];
> +    }
> +    [command_buffer retain];

I don't think this call of retain is necessary. The completion handler 
gets the command buffer via parameter, which implies the command buffer 
is automatically retained until the completion handler finishes.

> +    [command_buffer addCompletedHandler:
> +        ^(id<MTLCommandBuffer> cb)
> +        {
> +            dispatch_async(s->render_queue, ^{
> +                apple_gfx_render_frame_completed(s, vram, texture);
> +                [texture release];
> +            });
> +            [command_buffer release];
> +        }];
> +    [command_buffer commit];
> +}
> +
> +static void copy_mtl_texture_to_surface_mem(id<MTLTexture> texture, void *vram)
> +{
> +    /* TODO: Skip this entirely on a pure Metal or headless/guest-only
> +     * rendering path, else use a blit command encoder? Needs careful
> +     * (double?) buffering design. */
> +    size_t width = texture.width, height = texture.height;
> +    MTLRegion region = MTLRegionMake2D(0, 0, width, height);
> +    [texture getBytes:vram
> +          bytesPerRow:(width * 4)
> +        bytesPerImage:(width * height * 4)
> +           fromRegion:region
> +          mipmapLevel:0
> +                slice:0];
> +}
> +
> +static void apple_gfx_render_frame_completed(AppleGFXState *s, void *vram,
> +                                             id<MTLTexture> texture)
> +{
> +    --s->pending_frames;
> +    assert(s->pending_frames >= 0);
> +
> +    if (vram != s->vram) {
> +        /* Display mode has changed, drop this old frame. */
> +        assert(texture != s->texture);
> +        g_free(vram);

This management of buffers looks a bit convoluted. I suggest remembering 
the width and height instead of pointers and comparing them. This way 
you can free resources in set_mode().

> +    } else {
> +        copy_mtl_texture_to_surface_mem(texture, vram);

Writing vram outside BQL may result in tearing.

> +        if (s->gfx_update_requested) {
> +            s->gfx_update_requested = false;
> +            dpy_gfx_update_full(s->con);
> +            graphic_hw_update_done(s->con);
 > +            s->new_frame_ready = false;

This assignment is unnecessary as s->new_frame_ready is always false 
when s->gfx_update_requested. If you want to make sure 
s->gfx_update_requested and s->new_frame_ready are mutually exclusive, 
use one enum value instead of having two bools.

> +        } else {
> +            s->new_frame_ready = true;
> +        }
> +    }
> +    if (s->pending_frames > 0) {
> +        apple_gfx_render_new_frame(s);
> +    }
> +}
> +
> +static void apple_gfx_fb_update_display(void *opaque)
> +{
> +    AppleGFXState *s = opaque;
> +
> +    dispatch_async(s->render_queue, ^{
> +        if (s->pending_frames > 0) {

It should check for s->new_frame_ready as 
apple_gfx_render_frame_completed() doesn't check if
s->pending_frames > 0 before calling graphic_hw_update_done(), which is 
inconsistent.

Checking if s->pending_frames > 0 both in apple_gfx_fb_update_display() 
and apple_gfx_render_frame_completed() is also problematic as that can 
defer graphic_hw_update_done() indefinitely if we are getting new frames 
too fast.

> +            s->gfx_update_requested = true;
> +        } else {
> +            if (s->new_frame_ready) {
> +                dpy_gfx_update_full(s->con);
> +                s->new_frame_ready = false;
> +            }
> +            graphic_hw_update_done(s->con);
 > +        }> +    });
> +}
> +
> +static const GraphicHwOps apple_gfx_fb_ops = {
> +    .gfx_update = apple_gfx_fb_update_display,
> +    .gfx_update_async = true,
> +};
> +
> +static void update_cursor(AppleGFXState *s)
> +{
> +    dpy_mouse_set(s->con, s->pgdisp.cursorPosition.x,
> +                  s->pgdisp.cursorPosition.y, s->cursor_show);
> +}
> +
> +static void set_mode(AppleGFXState *s, uint32_t width, uint32_t height)
> +{
> +    void *vram = NULL;
> +    DisplaySurface *surface;
> +    MTLTextureDescriptor *textureDescriptor;
> +    id<MTLTexture> texture = nil;
> +    __block bool no_change = false;
> +
> +    dispatch_sync(s->render_queue,

Calling dispatch_sync() while holding BQL may result in deadlock.

> +        ^{
> +            if (s->surface &&
> +                width == surface_width(s->surface) &&
> +                height == surface_height(s->surface)) {
> +                no_change = true;
> +            }
> +        });
> +
> +    if (no_change) {
> +        return;
> +    }
> +
> +    vram = g_malloc0(width * height * 4);
> +    surface = qemu_create_displaysurface_from(width, height, PIXMAN_LE_a8r8g8b8,
> +                                              width * 4, vram);
> +
> +    @autoreleasepool {
> +        textureDescriptor =
> +            [MTLTextureDescriptor
> +                texture2DDescriptorWithPixelFormat:MTLPixelFormatBGRA8Unorm
> +                                             width:width
> +                                            height:height
> +                                         mipmapped:NO];
> +        textureDescriptor.usage = s->pgdisp.minimumTextureUsage;
> +        texture = [s->mtl newTextureWithDescriptor:textureDescriptor];
> +    }
> +
> +    s->using_managed_texture_storage =
> +        (texture.storageMode == MTLStorageModeManaged);
> +
> +    dispatch_sync(s->render_queue,
> +        ^{
> +            id<MTLTexture> old_texture = nil;
> +            void *old_vram = s->vram;
> +            s->vram = vram;
> +            s->surface = surface;
> +
> +            dpy_gfx_replace_surface(s->con, surface);
> +
> +            old_texture = s->texture;
> +            s->texture = texture;
> +            [old_texture release];

You can just do:
[s->texture release];
s->texture = texture;

This will make s->texture dangling between the two statements, but that 
don't matter since s->texture is not an atomic variable that can be 
safely observed from another thread anyway.

> +
> +            if (s->pending_frames == 0) {
> +                g_free(old_vram);
> +            }
> +        });
> +}
> +
> +static void create_fb(AppleGFXState *s)
> +{
> +    s->con = graphic_console_init(NULL, 0, &apple_gfx_fb_ops, s);
> +    set_mode(s, 1440, 1080);
> +
> +    s->cursor_show = true;
> +}
> +
> +static size_t apple_gfx_get_default_mmio_range_size(void)
> +{
> +    size_t mmio_range_size;
> +    @autoreleasepool {
> +        PGDeviceDescriptor *desc = [PGDeviceDescriptor new];
> +        mmio_range_size = desc.mmioLength;
> +        [desc release];
> +    }
> +    return mmio_range_size;
> +}
> +
> +void apple_gfx_common_init(Object *obj, AppleGFXState *s, const char* obj_name)

This function can be merged into apple_gfx_common_realize().

> +{
> +    Error *local_err = NULL;
> +    int r;
> +    size_t mmio_range_size = apple_gfx_get_default_mmio_range_size();
> +
> +    trace_apple_gfx_common_init(obj_name, mmio_range_size);
> +    memory_region_init_io(&s->iomem_gfx, obj, &apple_gfx_ops, s, obj_name,
> +                          mmio_range_size);
> +    s->iomem_gfx.disable_reentrancy_guard = true;

Why do you disable reentrancy guard?

> +
> +    /* TODO: PVG framework supports serialising device state: integrate it! */
> +    if (apple_gfx_mig_blocker == NULL) {
> +        error_setg(&apple_gfx_mig_blocker,
> +                  "Migration state blocked by apple-gfx display device");
> +        r = migrate_add_blocker(&apple_gfx_mig_blocker, &local_err);
> +        if (r < 0) {
> +            error_report_err(local_err);

Please report the error to the caller of apple_gfx_common_realize() instead.

> +        }
> +    }
 > +}> +
> +static void apple_gfx_register_task_mapping_handlers(AppleGFXState *s,
> +                                                     PGDeviceDescriptor *desc)
> +{
> +    desc.createTask = ^(uint64_t vmSize, void * _Nullable * _Nonnull baseAddress) {
> +        AppleGFXTask *task = apple_gfx_new_task(s, vmSize);
> +        *baseAddress = (void*)task->address;

nit: please write as (void *) instead of (void*).

> +        trace_apple_gfx_create_task(vmSize, *baseAddress);
> +        return task;
> +    };
> +
> +    desc.destroyTask = ^(AppleGFXTask * _Nonnull task) {
> +        trace_apple_gfx_destroy_task(task);
> +        QTAILQ_REMOVE(&s->tasks, task, node);
> +        mach_vm_deallocate(mach_task_self(), task->address, task->len);
> +        g_free(task);
> +    };
> +
> +    desc.mapMemory = ^(AppleGFXTask * _Nonnull task, uint32_t rangeCount,
> +                       uint64_t virtualOffset, bool readOnly,
> +                       PGPhysicalMemoryRange_t * _Nonnull ranges) {
> +        kern_return_t r;
> +        mach_vm_address_t target, source;
> +        trace_apple_gfx_map_memory(task, rangeCount, virtualOffset, readOnly);
> +        for (int i = 0; i < rangeCount; i++) {
> +            PGPhysicalMemoryRange_t *range = &ranges[i];
> +            MemoryRegion *tmp_mr;
> +            /* TODO: Bounds checks? r/o? */
> +            bql_lock();
> +
> +            trace_apple_gfx_map_memory_range(i, range->physicalAddress,
> +                                             range->physicalLength);
> +
> +            target = task->address + virtualOffset;
> +            source = (mach_vm_address_t)gpa2hva(&tmp_mr,
> +                                                range->physicalAddress,
> +                                                range->physicalLength, NULL);
> +            vm_prot_t cur_protection = 0;
> +            vm_prot_t max_protection = 0;
> +            // Map guest RAM at range->physicalAddress into PG task memory range
> +            r = mach_vm_remap(mach_task_self(),
> +                              &target, range->physicalLength, vm_page_size - 1,
> +                              VM_FLAGS_FIXED | VM_FLAGS_OVERWRITE,
> +                              mach_task_self(),
> +                              source, false /* shared mapping, no copy */,
> +                              &cur_protection, &max_protection,
> +                              VM_INHERIT_COPY);
> +            trace_apple_gfx_remap(r, source, target);
> +            g_assert(r == KERN_SUCCESS);
> +
> +            bql_unlock();
> +
> +            virtualOffset += range->physicalLength;
> +        }
> +        return (bool)true;
> +    };
> +
> +    desc.unmapMemory = ^(AppleGFXTask * _Nonnull task, uint64_t virtualOffset,
> +                         uint64_t length) {
> +        kern_return_t r;
> +        mach_vm_address_t range_address;
> +
> +        trace_apple_gfx_unmap_memory(task, virtualOffset, length);
> +
> +        /* Replace task memory range with fresh pages, undoing the mapping
> +         * from guest RAM. */
> +        range_address = task->address + virtualOffset;
> +        r = mach_vm_allocate(mach_task_self(), &range_address, length,
> +                             VM_FLAGS_FIXED | VM_FLAGS_OVERWRITE);
> +        g_assert(r == KERN_SUCCESS);
> +
> +        return (bool)true;
> +    };
> +
> +    desc.readMemory = ^(uint64_t physicalAddress, uint64_t length,
> +                        void * _Nonnull dst) {
> +        trace_apple_gfx_read_memory(physicalAddress, length, dst);
> +        cpu_physical_memory_read(physicalAddress, dst, length);
> +        return (bool)true;
> +    };
> +
> +}
> +
> +static PGDisplayDescriptor *apple_gfx_prepare_display_handlers(AppleGFXState *s)

This name is misleading as it does a bit more than preparing handlers 
but sets properties like name and sizeInMillimeters.

> +{
> +    PGDisplayDescriptor *disp_desc = [PGDisplayDescriptor new];
> +
> +    disp_desc.name = @"QEMU display";
> +    disp_desc.sizeInMillimeters = NSMakeSize(400., 300.); /* A 20" display */
> +    disp_desc.queue = dispatch_get_main_queue();
> +    disp_desc.newFrameEventHandler = ^(void) {
> +        trace_apple_gfx_new_frame();
> +        dispatch_async(s->render_queue, ^{
> +            /* Drop frames if we get too far ahead. */
> +            if (s->pending_frames >= 2)
> +                return;
> +            ++s->pending_frames;
> +            if (s->pending_frames > 1) {
> +                return;
> +            }
> +            @autoreleasepool {
> +                apple_gfx_render_new_frame(s);
> +            }
> +        });
> +    };
> +    disp_desc.modeChangeHandler = ^(PGDisplayCoord_t sizeInPixels,
> +                                    OSType pixelFormat) {
> +        trace_apple_gfx_mode_change(sizeInPixels.x, sizeInPixels.y);
> +        set_mode(s, sizeInPixels.x, sizeInPixels.y);
> +    };
> +    disp_desc.cursorGlyphHandler = ^(NSBitmapImageRep *glyph,
> +                                     PGDisplayCoord_t hotSpot) {
> +        uint32_t bpp = glyph.bitsPerPixel;
> +        size_t width = glyph.pixelsWide;
> +        size_t height = glyph.pixelsHigh;
> +        size_t padding_bytes_per_row = glyph.bytesPerRow - width * 4;
> +        const uint8_t* px_data = glyph.bitmapData;
> +
> +        trace_apple_gfx_cursor_set(bpp, width, height);
> +
> +        if (s->cursor) {
> +            cursor_unref(s->cursor);
> +            s->cursor = NULL;
> +        }
> +
> +        if (bpp == 32) { /* Shouldn't be anything else, but just to be safe...*/
> +            s->cursor = cursor_alloc(width, height);
> +            s->cursor->hot_x = hotSpot.x;
> +            s->cursor->hot_y = hotSpot.y;
> +
> +            uint32_t *dest_px = s->cursor->data;
> +
> +            for (size_t y = 0; y < height; ++y) {
> +                for (size_t x = 0; x < width; ++x) {
> +                    /* NSBitmapImageRep's red & blue channels are swapped
> +                     * compared to QEMUCursor's. */
> +                    *dest_px =
> +                        (px_data[0] << 16u) |
> +                        (px_data[1] <<  8u) |
> +                        (px_data[2] <<  0u) |
> +                        (px_data[3] << 24u);
> +                    ++dest_px;
> +                    px_data += 4;
> +                }
> +                px_data += padding_bytes_per_row;
> +            }
> +            dpy_cursor_define(s->con, s->cursor);
> +            update_cursor(s);
> +        }
> +    };
> +    disp_desc.cursorShowHandler = ^(BOOL show) {
> +        trace_apple_gfx_cursor_show(show);
> +        s->cursor_show = show;
> +        update_cursor(s);
> +    };
> +    disp_desc.cursorMoveHandler = ^(void) {
> +        trace_apple_gfx_cursor_move();
> +        update_cursor(s);
> +    };
> +
> +    return disp_desc;
> +}
> +
> +static NSArray<PGDisplayMode*>* apple_gfx_prepare_display_mode_array(void)
> +{
> +    PGDisplayMode *modes[ARRAY_SIZE(apple_gfx_modes)];
> +    NSArray<PGDisplayMode*>* mode_array = nil;
> +    int i;
> +
> +    for (i = 0; i < ARRAY_SIZE(apple_gfx_modes); i++) {
> +        modes[i] =
> +            [[PGDisplayMode alloc] initWithSizeInPixels:apple_gfx_modes[i] refreshRateInHz:60.];
> +    }
> +
> +    mode_array = [NSArray arrayWithObjects:modes count:ARRAY_SIZE(apple_gfx_modes)];
> +
> +    for (i = 0; i < ARRAY_SIZE(apple_gfx_modes); i++) {
> +        [modes[i] release];
> +        modes[i] = nil;
> +    }
> +
> +    return mode_array;
> +}
> +
> +static id<MTLDevice> copy_suitable_metal_device(void)
> +{
> +    id<MTLDevice> dev = nil;
> +    NSArray<id<MTLDevice>> *devs = MTLCopyAllDevices();
> +
> +    /* Prefer a unified memory GPU. Failing that, pick a non-removable GPU. */
> +    for (size_t i = 0; i < devs.count; ++i) {
> +        if (devs[i].hasUnifiedMemory) {
> +            dev = devs[i];
> +            break;
> +        }
> +        if (!devs[i].removable) {
> +            dev = devs[i];
> +        }
> +    }
> +
> +    if (dev != nil) {
> +        [dev retain];
> +    } else {
> +        dev = MTLCreateSystemDefaultDevice();
> +    }
> +    [devs release];
> +
> +    return dev;
> +}
> +
> +void apple_gfx_common_realize(AppleGFXState *s, PGDeviceDescriptor *desc)
> +{
> +    PGDisplayDescriptor *disp_desc = nil;
> +
> +    QTAILQ_INIT(&s->tasks);
> +    s->render_queue = dispatch_queue_create("apple-gfx.render",
> +                                            DISPATCH_QUEUE_SERIAL);
> +    s->mtl = copy_suitable_metal_device();
> +    s->mtl_queue = [s->mtl newCommandQueue];
> +
> +    desc.device = s->mtl;
> +
> +    apple_gfx_register_task_mapping_handlers(s, desc);
> +
> +    s->pgdev = PGNewDeviceWithDescriptor(desc);
> +
> +    disp_desc = apple_gfx_prepare_display_handlers(s);
> +    s->pgdisp = [s->pgdev newDisplayWithDescriptor:disp_desc
> +                                              port:0 serialNum:1234];
> +    [disp_desc release];
> +    s->pgdisp.modeList = apple_gfx_prepare_display_mode_array();
> +
> +    create_fb(s);
> +}
> diff --git a/hw/display/meson.build b/hw/display/meson.build
> index 7db05eace97..70d855749c0 100644
> --- a/hw/display/meson.build
> +++ b/hw/display/meson.build
> @@ -65,6 +65,8 @@ system_ss.add(when: 'CONFIG_ARTIST', if_true: files('artist.c'))
>   
>   system_ss.add(when: 'CONFIG_ATI_VGA', if_true: [files('ati.c', 'ati_2d.c', 'ati_dbg.c'), pixman])
>   
> +system_ss.add(when: 'CONFIG_MAC_PVG',         if_true: [files('apple-gfx.m'), pvg, metal])
> +system_ss.add(when: 'CONFIG_MAC_PVG_VMAPPLE', if_true: [files('apple-gfx-vmapple.m'), pvg, metal])
>   
>   if config_all_devices.has_key('CONFIG_VIRTIO_GPU')
>     virtio_gpu_ss = ss.source_set()
> diff --git a/hw/display/trace-events b/hw/display/trace-events
> index 781f8a33203..1809a358e36 100644
> --- a/hw/display/trace-events
> +++ b/hw/display/trace-events
> @@ -191,3 +191,29 @@ dm163_bits_ppi(unsigned dest_width) "dest_width : %u"
>   dm163_leds(int led, uint32_t value) "led %d: 0x%x"
>   dm163_channels(int channel, uint8_t value) "channel %d: 0x%x"
>   dm163_refresh_rate(uint32_t rr) "refresh rate %d"
> +
> +# apple-gfx.m
> +apple_gfx_read(uint64_t offset, uint64_t res) "offset=0x%"PRIx64" res=0x%"PRIx64
> +apple_gfx_write(uint64_t offset, uint64_t val) "offset=0x%"PRIx64" val=0x%"PRIx64
> +apple_gfx_create_task(uint32_t vm_size, void *va) "vm_size=0x%x base_addr=%p"
> +apple_gfx_destroy_task(void *task) "task=%p"
> +apple_gfx_map_memory(void *task, uint32_t range_count, uint64_t virtual_offset, uint32_t read_only) "task=%p range_count=0x%x virtual_offset=0x%"PRIx64" read_only=%d"
> +apple_gfx_map_memory_range(uint32_t i, uint64_t phys_addr, uint64_t phys_len) "[%d] phys_addr=0x%"PRIx64" phys_len=0x%"PRIx64
> +apple_gfx_remap(uint64_t retval, uint64_t source, uint64_t target) "retval=%"PRId64" source=0x%"PRIx64" target=0x%"PRIx64
> +apple_gfx_unmap_memory(void *task, uint64_t virtual_offset, uint64_t length) "task=%p virtual_offset=0x%"PRIx64" length=0x%"PRIx64
> +apple_gfx_read_memory(uint64_t phys_address, uint64_t length, void *dst) "phys_addr=0x%"PRIx64" length=0x%"PRIx64" dest=%p"
> +apple_gfx_raise_irq(uint32_t vector) "vector=0x%x"
> +apple_gfx_new_frame(void) ""
> +apple_gfx_mode_change(uint64_t x, uint64_t y) "x=%"PRId64" y=%"PRId64
> +apple_gfx_cursor_set(uint32_t bpp, uint64_t width, uint64_t height) "bpp=%d width=%"PRId64" height=0x%"PRId64
> +apple_gfx_cursor_show(uint32_t show) "show=%d"
> +apple_gfx_cursor_move(void) ""
> +apple_gfx_common_init(const char *device_name, size_t mmio_size) "device: %s; MMIO size: %zu bytes"
> +
> +# apple-gfx-vmapple.m
> +apple_iosfc_read(uint64_t offset, uint64_t res) "offset=0x%"PRIx64" res=0x%"PRIx64
> +apple_iosfc_write(uint64_t offset, uint64_t val) "offset=0x%"PRIx64" val=0x%"PRIx64
> +apple_iosfc_map_memory(uint64_t phys, uint64_t len, uint32_t ro, void *va, void *e, void *f) "phys=0x%"PRIx64" len=0x%"PRIx64" ro=%d va=%p e=%p f=%p"
> +apple_iosfc_unmap_memory(void *a, void *b, void *c, void *d, void *e, void *f) "a=%p b=%p c=%p d=%p e=%p f=%p"
> +apple_iosfc_raise_irq(uint32_t vector) "vector=0x%x"
> +
> diff --git a/meson.build b/meson.build
> index 10464466ff3..f09df3f09d5 100644
> --- a/meson.build
> +++ b/meson.build
> @@ -741,6 +741,8 @@ socket = []
>   version_res = []
>   coref = []
>   iokit = []
> +pvg = []
> +metal = []
>   emulator_link_args = []
>   midl = not_found
>   widl = not_found
> @@ -762,6 +764,8 @@ elif host_os == 'darwin'
>     coref = dependency('appleframeworks', modules: 'CoreFoundation')
>     iokit = dependency('appleframeworks', modules: 'IOKit', required: false)
>     host_dsosuf = '.dylib'
> +  pvg = dependency('appleframeworks', modules: 'ParavirtualizedGraphics')
> +  metal = dependency('appleframeworks', modules: 'Metal')
>   elif host_os == 'sunos'
>     socket = [cc.find_library('socket'),
>               cc.find_library('nsl'),

Re: [PATCH v3 02/14] hw/display/apple-gfx: Adds PCI implementation

[Akihiko Odaki]

On 2024/09/28 17:57, Phil Dennis-Jordan wrote:
> This change wires up the PCI variant of the paravirtualised
> graphics device, mainly useful for x86-64 macOS guests, implemented
> by macOS's ParavirtualizedGraphics.framework. It builds on code
> shared with the vmapple/mmio variant of the PVG device.
> 
> Signed-off-by: Phil Dennis-Jordan <phil@philjordan.eu>
> ---
>   hw/display/Kconfig         |   5 ++
>   hw/display/apple-gfx-pci.m | 138 +++++++++++++++++++++++++++++++++++++
>   hw/display/meson.build     |   1 +
>   3 files changed, 144 insertions(+)
>   create mode 100644 hw/display/apple-gfx-pci.m
> 
> diff --git a/hw/display/Kconfig b/hw/display/Kconfig
> index 179a479d220..c2ec268f8e9 100644
> --- a/hw/display/Kconfig
> +++ b/hw/display/Kconfig
> @@ -152,3 +152,8 @@ config MAC_PVG_VMAPPLE
>       bool
>       depends on MAC_PVG
>       depends on ARM
> +
> +config MAC_PVG_PCI
> +    bool
> +    depends on MAC_PVG && PCI
> +    default y if PCI_DEVICES
> diff --git a/hw/display/apple-gfx-pci.m b/hw/display/apple-gfx-pci.m
> new file mode 100644
> index 00000000000..9370258ee46
> --- /dev/null
> +++ b/hw/display/apple-gfx-pci.m
> @@ -0,0 +1,138 @@
> +/*
> + * QEMU Apple ParavirtualizedGraphics.framework device, PCI variant
> + *
> + * Copyright © 2023-2024 Phil Dennis-Jordan
> + *
> + * This work is licensed under the terms of the GNU GPL, version 2 or later.
> + * See the COPYING file in the top-level directory.

Please use SPDX-License-Identifier instead.

> + *
> + * ParavirtualizedGraphics.framework is a set of libraries that macOS provides
> + * which implements 3d graphics passthrough to the host as well as a
> + * proprietary guest communication channel to drive it. This device model
> + * implements support to drive that library from within QEMU as a PCI device
> + * aimed primarily at x86-64 macOS VMs.
> + */
> +
> +#include "apple-gfx.h"
> +#include "hw/pci/pci_device.h"
> +#include "hw/pci/msi.h"
> +#include "qapi/error.h"
> +#include "trace.h"
> +#import <ParavirtualizedGraphics/ParavirtualizedGraphics.h>

Please add #include "qemu/osdep.h" at top and reorder according to 
"Include directives" section in: docs/devel/style.rst

> +
> +typedef struct AppleGFXPCIState {
> +    PCIDevice parent_obj;
> +
> +    AppleGFXState common;
> +} AppleGFXPCIState;
> +
> +OBJECT_DECLARE_SIMPLE_TYPE(AppleGFXPCIState, APPLE_GFX_PCI)
> +
> +static const char* apple_gfx_pci_option_rom_path = NULL;
> +
> +static void apple_gfx_init_option_rom_path(void)
> +{
> +    NSURL *option_rom_url = PGCopyOptionROMURL();
> +    const char *option_rom_path = option_rom_url.fileSystemRepresentation;
> +    if (option_rom_url.fileURL && option_rom_path != NULL) {

option_rom_path != NULL is unnecessary; NSURL.h has 
NS_HEADER_AUDIT_BEGIN(nullability, sendability), which means any 
non-annotated member is non-nullable.

> +        apple_gfx_pci_option_rom_path = g_strdup(option_rom_path);
> +    }
> +    [option_rom_url release];
> +}
> +
> +static void apple_gfx_pci_init(Object *obj)
> +{
> +    AppleGFXPCIState *s = APPLE_GFX_PCI(obj);
> +
> +    if (!apple_gfx_pci_option_rom_path) {
> +        /* Done on device not class init to avoid -daemonize ObjC fork crash */

It is unclear what "-daemonize ObjC fork crash" means. Please add more 
details.

> +        PCIDeviceClass *pci = PCI_DEVICE_CLASS(object_get_class(obj));
> +        apple_gfx_init_option_rom_path();
> +        pci->romfile = apple_gfx_pci_option_rom_path;
> +    }
> +
> +    apple_gfx_common_init(obj, &s->common, TYPE_APPLE_GFX_PCI);
> +}
> +
> +static void apple_gfx_pci_interrupt(PCIDevice *dev, AppleGFXPCIState *s,

s is unused.

> +                                    uint32_t vector)
> +{
> +    bool msi_ok;
> +    trace_apple_gfx_raise_irq(vector);
> +
> +    msi_ok = msi_enabled(dev);
> +    if (msi_ok) {
> +        msi_notify(dev, vector);
> +    }
> +}
> +
> +static void apple_gfx_pci_realize(PCIDevice *dev, Error **errp)
> +{
> +    AppleGFXPCIState *s = APPLE_GFX_PCI(dev);
> +    Error *err = NULL;
> +    int ret;
> +
> +    pci_register_bar(dev, PG_PCI_BAR_MMIO,
> +                     PCI_BASE_ADDRESS_SPACE_MEMORY, &s->common.iomem_gfx);
> +
> +    ret = msi_init(dev, 0x0 /* config offset; 0 = find space */,
> +                   PG_PCI_MAX_MSI_VECTORS, true /* msi64bit */,
> +                   false /*msi_per_vector_mask*/, &err);
> +    if (ret != 0) {
> +        error_propagate(errp, err);

You can just pass errp to msi_init().

> +        return;
> +    }
> +
> +    @autoreleasepool {
> +        PGDeviceDescriptor *desc = [PGDeviceDescriptor new];
> +        desc.raiseInterrupt = ^(uint32_t vector) {
> +            apple_gfx_pci_interrupt(dev, s, vector);
> +        };
> +
> +        apple_gfx_common_realize(&s->common, desc);
> +        [desc release];
> +        desc = nil;
> +    }
> +}
> +
> +static void apple_gfx_pci_reset(Object *obj, ResetType type)
> +{
> +    AppleGFXPCIState *s = APPLE_GFX_PCI(obj);
> +    [s->common.pgdev reset];
> +}
> +
> +static void apple_gfx_pci_class_init(ObjectClass *klass, void *data)
> +{
> +    DeviceClass *dc = DEVICE_CLASS(klass);
> +    PCIDeviceClass *pci = PCI_DEVICE_CLASS(klass);
> +    ResettableClass *rc = RESETTABLE_CLASS(klass);
> +
> +    assert(rc->phases.hold == NULL);
> +    rc->phases.hold = apple_gfx_pci_reset;
> +    dc->desc = "macOS Paravirtualized Graphics PCI Display Controller";
> +    dc->hotpluggable = false;
> +    set_bit(DEVICE_CATEGORY_DISPLAY, dc->categories);
> +
> +    pci->vendor_id = PG_PCI_VENDOR_ID;
> +    pci->device_id = PG_PCI_DEVICE_ID;
> +    pci->class_id = PCI_CLASS_DISPLAY_OTHER;
> +    pci->realize = apple_gfx_pci_realize;
> +
> +    // TODO: Property for setting mode list
> +}
> +
> +static TypeInfo apple_gfx_pci_types[] = {
> +    {
> +        .name          = TYPE_APPLE_GFX_PCI,
> +        .parent        = TYPE_PCI_DEVICE,
> +        .instance_size = sizeof(AppleGFXPCIState),
> +        .class_init    = apple_gfx_pci_class_init,
> +        .instance_init = apple_gfx_pci_init,
> +        .interfaces = (InterfaceInfo[]) {
> +            { INTERFACE_PCIE_DEVICE },
> +            { },
> +        },
> +    }
> +};
> +DEFINE_TYPES(apple_gfx_pci_types)
> +
> diff --git a/hw/display/meson.build b/hw/display/meson.build
> index 70d855749c0..ceb7bb07612 100644
> --- a/hw/display/meson.build
> +++ b/hw/display/meson.build
> @@ -67,6 +67,7 @@ system_ss.add(when: 'CONFIG_ATI_VGA', if_true: [files('ati.c', 'ati_2d.c', 'ati_
>   
>   system_ss.add(when: 'CONFIG_MAC_PVG',         if_true: [files('apple-gfx.m'), pvg, metal])
>   system_ss.add(when: 'CONFIG_MAC_PVG_VMAPPLE', if_true: [files('apple-gfx-vmapple.m'), pvg, metal])
> +system_ss.add(when: 'CONFIG_MAC_PVG_PCI',     if_true: [files('apple-gfx-pci.m'), pvg, metal])
>   
>   if config_all_devices.has_key('CONFIG_VIRTIO_GPU')
>     virtio_gpu_ss = ss.source_set()

Re: [PATCH v3 03/14] ui/cocoa: Adds non-app runloop on main thread mode

[Akihiko Odaki]

On 2024/09/28 17:57, Phil Dennis-Jordan wrote:
> Various system frameworks on macOS and other Apple platforms
> require a main runloop to be processing events on the process’s
> main thread. The Cocoa UI’s requirement to run the process as a
> Cocoa application automatically enables this runloop, but it
> can be useful to have the runloop handling events even without
> the Cocoa UI active.
> 
> This change adds a non-app runloop mode to the cocoa_main
> function. This can be requested by other code, while the Cocoa UI
> additionally enables app mode. This arrangement ensures there is
> only one qemu_main function switcheroo, and the Cocoa UI’s app
> mode requirement and other subsystems’ runloop requests don’t
> conflict with each other.

gtk and sdl need to run in the main thread so stealing the main thread 
by setting qemu_main will break them. Please investigate the possibility 
of running CFRunLoop in another thread.

> 
> The main runloop is required for the AppleGFX PV graphics device,
> so the runloop request call has been added to its initialisation.

Please move this patch before any other patches that require it.

Regards,
Akihiko Odaki

Re: [PATCH v3 01/14] hw/display/apple-gfx: Introduce ParavirtualizedGraphics.Framework support

[Phil Dennis-Jordan]

[-- Attachment #1: Type: text/plain, Size: 32831 bytes --]

Hi,

Thanks for taking a close look at this.

There are some further comments, explanations, and also a few questions
inline below. Where I've not commented, I'll just go ahead and make the
suggested change for v4.

On Tue, 1 Oct 2024 at 11:40, Akihiko Odaki <akihiko.odaki@daynix.com> wrote:

>
> > This patch implements a QEMU device that drives PVG for the VMApple
> > variant of it.
>
> I think it is better to name it MMIO variant instead of VMApple. There
> is nothing specific to VMApple in: hw/display/apple-gfx-vmapple.m
>

I mean, I don't see it being useful for anything whatsoever outside the
vmapple machine type… But I guess there's little harm in renaming it.


> > +#include "apple-gfx.h"
> > +#include "monitor/monitor.h"
> > +#include "hw/sysbus.h"
> > +#include "hw/irq.h"
> > +#include "trace.h"
> > +#import <ParavirtualizedGraphics/ParavirtualizedGraphics.h>
> > +
> > +_Static_assert(__aarch64__, "");
>
> I don't think this assertion is worthwhile. This assertion will trigger
> if you accidentally remove depends on AARCH64 from Kconfig, but I don't
> think such code change happens by accident, and there is no reason to
> believe that this assertion won't be removed in such a case.
>

As far as I'm aware the Kconfig AARCH64 dependency is for the *target*
architecture, not the *host* architecture? The static assert checks for the
latter. The PGIOSurfaceHostDeviceDescriptor type isn't available at all on
non-aarch64 macOS hosts. I've not had any luck with using this variant of
the device on x86-64 hosts simply by disabling any surface mapper code.

Incidentally, if you know of a way to depend on a specific *host*
architecture in the Kconfig, that would be even better. I couldn't spot a
way of doing that though.

> +
> > +/*
> > + * ParavirtualizedGraphics.Framework only ships header files for the PCI
> > + * variant which does not include IOSFC descriptors and host devices.
> We add
> > + * their definitions here so that we can also work with the ARM version.
> > + */
> > +typedef bool(^IOSFCRaiseInterrupt)(uint32_t vector);
> > +typedef bool(^IOSFCUnmapMemory)(
> > +    void *a, void *b, void *c, void *d, void *e, void *f);
>
> Omit dummy parameter names.
>


> > +@end
> > +
> > +typedef struct AppleGFXVmappleState {
> > +    SysBusDevice parent_obj;
> > +
> > +    AppleGFXState common;
> > +
> > +    qemu_irq irq_gfx;
> > +    qemu_irq irq_iosfc;
> > +    MemoryRegion iomem_iosfc;
> > +    PGIOSurfaceHostDevice *pgiosfc;
> > +} AppleGFXVmappleState;
> > +
> > +OBJECT_DECLARE_SIMPLE_TYPE(AppleGFXVmappleState, APPLE_GFX_VMAPPLE)
> > +
> > +
> > +static uint64_t apple_iosfc_read(void *opaque, hwaddr offset, unsigned
> size)
> > +{
> > +    AppleGFXVmappleState *s = opaque;
> > +    uint64_t res = 0;
> > +
> > +    bql_unlock();
>
> It is dangerous to unlock BQL at an arbitrary place. Instead of
> unlocking, I suggest:
> - running [s->pgiosfc mmioReadAtOffset:offset] on another thread
> - using a bottom half to request operations that require BQL from the
> thread running [s->pgiosfc mmioReadAtOffset:offset]
> - calling AIO_WAIT_WHILE() to process the bottom half and to wait for
> the completion of [s->pgiosfc mmioReadAtOffset:offset]
>

OK, I think I see what you mean, I'll try to rework things around that
pattern. Any preference on how I kick off the job on the other thread? As
we necessarily need to use libdispatch in a bunch of places in this code
anyway, I guess dispatch_async() would probably be the simplest?


> > +    res = [s->pgiosfc mmioReadAtOffset:offset];
> > +    bql_lock();
> > +
> > +    trace_apple_iosfc_read(offset, res);
> > +
> > +    return res;
> > +}
> > +
> > +static void apple_iosfc_write(
> > +    void *opaque, hwaddr offset, uint64_t val, unsigned size)
> > +{
> > +    AppleGFXVmappleState *s = opaque;
> > +
> > +    trace_apple_iosfc_write(offset, val);
> > +
> > +    [s->pgiosfc mmioWriteAtOffset:offset value:val];
> > +}
> > +
> > +static const MemoryRegionOps apple_iosfc_ops = {
> > +    .read = apple_iosfc_read,
> > +    .write = apple_iosfc_write,
> > +    .endianness = DEVICE_LITTLE_ENDIAN,
> > +    .valid = {
> > +        .min_access_size = 4,
> > +        .max_access_size = 8,
> > +    },
> > +    .impl = {
> > +        .min_access_size = 4,
> > +        .max_access_size = 8,
> > +    },
> > +};
> > +
> > +static PGIOSurfaceHostDevice *apple_gfx_prepare_iosurface_host_device(
> > +    AppleGFXVmappleState *s)
> > +{
> > +    PGIOSurfaceHostDeviceDescriptor *iosfc_desc =
> > +        [PGIOSurfaceHostDeviceDescriptor new];
> > +    PGIOSurfaceHostDevice *iosfc_host_dev = nil;
> > +
> > +    iosfc_desc.mapMemory =
> > +        ^(uint64_t phys, uint64_t len, bool ro, void **va, void *e,
> void *f) {
> > +            trace_apple_iosfc_map_memory(phys, len, ro, va, e, f);
> > +            MemoryRegion *tmp_mr;
> > +            *va = gpa2hva(&tmp_mr, phys, len, NULL);
>
> Use: dma_memory_map()
>
>
That doesn't seem to be a precisely equivalent operation. It also says in
its headerdoc,

> Use only for reads OR writes - not for read-modify-write operations.

which I don't think we can guarantee here at all.

I guess I can call it twice, once for writing and once for reading, but
given that the dma_memory_unmap operation marks the area dirty, I'm not
it's intended for what I understand the use case here to be: As far as I
can tell, the PV graphics device uses (some) of this memory to exchange
data in a cache-coherent way between host and guest, e.g. as a lock-free
ring buffer, using atomic operations as necessary. (This works because it's
a PV device: it "knows" the other end just another CPU core (or even the
same one) executing in a different hypervisor context.) This doesn't really
match "traditional" DMA patterns where there's either a read or a write
happening.

Hunting around some more for alternative APIs, there's also
memory_region_get_ram_ptr(), but I'm not sure its restrictions apply here
either.

> +            return (bool)true;
>
> Why cast?
>

Good question. Not originally my code, so I've fixed all the instances I
could find now.


> > +        };
> > +
> > +    iosfc_desc.unmapMemory =
> > +        ^(void *a, void *b, void *c, void *d, void *e, void *f) {
> > +            trace_apple_iosfc_unmap_memory(a, b, c, d, e, f);
> > +            return (bool)true;
> > +        };
> > +
> > +    iosfc_desc.raiseInterrupt = ^(uint32_t vector) {
> > +        trace_apple_iosfc_raise_irq(vector);
> > +        bool locked = bql_locked();
> > +        if (!locked) {
> > +            bql_lock();
> > +        }
>  > +        qemu_irq_pulse(s->irq_iosfc);> +        if (!locked) {
> > +            bql_unlock();
> > +        }
> > +        return (bool)true;
> > +    };
> > +
> > +    iosfc_host_dev =
> > +        [[PGIOSurfaceHostDevice alloc] initWithDescriptor:iosfc_desc];
> > +    [iosfc_desc release];
> > +    return iosfc_host_dev;
> > +}
> > +
> > +static void apple_gfx_vmapple_realize(DeviceState *dev, Error **errp)
> > +{
> > +    @autoreleasepool {
>
> This autoreleasepool is not used.
>

It is definitely used inside the apple_gfx_common_realize() call. It's also
impossible to say whether [PGDeviceDescriptor new] uses autorelease
semantics internally, so it seemed safer to wrap the whole thing in an
outer pool.

> +        AppleGFXVmappleState *s = APPLE_GFX_VMAPPLE(dev);
> > +
> > +        PGDeviceDescriptor *desc = [PGDeviceDescriptor new];
> > +        desc.usingIOSurfaceMapper = true;
> > +        desc.raiseInterrupt = ^(uint32_t vector) {
> > +            bool locked;
> > +
> > +            trace_apple_gfx_raise_irq(vector);
> > +            locked = bql_locked();
> > +            if (!locked) {
> > +                bql_lock();
> > +            }
> > +            qemu_irq_pulse(s->irq_gfx);
> > +            if (!locked) {
> > +                bql_unlock();
> > +            }
> > +        };
> > +
> > +        s->pgiosfc = apple_gfx_prepare_iosurface_host_device(s);
> > +
> > +        apple_gfx_common_realize(&s->common, desc);
> > +        [desc release];
> > +        desc = nil;
> > +    }
> > +}
> > +
>
>

> > +++ b/hw/display/apple-gfx.h
> > @@ -0,0 +1,57 @@
> > +#ifndef QEMU_APPLE_GFX_H
> > +#define QEMU_APPLE_GFX_H
> > +
> > +#define TYPE_APPLE_GFX_VMAPPLE      "apple-gfx-vmapple"
> > +#define TYPE_APPLE_GFX_PCI          "apple-gfx-pci"
> > +
> > +#include "qemu/typedefs.h"
> > +
> > +typedef struct AppleGFXState AppleGFXState;
> > +
> > +void apple_gfx_common_init(Object *obj, AppleGFXState *s, const char*
> obj_name);
> > +
> > +#ifdef __OBJC__
>
> This ifdef is unnecessary.
>

Ah indeed; at one point, vmapple.c was #including this file, but that's no
longer necessary.


>
> > diff --git a/hw/display/apple-gfx.m b/hw/display/apple-gfx.m
> > new file mode 100644
> > index 00000000000..837300f9cd4
> > --- /dev/null
> > +++ b/hw/display/apple-gfx.m
> > @@ -0,0 +1,536 @@
> > +/*
> > + * QEMU Apple ParavirtualizedGraphics.framework device
> > + *
> > + * Copyright © 2023 Amazon.com, Inc. or its affiliates. All Rights
> Reserved.
> > + *
> > + * This work is licensed under the terms of the GNU GPL, version 2 or
> later.
> > + * See the COPYING file in the top-level directory.
> > + *
> > + * ParavirtualizedGraphics.framework is a set of libraries that macOS
> provides
> > + * which implements 3d graphics passthrough to the host as well as a
> > + * proprietary guest communication channel to drive it. This device
> model
> > + * implements support to drive that library from within QEMU.
> > + */
> > +
> > +#include "apple-gfx.h"
> > +#include "trace.h"
> > +#include "qemu/main-loop.h"
> > +#include "ui/console.h"
> > +#include "monitor/monitor.h"
> > +#include "qapi/error.h"
> > +#include "migration/blocker.h"
> > +#include <mach/mach_vm.h>
> > +#import <ParavirtualizedGraphics/ParavirtualizedGraphics.h>
> > +
> > +static const PGDisplayCoord_t apple_gfx_modes[] = {
> > +    { .x = 1440, .y = 1080 },
> > +    { .x = 1280, .y = 1024 },
> > +};
> > +
> > +typedef struct PGTask_s { // Name matches forward declaration in PG
> header
>
> Let's name it AppleGFXTask. It is a common practice to have the same tag
> name and typedef in QEMU.
>

This is defining a forward-declared type from framework headers which is
opaque from the framework's point of view. We do not get to choose its
struct name. The alternative is having casts wherever these objects are
being passed between our code and the framework. (See the original v1/v2
vmapple patch series for how messy this is.)


> > +static void apple_gfx_render_new_frame(AppleGFXState *s)
> > +{
> > +    BOOL r;
> > +    void *vram = s->vram;
> > +    uint32_t width = surface_width(s->surface);
> > +    uint32_t height = surface_height(s->surface);
> > +    MTLRegion region = MTLRegionMake2D(0, 0, width, height);
> > +    id<MTLCommandBuffer> command_buffer = [s->mtl_queue commandBuffer];
> > +    id<MTLTexture> texture = s->texture;
> > +    r = [s->pgdisp encodeCurrentFrameToCommandBuffer:command_buffer
> > +                                             texture:texture
> > +                                              region:region];
> > +    if (!r) {
> > +        return;
> > +    }
> > +    [texture retain];
> > +    if (s->using_managed_texture_storage) {
> > +        /* "Managed" textures exist in both VRAM and RAM and must be
> synced. */
> > +        id<MTLBlitCommandEncoder> blit = [command_buffer
> blitCommandEncoder];
> > +        [blit synchronizeResource:texture];
> > +        [blit endEncoding];
> > +    }
> > +    [command_buffer retain];
>
> I don't think this call of retain is necessary. The completion handler
> gets the command buffer via parameter, which implies the command buffer
> is automatically retained until the completion handler finishes.
>

I couldn't find any hard guarantees for this documented anywhere, hence
erring on the side of caution. I think the harm of an additional
retain/release pair here is quite minor.


> > +static void apple_gfx_render_frame_completed(AppleGFXState *s, void
> *vram,
> > +                                             id<MTLTexture> texture)
> > +{
> > +    --s->pending_frames;
> > +    assert(s->pending_frames >= 0);
> > +
> > +    if (vram != s->vram) {
> > +        /* Display mode has changed, drop this old frame. */
> > +        assert(texture != s->texture);
> > +        g_free(vram);
>
> This management of buffers looks a bit convoluted. I suggest remembering
> the width and height instead of pointers and comparing them. This way
> you can free resources in set_mode().
>

Yeah, I suppose that works, I can change that around.

> +    } else {
> > +        copy_mtl_texture_to_surface_mem(texture, vram);
>
> Writing vram outside BQL may result in tearing.
>

As far as I can tell(*), QXL does the same. I couldn't find any examples of
double-buffering in any of the existing display devices, which would be the
only way to do async updates efficiently and without tearing. In any case,
this solution is still vastly better than a regular VGA device, which
suffers from very visible tearing with macOS on the guest side anyway. And
in an ideal world, we'd pass through the rendered texture directly to the
Cocoa UI code rather than copying out only for the CPU to draw it back into
a window surface which is then passed to the GPU for host side rendering.
But I felt this patch is already very, very large, and if anyone cares, we
can fix imperfections in subsequent updates.

(*)The rendering code in that device is also fairly complex, so I may be
misreading it.


> > +        if (s->gfx_update_requested) {
> > +            s->gfx_update_requested = false;
> > +            dpy_gfx_update_full(s->con);
> > +            graphic_hw_update_done(s->con);
>  > +            s->new_frame_ready = false;
>
> This assignment is unnecessary as s->new_frame_ready is always false
> when s->gfx_update_requested. If you want to make sure
> s->gfx_update_requested and s->new_frame_ready are mutually exclusive,
> use one enum value instead of having two bools.
>

I'll need to refresh my memory and get back to you on this one, it's been
so many months since I actively worked on this code.


> > +        } else {
> > +            s->new_frame_ready = true;
> > +        }
> > +    }
> > +    if (s->pending_frames > 0) {
> > +        apple_gfx_render_new_frame(s);
> > +    }
> > +}
> > +
> > +static void apple_gfx_fb_update_display(void *opaque)
> > +{
> > +    AppleGFXState *s = opaque;
> > +
> > +    dispatch_async(s->render_queue, ^{
> > +        if (s->pending_frames > 0) {
>
> It should check for s->new_frame_ready as
> apple_gfx_render_frame_completed() doesn't check if
> s->pending_frames > 0 before calling graphic_hw_update_done(), which is
> inconsistent.
>

pending_frames is about guest-side frames that are queued to be rendered by
the host GPU.
new_frame_ready being true indicates that the contents of the Qemu console
surface has been updated with new frame data since the last gfx_update.
gfx_update_requested indicates that gfx_update is currently awaiting an
async completion (graphic_hw_update_done) but the surface has not received
a new frame content, but the GPU is stily busy drawing one.

apple_gfx_render_frame_completed is scheduled exactly once per pending
frame, so pending_frames > 0 is an invariant there. (Hence the assert.)

I don't think there is any inconsistency here, but I'll double check. It's
possible that there's an easier way to express the state machine, and I'll
take a look at that.



> Checking if s->pending_frames > 0 both in apple_gfx_fb_update_display()
> and apple_gfx_render_frame_completed() is also problematic as that can
> defer graphic_hw_update_done() indefinitely if we are getting new frames
> too fast.
>

I see what you mean about this part. I'll have to test it, but I guess we
should reverse the priority, like this:

        if (s->new_frame_ready) {
            dpy_gfx_update_full(s->con);
            s->new_frame_ready = false;
            graphic_hw_update_done(s->con);
        } else if (s->pending_frames > 0) {
            s->gfx_update_requested = true;
        } else {
            graphic_hw_update_done(s->con);
        }

1. If we already have a frame, ready to be displayed, return it immediately.
2. If the guest has reported that it's completed a frame and the GPU is
currently busy rendering it, defer graphic_hw_update_done until it's done.
3. If the guest reports no changes to its display, indicate this back to
Qemu as a no-op display update graphic_hw_update_done() with no
dpy_gfx_update* call.



> > +            s->gfx_update_requested = true;
> > +        } else {
> > +            if (s->new_frame_ready) {
> > +                dpy_gfx_update_full(s->con);
> > +                s->new_frame_ready = false;
> > +            }
> > +            graphic_hw_update_done(s->con);
>  > +        }> +    });
> > +}
> > +
> > +static const GraphicHwOps apple_gfx_fb_ops = {
> > +    .gfx_update = apple_gfx_fb_update_display,
> > +    .gfx_update_async = true,
> > +};
> > +
> > +static void update_cursor(AppleGFXState *s)
> > +{
> > +    dpy_mouse_set(s->con, s->pgdisp.cursorPosition.x,
> > +                  s->pgdisp.cursorPosition.y, s->cursor_show);
> > +}
> > +
> > +static void set_mode(AppleGFXState *s, uint32_t width, uint32_t height)
> > +{
> > +    void *vram = NULL;
> > +    DisplaySurface *surface;
> > +    MTLTextureDescriptor *textureDescriptor;
> > +    id<MTLTexture> texture = nil;
> > +    __block bool no_change = false;
> > +
> > +    dispatch_sync(s->render_queue,
>
> Calling dispatch_sync() while holding BQL may result in deadlock.
>

Only if any code executed on the same dispatch queue acquires the lock
either directly or transitively. I believe I have ensure this is not done
on the reqnder_queue, have you found anywhere this is the case?


> > +        ^{
> > +            if (s->surface &&
> > +                width == surface_width(s->surface) &&
> > +                height == surface_height(s->surface)) {
> > +                no_change = true;
> > +            }
> > +        });
> > +
> > +    if (no_change) {
> > +        return;
> > +    }
> > +
> > +    vram = g_malloc0(width * height * 4);
> > +    surface = qemu_create_displaysurface_from(width, height,
> PIXMAN_LE_a8r8g8b8,
> > +                                              width * 4, vram);
> > +
> > +    @autoreleasepool {
> > +        textureDescriptor =
> > +            [MTLTextureDescriptor
> > +
> texture2DDescriptorWithPixelFormat:MTLPixelFormatBGRA8Unorm
> > +                                             width:width
> > +                                            height:height
> > +                                         mipmapped:NO];
> > +        textureDescriptor.usage = s->pgdisp.minimumTextureUsage;
> > +        texture = [s->mtl newTextureWithDescriptor:textureDescriptor];
> > +    }
> > +
> > +    s->using_managed_texture_storage =
> > +        (texture.storageMode == MTLStorageModeManaged);
> > +
> > +    dispatch_sync(s->render_queue,
> > +        ^{
> > +            id<MTLTexture> old_texture = nil;
> > +            void *old_vram = s->vram;
> > +            s->vram = vram;
> > +            s->surface = surface;
> > +
> > +            dpy_gfx_replace_surface(s->con, surface);
> > +
> > +            old_texture = s->texture;
> > +            s->texture = texture;
> > +            [old_texture release];
>
> You can just do:
> [s->texture release];
> s->texture = texture;
>
> This will make s->texture dangling between the two statements, but that
> don't matter since s->texture is not an atomic variable that can be
> safely observed from another thread anyway.
>
> > +
> > +            if (s->pending_frames == 0) {
> > +                g_free(old_vram);
> > +            }
> > +        });
> > +}
> > +
> > +static void create_fb(AppleGFXState *s)
> > +{
> > +    s->con = graphic_console_init(NULL, 0, &apple_gfx_fb_ops, s);
> > +    set_mode(s, 1440, 1080);
> > +
> > +    s->cursor_show = true;
> > +}
> > +
> > +static size_t apple_gfx_get_default_mmio_range_size(void)
> > +{
> > +    size_t mmio_range_size;
> > +    @autoreleasepool {
> > +        PGDeviceDescriptor *desc = [PGDeviceDescriptor new];
> > +        mmio_range_size = desc.mmioLength;
> > +        [desc release];
> > +    }
> > +    return mmio_range_size;
> > +}
> > +
> > +void apple_gfx_common_init(Object *obj, AppleGFXState *s, const char*
> obj_name)
>
> This function can be merged into apple_gfx_common_realize().
>

Probably. I'll try it.


> > +{
> > +    Error *local_err = NULL;
> > +    int r;
> > +    size_t mmio_range_size = apple_gfx_get_default_mmio_range_size();
> > +
> > +    trace_apple_gfx_common_init(obj_name, mmio_range_size);
> > +    memory_region_init_io(&s->iomem_gfx, obj, &apple_gfx_ops, s,
> obj_name,
> > +                          mmio_range_size);
> > +    s->iomem_gfx.disable_reentrancy_guard = true;
>
> Why do you disable reentrancy guard?
>

Perhaps with the proposed AIO_WAIT_WHILE based I/O scheme, this won't be an
issue anymore, but the guard would otherwise keep dropping MMIOs which
immediately caused the PV graphics device to stop making progress. The MMIO
APIs in the PVG framework are thread- and reentrancy-safe, so we certainly
don't need to serialise them on our side.


> > +
> > +    /* TODO: PVG framework supports serialising device state: integrate
> it! */
> > +    if (apple_gfx_mig_blocker == NULL) {
> > +        error_setg(&apple_gfx_mig_blocker,
> > +                  "Migration state blocked by apple-gfx display
> device");
> > +        r = migrate_add_blocker(&apple_gfx_mig_blocker, &local_err);
> > +        if (r < 0) {
> > +            error_report_err(local_err);
>
> Please report the error to the caller of apple_gfx_common_realize()
> instead.
>
> > +        }
> > +    }
>  > +}> +
> > +static void apple_gfx_register_task_mapping_handlers(AppleGFXState *s,
> > +                                                     PGDeviceDescriptor
> *desc)
> > +{
> > +    desc.createTask = ^(uint64_t vmSize, void * _Nullable * _Nonnull
> baseAddress) {
> > +        AppleGFXTask *task = apple_gfx_new_task(s, vmSize);
> > +        *baseAddress = (void*)task->address;
>
> nit: please write as (void *) instead of (void*).
>
> > +        trace_apple_gfx_create_task(vmSize, *baseAddress);
> > +        return task;
> > +    };
> > +
>
> > +{
> > +    PGDisplayDescriptor *disp_desc = [PGDisplayDescriptor new];
> > +
> > +    disp_desc.name = @"QEMU display";
> > +    disp_desc.sizeInMillimeters = NSMakeSize(400., 300.); /* A 20"
> display */
> > +    disp_desc.queue = dispatch_get_main_queue();
> > +    disp_desc.newFrameEventHandler = ^(void) {
> > +        trace_apple_gfx_new_frame();
> > +        dispatch_async(s->render_queue, ^{
> > +            /* Drop frames if we get too far ahead. */
> > +            if (s->pending_frames >= 2)
> > +                return;
> > +            ++s->pending_frames;
> > +            if (s->pending_frames > 1) {
> > +                return;
> > +            }
> > +            @autoreleasepool {
> > +                apple_gfx_render_new_frame(s);
> > +            }
> > +        });
> > +    };
> > +    disp_desc.modeChangeHandler = ^(PGDisplayCoord_t sizeInPixels,
> > +                                    OSType pixelFormat) {
> > +        trace_apple_gfx_mode_change(sizeInPixels.x, sizeInPixels.y);
> > +        set_mode(s, sizeInPixels.x, sizeInPixels.y);
> > +    };
> > +    disp_desc.cursorGlyphHandler = ^(NSBitmapImageRep *glyph,
> > +                                     PGDisplayCoord_t hotSpot) {
> > +        uint32_t bpp = glyph.bitsPerPixel;
> > +        size_t width = glyph.pixelsWide;
> > +        size_t height = glyph.pixelsHigh;
> > +        size_t padding_bytes_per_row = glyph.bytesPerRow - width * 4;
> > +        const uint8_t* px_data = glyph.bitmapData;
> > +
> > +        trace_apple_gfx_cursor_set(bpp, width, height);
> > +
> > +        if (s->cursor) {
> > +            cursor_unref(s->cursor);
> > +            s->cursor = NULL;
> > +        }
> > +
> > +        if (bpp == 32) { /* Shouldn't be anything else, but just to be
> safe...*/
> > +            s->cursor = cursor_alloc(width, height);
> > +            s->cursor->hot_x = hotSpot.x;
> > +            s->cursor->hot_y = hotSpot.y;
> > +
> > +            uint32_t *dest_px = s->cursor->data;
> > +
> > +            for (size_t y = 0; y < height; ++y) {
> > +                for (size_t x = 0; x < width; ++x) {
> > +                    /* NSBitmapImageRep's red & blue channels are
> swapped
> > +                     * compared to QEMUCursor's. */
> > +                    *dest_px =
> > +                        (px_data[0] << 16u) |
> > +                        (px_data[1] <<  8u) |
> > +                        (px_data[2] <<  0u) |
> > +                        (px_data[3] << 24u);
> > +                    ++dest_px;
> > +                    px_data += 4;
> > +                }
> > +                px_data += padding_bytes_per_row;
> > +            }
> > +            dpy_cursor_define(s->con, s->cursor);
> > +            update_cursor(s);
> > +        }
> > +    };
> > +    disp_desc.cursorShowHandler = ^(BOOL show) {
> > +        trace_apple_gfx_cursor_show(show);
> > +        s->cursor_show = show;
> > +        update_cursor(s);
> > +    };
> > +    disp_desc.cursorMoveHandler = ^(void) {
> > +        trace_apple_gfx_cursor_move();
> > +        update_cursor(s);
> > +    };
> > +
> > +    return disp_desc;
> > +}
> > +
> > +static NSArray<PGDisplayMode*>*
> apple_gfx_prepare_display_mode_array(void)
> > +{
> > +    PGDisplayMode *modes[ARRAY_SIZE(apple_gfx_modes)];
> > +    NSArray<PGDisplayMode*>* mode_array = nil;
> > +    int i;
> > +
> > +    for (i = 0; i < ARRAY_SIZE(apple_gfx_modes); i++) {
> > +        modes[i] =
> > +            [[PGDisplayMode alloc]
> initWithSizeInPixels:apple_gfx_modes[i] refreshRateInHz:60.];
> > +    }
> > +
> > +    mode_array = [NSArray arrayWithObjects:modes
> count:ARRAY_SIZE(apple_gfx_modes)];
> > +
> > +    for (i = 0; i < ARRAY_SIZE(apple_gfx_modes); i++) {
> > +        [modes[i] release];
> > +        modes[i] = nil;
> > +    }
> > +
> > +    return mode_array;
> > +}
> > +
> > +static id<MTLDevice> copy_suitable_metal_device(void)
> > +{
> > +    id<MTLDevice> dev = nil;
> > +    NSArray<id<MTLDevice>> *devs = MTLCopyAllDevices();
> > +
> > +    /* Prefer a unified memory GPU. Failing that, pick a non-removable
> GPU. */
> > +    for (size_t i = 0; i < devs.count; ++i) {
> > +        if (devs[i].hasUnifiedMemory) {
> > +            dev = devs[i];
> > +            break;
> > +        }
> > +        if (!devs[i].removable) {
> > +            dev = devs[i];
> > +        }
> > +    }
> > +
> > +    if (dev != nil) {
> > +        [dev retain];
> > +    } else {
> > +        dev = MTLCreateSystemDefaultDevice();
> > +    }
> > +    [devs release];
> > +
> > +    return dev;
> > +}
> > +
> > +void apple_gfx_common_realize(AppleGFXState *s, PGDeviceDescriptor
> *desc)
> > +{
> > +    PGDisplayDescriptor *disp_desc = nil;
> > +
> > +    QTAILQ_INIT(&s->tasks);
> > +    s->render_queue = dispatch_queue_create("apple-gfx.render",
> > +                                            DISPATCH_QUEUE_SERIAL);
> > +    s->mtl = copy_suitable_metal_device();
> > +    s->mtl_queue = [s->mtl newCommandQueue];
> > +
> > +    desc.device = s->mtl;
> > +
> > +    apple_gfx_register_task_mapping_handlers(s, desc);
> > +
> > +    s->pgdev = PGNewDeviceWithDescriptor(desc);
> > +
> > +    disp_desc = apple_gfx_prepare_display_handlers(s);
> > +    s->pgdisp = [s->pgdev newDisplayWithDescriptor:disp_desc
> > +                                              port:0 serialNum:1234];
> > +    [disp_desc release];
> > +    s->pgdisp.modeList = apple_gfx_prepare_display_mode_array();
> > +
> > +    create_fb(s);
> > +}
> > diff --git a/hw/display/meson.build b/hw/display/meson.build
> > index 7db05eace97..70d855749c0 100644
> > --- a/hw/display/meson.build
> > +++ b/hw/display/meson.build
> > @@ -65,6 +65,8 @@ system_ss.add(when: 'CONFIG_ARTIST', if_true:
> files('artist.c'))
> >
> >   system_ss.add(when: 'CONFIG_ATI_VGA', if_true: [files('ati.c',
> 'ati_2d.c', 'ati_dbg.c'), pixman])
> >
> > +system_ss.add(when: 'CONFIG_MAC_PVG',         if_true:
> [files('apple-gfx.m'), pvg, metal])
> > +system_ss.add(when: 'CONFIG_MAC_PVG_VMAPPLE', if_true:
> [files('apple-gfx-vmapple.m'), pvg, metal])
> >
> >   if config_all_devices.has_key('CONFIG_VIRTIO_GPU')
> >     virtio_gpu_ss = ss.source_set()
> > diff --git a/hw/display/trace-events b/hw/display/trace-events
> > index 781f8a33203..1809a358e36 100644
> > --- a/hw/display/trace-events
> > +++ b/hw/display/trace-events
> > @@ -191,3 +191,29 @@ dm163_bits_ppi(unsigned dest_width) "dest_width :
> %u"
> >   dm163_leds(int led, uint32_t value) "led %d: 0x%x"
> >   dm163_channels(int channel, uint8_t value) "channel %d: 0x%x"
> >   dm163_refresh_rate(uint32_t rr) "refresh rate %d"
> > +
> > +# apple-gfx.m
> > +apple_gfx_read(uint64_t offset, uint64_t res) "offset=0x%"PRIx64"
> res=0x%"PRIx64
> > +apple_gfx_write(uint64_t offset, uint64_t val) "offset=0x%"PRIx64"
> val=0x%"PRIx64
> > +apple_gfx_create_task(uint32_t vm_size, void *va) "vm_size=0x%x
> base_addr=%p"
> > +apple_gfx_destroy_task(void *task) "task=%p"
> > +apple_gfx_map_memory(void *task, uint32_t range_count, uint64_t
> virtual_offset, uint32_t read_only) "task=%p range_count=0x%x
> virtual_offset=0x%"PRIx64" read_only=%d"
> > +apple_gfx_map_memory_range(uint32_t i, uint64_t phys_addr, uint64_t
> phys_len) "[%d] phys_addr=0x%"PRIx64" phys_len=0x%"PRIx64
> > +apple_gfx_remap(uint64_t retval, uint64_t source, uint64_t target)
> "retval=%"PRId64" source=0x%"PRIx64" target=0x%"PRIx64
> > +apple_gfx_unmap_memory(void *task, uint64_t virtual_offset, uint64_t
> length) "task=%p virtual_offset=0x%"PRIx64" length=0x%"PRIx64
> > +apple_gfx_read_memory(uint64_t phys_address, uint64_t length, void
> *dst) "phys_addr=0x%"PRIx64" length=0x%"PRIx64" dest=%p"
> > +apple_gfx_raise_irq(uint32_t vector) "vector=0x%x"
> > +apple_gfx_new_frame(void) ""
> > +apple_gfx_mode_change(uint64_t x, uint64_t y) "x=%"PRId64" y=%"PRId64
> > +apple_gfx_cursor_set(uint32_t bpp, uint64_t width, uint64_t height)
> "bpp=%d width=%"PRId64" height=0x%"PRId64
> > +apple_gfx_cursor_show(uint32_t show) "show=%d"
> > +apple_gfx_cursor_move(void) ""
> > +apple_gfx_common_init(const char *device_name, size_t mmio_size)
> "device: %s; MMIO size: %zu bytes"
> > +
> > +# apple-gfx-vmapple.m
> > +apple_iosfc_read(uint64_t offset, uint64_t res) "offset=0x%"PRIx64"
> res=0x%"PRIx64
> > +apple_iosfc_write(uint64_t offset, uint64_t val) "offset=0x%"PRIx64"
> val=0x%"PRIx64
> > +apple_iosfc_map_memory(uint64_t phys, uint64_t len, uint32_t ro, void
> *va, void *e, void *f) "phys=0x%"PRIx64" len=0x%"PRIx64" ro=%d va=%p e=%p
> f=%p"
> > +apple_iosfc_unmap_memory(void *a, void *b, void *c, void *d, void *e,
> void *f) "a=%p b=%p c=%p d=%p e=%p f=%p"
> > +apple_iosfc_raise_irq(uint32_t vector) "vector=0x%x"
> > +
> > diff --git a/meson.build b/meson.build
> > index 10464466ff3..f09df3f09d5 100644
> > --- a/meson.build
> > +++ b/meson.build
> > @@ -741,6 +741,8 @@ socket = []
> >   version_res = []
> >   coref = []
> >   iokit = []
> > +pvg = []
> > +metal = []
> >   emulator_link_args = []
> >   midl = not_found
> >   widl = not_found
> > @@ -762,6 +764,8 @@ elif host_os == 'darwin'
> >     coref = dependency('appleframeworks', modules: 'CoreFoundation')
> >     iokit = dependency('appleframeworks', modules: 'IOKit', required:
> false)
> >     host_dsosuf = '.dylib'
> > +  pvg = dependency('appleframeworks', modules:
> 'ParavirtualizedGraphics')
> > +  metal = dependency('appleframeworks', modules: 'Metal')
> >   elif host_os == 'sunos'
> >     socket = [cc.find_library('socket'),
> >               cc.find_library('nsl'),
>
>

[-- Attachment #2: Type: text/html, Size: 43057 bytes --]

Re: [PATCH v3 02/14] hw/display/apple-gfx: Adds PCI implementation

[Phil Dennis-Jordan]

[-- Attachment #1: Type: text/plain, Size: 8403 bytes --]

On Wed, 2 Oct 2024 at 09:14, Akihiko Odaki <akihiko.odaki@daynix.com> wrote:

> On 2024/09/28 17:57, Phil Dennis-Jordan wrote:
> > This change wires up the PCI variant of the paravirtualised
> > graphics device, mainly useful for x86-64 macOS guests, implemented
> > by macOS's ParavirtualizedGraphics.framework. It builds on code
> > shared with the vmapple/mmio variant of the PVG device.
> >
> > Signed-off-by: Phil Dennis-Jordan <phil@philjordan.eu>
> > ---
> >   hw/display/Kconfig         |   5 ++
> >   hw/display/apple-gfx-pci.m | 138 +++++++++++++++++++++++++++++++++++++
> >   hw/display/meson.build     |   1 +
> >   3 files changed, 144 insertions(+)
> >   create mode 100644 hw/display/apple-gfx-pci.m
> >
> > diff --git a/hw/display/Kconfig b/hw/display/Kconfig
> > index 179a479d220..c2ec268f8e9 100644
> > --- a/hw/display/Kconfig
> > +++ b/hw/display/Kconfig
> > @@ -152,3 +152,8 @@ config MAC_PVG_VMAPPLE
> >       bool
> >       depends on MAC_PVG
> >       depends on ARM
> > +
> > +config MAC_PVG_PCI
> > +    bool
> > +    depends on MAC_PVG && PCI
> > +    default y if PCI_DEVICES
> > diff --git a/hw/display/apple-gfx-pci.m b/hw/display/apple-gfx-pci.m
> > new file mode 100644
> > index 00000000000..9370258ee46
> > --- /dev/null
> > +++ b/hw/display/apple-gfx-pci.m
> > @@ -0,0 +1,138 @@
> > +/*
> > + * QEMU Apple ParavirtualizedGraphics.framework device, PCI variant
> > + *
> > + * Copyright © 2023-2024 Phil Dennis-Jordan
> > + *
> > + * This work is licensed under the terms of the GNU GPL, version 2 or
> later.
> > + * See the COPYING file in the top-level directory.
>
> Please use SPDX-License-Identifier instead.
>
> > + *
> > + * ParavirtualizedGraphics.framework is a set of libraries that macOS
> provides
> > + * which implements 3d graphics passthrough to the host as well as a
> > + * proprietary guest communication channel to drive it. This device
> model
> > + * implements support to drive that library from within QEMU as a PCI
> device
> > + * aimed primarily at x86-64 macOS VMs.
> > + */
> > +
> > +#include "apple-gfx.h"
> > +#include "hw/pci/pci_device.h"
> > +#include "hw/pci/msi.h"
> > +#include "qapi/error.h"
> > +#include "trace.h"
> > +#import <ParavirtualizedGraphics/ParavirtualizedGraphics.h>
>
> Please add #include "qemu/osdep.h" at top and reorder according to
> "Include directives" section in: docs/devel/style.rst
>
> > +
> > +typedef struct AppleGFXPCIState {
> > +    PCIDevice parent_obj;
> > +
> > +    AppleGFXState common;
> > +} AppleGFXPCIState;
> > +
> > +OBJECT_DECLARE_SIMPLE_TYPE(AppleGFXPCIState, APPLE_GFX_PCI)
> > +
> > +static const char* apple_gfx_pci_option_rom_path = NULL;
> > +
> > +static void apple_gfx_init_option_rom_path(void)
> > +{
> > +    NSURL *option_rom_url = PGCopyOptionROMURL();
> > +    const char *option_rom_path =
> option_rom_url.fileSystemRepresentation;
> > +    if (option_rom_url.fileURL && option_rom_path != NULL) {
>
> option_rom_path != NULL is unnecessary; NSURL.h has
> NS_HEADER_AUDIT_BEGIN(nullability, sendability), which means any
> non-annotated member is non-nullable.
>
> > +        apple_gfx_pci_option_rom_path = g_strdup(option_rom_path);
> > +    }
> > +    [option_rom_url release];
> > +}
> > +
> > +static void apple_gfx_pci_init(Object *obj)
> > +{
> > +    AppleGFXPCIState *s = APPLE_GFX_PCI(obj);
> > +
> > +    if (!apple_gfx_pci_option_rom_path) {
> > +        /* Done on device not class init to avoid -daemonize ObjC fork
> crash */
>
> It is unclear what "-daemonize ObjC fork crash" means. Please add more
> details.
>

When libvirt starts up a VM, it runs Qemu once in a dry-run state with the
-daemonize command line argument without actually running the VM. It then
queries various information about available features. It then shuts down
Qemu and re-runs it without -daemonize and actually lets the VM run.

The -daemonize command line argument causes fork() to be called. Apple's
Objective-C runtime does not support fork()ing and will immediately crash
the process when it detects it happening. QOM class init happens before the
fork(), so you can't call any Objective-C code from the class init
functions without causing a crash with -daemonize.

I can expand the comment somewhat for clarity.


> > +        PCIDeviceClass *pci = PCI_DEVICE_CLASS(object_get_class(obj));
> > +        apple_gfx_init_option_rom_path();
> > +        pci->romfile = apple_gfx_pci_option_rom_path;
> > +    }
> > +
> > +    apple_gfx_common_init(obj, &s->common, TYPE_APPLE_GFX_PCI);
> > +}
> > +
> > +static void apple_gfx_pci_interrupt(PCIDevice *dev, AppleGFXPCIState *s,
>
> s is unused.
>

Hmm, I suppose I should probably have a fallback legacy PCI IRQ trigger
here when MSI is not available.


> > +                                    uint32_t vector)
> > +{
> > +    bool msi_ok;
> > +    trace_apple_gfx_raise_irq(vector);
> > +
> > +    msi_ok = msi_enabled(dev);
> > +    if (msi_ok) {
> > +        msi_notify(dev, vector);
> > +    }
> > +}
> > +
> > +static void apple_gfx_pci_realize(PCIDevice *dev, Error **errp)
> > +{
> > +    AppleGFXPCIState *s = APPLE_GFX_PCI(dev);
> > +    Error *err = NULL;
> > +    int ret;
> > +
> > +    pci_register_bar(dev, PG_PCI_BAR_MMIO,
> > +                     PCI_BASE_ADDRESS_SPACE_MEMORY,
> &s->common.iomem_gfx);
> > +
> > +    ret = msi_init(dev, 0x0 /* config offset; 0 = find space */,
> > +                   PG_PCI_MAX_MSI_VECTORS, true /* msi64bit */,
> > +                   false /*msi_per_vector_mask*/, &err);
> > +    if (ret != 0) {
> > +        error_propagate(errp, err);
>
> You can just pass errp to msi_init().
>
> > +        return;
> > +    }
> > +
> > +    @autoreleasepool {
> > +        PGDeviceDescriptor *desc = [PGDeviceDescriptor new];
> > +        desc.raiseInterrupt = ^(uint32_t vector) {
> > +            apple_gfx_pci_interrupt(dev, s, vector);
> > +        };
> > +
> > +        apple_gfx_common_realize(&s->common, desc);
> > +        [desc release];
> > +        desc = nil;
> > +    }
> > +}
> > +
> > +static void apple_gfx_pci_reset(Object *obj, ResetType type)
> > +{
> > +    AppleGFXPCIState *s = APPLE_GFX_PCI(obj);
> > +    [s->common.pgdev reset];
> > +}
> > +
> > +static void apple_gfx_pci_class_init(ObjectClass *klass, void *data)
> > +{
> > +    DeviceClass *dc = DEVICE_CLASS(klass);
> > +    PCIDeviceClass *pci = PCI_DEVICE_CLASS(klass);
> > +    ResettableClass *rc = RESETTABLE_CLASS(klass);
> > +
> > +    assert(rc->phases.hold == NULL);
> > +    rc->phases.hold = apple_gfx_pci_reset;
> > +    dc->desc = "macOS Paravirtualized Graphics PCI Display Controller";
> > +    dc->hotpluggable = false;
> > +    set_bit(DEVICE_CATEGORY_DISPLAY, dc->categories);
> > +
> > +    pci->vendor_id = PG_PCI_VENDOR_ID;
> > +    pci->device_id = PG_PCI_DEVICE_ID;
> > +    pci->class_id = PCI_CLASS_DISPLAY_OTHER;
> > +    pci->realize = apple_gfx_pci_realize;
> > +
> > +    // TODO: Property for setting mode list
> > +}
> > +
> > +static TypeInfo apple_gfx_pci_types[] = {
> > +    {
> > +        .name          = TYPE_APPLE_GFX_PCI,
> > +        .parent        = TYPE_PCI_DEVICE,
> > +        .instance_size = sizeof(AppleGFXPCIState),
> > +        .class_init    = apple_gfx_pci_class_init,
> > +        .instance_init = apple_gfx_pci_init,
> > +        .interfaces = (InterfaceInfo[]) {
> > +            { INTERFACE_PCIE_DEVICE },
> > +            { },
> > +        },
> > +    }
> > +};
> > +DEFINE_TYPES(apple_gfx_pci_types)
> > +
> > diff --git a/hw/display/meson.build b/hw/display/meson.build
> > index 70d855749c0..ceb7bb07612 100644
> > --- a/hw/display/meson.build
> > +++ b/hw/display/meson.build
> > @@ -67,6 +67,7 @@ system_ss.add(when: 'CONFIG_ATI_VGA', if_true:
> [files('ati.c', 'ati_2d.c', 'ati_
> >
> >   system_ss.add(when: 'CONFIG_MAC_PVG',         if_true:
> [files('apple-gfx.m'), pvg, metal])
> >   system_ss.add(when: 'CONFIG_MAC_PVG_VMAPPLE', if_true:
> [files('apple-gfx-vmapple.m'), pvg, metal])
> > +system_ss.add(when: 'CONFIG_MAC_PVG_PCI',     if_true:
> [files('apple-gfx-pci.m'), pvg, metal])
> >
> >   if config_all_devices.has_key('CONFIG_VIRTIO_GPU')
> >     virtio_gpu_ss = ss.source_set()
>
>

[-- Attachment #2: Type: text/html, Size: 10657 bytes --]

Re: [PATCH v3 01/14] hw/display/apple-gfx: Introduce ParavirtualizedGraphics.Framework support

[Akihiko Odaki]

On 2024/10/02 22:33, Phil Dennis-Jordan wrote:
> Hi,
> 
> Thanks for taking a close look at this.
> 
> There are some further comments, explanations, and also a few questions 
> inline below. Where I've not commented, I'll just go ahead and make the 
> suggested change for v4.
> 
> On Tue, 1 Oct 2024 at 11:40, Akihiko Odaki <akihiko.odaki@daynix.com 
> <mailto:akihiko.odaki@daynix.com>> wrote:
> 
> 
>      > This patch implements a QEMU device that drives PVG for the VMApple
>      > variant of it.
> 
>     I think it is better to name it MMIO variant instead of VMApple. There
>     is nothing specific to VMApple in: hw/display/apple-gfx-vmapple.m
> 
> 
> I mean, I don't see it being useful for anything whatsoever outside the 
> vmapple machine type… But I guess there's little harm in renaming it.

I don't know but perhaps we may get vmapple2 or something in the future.

> 
> 
>      > +#include "apple-gfx.h"
>      > +#include "monitor/monitor.h"
>      > +#include "hw/sysbus.h"
>      > +#include "hw/irq.h"
>      > +#include "trace.h"
>      > +#import <ParavirtualizedGraphics/ParavirtualizedGraphics.h>
>      > +
>      > +_Static_assert(__aarch64__, "");
> 
>     I don't think this assertion is worthwhile. This assertion will trigger
>     if you accidentally remove depends on AARCH64 from Kconfig, but I don't
>     think such code change happens by accident, and there is no reason to
>     believe that this assertion won't be removed in such a case.
> 
> 
> As far as I'm aware the Kconfig AARCH64 dependency is for the /target/ 
> architecture, not the /host/ architecture? The static assert checks for 
> the latter. The PGIOSurfaceHostDeviceDescriptor type isn't available at 
> all on non-aarch64 macOS hosts. I've not had any luck with using this 
> variant of the device on x86-64 hosts simply by disabling any surface 
> mapper code.
> 
> Incidentally, if you know of a way to depend on a specific /host/ 
> architecture in the Kconfig, that would be even better. I couldn't spot 
> a way of doing that though.

I got your intention now. The correct way to do that is to check for cpu 
== 'aarch64'. Having assertion will break qemu-system-aarch64 on Intel Macs.

> 
>      > +
>      > +/*
>      > + * ParavirtualizedGraphics.Framework only ships header files for
>     the PCI
>      > + * variant which does not include IOSFC descriptors and host
>     devices. We add
>      > + * their definitions here so that we can also work with the ARM
>     version.
>      > + */
>      > +typedef bool(^IOSFCRaiseInterrupt)(uint32_t vector);
>      > +typedef bool(^IOSFCUnmapMemory)(
>      > +    void *a, void *b, void *c, void *d, void *e, void *f);
> 
>     Omit dummy parameter names.
> 
> 
>      > +@end
>      > +
>      > +typedef struct AppleGFXVmappleState {
>      > +    SysBusDevice parent_obj;
>      > +
>      > +    AppleGFXState common;
>      > +
>      > +    qemu_irq irq_gfx;
>      > +    qemu_irq irq_iosfc;
>      > +    MemoryRegion iomem_iosfc;
>      > +    PGIOSurfaceHostDevice *pgiosfc;
>      > +} AppleGFXVmappleState;
>      > +
>      > +OBJECT_DECLARE_SIMPLE_TYPE(AppleGFXVmappleState, APPLE_GFX_VMAPPLE)
>      > +
>      > +
>      > +static uint64_t apple_iosfc_read(void *opaque, hwaddr offset,
>     unsigned size)
>      > +{
>      > +    AppleGFXVmappleState *s = opaque;
>      > +    uint64_t res = 0;
>      > +
>      > +    bql_unlock();
> 
>     It is dangerous to unlock BQL at an arbitrary place. Instead of
>     unlocking, I suggest:
>     - running [s->pgiosfc mmioReadAtOffset:offset] on another thread
>     - using a bottom half to request operations that require BQL from the
>     thread running [s->pgiosfc mmioReadAtOffset:offset]
>     - calling AIO_WAIT_WHILE() to process the bottom half and to wait for
>     the completion of [s->pgiosfc mmioReadAtOffset:offset]
> 
> 
> OK, I think I see what you mean, I'll try to rework things around that 
> pattern. Any preference on how I kick off the job on the other thread? 
> As we necessarily need to use libdispatch in a bunch of places in this 
> code anyway, I guess dispatch_async() would probably be the simplest?

Perhaps so. The QEMU way is to use a bottom half with AioContext, but 
you can't simultaneously run a dispatch queue and AioContext in one 
thread so you have to use the dispatch queue if you need one.

> 
>      > +    res = [s->pgiosfc mmioReadAtOffset:offset];
>      > +    bql_lock();
>      > +
>      > +    trace_apple_iosfc_read(offset, res);
>      > +
>      > +    return res;
>      > +}
>      > +
>      > +static void apple_iosfc_write(
>      > +    void *opaque, hwaddr offset, uint64_t val, unsigned size)
>      > +{
>      > +    AppleGFXVmappleState *s = opaque;
>      > +
>      > +    trace_apple_iosfc_write(offset, val);
>      > +
>      > +    [s->pgiosfc mmioWriteAtOffset:offset value:val];
>      > +}
>      > +
>      > +static const MemoryRegionOps apple_iosfc_ops = {
>      > +    .read = apple_iosfc_read,
>      > +    .write = apple_iosfc_write,
>      > +    .endianness = DEVICE_LITTLE_ENDIAN,
>      > +    .valid = {
>      > +        .min_access_size = 4,
>      > +        .max_access_size = 8,
>      > +    },
>      > +    .impl = {
>      > +        .min_access_size = 4,
>      > +        .max_access_size = 8,
>      > +    },
>      > +};
>      > +
>      > +static PGIOSurfaceHostDevice
>     *apple_gfx_prepare_iosurface_host_device(
>      > +    AppleGFXVmappleState *s)
>      > +{
>      > +    PGIOSurfaceHostDeviceDescriptor *iosfc_desc =
>      > +        [PGIOSurfaceHostDeviceDescriptor new];
>      > +    PGIOSurfaceHostDevice *iosfc_host_dev = nil;
>      > +
>      > +    iosfc_desc.mapMemory =
>      > +        ^(uint64_t phys, uint64_t len, bool ro, void **va, void
>     *e, void *f) {
>      > +            trace_apple_iosfc_map_memory(phys, len, ro, va, e, f);
>      > +            MemoryRegion *tmp_mr;
>      > +            *va = gpa2hva(&tmp_mr, phys, len, NULL);
> 
>     Use: dma_memory_map()
> 
> 
> That doesn't seem to be a precisely equivalent operation. It also says 
> in its headerdoc,
> 
>     Use only for reads OR writes - not for read-modify-write operations.
> 
> which I don't think we can guarantee here at all.
> 
> I guess I can call it twice, once for writing and once for reading, but 
> given that the dma_memory_unmap operation marks the area dirty, I'm not 
> it's intended for what I understand the use case here to be: As far as I 
> can tell, the PV graphics device uses (some) of this memory to exchange 
> data in a cache-coherent way between host and guest, e.g. as a lock-free 
> ring buffer, using atomic operations as necessary. (This works because 
> it's a PV device: it "knows" the other end just another CPU core (or 
> even the same one) executing in a different hypervisor context.) This 
> doesn't really match "traditional" DMA patterns where there's either a 
> read or a write happening.

I think the story is a bit different for this VMApple variant. Probably 
the CPU and GPU in Apple Silicon is cache-coherent so you can map normal 
memory for GPU without any kind of cache maintenance.

Cache conherency of CPU and GPU in Apple Silicon is implied with Apple's 
documentation; it says you don't need to synchronize resources for 
MTLStorageModeShared, which is the default for Apple Silicon.
https://developer.apple.com/documentation/metal/resource_fundamentals/synchronizing_a_managed_resource_in_macos

The name "IOSurface" also implies it is used not only for e.g., ring 
buffer but also for real data.

> 
> Hunting around some more for alternative APIs, there's also 
> memory_region_get_ram_ptr(), but I'm not sure its restrictions apply 
> here either.

I think you can call memory_access_is_direct() to check if the 
requirement is satisfied.

It will still break dirty page tracking implemented by 
dma_memory_unmap() and others, but it's broken for hvf, which does not 
implement dirty page tracking either.

> 
>      > +            return (bool)true;
> 
>     Why cast?
> 
> 
> Good question. Not originally my code, so I've fixed all the instances I 
> could find now.
> 
>      > +        };
>      > +
>      > +    iosfc_desc.unmapMemory =
>      > +        ^(void *a, void *b, void *c, void *d, void *e, void *f) {
>      > +            trace_apple_iosfc_unmap_memory(a, b, c, d, e, f);
>      > +            return (bool)true;
>      > +        };
>      > +
>      > +    iosfc_desc.raiseInterrupt = ^(uint32_t vector) {
>      > +        trace_apple_iosfc_raise_irq(vector);
>      > +        bool locked = bql_locked();
>      > +        if (!locked) {
>      > +            bql_lock();
>      > +        }
>       > +        qemu_irq_pulse(s->irq_iosfc);> +        if (!locked) {
>      > +            bql_unlock();
>      > +        }
>      > +        return (bool)true;
>      > +    };
>      > +
>      > +    iosfc_host_dev =
>      > +        [[PGIOSurfaceHostDevice alloc]
>     initWithDescriptor:iosfc_desc];
>      > +    [iosfc_desc release];
>      > +    return iosfc_host_dev;
>      > +}
>      > +
>      > +static void apple_gfx_vmapple_realize(DeviceState *dev, Error
>     **errp)
>      > +{
>      > +    @autoreleasepool {
> 
>     This autoreleasepool is not used.
> 
> 
> It is definitely used inside the apple_gfx_common_realize() call. It's 
> also impossible to say whether [PGDeviceDescriptor new] uses autorelease 
> semantics internally, so it seemed safer to wrap the whole thing in an 
> outer pool.

Theoretically, It should be safe to assume the callee creates 
autoreleasepool by themselves as needed in general. We have bunch of 
code to call Objective-C APIs without creating autoreleasepool in the 
caller. Practically, [PGDeviceDescriptor new] is likely to be 
implemented with ARC, which wraps methods in autoreleasepool as necessary.

Functions that uses a method that returns autorelease resources should 
be wrapped with autoreleasepool instead of assuming the caller creates 
autoreleasepool for them.

> 
>      > +        AppleGFXVmappleState *s = APPLE_GFX_VMAPPLE(dev);
>      > +
>      > +        PGDeviceDescriptor *desc = [PGDeviceDescriptor new];
>      > +        desc.usingIOSurfaceMapper = true;
>      > +        desc.raiseInterrupt = ^(uint32_t vector) {
>      > +            bool locked;
>      > +
>      > +            trace_apple_gfx_raise_irq(vector);
>      > +            locked = bql_locked();
>      > +            if (!locked) {
>      > +                bql_lock();
>      > +            }
>      > +            qemu_irq_pulse(s->irq_gfx);
>      > +            if (!locked) {
>      > +                bql_unlock();
>      > +            }
>      > +        };
>      > +
>      > +        s->pgiosfc = apple_gfx_prepare_iosurface_host_device(s);
>      > +
>      > +        apple_gfx_common_realize(&s->common, desc);
>      > +        [desc release];
>      > +        desc = nil;
>      > +    }
>      > +}
>      > +
> 
>      > +++ b/hw/display/apple-gfx.h
>      > @@ -0,0 +1,57 @@
>      > +#ifndef QEMU_APPLE_GFX_H
>      > +#define QEMU_APPLE_GFX_H
>      > +
>      > +#define TYPE_APPLE_GFX_VMAPPLE      "apple-gfx-vmapple"
>      > +#define TYPE_APPLE_GFX_PCI          "apple-gfx-pci"
>      > +
>      > +#include "qemu/typedefs.h"
>      > +
>      > +typedef struct AppleGFXState AppleGFXState;
>      > +
>      > +void apple_gfx_common_init(Object *obj, AppleGFXState *s, const
>     char* obj_name);
>      > +
>      > +#ifdef __OBJC__
> 
>     This ifdef is unnecessary.
> 
> Ah indeed; at one point, vmapple.c was #including this file, but that's 
> no longer necessary.
> 
> 
>      > diff --git a/hw/display/apple-gfx.m b/hw/display/apple-gfx.m
>      > new file mode 100644
>      > index 00000000000..837300f9cd4
>      > --- /dev/null
>      > +++ b/hw/display/apple-gfx.m
>      > @@ -0,0 +1,536 @@
>      > +/*
>      > + * QEMU Apple ParavirtualizedGraphics.framework device
>      > + *
>      > + * Copyright © 2023 Amazon.com, Inc. or its affiliates. All
>     Rights Reserved.
>      > + *
>      > + * This work is licensed under the terms of the GNU GPL, version
>     2 or later.
>      > + * See the COPYING file in the top-level directory.
>      > + *
>      > + * ParavirtualizedGraphics.framework is a set of libraries that
>     macOS provides
>      > + * which implements 3d graphics passthrough to the host as well as a
>      > + * proprietary guest communication channel to drive it. This
>     device model
>      > + * implements support to drive that library from within QEMU.
>      > + */
>      > +
>      > +#include "apple-gfx.h"
>      > +#include "trace.h"
>      > +#include "qemu/main-loop.h"
>      > +#include "ui/console.h"
>      > +#include "monitor/monitor.h"
>      > +#include "qapi/error.h"
>      > +#include "migration/blocker.h"
>      > +#include <mach/mach_vm.h>
>      > +#import <ParavirtualizedGraphics/ParavirtualizedGraphics.h>
>      > +
>      > +static const PGDisplayCoord_t apple_gfx_modes[] = {
>      > +    { .x = 1440, .y = 1080 },
>      > +    { .x = 1280, .y = 1024 },
>      > +};
>      > +
>      > +typedef struct PGTask_s { // Name matches forward declaration in
>     PG header
> 
>     Let's name it AppleGFXTask. It is a common practice to have the same
>     tag
>     name and typedef in QEMU.
> 
> 
> This is defining a forward-declared type from framework headers which is 
> opaque from the framework's point of view. We do not get to choose its 
> struct name. The alternative is having casts wherever these objects are 
> being passed between our code and the framework. (See the original v1/v2 
> vmapple patch series for how messy this is.)

I got the idea. Let's not avoid the typedef then to clarify the naming 
is not under our control.

> 
>      > +static void apple_gfx_render_new_frame(AppleGFXState *s)
>      > +{
>      > +    BOOL r;
>      > +    void *vram = s->vram;
>      > +    uint32_t width = surface_width(s->surface);
>      > +    uint32_t height = surface_height(s->surface);
>      > +    MTLRegion region = MTLRegionMake2D(0, 0, width, height);
>      > +    id<MTLCommandBuffer> command_buffer = [s->mtl_queue
>     commandBuffer];
>      > +    id<MTLTexture> texture = s->texture;
>      > +    r = [s->pgdisp encodeCurrentFrameToCommandBuffer:command_buffer
>      > +                                             texture:texture
>      > +                                              region:region];
>      > +    if (!r) {
>      > +        return;
>      > +    }
>      > +    [texture retain];
>      > +    if (s->using_managed_texture_storage) {
>      > +        /* "Managed" textures exist in both VRAM and RAM and
>     must be synced. */
>      > +        id<MTLBlitCommandEncoder> blit = [command_buffer
>     blitCommandEncoder];
>      > +        [blit synchronizeResource:texture];
>      > +        [blit endEncoding];
>      > +    }
>      > +    [command_buffer retain];
> 
>     I don't think this call of retain is necessary. The completion handler
>     gets the command buffer via parameter, which implies the command buffer
>     is automatically retained until the completion handler finishes.
> 
> 
> I couldn't find any hard guarantees for this documented anywhere, hence 
> erring on the side of caution. I think the harm of an additional retain/ 
> release pair here is quite minor.

Apple's sample code doesn't retain for such a case so we can certainly 
omit it:
https://developer.apple.com/documentation/metal/indirect_command_encoding/encoding_indirect_command_buffers_on_the_cpu

It's better to avoid additional resource management when the underlying 
API automatically does things correct. There is a hazard of forgetting 
release anytime adding one call of retain.

> 
> 
>      > +static void apple_gfx_render_frame_completed(AppleGFXState *s,
>     void *vram,
>      > +                                             id<MTLTexture> texture)
>      > +{
>      > +    --s->pending_frames;
>      > +    assert(s->pending_frames >= 0);
>      > +
>      > +    if (vram != s->vram) {
>      > +        /* Display mode has changed, drop this old frame. */
>      > +        assert(texture != s->texture);
>      > +        g_free(vram);
> 
>     This management of buffers looks a bit convoluted. I suggest
>     remembering
>     the width and height instead of pointers and comparing them. This way
>     you can free resources in set_mode().
> 
> 
> Yeah, I suppose that works, I can change that around.
> 
>      > +    } else {
>      > +        copy_mtl_texture_to_surface_mem(texture, vram);
> 
>     Writing vram outside BQL may result in tearing.
> 
> 
> As far as I can tell(*), QXL does the same. I couldn't find any examples 
> of double-buffering in any of the existing display devices, which would 
> be the only way to do async updates efficiently and without tearing. In 
> any case, this solution is still vastly better than a regular VGA 
> device, which suffers from very visible tearing with macOS on the guest 
> side anyway. And in an ideal world, we'd pass through the rendered 
> texture directly to the Cocoa UI code rather than copying out only for 
> the CPU to draw it back into a window surface which is then passed to 
> the GPU for host side rendering. But I felt this patch is already very, 
> very large, and if anyone cares, we can fix imperfections in subsequent 
> updates.
> 
> (*)The rendering code in that device is also fairly complex, so I may be 
> misreading it.

QXL always modifies the surface with BQL. The surface is modified with 
qxl_blit(), which is called by qxl_render_update_area_unlocked(). 
qxl_render_update_area_unlocked() is called by either of 
qxl_render_update() and qxl_render_update_area_bh(). Both of them are 
called with BQL. The name includes "unlocked", but it means it is called 
without holding QXL-internal lock.

Most devices works entirely with BQL so they don't perform double 
buffering. apple-gfx can do the same.

> 
>      > +        if (s->gfx_update_requested) {
>      > +            s->gfx_update_requested = false;
>      > +            dpy_gfx_update_full(s->con);
>      > +            graphic_hw_update_done(s->con);
>       > +            s->new_frame_ready = false;
> 
>     This assignment is unnecessary as s->new_frame_ready is always false
>     when s->gfx_update_requested. If you want to make sure
>     s->gfx_update_requested and s->new_frame_ready are mutually exclusive,
>     use one enum value instead of having two bools.
> 
> 
> I'll need to refresh my memory and get back to you on this one, it's 
> been so many months since I actively worked on this code.
> 
>      > +        } else {
>      > +            s->new_frame_ready = true;
>      > +        }
>      > +    }
>      > +    if (s->pending_frames > 0) {
>      > +        apple_gfx_render_new_frame(s);
>      > +    }
>      > +}
>      > +
>      > +static void apple_gfx_fb_update_display(void *opaque)
>      > +{
>      > +    AppleGFXState *s = opaque;
>      > +
>      > +    dispatch_async(s->render_queue, ^{
>      > +        if (s->pending_frames > 0) {
> 
>     It should check for s->new_frame_ready as
>     apple_gfx_render_frame_completed() doesn't check if
>     s->pending_frames > 0 before calling graphic_hw_update_done(), which is
>     inconsistent.
> 
> 
> pending_frames is about guest-side frames that are queued to be rendered 
> by the host GPU.
> new_frame_ready being true indicates that the contents of the Qemu 
> console surface has been updated with new frame data since the last 
> gfx_update.
> gfx_update_requested indicates that gfx_update is currently awaiting an 
> async completion (graphic_hw_update_done) but the surface has not 
> received a new frame content, but the GPU is stily busy drawing one.
> 
> apple_gfx_render_frame_completed is scheduled exactly once per pending 
> frame, so pending_frames > 0 is an invariant there. (Hence the assert.)> > I don't think there is any inconsistency here, but I'll double check.
> It's possible that there's an easier way to express the state machine, 
> and I'll take a look at that.

I meant that apple_gfx_render_frame_completed() does not check if the 
frame is the last one currently pending. apple_gfx_fb_update_display() 
ignores a new ready frame when there is a more pending frame, but 
apple_gfx_render_frame_completed() unconditionally fires 
graphic_hw_update_done() even if there is a more pending frame. And I 
think apple_gfx_render_frame_completed() is right and 
apple_gfx_fb_update_display() is wrong in such a situation.

> 
>     Checking if s->pending_frames > 0 both in apple_gfx_fb_update_display()
>     and apple_gfx_render_frame_completed() is also problematic as that can
>     defer graphic_hw_update_done() indefinitely if we are getting new
>     frames
>     too fast.
> 
> 
> I see what you mean about this part. I'll have to test it, but I guess 
> we should reverse the priority, like this:
> 
>          if (s->new_frame_ready) {
>              dpy_gfx_update_full(s->con);
>              s->new_frame_ready = false;
>              graphic_hw_update_done(s->con);
>          } else if (s->pending_frames > 0) {
>              s->gfx_update_requested = true;
>          } else {
>              graphic_hw_update_done(s->con);
>          }
> 
> 1. If we already have a frame, ready to be displayed, return it immediately.
> 2. If the guest has reported that it's completed a frame and the GPU is 
> currently busy rendering it, defer graphic_hw_update_done until it's done.
> 3. If the guest reports no changes to its display, indicate this back to 
> Qemu as a no-op display update graphic_hw_update_done() with no 
> dpy_gfx_update* call.

Yes, that looks correct.

> 
>      > +            s->gfx_update_requested = true;
>      > +        } else {
>      > +            if (s->new_frame_ready) {
>      > +                dpy_gfx_update_full(s->con);
>      > +                s->new_frame_ready = false;
>      > +            }
>      > +            graphic_hw_update_done(s->con);
>       > +        }> +    });
>      > +}
>      > +
>      > +static const GraphicHwOps apple_gfx_fb_ops = {
>      > +    .gfx_update = apple_gfx_fb_update_display,
>      > +    .gfx_update_async = true,
>      > +};
>      > +
>      > +static void update_cursor(AppleGFXState *s)
>      > +{
>      > +    dpy_mouse_set(s->con, s->pgdisp.cursorPosition.x,
>      > +                  s->pgdisp.cursorPosition.y, s->cursor_show);
>      > +}
>      > +
>      > +static void set_mode(AppleGFXState *s, uint32_t width, uint32_t
>     height)
>      > +{
>      > +    void *vram = NULL;
>      > +    DisplaySurface *surface;
>      > +    MTLTextureDescriptor *textureDescriptor;
>      > +    id<MTLTexture> texture = nil;
>      > +    __block bool no_change = false;
>      > +
>      > +    dispatch_sync(s->render_queue,
> 
>     Calling dispatch_sync() while holding BQL may result in deadlock.
> 
> Only if any code executed on the same dispatch queue acquires the lock 
> either directly or transitively. I believe I have ensure this is not 
> done on the reqnder_queue, have you found anywhere this is the case?

The documentation is not clear what threads a dispatch queue runs on. We 
can have a deadlock if they lock the BQL.

> 
>      > +        ^{
>      > +            if (s->surface &&
>      > +                width == surface_width(s->surface) &&
>      > +                height == surface_height(s->surface)) {
>      > +                no_change = true;
>      > +            }
>      > +        });
>      > +
>      > +    if (no_change) {
>      > +        return;
>      > +    }
>      > +
>      > +    vram = g_malloc0(width * height * 4);
>      > +    surface = qemu_create_displaysurface_from(width, height,
>     PIXMAN_LE_a8r8g8b8,
>      > +                                              width * 4, vram);
>      > +
>      > +    @autoreleasepool {
>      > +        textureDescriptor =
>      > +            [MTLTextureDescriptor
>      > +               
>     texture2DDescriptorWithPixelFormat:MTLPixelFormatBGRA8Unorm
>      > +                                             width:width
>      > +                                            height:height
>      > +                                         mipmapped:NO];
>      > +        textureDescriptor.usage = s->pgdisp.minimumTextureUsage;
>      > +        texture = [s->mtl
>     newTextureWithDescriptor:textureDescriptor];
>      > +    }
>      > +
>      > +    s->using_managed_texture_storage =
>      > +        (texture.storageMode == MTLStorageModeManaged);
>      > +
>      > +    dispatch_sync(s->render_queue,
>      > +        ^{
>      > +            id<MTLTexture> old_texture = nil;
>      > +            void *old_vram = s->vram;
>      > +            s->vram = vram;
>      > +            s->surface = surface;
>      > +
>      > +            dpy_gfx_replace_surface(s->con, surface);
>      > +
>      > +            old_texture = s->texture;
>      > +            s->texture = texture;
>      > +            [old_texture release];
> 
>     You can just do:
>     [s->texture release];
>     s->texture = texture;
> 
>     This will make s->texture dangling between the two statements, but that
>     don't matter since s->texture is not an atomic variable that can be
>     safely observed from another thread anyway.
> 
>      > +
>      > +            if (s->pending_frames == 0) {
>      > +                g_free(old_vram);
>      > +            }
>      > +        });
>      > +}
>      > +
>      > +static void create_fb(AppleGFXState *s)
>      > +{
>      > +    s->con = graphic_console_init(NULL, 0, &apple_gfx_fb_ops, s);
>      > +    set_mode(s, 1440, 1080);
>      > +
>      > +    s->cursor_show = true;
>      > +}
>      > +
>      > +static size_t apple_gfx_get_default_mmio_range_size(void)
>      > +{
>      > +    size_t mmio_range_size;
>      > +    @autoreleasepool {
>      > +        PGDeviceDescriptor *desc = [PGDeviceDescriptor new];
>      > +        mmio_range_size = desc.mmioLength;
>      > +        [desc release];
>      > +    }
>      > +    return mmio_range_size;
>      > +}
>      > +
>      > +void apple_gfx_common_init(Object *obj, AppleGFXState *s, const
>     char* obj_name)
> 
>     This function can be merged into apple_gfx_common_realize().
> 
> 
> Probably. I'll try it.
> 
>      > +{
>      > +    Error *local_err = NULL;
>      > +    int r;
>      > +    size_t mmio_range_size =
>     apple_gfx_get_default_mmio_range_size();
>      > +
>      > +    trace_apple_gfx_common_init(obj_name, mmio_range_size);
>      > +    memory_region_init_io(&s->iomem_gfx, obj, &apple_gfx_ops, s,
>     obj_name,
>      > +                          mmio_range_size);
>      > +    s->iomem_gfx.disable_reentrancy_guard = true;
> 
>     Why do you disable reentrancy guard?
> 
> 
> Perhaps with the proposed AIO_WAIT_WHILE based I/O scheme, this won't be 
> an issue anymore, but the guard would otherwise keep dropping MMIOs 
> which immediately caused the PV graphics device to stop making progress. 
> The MMIO APIs in the PVG framework are thread- and reentrancy-safe, so 
> we certainly don't need to serialise them on our side.

It's better to understand why such reentrancy happens. Reentrancy itself 
is often a sign of bug.

> 
>      > +
>      > +    /* TODO: PVG framework supports serialising device state:
>     integrate it! */
>      > +    if (apple_gfx_mig_blocker == NULL) {
>      > +        error_setg(&apple_gfx_mig_blocker,
>      > +                  "Migration state blocked by apple-gfx display
>     device");
>      > +        r = migrate_add_blocker(&apple_gfx_mig_blocker, &local_err);
>      > +        if (r < 0) {
>      > +            error_report_err(local_err);
> 
>     Please report the error to the caller of apple_gfx_common_realize()
>     instead.
> 
>      > +        }
>      > +    }
>       > +}> +
>      > +static void
>     apple_gfx_register_task_mapping_handlers(AppleGFXState *s,
>      > +                                                   
>       PGDeviceDescriptor *desc)
>      > +{
>      > +    desc.createTask = ^(uint64_t vmSize, void * _Nullable *
>     _Nonnull baseAddress) {
>      > +        AppleGFXTask *task = apple_gfx_new_task(s, vmSize);
>      > +        *baseAddress = (void*)task->address;
> 
>     nit: please write as (void *) instead of (void*).
> 
>      > +        trace_apple_gfx_create_task(vmSize, *baseAddress);
>      > +        return task;
>      > +    };
>      > +
> 
>      > +{
>      > +    PGDisplayDescriptor *disp_desc = [PGDisplayDescriptor new];
>      > +
>      > + disp_desc.name <http://disp_desc.name> = @"QEMU display";
>      > +    disp_desc.sizeInMillimeters = NSMakeSize(400., 300.); /* A
>     20" display */
>      > +    disp_desc.queue = dispatch_get_main_queue();
>      > +    disp_desc.newFrameEventHandler = ^(void) {
>      > +        trace_apple_gfx_new_frame();
>      > +        dispatch_async(s->render_queue, ^{
>      > +            /* Drop frames if we get too far ahead. */
>      > +            if (s->pending_frames >= 2)
>      > +                return;
>      > +            ++s->pending_frames;
>      > +            if (s->pending_frames > 1) {
>      > +                return;
>      > +            }
>      > +            @autoreleasepool {
>      > +                apple_gfx_render_new_frame(s);
>      > +            }
>      > +        });
>      > +    };
>      > +    disp_desc.modeChangeHandler = ^(PGDisplayCoord_t sizeInPixels,
>      > +                                    OSType pixelFormat) {
>      > +        trace_apple_gfx_mode_change(sizeInPixels.x, sizeInPixels.y);
>      > +        set_mode(s, sizeInPixels.x, sizeInPixels.y);
>      > +    };
>      > +    disp_desc.cursorGlyphHandler = ^(NSBitmapImageRep *glyph,
>      > +                                     PGDisplayCoord_t hotSpot) {
>      > +        uint32_t bpp = glyph.bitsPerPixel;
>      > +        size_t width = glyph.pixelsWide;
>      > +        size_t height = glyph.pixelsHigh;
>      > +        size_t padding_bytes_per_row = glyph.bytesPerRow - width
>     * 4;
>      > +        const uint8_t* px_data = glyph.bitmapData;
>      > +
>      > +        trace_apple_gfx_cursor_set(bpp, width, height);
>      > +
>      > +        if (s->cursor) {
>      > +            cursor_unref(s->cursor);
>      > +            s->cursor = NULL;
>      > +        }
>      > +
>      > +        if (bpp == 32) { /* Shouldn't be anything else, but just
>     to be safe...*/
>      > +            s->cursor = cursor_alloc(width, height);
>      > +            s->cursor->hot_x = hotSpot.x;
>      > +            s->cursor->hot_y = hotSpot.y;
>      > +
>      > +            uint32_t *dest_px = s->cursor->data;
>      > +
>      > +            for (size_t y = 0; y < height; ++y) {
>      > +                for (size_t x = 0; x < width; ++x) {
>      > +                    /* NSBitmapImageRep's red & blue channels
>     are swapped
>      > +                     * compared to QEMUCursor's. */
>      > +                    *dest_px =
>      > +                        (px_data[0] << 16u) |
>      > +                        (px_data[1] <<  8u) |
>      > +                        (px_data[2] <<  0u) |
>      > +                        (px_data[3] << 24u);
>      > +                    ++dest_px;
>      > +                    px_data += 4;
>      > +                }
>      > +                px_data += padding_bytes_per_row;
>      > +            }
>      > +            dpy_cursor_define(s->con, s->cursor);
>      > +            update_cursor(s);
>      > +        }
>      > +    };
>      > +    disp_desc.cursorShowHandler = ^(BOOL show) {
>      > +        trace_apple_gfx_cursor_show(show);
>      > +        s->cursor_show = show;
>      > +        update_cursor(s);
>      > +    };
>      > +    disp_desc.cursorMoveHandler = ^(void) {
>      > +        trace_apple_gfx_cursor_move();
>      > +        update_cursor(s);
>      > +    };
>      > +
>      > +    return disp_desc;
>      > +}
>      > +
>      > +static NSArray<PGDisplayMode*>*
>     apple_gfx_prepare_display_mode_array(void)
>      > +{
>      > +    PGDisplayMode *modes[ARRAY_SIZE(apple_gfx_modes)];
>      > +    NSArray<PGDisplayMode*>* mode_array = nil;
>      > +    int i;
>      > +
>      > +    for (i = 0; i < ARRAY_SIZE(apple_gfx_modes); i++) {
>      > +        modes[i] =
>      > +            [[PGDisplayMode alloc]
>     initWithSizeInPixels:apple_gfx_modes[i] refreshRateInHz:60.];
>      > +    }
>      > +
>      > +    mode_array = [NSArray arrayWithObjects:modes
>     count:ARRAY_SIZE(apple_gfx_modes)];
>      > +
>      > +    for (i = 0; i < ARRAY_SIZE(apple_gfx_modes); i++) {
>      > +        [modes[i] release];
>      > +        modes[i] = nil;
>      > +    }
>      > +
>      > +    return mode_array;
>      > +}
>      > +
>      > +static id<MTLDevice> copy_suitable_metal_device(void)
>      > +{
>      > +    id<MTLDevice> dev = nil;
>      > +    NSArray<id<MTLDevice>> *devs = MTLCopyAllDevices();
>      > +
>      > +    /* Prefer a unified memory GPU. Failing that, pick a non-
>     removable GPU. */
>      > +    for (size_t i = 0; i < devs.count; ++i) {
>      > +        if (devs[i].hasUnifiedMemory) {
>      > +            dev = devs[i];
>      > +            break;
>      > +        }
>      > +        if (!devs[i].removable) {
>      > +            dev = devs[i];
>      > +        }
>      > +    }
>      > +
>      > +    if (dev != nil) {
>      > +        [dev retain];
>      > +    } else {
>      > +        dev = MTLCreateSystemDefaultDevice();
>      > +    }
>      > +    [devs release];
>      > +
>      > +    return dev;
>      > +}
>      > +
>      > +void apple_gfx_common_realize(AppleGFXState *s,
>     PGDeviceDescriptor *desc)
>      > +{
>      > +    PGDisplayDescriptor *disp_desc = nil;
>      > +
>      > +    QTAILQ_INIT(&s->tasks);
>      > +    s->render_queue = dispatch_queue_create("apple-gfx.render",
>      > +                                            DISPATCH_QUEUE_SERIAL);
>      > +    s->mtl = copy_suitable_metal_device();
>      > +    s->mtl_queue = [s->mtl newCommandQueue];
>      > +
>      > +    desc.device = s->mtl;
>      > +
>      > +    apple_gfx_register_task_mapping_handlers(s, desc);
>      > +
>      > +    s->pgdev = PGNewDeviceWithDescriptor(desc);
>      > +
>      > +    disp_desc = apple_gfx_prepare_display_handlers(s);
>      > +    s->pgdisp = [s->pgdev newDisplayWithDescriptor:disp_desc
>      > +                                              port:0
>     serialNum:1234];
>      > +    [disp_desc release];
>      > +    s->pgdisp.modeList = apple_gfx_prepare_display_mode_array();
>      > +
>      > +    create_fb(s);
>      > +}
>      > diff --git a/hw/display/meson.build b/hw/display/meson.build
>      > index 7db05eace97..70d855749c0 100644
>      > --- a/hw/display/meson.build
>      > +++ b/hw/display/meson.build
>      > @@ -65,6 +65,8 @@ system_ss.add(when: 'CONFIG_ARTIST', if_true:
>     files('artist.c'))
>      >
>      >   system_ss.add(when: 'CONFIG_ATI_VGA', if_true: [files('ati.c',
>     'ati_2d.c', 'ati_dbg.c'), pixman])
>      >
>      > +system_ss.add(when: 'CONFIG_MAC_PVG',         if_true:
>     [files('apple-gfx.m'), pvg, metal])
>      > +system_ss.add(when: 'CONFIG_MAC_PVG_VMAPPLE', if_true:
>     [files('apple-gfx-vmapple.m'), pvg, metal])
>      >
>      >   if config_all_devices.has_key('CONFIG_VIRTIO_GPU')
>      >     virtio_gpu_ss = ss.source_set()
>      > diff --git a/hw/display/trace-events b/hw/display/trace-events
>      > index 781f8a33203..1809a358e36 100644
>      > --- a/hw/display/trace-events
>      > +++ b/hw/display/trace-events
>      > @@ -191,3 +191,29 @@ dm163_bits_ppi(unsigned dest_width)
>     "dest_width : %u"
>      >   dm163_leds(int led, uint32_t value) "led %d: 0x%x"
>      >   dm163_channels(int channel, uint8_t value) "channel %d: 0x%x"
>      >   dm163_refresh_rate(uint32_t rr) "refresh rate %d"
>      > +
>      > +# apple-gfx.m
>      > +apple_gfx_read(uint64_t offset, uint64_t res)
>     "offset=0x%"PRIx64" res=0x%"PRIx64
>      > +apple_gfx_write(uint64_t offset, uint64_t val)
>     "offset=0x%"PRIx64" val=0x%"PRIx64
>      > +apple_gfx_create_task(uint32_t vm_size, void *va) "vm_size=0x%x
>     base_addr=%p"
>      > +apple_gfx_destroy_task(void *task) "task=%p"
>      > +apple_gfx_map_memory(void *task, uint32_t range_count, uint64_t
>     virtual_offset, uint32_t read_only) "task=%p range_count=0x%x
>     virtual_offset=0x%"PRIx64" read_only=%d"
>      > +apple_gfx_map_memory_range(uint32_t i, uint64_t phys_addr,
>     uint64_t phys_len) "[%d] phys_addr=0x%"PRIx64" phys_len=0x%"PRIx64
>      > +apple_gfx_remap(uint64_t retval, uint64_t source, uint64_t
>     target) "retval=%"PRId64" source=0x%"PRIx64" target=0x%"PRIx64
>      > +apple_gfx_unmap_memory(void *task, uint64_t virtual_offset,
>     uint64_t length) "task=%p virtual_offset=0x%"PRIx64" length=0x%"PRIx64
>      > +apple_gfx_read_memory(uint64_t phys_address, uint64_t length,
>     void *dst) "phys_addr=0x%"PRIx64" length=0x%"PRIx64" dest=%p"
>      > +apple_gfx_raise_irq(uint32_t vector) "vector=0x%x"
>      > +apple_gfx_new_frame(void) ""
>      > +apple_gfx_mode_change(uint64_t x, uint64_t y) "x=%"PRId64"
>     y=%"PRId64
>      > +apple_gfx_cursor_set(uint32_t bpp, uint64_t width, uint64_t
>     height) "bpp=%d width=%"PRId64" height=0x%"PRId64
>      > +apple_gfx_cursor_show(uint32_t show) "show=%d"
>      > +apple_gfx_cursor_move(void) ""
>      > +apple_gfx_common_init(const char *device_name, size_t mmio_size)
>     "device: %s; MMIO size: %zu bytes"
>      > +
>      > +# apple-gfx-vmapple.m
>      > +apple_iosfc_read(uint64_t offset, uint64_t res)
>     "offset=0x%"PRIx64" res=0x%"PRIx64
>      > +apple_iosfc_write(uint64_t offset, uint64_t val)
>     "offset=0x%"PRIx64" val=0x%"PRIx64
>      > +apple_iosfc_map_memory(uint64_t phys, uint64_t len, uint32_t ro,
>     void *va, void *e, void *f) "phys=0x%"PRIx64" len=0x%"PRIx64" ro=%d
>     va=%p e=%p f=%p"
>      > +apple_iosfc_unmap_memory(void *a, void *b, void *c, void *d,
>     void *e, void *f) "a=%p b=%p c=%p d=%p e=%p f=%p"
>      > +apple_iosfc_raise_irq(uint32_t vector) "vector=0x%x"
>      > +
>      > diff --git a/meson.build b/meson.build
>      > index 10464466ff3..f09df3f09d5 100644
>      > --- a/meson.build
>      > +++ b/meson.build
>      > @@ -741,6 +741,8 @@ socket = []
>      >   version_res = []
>      >   coref = []
>      >   iokit = []
>      > +pvg = []
>      > +metal = []
>      >   emulator_link_args = []
>      >   midl = not_found
>      >   widl = not_found
>      > @@ -762,6 +764,8 @@ elif host_os == 'darwin'
>      >     coref = dependency('appleframeworks', modules: 'CoreFoundation')
>      >     iokit = dependency('appleframeworks', modules: 'IOKit',
>     required: false)
>      >     host_dsosuf = '.dylib'
>      > +  pvg = dependency('appleframeworks', modules:
>     'ParavirtualizedGraphics')
>      > +  metal = dependency('appleframeworks', modules: 'Metal')
>      >   elif host_os == 'sunos'
>      >     socket = [cc.find_library('socket'),
>      >               cc.find_library('nsl'),
> 

Re: [PATCH v3 00/14] macOS PV Graphics and new vmapple machine type

[Alex Bennée]

Phil Dennis-Jordan <phil@philjordan.eu> writes:

> (Apologies to anyone who has received more than one version of this
> series of emails; my git-send-email was misconfigured and this is
> a new attempt.)
>
> This patch set introduces a new ARM and macOS HVF specific machine type
> called "vmapple", as well as a family of display devices based on the
> ParavirtualizedGraphics.framework in macOS. One of the display adapter
> variants, apple-gfx-vmapple, is required for the new machine type, while
> apple-gfx-pci can be used to enable 3D graphics acceleration with x86-64
> macOS guest OSes.
>
<snip>
>
> There are currently a few limitations to this which aren't intrinsic,
> just imperfect emulation of the VZF, but it's good enough to be just
> about usable for some purposes:
>
>  * macOS 12 guests only. Versions 13+ currently fail during early boot.
>  * macOS 11+ arm64 hosts only, with hvf accel. (Perhaps some differences
>    between Apple M series CPUs and TCG's aarch64 implementation? macOS
>    hosts only because ParavirtualizedGraphics.framework is a black box
>    implementing most of the logic behind the apple-gfx device.)

We don't currently have TCG CPU models for the Apple Silicon processors.
They are not too hard to add (basically setting the correct ID register
bits, c.f. aarch64_neoverse_n1_initfn for an example). However that
would only cover Aarch64 architectural features. We do no modelling of
the extra instructions that Apple added (although in theory that should
only be run in Apples own ML libraries).


-- 
Alex Bennée
Virtualisation Tech Lead @ Linaro

Re: [PATCH v3 04/14] hw/display/apple-gfx: Adds configurable mode list

[Akihiko Odaki]

On 2024/09/28 17:57, Phil Dennis-Jordan wrote:
> This change adds a property 'display_modes' on the graphics device
> which permits specifying a list of display modes. (screen resolution
> and refresh rate)
> 
> PCI variant of apple-gfx only for the moment.
> 
> Signed-off-by: Phil Dennis-Jordan <phil@philjordan.eu>
> ---
>   hw/display/apple-gfx-pci.m |  43 ++++++++++-
>   hw/display/apple-gfx.h     |  17 ++++-
>   hw/display/apple-gfx.m     | 151 ++++++++++++++++++++++++++++++++++---
>   3 files changed, 198 insertions(+), 13 deletions(-)
> 
> diff --git a/hw/display/apple-gfx-pci.m b/hw/display/apple-gfx-pci.m
> index 9370258ee46..ea86a1f4a21 100644
> --- a/hw/display/apple-gfx-pci.m
> +++ b/hw/display/apple-gfx-pci.m
> @@ -16,6 +16,7 @@
>   #include "apple-gfx.h"
>   #include "hw/pci/pci_device.h"
>   #include "hw/pci/msi.h"
> +#include "hw/qdev-properties.h"
>   #include "qapi/error.h"
>   #include "trace.h"
>   #import <ParavirtualizedGraphics/ParavirtualizedGraphics.h>
> @@ -101,6 +102,46 @@ static void apple_gfx_pci_reset(Object *obj, ResetType type)
>       [s->common.pgdev reset];
>   }
>   
> +static void apple_gfx_pci_get_display_modes(Object *obj, Visitor *v,
> +                                            const char *name, void *opaque,
> +                                            Error **errp)
> +{
> +    Property *prop = opaque;
> +    AppleGFXDisplayModeList *mode_list = object_field_prop_ptr(obj, prop);
> +
> +    apple_gfx_get_display_modes(mode_list, v, name, errp);
> +}
> +
> +static void apple_gfx_pci_set_display_modes(Object *obj, Visitor *v,
> +                                            const char *name, void *opaque,
> +                                            Error **errp)
> +{
> +    Property *prop = opaque;
> +    AppleGFXDisplayModeList *mode_list = object_field_prop_ptr(obj, prop);
> +
> +    apple_gfx_set_display_modes(mode_list, v, name, errp);
> +}
> +
> +const PropertyInfo apple_gfx_pci_prop_display_modes = {
> +    .name  = "display_modes",
> +    .description =
> +        "Colon-separated list of display modes; "
> +        "<width>x<height>@<refresh-rate>; the first mode is considered "
> +        "'native'. Example: 3840x2160@60:2560x1440@60:1920x1080@60",

Please use DEFINE_PROP_ARRAY() instead of inventing your own way to 
define an array.

> +    .get   = apple_gfx_pci_get_display_modes,
> +    .set   = apple_gfx_pci_set_display_modes,
> +};
> +
> +#define DEFINE_PROP_DISPLAY_MODES(_name, _state, _field) \
> +    DEFINE_PROP(_name, _state, _field, apple_gfx_pci_prop_display_modes, \
> +                AppleGFXDisplayModeList)
> +
> +static Property apple_gfx_pci_properties[] = {
> +    DEFINE_PROP_DISPLAY_MODES("display-modes", AppleGFXPCIState,
> +                              common.display_modes),
> +    DEFINE_PROP_END_OF_LIST(),
> +};
> +
>   static void apple_gfx_pci_class_init(ObjectClass *klass, void *data)
>   {
>       DeviceClass *dc = DEVICE_CLASS(klass);
> @@ -118,7 +159,7 @@ static void apple_gfx_pci_class_init(ObjectClass *klass, void *data)
>       pci->class_id = PCI_CLASS_DISPLAY_OTHER;
>       pci->realize = apple_gfx_pci_realize;
>   
> -    // TODO: Property for setting mode list
> +    device_class_set_props(dc, apple_gfx_pci_properties);
>   }
>   
>   static TypeInfo apple_gfx_pci_types[] = {
> diff --git a/hw/display/apple-gfx.h b/hw/display/apple-gfx.h
> index 995ecf7f4a7..baad4a98652 100644
> --- a/hw/display/apple-gfx.h
> +++ b/hw/display/apple-gfx.h
> @@ -5,14 +5,28 @@
>   #define TYPE_APPLE_GFX_PCI          "apple-gfx-pci"
>   
>   #include "qemu/typedefs.h"
> +#include "qemu/osdep.h"
>   
>   typedef struct AppleGFXState AppleGFXState;
>   
> +typedef struct AppleGFXDisplayMode {
> +    uint16_t width_px;
> +    uint16_t height_px;
> +    uint16_t refresh_rate_hz;
> +} AppleGFXDisplayMode;
> +
> +typedef struct AppleGFXDisplayModeList {
> +    GArray *modes;
> +} AppleGFXDisplayModeList;
> +
>   void apple_gfx_common_init(Object *obj, AppleGFXState *s, const char* obj_name);
> +void apple_gfx_get_display_modes(AppleGFXDisplayModeList *mode_list, Visitor *v,
> +                                 const char *name, Error **errp);
> +void apple_gfx_set_display_modes(AppleGFXDisplayModeList *mode_list, Visitor *v,
> +                                 const char *name, Error **errp);
>   
>   #ifdef __OBJC__
>   
> -#include "qemu/osdep.h"
>   #include "exec/memory.h"
>   #include "ui/surface.h"
>   #include <dispatch/dispatch.h>
> @@ -38,6 +52,7 @@ struct AppleGFXState {
>       bool new_frame;
>       bool cursor_show;
>       QEMUCursor *cursor;
> +    AppleGFXDisplayModeList display_modes;
>   
>       dispatch_queue_t render_queue;
>       /* The following fields should only be accessed from render_queue: */
> diff --git a/hw/display/apple-gfx.m b/hw/display/apple-gfx.m
> index 6ef1048d93d..358192db6a0 100644
> --- a/hw/display/apple-gfx.m
> +++ b/hw/display/apple-gfx.m
> @@ -16,6 +16,9 @@
>   #include "trace.h"
>   #include "qemu-main.h"
>   #include "qemu/main-loop.h"
> +#include "qemu/cutils.h"
> +#include "qapi/visitor.h"
> +#include "qapi/error.h"
>   #include "ui/console.h"
>   #include "monitor/monitor.h"
>   #include "qapi/error.h"
> @@ -23,9 +26,10 @@
>   #include <mach/mach_vm.h>
>   #import <ParavirtualizedGraphics/ParavirtualizedGraphics.h>
>   
> -static const PGDisplayCoord_t apple_gfx_modes[] = {
> -    { .x = 1440, .y = 1080 },
> -    { .x = 1280, .y = 1024 },
> +static const AppleGFXDisplayMode apple_gfx_default_modes[] = {
> +    { 1920, 1080, 60 },
> +    { 1440, 1080, 60 },
> +    { 1280, 1024, 60 },
>   };
>   
>   typedef struct PGTask_s { // Name matches forward declaration in PG header
> @@ -264,7 +268,6 @@ static void set_mode(AppleGFXState *s, uint32_t width, uint32_t height)
>   static void create_fb(AppleGFXState *s)
>   {
>       s->con = graphic_console_init(NULL, 0, &apple_gfx_fb_ops, s);
> -    set_mode(s, 1440, 1080);
>   
>       s->cursor_show = true;
>   }
> @@ -466,20 +469,24 @@ static void apple_gfx_register_task_mapping_handlers(AppleGFXState *s,
>       return disp_desc;
>   }
>   
> -static NSArray<PGDisplayMode*>* apple_gfx_prepare_display_mode_array(void)
> +static NSArray<PGDisplayMode*>* apple_gfx_create_display_mode_array(
> +    const AppleGFXDisplayMode display_modes[], int display_mode_count)
>   {
> -    PGDisplayMode *modes[ARRAY_SIZE(apple_gfx_modes)];
> +    PGDisplayMode **modes = alloca(sizeof(modes[0]) * display_mode_count);
>       NSArray<PGDisplayMode*>* mode_array = nil;
>       int i;
>   
> -    for (i = 0; i < ARRAY_SIZE(apple_gfx_modes); i++) {
> +    for (i = 0; i < display_mode_count; i++) {
> +        const AppleGFXDisplayMode *mode = &display_modes[i];
> +        PGDisplayCoord_t mode_size = { mode->width_px, mode->height_px };
>           modes[i] =
> -            [[PGDisplayMode alloc] initWithSizeInPixels:apple_gfx_modes[i] refreshRateInHz:60.];
> +            [[PGDisplayMode alloc] initWithSizeInPixels:mode_size
> +                                        refreshRateInHz:mode->refresh_rate_hz];
>       }
>   
> -    mode_array = [NSArray arrayWithObjects:modes count:ARRAY_SIZE(apple_gfx_modes)];
> +    mode_array = [NSArray arrayWithObjects:modes count:display_mode_count];
>   
> -    for (i = 0; i < ARRAY_SIZE(apple_gfx_modes); i++) {
> +    for (i = 0; i < display_mode_count; i++) {
>           [modes[i] release];
>           modes[i] = nil;
>       }
> @@ -516,6 +523,8 @@ static void apple_gfx_register_task_mapping_handlers(AppleGFXState *s,
>   void apple_gfx_common_realize(AppleGFXState *s, PGDeviceDescriptor *desc)
>   {
>       PGDisplayDescriptor *disp_desc = nil;
> +    const AppleGFXDisplayMode *display_modes = apple_gfx_default_modes;
> +    int num_display_modes = ARRAY_SIZE(apple_gfx_default_modes);
>   
>       QTAILQ_INIT(&s->tasks);
>       s->render_queue = dispatch_queue_create("apple-gfx.render",
> @@ -533,7 +542,127 @@ void apple_gfx_common_realize(AppleGFXState *s, PGDeviceDescriptor *desc)
>       s->pgdisp = [s->pgdev newDisplayWithDescriptor:disp_desc
>                                                 port:0 serialNum:1234];
>       [disp_desc release];
> -    s->pgdisp.modeList = apple_gfx_prepare_display_mode_array();
> +
> +    if (s->display_modes.modes != NULL && s->display_modes.modes->len > 0) {
> +        display_modes =
> +            &g_array_index(s->display_modes.modes, AppleGFXDisplayMode, 0);
> +        num_display_modes = s->display_modes.modes->len;
> +    }
> +    s->pgdisp.modeList =
> +        apple_gfx_create_display_mode_array(display_modes, num_display_modes);
>   
>       create_fb(s);
>   }
> +
> +void apple_gfx_get_display_modes(AppleGFXDisplayModeList *mode_list, Visitor *v,
> +                                 const char *name, Error **errp)
> +{
> +    GArray *modes = mode_list->modes;
> +    /* 3 uint16s (max 5 digits) and 3 separator characters per mode + nul. */
> +    size_t buffer_size = (5 + 1) * 3 * modes->len + 1;
> +
> +    char *buffer = alloca(buffer_size);
> +    char *pos = buffer;
> +
> +    unsigned used = 0;
> +    buffer[0] = '\0';
> +    for (guint i = 0; i < modes->len; ++i)
> +    {
> +        AppleGFXDisplayMode *mode =
> +            &g_array_index(modes, AppleGFXDisplayMode, i);
> +        int rc = snprintf(pos, buffer_size - used,
> +                          "%s%"PRIu16"x%"PRIu16"@%"PRIu16,
> +                          i > 0 ? ":" : "",
> +                          mode->width_px, mode->height_px,
> +                          mode->refresh_rate_hz);
> +        used += rc;
> +        pos += rc;
> +        assert(used < buffer_size);
> +    }
> +
> +    pos = buffer;
> +    visit_type_str(v, name, &pos, errp);
> +}
> +
> +void apple_gfx_set_display_modes(AppleGFXDisplayModeList *mode_list, Visitor *v,
> +                                 const char *name, Error **errp)
> +{
> +    Error *local_err = NULL;
> +    const char *endptr;
> +    char *str;
> +    int ret;
> +    unsigned int val;
> +    uint32_t num_modes;
> +    GArray *modes;
> +    uint32_t mode_idx;
> +
> +    visit_type_str(v, name, &str, &local_err);
> +    if (local_err) {
> +        error_propagate(errp, local_err);
> +        return;
> +    }
> +
> +    // Count colons to estimate modes. No leading/trailing colons so start at 1.
> +    num_modes = 1;
> +    for (size_t i = 0; str[i] != '\0'; ++i)
> +    {
> +        if (str[i] == ':') {
> +            ++num_modes;
> +        }
> +    }
> +
> +    modes = g_array_sized_new(false, true, sizeof(AppleGFXDisplayMode), num_modes);
> +
> +    endptr = str;
> +    for (mode_idx = 0; mode_idx < num_modes; ++mode_idx)
> +    {
> +        AppleGFXDisplayMode mode = {};
> +        if (mode_idx > 0)
> +        {
> +            if (*endptr != ':') {
> +                goto separator_error;
> +            }
> +            ++endptr;
> +        }
> +
> +        ret = qemu_strtoui(endptr, &endptr, 10, &val);
> +        if (ret || val > UINT16_MAX || val == 0) {
> +            error_setg(errp, "width of '%s' must be a decimal integer number "
> +                       "of pixels in the range 1..65535", name);
> +            goto out;
> +        }
> +        mode.width_px = val;
> +        if (*endptr != 'x') {
> +            goto separator_error;
> +        }
> +
> +        ret = qemu_strtoui(endptr + 1, &endptr, 10, &val);
> +        if (ret || val > UINT16_MAX || val == 0) {
> +            error_setg(errp, "height of '%s' must be a decimal integer number "
> +                       "of pixels in the range 1..65535", name);
> +            goto out;
> +        }
> +        mode.height_px = val;
> +        if (*endptr != '@') {
> +            goto separator_error;
> +        }
> +
> +        ret = qemu_strtoui(endptr + 1, &endptr, 10, &val);
> +        if (ret) {
> +            error_setg(errp, "refresh rate of '%s'"
> +                       " must be a non-negative decimal integer (Hertz)", name);
> +        }
> +        mode.refresh_rate_hz = val;
> +        g_array_append_val(modes, mode);
> +    }
> +
> +    mode_list->modes = modes;
> +    goto out;
> +
> +separator_error:
> +    error_setg(errp, "Each display mode takes the format "
> +               "'<width>x<height>@<rate>', modes are separated by colons. (:)");
> +out:
> +    g_free(str);
> +    return;
> +}

Re: [PATCH v3 09/14] gpex: Allow more than 4 legacy IRQs

[Akihiko Odaki]

On 2024/09/28 17:57, Phil Dennis-Jordan wrote:
> From: Alexander Graf <graf@amazon.com>
> 
> Some boards such as vmapple don't do real legacy PCI IRQ swizzling.
> Instead, they just keep allocating more board IRQ lines for each new
> legacy IRQ. Let's support that mode by giving instantiators a new
> "nr_irqs" property they can use to support more than 4 legacy IRQ lines.
> In this mode, GPEX will export more IRQ lines, one for each device.
> 
> Signed-off-by: Alexander Graf <graf@amazon.com>
> Signed-off-by: Phil Dennis-Jordan <phil@philjordan.eu>
> ---
>   hw/arm/sbsa-ref.c          |  2 +-
>   hw/arm/virt.c              |  2 +-
>   hw/i386/microvm.c          |  2 +-
>   hw/loongarch/virt.c        |  2 +-
>   hw/mips/loongson3_virt.c   |  2 +-
>   hw/openrisc/virt.c         | 12 ++++++------
>   hw/pci-host/gpex.c         | 36 +++++++++++++++++++++++++++++++-----
>   hw/riscv/virt.c            | 12 ++++++------
>   hw/xtensa/virt.c           |  2 +-
>   include/hw/pci-host/gpex.h |  7 +++----
>   10 files changed, 52 insertions(+), 27 deletions(-)
> 
> diff --git a/hw/arm/sbsa-ref.c b/hw/arm/sbsa-ref.c
> index e3195d54497..7e7322486c2 100644
> --- a/hw/arm/sbsa-ref.c
> +++ b/hw/arm/sbsa-ref.c
> @@ -673,7 +673,7 @@ static void create_pcie(SBSAMachineState *sms)
>       /* Map IO port space */
>       sysbus_mmio_map(SYS_BUS_DEVICE(dev), 2, base_pio);
>   
> -    for (i = 0; i < GPEX_NUM_IRQS; i++) {
> +    for (i = 0; i < PCI_NUM_PINS; i++) {
>           sysbus_connect_irq(SYS_BUS_DEVICE(dev), i,
>                              qdev_get_gpio_in(sms->gic, irq + i));
>           gpex_set_irq_num(GPEX_HOST(dev), i, irq + i);
> diff --git a/hw/arm/virt.c b/hw/arm/virt.c
> index 8b2b991d978..bd3b17be2ea 100644
> --- a/hw/arm/virt.c
> +++ b/hw/arm/virt.c
> @@ -1547,7 +1547,7 @@ static void create_pcie(VirtMachineState *vms)
>       /* Map IO port space */
>       sysbus_mmio_map(SYS_BUS_DEVICE(dev), 2, base_pio);
>   
> -    for (i = 0; i < GPEX_NUM_IRQS; i++) {
> +    for (i = 0; i < PCI_NUM_PINS; i++) {
>           sysbus_connect_irq(SYS_BUS_DEVICE(dev), i,
>                              qdev_get_gpio_in(vms->gic, irq + i));
>           gpex_set_irq_num(GPEX_HOST(dev), i, irq + i);
> diff --git a/hw/i386/microvm.c b/hw/i386/microvm.c
> index 40edcee7af2..216a169c413 100644
> --- a/hw/i386/microvm.c
> +++ b/hw/i386/microvm.c
> @@ -139,7 +139,7 @@ static void create_gpex(MicrovmMachineState *mms)
>                                       mms->gpex.mmio64.base, mmio64_alias);
>       }
>   
> -    for (i = 0; i < GPEX_NUM_IRQS; i++) {
> +    for (i = 0; i < PCI_NUM_PINS; i++) {
>           sysbus_connect_irq(SYS_BUS_DEVICE(dev), i,
>                              x86ms->gsi[mms->gpex.irq + i]);
>       }
> diff --git a/hw/loongarch/virt.c b/hw/loongarch/virt.c
> index 75980b6e3c7..8eb7277a8d1 100644
> --- a/hw/loongarch/virt.c
> +++ b/hw/loongarch/virt.c
> @@ -703,7 +703,7 @@ static void virt_devices_init(DeviceState *pch_pic,
>       memory_region_add_subregion(get_system_memory(), VIRT_PCI_IO_BASE,
>                                   pio_alias);
>   
> -    for (i = 0; i < GPEX_NUM_IRQS; i++) {
> +    for (i = 0; i < PCI_NUM_PINS; i++) {
>           sysbus_connect_irq(d, i,
>                              qdev_get_gpio_in(pch_pic, 16 + i));
>           gpex_set_irq_num(GPEX_HOST(gpex_dev), i, 16 + i);
> diff --git a/hw/mips/loongson3_virt.c b/hw/mips/loongson3_virt.c
> index 2067b4fecb5..acafd73129d 100644
> --- a/hw/mips/loongson3_virt.c
> +++ b/hw/mips/loongson3_virt.c
> @@ -458,7 +458,7 @@ static inline void loongson3_virt_devices_init(MachineState *machine,
>                                   virt_memmap[VIRT_PCIE_PIO].base, s->pio_alias);
>       sysbus_mmio_map(SYS_BUS_DEVICE(dev), 2, virt_memmap[VIRT_PCIE_PIO].base);
>   
> -    for (i = 0; i < GPEX_NUM_IRQS; i++) {
> +    for (i = 0; i < PCI_NUM_PINS; i++) {
>           irq = qdev_get_gpio_in(pic, PCIE_IRQ_BASE + i);
>           sysbus_connect_irq(SYS_BUS_DEVICE(dev), i, irq);
>           gpex_set_irq_num(GPEX_HOST(dev), i, PCIE_IRQ_BASE + i);
> diff --git a/hw/openrisc/virt.c b/hw/openrisc/virt.c
> index f8a68a6a6b1..16a5676c4bb 100644
> --- a/hw/openrisc/virt.c
> +++ b/hw/openrisc/virt.c
> @@ -318,7 +318,7 @@ static void create_pcie_irq_map(void *fdt, char *nodename, int irq_base,
>   {
>       int pin, dev;
>       uint32_t irq_map_stride = 0;
> -    uint32_t full_irq_map[GPEX_NUM_IRQS * GPEX_NUM_IRQS * 6] = {};
> +    uint32_t full_irq_map[PCI_NUM_PINS * PCI_NUM_PINS * 6] = {};
>       uint32_t *irq_map = full_irq_map;
>   
>       /*
> @@ -330,11 +330,11 @@ static void create_pcie_irq_map(void *fdt, char *nodename, int irq_base,
>        * possible slot) seeing the interrupt-map-mask will allow the table
>        * to wrap to any number of devices.
>        */
> -    for (dev = 0; dev < GPEX_NUM_IRQS; dev++) {
> +    for (dev = 0; dev < PCI_NUM_PINS; dev++) {
>           int devfn = dev << 3;
>   
> -        for (pin = 0; pin < GPEX_NUM_IRQS; pin++) {
> -            int irq_nr = irq_base + ((pin + PCI_SLOT(devfn)) % GPEX_NUM_IRQS);
> +        for (pin = 0; pin < PCI_NUM_PINS; pin++) {
> +            int irq_nr = irq_base + ((pin + PCI_SLOT(devfn)) % PCI_NUM_PINS);
>               int i = 0;
>   
>               /* Fill PCI address cells */
> @@ -357,7 +357,7 @@ static void create_pcie_irq_map(void *fdt, char *nodename, int irq_base,
>       }
>   
>       qemu_fdt_setprop(fdt, nodename, "interrupt-map", full_irq_map,
> -                     GPEX_NUM_IRQS * GPEX_NUM_IRQS *
> +                     PCI_NUM_PINS * PCI_NUM_PINS *
>                        irq_map_stride * sizeof(uint32_t));
>   
>       qemu_fdt_setprop_cells(fdt, nodename, "interrupt-map-mask",
> @@ -409,7 +409,7 @@ static void openrisc_virt_pcie_init(OR1KVirtState *state,
>       memory_region_add_subregion(get_system_memory(), pio_base, alias);
>   
>       /* Connect IRQ lines. */
> -    for (i = 0; i < GPEX_NUM_IRQS; i++) {
> +    for (i = 0; i < PCI_NUM_PINS; i++) {
>           pcie_irq = get_per_cpu_irq(cpus, num_cpus, irq_base + i);
>   
>           sysbus_connect_irq(SYS_BUS_DEVICE(dev), i, pcie_irq);
> diff --git a/hw/pci-host/gpex.c b/hw/pci-host/gpex.c
> index e9cf455bf52..4aca0a95771 100644
> --- a/hw/pci-host/gpex.c
> +++ b/hw/pci-host/gpex.c
> @@ -32,6 +32,7 @@
>   #include "qemu/osdep.h"
>   #include "qapi/error.h"
>   #include "hw/irq.h"
> +#include "hw/pci/pci_bus.h"
>   #include "hw/pci-host/gpex.h"
>   #include "hw/qdev-properties.h"
>   #include "migration/vmstate.h"
> @@ -50,7 +51,7 @@ static void gpex_set_irq(void *opaque, int irq_num, int level)
>   
>   int gpex_set_irq_num(GPEXHost *s, int index, int gsi)
>   {
> -    if (index >= GPEX_NUM_IRQS) {
> +    if (index >= s->nr_irqs) {
>           return -EINVAL;
>       }
>   
> @@ -74,14 +75,29 @@ static PCIINTxRoute gpex_route_intx_pin_to_irq(void *opaque, int pin)
>       return route;
>   }
>   
> +static int gpex_swizzle_map_irq_fn(PCIDevice *pci_dev, int pin)
> +{
> +    PCIBus *bus = pci_device_root_bus(pci_dev);
> +
> +    return (PCI_SLOT(pci_dev->devfn) + pin) % bus->nirq;
> +}
> +
>   static void gpex_host_realize(DeviceState *dev, Error **errp)
>   {
>       PCIHostState *pci = PCI_HOST_BRIDGE(dev);
>       GPEXHost *s = GPEX_HOST(dev);
>       SysBusDevice *sbd = SYS_BUS_DEVICE(dev);
>       PCIExpressHost *pex = PCIE_HOST_BRIDGE(dev);
> +    pci_map_irq_fn map_irq_fn = pci_swizzle_map_irq_fn;
>       int i;
>   
> +    s->irq = g_malloc0(s->nr_irqs * sizeof(*s->irq));
> +    s->irq_num = g_malloc0(s->nr_irqs * sizeof(*s->irq_num));

Use g_malloc0_n(). I also prefer to have one array of struct instead of 
allocating two arrays to save the number of allocations.

> +
> +    if (s->nr_irqs != PCI_NUM_PINS) {

I see little utlity with this conditional. There is no behavioral 
difference in these functions when s->nr_irqs == PCI_NUM_PINS. The 
nr_irqs property will not make sense if there is.

> +        map_irq_fn = gpex_swizzle_map_irq_fn;
> +    }
> +
>       pcie_host_mmcfg_init(pex, PCIE_MMCFG_SIZE_MAX);
>       sysbus_init_mmio(sbd, &pex->mmio);
>   
> @@ -128,19 +144,27 @@ static void gpex_host_realize(DeviceState *dev, Error **errp)
>           sysbus_init_mmio(sbd, &s->io_ioport);
>       }
>   
> -    for (i = 0; i < GPEX_NUM_IRQS; i++) {
> +    for (i = 0; i < s->nr_irqs; i++) {
>           sysbus_init_irq(sbd, &s->irq[i]);
>           s->irq_num[i] = -1;
>       }
>   
> -    pci->bus = pci_register_root_bus(dev, "pcie.0", gpex_set_irq,
> -                                     pci_swizzle_map_irq_fn, s, &s->io_mmio,
> -                                     &s->io_ioport, 0, 4, TYPE_PCIE_BUS);
> +    pci->bus = pci_register_root_bus(dev, "pcie.0", gpex_set_irq, map_irq_fn,
> +                                     s, &s->io_mmio, &s->io_ioport, 0,
> +                                     s->nr_irqs, TYPE_PCIE_BUS);
>   
>       pci_bus_set_route_irq_fn(pci->bus, gpex_route_intx_pin_to_irq);
>       qdev_realize(DEVICE(&s->gpex_root), BUS(pci->bus), &error_fatal);
>   }
>   
> +static void gpex_host_unrealize(DeviceState *dev)
> +{
> +    GPEXHost *s = GPEX_HOST(dev);
> +
> +    g_free(s->irq);
> +    g_free(s->irq_num);
> +}
> +
>   static const char *gpex_host_root_bus_path(PCIHostState *host_bridge,
>                                             PCIBus *rootbus)
>   {
> @@ -166,6 +190,7 @@ static Property gpex_host_properties[] = {
>                          gpex_cfg.mmio64.base, 0),
>       DEFINE_PROP_SIZE(PCI_HOST_ABOVE_4G_MMIO_SIZE, GPEXHost,
>                        gpex_cfg.mmio64.size, 0),
> +    DEFINE_PROP_UINT32("nr-irqs", GPEXHost, nr_irqs, PCI_NUM_PINS),

UINT32 is too big for the number of IRQs. I find code in many places use 
int for IRQ, which will break for a too big value.

Any IRQ value before swizzling fits in UINT8 so let's use it.

>       DEFINE_PROP_END_OF_LIST(),
>   };
>   
> @@ -176,6 +201,7 @@ static void gpex_host_class_init(ObjectClass *klass, void *data)
>   
>       hc->root_bus_path = gpex_host_root_bus_path;
>       dc->realize = gpex_host_realize;
> +    dc->unrealize = gpex_host_unrealize;
>       set_bit(DEVICE_CATEGORY_BRIDGE, dc->categories);
>       dc->fw_name = "pci";
>       device_class_set_props(dc, gpex_host_properties);
> diff --git a/hw/riscv/virt.c b/hw/riscv/virt.c
> index cef41c150aa..6c3ed4b3d9c 100644
> --- a/hw/riscv/virt.c
> +++ b/hw/riscv/virt.c
> @@ -167,7 +167,7 @@ static void create_pcie_irq_map(RISCVVirtState *s, void *fdt, char *nodename,
>   {
>       int pin, dev;
>       uint32_t irq_map_stride = 0;
> -    uint32_t full_irq_map[GPEX_NUM_IRQS * GPEX_NUM_IRQS *
> +    uint32_t full_irq_map[PCI_NUM_PINS * PCI_NUM_PINS *
>                             FDT_MAX_INT_MAP_WIDTH] = {};
>       uint32_t *irq_map = full_irq_map;
>   
> @@ -179,11 +179,11 @@ static void create_pcie_irq_map(RISCVVirtState *s, void *fdt, char *nodename,
>        * possible slot) seeing the interrupt-map-mask will allow the table
>        * to wrap to any number of devices.
>        */
> -    for (dev = 0; dev < GPEX_NUM_IRQS; dev++) {
> +    for (dev = 0; dev < PCI_NUM_PINS; dev++) {
>           int devfn = dev * 0x8;
>   
> -        for (pin = 0; pin < GPEX_NUM_IRQS; pin++) {
> -            int irq_nr = PCIE_IRQ + ((pin + PCI_SLOT(devfn)) % GPEX_NUM_IRQS);
> +        for (pin = 0; pin < PCI_NUM_PINS; pin++) {
> +            int irq_nr = PCIE_IRQ + ((pin + PCI_SLOT(devfn)) % PCI_NUM_PINS);
>               int i = 0;
>   
>               /* Fill PCI address cells */
> @@ -209,7 +209,7 @@ static void create_pcie_irq_map(RISCVVirtState *s, void *fdt, char *nodename,
>       }
>   
>       qemu_fdt_setprop(fdt, nodename, "interrupt-map", full_irq_map,
> -                     GPEX_NUM_IRQS * GPEX_NUM_IRQS *
> +                     PCI_NUM_PINS * PCI_NUM_PINS *
>                        irq_map_stride * sizeof(uint32_t));
>   
>       qemu_fdt_setprop_cells(fdt, nodename, "interrupt-map-mask",
> @@ -1157,7 +1157,7 @@ static inline DeviceState *gpex_pcie_init(MemoryRegion *sys_mem,
>   
>       sysbus_mmio_map(SYS_BUS_DEVICE(dev), 2, pio_base);
>   
> -    for (i = 0; i < GPEX_NUM_IRQS; i++) {
> +    for (i = 0; i < PCI_NUM_PINS; i++) {
>           irq = qdev_get_gpio_in(irqchip, PCIE_IRQ + i);
>   
>           sysbus_connect_irq(SYS_BUS_DEVICE(dev), i, irq);
> diff --git a/hw/xtensa/virt.c b/hw/xtensa/virt.c
> index 5310a888613..8f5c2009d29 100644
> --- a/hw/xtensa/virt.c
> +++ b/hw/xtensa/virt.c
> @@ -93,7 +93,7 @@ static void create_pcie(MachineState *ms, CPUXtensaState *env, int irq_base,
>       /* Connect IRQ lines. */
>       extints = xtensa_get_extints(env);
>   
> -    for (i = 0; i < GPEX_NUM_IRQS; i++) {
> +    for (i = 0; i < PCI_NUM_PINS; i++) {
>           void *q = extints[irq_base + i];
>   
>           sysbus_connect_irq(SYS_BUS_DEVICE(dev), i, q);
> diff --git a/include/hw/pci-host/gpex.h b/include/hw/pci-host/gpex.h
> index dce883573ba..8f87a3872cb 100644
> --- a/include/hw/pci-host/gpex.h
> +++ b/include/hw/pci-host/gpex.h
> @@ -32,8 +32,6 @@ OBJECT_DECLARE_SIMPLE_TYPE(GPEXHost, GPEX_HOST)
>   #define TYPE_GPEX_ROOT_DEVICE "gpex-root"
>   OBJECT_DECLARE_SIMPLE_TYPE(GPEXRootState, GPEX_ROOT_DEVICE)
>   
> -#define GPEX_NUM_IRQS 4
> -
>   struct GPEXRootState {
>       /*< private >*/
>       PCIDevice parent_obj;
> @@ -60,8 +58,9 @@ struct GPEXHost {
>       MemoryRegion io_mmio;
>       MemoryRegion io_ioport_window;
>       MemoryRegion io_mmio_window;
> -    qemu_irq irq[GPEX_NUM_IRQS];
> -    int irq_num[GPEX_NUM_IRQS];
> +    uint32_t nr_irqs;
> +    qemu_irq *irq;
> +    int *irq_num;
>   
>       bool allow_unmapped_accesses;
>   

Re: [PATCH v3 10/14] hw/vmapple/aes: Introduce aes engine

[Akihiko Odaki]

On 2024/09/28 17:57, Phil Dennis-Jordan wrote:
> From: Alexander Graf <graf@amazon.com>
> 
> VMApple contains an "aes" engine device that it uses to encrypt and
> decrypt its nvram. It has trivial hard coded keys it uses for that
> purpose.
> 
> Add device emulation for this device model.
> 
> Signed-off-by: Alexander Graf <graf@amazon.com>
> Co-authored-by: Phil Dennis-Jordan <phil@philjordan.eu>
> Signed-off-by: Phil Dennis-Jordan <phil@philjordan.eu>
> 
> ---
> v3:
> 
>   * Rebased on latest upstream and fixed minor breakages.
>   * Replaced legacy device reset method with Resettable method
> 
>   hw/vmapple/Kconfig      |   2 +
>   hw/vmapple/aes.c        | 584 ++++++++++++++++++++++++++++++++++++++++
>   hw/vmapple/meson.build  |   1 +
>   hw/vmapple/trace-events |  19 ++
>   4 files changed, 606 insertions(+)
>   create mode 100644 hw/vmapple/aes.c
> 
> diff --git a/hw/vmapple/Kconfig b/hw/vmapple/Kconfig
> index 8b137891791..a73504d5999 100644
> --- a/hw/vmapple/Kconfig
> +++ b/hw/vmapple/Kconfig
> @@ -1 +1,3 @@
> +config VMAPPLE_AES
> +    bool
>   
> diff --git a/hw/vmapple/aes.c b/hw/vmapple/aes.c
> new file mode 100644
> index 00000000000..074fbdd9c36
> --- /dev/null
> +++ b/hw/vmapple/aes.c
> @@ -0,0 +1,584 @@
> +/*
> + * QEMU Apple AES device emulation
> + *
> + * Copyright © 2023 Amazon.com, Inc. or its affiliates. All Rights Reserved.
> + *
> + * This work is licensed under the terms of the GNU GPL, version 2 or later.
> + * See the COPYING file in the top-level directory.
> + */
> +
> +#include "qemu/osdep.h"
> +#include "hw/irq.h"
> +#include "migration/vmstate.h"
> +#include "qemu/log.h"
> +#include "qemu/module.h"
> +#include "trace.h"
> +#include "hw/sysbus.h"
> +#include "crypto/hash.h"
> +#include "crypto/aes.h"
> +#include "crypto/cipher.h"
> +
> +#define TYPE_AES          "apple-aes"
> +#define MAX_FIFO_SIZE     9
> +
> +#define CMD_KEY           0x1
> +#define CMD_KEY_CONTEXT_SHIFT    27
> +#define CMD_KEY_CONTEXT_MASK     (0x1 << CMD_KEY_CONTEXT_SHIFT)
> +#define CMD_KEY_SELECT_SHIFT     24
> +#define CMD_KEY_SELECT_MASK      (0x7 << CMD_KEY_SELECT_SHIFT)
> +#define CMD_KEY_KEY_LEN_SHIFT    22
> +#define CMD_KEY_KEY_LEN_MASK     (0x3 << CMD_KEY_KEY_LEN_SHIFT)
> +#define CMD_KEY_ENCRYPT_SHIFT    20
> +#define CMD_KEY_ENCRYPT_MASK     (0x1 << CMD_KEY_ENCRYPT_SHIFT)
> +#define CMD_KEY_BLOCK_MODE_SHIFT 16
> +#define CMD_KEY_BLOCK_MODE_MASK  (0x3 << CMD_KEY_BLOCK_MODE_SHIFT)
> +#define CMD_IV            0x2
> +#define CMD_IV_CONTEXT_SHIFT     26
> +#define CMD_IV_CONTEXT_MASK      (0x3 << CMD_KEY_CONTEXT_SHIFT)
> +#define CMD_DSB           0x3
> +#define CMD_SKG           0x4
> +#define CMD_DATA          0x5
> +#define CMD_DATA_KEY_CTX_SHIFT   27
> +#define CMD_DATA_KEY_CTX_MASK    (0x1 << CMD_DATA_KEY_CTX_SHIFT)
> +#define CMD_DATA_IV_CTX_SHIFT    25
> +#define CMD_DATA_IV_CTX_MASK     (0x3 << CMD_DATA_IV_CTX_SHIFT)
> +#define CMD_DATA_LEN_MASK        0xffffff
> +#define CMD_STORE_IV      0x6
> +#define CMD_STORE_IV_ADDR_MASK   0xffffff
> +#define CMD_WRITE_REG     0x7
> +#define CMD_FLAG          0x8
> +#define CMD_FLAG_STOP_MASK       BIT(26)
> +#define CMD_FLAG_RAISE_IRQ_MASK  BIT(27)
> +#define CMD_FLAG_INFO_MASK       0xff
> +#define CMD_MAX           0x10
> +
> +#define CMD_SHIFT         28
> +
> +#define REG_STATUS            0xc
> +#define REG_STATUS_DMA_READ_RUNNING     BIT(0)
> +#define REG_STATUS_DMA_READ_PENDING     BIT(1)
> +#define REG_STATUS_DMA_WRITE_RUNNING    BIT(2)
> +#define REG_STATUS_DMA_WRITE_PENDING    BIT(3)
> +#define REG_STATUS_BUSY                 BIT(4)
> +#define REG_STATUS_EXECUTING            BIT(5)
> +#define REG_STATUS_READY                BIT(6)
> +#define REG_STATUS_TEXT_DPA_SEEDED      BIT(7)
> +#define REG_STATUS_UNWRAP_DPA_SEEDED    BIT(8)
> +
> +#define REG_IRQ_STATUS        0x18
> +#define REG_IRQ_STATUS_INVALID_CMD      BIT(2)
> +#define REG_IRQ_STATUS_FLAG             BIT(5)
> +#define REG_IRQ_ENABLE        0x1c
> +#define REG_WATERMARK         0x20
> +#define REG_Q_STATUS          0x24
> +#define REG_FLAG_INFO         0x30
> +#define REG_FIFO              0x200
> +
> +static const uint32_t key_lens[4] = {

I suggest deriving the size from CMD_KEY_KEY_LEN_MASK.

> +    [0] = 16,
> +    [1] = 24,
> +    [2] = 32,
> +    [3] = 64,
> +};
> +
> +struct key {

Please add typedef.

> +    uint32_t key_len;
> +    uint32_t key[8];
> +};
> +
> +struct iv {
> +    uint32_t iv[4];
> +};
> +
> +struct context {
> +    struct key key;
> +    struct iv iv;
> +};
> +
> +static struct key builtin_keys[7] = {
 > +    [1] = {> +        .key_len = 32,
> +        .key = { 0x1 },
> +    },
> +    [2] = {
> +        .key_len = 32,
> +        .key = { 0x2 },
> +    },
> +    [3] = {
> +        .key_len = 32,
> +        .key = { 0x3 },
> +    }
> +};
> +
> +typedef struct AESState {
> +    /* Private */

This private/public comments are unnecessary; this struct can be 
referenced only from this file and all members are automatically private.

> +    SysBusDevice parent_obj;
> +
> +    /* Public */
> +    qemu_irq irq;
> +    MemoryRegion iomem1;
> +    MemoryRegion iomem2;
> +
> +    uint32_t status;
> +    uint32_t q_status;
> +    uint32_t irq_status;
> +    uint32_t irq_enable;
> +    uint32_t watermark;
> +    uint32_t flag_info;
> +    uint32_t fifo[MAX_FIFO_SIZE];
> +    uint32_t fifo_idx;
> +    struct key key[2];
> +    struct iv iv[4];
> +    bool is_encrypt;
> +    QCryptoCipherMode block_mode;
> +} AESState;
> +
> +OBJECT_DECLARE_SIMPLE_TYPE(AESState, AES)
> +
> +static void aes_update_irq(AESState *s)
> +{
> +    qemu_set_irq(s->irq, !!(s->irq_status & s->irq_enable));
> +}
> +
> +static uint64_t aes1_read(void *opaque, hwaddr offset, unsigned size)
> +{
> +    AESState *s = opaque;
> +    uint64_t res = 0;
> +
> +    switch (offset) {
> +    case REG_STATUS:
> +        res = s->status;
> +        break;
> +    case REG_IRQ_STATUS:
> +        res = s->irq_status;
> +        break;
> +    case REG_IRQ_ENABLE:
> +        res = s->irq_enable;
> +        break;
> +    case REG_WATERMARK:
> +        res = s->watermark;
> +        break;
> +    case REG_Q_STATUS:
> +        res = s->q_status;
> +        break;
> +    case REG_FLAG_INFO:
> +        res = s->flag_info;
> +        break;
> +
> +    default:
> +        trace_aes_read_unknown(offset);

Use: LOG_UNIMP

> +        break;
> +    }
> +
> +    trace_aes_read(offset, res);
> +
> +    return res;
> +}
> +
> +static void fifo_append(AESState *s, uint64_t val)
> +{
> +    if (s->fifo_idx == MAX_FIFO_SIZE) {
> +        /* Exceeded the FIFO. Bail out */
> +        return;
> +    }
> +
> +    s->fifo[s->fifo_idx++] = val;
> +}
> +
> +static bool has_payload(AESState *s, uint32_t elems)
> +{
> +    return s->fifo_idx >= (elems + 1);
> +}
> +
> +static bool cmd_key(AESState *s)
> +{
> +    uint32_t cmd = s->fifo[0];
> +    uint32_t key_select = (cmd & CMD_KEY_SELECT_MASK) >> CMD_KEY_SELECT_SHIFT;
> +    uint32_t ctxt = (cmd & CMD_KEY_CONTEXT_MASK) >> CMD_KEY_CONTEXT_SHIFT;
> +    uint32_t key_len;
> +
> +    switch ((cmd & CMD_KEY_BLOCK_MODE_MASK) >> CMD_KEY_BLOCK_MODE_SHIFT) {
> +    case 0:
> +        s->block_mode = QCRYPTO_CIPHER_MODE_ECB;
> +        break;
> +    case 1:
> +        s->block_mode = QCRYPTO_CIPHER_MODE_CBC;
> +        break;
> +    default:
> +        return false;
> +    }
> +
> +    s->is_encrypt = !!((cmd & CMD_KEY_ENCRYPT_MASK) >> CMD_KEY_ENCRYPT_SHIFT);

CMD_KEY_ENCRYPT_SHIFT is unnecessary as in case of 
CMD_FLAG_RAISE_IRQ_MASK. "!!" is also unnecessary; it will be implicitly 
casted into bool.

> +    key_len = key_lens[((cmd & CMD_KEY_KEY_LEN_MASK) >> CMD_KEY_KEY_LEN_SHIFT)];
> +
> +    if (key_select) {
> +        trace_aes_cmd_key_select_builtin(ctxt, key_select,
> +                                         s->is_encrypt ? "en" : "de",
> +                                         QCryptoCipherMode_str(s->block_mode));
> +        s->key[ctxt] = builtin_keys[key_select];

I guess this should be: builtin_keys[key_select - 1]

> +    } else {
> +        trace_aes_cmd_key_select_new(ctxt, key_len,
> +                                     s->is_encrypt ? "en" : "de",
> +                                     QCryptoCipherMode_str(s->block_mode));
> +        if (key_len > sizeof(s->key[ctxt].key)) {
> +            return false;
> +        }
> +        if (!has_payload(s, key_len / sizeof(uint32_t))) {
> +            /* wait for payload */
> +            return false;
> +        }
> +        memcpy(&s->key[ctxt].key, &s->fifo[1], key_len);
> +        s->key[ctxt].key_len = key_len;
> +    }
> +
> +    return true;
> +}
> +
> +static bool cmd_iv(AESState *s)
> +{
> +    uint32_t cmd = s->fifo[0];
> +    uint32_t ctxt = (cmd & CMD_IV_CONTEXT_MASK) >> CMD_IV_CONTEXT_SHIFT;
> +
> +    if (!has_payload(s, 4)) {
> +        /* wait for payload */
> +        return false;
> +    }
> +    memcpy(&s->iv[ctxt].iv, &s->fifo[1], sizeof(s->iv[ctxt].iv));
> +    trace_aes_cmd_iv(ctxt, s->fifo[1], s->fifo[2], s->fifo[3], s->fifo[4]);
> +
> +    return true;
> +}
> +
> +static char hexdigit2str(uint8_t val)
> +{
> +    g_assert(val < 0x10);
> +    if (val >= 0xa) {
> +        return 'a' + (val - 0xa);
> +    } else {
> +        return '0' + val;
> +    }
> +}
> +
> +static void dump_data(const char *desc, const void *p, size_t len)
> +{
> +    char *hex = alloca((len * 2) + 1);

Don't use alloca(); use a fixed-length array instead.

> +    const uint8_t *data = p;
> +    char *hexp = hex;
> +    size_t i;
> +
> +    if (len > 0x1000) {
> +        /* Too large buffer, let's bail out */
> +        return;
> +    }
> +
> +    for (i = 0; i < len; i++) {
> +        uint8_t val = data[i];
> +        *(hexp++) = hexdigit2str(val >> 4);
> +        *(hexp++) = hexdigit2str(val & 0xf);
> +    }
> +    *hexp = '\0';

Let's move this to: util/hexdump.c

> +
> +    trace_aes_dump_data(desc, hex);
> +}
> +
> +static bool cmd_data(AESState *s)
> +{
> +    uint32_t cmd = s->fifo[0];
> +    uint32_t ctxt_iv = 0;
> +    uint32_t ctxt_key = (cmd & CMD_DATA_KEY_CTX_MASK) >> CMD_DATA_KEY_CTX_SHIFT;
> +    uint32_t len = cmd & CMD_DATA_LEN_MASK;
> +    uint64_t src_addr = s->fifo[2];
> +    uint64_t dst_addr = s->fifo[3];
> +    QCryptoCipherAlgo alg;
> +    QCryptoCipher *cipher;
> +    char *src;
> +    char *dst;
> +
> +    src_addr |= ((uint64_t)s->fifo[1] << 16) & 0xffff00000000ULL;
> +    dst_addr |= ((uint64_t)s->fifo[1] << 32) & 0xffff00000000ULL;
> +
> +    trace_aes_cmd_data(ctxt_key, ctxt_iv, src_addr, dst_addr, len);
> +
> +    if (!has_payload(s, 3)) {
> +        /* wait for payload */
> +        trace_aes_cmd_data_error("No payload");

Use: LOG_GUEST_ERROR

> +        return false;
> +    }
> +
> +    if (ctxt_key >= ARRAY_SIZE(s->key) ||
> +        ctxt_iv >= ARRAY_SIZE(s->iv)) {
> +        /* Invalid input */
> +        trace_aes_cmd_data_error("Invalid key or iv");
> +        return false;
> +    }
> +
> +    src = g_malloc0(len);
> +    dst = g_malloc0(len);
> +
> +    cpu_physical_memory_read(src_addr, src, len);

Use: dma_memory_read()

> +
> +    dump_data("cmd_data(): src_data=", src, len);
> +
> +    switch (s->key[ctxt_key].key_len) {
> +    case 128 / 8:
> +        alg = QCRYPTO_CIPHER_ALGO_AES_128;
> +        break;
> +    case 192 / 8:
> +        alg = QCRYPTO_CIPHER_ALGO_AES_192;
> +        break;
> +    case 256 / 8:
> +        alg = QCRYPTO_CIPHER_ALGO_AES_256;
> +        break;
> +    default:
> +        trace_aes_cmd_data_error("Invalid key len");
> +        goto err_free;

Use g_autoptr instead of goto and g_free().

> +    }
> +    cipher = qcrypto_cipher_new(alg, s->block_mode,
> +                                (void *)s->key[ctxt_key].key,
> +                                s->key[ctxt_key].key_len, NULL);
> +    g_assert(cipher != NULL);
Handle this error as you do for qcrypto_cipher_setiv(), 
qcrypto_cipher_encrypt() and qcrypto_cipher_decrypt().

> +    if (s->block_mode != QCRYPTO_CIPHER_MODE_ECB) {
> +        if (qcrypto_cipher_setiv(cipher, (void *)s->iv[ctxt_iv].iv,
> +                                 sizeof(s->iv[ctxt_iv].iv), NULL) != 0) {
> +            trace_aes_cmd_data_error("Failed to set IV");
> +            goto err_free_cipher;
> +        }
> +    }
> +    if (s->is_encrypt) {
> +        if (qcrypto_cipher_encrypt(cipher, src, dst, len, NULL) != 0) {
> +            trace_aes_cmd_data_error("Encrypt failed");
> +            goto err_free_cipher;
> +        }
> +    } else {
> +        if (qcrypto_cipher_decrypt(cipher, src, dst, len, NULL) != 0) {
> +            trace_aes_cmd_data_error("Decrypt failed");
> +            goto err_free_cipher;
> +        }
> +    }
> +    qcrypto_cipher_free(cipher);
> +
> +    dump_data("cmd_data(): dst_data=", dst, len);
> +    cpu_physical_memory_write(dst_addr, dst, len);
> +    g_free(src);
> +    g_free(dst);
> +
> +    return true;
> +
> +err_free_cipher:
> +    qcrypto_cipher_free(cipher);
> +err_free:
> +    g_free(src);
> +    g_free(dst);
> +    return false;
> +}
> +
> +static bool cmd_store_iv(AESState *s)
> +{
> +    uint32_t cmd = s->fifo[0];
> +    uint32_t ctxt = (cmd & CMD_IV_CONTEXT_MASK) >> CMD_IV_CONTEXT_SHIFT;
> +    uint64_t addr = s->fifo[1];
> +
> +    if (!has_payload(s, 1)) {
> +        /* wait for payload */

Add a log as it's done for cmd_data().

> +        return false;
> +    }
> +
> +    if (ctxt >= ARRAY_SIZE(s->iv)) {
> +        /* Invalid context selected */
> +        return false;
> +    }
> +
> +    addr |= ((uint64_t)cmd << 32) & 0xff00000000ULL;
> +    cpu_physical_memory_write(addr, &s->iv[ctxt].iv, sizeof(s->iv[ctxt].iv));
> +
> +    trace_aes_cmd_store_iv(ctxt, addr, s->iv[ctxt].iv[0], s->iv[ctxt].iv[1],
> +                           s->iv[ctxt].iv[2], s->iv[ctxt].iv[3]);
> +
> +    return true;
> +}
> +
> +static bool cmd_flag(AESState *s)
> +{
> +    uint32_t cmd = s->fifo[0];
> +    uint32_t raise_irq = cmd & CMD_FLAG_RAISE_IRQ_MASK;
> +
> +    /* We always process data when it's coming in, so fire an IRQ immediately */
> +    if (raise_irq) {
> +        s->irq_status |= REG_IRQ_STATUS_FLAG;
> +    }
> +
> +    s->flag_info = cmd & CMD_FLAG_INFO_MASK;
> +
> +    trace_aes_cmd_flag(!!raise_irq, s->flag_info);
> +
> +    return true;
> +}
> +
> +static void fifo_process(AESState *s)
> +{
> +    uint32_t cmd = s->fifo[0] >> CMD_SHIFT;
> +    bool success = false;
> +
> +    if (!s->fifo_idx) {
> +        return;
> +    }
> +
> +    switch (cmd) {
> +    case CMD_KEY:
> +        success = cmd_key(s);
> +        break;
> +    case CMD_IV:
> +        success = cmd_iv(s);
> +        break;
> +    case CMD_DATA:
> +        success = cmd_data(s);
> +        break;
> +    case CMD_STORE_IV:
> +        success = cmd_store_iv(s);
> +        break;
> +    case CMD_FLAG:
> +        success = cmd_flag(s);
> +        break;
> +    default:
> +        s->irq_status |= REG_IRQ_STATUS_INVALID_CMD;
> +        break;
> +    }
> +
> +    if (success) {
> +        s->fifo_idx = 0;
> +    }
> +
> +    trace_aes_fifo_process(cmd, success ? 1 : 0);
> +}
> +
> +static void aes1_write(void *opaque, hwaddr offset, uint64_t val, unsigned size)
> +{
> +    AESState *s = opaque;
> +
> +    trace_aes_write(offset, val);
> +
> +    switch (offset) {
> +    case REG_IRQ_STATUS:
> +        s->irq_status &= ~val;
> +        break;
> +    case REG_IRQ_ENABLE:
> +        s->irq_enable = val;
> +        break;
> +    case REG_FIFO:
> +        fifo_append(s, val);
> +        fifo_process(s);
> +        break;
> +    default:
> +        trace_aes_write_unknown(offset);
> +        return;
> +    }
> +
> +    aes_update_irq(s);
> +}
> +
> +static const MemoryRegionOps aes1_ops = {
> +    .read = aes1_read,
> +    .write = aes1_write,
> +    .endianness = DEVICE_NATIVE_ENDIAN,
> +    .valid = {
> +        .min_access_size = 4,
> +        .max_access_size = 8,
> +    },
> +    .impl = {
> +        .min_access_size = 4,
> +        .max_access_size = 4,
> +    },
> +};
> +
> +static uint64_t aes2_read(void *opaque, hwaddr offset, unsigned size)
> +{
> +    uint64_t res = 0;
> +
> +    switch (offset) {
> +    case 0:
> +        res = 0;
> +        break;
> +    default:
> +        trace_aes_2_read_unknown(offset);
> +        break;
> +    }
> +
> +    trace_aes_2_read(offset, res);
> +
> +    return res;
> +}
> +
> +static void aes2_write(void *opaque, hwaddr offset, uint64_t val, unsigned size)
> +{
> +    trace_aes_2_write(offset, val);
> +
> +    switch (offset) {
> +    default:
> +        trace_aes_2_write_unknown(offset);
> +        return;
> +    }
> +}
> +
> +static const MemoryRegionOps aes2_ops = {
> +    .read = aes2_read,
> +    .write = aes2_write,
> +    .endianness = DEVICE_NATIVE_ENDIAN,
> +    .valid = {
> +        .min_access_size = 4,
> +        .max_access_size = 8,
> +    },
> +    .impl = {
> +        .min_access_size = 4,
> +        .max_access_size = 4,
> +    },
> +};
> +
> +static void aes_reset(Object *obj, ResetType type)
> +{
> +    AESState *s = AES(obj);
> +
> +    s->status = 0x3f80;
> +    s->q_status = 2;
> +    s->irq_status = 0;
> +    s->irq_enable = 0;
> +    s->watermark = 0;
> +}
> +
> +static void aes_init(Object *obj)
> +{
> +    AESState *s = AES(obj);
> +
> +    memory_region_init_io(&s->iomem1, obj, &aes1_ops, s, TYPE_AES, 0x4000);
> +    memory_region_init_io(&s->iomem2, obj, &aes2_ops, s, TYPE_AES, 0x4000);
> +    sysbus_init_mmio(SYS_BUS_DEVICE(s), &s->iomem1);
> +    sysbus_init_mmio(SYS_BUS_DEVICE(s), &s->iomem2);
> +    sysbus_init_irq(SYS_BUS_DEVICE(s), &s->irq);
> +}
> +
> +static void aes_realize(DeviceState *dev, Error **errp)
> +{
> +}

This shouldn't be necessary.

> +
> +static void aes_class_init(ObjectClass *klass, void *data)
> +{
> +    DeviceClass *dc = DEVICE_CLASS(klass);
> +    ResettableClass *rc = RESETTABLE_CLASS(klass);
> +
> +    rc->phases.hold = aes_reset;
> +    dc->realize = aes_realize;
> +}
> +
> +static const TypeInfo aes_info = {
> +    .name          = TYPE_AES,
> +    .parent        = TYPE_SYS_BUS_DEVICE,
> +    .instance_size = sizeof(AESState),
> +    .class_init    = aes_class_init,
> +    .instance_init = aes_init,
> +};
> +
> +static void aes_register_types(void)
> +{
> +    type_register_static(&aes_info);
> +}
> +
> +type_init(aes_register_types)
> diff --git a/hw/vmapple/meson.build b/hw/vmapple/meson.build
> index e69de29bb2d..bcd4dcb28d2 100644
> --- a/hw/vmapple/meson.build
> +++ b/hw/vmapple/meson.build
> @@ -0,0 +1 @@
> +system_ss.add(when: 'CONFIG_VMAPPLE_AES',  if_true: files('aes.c'))
> diff --git a/hw/vmapple/trace-events b/hw/vmapple/trace-events
> index 9ccc5790487..1c9a3326eb4 100644
> --- a/hw/vmapple/trace-events
> +++ b/hw/vmapple/trace-events
> @@ -1,2 +1,21 @@
>   # See docs/devel/tracing.rst for syntax documentation.
>   
> +# aes.c
> +aes_read_unknown(uint64_t offset) "offset=0x%"PRIx64
> +aes_read(uint64_t offset, uint64_t res) "offset=0x%"PRIx64" res=0x%"PRIx64
> +aes_cmd_key_select_builtin(uint32_t ctx, uint32_t key_id, const char *direction, const char *cipher) "[%d] Selecting builtin key %d to %scrypt with %s"
> +aes_cmd_key_select_new(uint32_t ctx, uint32_t key_len, const char *direction, const char *cipher) "[%d] Selecting new key size=%d to %scrypt with %s"
> +aes_cmd_iv(uint32_t ctx, uint32_t iv0, uint32_t iv1, uint32_t iv2, uint32_t iv3) "[%d] 0x%08x 0x%08x 0x%08x 0x%08x"
> +aes_cmd_data(uint32_t key, uint32_t iv, uint64_t src, uint64_t dst, uint32_t len) "[key=%d iv=%d] src=0x%"PRIx64" dst=0x%"PRIx64" len=0x%x"
> +aes_cmd_data_error(const char *reason) "reason=%s"
> +aes_cmd_store_iv(uint32_t ctx, uint64_t addr, uint32_t iv0, uint32_t iv1, uint32_t iv2, uint32_t iv3) "[%d] addr=0x%"PRIx64"x -> 0x%08x 0x%08x 0x%08x 0x%08x"
> +aes_cmd_flag(uint32_t raise, uint32_t flag_info) "raise=%d flag_info=0x%x"
> +aes_fifo_process(uint32_t cmd, uint32_t success) "cmd=%d success=%d"
> +aes_write_unknown(uint64_t offset) "offset=0x%"PRIx64
> +aes_write(uint64_t offset, uint64_t val) "offset=0x%"PRIx64" val=0x%"PRIx64
> +aes_2_read_unknown(uint64_t offset) "offset=0x%"PRIx64
> +aes_2_read(uint64_t offset, uint64_t res) "offset=0x%"PRIx64" res=0x%"PRIx64
> +aes_2_write_unknown(uint64_t offset) "offset=0x%"PRIx64
> +aes_2_write(uint64_t offset, uint64_t val) "offset=0x%"PRIx64" val=0x%"PRIx64
> +aes_dump_data(const char *desc, const char *hex) "%s%s"
> +

Re: [PATCH v3 11/14] hw/vmapple/bdif: Introduce vmapple backdoor interface

[Akihiko Odaki]

On 2024/09/28 17:57, Phil Dennis-Jordan wrote:
> From: Alexander Graf <graf@amazon.com>
> 
> The VMApple machine exposes AUX and ROOT block devices (as well as USB OTG
> emulation) via virtio-pci as well as a special, simple backdoor platform
> device.
> 
> This patch implements this backdoor platform device to the best of my
> understanding. I left out any USB OTG parts; they're only needed for
> guest recovery and I don't understand the protocol yet.
> 
> Signed-off-by: Alexander Graf <graf@amazon.com>
> Signed-off-by: Phil Dennis-Jordan <phil@philjordan.eu>
> ---
>   hw/vmapple/Kconfig        |   3 +
>   hw/vmapple/bdif.c         | 245 ++++++++++++++++++++++++++++++++++++++
>   hw/vmapple/meson.build    |   1 +
>   hw/vmapple/trace-events   |   5 +
>   include/hw/vmapple/bdif.h |  31 +++++
>   5 files changed, 285 insertions(+)
>   create mode 100644 hw/vmapple/bdif.c
>   create mode 100644 include/hw/vmapple/bdif.h
> 
> diff --git a/hw/vmapple/Kconfig b/hw/vmapple/Kconfig
> index a73504d5999..68f88876eb9 100644
> --- a/hw/vmapple/Kconfig
> +++ b/hw/vmapple/Kconfig
> @@ -1,3 +1,6 @@
>   config VMAPPLE_AES
>       bool
>   
> +config VMAPPLE_BDIF
> +    bool
> +
> diff --git a/hw/vmapple/bdif.c b/hw/vmapple/bdif.c
> new file mode 100644
> index 00000000000..36b5915ff30
> --- /dev/null
> +++ b/hw/vmapple/bdif.c
> @@ -0,0 +1,245 @@
> +/*
> + * VMApple Backdoor Interface
> + *
> + * Copyright © 2023 Amazon.com, Inc. or its affiliates. All Rights Reserved.
> + *
> + * This work is licensed under the terms of the GNU GPL, version 2 or later.
> + * See the COPYING file in the top-level directory.
> + */
> +
> +#include "qemu/osdep.h"
> +#include "hw/vmapple/bdif.h"
> +#include "qemu/log.h"
> +#include "qemu/module.h"
> +#include "qapi/error.h"
> +#include "trace.h"
> +#include "hw/block/block.h"
> +#include "sysemu/block-backend.h"
> +
> +#define REG_DEVID_MASK      0xffff0000
> +#define DEVID_ROOT          0x00000000
> +#define DEVID_AUX           0x00010000
> +#define DEVID_USB           0x00100000
> +
> +#define REG_STATUS          0x0
> +#define REG_STATUS_ACTIVE     BIT(0)
> +#define REG_CFG             0x4
> +#define REG_CFG_ACTIVE        BIT(1)
> +#define REG_UNK1            0x8
> +#define REG_BUSY            0x10
> +#define REG_BUSY_READY        BIT(0)
> +#define REG_UNK2            0x400
> +#define REG_CMD             0x408
> +#define REG_NEXT_DEVICE     0x420
> +#define REG_UNK3            0x434
> +
> +typedef struct vblk_sector {

Please use VblkSector for the tag name too.

> +    uint32_t pad;
> +    uint32_t pad2;
> +    uint32_t sector;
> +    uint32_t pad3;
> +} VblkSector;
> +
> +typedef struct vblk_req_cmd {
> +    uint64_t addr;
> +    uint32_t len;
> +    uint32_t flags;
> +} VblkReqCmd;
> +
> +typedef struct vblk_req {
> +    VblkReqCmd sector;
> +    VblkReqCmd data;
> +    VblkReqCmd retval;
> +} VblkReq;
> +
> +#define VBLK_DATA_FLAGS_READ  0x00030001
> +#define VBLK_DATA_FLAGS_WRITE 0x00010001
> +
> +#define VBLK_RET_SUCCESS  0
> +#define VBLK_RET_FAILED   1
> +
> +static uint64_t bdif_read(void *opaque, hwaddr offset, unsigned size)
> +{
> +    uint64_t ret = -1;
> +    uint64_t devid = (offset & REG_DEVID_MASK);

The parenthes in this line are unnecessary.

> +
> +    switch (offset & ~REG_DEVID_MASK) {
> +    case REG_STATUS:
> +        ret = REG_STATUS_ACTIVE;
> +        break;
> +    case REG_CFG:
> +        ret = REG_CFG_ACTIVE;
> +        break;
> +    case REG_UNK1:
> +        ret = 0x420;
> +        break;
> +    case REG_BUSY:
> +        ret = REG_BUSY_READY;
> +        break;
> +    case REG_UNK2:
> +        ret = 0x1;
> +        break;
> +    case REG_UNK3:
> +        ret = 0x0;
> +        break;
> +    case REG_NEXT_DEVICE:
> +        switch (devid) {
> +        case DEVID_ROOT:
> +            ret = 0x8000000;
> +            break;
> +        case DEVID_AUX:
> +            ret = 0x10000;
> +            break;
> +        }
> +        break;
> +    }
> +
> +    trace_bdif_read(offset, size, ret);
> +    return ret;
> +}
> +
> +static void le2cpu_sector(VblkSector *sector)
> +{
> +    sector->sector = le32_to_cpu(sector->sector);
> +}
> +
> +static void le2cpu_reqcmd(VblkReqCmd *cmd)
> +{
> +    cmd->addr = le64_to_cpu(cmd->addr);
> +    cmd->len = le32_to_cpu(cmd->len);
> +    cmd->flags = le32_to_cpu(cmd->flags);
> +}
> +
> +static void le2cpu_req(VblkReq *req)
> +{
> +    le2cpu_reqcmd(&req->sector);
> +    le2cpu_reqcmd(&req->data);
> +    le2cpu_reqcmd(&req->retval);
> +}
> +
> +static void vblk_cmd(uint64_t devid, BlockBackend *blk, uint64_t value,
> +                     uint64_t static_off)
> +{
> +    VblkReq req;
> +    VblkSector sector;
> +    uint64_t off = 0;
> +    char *buf = NULL;
> +    uint8_t ret = VBLK_RET_FAILED;
> +    int r;
> +
> +    cpu_physical_memory_read(value, &req, sizeof(req));
> +    le2cpu_req(&req);
> +
> +    if (req.sector.len != sizeof(sector)) {
> +        ret = VBLK_RET_FAILED;
> +        goto out;
> +    }
> +
> +    /* Read the vblk command */
> +    cpu_physical_memory_read(req.sector.addr, &sector, sizeof(sector));
> +    le2cpu_sector(&sector);
> +
> +    off = sector.sector * 512ULL + static_off;
> +
> +    /* Sanity check that we're not allocating bogus sizes */
> +    if (req.data.len > (128 * 1024 * 1024)) {

Use MiB defined in: include/qemu/units.h
The parentheses on the right hand are also unnecessary.

> +        goto out;
> +    }
> +
> +    buf = g_malloc0(req.data.len);
> +    switch (req.data.flags) {
> +    case VBLK_DATA_FLAGS_READ:
> +        r = blk_pread(blk, off, req.data.len, buf, 0);
> +        trace_bdif_vblk_read(devid == DEVID_AUX ? "aux" : "root",
> +                             req.data.addr, off, req.data.len, r);
> +        if (r < 0) {
> +            goto out;
> +        }
> +        cpu_physical_memory_write(req.data.addr, buf, req.data.len);
> +        ret = VBLK_RET_SUCCESS;
> +        break;
> +    case VBLK_DATA_FLAGS_WRITE:
> +        /* Not needed, iBoot only reads */
> +        break;
> +    default:
> +        break;
> +    }
> +
> +out:
> +    g_free(buf);
> +    cpu_physical_memory_write(req.retval.addr, &ret, 1);
> +}
> +
> +static void bdif_write(void *opaque, hwaddr offset,
> +                       uint64_t value, unsigned size)
> +{
> +    VMAppleBdifState *s = opaque;
> +    uint64_t devid = (offset & REG_DEVID_MASK);
> +
> +    trace_bdif_write(offset, size, value);
> +
> +    switch (offset & ~REG_DEVID_MASK) {
> +    case REG_CMD:
> +        switch (devid) {
> +        case DEVID_ROOT:
> +            vblk_cmd(devid, s->root, value, 0x0);
> +            break;
> +        case DEVID_AUX:
> +            vblk_cmd(devid, s->aux, value, 0x0);
> +            break;
> +        }
> +        break;
> +    }
> +}
> +
> +static const MemoryRegionOps bdif_ops = {
> +    .read = bdif_read,
> +    .write = bdif_write,
> +    .endianness = DEVICE_NATIVE_ENDIAN,
> +    .valid = {
> +        .min_access_size = 1,
> +        .max_access_size = 8,
> +    },
> +    .impl = {
> +        .min_access_size = 1,
> +        .max_access_size = 8,
> +    },
> +};
> +
> +static void bdif_init(Object *obj)
> +{
> +    VMAppleBdifState *s = VMAPPLE_BDIF(obj);
> +
> +    memory_region_init_io(&s->mmio, obj, &bdif_ops, obj,
> +                         "VMApple Backdoor Interface", VMAPPLE_BDIF_SIZE);
> +    sysbus_init_mmio(SYS_BUS_DEVICE(obj), &s->mmio);
> +}
> +
> +static Property bdif_properties[] = {
> +    DEFINE_PROP_DRIVE("aux", VMAppleBdifState, aux),
> +    DEFINE_PROP_DRIVE("root", VMAppleBdifState, root),
> +    DEFINE_PROP_END_OF_LIST(),
> +};
> +
> +static void bdif_class_init(ObjectClass *klass, void *data)
> +{
> +    DeviceClass *dc = DEVICE_CLASS(klass);
> +
> +    dc->desc = "VMApple Backdoor Interface";
> +    device_class_set_props(dc, bdif_properties);
> +}
> +
> +static const TypeInfo bdif_info = {
> +    .name          = TYPE_VMAPPLE_BDIF,
> +    .parent        = TYPE_SYS_BUS_DEVICE,
> +    .instance_size = sizeof(VMAppleBdifState),
> +    .instance_init = bdif_init,
> +    .class_init    = bdif_class_init,
> +};
> +
> +static void bdif_register_types(void)
> +{
> +    type_register_static(&bdif_info);
> +}
> +
> +type_init(bdif_register_types)
> diff --git a/hw/vmapple/meson.build b/hw/vmapple/meson.build
> index bcd4dcb28d2..d4624713deb 100644
> --- a/hw/vmapple/meson.build
> +++ b/hw/vmapple/meson.build
> @@ -1 +1,2 @@
>   system_ss.add(when: 'CONFIG_VMAPPLE_AES',  if_true: files('aes.c'))
> +system_ss.add(when: 'CONFIG_VMAPPLE_BDIF', if_true: files('bdif.c'))
> diff --git a/hw/vmapple/trace-events b/hw/vmapple/trace-events
> index 1c9a3326eb4..fc8e9cc5897 100644
> --- a/hw/vmapple/trace-events
> +++ b/hw/vmapple/trace-events
> @@ -19,3 +19,8 @@ aes_2_write_unknown(uint64_t offset) "offset=0x%"PRIx64
>   aes_2_write(uint64_t offset, uint64_t val) "offset=0x%"PRIx64" val=0x%"PRIx64
>   aes_dump_data(const char *desc, const char *hex) "%s%s"
>   
> +# bdif.c
> +bdif_read(uint64_t offset, uint32_t size, uint64_t value) "offset=0x%"PRIx64" size=0x%x value=0x%"PRIx64
> +bdif_write(uint64_t offset, uint32_t size, uint64_t value) "offset=0x%"PRIx64" size=0x%x value=0x%"PRIx64
> +bdif_vblk_read(const char *dev, uint64_t addr, uint64_t offset, uint32_t len, int r) "dev=%s addr=0x%"PRIx64" off=0x%"PRIx64" size=0x%x r=%d"
> +
> diff --git a/include/hw/vmapple/bdif.h b/include/hw/vmapple/bdif.h
> new file mode 100644
> index 00000000000..65ee43457b9
> --- /dev/null
> +++ b/include/hw/vmapple/bdif.h
> @@ -0,0 +1,31 @@
> +/*
> + * VMApple Backdoor Interface
> + *
> + * Copyright © 2023 Amazon.com, Inc. or its affiliates. All Rights Reserved.
> + *
> + * This work is licensed under the terms of the GNU GPL, version 2 or later.
> + * See the COPYING file in the top-level directory.
> + */
> +
> +#ifndef HW_VMAPPLE_BDIF_H
> +#define HW_VMAPPLE_BDIF_H
> +
> +#include "hw/sysbus.h"
> +#include "qom/object.h"
> +
> +#define TYPE_VMAPPLE_BDIF "vmapple-bdif"
> +OBJECT_DECLARE_SIMPLE_TYPE(VMAppleBdifState, VMAPPLE_BDIF)
> +
> +struct VMAppleBdifState {
> +    /* <private> */
> +    SysBusDevice parent_obj;
> +
> +    /* <public> */
> +    BlockBackend *aux;
> +    BlockBackend *root;
> +    MemoryRegion mmio;
> +};
> +
> +#define VMAPPLE_BDIF_SIZE 0x00200000

Please move VMAppleBdifState and VMAPPLE_BDIF_SIZE into: hw/vmapple/bdif.c
They are both private.

> +
> +#endif /* HW_VMAPPLE_BDIF_H */

Re: [PATCH v3 12/14] hw/vmapple/cfg: Introduce vmapple cfg region

[Akihiko Odaki]

On 2024/09/28 17:57, Phil Dennis-Jordan wrote:
> From: Alexander Graf <graf@amazon.com>
> 
> Instead of device tree or other more standardized means, VMApple passes
> platform configuration to the first stage boot loader in a binary encoded
> format that resides at a dedicated RAM region in physical address space.
> 
> This patch models this configuration space as a qdev device which we can
> then map at the fixed location in the address space. That way, we can
> influence and annotate all configuration fields easily.
> 
> Signed-off-by: Alexander Graf <graf@amazon.com>
> Signed-off-by: Phil Dennis-Jordan <phil@philjordan.eu>
> 
> ---
> v3:
> 
>   * Replaced legacy device reset method with Resettable method
> 
>   hw/vmapple/Kconfig       |   3 ++
>   hw/vmapple/cfg.c         | 106 +++++++++++++++++++++++++++++++++++++++
>   hw/vmapple/meson.build   |   1 +
>   include/hw/vmapple/cfg.h |  68 +++++++++++++++++++++++++
>   4 files changed, 178 insertions(+)
>   create mode 100644 hw/vmapple/cfg.c
>   create mode 100644 include/hw/vmapple/cfg.h
> 
> diff --git a/hw/vmapple/Kconfig b/hw/vmapple/Kconfig
> index 68f88876eb9..8bbeb9a9237 100644
> --- a/hw/vmapple/Kconfig
> +++ b/hw/vmapple/Kconfig
> @@ -4,3 +4,6 @@ config VMAPPLE_AES
>   config VMAPPLE_BDIF
>       bool
>   
> +config VMAPPLE_CFG
> +    bool
> +
> diff --git a/hw/vmapple/cfg.c b/hw/vmapple/cfg.c
> new file mode 100644
> index 00000000000..a5e5c62f59f
> --- /dev/null
> +++ b/hw/vmapple/cfg.c
> @@ -0,0 +1,106 @@
> +/*
> + * VMApple Configuration Region
> + *
> + * Copyright © 2023 Amazon.com, Inc. or its affiliates. All Rights Reserved.
> + *
> + * This work is licensed under the terms of the GNU GPL, version 2 or later.
> + * See the COPYING file in the top-level directory.
> + */
> +
> +#include "qemu/osdep.h"
> +#include "hw/vmapple/cfg.h"
> +#include "qemu/log.h"
> +#include "qemu/module.h"
> +#include "qapi/error.h"
> +
> +static void vmapple_cfg_reset(Object *obj, ResetType type)
> +{
> +    VMAppleCfgState *s = VMAPPLE_CFG(obj);
> +    VMAppleCfg *cfg;
> +
> +    cfg = memory_region_get_ram_ptr(&s->mem);
> +    memset((void *)cfg, 0, VMAPPLE_CFG_SIZE);
> +    *cfg = s->cfg;
 > +}> +
> +static void vmapple_cfg_realize(DeviceState *dev, Error **errp)
> +{
> +    VMAppleCfgState *s = VMAPPLE_CFG(dev);
> +    uint32_t i;
> +
> +    strncpy(s->cfg.serial, s->serial, sizeof(s->cfg.serial));
> +    strncpy(s->cfg.model, s->model, sizeof(s->cfg.model));
> +    strncpy(s->cfg.soc_name, s->soc_name, sizeof(s->cfg.soc_name));
> +    strncpy(s->cfg.unk8, "D/A", sizeof(s->cfg.soc_name));

Use qemu_strnlen() to report an error for too long strings.

> +    s->cfg.ecid = cpu_to_be64(s->cfg.ecid);
> +    s->cfg.version = 2;
> +    s->cfg.unk1 = 1;
> +    s->cfg.unk2 = 1;
> +    s->cfg.unk3 = 0x20;
> +    s->cfg.unk4 = 0;
> +    s->cfg.unk5 = 1;
> +    s->cfg.unk6 = 1;
> +    s->cfg.unk7 = 0;
> +    s->cfg.unk10 = 1;
> +
> +    g_assert(s->cfg.nr_cpus < ARRAY_SIZE(s->cfg.cpu_ids));

Report an error instead of asserting.

> +    for (i = 0; i < s->cfg.nr_cpus; i++) {
> +        s->cfg.cpu_ids[i] = i;
> +    }
 > +}> +
> +static void vmapple_cfg_init(Object *obj)
> +{
> +    VMAppleCfgState *s = VMAPPLE_CFG(obj);
> +
> +    memory_region_init_ram(&s->mem, obj, "VMApple Config", VMAPPLE_CFG_SIZE,
> +                           &error_fatal);
> +    sysbus_init_mmio(SYS_BUS_DEVICE(obj), &s->mem);
> +
> +    s->serial = (char *)"1234";
> +    s->model = (char *)"VM0001";
> +    s->soc_name = (char *)"Apple M1 (Virtual)";

These casts are unsafe; these pointers will be freed when this device is 
freed.

> +}
> +
> +static Property vmapple_cfg_properties[] = {
> +    DEFINE_PROP_UINT32("nr-cpus", VMAppleCfgState, cfg.nr_cpus, 1),
> +    DEFINE_PROP_UINT64("ecid", VMAppleCfgState, cfg.ecid, 0),
> +    DEFINE_PROP_UINT64("ram-size", VMAppleCfgState, cfg.ram_size, 0),
> +    DEFINE_PROP_UINT32("run_installer1", VMAppleCfgState, cfg.run_installer1, 0),
> +    DEFINE_PROP_UINT32("run_installer2", VMAppleCfgState, cfg.run_installer2, 0),
> +    DEFINE_PROP_UINT32("rnd", VMAppleCfgState, cfg.rnd, 0),
> +    DEFINE_PROP_MACADDR("mac-en0", VMAppleCfgState, cfg.mac_en0),
> +    DEFINE_PROP_MACADDR("mac-en1", VMAppleCfgState, cfg.mac_en1),
> +    DEFINE_PROP_MACADDR("mac-wifi0", VMAppleCfgState, cfg.mac_wifi0),
> +    DEFINE_PROP_MACADDR("mac-bt0", VMAppleCfgState, cfg.mac_bt0),
> +    DEFINE_PROP_STRING("serial", VMAppleCfgState, serial),
> +    DEFINE_PROP_STRING("model", VMAppleCfgState, model),
> +    DEFINE_PROP_STRING("soc_name", VMAppleCfgState, soc_name),
> +    DEFINE_PROP_END_OF_LIST(),
> +};
> +
> +static void vmapple_cfg_class_init(ObjectClass *klass, void *data)
> +{
> +    DeviceClass *dc = DEVICE_CLASS(klass);
> +    ResettableClass *rc = RESETTABLE_CLASS(klass);
> +
> +    dc->realize = vmapple_cfg_realize;
> +    dc->desc = "VMApple Configuration Region";
> +    device_class_set_props(dc, vmapple_cfg_properties);
> +    rc->phases.hold = vmapple_cfg_reset;
> +}
> +
> +static const TypeInfo vmapple_cfg_info = {
> +    .name          = TYPE_VMAPPLE_CFG,
> +    .parent        = TYPE_SYS_BUS_DEVICE,
> +    .instance_size = sizeof(VMAppleCfgState),
> +    .instance_init = vmapple_cfg_init,
> +    .class_init    = vmapple_cfg_class_init,
> +};
> +
> +static void vmapple_cfg_register_types(void)
> +{
> +    type_register_static(&vmapple_cfg_info);
> +}
> +
> +type_init(vmapple_cfg_register_types)
> diff --git a/hw/vmapple/meson.build b/hw/vmapple/meson.build
> index d4624713deb..64b78693a31 100644
> --- a/hw/vmapple/meson.build
> +++ b/hw/vmapple/meson.build
> @@ -1,2 +1,3 @@
>   system_ss.add(when: 'CONFIG_VMAPPLE_AES',  if_true: files('aes.c'))
>   system_ss.add(when: 'CONFIG_VMAPPLE_BDIF', if_true: files('bdif.c'))
> +system_ss.add(when: 'CONFIG_VMAPPLE_CFG',  if_true: files('cfg.c'))
> diff --git a/include/hw/vmapple/cfg.h b/include/hw/vmapple/cfg.h
> new file mode 100644
> index 00000000000..3337064e447
> --- /dev/null
> +++ b/include/hw/vmapple/cfg.h
> @@ -0,0 +1,68 @@
> +/*
> + * VMApple Configuration Region
> + *
> + * Copyright © 2023 Amazon.com, Inc. or its affiliates. All Rights Reserved.
> + *
> + * This work is licensed under the terms of the GNU GPL, version 2 or later.
> + * See the COPYING file in the top-level directory.
> + */
> +
> +#ifndef HW_VMAPPLE_CFG_H
> +#define HW_VMAPPLE_CFG_H
> +
> +#include "hw/sysbus.h"
> +#include "qom/object.h"
> +#include "net/net.h"
> +
> +typedef struct VMAppleCfg {
> +    uint32_t version;         /* 0x000 */
> +    uint32_t nr_cpus;         /* 0x004 */
> +    uint32_t unk1;            /* 0x008 */
> +    uint32_t unk2;            /* 0x00c */
> +    uint32_t unk3;            /* 0x010 */
> +    uint32_t unk4;            /* 0x014 */
> +    uint64_t ecid;            /* 0x018 */
> +    uint64_t ram_size;        /* 0x020 */
> +    uint32_t run_installer1;  /* 0x028 */
> +    uint32_t unk5;            /* 0x02c */
> +    uint32_t unk6;            /* 0x030 */
> +    uint32_t run_installer2;  /* 0x034 */
> +    uint32_t rnd;             /* 0x038 */
> +    uint32_t unk7;            /* 0x03c */
> +    MACAddr mac_en0;          /* 0x040 */
> +    uint8_t pad1[2];
> +    MACAddr mac_en1;          /* 0x048 */
> +    uint8_t pad2[2];
> +    MACAddr mac_wifi0;        /* 0x050 */
> +    uint8_t pad3[2];
> +    MACAddr mac_bt0;          /* 0x058 */
> +    uint8_t pad4[2];
> +    uint8_t reserved[0xa0];   /* 0x060 */
> +    uint32_t cpu_ids[0x80];   /* 0x100 */
> +    uint8_t scratch[0x200];   /* 0x180 */
> +    char serial[32];          /* 0x380 */
> +    char unk8[32];            /* 0x3a0 */
> +    char model[32];           /* 0x3c0 */
> +    uint8_t unk9[32];         /* 0x3e0 */
> +    uint32_t unk10;           /* 0x400 */
> +    char soc_name[32];        /* 0x404 */
> +} VMAppleCfg;
> +
> +#define TYPE_VMAPPLE_CFG "vmapple-cfg"
> +OBJECT_DECLARE_SIMPLE_TYPE(VMAppleCfgState, VMAPPLE_CFG)
> +
> +struct VMAppleCfgState {
> +    /* <private> */
> +    SysBusDevice parent_obj;
> +    VMAppleCfg cfg;
> +
> +    /* <public> */
> +    MemoryRegion mem;
> +    char *serial;
> +    char *model;
> +    char *soc_name;
> +};
> +
> +#define VMAPPLE_CFG_SIZE 0x00010000
> +
> +#endif /* HW_VMAPPLE_CFG_H */

Re: [PATCH v3 13/14] hw/vmapple/virtio-blk: Add support for apple virtio-blk

[Akihiko Odaki]

On 2024/09/28 17:57, Phil Dennis-Jordan wrote:
> From: Alexander Graf <graf@amazon.com>
> 
> Apple has its own virtio-blk PCI device ID where it deviates from the
> official virtio-pci spec slightly: It puts a new "apple type"
> field at a static offset in config space and introduces a new barrier
> command.
> 
> This patch first creates a mechanism for virtio-blk downstream classes to
> handle unknown commands. It then creates such a downstream class and a new
> vmapple-virtio-blk-pci class which support the additional apple type config
> identifier as well as the barrier command.
> 
> It then exposes 2 subclasses from that that we can use to expose root and
> aux virtio-blk devices: "vmapple-virtio-root" and "vmapple-virtio-aux".
> 
> Signed-off-by: Alexander Graf <graf@amazon.com>
> Signed-off-by: Phil Dennis-Jordan <phil@philjordan.eu>
> ---
>   hw/block/virtio-blk.c           |  19 ++-
>   hw/vmapple/Kconfig              |   3 +
>   hw/vmapple/meson.build          |   1 +
>   hw/vmapple/virtio-blk.c         | 212 ++++++++++++++++++++++++++++++++
>   include/hw/pci/pci_ids.h        |   1 +
>   include/hw/virtio/virtio-blk.h  |  12 +-
>   include/hw/vmapple/virtio-blk.h |  39 ++++++
>   7 files changed, 282 insertions(+), 5 deletions(-)
>   create mode 100644 hw/vmapple/virtio-blk.c
>   create mode 100644 include/hw/vmapple/virtio-blk.h
> 
> diff --git a/hw/block/virtio-blk.c b/hw/block/virtio-blk.c
> index 115795392c4..cecc4cef9e4 100644
> --- a/hw/block/virtio-blk.c
> +++ b/hw/block/virtio-blk.c
> @@ -50,12 +50,12 @@ static void virtio_blk_init_request(VirtIOBlock *s, VirtQueue *vq,
>       req->mr_next = NULL;
>   }
>   
> -static void virtio_blk_free_request(VirtIOBlockReq *req)
> +void virtio_blk_free_request(VirtIOBlockReq *req)
>   {
>       g_free(req);
>   }

This function is identical with g_free(). Perhaps it's better to remove 
it instead of updating it.

>   
> -static void virtio_blk_req_complete(VirtIOBlockReq *req, unsigned char status)
> +void virtio_blk_req_complete(VirtIOBlockReq *req, unsigned char status)
>   {
>       VirtIOBlock *s = req->dev;
>       VirtIODevice *vdev = VIRTIO_DEVICE(s);
> @@ -966,8 +966,18 @@ static int virtio_blk_handle_request(VirtIOBlockReq *req, MultiReqBuffer *mrb)
>           break;
>       }
>       default:
> -        virtio_blk_req_complete(req, VIRTIO_BLK_S_UNSUPP);
> -        virtio_blk_free_request(req);
> +    {
> +        /*
> +         * Give subclasses a chance to handle unknown requests. This way the
> +         * class lookup is not in the hot path.
> +         */
> +        VirtIOBlkClass *vbk = VIRTIO_BLK_GET_CLASS(s);
> +        if (!vbk->handle_unknown_request ||
> +            !vbk->handle_unknown_request(req, mrb, type)) {
> +            virtio_blk_req_complete(req, VIRTIO_BLK_S_UNSUPP);
> +            virtio_blk_free_request(req);
> +        }
> +    }
>       }
>       return 0;
>   }
> @@ -2044,6 +2054,7 @@ static const TypeInfo virtio_blk_info = {
>       .instance_size = sizeof(VirtIOBlock),
>       .instance_init = virtio_blk_instance_init,
>       .class_init = virtio_blk_class_init,
> +    .class_size = sizeof(VirtIOBlkClass),
>   };
>   
>   static void virtio_register_types(void)
> diff --git a/hw/vmapple/Kconfig b/hw/vmapple/Kconfig
> index 8bbeb9a9237..bcd1be63e3c 100644
> --- a/hw/vmapple/Kconfig
> +++ b/hw/vmapple/Kconfig
> @@ -7,3 +7,6 @@ config VMAPPLE_BDIF
>   config VMAPPLE_CFG
>       bool
>   
> +config VMAPPLE_VIRTIO_BLK
> +    bool
> +
> diff --git a/hw/vmapple/meson.build b/hw/vmapple/meson.build
> index 64b78693a31..bf17cf906c9 100644
> --- a/hw/vmapple/meson.build
> +++ b/hw/vmapple/meson.build
> @@ -1,3 +1,4 @@
>   system_ss.add(when: 'CONFIG_VMAPPLE_AES',  if_true: files('aes.c'))
>   system_ss.add(when: 'CONFIG_VMAPPLE_BDIF', if_true: files('bdif.c'))
>   system_ss.add(when: 'CONFIG_VMAPPLE_CFG',  if_true: files('cfg.c'))
> +system_ss.add(when: 'CONFIG_VMAPPLE_VIRTIO_BLK',  if_true: files('virtio-blk.c'))
> diff --git a/hw/vmapple/virtio-blk.c b/hw/vmapple/virtio-blk.c
> new file mode 100644
> index 00000000000..720eaa61a86
> --- /dev/null
> +++ b/hw/vmapple/virtio-blk.c
> @@ -0,0 +1,212 @@
> +/*
> + * VMApple specific VirtIO Block implementation
> + *
> + * Copyright © 2023 Amazon.com, Inc. or its affiliates. All Rights Reserved.
> + *
> + * This work is licensed under the terms of the GNU GPL, version 2 or later.
> + * See the COPYING file in the top-level directory.
> + *
> + * VMApple uses almost standard VirtIO Block, but with a few key differences:
> + *
> + *  - Different PCI device/vendor ID
> + *  - An additional "type" identifier to differentiate AUX and Root volumes
> + *  - An additional BARRIER command
> + */
> +
> +#include "qemu/osdep.h"
> +#include "hw/vmapple/virtio-blk.h"
> +#include "qemu/log.h"
> +#include "qemu/module.h"
> +#include "qapi/error.h"
> +
> +#define VIRTIO_BLK_T_APPLE_BARRIER     0x10000
> +
> +#define VIRTIO_APPLE_TYPE_ROOT 1
> +#define VIRTIO_APPLE_TYPE_AUX  2
> +
> +static bool vmapple_virtio_blk_handle_unknown_request(VirtIOBlockReq *req,
> +                                                      MultiReqBuffer *mrb,
> +                                                      uint32_t type)
> +{
> +    switch (type) {
> +    case VIRTIO_BLK_T_APPLE_BARRIER:
> +        /* We ignore barriers for now. YOLO. */

It should be LOG_UNIMP instead of a mere comment.

> +        virtio_blk_req_complete(req, VIRTIO_BLK_S_OK);
> +        virtio_blk_free_request(req);
> +        return true;
> +    default:
> +        return false;
> +    }
> +}
> +
> +/*
> + * VMApple virtio-blk uses the same config format as normal virtio, with one
> + * exception: It adds an "apple type" specififer at the same location that
> + * the spec reserves for max_secure_erase_sectors. Let's hook into the
> + * get_config code path here, run it as usual and then patch in the apple type.
> + */
> +static void vmapple_virtio_blk_get_config(VirtIODevice *vdev, uint8_t *config)
> +{
> +    VMAppleVirtIOBlk *dev = VMAPPLE_VIRTIO_BLK(vdev);
> +    VMAppleVirtIOBlkClass *vvbk = VMAPPLE_VIRTIO_BLK_GET_CLASS(dev);
> +    struct virtio_blk_config *blkcfg = (struct virtio_blk_config *)config;
> +
> +    vvbk->get_config(vdev, config);
> +
> +    g_assert(dev->parent_obj.config_size >= endof(struct virtio_blk_config, zoned));
> +
> +    /* Apple abuses the field for max_secure_erase_sectors as type id */
> +    blkcfg->max_secure_erase_sectors = dev->apple_type;
> +}
> +
> +static Property vmapple_virtio_blk_properties[] = {
> +    DEFINE_PROP_UINT32("apple-type", VMAppleVirtIOBlk, apple_type, 0),
> +    DEFINE_PROP_END_OF_LIST(),
> +};
> +
> +static void vmapple_virtio_blk_class_init(ObjectClass *klass, void *data)
> +{
> +    DeviceClass *dc = DEVICE_CLASS(klass);
> +    VirtIOBlkClass *vbk = VIRTIO_BLK_CLASS(klass);
> +    VirtioDeviceClass *vdc = VIRTIO_DEVICE_CLASS(klass);
> +    VMAppleVirtIOBlkClass *vvbk = VMAPPLE_VIRTIO_BLK_CLASS(klass);
> +
> +    vbk->handle_unknown_request = vmapple_virtio_blk_handle_unknown_request;
> +    vvbk->get_config = vdc->get_config;
> +    vdc->get_config = vmapple_virtio_blk_get_config;
> +    device_class_set_props(dc, vmapple_virtio_blk_properties);
> +}
> +
> +static const TypeInfo vmapple_virtio_blk_info = {
> +    .name          = TYPE_VMAPPLE_VIRTIO_BLK,
> +    .parent        = TYPE_VIRTIO_BLK,
> +    .instance_size = sizeof(VMAppleVirtIOBlk),
> +    .class_init    = vmapple_virtio_blk_class_init,
> +};
> +
> +/* PCI Devices */
> +
> +typedef struct VMAppleVirtIOBlkPCI {
> +    VirtIOPCIProxy parent_obj;
> +    VMAppleVirtIOBlk vdev;
> +    uint32_t apple_type;
> +} VMAppleVirtIOBlkPCI;
> +
> +/*
> + * vmapple-virtio-blk-pci: This extends VirtioPCIProxy.
> + */
> +#define TYPE_VMAPPLE_VIRTIO_BLK_PCI "vmapple-virtio-blk-pci-base"
> +DECLARE_INSTANCE_CHECKER(VMAppleVirtIOBlkPCI, VMAPPLE_VIRTIO_BLK_PCI,
> +                         TYPE_VMAPPLE_VIRTIO_BLK_PCI)
> +
> +static Property vmapple_virtio_blk_pci_properties[] = {
> +    DEFINE_PROP_UINT32("class", VirtIOPCIProxy, class_code, 0),
> +    DEFINE_PROP_BIT("ioeventfd", VirtIOPCIProxy, flags,
> +                    VIRTIO_PCI_FLAG_USE_IOEVENTFD_BIT, true),
> +    DEFINE_PROP_UINT32("vectors", VirtIOPCIProxy, nvectors,
> +                       DEV_NVECTORS_UNSPECIFIED),
> +    DEFINE_PROP_END_OF_LIST(),
> +};
> +
> +static void vmapple_virtio_blk_pci_realize(VirtIOPCIProxy *vpci_dev, Error **errp)
> +{
> +    VMAppleVirtIOBlkPCI *dev = VMAPPLE_VIRTIO_BLK_PCI(vpci_dev);
> +    DeviceState *vdev = DEVICE(&dev->vdev);
> +    VirtIOBlkConf *conf = &dev->vdev.parent_obj.conf;
> +
> +    if (conf->num_queues == VIRTIO_BLK_AUTO_NUM_QUEUES) {
> +        conf->num_queues = virtio_pci_optimal_num_queues(0);
> +    }
> +
> +    if (vpci_dev->nvectors == DEV_NVECTORS_UNSPECIFIED) {
> +        vpci_dev->nvectors = conf->num_queues + 1;
> +    }
> +
> +    /*
> +     * We don't support zones, but we need the additional config space size.
> +     * Let's just expose the feature so the rest of the virtio-blk logic
> +     * allocates enough space for us. The guest will ignore zones anyway.
> +     */
> +    virtio_add_feature(&dev->vdev.parent_obj.host_features, VIRTIO_BLK_F_ZONED);
> +    /* Propagate the apple type down to the virtio-blk device */
> +    qdev_prop_set_uint32(DEVICE(&dev->vdev), "apple-type", dev->apple_type);
> +    /* and spawn the virtio-blk device */
> +    qdev_realize(vdev, BUS(&vpci_dev->bus), errp);
> +
> +    /*
> +     * The virtio-pci machinery adjusts its vendor/device ID based on whether
> +     * we support modern or legacy virtio. Let's patch it back to the Apple
> +     * identifiers here.
> +     */
> +    pci_config_set_vendor_id(vpci_dev->pci_dev.config, PCI_VENDOR_ID_APPLE);
> +    pci_config_set_device_id(vpci_dev->pci_dev.config, PCI_DEVICE_ID_APPLE_VIRTIO_BLK);
> +}
> +
> +static void vmapple_virtio_blk_pci_class_init(ObjectClass *klass, void *data)
> +{
> +    DeviceClass *dc = DEVICE_CLASS(klass);
> +    VirtioPCIClass *k = VIRTIO_PCI_CLASS(klass);
> +    PCIDeviceClass *pcidev_k = PCI_DEVICE_CLASS(klass);
> +
> +    set_bit(DEVICE_CATEGORY_STORAGE, dc->categories);
> +    device_class_set_props(dc, vmapple_virtio_blk_pci_properties);
> +    k->realize = vmapple_virtio_blk_pci_realize;
> +    pcidev_k->vendor_id = PCI_VENDOR_ID_APPLE;
> +    pcidev_k->device_id = PCI_DEVICE_ID_APPLE_VIRTIO_BLK;
> +    pcidev_k->revision = VIRTIO_PCI_ABI_VERSION;
> +    pcidev_k->class_id = PCI_CLASS_STORAGE_SCSI;
> +}
> +
> +static void vmapple_virtio_blk_pci_instance_init(Object *obj)
> +{
> +    VMAppleVirtIOBlkPCI *dev = VMAPPLE_VIRTIO_BLK_PCI(obj);
> +
> +    virtio_instance_init_common(obj, &dev->vdev, sizeof(dev->vdev),
> +                                TYPE_VMAPPLE_VIRTIO_BLK);
> +}
> +
> +static const VirtioPCIDeviceTypeInfo vmapple_virtio_blk_pci_info = {
> +    .base_name     = TYPE_VMAPPLE_VIRTIO_BLK_PCI,
> +    .generic_name  = "vmapple-virtio-blk-pci",
> +    .instance_size = sizeof(VMAppleVirtIOBlkPCI),
> +    .instance_init = vmapple_virtio_blk_pci_instance_init,
> +    .class_init    = vmapple_virtio_blk_pci_class_init,
> +};
> +
> +static void vmapple_virtio_root_instance_init(Object *obj)
> +{
> +    VMAppleVirtIOBlkPCI *dev = VMAPPLE_VIRTIO_BLK_PCI(obj);
> +
> +    dev->apple_type = VIRTIO_APPLE_TYPE_ROOT;
> +}
> +
> +static const TypeInfo vmapple_virtio_root_info = {
> +    .name          = TYPE_VMAPPLE_VIRTIO_ROOT,
> +    .parent        = "vmapple-virtio-blk-pci",
> +    .instance_size = sizeof(VMAppleVirtIOBlkPCI),
> +    .instance_init = vmapple_virtio_root_instance_init,
> +};
> +
> +static void vmapple_virtio_aux_instance_init(Object *obj)
> +{
> +    VMAppleVirtIOBlkPCI *dev = VMAPPLE_VIRTIO_BLK_PCI(obj);
> +
> +    dev->apple_type = VIRTIO_APPLE_TYPE_AUX;
> +}
> +
> +static const TypeInfo vmapple_virtio_aux_info = {
> +    .name          = TYPE_VMAPPLE_VIRTIO_AUX,
> +    .parent        = "vmapple-virtio-blk-pci",
> +    .instance_size = sizeof(VMAppleVirtIOBlkPCI),
> +    .instance_init = vmapple_virtio_aux_instance_init,
> +};
> +
> +static void vmapple_virtio_blk_register_types(void)
> +{
> +    type_register_static(&vmapple_virtio_blk_info);
> +    virtio_pci_types_register(&vmapple_virtio_blk_pci_info);
> +    type_register_static(&vmapple_virtio_root_info);
> +    type_register_static(&vmapple_virtio_aux_info);
> +}
> +
> +type_init(vmapple_virtio_blk_register_types)
> diff --git a/include/hw/pci/pci_ids.h b/include/hw/pci/pci_ids.h
> index f1a53fea8d6..33e2898be95 100644
> --- a/include/hw/pci/pci_ids.h
> +++ b/include/hw/pci/pci_ids.h
> @@ -191,6 +191,7 @@
>   #define PCI_DEVICE_ID_APPLE_UNI_N_AGP    0x0020
>   #define PCI_DEVICE_ID_APPLE_U3_AGP       0x004b
>   #define PCI_DEVICE_ID_APPLE_UNI_N_GMAC   0x0021
> +#define PCI_DEVICE_ID_APPLE_VIRTIO_BLK   0x1a00
>   
>   #define PCI_VENDOR_ID_SUN                0x108e
>   #define PCI_DEVICE_ID_SUN_EBUS           0x1000
> diff --git a/include/hw/virtio/virtio-blk.h b/include/hw/virtio/virtio-blk.h
> index 5c14110c4b1..28d5046ea6c 100644
> --- a/include/hw/virtio/virtio-blk.h
> +++ b/include/hw/virtio/virtio-blk.h
> @@ -24,7 +24,7 @@
>   #include "qapi/qapi-types-virtio.h"
>   
>   #define TYPE_VIRTIO_BLK "virtio-blk-device"
> -OBJECT_DECLARE_SIMPLE_TYPE(VirtIOBlock, VIRTIO_BLK)
> +OBJECT_DECLARE_TYPE(VirtIOBlock, VirtIOBlkClass, VIRTIO_BLK)
>   
>   /* This is the last element of the write scatter-gather list */
>   struct virtio_blk_inhdr
> @@ -100,6 +100,16 @@ typedef struct MultiReqBuffer {
>       bool is_write;
>   } MultiReqBuffer;
>   
> +typedef struct VirtIOBlkClass {
> +    /*< private >*/
> +    VirtioDeviceClass parent;
> +    /*< public >*/
> +    bool (*handle_unknown_request)(VirtIOBlockReq *req, MultiReqBuffer *mrb,
> +                                   uint32_t type);
> +} VirtIOBlkClass;
> +
>   void virtio_blk_handle_vq(VirtIOBlock *s, VirtQueue *vq);
> +void virtio_blk_free_request(VirtIOBlockReq *req);
> +void virtio_blk_req_complete(VirtIOBlockReq *req, unsigned char status);
>   
>   #endif
> diff --git a/include/hw/vmapple/virtio-blk.h b/include/hw/vmapple/virtio-blk.h
> new file mode 100644
> index 00000000000..b23106a3dfb
> --- /dev/null
> +++ b/include/hw/vmapple/virtio-blk.h
> @@ -0,0 +1,39 @@
> +/*
> + * VMApple specific VirtIO Block implementation
> + *
> + * Copyright © 2023 Amazon.com, Inc. or its affiliates. All Rights Reserved.
> + *
> + * This work is licensed under the terms of the GNU GPL, version 2 or later.
> + * See the COPYING file in the top-level directory.
> + */
> +
> +#ifndef HW_VMAPPLE_CFG_H
> +#define HW_VMAPPLE_CFG_H
> +
> +#include "hw/sysbus.h"
> +#include "qom/object.h"
> +#include "hw/virtio/virtio-pci.h"
> +#include "hw/virtio/virtio-blk.h"
> +
> +#define TYPE_VMAPPLE_VIRTIO_BLK "vmapple-virtio-blk"
> +#define TYPE_VMAPPLE_VIRTIO_ROOT "vmapple-virtio-root"
> +#define TYPE_VMAPPLE_VIRTIO_AUX "vmapple-virtio-aux"
> +
> +OBJECT_DECLARE_TYPE(VMAppleVirtIOBlk, VMAppleVirtIOBlkClass, VMAPPLE_VIRTIO_BLK)
> +
> +typedef struct VMAppleVirtIOBlkClass {
> +    /*< private >*/
> +    VirtIOBlkClass parent;
> +    /*< public >*/
> +    void (*get_config)(VirtIODevice *vdev, uint8_t *config);
> +} VMAppleVirtIOBlkClass;
> +
> +typedef struct VMAppleVirtIOBlk {
> +    /* <private> */
> +    VirtIOBlock parent_obj;
> +
> +    /* <public> */
> +    uint32_t apple_type;
> +} VMAppleVirtIOBlk;
> +
> +#endif /* HW_VMAPPLE_CFG_H */

Re: [PATCH v3 14/14] hw/vmapple/vmapple: Add vmapple machine type

[Akihiko Odaki]

On 2024/09/28 17:57, Phil Dennis-Jordan wrote:
> From: Alexander Graf <graf@amazon.com>
> 
> Apple defines a new "vmapple" machine type as part of its proprietary
> macOS Virtualization.Framework vmm. This machine type is similar to the
> virt one, but with subtle differences in base devices, a few special
> vmapple device additions and a vastly different boot chain.
> 
> This patch reimplements this machine type in QEMU. To use it, you
> have to have a readily installed version of macOS for VMApple,
> run on macOS with -accel hvf, pass the Virtualization.Framework
> boot rom (AVPBooter) in via -bios, pass the aux and root volume as pflash
> and pass aux and root volume as virtio drives. In addition, you also
> need to find the machine UUID and pass that as -M vmapple,uuid= parameter:
> 
> $ qemu-system-aarch64 -accel hvf -M vmapple,uuid=0x1234 -m 4G \
>      -bios /System/Library/Frameworks/Virtualization.framework/Versions/A/Resources/AVPBooter.vmapple2.bin
>      -drive file=aux,if=pflash,format=raw \
>      -drive file=root,if=pflash,format=raw \
>      -drive file=aux,if=none,id=aux,format=raw \
>      -device vmapple-virtio-aux,drive=aux \
>      -drive file=root,if=none,id=root,format=raw \
>      -device vmapple-virtio-root,drive=root
> 
> With all these in place, you should be able to see macOS booting
> successfully.
> 
> Known issues:
>   - Keyboard and mouse/tablet input is laggy. The reason for this is
>     either that macOS's XHCI driver is broken when the device/platform
>     does not support MSI/MSI-X, or there's some unfortunate interplay
>     with Qemu's XHCI implementation in this scenario.
>   - Currently only macOS 12 guests are supported. The boot process for
>     13+ will need further investigation and adjustment.
> 
> Signed-off-by: Alexander Graf <graf@amazon.com>
> Co-authored-by: Phil Dennis-Jordan <phil@philjordan.eu>
> Signed-off-by: Phil Dennis-Jordan <phil@philjordan.eu>
> 
> ---
> v3:
>   * Rebased on latest upstream, updated affinity and NIC creation
> API usage
>   * Included Apple-variant virtio-blk in build dependency
>   * Updated API usage for setting 'redist-region-count' array-typed property on GIC.
>   * Switched from virtio HID devices (for which macOS 12 does not contain drivers) to an XHCI USB controller and USB HID devices.
> 
>   MAINTAINERS                 |   1 +
>   docs/system/arm/vmapple.rst |  63 ++++
>   docs/system/target-arm.rst  |   1 +
>   hw/vmapple/Kconfig          |  20 ++
>   hw/vmapple/meson.build      |   1 +
>   hw/vmapple/vmapple.c        | 661 ++++++++++++++++++++++++++++++++++++
>   6 files changed, 747 insertions(+)
>   create mode 100644 docs/system/arm/vmapple.rst
>   create mode 100644 hw/vmapple/vmapple.c
> 
> diff --git a/MAINTAINERS b/MAINTAINERS
> index 4e7f25e5299..89ef071a01a 100644
> --- a/MAINTAINERS
> +++ b/MAINTAINERS
> @@ -2783,6 +2783,7 @@ R: Phil Dennis-Jordan <phil@philjordan.eu>
>   S: Maintained
>   F: hw/vmapple/*
>   F: include/hw/vmapple/*
> +F: docs/system/arm/vmapple.rst
>   
>   Subsystems
>   ----------
> diff --git a/docs/system/arm/vmapple.rst b/docs/system/arm/vmapple.rst
> new file mode 100644
> index 00000000000..acb921ffb35
> --- /dev/null
> +++ b/docs/system/arm/vmapple.rst
> @@ -0,0 +1,63 @@
> +VMApple machine emulation
> +========================================================================================
> +
> +VMApple is the device model that the macOS built-in hypervisor called "Virtualization.framework"
> +exposes to Apple Silicon macOS guests. The "vmapple" machine model in QEMU implements the same
> +device model, but does not use any code from Virtualization.Framework.
> +
> +Prerequisites
> +-------------
> +
> +To run the vmapple machine model, you need to
> +
> + * Run on Apple Silicon
> + * Run on macOS 12.0 or above
> + * Have an already installed copy of a Virtualization.Framework macOS 12 virtual machine. I will
> +   assume that you installed it using the macosvm CLI.
> +
> +First, we need to extract the UUID from the virtual machine that you installed. You can do this
> +by running the following shell script:
> +
> +.. code-block:: bash
> +  :caption: uuid.sh script to extract the UUID from a macosvm.json file
> +
> +  #!/bin/bash
> +
> +  MID=$(cat "$1" | python3 -c 'import json,sys;obj=json.load(sys.stdin);print(obj["machineId"]);')
> +  echo "$MID" | base64 -d | plutil -extract ECID raw -
> +
> +Now we also need to trim the aux partition. It contains metadata that we can just discard:
> +
> +.. code-block:: bash
> +  :caption: Command to trim the aux file
> +
> +  $ dd if="aux.img" of="aux.img.trimmed" bs=$(( 0x4000 )) skip=1
> +
> +How to run
> +----------
> +
> +Then, we can launch QEMU with the Virtualization.Framework pre-boot environment and the readily
> +installed target disk images. I recommend to port forward the VM's ssh and vnc ports to the host
> +to get better interactive access into the target system:
> +
> +.. code-block:: bash
> +  :caption: Example execution command line
> +
> +  $ UUID=$(uuid.sh macosvm.json)
> +  $ AVPBOOTER=/System/Library/Frameworks/Virtualization.framework/Resources/AVPBooter.vmapple2.bin
> +  $ AUX=aux.img.trimmed
> +  $ DISK=disk.img
> +  $ qemu-system-aarch64 \
> +       -serial mon:stdio \
> +       -m 4G \
> +       -accel hvf \
> +       -M vmapple,uuid=$UUID \
> +       -bios $AVPBOOTER \
> +        -drive file="$AUX",if=pflash,format=raw \
> +        -drive file="$DISK",if=pflash,format=raw \
> +       -drive file="$AUX",if=none,id=aux,format=raw \
> +       -drive file="$DISK",if=none,id=root,format=raw \
> +       -device vmapple-virtio-aux,drive=aux \
> +       -device vmapple-virtio-root,drive=root \
> +       -net user,ipv6=off,hostfwd=tcp::2222-:22,hostfwd=tcp::5901-:5900 \
> +       -net nic,model=virtio-net-pci \
> diff --git a/docs/system/target-arm.rst b/docs/system/target-arm.rst
> index 7b992722846..f1948abb545 100644
> --- a/docs/system/target-arm.rst
> +++ b/docs/system/target-arm.rst
> @@ -107,6 +107,7 @@ undocumented; you can get a complete list by running
>      arm/stellaris
>      arm/stm32
>      arm/virt
> +   arm/vmapple
>      arm/xenpvh
>      arm/xlnx-versal-virt
>      arm/xlnx-zynq
> diff --git a/hw/vmapple/Kconfig b/hw/vmapple/Kconfig
> index bcd1be63e3c..0f83d4259fc 100644
> --- a/hw/vmapple/Kconfig
> +++ b/hw/vmapple/Kconfig
> @@ -10,3 +10,23 @@ config VMAPPLE_CFG
>   config VMAPPLE_VIRTIO_BLK
>       bool
>   
> +config VMAPPLE
> +    bool
> +    depends on ARM
> +    depends on HVF
> +    default y if ARM
> +    imply PCI_DEVICES
> +    select ARM_GIC
> +    select PLATFORM_BUS
> +    select PCI_EXPRESS
> +    select PCI_EXPRESS_GENERIC_BRIDGE
> +    select PL011 # UART
> +    select PL031 # RTC
> +    select PL061 # GPIO
> +    select GPIO_PWR
> +    select PVPANIC_MMIO
> +    select VMAPPLE_AES
> +    select VMAPPLE_BDIF
> +    select VMAPPLE_CFG
> +    select MAC_PVG_VMAPPLE
> +    select VMAPPLE_VIRTIO_BLK
> diff --git a/hw/vmapple/meson.build b/hw/vmapple/meson.build
> index bf17cf906c9..e572f7d5602 100644
> --- a/hw/vmapple/meson.build
> +++ b/hw/vmapple/meson.build
> @@ -2,3 +2,4 @@ system_ss.add(when: 'CONFIG_VMAPPLE_AES',  if_true: files('aes.c'))
>   system_ss.add(when: 'CONFIG_VMAPPLE_BDIF', if_true: files('bdif.c'))
>   system_ss.add(when: 'CONFIG_VMAPPLE_CFG',  if_true: files('cfg.c'))
>   system_ss.add(when: 'CONFIG_VMAPPLE_VIRTIO_BLK',  if_true: files('virtio-blk.c'))
> +specific_ss.add(when: 'CONFIG_VMAPPLE',     if_true: files('vmapple.c'))
> diff --git a/hw/vmapple/vmapple.c b/hw/vmapple/vmapple.c
> new file mode 100644
> index 00000000000..f0060a6f7ee
> --- /dev/null
> +++ b/hw/vmapple/vmapple.c
> @@ -0,0 +1,661 @@
> +/*
> + * VMApple machine emulation
> + *
> + * Copyright © 2023 Amazon.com, Inc. or its affiliates. All Rights Reserved.
> + *
> + * This work is licensed under the terms of the GNU GPL, version 2 or later.
> + * See the COPYING file in the top-level directory.
> + *
> + * VMApple is the device model that the macOS built-in hypervisor called
> + * "Virtualization.framework" exposes to Apple Silicon macOS guests. The
> + * machine model in this file implements the same device model in QEMU, but
> + * does not use any code from Virtualization.Framework.
> + */
> +
> +#include "qemu/osdep.h"
> +#include "qemu/help-texts.h"
> +#include "qemu/datadir.h"
> +#include "qemu/units.h"
> +#include "qemu/option.h"
> +#include "monitor/qdev.h"
> +#include "hw/sysbus.h"
> +#include "hw/arm/boot.h"
> +#include "hw/arm/primecell.h"
> +#include "hw/boards.h"
> +#include "hw/usb.h"
> +#include "net/net.h"
> +#include "sysemu/sysemu.h"
> +#include "sysemu/runstate.h"
> +#include "sysemu/kvm.h"
> +#include "sysemu/hvf.h"
> +#include "hw/loader.h"
> +#include "qapi/error.h"
> +#include "qapi/qmp/qlist.h"
> +#include "qemu/bitops.h"
> +#include "qemu/error-report.h"
> +#include "qemu/module.h"
> +#include "hw/pci-host/gpex.h"
> +#include "hw/virtio/virtio-pci.h"
> +#include "hw/qdev-properties.h"
> +#include "hw/intc/arm_gic.h"
> +#include "hw/intc/arm_gicv3_common.h"
> +#include "hw/irq.h"
> +#include "hw/usb/xhci.h"
> +#include "qapi/visitor.h"
> +#include "qapi/qapi-visit-common.h"
> +#include "standard-headers/linux/input.h"
> +#include "target/arm/internals.h"
> +#include "target/arm/kvm_arm.h"
> +#include "hw/char/pl011.h"
> +#include "qemu/guest-random.h"
> +#include "sysemu/reset.h"
> +#include "qemu/log.h"
> +#include "hw/vmapple/cfg.h"
> +#include "hw/misc/pvpanic.h"
> +#include "hw/vmapple/bdif.h"
> +
> +struct VMAppleMachineClass {
> +    MachineClass parent;
> +};
> +
> +struct VMAppleMachineState {
> +    MachineState parent;
> +
> +    Notifier machine_done;
> +    struct arm_boot_info bootinfo;
> +    MemMapEntry *memmap;
> +    const int *irqmap;
> +    DeviceState *gic;
> +    DeviceState *cfg;
> +    Notifier powerdown_notifier;
> +    PCIBus *bus;
> +    MemoryRegion fw_mr;
> +    uint64_t uuid;
> +};
> +
> +#define DEFINE_VMAPPLE_MACHINE_LATEST(major, minor, latest) \
> +    static void vmapple##major##_##minor##_class_init(ObjectClass *oc, \
> +                                                    void *data) \
> +    { \
> +        MachineClass *mc = MACHINE_CLASS(oc); \
> +        vmapple_machine_##major##_##minor##_options(mc); \
> +        mc->desc = "QEMU " # major "." # minor " Apple Virtual Machine"; \
> +        if (latest) { \
> +            mc->alias = "vmapple"; \
> +        } \
> +    } \
> +    static const TypeInfo machvmapple##major##_##minor##_info = { \
> +        .name = MACHINE_TYPE_NAME("vmapple-" # major "." # minor), \
> +        .parent = TYPE_VMAPPLE_MACHINE, \
> +        .class_init = vmapple##major##_##minor##_class_init, \
> +    }; \
> +    static void machvmapple_machine_##major##_##minor##_init(void) \
> +    { \
> +        type_register_static(&machvmapple##major##_##minor##_info); \
> +    } \
> +    type_init(machvmapple_machine_##major##_##minor##_init);
> +
> +#define DEFINE_VMAPPLE_MACHINE_AS_LATEST(major, minor) \
> +    DEFINE_VMAPPLE_MACHINE_LATEST(major, minor, true)
> +#define DEFINE_VMAPPLE_MACHINE(major, minor) \
> +    DEFINE_VMAPPLE_MACHINE_LATEST(major, minor, false)
> +
> +#define TYPE_VMAPPLE_MACHINE   MACHINE_TYPE_NAME("vmapple")
> +OBJECT_DECLARE_TYPE(VMAppleMachineState, VMAppleMachineClass, VMAPPLE_MACHINE)
> +
> +/* Number of external interrupt lines to configure the GIC with */
> +#define NUM_IRQS 256
> +
> +enum {
> +    VMAPPLE_FIRMWARE,
> +    VMAPPLE_CONFIG,
> +    VMAPPLE_MEM,
> +    VMAPPLE_GIC_DIST,
> +    VMAPPLE_GIC_REDIST,
> +    VMAPPLE_UART,
> +    VMAPPLE_RTC,
> +    VMAPPLE_PCIE,
> +    VMAPPLE_PCIE_MMIO,
> +    VMAPPLE_PCIE_ECAM,
> +    VMAPPLE_GPIO,
> +    VMAPPLE_PVPANIC,
> +    VMAPPLE_APV_GFX,
> +    VMAPPLE_APV_IOSFC,
> +    VMAPPLE_AES_1,
> +    VMAPPLE_AES_2,
> +    VMAPPLE_BDOOR,
> +    VMAPPLE_MEMMAP_LAST,
> +};
> +
> +static MemMapEntry memmap[] = {
> +    [VMAPPLE_FIRMWARE] =           { 0x00100000, 0x00100000 },
> +    [VMAPPLE_CONFIG] =             { 0x00400000, 0x00010000 },
> +
> +    [VMAPPLE_GIC_DIST] =           { 0x10000000, 0x00010000 },
> +    [VMAPPLE_GIC_REDIST] =         { 0x10010000, 0x00400000 },
> +
> +    [VMAPPLE_UART] =               { 0x20010000, 0x00010000 },
> +    [VMAPPLE_RTC] =                { 0x20050000, 0x00001000 },
> +    [VMAPPLE_GPIO] =               { 0x20060000, 0x00001000 },
> +    [VMAPPLE_PVPANIC] =            { 0x20070000, 0x00000002 },
> +    [VMAPPLE_BDOOR] =              { 0x30000000, 0x00200000 },
> +    [VMAPPLE_APV_GFX] =            { 0x30200000, 0x00010000 },
> +    [VMAPPLE_APV_IOSFC] =          { 0x30210000, 0x00010000 },
> +    [VMAPPLE_AES_1] =              { 0x30220000, 0x00004000 },
> +    [VMAPPLE_AES_2] =              { 0x30230000, 0x00004000 },
> +    [VMAPPLE_PCIE_ECAM] =          { 0x40000000, 0x10000000 },
> +    [VMAPPLE_PCIE_MMIO] =          { 0x50000000, 0x1fff0000 },
> +
> +    /* Actual RAM size depends on configuration */
> +    [VMAPPLE_MEM] =                { 0x70000000ULL, GiB},
> +};
> +
> +static const int irqmap[] = {
> +    [VMAPPLE_UART] = 1,
> +    [VMAPPLE_RTC] = 2,
> +    [VMAPPLE_GPIO] = 0x5,
> +    [VMAPPLE_APV_IOSFC] = 0x10,
> +    [VMAPPLE_APV_GFX] = 0x11,
> +    [VMAPPLE_AES_1] = 0x12,
> +    [VMAPPLE_PCIE] = 0x20,
> +};
> +
> +#define GPEX_NUM_IRQS 16
> +
> +static void create_bdif(VMAppleMachineState *vms, MemoryRegion *mem)
> +{
> +    DeviceState *bdif;
> +    SysBusDevice *bdif_sb;
> +    DriveInfo *di_aux = drive_get(IF_PFLASH, 0, 0);
> +    DriveInfo *di_root = drive_get(IF_PFLASH, 0, 1);
> +
> +    if (!di_aux) {
> +        error_report("No AUX device found. Please specify one as pflash drive");
> +        exit(1);
> +    }
> +
> +    if (!di_root) {
> +        /* Fall back to the first IF_VIRTIO device as root device */
> +        di_root = drive_get(IF_VIRTIO, 0, 0);
> +    }
> +
> +    if (!di_root) {
> +        error_report("No root device found. Please specify one as virtio drive");
> +        exit(1);
> +    }
> +
> +    /* PV backdoor device */
> +    bdif = qdev_new(TYPE_VMAPPLE_BDIF);
> +    bdif_sb = SYS_BUS_DEVICE(bdif);
> +    sysbus_mmio_map(bdif_sb, 0, vms->memmap[VMAPPLE_BDOOR].base);
> +
> +    qdev_prop_set_drive(DEVICE(bdif), "aux", blk_by_legacy_dinfo(di_aux));
> +    qdev_prop_set_drive(DEVICE(bdif), "root", blk_by_legacy_dinfo(di_root));
> +
> +    sysbus_realize_and_unref(bdif_sb, &error_fatal);
> +}
> +
> +static void create_pvpanic(VMAppleMachineState *vms, MemoryRegion *mem)
> +{
> +    SysBusDevice *cfg;
> +
> +    vms->cfg = qdev_new(TYPE_PVPANIC_MMIO_DEVICE);
> +    cfg = SYS_BUS_DEVICE(vms->cfg);
> +    sysbus_mmio_map(cfg, 0, vms->memmap[VMAPPLE_PVPANIC].base);
> +
> +    sysbus_realize_and_unref(cfg, &error_fatal);
> +}
> +
> +static void create_cfg(VMAppleMachineState *vms, MemoryRegion *mem)
> +{
> +    SysBusDevice *cfg;
> +    MachineState *machine = MACHINE(vms);
> +    uint32_t rnd = 1;
> +
> +    vms->cfg = qdev_new(TYPE_VMAPPLE_CFG);
> +    cfg = SYS_BUS_DEVICE(vms->cfg);
> +    sysbus_mmio_map(cfg, 0, vms->memmap[VMAPPLE_CONFIG].base);
> +
> +    qemu_guest_getrandom_nofail(&rnd, sizeof(rnd));
> +
> +    qdev_prop_set_uint32(vms->cfg, "nr-cpus", machine->smp.cpus);
> +    qdev_prop_set_uint64(vms->cfg, "ecid", vms->uuid);
> +    qdev_prop_set_uint64(vms->cfg, "ram-size", machine->ram_size);
> +    qdev_prop_set_uint32(vms->cfg, "rnd", rnd);
> +
> +    sysbus_realize_and_unref(cfg, &error_fatal);
> +}
> +
> +static void create_gfx(VMAppleMachineState *vms, MemoryRegion *mem)
> +{
> +    int irq_gfx = vms->irqmap[VMAPPLE_APV_GFX];
> +    int irq_iosfc = vms->irqmap[VMAPPLE_APV_IOSFC];
> +    SysBusDevice *aes;
> +
> +    aes = SYS_BUS_DEVICE(qdev_new("apple-gfx-vmapple"));
> +    sysbus_mmio_map(aes, 0, vms->memmap[VMAPPLE_APV_GFX].base);
> +    sysbus_mmio_map(aes, 1, vms->memmap[VMAPPLE_APV_IOSFC].base);
> +    sysbus_connect_irq(aes, 0, qdev_get_gpio_in(vms->gic, irq_gfx));
> +    sysbus_connect_irq(aes, 1, qdev_get_gpio_in(vms->gic, irq_iosfc));
> +    sysbus_realize_and_unref(aes, &error_fatal);
> +}
> +
> +static void create_aes(VMAppleMachineState *vms, MemoryRegion *mem)
> +{
> +    int irq = vms->irqmap[VMAPPLE_AES_1];
> +    SysBusDevice *aes;
> +
> +    aes = SYS_BUS_DEVICE(qdev_new("apple-aes"));
> +    sysbus_mmio_map(aes, 0, vms->memmap[VMAPPLE_AES_1].base);
> +    sysbus_mmio_map(aes, 1, vms->memmap[VMAPPLE_AES_2].base);
> +    sysbus_connect_irq(aes, 0, qdev_get_gpio_in(vms->gic, irq));
> +    sysbus_realize_and_unref(aes, &error_fatal);
> +}
> +
> +static inline int arm_gic_ppi_index(int cpu_nr, int ppi_index)
> +{
> +    return NUM_IRQS + cpu_nr * GIC_INTERNAL + ppi_index;
> +}
> +
> +static void create_gic(VMAppleMachineState *vms, MemoryRegion *mem)
> +{
> +    MachineState *ms = MACHINE(vms);
> +    /* We create a standalone GIC */
> +    SysBusDevice *gicbusdev;
> +    QList *redist_region_count;
> +    int i;
> +    unsigned int smp_cpus = ms->smp.cpus;
> +
> +    vms->gic = qdev_new(gicv3_class_name());
> +    qdev_prop_set_uint32(vms->gic, "revision", 3);
> +    qdev_prop_set_uint32(vms->gic, "num-cpu", smp_cpus);
> +    /*
> +     * Note that the num-irq property counts both internal and external
> +     * interrupts; there are always 32 of the former (mandated by GIC spec).
> +     */
> +    qdev_prop_set_uint32(vms->gic, "num-irq", NUM_IRQS + 32);
> +
> +    uint32_t redist0_capacity =
> +                vms->memmap[VMAPPLE_GIC_REDIST].size / GICV3_REDIST_SIZE;
> +    uint32_t redist0_count = MIN(smp_cpus, redist0_capacity);
> +
> +    redist_region_count = qlist_new();
> +    qlist_append_int(redist_region_count, redist0_count);
> +    qdev_prop_set_array(vms->gic, "redist-region-count", redist_region_count);
> +
> +    gicbusdev = SYS_BUS_DEVICE(vms->gic);
> +    sysbus_realize_and_unref(gicbusdev, &error_fatal);
> +    sysbus_mmio_map(gicbusdev, 0, vms->memmap[VMAPPLE_GIC_DIST].base);
> +    sysbus_mmio_map(gicbusdev, 1, vms->memmap[VMAPPLE_GIC_REDIST].base);
> +
> +    /*
> +     * Wire the outputs from each CPU's generic timer and the GICv3
> +     * maintenance interrupt signal to the appropriate GIC PPI inputs,
> +     * and the GIC's IRQ/FIQ/VIRQ/VFIQ interrupt outputs to the CPU's inputs.
> +     */
> +    for (i = 0; i < smp_cpus; i++) {
> +        DeviceState *cpudev = DEVICE(qemu_get_cpu(i));
> +
> +        /* Map the virt timer to PPI 27 */
> +        qdev_connect_gpio_out(cpudev, GTIMER_VIRT,
> +                              qdev_get_gpio_in(vms->gic,
> +                                               arm_gic_ppi_index(i, 27)));
> +
> +        /* Map the GIC IRQ and FIQ lines to CPU */
> +        sysbus_connect_irq(gicbusdev, i, qdev_get_gpio_in(cpudev, ARM_CPU_IRQ));
> +        sysbus_connect_irq(gicbusdev, i + smp_cpus,
> +                           qdev_get_gpio_in(cpudev, ARM_CPU_FIQ));
> +    }
> +}
> +
> +static void create_uart(const VMAppleMachineState *vms, int uart,
> +                        MemoryRegion *mem, Chardev *chr)
> +{
> +    hwaddr base = vms->memmap[uart].base;
> +    int irq = vms->irqmap[uart];
> +    DeviceState *dev = qdev_new(TYPE_PL011);
> +    SysBusDevice *s = SYS_BUS_DEVICE(dev);
> +
> +    qdev_prop_set_chr(dev, "chardev", chr);
> +    sysbus_realize_and_unref(SYS_BUS_DEVICE(dev), &error_fatal);
> +    memory_region_add_subregion(mem, base,
> +                                sysbus_mmio_get_region(s, 0));
> +    sysbus_connect_irq(s, 0, qdev_get_gpio_in(vms->gic, irq));
> +}
> +
> +static void create_rtc(const VMAppleMachineState *vms)
> +{
> +    hwaddr base = vms->memmap[VMAPPLE_RTC].base;
> +    int irq = vms->irqmap[VMAPPLE_RTC];
> +
> +    sysbus_create_simple("pl031", base, qdev_get_gpio_in(vms->gic, irq));
> +}
> +
> +static DeviceState *gpio_key_dev;
> +static void vmapple_powerdown_req(Notifier *n, void *opaque)
> +{
> +    /* use gpio Pin 3 for power button event */
> +    qemu_set_irq(qdev_get_gpio_in(gpio_key_dev, 0), 1);
> +}
> +
> +static void create_gpio_devices(const VMAppleMachineState *vms, int gpio,
> +                                MemoryRegion *mem)
> +{
> +    DeviceState *pl061_dev;
> +    hwaddr base = vms->memmap[gpio].base;
> +    int irq = vms->irqmap[gpio];
> +    SysBusDevice *s;
> +
> +    pl061_dev = qdev_new("pl061");
> +    /* Pull lines down to 0 if not driven by the PL061 */
> +    qdev_prop_set_uint32(pl061_dev, "pullups", 0);
> +    qdev_prop_set_uint32(pl061_dev, "pulldowns", 0xff);
> +    s = SYS_BUS_DEVICE(pl061_dev);
> +    sysbus_realize_and_unref(s, &error_fatal);
> +    memory_region_add_subregion(mem, base, sysbus_mmio_get_region(s, 0));
> +    sysbus_connect_irq(s, 0, qdev_get_gpio_in(vms->gic, irq));
> +    gpio_key_dev = sysbus_create_simple("gpio-key", -1,
> +                                        qdev_get_gpio_in(pl061_dev, 3));
> +}
> +
> +static void vmapple_firmware_init(VMAppleMachineState *vms,
> +                                  MemoryRegion *sysmem)
> +{
> +    hwaddr size = vms->memmap[VMAPPLE_FIRMWARE].size;
> +    hwaddr base = vms->memmap[VMAPPLE_FIRMWARE].base;
> +    const char *bios_name;
> +    int image_size;
> +    char *fname;
> +
> +    bios_name = MACHINE(vms)->firmware;
> +    if (!bios_name) {
> +        error_report("No firmware specified");
> +        exit(1);
> +    }
> +
> +    fname = qemu_find_file(QEMU_FILE_TYPE_BIOS, bios_name);
> +    if (!fname) {
> +        error_report("Could not find ROM image '%s'", bios_name);
> +        exit(1);
> +    }
> +
> +    memory_region_init_ram(&vms->fw_mr, NULL, "firmware", size, NULL);

Pass: &error_fatal

> +    image_size = load_image_mr(fname, &vms->fw_mr);
> +
> +    g_free(fname);
> +    if (image_size < 0) {
> +        error_report("Could not load ROM image '%s'", bios_name);
> +        exit(1);
> +    }
> +
> +    memory_region_add_subregion(get_system_memory(), base, &vms->fw_mr);
> +}
> +
> +static void create_pcie(VMAppleMachineState *vms)
> +{
> +    hwaddr base_mmio = vms->memmap[VMAPPLE_PCIE_MMIO].base;
> +    hwaddr size_mmio = vms->memmap[VMAPPLE_PCIE_MMIO].size;
> +    hwaddr base_ecam = vms->memmap[VMAPPLE_PCIE_ECAM].base;
> +    hwaddr size_ecam = vms->memmap[VMAPPLE_PCIE_ECAM].size;
> +    int irq = vms->irqmap[VMAPPLE_PCIE];
> +    MemoryRegion *mmio_alias;
> +    MemoryRegion *mmio_reg;
> +    MemoryRegion *ecam_alias;
> +    MemoryRegion *ecam_reg;
> +    DeviceState *dev;
> +    int i;
> +    PCIHostState *pci;
> +    DeviceState *usb_controller;
> +    USBBus *usb_bus;
> +
> +    dev = qdev_new(TYPE_GPEX_HOST);
> +    qdev_prop_set_uint32(dev, "nr-irqs", GPEX_NUM_IRQS);
> +    sysbus_realize_and_unref(SYS_BUS_DEVICE(dev), &error_fatal);
> +
> +    /* Map only the first size_ecam bytes of ECAM space */
> +    ecam_alias = g_new0(MemoryRegion, 1);
> +    ecam_reg = sysbus_mmio_get_region(SYS_BUS_DEVICE(dev), 0);
> +    memory_region_init_alias(ecam_alias, OBJECT(dev), "pcie-ecam",
> +                             ecam_reg, 0, size_ecam);
> +    memory_region_add_subregion(get_system_memory(), base_ecam, ecam_alias);
> +
> +    /*
> +     * Map the MMIO window from [0x50000000-0x7fff0000] in PCI space into
> +     * system address space at [0x50000000-0x7fff0000].
> +     */
> +    mmio_alias = g_new0(MemoryRegion, 1);
> +    mmio_reg = sysbus_mmio_get_region(SYS_BUS_DEVICE(dev), 1);
> +    memory_region_init_alias(mmio_alias, OBJECT(dev), "pcie-mmio",
> +                             mmio_reg, base_mmio, size_mmio);
> +    memory_region_add_subregion(get_system_memory(), base_mmio, mmio_alias);
> +
> +    for (i = 0; i < GPEX_NUM_IRQS; i++) {
> +        sysbus_connect_irq(SYS_BUS_DEVICE(dev), i,
> +                           qdev_get_gpio_in(vms->gic, irq + i));
> +        gpex_set_irq_num(GPEX_HOST(dev), i, irq + i);
> +    }
> +
> +    pci = PCI_HOST_BRIDGE(dev);
> +    vms->bus = pci->bus;
> +    g_assert_nonnull(vms->bus);
> +
> +    while ((dev = qemu_create_nic_device("virtio-net-pci", true, NULL))) {
> +        qdev_realize_and_unref(dev, BUS(vms->bus), &error_fatal);
> +    }
> +
> +    usb_controller = qdev_new(TYPE_QEMU_XHCI);
> +    qdev_realize_and_unref(usb_controller, BUS(pci->bus), &error_fatal);
> +
> +    usb_bus = USB_BUS(object_resolve_type_unambiguous(TYPE_USB_BUS,
> +                                                      &error_fatal));
> +    usb_create_simple(usb_bus, "usb-kbd");
> +    usb_create_simple(usb_bus, "usb-tablet");
> +}
> +
> +static void vmapple_reset(void *opaque)
> +{
> +    VMAppleMachineState *vms = opaque;
> +    hwaddr base = vms->memmap[VMAPPLE_FIRMWARE].base;
> +
> +    cpu_set_pc(first_cpu, base);
> +}
> +
> +static void mach_vmapple_init(MachineState *machine)
> +{
> +    VMAppleMachineState *vms = VMAPPLE_MACHINE(machine);
> +    MachineClass *mc = MACHINE_GET_CLASS(machine);
> +    const CPUArchIdList *possible_cpus;
> +    MemoryRegion *sysmem = get_system_memory();
> +    int n;
> +    unsigned int smp_cpus = machine->smp.cpus;
> +    unsigned int max_cpus = machine->smp.max_cpus;
> +
> +    vms->memmap = memmap;
> +    machine->usb = true;
> +
> +    possible_cpus = mc->possible_cpu_arch_ids(machine);
> +    assert(possible_cpus->len == max_cpus);
> +    for (n = 0; n < possible_cpus->len; n++) {
> +        Object *cpu;
> +        CPUState *cs;
> +
> +        if (n >= smp_cpus) {
> +            break;
> +        }
> +
> +        cpu = object_new(possible_cpus->cpus[n].type);
> +        object_property_set_int(cpu, "mp-affinity",
> +                                possible_cpus->cpus[n].arch_id, NULL);
> +
> +        cs = CPU(cpu);
> +        cs->cpu_index = n;
> +
> +        numa_cpu_pre_plug(&possible_cpus->cpus[cs->cpu_index], DEVICE(cpu),
> +                          &error_fatal);
> +
> +        object_property_set_bool(cpu, "has_el3", false, NULL);
> +        object_property_set_bool(cpu, "has_el2", false, NULL);
> +        object_property_set_int(cpu, "psci-conduit", QEMU_PSCI_CONDUIT_HVC,
> +                                NULL);
> +
> +        /* Secondary CPUs start in PSCI powered-down state */
> +        if (n > 0) {
> +            object_property_set_bool(cpu, "start-powered-off", true, NULL);
> +        }
> +
> +        object_property_set_link(cpu, "memory", OBJECT(sysmem), &error_abort);
> +        qdev_realize(DEVICE(cpu), NULL, &error_fatal);
> +        object_unref(cpu);
> +    }
> +
> +    memory_region_add_subregion(sysmem, vms->memmap[VMAPPLE_MEM].base,
> +                                machine->ram);
> +
> +    create_gic(vms, sysmem);
> +    create_bdif(vms, sysmem);
> +    create_pvpanic(vms, sysmem);
> +    create_aes(vms, sysmem);
> +    create_gfx(vms, sysmem);
> +    create_uart(vms, VMAPPLE_UART, sysmem, serial_hd(0));
> +    create_rtc(vms);
> +    create_pcie(vms);
> +
> +    create_gpio_devices(vms, VMAPPLE_GPIO, sysmem);
> +
> +    vmapple_firmware_init(vms, sysmem);
> +    create_cfg(vms, sysmem);
> +
> +    /* connect powerdown request */
> +    vms->powerdown_notifier.notify = vmapple_powerdown_req;
> +    qemu_register_powerdown_notifier(&vms->powerdown_notifier);
> +
> +    vms->bootinfo.ram_size = machine->ram_size;
> +    vms->bootinfo.board_id = -1;
> +    vms->bootinfo.loader_start = vms->memmap[VMAPPLE_MEM].base;
> +    vms->bootinfo.skip_dtb_autoload = true;
> +    vms->bootinfo.firmware_loaded = true;
> +    arm_load_kernel(ARM_CPU(first_cpu), machine, &vms->bootinfo);
> +
> +    qemu_register_reset(vmapple_reset, vms);
> +}
> +
> +static CpuInstanceProperties
> +vmapple_cpu_index_to_props(MachineState *ms, unsigned cpu_index)
> +{
> +    MachineClass *mc = MACHINE_GET_CLASS(ms);
> +    const CPUArchIdList *possible_cpus = mc->possible_cpu_arch_ids(ms);
> +
> +    assert(cpu_index < possible_cpus->len);
> +    return possible_cpus->cpus[cpu_index].props;
> +}
> +
> +
> +static int64_t vmapple_get_default_cpu_node_id(const MachineState *ms, int idx)
> +{
> +    return idx % ms->numa_state->num_nodes;
> +}
> +
> +static const CPUArchIdList *vmapple_possible_cpu_arch_ids(MachineState *ms)
> +{
> +    int n;
> +    unsigned int max_cpus = ms->smp.max_cpus;
> +
> +    if (ms->possible_cpus) {
> +        assert(ms->possible_cpus->len == max_cpus);
> +        return ms->possible_cpus;
> +    }
> +
> +    ms->possible_cpus = g_malloc0(sizeof(CPUArchIdList) +
> +                                  sizeof(CPUArchId) * max_cpus);
> +    ms->possible_cpus->len = max_cpus;
> +    for (n = 0; n < ms->possible_cpus->len; n++) {
> +        ms->possible_cpus->cpus[n].type = ms->cpu_type;
> +        ms->possible_cpus->cpus[n].arch_id =
> +            arm_build_mp_affinity(n, GICV3_TARGETLIST_BITS);
> +        ms->possible_cpus->cpus[n].props.has_thread_id = true;
> +        ms->possible_cpus->cpus[n].props.thread_id = n;
> +    }
> +    return ms->possible_cpus;
> +}
> +
> +static void vmapple_get_uuid(Object *obj, Visitor *v, const char *name,
> +                             void *opaque, Error **errp)
> +{
> +    VMAppleMachineState *vms = VMAPPLE_MACHINE(obj);
> +    uint64_t value = be64_to_cpu(vms->uuid);
> +
> +    visit_type_uint64(v, name, &value, errp);
> +}
> +
> +static void vmapple_set_uuid(Object *obj, Visitor *v, const char *name,
> +                             void *opaque, Error **errp)
> +{
> +    VMAppleMachineState *vms = VMAPPLE_MACHINE(obj);
> +    Error *error = NULL;
> +    uint64_t value;
> +
> +    visit_type_uint64(v, name, &value, &error);
> +    if (error) {
> +        error_propagate(errp, error);
> +        return;
> +    }
> +
> +    vms->uuid = cpu_to_be64(value);
> +}

vmapple converts the value to big endian here and vmapple-cfg converts 
it back later. What's the intention?

 > +> +static void vmapple_machine_class_init(ObjectClass *oc, void *data)
> +{
> +    MachineClass *mc = MACHINE_CLASS(oc);
> +
> +    mc->init = mach_vmapple_init;
> +    mc->max_cpus = 32;
> +    mc->block_default_type = IF_VIRTIO;
> +    mc->no_cdrom = 1;
> +    mc->pci_allow_0_address = true;
> +    mc->minimum_page_bits = 12;
> +    mc->possible_cpu_arch_ids = vmapple_possible_cpu_arch_ids;
> +    mc->cpu_index_to_instance_props = vmapple_cpu_index_to_props;
> +    if (hvf_enabled()) {
> +        mc->default_cpu_type = ARM_CPU_TYPE_NAME("host");
> +    } else {
> +        mc->default_cpu_type = ARM_CPU_TYPE_NAME("max");
> +    }

Remove this conditional. VMApple only works with the host model.

(I wonder if this works on KVM on Asahi Linux by the way. 
apple-gfx-vmapple won't work, but perhaps anything else may just work.)

> +    mc->get_default_cpu_node_id = vmapple_get_default_cpu_node_id;
> +    mc->default_ram_id = "mach-vmapple.ram";
> +
> +    object_register_sugar_prop(TYPE_VIRTIO_PCI, "disable-legacy",
> +                               "on", true);
> +
> +    object_class_property_add(oc, "uuid", "uint64", vmapple_get_uuid,
> +                              vmapple_set_uuid, NULL, NULL);
> +    object_class_property_set_description(oc, "uuid", "Machine UUID (SDOM)");
> +}
> +
> +static void vmapple_instance_init(Object *obj)
> +{
> +    VMAppleMachineState *vms = VMAPPLE_MACHINE(obj);
> +
> +    vms->irqmap = irqmap;
> +}
> +
> +static const TypeInfo vmapple_machine_info = {
> +    .name          = TYPE_VMAPPLE_MACHINE,
> +    .parent        = TYPE_MACHINE,
> +    .abstract      = true,
> +    .instance_size = sizeof(VMAppleMachineState),
> +    .class_size    = sizeof(VMAppleMachineClass),
> +    .class_init    = vmapple_machine_class_init,
> +    .instance_init = vmapple_instance_init,
> +};
> +
> +static void machvmapple_machine_init(void)
> +{
> +    type_register_static(&vmapple_machine_info);
> +}
> +type_init(machvmapple_machine_init);
> +
> +static void vmapple_machine_8_1_options(MachineClass *mc)
> +{
> +}
> +DEFINE_VMAPPLE_MACHINE_AS_LATEST(8, 1)

Please update this.

> +

Re: [PATCH v3 06/14] hw: Add vmapple subdir

[Akihiko Odaki]

On 2024/09/28 17:57, Phil Dennis-Jordan wrote:
> From: Alexander Graf <graf@amazon.com>
> 
> We will introduce a number of devices that are specific to the vmapple
> target machine. To keep them all tidily together, let's put them into
> a single target directory.
> 
> Signed-off-by: Alexander Graf <graf@amazon.com>
> Signed-off-by: Phil Dennis-Jordan <phil@philjordan.eu>

Reviewed-by: Akihiko Odaki <akihiko.odaki@daynix.com>

Re: [PATCH v3 07/14] hw/misc/pvpanic: Add MMIO interface

[Akihiko Odaki]

On 2024/09/28 17:57, Phil Dennis-Jordan wrote:
> From: Alexander Graf <graf@amazon.com>
> 
> In addition to the ISA and PCI variants of pvpanic, let's add an MMIO
> platform device that we can use in embedded arm environments.
> 
> Signed-off-by: Alexander Graf <graf@amazon.com>
> Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org>
> Tested-by: Philippe Mathieu-Daudé <philmd@linaro.org>
> Signed-off-by: Phil Dennis-Jordan <phil@philjordan.eu>

Reviewed-by: Akihiko Odaki <akihiko.odaki@daynix.com>

Re: [PATCH v3 08/14] hvf: arm: Ignore writes to CNTP_CTL_EL0

[Akihiko Odaki]

On 2024/09/28 17:57, Phil Dennis-Jordan wrote:
> From: Alexander Graf <graf@amazon.com>
> 
> MacOS unconditionally disables interrupts of the physical timer on boot
> and then continues to use the virtual one. We don't really want to support
> a full physical timer emulation, so let's just ignore those writes.
> 
> Signed-off-by: Alexander Graf <graf@amazon.com>
> Signed-off-by: Phil Dennis-Jordan <phil@philjordan.eu>

Reviewed-by: Akihiko Odaki <akihiko.odaki@daynix.com>

Re: [PATCH v3 01/14] hw/display/apple-gfx: Introduce ParavirtualizedGraphics.Framework support

[Phil Dennis-Jordan]

[-- Attachment #1: Type: text/plain, Size: 40513 bytes --]

On Thu, 3 Oct 2024 at 09:09, Akihiko Odaki <akihiko.odaki@daynix.com> wrote:

> On 2024/10/02 22:33, Phil Dennis-Jordan wrote:
> >
> >
> >      > +#include "apple-gfx.h"
> >      > +#include "monitor/monitor.h"
> >      > +#include "hw/sysbus.h"
> >      > +#include "hw/irq.h"
> >      > +#include "trace.h"
> >      > +#import <ParavirtualizedGraphics/ParavirtualizedGraphics.h>
> >      > +
> >      > +_Static_assert(__aarch64__, "");
> >
> >     I don't think this assertion is worthwhile. This assertion will
> trigger
> >     if you accidentally remove depends on AARCH64 from Kconfig, but I
> don't
> >     think such code change happens by accident, and there is no reason to
> >     believe that this assertion won't be removed in such a case.
> >
> >
> > As far as I'm aware the Kconfig AARCH64 dependency is for the /target/
> > architecture, not the /host/ architecture? The static assert checks for
> > the latter. The PGIOSurfaceHostDeviceDescriptor type isn't available at
> > all on non-aarch64 macOS hosts. I've not had any luck with using this
> > variant of the device on x86-64 hosts simply by disabling any surface
> > mapper code.
> >
> > Incidentally, if you know of a way to depend on a specific /host/
> > architecture in the Kconfig, that would be even better. I couldn't spot
> > a way of doing that though.
>
> I got your intention now. The correct way to do that is to check for cpu
> == 'aarch64'. Having assertion will break qemu-system-aarch64 on Intel
> Macs.
>

OK, looks Iike that needs to be done at the meson.build level not Kconfig,
but this seems to work:

if cpu == 'aarch64'
  system_ss.add(when: 'CONFIG_MAC_PVG_MMIO',  if_true:
[files('apple-gfx-mmio.m'), pvg, metal])
endif



> >
> >     It is dangerous to unlock BQL at an arbitrary place. Instead of
> >     unlocking, I suggest:
> >     - running [s->pgiosfc mmioReadAtOffset:offset] on another thread
> >     - using a bottom half to request operations that require BQL from the
> >     thread running [s->pgiosfc mmioReadAtOffset:offset]
> >     - calling AIO_WAIT_WHILE() to process the bottom half and to wait for
> >     the completion of [s->pgiosfc mmioReadAtOffset:offset]
> >
> >
> > OK, I think I see what you mean, I'll try to rework things around that
> > pattern. Any preference on how I kick off the job on the other thread?
> > As we necessarily need to use libdispatch in a bunch of places in this
> > code anyway, I guess dispatch_async() would probably be the simplest?
>
> Perhaps so. The QEMU way is to use a bottom half with AioContext, but
> you can't simultaneously run a dispatch queue and AioContext in one
> thread so you have to use the dispatch queue if you need one.
>
> >
> >      > +    res = [s->pgiosfc mmioReadAtOffset:offset];
> >      > +    bql_lock();
> >      > +
> >      > +    trace_apple_iosfc_read(offset, res);
> >      > +
> >      > +    return res;
> >      > +}
> >      > +
> >      > +static void apple_iosfc_write(
> >      > +    void *opaque, hwaddr offset, uint64_t val, unsigned size)
> >      > +{
> >      > +    AppleGFXVmappleState *s = opaque;
> >      > +
> >      > +    trace_apple_iosfc_write(offset, val);
> >      > +
> >      > +    [s->pgiosfc mmioWriteAtOffset:offset value:val];
> >      > +}
> >      > +
> >      > +static const MemoryRegionOps apple_iosfc_ops = {
> >      > +    .read = apple_iosfc_read,
> >      > +    .write = apple_iosfc_write,
> >      > +    .endianness = DEVICE_LITTLE_ENDIAN,
> >      > +    .valid = {
> >      > +        .min_access_size = 4,
> >      > +        .max_access_size = 8,
> >      > +    },
> >      > +    .impl = {
> >      > +        .min_access_size = 4,
> >      > +        .max_access_size = 8,
> >      > +    },
> >      > +};
> >      > +
> >      > +static PGIOSurfaceHostDevice
> >     *apple_gfx_prepare_iosurface_host_device(
> >      > +    AppleGFXVmappleState *s)
> >      > +{
> >      > +    PGIOSurfaceHostDeviceDescriptor *iosfc_desc =
> >      > +        [PGIOSurfaceHostDeviceDescriptor new];
> >      > +    PGIOSurfaceHostDevice *iosfc_host_dev = nil;
> >      > +
> >      > +    iosfc_desc.mapMemory =
> >      > +        ^(uint64_t phys, uint64_t len, bool ro, void **va, void
> >     *e, void *f) {
> >      > +            trace_apple_iosfc_map_memory(phys, len, ro, va, e,
> f);
> >      > +            MemoryRegion *tmp_mr;
> >      > +            *va = gpa2hva(&tmp_mr, phys, len, NULL);
> >
> >     Use: dma_memory_map()
> >
> >
> > That doesn't seem to be a precisely equivalent operation. It also says
> > in its headerdoc,
> >
> >     Use only for reads OR writes - not for read-modify-write operations.
> >
> > which I don't think we can guarantee here at all.
> >
> > I guess I can call it twice, once for writing and once for reading, but
> > given that the dma_memory_unmap operation marks the area dirty, I'm not
> > it's intended for what I understand the use case here to be: As far as I
> > can tell, the PV graphics device uses (some) of this memory to exchange
> > data in a cache-coherent way between host and guest, e.g. as a lock-free
> > ring buffer, using atomic operations as necessary. (This works because
> > it's a PV device: it "knows" the other end just another CPU core (or
> > even the same one) executing in a different hypervisor context.) This
> > doesn't really match "traditional" DMA patterns where there's either a
> > read or a write happening.
>
> I think the story is a bit different for this VMApple variant. Probably
> the CPU and GPU in Apple Silicon is cache-coherent so you can map normal
> memory for GPU without any kind of cache maintenance.
>
> Cache conherency of CPU and GPU in Apple Silicon is implied with Apple's
> documentation; it says you don't need to synchronize resources for
> MTLStorageModeShared, which is the default for Apple Silicon.
>
> https://developer.apple.com/documentation/metal/resource_fundamentals/synchronizing_a_managed_resource_in_macos
>
> The name "IOSurface" also implies it is used not only for e.g., ring
> buffer but also for real data.
>

Note that the PGTask map/unmap callbacks appear to have equivalent
semantics, so it's not just the surface mapping.


> >
> > Hunting around some more for alternative APIs, there's also
> > memory_region_get_ram_ptr(), but I'm not sure its restrictions apply
> > here either.
>
> I think you can call memory_access_is_direct() to check if the
> requirement is satisfied.
>
> It will still break dirty page tracking implemented by
> dma_memory_unmap() and others, but it's broken for hvf, which does not
> implement dirty page tracking either.
>



>
> >      > +            return (bool)true;
> >
> >     Why cast?
> >
> >
> > Good question. Not originally my code, so I've fixed all the instances I
> > could find now.
>

OK, it turns out the reason for this is that C treats 'true' as an int,
which then becomes the block's inferred return type - and the callbacks are
expecting bool-returning blocks.

I've fixed it by explicitly specifying the block return type and removing
the cast in the return statement:

iosfc_desc.unmapMemory =
       ^bool(…) {
           …
           return true;
       };


> >      > +
> >      > +    iosfc_desc.unmapMemory =
> >      > +        ^(void *a, void *b, void *c, void *d, void *e, void *f) {
> >      > +            trace_apple_iosfc_unmap_memory(a, b, c, d, e, f);
> >      > +            return (bool)true;
> >      > +        };
> >      > +
> >      > +    iosfc_desc.raiseInterrupt = ^(uint32_t vector) {
> >      > +        trace_apple_iosfc_raise_irq(vector);
> >      > +        bool locked = bql_locked();
> >      > +        if (!locked) {
> >      > +            bql_lock();
> >      > +        }
> >       > +        qemu_irq_pulse(s->irq_iosfc);> +        if (!locked) {
> >      > +            bql_unlock();
> >      > +        }
> >      > +        return (bool)true;
> >      > +    };
> >      > +
> >      > +    iosfc_host_dev =
> >      > +        [[PGIOSurfaceHostDevice alloc]
> >     initWithDescriptor:iosfc_desc];
> >      > +    [iosfc_desc release];
> >      > +    return iosfc_host_dev;
> >      > +}
> >      > +
> >      > +static void apple_gfx_vmapple_realize(DeviceState *dev, Error
> >     **errp)
> >      > +{
> >      > +    @autoreleasepool {
> >
> >     This autoreleasepool is not used.
> >
> >
> > It is definitely used inside the apple_gfx_common_realize() call. It's
> > also impossible to say whether [PGDeviceDescriptor new] uses autorelease
> > semantics internally, so it seemed safer to wrap the whole thing in an
> > outer pool.
>
> Theoretically, It should be safe to assume the callee creates
> autoreleasepool by themselves as needed in general. We have bunch of
> code to call Objective-C APIs without creating autoreleasepool in the
> caller. Practically, [PGDeviceDescriptor new] is likely to be
> implemented with ARC, which wraps methods in autoreleasepool as necessary.
>

As far as I'm aware, ARC does NOT automatically insert autorelease pool
blocks. The reason you rarely need to create autoreleasepool blocks in
"plain" Objective-C programming is that Cocoa/CFRunloop/libdispatch event
handlers run each event in an autoreleasepool. So you don't need to create
them explicitly when using dispatch_async and similar, or when running code
on the main thread (which runs inside
NSApplicationMain/CFRunloopRun/dispatch_main).

As far as I'm aware, if you don't explicitly define autoreleasepools in raw
threads created with the pthreads API, any autoreleased objects will leak.
At least I've not found any specification/documentation contradicting this.
And most code in Qemu runs on such raw threads, so we need to play it safe
with regard to autorelease semantics.

Whether the existing Qemu Objective-C code is safe in this regard I don't
know for certain, but I've certainly paid attention to this aspect when
modifying ui/cocoa.m in the past, and indeed most of that code runs on the
main thread. Note also how I wrap the apple_gfx_render_new_frame call in a
pool when it can't be guaranteed it's running on a dispatch queue because
the command buffer inside that uses autorelease semantics.

Functions that uses a method that returns autorelease resources should
> be wrapped with autoreleasepool instead of assuming the caller creates
> autoreleasepool for them.
>

I'm treating apple_gfx_common_realize as an internal API, and I don't think
expecting its callers to wrap it in an autoreleasepool block is
unreasonable. I can certainly explicitly document this in a comment.



> >      > diff --git a/hw/display/apple-gfx.m b/hw/display/apple-gfx.m
> >      > new file mode 100644
> >      > index 00000000000..837300f9cd4
> >      > --- /dev/null
> >      > +++ b/hw/display/apple-gfx.m
> >      > @@ -0,0 +1,536 @@
> >      > +/*
> >      > + * QEMU Apple ParavirtualizedGraphics.framework device
> >      > + *
> >      > + * Copyright © 2023 Amazon.com, Inc. or its affiliates. All
> >     Rights Reserved.
> >      > + *
> >      > + * This work is licensed under the terms of the GNU GPL, version
> >     2 or later.
> >      > + * See the COPYING file in the top-level directory.
> >      > + *
> >      > + * ParavirtualizedGraphics.framework is a set of libraries that
> >     macOS provides
> >      > + * which implements 3d graphics passthrough to the host as well
> as a
> >      > + * proprietary guest communication channel to drive it. This
> >     device model
> >      > + * implements support to drive that library from within QEMU.
> >      > + */
> >      > +
> >      > +#include "apple-gfx.h"
> >      > +#include "trace.h"
> >      > +#include "qemu/main-loop.h"
> >      > +#include "ui/console.h"
> >      > +#include "monitor/monitor.h"
> >      > +#include "qapi/error.h"
> >      > +#include "migration/blocker.h"
> >      > +#include <mach/mach_vm.h>
> >      > +#import <ParavirtualizedGraphics/ParavirtualizedGraphics.h>
> >      > +
> >      > +static const PGDisplayCoord_t apple_gfx_modes[] = {
> >      > +    { .x = 1440, .y = 1080 },
> >      > +    { .x = 1280, .y = 1024 },
> >      > +};
> >      > +
> >      > +typedef struct PGTask_s { // Name matches forward declaration in
> >     PG header
> >
> >     Let's name it AppleGFXTask. It is a common practice to have the same
> >     tag
> >     name and typedef in QEMU.
> >
> >
> > This is defining a forward-declared type from framework headers which is
> > opaque from the framework's point of view. We do not get to choose its
> > struct name. The alternative is having casts wherever these objects are
> > being passed between our code and the framework. (See the original v1/v2
> > vmapple patch series for how messy this is.)
>
> I got the idea. Let's not avoid the typedef then to clarify the naming
> is not under our control.
>

I'm not sure what you mean by this double negative. Are you saying, don't
add our own typedef for struct PGTask_s at all, just use the
framework-supplied PGTask_t where appropriate?


>
> >
> >      > +static void apple_gfx_render_frame_completed(AppleGFXState *s,
> >     void *vram,
> >      > +                                             id<MTLTexture>
> texture)
> >      > +{
> >      > +    --s->pending_frames;
> >      > +    assert(s->pending_frames >= 0);
> >      > +
> >      > +    if (vram != s->vram) {
> >      > +        /* Display mode has changed, drop this old frame. */
> >      > +        assert(texture != s->texture);
> >      > +        g_free(vram);
> >
> >     This management of buffers looks a bit convoluted. I suggest
> >     remembering
> >     the width and height instead of pointers and comparing them. This way
> >     you can free resources in set_mode().
> >
> >
> > Yeah, I suppose that works, I can change that around.
> >
> >      > +    } else {
> >      > +        copy_mtl_texture_to_surface_mem(texture, vram);
> >
> >     Writing vram outside BQL may result in tearing.
> >
> >
> > As far as I can tell(*), QXL does the same. I couldn't find any examples
> > of double-buffering in any of the existing display devices, which would
> > be the only way to do async updates efficiently and without tearing. In
> > any case, this solution is still vastly better than a regular VGA
> > device, which suffers from very visible tearing with macOS on the guest
> > side anyway. And in an ideal world, we'd pass through the rendered
> > texture directly to the Cocoa UI code rather than copying out only for
> > the CPU to draw it back into a window surface which is then passed to
> > the GPU for host side rendering. But I felt this patch is already very,
> > very large, and if anyone cares, we can fix imperfections in subsequent
> > updates.
> >
> > (*)The rendering code in that device is also fairly complex, so I may be
> > misreading it.
>
> QXL always modifies the surface with BQL. The surface is modified with
> qxl_blit(), which is called by qxl_render_update_area_unlocked().
> qxl_render_update_area_unlocked() is called by either of
> qxl_render_update() and qxl_render_update_area_bh(). Both of them are
> called with BQL. The name includes "unlocked", but it means it is called
> without holding QXL-internal lock.
>
> Most devices works entirely with BQL so they don't perform double
> buffering. apple-gfx can do the same.
>

I think we can safely move apple-gfx's framebuffer state management back
inside the BQL, yes. I just figured that copying dozens of megabytes of
framebuffer data on every frame while holding the lock was not going to
help BQL contention. Especially as PVG does not have a concept of dirty
areas, so we must copy the whole framebuffer every time. (Unless we were to
implement dirty area detection ourselves.)

Unfortunately, implementing double-buffering would require a major rework
of Qemu's whole surface management, console code, and probably most of the
UI implementations. I'm guessing the OpenGL fast-path sidesteps all of
this, so replicating that with Metal would probably be the easier way
forward. (Although doing all this graphics stuff inside the BQL generally
seems like a major architectural flaw; I suppose most enterprise use of
Qemu does not involve the framebuffer, so it's not shown up in BQL
contention profiling there. It certainly does in desktop use, although at
least on macOS hosts there are far worse culprits in that regard.)

>
> >      > +        if (s->gfx_update_requested) {
> >      > +            s->gfx_update_requested = false;
> >      > +            dpy_gfx_update_full(s->con);
> >      > +            graphic_hw_update_done(s->con);
> >       > +            s->new_frame_ready = false;
> >
> >     This assignment is unnecessary as s->new_frame_ready is always false
> >     when s->gfx_update_requested. If you want to make sure
> >     s->gfx_update_requested and s->new_frame_ready are mutually
> exclusive,
> >     use one enum value instead of having two bools.
> >
> >
> > I'll need to refresh my memory and get back to you on this one, it's
> > been so many months since I actively worked on this code.
> >
> >      > +        } else {
> >      > +            s->new_frame_ready = true;
> >      > +        }
> >      > +    }
> >      > +    if (s->pending_frames > 0) {
> >      > +        apple_gfx_render_new_frame(s);
> >      > +    }
> >      > +}
> >      > +
> >      > +static void apple_gfx_fb_update_display(void *opaque)
> >      > +{
> >      > +    AppleGFXState *s = opaque;
> >      > +
> >      > +    dispatch_async(s->render_queue, ^{
> >      > +        if (s->pending_frames > 0) {
> >
> >     It should check for s->new_frame_ready as
> >     apple_gfx_render_frame_completed() doesn't check if
> >     s->pending_frames > 0 before calling graphic_hw_update_done(), which
> is
> >     inconsistent.
> >
> >
> > pending_frames is about guest-side frames that are queued to be rendered
> > by the host GPU.
> > new_frame_ready being true indicates that the contents of the Qemu
> > console surface has been updated with new frame data since the last
> > gfx_update.
> > gfx_update_requested indicates that gfx_update is currently awaiting an
> > async completion (graphic_hw_update_done) but the surface has not
> > received a new frame content, but the GPU is stily busy drawing one.
> >
> > apple_gfx_render_frame_completed is scheduled exactly once per pending
> > frame, so pending_frames > 0 is an invariant there. (Hence the assert.)>
> > I don't think there is any inconsistency here, but I'll double check.
> > It's possible that there's an easier way to express the state machine,
> > and I'll take a look at that.
>
> I meant that apple_gfx_render_frame_completed() does not check if the
> frame is the last one currently pending. apple_gfx_fb_update_display()
> ignores a new ready frame when there is a more pending frame, but
> apple_gfx_render_frame_completed() unconditionally fires
> graphic_hw_update_done() even if there is a more pending frame. And I
> think apple_gfx_render_frame_completed() is right and
> apple_gfx_fb_update_display() is wrong in such a situation.
>
>
OK, got it. And yes, I agree.


> >
> >     Checking if s->pending_frames > 0 both in
> apple_gfx_fb_update_display()
> >     and apple_gfx_render_frame_completed() is also problematic as that
> can
> >     defer graphic_hw_update_done() indefinitely if we are getting new
> >     frames
> >     too fast.
> >
> >
> > I see what you mean about this part. I'll have to test it, but I guess
> > we should reverse the priority, like this:
> >
> >          if (s->new_frame_ready) {
> >              dpy_gfx_update_full(s->con);
> >              s->new_frame_ready = false;
> >              graphic_hw_update_done(s->con);
> >          } else if (s->pending_frames > 0) {
> >              s->gfx_update_requested = true;
> >          } else {
> >              graphic_hw_update_done(s->con);
> >          }
> >
> > 1. If we already have a frame, ready to be displayed, return it
> immediately.
> > 2. If the guest has reported that it's completed a frame and the GPU is
> > currently busy rendering it, defer graphic_hw_update_done until it's
> done.
> > 3. If the guest reports no changes to its display, indicate this back to
> > Qemu as a no-op display update graphic_hw_update_done() with no
> > dpy_gfx_update* call.
>
> Yes, that looks correct.
>
> >
> >      > +            s->gfx_update_requested = true;
> >      > +        } else {
> >      > +            if (s->new_frame_ready) {
> >      > +                dpy_gfx_update_full(s->con);
> >      > +                s->new_frame_ready = false;
> >      > +            }
> >      > +            graphic_hw_update_done(s->con);
> >       > +        }> +    });
> >      > +}
> >      > +
> >      > +static const GraphicHwOps apple_gfx_fb_ops = {
> >      > +    .gfx_update = apple_gfx_fb_update_display,
> >      > +    .gfx_update_async = true,
> >      > +};
> >      > +
> >      > +static void update_cursor(AppleGFXState *s)
> >      > +{
> >      > +    dpy_mouse_set(s->con, s->pgdisp.cursorPosition.x,
> >      > +                  s->pgdisp.cursorPosition.y, s->cursor_show);
> >      > +}
> >      > +
> >      > +static void set_mode(AppleGFXState *s, uint32_t width, uint32_t
> >     height)
> >      > +{
> >      > +    void *vram = NULL;
> >      > +    DisplaySurface *surface;
> >      > +    MTLTextureDescriptor *textureDescriptor;
> >      > +    id<MTLTexture> texture = nil;
> >      > +    __block bool no_change = false;
> >      > +
> >      > +    dispatch_sync(s->render_queue,
> >
> >     Calling dispatch_sync() while holding BQL may result in deadlock.
> >
> > Only if any code executed on the same dispatch queue acquires the lock
> > either directly or transitively. I believe I have ensure this is not
> > done on the reqnder_queue, have you found anywhere this is the case?
>
> The documentation is not clear what threads a dispatch queue runs on. We
> can have a deadlock if they lock the BQL.
>

dispatch_sync is a synchronisation primitive (it waits for and asserts
exclusive access to the given queue), it doesn't actually do any thread
scheduling. Work scheduled asynchronously to non-main dispatch queues will
otherwise run on libdispatch pool threads. Running blocks on dispatch
queues will not preempt and schedule it on other threads which may or may
not be holding some locks.

So the only way this code will deadlock is if any code scheduled to
render_queue directly or transitively acquires the BQL. None of it does,
although updating the console while holding the BQL rather complicates this.


> >
> >      > +        ^{
> >      > +            if (s->surface &&
> >      > +                width == surface_width(s->surface) &&
> >      > +                height == surface_height(s->surface)) {
> >      > +                no_change = true;
> >      > +            }
> >      > +        });
> >      > +
> >      > +    if (no_change) {
> >      > +        return;
> >      > +    }
> >      > +
> >      > +    vram = g_malloc0(width * height * 4);
> >      > +    surface = qemu_create_displaysurface_from(width, height,
> >     PIXMAN_LE_a8r8g8b8,
> >      > +                                              width * 4, vram);
> >      > +
> >      > +    @autoreleasepool {
> >      > +        textureDescriptor =
> >      > +            [MTLTextureDescriptor
> >      > +
> >     texture2DDescriptorWithPixelFormat:MTLPixelFormatBGRA8Unorm
> >      > +                                             width:width
> >      > +                                            height:height
> >      > +                                         mipmapped:NO];
> >      > +        textureDescriptor.usage = s->pgdisp.minimumTextureUsage;
> >      > +        texture = [s->mtl
> >     newTextureWithDescriptor:textureDescriptor];
> >      > +    }
> >      > +
> >      > +    s->using_managed_texture_storage =
> >      > +        (texture.storageMode == MTLStorageModeManaged);
> >      > +
> >      > +    dispatch_sync(s->render_queue,
> >      > +        ^{
> >      > +            id<MTLTexture> old_texture = nil;
> >      > +            void *old_vram = s->vram;
> >      > +            s->vram = vram;
> >      > +            s->surface = surface;
> >      > +
> >      > +            dpy_gfx_replace_surface(s->con, surface);
> >      > +
> >      > +            old_texture = s->texture;
> >      > +            s->texture = texture;
> >      > +            [old_texture release];
> >
> >     You can just do:
> >     [s->texture release];
> >     s->texture = texture;
> >
> >     This will make s->texture dangling between the two statements, but
> that
> >     don't matter since s->texture is not an atomic variable that can be
> >     safely observed from another thread anyway.
> >
> >      > +
> >      > +            if (s->pending_frames == 0) {
> >      > +                g_free(old_vram);
> >      > +            }
> >      > +        });
> >      > +}
> >      > +
> >      > +static void create_fb(AppleGFXState *s)
> >      > +{
> >      > +    s->con = graphic_console_init(NULL, 0, &apple_gfx_fb_ops, s);
> >      > +    set_mode(s, 1440, 1080);
> >      > +
> >      > +    s->cursor_show = true;
> >      > +}
> >      > +
> >      > +static size_t apple_gfx_get_default_mmio_range_size(void)
> >      > +{
> >      > +    size_t mmio_range_size;
> >      > +    @autoreleasepool {
> >      > +        PGDeviceDescriptor *desc = [PGDeviceDescriptor new];
> >      > +        mmio_range_size = desc.mmioLength;
> >      > +        [desc release];
> >      > +    }
> >      > +    return mmio_range_size;
> >      > +}
> >      > +
> >      > +void apple_gfx_common_init(Object *obj, AppleGFXState *s, const
> >     char* obj_name)
> >
> >     This function can be merged into apple_gfx_common_realize().
> >
> >
> > Probably. I'll try it.
>

Upon further inspection, we need to call
cocoa_enable_runloop_on_main_thread() during the init phase, not realize().
So we can't get rid of this entirely. Is there any value in moving the
other init code into _realize()?


> >      > +{
> >      > +    Error *local_err = NULL;
> >      > +    int r;
> >      > +    size_t mmio_range_size =
> >     apple_gfx_get_default_mmio_range_size();
> >      > +
> >      > +    trace_apple_gfx_common_init(obj_name, mmio_range_size);
> >      > +    memory_region_init_io(&s->iomem_gfx, obj, &apple_gfx_ops, s,
> >     obj_name,
> >      > +                          mmio_range_size);
> >      > +    s->iomem_gfx.disable_reentrancy_guard = true;
> >
> >     Why do you disable reentrancy guard?
> >
> >
> > Perhaps with the proposed AIO_WAIT_WHILE based I/O scheme, this won't be
> > an issue anymore, but the guard would otherwise keep dropping MMIOs
> > which immediately caused the PV graphics device to stop making progress.
> > The MMIO APIs in the PVG framework are thread- and reentrancy-safe, so
> > we certainly don't need to serialise them on our side.
>
> It's better to understand why such reentrancy happens. Reentrancy itself
> is often a sign of bug.
>
> >
> >      > +
> >      > +    /* TODO: PVG framework supports serialising device state:
> >     integrate it! */
> >      > +    if (apple_gfx_mig_blocker == NULL) {
> >      > +        error_setg(&apple_gfx_mig_blocker,
> >      > +                  "Migration state blocked by apple-gfx display
> >     device");
> >      > +        r = migrate_add_blocker(&apple_gfx_mig_blocker,
> &local_err);
> >      > +        if (r < 0) {
> >      > +            error_report_err(local_err);
> >
> >     Please report the error to the caller of apple_gfx_common_realize()
> >     instead.
> >
> >      > +        }
> >      > +    }
> >       > +}> +
> >      > +static void
> >     apple_gfx_register_task_mapping_handlers(AppleGFXState *s,
> >      > +
> >       PGDeviceDescriptor *desc)
> >      > +{
> >      > +    desc.createTask = ^(uint64_t vmSize, void * _Nullable *
> >     _Nonnull baseAddress) {
> >      > +        AppleGFXTask *task = apple_gfx_new_task(s, vmSize);
> >      > +        *baseAddress = (void*)task->address;
> >
> >     nit: please write as (void *) instead of (void*).
> >
> >      > +        trace_apple_gfx_create_task(vmSize, *baseAddress);
> >      > +        return task;
> >      > +    };
> >      > +
> >
> >      > +{
> >      > +    PGDisplayDescriptor *disp_desc = [PGDisplayDescriptor new];
> >      > +
> >      > + disp_desc.name <http://disp_desc.name> = @"QEMU display";
> >      > +    disp_desc.sizeInMillimeters = NSMakeSize(400., 300.); /* A
> >     20" display */
> >      > +    disp_desc.queue = dispatch_get_main_queue();
> >      > +    disp_desc.newFrameEventHandler = ^(void) {
> >      > +        trace_apple_gfx_new_frame();
> >      > +        dispatch_async(s->render_queue, ^{
> >      > +            /* Drop frames if we get too far ahead. */
> >      > +            if (s->pending_frames >= 2)
> >      > +                return;
> >      > +            ++s->pending_frames;
> >      > +            if (s->pending_frames > 1) {
> >      > +                return;
> >      > +            }
> >      > +            @autoreleasepool {
> >      > +                apple_gfx_render_new_frame(s);
> >      > +            }
> >      > +        });
> >      > +    };
> >      > +    disp_desc.modeChangeHandler = ^(PGDisplayCoord_t
> sizeInPixels,
> >      > +                                    OSType pixelFormat) {
> >      > +        trace_apple_gfx_mode_change(sizeInPixels.x,
> sizeInPixels.y);
> >      > +        set_mode(s, sizeInPixels.x, sizeInPixels.y);
> >      > +    };
> >      > +    disp_desc.cursorGlyphHandler = ^(NSBitmapImageRep *glyph,
> >      > +                                     PGDisplayCoord_t hotSpot) {
> >      > +        uint32_t bpp = glyph.bitsPerPixel;
> >      > +        size_t width = glyph.pixelsWide;
> >      > +        size_t height = glyph.pixelsHigh;
> >      > +        size_t padding_bytes_per_row = glyph.bytesPerRow - width
> >     * 4;
> >      > +        const uint8_t* px_data = glyph.bitmapData;
> >      > +
> >      > +        trace_apple_gfx_cursor_set(bpp, width, height);
> >      > +
> >      > +        if (s->cursor) {
> >      > +            cursor_unref(s->cursor);
> >      > +            s->cursor = NULL;
> >      > +        }
> >      > +
> >      > +        if (bpp == 32) { /* Shouldn't be anything else, but just
> >     to be safe...*/
> >      > +            s->cursor = cursor_alloc(width, height);
> >      > +            s->cursor->hot_x = hotSpot.x;
> >      > +            s->cursor->hot_y = hotSpot.y;
> >      > +
> >      > +            uint32_t *dest_px = s->cursor->data;
> >      > +
> >      > +            for (size_t y = 0; y < height; ++y) {
> >      > +                for (size_t x = 0; x < width; ++x) {
> >      > +                    /* NSBitmapImageRep's red & blue channels
> >     are swapped
> >      > +                     * compared to QEMUCursor's. */
> >      > +                    *dest_px =
> >      > +                        (px_data[0] << 16u) |
> >      > +                        (px_data[1] <<  8u) |
> >      > +                        (px_data[2] <<  0u) |
> >      > +                        (px_data[3] << 24u);
> >      > +                    ++dest_px;
> >      > +                    px_data += 4;
> >      > +                }
> >      > +                px_data += padding_bytes_per_row;
> >      > +            }
> >      > +            dpy_cursor_define(s->con, s->cursor);
> >      > +            update_cursor(s);
> >      > +        }
> >      > +    };
> >      > +    disp_desc.cursorShowHandler = ^(BOOL show) {
> >      > +        trace_apple_gfx_cursor_show(show);
> >      > +        s->cursor_show = show;
> >      > +        update_cursor(s);
> >      > +    };
> >      > +    disp_desc.cursorMoveHandler = ^(void) {
> >      > +        trace_apple_gfx_cursor_move();
> >      > +        update_cursor(s);
> >      > +    };
> >      > +
> >      > +    return disp_desc;
> >      > +}
> >      > +
> >      > +static NSArray<PGDisplayMode*>*
> >     apple_gfx_prepare_display_mode_array(void)
> >      > +{
> >      > +    PGDisplayMode *modes[ARRAY_SIZE(apple_gfx_modes)];
> >      > +    NSArray<PGDisplayMode*>* mode_array = nil;
> >      > +    int i;
> >      > +
> >      > +    for (i = 0; i < ARRAY_SIZE(apple_gfx_modes); i++) {
> >      > +        modes[i] =
> >      > +            [[PGDisplayMode alloc]
> >     initWithSizeInPixels:apple_gfx_modes[i] refreshRateInHz:60.];
> >      > +    }
> >      > +
> >      > +    mode_array = [NSArray arrayWithObjects:modes
> >     count:ARRAY_SIZE(apple_gfx_modes)];
> >      > +
> >      > +    for (i = 0; i < ARRAY_SIZE(apple_gfx_modes); i++) {
> >      > +        [modes[i] release];
> >      > +        modes[i] = nil;
> >      > +    }
> >      > +
> >      > +    return mode_array;
> >      > +}
> >      > +
> >      > +static id<MTLDevice> copy_suitable_metal_device(void)
> >      > +{
> >      > +    id<MTLDevice> dev = nil;
> >      > +    NSArray<id<MTLDevice>> *devs = MTLCopyAllDevices();
> >      > +
> >      > +    /* Prefer a unified memory GPU. Failing that, pick a non-
> >     removable GPU. */
> >      > +    for (size_t i = 0; i < devs.count; ++i) {
> >      > +        if (devs[i].hasUnifiedMemory) {
> >      > +            dev = devs[i];
> >      > +            break;
> >      > +        }
> >      > +        if (!devs[i].removable) {
> >      > +            dev = devs[i];
> >      > +        }
> >      > +    }
> >      > +
> >      > +    if (dev != nil) {
> >      > +        [dev retain];
> >      > +    } else {
> >      > +        dev = MTLCreateSystemDefaultDevice();
> >      > +    }
> >      > +    [devs release];
> >      > +
> >      > +    return dev;
> >      > +}
> >      > +
> >      > +void apple_gfx_common_realize(AppleGFXState *s,
> >     PGDeviceDescriptor *desc)
> >      > +{
> >      > +    PGDisplayDescriptor *disp_desc = nil;
> >      > +
> >      > +    QTAILQ_INIT(&s->tasks);
> >      > +    s->render_queue = dispatch_queue_create("apple-gfx.render",
> >      > +
> DISPATCH_QUEUE_SERIAL);
> >      > +    s->mtl = copy_suitable_metal_device();
> >      > +    s->mtl_queue = [s->mtl newCommandQueue];
> >      > +
> >      > +    desc.device = s->mtl;
> >      > +
> >      > +    apple_gfx_register_task_mapping_handlers(s, desc);
> >      > +
> >      > +    s->pgdev = PGNewDeviceWithDescriptor(desc);
> >      > +
> >      > +    disp_desc = apple_gfx_prepare_display_handlers(s);
> >      > +    s->pgdisp = [s->pgdev newDisplayWithDescriptor:disp_desc
> >      > +                                              port:0
> >     serialNum:1234];
> >      > +    [disp_desc release];
> >      > +    s->pgdisp.modeList = apple_gfx_prepare_display_mode_array();
> >      > +
> >      > +    create_fb(s);
> >      > +}
> >      > diff --git a/hw/display/meson.build b/hw/display/meson.build
> >      > index 7db05eace97..70d855749c0 100644
> >      > --- a/hw/display/meson.build
> >      > +++ b/hw/display/meson.build
> >      > @@ -65,6 +65,8 @@ system_ss.add(when: 'CONFIG_ARTIST', if_true:
> >     files('artist.c'))
> >      >
> >      >   system_ss.add(when: 'CONFIG_ATI_VGA', if_true: [files('ati.c',
> >     'ati_2d.c', 'ati_dbg.c'), pixman])
> >      >
> >      > +system_ss.add(when: 'CONFIG_MAC_PVG',         if_true:
> >     [files('apple-gfx.m'), pvg, metal])
> >      > +system_ss.add(when: 'CONFIG_MAC_PVG_VMAPPLE', if_true:
> >     [files('apple-gfx-vmapple.m'), pvg, metal])
> >      >
> >      >   if config_all_devices.has_key('CONFIG_VIRTIO_GPU')
> >      >     virtio_gpu_ss = ss.source_set()
> >      > diff --git a/hw/display/trace-events b/hw/display/trace-events
> >      > index 781f8a33203..1809a358e36 100644
> >      > --- a/hw/display/trace-events
> >      > +++ b/hw/display/trace-events
> >      > @@ -191,3 +191,29 @@ dm163_bits_ppi(unsigned dest_width)
> >     "dest_width : %u"
> >      >   dm163_leds(int led, uint32_t value) "led %d: 0x%x"
> >      >   dm163_channels(int channel, uint8_t value) "channel %d: 0x%x"
> >      >   dm163_refresh_rate(uint32_t rr) "refresh rate %d"
> >      > +
> >      > +# apple-gfx.m
> >      > +apple_gfx_read(uint64_t offset, uint64_t res)
> >     "offset=0x%"PRIx64" res=0x%"PRIx64
> >      > +apple_gfx_write(uint64_t offset, uint64_t val)
> >     "offset=0x%"PRIx64" val=0x%"PRIx64
> >      > +apple_gfx_create_task(uint32_t vm_size, void *va) "vm_size=0x%x
> >     base_addr=%p"
> >      > +apple_gfx_destroy_task(void *task) "task=%p"
> >      > +apple_gfx_map_memory(void *task, uint32_t range_count, uint64_t
> >     virtual_offset, uint32_t read_only) "task=%p range_count=0x%x
> >     virtual_offset=0x%"PRIx64" read_only=%d"
> >      > +apple_gfx_map_memory_range(uint32_t i, uint64_t phys_addr,
> >     uint64_t phys_len) "[%d] phys_addr=0x%"PRIx64" phys_len=0x%"PRIx64
> >      > +apple_gfx_remap(uint64_t retval, uint64_t source, uint64_t
> >     target) "retval=%"PRId64" source=0x%"PRIx64" target=0x%"PRIx64
> >      > +apple_gfx_unmap_memory(void *task, uint64_t virtual_offset,
> >     uint64_t length) "task=%p virtual_offset=0x%"PRIx64"
> length=0x%"PRIx64
> >      > +apple_gfx_read_memory(uint64_t phys_address, uint64_t length,
> >     void *dst) "phys_addr=0x%"PRIx64" length=0x%"PRIx64" dest=%p"
> >      > +apple_gfx_raise_irq(uint32_t vector) "vector=0x%x"
> >      > +apple_gfx_new_frame(void) ""
> >      > +apple_gfx_mode_change(uint64_t x, uint64_t y) "x=%"PRId64"
> >     y=%"PRId64
> >      > +apple_gfx_cursor_set(uint32_t bpp, uint64_t width, uint64_t
> >     height) "bpp=%d width=%"PRId64" height=0x%"PRId64
> >      > +apple_gfx_cursor_show(uint32_t show) "show=%d"
> >      > +apple_gfx_cursor_move(void) ""
> >      > +apple_gfx_common_init(const char *device_name, size_t mmio_size)
> >     "device: %s; MMIO size: %zu bytes"
> >      > +
> >      > +# apple-gfx-vmapple.m
> >      > +apple_iosfc_read(uint64_t offset, uint64_t res)
> >     "offset=0x%"PRIx64" res=0x%"PRIx64
> >      > +apple_iosfc_write(uint64_t offset, uint64_t val)
> >     "offset=0x%"PRIx64" val=0x%"PRIx64
> >      > +apple_iosfc_map_memory(uint64_t phys, uint64_t len, uint32_t ro,
> >     void *va, void *e, void *f) "phys=0x%"PRIx64" len=0x%"PRIx64" ro=%d
> >     va=%p e=%p f=%p"
> >      > +apple_iosfc_unmap_memory(void *a, void *b, void *c, void *d,
> >     void *e, void *f) "a=%p b=%p c=%p d=%p e=%p f=%p"
> >      > +apple_iosfc_raise_irq(uint32_t vector) "vector=0x%x"
> >      > +
> >      > diff --git a/meson.build b/meson.build
> >      > index 10464466ff3..f09df3f09d5 100644
> >      > --- a/meson.build
> >      > +++ b/meson.build
> >      > @@ -741,6 +741,8 @@ socket = []
> >      >   version_res = []
> >      >   coref = []
> >      >   iokit = []
> >      > +pvg = []
> >      > +metal = []
> >      >   emulator_link_args = []
> >      >   midl = not_found
> >      >   widl = not_found
> >      > @@ -762,6 +764,8 @@ elif host_os == 'darwin'
> >      >     coref = dependency('appleframeworks', modules:
> 'CoreFoundation')
> >      >     iokit = dependency('appleframeworks', modules: 'IOKit',
> >     required: false)
> >      >     host_dsosuf = '.dylib'
> >      > +  pvg = dependency('appleframeworks', modules:
> >     'ParavirtualizedGraphics')
> >      > +  metal = dependency('appleframeworks', modules: 'Metal')
> >      >   elif host_os == 'sunos'
> >      >     socket = [cc.find_library('socket'),
> >      >               cc.find_library('nsl'),
> >
>
>

[-- Attachment #2: Type: text/html, Size: 53722 bytes --]

Re: [PATCH v3 01/14] hw/display/apple-gfx: Introduce ParavirtualizedGraphics.Framework support

[Akihiko Odaki]

On 2024/10/06 19:39, Phil Dennis-Jordan wrote:
> 
> 
> On Thu, 3 Oct 2024 at 09:09, Akihiko Odaki <akihiko.odaki@daynix.com 
> <mailto:akihiko.odaki@daynix.com>> wrote:
> 
>     On 2024/10/02 22:33, Phil Dennis-Jordan wrote:
>      >
>      >
>      >      > +#include "apple-gfx.h"
>      >      > +#include "monitor/monitor.h"
>      >      > +#include "hw/sysbus.h"
>      >      > +#include "hw/irq.h"
>      >      > +#include "trace.h"
>      >      > +#import <ParavirtualizedGraphics/ParavirtualizedGraphics.h>
>      >      > +
>      >      > +_Static_assert(__aarch64__, "");
>      >
>      >     I don't think this assertion is worthwhile. This assertion
>     will trigger
>      >     if you accidentally remove depends on AARCH64 from Kconfig,
>     but I don't
>      >     think such code change happens by accident, and there is no
>     reason to
>      >     believe that this assertion won't be removed in such a case.
>      >
>      >
>      > As far as I'm aware the Kconfig AARCH64 dependency is for the /
>     target/
>      > architecture, not the /host/ architecture? The static assert
>     checks for
>      > the latter. The PGIOSurfaceHostDeviceDescriptor type isn't
>     available at
>      > all on non-aarch64 macOS hosts. I've not had any luck with using
>     this
>      > variant of the device on x86-64 hosts simply by disabling any
>     surface
>      > mapper code.
>      >
>      > Incidentally, if you know of a way to depend on a specific /host/
>      > architecture in the Kconfig, that would be even better. I
>     couldn't spot
>      > a way of doing that though.
> 
>     I got your intention now. The correct way to do that is to check for
>     cpu
>     == 'aarch64'. Having assertion will break qemu-system-aarch64 on
>     Intel Macs.
> 
> 
> OK, looks Iike that needs to be done at the meson.build level not 
> Kconfig, but this seems to work:
> 
> if cpu == 'aarch64'
>    system_ss.add(when: 'CONFIG_MAC_PVG_MMIO',  if_true: [files('apple- 
> gfx-mmio.m'), pvg, metal])
> endif
> 
>      >
>      >     It is dangerous to unlock BQL at an arbitrary place. Instead of
>      >     unlocking, I suggest:
>      >     - running [s->pgiosfc mmioReadAtOffset:offset] on another thread
>      >     - using a bottom half to request operations that require BQL
>     from the
>      >     thread running [s->pgiosfc mmioReadAtOffset:offset]
>      >     - calling AIO_WAIT_WHILE() to process the bottom half and to
>     wait for
>      >     the completion of [s->pgiosfc mmioReadAtOffset:offset]
>      >
>      >
>      > OK, I think I see what you mean, I'll try to rework things around
>     that
>      > pattern. Any preference on how I kick off the job on the other
>     thread?
>      > As we necessarily need to use libdispatch in a bunch of places in
>     this
>      > code anyway, I guess dispatch_async() would probably be the simplest?
> 
>     Perhaps so. The QEMU way is to use a bottom half with AioContext, but
>     you can't simultaneously run a dispatch queue and AioContext in one
>     thread so you have to use the dispatch queue if you need one.
> 
>      >
>      >      > +    res = [s->pgiosfc mmioReadAtOffset:offset];
>      >      > +    bql_lock();
>      >      > +
>      >      > +    trace_apple_iosfc_read(offset, res);
>      >      > +
>      >      > +    return res;
>      >      > +}
>      >      > +
>      >      > +static void apple_iosfc_write(
>      >      > +    void *opaque, hwaddr offset, uint64_t val, unsigned size)
>      >      > +{
>      >      > +    AppleGFXVmappleState *s = opaque;
>      >      > +
>      >      > +    trace_apple_iosfc_write(offset, val);
>      >      > +
>      >      > +    [s->pgiosfc mmioWriteAtOffset:offset value:val];
>      >      > +}
>      >      > +
>      >      > +static const MemoryRegionOps apple_iosfc_ops = {
>      >      > +    .read = apple_iosfc_read,
>      >      > +    .write = apple_iosfc_write,
>      >      > +    .endianness = DEVICE_LITTLE_ENDIAN,
>      >      > +    .valid = {
>      >      > +        .min_access_size = 4,
>      >      > +        .max_access_size = 8,
>      >      > +    },
>      >      > +    .impl = {
>      >      > +        .min_access_size = 4,
>      >      > +        .max_access_size = 8,
>      >      > +    },
>      >      > +};
>      >      > +
>      >      > +static PGIOSurfaceHostDevice
>      >     *apple_gfx_prepare_iosurface_host_device(
>      >      > +    AppleGFXVmappleState *s)
>      >      > +{
>      >      > +    PGIOSurfaceHostDeviceDescriptor *iosfc_desc =
>      >      > +        [PGIOSurfaceHostDeviceDescriptor new];
>      >      > +    PGIOSurfaceHostDevice *iosfc_host_dev = nil;
>      >      > +
>      >      > +    iosfc_desc.mapMemory =
>      >      > +        ^(uint64_t phys, uint64_t len, bool ro, void
>     **va, void
>      >     *e, void *f) {
>      >      > +            trace_apple_iosfc_map_memory(phys, len, ro,
>     va, e, f);
>      >      > +            MemoryRegion *tmp_mr;
>      >      > +            *va = gpa2hva(&tmp_mr, phys, len, NULL);
>      >
>      >     Use: dma_memory_map()
>      >
>      >
>      > That doesn't seem to be a precisely equivalent operation. It also
>     says
>      > in its headerdoc,
>      >
>      >     Use only for reads OR writes - not for read-modify-write
>     operations.
>      >
>      > which I don't think we can guarantee here at all.
>      >
>      > I guess I can call it twice, once for writing and once for
>     reading, but
>      > given that the dma_memory_unmap operation marks the area dirty,
>     I'm not
>      > it's intended for what I understand the use case here to be: As
>     far as I
>      > can tell, the PV graphics device uses (some) of this memory to
>     exchange
>      > data in a cache-coherent way between host and guest, e.g. as a
>     lock-free
>      > ring buffer, using atomic operations as necessary. (This works
>     because
>      > it's a PV device: it "knows" the other end just another CPU core (or
>      > even the same one) executing in a different hypervisor context.)
>     This
>      > doesn't really match "traditional" DMA patterns where there's
>     either a
>      > read or a write happening.
> 
>     I think the story is a bit different for this VMApple variant. Probably
>     the CPU and GPU in Apple Silicon is cache-coherent so you can map
>     normal
>     memory for GPU without any kind of cache maintenance.
> 
>     Cache conherency of CPU and GPU in Apple Silicon is implied with
>     Apple's
>     documentation; it says you don't need to synchronize resources for
>     MTLStorageModeShared, which is the default for Apple Silicon.
>     https://developer.apple.com/documentation/metal/
>     resource_fundamentals/synchronizing_a_managed_resource_in_macos
>     <https://developer.apple.com/documentation/metal/
>     resource_fundamentals/synchronizing_a_managed_resource_in_macos>
> 
>     The name "IOSurface" also implies it is used not only for e.g., ring
>     buffer but also for real data.
> 
> Note that the PGTask map/unmap callbacks appear to have equivalent 
> semantics, so it's not just the surface mapping.
> 
>      >
>      > Hunting around some more for alternative APIs, there's also
>      > memory_region_get_ram_ptr(), but I'm not sure its restrictions apply
>      > here either.
> 
>     I think you can call memory_access_is_direct() to check if the
>     requirement is satisfied.
> 
>     It will still break dirty page tracking implemented by
>     dma_memory_unmap() and others, but it's broken for hvf, which does not
>     implement dirty page tracking either.
> 
> 
> 
> 
>      >
>      >      > +            return (bool)true;
>      >
>      >     Why cast?
>      >
>      >
>      > Good question. Not originally my code, so I've fixed all the
>     instances I
>      > could find now.
> 
> 
> OK, it turns out the reason for this is that C treats 'true' as an int, 
> which then becomes the block's inferred return type - and the callbacks 
> are expecting bool-returning blocks.
> 
> I've fixed it by explicitly specifying the block return type and 
> removing the cast in the return statement:
> 
> iosfc_desc.unmapMemory =
>         ^bool(…) {
>             …
>             return true;
>         };
> 
>      >      > +
>      >      > +    iosfc_desc.unmapMemory =
>      >      > +        ^(void *a, void *b, void *c, void *d, void *e,
>     void *f) {
>      >      > +            trace_apple_iosfc_unmap_memory(a, b, c, d, e, f);
>      >      > +            return (bool)true;
>      >      > +        };
>      >      > +
>      >      > +    iosfc_desc.raiseInterrupt = ^(uint32_t vector) {
>      >      > +        trace_apple_iosfc_raise_irq(vector);
>      >      > +        bool locked = bql_locked();
>      >      > +        if (!locked) {
>      >      > +            bql_lock();
>      >      > +        }
>      >       > +        qemu_irq_pulse(s->irq_iosfc);> +        if (!
>     locked) {
>      >      > +            bql_unlock();
>      >      > +        }
>      >      > +        return (bool)true;
>      >      > +    };
>      >      > +
>      >      > +    iosfc_host_dev =
>      >      > +        [[PGIOSurfaceHostDevice alloc]
>      >     initWithDescriptor:iosfc_desc];
>      >      > +    [iosfc_desc release];
>      >      > +    return iosfc_host_dev;
>      >      > +}
>      >      > +
>      >      > +static void apple_gfx_vmapple_realize(DeviceState *dev, Error
>      >     **errp)
>      >      > +{
>      >      > +    @autoreleasepool {
>      >
>      >     This autoreleasepool is not used.
>      >
>      >
>      > It is definitely used inside the apple_gfx_common_realize() call.
>     It's
>      > also impossible to say whether [PGDeviceDescriptor new] uses
>     autorelease
>      > semantics internally, so it seemed safer to wrap the whole thing
>     in an
>      > outer pool.
> 
>     Theoretically, It should be safe to assume the callee creates
>     autoreleasepool by themselves as needed in general. We have bunch of
>     code to call Objective-C APIs without creating autoreleasepool in the
>     caller. Practically, [PGDeviceDescriptor new] is likely to be
>     implemented with ARC, which wraps methods in autoreleasepool as
>     necessary.
> 
> 
> As far as I'm aware, ARC does NOT automatically insert autorelease pool 
> blocks. The reason you rarely need to create autoreleasepool blocks in 
> "plain" Objective-C programming is that Cocoa/CFRunloop/libdispatch 
> event handlers run each event in an autoreleasepool. So you don't need 
> to create them explicitly when using dispatch_async and similar, or when 
> running code on the main thread (which runs inside NSApplicationMain/ 
> CFRunloopRun/dispatch_main).

My statement regarding ARC was wrong; It seems ARC just retains a value 
annoted as autoreleased.

> 
> As far as I'm aware, if you don't explicitly define autoreleasepools in 
> raw threads created with the pthreads API, any autoreleased objects will 
> leak. At least I've not found any specification/documentation 
> contradicting this. And most code in Qemu runs on such raw threads, so 
> we need to play it safe with regard to autorelease semantics.

I decided to dig deeper and found this documentation:
https://developer.apple.com/library/archive/documentation/Cocoa/Conceptual/MemoryMgmt/Articles/mmAutoreleasePools.html

It says:
 > Cocoa always expects code to be executed within an autorelease pool
 > block, otherwise autoreleased objects do not get released and your
 > application leaks memory.

So yes, we must wrap everything in @autoreleasepool at least for Cocoa. 
It is probably a good practice to wrap all Objective-C code in 
@autoreleasepool.

> 
> Whether the existing Qemu Objective-C code is safe in this regard I 
> don't know for certain, but I've certainly paid attention to this aspect 
> when modifying ui/cocoa.m in the past, and indeed most of that code runs 
> on the main thread. Note also how I wrap the apple_gfx_render_new_frame 
> call in a pool when it can't be guaranteed it's running on a dispatch 
> queue because the command buffer inside that uses autorelease semantics.

It is more about event loop rather than thread. Resources allocated 
before [NSApp run] will leak even if they are autoreleased in the main 
thread. apple_gfx_common_realize() is one of such functions that run in 
the main thread before [NSApp run]. In ui/cocoa, cocoa_display_init() 
runs before [NSApp run]. Fortunately we already have NSAutoreleasePool 
for this function.

> 
>     Functions that uses a method that returns autorelease resources should
>     be wrapped with autoreleasepool instead of assuming the caller creates
>     autoreleasepool for them.
> 
> 
> I'm treating apple_gfx_common_realize as an internal API, and I don't 
> think expecting its callers to wrap it in an autoreleasepool block is 
> unreasonable. I can certainly explicitly document this in a comment.

We don't have a comment for cocoa_display_init() and it's more about 
generic macOS programming so it's not necessary.

 > >      >      > diff --git a/hw/display/apple-gfx.m 
b/hw/display/apple-gfx.m
>      >      > new file mode 100644
>      >      > index 00000000000..837300f9cd4
>      >      > --- /dev/null
>      >      > +++ b/hw/display/apple-gfx.m
>      >      > @@ -0,0 +1,536 @@
>      >      > +/*
>      >      > + * QEMU Apple ParavirtualizedGraphics.framework device
>      >      > + *
>      >      > + * Copyright © 2023 Amazon.com, Inc. or its affiliates. All
>      >     Rights Reserved.
>      >      > + *
>      >      > + * This work is licensed under the terms of the GNU GPL,
>     version
>      >     2 or later.
>      >      > + * See the COPYING file in the top-level directory.
>      >      > + *
>      >      > + * ParavirtualizedGraphics.framework is a set of
>     libraries that
>      >     macOS provides
>      >      > + * which implements 3d graphics passthrough to the host
>     as well as a
>      >      > + * proprietary guest communication channel to drive it. This
>      >     device model
>      >      > + * implements support to drive that library from within QEMU.
>      >      > + */
>      >      > +
>      >      > +#include "apple-gfx.h"
>      >      > +#include "trace.h"
>      >      > +#include "qemu/main-loop.h"
>      >      > +#include "ui/console.h"
>      >      > +#include "monitor/monitor.h"
>      >      > +#include "qapi/error.h"
>      >      > +#include "migration/blocker.h"
>      >      > +#include <mach/mach_vm.h>
>      >      > +#import <ParavirtualizedGraphics/ParavirtualizedGraphics.h>
>      >      > +
>      >      > +static const PGDisplayCoord_t apple_gfx_modes[] = {
>      >      > +    { .x = 1440, .y = 1080 },
>      >      > +    { .x = 1280, .y = 1024 },
>      >      > +};
>      >      > +
>      >      > +typedef struct PGTask_s { // Name matches forward
>     declaration in
>      >     PG header
>      >
>      >     Let's name it AppleGFXTask. It is a common practice to have
>     the same
>      >     tag
>      >     name and typedef in QEMU.
>      >
>      >
>      > This is defining a forward-declared type from framework headers
>     which is
>      > opaque from the framework's point of view. We do not get to
>     choose its
>      > struct name. The alternative is having casts wherever these
>     objects are
>      > being passed between our code and the framework. (See the
>     original v1/v2
>      > vmapple patch series for how messy this is.)
> 
>     I got the idea. Let's not avoid the typedef then to clarify the naming
>     is not under our control.
> 
> 
> I'm not sure what you mean by this double negative. Are you saying, 
> don't add our own typedef for struct PGTask_s at all, just use the 
> framework-supplied PGTask_t where appropriate?

Yes.

> 
> 
>      >
>      >      > +static void
>     apple_gfx_render_frame_completed(AppleGFXState *s,
>      >     void *vram,
>      >      > +                                           
>       id<MTLTexture> texture)
>      >      > +{
>      >      > +    --s->pending_frames;
>      >      > +    assert(s->pending_frames >= 0);
>      >      > +
>      >      > +    if (vram != s->vram) {
>      >      > +        /* Display mode has changed, drop this old frame. */
>      >      > +        assert(texture != s->texture);
>      >      > +        g_free(vram);
>      >
>      >     This management of buffers looks a bit convoluted. I suggest
>      >     remembering
>      >     the width and height instead of pointers and comparing them.
>     This way
>      >     you can free resources in set_mode().
>      >
>      >
>      > Yeah, I suppose that works, I can change that around.
>      >
>      >      > +    } else {
>      >      > +        copy_mtl_texture_to_surface_mem(texture, vram);
>      >
>      >     Writing vram outside BQL may result in tearing.
>      >
>      >
>      > As far as I can tell(*), QXL does the same. I couldn't find any
>     examples
>      > of double-buffering in any of the existing display devices, which
>     would
>      > be the only way to do async updates efficiently and without
>     tearing. In
>      > any case, this solution is still vastly better than a regular VGA
>      > device, which suffers from very visible tearing with macOS on the
>     guest
>      > side anyway. And in an ideal world, we'd pass through the rendered
>      > texture directly to the Cocoa UI code rather than copying out
>     only for
>      > the CPU to draw it back into a window surface which is then
>     passed to
>      > the GPU for host side rendering. But I felt this patch is already
>     very,
>      > very large, and if anyone cares, we can fix imperfections in
>     subsequent
>      > updates.
>      >
>      > (*)The rendering code in that device is also fairly complex, so I
>     may be
>      > misreading it.
> 
>     QXL always modifies the surface with BQL. The surface is modified with
>     qxl_blit(), which is called by qxl_render_update_area_unlocked().
>     qxl_render_update_area_unlocked() is called by either of
>     qxl_render_update() and qxl_render_update_area_bh(). Both of them are
>     called with BQL. The name includes "unlocked", but it means it is
>     called
>     without holding QXL-internal lock.
> 
>     Most devices works entirely with BQL so they don't perform double
>     buffering. apple-gfx can do the same.
> 
> 
> I think we can safely move apple-gfx's framebuffer state management back 
> inside the BQL, yes. I just figured that copying dozens of megabytes of 
> framebuffer data on every frame while holding the lock was not going to 
> help BQL contention. Especially as PVG does not have a concept of dirty 
> areas, so we must copy the whole framebuffer every time. (Unless we were 
> to implement dirty area detection ourselves.)
> 
> Unfortunately, implementing double-buffering would require a major 
> rework of Qemu's whole surface management, console code, and probably 
> most of the UI implementations. I'm guessing the OpenGL fast-path 
> sidesteps all of this, so replicating that with Metal would probably be 
> the easier way forward. (Although doing all this graphics stuff inside 
> the BQL generally seems like a major architectural flaw; I suppose most 
> enterprise use of Qemu does not involve the framebuffer, so it's not 
> shown up in BQL contention profiling there. It certainly does in desktop 
> use, although at least on macOS hosts there are far worse culprits in 
> that regard.)

We don't need double-buffering; instead we can have one thread that 
renders the UI and also
[PGDisplay -encodeCurrentFrameToCommandBuffer:texture:region:]. 
Rendering the UI in machine event loop in one thread is a bad idea in 
the first place so it will make sense to extract all UI work into a 
separate thread. This still requires a lot of work because the UI code 
assumes BQL everywhere.

In my understanding BQL is less problematic for KVM users because QEMU 
can (sometimes) avoid locking BQL in the vCPU threads.

> 
>      >
>      >      > +        if (s->gfx_update_requested) {
>      >      > +            s->gfx_update_requested = false;
>      >      > +            dpy_gfx_update_full(s->con);
>      >      > +            graphic_hw_update_done(s->con);
>      >       > +            s->new_frame_ready = false;
>      >
>      >     This assignment is unnecessary as s->new_frame_ready is
>     always false
>      >     when s->gfx_update_requested. If you want to make sure
>      >     s->gfx_update_requested and s->new_frame_ready are mutually
>     exclusive,
>      >     use one enum value instead of having two bools.
>      >
>      >
>      > I'll need to refresh my memory and get back to you on this one, it's
>      > been so many months since I actively worked on this code.
>      >
>      >      > +        } else {
>      >      > +            s->new_frame_ready = true;
>      >      > +        }
>      >      > +    }
>      >      > +    if (s->pending_frames > 0) {
>      >      > +        apple_gfx_render_new_frame(s);
>      >      > +    }
>      >      > +}
>      >      > +
>      >      > +static void apple_gfx_fb_update_display(void *opaque)
>      >      > +{
>      >      > +    AppleGFXState *s = opaque;
>      >      > +
>      >      > +    dispatch_async(s->render_queue, ^{
>      >      > +        if (s->pending_frames > 0) {
>      >
>      >     It should check for s->new_frame_ready as
>      >     apple_gfx_render_frame_completed() doesn't check if
>      >     s->pending_frames > 0 before calling
>     graphic_hw_update_done(), which is
>      >     inconsistent.
>      >
>      >
>      > pending_frames is about guest-side frames that are queued to be
>     rendered
>      > by the host GPU.
>      > new_frame_ready being true indicates that the contents of the Qemu
>      > console surface has been updated with new frame data since the last
>      > gfx_update.
>      > gfx_update_requested indicates that gfx_update is currently
>     awaiting an
>      > async completion (graphic_hw_update_done) but the surface has not
>      > received a new frame content, but the GPU is stily busy drawing one.
>      >
>      > apple_gfx_render_frame_completed is scheduled exactly once per
>     pending
>      > frame, so pending_frames > 0 is an invariant there. (Hence the
>     assert.)> > I don't think there is any inconsistency here, but I'll
>     double check.
>      > It's possible that there's an easier way to express the state
>     machine,
>      > and I'll take a look at that.
> 
>     I meant that apple_gfx_render_frame_completed() does not check if the
>     frame is the last one currently pending. apple_gfx_fb_update_display()
>     ignores a new ready frame when there is a more pending frame, but
>     apple_gfx_render_frame_completed() unconditionally fires
>     graphic_hw_update_done() even if there is a more pending frame. And I
>     think apple_gfx_render_frame_completed() is right and
>     apple_gfx_fb_update_display() is wrong in such a situation.
> 
> 
> OK, got it. And yes, I agree.
> 
>      >
>      >     Checking if s->pending_frames > 0 both in
>     apple_gfx_fb_update_display()
>      >     and apple_gfx_render_frame_completed() is also problematic as
>     that can
>      >     defer graphic_hw_update_done() indefinitely if we are getting new
>      >     frames
>      >     too fast.
>      >
>      >
>      > I see what you mean about this part. I'll have to test it, but I
>     guess
>      > we should reverse the priority, like this:
>      >
>      >          if (s->new_frame_ready) {
>      >              dpy_gfx_update_full(s->con);
>      >              s->new_frame_ready = false;
>      >              graphic_hw_update_done(s->con);
>      >          } else if (s->pending_frames > 0) {
>      >              s->gfx_update_requested = true;
>      >          } else {
>      >              graphic_hw_update_done(s->con);
>      >          }
>      >
>      > 1. If we already have a frame, ready to be displayed, return it
>     immediately.
>      > 2. If the guest has reported that it's completed a frame and the
>     GPU is
>      > currently busy rendering it, defer graphic_hw_update_done until
>     it's done.
>      > 3. If the guest reports no changes to its display, indicate this
>     back to
>      > Qemu as a no-op display update graphic_hw_update_done() with no
>      > dpy_gfx_update* call.
> 
>     Yes, that looks correct.
> 
>      >
>      >      > +            s->gfx_update_requested = true;
>      >      > +        } else {
>      >      > +            if (s->new_frame_ready) {
>      >      > +                dpy_gfx_update_full(s->con);
>      >      > +                s->new_frame_ready = false;
>      >      > +            }
>      >      > +            graphic_hw_update_done(s->con);
>      >       > +        }> +    });
>      >      > +}
>      >      > +
>      >      > +static const GraphicHwOps apple_gfx_fb_ops = {
>      >      > +    .gfx_update = apple_gfx_fb_update_display,
>      >      > +    .gfx_update_async = true,
>      >      > +};
>      >      > +
>      >      > +static void update_cursor(AppleGFXState *s)
>      >      > +{
>      >      > +    dpy_mouse_set(s->con, s->pgdisp.cursorPosition.x,
>      >      > +                  s->pgdisp.cursorPosition.y, s-
>      >cursor_show);
>      >      > +}
>      >      > +
>      >      > +static void set_mode(AppleGFXState *s, uint32_t width,
>     uint32_t
>      >     height)
>      >      > +{
>      >      > +    void *vram = NULL;
>      >      > +    DisplaySurface *surface;
>      >      > +    MTLTextureDescriptor *textureDescriptor;
>      >      > +    id<MTLTexture> texture = nil;
>      >      > +    __block bool no_change = false;
>      >      > +
>      >      > +    dispatch_sync(s->render_queue,
>      >
>      >     Calling dispatch_sync() while holding BQL may result in deadlock.
>      >
>      > Only if any code executed on the same dispatch queue acquires the
>     lock
>      > either directly or transitively. I believe I have ensure this is not
>      > done on the reqnder_queue, have you found anywhere this is the case?
> 
>     The documentation is not clear what threads a dispatch queue runs
>     on. We
>     can have a deadlock if they lock the BQL.
> 
> 
> dispatch_sync is a synchronisation primitive (it waits for and asserts 
> exclusive access to the given queue), it doesn't actually do any thread 
> scheduling. Work scheduled asynchronously to non-main dispatch queues 
> will otherwise run on libdispatch pool threads. Running blocks on 
> dispatch queues will not preempt and schedule it on other threads which 
> may or may not be holding some locks.

What if all pool threads are waiting for BQL?

> 
> So the only way this code will deadlock is if any code scheduled to 
> render_queue directly or transitively acquires the BQL. None of it does, 
> although updating the console while holding the BQL rather complicates this.
> 
>      >
>      >      > +        ^{
>      >      > +            if (s->surface &&
>      >      > +                width == surface_width(s->surface) &&
>      >      > +                height == surface_height(s->surface)) {
>      >      > +                no_change = true;
>      >      > +            }
>      >      > +        });
>      >      > +
>      >      > +    if (no_change) {
>      >      > +        return;
>      >      > +    }
>      >      > +
>      >      > +    vram = g_malloc0(width * height * 4);
>      >      > +    surface = qemu_create_displaysurface_from(width, height,
>      >     PIXMAN_LE_a8r8g8b8,
>      >      > +                                              width * 4,
>     vram);
>      >      > +
>      >      > +    @autoreleasepool {
>      >      > +        textureDescriptor =
>      >      > +            [MTLTextureDescriptor
>      >      > +
>      >     texture2DDescriptorWithPixelFormat:MTLPixelFormatBGRA8Unorm
>      >      > +                                             width:width
>      >      > +                                            height:height
>      >      > +                                         mipmapped:NO];
>      >      > +        textureDescriptor.usage = s-
>      >pgdisp.minimumTextureUsage;
>      >      > +        texture = [s->mtl
>      >     newTextureWithDescriptor:textureDescriptor];
>      >      > +    }
>      >      > +
>      >      > +    s->using_managed_texture_storage =
>      >      > +        (texture.storageMode == MTLStorageModeManaged);
>      >      > +
>      >      > +    dispatch_sync(s->render_queue,
>      >      > +        ^{
>      >      > +            id<MTLTexture> old_texture = nil;
>      >      > +            void *old_vram = s->vram;
>      >      > +            s->vram = vram;
>      >      > +            s->surface = surface;
>      >      > +
>      >      > +            dpy_gfx_replace_surface(s->con, surface);
>      >      > +
>      >      > +            old_texture = s->texture;
>      >      > +            s->texture = texture;
>      >      > +            [old_texture release];
>      >
>      >     You can just do:
>      >     [s->texture release];
>      >     s->texture = texture;
>      >
>      >     This will make s->texture dangling between the two
>     statements, but that
>      >     don't matter since s->texture is not an atomic variable that
>     can be
>      >     safely observed from another thread anyway.
>      >
>      >      > +
>      >      > +            if (s->pending_frames == 0) {
>      >      > +                g_free(old_vram);
>      >      > +            }
>      >      > +        });
>      >      > +}
>      >      > +
>      >      > +static void create_fb(AppleGFXState *s)
>      >      > +{
>      >      > +    s->con = graphic_console_init(NULL, 0,
>     &apple_gfx_fb_ops, s);
>      >      > +    set_mode(s, 1440, 1080);
>      >      > +
>      >      > +    s->cursor_show = true;
>      >      > +}
>      >      > +
>      >      > +static size_t apple_gfx_get_default_mmio_range_size(void)
>      >      > +{
>      >      > +    size_t mmio_range_size;
>      >      > +    @autoreleasepool {
>      >      > +        PGDeviceDescriptor *desc = [PGDeviceDescriptor new];
>      >      > +        mmio_range_size = desc.mmioLength;
>      >      > +        [desc release];
>      >      > +    }
>      >      > +    return mmio_range_size;
>      >      > +}
>      >      > +
>      >      > +void apple_gfx_common_init(Object *obj, AppleGFXState *s,
>     const
>      >     char* obj_name)
>      >
>      >     This function can be merged into apple_gfx_common_realize().
>      >
>      >
>      > Probably. I'll try it.
> 
> 
> Upon further inspection, we need to call 
> cocoa_enable_runloop_on_main_thread() during the init phase, not 
> realize(). So we can't get rid of this entirely. Is there any value in 
> moving the other init code into _realize()?

Calling cocoa_enable_runloop_on_main_thread() should be avoided even in 
apple_gfx_common_init(). QEMU can plug a device at runtime instead of 
initialization time, and in such a case, apple_gfx_common_init() will 
run after calling qemu_main.

I had a closer look and found it has a memory_region_init_io() call, 
which should remain in apple_gfx_common_init(). This leads to the same 
conclusion that we cannot remove this function so let's only move 
migrate_add_blocker() to apple_gfx_common_realize() to report its error.

>  
>      >      > +{
>      >      > +    Error *local_err = NULL;
>      >      > +    int r;
>      >      > +    size_t mmio_range_size =
>      >     apple_gfx_get_default_mmio_range_size();
>      >      > +
>      >      > +    trace_apple_gfx_common_init(obj_name, mmio_range_size);
>      >      > +    memory_region_init_io(&s->iomem_gfx, obj,
>     &apple_gfx_ops, s,
>      >     obj_name,
>      >      > +                          mmio_range_size);
>      >      > +    s->iomem_gfx.disable_reentrancy_guard = true;
>      >
>      >     Why do you disable reentrancy guard?
>      >
>      >
>      > Perhaps with the proposed AIO_WAIT_WHILE based I/O scheme, this
>     won't be
>      > an issue anymore, but the guard would otherwise keep dropping MMIOs
>      > which immediately caused the PV graphics device to stop making
>     progress.
>      > The MMIO APIs in the PVG framework are thread- and reentrancy-
>     safe, so
>      > we certainly don't need to serialise them on our side.
> 
>     It's better to understand why such reentrancy happens. Reentrancy
>     itself
>     is often a sign of bug.
> 
>      >
>      >      > +
>      >      > +    /* TODO: PVG framework supports serialising device state:
>      >     integrate it! */
>      >      > +    if (apple_gfx_mig_blocker == NULL) {
>      >      > +        error_setg(&apple_gfx_mig_blocker,
>      >      > +                  "Migration state blocked by apple-gfx
>     display
>      >     device");
>      >      > +        r = migrate_add_blocker(&apple_gfx_mig_blocker,
>     &local_err);
>      >      > +        if (r < 0) {
>      >      > +            error_report_err(local_err);
>      >
>      >     Please report the error to the caller of
>     apple_gfx_common_realize()
>      >     instead.
>      >
>      >      > +        }
>      >      > +    }
>      >       > +}> +
>      >      > +static void
>      >     apple_gfx_register_task_mapping_handlers(AppleGFXState *s,
>      >      > +
>      >       PGDeviceDescriptor *desc)
>      >      > +{
>      >      > +    desc.createTask = ^(uint64_t vmSize, void * _Nullable *
>      >     _Nonnull baseAddress) {
>      >      > +        AppleGFXTask *task = apple_gfx_new_task(s, vmSize);
>      >      > +        *baseAddress = (void*)task->address;
>      >
>      >     nit: please write as (void *) instead of (void*).
>      >
>      >      > +        trace_apple_gfx_create_task(vmSize, *baseAddress);
>      >      > +        return task;
>      >      > +    };
>      >      > +
>      >
>      >      > +{
>      >      > +    PGDisplayDescriptor *disp_desc = [PGDisplayDescriptor
>     new];
>      >      > +
>      >      > + disp_desc.name <http://disp_desc.name> <http://
>     disp_desc.name <http://disp_desc.name>> = @"QEMU display";
>      >      > +    disp_desc.sizeInMillimeters = NSMakeSize(400.,
>     300.); /* A
>      >     20" display */
>      >      > +    disp_desc.queue = dispatch_get_main_queue();
>      >      > +    disp_desc.newFrameEventHandler = ^(void) {
>      >      > +        trace_apple_gfx_new_frame();
>      >      > +        dispatch_async(s->render_queue, ^{
>      >      > +            /* Drop frames if we get too far ahead. */
>      >      > +            if (s->pending_frames >= 2)
>      >      > +                return;
>      >      > +            ++s->pending_frames;
>      >      > +            if (s->pending_frames > 1) {
>      >      > +                return;
>      >      > +            }
>      >      > +            @autoreleasepool {
>      >      > +                apple_gfx_render_new_frame(s);
>      >      > +            }
>      >      > +        });
>      >      > +    };
>      >      > +    disp_desc.modeChangeHandler = ^(PGDisplayCoord_t
>     sizeInPixels,
>      >      > +                                    OSType pixelFormat) {
>      >      > +        trace_apple_gfx_mode_change(sizeInPixels.x,
>     sizeInPixels.y);
>      >      > +        set_mode(s, sizeInPixels.x, sizeInPixels.y);
>      >      > +    };
>      >      > +    disp_desc.cursorGlyphHandler = ^(NSBitmapImageRep *glyph,
>      >      > +                                     PGDisplayCoord_t
>     hotSpot) {
>      >      > +        uint32_t bpp = glyph.bitsPerPixel;
>      >      > +        size_t width = glyph.pixelsWide;
>      >      > +        size_t height = glyph.pixelsHigh;
>      >      > +        size_t padding_bytes_per_row = glyph.bytesPerRow
>     - width
>      >     * 4;
>      >      > +        const uint8_t* px_data = glyph.bitmapData;
>      >      > +
>      >      > +        trace_apple_gfx_cursor_set(bpp, width, height);
>      >      > +
>      >      > +        if (s->cursor) {
>      >      > +            cursor_unref(s->cursor);
>      >      > +            s->cursor = NULL;
>      >      > +        }
>      >      > +
>      >      > +        if (bpp == 32) { /* Shouldn't be anything else,
>     but just
>      >     to be safe...*/
>      >      > +            s->cursor = cursor_alloc(width, height);
>      >      > +            s->cursor->hot_x = hotSpot.x;
>      >      > +            s->cursor->hot_y = hotSpot.y;
>      >      > +
>      >      > +            uint32_t *dest_px = s->cursor->data;
>      >      > +
>      >      > +            for (size_t y = 0; y < height; ++y) {
>      >      > +                for (size_t x = 0; x < width; ++x) {
>      >      > +                    /* NSBitmapImageRep's red & blue channels
>      >     are swapped
>      >      > +                     * compared to QEMUCursor's. */
>      >      > +                    *dest_px =
>      >      > +                        (px_data[0] << 16u) |
>      >      > +                        (px_data[1] <<  8u) |
>      >      > +                        (px_data[2] <<  0u) |
>      >      > +                        (px_data[3] << 24u);
>      >      > +                    ++dest_px;
>      >      > +                    px_data += 4;
>      >      > +                }
>      >      > +                px_data += padding_bytes_per_row;
>      >      > +            }
>      >      > +            dpy_cursor_define(s->con, s->cursor);
>      >      > +            update_cursor(s);
>      >      > +        }
>      >      > +    };
>      >      > +    disp_desc.cursorShowHandler = ^(BOOL show) {
>      >      > +        trace_apple_gfx_cursor_show(show);
>      >      > +        s->cursor_show = show;
>      >      > +        update_cursor(s);
>      >      > +    };
>      >      > +    disp_desc.cursorMoveHandler = ^(void) {
>      >      > +        trace_apple_gfx_cursor_move();
>      >      > +        update_cursor(s);
>      >      > +    };
>      >      > +
>      >      > +    return disp_desc;
>      >      > +}
>      >      > +
>      >      > +static NSArray<PGDisplayMode*>*
>      >     apple_gfx_prepare_display_mode_array(void)
>      >      > +{
>      >      > +    PGDisplayMode *modes[ARRAY_SIZE(apple_gfx_modes)];
>      >      > +    NSArray<PGDisplayMode*>* mode_array = nil;
>      >      > +    int i;
>      >      > +
>      >      > +    for (i = 0; i < ARRAY_SIZE(apple_gfx_modes); i++) {
>      >      > +        modes[i] =
>      >      > +            [[PGDisplayMode alloc]
>      >     initWithSizeInPixels:apple_gfx_modes[i] refreshRateInHz:60.];
>      >      > +    }
>      >      > +
>      >      > +    mode_array = [NSArray arrayWithObjects:modes
>      >     count:ARRAY_SIZE(apple_gfx_modes)];
>      >      > +
>      >      > +    for (i = 0; i < ARRAY_SIZE(apple_gfx_modes); i++) {
>      >      > +        [modes[i] release];
>      >      > +        modes[i] = nil;
>      >      > +    }
>      >      > +
>      >      > +    return mode_array;
>      >      > +}
>      >      > +
>      >      > +static id<MTLDevice> copy_suitable_metal_device(void)
>      >      > +{
>      >      > +    id<MTLDevice> dev = nil;
>      >      > +    NSArray<id<MTLDevice>> *devs = MTLCopyAllDevices();
>      >      > +
>      >      > +    /* Prefer a unified memory GPU. Failing that, pick a non-
>      >     removable GPU. */
>      >      > +    for (size_t i = 0; i < devs.count; ++i) {
>      >      > +        if (devs[i].hasUnifiedMemory) {
>      >      > +            dev = devs[i];
>      >      > +            break;
>      >      > +        }
>      >      > +        if (!devs[i].removable) {
>      >      > +            dev = devs[i];
>      >      > +        }
>      >      > +    }
>      >      > +
>      >      > +    if (dev != nil) {
>      >      > +        [dev retain];
>      >      > +    } else {
>      >      > +        dev = MTLCreateSystemDefaultDevice();
>      >      > +    }
>      >      > +    [devs release];
>      >      > +
>      >      > +    return dev;
>      >      > +}
>      >      > +
>      >      > +void apple_gfx_common_realize(AppleGFXState *s,
>      >     PGDeviceDescriptor *desc)
>      >      > +{
>      >      > +    PGDisplayDescriptor *disp_desc = nil;
>      >      > +
>      >      > +    QTAILQ_INIT(&s->tasks);
>      >      > +    s->render_queue = dispatch_queue_create("apple-
>     gfx.render",
>      >      > +                                           
>     DISPATCH_QUEUE_SERIAL);
>      >      > +    s->mtl = copy_suitable_metal_device();
>      >      > +    s->mtl_queue = [s->mtl newCommandQueue];
>      >      > +
>      >      > +    desc.device = s->mtl;
>      >      > +
>      >      > +    apple_gfx_register_task_mapping_handlers(s, desc);
>      >      > +
>      >      > +    s->pgdev = PGNewDeviceWithDescriptor(desc);
>      >      > +
>      >      > +    disp_desc = apple_gfx_prepare_display_handlers(s);
>      >      > +    s->pgdisp = [s->pgdev newDisplayWithDescriptor:disp_desc
>      >      > +                                              port:0
>      >     serialNum:1234];
>      >      > +    [disp_desc release];
>      >      > +    s->pgdisp.modeList =
>     apple_gfx_prepare_display_mode_array();
>      >      > +
>      >      > +    create_fb(s);
>      >      > +}
>      >      > diff --git a/hw/display/meson.build b/hw/display/meson.build
>      >      > index 7db05eace97..70d855749c0 100644
>      >      > --- a/hw/display/meson.build
>      >      > +++ b/hw/display/meson.build
>      >      > @@ -65,6 +65,8 @@ system_ss.add(when: 'CONFIG_ARTIST',
>     if_true:
>      >     files('artist.c'))
>      >      >
>      >      >   system_ss.add(when: 'CONFIG_ATI_VGA', if_true:
>     [files('ati.c',
>      >     'ati_2d.c', 'ati_dbg.c'), pixman])
>      >      >
>      >      > +system_ss.add(when: 'CONFIG_MAC_PVG',         if_true:
>      >     [files('apple-gfx.m'), pvg, metal])
>      >      > +system_ss.add(when: 'CONFIG_MAC_PVG_VMAPPLE', if_true:
>      >     [files('apple-gfx-vmapple.m'), pvg, metal])
>      >      >
>      >      >   if config_all_devices.has_key('CONFIG_VIRTIO_GPU')
>      >      >     virtio_gpu_ss = ss.source_set()
>      >      > diff --git a/hw/display/trace-events b/hw/display/trace-events
>      >      > index 781f8a33203..1809a358e36 100644
>      >      > --- a/hw/display/trace-events
>      >      > +++ b/hw/display/trace-events
>      >      > @@ -191,3 +191,29 @@ dm163_bits_ppi(unsigned dest_width)
>      >     "dest_width : %u"
>      >      >   dm163_leds(int led, uint32_t value) "led %d: 0x%x"
>      >      >   dm163_channels(int channel, uint8_t value) "channel %d:
>     0x%x"
>      >      >   dm163_refresh_rate(uint32_t rr) "refresh rate %d"
>      >      > +
>      >      > +# apple-gfx.m
>      >      > +apple_gfx_read(uint64_t offset, uint64_t res)
>      >     "offset=0x%"PRIx64" res=0x%"PRIx64
>      >      > +apple_gfx_write(uint64_t offset, uint64_t val)
>      >     "offset=0x%"PRIx64" val=0x%"PRIx64
>      >      > +apple_gfx_create_task(uint32_t vm_size, void *va)
>     "vm_size=0x%x
>      >     base_addr=%p"
>      >      > +apple_gfx_destroy_task(void *task) "task=%p"
>      >      > +apple_gfx_map_memory(void *task, uint32_t range_count,
>     uint64_t
>      >     virtual_offset, uint32_t read_only) "task=%p range_count=0x%x
>      >     virtual_offset=0x%"PRIx64" read_only=%d"
>      >      > +apple_gfx_map_memory_range(uint32_t i, uint64_t phys_addr,
>      >     uint64_t phys_len) "[%d] phys_addr=0x%"PRIx64"
>     phys_len=0x%"PRIx64
>      >      > +apple_gfx_remap(uint64_t retval, uint64_t source, uint64_t
>      >     target) "retval=%"PRId64" source=0x%"PRIx64" target=0x%"PRIx64
>      >      > +apple_gfx_unmap_memory(void *task, uint64_t virtual_offset,
>      >     uint64_t length) "task=%p virtual_offset=0x%"PRIx64"
>     length=0x%"PRIx64
>      >      > +apple_gfx_read_memory(uint64_t phys_address, uint64_t length,
>      >     void *dst) "phys_addr=0x%"PRIx64" length=0x%"PRIx64" dest=%p"
>      >      > +apple_gfx_raise_irq(uint32_t vector) "vector=0x%x"
>      >      > +apple_gfx_new_frame(void) ""
>      >      > +apple_gfx_mode_change(uint64_t x, uint64_t y) "x=%"PRId64"
>      >     y=%"PRId64
>      >      > +apple_gfx_cursor_set(uint32_t bpp, uint64_t width, uint64_t
>      >     height) "bpp=%d width=%"PRId64" height=0x%"PRId64
>      >      > +apple_gfx_cursor_show(uint32_t show) "show=%d"
>      >      > +apple_gfx_cursor_move(void) ""
>      >      > +apple_gfx_common_init(const char *device_name, size_t
>     mmio_size)
>      >     "device: %s; MMIO size: %zu bytes"
>      >      > +
>      >      > +# apple-gfx-vmapple.m
>      >      > +apple_iosfc_read(uint64_t offset, uint64_t res)
>      >     "offset=0x%"PRIx64" res=0x%"PRIx64
>      >      > +apple_iosfc_write(uint64_t offset, uint64_t val)
>      >     "offset=0x%"PRIx64" val=0x%"PRIx64
>      >      > +apple_iosfc_map_memory(uint64_t phys, uint64_t len,
>     uint32_t ro,
>      >     void *va, void *e, void *f) "phys=0x%"PRIx64" len=0x%"PRIx64"
>     ro=%d
>      >     va=%p e=%p f=%p"
>      >      > +apple_iosfc_unmap_memory(void *a, void *b, void *c, void *d,
>      >     void *e, void *f) "a=%p b=%p c=%p d=%p e=%p f=%p"
>      >      > +apple_iosfc_raise_irq(uint32_t vector) "vector=0x%x"
>      >      > +
>      >      > diff --git a/meson.build b/meson.build
>      >      > index 10464466ff3..f09df3f09d5 100644
>      >      > --- a/meson.build
>      >      > +++ b/meson.build
>      >      > @@ -741,6 +741,8 @@ socket = []
>      >      >   version_res = []
>      >      >   coref = []
>      >      >   iokit = []
>      >      > +pvg = []
>      >      > +metal = []
>      >      >   emulator_link_args = []
>      >      >   midl = not_found
>      >      >   widl = not_found
>      >      > @@ -762,6 +764,8 @@ elif host_os == 'darwin'
>      >      >     coref = dependency('appleframeworks', modules:
>     'CoreFoundation')
>      >      >     iokit = dependency('appleframeworks', modules: 'IOKit',
>      >     required: false)
>      >      >     host_dsosuf = '.dylib'
>      >      > +  pvg = dependency('appleframeworks', modules:
>      >     'ParavirtualizedGraphics')
>      >      > +  metal = dependency('appleframeworks', modules: 'Metal')
>      >      >   elif host_os == 'sunos'
>      >      >     socket = [cc.find_library('socket'),
>      >      >               cc.find_library('nsl'),
>      >
> 

Re: [PATCH v3 12/14] hw/vmapple/cfg: Introduce vmapple cfg region

[Phil Dennis-Jordan]

[-- Attachment #1: Type: text/plain, Size: 9867 bytes --]

On Sat, 5 Oct 2024 at 07:35, Akihiko Odaki <akihiko.odaki@daynix.com> wrote:

> On 2024/09/28 17:57, Phil Dennis-Jordan wrote:
> > From: Alexander Graf <graf@amazon.com>
> >
> > Instead of device tree or other more standardized means, VMApple passes
> > platform configuration to the first stage boot loader in a binary encoded
> > format that resides at a dedicated RAM region in physical address space.
> >
> > This patch models this configuration space as a qdev device which we can
> > then map at the fixed location in the address space. That way, we can
> > influence and annotate all configuration fields easily.
> >
> > Signed-off-by: Alexander Graf <graf@amazon.com>
> > Signed-off-by: Phil Dennis-Jordan <phil@philjordan.eu>
> >
> > ---
> > v3:
> >
> >   * Replaced legacy device reset method with Resettable method
> >
> >   hw/vmapple/Kconfig       |   3 ++
> >   hw/vmapple/cfg.c         | 106 +++++++++++++++++++++++++++++++++++++++
> >   hw/vmapple/meson.build   |   1 +
> >   include/hw/vmapple/cfg.h |  68 +++++++++++++++++++++++++
> >   4 files changed, 178 insertions(+)
> >   create mode 100644 hw/vmapple/cfg.c
> >   create mode 100644 include/hw/vmapple/cfg.h
> >
> > diff --git a/hw/vmapple/Kconfig b/hw/vmapple/Kconfig
> > index 68f88876eb9..8bbeb9a9237 100644
> > --- a/hw/vmapple/Kconfig
> > +++ b/hw/vmapple/Kconfig
> > @@ -4,3 +4,6 @@ config VMAPPLE_AES
> >   config VMAPPLE_BDIF
> >       bool
> >
> > +config VMAPPLE_CFG
> > +    bool
> > +
> > diff --git a/hw/vmapple/cfg.c b/hw/vmapple/cfg.c
> > new file mode 100644
> > index 00000000000..a5e5c62f59f
> > --- /dev/null
> > +++ b/hw/vmapple/cfg.c
> > @@ -0,0 +1,106 @@
> > +/*
> > + * VMApple Configuration Region
> > + *
> > + * Copyright © 2023 Amazon.com, Inc. or its affiliates. All Rights
> Reserved.
> > + *
> > + * This work is licensed under the terms of the GNU GPL, version 2 or
> later.
> > + * See the COPYING file in the top-level directory.
> > + */
> > +
> > +#include "qemu/osdep.h"
> > +#include "hw/vmapple/cfg.h"
> > +#include "qemu/log.h"
> > +#include "qemu/module.h"
> > +#include "qapi/error.h"
> > +
> > +static void vmapple_cfg_reset(Object *obj, ResetType type)
> > +{
> > +    VMAppleCfgState *s = VMAPPLE_CFG(obj);
> > +    VMAppleCfg *cfg;
> > +
> > +    cfg = memory_region_get_ram_ptr(&s->mem);
> > +    memset((void *)cfg, 0, VMAPPLE_CFG_SIZE);
> > +    *cfg = s->cfg;
>  > +}> +
> > +static void vmapple_cfg_realize(DeviceState *dev, Error **errp)
> > +{
> > +    VMAppleCfgState *s = VMAPPLE_CFG(dev);
> > +    uint32_t i;
> > +
> > +    strncpy(s->cfg.serial, s->serial, sizeof(s->cfg.serial));
> > +    strncpy(s->cfg.model, s->model, sizeof(s->cfg.model));
> > +    strncpy(s->cfg.soc_name, s->soc_name, sizeof(s->cfg.soc_name));
> > +    strncpy(s->cfg.unk8, "D/A", sizeof(s->cfg.soc_name));
>
> Use qemu_strnlen() to report an error for too long strings.
>

Hmm, I don't see any existing instances of such a pattern. I do however see
a couple of uses of g_strlcpy in the Qemu codebase - that would be a better
candidate for error checked string copying, though it still involves some
awkward return value checks. I'm going to wrap that in a helper function
and macro to replace all 4 strncpy instances here. If the same thing is
useful elsewhere later, it can be promoted to cutils or similar.

(Also, I notice that last strncpy actually uses the wrong destination size;
my wrapper macro uses ARRAY_SIZE to avoid this mistake altogether.)

> +    s->cfg.ecid = cpu_to_be64(s->cfg.ecid);
> > +    s->cfg.version = 2;
> > +    s->cfg.unk1 = 1;
> > +    s->cfg.unk2 = 1;
> > +    s->cfg.unk3 = 0x20;
> > +    s->cfg.unk4 = 0;
> > +    s->cfg.unk5 = 1;
> > +    s->cfg.unk6 = 1;
> > +    s->cfg.unk7 = 0;
> > +    s->cfg.unk10 = 1;
> > +
> > +    g_assert(s->cfg.nr_cpus < ARRAY_SIZE(s->cfg.cpu_ids));
>
> Report an error instead of asserting.
>
> > +    for (i = 0; i < s->cfg.nr_cpus; i++) {
> > +        s->cfg.cpu_ids[i] = i;
> > +    }
>  > +}> +
> > +static void vmapple_cfg_init(Object *obj)
> > +{
> > +    VMAppleCfgState *s = VMAPPLE_CFG(obj);
> > +
> > +    memory_region_init_ram(&s->mem, obj, "VMApple Config",
> VMAPPLE_CFG_SIZE,
> > +                           &error_fatal);
> > +    sysbus_init_mmio(SYS_BUS_DEVICE(obj), &s->mem);
> > +
> > +    s->serial = (char *)"1234";
> > +    s->model = (char *)"VM0001";
> > +    s->soc_name = (char *)"Apple M1 (Virtual)";
>
> These casts are unsafe; these pointers will be freed when this device is
> freed.
>

Good catch! The more usual pattern for default string property values seems
to be to fill them in _realize() (using g_strdup()) if no other value was
previously set, so I've applied that here for the next version of the patch.


>
> > +}
> > +
> > +static Property vmapple_cfg_properties[] = {
> > +    DEFINE_PROP_UINT32("nr-cpus", VMAppleCfgState, cfg.nr_cpus, 1),
> > +    DEFINE_PROP_UINT64("ecid", VMAppleCfgState, cfg.ecid, 0),
> > +    DEFINE_PROP_UINT64("ram-size", VMAppleCfgState, cfg.ram_size, 0),
> > +    DEFINE_PROP_UINT32("run_installer1", VMAppleCfgState,
> cfg.run_installer1, 0),
> > +    DEFINE_PROP_UINT32("run_installer2", VMAppleCfgState,
> cfg.run_installer2, 0),
> > +    DEFINE_PROP_UINT32("rnd", VMAppleCfgState, cfg.rnd, 0),
> > +    DEFINE_PROP_MACADDR("mac-en0", VMAppleCfgState, cfg.mac_en0),
> > +    DEFINE_PROP_MACADDR("mac-en1", VMAppleCfgState, cfg.mac_en1),
> > +    DEFINE_PROP_MACADDR("mac-wifi0", VMAppleCfgState, cfg.mac_wifi0),
> > +    DEFINE_PROP_MACADDR("mac-bt0", VMAppleCfgState, cfg.mac_bt0),
> > +    DEFINE_PROP_STRING("serial", VMAppleCfgState, serial),
> > +    DEFINE_PROP_STRING("model", VMAppleCfgState, model),
> > +    DEFINE_PROP_STRING("soc_name", VMAppleCfgState, soc_name),
> > +    DEFINE_PROP_END_OF_LIST(),
> > +};
> > +
> > +static void vmapple_cfg_class_init(ObjectClass *klass, void *data)
> > +{
> > +    DeviceClass *dc = DEVICE_CLASS(klass);
> > +    ResettableClass *rc = RESETTABLE_CLASS(klass);
> > +
> > +    dc->realize = vmapple_cfg_realize;
> > +    dc->desc = "VMApple Configuration Region";
> > +    device_class_set_props(dc, vmapple_cfg_properties);
> > +    rc->phases.hold = vmapple_cfg_reset;
> > +}
> > +
> > +static const TypeInfo vmapple_cfg_info = {
> > +    .name          = TYPE_VMAPPLE_CFG,
> > +    .parent        = TYPE_SYS_BUS_DEVICE,
> > +    .instance_size = sizeof(VMAppleCfgState),
> > +    .instance_init = vmapple_cfg_init,
> > +    .class_init    = vmapple_cfg_class_init,
> > +};
> > +
> > +static void vmapple_cfg_register_types(void)
> > +{
> > +    type_register_static(&vmapple_cfg_info);
> > +}
> > +
> > +type_init(vmapple_cfg_register_types)
> > diff --git a/hw/vmapple/meson.build b/hw/vmapple/meson.build
> > index d4624713deb..64b78693a31 100644
> > --- a/hw/vmapple/meson.build
> > +++ b/hw/vmapple/meson.build
> > @@ -1,2 +1,3 @@
> >   system_ss.add(when: 'CONFIG_VMAPPLE_AES',  if_true: files('aes.c'))
> >   system_ss.add(when: 'CONFIG_VMAPPLE_BDIF', if_true: files('bdif.c'))
> > +system_ss.add(when: 'CONFIG_VMAPPLE_CFG',  if_true: files('cfg.c'))
> > diff --git a/include/hw/vmapple/cfg.h b/include/hw/vmapple/cfg.h
> > new file mode 100644
> > index 00000000000..3337064e447
> > --- /dev/null
> > +++ b/include/hw/vmapple/cfg.h
> > @@ -0,0 +1,68 @@
> > +/*
> > + * VMApple Configuration Region
> > + *
> > + * Copyright © 2023 Amazon.com, Inc. or its affiliates. All Rights
> Reserved.
> > + *
> > + * This work is licensed under the terms of the GNU GPL, version 2 or
> later.
> > + * See the COPYING file in the top-level directory.
> > + */
> > +
> > +#ifndef HW_VMAPPLE_CFG_H
> > +#define HW_VMAPPLE_CFG_H
> > +
> > +#include "hw/sysbus.h"
> > +#include "qom/object.h"
> > +#include "net/net.h"
> > +
> > +typedef struct VMAppleCfg {
> > +    uint32_t version;         /* 0x000 */
> > +    uint32_t nr_cpus;         /* 0x004 */
> > +    uint32_t unk1;            /* 0x008 */
> > +    uint32_t unk2;            /* 0x00c */
> > +    uint32_t unk3;            /* 0x010 */
> > +    uint32_t unk4;            /* 0x014 */
> > +    uint64_t ecid;            /* 0x018 */
> > +    uint64_t ram_size;        /* 0x020 */
> > +    uint32_t run_installer1;  /* 0x028 */
> > +    uint32_t unk5;            /* 0x02c */
> > +    uint32_t unk6;            /* 0x030 */
> > +    uint32_t run_installer2;  /* 0x034 */
> > +    uint32_t rnd;             /* 0x038 */
> > +    uint32_t unk7;            /* 0x03c */
> > +    MACAddr mac_en0;          /* 0x040 */
> > +    uint8_t pad1[2];
> > +    MACAddr mac_en1;          /* 0x048 */
> > +    uint8_t pad2[2];
> > +    MACAddr mac_wifi0;        /* 0x050 */
> > +    uint8_t pad3[2];
> > +    MACAddr mac_bt0;          /* 0x058 */
> > +    uint8_t pad4[2];
> > +    uint8_t reserved[0xa0];   /* 0x060 */
> > +    uint32_t cpu_ids[0x80];   /* 0x100 */
> > +    uint8_t scratch[0x200];   /* 0x180 */
> > +    char serial[32];          /* 0x380 */
> > +    char unk8[32];            /* 0x3a0 */
> > +    char model[32];           /* 0x3c0 */
> > +    uint8_t unk9[32];         /* 0x3e0 */
> > +    uint32_t unk10;           /* 0x400 */
> > +    char soc_name[32];        /* 0x404 */
> > +} VMAppleCfg;
> > +
> > +#define TYPE_VMAPPLE_CFG "vmapple-cfg"
> > +OBJECT_DECLARE_SIMPLE_TYPE(VMAppleCfgState, VMAPPLE_CFG)
> > +
> > +struct VMAppleCfgState {
> > +    /* <private> */
> > +    SysBusDevice parent_obj;
> > +    VMAppleCfg cfg;
> > +
> > +    /* <public> */
> > +    MemoryRegion mem;
> > +    char *serial;
> > +    char *model;
> > +    char *soc_name;
> > +};
> > +
> > +#define VMAPPLE_CFG_SIZE 0x00010000
> > +
> > +#endif /* HW_VMAPPLE_CFG_H */
>
>

[-- Attachment #2: Type: text/html, Size: 12759 bytes --]

Re: [PATCH v3 13/14] hw/vmapple/virtio-blk: Add support for apple virtio-blk

[Phil Dennis-Jordan]

[-- Attachment #1: Type: text/plain, Size: 17196 bytes --]

On Sat, 5 Oct 2024 at 07:47, Akihiko Odaki <akihiko.odaki@daynix.com> wrote:

> On 2024/09/28 17:57, Phil Dennis-Jordan wrote:
> > From: Alexander Graf <graf@amazon.com>
> >
> > Apple has its own virtio-blk PCI device ID where it deviates from the
> > official virtio-pci spec slightly: It puts a new "apple type"
> > field at a static offset in config space and introduces a new barrier
> > command.
> >
> > This patch first creates a mechanism for virtio-blk downstream classes to
> > handle unknown commands. It then creates such a downstream class and a
> new
> > vmapple-virtio-blk-pci class which support the additional apple type
> config
> > identifier as well as the barrier command.
> >
> > It then exposes 2 subclasses from that that we can use to expose root and
> > aux virtio-blk devices: "vmapple-virtio-root" and "vmapple-virtio-aux".
> >
> > Signed-off-by: Alexander Graf <graf@amazon.com>
> > Signed-off-by: Phil Dennis-Jordan <phil@philjordan.eu>
> > ---
> >   hw/block/virtio-blk.c           |  19 ++-
> >   hw/vmapple/Kconfig              |   3 +
> >   hw/vmapple/meson.build          |   1 +
> >   hw/vmapple/virtio-blk.c         | 212 ++++++++++++++++++++++++++++++++
> >   include/hw/pci/pci_ids.h        |   1 +
> >   include/hw/virtio/virtio-blk.h  |  12 +-
> >   include/hw/vmapple/virtio-blk.h |  39 ++++++
> >   7 files changed, 282 insertions(+), 5 deletions(-)
> >   create mode 100644 hw/vmapple/virtio-blk.c
> >   create mode 100644 include/hw/vmapple/virtio-blk.h
> >
> > diff --git a/hw/block/virtio-blk.c b/hw/block/virtio-blk.c
> > index 115795392c4..cecc4cef9e4 100644
> > --- a/hw/block/virtio-blk.c
> > +++ b/hw/block/virtio-blk.c
> > @@ -50,12 +50,12 @@ static void virtio_blk_init_request(VirtIOBlock *s,
> VirtQueue *vq,
> >       req->mr_next = NULL;
> >   }
> >
> > -static void virtio_blk_free_request(VirtIOBlockReq *req)
> > +void virtio_blk_free_request(VirtIOBlockReq *req)
> >   {
> >       g_free(req);
> >   }
>
> This function is identical with g_free(). Perhaps it's better to remove
> it instead of updating it.
>

I'm not sure that's something I should be doing in such a tangential patch
series, and it's very much a matter of taste - other operations on
VirtIOBlockReq have similar, consistent naming, and this function seemingly
hasn't been touched for 7 years.

Perhaps virtio-blk maintainer Stefan Hajnoczi could weigh in? (tagged in
To:)



> > -static void virtio_blk_req_complete(VirtIOBlockReq *req, unsigned char
> status)
> > +void virtio_blk_req_complete(VirtIOBlockReq *req, unsigned char status)
> >   {
> >       VirtIOBlock *s = req->dev;
> >       VirtIODevice *vdev = VIRTIO_DEVICE(s);
> > @@ -966,8 +966,18 @@ static int virtio_blk_handle_request(VirtIOBlockReq
> *req, MultiReqBuffer *mrb)
> >           break;
> >       }
> >       default:
> > -        virtio_blk_req_complete(req, VIRTIO_BLK_S_UNSUPP);
> > -        virtio_blk_free_request(req);
> > +    {
> > +        /*
> > +         * Give subclasses a chance to handle unknown requests. This
> way the
> > +         * class lookup is not in the hot path.
> > +         */
> > +        VirtIOBlkClass *vbk = VIRTIO_BLK_GET_CLASS(s);
> > +        if (!vbk->handle_unknown_request ||
> > +            !vbk->handle_unknown_request(req, mrb, type)) {
> > +            virtio_blk_req_complete(req, VIRTIO_BLK_S_UNSUPP);
> > +            virtio_blk_free_request(req);
> > +        }
> > +    }
> >       }
> >       return 0;
> >   }
> > @@ -2044,6 +2054,7 @@ static const TypeInfo virtio_blk_info = {
> >       .instance_size = sizeof(VirtIOBlock),
> >       .instance_init = virtio_blk_instance_init,
> >       .class_init = virtio_blk_class_init,
> > +    .class_size = sizeof(VirtIOBlkClass),
> >   };
> >
> >   static void virtio_register_types(void)
> > diff --git a/hw/vmapple/Kconfig b/hw/vmapple/Kconfig
> > index 8bbeb9a9237..bcd1be63e3c 100644
> > --- a/hw/vmapple/Kconfig
> > +++ b/hw/vmapple/Kconfig
> > @@ -7,3 +7,6 @@ config VMAPPLE_BDIF
> >   config VMAPPLE_CFG
> >       bool
> >
> > +config VMAPPLE_VIRTIO_BLK
> > +    bool
> > +
> > diff --git a/hw/vmapple/meson.build b/hw/vmapple/meson.build
> > index 64b78693a31..bf17cf906c9 100644
> > --- a/hw/vmapple/meson.build
> > +++ b/hw/vmapple/meson.build
> > @@ -1,3 +1,4 @@
> >   system_ss.add(when: 'CONFIG_VMAPPLE_AES',  if_true: files('aes.c'))
> >   system_ss.add(when: 'CONFIG_VMAPPLE_BDIF', if_true: files('bdif.c'))
> >   system_ss.add(when: 'CONFIG_VMAPPLE_CFG',  if_true: files('cfg.c'))
> > +system_ss.add(when: 'CONFIG_VMAPPLE_VIRTIO_BLK',  if_true:
> files('virtio-blk.c'))
> > diff --git a/hw/vmapple/virtio-blk.c b/hw/vmapple/virtio-blk.c
> > new file mode 100644
> > index 00000000000..720eaa61a86
> > --- /dev/null
> > +++ b/hw/vmapple/virtio-blk.c
> > @@ -0,0 +1,212 @@
> > +/*
> > + * VMApple specific VirtIO Block implementation
> > + *
> > + * Copyright © 2023 Amazon.com, Inc. or its affiliates. All Rights
> Reserved.
> > + *
> > + * This work is licensed under the terms of the GNU GPL, version 2 or
> later.
> > + * See the COPYING file in the top-level directory.
> > + *
> > + * VMApple uses almost standard VirtIO Block, but with a few key
> differences:
> > + *
> > + *  - Different PCI device/vendor ID
> > + *  - An additional "type" identifier to differentiate AUX and Root
> volumes
> > + *  - An additional BARRIER command
> > + */
> > +
> > +#include "qemu/osdep.h"
> > +#include "hw/vmapple/virtio-blk.h"
> > +#include "qemu/log.h"
> > +#include "qemu/module.h"
> > +#include "qapi/error.h"
> > +
> > +#define VIRTIO_BLK_T_APPLE_BARRIER     0x10000
> > +
> > +#define VIRTIO_APPLE_TYPE_ROOT 1
> > +#define VIRTIO_APPLE_TYPE_AUX  2
> > +
> > +static bool vmapple_virtio_blk_handle_unknown_request(VirtIOBlockReq
> *req,
> > +                                                      MultiReqBuffer
> *mrb,
> > +                                                      uint32_t type)
> > +{
> > +    switch (type) {
> > +    case VIRTIO_BLK_T_APPLE_BARRIER:
> > +        /* We ignore barriers for now. YOLO. */
>
> It should be LOG_UNIMP instead of a mere comment.
>

Fixed in next patch version.


> > +        virtio_blk_req_complete(req, VIRTIO_BLK_S_OK);
> > +        virtio_blk_free_request(req);
> > +        return true;
> > +    default:
> > +        return false;
> > +    }
> > +}
> > +
> > +/*
> > + * VMApple virtio-blk uses the same config format as normal virtio,
> with one
> > + * exception: It adds an "apple type" specififer at the same location
> that
> > + * the spec reserves for max_secure_erase_sectors. Let's hook into the
> > + * get_config code path here, run it as usual and then patch in the
> apple type.
> > + */
> > +static void vmapple_virtio_blk_get_config(VirtIODevice *vdev, uint8_t
> *config)
> > +{
> > +    VMAppleVirtIOBlk *dev = VMAPPLE_VIRTIO_BLK(vdev);
> > +    VMAppleVirtIOBlkClass *vvbk = VMAPPLE_VIRTIO_BLK_GET_CLASS(dev);
> > +    struct virtio_blk_config *blkcfg = (struct virtio_blk_config
> *)config;
> > +
> > +    vvbk->get_config(vdev, config);
> > +
> > +    g_assert(dev->parent_obj.config_size >= endof(struct
> virtio_blk_config, zoned));
> > +
> > +    /* Apple abuses the field for max_secure_erase_sectors as type id */
> > +    blkcfg->max_secure_erase_sectors = dev->apple_type;
> > +}
> > +
> > +static Property vmapple_virtio_blk_properties[] = {
> > +    DEFINE_PROP_UINT32("apple-type", VMAppleVirtIOBlk, apple_type, 0),
> > +    DEFINE_PROP_END_OF_LIST(),
> > +};
> > +
> > +static void vmapple_virtio_blk_class_init(ObjectClass *klass, void
> *data)
> > +{
> > +    DeviceClass *dc = DEVICE_CLASS(klass);
> > +    VirtIOBlkClass *vbk = VIRTIO_BLK_CLASS(klass);
> > +    VirtioDeviceClass *vdc = VIRTIO_DEVICE_CLASS(klass);
> > +    VMAppleVirtIOBlkClass *vvbk = VMAPPLE_VIRTIO_BLK_CLASS(klass);
> > +
> > +    vbk->handle_unknown_request =
> vmapple_virtio_blk_handle_unknown_request;
> > +    vvbk->get_config = vdc->get_config;
> > +    vdc->get_config = vmapple_virtio_blk_get_config;
> > +    device_class_set_props(dc, vmapple_virtio_blk_properties);
> > +}
> > +
> > +static const TypeInfo vmapple_virtio_blk_info = {
> > +    .name          = TYPE_VMAPPLE_VIRTIO_BLK,
> > +    .parent        = TYPE_VIRTIO_BLK,
> > +    .instance_size = sizeof(VMAppleVirtIOBlk),
> > +    .class_init    = vmapple_virtio_blk_class_init,
> > +};
> > +
> > +/* PCI Devices */
> > +
> > +typedef struct VMAppleVirtIOBlkPCI {
> > +    VirtIOPCIProxy parent_obj;
> > +    VMAppleVirtIOBlk vdev;
> > +    uint32_t apple_type;
> > +} VMAppleVirtIOBlkPCI;
> > +
> > +/*
> > + * vmapple-virtio-blk-pci: This extends VirtioPCIProxy.
> > + */
> > +#define TYPE_VMAPPLE_VIRTIO_BLK_PCI "vmapple-virtio-blk-pci-base"
> > +DECLARE_INSTANCE_CHECKER(VMAppleVirtIOBlkPCI, VMAPPLE_VIRTIO_BLK_PCI,
> > +                         TYPE_VMAPPLE_VIRTIO_BLK_PCI)
> > +
> > +static Property vmapple_virtio_blk_pci_properties[] = {
> > +    DEFINE_PROP_UINT32("class", VirtIOPCIProxy, class_code, 0),
> > +    DEFINE_PROP_BIT("ioeventfd", VirtIOPCIProxy, flags,
> > +                    VIRTIO_PCI_FLAG_USE_IOEVENTFD_BIT, true),
> > +    DEFINE_PROP_UINT32("vectors", VirtIOPCIProxy, nvectors,
> > +                       DEV_NVECTORS_UNSPECIFIED),
> > +    DEFINE_PROP_END_OF_LIST(),
> > +};
> > +
> > +static void vmapple_virtio_blk_pci_realize(VirtIOPCIProxy *vpci_dev,
> Error **errp)
> > +{
> > +    VMAppleVirtIOBlkPCI *dev = VMAPPLE_VIRTIO_BLK_PCI(vpci_dev);
> > +    DeviceState *vdev = DEVICE(&dev->vdev);
> > +    VirtIOBlkConf *conf = &dev->vdev.parent_obj.conf;
> > +
> > +    if (conf->num_queues == VIRTIO_BLK_AUTO_NUM_QUEUES) {
> > +        conf->num_queues = virtio_pci_optimal_num_queues(0);
> > +    }
> > +
> > +    if (vpci_dev->nvectors == DEV_NVECTORS_UNSPECIFIED) {
> > +        vpci_dev->nvectors = conf->num_queues + 1;
> > +    }
> > +
> > +    /*
> > +     * We don't support zones, but we need the additional config space
> size.
> > +     * Let's just expose the feature so the rest of the virtio-blk logic
> > +     * allocates enough space for us. The guest will ignore zones
> anyway.
> > +     */
> > +    virtio_add_feature(&dev->vdev.parent_obj.host_features,
> VIRTIO_BLK_F_ZONED);
> > +    /* Propagate the apple type down to the virtio-blk device */
> > +    qdev_prop_set_uint32(DEVICE(&dev->vdev), "apple-type",
> dev->apple_type);
> > +    /* and spawn the virtio-blk device */
> > +    qdev_realize(vdev, BUS(&vpci_dev->bus), errp);
> > +
> > +    /*
> > +     * The virtio-pci machinery adjusts its vendor/device ID based on
> whether
> > +     * we support modern or legacy virtio. Let's patch it back to the
> Apple
> > +     * identifiers here.
> > +     */
> > +    pci_config_set_vendor_id(vpci_dev->pci_dev.config,
> PCI_VENDOR_ID_APPLE);
> > +    pci_config_set_device_id(vpci_dev->pci_dev.config,
> PCI_DEVICE_ID_APPLE_VIRTIO_BLK);
> > +}
> > +
> > +static void vmapple_virtio_blk_pci_class_init(ObjectClass *klass, void
> *data)
> > +{
> > +    DeviceClass *dc = DEVICE_CLASS(klass);
> > +    VirtioPCIClass *k = VIRTIO_PCI_CLASS(klass);
> > +    PCIDeviceClass *pcidev_k = PCI_DEVICE_CLASS(klass);
> > +
> > +    set_bit(DEVICE_CATEGORY_STORAGE, dc->categories);
> > +    device_class_set_props(dc, vmapple_virtio_blk_pci_properties);
> > +    k->realize = vmapple_virtio_blk_pci_realize;
> > +    pcidev_k->vendor_id = PCI_VENDOR_ID_APPLE;
> > +    pcidev_k->device_id = PCI_DEVICE_ID_APPLE_VIRTIO_BLK;
> > +    pcidev_k->revision = VIRTIO_PCI_ABI_VERSION;
> > +    pcidev_k->class_id = PCI_CLASS_STORAGE_SCSI;
> > +}
> > +
> > +static void vmapple_virtio_blk_pci_instance_init(Object *obj)
> > +{
> > +    VMAppleVirtIOBlkPCI *dev = VMAPPLE_VIRTIO_BLK_PCI(obj);
> > +
> > +    virtio_instance_init_common(obj, &dev->vdev, sizeof(dev->vdev),
> > +                                TYPE_VMAPPLE_VIRTIO_BLK);
> > +}
> > +
> > +static const VirtioPCIDeviceTypeInfo vmapple_virtio_blk_pci_info = {
> > +    .base_name     = TYPE_VMAPPLE_VIRTIO_BLK_PCI,
> > +    .generic_name  = "vmapple-virtio-blk-pci",
> > +    .instance_size = sizeof(VMAppleVirtIOBlkPCI),
> > +    .instance_init = vmapple_virtio_blk_pci_instance_init,
> > +    .class_init    = vmapple_virtio_blk_pci_class_init,
> > +};
> > +
> > +static void vmapple_virtio_root_instance_init(Object *obj)
> > +{
> > +    VMAppleVirtIOBlkPCI *dev = VMAPPLE_VIRTIO_BLK_PCI(obj);
> > +
> > +    dev->apple_type = VIRTIO_APPLE_TYPE_ROOT;
> > +}
> > +
> > +static const TypeInfo vmapple_virtio_root_info = {
> > +    .name          = TYPE_VMAPPLE_VIRTIO_ROOT,
> > +    .parent        = "vmapple-virtio-blk-pci",
> > +    .instance_size = sizeof(VMAppleVirtIOBlkPCI),
> > +    .instance_init = vmapple_virtio_root_instance_init,
> > +};
> > +
> > +static void vmapple_virtio_aux_instance_init(Object *obj)
> > +{
> > +    VMAppleVirtIOBlkPCI *dev = VMAPPLE_VIRTIO_BLK_PCI(obj);
> > +
> > +    dev->apple_type = VIRTIO_APPLE_TYPE_AUX;
> > +}
> > +
> > +static const TypeInfo vmapple_virtio_aux_info = {
> > +    .name          = TYPE_VMAPPLE_VIRTIO_AUX,
> > +    .parent        = "vmapple-virtio-blk-pci",
> > +    .instance_size = sizeof(VMAppleVirtIOBlkPCI),
> > +    .instance_init = vmapple_virtio_aux_instance_init,
> > +};
> > +
> > +static void vmapple_virtio_blk_register_types(void)
> > +{
> > +    type_register_static(&vmapple_virtio_blk_info);
> > +    virtio_pci_types_register(&vmapple_virtio_blk_pci_info);
> > +    type_register_static(&vmapple_virtio_root_info);
> > +    type_register_static(&vmapple_virtio_aux_info);
> > +}
> > +
> > +type_init(vmapple_virtio_blk_register_types)
> > diff --git a/include/hw/pci/pci_ids.h b/include/hw/pci/pci_ids.h
> > index f1a53fea8d6..33e2898be95 100644
> > --- a/include/hw/pci/pci_ids.h
> > +++ b/include/hw/pci/pci_ids.h
> > @@ -191,6 +191,7 @@
> >   #define PCI_DEVICE_ID_APPLE_UNI_N_AGP    0x0020
> >   #define PCI_DEVICE_ID_APPLE_U3_AGP       0x004b
> >   #define PCI_DEVICE_ID_APPLE_UNI_N_GMAC   0x0021
> > +#define PCI_DEVICE_ID_APPLE_VIRTIO_BLK   0x1a00
> >
> >   #define PCI_VENDOR_ID_SUN                0x108e
> >   #define PCI_DEVICE_ID_SUN_EBUS           0x1000
> > diff --git a/include/hw/virtio/virtio-blk.h
> b/include/hw/virtio/virtio-blk.h
> > index 5c14110c4b1..28d5046ea6c 100644
> > --- a/include/hw/virtio/virtio-blk.h
> > +++ b/include/hw/virtio/virtio-blk.h
> > @@ -24,7 +24,7 @@
> >   #include "qapi/qapi-types-virtio.h"
> >
> >   #define TYPE_VIRTIO_BLK "virtio-blk-device"
> > -OBJECT_DECLARE_SIMPLE_TYPE(VirtIOBlock, VIRTIO_BLK)
> > +OBJECT_DECLARE_TYPE(VirtIOBlock, VirtIOBlkClass, VIRTIO_BLK)
> >
> >   /* This is the last element of the write scatter-gather list */
> >   struct virtio_blk_inhdr
> > @@ -100,6 +100,16 @@ typedef struct MultiReqBuffer {
> >       bool is_write;
> >   } MultiReqBuffer;
> >
> > +typedef struct VirtIOBlkClass {
> > +    /*< private >*/
> > +    VirtioDeviceClass parent;
> > +    /*< public >*/
> > +    bool (*handle_unknown_request)(VirtIOBlockReq *req, MultiReqBuffer
> *mrb,
> > +                                   uint32_t type);
> > +} VirtIOBlkClass;
> > +
> >   void virtio_blk_handle_vq(VirtIOBlock *s, VirtQueue *vq);
> > +void virtio_blk_free_request(VirtIOBlockReq *req);
> > +void virtio_blk_req_complete(VirtIOBlockReq *req, unsigned char status);
> >
> >   #endif
> > diff --git a/include/hw/vmapple/virtio-blk.h
> b/include/hw/vmapple/virtio-blk.h
> > new file mode 100644
> > index 00000000000..b23106a3dfb
> > --- /dev/null
> > +++ b/include/hw/vmapple/virtio-blk.h
> > @@ -0,0 +1,39 @@
> > +/*
> > + * VMApple specific VirtIO Block implementation
> > + *
> > + * Copyright © 2023 Amazon.com, Inc. or its affiliates. All Rights
> Reserved.
> > + *
> > + * This work is licensed under the terms of the GNU GPL, version 2 or
> later.
> > + * See the COPYING file in the top-level directory.
> > + */
> > +
> > +#ifndef HW_VMAPPLE_CFG_H
> > +#define HW_VMAPPLE_CFG_H
> > +
> > +#include "hw/sysbus.h"
> > +#include "qom/object.h"
> > +#include "hw/virtio/virtio-pci.h"
> > +#include "hw/virtio/virtio-blk.h"
> > +
> > +#define TYPE_VMAPPLE_VIRTIO_BLK "vmapple-virtio-blk"
> > +#define TYPE_VMAPPLE_VIRTIO_ROOT "vmapple-virtio-root"
> > +#define TYPE_VMAPPLE_VIRTIO_AUX "vmapple-virtio-aux"
> > +
> > +OBJECT_DECLARE_TYPE(VMAppleVirtIOBlk, VMAppleVirtIOBlkClass,
> VMAPPLE_VIRTIO_BLK)
> > +
> > +typedef struct VMAppleVirtIOBlkClass {
> > +    /*< private >*/
> > +    VirtIOBlkClass parent;
> > +    /*< public >*/
> > +    void (*get_config)(VirtIODevice *vdev, uint8_t *config);
> > +} VMAppleVirtIOBlkClass;
> > +
> > +typedef struct VMAppleVirtIOBlk {
> > +    /* <private> */
> > +    VirtIOBlock parent_obj;
> > +
> > +    /* <public> */
> > +    uint32_t apple_type;
> > +} VMAppleVirtIOBlk;
> > +
> > +#endif /* HW_VMAPPLE_CFG_H */
>
>

[-- Attachment #2: Type: text/html, Size: 21225 bytes --]

Re: [PATCH v3 12/14] hw/vmapple/cfg: Introduce vmapple cfg region

[Akihiko Odaki]

On 2024/10/07 23:10, Phil Dennis-Jordan wrote:
> 
> 
> On Sat, 5 Oct 2024 at 07:35, Akihiko Odaki <akihiko.odaki@daynix.com 
> <mailto:akihiko.odaki@daynix.com>> wrote:
> 
>     On 2024/09/28 17:57, Phil Dennis-Jordan wrote:
>      > From: Alexander Graf <graf@amazon.com <mailto:graf@amazon.com>>
>      >
>      > Instead of device tree or other more standardized means, VMApple
>     passes
>      > platform configuration to the first stage boot loader in a binary
>     encoded
>      > format that resides at a dedicated RAM region in physical address
>     space.
>      >
>      > This patch models this configuration space as a qdev device which
>     we can
>      > then map at the fixed location in the address space. That way, we can
>      > influence and annotate all configuration fields easily.
>      >
>      > Signed-off-by: Alexander Graf <graf@amazon.com
>     <mailto:graf@amazon.com>>
>      > Signed-off-by: Phil Dennis-Jordan <phil@philjordan.eu
>     <mailto:phil@philjordan.eu>>
>      >
>      > ---
>      > v3:
>      >
>      >   * Replaced legacy device reset method with Resettable method
>      >
>      >   hw/vmapple/Kconfig       |   3 ++
>      >   hw/vmapple/cfg.c         | 106 ++++++++++++++++++++++++++++++++
>     +++++++
>      >   hw/vmapple/meson.build   |   1 +
>      >   include/hw/vmapple/cfg.h |  68 +++++++++++++++++++++++++
>      >   4 files changed, 178 insertions(+)
>      >   create mode 100644 hw/vmapple/cfg.c
>      >   create mode 100644 include/hw/vmapple/cfg.h
>      >
>      > diff --git a/hw/vmapple/Kconfig b/hw/vmapple/Kconfig
>      > index 68f88876eb9..8bbeb9a9237 100644
>      > --- a/hw/vmapple/Kconfig
>      > +++ b/hw/vmapple/Kconfig
>      > @@ -4,3 +4,6 @@ config VMAPPLE_AES
>      >   config VMAPPLE_BDIF
>      >       bool
>      >
>      > +config VMAPPLE_CFG
>      > +    bool
>      > +
>      > diff --git a/hw/vmapple/cfg.c b/hw/vmapple/cfg.c
>      > new file mode 100644
>      > index 00000000000..a5e5c62f59f
>      > --- /dev/null
>      > +++ b/hw/vmapple/cfg.c
>      > @@ -0,0 +1,106 @@
>      > +/*
>      > + * VMApple Configuration Region
>      > + *
>      > + * Copyright © 2023 Amazon.com, Inc. or its affiliates. All
>     Rights Reserved.
>      > + *
>      > + * This work is licensed under the terms of the GNU GPL, version
>     2 or later.
>      > + * See the COPYING file in the top-level directory.
>      > + */
>      > +
>      > +#include "qemu/osdep.h"
>      > +#include "hw/vmapple/cfg.h"
>      > +#include "qemu/log.h"
>      > +#include "qemu/module.h"
>      > +#include "qapi/error.h"
>      > +
>      > +static void vmapple_cfg_reset(Object *obj, ResetType type)
>      > +{
>      > +    VMAppleCfgState *s = VMAPPLE_CFG(obj);
>      > +    VMAppleCfg *cfg;
>      > +
>      > +    cfg = memory_region_get_ram_ptr(&s->mem);
>      > +    memset((void *)cfg, 0, VMAPPLE_CFG_SIZE);
>      > +    *cfg = s->cfg;
>       > +}> +
>      > +static void vmapple_cfg_realize(DeviceState *dev, Error **errp)
>      > +{
>      > +    VMAppleCfgState *s = VMAPPLE_CFG(dev);
>      > +    uint32_t i;
>      > +
>      > +    strncpy(s->cfg.serial, s->serial, sizeof(s->cfg.serial));
>      > +    strncpy(s->cfg.model, s->model, sizeof(s->cfg.model));
>      > +    strncpy(s->cfg.soc_name, s->soc_name, sizeof(s->cfg.soc_name));
>      > +    strncpy(s->cfg.unk8, "D/A", sizeof(s->cfg.soc_name));
> 
>     Use qemu_strnlen() to report an error for too long strings.
> 
> 
> Hmm, I don't see any existing instances of such a pattern. I do however 
> see a couple of uses of g_strlcpy in the Qemu codebase - that would be a 
> better candidate for error checked string copying, though it still 
> involves some awkward return value checks. I'm going to wrap that in a 
> helper function and macro to replace all 4 strncpy instances here. If 
> the same thing is useful elsewhere later, it can be promoted to cutils 
> or similar.

g_strlcpy() internally performs strlen(), which is worse than 
qemu_strnlen().

It is nice to have a helper function. Linux also has something similar 
called strscpy():
https://www.kernel.org/doc/html/latest/core-api/kernel-api.html#c.strscpy

> 
> (Also, I notice that last strncpy actually uses the wrong destination 
> size; my wrapper macro uses ARRAY_SIZE to avoid this mistake altogether.)
> 
>      > +    s->cfg.ecid = cpu_to_be64(s->cfg.ecid);
>      > +    s->cfg.version = 2;
>      > +    s->cfg.unk1 = 1;
>      > +    s->cfg.unk2 = 1;
>      > +    s->cfg.unk3 = 0x20;
>      > +    s->cfg.unk4 = 0;
>      > +    s->cfg.unk5 = 1;
>      > +    s->cfg.unk6 = 1;
>      > +    s->cfg.unk7 = 0;
>      > +    s->cfg.unk10 = 1;
>      > +
>      > +    g_assert(s->cfg.nr_cpus < ARRAY_SIZE(s->cfg.cpu_ids));
> 
>     Report an error instead of asserting.
> 
>      > +    for (i = 0; i < s->cfg.nr_cpus; i++) {
>      > +        s->cfg.cpu_ids[i] = i;
>      > +    }
>       > +}> +
>      > +static void vmapple_cfg_init(Object *obj)
>      > +{
>      > +    VMAppleCfgState *s = VMAPPLE_CFG(obj);
>      > +
>      > +    memory_region_init_ram(&s->mem, obj, "VMApple Config",
>     VMAPPLE_CFG_SIZE,
>      > +                           &error_fatal);
>      > +    sysbus_init_mmio(SYS_BUS_DEVICE(obj), &s->mem);
>      > +
>      > +    s->serial = (char *)"1234";
>      > +    s->model = (char *)"VM0001";
>      > +    s->soc_name = (char *)"Apple M1 (Virtual)";
> 
>     These casts are unsafe; these pointers will be freed when this
>     device is
>     freed.
> 
> 
> Good catch! The more usual pattern for default string property values 
> seems to be to fill them in _realize() (using g_strdup()) if no other 
> value was previously set, so I've applied that here for the next version 
> of the patch.
> 
> 
>      > +}
>      > +
>      > +static Property vmapple_cfg_properties[] = {
>      > +    DEFINE_PROP_UINT32("nr-cpus", VMAppleCfgState, cfg.nr_cpus, 1),
>      > +    DEFINE_PROP_UINT64("ecid", VMAppleCfgState, cfg.ecid, 0),
>      > +    DEFINE_PROP_UINT64("ram-size", VMAppleCfgState,
>     cfg.ram_size, 0),
>      > +    DEFINE_PROP_UINT32("run_installer1", VMAppleCfgState,
>     cfg.run_installer1, 0),
>      > +    DEFINE_PROP_UINT32("run_installer2", VMAppleCfgState,
>     cfg.run_installer2, 0),
>      > +    DEFINE_PROP_UINT32("rnd", VMAppleCfgState, cfg.rnd, 0),
>      > +    DEFINE_PROP_MACADDR("mac-en0", VMAppleCfgState, cfg.mac_en0),
>      > +    DEFINE_PROP_MACADDR("mac-en1", VMAppleCfgState, cfg.mac_en1),
>      > +    DEFINE_PROP_MACADDR("mac-wifi0", VMAppleCfgState,
>     cfg.mac_wifi0),
>      > +    DEFINE_PROP_MACADDR("mac-bt0", VMAppleCfgState, cfg.mac_bt0),
>      > +    DEFINE_PROP_STRING("serial", VMAppleCfgState, serial),
>      > +    DEFINE_PROP_STRING("model", VMAppleCfgState, model),
>      > +    DEFINE_PROP_STRING("soc_name", VMAppleCfgState, soc_name),
>      > +    DEFINE_PROP_END_OF_LIST(),
>      > +};
>      > +
>      > +static void vmapple_cfg_class_init(ObjectClass *klass, void *data)
>      > +{
>      > +    DeviceClass *dc = DEVICE_CLASS(klass);
>      > +    ResettableClass *rc = RESETTABLE_CLASS(klass);
>      > +
>      > +    dc->realize = vmapple_cfg_realize;
>      > +    dc->desc = "VMApple Configuration Region";
>      > +    device_class_set_props(dc, vmapple_cfg_properties);
>      > +    rc->phases.hold = vmapple_cfg_reset;
>      > +}
>      > +
>      > +static const TypeInfo vmapple_cfg_info = {
>      > +    .name          = TYPE_VMAPPLE_CFG,
>      > +    .parent        = TYPE_SYS_BUS_DEVICE,
>      > +    .instance_size = sizeof(VMAppleCfgState),
>      > +    .instance_init = vmapple_cfg_init,
>      > +    .class_init    = vmapple_cfg_class_init,
>      > +};
>      > +
>      > +static void vmapple_cfg_register_types(void)
>      > +{
>      > +    type_register_static(&vmapple_cfg_info);
>      > +}
>      > +
>      > +type_init(vmapple_cfg_register_types)
>      > diff --git a/hw/vmapple/meson.build b/hw/vmapple/meson.build
>      > index d4624713deb..64b78693a31 100644
>      > --- a/hw/vmapple/meson.build
>      > +++ b/hw/vmapple/meson.build
>      > @@ -1,2 +1,3 @@
>      >   system_ss.add(when: 'CONFIG_VMAPPLE_AES',  if_true: files('aes.c'))
>      >   system_ss.add(when: 'CONFIG_VMAPPLE_BDIF', if_true:
>     files('bdif.c'))
>      > +system_ss.add(when: 'CONFIG_VMAPPLE_CFG',  if_true: files('cfg.c'))
>      > diff --git a/include/hw/vmapple/cfg.h b/include/hw/vmapple/cfg.h
>      > new file mode 100644
>      > index 00000000000..3337064e447
>      > --- /dev/null
>      > +++ b/include/hw/vmapple/cfg.h
>      > @@ -0,0 +1,68 @@
>      > +/*
>      > + * VMApple Configuration Region
>      > + *
>      > + * Copyright © 2023 Amazon.com, Inc. or its affiliates. All
>     Rights Reserved.
>      > + *
>      > + * This work is licensed under the terms of the GNU GPL, version
>     2 or later.
>      > + * See the COPYING file in the top-level directory.
>      > + */
>      > +
>      > +#ifndef HW_VMAPPLE_CFG_H
>      > +#define HW_VMAPPLE_CFG_H
>      > +
>      > +#include "hw/sysbus.h"
>      > +#include "qom/object.h"
>      > +#include "net/net.h"
>      > +
>      > +typedef struct VMAppleCfg {
>      > +    uint32_t version;         /* 0x000 */
>      > +    uint32_t nr_cpus;         /* 0x004 */
>      > +    uint32_t unk1;            /* 0x008 */
>      > +    uint32_t unk2;            /* 0x00c */
>      > +    uint32_t unk3;            /* 0x010 */
>      > +    uint32_t unk4;            /* 0x014 */
>      > +    uint64_t ecid;            /* 0x018 */
>      > +    uint64_t ram_size;        /* 0x020 */
>      > +    uint32_t run_installer1;  /* 0x028 */
>      > +    uint32_t unk5;            /* 0x02c */
>      > +    uint32_t unk6;            /* 0x030 */
>      > +    uint32_t run_installer2;  /* 0x034 */
>      > +    uint32_t rnd;             /* 0x038 */
>      > +    uint32_t unk7;            /* 0x03c */
>      > +    MACAddr mac_en0;          /* 0x040 */
>      > +    uint8_t pad1[2];
>      > +    MACAddr mac_en1;          /* 0x048 */
>      > +    uint8_t pad2[2];
>      > +    MACAddr mac_wifi0;        /* 0x050 */
>      > +    uint8_t pad3[2];
>      > +    MACAddr mac_bt0;          /* 0x058 */
>      > +    uint8_t pad4[2];
>      > +    uint8_t reserved[0xa0];   /* 0x060 */
>      > +    uint32_t cpu_ids[0x80];   /* 0x100 */
>      > +    uint8_t scratch[0x200];   /* 0x180 */
>      > +    char serial[32];          /* 0x380 */
>      > +    char unk8[32];            /* 0x3a0 */
>      > +    char model[32];           /* 0x3c0 */
>      > +    uint8_t unk9[32];         /* 0x3e0 */
>      > +    uint32_t unk10;           /* 0x400 */
>      > +    char soc_name[32];        /* 0x404 */
>      > +} VMAppleCfg;
>      > +
>      > +#define TYPE_VMAPPLE_CFG "vmapple-cfg"
>      > +OBJECT_DECLARE_SIMPLE_TYPE(VMAppleCfgState, VMAPPLE_CFG)
>      > +
>      > +struct VMAppleCfgState {
>      > +    /* <private> */
>      > +    SysBusDevice parent_obj;
>      > +    VMAppleCfg cfg;
>      > +
>      > +    /* <public> */
>      > +    MemoryRegion mem;
>      > +    char *serial;
>      > +    char *model;
>      > +    char *soc_name;
>      > +};
>      > +
>      > +#define VMAPPLE_CFG_SIZE 0x00010000
>      > +
>      > +#endif /* HW_VMAPPLE_CFG_H */
> 

Re: [PATCH v3 13/14] hw/vmapple/virtio-blk: Add support for apple virtio-blk

[Akihiko Odaki]

On 2024/10/07 23:31, Phil Dennis-Jordan wrote:
> 
> On Sat, 5 Oct 2024 at 07:47, Akihiko Odaki <akihiko.odaki@daynix.com 
> <mailto:akihiko.odaki@daynix.com>> wrote:
> 
>     On 2024/09/28 17:57, Phil Dennis-Jordan wrote:
>      > From: Alexander Graf <graf@amazon.com <mailto:graf@amazon.com>>
>      >
>      > Apple has its own virtio-blk PCI device ID where it deviates from the
>      > official virtio-pci spec slightly: It puts a new "apple type"
>      > field at a static offset in config space and introduces a new barrier
>      > command.
>      >
>      > This patch first creates a mechanism for virtio-blk downstream
>     classes to
>      > handle unknown commands. It then creates such a downstream class
>     and a new
>      > vmapple-virtio-blk-pci class which support the additional apple
>     type config
>      > identifier as well as the barrier command.
>      >
>      > It then exposes 2 subclasses from that that we can use to expose
>     root and
>      > aux virtio-blk devices: "vmapple-virtio-root" and "vmapple-
>     virtio-aux".
>      >
>      > Signed-off-by: Alexander Graf <graf@amazon.com
>     <mailto:graf@amazon.com>>
>      > Signed-off-by: Phil Dennis-Jordan <phil@philjordan.eu
>     <mailto:phil@philjordan.eu>>
>      > ---
>      >   hw/block/virtio-blk.c           |  19 ++-
>      >   hw/vmapple/Kconfig              |   3 +
>      >   hw/vmapple/meson.build          |   1 +
>      >   hw/vmapple/virtio-blk.c         | 212 +++++++++++++++++++++++++
>     +++++++
>      >   include/hw/pci/pci_ids.h        |   1 +
>      >   include/hw/virtio/virtio-blk.h  |  12 +-
>      >   include/hw/vmapple/virtio-blk.h |  39 ++++++
>      >   7 files changed, 282 insertions(+), 5 deletions(-)
>      >   create mode 100644 hw/vmapple/virtio-blk.c
>      >   create mode 100644 include/hw/vmapple/virtio-blk.h
>      >
>      > diff --git a/hw/block/virtio-blk.c b/hw/block/virtio-blk.c
>      > index 115795392c4..cecc4cef9e4 100644
>      > --- a/hw/block/virtio-blk.c
>      > +++ b/hw/block/virtio-blk.c
>      > @@ -50,12 +50,12 @@ static void
>     virtio_blk_init_request(VirtIOBlock *s, VirtQueue *vq,
>      >       req->mr_next = NULL;
>      >   }
>      >
>      > -static void virtio_blk_free_request(VirtIOBlockReq *req)
>      > +void virtio_blk_free_request(VirtIOBlockReq *req)
>      >   {
>      >       g_free(req);
>      >   }
> 
>     This function is identical with g_free(). Perhaps it's better to remove
>     it instead of updating it.
> 
> 
> I'm not sure that's something I should be doing in such a tangential 
> patch series, and it's very much a matter of taste - other operations on 
> VirtIOBlockReq have similar, consistent naming, and this function 
> seemingly hasn't been touched for 7 years.

It is not really consistent when it comes to memory allocation. It 
simply uses virtqueue_pop() or qemu_get_virtqueue_element() without 
function wrappers to get VirtIOBlockReq allocated. It shouldn't be 
necessary for deallocation either.

git blame reveals virtio_blk_free_request() used to be more complex. 
It's more likely that just nobody had a need to remove this function.

> 
> Perhaps virtio-blk maintainer Stefan Hajnoczi could weigh in? (tagged in 
> To:)
> 
> 
>      > -static void virtio_blk_req_complete(VirtIOBlockReq *req,
>     unsigned char status)
>      > +void virtio_blk_req_complete(VirtIOBlockReq *req, unsigned char
>     status)
>      >   {
>      >       VirtIOBlock *s = req->dev;
>      >       VirtIODevice *vdev = VIRTIO_DEVICE(s);
>      > @@ -966,8 +966,18 @@ static int
>     virtio_blk_handle_request(VirtIOBlockReq *req, MultiReqBuffer *mrb)
>      >           break;
>      >       }
>      >       default:
>      > -        virtio_blk_req_complete(req, VIRTIO_BLK_S_UNSUPP);
>      > -        virtio_blk_free_request(req);
>      > +    {
>      > +        /*
>      > +         * Give subclasses a chance to handle unknown requests.
>     This way the
>      > +         * class lookup is not in the hot path.
>      > +         */
>      > +        VirtIOBlkClass *vbk = VIRTIO_BLK_GET_CLASS(s);
>      > +        if (!vbk->handle_unknown_request ||
>      > +            !vbk->handle_unknown_request(req, mrb, type)) {
>      > +            virtio_blk_req_complete(req, VIRTIO_BLK_S_UNSUPP);
>      > +            virtio_blk_free_request(req);
>      > +        }
>      > +    }
>      >       }
>      >       return 0;
>      >   }
>      > @@ -2044,6 +2054,7 @@ static const TypeInfo virtio_blk_info = {
>      >       .instance_size = sizeof(VirtIOBlock),
>      >       .instance_init = virtio_blk_instance_init,
>      >       .class_init = virtio_blk_class_init,
>      > +    .class_size = sizeof(VirtIOBlkClass),
>      >   };
>      >
>      >   static void virtio_register_types(void)
>      > diff --git a/hw/vmapple/Kconfig b/hw/vmapple/Kconfig
>      > index 8bbeb9a9237..bcd1be63e3c 100644
>      > --- a/hw/vmapple/Kconfig
>      > +++ b/hw/vmapple/Kconfig
>      > @@ -7,3 +7,6 @@ config VMAPPLE_BDIF
>      >   config VMAPPLE_CFG
>      >       bool
>      >
>      > +config VMAPPLE_VIRTIO_BLK
>      > +    bool
>      > +
>      > diff --git a/hw/vmapple/meson.build b/hw/vmapple/meson.build
>      > index 64b78693a31..bf17cf906c9 100644
>      > --- a/hw/vmapple/meson.build
>      > +++ b/hw/vmapple/meson.build
>      > @@ -1,3 +1,4 @@
>      >   system_ss.add(when: 'CONFIG_VMAPPLE_AES',  if_true: files('aes.c'))
>      >   system_ss.add(when: 'CONFIG_VMAPPLE_BDIF', if_true:
>     files('bdif.c'))
>      >   system_ss.add(when: 'CONFIG_VMAPPLE_CFG',  if_true: files('cfg.c'))
>      > +system_ss.add(when: 'CONFIG_VMAPPLE_VIRTIO_BLK',  if_true:
>     files('virtio-blk.c'))
>      > diff --git a/hw/vmapple/virtio-blk.c b/hw/vmapple/virtio-blk.c
>      > new file mode 100644
>      > index 00000000000..720eaa61a86
>      > --- /dev/null
>      > +++ b/hw/vmapple/virtio-blk.c
>      > @@ -0,0 +1,212 @@
>      > +/*
>      > + * VMApple specific VirtIO Block implementation
>      > + *
>      > + * Copyright © 2023 Amazon.com, Inc. or its affiliates. All
>     Rights Reserved.
>      > + *
>      > + * This work is licensed under the terms of the GNU GPL, version
>     2 or later.
>      > + * See the COPYING file in the top-level directory.
>      > + *
>      > + * VMApple uses almost standard VirtIO Block, but with a few key
>     differences:
>      > + *
>      > + *  - Different PCI device/vendor ID
>      > + *  - An additional "type" identifier to differentiate AUX and
>     Root volumes
>      > + *  - An additional BARRIER command
>      > + */
>      > +
>      > +#include "qemu/osdep.h"
>      > +#include "hw/vmapple/virtio-blk.h"
>      > +#include "qemu/log.h"
>      > +#include "qemu/module.h"
>      > +#include "qapi/error.h"
>      > +
>      > +#define VIRTIO_BLK_T_APPLE_BARRIER     0x10000
>      > +
>      > +#define VIRTIO_APPLE_TYPE_ROOT 1
>      > +#define VIRTIO_APPLE_TYPE_AUX  2
>      > +
>      > +static bool
>     vmapple_virtio_blk_handle_unknown_request(VirtIOBlockReq *req,
>      > +                                                     
>     MultiReqBuffer *mrb,
>      > +                                                      uint32_t type)
>      > +{
>      > +    switch (type) {
>      > +    case VIRTIO_BLK_T_APPLE_BARRIER:
>      > +        /* We ignore barriers for now. YOLO. */
> 
>     It should be LOG_UNIMP instead of a mere comment.
> 
> 
> Fixed in next patch version.
> 
>      > +        virtio_blk_req_complete(req, VIRTIO_BLK_S_OK);
>      > +        virtio_blk_free_request(req);
>      > +        return true;
>      > +    default:
>      > +        return false;
>      > +    }
>      > +}
>      > +
>      > +/*
>      > + * VMApple virtio-blk uses the same config format as normal
>     virtio, with one
>      > + * exception: It adds an "apple type" specififer at the same
>     location that
>      > + * the spec reserves for max_secure_erase_sectors. Let's hook
>     into the
>      > + * get_config code path here, run it as usual and then patch in
>     the apple type.
>      > + */
>      > +static void vmapple_virtio_blk_get_config(VirtIODevice *vdev,
>     uint8_t *config)
>      > +{
>      > +    VMAppleVirtIOBlk *dev = VMAPPLE_VIRTIO_BLK(vdev);
>      > +    VMAppleVirtIOBlkClass *vvbk = VMAPPLE_VIRTIO_BLK_GET_CLASS(dev);
>      > +    struct virtio_blk_config *blkcfg = (struct virtio_blk_config
>     *)config;
>      > +
>      > +    vvbk->get_config(vdev, config);
>      > +
>      > +    g_assert(dev->parent_obj.config_size >= endof(struct
>     virtio_blk_config, zoned));
>      > +
>      > +    /* Apple abuses the field for max_secure_erase_sectors as
>     type id */
>      > +    blkcfg->max_secure_erase_sectors = dev->apple_type;
>      > +}
>      > +
>      > +static Property vmapple_virtio_blk_properties[] = {
>      > +    DEFINE_PROP_UINT32("apple-type", VMAppleVirtIOBlk,
>     apple_type, 0),
>      > +    DEFINE_PROP_END_OF_LIST(),
>      > +};
>      > +
>      > +static void vmapple_virtio_blk_class_init(ObjectClass *klass,
>     void *data)
>      > +{
>      > +    DeviceClass *dc = DEVICE_CLASS(klass);
>      > +    VirtIOBlkClass *vbk = VIRTIO_BLK_CLASS(klass);
>      > +    VirtioDeviceClass *vdc = VIRTIO_DEVICE_CLASS(klass);
>      > +    VMAppleVirtIOBlkClass *vvbk = VMAPPLE_VIRTIO_BLK_CLASS(klass);
>      > +
>      > +    vbk->handle_unknown_request =
>     vmapple_virtio_blk_handle_unknown_request;
>      > +    vvbk->get_config = vdc->get_config;
>      > +    vdc->get_config = vmapple_virtio_blk_get_config;
>      > +    device_class_set_props(dc, vmapple_virtio_blk_properties);
>      > +}
>      > +
>      > +static const TypeInfo vmapple_virtio_blk_info = {
>      > +    .name          = TYPE_VMAPPLE_VIRTIO_BLK,
>      > +    .parent        = TYPE_VIRTIO_BLK,
>      > +    .instance_size = sizeof(VMAppleVirtIOBlk),
>      > +    .class_init    = vmapple_virtio_blk_class_init,
>      > +};
>      > +
>      > +/* PCI Devices */
>      > +
>      > +typedef struct VMAppleVirtIOBlkPCI {
>      > +    VirtIOPCIProxy parent_obj;
>      > +    VMAppleVirtIOBlk vdev;
>      > +    uint32_t apple_type;
>      > +} VMAppleVirtIOBlkPCI;
>      > +
>      > +/*
>      > + * vmapple-virtio-blk-pci: This extends VirtioPCIProxy.
>      > + */
>      > +#define TYPE_VMAPPLE_VIRTIO_BLK_PCI "vmapple-virtio-blk-pci-base"
>      > +DECLARE_INSTANCE_CHECKER(VMAppleVirtIOBlkPCI,
>     VMAPPLE_VIRTIO_BLK_PCI,
>      > +                         TYPE_VMAPPLE_VIRTIO_BLK_PCI)
>      > +
>      > +static Property vmapple_virtio_blk_pci_properties[] = {
>      > +    DEFINE_PROP_UINT32("class", VirtIOPCIProxy, class_code, 0),
>      > +    DEFINE_PROP_BIT("ioeventfd", VirtIOPCIProxy, flags,
>      > +                    VIRTIO_PCI_FLAG_USE_IOEVENTFD_BIT, true),
>      > +    DEFINE_PROP_UINT32("vectors", VirtIOPCIProxy, nvectors,
>      > +                       DEV_NVECTORS_UNSPECIFIED),
>      > +    DEFINE_PROP_END_OF_LIST(),
>      > +};
>      > +
>      > +static void vmapple_virtio_blk_pci_realize(VirtIOPCIProxy
>     *vpci_dev, Error **errp)
>      > +{
>      > +    VMAppleVirtIOBlkPCI *dev = VMAPPLE_VIRTIO_BLK_PCI(vpci_dev);
>      > +    DeviceState *vdev = DEVICE(&dev->vdev);
>      > +    VirtIOBlkConf *conf = &dev->vdev.parent_obj.conf;
>      > +
>      > +    if (conf->num_queues == VIRTIO_BLK_AUTO_NUM_QUEUES) {
>      > +        conf->num_queues = virtio_pci_optimal_num_queues(0);
>      > +    }
>      > +
>      > +    if (vpci_dev->nvectors == DEV_NVECTORS_UNSPECIFIED) {
>      > +        vpci_dev->nvectors = conf->num_queues + 1;
>      > +    }
>      > +
>      > +    /*
>      > +     * We don't support zones, but we need the additional config
>     space size.
>      > +     * Let's just expose the feature so the rest of the virtio-
>     blk logic
>      > +     * allocates enough space for us. The guest will ignore
>     zones anyway.
>      > +     */
>      > +    virtio_add_feature(&dev->vdev.parent_obj.host_features,
>     VIRTIO_BLK_F_ZONED);
>      > +    /* Propagate the apple type down to the virtio-blk device */
>      > +    qdev_prop_set_uint32(DEVICE(&dev->vdev), "apple-type", dev-
>      >apple_type);
>      > +    /* and spawn the virtio-blk device */
>      > +    qdev_realize(vdev, BUS(&vpci_dev->bus), errp);
>      > +
>      > +    /*
>      > +     * The virtio-pci machinery adjusts its vendor/device ID
>     based on whether
>      > +     * we support modern or legacy virtio. Let's patch it back
>     to the Apple
>      > +     * identifiers here.
>      > +     */
>      > +    pci_config_set_vendor_id(vpci_dev->pci_dev.config,
>     PCI_VENDOR_ID_APPLE);
>      > +    pci_config_set_device_id(vpci_dev->pci_dev.config,
>     PCI_DEVICE_ID_APPLE_VIRTIO_BLK);
>      > +}
>      > +
>      > +static void vmapple_virtio_blk_pci_class_init(ObjectClass
>     *klass, void *data)
>      > +{
>      > +    DeviceClass *dc = DEVICE_CLASS(klass);
>      > +    VirtioPCIClass *k = VIRTIO_PCI_CLASS(klass);
>      > +    PCIDeviceClass *pcidev_k = PCI_DEVICE_CLASS(klass);
>      > +
>      > +    set_bit(DEVICE_CATEGORY_STORAGE, dc->categories);
>      > +    device_class_set_props(dc, vmapple_virtio_blk_pci_properties);
>      > +    k->realize = vmapple_virtio_blk_pci_realize;
>      > +    pcidev_k->vendor_id = PCI_VENDOR_ID_APPLE;
>      > +    pcidev_k->device_id = PCI_DEVICE_ID_APPLE_VIRTIO_BLK;
>      > +    pcidev_k->revision = VIRTIO_PCI_ABI_VERSION;
>      > +    pcidev_k->class_id = PCI_CLASS_STORAGE_SCSI;
>      > +}
>      > +
>      > +static void vmapple_virtio_blk_pci_instance_init(Object *obj)
>      > +{
>      > +    VMAppleVirtIOBlkPCI *dev = VMAPPLE_VIRTIO_BLK_PCI(obj);
>      > +
>      > +    virtio_instance_init_common(obj, &dev->vdev, sizeof(dev->vdev),
>      > +                                TYPE_VMAPPLE_VIRTIO_BLK);
>      > +}
>      > +
>      > +static const VirtioPCIDeviceTypeInfo vmapple_virtio_blk_pci_info = {
>      > +    .base_name     = TYPE_VMAPPLE_VIRTIO_BLK_PCI,
>      > +    .generic_name  = "vmapple-virtio-blk-pci",
>      > +    .instance_size = sizeof(VMAppleVirtIOBlkPCI),
>      > +    .instance_init = vmapple_virtio_blk_pci_instance_init,
>      > +    .class_init    = vmapple_virtio_blk_pci_class_init,
>      > +};
>      > +
>      > +static void vmapple_virtio_root_instance_init(Object *obj)
>      > +{
>      > +    VMAppleVirtIOBlkPCI *dev = VMAPPLE_VIRTIO_BLK_PCI(obj);
>      > +
>      > +    dev->apple_type = VIRTIO_APPLE_TYPE_ROOT;
>      > +}
>      > +
>      > +static const TypeInfo vmapple_virtio_root_info = {
>      > +    .name          = TYPE_VMAPPLE_VIRTIO_ROOT,
>      > +    .parent        = "vmapple-virtio-blk-pci",
>      > +    .instance_size = sizeof(VMAppleVirtIOBlkPCI),
>      > +    .instance_init = vmapple_virtio_root_instance_init,
>      > +};
>      > +
>      > +static void vmapple_virtio_aux_instance_init(Object *obj)
>      > +{
>      > +    VMAppleVirtIOBlkPCI *dev = VMAPPLE_VIRTIO_BLK_PCI(obj);
>      > +
>      > +    dev->apple_type = VIRTIO_APPLE_TYPE_AUX;
>      > +}
>      > +
>      > +static const TypeInfo vmapple_virtio_aux_info = {
>      > +    .name          = TYPE_VMAPPLE_VIRTIO_AUX,
>      > +    .parent        = "vmapple-virtio-blk-pci",
>      > +    .instance_size = sizeof(VMAppleVirtIOBlkPCI),
>      > +    .instance_init = vmapple_virtio_aux_instance_init,
>      > +};
>      > +
>      > +static void vmapple_virtio_blk_register_types(void)
>      > +{
>      > +    type_register_static(&vmapple_virtio_blk_info);
>      > +    virtio_pci_types_register(&vmapple_virtio_blk_pci_info);
>      > +    type_register_static(&vmapple_virtio_root_info);
>      > +    type_register_static(&vmapple_virtio_aux_info);
>      > +}
>      > +
>      > +type_init(vmapple_virtio_blk_register_types)
>      > diff --git a/include/hw/pci/pci_ids.h b/include/hw/pci/pci_ids.h
>      > index f1a53fea8d6..33e2898be95 100644
>      > --- a/include/hw/pci/pci_ids.h
>      > +++ b/include/hw/pci/pci_ids.h
>      > @@ -191,6 +191,7 @@
>      >   #define PCI_DEVICE_ID_APPLE_UNI_N_AGP    0x0020
>      >   #define PCI_DEVICE_ID_APPLE_U3_AGP       0x004b
>      >   #define PCI_DEVICE_ID_APPLE_UNI_N_GMAC   0x0021
>      > +#define PCI_DEVICE_ID_APPLE_VIRTIO_BLK   0x1a00
>      >
>      >   #define PCI_VENDOR_ID_SUN                0x108e
>      >   #define PCI_DEVICE_ID_SUN_EBUS           0x1000
>      > diff --git a/include/hw/virtio/virtio-blk.h b/include/hw/virtio/
>     virtio-blk.h
>      > index 5c14110c4b1..28d5046ea6c 100644
>      > --- a/include/hw/virtio/virtio-blk.h
>      > +++ b/include/hw/virtio/virtio-blk.h
>      > @@ -24,7 +24,7 @@
>      >   #include "qapi/qapi-types-virtio.h"
>      >
>      >   #define TYPE_VIRTIO_BLK "virtio-blk-device"
>      > -OBJECT_DECLARE_SIMPLE_TYPE(VirtIOBlock, VIRTIO_BLK)
>      > +OBJECT_DECLARE_TYPE(VirtIOBlock, VirtIOBlkClass, VIRTIO_BLK)
>      >
>      >   /* This is the last element of the write scatter-gather list */
>      >   struct virtio_blk_inhdr
>      > @@ -100,6 +100,16 @@ typedef struct MultiReqBuffer {
>      >       bool is_write;
>      >   } MultiReqBuffer;
>      >
>      > +typedef struct VirtIOBlkClass {
>      > +    /*< private >*/
>      > +    VirtioDeviceClass parent;
>      > +    /*< public >*/
>      > +    bool (*handle_unknown_request)(VirtIOBlockReq *req,
>     MultiReqBuffer *mrb,
>      > +                                   uint32_t type);
>      > +} VirtIOBlkClass;
>      > +
>      >   void virtio_blk_handle_vq(VirtIOBlock *s, VirtQueue *vq);
>      > +void virtio_blk_free_request(VirtIOBlockReq *req);
>      > +void virtio_blk_req_complete(VirtIOBlockReq *req, unsigned char
>     status);
>      >
>      >   #endif
>      > diff --git a/include/hw/vmapple/virtio-blk.h b/include/hw/
>     vmapple/virtio-blk.h
>      > new file mode 100644
>      > index 00000000000..b23106a3dfb
>      > --- /dev/null
>      > +++ b/include/hw/vmapple/virtio-blk.h
>      > @@ -0,0 +1,39 @@
>      > +/*
>      > + * VMApple specific VirtIO Block implementation
>      > + *
>      > + * Copyright © 2023 Amazon.com, Inc. or its affiliates. All
>     Rights Reserved.
>      > + *
>      > + * This work is licensed under the terms of the GNU GPL, version
>     2 or later.
>      > + * See the COPYING file in the top-level directory.
>      > + */
>      > +
>      > +#ifndef HW_VMAPPLE_CFG_H
>      > +#define HW_VMAPPLE_CFG_H
>      > +
>      > +#include "hw/sysbus.h"
>      > +#include "qom/object.h"
>      > +#include "hw/virtio/virtio-pci.h"
>      > +#include "hw/virtio/virtio-blk.h"
>      > +
>      > +#define TYPE_VMAPPLE_VIRTIO_BLK "vmapple-virtio-blk"
>      > +#define TYPE_VMAPPLE_VIRTIO_ROOT "vmapple-virtio-root"
>      > +#define TYPE_VMAPPLE_VIRTIO_AUX "vmapple-virtio-aux"
>      > +
>      > +OBJECT_DECLARE_TYPE(VMAppleVirtIOBlk, VMAppleVirtIOBlkClass,
>     VMAPPLE_VIRTIO_BLK)
>      > +
>      > +typedef struct VMAppleVirtIOBlkClass {
>      > +    /*< private >*/
>      > +    VirtIOBlkClass parent;
>      > +    /*< public >*/
>      > +    void (*get_config)(VirtIODevice *vdev, uint8_t *config);
>      > +} VMAppleVirtIOBlkClass;
>      > +
>      > +typedef struct VMAppleVirtIOBlk {
>      > +    /* <private> */
>      > +    VirtIOBlock parent_obj;
>      > +
>      > +    /* <public> */
>      > +    uint32_t apple_type;
>      > +} VMAppleVirtIOBlk;
>      > +
>      > +#endif /* HW_VMAPPLE_CFG_H */
> 

Re: [PATCH v3 14/14] hw/vmapple/vmapple: Add vmapple machine type

[Phil Dennis-Jordan]

[-- Attachment #1: Type: text/plain, Size: 35356 bytes --]

On Sat, 5 Oct 2024 at 08:11, Akihiko Odaki <akihiko.odaki@daynix.com> wrote:

> On 2024/09/28 17:57, Phil Dennis-Jordan wrote:
> > From: Alexander Graf <graf@amazon.com>
> >
> > Apple defines a new "vmapple" machine type as part of its proprietary
> > macOS Virtualization.Framework vmm. This machine type is similar to the
> > virt one, but with subtle differences in base devices, a few special
> > vmapple device additions and a vastly different boot chain.
> >
> > This patch reimplements this machine type in QEMU. To use it, you
> > have to have a readily installed version of macOS for VMApple,
> > run on macOS with -accel hvf, pass the Virtualization.Framework
> > boot rom (AVPBooter) in via -bios, pass the aux and root volume as pflash
> > and pass aux and root volume as virtio drives. In addition, you also
> > need to find the machine UUID and pass that as -M vmapple,uuid=
> parameter:
> >
> > $ qemu-system-aarch64 -accel hvf -M vmapple,uuid=0x1234 -m 4G \
> >      -bios
> /System/Library/Frameworks/Virtualization.framework/Versions/A/Resources/AVPBooter.vmapple2.bin
> >      -drive file=aux,if=pflash,format=raw \
> >      -drive file=root,if=pflash,format=raw \
> >      -drive file=aux,if=none,id=aux,format=raw \
> >      -device vmapple-virtio-aux,drive=aux \
> >      -drive file=root,if=none,id=root,format=raw \
> >      -device vmapple-virtio-root,drive=root
> >
> > With all these in place, you should be able to see macOS booting
> > successfully.
> >
> > Known issues:
> >   - Keyboard and mouse/tablet input is laggy. The reason for this is
> >     either that macOS's XHCI driver is broken when the device/platform
> >     does not support MSI/MSI-X, or there's some unfortunate interplay
> >     with Qemu's XHCI implementation in this scenario.
> >   - Currently only macOS 12 guests are supported. The boot process for
> >     13+ will need further investigation and adjustment.
> >
> > Signed-off-by: Alexander Graf <graf@amazon.com>
> > Co-authored-by: Phil Dennis-Jordan <phil@philjordan.eu>
> > Signed-off-by: Phil Dennis-Jordan <phil@philjordan.eu>
> >
> > ---
> > v3:
> >   * Rebased on latest upstream, updated affinity and NIC creation
> > API usage
> >   * Included Apple-variant virtio-blk in build dependency
> >   * Updated API usage for setting 'redist-region-count' array-typed
> property on GIC.
> >   * Switched from virtio HID devices (for which macOS 12 does not
> contain drivers) to an XHCI USB controller and USB HID devices.
> >
> >   MAINTAINERS                 |   1 +
> >   docs/system/arm/vmapple.rst |  63 ++++
> >   docs/system/target-arm.rst  |   1 +
> >   hw/vmapple/Kconfig          |  20 ++
> >   hw/vmapple/meson.build      |   1 +
> >   hw/vmapple/vmapple.c        | 661 ++++++++++++++++++++++++++++++++++++
> >   6 files changed, 747 insertions(+)
> >   create mode 100644 docs/system/arm/vmapple.rst
> >   create mode 100644 hw/vmapple/vmapple.c
> >
> > diff --git a/MAINTAINERS b/MAINTAINERS
> > index 4e7f25e5299..89ef071a01a 100644
> > --- a/MAINTAINERS
> > +++ b/MAINTAINERS
> > @@ -2783,6 +2783,7 @@ R: Phil Dennis-Jordan <phil@philjordan.eu>
> >   S: Maintained
> >   F: hw/vmapple/*
> >   F: include/hw/vmapple/*
> > +F: docs/system/arm/vmapple.rst
> >
> >   Subsystems
> >   ----------
> > diff --git a/docs/system/arm/vmapple.rst b/docs/system/arm/vmapple.rst
> > new file mode 100644
> > index 00000000000..acb921ffb35
> > --- /dev/null
> > +++ b/docs/system/arm/vmapple.rst
> > @@ -0,0 +1,63 @@
> > +VMApple machine emulation
> >
> +========================================================================================
> > +
> > +VMApple is the device model that the macOS built-in hypervisor called
> "Virtualization.framework"
> > +exposes to Apple Silicon macOS guests. The "vmapple" machine model in
> QEMU implements the same
> > +device model, but does not use any code from Virtualization.Framework.
> > +
> > +Prerequisites
> > +-------------
> > +
> > +To run the vmapple machine model, you need to
> > +
> > + * Run on Apple Silicon
> > + * Run on macOS 12.0 or above
> > + * Have an already installed copy of a Virtualization.Framework macOS
> 12 virtual machine. I will
> > +   assume that you installed it using the macosvm CLI.
> > +
> > +First, we need to extract the UUID from the virtual machine that you
> installed. You can do this
> > +by running the following shell script:
> > +
> > +.. code-block:: bash
> > +  :caption: uuid.sh script to extract the UUID from a macosvm.json file
> > +
> > +  #!/bin/bash
> > +
> > +  MID=$(cat "$1" | python3 -c 'import
> json,sys;obj=json.load(sys.stdin);print(obj["machineId"]);')
> > +  echo "$MID" | base64 -d | plutil -extract ECID raw -
> > +
> > +Now we also need to trim the aux partition. It contains metadata that
> we can just discard:
> > +
> > +.. code-block:: bash
> > +  :caption: Command to trim the aux file
> > +
> > +  $ dd if="aux.img" of="aux.img.trimmed" bs=$(( 0x4000 )) skip=1
> > +
> > +How to run
> > +----------
> > +
> > +Then, we can launch QEMU with the Virtualization.Framework pre-boot
> environment and the readily
> > +installed target disk images. I recommend to port forward the VM's ssh
> and vnc ports to the host
> > +to get better interactive access into the target system:
> > +
> > +.. code-block:: bash
> > +  :caption: Example execution command line
> > +
> > +  $ UUID=$(uuid.sh macosvm.json)
> > +  $
> AVPBOOTER=/System/Library/Frameworks/Virtualization.framework/Resources/AVPBooter.vmapple2.bin
> > +  $ AUX=aux.img.trimmed
> > +  $ DISK=disk.img
> > +  $ qemu-system-aarch64 \
> > +       -serial mon:stdio \
> > +       -m 4G \
> > +       -accel hvf \
> > +       -M vmapple,uuid=$UUID \
> > +       -bios $AVPBOOTER \
> > +        -drive file="$AUX",if=pflash,format=raw \
> > +        -drive file="$DISK",if=pflash,format=raw \
> > +       -drive file="$AUX",if=none,id=aux,format=raw \
> > +       -drive file="$DISK",if=none,id=root,format=raw \
> > +       -device vmapple-virtio-aux,drive=aux \
> > +       -device vmapple-virtio-root,drive=root \
> > +       -net user,ipv6=off,hostfwd=tcp::2222-:22,hostfwd=tcp::5901-:5900
> \
> > +       -net nic,model=virtio-net-pci \
> > diff --git a/docs/system/target-arm.rst b/docs/system/target-arm.rst
> > index 7b992722846..f1948abb545 100644
> > --- a/docs/system/target-arm.rst
> > +++ b/docs/system/target-arm.rst
> > @@ -107,6 +107,7 @@ undocumented; you can get a complete list by running
> >      arm/stellaris
> >      arm/stm32
> >      arm/virt
> > +   arm/vmapple
> >      arm/xenpvh
> >      arm/xlnx-versal-virt
> >      arm/xlnx-zynq
> > diff --git a/hw/vmapple/Kconfig b/hw/vmapple/Kconfig
> > index bcd1be63e3c..0f83d4259fc 100644
> > --- a/hw/vmapple/Kconfig
> > +++ b/hw/vmapple/Kconfig
> > @@ -10,3 +10,23 @@ config VMAPPLE_CFG
> >   config VMAPPLE_VIRTIO_BLK
> >       bool
> >
> > +config VMAPPLE
> > +    bool
> > +    depends on ARM
> > +    depends on HVF
> > +    default y if ARM
> > +    imply PCI_DEVICES
> > +    select ARM_GIC
> > +    select PLATFORM_BUS
> > +    select PCI_EXPRESS
> > +    select PCI_EXPRESS_GENERIC_BRIDGE
> > +    select PL011 # UART
> > +    select PL031 # RTC
> > +    select PL061 # GPIO
> > +    select GPIO_PWR
> > +    select PVPANIC_MMIO
> > +    select VMAPPLE_AES
> > +    select VMAPPLE_BDIF
> > +    select VMAPPLE_CFG
> > +    select MAC_PVG_VMAPPLE
> > +    select VMAPPLE_VIRTIO_BLK
> > diff --git a/hw/vmapple/meson.build b/hw/vmapple/meson.build
> > index bf17cf906c9..e572f7d5602 100644
> > --- a/hw/vmapple/meson.build
> > +++ b/hw/vmapple/meson.build
> > @@ -2,3 +2,4 @@ system_ss.add(when: 'CONFIG_VMAPPLE_AES',  if_true:
> files('aes.c'))
> >   system_ss.add(when: 'CONFIG_VMAPPLE_BDIF', if_true: files('bdif.c'))
> >   system_ss.add(when: 'CONFIG_VMAPPLE_CFG',  if_true: files('cfg.c'))
> >   system_ss.add(when: 'CONFIG_VMAPPLE_VIRTIO_BLK',  if_true:
> files('virtio-blk.c'))
> > +specific_ss.add(when: 'CONFIG_VMAPPLE',     if_true: files('vmapple.c'))
> > diff --git a/hw/vmapple/vmapple.c b/hw/vmapple/vmapple.c
> > new file mode 100644
> > index 00000000000..f0060a6f7ee
> > --- /dev/null
> > +++ b/hw/vmapple/vmapple.c
> > @@ -0,0 +1,661 @@
> > +/*
> > + * VMApple machine emulation
> > + *
> > + * Copyright © 2023 Amazon.com, Inc. or its affiliates. All Rights
> Reserved.
> > + *
> > + * This work is licensed under the terms of the GNU GPL, version 2 or
> later.
> > + * See the COPYING file in the top-level directory.
> > + *
> > + * VMApple is the device model that the macOS built-in hypervisor called
> > + * "Virtualization.framework" exposes to Apple Silicon macOS guests. The
> > + * machine model in this file implements the same device model in QEMU,
> but
> > + * does not use any code from Virtualization.Framework.
> > + */
> > +
> > +#include "qemu/osdep.h"
> > +#include "qemu/help-texts.h"
> > +#include "qemu/datadir.h"
> > +#include "qemu/units.h"
> > +#include "qemu/option.h"
> > +#include "monitor/qdev.h"
> > +#include "hw/sysbus.h"
> > +#include "hw/arm/boot.h"
> > +#include "hw/arm/primecell.h"
> > +#include "hw/boards.h"
> > +#include "hw/usb.h"
> > +#include "net/net.h"
> > +#include "sysemu/sysemu.h"
> > +#include "sysemu/runstate.h"
> > +#include "sysemu/kvm.h"
> > +#include "sysemu/hvf.h"
> > +#include "hw/loader.h"
> > +#include "qapi/error.h"
> > +#include "qapi/qmp/qlist.h"
> > +#include "qemu/bitops.h"
> > +#include "qemu/error-report.h"
> > +#include "qemu/module.h"
> > +#include "hw/pci-host/gpex.h"
> > +#include "hw/virtio/virtio-pci.h"
> > +#include "hw/qdev-properties.h"
> > +#include "hw/intc/arm_gic.h"
> > +#include "hw/intc/arm_gicv3_common.h"
> > +#include "hw/irq.h"
> > +#include "hw/usb/xhci.h"
> > +#include "qapi/visitor.h"
> > +#include "qapi/qapi-visit-common.h"
> > +#include "standard-headers/linux/input.h"
> > +#include "target/arm/internals.h"
> > +#include "target/arm/kvm_arm.h"
> > +#include "hw/char/pl011.h"
> > +#include "qemu/guest-random.h"
> > +#include "sysemu/reset.h"
> > +#include "qemu/log.h"
> > +#include "hw/vmapple/cfg.h"
> > +#include "hw/misc/pvpanic.h"
> > +#include "hw/vmapple/bdif.h"
> > +
> > +struct VMAppleMachineClass {
> > +    MachineClass parent;
> > +};
> > +
> > +struct VMAppleMachineState {
> > +    MachineState parent;
> > +
> > +    Notifier machine_done;
> > +    struct arm_boot_info bootinfo;
> > +    MemMapEntry *memmap;
> > +    const int *irqmap;
> > +    DeviceState *gic;
> > +    DeviceState *cfg;
> > +    Notifier powerdown_notifier;
> > +    PCIBus *bus;
> > +    MemoryRegion fw_mr;
> > +    uint64_t uuid;
> > +};
> > +
> > +#define DEFINE_VMAPPLE_MACHINE_LATEST(major, minor, latest) \
> > +    static void vmapple##major##_##minor##_class_init(ObjectClass *oc, \
> > +                                                    void *data) \
> > +    { \
> > +        MachineClass *mc = MACHINE_CLASS(oc); \
> > +        vmapple_machine_##major##_##minor##_options(mc); \
> > +        mc->desc = "QEMU " # major "." # minor " Apple Virtual
> Machine"; \
> > +        if (latest) { \
> > +            mc->alias = "vmapple"; \
> > +        } \
> > +    } \
> > +    static const TypeInfo machvmapple##major##_##minor##_info = { \
> > +        .name = MACHINE_TYPE_NAME("vmapple-" # major "." # minor), \
> > +        .parent = TYPE_VMAPPLE_MACHINE, \
> > +        .class_init = vmapple##major##_##minor##_class_init, \
> > +    }; \
> > +    static void machvmapple_machine_##major##_##minor##_init(void) \
> > +    { \
> > +        type_register_static(&machvmapple##major##_##minor##_info); \
> > +    } \
> > +    type_init(machvmapple_machine_##major##_##minor##_init);
> > +
> > +#define DEFINE_VMAPPLE_MACHINE_AS_LATEST(major, minor) \
> > +    DEFINE_VMAPPLE_MACHINE_LATEST(major, minor, true)
> > +#define DEFINE_VMAPPLE_MACHINE(major, minor) \
> > +    DEFINE_VMAPPLE_MACHINE_LATEST(major, minor, false)
> > +
> > +#define TYPE_VMAPPLE_MACHINE   MACHINE_TYPE_NAME("vmapple")
> > +OBJECT_DECLARE_TYPE(VMAppleMachineState, VMAppleMachineClass,
> VMAPPLE_MACHINE)
> > +
> > +/* Number of external interrupt lines to configure the GIC with */
> > +#define NUM_IRQS 256
> > +
> > +enum {
> > +    VMAPPLE_FIRMWARE,
> > +    VMAPPLE_CONFIG,
> > +    VMAPPLE_MEM,
> > +    VMAPPLE_GIC_DIST,
> > +    VMAPPLE_GIC_REDIST,
> > +    VMAPPLE_UART,
> > +    VMAPPLE_RTC,
> > +    VMAPPLE_PCIE,
> > +    VMAPPLE_PCIE_MMIO,
> > +    VMAPPLE_PCIE_ECAM,
> > +    VMAPPLE_GPIO,
> > +    VMAPPLE_PVPANIC,
> > +    VMAPPLE_APV_GFX,
> > +    VMAPPLE_APV_IOSFC,
> > +    VMAPPLE_AES_1,
> > +    VMAPPLE_AES_2,
> > +    VMAPPLE_BDOOR,
> > +    VMAPPLE_MEMMAP_LAST,
> > +};
> > +
> > +static MemMapEntry memmap[] = {
> > +    [VMAPPLE_FIRMWARE] =           { 0x00100000, 0x00100000 },
> > +    [VMAPPLE_CONFIG] =             { 0x00400000, 0x00010000 },
> > +
> > +    [VMAPPLE_GIC_DIST] =           { 0x10000000, 0x00010000 },
> > +    [VMAPPLE_GIC_REDIST] =         { 0x10010000, 0x00400000 },
> > +
> > +    [VMAPPLE_UART] =               { 0x20010000, 0x00010000 },
> > +    [VMAPPLE_RTC] =                { 0x20050000, 0x00001000 },
> > +    [VMAPPLE_GPIO] =               { 0x20060000, 0x00001000 },
> > +    [VMAPPLE_PVPANIC] =            { 0x20070000, 0x00000002 },
> > +    [VMAPPLE_BDOOR] =              { 0x30000000, 0x00200000 },
> > +    [VMAPPLE_APV_GFX] =            { 0x30200000, 0x00010000 },
> > +    [VMAPPLE_APV_IOSFC] =          { 0x30210000, 0x00010000 },
> > +    [VMAPPLE_AES_1] =              { 0x30220000, 0x00004000 },
> > +    [VMAPPLE_AES_2] =              { 0x30230000, 0x00004000 },
> > +    [VMAPPLE_PCIE_ECAM] =          { 0x40000000, 0x10000000 },
> > +    [VMAPPLE_PCIE_MMIO] =          { 0x50000000, 0x1fff0000 },
> > +
> > +    /* Actual RAM size depends on configuration */
> > +    [VMAPPLE_MEM] =                { 0x70000000ULL, GiB},
> > +};
> > +
> > +static const int irqmap[] = {
> > +    [VMAPPLE_UART] = 1,
> > +    [VMAPPLE_RTC] = 2,
> > +    [VMAPPLE_GPIO] = 0x5,
> > +    [VMAPPLE_APV_IOSFC] = 0x10,
> > +    [VMAPPLE_APV_GFX] = 0x11,
> > +    [VMAPPLE_AES_1] = 0x12,
> > +    [VMAPPLE_PCIE] = 0x20,
> > +};
> > +
> > +#define GPEX_NUM_IRQS 16
> > +
> > +static void create_bdif(VMAppleMachineState *vms, MemoryRegion *mem)
> > +{
> > +    DeviceState *bdif;
> > +    SysBusDevice *bdif_sb;
> > +    DriveInfo *di_aux = drive_get(IF_PFLASH, 0, 0);
> > +    DriveInfo *di_root = drive_get(IF_PFLASH, 0, 1);
> > +
> > +    if (!di_aux) {
> > +        error_report("No AUX device found. Please specify one as pflash
> drive");
> > +        exit(1);
> > +    }
> > +
> > +    if (!di_root) {
> > +        /* Fall back to the first IF_VIRTIO device as root device */
> > +        di_root = drive_get(IF_VIRTIO, 0, 0);
> > +    }
> > +
> > +    if (!di_root) {
> > +        error_report("No root device found. Please specify one as
> virtio drive");
> > +        exit(1);
> > +    }
> > +
> > +    /* PV backdoor device */
> > +    bdif = qdev_new(TYPE_VMAPPLE_BDIF);
> > +    bdif_sb = SYS_BUS_DEVICE(bdif);
> > +    sysbus_mmio_map(bdif_sb, 0, vms->memmap[VMAPPLE_BDOOR].base);
> > +
> > +    qdev_prop_set_drive(DEVICE(bdif), "aux",
> blk_by_legacy_dinfo(di_aux));
> > +    qdev_prop_set_drive(DEVICE(bdif), "root",
> blk_by_legacy_dinfo(di_root));
> > +
> > +    sysbus_realize_and_unref(bdif_sb, &error_fatal);
> > +}
> > +
> > +static void create_pvpanic(VMAppleMachineState *vms, MemoryRegion *mem)
> > +{
> > +    SysBusDevice *cfg;
> > +
> > +    vms->cfg = qdev_new(TYPE_PVPANIC_MMIO_DEVICE);
> > +    cfg = SYS_BUS_DEVICE(vms->cfg);
> > +    sysbus_mmio_map(cfg, 0, vms->memmap[VMAPPLE_PVPANIC].base);
> > +
> > +    sysbus_realize_and_unref(cfg, &error_fatal);
> > +}
> > +
> > +static void create_cfg(VMAppleMachineState *vms, MemoryRegion *mem)
> > +{
> > +    SysBusDevice *cfg;
> > +    MachineState *machine = MACHINE(vms);
> > +    uint32_t rnd = 1;
> > +
> > +    vms->cfg = qdev_new(TYPE_VMAPPLE_CFG);
> > +    cfg = SYS_BUS_DEVICE(vms->cfg);
> > +    sysbus_mmio_map(cfg, 0, vms->memmap[VMAPPLE_CONFIG].base);
> > +
> > +    qemu_guest_getrandom_nofail(&rnd, sizeof(rnd));
> > +
> > +    qdev_prop_set_uint32(vms->cfg, "nr-cpus", machine->smp.cpus);
> > +    qdev_prop_set_uint64(vms->cfg, "ecid", vms->uuid);
> > +    qdev_prop_set_uint64(vms->cfg, "ram-size", machine->ram_size);
> > +    qdev_prop_set_uint32(vms->cfg, "rnd", rnd);
> > +
> > +    sysbus_realize_and_unref(cfg, &error_fatal);
> > +}
> > +
> > +static void create_gfx(VMAppleMachineState *vms, MemoryRegion *mem)
> > +{
> > +    int irq_gfx = vms->irqmap[VMAPPLE_APV_GFX];
> > +    int irq_iosfc = vms->irqmap[VMAPPLE_APV_IOSFC];
> > +    SysBusDevice *aes;
> > +
> > +    aes = SYS_BUS_DEVICE(qdev_new("apple-gfx-vmapple"));
> > +    sysbus_mmio_map(aes, 0, vms->memmap[VMAPPLE_APV_GFX].base);
> > +    sysbus_mmio_map(aes, 1, vms->memmap[VMAPPLE_APV_IOSFC].base);
> > +    sysbus_connect_irq(aes, 0, qdev_get_gpio_in(vms->gic, irq_gfx));
> > +    sysbus_connect_irq(aes, 1, qdev_get_gpio_in(vms->gic, irq_iosfc));
> > +    sysbus_realize_and_unref(aes, &error_fatal);
> > +}
> > +
> > +static void create_aes(VMAppleMachineState *vms, MemoryRegion *mem)
> > +{
> > +    int irq = vms->irqmap[VMAPPLE_AES_1];
> > +    SysBusDevice *aes;
> > +
> > +    aes = SYS_BUS_DEVICE(qdev_new("apple-aes"));
> > +    sysbus_mmio_map(aes, 0, vms->memmap[VMAPPLE_AES_1].base);
> > +    sysbus_mmio_map(aes, 1, vms->memmap[VMAPPLE_AES_2].base);
> > +    sysbus_connect_irq(aes, 0, qdev_get_gpio_in(vms->gic, irq));
> > +    sysbus_realize_and_unref(aes, &error_fatal);
> > +}
> > +
> > +static inline int arm_gic_ppi_index(int cpu_nr, int ppi_index)
> > +{
> > +    return NUM_IRQS + cpu_nr * GIC_INTERNAL + ppi_index;
> > +}
> > +
> > +static void create_gic(VMAppleMachineState *vms, MemoryRegion *mem)
> > +{
> > +    MachineState *ms = MACHINE(vms);
> > +    /* We create a standalone GIC */
> > +    SysBusDevice *gicbusdev;
> > +    QList *redist_region_count;
> > +    int i;
> > +    unsigned int smp_cpus = ms->smp.cpus;
> > +
> > +    vms->gic = qdev_new(gicv3_class_name());
> > +    qdev_prop_set_uint32(vms->gic, "revision", 3);
> > +    qdev_prop_set_uint32(vms->gic, "num-cpu", smp_cpus);
> > +    /*
> > +     * Note that the num-irq property counts both internal and external
> > +     * interrupts; there are always 32 of the former (mandated by GIC
> spec).
> > +     */
> > +    qdev_prop_set_uint32(vms->gic, "num-irq", NUM_IRQS + 32);
> > +
> > +    uint32_t redist0_capacity =
> > +                vms->memmap[VMAPPLE_GIC_REDIST].size /
> GICV3_REDIST_SIZE;
> > +    uint32_t redist0_count = MIN(smp_cpus, redist0_capacity);
> > +
> > +    redist_region_count = qlist_new();
> > +    qlist_append_int(redist_region_count, redist0_count);
> > +    qdev_prop_set_array(vms->gic, "redist-region-count",
> redist_region_count);
> > +
> > +    gicbusdev = SYS_BUS_DEVICE(vms->gic);
> > +    sysbus_realize_and_unref(gicbusdev, &error_fatal);
> > +    sysbus_mmio_map(gicbusdev, 0, vms->memmap[VMAPPLE_GIC_DIST].base);
> > +    sysbus_mmio_map(gicbusdev, 1, vms->memmap[VMAPPLE_GIC_REDIST].base);
> > +
> > +    /*
> > +     * Wire the outputs from each CPU's generic timer and the GICv3
> > +     * maintenance interrupt signal to the appropriate GIC PPI inputs,
> > +     * and the GIC's IRQ/FIQ/VIRQ/VFIQ interrupt outputs to the CPU's
> inputs.
> > +     */
> > +    for (i = 0; i < smp_cpus; i++) {
> > +        DeviceState *cpudev = DEVICE(qemu_get_cpu(i));
> > +
> > +        /* Map the virt timer to PPI 27 */
> > +        qdev_connect_gpio_out(cpudev, GTIMER_VIRT,
> > +                              qdev_get_gpio_in(vms->gic,
> > +                                               arm_gic_ppi_index(i,
> 27)));
> > +
> > +        /* Map the GIC IRQ and FIQ lines to CPU */
> > +        sysbus_connect_irq(gicbusdev, i, qdev_get_gpio_in(cpudev,
> ARM_CPU_IRQ));
> > +        sysbus_connect_irq(gicbusdev, i + smp_cpus,
> > +                           qdev_get_gpio_in(cpudev, ARM_CPU_FIQ));
> > +    }
> > +}
> > +
> > +static void create_uart(const VMAppleMachineState *vms, int uart,
> > +                        MemoryRegion *mem, Chardev *chr)
> > +{
> > +    hwaddr base = vms->memmap[uart].base;
> > +    int irq = vms->irqmap[uart];
> > +    DeviceState *dev = qdev_new(TYPE_PL011);
> > +    SysBusDevice *s = SYS_BUS_DEVICE(dev);
> > +
> > +    qdev_prop_set_chr(dev, "chardev", chr);
> > +    sysbus_realize_and_unref(SYS_BUS_DEVICE(dev), &error_fatal);
> > +    memory_region_add_subregion(mem, base,
> > +                                sysbus_mmio_get_region(s, 0));
> > +    sysbus_connect_irq(s, 0, qdev_get_gpio_in(vms->gic, irq));
> > +}
> > +
> > +static void create_rtc(const VMAppleMachineState *vms)
> > +{
> > +    hwaddr base = vms->memmap[VMAPPLE_RTC].base;
> > +    int irq = vms->irqmap[VMAPPLE_RTC];
> > +
> > +    sysbus_create_simple("pl031", base, qdev_get_gpio_in(vms->gic,
> irq));
> > +}
> > +
> > +static DeviceState *gpio_key_dev;
> > +static void vmapple_powerdown_req(Notifier *n, void *opaque)
> > +{
> > +    /* use gpio Pin 3 for power button event */
> > +    qemu_set_irq(qdev_get_gpio_in(gpio_key_dev, 0), 1);
> > +}
> > +
> > +static void create_gpio_devices(const VMAppleMachineState *vms, int
> gpio,
> > +                                MemoryRegion *mem)
> > +{
> > +    DeviceState *pl061_dev;
> > +    hwaddr base = vms->memmap[gpio].base;
> > +    int irq = vms->irqmap[gpio];
> > +    SysBusDevice *s;
> > +
> > +    pl061_dev = qdev_new("pl061");
> > +    /* Pull lines down to 0 if not driven by the PL061 */
> > +    qdev_prop_set_uint32(pl061_dev, "pullups", 0);
> > +    qdev_prop_set_uint32(pl061_dev, "pulldowns", 0xff);
> > +    s = SYS_BUS_DEVICE(pl061_dev);
> > +    sysbus_realize_and_unref(s, &error_fatal);
> > +    memory_region_add_subregion(mem, base, sysbus_mmio_get_region(s,
> 0));
> > +    sysbus_connect_irq(s, 0, qdev_get_gpio_in(vms->gic, irq));
> > +    gpio_key_dev = sysbus_create_simple("gpio-key", -1,
> > +                                        qdev_get_gpio_in(pl061_dev, 3));
> > +}
> > +
> > +static void vmapple_firmware_init(VMAppleMachineState *vms,
> > +                                  MemoryRegion *sysmem)
> > +{
> > +    hwaddr size = vms->memmap[VMAPPLE_FIRMWARE].size;
> > +    hwaddr base = vms->memmap[VMAPPLE_FIRMWARE].base;
> > +    const char *bios_name;
> > +    int image_size;
> > +    char *fname;
> > +
> > +    bios_name = MACHINE(vms)->firmware;
> > +    if (!bios_name) {
> > +        error_report("No firmware specified");
> > +        exit(1);
> > +    }
> > +
> > +    fname = qemu_find_file(QEMU_FILE_TYPE_BIOS, bios_name);
> > +    if (!fname) {
> > +        error_report("Could not find ROM image '%s'", bios_name);
> > +        exit(1);
> > +    }
> > +
> > +    memory_region_init_ram(&vms->fw_mr, NULL, "firmware", size, NULL);
>
> Pass: &error_fatal
>
> > +    image_size = load_image_mr(fname, &vms->fw_mr);
> > +
> > +    g_free(fname);
> > +    if (image_size < 0) {
> > +        error_report("Could not load ROM image '%s'", bios_name);
> > +        exit(1);
> > +    }
> > +
> > +    memory_region_add_subregion(get_system_memory(), base, &vms->fw_mr);
> > +}
> > +
> > +static void create_pcie(VMAppleMachineState *vms)
> > +{
> > +    hwaddr base_mmio = vms->memmap[VMAPPLE_PCIE_MMIO].base;
> > +    hwaddr size_mmio = vms->memmap[VMAPPLE_PCIE_MMIO].size;
> > +    hwaddr base_ecam = vms->memmap[VMAPPLE_PCIE_ECAM].base;
> > +    hwaddr size_ecam = vms->memmap[VMAPPLE_PCIE_ECAM].size;
> > +    int irq = vms->irqmap[VMAPPLE_PCIE];
> > +    MemoryRegion *mmio_alias;
> > +    MemoryRegion *mmio_reg;
> > +    MemoryRegion *ecam_alias;
> > +    MemoryRegion *ecam_reg;
> > +    DeviceState *dev;
> > +    int i;
> > +    PCIHostState *pci;
> > +    DeviceState *usb_controller;
> > +    USBBus *usb_bus;
> > +
> > +    dev = qdev_new(TYPE_GPEX_HOST);
> > +    qdev_prop_set_uint32(dev, "nr-irqs", GPEX_NUM_IRQS);
> > +    sysbus_realize_and_unref(SYS_BUS_DEVICE(dev), &error_fatal);
> > +
> > +    /* Map only the first size_ecam bytes of ECAM space */
> > +    ecam_alias = g_new0(MemoryRegion, 1);
> > +    ecam_reg = sysbus_mmio_get_region(SYS_BUS_DEVICE(dev), 0);
> > +    memory_region_init_alias(ecam_alias, OBJECT(dev), "pcie-ecam",
> > +                             ecam_reg, 0, size_ecam);
> > +    memory_region_add_subregion(get_system_memory(), base_ecam,
> ecam_alias);
> > +
> > +    /*
> > +     * Map the MMIO window from [0x50000000-0x7fff0000] in PCI space
> into
> > +     * system address space at [0x50000000-0x7fff0000].
> > +     */
> > +    mmio_alias = g_new0(MemoryRegion, 1);
> > +    mmio_reg = sysbus_mmio_get_region(SYS_BUS_DEVICE(dev), 1);
> > +    memory_region_init_alias(mmio_alias, OBJECT(dev), "pcie-mmio",
> > +                             mmio_reg, base_mmio, size_mmio);
> > +    memory_region_add_subregion(get_system_memory(), base_mmio,
> mmio_alias);
> > +
> > +    for (i = 0; i < GPEX_NUM_IRQS; i++) {
> > +        sysbus_connect_irq(SYS_BUS_DEVICE(dev), i,
> > +                           qdev_get_gpio_in(vms->gic, irq + i));
> > +        gpex_set_irq_num(GPEX_HOST(dev), i, irq + i);
> > +    }
> > +
> > +    pci = PCI_HOST_BRIDGE(dev);
> > +    vms->bus = pci->bus;
> > +    g_assert_nonnull(vms->bus);
> > +
> > +    while ((dev = qemu_create_nic_device("virtio-net-pci", true,
> NULL))) {
> > +        qdev_realize_and_unref(dev, BUS(vms->bus), &error_fatal);
> > +    }
> > +
> > +    usb_controller = qdev_new(TYPE_QEMU_XHCI);
> > +    qdev_realize_and_unref(usb_controller, BUS(pci->bus), &error_fatal);
> > +
> > +    usb_bus = USB_BUS(object_resolve_type_unambiguous(TYPE_USB_BUS,
> > +                                                      &error_fatal));
> > +    usb_create_simple(usb_bus, "usb-kbd");
> > +    usb_create_simple(usb_bus, "usb-tablet");
> > +}
> > +
> > +static void vmapple_reset(void *opaque)
> > +{
> > +    VMAppleMachineState *vms = opaque;
> > +    hwaddr base = vms->memmap[VMAPPLE_FIRMWARE].base;
> > +
> > +    cpu_set_pc(first_cpu, base);
> > +}
> > +
> > +static void mach_vmapple_init(MachineState *machine)
> > +{
> > +    VMAppleMachineState *vms = VMAPPLE_MACHINE(machine);
> > +    MachineClass *mc = MACHINE_GET_CLASS(machine);
> > +    const CPUArchIdList *possible_cpus;
> > +    MemoryRegion *sysmem = get_system_memory();
> > +    int n;
> > +    unsigned int smp_cpus = machine->smp.cpus;
> > +    unsigned int max_cpus = machine->smp.max_cpus;
> > +
> > +    vms->memmap = memmap;
> > +    machine->usb = true;
> > +
> > +    possible_cpus = mc->possible_cpu_arch_ids(machine);
> > +    assert(possible_cpus->len == max_cpus);
> > +    for (n = 0; n < possible_cpus->len; n++) {
> > +        Object *cpu;
> > +        CPUState *cs;
> > +
> > +        if (n >= smp_cpus) {
> > +            break;
> > +        }
> > +
> > +        cpu = object_new(possible_cpus->cpus[n].type);
> > +        object_property_set_int(cpu, "mp-affinity",
> > +                                possible_cpus->cpus[n].arch_id, NULL);
> > +
> > +        cs = CPU(cpu);
> > +        cs->cpu_index = n;
> > +
> > +        numa_cpu_pre_plug(&possible_cpus->cpus[cs->cpu_index],
> DEVICE(cpu),
> > +                          &error_fatal);
> > +
> > +        object_property_set_bool(cpu, "has_el3", false, NULL);
> > +        object_property_set_bool(cpu, "has_el2", false, NULL);
> > +        object_property_set_int(cpu, "psci-conduit",
> QEMU_PSCI_CONDUIT_HVC,
> > +                                NULL);
> > +
> > +        /* Secondary CPUs start in PSCI powered-down state */
> > +        if (n > 0) {
> > +            object_property_set_bool(cpu, "start-powered-off", true,
> NULL);
> > +        }
> > +
> > +        object_property_set_link(cpu, "memory", OBJECT(sysmem),
> &error_abort);
> > +        qdev_realize(DEVICE(cpu), NULL, &error_fatal);
> > +        object_unref(cpu);
> > +    }
> > +
> > +    memory_region_add_subregion(sysmem, vms->memmap[VMAPPLE_MEM].base,
> > +                                machine->ram);
> > +
> > +    create_gic(vms, sysmem);
> > +    create_bdif(vms, sysmem);
> > +    create_pvpanic(vms, sysmem);
> > +    create_aes(vms, sysmem);
> > +    create_gfx(vms, sysmem);
> > +    create_uart(vms, VMAPPLE_UART, sysmem, serial_hd(0));
> > +    create_rtc(vms);
> > +    create_pcie(vms);
> > +
> > +    create_gpio_devices(vms, VMAPPLE_GPIO, sysmem);
> > +
> > +    vmapple_firmware_init(vms, sysmem);
> > +    create_cfg(vms, sysmem);
> > +
> > +    /* connect powerdown request */
> > +    vms->powerdown_notifier.notify = vmapple_powerdown_req;
> > +    qemu_register_powerdown_notifier(&vms->powerdown_notifier);
> > +
> > +    vms->bootinfo.ram_size = machine->ram_size;
> > +    vms->bootinfo.board_id = -1;
> > +    vms->bootinfo.loader_start = vms->memmap[VMAPPLE_MEM].base;
> > +    vms->bootinfo.skip_dtb_autoload = true;
> > +    vms->bootinfo.firmware_loaded = true;
> > +    arm_load_kernel(ARM_CPU(first_cpu), machine, &vms->bootinfo);
> > +
> > +    qemu_register_reset(vmapple_reset, vms);
> > +}
> > +
> > +static CpuInstanceProperties
> > +vmapple_cpu_index_to_props(MachineState *ms, unsigned cpu_index)
> > +{
> > +    MachineClass *mc = MACHINE_GET_CLASS(ms);
> > +    const CPUArchIdList *possible_cpus = mc->possible_cpu_arch_ids(ms);
> > +
> > +    assert(cpu_index < possible_cpus->len);
> > +    return possible_cpus->cpus[cpu_index].props;
> > +}
> > +
> > +
> > +static int64_t vmapple_get_default_cpu_node_id(const MachineState *ms,
> int idx)
> > +{
> > +    return idx % ms->numa_state->num_nodes;
> > +}
> > +
> > +static const CPUArchIdList *vmapple_possible_cpu_arch_ids(MachineState
> *ms)
> > +{
> > +    int n;
> > +    unsigned int max_cpus = ms->smp.max_cpus;
> > +
> > +    if (ms->possible_cpus) {
> > +        assert(ms->possible_cpus->len == max_cpus);
> > +        return ms->possible_cpus;
> > +    }
> > +
> > +    ms->possible_cpus = g_malloc0(sizeof(CPUArchIdList) +
> > +                                  sizeof(CPUArchId) * max_cpus);
> > +    ms->possible_cpus->len = max_cpus;
> > +    for (n = 0; n < ms->possible_cpus->len; n++) {
> > +        ms->possible_cpus->cpus[n].type = ms->cpu_type;
> > +        ms->possible_cpus->cpus[n].arch_id =
> > +            arm_build_mp_affinity(n, GICV3_TARGETLIST_BITS);
> > +        ms->possible_cpus->cpus[n].props.has_thread_id = true;
> > +        ms->possible_cpus->cpus[n].props.thread_id = n;
> > +    }
> > +    return ms->possible_cpus;
> > +}
> > +
> > +static void vmapple_get_uuid(Object *obj, Visitor *v, const char *name,
> > +                             void *opaque, Error **errp)
> > +{
> > +    VMAppleMachineState *vms = VMAPPLE_MACHINE(obj);
> > +    uint64_t value = be64_to_cpu(vms->uuid);
> > +
> > +    visit_type_uint64(v, name, &value, errp);
> > +}
> > +
> > +static void vmapple_set_uuid(Object *obj, Visitor *v, const char *name,
> > +                             void *opaque, Error **errp)
> > +{
> > +    VMAppleMachineState *vms = VMAPPLE_MACHINE(obj);
> > +    Error *error = NULL;
> > +    uint64_t value;
> > +
> > +    visit_type_uint64(v, name, &value, &error);
> > +    if (error) {
> > +        error_propagate(errp, error);
> > +        return;
> > +    }
> > +
> > +    vms->uuid = cpu_to_be64(value);
> > +}
>
> vmapple converts the value to big endian here and vmapple-cfg converts
> it back later. What's the intention?
>

Good question. This isn't originally my code, so I could at best guess.
Removing all the conversions seems to have no negative effects in practice.
(As you'd expect.) The Cfg area is memory-mapped into the guest, but none
of the other fields are endian-adjusted, and as you say it's done twice so
there doesn't really seem to be a good reason for any conversion here.


>  > +> +static void vmapple_machine_class_init(ObjectClass *oc, void *data)
> > +{
> > +    MachineClass *mc = MACHINE_CLASS(oc);
> > +
> > +    mc->init = mach_vmapple_init;
> > +    mc->max_cpus = 32;
> > +    mc->block_default_type = IF_VIRTIO;
> > +    mc->no_cdrom = 1;
> > +    mc->pci_allow_0_address = true;
> > +    mc->minimum_page_bits = 12;
> > +    mc->possible_cpu_arch_ids = vmapple_possible_cpu_arch_ids;
> > +    mc->cpu_index_to_instance_props = vmapple_cpu_index_to_props;
> > +    if (hvf_enabled()) {
> > +        mc->default_cpu_type = ARM_CPU_TYPE_NAME("host");
> > +    } else {
> > +        mc->default_cpu_type = ARM_CPU_TYPE_NAME("max");
> > +    }
>
> Remove this conditional. VMApple only works with the host model.
>
> (I wonder if this works on KVM on Asahi Linux by the way.
> apple-gfx-vmapple won't work, but perhaps anything else may just work.)
>

I don't think macOS boots without a graphics device, and on aarch64 there's
no UEFI framebuffer or VGA fallback device driver available. In fact the
vmapple kernel doesn't even contain the driver for the physical GPUs found
on Macs, and vice versa. I don't know if the PVG guest driver supports some
kind of framebuffer-only mode similar to the UEFI driver for it, I'll leave
that for someone else to figure out. :-)


> > +    mc->get_default_cpu_node_id = vmapple_get_default_cpu_node_id;
> > +    mc->default_ram_id = "mach-vmapple.ram";
> > +
> > +    object_register_sugar_prop(TYPE_VIRTIO_PCI, "disable-legacy",
> > +                               "on", true);
> > +
> > +    object_class_property_add(oc, "uuid", "uint64", vmapple_get_uuid,
> > +                              vmapple_set_uuid, NULL, NULL);
> > +    object_class_property_set_description(oc, "uuid", "Machine UUID
> (SDOM)");
> > +}
> > +
> > +static void vmapple_instance_init(Object *obj)
> > +{
> > +    VMAppleMachineState *vms = VMAPPLE_MACHINE(obj);
> > +
> > +    vms->irqmap = irqmap;
> > +}
> > +
> > +static const TypeInfo vmapple_machine_info = {
> > +    .name          = TYPE_VMAPPLE_MACHINE,
> > +    .parent        = TYPE_MACHINE,
> > +    .abstract      = true,
> > +    .instance_size = sizeof(VMAppleMachineState),
> > +    .class_size    = sizeof(VMAppleMachineClass),
> > +    .class_init    = vmapple_machine_class_init,
> > +    .instance_init = vmapple_instance_init,
> > +};
> > +
> > +static void machvmapple_machine_init(void)
> > +{
> > +    type_register_static(&vmapple_machine_info);
> > +}
> > +type_init(machvmapple_machine_init);
> > +
> > +static void vmapple_machine_8_1_options(MachineClass *mc)
> > +{
> > +}
> > +DEFINE_VMAPPLE_MACHINE_AS_LATEST(8, 1)
>
> Please update this.
>
> > +
>
>

[-- Attachment #2: Type: text/html, Size: 43907 bytes --]

Re: [PATCH v3 10/14] hw/vmapple/aes: Introduce aes engine

[Phil Dennis-Jordan]

[-- Attachment #1: Type: text/plain, Size: 22864 bytes --]

On Fri, 4 Oct 2024 at 07:32, Akihiko Odaki <akihiko.odaki@daynix.com> wrote:

> On 2024/09/28 17:57, Phil Dennis-Jordan wrote:
> > From: Alexander Graf <graf@amazon.com>
> >
> > VMApple contains an "aes" engine device that it uses to encrypt and
> > decrypt its nvram. It has trivial hard coded keys it uses for that
> > purpose.
> >
> > Add device emulation for this device model.
> >
> > Signed-off-by: Alexander Graf <graf@amazon.com>
> > Co-authored-by: Phil Dennis-Jordan <phil@philjordan.eu>
> > Signed-off-by: Phil Dennis-Jordan <phil@philjordan.eu>
> >
> > ---
> > v3:
> >
> >   * Rebased on latest upstream and fixed minor breakages.
> >   * Replaced legacy device reset method with Resettable method
> >
> >   hw/vmapple/Kconfig      |   2 +
> >   hw/vmapple/aes.c        | 584 ++++++++++++++++++++++++++++++++++++++++
> >   hw/vmapple/meson.build  |   1 +
> >   hw/vmapple/trace-events |  19 ++
> >   4 files changed, 606 insertions(+)
> >   create mode 100644 hw/vmapple/aes.c
> >
> > diff --git a/hw/vmapple/Kconfig b/hw/vmapple/Kconfig
> > index 8b137891791..a73504d5999 100644
> > --- a/hw/vmapple/Kconfig
> > +++ b/hw/vmapple/Kconfig
> > @@ -1 +1,3 @@
> > +config VMAPPLE_AES
> > +    bool
> >
> > diff --git a/hw/vmapple/aes.c b/hw/vmapple/aes.c
> > new file mode 100644
> > index 00000000000..074fbdd9c36
> > --- /dev/null
> > +++ b/hw/vmapple/aes.c
> > @@ -0,0 +1,584 @@
> > +/*
> > + * QEMU Apple AES device emulation
> > + *
> > + * Copyright © 2023 Amazon.com, Inc. or its affiliates. All Rights
> Reserved.
> > + *
> > + * This work is licensed under the terms of the GNU GPL, version 2 or
> later.
> > + * See the COPYING file in the top-level directory.
> > + */
> > +
> > +#include "qemu/osdep.h"
> > +#include "hw/irq.h"
> > +#include "migration/vmstate.h"
> > +#include "qemu/log.h"
> > +#include "qemu/module.h"
> > +#include "trace.h"
> > +#include "hw/sysbus.h"
> > +#include "crypto/hash.h"
> > +#include "crypto/aes.h"
> > +#include "crypto/cipher.h"
> > +
> > +#define TYPE_AES          "apple-aes"
> > +#define MAX_FIFO_SIZE     9
> > +
> > +#define CMD_KEY           0x1
> > +#define CMD_KEY_CONTEXT_SHIFT    27
> > +#define CMD_KEY_CONTEXT_MASK     (0x1 << CMD_KEY_CONTEXT_SHIFT)
> > +#define CMD_KEY_SELECT_SHIFT     24
> > +#define CMD_KEY_SELECT_MASK      (0x7 << CMD_KEY_SELECT_SHIFT)
> > +#define CMD_KEY_KEY_LEN_SHIFT    22
> > +#define CMD_KEY_KEY_LEN_MASK     (0x3 << CMD_KEY_KEY_LEN_SHIFT)
> > +#define CMD_KEY_ENCRYPT_SHIFT    20
> > +#define CMD_KEY_ENCRYPT_MASK     (0x1 << CMD_KEY_ENCRYPT_SHIFT)
> > +#define CMD_KEY_BLOCK_MODE_SHIFT 16
> > +#define CMD_KEY_BLOCK_MODE_MASK  (0x3 << CMD_KEY_BLOCK_MODE_SHIFT)
> > +#define CMD_IV            0x2
> > +#define CMD_IV_CONTEXT_SHIFT     26
> > +#define CMD_IV_CONTEXT_MASK      (0x3 << CMD_KEY_CONTEXT_SHIFT)
> > +#define CMD_DSB           0x3
> > +#define CMD_SKG           0x4
> > +#define CMD_DATA          0x5
> > +#define CMD_DATA_KEY_CTX_SHIFT   27
> > +#define CMD_DATA_KEY_CTX_MASK    (0x1 << CMD_DATA_KEY_CTX_SHIFT)
> > +#define CMD_DATA_IV_CTX_SHIFT    25
> > +#define CMD_DATA_IV_CTX_MASK     (0x3 << CMD_DATA_IV_CTX_SHIFT)
> > +#define CMD_DATA_LEN_MASK        0xffffff
> > +#define CMD_STORE_IV      0x6
> > +#define CMD_STORE_IV_ADDR_MASK   0xffffff
> > +#define CMD_WRITE_REG     0x7
> > +#define CMD_FLAG          0x8
> > +#define CMD_FLAG_STOP_MASK       BIT(26)
> > +#define CMD_FLAG_RAISE_IRQ_MASK  BIT(27)
> > +#define CMD_FLAG_INFO_MASK       0xff
> > +#define CMD_MAX           0x10
> > +
> > +#define CMD_SHIFT         28
> > +
> > +#define REG_STATUS            0xc
> > +#define REG_STATUS_DMA_READ_RUNNING     BIT(0)
> > +#define REG_STATUS_DMA_READ_PENDING     BIT(1)
> > +#define REG_STATUS_DMA_WRITE_RUNNING    BIT(2)
> > +#define REG_STATUS_DMA_WRITE_PENDING    BIT(3)
> > +#define REG_STATUS_BUSY                 BIT(4)
> > +#define REG_STATUS_EXECUTING            BIT(5)
> > +#define REG_STATUS_READY                BIT(6)
> > +#define REG_STATUS_TEXT_DPA_SEEDED      BIT(7)
> > +#define REG_STATUS_UNWRAP_DPA_SEEDED    BIT(8)
> > +
> > +#define REG_IRQ_STATUS        0x18
> > +#define REG_IRQ_STATUS_INVALID_CMD      BIT(2)
> > +#define REG_IRQ_STATUS_FLAG             BIT(5)
> > +#define REG_IRQ_ENABLE        0x1c
> > +#define REG_WATERMARK         0x20
> > +#define REG_Q_STATUS          0x24
> > +#define REG_FLAG_INFO         0x30
> > +#define REG_FIFO              0x200
> > +
> > +static const uint32_t key_lens[4] = {
>
> I suggest deriving the size from CMD_KEY_KEY_LEN_MASK.
>
> > +    [0] = 16,
> > +    [1] = 24,
> > +    [2] = 32,
> > +    [3] = 64,
> > +};
> > +
> > +struct key {
>
> Please add typedef.
>
> > +    uint32_t key_len;
> > +    uint32_t key[8];
> > +};
> > +
> > +struct iv {
> > +    uint32_t iv[4];
> > +};
> > +
> > +struct context {
> > +    struct key key;
> > +    struct iv iv;
> > +};
> > +
> > +static struct key builtin_keys[7] = {
>  > +    [1] = {> +        .key_len = 32,
> > +        .key = { 0x1 },
> > +    },
> > +    [2] = {
> > +        .key_len = 32,
> > +        .key = { 0x2 },
> > +    },
> > +    [3] = {
> > +        .key_len = 32,
> > +        .key = { 0x3 },
> > +    }
> > +};
> > +
> > +typedef struct AESState {
> > +    /* Private */
>
> This private/public comments are unnecessary; this struct can be
> referenced only from this file and all members are automatically private.
>
> > +    SysBusDevice parent_obj;
> > +
> > +    /* Public */
> > +    qemu_irq irq;
> > +    MemoryRegion iomem1;
> > +    MemoryRegion iomem2;
> > +
> > +    uint32_t status;
> > +    uint32_t q_status;
> > +    uint32_t irq_status;
> > +    uint32_t irq_enable;
> > +    uint32_t watermark;
> > +    uint32_t flag_info;
> > +    uint32_t fifo[MAX_FIFO_SIZE];
> > +    uint32_t fifo_idx;
> > +    struct key key[2];
> > +    struct iv iv[4];
> > +    bool is_encrypt;
> > +    QCryptoCipherMode block_mode;
> > +} AESState;
> > +
> > +OBJECT_DECLARE_SIMPLE_TYPE(AESState, AES)
> > +
> > +static void aes_update_irq(AESState *s)
> > +{
> > +    qemu_set_irq(s->irq, !!(s->irq_status & s->irq_enable));
> > +}
> > +
> > +static uint64_t aes1_read(void *opaque, hwaddr offset, unsigned size)
> > +{
> > +    AESState *s = opaque;
> > +    uint64_t res = 0;
> > +
> > +    switch (offset) {
> > +    case REG_STATUS:
> > +        res = s->status;
> > +        break;
> > +    case REG_IRQ_STATUS:
> > +        res = s->irq_status;
> > +        break;
> > +    case REG_IRQ_ENABLE:
> > +        res = s->irq_enable;
> > +        break;
> > +    case REG_WATERMARK:
> > +        res = s->watermark;
> > +        break;
> > +    case REG_Q_STATUS:
> > +        res = s->q_status;
> > +        break;
> > +    case REG_FLAG_INFO:
> > +        res = s->flag_info;
> > +        break;
> > +
> > +    default:
> > +        trace_aes_read_unknown(offset);
>
> Use: LOG_UNIMP
>
> > +        break;
> > +    }
> > +
> > +    trace_aes_read(offset, res);
> > +
> > +    return res;
> > +}
> > +
> > +static void fifo_append(AESState *s, uint64_t val)
> > +{
> > +    if (s->fifo_idx == MAX_FIFO_SIZE) {
> > +        /* Exceeded the FIFO. Bail out */
> > +        return;
> > +    }
> > +
> > +    s->fifo[s->fifo_idx++] = val;
> > +}
> > +
> > +static bool has_payload(AESState *s, uint32_t elems)
> > +{
> > +    return s->fifo_idx >= (elems + 1);
> > +}
> > +
> > +static bool cmd_key(AESState *s)
> > +{
> > +    uint32_t cmd = s->fifo[0];
> > +    uint32_t key_select = (cmd & CMD_KEY_SELECT_MASK) >>
> CMD_KEY_SELECT_SHIFT;
> > +    uint32_t ctxt = (cmd & CMD_KEY_CONTEXT_MASK) >>
> CMD_KEY_CONTEXT_SHIFT;
> > +    uint32_t key_len;
> > +
> > +    switch ((cmd & CMD_KEY_BLOCK_MODE_MASK) >>
> CMD_KEY_BLOCK_MODE_SHIFT) {
> > +    case 0:
> > +        s->block_mode = QCRYPTO_CIPHER_MODE_ECB;
> > +        break;
> > +    case 1:
> > +        s->block_mode = QCRYPTO_CIPHER_MODE_CBC;
> > +        break;
> > +    default:
> > +        return false;
> > +    }
> > +
> > +    s->is_encrypt = !!((cmd & CMD_KEY_ENCRYPT_MASK) >>
> CMD_KEY_ENCRYPT_SHIFT);
>
> CMD_KEY_ENCRYPT_SHIFT is unnecessary as in case of
> CMD_FLAG_RAISE_IRQ_MASK. "!!" is also unnecessary; it will be implicitly
> casted into bool.
>
> > +    key_len = key_lens[((cmd & CMD_KEY_KEY_LEN_MASK) >>
> CMD_KEY_KEY_LEN_SHIFT)];
> > +
> > +    if (key_select) {
> > +        trace_aes_cmd_key_select_builtin(ctxt, key_select,
> > +                                         s->is_encrypt ? "en" : "de",
> > +
>  QCryptoCipherMode_str(s->block_mode));
> > +        s->key[ctxt] = builtin_keys[key_select];
>
> I guess this should be: builtin_keys[key_select - 1]
>

I don't think it should: the builtin keys start at index 1. This seems
fine, and I'd rather leave the indices alone to avoid needing to do
arithmetic on them. However in that case, builtin_keys needs to have 8
elements not 7 as it currently does to avoid overflow. I'll fix this and
use symbolic constants.

(Thanks for the other review comments, I've reworked those parts for v4.)


> > +    } else {
> > +        trace_aes_cmd_key_select_new(ctxt, key_len,
> > +                                     s->is_encrypt ? "en" : "de",
> > +
>  QCryptoCipherMode_str(s->block_mode));
> > +        if (key_len > sizeof(s->key[ctxt].key)) {
> > +            return false;
> > +        }
> > +        if (!has_payload(s, key_len / sizeof(uint32_t))) {
> > +            /* wait for payload */
> > +            return false;
> > +        }
> > +        memcpy(&s->key[ctxt].key, &s->fifo[1], key_len);
> > +        s->key[ctxt].key_len = key_len;
> > +    }
> > +
> > +    return true;
> > +}
> > +
> > +static bool cmd_iv(AESState *s)
> > +{
> > +    uint32_t cmd = s->fifo[0];
> > +    uint32_t ctxt = (cmd & CMD_IV_CONTEXT_MASK) >> CMD_IV_CONTEXT_SHIFT;
> > +
> > +    if (!has_payload(s, 4)) {
> > +        /* wait for payload */
> > +        return false;
> > +    }
> > +    memcpy(&s->iv[ctxt].iv, &s->fifo[1], sizeof(s->iv[ctxt].iv));
> > +    trace_aes_cmd_iv(ctxt, s->fifo[1], s->fifo[2], s->fifo[3],
> s->fifo[4]);
> > +
> > +    return true;
> > +}
> > +
> > +static char hexdigit2str(uint8_t val)
> > +{
> > +    g_assert(val < 0x10);
> > +    if (val >= 0xa) {
> > +        return 'a' + (val - 0xa);
> > +    } else {
> > +        return '0' + val;
> > +    }
> > +}
> > +
> > +static void dump_data(const char *desc, const void *p, size_t len)
> > +{
> > +    char *hex = alloca((len * 2) + 1);
>
> Don't use alloca(); use a fixed-length array instead.
>
> > +    const uint8_t *data = p;
> > +    char *hexp = hex;
> > +    size_t i;
> > +
> > +    if (len > 0x1000) {
> > +        /* Too large buffer, let's bail out */
> > +        return;
> > +    }
> > +
> > +    for (i = 0; i < len; i++) {
> > +        uint8_t val = data[i];
> > +        *(hexp++) = hexdigit2str(val >> 4);
> > +        *(hexp++) = hexdigit2str(val & 0xf);
> > +    }
> > +    *hexp = '\0';
>
> Let's move this to: util/hexdump.c
>
> > +
> > +    trace_aes_dump_data(desc, hex);
> > +}
> > +
> > +static bool cmd_data(AESState *s)
> > +{
> > +    uint32_t cmd = s->fifo[0];
> > +    uint32_t ctxt_iv = 0;
> > +    uint32_t ctxt_key = (cmd & CMD_DATA_KEY_CTX_MASK) >>
> CMD_DATA_KEY_CTX_SHIFT;
> > +    uint32_t len = cmd & CMD_DATA_LEN_MASK;
> > +    uint64_t src_addr = s->fifo[2];
> > +    uint64_t dst_addr = s->fifo[3];
> > +    QCryptoCipherAlgo alg;
> > +    QCryptoCipher *cipher;
> > +    char *src;
> > +    char *dst;
> > +
> > +    src_addr |= ((uint64_t)s->fifo[1] << 16) & 0xffff00000000ULL;
> > +    dst_addr |= ((uint64_t)s->fifo[1] << 32) & 0xffff00000000ULL;
> > +
> > +    trace_aes_cmd_data(ctxt_key, ctxt_iv, src_addr, dst_addr, len);
> > +
> > +    if (!has_payload(s, 3)) {
> > +        /* wait for payload */
> > +        trace_aes_cmd_data_error("No payload");
>
> Use: LOG_GUEST_ERROR
>
> > +        return false;
> > +    }
> > +
> > +    if (ctxt_key >= ARRAY_SIZE(s->key) ||
> > +        ctxt_iv >= ARRAY_SIZE(s->iv)) {
> > +        /* Invalid input */
> > +        trace_aes_cmd_data_error("Invalid key or iv");
> > +        return false;
> > +    }
> > +
> > +    src = g_malloc0(len);
> > +    dst = g_malloc0(len);
> > +
> > +    cpu_physical_memory_read(src_addr, src, len);
>
> Use: dma_memory_read()
>
> > +
> > +    dump_data("cmd_data(): src_data=", src, len);
> > +
> > +    switch (s->key[ctxt_key].key_len) {
> > +    case 128 / 8:
> > +        alg = QCRYPTO_CIPHER_ALGO_AES_128;
> > +        break;
> > +    case 192 / 8:
> > +        alg = QCRYPTO_CIPHER_ALGO_AES_192;
> > +        break;
> > +    case 256 / 8:
> > +        alg = QCRYPTO_CIPHER_ALGO_AES_256;
> > +        break;
> > +    default:
> > +        trace_aes_cmd_data_error("Invalid key len");
> > +        goto err_free;
>
> Use g_autoptr instead of goto and g_free().
>
> > +    }
> > +    cipher = qcrypto_cipher_new(alg, s->block_mode,
> > +                                (void *)s->key[ctxt_key].key,
> > +                                s->key[ctxt_key].key_len, NULL);
> > +    g_assert(cipher != NULL);
> Handle this error as you do for qcrypto_cipher_setiv(),
> qcrypto_cipher_encrypt() and qcrypto_cipher_decrypt().
>
> > +    if (s->block_mode != QCRYPTO_CIPHER_MODE_ECB) {
> > +        if (qcrypto_cipher_setiv(cipher, (void *)s->iv[ctxt_iv].iv,
> > +                                 sizeof(s->iv[ctxt_iv].iv), NULL) != 0)
> {
> > +            trace_aes_cmd_data_error("Failed to set IV");
> > +            goto err_free_cipher;
> > +        }
> > +    }
> > +    if (s->is_encrypt) {
> > +        if (qcrypto_cipher_encrypt(cipher, src, dst, len, NULL) != 0) {
> > +            trace_aes_cmd_data_error("Encrypt failed");
> > +            goto err_free_cipher;
> > +        }
> > +    } else {
> > +        if (qcrypto_cipher_decrypt(cipher, src, dst, len, NULL) != 0) {
> > +            trace_aes_cmd_data_error("Decrypt failed");
> > +            goto err_free_cipher;
> > +        }
> > +    }
> > +    qcrypto_cipher_free(cipher);
> > +
> > +    dump_data("cmd_data(): dst_data=", dst, len);
> > +    cpu_physical_memory_write(dst_addr, dst, len);
> > +    g_free(src);
> > +    g_free(dst);
> > +
> > +    return true;
> > +
> > +err_free_cipher:
> > +    qcrypto_cipher_free(cipher);
> > +err_free:
> > +    g_free(src);
> > +    g_free(dst);
> > +    return false;
> > +}
> > +
> > +static bool cmd_store_iv(AESState *s)
> > +{
> > +    uint32_t cmd = s->fifo[0];
> > +    uint32_t ctxt = (cmd & CMD_IV_CONTEXT_MASK) >> CMD_IV_CONTEXT_SHIFT;
> > +    uint64_t addr = s->fifo[1];
> > +
> > +    if (!has_payload(s, 1)) {
> > +        /* wait for payload */
>
> Add a log as it's done for cmd_data().
>
> > +        return false;
> > +    }
> > +
> > +    if (ctxt >= ARRAY_SIZE(s->iv)) {
> > +        /* Invalid context selected */
> > +        return false;
> > +    }
> > +
> > +    addr |= ((uint64_t)cmd << 32) & 0xff00000000ULL;
> > +    cpu_physical_memory_write(addr, &s->iv[ctxt].iv,
> sizeof(s->iv[ctxt].iv));
> > +
> > +    trace_aes_cmd_store_iv(ctxt, addr, s->iv[ctxt].iv[0],
> s->iv[ctxt].iv[1],
> > +                           s->iv[ctxt].iv[2], s->iv[ctxt].iv[3]);
> > +
> > +    return true;
> > +}
> > +
> > +static bool cmd_flag(AESState *s)
> > +{
> > +    uint32_t cmd = s->fifo[0];
> > +    uint32_t raise_irq = cmd & CMD_FLAG_RAISE_IRQ_MASK;
> > +
> > +    /* We always process data when it's coming in, so fire an IRQ
> immediately */
> > +    if (raise_irq) {
> > +        s->irq_status |= REG_IRQ_STATUS_FLAG;
> > +    }
> > +
> > +    s->flag_info = cmd & CMD_FLAG_INFO_MASK;
> > +
> > +    trace_aes_cmd_flag(!!raise_irq, s->flag_info);
> > +
> > +    return true;
> > +}
> > +
> > +static void fifo_process(AESState *s)
> > +{
> > +    uint32_t cmd = s->fifo[0] >> CMD_SHIFT;
> > +    bool success = false;
> > +
> > +    if (!s->fifo_idx) {
> > +        return;
> > +    }
> > +
> > +    switch (cmd) {
> > +    case CMD_KEY:
> > +        success = cmd_key(s);
> > +        break;
> > +    case CMD_IV:
> > +        success = cmd_iv(s);
> > +        break;
> > +    case CMD_DATA:
> > +        success = cmd_data(s);
> > +        break;
> > +    case CMD_STORE_IV:
> > +        success = cmd_store_iv(s);
> > +        break;
> > +    case CMD_FLAG:
> > +        success = cmd_flag(s);
> > +        break;
> > +    default:
> > +        s->irq_status |= REG_IRQ_STATUS_INVALID_CMD;
> > +        break;
> > +    }
> > +
> > +    if (success) {
> > +        s->fifo_idx = 0;
> > +    }
> > +
> > +    trace_aes_fifo_process(cmd, success ? 1 : 0);
> > +}
> > +
> > +static void aes1_write(void *opaque, hwaddr offset, uint64_t val,
> unsigned size)
> > +{
> > +    AESState *s = opaque;
> > +
> > +    trace_aes_write(offset, val);
> > +
> > +    switch (offset) {
> > +    case REG_IRQ_STATUS:
> > +        s->irq_status &= ~val;
> > +        break;
> > +    case REG_IRQ_ENABLE:
> > +        s->irq_enable = val;
> > +        break;
> > +    case REG_FIFO:
> > +        fifo_append(s, val);
> > +        fifo_process(s);
> > +        break;
> > +    default:
> > +        trace_aes_write_unknown(offset);
> > +        return;
> > +    }
> > +
> > +    aes_update_irq(s);
> > +}
> > +
> > +static const MemoryRegionOps aes1_ops = {
> > +    .read = aes1_read,
> > +    .write = aes1_write,
> > +    .endianness = DEVICE_NATIVE_ENDIAN,
> > +    .valid = {
> > +        .min_access_size = 4,
> > +        .max_access_size = 8,
> > +    },
> > +    .impl = {
> > +        .min_access_size = 4,
> > +        .max_access_size = 4,
> > +    },
> > +};
> > +
> > +static uint64_t aes2_read(void *opaque, hwaddr offset, unsigned size)
> > +{
> > +    uint64_t res = 0;
> > +
> > +    switch (offset) {
> > +    case 0:
> > +        res = 0;
> > +        break;
> > +    default:
> > +        trace_aes_2_read_unknown(offset);
> > +        break;
> > +    }
> > +
> > +    trace_aes_2_read(offset, res);
> > +
> > +    return res;
> > +}
> > +
> > +static void aes2_write(void *opaque, hwaddr offset, uint64_t val,
> unsigned size)
> > +{
> > +    trace_aes_2_write(offset, val);
> > +
> > +    switch (offset) {
> > +    default:
> > +        trace_aes_2_write_unknown(offset);
> > +        return;
> > +    }
> > +}
> > +
> > +static const MemoryRegionOps aes2_ops = {
> > +    .read = aes2_read,
> > +    .write = aes2_write,
> > +    .endianness = DEVICE_NATIVE_ENDIAN,
> > +    .valid = {
> > +        .min_access_size = 4,
> > +        .max_access_size = 8,
> > +    },
> > +    .impl = {
> > +        .min_access_size = 4,
> > +        .max_access_size = 4,
> > +    },
> > +};
> > +
> > +static void aes_reset(Object *obj, ResetType type)
> > +{
> > +    AESState *s = AES(obj);
> > +
> > +    s->status = 0x3f80;
> > +    s->q_status = 2;
> > +    s->irq_status = 0;
> > +    s->irq_enable = 0;
> > +    s->watermark = 0;
> > +}
> > +
> > +static void aes_init(Object *obj)
> > +{
> > +    AESState *s = AES(obj);
> > +
> > +    memory_region_init_io(&s->iomem1, obj, &aes1_ops, s, TYPE_AES,
> 0x4000);
> > +    memory_region_init_io(&s->iomem2, obj, &aes2_ops, s, TYPE_AES,
> 0x4000);
> > +    sysbus_init_mmio(SYS_BUS_DEVICE(s), &s->iomem1);
> > +    sysbus_init_mmio(SYS_BUS_DEVICE(s), &s->iomem2);
> > +    sysbus_init_irq(SYS_BUS_DEVICE(s), &s->irq);
> > +}
> > +
> > +static void aes_realize(DeviceState *dev, Error **errp)
> > +{
> > +}
>
> This shouldn't be necessary.
>
> > +
> > +static void aes_class_init(ObjectClass *klass, void *data)
> > +{
> > +    DeviceClass *dc = DEVICE_CLASS(klass);
> > +    ResettableClass *rc = RESETTABLE_CLASS(klass);
> > +
> > +    rc->phases.hold = aes_reset;
> > +    dc->realize = aes_realize;
> > +}
> > +
> > +static const TypeInfo aes_info = {
> > +    .name          = TYPE_AES,
> > +    .parent        = TYPE_SYS_BUS_DEVICE,
> > +    .instance_size = sizeof(AESState),
> > +    .class_init    = aes_class_init,
> > +    .instance_init = aes_init,
> > +};
> > +
> > +static void aes_register_types(void)
> > +{
> > +    type_register_static(&aes_info);
> > +}
> > +
> > +type_init(aes_register_types)
> > diff --git a/hw/vmapple/meson.build b/hw/vmapple/meson.build
> > index e69de29bb2d..bcd4dcb28d2 100644
> > --- a/hw/vmapple/meson.build
> > +++ b/hw/vmapple/meson.build
> > @@ -0,0 +1 @@
> > +system_ss.add(when: 'CONFIG_VMAPPLE_AES',  if_true: files('aes.c'))
> > diff --git a/hw/vmapple/trace-events b/hw/vmapple/trace-events
> > index 9ccc5790487..1c9a3326eb4 100644
> > --- a/hw/vmapple/trace-events
> > +++ b/hw/vmapple/trace-events
> > @@ -1,2 +1,21 @@
> >   # See docs/devel/tracing.rst for syntax documentation.
> >
> > +# aes.c
> > +aes_read_unknown(uint64_t offset) "offset=0x%"PRIx64
> > +aes_read(uint64_t offset, uint64_t res) "offset=0x%"PRIx64"
> res=0x%"PRIx64
> > +aes_cmd_key_select_builtin(uint32_t ctx, uint32_t key_id, const char
> *direction, const char *cipher) "[%d] Selecting builtin key %d to %scrypt
> with %s"
> > +aes_cmd_key_select_new(uint32_t ctx, uint32_t key_len, const char
> *direction, const char *cipher) "[%d] Selecting new key size=%d to %scrypt
> with %s"
> > +aes_cmd_iv(uint32_t ctx, uint32_t iv0, uint32_t iv1, uint32_t iv2,
> uint32_t iv3) "[%d] 0x%08x 0x%08x 0x%08x 0x%08x"
> > +aes_cmd_data(uint32_t key, uint32_t iv, uint64_t src, uint64_t dst,
> uint32_t len) "[key=%d iv=%d] src=0x%"PRIx64" dst=0x%"PRIx64" len=0x%x"
> > +aes_cmd_data_error(const char *reason) "reason=%s"
> > +aes_cmd_store_iv(uint32_t ctx, uint64_t addr, uint32_t iv0, uint32_t
> iv1, uint32_t iv2, uint32_t iv3) "[%d] addr=0x%"PRIx64"x -> 0x%08x 0x%08x
> 0x%08x 0x%08x"
> > +aes_cmd_flag(uint32_t raise, uint32_t flag_info) "raise=%d
> flag_info=0x%x"
> > +aes_fifo_process(uint32_t cmd, uint32_t success) "cmd=%d success=%d"
> > +aes_write_unknown(uint64_t offset) "offset=0x%"PRIx64
> > +aes_write(uint64_t offset, uint64_t val) "offset=0x%"PRIx64"
> val=0x%"PRIx64
> > +aes_2_read_unknown(uint64_t offset) "offset=0x%"PRIx64
> > +aes_2_read(uint64_t offset, uint64_t res) "offset=0x%"PRIx64"
> res=0x%"PRIx64
> > +aes_2_write_unknown(uint64_t offset) "offset=0x%"PRIx64
> > +aes_2_write(uint64_t offset, uint64_t val) "offset=0x%"PRIx64"
> val=0x%"PRIx64
> > +aes_dump_data(const char *desc, const char *hex) "%s%s"
> > +
>
>

[-- Attachment #2: Type: text/html, Size: 29387 bytes --]

Re: [PATCH v3 13/14] hw/vmapple/virtio-blk: Add support for apple virtio-blk

[Phil Dennis-Jordan]

[-- Attachment #1: Type: text/plain, Size: 21247 bytes --]

On Mon, 7 Oct 2024 at 20:10, Akihiko Odaki <akihiko.odaki@daynix.com> wrote:

> On 2024/10/07 23:31, Phil Dennis-Jordan wrote:
> >
> > On Sat, 5 Oct 2024 at 07:47, Akihiko Odaki <akihiko.odaki@daynix.com
> > <mailto:akihiko.odaki@daynix.com>> wrote:
> >
> >     On 2024/09/28 17:57, Phil Dennis-Jordan wrote:
> >      > From: Alexander Graf <graf@amazon.com <mailto:graf@amazon.com>>
> >      >
> >      > Apple has its own virtio-blk PCI device ID where it deviates from
> the
> >      > official virtio-pci spec slightly: It puts a new "apple type"
> >      > field at a static offset in config space and introduces a new
> barrier
> >      > command.
> >      >
> >      > This patch first creates a mechanism for virtio-blk downstream
> >     classes to
> >      > handle unknown commands. It then creates such a downstream class
> >     and a new
> >      > vmapple-virtio-blk-pci class which support the additional apple
> >     type config
> >      > identifier as well as the barrier command.
> >      >
> >      > It then exposes 2 subclasses from that that we can use to expose
> >     root and
> >      > aux virtio-blk devices: "vmapple-virtio-root" and "vmapple-
> >     virtio-aux".
> >      >
> >      > Signed-off-by: Alexander Graf <graf@amazon.com
> >     <mailto:graf@amazon.com>>
> >      > Signed-off-by: Phil Dennis-Jordan <phil@philjordan.eu
> >     <mailto:phil@philjordan.eu>>
> >      > ---
> >      >   hw/block/virtio-blk.c           |  19 ++-
> >      >   hw/vmapple/Kconfig              |   3 +
> >      >   hw/vmapple/meson.build          |   1 +
> >      >   hw/vmapple/virtio-blk.c         | 212 +++++++++++++++++++++++++
> >     +++++++
> >      >   include/hw/pci/pci_ids.h        |   1 +
> >      >   include/hw/virtio/virtio-blk.h  |  12 +-
> >      >   include/hw/vmapple/virtio-blk.h |  39 ++++++
> >      >   7 files changed, 282 insertions(+), 5 deletions(-)
> >      >   create mode 100644 hw/vmapple/virtio-blk.c
> >      >   create mode 100644 include/hw/vmapple/virtio-blk.h
> >      >
> >      > diff --git a/hw/block/virtio-blk.c b/hw/block/virtio-blk.c
> >      > index 115795392c4..cecc4cef9e4 100644
> >      > --- a/hw/block/virtio-blk.c
> >      > +++ b/hw/block/virtio-blk.c
> >      > @@ -50,12 +50,12 @@ static void
> >     virtio_blk_init_request(VirtIOBlock *s, VirtQueue *vq,
> >      >       req->mr_next = NULL;
> >      >   }
> >      >
> >      > -static void virtio_blk_free_request(VirtIOBlockReq *req)
> >      > +void virtio_blk_free_request(VirtIOBlockReq *req)
> >      >   {
> >      >       g_free(req);
> >      >   }
> >
> >     This function is identical with g_free(). Perhaps it's better to
> remove
> >     it instead of updating it.
> >
> >
> > I'm not sure that's something I should be doing in such a tangential
> > patch series, and it's very much a matter of taste - other operations on
> > VirtIOBlockReq have similar, consistent naming, and this function
> > seemingly hasn't been touched for 7 years.
>
> It is not really consistent when it comes to memory allocation. It
> simply uses virtqueue_pop() or qemu_get_virtqueue_element() without
> function wrappers to get VirtIOBlockReq allocated. It shouldn't be
> necessary for deallocation either.
>
> git blame reveals virtio_blk_free_request() used to be more complex.
> It's more likely that just nobody had a need to remove this function.
>

I've included this code transformation as an optional second commit in v4,
it can then either be pulled or dropped from the pull depending on popular
opinion.


> >
> > Perhaps virtio-blk maintainer Stefan Hajnoczi could weigh in? (tagged in
> > To:)
> >
> >
> >      > -static void virtio_blk_req_complete(VirtIOBlockReq *req,
> >     unsigned char status)
> >      > +void virtio_blk_req_complete(VirtIOBlockReq *req, unsigned char
> >     status)
> >      >   {
> >      >       VirtIOBlock *s = req->dev;
> >      >       VirtIODevice *vdev = VIRTIO_DEVICE(s);
> >      > @@ -966,8 +966,18 @@ static int
> >     virtio_blk_handle_request(VirtIOBlockReq *req, MultiReqBuffer *mrb)
> >      >           break;
> >      >       }
> >      >       default:
> >      > -        virtio_blk_req_complete(req, VIRTIO_BLK_S_UNSUPP);
> >      > -        virtio_blk_free_request(req);
> >      > +    {
> >      > +        /*
> >      > +         * Give subclasses a chance to handle unknown requests.
> >     This way the
> >      > +         * class lookup is not in the hot path.
> >      > +         */
> >      > +        VirtIOBlkClass *vbk = VIRTIO_BLK_GET_CLASS(s);
> >      > +        if (!vbk->handle_unknown_request ||
> >      > +            !vbk->handle_unknown_request(req, mrb, type)) {
> >      > +            virtio_blk_req_complete(req, VIRTIO_BLK_S_UNSUPP);
> >      > +            virtio_blk_free_request(req);
> >      > +        }
> >      > +    }
> >      >       }
> >      >       return 0;
> >      >   }
> >      > @@ -2044,6 +2054,7 @@ static const TypeInfo virtio_blk_info = {
> >      >       .instance_size = sizeof(VirtIOBlock),
> >      >       .instance_init = virtio_blk_instance_init,
> >      >       .class_init = virtio_blk_class_init,
> >      > +    .class_size = sizeof(VirtIOBlkClass),
> >      >   };
> >      >
> >      >   static void virtio_register_types(void)
> >      > diff --git a/hw/vmapple/Kconfig b/hw/vmapple/Kconfig
> >      > index 8bbeb9a9237..bcd1be63e3c 100644
> >      > --- a/hw/vmapple/Kconfig
> >      > +++ b/hw/vmapple/Kconfig
> >      > @@ -7,3 +7,6 @@ config VMAPPLE_BDIF
> >      >   config VMAPPLE_CFG
> >      >       bool
> >      >
> >      > +config VMAPPLE_VIRTIO_BLK
> >      > +    bool
> >      > +
> >      > diff --git a/hw/vmapple/meson.build b/hw/vmapple/meson.build
> >      > index 64b78693a31..bf17cf906c9 100644
> >      > --- a/hw/vmapple/meson.build
> >      > +++ b/hw/vmapple/meson.build
> >      > @@ -1,3 +1,4 @@
> >      >   system_ss.add(when: 'CONFIG_VMAPPLE_AES',  if_true:
> files('aes.c'))
> >      >   system_ss.add(when: 'CONFIG_VMAPPLE_BDIF', if_true:
> >     files('bdif.c'))
> >      >   system_ss.add(when: 'CONFIG_VMAPPLE_CFG',  if_true:
> files('cfg.c'))
> >      > +system_ss.add(when: 'CONFIG_VMAPPLE_VIRTIO_BLK',  if_true:
> >     files('virtio-blk.c'))
> >      > diff --git a/hw/vmapple/virtio-blk.c b/hw/vmapple/virtio-blk.c
> >      > new file mode 100644
> >      > index 00000000000..720eaa61a86
> >      > --- /dev/null
> >      > +++ b/hw/vmapple/virtio-blk.c
> >      > @@ -0,0 +1,212 @@
> >      > +/*
> >      > + * VMApple specific VirtIO Block implementation
> >      > + *
> >      > + * Copyright © 2023 Amazon.com, Inc. or its affiliates. All
> >     Rights Reserved.
> >      > + *
> >      > + * This work is licensed under the terms of the GNU GPL, version
> >     2 or later.
> >      > + * See the COPYING file in the top-level directory.
> >      > + *
> >      > + * VMApple uses almost standard VirtIO Block, but with a few key
> >     differences:
> >      > + *
> >      > + *  - Different PCI device/vendor ID
> >      > + *  - An additional "type" identifier to differentiate AUX and
> >     Root volumes
> >      > + *  - An additional BARRIER command
> >      > + */
> >      > +
> >      > +#include "qemu/osdep.h"
> >      > +#include "hw/vmapple/virtio-blk.h"
> >      > +#include "qemu/log.h"
> >      > +#include "qemu/module.h"
> >      > +#include "qapi/error.h"
> >      > +
> >      > +#define VIRTIO_BLK_T_APPLE_BARRIER     0x10000
> >      > +
> >      > +#define VIRTIO_APPLE_TYPE_ROOT 1
> >      > +#define VIRTIO_APPLE_TYPE_AUX  2
> >      > +
> >      > +static bool
> >     vmapple_virtio_blk_handle_unknown_request(VirtIOBlockReq *req,
> >      > +
> >     MultiReqBuffer *mrb,
> >      > +                                                      uint32_t
> type)
> >      > +{
> >      > +    switch (type) {
> >      > +    case VIRTIO_BLK_T_APPLE_BARRIER:
> >      > +        /* We ignore barriers for now. YOLO. */
> >
> >     It should be LOG_UNIMP instead of a mere comment.
> >
> >
> > Fixed in next patch version.
> >
> >      > +        virtio_blk_req_complete(req, VIRTIO_BLK_S_OK);
> >      > +        virtio_blk_free_request(req);
> >      > +        return true;
> >      > +    default:
> >      > +        return false;
> >      > +    }
> >      > +}
> >      > +
> >      > +/*
> >      > + * VMApple virtio-blk uses the same config format as normal
> >     virtio, with one
> >      > + * exception: It adds an "apple type" specififer at the same
> >     location that
> >      > + * the spec reserves for max_secure_erase_sectors. Let's hook
> >     into the
> >      > + * get_config code path here, run it as usual and then patch in
> >     the apple type.
> >      > + */
> >      > +static void vmapple_virtio_blk_get_config(VirtIODevice *vdev,
> >     uint8_t *config)
> >      > +{
> >      > +    VMAppleVirtIOBlk *dev = VMAPPLE_VIRTIO_BLK(vdev);
> >      > +    VMAppleVirtIOBlkClass *vvbk =
> VMAPPLE_VIRTIO_BLK_GET_CLASS(dev);
> >      > +    struct virtio_blk_config *blkcfg = (struct virtio_blk_config
> >     *)config;
> >      > +
> >      > +    vvbk->get_config(vdev, config);
> >      > +
> >      > +    g_assert(dev->parent_obj.config_size >= endof(struct
> >     virtio_blk_config, zoned));
> >      > +
> >      > +    /* Apple abuses the field for max_secure_erase_sectors as
> >     type id */
> >      > +    blkcfg->max_secure_erase_sectors = dev->apple_type;
> >      > +}
> >      > +
> >      > +static Property vmapple_virtio_blk_properties[] = {
> >      > +    DEFINE_PROP_UINT32("apple-type", VMAppleVirtIOBlk,
> >     apple_type, 0),
> >      > +    DEFINE_PROP_END_OF_LIST(),
> >      > +};
> >      > +
> >      > +static void vmapple_virtio_blk_class_init(ObjectClass *klass,
> >     void *data)
> >      > +{
> >      > +    DeviceClass *dc = DEVICE_CLASS(klass);
> >      > +    VirtIOBlkClass *vbk = VIRTIO_BLK_CLASS(klass);
> >      > +    VirtioDeviceClass *vdc = VIRTIO_DEVICE_CLASS(klass);
> >      > +    VMAppleVirtIOBlkClass *vvbk =
> VMAPPLE_VIRTIO_BLK_CLASS(klass);
> >      > +
> >      > +    vbk->handle_unknown_request =
> >     vmapple_virtio_blk_handle_unknown_request;
> >      > +    vvbk->get_config = vdc->get_config;
> >      > +    vdc->get_config = vmapple_virtio_blk_get_config;
> >      > +    device_class_set_props(dc, vmapple_virtio_blk_properties);
> >      > +}
> >      > +
> >      > +static const TypeInfo vmapple_virtio_blk_info = {
> >      > +    .name          = TYPE_VMAPPLE_VIRTIO_BLK,
> >      > +    .parent        = TYPE_VIRTIO_BLK,
> >      > +    .instance_size = sizeof(VMAppleVirtIOBlk),
> >      > +    .class_init    = vmapple_virtio_blk_class_init,
> >      > +};
> >      > +
> >      > +/* PCI Devices */
> >      > +
> >      > +typedef struct VMAppleVirtIOBlkPCI {
> >      > +    VirtIOPCIProxy parent_obj;
> >      > +    VMAppleVirtIOBlk vdev;
> >      > +    uint32_t apple_type;
> >      > +} VMAppleVirtIOBlkPCI;
> >      > +
> >      > +/*
> >      > + * vmapple-virtio-blk-pci: This extends VirtioPCIProxy.
> >      > + */
> >      > +#define TYPE_VMAPPLE_VIRTIO_BLK_PCI "vmapple-virtio-blk-pci-base"
> >      > +DECLARE_INSTANCE_CHECKER(VMAppleVirtIOBlkPCI,
> >     VMAPPLE_VIRTIO_BLK_PCI,
> >      > +                         TYPE_VMAPPLE_VIRTIO_BLK_PCI)
> >      > +
> >      > +static Property vmapple_virtio_blk_pci_properties[] = {
> >      > +    DEFINE_PROP_UINT32("class", VirtIOPCIProxy, class_code, 0),
> >      > +    DEFINE_PROP_BIT("ioeventfd", VirtIOPCIProxy, flags,
> >      > +                    VIRTIO_PCI_FLAG_USE_IOEVENTFD_BIT, true),
> >      > +    DEFINE_PROP_UINT32("vectors", VirtIOPCIProxy, nvectors,
> >      > +                       DEV_NVECTORS_UNSPECIFIED),
> >      > +    DEFINE_PROP_END_OF_LIST(),
> >      > +};
> >      > +
> >      > +static void vmapple_virtio_blk_pci_realize(VirtIOPCIProxy
> >     *vpci_dev, Error **errp)
> >      > +{
> >      > +    VMAppleVirtIOBlkPCI *dev = VMAPPLE_VIRTIO_BLK_PCI(vpci_dev);
> >      > +    DeviceState *vdev = DEVICE(&dev->vdev);
> >      > +    VirtIOBlkConf *conf = &dev->vdev.parent_obj.conf;
> >      > +
> >      > +    if (conf->num_queues == VIRTIO_BLK_AUTO_NUM_QUEUES) {
> >      > +        conf->num_queues = virtio_pci_optimal_num_queues(0);
> >      > +    }
> >      > +
> >      > +    if (vpci_dev->nvectors == DEV_NVECTORS_UNSPECIFIED) {
> >      > +        vpci_dev->nvectors = conf->num_queues + 1;
> >      > +    }
> >      > +
> >      > +    /*
> >      > +     * We don't support zones, but we need the additional config
> >     space size.
> >      > +     * Let's just expose the feature so the rest of the virtio-
> >     blk logic
> >      > +     * allocates enough space for us. The guest will ignore
> >     zones anyway.
> >      > +     */
> >      > +    virtio_add_feature(&dev->vdev.parent_obj.host_features,
> >     VIRTIO_BLK_F_ZONED);
> >      > +    /* Propagate the apple type down to the virtio-blk device */
> >      > +    qdev_prop_set_uint32(DEVICE(&dev->vdev), "apple-type", dev-
> >      >apple_type);
> >      > +    /* and spawn the virtio-blk device */
> >      > +    qdev_realize(vdev, BUS(&vpci_dev->bus), errp);
> >      > +
> >      > +    /*
> >      > +     * The virtio-pci machinery adjusts its vendor/device ID
> >     based on whether
> >      > +     * we support modern or legacy virtio. Let's patch it back
> >     to the Apple
> >      > +     * identifiers here.
> >      > +     */
> >      > +    pci_config_set_vendor_id(vpci_dev->pci_dev.config,
> >     PCI_VENDOR_ID_APPLE);
> >      > +    pci_config_set_device_id(vpci_dev->pci_dev.config,
> >     PCI_DEVICE_ID_APPLE_VIRTIO_BLK);
> >      > +}
> >      > +
> >      > +static void vmapple_virtio_blk_pci_class_init(ObjectClass
> >     *klass, void *data)
> >      > +{
> >      > +    DeviceClass *dc = DEVICE_CLASS(klass);
> >      > +    VirtioPCIClass *k = VIRTIO_PCI_CLASS(klass);
> >      > +    PCIDeviceClass *pcidev_k = PCI_DEVICE_CLASS(klass);
> >      > +
> >      > +    set_bit(DEVICE_CATEGORY_STORAGE, dc->categories);
> >      > +    device_class_set_props(dc,
> vmapple_virtio_blk_pci_properties);
> >      > +    k->realize = vmapple_virtio_blk_pci_realize;
> >      > +    pcidev_k->vendor_id = PCI_VENDOR_ID_APPLE;
> >      > +    pcidev_k->device_id = PCI_DEVICE_ID_APPLE_VIRTIO_BLK;
> >      > +    pcidev_k->revision = VIRTIO_PCI_ABI_VERSION;
> >      > +    pcidev_k->class_id = PCI_CLASS_STORAGE_SCSI;
> >      > +}
> >      > +
> >      > +static void vmapple_virtio_blk_pci_instance_init(Object *obj)
> >      > +{
> >      > +    VMAppleVirtIOBlkPCI *dev = VMAPPLE_VIRTIO_BLK_PCI(obj);
> >      > +
> >      > +    virtio_instance_init_common(obj, &dev->vdev,
> sizeof(dev->vdev),
> >      > +                                TYPE_VMAPPLE_VIRTIO_BLK);
> >      > +}
> >      > +
> >      > +static const VirtioPCIDeviceTypeInfo vmapple_virtio_blk_pci_info
> = {
> >      > +    .base_name     = TYPE_VMAPPLE_VIRTIO_BLK_PCI,
> >      > +    .generic_name  = "vmapple-virtio-blk-pci",
> >      > +    .instance_size = sizeof(VMAppleVirtIOBlkPCI),
> >      > +    .instance_init = vmapple_virtio_blk_pci_instance_init,
> >      > +    .class_init    = vmapple_virtio_blk_pci_class_init,
> >      > +};
> >      > +
> >      > +static void vmapple_virtio_root_instance_init(Object *obj)
> >      > +{
> >      > +    VMAppleVirtIOBlkPCI *dev = VMAPPLE_VIRTIO_BLK_PCI(obj);
> >      > +
> >      > +    dev->apple_type = VIRTIO_APPLE_TYPE_ROOT;
> >      > +}
> >      > +
> >      > +static const TypeInfo vmapple_virtio_root_info = {
> >      > +    .name          = TYPE_VMAPPLE_VIRTIO_ROOT,
> >      > +    .parent        = "vmapple-virtio-blk-pci",
> >      > +    .instance_size = sizeof(VMAppleVirtIOBlkPCI),
> >      > +    .instance_init = vmapple_virtio_root_instance_init,
> >      > +};
> >      > +
> >      > +static void vmapple_virtio_aux_instance_init(Object *obj)
> >      > +{
> >      > +    VMAppleVirtIOBlkPCI *dev = VMAPPLE_VIRTIO_BLK_PCI(obj);
> >      > +
> >      > +    dev->apple_type = VIRTIO_APPLE_TYPE_AUX;
> >      > +}
> >      > +
> >      > +static const TypeInfo vmapple_virtio_aux_info = {
> >      > +    .name          = TYPE_VMAPPLE_VIRTIO_AUX,
> >      > +    .parent        = "vmapple-virtio-blk-pci",
> >      > +    .instance_size = sizeof(VMAppleVirtIOBlkPCI),
> >      > +    .instance_init = vmapple_virtio_aux_instance_init,
> >      > +};
> >      > +
> >      > +static void vmapple_virtio_blk_register_types(void)
> >      > +{
> >      > +    type_register_static(&vmapple_virtio_blk_info);
> >      > +    virtio_pci_types_register(&vmapple_virtio_blk_pci_info);
> >      > +    type_register_static(&vmapple_virtio_root_info);
> >      > +    type_register_static(&vmapple_virtio_aux_info);
> >      > +}
> >      > +
> >      > +type_init(vmapple_virtio_blk_register_types)
> >      > diff --git a/include/hw/pci/pci_ids.h b/include/hw/pci/pci_ids.h
> >      > index f1a53fea8d6..33e2898be95 100644
> >      > --- a/include/hw/pci/pci_ids.h
> >      > +++ b/include/hw/pci/pci_ids.h
> >      > @@ -191,6 +191,7 @@
> >      >   #define PCI_DEVICE_ID_APPLE_UNI_N_AGP    0x0020
> >      >   #define PCI_DEVICE_ID_APPLE_U3_AGP       0x004b
> >      >   #define PCI_DEVICE_ID_APPLE_UNI_N_GMAC   0x0021
> >      > +#define PCI_DEVICE_ID_APPLE_VIRTIO_BLK   0x1a00
> >      >
> >      >   #define PCI_VENDOR_ID_SUN                0x108e
> >      >   #define PCI_DEVICE_ID_SUN_EBUS           0x1000
> >      > diff --git a/include/hw/virtio/virtio-blk.h b/include/hw/virtio/
> >     virtio-blk.h
> >      > index 5c14110c4b1..28d5046ea6c 100644
> >      > --- a/include/hw/virtio/virtio-blk.h
> >      > +++ b/include/hw/virtio/virtio-blk.h
> >      > @@ -24,7 +24,7 @@
> >      >   #include "qapi/qapi-types-virtio.h"
> >      >
> >      >   #define TYPE_VIRTIO_BLK "virtio-blk-device"
> >      > -OBJECT_DECLARE_SIMPLE_TYPE(VirtIOBlock, VIRTIO_BLK)
> >      > +OBJECT_DECLARE_TYPE(VirtIOBlock, VirtIOBlkClass, VIRTIO_BLK)
> >      >
> >      >   /* This is the last element of the write scatter-gather list */
> >      >   struct virtio_blk_inhdr
> >      > @@ -100,6 +100,16 @@ typedef struct MultiReqBuffer {
> >      >       bool is_write;
> >      >   } MultiReqBuffer;
> >      >
> >      > +typedef struct VirtIOBlkClass {
> >      > +    /*< private >*/
> >      > +    VirtioDeviceClass parent;
> >      > +    /*< public >*/
> >      > +    bool (*handle_unknown_request)(VirtIOBlockReq *req,
> >     MultiReqBuffer *mrb,
> >      > +                                   uint32_t type);
> >      > +} VirtIOBlkClass;
> >      > +
> >      >   void virtio_blk_handle_vq(VirtIOBlock *s, VirtQueue *vq);
> >      > +void virtio_blk_free_request(VirtIOBlockReq *req);
> >      > +void virtio_blk_req_complete(VirtIOBlockReq *req, unsigned char
> >     status);
> >      >
> >      >   #endif
> >      > diff --git a/include/hw/vmapple/virtio-blk.h b/include/hw/
> >     vmapple/virtio-blk.h
> >      > new file mode 100644
> >      > index 00000000000..b23106a3dfb
> >      > --- /dev/null
> >      > +++ b/include/hw/vmapple/virtio-blk.h
> >      > @@ -0,0 +1,39 @@
> >      > +/*
> >      > + * VMApple specific VirtIO Block implementation
> >      > + *
> >      > + * Copyright © 2023 Amazon.com, Inc. or its affiliates. All
> >     Rights Reserved.
> >      > + *
> >      > + * This work is licensed under the terms of the GNU GPL, version
> >     2 or later.
> >      > + * See the COPYING file in the top-level directory.
> >      > + */
> >      > +
> >      > +#ifndef HW_VMAPPLE_CFG_H
> >      > +#define HW_VMAPPLE_CFG_H
> >      > +
> >      > +#include "hw/sysbus.h"
> >      > +#include "qom/object.h"
> >      > +#include "hw/virtio/virtio-pci.h"
> >      > +#include "hw/virtio/virtio-blk.h"
> >      > +
> >      > +#define TYPE_VMAPPLE_VIRTIO_BLK "vmapple-virtio-blk"
> >      > +#define TYPE_VMAPPLE_VIRTIO_ROOT "vmapple-virtio-root"
> >      > +#define TYPE_VMAPPLE_VIRTIO_AUX "vmapple-virtio-aux"
> >      > +
> >      > +OBJECT_DECLARE_TYPE(VMAppleVirtIOBlk, VMAppleVirtIOBlkClass,
> >     VMAPPLE_VIRTIO_BLK)
> >      > +
> >      > +typedef struct VMAppleVirtIOBlkClass {
> >      > +    /*< private >*/
> >      > +    VirtIOBlkClass parent;
> >      > +    /*< public >*/
> >      > +    void (*get_config)(VirtIODevice *vdev, uint8_t *config);
> >      > +} VMAppleVirtIOBlkClass;
> >      > +
> >      > +typedef struct VMAppleVirtIOBlk {
> >      > +    /* <private> */
> >      > +    VirtIOBlock parent_obj;
> >      > +
> >      > +    /* <public> */
> >      > +    uint32_t apple_type;
> >      > +} VMAppleVirtIOBlk;
> >      > +
> >      > +#endif /* HW_VMAPPLE_CFG_H */
> >
>
>

[-- Attachment #2: Type: text/html, Size: 28471 bytes --]

Re: [PATCH v3 12/14] hw/vmapple/cfg: Introduce vmapple cfg region

[Phil Dennis-Jordan]

[-- Attachment #1: Type: text/plain, Size: 13104 bytes --]

On Mon, 7 Oct 2024 at 20:04, Akihiko Odaki <akihiko.odaki@daynix.com> wrote:

> On 2024/10/07 23:10, Phil Dennis-Jordan wrote:
> >
> >
> > On Sat, 5 Oct 2024 at 07:35, Akihiko Odaki <akihiko.odaki@daynix.com
> > <mailto:akihiko.odaki@daynix.com>> wrote:
> >
> >     On 2024/09/28 17:57, Phil Dennis-Jordan wrote:
> >      > From: Alexander Graf <graf@amazon.com <mailto:graf@amazon.com>>
> >      >
> >      > Instead of device tree or other more standardized means, VMApple
> >     passes
> >      > platform configuration to the first stage boot loader in a binary
> >     encoded
> >      > format that resides at a dedicated RAM region in physical address
> >     space.
> >      >
> >      > This patch models this configuration space as a qdev device which
> >     we can
> >      > then map at the fixed location in the address space. That way, we
> can
> >      > influence and annotate all configuration fields easily.
> >      >
> >      > Signed-off-by: Alexander Graf <graf@amazon.com
> >     <mailto:graf@amazon.com>>
> >      > Signed-off-by: Phil Dennis-Jordan <phil@philjordan.eu
> >     <mailto:phil@philjordan.eu>>
> >      >
> >      > ---
> >      > v3:
> >      >
> >      >   * Replaced legacy device reset method with Resettable method
> >      >
> >      >   hw/vmapple/Kconfig       |   3 ++
> >      >   hw/vmapple/cfg.c         | 106 ++++++++++++++++++++++++++++++++
> >     +++++++
> >      >   hw/vmapple/meson.build   |   1 +
> >      >   include/hw/vmapple/cfg.h |  68 +++++++++++++++++++++++++
> >      >   4 files changed, 178 insertions(+)
> >      >   create mode 100644 hw/vmapple/cfg.c
> >      >   create mode 100644 include/hw/vmapple/cfg.h
> >      >
> >      > diff --git a/hw/vmapple/Kconfig b/hw/vmapple/Kconfig
> >      > index 68f88876eb9..8bbeb9a9237 100644
> >      > --- a/hw/vmapple/Kconfig
> >      > +++ b/hw/vmapple/Kconfig
> >      > @@ -4,3 +4,6 @@ config VMAPPLE_AES
> >      >   config VMAPPLE_BDIF
> >      >       bool
> >      >
> >      > +config VMAPPLE_CFG
> >      > +    bool
> >      > +
> >      > diff --git a/hw/vmapple/cfg.c b/hw/vmapple/cfg.c
> >      > new file mode 100644
> >      > index 00000000000..a5e5c62f59f
> >      > --- /dev/null
> >      > +++ b/hw/vmapple/cfg.c
> >      > @@ -0,0 +1,106 @@
> >      > +/*
> >      > + * VMApple Configuration Region
> >      > + *
> >      > + * Copyright © 2023 Amazon.com, Inc. or its affiliates. All
> >     Rights Reserved.
> >      > + *
> >      > + * This work is licensed under the terms of the GNU GPL, version
> >     2 or later.
> >      > + * See the COPYING file in the top-level directory.
> >      > + */
> >      > +
> >      > +#include "qemu/osdep.h"
> >      > +#include "hw/vmapple/cfg.h"
> >      > +#include "qemu/log.h"
> >      > +#include "qemu/module.h"
> >      > +#include "qapi/error.h"
> >      > +
> >      > +static void vmapple_cfg_reset(Object *obj, ResetType type)
> >      > +{
> >      > +    VMAppleCfgState *s = VMAPPLE_CFG(obj);
> >      > +    VMAppleCfg *cfg;
> >      > +
> >      > +    cfg = memory_region_get_ram_ptr(&s->mem);
> >      > +    memset((void *)cfg, 0, VMAPPLE_CFG_SIZE);
> >      > +    *cfg = s->cfg;
> >       > +}> +
> >      > +static void vmapple_cfg_realize(DeviceState *dev, Error **errp)
> >      > +{
> >      > +    VMAppleCfgState *s = VMAPPLE_CFG(dev);
> >      > +    uint32_t i;
> >      > +
> >      > +    strncpy(s->cfg.serial, s->serial, sizeof(s->cfg.serial));
> >      > +    strncpy(s->cfg.model, s->model, sizeof(s->cfg.model));
> >      > +    strncpy(s->cfg.soc_name, s->soc_name,
> sizeof(s->cfg.soc_name));
> >      > +    strncpy(s->cfg.unk8, "D/A", sizeof(s->cfg.soc_name));
> >
> >     Use qemu_strnlen() to report an error for too long strings.
> >
> >
> > Hmm, I don't see any existing instances of such a pattern. I do however
> > see a couple of uses of g_strlcpy in the Qemu codebase - that would be a
> > better candidate for error checked string copying, though it still
> > involves some awkward return value checks. I'm going to wrap that in a
> > helper function and macro to replace all 4 strncpy instances here. If
> > the same thing is useful elsewhere later, it can be promoted to cutils
> > or similar.
>
> g_strlcpy() internally performs strlen(), which is worse than
> qemu_strnlen().
>

Worse in what sense? It really depends what you're defending against. Sure,
strlcpy blows up if the source string isn't nul-terminated. All the source
strings here are expected to be nul-terminated here though.


> It is nice to have a helper function. Linux also has something similar
> called strscpy():
> https://www.kernel.org/doc/html/latest/core-api/kernel-api.html#c.strscpy
>

I'm not convinced this is the right patch set to be reinventing Qemu's
string copying functions. Instances abound of the even less safe strcpy()
(and sprintf, etc.) being used in Qemu. Some of those are likely even
problematic, but it seems that should be subject to a more holistic
investigation into what kinds of usage patterns there are, and then a
handful of safe solutions can be found that would work for 99% of string
copying in the code base. Not by adding yet another strcpy variant in a
device backend and machine type patch set where one of the existing
variants works just fine.


> >
> > (Also, I notice that last strncpy actually uses the wrong destination
> > size; my wrapper macro uses ARRAY_SIZE to avoid this mistake altogether.)
> >
> >      > +    s->cfg.ecid = cpu_to_be64(s->cfg.ecid);
> >      > +    s->cfg.version = 2;
> >      > +    s->cfg.unk1 = 1;
> >      > +    s->cfg.unk2 = 1;
> >      > +    s->cfg.unk3 = 0x20;
> >      > +    s->cfg.unk4 = 0;
> >      > +    s->cfg.unk5 = 1;
> >      > +    s->cfg.unk6 = 1;
> >      > +    s->cfg.unk7 = 0;
> >      > +    s->cfg.unk10 = 1;
> >      > +
> >      > +    g_assert(s->cfg.nr_cpus < ARRAY_SIZE(s->cfg.cpu_ids));
> >
> >     Report an error instead of asserting.
> >
> >      > +    for (i = 0; i < s->cfg.nr_cpus; i++) {
> >      > +        s->cfg.cpu_ids[i] = i;
> >      > +    }
> >       > +}> +
> >      > +static void vmapple_cfg_init(Object *obj)
> >      > +{
> >      > +    VMAppleCfgState *s = VMAPPLE_CFG(obj);
> >      > +
> >      > +    memory_region_init_ram(&s->mem, obj, "VMApple Config",
> >     VMAPPLE_CFG_SIZE,
> >      > +                           &error_fatal);
> >      > +    sysbus_init_mmio(SYS_BUS_DEVICE(obj), &s->mem);
> >      > +
> >      > +    s->serial = (char *)"1234";
> >      > +    s->model = (char *)"VM0001";
> >      > +    s->soc_name = (char *)"Apple M1 (Virtual)";
> >
> >     These casts are unsafe; these pointers will be freed when this
> >     device is
> >     freed.
> >
> >
> > Good catch! The more usual pattern for default string property values
> > seems to be to fill them in _realize() (using g_strdup()) if no other
> > value was previously set, so I've applied that here for the next version
> > of the patch.
> >
> >
> >      > +}
> >      > +
> >      > +static Property vmapple_cfg_properties[] = {
> >      > +    DEFINE_PROP_UINT32("nr-cpus", VMAppleCfgState, cfg.nr_cpus,
> 1),
> >      > +    DEFINE_PROP_UINT64("ecid", VMAppleCfgState, cfg.ecid, 0),
> >      > +    DEFINE_PROP_UINT64("ram-size", VMAppleCfgState,
> >     cfg.ram_size, 0),
> >      > +    DEFINE_PROP_UINT32("run_installer1", VMAppleCfgState,
> >     cfg.run_installer1, 0),
> >      > +    DEFINE_PROP_UINT32("run_installer2", VMAppleCfgState,
> >     cfg.run_installer2, 0),
> >      > +    DEFINE_PROP_UINT32("rnd", VMAppleCfgState, cfg.rnd, 0),
> >      > +    DEFINE_PROP_MACADDR("mac-en0", VMAppleCfgState, cfg.mac_en0),
> >      > +    DEFINE_PROP_MACADDR("mac-en1", VMAppleCfgState, cfg.mac_en1),
> >      > +    DEFINE_PROP_MACADDR("mac-wifi0", VMAppleCfgState,
> >     cfg.mac_wifi0),
> >      > +    DEFINE_PROP_MACADDR("mac-bt0", VMAppleCfgState, cfg.mac_bt0),
> >      > +    DEFINE_PROP_STRING("serial", VMAppleCfgState, serial),
> >      > +    DEFINE_PROP_STRING("model", VMAppleCfgState, model),
> >      > +    DEFINE_PROP_STRING("soc_name", VMAppleCfgState, soc_name),
> >      > +    DEFINE_PROP_END_OF_LIST(),
> >      > +};
> >      > +
> >      > +static void vmapple_cfg_class_init(ObjectClass *klass, void
> *data)
> >      > +{
> >      > +    DeviceClass *dc = DEVICE_CLASS(klass);
> >      > +    ResettableClass *rc = RESETTABLE_CLASS(klass);
> >      > +
> >      > +    dc->realize = vmapple_cfg_realize;
> >      > +    dc->desc = "VMApple Configuration Region";
> >      > +    device_class_set_props(dc, vmapple_cfg_properties);
> >      > +    rc->phases.hold = vmapple_cfg_reset;
> >      > +}
> >      > +
> >      > +static const TypeInfo vmapple_cfg_info = {
> >      > +    .name          = TYPE_VMAPPLE_CFG,
> >      > +    .parent        = TYPE_SYS_BUS_DEVICE,
> >      > +    .instance_size = sizeof(VMAppleCfgState),
> >      > +    .instance_init = vmapple_cfg_init,
> >      > +    .class_init    = vmapple_cfg_class_init,
> >      > +};
> >      > +
> >      > +static void vmapple_cfg_register_types(void)
> >      > +{
> >      > +    type_register_static(&vmapple_cfg_info);
> >      > +}
> >      > +
> >      > +type_init(vmapple_cfg_register_types)
> >      > diff --git a/hw/vmapple/meson.build b/hw/vmapple/meson.build
> >      > index d4624713deb..64b78693a31 100644
> >      > --- a/hw/vmapple/meson.build
> >      > +++ b/hw/vmapple/meson.build
> >      > @@ -1,2 +1,3 @@
> >      >   system_ss.add(when: 'CONFIG_VMAPPLE_AES',  if_true:
> files('aes.c'))
> >      >   system_ss.add(when: 'CONFIG_VMAPPLE_BDIF', if_true:
> >     files('bdif.c'))
> >      > +system_ss.add(when: 'CONFIG_VMAPPLE_CFG',  if_true:
> files('cfg.c'))
> >      > diff --git a/include/hw/vmapple/cfg.h b/include/hw/vmapple/cfg.h
> >      > new file mode 100644
> >      > index 00000000000..3337064e447
> >      > --- /dev/null
> >      > +++ b/include/hw/vmapple/cfg.h
> >      > @@ -0,0 +1,68 @@
> >      > +/*
> >      > + * VMApple Configuration Region
> >      > + *
> >      > + * Copyright © 2023 Amazon.com, Inc. or its affiliates. All
> >     Rights Reserved.
> >      > + *
> >      > + * This work is licensed under the terms of the GNU GPL, version
> >     2 or later.
> >      > + * See the COPYING file in the top-level directory.
> >      > + */
> >      > +
> >      > +#ifndef HW_VMAPPLE_CFG_H
> >      > +#define HW_VMAPPLE_CFG_H
> >      > +
> >      > +#include "hw/sysbus.h"
> >      > +#include "qom/object.h"
> >      > +#include "net/net.h"
> >      > +
> >      > +typedef struct VMAppleCfg {
> >      > +    uint32_t version;         /* 0x000 */
> >      > +    uint32_t nr_cpus;         /* 0x004 */
> >      > +    uint32_t unk1;            /* 0x008 */
> >      > +    uint32_t unk2;            /* 0x00c */
> >      > +    uint32_t unk3;            /* 0x010 */
> >      > +    uint32_t unk4;            /* 0x014 */
> >      > +    uint64_t ecid;            /* 0x018 */
> >      > +    uint64_t ram_size;        /* 0x020 */
> >      > +    uint32_t run_installer1;  /* 0x028 */
> >      > +    uint32_t unk5;            /* 0x02c */
> >      > +    uint32_t unk6;            /* 0x030 */
> >      > +    uint32_t run_installer2;  /* 0x034 */
> >      > +    uint32_t rnd;             /* 0x038 */
> >      > +    uint32_t unk7;            /* 0x03c */
> >      > +    MACAddr mac_en0;          /* 0x040 */
> >      > +    uint8_t pad1[2];
> >      > +    MACAddr mac_en1;          /* 0x048 */
> >      > +    uint8_t pad2[2];
> >      > +    MACAddr mac_wifi0;        /* 0x050 */
> >      > +    uint8_t pad3[2];
> >      > +    MACAddr mac_bt0;          /* 0x058 */
> >      > +    uint8_t pad4[2];
> >      > +    uint8_t reserved[0xa0];   /* 0x060 */
> >      > +    uint32_t cpu_ids[0x80];   /* 0x100 */
> >      > +    uint8_t scratch[0x200];   /* 0x180 */
> >      > +    char serial[32];          /* 0x380 */
> >      > +    char unk8[32];            /* 0x3a0 */
> >      > +    char model[32];           /* 0x3c0 */
> >      > +    uint8_t unk9[32];         /* 0x3e0 */
> >      > +    uint32_t unk10;           /* 0x400 */
> >      > +    char soc_name[32];        /* 0x404 */
> >      > +} VMAppleCfg;
> >      > +
> >      > +#define TYPE_VMAPPLE_CFG "vmapple-cfg"
> >      > +OBJECT_DECLARE_SIMPLE_TYPE(VMAppleCfgState, VMAPPLE_CFG)
> >      > +
> >      > +struct VMAppleCfgState {
> >      > +    /* <private> */
> >      > +    SysBusDevice parent_obj;
> >      > +    VMAppleCfg cfg;
> >      > +
> >      > +    /* <public> */
> >      > +    MemoryRegion mem;
> >      > +    char *serial;
> >      > +    char *model;
> >      > +    char *soc_name;
> >      > +};
> >      > +
> >      > +#define VMAPPLE_CFG_SIZE 0x00010000
> >      > +
> >      > +#endif /* HW_VMAPPLE_CFG_H */
> >
>
>

[-- Attachment #2: Type: text/html, Size: 18137 bytes --]

Re: [PATCH v3 11/14] hw/vmapple/bdif: Introduce vmapple backdoor interface

[Phil Dennis-Jordan]

[-- Attachment #1: Type: text/plain, Size: 11829 bytes --]

On Sat, 5 Oct 2024 at 07:12, Akihiko Odaki <akihiko.odaki@daynix.com> wrote:

> On 2024/09/28 17:57, Phil Dennis-Jordan wrote:
> > From: Alexander Graf <graf@amazon.com>
> >
> > The VMApple machine exposes AUX and ROOT block devices (as well as USB
> OTG
> > emulation) via virtio-pci as well as a special, simple backdoor platform
> > device.
> >
> > This patch implements this backdoor platform device to the best of my
> > understanding. I left out any USB OTG parts; they're only needed for
> > guest recovery and I don't understand the protocol yet.
> >
> > Signed-off-by: Alexander Graf <graf@amazon.com>
> > Signed-off-by: Phil Dennis-Jordan <phil@philjordan.eu>
> > ---
> >   hw/vmapple/Kconfig        |   3 +
> >   hw/vmapple/bdif.c         | 245 ++++++++++++++++++++++++++++++++++++++
> >   hw/vmapple/meson.build    |   1 +
> >   hw/vmapple/trace-events   |   5 +
> >   include/hw/vmapple/bdif.h |  31 +++++
> >   5 files changed, 285 insertions(+)
> >   create mode 100644 hw/vmapple/bdif.c
> >   create mode 100644 include/hw/vmapple/bdif.h
> >
> > diff --git a/hw/vmapple/Kconfig b/hw/vmapple/Kconfig
> > index a73504d5999..68f88876eb9 100644
> > --- a/hw/vmapple/Kconfig
> > +++ b/hw/vmapple/Kconfig
> > @@ -1,3 +1,6 @@
> >   config VMAPPLE_AES
> >       bool
> >
> > +config VMAPPLE_BDIF
> > +    bool
> > +
> > diff --git a/hw/vmapple/bdif.c b/hw/vmapple/bdif.c
> > new file mode 100644
> > index 00000000000..36b5915ff30
> > --- /dev/null
> > +++ b/hw/vmapple/bdif.c
> > @@ -0,0 +1,245 @@
> > +/*
> > + * VMApple Backdoor Interface
> > + *
> > + * Copyright © 2023 Amazon.com, Inc. or its affiliates. All Rights
> Reserved.
> > + *
> > + * This work is licensed under the terms of the GNU GPL, version 2 or
> later.
> > + * See the COPYING file in the top-level directory.
> > + */
> > +
> > +#include "qemu/osdep.h"
> > +#include "hw/vmapple/bdif.h"
> > +#include "qemu/log.h"
> > +#include "qemu/module.h"
> > +#include "qapi/error.h"
> > +#include "trace.h"
> > +#include "hw/block/block.h"
> > +#include "sysemu/block-backend.h"
> > +
> > +#define REG_DEVID_MASK      0xffff0000
> > +#define DEVID_ROOT          0x00000000
> > +#define DEVID_AUX           0x00010000
> > +#define DEVID_USB           0x00100000
> > +
> > +#define REG_STATUS          0x0
> > +#define REG_STATUS_ACTIVE     BIT(0)
> > +#define REG_CFG             0x4
> > +#define REG_CFG_ACTIVE        BIT(1)
> > +#define REG_UNK1            0x8
> > +#define REG_BUSY            0x10
> > +#define REG_BUSY_READY        BIT(0)
> > +#define REG_UNK2            0x400
> > +#define REG_CMD             0x408
> > +#define REG_NEXT_DEVICE     0x420
> > +#define REG_UNK3            0x434
> > +
> > +typedef struct vblk_sector {
>
> Please use VblkSector for the tag name too.
>
> > +    uint32_t pad;
> > +    uint32_t pad2;
> > +    uint32_t sector;
> > +    uint32_t pad3;
> > +} VblkSector;
> > +
> > +typedef struct vblk_req_cmd {
> > +    uint64_t addr;
> > +    uint32_t len;
> > +    uint32_t flags;
> > +} VblkReqCmd;
> > +
> > +typedef struct vblk_req {
> > +    VblkReqCmd sector;
> > +    VblkReqCmd data;
> > +    VblkReqCmd retval;
> > +} VblkReq;
> > +
> > +#define VBLK_DATA_FLAGS_READ  0x00030001
> > +#define VBLK_DATA_FLAGS_WRITE 0x00010001
> > +
> > +#define VBLK_RET_SUCCESS  0
> > +#define VBLK_RET_FAILED   1
> > +
> > +static uint64_t bdif_read(void *opaque, hwaddr offset, unsigned size)
> > +{
> > +    uint64_t ret = -1;
> > +    uint64_t devid = (offset & REG_DEVID_MASK);
>
> The parenthes in this line are unnecessary.
>
> > +
> > +    switch (offset & ~REG_DEVID_MASK) {
> > +    case REG_STATUS:
> > +        ret = REG_STATUS_ACTIVE;
> > +        break;
> > +    case REG_CFG:
> > +        ret = REG_CFG_ACTIVE;
> > +        break;
> > +    case REG_UNK1:
> > +        ret = 0x420;
> > +        break;
> > +    case REG_BUSY:
> > +        ret = REG_BUSY_READY;
> > +        break;
> > +    case REG_UNK2:
> > +        ret = 0x1;
> > +        break;
> > +    case REG_UNK3:
> > +        ret = 0x0;
> > +        break;
> > +    case REG_NEXT_DEVICE:
> > +        switch (devid) {
> > +        case DEVID_ROOT:
> > +            ret = 0x8000000;
> > +            break;
> > +        case DEVID_AUX:
> > +            ret = 0x10000;
> > +            break;
> > +        }
> > +        break;
> > +    }
> > +
> > +    trace_bdif_read(offset, size, ret);
> > +    return ret;
> > +}
> > +
> > +static void le2cpu_sector(VblkSector *sector)
> > +{
> > +    sector->sector = le32_to_cpu(sector->sector);
> > +}
> > +
> > +static void le2cpu_reqcmd(VblkReqCmd *cmd)
> > +{
> > +    cmd->addr = le64_to_cpu(cmd->addr);
> > +    cmd->len = le32_to_cpu(cmd->len);
> > +    cmd->flags = le32_to_cpu(cmd->flags);
> > +}
> > +
> > +static void le2cpu_req(VblkReq *req)
> > +{
> > +    le2cpu_reqcmd(&req->sector);
> > +    le2cpu_reqcmd(&req->data);
> > +    le2cpu_reqcmd(&req->retval);
> > +}
> > +
> > +static void vblk_cmd(uint64_t devid, BlockBackend *blk, uint64_t value,
> > +                     uint64_t static_off)
> > +{
> > +    VblkReq req;
> > +    VblkSector sector;
> > +    uint64_t off = 0;
> > +    char *buf = NULL;
> > +    uint8_t ret = VBLK_RET_FAILED;
> > +    int r;
> > +
> > +    cpu_physical_memory_read(value, &req, sizeof(req));
> > +    le2cpu_req(&req);
> > +
> > +    if (req.sector.len != sizeof(sector)) {
> > +        ret = VBLK_RET_FAILED;
> > +        goto out;
> > +    }
> > +
> > +    /* Read the vblk command */
> > +    cpu_physical_memory_read(req.sector.addr, &sector, sizeof(sector));
> > +    le2cpu_sector(&sector);
> > +
> > +    off = sector.sector * 512ULL + static_off;
> > +
> > +    /* Sanity check that we're not allocating bogus sizes */
> > +    if (req.data.len > (128 * 1024 * 1024)) {
>
> Use MiB defined in: include/qemu/units.h
> The parentheses on the right hand are also unnecessary.
>
> > +        goto out;
> > +    }
> > +
> > +    buf = g_malloc0(req.data.len);
> > +    switch (req.data.flags) {
> > +    case VBLK_DATA_FLAGS_READ:
> > +        r = blk_pread(blk, off, req.data.len, buf, 0);
> > +        trace_bdif_vblk_read(devid == DEVID_AUX ? "aux" : "root",
> > +                             req.data.addr, off, req.data.len, r);
> > +        if (r < 0) {
> > +            goto out;
> > +        }
> > +        cpu_physical_memory_write(req.data.addr, buf, req.data.len);
> > +        ret = VBLK_RET_SUCCESS;
> > +        break;
> > +    case VBLK_DATA_FLAGS_WRITE:
> > +        /* Not needed, iBoot only reads */
> > +        break;
> > +    default:
> > +        break;
> > +    }
> > +
> > +out:
> > +    g_free(buf);
> > +    cpu_physical_memory_write(req.retval.addr, &ret, 1);
> > +}
> > +
> > +static void bdif_write(void *opaque, hwaddr offset,
> > +                       uint64_t value, unsigned size)
> > +{
> > +    VMAppleBdifState *s = opaque;
> > +    uint64_t devid = (offset & REG_DEVID_MASK);
> > +
> > +    trace_bdif_write(offset, size, value);
> > +
> > +    switch (offset & ~REG_DEVID_MASK) {
> > +    case REG_CMD:
> > +        switch (devid) {
> > +        case DEVID_ROOT:
> > +            vblk_cmd(devid, s->root, value, 0x0);
> > +            break;
> > +        case DEVID_AUX:
> > +            vblk_cmd(devid, s->aux, value, 0x0);
> > +            break;
> > +        }
> > +        break;
> > +    }
> > +}
> > +
> > +static const MemoryRegionOps bdif_ops = {
> > +    .read = bdif_read,
> > +    .write = bdif_write,
> > +    .endianness = DEVICE_NATIVE_ENDIAN,
> > +    .valid = {
> > +        .min_access_size = 1,
> > +        .max_access_size = 8,
> > +    },
> > +    .impl = {
> > +        .min_access_size = 1,
> > +        .max_access_size = 8,
> > +    },
> > +};
> > +
> > +static void bdif_init(Object *obj)
> > +{
> > +    VMAppleBdifState *s = VMAPPLE_BDIF(obj);
> > +
> > +    memory_region_init_io(&s->mmio, obj, &bdif_ops, obj,
> > +                         "VMApple Backdoor Interface",
> VMAPPLE_BDIF_SIZE);
> > +    sysbus_init_mmio(SYS_BUS_DEVICE(obj), &s->mmio);
> > +}
> > +
> > +static Property bdif_properties[] = {
> > +    DEFINE_PROP_DRIVE("aux", VMAppleBdifState, aux),
> > +    DEFINE_PROP_DRIVE("root", VMAppleBdifState, root),
> > +    DEFINE_PROP_END_OF_LIST(),
> > +};
> > +
> > +static void bdif_class_init(ObjectClass *klass, void *data)
> > +{
> > +    DeviceClass *dc = DEVICE_CLASS(klass);
> > +
> > +    dc->desc = "VMApple Backdoor Interface";
> > +    device_class_set_props(dc, bdif_properties);
> > +}
> > +
> > +static const TypeInfo bdif_info = {
> > +    .name          = TYPE_VMAPPLE_BDIF,
> > +    .parent        = TYPE_SYS_BUS_DEVICE,
> > +    .instance_size = sizeof(VMAppleBdifState),
> > +    .instance_init = bdif_init,
> > +    .class_init    = bdif_class_init,
> > +};
> > +
> > +static void bdif_register_types(void)
> > +{
> > +    type_register_static(&bdif_info);
> > +}
> > +
> > +type_init(bdif_register_types)
> > diff --git a/hw/vmapple/meson.build b/hw/vmapple/meson.build
> > index bcd4dcb28d2..d4624713deb 100644
> > --- a/hw/vmapple/meson.build
> > +++ b/hw/vmapple/meson.build
> > @@ -1 +1,2 @@
> >   system_ss.add(when: 'CONFIG_VMAPPLE_AES',  if_true: files('aes.c'))
> > +system_ss.add(when: 'CONFIG_VMAPPLE_BDIF', if_true: files('bdif.c'))
> > diff --git a/hw/vmapple/trace-events b/hw/vmapple/trace-events
> > index 1c9a3326eb4..fc8e9cc5897 100644
> > --- a/hw/vmapple/trace-events
> > +++ b/hw/vmapple/trace-events
> > @@ -19,3 +19,8 @@ aes_2_write_unknown(uint64_t offset) "offset=0x%"PRIx64
> >   aes_2_write(uint64_t offset, uint64_t val) "offset=0x%"PRIx64"
> val=0x%"PRIx64
> >   aes_dump_data(const char *desc, const char *hex) "%s%s"
> >
> > +# bdif.c
> > +bdif_read(uint64_t offset, uint32_t size, uint64_t value)
> "offset=0x%"PRIx64" size=0x%x value=0x%"PRIx64
> > +bdif_write(uint64_t offset, uint32_t size, uint64_t value)
> "offset=0x%"PRIx64" size=0x%x value=0x%"PRIx64
> > +bdif_vblk_read(const char *dev, uint64_t addr, uint64_t offset,
> uint32_t len, int r) "dev=%s addr=0x%"PRIx64" off=0x%"PRIx64" size=0x%x
> r=%d"
> > +
> > diff --git a/include/hw/vmapple/bdif.h b/include/hw/vmapple/bdif.h
> > new file mode 100644
> > index 00000000000..65ee43457b9
> > --- /dev/null
> > +++ b/include/hw/vmapple/bdif.h
> > @@ -0,0 +1,31 @@
> > +/*
> > + * VMApple Backdoor Interface
> > + *
> > + * Copyright © 2023 Amazon.com, Inc. or its affiliates. All Rights
> Reserved.
> > + *
> > + * This work is licensed under the terms of the GNU GPL, version 2 or
> later.
> > + * See the COPYING file in the top-level directory.
> > + */
> > +
> > +#ifndef HW_VMAPPLE_BDIF_H
> > +#define HW_VMAPPLE_BDIF_H
> > +
> > +#include "hw/sysbus.h"
> > +#include "qom/object.h"
> > +
> > +#define TYPE_VMAPPLE_BDIF "vmapple-bdif"
> > +OBJECT_DECLARE_SIMPLE_TYPE(VMAppleBdifState, VMAPPLE_BDIF)
> > +
> > +struct VMAppleBdifState {
> > +    /* <private> */
> > +    SysBusDevice parent_obj;
> > +
> > +    /* <public> */
> > +    BlockBackend *aux;
> > +    BlockBackend *root;
> > +    MemoryRegion mmio;
> > +};
> > +
> > +#define VMAPPLE_BDIF_SIZE 0x00200000
>
> Please move VMAppleBdifState and VMAPPLE_BDIF_SIZE into: hw/vmapple/bdif.c
> They are both private.
>

Hmm, the same thing applies to the cfg device and the virtio-blk extension;
the only thing that's actually interesting to have in a header file are the
device type string definitions, so I think I'll put all of them in a single
/include/hw/vmapple/vmapple.h file.


> > +
> > +#endif /* HW_VMAPPLE_BDIF_H */
>
>

[-- Attachment #2: Type: text/html, Size: 15246 bytes --]

Re: [PATCH v3 04/14] hw/display/apple-gfx: Adds configurable mode list

[Phil Dennis-Jordan]

[-- Attachment #1: Type: text/plain, Size: 13252 bytes --]

On Fri, 4 Oct 2024 at 06:17, Akihiko Odaki <akihiko.odaki@daynix.com> wrote:

> On 2024/09/28 17:57, Phil Dennis-Jordan wrote:
> > This change adds a property 'display_modes' on the graphics device
> > which permits specifying a list of display modes. (screen resolution
> > and refresh rate)
> >
> > PCI variant of apple-gfx only for the moment.
> >
> > Signed-off-by: Phil Dennis-Jordan <phil@philjordan.eu>
> > ---
> >   hw/display/apple-gfx-pci.m |  43 ++++++++++-
> >   hw/display/apple-gfx.h     |  17 ++++-
> >   hw/display/apple-gfx.m     | 151 ++++++++++++++++++++++++++++++++++---
> >   3 files changed, 198 insertions(+), 13 deletions(-)
> >
> > diff --git a/hw/display/apple-gfx-pci.m b/hw/display/apple-gfx-pci.m
> > index 9370258ee46..ea86a1f4a21 100644
> > --- a/hw/display/apple-gfx-pci.m
> > +++ b/hw/display/apple-gfx-pci.m
> > @@ -16,6 +16,7 @@
> >   #include "apple-gfx.h"
> >   #include "hw/pci/pci_device.h"
> >   #include "hw/pci/msi.h"
> > +#include "hw/qdev-properties.h"
> >   #include "qapi/error.h"
> >   #include "trace.h"
> >   #import <ParavirtualizedGraphics/ParavirtualizedGraphics.h>
> > @@ -101,6 +102,46 @@ static void apple_gfx_pci_reset(Object *obj,
> ResetType type)
> >       [s->common.pgdev reset];
> >   }
> >
> > +static void apple_gfx_pci_get_display_modes(Object *obj, Visitor *v,
> > +                                            const char *name, void
> *opaque,
> > +                                            Error **errp)
> > +{
> > +    Property *prop = opaque;
> > +    AppleGFXDisplayModeList *mode_list = object_field_prop_ptr(obj,
> prop);
> > +
> > +    apple_gfx_get_display_modes(mode_list, v, name, errp);
> > +}
> > +
> > +static void apple_gfx_pci_set_display_modes(Object *obj, Visitor *v,
> > +                                            const char *name, void
> *opaque,
> > +                                            Error **errp)
> > +{
> > +    Property *prop = opaque;
> > +    AppleGFXDisplayModeList *mode_list = object_field_prop_ptr(obj,
> prop);
> > +
> > +    apple_gfx_set_display_modes(mode_list, v, name, errp);
> > +}
> > +
> > +const PropertyInfo apple_gfx_pci_prop_display_modes = {
> > +    .name  = "display_modes",
> > +    .description =
> > +        "Colon-separated list of display modes; "
> > +        "<width>x<height>@<refresh-rate>; the first mode is considered "
> > +        "'native'. Example: 3840x2160@60:2560x1440@60:1920x1080@60",
>
> Please use DEFINE_PROP_ARRAY() instead of inventing your own way to
> define an array.
>

Looks like the ability to specify array properties on the Qemu command line
has recently been added. I'm pretty sure I still need the custom set/get
implementations for each array element (display mode) though.


> > +    .get   = apple_gfx_pci_get_display_modes,
> > +    .set   = apple_gfx_pci_set_display_modes,
> > +};
> > +
> > +#define DEFINE_PROP_DISPLAY_MODES(_name, _state, _field) \
> > +    DEFINE_PROP(_name, _state, _field,
> apple_gfx_pci_prop_display_modes, \
> > +                AppleGFXDisplayModeList)
> > +
> > +static Property apple_gfx_pci_properties[] = {
> > +    DEFINE_PROP_DISPLAY_MODES("display-modes", AppleGFXPCIState,
> > +                              common.display_modes),
> > +    DEFINE_PROP_END_OF_LIST(),
> > +};
> > +
> >   static void apple_gfx_pci_class_init(ObjectClass *klass, void *data)
> >   {
> >       DeviceClass *dc = DEVICE_CLASS(klass);
> > @@ -118,7 +159,7 @@ static void apple_gfx_pci_class_init(ObjectClass
> *klass, void *data)
> >       pci->class_id = PCI_CLASS_DISPLAY_OTHER;
> >       pci->realize = apple_gfx_pci_realize;
> >
> > -    // TODO: Property for setting mode list
> > +    device_class_set_props(dc, apple_gfx_pci_properties);
> >   }
> >
> >   static TypeInfo apple_gfx_pci_types[] = {
> > diff --git a/hw/display/apple-gfx.h b/hw/display/apple-gfx.h
> > index 995ecf7f4a7..baad4a98652 100644
> > --- a/hw/display/apple-gfx.h
> > +++ b/hw/display/apple-gfx.h
> > @@ -5,14 +5,28 @@
> >   #define TYPE_APPLE_GFX_PCI          "apple-gfx-pci"
> >
> >   #include "qemu/typedefs.h"
> > +#include "qemu/osdep.h"
> >
> >   typedef struct AppleGFXState AppleGFXState;
> >
> > +typedef struct AppleGFXDisplayMode {
> > +    uint16_t width_px;
> > +    uint16_t height_px;
> > +    uint16_t refresh_rate_hz;
> > +} AppleGFXDisplayMode;
> > +
> > +typedef struct AppleGFXDisplayModeList {
> > +    GArray *modes;
> > +} AppleGFXDisplayModeList;
> > +
> >   void apple_gfx_common_init(Object *obj, AppleGFXState *s, const char*
> obj_name);
> > +void apple_gfx_get_display_modes(AppleGFXDisplayModeList *mode_list,
> Visitor *v,
> > +                                 const char *name, Error **errp);
> > +void apple_gfx_set_display_modes(AppleGFXDisplayModeList *mode_list,
> Visitor *v,
> > +                                 const char *name, Error **errp);
> >
> >   #ifdef __OBJC__
> >
> > -#include "qemu/osdep.h"
> >   #include "exec/memory.h"
> >   #include "ui/surface.h"
> >   #include <dispatch/dispatch.h>
> > @@ -38,6 +52,7 @@ struct AppleGFXState {
> >       bool new_frame;
> >       bool cursor_show;
> >       QEMUCursor *cursor;
> > +    AppleGFXDisplayModeList display_modes;
> >
> >       dispatch_queue_t render_queue;
> >       /* The following fields should only be accessed from render_queue:
> */
> > diff --git a/hw/display/apple-gfx.m b/hw/display/apple-gfx.m
> > index 6ef1048d93d..358192db6a0 100644
> > --- a/hw/display/apple-gfx.m
> > +++ b/hw/display/apple-gfx.m
> > @@ -16,6 +16,9 @@
> >   #include "trace.h"
> >   #include "qemu-main.h"
> >   #include "qemu/main-loop.h"
> > +#include "qemu/cutils.h"
> > +#include "qapi/visitor.h"
> > +#include "qapi/error.h"
> >   #include "ui/console.h"
> >   #include "monitor/monitor.h"
> >   #include "qapi/error.h"
> > @@ -23,9 +26,10 @@
> >   #include <mach/mach_vm.h>
> >   #import <ParavirtualizedGraphics/ParavirtualizedGraphics.h>
> >
> > -static const PGDisplayCoord_t apple_gfx_modes[] = {
> > -    { .x = 1440, .y = 1080 },
> > -    { .x = 1280, .y = 1024 },
> > +static const AppleGFXDisplayMode apple_gfx_default_modes[] = {
> > +    { 1920, 1080, 60 },
> > +    { 1440, 1080, 60 },
> > +    { 1280, 1024, 60 },
> >   };
> >
> >   typedef struct PGTask_s { // Name matches forward declaration in PG
> header
> > @@ -264,7 +268,6 @@ static void set_mode(AppleGFXState *s, uint32_t
> width, uint32_t height)
> >   static void create_fb(AppleGFXState *s)
> >   {
> >       s->con = graphic_console_init(NULL, 0, &apple_gfx_fb_ops, s);
> > -    set_mode(s, 1440, 1080);
> >
> >       s->cursor_show = true;
> >   }
> > @@ -466,20 +469,24 @@ static void
> apple_gfx_register_task_mapping_handlers(AppleGFXState *s,
> >       return disp_desc;
> >   }
> >
> > -static NSArray<PGDisplayMode*>*
> apple_gfx_prepare_display_mode_array(void)
> > +static NSArray<PGDisplayMode*>* apple_gfx_create_display_mode_array(
> > +    const AppleGFXDisplayMode display_modes[], int display_mode_count)
> >   {
> > -    PGDisplayMode *modes[ARRAY_SIZE(apple_gfx_modes)];
> > +    PGDisplayMode **modes = alloca(sizeof(modes[0]) *
> display_mode_count);
> >       NSArray<PGDisplayMode*>* mode_array = nil;
> >       int i;
> >
> > -    for (i = 0; i < ARRAY_SIZE(apple_gfx_modes); i++) {
> > +    for (i = 0; i < display_mode_count; i++) {
> > +        const AppleGFXDisplayMode *mode = &display_modes[i];
> > +        PGDisplayCoord_t mode_size = { mode->width_px, mode->height_px
> };
> >           modes[i] =
> > -            [[PGDisplayMode alloc]
> initWithSizeInPixels:apple_gfx_modes[i] refreshRateInHz:60.];
> > +            [[PGDisplayMode alloc] initWithSizeInPixels:mode_size
> > +
> refreshRateInHz:mode->refresh_rate_hz];
> >       }
> >
> > -    mode_array = [NSArray arrayWithObjects:modes
> count:ARRAY_SIZE(apple_gfx_modes)];
> > +    mode_array = [NSArray arrayWithObjects:modes
> count:display_mode_count];
> >
> > -    for (i = 0; i < ARRAY_SIZE(apple_gfx_modes); i++) {
> > +    for (i = 0; i < display_mode_count; i++) {
> >           [modes[i] release];
> >           modes[i] = nil;
> >       }
> > @@ -516,6 +523,8 @@ static void
> apple_gfx_register_task_mapping_handlers(AppleGFXState *s,
> >   void apple_gfx_common_realize(AppleGFXState *s, PGDeviceDescriptor
> *desc)
> >   {
> >       PGDisplayDescriptor *disp_desc = nil;
> > +    const AppleGFXDisplayMode *display_modes = apple_gfx_default_modes;
> > +    int num_display_modes = ARRAY_SIZE(apple_gfx_default_modes);
> >
> >       QTAILQ_INIT(&s->tasks);
> >       s->render_queue = dispatch_queue_create("apple-gfx.render",
> > @@ -533,7 +542,127 @@ void apple_gfx_common_realize(AppleGFXState *s,
> PGDeviceDescriptor *desc)
> >       s->pgdisp = [s->pgdev newDisplayWithDescriptor:disp_desc
> >                                                 port:0 serialNum:1234];
> >       [disp_desc release];
> > -    s->pgdisp.modeList = apple_gfx_prepare_display_mode_array();
> > +
> > +    if (s->display_modes.modes != NULL && s->display_modes.modes->len >
> 0) {
> > +        display_modes =
> > +            &g_array_index(s->display_modes.modes, AppleGFXDisplayMode,
> 0);
> > +        num_display_modes = s->display_modes.modes->len;
> > +    }
> > +    s->pgdisp.modeList =
> > +        apple_gfx_create_display_mode_array(display_modes,
> num_display_modes);
> >
> >       create_fb(s);
> >   }
> > +
> > +void apple_gfx_get_display_modes(AppleGFXDisplayModeList *mode_list,
> Visitor *v,
> > +                                 const char *name, Error **errp)
> > +{
> > +    GArray *modes = mode_list->modes;
> > +    /* 3 uint16s (max 5 digits) and 3 separator characters per mode +
> nul. */
> > +    size_t buffer_size = (5 + 1) * 3 * modes->len + 1;
> > +
> > +    char *buffer = alloca(buffer_size);
> > +    char *pos = buffer;
> > +
> > +    unsigned used = 0;
> > +    buffer[0] = '\0';
> > +    for (guint i = 0; i < modes->len; ++i)
> > +    {
> > +        AppleGFXDisplayMode *mode =
> > +            &g_array_index(modes, AppleGFXDisplayMode, i);
> > +        int rc = snprintf(pos, buffer_size - used,
> > +                          "%s%"PRIu16"x%"PRIu16"@%"PRIu16,
> > +                          i > 0 ? ":" : "",
> > +                          mode->width_px, mode->height_px,
> > +                          mode->refresh_rate_hz);
> > +        used += rc;
> > +        pos += rc;
> > +        assert(used < buffer_size);
> > +    }
> > +
> > +    pos = buffer;
> > +    visit_type_str(v, name, &pos, errp);
> > +}
> > +
> > +void apple_gfx_set_display_modes(AppleGFXDisplayModeList *mode_list,
> Visitor *v,
> > +                                 const char *name, Error **errp)
> > +{
> > +    Error *local_err = NULL;
> > +    const char *endptr;
> > +    char *str;
> > +    int ret;
> > +    unsigned int val;
> > +    uint32_t num_modes;
> > +    GArray *modes;
> > +    uint32_t mode_idx;
> > +
> > +    visit_type_str(v, name, &str, &local_err);
> > +    if (local_err) {
> > +        error_propagate(errp, local_err);
> > +        return;
> > +    }
> > +
> > +    // Count colons to estimate modes. No leading/trailing colons so
> start at 1.
> > +    num_modes = 1;
> > +    for (size_t i = 0; str[i] != '\0'; ++i)
> > +    {
> > +        if (str[i] == ':') {
> > +            ++num_modes;
> > +        }
> > +    }
> > +
> > +    modes = g_array_sized_new(false, true, sizeof(AppleGFXDisplayMode),
> num_modes);
> > +
> > +    endptr = str;
> > +    for (mode_idx = 0; mode_idx < num_modes; ++mode_idx)
> > +    {
> > +        AppleGFXDisplayMode mode = {};
> > +        if (mode_idx > 0)
> > +        {
> > +            if (*endptr != ':') {
> > +                goto separator_error;
> > +            }
> > +            ++endptr;
> > +        }
> > +
> > +        ret = qemu_strtoui(endptr, &endptr, 10, &val);
> > +        if (ret || val > UINT16_MAX || val == 0) {
> > +            error_setg(errp, "width of '%s' must be a decimal integer
> number "
> > +                       "of pixels in the range 1..65535", name);
> > +            goto out;
> > +        }
> > +        mode.width_px = val;
> > +        if (*endptr != 'x') {
> > +            goto separator_error;
> > +        }
> > +
> > +        ret = qemu_strtoui(endptr + 1, &endptr, 10, &val);
> > +        if (ret || val > UINT16_MAX || val == 0) {
> > +            error_setg(errp, "height of '%s' must be a decimal integer
> number "
> > +                       "of pixels in the range 1..65535", name);
> > +            goto out;
> > +        }
> > +        mode.height_px = val;
> > +        if (*endptr != '@') {
> > +            goto separator_error;
> > +        }
> > +
> > +        ret = qemu_strtoui(endptr + 1, &endptr, 10, &val);
> > +        if (ret) {
> > +            error_setg(errp, "refresh rate of '%s'"
> > +                       " must be a non-negative decimal integer
> (Hertz)", name);
> > +        }
> > +        mode.refresh_rate_hz = val;
> > +        g_array_append_val(modes, mode);
> > +    }
> > +
> > +    mode_list->modes = modes;
> > +    goto out;
> > +
> > +separator_error:
> > +    error_setg(errp, "Each display mode takes the format "
> > +               "'<width>x<height>@<rate>', modes are separated by
> colons. (:)");
> > +out:
> > +    g_free(str);
> > +    return;
> > +}
>
>

[-- Attachment #2: Type: text/html, Size: 17533 bytes --]

Re: [PATCH v3 01/14] hw/display/apple-gfx: Introduce ParavirtualizedGraphics.Framework support

[Phil Dennis-Jordan]

[-- Attachment #1: Type: text/plain, Size: 42231 bytes --]

On Mon, 7 Oct 2024 at 10:25, Akihiko Odaki <akihiko.odaki@daynix.com> wrote:

>
> >     Theoretically, It should be safe to assume the callee creates
> >     autoreleasepool by themselves as needed in general. We have bunch of
> >     code to call Objective-C APIs without creating autoreleasepool in the
> >     caller. Practically, [PGDeviceDescriptor new] is likely to be
> >     implemented with ARC, which wraps methods in autoreleasepool as
> >     necessary.
> >
> >
> > As far as I'm aware, ARC does NOT automatically insert autorelease pool
> > blocks. The reason you rarely need to create autoreleasepool blocks in
> > "plain" Objective-C programming is that Cocoa/CFRunloop/libdispatch
> > event handlers run each event in an autoreleasepool. So you don't need
> > to create them explicitly when using dispatch_async and similar, or when
> > running code on the main thread (which runs inside NSApplicationMain/
> > CFRunloopRun/dispatch_main).
>
> My statement regarding ARC was wrong; It seems ARC just retains a value
> annoted as autoreleased.
>

Yes, the runtime maintains a sort of shadow stack of pointers to
autoreleased objects for each thread. When you enter a pool block, it
remembers the position in the shadow stack. On exiting the pool, all
objects added since entering the pool are popped from the shadow stack and
released.

Hence, if objects are added to the stack outside of ANY autorelease pool,
those are never popped and released.


> >
> > As far as I'm aware, if you don't explicitly define autoreleasepools in
> > raw threads created with the pthreads API, any autoreleased objects will
> > leak. At least I've not found any specification/documentation
> > contradicting this. And most code in Qemu runs on such raw threads, so
> > we need to play it safe with regard to autorelease semantics.
>
> I decided to dig deeper and found this documentation:
>
> https://developer.apple.com/library/archive/documentation/Cocoa/Conceptual/MemoryMgmt/Articles/mmAutoreleasePools.html
>
> It says:
>  > Cocoa always expects code to be executed within an autorelease pool
>  > block, otherwise autoreleased objects do not get released and your
>  > application leaks memory.
>
> So yes, we must wrap everything in @autoreleasepool at least for Cocoa.
> It is probably a good practice to wrap all Objective-C code in
> @autoreleasepool.
>

Yes; any method or function returning an Objective-C object pointer using
autorelease semantics - either annotated or by naming convention - will use
the pool. But unless it's all your own code all the way down, you don't
know if anything inside a method whose own return value is not autoreleased
might use autorelease semantics.

(The naming convention says most methods return autoreleased, except those
named create*, new*, or similar.)

> Whether the existing Qemu Objective-C code is safe in this regard I
> > don't know for certain, but I've certainly paid attention to this aspect
> > when modifying ui/cocoa.m in the past, and indeed most of that code runs
> > on the main thread. Note also how I wrap the apple_gfx_render_new_frame
> > call in a pool when it can't be guaranteed it's running on a dispatch
> > queue because the command buffer inside that uses autorelease semantics.
>
> It is more about event loop rather than thread. Resources allocated
> before [NSApp run] will leak even if they are autoreleased in the main
> thread. apple_gfx_common_realize() is one of such functions that run in
> the main thread before [NSApp run]. In ui/cocoa, cocoa_display_init()
> runs before [NSApp run]. Fortunately we already have NSAutoreleasePool
> for this function.
>

Yes. Unless it's very clear that the code will run only in a Cocoa event
handler or dispatch queue context, you need to use an autoreleasepool.


> >
> >     Functions that uses a method that returns autorelease resources
> should
> >     be wrapped with autoreleasepool instead of assuming the caller
> creates
> >     autoreleasepool for them.
> >
> >
> > I'm treating apple_gfx_common_realize as an internal API, and I don't
> > think expecting its callers to wrap it in an autoreleasepool block is
> > unreasonable. I can certainly explicitly document this in a comment.
>
> We don't have a comment for cocoa_display_init() and it's more about
> generic macOS programming so it's not necessary.
>
>  > >      >      > diff --git a/hw/display/apple-gfx.m
> b/hw/display/apple-gfx.m
> >      >      > new file mode 100644
> >      >      > index 00000000000..837300f9cd4
> >      >      > --- /dev/null
> >      >      > +++ b/hw/display/apple-gfx.m
> >      >      > @@ -0,0 +1,536 @@
> >      >      > +/*
> >      >      > + * QEMU Apple ParavirtualizedGraphics.framework device
> >      >      > + *
> >      >      > + * Copyright © 2023 Amazon.com, Inc. or its affiliates.
> All
> >      >     Rights Reserved.
> >      >      > + *
> >      >      > + * This work is licensed under the terms of the GNU GPL,
> >     version
> >      >     2 or later.
> >      >      > + * See the COPYING file in the top-level directory.
> >      >      > + *
> >      >      > + * ParavirtualizedGraphics.framework is a set of
> >     libraries that
> >      >     macOS provides
> >      >      > + * which implements 3d graphics passthrough to the host
> >     as well as a
> >      >      > + * proprietary guest communication channel to drive it.
> This
> >      >     device model
> >      >      > + * implements support to drive that library from within
> QEMU.
> >      >      > + */
> >      >      > +
> >      >      > +#include "apple-gfx.h"
> >      >      > +#include "trace.h"
> >      >      > +#include "qemu/main-loop.h"
> >      >      > +#include "ui/console.h"
> >      >      > +#include "monitor/monitor.h"
> >      >      > +#include "qapi/error.h"
> >      >      > +#include "migration/blocker.h"
> >      >      > +#include <mach/mach_vm.h>
> >      >      > +#import
> <ParavirtualizedGraphics/ParavirtualizedGraphics.h>
> >      >      > +
> >      >      > +static const PGDisplayCoord_t apple_gfx_modes[] = {
> >      >      > +    { .x = 1440, .y = 1080 },
> >      >      > +    { .x = 1280, .y = 1024 },
> >      >      > +};
> >      >      > +
> >      >      > +typedef struct PGTask_s { // Name matches forward
> >     declaration in
> >      >     PG header
> >      >
> >      >     Let's name it AppleGFXTask. It is a common practice to have
> >     the same
> >      >     tag
> >      >     name and typedef in QEMU.
> >      >
> >      >
> >      > This is defining a forward-declared type from framework headers
> >     which is
> >      > opaque from the framework's point of view. We do not get to
> >     choose its
> >      > struct name. The alternative is having casts wherever these
> >     objects are
> >      > being passed between our code and the framework. (See the
> >     original v1/v2
> >      > vmapple patch series for how messy this is.)
> >
> >     I got the idea. Let's not avoid the typedef then to clarify the
> naming
> >     is not under our control.
> >
> >
> > I'm not sure what you mean by this double negative. Are you saying,
> > don't add our own typedef for struct PGTask_s at all, just use the
> > framework-supplied PGTask_t where appropriate?
>
> Yes.
>
> >
> >
> >      >
> >      >      > +static void
> >     apple_gfx_render_frame_completed(AppleGFXState *s,
> >      >     void *vram,
> >      >      > +
> >       id<MTLTexture> texture)
> >      >      > +{
> >      >      > +    --s->pending_frames;
> >      >      > +    assert(s->pending_frames >= 0);
> >      >      > +
> >      >      > +    if (vram != s->vram) {
> >      >      > +        /* Display mode has changed, drop this old frame.
> */
> >      >      > +        assert(texture != s->texture);
> >      >      > +        g_free(vram);
> >      >
> >      >     This management of buffers looks a bit convoluted. I suggest
> >      >     remembering
> >      >     the width and height instead of pointers and comparing them.
> >     This way
> >      >     you can free resources in set_mode().
> >      >
> >      >
> >      > Yeah, I suppose that works, I can change that around.
> >      >
> >      >      > +    } else {
> >      >      > +        copy_mtl_texture_to_surface_mem(texture, vram);
> >      >
> >      >     Writing vram outside BQL may result in tearing.
> >      >
> >      >
> >      > As far as I can tell(*), QXL does the same. I couldn't find any
> >     examples
> >      > of double-buffering in any of the existing display devices, which
> >     would
> >      > be the only way to do async updates efficiently and without
> >     tearing. In
> >      > any case, this solution is still vastly better than a regular VGA
> >      > device, which suffers from very visible tearing with macOS on the
> >     guest
> >      > side anyway. And in an ideal world, we'd pass through the rendered
> >      > texture directly to the Cocoa UI code rather than copying out
> >     only for
> >      > the CPU to draw it back into a window surface which is then
> >     passed to
> >      > the GPU for host side rendering. But I felt this patch is already
> >     very,
> >      > very large, and if anyone cares, we can fix imperfections in
> >     subsequent
> >      > updates.
> >      >
> >      > (*)The rendering code in that device is also fairly complex, so I
> >     may be
> >      > misreading it.
> >
> >     QXL always modifies the surface with BQL. The surface is modified
> with
> >     qxl_blit(), which is called by qxl_render_update_area_unlocked().
> >     qxl_render_update_area_unlocked() is called by either of
> >     qxl_render_update() and qxl_render_update_area_bh(). Both of them are
> >     called with BQL. The name includes "unlocked", but it means it is
> >     called
> >     without holding QXL-internal lock.
> >
> >     Most devices works entirely with BQL so they don't perform double
> >     buffering. apple-gfx can do the same.
> >
> >
> > I think we can safely move apple-gfx's framebuffer state management back
> > inside the BQL, yes. I just figured that copying dozens of megabytes of
> > framebuffer data on every frame while holding the lock was not going to
> > help BQL contention. Especially as PVG does not have a concept of dirty
> > areas, so we must copy the whole framebuffer every time. (Unless we were
> > to implement dirty area detection ourselves.)
> >
> > Unfortunately, implementing double-buffering would require a major
> > rework of Qemu's whole surface management, console code, and probably
> > most of the UI implementations. I'm guessing the OpenGL fast-path
> > sidesteps all of this, so replicating that with Metal would probably be
> > the easier way forward. (Although doing all this graphics stuff inside
> > the BQL generally seems like a major architectural flaw; I suppose most
> > enterprise use of Qemu does not involve the framebuffer, so it's not
> > shown up in BQL contention profiling there. It certainly does in desktop
> > use, although at least on macOS hosts there are far worse culprits in
> > that regard.)
>
> We don't need double-buffering; instead we can have one thread that
> renders the UI and also
> [PGDisplay -encodeCurrentFrameToCommandBuffer:texture:region:].
> Rendering the UI in machine event loop in one thread is a bad idea in
> the first place so it will make sense to extract all UI work into a
> separate thread. This still requires a lot of work because the UI code
> assumes BQL everywhere.
>

I'm not reworking the UI layer in this patch set, so it sounds like the
best we can do is to acquire the BQL for updating the console framebuffer
and just take the lock contention hit.


> In my understanding BQL is less problematic for KVM users because QEMU
> can (sometimes) avoid locking BQL in the vCPU threads.
>

Yeah, the current hvf implementation is unfortunately still very BQL heavy.
Part of that is down to the interrupt controller: with the KVM accel, you
typically use the interrupt controller provided by the kernel. With the hvf
accel, you get a lot of VM exits for servicing interrupt controller I/O; at
least on x86, the APIC implementation only works inside the BQL. HVF on
x86-64 does technically have an in-kernel APIC implementation;
unfortunately, it has some severe bugs which prevent using it in the
general case. (EOI signalling to the IOAPIC does not work/FB14425590; the
behaviour for coalescing and retriggering of level-triggered interrupts is
also flawed/FB14425412.) On the aarch64 side, an in-kernel GIC
implementation was only just added to hvf with macOS 15.

I have submitted a patch that at least adds x2apic support to the hvf accel
and thus significantly reduces BQL pressure a few times, but the
maintainers of the x86 hvf backend don't seem to be active at the moment.


> >
> >      >
> >      >      > +        if (s->gfx_update_requested) {
> >      >      > +            s->gfx_update_requested = false;
> >      >      > +            dpy_gfx_update_full(s->con);
> >      >      > +            graphic_hw_update_done(s->con);
> >      >       > +            s->new_frame_ready = false;
> >      >
> >      >     This assignment is unnecessary as s->new_frame_ready is
> >     always false
> >      >     when s->gfx_update_requested. If you want to make sure
> >      >     s->gfx_update_requested and s->new_frame_ready are mutually
> >     exclusive,
> >      >     use one enum value instead of having two bools.
> >      >
> >      >
> >      > I'll need to refresh my memory and get back to you on this one,
> it's
> >      > been so many months since I actively worked on this code.
> >      >
> >      >      > +        } else {
> >      >      > +            s->new_frame_ready = true;
> >      >      > +        }
> >      >      > +    }
> >      >      > +    if (s->pending_frames > 0) {
> >      >      > +        apple_gfx_render_new_frame(s);
> >      >      > +    }
> >      >      > +}
> >      >      > +
> >      >      > +static void apple_gfx_fb_update_display(void *opaque)
> >      >      > +{
> >      >      > +    AppleGFXState *s = opaque;
> >      >      > +
> >      >      > +    dispatch_async(s->render_queue, ^{
> >      >      > +        if (s->pending_frames > 0) {
> >      >
> >      >     It should check for s->new_frame_ready as
> >      >     apple_gfx_render_frame_completed() doesn't check if
> >      >     s->pending_frames > 0 before calling
> >     graphic_hw_update_done(), which is
> >      >     inconsistent.
> >      >
> >      >
> >      > pending_frames is about guest-side frames that are queued to be
> >     rendered
> >      > by the host GPU.
> >      > new_frame_ready being true indicates that the contents of the Qemu
> >      > console surface has been updated with new frame data since the
> last
> >      > gfx_update.
> >      > gfx_update_requested indicates that gfx_update is currently
> >     awaiting an
> >      > async completion (graphic_hw_update_done) but the surface has not
> >      > received a new frame content, but the GPU is stily busy drawing
> one.
> >      >
> >      > apple_gfx_render_frame_completed is scheduled exactly once per
> >     pending
> >      > frame, so pending_frames > 0 is an invariant there. (Hence the
> >     assert.)> > I don't think there is any inconsistency here, but I'll
> >     double check.
> >      > It's possible that there's an easier way to express the state
> >     machine,
> >      > and I'll take a look at that.
> >
> >     I meant that apple_gfx_render_frame_completed() does not check if the
> >     frame is the last one currently pending.
> apple_gfx_fb_update_display()
> >     ignores a new ready frame when there is a more pending frame, but
> >     apple_gfx_render_frame_completed() unconditionally fires
> >     graphic_hw_update_done() even if there is a more pending frame. And I
> >     think apple_gfx_render_frame_completed() is right and
> >     apple_gfx_fb_update_display() is wrong in such a situation.
> >
> >
> > OK, got it. And yes, I agree.
> >
> >      >
> >      >     Checking if s->pending_frames > 0 both in
> >     apple_gfx_fb_update_display()
> >      >     and apple_gfx_render_frame_completed() is also problematic as
> >     that can
> >      >     defer graphic_hw_update_done() indefinitely if we are getting
> new
> >      >     frames
> >      >     too fast.
> >      >
> >      >
> >      > I see what you mean about this part. I'll have to test it, but I
> >     guess
> >      > we should reverse the priority, like this:
> >      >
> >      >          if (s->new_frame_ready) {
> >      >              dpy_gfx_update_full(s->con);
> >      >              s->new_frame_ready = false;
> >      >              graphic_hw_update_done(s->con);
> >      >          } else if (s->pending_frames > 0) {
> >      >              s->gfx_update_requested = true;
> >      >          } else {
> >      >              graphic_hw_update_done(s->con);
> >      >          }
> >      >
> >      > 1. If we already have a frame, ready to be displayed, return it
> >     immediately.
> >      > 2. If the guest has reported that it's completed a frame and the
> >     GPU is
> >      > currently busy rendering it, defer graphic_hw_update_done until
> >     it's done.
> >      > 3. If the guest reports no changes to its display, indicate this
> >     back to
> >      > Qemu as a no-op display update graphic_hw_update_done() with no
> >      > dpy_gfx_update* call.
> >
> >     Yes, that looks correct.
> >
> >      >
> >      >      > +            s->gfx_update_requested = true;
> >      >      > +        } else {
> >      >      > +            if (s->new_frame_ready) {
> >      >      > +                dpy_gfx_update_full(s->con);
> >      >      > +                s->new_frame_ready = false;
> >      >      > +            }
> >      >      > +            graphic_hw_update_done(s->con);
> >      >       > +        }> +    });
> >      >      > +}
> >      >      > +
> >      >      > +static const GraphicHwOps apple_gfx_fb_ops = {
> >      >      > +    .gfx_update = apple_gfx_fb_update_display,
> >      >      > +    .gfx_update_async = true,
> >      >      > +};
> >      >      > +
> >      >      > +static void update_cursor(AppleGFXState *s)
> >      >      > +{
> >      >      > +    dpy_mouse_set(s->con, s->pgdisp.cursorPosition.x,
> >      >      > +                  s->pgdisp.cursorPosition.y, s-
> >      >cursor_show);
> >      >      > +}
> >      >      > +
> >      >      > +static void set_mode(AppleGFXState *s, uint32_t width,
> >     uint32_t
> >      >     height)
> >      >      > +{
> >      >      > +    void *vram = NULL;
> >      >      > +    DisplaySurface *surface;
> >      >      > +    MTLTextureDescriptor *textureDescriptor;
> >      >      > +    id<MTLTexture> texture = nil;
> >      >      > +    __block bool no_change = false;
> >      >      > +
> >      >      > +    dispatch_sync(s->render_queue,
> >      >
> >      >     Calling dispatch_sync() while holding BQL may result in
> deadlock.
> >      >
> >      > Only if any code executed on the same dispatch queue acquires the
> >     lock
> >      > either directly or transitively. I believe I have ensure this is
> not
> >      > done on the reqnder_queue, have you found anywhere this is the
> case?
> >
> >     The documentation is not clear what threads a dispatch queue runs
> >     on. We
> >     can have a deadlock if they lock the BQL.
> >
> >
> > dispatch_sync is a synchronisation primitive (it waits for and asserts
> > exclusive access to the given queue), it doesn't actually do any thread
> > scheduling. Work scheduled asynchronously to non-main dispatch queues
> > will otherwise run on libdispatch pool threads. Running blocks on
> > dispatch queues will not preempt and schedule it on other threads which
> > may or may not be holding some locks.
>
> What if all pool threads are waiting for BQL?
>

The pool is elastic, so if threads are blocked (in syscalls) and there are
queued tasks ready to run, the runtime will create more threads as
necessary.


> >
> > So the only way this code will deadlock is if any code scheduled to
> > render_queue directly or transitively acquires the BQL. None of it does,
> > although updating the console while holding the BQL rather complicates
> this.
> >
> >      >
> >      >      > +        ^{
> >      >      > +            if (s->surface &&
> >      >      > +                width == surface_width(s->surface) &&
> >      >      > +                height == surface_height(s->surface)) {
> >      >      > +                no_change = true;
> >      >      > +            }
> >      >      > +        });
> >      >      > +
> >      >      > +    if (no_change) {
> >      >      > +        return;
> >      >      > +    }
> >      >      > +
> >      >      > +    vram = g_malloc0(width * height * 4);
> >      >      > +    surface = qemu_create_displaysurface_from(width,
> height,
> >      >     PIXMAN_LE_a8r8g8b8,
> >      >      > +                                              width * 4,
> >     vram);
> >      >      > +
> >      >      > +    @autoreleasepool {
> >      >      > +        textureDescriptor =
> >      >      > +            [MTLTextureDescriptor
> >      >      > +
> >      >     texture2DDescriptorWithPixelFormat:MTLPixelFormatBGRA8Unorm
> >      >      > +                                             width:width
> >      >      > +                                            height:height
> >      >      > +                                         mipmapped:NO];
> >      >      > +        textureDescriptor.usage = s-
> >      >pgdisp.minimumTextureUsage;
> >      >      > +        texture = [s->mtl
> >      >     newTextureWithDescriptor:textureDescriptor];
> >      >      > +    }
> >      >      > +
> >      >      > +    s->using_managed_texture_storage =
> >      >      > +        (texture.storageMode == MTLStorageModeManaged);
> >      >      > +
> >      >      > +    dispatch_sync(s->render_queue,
> >      >      > +        ^{
> >      >      > +            id<MTLTexture> old_texture = nil;
> >      >      > +            void *old_vram = s->vram;
> >      >      > +            s->vram = vram;
> >      >      > +            s->surface = surface;
> >      >      > +
> >      >      > +            dpy_gfx_replace_surface(s->con, surface);
> >      >      > +
> >      >      > +            old_texture = s->texture;
> >      >      > +            s->texture = texture;
> >      >      > +            [old_texture release];
> >      >
> >      >     You can just do:
> >      >     [s->texture release];
> >      >     s->texture = texture;
> >      >
> >      >     This will make s->texture dangling between the two
> >     statements, but that
> >      >     don't matter since s->texture is not an atomic variable that
> >     can be
> >      >     safely observed from another thread anyway.
> >      >
> >      >      > +
> >      >      > +            if (s->pending_frames == 0) {
> >      >      > +                g_free(old_vram);
> >      >      > +            }
> >      >      > +        });
> >      >      > +}
> >      >      > +
> >      >      > +static void create_fb(AppleGFXState *s)
> >      >      > +{
> >      >      > +    s->con = graphic_console_init(NULL, 0,
> >     &apple_gfx_fb_ops, s);
> >      >      > +    set_mode(s, 1440, 1080);
> >      >      > +
> >      >      > +    s->cursor_show = true;
> >      >      > +}
> >      >      > +
> >      >      > +static size_t apple_gfx_get_default_mmio_range_size(void)
> >      >      > +{
> >      >      > +    size_t mmio_range_size;
> >      >      > +    @autoreleasepool {
> >      >      > +        PGDeviceDescriptor *desc = [PGDeviceDescriptor
> new];
> >      >      > +        mmio_range_size = desc.mmioLength;
> >      >      > +        [desc release];
> >      >      > +    }
> >      >      > +    return mmio_range_size;
> >      >      > +}
> >      >      > +
> >      >      > +void apple_gfx_common_init(Object *obj, AppleGFXState *s,
> >     const
> >      >     char* obj_name)
> >      >
> >      >     This function can be merged into apple_gfx_common_realize().
> >      >
> >      >
> >      > Probably. I'll try it.
> >
> >
> > Upon further inspection, we need to call
> > cocoa_enable_runloop_on_main_thread() during the init phase, not
> > realize(). So we can't get rid of this entirely. Is there any value in
> > moving the other init code into _realize()?
>
> Calling cocoa_enable_runloop_on_main_thread() should be avoided even in
> apple_gfx_common_init(). QEMU can plug a device at runtime instead of
> initialization time, and in such a case, apple_gfx_common_init() will
> run after calling qemu_main.
>

Good point, it looks like I forgot to mark the mmio variant of apple-gfx
devices as non-hotpluggable; I've done that now. I strongly suspect the
guest driver doesn't support hotplugging anyway.


> I had a closer look and found it has a memory_region_init_io() call,
> which should remain in apple_gfx_common_init(). This leads to the same
> conclusion that we cannot remove this function so let's only move
> migrate_add_blocker() to apple_gfx_common_realize() to report its error.
>

Sounds good.



> >
> >      >      > +{
> >      >      > +    Error *local_err = NULL;
> >      >      > +    int r;
> >      >      > +    size_t mmio_range_size =
> >      >     apple_gfx_get_default_mmio_range_size();
> >      >      > +
> >      >      > +    trace_apple_gfx_common_init(obj_name,
> mmio_range_size);
> >      >      > +    memory_region_init_io(&s->iomem_gfx, obj,
> >     &apple_gfx_ops, s,
> >      >     obj_name,
> >      >      > +                          mmio_range_size);
> >      >      > +    s->iomem_gfx.disable_reentrancy_guard = true;
> >      >
> >      >     Why do you disable reentrancy guard?
> >      >
> >      >
> >      > Perhaps with the proposed AIO_WAIT_WHILE based I/O scheme, this
> >     won't be
> >      > an issue anymore, but the guard would otherwise keep dropping
> MMIOs
> >      > which immediately caused the PV graphics device to stop making
> >     progress.
> >      > The MMIO APIs in the PVG framework are thread- and reentrancy-
> >     safe, so
> >      > we certainly don't need to serialise them on our side.
> >
> >     It's better to understand why such reentrancy happens. Reentrancy
> >     itself
> >     is often a sign of bug.
> >
> >      >
> >      >      > +
> >      >      > +    /* TODO: PVG framework supports serialising device
> state:
> >      >     integrate it! */
> >      >      > +    if (apple_gfx_mig_blocker == NULL) {
> >      >      > +        error_setg(&apple_gfx_mig_blocker,
> >      >      > +                  "Migration state blocked by apple-gfx
> >     display
> >      >     device");
> >      >      > +        r = migrate_add_blocker(&apple_gfx_mig_blocker,
> >     &local_err);
> >      >      > +        if (r < 0) {
> >      >      > +            error_report_err(local_err);
> >      >
> >      >     Please report the error to the caller of
> >     apple_gfx_common_realize()
> >      >     instead.
> >      >
> >      >      > +        }
> >      >      > +    }
> >      >       > +}> +
> >      >      > +static void
> >      >     apple_gfx_register_task_mapping_handlers(AppleGFXState *s,
> >      >      > +
> >      >       PGDeviceDescriptor *desc)
> >      >      > +{
> >      >      > +    desc.createTask = ^(uint64_t vmSize, void * _Nullable
> *
> >      >     _Nonnull baseAddress) {
> >      >      > +        AppleGFXTask *task = apple_gfx_new_task(s,
> vmSize);
> >      >      > +        *baseAddress = (void*)task->address;
> >      >
> >      >     nit: please write as (void *) instead of (void*).
> >      >
> >      >      > +        trace_apple_gfx_create_task(vmSize, *baseAddress);
> >      >      > +        return task;
> >      >      > +    };
> >      >      > +
> >      >
> >      >      > +{
> >      >      > +    PGDisplayDescriptor *disp_desc = [PGDisplayDescriptor
> >     new];
> >      >      > +
> >      >      > + disp_desc.name <http://disp_desc.name> <http://
> >     disp_desc.name <http://disp_desc.name>> = @"QEMU display";
> >      >      > +    disp_desc.sizeInMillimeters = NSMakeSize(400.,
> >     300.); /* A
> >      >     20" display */
> >      >      > +    disp_desc.queue = dispatch_get_main_queue();
> >      >      > +    disp_desc.newFrameEventHandler = ^(void) {
> >      >      > +        trace_apple_gfx_new_frame();
> >      >      > +        dispatch_async(s->render_queue, ^{
> >      >      > +            /* Drop frames if we get too far ahead. */
> >      >      > +            if (s->pending_frames >= 2)
> >      >      > +                return;
> >      >      > +            ++s->pending_frames;
> >      >      > +            if (s->pending_frames > 1) {
> >      >      > +                return;
> >      >      > +            }
> >      >      > +            @autoreleasepool {
> >      >      > +                apple_gfx_render_new_frame(s);
> >      >      > +            }
> >      >      > +        });
> >      >      > +    };
> >      >      > +    disp_desc.modeChangeHandler = ^(PGDisplayCoord_t
> >     sizeInPixels,
> >      >      > +                                    OSType pixelFormat) {
> >      >      > +        trace_apple_gfx_mode_change(sizeInPixels.x,
> >     sizeInPixels.y);
> >      >      > +        set_mode(s, sizeInPixels.x, sizeInPixels.y);
> >      >      > +    };
> >      >      > +    disp_desc.cursorGlyphHandler = ^(NSBitmapImageRep
> *glyph,
> >      >      > +                                     PGDisplayCoord_t
> >     hotSpot) {
> >      >      > +        uint32_t bpp = glyph.bitsPerPixel;
> >      >      > +        size_t width = glyph.pixelsWide;
> >      >      > +        size_t height = glyph.pixelsHigh;
> >      >      > +        size_t padding_bytes_per_row = glyph.bytesPerRow
> >     - width
> >      >     * 4;
> >      >      > +        const uint8_t* px_data = glyph.bitmapData;
> >      >      > +
> >      >      > +        trace_apple_gfx_cursor_set(bpp, width, height);
> >      >      > +
> >      >      > +        if (s->cursor) {
> >      >      > +            cursor_unref(s->cursor);
> >      >      > +            s->cursor = NULL;
> >      >      > +        }
> >      >      > +
> >      >      > +        if (bpp == 32) { /* Shouldn't be anything else,
> >     but just
> >      >     to be safe...*/
> >      >      > +            s->cursor = cursor_alloc(width, height);
> >      >      > +            s->cursor->hot_x = hotSpot.x;
> >      >      > +            s->cursor->hot_y = hotSpot.y;
> >      >      > +
> >      >      > +            uint32_t *dest_px = s->cursor->data;
> >      >      > +
> >      >      > +            for (size_t y = 0; y < height; ++y) {
> >      >      > +                for (size_t x = 0; x < width; ++x) {
> >      >      > +                    /* NSBitmapImageRep's red & blue
> channels
> >      >     are swapped
> >      >      > +                     * compared to QEMUCursor's. */
> >      >      > +                    *dest_px =
> >      >      > +                        (px_data[0] << 16u) |
> >      >      > +                        (px_data[1] <<  8u) |
> >      >      > +                        (px_data[2] <<  0u) |
> >      >      > +                        (px_data[3] << 24u);
> >      >      > +                    ++dest_px;
> >      >      > +                    px_data += 4;
> >      >      > +                }
> >      >      > +                px_data += padding_bytes_per_row;
> >      >      > +            }
> >      >      > +            dpy_cursor_define(s->con, s->cursor);
> >      >      > +            update_cursor(s);
> >      >      > +        }
> >      >      > +    };
> >      >      > +    disp_desc.cursorShowHandler = ^(BOOL show) {
> >      >      > +        trace_apple_gfx_cursor_show(show);
> >      >      > +        s->cursor_show = show;
> >      >      > +        update_cursor(s);
> >      >      > +    };
> >      >      > +    disp_desc.cursorMoveHandler = ^(void) {
> >      >      > +        trace_apple_gfx_cursor_move();
> >      >      > +        update_cursor(s);
> >      >      > +    };
> >      >      > +
> >      >      > +    return disp_desc;
> >      >      > +}
> >      >      > +
> >      >      > +static NSArray<PGDisplayMode*>*
> >      >     apple_gfx_prepare_display_mode_array(void)
> >      >      > +{
> >      >      > +    PGDisplayMode *modes[ARRAY_SIZE(apple_gfx_modes)];
> >      >      > +    NSArray<PGDisplayMode*>* mode_array = nil;
> >      >      > +    int i;
> >      >      > +
> >      >      > +    for (i = 0; i < ARRAY_SIZE(apple_gfx_modes); i++) {
> >      >      > +        modes[i] =
> >      >      > +            [[PGDisplayMode alloc]
> >      >     initWithSizeInPixels:apple_gfx_modes[i] refreshRateInHz:60.];
> >      >      > +    }
> >      >      > +
> >      >      > +    mode_array = [NSArray arrayWithObjects:modes
> >      >     count:ARRAY_SIZE(apple_gfx_modes)];
> >      >      > +
> >      >      > +    for (i = 0; i < ARRAY_SIZE(apple_gfx_modes); i++) {
> >      >      > +        [modes[i] release];
> >      >      > +        modes[i] = nil;
> >      >      > +    }
> >      >      > +
> >      >      > +    return mode_array;
> >      >      > +}
> >      >      > +
> >      >      > +static id<MTLDevice> copy_suitable_metal_device(void)
> >      >      > +{
> >      >      > +    id<MTLDevice> dev = nil;
> >      >      > +    NSArray<id<MTLDevice>> *devs = MTLCopyAllDevices();
> >      >      > +
> >      >      > +    /* Prefer a unified memory GPU. Failing that, pick a
> non-
> >      >     removable GPU. */
> >      >      > +    for (size_t i = 0; i < devs.count; ++i) {
> >      >      > +        if (devs[i].hasUnifiedMemory) {
> >      >      > +            dev = devs[i];
> >      >      > +            break;
> >      >      > +        }
> >      >      > +        if (!devs[i].removable) {
> >      >      > +            dev = devs[i];
> >      >      > +        }
> >      >      > +    }
> >      >      > +
> >      >      > +    if (dev != nil) {
> >      >      > +        [dev retain];
> >      >      > +    } else {
> >      >      > +        dev = MTLCreateSystemDefaultDevice();
> >      >      > +    }
> >      >      > +    [devs release];
> >      >      > +
> >      >      > +    return dev;
> >      >      > +}
> >      >      > +
> >      >      > +void apple_gfx_common_realize(AppleGFXState *s,
> >      >     PGDeviceDescriptor *desc)
> >      >      > +{
> >      >      > +    PGDisplayDescriptor *disp_desc = nil;
> >      >      > +
> >      >      > +    QTAILQ_INIT(&s->tasks);
> >      >      > +    s->render_queue = dispatch_queue_create("apple-
> >     gfx.render",
> >      >      > +
> >     DISPATCH_QUEUE_SERIAL);
> >      >      > +    s->mtl = copy_suitable_metal_device();
> >      >      > +    s->mtl_queue = [s->mtl newCommandQueue];
> >      >      > +
> >      >      > +    desc.device = s->mtl;
> >      >      > +
> >      >      > +    apple_gfx_register_task_mapping_handlers(s, desc);
> >      >      > +
> >      >      > +    s->pgdev = PGNewDeviceWithDescriptor(desc);
> >      >      > +
> >      >      > +    disp_desc = apple_gfx_prepare_display_handlers(s);
> >      >      > +    s->pgdisp = [s->pgdev
> newDisplayWithDescriptor:disp_desc
> >      >      > +                                              port:0
> >      >     serialNum:1234];
> >      >      > +    [disp_desc release];
> >      >      > +    s->pgdisp.modeList =
> >     apple_gfx_prepare_display_mode_array();
> >      >      > +
> >      >      > +    create_fb(s);
> >      >      > +}
> >      >      > diff --git a/hw/display/meson.build
> b/hw/display/meson.build
> >      >      > index 7db05eace97..70d855749c0 100644
> >      >      > --- a/hw/display/meson.build
> >      >      > +++ b/hw/display/meson.build
> >      >      > @@ -65,6 +65,8 @@ system_ss.add(when: 'CONFIG_ARTIST',
> >     if_true:
> >      >     files('artist.c'))
> >      >      >
> >      >      >   system_ss.add(when: 'CONFIG_ATI_VGA', if_true:
> >     [files('ati.c',
> >      >     'ati_2d.c', 'ati_dbg.c'), pixman])
> >      >      >
> >      >      > +system_ss.add(when: 'CONFIG_MAC_PVG',         if_true:
> >      >     [files('apple-gfx.m'), pvg, metal])
> >      >      > +system_ss.add(when: 'CONFIG_MAC_PVG_VMAPPLE', if_true:
> >      >     [files('apple-gfx-vmapple.m'), pvg, metal])
> >      >      >
> >      >      >   if config_all_devices.has_key('CONFIG_VIRTIO_GPU')
> >      >      >     virtio_gpu_ss = ss.source_set()
> >      >      > diff --git a/hw/display/trace-events
> b/hw/display/trace-events
> >      >      > index 781f8a33203..1809a358e36 100644
> >      >      > --- a/hw/display/trace-events
> >      >      > +++ b/hw/display/trace-events
> >      >      > @@ -191,3 +191,29 @@ dm163_bits_ppi(unsigned dest_width)
> >      >     "dest_width : %u"
> >      >      >   dm163_leds(int led, uint32_t value) "led %d: 0x%x"
> >      >      >   dm163_channels(int channel, uint8_t value) "channel %d:
> >     0x%x"
> >      >      >   dm163_refresh_rate(uint32_t rr) "refresh rate %d"
> >      >      > +
> >      >      > +# apple-gfx.m
> >      >      > +apple_gfx_read(uint64_t offset, uint64_t res)
> >      >     "offset=0x%"PRIx64" res=0x%"PRIx64
> >      >      > +apple_gfx_write(uint64_t offset, uint64_t val)
> >      >     "offset=0x%"PRIx64" val=0x%"PRIx64
> >      >      > +apple_gfx_create_task(uint32_t vm_size, void *va)
> >     "vm_size=0x%x
> >      >     base_addr=%p"
> >      >      > +apple_gfx_destroy_task(void *task) "task=%p"
> >      >      > +apple_gfx_map_memory(void *task, uint32_t range_count,
> >     uint64_t
> >      >     virtual_offset, uint32_t read_only) "task=%p range_count=0x%x
> >      >     virtual_offset=0x%"PRIx64" read_only=%d"
> >      >      > +apple_gfx_map_memory_range(uint32_t i, uint64_t phys_addr,
> >      >     uint64_t phys_len) "[%d] phys_addr=0x%"PRIx64"
> >     phys_len=0x%"PRIx64
> >      >      > +apple_gfx_remap(uint64_t retval, uint64_t source, uint64_t
> >      >     target) "retval=%"PRId64" source=0x%"PRIx64" target=0x%"PRIx64
> >      >      > +apple_gfx_unmap_memory(void *task, uint64_t
> virtual_offset,
> >      >     uint64_t length) "task=%p virtual_offset=0x%"PRIx64"
> >     length=0x%"PRIx64
> >      >      > +apple_gfx_read_memory(uint64_t phys_address, uint64_t
> length,
> >      >     void *dst) "phys_addr=0x%"PRIx64" length=0x%"PRIx64" dest=%p"
> >      >      > +apple_gfx_raise_irq(uint32_t vector) "vector=0x%x"
> >      >      > +apple_gfx_new_frame(void) ""
> >      >      > +apple_gfx_mode_change(uint64_t x, uint64_t y) "x=%"PRId64"
> >      >     y=%"PRId64
> >      >      > +apple_gfx_cursor_set(uint32_t bpp, uint64_t width,
> uint64_t
> >      >     height) "bpp=%d width=%"PRId64" height=0x%"PRId64
> >      >      > +apple_gfx_cursor_show(uint32_t show) "show=%d"
> >      >      > +apple_gfx_cursor_move(void) ""
> >      >      > +apple_gfx_common_init(const char *device_name, size_t
> >     mmio_size)
> >      >     "device: %s; MMIO size: %zu bytes"
> >      >      > +
> >      >      > +# apple-gfx-vmapple.m
> >      >      > +apple_iosfc_read(uint64_t offset, uint64_t res)
> >      >     "offset=0x%"PRIx64" res=0x%"PRIx64
> >      >      > +apple_iosfc_write(uint64_t offset, uint64_t val)
> >      >     "offset=0x%"PRIx64" val=0x%"PRIx64
> >      >      > +apple_iosfc_map_memory(uint64_t phys, uint64_t len,
> >     uint32_t ro,
> >      >     void *va, void *e, void *f) "phys=0x%"PRIx64" len=0x%"PRIx64"
> >     ro=%d
> >      >     va=%p e=%p f=%p"
> >      >      > +apple_iosfc_unmap_memory(void *a, void *b, void *c, void
> *d,
> >      >     void *e, void *f) "a=%p b=%p c=%p d=%p e=%p f=%p"
> >      >      > +apple_iosfc_raise_irq(uint32_t vector) "vector=0x%x"
> >      >      > +
> >      >      > diff --git a/meson.build b/meson.build
> >      >      > index 10464466ff3..f09df3f09d5 100644
> >      >      > --- a/meson.build
> >      >      > +++ b/meson.build
> >      >      > @@ -741,6 +741,8 @@ socket = []
> >      >      >   version_res = []
> >      >      >   coref = []
> >      >      >   iokit = []
> >      >      > +pvg = []
> >      >      > +metal = []
> >      >      >   emulator_link_args = []
> >      >      >   midl = not_found
> >      >      >   widl = not_found
> >      >      > @@ -762,6 +764,8 @@ elif host_os == 'darwin'
> >      >      >     coref = dependency('appleframeworks', modules:
> >     'CoreFoundation')
> >      >      >     iokit = dependency('appleframeworks', modules: 'IOKit',
> >      >     required: false)
> >      >      >     host_dsosuf = '.dylib'
> >      >      > +  pvg = dependency('appleframeworks', modules:
> >      >     'ParavirtualizedGraphics')
> >      >      > +  metal = dependency('appleframeworks', modules: 'Metal')
> >      >      >   elif host_os == 'sunos'
> >      >      >     socket = [cc.find_library('socket'),
> >      >      >               cc.find_library('nsl'),
> >      >
> >
>
>

[-- Attachment #2: Type: text/html, Size: 57954 bytes --]

Re: [PATCH v3 12/14] hw/vmapple/cfg: Introduce vmapple cfg region

[Akihiko Odaki]

On 2024/10/09 22:08, Phil Dennis-Jordan wrote:
> 
> 
> On Mon, 7 Oct 2024 at 20:04, Akihiko Odaki <akihiko.odaki@daynix.com 
> <mailto:akihiko.odaki@daynix.com>> wrote:
> 
>     On 2024/10/07 23:10, Phil Dennis-Jordan wrote:
>      >
>      >
>      > On Sat, 5 Oct 2024 at 07:35, Akihiko Odaki
>     <akihiko.odaki@daynix.com <mailto:akihiko.odaki@daynix.com>
>      > <mailto:akihiko.odaki@daynix.com
>     <mailto:akihiko.odaki@daynix.com>>> wrote:
>      >
>      >     On 2024/09/28 17:57, Phil Dennis-Jordan wrote:
>      >      > From: Alexander Graf <graf@amazon.com
>     <mailto:graf@amazon.com> <mailto:graf@amazon.com
>     <mailto:graf@amazon.com>>>
>      >      >
>      >      > Instead of device tree or other more standardized means,
>     VMApple
>      >     passes
>      >      > platform configuration to the first stage boot loader in a
>     binary
>      >     encoded
>      >      > format that resides at a dedicated RAM region in physical
>     address
>      >     space.
>      >      >
>      >      > This patch models this configuration space as a qdev
>     device which
>      >     we can
>      >      > then map at the fixed location in the address space. That
>     way, we can
>      >      > influence and annotate all configuration fields easily.
>      >      >
>      >      > Signed-off-by: Alexander Graf <graf@amazon.com
>     <mailto:graf@amazon.com>
>      >     <mailto:graf@amazon.com <mailto:graf@amazon.com>>>
>      >      > Signed-off-by: Phil Dennis-Jordan <phil@philjordan.eu
>     <mailto:phil@philjordan.eu>
>      >     <mailto:phil@philjordan.eu <mailto:phil@philjordan.eu>>>
>      >      >
>      >      > ---
>      >      > v3:
>      >      >
>      >      >   * Replaced legacy device reset method with Resettable method
>      >      >
>      >      >   hw/vmapple/Kconfig       |   3 ++
>      >      >   hw/vmapple/cfg.c         | 106 +++++++++++++++++++++++++
>     +++++++
>      >     +++++++
>      >      >   hw/vmapple/meson.build   |   1 +
>      >      >   include/hw/vmapple/cfg.h |  68 +++++++++++++++++++++++++
>      >      >   4 files changed, 178 insertions(+)
>      >      >   create mode 100644 hw/vmapple/cfg.c
>      >      >   create mode 100644 include/hw/vmapple/cfg.h
>      >      >
>      >      > diff --git a/hw/vmapple/Kconfig b/hw/vmapple/Kconfig
>      >      > index 68f88876eb9..8bbeb9a9237 100644
>      >      > --- a/hw/vmapple/Kconfig
>      >      > +++ b/hw/vmapple/Kconfig
>      >      > @@ -4,3 +4,6 @@ config VMAPPLE_AES
>      >      >   config VMAPPLE_BDIF
>      >      >       bool
>      >      >
>      >      > +config VMAPPLE_CFG
>      >      > +    bool
>      >      > +
>      >      > diff --git a/hw/vmapple/cfg.c b/hw/vmapple/cfg.c
>      >      > new file mode 100644
>      >      > index 00000000000..a5e5c62f59f
>      >      > --- /dev/null
>      >      > +++ b/hw/vmapple/cfg.c
>      >      > @@ -0,0 +1,106 @@
>      >      > +/*
>      >      > + * VMApple Configuration Region
>      >      > + *
>      >      > + * Copyright © 2023 Amazon.com, Inc. or its affiliates. All
>      >     Rights Reserved.
>      >      > + *
>      >      > + * This work is licensed under the terms of the GNU GPL,
>     version
>      >     2 or later.
>      >      > + * See the COPYING file in the top-level directory.
>      >      > + */
>      >      > +
>      >      > +#include "qemu/osdep.h"
>      >      > +#include "hw/vmapple/cfg.h"
>      >      > +#include "qemu/log.h"
>      >      > +#include "qemu/module.h"
>      >      > +#include "qapi/error.h"
>      >      > +
>      >      > +static void vmapple_cfg_reset(Object *obj, ResetType type)
>      >      > +{
>      >      > +    VMAppleCfgState *s = VMAPPLE_CFG(obj);
>      >      > +    VMAppleCfg *cfg;
>      >      > +
>      >      > +    cfg = memory_region_get_ram_ptr(&s->mem);
>      >      > +    memset((void *)cfg, 0, VMAPPLE_CFG_SIZE);
>      >      > +    *cfg = s->cfg;
>      >       > +}> +
>      >      > +static void vmapple_cfg_realize(DeviceState *dev, Error
>     **errp)
>      >      > +{
>      >      > +    VMAppleCfgState *s = VMAPPLE_CFG(dev);
>      >      > +    uint32_t i;
>      >      > +
>      >      > +    strncpy(s->cfg.serial, s->serial, sizeof(s->cfg.serial));
>      >      > +    strncpy(s->cfg.model, s->model, sizeof(s->cfg.model));
>      >      > +    strncpy(s->cfg.soc_name, s->soc_name, sizeof(s-
>      >cfg.soc_name));
>      >      > +    strncpy(s->cfg.unk8, "D/A", sizeof(s->cfg.soc_name));
>      >
>      >     Use qemu_strnlen() to report an error for too long strings.
>      >
>      >
>      > Hmm, I don't see any existing instances of such a pattern. I do
>     however
>      > see a couple of uses of g_strlcpy in the Qemu codebase - that
>     would be a
>      > better candidate for error checked string copying, though it still
>      > involves some awkward return value checks. I'm going to wrap that
>     in a
>      > helper function and macro to replace all 4 strncpy instances
>     here. If
>      > the same thing is useful elsewhere later, it can be promoted to
>     cutils
>      > or similar.
> 
>     g_strlcpy() internally performs strlen(), which is worse than
>     qemu_strnlen().
> 
> 
> Worse in what sense? It really depends what you're defending against. 
> Sure, strlcpy blows up if the source string isn't nul-terminated. All 
> the source strings here are expected to be nul-terminated here though.
> 
>     It is nice to have a helper function. Linux also has something similar
>     called strscpy():
>     https://www.kernel.org/doc/html/latest/core-api/kernel-
>     api.html#c.strscpy <https://www.kernel.org/doc/html/latest/core-api/
>     kernel-api.html#c.strscpy>
> 
> 
> I'm not convinced this is the right patch set to be reinventing Qemu's 
> string copying functions. Instances abound of the even less safe 
> strcpy() (and sprintf, etc.) being used in Qemu. Some of those are 
> likely even problematic, but it seems that should be subject to a more 
> holistic investigation into what kinds of usage patterns there are, and 
> then a handful of safe solutions can be found that would work for 99% of 
> string copying in the code base. Not by adding yet another strcpy 
> variant in a device backend and machine type patch set where one of the 
> existing variants works just fine.

My suggestion were for the case you were going to write one to avoid 
cumbersome return value checks as you told in the earlier email. It is 
fine if your helper function is to be specific to this device to add 
device-specific error reporting code, for example.

> 
>      >
>      > (Also, I notice that last strncpy actually uses the wrong
>     destination
>      > size; my wrapper macro uses ARRAY_SIZE to avoid this mistake
>     altogether.)
>      >
>      >      > +    s->cfg.ecid = cpu_to_be64(s->cfg.ecid);
>      >      > +    s->cfg.version = 2;
>      >      > +    s->cfg.unk1 = 1;
>      >      > +    s->cfg.unk2 = 1;
>      >      > +    s->cfg.unk3 = 0x20;
>      >      > +    s->cfg.unk4 = 0;
>      >      > +    s->cfg.unk5 = 1;
>      >      > +    s->cfg.unk6 = 1;
>      >      > +    s->cfg.unk7 = 0;
>      >      > +    s->cfg.unk10 = 1;
>      >      > +
>      >      > +    g_assert(s->cfg.nr_cpus < ARRAY_SIZE(s->cfg.cpu_ids));
>      >
>      >     Report an error instead of asserting.
>      >
>      >      > +    for (i = 0; i < s->cfg.nr_cpus; i++) {
>      >      > +        s->cfg.cpu_ids[i] = i;
>      >      > +    }
>      >       > +}> +
>      >      > +static void vmapple_cfg_init(Object *obj)
>      >      > +{
>      >      > +    VMAppleCfgState *s = VMAPPLE_CFG(obj);
>      >      > +
>      >      > +    memory_region_init_ram(&s->mem, obj, "VMApple Config",
>      >     VMAPPLE_CFG_SIZE,
>      >      > +                           &error_fatal);
>      >      > +    sysbus_init_mmio(SYS_BUS_DEVICE(obj), &s->mem);
>      >      > +
>      >      > +    s->serial = (char *)"1234";
>      >      > +    s->model = (char *)"VM0001";
>      >      > +    s->soc_name = (char *)"Apple M1 (Virtual)";
>      >
>      >     These casts are unsafe; these pointers will be freed when this
>      >     device is
>      >     freed.
>      >
>      >
>      > Good catch! The more usual pattern for default string property
>     values
>      > seems to be to fill them in _realize() (using g_strdup()) if no
>     other
>      > value was previously set, so I've applied that here for the next
>     version
>      > of the patch.
>      >
>      >
>      >      > +}
>      >      > +
>      >      > +static Property vmapple_cfg_properties[] = {
>      >      > +    DEFINE_PROP_UINT32("nr-cpus", VMAppleCfgState,
>     cfg.nr_cpus, 1),
>      >      > +    DEFINE_PROP_UINT64("ecid", VMAppleCfgState, cfg.ecid, 0),
>      >      > +    DEFINE_PROP_UINT64("ram-size", VMAppleCfgState,
>      >     cfg.ram_size, 0),
>      >      > +    DEFINE_PROP_UINT32("run_installer1", VMAppleCfgState,
>      >     cfg.run_installer1, 0),
>      >      > +    DEFINE_PROP_UINT32("run_installer2", VMAppleCfgState,
>      >     cfg.run_installer2, 0),
>      >      > +    DEFINE_PROP_UINT32("rnd", VMAppleCfgState, cfg.rnd, 0),
>      >      > +    DEFINE_PROP_MACADDR("mac-en0", VMAppleCfgState,
>     cfg.mac_en0),
>      >      > +    DEFINE_PROP_MACADDR("mac-en1", VMAppleCfgState,
>     cfg.mac_en1),
>      >      > +    DEFINE_PROP_MACADDR("mac-wifi0", VMAppleCfgState,
>      >     cfg.mac_wifi0),
>      >      > +    DEFINE_PROP_MACADDR("mac-bt0", VMAppleCfgState,
>     cfg.mac_bt0),
>      >      > +    DEFINE_PROP_STRING("serial", VMAppleCfgState, serial),
>      >      > +    DEFINE_PROP_STRING("model", VMAppleCfgState, model),
>      >      > +    DEFINE_PROP_STRING("soc_name", VMAppleCfgState,
>     soc_name),
>      >      > +    DEFINE_PROP_END_OF_LIST(),
>      >      > +};
>      >      > +
>      >      > +static void vmapple_cfg_class_init(ObjectClass *klass,
>     void *data)
>      >      > +{
>      >      > +    DeviceClass *dc = DEVICE_CLASS(klass);
>      >      > +    ResettableClass *rc = RESETTABLE_CLASS(klass);
>      >      > +
>      >      > +    dc->realize = vmapple_cfg_realize;
>      >      > +    dc->desc = "VMApple Configuration Region";
>      >      > +    device_class_set_props(dc, vmapple_cfg_properties);
>      >      > +    rc->phases.hold = vmapple_cfg_reset;
>      >      > +}
>      >      > +
>      >      > +static const TypeInfo vmapple_cfg_info = {
>      >      > +    .name          = TYPE_VMAPPLE_CFG,
>      >      > +    .parent        = TYPE_SYS_BUS_DEVICE,
>      >      > +    .instance_size = sizeof(VMAppleCfgState),
>      >      > +    .instance_init = vmapple_cfg_init,
>      >      > +    .class_init    = vmapple_cfg_class_init,
>      >      > +};
>      >      > +
>      >      > +static void vmapple_cfg_register_types(void)
>      >      > +{
>      >      > +    type_register_static(&vmapple_cfg_info);
>      >      > +}
>      >      > +
>      >      > +type_init(vmapple_cfg_register_types)
>      >      > diff --git a/hw/vmapple/meson.build b/hw/vmapple/meson.build
>      >      > index d4624713deb..64b78693a31 100644
>      >      > --- a/hw/vmapple/meson.build
>      >      > +++ b/hw/vmapple/meson.build
>      >      > @@ -1,2 +1,3 @@
>      >      >   system_ss.add(when: 'CONFIG_VMAPPLE_AES',  if_true:
>     files('aes.c'))
>      >      >   system_ss.add(when: 'CONFIG_VMAPPLE_BDIF', if_true:
>      >     files('bdif.c'))
>      >      > +system_ss.add(when: 'CONFIG_VMAPPLE_CFG',  if_true:
>     files('cfg.c'))
>      >      > diff --git a/include/hw/vmapple/cfg.h b/include/hw/
>     vmapple/cfg.h
>      >      > new file mode 100644
>      >      > index 00000000000..3337064e447
>      >      > --- /dev/null
>      >      > +++ b/include/hw/vmapple/cfg.h
>      >      > @@ -0,0 +1,68 @@
>      >      > +/*
>      >      > + * VMApple Configuration Region
>      >      > + *
>      >      > + * Copyright © 2023 Amazon.com, Inc. or its affiliates. All
>      >     Rights Reserved.
>      >      > + *
>      >      > + * This work is licensed under the terms of the GNU GPL,
>     version
>      >     2 or later.
>      >      > + * See the COPYING file in the top-level directory.
>      >      > + */
>      >      > +
>      >      > +#ifndef HW_VMAPPLE_CFG_H
>      >      > +#define HW_VMAPPLE_CFG_H
>      >      > +
>      >      > +#include "hw/sysbus.h"
>      >      > +#include "qom/object.h"
>      >      > +#include "net/net.h"
>      >      > +
>      >      > +typedef struct VMAppleCfg {
>      >      > +    uint32_t version;         /* 0x000 */
>      >      > +    uint32_t nr_cpus;         /* 0x004 */
>      >      > +    uint32_t unk1;            /* 0x008 */
>      >      > +    uint32_t unk2;            /* 0x00c */
>      >      > +    uint32_t unk3;            /* 0x010 */
>      >      > +    uint32_t unk4;            /* 0x014 */
>      >      > +    uint64_t ecid;            /* 0x018 */
>      >      > +    uint64_t ram_size;        /* 0x020 */
>      >      > +    uint32_t run_installer1;  /* 0x028 */
>      >      > +    uint32_t unk5;            /* 0x02c */
>      >      > +    uint32_t unk6;            /* 0x030 */
>      >      > +    uint32_t run_installer2;  /* 0x034 */
>      >      > +    uint32_t rnd;             /* 0x038 */
>      >      > +    uint32_t unk7;            /* 0x03c */
>      >      > +    MACAddr mac_en0;          /* 0x040 */
>      >      > +    uint8_t pad1[2];
>      >      > +    MACAddr mac_en1;          /* 0x048 */
>      >      > +    uint8_t pad2[2];
>      >      > +    MACAddr mac_wifi0;        /* 0x050 */
>      >      > +    uint8_t pad3[2];
>      >      > +    MACAddr mac_bt0;          /* 0x058 */
>      >      > +    uint8_t pad4[2];
>      >      > +    uint8_t reserved[0xa0];   /* 0x060 */
>      >      > +    uint32_t cpu_ids[0x80];   /* 0x100 */
>      >      > +    uint8_t scratch[0x200];   /* 0x180 */
>      >      > +    char serial[32];          /* 0x380 */
>      >      > +    char unk8[32];            /* 0x3a0 */
>      >      > +    char model[32];           /* 0x3c0 */
>      >      > +    uint8_t unk9[32];         /* 0x3e0 */
>      >      > +    uint32_t unk10;           /* 0x400 */
>      >      > +    char soc_name[32];        /* 0x404 */
>      >      > +} VMAppleCfg;
>      >      > +
>      >      > +#define TYPE_VMAPPLE_CFG "vmapple-cfg"
>      >      > +OBJECT_DECLARE_SIMPLE_TYPE(VMAppleCfgState, VMAPPLE_CFG)
>      >      > +
>      >      > +struct VMAppleCfgState {
>      >      > +    /* <private> */
>      >      > +    SysBusDevice parent_obj;
>      >      > +    VMAppleCfg cfg;
>      >      > +
>      >      > +    /* <public> */
>      >      > +    MemoryRegion mem;
>      >      > +    char *serial;
>      >      > +    char *model;
>      >      > +    char *soc_name;
>      >      > +};
>      >      > +
>      >      > +#define VMAPPLE_CFG_SIZE 0x00010000
>      >      > +
>      >      > +#endif /* HW_VMAPPLE_CFG_H */
>      >
> 

Re: [PATCH v3 00/14] macOS PV Graphics and new vmapple machine type

[Phil Dennis-Jordan]

[-- Attachment #1: Type: text/plain, Size: 1413 bytes --]

On Thu, 3 Oct 2024 at 10:06, Alex Bennée <alex.bennee@linaro.org> wrote:

>
> > There are currently a few limitations to this which aren't intrinsic,
> > just imperfect emulation of the VZF, but it's good enough to be just
> > about usable for some purposes:
> >
> >  * macOS 12 guests only. Versions 13+ currently fail during early boot.
> >  * macOS 11+ arm64 hosts only, with hvf accel. (Perhaps some differences
> >    between Apple M series CPUs and TCG's aarch64 implementation? macOS
> >    hosts only because ParavirtualizedGraphics.framework is a black box
> >    implementing most of the logic behind the apple-gfx device.)
>
> We don't currently have TCG CPU models for the Apple Silicon processors.
> They are not too hard to add (basically setting the correct ID register
> bits, c.f. aarch64_neoverse_n1_initfn for an example). However that
> would only cover Aarch64 architectural features. We do no modelling of
> the extra instructions that Apple added (although in theory that should
> only be run in Apples own ML libraries).
>

This really isn't my area of expertise, and I don't see myself attempting
to make it work with TCG. Given that the OS only boots with the PV graphics
device, you can only really use this machine type on a macOS host, so there
aren't many reasons to use TCG over HVF. I suppose it might make debugging
the myriad other rough edges easier!

[-- Attachment #2: Type: text/html, Size: 1862 bytes --]