[Qemu-devel] [RFC 0/6] vSMMU initialization

[Qemu-devel] [RFC 0/6] vSMMU initialization

[Baptiste Reynal]

The ARM SMMU has support for 2-stages address translations, allowing a virtual
address to be translated at two levels:
- Stage 1 translates a virtual address (VA) into an intermediate physical
address (IPA)
- Stage 2 translates an IPA into a physical address (PA)

Will Deacon introduced a virtual SMMU interface for KVM, which gives a virtual
machine the possibility to use an IOMMU with native drivers. While the VM will
program the first stage of translation (stage 1), the interface will program
the second (stage 2) on the physical SMMU.

This RFC brings support for the vSMMU on QEMU-side, introducing the code for
vSMMU initialization and the device tree operations needed to attach devices
to it.

The main purpose of such vSMMU device is to allow VFIO devices to benefit
from an IOMMU on the guest side, for userspace drivers or nested VFIO.

It is an early state of development so:
- Interrupts are not handled for now
- Only one device can be bound to the vSMMU

I'm open to any comments or suggestions.

Baptiste Reynal (6):
  headers sync
  hw/core/platform-bus: initialization notifier
  hw/core/platform-bus: add base_address field
  hw/vfio: vsmmu device
  hw/arm/sysbus-fdt: enable vsmmu dynamic instantiation
  hw/arm/sysbus-fdt: add smmu masters in device tree

 hw/arm/sysbus-fdt.c             |  78 ++++++++++++++++++++
 hw/arm/virt.c                   |   4 +
 hw/core/platform-bus.c          |  13 ++++
 hw/vfio/Makefile.objs           |   1 +
 hw/vfio/common.c                |   8 +-
 hw/vfio/platform.c              |   2 +
 hw/vfio/smmu.c                  | 157 ++++++++++++++++++++++++++++++++++++++++
 include/hw/platform-bus.h       |   3 +
 include/hw/vfio/vfio-platform.h |   1 +
 include/hw/vfio/vfio-smmu.h     |  50 +++++++++++++
 linux-headers/linux/kvm.h       |  15 ++++
 11 files changed, 330 insertions(+), 2 deletions(-)
 create mode 100644 hw/vfio/smmu.c
 create mode 100644 include/hw/vfio/vfio-smmu.h

-- 
2.4.3

[Qemu-devel] [RFC 1/6] headers sync

[Baptiste Reynal]

Signed-off-by: Baptiste Reynal <b.reynal@virtualopensystems.com>
---
 linux-headers/linux/kvm.h | 15 +++++++++++++++
 1 file changed, 15 insertions(+)

diff --git a/linux-headers/linux/kvm.h b/linux-headers/linux/kvm.h
index fad9e5c..d48d7b1 100644
--- a/linux-headers/linux/kvm.h
+++ b/linux-headers/linux/kvm.h
@@ -1003,6 +1003,19 @@ struct kvm_device_attr {
 #define   KVM_DEV_VFIO_GROUP_ADD			1
 #define   KVM_DEV_VFIO_GROUP_DEL			2
 
+#define  KVM_DEV_ARM_SMMU_V2_CFG		1
+#define   KVM_DEV_ARM_SMMU_V2_CFG_INIT			1
+#define   KVM_DEV_ARM_SMMU_V2_CFG_IRQ			2
+#define   KVM_DEV_ARM_SMMU_V2_CFG_SIZE			3
+#define  KVM_DEV_ARM_SMMU_V2_VFIO		2
+#define   KVM_DEV_ARM_SMMU_V2_VFIO_GROUP_ADD		1
+#define   KVM_DEV_ARM_SMMU_V2_VFIO_GROUP_DEL		2
+
+struct arm_smmu_v2_vfio_group_sid {
+	__u32	fd;	/* file descriptor of the VFIO group */
+	__u16	sid;
+};
+
 enum kvm_device_type {
 	KVM_DEV_TYPE_FSL_MPIC_20	= 1,
 #define KVM_DEV_TYPE_FSL_MPIC_20	KVM_DEV_TYPE_FSL_MPIC_20
@@ -1018,6 +1031,8 @@ enum kvm_device_type {
 #define KVM_DEV_TYPE_FLIC		KVM_DEV_TYPE_FLIC
 	KVM_DEV_TYPE_ARM_VGIC_V3,
 #define KVM_DEV_TYPE_ARM_VGIC_V3	KVM_DEV_TYPE_ARM_VGIC_V3
+	KVM_DEV_TYPE_ARM_SMMU_V2,
+#define KVM_DEV_TYPE_ARM_SMMU_V2	KVM_DEV_TYPE_ARM_SMMU_V2
 	KVM_DEV_TYPE_MAX,
 };
 
-- 
2.4.3

[Qemu-devel] [RFC 2/6] hw/core/platform-bus: initialization notifier

[Baptiste Reynal]

Add a platform_bus_link_done_notifier for devices to be notified once
the platform bus is initialized. Address and IRQs for platform devices
are unknown at device realization, we need to notify the device once
the platform bus gathered all devices, i.e. when those resources are
ready. This is required for some device initialization
(e.g. the vSMMU base address needs to be set on the KVM device).

Signed-off-by: Baptiste Reynal <b.reynal@virtualopensystems.com>
---
 hw/core/platform-bus.c    | 13 +++++++++++++
 include/hw/platform-bus.h |  1 +
 2 files changed, 14 insertions(+)

diff --git a/hw/core/platform-bus.c b/hw/core/platform-bus.c
index 0f052b3..8642dfc 100644
--- a/hw/core/platform-bus.c
+++ b/hw/core/platform-bus.c
@@ -24,6 +24,18 @@
 #include "exec/address-spaces.h"
 #include "sysemu/sysemu.h"
 
+static NotifierList platform_bus_link_done_notifiers =
+    NOTIFIER_LIST_INITIALIZER(platform_bus_link_done_notifiers);
+
+void qemu_add_platform_bus_link_done_notifier(Notifier *notify)
+{
+    notifier_list_add(&platform_bus_link_done_notifiers, notify);
+}
+
+static void qemu_run_platform_bus_link_done_notifiers(void)
+{
+    notifier_list_notify(&platform_bus_link_done_notifiers, NULL);
+}
 
 /*
  * Returns the PlatformBus IRQ number for a SysBusDevice irq number or -1 if
@@ -196,6 +208,7 @@ static void platform_bus_init_notify(Notifier *notifier, void *data)
     plaform_bus_refresh_irqs(pb);
 
     foreach_dynamic_sysbus_device(link_sysbus_device, pb);
+    qemu_run_platform_bus_link_done_notifiers();
 }
 
 static void platform_bus_realize(DeviceState *dev, Error **errp)
diff --git a/include/hw/platform-bus.h b/include/hw/platform-bus.h
index bd42b83..3c3f96b 100644
--- a/include/hw/platform-bus.h
+++ b/include/hw/platform-bus.h
@@ -53,5 +53,6 @@ int platform_bus_get_irqn(PlatformBusDevice *platform_bus, SysBusDevice *sbdev,
                           int n);
 hwaddr platform_bus_get_mmio_addr(PlatformBusDevice *pbus, SysBusDevice *sbdev,
                                   int n);
+void qemu_add_platform_bus_link_done_notifier(Notifier *notify);
 
 #endif /* !HW_PLATFORM_BUS_H */
-- 
2.4.3

[Qemu-devel] [RFC 3/6] hw/core/platform-bus: add base_address field

[Baptiste Reynal]

Add base_address field to platform_bus structure and initialize it.

As the platform devices addresses are relative to the platform bus, we
need it to get the absolute address of a device.

Signed-off-by: Baptiste Reynal <b.reynal@virtualopensystems.com>
---
 hw/arm/virt.c             | 4 ++++
 include/hw/platform-bus.h | 2 ++
 2 files changed, 6 insertions(+)

diff --git a/hw/arm/virt.c b/hw/arm/virt.c
index 1b1cc71..33361c5 100644
--- a/hw/arm/virt.c
+++ b/hw/arm/virt.c
@@ -740,6 +740,7 @@ static void create_platform_bus(VirtBoardInfo *vbi, qemu_irq *pic)
 {
     DeviceState *dev;
     SysBusDevice *s;
+    PlatformBusDevice *pbus;
     int i;
     ARMPlatformBusFDTParams *fdt_params = g_new(ARMPlatformBusFDTParams, 1);
     MemoryRegion *sysmem = get_system_memory();
@@ -766,6 +767,9 @@ static void create_platform_bus(VirtBoardInfo *vbi, qemu_irq *pic)
         platform_bus_params.platform_bus_size);
     qdev_init_nofail(dev);
     s = SYS_BUS_DEVICE(dev);
+    pbus = PLATFORM_BUS_DEVICE(dev);
+
+    pbus->base_address = vbi->memmap[VIRT_PLATFORM_BUS].base;
 
     for (i = 0; i < platform_bus_params.platform_bus_num_irqs; i++) {
         int irqn = platform_bus_params.platform_bus_first_irq + i;
diff --git a/include/hw/platform-bus.h b/include/hw/platform-bus.h
index 3c3f96b..13efc82 100644
--- a/include/hw/platform-bus.h
+++ b/include/hw/platform-bus.h
@@ -41,6 +41,8 @@ struct PlatformBusDevice {
     bool done_gathering;
 
     /*< public >*/
+    hwaddr base_address;
+
     uint32_t mmio_size;
     MemoryRegion mmio;
 
-- 
2.4.3

[Qemu-devel] [RFC 4/6] hw/vfio: vsmmu device

[Baptiste Reynal]

This patches introduces support for ARM virtual SMMU, enabling two stages
address translation.

The vSMMU device can be instantiated from the command line using following
option:

-device vsmmu,x-group=1

Signed-off-by: Baptiste Reynal <b.reynal@virtualopensystems.com>
---
 hw/vfio/Makefile.objs       |   1 +
 hw/vfio/common.c            |   8 ++-
 hw/vfio/platform.c          |   1 +
 hw/vfio/smmu.c              | 157 ++++++++++++++++++++++++++++++++++++++++++++
 include/hw/vfio/vfio-smmu.h |  50 ++++++++++++++
 5 files changed, 215 insertions(+), 2 deletions(-)
 create mode 100644 hw/vfio/smmu.c
 create mode 100644 include/hw/vfio/vfio-smmu.h

diff --git a/hw/vfio/Makefile.objs b/hw/vfio/Makefile.objs
index d540c9d..4b91901 100644
--- a/hw/vfio/Makefile.objs
+++ b/hw/vfio/Makefile.objs
@@ -3,4 +3,5 @@ obj-$(CONFIG_SOFTMMU) += common.o
 obj-$(CONFIG_PCI) += pci.o
 obj-$(CONFIG_SOFTMMU) += platform.o
 obj-$(CONFIG_SOFTMMU) += calxeda-xgmac.o
+obj-$(CONFIG_SOFTMMU) += smmu.o
 endif
diff --git a/hw/vfio/common.c b/hw/vfio/common.c
index b1045da..a3ee72f 100644
--- a/hw/vfio/common.c
+++ b/hw/vfio/common.c
@@ -676,8 +676,12 @@ static int vfio_connect_container(VFIOGroup *group, AddressSpace *as)
             goto free_container_exit;
         }
 
-        ret = ioctl(fd, VFIO_SET_IOMMU,
-                    v2 ? VFIO_TYPE1v2_IOMMU : VFIO_TYPE1_IOMMU);
+        if (ioctl(fd, VFIO_CHECK_EXTENSION, VFIO_TYPE1_NESTING_IOMMU)) {
+            ret = ioctl(fd, VFIO_SET_IOMMU, VFIO_TYPE1_NESTING_IOMMU);
+        } else {
+            ret = ioctl(fd, VFIO_SET_IOMMU,
+			    v2 ? VFIO_TYPE1v2_IOMMU : VFIO_TYPE1_IOMMU);
+        }
         if (ret) {
             error_report("vfio: failed to set iommu for container: %m");
             ret = -errno;
diff --git a/hw/vfio/platform.c b/hw/vfio/platform.c
index 9382bb7..6192458 100644
--- a/hw/vfio/platform.c
+++ b/hw/vfio/platform.c
@@ -475,6 +475,7 @@ static int vfio_base_device_init(VFIODevice *vbasedev)
         error_report("vfio: failed to get group %d", groupid);
         return -ENOENT;
     }
+    vbasedev->group = group;
 
     g_snprintf(path, sizeof(path), "%s", vbasedev->name);
 
diff --git a/hw/vfio/smmu.c b/hw/vfio/smmu.c
new file mode 100644
index 0000000..467abbe
--- /dev/null
+++ b/hw/vfio/smmu.c
@@ -0,0 +1,157 @@
+/*
+ * support for vsmmu interface to use it with vfio devices
+ *
+ * Copyright (C) 2015 - Virtual Open Systems
+ *
+ * Author: Baptiste Reynal <b.reynal@virtualopensystems.com>
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2.  See
+ * the COPYING file in the top-level directory.
+ */
+
+#include <sys/ioctl.h>
+
+#include "hw/vfio/vfio-smmu.h"
+#include "sysemu/kvm.h"
+#include "sysemu/sysemu.h"
+#include "qemu/error-report.h"
+#include "hw/platform-bus.h"
+#include "hw/vfio/vfio.h"
+#include "hw/vfio/vfio-common.h"
+
+static void vsmmu_notify(Notifier *notifier, void *data)
+{
+    SmmuNotifierParams *p = DO_UPCAST(SmmuNotifierParams,
+                                                notifier, notifier);
+
+    DeviceState *dev;
+    PlatformBusDevice *pbus;
+    SysBusDevice *sbdev = SYS_BUS_DEVICE(p->vsmmu);
+
+    dev = qdev_find_recursive(sysbus_get_default(), TYPE_PLATFORM_BUS_DEVICE);
+    pbus = PLATFORM_BUS_DEVICE(dev);
+    assert(pbus->done_gathering);
+
+    hwaddr base = platform_bus_get_mmio_addr(pbus, sbdev, 0);
+    base += pbus->base_address;
+
+    struct kvm_device_attr attr = {
+        .group = KVM_DEV_ARM_SMMU_V2_CFG,
+        .attr = KVM_DEV_ARM_SMMU_V2_CFG_INIT,
+        .addr = (uint64_t)(unsigned long) &base,
+    };
+
+    if (ioctl(p->vsmmu->kvm_device, KVM_SET_DEVICE_ATTR, &attr)) {
+        error_report("Error during vSMMU initialization: %m.\n");
+    } else {
+        p->vsmmu->base = base;
+    }
+}
+
+static void smmu_realize(DeviceState *dev, Error **errp)
+{
+    VFIOSmmuDevice *vsmmu = VFIO_SMMU_DEVICE(dev);
+    SysBusDevice *sbdev = SYS_BUS_DEVICE(dev);
+    VFIOGroup *group;
+    int ret;
+    const char *name = "smmu memory";
+
+    group = vfio_get_group(vsmmu->group, &address_space_memory);
+    if (!group) {
+        error_report("vfio: failed to get group %d", vsmmu->group);
+        goto fail;
+    }
+
+    /* Create ARM_SMMU_V2 */
+    struct kvm_create_device cd = {
+        .type = KVM_DEV_TYPE_ARM_SMMU_V2,
+    };
+
+    ret = kvm_vm_ioctl(kvm_state, KVM_CREATE_DEVICE, &cd);
+    if (ret < 0) {
+        error_report("vfio: error creating vSMMU: %m");
+        goto fail;
+    };
+    vsmmu->kvm_device = cd.fd;
+
+    /* Add group */
+    struct arm_smmu_v2_vfio_group_sid group_sid = {
+        .fd = group->fd,
+        .sid = group->groupid,
+    };
+
+    struct kvm_device_attr attr = {
+        .group = KVM_DEV_ARM_SMMU_V2_VFIO,
+        .attr = KVM_DEV_ARM_SMMU_V2_VFIO_GROUP_ADD,
+        .addr = (uint64_t)(unsigned long) &group_sid,
+    };
+
+    if (ioctl(cd.fd, KVM_SET_DEVICE_ATTR, &attr)) {
+        error_report("Failed to add group %d to vSMMU: %m",
+                 group->groupid);
+        goto fail;
+    }
+
+    /* Initialize the virtual device */
+    /* Get vSMMU size for allocation */
+    int size;
+
+    attr.group = KVM_DEV_ARM_SMMU_V2_CFG;
+    attr.attr = KVM_DEV_ARM_SMMU_V2_CFG_SIZE;
+    attr.addr = (uint64_t)(unsigned long) &size;
+
+    ret = ioctl(cd.fd, KVM_GET_DEVICE_ATTR, &attr);
+
+    if (ret) {
+        error_report("Failed to get vSMMU size: %m");
+        goto fail;
+    }
+
+    vsmmu->size = size;
+
+    memory_region_init_reservation(&vsmmu->mem, OBJECT(vsmmu), name, size);
+    sysbus_init_mmio(sbdev, &vsmmu->mem);
+
+    /* Register a machine init done notifier to initialize the
+     * vSMMU at the right address */
+    SmmuNotifierParams *p = g_new(SmmuNotifierParams, 1);
+    p->vsmmu = vsmmu;
+    p->notifier.notify = vsmmu_notify;
+    qemu_add_platform_bus_link_done_notifier(&p->notifier);
+fail:
+    return;
+}
+
+static const VMStateDescription vfio_platform_vmstate = {
+    .name = TYPE_VFIO_SMMU,
+    .unmigratable = 1,
+};
+
+static Property vfio_smmu_dev_properties[] = {
+    DEFINE_PROP_UINT32("x-group", VFIOSmmuDevice,
+                       group, -1),
+    DEFINE_PROP_END_OF_LIST(),
+};
+
+static void vfio_smmu_class_init(ObjectClass *klass, void *data)
+{
+    DeviceClass *dc = DEVICE_CLASS(klass);
+    dc->realize = smmu_realize;
+    dc->desc = "VFIO SMMU";
+    dc->props = vfio_smmu_dev_properties;
+}
+
+static const TypeInfo vfio_smmu_dev_info = {
+    .name = TYPE_VFIO_SMMU,
+    .parent = TYPE_SYS_BUS_DEVICE,
+    .instance_size = sizeof(VFIOSmmuDevice),
+    .class_init = vfio_smmu_class_init,
+    .class_size = sizeof(VFIOSmmuDeviceClass),
+};
+
+static void register_smmu_dev_type(void)
+{
+    type_register_static(&vfio_smmu_dev_info);
+}
+
+type_init(register_smmu_dev_type)
diff --git a/include/hw/vfio/vfio-smmu.h b/include/hw/vfio/vfio-smmu.h
new file mode 100644
index 0000000..c14ff8e
--- /dev/null
+++ b/include/hw/vfio/vfio-smmu.h
@@ -0,0 +1,50 @@
+/*
+ * support for vsmmu interface to use it with vfio devices
+ *
+ * Copyright (C) 2015 - Virtual Open Systems
+ *
+ * Author: Baptiste Reynal <b.reynal@virtualopensystems.com>
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2.  See
+ * the COPYING file in the top-level directory.
+ */
+
+#ifndef HW_VFIO_SMMU_H
+#define HW_VFIO_SMMU_H
+
+#include "hw/sysbus.h"
+
+#define TYPE_VFIO_SMMU "vfio-smmu"
+
+typedef struct VFIOSmmuDevice {
+    SysBusDevice sbdev;
+    int kvm_device;
+    int size;
+    hwaddr base;
+
+    MemoryRegion mem;
+
+    uint32_t group;
+} VFIOSmmuDevice;
+
+typedef struct SmmuNotifierParams {
+    Notifier notifier;
+    VFIOSmmuDevice *vsmmu;
+} SmmuNotifierParams;
+
+typedef struct VFIOSmmuDeviceClass {
+    /*< private >*/
+    SysBusDeviceClass parent_class;
+    /*< public >*/
+} VFIOSmmuDeviceClass;
+
+#define VFIO_SMMU_DEVICE(obj) \
+     OBJECT_CHECK(VFIOSmmuDevice, (obj), TYPE_VFIO_SMMU)
+#define VFIO_SMMU_DEVICE_CLASS(klass) \
+     OBJECT_CLASS_CHECK(VFIOSmmuDeviceClass, (klass), \
+                        TYPE_VFIO_SMMU)
+#define VFIO_SMMU_DEVICE_GET_CLASS(obj) \
+     OBJECT_GET_CLASS(VFIOSmmuDeviceClass, (obj), \
+                        TYPE_VFIO_SMMU)
+
+#endif
-- 
2.4.3

[Qemu-devel] [RFC 5/6] hw/arm/sysbus-fdt: enable vsmmu dynamic instantiation

[Baptiste Reynal]

This patch adds the device tree node creation for the vSMMU.

Signed-off-by: Baptiste Reynal <b.reynal@virtualopensystems.com>
---
 hw/arm/sysbus-fdt.c | 43 +++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 43 insertions(+)

diff --git a/hw/arm/sysbus-fdt.c b/hw/arm/sysbus-fdt.c
index 3038b94..67f4425 100644
--- a/hw/arm/sysbus-fdt.c
+++ b/hw/arm/sysbus-fdt.c
@@ -26,6 +26,10 @@
 #include "sysemu/device_tree.h"
 #include "hw/platform-bus.h"
 #include "sysemu/sysemu.h"
+#include "hw/vfio/vfio-platform.h"
+#include "hw/vfio/vfio-smmu.h"
+
+#include <libfdt.h>
 
 /*
  * internal struct that contains the information to create dynamic
@@ -53,8 +57,47 @@ typedef struct NodeCreationPair {
     int (*add_fdt_node_fn)(SysBusDevice *sbdev, void *opaque);
 } NodeCreationPair;
 
+VFIOSmmuDevice *vsmmu = NULL;
+
+static int add_arm_smmu_fdt_node(SysBusDevice *sbdev, void *opaque)
+{
+    PlatformBusFDTData *data = opaque;
+    vsmmu = VFIO_SMMU_DEVICE(sbdev);
+    void *fdt = data->fdt;
+    const char *nodename;
+    uint32_t reg_attr[4], irq_attr[3];
+    uint32_t gint = 0;
+
+    nodename = g_strdup_printf("/smmu@%" PRIx64, vsmmu->base);
+
+    qemu_fdt_add_subnode(fdt, nodename);
+    qemu_fdt_setprop(fdt, nodename, "compatible", "arm,smmu-v2", 12);
+
+    reg_attr[0] = 0;
+    reg_attr[1] = cpu_to_be32(vsmmu->base);
+    reg_attr[2] = 0;
+    reg_attr[3] = cpu_to_be32(vsmmu->size);
+
+    qemu_fdt_setprop(fdt, nodename, "reg",
+            reg_attr, 4*sizeof(uint32_t));
+
+    /* Add dummy interrupt */
+    irq_attr[0] = 0;
+    irq_attr[1] = 0;
+    irq_attr[2] = 0;
+
+    qemu_fdt_setprop(fdt, nodename, "interrupts",
+            irq_attr, 3*sizeof(uint32_t));
+
+    qemu_fdt_setprop(fdt, nodename, "#global-interrupts", &gint,
+            sizeof(uint32_t));
+
+    return 0;
+}
+
 /* list of supported dynamic sysbus devices */
 static const NodeCreationPair add_fdt_node_functions[] = {
+    {TYPE_VFIO_SMMU, add_arm_smmu_fdt_node},
     {"", NULL}, /* last element */
 };
 
-- 
2.4.3

[Qemu-devel] [RFC 6/6] hw/arm/sysbus-fdt: add smmu masters in device tree

[Baptiste Reynal]

This patch adds a function to bind a device as an SMMU master in the
device tree. Currently, only one device is supported.

A device can be set as SMMU master by using the "x-iommu" argument on
the command line:

-device pl330,x-iommu=true

Signed-off-by: Baptiste Reynal <b.reynal@virtualopensystems.com>
---
 hw/arm/sysbus-fdt.c             | 35 +++++++++++++++++++++++++++++++++++
 hw/vfio/platform.c              |  1 +
 include/hw/vfio/vfio-platform.h |  1 +
 3 files changed, 37 insertions(+)

diff --git a/hw/arm/sysbus-fdt.c b/hw/arm/sysbus-fdt.c
index 67f4425..356f644 100644
--- a/hw/arm/sysbus-fdt.c
+++ b/hw/arm/sysbus-fdt.c
@@ -101,6 +101,38 @@ static const NodeCreationPair add_fdt_node_functions[] = {
     {"", NULL}, /* last element */
 };
 
+static int add_smmu_master(SysBusDevice *sbdev, void *opaque)
+{
+    if (object_dynamic_cast(OBJECT(sbdev), TYPE_VFIO_PLATFORM) &&
+	    VFIO_PLATFORM_DEVICE(sbdev)->iommu_master) {
+        int master[2];
+        char *smmu_nodename, *master_nodename;
+        uint64_t mmio_base;
+        PlatformBusFDTData *data = opaque;
+        PlatformBusDevice *pbus = data->pbus;
+        void *fdt = data->fdt;
+        VFIOPlatformDevice *vdev = VFIO_PLATFORM_DEVICE(sbdev);
+        VFIODevice *vbasedev = &vdev->vbasedev;
+        const char *parent_node = data->pbus_node_name;
+
+        smmu_nodename = g_strdup_printf("/smmu@%" PRIx64, vsmmu->base);
+        mmio_base = platform_bus_get_mmio_addr(pbus, sbdev, 0);
+
+        master_nodename = g_strdup_printf("%s/%s@%" PRIx64, parent_node,
+                vbasedev->name,
+                mmio_base);
+
+        master[0] = cpu_to_be32(qemu_fdt_get_phandle(fdt, master_nodename));
+        master[1] = cpu_to_be32(vbasedev->group->groupid);
+
+        qemu_fdt_setprop(fdt, smmu_nodename, "mmu-masters",
+                master, 2*sizeof(int));
+
+    }
+
+    return 0;
+}
+
 /**
  * add_fdt_node - add the device tree node of a dynamic sysbus device
  *
@@ -193,6 +225,9 @@ static void add_all_platform_bus_fdt_nodes(ARMPlatformBusFDTParams *fdt_params)
 
     /* Loop through all dynamic sysbus devices and create their node */
     foreach_dynamic_sysbus_device(add_fdt_node, &data);
+    if (vsmmu) {
+        foreach_dynamic_sysbus_device(add_smmu_master, &data);
+    }
 
     g_free(node);
 }
diff --git a/hw/vfio/platform.c b/hw/vfio/platform.c
index 6192458..d4ea297 100644
--- a/hw/vfio/platform.c
+++ b/hw/vfio/platform.c
@@ -585,6 +585,7 @@ static Property vfio_platform_dev_properties[] = {
     DEFINE_PROP_BOOL("x-mmap", VFIOPlatformDevice, vbasedev.allow_mmap, true),
     DEFINE_PROP_UINT32("mmap-timeout-ms", VFIOPlatformDevice,
                        mmap_timeout, 1100),
+    DEFINE_PROP_BOOL("x-iommu", VFIOPlatformDevice, iommu_master, false),
     DEFINE_PROP_END_OF_LIST(),
 };
 
diff --git a/include/hw/vfio/vfio-platform.h b/include/hw/vfio/vfio-platform.h
index 26b2ad6..8f5d980 100644
--- a/include/hw/vfio/vfio-platform.h
+++ b/include/hw/vfio/vfio-platform.h
@@ -57,6 +57,7 @@ typedef struct VFIOPlatformDevice {
     uint32_t mmap_timeout; /* delay to re-enable mmaps after interrupt */
     QEMUTimer *mmap_timer; /* allows fast-path resume after IRQ hit */
     QemuMutex intp_mutex; /* protect the intp_list IRQ state */
+    bool iommu_master; /* is the device a master to the vSMMU ? */
 } VFIOPlatformDevice;
 
 typedef struct VFIOPlatformDeviceClass {
-- 
2.4.3

Re: [Qemu-devel] [RFC 0/6] vSMMU initialization

[Will Deacon]

On Fri, Jun 12, 2015 at 03:20:04PM +0100, Baptiste Reynal wrote:
> The ARM SMMU has support for 2-stages address translations, allowing a virtual
> address to be translated at two levels:
> - Stage 1 translates a virtual address (VA) into an intermediate physical
> address (IPA)
> - Stage 2 translates an IPA into a physical address (PA)
> 
> Will Deacon introduced a virtual SMMU interface for KVM, which gives a virtual
> machine the possibility to use an IOMMU with native drivers. While the VM will
> program the first stage of translation (stage 1), the interface will program
> the second (stage 2) on the physical SMMU.

Please note that I have no plans to merge the kernel-side of this at the
moment. It was merely an exploratory tool to see what a non-PV vSMMU
implementation might look like and certainly not intended to be used in
anger.

Will

Re: [Qemu-devel] [RFC 0/6] vSMMU initialization

[Varun Sethi]

Hi Will,

> -----Original Message-----
> From: iommu-bounces@lists.linux-foundation.org [mailto:iommu-
> bounces@lists.linux-foundation.org] On Behalf Of Will Deacon
> Sent: Friday, June 12, 2015 7:53 PM
> To: Baptiste Reynal
> Cc: iommu@lists.linux-foundation.org; tech@virtualopensystems.com;
> qemu-devel@nongnu.org
> Subject: Re: [RFC 0/6] vSMMU initialization
> 
> On Fri, Jun 12, 2015 at 03:20:04PM +0100, Baptiste Reynal wrote:
> > The ARM SMMU has support for 2-stages address translations, allowing a
> > virtual address to be translated at two levels:
> > - Stage 1 translates a virtual address (VA) into an intermediate
> > physical address (IPA)
> > - Stage 2 translates an IPA into a physical address (PA)
> >
> > Will Deacon introduced a virtual SMMU interface for KVM, which gives a
> > virtual machine the possibility to use an IOMMU with native drivers.
> > While the VM will program the first stage of translation (stage 1),
> > the interface will program the second (stage 2) on the physical SMMU.
> 
> Please note that I have no plans to merge the kernel-side of this at the
> moment. It was merely an exploratory tool to see what a non-PV vSMMU
> implementation might look like and certainly not intended to be used in
> anger.
How do you see the context fault reporting work for the PV interface? Currently the vSMMU interface does seem quiet restrictive and it may simplify things by having PV iommu interface. But, do you see this even true in case of SMMUv3?
Just wondering if we can give more control with respect memory transaction attributes to the guest. Also, would it make sense to give guest control of the fault handling attributes i.e. fault/terminate model.

-Varun

Re: [Qemu-devel] [RFC 0/6] vSMMU initialization

[Will Deacon]

On Tue, Jul 14, 2015 at 03:21:03AM +0100, Varun Sethi wrote:
> Hi Will,

Hi Varun,

> > On Fri, Jun 12, 2015 at 03:20:04PM +0100, Baptiste Reynal wrote:
> > > The ARM SMMU has support for 2-stages address translations, allowing a
> > > virtual address to be translated at two levels:
> > > - Stage 1 translates a virtual address (VA) into an intermediate
> > > physical address (IPA)
> > > - Stage 2 translates an IPA into a physical address (PA)
> > >
> > > Will Deacon introduced a virtual SMMU interface for KVM, which gives a
> > > virtual machine the possibility to use an IOMMU with native drivers.
> > > While the VM will program the first stage of translation (stage 1),
> > > the interface will program the second (stage 2) on the physical SMMU.
> > 
> > Please note that I have no plans to merge the kernel-side of this at the
> > moment. It was merely an exploratory tool to see what a non-PV vSMMU
> > implementation might look like and certainly not intended to be used in
> > anger.
> How do you see the context fault reporting work for the PV interface?

We could have an interrupt, for the PV IOMMU and have the hypervisor
inject that, no?

> Currently the vSMMU interface does seem quiet restrictive and it may
> simplify things by having PV iommu interface. But, do you see this even
> true in case of SMMUv3?

I think SMMUv3 is *far* more amenable to the vSMMU approach, largely
because it moves many of the data structures into memory, but also because
it has support for things like ATS and PRI, so sharing page tables with
the CPU becomes a real possibility (and is something that doesn't work
well with a PV model).

> Just wondering if we can give more control with respect memory transaction
> attributes to the guest. Also, would it make sense to give guest control
> of the fault handling attributes i.e. fault/terminate model.

I'd like to see the basics prototyped before we start trying to design
for these more advanced use-cases. I'm confident there are plenty of things
we haven't even considered at the moment.

Will

Re: [Qemu-devel] [RFC 0/6] vSMMU initialization

[Baptiste Reynal]

On Tue, Jul 14, 2015 at 1:04 PM, Will Deacon <will.deacon@arm.com> wrote:
> On Tue, Jul 14, 2015 at 03:21:03AM +0100, Varun Sethi wrote:
>> Hi Will,
>
> Hi Varun,
>
>> > On Fri, Jun 12, 2015 at 03:20:04PM +0100, Baptiste Reynal wrote:
>> > > The ARM SMMU has support for 2-stages address translations, allowing a
>> > > virtual address to be translated at two levels:
>> > > - Stage 1 translates a virtual address (VA) into an intermediate
>> > > physical address (IPA)
>> > > - Stage 2 translates an IPA into a physical address (PA)
>> > >
>> > > Will Deacon introduced a virtual SMMU interface for KVM, which gives a
>> > > virtual machine the possibility to use an IOMMU with native drivers.
>> > > While the VM will program the first stage of translation (stage 1),
>> > > the interface will program the second (stage 2) on the physical SMMU.
>> >
>> > Please note that I have no plans to merge the kernel-side of this at the
>> > moment. It was merely an exploratory tool to see what a non-PV vSMMU
>> > implementation might look like and certainly not intended to be used in
>> > anger.
>> How do you see the context fault reporting work for the PV interface?
>
> We could have an interrupt, for the PV IOMMU and have the hypervisor
> inject that, no?
>
>> Currently the vSMMU interface does seem quiet restrictive and it may
>> simplify things by having PV iommu interface. But, do you see this even
>> true in case of SMMUv3?

Varun, may I know what do you mean by "more restrictive" ? Do you have
in mind any use case which should apply to the PV interface and not
the vSMMU ?

>
> I think SMMUv3 is *far* more amenable to the vSMMU approach, largely
> because it moves many of the data structures into memory, but also because
> it has support for things like ATS and PRI, so sharing page tables with
> the CPU becomes a real possibility (and is something that doesn't work
> well with a PV model).
>
>> Just wondering if we can give more control with respect memory transaction
>> attributes to the guest. Also, would it make sense to give guest control
>> of the fault handling attributes i.e. fault/terminate model.
>
> I'd like to see the basics prototyped before we start trying to design
> for these more advanced use-cases. I'm confident there are plenty of things
> we haven't even considered at the moment.

Hi Will,

>From my current understanding, vSMMU and PV interface seems
complementary and have different target. While the PV interface
targets broad compatibility with hardware and page table abstraction,
the vSMMU relies on specific hardware capabilities for better
performances (with dual-stage support and future ATS/PRI). As no PV
interface exists for now, we decided to focus our effort on the vSMMU.
Unless PV interface is strictly needed, we'd like to continue with the
implementation of the vSMMU.

In my opinion, both solutions are complementary and can co-exist once
someone shows interest for the PV.
>
> Will

Best regards,
Baptiste

Re: [Qemu-devel] [RFC 0/6] vSMMU initialization

[Will Deacon]

On Wed, Jul 15, 2015 at 02:38:15PM +0100, Baptiste Reynal wrote:
> On Tue, Jul 14, 2015 at 1:04 PM, Will Deacon <will.deacon@arm.com> wrote:
> > I think SMMUv3 is *far* more amenable to the vSMMU approach, largely
> > because it moves many of the data structures into memory, but also because
> > it has support for things like ATS and PRI, so sharing page tables with
> > the CPU becomes a real possibility (and is something that doesn't work
> > well with a PV model).
> >
> >> Just wondering if we can give more control with respect memory transaction
> >> attributes to the guest. Also, would it make sense to give guest control
> >> of the fault handling attributes i.e. fault/terminate model.
> >
> > I'd like to see the basics prototyped before we start trying to design
> > for these more advanced use-cases. I'm confident there are plenty of things
> > we haven't even considered at the moment.
> 
> From my current understanding, vSMMU and PV interface seems
> complementary and have different target. While the PV interface
> targets broad compatibility with hardware and page table abstraction,
> the vSMMU relies on specific hardware capabilities for better
> performances (with dual-stage support and future ATS/PRI). As no PV
> interface exists for now, we decided to focus our effort on the vSMMU.
> Unless PV interface is strictly needed, we'd like to continue with the
> implementation of the vSMMU.

On the contrary, I'm not going to merge vSMMU code unless there are strong
technical reasons against PV. I don't see your performance argument (I
would actually expect the vSMMU to be *slower* with SMMUv2) and I really
don't want fragmentation where user ABI is concerned.

Also, your argument about focussing on vSMMU because PV doesn't exist yet
doesn't make sense. Mainline doesn't support either of these for ARM.

> In my opinion, both solutions are complementary and can co-exist once
> someone shows interest for the PV.

I think you have this the wrong way around. We should start with PV (one
portable interface) and only add vSMMU interfaces where there are good
reasons to do so.

Will

Re: [Qemu-devel] [RFC 0/6] vSMMU initialization

[Varun Sethi]

Hi Baptiste,

> -----Original Message-----
> From: Baptiste Reynal [mailto:b.reynal@virtualopensystems.com]
> Sent: Wednesday, July 15, 2015 7:08 PM
> To: Will Deacon
> Cc: Sethi Varun-B16395; iommu@lists.linux-foundation.org;
> tech@virtualopensystems.com; qemu-devel@nongnu.org
> Subject: Re: [RFC 0/6] vSMMU initialization
> 
> On Tue, Jul 14, 2015 at 1:04 PM, Will Deacon <will.deacon@arm.com> wrote:
> > On Tue, Jul 14, 2015 at 03:21:03AM +0100, Varun Sethi wrote:
> >> Hi Will,
> >
> > Hi Varun,
> >
> >> > On Fri, Jun 12, 2015 at 03:20:04PM +0100, Baptiste Reynal wrote:
> >> > > The ARM SMMU has support for 2-stages address translations,
> >> > > allowing a virtual address to be translated at two levels:
> >> > > - Stage 1 translates a virtual address (VA) into an intermediate
> >> > > physical address (IPA)
> >> > > - Stage 2 translates an IPA into a physical address (PA)
> >> > >
> >> > > Will Deacon introduced a virtual SMMU interface for KVM, which
> >> > > gives a virtual machine the possibility to use an IOMMU with native
> drivers.
> >> > > While the VM will program the first stage of translation (stage
> >> > > 1), the interface will program the second (stage 2) on the physical
> SMMU.
> >> >
> >> > Please note that I have no plans to merge the kernel-side of this
> >> > at the moment. It was merely an exploratory tool to see what a
> >> > non-PV vSMMU implementation might look like and certainly not
> >> > intended to be used in anger.
> >> How do you see the context fault reporting work for the PV interface?
> >
> > We could have an interrupt, for the PV IOMMU and have the hypervisor
> > inject that, no?
> >
> >> Currently the vSMMU interface does seem quiet restrictive and it may
> >> simplify things by having PV iommu interface. But, do you see this
> >> even true in case of SMMUv3?
> 
> Varun, may I know what do you mean by "more restrictive" ? Do you have in
> mind any use case which should apply to the PV interface and not the
> vSMMU ?
> 
[varun] What I meant was that vSMMU allows for setting up of the stage 1 translation, but doesn't specifically allow access to other SMMU hardware functionality. We can certainly extend the vSMMU interface for providing additional functionality to the guest VM. I do agree with Will, that for extending the vSMMU interface prototyping is necessary. We need to come up with specific use cases for that.
For controlling stage 1 translation PV interface may be a bit simpler, but then we would need a generic interface to bind to most IOMMUs in the host.

-Varun

Re: [Qemu-devel] [RFC 0/6] vSMMU initialization

[Varun Sethi]

Hi Will,

> -----Original Message-----
> From: Will Deacon [mailto:will.deacon@arm.com]
> Sent: Tuesday, July 14, 2015 4:34 PM
> To: Sethi Varun-B16395
> Cc: Baptiste Reynal; iommu@lists.linux-foundation.org;
> tech@virtualopensystems.com; qemu-devel@nongnu.org
> Subject: Re: [RFC 0/6] vSMMU initialization
> 
> On Tue, Jul 14, 2015 at 03:21:03AM +0100, Varun Sethi wrote:
> > Hi Will,
> 
> Hi Varun,
> 
> > > On Fri, Jun 12, 2015 at 03:20:04PM +0100, Baptiste Reynal wrote:
> > > > The ARM SMMU has support for 2-stages address translations,
> > > > allowing a virtual address to be translated at two levels:
> > > > - Stage 1 translates a virtual address (VA) into an intermediate
> > > > physical address (IPA)
> > > > - Stage 2 translates an IPA into a physical address (PA)
> > > >
> > > > Will Deacon introduced a virtual SMMU interface for KVM, which
> > > > gives a virtual machine the possibility to use an IOMMU with native
> drivers.
> > > > While the VM will program the first stage of translation (stage
> > > > 1), the interface will program the second (stage 2) on the physical
> SMMU.
> > >
> > > Please note that I have no plans to merge the kernel-side of this at
> > > the moment. It was merely an exploratory tool to see what a non-PV
> > > vSMMU implementation might look like and certainly not intended to
> > > be used in anger.
> > How do you see the context fault reporting work for the PV interface?
> 
> We could have an interrupt, for the PV IOMMU and have the hypervisor
> inject that, no?
> 
Can you please elaborate on the PV IOMMU interface. I want to understand how context fault information would be communicated to the guest.

-Varun

Re: [Qemu-devel] [RFC 0/6] vSMMU initialization

[Will Deacon]

On Wed, Jul 15, 2015 at 06:28:54PM +0100, Varun Sethi wrote:
> > > > On Fri, Jun 12, 2015 at 03:20:04PM +0100, Baptiste Reynal wrote:
> > > > > The ARM SMMU has support for 2-stages address translations,
> > > > > allowing a virtual address to be translated at two levels:
> > > > > - Stage 1 translates a virtual address (VA) into an intermediate
> > > > > physical address (IPA)
> > > > > - Stage 2 translates an IPA into a physical address (PA)
> > > > >
> > > > > Will Deacon introduced a virtual SMMU interface for KVM, which
> > > > > gives a virtual machine the possibility to use an IOMMU with native
> > drivers.
> > > > > While the VM will program the first stage of translation (stage
> > > > > 1), the interface will program the second (stage 2) on the physical
> > SMMU.
> > > >
> > > > Please note that I have no plans to merge the kernel-side of this at
> > > > the moment. It was merely an exploratory tool to see what a non-PV
> > > > vSMMU implementation might look like and certainly not intended to
> > > > be used in anger.
> > > How do you see the context fault reporting work for the PV interface?
> > 
> > We could have an interrupt, for the PV IOMMU and have the hypervisor
> > inject that, no?
> > 
> Can you please elaborate on the PV IOMMU interface. I want to understand
> how context fault information would be communicated to the guest.

I replied to this the other day!

My assumption is that we'd have an irq and some memory region to describe
the fault in as general a way as possible. Whether that memory region
looks like MMIO registers or something like a virtio ring buffer is an
implementation detail to be resolved by prototyping.

Will