* [PATCH v3 01/13] x86/asm: add cpu_relax()
2016-11-14 22:27 [PATCH v3 00/13] VT-d unit test Peter Xu
@ 2016-11-14 22:27 ` Peter Xu
2016-11-14 22:28 ` [PATCH v3 02/13] libcflat: introduce is_power_of_2() Peter Xu
` (13 subsequent siblings)
14 siblings, 0 replies; 29+ messages in thread
From: Peter Xu @ 2016-11-14 22:27 UTC (permalink / raw)
To: kvm; +Cc: drjones, agordeev, jan.kiszka, rkrcmar, pbonzini, peterx
This will be useful to be put inside loops.
Suggested-by: Andrew Jones <drjones@redhat.com>
Reviewed-by: Andrew Jones <drjones@redhat.com>
Signed-off-by: Peter Xu <peterx@redhat.com>
---
lib/x86/asm/barrier.h | 11 +++++++++++
1 file changed, 11 insertions(+)
diff --git a/lib/x86/asm/barrier.h b/lib/x86/asm/barrier.h
index 7c108bd..193fb4c 100644
--- a/lib/x86/asm/barrier.h
+++ b/lib/x86/asm/barrier.h
@@ -13,4 +13,15 @@
#define smp_rmb() barrier()
#define smp_wmb() barrier()
+/* REP NOP (PAUSE) is a good thing to insert into busy-wait loops. */
+static inline void rep_nop(void)
+{
+ asm volatile("rep; nop" ::: "memory");
+}
+
+static inline void cpu_relax(void)
+{
+ rep_nop();
+}
+
#endif
--
2.7.4
^ permalink raw reply related [flat|nested] 29+ messages in thread* [PATCH v3 02/13] libcflat: introduce is_power_of_2()
2016-11-14 22:27 [PATCH v3 00/13] VT-d unit test Peter Xu
2016-11-14 22:27 ` [PATCH v3 01/13] x86/asm: add cpu_relax() Peter Xu
@ 2016-11-14 22:28 ` Peter Xu
2016-11-14 22:28 ` [PATCH v3 03/13] x86: intel-iommu: add vt-d init test Peter Xu
` (12 subsequent siblings)
14 siblings, 0 replies; 29+ messages in thread
From: Peter Xu @ 2016-11-14 22:28 UTC (permalink / raw)
To: kvm; +Cc: drjones, agordeev, jan.kiszka, rkrcmar, pbonzini, peterx
Suggested-by: Andrew Jones <drjones@redhat.com>
Reviewed-by: Andrew Jones <drjones@redhat.com>
Signed-off-by: Peter Xu <peterx@redhat.com>
---
lib/libcflat.h | 5 +++++
1 file changed, 5 insertions(+)
diff --git a/lib/libcflat.h b/lib/libcflat.h
index 72b1bf9..038ea1d 100644
--- a/lib/libcflat.h
+++ b/lib/libcflat.h
@@ -103,4 +103,9 @@ do { \
} \
} while (0)
+static inline bool is_power_of_2(unsigned long n)
+{
+ return n && !(n & (n - 1));
+}
+
#endif
--
2.7.4
^ permalink raw reply related [flat|nested] 29+ messages in thread* [PATCH v3 03/13] x86: intel-iommu: add vt-d init test
2016-11-14 22:27 [PATCH v3 00/13] VT-d unit test Peter Xu
2016-11-14 22:27 ` [PATCH v3 01/13] x86/asm: add cpu_relax() Peter Xu
2016-11-14 22:28 ` [PATCH v3 02/13] libcflat: introduce is_power_of_2() Peter Xu
@ 2016-11-14 22:28 ` Peter Xu
2016-11-15 8:41 ` Andrew Jones
2016-11-14 22:28 ` [PATCH v3 04/13] libcflat: add IS_ALIGNED() macro, and page sizes Peter Xu
` (11 subsequent siblings)
14 siblings, 1 reply; 29+ messages in thread
From: Peter Xu @ 2016-11-14 22:28 UTC (permalink / raw)
To: kvm; +Cc: drjones, agordeev, jan.kiszka, rkrcmar, pbonzini, peterx
Adding fundamental init test for Intel IOMMU. This includes basic
initialization of Intel IOMMU device, like DMAR (DMA Remapping),
IR (Interrupt Remapping), QI (Queue Invalidation), etc.
Further tests can use vtd_init() to initialize Intel IOMMU environment.
x86/unittests is updated to add this test.
Signed-off-by: Peter Xu <peterx@redhat.com>
---
lib/x86/intel-iommu.c | 88 +++++++++++++++++++++++++++++++++++++
lib/x86/intel-iommu.h | 118 ++++++++++++++++++++++++++++++++++++++++++++++++++
x86/Makefile.x86_64 | 2 +
x86/intel-iommu.c | 27 ++++++++++++
x86/unittests.cfg | 7 +++
5 files changed, 242 insertions(+)
create mode 100644 lib/x86/intel-iommu.c
create mode 100644 lib/x86/intel-iommu.h
create mode 100644 x86/intel-iommu.c
diff --git a/lib/x86/intel-iommu.c b/lib/x86/intel-iommu.c
new file mode 100644
index 0000000..9890f34
--- /dev/null
+++ b/lib/x86/intel-iommu.c
@@ -0,0 +1,88 @@
+/*
+ * Intel IOMMU APIs
+ *
+ * Copyright (C) 2016 Red Hat, Inc.
+ *
+ * Authors:
+ * Peter Xu <peterx@redhat.com>,
+ *
+ * This work is licensed under the terms of the GNU LGPL, version 2 or
+ * later.
+ */
+
+#include "intel-iommu.h"
+
+#define VTD_RTA_MASK (PAGE_MASK)
+#define VTD_IRTA_MASK (PAGE_MASK)
+
+static uint64_t vtd_root_table(void)
+{
+ /* No extend root table support yet */
+ return vtd_readq(DMAR_RTADDR_REG) & VTD_RTA_MASK;
+}
+
+static uint64_t vtd_ir_table(void)
+{
+ return vtd_readq(DMAR_IRTA_REG) & VTD_IRTA_MASK;
+}
+
+static void vtd_gcmd_or(uint32_t cmd)
+{
+ uint32_t status;
+
+ /* We only allow set one bit for each time */
+ assert(is_power_of_2(cmd));
+
+ status = vtd_readl(DMAR_GSTS_REG);
+ vtd_writel(DMAR_GCMD_REG, status | cmd);
+
+ if (cmd & VTD_GCMD_ONE_SHOT_BITS) {
+ /* One-shot bits are taking effect immediately */
+ return;
+ }
+
+ /* Make sure IOMMU handled our command request */
+ while (!(vtd_readl(DMAR_GSTS_REG) & cmd))
+ cpu_relax();
+}
+
+static void vtd_dump_init_info(void)
+{
+ printf("VT-d version: 0x%x\n", vtd_readl(DMAR_VER_REG));
+ printf(" cap: 0x%016lx\n", vtd_readq(DMAR_CAP_REG));
+ printf(" ecap: 0x%016lx\n", vtd_readq(DMAR_ECAP_REG));
+}
+
+static void vtd_setup_root_table(void)
+{
+ void *root = alloc_page();
+
+ memset(root, 0, PAGE_SIZE);
+ vtd_writeq(DMAR_RTADDR_REG, virt_to_phys(root));
+ vtd_gcmd_or(VTD_GCMD_ROOT);
+ printf("DMAR table address: 0x%016lx\n", vtd_root_table());
+}
+
+static void vtd_setup_ir_table(void)
+{
+ void *root = alloc_page();
+
+ memset(root, 0, PAGE_SIZE);
+ /* 0xf stands for table size (2^(0xf+1) == 65536) */
+ vtd_writeq(DMAR_IRTA_REG, virt_to_phys(root) | 0xf);
+ vtd_gcmd_or(VTD_GCMD_IR_TABLE);
+ printf("IR table address: 0x%016lx\n", vtd_ir_table());
+}
+
+void vtd_init(void)
+{
+ setup_vm();
+ smp_init();
+
+ vtd_dump_init_info();
+ vtd_gcmd_or(VTD_GCMD_QI); /* Enable QI */
+ vtd_setup_root_table();
+ vtd_setup_ir_table();
+ vtd_gcmd_or(VTD_GCMD_DMAR); /* Enable DMAR */
+ vtd_gcmd_or(VTD_GCMD_IR); /* Enable IR */
+}
diff --git a/lib/x86/intel-iommu.h b/lib/x86/intel-iommu.h
new file mode 100644
index 0000000..fae9ae5
--- /dev/null
+++ b/lib/x86/intel-iommu.h
@@ -0,0 +1,118 @@
+/*
+ * Intel IOMMU header
+ *
+ * Copyright (C) 2016 Red Hat, Inc.
+ *
+ * Authors:
+ * Peter Xu <peterx@redhat.com>,
+ *
+ * This work is licensed under the terms of the GNU LGPL, version 2 or
+ * later.
+ *
+ * (From include/linux/intel-iommu.h)
+ */
+
+#ifndef __INTEL_IOMMU_H__
+#define __INTEL_IOMMU_H__
+
+#include "libcflat.h"
+#include "vm.h"
+#include "isr.h"
+#include "smp.h"
+#include "desc.h"
+#include "asm/io.h"
+
+#define Q35_HOST_BRIDGE_IOMMU_ADDR 0xfed90000ULL
+
+/*
+ * Intel IOMMU register specification
+ */
+#define DMAR_VER_REG 0x0 /* Arch version supported by this IOMMU */
+#define DMAR_CAP_REG 0x8 /* Hardware supported capabilities */
+#define DMAR_CAP_REG_HI 0xc /* High 32-bit of DMAR_CAP_REG */
+#define DMAR_ECAP_REG 0x10 /* Extended capabilities supported */
+#define DMAR_ECAP_REG_HI 0X14
+#define DMAR_GCMD_REG 0x18 /* Global command */
+#define DMAR_GSTS_REG 0x1c /* Global status */
+#define DMAR_RTADDR_REG 0x20 /* Root entry table */
+#define DMAR_RTADDR_REG_HI 0X24
+#define DMAR_CCMD_REG 0x28 /* Context command */
+#define DMAR_CCMD_REG_HI 0x2c
+#define DMAR_FSTS_REG 0x34 /* Fault status */
+#define DMAR_FECTL_REG 0x38 /* Fault control */
+#define DMAR_FEDATA_REG 0x3c /* Fault event interrupt data */
+#define DMAR_FEADDR_REG 0x40 /* Fault event interrupt addr */
+#define DMAR_FEUADDR_REG 0x44 /* Upper address */
+#define DMAR_AFLOG_REG 0x58 /* Advanced fault control */
+#define DMAR_AFLOG_REG_HI 0X5c
+#define DMAR_PMEN_REG 0x64 /* Enable protected memory region */
+#define DMAR_PLMBASE_REG 0x68 /* PMRR low addr */
+#define DMAR_PLMLIMIT_REG 0x6c /* PMRR low limit */
+#define DMAR_PHMBASE_REG 0x70 /* PMRR high base addr */
+#define DMAR_PHMBASE_REG_HI 0X74
+#define DMAR_PHMLIMIT_REG 0x78 /* PMRR high limit */
+#define DMAR_PHMLIMIT_REG_HI 0x7c
+#define DMAR_IQH_REG 0x80 /* Invalidation queue head */
+#define DMAR_IQH_REG_HI 0X84
+#define DMAR_IQT_REG 0x88 /* Invalidation queue tail */
+#define DMAR_IQT_REG_HI 0X8c
+#define DMAR_IQA_REG 0x90 /* Invalidation queue addr */
+#define DMAR_IQA_REG_HI 0x94
+#define DMAR_ICS_REG 0x9c /* Invalidation complete status */
+#define DMAR_IRTA_REG 0xb8 /* Interrupt remapping table addr */
+#define DMAR_IRTA_REG_HI 0xbc
+#define DMAR_IECTL_REG 0xa0 /* Invalidation event control */
+#define DMAR_IEDATA_REG 0xa4 /* Invalidation event data */
+#define DMAR_IEADDR_REG 0xa8 /* Invalidation event address */
+#define DMAR_IEUADDR_REG 0xac /* Invalidation event address */
+#define DMAR_PQH_REG 0xc0 /* Page request queue head */
+#define DMAR_PQH_REG_HI 0xc4
+#define DMAR_PQT_REG 0xc8 /* Page request queue tail*/
+#define DMAR_PQT_REG_HI 0xcc
+#define DMAR_PQA_REG 0xd0 /* Page request queue address */
+#define DMAR_PQA_REG_HI 0xd4
+#define DMAR_PRS_REG 0xdc /* Page request status */
+#define DMAR_PECTL_REG 0xe0 /* Page request event control */
+#define DMAR_PEDATA_REG 0xe4 /* Page request event data */
+#define DMAR_PEADDR_REG 0xe8 /* Page request event address */
+#define DMAR_PEUADDR_REG 0xec /* Page event upper address */
+#define DMAR_MTRRCAP_REG 0x100 /* MTRR capability */
+#define DMAR_MTRRCAP_REG_HI 0x104
+#define DMAR_MTRRDEF_REG 0x108 /* MTRR default type */
+#define DMAR_MTRRDEF_REG_HI 0x10c
+
+#define VTD_GCMD_IR_TABLE 0x1000000
+#define VTD_GCMD_IR 0x2000000
+#define VTD_GCMD_QI 0x4000000
+#define VTD_GCMD_WBF 0x8000000 /* Write Buffer Flush */
+#define VTD_GCMD_SFL 0x20000000 /* Set Fault Log */
+#define VTD_GCMD_ROOT 0x40000000
+#define VTD_GCMD_DMAR 0x80000000
+#define VTD_GCMD_ONE_SHOT_BITS (VTD_GCMD_IR_TABLE | VTD_GCMD_WBF | \
+ VTD_GCMD_SFL | VTD_GCMD_ROOT)
+
+#define vtd_reg(reg) ((volatile void *)(Q35_HOST_BRIDGE_IOMMU_ADDR + reg))
+
+static inline void vtd_writel(unsigned int reg, uint32_t value)
+{
+ __raw_writel(value, vtd_reg(reg));
+}
+
+static inline void vtd_writeq(unsigned int reg, uint64_t value)
+{
+ __raw_writeq(value, vtd_reg(reg));
+}
+
+static inline uint32_t vtd_readl(unsigned int reg)
+{
+ return __raw_readl(vtd_reg(reg));
+}
+
+static inline uint64_t vtd_readq(unsigned int reg)
+{
+ return __raw_readq(vtd_reg(reg));
+}
+
+void vtd_init(void);
+
+#endif
diff --git a/x86/Makefile.x86_64 b/x86/Makefile.x86_64
index f82492b..3e2821e 100644
--- a/x86/Makefile.x86_64
+++ b/x86/Makefile.x86_64
@@ -4,6 +4,7 @@ ldarch = elf64-x86-64
CFLAGS += -mno-red-zone
cflatobjs += lib/x86/setjmp64.o
+cflatobjs += lib/x86/intel-iommu.o
tests = $(TEST_DIR)/access.flat $(TEST_DIR)/apic.flat \
$(TEST_DIR)/emulator.flat $(TEST_DIR)/idt_test.flat \
@@ -14,6 +15,7 @@ tests = $(TEST_DIR)/access.flat $(TEST_DIR)/apic.flat \
tests += $(TEST_DIR)/svm.flat
tests += $(TEST_DIR)/vmx.flat
tests += $(TEST_DIR)/tscdeadline_latency.flat
+tests += $(TEST_DIR)/intel-iommu.flat
include $(TEST_DIR)/Makefile.common
diff --git a/x86/intel-iommu.c b/x86/intel-iommu.c
new file mode 100644
index 0000000..f247913
--- /dev/null
+++ b/x86/intel-iommu.c
@@ -0,0 +1,27 @@
+/*
+ * Intel IOMMU unit test.
+ *
+ * Copyright (C) 2016 Red Hat, Inc.
+ *
+ * Authors:
+ * Peter Xu <peterx@redhat.com>,
+ *
+ * This work is licensed under the terms of the GNU LGPL, version 2 or
+ * later.
+ */
+
+#include "intel-iommu.h"
+
+int main(int argc, char *argv[])
+{
+ vtd_init();
+
+ report("fault status check", vtd_readl(DMAR_FSTS_REG) == 0);
+ report("QI enablement", vtd_readl(DMAR_GSTS_REG) & VTD_GCMD_QI);
+ report("DMAR table setup", vtd_readl(DMAR_GSTS_REG) & VTD_GCMD_ROOT);
+ report("IR table setup", vtd_readl(DMAR_GSTS_REG) & VTD_GCMD_IR_TABLE);
+ report("DMAR enablement", vtd_readl(DMAR_GSTS_REG) & VTD_GCMD_DMAR);
+ report("IR enablement", vtd_readl(DMAR_GSTS_REG) & VTD_GCMD_IR);
+
+ return report_summary();
+}
diff --git a/x86/unittests.cfg b/x86/unittests.cfg
index 23395c6..5413838 100644
--- a/x86/unittests.cfg
+++ b/x86/unittests.cfg
@@ -217,3 +217,10 @@ extra_params = -cpu kvm64,hv_time,hv_synic,hv_stimer -device hyperv-testdev
file = hyperv_clock.flat
smp = 2
extra_params = -cpu kvm64,hv_time
+
+[intel_iommu]
+file = intel-iommu.flat
+arch = x86_64
+timeout = 30
+smp = 4
+extra_params = -M q35,kernel-irqchip=split -device intel-iommu,intremap=on,eim=off -device edu
--
2.7.4
^ permalink raw reply related [flat|nested] 29+ messages in thread* Re: [PATCH v3 03/13] x86: intel-iommu: add vt-d init test
2016-11-14 22:28 ` [PATCH v3 03/13] x86: intel-iommu: add vt-d init test Peter Xu
@ 2016-11-15 8:41 ` Andrew Jones
0 siblings, 0 replies; 29+ messages in thread
From: Andrew Jones @ 2016-11-15 8:41 UTC (permalink / raw)
To: Peter Xu; +Cc: kvm, agordeev, jan.kiszka, rkrcmar, pbonzini
On Mon, Nov 14, 2016 at 05:28:01PM -0500, Peter Xu wrote:
> Adding fundamental init test for Intel IOMMU. This includes basic
> initialization of Intel IOMMU device, like DMAR (DMA Remapping),
> IR (Interrupt Remapping), QI (Queue Invalidation), etc.
>
> Further tests can use vtd_init() to initialize Intel IOMMU environment.
>
> x86/unittests is updated to add this test.
>
> Signed-off-by: Peter Xu <peterx@redhat.com>
> ---
> lib/x86/intel-iommu.c | 88 +++++++++++++++++++++++++++++++++++++
> lib/x86/intel-iommu.h | 118 ++++++++++++++++++++++++++++++++++++++++++++++++++
> x86/Makefile.x86_64 | 2 +
> x86/intel-iommu.c | 27 ++++++++++++
> x86/unittests.cfg | 7 +++
> 5 files changed, 242 insertions(+)
> create mode 100644 lib/x86/intel-iommu.c
> create mode 100644 lib/x86/intel-iommu.h
> create mode 100644 x86/intel-iommu.c
>
> diff --git a/lib/x86/intel-iommu.c b/lib/x86/intel-iommu.c
> new file mode 100644
> index 0000000..9890f34
> --- /dev/null
> +++ b/lib/x86/intel-iommu.c
> @@ -0,0 +1,88 @@
> +/*
> + * Intel IOMMU APIs
> + *
> + * Copyright (C) 2016 Red Hat, Inc.
> + *
> + * Authors:
> + * Peter Xu <peterx@redhat.com>,
> + *
> + * This work is licensed under the terms of the GNU LGPL, version 2 or
> + * later.
> + */
> +
> +#include "intel-iommu.h"
> +
> +#define VTD_RTA_MASK (PAGE_MASK)
> +#define VTD_IRTA_MASK (PAGE_MASK)
> +
> +static uint64_t vtd_root_table(void)
> +{
> + /* No extend root table support yet */
> + return vtd_readq(DMAR_RTADDR_REG) & VTD_RTA_MASK;
> +}
> +
> +static uint64_t vtd_ir_table(void)
> +{
> + return vtd_readq(DMAR_IRTA_REG) & VTD_IRTA_MASK;
> +}
> +
> +static void vtd_gcmd_or(uint32_t cmd)
> +{
> + uint32_t status;
> +
> + /* We only allow set one bit for each time */
> + assert(is_power_of_2(cmd));
> +
> + status = vtd_readl(DMAR_GSTS_REG);
> + vtd_writel(DMAR_GCMD_REG, status | cmd);
> +
> + if (cmd & VTD_GCMD_ONE_SHOT_BITS) {
> + /* One-shot bits are taking effect immediately */
> + return;
> + }
> +
> + /* Make sure IOMMU handled our command request */
> + while (!(vtd_readl(DMAR_GSTS_REG) & cmd))
> + cpu_relax();
> +}
> +
> +static void vtd_dump_init_info(void)
> +{
> + printf("VT-d version: 0x%x\n", vtd_readl(DMAR_VER_REG));
> + printf(" cap: 0x%016lx\n", vtd_readq(DMAR_CAP_REG));
> + printf(" ecap: 0x%016lx\n", vtd_readq(DMAR_ECAP_REG));
> +}
> +
> +static void vtd_setup_root_table(void)
> +{
> + void *root = alloc_page();
> +
> + memset(root, 0, PAGE_SIZE);
> + vtd_writeq(DMAR_RTADDR_REG, virt_to_phys(root));
> + vtd_gcmd_or(VTD_GCMD_ROOT);
> + printf("DMAR table address: 0x%016lx\n", vtd_root_table());
> +}
> +
> +static void vtd_setup_ir_table(void)
> +{
> + void *root = alloc_page();
> +
> + memset(root, 0, PAGE_SIZE);
> + /* 0xf stands for table size (2^(0xf+1) == 65536) */
> + vtd_writeq(DMAR_IRTA_REG, virt_to_phys(root) | 0xf);
> + vtd_gcmd_or(VTD_GCMD_IR_TABLE);
> + printf("IR table address: 0x%016lx\n", vtd_ir_table());
> +}
> +
> +void vtd_init(void)
> +{
> + setup_vm();
> + smp_init();
> +
> + vtd_dump_init_info();
> + vtd_gcmd_or(VTD_GCMD_QI); /* Enable QI */
> + vtd_setup_root_table();
> + vtd_setup_ir_table();
> + vtd_gcmd_or(VTD_GCMD_DMAR); /* Enable DMAR */
> + vtd_gcmd_or(VTD_GCMD_IR); /* Enable IR */
> +}
> diff --git a/lib/x86/intel-iommu.h b/lib/x86/intel-iommu.h
> new file mode 100644
> index 0000000..fae9ae5
> --- /dev/null
> +++ b/lib/x86/intel-iommu.h
> @@ -0,0 +1,118 @@
> +/*
> + * Intel IOMMU header
> + *
> + * Copyright (C) 2016 Red Hat, Inc.
> + *
> + * Authors:
> + * Peter Xu <peterx@redhat.com>,
> + *
> + * This work is licensed under the terms of the GNU LGPL, version 2 or
> + * later.
> + *
> + * (From include/linux/intel-iommu.h)
> + */
> +
> +#ifndef __INTEL_IOMMU_H__
> +#define __INTEL_IOMMU_H__
> +
> +#include "libcflat.h"
> +#include "vm.h"
> +#include "isr.h"
> +#include "smp.h"
> +#include "desc.h"
> +#include "asm/io.h"
> +
> +#define Q35_HOST_BRIDGE_IOMMU_ADDR 0xfed90000ULL
> +
> +/*
> + * Intel IOMMU register specification
> + */
> +#define DMAR_VER_REG 0x0 /* Arch version supported by this IOMMU */
> +#define DMAR_CAP_REG 0x8 /* Hardware supported capabilities */
> +#define DMAR_CAP_REG_HI 0xc /* High 32-bit of DMAR_CAP_REG */
> +#define DMAR_ECAP_REG 0x10 /* Extended capabilities supported */
> +#define DMAR_ECAP_REG_HI 0X14
> +#define DMAR_GCMD_REG 0x18 /* Global command */
> +#define DMAR_GSTS_REG 0x1c /* Global status */
> +#define DMAR_RTADDR_REG 0x20 /* Root entry table */
> +#define DMAR_RTADDR_REG_HI 0X24
> +#define DMAR_CCMD_REG 0x28 /* Context command */
> +#define DMAR_CCMD_REG_HI 0x2c
> +#define DMAR_FSTS_REG 0x34 /* Fault status */
> +#define DMAR_FECTL_REG 0x38 /* Fault control */
> +#define DMAR_FEDATA_REG 0x3c /* Fault event interrupt data */
> +#define DMAR_FEADDR_REG 0x40 /* Fault event interrupt addr */
> +#define DMAR_FEUADDR_REG 0x44 /* Upper address */
> +#define DMAR_AFLOG_REG 0x58 /* Advanced fault control */
> +#define DMAR_AFLOG_REG_HI 0X5c
> +#define DMAR_PMEN_REG 0x64 /* Enable protected memory region */
> +#define DMAR_PLMBASE_REG 0x68 /* PMRR low addr */
> +#define DMAR_PLMLIMIT_REG 0x6c /* PMRR low limit */
> +#define DMAR_PHMBASE_REG 0x70 /* PMRR high base addr */
> +#define DMAR_PHMBASE_REG_HI 0X74
> +#define DMAR_PHMLIMIT_REG 0x78 /* PMRR high limit */
> +#define DMAR_PHMLIMIT_REG_HI 0x7c
> +#define DMAR_IQH_REG 0x80 /* Invalidation queue head */
> +#define DMAR_IQH_REG_HI 0X84
> +#define DMAR_IQT_REG 0x88 /* Invalidation queue tail */
> +#define DMAR_IQT_REG_HI 0X8c
> +#define DMAR_IQA_REG 0x90 /* Invalidation queue addr */
> +#define DMAR_IQA_REG_HI 0x94
> +#define DMAR_ICS_REG 0x9c /* Invalidation complete status */
> +#define DMAR_IRTA_REG 0xb8 /* Interrupt remapping table addr */
> +#define DMAR_IRTA_REG_HI 0xbc
> +#define DMAR_IECTL_REG 0xa0 /* Invalidation event control */
> +#define DMAR_IEDATA_REG 0xa4 /* Invalidation event data */
> +#define DMAR_IEADDR_REG 0xa8 /* Invalidation event address */
> +#define DMAR_IEUADDR_REG 0xac /* Invalidation event address */
> +#define DMAR_PQH_REG 0xc0 /* Page request queue head */
> +#define DMAR_PQH_REG_HI 0xc4
> +#define DMAR_PQT_REG 0xc8 /* Page request queue tail*/
> +#define DMAR_PQT_REG_HI 0xcc
> +#define DMAR_PQA_REG 0xd0 /* Page request queue address */
> +#define DMAR_PQA_REG_HI 0xd4
> +#define DMAR_PRS_REG 0xdc /* Page request status */
> +#define DMAR_PECTL_REG 0xe0 /* Page request event control */
> +#define DMAR_PEDATA_REG 0xe4 /* Page request event data */
> +#define DMAR_PEADDR_REG 0xe8 /* Page request event address */
> +#define DMAR_PEUADDR_REG 0xec /* Page event upper address */
> +#define DMAR_MTRRCAP_REG 0x100 /* MTRR capability */
> +#define DMAR_MTRRCAP_REG_HI 0x104
> +#define DMAR_MTRRDEF_REG 0x108 /* MTRR default type */
> +#define DMAR_MTRRDEF_REG_HI 0x10c
> +
> +#define VTD_GCMD_IR_TABLE 0x1000000
> +#define VTD_GCMD_IR 0x2000000
> +#define VTD_GCMD_QI 0x4000000
> +#define VTD_GCMD_WBF 0x8000000 /* Write Buffer Flush */
> +#define VTD_GCMD_SFL 0x20000000 /* Set Fault Log */
> +#define VTD_GCMD_ROOT 0x40000000
> +#define VTD_GCMD_DMAR 0x80000000
> +#define VTD_GCMD_ONE_SHOT_BITS (VTD_GCMD_IR_TABLE | VTD_GCMD_WBF | \
> + VTD_GCMD_SFL | VTD_GCMD_ROOT)
> +
> +#define vtd_reg(reg) ((volatile void *)(Q35_HOST_BRIDGE_IOMMU_ADDR + reg))
> +
> +static inline void vtd_writel(unsigned int reg, uint32_t value)
> +{
> + __raw_writel(value, vtd_reg(reg));
> +}
> +
> +static inline void vtd_writeq(unsigned int reg, uint64_t value)
> +{
> + __raw_writeq(value, vtd_reg(reg));
> +}
> +
> +static inline uint32_t vtd_readl(unsigned int reg)
> +{
> + return __raw_readl(vtd_reg(reg));
> +}
> +
> +static inline uint64_t vtd_readq(unsigned int reg)
> +{
> + return __raw_readq(vtd_reg(reg));
> +}
> +
> +void vtd_init(void);
> +
> +#endif
> diff --git a/x86/Makefile.x86_64 b/x86/Makefile.x86_64
> index f82492b..3e2821e 100644
> --- a/x86/Makefile.x86_64
> +++ b/x86/Makefile.x86_64
> @@ -4,6 +4,7 @@ ldarch = elf64-x86-64
> CFLAGS += -mno-red-zone
>
> cflatobjs += lib/x86/setjmp64.o
> +cflatobjs += lib/x86/intel-iommu.o
>
> tests = $(TEST_DIR)/access.flat $(TEST_DIR)/apic.flat \
> $(TEST_DIR)/emulator.flat $(TEST_DIR)/idt_test.flat \
> @@ -14,6 +15,7 @@ tests = $(TEST_DIR)/access.flat $(TEST_DIR)/apic.flat \
> tests += $(TEST_DIR)/svm.flat
> tests += $(TEST_DIR)/vmx.flat
> tests += $(TEST_DIR)/tscdeadline_latency.flat
> +tests += $(TEST_DIR)/intel-iommu.flat
>
> include $(TEST_DIR)/Makefile.common
>
> diff --git a/x86/intel-iommu.c b/x86/intel-iommu.c
> new file mode 100644
> index 0000000..f247913
> --- /dev/null
> +++ b/x86/intel-iommu.c
> @@ -0,0 +1,27 @@
> +/*
> + * Intel IOMMU unit test.
> + *
> + * Copyright (C) 2016 Red Hat, Inc.
> + *
> + * Authors:
> + * Peter Xu <peterx@redhat.com>,
> + *
> + * This work is licensed under the terms of the GNU LGPL, version 2 or
> + * later.
> + */
> +
> +#include "intel-iommu.h"
> +
> +int main(int argc, char *argv[])
> +{
> + vtd_init();
> +
> + report("fault status check", vtd_readl(DMAR_FSTS_REG) == 0);
> + report("QI enablement", vtd_readl(DMAR_GSTS_REG) & VTD_GCMD_QI);
> + report("DMAR table setup", vtd_readl(DMAR_GSTS_REG) & VTD_GCMD_ROOT);
> + report("IR table setup", vtd_readl(DMAR_GSTS_REG) & VTD_GCMD_IR_TABLE);
> + report("DMAR enablement", vtd_readl(DMAR_GSTS_REG) & VTD_GCMD_DMAR);
> + report("IR enablement", vtd_readl(DMAR_GSTS_REG) & VTD_GCMD_IR);
> +
> + return report_summary();
> +}
> diff --git a/x86/unittests.cfg b/x86/unittests.cfg
> index 23395c6..5413838 100644
> --- a/x86/unittests.cfg
> +++ b/x86/unittests.cfg
> @@ -217,3 +217,10 @@ extra_params = -cpu kvm64,hv_time,hv_synic,hv_stimer -device hyperv-testdev
> file = hyperv_clock.flat
> smp = 2
> extra_params = -cpu kvm64,hv_time
> +
> +[intel_iommu]
> +file = intel-iommu.flat
> +arch = x86_64
> +timeout = 30
> +smp = 4
> +extra_params = -M q35,kernel-irqchip=split -device intel-iommu,intremap=on,eim=off -device edu
> --
> 2.7.4
>
Looks good to me, but I still haven't opened a spec. I defer to
Paolo and Radim for that.
Thanks,
drew
^ permalink raw reply [flat|nested] 29+ messages in thread
* [PATCH v3 04/13] libcflat: add IS_ALIGNED() macro, and page sizes
2016-11-14 22:27 [PATCH v3 00/13] VT-d unit test Peter Xu
` (2 preceding siblings ...)
2016-11-14 22:28 ` [PATCH v3 03/13] x86: intel-iommu: add vt-d init test Peter Xu
@ 2016-11-14 22:28 ` Peter Xu
2016-11-15 8:43 ` Andrew Jones
2016-11-14 22:28 ` [PATCH v3 05/13] libcflat: moving MIN/MAX here Peter Xu
` (10 subsequent siblings)
14 siblings, 1 reply; 29+ messages in thread
From: Peter Xu @ 2016-11-14 22:28 UTC (permalink / raw)
To: kvm; +Cc: drjones, agordeev, jan.kiszka, rkrcmar, pbonzini, peterx
These macros will be useful to do page alignment checks.
Reviewed-by: Andrew Jones <drjones@redhat.com>
Signed-off-by: Peter Xu <peterx@redhat.com>
---
lib/libcflat.h | 6 ++++++
1 file changed, 6 insertions(+)
diff --git a/lib/libcflat.h b/lib/libcflat.h
index 038ea1d..d0b0fd1 100644
--- a/lib/libcflat.h
+++ b/lib/libcflat.h
@@ -33,6 +33,12 @@
#define __ALIGN_MASK(x, mask) (((x) + (mask)) & ~(mask))
#define __ALIGN(x, a) __ALIGN_MASK(x, (typeof(x))(a) - 1)
#define ALIGN(x, a) __ALIGN((x), (a))
+#define IS_ALIGNED(x, a) (((x) & ((typeof(x))(a) - 1)) == 0)
+
+#define SZ_4K 0x1000
+#define SZ_64K 0x10000
+#define SZ_2M 0x200000
+#define SZ_1G 0x40000000
typedef uint8_t u8;
typedef int8_t s8;
--
2.7.4
^ permalink raw reply related [flat|nested] 29+ messages in thread* Re: [PATCH v3 04/13] libcflat: add IS_ALIGNED() macro, and page sizes
2016-11-14 22:28 ` [PATCH v3 04/13] libcflat: add IS_ALIGNED() macro, and page sizes Peter Xu
@ 2016-11-15 8:43 ` Andrew Jones
2016-11-15 16:26 ` Peter Xu
0 siblings, 1 reply; 29+ messages in thread
From: Andrew Jones @ 2016-11-15 8:43 UTC (permalink / raw)
To: Peter Xu; +Cc: kvm, agordeev, jan.kiszka, rkrcmar, pbonzini
On Mon, Nov 14, 2016 at 05:28:02PM -0500, Peter Xu wrote:
> These macros will be useful to do page alignment checks.
>
> Reviewed-by: Andrew Jones <drjones@redhat.com>
> Signed-off-by: Peter Xu <peterx@redhat.com>
> ---
> lib/libcflat.h | 6 ++++++
> 1 file changed, 6 insertions(+)
>
> diff --git a/lib/libcflat.h b/lib/libcflat.h
> index 038ea1d..d0b0fd1 100644
> --- a/lib/libcflat.h
> +++ b/lib/libcflat.h
> @@ -33,6 +33,12 @@
> #define __ALIGN_MASK(x, mask) (((x) + (mask)) & ~(mask))
> #define __ALIGN(x, a) __ALIGN_MASK(x, (typeof(x))(a) - 1)
> #define ALIGN(x, a) __ALIGN((x), (a))
> +#define IS_ALIGNED(x, a) (((x) & ((typeof(x))(a) - 1)) == 0)
> +
> +#define SZ_4K 0x1000
> +#define SZ_64K 0x10000
> +#define SZ_2M 0x200000
> +#define SZ_1G 0x40000000
I've changed this to bit shifts in v6 of the gic series. I hope that
series is pretty much on it's way in now, but if not, then please
pick up the new version.
Thanks,
drew
>
> typedef uint8_t u8;
> typedef int8_t s8;
> --
> 2.7.4
>
> --
> To unsubscribe from this list: send the line "unsubscribe kvm" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: [PATCH v3 04/13] libcflat: add IS_ALIGNED() macro, and page sizes
2016-11-15 8:43 ` Andrew Jones
@ 2016-11-15 16:26 ` Peter Xu
0 siblings, 0 replies; 29+ messages in thread
From: Peter Xu @ 2016-11-15 16:26 UTC (permalink / raw)
To: Andrew Jones; +Cc: kvm, agordeev, jan.kiszka, rkrcmar, pbonzini
On Tue, Nov 15, 2016 at 09:43:40AM +0100, Andrew Jones wrote:
[...]
> > +#define SZ_4K 0x1000
> > +#define SZ_64K 0x10000
> > +#define SZ_2M 0x200000
> > +#define SZ_1G 0x40000000
>
> I've changed this to bit shifts in v6 of the gic series. I hope that
> series is pretty much on it's way in now, but if not, then please
> pick up the new version.
Will do. Thanks,
-- peterx
^ permalink raw reply [flat|nested] 29+ messages in thread
* [PATCH v3 05/13] libcflat: moving MIN/MAX here
2016-11-14 22:27 [PATCH v3 00/13] VT-d unit test Peter Xu
` (3 preceding siblings ...)
2016-11-14 22:28 ` [PATCH v3 04/13] libcflat: add IS_ALIGNED() macro, and page sizes Peter Xu
@ 2016-11-14 22:28 ` Peter Xu
2016-11-14 22:28 ` [PATCH v3 06/13] vm/page: provide PGDIR_OFFSET() macro Peter Xu
` (9 subsequent siblings)
14 siblings, 0 replies; 29+ messages in thread
From: Peter Xu @ 2016-11-14 22:28 UTC (permalink / raw)
To: kvm; +Cc: drjones, agordeev, jan.kiszka, rkrcmar, pbonzini, peterx
That's something can be used outside alloc.c.
Reviewed-by: Andrew Jones <drjones@redhat.com>
Signed-off-by: Peter Xu <peterx@redhat.com>
---
lib/alloc.c | 3 ---
lib/libcflat.h | 3 +++
2 files changed, 3 insertions(+), 3 deletions(-)
diff --git a/lib/alloc.c b/lib/alloc.c
index e1d7b8a..58af52b 100644
--- a/lib/alloc.c
+++ b/lib/alloc.c
@@ -7,9 +7,6 @@
#include "asm/spinlock.h"
#include "asm/io.h"
-#define MIN(a, b) ((a) < (b) ? (a) : (b))
-#define MAX(a, b) ((a) > (b) ? (a) : (b))
-
#define PHYS_ALLOC_NR_REGIONS 256
struct phys_alloc_region {
diff --git a/lib/libcflat.h b/lib/libcflat.h
index d0b0fd1..79695f6 100644
--- a/lib/libcflat.h
+++ b/lib/libcflat.h
@@ -40,6 +40,9 @@
#define SZ_2M 0x200000
#define SZ_1G 0x40000000
+#define MIN(a, b) ((a) < (b) ? (a) : (b))
+#define MAX(a, b) ((a) > (b) ? (a) : (b))
+
typedef uint8_t u8;
typedef int8_t s8;
typedef uint16_t u16;
--
2.7.4
^ permalink raw reply related [flat|nested] 29+ messages in thread* [PATCH v3 06/13] vm/page: provide PGDIR_OFFSET() macro
2016-11-14 22:27 [PATCH v3 00/13] VT-d unit test Peter Xu
` (4 preceding siblings ...)
2016-11-14 22:28 ` [PATCH v3 05/13] libcflat: moving MIN/MAX here Peter Xu
@ 2016-11-14 22:28 ` Peter Xu
2016-11-14 22:28 ` [PATCH v3 07/13] pci: introduce struct pci_dev Peter Xu
` (8 subsequent siblings)
14 siblings, 0 replies; 29+ messages in thread
From: Peter Xu @ 2016-11-14 22:28 UTC (permalink / raw)
To: kvm; +Cc: drjones, agordeev, jan.kiszka, rkrcmar, pbonzini, peterx
This can be used in further patches.
Reviewed-by: Andrew Jones <drjones@redhat.com>
Signed-off-by: Peter Xu <peterx@redhat.com>
---
lib/x86/asm/page.h | 3 +++
lib/x86/vm.c | 4 ++--
2 files changed, 5 insertions(+), 2 deletions(-)
diff --git a/lib/x86/asm/page.h b/lib/x86/asm/page.h
index 5044a49..c43bab2 100644
--- a/lib/x86/asm/page.h
+++ b/lib/x86/asm/page.h
@@ -41,5 +41,8 @@
#define PGDIR_MASK 1023
#endif
+#define PGDIR_BITS(lvl) (((lvl) - 1) * PGDIR_WIDTH + PAGE_SHIFT)
+#define PGDIR_OFFSET(va, lvl) (((va) >> PGDIR_BITS(lvl)) & PGDIR_MASK)
+
#endif /* !__ASSEMBLY__ */
#endif
diff --git a/lib/x86/vm.c b/lib/x86/vm.c
index f7e778b..cda4c5f 100644
--- a/lib/x86/vm.c
+++ b/lib/x86/vm.c
@@ -48,7 +48,7 @@ unsigned long *install_pte(unsigned long *cr3,
unsigned offset;
for (level = PAGE_LEVEL; level > pte_level; --level) {
- offset = ((unsigned long)virt >> ((level-1) * PGDIR_WIDTH + 12)) & PGDIR_MASK;
+ offset = PGDIR_OFFSET((unsigned long)virt, level);
if (!(pt[offset] & PT_PRESENT_MASK)) {
unsigned long *new_pt = pt_page;
if (!new_pt)
@@ -60,7 +60,7 @@ unsigned long *install_pte(unsigned long *cr3,
}
pt = phys_to_virt(pt[offset] & PT_ADDR_MASK);
}
- offset = ((unsigned long)virt >> ((level-1) * PGDIR_WIDTH + 12)) & PGDIR_MASK;
+ offset = PGDIR_OFFSET((unsigned long)virt, level);
pt[offset] = pte;
return &pt[offset];
}
--
2.7.4
^ permalink raw reply related [flat|nested] 29+ messages in thread* [PATCH v3 07/13] pci: introduce struct pci_dev
2016-11-14 22:27 [PATCH v3 00/13] VT-d unit test Peter Xu
` (5 preceding siblings ...)
2016-11-14 22:28 ` [PATCH v3 06/13] vm/page: provide PGDIR_OFFSET() macro Peter Xu
@ 2016-11-14 22:28 ` Peter Xu
2016-11-14 22:28 ` [PATCH v3 08/13] pci: provide pci_scan_bars() Peter Xu
` (7 subsequent siblings)
14 siblings, 0 replies; 29+ messages in thread
From: Peter Xu @ 2016-11-14 22:28 UTC (permalink / raw)
To: kvm; +Cc: drjones, agordeev, jan.kiszka, rkrcmar, pbonzini, peterx
To extend current PCI framework, we need a per-device struct to store
device specific information. Time to have a pci_dev struct. Most of the
current PCI APIs are converted to use this pci_dev object as the first
argument. Currently it only contains one field "bdf", which is the bdf
of current device.
For a few APIs like pci_config_*() ops or pci_find_dev(), I kept the old
interface (use PCI BDF value rather than "struct pci_dev") since they
can be used in a open context that without any specific PCI device.
Reviewed-by: Andrew Jones <drjones@redhat.com>
Signed-off-by: Peter Xu <peterx@redhat.com>
---
lib/pci-host-generic.c | 9 +++++---
lib/pci-testdev.c | 10 +++++----
lib/pci.c | 56 ++++++++++++++++++++++++++++++--------------------
lib/pci.h | 24 ++++++++++++++--------
x86/vmexit.c | 18 ++++++++--------
5 files changed, 72 insertions(+), 45 deletions(-)
diff --git a/lib/pci-host-generic.c b/lib/pci-host-generic.c
index 6ac0f15..ea83d72 100644
--- a/lib/pci-host-generic.c
+++ b/lib/pci-host-generic.c
@@ -211,6 +211,7 @@ static bool pci_alloc_resource(pcidevaddr_t dev, int bar_num, u64 *addr)
bool pci_probe(void)
{
+ struct pci_dev pci_dev;
pcidevaddr_t dev;
u8 header;
u32 cmd;
@@ -225,6 +226,8 @@ bool pci_probe(void)
if (!pci_dev_exists(dev))
continue;
+ pci_dev_init(&pci_dev, dev);
+
/* We are only interested in normal PCI devices */
header = pci_config_readb(dev, PCI_HEADER_TYPE);
if ((header & PCI_HEADER_TYPE_MASK) != PCI_HEADER_TYPE_NORMAL)
@@ -236,15 +239,15 @@ bool pci_probe(void)
u64 addr;
if (pci_alloc_resource(dev, i, &addr)) {
- pci_bar_set_addr(dev, i, addr);
+ pci_bar_set_addr(&pci_dev, i, addr);
- if (pci_bar_is_memory(dev, i))
+ if (pci_bar_is_memory(&pci_dev, i))
cmd |= PCI_COMMAND_MEMORY;
else
cmd |= PCI_COMMAND_IO;
}
- if (pci_bar_is64(dev, i))
+ if (pci_bar_is64(&pci_dev, i))
i++;
}
diff --git a/lib/pci-testdev.c b/lib/pci-testdev.c
index ad482d3..7d298e6 100644
--- a/lib/pci-testdev.c
+++ b/lib/pci-testdev.c
@@ -163,9 +163,10 @@ static int pci_testdev_all(struct pci_test_dev_hdr *test,
int pci_testdev(void)
{
+ struct pci_dev pci_dev;
+ pcidevaddr_t dev;
phys_addr_t addr;
void __iomem *mem, *io;
- pcidevaddr_t dev;
int nr_tests = 0;
bool ret;
@@ -175,14 +176,15 @@ int pci_testdev(void)
"check QEMU '-device pci-testdev' parameter\n");
return -1;
}
+ pci_dev_init(&pci_dev, dev);
- ret = pci_bar_is_valid(dev, 0) && pci_bar_is_valid(dev, 1);
+ ret = pci_bar_is_valid(&pci_dev, 0) && pci_bar_is_valid(&pci_dev, 1);
assert(ret);
- addr = pci_bar_get_addr(dev, 0);
+ addr = pci_bar_get_addr(&pci_dev, 0);
mem = ioremap(addr, PAGE_SIZE);
- addr = pci_bar_get_addr(dev, 1);
+ addr = pci_bar_get_addr(&pci_dev, 1);
io = (void *)(unsigned long)addr;
nr_tests += pci_testdev_all(mem, &pci_testdev_mem_ops);
diff --git a/lib/pci.c b/lib/pci.c
index 6bd54cb..8f2356d 100644
--- a/lib/pci.c
+++ b/lib/pci.c
@@ -13,12 +13,18 @@ bool pci_dev_exists(pcidevaddr_t dev)
pci_config_readw(dev, PCI_DEVICE_ID) != 0xffff);
}
+void pci_dev_init(struct pci_dev *dev, pcidevaddr_t bdf)
+{
+ memset(dev, 0, sizeof(*dev));
+ dev->bdf = bdf;
+}
+
/* Scan bus look for a specific device. Only bus 0 scanned for now. */
pcidevaddr_t pci_find_dev(uint16_t vendor_id, uint16_t device_id)
{
pcidevaddr_t dev;
- for (dev = 0; dev < 256; ++dev) {
+ for (dev = 0; dev < PCI_DEVFN_MAX; ++dev) {
if (pci_config_readw(dev, PCI_VENDOR_ID) == vendor_id &&
pci_config_readw(dev, PCI_DEVICE_ID) == device_id)
return dev;
@@ -33,12 +39,13 @@ uint32_t pci_bar_mask(uint32_t bar)
PCI_BASE_ADDRESS_IO_MASK : PCI_BASE_ADDRESS_MEM_MASK;
}
-uint32_t pci_bar_get(pcidevaddr_t dev, int bar_num)
+uint32_t pci_bar_get(struct pci_dev *dev, int bar_num)
{
- return pci_config_readl(dev, PCI_BASE_ADDRESS_0 + bar_num * 4);
+ return pci_config_readl(dev->bdf, PCI_BASE_ADDRESS_0 +
+ bar_num * 4);
}
-phys_addr_t pci_bar_get_addr(pcidevaddr_t dev, int bar_num)
+phys_addr_t pci_bar_get_addr(struct pci_dev *dev, int bar_num)
{
uint32_t bar = pci_bar_get(dev, bar_num);
uint32_t mask = pci_bar_mask(bar);
@@ -47,17 +54,18 @@ phys_addr_t pci_bar_get_addr(pcidevaddr_t dev, int bar_num)
if (pci_bar_is64(dev, bar_num))
addr |= (uint64_t)pci_bar_get(dev, bar_num + 1) << 32;
- return pci_translate_addr(dev, addr);
+ return pci_translate_addr(dev->bdf, addr);
}
-void pci_bar_set_addr(pcidevaddr_t dev, int bar_num, phys_addr_t addr)
+void pci_bar_set_addr(struct pci_dev *dev, int bar_num, phys_addr_t addr)
{
int off = PCI_BASE_ADDRESS_0 + bar_num * 4;
- pci_config_writel(dev, off, (uint32_t)addr);
+ pci_config_writel(dev->bdf, off, (uint32_t)addr);
if (pci_bar_is64(dev, bar_num))
- pci_config_writel(dev, off + 4, (uint32_t)(addr >> 32));
+ pci_config_writel(dev->bdf, off + 4,
+ (uint32_t)(addr >> 32));
}
/*
@@ -70,20 +78,21 @@ void pci_bar_set_addr(pcidevaddr_t dev, int bar_num, phys_addr_t addr)
* The following pci_bar_size_helper() and pci_bar_size() functions
* implement the algorithm.
*/
-static uint32_t pci_bar_size_helper(pcidevaddr_t dev, int bar_num)
+static uint32_t pci_bar_size_helper(struct pci_dev *dev, int bar_num)
{
int off = PCI_BASE_ADDRESS_0 + bar_num * 4;
+ uint16_t bdf = dev->bdf;
uint32_t bar, val;
- bar = pci_config_readl(dev, off);
- pci_config_writel(dev, off, ~0u);
- val = pci_config_readl(dev, off);
- pci_config_writel(dev, off, bar);
+ bar = pci_config_readl(bdf, off);
+ pci_config_writel(bdf, off, ~0u);
+ val = pci_config_readl(bdf, off);
+ pci_config_writel(bdf, off, bar);
return val;
}
-phys_addr_t pci_bar_size(pcidevaddr_t dev, int bar_num)
+phys_addr_t pci_bar_size(struct pci_dev *dev, int bar_num)
{
uint32_t bar, size;
@@ -104,19 +113,19 @@ phys_addr_t pci_bar_size(pcidevaddr_t dev, int bar_num)
}
}
-bool pci_bar_is_memory(pcidevaddr_t dev, int bar_num)
+bool pci_bar_is_memory(struct pci_dev *dev, int bar_num)
{
uint32_t bar = pci_bar_get(dev, bar_num);
return !(bar & PCI_BASE_ADDRESS_SPACE_IO);
}
-bool pci_bar_is_valid(pcidevaddr_t dev, int bar_num)
+bool pci_bar_is_valid(struct pci_dev *dev, int bar_num)
{
return pci_bar_get(dev, bar_num);
}
-bool pci_bar_is64(pcidevaddr_t dev, int bar_num)
+bool pci_bar_is64(struct pci_dev *dev, int bar_num)
{
uint32_t bar = pci_bar_get(dev, bar_num);
@@ -127,7 +136,7 @@ bool pci_bar_is64(pcidevaddr_t dev, int bar_num)
PCI_BASE_ADDRESS_MEM_TYPE_64;
}
-void pci_bar_print(pcidevaddr_t dev, int bar_num)
+void pci_bar_print(struct pci_dev *dev, int bar_num)
{
phys_addr_t size, start, end;
uint32_t bar;
@@ -186,8 +195,11 @@ static void pci_dev_print(pcidevaddr_t dev)
uint8_t progif = pci_config_readb(dev, PCI_CLASS_PROG);
uint8_t subclass = pci_config_readb(dev, PCI_CLASS_DEVICE);
uint8_t class = pci_config_readb(dev, PCI_CLASS_DEVICE + 1);
+ struct pci_dev pci_dev;
int i;
+ pci_dev_init(&pci_dev, dev);
+
pci_dev_print_id(dev);
printf(" type %02x progif %02x class %02x subclass %02x\n",
header, progif, class, subclass);
@@ -196,12 +208,12 @@ static void pci_dev_print(pcidevaddr_t dev)
return;
for (i = 0; i < 6; i++) {
- if (pci_bar_size(dev, i)) {
+ if (pci_bar_size(&pci_dev, i)) {
printf("\t");
- pci_bar_print(dev, i);
+ pci_bar_print(&pci_dev, i);
printf("\n");
}
- if (pci_bar_is64(dev, i))
+ if (pci_bar_is64(&pci_dev, i))
i++;
}
}
@@ -210,7 +222,7 @@ void pci_print(void)
{
pcidevaddr_t dev;
- for (dev = 0; dev < 256; ++dev) {
+ for (dev = 0; dev < PCI_DEVFN_MAX; ++dev) {
if (pci_dev_exists(dev))
pci_dev_print(dev);
}
diff --git a/lib/pci.h b/lib/pci.h
index 30f5381..d7140dd 100644
--- a/lib/pci.h
+++ b/lib/pci.h
@@ -15,6 +15,14 @@ enum {
PCIDEVADDR_INVALID = 0xffff,
};
+#define PCI_DEVFN_MAX 256
+
+struct pci_dev {
+ uint16_t bdf;
+};
+
+void pci_dev_init(struct pci_dev *dev, pcidevaddr_t bdf);
+
extern bool pci_probe(void);
extern void pci_print(void);
extern bool pci_dev_exists(pcidevaddr_t dev);
@@ -32,15 +40,15 @@ extern pcidevaddr_t pci_find_dev(uint16_t vendor_id, uint16_t device_id);
* It is expected the caller is aware of the device BAR layout and never
* tries to address the middle of a 64-bit register.
*/
-extern phys_addr_t pci_bar_get_addr(pcidevaddr_t dev, int bar_num);
-extern void pci_bar_set_addr(pcidevaddr_t dev, int bar_num, phys_addr_t addr);
-extern phys_addr_t pci_bar_size(pcidevaddr_t dev, int bar_num);
-extern uint32_t pci_bar_get(pcidevaddr_t dev, int bar_num);
+extern phys_addr_t pci_bar_get_addr(struct pci_dev *dev, int bar_num);
+extern void pci_bar_set_addr(struct pci_dev *dev, int bar_num, phys_addr_t addr);
+extern phys_addr_t pci_bar_size(struct pci_dev *dev, int bar_num);
+extern uint32_t pci_bar_get(struct pci_dev *dev, int bar_num);
extern uint32_t pci_bar_mask(uint32_t bar);
-extern bool pci_bar_is64(pcidevaddr_t dev, int bar_num);
-extern bool pci_bar_is_memory(pcidevaddr_t dev, int bar_num);
-extern bool pci_bar_is_valid(pcidevaddr_t dev, int bar_num);
-extern void pci_bar_print(pcidevaddr_t dev, int bar_num);
+extern bool pci_bar_is64(struct pci_dev *dev, int bar_num);
+extern bool pci_bar_is_memory(struct pci_dev *dev, int bar_num);
+extern bool pci_bar_is_valid(struct pci_dev *dev, int bar_num);
+extern void pci_bar_print(struct pci_dev *dev, int bar_num);
extern void pci_dev_print_id(pcidevaddr_t dev);
int pci_testdev(void);
diff --git a/x86/vmexit.c b/x86/vmexit.c
index 2d99d5f..63fa070 100644
--- a/x86/vmexit.c
+++ b/x86/vmexit.c
@@ -372,7 +372,8 @@ int main(int ac, char **av)
struct fadt_descriptor_rev1 *fadt;
int i;
unsigned long membar = 0;
- pcidevaddr_t pcidev;
+ struct pci_dev pcidev;
+ int ret;
smp_init();
setup_vm();
@@ -385,21 +386,22 @@ int main(int ac, char **av)
pm_tmr_blk = fadt->pm_tmr_blk;
printf("PM timer port is %x\n", pm_tmr_blk);
- pcidev = pci_find_dev(PCI_VENDOR_ID_REDHAT, PCI_DEVICE_ID_REDHAT_TEST);
- if (pcidev != PCIDEVADDR_INVALID) {
+ ret = pci_find_dev(PCI_VENDOR_ID_REDHAT, PCI_DEVICE_ID_REDHAT_TEST);
+ if (ret != PCIDEVADDR_INVALID) {
+ pci_dev_init(&pcidev, ret);
for (i = 0; i < PCI_TESTDEV_NUM_BARS; i++) {
- if (!pci_bar_is_valid(pcidev, i)) {
+ if (!pci_bar_is_valid(&pcidev, i)) {
continue;
}
- if (pci_bar_is_memory(pcidev, i)) {
- membar = pci_bar_get_addr(pcidev, i);
+ if (pci_bar_is_memory(&pcidev, i)) {
+ membar = pci_bar_get_addr(&pcidev, i);
pci_test.memaddr = ioremap(membar, PAGE_SIZE);
} else {
- pci_test.iobar = pci_bar_get_addr(pcidev, i);
+ pci_test.iobar = pci_bar_get_addr(&pcidev, i);
}
}
printf("pci-testdev at 0x%x membar %lx iobar %x\n",
- pcidev, membar, pci_test.iobar);
+ pcidev.bdf, membar, pci_test.iobar);
}
for (i = 0; i < ARRAY_SIZE(tests); ++i)
--
2.7.4
^ permalink raw reply related [flat|nested] 29+ messages in thread* [PATCH v3 08/13] pci: provide pci_scan_bars()
2016-11-14 22:27 [PATCH v3 00/13] VT-d unit test Peter Xu
` (6 preceding siblings ...)
2016-11-14 22:28 ` [PATCH v3 07/13] pci: introduce struct pci_dev Peter Xu
@ 2016-11-14 22:28 ` Peter Xu
2016-11-15 8:54 ` Andrew Jones
2016-11-14 22:28 ` [PATCH v3 09/13] pci: provide pci_enable_defaults() Peter Xu
` (6 subsequent siblings)
14 siblings, 1 reply; 29+ messages in thread
From: Peter Xu @ 2016-11-14 22:28 UTC (permalink / raw)
To: kvm; +Cc: drjones, agordeev, jan.kiszka, rkrcmar, pbonzini, peterx
Let's provide a more general way to scan PCI bars, rather than read the
config registers every time.
Then let x86/vmexit.c leverage pci_scan_bars()
Signed-off-by: Peter Xu <peterx@redhat.com>
---
lib/pci.c | 17 ++++++++++++++++-
lib/pci.h | 5 +++++
x86/vmexit.c | 17 ++++++-----------
3 files changed, 27 insertions(+), 12 deletions(-)
diff --git a/lib/pci.c b/lib/pci.c
index 8f2356d..462c370 100644
--- a/lib/pci.c
+++ b/lib/pci.c
@@ -207,7 +207,7 @@ static void pci_dev_print(pcidevaddr_t dev)
if ((header & PCI_HEADER_TYPE_MASK) != PCI_HEADER_TYPE_NORMAL)
return;
- for (i = 0; i < 6; i++) {
+ for (i = 0; i < PCI_BAR_NUM; i++) {
if (pci_bar_size(&pci_dev, i)) {
printf("\t");
pci_bar_print(&pci_dev, i);
@@ -227,3 +227,18 @@ void pci_print(void)
pci_dev_print(dev);
}
}
+
+void pci_scan_bars(struct pci_dev *dev)
+{
+ int i;
+
+ for (i = 0; i < PCI_BAR_NUM; i++) {
+ if (!pci_bar_is_valid(dev, i))
+ continue;
+ dev->bar[i] = pci_bar_get_addr(dev, i);
+ if (pci_bar_is64(dev, i)) {
+ i++;
+ dev->bar[i] = (phys_addr_t)0;
+ }
+ }
+}
diff --git a/lib/pci.h b/lib/pci.h
index d7140dd..338ad51 100644
--- a/lib/pci.h
+++ b/lib/pci.h
@@ -15,13 +15,16 @@ enum {
PCIDEVADDR_INVALID = 0xffff,
};
+#define PCI_BAR_NUM 6
#define PCI_DEVFN_MAX 256
struct pci_dev {
uint16_t bdf;
+ phys_addr_t bar[PCI_BAR_NUM];
};
void pci_dev_init(struct pci_dev *dev, pcidevaddr_t bdf);
+void pci_scan_bars(struct pci_dev *dev);
extern bool pci_probe(void);
extern void pci_print(void);
@@ -65,6 +68,8 @@ int pci_testdev(void);
* pci-testdev supports at least three types of tests (via mmio and
* portio BARs): no-eventfd, wildcard-eventfd and datamatch-eventfd
*/
+#define PCI_TESTDEV_BAR_MEM 0
+#define PCI_TESTDEV_BAR_IO 1
#define PCI_TESTDEV_NUM_BARS 2
#define PCI_TESTDEV_NUM_TESTS 3
diff --git a/x86/vmexit.c b/x86/vmexit.c
index 63fa070..a22f43f 100644
--- a/x86/vmexit.c
+++ b/x86/vmexit.c
@@ -389,17 +389,12 @@ int main(int ac, char **av)
ret = pci_find_dev(PCI_VENDOR_ID_REDHAT, PCI_DEVICE_ID_REDHAT_TEST);
if (ret != PCIDEVADDR_INVALID) {
pci_dev_init(&pcidev, ret);
- for (i = 0; i < PCI_TESTDEV_NUM_BARS; i++) {
- if (!pci_bar_is_valid(&pcidev, i)) {
- continue;
- }
- if (pci_bar_is_memory(&pcidev, i)) {
- membar = pci_bar_get_addr(&pcidev, i);
- pci_test.memaddr = ioremap(membar, PAGE_SIZE);
- } else {
- pci_test.iobar = pci_bar_get_addr(&pcidev, i);
- }
- }
+ pci_scan_bars(&pcidev);
+ assert(pci_bar_is_memory(&pcidev, PCI_TESTDEV_BAR_MEM));
+ assert(!pci_bar_is_memory(&pcidev, PCI_TESTDEV_BAR_IO));
+ membar = pcidev.bar[PCI_TESTDEV_BAR_MEM];
+ pci_test.memaddr = ioremap(membar, PAGE_SIZE);
+ pci_test.iobar = pcidev.bar[PCI_TESTDEV_BAR_IO];
printf("pci-testdev at 0x%x membar %lx iobar %x\n",
pcidev.bdf, membar, pci_test.iobar);
}
--
2.7.4
^ permalink raw reply related [flat|nested] 29+ messages in thread* Re: [PATCH v3 08/13] pci: provide pci_scan_bars()
2016-11-14 22:28 ` [PATCH v3 08/13] pci: provide pci_scan_bars() Peter Xu
@ 2016-11-15 8:54 ` Andrew Jones
2016-11-15 16:32 ` Peter Xu
0 siblings, 1 reply; 29+ messages in thread
From: Andrew Jones @ 2016-11-15 8:54 UTC (permalink / raw)
To: Peter Xu; +Cc: kvm, agordeev, jan.kiszka, rkrcmar, pbonzini
On Mon, Nov 14, 2016 at 05:28:06PM -0500, Peter Xu wrote:
> Let's provide a more general way to scan PCI bars, rather than read the
> config registers every time.
>
> Then let x86/vmexit.c leverage pci_scan_bars()
>
> Signed-off-by: Peter Xu <peterx@redhat.com>
> ---
> lib/pci.c | 17 ++++++++++++++++-
> lib/pci.h | 5 +++++
> x86/vmexit.c | 17 ++++++-----------
> 3 files changed, 27 insertions(+), 12 deletions(-)
>
> diff --git a/lib/pci.c b/lib/pci.c
> index 8f2356d..462c370 100644
> --- a/lib/pci.c
> +++ b/lib/pci.c
> @@ -207,7 +207,7 @@ static void pci_dev_print(pcidevaddr_t dev)
> if ((header & PCI_HEADER_TYPE_MASK) != PCI_HEADER_TYPE_NORMAL)
> return;
>
> - for (i = 0; i < 6; i++) {
> + for (i = 0; i < PCI_BAR_NUM; i++) {
> if (pci_bar_size(&pci_dev, i)) {
> printf("\t");
> pci_bar_print(&pci_dev, i);
> @@ -227,3 +227,18 @@ void pci_print(void)
> pci_dev_print(dev);
> }
> }
> +
> +void pci_scan_bars(struct pci_dev *dev)
> +{
> + int i;
> +
> + for (i = 0; i < PCI_BAR_NUM; i++) {
> + if (!pci_bar_is_valid(dev, i))
> + continue;
> + dev->bar[i] = pci_bar_get_addr(dev, i);
> + if (pci_bar_is64(dev, i)) {
> + i++;
> + dev->bar[i] = (phys_addr_t)0;
> + }
> + }
> +}
> diff --git a/lib/pci.h b/lib/pci.h
> index d7140dd..338ad51 100644
> --- a/lib/pci.h
> +++ b/lib/pci.h
> @@ -15,13 +15,16 @@ enum {
> PCIDEVADDR_INVALID = 0xffff,
> };
>
> +#define PCI_BAR_NUM 6
> #define PCI_DEVFN_MAX 256
>
> struct pci_dev {
> uint16_t bdf;
> + phys_addr_t bar[PCI_BAR_NUM];
> };
>
> void pci_dev_init(struct pci_dev *dev, pcidevaddr_t bdf);
> +void pci_scan_bars(struct pci_dev *dev);
>
> extern bool pci_probe(void);
> extern void pci_print(void);
> @@ -65,6 +68,8 @@ int pci_testdev(void);
> * pci-testdev supports at least three types of tests (via mmio and
> * portio BARs): no-eventfd, wildcard-eventfd and datamatch-eventfd
> */
> +#define PCI_TESTDEV_BAR_MEM 0
> +#define PCI_TESTDEV_BAR_IO 1
> #define PCI_TESTDEV_NUM_BARS 2
> #define PCI_TESTDEV_NUM_TESTS 3
>
> diff --git a/x86/vmexit.c b/x86/vmexit.c
> index 63fa070..a22f43f 100644
> --- a/x86/vmexit.c
> +++ b/x86/vmexit.c
> @@ -389,17 +389,12 @@ int main(int ac, char **av)
> ret = pci_find_dev(PCI_VENDOR_ID_REDHAT, PCI_DEVICE_ID_REDHAT_TEST);
> if (ret != PCIDEVADDR_INVALID) {
> pci_dev_init(&pcidev, ret);
> - for (i = 0; i < PCI_TESTDEV_NUM_BARS; i++) {
> - if (!pci_bar_is_valid(&pcidev, i)) {
> - continue;
> - }
> - if (pci_bar_is_memory(&pcidev, i)) {
> - membar = pci_bar_get_addr(&pcidev, i);
> - pci_test.memaddr = ioremap(membar, PAGE_SIZE);
> - } else {
> - pci_test.iobar = pci_bar_get_addr(&pcidev, i);
> - }
> - }
> + pci_scan_bars(&pcidev);
> + assert(pci_bar_is_memory(&pcidev, PCI_TESTDEV_BAR_MEM));
> + assert(!pci_bar_is_memory(&pcidev, PCI_TESTDEV_BAR_IO));
> + membar = pcidev.bar[PCI_TESTDEV_BAR_MEM];
> + pci_test.memaddr = ioremap(membar, PAGE_SIZE);
> + pci_test.iobar = pcidev.bar[PCI_TESTDEV_BAR_IO];
> printf("pci-testdev at 0x%x membar %lx iobar %x\n",
> pcidev.bdf, membar, pci_test.iobar);
> }
> --
> 2.7.4
>
You're still missing at least one 256 and one 6 that you wanted to change
to PCI_DEVFN_MAX, PCI_BAR_NUM. See lib/pci-host-generic.c:pci_probe. Also,
Alex's series is merged now so you can rebase on master to get it.
Otherwise
Reviewed-by: Andrew Jones <drjones@redhat.com>
Thanks,
drew
^ permalink raw reply [flat|nested] 29+ messages in thread* Re: [PATCH v3 08/13] pci: provide pci_scan_bars()
2016-11-15 8:54 ` Andrew Jones
@ 2016-11-15 16:32 ` Peter Xu
0 siblings, 0 replies; 29+ messages in thread
From: Peter Xu @ 2016-11-15 16:32 UTC (permalink / raw)
To: Andrew Jones; +Cc: kvm, agordeev, jan.kiszka, rkrcmar, pbonzini
On Tue, Nov 15, 2016 at 09:54:54AM +0100, Andrew Jones wrote:
[...]
> You're still missing at least one 256 and one 6 that you wanted to change
> to PCI_DEVFN_MAX, PCI_BAR_NUM. See lib/pci-host-generic.c:pci_probe. Also,
> Alex's series is merged now so you can rebase on master to get it.
Rebased and replaced.
>
> Otherwise
> Reviewed-by: Andrew Jones <drjones@redhat.com>
Thanks,
-- peterx
^ permalink raw reply [flat|nested] 29+ messages in thread
* [PATCH v3 09/13] pci: provide pci_enable_defaults()
2016-11-14 22:27 [PATCH v3 00/13] VT-d unit test Peter Xu
` (7 preceding siblings ...)
2016-11-14 22:28 ` [PATCH v3 08/13] pci: provide pci_scan_bars() Peter Xu
@ 2016-11-14 22:28 ` Peter Xu
2016-11-15 8:58 ` Andrew Jones
2016-11-14 22:28 ` [PATCH v3 10/13] pci: edu: introduce pci-edu helpers Peter Xu
` (5 subsequent siblings)
14 siblings, 1 reply; 29+ messages in thread
From: Peter Xu @ 2016-11-14 22:28 UTC (permalink / raw)
To: kvm; +Cc: drjones, agordeev, jan.kiszka, rkrcmar, pbonzini, peterx
One helper function to set/clear specific bit in PCI_COMMAND register is
introduced. Then, provide a function to do most of the common PCI init
work.
Suggested-by: Andrew Jones <drjones@redhat.com>
Signed-off-by: Peter Xu <peterx@redhat.com>
---
lib/pci.c | 19 +++++++++++++++++++
lib/pci.h | 2 ++
2 files changed, 21 insertions(+)
diff --git a/lib/pci.c b/lib/pci.c
index 462c370..1c59e13 100644
--- a/lib/pci.c
+++ b/lib/pci.c
@@ -7,6 +7,18 @@
#include "pci.h"
#include "asm/pci.h"
+void pci_cmd_set_clr(struct pci_dev *dev, uint16_t set, uint16_t clr)
+{
+ uint16_t val = pci_config_readw(dev->bdf, PCI_COMMAND);
+
+ /* No overlap is allowed */
+ assert((set & clr) == 0);
+ val |= set;
+ val &= ~clr;
+
+ pci_config_writew(dev->bdf, PCI_COMMAND, val);
+}
+
bool pci_dev_exists(pcidevaddr_t dev)
{
return (pci_config_readw(dev, PCI_VENDOR_ID) != 0xffff &&
@@ -242,3 +254,10 @@ void pci_scan_bars(struct pci_dev *dev)
}
}
}
+
+void pci_enable_defaults(struct pci_dev *dev)
+{
+ pci_scan_bars(dev);
+ /* Enable device DMA operations */
+ pci_cmd_set_clr(dev, PCI_COMMAND_MASTER, 0);
+}
diff --git a/lib/pci.h b/lib/pci.h
index 338ad51..1e7be27 100644
--- a/lib/pci.h
+++ b/lib/pci.h
@@ -25,6 +25,8 @@ struct pci_dev {
void pci_dev_init(struct pci_dev *dev, pcidevaddr_t bdf);
void pci_scan_bars(struct pci_dev *dev);
+void pci_cmd_set_clr(struct pci_dev *dev, uint16_t set, uint16_t clr);
+void pci_enable_defaults(struct pci_dev *dev);
extern bool pci_probe(void);
extern void pci_print(void);
--
2.7.4
^ permalink raw reply related [flat|nested] 29+ messages in thread* Re: [PATCH v3 09/13] pci: provide pci_enable_defaults()
2016-11-14 22:28 ` [PATCH v3 09/13] pci: provide pci_enable_defaults() Peter Xu
@ 2016-11-15 8:58 ` Andrew Jones
2016-11-15 16:42 ` Peter Xu
0 siblings, 1 reply; 29+ messages in thread
From: Andrew Jones @ 2016-11-15 8:58 UTC (permalink / raw)
To: Peter Xu; +Cc: kvm, agordeev, jan.kiszka, rkrcmar, pbonzini
On Mon, Nov 14, 2016 at 05:28:07PM -0500, Peter Xu wrote:
> One helper function to set/clear specific bit in PCI_COMMAND register is
> introduced. Then, provide a function to do most of the common PCI init
> work.
>
> Suggested-by: Andrew Jones <drjones@redhat.com>
> Signed-off-by: Peter Xu <peterx@redhat.com>
> ---
> lib/pci.c | 19 +++++++++++++++++++
> lib/pci.h | 2 ++
> 2 files changed, 21 insertions(+)
>
> diff --git a/lib/pci.c b/lib/pci.c
> index 462c370..1c59e13 100644
> --- a/lib/pci.c
> +++ b/lib/pci.c
> @@ -7,6 +7,18 @@
> #include "pci.h"
> #include "asm/pci.h"
>
> +void pci_cmd_set_clr(struct pci_dev *dev, uint16_t set, uint16_t clr)
> +{
> + uint16_t val = pci_config_readw(dev->bdf, PCI_COMMAND);
> +
> + /* No overlap is allowed */
> + assert((set & clr) == 0);
> + val |= set;
> + val &= ~clr;
> +
> + pci_config_writew(dev->bdf, PCI_COMMAND, val);
> +}
> +
> bool pci_dev_exists(pcidevaddr_t dev)
> {
> return (pci_config_readw(dev, PCI_VENDOR_ID) != 0xffff &&
> @@ -242,3 +254,10 @@ void pci_scan_bars(struct pci_dev *dev)
> }
> }
> }
> +
> +void pci_enable_defaults(struct pci_dev *dev)
> +{
> + pci_scan_bars(dev);
> + /* Enable device DMA operations */
> + pci_cmd_set_clr(dev, PCI_COMMAND_MASTER, 0);
> +}
> diff --git a/lib/pci.h b/lib/pci.h
> index 338ad51..1e7be27 100644
> --- a/lib/pci.h
> +++ b/lib/pci.h
> @@ -25,6 +25,8 @@ struct pci_dev {
>
> void pci_dev_init(struct pci_dev *dev, pcidevaddr_t bdf);
> void pci_scan_bars(struct pci_dev *dev);
> +void pci_cmd_set_clr(struct pci_dev *dev, uint16_t set, uint16_t clr);
> +void pci_enable_defaults(struct pci_dev *dev);
Don't forget the [pointless] externs on these functions for this patch,
and the ones that introduced the functions above.
>
> extern bool pci_probe(void);
> extern void pci_print(void);
> --
> 2.7.4
I think I already gave my r-b on this patch last time, but anyway, here it
is again
Reviewed-by: Andrew Jones <drjones@redhat.com>
drew
^ permalink raw reply [flat|nested] 29+ messages in thread* Re: [PATCH v3 09/13] pci: provide pci_enable_defaults()
2016-11-15 8:58 ` Andrew Jones
@ 2016-11-15 16:42 ` Peter Xu
2016-11-15 17:15 ` Andrew Jones
0 siblings, 1 reply; 29+ messages in thread
From: Peter Xu @ 2016-11-15 16:42 UTC (permalink / raw)
To: Andrew Jones; +Cc: kvm, agordeev, jan.kiszka, rkrcmar, pbonzini
On Tue, Nov 15, 2016 at 09:58:40AM +0100, Andrew Jones wrote:
[...]
> > @@ -25,6 +25,8 @@ struct pci_dev {
> >
> > void pci_dev_init(struct pci_dev *dev, pcidevaddr_t bdf);
> > void pci_scan_bars(struct pci_dev *dev);
> > +void pci_cmd_set_clr(struct pci_dev *dev, uint16_t set, uint16_t clr);
> > +void pci_enable_defaults(struct pci_dev *dev);
>
> Don't forget the [pointless] externs on these functions for this patch,
> and the ones that introduced the functions above.
Adding all of them missing for pci.h. Actually I see there's still one
missing:
int pci_testdev(void);
But I won't touch it since that's irrelevant to this series (and I am
always okay /wo "extern" :).
[...]
> Reviewed-by: Andrew Jones <drjones@redhat.com>
Adding this too. Thanks,
-- peterx
^ permalink raw reply [flat|nested] 29+ messages in thread* Re: [PATCH v3 09/13] pci: provide pci_enable_defaults()
2016-11-15 16:42 ` Peter Xu
@ 2016-11-15 17:15 ` Andrew Jones
0 siblings, 0 replies; 29+ messages in thread
From: Andrew Jones @ 2016-11-15 17:15 UTC (permalink / raw)
To: Peter Xu; +Cc: kvm, agordeev, jan.kiszka, rkrcmar, pbonzini
On Tue, Nov 15, 2016 at 11:42:43AM -0500, Peter Xu wrote:
> On Tue, Nov 15, 2016 at 09:58:40AM +0100, Andrew Jones wrote:
>
> [...]
>
> > > @@ -25,6 +25,8 @@ struct pci_dev {
> > >
> > > void pci_dev_init(struct pci_dev *dev, pcidevaddr_t bdf);
> > > void pci_scan_bars(struct pci_dev *dev);
> > > +void pci_cmd_set_clr(struct pci_dev *dev, uint16_t set, uint16_t clr);
> > > +void pci_enable_defaults(struct pci_dev *dev);
> >
> > Don't forget the [pointless] externs on these functions for this patch,
> > and the ones that introduced the functions above.
>
> Adding all of them missing for pci.h. Actually I see there's still one
> missing:
>
> int pci_testdev(void);
>
> But I won't touch it since that's irrelevant to this series (and I am
> always okay /wo "extern" :).
"fix" that one too please.
thanks,
drew
>
> [...]
>
> > Reviewed-by: Andrew Jones <drjones@redhat.com>
>
> Adding this too. Thanks,
>
> -- peterx
> --
> To unsubscribe from this list: send the line "unsubscribe kvm" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 29+ messages in thread
* [PATCH v3 10/13] pci: edu: introduce pci-edu helpers
2016-11-14 22:27 [PATCH v3 00/13] VT-d unit test Peter Xu
` (8 preceding siblings ...)
2016-11-14 22:28 ` [PATCH v3 09/13] pci: provide pci_enable_defaults() Peter Xu
@ 2016-11-14 22:28 ` Peter Xu
2016-11-15 9:02 ` Andrew Jones
2016-11-14 22:28 ` [PATCH v3 11/13] x86: intel-iommu: add dmar test Peter Xu
` (4 subsequent siblings)
14 siblings, 1 reply; 29+ messages in thread
From: Peter Xu @ 2016-11-14 22:28 UTC (permalink / raw)
To: kvm; +Cc: drjones, agordeev, jan.kiszka, rkrcmar, pbonzini, peterx
QEMU edu device is a pci device that is originally written for
educational purpose, however it also suits for IOMMU unit test. Adding
helpers for this specific device to implement the device logic.
The device supports lots of functions, here only DMA operation is
supported.
The spec of the device can be found at:
https://github.com/qemu/qemu/blob/master/docs/specs/edu.txt
Reviewed-by: Andrew Jones <drjones@redhat.com>
Signed-off-by: Peter Xu <peterx@redhat.com>
---
lib/pci-edu.c | 73 ++++++++++++++++++++++++++++++++++++++++++++++++++++
lib/pci-edu.h | 82 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
2 files changed, 155 insertions(+)
create mode 100644 lib/pci-edu.c
create mode 100644 lib/pci-edu.h
diff --git a/lib/pci-edu.c b/lib/pci-edu.c
new file mode 100644
index 0000000..c9a8b0c
--- /dev/null
+++ b/lib/pci-edu.c
@@ -0,0 +1,73 @@
+/*
+ * Edu PCI device.
+ *
+ * Copyright (C) 2016 Red Hat, Inc.
+ *
+ * Authors:
+ * Peter Xu <peterx@redhat.com>,
+ *
+ * This work is licensed under the terms of the GNU LGPL, version 2 or
+ * later.
+ */
+
+#include "pci-edu.h"
+#include "asm/barrier.h"
+
+/* Return true if alive */
+static inline bool edu_check_alive(struct pci_edu_dev *dev)
+{
+ static uint32_t live_count = 1;
+ uint32_t value;
+
+ edu_reg_writel(dev, EDU_REG_ALIVE, live_count++);
+ value = edu_reg_readl(dev, EDU_REG_ALIVE);
+ return (live_count - 1 == ~value);
+}
+
+bool edu_init(struct pci_edu_dev *dev)
+{
+ pcidevaddr_t dev_addr;
+
+ dev_addr = pci_find_dev(PCI_VENDOR_ID_QEMU, PCI_DEVICE_ID_EDU);
+ if (dev_addr == PCIDEVADDR_INVALID)
+ return false;
+
+ pci_dev_init(&dev->pci_dev, dev_addr);
+ pci_enable_defaults(&dev->pci_dev);
+ assert(edu_check_alive(dev));
+ return true;
+}
+
+void edu_dma(struct pci_edu_dev *dev, iova_t iova,
+ size_t size, int dev_offset, bool from_device)
+{
+ uint64_t from, to;
+ uint32_t cmd = EDU_CMD_DMA_START;
+
+ assert(size <= EDU_DMA_SIZE_MAX);
+ assert(dev_offset < EDU_DMA_SIZE_MAX &&
+ dev_offset >= 0);
+
+ printf("edu device DMA start %s addr %p size 0x%lu off 0x%x\n",
+ from_device ? "FROM" : "TO",
+ (void *)iova, size, dev_offset);
+
+ if (from_device) {
+ from = dev_offset + EDU_DMA_START;
+ to = iova;
+ cmd |= EDU_CMD_DMA_FROM;
+ } else {
+ from = iova;
+ to = EDU_DMA_START + dev_offset;
+ cmd |= EDU_CMD_DMA_TO;
+ }
+
+ edu_reg_writeq(dev, EDU_REG_DMA_SRC, from);
+ edu_reg_writeq(dev, EDU_REG_DMA_DST, to);
+ edu_reg_writeq(dev, EDU_REG_DMA_COUNT, size);
+ edu_reg_writel(dev, EDU_REG_DMA_CMD, cmd);
+
+ /* Wait until DMA finished */
+ while (edu_reg_readl(dev, EDU_REG_DMA_CMD) & EDU_CMD_DMA_START)
+ cpu_relax();
+}
diff --git a/lib/pci-edu.h b/lib/pci-edu.h
new file mode 100644
index 0000000..b3a6e2b
--- /dev/null
+++ b/lib/pci-edu.h
@@ -0,0 +1,82 @@
+/*
+ * Edu PCI device header.
+ *
+ * Copyright (C) 2016 Red Hat, Inc.
+ *
+ * Authors:
+ * Peter Xu <peterx@redhat.com>,
+ *
+ * This work is licensed under the terms of the GNU LGPL, version 2 or
+ * later.
+ *
+ * Edu device is a virtualized device in QEMU. Please refer to
+ * docs/specs/edu.txt in QEMU repository for EDU device manual.
+ */
+#ifndef __PCI_EDU_H__
+#define __PCI_EDU_H__
+
+#include "pci.h"
+#include "asm/io.h"
+
+#define PCI_VENDOR_ID_QEMU 0x1234
+#define PCI_DEVICE_ID_EDU 0x11e8
+
+/* The only bar used by EDU device */
+#define EDU_BAR_MEM 0
+#define EDU_MAGIC 0xed
+#define EDU_VERSION 0x100
+#define EDU_DMA_BUF_SIZE (1 << 20)
+#define EDU_INPUT_BUF_SIZE 256
+
+#define EDU_REG_ID 0x0
+#define EDU_REG_ALIVE 0x4
+#define EDU_REG_FACTORIAL 0x8
+#define EDU_REG_STATUS 0x20
+#define EDU_REG_DMA_SRC 0x80
+#define EDU_REG_DMA_DST 0x88
+#define EDU_REG_DMA_COUNT 0x90
+#define EDU_REG_DMA_CMD 0x98
+
+#define EDU_CMD_DMA_START 0x01
+#define EDU_CMD_DMA_FROM 0x02
+#define EDU_CMD_DMA_TO 0x00
+
+#define EDU_STATUS_FACTORIAL 0x1
+#define EDU_STATUS_INT_ENABLE 0x80
+
+#define EDU_DMA_START 0x40000
+#define EDU_DMA_SIZE_MAX 4096
+
+struct pci_edu_dev {
+ struct pci_dev pci_dev;
+};
+
+#define edu_reg(d, r) (volatile void *)((d)->pci_dev.bar[EDU_BAR_MEM] + (r))
+
+static inline uint64_t edu_reg_readq(struct pci_edu_dev *dev, int reg)
+{
+ return __raw_readq(edu_reg(dev, reg));
+}
+
+static inline uint32_t edu_reg_readl(struct pci_edu_dev *dev, int reg)
+{
+ return __raw_readl(edu_reg(dev, reg));
+}
+
+static inline void edu_reg_writeq(struct pci_edu_dev *dev, int reg,
+ uint64_t val)
+{
+ __raw_writeq(val, edu_reg(dev, reg));
+}
+
+static inline void edu_reg_writel(struct pci_edu_dev *dev, int reg,
+ uint32_t val)
+{
+ __raw_writel(val, edu_reg(dev, reg));
+}
+
+bool edu_init(struct pci_edu_dev *dev);
+void edu_dma(struct pci_edu_dev *dev, iova_t iova,
+ size_t size, int dev_offset, bool from_device);
+
+#endif
--
2.7.4
^ permalink raw reply related [flat|nested] 29+ messages in thread* Re: [PATCH v3 10/13] pci: edu: introduce pci-edu helpers
2016-11-14 22:28 ` [PATCH v3 10/13] pci: edu: introduce pci-edu helpers Peter Xu
@ 2016-11-15 9:02 ` Andrew Jones
0 siblings, 0 replies; 29+ messages in thread
From: Andrew Jones @ 2016-11-15 9:02 UTC (permalink / raw)
To: Peter Xu; +Cc: kvm, agordeev, jan.kiszka, rkrcmar, pbonzini
On Mon, Nov 14, 2016 at 05:28:08PM -0500, Peter Xu wrote:
> QEMU edu device is a pci device that is originally written for
> educational purpose, however it also suits for IOMMU unit test. Adding
> helpers for this specific device to implement the device logic.
>
> The device supports lots of functions, here only DMA operation is
> supported.
>
> The spec of the device can be found at:
>
> https://github.com/qemu/qemu/blob/master/docs/specs/edu.txt
>
> Reviewed-by: Andrew Jones <drjones@redhat.com>
> Signed-off-by: Peter Xu <peterx@redhat.com>
> ---
> lib/pci-edu.c | 73 ++++++++++++++++++++++++++++++++++++++++++++++++++++
> lib/pci-edu.h | 82 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> 2 files changed, 155 insertions(+)
> create mode 100644 lib/pci-edu.c
> create mode 100644 lib/pci-edu.h
>
> diff --git a/lib/pci-edu.c b/lib/pci-edu.c
> new file mode 100644
> index 0000000..c9a8b0c
> --- /dev/null
> +++ b/lib/pci-edu.c
> @@ -0,0 +1,73 @@
> +/*
> + * Edu PCI device.
> + *
> + * Copyright (C) 2016 Red Hat, Inc.
> + *
> + * Authors:
> + * Peter Xu <peterx@redhat.com>,
> + *
> + * This work is licensed under the terms of the GNU LGPL, version 2 or
> + * later.
> + */
> +
> +#include "pci-edu.h"
> +#include "asm/barrier.h"
> +
> +/* Return true if alive */
> +static inline bool edu_check_alive(struct pci_edu_dev *dev)
> +{
> + static uint32_t live_count = 1;
> + uint32_t value;
> +
> + edu_reg_writel(dev, EDU_REG_ALIVE, live_count++);
> + value = edu_reg_readl(dev, EDU_REG_ALIVE);
> + return (live_count - 1 == ~value);
> +}
> +
> +bool edu_init(struct pci_edu_dev *dev)
> +{
> + pcidevaddr_t dev_addr;
> +
> + dev_addr = pci_find_dev(PCI_VENDOR_ID_QEMU, PCI_DEVICE_ID_EDU);
> + if (dev_addr == PCIDEVADDR_INVALID)
> + return false;
> +
> + pci_dev_init(&dev->pci_dev, dev_addr);
> + pci_enable_defaults(&dev->pci_dev);
> + assert(edu_check_alive(dev));
If we were to compile with assert as a no-op (which we never do...),
then we won't execute edu_check_alive. To be safe, you want
ret = edu_check_alive(dev)
assert(ret)
here and anywhere you use asserts.
> + return true;
> +}
> +
> +void edu_dma(struct pci_edu_dev *dev, iova_t iova,
> + size_t size, int dev_offset, bool from_device)
> +{
> + uint64_t from, to;
> + uint32_t cmd = EDU_CMD_DMA_START;
> +
> + assert(size <= EDU_DMA_SIZE_MAX);
> + assert(dev_offset < EDU_DMA_SIZE_MAX &&
> + dev_offset >= 0);
> +
> + printf("edu device DMA start %s addr %p size 0x%lu off 0x%x\n",
> + from_device ? "FROM" : "TO",
> + (void *)iova, size, dev_offset);
> +
> + if (from_device) {
> + from = dev_offset + EDU_DMA_START;
> + to = iova;
> + cmd |= EDU_CMD_DMA_FROM;
> + } else {
> + from = iova;
> + to = EDU_DMA_START + dev_offset;
> + cmd |= EDU_CMD_DMA_TO;
> + }
> +
> + edu_reg_writeq(dev, EDU_REG_DMA_SRC, from);
> + edu_reg_writeq(dev, EDU_REG_DMA_DST, to);
> + edu_reg_writeq(dev, EDU_REG_DMA_COUNT, size);
> + edu_reg_writel(dev, EDU_REG_DMA_CMD, cmd);
> +
> + /* Wait until DMA finished */
> + while (edu_reg_readl(dev, EDU_REG_DMA_CMD) & EDU_CMD_DMA_START)
> + cpu_relax();
> +}
> diff --git a/lib/pci-edu.h b/lib/pci-edu.h
> new file mode 100644
> index 0000000..b3a6e2b
> --- /dev/null
> +++ b/lib/pci-edu.h
> @@ -0,0 +1,82 @@
> +/*
> + * Edu PCI device header.
> + *
> + * Copyright (C) 2016 Red Hat, Inc.
> + *
> + * Authors:
> + * Peter Xu <peterx@redhat.com>,
> + *
> + * This work is licensed under the terms of the GNU LGPL, version 2 or
> + * later.
> + *
> + * Edu device is a virtualized device in QEMU. Please refer to
> + * docs/specs/edu.txt in QEMU repository for EDU device manual.
> + */
> +#ifndef __PCI_EDU_H__
> +#define __PCI_EDU_H__
> +
> +#include "pci.h"
> +#include "asm/io.h"
> +
> +#define PCI_VENDOR_ID_QEMU 0x1234
> +#define PCI_DEVICE_ID_EDU 0x11e8
> +
> +/* The only bar used by EDU device */
> +#define EDU_BAR_MEM 0
> +#define EDU_MAGIC 0xed
> +#define EDU_VERSION 0x100
> +#define EDU_DMA_BUF_SIZE (1 << 20)
> +#define EDU_INPUT_BUF_SIZE 256
> +
> +#define EDU_REG_ID 0x0
> +#define EDU_REG_ALIVE 0x4
> +#define EDU_REG_FACTORIAL 0x8
> +#define EDU_REG_STATUS 0x20
> +#define EDU_REG_DMA_SRC 0x80
> +#define EDU_REG_DMA_DST 0x88
> +#define EDU_REG_DMA_COUNT 0x90
> +#define EDU_REG_DMA_CMD 0x98
> +
> +#define EDU_CMD_DMA_START 0x01
> +#define EDU_CMD_DMA_FROM 0x02
> +#define EDU_CMD_DMA_TO 0x00
> +
> +#define EDU_STATUS_FACTORIAL 0x1
> +#define EDU_STATUS_INT_ENABLE 0x80
> +
> +#define EDU_DMA_START 0x40000
> +#define EDU_DMA_SIZE_MAX 4096
> +
> +struct pci_edu_dev {
> + struct pci_dev pci_dev;
> +};
> +
> +#define edu_reg(d, r) (volatile void *)((d)->pci_dev.bar[EDU_BAR_MEM] + (r))
> +
> +static inline uint64_t edu_reg_readq(struct pci_edu_dev *dev, int reg)
> +{
> + return __raw_readq(edu_reg(dev, reg));
> +}
> +
> +static inline uint32_t edu_reg_readl(struct pci_edu_dev *dev, int reg)
> +{
> + return __raw_readl(edu_reg(dev, reg));
> +}
> +
> +static inline void edu_reg_writeq(struct pci_edu_dev *dev, int reg,
> + uint64_t val)
> +{
> + __raw_writeq(val, edu_reg(dev, reg));
> +}
> +
> +static inline void edu_reg_writel(struct pci_edu_dev *dev, int reg,
> + uint32_t val)
> +{
> + __raw_writel(val, edu_reg(dev, reg));
> +}
> +
> +bool edu_init(struct pci_edu_dev *dev);
> +void edu_dma(struct pci_edu_dev *dev, iova_t iova,
> + size_t size, int dev_offset, bool from_device);
> +
> +#endif
> --
> 2.7.4
>
Thanks,
drew
^ permalink raw reply [flat|nested] 29+ messages in thread
* [PATCH v3 11/13] x86: intel-iommu: add dmar test
2016-11-14 22:27 [PATCH v3 00/13] VT-d unit test Peter Xu
` (9 preceding siblings ...)
2016-11-14 22:28 ` [PATCH v3 10/13] pci: edu: introduce pci-edu helpers Peter Xu
@ 2016-11-14 22:28 ` Peter Xu
2016-11-15 9:08 ` Andrew Jones
2016-11-14 22:28 ` [PATCH v3 12/13] pci: add msi support for 32/64bit address Peter Xu
` (3 subsequent siblings)
14 siblings, 1 reply; 29+ messages in thread
From: Peter Xu @ 2016-11-14 22:28 UTC (permalink / raw)
To: kvm; +Cc: drjones, agordeev, jan.kiszka, rkrcmar, pbonzini, peterx
DMAR test is based on QEMU edu device. A 4B DMA memory copy is carried
out as the simplest DMAR test.
Signed-off-by: Peter Xu <peterx@redhat.com>
---
lib/pci.h | 5 ++
lib/x86/intel-iommu.c | 131 ++++++++++++++++++++++++++++++++++++++++++++++++++
lib/x86/intel-iommu.h | 23 +++++++++
x86/Makefile.common | 1 +
x86/intel-iommu.c | 51 ++++++++++++++++++++
5 files changed, 211 insertions(+)
diff --git a/lib/pci.h b/lib/pci.h
index 1e7be27..a2002ad 100644
--- a/lib/pci.h
+++ b/lib/pci.h
@@ -18,6 +18,9 @@ enum {
#define PCI_BAR_NUM 6
#define PCI_DEVFN_MAX 256
+#define PCI_BDF_GET_DEVFN(x) ((x) & 0xff)
+#define PCI_BDF_GET_BUS(x) (((x) >> 8) & 0xff)
+
struct pci_dev {
uint16_t bdf;
phys_addr_t bar[PCI_BAR_NUM];
@@ -28,6 +31,8 @@ void pci_scan_bars(struct pci_dev *dev);
void pci_cmd_set_clr(struct pci_dev *dev, uint16_t set, uint16_t clr);
void pci_enable_defaults(struct pci_dev *dev);
+typedef phys_addr_t iova_t;
+
extern bool pci_probe(void);
extern void pci_print(void);
extern bool pci_dev_exists(pcidevaddr_t dev);
diff --git a/lib/x86/intel-iommu.c b/lib/x86/intel-iommu.c
index 9890f34..c1b6164 100644
--- a/lib/x86/intel-iommu.c
+++ b/lib/x86/intel-iommu.c
@@ -11,6 +11,42 @@
*/
#include "intel-iommu.h"
+#include "libcflat.h"
+
+/*
+ * VT-d in QEMU currently only support 39 bits address width, which is
+ * 3-level translation.
+ */
+#define VTD_PAGE_LEVEL 3
+#define VTD_CE_AW_39BIT 0x1
+
+typedef uint64_t vtd_pte_t;
+
+struct vtd_root_entry {
+ /* Quad 1 */
+ uint64_t present:1;
+ uint64_t __reserved:11;
+ uint64_t context_table_p:52;
+ /* Quad 2 */
+ uint64_t __reserved_2;
+} __attribute__ ((packed));
+typedef struct vtd_root_entry vtd_re_t;
+
+struct vtd_context_entry {
+ /* Quad 1 */
+ uint64_t present:1;
+ uint64_t disable_fault_report:1;
+ uint64_t trans_type:2;
+ uint64_t __reserved:8;
+ uint64_t slptptr:52;
+ /* Quad 2 */
+ uint64_t addr_width:3;
+ uint64_t __ignore:4;
+ uint64_t __reserved_2:1;
+ uint64_t domain_id:16;
+ uint64_t __reserved_3:40;
+} __attribute__ ((packed));
+typedef struct vtd_context_entry vtd_ce_t;
#define VTD_RTA_MASK (PAGE_MASK)
#define VTD_IRTA_MASK (PAGE_MASK)
@@ -74,6 +110,101 @@ static void vtd_setup_ir_table(void)
printf("IR table address: 0x%016lx\n", vtd_ir_table());
}
+static void vtd_install_pte(vtd_pte_t *root, iova_t iova,
+ phys_addr_t pa, int level_target)
+{
+ int level;
+ unsigned int offset;
+ void *page;
+
+ for (level = VTD_PAGE_LEVEL; level > level_target; level--) {
+ offset = PGDIR_OFFSET(iova, level);
+ if (!(root[offset] & VTD_PTE_RW)) {
+ page = alloc_page();
+ memset(page, 0, PAGE_SIZE);
+ root[offset] = virt_to_phys(page) | VTD_PTE_RW;
+ }
+ root = (uint64_t *)(root[offset] & VTD_PTE_ADDR);
+ }
+
+ offset = PGDIR_OFFSET(iova, level);
+ root[offset] = pa | VTD_PTE_RW;
+ if (level != 1)
+ /* This is huge page */
+ root[offset] |= VTD_PTE_HUGE;
+}
+
+#define VTD_FETCH_VIRT_ADDR(x) \
+ ((void *)(((uint64_t)phys_to_virt(x)) >> PAGE_SHIFT))
+
+/**
+ * vtd_map_range: setup IO address mapping for specific memory range
+ *
+ * @sid: source ID of the device to setup
+ * @iova: start IO virtual address
+ * @pa: start physical address
+ * @size: size of the mapping area
+ */
+void vtd_map_range(uint16_t sid, iova_t iova, phys_addr_t pa, size_t size)
+{
+ uint8_t bus_n, devfn;
+ void *slptptr;
+ vtd_ce_t *ce;
+ vtd_re_t *re = phys_to_virt(vtd_root_table());
+
+ assert(IS_ALIGNED(iova, SZ_4K));
+ assert(IS_ALIGNED(pa, SZ_4K));
+ assert(IS_ALIGNED(size, SZ_4K));
+
+ bus_n = PCI_BDF_GET_BUS(sid);
+ devfn = PCI_BDF_GET_DEVFN(sid);
+
+ /* Point to the correct root entry */
+ re += bus_n;
+
+ if (!re->present) {
+ ce = alloc_page();
+ memset(ce, 0, PAGE_SIZE);
+ memset(re, 0, sizeof(*re));
+ re->context_table_p = virt_to_phys(ce) >> PAGE_SHIFT;
+ re->present = 1;
+ printf("allocated vt-d root entry for PCI bus %d\n",
+ bus_n);
+ } else
+ ce = VTD_FETCH_VIRT_ADDR(re->context_table_p);
+
+ /* Point to the correct context entry */
+ ce += devfn;
+
+ if (!ce->present) {
+ slptptr = alloc_page();
+ memset(slptptr, 0, PAGE_SIZE);
+ memset(ce, 0, sizeof(*ce));
+ /* To make it simple, domain ID is the same as SID */
+ ce->domain_id = sid;
+ /* We only test 39 bits width case (3-level paging) */
+ ce->addr_width = VTD_CE_AW_39BIT;
+ ce->slptptr = virt_to_phys(slptptr) >> PAGE_SHIFT;
+ ce->trans_type = VTD_CONTEXT_TT_MULTI_LEVEL;
+ ce->present = 1;
+ /* No error reporting yet */
+ ce->disable_fault_report = 1;
+ printf("allocated vt-d context entry for devfn 0x%x\n",
+ devfn);
+ } else
+ slptptr = VTD_FETCH_VIRT_ADDR(ce->slptptr);
+
+ while (size) {
+ /* TODO: currently we only map 4K pages (level = 1) */
+ printf("map 4K page IOVA 0x%lx to 0x%lx (sid=0x%04x)\n",
+ iova, pa, sid);
+ vtd_install_pte(slptptr, iova, pa, 1);
+ size -= PAGE_SIZE;
+ iova += PAGE_SIZE;
+ pa += PAGE_SIZE;
+ }
+}
+
void vtd_init(void)
{
setup_vm();
diff --git a/lib/x86/intel-iommu.h b/lib/x86/intel-iommu.h
index fae9ae5..1352639 100644
--- a/lib/x86/intel-iommu.h
+++ b/lib/x86/intel-iommu.h
@@ -20,6 +20,7 @@
#include "isr.h"
#include "smp.h"
#include "desc.h"
+#include "pci.h"
#include "asm/io.h"
#define Q35_HOST_BRIDGE_IOMMU_ADDR 0xfed90000ULL
@@ -91,6 +92,27 @@
#define VTD_GCMD_ONE_SHOT_BITS (VTD_GCMD_IR_TABLE | VTD_GCMD_WBF | \
VTD_GCMD_SFL | VTD_GCMD_ROOT)
+/* Supported Adjusted Guest Address Widths */
+#define VTD_CAP_SAGAW_SHIFT 8
+/* 39-bit AGAW, 3-level page-table */
+#define VTD_CAP_SAGAW_39bit (0x2ULL << VTD_CAP_SAGAW_SHIFT)
+/* 48-bit AGAW, 4-level page-table */
+#define VTD_CAP_SAGAW_48bit (0x4ULL << VTD_CAP_SAGAW_SHIFT)
+#define VTD_CAP_SAGAW VTD_CAP_SAGAW_39bit
+
+/* Both 1G/2M huge pages */
+#define VTD_CAP_SLLPS ((1ULL << 34) | (1ULL << 35))
+
+#define VTD_CONTEXT_TT_MULTI_LEVEL 0
+#define VTD_CONTEXT_TT_DEV_IOTLB 1
+#define VTD_CONTEXT_TT_PASS_THROUGH 2
+
+#define VTD_PTE_R (1 << 0)
+#define VTD_PTE_W (1 << 1)
+#define VTD_PTE_RW (VTD_PTE_R | VTD_PTE_W)
+#define VTD_PTE_ADDR GENMASK_ULL(51, 12)
+#define VTD_PTE_HUGE (1 << 7)
+
#define vtd_reg(reg) ((volatile void *)(Q35_HOST_BRIDGE_IOMMU_ADDR + reg))
static inline void vtd_writel(unsigned int reg, uint32_t value)
@@ -114,5 +136,6 @@ static inline uint64_t vtd_readq(unsigned int reg)
}
void vtd_init(void);
+void vtd_map_range(uint16_t sid, phys_addr_t iova, phys_addr_t pa, size_t size);
#endif
diff --git a/x86/Makefile.common b/x86/Makefile.common
index 356d879..1dad18b 100644
--- a/x86/Makefile.common
+++ b/x86/Makefile.common
@@ -3,6 +3,7 @@
all: test_cases
cflatobjs += lib/pci.o
+cflatobjs += lib/pci-edu.o
cflatobjs += lib/x86/io.o
cflatobjs += lib/x86/smp.o
cflatobjs += lib/x86/vm.o
diff --git a/x86/intel-iommu.c b/x86/intel-iommu.c
index f247913..21fd57f 100644
--- a/x86/intel-iommu.c
+++ b/x86/intel-iommu.c
@@ -11,9 +11,50 @@
*/
#include "intel-iommu.h"
+#include "pci-edu.h"
+
+#define VTD_TEST_DMAR_4B ("DMAR 4B memcpy test")
+
+void vtd_test_dmar(struct pci_edu_dev *dev)
+{
+ void *page = alloc_page();
+
+#define DMA_TEST_WORD (0x12345678)
+ /* Modify the first 4 bytes of the page */
+ *(uint32_t *)page = DMA_TEST_WORD;
+
+ /*
+ * Map the newly allocated page into IOVA address 0 (size 4K)
+ * of the device address space. Root entry and context entry
+ * will be automatically created when needed.
+ */
+ vtd_map_range(dev->pci_dev.bdf, 0, virt_to_phys(page), PAGE_SIZE);
+
+ /*
+ * DMA the first 4 bytes of the page to EDU device buffer
+ * offset 0.
+ */
+ edu_dma(dev, 0, 4, 0, false);
+
+ /*
+ * DMA the first 4 bytes of EDU device buffer into the page
+ * with offset 4 (so it'll be using 4-7 bytes).
+ */
+ edu_dma(dev, 4, 4, 0, true);
+
+ /*
+ * Check data match between 0-3 bytes and 4-7 bytes of the
+ * page.
+ */
+ report(VTD_TEST_DMAR_4B, *((uint32_t *)page + 1) == DMA_TEST_WORD);
+
+ free_page(page);
+}
int main(int argc, char *argv[])
{
+ struct pci_edu_dev dev;
+
vtd_init();
report("fault status check", vtd_readl(DMAR_FSTS_REG) == 0);
@@ -22,6 +63,16 @@ int main(int argc, char *argv[])
report("IR table setup", vtd_readl(DMAR_GSTS_REG) & VTD_GCMD_IR_TABLE);
report("DMAR enablement", vtd_readl(DMAR_GSTS_REG) & VTD_GCMD_DMAR);
report("IR enablement", vtd_readl(DMAR_GSTS_REG) & VTD_GCMD_IR);
+ report("DMAR support 39 bits address width",
+ vtd_readq(DMAR_CAP_REG) & VTD_CAP_SAGAW);
+ report("DMAR support huge pages", vtd_readq(DMAR_CAP_REG) & VTD_CAP_SLLPS);
+
+ if (!edu_init(&dev)) {
+ printf("Please specify \"-device edu\" to do "
+ "further IOMMU tests.\n");
+ report_skip(VTD_TEST_DMAR_4B);
+ } else
+ vtd_test_dmar(&dev);
return report_summary();
}
--
2.7.4
^ permalink raw reply related [flat|nested] 29+ messages in thread* Re: [PATCH v3 11/13] x86: intel-iommu: add dmar test
2016-11-14 22:28 ` [PATCH v3 11/13] x86: intel-iommu: add dmar test Peter Xu
@ 2016-11-15 9:08 ` Andrew Jones
2016-11-15 17:01 ` Peter Xu
0 siblings, 1 reply; 29+ messages in thread
From: Andrew Jones @ 2016-11-15 9:08 UTC (permalink / raw)
To: Peter Xu; +Cc: kvm, agordeev, jan.kiszka, rkrcmar, pbonzini
On Mon, Nov 14, 2016 at 05:28:09PM -0500, Peter Xu wrote:
> DMAR test is based on QEMU edu device. A 4B DMA memory copy is carried
> out as the simplest DMAR test.
>
> Signed-off-by: Peter Xu <peterx@redhat.com>
> ---
> lib/pci.h | 5 ++
> lib/x86/intel-iommu.c | 131 ++++++++++++++++++++++++++++++++++++++++++++++++++
> lib/x86/intel-iommu.h | 23 +++++++++
> x86/Makefile.common | 1 +
> x86/intel-iommu.c | 51 ++++++++++++++++++++
> 5 files changed, 211 insertions(+)
>
> diff --git a/lib/pci.h b/lib/pci.h
> index 1e7be27..a2002ad 100644
> --- a/lib/pci.h
> +++ b/lib/pci.h
> @@ -18,6 +18,9 @@ enum {
> #define PCI_BAR_NUM 6
> #define PCI_DEVFN_MAX 256
>
> +#define PCI_BDF_GET_DEVFN(x) ((x) & 0xff)
> +#define PCI_BDF_GET_BUS(x) (((x) >> 8) & 0xff)
> +
> struct pci_dev {
> uint16_t bdf;
> phys_addr_t bar[PCI_BAR_NUM];
> @@ -28,6 +31,8 @@ void pci_scan_bars(struct pci_dev *dev);
> void pci_cmd_set_clr(struct pci_dev *dev, uint16_t set, uint16_t clr);
> void pci_enable_defaults(struct pci_dev *dev);
>
> +typedef phys_addr_t iova_t;
> +
> extern bool pci_probe(void);
> extern void pci_print(void);
> extern bool pci_dev_exists(pcidevaddr_t dev);
> diff --git a/lib/x86/intel-iommu.c b/lib/x86/intel-iommu.c
> index 9890f34..c1b6164 100644
> --- a/lib/x86/intel-iommu.c
> +++ b/lib/x86/intel-iommu.c
> @@ -11,6 +11,42 @@
> */
>
> #include "intel-iommu.h"
> +#include "libcflat.h"
> +
> +/*
> + * VT-d in QEMU currently only support 39 bits address width, which is
> + * 3-level translation.
> + */
> +#define VTD_PAGE_LEVEL 3
> +#define VTD_CE_AW_39BIT 0x1
> +
> +typedef uint64_t vtd_pte_t;
> +
> +struct vtd_root_entry {
> + /* Quad 1 */
> + uint64_t present:1;
> + uint64_t __reserved:11;
> + uint64_t context_table_p:52;
> + /* Quad 2 */
> + uint64_t __reserved_2;
> +} __attribute__ ((packed));
> +typedef struct vtd_root_entry vtd_re_t;
> +
> +struct vtd_context_entry {
> + /* Quad 1 */
> + uint64_t present:1;
> + uint64_t disable_fault_report:1;
> + uint64_t trans_type:2;
> + uint64_t __reserved:8;
> + uint64_t slptptr:52;
> + /* Quad 2 */
> + uint64_t addr_width:3;
> + uint64_t __ignore:4;
> + uint64_t __reserved_2:1;
> + uint64_t domain_id:16;
> + uint64_t __reserved_3:40;
> +} __attribute__ ((packed));
> +typedef struct vtd_context_entry vtd_ce_t;
>
> #define VTD_RTA_MASK (PAGE_MASK)
> #define VTD_IRTA_MASK (PAGE_MASK)
> @@ -74,6 +110,101 @@ static void vtd_setup_ir_table(void)
> printf("IR table address: 0x%016lx\n", vtd_ir_table());
> }
>
> +static void vtd_install_pte(vtd_pte_t *root, iova_t iova,
> + phys_addr_t pa, int level_target)
> +{
> + int level;
> + unsigned int offset;
> + void *page;
> +
> + for (level = VTD_PAGE_LEVEL; level > level_target; level--) {
> + offset = PGDIR_OFFSET(iova, level);
> + if (!(root[offset] & VTD_PTE_RW)) {
> + page = alloc_page();
> + memset(page, 0, PAGE_SIZE);
> + root[offset] = virt_to_phys(page) | VTD_PTE_RW;
> + }
> + root = (uint64_t *)(root[offset] & VTD_PTE_ADDR);
> + }
> +
> + offset = PGDIR_OFFSET(iova, level);
> + root[offset] = pa | VTD_PTE_RW;
> + if (level != 1)
> + /* This is huge page */
> + root[offset] |= VTD_PTE_HUGE;
I'd use {} on this if because you have two lines (even though one is
a comment)
> +}
> +
> +#define VTD_FETCH_VIRT_ADDR(x) \
> + ((void *)(((uint64_t)phys_to_virt(x)) >> PAGE_SHIFT))
> +
> +/**
> + * vtd_map_range: setup IO address mapping for specific memory range
> + *
> + * @sid: source ID of the device to setup
> + * @iova: start IO virtual address
> + * @pa: start physical address
> + * @size: size of the mapping area
> + */
> +void vtd_map_range(uint16_t sid, iova_t iova, phys_addr_t pa, size_t size)
> +{
> + uint8_t bus_n, devfn;
> + void *slptptr;
> + vtd_ce_t *ce;
> + vtd_re_t *re = phys_to_virt(vtd_root_table());
> +
> + assert(IS_ALIGNED(iova, SZ_4K));
> + assert(IS_ALIGNED(pa, SZ_4K));
> + assert(IS_ALIGNED(size, SZ_4K));
> +
> + bus_n = PCI_BDF_GET_BUS(sid);
> + devfn = PCI_BDF_GET_DEVFN(sid);
> +
> + /* Point to the correct root entry */
> + re += bus_n;
> +
> + if (!re->present) {
> + ce = alloc_page();
> + memset(ce, 0, PAGE_SIZE);
> + memset(re, 0, sizeof(*re));
> + re->context_table_p = virt_to_phys(ce) >> PAGE_SHIFT;
> + re->present = 1;
> + printf("allocated vt-d root entry for PCI bus %d\n",
> + bus_n);
> + } else
> + ce = VTD_FETCH_VIRT_ADDR(re->context_table_p);
> +
> + /* Point to the correct context entry */
> + ce += devfn;
> +
> + if (!ce->present) {
> + slptptr = alloc_page();
> + memset(slptptr, 0, PAGE_SIZE);
> + memset(ce, 0, sizeof(*ce));
> + /* To make it simple, domain ID is the same as SID */
> + ce->domain_id = sid;
> + /* We only test 39 bits width case (3-level paging) */
> + ce->addr_width = VTD_CE_AW_39BIT;
> + ce->slptptr = virt_to_phys(slptptr) >> PAGE_SHIFT;
> + ce->trans_type = VTD_CONTEXT_TT_MULTI_LEVEL;
> + ce->present = 1;
> + /* No error reporting yet */
> + ce->disable_fault_report = 1;
> + printf("allocated vt-d context entry for devfn 0x%x\n",
> + devfn);
> + } else
> + slptptr = VTD_FETCH_VIRT_ADDR(ce->slptptr);
> +
> + while (size) {
> + /* TODO: currently we only map 4K pages (level = 1) */
> + printf("map 4K page IOVA 0x%lx to 0x%lx (sid=0x%04x)\n",
> + iova, pa, sid);
> + vtd_install_pte(slptptr, iova, pa, 1);
> + size -= PAGE_SIZE;
> + iova += PAGE_SIZE;
> + pa += PAGE_SIZE;
> + }
> +}
> +
> void vtd_init(void)
> {
> setup_vm();
> diff --git a/lib/x86/intel-iommu.h b/lib/x86/intel-iommu.h
> index fae9ae5..1352639 100644
> --- a/lib/x86/intel-iommu.h
> +++ b/lib/x86/intel-iommu.h
> @@ -20,6 +20,7 @@
> #include "isr.h"
> #include "smp.h"
> #include "desc.h"
> +#include "pci.h"
> #include "asm/io.h"
>
> #define Q35_HOST_BRIDGE_IOMMU_ADDR 0xfed90000ULL
> @@ -91,6 +92,27 @@
> #define VTD_GCMD_ONE_SHOT_BITS (VTD_GCMD_IR_TABLE | VTD_GCMD_WBF | \
> VTD_GCMD_SFL | VTD_GCMD_ROOT)
>
> +/* Supported Adjusted Guest Address Widths */
> +#define VTD_CAP_SAGAW_SHIFT 8
> +/* 39-bit AGAW, 3-level page-table */
> +#define VTD_CAP_SAGAW_39bit (0x2ULL << VTD_CAP_SAGAW_SHIFT)
> +/* 48-bit AGAW, 4-level page-table */
> +#define VTD_CAP_SAGAW_48bit (0x4ULL << VTD_CAP_SAGAW_SHIFT)
> +#define VTD_CAP_SAGAW VTD_CAP_SAGAW_39bit
> +
> +/* Both 1G/2M huge pages */
> +#define VTD_CAP_SLLPS ((1ULL << 34) | (1ULL << 35))
> +
> +#define VTD_CONTEXT_TT_MULTI_LEVEL 0
> +#define VTD_CONTEXT_TT_DEV_IOTLB 1
> +#define VTD_CONTEXT_TT_PASS_THROUGH 2
> +
> +#define VTD_PTE_R (1 << 0)
> +#define VTD_PTE_W (1 << 1)
> +#define VTD_PTE_RW (VTD_PTE_R | VTD_PTE_W)
> +#define VTD_PTE_ADDR GENMASK_ULL(51, 12)
> +#define VTD_PTE_HUGE (1 << 7)
> +
> #define vtd_reg(reg) ((volatile void *)(Q35_HOST_BRIDGE_IOMMU_ADDR + reg))
>
> static inline void vtd_writel(unsigned int reg, uint32_t value)
> @@ -114,5 +136,6 @@ static inline uint64_t vtd_readq(unsigned int reg)
> }
>
> void vtd_init(void);
> +void vtd_map_range(uint16_t sid, phys_addr_t iova, phys_addr_t pa, size_t size);
>
> #endif
> diff --git a/x86/Makefile.common b/x86/Makefile.common
> index 356d879..1dad18b 100644
> --- a/x86/Makefile.common
> +++ b/x86/Makefile.common
> @@ -3,6 +3,7 @@
> all: test_cases
>
> cflatobjs += lib/pci.o
> +cflatobjs += lib/pci-edu.o
> cflatobjs += lib/x86/io.o
> cflatobjs += lib/x86/smp.o
> cflatobjs += lib/x86/vm.o
> diff --git a/x86/intel-iommu.c b/x86/intel-iommu.c
> index f247913..21fd57f 100644
> --- a/x86/intel-iommu.c
> +++ b/x86/intel-iommu.c
> @@ -11,9 +11,50 @@
> */
>
> #include "intel-iommu.h"
> +#include "pci-edu.h"
> +
> +#define VTD_TEST_DMAR_4B ("DMAR 4B memcpy test")
> +
> +void vtd_test_dmar(struct pci_edu_dev *dev)
> +{
> + void *page = alloc_page();
> +
> +#define DMA_TEST_WORD (0x12345678)
> + /* Modify the first 4 bytes of the page */
> + *(uint32_t *)page = DMA_TEST_WORD;
> +
> + /*
> + * Map the newly allocated page into IOVA address 0 (size 4K)
> + * of the device address space. Root entry and context entry
> + * will be automatically created when needed.
> + */
> + vtd_map_range(dev->pci_dev.bdf, 0, virt_to_phys(page), PAGE_SIZE);
> +
> + /*
> + * DMA the first 4 bytes of the page to EDU device buffer
> + * offset 0.
> + */
> + edu_dma(dev, 0, 4, 0, false);
> +
> + /*
> + * DMA the first 4 bytes of EDU device buffer into the page
> + * with offset 4 (so it'll be using 4-7 bytes).
> + */
> + edu_dma(dev, 4, 4, 0, true);
> +
> + /*
> + * Check data match between 0-3 bytes and 4-7 bytes of the
> + * page.
> + */
> + report(VTD_TEST_DMAR_4B, *((uint32_t *)page + 1) == DMA_TEST_WORD);
> +
> + free_page(page);
> +}
>
> int main(int argc, char *argv[])
> {
> + struct pci_edu_dev dev;
> +
> vtd_init();
>
> report("fault status check", vtd_readl(DMAR_FSTS_REG) == 0);
> @@ -22,6 +63,16 @@ int main(int argc, char *argv[])
> report("IR table setup", vtd_readl(DMAR_GSTS_REG) & VTD_GCMD_IR_TABLE);
> report("DMAR enablement", vtd_readl(DMAR_GSTS_REG) & VTD_GCMD_DMAR);
> report("IR enablement", vtd_readl(DMAR_GSTS_REG) & VTD_GCMD_IR);
> + report("DMAR support 39 bits address width",
> + vtd_readq(DMAR_CAP_REG) & VTD_CAP_SAGAW);
> + report("DMAR support huge pages", vtd_readq(DMAR_CAP_REG) & VTD_CAP_SLLPS);
> +
> + if (!edu_init(&dev)) {
> + printf("Please specify \"-device edu\" to do "
> + "further IOMMU tests.\n");
> + report_skip(VTD_TEST_DMAR_4B);
> + } else
> + vtd_test_dmar(&dev);
>
> return report_summary();
> }
> --
> 2.7.4
>
I defer the real review to Paolo and Radim, but I have no complaints.
Thanks,
drew
^ permalink raw reply [flat|nested] 29+ messages in thread* Re: [PATCH v3 11/13] x86: intel-iommu: add dmar test
2016-11-15 9:08 ` Andrew Jones
@ 2016-11-15 17:01 ` Peter Xu
0 siblings, 0 replies; 29+ messages in thread
From: Peter Xu @ 2016-11-15 17:01 UTC (permalink / raw)
To: Andrew Jones; +Cc: kvm, agordeev, jan.kiszka, rkrcmar, pbonzini
On Tue, Nov 15, 2016 at 10:08:41AM +0100, Andrew Jones wrote:
[...]
> > +static void vtd_install_pte(vtd_pte_t *root, iova_t iova,
> > + phys_addr_t pa, int level_target)
> > +{
> > + int level;
> > + unsigned int offset;
> > + void *page;
> > +
> > + for (level = VTD_PAGE_LEVEL; level > level_target; level--) {
> > + offset = PGDIR_OFFSET(iova, level);
> > + if (!(root[offset] & VTD_PTE_RW)) {
> > + page = alloc_page();
> > + memset(page, 0, PAGE_SIZE);
> > + root[offset] = virt_to_phys(page) | VTD_PTE_RW;
> > + }
> > + root = (uint64_t *)(root[offset] & VTD_PTE_ADDR);
> > + }
> > +
> > + offset = PGDIR_OFFSET(iova, level);
> > + root[offset] = pa | VTD_PTE_RW;
> > + if (level != 1)
> > + /* This is huge page */
> > + root[offset] |= VTD_PTE_HUGE;
>
> I'd use {} on this if because you have two lines (even though one is
> a comment)
Hmm obviously I missed this one. Thanks,
-- peterx
^ permalink raw reply [flat|nested] 29+ messages in thread
* [PATCH v3 12/13] pci: add msi support for 32/64bit address
2016-11-14 22:27 [PATCH v3 00/13] VT-d unit test Peter Xu
` (10 preceding siblings ...)
2016-11-14 22:28 ` [PATCH v3 11/13] x86: intel-iommu: add dmar test Peter Xu
@ 2016-11-14 22:28 ` Peter Xu
2016-11-15 9:10 ` Andrew Jones
2016-11-14 22:28 ` [PATCH v3 13/13] x86: intel-iommu: add IR MSI test Peter Xu
` (2 subsequent siblings)
14 siblings, 1 reply; 29+ messages in thread
From: Peter Xu @ 2016-11-14 22:28 UTC (permalink / raw)
To: kvm; +Cc: drjones, agordeev, jan.kiszka, rkrcmar, pbonzini, peterx
pci_cap_walk() is provided to allow walk through all the capability bits
for a specific PCI device. If a cap handler is provided, it'll be
triggered if the cap is detected along the cap walk.
MSI cap handler is the first one supported. We can add more cap handler
in the future.
Meanwhile, pci_setup_msi() API is provided to support basic 32/64 bit
address MSI setup.
Reviewed-by: Andrew Jones <drjones@redhat.com>
Signed-off-by: Peter Xu <peterx@redhat.com>
---
lib/pci.c | 65 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
lib/pci.h | 3 +++
2 files changed, 68 insertions(+)
diff --git a/lib/pci.c b/lib/pci.c
index 1c59e13..4eb5574 100644
--- a/lib/pci.c
+++ b/lib/pci.c
@@ -7,6 +7,70 @@
#include "pci.h"
#include "asm/pci.h"
+typedef void (*pci_cap_handler)(struct pci_dev *dev, int cap_offset);
+
+static void pci_cap_msi_handler(struct pci_dev *dev, int cap_offset)
+{
+ printf("Detected MSI for device 0x%x offset 0x%x\n",
+ dev->bdf, cap_offset);
+ dev->msi_offset = cap_offset;
+}
+
+static pci_cap_handler cap_handlers[PCI_CAP_ID_MAX + 1] = {
+ [PCI_CAP_ID_MSI] = pci_cap_msi_handler,
+};
+
+void pci_cap_walk(struct pci_dev *dev)
+{
+ uint8_t cap_offset;
+ uint8_t cap_id;
+
+ cap_offset = pci_config_readb(dev->bdf, PCI_CAPABILITY_LIST);
+ while (cap_offset) {
+ cap_id = pci_config_readb(dev->bdf, cap_offset);
+ printf("PCI detected cap 0x%x\n", cap_id);
+ if (cap_handlers[cap_id])
+ cap_handlers[cap_id](dev, cap_offset);
+ cap_offset = pci_config_readb(dev->bdf, cap_offset + 1);
+ }
+}
+
+bool pci_setup_msi(struct pci_dev *dev, uint64_t msi_addr, uint32_t msi_data)
+{
+ uint16_t msi_control;
+ uint16_t offset;
+ pcidevaddr_t addr;
+
+ assert(dev);
+
+ if (!dev->msi_offset) {
+ printf("MSI: dev 0x%x does not support MSI.\n", dev->bdf);
+ return false;
+ }
+
+ addr = dev->bdf;
+ offset = dev->msi_offset;
+ msi_control = pci_config_readw(addr, offset + PCI_MSI_FLAGS);
+ pci_config_writel(addr, offset + PCI_MSI_ADDRESS_LO,
+ msi_addr & 0xffffffff);
+
+ if (msi_control & PCI_MSI_FLAGS_64BIT) {
+ pci_config_writel(addr, offset + PCI_MSI_ADDRESS_HI,
+ (uint32_t)(msi_addr >> 32));
+ pci_config_writel(addr, offset + PCI_MSI_DATA_64, msi_data);
+ printf("MSI: dev 0x%x init 64bit address: ", addr);
+ } else {
+ pci_config_writel(addr, offset + PCI_MSI_DATA_32, msi_data);
+ printf("MSI: dev 0x%x init 32bit address: ", addr);
+ }
+ printf("addr=0x%lx, data=0x%x\n", msi_addr, msi_data);
+
+ msi_control |= PCI_MSI_FLAGS_ENABLE;
+ pci_config_writew(addr, offset + PCI_MSI_FLAGS, msi_control);
+
+ return true;
+}
+
void pci_cmd_set_clr(struct pci_dev *dev, uint16_t set, uint16_t clr)
{
uint16_t val = pci_config_readw(dev->bdf, PCI_COMMAND);
@@ -260,4 +324,5 @@ void pci_enable_defaults(struct pci_dev *dev)
pci_scan_bars(dev);
/* Enable device DMA operations */
pci_cmd_set_clr(dev, PCI_COMMAND_MASTER, 0);
+ pci_cap_walk(dev);
}
diff --git a/lib/pci.h b/lib/pci.h
index a2002ad..221db2f 100644
--- a/lib/pci.h
+++ b/lib/pci.h
@@ -23,13 +23,16 @@ enum {
struct pci_dev {
uint16_t bdf;
+ uint16_t msi_offset;
phys_addr_t bar[PCI_BAR_NUM];
};
void pci_dev_init(struct pci_dev *dev, pcidevaddr_t bdf);
void pci_scan_bars(struct pci_dev *dev);
void pci_cmd_set_clr(struct pci_dev *dev, uint16_t set, uint16_t clr);
+void pci_cap_walk(struct pci_dev *dev);
void pci_enable_defaults(struct pci_dev *dev);
+bool pci_setup_msi(struct pci_dev *dev, uint64_t msi_addr, uint32_t msi_data);
typedef phys_addr_t iova_t;
--
2.7.4
^ permalink raw reply related [flat|nested] 29+ messages in thread* Re: [PATCH v3 12/13] pci: add msi support for 32/64bit address
2016-11-14 22:28 ` [PATCH v3 12/13] pci: add msi support for 32/64bit address Peter Xu
@ 2016-11-15 9:10 ` Andrew Jones
0 siblings, 0 replies; 29+ messages in thread
From: Andrew Jones @ 2016-11-15 9:10 UTC (permalink / raw)
To: Peter Xu; +Cc: kvm, agordeev, jan.kiszka, rkrcmar, pbonzini
On Mon, Nov 14, 2016 at 05:28:10PM -0500, Peter Xu wrote:
> pci_cap_walk() is provided to allow walk through all the capability bits
> for a specific PCI device. If a cap handler is provided, it'll be
> triggered if the cap is detected along the cap walk.
>
> MSI cap handler is the first one supported. We can add more cap handler
> in the future.
>
> Meanwhile, pci_setup_msi() API is provided to support basic 32/64 bit
> address MSI setup.
>
> Reviewed-by: Andrew Jones <drjones@redhat.com>
> Signed-off-by: Peter Xu <peterx@redhat.com>
> ---
> lib/pci.c | 65 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> lib/pci.h | 3 +++
> 2 files changed, 68 insertions(+)
>
> diff --git a/lib/pci.c b/lib/pci.c
> index 1c59e13..4eb5574 100644
> --- a/lib/pci.c
> +++ b/lib/pci.c
> @@ -7,6 +7,70 @@
> #include "pci.h"
> #include "asm/pci.h"
>
> +typedef void (*pci_cap_handler)(struct pci_dev *dev, int cap_offset);
> +
> +static void pci_cap_msi_handler(struct pci_dev *dev, int cap_offset)
> +{
> + printf("Detected MSI for device 0x%x offset 0x%x\n",
> + dev->bdf, cap_offset);
> + dev->msi_offset = cap_offset;
> +}
> +
> +static pci_cap_handler cap_handlers[PCI_CAP_ID_MAX + 1] = {
> + [PCI_CAP_ID_MSI] = pci_cap_msi_handler,
> +};
> +
> +void pci_cap_walk(struct pci_dev *dev)
> +{
> + uint8_t cap_offset;
> + uint8_t cap_id;
> +
> + cap_offset = pci_config_readb(dev->bdf, PCI_CAPABILITY_LIST);
> + while (cap_offset) {
> + cap_id = pci_config_readb(dev->bdf, cap_offset);
> + printf("PCI detected cap 0x%x\n", cap_id);
> + if (cap_handlers[cap_id])
> + cap_handlers[cap_id](dev, cap_offset);
> + cap_offset = pci_config_readb(dev->bdf, cap_offset + 1);
> + }
> +}
> +
> +bool pci_setup_msi(struct pci_dev *dev, uint64_t msi_addr, uint32_t msi_data)
> +{
> + uint16_t msi_control;
> + uint16_t offset;
> + pcidevaddr_t addr;
> +
> + assert(dev);
> +
> + if (!dev->msi_offset) {
> + printf("MSI: dev 0x%x does not support MSI.\n", dev->bdf);
> + return false;
> + }
> +
> + addr = dev->bdf;
> + offset = dev->msi_offset;
> + msi_control = pci_config_readw(addr, offset + PCI_MSI_FLAGS);
> + pci_config_writel(addr, offset + PCI_MSI_ADDRESS_LO,
> + msi_addr & 0xffffffff);
> +
> + if (msi_control & PCI_MSI_FLAGS_64BIT) {
> + pci_config_writel(addr, offset + PCI_MSI_ADDRESS_HI,
> + (uint32_t)(msi_addr >> 32));
> + pci_config_writel(addr, offset + PCI_MSI_DATA_64, msi_data);
> + printf("MSI: dev 0x%x init 64bit address: ", addr);
> + } else {
> + pci_config_writel(addr, offset + PCI_MSI_DATA_32, msi_data);
> + printf("MSI: dev 0x%x init 32bit address: ", addr);
> + }
> + printf("addr=0x%lx, data=0x%x\n", msi_addr, msi_data);
> +
> + msi_control |= PCI_MSI_FLAGS_ENABLE;
> + pci_config_writew(addr, offset + PCI_MSI_FLAGS, msi_control);
> +
> + return true;
> +}
> +
> void pci_cmd_set_clr(struct pci_dev *dev, uint16_t set, uint16_t clr)
> {
> uint16_t val = pci_config_readw(dev->bdf, PCI_COMMAND);
> @@ -260,4 +324,5 @@ void pci_enable_defaults(struct pci_dev *dev)
> pci_scan_bars(dev);
> /* Enable device DMA operations */
> pci_cmd_set_clr(dev, PCI_COMMAND_MASTER, 0);
> + pci_cap_walk(dev);
> }
> diff --git a/lib/pci.h b/lib/pci.h
> index a2002ad..221db2f 100644
> --- a/lib/pci.h
> +++ b/lib/pci.h
> @@ -23,13 +23,16 @@ enum {
>
> struct pci_dev {
> uint16_t bdf;
> + uint16_t msi_offset;
> phys_addr_t bar[PCI_BAR_NUM];
> };
>
> void pci_dev_init(struct pci_dev *dev, pcidevaddr_t bdf);
> void pci_scan_bars(struct pci_dev *dev);
> void pci_cmd_set_clr(struct pci_dev *dev, uint16_t set, uint16_t clr);
> +void pci_cap_walk(struct pci_dev *dev);
> void pci_enable_defaults(struct pci_dev *dev);
> +bool pci_setup_msi(struct pci_dev *dev, uint64_t msi_addr, uint32_t msi_data);
missing more pointless externs here
>
> typedef phys_addr_t iova_t;
>
> --
> 2.7.4
Thanks,
drew
^ permalink raw reply [flat|nested] 29+ messages in thread
* [PATCH v3 13/13] x86: intel-iommu: add IR MSI test
2016-11-14 22:27 [PATCH v3 00/13] VT-d unit test Peter Xu
` (11 preceding siblings ...)
2016-11-14 22:28 ` [PATCH v3 12/13] pci: add msi support for 32/64bit address Peter Xu
@ 2016-11-14 22:28 ` Peter Xu
2016-11-15 9:13 ` Andrew Jones
2016-11-14 22:36 ` THIS SERIES SHOULD BE: "[PATCH v4 00/13] VT-d unit test" Peter Xu
2016-11-15 9:17 ` [PATCH v3 00/13] VT-d unit test Andrew Jones
14 siblings, 1 reply; 29+ messages in thread
From: Peter Xu @ 2016-11-14 22:28 UTC (permalink / raw)
To: kvm; +Cc: drjones, agordeev, jan.kiszka, rkrcmar, pbonzini, peterx
First of all, vtd_setup_msi() is provided. It setup IRTE entries,
meanwhile, setup PCI device MSI vectors corresponding to VT-d spec.
Meanwhile, IR MSI test is added to intel IOMMU unit test. The basic IR
test is carried out by a edu INTR raise request. When write to the intr
raise register, interrupt will be generated. Type of interrupt will
depend on the setup (either INTx or MSI).
Suggested-by: Andrew Jones <drjones@redhat.com>
Signed-off-by: Peter Xu <peterx@redhat.com>
---
lib/pci-edu.h | 1 +
lib/x86/intel-iommu.c | 94 +++++++++++++++++++++++++++++++++++++++++++++++++++
lib/x86/intel-iommu.h | 1 +
x86/intel-iommu.c | 44 +++++++++++++++++++++++-
4 files changed, 139 insertions(+), 1 deletion(-)
diff --git a/lib/pci-edu.h b/lib/pci-edu.h
index b3a6e2b..a87219a 100644
--- a/lib/pci-edu.h
+++ b/lib/pci-edu.h
@@ -32,6 +32,7 @@
#define EDU_REG_ALIVE 0x4
#define EDU_REG_FACTORIAL 0x8
#define EDU_REG_STATUS 0x20
+#define EDU_REG_INTR_RAISE 0x60
#define EDU_REG_DMA_SRC 0x80
#define EDU_REG_DMA_DST 0x88
#define EDU_REG_DMA_COUNT 0x90
diff --git a/lib/x86/intel-iommu.c b/lib/x86/intel-iommu.c
index c1b6164..9f1acea 100644
--- a/lib/x86/intel-iommu.c
+++ b/lib/x86/intel-iommu.c
@@ -12,6 +12,7 @@
#include "intel-iommu.h"
#include "libcflat.h"
+#include "pci.h"
/*
* VT-d in QEMU currently only support 39 bits address width, which is
@@ -48,6 +49,26 @@ struct vtd_context_entry {
} __attribute__ ((packed));
typedef struct vtd_context_entry vtd_ce_t;
+struct vtd_irte {
+ uint32_t present:1;
+ uint32_t fault_disable:1; /* Fault Processing Disable */
+ uint32_t dest_mode:1; /* Destination Mode */
+ uint32_t redir_hint:1; /* Redirection Hint */
+ uint32_t trigger_mode:1; /* Trigger Mode */
+ uint32_t delivery_mode:3; /* Delivery Mode */
+ uint32_t __avail:4; /* Available spaces for software */
+ uint32_t __reserved_0:3; /* Reserved 0 */
+ uint32_t irte_mode:1; /* IRTE Mode */
+ uint32_t vector:8; /* Interrupt Vector */
+ uint32_t __reserved_1:8; /* Reserved 1 */
+ uint32_t dest_id; /* Destination ID */
+ uint16_t source_id:16; /* Source-ID */
+ uint64_t sid_q:2; /* Source-ID Qualifier */
+ uint64_t sid_vtype:2; /* Source-ID Validation Type */
+ uint64_t __reserved_2:44; /* Reserved 2 */
+} __attribute__ ((packed));
+typedef struct vtd_irte vtd_irte_t;
+
#define VTD_RTA_MASK (PAGE_MASK)
#define VTD_IRTA_MASK (PAGE_MASK)
@@ -205,6 +226,79 @@ void vtd_map_range(uint16_t sid, iova_t iova, phys_addr_t pa, size_t size)
}
}
+static uint16_t vtd_intr_index_alloc(void)
+{
+ static int index_ctr = 0;
+ assert(index_ctr < 65535);
+ return index_ctr++;
+}
+
+static void vtd_setup_irte(struct pci_dev *dev, vtd_irte_t *irte,
+ int vector, int dest_id)
+{
+ assert(sizeof(vtd_irte_t) == 16);
+ memset(irte, 0, sizeof(*irte));
+ irte->fault_disable = 1;
+ irte->dest_mode = 0; /* physical */
+ irte->trigger_mode = 0; /* edge */
+ irte->delivery_mode = 0; /* fixed */
+ irte->irte_mode = 0; /* remapped */
+ irte->vector = vector;
+ irte->dest_id = dest_id;
+ irte->source_id = dev->bdf;
+ irte->sid_q = 0;
+ irte->sid_vtype = 1; /* full-sid verify */
+ irte->present = 1;
+}
+
+struct vtd_msi_addr {
+ uint32_t __dont_care:2;
+ uint32_t handle_15:1; /* handle[15] */
+ uint32_t shv:1;
+ uint32_t interrupt_format:1;
+ uint32_t handle_0_14:15; /* handle[0:14] */
+ uint32_t head:12; /* 0xfee */
+ uint32_t addr_hi; /* not used except with x2apic */
+} __attribute__ ((packed));
+typedef struct vtd_msi_addr vtd_msi_addr_t;
+
+struct vtd_msi_data {
+ uint16_t __reserved;
+ uint16_t subhandle;
+} __attribute__ ((packed));
+typedef struct vtd_msi_data vtd_msi_data_t;
+
+/**
+ * vtd_setup_msi - setup MSI message for a device
+ *
+ * @dev: PCI device to setup MSI
+ * @vector: interrupt vector
+ * @dest_id: destination processor
+ */
+bool vtd_setup_msi(struct pci_dev *dev, int vector, int dest_id)
+{
+ vtd_msi_data_t msi_data = {};
+ vtd_msi_addr_t msi_addr = {};
+ vtd_irte_t *irte = phys_to_virt(vtd_ir_table());
+ uint16_t index = vtd_intr_index_alloc();
+
+ assert(sizeof(vtd_msi_addr_t) == 8);
+ assert(sizeof(vtd_msi_data_t) == 4);
+
+ printf("INTR: setup IRTE index %d\n", index);
+ vtd_setup_irte(dev, irte + index, vector, dest_id);
+
+ msi_addr.handle_15 = index >> 15 & 1;
+ msi_addr.shv = 0;
+ msi_addr.interrupt_format = 1;
+ msi_addr.handle_0_14 = index & 0x7fff;
+ msi_addr.head = 0xfee;
+ msi_data.subhandle = 0;
+
+ return pci_setup_msi(dev, *(uint64_t *)&msi_addr,
+ *(uint32_t *)&msi_data);
+}
+
void vtd_init(void)
{
setup_vm();
diff --git a/lib/x86/intel-iommu.h b/lib/x86/intel-iommu.h
index 1352639..13feafa 100644
--- a/lib/x86/intel-iommu.h
+++ b/lib/x86/intel-iommu.h
@@ -137,5 +137,6 @@ static inline uint64_t vtd_readq(unsigned int reg)
void vtd_init(void);
void vtd_map_range(uint16_t sid, phys_addr_t iova, phys_addr_t pa, size_t size);
+bool vtd_setup_msi(struct pci_dev *dev, int vector, int dest_id);
#endif
diff --git a/x86/intel-iommu.c b/x86/intel-iommu.c
index 21fd57f..753f90e 100644
--- a/x86/intel-iommu.c
+++ b/x86/intel-iommu.c
@@ -12,8 +12,10 @@
#include "intel-iommu.h"
#include "pci-edu.h"
+#include "x86/apic.h"
#define VTD_TEST_DMAR_4B ("DMAR 4B memcpy test")
+#define VTD_TEST_IR_MSI ("IR MSI")
void vtd_test_dmar(struct pci_edu_dev *dev)
{
@@ -51,6 +53,43 @@ void vtd_test_dmar(struct pci_edu_dev *dev)
free_page(page);
}
+static volatile bool edu_intr_recved;
+
+static void edu_isr(isr_regs_t *regs)
+{
+ edu_intr_recved = true;
+ eoi();
+}
+
+static void vtd_test_ir(struct pci_edu_dev *dev)
+{
+#define VTD_TEST_VECTOR (0xee)
+ /*
+ * Setup EDU PCI device MSI, using interrupt remapping. By
+ * default, EDU device is using INTx.
+ */
+ if (!vtd_setup_msi(&dev->pci_dev, VTD_TEST_VECTOR, 0)) {
+ printf("edu device does not support MSI, skip test\n");
+ report_skip(VTD_TEST_IR_MSI);
+ return;
+ }
+
+ handle_irq(VTD_TEST_VECTOR, edu_isr);
+ irq_enable();
+
+ /* Manually trigger INTR */
+ edu_reg_writel(dev, EDU_REG_INTR_RAISE, 1);
+
+ while (!edu_intr_recved)
+ cpu_relax();
+
+ /* Clear INTR bits */
+ edu_reg_writel(dev, EDU_REG_INTR_RAISE, 0);
+
+ /* We are good as long as we reach here */
+ report(VTD_TEST_IR_MSI, true);
+}
+
int main(int argc, char *argv[])
{
struct pci_edu_dev dev;
@@ -71,8 +110,11 @@ int main(int argc, char *argv[])
printf("Please specify \"-device edu\" to do "
"further IOMMU tests.\n");
report_skip(VTD_TEST_DMAR_4B);
- } else
+ report_skip(VTD_TEST_IR_MSI);
+ } else {
vtd_test_dmar(&dev);
+ vtd_test_ir(&dev);
+ }
return report_summary();
}
--
2.7.4
^ permalink raw reply related [flat|nested] 29+ messages in thread* Re: [PATCH v3 13/13] x86: intel-iommu: add IR MSI test
2016-11-14 22:28 ` [PATCH v3 13/13] x86: intel-iommu: add IR MSI test Peter Xu
@ 2016-11-15 9:13 ` Andrew Jones
0 siblings, 0 replies; 29+ messages in thread
From: Andrew Jones @ 2016-11-15 9:13 UTC (permalink / raw)
To: Peter Xu; +Cc: kvm, agordeev, jan.kiszka, rkrcmar, pbonzini
On Mon, Nov 14, 2016 at 05:28:11PM -0500, Peter Xu wrote:
> First of all, vtd_setup_msi() is provided. It setup IRTE entries,
> meanwhile, setup PCI device MSI vectors corresponding to VT-d spec.
>
> Meanwhile, IR MSI test is added to intel IOMMU unit test. The basic IR
> test is carried out by a edu INTR raise request. When write to the intr
> raise register, interrupt will be generated. Type of interrupt will
> depend on the setup (either INTx or MSI).
>
> Suggested-by: Andrew Jones <drjones@redhat.com>
I don't recall what I suggested in this patch, but certainly the idea
for the test is all yours, so you can drop this tag.
> Signed-off-by: Peter Xu <peterx@redhat.com>
> ---
> lib/pci-edu.h | 1 +
> lib/x86/intel-iommu.c | 94 +++++++++++++++++++++++++++++++++++++++++++++++++++
> lib/x86/intel-iommu.h | 1 +
> x86/intel-iommu.c | 44 +++++++++++++++++++++++-
> 4 files changed, 139 insertions(+), 1 deletion(-)
>
> diff --git a/lib/pci-edu.h b/lib/pci-edu.h
> index b3a6e2b..a87219a 100644
> --- a/lib/pci-edu.h
> +++ b/lib/pci-edu.h
> @@ -32,6 +32,7 @@
> #define EDU_REG_ALIVE 0x4
> #define EDU_REG_FACTORIAL 0x8
> #define EDU_REG_STATUS 0x20
> +#define EDU_REG_INTR_RAISE 0x60
> #define EDU_REG_DMA_SRC 0x80
> #define EDU_REG_DMA_DST 0x88
> #define EDU_REG_DMA_COUNT 0x90
> diff --git a/lib/x86/intel-iommu.c b/lib/x86/intel-iommu.c
> index c1b6164..9f1acea 100644
> --- a/lib/x86/intel-iommu.c
> +++ b/lib/x86/intel-iommu.c
> @@ -12,6 +12,7 @@
>
> #include "intel-iommu.h"
> #include "libcflat.h"
> +#include "pci.h"
>
> /*
> * VT-d in QEMU currently only support 39 bits address width, which is
> @@ -48,6 +49,26 @@ struct vtd_context_entry {
> } __attribute__ ((packed));
> typedef struct vtd_context_entry vtd_ce_t;
>
> +struct vtd_irte {
> + uint32_t present:1;
> + uint32_t fault_disable:1; /* Fault Processing Disable */
> + uint32_t dest_mode:1; /* Destination Mode */
> + uint32_t redir_hint:1; /* Redirection Hint */
> + uint32_t trigger_mode:1; /* Trigger Mode */
> + uint32_t delivery_mode:3; /* Delivery Mode */
> + uint32_t __avail:4; /* Available spaces for software */
> + uint32_t __reserved_0:3; /* Reserved 0 */
> + uint32_t irte_mode:1; /* IRTE Mode */
> + uint32_t vector:8; /* Interrupt Vector */
> + uint32_t __reserved_1:8; /* Reserved 1 */
> + uint32_t dest_id; /* Destination ID */
> + uint16_t source_id:16; /* Source-ID */
> + uint64_t sid_q:2; /* Source-ID Qualifier */
> + uint64_t sid_vtype:2; /* Source-ID Validation Type */
> + uint64_t __reserved_2:44; /* Reserved 2 */
> +} __attribute__ ((packed));
> +typedef struct vtd_irte vtd_irte_t;
> +
> #define VTD_RTA_MASK (PAGE_MASK)
> #define VTD_IRTA_MASK (PAGE_MASK)
>
> @@ -205,6 +226,79 @@ void vtd_map_range(uint16_t sid, iova_t iova, phys_addr_t pa, size_t size)
> }
> }
>
> +static uint16_t vtd_intr_index_alloc(void)
> +{
> + static int index_ctr = 0;
> + assert(index_ctr < 65535);
> + return index_ctr++;
> +}
> +
> +static void vtd_setup_irte(struct pci_dev *dev, vtd_irte_t *irte,
> + int vector, int dest_id)
> +{
> + assert(sizeof(vtd_irte_t) == 16);
> + memset(irte, 0, sizeof(*irte));
> + irte->fault_disable = 1;
> + irte->dest_mode = 0; /* physical */
> + irte->trigger_mode = 0; /* edge */
> + irte->delivery_mode = 0; /* fixed */
> + irte->irte_mode = 0; /* remapped */
> + irte->vector = vector;
> + irte->dest_id = dest_id;
> + irte->source_id = dev->bdf;
> + irte->sid_q = 0;
> + irte->sid_vtype = 1; /* full-sid verify */
> + irte->present = 1;
> +}
> +
> +struct vtd_msi_addr {
> + uint32_t __dont_care:2;
> + uint32_t handle_15:1; /* handle[15] */
> + uint32_t shv:1;
> + uint32_t interrupt_format:1;
> + uint32_t handle_0_14:15; /* handle[0:14] */
> + uint32_t head:12; /* 0xfee */
> + uint32_t addr_hi; /* not used except with x2apic */
> +} __attribute__ ((packed));
> +typedef struct vtd_msi_addr vtd_msi_addr_t;
> +
> +struct vtd_msi_data {
> + uint16_t __reserved;
> + uint16_t subhandle;
> +} __attribute__ ((packed));
> +typedef struct vtd_msi_data vtd_msi_data_t;
> +
> +/**
> + * vtd_setup_msi - setup MSI message for a device
> + *
> + * @dev: PCI device to setup MSI
> + * @vector: interrupt vector
> + * @dest_id: destination processor
> + */
> +bool vtd_setup_msi(struct pci_dev *dev, int vector, int dest_id)
> +{
> + vtd_msi_data_t msi_data = {};
> + vtd_msi_addr_t msi_addr = {};
> + vtd_irte_t *irte = phys_to_virt(vtd_ir_table());
> + uint16_t index = vtd_intr_index_alloc();
> +
> + assert(sizeof(vtd_msi_addr_t) == 8);
> + assert(sizeof(vtd_msi_data_t) == 4);
> +
> + printf("INTR: setup IRTE index %d\n", index);
> + vtd_setup_irte(dev, irte + index, vector, dest_id);
> +
> + msi_addr.handle_15 = index >> 15 & 1;
> + msi_addr.shv = 0;
> + msi_addr.interrupt_format = 1;
> + msi_addr.handle_0_14 = index & 0x7fff;
> + msi_addr.head = 0xfee;
> + msi_data.subhandle = 0;
> +
> + return pci_setup_msi(dev, *(uint64_t *)&msi_addr,
> + *(uint32_t *)&msi_data);
> +}
> +
> void vtd_init(void)
> {
> setup_vm();
> diff --git a/lib/x86/intel-iommu.h b/lib/x86/intel-iommu.h
> index 1352639..13feafa 100644
> --- a/lib/x86/intel-iommu.h
> +++ b/lib/x86/intel-iommu.h
> @@ -137,5 +137,6 @@ static inline uint64_t vtd_readq(unsigned int reg)
>
> void vtd_init(void);
> void vtd_map_range(uint16_t sid, phys_addr_t iova, phys_addr_t pa, size_t size);
> +bool vtd_setup_msi(struct pci_dev *dev, int vector, int dest_id);
>
> #endif
> diff --git a/x86/intel-iommu.c b/x86/intel-iommu.c
> index 21fd57f..753f90e 100644
> --- a/x86/intel-iommu.c
> +++ b/x86/intel-iommu.c
> @@ -12,8 +12,10 @@
>
> #include "intel-iommu.h"
> #include "pci-edu.h"
> +#include "x86/apic.h"
>
> #define VTD_TEST_DMAR_4B ("DMAR 4B memcpy test")
> +#define VTD_TEST_IR_MSI ("IR MSI")
>
> void vtd_test_dmar(struct pci_edu_dev *dev)
> {
> @@ -51,6 +53,43 @@ void vtd_test_dmar(struct pci_edu_dev *dev)
> free_page(page);
> }
>
> +static volatile bool edu_intr_recved;
> +
> +static void edu_isr(isr_regs_t *regs)
> +{
> + edu_intr_recved = true;
> + eoi();
> +}
> +
> +static void vtd_test_ir(struct pci_edu_dev *dev)
> +{
> +#define VTD_TEST_VECTOR (0xee)
> + /*
> + * Setup EDU PCI device MSI, using interrupt remapping. By
> + * default, EDU device is using INTx.
> + */
> + if (!vtd_setup_msi(&dev->pci_dev, VTD_TEST_VECTOR, 0)) {
> + printf("edu device does not support MSI, skip test\n");
> + report_skip(VTD_TEST_IR_MSI);
> + return;
> + }
> +
> + handle_irq(VTD_TEST_VECTOR, edu_isr);
> + irq_enable();
> +
> + /* Manually trigger INTR */
> + edu_reg_writel(dev, EDU_REG_INTR_RAISE, 1);
> +
> + while (!edu_intr_recved)
> + cpu_relax();
> +
> + /* Clear INTR bits */
> + edu_reg_writel(dev, EDU_REG_INTR_RAISE, 0);
> +
> + /* We are good as long as we reach here */
> + report(VTD_TEST_IR_MSI, true);
> +}
> +
> int main(int argc, char *argv[])
> {
> struct pci_edu_dev dev;
> @@ -71,8 +110,11 @@ int main(int argc, char *argv[])
> printf("Please specify \"-device edu\" to do "
> "further IOMMU tests.\n");
> report_skip(VTD_TEST_DMAR_4B);
> - } else
> + report_skip(VTD_TEST_IR_MSI);
> + } else {
> vtd_test_dmar(&dev);
> + vtd_test_ir(&dev);
> + }
>
> return report_summary();
> }
> --
> 2.7.4
>
Deferring real-review to x86 people. Looks good to me.
Thanks,
drew
^ permalink raw reply [flat|nested] 29+ messages in thread
* THIS SERIES SHOULD BE: "[PATCH v4 00/13] VT-d unit test"
2016-11-14 22:27 [PATCH v3 00/13] VT-d unit test Peter Xu
` (12 preceding siblings ...)
2016-11-14 22:28 ` [PATCH v3 13/13] x86: intel-iommu: add IR MSI test Peter Xu
@ 2016-11-14 22:36 ` Peter Xu
2016-11-15 9:17 ` [PATCH v3 00/13] VT-d unit test Andrew Jones
14 siblings, 0 replies; 29+ messages in thread
From: Peter Xu @ 2016-11-14 22:36 UTC (permalink / raw)
To: kvm; +Cc: drjones, agordeev, jan.kiszka, rkrcmar, pbonzini
On Mon, Nov 14, 2016 at 05:27:58PM -0500, Peter Xu wrote:
> (Please ignore previous v3 version and use this one. My fault to not
> configure base branch before publish :-( Sorry for the noise.)
>
> This is v4 of vt-d unit test series.
>
> Patch "libcflat: add IS_ALIGNED() macro, and page sizes" is picked up
> by Drew in the ARM GIC framework series, so please feel free to drop
> it when needed.
>
> Online repo:
>
> https://github.com/xzpeter/kvm-unit-tests.git iommu-ut-v4
>
> Please review. Thanks,
Ok this time the patches are correct but I forget to manually increase
the version number for git-publish.
Please... s/v3/v4/ in all patch titles and cover letter as well. Sorry
again. I won't repost if no one disagree.
Thanks,
-- peterx
^ permalink raw reply [flat|nested] 29+ messages in thread* Re: [PATCH v3 00/13] VT-d unit test
2016-11-14 22:27 [PATCH v3 00/13] VT-d unit test Peter Xu
` (13 preceding siblings ...)
2016-11-14 22:36 ` THIS SERIES SHOULD BE: "[PATCH v4 00/13] VT-d unit test" Peter Xu
@ 2016-11-15 9:17 ` Andrew Jones
14 siblings, 0 replies; 29+ messages in thread
From: Andrew Jones @ 2016-11-15 9:17 UTC (permalink / raw)
To: Peter Xu; +Cc: kvm, agordeev, jan.kiszka, rkrcmar, pbonzini
On Mon, Nov 14, 2016 at 05:27:58PM -0500, Peter Xu wrote:
> (Please ignore previous v3 version and use this one. My fault to not
> configure base branch before publish :-( Sorry for the noise.)
>
> This is v4 of vt-d unit test series.
>
> Patch "libcflat: add IS_ALIGNED() macro, and page sizes" is picked up
> by Drew in the ARM GIC framework series, so please feel free to drop
> it when needed.
>
> Online repo:
>
> https://github.com/xzpeter/kvm-unit-tests.git iommu-ut-v4
>
> Please review. Thanks,
Reviewed. I didn't find anything major. We just need Paolo or Radim
to take a look now.
Thanks,
drew
^ permalink raw reply [flat|nested] 29+ messages in thread