xen-devel.lists.xenproject.org archive mirror
 help / color / mirror / Atom feed
* [PATCH 0 of 6 V6] amd iommu: support ats/gpgpu passthru on iommuv2 systems
@ 2012-03-08 13:21 Wei Wang
  2012-03-08 13:21 ` [PATCH 1 of 6 V6] amd iommu: Add 2 hypercalls for libxc Wei Wang
                   ` (7 more replies)
  0 siblings, 8 replies; 21+ messages in thread
From: Wei Wang @ 2012-03-08 13:21 UTC (permalink / raw)
  To: Ian.Jackson, Ian.Campbell, JBeulich, keir; +Cc: xen-devel

Hi,
This is patch set v6. It includes all pending patches that are needed to enable gpgpu passthrough and heterogeneous computing for guests.
thanks,
Wei

For more details, please refer to old threads.
http://lists.xen.org/archives/html/xen-devel/2012-02/msg00889.html
http://lists.xen.org/archives/html/xen-devel/2012-01/msg01646.html

and, for an overview of the design, please refer to
http://www.amd64.org/pub/iommuv2.png

======================================================================
changes in v6:
* Fix indentation issues.
* Fix definition of iommu_set_msi.
* Rebase on top of tip.

changes in v5:
* Remove patch 2 after upstream c/s 24729:6f6a6d1d2fb6

changes in v4:
* Only tool part in this version, since hypervisor patches have already been committed.
* rename guest config option from "iommu = {0,1}" to "guest_iommu = {0,1}"
* add description into docs/man/xl.cfg.pod.5


changes in v3:
* Use xenstore to receive guest iommu configuration instead of adding in a new field in hvm_info_table.
* Support pci segment in vbdf to mbdf bind.
* Make hypercalls visible for non-x86 platforms.
* A few code cleanups according to comments from Jan and Ian.

Changes in v2:
* Do not use linked list to access guest iommu tables.
* Do not parse iommu parameter in libxl_device_model_info again.
* Fix incorrect logical calculation in patch 11.
* Fix hypercall definition for non-x86 systems. 

^ permalink raw reply	[flat|nested] 21+ messages in thread
* [PATCH 3 of 6 V6] hvmloader: Build IVRS table
@ 2012-09-26 14:47 Wei Wang
  0 siblings, 0 replies; 21+ messages in thread
From: Wei Wang @ 2012-09-26 14:47 UTC (permalink / raw)
  To: xen-devel@lists.xensource.com, jbeulich, Keir Fraser,
	Ian Campbell, Ian Jackson

[-- Attachment #1: Type: text/plain, Size: 2 bytes --]




[-- Attachment #2: 0003-hvmloader-Build-IVRS-table.patch --]
[-- Type: text/plain, Size: 9130 bytes --]

From 27b1830d7bf046bdb63962537a9898c94fa3af01 Mon Sep 17 00:00:00 2001
From: Wei Wang <wei.wang2@amd.com>
Date: Wed, 26 Sep 2012 11:46:34 +0200
Subject: [PATCH 3/6] hvmloader: Build IVRS table.

There are 32 ivrs padding entries allocated at the beginning. If a passthru
device has been found from qemu bus, a padding entry will be replaced by a
real device entry. This patch has been tested with both rombios and seabios

Signed-off-by: Wei Wang <wei.wang2@amd.com>
---
 Config.mk                               |    2 +-
 tools/firmware/hvmloader/acpi/acpi2_0.h |   54 ++++++++++++++++++
 tools/firmware/hvmloader/acpi/build.c   |   91 +++++++++++++++++++++++++++++++
 tools/firmware/hvmloader/pci.c          |   30 ++++++++++-
 4 files changed, 175 insertions(+), 2 deletions(-)

diff --git a/Config.mk b/Config.mk
index d99b9a1..975a7e0 100644
--- a/Config.mk
+++ b/Config.mk
@@ -202,7 +202,7 @@ QEMU_UPSTREAM_URL ?= http://xenbits.xen.org/git-http/qemu-upstream-unstable.git
 SEABIOS_UPSTREAM_URL ?= http://xenbits.xen.org/git-http/seabios.git
 else
 OVMF_UPSTREAM_URL ?= git://xenbits.xen.org/ovmf.git
-QEMU_UPSTREAM_URL ?= git://xenbits.xen.org/qemu-upstream-unstable.git
+QEMU_UPSTREAM_URL ?= git://git.qemu.org/qemu.git
 SEABIOS_UPSTREAM_URL ?= git://xenbits.xen.org/seabios.git
 endif
 OVMF_UPSTREAM_REVISION ?= b0855f925c6e2e0b21fbb03fab4b5fb5b6876871
diff --git a/tools/firmware/hvmloader/acpi/acpi2_0.h b/tools/firmware/hvmloader/acpi/acpi2_0.h
index b281ec0..669305b 100644
--- a/tools/firmware/hvmloader/acpi/acpi2_0.h
+++ b/tools/firmware/hvmloader/acpi/acpi2_0.h
@@ -389,6 +389,60 @@ struct acpi_20_madt_intsrcovr {
 #define ACPI_2_0_WAET_REVISION 0x01
 #define ACPI_1_0_FADT_REVISION 0x01
 
+#define IVRS_SIGNATURE ASCII32('I','V','R','S')
+#define IVRS_REVISION           1
+#define IVRS_VASIZE             64
+#define IVRS_PASIZE             52
+#define IVRS_GVASIZE            64
+
+#define IVHD_BLOCK_TYPE         0x10
+#define IVHD_FLAG_HTTUNEN       (1 << 0)
+#define IVHD_FLAG_PASSPW        (1 << 1)
+#define IVHD_FLAG_RESPASSPW     (1 << 2)
+#define IVHD_FLAG_ISOC          (1 << 3)
+#define IVHD_FLAG_IOTLBSUP      (1 << 4)
+#define IVHD_FLAG_COHERENT      (1 << 5)
+#define IVHD_FLAG_PREFSUP       (1 << 6)
+#define IVHD_FLAG_PPRSUP        (1 << 7)
+
+#define IVHD_EFR_GTSUP          (1 << 2)
+#define IVHD_EFR_IASUP          (1 << 5)
+
+#define IVHD_SELECT_4_BYTE      0x2
+
+struct ivrs_ivhd_block
+{
+    uint8_t    type;
+    uint8_t    flags;
+    uint16_t   length;
+    uint16_t   devid;
+    uint16_t   cap_offset;
+    uint64_t   iommu_base_addr;
+    uint16_t   pci_segment;
+    uint16_t   iommu_info;
+    uint32_t   reserved;
+};
+
+/* IVHD 4-byte device entries */
+struct ivrs_ivhd_device
+{
+   uint8_t  type;
+   uint16_t dev_id;
+   uint8_t  flags;
+};
+
+#define PT_DEV_MAX_NR           32
+#define IOMMU_CAP_OFFSET        0x40
+struct acpi_40_ivrs
+{
+    struct acpi_header                      header;
+    uint32_t                                iv_info;
+    uint32_t                                reserved[2];
+    struct ivrs_ivhd_block                  ivhd_block;
+    struct ivrs_ivhd_device                 ivhd_device[PT_DEV_MAX_NR];
+};
+
+
 #pragma pack ()
 
 struct acpi_config {
diff --git a/tools/firmware/hvmloader/acpi/build.c b/tools/firmware/hvmloader/acpi/build.c
index d09785d..ca3697b 100644
--- a/tools/firmware/hvmloader/acpi/build.c
+++ b/tools/firmware/hvmloader/acpi/build.c
@@ -23,6 +23,8 @@
 #include "ssdt_pm.h"
 #include "../config.h"
 #include "../util.h"
+#include "../hypercall.h"
+#include <xen/hvm/params.h>
 
 #define align16(sz)        (((sz) + 15) & ~15)
 #define fixed_strcpy(d, s) strncpy((d), (s), sizeof(d))
@@ -198,6 +200,87 @@ static struct acpi_20_waet *construct_waet(void)
     return waet;
 }
 
+extern uint32_t ptdev_bdf[PT_DEV_MAX_NR];
+extern uint32_t ptdev_nr;
+extern uint32_t iommu_bdf;
+static struct acpi_40_ivrs* construct_ivrs(void)
+{
+    struct acpi_40_ivrs *ivrs;
+    uint64_t mmio;
+    struct ivrs_ivhd_block *ivhd;
+    struct ivrs_ivhd_device *dev_entry;
+    struct xen_hvm_param p;
+
+    if (ptdev_nr == 0 || iommu_bdf == 0) return NULL;
+
+    ivrs = mem_alloc(sizeof(*ivrs), 16);
+    if (!ivrs) 
+    {
+        printf("unable to build IVRS tables: out of memory\n");
+        return NULL;
+    }
+    memset(ivrs, 0, sizeof(*ivrs));
+
+    /* initialize acpi header */
+    ivrs->header.signature = IVRS_SIGNATURE;
+    ivrs->header.revision = IVRS_REVISION;
+    fixed_strcpy(ivrs->header.oem_id, ACPI_OEM_ID);
+    fixed_strcpy(ivrs->header.oem_table_id, ACPI_OEM_TABLE_ID);
+
+    ivrs->header.oem_revision = ACPI_OEM_REVISION;
+    ivrs->header.creator_id   = ACPI_CREATOR_ID;
+    ivrs->header.creator_revision = ACPI_CREATOR_REVISION;
+
+    ivrs->header.length = sizeof(*ivrs);
+
+    /* initialize IVHD Block */
+    ivhd = &ivrs->ivhd_block;
+    ivrs->iv_info = (IVRS_VASIZE << 15) | (IVRS_PASIZE << 8) |
+                    (IVRS_GVASIZE << 5);
+
+    ivhd->type          = IVHD_BLOCK_TYPE;
+    ivhd->flags         = IVHD_FLAG_PPRSUP | IVHD_FLAG_IOTLBSUP;
+    ivhd->devid         = iommu_bdf;
+    ivhd->cap_offset    = IOMMU_CAP_OFFSET;
+
+    /*reserve 32K IOMMU MMIO space */
+    mmio = virt_to_phys(mem_alloc(0x8000, 0x1000));
+    if (!mmio) 
+    {   
+        printf("unable to reserve iommu mmio pages: out of memory\n");
+        return NULL;
+    }
+    
+    p.domid = DOMID_SELF;
+    p.index = HVM_PARAM_IOMMU_BASE;
+    p.value = mmio;
+
+    /* Return non-zero if IOMMUv2 hardware is not avaliable */
+    if ( hypercall_hvm_op(HVMOP_set_param, &p) )
+    {
+        printf("unable to set iommu mmio base address\n");
+        return NULL;
+    }
+        
+    ivhd->iommu_base_addr = mmio;
+    ivhd->reserved = IVHD_EFR_IASUP | IVHD_EFR_GTSUP;
+
+    /* Build IVHD device entries */
+    dev_entry = ivrs->ivhd_device;
+    for ( int i = 0; i < ptdev_nr; i++ )
+    {
+        dev_entry[i].type   = IVHD_SELECT_4_BYTE;
+        dev_entry[i].dev_id = ptdev_bdf[i];
+        dev_entry[i].flags  = 0;
+    }
+
+    ivhd->length = sizeof(*ivhd) + sizeof(*dev_entry) * PT_DEV_MAX_NR;
+    set_checksum(ivrs, offsetof(struct acpi_header, checksum), 
+                 ivrs->header.length);
+
+    return ivrs;
+}
+
 static int construct_secondary_tables(unsigned long *table_ptrs,
                                       struct acpi_info *info)
 {
@@ -206,6 +289,7 @@ static int construct_secondary_tables(unsigned long *table_ptrs,
     struct acpi_20_hpet *hpet;
     struct acpi_20_waet *waet;
     struct acpi_20_tcpa *tcpa;
+    struct acpi_40_ivrs *ivrs;
     unsigned char *ssdt;
     static const uint16_t tis_signature[] = {0x0001, 0x0001, 0x0001};
     uint16_t *tis_hdr;
@@ -293,6 +377,13 @@ static int construct_secondary_tables(unsigned long *table_ptrs,
         }
     }
 
+    if ( !strncmp(xenstore_read("guest_iommu", "1"), "1", 1) )
+    {
+        ivrs = construct_ivrs();
+        if ( ivrs != NULL )
+            table_ptrs[nr_tables++] = (unsigned long)ivrs;
+    }
+
     table_ptrs[nr_tables] = 0;
     return nr_tables;
 }
diff --git a/tools/firmware/hvmloader/pci.c b/tools/firmware/hvmloader/pci.c
index fd56e50..cd8e821 100644
--- a/tools/firmware/hvmloader/pci.c
+++ b/tools/firmware/hvmloader/pci.c
@@ -34,11 +34,17 @@ unsigned long pci_mem_end = PCI_MEM_END;
 enum virtual_vga virtual_vga = VGA_none;
 unsigned long igd_opregion_pgbase = 0;
 
+/* support up to 32 passthrough devices */
+#define PT_DEV_MAX_NR           32
+uint32_t ptdev_bdf[PT_DEV_MAX_NR];
+uint32_t ptdev_nr;
+uint32_t iommu_bdf = 0;
+
 void pci_setup(void)
 {
     uint32_t base, devfn, bar_reg, bar_data, bar_sz, cmd, mmio_total = 0;
     uint32_t vga_devfn = 256;
-    uint16_t class, vendor_id, device_id;
+    uint16_t class, vendor_id, device_id, sub_vendor_id;
     unsigned int bar, pin, link, isa_irq;
 
     /* Resources assignable to PCI devices via BARs. */
@@ -72,12 +78,34 @@ void pci_setup(void)
         class     = pci_readw(devfn, PCI_CLASS_DEVICE);
         vendor_id = pci_readw(devfn, PCI_VENDOR_ID);
         device_id = pci_readw(devfn, PCI_DEVICE_ID);
+        sub_vendor_id = pci_readw(devfn, PCI_SUBSYSTEM_VENDOR_ID);
+
         if ( (vendor_id == 0xffff) && (device_id == 0xffff) )
             continue;
 
         ASSERT((devfn != PCI_ISA_DEVFN) ||
                ((vendor_id == 0x8086) && (device_id == 0x7000)));
 
+        /* Found amd iommu device. */
+        if ( class == 0x0806 && vendor_id == 0x1022 )
+        {
+            iommu_bdf = devfn;
+            continue;
+        }
+        /* IVRS: Detecting passthrough devices.
+         * sub_vendor_id != citrix && sub_vendor_id != qemu */
+        if ( sub_vendor_id != 0x5853 && sub_vendor_id != 0x1af4 )
+        {
+            /* found a passthru device */
+            if ( ptdev_nr < PT_DEV_MAX_NR )
+            {
+                ptdev_bdf[ptdev_nr] = devfn;
+                ptdev_nr++;
+            }
+            else
+                printf("Number of passthru devices > PT_DEV_MAX_NR \n");
+        }
+
         switch ( class )
         {
         case 0x0300:
-- 
1.7.4



[-- Attachment #3: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

^ permalink raw reply related	[flat|nested] 21+ messages in thread
end of thread, other threads:[~2012-09-26 14:47 UTC | newest]

Thread overview: 21+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2012-03-08 13:21 [PATCH 0 of 6 V6] amd iommu: support ats/gpgpu passthru on iommuv2 systems Wei Wang
2012-03-08 13:21 ` [PATCH 1 of 6 V6] amd iommu: Add 2 hypercalls for libxc Wei Wang
2012-03-08 13:35   ` Jan Beulich
2012-03-08 14:18     ` Wei Wang
2012-03-08 14:41       ` Jan Beulich
2012-03-12 11:25       ` Ian Jackson
2012-03-12 15:43         ` Wei Wang
2012-03-14 11:55           ` Ian Jackson
2012-03-08 13:21 ` [PATCH 2 of 6 V6] amd iommu: call guest_iommu_set_base from hvmloader Wei Wang
2012-03-08 13:21 ` [PATCH 3 of 6 V6] hvmloader: Build IVRS table Wei Wang
2012-03-08 14:22   ` Zhang, Xiantao
2012-03-08 14:49     ` Wei Wang
2012-03-09  0:55       ` Zhang, Xiantao
2012-03-08 13:21 ` [PATCH 4 of 6 V6] libxc: add wrappers for new hypercalls Wei Wang
2012-03-08 13:21 ` [PATCH 5 of 6 V6] libxl: bind virtual bdf to physical bdf after device assignment Wei Wang
2012-03-08 13:21 ` [PATCH 6 of 6 V6] libxl: Introduce a new guest config file parameter Wei Wang
2012-03-14 14:02 ` [PATCH 0 of 6 V6] amd iommu: support ats/gpgpu passthru on iommuv2 systems Ian Jackson
2012-03-14 14:32   ` Wei Wang
2012-03-14 14:09 ` Ian Campbell
2012-03-14 14:48   ` Wei Wang
  -- strict thread matches above, loose matches on Subject: below --
2012-09-26 14:47 [PATCH 3 of 6 V6] hvmloader: Build IVRS table Wei Wang

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).