Re: Multi-bridged PCIe devices (Was: Re: iommuu/vt-d issues with LSI MegaSAS (PERC5i))

[Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>]

[-- Attachment #1: Type: text/plain, Size: 6898 bytes --]

On Tue, Jan 07, 2014 at 10:44:29AM +0000, Gordan Bobic wrote:
> On 2014-01-07 10:38, Andrew Cooper wrote:
> >On 07/01/14 10:35, Gordan Bobic wrote:
> >>On 2014-01-07 03:17, Zhang, Yang Z wrote:
> >>>Konrad Rzeszutek Wilk wrote on 2014-01-07:
> >>>>>Which would look like this:
> >>>>>
> >>>>>C220 ---> Tundra Bridge -----> (HB6 PCI bridge -> Brooktree BDFs)
> >>>>>on the card
> >>>>>          \--------------> IEEE-1394a
> >>>>>
> >>>>>I am actually wondering if this 07:00.0 device is the one that
> >>>>>reports itself as 08:00.0 (which I think is what you alluding to
> >>>>>Jan)
> >>>>>
> >>>>
> >>>>And to double check that theory I decided to pass in the IEEE-1394a
> >>>>to a guest:
> >>>>
> >>>>           +-1c.5-[07-08]----00.0-[08]----03.0  Texas Instruments
> >>>>TSB43AB22A IEEE-1394a-2000 Controller (PHY/Link) [iOHCI-Lynx]
> >>>>
> >>>>
> >>>>(XEN) [VT-D]iommu.c:885: iommu_fault_status: Fault Overflow (XEN)
> >>>>[VT-D]iommu.c:887: iommu_fault_status: Primary Pending Fault (XEN)
> >>>>[VT-D]iommu.c:865: DMAR:[DMA Read] Request device
> >>>>[0000:08:00.0] fault
> >>>>addr 370f1000, iommu reg = ffff82c3ffd53000 (XEN) DMAR:[fault reason
> >>>>02h] Present bit in context entry is clear (XEN) print_vtd_entries:
> >>>>iommu ffff83083d4939b0 dev 0000:08:00.0 gmfn 370f1 (XEN)
> >>>>root_entry
> >>>>= ffff83083d47f000 (XEN)     root_entry[8] = 72569b001 (XEN)
> >>>>context
> >>>>= ffff83072569b000 (XEN)     context[0] = 0_0 (XEN)
> >>>>ctxt_entry[0]
> >>>>not present
> >>>>
> >>>>So, capture card OK - Likely the Tundra bridge has an issue:
> >>>>
> >>>>07:00.0 PCI bridge: Tundra Semiconductor Corp. Device 8113 (rev 01)
> >>>>(prog-if 01 [Subtractive decode])
> >>>>        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop-
> >>>>        ParErr- Stepping- SERR- FastB2B- DisINTx- Status:
> >>>>Cap+ 66MHz-
> >>>>        UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort+
> >>>>        >SERR- <PERR- INTx- Latency: 0 Bus: primary=07,
> >>>>secondary=08,
> >>>>        subordinate=08, sec-latency=32 Memory behind bridge:
> >>>>        f0600000-f06fffff Secondary status: 66MHz+ FastB2B+ ParErr-
> >>>>        DEVSEL=medium TAbort- <TAbort- <MAbort+ <SERR- <PERR-
> >>>>BridgeCtl:
> >>>>        Parity- SERR- NoISA- VGA- MAbort- >Reset- FastB2B-
> >>>>                PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
> >>>>        Capabilities: [60] Subsystem: Super Micro Computer Inc
> >>>>Device 0805
> >>>>        Capabilities: [a0] Power Management version 3
> >>>>                Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
> >>>>                PME(D0+,D1+,D2+,D3hot+,D3cold+) Status: D0
> >>>>NoSoftRst+
> >>>>                PME-Enable- DSel=0 DScale=0 PME-
> >>>>
> >>>>or there is some unknown bridge in the motherboard.
> >>>
> >>>According your description above, the upstream Linux should also have
> >>>the same problem. Did you see it with upstream Linux?
> >>
> >>The problem I was seeing with LSI cards (phantom device doing DMA)
> >>does, indeed, also occur in upstream Linux. If I enable intel-iommu on
> >>bare metal Linux, the same problem occurs as with Xen.
> >>
> >>>There may be some buggy device that generate DMA request with
> >>>internal
> >>>BDF but it didn't expose it(not like Phantom device). For those
> >>>devices, I think we need to setup the VT-d page table manually.
> >>
> >>I think what is needed is a pci-phantom style override that tells the
> >>hypervisor to tell the IOMMU to allow DMA traffic from a specific
> >>invisible device ID.
> >>
> >>Gordan
> >
> >There is.  See "pci-phantom" in
> >http://xenbits.xen.org/docs/unstable/misc/xen-command-line.html
> 
> I thought this was only applicable to phantom _functions_ (number
> after the
> dot) rather than whole phantom _devices_. Is that not the case?

My new Supermicro X10SAE has this issue as well and I wrote a patch to 
"fix" this. Just to be clear - this is an issue with the motherboard
(or rather the PCI-to-PCI bridge) is that it violates VT-d chapter
3.9.2:
"For devices behind conventional PCI bridges, the source-id in the DMA
requests is the requestor-id of the bridge device.".

In my case the bridge is is:

06:00.0 PCI bridge: Tundra Semiconductor Corp. Device 8113 (rev 01)

But all the requests from the devices behind the bridge show up with
request-id +1 (aka 07:00.0):


(XEN) [2014-02-05 22:23:23] [VT-D]iommu.c:880: iommu_fau3] [VT-D]iommu.c:882: iommu_fault_status: Primary Pending Fault
(XEN) [2014-02-05 22:23:23] [VT-D]iommu.c:860: DMAR:[DMA Read] Request device [0000:07:00.0] fault addr a57fc000, iommu reg = ffff82c000203000
(XEN) [2014-02-05 22:23:23] DMAR:[fault reason 02h] Present bit in context entry is clear
(XEN) [2014-02-05 22:23:23] print_vtd_entries: iommu ffff83023948b9c0 dev 0000:07:00.0 gmfn a57fc
(XEN) [2014-02-05 22:23:23]     root_entry = ffff82004000d000
(XEN) [2014-02-05 22:23:23]     root_entry[7] = 1b8eba001
(XEN) [2014-02-05 22:23:23]     context = ffff82004000e000
(XEN) [2014-02-05 22:23:23]     context[0] = 0_0
(XEN) [2014-02-05 22:23:23]     ctxt_entry[0] not present
[   21.305708] firewire_ohci 0000:07:03.0: added OHCI v1.10 device as card 0, 4 IR + 8 IT contexts, quirks 0x2
(XEN) [2014-02-05 22:23:24] [VT-D]iommu.c:880: iommu_fault_status: Fault Overflow
(XEN) [2014-02-05 22:23:24] [VT-D]iommu.c:882: iommu_fault_status: Primary Pending Fault
(XEN) [2014-02-05 22:23:24] [VT-D]iommu.c:860: DMAR:[DMA Read] Request device [0000:07:00.0] fault addr b76cd000, iommu reg = ffff82c000203000
(XEN) [2014-02-05 22:23:24] DMAR:[fault reason 02h] Present bit in context entry is clear
(XEN) [2014-02-05 22:23:24] print_vtd_entries: iommu ffff83023948b9c0 dev 0000:07:00.0 gmfn b76cd
(XEN) [2014-02-05 22:23:24]     root_entry = ffff82004000d000
(XEN) [2014-02-05 22:23:24]     root_entry[7] = 1b8eba001
(XEN) [2014-02-05 22:23:24]     context = ffff82004000e000
(XEN) [2014-02-05 22:23:24]     context[0] = 0_0
(XEN) [2014-02-05 22:23:24]     ctxt_entry[0] not present
[   21.386993] firewire_ohci 0000:07:03.0: bad self ID 0/1 (00000000 != ~00000000)
[   21.402892] initcall fw_ohci_init+0x0/0x1000 [firewire_ohci] returned 0 after 254567 usecs


Anyhow, the "solution" was to provide a "link" to the device
being passed in and create a fake device in Xen so that when
I passthrough my fireware: 07:03.0 it will also setup
a context for 07:00.0 device (which does not exist at all).

Attached is the Linux kernel module I use to let the hypervisor
know about the new device (this could have been written as
an user application too).

And the patch for the hypervisor, along with some extra debug
patches so when doing 'xl debug-keys Q' you can get a better
sense of what is what.

The 0004-xen-pci-Introduce-a-way-to-deal-with-buggy-hardware-.patch
is what is interesting.

If there is an interest in upstream this I can take a look -
but I will need guidance from Jan how he would like to do it.


[-- Attachment #2: hack.c --]
[-- Type: text/plain, Size: 1995 bytes --]


#include <linux/module.h>
#include <linux/string.h>
#include <linux/types.h>
#include <linux/init.h>
#include <linux/stat.h>
#include <linux/err.h>
#include <linux/ctype.h>
#include <linux/slab.h>
#include <linux/limits.h>
#include <linux/device.h>
#include <linux/pci.h>
#include <linux/device.h>

#include <linux/pci.h>

#include <xen/interface/xen.h>
#include <xen/interface/physdev.h>

#include <asm/xen/hypervisor.h>
#include <asm/xen/hypercall.h>

#define LSI_HACK  "0.1"

MODULE_AUTHOR("Konrad Rzeszutek Wilk <konrad.wilk@xxxxxxxxxx>");
MODULE_DESCRIPTION("lsi hack");
MODULE_LICENSE("GPL");
MODULE_VERSION(LSI_HACK);

/* Want to link to the device passed in the guest */
int bus = 7;
module_param(bus, int, 0644);
MODULE_PARM_DESC(bus, "bus");
int slot = 3;
module_param(slot, int, 0644);
MODULE_PARM_DESC(slot, "slot");
int func = 0;
module_param(func, int, 0644);
MODULE_PARM_DESC(func, "slot");

#define XEN_PCI_DEV_LINK 0x8
static int __init lsi_hack_init(void)
{
        int r = 0;

        struct physdev_pci_device_add add = {
			.seg	= 0,
                        .bus    = 0x7,
                        .devfn  = PCI_DEVFN(0,0),
			.physfn.bus	= bus, /* The device we want to copy from */
			.physfn.devfn	= PCI_DEVFN(slot,func),
			.flags = XEN_PCI_DEV_LINK,
                };
	printk("%s: %02x:%02x.%u, %02x:%02x.%u, %x\n",
		__func__, add.bus, PCI_SLOT(add.devfn),
		PCI_FUNC(add.devfn), add.physfn.bus,
		PCI_SLOT(add.physfn.devfn), PCI_SLOT(add.physfn.devfn),
		add.flags);
        r = HYPERVISOR_physdev_op(PHYSDEVOP_pci_device_add, &add);

        return r;
}

static void __exit lsi_hack_exit(void)
{
        int r = 0;
        struct physdev_manage_pci manage_pci;

        manage_pci.bus = 0x7;
        manage_pci.devfn = PCI_DEVFN(0,0);

        r = HYPERVISOR_physdev_op(PHYSDEVOP_manage_pci_remove,
                &manage_pci);
        if (r)
                printk(KERN_ERR "%s: %d\n", __FUNCTION__, r);
}

module_init(lsi_hack_init);
module_exit(lsi_hack_exit);

[-- Attachment #3: 0004-xen-pci-Introduce-a-way-to-deal-with-buggy-hardware-.patch --]
[-- Type: text/plain, Size: 11687 bytes --]

>From cb165429726978952f5b9e75bece1dcb5630667f Mon Sep 17 00:00:00 2001
From: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
Date: Wed, 19 Feb 2014 10:58:19 -0500
Subject: [PATCH 4/6] xen/pci: Introduce a way to deal with buggy hardware with
 "hidden" PCI buses.

Signed-off-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
---
 xen/arch/x86/physdev.c              | 14 +++++-
 xen/drivers/passthrough/pci.c       | 89 +++++++++++++++++++++++++++++++++----
 xen/drivers/passthrough/vtd/iommu.c | 31 +++++++++++++
 xen/include/public/physdev.h        |  1 +
 xen/include/xen/pci.h               |  3 ++
 5 files changed, 128 insertions(+), 10 deletions(-)

diff --git a/xen/arch/x86/physdev.c b/xen/arch/x86/physdev.c
index bc0634c..f843c49 100644
--- a/xen/arch/x86/physdev.c
+++ b/xen/arch/x86/physdev.c
@@ -609,7 +609,11 @@ ret_t do_physdev_op(int cmd, XEN_GUEST_HANDLE_PARAM(void) arg)
         if ( copy_from_guest(&add, arg, 1) != 0 )
             break;
 
-        pdev_info.is_extfn = !!(add.flags & XEN_PCI_DEV_EXTFN);
+        if ( add.flags & XEN_PCI_DEV_EXTFN)
+            pdev_info.is_extfn = 1;
+        else
+            pdev_info.is_extfn = 0;
+
         if ( add.flags & XEN_PCI_DEV_VIRTFN )
         {
             pdev_info.is_virtfn = 1;
@@ -618,6 +622,14 @@ ret_t do_physdev_op(int cmd, XEN_GUEST_HANDLE_PARAM(void) arg)
         }
         else
             pdev_info.is_virtfn = 0;
+
+        if ( add.flags & XEN_PCI_DEV_LINK )
+        {
+            pdev_info.is_link = 1;
+            pdev_info.physfn.bus = add.physfn.bus;
+            pdev_info.physfn.devfn = add.physfn.devfn;
+        } else
+            pdev_info.is_link = 0;
         ret = pci_add_device(add.seg, add.bus, add.devfn, &pdev_info);
         break;
     }
diff --git a/xen/drivers/passthrough/pci.c b/xen/drivers/passthrough/pci.c
index 2a6eaa4..0e59216 100644
--- a/xen/drivers/passthrough/pci.c
+++ b/xen/drivers/passthrough/pci.c
@@ -153,7 +153,8 @@ static void __init parse_phantom_dev(char *str) {
 }
 custom_param("pci-phantom", parse_phantom_dev);
 
-static struct pci_dev *alloc_pdev(struct pci_seg *pseg, u8 bus, u8 devfn)
+static struct pci_dev *alloc_pdev(struct pci_seg *pseg, u8 bus, u8 devfn,
+                                  int link, u8 orig_bus, u8 orig_devfn)
 {
     struct pci_dev *pdev;
 
@@ -169,8 +170,38 @@ static struct pci_dev *alloc_pdev(struct pci_seg *pseg, u8 bus, u8 devfn)
     *((u8*) &pdev->bus) = bus;
     *((u8*) &pdev->devfn) = devfn;
     pdev->domain = NULL;
+    pdev->link = NULL;
     INIT_LIST_HEAD(&pdev->msi_list);
 
+    if ( link )
+    {
+        struct pci_dev *dev;
+        list_for_each_entry ( dev, &pseg->alldevs_list, alldevs_list )
+        {
+            if ( dev->bus == orig_bus && dev->devfn == orig_devfn )
+            {
+                /* N.B. The 'bus' passed is 'new' one, while 'orig_bus' are
+                 * the ones we expect to exist. We over-write 'bus' and
+                 * 'devfn' with the original one so that this new device
+                 * will be created with the original device properties.
+                 */
+                if ( dev->link )
+                {
+                    xfree (pdev);
+                    return NULL;
+                }
+                bus = dev->bus;
+                devfn = dev->devfn;
+                dev->link = pdev;
+                pdev->link = dev;
+                pdev->info.is_link = 1;
+                printk("%04x:%02x:%02x.%u linked with %02x:%02x.%u\n",
+                       pseg->nr, pdev->bus, PCI_SLOT(pdev->devfn),
+                       PCI_FUNC(pdev->devfn), dev->bus, PCI_SLOT(dev->devfn),
+                       PCI_FUNC(dev->devfn));
+            }
+        }
+    }
     if ( pci_find_cap_offset(pseg->nr, bus, PCI_SLOT(devfn), PCI_FUNC(devfn),
                              PCI_CAP_ID_MSIX) )
     {
@@ -201,12 +232,32 @@ static struct pci_dev *alloc_pdev(struct pci_seg *pseg, u8 bus, u8 devfn)
             sub_bus = pci_conf_read8(pseg->nr, bus, PCI_SLOT(devfn),
                                      PCI_FUNC(devfn), PCI_SUBORDINATE_BUS);
 
+            if ( pdev->info.is_link )
+            {
+                if ( sec_bus >= pdev->bus && pdev->bus <= sub_bus )
+                {
+#if 0
+                    u8 i = sec_bus;
+                    /* We can create an loop in bus2bridge by pointing to ourselves.
+                     * Hence destroy sec_bus up to pdev_bus values */
+                    spin_lock(&pseg->bus2bridge_lock);
+                    for ( ; i <= pdev->bus; i++ )
+                        pseg->bus2bridge[sec_bus].map = 0;
+                    spin_unlock(&pseg->bus2bridge_lock);
+                    /* And increment it so it won't cover us again*/
+                    sec_bus = pdev->bus + 1;
+                    printk("Link corrected [%02x:%02x:%u] spanning %x->%x\n", pdev->bus,
+                           PCI_SLOT(pdev->devfn), PCI_FUNC(pdev->devfn),sec_bus, sub_bus);
+#endif
+                    break;
+                }
+            }
             spin_lock(&pseg->bus2bridge_lock);
             for ( ; sec_bus <= sub_bus; sec_bus++ )
             {
                 pseg->bus2bridge[sec_bus].map = 1;
-                pseg->bus2bridge[sec_bus].bus = bus;
-                pseg->bus2bridge[sec_bus].devfn = devfn;
+                pseg->bus2bridge[sec_bus].bus = pdev->bus;
+                pseg->bus2bridge[sec_bus].devfn = pdev->devfn;
             }
             spin_unlock(&pseg->bus2bridge_lock);
             break;
@@ -299,7 +350,7 @@ int __init pci_hide_device(int bus, int devfn)
     int rc = -ENOMEM;
 
     spin_lock(&pcidevs_lock);
-    pdev = alloc_pdev(get_pseg(0), bus, devfn);
+    pdev = alloc_pdev(get_pseg(0), bus, devfn, 0, 0, 0);
     if ( pdev )
     {
         _pci_hide_device(pdev);
@@ -317,7 +368,7 @@ int __init pci_ro_device(int seg, int bus, int devfn)
 
     if ( !pseg )
         return -ENOMEM;
-    pdev = alloc_pdev(pseg, bus, devfn);
+    pdev = alloc_pdev(pseg, bus, devfn, 0, 0, 0);
     if ( !pdev )
         return -ENOMEM;
 
@@ -458,6 +509,7 @@ int pci_add_device(u16 seg, u8 bus, u8 devfn, const struct pci_dev_info *info)
     struct pci_seg *pseg;
     struct pci_dev *pdev;
     unsigned int slot = PCI_SLOT(devfn), func = PCI_FUNC(devfn);
+    u8 bus_link = 0, devfn_link = 0;
     const char *pdev_type;
     int ret;
 
@@ -474,6 +526,12 @@ int pci_add_device(u16 seg, u8 bus, u8 devfn, const struct pci_dev_info *info)
             pci_add_device(seg, info->physfn.bus, info->physfn.devfn, NULL);
         pdev_type = "virtual function";
     }
+    else if (info->is_link)
+    {
+        bus_link = info->physfn.bus;
+        devfn_link = info->physfn.devfn;
+        pdev_type = "link";
+    }
     else
     {
         info = NULL;
@@ -490,7 +548,7 @@ int pci_add_device(u16 seg, u8 bus, u8 devfn, const struct pci_dev_info *info)
     pseg = alloc_pseg(seg);
     if ( !pseg )
         goto out;
-    pdev = alloc_pdev(pseg, bus, devfn);
+    pdev = alloc_pdev(pseg, bus, devfn, (info && info->is_link), bus_link, devfn_link);
     if ( !pdev )
         goto out;
 
@@ -604,7 +662,7 @@ out:
 int pci_remove_device(u16 seg, u8 bus, u8 devfn)
 {
     struct pci_seg *pseg = get_pseg(seg);
-    struct pci_dev *pdev;
+    struct pci_dev *pdev, *link = NULL;
     int ret;
 
     ret = xsm_resource_unplug_pci(XSM_PRIV, (seg << 16) | (bus << 8) | devfn);
@@ -617,16 +675,29 @@ int pci_remove_device(u16 seg, u8 bus, u8 devfn)
         return -ENODEV;
 
     spin_lock(&pcidevs_lock);
+retry:
     list_for_each_entry ( pdev, &pseg->alldevs_list, alldevs_list )
         if ( pdev->bus == bus && pdev->devfn == devfn )
         {
             ret = iommu_remove_device(pdev);
             if ( pdev->domain )
                 list_del(&pdev->domain_list);
-            pci_cleanup_msi(pdev);
+            if ( !pdev->info.is_link ) /* If we are not the 'fake device' */
+                pci_cleanup_msi(pdev);
+            if ( pdev->link ) {
+                /* Can be NULL if the other device was removed first. */
+                link = pdev->link;
+            }
             free_pdev(pseg, pdev);
             printk(XENLOG_DEBUG "PCI remove device %04x:%02x:%02x.%u\n",
                    seg, bus, PCI_SLOT(devfn), PCI_FUNC(devfn));
+            if ( link )
+            {
+                bus = link->bus;
+                devfn = link->devfn;
+                link->link = NULL;
+                goto retry;
+            }
             break;
         }
 
@@ -838,7 +909,7 @@ static int __init _scan_pci_devices(struct pci_seg *pseg, void *arg)
                     continue;
                 }
 
-                pdev = alloc_pdev(pseg, bus, PCI_DEVFN(dev, func));
+                pdev = alloc_pdev(pseg, bus, PCI_DEVFN(dev, func), 0, 0, 0);
                 if ( !pdev )
                 {
                     printk("%s: alloc_pdev failed.\n", __func__);
diff --git a/xen/drivers/passthrough/vtd/iommu.c b/xen/drivers/passthrough/vtd/iommu.c
index 5f10034..a5a4664 100644
--- a/xen/drivers/passthrough/vtd/iommu.c
+++ b/xen/drivers/passthrough/vtd/iommu.c
@@ -1468,6 +1468,25 @@ static int domain_context_mapping(
         if ( ret )
             break;
 
+        if ( pdev->link )
+        {
+            u8 bus_link, devfn_link;
+            struct pci_dev *link_dev = pdev->link;
+
+            ASSERT ( link_dev );
+
+            bus_link = link_dev->bus;
+            devfn_link = link_dev->devfn;
+
+            if ( iommu_verbose )
+                dprintk(VTDPREFIX, "d%d:PCI: map %04x:%02x:%02x.%u\n",
+                        domain->domain_id, seg, bus_link,
+                        PCI_SLOT(devfn_link), PCI_FUNC(devfn_link));
+
+
+            ret = domain_context_mapping_one(domain, drhd->iommu, bus_link, devfn_link,
+                                             pci_get_pdev(seg, bus, devfn));
+        }
         if ( find_upstream_bridge(seg, &bus, &devfn, &secbus) < 1 )
             break;
 
@@ -1603,6 +1622,18 @@ static int domain_context_unmap(
         if ( ret )
             break;
 
+        if ( pdev->link )
+        {
+            struct pci_dev *link = pdev->link;
+
+            ASSERT(link->link == pdev);
+            tmp_bus = link->bus;
+            tmp_devfn = link->devfn;
+            if ( iommu_verbose )
+                dprintk(VTDPREFIX, "d%d:PCI: unmap %04x:%02x:%02x.%u\n",
+                        domain->domain_id, seg, tmp_bus, PCI_SLOT(tmp_devfn), PCI_FUNC(tmp_devfn));
+            ret = domain_context_unmap_one(domain, iommu, tmp_bus, tmp_devfn);
+        }
         tmp_bus = bus;
         tmp_devfn = devfn;
         if ( find_upstream_bridge(seg, &tmp_bus, &tmp_devfn, &secbus) < 1 )
diff --git a/xen/include/public/physdev.h b/xen/include/public/physdev.h
index d547928..c476175 100644
--- a/xen/include/public/physdev.h
+++ b/xen/include/public/physdev.h
@@ -281,6 +281,7 @@ DEFINE_XEN_GUEST_HANDLE(physdev_pci_mmcfg_reserved_t);
 #define XEN_PCI_DEV_EXTFN              0x1
 #define XEN_PCI_DEV_VIRTFN             0x2
 #define XEN_PCI_DEV_PXM                0x4
+#define XEN_PCI_DEV_LINK               0x8
 
 #define PHYSDEVOP_pci_device_add        25
 struct physdev_pci_device_add {
diff --git a/xen/include/xen/pci.h b/xen/include/xen/pci.h
index cadb525..b883c28 100644
--- a/xen/include/xen/pci.h
+++ b/xen/include/xen/pci.h
@@ -39,6 +39,7 @@ struct pci_dev_info {
         u8 bus;
         u8 devfn;
     } physfn;
+    bool_t is_link;
 };
 
 struct pci_dev {
@@ -75,6 +76,8 @@ struct pci_dev {
 #define PT_FAULT_THRESHOLD 10
     } fault;
     u64 vf_rlen[6];
+
+    struct pci_dev *link;
 };
 
 #define for_each_pdev(domain, pdev) \
-- 
1.8.3.1


[-- Attachment #4: 0001-pci-On-PCI-dump-keyhandler-include-bus2bridge-inform.patch --]
[-- Type: text/plain, Size: 1502 bytes --]

>From 3edf6e0b1b646a358ae14c64e726ad24350ad510 Mon Sep 17 00:00:00 2001
From: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
Date: Tue, 4 Feb 2014 12:54:53 -0500
Subject: [PATCH 1/6] pci: On PCI dump keyhandler include bus2bridge
 information

As it helps in figuring out whether they match reality if the
initial domain has altered the bus topology.

Signed-off-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
---
 xen/drivers/passthrough/pci.c | 12 ++++++++++++
 1 file changed, 12 insertions(+)

diff --git a/xen/drivers/passthrough/pci.c b/xen/drivers/passthrough/pci.c
index c5c8344..64dfd73 100644
--- a/xen/drivers/passthrough/pci.c
+++ b/xen/drivers/passthrough/pci.c
@@ -942,6 +942,7 @@ static int _dump_pci_devices(struct pci_seg *pseg, void *arg)
 {
     struct pci_dev *pdev;
     struct msi_desc *msi;
+    int i;
 
     printk("==== segment %04x ====\n", pseg->nr);
 
@@ -955,6 +956,17 @@ static int _dump_pci_devices(struct pci_seg *pseg, void *arg)
                printk("%d ", msi->irq);
         printk(">\n");
     }
+    printk("==== Bus2Bridge %04x ====\n", pseg->nr);
+    spin_lock(&pseg->bus2bridge_lock);
+    for ( i = 0; i < MAX_BUSES; i++)
+    {
+        if ( !pseg->bus2bridge[i].map )
+            continue;
+        printk("%02x -> %02x:%02x.%u\n", i, pseg->bus2bridge[i].bus,
+               PCI_SLOT(pseg->bus2bridge[i].devfn),
+               PCI_FUNC(pseg->bus2bridge[i].devfn));
+    }
+    spin_unlock(&pseg->bus2bridge_lock);
 
     return 0;
 }
-- 
1.8.3.1


[-- Attachment #5: 0002-pci-On-PCI-dump-device-keyhandler-include-Device-and.patch --]
[-- Type: text/plain, Size: 1458 bytes --]

>From ad49978e083123a1068461cd7f2a8e0c2becca17 Mon Sep 17 00:00:00 2001
From: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
Date: Tue, 4 Feb 2014 12:52:35 -0500
Subject: [PATCH 2/6] pci: On PCI dump device keyhandler include Device and
 Vendor ID

As it helps in troubleshooting if the initial domain has
re-numbered the bus numbers and what Xen sees is not the
reality.

Signed-off-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
---
 xen/drivers/passthrough/pci.c | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/xen/drivers/passthrough/pci.c b/xen/drivers/passthrough/pci.c
index 64dfd73..1ad4f17 100644
--- a/xen/drivers/passthrough/pci.c
+++ b/xen/drivers/passthrough/pci.c
@@ -948,9 +948,12 @@ static int _dump_pci_devices(struct pci_seg *pseg, void *arg)
 
     list_for_each_entry ( pdev, &pseg->alldevs_list, alldevs_list )
     {
-        printk("%04x:%02x:%02x.%u - dom %-3d - MSIs < ",
+        int id = pci_conf_read32(pseg->nr, pdev->bus, PCI_SLOT(pdev->devfn),
+                                 PCI_FUNC(pdev->devfn), 0);
+        printk("%04x:%02x:%02x.%u (%04x:%04x)- dom %-3d - MSIs < ",
                pseg->nr, pdev->bus,
                PCI_SLOT(pdev->devfn), PCI_FUNC(pdev->devfn),
+               id & 0xffff, (id >> 16) & 0xffff,
                pdev->domain ? pdev->domain->domain_id : -1);
         list_for_each_entry ( msi, &pdev->msi_list, list )
                printk("%d ", msi->irq);
-- 
1.8.3.1


[-- Attachment #6: 0003-DEBUG-Include-upstream-bridge-information.patch --]
[-- Type: text/plain, Size: 1679 bytes --]

>From fb4cf64dcced21b1f98b1e60fbb9dfa70f61fc3c Mon Sep 17 00:00:00 2001
From: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
Date: Tue, 4 Feb 2014 17:01:42 -0500
Subject: [PATCH 3/6] DEBUG: Include upstream bridge information.

Signed-off-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
---
 xen/drivers/passthrough/pci.c | 12 +++++++++++-
 1 file changed, 11 insertions(+), 1 deletion(-)

diff --git a/xen/drivers/passthrough/pci.c b/xen/drivers/passthrough/pci.c
index 1ad4f17..2a6eaa4 100644
--- a/xen/drivers/passthrough/pci.c
+++ b/xen/drivers/passthrough/pci.c
@@ -950,6 +950,9 @@ static int _dump_pci_devices(struct pci_seg *pseg, void *arg)
     {
         int id = pci_conf_read32(pseg->nr, pdev->bus, PCI_SLOT(pdev->devfn),
                                  PCI_FUNC(pdev->devfn), 0);
+        int rc = 0;
+        u8 bus, devfn, secbus;
+
         printk("%04x:%02x:%02x.%u (%04x:%04x)- dom %-3d - MSIs < ",
                pseg->nr, pdev->bus,
                PCI_SLOT(pdev->devfn), PCI_FUNC(pdev->devfn),
@@ -957,7 +960,14 @@ static int _dump_pci_devices(struct pci_seg *pseg, void *arg)
                pdev->domain ? pdev->domain->domain_id : -1);
         list_for_each_entry ( msi, &pdev->msi_list, list )
                printk("%d ", msi->irq);
-        printk(">\n");
+        bus = pdev->bus;
+        devfn = pdev->devfn;
+
+        rc = find_upstream_bridge( pseg->nr, &bus, &devfn, &secbus );
+        if ( rc < 0)
+            printk(">\n");
+        else
+            printk(">[%02x:%02x.%u]\n", bus, PCI_SLOT(devfn), PCI_FUNC(devfn));
     }
     printk("==== Bus2Bridge %04x ====\n", pseg->nr);
     spin_lock(&pseg->bus2bridge_lock);
-- 
1.8.3.1


[-- Attachment #7: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel