[PATCH][VTD] enabling PCI ACS P2P upstream forwarding

[PATCH][VTD] enabling PCI ACS P2P upstream forwarding

[Kay, Allen M]

[-- Attachment #1: Type: text/plain, Size: 1112 bytes --]

This patch enables P2P upstream forwarding in ACS capable PCIe switches.  The enabling is conditioned on iommu_enabled variable.  This code solves two potential problems in virtualization environment where a PCIe device is assigned to a guest domain using a HW iommu such as VT-d:

1) Unintentional failure caused by guest physical address programmed into the device's DMA that happens to match the memory address range of other downstream ports in the same PCIe switch.  This causes the PCI transaction to go to the matching downstream port instead of go to the root complex to get translated by VT-d as it should be.

2) Malicious guest software intentionally attacks another downstream PCIe device by programming the DMA address into the assigned device that matches memory address range of the downstream PCIe port.

Corresponding ACS filtering code is already in upstream control panel code that do not allow PCI device passthrough to guests if it is behind a PCIe switch that does not have ACS capability or with ACS capability but is not enabled. 

Signed-off-by: Allen Kay allen.m.kay@intel.com

[-- Attachment #2: acs_xen1117.patch --]
[-- Type: application/octet-stream, Size: 4104 bytes --]

diff -r c6c51f99768a xen/arch/ia64/xen/pci.c
--- a/xen/arch/ia64/xen/pci.c	Tue Nov 17 13:07:16 2009 +0000
+++ b/xen/arch/ia64/xen/pci.c	Tue Nov 17 13:36:59 2009 -0800
@@ -132,3 +132,8 @@
     BUG_ON((bus > 255) || (dev > 31) || (func > 7) || (reg > 255));
     pci_sal_write(0, bus, (dev<<3)|func, reg, 4, data);
 }
+
+int pci_find_ext_capability(int seg, int bus, int devfn, int cap)
+{
+    return 0;
+}
diff -r c6c51f99768a xen/drivers/passthrough/pci.c
--- a/xen/drivers/passthrough/pci.c	Tue Nov 17 13:07:16 2009 +0000
+++ b/xen/drivers/passthrough/pci.c	Tue Nov 17 13:36:59 2009 -0800
@@ -100,6 +100,45 @@
     return NULL;
 }
 
+/**
+ * pci_enable_acs - enable ACS if hardware support it
+ * @dev: the PCI device
+ */
+void pci_enable_acs(struct pci_dev *pdev)
+{
+    int pos;
+    u16 cap;
+    u16 ctrl;
+
+    u8 bus = pdev->bus;
+    u8 dev = PCI_SLOT(pdev->devfn);
+    u8 func = PCI_FUNC(pdev->devfn);
+
+    if ( !iommu_enabled )
+        return;
+
+    pos = pci_find_ext_capability(0, bus, pdev->devfn, PCI_EXT_CAP_ID_ACS);
+    if (!pos)
+        return;
+
+    cap = pci_conf_read16(bus, dev, func, pos + PCI_ACS_CAP);
+    ctrl = pci_conf_read16(bus, dev, func, pos + PCI_ACS_CTRL);
+
+    /* Source Validation */
+    ctrl |= (cap & PCI_ACS_SV);
+
+    /* P2P Request Redirect */
+    ctrl |= (cap & PCI_ACS_RR);
+
+    /* P2P Completion Redirect */
+    ctrl |= (cap & PCI_ACS_CR);
+
+    /* Upstream Forwarding */
+    ctrl |= (cap & PCI_ACS_UF);
+
+    pci_conf_write16(bus, dev, func, pos + PCI_ACS_CTRL, ctrl);
+}
+
 int pci_add_device(u8 bus, u8 devfn)
 {
     struct pci_dev *pdev;
@@ -119,6 +158,7 @@
             goto out;
 
         list_add(&pdev->domain_list, &dom0->arch.pdev_list);
+        pci_enable_acs(pdev);
     }
 
 out:
diff -r c6c51f99768a xen/drivers/passthrough/vtd/iommu.c
--- a/xen/drivers/passthrough/vtd/iommu.c	Tue Nov 17 13:07:16 2009 +0000
+++ b/xen/drivers/passthrough/vtd/iommu.c	Tue Nov 17 13:36:59 2009 -0800
@@ -1628,6 +1628,7 @@
             pdev->domain = d;
             list_add(&pdev->domain_list, &d->arch.pdev_list);
             domain_context_mapping(d, pdev->bus, pdev->devfn);
+            pci_enable_acs(pdev);
             if ( ats_device(0, pdev->bus, pdev->devfn) )
                 enable_ats_device(0, pdev->bus, pdev->devfn);
         }
diff -r c6c51f99768a xen/include/xen/pci.h
--- a/xen/include/xen/pci.h	Tue Nov 17 13:07:16 2009 +0000
+++ b/xen/include/xen/pci.h	Tue Nov 17 13:36:59 2009 -0800
@@ -116,5 +116,6 @@
 
 int msixtbl_pt_register(struct domain *d, int pirq, uint64_t gtable);
 void msixtbl_pt_unregister(struct domain *d, int pirq);
+void pci_enable_acs(struct pci_dev *pdev);
 
 #endif /* __XEN_PCI_H__ */
diff -r c6c51f99768a xen/include/xen/pci_regs.h
--- a/xen/include/xen/pci_regs.h	Tue Nov 17 13:07:16 2009 +0000
+++ b/xen/include/xen/pci_regs.h	Tue Nov 17 13:36:59 2009 -0800
@@ -419,9 +419,10 @@
 #define PCI_EXT_CAP_ID_VC	2
 #define PCI_EXT_CAP_ID_DSN	3
 #define PCI_EXT_CAP_ID_PWR	4
-#define PCI_EXT_CAP_ID_ARI	0xE
-#define PCI_EXT_CAP_ID_ATS	0xF
-#define PCI_EXT_CAP_ID_IOV	0x10
+#define PCI_EXT_CAP_ID_ACS	13
+#define PCI_EXT_CAP_ID_ARI	14
+#define PCI_EXT_CAP_ID_ATS	15
+#define PCI_EXT_CAP_ID_IOV	16
 
 /* Advanced Error Reporting */
 #define PCI_ERR_UNCOR_STATUS	4	/* Uncorrectable Error Status */
@@ -529,5 +530,16 @@
 #define HT_CAPTYPE_GEN3		0xD0	/* Generation 3 hypertransport configuration */
 #define HT_CAPTYPE_PM		0xE0	/* Hypertransport powermanagement configuration */
 
+/* Access Control Service */
+#define PCI_ACS_CAP		0x04	/* ACS Capability Register */
+#define  PCI_ACS_SV		0x01	/* Source Validation */
+#define  PCI_ACS_TB		0x02	/* Translation Blocking */
+#define  PCI_ACS_RR		0x04	/* P2P Request Redirect */
+#define  PCI_ACS_CR		0x08	/* P2P Completion Redirect */
+#define  PCI_ACS_UF		0x10	/* Upstream Forwarding */
+#define  PCI_ACS_EC		0x20	/* P2P Egress Control */
+#define  PCI_ACS_DT		0x40	/* Direct Translated P2P */
+#define PCI_ACS_CTRL		0x06	/* ACS Control Register */
+#define PCI_ACS_EGRESS_CTL_V	0x08	/* ACS Egress Control Vector */
 
 #endif /* LINUX_PCI_REGS_H */

[-- Attachment #3: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: [PATCH][VTD] enabling PCI ACS P2P upstream forwarding

[Jan Beulich]

I'm pretty certain a call to pci_enable_acs() would also be needed from
pci_add_device_ext()...

Jan

PS: Generating patches with -p makes it easier for reviewers to identfiy
where hunks belong.

>>> "Kay, Allen M" <allen.m.kay@intel.com> 17.11.09 22:49 >>>
This patch enables P2P upstream forwarding in ACS capable PCIe switches.  The enabling is conditioned on iommu_enabled variable.  This code solves two potential problems in virtualization environment where a PCIe device is assigned to a guest domain using a HW iommu such as VT-d:

1) Unintentional failure caused by guest physical address programmed into the device's DMA that happens to match the memory address range of other downstream ports in the same PCIe switch.  This causes the PCI transaction to go to the matching downstream port instead of go to the root complex to get translated by VT-d as it should be.

2) Malicious guest software intentionally attacks another downstream PCIe device by programming the DMA address into the assigned device that matches memory address range of the downstream PCIe port.

Corresponding ACS filtering code is already in upstream control panel code that do not allow PCI device passthrough to guests if it is behind a PCIe switch that does not have ACS capability or with ACS capability but is not enabled. 

Signed-off-by: Allen Kay allen.m.kay@intel.com

Re: [PATCH][VTD] enabling PCI ACS P2P upstream forwarding

[Konrad Rzeszutek Wilk]

On Tue, Nov 17, 2009 at 01:49:09PM -0800, Kay, Allen M wrote:
> This patch enables P2P upstream forwarding in ACS capable PCIe switches.  The enabling is conditioned on iommu_enabled variable.  This code solves two potential problems in virtualization environment where a PCIe device is assigned to a guest domain using a HW iommu such as VT-d:
> 
> 1) Unintentional failure caused by guest physical address programmed into the device's DMA that happens to match the memory address range of other downstream ports in the same PCIe switch.  This causes the PCI transaction to go to the matching downstream port instead of go to the root complex to get translated by VT-d as it should be.
> 
> 2) Malicious guest software intentionally attacks another downstream PCIe device by programming the DMA address into the assigned device that matches memory address range of the downstream PCIe port.
> 
> Corresponding ACS filtering code is already in upstream control panel code that do not allow PCI device passthrough to guests if it is behind a PCIe switch that does not have ACS capability or with ACS capability but is not enabled. 

Based on your description it sounds like the function should be called: pci_reset_acs.
Should there be a corresponding function to disable the P2P upstream forwarding?

RE: [PATCH][VTD] enabling PCI ACS P2P upstream forwarding

[Kay, Allen M]

> Based on your description it sounds like the function should be called: pci_reset_acs.

I'm not following ...  The patch is for enabling ACS capability.

> Should there be a corresponding function to disable the P2P upstream forwarding?

No, ACS is enabled during system boot phase - similar to other PCI capabilities.

-----Original Message-----
From: Konrad Rzeszutek Wilk [mailto:konrad.wilk@oracle.com] 
Sent: Wednesday, November 18, 2009 6:54 AM
To: Kay, Allen M
Cc: 'xen-devel@lists.xensource.com'; Dugger, Donald D; Jan Beulich; 'keir.fraser@eu.citrix.com'
Subject: Re: [Xen-devel] [PATCH][VTD] enabling PCI ACS P2P upstream forwarding

On Tue, Nov 17, 2009 at 01:49:09PM -0800, Kay, Allen M wrote:
> This patch enables P2P upstream forwarding in ACS capable PCIe switches.  The enabling is conditioned on iommu_enabled variable.  This code solves two potential problems in virtualization environment where a PCIe device is assigned to a guest domain using a HW iommu such as VT-d:
> 
> 1) Unintentional failure caused by guest physical address programmed into the device's DMA that happens to match the memory address range of other downstream ports in the same PCIe switch.  This causes the PCI transaction to go to the matching downstream port instead of go to the root complex to get translated by VT-d as it should be.
> 
> 2) Malicious guest software intentionally attacks another downstream PCIe device by programming the DMA address into the assigned device that matches memory address range of the downstream PCIe port.
> 
> Corresponding ACS filtering code is already in upstream control panel code that do not allow PCI device passthrough to guests if it is behind a PCIe switch that does not have ACS capability or with ACS capability but is not enabled. 

Based on your description it sounds like the function should be called: pci_reset_acs.
Should there be a corresponding function to disable the P2P upstream forwarding?