[PATCH v4 00/12] DMA-API/PCI map_peer_resource support for peer-to-peer

[PATCH v4 00/12] DMA-API/PCI map_peer_resource support for peer-to-peer

[Will Davis]

Hi,

This is the fourth version of a patchset to add the DMA APIs necessary to
map and unmap a PCI device's BAR to and from another PCI device's IOVA
domain. This enables PCI peer-to-peer traffic on x86 platforms where an
IOMMU is enabled (this requires that peer-to-peer traffic be supported by
the underlying chipset as well, of course).

Thanks,
Will

v4:
- added HAS_DMA_P2P Kconfig option to guard new APIs, which are only
  supported on x86 with this patch series
- added a new dma_peer_addr_t type to handle future platform support
  where the dma_peer_addr_t needs to have the same width as the bus
  address, which can be wider than the dma_addr_t returned by other
  DMA APIs
- added a new dma_peer_mapping_error() and associated PCI wrapper to
  detect errors returned via dma_map_peer_resource() API and its
  PCI wrapper
- modified the interface of the new API to explicitly take the peer
  device as an argument
- renamed the (un)map_resource APIs to (un)map_peer_resource, since
  they directly receive a 'peer' argument now
- modified the new PCI interface to take the BAR index instead of the
  actual struct resource
- modified the pci-nommu implementation to perform topology and ACS
  checks before allowing
- modified both the AMD and Intel IOMMU implementations to require
  that both devices be behind the same host bridge
- exposed a couple of preexisting PCI utility functions to aid in the
  topology checks for all of the x86 implementations

v3: http://www.spinics.net/lists/linux-pci/msg41703.html
- changed dma_map_resource() to BUG() if the map_resource DMA op is not
  provided, instead of returning 0
- updated documentation to remove requirement to check for 0 return value
  as an error
- remove const keyword from struct dma_map_ops in new DMA APIs for
  consistency with other APIs

v2: http://www.spinics.net/lists/linux-pci/msg41192.html
- added documentation for the new DMA APIs
- fixed physical-to-bus address conversion in the nommu implementation

v1: http://www.spinics.net/lists/linux-pci/msg40747.html



Will Davis (12):
  lib/Kconfig: add HAS_DMA_P2P for peer-to-peer support
  linux/types.h: Add dma_peer_addr_t type
  dma-debug: add checking for map/unmap_peer_resource
  DMA-API: Introduce dma_(un)map_peer_resource
  dma-mapping: pci: add pci_(un)map_peer_resource
  DMA-API: Add peer resource mapping documentation
  pci: expose pci_find_host_bridge()
  iommu/amd: Implement (un)map_peer_resource
  iommu/vt-d: implement (un)map_peer_resource
  pci: add pci_find_common_upstream_dev()
  x86: add pci-nommu implementation of map_peer_resource
  x86: declare support for DMA P2P

 Documentation/DMA-API-HOWTO.txt          | 46 ++++++++++++++-
 Documentation/DMA-API.txt                | 36 ++++++++++--
 arch/x86/Kconfig                         |  1 +
 arch/x86/include/asm/dma-mapping.h       | 18 +++++-
 arch/x86/kernel/pci-nommu.c              | 96 +++++++++++++++++++++++++++++++
 drivers/iommu/amd_iommu.c                | 99 +++++++++++++++++++++++++++-----
 drivers/iommu/intel-iommu.c              | 38 ++++++++++++
 drivers/pci/host-bridge.c                | 19 +-----
 drivers/pci/search.c                     | 25 ++++++++
 include/asm-generic/dma-mapping-common.h | 43 ++++++++++++++
 include/asm-generic/pci-dma-compat.h     | 23 ++++++++
 include/linux/dma-debug.h                | 39 ++++++++++++-
 include/linux/dma-mapping.h              | 12 ++++
 include/linux/pci.h                      |  9 +++
 include/linux/types.h                    | 13 ++++-
 lib/Kconfig                              |  5 ++
 lib/dma-debug.c                          | 86 ++++++++++++++++++++++++++-
 17 files changed, 564 insertions(+), 44 deletions(-)

-- 
2.4.6

[PATCH v4 01/12] lib/Kconfig: add HAS_DMA_P2P for peer-to-peer support

[Will Davis]

This option is dependent on HAS_DMA, and indicates whether or not the DMA
API should include the peer resource (P2P) interfaces.

Signed-off-by: Will Davis <wdavis@nvidia.com>
---
 lib/Kconfig | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/lib/Kconfig b/lib/Kconfig
index 3a2ef67..0b670ab 100644
--- a/lib/Kconfig
+++ b/lib/Kconfig
@@ -392,6 +392,11 @@ config HAS_DMA
 	depends on !NO_DMA
 	default y
 
+config HAS_DMA_P2P
+	bool
+	depends on HAS_DMA
+	default n
+
 config CHECK_SIGNATURE
 	bool
 
-- 
2.4.6

[PATCH v4 02/12] linux/types.h: Add dma_peer_addr_t type

[Will Davis]

On some platforms, a dma_addr_t is only 32 bits wide, but the bus addresses
can be 64 bits wide. For peer-to-peer DMA, a device needs to use the full
64-bit address. Therefore, introduce a DMA address type that is not
constrained by the CPU DMA address size.

This is only needed for peer-to-peer DMA, so guard this definition with a
CONFIG_HAS_DMA_P2P check.

Signed-off-by: Will Davis <wdavis@nvidia.com>
---
 include/linux/types.h | 13 +++++++++++--
 1 file changed, 11 insertions(+), 2 deletions(-)

diff --git a/include/linux/types.h b/include/linux/types.h
index 8715287..dd85173 100644
--- a/include/linux/types.h
+++ b/include/linux/types.h
@@ -140,13 +140,13 @@ typedef unsigned long blkcnt_t;
 #define pgoff_t unsigned long
 
 /*
- * A dma_addr_t can hold any valid DMA address, i.e., any address returned
+ * A dma_addr_t can hold any valid DMA address, i.e., most addresses returned
  * by the DMA API.
  *
  * If the DMA API only uses 32-bit addresses, dma_addr_t need only be 32
  * bits wide.  Bus addresses, e.g., PCI BARs, may be wider than 32 bits,
  * but drivers do memory-mapped I/O to ioremapped kernel virtual addresses,
- * so they don't care about the size of the actual bus addresses.
+ * so they don't care about the size of the actual bus addresses in most cases.
  */
 #ifdef CONFIG_ARCH_DMA_ADDR_T_64BIT
 typedef u64 dma_addr_t;
@@ -154,6 +154,15 @@ typedef u64 dma_addr_t;
 typedef u32 dma_addr_t;
 #endif
 
+/*
+ * Unlike dma_addr_t, a dma_peer_addr_t can hold any bus address for the
+ * platform. Drivers should only need this for peer-to-peer use cases, where
+ * the DMA target is also on the bus.
+ */
+#ifdef CONFIG_HAS_DMA_P2P
+typedef u64 dma_peer_addr_t;
+#endif
+
 typedef unsigned __bitwise__ gfp_t;
 typedef unsigned __bitwise__ fmode_t;
 typedef unsigned __bitwise__ oom_flags_t;
-- 
2.4.6

[PATCH v4 03/12] dma-debug: add checking for map/unmap_peer_resource

[Will Davis]

Add debug callbacks for the new dma_map_peer_resource and
dma_unmap_peer_resource functions.

Also add a callback for dma_peer_mapping_error. Most of the time this will
be identical to dma_mapping_error, except where the dma_addr_t is not as
wide as dma_peer_addr_t.

Guard these new callbacks behind a CONFIG_HAS_DMA_P2P check.

Signed-off-by: Will Davis <wdavis@nvidia.com>
Reviewed-by: Terence Ripperda <tripperda@nvidia.com>
Reviewed-by: John Hubbard <jhubbard@nvidia.com>
---
 include/linux/dma-debug.h | 39 ++++++++++++++++++++-
 lib/dma-debug.c           | 86 +++++++++++++++++++++++++++++++++++++++++++++--
 2 files changed, 122 insertions(+), 3 deletions(-)

diff --git a/include/linux/dma-debug.h b/include/linux/dma-debug.h
index fe8cb61..4ff8364 100644
--- a/include/linux/dma-debug.h
+++ b/include/linux/dma-debug.h
@@ -43,7 +43,21 @@ extern void debug_dma_mapping_error(struct device *dev, dma_addr_t dma_addr);
 
 extern void debug_dma_unmap_page(struct device *dev, dma_addr_t addr,
 				 size_t size, int direction, bool map_single);
-
+#ifdef CONFIG_HAS_DMA_P2P
+extern void debug_dma_map_peer_resource(struct device *dev,
+					struct device *peer,
+					struct resource *resource,
+					size_t offset, size_t size,
+					int direction,
+					dma_peer_addr_t dma_addr);
+
+extern void debug_dma_peer_mapping_error(struct device *dev,
+					 dma_peer_addr_t dma_addr);
+
+extern void debug_dma_unmap_peer_resource(struct device *dev,
+					  dma_peer_addr_t addr,
+					  size_t size, int direction);
+#endif
 extern void debug_dma_map_sg(struct device *dev, struct scatterlist *sg,
 			     int nents, int mapped_ents, int direction);
 
@@ -120,6 +134,29 @@ static inline void debug_dma_unmap_page(struct device *dev, dma_addr_t addr,
 {
 }
 
+#ifdef CONFIG_HAS_DMA_P2P
+static inline void debug_dma_map_peer_resource(struct device *dev,
+					       struct device *peer,
+					       struct resource *resource,
+					       size_t offset, size_t size,
+					       int direction,
+					       dma_peer_addr_t dma_addr)
+{
+}
+
+static inline void debug_dma_peer_mapping_error(struct device *dev,
+						dma_peer_addr_t dma_addr)
+{
+}
+
+static inline void debug_dma_unmap_peer_resource(struct device *dev,
+						 dma_peer_addr_t addr,
+						 size_t size,
+						 int direction)
+{
+}
+#endif
+
 static inline void debug_dma_map_sg(struct device *dev, struct scatterlist *sg,
 				    int nents, int mapped_ents, int direction)
 {
diff --git a/lib/dma-debug.c b/lib/dma-debug.c
index ae4b65e..622f569 100644
--- a/lib/dma-debug.c
+++ b/lib/dma-debug.c
@@ -43,6 +43,7 @@ enum {
 	dma_debug_page,
 	dma_debug_sg,
 	dma_debug_coherent,
+	dma_debug_peer_resource,
 };
 
 enum map_err_types {
@@ -64,7 +65,8 @@ enum map_err_types {
  * @direction: enum dma_data_direction
  * @sg_call_ents: 'nents' from dma_map_sg
  * @sg_mapped_ents: 'mapped_ents' from dma_map_sg
- * @map_err_type: track whether dma_mapping_error() was checked
+ * @map_err_type: track whether dma_mapping_error() or dma_peer_mapping_error()
+ *		  was called
  * @stacktrace: support backtraces when a violation is detected
  */
 struct dma_debug_entry {
@@ -1250,6 +1252,23 @@ out:
 	put_hash_bucket(bucket, &flags);
 }
 
+static void check_mapping_error(struct dma_debug_entry *entry, bool peer)
+{
+	if (peer && !(entry->type == dma_debug_peer_resource))
+		err_printk(entry->dev, entry, "DMA-API: device driver calls "
+				"dma_peer_mapping_error() on dma_addr_t "
+				"[device address=0x%016llx] [size=%llu bytes]\n",
+				(unsigned long long)entry->dev_addr,
+				entry->size);
+
+	if (!peer && (entry->type == dma_debug_peer_resource))
+		err_printk(entry->dev, entry, "DMA-API: device driver calls "
+				"dma_mapping_error() on dma_peer_addr_t "
+				"[device address=0x%016llx] [size=%llu bytes]\n",
+				(unsigned long long)entry->dev_addr,
+				entry->size);
+}
+
 void debug_dma_map_page(struct device *dev, struct page *page, size_t offset,
 			size_t size, int direction, dma_addr_t dma_addr,
 			bool map_single)
@@ -1289,7 +1308,9 @@ void debug_dma_map_page(struct device *dev, struct page *page, size_t offset,
 }
 EXPORT_SYMBOL(debug_dma_map_page);
 
-void debug_dma_mapping_error(struct device *dev, dma_addr_t dma_addr)
+static void debug_dma_mapping_error_common(struct device *dev,
+					   u64 dma_addr,
+					   bool peer)
 {
 	struct dma_debug_entry ref;
 	struct dma_debug_entry *entry;
@@ -1319,12 +1340,18 @@ void debug_dma_mapping_error(struct device *dev, dma_addr_t dma_addr)
 		 */
 		if (entry->map_err_type == MAP_ERR_NOT_CHECKED) {
 			entry->map_err_type = MAP_ERR_CHECKED;
+			check_mapping_error(entry, peer);
 			break;
 		}
 	}
 
 	put_hash_bucket(bucket, &flags);
 }
+
+void debug_dma_mapping_error(struct device *dev, dma_addr_t dma_addr)
+{
+	debug_dma_mapping_error_common(dev, (u64)dma_addr, false);
+}
 EXPORT_SYMBOL(debug_dma_mapping_error);
 
 void debug_dma_unmap_page(struct device *dev, dma_addr_t addr,
@@ -1348,6 +1375,61 @@ void debug_dma_unmap_page(struct device *dev, dma_addr_t addr,
 }
 EXPORT_SYMBOL(debug_dma_unmap_page);
 
+#ifdef CONFIG_HAS_DMA_P2P
+void debug_dma_map_peer_resource(struct device *dev, struct device *peer,
+				 struct resource *resource, size_t offset,
+				 size_t size, int direction,
+				 dma_peer_addr_t dma_addr)
+{
+	struct dma_debug_entry *entry;
+
+	if (unlikely(dma_debug_disabled()))
+		return;
+
+	if (dma_mapping_error(dev, dma_addr))
+		return;
+
+	entry = dma_entry_alloc();
+	if (!entry)
+		return;
+
+	entry->dev       = dev;
+	entry->type      = dma_debug_peer_resource;
+	entry->pfn       = resource->start >> PAGE_SHIFT;
+	entry->offset    = offset,
+	entry->dev_addr  = dma_addr;
+	entry->size      = size;
+	entry->direction = direction;
+
+	add_dma_entry(entry);
+}
+EXPORT_SYMBOL(debug_dma_map_peer_resource);
+
+void debug_dma_peer_mapping_error(struct device *dev, dma_peer_addr_t dma_addr)
+{
+	debug_dma_mapping_error_common(dev, (u64)dma_addr, true);
+}
+EXPORT_SYMBOL(debug_dma_peer_mapping_error);
+
+void debug_dma_unmap_peer_resource(struct device *dev, dma_peer_addr_t addr,
+				   size_t size, int direction)
+{
+	struct dma_debug_entry ref = {
+		.type           = dma_debug_peer_resource,
+		.dev            = dev,
+		.dev_addr       = addr,
+		.size           = size,
+		.direction      = direction,
+	};
+
+	if (unlikely(dma_debug_disabled()))
+		return;
+
+	check_unmap(&ref);
+}
+EXPORT_SYMBOL(debug_dma_unmap_peer_resource);
+#endif
+
 void debug_dma_map_sg(struct device *dev, struct scatterlist *sg,
 		      int nents, int mapped_ents, int direction)
 {
-- 
2.4.6

[PATCH v4 04/12] DMA-API: Introduce dma_(un)map_peer_resource

[Will Davis]

Add functions to DMA-map and -unmap a peer device's resource for a given
device. This will allow devices to DMA-map, for example, another device's
BAR region on PCI to enable peer-to-peer transactions.

Guard these new functions behind CONFIG_HAS_DMA_P2P.

Signed-off-by: Will Davis <wdavis@nvidia.com>
---
 include/asm-generic/dma-mapping-common.h | 43 ++++++++++++++++++++++++++++++++
 include/linux/dma-mapping.h              | 12 +++++++++
 2 files changed, 55 insertions(+)

diff --git a/include/asm-generic/dma-mapping-common.h b/include/asm-generic/dma-mapping-common.h
index 940d5ec..45eec17 100644
--- a/include/asm-generic/dma-mapping-common.h
+++ b/include/asm-generic/dma-mapping-common.h
@@ -73,6 +73,42 @@ static inline void dma_unmap_sg_attrs(struct device *dev, struct scatterlist *sg
 		ops->unmap_sg(dev, sg, nents, dir, attrs);
 }
 
+#ifdef CONFIG_HAS_DMA_P2P
+static inline dma_peer_addr_t dma_map_peer_resource_attrs(struct device *dev,
+							  struct device *peer,
+							  struct resource *res,
+							  size_t offset,
+							  size_t size,
+							  enum dma_data_direction dir,
+							  struct dma_attrs *attrs)
+{
+	struct dma_map_ops *ops = get_dma_ops(dev);
+	dma_peer_addr_t addr;
+
+	BUG_ON(!valid_dma_direction(dir));
+	BUG_ON(ops->map_peer_resource == NULL);
+	addr = ops->map_peer_resource(dev, peer, res, offset, size, dir,
+				      attrs);
+	debug_dma_map_peer_resource(dev, peer, res, offset, size, dir, addr);
+
+	return addr;
+}
+
+static inline void dma_unmap_peer_resource_attrs(struct device *dev,
+						 dma_peer_addr_t addr,
+						 size_t size,
+						 enum dma_data_direction dir,
+						 struct dma_attrs *attrs)
+{
+	struct dma_map_ops *ops = get_dma_ops(dev);
+
+	BUG_ON(!valid_dma_direction(dir));
+	if (ops->unmap_peer_resource)
+		ops->unmap_peer_resource(dev, addr, size, dir, attrs);
+	debug_dma_unmap_peer_resource(dev, addr, size, dir);
+}
+#endif
+
 static inline dma_addr_t dma_map_page(struct device *dev, struct page *page,
 				      size_t offset, size_t size,
 				      enum dma_data_direction dir)
@@ -181,6 +217,13 @@ dma_sync_sg_for_device(struct device *dev, struct scatterlist *sg,
 #define dma_map_sg(d, s, n, r) dma_map_sg_attrs(d, s, n, r, NULL)
 #define dma_unmap_sg(d, s, n, r) dma_unmap_sg_attrs(d, s, n, r, NULL)
 
+#ifdef CONFIG_HAS_DMA_P2P
+#define dma_map_peer_resource(d, p, e, o, s, r) \
+	dma_map_peer_resource_attrs(d, p, e, o, s, r, NULL)
+#define dma_unmap_peer_resource(d, a, s, r) \
+	dma_unmap_peer_resource_attrs(d, a, s, r, NULL)
+#endif
+
 extern int dma_common_mmap(struct device *dev, struct vm_area_struct *vma,
 			   void *cpu_addr, dma_addr_t dma_addr, size_t size);
 
diff --git a/include/linux/dma-mapping.h b/include/linux/dma-mapping.h
index ac07ff0..7b8fddc 100644
--- a/include/linux/dma-mapping.h
+++ b/include/linux/dma-mapping.h
@@ -34,6 +34,18 @@ struct dma_map_ops {
 	void (*unmap_page)(struct device *dev, dma_addr_t dma_handle,
 			   size_t size, enum dma_data_direction dir,
 			   struct dma_attrs *attrs);
+#ifdef CONFIG_HAS_DMA_P2P
+	dma_peer_addr_t (*map_peer_resource)(struct device *dev,
+					     struct device *peer,
+					     struct resource *res,
+					     unsigned long offset, size_t size,
+					     enum dma_data_direction dir,
+					     struct dma_attrs *attrs);
+	void (*unmap_peer_resource)(struct device *dev,
+				    dma_peer_addr_t dma_handle,
+				    size_t size, enum dma_data_direction dir,
+				    struct dma_attrs *attrs);
+#endif
 	/*
 	 * map_sg returns 0 on error and a value > 0 on success.
 	 * It should never return a value < 0.
-- 
2.4.6

[PATCH v4 05/12] dma-mapping: pci: add pci_(un)map_peer_resource

[Will Davis]

Simply route these through to the new dma_(un)map_peer_resource APIs.

Also route pci_peer_mapping_error through to the new dma_peer_mapping_error
API.

Guard these new functions behind CONFIG_HAS_DMA_P2P.

Signed-off-by: Will Davis <wdavis@nvidia.com>
Reviewed-by: Terence Ripperda <tripperda@nvidia.com>
Reviewed-by: John Hubbard <jhubbard@nvidia.com>
---
 include/asm-generic/pci-dma-compat.h | 23 +++++++++++++++++++++++
 1 file changed, 23 insertions(+)

diff --git a/include/asm-generic/pci-dma-compat.h b/include/asm-generic/pci-dma-compat.h
index c110843..7baf09e 100644
--- a/include/asm-generic/pci-dma-compat.h
+++ b/include/asm-generic/pci-dma-compat.h
@@ -61,6 +61,29 @@ pci_unmap_page(struct pci_dev *hwdev, dma_addr_t dma_address,
 	dma_unmap_page(hwdev == NULL ? NULL : &hwdev->dev, dma_address, size, (enum dma_data_direction)direction);
 }
 
+#ifdef CONFIG_HAS_DMA_P2P
+static inline dma_peer_addr_t
+pci_map_peer_resource(struct pci_dev *hwdev, struct pci_dev *peer,
+		      int res, unsigned long offset, size_t size,
+		      int direction)
+{
+	return dma_map_peer_resource(hwdev == NULL ? NULL : &hwdev->dev, peer == NULL ? NULL : &peer->dev, peer == NULL ? NULL : &peer->resource[res], offset, size, (enum dma_data_direction)direction);
+}
+
+static inline int
+pci_dma_peer_mapping_error(struct pci_dev *pdev, dma_peer_addr_t dma_addr)
+{
+	return dma_peer_mapping_error(&pdev->dev, dma_addr);
+}
+
+static inline void
+pci_unmap_peer_resource(struct pci_dev *hwdev, dma_peer_addr_t dma_address,
+			size_t size, int direction)
+{
+	dma_unmap_peer_resource(hwdev == NULL ? NULL : &hwdev->dev, dma_address, size, (enum dma_data_direction)direction);
+}
+#endif
+
 static inline int
 pci_map_sg(struct pci_dev *hwdev, struct scatterlist *sg,
 	   int nents, int direction)
-- 
2.4.6

[PATCH v4 06/12] DMA-API: Add peer resource mapping documentation

[Will Davis]

Add references to both the general API documentation, as well as the HOWTO,
about the dma_(un)map_peer_resource() and dma_peer_mapping_error() APIs.

Signed-off-by: Will Davis <wdavis@nvidia.com>
---
 Documentation/DMA-API-HOWTO.txt | 69 +++++++++++++++++++++++++++++++++++++++++
 Documentation/DMA-API.txt       | 38 ++++++++++++++++++++---
 2 files changed, 102 insertions(+), 5 deletions(-)

diff --git a/Documentation/DMA-API-HOWTO.txt b/Documentation/DMA-API-HOWTO.txt
index 55b70b9..053a18f 100644
--- a/Documentation/DMA-API-HOWTO.txt
+++ b/Documentation/DMA-API-HOWTO.txt
@@ -143,6 +143,10 @@ What about block I/O and networking buffers?  The block I/O and
 networking subsystems make sure that the buffers they use are valid
 for you to DMA from/to.
 
+In some systems, it may also be possible to DMA to and/or from a peer
+device's MMIO region, as described by a 'struct resource'. This is
+referred to as a peer-to-peer mapping.
+
 			DMA addressing limitations
 
 Does your device have any DMA addressing limitations?  For example, is
@@ -653,6 +657,45 @@ Every dma_map_{single,sg}() call should have its dma_unmap_{single,sg}()
 counterpart, because the DMA address space is a shared resource and
 you could render the machine unusable by consuming all DMA addresses.
 
+Peer-to-peer DMA mappings can be obtained using dma_map_peer_resource() to
+map another device's MMIO region for the given device:
+
+	dma_peer_addr_t dma_handle = dma_map_peer_resource(dev, peer, res,
+							   offset, size,
+							   direction);
+	if (dma_peer_mapping_error(dev, dma_handle))
+	{
+		/*
+		 * reduce current DMA mapping usage,
+		 * delay and try again later or
+		 * reset driver. P2P DMA may not be
+		 * supported between the devices.
+		 */
+		goto map_error_handling;
+	}
+
+	...
+
+	dma_unmap_peer_resource(dev, dma_handle, size, direction);
+
+Here, "res" is the struct resource * of the MMIO region, and "offset" means
+byte offset within the given resource.
+
+You should call dma_peer_mapping_error() as dma_map_peer_resource() could fail
+and return error.
+
+Not all implementations support dma_map_peer_resource(), but those that do
+should also implement dma_peer_mapping_error(). dma_map_peer_resource() may
+fail for various reasons depending on the implementation, but generally
+may fail for the same reasons as those outlined under the dma_map_single()
+discussion. In addition, the success of dma_map_peer_resource() may depend
+on the physical topology between the two devices. For example, in the x86
+implementations, peer-to-peer mappings will fail if the devices are behind
+different host bridges.
+
+You should call dma_unmap_peer_resource() when DMA activity is finished,
+e.g., from the interrupt which told you that the DMA transfer is done.
+
 If you need to use the same streaming DMA region multiple times and touch
 the data in between the DMA transfers, the buffer needs to be synced
 properly in order for the CPU and device to see the most up-to-date and
@@ -785,6 +828,23 @@ failure can be determined by:
 		goto map_error_handling;
 	}
 
+- checking the dma_peer_addr_t returned from dma_map_peer_resource() by using
+  dma_peer_mapping_error():
+
+  	dma_peer_addr_t dma_handle;
+
+	dma_handle = dma_map_peer_resource(dev, peer, res, offset, size,
+					   direction);
+	if (dma_peer_mapping_error(dev, dma_handle)) {
+		/*
+		 * reduce current DMA mapping usage,
+		 * delay and try again later or
+		 * reset driver. P2P DMA may not be
+		 * supported between the devices.
+		 */
+		goto map_error_handling;
+	}
+
 - unmap pages that are already mapped, when mapping error occurs in the middle
   of a multiple page mapping attempt. These example are applicable to
   dma_map_page() as well.
@@ -956,6 +1016,15 @@ to "Closing".
    support multiple types of IOMMUs in a single system, the example of
    x86 or powerpc helps.
 
+4) Peer-to-peer requirements.
+
+   If your platform can support peer-to-peer DMA, enable HAS_DMA_P2P to
+   include the peer-resource mapping interfaces in the DMA API, and
+   provide an implementation of dma_peer_mapping_error(). The latter part
+   is necessary because on some platforms, the dma_addr_t is not as wide
+   as the full bus address, and therefore the existing dma_mapping_error()
+   API may not be sufficient.
+
 			   Closing
 
 This document, and the API itself, would not be in its current
diff --git a/Documentation/DMA-API.txt b/Documentation/DMA-API.txt
index 7eba542..c6f11a1 100644
--- a/Documentation/DMA-API.txt
+++ b/Documentation/DMA-API.txt
@@ -283,14 +283,42 @@ and <size> parameters are provided to do partial page mapping, it is
 recommended that you never use these unless you really know what the
 cache width is.
 
+dma_peer_addr_t
+dma_map_peer_resource(struct device *dev, struct device *peer,
+		      struct resource *res, unsigned long offset,
+		      size_t size, enum dma_data_direction_direction)
+
+API for mapping resources. This API allows a driver to map a peer
+device's resource for DMA. All the notes and warnings for the other
+APIs apply here. Also, the success of this API does not validate or
+guarantee that peer-to-peer transactions between the device and its
+peer will be functional. They only grant access so that if such
+transactions are possible, an IOMMU will not prevent them from
+succeeding. However, a failure of this API may indicate that peer-to-peer
+transactions are not expected to succeed across the underlying topology
+between the devices.
+
+void
+dma_unmap_peer_resource(struct device *dev, dma_peer_addr_t dma_address,
+			size_t size, enum dma_data_direction direction)
+
+Unmaps the resource previously mapped. All the parameters passed in
+must be identical to those passed in to (and returned by) the mapping
+API.
+
 int
 dma_mapping_error(struct device *dev, dma_addr_t dma_addr)
 
-In some circumstances dma_map_single() and dma_map_page() will fail to create
-a mapping. A driver can check for these errors by testing the returned
-DMA address with dma_mapping_error(). A non-zero return value means the mapping
-could not be created and the driver should take appropriate action (e.g.
-reduce current DMA mapping usage or delay and try again later).
+int
+dma_peer_mapping_error(struct device *dev, dma_peer_addr_t dma_addr)
+
+In some circumstances dma_map_single(), dma_map_page() and
+dma_map_peer_resource() will fail to create a mapping. A driver can check for
+these errors by testing the returned DMA address with dma_mapping_error()
+(or dma_peer_mapping_error() for dma_map_peer_resource()). A non-zero return
+value means the mapping could not be created and the driver should take
+appropriate action (e.g. reduce current DMA mapping usage or delay and try
+again later).
 
 	int
 	dma_map_sg(struct device *dev, struct scatterlist *sg,
-- 
2.4.6

[PATCH v4 07/12] pci: expose pci_find_host_bridge()

[Will Davis]

This function takes a struct pci_bus * and returns the associated struct
pci_host_bridge * upstream.

Signed-off-by: Will Davis <wdavis@nvidia.com>
---
 drivers/pci/host-bridge.c | 19 ++-----------------
 include/linux/pci.h       |  7 +++++++
 2 files changed, 9 insertions(+), 17 deletions(-)

diff --git a/drivers/pci/host-bridge.c b/drivers/pci/host-bridge.c
index 5f4a2e0..6c2c82a 100644
--- a/drivers/pci/host-bridge.c
+++ b/drivers/pci/host-bridge.c
@@ -8,25 +8,10 @@
 
 #include "pci.h"
 
-static struct pci_bus *find_pci_root_bus(struct pci_bus *bus)
-{
-	while (bus->parent)
-		bus = bus->parent;
-
-	return bus;
-}
-
-struct pci_host_bridge *pci_find_host_bridge(struct pci_bus *bus)
-{
-	struct pci_bus *root_bus = find_pci_root_bus(bus);
-
-	return to_pci_host_bridge(root_bus->bridge);
-}
-
 struct device *pci_get_host_bridge_device(struct pci_dev *dev)
 {
-	struct pci_bus *root_bus = find_pci_root_bus(dev->bus);
-	struct device *bridge = root_bus->bridge;
+	struct pci_host_bridge *host_bridge = pci_find_host_bridge(dev->bus);
+	struct device *bridge = &host_bridge->dev;
 
 	kobject_get(&bridge->kobj);
 	return bridge;
diff --git a/include/linux/pci.h b/include/linux/pci.h
index 8a0321a..d58063e 100644
--- a/include/linux/pci.h
+++ b/include/linux/pci.h
@@ -512,6 +512,13 @@ static inline struct pci_dev *pci_upstream_bridge(struct pci_dev *dev)
 	return dev->bus->self;
 }
 
+static inline struct pci_host_bridge *pci_find_host_bridge(struct pci_bus *bus)
+{
+	while (bus->parent)
+		bus = bus->parent;
+	return to_pci_host_bridge(bus->bridge);
+}
+
 struct device *pci_get_host_bridge_device(struct pci_dev *dev);
 void pci_put_host_bridge_device(struct device *dev);
 
-- 
2.4.6

Re: [PATCH v4 07/12] pci: expose pci_find_host_bridge()

[Bjorn Helgaas]

On Wed, Jul 22, 2015 at 04:39:52PM -0500, Will Davis wrote:
> This function takes a struct pci_bus * and returns the associated struct
> pci_host_bridge * upstream.

Nits: for the PCI parts, please run "git log --oneline" and make yours
match the style of the previous history.  Same for the changelog text
itself, which can be an imperative sentence, e.g., something like:

  Return the struct pci_host_bridge corresponding to the given pci_bus.

Text like "This function ..." or "This patch ..." is superfluous because we
already know the changelog applies to *this* patch.

Bjorn

[PATCH v4 08/12] iommu/amd: Implement (un)map_peer_resource

[Will Davis]

Implement 'map_peer_resource' for the AMD IOMMU driver. Generalize the
existing map_page implementation to operate on a physical address, and
make both map_page and map_resource wrappers around that helper (and
similarly, for unmap_page and unmap_resource).

This allows a device to map another's resource, to enable peer-to-peer
transactions.

This is guarded behind CONFIG_HAS_DMA_P2P, since the struct dma_map_ops
members are as well.

Signed-off-by: Will Davis <wdavis@nvidia.com>
Reviewed-by: Terence Ripperda <tripperda@nvidia.com>
Reviewed-by: John Hubbard <jhubbard@nvidia.com>
---
 drivers/iommu/amd_iommu.c | 99 ++++++++++++++++++++++++++++++++++++++++-------
 1 file changed, 86 insertions(+), 13 deletions(-)

diff --git a/drivers/iommu/amd_iommu.c b/drivers/iommu/amd_iommu.c
index a57e9b7..adf9496 100644
--- a/drivers/iommu/amd_iommu.c
+++ b/drivers/iommu/amd_iommu.c
@@ -471,6 +471,10 @@ DECLARE_STATS_COUNTER(cnt_map_single);
 DECLARE_STATS_COUNTER(cnt_unmap_single);
 DECLARE_STATS_COUNTER(cnt_map_sg);
 DECLARE_STATS_COUNTER(cnt_unmap_sg);
+#ifdef CONFIG_HAS_DMA_P2P
+DECLARE_STATS_COUNTER(cnt_map_peer_resource);
+DECLARE_STATS_COUNTER(cnt_unmap_peer_resource);
+#endif
 DECLARE_STATS_COUNTER(cnt_alloc_coherent);
 DECLARE_STATS_COUNTER(cnt_free_coherent);
 DECLARE_STATS_COUNTER(cross_page);
@@ -509,6 +513,10 @@ static void amd_iommu_stats_init(void)
 	amd_iommu_stats_add(&cnt_unmap_single);
 	amd_iommu_stats_add(&cnt_map_sg);
 	amd_iommu_stats_add(&cnt_unmap_sg);
+#ifdef CONFIG_HAS_DMA_P2P
+	amd_iommu_stats_add(&cnt_map_peer_resource);
+	amd_iommu_stats_add(&cnt_unmap_peer_resource);
+#endif
 	amd_iommu_stats_add(&cnt_alloc_coherent);
 	amd_iommu_stats_add(&cnt_free_coherent);
 	amd_iommu_stats_add(&cross_page);
@@ -2585,20 +2593,16 @@ static void __unmap_single(struct dma_ops_domain *dma_dom,
 }
 
 /*
- * The exported map_single function for dma_ops.
+ * Wrapper function that contains code common to mapping a physical address
+ * range from a page or a resource.
  */
-static dma_addr_t map_page(struct device *dev, struct page *page,
-			   unsigned long offset, size_t size,
-			   enum dma_data_direction dir,
-			   struct dma_attrs *attrs)
+static dma_addr_t __map_phys(struct device *dev, phys_addr_t paddr,
+			     size_t size, enum dma_data_direction dir)
 {
 	unsigned long flags;
 	struct protection_domain *domain;
 	dma_addr_t addr;
 	u64 dma_mask;
-	phys_addr_t paddr = page_to_phys(page) + offset;
-
-	INC_STATS_COUNTER(cnt_map_single);
 
 	domain = get_domain(dev);
 	if (PTR_ERR(domain) == -EINVAL)
@@ -2624,16 +2628,15 @@ out:
 }
 
 /*
- * The exported unmap_single function for dma_ops.
+ * Wrapper function that contains code common to unmapping a physical address
+ * range from a page or a resource.
  */
-static void unmap_page(struct device *dev, dma_addr_t dma_addr, size_t size,
-		       enum dma_data_direction dir, struct dma_attrs *attrs)
+static void __unmap_phys(struct device *dev, dma_addr_t dma_addr, size_t size,
+			 enum dma_data_direction dir)
 {
 	unsigned long flags;
 	struct protection_domain *domain;
 
-	INC_STATS_COUNTER(cnt_unmap_single);
-
 	domain = get_domain(dev);
 	if (IS_ERR(domain))
 		return;
@@ -2707,6 +2710,72 @@ unmap:
 }
 
 /*
+ * The exported map_single function for dma_ops.
+ */
+static dma_addr_t map_page(struct device *dev, struct page *page,
+			   unsigned long offset, size_t size,
+			   enum dma_data_direction dir,
+			   struct dma_attrs *attrs)
+{
+	INC_STATS_COUNTER(cnt_map_single);
+
+	return __map_phys(dev, page_to_phys(page) + offset, size, dir);
+}
+
+/*
+ * The exported unmap_single function for dma_ops.
+ */
+static void unmap_page(struct device *dev, dma_addr_t dma_addr, size_t size,
+		       enum dma_data_direction dir, struct dma_attrs *attrs)
+{
+	INC_STATS_COUNTER(cnt_unmap_single);
+
+	__unmap_phys(dev, dma_addr, size, dir);
+}
+
+#ifdef CONFIG_HAS_DMA_P2P
+/*
+ * The exported map_peer_resource function for dma_ops.
+ */
+static dma_peer_addr_t map_peer_resource(struct device *dev,
+					 struct device *peer,
+					 struct resource *res,
+					 unsigned long offset,
+					 size_t size,
+					 enum dma_data_direction dir,
+					 struct dma_attrs *attrs)
+{
+	struct pci_dev *pdev = to_pci_dev(dev);
+	struct pci_dev *ppeer = to_pci_dev(peer);
+	struct pci_host_bridge *hbdev = pci_find_host_bridge(pdev->bus);
+	struct pci_host_bridge *hbpeer = pci_find_host_bridge(ppeer->bus);
+
+	INC_STATS_COUNTER(cnt_map_peer_resource);
+
+	/*
+	 * Disallow the peer-to-peer mapping if the devices do not share a host
+	 * bridge.
+	 */
+	if (hbdev != hbpeer)
+		return DMA_ERROR_CODE;
+
+	return __map_phys(dev, res->start + offset, size, dir);
+}
+
+/*
+ * The exported unmap_peer_resource function for dma_ops.
+ */
+static void unmap_peer_resource(struct device *dev, dma_peer_addr_t dma_addr,
+				size_t size, enum dma_data_direction dir,
+				struct dma_attrs *attrs)
+{
+	INC_STATS_COUNTER(cnt_unmap_peer_resource);
+
+	__unmap_phys(dev, dma_addr, size, dir);
+}
+#endif
+
+/*
  * The exported map_sg function for dma_ops (handles scatter-gather
  * lists).
  */
@@ -2852,6 +2921,10 @@ static struct dma_map_ops amd_iommu_dma_ops = {
 	.unmap_page = unmap_page,
 	.map_sg = map_sg,
 	.unmap_sg = unmap_sg,
+#ifdef CONFIG_HAS_DMA_P2P
+	.map_peer_resource = map_peer_resource,
+	.unmap_peer_resource = unmap_peer_resource,
+#endif
 	.dma_supported = amd_iommu_dma_supported,
 };
 
-- 
2.4.6

[PATCH v4 09/12] iommu/vt-d: implement (un)map_peer_resource

[Will Davis]

Implement 'map_peer_resource' for the Intel IOMMU driver. Simply translate
the resource to a physical address and route it to the same handlers used
by the 'map_page' API.

This allows a device to map another's resource, to enable peer-to-peer
transactions.

These functions are guarded behind CONFIG_HAS_DMA_P2P, since the
dma_map_ops members are as well.

Signed-off-by: Will Davis <wdavis@nvidia.com>
Reviewed-by: Terence Ripperda <tripperda@nvidia.com>
Reviewed-by: John Hubbard <jhubbard@nvidia.com>
---
 drivers/iommu/intel-iommu.c | 38 ++++++++++++++++++++++++++++++++++++++
 1 file changed, 38 insertions(+)

diff --git a/drivers/iommu/intel-iommu.c b/drivers/iommu/intel-iommu.c
index a98a7b2..66b371c 100644
--- a/drivers/iommu/intel-iommu.c
+++ b/drivers/iommu/intel-iommu.c
@@ -3544,6 +3544,40 @@ static void intel_unmap_page(struct device *dev, dma_addr_t dev_addr,
 	intel_unmap(dev, dev_addr);
 }
 
+#ifdef CONFIG_HAS_DMA_P2P
+static dma_peer_addr_t intel_map_peer_resource(struct device *dev,
+					       struct device *peer,
+					       struct resource *res,
+					       unsigned long offset,
+					       size_t size,
+					       enum dma_data_direction dir,
+					       struct dma_attrs *attrs)
+{
+	struct pci_dev *pdev = to_pci_dev(dev);
+	struct pci_dev *ppeer = to_pci_dev(peer);
+	struct pci_host_bridge *hbdev = pci_find_host_bridge(pdev->bus);
+	struct pci_host_bridge *hbpeer = pci_find_host_bridge(ppeer->bus);
+
+	/*
+	 * Disallow the peer-to-peer mapping if the devices do not share a host
+	 * bridge.
+	 */
+	if (hbdev != hbpeer)
+		return 0;
+
+	return __intel_map_single(dev, res->start + offset, size,
+				  dir, *dev->dma_mask);
+}
+
+static void intel_unmap_peer_resource(struct device *dev,
+				      dma_peer_addr_t dev_addr,
+				      size_t size, enum dma_data_direction dir,
+				      struct dma_attrs *attrs)
+{
+	intel_unmap(dev, dev_addr);
+}
+#endif
+
 static void *intel_alloc_coherent(struct device *dev, size_t size,
 				  dma_addr_t *dma_handle, gfp_t flags,
 				  struct dma_attrs *attrs)
@@ -3700,6 +3734,10 @@ struct dma_map_ops intel_dma_ops = {
 	.unmap_sg = intel_unmap_sg,
 	.map_page = intel_map_page,
 	.unmap_page = intel_unmap_page,
+#ifdef CONFIG_HAS_DMA_P2P
+	.map_peer_resource = intel_map_peer_resource,
+	.unmap_peer_resource = intel_unmap_peer_resource,
+#endif
 	.mapping_error = intel_mapping_error,
 };
 
-- 
2.4.6

Re: [PATCH v4 09/12] iommu/vt-d: implement (un)map_peer_resource

[Bjorn Helgaas]

On Wed, Jul 22, 2015 at 04:39:54PM -0500, Will Davis wrote:
> Implement 'map_peer_resource' for the Intel IOMMU driver. Simply translate
> the resource to a physical address and route it to the same handlers used
> by the 'map_page' API.
> 
> This allows a device to map another's resource, to enable peer-to-peer
> transactions.
> 
> These functions are guarded behind CONFIG_HAS_DMA_P2P, since the
> dma_map_ops members are as well.
> 
> Signed-off-by: Will Davis <wdavis@nvidia.com>
> Reviewed-by: Terence Ripperda <tripperda@nvidia.com>
> Reviewed-by: John Hubbard <jhubbard@nvidia.com>
> ---
>  drivers/iommu/intel-iommu.c | 38 ++++++++++++++++++++++++++++++++++++++
>  1 file changed, 38 insertions(+)
> 
> diff --git a/drivers/iommu/intel-iommu.c b/drivers/iommu/intel-iommu.c
> index a98a7b2..66b371c 100644
> --- a/drivers/iommu/intel-iommu.c
> +++ b/drivers/iommu/intel-iommu.c
> @@ -3544,6 +3544,40 @@ static void intel_unmap_page(struct device *dev, dma_addr_t dev_addr,
>  	intel_unmap(dev, dev_addr);
>  }
>  
> +#ifdef CONFIG_HAS_DMA_P2P
> +static dma_peer_addr_t intel_map_peer_resource(struct device *dev,
> +					       struct device *peer,
> +					       struct resource *res,
> +					       unsigned long offset,
> +					       size_t size,
> +					       enum dma_data_direction dir,
> +					       struct dma_attrs *attrs)
> +{
> +	struct pci_dev *pdev = to_pci_dev(dev);
> +	struct pci_dev *ppeer = to_pci_dev(peer);

You got "struct device" pointers, so they might not be PCI devices.  You
should probably return an error if one is not PCI.

> +	struct pci_host_bridge *hbdev = pci_find_host_bridge(pdev->bus);
> +	struct pci_host_bridge *hbpeer = pci_find_host_bridge(ppeer->bus);
> +
> +	/*
> +	 * Disallow the peer-to-peer mapping if the devices do not share a host
> +	 * bridge.
> +	 */
> +	if (hbdev != hbpeer)
> +		return 0;

I think the only thing guaranteed by the spec is peer-to-peer between
devices in the same hierarchy, where the hierarchy is rooted at a PCIe Root
Port or a conventional PCI host bridge.  I don't remember anything about
transactions between Root Ports or conventional PCI host bridges.

If there is some language in the spec about those transactions, a citation
here would be nice.

This decision might be better made by a PCI interface.  This same basic
logic is repeated here, in the AMD equivalent, and in the x86 code, and I
suspect it will end up being a little more complicated for the reasons
above.

> +
> +	return __intel_map_single(dev, res->start + offset, size,
> +				  dir, *dev->dma_mask);
> +}
> +
> +static void intel_unmap_peer_resource(struct device *dev,
> +				      dma_peer_addr_t dev_addr,
> +				      size_t size, enum dma_data_direction dir,
> +				      struct dma_attrs *attrs)
> +{
> +	intel_unmap(dev, dev_addr);
> +}
> +#endif
> +
>  static void *intel_alloc_coherent(struct device *dev, size_t size,
>  				  dma_addr_t *dma_handle, gfp_t flags,
>  				  struct dma_attrs *attrs)
> @@ -3700,6 +3734,10 @@ struct dma_map_ops intel_dma_ops = {
>  	.unmap_sg = intel_unmap_sg,
>  	.map_page = intel_map_page,
>  	.unmap_page = intel_unmap_page,
> +#ifdef CONFIG_HAS_DMA_P2P
> +	.map_peer_resource = intel_map_peer_resource,
> +	.unmap_peer_resource = intel_unmap_peer_resource,
> +#endif
>  	.mapping_error = intel_mapping_error,
>  };
>  
> -- 
> 2.4.6
> 

RE: [PATCH v4 09/12] iommu/vt-d: implement (un)map_peer_resource

[William Davis]

On Monday, August 10, 2015 12:02 PM, Bjorn Helgaas wrote:
> On Wed, Jul 22, 2015 at 04:39:54PM -0500, Will Davis wrote:
> > Implement 'map_peer_resource' for the Intel IOMMU driver. Simply translate
> > the resource to a physical address and route it to the same handlers used
> > by the 'map_page' API.
> >
> > This allows a device to map another's resource, to enable peer-to-peer
> > transactions.
> >
> > These functions are guarded behind CONFIG_HAS_DMA_P2P, since the
> > dma_map_ops members are as well.
> >
> > Signed-off-by: Will Davis <wdavis@nvidia.com>
> > Reviewed-by: Terence Ripperda <tripperda@nvidia.com>
> > Reviewed-by: John Hubbard <jhubbard@nvidia.com>
> > ---
> >  drivers/iommu/intel-iommu.c | 38 ++++++++++++++++++++++++++++++++++++++
> >  1 file changed, 38 insertions(+)
> >
> > diff --git a/drivers/iommu/intel-iommu.c b/drivers/iommu/intel-iommu.c
> > index a98a7b2..66b371c 100644
> > --- a/drivers/iommu/intel-iommu.c
> > +++ b/drivers/iommu/intel-iommu.c
> > @@ -3544,6 +3544,40 @@ static void intel_unmap_page(struct device *dev, dma_addr_t dev_addr,
> >  	intel_unmap(dev, dev_addr);
> >  }
> >
> > +#ifdef CONFIG_HAS_DMA_P2P
> > +static dma_peer_addr_t intel_map_peer_resource(struct device *dev,
> > +					       struct device *peer,
> > +					       struct resource *res,
> > +					       unsigned long offset,
> > +					       size_t size,
> > +					       enum dma_data_direction dir,
> > +					       struct dma_attrs *attrs)
> > +{
> > +	struct pci_dev *pdev = to_pci_dev(dev);
> > +	struct pci_dev *ppeer = to_pci_dev(peer);
> 
> You got "struct device" pointers, so they might not be PCI devices.  You
> should probably return an error if one is not PCI.
> 
> > +	struct pci_host_bridge *hbdev = pci_find_host_bridge(pdev->bus);
> > +	struct pci_host_bridge *hbpeer = pci_find_host_bridge(ppeer->bus);
> > +
> > +	/*
> > +	 * Disallow the peer-to-peer mapping if the devices do not share a host
> > +	 * bridge.
> > +	 */
> > +	if (hbdev != hbpeer)
> > +		return 0;
> 
> I think the only thing guaranteed by the spec is peer-to-peer between
> devices in the same hierarchy, where the hierarchy is rooted at a PCIe Root
> Port or a conventional PCI host bridge.  I don't remember anything about
> transactions between Root Ports or conventional PCI host bridges.
> 
> If there is some language in the spec about those transactions, a citation
> here would be nice.
> 

The only thing I could find in the conventional PCI spec was in section
3.10.7 (PCI Local Bus Specification Revision 3.0):

"7.  Peer-to-peer transactions crossing multiple host bridges
     PCI host bridges may, but are not required to, support PCI peer-to-peer
     transactions that traverse multiple PCI host bridges."

In the PCIe spec (PCI Express Base Specification Revision 3.0), section 1.3.1:
"The capability to route peer-to-peer transactions between hierarchy domains
 through a Root Complex is optional and implementation dependent. For example,
 an implementation may incorporate a real or virtual Switch internally within
 the Root Complex to enable full peer-to-peer support in a software transparent
 way."

And then as noted in a previous thread, the ACS part of the spec does not
make it possible to know for sure (section 6.12.1.1 and footnote 80):

"ACS P2P Request Redirect: must be implemented by Root Ports that support
 peer-to-peer traffic with other Root Ports;80 must be implemented by Switch
 Downstream Ports.
 ...
 80 Root Port indication of ACS P2P Request Redirect or ACS P2P Completion
 Redirect support does not imply any particular level of peer-to-peer support
 by the Root Complex, or that peer-to-peer traffic is supported at all"

I could add these citations to the comment, noting that we disable this case
because we don't have any way to know for sure (short of a chipset whitelist).

> This decision might be better made by a PCI interface.  This same basic
> logic is repeated here, in the AMD equivalent, and in the x86 code, and I
> suspect it will end up being a little more complicated for the reasons
> above.
>

Agreed, I will move this to a separate interface (something like
pci_is_p2p_possible(dev, peer)).

Thanks,
Will
-----------------------------------------------------------------------------------
This email message is for the sole use of the intended recipient(s) and may contain
confidential information.  Any unauthorized review, use, disclosure or distribution
is prohibited.  If you are not the intended recipient, please contact the sender by
reply email and destroy all copies of the original message.
-----------------------------------------------------------------------------------

[PATCH v4 10/12] pci: add pci_find_common_upstream_dev()

[Will Davis]

Adds an interface to find the first device which is upstream of both
devices.

Signed-off-by: Will Davis <wdavis@nvidia.com>
---
 drivers/pci/search.c | 25 +++++++++++++++++++++++++
 include/linux/pci.h  |  2 ++
 2 files changed, 27 insertions(+)

diff --git a/drivers/pci/search.c b/drivers/pci/search.c
index a20ce7d..cbd8163 100644
--- a/drivers/pci/search.c
+++ b/drivers/pci/search.c
@@ -384,3 +384,28 @@ int pci_dev_present(const struct pci_device_id *ids)
 	return 0;
 }
 EXPORT_SYMBOL(pci_dev_present);
+
+/**
+ * pci_find_common_upstream_dev - Returns the first common upstream device
+ * @from: the PCI device to search for a common upstream device from
+ * @to: the PCI device to search for a common upstream device to
+ *
+ * Walks up the bus hierarchy from the @from device, looking for the first bus
+ * which the @to device is downstream of. Returns %NULL if the devices do not
+ * share any upstream devices.
+ */
+struct pci_dev *pci_find_common_upstream_dev(struct pci_dev *from,
+					     struct pci_dev *to)
+{
+	struct pci_dev *pdev = from;
+
+	while (pdev != NULL) {
+		if ((to->bus->number >= pdev->bus->busn_res.start) &&
+		    (to->bus->number <= pdev->bus->busn_res.end))
+			return pdev;
+		pdev = pdev->bus->self;
+	}
+
+	return NULL;
+}
+EXPORT_SYMBOL(pci_find_common_upstream_dev);
diff --git a/include/linux/pci.h b/include/linux/pci.h
index d58063e..8262b9e 100644
--- a/include/linux/pci.h
+++ b/include/linux/pci.h
@@ -860,6 +860,8 @@ static inline struct pci_dev *pci_get_bus_and_slot(unsigned int bus,
 }
 struct pci_dev *pci_get_class(unsigned int class, struct pci_dev *from);
 int pci_dev_present(const struct pci_device_id *ids);
+struct pci_dev *pci_find_common_upstream_dev(struct pci_dev *from,
+					     struct pci_dev *to);
 
 int pci_bus_read_config_byte(struct pci_bus *bus, unsigned int devfn,
 			     int where, u8 *val);
-- 
2.4.6

Re: [PATCH v4 10/12] pci: add pci_find_common_upstream_dev()

[Bjorn Helgaas]

On Wed, Jul 22, 2015 at 04:39:55PM -0500, Will Davis wrote:
> Adds an interface to find the first device which is upstream of both
> devices.
> 
> Signed-off-by: Will Davis <wdavis@nvidia.com>
> ---
>  drivers/pci/search.c | 25 +++++++++++++++++++++++++
>  include/linux/pci.h  |  2 ++
>  2 files changed, 27 insertions(+)
> 
> diff --git a/drivers/pci/search.c b/drivers/pci/search.c
> index a20ce7d..cbd8163 100644
> --- a/drivers/pci/search.c
> +++ b/drivers/pci/search.c
> @@ -384,3 +384,28 @@ int pci_dev_present(const struct pci_device_id *ids)
>  	return 0;
>  }
>  EXPORT_SYMBOL(pci_dev_present);
> +
> +/**
> + * pci_find_common_upstream_dev - Returns the first common upstream device
> + * @from: the PCI device to search for a common upstream device from
> + * @to: the PCI device to search for a common upstream device to
> + *
> + * Walks up the bus hierarchy from the @from device, looking for the first bus
> + * which the @to device is downstream of. Returns %NULL if the devices do not
> + * share any upstream devices.

This interface looks like it should be symmetric: you should get the same
answer regardless of which device is "from" and which is "to".  I would
rename the arguments to something more neutral to reflect that.

> + */
> +struct pci_dev *pci_find_common_upstream_dev(struct pci_dev *from,
> +					     struct pci_dev *to)
> +{
> +	struct pci_dev *pdev = from;
> +
> +	while (pdev != NULL) {
> +		if ((to->bus->number >= pdev->bus->busn_res.start) &&
> +		    (to->bus->number <= pdev->bus->busn_res.end))
> +			return pdev;
> +		pdev = pdev->bus->self;

I think you want "pdev = pci_upstream_bridge(pdev)" here; otherwise you'll
exit the loop early if "from" is a VF on a virtual bus.

> +	}
> +
> +	return NULL;
> +}
> +EXPORT_SYMBOL(pci_find_common_upstream_dev);
> diff --git a/include/linux/pci.h b/include/linux/pci.h
> index d58063e..8262b9e 100644
> --- a/include/linux/pci.h
> +++ b/include/linux/pci.h
> @@ -860,6 +860,8 @@ static inline struct pci_dev *pci_get_bus_and_slot(unsigned int bus,
>  }
>  struct pci_dev *pci_get_class(unsigned int class, struct pci_dev *from);
>  int pci_dev_present(const struct pci_device_id *ids);
> +struct pci_dev *pci_find_common_upstream_dev(struct pci_dev *from,
> +					     struct pci_dev *to);
>  
>  int pci_bus_read_config_byte(struct pci_bus *bus, unsigned int devfn,
>  			     int where, u8 *val);
> -- 
> 2.4.6
> 

[PATCH v4 11/12] x86: add pci-nommu implementation of map_peer_resource

[Will Davis]

Perform various checks on whether the mapping should be allowed, based on
PCIe ACS (Access Control Services) settings and the topology between the
two peers.

Signed-off-by: Will Davis <wdavis@nvidia.com>
---
 arch/x86/kernel/pci-nommu.c | 96 +++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 96 insertions(+)

diff --git a/arch/x86/kernel/pci-nommu.c b/arch/x86/kernel/pci-nommu.c
index da15918..58f5296 100644
--- a/arch/x86/kernel/pci-nommu.c
+++ b/arch/x86/kernel/pci-nommu.c
@@ -38,6 +38,101 @@ static dma_addr_t nommu_map_page(struct device *dev, struct page *page,
 	return bus;
 }
 
+static dma_peer_addr_t nommu_map_peer_resource(struct device *dev,
+					       struct device *peer,
+					       struct resource *res,
+					       unsigned long offset,
+					       size_t size,
+					       enum dma_data_direction dir,
+					       struct dma_attrs *attrs)
+{
+	struct pci_dev *pdev = to_pci_dev(dev);
+	struct pci_dev *ppeer = to_pci_dev(peer);
+	struct pci_dev *rpdev, *rppeer, *common_upstream;
+	struct pci_host_bridge *dev_host_bridge;
+	struct pci_host_bridge *peer_host_bridge;
+	struct pci_bus_region region;
+	dma_peer_addr_t dma_address;
+	int pos;
+	u16 cap;
+
+	/*
+	 * Disallow the peer-to-peer mapping if the devices do not share a host
+	 * bridge.
+	 */
+	dev_host_bridge = pci_find_host_bridge(pdev->bus);
+	peer_host_bridge = pci_find_host_bridge(ppeer->bus);
+	if (dev_host_bridge != peer_host_bridge)
+		return DMA_ERROR_CODE;
+
+	if (!pci_is_pcie(pdev))
+		goto out;
+
+	/*
+	 * Access Control Services (ACS) Checks
+	 *
+	 * ACS has a capability bit for P2P Request Redirects, but
+	 * unfortunately it doesn't tell us much about the real capabilities of
+	 * the hardware.
+	 */
+	rpdev = pdev->bus->self;
+	rppeer = ppeer->bus->self;
+
+	while ((rpdev) && (pci_is_pcie(rpdev)) &&
+	       (pci_pcie_type(rpdev) != PCI_EXP_TYPE_ROOT_PORT))
+		rpdev = rpdev->bus->self;
+
+	while ((rppeer) && (pci_is_pcie(rppeer)) &&
+	       (pci_pcie_type(rppeer) != PCI_EXP_TYPE_ROOT_PORT))
+		rppeer = rppeer->bus->self;
+
+	common_upstream = pci_find_common_upstream_dev(pdev, ppeer);
+
+	/* If ACS is not implemented, we have no idea about P2P support. */
+	pos = pci_find_ext_capability(rpdev, PCI_EXT_CAP_ID_ACS);
+	if (!pos)
+		goto out;
+
+	/*
+	 * If the devices are under the same root port and have a common
+	 * upstream device, allow if the root port is further upstream from the
+	 * common upstream device and the common upstream device has Upstream
+	 * Forwarding disabled, or if the root port is the common upstream
+	 * device and ACS is not implemented.
+	 */
+	pci_read_config_word(rpdev, pos + PCI_ACS_CAP, &cap);
+	if ((rpdev == rppeer && common_upstream) &&
+	    (((common_upstream != rpdev) &&
+	      !pci_acs_enabled(common_upstream, PCI_ACS_UF)) ||
+	     ((common_upstream == rpdev) && ((cap & PCI_ACS_RR) == 0))))
+		goto out;
+
+	if (cap & PCI_ACS_RR) {
+		/* If ACS RR is implemented and enabled, allow the mapping */
+		if (pci_acs_enabled(rpdev, PCI_ACS_RR))
+			goto out;
+
+		/*
+		 * If ACS RR is implemented and disabled, allow if the devices
+		 * are under the same root port.
+		 */
+		if (!pci_acs_enabled(rpdev, PCI_ACS_RR) && rpdev == rppeer)
+			goto out;
+	}
+
+	return DMA_ERROR_CODE;
+
+out:
+	pcibios_resource_to_bus(pdev->bus, &region, res);
+	dma_address = region.start + offset;
+	WARN_ON(size == 0);
+	if (!check_addr("map_peer_resource", dev, dma_address, size))
+		return DMA_ERROR_CODE;
+	flush_write_buffers();
+	return dma_address;
+}
+
+
 /* Map a set of buffers described by scatterlist in streaming
  * mode for DMA.  This is the scatter-gather version of the
  * above pci_map_single interface.  Here the scatter gather list
@@ -93,6 +188,7 @@ struct dma_map_ops nommu_dma_ops = {
 	.free			= dma_generic_free_coherent,
 	.map_sg			= nommu_map_sg,
 	.map_page		= nommu_map_page,
+	.map_peer_resource	= nommu_map_peer_resource,
 	.sync_single_for_device = nommu_sync_single_for_device,
 	.sync_sg_for_device	= nommu_sync_sg_for_device,
 	.is_phys		= 1,
-- 
2.4.6

[PATCH v4 12/12] x86: declare support for DMA P2P

[Will Davis]

Use CONFIG_HAS_DMA_P2P to declare that we want the DMA peer resource APIs
enabled on x86, and provide the requisite dma_peer_mapping_error()
implementation.

Signed-off-by: Will Davis <wdavis@nvidia.com>
---
 arch/x86/Kconfig                   |  1 +
 arch/x86/include/asm/dma-mapping.h | 18 ++++++++++++++++--
 2 files changed, 17 insertions(+), 2 deletions(-)

diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig
index 3dbb7e7..581d1ad 100644
--- a/arch/x86/Kconfig
+++ b/arch/x86/Kconfig
@@ -149,6 +149,7 @@ config X86
 	select VIRT_TO_BUS
 	select X86_DEV_DMA_OPS			if X86_64
 	select X86_FEATURE_NAMES		if PROC_FS
+	select HAS_DMA_P2P
 
 config INSTRUCTION_DECODER
 	def_bool y
diff --git a/arch/x86/include/asm/dma-mapping.h b/arch/x86/include/asm/dma-mapping.h
index 1f5b728..64472d8 100644
--- a/arch/x86/include/asm/dma-mapping.h
+++ b/arch/x86/include/asm/dma-mapping.h
@@ -44,16 +44,30 @@ static inline struct dma_map_ops *get_dma_ops(struct device *dev)
 #include <asm-generic/dma-mapping-common.h>
 
 /* Make sure we keep the same behaviour */
-static inline int dma_mapping_error(struct device *dev, dma_addr_t dma_addr)
+static inline int dma_mapping_error_common(struct device *dev, u64 dma_addr)
 {
 	struct dma_map_ops *ops = get_dma_ops(dev);
-	debug_dma_mapping_error(dev, dma_addr);
 	if (ops->mapping_error)
 		return ops->mapping_error(dev, dma_addr);
 
 	return (dma_addr == DMA_ERROR_CODE);
 }
 
+static inline int dma_mapping_error(struct device *dev, dma_addr_t dma_addr)
+{
+	debug_dma_mapping_error(dev, dma_addr);
+	return dma_mapping_error_common(dev, (u64)dma_addr);
+}
+
+#ifdef CONFIG_HAS_DMA_P2P
+static inline int dma_peer_mapping_error(struct device *dev,
+					 dma_peer_addr_t dma_addr)
+{
+	debug_dma_peer_mapping_error(dev, dma_addr);
+	return dma_mapping_error_common(dev, (u64)dma_addr);
+}
+#endif
+
 #define dma_alloc_noncoherent(d, s, h, f) dma_alloc_coherent(d, s, h, f)
 #define dma_free_noncoherent(d, s, v, h) dma_free_coherent(d, s, v, h)
 
-- 
2.4.6

Re: [PATCH v4 00/12] DMA-API/PCI map_peer_resource support for peer-to-peer

[Will Davis]

> Hi,
> 
> This is the fourth version of a patchset to add the DMA APIs necessary to
> map and unmap a PCI device's BAR to and from another PCI device's IOVA
> domain. This enables PCI peer-to-peer traffic on x86 platforms where an
> IOMMU is enabled (this requires that peer-to-peer traffic be supported by
> the underlying chipset as well, of course).
> 

Bjorn, does this patch series adequately address the concerns with the
previous versions?

> Thanks,
> Will
> 
> v4:
> - added HAS_DMA_P2P Kconfig option to guard new APIs, which are only
>   supported on x86 with this patch series
> - added a new dma_peer_addr_t type to handle future platform support
>   where the dma_peer_addr_t needs to have the same width as the bus
>   address, which can be wider than the dma_addr_t returned by other
>   DMA APIs
> - added a new dma_peer_mapping_error() and associated PCI wrapper to
>   detect errors returned via dma_map_peer_resource() API and its
>   PCI wrapper
> - modified the interface of the new API to explicitly take the peer
>   device as an argument
> - renamed the (un)map_resource APIs to (un)map_peer_resource, since
>   they directly receive a 'peer' argument now
> - modified the new PCI interface to take the BAR index instead of the
>   actual struct resource
> - modified the pci-nommu implementation to perform topology and ACS
>   checks before allowing
> - modified both the AMD and Intel IOMMU implementations to require
>   that both devices be behind the same host bridge
> - exposed a couple of preexisting PCI utility functions to aid in the
>   topology checks for all of the x86 implementations
> 
> v3: http://www.spinics.net/lists/linux-pci/msg41703.html
> - changed dma_map_resource() to BUG() if the map_resource DMA op is not
>   provided, instead of returning 0
> - updated documentation to remove requirement to check for 0 return value
>   as an error
> - remove const keyword from struct dma_map_ops in new DMA APIs for
>   consistency with other APIs
> 
> v2: http://www.spinics.net/lists/linux-pci/msg41192.html
> - added documentation for the new DMA APIs
> - fixed physical-to-bus address conversion in the nommu implementation
> 
> v1: http://www.spinics.net/lists/linux-pci/msg40747.html
> 
> 
> 
> Will Davis (12):
>   lib/Kconfig: add HAS_DMA_P2P for peer-to-peer support
>   linux/types.h: Add dma_peer_addr_t type
>   dma-debug: add checking for map/unmap_peer_resource
>   DMA-API: Introduce dma_(un)map_peer_resource
>   dma-mapping: pci: add pci_(un)map_peer_resource
>   DMA-API: Add peer resource mapping documentation
>   pci: expose pci_find_host_bridge()
>   iommu/amd: Implement (un)map_peer_resource
>   iommu/vt-d: implement (un)map_peer_resource
>   pci: add pci_find_common_upstream_dev()
>   x86: add pci-nommu implementation of map_peer_resource
>   x86: declare support for DMA P2P
> 
>  Documentation/DMA-API-HOWTO.txt          | 46 ++++++++++++++-
>  Documentation/DMA-API.txt                | 36 ++++++++++--
>  arch/x86/Kconfig                         |  1 +
>  arch/x86/include/asm/dma-mapping.h       | 18 +++++-
>  arch/x86/kernel/pci-nommu.c              | 96 +++++++++++++++++++++++++++++++
>  drivers/iommu/amd_iommu.c                | 99 +++++++++++++++++++++++++++-----
>  drivers/iommu/intel-iommu.c              | 38 ++++++++++++
>  drivers/pci/host-bridge.c                | 19 +-----
>  drivers/pci/search.c                     | 25 ++++++++
>  include/asm-generic/dma-mapping-common.h | 43 ++++++++++++++
>  include/asm-generic/pci-dma-compat.h     | 23 ++++++++
>  include/linux/dma-debug.h                | 39 ++++++++++++-
>  include/linux/dma-mapping.h              | 12 ++++
>  include/linux/pci.h                      |  9 +++
>  include/linux/types.h                    | 13 ++++-
>  lib/Kconfig                              |  5 ++
>  lib/dma-debug.c                          | 86 ++++++++++++++++++++++++++-
>  17 files changed, 564 insertions(+), 44 deletions(-)
> 
> -- 
> 2.4.6