[PATCH 0/4] arm64: IOMMU-backed DMA mapping

[PATCH 0/4] arm64: IOMMU-backed DMA mapping

[Robin Murphy]

Hi all,

Here's the cleaned-up and nominally ready-to-go IOMMU DMA mapping series,
with (I think) all the outstanding comments[1] addressed. I appreciate
I've probably missed the boat for 4.2, but I'm ever the optimist...

Major changes since RFC:
- Renamed things, shuffled things around, and removed some of the less
  useful abstraction for simplicity and clarity.
- Fixed iommu_dma_alloc to handle compound pages and map buffers as
  scatterlists.
- Filled the API holes like size-aligned allocations and get_sgtable.
- Solved the problem of attching devices to domains before the devices
  are created, which confuses IOMMU drivers like the ARM SMMU.

A branch is available at:
   git://linux-arm.org/linux-rm iommu/dma

In terms of where we go from here, this is more or less what I envisage:
- Get the current functionality merged to unblock arm64 IOMMU work
- Implement proper managed IOMMU domains around this, integrating the
  iova_domain into the iommu_domain itself, and getting rid of the
  iommu_dma_domain wrapper.
- Find all of the ad-hoc "managed domain" implementations in drivers
  currently using the arm_iommu_* functions and convert them to "proper"
  managed domains.
- Finally, with the external users taken care of, bring arch/arm over
  to the common DMA mapping code.

In the meantime, Laurent's proposal for probe deferral[2] offers hope
that the bus notifier dance may only need to be short-lived, and I'm
hoping to spend some more time on DT-based IOMMU group handling for
platform devices, which should hopefully tie in with default domains
to make the add_device callback mostly redundant.

[1]:http://thread.gmane.org/gmane.linux.kernel.iommu/8773
[2]:http://thread.gmane.org/gmane.linux.kernel.iommu/9552

Robin Murphy (4):
  iommu/iova: Avoid over-allocating when size-aligned
  iommu: Implement common IOMMU ops for DMA mapping
  arm64: Add IOMMU dma_ops
  arm64: Hook up IOMMU dma_ops

 arch/arm64/Kconfig                   |   1 +
 arch/arm64/include/asm/device.h      |   3 +
 arch/arm64/include/asm/dma-mapping.h |  25 +-
 arch/arm64/mm/dma-mapping.c          | 357 ++++++++++++++++++++++
 drivers/iommu/Kconfig                |   7 +
 drivers/iommu/Makefile               |   1 +
 drivers/iommu/dma-iommu.c            | 560 +++++++++++++++++++++++++++++++++++
 drivers/iommu/intel-iommu.c          |   2 +
 drivers/iommu/iova.c                 |  23 +-
 include/linux/dma-iommu.h            |  94 ++++++
 10 files changed, 1051 insertions(+), 22 deletions(-)
 create mode 100644 drivers/iommu/dma-iommu.c
 create mode 100644 include/linux/dma-iommu.h

-- 
1.9.1

[PATCH 1/4] iommu/iova: Avoid over-allocating when size-aligned

[Robin Murphy]

Currently, allocating a size-aligned IOVA region quietly adjusts the
actual allocation size in the process, returning a rounded-up
power-of-two-sized allocation. This results in mismatched behaviour in
the IOMMU driver if the original size was not a power of two, where the
original size is mapped, but the rounded-up IOVA size is unmapped.

Whilst some IOMMUs will happily unmap already-unmapped pages, others
consider this an error, so fix it by computing the necessary alignment
padding without altering the actual allocation size. Also clean up by
making pad_size unsigned, since negative padding makes no sense.

CC: David Woodhouse <dwmw2-wEGCiKHe2LqWVfeAwA7xHQ@public.gmane.org>
Signed-off-by: Robin Murphy <robin.murphy-5wv7dgnIgG8@public.gmane.org>
---
 drivers/iommu/intel-iommu.c |  2 ++
 drivers/iommu/iova.c        | 23 ++++++-----------------
 2 files changed, 8 insertions(+), 17 deletions(-)

diff --git a/drivers/iommu/intel-iommu.c b/drivers/iommu/intel-iommu.c
index 68d43be..ada1fb9 100644
--- a/drivers/iommu/intel-iommu.c
+++ b/drivers/iommu/intel-iommu.c
@@ -2910,6 +2910,8 @@ static struct iova *intel_alloc_iova(struct device *dev,
 
 	/* Restrict dma_mask to the width that the iommu can handle */
 	dma_mask = min_t(uint64_t, DOMAIN_MAX_ADDR(domain->gaw), dma_mask);
+	/* Ensure we reserve the whole size-aligned region */
+	nrpages = __roundup_pow_of_two(nrpages);
 
 	if (!dmar_forcedac && dma_mask > DMA_BIT_MASK(32)) {
 		/*
diff --git a/drivers/iommu/iova.c b/drivers/iommu/iova.c
index 9dd8208..2c4a51b 100644
--- a/drivers/iommu/iova.c
+++ b/drivers/iommu/iova.c
@@ -120,19 +120,14 @@ __cached_rbnode_delete_update(struct iova_domain *iovad, struct iova *free)
 	}
 }
 
-/* Computes the padding size required, to make the
- * the start address naturally aligned on its size
+/*
+ * Computes the padding size required, to make the start address
+ * naturally aligned on the power-of-two order of its size
  */
-static int
-iova_get_pad_size(int size, unsigned int limit_pfn)
+static unsigned int
+iova_get_pad_size(unsigned int size, unsigned int limit_pfn)
 {
-	unsigned int pad_size = 0;
-	unsigned int order = ilog2(size);
-
-	if (order)
-		pad_size = (limit_pfn + 1) % (1 << order);
-
-	return pad_size;
+	return (limit_pfn + 1 - size) & (__roundup_pow_of_two(size) - 1);
 }
 
 static int __alloc_and_insert_iova_range(struct iova_domain *iovad,
@@ -264,12 +259,6 @@ alloc_iova(struct iova_domain *iovad, unsigned long size,
 	if (!new_iova)
 		return NULL;
 
-	/* If size aligned is set then round the size to
-	 * to next power of two.
-	 */
-	if (size_aligned)
-		size = __roundup_pow_of_two(size);
-
 	ret = __alloc_and_insert_iova_range(iovad, size, limit_pfn,
 			new_iova, size_aligned);
 
-- 
1.9.1

[PATCH 2/4] iommu: Implement common IOMMU ops for DMA mapping

[Robin Murphy]

Taking inspiration from the existing arch/arm code, break out some
generic functions to interface the DMA-API to the IOMMU-API. This will
do the bulk of the heavy lifting for IOMMU-backed dma-mapping.

Signed-off-by: Robin Murphy <robin.murphy-5wv7dgnIgG8@public.gmane.org>
---
 drivers/iommu/Kconfig     |   7 +
 drivers/iommu/Makefile    |   1 +
 drivers/iommu/dma-iommu.c | 560 ++++++++++++++++++++++++++++++++++++++++++++++
 include/linux/dma-iommu.h |  94 ++++++++
 4 files changed, 662 insertions(+)
 create mode 100644 drivers/iommu/dma-iommu.c
 create mode 100644 include/linux/dma-iommu.h

diff --git a/drivers/iommu/Kconfig b/drivers/iommu/Kconfig
index 1ae4e54..9107b6e 100644
--- a/drivers/iommu/Kconfig
+++ b/drivers/iommu/Kconfig
@@ -48,6 +48,13 @@ config OF_IOMMU
        def_bool y
        depends on OF && IOMMU_API
 
+# IOMMU-agnostic DMA-mapping layer
+config IOMMU_DMA
+	bool
+	depends on NEED_SG_DMA_LENGTH
+	select IOMMU_API
+	select IOMMU_IOVA
+
 config FSL_PAMU
 	bool "Freescale IOMMU support"
 	depends on PPC32
diff --git a/drivers/iommu/Makefile b/drivers/iommu/Makefile
index 080ffab..574b241 100644
--- a/drivers/iommu/Makefile
+++ b/drivers/iommu/Makefile
@@ -1,6 +1,7 @@
 obj-$(CONFIG_IOMMU_API) += iommu.o
 obj-$(CONFIG_IOMMU_API) += iommu-traces.o
 obj-$(CONFIG_IOMMU_API) += iommu-sysfs.o
+obj-$(CONFIG_IOMMU_DMA) += dma-iommu.o
 obj-$(CONFIG_IOMMU_IO_PGTABLE) += io-pgtable.o
 obj-$(CONFIG_IOMMU_IO_PGTABLE_LPAE) += io-pgtable-arm.o
 obj-$(CONFIG_IOMMU_IOVA) += iova.o
diff --git a/drivers/iommu/dma-iommu.c b/drivers/iommu/dma-iommu.c
new file mode 100644
index 0000000..5b2b065
--- /dev/null
+++ b/drivers/iommu/dma-iommu.c
@@ -0,0 +1,560 @@
+/*
+ * A fairly generic DMA-API to IOMMU-API glue layer.
+ *
+ * Copyright (C) 2014-2015 ARM Ltd.
+ *
+ * based in part on arch/arm/mm/dma-mapping.c:
+ * Copyright (C) 2000-2004 Russell King
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program.  If not, see <http://www.gnu.org/licenses/>.
+ */
+
+#define pr_fmt(fmt)	"%s: " fmt, __func__
+
+#include <linux/device.h>
+#include <linux/dma-iommu.h>
+#include <linux/huge_mm.h>
+#include <linux/iommu.h>
+#include <linux/iova.h>
+#include <linux/mm.h>
+
+int iommu_dma_init(void)
+{
+	return iommu_iova_cache_init();
+}
+
+struct iommu_dma_domain {
+	struct iommu_domain *domain;
+	struct iova_domain *iovad;
+	struct kref kref;
+};
+
+/**
+ * iommu_dma_create_domain - Create a DMA mapping domain
+ * @ops: iommu_ops representing the IOMMU backing this domain. It is down to
+ *       the IOMMU driver whether a domain may span multiple IOMMU instances
+ * @base: IOVA at which the mappable address space starts
+ * @size: Size of IOVA space
+ *
+ * @base and @size should be exact multiples of IOMMU page granularity to
+ * avoid rounding surprises. If necessary, we reserve the page at address 0
+ * to ensure it is an invalid IOVA.
+ *
+ * Return: Pointer to a domain initialised with the given IOVA range,
+ *         or NULL on failure. If successful, the caller holds an initial
+ *         reference, which may be released with iommu_dma_release_domain()
+ *         once a device is attached.
+ */
+struct iommu_dma_domain *iommu_dma_create_domain(const struct iommu_ops *ops,
+		dma_addr_t base, u64 size)
+{
+	struct iommu_dma_domain *dom;
+	struct iommu_domain *domain;
+	struct iova_domain *iovad;
+	unsigned long order, base_pfn, end_pfn;
+
+	dom = kzalloc(sizeof(*dom), GFP_KERNEL);
+	if (!dom)
+		return NULL;
+	/*
+	 * HACK(sort of): These domains currently belong to this layer and are
+	 * opaque from outside it, so they are "unmanaged" by the IOMMU API
+	 * itself. Once we have default domain support worked out, then we can
+	 * turn things inside out and put these inside managed IOMMU domains...
+	 */
+	domain = ops->domain_alloc(IOMMU_DOMAIN_UNMANAGED);
+	if (!domain)
+		goto out_free_dma_domain;
+
+	domain->ops = ops;
+	domain->type = IOMMU_DOMAIN_UNMANAGED;
+
+	/* Use the smallest supported page size for IOVA granularity */
+	order = __ffs(ops->pgsize_bitmap);
+	base_pfn = max_t(unsigned long, 1, base >> order);
+	end_pfn = (base + size - 1) >> order;
+
+	/* Check the domain allows at least some access to the device... */
+	if (domain->geometry.force_aperture) {
+		if (base > domain->geometry.aperture_end ||
+		    base + size <= domain->geometry.aperture_start) {
+			pr_warn("specified DMA range outside IOMMU capability\n");
+			goto out_free_iommu_domain;
+		}
+		/* ...then finally give it a kicking to make sure it fits */
+		base_pfn = max_t(unsigned long, base_pfn,
+				domain->geometry.aperture_start >> order);
+		end_pfn = min_t(unsigned long, end_pfn,
+				domain->geometry.aperture_end >> order);
+	}
+	/*
+	 * Note that this almost certainly breaks the case where multiple
+	 * devices with different DMA capabilities need to share a domain,
+	 * but we don't have the necessary information to handle that here
+	 * anyway - "proper" group and domain allocation needs to involve
+	 * the IOMMU driver and a complete view of the bus.
+	 */
+
+	iovad = kzalloc(sizeof(*iovad), GFP_KERNEL);
+	if (!iovad)
+		goto out_free_iommu_domain;
+
+	init_iova_domain(iovad, 1UL << order, base_pfn, end_pfn);
+
+	dom->domain = domain;
+	dom->iovad = iovad;
+	kref_init(&dom->kref);
+	return dom;
+
+out_free_iommu_domain:
+	ops->domain_free(domain);
+out_free_dma_domain:
+	kfree(dom);
+	return NULL;
+}
+
+static void __iommu_dma_free_domain(struct kref *kref)
+{
+	struct iommu_dma_domain *dom;
+
+	dom = container_of(kref, struct iommu_dma_domain, kref);
+	put_iova_domain(dom->iovad);
+	iommu_domain_free(dom->domain);
+	kfree(dom);
+}
+
+void iommu_dma_release_domain(struct iommu_dma_domain *dom)
+{
+	kref_put(&dom->kref, __iommu_dma_free_domain);
+}
+
+struct iommu_domain *iommu_dma_raw_domain(struct iommu_dma_domain *dom)
+{
+	return dom->domain;
+}
+
+int iommu_dma_attach_device(struct device *dev, struct iommu_dma_domain *dom)
+{
+	int ret;
+
+	kref_get(&dom->kref);
+	ret = iommu_attach_device(dom->domain, dev);
+	if (!ret)
+		arch_set_dma_domain(dev, dom);
+	return ret;
+}
+
+void iommu_dma_detach_device(struct device *dev)
+{
+	struct iommu_dma_domain *dom = arch_get_dma_domain(dev);
+
+	arch_set_dma_domain(dev, NULL);
+	iommu_detach_device(dom->domain, dev);
+	iommu_dma_release_domain(dom);
+}
+
+/*
+ * IOVAs are IOMMU _input_ addresses, so there still exists the possibility
+ * for static bus translation between device output and IOMMU input (yuck).
+ */
+static inline dma_addr_t dev_dma_addr(struct device *dev, dma_addr_t addr)
+{
+	dma_addr_t offset = (dma_addr_t)dev->dma_pfn_offset << PAGE_SHIFT;
+
+	BUG_ON(addr < offset);
+	return addr - offset;
+}
+
+/**
+ * dma_direction_to_prot - Translate DMA API directions to IOMMU API page flags
+ * @dir: Direction of DMA transfer
+ * @coherent: Is the DMA master cache-coherent?
+ *
+ * Return: corresponding IOMMU API page protection flags
+ */
+int dma_direction_to_prot(enum dma_data_direction dir, bool coherent)
+{
+	int prot = coherent ? IOMMU_CACHE : 0;
+
+	switch (dir) {
+	case DMA_BIDIRECTIONAL:
+		return prot | IOMMU_READ | IOMMU_WRITE;
+	case DMA_TO_DEVICE:
+		return prot | IOMMU_READ;
+	case DMA_FROM_DEVICE:
+		return prot | IOMMU_WRITE;
+	default:
+		return 0;
+	}
+}
+
+static struct iova *__alloc_iova(struct device *dev, size_t size, bool coherent)
+{
+	struct iommu_dma_domain *dom = arch_get_dma_domain(dev);
+	struct iova_domain *iovad = dom->iovad;
+	unsigned long shift = iova_shift(iovad);
+	unsigned long length = iova_align(iovad, size) >> shift;
+	u64 dma_limit = coherent ? dev->coherent_dma_mask : dma_get_mask(dev);
+
+	/*
+	 * Enforce size-alignment to be safe - there should probably be
+	 * an attribute to control this per-device, or at least per-domain...
+	 */
+	return alloc_iova(iovad, length, dma_limit >> shift, true);
+}
+
+/* The IOVA allocator knows what we mapped, so just unmap whatever that was */
+static void __iommu_dma_unmap(struct iommu_dma_domain *dom, dma_addr_t dma_addr)
+{
+	struct iova_domain *iovad = dom->iovad;
+	unsigned long shift = iova_shift(iovad);
+	unsigned long pfn = dma_addr >> shift;
+	struct iova *iova = find_iova(iovad, pfn);
+	size_t size = iova_size(iova) << shift;
+
+	/* ...and if we can't, then something is horribly, horribly wrong */
+	BUG_ON(iommu_unmap(dom->domain, pfn << shift, size) < size);
+	__free_iova(iovad, iova);
+}
+
+static void __iommu_dma_free_pages(struct page **pages, int count)
+{
+	while (count--)
+		__free_page(pages[count]);
+	kvfree(pages);
+}
+
+static struct page **__iommu_dma_alloc_pages(unsigned int count, gfp_t gfp)
+{
+	struct page **pages;
+	unsigned int i = 0, array_size = count * sizeof(*pages);
+
+	if (array_size <= PAGE_SIZE)
+		pages = kzalloc(array_size, GFP_KERNEL);
+	else
+		pages = vzalloc(array_size);
+	if (!pages)
+		return NULL;
+
+	while (count) {
+		struct page *page = NULL;
+		int j, order = __fls(count);
+
+		/*
+		 * Higher-order allocations are a convenience rather
+		 * than a necessity, hence using __GFP_NORETRY until
+		 * falling back to single-page allocations.
+		 */
+		for (order = min(order, MAX_ORDER); order > 0; order--) {
+			page = alloc_pages(gfp | __GFP_NORETRY, order);
+			if (!page)
+				continue;
+			if (PageCompound(page)) {
+				if (!split_huge_page(page))
+					break;
+				__free_pages(page, order);
+			} else {
+				split_page(page, order);
+				break;
+			}
+		}
+		if (!page)
+			page = alloc_page(gfp);
+		if (!page) {
+			__iommu_dma_free_pages(pages, i);
+			return NULL;
+		}
+		j = 1 << order;
+		count -= j;
+		while (j--)
+			pages[i++] = page++;
+	}
+	return pages;
+}
+
+/**
+ * iommu_dma_free - Free a buffer allocated by iommu_dma_alloc()
+ * @dev: Device which owns this buffer
+ * @pages: Array of buffer pages as returned by iommu_dma_alloc()
+ * @size: Size of buffer in bytes
+ * @handle: DMA address of buffer
+ *
+ * Frees both the pages associated with the buffer, and the array
+ * describing them
+ */
+void iommu_dma_free(struct device *dev, struct page **pages, size_t size,
+		dma_addr_t *handle)
+{
+	__iommu_dma_unmap(arch_get_dma_domain(dev), *handle);
+	__iommu_dma_free_pages(pages, PAGE_ALIGN(size) >> PAGE_SHIFT);
+	*handle = DMA_ERROR_CODE;
+}
+
+/**
+ * iommu_dma_alloc - Allocate and map a buffer contiguous in IOVA space
+ * @dev: Device to allocate memory for. Must be a real device
+ *	 attached to an iommu_dma_domain
+ * @size: Size of buffer in bytes
+ * @gfp: Allocation flags
+ * @prot: IOMMU mapping flags
+ * @coherent: Which dma_mask to base IOVA allocation on
+ * @handle: Out argument for allocated DMA handle
+ * @flush_page: Arch callback to flush a single page from caches as
+ *		necessary. May be NULL for coherent allocations
+ *
+ * If @size is less than PAGE_SIZE, then a full CPU page will be allocated,
+ * but an IOMMU which supports smaller pages might not map the whole thing.
+ * For now, the buffer is unconditionally zeroed for compatibility
+ *
+ * Return: Array of struct page pointers describing the buffer,
+ *	   or NULL on failure.
+ */
+struct page **iommu_dma_alloc(struct device *dev, size_t size, gfp_t gfp,
+		int prot, bool coherent, dma_addr_t *handle,
+		void (*flush_page)(const void *, phys_addr_t))
+{
+	struct iommu_dma_domain *dom = arch_get_dma_domain(dev);
+	struct iova_domain *iovad = dom->iovad;
+	struct iova *iova;
+	struct page **pages;
+	struct sg_table sgt;
+	struct sg_mapping_iter miter;
+	dma_addr_t dma_addr;
+	unsigned int count = PAGE_ALIGN(size) >> PAGE_SHIFT;
+
+	*handle = DMA_ERROR_CODE;
+
+	/* IOMMU can map any pages, so himem can also be used here */
+	gfp |= __GFP_NOWARN | __GFP_HIGHMEM;
+	pages = __iommu_dma_alloc_pages(count, gfp);
+	if (!pages)
+		return NULL;
+
+	iova = __alloc_iova(dev, size, coherent);
+	if (!iova)
+		goto out_free_pages;
+
+	if (sg_alloc_table_from_pages(&sgt, pages, count, 0, size, GFP_KERNEL))
+		goto out_free_iova;
+
+	dma_addr = iova_dma_addr(iovad, iova);
+	if (iommu_map_sg(dom->domain, dma_addr, sgt.sgl, sgt.orig_nents, prot)
+			< size)
+		goto out_free_sg;
+
+	/* Using the non-flushing flag since we're doing our own */
+	sg_miter_start(&miter, sgt.sgl, sgt.orig_nents, SG_MITER_FROM_SG);
+	while (sg_miter_next(&miter)) {
+		memset(miter.addr, 0, PAGE_SIZE);
+		if (flush_page)
+			flush_page(miter.addr, page_to_phys(miter.page));
+	}
+	sg_miter_stop(&miter);
+	sg_free_table(&sgt);
+
+	*handle = dma_addr;
+	return pages;
+
+out_free_sg:
+	sg_free_table(&sgt);
+out_free_iova:
+	__free_iova(iovad, iova);
+out_free_pages:
+	__iommu_dma_free_pages(pages, count);
+	return NULL;
+}
+
+/**
+ * iommu_dma_mmap - Map a buffer into provided user VMA
+ * @pages: Array representing buffer from iommu_dma_alloc()
+ * @size: Size of buffer in bytes
+ * @vma: VMA describing requested userspace mapping
+ *
+ * Maps the pages of the buffer in @pages into @vma. The caller is responsible
+ * for verifying the correct size and protection of @vma beforehand.
+ */
+
+int iommu_dma_mmap(struct page **pages, size_t size, struct vm_area_struct *vma)
+{
+	unsigned long uaddr = vma->vm_start;
+	unsigned int i, count = PAGE_ALIGN(size) >> PAGE_SHIFT;
+	int ret = -ENXIO;
+
+	for (i = vma->vm_pgoff; i < count && uaddr < vma->vm_end; i++) {
+		ret = vm_insert_page(vma, uaddr, pages[i]);
+		if (ret)
+			break;
+		uaddr += PAGE_SIZE;
+	}
+	return ret;
+}
+
+dma_addr_t iommu_dma_map_page(struct device *dev, struct page *page,
+		unsigned long offset, size_t size, int prot, bool coherent)
+{
+	dma_addr_t dma_addr;
+	struct iommu_dma_domain *dom = arch_get_dma_domain(dev);
+	struct iova_domain *iovad = dom->iovad;
+	phys_addr_t phys = page_to_phys(page) + offset;
+	size_t iova_off = iova_offset(iovad, phys);
+	size_t len = iova_align(iovad, size + iova_off);
+	struct iova *iova = __alloc_iova(dev, len, coherent);
+
+	if (!iova)
+		return DMA_ERROR_CODE;
+
+	dma_addr = iova_dma_addr(iovad, iova);
+	if (!iommu_map(dom->domain, dma_addr, phys - iova_off, len, prot))
+		return dev_dma_addr(dev, dma_addr + iova_off);
+
+	__free_iova(iovad, iova);
+	return DMA_ERROR_CODE;
+}
+
+void iommu_dma_unmap_page(struct device *dev, dma_addr_t handle, size_t size,
+		enum dma_data_direction dir, struct dma_attrs *attrs)
+{
+	__iommu_dma_unmap(arch_get_dma_domain(dev), handle);
+}
+
+static int __finalise_sg(struct device *dev, struct scatterlist *sg, int nents,
+		dma_addr_t dma_addr)
+{
+	struct scatterlist *s, *seg = sg;
+	unsigned long seg_mask = dma_get_seg_boundary(dev);
+	unsigned int max_len = dma_get_max_seg_size(dev);
+	unsigned int seg_len = 0, seg_dma = 0;
+	int i, count = 1;
+
+	for_each_sg(sg, s, nents, i) {
+		/* Un-swizzling the fields here, hence the naming mismatch */
+		unsigned int s_offset = sg_dma_address(s);
+		unsigned int s_length = sg_dma_len(s);
+		unsigned int s_dma_len = s->length;
+
+		s->offset = s_offset;
+		s->length = s_length;
+		sg_dma_address(s) = DMA_ERROR_CODE;
+		sg_dma_len(s) = 0;
+
+		if (seg_len && (seg_dma + seg_len == dma_addr + s_offset) &&
+		    (seg_len + s_dma_len <= max_len) &&
+		    ((seg_dma & seg_mask) <= seg_mask - (seg_len + s_length))
+		   ) {
+			sg_dma_len(seg) += s_dma_len;
+		} else {
+			if (seg_len) {
+				seg = sg_next(seg);
+				count++;
+			}
+			sg_dma_len(seg) = s_dma_len;
+			sg_dma_address(seg) = dma_addr + s_offset;
+
+			seg_len = s_offset;
+			seg_dma = dma_addr + s_offset;
+		}
+		seg_len += s_length;
+		dma_addr += s_dma_len;
+	}
+	return count;
+}
+
+static void __invalidate_sg(struct scatterlist *sg, int nents)
+{
+	struct scatterlist *s;
+	int i;
+
+	for_each_sg(sg, s, nents, i) {
+		if (sg_dma_address(s) != DMA_ERROR_CODE)
+			s->offset = sg_dma_address(s);
+		if (sg_dma_len(s))
+			s->length = sg_dma_len(s);
+		sg_dma_address(s) = DMA_ERROR_CODE;
+		sg_dma_len(s) = 0;
+	}
+}
+
+int iommu_dma_map_sg(struct device *dev, struct scatterlist *sg,
+		int nents, int prot, bool coherent)
+{
+	struct iommu_dma_domain *dom = arch_get_dma_domain(dev);
+	struct iova_domain *iovad = dom->iovad;
+	struct iova *iova;
+	struct scatterlist *s;
+	dma_addr_t dma_addr;
+	size_t iova_len = 0;
+	int i;
+
+	/*
+	 * Work out how much IOVA space we need, and align the segments to
+	 * IOVA granules for the IOMMU driver to handle. With some clever
+	 * trickery we can modify the list in a reversible manner.
+	 */
+	for_each_sg(sg, s, nents, i) {
+		size_t s_offset = iova_offset(iovad, s->offset);
+		size_t s_length = s->length;
+
+		sg_dma_address(s) = s->offset;
+		sg_dma_len(s) = s_length;
+		s->offset -= s_offset;
+		s_length = iova_align(iovad, s_length + s_offset);
+		s->length = s_length;
+
+		iova_len += s_length;
+	}
+
+	iova = __alloc_iova(dev, iova_len, coherent);
+	if (!iova)
+		goto out_restore_sg;
+
+	/*
+	 * We'll leave any physical concatenation to the IOMMU driver's
+	 * implementation - it knows better than we do.
+	 */
+	dma_addr = iova_dma_addr(iovad, iova);
+	if (iommu_map_sg(dom->domain, dma_addr, sg, nents, prot) < iova_len)
+		goto out_free_iova;
+
+	return __finalise_sg(dev, sg, nents, dev_dma_addr(dev, dma_addr));
+
+out_free_iova:
+	__free_iova(iovad, iova);
+out_restore_sg:
+	__invalidate_sg(sg, nents);
+	return 0;
+}
+
+void iommu_dma_unmap_sg(struct device *dev, struct scatterlist *sg, int nents,
+		enum dma_data_direction dir, struct dma_attrs *attrs)
+{
+	/*
+	 * The scatterlist segments are mapped contiguously
+	 * in IOVA space, so this is incredibly easy.
+	 */
+	__iommu_dma_unmap(arch_get_dma_domain(dev), sg_dma_address(sg));
+}
+
+int iommu_dma_supported(struct device *dev, u64 mask)
+{
+	/*
+	 * 'Special' IOMMUs which don't have the same addressing capability
+	 * as the CPU will have to wait until we have some way to query that
+	 * before they'll be able to use this framework.
+	 */
+	return 1;
+}
+
+int iommu_dma_mapping_error(struct device *dev, dma_addr_t dma_addr)
+{
+	return dma_addr == DMA_ERROR_CODE;
+}
diff --git a/include/linux/dma-iommu.h b/include/linux/dma-iommu.h
new file mode 100644
index 0000000..6708a8a
--- /dev/null
+++ b/include/linux/dma-iommu.h
@@ -0,0 +1,94 @@
+/*
+ * Copyright (C) 2014-2015 ARM Ltd.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program.  If not, see <http://www.gnu.org/licenses/>.
+ */
+#ifndef __DMA_IOMMU_H
+#define __DMA_IOMMU_H
+
+#ifdef __KERNEL__
+
+#include <linux/dma-mapping.h>
+
+#ifdef CONFIG_IOMMU_DMA
+
+int iommu_dma_init(void);
+
+/* Fundamental domain management */
+struct iommu_dma_domain *iommu_dma_create_domain(const struct iommu_ops *ops,
+		dma_addr_t base, u64 size);
+void iommu_dma_release_domain(struct iommu_dma_domain *dma_domain);
+
+int iommu_dma_attach_device(struct device *dev, struct iommu_dma_domain *domain);
+void iommu_dma_detach_device(struct device *dev);
+
+/* Helpers for DMA-API <-> IOMMU-API interaction */
+struct iommu_domain *iommu_dma_raw_domain(struct iommu_dma_domain *dma_domain);
+int dma_direction_to_prot(enum dma_data_direction dir, bool coherent);
+
+/*
+ * Arch code must provide implementations of these - it can associate domains
+ * with devices however it likes, provided the lookup is efficient
+ */
+static inline struct iommu_dma_domain *arch_get_dma_domain(struct device *dev);
+static inline void arch_set_dma_domain(struct device *dev,
+		struct iommu_dma_domain *dma_domain);
+
+/*
+ * These implement the bulk of the relevant DMA mapping callbacks, but require
+ * the arch code to handle attributes and provide IOMMU-specific parameters
+ */
+struct page **iommu_dma_alloc(struct device *dev, size_t size, gfp_t gfp,
+		int prot, bool coherent, dma_addr_t *handle,
+		void (*flush_page)(const void *, phys_addr_t));
+void iommu_dma_free(struct device *dev, struct page **pages, size_t size,
+		dma_addr_t *handle);
+
+int iommu_dma_mmap(struct page **pages, size_t size, struct vm_area_struct *vma);
+
+dma_addr_t iommu_dma_map_page(struct device *dev, struct page *page,
+		unsigned long offset, size_t size, int prot, bool coherent);
+int iommu_dma_map_sg(struct device *dev, struct scatterlist *sg,
+		int nents, int prot, bool coherent);
+
+/*
+ * Arch code with no special attribute handling may use these
+ * directly as DMA mapping callbacks for simplicity
+ */
+void iommu_dma_unmap_page(struct device *dev, dma_addr_t handle, size_t size,
+		enum dma_data_direction dir, struct dma_attrs *attrs);
+void iommu_dma_unmap_sg(struct device *dev, struct scatterlist *sg, int nents,
+		enum dma_data_direction dir, struct dma_attrs *attrs);
+int iommu_dma_supported(struct device *dev, u64 mask);
+int iommu_dma_mapping_error(struct device *dev, dma_addr_t dma_addr);
+
+#else
+
+static inline int iommu_dma_init(void)
+{
+	return 0;
+}
+
+#endif  /* CONFIG_IOMMU_DMA */
+
+static inline struct iommu_dma_domain *arch_get_dma_domain(struct device *dev)
+{
+	return NULL;
+}
+
+static inline void arch_set_dma_domain(struct device *dev,
+		struct iommu_dma_domain *dma_domain)
+{ }
+
+#endif	/* __KERNEL__ */
+#endif	/* __DMA_IOMMU_H */
-- 
1.9.1

Re: [PATCH 2/4] iommu: Implement common IOMMU ops for DMA mapping

[Yong Wu]

Hi Robin,
    Thanks.

    While we test venc in v4l2, we get a problem:
    When we enter the funtion[0], it will be break unexpectedly in the
funcion[1] while the offset of sg table is not zero. It is ok if the
offset is zero. Then I add more log in dma-iommu.c, please help check
below.
    All we tested it based on dma v2. and have not tested it on v3 yet.
The code of iommu-map-sg seems the same. if it's fixed in v3, I'm very
sorry. The map_sg in mtk-iommu use default_iommu_map_sg.
    Any question please tell me, Thanks very much. 

[0]http://lxr.free-electrons.com/source/drivers/media/v4l2-core/videobuf2-dma-contig.c#L564	[1]http://lxr.free-electrons.com/source/drivers/media/v4l2-core/videobuf2-dma-contig.c#L70


On Wed, 2015-05-27 at 15:09 +0100, Robin Murphy wrote:
> Taking inspiration from the existing arch/arm code, break out some
> generic functions to interface the DMA-API to the IOMMU-API. This will
> do the bulk of the heavy lifting for IOMMU-backed dma-mapping.
> 
> Signed-off-by: Robin Murphy <robin.murphy-5wv7dgnIgG8@public.gmane.org>
> ---
>  drivers/iommu/Kconfig     |   7 +
>  drivers/iommu/Makefile    |   1 +
>  drivers/iommu/dma-iommu.c | 560 ++++++++++++++++++++++++++++++++++++++++++++++
>  include/linux/dma-iommu.h |  94 ++++++++
>  4 files changed, 662 insertions(+)
>  create mode 100644 drivers/iommu/dma-iommu.c
>  create mode 100644 include/linux/dma-iommu.h
> 
[snip]
> +static int __finalise_sg(struct device *dev, struct scatterlist *sg, int nents,
> +		dma_addr_t dma_addr)
> +{
> +	struct scatterlist *s, *seg = sg;
> +	unsigned long seg_mask = dma_get_seg_boundary(dev);
> +	unsigned int max_len = dma_get_max_seg_size(dev);
> +	unsigned int seg_len = 0, seg_dma = 0;
> +	int i, count = 1;
> +
> +	for_each_sg(sg, s, nents, i) {
> +		/* Un-swizzling the fields here, hence the naming mismatch */
> +		unsigned int s_offset = sg_dma_address(s);
> +		unsigned int s_length = sg_dma_len(s);
> +		unsigned int s_dma_len = s->length;
> +
> +		s->offset = s_offset;
> +		s->length = s_length;
> +		sg_dma_address(s) = DMA_ERROR_CODE;
> +		sg_dma_len(s) = 0;
> +
> +		if (seg_len && (seg_dma + seg_len == dma_addr + s_offset) &&
> +		    (seg_len + s_dma_len <= max_len) &&
> +		    ((seg_dma & seg_mask) <= seg_mask - (seg_len + s_length))
> +		   ) {
> +			sg_dma_len(seg) += s_dma_len;
> +		} else {
> +			if (seg_len) {
> +				seg = sg_next(seg);
> +				count++;
> +			}
> +			sg_dma_len(seg) = s_dma_len;
> +			sg_dma_address(seg) = dma_addr + s_offset;
   Here the value of sg_dma_address have added s_offset, but
sg_dma_len(seg) still is s_dma_len.
   In the first loop, s_dma_len is from s->length which is alignd by
"s_length = iova_align(iovad, s_length + s_offset);" in
the interface iommu_dma_map_sg.
> +
> +			seg_len = s_offset;
> +			seg_dma = dma_addr + s_offset;
> +		}
> +		seg_len += s_length;
> +		dma_addr += s_dma_len;
> +	}
> +	return count;
> +}
> +
> +static void __invalidate_sg(struct scatterlist *sg, int nents)
> +{
> +	struct scatterlist *s;
> +	int i;
> +
> +	for_each_sg(sg, s, nents, i) {
> +		if (sg_dma_address(s) != DMA_ERROR_CODE)
> +			s->offset = sg_dma_address(s);
> +		if (sg_dma_len(s))
> +			s->length = sg_dma_len(s);
> +		sg_dma_address(s) = DMA_ERROR_CODE;
> +		sg_dma_len(s) = 0;
> +	}
> +}
> +
> +int iommu_dma_map_sg(struct device *dev, struct scatterlist *sg,
> +		int nents, int prot, bool coherent)
> +{
> +	struct iommu_dma_domain *dom = arch_get_dma_domain(dev);
> +	struct iova_domain *iovad = dom->iovad;
> +	struct iova *iova;
> +	struct scatterlist *s;
> +	dma_addr_t dma_addr;
> +	size_t iova_len = 0;
> +	int i;
> +
> +	/*
> +	 * Work out how much IOVA space we need, and align the segments to
> +	 * IOVA granules for the IOMMU driver to handle. With some clever
> +	 * trickery we can modify the list in a reversible manner.
> +	 */
> +	for_each_sg(sg, s, nents, i) {
> +		size_t s_offset = iova_offset(iovad, s->offset);
> +		size_t s_length = s->length;
> +
> +		sg_dma_address(s) = s->offset;
> +		sg_dma_len(s) = s_length;
> +		s->offset -= s_offset;
> +		s_length = iova_align(iovad, s_length + s_offset);
> +		s->length = s_length;
At the begging, s->length is the length of valid data. but it's aligned
here.
> +
> +		iova_len += s_length;
> +	}
> +
> +	iova = __alloc_iova(dev, iova_len, coherent);
> +	if (!iova)
> +		goto out_restore_sg;
> +
> +	/*
> +	 * We'll leave any physical concatenation to the IOMMU driver's
> +	 * implementation - it knows better than we do.
> +	 */
> +	dma_addr = iova_dma_addr(iovad, iova);
> +	if (iommu_map_sg(dom->domain, dma_addr, sg, nents, prot) < iova_len)
> +		goto out_free_iova;
> +
> +	return __finalise_sg(dev, sg, nents, dev_dma_addr(dev, dma_addr));
> +
> +out_free_iova:
> +	__free_iova(iovad, iova);
> +out_restore_sg:
> +	__invalidate_sg(sg, nents);
> +	return 0;
> +}
> +
[snip]

Re: [PATCH 2/4] iommu: Implement common IOMMU ops for DMA mapping

[Yingjoe Chen]

Hi Robin,

More info, hope this make it clearer. We are calling dma_map_sg_attrs
with the following 2 sg. With IOMMU, we are expecting it got merged into
1 contiguous va range, but instead we get 2 va range.

sg0 dma_address 0xfeeddc00 size 0x400, offset 0xc00
sg1 dma_address 0xfeede000 size 0x1000, offset 0x0

Joe.C

On Fri, 2015-05-29 at 13:26 +0800, Yong Wu wrote:
> Hi Robin,
>     Thanks.
> 
>     While we test venc in v4l2, we get a problem:
>     When we enter the funtion[0], it will be break unexpectedly in the
> funcion[1] while the offset of sg table is not zero. It is ok if the
> offset is zero. Then I add more log in dma-iommu.c, please help check
> below.
>     All we tested it based on dma v2. and have not tested it on v3 yet.
> The code of iommu-map-sg seems the same. if it's fixed in v3, I'm very
> sorry. The map_sg in mtk-iommu use default_iommu_map_sg.
>     Any question please tell me, Thanks very much. 
> 
> [0]http://lxr.free-electrons.com/source/drivers/media/v4l2-core/videobuf2-dma-contig.c#L564	[1]http://lxr.free-electrons.com/source/drivers/media/v4l2-core/videobuf2-dma-contig.c#L70
> 
> 
> On Wed, 2015-05-27 at 15:09 +0100, Robin Murphy wrote:
> > Taking inspiration from the existing arch/arm code, break out some
> > generic functions to interface the DMA-API to the IOMMU-API. This will
> > do the bulk of the heavy lifting for IOMMU-backed dma-mapping.
> > 
> > Signed-off-by: Robin Murphy <robin.murphy@arm.com>
> > ---
> >  drivers/iommu/Kconfig     |   7 +
> >  drivers/iommu/Makefile    |   1 +
> >  drivers/iommu/dma-iommu.c | 560 ++++++++++++++++++++++++++++++++++++++++++++++
> >  include/linux/dma-iommu.h |  94 ++++++++
> >  4 files changed, 662 insertions(+)
> >  create mode 100644 drivers/iommu/dma-iommu.c
> >  create mode 100644 include/linux/dma-iommu.h
> > 
> [snip]
> > +static int __finalise_sg(struct device *dev, struct scatterlist *sg, int nents,
> > +		dma_addr_t dma_addr)
> > +{
> > +	struct scatterlist *s, *seg = sg;
> > +	unsigned long seg_mask = dma_get_seg_boundary(dev);
> > +	unsigned int max_len = dma_get_max_seg_size(dev);
> > +	unsigned int seg_len = 0, seg_dma = 0;
> > +	int i, count = 1;
> > +
> > +	for_each_sg(sg, s, nents, i) {
> > +		/* Un-swizzling the fields here, hence the naming mismatch */
> > +		unsigned int s_offset = sg_dma_address(s);
> > +		unsigned int s_length = sg_dma_len(s);
> > +		unsigned int s_dma_len = s->length;
> > +
> > +		s->offset = s_offset;
> > +		s->length = s_length;
> > +		sg_dma_address(s) = DMA_ERROR_CODE;
> > +		sg_dma_len(s) = 0;
> > +
> > +		if (seg_len && (seg_dma + seg_len == dma_addr + s_offset) &&
> > +		    (seg_len + s_dma_len <= max_len) &&
> > +		    ((seg_dma & seg_mask) <= seg_mask - (seg_len + s_length))
> > +		   ) {
> > +			sg_dma_len(seg) += s_dma_len;
> > +		} else {
> > +			if (seg_len) {
> > +				seg = sg_next(seg);
> > +				count++;
> > +			}
> > +			sg_dma_len(seg) = s_dma_len;
> > +			sg_dma_address(seg) = dma_addr + s_offset;
>    Here the value of sg_dma_address have added s_offset, but
> sg_dma_len(seg) still is s_dma_len.
>    In the first loop, s_dma_len is from s->length which is alignd by
> "s_length = iova_align(iovad, s_length + s_offset);" in
> the interface iommu_dma_map_sg.
> > +
> > +			seg_len = s_offset;
> > +			seg_dma = dma_addr + s_offset;
> > +		}
> > +		seg_len += s_length;
> > +		dma_addr += s_dma_len;
> > +	}
> > +	return count;
> > +}
> > +
> > +static void __invalidate_sg(struct scatterlist *sg, int nents)
> > +{
> > +	struct scatterlist *s;
> > +	int i;
> > +
> > +	for_each_sg(sg, s, nents, i) {
> > +		if (sg_dma_address(s) != DMA_ERROR_CODE)
> > +			s->offset = sg_dma_address(s);
> > +		if (sg_dma_len(s))
> > +			s->length = sg_dma_len(s);
> > +		sg_dma_address(s) = DMA_ERROR_CODE;
> > +		sg_dma_len(s) = 0;
> > +	}
> > +}
> > +
> > +int iommu_dma_map_sg(struct device *dev, struct scatterlist *sg,
> > +		int nents, int prot, bool coherent)
> > +{
> > +	struct iommu_dma_domain *dom = arch_get_dma_domain(dev);
> > +	struct iova_domain *iovad = dom->iovad;
> > +	struct iova *iova;
> > +	struct scatterlist *s;
> > +	dma_addr_t dma_addr;
> > +	size_t iova_len = 0;
> > +	int i;
> > +
> > +	/*
> > +	 * Work out how much IOVA space we need, and align the segments to
> > +	 * IOVA granules for the IOMMU driver to handle. With some clever
> > +	 * trickery we can modify the list in a reversible manner.
> > +	 */
> > +	for_each_sg(sg, s, nents, i) {
> > +		size_t s_offset = iova_offset(iovad, s->offset);
> > +		size_t s_length = s->length;
> > +
> > +		sg_dma_address(s) = s->offset;
> > +		sg_dma_len(s) = s_length;
> > +		s->offset -= s_offset;
> > +		s_length = iova_align(iovad, s_length + s_offset);
> > +		s->length = s_length;
> At the begging, s->length is the length of valid data. but it's aligned
> here.
> > +
> > +		iova_len += s_length;
> > +	}
> > +
> > +	iova = __alloc_iova(dev, iova_len, coherent);
> > +	if (!iova)
> > +		goto out_restore_sg;
> > +
> > +	/*
> > +	 * We'll leave any physical concatenation to the IOMMU driver's
> > +	 * implementation - it knows better than we do.
> > +	 */
> > +	dma_addr = iova_dma_addr(iovad, iova);
> > +	if (iommu_map_sg(dom->domain, dma_addr, sg, nents, prot) < iova_len)
> > +		goto out_free_iova;
> > +
> > +	return __finalise_sg(dev, sg, nents, dev_dma_addr(dev, dma_addr));
> > +
> > +out_free_iova:
> > +	__free_iova(iovad, iova);
> > +out_restore_sg:
> > +	__invalidate_sg(sg, nents);
> > +	return 0;
> > +}
> > +
> [snip]
> 
> 

Re: [PATCH 2/4] iommu: Implement common IOMMU ops for DMA mapping

[Robin Murphy]

On 29/05/15 06:26, Yong Wu wrote:
> Hi Robin,
>      Thanks.
>
>      While we test venc in v4l2, we get a problem:

Thanks as always for testing!

>      When we enter the funtion[0], it will be break unexpectedly in the
> funcion[1] while the offset of sg table is not zero. It is ok if the
> offset is zero. Then I add more log in dma-iommu.c, please help check
> below.

Bah, non-zero offset was about the only scatterlist case I wasn't able 
to get out of the EHCI driver...

>      All we tested it based on dma v2. and have not tested it on v3 yet.
> The code of iommu-map-sg seems the same. if it's fixed in v3, I'm very
> sorry. The map_sg in mtk-iommu use default_iommu_map_sg.
>      Any question please tell me, Thanks very much.
>
> [0]http://lxr.free-electrons.com/source/drivers/media/v4l2-core/videobuf2-dma-contig.c#L564	[1]http://lxr.free-electrons.com/source/drivers/media/v4l2-core/videobuf2-dma-contig.c#L70
>
>
> On Wed, 2015-05-27 at 15:09 +0100, Robin Murphy wrote:
>> Taking inspiration from the existing arch/arm code, break out some
>> generic functions to interface the DMA-API to the IOMMU-API. This will
>> do the bulk of the heavy lifting for IOMMU-backed dma-mapping.
>>
>> Signed-off-by: Robin Murphy <robin.murphy-5wv7dgnIgG8@public.gmane.org>
>> ---
>>   drivers/iommu/Kconfig     |   7 +
>>   drivers/iommu/Makefile    |   1 +
>>   drivers/iommu/dma-iommu.c | 560 ++++++++++++++++++++++++++++++++++++++++++++++
>>   include/linux/dma-iommu.h |  94 ++++++++
>>   4 files changed, 662 insertions(+)
>>   create mode 100644 drivers/iommu/dma-iommu.c
>>   create mode 100644 include/linux/dma-iommu.h
>>
> [snip]
>> +static int __finalise_sg(struct device *dev, struct scatterlist *sg, int nents,
>> +		dma_addr_t dma_addr)
>> +{
>> +	struct scatterlist *s, *seg = sg;
>> +	unsigned long seg_mask = dma_get_seg_boundary(dev);
>> +	unsigned int max_len = dma_get_max_seg_size(dev);
>> +	unsigned int seg_len = 0, seg_dma = 0;
>> +	int i, count = 1;
>> +
>> +	for_each_sg(sg, s, nents, i) {
>> +		/* Un-swizzling the fields here, hence the naming mismatch */
>> +		unsigned int s_offset = sg_dma_address(s);
>> +		unsigned int s_length = sg_dma_len(s);
>> +		unsigned int s_dma_len = s->length;
>> +
>> +		s->offset = s_offset;
>> +		s->length = s_length;
>> +		sg_dma_address(s) = DMA_ERROR_CODE;
>> +		sg_dma_len(s) = 0;
>> +
>> +		if (seg_len && (seg_dma + seg_len == dma_addr + s_offset) &&
>> +		    (seg_len + s_dma_len <= max_len) &&
>> +		    ((seg_dma & seg_mask) <= seg_mask - (seg_len + s_length))
>> +		   ) {
>> +			sg_dma_len(seg) += s_dma_len;
>> +		} else {
>> +			if (seg_len) {
>> +				seg = sg_next(seg);
>> +				count++;
>> +			}
>> +			sg_dma_len(seg) = s_dma_len;

Does changing this to
			sg_dma_len(seg) = s_dma_len - s_offset;

make things behave as expected? Since the new unmap code no longer 
relies on adding up the individual segments to equal the IOVA size I 
don't _think_ that'll break anything here.

Robin.

[PATCH 3/4] arm64: Add IOMMU dma_ops

[Robin Murphy]

Taking some inspiration from the arch/arm code, implement the
arch-specific side of the DMA mapping ops using the new IOMMU-DMA layer.

Whilst proliferating per-device private IOMMU data via dev->archdata is
less than ideal, it will do the job for now, especially since we can't
easily handle the kind of problematic system topologies in the current
IOMMU API anyway.

Signed-off-by: Robin Murphy <robin.murphy-5wv7dgnIgG8@public.gmane.org>
---
 arch/arm64/include/asm/device.h      |   3 +
 arch/arm64/include/asm/dma-mapping.h |  14 ++
 arch/arm64/mm/dma-mapping.c          | 342 +++++++++++++++++++++++++++++++++++
 include/linux/dma-iommu.h            |   4 +-
 4 files changed, 361 insertions(+), 2 deletions(-)

diff --git a/arch/arm64/include/asm/device.h b/arch/arm64/include/asm/device.h
index 243ef25..510cee1 100644
--- a/arch/arm64/include/asm/device.h
+++ b/arch/arm64/include/asm/device.h
@@ -20,6 +20,9 @@ struct dev_archdata {
 	struct dma_map_ops *dma_ops;
 #ifdef CONFIG_IOMMU_API
 	void *iommu;			/* private IOMMU data */
+#ifdef CONFIG_IOMMU_DMA
+	struct iommu_dma_domain *dma_domain;
+#endif
 #endif
 	bool dma_coherent;
 };
diff --git a/arch/arm64/include/asm/dma-mapping.h b/arch/arm64/include/asm/dma-mapping.h
index 9437e3d..835a8d1 100644
--- a/arch/arm64/include/asm/dma-mapping.h
+++ b/arch/arm64/include/asm/dma-mapping.h
@@ -61,6 +61,20 @@ static inline bool is_device_dma_coherent(struct device *dev)
 }
 
 #include <asm-generic/dma-mapping-common.h>
+#include <linux/dma-iommu.h>
+
+#ifdef CONFIG_IOMMU_DMA
+static inline struct iommu_dma_domain *arch_get_dma_domain(struct device *dev)
+{
+	return dev->archdata.dma_domain;
+}
+
+static inline void arch_set_dma_domain(struct device *dev,
+				  struct iommu_dma_domain *dma_domain)
+{
+	dev->archdata.dma_domain = dma_domain;
+}
+#endif
 
 static inline dma_addr_t phys_to_dma(struct device *dev, phys_addr_t paddr)
 {
diff --git a/arch/arm64/mm/dma-mapping.c b/arch/arm64/mm/dma-mapping.c
index b0bd4e5..189477b 100644
--- a/arch/arm64/mm/dma-mapping.c
+++ b/arch/arm64/mm/dma-mapping.c
@@ -434,3 +434,345 @@ static int __init dma_debug_do_init(void)
 	return 0;
 }
 fs_initcall(dma_debug_do_init);
+
+
+#ifdef CONFIG_IOMMU_DMA
+#include <linux/iommu.h>
+#include <linux/platform_device.h>
+#include <linux/amba/bus.h>
+
+/* Thankfully, all cache ops are by VA so we can ignore phys here */
+static void flush_page(const void *virt, phys_addr_t phys)
+{
+	__dma_flush_range(virt, virt + PAGE_SIZE);
+}
+
+static void *__iommu_alloc_attrs(struct device *dev, size_t size,
+				 dma_addr_t *handle, gfp_t gfp,
+				 struct dma_attrs *attrs)
+{
+	bool coherent = is_device_dma_coherent(dev);
+	pgprot_t pgprot = coherent ? __pgprot(PROT_NORMAL) :
+				     __pgprot(PROT_NORMAL_NC);
+	int ioprot;
+	void *addr;
+
+	if (WARN(!dev, "cannot create IOMMU mapping for unknown device\n"))
+		return NULL;
+
+	if (!(gfp & __GFP_WAIT)) {
+		struct page *page;
+
+		addr = __alloc_from_pool(size, &page, gfp);
+		if (!addr)
+			return NULL;
+
+		ioprot = dma_direction_to_prot(DMA_BIDIRECTIONAL, false);
+		*handle = iommu_dma_map_page(dev, page, 0, size, ioprot, coherent);
+		if (iommu_dma_mapping_error(dev, *handle)) {
+			__free_from_pool(addr, size);
+			addr = NULL;
+		}
+	} else {
+		struct page **pages;
+
+		ioprot = dma_direction_to_prot(DMA_BIDIRECTIONAL, coherent);
+		pages = iommu_dma_alloc(dev, size, gfp, ioprot,	coherent,
+					handle, coherent ? NULL : flush_page);
+		if (!pages)
+			return NULL;
+
+		addr = dma_common_pages_remap(pages, size, VM_USERMAP,
+				__get_dma_pgprot(attrs, pgprot, coherent),
+				__builtin_return_address(0));
+		if (!addr)
+			iommu_dma_free(dev, pages, size, handle);
+	}
+	return addr;
+}
+
+static void __iommu_free_attrs(struct device *dev, size_t size, void *cpu_addr,
+			       dma_addr_t handle, struct dma_attrs *attrs)
+{
+	if (__free_from_pool(cpu_addr, size)) {
+		iommu_dma_unmap_page(dev, handle, size, 0, NULL);
+	} else {
+		struct vm_struct *area = find_vm_area(cpu_addr);
+
+		if (WARN_ON(!area || !area->pages))
+			return;
+		iommu_dma_free(dev, area->pages, size, &handle);
+		dma_common_free_remap(cpu_addr, size, VM_USERMAP);
+	}
+}
+
+static int __iommu_mmap_attrs(struct device *dev, struct vm_area_struct *vma,
+			      void *cpu_addr, dma_addr_t dma_addr, size_t size,
+			      struct dma_attrs *attrs)
+{
+	struct vm_struct *area;
+	int ret;
+
+	vma->vm_page_prot = __get_dma_pgprot(attrs, vma->vm_page_prot,
+					     is_device_dma_coherent(dev));
+
+	if (dma_mmap_from_coherent(dev, vma, cpu_addr, size, &ret))
+		return ret;
+
+	area = find_vm_area(cpu_addr);
+	if (WARN_ON(!area || area->pages))
+		return -ENXIO;
+
+	return iommu_dma_mmap(area->pages, size, vma);
+}
+
+static int __iommu_get_sgtable(struct device *dev, struct sg_table *sgt,
+			       void *cpu_addr, dma_addr_t dma_addr,
+			       size_t size, struct dma_attrs *attrs)
+{
+	unsigned int count = PAGE_ALIGN(size) >> PAGE_SHIFT;
+	struct vm_struct *area = find_vm_area(cpu_addr);
+
+	if (WARN_ON(!area || !area->pages))
+		return -ENXIO;
+
+	return sg_alloc_table_from_pages(sgt, area->pages, count, 0, size,
+					 GFP_KERNEL);
+}
+
+static void __iommu_sync_single_for_cpu(struct device *dev,
+					dma_addr_t dev_addr, size_t size,
+					enum dma_data_direction dir)
+{
+	struct iommu_dma_domain *dma_domain = arch_get_dma_domain(dev);
+	phys_addr_t phys;
+
+	if (is_device_dma_coherent(dev))
+		return;
+
+	phys = iommu_iova_to_phys(iommu_dma_raw_domain(dma_domain), dev_addr);
+	__dma_unmap_area(phys_to_virt(phys), size, dir);
+}
+
+static void __iommu_sync_single_for_device(struct device *dev,
+					   dma_addr_t dev_addr, size_t size,
+					   enum dma_data_direction dir)
+{
+	struct iommu_dma_domain *dma_domain = arch_get_dma_domain(dev);
+	phys_addr_t phys;
+
+	if (is_device_dma_coherent(dev))
+		return;
+
+	phys = iommu_iova_to_phys(iommu_dma_raw_domain(dma_domain), dev_addr);
+	__dma_map_area(phys_to_virt(phys), size, dir);
+}
+
+static dma_addr_t __iommu_map_page(struct device *dev, struct page *page,
+				   unsigned long offset, size_t size,
+				   enum dma_data_direction dir,
+				   struct dma_attrs *attrs)
+{
+	bool coherent = is_device_dma_coherent(dev);
+	int prot = dma_direction_to_prot(dir, coherent);
+	dma_addr_t dev_addr = iommu_dma_map_page(dev, page, offset, size,
+						 prot, coherent);
+
+	if (!iommu_dma_mapping_error(dev, dev_addr) &&
+	    !dma_get_attr(DMA_ATTR_SKIP_CPU_SYNC, attrs))
+		__iommu_sync_single_for_device(dev, dev_addr, size, dir);
+
+	return dev_addr;
+}
+
+static void __iommu_unmap_page(struct device *dev, dma_addr_t dev_addr,
+			       size_t size, enum dma_data_direction dir,
+			       struct dma_attrs *attrs)
+{
+	if (!dma_get_attr(DMA_ATTR_SKIP_CPU_SYNC, attrs))
+		__iommu_sync_single_for_cpu(dev, dev_addr, size, dir);
+
+	iommu_dma_unmap_page(dev, dev_addr, size, dir, attrs);
+}
+
+static void __iommu_sync_sg_for_cpu(struct device *dev,
+				    struct scatterlist *sgl, int nelems,
+				    enum dma_data_direction dir)
+{
+	struct scatterlist *sg;
+	int i;
+
+	if (is_device_dma_coherent(dev))
+		return;
+
+	for_each_sg(sgl, sg, nelems, i)
+		__dma_unmap_area(sg_virt(sg), sg->length, dir);
+}
+
+static void __iommu_sync_sg_for_device(struct device *dev,
+				       struct scatterlist *sgl, int nelems,
+				       enum dma_data_direction dir)
+{
+	struct scatterlist *sg;
+	int i;
+
+	if (is_device_dma_coherent(dev))
+		return;
+
+	for_each_sg(sgl, sg, nelems, i)
+		__dma_map_area(sg_virt(sg), sg->length, dir);
+}
+
+static int __iommu_map_sg_attrs(struct device *dev, struct scatterlist *sgl,
+				int nelems, enum dma_data_direction dir,
+				struct dma_attrs *attrs)
+{
+	bool coherent = is_device_dma_coherent(dev);
+
+	if (!dma_get_attr(DMA_ATTR_SKIP_CPU_SYNC, attrs))
+		__iommu_sync_sg_for_device(dev, sgl, nelems, dir);
+
+	return iommu_dma_map_sg(dev, sgl, nelems,
+			dma_direction_to_prot(dir, coherent),
+			coherent);
+}
+
+static void __iommu_unmap_sg_attrs(struct device *dev,
+				   struct scatterlist *sgl, int nelems,
+				   enum dma_data_direction dir,
+				   struct dma_attrs *attrs)
+{
+	if (!dma_get_attr(DMA_ATTR_SKIP_CPU_SYNC, attrs))
+		__iommu_sync_sg_for_cpu(dev, sgl, nelems, dir);
+
+	iommu_dma_unmap_sg(dev, sgl, nelems, dir, attrs);
+}
+
+static struct dma_map_ops iommu_dma_ops = {
+	.alloc = __iommu_alloc_attrs,
+	.free = __iommu_free_attrs,
+	.mmap = __iommu_mmap_attrs,
+	.get_sgtable = __iommu_get_sgtable,
+	.map_page = __iommu_map_page,
+	.unmap_page = __iommu_unmap_page,
+	.map_sg = __iommu_map_sg_attrs,
+	.unmap_sg = __iommu_unmap_sg_attrs,
+	.sync_single_for_cpu = __iommu_sync_single_for_cpu,
+	.sync_single_for_device = __iommu_sync_single_for_device,
+	.sync_sg_for_cpu = __iommu_sync_sg_for_cpu,
+	.sync_sg_for_device = __iommu_sync_sg_for_device,
+	.dma_supported = iommu_dma_supported,
+	.mapping_error = iommu_dma_mapping_error,
+};
+
+struct iommu_dma_notifier_data {
+	struct list_head list;
+	struct device *dev;
+	struct iommu_dma_domain *dma_domain;
+};
+static LIST_HEAD(iommu_dma_masters);
+static DEFINE_MUTEX(iommu_dma_notifier_lock);
+
+static int __iommu_attach_notifier(struct notifier_block *nb,
+				   unsigned long action, void *data)
+{
+	struct iommu_dma_notifier_data *master, *tmp;
+
+	if (action != BUS_NOTIFY_ADD_DEVICE)
+		return 0;
+	/*
+	 * We expect the list to only contain the most recent addition,
+	 * which *should* be the same device as @data, so just process the
+	 * whole thing blindly. If any previous attachments did happen to
+	 * fail, they get a free retry since the domains are still live.
+	 */
+	mutex_lock(&iommu_dma_notifier_lock);
+	list_for_each_entry_safe(master, tmp, &iommu_dma_masters, list) {
+		if (iommu_dma_attach_device(master->dev, master->dma_domain)) {
+			pr_warn("Failed to attach device %s to IOMMU mapping; retaining platform DMA ops\n",
+				dev_name(master->dev));
+		} else {
+			master->dev->archdata.dma_ops = &iommu_dma_ops;
+			/* it's safe to drop the initial refcount now */
+			iommu_dma_release_domain(master->dma_domain);
+			list_del(&master->list);
+			kfree(master);
+		}
+	}
+	mutex_unlock(&iommu_dma_notifier_lock);
+	return 0;
+}
+
+static int register_iommu_dma_ops_notifier(struct bus_type *bus)
+{
+	int ret;
+	struct notifier_block *nb = kzalloc(sizeof(*nb), GFP_KERNEL);
+
+	/*
+	 * The device must be attached to a domain before its driver probe,
+	 * in case the driver allocates DMA buffers immediately. However, most
+	 * IOMMU drivers are currently configuring groups in their add_device
+	 * callback, so the attach should happen after that. Since the IOMMU
+	 * core uses a bus notifier for add_device, do the same but with a
+	 * stupidly low priority to ensure the appropriate ordering.
+	 *
+	 * This can hopefully all go away once we have default domains in the
+	 * IOMMU core.
+	 */
+	nb->notifier_call = __iommu_attach_notifier;
+	nb->priority = INT_MIN;
+
+	ret = bus_register_notifier(bus, nb);
+	if (ret) {
+		pr_warn("Failed to register DMA domain notifier; IOMMU DMA ops unavailable on bus '%s'\n",
+			bus->name);
+		kfree(nb);
+	}
+	return ret;
+}
+
+static int __init arm64_iommu_dma_init(void)
+{
+	int ret;
+
+	ret = iommu_dma_init();
+	if (!ret)
+		ret = register_iommu_dma_ops_notifier(&platform_bus_type);
+	if (!ret)
+		ret = register_iommu_dma_ops_notifier(&amba_bustype);
+	return ret;
+}
+arch_initcall(arm64_iommu_dma_init);
+
+static void __iommu_setup_dma_ops(struct device *dev, u64 dma_base, u64 size,
+				  const struct iommu_ops *ops)
+{
+	struct iommu_dma_notifier_data *iommudata;
+
+	if (!ops)
+		return;
+
+	iommudata = kzalloc(sizeof(*iommudata), GFP_KERNEL);
+	if (!iommudata)
+		return;
+
+	iommudata->dev = dev;
+	iommudata->dma_domain = iommu_dma_create_domain(ops, dma_base, size);
+	if (!iommudata->dma_domain) {
+		pr_warn("Failed to create %llu-byte IOMMU mapping for device %s\n",
+				size, dev_name(dev));
+		kfree(iommudata);
+		return;
+	}
+	mutex_lock(&iommu_dma_notifier_lock);
+	list_add(&iommudata->list, &iommu_dma_masters);
+	mutex_unlock(&iommu_dma_notifier_lock);
+}
+
+#else
+
+static void __iommu_setup_dma_ops(struct device *dev, u64 dma_base, u64 size,
+				  struct iommu_ops *iommu)
+{ }
+
+#endif  /* CONFIG_IOMMU_DMA */
diff --git a/include/linux/dma-iommu.h b/include/linux/dma-iommu.h
index 6708a8a..8062ed7 100644
--- a/include/linux/dma-iommu.h
+++ b/include/linux/dma-iommu.h
@@ -79,8 +79,6 @@ static inline int iommu_dma_init(void)
 	return 0;
 }
 
-#endif  /* CONFIG_IOMMU_DMA */
-
 static inline struct iommu_dma_domain *arch_get_dma_domain(struct device *dev)
 {
 	return NULL;
@@ -90,5 +88,7 @@ static inline void arch_set_dma_domain(struct device *dev,
 		struct iommu_dma_domain *dma_domain)
 { }
 
+#endif	/* CONFIG_IOMMU_DMA */
+
 #endif	/* __KERNEL__ */
 #endif	/* __DMA_IOMMU_H */
-- 
1.9.1

Re: [PATCH 3/4] arm64: Add IOMMU dma_ops

[Yong Wu]

Hi Robin,
     Sorry to disturb you. When prepare iommu based on this dma-v3. I
meet some problem about how to use the dma-iommu.
     I list some sample code like below:
//========
int mtk_iommu_of_xlate(struct device *dev, struct of_phandle_args *args)
{
        if (args->args_count != 2) {            
                return -EINVAL;
        }

        /* save the private data from args  
         * the private data is larbid and portid in our platform */
        priv->larbid = args->args[0];
        priv->portid = args->args[1];

        /*list the private data */
        list_add_tail(&priv->client, &head->client);

        /* save the list to dev->archdata->iommu */
        ...
}
static int mtk_iommu_init_fn(struct device_node *np)
{
        of_iommu_set_ops(np, &mtk_iommu_ops);   
        return 0;
}

IOMMU_OF_DECLARE(mtkm4u, "mediatek,mt8173-m4u", mtk_iommu_init_fn);
//=======

Question :
  1: if there are 2 client devices use iommu, it will be like this in
dtsi,
                disp:testiommu@1400c000 {
                        compatible = "mediatek,mt8173-test";
                        iommus = <&iommu M4U_LARB0_ID 0>,
                                 <&iommu M4U_LARB1_ID 3>;
                };
                disp1:testiommu@1400D000 {
                        compatible = "mediatek,mt8173-disptest1";
                        iommus = <&iommu M4U_LARB0_ID 8>, 
                                 <&iommu M4U_LARB1_ID 2>;
                };
    Then both devices will enter arch_setup_dma_ops and their parameter
"struct iommu_ops *" is not zero, then it will create two
dma-iommu-domain.
    If we expect all the client device share a iommu domain, then how
should i do?

2. iommu_dma_attach_device will be called automatically in the notify
"__iommu_attach_notifier". But it run before the probe of our iommu
device.
    Then we can't write the register to enable iommu in *_attach_device
because we have not parsed the dtsi at that time.
   ->I have tried to move parse dtsi into mtk_iommu_of_xlate, in order
to read/write register in attach_device.
But it will be warning like this:
(150604_09:43:41.094)WARNING: CPU: 0 PID: 1
at /proj/mtk40525/upstreamdev/chromebook_kernelonly/kernel/mediatek/drivers/base/dd.c:286 driver_probe_device+0x25c/0x29c()

3.In mtk_iommu_probe, we can't get "struct iommu_domain *" from the
current iommu device, we should use it while devm_request_irq.
I have to use the global variable here?!

4.If I don't implement the of_xlate and IOMMU_OF_DECLARE, I call
iommu_dma_create_domain in the mtk_iommu_probe, but I can not
get the "iommu_dma_ops" which is static in arch/arm64/mm/dma-mapping.c
to assign to dev->archdata->dma_ops for the iommu client device.

If i miss something, please tell me and help give some suggestion.
Thanks very much.

On Wed, 2015-05-27 at 15:09 +0100, Robin Murphy wrote:
> Taking some inspiration from the arch/arm code, implement the
> arch-specific side of the DMA mapping ops using the new IOMMU-DMA layer.
> 
> Whilst proliferating per-device private IOMMU data via dev->archdata is
> less than ideal, it will do the job for now, especially since we can't
> easily handle the kind of problematic system topologies in the current
> IOMMU API anyway.
> 
> Signed-off-by: Robin Murphy <robin.murphy-5wv7dgnIgG8@public.gmane.org>
> ---
>  arch/arm64/include/asm/device.h      |   3 +
>  arch/arm64/include/asm/dma-mapping.h |  14 ++
>  arch/arm64/mm/dma-mapping.c          | 342 +++++++++++++++++++++++++++++++++++
>  include/linux/dma-iommu.h            |   4 +-
>  4 files changed, 361 insertions(+), 2 deletions(-)
> 
[snip]
> +struct iommu_dma_notifier_data {
> +	struct list_head list;
> +	struct device *dev;
> +	struct iommu_dma_domain *dma_domain;
> +};
> +static LIST_HEAD(iommu_dma_masters);
> +static DEFINE_MUTEX(iommu_dma_notifier_lock);
> +
> +static int __iommu_attach_notifier(struct notifier_block *nb,
> +				   unsigned long action, void *data)
> +{
> +	struct iommu_dma_notifier_data *master, *tmp;
> +
> +	if (action != BUS_NOTIFY_ADD_DEVICE)
> +		return 0;
> +	/*
> +	 * We expect the list to only contain the most recent addition,
> +	 * which *should* be the same device as @data, so just process the
> +	 * whole thing blindly. If any previous attachments did happen to
> +	 * fail, they get a free retry since the domains are still live.
> +	 */
> +	mutex_lock(&iommu_dma_notifier_lock);
> +	list_for_each_entry_safe(master, tmp, &iommu_dma_masters, list) {
> +		if (iommu_dma_attach_device(master->dev, master->dma_domain)) {
> +			pr_warn("Failed to attach device %s to IOMMU mapping; retaining platform DMA ops\n",
> +				dev_name(master->dev));
> +		} else {
> +			master->dev->archdata.dma_ops = &iommu_dma_ops;
> +			/* it's safe to drop the initial refcount now */
> +			iommu_dma_release_domain(master->dma_domain);
> +			list_del(&master->list);
> +			kfree(master);
> +		}
> +	}
> +	mutex_unlock(&iommu_dma_notifier_lock);
> +	return 0;
> +}
> +
> +static int register_iommu_dma_ops_notifier(struct bus_type *bus)
> +{
> +	int ret;
> +	struct notifier_block *nb = kzalloc(sizeof(*nb), GFP_KERNEL);
> +
> +	/*
> +	 * The device must be attached to a domain before its driver probe,
> +	 * in case the driver allocates DMA buffers immediately. However, most
> +	 * IOMMU drivers are currently configuring groups in their add_device
> +	 * callback, so the attach should happen after that. Since the IOMMU
> +	 * core uses a bus notifier for add_device, do the same but with a
> +	 * stupidly low priority to ensure the appropriate ordering.
> +	 *
> +	 * This can hopefully all go away once we have default domains in the
> +	 * IOMMU core.
> +	 */
> +	nb->notifier_call = __iommu_attach_notifier;
> +	nb->priority = INT_MIN;
> +
> +	ret = bus_register_notifier(bus, nb);
> +	if (ret) {
> +		pr_warn("Failed to register DMA domain notifier; IOMMU DMA ops unavailable on bus '%s'\n",
> +			bus->name);
> +		kfree(nb);
> +	}
> +	return ret;
> +}
> +
> +static int __init arm64_iommu_dma_init(void)
> +{
> +	int ret;
> +
> +	ret = iommu_dma_init();
> +	if (!ret)
> +		ret = register_iommu_dma_ops_notifier(&platform_bus_type);
> +	if (!ret)
> +		ret = register_iommu_dma_ops_notifier(&amba_bustype);
> +	return ret;
> +}
> +arch_initcall(arm64_iommu_dma_init);
> +
> +static void __iommu_setup_dma_ops(struct device *dev, u64 dma_base, u64 size,
> +				  const struct iommu_ops *ops)
> +{
> +	struct iommu_dma_notifier_data *iommudata;
> +
> +	if (!ops)
> +		return;
> +
> +	iommudata = kzalloc(sizeof(*iommudata), GFP_KERNEL);
> +	if (!iommudata)
> +		return;
> +
> +	iommudata->dev = dev;
> +	iommudata->dma_domain = iommu_dma_create_domain(ops, dma_base, size);
> +	if (!iommudata->dma_domain) {
> +		pr_warn("Failed to create %llu-byte IOMMU mapping for device %s\n",
> +				size, dev_name(dev));
> +		kfree(iommudata);
> +		return;
> +	}
> +	mutex_lock(&iommu_dma_notifier_lock);
> +	list_add(&iommudata->list, &iommu_dma_masters);
> +	mutex_unlock(&iommu_dma_notifier_lock);
> +}
> +

Re: [PATCH 3/4] arm64: Add IOMMU dma_ops

[Robin Murphy]

On 04/06/15 13:29, Yong Wu wrote:
> Hi Robin,
>       Sorry to disturb you. When prepare iommu based on this dma-v3. I
> meet some problem about how to use the dma-iommu.
>       I list some sample code like below:
> //========
> int mtk_iommu_of_xlate(struct device *dev, struct of_phandle_args *args)
> {
>          if (args->args_count != 2) {
>                  return -EINVAL;
>          }
>
>          /* save the private data from args
>           * the private data is larbid and portid in our platform */
>          priv->larbid = args->args[0];
>          priv->portid = args->args[1];
>
>          /*list the private data */
>          list_add_tail(&priv->client, &head->client);
>
>          /* save the list to dev->archdata->iommu */
>          ...
> }
> static int mtk_iommu_init_fn(struct device_node *np)
> {
>          of_iommu_set_ops(np, &mtk_iommu_ops);
>          return 0;
> }
>
> IOMMU_OF_DECLARE(mtkm4u, "mediatek,mt8173-m4u", mtk_iommu_init_fn);
> //=======
>
> Question :
>    1: if there are 2 client devices use iommu, it will be like this in
> dtsi,
>                  disp:testiommu@1400c000 {
>                          compatible = "mediatek,mt8173-test";
>                          iommus = <&iommu M4U_LARB0_ID 0>,
>                                   <&iommu M4U_LARB1_ID 3>;
>                  };
>                  disp1:testiommu@1400D000 {
>                          compatible = "mediatek,mt8173-disptest1";
>                          iommus = <&iommu M4U_LARB0_ID 8>,
>                                   <&iommu M4U_LARB1_ID 2>;
>                  };
>      Then both devices will enter arch_setup_dma_ops and their parameter
> "struct iommu_ops *" is not zero, then it will create two
> dma-iommu-domain.
>      If we expect all the client device share a iommu domain, then how
> should i do?

Until we get IOMMU core default domains sorted out, there's no nice way 
around this. The DMA mapping layer doesn't know enough to be able to 
create per-IOMMU DMA domains, so the best it can do is create one DMA 
domain per device (a single global domain wouldn't work across multiple 
IOMMUs). For now, the only reasonable option I can come up with is your 
driver using yet another bus notifier to run after the dma-iommu one, 
detaching the relevant devices from their automatic DMA domains and 
attaching them to its own private one.

> 2. iommu_dma_attach_device will be called automatically in the notify
> "__iommu_attach_notifier". But it run before the probe of our iommu
> device.
>      Then we can't write the register to enable iommu in *_attach_device
> because we have not parsed the dtsi at that time.
>     ->I have tried to move parse dtsi into mtk_iommu_of_xlate, in order
> to read/write register in attach_device.
> But it will be warning like this:
> (150604_09:43:41.094)WARNING: CPU: 0 PID: 1
> at /proj/mtk40525/upstreamdev/chromebook_kernelonly/kernel/mediatek/drivers/base/dd.c:286 driver_probe_device+0x25c/0x29c()

The current sort-of-acceptable workaround for that is to explicitly call 
of_platform_device_create from mtk_iommu_init_fn - see Marek's 
exynos-iommu implementation[1] or my work-in-progress iommu/smmu branch 
for examples. Please take a look at Laurent's probe-deferral series[2] 
if you'd like to help with the "proper" solution.

> 3.In mtk_iommu_probe, we can't get "struct iommu_domain *" from the
> current iommu device, we should use it while devm_request_irq.
> I have to use the global variable here?!

I'm not sure I follow - I was imagining you'd allocate your private DMA 
domain during IOMMU device creation, then stash it in your 
mtk_iommu_info and retrieve it as needed. If it's a "global" interrupt 
it seems more logical to pass that iommu data to the handler anyway, but 
if it's purely a "domain fault" interrupt couldn't you register it when 
creating the domain, rather than at device probe? That seems to be how 
we do it in the ARM SMMU driver, for example.

Robin.

[1]:http://article.gmane.org/gmane.linux.kernel.samsung-soc/45423
[2]:http://thread.gmane.org/gmane.linux.kernel.iommu/9552

> 4.If I don't implement the of_xlate and IOMMU_OF_DECLARE, I call
> iommu_dma_create_domain in the mtk_iommu_probe, but I can not
> get the "iommu_dma_ops" which is static in arch/arm64/mm/dma-mapping.c
> to assign to dev->archdata->dma_ops for the iommu client device.
>
> If i miss something, please tell me and help give some suggestion.
> Thanks very much.
>
> On Wed, 2015-05-27 at 15:09 +0100, Robin Murphy wrote:
>> Taking some inspiration from the arch/arm code, implement the
>> arch-specific side of the DMA mapping ops using the new IOMMU-DMA layer.
>>
>> Whilst proliferating per-device private IOMMU data via dev->archdata is
>> less than ideal, it will do the job for now, especially since we can't
>> easily handle the kind of problematic system topologies in the current
>> IOMMU API anyway.
>>
>> Signed-off-by: Robin Murphy <robin.murphy-5wv7dgnIgG8@public.gmane.org>
>> ---
>>   arch/arm64/include/asm/device.h      |   3 +
>>   arch/arm64/include/asm/dma-mapping.h |  14 ++
>>   arch/arm64/mm/dma-mapping.c          | 342 +++++++++++++++++++++++++++++++++++
>>   include/linux/dma-iommu.h            |   4 +-
>>   4 files changed, 361 insertions(+), 2 deletions(-)
>>
> [snip]
>> +struct iommu_dma_notifier_data {
>> +	struct list_head list;
>> +	struct device *dev;
>> +	struct iommu_dma_domain *dma_domain;
>> +};
>> +static LIST_HEAD(iommu_dma_masters);
>> +static DEFINE_MUTEX(iommu_dma_notifier_lock);
>> +
>> +static int __iommu_attach_notifier(struct notifier_block *nb,
>> +				   unsigned long action, void *data)
>> +{
>> +	struct iommu_dma_notifier_data *master, *tmp;
>> +
>> +	if (action != BUS_NOTIFY_ADD_DEVICE)
>> +		return 0;
>> +	/*
>> +	 * We expect the list to only contain the most recent addition,
>> +	 * which *should* be the same device as @data, so just process the
>> +	 * whole thing blindly. If any previous attachments did happen to
>> +	 * fail, they get a free retry since the domains are still live.
>> +	 */
>> +	mutex_lock(&iommu_dma_notifier_lock);
>> +	list_for_each_entry_safe(master, tmp, &iommu_dma_masters, list) {
>> +		if (iommu_dma_attach_device(master->dev, master->dma_domain)) {
>> +			pr_warn("Failed to attach device %s to IOMMU mapping; retaining platform DMA ops\n",
>> +				dev_name(master->dev));
>> +		} else {
>> +			master->dev->archdata.dma_ops = &iommu_dma_ops;
>> +			/* it's safe to drop the initial refcount now */
>> +			iommu_dma_release_domain(master->dma_domain);
>> +			list_del(&master->list);
>> +			kfree(master);
>> +		}
>> +	}
>> +	mutex_unlock(&iommu_dma_notifier_lock);
>> +	return 0;
>> +}
>> +
>> +static int register_iommu_dma_ops_notifier(struct bus_type *bus)
>> +{
>> +	int ret;
>> +	struct notifier_block *nb = kzalloc(sizeof(*nb), GFP_KERNEL);
>> +
>> +	/*
>> +	 * The device must be attached to a domain before its driver probe,
>> +	 * in case the driver allocates DMA buffers immediately. However, most
>> +	 * IOMMU drivers are currently configuring groups in their add_device
>> +	 * callback, so the attach should happen after that. Since the IOMMU
>> +	 * core uses a bus notifier for add_device, do the same but with a
>> +	 * stupidly low priority to ensure the appropriate ordering.
>> +	 *
>> +	 * This can hopefully all go away once we have default domains in the
>> +	 * IOMMU core.
>> +	 */
>> +	nb->notifier_call = __iommu_attach_notifier;
>> +	nb->priority = INT_MIN;
>> +
>> +	ret = bus_register_notifier(bus, nb);
>> +	if (ret) {
>> +		pr_warn("Failed to register DMA domain notifier; IOMMU DMA ops unavailable on bus '%s'\n",
>> +			bus->name);
>> +		kfree(nb);
>> +	}
>> +	return ret;
>> +}
>> +
>> +static int __init arm64_iommu_dma_init(void)
>> +{
>> +	int ret;
>> +
>> +	ret = iommu_dma_init();
>> +	if (!ret)
>> +		ret = register_iommu_dma_ops_notifier(&platform_bus_type);
>> +	if (!ret)
>> +		ret = register_iommu_dma_ops_notifier(&amba_bustype);
>> +	return ret;
>> +}
>> +arch_initcall(arm64_iommu_dma_init);
>> +
>> +static void __iommu_setup_dma_ops(struct device *dev, u64 dma_base, u64 size,
>> +				  const struct iommu_ops *ops)
>> +{
>> +	struct iommu_dma_notifier_data *iommudata;
>> +
>> +	if (!ops)
>> +		return;
>> +
>> +	iommudata = kzalloc(sizeof(*iommudata), GFP_KERNEL);
>> +	if (!iommudata)
>> +		return;
>> +
>> +	iommudata->dev = dev;
>> +	iommudata->dma_domain = iommu_dma_create_domain(ops, dma_base, size);
>> +	if (!iommudata->dma_domain) {
>> +		pr_warn("Failed to create %llu-byte IOMMU mapping for device %s\n",
>> +				size, dev_name(dev));
>> +		kfree(iommudata);
>> +		return;
>> +	}
>> +	mutex_lock(&iommu_dma_notifier_lock);
>> +	list_add(&iommudata->list, &iommu_dma_masters);
>> +	mutex_unlock(&iommu_dma_notifier_lock);
>> +}
>> +
>
>

[PATCH 4/4] arm64: Hook up IOMMU dma_ops

[Robin Murphy]

With iommu_dma_ops in place, hook them up to the configuration code, so
IOMMU-fronted devices will get them automatically.

Signed-off-by: Robin Murphy <robin.murphy-5wv7dgnIgG8@public.gmane.org>
---
 arch/arm64/Kconfig                   |  1 +
 arch/arm64/include/asm/dma-mapping.h | 11 ++++++-----
 arch/arm64/mm/dma-mapping.c          | 15 +++++++++++++++
 3 files changed, 22 insertions(+), 5 deletions(-)

diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig
index 7796af4..257cd7e 100644
--- a/arch/arm64/Kconfig
+++ b/arch/arm64/Kconfig
@@ -70,6 +70,7 @@ config ARM64
 	select HAVE_PERF_USER_STACK_DUMP
 	select HAVE_RCU_TABLE_FREE
 	select HAVE_SYSCALL_TRACEPOINTS
+	select IOMMU_DMA if IOMMU_SUPPORT
 	select IRQ_DOMAIN
 	select MODULES_USE_ELF_RELA
 	select NO_BOOTMEM
diff --git a/arch/arm64/include/asm/dma-mapping.h b/arch/arm64/include/asm/dma-mapping.h
index 835a8d1..a87b507 100644
--- a/arch/arm64/include/asm/dma-mapping.h
+++ b/arch/arm64/include/asm/dma-mapping.h
@@ -45,11 +45,8 @@ static inline struct dma_map_ops *get_dma_ops(struct device *dev)
 		return __generic_dma_ops(dev);
 }
 
-static inline void arch_setup_dma_ops(struct device *dev, u64 dma_base, u64 size,
-				      struct iommu_ops *iommu, bool coherent)
-{
-	dev->archdata.dma_coherent = coherent;
-}
+void arch_setup_dma_ops(struct device *dev, u64 dma_base, u64 size,
+			struct iommu_ops *iommu, bool coherent);
 #define arch_setup_dma_ops	arch_setup_dma_ops
 
 /* do not use this function in a driver */
@@ -64,6 +61,10 @@ static inline bool is_device_dma_coherent(struct device *dev)
 #include <linux/dma-iommu.h>
 
 #ifdef CONFIG_IOMMU_DMA
+
+void arch_teardown_dma_ops(struct device *dev);
+#define arch_teardown_dma_ops	arch_teardown_dma_ops
+
 static inline struct iommu_dma_domain *arch_get_dma_domain(struct device *dev)
 {
 	return dev->archdata.dma_domain;
diff --git a/arch/arm64/mm/dma-mapping.c b/arch/arm64/mm/dma-mapping.c
index 189477b..f9d4184 100644
--- a/arch/arm64/mm/dma-mapping.c
+++ b/arch/arm64/mm/dma-mapping.c
@@ -769,6 +769,14 @@ static void __iommu_setup_dma_ops(struct device *dev, u64 dma_base, u64 size,
 	mutex_unlock(&iommu_dma_notifier_lock);
 }
 
+void arch_teardown_dma_ops(struct device *dev)
+{
+	if (dev->archdata.dma_domain) {
+		iommu_dma_detach_device(dev);
+		dev->archdata.dma_ops = NULL;
+	}
+}
+
 #else
 
 static void __iommu_setup_dma_ops(struct device *dev, u64 dma_base, u64 size,
@@ -776,3 +784,10 @@ static void __iommu_setup_dma_ops(struct device *dev, u64 dma_base, u64 size,
 { }
 
 #endif  /* CONFIG_IOMMU_DMA */
+
+void arch_setup_dma_ops(struct device *dev, u64 dma_base, u64 size,
+			struct iommu_ops *iommu, bool coherent)
+{
+	dev->archdata.dma_coherent = coherent;
+	__iommu_setup_dma_ops(dev, dma_base, size, iommu);
+}
-- 
1.9.1

Re: [PATCH 0/4] arm64: IOMMU-backed DMA mapping

[Joerg Roedel]

Hi Robin,

Thanks for your work. We are really moving forward to a common DMA-API
implementation for all iommu drivers.

On Wed, May 27, 2015 at 03:09:14PM +0100, Robin Murphy wrote:
> In the meantime, Laurent's proposal for probe deferral[2] offers hope
> that the bus notifier dance may only need to be short-lived, and I'm
> hoping to spend some more time on DT-based IOMMU group handling for
> platform devices, which should hopefully tie in with default domains
> to make the add_device callback mostly redundant.

Yes, this would be great. I think it makes a lot of sense to make use of
the default domain feature. Can you have a look at how well this
integrates with my current patches for default domains and provide some
feedback?

Thanks,

	Joerg