* [PATCH 0/4] arm:dma-mapping Refactoring iommu dma-mapping code @ 2014-06-02 10:19 ritesh.harjani at gmail.com 2014-06-02 10:19 ` [PATCH 1/4] arm: dma-iommu: Move out dma_iommu_mapping struct ritesh.harjani at gmail.com 2014-06-03 13:01 ` [PATCH 0/4] arm:dma-mapping Refactoring iommu dma-mapping code Will Deacon 0 siblings, 2 replies; 6+ messages in thread From: ritesh.harjani at gmail.com @ 2014-06-02 10:19 UTC (permalink / raw) To: linux-arm-kernel From: Ritesh Harjani <ritesh.harjani@gmail.com> Hi All, This patch series is to refractor iommu related common code from arch/arm/dma-mapping.c to lib/iommu-helper.c based on the various discussions with the maintainers/experts [1]. Currently the only user of the common lib/iommu-helper code will be ARM & ARM64 but later various architecture might try to use this iommu lib helper functions. Major change of this refactoring starts with bringing out struct dma_iommu_mapping *mapping variable from arch/arm/include/asm/device.h to include/linux/device.h and by moving out complete structure defination of dma_iommu_mapping to inclue/linux/iommu-helper.h. Link [2] give more details on why this was done, also this change got approval from Will Daecon [2]. There are 1/2 more function definitions which I can think of moving out, but those can be done once this patch series is approved as those are not very big changes. [1]: https://www.mail-archive.com/iommu at lists.linux-foundation.org/msg03458.html [2]: https://www.mail-archive.com/iommu at lists.linux-foundation.org/msg04272.html Ritesh Harjani (4): arm: dma-iommu: Move out dma_iommu_mapping struct arm: dma-mapping: Refractor attach/detach dev function calls arm: dma-mapping: Refractor iommu_alloc/free funcs arm:dma-iommu: Move out complete func defs arch/arm/Kconfig | 46 +-- arch/arm/include/asm/device.h | 9 - arch/arm/include/asm/dma-iommu.h | 16 +- arch/arm/mm/dma-mapping.c | 566 +++--------------------------- drivers/gpu/drm/exynos/exynos_drm_iommu.c | 10 +- include/linux/device.h | 4 + include/linux/iommu-helper.h | 64 ++++ lib/iommu-helper.c | 561 +++++++++++++++++++++++++++++ 8 files changed, 705 insertions(+), 571 deletions(-) -- 1.8.1.3 ^ permalink raw reply [flat|nested] 6+ messages in thread
* [PATCH 1/4] arm: dma-iommu: Move out dma_iommu_mapping struct 2014-06-02 10:19 [PATCH 0/4] arm:dma-mapping Refactoring iommu dma-mapping code ritesh.harjani at gmail.com @ 2014-06-02 10:19 ` ritesh.harjani at gmail.com 2014-06-02 10:19 ` [PATCH 2/4] arm: dma-mapping: Refractor attach/detach dev function calls ritesh.harjani at gmail.com 2014-06-03 13:01 ` [PATCH 0/4] arm:dma-mapping Refactoring iommu dma-mapping code Will Deacon 1 sibling, 1 reply; 6+ messages in thread From: ritesh.harjani at gmail.com @ 2014-06-02 10:19 UTC (permalink / raw) To: linux-arm-kernel From: Ritesh Harjani <ritesh.harjani@gmail.com> This patch is the 1st in the refractoring patch series of arm, dma-iommu-mapping code, which brings dma_iommu_mapping structure out to include/linux/iommu-helper.h 1. take out dma_iommu_mapping variable from arch/arm/asm/include/device.h to include/linux/device.h 2. take out complete structure defination of dma_iommu_mapping to include/linux/iommu-helper.h 3. Protect this variable with a config and let the archs define this config based on their use. Change-Id: Ic9d5e6e2346258698348153de39ded20d730fe72 Signed-off-by: Ritesh Harjani <ritesh.harjani@gmail.com> --- arch/arm/Kconfig | 4 ++++ arch/arm/include/asm/device.h | 9 --------- arch/arm/include/asm/dma-iommu.h | 16 +--------------- arch/arm/mm/dma-mapping.c | 20 ++++++++++---------- drivers/gpu/drm/exynos/exynos_drm_iommu.c | 10 +++++----- include/linux/device.h | 4 ++++ include/linux/iommu-helper.h | 22 ++++++++++++++++++++++ 7 files changed, 46 insertions(+), 39 deletions(-) diff --git a/arch/arm/Kconfig b/arch/arm/Kconfig index c0b31fc..20717fb 100644 --- a/arch/arm/Kconfig +++ b/arch/arm/Kconfig @@ -78,6 +78,7 @@ config ARM_DMA_USE_IOMMU bool select ARM_HAS_SG_CHAIN select NEED_SG_DMA_LENGTH + select DMA_USE_IOMMU_HELPER_MAPPING if ARM_DMA_USE_IOMMU @@ -1945,6 +1946,9 @@ config SWIOTLB config IOMMU_HELPER def_bool SWIOTLB +config DMA_USE_IOMMU_HELPER_MAPPING + def_bool n + config XEN_DOM0 def_bool y depends on XEN diff --git a/arch/arm/include/asm/device.h b/arch/arm/include/asm/device.h index dc662fc..6e2cb0e 100644 --- a/arch/arm/include/asm/device.h +++ b/arch/arm/include/asm/device.h @@ -14,9 +14,6 @@ struct dev_archdata { #ifdef CONFIG_IOMMU_API void *iommu; /* private IOMMU data */ #endif -#ifdef CONFIG_ARM_DMA_USE_IOMMU - struct dma_iommu_mapping *mapping; -#endif }; struct omap_device; @@ -27,10 +24,4 @@ struct pdev_archdata { #endif }; -#ifdef CONFIG_ARM_DMA_USE_IOMMU -#define to_dma_iommu_mapping(dev) ((dev)->archdata.mapping) -#else -#define to_dma_iommu_mapping(dev) NULL -#endif - #endif diff --git a/arch/arm/include/asm/dma-iommu.h b/arch/arm/include/asm/dma-iommu.h index 8e3fcb9..50c010b 100644 --- a/arch/arm/include/asm/dma-iommu.h +++ b/arch/arm/include/asm/dma-iommu.h @@ -8,21 +8,7 @@ #include <linux/dma-debug.h> #include <linux/kmemcheck.h> #include <linux/kref.h> - -struct dma_iommu_mapping { - /* iommu specific data */ - struct iommu_domain *domain; - - unsigned long **bitmaps; /* array of bitmaps */ - unsigned int nr_bitmaps; /* nr of elements in array */ - unsigned int extensions; - size_t bitmap_size; /* size of a single bitmap */ - size_t bits; /* per bitmap */ - dma_addr_t base; - - spinlock_t lock; - struct kref kref; -}; +#include <linux/iommu-helper.h> struct dma_iommu_mapping * arm_iommu_create_mapping(struct bus_type *bus, dma_addr_t base, size_t size); diff --git a/arch/arm/mm/dma-mapping.c b/arch/arm/mm/dma-mapping.c index 3d43c41..b82561e 100644 --- a/arch/arm/mm/dma-mapping.c +++ b/arch/arm/mm/dma-mapping.c @@ -1292,7 +1292,7 @@ err: static dma_addr_t __iommu_create_mapping(struct device *dev, struct page **pages, size_t size) { - struct dma_iommu_mapping *mapping = dev->archdata.mapping; + struct dma_iommu_mapping *mapping = dev->mapping; unsigned int count = PAGE_ALIGN(size) >> PAGE_SHIFT; dma_addr_t dma_addr, iova; int i, ret = DMA_ERROR_CODE; @@ -1328,7 +1328,7 @@ fail: static int __iommu_remove_mapping(struct device *dev, dma_addr_t iova, size_t size) { - struct dma_iommu_mapping *mapping = dev->archdata.mapping; + struct dma_iommu_mapping *mapping = dev->mapping; /* * add optional in-page offset from iova to size and align @@ -1541,7 +1541,7 @@ static int __map_sg_chunk(struct device *dev, struct scatterlist *sg, enum dma_data_direction dir, struct dma_attrs *attrs, bool is_coherent) { - struct dma_iommu_mapping *mapping = dev->archdata.mapping; + struct dma_iommu_mapping *mapping = dev->mapping; dma_addr_t iova, iova_base; int ret = 0; unsigned int count; @@ -1762,7 +1762,7 @@ static dma_addr_t arm_coherent_iommu_map_page(struct device *dev, struct page *p unsigned long offset, size_t size, enum dma_data_direction dir, struct dma_attrs *attrs) { - struct dma_iommu_mapping *mapping = dev->archdata.mapping; + struct dma_iommu_mapping *mapping = dev->mapping; dma_addr_t dma_addr; int ret, prot, len = PAGE_ALIGN(size + offset); @@ -1815,7 +1815,7 @@ static void arm_coherent_iommu_unmap_page(struct device *dev, dma_addr_t handle, size_t size, enum dma_data_direction dir, struct dma_attrs *attrs) { - struct dma_iommu_mapping *mapping = dev->archdata.mapping; + struct dma_iommu_mapping *mapping = dev->mapping; dma_addr_t iova = handle & PAGE_MASK; int offset = handle & ~PAGE_MASK; int len = PAGE_ALIGN(size + offset); @@ -1840,7 +1840,7 @@ static void arm_iommu_unmap_page(struct device *dev, dma_addr_t handle, size_t size, enum dma_data_direction dir, struct dma_attrs *attrs) { - struct dma_iommu_mapping *mapping = dev->archdata.mapping; + struct dma_iommu_mapping *mapping = dev->mapping; dma_addr_t iova = handle & PAGE_MASK; struct page *page = phys_to_page(iommu_iova_to_phys(mapping->domain, iova)); int offset = handle & ~PAGE_MASK; @@ -1859,7 +1859,7 @@ static void arm_iommu_unmap_page(struct device *dev, dma_addr_t handle, static void arm_iommu_sync_single_for_cpu(struct device *dev, dma_addr_t handle, size_t size, enum dma_data_direction dir) { - struct dma_iommu_mapping *mapping = dev->archdata.mapping; + struct dma_iommu_mapping *mapping = dev->mapping; dma_addr_t iova = handle & PAGE_MASK; struct page *page = phys_to_page(iommu_iova_to_phys(mapping->domain, iova)); unsigned int offset = handle & ~PAGE_MASK; @@ -1873,7 +1873,7 @@ static void arm_iommu_sync_single_for_cpu(struct device *dev, static void arm_iommu_sync_single_for_device(struct device *dev, dma_addr_t handle, size_t size, enum dma_data_direction dir) { - struct dma_iommu_mapping *mapping = dev->archdata.mapping; + struct dma_iommu_mapping *mapping = dev->mapping; dma_addr_t iova = handle & PAGE_MASK; struct page *page = phys_to_page(iommu_iova_to_phys(mapping->domain, iova)); unsigned int offset = handle & ~PAGE_MASK; @@ -2045,7 +2045,7 @@ int arm_iommu_attach_device(struct device *dev, return err; kref_get(&mapping->kref); - dev->archdata.mapping = mapping; + dev->mapping = mapping; set_dma_ops(dev, &iommu_ops); pr_debug("Attached IOMMU controller to %s device.\n", dev_name(dev)); @@ -2072,7 +2072,7 @@ void arm_iommu_detach_device(struct device *dev) iommu_detach_device(mapping->domain, dev); kref_put(&mapping->kref, release_iommu_mapping); - dev->archdata.mapping = NULL; + dev->mapping = NULL; set_dma_ops(dev, NULL); pr_debug("Detached IOMMU controller from %s device.\n", dev_name(dev)); diff --git a/drivers/gpu/drm/exynos/exynos_drm_iommu.c b/drivers/gpu/drm/exynos/exynos_drm_iommu.c index 091068f..2dabdbf 100644 --- a/drivers/gpu/drm/exynos/exynos_drm_iommu.c +++ b/drivers/gpu/drm/exynos/exynos_drm_iommu.c @@ -46,7 +46,7 @@ int drm_create_iommu_mapping(struct drm_device *drm_dev) dev->dma_parms = devm_kzalloc(dev, sizeof(*dev->dma_parms), GFP_KERNEL); dma_set_max_seg_size(dev, 0xffffffffu); - dev->archdata.mapping = mapping; + dev->mapping = mapping; return 0; } @@ -63,7 +63,7 @@ void drm_release_iommu_mapping(struct drm_device *drm_dev) { struct device *dev = drm_dev->dev; - arm_iommu_release_mapping(dev->archdata.mapping); + arm_iommu_release_mapping(dev->mapping); } /* @@ -81,7 +81,7 @@ int drm_iommu_attach_device(struct drm_device *drm_dev, struct device *dev = drm_dev->dev; int ret; - if (!dev->archdata.mapping) { + if (!dev->mapping) { DRM_ERROR("iommu_mapping is null.\n"); return -EFAULT; } @@ -91,7 +91,7 @@ int drm_iommu_attach_device(struct drm_device *drm_dev, GFP_KERNEL); dma_set_max_seg_size(subdrv_dev, 0xffffffffu); - ret = arm_iommu_attach_device(subdrv_dev, dev->archdata.mapping); + ret = arm_iommu_attach_device(subdrv_dev, dev->mapping); if (ret < 0) { DRM_DEBUG_KMS("failed iommu attach.\n"); return ret; @@ -124,7 +124,7 @@ void drm_iommu_detach_device(struct drm_device *drm_dev, struct device *subdrv_dev) { struct device *dev = drm_dev->dev; - struct dma_iommu_mapping *mapping = dev->archdata.mapping; + struct dma_iommu_mapping *mapping = dev->mapping; if (!mapping || !mapping->domain) return; diff --git a/include/linux/device.h b/include/linux/device.h index c0a1261..c73df6f 100644 --- a/include/linux/device.h +++ b/include/linux/device.h @@ -705,6 +705,10 @@ struct device { /* arch specific additions */ struct dev_archdata archdata; +#ifdef CONFIG_DMA_USE_IOMMU_HELPER_MAPPING + struct dma_iommu_mapping *mapping; +#endif + struct device_node *of_node; /* associated device tree node */ struct acpi_dev_node acpi_node; /* associated ACPI device node */ diff --git a/include/linux/iommu-helper.h b/include/linux/iommu-helper.h index 86bdeff..0c5e4c7 100644 --- a/include/linux/iommu-helper.h +++ b/include/linux/iommu-helper.h @@ -3,6 +3,28 @@ #include <linux/kernel.h> +#ifdef CONFIG_DMA_USE_IOMMU_HELPER_MAPPING +struct dma_iommu_mapping { + /* iommu specific data */ + struct iommu_domain *domain; + + unsigned long **bitmaps; /* array of bitmaps */ + unsigned int nr_bitmaps; /* nr of elements in array */ + unsigned int extensions; + size_t bitmap_size; /* size of a single bitmap */ + size_t bits; /* per bitmap */ + dma_addr_t base; + + spinlock_t lock; + struct kref kref; +}; + +#define to_dma_iommu_mapping(dev) ((dev)->mapping) +#else +#define to_dma_iommu_mapping(dev) NULL +#endif + + static inline unsigned long iommu_device_max_index(unsigned long size, unsigned long offset, u64 dma_mask) -- 1.8.1.3 ^ permalink raw reply related [flat|nested] 6+ messages in thread
* [PATCH 2/4] arm: dma-mapping: Refractor attach/detach dev function calls 2014-06-02 10:19 ` [PATCH 1/4] arm: dma-iommu: Move out dma_iommu_mapping struct ritesh.harjani at gmail.com @ 2014-06-02 10:19 ` ritesh.harjani at gmail.com 2014-06-02 10:19 ` [PATCH 3/4] arm: dma-mapping: Refractor iommu_alloc/free funcs ritesh.harjani at gmail.com 0 siblings, 1 reply; 6+ messages in thread From: ritesh.harjani at gmail.com @ 2014-06-02 10:19 UTC (permalink / raw) To: linux-arm-kernel From: Ritesh Harjani <ritesh.harjani@gmail.com> Refractor following function calls to lib/iommu-helper.c arm_iommu_attach/detach device function calls. arm_iommu_init/release_mapping function calls. Change-Id: Ic69a8b6b7008599a6e98b670b11a61ff6a5bac99 Signed-off-by: Ritesh Harjani <ritesh.harjani@gmail.com> --- arch/arm/mm/dma-mapping.c | 101 +++++-------------------------- include/linux/iommu-helper.h | 10 ++++ lib/iommu-helper.c | 140 +++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 165 insertions(+), 86 deletions(-) diff --git a/arch/arm/mm/dma-mapping.c b/arch/arm/mm/dma-mapping.c index b82561e..38fc146 100644 --- a/arch/arm/mm/dma-mapping.c +++ b/arch/arm/mm/dma-mapping.c @@ -1924,7 +1924,8 @@ struct dma_map_ops iommu_coherent_ops = { * @base: start address of the valid IO address space * @size: maximum size of the valid IO address space * - * Creates a mapping structure which holds information about used/unused + * Calls for lib/iommu-helper function which creates a mapping + * structure which holds information about used/unused * IO address ranges, which is required to perform memory allocation and * mapping with IOMMU aware functions. * @@ -1934,71 +1935,10 @@ struct dma_map_ops iommu_coherent_ops = { struct dma_iommu_mapping * arm_iommu_create_mapping(struct bus_type *bus, dma_addr_t base, size_t size) { - unsigned int bits = size >> PAGE_SHIFT; - unsigned int bitmap_size = BITS_TO_LONGS(bits) * sizeof(long); - struct dma_iommu_mapping *mapping; - int extensions = 1; - int err = -ENOMEM; - - if (!bitmap_size) - return ERR_PTR(-EINVAL); - - if (bitmap_size > PAGE_SIZE) { - extensions = bitmap_size / PAGE_SIZE; - bitmap_size = PAGE_SIZE; - } - - mapping = kzalloc(sizeof(struct dma_iommu_mapping), GFP_KERNEL); - if (!mapping) - goto err; - - mapping->bitmap_size = bitmap_size; - mapping->bitmaps = kzalloc(extensions * sizeof(unsigned long *), - GFP_KERNEL); - if (!mapping->bitmaps) - goto err2; - - mapping->bitmaps[0] = kzalloc(bitmap_size, GFP_KERNEL); - if (!mapping->bitmaps[0]) - goto err3; - - mapping->nr_bitmaps = 1; - mapping->extensions = extensions; - mapping->base = base; - mapping->bits = BITS_PER_BYTE * bitmap_size; - - spin_lock_init(&mapping->lock); - - mapping->domain = iommu_domain_alloc(bus); - if (!mapping->domain) - goto err4; - - kref_init(&mapping->kref); - return mapping; -err4: - kfree(mapping->bitmaps[0]); -err3: - kfree(mapping->bitmaps); -err2: - kfree(mapping); -err: - return ERR_PTR(err); + return __iommu_init_mapping(bus, base, size); } EXPORT_SYMBOL_GPL(arm_iommu_create_mapping); -static void release_iommu_mapping(struct kref *kref) -{ - int i; - struct dma_iommu_mapping *mapping = - container_of(kref, struct dma_iommu_mapping, kref); - - iommu_domain_free(mapping->domain); - for (i = 0; i < mapping->nr_bitmaps; i++) - kfree(mapping->bitmaps[i]); - kfree(mapping->bitmaps); - kfree(mapping); -} - static int extend_iommu_mapping(struct dma_iommu_mapping *mapping) { int next_bitmap; @@ -2019,8 +1959,7 @@ static int extend_iommu_mapping(struct dma_iommu_mapping *mapping) void arm_iommu_release_mapping(struct dma_iommu_mapping *mapping) { - if (mapping) - kref_put(&mapping->kref, release_iommu_mapping); + __iommu_release_mapping(mapping); } EXPORT_SYMBOL_GPL(arm_iommu_release_mapping); @@ -2030,8 +1969,9 @@ EXPORT_SYMBOL_GPL(arm_iommu_release_mapping); * @mapping: io address space mapping structure (returned from * arm_iommu_create_mapping) * - * Attaches specified io address space mapping to the provided device, - * this replaces the dma operations (dma_map_ops pointer) with the + * Calls for lib/iommu-helper which attaches specified io + * address space mapping to the provided device, this + * replaces the dma operations (dma_map_ops pointer) with the * IOMMU aware version. More than one client might be attached to * the same io address space mapping. */ @@ -2040,13 +1980,12 @@ int arm_iommu_attach_device(struct device *dev, { int err; - err = iommu_attach_device(mapping->domain, dev); - if (err) - return err; + err = __iommu_attach_device(dev, mapping); - kref_get(&mapping->kref); - dev->mapping = mapping; - set_dma_ops(dev, &iommu_ops); + if (!err) + set_dma_ops(dev, &iommu_ops); + else + return err; pr_debug("Attached IOMMU controller to %s device.\n", dev_name(dev)); return 0; @@ -2057,24 +1996,14 @@ EXPORT_SYMBOL_GPL(arm_iommu_attach_device); * arm_iommu_detach_device * @dev: valid struct device pointer * - * Detaches the provided device from a previously attached map. + * Calls for lib/iommu-helper which detaches the provided + * device from a previously attached map. * This voids the dma operations (dma_map_ops pointer) */ void arm_iommu_detach_device(struct device *dev) { - struct dma_iommu_mapping *mapping; - - mapping = to_dma_iommu_mapping(dev); - if (!mapping) { - dev_warn(dev, "Not attached\n"); - return; - } - - iommu_detach_device(mapping->domain, dev); - kref_put(&mapping->kref, release_iommu_mapping); - dev->mapping = NULL; + __iommu_detach_device(dev); set_dma_ops(dev, NULL); - pr_debug("Detached IOMMU controller from %s device.\n", dev_name(dev)); } EXPORT_SYMBOL_GPL(arm_iommu_detach_device); diff --git a/include/linux/iommu-helper.h b/include/linux/iommu-helper.h index 0c5e4c7..c6a315d 100644 --- a/include/linux/iommu-helper.h +++ b/include/linux/iommu-helper.h @@ -19,6 +19,16 @@ struct dma_iommu_mapping { struct kref kref; }; +extern void __iommu_detach_device(struct device *dev); + +extern void __iommu_release_mapping(struct dma_iommu_mapping *mapping); + +extern int __iommu_attach_device(struct device *dev, + struct dma_iommu_mapping *mapping); + +extern struct dma_iommu_mapping * +__iommu_init_mapping(struct bus_type *bus, dma_addr_t base, size_t size); + #define to_dma_iommu_mapping(dev) ((dev)->mapping) #else #define to_dma_iommu_mapping(dev) NULL diff --git a/lib/iommu-helper.c b/lib/iommu-helper.c index c27e269..28daaa5 100644 --- a/lib/iommu-helper.c +++ b/lib/iommu-helper.c @@ -6,6 +6,15 @@ #include <linux/bitmap.h> #include <linux/bug.h> +#ifdef CONFIG_DMA_USE_IOMMU_HELPER_MAPPING +#include <linux/iommu.h> +#include <linux/device.h> +#include <linux/iommu-helper.h> +#include <linux/slab.h> +#include <linux/vmalloc.h> +#include <linux/errno.h> +#endif + int iommu_is_span_boundary(unsigned int index, unsigned int nr, unsigned long shift, unsigned long boundary_size) @@ -39,3 +48,134 @@ again: return -1; } EXPORT_SYMBOL(iommu_area_alloc); + +#ifdef CONFIG_DMA_USE_IOMMU_HELPER_MAPPING + +/** + * __iommu_init_mapping + * @bus: pointer to the bus holding the client device (for IOMMU calls) + * @base: start address of the valid IO address space + * @size: maximum size of the valid IO address space + * + * Creates a mapping structure which holds information about used/unused + * IO address ranges, which is required to perform memory allocation and + * mapping with IOMMU aware functions. + * + */ + +struct dma_iommu_mapping * +__iommu_init_mapping(struct bus_type *bus, dma_addr_t base, size_t size) +{ + unsigned int bits = size >> PAGE_SHIFT; + unsigned int bitmap_size = BITS_TO_LONGS(bits) * sizeof(long); + struct dma_iommu_mapping *mapping; + int extensions = 1; + int err = -ENOMEM; + + if (!bitmap_size) + return ERR_PTR(-EINVAL); + + if (bitmap_size > PAGE_SIZE) { + extensions = bitmap_size / PAGE_SIZE; + bitmap_size = PAGE_SIZE; + } + + mapping = kzalloc(sizeof(struct dma_iommu_mapping), GFP_KERNEL); + if (!mapping) + goto err; + + mapping->bitmap_size = bitmap_size; + mapping->bitmaps = kzalloc(extensions * sizeof(unsigned long *), + GFP_KERNEL); + if (!mapping->bitmaps) + goto err2; + + mapping->bitmaps[0] = kzalloc(bitmap_size, GFP_KERNEL); + if (!mapping->bitmaps[0]) + goto err3; + + mapping->nr_bitmaps = 1; + mapping->extensions = extensions; + mapping->base = base; + mapping->bits = BITS_PER_BYTE * bitmap_size; + + spin_lock_init(&mapping->lock); + + mapping->domain = iommu_domain_alloc(bus); + if (!mapping->domain) + goto err4; + + kref_init(&mapping->kref); + return mapping; +err4: + kfree(mapping->bitmaps[0]); +err3: + kfree(mapping->bitmaps); +err2: + kfree(mapping); +err: + return ERR_PTR(err); +} + +static void release_iommu_mapping(struct kref *kref) +{ + int i; + struct dma_iommu_mapping *mapping = + container_of(kref, struct dma_iommu_mapping, kref); + + iommu_domain_free(mapping->domain); + for (i = 0; i < mapping->nr_bitmaps; i++) + kfree(mapping->bitmaps[i]); + kfree(mapping->bitmaps); + kfree(mapping); +} + + +void __iommu_release_mapping(struct dma_iommu_mapping *mapping) +{ + if (mapping) + kref_put(&mapping->kref, release_iommu_mapping); +} + +/** + * __iommu_detach_device + * @dev: valid struct device pointer + * + * Detaches the provided device from a previously attached map. + */ +void __iommu_detach_device(struct device *dev) +{ + struct dma_iommu_mapping *mapping; + + mapping = to_dma_iommu_mapping(dev); + if (!mapping) { + dev_warn(dev, "Not attached\n"); + return; + } + + iommu_detach_device(mapping->domain, dev); + kref_put(&mapping->kref, release_iommu_mapping); + dev->mapping = NULL; +} + +/** + * __iommu_attach_device + * @dev: valid struct device pointer + * @mapping: io address space mapping structure + * + * Attaches specified io address space mapping to the provided device. + */ +int __iommu_attach_device(struct device *dev, + struct dma_iommu_mapping *mapping) +{ + int err; + + err = iommu_attach_device(mapping->domain, dev); + if (err) + return err; + + kref_get(&mapping->kref); + dev->mapping = mapping; + return 0; +} +#endif -- 1.8.1.3 ^ permalink raw reply related [flat|nested] 6+ messages in thread
* [PATCH 3/4] arm: dma-mapping: Refractor iommu_alloc/free funcs 2014-06-02 10:19 ` [PATCH 2/4] arm: dma-mapping: Refractor attach/detach dev function calls ritesh.harjani at gmail.com @ 2014-06-02 10:19 ` ritesh.harjani at gmail.com 2014-06-02 10:19 ` [PATCH 4/4] arm:dma-iommu: Move out complete func defs ritesh.harjani at gmail.com 0 siblings, 1 reply; 6+ messages in thread From: ritesh.harjani at gmail.com @ 2014-06-02 10:19 UTC (permalink / raw) To: linux-arm-kernel From: Ritesh Harjani <ritesh.harjani@gmail.com> iommu_alloc/free_buffer can be moved out to lib/iommu_helper.c as a part of refactoring arm iommu dma-mapping code. Change-Id: I5fba02f64cb4913f6d0200189267826df47df8d6 Signed-off-by: Ritesh Harjani <ritesh.harjani@gmail.com> --- arch/arm/mm/dma-mapping.c | 95 +------------------------------------------- include/linux/iommu-helper.h | 8 ++++ lib/iommu-helper.c | 95 ++++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 105 insertions(+), 93 deletions(-) diff --git a/arch/arm/mm/dma-mapping.c b/arch/arm/mm/dma-mapping.c index 38fc146..268004c 100644 --- a/arch/arm/mm/dma-mapping.c +++ b/arch/arm/mm/dma-mapping.c @@ -1161,98 +1161,6 @@ static inline void __free_iova(struct dma_iommu_mapping *mapping, spin_unlock_irqrestore(&mapping->lock, flags); } -static struct page **__iommu_alloc_buffer(struct device *dev, size_t size, - gfp_t gfp, struct dma_attrs *attrs) -{ - struct page **pages; - int count = size >> PAGE_SHIFT; - int array_size = count * sizeof(struct page *); - int i = 0; - - if (array_size <= PAGE_SIZE) - pages = kzalloc(array_size, gfp); - else - pages = vzalloc(array_size); - if (!pages) - return NULL; - - if (dma_get_attr(DMA_ATTR_FORCE_CONTIGUOUS, attrs)) - { - unsigned long order = get_order(size); - struct page *page; - - page = dma_alloc_from_contiguous(dev, count, order); - if (!page) - goto error; - - __dma_clear_buffer(page, size); - - for (i = 0; i < count; i++) - pages[i] = page + i; - - return pages; - } - - /* - * IOMMU can map any pages, so himem can also be used here - */ - gfp |= __GFP_NOWARN | __GFP_HIGHMEM; - - while (count) { - int j, order = __fls(count); - - pages[i] = alloc_pages(gfp, order); - while (!pages[i] && order) - pages[i] = alloc_pages(gfp, --order); - if (!pages[i]) - goto error; - - if (order) { - split_page(pages[i], order); - j = 1 << order; - while (--j) - pages[i + j] = pages[i] + j; - } - - __dma_clear_buffer(pages[i], PAGE_SIZE << order); - i += 1 << order; - count -= 1 << order; - } - - return pages; -error: - while (i--) - if (pages[i]) - __free_pages(pages[i], 0); - if (array_size <= PAGE_SIZE) - kfree(pages); - else - vfree(pages); - return NULL; -} - -static int __iommu_free_buffer(struct device *dev, struct page **pages, - size_t size, struct dma_attrs *attrs) -{ - int count = size >> PAGE_SHIFT; - int array_size = count * sizeof(struct page *); - int i; - - if (dma_get_attr(DMA_ATTR_FORCE_CONTIGUOUS, attrs)) { - dma_release_from_contiguous(dev, pages[0], count); - } else { - for (i = 0; i < count; i++) - if (pages[i]) - __free_pages(pages[i], 0); - } - - if (array_size <= PAGE_SIZE) - kfree(pages); - else - vfree(pages); - return 0; -} - /* * Create a CPU mapping for a specified pages */ @@ -1417,7 +1325,8 @@ static void *arm_iommu_alloc_attrs(struct device *dev, size_t size, */ gfp &= ~(__GFP_COMP); - pages = __iommu_alloc_buffer(dev, size, gfp, attrs); + pages = __iommu_alloc_buffer(dev, size, gfp, attrs, + __dma_clear_buffer); if (!pages) return NULL; diff --git a/include/linux/iommu-helper.h b/include/linux/iommu-helper.h index c6a315d..b27b7cb8 100644 --- a/include/linux/iommu-helper.h +++ b/include/linux/iommu-helper.h @@ -2,6 +2,7 @@ #define _LINUX_IOMMU_HELPER_H #include <linux/kernel.h> +#include <linux/dma-attrs.h> #ifdef CONFIG_DMA_USE_IOMMU_HELPER_MAPPING struct dma_iommu_mapping { @@ -19,6 +20,13 @@ struct dma_iommu_mapping { struct kref kref; }; +extern struct page **__iommu_alloc_buffer(struct device *dev, size_t size, + gfp_t gfp, struct dma_attrs *attrs, + void (*arch_clear_buffer_cb)(struct page*, size_t)); + +extern int __iommu_free_buffer(struct device *dev, struct page **pages, + size_t size, struct dma_attrs *attrs); + extern void __iommu_detach_device(struct device *dev); extern void __iommu_release_mapping(struct dma_iommu_mapping *mapping); diff --git a/lib/iommu-helper.c b/lib/iommu-helper.c index 28daaa5..e0f643a 100644 --- a/lib/iommu-helper.c +++ b/lib/iommu-helper.c @@ -13,6 +13,8 @@ #include <linux/slab.h> #include <linux/vmalloc.h> #include <linux/errno.h> +#include <linux/dma-contiguous.h> +#include <linux/mm.h> #endif int iommu_is_span_boundary(unsigned int index, unsigned int nr, @@ -51,6 +53,99 @@ EXPORT_SYMBOL(iommu_area_alloc); #ifdef CONFIG_DMA_USE_IOMMU_HELPER_MAPPING +struct page **__iommu_alloc_buffer(struct device *dev, size_t size, + gfp_t gfp, struct dma_attrs *attrs, + void (*arch_clear_buffer_cb)(struct page*, size_t)) +{ + struct page **pages; + int count = size >> PAGE_SHIFT; + int array_size = count * sizeof(struct page *); + int i = 0; + + if (array_size <= PAGE_SIZE) + pages = kzalloc(array_size, gfp); + else + pages = vzalloc(array_size); + if (!pages) + return NULL; + + if (dma_get_attr(DMA_ATTR_FORCE_CONTIGUOUS, attrs)) { + unsigned long order = get_order(size); + struct page *page; + + page = dma_alloc_from_contiguous(dev, count, order); + if (!page) + goto error; + + if (arch_clear_buffer_cb) + arch_clear_buffer_cb(page, size); + + for (i = 0; i < count; i++) + pages[i] = page + i; + + return pages; + } + + /* + * IOMMU can map any pages, so himem can also be used here + */ + gfp |= __GFP_NOWARN | __GFP_HIGHMEM; + + while (count) { + int j, order = __fls(count); + + pages[i] = alloc_pages(gfp, order); + while (!pages[i] && order) + pages[i] = alloc_pages(gfp, --order); + if (!pages[i]) + goto error; + + if (order) { + split_page(pages[i], order); + j = 1 << order; + while (--j) + pages[i + j] = pages[i] + j; + } + if (arch_clear_buffer_cb) + arch_clear_buffer_cb(pages[i], PAGE_SIZE << order); + i += 1 << order; + count -= 1 << order; + } + + return pages; +error: + while (i--) + if (pages[i]) + __free_pages(pages[i], 0); + if (array_size <= PAGE_SIZE) + kfree(pages); + else + vfree(pages); + return NULL; +} + +int __iommu_free_buffer(struct device *dev, struct page **pages, + size_t size, struct dma_attrs *attrs) +{ + int count = size >> PAGE_SHIFT; + int array_size = count * sizeof(struct page *); + int i; + + if (dma_get_attr(DMA_ATTR_FORCE_CONTIGUOUS, attrs)) { + dma_release_from_contiguous(dev, pages[0], count); + } else { + for (i = 0; i < count; i++) + if (pages[i]) + __free_pages(pages[i], 0); + } + + if (array_size <= PAGE_SIZE) + kfree(pages); + else + vfree(pages); + return 0; +} + /** * __iommu_init_mapping * @bus: pointer to the bus holding the client device (for IOMMU calls) -- 1.8.1.3 ^ permalink raw reply related [flat|nested] 6+ messages in thread
* [PATCH 4/4] arm:dma-iommu: Move out complete func defs 2014-06-02 10:19 ` [PATCH 3/4] arm: dma-mapping: Refractor iommu_alloc/free funcs ritesh.harjani at gmail.com @ 2014-06-02 10:19 ` ritesh.harjani at gmail.com 0 siblings, 0 replies; 6+ messages in thread From: ritesh.harjani at gmail.com @ 2014-06-02 10:19 UTC (permalink / raw) To: linux-arm-kernel From: Ritesh Harjani <ritesh.harjani@gmail.com> Move out complete function definations from arch/arm/dma-mapping to lib/iommu-helper 1. Moved out iova alloc/free routine and make them statically defined. 2. Moved out complete function definations calling for alloc/free_iova routing to lib/iommu-helper.c 3. Seprated out cache maintainance from iommu_map/unmap function routine, to be called from within arch/arm/dma-mapping.c Change-Id: I6abfa820fe1450b7de70de1a1ace15263a29dfc4 Signed-off-by: Ritesh Harjani <ritesh.harjani@gmail.com> --- arch/arm/Kconfig | 42 ++--- arch/arm/mm/dma-mapping.c | 362 +++---------------------------------------- include/linux/iommu-helper.h | 28 +++- lib/iommu-helper.c | 329 ++++++++++++++++++++++++++++++++++++++- 4 files changed, 399 insertions(+), 362 deletions(-) diff --git a/arch/arm/Kconfig b/arch/arm/Kconfig index 20717fb..977427d 100644 --- a/arch/arm/Kconfig +++ b/arch/arm/Kconfig @@ -80,27 +80,6 @@ config ARM_DMA_USE_IOMMU select NEED_SG_DMA_LENGTH select DMA_USE_IOMMU_HELPER_MAPPING -if ARM_DMA_USE_IOMMU - -config ARM_DMA_IOMMU_ALIGNMENT - int "Maximum PAGE_SIZE order of alignment for DMA IOMMU buffers" - range 4 9 - default 8 - help - DMA mapping framework by default aligns all buffers to the smallest - PAGE_SIZE order which is greater than or equal to the requested buffer - size. This works well for buffers up to a few hundreds kilobytes, but - for larger buffers it just a waste of address space. Drivers which has - relatively small addressing window (like 64Mib) might run out of - virtual space with just a few allocations. - - With this parameter you can specify the maximum PAGE_SIZE order for - DMA IOMMU buffers. Larger buffers will be aligned only to this - specified order. The order is expressed as a power of two multiplied - by the PAGE_SIZE. - -endif - config HAVE_PWM bool @@ -1949,6 +1928,27 @@ config IOMMU_HELPER config DMA_USE_IOMMU_HELPER_MAPPING def_bool n +if DMA_USE_IOMMU_HELPER_MAPPING + +config DMA_IOMMU_ALIGNMENT + int "Maximum PAGE_SIZE order of alignment for DMA IOMMU buffers" + range 4 9 + default 8 + help + DMA mapping framework by default aligns all buffers to the smallest + PAGE_SIZE order which is greater than or equal to the requested buffer + size. This works well for buffers up to a few hundreds kilobytes, but + for larger buffers it just a waste of address space. Drivers which has + relatively small addressing window (like 64Mib) might run out of + virtual space with just a few allocations. + + With this parameter you can specify the maximum PAGE_SIZE order for + DMA IOMMU buffers. Larger buffers will be aligned only to this + specified order. The order is expressed as a power of two multiplied + by the PAGE_SIZE. + +endif + config XEN_DOM0 def_bool y depends on XEN diff --git a/arch/arm/mm/dma-mapping.c b/arch/arm/mm/dma-mapping.c index 268004c..6546fba 100644 --- a/arch/arm/mm/dma-mapping.c +++ b/arch/arm/mm/dma-mapping.c @@ -1066,101 +1066,6 @@ fs_initcall(dma_debug_do_init); /* IOMMU */ -static int extend_iommu_mapping(struct dma_iommu_mapping *mapping); - -static inline dma_addr_t __alloc_iova(struct dma_iommu_mapping *mapping, - size_t size) -{ - unsigned int order = get_order(size); - unsigned int align = 0; - unsigned int count, start; - size_t mapping_size = mapping->bits << PAGE_SHIFT; - unsigned long flags; - dma_addr_t iova; - int i; - - if (order > CONFIG_ARM_DMA_IOMMU_ALIGNMENT) - order = CONFIG_ARM_DMA_IOMMU_ALIGNMENT; - - count = PAGE_ALIGN(size) >> PAGE_SHIFT; - align = (1 << order) - 1; - - spin_lock_irqsave(&mapping->lock, flags); - for (i = 0; i < mapping->nr_bitmaps; i++) { - start = bitmap_find_next_zero_area(mapping->bitmaps[i], - mapping->bits, 0, count, align); - - if (start > mapping->bits) - continue; - - bitmap_set(mapping->bitmaps[i], start, count); - break; - } - - /* - * No unused range found. Try to extend the existing mapping - * and perform a second attempt to reserve an IO virtual - * address range of size bytes. - */ - if (i == mapping->nr_bitmaps) { - if (extend_iommu_mapping(mapping)) { - spin_unlock_irqrestore(&mapping->lock, flags); - return DMA_ERROR_CODE; - } - - start = bitmap_find_next_zero_area(mapping->bitmaps[i], - mapping->bits, 0, count, align); - - if (start > mapping->bits) { - spin_unlock_irqrestore(&mapping->lock, flags); - return DMA_ERROR_CODE; - } - - bitmap_set(mapping->bitmaps[i], start, count); - } - spin_unlock_irqrestore(&mapping->lock, flags); - - iova = mapping->base + (mapping_size * i); - iova += start << PAGE_SHIFT; - - return iova; -} - -static inline void __free_iova(struct dma_iommu_mapping *mapping, - dma_addr_t addr, size_t size) -{ - unsigned int start, count; - size_t mapping_size = mapping->bits << PAGE_SHIFT; - unsigned long flags; - dma_addr_t bitmap_base; - u32 bitmap_index; - - if (!size) - return; - - bitmap_index = (u32) (addr - mapping->base) / (u32) mapping_size; - BUG_ON(addr < mapping->base || bitmap_index > mapping->extensions); - - bitmap_base = mapping->base + mapping_size * bitmap_index; - - start = (addr - bitmap_base) >> PAGE_SHIFT; - - if (addr + size > bitmap_base + mapping_size) { - /* - * The address range to be freed reaches into the iova - * range of the next bitmap. This should not happen as - * we don't allow this in __alloc_iova (at the - * moment). - */ - BUG(); - } else - count = size >> PAGE_SHIFT; - - spin_lock_irqsave(&mapping->lock, flags); - bitmap_clear(mapping->bitmaps[bitmap_index], start, count); - spin_unlock_irqrestore(&mapping->lock, flags); -} - /* * Create a CPU mapping for a specified pages */ @@ -1194,62 +1099,6 @@ err: return NULL; } -/* - * Create a mapping in device IO address space for specified pages - */ -static dma_addr_t -__iommu_create_mapping(struct device *dev, struct page **pages, size_t size) -{ - struct dma_iommu_mapping *mapping = dev->mapping; - unsigned int count = PAGE_ALIGN(size) >> PAGE_SHIFT; - dma_addr_t dma_addr, iova; - int i, ret = DMA_ERROR_CODE; - - dma_addr = __alloc_iova(mapping, size); - if (dma_addr == DMA_ERROR_CODE) - return dma_addr; - - iova = dma_addr; - for (i = 0; i < count; ) { - unsigned int next_pfn = page_to_pfn(pages[i]) + 1; - phys_addr_t phys = page_to_phys(pages[i]); - unsigned int len, j; - - for (j = i + 1; j < count; j++, next_pfn++) - if (page_to_pfn(pages[j]) != next_pfn) - break; - - len = (j - i) << PAGE_SHIFT; - ret = iommu_map(mapping->domain, iova, phys, len, - IOMMU_READ|IOMMU_WRITE); - if (ret < 0) - goto fail; - iova += len; - i = j; - } - return dma_addr; -fail: - iommu_unmap(mapping->domain, dma_addr, iova-dma_addr); - __free_iova(mapping, dma_addr, size); - return DMA_ERROR_CODE; -} - -static int __iommu_remove_mapping(struct device *dev, dma_addr_t iova, size_t size) -{ - struct dma_iommu_mapping *mapping = dev->mapping; - - /* - * add optional in-page offset from iova to size and align - * result to page size - */ - size = PAGE_ALIGN((iova & ~PAGE_MASK) + size); - iova &= PAGE_MASK; - - iommu_unmap(mapping->domain, iova, size); - __free_iova(mapping, iova, size); - return 0; -} - static struct page **__atomic_get_pages(void *addr) { struct dma_pool *pool = &atomic_pool; @@ -1421,120 +1270,6 @@ static int arm_iommu_get_sgtable(struct device *dev, struct sg_table *sgt, GFP_KERNEL); } -static int __dma_direction_to_prot(enum dma_data_direction dir) -{ - int prot; - - switch (dir) { - case DMA_BIDIRECTIONAL: - prot = IOMMU_READ | IOMMU_WRITE; - break; - case DMA_TO_DEVICE: - prot = IOMMU_READ; - break; - case DMA_FROM_DEVICE: - prot = IOMMU_WRITE; - break; - default: - prot = 0; - } - - return prot; -} - -/* - * Map a part of the scatter-gather list into contiguous io address space - */ -static int __map_sg_chunk(struct device *dev, struct scatterlist *sg, - size_t size, dma_addr_t *handle, - enum dma_data_direction dir, struct dma_attrs *attrs, - bool is_coherent) -{ - struct dma_iommu_mapping *mapping = dev->mapping; - dma_addr_t iova, iova_base; - int ret = 0; - unsigned int count; - struct scatterlist *s; - int prot; - - size = PAGE_ALIGN(size); - *handle = DMA_ERROR_CODE; - - iova_base = iova = __alloc_iova(mapping, size); - if (iova == DMA_ERROR_CODE) - return -ENOMEM; - - for (count = 0, s = sg; count < (size >> PAGE_SHIFT); s = sg_next(s)) { - phys_addr_t phys = page_to_phys(sg_page(s)); - unsigned int len = PAGE_ALIGN(s->offset + s->length); - - if (!is_coherent && - !dma_get_attr(DMA_ATTR_SKIP_CPU_SYNC, attrs)) - __dma_page_cpu_to_dev(sg_page(s), s->offset, s->length, dir); - - prot = __dma_direction_to_prot(dir); - - ret = iommu_map(mapping->domain, iova, phys, len, prot); - if (ret < 0) - goto fail; - count += len >> PAGE_SHIFT; - iova += len; - } - *handle = iova_base; - - return 0; -fail: - iommu_unmap(mapping->domain, iova_base, count * PAGE_SIZE); - __free_iova(mapping, iova_base, size); - return ret; -} - -static int __iommu_map_sg(struct device *dev, struct scatterlist *sg, int nents, - enum dma_data_direction dir, struct dma_attrs *attrs, - bool is_coherent) -{ - struct scatterlist *s = sg, *dma = sg, *start = sg; - int i, count = 0; - unsigned int offset = s->offset; - unsigned int size = s->offset + s->length; - unsigned int max = dma_get_max_seg_size(dev); - - for (i = 1; i < nents; i++) { - s = sg_next(s); - - s->dma_address = DMA_ERROR_CODE; - s->dma_length = 0; - - if (s->offset || (size & ~PAGE_MASK) || size + s->length > max) { - if (__map_sg_chunk(dev, start, size, &dma->dma_address, - dir, attrs, is_coherent) < 0) - goto bad_mapping; - - dma->dma_address += offset; - dma->dma_length = size - offset; - - size = offset = s->offset; - start = s; - dma = sg_next(dma); - count += 1; - } - size += s->length; - } - if (__map_sg_chunk(dev, start, size, &dma->dma_address, dir, attrs, - is_coherent) < 0) - goto bad_mapping; - - dma->dma_address += offset; - dma->dma_length = size - offset; - - return count+1; - -bad_mapping: - for_each_sg(sg, s, count, i) - __iommu_remove_mapping(dev, sg_dma_address(s), sg_dma_len(s)); - return 0; -} - /** * arm_coherent_iommu_map_sg - map a set of SG buffers for streaming mode DMA * @dev: valid struct device pointer @@ -1550,7 +1285,7 @@ bad_mapping: int arm_coherent_iommu_map_sg(struct device *dev, struct scatterlist *sg, int nents, enum dma_data_direction dir, struct dma_attrs *attrs) { - return __iommu_map_sg(dev, sg, nents, dir, attrs, true); + return __iommu_map_sg(dev, sg, nents, dir, attrs); } /** @@ -1568,25 +1303,15 @@ int arm_coherent_iommu_map_sg(struct device *dev, struct scatterlist *sg, int arm_iommu_map_sg(struct device *dev, struct scatterlist *sg, int nents, enum dma_data_direction dir, struct dma_attrs *attrs) { - return __iommu_map_sg(dev, sg, nents, dir, attrs, false); -} - -static void __iommu_unmap_sg(struct device *dev, struct scatterlist *sg, - int nents, enum dma_data_direction dir, struct dma_attrs *attrs, - bool is_coherent) -{ struct scatterlist *s; - int i; - - for_each_sg(sg, s, nents, i) { - if (sg_dma_len(s)) - __iommu_remove_mapping(dev, sg_dma_address(s), - sg_dma_len(s)); - if (!is_coherent && - !dma_get_attr(DMA_ATTR_SKIP_CPU_SYNC, attrs)) - __dma_page_dev_to_cpu(sg_page(s), s->offset, - s->length, dir); + int i, ret; + ret = __iommu_map_sg(dev, sg, nents, dir, attrs); + if (!dma_get_attr(DMA_ATTR_SKIP_CPU_SYNC, attrs)) { + for_each_sg(sg, s, ret, i) + __dma_page_cpu_to_dev(sg_page(s), s->offset, + s->length, dir); } + return ret; } /** @@ -1602,7 +1327,7 @@ static void __iommu_unmap_sg(struct device *dev, struct scatterlist *sg, void arm_coherent_iommu_unmap_sg(struct device *dev, struct scatterlist *sg, int nents, enum dma_data_direction dir, struct dma_attrs *attrs) { - __iommu_unmap_sg(dev, sg, nents, dir, attrs, true); + __iommu_unmap_sg(dev, sg, nents, dir, attrs); } /** @@ -1618,7 +1343,16 @@ void arm_coherent_iommu_unmap_sg(struct device *dev, struct scatterlist *sg, void arm_iommu_unmap_sg(struct device *dev, struct scatterlist *sg, int nents, enum dma_data_direction dir, struct dma_attrs *attrs) { - __iommu_unmap_sg(dev, sg, nents, dir, attrs, false); + struct scatterlist *s; + int i; + + __iommu_unmap_sg(dev, sg, nents, dir, attrs); + + for_each_sg(sg, s, nents, i) { + if (!dma_get_attr(DMA_ATTR_SKIP_CPU_SYNC, attrs)) + __dma_page_dev_to_cpu(sg_page(s), s->offset, + s->length, dir); + } } /** @@ -1671,24 +1405,7 @@ static dma_addr_t arm_coherent_iommu_map_page(struct device *dev, struct page *p unsigned long offset, size_t size, enum dma_data_direction dir, struct dma_attrs *attrs) { - struct dma_iommu_mapping *mapping = dev->mapping; - dma_addr_t dma_addr; - int ret, prot, len = PAGE_ALIGN(size + offset); - - dma_addr = __alloc_iova(mapping, len); - if (dma_addr == DMA_ERROR_CODE) - return dma_addr; - - prot = __dma_direction_to_prot(dir); - - ret = iommu_map(mapping->domain, dma_addr, page_to_phys(page), len, prot); - if (ret < 0) - goto fail; - - return dma_addr + offset; -fail: - __free_iova(mapping, dma_addr, len); - return DMA_ERROR_CODE; + return __iommu_map_page(dev, page, offset, size, dir); } /** @@ -1708,7 +1425,7 @@ static dma_addr_t arm_iommu_map_page(struct device *dev, struct page *page, if (!dma_get_attr(DMA_ATTR_SKIP_CPU_SYNC, attrs)) __dma_page_cpu_to_dev(page, offset, size, dir); - return arm_coherent_iommu_map_page(dev, page, offset, size, dir, attrs); + return __iommu_map_page(dev, page, offset, size, dir); } /** @@ -1724,16 +1441,7 @@ static void arm_coherent_iommu_unmap_page(struct device *dev, dma_addr_t handle, size_t size, enum dma_data_direction dir, struct dma_attrs *attrs) { - struct dma_iommu_mapping *mapping = dev->mapping; - dma_addr_t iova = handle & PAGE_MASK; - int offset = handle & ~PAGE_MASK; - int len = PAGE_ALIGN(size + offset); - - if (!iova) - return; - - iommu_unmap(mapping->domain, iova, len); - __free_iova(mapping, iova, len); + __iommu_unmap_page(dev, handle, size, dir); } /** @@ -1753,16 +1461,12 @@ static void arm_iommu_unmap_page(struct device *dev, dma_addr_t handle, dma_addr_t iova = handle & PAGE_MASK; struct page *page = phys_to_page(iommu_iova_to_phys(mapping->domain, iova)); int offset = handle & ~PAGE_MASK; - int len = PAGE_ALIGN(size + offset); - - if (!iova) - return; if (!dma_get_attr(DMA_ATTR_SKIP_CPU_SYNC, attrs)) __dma_page_dev_to_cpu(page, offset, size, dir); - iommu_unmap(mapping->domain, iova, len); - __free_iova(mapping, iova, len); + __iommu_unmap_page(dev, handle, size, dir); + } static void arm_iommu_sync_single_for_cpu(struct device *dev, @@ -1848,24 +1552,6 @@ arm_iommu_create_mapping(struct bus_type *bus, dma_addr_t base, size_t size) } EXPORT_SYMBOL_GPL(arm_iommu_create_mapping); -static int extend_iommu_mapping(struct dma_iommu_mapping *mapping) -{ - int next_bitmap; - - if (mapping->nr_bitmaps > mapping->extensions) - return -EINVAL; - - next_bitmap = mapping->nr_bitmaps; - mapping->bitmaps[next_bitmap] = kzalloc(mapping->bitmap_size, - GFP_ATOMIC); - if (!mapping->bitmaps[next_bitmap]) - return -ENOMEM; - - mapping->nr_bitmaps++; - - return 0; -} - void arm_iommu_release_mapping(struct dma_iommu_mapping *mapping) { __iommu_release_mapping(mapping); diff --git a/include/linux/iommu-helper.h b/include/linux/iommu-helper.h index b27b7cb8..1e2e5f2 100644 --- a/include/linux/iommu-helper.h +++ b/include/linux/iommu-helper.h @@ -5,6 +5,12 @@ #include <linux/dma-attrs.h> #ifdef CONFIG_DMA_USE_IOMMU_HELPER_MAPPING +#include <linux/mm_types.h> +#include <linux/dma-debug.h> +#include <linux/kmemcheck.h> +#include <linux/kref.h> +#include <linux/dma-mapping.h> + struct dma_iommu_mapping { /* iommu specific data */ struct iommu_domain *domain; @@ -20,6 +26,25 @@ struct dma_iommu_mapping { struct kref kref; }; +extern dma_addr_t __iommu_create_mapping(struct device *dev, struct page **pages, + size_t size); + +extern int __iommu_remove_mapping(struct device *dev, dma_addr_t iova, + size_t size); + +extern dma_addr_t __iommu_map_page(struct device *dev, struct page *page, + unsigned long offset, size_t size, enum dma_data_direction dir); + +extern void __iommu_unmap_page(struct device *dev, dma_addr_t handle, + size_t size, enum dma_data_direction dir); + +extern int __iommu_map_sg(struct device *dev, struct scatterlist *sg, int nents, + enum dma_data_direction dir, struct dma_attrs *attrs); + +extern void __iommu_unmap_sg(struct device *dev, struct scatterlist *sg, + int nents, enum dma_data_direction dir, + struct dma_attrs *attrs); + extern struct page **__iommu_alloc_buffer(struct device *dev, size_t size, gfp_t gfp, struct dma_attrs *attrs, void (*arch_clear_buffer_cb)(struct page*, size_t)); @@ -29,14 +54,13 @@ extern int __iommu_free_buffer(struct device *dev, struct page **pages, extern void __iommu_detach_device(struct device *dev); -extern void __iommu_release_mapping(struct dma_iommu_mapping *mapping); - extern int __iommu_attach_device(struct device *dev, struct dma_iommu_mapping *mapping); extern struct dma_iommu_mapping * __iommu_init_mapping(struct bus_type *bus, dma_addr_t base, size_t size); +extern void __iommu_release_mapping(struct dma_iommu_mapping *mapping); #define to_dma_iommu_mapping(dev) ((dev)->mapping) #else #define to_dma_iommu_mapping(dev) NULL diff --git a/lib/iommu-helper.c b/lib/iommu-helper.c index e0f643a..75d900e 100644 --- a/lib/iommu-helper.c +++ b/lib/iommu-helper.c @@ -8,13 +8,14 @@ #ifdef CONFIG_DMA_USE_IOMMU_HELPER_MAPPING #include <linux/iommu.h> -#include <linux/device.h> #include <linux/iommu-helper.h> #include <linux/slab.h> #include <linux/vmalloc.h> #include <linux/errno.h> #include <linux/dma-contiguous.h> #include <linux/mm.h> + +#include <asm/dma-mapping.h> #endif int iommu_is_span_boundary(unsigned int index, unsigned int nr, @@ -53,6 +54,195 @@ EXPORT_SYMBOL(iommu_area_alloc); #ifdef CONFIG_DMA_USE_IOMMU_HELPER_MAPPING +/* IOMMU */ +static int extend_iommu_mapping(struct dma_iommu_mapping *mapping) +{ + int next_bitmap; + + if (mapping->nr_bitmaps > mapping->extensions) + return -EINVAL; + + next_bitmap = mapping->nr_bitmaps; + mapping->bitmaps[next_bitmap] = kzalloc(mapping->bitmap_size, + GFP_ATOMIC); + if (!mapping->bitmaps[next_bitmap]) + return -ENOMEM; + + mapping->nr_bitmaps++; + + return 0; +} + +static int __dma_direction_to_prot(enum dma_data_direction dir) +{ + int prot; + + switch (dir) { + case DMA_BIDIRECTIONAL: + prot = IOMMU_READ | IOMMU_WRITE; + break; + case DMA_TO_DEVICE: + prot = IOMMU_READ; + break; + case DMA_FROM_DEVICE: + prot = IOMMU_WRITE; + break; + default: + prot = 0; + } + + return prot; +} + +static inline dma_addr_t __alloc_iova(struct dma_iommu_mapping *mapping, + size_t size) +{ + unsigned int order = get_order(size); + unsigned int align = 0; + unsigned int count, start; + size_t mapping_size = mapping->bits << PAGE_SHIFT; + unsigned long flags; + dma_addr_t iova; + int i; + + if (order > CONFIG_DMA_IOMMU_ALIGNMENT) + order = CONFIG_DMA_IOMMU_ALIGNMENT; + + count = PAGE_ALIGN(size) >> PAGE_SHIFT; + align = (1 << order) - 1; + + spin_lock_irqsave(&mapping->lock, flags); + for (i = 0; i < mapping->nr_bitmaps; i++) { + start = bitmap_find_next_zero_area(mapping->bitmaps[i], + mapping->bits, 0, count, align); + + if (start > mapping->bits) + continue; + + bitmap_set(mapping->bitmaps[i], start, count); + break; + } + + /* + * No unused range found. Try to extend the existing mapping + * and perform a second attempt to reserve an IO virtual + * address range of size bytes. + */ + if (i == mapping->nr_bitmaps) { + if (extend_iommu_mapping(mapping)) { + spin_unlock_irqrestore(&mapping->lock, flags); + return DMA_ERROR_CODE; + } + + start = bitmap_find_next_zero_area(mapping->bitmaps[i], + mapping->bits, 0, count, align); + + if (start > mapping->bits) { + spin_unlock_irqrestore(&mapping->lock, flags); + return DMA_ERROR_CODE; + } + + bitmap_set(mapping->bitmaps[i], start, count); + } + spin_unlock_irqrestore(&mapping->lock, flags); + + iova = mapping->base + (mapping_size * i); + iova += start << PAGE_SHIFT; + + return iova; +} + +static inline void __free_iova(struct dma_iommu_mapping *mapping, + dma_addr_t addr, size_t size) +{ + unsigned int start, count; + size_t mapping_size = mapping->bits << PAGE_SHIFT; + unsigned long flags; + dma_addr_t bitmap_base; + u32 bitmap_index; + + if (!size) + return; + + bitmap_index = (u32) (addr - mapping->base) / (u32) mapping_size; + BUG_ON(addr < mapping->base || bitmap_index > mapping->extensions); + + bitmap_base = mapping->base + mapping_size * bitmap_index; + + start = (addr - bitmap_base) >> PAGE_SHIFT; + + if (addr + size > bitmap_base + mapping_size) { + /* + * The address range to be freed reaches into the iova + * range of the next bitmap. This should not happen as + * we don't allow this in __alloc_iova (at the + * moment). + */ + BUG(); + } else + count = size >> PAGE_SHIFT; + + spin_lock_irqsave(&mapping->lock, flags); + bitmap_clear(mapping->bitmaps[bitmap_index], start, count); + spin_unlock_irqrestore(&mapping->lock, flags); +} + +/* + * Create a mapping in device IO address space for specified pages + */ +dma_addr_t +__iommu_create_mapping(struct device *dev, struct page **pages, size_t size) +{ + struct dma_iommu_mapping *mapping = dev->mapping; + unsigned int count = PAGE_ALIGN(size) >> PAGE_SHIFT; + dma_addr_t dma_addr, iova; + int i, ret = DMA_ERROR_CODE; + + dma_addr = __alloc_iova(mapping, size); + if (dma_addr == DMA_ERROR_CODE) + return dma_addr; + + iova = dma_addr; + for (i = 0; i < count; ) { + unsigned int next_pfn = page_to_pfn(pages[i]) + 1; + phys_addr_t phys = page_to_phys(pages[i]); + unsigned int len, j; + + for (j = i + 1; j < count; j++, next_pfn++) + if (page_to_pfn(pages[j]) != next_pfn) + break; + + len = (j - i) << PAGE_SHIFT; + ret = iommu_map(mapping->domain, iova, phys, len, + IOMMU_READ|IOMMU_WRITE); + if (ret < 0) + goto fail; + iova += len; + i = j; + } + return dma_addr; +fail: + iommu_unmap(mapping->domain, dma_addr, iova-dma_addr); + __free_iova(mapping, dma_addr, size); + return DMA_ERROR_CODE; +} + +int __iommu_remove_mapping(struct device *dev, dma_addr_t iova, size_t size) +{ + struct dma_iommu_mapping *mapping = dev->mapping; + + /* + * add optional in-page offset from iova to size and align + * result to page size + */ + size = PAGE_ALIGN((iova & ~PAGE_MASK) + size); + iova &= PAGE_MASK; + + iommu_unmap(mapping->domain, iova, size); + __free_iova(mapping, iova, size); + return 0; +} + struct page **__iommu_alloc_buffer(struct device *dev, size_t size, gfp_t gfp, struct dma_attrs *attrs, void (*arch_clear_buffer_cb)(struct page*, size_t)) @@ -146,6 +336,143 @@ int __iommu_free_buffer(struct device *dev, struct page **pages, return 0; } +/* + * Map a part of the scatter-gather list into contiguous io address space + */ +static int __map_sg_chunk(struct device *dev, struct scatterlist *sg, + size_t size, dma_addr_t *handle, + enum dma_data_direction dir, struct dma_attrs *attrs) +{ + struct dma_iommu_mapping *mapping = dev->mapping; + dma_addr_t iova, iova_base; + int ret = 0; + unsigned int count; + struct scatterlist *s; + int prot; + + size = PAGE_ALIGN(size); + *handle = DMA_ERROR_CODE; + + iova_base = iova = __alloc_iova(mapping, size); + if (iova == DMA_ERROR_CODE) + return -ENOMEM; + + for (count = 0, s = sg; count < (size >> PAGE_SHIFT); s = sg_next(s)) { + phys_addr_t phys = page_to_phys(sg_page(s)); + unsigned int len = PAGE_ALIGN(s->offset + s->length); + + prot = __dma_direction_to_prot(dir); + + ret = iommu_map(mapping->domain, iova, phys, len, prot); + if (ret < 0) + goto fail; + count += len >> PAGE_SHIFT; + iova += len; + } + *handle = iova_base; + + return 0; +fail: + iommu_unmap(mapping->domain, iova_base, count * PAGE_SIZE); + __free_iova(mapping, iova_base, size); + return ret; +} + +int __iommu_map_sg(struct device *dev, struct scatterlist *sg, int nents, + enum dma_data_direction dir, struct dma_attrs *attrs) +{ + struct scatterlist *s = sg, *dma = sg, *start = sg; + int i, count = 0; + unsigned int offset = s->offset; + unsigned int size = s->offset + s->length; + unsigned int max = dma_get_max_seg_size(dev); + + for (i = 1; i < nents; i++) { + s = sg_next(s); + + s->dma_address = DMA_ERROR_CODE; + s->dma_length = 0; + + if (s->offset || (size & ~PAGE_MASK) || size + s->length > max) { + if (__map_sg_chunk(dev, start, size, &dma->dma_address, + dir, attrs) < 0) + goto bad_mapping; + + dma->dma_address += offset; + dma->dma_length = size - offset; + + size = offset = s->offset; + start = s; + dma = sg_next(dma); + count += 1; + } + size += s->length; + } + if (__map_sg_chunk(dev, start, size, &dma->dma_address, dir, attrs) < 0) + goto bad_mapping; + + dma->dma_address += offset; + dma->dma_length = size - offset; + + return count+1; + +bad_mapping: + for_each_sg(sg, s, count, i) + __iommu_remove_mapping(dev, sg_dma_address(s), sg_dma_len(s)); + return 0; +} + +void __iommu_unmap_sg(struct device *dev, struct scatterlist *sg, + int nents, enum dma_data_direction dir, struct dma_attrs *attrs) +{ + struct scatterlist *s; + int i; + + for_each_sg(sg, s, nents, i) { + if (sg_dma_len(s)) + __iommu_remove_mapping(dev, sg_dma_address(s), + sg_dma_len(s)); +} +} + +dma_addr_t __iommu_map_page(struct device *dev, struct page *page, + unsigned long offset, size_t size, enum dma_data_direction dir) +{ + struct dma_iommu_mapping *mapping = dev->mapping; + dma_addr_t dma_addr; + int ret, prot, len = PAGE_ALIGN(size + offset); + + dma_addr = __alloc_iova(mapping, len); + if (dma_addr == DMA_ERROR_CODE) + return dma_addr; + + prot = __dma_direction_to_prot(dir); + + ret = iommu_map(mapping->domain, dma_addr, page_to_phys(page), len, prot); + if (ret < 0) + goto fail; + + return dma_addr + offset; +fail: + __free_iova(mapping, dma_addr, len); + return DMA_ERROR_CODE; +} + +void __iommu_unmap_page(struct device *dev, dma_addr_t handle, + size_t size, enum dma_data_direction dir) +{ + struct dma_iommu_mapping *mapping = dev->mapping; + dma_addr_t iova = handle & PAGE_MASK; + int offset = handle & ~PAGE_MASK; + int len = PAGE_ALIGN(size + offset); + + if (!iova) + return; + + iommu_unmap(mapping->domain, iova, len); + __free_iova(mapping, iova, len); +} + /** * __iommu_init_mapping * @bus: pointer to the bus holding the client device (for IOMMU calls) -- 1.8.1.3 ^ permalink raw reply related [flat|nested] 6+ messages in thread
* [PATCH 0/4] arm:dma-mapping Refactoring iommu dma-mapping code 2014-06-02 10:19 [PATCH 0/4] arm:dma-mapping Refactoring iommu dma-mapping code ritesh.harjani at gmail.com 2014-06-02 10:19 ` [PATCH 1/4] arm: dma-iommu: Move out dma_iommu_mapping struct ritesh.harjani at gmail.com @ 2014-06-03 13:01 ` Will Deacon 1 sibling, 0 replies; 6+ messages in thread From: Will Deacon @ 2014-06-03 13:01 UTC (permalink / raw) To: linux-arm-kernel On Mon, Jun 02, 2014 at 11:19:19AM +0100, ritesh.harjani at gmail.com wrote: > From: Ritesh Harjani <ritesh.harjani@gmail.com> > Hi All, Hi Ritesh, Thanks for the new patches. I have a few comments on the series as a whole. > This patch series is to refractor iommu related common code from > arch/arm/dma-mapping.c to lib/iommu-helper.c based on the various > discussions with the maintainers/experts [1]. > > Currently the only user of the common lib/iommu-helper code will > be ARM & ARM64 but later various architecture might try to use this > iommu lib helper functions. > > Major change of this refactoring starts with bringing out struct dma_iommu_mapping > *mapping variable from arch/arm/include/asm/device.h to include/linux/device.h > and by moving out complete structure defination of dma_iommu_mapping to > inclue/linux/iommu-helper.h. Link [2] give more details on why this was done, > also this change got approval from Will Daecon [2]. Well, I can't approve changes to include/linux/device.h, so that probably needs to be acked by Grant and/or Greg. Could you split that patch out into a separate change please, so that it can go in independently? Also, I think you could merge patches 2 and 3, no? > There are 1/2 more function definitions which I can think of moving out, but > those can be done once this patch series is approved as those are not very > big changes. This certainly looks good to start with, although I think you should consider renaming the functions in the helper library so that they aren't prefixed with double underscores. Maybe iommu_helper_* instead? Finally, please drop the ChangeId entries from your commit messages (and you've consistently misspelled refactor as refractor). Will ^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2014-06-03 13:01 UTC | newest] Thread overview: 6+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2014-06-02 10:19 [PATCH 0/4] arm:dma-mapping Refactoring iommu dma-mapping code ritesh.harjani at gmail.com 2014-06-02 10:19 ` [PATCH 1/4] arm: dma-iommu: Move out dma_iommu_mapping struct ritesh.harjani at gmail.com 2014-06-02 10:19 ` [PATCH 2/4] arm: dma-mapping: Refractor attach/detach dev function calls ritesh.harjani at gmail.com 2014-06-02 10:19 ` [PATCH 3/4] arm: dma-mapping: Refractor iommu_alloc/free funcs ritesh.harjani at gmail.com 2014-06-02 10:19 ` [PATCH 4/4] arm:dma-iommu: Move out complete func defs ritesh.harjani at gmail.com 2014-06-03 13:01 ` [PATCH 0/4] arm:dma-mapping Refactoring iommu dma-mapping code Will Deacon
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.