[PATCH v5 0/2] iommu/smmu-v3: Workaround for hisilicon 161010801 erratum(reserve HW MSI)

[PATCH v5 0/2] iommu/smmu-v3: Workaround for hisilicon 161010801 erratum(reserve HW MSI)

[Shameer Kolothum]

On certain HiSilicon platforms (Hip06/Hip07) the GIC ITS and
PCIe RC deviates from the standard implementation and this breaks
PCIe MSI functionality when SMMU is enabled.

The HiSilicon erratum 161010801 describes this limitation of certain
HiSilicon platforms to support the SMMU mappings for MSI transactions.
On these platforms GICv3 ITS translator is presented with the deviceID
by extending the MSI payload data to 64 bits to include the deviceID.
Hence, the PCIe controller on this platforms has to differentiate the
MSI payload against other DMA payload and has to modify the MSI payload.
This basically makes it difficult for this platforms to have a SMMU
translation for MSI.

To implement this quirk, the following changes are incorporated:

1. Added a generic helper function to IORT code to retrieve and
   reserve the HW ITS address regions.
2. Modified iommu_dma_get_resv_regions() to reserve the hw msi
   regions which means these address regions will not be translated and
   will be excluded from iova allocations.

Note: This requires patches [1] and [2] for vfio pass-through support
on these platforms.

Thanks,
Shameer

[1]: https://lkml.org/lkml/2017/7/27/388
[2]: https://lkml.org/lkml/2017/7/27/386

Changelog:

v4 --> v5
Addressed comments from Robin and Lorenzo:
-Added a comment to make it clear that, for now, only straightforward 
 HW topologies are handled while reserving ITS regions(patch #1).

v3 --> v4
Rebased on 4.13-rc1.
Addressed comments from Robin, Will and Lorenzo:
-As suggested by Robin, moved the ITS msi reservation into
 iommu_dma_get_resv_regions().
-Added its_count != resv region failure case(patch #1).

v2 --> v3
Addressed comments from Lorenzo and Robin:
-Removed dev_is_pci() check in smmuV3 driver.
-Don't treat device not having an ITS mapping as an error in
 iort helper function.

v1 --> v2
-patch 2/2: Invoke iort helper fn based on fwnode type(acpi).

RFCv2 -->PATCH
-Incorporated Lorenzo's review comments.

RFC v1 --> RFC v2
Based on Robin's review comments,
-Removed  the generic erratum framework.
-Using IORT/MADT tables to retrieve the ITS base addr instead
 of vendor specific CSRT table.

Shameer Kolothum (2):
  ACPI/IORT: Add ITS address regions reservation helper
  iommu/dma: Add HW MSI address regions reservation

 drivers/acpi/arm64/iort.c        | 95 ++++++++++++++++++++++++++++++++++++++--
 drivers/iommu/dma-iommu.c        |  8 +++-
 drivers/irqchip/irq-gic-v3-its.c |  3 +-
 include/linux/acpi_iort.h        |  7 ++-
 4 files changed, 107 insertions(+), 6 deletions(-)

-- 
1.9.1

[PATCH v5 1/2] ACPI/IORT: Add ITS address regions reservation helper

[Shameer Kolothum]

On some platforms ITS address regions have to be excluded from normal
IOVA allocation in that they are detected and decoded in a HW specific
way by system components and so they cannot be considered normal IOVA
address space.

Add an helper function that retrieves ITS address regions through IORT
device <-> ITS mappings and reserves it so that these regions will not
be translated by IOMMU and will be excluded from IOVA allocations.

Signed-off-by: Shameer Kolothum <shameerali.kolothum.thodi-hv44wF8Li93QT0dZR+AlfA@public.gmane.org>
[lorenzo.pieralisi-5wv7dgnIgG8@public.gmane.org: updated commit log/added comments]
Signed-off-by: Lorenzo Pieralisi <lorenzo.pieralisi-5wv7dgnIgG8@public.gmane.org>
---
 drivers/acpi/arm64/iort.c        | 95 ++++++++++++++++++++++++++++++++++++++--
 drivers/irqchip/irq-gic-v3-its.c |  3 +-
 include/linux/acpi_iort.h        |  7 ++-
 3 files changed, 100 insertions(+), 5 deletions(-)

diff --git a/drivers/acpi/arm64/iort.c b/drivers/acpi/arm64/iort.c
index a3215ee..86b5a51 100644
--- a/drivers/acpi/arm64/iort.c
+++ b/drivers/acpi/arm64/iort.c
@@ -39,6 +39,7 @@
 struct iort_its_msi_chip {
 	struct list_head	list;
 	struct fwnode_handle	*fw_node;
+	phys_addr_t		base_addr;
 	u32			translation_id;
 };
 
@@ -136,14 +137,16 @@ typedef acpi_status (*iort_find_node_callback)
 static DEFINE_SPINLOCK(iort_msi_chip_lock);
 
 /**
- * iort_register_domain_token() - register domain token and related ITS ID
- * to the list from where we can get it back later on.
+ * iort_register_domain_token() - register domain token along with related
+ * ITS ID and base address to the list from where we can get it back later on.
  * @trans_id: ITS ID.
+ * @base: ITS base address.
  * @fw_node: Domain token.
  *
  * Returns: 0 on success, -ENOMEM if no memory when allocating list element
  */
-int iort_register_domain_token(int trans_id, struct fwnode_handle *fw_node)
+int iort_register_domain_token(int trans_id, phys_addr_t base,
+			       struct fwnode_handle *fw_node)
 {
 	struct iort_its_msi_chip *its_msi_chip;
 
@@ -153,6 +156,7 @@ int iort_register_domain_token(int trans_id, struct fwnode_handle *fw_node)
 
 	its_msi_chip->fw_node = fw_node;
 	its_msi_chip->translation_id = trans_id;
+	its_msi_chip->base_addr = base;
 
 	spin_lock(&iort_msi_chip_lock);
 	list_add(&its_msi_chip->list, &iort_msi_chip_list);
@@ -481,6 +485,24 @@ int iort_pmsi_get_dev_id(struct device *dev, u32 *dev_id)
 	return -ENODEV;
 }
 
+static int __maybe_unused iort_find_its_base(u32 its_id, phys_addr_t *base)
+{
+	struct iort_its_msi_chip *its_msi_chip;
+	bool match = false;
+
+	spin_lock(&iort_msi_chip_lock);
+	list_for_each_entry(its_msi_chip, &iort_msi_chip_list, list) {
+		if (its_msi_chip->translation_id == its_id) {
+			*base = its_msi_chip->base_addr;
+			match = true;
+			break;
+		}
+	}
+	spin_unlock(&iort_msi_chip_lock);
+
+	return match ? 0 : -ENODEV;
+}
+
 /**
  * iort_dev_find_its_id() - Find the ITS identifier for a device
  * @dev: The device.
@@ -639,6 +661,71 @@ int iort_add_device_replay(const struct iommu_ops *ops, struct device *dev)
 
 	return err;
 }
+
+/**
+ * iort_iommu_its_get_resv_regions - Reserved region driver helper
+ * @dev: Device from iommu_get_resv_regions()
+ * @list: Reserved region list from iommu_get_resv_regions()
+ *
+ * Returns: Number of reserved regions on success(0 if no associated ITS),
+ *          appropriate error value otherwise.
+ */
+int iort_iommu_its_get_resv_regions(struct device *dev, struct list_head *head)
+{
+	struct acpi_iort_its_group *its;
+	struct acpi_iort_node *node, *its_node = NULL;
+	int i, resv = 0;
+
+	node = iort_find_dev_node(dev);
+	if (!node)
+		return -ENODEV;
+
+	/*
+	 * Current logic to reserve ITS regions relies on HW topologies
+	 * where a given PCI or named component maps its IDs to only one
+	 * ITS group; if a PCI or named component can map its IDs to
+	 * different ITS groups through IORT mappings this function has
+	 * to be reworked to ensure we reserve regions for all ITS groups
+	 * a given PCI or named component may map IDs to.
+	 */
+	if (dev_is_pci(dev)) {
+		u32 rid;
+
+		pci_for_each_dma_alias(to_pci_dev(dev), __get_pci_rid, &rid);
+		its_node = iort_node_map_id(node, rid, NULL, IORT_MSI_TYPE);
+	} else {
+		for (i = 0; i < node->mapping_count; i++) {
+			its_node = iort_node_map_platform_id(node, NULL,
+							 IORT_MSI_TYPE, i);
+			if (its_node)
+				break;
+		}
+	}
+
+	if (!its_node)
+		return 0;
+
+	/* Move to ITS specific data */
+	its = (struct acpi_iort_its_group *)its_node->node_data;
+
+	for (i = 0; i < its->its_count; i++) {
+		phys_addr_t base;
+
+		if (!iort_find_its_base(its->identifiers[i], &base)) {
+			int prot = IOMMU_WRITE | IOMMU_NOEXEC | IOMMU_MMIO;
+			struct iommu_resv_region *region;
+
+			region = iommu_alloc_resv_region(base, SZ_128K, prot,
+							 IOMMU_RESV_MSI);
+			if (region) {
+				list_add_tail(&region->list, head);
+				resv++;
+			}
+		}
+	}
+
+	return (resv == its->its_count) ? resv : -ENODEV;
+}
 #else
 static inline
 const struct iommu_ops *iort_fwspec_iommu_ops(struct iommu_fwspec *fwspec)
@@ -646,6 +733,8 @@ const struct iommu_ops *iort_fwspec_iommu_ops(struct iommu_fwspec *fwspec)
 static inline
 int iort_add_device_replay(const struct iommu_ops *ops, struct device *dev)
 { return 0; }
+int iort_iommu_its_get_resv_regions(struct device *dev, struct list_head *head)
+{ return -ENODEV; }
 #endif
 
 static const struct iommu_ops *iort_iommu_xlate(struct device *dev,
diff --git a/drivers/irqchip/irq-gic-v3-its.c b/drivers/irqchip/irq-gic-v3-its.c
index 6893287..77322b3 100644
--- a/drivers/irqchip/irq-gic-v3-its.c
+++ b/drivers/irqchip/irq-gic-v3-its.c
@@ -1928,7 +1928,8 @@ static int __init gic_acpi_parse_madt_its(struct acpi_subtable_header *header,
 		return -ENOMEM;
 	}
 
-	err = iort_register_domain_token(its_entry->translation_id, dom_handle);
+	err = iort_register_domain_token(its_entry->translation_id, res.start,
+					 dom_handle);
 	if (err) {
 		pr_err("ITS@%pa: Unable to register GICv3 ITS domain token (ITS ID %d) to IORT\n",
 		       &res.start, its_entry->translation_id);
diff --git a/include/linux/acpi_iort.h b/include/linux/acpi_iort.h
index 8379d40..d7ed49c 100644
--- a/include/linux/acpi_iort.h
+++ b/include/linux/acpi_iort.h
@@ -26,7 +26,8 @@
 #define IORT_IRQ_MASK(irq)		(irq & 0xffffffffULL)
 #define IORT_IRQ_TRIGGER_MASK(irq)	((irq >> 32) & 0xffffffffULL)
 
-int iort_register_domain_token(int trans_id, struct fwnode_handle *fw_node);
+int iort_register_domain_token(int trans_id, phys_addr_t base,
+			       struct fwnode_handle *fw_node);
 void iort_deregister_domain_token(int trans_id);
 struct fwnode_handle *iort_find_domain_token(int trans_id);
 #ifdef CONFIG_ACPI_IORT
@@ -38,6 +39,7 @@
 /* IOMMU interface */
 void iort_set_dma_mask(struct device *dev);
 const struct iommu_ops *iort_iommu_configure(struct device *dev);
+int iort_iommu_its_get_resv_regions(struct device *dev, struct list_head *head);
 #else
 static inline void acpi_iort_init(void) { }
 static inline u32 iort_msi_map_rid(struct device *dev, u32 req_id)
@@ -51,6 +53,9 @@ static inline void iort_set_dma_mask(struct device *dev) { }
 static inline
 const struct iommu_ops *iort_iommu_configure(struct device *dev)
 { return NULL; }
+static inline
+int iort_iommu_its_get_resv_regions(struct device *dev, struct list_head *head)
+{ return -ENODEV; }
 #endif
 
 #endif /* __ACPI_IORT_H__ */
-- 
1.9.1

Re: [PATCH v5 1/2] ACPI/IORT: Add ITS address regions reservation helper

[Robin Murphy]

On 01/08/17 11:49, Shameer Kolothum wrote:
> On some platforms ITS address regions have to be excluded from normal
> IOVA allocation in that they are detected and decoded in a HW specific
> way by system components and so they cannot be considered normal IOVA
> address space.
> 
> Add an helper function that retrieves ITS address regions through IORT
> device <-> ITS mappings and reserves it so that these regions will not
> be translated by IOMMU and will be excluded from IOVA allocations.

I've just realised that we no longer seem to have a check that ensures
the regions are only reserved on platforms that need it - if not, then
we're going to break everything else that does have an ITS behind SMMU
translation as expected.

It feels like IORT should know enough to be able to make that decision
internally, but if not (or if it would be hideous to do so), then I
guess my idea for patch #2 was a bust and we probably do need to go back
to calling directly from the SMMU driver based on the SMMU model.

Robin.

> Signed-off-by: Shameer Kolothum <shameerali.kolothum.thodi@huawei.com>
> [lorenzo.pieralisi@arm.com: updated commit log/added comments]
> Signed-off-by: Lorenzo Pieralisi <lorenzo.pieralisi@arm.com>
> ---
>  drivers/acpi/arm64/iort.c        | 95 ++++++++++++++++++++++++++++++++++++++--
>  drivers/irqchip/irq-gic-v3-its.c |  3 +-
>  include/linux/acpi_iort.h        |  7 ++-
>  3 files changed, 100 insertions(+), 5 deletions(-)
> 
> diff --git a/drivers/acpi/arm64/iort.c b/drivers/acpi/arm64/iort.c
> index a3215ee..86b5a51 100644
> --- a/drivers/acpi/arm64/iort.c
> +++ b/drivers/acpi/arm64/iort.c
> @@ -39,6 +39,7 @@
>  struct iort_its_msi_chip {
>  	struct list_head	list;
>  	struct fwnode_handle	*fw_node;
> +	phys_addr_t		base_addr;
>  	u32			translation_id;
>  };
>  
> @@ -136,14 +137,16 @@ typedef acpi_status (*iort_find_node_callback)
>  static DEFINE_SPINLOCK(iort_msi_chip_lock);
>  
>  /**
> - * iort_register_domain_token() - register domain token and related ITS ID
> - * to the list from where we can get it back later on.
> + * iort_register_domain_token() - register domain token along with related
> + * ITS ID and base address to the list from where we can get it back later on.
>   * @trans_id: ITS ID.
> + * @base: ITS base address.
>   * @fw_node: Domain token.
>   *
>   * Returns: 0 on success, -ENOMEM if no memory when allocating list element
>   */
> -int iort_register_domain_token(int trans_id, struct fwnode_handle *fw_node)
> +int iort_register_domain_token(int trans_id, phys_addr_t base,
> +			       struct fwnode_handle *fw_node)
>  {
>  	struct iort_its_msi_chip *its_msi_chip;
>  
> @@ -153,6 +156,7 @@ int iort_register_domain_token(int trans_id, struct fwnode_handle *fw_node)
>  
>  	its_msi_chip->fw_node = fw_node;
>  	its_msi_chip->translation_id = trans_id;
> +	its_msi_chip->base_addr = base;
>  
>  	spin_lock(&iort_msi_chip_lock);
>  	list_add(&its_msi_chip->list, &iort_msi_chip_list);
> @@ -481,6 +485,24 @@ int iort_pmsi_get_dev_id(struct device *dev, u32 *dev_id)
>  	return -ENODEV;
>  }
>  
> +static int __maybe_unused iort_find_its_base(u32 its_id, phys_addr_t *base)
> +{
> +	struct iort_its_msi_chip *its_msi_chip;
> +	bool match = false;
> +
> +	spin_lock(&iort_msi_chip_lock);
> +	list_for_each_entry(its_msi_chip, &iort_msi_chip_list, list) {
> +		if (its_msi_chip->translation_id == its_id) {
> +			*base = its_msi_chip->base_addr;
> +			match = true;
> +			break;
> +		}
> +	}
> +	spin_unlock(&iort_msi_chip_lock);
> +
> +	return match ? 0 : -ENODEV;
> +}
> +
>  /**
>   * iort_dev_find_its_id() - Find the ITS identifier for a device
>   * @dev: The device.
> @@ -639,6 +661,71 @@ int iort_add_device_replay(const struct iommu_ops *ops, struct device *dev)
>  
>  	return err;
>  }
> +
> +/**
> + * iort_iommu_its_get_resv_regions - Reserved region driver helper
> + * @dev: Device from iommu_get_resv_regions()
> + * @list: Reserved region list from iommu_get_resv_regions()
> + *
> + * Returns: Number of reserved regions on success(0 if no associated ITS),
> + *          appropriate error value otherwise.
> + */
> +int iort_iommu_its_get_resv_regions(struct device *dev, struct list_head *head)
> +{
> +	struct acpi_iort_its_group *its;
> +	struct acpi_iort_node *node, *its_node = NULL;
> +	int i, resv = 0;
> +
> +	node = iort_find_dev_node(dev);
> +	if (!node)
> +		return -ENODEV;
> +
> +	/*
> +	 * Current logic to reserve ITS regions relies on HW topologies
> +	 * where a given PCI or named component maps its IDs to only one
> +	 * ITS group; if a PCI or named component can map its IDs to
> +	 * different ITS groups through IORT mappings this function has
> +	 * to be reworked to ensure we reserve regions for all ITS groups
> +	 * a given PCI or named component may map IDs to.
> +	 */
> +	if (dev_is_pci(dev)) {
> +		u32 rid;
> +
> +		pci_for_each_dma_alias(to_pci_dev(dev), __get_pci_rid, &rid);
> +		its_node = iort_node_map_id(node, rid, NULL, IORT_MSI_TYPE);
> +	} else {
> +		for (i = 0; i < node->mapping_count; i++) {
> +			its_node = iort_node_map_platform_id(node, NULL,
> +							 IORT_MSI_TYPE, i);
> +			if (its_node)
> +				break;
> +		}
> +	}
> +
> +	if (!its_node)
> +		return 0;
> +
> +	/* Move to ITS specific data */
> +	its = (struct acpi_iort_its_group *)its_node->node_data;
> +
> +	for (i = 0; i < its->its_count; i++) {
> +		phys_addr_t base;
> +
> +		if (!iort_find_its_base(its->identifiers[i], &base)) {
> +			int prot = IOMMU_WRITE | IOMMU_NOEXEC | IOMMU_MMIO;
> +			struct iommu_resv_region *region;
> +
> +			region = iommu_alloc_resv_region(base, SZ_128K, prot,
> +							 IOMMU_RESV_MSI);
> +			if (region) {
> +				list_add_tail(&region->list, head);
> +				resv++;
> +			}
> +		}
> +	}
> +
> +	return (resv == its->its_count) ? resv : -ENODEV;
> +}
>  #else
>  static inline
>  const struct iommu_ops *iort_fwspec_iommu_ops(struct iommu_fwspec *fwspec)
> @@ -646,6 +733,8 @@ const struct iommu_ops *iort_fwspec_iommu_ops(struct iommu_fwspec *fwspec)
>  static inline
>  int iort_add_device_replay(const struct iommu_ops *ops, struct device *dev)
>  { return 0; }
> +int iort_iommu_its_get_resv_regions(struct device *dev, struct list_head *head)
> +{ return -ENODEV; }
>  #endif
>  
>  static const struct iommu_ops *iort_iommu_xlate(struct device *dev,
> diff --git a/drivers/irqchip/irq-gic-v3-its.c b/drivers/irqchip/irq-gic-v3-its.c
> index 6893287..77322b3 100644
> --- a/drivers/irqchip/irq-gic-v3-its.c
> +++ b/drivers/irqchip/irq-gic-v3-its.c
> @@ -1928,7 +1928,8 @@ static int __init gic_acpi_parse_madt_its(struct acpi_subtable_header *header,
>  		return -ENOMEM;
>  	}
>  
> -	err = iort_register_domain_token(its_entry->translation_id, dom_handle);
> +	err = iort_register_domain_token(its_entry->translation_id, res.start,
> +					 dom_handle);
>  	if (err) {
>  		pr_err("ITS@%pa: Unable to register GICv3 ITS domain token (ITS ID %d) to IORT\n",
>  		       &res.start, its_entry->translation_id);
> diff --git a/include/linux/acpi_iort.h b/include/linux/acpi_iort.h
> index 8379d40..d7ed49c 100644
> --- a/include/linux/acpi_iort.h
> +++ b/include/linux/acpi_iort.h
> @@ -26,7 +26,8 @@
>  #define IORT_IRQ_MASK(irq)		(irq & 0xffffffffULL)
>  #define IORT_IRQ_TRIGGER_MASK(irq)	((irq >> 32) & 0xffffffffULL)
>  
> -int iort_register_domain_token(int trans_id, struct fwnode_handle *fw_node);
> +int iort_register_domain_token(int trans_id, phys_addr_t base,
> +			       struct fwnode_handle *fw_node);
>  void iort_deregister_domain_token(int trans_id);
>  struct fwnode_handle *iort_find_domain_token(int trans_id);
>  #ifdef CONFIG_ACPI_IORT
> @@ -38,6 +39,7 @@
>  /* IOMMU interface */
>  void iort_set_dma_mask(struct device *dev);
>  const struct iommu_ops *iort_iommu_configure(struct device *dev);
> +int iort_iommu_its_get_resv_regions(struct device *dev, struct list_head *head);
>  #else
>  static inline void acpi_iort_init(void) { }
>  static inline u32 iort_msi_map_rid(struct device *dev, u32 req_id)
> @@ -51,6 +53,9 @@ static inline void iort_set_dma_mask(struct device *dev) { }
>  static inline
>  const struct iommu_ops *iort_iommu_configure(struct device *dev)
>  { return NULL; }
> +static inline
> +int iort_iommu_its_get_resv_regions(struct device *dev, struct list_head *head)
> +{ return -ENODEV; }
>  #endif
>  
>  #endif /* __ACPI_IORT_H__ */
> 

RE: [PATCH v5 1/2] ACPI/IORT: Add ITS address regions reservation helper

[Shameerali Kolothum Thodi]

> -----Original Message-----
> From: Robin Murphy [mailto:robin.murphy@arm.com]
> Sent: Friday, August 04, 2017 5:57 PM
> To: Shameerali Kolothum Thodi; lorenzo.pieralisi@arm.com;
> marc.zyngier@arm.com; sudeep.holla@arm.com; will.deacon@arm.com;
> hanjun.guo@linaro.org
> Cc: Gabriele Paoloni; John Garry; iommu@lists.linux-foundation.org; linux-
> arm-kernel@lists.infradead.org; linux-acpi@vger.kernel.org;
> devel@acpica.org; Linuxarm; Wangzhou (B); Guohanjun (Hanjun Guo)
> Subject: Re: [PATCH v5 1/2] ACPI/IORT: Add ITS address regions reservation
> helper
> 
> On 01/08/17 11:49, Shameer Kolothum wrote:
> > On some platforms ITS address regions have to be excluded from normal
> > IOVA allocation in that they are detected and decoded in a HW specific
> > way by system components and so they cannot be considered normal
> IOVA
> > address space.
> >
> > Add an helper function that retrieves ITS address regions through IORT
> > device <-> ITS mappings and reserves it so that these regions will not
> > be translated by IOMMU and will be excluded from IOVA allocations.
> 
> I've just realised that we no longer seem to have a check that ensures
> the regions are only reserved on platforms that need it - if not, then
> we're going to break everything else that does have an ITS behind SMMU
> translation as expected.

Right. I had this doubt, but then my thinking was that we will have the SW_MSI
regions for those and will end up  using that. But that doesn’t seems
to be the case now.

> It feels like IORT should know enough to be able to make that decision
> internally, but if not (or if it would be hideous to do so), then I
> guess my idea for patch #2 was a bust and we probably do need to go back
> to calling directly from the SMMU driver based on the SMMU model.

It might be possible to do that check inside iort code, but then we have to find
the  smmu node first and check the model number. I think it will be more
cleaner if SMMU driver makes that check and call the
iommu_dma_get_msi_resv_regions().

If you are Ok with that I will quickly rework and send out the next version.

Thanks,
Shameer

_______________________________________________
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu

Re: [PATCH v5 1/2] ACPI/IORT: Add ITS address regions reservation helper

[Lorenzo Pieralisi]

On Mon, Aug 07, 2017 at 08:21:40AM +0000, Shameerali Kolothum Thodi wrote:
> 
> 
> > -----Original Message-----
> > From: Robin Murphy [mailto:robin.murphy@arm.com]
> > Sent: Friday, August 04, 2017 5:57 PM
> > To: Shameerali Kolothum Thodi; lorenzo.pieralisi@arm.com;
> > marc.zyngier@arm.com; sudeep.holla@arm.com; will.deacon@arm.com;
> > hanjun.guo@linaro.org
> > Cc: Gabriele Paoloni; John Garry; iommu@lists.linux-foundation.org; linux-
> > arm-kernel@lists.infradead.org; linux-acpi@vger.kernel.org;
> > devel@acpica.org; Linuxarm; Wangzhou (B); Guohanjun (Hanjun Guo)
> > Subject: Re: [PATCH v5 1/2] ACPI/IORT: Add ITS address regions reservation
> > helper
> > 
> > On 01/08/17 11:49, Shameer Kolothum wrote:
> > > On some platforms ITS address regions have to be excluded from normal
> > > IOVA allocation in that they are detected and decoded in a HW specific
> > > way by system components and so they cannot be considered normal
> > IOVA
> > > address space.
> > >
> > > Add an helper function that retrieves ITS address regions through IORT
> > > device <-> ITS mappings and reserves it so that these regions will not
> > > be translated by IOMMU and will be excluded from IOVA allocations.
> > 
> > I've just realised that we no longer seem to have a check that ensures
> > the regions are only reserved on platforms that need it - if not, then
> > we're going to break everything else that does have an ITS behind SMMU
> > translation as expected.
> 
> Right. I had this doubt, but then my thinking was that we will have the SW_MSI
> regions for those and will end up  using that. But that doesn’t seems
> to be the case now.
> 
> > It feels like IORT should know enough to be able to make that decision
> > internally, but if not (or if it would be hideous to do so), then I
> > guess my idea for patch #2 was a bust and we probably do need to go back
> > to calling directly from the SMMU driver based on the SMMU model.
> 
> It might be possible to do that check inside iort code, but then we have to find
> the  smmu node first and check the model number. I think it will be more
> cleaner if SMMU driver makes that check and call the
> iommu_dma_get_msi_resv_regions().

+1 on this one - we can do it in IORT but I think it is more logical
to have a flag in the SMMU driver (keeping the DT/ACPI fwnode switch in
generic IOMMU layer though).

Side note I: AFAICS iommu_dma_get_resv_regions() is only used on ARM, if
it is not patch 2 would break miserably on arches that do not select
IORT - you should rework the return code on !CONFIG_ACPI_IORT configs.

Side note II(nit): why not call the function iort_iommu_get_resv_regions() ?

Lorenzo

> If you are Ok with that I will quickly rework and send out the next version.
> 
> Thanks,
> Shameer
> 
> _______________________________________________
> linux-arm-kernel mailing list
> linux-arm-kernel@lists.infradead.org
> http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

RE: [PATCH v5 1/2] ACPI/IORT: Add ITS address regions reservation helper

[Shameerali Kolothum Thodi]

> -----Original Message-----
> From: Lorenzo Pieralisi [mailto:lorenzo.pieralisi@arm.com]
> Sent: Monday, August 07, 2017 6:09 PM
> To: Shameerali Kolothum Thodi
> Cc: Robin Murphy; marc.zyngier@arm.com; sudeep.holla@arm.com;
> will.deacon@arm.com; hanjun.guo@linaro.org; Gabriele Paoloni; John Garry;
> Linuxarm; linux-acpi@vger.kernel.org; iommu@lists.linux-foundation.org;
> Wangzhou (B); Guohanjun (Hanjun Guo); linux-arm-
> kernel@lists.infradead.org; devel@acpica.org
> Subject: Re: [PATCH v5 1/2] ACPI/IORT: Add ITS address regions reservation
> helper
> 
> On Mon, Aug 07, 2017 at 08:21:40AM +0000, Shameerali Kolothum Thodi
> wrote:
> >
> >
> > > -----Original Message-----
> > > From: Robin Murphy [mailto:robin.murphy@arm.com]
> > > Sent: Friday, August 04, 2017 5:57 PM
> > > To: Shameerali Kolothum Thodi; lorenzo.pieralisi@arm.com;
> > > marc.zyngier@arm.com; sudeep.holla@arm.com; will.deacon@arm.com;
> > > hanjun.guo@linaro.org
> > > Cc: Gabriele Paoloni; John Garry; iommu@lists.linux-foundation.org;
> linux-
> > > arm-kernel@lists.infradead.org; linux-acpi@vger.kernel.org;
> > > devel@acpica.org; Linuxarm; Wangzhou (B); Guohanjun (Hanjun Guo)
> > > Subject: Re: [PATCH v5 1/2] ACPI/IORT: Add ITS address regions
> reservation
> > > helper
> > >
> > > On 01/08/17 11:49, Shameer Kolothum wrote:
> > > > On some platforms ITS address regions have to be excluded from
> normal
> > > > IOVA allocation in that they are detected and decoded in a HW specific
> > > > way by system components and so they cannot be considered normal
> > > IOVA
> > > > address space.
> > > >
> > > > Add an helper function that retrieves ITS address regions through IORT
> > > > device <-> ITS mappings and reserves it so that these regions will not
> > > > be translated by IOMMU and will be excluded from IOVA allocations.
> > >
> > > I've just realised that we no longer seem to have a check that ensures
> > > the regions are only reserved on platforms that need it - if not, then
> > > we're going to break everything else that does have an ITS behind SMMU
> > > translation as expected.
> >
> > Right. I had this doubt, but then my thinking was that we will have the
> SW_MSI
> > regions for those and will end up  using that. But that doesn’t seems
> > to be the case now.
> >
> > > It feels like IORT should know enough to be able to make that decision
> > > internally, but if not (or if it would be hideous to do so), then I
> > > guess my idea for patch #2 was a bust and we probably do need to go
> back
> > > to calling directly from the SMMU driver based on the SMMU model.
> >
> > It might be possible to do that check inside iort code, but then we have to
> find
> > the  smmu node first and check the model number. I think it will be more
> > cleaner if SMMU driver makes that check and call the
> > iommu_dma_get_msi_resv_regions().
> 
> +1 on this one - we can do it in IORT but I think it is more logical
> to have a flag in the SMMU driver (keeping the DT/ACPI fwnode switch in
> generic IOMMU layer though).

Please find [1] for the generic iommu helper function which will be called
from SMMU driver based on the model.

> Side note I: AFAICS iommu_dma_get_resv_regions() is only used on ARM, if
> it is not patch 2 would break miserably on arches that do not select
> IORT - you should rework the return code on !CONFIG_ACPI_IORT configs.

Yes. But so far it is only used on ARM. In any way this function is not going to be
changed and will introduce a new iommu_dma_get_msi_resv_regions() fn  as
proposed in [1].

Please take a look and let me know. I will submit the series again if this is fine.

Thanks,
Shameer

[1] This will replace patch 2 and will be followed by smmu driver patch to invoke
iommu_dma_get_msi_resv_regions() based on SMMU model.

-->8--
Subject: [PATCH] iommu/dma: Add a helper  function to reserve HW MSI address regions for IOMMU drivers

IOMMU drivers can use this to implement their .get_resv_regions callback
for HW MSI specific reservations(e.g. ARM GICv3 ITS MSI region).

Signed-off-by: Shameer Kolothum <shameerali.kolothum.thodi@huawei.com>
---
 drivers/iommu/dma-iommu.c | 19 +++++++++++++++++++
 include/linux/dma-iommu.h |  7 +++++++
 2 files changed, 26 insertions(+)

diff --git a/drivers/iommu/dma-iommu.c b/drivers/iommu/dma-iommu.c
index 9d1cebe..952ecdd 100644
--- a/drivers/iommu/dma-iommu.c
+++ b/drivers/iommu/dma-iommu.c
@@ -19,6 +19,7 @@
  * along with this program.  If not, see <http://www.gnu.org/licenses/>.
  */
 
+#include <linux/acpi_iort.h>
 #include <linux/device.h>
 #include <linux/dma-iommu.h>
 #include <linux/gfp.h>
@@ -198,6 +199,24 @@ void iommu_dma_get_resv_regions(struct device *dev, struct list_head *list)
 }
 EXPORT_SYMBOL(iommu_dma_get_resv_regions);
 
+/**
+ * iommu_dma_get_msi_resv_regions - Reserved region driver helper
+ * @dev: Device from iommu_get_resv_regions()
+ * @list: Reserved region list from iommu_get_resv_regions()
+ *
+ * IOMMU drivers can use this to implement their .get_resv_regions
+ * callback for HW MSI specific reservations. For now, this only
+ * covers ITS MSI region reservation using ACPI IORT helper function.
+ */
+int iommu_dma_get_msi_resv_regions(struct device *dev, struct list_head *list)
+{
+	if (!is_of_node(dev->iommu_fwspec->iommu_fwnode))
+		return iort_iommu_its_get_resv_regions(dev, list);
+
+	return -ENODEV;
+}
+EXPORT_SYMBOL(iommu_dma_get_msi_resv_regions);
+
 static int cookie_init_hw_msi_region(struct iommu_dma_cookie *cookie,
 		phys_addr_t start, phys_addr_t end)
 {
diff --git a/include/linux/dma-iommu.h b/include/linux/dma-iommu.h
index 92f2083..6062ef0 100644
--- a/include/linux/dma-iommu.h
+++ b/include/linux/dma-iommu.h
@@ -74,6 +74,8 @@ void iommu_dma_unmap_resource(struct device *dev, dma_addr_t handle,
 void iommu_dma_map_msi_msg(int irq, struct msi_msg *msg);
 void iommu_dma_get_resv_regions(struct device *dev, struct list_head *list);
 
+int iommu_dma_get_msi_resv_regions(struct device *dev, struct list_head *list);
+
 #else
 
 struct iommu_domain;
@@ -107,6 +109,11 @@ static inline void iommu_dma_get_resv_regions(struct device *dev, struct list_he
 {
 }
 
+static inline int iommu_dma_get_msi_resv_regions(struct device *dev, struct list_head *list)
+{
+	return -ENODEV;
+}
+
 #endif	/* CONFIG_IOMMU_DMA */
 #endif	/* __KERNEL__ */
 #endif	/* __DMA_IOMMU_H */
-- 
1.9.1

[PATCH v5 2/2] iommu/dma: Add HW MSI address regions reservation

[Shameer Kolothum]

Modified iommu_dma_get_resv_regions() to include HW MSI
(ARM GICv3 ITS MSI) specific reservations if available.

Suggested-by: Robin Murphy <robin.murphy-5wv7dgnIgG8@public.gmane.org>
Signed-off-by: Shameer Kolothum <shameerali.kolothum.thodi-hv44wF8Li93QT0dZR+AlfA@public.gmane.org>
---
 drivers/iommu/dma-iommu.c | 8 +++++++-
 1 file changed, 7 insertions(+), 1 deletion(-)

diff --git a/drivers/iommu/dma-iommu.c b/drivers/iommu/dma-iommu.c
index 9d1cebe..44eca1e 100644
--- a/drivers/iommu/dma-iommu.c
+++ b/drivers/iommu/dma-iommu.c
@@ -19,6 +19,7 @@
  * along with this program.  If not, see <http://www.gnu.org/licenses/>.
  */
 
+#include <linux/acpi_iort.h>
 #include <linux/device.h>
 #include <linux/dma-iommu.h>
 #include <linux/gfp.h>
@@ -167,13 +168,18 @@ void iommu_put_dma_cookie(struct iommu_domain *domain)
  *
  * IOMMU drivers can use this to implement their .get_resv_regions callback
  * for general non-IOMMU-specific reservations. Currently, this covers host
- * bridge windows for PCI devices.
+ * bridge windows for PCI devices and HW MSI(ARM GICv3 ITS MSI) region
+ * reservations if available.
  */
 void iommu_dma_get_resv_regions(struct device *dev, struct list_head *list)
 {
 	struct pci_host_bridge *bridge;
 	struct resource_entry *window;
 
+	if (!is_of_node(dev->iommu_fwspec->iommu_fwnode) &&
+		iort_iommu_its_get_resv_regions(dev, list) < 0)
+		return;
+
 	if (!dev_is_pci(dev))
 		return;
 
-- 
1.9.1