[PATCH 2/2] iommu/vt-d: Serialize SVA unbind teardown with intel_mm_release()

[Lu Baolu <baolu.lu@linux.intel.com>]

intel_mm_release() can run concurrently with an unbind of the same PASID.
The unbind clears the Present bit, drops iommu->lock and then invalidates
the PASID cache, IOTLB and device TLB:

  iommu_sva_unbind_device()              exit_mmap()
    blocking_domain_set_dev_pasid()        intel_mm_release()
      intel_pasid_tear_down_entry()
        pasid_clear_present()
        spin_unlock(&iommu->lock)
                                             intel_pasid_tear_down_entry()
                                               !present && !FPD -> return
                                           ... free page tables ...
        ... invalidate caches ...

intel_mm_release() then returns before the invalidations complete, and
exit_mmap() can free the page tables while the IOMMU and the device still
hold translations that reference them.

intel_mm_release() walks domain->dev_pasids and tears down each entry
under domain->lock. Do the same in the blocking attach for an SVA domain:
tear down the entry with fault_ignore under domain->lock, so that either
intel_mm_release() completes the invalidations first, or it waits for the
unbind to complete them.

Fixes: 4f0bdab175d6 ("iommu/vt-d: Make the blocked domain support PASID")
Signed-off-by: Lu Baolu <baolu.lu@linux.intel.com>
---
 drivers/iommu/intel/iommu.c | 10 ++++++++++
 drivers/iommu/intel/pasid.c |  2 +-
 2 files changed, 11 insertions(+), 1 deletion(-)

diff --git a/drivers/iommu/intel/iommu.c b/drivers/iommu/intel/iommu.c
index 5a2e705a2f2e..02d2c97da86b 100644
--- a/drivers/iommu/intel/iommu.c
+++ b/drivers/iommu/intel/iommu.c
@@ -3699,6 +3699,16 @@ static int blocking_domain_set_dev_pasid(struct iommu_domain *domain,
 {
 	struct device_domain_info *info = dev_iommu_priv_get(dev);
 
+	if (old && old->type == IOMMU_DOMAIN_SVA) {
+		struct dmar_domain *dmar_domain = to_dmar_domain(old);
+		unsigned long flags;
+
+		/* intel_mm_release() must not return before invalidation. */
+		spin_lock_irqsave(&dmar_domain->lock, flags);
+		intel_pasid_tear_down_entry(info->iommu, dev, pasid, true);
+		spin_unlock_irqrestore(&dmar_domain->lock, flags);
+	}
+
 	intel_pasid_tear_down_entry(info->iommu, dev, pasid, false);
 	iopf_for_domain_remove(old, dev);
 	domain_remove_dev_pasid(old, dev, pasid);
diff --git a/drivers/iommu/intel/pasid.c b/drivers/iommu/intel/pasid.c
index d1c08d180788..00f9b0722eb5 100644
--- a/drivers/iommu/intel/pasid.c
+++ b/drivers/iommu/intel/pasid.c
@@ -237,7 +237,7 @@ void intel_pasid_tear_down_entry(struct intel_iommu *iommu, struct device *dev,
 	}
 
 	if (!pasid_pte_is_present(pte)) {
-		if (!pasid_pte_is_fault_disabled(pte)) {
+		if (!pasid_pte_is_fault_disabled(pte) || fault_ignore) {
 			spin_unlock(&iommu->lock);
 			return;
 		}
-- 
2.43.0