[PATCH 1/1] iommu/vt-d: Optimize iotlb_sync_map for non-caching/non-RWBF modes

[PATCH 1/1] iommu/vt-d: Optimize iotlb_sync_map for non-caching/non-RWBF modes

[Lu Baolu]

The iotlb_sync_map iommu ops allows drivers to perform necessary cache
flushes when new mappings are established. For the Intel iommu driver,
this callback specifically serves two purposes:

- To flush caches when a second-stage page table is attached to a device
  whose iommu is operating in caching mode (CAP_REG.CM==1).
- To explicitly flush internal write buffers to ensure updates to memory-
  resident remapping structures are visible to hardware (CAP_REG.RWBF==1).

However, in scenarios where neither caching mode nor the RWBF flag is
active, the cache_tag_flush_range_np() helper, which is called in the
iotlb_sync_map path, effectively becomes a no-op.

Despite being a no-op, cache_tag_flush_range_np() involves iterating
through all cache tags of the iommu's attached to the domain, protected
by a spinlock. This unnecessary execution path introduces overhead,
leading to a measurable I/O performance regression. On systems with NVMes
under the same bridge, performance was observed to drop from approximately
~6150 MiB/s down to ~4985 MiB/s.

Introduce a flag in the dmar_domain structure. This flag will only be set
when iotlb_sync_map is required (i.e., when CM or RWBF is set). The
cache_tag_flush_range_np() is called only for domains where this flag is
set.

Reported-by: Ioanna Alifieraki <ioanna-maria.alifieraki@canonical.com>
Closes: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2115738
Link: https://lore.kernel.org/r/20250701171154.52435-1-ioanna-maria.alifieraki@canonical.com
Fixes: 129dab6e1286 ("iommu/vt-d: Use cache_tag_flush_range_np() in iotlb_sync_map")
Cc: stable@vger.kernel.org
Signed-off-by: Lu Baolu <baolu.lu@linux.intel.com>
---
 drivers/iommu/intel/iommu.c | 19 ++++++++++++++++++-
 drivers/iommu/intel/iommu.h |  3 +++
 2 files changed, 21 insertions(+), 1 deletion(-)

diff --git a/drivers/iommu/intel/iommu.c b/drivers/iommu/intel/iommu.c
index 7aa3932251b2..f60201ee4be0 100644
--- a/drivers/iommu/intel/iommu.c
+++ b/drivers/iommu/intel/iommu.c
@@ -1796,6 +1796,18 @@ static int domain_setup_first_level(struct intel_iommu *iommu,
 					  (pgd_t *)pgd, flags, old);
 }
 
+static bool domain_need_iotlb_sync_map(struct dmar_domain *domain,
+				       struct intel_iommu *iommu)
+{
+	if (cap_caching_mode(iommu->cap) && !domain->use_first_level)
+		return true;
+
+	if (rwbf_quirk || cap_rwbf(iommu->cap))
+		return true;
+
+	return false;
+}
+
 static int dmar_domain_attach_device(struct dmar_domain *domain,
 				     struct device *dev)
 {
@@ -1833,6 +1845,8 @@ static int dmar_domain_attach_device(struct dmar_domain *domain,
 	if (ret)
 		goto out_block_translation;
 
+	domain->iotlb_sync_map |= domain_need_iotlb_sync_map(domain, iommu);
+
 	return 0;
 
 out_block_translation:
@@ -3945,7 +3959,10 @@ static bool risky_device(struct pci_dev *pdev)
 static int intel_iommu_iotlb_sync_map(struct iommu_domain *domain,
 				      unsigned long iova, size_t size)
 {
-	cache_tag_flush_range_np(to_dmar_domain(domain), iova, iova + size - 1);
+	struct dmar_domain *dmar_domain = to_dmar_domain(domain);
+
+	if (dmar_domain->iotlb_sync_map)
+		cache_tag_flush_range_np(dmar_domain, iova, iova + size - 1);
 
 	return 0;
 }
diff --git a/drivers/iommu/intel/iommu.h b/drivers/iommu/intel/iommu.h
index 3ddbcc603de2..7ab2c34a5ecc 100644
--- a/drivers/iommu/intel/iommu.h
+++ b/drivers/iommu/intel/iommu.h
@@ -614,6 +614,9 @@ struct dmar_domain {
 	u8 has_mappings:1;		/* Has mappings configured through
 					 * iommu_map() interface.
 					 */
+	u8 iotlb_sync_map:1;		/* Need to flush IOTLB cache or write
+					 * buffer when creating mappings.
+					 */
 
 	spinlock_t lock;		/* Protect device tracking lists */
 	struct list_head devices;	/* all devices' list */
-- 
2.43.0

RE: [PATCH 1/1] iommu/vt-d: Optimize iotlb_sync_map for non-caching/non-RWBF modes

[Tian, Kevin]

> From: Lu Baolu <baolu.lu@linux.intel.com>
> Sent: Thursday, July 3, 2025 11:16 AM
> 
> The iotlb_sync_map iommu ops allows drivers to perform necessary cache
> flushes when new mappings are established. For the Intel iommu driver,
> this callback specifically serves two purposes:
> 
> - To flush caches when a second-stage page table is attached to a device
>   whose iommu is operating in caching mode (CAP_REG.CM==1).
> - To explicitly flush internal write buffers to ensure updates to memory-
>   resident remapping structures are visible to hardware (CAP_REG.RWBF==1).
> 
> However, in scenarios where neither caching mode nor the RWBF flag is
> active, the cache_tag_flush_range_np() helper, which is called in the
> iotlb_sync_map path, effectively becomes a no-op.
> 
> Despite being a no-op, cache_tag_flush_range_np() involves iterating
> through all cache tags of the iommu's attached to the domain, protected
> by a spinlock. This unnecessary execution path introduces overhead,
> leading to a measurable I/O performance regression. On systems with
> NVMes
> under the same bridge, performance was observed to drop from
> approximately
> ~6150 MiB/s down to ~4985 MiB/s.

so for the same bridge case two NVMe disks likely are in the same 
iommu group sharing a domain. Then there is contention on the
spinlock from two parallel threads on two disks. when disks come
from different bridges they are attached to different domains hence
no contention.

is it a correct description for the difference between same vs.
different bridge?

> @@ -1833,6 +1845,8 @@ static int dmar_domain_attach_device(struct
> dmar_domain *domain,
>  	if (ret)
>  		goto out_block_translation;
> 
> +	domain->iotlb_sync_map |= domain_need_iotlb_sync_map(domain,
> iommu);
> +
>  	return 0;
> 

also need to update the flag upon detach.

with it:

Reviewed-by: Kevin Tian <kevin.tian@intel.com>

Re: [PATCH 1/1] iommu/vt-d: Optimize iotlb_sync_map for non-caching/non-RWBF modes

[Baolu Lu]

On 7/3/25 15:16, Tian, Kevin wrote:
>> From: Lu Baolu<baolu.lu@linux.intel.com>
>> Sent: Thursday, July 3, 2025 11:16 AM
>>
>> The iotlb_sync_map iommu ops allows drivers to perform necessary cache
>> flushes when new mappings are established. For the Intel iommu driver,
>> this callback specifically serves two purposes:
>>
>> - To flush caches when a second-stage page table is attached to a device
>>    whose iommu is operating in caching mode (CAP_REG.CM==1).
>> - To explicitly flush internal write buffers to ensure updates to memory-
>>    resident remapping structures are visible to hardware (CAP_REG.RWBF==1).
>>
>> However, in scenarios where neither caching mode nor the RWBF flag is
>> active, the cache_tag_flush_range_np() helper, which is called in the
>> iotlb_sync_map path, effectively becomes a no-op.
>>
>> Despite being a no-op, cache_tag_flush_range_np() involves iterating
>> through all cache tags of the iommu's attached to the domain, protected
>> by a spinlock. This unnecessary execution path introduces overhead,
>> leading to a measurable I/O performance regression. On systems with
>> NVMes
>> under the same bridge, performance was observed to drop from
>> approximately
>> ~6150 MiB/s down to ~4985 MiB/s.
> so for the same bridge case two NVMe disks likely are in the same
> iommu group sharing a domain. Then there is contention on the
> spinlock from two parallel threads on two disks. when disks come
> from different bridges they are attached to different domains hence
> no contention.
> 
> is it a correct description for the difference between same vs.
> different bridge?

Yes. I have the same understanding.

Thanks,
baolu

Re: [PATCH 1/1] iommu/vt-d: Optimize iotlb_sync_map for non-caching/non-RWBF modes

[Baolu Lu]

On 7/3/25 11:15, Lu Baolu wrote:
> The iotlb_sync_map iommu ops allows drivers to perform necessary cache
> flushes when new mappings are established. For the Intel iommu driver,
> this callback specifically serves two purposes:
> 
> - To flush caches when a second-stage page table is attached to a device
>    whose iommu is operating in caching mode (CAP_REG.CM==1).
> - To explicitly flush internal write buffers to ensure updates to memory-
>    resident remapping structures are visible to hardware (CAP_REG.RWBF==1).
> 
> However, in scenarios where neither caching mode nor the RWBF flag is
> active, the cache_tag_flush_range_np() helper, which is called in the
> iotlb_sync_map path, effectively becomes a no-op.
> 
> Despite being a no-op, cache_tag_flush_range_np() involves iterating
> through all cache tags of the iommu's attached to the domain, protected
> by a spinlock. This unnecessary execution path introduces overhead,
> leading to a measurable I/O performance regression. On systems with NVMes
> under the same bridge, performance was observed to drop from approximately
> ~6150 MiB/s down to ~4985 MiB/s.
> 
> Introduce a flag in the dmar_domain structure. This flag will only be set
> when iotlb_sync_map is required (i.e., when CM or RWBF is set). The
> cache_tag_flush_range_np() is called only for domains where this flag is
> set.
> 
> Reported-by: Ioanna Alifieraki<ioanna-maria.alifieraki@canonical.com>
> Closes:https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2115738
> Link:https://lore.kernel.org/r/20250701171154.52435-1-ioanna- 
> maria.alifieraki@canonical.com
> Fixes: 129dab6e1286 ("iommu/vt-d: Use cache_tag_flush_range_np() in iotlb_sync_map")
> Cc:stable@vger.kernel.org
> Signed-off-by: Lu Baolu<baolu.lu@linux.intel.com>
> ---
>   drivers/iommu/intel/iommu.c | 19 ++++++++++++++++++-
>   drivers/iommu/intel/iommu.h |  3 +++
>   2 files changed, 21 insertions(+), 1 deletion(-)

Queued for linux-next.

--
baolu