[PATCH rc] iommupt: Only cache flush memory changed by unmap

[PATCH rc] iommupt: Only cache flush memory changed by unmap

[Jason Gunthorpe]

The cache flush was happening on every level across the whole range of
iteration, even if no leafs or tables were cleared. Instead flush only the
sub range that was actually written.

Overflushing isn't a correctness problem but it does impact the
performance of unmap.

After this series the performance compared to the original VT-d
implementation with cache flushing turned on is:

map_pages
   pgsz  ,avg new,old ns, min new,old ns  , min % (+ve is better)
     2^12,    253,266   ,     213,227     ,   6.06
     2^21,    246,244   ,     221,219     ,   0.00
     2^30,    231,240   ,     209,217     ,   3.03
 256*2^12,   2604,2668  ,    2415,2540    ,   4.04
 256*2^21,   2495,2824  ,    2390,2734    ,  12.12
 256*2^30,   2542,2845  ,    2380,2718    ,  12.12

unmap_pages
   pgsz  ,avg new,old ns, min new,old ns  , min % (+ve is better)
     2^12,    259,292   ,     222,251     ,  11.11
     2^21,    255,259   ,     227,236     ,   3.03
     2^30,    238,254   ,     217,230     ,   5.05
 256*2^12,   2751,2620  ,    2417,2437    ,   0.00
 256*2^21,   2461,2526  ,    2377,2423    ,   1.01
 256*2^30,   2498,2543  ,    2370,2404    ,   1.01

Fixes: efa03dab7ce4 ("iommupt: Flush the CPU cache after any writes to the page table")
Reported-by: Francois Dugast <francois.dugast@intel.com>
Closes: https://lore.kernel.org/all/20260121130233.257428-1-francois.dugast@intel.com/
Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
---
 drivers/iommu/generic_pt/iommu_pt.h | 11 ++++++++++-
 1 file changed, 10 insertions(+), 1 deletion(-)

diff --git a/drivers/iommu/generic_pt/iommu_pt.h b/drivers/iommu/generic_pt/iommu_pt.h
index 3327116a441cac..687bb4e3d8ca5a 100644
--- a/drivers/iommu/generic_pt/iommu_pt.h
+++ b/drivers/iommu/generic_pt/iommu_pt.h
@@ -931,6 +931,8 @@ static __maybe_unused int __unmap_range(struct pt_range *range, void *arg,
 					struct pt_table_p *table)
 {
 	struct pt_state pts = pt_init(range, level, table);
+	unsigned int flush_start_index = UINT_MAX;
+	unsigned int flush_end_index = UINT_MAX;
 	struct pt_unmap_args *unmap = arg;
 	unsigned int num_oas = 0;
 	unsigned int start_index;
@@ -986,6 +988,9 @@ static __maybe_unused int __unmap_range(struct pt_range *range, void *arg,
 				iommu_pages_list_add(&unmap->free_list,
 						     pts.table_lower);
 				pt_clear_entries(&pts, ilog2(1));
+				if (pts.index < flush_start_index)
+					flush_start_index = pts.index;
+				flush_end_index = pts.index + 1;
 			}
 			pts.index++;
 		} else {
@@ -999,7 +1004,10 @@ static __maybe_unused int __unmap_range(struct pt_range *range, void *arg,
 			num_contig_lg2 = pt_entry_num_contig_lg2(&pts);
 			pt_clear_entries(&pts, num_contig_lg2);
 			num_oas += log2_to_int(num_contig_lg2);
+			if (pts.index < flush_start_index)
+				flush_start_index = pts.index;
 			pts.index += log2_to_int(num_contig_lg2);
+			flush_end_index = pts.index;
 		}
 		if (pts.index >= pts.end_index)
 			break;
@@ -1007,7 +1015,8 @@ static __maybe_unused int __unmap_range(struct pt_range *range, void *arg,
 	} while (true);
 
 	unmap->unmapped += log2_mul(num_oas, pt_table_item_lg2sz(&pts));
-	flush_writes_range(&pts, start_index, pts.index);
+	if (flush_start_index != flush_end_index)
+		flush_writes_range(&pts, flush_start_index, flush_end_index);
 
 	return ret;
 }

base-commit: 24d479d26b25bce5faea3ddd9fa8f3a6c3129ea7
-- 
2.43.0

RE: [PATCH rc] iommupt: Only cache flush memory changed by unmap

[Tian, Kevin]

> From: Jason Gunthorpe <jgg@nvidia.com>
> Sent: Sunday, January 25, 2026 5:00 AM
> 
> The cache flush was happening on every level across the whole range of
> iteration, even if no leafs or tables were cleared. Instead flush only the
> sub range that was actually written.
> 
> Overflushing isn't a correctness problem but it does impact the
> performance of unmap.
> 
> After this series the performance compared to the original VT-d
> implementation with cache flushing turned on is:
> 
> map_pages
>    pgsz  ,avg new,old ns, min new,old ns  , min % (+ve is better)
>      2^12,    253,266   ,     213,227     ,   6.06
>      2^21,    246,244   ,     221,219     ,   0.00
>      2^30,    231,240   ,     209,217     ,   3.03
>  256*2^12,   2604,2668  ,    2415,2540    ,   4.04
>  256*2^21,   2495,2824  ,    2390,2734    ,  12.12
>  256*2^30,   2542,2845  ,    2380,2718    ,  12.12
> 
> unmap_pages
>    pgsz  ,avg new,old ns, min new,old ns  , min % (+ve is better)
>      2^12,    259,292   ,     222,251     ,  11.11
>      2^21,    255,259   ,     227,236     ,   3.03
>      2^30,    238,254   ,     217,230     ,   5.05
>  256*2^12,   2751,2620  ,    2417,2437    ,   0.00
>  256*2^21,   2461,2526  ,    2377,2423    ,   1.01
>  256*2^30,   2498,2543  ,    2370,2404    ,   1.01
> 

that is great!

Out of curiosity - why would an optimization in the unmap path also
lead to better efficiency in the map path (in some cases map improves
even more than unmap, e.g. with 256 pages)?

At a glance I didn't see how __unmap_range is relevant to map...

Re: [PATCH rc] iommupt: Only cache flush memory changed by unmap

[Jason Gunthorpe]

On Mon, Jan 26, 2026 at 07:33:21AM +0000, Tian, Kevin wrote:
> > From: Jason Gunthorpe <jgg@nvidia.com>
> > Sent: Sunday, January 25, 2026 5:00 AM
> > 
> > The cache flush was happening on every level across the whole range of
> > iteration, even if no leafs or tables were cleared. Instead flush only the
> > sub range that was actually written.
> > 
> > Overflushing isn't a correctness problem but it does impact the
> > performance of unmap.
> > 
> > After this series the performance compared to the original VT-d
> > implementation with cache flushing turned on is:
> > 
> > map_pages
> >    pgsz  ,avg new,old ns, min new,old ns  , min % (+ve is better)
> >      2^12,    253,266   ,     213,227     ,   6.06
> >      2^21,    246,244   ,     221,219     ,   0.00
> >      2^30,    231,240   ,     209,217     ,   3.03
> >  256*2^12,   2604,2668  ,    2415,2540    ,   4.04
> >  256*2^21,   2495,2824  ,    2390,2734    ,  12.12
> >  256*2^30,   2542,2845  ,    2380,2718    ,  12.12
> > 
> > unmap_pages
> >    pgsz  ,avg new,old ns, min new,old ns  , min % (+ve is better)
> >      2^12,    259,292   ,     222,251     ,  11.11
> >      2^21,    255,259   ,     227,236     ,   3.03
> >      2^30,    238,254   ,     217,230     ,   5.05
> >  256*2^12,   2751,2620  ,    2417,2437    ,   0.00
> >  256*2^21,   2461,2526  ,    2377,2423    ,   1.01
> >  256*2^30,   2498,2543  ,    2370,2404    ,   1.01
> > 
> 
> that is great!
> 
> Out of curiosity - why would an optimization in the unmap path also
> lead to better efficiency in the map path (in some cases map improves
> even more than unmap, e.g. with 256 pages)?

I cannot explain this, but it is how the test behaves on my system -
before this patch:

map_pages
   pgsz  ,avg new,old ns, min new,old ns  , min % (+ve is better)
     2^12,    530,237   ,     450,207     , -117.17
     2^21,    417,231   ,     375,208     , -80.80
     2^30,    325,233   ,     275,217     , -26.26
 256*2^12,   3020,2658  ,    2599,2492    ,  -4.04
 256*2^21,   2892,2839  ,    2585,2721    ,   4.04
 256*2^30,   2794,2836  ,    2548,2712    ,   6.06

unmap_pages
   pgsz  ,avg new,old ns, min new,old ns  , min % (+ve is better)
     2^12,    444,257   ,     410,230     , -78.78
     2^21,    378,244   ,     353,225     , -56.56
     2^30,    300,245   ,     281,229     , -22.22
 256*2^12,   2975,2604  ,    2536,2403    ,  -5.05
 256*2^21,   2887,2523  ,    2489,2417    ,  -2.02
 256*2^30,   2799,2525  ,    2471,2397    ,  -3.03

Perhaps it is an artifact of how the test is constructed and
subsequent maps are suffering because the cache for the already
populated levels is in a poor state due to the extra flush?

Mapping requires an atomic on the cache line so maybe that interacts
poorly with a clflush somehow?

Jason

Re: [PATCH rc] iommupt: Only cache flush memory changed by unmap

[Francois Dugast]

On Sat, Jan 24, 2026 at 05:00:21PM -0400, Jason Gunthorpe wrote:
> The cache flush was happening on every level across the whole range of
> iteration, even if no leafs or tables were cleared. Instead flush only the
> sub range that was actually written.
> 
> Overflushing isn't a correctness problem but it does impact the
> performance of unmap.
> 
> After this series the performance compared to the original VT-d
> implementation with cache flushing turned on is:
> 
> map_pages
>    pgsz  ,avg new,old ns, min new,old ns  , min % (+ve is better)
>      2^12,    253,266   ,     213,227     ,   6.06
>      2^21,    246,244   ,     221,219     ,   0.00
>      2^30,    231,240   ,     209,217     ,   3.03
>  256*2^12,   2604,2668  ,    2415,2540    ,   4.04
>  256*2^21,   2495,2824  ,    2390,2734    ,  12.12
>  256*2^30,   2542,2845  ,    2380,2718    ,  12.12
> 
> unmap_pages
>    pgsz  ,avg new,old ns, min new,old ns  , min % (+ve is better)
>      2^12,    259,292   ,     222,251     ,  11.11
>      2^21,    255,259   ,     227,236     ,   3.03
>      2^30,    238,254   ,     217,230     ,   5.05
>  256*2^12,   2751,2620  ,    2417,2437    ,   0.00
>  256*2^21,   2461,2526  ,    2377,2423    ,   1.01
>  256*2^30,   2498,2543  ,    2370,2404    ,   1.01
> 
> Fixes: efa03dab7ce4 ("iommupt: Flush the CPU cache after any writes to the page table")
> Reported-by: Francois Dugast <francois.dugast@intel.com>
> Closes: https://lore.kernel.org/all/20260121130233.257428-1-francois.dugast@intel.com/
> Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>

This reverts the performance drop I reported, thanks.

Tested-by: Francois Dugast <francois.dugast@intel.com>

Francois

RE: [PATCH rc] iommupt: Only cache flush memory changed by unmap

[Tian, Kevin]

> From: Jason Gunthorpe <jgg@nvidia.com>
> Sent: Monday, January 26, 2026 10:14 PM
> 
> On Mon, Jan 26, 2026 at 07:33:21AM +0000, Tian, Kevin wrote:
> > > From: Jason Gunthorpe <jgg@nvidia.com>
> > > Sent: Sunday, January 25, 2026 5:00 AM
> > >
> > > The cache flush was happening on every level across the whole range of
> > > iteration, even if no leafs or tables were cleared. Instead flush only the
> > > sub range that was actually written.
> > >
> > > Overflushing isn't a correctness problem but it does impact the
> > > performance of unmap.
> > >
> > > After this series the performance compared to the original VT-d
> > > implementation with cache flushing turned on is:
> > >
> > > map_pages
> > >    pgsz  ,avg new,old ns, min new,old ns  , min % (+ve is better)
> > >      2^12,    253,266   ,     213,227     ,   6.06
> > >      2^21,    246,244   ,     221,219     ,   0.00
> > >      2^30,    231,240   ,     209,217     ,   3.03
> > >  256*2^12,   2604,2668  ,    2415,2540    ,   4.04
> > >  256*2^21,   2495,2824  ,    2390,2734    ,  12.12
> > >  256*2^30,   2542,2845  ,    2380,2718    ,  12.12
> > >
> > > unmap_pages
> > >    pgsz  ,avg new,old ns, min new,old ns  , min % (+ve is better)
> > >      2^12,    259,292   ,     222,251     ,  11.11
> > >      2^21,    255,259   ,     227,236     ,   3.03
> > >      2^30,    238,254   ,     217,230     ,   5.05
> > >  256*2^12,   2751,2620  ,    2417,2437    ,   0.00
> > >  256*2^21,   2461,2526  ,    2377,2423    ,   1.01
> > >  256*2^30,   2498,2543  ,    2370,2404    ,   1.01
> > >
> >
> > that is great!
> >
> > Out of curiosity - why would an optimization in the unmap path also
> > lead to better efficiency in the map path (in some cases map improves
> > even more than unmap, e.g. with 256 pages)?
> 
> I cannot explain this, but it is how the test behaves on my system -
> before this patch:
> 
> map_pages
>    pgsz  ,avg new,old ns, min new,old ns  , min % (+ve is better)
>      2^12,    530,237   ,     450,207     , -117.17
>      2^21,    417,231   ,     375,208     , -80.80
>      2^30,    325,233   ,     275,217     , -26.26
>  256*2^12,   3020,2658  ,    2599,2492    ,  -4.04
>  256*2^21,   2892,2839  ,    2585,2721    ,   4.04
>  256*2^30,   2794,2836  ,    2548,2712    ,   6.06
> 
> unmap_pages
>    pgsz  ,avg new,old ns, min new,old ns  , min % (+ve is better)
>      2^12,    444,257   ,     410,230     , -78.78
>      2^21,    378,244   ,     353,225     , -56.56
>      2^30,    300,245   ,     281,229     , -22.22
>  256*2^12,   2975,2604  ,    2536,2403    ,  -5.05
>  256*2^21,   2887,2523  ,    2489,2417    ,  -2.02
>  256*2^30,   2799,2525  ,    2471,2397    ,  -3.03
> 
> Perhaps it is an artifact of how the test is constructed and
> subsequent maps are suffering because the cache for the already
> populated levels is in a poor state due to the extra flush?

that's a possible reason.

> 
> Mapping requires an atomic on the cache line so maybe that interacts
> poorly with a clflush somehow?
> 

depend on the concurrency of mappings vs. clflush...

anyway since it's fixed the regression:

Reviewed-by: Kevin Tian <kevin.tian@intel.com>

Re: [PATCH rc] iommupt: Only cache flush memory changed by unmap

[Baolu Lu]

On 1/25/26 05:00, Jason Gunthorpe wrote:
> The cache flush was happening on every level across the whole range of
> iteration, even if no leafs or tables were cleared. Instead flush only the
> sub range that was actually written.
> 
> Overflushing isn't a correctness problem but it does impact the
> performance of unmap.
> 
> After this series the performance compared to the original VT-d
> implementation with cache flushing turned on is:
> 
> map_pages
>     pgsz  ,avg new,old ns, min new,old ns  , min % (+ve is better)
>       2^12,    253,266   ,     213,227     ,   6.06
>       2^21,    246,244   ,     221,219     ,   0.00
>       2^30,    231,240   ,     209,217     ,   3.03
>   256*2^12,   2604,2668  ,    2415,2540    ,   4.04
>   256*2^21,   2495,2824  ,    2390,2734    ,  12.12
>   256*2^30,   2542,2845  ,    2380,2718    ,  12.12
> 
> unmap_pages
>     pgsz  ,avg new,old ns, min new,old ns  , min % (+ve is better)
>       2^12,    259,292   ,     222,251     ,  11.11
>       2^21,    255,259   ,     227,236     ,   3.03
>       2^30,    238,254   ,     217,230     ,   5.05
>   256*2^12,   2751,2620  ,    2417,2437    ,   0.00
>   256*2^21,   2461,2526  ,    2377,2423    ,   1.01
>   256*2^30,   2498,2543  ,    2370,2404    ,   1.01
> 
> Fixes: efa03dab7ce4 ("iommupt: Flush the CPU cache after any writes to the page table")
> Reported-by: Francois Dugast<francois.dugast@intel.com>
> Closes:https://lore.kernel.org/all/20260121130233.257428-1- 
> francois.dugast@intel.com/
> Signed-off-by: Jason Gunthorpe<jgg@nvidia.com>
> ---
>   drivers/iommu/generic_pt/iommu_pt.h | 11 ++++++++++-
>   1 file changed, 10 insertions(+), 1 deletion(-)

Thanks for fixing this.

Reviewed-by: Lu Baolu <baolu.lu@linux.intel.com>

Re: [PATCH rc] iommupt: Only cache flush memory changed by unmap

[Joerg Roedel]

On Sat, Jan 24, 2026 at 05:00:21PM -0400, Jason Gunthorpe wrote:
>  drivers/iommu/generic_pt/iommu_pt.h | 11 ++++++++++-
>  1 file changed, 10 insertions(+), 1 deletion(-)

Applied, thanks.