Re: [PATCH rc] iommupt: Only cache flush memory changed by unmap

[Jason Gunthorpe <jgg@nvidia.com>]

On Mon, Jan 26, 2026 at 07:33:21AM +0000, Tian, Kevin wrote:
> > From: Jason Gunthorpe <jgg@nvidia.com>
> > Sent: Sunday, January 25, 2026 5:00 AM
> > 
> > The cache flush was happening on every level across the whole range of
> > iteration, even if no leafs or tables were cleared. Instead flush only the
> > sub range that was actually written.
> > 
> > Overflushing isn't a correctness problem but it does impact the
> > performance of unmap.
> > 
> > After this series the performance compared to the original VT-d
> > implementation with cache flushing turned on is:
> > 
> > map_pages
> >    pgsz  ,avg new,old ns, min new,old ns  , min % (+ve is better)
> >      2^12,    253,266   ,     213,227     ,   6.06
> >      2^21,    246,244   ,     221,219     ,   0.00
> >      2^30,    231,240   ,     209,217     ,   3.03
> >  256*2^12,   2604,2668  ,    2415,2540    ,   4.04
> >  256*2^21,   2495,2824  ,    2390,2734    ,  12.12
> >  256*2^30,   2542,2845  ,    2380,2718    ,  12.12
> > 
> > unmap_pages
> >    pgsz  ,avg new,old ns, min new,old ns  , min % (+ve is better)
> >      2^12,    259,292   ,     222,251     ,  11.11
> >      2^21,    255,259   ,     227,236     ,   3.03
> >      2^30,    238,254   ,     217,230     ,   5.05
> >  256*2^12,   2751,2620  ,    2417,2437    ,   0.00
> >  256*2^21,   2461,2526  ,    2377,2423    ,   1.01
> >  256*2^30,   2498,2543  ,    2370,2404    ,   1.01
> > 
> 
> that is great!
> 
> Out of curiosity - why would an optimization in the unmap path also
> lead to better efficiency in the map path (in some cases map improves
> even more than unmap, e.g. with 256 pages)?

I cannot explain this, but it is how the test behaves on my system -
before this patch:

map_pages
   pgsz  ,avg new,old ns, min new,old ns  , min % (+ve is better)
     2^12,    530,237   ,     450,207     , -117.17
     2^21,    417,231   ,     375,208     , -80.80
     2^30,    325,233   ,     275,217     , -26.26
 256*2^12,   3020,2658  ,    2599,2492    ,  -4.04
 256*2^21,   2892,2839  ,    2585,2721    ,   4.04
 256*2^30,   2794,2836  ,    2548,2712    ,   6.06

unmap_pages
   pgsz  ,avg new,old ns, min new,old ns  , min % (+ve is better)
     2^12,    444,257   ,     410,230     , -78.78
     2^21,    378,244   ,     353,225     , -56.56
     2^30,    300,245   ,     281,229     , -22.22
 256*2^12,   2975,2604  ,    2536,2403    ,  -5.05
 256*2^21,   2887,2523  ,    2489,2417    ,  -2.02
 256*2^30,   2799,2525  ,    2471,2397    ,  -3.03

Perhaps it is an artifact of how the test is constructed and
subsequent maps are suffering because the cache for the already
populated levels is in a poor state due to the extra flush?

Mapping requires an atomic on the cache line so maybe that interacts
poorly with a clflush somehow?

Jason