Re: Xe performance regression with recent IOMMU changes

[Jason Gunthorpe <jgg@nvidia.com>]

On Wed, Jan 21, 2026 at 09:11:35AM -0400, Jason Gunthorpe wrote:
> On Wed, Jan 21, 2026 at 02:02:16PM +0100, Francois Dugast wrote:
> > I am reporting a slowdown in Xe caused by a couple of IOMMU changes. It
> > can be observed during DMA mappings/unmappings required to issue copies
> > between system memory and the device, when handling GPU faults. Not sure
> > how other use cases or vendors are affected but below is the impact on
> > execution times for BMG:
> > 
> > Before changes:
> >   4KB
> >     drm_pagemap_migrate_map_pages: 0.4 us
> >     drm_pagemap_migrate_unmap_pages: 0.4 us
> >   64KB
> >     drm_pagemap_migrate_map_pages: 2.5 us
> >     drm_pagemap_migrate_unmap_pages: 3.5 us
> >   2MB
> >     drm_pagemap_migrate_map_pages: 88 us
> >     drm_pagemap_migrate_unmap_pages: 108 us
> > 
> > After changes:
> >   4KB
> >     drm_pagemap_migrate_map_pages: 0.7 us
> >     drm_pagemap_migrate_unmap_pages: 0.7 us
> >   64KB
> >     drm_pagemap_migrate_map_pages: 3.5 us
> >     drm_pagemap_migrate_unmap_pages: 10.5 us
> >   2MB
> >     drm_pagemap_migrate_map_pages: 102 us
> >     drm_pagemap_migrate_unmap_pages: 330 us
> 
> I posted some more optimizations for these cases, it should reduce the
> numbers.
> 
> This is the opposite of the benchmark numbers I ran which showed
> significant gains as the page count and sizes increased.
> 
> But something weird is going on to see a 3x increase in unmap, that
> shouldn't be just algorithm overhead. That almost seems like
> additional IOTLB invalidation overhead or something else going wrong.
> 
> Is this from a system with the VT-d cache flushing requirement? That
> logic changed around too and could have this kind of big impact.

Oh looking at the code a bit you've got pretty much the slowest
possible thing you can do here:

	for (i = 0; i < npages;) {
		if (!pagemap_addr[i].addr || dma_mapping_error(dev, pagemap_addr[i].addr))
			goto next;

		dma_unmap_page(dev, pagemap_addr[i].addr, PAGE_SIZE << pagemap_addr[i].order, dir);

It is weird though:

0.7 us * 512 = 358us so it is about the reported speed.

But the old one is 0.4 us * 512 = 204 us which is twice as
slow as reported?? It got 2x faster the more times you loop it? Huh?

The real way to fix this up is to use the new DMA API so this can be
collapsed into a single unmap. Then it will take < 1us for all those cases.

Look at the patches Leon made for the RDMA ODP stuff, it has a similar
looking workflow.

The optimizations I posted will help this noticably.

Jason