Re: iommu/smmu/PCI ATS + TTM unmaps + nouveau best way forward

[Jason Gunthorpe <jgg@nvidia.com>]

On Tue, Jun 30, 2026 at 07:39:44PM +1000, Dave Airlie wrote:
> I've been bringing up nouveau on NVIDIA Spark GB10 and fallen down a
> hole of why is the GPU writing to pages I've released back to the CPU
> page allocator. I've wasted a lot of time on GPU L2 cache and explicit
> handling for it, which might be needed but hasn't solved my problem.
> I've also invalidated and flushed the GPU TLB excessively.
>
> Today I finally hit up iommu.strict=1 makes things a lot happier,
> non-strict IOMMU seems to allow a race between dma_unmap_page and
> free_page where the unmap goes into the IOMMU flush queue, where the
> actual unmap are delayed until something triggers a flush later, and
> the ATS translations stay alive past when they should, and after the
> page has been allocated by some subsequent user.

It is a driver/device bug if it continues to DMA to memory after
dma_unmap is called. Strict mode is intended to increase security
against malicious devices and to aid debugging buggy drivers, it must
not be relied upon to have any functional effect on a correct driver.

This is not an API issue or an impedence mismatch, you have to follow
the DMA API contract when using the DMA API - meaning before unmap is
called the device is not doing DMA any more. It is a purely a
driver/device bug that the device is not quieted prior to unmapping.

You cannot fix this with any of your proposed hacks, the driver must
have control over the device and it must know when the device is done
DMA. Presumably there is a missing flush or fence that is allowing the
GPU to continue to DMA even after the OS thinks it has told it to
stop. There are many unique things about the coherent GPUs, I would
not be surprised if the driver needs some additional fencing
operations to clear the fabric beyond simple cache invalidation, but I
don't know.

Normally strict would convert these bugs from quiet corruptions into
noisy IOMMU logging, however in this case the GPU is using ATS and the
IOMMU does not log non-present ATS responses. Instead the GPU's
internal attempt to DMA will fail and the GPU will do whatever it does
to propogate that failure. Probably the driver has not noticed the
fall out, or it ignored it.

Jason