Re: iommu/smmu/PCI ATS + TTM unmaps + nouveau best way forward

[Jason Gunthorpe <jgg@nvidia.com>]

On Tue, Jun 30, 2026 at 01:24:57PM +0200, Christian König wrote:

> > Today I finally hit up iommu.strict=1 makes things a lot happier,
> > non-strict IOMMU seems to allow a race between dma_unmap_page and
> > free_page where the unmap goes into the IOMMU flush queue, where the
> > actual unmap are delayed until something triggers a flush later, and
> > the ATS translations stay alive past when they should, and after the
> > page has been allocated by some subsequent user.
> 
> Try disabling ATS, that was a common source of problems during HW
> bringup for us in the past.

The spark HW cannot run without ATS.

> > 2. split ttm page unmap/free into two stages, and force an iommu
> > flush/ATS sync operation after the unmap before the free. (I don't
> > think iommu has an interface for this).
> 
> Yeah that idea already came up on similar issues as well. I
> certainly wouldn't be opposed to that.

Only for debugging, it is completely wrong to rely on this for
functionality.

> You either have to use memory fences which use the same path as the
> DMA operations itself

Yes, in a PCIe system generally something like the GPU TLB
invalidation command will complete and flush the fabric path to quiet
the DMAs. That won't be true on spark though.

Dave could be facing a fabric issue. AFAIK the ATS invalidation that
the iommu=strict pushses should fence the fabric so would resolve a
missing fabric fence. I suppose if you don't see any evidence of a
non-present ATS response explosion with strict then this would be a
really good theory.

However, the GPU must have another way to inject a fabric fence. The
production driver does not rely on ATS fencing to synchronize memory
operations!

> So far the record for grace periods you need after a TLB flush is 6
> seconds on production HW with up to 10 seconds on engineering
> samples (because of HW bugs).

Yeah, if the HW mechanism to fence the fabric don't work spark would
need similar times to guarentee a fabric drain.

Jason