Re: iommu/smmu/PCI ATS + TTM unmaps + nouveau best way forward

["Christian König" <christian.koenig@amd.com>]

On 6/30/26 11:39, Dave Airlie wrote:
> I've been bringing up nouveau on NVIDIA Spark GB10 and fallen down a
> hole of why is the GPU writing to pages I've released back to the CPU
> page allocator. I've wasted a lot of time on GPU L2 cache and explicit
> handling for it, which might be needed but hasn't solved my problem.
> I've also invalidated and flushed the GPU TLB excessively.

How does nouveau make sure that all DMA transactions have completed (e.g. caches flushed etc...)?

> Today I finally hit up iommu.strict=1 makes things a lot happier,
> non-strict IOMMU seems to allow a race between dma_unmap_page and
> free_page where the unmap goes into the IOMMU flush queue, where the
> actual unmap are delayed until something triggers a flush later, and
> the ATS translations stay alive past when they should, and after the
> page has been allocated by some subsequent user.

Try disabling ATS, that was a common source of problems during HW bringup for us in the past.

It obviously cripples the performance, but when the problem doesn't happen any more you have at least an additional data point to narrow things down.

> The options I seem to have: (but I'm probably very wrong)
> 1. force my own iommu domain and use iommu_map/unmap explicitly in the
> driver which avoids the flush queue?
> 2. split ttm page unmap/free into two stages, and force an iommu
> flush/ATS sync operation after the unmap before the free. (I don't
> think iommu has an interface for this).

Yeah that idea already came up on similar issues as well. I certainly wouldn't be opposed to that.

> 3. force untrusted on the PCI device?
> 
> I just want to see if anyone else has hit this problem before in this
> space and I'm missing something, just feel solving this in the driver
> is a workaround for ttm/dma API impedance mismatch.

Yeah we have fought with stuff like that on basic every AMD GPU HW generation.

From experience I would say that both MMU implementations inside GPUs/accelerators as well as well as IOMMU are really hard to get 100% reliable in preventing random memory writes.

You either have to use memory fences which use the same path as the DMA operations itself or you have extra heavyweight TLB flushes with just insane overhead or grace periods to make sure that all transactions have actually reached system memory.

So far the record for grace periods you need after a TLB flush is 6 seconds on production HW with up to 10 seconds on engineering samples (because of HW bugs).

Regards,
Christian.

> 
> Thanks,
> Dave.