Re: iommu/smmu/PCI ATS + TTM unmaps + nouveau best way forward

[Alistair Popple <apopple@nvidia.com>]

On 2026-07-01 at 09:21 +1000, Matthew Brost <matthew.brost@intel.com> wrote...
> On Tue, Jun 30, 2026 at 08:39:19AM -0300, Jason Gunthorpe wrote:
> > On Tue, Jun 30, 2026 at 07:39:44PM +1000, Dave Airlie wrote:
> > > I've been bringing up nouveau on NVIDIA Spark GB10 and fallen down a
> > > hole of why is the GPU writing to pages I've released back to the CPU
> > > page allocator. I've wasted a lot of time on GPU L2 cache and explicit
> > > handling for it, which might be needed but hasn't solved my problem.
> > > I've also invalidated and flushed the GPU TLB excessively.
> > >
> > > Today I finally hit up iommu.strict=1 makes things a lot happier,
> > > non-strict IOMMU seems to allow a race between dma_unmap_page and
> > > free_page where the unmap goes into the IOMMU flush queue, where the
> > > actual unmap are delayed until something triggers a flush later, and
> > > the ATS translations stay alive past when they should, and after the
> > > page has been allocated by some subsequent user.
> > 
> > It is a driver/device bug if it continues to DMA to memory after
> > dma_unmap is called. Strict mode is intended to increase security
> > against malicious devices and to aid debugging buggy drivers, it must
> > not be relied upon to have any functional effect on a correct driver.
> > 
> 
> I think I agree here. In Xe, AFAIK, we never call dma_unmap until all
> fences have been signaled (dma-fence or preempt-fence mode), after which
> GPU translations are rebuilt before the next fence is installed, or
> until the GPU page tables have been invalidated (GPU page faults). TTM
> is a little harder to reason about, as we DMA-unmap via
> ->ttm_tt_unpopulate(), but I just audited the call sites and TTM looks
> correct.
> 
> > This is not an API issue or an impedence mismatch, you have to follow
> > the DMA API contract when using the DMA API - meaning before unmap is
> > called the device is not doing DMA any more. It is a purely a
> > driver/device bug that the device is not quieted prior to unmapping.
> > 
> > You cannot fix this with any of your proposed hacks, the driver must
> > have control over the device and it must know when the device is done
> > DMA. Presumably there is a missing flush or fence that is allowing the
> > GPU to continue to DMA even after the OS thinks it has told it to
> > stop. There are many unique things about the coherent GPUs, I would
> > not be surprised if the driver needs some additional fencing
> > operations to clear the fabric beyond simple cache invalidation, but I
> > don't know.
> > 
> > Normally strict would convert these bugs from quiet corruptions into
> > noisy IOMMU logging, however in this case the GPU is using ATS and the
> > IOMMU does not log non-present ATS responses. Instead the GPU's
> > internal attempt to DMA will fail and the GPU will do whatever it does
> > to propogate that failure. Probably the driver has not noticed the
> > fall out, or it ignored it.
> 
> Right. We only have ATS enabled in Xe for experimentation, but whenever
> we've had issues with dma_unmap ordering in Xe without ATS, they have
> resulted in DMAR-related dmesg spam.
> 
> So TL;DR I'm guessing Nouveau has dma_unmap before GPU idle bug
> somewhere.

Agree with most of the above and this seems the most likely explaination. ATS
doesn't change much here - most of our driver still goes via dma_map/unmap to
access host memory via an IOMMU IOVA so non-present entries there would be fatal
at the IOMMU level. So even with ATS enabled a non-present DMA mapping should
still result in noisy IOMMU faults.

 - Alistair

> Matt
> 
> > 
> > Jason