Re: [PATCH RFC v2 0/4] Disable ATS via iommu during PCI resets

[Nicolin Chen <nicolinc@nvidia.com>]

On Thu, Jul 24, 2025 at 02:50:53PM +0800, Ethan Zhao wrote:
> On 6/28/2025 3:42 PM, Nicolin Chen wrote:
> > PCIe permits a device to ignore ATS invalidation TLPs, while processing a
> > reset. This creates a problem visible to the OS where an ATS invalidation
> > command will time out: e.g. an SVA domain will have no coordination with a
> > reset event and can racily issue ATS invalidations to a resetting device.
> > 
> > The OS should do something to mitigate this as we do not want production
> > systems to be reporting critical ATS failures, especially in a hypervisor
> > environment. Broadly, OS could arrange to ignore the timeouts, block page
> > table mutations to prevent invalidations, or disable and block ATS.
> > 
> > The PCIe spec in sec 10.3.1 IMPLEMENTATION NOTE recommends to disable and
> > block ATS before initiating a Function Level Reset. It also mentions that
> > other reset methods could have the same vulnerability as well.
> > 
> > Provide a callback from the PCI subsystem that will enclose the reset and
> > have the iommu core temporarily change all the attached domain to BLOCKED.
> > After attaching a BLOCKED domain, IOMMU drivers should fence any incoming
> > ATS queries, synchronously stop issuing new ATS invalidations, and wait
> > for all ATS invalidations to complete. This can avoid any ATS invaliation
> > timeouts.
> 
> This approach seems effective for reset operations initiated through
> software interface functions, but how would we handle those triggered by
> hardware mechanisms? For example, resets caused by PCIe DPC mechanisms,
> device firmware, or manual hot-plug operations?

That's a good point. But I am not sure what SW can do about those.

IIUIC, DPC resets PCI at the HW level, SW only gets a notification
after the HW reset finishes. So, during this HW reset, iommu might
issue ATC invalidations (resulting in invalidation timeout noises)
since at the SW level the device is still actively attached to an
IOMMU instance. Right?

Nicolin