Re: [PATCH rc v2] iommu/arm-smmu-v3: Fix inconsistent ATS state tracking

[Pranjal Shrivastava <praan@google.com>]

On Tue, May 05, 2026 at 09:14:03PM +0000, Pranjal Shrivastava wrote:
> On Tue, May 05, 2026 at 01:11:26PM -0300, Jason Gunthorpe wrote:
> > On Mon, May 04, 2026 at 07:33:07PM +0000, Pranjal Shrivastava wrote:
> > 
> > > > > The issue was exposed under heavy load when running a VFIO-based DMA map
> > > > > stress test: iova_stress [1]
> > > > 
> > > > I wonder what's the real reason for pci_enable_ats() to fail:
> > > > 
> > > 
> > > Yes, It's the dev->is_virtfn case (the one you suspect below)
> > 
> > Oh.... This is a much larger problem then
> > 
> > The VF should not fail to enable ATS, that is a meaningful bug, I'm
> > not sure why are are just discovering this?
> > 
> > It looks like the PF unconditionally calls pci_prepare_ats() during
> > arm_smmu_probe_device(), so how does this happen:
> > 
> > > >         if (dev->is_virtfn) {
> > > >                 pdev = pci_physfn(dev);
> > > >                 if (pdev->ats_stu != ps)
> > > >                         return -EINVAL;		// maybe this one?
> > 
> > ??
> > 
> > So, I think the right error handling for a case that shouldn't happen is
> > to fail the attachment not to try to continue it broken?
> > 
> 
> Hmm.. you mean we should fail the VF attach if it's PF's ATS isn't
> enabled?
> 
> I agree prepare_ats is called regardless on every device and we return
> early without setting the STU if (dev->is_virtfn) but the catch is we
> never check for pci_prepare_ats's return value in probe_device :/
> (The PF might be untrusted or non ATS capable).
> 
> Here's the EXACT failing case:
> 
> 1. arm_smmu_probe_device(PF) -> we call pci_prepare_ats & ignore retval
> 2. The PF attaches to Identity domain (pci_enable_ats skipped, no logs)
> 3. We create VFs and try to assign them (new group -> unmanaged domain)
> 4. We try to enable ATS in attach_commit, fail and see the failure log
> 5. arm-smmu-v3 driver tracks the state as "ATS enabled" for VF
> 6. We try to disable ATS which was never enabled and hit the WARN
> 
> The catch is in step 1, pci_prepare_ats(PF) returned -EINVAL as either
> the ATS capability didn't exist OR the device was untrusted but the
> driver let it slide (ignored the retval in probe_device).
> I'll debug further to see which one is it (because in that case
> arm_smmu_ats_supported should've failed too).
> 

I added some logs to check the reason of failure.
To make the matter worse, I believe I spotted this due to a broken HW.
(Although, we can face something similar with untrusted etc. in the mix)

The PF shows no ATS Cap:

[    0.575749] PCI host bridge to bus 000X:00
[    0.575773] pci 000X:00:00.0: set_pcie_untrusted: no parent bridge, removable=0, external_facing=0
[    0.575774] pci 000X:00:00.0: [abab:cdcd] type 00 class 0x060000 conventional PCI endpoint
[    0.575786] pci 000X:00:00.0: pci_ats_supported: NO ATS CAPABILITY FOUND (ats_cap=0)
[    0.575787] pci 000X:00:00.0: pci_ats_supported: NO ATS CAPABILITY FOUND (ats_cap=0)
[    0.575819] pci 000X:00:01.0: set_pcie_untrusted: no parent bridge, removable=0, external_facing=0
[    0.575821] pci 000X:00:01.0: [abab:cdcd] type 01 class 0x060400 PCIe Root Port
[    2.591975] pci 000X:00:00.0: pci_ats_supported: NO ATS CAPABILITY FOUND (ats_cap=0)
[    2.591979] pci 000X:00:00.0: pci_prepare_ats: ps=12, is_virtfn=0, ats_cap=0x0
[    2.591980] pci 000X:00:00.0: pci_ats_supported: NO ATS CAPABILITY FOUND (ats_cap=0)
[    2.591981] pci 000X:00:00.0: pci_prepare_ats failed: not supported
[    2.592001] pci 000X:00:00.0: Adding to iommu group 19
[    2.592007] pci 000X:00:01.0: pci_ats_supported: NO ATS CAPABILITY FOUND (ats_cap=0)
[    2.592009] pci 000X:00:01.0: pci_prepare_ats: ps=12, is_virtfn=0, ats_cap=0x0

Whereas the VF presents ATS cap:

[   16.803702] pci 000X:02:00.0: [abab:cdcd] type 00 class 0x020000 PCIe Endpoint
[   16.803869] pci 000X:02:00.0: pci_prepare_ats: ps=12, is_virtfn=1, ats_cap=0x1e4
[   16.803871] pci 000X:02:00.0: pci_prepare_ats: device is VF, returning success
[   16.803927] pci 000X:02:00.0: Adding to iommu group <redacted>
[   16.803941] pci 000X:02:00.0: arm_smmu_ats_supported: dev is PCI, ats_cap=0x1e4, untrusted=0, pci_ats_supported=1
[   16.803949] pci 000X:02:00.0: Calling pci_enable_ats(STU 12)
[   16.803950] pci 000X:02:00.0: pci_enable_ats: ps=12, is_virtfn=1, ats_cap=0x1e4
[   16.803951] pci 000X:02:00.0: pci_enable_ats: checking PF (000X:01:00.0) -> ats_cap=0x0, ats_enabled=0, ats_stu=0
[   16.803952] pci 000X:02:00.0: pci_enable_ats failed: STU mismatch (PF STU = 0 vs VF STU = 12)
[   16.803954] pci 000X:02:00.0: Failed to enable ATS (STU 12) error -22
[   16.804168] pci 000X:02:00.0: disabling ATS

which means pci_ats_supported returns true for such broken HW. Ideally,
it should return false for VF if the PF doesn't support ATS!

Currently, we don't *really* enable ATS but I believe pci_ats_supported
should fail in such cases.

Thanks,
Praan