Re: [PATCH rc v2] iommu/arm-smmu-v3: Fix inconsistent ATS state tracking

[Pranjal Shrivastava <praan@google.com>]

On Wed, May 06, 2026 at 06:46:30AM -0300, Jason Gunthorpe wrote:
> On Tue, May 05, 2026 at 10:32:40PM +0000, Pranjal Shrivastava wrote:
> > The PF shows no ATS Cap:
> 
> Yeah, that's complete broken and unusable. We should be failing things
> not trying to patch around it. quirks should be used to deal with
> this non-compliant HW.
> 

IMPORTANT correction, the HW is NOT broken, I mis-read the logs & pasted
the wrong ones. The BDF logs were supposed to be:

WRONG BDF logs: 000X:00:01.0 
RIGHT BDF logs: 000X:01:00.0

Infact we're seeing an in-tree quirk[1] in action:

PF logs:

[    0.575983] pci 000X:01:00.0: [8086:1452] type 00 class 0x020000 PCIe Endpoint
[    0.576003] pci 000X:01:00.0: BAR 0 [mem 0x640900000000-0x64090fffffff 64bit pref]
[    0.576005] pci 000X:01:00.0: BAR 2 [mem 0x640a6cc00000-0x640a6cc3ffff 64bit pref]
[    0.576089] pci 000X:01:00.0: VF BAR 0 [mem 0x640994800000-0x640994bfffff 64bit pref]
[    0.576091] pci 000X:01:00.0: VF BAR 0 [mem 0x640994800000-0x640a547fffff 64bit pref]: contains BAR 0 for 768 VFs
[    0.576093] pci 000X:01:00.0: VF BAR 2 [mem 0x640a6cc80000-0x640a6cc8ffff 64bit pref]
[    0.576094] pci 000X:01:00.0: VF BAR 2 [mem 0x640a6cc80000-0x640a6fc7ffff 64bit pref]: contains BAR 2 for 768 VFs

[    0.832175] pci 000X:01:00.0: quirk_intel_e2000_no_ats: revision 0x11 < 0x20, disabling ATS 
[    0.832177] pci 000X:01:00.0: quirk_no_ats: SETTING ats_cap TO 0 (was 0x2dc) <--- pdev->ats_cap = 0

[    2.365656] pci 000X:01:00.0: pci_prepare_ats: ps=12, is_virtfn=0, ats_cap=0x0
[    2.365663] pci 000X:01:00.0: pci_prepare_ats: MANUAL SCAN FOUND ATS at 0x2dc (header 0x1000f, ctrl 0x0)
[    2.365666] pci 000X:01:00.0: pci_prepare_ats failed: not supported

The PCI ID : Vendor ID => [8086:1452] is for Intel e2000 NICs which has
a quirk for ATS: quirk_intel_e2000_no_ats[1]

VF logs:
[   16.634000] pci 000X:02:00.0: [8086:145c] type 00 class 0x020000 PCIe Endpoint
[   16.634060] pci 000X:02:00.0: pci_ats_init: pci_find_ext_capability(ATS) returned 0x1e4

[   16.634196] pci 000X:02:00.0: pci_prepare_ats: ps=12, is_virtfn=1, ats_cap=0x1e4
[   16.634201] pci 000X:02:00.0: pci_prepare_ats: MANUAL SCAN FOUND ATS at 0x1e4
[   16.634202] pci 000X:02:00.0: pci_prepare_ats: device is VF, returning success

[   16.634272] pci 000X:02:00.0: arm_smmu_ats_supported: dev is PCI, ats_cap=0x1e4, pci_ats_supported=1

[   16.634282] pci 000X:02:00.0: pci_enable_ats entry: ps=12, is_virtfn=1, ats_cap=0x1e4
[   16.634283] pci 000X:02:00.0: pci_enable_ats (VF): checking PF (0005:01:00.0) -> ats_cap=0x0, ats_enabled=0, ats_stu=0
[   16.634285] pci 000X:02:00.0: pci_enable_ats EXIT: STU mismatch (PF 0 vs VF 12)
[   16.634286] pci 000X:02:00.0: Failed to enable ATS (STU 12) error -22

[   16.634513] pci 000X:02:00.0: quirk_intel_e2000_no_ats: revision 0x11 < 0x20, disabling ATS
[   16.634515] pci 000X:02:00.0: quirk_no_ats: SETTING ats_cap TO 0 (was 0x1e4)

The PCI ID : Vendor ID => [8086:145c] also has the same: quirk_intel_e2000_no_ats[1]

Thus, this sequence occurs:

1. The Intel quirk sets pdev->ats_cap = 0 for the PF at boot.
2. pci_prepare_ats(PF) fails during arm_smmu_probe_device() as the
   quirk causes pci_ats_supported to return false & PF->ats_stu
   remains 0. (arm-smmu-v3 ignores the failure)

3. PF attaches to identity domain: pci_enable_ats is skipped
4. VF's arm_smmu_probe_device returns early from pci_prepare_ats() due 
   to the if (dev->is_virtfn) return 0; check

5. SMMU calls pci_ats_supported() for the VF in attach_prepare() which
   returns true as the quirk hasn't been called for the VF yet. 
   The SMMU driver sets state->ats_enabled = true

6. We call pci_enable_ats which hits the STU mismatch (PF never sets it)
7. Later while disabling ATS for VF via pci_disable_ats, we hit the WARN

I guess we must think about the following:

1. Check return values of pci_enable_ats / pci_prepare_ats in SMMUv3
2. PCIe core to provide a proper ATS flag that iommu driver can rely on
   and maybe define a policy around pci_ats_supported() to make sure it 
   also handles such quirks in pci_ats_supported().

LMK, your thoughts?

Thanks!
Praan