Re: [PATCH rc v2] iommu/arm-smmu-v3: Fix inconsistent ATS state tracking

[Samiullah Khawaja <skhawaja@google.com>]

On Wed, May 06, 2026 at 09:57:55PM +0000, Pranjal Shrivastava wrote:
>On Wed, May 06, 2026 at 06:46:30AM -0300, Jason Gunthorpe wrote:
>> On Tue, May 05, 2026 at 10:32:40PM +0000, Pranjal Shrivastava wrote:
>> > The PF shows no ATS Cap:
>>
>> Yeah, that's complete broken and unusable. We should be failing things
>> not trying to patch around it. quirks should be used to deal with
>> this non-compliant HW.
>>
>
>IMPORTANT correction, the HW is NOT broken, I mis-read the logs & pasted
>the wrong ones. The BDF logs were supposed to be:
>
>WRONG BDF logs: 000X:00:01.0
>RIGHT BDF logs: 000X:01:00.0
>
>Infact we're seeing an in-tree quirk[1] in action:
>
>PF logs:
>
>[    0.575983] pci 000X:01:00.0: [8086:1452] type 00 class 0x020000 PCIe Endpoint
>[    0.576003] pci 000X:01:00.0: BAR 0 [mem 0x640900000000-0x64090fffffff 64bit pref]
>[    0.576005] pci 000X:01:00.0: BAR 2 [mem 0x640a6cc00000-0x640a6cc3ffff 64bit pref]
>[    0.576089] pci 000X:01:00.0: VF BAR 0 [mem 0x640994800000-0x640994bfffff 64bit pref]
>[    0.576091] pci 000X:01:00.0: VF BAR 0 [mem 0x640994800000-0x640a547fffff 64bit pref]: contains BAR 0 for 768 VFs
>[    0.576093] pci 000X:01:00.0: VF BAR 2 [mem 0x640a6cc80000-0x640a6cc8ffff 64bit pref]
>[    0.576094] pci 000X:01:00.0: VF BAR 2 [mem 0x640a6cc80000-0x640a6fc7ffff 64bit pref]: contains BAR 2 for 768 VFs
>
>[    0.832175] pci 000X:01:00.0: quirk_intel_e2000_no_ats: revision 0x11 < 0x20, disabling ATS
>[    0.832177] pci 000X:01:00.0: quirk_no_ats: SETTING ats_cap TO 0 (was 0x2dc) <--- pdev->ats_cap = 0
>
>[    2.365656] pci 000X:01:00.0: pci_prepare_ats: ps=12, is_virtfn=0, ats_cap=0x0
>[    2.365663] pci 000X:01:00.0: pci_prepare_ats: MANUAL SCAN FOUND ATS at 0x2dc (header 0x1000f, ctrl 0x0)
>[    2.365666] pci 000X:01:00.0: pci_prepare_ats failed: not supported
>
>The PCI ID : Vendor ID => [8086:1452] is for Intel e2000 NICs which has
>a quirk for ATS: quirk_intel_e2000_no_ats[1]
>
>VF logs:
>[   16.634000] pci 000X:02:00.0: [8086:145c] type 00 class 0x020000 PCIe Endpoint
>[   16.634060] pci 000X:02:00.0: pci_ats_init: pci_find_ext_capability(ATS) returned 0x1e4
>
>[   16.634196] pci 000X:02:00.0: pci_prepare_ats: ps=12, is_virtfn=1, ats_cap=0x1e4
>[   16.634201] pci 000X:02:00.0: pci_prepare_ats: MANUAL SCAN FOUND ATS at 0x1e4
>[   16.634202] pci 000X:02:00.0: pci_prepare_ats: device is VF, returning success
>
>[   16.634272] pci 000X:02:00.0: arm_smmu_ats_supported: dev is PCI, ats_cap=0x1e4, pci_ats_supported=1
>
>[   16.634282] pci 000X:02:00.0: pci_enable_ats entry: ps=12, is_virtfn=1, ats_cap=0x1e4
>[   16.634283] pci 000X:02:00.0: pci_enable_ats (VF): checking PF (0005:01:00.0) -> ats_cap=0x0, ats_enabled=0, ats_stu=0
>[   16.634285] pci 000X:02:00.0: pci_enable_ats EXIT: STU mismatch (PF 0 vs VF 12)
>[   16.634286] pci 000X:02:00.0: Failed to enable ATS (STU 12) error -22
>
>[   16.634513] pci 000X:02:00.0: quirk_intel_e2000_no_ats: revision 0x11 < 0x20, disabling ATS
>[   16.634515] pci 000X:02:00.0: quirk_no_ats: SETTING ats_cap TO 0 (was 0x1e4)
>
>The PCI ID : Vendor ID => [8086:145c] also has the same: quirk_intel_e2000_no_ats[1]
>
>Thus, this sequence occurs:
>
>1. The Intel quirk sets pdev->ats_cap = 0 for the PF at boot.
>2. pci_prepare_ats(PF) fails during arm_smmu_probe_device() as the
>   quirk causes pci_ats_supported to return false & PF->ats_stu
>   remains 0. (arm-smmu-v3 ignores the failure)

Ok that explains how PF stu is zero.
>
>3. PF attaches to identity domain: pci_enable_ats is skipped
>4. VF's arm_smmu_probe_device returns early from pci_prepare_ats() due
>   to the if (dev->is_virtfn) return 0; check
>
>5. SMMU calls pci_ats_supported() for the VF in attach_prepare() which
>   returns true as the quirk hasn't been called for the VF yet.
>   The SMMU driver sets state->ats_enabled = true
>
>6. We call pci_enable_ats which hits the STU mismatch (PF never sets it)
>7. Later while disabling ATS for VF via pci_disable_ats, we hit the WARN
>
>I guess we must think about the following:
>
>1. Check return values of pci_enable_ats / pci_prepare_ats in SMMUv3
>2. PCIe core to provide a proper ATS flag that iommu driver can rely on
>   and maybe define a policy around pci_ats_supported() to make sure it
>   also handles such quirks in pci_ats_supported().

It seems like that ats_supported() checking only ats_cap of the VF is
not enough, since ATS can be enabled for a VF only when the stu matches
with the PF? So even if ats_supported() for the VF returns true, its not
really supported if the stu doesn't match. And the stu would be set
properly if the PF had ats_cap. Should the ats_supported() check for
the ats_cap for the PF also?

We can check the PF ats_cap in the ats_supported() or handle the
enable_ats() error in arm smmuv3. Intel and AMD drivers seems to be
handling the error.
>
>LMK, your thoughts?
>
>Thanks!
>Praan