Re: [PATCH rc v2] iommu/arm-smmu-v3: Fix inconsistent ATS state tracking

[Pranjal Shrivastava <praan@google.com>]

On Wed, May 06, 2026 at 09:57:55PM +0000, Pranjal Shrivastava wrote:
> On Wed, May 06, 2026 at 06:46:30AM -0300, Jason Gunthorpe wrote:
> > On Tue, May 05, 2026 at 10:32:40PM +0000, Pranjal Shrivastava wrote:
> > > The PF shows no ATS Cap:
> > 
> > Yeah, that's complete broken and unusable. We should be failing things
> > not trying to patch around it. quirks should be used to deal with
> > this non-compliant HW.
> > 
> 
> IMPORTANT correction, the HW is NOT broken, I mis-read the logs & pasted
> the wrong ones. The BDF logs were supposed to be:
> 
> WRONG BDF logs: 000X:00:01.0 
> RIGHT BDF logs: 000X:01:00.0
> 
> Infact we're seeing an in-tree quirk[1] in action:
> 
> PF logs:
> 
> [    0.575983] pci 000X:01:00.0: [8086:1452] type 00 class 0x020000 PCIe Endpoint
> [    0.576003] pci 000X:01:00.0: BAR 0 [mem 0x640900000000-0x64090fffffff 64bit pref]
> [    0.576005] pci 000X:01:00.0: BAR 2 [mem 0x640a6cc00000-0x640a6cc3ffff 64bit pref]
> [    0.576089] pci 000X:01:00.0: VF BAR 0 [mem 0x640994800000-0x640994bfffff 64bit pref]
> [    0.576091] pci 000X:01:00.0: VF BAR 0 [mem 0x640994800000-0x640a547fffff 64bit pref]: contains BAR 0 for 768 VFs
> [    0.576093] pci 000X:01:00.0: VF BAR 2 [mem 0x640a6cc80000-0x640a6cc8ffff 64bit pref]
> [    0.576094] pci 000X:01:00.0: VF BAR 2 [mem 0x640a6cc80000-0x640a6fc7ffff 64bit pref]: contains BAR 2 for 768 VFs
> 
> [    0.832175] pci 000X:01:00.0: quirk_intel_e2000_no_ats: revision 0x11 < 0x20, disabling ATS 
> [    0.832177] pci 000X:01:00.0: quirk_no_ats: SETTING ats_cap TO 0 (was 0x2dc) <--- pdev->ats_cap = 0
> 
> [    2.365656] pci 000X:01:00.0: pci_prepare_ats: ps=12, is_virtfn=0, ats_cap=0x0
> [    2.365663] pci 000X:01:00.0: pci_prepare_ats: MANUAL SCAN FOUND ATS at 0x2dc (header 0x1000f, ctrl 0x0)
> [    2.365666] pci 000X:01:00.0: pci_prepare_ats failed: not supported
> 
> The PCI ID : Vendor ID => [8086:1452] is for Intel e2000 NICs which has
> a quirk for ATS: quirk_intel_e2000_no_ats[1]
> 
> VF logs:
> [   16.634000] pci 000X:02:00.0: [8086:145c] type 00 class 0x020000 PCIe Endpoint
> [   16.634060] pci 000X:02:00.0: pci_ats_init: pci_find_ext_capability(ATS) returned 0x1e4
> 
> [   16.634196] pci 000X:02:00.0: pci_prepare_ats: ps=12, is_virtfn=1, ats_cap=0x1e4
> [   16.634201] pci 000X:02:00.0: pci_prepare_ats: MANUAL SCAN FOUND ATS at 0x1e4
> [   16.634202] pci 000X:02:00.0: pci_prepare_ats: device is VF, returning success
> 
> [   16.634272] pci 000X:02:00.0: arm_smmu_ats_supported: dev is PCI, ats_cap=0x1e4, pci_ats_supported=1
> 
> [   16.634282] pci 000X:02:00.0: pci_enable_ats entry: ps=12, is_virtfn=1, ats_cap=0x1e4
> [   16.634283] pci 000X:02:00.0: pci_enable_ats (VF): checking PF (0005:01:00.0) -> ats_cap=0x0, ats_enabled=0, ats_stu=0
> [   16.634285] pci 000X:02:00.0: pci_enable_ats EXIT: STU mismatch (PF 0 vs VF 12)
> [   16.634286] pci 000X:02:00.0: Failed to enable ATS (STU 12) error -22
> 
> [   16.634513] pci 000X:02:00.0: quirk_intel_e2000_no_ats: revision 0x11 < 0x20, disabling ATS
> [   16.634515] pci 000X:02:00.0: quirk_no_ats: SETTING ats_cap TO 0 (was 0x1e4)
> 
> The PCI ID : Vendor ID => [8086:145c] also has the same: quirk_intel_e2000_no_ats[1]
> 
> Thus, this sequence occurs:
> 
> 1. The Intel quirk sets pdev->ats_cap = 0 for the PF at boot.
> 2. pci_prepare_ats(PF) fails during arm_smmu_probe_device() as the
>    quirk causes pci_ats_supported to return false & PF->ats_stu
>    remains 0. (arm-smmu-v3 ignores the failure)
> 
> 3. PF attaches to identity domain: pci_enable_ats is skipped
> 4. VF's arm_smmu_probe_device returns early from pci_prepare_ats() due 
>    to the if (dev->is_virtfn) return 0; check
> 
> 5. SMMU calls pci_ats_supported() for the VF in attach_prepare() which
>    returns true as the quirk hasn't been called for the VF yet. 
>    The SMMU driver sets state->ats_enabled = true
> 
> 6. We call pci_enable_ats which hits the STU mismatch (PF never sets it)
> 7. Later while disabling ATS for VF via pci_disable_ats, we hit the WARN
> 
> I guess we must think about the following:
> 
> 1. Check return values of pci_enable_ats / pci_prepare_ats in SMMUv3
> 2. PCIe core to provide a proper ATS flag that iommu driver can rely on
>    and maybe define a policy around pci_ats_supported() to make sure it 
>    also handles such quirks in pci_ats_supported().
> 
> LMK, your thoughts?
> 
> Thanks!
> Praan

[1] https://elixir.bootlin.com/linux/v7.0.1/source/drivers/pci/quirks.c#L5703