Re: [PATCH 03/11] iommu: Compute iommu_groups properly for PCIe switches

[Alex Williamson <alex.williamson@redhat.com>]

On Fri, 18 Jul 2025 10:32:59 -0300
Jason Gunthorpe <jgg@nvidia.com> wrote:

> On Thu, Jul 17, 2025 at 10:31:42PM -0400, Donald Dutile wrote:
>  
> > > > If no (optional) ACS P2P Egress control, and no other ACS control, then I read/decode
> > > > the spec to mean no p2p btwn functions is possible, b/c if it is possible, by spec,
> > > > it must have an ACS cap to control it; ergo, no ACS cap, no p2p capability/routing.  
> > > 
> > > Where did you see this? Linux has never worked this way, we have
> > > extensive ACS quirks specifically because we've assumed no ACS cap
> > > means P2P is possible and not controllable.
> > >   
> > e.g., Section 6.12.1.2 ACS Functions in SR-IOV, SIOV, and Multi-Function Devices
> >  ...
> >  ACS P2P Request Redirect: must be implemented by Functions that support peer-to-peer traffic with other Functions.
> >                            ^^^^
> > 
> > It's been noted/stated/admitted that MFDs have not followed the ACS
> > rules, and thus the quirks may/are needed.
> > 
> > Linux default code should not be opposite of the spec, i.e., if no
> > ACS, then P2P is possible, thus all fcns are part of an IOMMU group.
> > The spec states that ACS support must be provided if p2p traffic
> > with other functions is supported.  
> 
> Linux is definately the opposite of this.
> 
> Alex would you agree to reverse this logic for MFDs? If the MFD does
> not have ACS cap then the MFD does not do internal loopback P2P?
> 
> I think that solves all the MFD related problems.

Sorry, I'm way, way late to responding to this, but I interpret this to
mean that if a multifunction device implements an ACS capability, it
must implement ACS P2P RR if it supports P2P between functions.  If the
ACS capability does not implement ACS P2P RR then we can assume that the
device does not support P2P between functions, but if the device does
not implement an ACS capability at all, we cannot assume anything.

This is effectively why NIC vendors like Intel started implementing an
empty ACS capability, such that we can infer that there is no P2P
between functions.

I'm just catching up from some extended PTO, but this statement in the
cover of the new series is setting off red flags for me:

  For multi-function-devices, a PCIe topology like:

                    -- MFD 00:1f.0 ACS not supported
    Root 00:00.00 --|- MFD 00:1f.2 ACS not supported
                    |- MFD 00:1f.6 ACS = REQ_ACS_FLAGS

  Will group [1f.0, 1f.2] and 1f.6 gets a single device group. However from
  a spec perspective each device should get its own group, because ACS not
  supported can assume no loopback is possible by spec.

The ACS capability was only introduced in PCIe 2.0 and vendors have
only become more diligent about implementing it as it's become
important for device isolation and assignment.  IMO, we can't assume
anything at all about a multifunction device that does not implement
ACS.  Thanks,

Alex