Re: [PATCH 02/11] PCI: Add pci_bus_isolation()

[Jason Gunthorpe <jgg@nvidia.com>]

On Thu, Jul 03, 2025 at 04:17:27PM -0600, Alex Williamson wrote:
> > So I will change it to be like:
> > 
> > 	/*
> > 	 * This bus was created by pci_register_host_bridge(). There is nothing
> > 	 * upstream of this, assume it contains the TA and that the root complex
> > 	 * does not allow P2P without going through the IOMMU.
> > 	 */
> > 	if (pci_is_root_bus(bus))
> > 		return PCIE_ISOLATED;
> 
> Ok, but did we sidestep the question of whether the root bus can be
> conventional?

The root bus doesn't exist, it is the thing on the other side of the
host bridge.. So it is implementation specific and I don't think we
can make any guesses about it.

> > And now it seems we never took care with SRIOV, along with the PF
> > every SRIOV VF needs to have its ACS checked as though it was a MFD..
> 
> There's actually evidence that we did take care to make sure VFs never
> flag themselves as multifunction in order to avoid the multifunction
> ACS tests.

That's not what I mean.. The spec says:

 6.12 Access Control Services (ACS)

 ACS defines a set of control points within a PCI Express topology to
 determine whether a TLP is to be routed normally, blocked, or
 redirected. ACS is applicable to RCs, Switches, and Multi-Function
 Devices. For ACS requirements, single-Function devices that are
 SR-IOV capable must be handled as if they were Multi-Function
 Devices, since they essentially behave as Multi-Function Devices
 after their Virtual Functions (VFs) are enabled.

I read "essentially behave as Multi-Function Devices" as meaning the
VFs and PFs are all together and have possible internal loopback
similar to a MFDs.

Meaning we can have P2P between VFs and ACS is present to inhibit
that.

>  I think we'd see lots of devices suddenly unusable for one
> of their intended use cases if we grouped VFs that don't expose an ACS
> capability.  

That may be, but how should the spec be understood?

> Also VFs from multiple PFs exist on the same virtual bus,
> so I imagine if the PF supports ACS but the VF doesn't, you'd end up
> with multiple isolation domains on the same bus.  

AFAICT the virtual bus thing is a Linux-ism to handle the bus numbers
taken over by SRIOV VF RIDS going past a single bus number. I've
revised things to better handle that by processing only the physical
busses.

I wasn't thinking multiple groups because ACS is all or nothing - if
one VF/PF has ACS that permits it to P2P internally to all other
functions then the entire PF and all VFs are one group. Same logic as
MFDs.

> Thus, we've so far take the approach that "surely the hw vendor
> intended these to be used independently", and only considered the
> isolation upstream from the VFs.  Thanks,

So maybe if the ACS capability is not present we assume that it means
there is no internal loopback, otherwise the ACS must be correct?

Jason