Re: [PATCH v9 02/18] PCI/P2PDMA: Derive routing from directional ACS controls

[Leon Romanovsky <leon@kernel.org>]

On Wed, Oct 07, 2026 at 03:40:45PM -0500, Bjorn Helgaas wrote:
> On Wed, Oct 07, 2026 at 09:32:27AM +0300, Leon Romanovsky wrote:
> > On Tue, Oct 06, 2026 at 03:49:47PM -0500, Bjorn Helgaas wrote:
> > > On Thu, Oct 01, 2026 at 02:55:10PM +0300, Leon Romanovsky wrote:
> > > > From: Leon Romanovsky <leonro@nvidia.com>
> > > > 
> > > > pci_bridge_has_acs_redir() treats Request and Completion Redirect as
> > > > interchangeable. On asymmetric fabrics, a control for only the reverse TLP
> > > > direction can unnecessarily force P2PDMA through the host bridge.
> > > 
> > > Does "the reverse TLP direction" refer to Completions?
> > 
> > In general, the P2P code treats TLPs flowing from device A to device B the
> > same as TLPs flowing from device B to device A.
> > 
> > However, in the context of this commit message, yes: completions flow in the
> > opposite direction from the device's perspective.
> 
> And I guess asymmetric fabrics must mean fabrics where Request
> Redirect and Completion Redirect are not set the same way?

Yes, in some of our systems, deviceA to deviceB is going through
different route than deviceB to deviceA.

> 
> > > > Evaluate Request Redirect for client Requests and Completion Redirect for
> > > > provider read Completions. Continue treating enabled Egress Control
> > > > conservatively as a Request redirect.
> > > 
> > > Completion Redirect is intended to avoid ordering rule violations
> > > between Completions and Requests when Requests are redirected (PCIe
> > > r7.0, sec 6.12.1.1).  I assume this patch preserves the ordering rule,
> > > but does the commit log need to say something about that?  I don't
> > > know enough about P2P DMA for it to be obvious to me.
> > 
> > I don't think so, i didn't change anything related to ordering.
> 
> I don't think there's anything in this whole series that changes any
> ACS settings, so I shouldn't have wondered about *preserving* the
> ordering rule.
> 
> But I asked about ordering because it sounds like this patch expects
> to encounter asymmetric fabrics where Request Redirect and Completion
> Redirect may not be set the same way, and the spec implies that
> asymmetry may result in ordering violations.

My guess is that the hardware and system architects have ensured this
doesn't happen. At least, I haven't received any complaints about this
series during internal testing.

> 
> > > Not really a question for this series, but p2pdma.c and p2pdma.rst
> > > refer to "clients" and "providers", neither of which are mentioned in
> > > the PCIe spec.  In this case it sounds like a client is a Requester
> > > and a provider is a Completer in spec terms.  Is that always the case?
> > > If so, "client" and "provider" in this paragraph are not adding any
> > > information.
> > > 
> > > If "client" is not the same concept as "Requester" and "provider" not
> > > the same as "Completer", maybe p2pdma.rst could explain the
> > > difference?
> > 
> > Client vs. provider are actual target vs. initiator. They express the
> > device role in the flow.
> 
> I'd rather use "Requester" and "Completer" when possible because they
> have specific meanings in the PCI spec and they correspond to the ACS
> control bits.  Client, provider, target, initiator are all from the
> outer world that makes use of PCIe constructs, but they don't mean
> anything inside the PCIe world.

These names come from p2p users such as dma-buf and NVMe. I didn't want
to rename them as they already exists and is in use inside p2p code.

Thanks

> 
> Bjorn