Re: [PATCH for-next] RDMA/efa: Expose device P2P DMA support via device query

[Jason Gunthorpe <jgg@nvidia.com>]

On Thu, Sep 24, 2026 at 07:30:26PM +0000, Michael Margolin wrote:
> On Thu, Sep 24, 2026 at 03:22:01PM -0300, Jason Gunthorpe wrote:
> > On Thu, Sep 24, 2026 at 03:29:13PM +0000, Michael Margolin wrote:
> > 
> > > > > 2. Expose EFA_DEV_CAP(dev, PCIE_PEER_ACCESS_ENABLED)
> > > > >    directly to userspace. As you already stated, this is a
> > > > >    bit awkward as a device isn't expected to be aware of
> > > > >    system topology, but since EFA lives in the cloud it is
> > > > >    partially aware of the platform configuration and in
> > > > >    particular the ability to access peer devices over PCIe.
> > > > 
> > > > If you already know your VMs are always safe what is even the issue?
> > > > What are you probing for?
> > > > 
> > > > Lets please not hack around bad userspace with even worse kernel uAPI.
> > > 
> > > The EFA device already knows this attribute. The problem is that this
> > > information is not exposed to userspace today.
> > > 
> > > So if I go back to our original proposal - exposing the
> > > EFA_DEV_CAP(dev, P2P_DMA) bit through query device - this is a firmware
> > > attribute that says "this device does not block PCIe peer access on its
> > > end." It's not a topology claim, it's a device property, similar to how
> > > we already expose RDMA read/write support.
> > 
> > And how does that help you if you actually need reachability? You
> > haven't said directly what you are even trying to achieve (though I
> > have a pretty good guess).
> 
> On some platforms P2P is blocked at the device level despite the fact
> that the peer GPU is reachable over PCIe.

This idea just doesn't exist in Linux.. It is completely inappropriate
for a PCI device to declare it knows something about the
topology. That's not how PCI works, it is the wrong place for your VM
to signal information about the PCI routing through the EFA driver.

That needs to come up to the P2P subsystem.

> This mainly happens in multi-instance setups where there is a need
> to manage PCIe bandwidth or prevent noisy neighbor effects. Today we
> check this attribute on the device during MR registration, so
> libfabric's EFA provider allocates GPU memory and attempts to
> register an MR to get this information.  This process is costly,
> thus we are trying to replace it with a simple capability check.

Which is the right thing to do, again if it is slow go talk to the
people who are making it slow..

> > Precompute it and use a topology file like everyone else?
> 
> This is an option but it seems like a big overkill for the simple
> check that we need.

Maybe there is some other place you can inject into the VM a 'this vm
does not support p2p' flag that userspace can see?

Leon has been working on some ACPI enhancements for this, perhaps that
new table could be distilled down to a single sysfs under an acpi
directory?

Jason