PCI_P2PDMA_MAP_BUS_ADDR and PCIe ATS?

PCI_P2PDMA_MAP_BUS_ADDR and PCIe ATS?

[Thomas Hellström]

Hi!

I was looking at the way pci_p2pdma_distance() returns
PCI_P2PDMA_MAP_BUS_ADDR if devices sit beneath a local switch with ACS
disabled or not present.

The client then programs the bus address directly to perform p2p dma,
as handed out by the dma layer.

However if the device (p2pdma client) has IOMMU turned on and ATS
enabled, the device may or may not send a translation request to the
IOMMU. Wouldn't that fail returnint an IOMMU fault if the device is
programmed with the bus address rather than the IOVA?

So if ATS is enabled, shouldn't the device always be handed the IOVA to
avoid this? Granted, untranslated transactions will then take the
round-trip to the host bridge, but pre-translated transactions will
still benefit from the direct routing through the switch, although that
is not directly visible to the kernel p2pdma layer?

Does pci_p2pdma_distance() need to be client ATS-enabled aware?

Any input appreciated,
Thanks,
Thomas

Re: PCI_P2PDMA_MAP_BUS_ADDR and PCIe ATS?

[Jason Gunthorpe]

On Tue, Aug 18, 2026 at 11:33:07AM +0200, Thomas Hellström wrote:
> Hi!
> 
> I was looking at the way pci_p2pdma_distance() returns
> PCI_P2PDMA_MAP_BUS_ADDR if devices sit beneath a local switch with ACS
> disabled or not present.
> 
> The client then programs the bus address directly to perform p2p dma,
> as handed out by the dma layer.
> 
> However if the device (p2pdma client) has IOMMU turned on and ATS
> enabled, the device may or may not send a translation request to the
> IOMMU. Wouldn't that fail returnint an IOMMU fault if the device is
> programmed with the bus address rather than the IOVA?

There are many goofy configurations you can setup on PCIe, something
like this would be one of them :)

Linux only (barely) works with a few mainstream ones.
 
> So if ATS is enabled, shouldn't the device always be handed the IOVA to
> avoid this? 

No, Linux has no idea if the device will use ATS or not for a certain
request. ATS is not an all or nothing feature in PCI.

If you want this kind of configuration then set the ACS flags to
achieve it, or disable ATS in the device.

IMHO such a PCIe fabric is misconfigured. I can't see a reason why
anyone would want to have ACS flags allowing fast path and then Linux
setup to never supply an IOVA that can use it.

Jason