PCI_P2PDMA_MAP_BUS_ADDR and PCIe ATS?

PCI_P2PDMA_MAP_BUS_ADDR and PCIe ATS?

[Thomas Hellström]

Hi!

I was looking at the way pci_p2pdma_distance() returns
PCI_P2PDMA_MAP_BUS_ADDR if devices sit beneath a local switch with ACS
disabled or not present.

The client then programs the bus address directly to perform p2p dma,
as handed out by the dma layer.

However if the device (p2pdma client) has IOMMU turned on and ATS
enabled, the device may or may not send a translation request to the
IOMMU. Wouldn't that fail returnint an IOMMU fault if the device is
programmed with the bus address rather than the IOVA?

So if ATS is enabled, shouldn't the device always be handed the IOVA to
avoid this? Granted, untranslated transactions will then take the
round-trip to the host bridge, but pre-translated transactions will
still benefit from the direct routing through the switch, although that
is not directly visible to the kernel p2pdma layer?

Does pci_p2pdma_distance() need to be client ATS-enabled aware?

Any input appreciated,
Thanks,
Thomas

Re: PCI_P2PDMA_MAP_BUS_ADDR and PCIe ATS?

[Jason Gunthorpe]

On Tue, Aug 18, 2026 at 11:33:07AM +0200, Thomas Hellström wrote:
> Hi!
> 
> I was looking at the way pci_p2pdma_distance() returns
> PCI_P2PDMA_MAP_BUS_ADDR if devices sit beneath a local switch with ACS
> disabled or not present.
> 
> The client then programs the bus address directly to perform p2p dma,
> as handed out by the dma layer.
> 
> However if the device (p2pdma client) has IOMMU turned on and ATS
> enabled, the device may or may not send a translation request to the
> IOMMU. Wouldn't that fail returnint an IOMMU fault if the device is
> programmed with the bus address rather than the IOVA?

There are many goofy configurations you can setup on PCIe, something
like this would be one of them :)

Linux only (barely) works with a few mainstream ones.
 
> So if ATS is enabled, shouldn't the device always be handed the IOVA to
> avoid this? 

No, Linux has no idea if the device will use ATS or not for a certain
request. ATS is not an all or nothing feature in PCI.

If you want this kind of configuration then set the ACS flags to
achieve it, or disable ATS in the device.

IMHO such a PCIe fabric is misconfigured. I can't see a reason why
anyone would want to have ACS flags allowing fast path and then Linux
setup to never supply an IOVA that can use it.

Jason

Re: PCI_P2PDMA_MAP_BUS_ADDR and PCIe ATS?

[Thomas Hellström]

On Tue, 2026-08-18 at 09:43 -0300, Jason Gunthorpe wrote:
> On Tue, Aug 18, 2026 at 11:33:07AM +0200, Thomas Hellström wrote:
> > Hi!
> > 
> > I was looking at the way pci_p2pdma_distance() returns
> > PCI_P2PDMA_MAP_BUS_ADDR if devices sit beneath a local switch with
> > ACS
> > disabled or not present.
> > 
> > The client then programs the bus address directly to perform p2p
> > dma,
> > as handed out by the dma layer.
> > 
> > However if the device (p2pdma client) has IOMMU turned on and ATS
> > enabled, the device may or may not send a translation request to
> > the
> > IOMMU. Wouldn't that fail returnint an IOMMU fault if the device is
> > programmed with the bus address rather than the IOVA?
> 
> There are many goofy configurations you can setup on PCIe, something
> like this would be one of them :)
> 
> Linux only (barely) works with a few mainstream ones.
>  
> > So if ATS is enabled, shouldn't the device always be handed the
> > IOVA to
> > avoid this? 
> 
> No, Linux has no idea if the device will use ATS or not for a certain
> request. ATS is not an all or nothing feature in PCI.
> 
> If you want this kind of configuration then set the ACS flags to
> achieve it, or disable ATS in the device.
> 
> IMHO such a PCIe fabric is misconfigured. I can't see a reason why
> anyone would want to have ACS flags allowing fast path and then Linux
> setup to never supply an IOVA that can use it.

From my understanding, ATS translated requests would use the fast-path
as soon as a translation is established in the device, without the BUS
address being explicitly programmed. ACS redirect would block that. 

Admittedly untranslated requests would take the detour around the host
bridge, but a GPU device that would otherwise enable ATS by default to
offload the IOMMU TLB would outright fail with the current code with
little clue what was going on.

So wouldn't it be better to be safe than sorry and return a host bridge
map and make something usable and potentially fast out of something
currently unusable?

The only way I can see one could support the fast-path for both
untranslated and translated request would be to set up a 1:1 IOVA for
the particular mapping.

/Thomas









> 
> Jason

Re: PCI_P2PDMA_MAP_BUS_ADDR and PCIe ATS?

[Jason Gunthorpe]

On Tue, Aug 18, 2026 at 05:20:51PM +0200, Thomas Hellström wrote:

> From my understanding, ATS translated requests would use the fast-path
> as soon as a translation is established in the device, without the BUS
> address being explicitly programmed. ACS redirect would block that. 

Oh it's super complex. There are seperate ACS bits for translated and
untranslated requests. You need a PCI expert to design your fabric and
devices to get a working system unfortunately.

If you use ATS and don't set the ACS to route all translations to host
then your IOVA space is no longer uniform, you have different results
depending on which fabric route you take. That's a pretty unreasonable
fabric design, and can't work securely with virtualization.

> So wouldn't it be better to be safe than sorry and return a host bridge
> map and make something usable and potentially fast out of something
> currently unusable?

What we have today works for lots of devices and if you change it like
this then you'll break existing users.

Probably what you want to do is add a flag that the device is going to
use ATS for the P2P DMA. Then we can force setup an IOVA and check the
fabric path for translated request routing to make sure it will work.

And yes, there is an existing issue with DMABUF not being able to
negotiate these little pathing details with the importer when the
exporter forms the scatterlist..

Jason

Re: PCI_P2PDMA_MAP_BUS_ADDR and PCIe ATS?

[Thomas Hellström]

On Tue, 2026-08-18 at 16:03 -0300, Jason Gunthorpe wrote:
> On Tue, Aug 18, 2026 at 05:20:51PM +0200, Thomas Hellström wrote:
> 
> > From my understanding, ATS translated requests would use the fast-
> > path
> > as soon as a translation is established in the device, without the
> > BUS
> > address being explicitly programmed. ACS redirect would block that.
> 
> Oh it's super complex. There are seperate ACS bits for translated and
> untranslated requests. You need a PCI expert to design your fabric
> and
> devices to get a working system unfortunately.
> 
> If you use ATS and don't set the ACS to route all translations to
> host
> then your IOVA space is no longer uniform, you have different results
> depending on which fabric route you take. That's a pretty
> unreasonable
> fabric design, and can't work securely with virtualization.
> 
> > So wouldn't it be better to be safe than sorry and return a host
> > bridge
> > map and make something usable and potentially fast out of something
> > currently unusable?
> 
> What we have today works for lots of devices and if you change it
> like
> this then you'll break existing users.

I guess if there are setups with devices with PCIe ATS enabled but only
ever sending untranslated requests, indeed they would see a performance
regression with this approach.

> 
> Probably what you want to do is add a flag that the device is going
> to
> use ATS for the P2P DMA. Then we can force setup an IOVA and check
> the
> fabric path for translated request routing to make sure it will work.

Yes, sounds like a way forward. To be honest, we don't have an existing
use-case yet, but taking the union of how we want to configure our GPUs
moving forward and the support requests we see from p2pdma users, this
is probably something we are going to hit sooner than later.

> 
> And yes, there is an existing issue with DMABUF not being able to
> negotiate these little pathing details with the importer when the
> exporter forms the scatterlist..

Yes, it was when looking into that I actually stumbled upon the ATS
issue as well. I recall you and Christian have touched upon this in the
past. What is an acceptable way forward here? A dma_map_resource_p2p()
interface that takes into account the mapping type?

Thanks,
Thomas


> 
> Jason

Re: PCI_P2PDMA_MAP_BUS_ADDR and PCIe ATS?

[Jason Gunthorpe]

On Wed, Aug 19, 2026 at 08:42:10AM +0200, Thomas Hellström wrote:

> > Probably what you want to do is add a flag that the device is going
> > to
> > use ATS for the P2P DMA. Then we can force setup an IOVA and check
> > the
> > fabric path for translated request routing to make sure it will work.
> 
> Yes, sounds like a way forward. To be honest, we don't have an existing
> use-case yet, but taking the union of how we want to configure our GPUs
> moving forward and the support requests we see from p2pdma users, this
> is probably something we are going to hit sooner than later.

I will say we've had to deploy alot of experts to design our systems
and it is really hard to get something that actually works :|

> > And yes, there is an existing issue with DMABUF not being able to
> > negotiate these little pathing details with the importer when the
> > exporter forms the scatterlist..
> 
> Yes, it was when looking into that I actually stumbled upon the ATS
> issue as well. I recall you and Christian have touched upon this in the
> past. What is an acceptable way forward here? A dma_map_resource_p2p()
> interface that takes into account the mapping type?

IDK, it is painful. There is alot of stuff coming that the importer
will need to customize when performing the mapping. Locking the DMA
mapping process in the exporter is not helpful.  For example your
device might need to say 'I will always use ATS', while mlx5 would
prefer 'Should I disable ATS for this mapping'.

Christian seems unrelenting on this point so I don't really have a
good solution beyond some endless list of single user flags.

I wanted to rebase and revise the mapping type series but I've been
just buried in security things, patch reviews and travel lately. Maybe
I will get there after this merge window

Jason