Re: [PATCH v4 1/5] PCI/P2PDMA: Don't enforce ACS check for device functions of Intel GPUs

["Christian König" <christian.koenig@amd.com>]

Hi guys,

On 22.09.25 08:59, Kasireddy, Vivek wrote:
> Hi Jason,
> 
>> Subject: Re: [PATCH v4 1/5] PCI/P2PDMA: Don't enforce ACS check for device
>> functions of Intel GPUs
>>
>> On Fri, Sep 19, 2025 at 06:22:45AM +0000, Kasireddy, Vivek wrote:
>>>> In this case messing with ACS is completely wrong. If the intention is
>>>> to convay a some kind of "private" address representing the physical
>>>> VRAM then you need to use a DMABUF mechanism to do that, not
>> deliver a
>>>> P2P address that the other side cannot access.
>>
>>> I think using a PCI BAR Address works just fine in this case because the Xe
>>> driver bound to PF on the Host can easily determine that it belongs to one
>>> of the VFs and translate it into VRAM Address.
>>
>> That isn't how the P2P or ACS mechansim works in Linux, it is about
>> the actual address used for DMA.
> Right, but this is not dealing with P2P DMA access between two random,
> unrelated devices. Instead, this is a special situation involving a GPU PF
> trying to access the VRAM of a VF that it provisioned and holds a reference
> on (note that the backing object for VF's VRAM is pinned by Xe on Host
> as part of resource provisioning). But it gets treated as regular P2P DMA
> because the exporters rely on pci_p2pdma_distance() or
> pci_p2pdma_map_type() to determine P2P compatibility.
> 
> In other words, I am trying to look at this problem differently: how can the
> PF be allowed to access the VF's resource that it provisioned, particularly
> when the VF itself requests the PF to access it and when a hardware path
> (via PCIe fabric) is not required/supported or doesn't exist at all?

Well what exactly is happening here? You have a PF assigned to the host and a VF passed through to a guest, correct?

And now the PF (from the host side) wants to access a BAR of the VF?

Regards,
Christian.

> 
> Furthermore, note that on a server system with a whitelisted PCIe upstream
> bridge, this quirk would not be needed at all as pci_p2pdma_map_type()
> would not have failed and this would have been a purely Xe driver specific
> problem to solve that would have required just the translation logic and no
> further changes anywhere. But my goal is to fix it across systems like
> workstations/desktops that do not typically have whitelisted PCIe upstream
> bridges.
> 
>>
>> You can't translate a dma_addr_t to anything in the Xe PF driver
>> anyhow, once it goes through the IOMMU the necessary information is lost.
> Well, I already tested this path (via IOMMU, with your earlier vfio-pci +
> dmabuf patch that used dma_map_resource() and also with Leon's latest
> version) and found that I could still do the translation in the Xe PF driver
> after first calling iommu_iova_to_phys().
> 
>> This is a fundamentally broken design to dma map something and
>> then try to reverse engineer the dma_addr_t back to something with
>> meaning.
> IIUC, I don't think this is a new or radical idea. I think the concept is slightly
> similar to using bounce buffers to address hardware DMA limitations except
> that there are no memory copies and the CPU is not involved. And, I don't see
> any other way to do this because I don't believe the exporter can provide a
> DMA address that the importer can use directly without any translation, which
> seems unavoidable in this case.
> 
>>
>>>> Christian told me dmabuf has such a private address mechanism, so
>>>> please figure out a way to use it..
>>>
>>> Even if such as a mechanism exists, we still need a way to prevent
>>> pci_p2pdma_map_type() from failing when invoked by the exporter (vfio-
>> pci).
>>> Does it make sense to move this quirk into the exporter?
>>
>> When you export a private address through dmabuf the VFIO exporter
>> will not call p2pdma paths when generating it.
> I have cc'd Christian and Simona. Hopefully, they can help explain how
> the dmabuf private address mechanism can be used to address my
> use-case. And, I sincerely hope that it will work, otherwise I don't see
> any viable path forward for what I am trying to do other than using this
> quirk and translation. Note that the main reason why I am doing this
> is because I am seeing at-least ~35% performance gain when running
> light 3D/Gfx workloads.
> 
>>
>>> Also, AFAICS, translating BAR Address to VRAM Address can only be
>>> done by the Xe driver bound to PF because it has access to provisioning
>>> data. In other words, vfio-pci would not be able to share any other
>>> address other than the BAR Address because it wouldn't know how to
>>> translate it to VRAM Address.
>>
>> If you have a vfio varient driver then the VF vfio driver could call
>> the Xe driver to create a suitable dmabuf using the private
>> addressing. This is probably what is required here if this is what you
>> are trying to do.
> Could this not be done via the vendor agnostic vfio-pci (+ dmabuf) driver
> instead of having to use a separate VF/vfio variant driver?
> 
>>
>>>> No, don't, it is completely wrong to mess with ACS flags for the
>>>> problem you are trying to solve.
>>
>>> But I am not messing with any ACS flags here. I am just adding a quirk to
>>> sidestep the ACS enforcement check given that the PF to VF access does
>>> not involve the PCIe fabric in this case.
>>
>> Which is completely wrong. These are all based on fabric capability,
>> not based on code in drivers to wrongly "translate" the dma_addr_t.
> I am not sure why you consider translation to be wrong in this case
> given that it is done by a trusted entity (Xe PF driver) that is bound to
> the GPU PF and provisioned the resource that it is trying to access.
> What limitations do you see with this approach?
> 
> Also, the quirk being added in this patch is indeed meant to address a
> specific case (GPU PF to VF access) to workaround a potential hardware
> limitation (non-existence of a direct PF to VF DMA access path via the
> PCIe fabric). Isn't that one of the main ideas behind using quirks -- to
> address hardware limitations?
> 
> Thanks,
> Vivek
> 
>>
>> Jason