Re: [PATCH v3 1/1] PCI: Add translated request only flag for pci_enable_pasid()

[Bjorn Helgaas <helgaas@kernel.org>]

On Tue, Jan 31, 2023 at 08:25:05PM +0800, Baolu Lu wrote:
> On 2023/1/31 2:38, Bjorn Helgaas wrote:
> > PCIe r6.0, sec 6.20.1:
> > 
> >    A Function is not permitted to generate Requests using Translated
> >    Addresses and a PASID unless both PASID Enable and Translated
> >    Requests with PASID Enable are Set.
> > 
> > You want AMD graphics devices to do DMA with translated addresses and
> > PASID, right?  pci_enable_pasid() sets PASID Enable
> > (PCI_PASID_CTRL_ENABLE), but I don't see where "Translated Requests
> > with PASID Enable" is set.  We don't even have a #define for it.
> > 
> > I would think we should check "Translated Requests with PASID
> > Supported" before setting "Translated Requests with PASID Enable",
> > too?
> 
> This seems to be an ECN for PCIe 5.x:
> 
> https://members.pcisig.com/wg/PCI-SIG/document/14929
> 
> What I read from this ECN is that,
> 
> With this ECN, translated memory requests for PASIDs are not allowed to
> carry a PASID prefix if "Translated Requests with PASID Enabled" is not
> set. It does not mean whether the device can generate translated memory
> requests for PASID, but whether the memory request can carry a PASID
> prefix.

My assumption that "you want AMD graphics devices to do DMA with
translated addresses and PASID" was wrong.

Per Jason [1], it sounds like the AMD GPU generates Translation
Requests (sec 10.2.2) with a PASID.  The GPU will cache the translated
address from the Translation Completion in its local ATC, and will do
DMA (MemRd/Wr) with that translated address but *without* PASID
prefixes.

That makes sense because (PASID, IOVA) maps to a translated address,
e.g., a a CPU physical address, and the GPU can DMA to that address
directly without needing the PASID.

Bjorn

[1] https://lore.kernel.org/r/Y9nQK9P3HOxEeZ4U@nvidia.com