From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4AD578483 for ; Wed, 1 Feb 2023 16:58:54 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id C4D7CC433EF; Wed, 1 Feb 2023 16:58:53 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1675270734; bh=9Cau6PyVE2GtVxFGlwOHuWzyHir/3fJpWhTQRdW+NkI=; h=Date:From:To:Cc:Subject:In-Reply-To:From; b=ipvF/xaMGsJj1RPkbilfTBacfCIfc+CnK8ILbhc7Rzg0XOfdyBC0z9uLdGlLdaRMg lANpe4Ci0BO9cv/Wo3kqTttJUkQvb0L7266E6398ztRx0NBbw1CYRA+1VPn32Pl1Ng Zd6H2FDYHAcFx1Uh1FJARqDqD9XTG001++RAWsRWmt9RIL6gCsJ96aovqsR6xmc24a SDel2H0tkqxQzwgMpt+QvNNDRs64Z2IZv2AVtzDMxUO32boZAzU2msep7wcbN4UorQ 7+EvjNSNGRSmxqRaC8AkMr7wjP8MqLseRGZ6vORyEkoDvr/xHE6KRJ/XF+RF5inE2A 7dhJIj7SSsBVw== Date: Wed, 1 Feb 2023 10:58:52 -0600 From: Bjorn Helgaas To: Baolu Lu Cc: Bjorn Helgaas , Joerg Roedel , Matt Fagnani , Christian =?iso-8859-1?Q?K=F6nig?= , Jason Gunthorpe , Kevin Tian , Vasant Hegde , Tony Zhu , linux-pci@vger.kernel.org, iommu@lists.linux.dev, linux-kernel@vger.kernel.org Subject: Re: [PATCH v3 1/1] PCI: Add translated request only flag for pci_enable_pasid() Message-ID: <20230201165852.GA1775862@bhelgaas> Precedence: bulk X-Mailing-List: iommu@lists.linux.dev List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <8cb478bf-db44-0a8f-8521-771529ca0706@linux.intel.com> On Tue, Jan 31, 2023 at 08:25:05PM +0800, Baolu Lu wrote: > On 2023/1/31 2:38, Bjorn Helgaas wrote: > > PCIe r6.0, sec 6.20.1: > > > > A Function is not permitted to generate Requests using Translated > > Addresses and a PASID unless both PASID Enable and Translated > > Requests with PASID Enable are Set. > > > > You want AMD graphics devices to do DMA with translated addresses and > > PASID, right? pci_enable_pasid() sets PASID Enable > > (PCI_PASID_CTRL_ENABLE), but I don't see where "Translated Requests > > with PASID Enable" is set. We don't even have a #define for it. > > > > I would think we should check "Translated Requests with PASID > > Supported" before setting "Translated Requests with PASID Enable", > > too? > > This seems to be an ECN for PCIe 5.x: > > https://members.pcisig.com/wg/PCI-SIG/document/14929 > > What I read from this ECN is that, > > With this ECN, translated memory requests for PASIDs are not allowed to > carry a PASID prefix if "Translated Requests with PASID Enabled" is not > set. It does not mean whether the device can generate translated memory > requests for PASID, but whether the memory request can carry a PASID > prefix. My assumption that "you want AMD graphics devices to do DMA with translated addresses and PASID" was wrong. Per Jason [1], it sounds like the AMD GPU generates Translation Requests (sec 10.2.2) with a PASID. The GPU will cache the translated address from the Translation Completion in its local ATC, and will do DMA (MemRd/Wr) with that translated address but *without* PASID prefixes. That makes sense because (PASID, IOVA) maps to a translated address, e.g., a a CPU physical address, and the GPU can DMA to that address directly without needing the PASID. Bjorn [1] https://lore.kernel.org/r/Y9nQK9P3HOxEeZ4U@nvidia.com