From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 5557FC52D7B for ; Tue, 13 Aug 2024 08:54:22 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:In-Reply-To:Content-Type: MIME-Version:References:Message-ID:Subject:Cc:To:From:Date:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=ekHJjFkL1mIU9h8XFWVaUO3kJo31VqjD0RAZHJBeqU0=; b=dcqQlOsSNUBzfsU+A5rsT1E6wl P2wEXy0SXJyi1mfy8v/sTQBElKq2pY01RwIfmMQorJ6xeP/Fkb0bawg/6uL9f6mP1cynYzu0imJw9 v/zQqyHiAwZyYRYgTs9yZed+SR8OxSkSbuTp7zBOcrnumWAPjw1U2UPY77MzOJL+oYlfCepOJ2ru6 rTOwH1K4m4FRaVJ8DtfxGJjUZ3lKhQhGlFzgZG3wshoQIE0SuqLEMZyKSNUGmrsHZJx9n6brvjUqp yliHd0bvQ0J1wpeENhIid5maRwU484wVN4rpptrWmEOo/e59PSgo3aeNSXdCZwJheIgZCT4uPTzw8 4Lm6ZCjw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1sdnIC-000000031bv-3XJR; Tue, 13 Aug 2024 08:54:12 +0000 Received: from mail.8bytes.org ([2a01:238:42d9:3f00:e505:6202:4f0c:f051]) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1sdnGX-00000003135-37BJ for linux-arm-kernel@lists.infradead.org; Tue, 13 Aug 2024 08:52:31 +0000 Received: from 8bytes.org (pd9fe9dd8.dip0.t-ipconnect.de [217.254.157.216]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange ECDHE (P-256) server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mail.8bytes.org (Postfix) with ESMTPSA id 06BAA2A50EF; Tue, 13 Aug 2024 10:52:24 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=8bytes.org; s=default; t=1723539144; bh=YMj67aL6wIKEzLDPFc93NrmMN0mrzP4ToduOyz628Qs=; h=Date:From:To:Cc:Subject:References:In-Reply-To:From; b=ieUQMxe0HigcQJaqZh3zZsFiLkq0a1eBi2t5Pzt3YFymmi01Ww7+zL7cQPyhBKe00 Tv+IxDclbBH33noPMUZuJQ9t6T0YDDv95L+RRshXSwKEhqyZjHhC4pCMsJQk16HeNA kOcqb6zF9TORWNX5c5ULAvMValaCgFGYUf8E6LEOsBK/MDe3BTwb3QcHe9mwiczHJD AbrCE2QzIFnVacFRHILEeAHiQxJO3LUqMXn/qlZOjbttijsa3YQO7SqIeQ4SAAOvKT DAoIAy9lpwIR30CvgRCaqf23OwAuncjTWDSDmSFFCObdQAUuEbBXuTYunB7Fu9UbMc kv2dvXaUOmdEw== Date: Tue, 13 Aug 2024 10:52:22 +0200 From: Joerg Roedel To: Jason Gunthorpe Cc: Lu Baolu , Bjorn Helgaas , David Woodhouse , iommu@lists.linux.dev, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-pci@vger.kernel.org, Robin Murphy , Suravee Suthikulpanit , Will Deacon , patches@lists.linux.dev Subject: Re: [PATCH] iommu: Allow ATS to work on VFs when the PF uses IDENTITY Message-ID: References: <0-v1-0fb4d2ab6770+7e706-ats_vf_jgg@nvidia.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <0-v1-0fb4d2ab6770+7e706-ats_vf_jgg@nvidia.com> X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240813_015230_054562_46A0FB4F X-CRM114-Status: GOOD ( 23.48 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org On Wed, Aug 07, 2024 at 03:19:20PM -0300, Jason Gunthorpe wrote: > PCI ATS has a global Smallest Translation Unit field that is located in > the PF but shared by all of the VFs. > > The expectation is that the STU will be set to the root port's global STU > capability which is driven by the IO page table configuration of the iommu > HW. Today it becomes set when the iommu driver first enables ATS. > > Thus, to enable ATS on the VF, the PF must have already had the correct > STU programmed, even if ATS is off on the PF. > > Unfortunately the PF only programs the STU when the PF enables ATS. The > iommu drivers tend to leave ATS disabled when IDENTITY translation is > being used. > > Thus we can get into a state where the PF is setup to use IDENTITY with > the DMA API while the VF would like to use VFIO with a PAGING domain and > have ATS turned on. This fails because the PF never loaded a PAGING domain > and so it never setup the STU, and the VF can't do it. > > The simplest solution is to have the iommu driver set the ATS STU when it > probes the device. This way the ATS STU is loaded immediately at boot time > to all PFs and there is no issue when a VF comes to use it. > > Add a new call pci_prepare_ats() which should be called by iommu drivers > in their probe_device() op for every PCI device if the iommu driver > supports ATS. This will setup the STU based on whatever page size > capability the iommu HW has. > > Signed-off-by: Jason Gunthorpe > --- > drivers/iommu/amd/iommu.c | 3 ++ > drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 6 ++++ > drivers/iommu/intel/iommu.c | 1 + > drivers/pci/ats.c | 33 +++++++++++++++++++++ > include/linux/pci-ats.h | 1 + > 5 files changed, 44 insertions(+) Applied, thanks.