From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 08058C3DA4A for ; Mon, 12 Aug 2024 02:25:28 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Transfer-Encoding: Content-Type:In-Reply-To:From:References:To:Subject:Cc:MIME-Version:Date: Message-ID:Reply-To:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=4cbGV5hDPFhQgs/iKa4S/6EN64s6B2ySJxpNT4V6dhM=; b=maB3kU3l/Pe/NTThaobAlL8r9A bsW+JYiUIZ84keYn7WOnoOlomm5AxlnPHgYGjuQ5ntCXZnsziO4xKwvKBgTk1QCS8hhdB3tbDI/Q+ qLKcTFIvcrIX+Zr0djQskVzMnj4k66sMbLmCe7YVZvj2YwGkfN2ZUFYAwPetbXpmIfLhyC23r/Nr/ x7vDXi0S2/Bmn+h3ZodF3qbKj8OUcERQ41FEFA4HsXRJQH9MAkg5j+I53XqMFBnj6FwI5YiQgH+vl 8IxGyMtLqmoFdSXsIGBIW8foKjCjzhID1hOwYeZ9Ut+QUoiIoyzrAvfnqKA4N7Wfz1hov/2G6wrGE ZeMUkLGQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1sdKkG-0000000Gi3u-3JGR; Mon, 12 Aug 2024 02:25:16 +0000 Received: from mgamail.intel.com ([192.198.163.9]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1sdKj5-0000000Ghq6-3vWG for linux-arm-kernel@lists.infradead.org; Mon, 12 Aug 2024 02:24:05 +0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1723429444; x=1754965444; h=message-id:date:mime-version:cc:subject:to:references: from:in-reply-to:content-transfer-encoding; bh=FqGg5DIw49XQ7Uc49eBDwirJ08MyI6X0CPTIL7COrkM=; b=NavN4KGDtPDTn+HJcbhlA45OXBHWkLWpe52EeBm7YJ1EcT5fbTvH2aVf wecgEsbrsTRATRIh/hXgXlq5gI0XHjQ/GrkK5sG0tGYYN5QQd1IwuuS/M BBcVrWCDdtiGX8sKL9Ig+bHoyMWT7eVv99uL3YjOouq0zey3xEll2m1ip 8F1wBDhmWoh28gHu83A61qdPPgiKlSM+veOYUzehkE7MPBS2dVJdVm5Pr QLbdTOOhfpfj9pViUqWlzLF03+/uzgBguyfxiwLx6KALImgCNb7GlBtGE IvgJaHnL+drd58wSvnzH5JC/qarRO/bFRpY833SoRWTBmmwXE9QrECZFo g==; X-CSE-ConnectionGUID: pMqcKeaJTo6heutd4+hzbA== X-CSE-MsgGUID: GeND8CmoSUarv1KGZpNQFw== X-IronPort-AV: E=McAfee;i="6700,10204,11161"; a="32197518" X-IronPort-AV: E=Sophos;i="6.09,282,1716274800"; d="scan'208";a="32197518" Received: from orviesa009.jf.intel.com ([10.64.159.149]) by fmvoesa103.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 11 Aug 2024 19:24:03 -0700 X-CSE-ConnectionGUID: GYcgJn7UReyXGjarYyDrkA== X-CSE-MsgGUID: 7I8pB5x5QceZjJsh9TtACw== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.09,282,1716274800"; d="scan'208";a="58020573" Received: from allen-box.sh.intel.com (HELO [10.239.159.127]) ([10.239.159.127]) by orviesa009.jf.intel.com with ESMTP; 11 Aug 2024 19:24:00 -0700 Message-ID: Date: Mon, 12 Aug 2024 10:20:32 +0800 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Cc: baolu.lu@linux.intel.com, patches@lists.linux.dev Subject: Re: [PATCH] iommu: Allow ATS to work on VFs when the PF uses IDENTITY To: Jason Gunthorpe , Bjorn Helgaas , David Woodhouse , iommu@lists.linux.dev, Joerg Roedel , kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-pci@vger.kernel.org, Robin Murphy , Suravee Suthikulpanit , Will Deacon References: <0-v1-0fb4d2ab6770+7e706-ats_vf_jgg@nvidia.com> Content-Language: en-US From: Baolu Lu In-Reply-To: <0-v1-0fb4d2ab6770+7e706-ats_vf_jgg@nvidia.com> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240811_192404_024178_BFA860AD X-CRM114-Status: GOOD ( 22.79 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org On 8/8/24 2:19 AM, Jason Gunthorpe wrote: > PCI ATS has a global Smallest Translation Unit field that is located in > the PF but shared by all of the VFs. > > The expectation is that the STU will be set to the root port's global STU > capability which is driven by the IO page table configuration of the iommu > HW. Today it becomes set when the iommu driver first enables ATS. > > Thus, to enable ATS on the VF, the PF must have already had the correct > STU programmed, even if ATS is off on the PF. > > Unfortunately the PF only programs the STU when the PF enables ATS. The > iommu drivers tend to leave ATS disabled when IDENTITY translation is > being used. > > Thus we can get into a state where the PF is setup to use IDENTITY with > the DMA API while the VF would like to use VFIO with a PAGING domain and > have ATS turned on. This fails because the PF never loaded a PAGING domain > and so it never setup the STU, and the VF can't do it. > > The simplest solution is to have the iommu driver set the ATS STU when it > probes the device. This way the ATS STU is loaded immediately at boot time > to all PFs and there is no issue when a VF comes to use it. > > Add a new call pci_prepare_ats() which should be called by iommu drivers > in their probe_device() op for every PCI device if the iommu driver > supports ATS. This will setup the STU based on whatever page size > capability the iommu HW has. > > Signed-off-by: Jason Gunthorpe > --- > drivers/iommu/amd/iommu.c | 3 ++ > drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 6 ++++ > drivers/iommu/intel/iommu.c | 1 + > drivers/pci/ats.c | 33 +++++++++++++++++++++ > include/linux/pci-ats.h | 1 + > 5 files changed, 44 insertions(+) For Intel VT-d, Reviewed-by: Lu Baolu Thanks, baolu