From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 1B375C25B77 for ; Wed, 22 May 2024 09:57:54 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:Content-Type: Content-Transfer-Encoding:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:In-Reply-To:From:References:To:Subject:Cc: MIME-Version:Date:Message-ID:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=Ce8KhOHE8xWu0a/ePLS0Ue68S84/O3FgFqN4jd+znLM=; b=V/5G3lKGJaKB2R 6+po0yiA9G0XgaxO85+WoAHpUjankWcHL5PZUgbM28Js1gx8hRfm+nhn1hHHnYQYUgDQSajY1TD7o xndlwvtjHpaFcwweDeEC2dY+57uHLq/n8ceu+QKopMc8jjWAPUBvrDVKR3LRl+kWwhIUdphFLLu1D rvFCramf3IBZxmhcM4dFwYmujsZ/MBI1opC0F0gUKDhc1k9Ok1dfx9s7rcD4AdoPgWV86fmzJP1eS UKV3JWsU8fR1JadWhdRk00R4bsl+vhmQwEDGjYKPJs7uV+jR16teHlNwVmubY7UPwlz9+zjWlcdHQ NBYlgZEKHBofaj5Y9CHA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1s9ijA-00000002ZJ4-3jfu; Wed, 22 May 2024 09:57:44 +0000 Received: from mgamail.intel.com ([198.175.65.13]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1s9ij8-00000002ZHb-3iYe for linux-arm-kernel@lists.infradead.org; Wed, 22 May 2024 09:57:44 +0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1716371862; x=1747907862; h=message-id:date:mime-version:cc:subject:to:references: from:in-reply-to:content-transfer-encoding; bh=dxxOdiXH9g1wuJ/3cBi6DHlb9NEtTWrT0Fmqvonz0d4=; b=kTXfM5I49Xo6wOzOrdpG9MbbRiKZRNmfmIGeXfQ7z5zYP4lNgp4uYszN ZY+RlfZJwRPbBamYs5KMhmLfQgIzcOHATg+YveGuuQLmUNhEtfvbMd41A XpcDjqJSASt+Hsu892dkBAtqBZffyZpAtyeoOfQldC19XWahqsz2i0Hqo CspwQdAp9esZqN7dWG8PdJaISEfAaVnKiCY4yqz2TjQ4UtvUDpvhJ8RaQ 7sbHNVXPYA95f5CR60O2nN6sieMUjeR7yLmfi2NUDnu61BOqsvyWPpO7v LjOWfB5skSc1Y+hfIh7pDu2b2zB4pD94L4hIxJfbetxYC+RKfETEIN9EP Q==; X-CSE-ConnectionGUID: g84B3CJIQDeAErGVkQ9HUw== X-CSE-MsgGUID: oVoPo4TgROOFgY1XJ6SaEA== X-IronPort-AV: E=McAfee;i="6600,9927,11079"; a="23742136" X-IronPort-AV: E=Sophos;i="6.08,179,1712646000"; d="scan'208";a="23742136" Received: from fmviesa005.fm.intel.com ([10.60.135.145]) by orvoesa105.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 22 May 2024 02:57:33 -0700 X-CSE-ConnectionGUID: yYYhyTfXTjiSS1atoHXAkw== X-CSE-MsgGUID: tHrzgwNSSZ+GNrt630I2bQ== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.08,179,1712646000"; d="scan'208";a="37627845" Received: from blu2-mobl.ccr.corp.intel.com (HELO [10.125.248.220]) ([10.125.248.220]) by fmviesa005-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 22 May 2024 02:57:28 -0700 Message-ID: Date: Wed, 22 May 2024 17:57:26 +0800 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Cc: baolu.lu@linux.intel.com, "will@kernel.org" , "robin.murphy@arm.com" , "suravee.suthikulpanit@amd.com" , "joro@8bytes.org" , "linux-kernel@vger.kernel.org" , "iommu@lists.linux.dev" , "linux-arm-kernel@lists.infradead.org" , "linux-tegra@vger.kernel.org" , "Liu, Yi L" , "eric.auger@redhat.com" , "vasant.hegde@amd.com" , "jon.grimm@amd.com" , "santosh.shukla@amd.com" , "Dhaval.Giani@amd.com" , "shameerali.kolothum.thodi@huawei.com" Subject: Re: [PATCH RFCv1 04/14] iommufd: Add struct iommufd_viommu and iommufd_viommu_ops To: "Tian, Kevin" , Jason Gunthorpe , Nicolin Chen References: <8610498e3fc00000e78bb9cef6fac9f6a54978a4.1712978212.git.nicolinc@nvidia.com> Content-Language: en-US From: Baolu Lu In-Reply-To: X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240522_025743_006895_5F0CF5C3 X-CRM114-Status: GOOD ( 25.90 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset="us-ascii"; Format="flowed" Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org On 2024/5/22 16:58, Tian, Kevin wrote: >> From: Jason Gunthorpe >> Sent: Tuesday, May 14, 2024 11:56 PM >> >> On Sun, May 12, 2024 at 08:34:02PM -0700, Nicolin Chen wrote: >>> On Sun, May 12, 2024 at 11:03:53AM -0300, Jason Gunthorpe wrote: >>>> On Fri, Apr 12, 2024 at 08:47:01PM -0700, Nicolin Chen wrote: >>>>> Add a new iommufd_viommu core structure to represent a vIOMMU >> instance in >>>>> the user space, typically backed by a HW-accelerated feature of an >> IOMMU, >>>>> e.g. NVIDIA CMDQ-Virtualization (an ARM SMMUv3 extension) and >> AMD Hardware >>>>> Accelerated Virtualized IOMMU (vIOMMU). >>>> I expect this will also be the only way to pass in an associated KVM, >>>> userspace would supply the kvm when creating the viommu. >>>> >>>> The tricky bit of this flow is how to manage the S2. It is necessary >>>> that the S2 be linked to the viommu: >>>> >>>> 1) ARM BTM requires the VMID to be shared with KVM >>>> 2) AMD and others need the S2 translation because some of the HW >>>> acceleration is done inside the guest address space >>>> >>>> I haven't looked closely at AMD but presumably the VIOMMU create will >>>> have to install the S2 into a DID or something? >>>> >>>> So we need the S2 to exist before the VIOMMU is created, but the >>>> drivers are going to need some more fixing before that will fully >>>> work. > Can you elaborate on this point? VIOMMU is a dummy container when > it's created and the association to S2 comes relevant only until when > VQUEUE is created inside and linked to a device? then there should be > a window in between allowing the userspace to configure S2. > > Not saying against setting S2 up before vIOMMU creation. Just want > to better understand the rationale here. > >>>> Does the nesting domain create need the viommu as well (in place of >>>> the S2 hwpt)? That feels sort of natural. >>> Yes, I had a similar thought initially: each viommu is backed by >>> a nested IOMMU HW, and a special HW accelerator like VCMDQ could >>> be treated as an extension on top of that. It might not be very >>> straightforward like the current design having vintf<->viommu and >>> vcmdq <-> vqueue though... >> vqueue should be considered a sub object of the viommu and hold a >> refcount on the viommu object for its lifetime. >> >>> In that case, we can then support viommu_cache_invalidate, which >>> is quite natural for SMMUv3. Yet, I recall Kevin said that VT-d >>> doesn't want or need that. >> Right, Intel currently doesn't need it, but I feel like everyone will >> need this eventually as the fast invalidation path is quite important. >> > yes, there is no need but I don't see any harm of preparing for such > extension on VT-d. Logically it's clearer, e.g. if we decide to move > device TLB invalidation to a separate uAPI then vIOMMU is certainly > a clearer object to carry it. and hardware extensions really looks like > optimization on software implementations. > > and we do need make a decision now, given if we make vIOMMU as > a generic object for all vendors it may have potential impact on > the user page fault support which Baolu is working on. the so-called > fault object will be contained in vIOMMU, which is software managed > on VT-d/SMMU but passed through on AMD. And probably we don't > need another handle mechanism in the attach path, suppose the > vIOMMU object already contains necessary information to find out > iommufd_object for a reported fault. Yes, if the vIOMMU object tracks all iommufd devices that it manages. Best regards, baolu _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel