From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id AE0EFCE7B02 for ; Fri, 6 Sep 2024 11:15:44 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:In-Reply-To: Content-Transfer-Encoding:Content-Type:MIME-Version:References:Message-ID: Subject:Cc:To:From:Date:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=1kfdlvi3htUX5t/uzYteyfrsdGm7wcbhZeedCN8JXH8=; b=DSo3PmqV5e/SHF+ekAYoCLEc+3 UkXNwj98pdNoYVvJx1CO9NOTUdNYdkullaQ5rpK1Re/fNHov/+QW9qrtaysBCHpKCF3OCyQGz9dvP sm2hfSF1xS5veipF1Aj3SSIZW4CcoemsS7N+Q0uCl2xiZLwJpvr68sLWF65V0B0atQVXew8JoRyE3 AXxwCZDVAPaMMTnmgrgBeeJK2Z6sdOB0z5wMKQqMjGwiYLwijpeZ3u6uPueCzHY17Q+xTGxtLHwYn 9tFEWYUYVEmMb6XLOz/Sx6jXjse7UYrypc1V5Kl0ivvytb9N4np8Z/vk7eUwihN+uBzw9wXh+5K1K EFKiu/OA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1smWwA-0000000BvGs-44C6; Fri, 06 Sep 2024 11:15:34 +0000 Received: from mail-wm1-x32d.google.com ([2a00:1450:4864:20::32d]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1smWol-0000000BtmP-3JCD for linux-arm-kernel@lists.infradead.org; Fri, 06 Sep 2024 11:07:57 +0000 Received: by mail-wm1-x32d.google.com with SMTP id 5b1f17b1804b1-42ba8928f1dso35445e9.1 for ; Fri, 06 Sep 2024 04:07:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1725620873; x=1726225673; darn=lists.infradead.org; h=in-reply-to:content-transfer-encoding:content-disposition :mime-version:references:message-id:subject:cc:to:from:date:from:to :cc:subject:date:message-id:reply-to; bh=1kfdlvi3htUX5t/uzYteyfrsdGm7wcbhZeedCN8JXH8=; b=w8f2VOy9y1tQQERVP3oWKGYx/bjMJCRPOXUf9Yn325L/Fh/iwYFQEmva6mQGyVRdno A/64CRcCckgmDP2rzffraf1SeplWhv7ifNf2fAHWSj0HeoWABRe8dwh9zsRt0eXTIpkW 9TfSyPnS7Mj2QBVCx6Tg9GWO7sEQ6J4xuwKxZ4E59rcUZ1GfoXDOu+zBbuuRYwfr1hpc sdKZXeQA6DP2V4Tgli554kAHLNZ5sCa/2pexTPnVWRjV3nj8fsTBxJLY3ZVBq0gGMihY jXY6/2q9C1dBAvW9e4S1STsXbGkVAkA+mzeh5/ikdEfGSdefzD6PI5B3BXrO9NmIm3lC vTHg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1725620873; x=1726225673; h=in-reply-to:content-transfer-encoding:content-disposition :mime-version:references:message-id:subject:cc:to:from:date :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=1kfdlvi3htUX5t/uzYteyfrsdGm7wcbhZeedCN8JXH8=; b=D1XyvEGK8imYOsDcP/yDOB0yZJr5mLunkiO84ehTykgAqehDvDTq/iMIbG81Ovdfja Pya/WEjRVvs3OPNm7CHVo1pmu17RHcCGf0/PPDq7giToAEYDgoeoHtxJC4pVDFuiB7rw xuYbu4cAk/pUxX5a0dW/uu8DS2ZDlfwBApAPh8U55Gez9L6Y5d3NetGZ2WjFQhVsUlOL PQZRVkIt9ugjFy6cfmkO3RAVT4/u282RsH1UMR28/LHt09epoajJR5q1Tok6T5tretca ECaZgoTLoByMNQLj9SKYWdCF9rfZAAJBI3keLPh4c8XG51IP1TvllxvseocBNglx3kIm L1dw== X-Forwarded-Encrypted: i=1; AJvYcCV8os9BpXg57rEYwsi5vptXj7WjjHwAed+ZSszai7w4QMjX0LzwZpe8+zkI3fBhUBvEVvuMhTvgDvMuuskRpLai@lists.infradead.org X-Gm-Message-State: AOJu0Yx3e3zNMQftNJf0eNxYZQ6Un8S4V54bH9FZufPzcOatjxCXuViu jItoVGPTTyGUm3DAex/AMtfTGqpJuojZzHrnkXPCmdP+LlynPqnEK7eSBTnE9w== X-Google-Smtp-Source: AGHT+IErQWOce1aTO7QUQS6iDDj2x0dGhvkowMBDS1btzgPV98ts3aP/uP28acSxLiOr1pFciOVHDg== X-Received: by 2002:a05:600c:b93:b0:42b:8ff7:bee2 with SMTP id 5b1f17b1804b1-42ca05fbaccmr693485e9.5.1725620872291; Fri, 06 Sep 2024 04:07:52 -0700 (PDT) Received: from google.com (109.36.187.35.bc.googleusercontent.com. [35.187.36.109]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-374c2ffe062sm15554875f8f.29.2024.09.06.04.07.51 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 06 Sep 2024 04:07:51 -0700 (PDT) Date: Fri, 6 Sep 2024 11:07:47 +0000 From: Mostafa Saleh To: Jason Gunthorpe Cc: acpica-devel@lists.linux.dev, Hanjun Guo , iommu@lists.linux.dev, Joerg Roedel , Kevin Tian , kvm@vger.kernel.org, Len Brown , linux-acpi@vger.kernel.org, linux-arm-kernel@lists.infradead.org, Lorenzo Pieralisi , "Rafael J. Wysocki" , Robert Moore , Robin Murphy , Sudeep Holla , Will Deacon , Alex Williamson , Eric Auger , Jean-Philippe Brucker , Moritz Fischer , Michael Shavit , Nicolin Chen , patches@lists.linux.dev, Shameerali Kolothum Thodi Subject: Re: [PATCH v2 8/8] iommu/arm-smmu-v3: Support IOMMU_DOMAIN_NESTED Message-ID: References: <0-v2-621370057090+91fec-smmuv3_nesting_jgg@nvidia.com> <8-v2-621370057090+91fec-smmuv3_nesting_jgg@nvidia.com> <20240830170426.GV3773488@nvidia.com> <20240903003022.GF3773488@nvidia.com> <20240903235532.GJ3773488@nvidia.com> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <20240903235532.GJ3773488@nvidia.com> X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240906_040755_863980_C08AC6DB X-CRM114-Status: GOOD ( 65.38 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org On Tue, Sep 03, 2024 at 08:55:32PM -0300, Jason Gunthorpe wrote: > On Tue, Sep 03, 2024 at 09:00:32AM +0000, Mostafa Saleh wrote: > > On Mon, Sep 02, 2024 at 09:30:22PM -0300, Jason Gunthorpe wrote: > > > On Mon, Sep 02, 2024 at 09:57:45AM +0000, Mostafa Saleh wrote: > > > > > > 2) Is there a reason the UAPI is designed this way? > > > > > > The way I imagined this, is that userspace will pass the pointer to the CD > > > > > > (+ format) not the STE (or part of it). > > > > > > > > > > Yes, we need more information from the STE than just that. EATS and > > > > > STALL for instance. And the cachability below. Who knows what else in > > > > > the future. > > > > > > > > But for example if that was extended later, how can user space know > > > > which fields are allowed and which are not? > > > > > > Changes the vSTE rules that require userspace being aware would have > > > to be signaled in the GET_INFO answer. This is the same process no > > > matter how you encode the STE bits in the structure. > > > > How? > > --- a/include/uapi/linux/iommufd.h > +++ b/include/uapi/linux/iommufd.h > @@ -504,6 +504,11 @@ struct iommu_hw_info_vtd { > __aligned_u64 ecap_reg; > }; > > +enum { > + /* The kernel understand field NEW in the STE */ > + IOMMU_HW_INFO_ARM_SMMUV3_VSTE_NEW = 1 << 0, > +}; > + > /** > * struct iommu_hw_info_arm_smmuv3 - ARM SMMUv3 hardware information > * (IOMMU_HW_INFO_TYPE_ARM_SMMUV3) > @@ -514,6 +519,7 @@ struct iommu_hw_info_vtd { > * @iidr: Information about the implementation and implementer of ARM SMMU, > * and architecture version supported > * @aidr: ARM SMMU architecture version > + * @kernel_capabilities: Bitmask of IOMMU_HW_INFO_ARM_SMMUV3_* > * > * For the details of @idr, @iidr and @aidr, please refer to the chapters > * from 6.3.1 to 6.3.6 in the SMMUv3 Spec. > @@ -535,6 +541,7 @@ struct iommu_hw_info_arm_smmuv3 { > __u32 idr[6]; > __u32 iidr; > __u32 aidr; > + __u32 kernel_capabilities; > }; > > /** > > For example. There are all sorts of rules about 0 filling and things > that make this work trivially for the userspace. I see, that makes sense to have. However, I believe the UAPI can be more clear and solid in terms of what is supported (maybe a typical struct with the CD, and some extra configs?) I will give it a think. > > > And why changing that in the future is not a problem as > > sanitising IDRs? > > Reporting a static kernel capability through GET_INFO output is > easier/saner than providing some kind of policy flags in the GET_INFO > input to specify how the sanitization should work. I don’t think it’s “policy”, it’s just giving userspace the minimum knowledge it needs to create the vSMMU, but again no really strong opinion about that. > > > > This confirmation of kernel support would then be reflected in the > > > vIDRs to the VM and the VM could know to set the extended bits. > > > > > > Otherwise setting an invalidate vSTE will fail the ioctl, the VMM can > > > log the event, generate an event and install an abort vSTE. > > > > > > > > Overall this sort of direct transparency is how I prefer to see these > > > > > kinds of iommufd HW specific interfaces designed. From a lot of > > > > > experience here, arbitary marshall/unmarshall is often an > > > > > antipattern :) > > > > > > > > Is there any documentation for the (proposed) SMMUv3 UAPI for IOMMUFD? > > > > > > Just the comments in this series? > > > > But this is a UAPI. How can userspace implement that if it has no > > documentation, and how can it be maintained if there is no clear > > interface with userspace with what is expected/returned... > > I'm not sure what you are looking for here? I don't think an entire > tutorial on how to build a paravirtualized vSMMU is appropriate to > put in comments? Sorry, I don’t think I was clear, I meant actual documentation for the UAPI, as in RST files for example. If I want to support that in kvmtool how can I implement it? I think we should have clear docs for the UAPI with what is exposed from the driver, what are the possible returns, expected behaviour of abort, bypass in the vSTE..., it also makes it easier to reason about some of the choices. > > The behavior of the vSTE processing as a single feature should be > understandable, and I think it is from the comments and code. If it > isn't, lets improve that. > > There is definitely a jump from knowing how these point items work to > knowing how to build a para virtualized vSMMU in your VMM. This is > likely a gap of thousands of lines of code in userspace :\ > > > But we have a different model, with virtio-iommu, it typically presents > > the device to the VM and on the backend it calls VFIO MAP/UNMAP. > > I thought pkvm's model was also map/unmap - so it could suppor HW > without nesting? > Yes, it’s map/unmap based, but it has to be implemented in the hypervisor, it doesn’t rely on VFIO. Also, I have been looking at nesting recently (but for the host). > > > Although technically we can have virtio-iommu in the hypervisor (EL2), > > that is a lot of complexit and increase in the TCB of pKVM. > > That is too bad, it would be nice to not have to do everything new > from scratch to just get to the same outcome. :( > Yeah, I agree, yet a new pv interface :/ Although, it’s quite simple as it follows Linux IOMMU semantics with HVC as transport and no in-memory data structures, queues... Not implementing virtio in the hypervisor was an initial design choice as it would be very challenging in terms of reasoning about the TCB. > > I haven’t been keeping up with iommufd lately, I will try to spend more > > time on that in the future. > > But my idea is that we would create an IOMMUFD, attach it to a device and then > > through some extra IOCTLs, we can configure some “virtual” topology for it which > > then relies on KVM, again this is very early, and we need to support pKVM IOMMUs > > in the host first (I plan to send v2 RFC soon for that) > > Most likely your needs here will match the needs of the confidential > compute people which are basically doing that same stuf. The way pKVM > wants to operate looks really similar to me to how the confidential > compute stuff wants to work where the VMM is untrusted and operations > are delegated to some kind of secure world. > Exactly. > So, for instance, AMD recently posted patches about how they would > create vPCI devices in their secure world, and there are various > things in the works for secure IOMMUs and so forth all with the > intention of not trusting the VMM, or permitting the VMM to compromise > the VM. > I have seen those, but didn't get the time to read them > I would *really* like everyone to sit down and figure out how to > manage virtual device lifecycle in a single language! Yes, just like the guest_memfd work. There has been also some work to unify some of the guest HVC bits: https://lore.kernel.org/all/20240830130150.8568-1-will@kernel.org/ We should do the same for IOMMUs and IO. > > > > I haven't heard if someone is going to KVM forum to talk about > > > vSMMUv3? Eric? Nicolin do you know? > > > > I see, I won’t be in KVM forum, but I plan to attend LPC, we can discuss > > further there if people are interested. > > Sure, definately, look forward to meeting you! Great, me too! Thanks, Mostafa > > Jason