From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id C17A9CD13CF for ; Mon, 2 Sep 2024 09:59:24 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:In-Reply-To: Content-Transfer-Encoding:Content-Type:MIME-Version:References:Message-ID: Subject:Cc:To:From:Date:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=lZQX/cMZETZqukPtivvikiv242o9q96eLnQhbaJJJkM=; b=3wC2yjafilqWLo+TzGbBkd7CND 31Hwvnbd6P1v1ewsv1ePg4GWRUhMuw4IxV0GfIysxjEY3hgeDEhg/nu1rDeRUSev6+yl7iTUoKlNy p8CRyymC2GzgmpD0Ml/sfwdWC/fYNfZFJuKS7hGUMSyF5AADP6d8jdV7Rwrcn5BfPLR/UUhnuFZmf oe1vJQK7e1ElGVSl6F+OZirOhmNuGdVLEsGowg9HxRg2zuqmUL7KI4wK6UqU0/7YoicENwt6hwnVG O95wmOP+2yqXXwAAwwcXuXmKxbfSza7jMvWpwGyhd6jzQ2YuZnhA13BDbEBIN2e1fjhNF8WFtxSIS w9FNEjqA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1sl3q5-0000000DoLV-0PwF; Mon, 02 Sep 2024 09:59:13 +0000 Received: from mail-wm1-x329.google.com ([2a00:1450:4864:20::329]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1sl3ol-0000000Dnkx-303z for linux-arm-kernel@lists.infradead.org; Mon, 02 Sep 2024 09:57:53 +0000 Received: by mail-wm1-x329.google.com with SMTP id 5b1f17b1804b1-427fc9834deso67575e9.0 for ; Mon, 02 Sep 2024 02:57:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1725271070; x=1725875870; darn=lists.infradead.org; h=in-reply-to:content-transfer-encoding:content-disposition :mime-version:references:message-id:subject:cc:to:from:date:from:to :cc:subject:date:message-id:reply-to; bh=lZQX/cMZETZqukPtivvikiv242o9q96eLnQhbaJJJkM=; b=p+Vt+EIC0YcQU/RrNGmm6vuqW++q31PEJFDv63Bu8VRALj7N89Xgki4yEngDAVhnV4 Co2TP4G2OkCVmeR6KVgVH0AxLqIXLaeLwnUM5IzeT9QSE4Bf8jDRl2jEbSq5ZA3CULco lN6fgvpDE+DVv9J77ZLx2anbkFvv+k+XJii062ZuPjALdNMh6ULmbXZdwJghoCSh3tnd uiECgro307O/O3Ara5ZsiPAAzAIZPfHR6FTePF0O+t0dRVjBaFyaVCe/Uff9K9lUMvNl 9hJ+sDCrYVoEW6XtrpxPg10p30Hg+u8kZHQC3o0sPFIG/R/Ym7w5C0sL9QPsVQq4QXjB iN3g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1725271070; x=1725875870; h=in-reply-to:content-transfer-encoding:content-disposition :mime-version:references:message-id:subject:cc:to:from:date :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=lZQX/cMZETZqukPtivvikiv242o9q96eLnQhbaJJJkM=; b=SdhgaUrwDKZzd9nw0IBBOoB39XbydgsaSnDexRRxmMYewLGewC0oZbbgoGcveYwYkI d1kthmJx5RRxAIHw0CcfNfqJx3PJPhJgozbUQK6W+uaahzilCMMrDN2zWCmE5yJ6Ov5n NvVqnajOk5A7J6Bm+7MRUm5M9ymrCw7Qi9AURZboywO8KZS6ptLOESw48UeNl84917n0 mRu2ERTflbns8MqHvnkgVTc3tzr9SIpIEGLWJM2nsD3vJashfMkl+2xuEHlbh10QBE3u RCMpngTWUxpj1rV3J3Fx50ylZ6MEu3wWVCksrAXThU+9IakV/H4OMaIYnEYDsRL13ua4 s8Nw== X-Forwarded-Encrypted: i=1; AJvYcCU+m4SgqikV+HKLOAr543LEpGKpp0KWPK+oFfe+de7AileiO6yFiTS4lxFjWbtQZCFUYBYdKjwbdbm7uWp78zak@lists.infradead.org X-Gm-Message-State: AOJu0YzxfhEo0Se/DwxDV6iJIIJdHHghWc2YYHVsiZxBcQC29riuCkVo QMDp8Vw3B+AQJM1MMtFyjxdzY+W2e1+RtSITXQuUQ3zVLSTNoy/R1uy4Y6FI2A== X-Google-Smtp-Source: AGHT+IFSZcv0nult7Zja5pAokUpRh3eCHY8+dggfufZa4QfUGMc7wSMOSbQ5AYJaIDsiCUY7g7vMyA== X-Received: by 2002:a05:600c:500f:b0:426:5d89:896d with SMTP id 5b1f17b1804b1-42c78767f7amr1965085e9.1.1725271069765; Mon, 02 Sep 2024 02:57:49 -0700 (PDT) Received: from google.com (109.36.187.35.bc.googleusercontent.com. [35.187.36.109]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-3749ee4d391sm11015220f8f.3.2024.09.02.02.57.49 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 02 Sep 2024 02:57:49 -0700 (PDT) Date: Mon, 2 Sep 2024 09:57:45 +0000 From: Mostafa Saleh To: Jason Gunthorpe Cc: acpica-devel@lists.linux.dev, Hanjun Guo , iommu@lists.linux.dev, Joerg Roedel , Kevin Tian , kvm@vger.kernel.org, Len Brown , linux-acpi@vger.kernel.org, linux-arm-kernel@lists.infradead.org, Lorenzo Pieralisi , "Rafael J. Wysocki" , Robert Moore , Robin Murphy , Sudeep Holla , Will Deacon , Alex Williamson , Eric Auger , Jean-Philippe Brucker , Moritz Fischer , Michael Shavit , Nicolin Chen , patches@lists.linux.dev, Shameerali Kolothum Thodi Subject: Re: [PATCH v2 8/8] iommu/arm-smmu-v3: Support IOMMU_DOMAIN_NESTED Message-ID: References: <0-v2-621370057090+91fec-smmuv3_nesting_jgg@nvidia.com> <8-v2-621370057090+91fec-smmuv3_nesting_jgg@nvidia.com> <20240830170426.GV3773488@nvidia.com> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <20240830170426.GV3773488@nvidia.com> X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240902_025751_783121_EA1DA873 X-CRM114-Status: GOOD ( 47.19 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org On Fri, Aug 30, 2024 at 02:04:26PM -0300, Jason Gunthorpe wrote: > On Fri, Aug 30, 2024 at 04:09:04PM +0000, Mostafa Saleh wrote: > > Hi Jason, > > > > On Tue, Aug 27, 2024 at 12:51:38PM -0300, Jason Gunthorpe wrote: > > > For SMMUv3 a IOMMU_DOMAIN_NESTED is composed of a S2 iommu_domain acting > > > as the parent and a user provided STE fragment that defines the CD table > > > and related data with addresses translated by the S2 iommu_domain. > > > > > > The kernel only permits userspace to control certain allowed bits of the > > > STE that are safe for user/guest control. > > > > > > IOTLB maintenance is a bit subtle here, the S1 implicitly includes the S2 > > > translation, but there is no way of knowing which S1 entries refer to a > > > range of S2. > > > > > > For the IOTLB we follow ARM's guidance and issue a CMDQ_OP_TLBI_NH_ALL to > > > flush all ASIDs from the VMID after flushing the S2 on any change to the > > > S2. > > > > > > Similarly we have to flush the entire ATC if the S2 is changed. > > > > > > > I am still reviewing this patch, but just some quick questions. > > > > 1) How does userspace do IOTLB maintenance for S1 in that case? > > See > > https://lore.kernel.org/linux-iommu/cover.1724776335.git.nicolinc@nvidia.com > > Patch 17 Thanks, I had this series on my radar, I will check it by this week. > > Really, this series and that series must be together. We have a patch > planning issue to sort out here as well, all 27 should go together > into the same merge window. > > > 2) Is there a reason the UAPI is designed this way? > > The way I imagined this, is that userspace will pass the pointer to the CD > > (+ format) not the STE (or part of it). > > Yes, we need more information from the STE than just that. EATS and > STALL for instance. And the cachability below. Who knows what else in > the future. But for example if that was extended later, how can user space know which fields are allowed and which are not? > > We also want to support the V=0, Bypass and Abort STE configurations > under the nesting domain (V, CFG required) so that the VIOMMU can > remain affiliated with the STE in all cases. This is necessary to > allow VIOMMU event reporting to always work. > > Looking at the masks: > > STRTAB_STE_0_NESTING_ALLOWED = 0xf80fffffffffffff > STRTAB_STE_1_NESTING_ALLOWED = 0x380000ff > > So we do use alot of the bits. Reformatting from the native HW format > into something else doesn't seem better for VMM or kernel.. > > This is similar to the invalidation design where we also just forward > the invalidation command as is in native HW format, and how IDR is > done the same. > > Overall this sort of direct transparency is how I prefer to see these > kinds of iommufd HW specific interfaces designed. From a lot of > experience here, arbitary marshall/unmarshall is often an > antipattern :) Is there any documentation for the (proposed) SMMUv3 UAPI for IOMMUFD? I can understand reading IDRs from userspace (with some sanitation), but adding some more logic to map vSTE to STE needs more care of what kind of semantics are provided. Also, I am working on similar interface for pKVM where we “paravirtualize” the SMMU access for guests, it’s different semantics, but I hope we can align that with IOMMUFD (but it’s nowhere near upstream now) I see you are talking in LPC about IOMMUFD: https://lore.kernel.org/linux-iommu/0-v1-01fa10580981+1d-iommu_pt_jgg@nvidia.com/T/#m2dbb08f3bf8506a492bc7dda2de662e42371e683 Do you have any plans to talk about this also? Thanks, Mostafa > > > Making user space messing with shareability and cacheability of S1 CD access > > feels odd. (Although CD configure page table access which is similar). > > As I understand it, the walk of the CD table will be constrained by > the S2FWB, just like all the other accesses by the guest. > > So we just take a consistent approach of allowing the guest to provide > memattrs in the vSTE, CD, and S1 page table and rely on the HW's S2FWB > to police it. > > As you say there are lots of memattr type bits under direct guest > control, it doesn't necessarily make alot of sense to permit > everything in those contexts and then add extra code to do something > different here. > > Though I agree it looks odd, it is self-consistent. > > Jason