From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id C8AA1CA0EDC for ; Fri, 30 Aug 2024 17:00:45 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:In-Reply-To:Content-Type: MIME-Version:References:Message-ID:Subject:CC:To:From:Date:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=RENsyliQMBVhaPEPIYXZwpoQaf0SD4eb4IOhjdJVmj0=; b=eue759XOWvmkxeLx+2NSg0h/6K 0yJr21dsJjWh+psb1QT/NcDHW9qipg7W1IUWZy7J6pKUxCdq/ddga9RmE7zYyUiyGEASdeeqtcI4T ziMGntU51tLIFDB+bJ34t5PukMXAog+HCgaIHz0CFhhkggMTV3zgO2+MKwclmkRVF4PuxI7njh5QO B9vmNy8CK9MNKAbwpZuQdYN5Ba98VPcl6EglkogpQ0HQAYGTqtY5/bDQ/ofM/CemugSRH1U6T1JMT +yr90TYYn7mO2UmVIj9t/fcok2ne5Oc/GyjvGmt4CBwqIpsLZ9sBT99GPd2CuUBOjz1kBMXZihNb7 Pepra7uA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1sk4z9-000000076qT-16JA; Fri, 30 Aug 2024 17:00:31 +0000 Received: from mail-mw2nam10on20609.outbound.protection.outlook.com ([2a01:111:f403:2412::609] helo=NAM10-MW2-obe.outbound.protection.outlook.com) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1sk4yF-000000076gC-4BBw for linux-arm-kernel@lists.infradead.org; Fri, 30 Aug 2024 16:59:37 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=e1pzcaLekLYxuKywIZwxkLVhU7fFG28FcIyxrDgnMu1QItiNrC2DZCRZIFviA0dSi0RHBSyEuXkf4s8zceQMvx9Qe2YLgho7FsImN+HmYADo6iJGGALez4R5Ey+oRDOgEJisivFt6qUftpER+2jTCBKlklCqOhozjzsGS1ItDFXSToaQcHiSw888OPPBN/BNhD6WkVha393OhABEmC9tfFLOCYw3FZP3dSHlMzE/GuohLBUtwc8Tspbm9tWlVW4zVMSe4hbKR67O9wZe5WV9PdYkYxxpdcPcER7x3ntMrH6dOo1SNdjiWcLCMtabr3N2Tt3TUb6JVyyDdrgymjyWHQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=RENsyliQMBVhaPEPIYXZwpoQaf0SD4eb4IOhjdJVmj0=; b=FaGd7CQYk1Y7d79OZ6SzEu7qN3sEYo53f/ApgPyyXtEGaJ2GxzzDwarDwceRELSIJCw7f3RXe05TYJLSGOquMlmF1FKVcH8WXM9LVAbuvcEtsF2nXvm3CfO2kRCMw+jVsJn3HVKLg/vF6rzxpusf1BIfQNMWCFulVaRdedXwsbfPA+znm822VhNHY30JqBF9PH3fbNOGWeu8JqQbIs+pK4wy1eA/NSA2EHpBHPei9UQciBfR2HOJb2wF5FB1yfHiaVErZRODO4NNGlLLNpmTAQWwdy0vxwVE6+Hu0gV+pp7ZHN9KUh2dyg5+1zeFpGzNzlLN2Q5pWGePXjN88QiEvw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.117.161) smtp.rcpttodomain=google.com smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=RENsyliQMBVhaPEPIYXZwpoQaf0SD4eb4IOhjdJVmj0=; b=aVsp92c+W7ZSLkdpbuwSqy03C91TwpRL5XVdjC4KDwq06ZYguPEbP/HGwH+h0edG01XgMCvDLHpXRZ07tJuoS7DQSyy8E9XhDHSWh46QQl2HNMDj1bEBQoOMDUSwIXkyGu53ddfJR6O66ETgKNjFTdZoL1nJbGDHZPyHS9Epzf4g+biK6OSmJLFxKZUtOKdagUNukKKT820uFWhhwyqSy1yLRd/EN6QCj5sbrt9hEfYlViBhMu0Pvoymcwc2LpG9E/3w785+vNAHGwFTXIZW2ieu6AjssSM58JS03LS0sTpkshjz8ZDA1qDKojmFJL0npEQdoICKLpg+lkp5SxHEIQ== Received: from SJ0PR13CA0190.namprd13.prod.outlook.com (2603:10b6:a03:2c3::15) by LV8PR12MB9417.namprd12.prod.outlook.com (2603:10b6:408:204::18) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7897.26; Fri, 30 Aug 2024 16:59:28 +0000 Received: from MWH0EPF000989E9.namprd02.prod.outlook.com (2603:10b6:a03:2c3:cafe::17) by SJ0PR13CA0190.outlook.office365.com (2603:10b6:a03:2c3::15) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7918.21 via Frontend Transport; Fri, 30 Aug 2024 16:59:28 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.117.161) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.117.161 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.117.161; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.117.161) by MWH0EPF000989E9.mail.protection.outlook.com (10.167.241.136) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7918.13 via Frontend Transport; Fri, 30 Aug 2024 16:59:28 +0000 Received: from rnnvmail202.nvidia.com (10.129.68.7) by mail.nvidia.com (10.129.200.67) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.4; Fri, 30 Aug 2024 09:59:11 -0700 Received: from rnnvmail204.nvidia.com (10.129.68.6) by rnnvmail202.nvidia.com (10.129.68.7) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.4; Fri, 30 Aug 2024 09:59:10 -0700 Received: from Asurada-Nvidia (10.127.8.13) by mail.nvidia.com (10.129.68.6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.4 via Frontend Transport; Fri, 30 Aug 2024 09:59:08 -0700 Date: Fri, 30 Aug 2024 09:59:07 -0700 From: Nicolin Chen To: Mostafa Saleh CC: Jason Gunthorpe , , "Hanjun Guo" , , Joerg Roedel , Kevin Tian , , Len Brown , , , Lorenzo Pieralisi , "Rafael J. Wysocki" , "Robert Moore" , Robin Murphy , "Sudeep Holla" , Will Deacon , Alex Williamson , Eric Auger , Jean-Philippe Brucker , Moritz Fischer , Michael Shavit , , Shameerali Kolothum Thodi Subject: Re: [PATCH v2 8/8] iommu/arm-smmu-v3: Support IOMMU_DOMAIN_NESTED Message-ID: References: <0-v2-621370057090+91fec-smmuv3_nesting_jgg@nvidia.com> <8-v2-621370057090+91fec-smmuv3_nesting_jgg@nvidia.com> MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Disposition: inline In-Reply-To: X-NV-OnPremToCloud: ExternallySecured X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: MWH0EPF000989E9:EE_|LV8PR12MB9417:EE_ X-MS-Office365-Filtering-Correlation-Id: 9266d592-0308-4172-84db-08dcc9151c00 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|1800799024|82310400026|36860700013|7416014|376014; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?Ja57kSFCi2t5STLnIyqNL8l748yY7pX8hfqaHdH6HCNz5qTkbPrVTpG0GtKH?= =?us-ascii?Q?a54ffgBoMZIEcrgF5Eo5A++1rMRvCprvBOK2fqK+qOJ5JkzSLPBNeZCVeFmD?= =?us-ascii?Q?HW6vY6mJ75sTnWLqrGhmzIm4sxUgV8gH/yxdpR7C5qjkoIqZnoIKY2Fn0uDD?= =?us-ascii?Q?SmWnMwUPlpXy2xyTWdQ6yl8U74puxqtTnmo2TCr/UGZg5mybHmH2APFWOP86?= =?us-ascii?Q?TlFeYMmugiyfb2hcBqrjMn66Nj1RudAxPJxX6Q357+ZNcWqP9VCropmSrPeK?= =?us-ascii?Q?TGdkybF5Jbgjvu42AM4l4RLh9Dzps9SMC6N/BfwNgr0zCuM6H6E02Rbk7ND7?= =?us-ascii?Q?BBwZm/Obux1dnL7/VYy7hxFTPHWSVyUYGFxzGJ38G2BY4HUjulGwwszA7ahr?= =?us-ascii?Q?SjxdcGt1ufK1ftcQKXlzt/Kv0TIP4Sdn2ECeXRtInFlSwEw/yuIb34SH1+OC?= =?us-ascii?Q?jgE7zJtwLjcKfohHy/hdIMlDVN9SsSgMuWWrZIoTQdzhgAjFarH4wblUcDXc?= =?us-ascii?Q?O0Baou7bMuOYd1KmUqIomLxwEV1COwIszY/wf4mDwstT6rOQuARN5s0Gz114?= =?us-ascii?Q?T1qPVIe0Igt9ObsbOVUok5O2coiZ7vJ03BpeoYqtNx5d05zOfUi6hv+wtO9j?= =?us-ascii?Q?kaL80/1O3KgDB+qXSLhpNdPQc6LkHjRXn/OX2Tu1m91Lb68XLScVX6WxXNYT?= =?us-ascii?Q?5ejUYtn+1SPdMHmva8g2sgdIPhb/uXd2XMhA0PsKMUiVq/RzNt0EpUVWMMTJ?= =?us-ascii?Q?nZWVz3pY+eLhOtwySz3Ctfxa+3ZrCvfRSXwVfYDHISQSt9yb95EBgjmAZhSu?= =?us-ascii?Q?lPk+t/ZYi9WlTJhzyeuox/HhdThYUWOTdLNOZJRaVMVDEfg9iS9BvDw3RYGl?= =?us-ascii?Q?TEMYB8j/+j9V3LYiUPrTl7G8a2HMuS96oyf2+2W0jcatMJK9HKklfkTH7YOc?= =?us-ascii?Q?ZN1D7eiMOZGlZsPWVun5qN48K5hOdnhPHYx6GIIxKbFYclz3sbbwjFmn/cok?= =?us-ascii?Q?3qZ16RYlyO493utCV58ccOhY/0/0O8EG7TOVONVh0fAM8fx3pVGxWHboECSY?= =?us-ascii?Q?L1K5cVOVCMGY9lBEpTO0YYnqz6G62ZnEaOPzRlCY6oFODsNuRO2w4vTbhh9O?= =?us-ascii?Q?RRWDsBXkcK6DPKCgC4DJl/0uJOa7adYa6vJls0qCYeB+mc3J6r/RpDHcL1sM?= =?us-ascii?Q?5rqW/cTejGnI1bFIRmYKYyidmW6b3WsyXfe1UprTe0/V9BD7x9b0IvUriNRy?= =?us-ascii?Q?D4Iz8m5dT6NP8ZJunr2Fo4PVAUpJqyvBxkRoTIyVNcOTqNXH2jk1dVEYRp5+?= =?us-ascii?Q?/+AW8Pzn1TL8SJaRk4DeELhnj6llUFaugIutfscQj+f2Kbz4gWVoKxOoSm+G?= =?us-ascii?Q?JMiIKByL2XE9phqmR6Vt8oTdP+5NLAv9Z58Sm2PNly2BwBInkOo+0JKBofPn?= =?us-ascii?Q?LLi7TAUCJKG+hZdqbGM08HDUerirG/CQ?= X-Forefront-Antispam-Report: CIP:216.228.117.161;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:dc6edge2.nvidia.com;CAT:NONE;SFS:(13230040)(1800799024)(82310400026)(36860700013)(7416014)(376014);DIR:OUT;SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 30 Aug 2024 16:59:28.0122 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 9266d592-0308-4172-84db-08dcc9151c00 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.117.161];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: MWH0EPF000989E9.namprd02.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: LV8PR12MB9417 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240830_095936_062878_809DEF89 X-CRM114-Status: GOOD ( 23.36 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org On Fri, Aug 30, 2024 at 04:09:04PM +0000, Mostafa Saleh wrote: > On Tue, Aug 27, 2024 at 12:51:38PM -0300, Jason Gunthorpe wrote: > > For SMMUv3 a IOMMU_DOMAIN_NESTED is composed of a S2 iommu_domain acting > > as the parent and a user provided STE fragment that defines the CD table > > and related data with addresses translated by the S2 iommu_domain. > > > > The kernel only permits userspace to control certain allowed bits of the > > STE that are safe for user/guest control. > > > > IOTLB maintenance is a bit subtle here, the S1 implicitly includes the S2 > > translation, but there is no way of knowing which S1 entries refer to a > > range of S2. > > > > For the IOTLB we follow ARM's guidance and issue a CMDQ_OP_TLBI_NH_ALL to > > flush all ASIDs from the VMID after flushing the S2 on any change to the > > S2. > > > > Similarly we have to flush the entire ATC if the S2 is changed. > > > > I am still reviewing this patch, but just some quick questions. > > 1) How does userspace do IOTLB maintenance for S1 in that case? We do all the TLBI/ATC/CD invalidations via the VIOMMU uapi: https://lore.kernel.org/linux-iommu/cover.1724776335.git.nicolinc@nvidia.com/ In another word, nesting support requires the VIOMMU p1 series at least. > 2) Is there a reason the UAPI is designed this way? > The way I imagined this, is that userspace will pass the pointer to the CD > (+ format) not the STE (or part of it). > Making user space messing with shareability and cacheability of S1 CD access > feels odd. (Although CD configure page table access which is similar). Given that STE.S1ContextPtr itself is an IPA/GPA, it feels to me that the HW is designed in such a fashion of user space managing the CD table and its entries? CD cache will be flushed if CFGI_CD{_ALL} is trapped. This would be the same if we pass CD info via the uAPI. What's the concern for shareability? Thanks Nicolin