From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 4D274C83030 for ; Thu, 3 Jul 2025 21:19:10 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:MIME-Version:In-Reply-To: Content-Type:References:Message-ID:Subject:Cc:To:From:Date:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=MY7nZjyoVGoBsEEsCJ7kgtHsKCFOhRhsRxFnuIQLR3U=; b=UWae8hdGPMqyZnXzR2uQZP5BA7 h1f5HtYL94fpYH1Y4rf60JjoNtjE2p/BXXFqkUDcMl/uqNmLY0yC9bmGsZsHx8Krra8wPSZbO7nHO rf/S8hIAvvUR+8MpNusHwW7iA1tLxEajujFMXl/LHtEbe12naUp0vaeh5gs/iJHma65rVDmQ5aEUc 2/2QRBpfG8SSWuhP28QaSVHmqECUF/yCZb6ebICBkGWMCt/hEnJ1c/+iG5KWiyqAs8gSt66XoUvRt Dn8j+bo4r1ytojivu4KLXfgA2Nl9FkHezqZRvpu2GDX9FNWVdt99ZKBHHLN0LPi9rNvks9uwHx8Td cAO+dI6A==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98.2 #2 (Red Hat Linux)) id 1uXRKh-0000000CbZx-2Wnj; Thu, 03 Jul 2025 21:19:03 +0000 Received: from mail-mw2nam12on20620.outbound.protection.outlook.com ([2a01:111:f403:200a::620] helo=NAM12-MW2-obe.outbound.protection.outlook.com) by bombadil.infradead.org with esmtps (Exim 4.98.2 #2 (Red Hat Linux)) id 1uXO9q-0000000CBqz-2yrn for linux-arm-kernel@lists.infradead.org; Thu, 03 Jul 2025 17:55:40 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=j2Ijk6NBUkDduksvbmXxc5+Yogll4HkiLollhFYmUG7vZfZ15+N0i3d1x/sJdgfhpv9fZpcSb8vfiBqzTj1P79WdjlR8Sibn79MLauKamMxf2Zy6BebPtTSFbvAV+HTXV/Z7GkTdwXXzojHEV6ToNMwuIUutF/5kSJmvmmmIflVnKb9Q49iXLh6uIsR2QElC98q6Dcm8PE8ya9Aopl3JDWWj/EyuUAWCk2GilMnAyJ0fAiJHiGFF9Dojzq1Izz/2UGggXsDr6uOQHlDy6eJqFttkgc6RP/cNz//kmC97Y9mun+r0BI0yAR0ftiTSHYHqMhDEiwO6134pjxc8wUy/Bw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=MY7nZjyoVGoBsEEsCJ7kgtHsKCFOhRhsRxFnuIQLR3U=; b=COkZAF68r2ZvE1ZRtIQ+Sv8WKHGL37qSjnJHxc59IJqi75PnHBajG3PzczUa/0wABBlJpmQnd11KN7s2+KQU47E/BSCdtaVABuWOAozO2ELc0f6FZthGWsUIRhPITrhd9PnmWXeMS73c+LySzeW7i8szMEc0n7t6h/BP7dtVGW2f8bAqqniLqwflSC9fI0rZc04u7vtE9o4k03VPIFqV5Z4yd4xwlbWauokJJb3ohvHXBWGPiB5HmHGmHDDmm00oYwEc0LcqUjiVOYBRBpSEBlyQAw5ClVF/AP412uHZACthBBSHDB8KKQWinoetLN1xc6OlGEY5A5b19oYrpxhXVA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=MY7nZjyoVGoBsEEsCJ7kgtHsKCFOhRhsRxFnuIQLR3U=; b=AYOqxv/c8b7ntIpfpXP4tBQwCwfhr1T7eeEc/JKQ9obmfAm2WZ7CylsS2i5hpx7ipvpMq+c6gihHz8EfFF3u86eIYg9cpjrL1AaUM/WagIy7mXYI0nzS4FvjfTXgqaSP2ekUnnln1/hcFpJZal8uOvMgLWaON8Ylpovv7gY1CVStAobcICqKHU40yBCAqTSAKo39c6t4NrvBMgRaORo4u0tJwgHTODrm5WfDQpUhsCWCHGXW8Nj3wKGOXY+0O75FWj8SAzDrVrzBp98SekfZ3YHhTbARqiwu2hX0Tf+WbWAS+GZX5111DiGZoKee08kVcNxXJXhdZgQS6XSCpax7EQ== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; Received: from CH3PR12MB8659.namprd12.prod.outlook.com (2603:10b6:610:17c::13) by SA1PR12MB7127.namprd12.prod.outlook.com (2603:10b6:806:29e::18) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8880.17; Thu, 3 Jul 2025 17:55:34 +0000 Received: from CH3PR12MB8659.namprd12.prod.outlook.com ([fe80::6eb6:7d37:7b4b:1732]) by CH3PR12MB8659.namprd12.prod.outlook.com ([fe80::6eb6:7d37:7b4b:1732%7]) with mapi id 15.20.8880.030; Thu, 3 Jul 2025 17:55:34 +0000 Date: Thu, 3 Jul 2025 14:55:32 -0300 From: Jason Gunthorpe To: Pranjal Shrivastava Cc: Nicolin Chen , kevin.tian@intel.com, corbet@lwn.net, will@kernel.org, bagasdotme@gmail.com, robin.murphy@arm.com, joro@8bytes.org, thierry.reding@gmail.com, vdumpa@nvidia.com, jonathanh@nvidia.com, shuah@kernel.org, jsnitsel@redhat.com, nathan@kernel.org, peterz@infradead.org, yi.l.liu@intel.com, mshavit@google.com, zhangzekun11@huawei.com, iommu@lists.linux.dev, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-tegra@vger.kernel.org, linux-kselftest@vger.kernel.org, patches@lists.linux.dev, mochs@nvidia.com, alok.a.tiwari@oracle.com, vasant.hegde@amd.com, dwmw2@infradead.org, baolu.lu@linux.intel.com Subject: Re: [PATCH v7 27/28] iommu/tegra241-cmdqv: Add user-space use support Message-ID: <20250703175532.GF1209783@nvidia.com> References: <20250702180541.GD1139770@nvidia.com> Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-ClientProxiedBy: SA9PR13CA0130.namprd13.prod.outlook.com (2603:10b6:806:27::15) To CH3PR12MB8659.namprd12.prod.outlook.com (2603:10b6:610:17c::13) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CH3PR12MB8659:EE_|SA1PR12MB7127:EE_ X-MS-Office365-Filtering-Correlation-Id: 58f9f43e-8e7e-44b6-6e62-08ddba5acf1c X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|376014|7416014|1800799024|366016; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?4sODyGP6MIzYr0wLqsEuD7dntAXtBLwvhdUgj9y8JS3hIVugDvFPS2kriLm5?= =?us-ascii?Q?pkkmV+w+Mf91SwcUIXGKNDy3K1xZL4wCX4ScnggQThbr41qMPimH49BETWgZ?= =?us-ascii?Q?ode1DkQ+h7lqr/QZJp19AtAKblVXhhy3obL4GgiF0vqZcGXY341suVS/bQzz?= =?us-ascii?Q?n/xfXkBiUTBFFVgct55fUhkyTHkB47hAKNmpsqDljDZ7haSrcp1Ak9SDTZ8D?= =?us-ascii?Q?tKFUjnBmBM06otoQo5zlqeMX+QhZFOmHqAhyHrbXaI0WwDePnVgr6rQITJqX?= =?us-ascii?Q?h8mAgdcKWKfzTUiPCIA7s6etrNG7Kz/5VbcDeNouM6bj8/ySTeoNWBbNDeX2?= =?us-ascii?Q?3rpwB/thlcdc8BXTUiumxVuIMLOOKmA2ZOOf7qzcjfvuck5l1487vxHCbZQw?= =?us-ascii?Q?YuVJeDKPsP2CgAOJd0obNIO1oOM0Koq+gqlWB5Zq8Pt7GriykLRgp65f9D3j?= =?us-ascii?Q?9HToSGg8Z9dzo+V/qJXXubalbmJ7+pY981M7JlXA6H3SW9zhME2lxldjX0+L?= =?us-ascii?Q?4j2JcVkJA1iCt6O+5ITNMD/+U0fzhPKzQzPxe+NyP5gW6mmhXg+gOOhr3Q3d?= =?us-ascii?Q?DxANTXAU5pNMAOvp1DIIJOGfJFt9uc54a2DEoQxHerDeMDyapru179NjxZ79?= =?us-ascii?Q?ue84JT5GFJRowXydaz2XzCX+SLQV3czZO98sTt1uHbE1B8x1fDAKvrVXa9yS?= =?us-ascii?Q?1WRc2V2Usp4YI6WZWgyY0XWl5CkVEr9/Y6oJ40Xu2Kv9lAHXC6G2hmFJ82ou?= =?us-ascii?Q?82iDf2ahSLst6F6EInqg5kpELj7C+PjY7CoefNmPyTYRVS/4/6pTRZBCMoTX?= =?us-ascii?Q?HSOUbKgHFk3XG54jj/tUlJ4Qrr/O1RsIeSnG3rSxr3P04O8Fv8BgKxL5OyAn?= =?us-ascii?Q?fSnpt+7rabrr5ZvhovmFMuCh+sfLmrYeevic9ebwxBdYSNk1elczIomUUB6B?= =?us-ascii?Q?ShAxOCWTAo2atQlBU0gkNzAzvtuw4tYYGGBplcGQ+iyUPwJjEXWLW14FYixy?= =?us-ascii?Q?gZpSEW3VBw8aS+7TrXxCYpS0FEkgl5p8bh7f82flA2K6rEw0tJkeFTfIGhOy?= =?us-ascii?Q?l3T93apN9jcBaVOPytnsE/NDosnh/8wHubBlwg6Z7555HhAukjgtCYGPrt0e?= =?us-ascii?Q?0MiOSDu0+6NV7LsLBEOgjVGambhmAkymVW1sDiUlxtrd++VKPeGzW7PRyc8j?= =?us-ascii?Q?SesjThIKwPvjPWHFrMYEVCU2loPfPQqAMSJg67N9zb0y+jTAB2wDPfJFAc9e?= =?us-ascii?Q?PGGp5Jc6RTOVXpg823r+wUaVMwRr/yagIg5OPdyOJQhvaUbSaNKl9eTIZXxn?= =?us-ascii?Q?58b6wknLRq57htE20SXxLeIXeRX+J10/+w7LVdbo2vcCxgNAtDgx3wBz+SPw?= =?us-ascii?Q?3zHMrR7ufdoVDQ9wy4Q8EKFWjaoYNFFz1tHSaD7dZqPWmxGZUw=3D=3D?= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:CH3PR12MB8659.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230040)(376014)(7416014)(1800799024)(366016);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?qYMxpzotLOmcfL2diPLMwfhxKff2tIz47JS9kwTqcvVbMR5OHuc+YCw0jaYJ?= =?us-ascii?Q?WD9f8Y0t9t90YiciktzussSiynTPGW909+6j4cR71kT3leaHX3070qUD314f?= =?us-ascii?Q?RVexf21q6ZHbFVUpV1PdTASAlhjAPySX0ykUnDTD14mQhKZGx/n4CoRD2AtE?= =?us-ascii?Q?rqRQzCSX/hpZjyjPi9t26/9J9nuRcrQtddq58T56AntUPAWoZc4t+y9KILxZ?= =?us-ascii?Q?fBuS7YYHrxEGkqtUP8I/24r/mX+uiG6PSgcfUlABu2XkPsF6+JMGF9uHJ+JZ?= =?us-ascii?Q?lX1FB0P+sWJ5K3eI9zYKwe9CsAfAr3PXKpjWoNa9eJmo7mi8WStPL3u/jrmu?= =?us-ascii?Q?8LVxiZuTMCoGZWg5E8SxkZG7ynuQdQSo5yn0BQ808H7/cBNOpr2+Ej5hdPt/?= =?us-ascii?Q?beyjiyepRnqsa2d24O1xzLoDhzg2dvbejkIzBd71daLdWtVAMiIM4BPPrE+i?= =?us-ascii?Q?vvT6SERrEq5RLVOM6WwdBx9DKykQg0L9wpQ+aJuWI+j5PYSHnknOO95Ub9nt?= =?us-ascii?Q?feBqoJse79qIThE2uSqHaKNv2LtcShuyc35Vbu9QYc4m26cjwxJieTtJqBz7?= =?us-ascii?Q?My9d6lItNoEi6kmt+BM+SJgn2Il1mZXEnEzrkLgNIY6E02teCwhGFeX8CpwG?= =?us-ascii?Q?Dqv3UnUvVyZCg5lLZMnI7qvgFpPNstVXif2hlGhDffnzedhKIwPlcjqvoEcp?= =?us-ascii?Q?Zq+2t/aVz7MgR+b0j1HGgjIBT755PEBrnVXNs9Fcz6VY1fcCOYfjjO6U9/0D?= =?us-ascii?Q?oAF6p7EsMuJsJj2Jfx+tPyyIugzfEcl5/yHJh2VOfPRbLW3QXeTNaWFDVLmG?= =?us-ascii?Q?+53PNYYfwGhTAt/1NY9u5boEb96Lb1NHxYR6Dx8ToHNdYC4pvNMQgTswwOh2?= =?us-ascii?Q?ESlKzRmfxWdsta9H3o1FzyTojjQ1r+aBbSFOMG6qbiJHQKSfN6MPwz1xyny/?= =?us-ascii?Q?M53ckPW7KEPIgZLDrJlV8EtwiLYoDTlj6v5+djwcolqGsIz0tQ6ZVpwb0UTq?= =?us-ascii?Q?dVHKHCwJwQMXTTwG8T6y3PMsTo3bEfU0ohVAuU+w22l+4iBAvRJAClMb7FO6?= =?us-ascii?Q?c2W26uJ9q+Ad3zrOr2QBcVNGjEUPQc4+y9ifaLV73/tIWD0KA64nsgxcUk0E?= =?us-ascii?Q?PcOmDk+aPcBda8Ta1VwqnwQ4hM+0VtAAuudvVCEfYn+VU/Ltb/8Mq1nGeY0M?= =?us-ascii?Q?8UZYV993k4TR/emRQ+hHYfiUvTCg0k8RovC8xsBbh44wD5uOvSe3eRdIy8Cv?= =?us-ascii?Q?KT5HLlUN2i2izO18DjEkm3kZf4B18VBVuSC3ry2EMWa1fZ1HBlzqlTQsGXOX?= =?us-ascii?Q?IiHHnvnBtz+EBpX5uYEcfAURa8rkyB0Id/rpcT+FmuXWfdduAtYVk6xwWlNK?= =?us-ascii?Q?CB+0CVyLPWmM9lUT70SMLEmT5toYBc0CURsRqV4yrEu1sVDneor6R2ouZSzC?= =?us-ascii?Q?1IsFibs3E2NMsVKvQIEWxdHZwo6+hYTruRrKyGHVf6O9yMiuswX0qX6uAbem?= =?us-ascii?Q?5NF9FronLD5YCeqsjV8Eu6PFV5X0mVumGAXKv5OOwbP/LE7mujAb5J8PK+ww?= =?us-ascii?Q?snHnuPY4BEewfyypDTXTZ8zrecCrv4FworeuApjZ?= X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: 58f9f43e-8e7e-44b6-6e62-08ddba5acf1c X-MS-Exchange-CrossTenant-AuthSource: CH3PR12MB8659.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 03 Jul 2025 17:55:34.4832 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: CcTSHCWdDj19siofnJUY7SwUjXDgR1trpwCD/49gGvUjcI71jYsWIVpj8oylv7Qj X-MS-Exchange-Transport-CrossTenantHeadersStamped: SA1PR12MB7127 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20250703_105539_111452_2F6DFAB4 X-CRM114-Status: GOOD ( 13.20 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org On Thu, Jul 03, 2025 at 02:46:03PM +0000, Pranjal Shrivastava wrote: > Right.. I was however hoping we'd also trap commands like CMD_PRI_RESP > and CMD_RESUME...I'm not sure if they should be accelerated via CMDQV.. > I guess I'll need to look and understand a little more if they are.. Right now these commands are not supported by vSMMUv3 in Linux. They probably should be trapped, but completing a PRI (or resuming a stall which we will treat the same) will go through the PRI/page fault logic in iommufd not the cache invalidate. > > The goal of the SMMU driver when it detects CMDQV support is to route > > all supported invalidations to CMDQV queues and then balance those > > queues across CPUs to reduce lock contention. > > I see.. that makes sense.. so it's a relatively small gain (but a nice > one). Thanks for clarifying! On bare metal the gain is small (due to locking and balancing), while on virtualization the gain is huge (due to no trapping). Regardless the SMMU driver uses cmdqv support if the HW says it is there. Jason