From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 87590C4345F for ; Thu, 2 May 2024 16:25:58 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:In-Reply-To:References: Message-ID:Subject:Cc:To:From:Date:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=WiBYAjA/rjlz7xtVzaJmRLvwQoNxAqmjUzO7lSdIQYM=; b=VuZsL/1wsBAolp HoW7+fLiyB1UReJ6STVeFQ9q4kcNwPBcizIka9dCzkp0Hox1zOOylqaT9DcvRHaymjpATm416t2Mj bKTZ9u4aNdkQvjSYNcCihSw5fpmC2DAcnc88bqS3GBddHfD/JvpbpA80M+wDGRNMwgOy8Yzyc+zl9 29U4PTKMe0YjuC2mIGO9F1vP7WPDxkmHhgNVNXIUOqdPwJYQqEG/+ifwb92opg/D2Zatn6O7/8A9R pzXbpVNJ7bI4r2B0CtPzFsUFCauifH4Y/968VbbagysyyacHD3FZT9g+rA6fKSdoCRNTKA5Tpwlyb 24IwzwKYG3rVInUt3g4A==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1s2ZFh-0000000DHeU-07zp; Thu, 02 May 2024 16:25:45 +0000 Received: from mail-mw2nam04on20600.outbound.protection.outlook.com ([2a01:111:f403:240a::600] helo=NAM04-MW2-obe.outbound.protection.outlook.com) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1s2ZFe-0000000DHcF-1PCm for linux-arm-kernel@lists.infradead.org; Thu, 02 May 2024 16:25:43 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=LD/VWlT5p63ke17HsgKKn6i8t540S9idiW/t3oA/Oa2D7HtJzPwJq76FTRe1m9vetfTF4sWribdgajQ3sx69dCUMJ2eZhF9MDOzjH69lKcelh8IzDK3VOycnwBYwXlSzaER1OGYR3Nx6sXYwJrZgFWdV6zpBCJwvjjC+XabYgyZUNBz7CcXeLsw0wlROs+KujlowdOUseVNut1Yoj0QnoGKcfu/t7m7pa592jbIyyZk33fLCHGF1rI2fmr71Y7bwxt9f4g6VhxHKwJjZcNNb1wLaS1RM+1P9Dk8qj6fz4+ZHL9gmBn/KWhHzct6b1UYbRuf3qa5tL8Hsrs6GygMPHw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=PmNbJufrwkgirVbLBjf1Mj5gxvEFeUgfPMh+QYO8NLI=; b=gsB4QyVl/CH3z5DN3QLqF7gNOtXkXGWip2Uds5oJp/MT08cNmikFVzPGfc15j6O9GPW4qOY+E9YTPn4Pv50tU44m1NAEEjPz8U6bJGXKeQAj0O1eN6La8MYlLjks+iXI7ut4F9vq1S1D4zaKXrk6MYQu+U1dtHmCtpmB5MHMNIM0HzBTT9rwApUrNUqKeOx2EM88bfd4bhYlewCpygW26wW/xglfUBCEmXRSPfyR7OYH6E+mSrtevsahpHZ1L/97aWw3qgAXX5AFD3dCZywgH6T41TRNdrLmNWZcXjMrj9UDv2NWNZGsT46vnPJbnyPQSkYFyZ7maJGsRiMhb0qAmg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=PmNbJufrwkgirVbLBjf1Mj5gxvEFeUgfPMh+QYO8NLI=; b=HiGirDSkKHZAroXk96dvwbzqhZLsBRSy6LKORTppw2a3xwHE7Xddc0sDnpQNXkFA/r11SBrpCdPWio0IPg7WinnxtA7oe+9KIM2VvTpMuLq0wEulLd3ZQCye7io02mnqAOqcZzzDQI6z+3Wm0HlA55zMR6GIgkeInwfR86Hinb7XQvugNMeJBLas17cf2QQmfpQxg1dZOclqF7s5Qx7iU0iNIrY/bZVkSTypqITChF/FMSR8nA8ZQbVaqHUr7hOJosPSOCeVo6ZOsGk6r/j1DxNYdTfX2SdD9BwPkx35++HioWRaRBXz0oD3cisejpF/Dp6VL2mSw+7d2+sC5dRzfQ== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; Received: from DM6PR12MB3849.namprd12.prod.outlook.com (2603:10b6:5:1c7::26) by PH7PR12MB9128.namprd12.prod.outlook.com (2603:10b6:510:2f7::7) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7544.29; Thu, 2 May 2024 16:25:33 +0000 Received: from DM6PR12MB3849.namprd12.prod.outlook.com ([fe80::c296:774b:a5fc:965e]) by DM6PR12MB3849.namprd12.prod.outlook.com ([fe80::c296:774b:a5fc:965e%3]) with mapi id 15.20.7544.029; Thu, 2 May 2024 16:25:33 +0000 Date: Thu, 2 May 2024 13:25:30 -0300 From: Jason Gunthorpe To: Tanmay Jagdale Cc: will@kernel.org, robin.murphy@arm.com, joro@8bytes.org, nicolinc@nvidia.com, mshavit@google.com, baolu.lu@linux.intel.com, thunder.leizhen@huawei.com, set_pte_at@outlook.com, smostafa@google.com, sgoutham@marvell.com, gcherian@marvell.com, jcm@jonmasters.org, linux-arm-kernel@lists.infradead.org, iommu@lists.linux.dev, linux-kernel@vger.kernel.org Subject: Re: [PATCH V3 0/2] iommu/arm-smmu-v3: Add support for ECMDQ register mode Message-ID: <20240502162530.GA578685@nvidia.com> References: <20240425144152.52352-1-tanmay@marvell.com> Content-Disposition: inline In-Reply-To: <20240425144152.52352-1-tanmay@marvell.com> X-ClientProxiedBy: SN6PR04CA0074.namprd04.prod.outlook.com (2603:10b6:805:f2::15) To DM6PR12MB3849.namprd12.prod.outlook.com (2603:10b6:5:1c7::26) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DM6PR12MB3849:EE_|PH7PR12MB9128:EE_ X-MS-Office365-Filtering-Correlation-Id: 8711b39a-718c-43e0-1ff0-08dc6ac47d78 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230031|7416005|366007|376005|1800799015; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?ISCwDyGA9lisdYsqZG8mHtCzlQmbOfTiy1JrY12dNF5O3TcIRlzGOTBngemU?= =?us-ascii?Q?jSiq11qBcGjVCLc5nfr4d0HQ0CRFl7mROeScBT8SA7gS3blG8akXFOywFVUL?= =?us-ascii?Q?6TLO+m9RQloaCz6DQ1geusB0QwnbYmtvT5VV6R2qnHhRnluTgaLDaxkOMBFd?= =?us-ascii?Q?XlBZw3w5rI+B/QFadwyZBFEn9CC3IsKhWtzLGqbZTQHMYBElNkOz9GGRyO2V?= =?us-ascii?Q?P2gsnDzYdWTxy6mR7yr2mzZhAU0A7mr+kiQDfhL9lkYDw/9GJ+ES1TfEZFDb?= =?us-ascii?Q?/Wc/crBgTjVFe4Jjgin3browT3A3AP+vgvdknABrb1fe+1T0zuIN5jIGCgYo?= =?us-ascii?Q?6UlbHfoMtyjvOkoG7FwFmPWI21hQxDsuoVdlwi7tNUkFPSc21DbI355803Zt?= =?us-ascii?Q?vwVWTN1dmbEf6fbHviAj7NJvwc0BXlL0TiV/ZO9s2LrTcfLETAuABpMXfjZ+?= =?us-ascii?Q?PkyEV4RUNvhVnxsG3WAxxswNelbSVmmrp3+cDNtzHaQxojhiUbZgfZ9l9nmq?= =?us-ascii?Q?RZFz7SDQyuPUhLE7xxP6fpMumBqGhRVnOHAktSY/rcC7eo4U/qzVtGseQLq2?= =?us-ascii?Q?9e5IpQ36EqmCIxvjpBRYlDYk8dNCP11a7883vnD5xV6SOFIJLsZNpRnfUEYt?= =?us-ascii?Q?5z7rTjUtC79MYSOhMKJXLPdQyaliedZcGoBchUEv5aDVtivwFAgiOVUGa1eD?= =?us-ascii?Q?ugq/gvqb3xgTANc26bC8X/yqkgeo75+dFSAdI+vhwTEzJUqxrUe13JWEiugZ?= =?us-ascii?Q?T93oypdtMv4wEb9upIzPNP4pTw+9AImyuFBM/CZN8tj//veB4AfuIVTnHFlW?= =?us-ascii?Q?7IKKLV5FOufQfKZhpQJT3ekpiQ7/H+8SYG6iembJy4njg3pKLtvPXczY5Jf/?= =?us-ascii?Q?TDU63WMmFAOPLV91SxhmxSZuzI1kc779AHcJZZUZcfJ4Y2v6UGRwrt2iT5UY?= =?us-ascii?Q?26NT1eONPIgvBWZcMwFzB8zbCaciMSz30yxD5f++6Z3JVQ61GSH1YK3qsc9T?= =?us-ascii?Q?87OucrASmUSQ64O8RhH+DbadD0StN5GDNfP+b0EqRfFGCANEtwtxkWWdY8gR?= =?us-ascii?Q?dTwyqD4M9J51B4p4t5yeU3r8DEFN9kixpsNekC/7WU7EESiyP1QZllvz8PI7?= =?us-ascii?Q?steuIoN3iJKJyL3zBjTMgba2YYHNpQHpyYB8ZuUD6GKchuLuC9D2/jsPj7Md?= =?us-ascii?Q?RKOsw58o/HZCl8T/KZR13BEDUW9k5oxFv2+72N+SjejJHa45EACg4Mp+NLR+?= =?us-ascii?Q?QOc8ViqOR46K4no2zZnq4un+JsZHwk2I+tZ7ag5AdQ=3D=3D?= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DM6PR12MB3849.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230031)(7416005)(366007)(376005)(1800799015);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?AyQrBS7SMQIM72zGjcitWFNVFLpBcMSWTbcESVIcFUxi17yNdTr1NbgbCbIE?= =?us-ascii?Q?kjXb0DFFcp2TozpU4nQWiUOAx6GwG7Bgm8FmYp6i8SzHmXeKj5aVJF9xF7vf?= =?us-ascii?Q?jmI78W59EeXHbDlFLxrok2D3/S8guMfj3EZE5Ff7pzI2RsK8zj3+uWJT+5+u?= =?us-ascii?Q?hKJBm7sEYlamdsbtaKlQZ4uIlwwVEd9y14rcSq8n5uz8JmveBWvP40ql3V9u?= =?us-ascii?Q?Jhm1isHbc7e0TXymIydu/XcJB5sCc1h7Bkg2BCxjTFRBS439Tt0MjqR7ewGo?= =?us-ascii?Q?5se1FvEVtuUiauXd9NmEaWcwwrjgQKVKm/C8LidW2MALZQgta4JHcZtkTTDw?= =?us-ascii?Q?paD9D2YtwoHIFcIbx/Ef5RdxzB0yKS/WNiWxldUFTjeLTmkDbECDChuToyvU?= =?us-ascii?Q?15+T65uvxApvVO3b8vq7gNIkMLxhnYzcOYLNdHImFMFR4eamVJWHL4WUa8o1?= =?us-ascii?Q?IlwBR1hAFXaoIgZSCN227CXNW+kfjGh4j8QaThaUjtRQQwhMtQTJlLMsY9hm?= =?us-ascii?Q?MSGmvqgp2DWT4QS/rODsqi1+/wY8yMTx9OQyVXWwIzPnyDF/NpGBNxCG7MdL?= =?us-ascii?Q?QbtbJ6duKF5L/ZcvLJnpXvmdmps0Afkjh26xoO4wMu8N5qQHYwQD+JzVK+xu?= =?us-ascii?Q?R93S0GnkWR2ATYPqcCDdYQJbrYibXpF+ZIv5pn2gnCGNqMPbeKI6lhfumeH0?= =?us-ascii?Q?9LonUv06RLDY91NTgGMmV+tY15R6Zq0l3YN9oVB/eel7tnHpqdbWu/HhNh6d?= =?us-ascii?Q?gp5BlXHdOjPDrIGuwny/0IdhMEG+lely1jWuBYmip/OfOvXKnqk/d4Arphat?= =?us-ascii?Q?hC+LDZE99TagMbDStLPZ97KdC2Ck3z7hNSeHvzwOK6mGQKifcFDTZWVSPlUh?= =?us-ascii?Q?hNzpDvum7oJoQp9wzUUHYezzh1xpjwOLcXj4Dpt27udjrtE+dfh+mYHeDYTR?= =?us-ascii?Q?mJgn12gKWHQ8koNH11zHWlJZqDD+FxnXlzs6ps+nsGnpC0CMXqOqRWzxOAP7?= =?us-ascii?Q?KqVm2wZMgmO1NvXN/OQMPMB9GZoDcM++OoPl4M193tYHhk6ltZRrBeJNr6/I?= =?us-ascii?Q?du1O223ULI2W9m7taTwy0FgUUQG46JEqN7LNeZVOvnWD8uGJWVgcwTv3rfvx?= =?us-ascii?Q?92pLSjuIPxCXjeCO601ULDZ+4c3ETyrupmOc9DCCKTkVeWzbv1mr8Wj28z6T?= =?us-ascii?Q?npbZfM3RnQhPm0SomjT/skAd0zRwF1eknLZSawdRqS9JsXsGdzDU18FPWvNb?= =?us-ascii?Q?I7Q6Iem+tVXabzlJdc6XL8Vr53sxttlGj5MohrzIFhimDLC7TiEclBAyR5PY?= =?us-ascii?Q?Qf/VkCNoSEmHbi7fGsJ9yObU8WzKAWP3iriKwrBlphr54emDyiNhss18l38x?= =?us-ascii?Q?d74NFac30lniycbKVVwnWqSLcBJUe6e+0K+kCIc/uIuhRz8j8NwdM365NwJw?= =?us-ascii?Q?VQGUTXL9tFzeOpd/tekhehDhDpihH2xOUjb9mfh5SuvTy+/kLkRp2vcm1RfJ?= =?us-ascii?Q?e7kvKzep0u1IERhHq+SzP26luZ8GFiiLLKwFd0DWGIkCAoH67IpN+zGd5vDU?= =?us-ascii?Q?71xtx5Y7LRRwsliq4gCSJ4FLOWd5YFvVsh6Qc+RW?= X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: 8711b39a-718c-43e0-1ff0-08dc6ac47d78 X-MS-Exchange-CrossTenant-AuthSource: DM6PR12MB3849.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 02 May 2024 16:25:33.2716 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: ppycXuNTRLElPdLkUFfioFXyEvcPLpUP/BxgtlWlg0Tvmqldzz6zo6D2brzPNaLz X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH7PR12MB9128 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240502_092542_541467_66FE42E1 X-CRM114-Status: GOOD ( 13.23 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org On Thu, Apr 25, 2024 at 07:41:50AM -0700, Tanmay Jagdale wrote: > Resending the patches by Zhen Lei that add > support for SMMU ECMDQ feature. > > Tested this feature on a Marvell SoC by implementing a smmu-test driver. > This test driver spawns a thread per CPU and each thread keeps sending > map, table-walk and unmap requests for a fixed duration. So this is not just measuring invalidation performance but basically the DMA API performance to do map/unmap operations? What is "batch size" ? Does this HW support the range invalidation? How many invalidation commands does earch test cycle generate? > Total Requests Average Requests Difference > Per CPU wrt ECMDQ > ----------------------------------------------------------------- > ECMDQ 239286381 2991079 > CMDQ Batch Size 1 228232187 2852902 -4.62% > CMDQ Batch Size 32 233465784 2918322 -2.43% > CMDQ Batch Size 64 231679588 2895994 -3.18% > CMDQ Batch Size 128 233189030 2914862 -2.55% > CMDQ Batch Size 256 230965773 2887072 -3.48% If this is really 5% for a typical DMA API map/unmap cycle then that seems interesting to me. If it is 5% for just the invalidation command then it is harder to say. I'd suggest to present your results in terms of latency to do a dma API map/unmap cycle, and to show how the results scale as you add more threads. Does even 2 threads start to show a 4-5% gain? Also, I'm wondering how ATS would interact, I think we have a head-of-line blocking issue with ATS.. Allowing ATS to progress concurrently with unrelated parallel invalidations may also be interesting, esepcially for multi-SVA workloads. Jason _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel