From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 1D92BC47DAF for ; Mon, 22 Jan 2024 17:57:53 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:In-Reply-To:References: Message-ID:Subject:Cc:To:From:Date:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=UU/iMs0avcaJkIV1qtGtohRhbgK36Tz8WBuXejJsUOc=; b=LCK5glqprcJc6x FZSeuuV6e5w0tNg0MM4YkdqGn3Ms19zPjuicnuu3UyO+XCwT4YUI/656ROJcqqkKEgmYcvGdle/7J B8GlwlB9uYXMVffGm8rkM/P3LVAtvU4+YQlKHi2elNGth6p4T6D54F5ypi7DH87Vl+WZoW5PCJhfE D111UD7JGxTym9+XjFsaZVMjhSHk2nwLjdIcTgumRqoivMUYUMcypb/1FiYBEMU4mV+5euOb+B5Dd wG98x5pBNGxOK6yjYbwi2MbQq4aWyNOvTnAy5+2Zi5Ol9Hb1xzp0ahh/O3sQ0HQD6w6CMQr1gT+t1 s1dI6pMQJMbvjnfRna1w==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1rRyY0-00DNf2-1a; Mon, 22 Jan 2024 17:57:24 +0000 Received: from mail-sn1nam02on2076.outbound.protection.outlook.com ([40.107.96.76] helo=NAM02-SN1-obe.outbound.protection.outlook.com) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1rRyXx-00DNcz-3D for linux-arm-kernel@lists.infradead.org; Mon, 22 Jan 2024 17:57:23 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=kDWnrny3xGJK8JmJuWN/AfGtTH50MntpZKilfE/d/qRnEQ+kbIp085P15nGrMG3EW5zbcngxiUM/9s7fKCHYjm+OfrqN6y9x1qGM0DCyGiibUGxs3AmOlZILVl59jMxwoCPpLTrTQZjttqBKoYKwlSn/8Bqs2mwRpTlYwR+sPGfQmXzLcxpbX/TwcXHB6Z2LKQzRz9E0SS1BFjW4OsCPNvduS3q2It3DcaMdMKSvKAosTXEa5hadrfMzTa95L3h2nnZ9RazG6lT4+BnfdG/AnlJxPklWhif2McK/45TAm9kZe5SAAu8cPTz/f/+BT59fpNdiukwfjPd+9NHPGs8noQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=3gNV6i3XZ8sR4s38Bl74F33Xd1Xz/0sKK3b2Fec+Lr0=; b=PLW5D3PtyAl34lFqkfzUKpWeeS8z3cfGch5cT0N3JT5zzkilBmzx8tW3LvCi7ikEP0kJmAu04XLM4k+9ktENT1t3MSeS5OwGWEnbk5TPDa+S5DUrfsLM/c139r9xzU+jL2+wo5wuPXnLV4bPsGH8ihjVEq46KQqhSAfASUzNZRpWbOZIlA9uGLFE99p9t25pDZlZRWzMGi//YVcPIMJdxB8+fomTuYKFZSoiE3drrH3ZprHdbJjp4wuJr0zHwzmR2gzutKM5IC9dbrSVZ1/3PnVUbeHe5v2id2ptJWOXAWrFdjU6OHsJZ6MoGZPBHo4HatQPVzFJE8JdNH6gtdMWXw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=3gNV6i3XZ8sR4s38Bl74F33Xd1Xz/0sKK3b2Fec+Lr0=; b=CcIX0jSFJO5tv02LI5cwK6QDurUj2cbmBdb7OLclcAOkUGPcKL4KoSePzb8ncQyw/zMs4GKmch5gU3CHpj2WlwT4fSnYvxwc3eCr1nprGIW2WxQ4aIujf0cc+FaazS/qzyIy57M4fSAP6628Ik28PoO2AZtSHTc7ES+aBU/viXgASVANYBVjZvPkjWb49/q+V/Ezjva+FRcD8S0y2HA8ecauKDObSq+mnNZZHeQypYcVaJbr85FGrufpcvQjyEnakiwwEh9lLE8BgDJOFDs6N9zS6J2YFqDmVeatiG26FSiQWbVXvueOxOgdaPN4y41ru/BYRxXIEjZP205T9c0LHQ== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; Received: from LV2PR12MB5869.namprd12.prod.outlook.com (2603:10b6:408:176::16) by SN7PR12MB8025.namprd12.prod.outlook.com (2603:10b6:806:340::12) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7181.31; Mon, 22 Jan 2024 17:57:02 +0000 Received: from LV2PR12MB5869.namprd12.prod.outlook.com ([fe80::96dd:1160:6472:9873]) by LV2PR12MB5869.namprd12.prod.outlook.com ([fe80::96dd:1160:6472:9873%6]) with mapi id 15.20.7202.031; Mon, 22 Jan 2024 17:57:02 +0000 Date: Mon, 22 Jan 2024 13:57:00 -0400 From: Jason Gunthorpe To: Nicolin Chen Cc: "will@kernel.org" , Robin Murphy , "joro@8bytes.org" , "jean-philippe@linaro.org" , Alistair Popple , "linux-kernel@vger.kernel.org" , "linux-arm-kernel@lists.infradead.org" , "iommu@lists.linux.dev" Subject: Re: [PATCH 1/3] iommu/io-pgtable-arm: Add nents_per_pgtable in struct io_pgtable_cfg Message-ID: <20240122175700.GB1455070@nvidia.com> References: <0fe68babdb3a07adf024ed471fead4e3eb7e703f.1692693557.git.nicolinc@nvidia.com> <61f9b371-7c45-26b1-ec0f-600765280c89@arm.com> <20240122130152.GP734935@nvidia.com> Content-Disposition: inline In-Reply-To: X-ClientProxiedBy: DM5PR07CA0115.namprd07.prod.outlook.com (2603:10b6:4:ae::44) To LV2PR12MB5869.namprd12.prod.outlook.com (2603:10b6:408:176::16) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: LV2PR12MB5869:EE_|SN7PR12MB8025:EE_ X-MS-Office365-Filtering-Correlation-Id: 2a767ad6-7c3f-492c-b9f6-08dc1b7389b5 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: FAqLoYchVAGzqI5b7pAtsTNFzk/p/JFFTLGNCsBDO1TiHRS/EiKLnhEddz/BTgtUsCLBYr8FzeGcMidIgbZUpMjTsNbKL5i0jzO6LKGTwSRbOIyv02XsjGc1ay6VVuWJlEoJjE5eyg7K/EZmlWaRYBZjhQMNMheKmlh/9eVf1IKqp2naxHarPc6i/IWNp8U3gm4vD1XRxbVGhvFgqPoXoIpzZZoFDZpsgqoHjQdjwcgaJh8mkGT6TxdeaU4bTf4vAxPdAj3uEXOj8VVIUVLntYyo1kIldXtFTxTDHHyZ+gzS4Wa8m3FnziC8iCDysCfeVLeaFOSpooEmmrMUnu7nqWlktaCgXxyhuDlS4lF0+PV6ljw/AgO2xHT2wlsjjOa7BOvbpqBk+7Lpadl+P8jqvib+mR2QpCuytVR8PfhShVAkkzwH2X78ldgqJGcMvJHqW1UoGGiL3SGlnlDH0BM+rITk/1KJHkXbo/XODWIqC9TzIUBsIbtMB1jm1ru/xTN9ec7WfbkYlJTD9veVZa8aDUL0aTgrQq37vsro5PZw1wmB+3z3yekirPfKDwIlsR+ApfbY6E6oL+vusNx55/XbIqtqyOOTamWCfzAlKirJfYeJUm36jwV6A41SeKtHKhLJRTtxMjjs532sT2X7I8TOUw== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:LV2PR12MB5869.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230031)(136003)(39860400002)(396003)(346002)(376002)(366004)(230922051799003)(64100799003)(186009)(451199024)(1800799012)(66899024)(83380400001)(86362001)(33656002)(36756003)(4326008)(8936002)(8676002)(38100700002)(6862004)(26005)(2616005)(66556008)(6486002)(66946007)(966005)(6636002)(66476007)(5660300002)(37006003)(1076003)(478600001)(6512007)(316002)(2906002)(54906003)(41300700001)(6506007);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?LdVAc6CCL2dDI3fKOVSLYvm5k9a7FdahbZS3AgMeZY2upPn6tC39tIGYAS+j?= =?us-ascii?Q?qAMWmmpE8c1gJc/FJvDDj1MgDUUNi+sAR4st8dY2tynkRrGiDt/PJ8lND3uK?= =?us-ascii?Q?ciphfCgPJ0OKfYnz7gaP9leoPkntZWzga04OmxnYvw1cHuc1vpfj5C8uOXf5?= =?us-ascii?Q?qxbfgRcpZN84DCLydFmCCRIJLExQUF7KGTAzkKoWwjZIXO3+XQm6ovcB3RKv?= =?us-ascii?Q?x+bgPGUYFztZ9YOwqOU2TnWeEKkzZ1p64gEwQo6UyxbYljRZ2Bx14ETQOJHP?= =?us-ascii?Q?nKWgNB2/wjXc5Fv0bKQ1ub5qbDb8nKLem51uArmCBP/QsBA8rSJiAYBvVofc?= =?us-ascii?Q?8Zh9cCn7xDPpN42E1aJK1tQf7jDETcifDEHFdW0L9D5wpmA6EjUB8pZB53Zk?= =?us-ascii?Q?bt5TBGnMOw35T2HCgHozQS7pGhWemu8j3uS+jdiGECSxBkBn04hm9dUTlObD?= =?us-ascii?Q?DlKhyPL7sRmWU3sMDfZmhH0TlttlcbrT/UO9FPXvIJq1fuxbv8ql0MGzQTUg?= =?us-ascii?Q?NdhM5xJRmMd9c+OJIqRMApVM1PHvE+vh0f3IQeAbXldTBAHs9aVh72Ar+Qmo?= =?us-ascii?Q?aWY3hsYiy5hec6FfSpWEcqWlMfDbborzCCm5HC15beE1v1f30Y8J9nNf3r46?= =?us-ascii?Q?GrCjcBeOosO8Z6EPEim9fGIlk1/QoSmPQyLvVySfQqMk7tc5/NwTweCWwpK7?= =?us-ascii?Q?hy1B8d89KltJ4R3V+7tiZPn6jKwlhC0CnE4DQX/3rI7J+ZrY/DlA1wKT4BCw?= =?us-ascii?Q?pQqfmU3dI+heTgIEvHXrnmSVxlUVyqxqZLPLEgtp7ESbr5cxxXK3LiZAT04j?= =?us-ascii?Q?ri/zkjSfFoS0OMI8IDm28mJOJnU9Vi+jm2/NHKuaAKxvhYk1FurqNz4wPI4Y?= =?us-ascii?Q?w3ABn7Z7vEpa/YA6HNOjZN+Writlqvr3Z6ejLqYo62pT7WTfnKxKatdUlKLc?= =?us-ascii?Q?mByoXM//XR/Y3z8H3xYE8b40YlU9ijvr3EbC9rO3q2Od41BtlXMquPsqxFFD?= =?us-ascii?Q?OJOLSJTUBBmSVgku4fuPfEDJgoYF2zpimxriw2/PYzSBcnlJJuEirGl3NS3E?= =?us-ascii?Q?d/A0Ci1CV1LScQAmVnwT9xtOjZRmH0DDdbGCRYaMCKEE4S/ANb32SbQNhMay?= =?us-ascii?Q?DLYJo3v6GNJuUsG7UHC3hetq2WvDwWdw/EzJjkeEnKL90Cca6Ch6TJ+AgygU?= =?us-ascii?Q?rUv8rETu1riWJztauPFaxy4AJNeXLI4rft9kB6OmFTjSuLBQDbLkNeh36XAl?= =?us-ascii?Q?PBmtZMEfpA3CSio108cZngpyPQC68G0t0wmDJvlE7sBLT0CrfrES0IiVIW7p?= =?us-ascii?Q?m49kmqNcnDAxBNfarjRtWKBTq7QO9ROxYKp+hsjuScm22/5nh53oEAWKcWBg?= =?us-ascii?Q?vQDKOVeWdRl5iAhEIT/OD1NomT2ulyFjfxBg/SGzMiuLaYshqGPmgdN4UvK/?= =?us-ascii?Q?evjFUB4KSlsEKzQXuM94NUv/mm+RhGd06v8qX8mVAxG9aLl/NFsk1KM0aRoJ?= =?us-ascii?Q?x74Hu/EcRJa6/QjDsSu/4FPIJZSjmQQUhuq74N6tItQnvgwByvMlm4w62ntZ?= =?us-ascii?Q?TO8dq32sTD541mSm+mnV3Vcoic1rkReOjvJbGGLx?= X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: 2a767ad6-7c3f-492c-b9f6-08dc1b7389b5 X-MS-Exchange-CrossTenant-AuthSource: LV2PR12MB5869.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 22 Jan 2024 17:57:02.6502 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: s/UNgAaYmS/jHeT+NyjlA7/0i2cl8dz2lgSDz4yJP4b4gPtEWd4awTLPwHpdsFhk X-MS-Exchange-Transport-CrossTenantHeadersStamped: SN7PR12MB8025 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240122_095722_104259_E7672F61 X-CRM114-Status: GOOD ( 23.60 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org On Mon, Jan 22, 2024 at 09:24:08AM -0800, Nicolin Chen wrote: > > Or do we need to measure at boot time invalidation performance and set > > a threshold that way? > > I see. We can run an invalidation at default max_tlbi_ops to > get its delay in msec or usec, and then set as the threshold > "xx ms" in the idea one. > > > Also, it seems to me that SVA use cases and, say, DMA API cases are > > somewhat different where we may be willing to wait longer for DMA API. > > Hmm, the lockup that my patch fixed was for an SVA case that > doesn't seem to involve DMA API: > https://lore.kernel.org/linux-iommu/20230901203904.4073-1-nicolinc@nvidia.com/ > > And the other lockup fix for a non-SVA case from Zhang doesn't > seem to involve DMA API either: > https://lore.kernel.org/linux-iommu/e74ea905-d107-4202-97ca-c2c509e7aa1e@huawei.com/ > > Maybe we can treat DMA API a bit different. But I am not sure > about the justification of leaving it to wait longer. Mind > elaborating? Well, there are two issues.. The first is the soft lockup, that should just be reliably prevented. The timer, for instance, is a reasonable stab at making that universally safe. Then there is the issue of just raw invalidation performance, where SVA particularly is linked to the mm and the longer invalidation takes the slower the apps will be. We don't have any idea where future DMA might hit the cache, so it is hard to know if all invalidation is not the right thing.. DMA api is often lazy and the active DMA is a bit more predictable, so perhaps there is more merit in spending more time to narrow the invalidation. The other case was vfio unmap for VM tear down, which ideally would use whole ASID invalidation. If your issue is softlockup, not performance, then that should be prevented strongly. Broadly speaking if SVA is pushing too high an invalidation workload then we need to agressively trim it, and do so dynamically. Certainly we should not have a tunable that has to be set right to avoid soft lockup. A tunable to improve performance, perhaps, but not to achieve basic correctness. Maybe it is really just a simple thing - compute how many invalidation commands are needed, if they don't all fit in the current queue space, then do an invalidate all instead? Jason _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel