From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id C46A7CD5BA4 for ; Thu, 21 May 2026 10:46:30 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:MIME-Version:In-Reply-To: Content-Type:References:Message-ID:Subject:Cc:To:From:Date:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=D75tCjitQ1odxw9JZsctvqpHrXyBP10/ZfCuiJk1724=; b=KvTo273NquR37rLA9Rt5i2I8+f BFYbK5/4YYHm6qgfcdm+EuCq5BQziaCUnY4SKDR0nl/Io1ux0f/AGjffEkpWkHjtbUSdZ5nJxfpfJ QHKmvptbXd9hs55znmRh1IogX/BmlMwMN1Ln7J9LY+3CMX3v8Mzuu81QNltfmdvFH8sFvXYHr4TXK XKCmnjNe13R9TKA+2tA04S6PnKfIxa7y3uUzkLIHB2oIqrVFYtFjRouc5Djg+kRz3PDq155s8yT4k XNX+w0VIJIrEnJvU95lkUi8aGTQ/STqAjTtYhw1LiAT/eyHGafsOIC6uiTeF23xjOPSaWpJBNj2xT /y+XQlug==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.99.1 #2 (Red Hat Linux)) id 1wQ0v2-00000007W8M-2C8I; Thu, 21 May 2026 10:46:24 +0000 Received: from mail-westusazlp170120002.outbound.protection.outlook.com ([2a01:111:f403:c001::2] helo=SJ2PR03CU001.outbound.protection.outlook.com) by bombadil.infradead.org with esmtps (Exim 4.99.1 #2 (Red Hat Linux)) id 1wQ0uz-00000007W7g-2C7Y for linux-arm-kernel@lists.infradead.org; Thu, 21 May 2026 10:46:22 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=rfhqCzR4JZoSzO38agpYl79s7BTpafYUCYJoQ5JRxhixJc3L7fWP8ZCrQe4kwWknGS1RG65NoRT2kAFKrfz5zciC+Or3AekJK73Cd6iMmT4AeqWTsvf/LdbbNwNnDpDEzRXyOG350IxUgt3v0RTiKFFZ1/y+ovUFMm03BjnKyM77QCAdE+MPCNeD1NKvac5cZEj7athbshEfxFP9NF015Fzfi8oylUpGrzn1iwBN4P7Fjj9vZ0cLPMZYZ1oXZPkhO+fVTtHNn5IY9itV91sCUvsA6qGx/rWtnoiX97qCl+Es8rTshoxX9aa9rnCH6mQqmcJB5IDCee9FmTBnMhde2A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=D75tCjitQ1odxw9JZsctvqpHrXyBP10/ZfCuiJk1724=; b=rwPh3Ja87HfF8ZzTcnns8cXdLNdbN73FYmdZofTzPrEvq7Yo4XfALXb1z5Bv3mE8jlzZbYvtlN66IeFgif3RGMb0zdr0XwJs03RhySOThg0xp6RtakN+PpJlZD6EYTC+V1qffM8gDlhirh5UUZ+zgRUmPhmdN+h0/6cpn8V4prtAyRNKiXK3pKM/gBFwA6KzmiIedv16/fXOa3TC0UBaJeK+zGIGAnJtB573eXFqyaLaBi24ZmOUm984kn4esK1Z1QdzlxsDg9WUsvdP4ZQyN4QB1qYZtCA8sLSTkHHDn7xLQ9jfDs9z+zrZinwg6QMAFLsl/0ggCKdfUSp9/ilreg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=D75tCjitQ1odxw9JZsctvqpHrXyBP10/ZfCuiJk1724=; b=bVTmzZ7w8Fyr+Kr97fUGSTj237Gd/uI50Z1dobemGWQO8dsPdBenMKn8wIIVYzLqRVW0sLYyBaNCMXtiqpNXWtZ+Rp8yf1nSyb5U/U65Gsfe/KzRDt/aXl9rcZeQATQMYjzq4WmR0kFhOz8+aveMvJeoc3rJLciLq1NqGUqjJvuXiHhYWPySCxUPA+W8UleGNcR9A+7SWe8KNxZzCueJWhT6KfBc86x8IqN2ixu12Y3suPYLklZRaLpDNMk1yQJsjXIZpHyO1J10+QdO8X9Plwsm+tv/er/ubJoJazSMkgfZmOiXnFVULfYZ6hmXxzWG/EkJUrjqGKVveg1sDOoH/A== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; Received: from PH8PR12MB7277.namprd12.prod.outlook.com (2603:10b6:510:223::13) by CH1PPF5A8F51299.namprd12.prod.outlook.com (2603:10b6:61f:fc00::60f) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.9913.12; Thu, 21 May 2026 10:46:08 +0000 Received: from PH8PR12MB7277.namprd12.prod.outlook.com ([fe80::2920:e6d9:4461:e2b4]) by PH8PR12MB7277.namprd12.prod.outlook.com ([fe80::2920:e6d9:4461:e2b4%5]) with mapi id 15.21.0048.016; Thu, 21 May 2026 10:46:07 +0000 Date: Thu, 21 May 2026 20:46:02 +1000 From: Balbir Singh To: Ryan Roberts Cc: Alistair Popple , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, catalin.marinas@arm.com, will@kernel.org, david@kernel.org, anshuman.khandual@arm.com, dev.jain@arm.com, jhubbard@nvidia.com Subject: Re: [PATCH] [RFC] arm64: mmu: use range based TLB flushing when hot unplugging memory Message-ID: References: <20260521042426.2128731-1-apopple@nvidia.com> Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-ClientProxiedBy: ME3P282CA0127.AUSP282.PROD.OUTLOOK.COM (2603:10c6:220:1c4::17) To PH8PR12MB7277.namprd12.prod.outlook.com (2603:10b6:510:223::13) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: PH8PR12MB7277:EE_|CH1PPF5A8F51299:EE_ X-MS-Office365-Filtering-Correlation-Id: ba960f1a-098b-4f4c-a183-08deb72629c0 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|1800799024|366016|376014|22082099003|56012099003|18002099003|11063799006|4143699003|6133799003; X-Microsoft-Antispam-Message-Info: uAhM/KdAyRmD1/CujTewR4uRbygKbGbtT7qThZ0DxI6V6KRKKbbMKYv5ruo3qcZ1OkGmCqR32S25v+5xTgr9Z5LQuXwbIibbffUQvoDVBGp+C6OKN58E7m1K1JH8OTpd2/eJ6zAfwreoStdhB2Z2jf3pmck0ViSSTwVP45dpRo2KBnO0ZCVj5d6SB6ZqacIieKKeIEw2tDrfMvt1d12atuWPtqJKJqFKW5FUnfw3zvUGgsISbXl6dhMcVkp4ke9DpqITeuf0PslQeCSTltZKOYhr5zgNRm5oWm8wRuDDt4GPeyD5b7BugolNhLVQ+KWkzPluN3QQnDLlzlA4irF2MW8DWEj/i1i3yW59es3sXNhFVxNmh3f6JlnV/An5CX3VDKKJa7SMDfKF2iVq3rA+3LCZ4bNHzXZiaZU16K1X2wt0sLXOfrQI6xOEPU8KNvUPohs1uIQvO1kdvHnFOmWyjPQZIVGIMXgx8k2Jp6iAshbj96ie+f2DO6KIi60KLsOIPycnGvFAkkjdlsGV5njhHjfbaoMxMeaXSOlwQsPxLjkFCqUiJ8mGh8SzUI2f+NQmBWusQ/ZXU438qwJvJ9STgqLBBIbEIoFGLSe1/8hVdy6x5q38TynAKdJb0JWi6wvZQEH/PEp135T4mpCIG171MGuTgbR7NuwEXxiAYtV00VKbXFchE1osO2k6fmdLzY3d X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:PH8PR12MB7277.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230040)(1800799024)(366016)(376014)(22082099003)(56012099003)(18002099003)(11063799006)(4143699003)(6133799003);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?zM/cLqMq6OG4awWze8LCmLJMmPIsIPX5D3X4jz1scDzLOJnCByoFrbdZLAMX?= =?us-ascii?Q?iiPUF2Fo9HqLO/DGiUKYs4xTzY8bOcLodAFE7487GaUfrw//aWVAxVl7EqFL?= =?us-ascii?Q?IwZvsQtJm+cUkNAIp3DOuL9ZiVS4/PnP6d82j7v5glEjWb2IXuRLjYUIQPpv?= =?us-ascii?Q?4bzHslij7ILViPpiiSGJiju2RAPUCAH2tgfk6Tx2aSB1fDZqcnwqrzQMNQfq?= =?us-ascii?Q?uVRU8FmFxBWHikC/v5VLOkSatF8pPHdgN/q4Mq/TLtzeeYZsXnjeZ584dCbu?= =?us-ascii?Q?/xjAbhOJYLyNwnt3fVc9GHQg/NtYhK7MRFkRRpja8IB27WQcPtIpVO/ge2lg?= =?us-ascii?Q?qsl9fv+buSiAV40/FZiUh4ad31O00Fwb3Ougf3/FGHeclyWVpUzIvPWZFsso?= =?us-ascii?Q?7eAQKEj9+IWKfNFPBLHxGccYt5aOLSJsI9nlkM1QFTqHPYn1Yec8xwxo711l?= =?us-ascii?Q?yFRR2ZJ4+bHo4EgRX/BbHdVhnTzK8HmD8AU/y2DKNF22XEpb3ZytrEQt8agp?= =?us-ascii?Q?y++x31jb5BT4T3+kaO5mzKm5/XHFtgdM646hj7LzjEUqKNR8+BJopvqNsUIt?= =?us-ascii?Q?bc3zLQknabUFqhH4lKubVSo8eLHkz5QzyUZqt4hCdnXSJ/jqkQuzWzGTvQV4?= =?us-ascii?Q?JUVpPp1pOg/82Nu4G+F7qf2wLALtXvUpeH+pDMaMJpn7/k+fn/qtIg6+YlV3?= =?us-ascii?Q?5pnl9ejUWbxW1+iuIPldau1vYBvldXTKjronrX6g1duAfO9kwzucMJIru+4C?= =?us-ascii?Q?IAmMaAoBUKGU+rsOtWY+IDoJtUBfzQS4T2xafZoENLJU/QzIbqiYyeQdflum?= =?us-ascii?Q?PeaSYC+HFxkywlqzXveTQ+euajSotgeYWo6Vd4Fa4j4j9JtdA0aSrMOEH+GK?= =?us-ascii?Q?a+0eIFG/eMeDR0Gpjrr3KwfSV1xxFAGpNBmcV6b4StQbUvQotsa2MIZ1IheY?= =?us-ascii?Q?8fKbZF+KO3KHErimy6cQ/ZbHkqQM8fRLcIrWvZwfxhF0n9c/EOWQl3LaIpIT?= =?us-ascii?Q?MNZ/JoQhWqXUQVktY6ZxZxLEFzCzK/D41PvyW6iPZm3SFbzLOckW3Y0QkcBr?= =?us-ascii?Q?ZLsB6DpLsf2WX/oWJ1TC/OtXBUlYHK6DW/CJ0DDTgSDUA+/3h4aGd0Kx/dMw?= =?us-ascii?Q?F6+QGRajVl20zSzaur0DCsfsb7mgHyBjqZw6hJPRSx6TKENlG5Py8XO3ESxj?= =?us-ascii?Q?9BzwuIcdQtuML5XZ8T39LhIUd0PrHI/oFfV5JAcqP8LDeKaiRvTJjDYGzrUw?= =?us-ascii?Q?wRBfDAIWOO6YXNgXGfBbZ1YjGdKyyTSy4DX78HUvfaIc0KCRyem9e8xCskea?= =?us-ascii?Q?ifkfJo32zpf8BteJY761p5DKq5Cv5LlmQAS6NMhMxxwDeqLbnla9g/oQbjwA?= =?us-ascii?Q?U2QfkoR8rEJRJkOqHSzg1eaBVggan6heKKKQitTORjdtg9mHKNdldSzWNOuB?= =?us-ascii?Q?9l50TN0eVzY2ImPCfH3OaMPbyj1aX6DSIFbfyIdVSD+KTKnwFxoYlgNGv5Ak?= =?us-ascii?Q?GzUczouPHYLEEeHLy+w6YRPZQSzVDgWtHqjX5uwEA0QujhJVBkv2w5zTkV52?= =?us-ascii?Q?3YiweqiUeisLyiRCEIpoXuKCnEEOTXl9dZHr3NhMm6yRfQsDvREhGcpGrXn+?= =?us-ascii?Q?lNOUxp5kGthCnsT6TZ/2TrzH+6f/Rg8523Z+Zz8S6m9aRvCOT1vAGIRFd4U5?= =?us-ascii?Q?1tnJDb0oOYrpde3UX6tpntnMAUDpMuNygJW8nGnXVIKZexDh63kGuPJXzsDa?= =?us-ascii?Q?c7LU3yRLRQ=3D=3D?= X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: ba960f1a-098b-4f4c-a183-08deb72629c0 X-MS-Exchange-CrossTenant-AuthSource: PH8PR12MB7277.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 21 May 2026 10:46:07.4589 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: lRvGG5nDBRogWz+ZyXhql2/h1yJV4hdasYF5pqVl52wD/iamZbINpuaNPJh1aF1W0CfCqk430gnjxjj8FNWe2A== X-MS-Exchange-Transport-CrossTenantHeadersStamped: CH1PPF5A8F51299 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.9.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20260521_034621_562368_EA71924F X-CRM114-Status: GOOD ( 31.50 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org On Thu, May 21, 2026 at 09:50:04AM +0100, Ryan Roberts wrote: > On 21/05/2026 05:24, Alistair Popple wrote: > > Hot unplugging memory on ARM64 requires a TLB invalidate after unmapping > > the page to be hot unplugged from the direct map. Currently that happens > > one page at a time, meaning range based invalidates cannot be used. The > > result of this is that removing large amounts of memory takes a long > > time and in some cases can trigger an RCU stall warning. > > > > For example on one system hot unplugging 480GB of memory takes ~1 > > minute. With this change the same operation took ~1 second, a 60x > > improvement. > > > > Signed-off-by: Alistair Popple > > > > --- > > > > This is an RFC, because I'm not sure the change is correct as it frees > > the PTE page before flushing the TLB. I'm not familiar enough with ARM64 > > architecture to be sure this is safe, for example I don't know if HW > > can update PTE bits such as access/dirty in the page through a stale > > TLB entry. > > > > If so this would open a window during which the page is free but could > > still be written to. Likely the safe option would be to collect all the > > pages to be free on a list and free them after doing the range based TLB > > flush, but wanted to get feedback on the approach before implementing it > > which is the goal of this RFC. > > Hi Alistair, > > This patch doesn't apply on v7.1-rc4 because it conflicts with this patch: > > Commit 48478b9f79137 ("arm64/mm: Enable batched TLB flush in unmap_hotplug_range()") > > which has a very similar performance improvement, so hopefully it solves your > problem? > > There are two paths which use this logic; unmapping the linear map and unmapping > the corresponding vmemmap. In the latter case, the memory is also freed, so we > can't safely do the range optimizaiton there since the TLB needs to be flushed > before freeing the memory. But the linear map is the big, slow bit so hopefully > it's sufficent for you? > I assume vmemmap path is for tearing down the struct pages corresponding to the physical memory and vmemmap teardowns taking a flush should be OK. It is worth checking if the issue is already fixed. Balbir