From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 1D351CCFA13 for ; Wed, 29 Apr 2026 17:09:13 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:MIME-Version:Content-Type: Content-Transfer-Encoding:References:In-Reply-To:Message-ID:Date:Subject:Cc: To:From:Reply-To:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=7tlHlWRAO2UmHsFvNbnNZA38HOlN2Fp5E0nYo+QXLLM=; b=QCBuW4IBPUImSTjHTH+OzVKdEp IanoiZuOWf1dKqDHxPFIvJAOZadmMPceZd9vuW6rBuRsfuAdupZD756baYWpL53TEwBqUZXCpx2Rg AvvYxBpyKK70+PAZ3sII8zjOwK0l6N24v3UoDKjkuzYAAPfYB7qo2wTtEDtWtOGLJxueDwkFZr48D 50eVSwja+CXvyRdFmze1+4yaXqX6CWhtBYXX6wEs3aZEpYKvFU7RIzw7p7FakNe2Mwn4E4picogKA PsNNYbvOPAnUOYbpFltjBXIvSuYRoCG1DzeubsbiliedgK0ddkThSp/Bc9kkPfbG/KPPZNXHLLybs Jd4UgNsA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98.2 #2 (Red Hat Linux)) id 1wI8PK-00000003w9X-3jjj; Wed, 29 Apr 2026 17:09:06 +0000 Received: from mail-southcentralusazlp170120001.outbound.protection.outlook.com ([2a01:111:f403:c10d::1] helo=SN4PR2101CU001.outbound.protection.outlook.com) by bombadil.infradead.org with esmtps (Exim 4.98.2 #2 (Red Hat Linux)) id 1wI8PH-00000003w7G-3uA6 for linux-arm-kernel@lists.infradead.org; Wed, 29 Apr 2026 17:09:05 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=ewyavoI0azou8iYv7caOliqtUou8eu+5rkpZR8Pp6r0sWC0pBhe1+3fRixv0CS7Ownwigfd3Wo7XUwoxmLuFmtJB2SWnYE74gGs9ZyXO/UF/fIPtmpTPy8pM2W2i1EZ2Vva0A5HELh2ckldQ1fJUpcs9vwkbjMS7KQd1meRMbBtgfeDMxbNqfo4yrk+uHpKfVCX4GaS99Vt+s050LwpXumskE9jtJAYdJgopNfX6He/qMIxL7+R1ffYHUWnDLiRMRXoILMybp0t9hswZTSbd6Sb7/Yd5LsuTxxM/GA6uDpm0Wu1DRT6hELC2OrtOf0HLooQxgGG222hs9+U+LfyGhw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=7tlHlWRAO2UmHsFvNbnNZA38HOlN2Fp5E0nYo+QXLLM=; b=HvB65jXxwCbCnGKtLpi5Uu9xuwtEwE4Bz5HzPUiZ75eHVAleoic4cXxsAF5ycbuJhlndMWMvAWrFMEeEuFgGbc2Rvz4dphBTnyA5pTGHPU8XUeCugojcPUL1daWtuhS2iadrGiKENBcvu9VbzJTyXe2gUFmDheQpW2/91KawpSjQRWnhc+O/yOpcld/8WUmJkH4rG4Y1LMnsbmwL7JT3mvp/5g8CdHmYVh9fMmGy+GrT94qYGPzTuxTCzhTLu/8X5iV1OAymRwPgbEcZWgnP2/QTeqJVYkJ7JUNcLe5he+h4o3VdmXR9wQ3RZdXonl8pbsBaJCABR8+ONMWuyTpxug== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=os.amperecomputing.com; dmarc=pass action=none header.from=os.amperecomputing.com; dkim=pass header.d=os.amperecomputing.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=os.amperecomputing.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=7tlHlWRAO2UmHsFvNbnNZA38HOlN2Fp5E0nYo+QXLLM=; b=t4zm0FuQ+CEbNLti2rs60K9ua3M2iBkIjGnIlCuLS6g8uN4NOXaXU2eq+8bGsEkVVXr7eELOeZfe4Hv61aK19/ixWkra55mQfDwMD++oGT6LxpRmBt95666lbd1+xlvT4UmzWEAo2Wru0RmLyvXKiVH/pIJXUaYSq7zROeD/GJo= Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=os.amperecomputing.com; Received: from CH0PR01MB6873.prod.exchangelabs.com (2603:10b6:610:112::22) by SA1PR01MB6671.prod.exchangelabs.com (2603:10b6:806:18b::10) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.9846.26; Wed, 29 Apr 2026 17:08:55 +0000 Received: from CH0PR01MB6873.prod.exchangelabs.com ([fe80::46eb:64a3:667c:c1a0]) by CH0PR01MB6873.prod.exchangelabs.com ([fe80::46eb:64a3:667c:c1a0%4]) with mapi id 15.20.9870.020; Wed, 29 Apr 2026 17:08:55 +0000 From: Yang Shi To: cl@gentwo.org, dennis@kernel.org, tj@kernel.org, urezki@gmail.com, catalin.marinas@arm.com, will@kernel.org, ryan.roberts@arm.com, david@kernel.org, akpm@linux-foundation.org, hca@linux.ibm.com, gor@linux.ibm.com, agordeev@linux.ibm.com Cc: yang@os.amperecomputing.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org Subject: [PATCH 02/11] arm64: mm: define percpu virtual space area Date: Wed, 29 Apr 2026 10:04:30 -0700 Message-ID: <20260429170758.3018959-3-yang@os.amperecomputing.com> X-Mailer: git-send-email 2.52.0 In-Reply-To: <20260429170758.3018959-1-yang@os.amperecomputing.com> References: <20260429170758.3018959-1-yang@os.amperecomputing.com> Content-Transfer-Encoding: 8bit Content-Type: text/plain X-ClientProxiedBy: SA0PR13CA0008.namprd13.prod.outlook.com (2603:10b6:806:130::13) To CH0PR01MB6873.prod.exchangelabs.com (2603:10b6:610:112::22) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CH0PR01MB6873:EE_|SA1PR01MB6671:EE_ X-MS-Office365-Filtering-Correlation-Id: b468d02f-6f97-48bd-0d39-08dea611fe88 X-MS-Exchange-AtpMessageProperties: SA X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|1800799024|376014|7416014|366016|18002099003|56012099003|22082099003|55112099003|921020; X-Microsoft-Antispam-Message-Info: eGZfahnj+uGAOYQBcBA3BGhwnqTN78NMCaRzFqqH9wpvNlulkUMNI5OzBdDeRzAM88dvhNpIiyUpzF+vs79zFmKgKFjK0gUHYRbIzPOjrD0KfVyjtiC5/0Gosyhlej3ecV3BRodFhFnDB2ajUU0bF/Xn2lDcqiODJqnI9g8GrGyQ2/YDeEpqwVqP97V3j6q+KKJO5cCmQkIQiIZH/eGq9uUKp6bcR/TVpNDFSIHr0C/ZafAWMuxBHLwlJLpSBl8blifWdiF3JZ4tp4DOkfcu36os4j0VlfHKv1RI0M+X7yNnqH+Wyf/c103L3mjJDjvx6TuWiEkmUtViOtmXxz3Bhm6zLpRR3nRlTjOCLUGuVRQ0jgXuCd6TyOLwA+YADmx0xV/BR30Mz13zp4kgzaMyKtS0ehiHzuJ+yX8eQskxi7YMxGD+GGkuHsWrX4KCvelUr//tslUhtPPzna53ydOenxxM+jHa43FPOdp+xec5A1xJqBiRr7UrGCpWiFSt05H/wmkOm4aXzmdTEkPaeMu/MKwNOl6GwQNbmJCKR3YGKqn1F6MroG5/fDe/KS75rdYKseLFltSt4DZ8t9N0QwE+GNeftfE+Ii8ACCBhHp8p1Hbou814COB3zm7YIg/d4buj2LnjEs4m/d9myFhjE5+hERIqPnXWrhTLCZ/WdyKedy6oR7k7ZWWc+S93r6dcA2/AOf7GFbXiBJ4g0hCJ3/zZqIwii/UiXc89X4GHS3opWf1LMuDJiRHyKxnfDIswPHRNTM1EUpzxJG585IdOIQiS2g== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:CH0PR01MB6873.prod.exchangelabs.com;PTR:;CAT:NONE;SFS:(13230040)(1800799024)(376014)(7416014)(366016)(18002099003)(56012099003)(22082099003)(55112099003)(921020);DIR:OUT;SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?S6oSVvyXcGxjfsDpjjA+aG2UTmoHu8KlO2h0y73D4V++0tgt86bsLhLtUdzq?= =?us-ascii?Q?4i7IQ4jSyR7YWgTJp70wc3nyj9UK62vJdxpFSzTSzOTc59NKnVSIP567I+T+?= =?us-ascii?Q?p+Hj7BcJJ53j19g+MlfsXOwclEzFzulGvjTOwXx7YlITRQHM3u7z6mGdizJz?= =?us-ascii?Q?kjl1XLGIZxRFv2y+eZ8e/wathSX4PNHKu8IT4fKiQ5Rqldh2tFYuxk11OHRr?= =?us-ascii?Q?4kn99y6BWCOqqmVXWe/EURdW3SiXTllJMmLLkxBL61S19srNPDA5VZUfLH+3?= =?us-ascii?Q?ehU+PLiVdNZKnUiTzmbvG11DwawCzDi/QeypmGz+rU5Qibpd4f6He8+ygZAm?= =?us-ascii?Q?XAXwI7PhFA84EE/jToXzW3HW7e91CpPPcdEqGZQs9zCPFj3lUfnhn/0hqLUP?= =?us-ascii?Q?0iiIr02ZZp7SZk4w9a3wwVvVQmMwcBpP1d8AYUpxR3NXMHHMZ7vrJ43cPs2M?= =?us-ascii?Q?kZN12g6Oae9rgQjI/GUgZWP0G3b3ynxixT1KVz7VR2+YeaDNo7wYxr1gXXud?= =?us-ascii?Q?jDN4GQk2j238EN0ZzXlseGGVL1+ZjuMlQSHOMS0KoBL3lD15idgOqAwYMX7a?= =?us-ascii?Q?HknK3/HZ1RMRLuDMbfd544d9+fUUWoK9SPyShPahMuWwSiPQ2h5pHczpwafa?= =?us-ascii?Q?t+B393+00byF06hE2MxK6TtDz2Uws/8hgwmh4kfOKKUTauaOMYHJu1bZIslP?= =?us-ascii?Q?Sra9Ioo79LQAhiNre6uIg0TCqlLzoF6T0Lkx0sGFNCaM8zqzgNeXZRJ9bfVf?= =?us-ascii?Q?oJDu6crudpMgN5P65UTbUytIoZncsZySCTV7Y8yEOKHBfw4TWU46NnpB3LMI?= =?us-ascii?Q?UIu2Nq8dWz3osTqcM7+JxHKjrGSgoCFpbpXDLNTi/ONuB4ADoT4sWwwugXdj?= =?us-ascii?Q?mhxNojDslU/IO3UBCpbQBZQCIj/y0h/MQKkoRe/ddfFgU69QQDyJnEUuuW+6?= =?us-ascii?Q?3zJCRuAjB24dDW9XJU/QVdfiSJhDqP6Q95nFTsF7M6zvbVyhqEqvBfiqgJmt?= =?us-ascii?Q?DfZGjKSzjR7oJ8LhIScDw/WKHhW63EsPvG7LVRZwSZ7B+n3ycg7FaGdT+Vyk?= =?us-ascii?Q?2ToKMRhBdblwMjIaxcR6ZOBn4BoztObxw6P/lyiML18vRDK1zAvzqifH9Cbt?= =?us-ascii?Q?cC73B+KPjZTsGClY8YqBvr1ZJCXF8l3YLGfrHER6v40A2iimQguHiKFoyjwh?= =?us-ascii?Q?BJqyGYVoh1zCuH7Sif3ieF59gfjzZXJOLDh38qlWC5KbnAaZBOI9s5YwL8pE?= =?us-ascii?Q?k1wcqmxwWfsPKQUFSAYGPvPkvOW/RwUCuTBw8dNubuna5ZxQ11C3vX0WuZms?= =?us-ascii?Q?hr8iZKhhhQGNG8eCWW1PfOdD1SHMwaajzd+KoN8/178QnTImGsq8f0Fl3nbX?= =?us-ascii?Q?RQQpQDN21dE9GFt8Th27y360s67zRL8oe1jvKpZqFr5pbeDnEEgs+GR89t+W?= =?us-ascii?Q?AdIBfY9mmRSaZdml5WNWD1mqRAh2fyoHRV7Gwmgrsv3EE4Txbv8h8JsFo4F/?= =?us-ascii?Q?lHD+mxKj3w+TS6xK0HYMBmMWIkybhNuIDYH2ZK5QSd9ZOdN3fDsFOdsGVChe?= =?us-ascii?Q?IHFP3iDOgvEiSVtpYZdneTiuWjmMzzE2/T3Q2wQDZPW6ivgA91u/xuo5e5xy?= =?us-ascii?Q?7PtiHf7OOF6ImDR/qWiqN967/aSI8UjMrEcAIypxPWwHgDHWROR+AZACDhLw?= =?us-ascii?Q?aBVrRgPGSL4JfbYfnXWZqsu8pa90oF1/uVTPGzLUJq8w8UmsO4K0i//KwJOz?= =?us-ascii?Q?pim05LSSZQFCMemZmjzbGvYUewEkS6k=3D?= X-OriginatorOrg: os.amperecomputing.com X-MS-Exchange-CrossTenant-Network-Message-Id: b468d02f-6f97-48bd-0d39-08dea611fe88 X-MS-Exchange-CrossTenant-AuthSource: CH0PR01MB6873.prod.exchangelabs.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 29 Apr 2026 17:08:55.0033 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 3bc2b170-fd94-476d-b0ce-4229bdc904a7 X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: Kl0fYc8kFE9rJDKDAr6XpPad3/iHBekRoKPFjHCls5jep0BIp2EMqWhISsS5Qi1mwqEnbbRkWxzHRihCF5gy5W5SVQIrlTzHJV12lDQiYto= X-MS-Exchange-Transport-CrossTenantHeadersStamped: SA1PR01MB6671 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20260429_100903_973324_F27A94FD X-CRM114-Status: GOOD ( 17.40 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org The percpu allocator returns offset from percpu base address. The percpu base address is determined by the first chunk which is typically in the low address of vmalloc space, however percpu varriables are typically allocated from the high address of vmalloc space. So the offset could be quite big. It may be the whole size of vmalloc space. To support local percpu mapping in order to optimize this_cpu_*() ops, the percpu allocator needs to allocate memory from local percpu area too in the following patch and the offset to local percpu base address must be same because the offset returned by percpu allocator must be used to access both global percpu and local percpu. We can half vmalloc space to have either half dedicated to local percpu, but it wastes too much address space. So carve out dedicated global percpu and local percpu areas. Each area size is 2 * PGDIR_SIZE. It is 1TB with 4K page size, should be big enough for percpu. The percpu areas are PGDIR_SIZE aligned in order to just need to sync percpu page table at pgd level to minimize page table sync overhead. The kernel virtual address space layout now looks like: +-----------------+ | Linear mapping | +-----------------+ | Modules | +-----------------+ | Vmalloc | +-----------------+ | Global Percpu | +-----------------+ | Local Percpu | +-----------------+ | Vmemap | +-----------------+ | PCI I/O | +-----------------+ | Fixed map | +-----------------+ Signed-off-by: Yang Shi --- arch/arm64/include/asm/pgtable.h | 21 ++++++++++++++++++--- arch/arm64/mm/mmu.c | 4 ++++ arch/arm64/mm/ptdump.c | 4 ++++ 3 files changed, 26 insertions(+), 3 deletions(-) diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h index 38eec71ec383..9043b976682c 100644 --- a/arch/arm64/include/asm/pgtable.h +++ b/arch/arm64/include/asm/pgtable.h @@ -18,14 +18,29 @@ * VMALLOC range. * * VMALLOC_START: beginning of the kernel vmalloc space - * VMALLOC_END: extends to the available space below vmemmap + * VMALLOC_END: extends to the space below global percpu area */ #define VMALLOC_START (MODULES_END) +#define VMALLOC_END (PERCPU_START - SZ_8M) + +/* + * PERCPU range + * + * PERCPU_START: beginning of global percpu area + * PERCPU_END: end of global percpu area + * LOCAL_PERCPU_START: beginning of local percpu area + * LOCAL_PERCPU_END: end of local percpu area, extend to the available + * space below vmemap + */ +#define PERCPU_SIZE (2 * PGDIR_SIZE) +#define PERCPU_START (PERCPU_END - PERCPU_SIZE) +#define PERCPU_END (LOCAL_PERCPU_START) +#define LOCAL_PERCPU_START (LOCAL_PERCPU_END - PERCPU_SIZE) #if VA_BITS == VA_BITS_MIN -#define VMALLOC_END (VMEMMAP_START - SZ_8M) +#define LOCAL_PERCPU_END (ALIGN_DOWN(VMEMMAP_START, PGDIR_SIZE)) #else #define VMEMMAP_UNUSED_NPAGES ((_PAGE_OFFSET(vabits_actual) - PAGE_OFFSET) >> PAGE_SHIFT) -#define VMALLOC_END (VMEMMAP_START + VMEMMAP_UNUSED_NPAGES * sizeof(struct page) - SZ_8M) +#define LOCAL_PERCPU_END (ALIGN_DOWN((VMEMMAP_START + VMEMMAP_UNUSED_NPAGES * sizeof(struct page)), PGDIR_SIZE)) #endif #define vmemmap ((struct page *)VMEMMAP_START - (memstart_addr >> PAGE_SHIFT)) diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c index ed1545baa045..7708dcc1b6a9 100644 --- a/arch/arm64/mm/mmu.c +++ b/arch/arm64/mm/mmu.c @@ -108,6 +108,10 @@ void arch_sync_kernel_mappings(unsigned long start, unsigned long end) if (!percpu_pgd_setup_done) return; + /* Don't sync local percpu area page table */ + if (start >= LOCAL_PERCPU_START && end < LOCAL_PERCPU_END) + return; + addr = start; do { pgd = READ_ONCE(*pgdp); diff --git a/arch/arm64/mm/ptdump.c b/arch/arm64/mm/ptdump.c index ab9899ca1e5f..7d5696a48917 100644 --- a/arch/arm64/mm/ptdump.c +++ b/arch/arm64/mm/ptdump.c @@ -389,6 +389,10 @@ static int __init ptdump_init(void) { MODULES_END, "Modules end" }, { VMALLOC_START, "vmalloc() area" }, { VMALLOC_END, "vmalloc() end" }, + { PERCPU_START, "Global percpu start" }, + { PERCPU_END, "Global percpu end" }, + { LOCAL_PERCPU_START, "Local percpu start" }, + { LOCAL_PERCPU_END, "Local percpu end" }, { vmemmap_start, "vmemmap start" }, { VMEMMAP_END, "vmemmap end" }, { PCI_IO_START, "PCI I/O start" }, -- 2.47.0