From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (prime256v1) server-digest SHA256) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 89DDD606EEBF for ; Tue, 3 Nov 2026 16:01:19 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key, unprotected) header.d=lists.infradead.org header.i=@lists.infradead.org header.a=rsa-sha256 header.s=bombadil.20210309 header.b=kAkun3NT; dkim=pass (2048-bit key, unprotected) header.d=kernel.org header.i=@kernel.org header.a=rsa-sha256 header.s=k20260515 header.b=IMVFroQ1 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Type:MIME-Version: Message-ID:Date:References:In-Reply-To:Subject:Cc:To:From:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Unsubscribe-Post:List-Owner; bh=PSyuWChpGynvLRHVjaO/iLjkmV2JYaTq3nj516QvjuM=; b=kAkun3NTx99HrHg0upNFMX5kNu bJSiTV6W8PJOSSQoRma+STjx7kxfQWsSXLMnSL4MNtvY26aayZsWz10khiOrg8c9yIHglZHBQw3v7 Qe571JtdJ1JfR0vXLNA1hH5QXapkN+AZCBE1r4jFXPTi/YEU1YImdxHyGvX6X2iu5/TAZdMMaRvOZ aHH1k2N837DUbV1TzZ3xZUCcne2iV4l+IRmUM4RUE7h0CRWkiogUEGzqwmdI79rDjb/OAKMX9wcj/ 7xUgw4X7Jr1Ffq6+k2uftMDO35/edXVHqttyZZytLz3GgWLfYbGnqFDTVUZKLsgnxPCCWWN6AEhhQ LjZqBdAg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.100.1 #2 (Red Hat Linux)) id 1xOGwp-000000032tN-0Zas; Tue, 03 Nov 2026 16:01:19 +0000 Received: from sea.source.kernel.org ([2600:3c0a:e001:78e:0:1991:8:25]) by bombadil.infradead.org with esmtps (Exim 4.100.1 #2 (Red Hat Linux)) id 1xOGwi-000000032ss-2I3O for kexec@lists.infradead.org; Tue, 03 Nov 2026 16:01:12 +0000 Received: from smtp.kernel.org (quasi.space.kernel.org [100.103.45.18]) by sea.source.kernel.org (Postfix) with ESMTP id 4FFB141950; Tue, 3 Nov 2026 16:01:12 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 3FD601F000C1; Tue, 3 Nov 2026 16:01:10 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel.org; s=k20260515; t=1793721672; bh=PSyuWChpGynvLRHVjaO/iLjkmV2JYaTq3nj516QvjuM=; h=From:To:Cc:Subject:In-Reply-To:References:Date; b=IMVFroQ1qx1z6bfBCYYz2bd8YHfKv5JB8R3RCU2WD1829BGMlrHnRpsj+rN4xW3vd ZHbnv7fZjTfyrBTs/osD2c56v7mzj5rkowlYlds9EKi9XFi9qNHQA59tC1YV+3WDX9 +NBpWvuqgOcrnCNFqbYFU3q0mPYQdazovNqWNMSixO/UKk2+G5tY6j/E1YpuqVwfV9 zArRzpk0pAAyrJVt5qr2GHNQqvb6FXHKeuhiM20TrQH1TprTy9ER+nGvf4xM6LtL+5 wPS7M2ctjVhyFjN3Lr7BxkWryUmDi+wIBrA7mBrsP0ghIhLE8r2vVU4lGuxKfppnOx vSXb6Ati0bilA== From: Pratyush Yadav To: "Mike Rapoport (Microsoft)" Cc: Pasha Tatashin , Pratyush Yadav , Alexander Graf , Andrew Morton , Chen Pei , George Guo , Kexin Liu , Sourabh Jain , kexec@lists.infradead.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: Re: [PATCH 2/2] kho: fix bootmem sizing In-Reply-To: <20261103-kho-bootmem-sizing-v1-2-45aa74098f34@kernel.org> (Mike Rapoport's message of "Tue, 03 Nov 2026 13:49:50 +0200") References: <20261103-kho-bootmem-sizing-v1-0-45aa74098f34@kernel.org> <20261103-kho-bootmem-sizing-v1-2-45aa74098f34@kernel.org> Date: Tue, 03 Nov 2026 17:01:08 +0100 Message-ID: <2vxz5wydnbuj.fsf@kernel.org> User-Agent: Gnus/5.13 (Gnus v5.13) X-Archived-By: MIME-Version: 1.0 Content-Type: text/plain X-BeenThere: kexec@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "kexec" X-Spam-Status: No, score=-17.4 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_SANE_1 autolearn=unavailable autolearn_force=no version=4.0.1 X-Spam-Checker-Version: SpamAssassin 4.0.1 (2024-03-26) on aws-us-west-2-korg-listserv-alma10-1 Hi Mike, On Tue, Nov 03 2026, Mike Rapoport (Microsoft) wrote: > Bootmem areas are sized from overlapping views of the memory reserved in > memblock: > > * the global area is sized from all reserved memory, including memory > allocated on specific nodes, so on NUMA systems everything allocated > on a node is accounted in both the global area and the area of that > node. > * per-node sizes are calculated after the lowmem and the global areas > are reserved. With CONFIG_NUMA=n all memblock regions belong to node 0, > so the lowmem and the global areas are accounted in the size of node 0 > area and scaled once again. > > As a result, bootmem is about twice bigger than it should be on NUMA > systems and about four times bigger with CONFIG_NUMA=n. On small systems > reservation of the bootmem may fail and then KHO gets disabled. > > Make the areas cover separate parts of the reserved memory: the lowmem > area covers allocations below ARCH_LOW_ADDRESS_LIMIT, per-node areas > cover allocations above it on each node and the global area covers > allocations that are not bound to any node. Calculate all sizes before > reserving any of the areas. > > With CONFIG_NUMA=n the global area becomes empty and it is not reserved. > > Bootmem sizes in MiB in x86 QEMU VMs, before -> after: > > CONFIG_NUMA=y CONFIG_NUMA=n > 1G, 1 node 176 -> 92 320 -> 80 > 4G, 1 node 508 -> 332 1264 -> 316 > 16G, 2 nodes 1268 -> 704 2944 -> 736 > 32G, 4 nodes 2296 -> 1224 5196 -> 1300 > 64G, 4 nodes 4344 -> 2248 9664 -> 2420 > > Fixes: 3dc92c311498 ("kexec: add Kexec HandOver (KHO) generation helpers") > Reported-by: Kexin Liu > Reported-by: George Guo > Closes: https://lore.kernel.org/all/20260904025101.9959-1-dongtai.guo@linux.dev > Reported-by: Sourabh Jain > Closes: https://lore.kernel.org/all/20260922131217.698809-1-sourabhjain@linux.ibm.com > Signed-off-by: Mike Rapoport (Microsoft) Thanks for the patch! My LLM caught an interesting problems that is worth considering. On arm64, ARCH_LOW_ADDRESS_LIMIT is arm64_dma_phys_limit - 1, and dma_limits_init() sets the default value to PHYS_MASK + 1, and it can also be memblock_end_of_dram() in some cases. So on arm64, it is possible for lowmem to span the entire RAM. Currently, that results in over-allocation of bootmem, and it ends up being 4x what it should be. While obviously problematic, it should still work. With this patch, all memory is in lowmem, and nothing ever goes to per-node bootmem (except for the node that hosts the lowmem area). All this is from reading the code, I haven't tried anything out yet. Not sure how to fix this though. Should we perhaps use min(ARCH_LOW_ADDRESS_LIMIT, U32_MAX) for the lowmem limit? Also, a couple nitpicks below. > --- > kernel/liveupdate/kexec_handover.c | 83 ++++++++++++++++++++------------------ > 1 file changed, 44 insertions(+), 39 deletions(-) > > diff --git a/kernel/liveupdate/kexec_handover.c b/kernel/liveupdate/kexec_handover.c > index f268c57413a7e..eaeb895bb7595 100644 > --- a/kernel/liveupdate/kexec_handover.c > +++ b/kernel/liveupdate/kexec_handover.c > @@ -684,6 +684,7 @@ static unsigned int bootmem_scale __initdata = 200; > static phys_addr_t bootmem_size_global __initdata; > static phys_addr_t bootmem_size_pernode __initdata; > static phys_addr_t bootmem_size_lowmem __initdata; > +static phys_addr_t bootmem_size_node[MAX_NUMNODES] __initdata; > > static int __init kho_parse_bootmem_size(char *p) > { > @@ -753,29 +754,48 @@ static int __init kho_parse_bootmem_size(char *p) > } > early_param("kho_bootmem", kho_parse_bootmem_size); > > +static phys_addr_t __init reserved_mem_size(phys_addr_t limit, int nid) > +{ > + return memblock_reserved_kern_size(limit, nid) - > + memblock_reserved_hugetlb_size(limit, nid); > +} > + > +static phys_addr_t __init bootmem_scale_size(phys_addr_t size) > +{ > + return size * bootmem_scale / 100; > +} > + > +/* > + * The lowmem area covers allocations below ARCH_LOW_ADDRESS_LIMIT, per-node > + * areas cover allocations above it on each node and the global area covers > + * allocations that are not bound to any node. > + * > + * All sizes must be calculated before any area is reserved, otherwise the > + * areas reserved first will be accounted in the sizes of the others. > + */ > static void __init bootmem_size_update(void) > { > - /* > - * If fixed sizes are not provided via command line, calculate them now. > - * Remove HugeTLB allocations from it because they never get allocated > - * from bootmem. > - */ > + int nid; > + > if (bootmem_scale) { > - phys_addr_t size; > - > - size = memblock_reserved_kern_size(ARCH_LOW_ADDRESS_LIMIT, > - NUMA_NO_NODE); > - size -= memblock_reserved_hugetlb_size(ARCH_LOW_ADDRESS_LIMIT, > - NUMA_NO_NODE); > - size = size * bootmem_scale / 100; > - bootmem_size_lowmem = size; > - > - size = memblock_reserved_kern_size(MEMBLOCK_ALLOC_ANYWHERE, > - NUMA_NO_NODE); > - size -= memblock_reserved_hugetlb_size(MEMBLOCK_ALLOC_ANYWHERE, > - NUMA_NO_NODE); > - size = size * bootmem_scale / 100 - bootmem_size_lowmem; > - bootmem_size_global = size; > + phys_addr_t lowmem, global, size; > + > + lowmem = reserved_mem_size(ARCH_LOW_ADDRESS_LIMIT, NUMA_NO_NODE); > + global = reserved_mem_size(MEMBLOCK_ALLOC_ANYWHERE, NUMA_NO_NODE); > + global -= lowmem; Nit: Please add a short comment above this to explain why you remove lowmem from global. > + > + for_each_node_state(nid, N_MEMORY) { > + size = reserved_mem_size(MEMBLOCK_ALLOC_ANYWHERE, nid) - > + reserved_mem_size(ARCH_LOW_ADDRESS_LIMIT, nid); > + bootmem_size_node[nid] = bootmem_scale_size(size); > + global -= size; Nit: same here, a comment here would make it easier to understand. > + } > + > + bootmem_size_lowmem = bootmem_scale_size(lowmem); > + bootmem_size_global = bootmem_scale_size(global); > + } else { > + for_each_node_state(nid, N_MEMORY) > + bootmem_size_node[nid] = bootmem_size_pernode; > } > > /* > @@ -784,24 +804,9 @@ static void __init bootmem_size_update(void) > */ > bootmem_size_lowmem = round_up(bootmem_size_lowmem, BOOTMEM_ALIGNMENT_BYTES); > bootmem_size_global = round_up(bootmem_size_global, BOOTMEM_ALIGNMENT_BYTES); > -} > - > -static phys_addr_t __init bootmem_size_node(int nid) > -{ > - phys_addr_t size; > - > - if (bootmem_scale) { > - size = memblock_reserved_kern_size(MEMBLOCK_ALLOC_ANYWHERE, > - nid); > - /* Do not count HugeTLB pages. */ > - size -= memblock_reserved_hugetlb_size(MEMBLOCK_ALLOC_ANYWHERE, > - nid); > - size = size * bootmem_scale / 100; > - } else { > - size = bootmem_size_pernode; > - } > - > - return round_up(size, BOOTMEM_ALIGNMENT_BYTES); > + for_each_node_state(nid, N_MEMORY) > + bootmem_size_node[nid] = round_up(bootmem_size_node[nid], > + BOOTMEM_ALIGNMENT_BYTES); > } > > bool kho_bootmem_overlap(phys_addr_t phys, size_t size) > @@ -891,7 +896,7 @@ static void __init kho_reserve_bootmem(void) > * memoryless nodes, as we can not allocate bootmem areas there. > */ > for_each_node_state(nid, N_MEMORY) { > - if (kho_reserve_bootmem_area(&i, bootmem_size_node(nid), > + if (kho_reserve_bootmem_area(&i, bootmem_size_node[nid], > MEMBLOCK_ALLOC_ACCESSIBLE, nid)) { > pr_err("Failed to reserve nid %d bootmem\n", nid); > goto err_free_bootmem_areas; -- Regards, Pratyush Yadav