* [PATCH v4 02/21] kho: make radix max key width more obvious
2026-07-25 17:21 [PATCH v4 00/21] kho: make boot time huge page allocation work nicely with KHO Pratyush Yadav
@ 2026-07-25 17:21 ` Pratyush Yadav
2026-07-25 17:21 ` [PATCH v4 04/21] kho: return virtual address of mem_map from kho_get_mem_map() Pratyush Yadav
` (4 subsequent siblings)
5 siblings, 0 replies; 7+ messages in thread
From: Pratyush Yadav @ 2026-07-25 17:21 UTC (permalink / raw)
To: Mike Rapoport, Pasha Tatashin, Pratyush Yadav, Alexander Graf,
Muchun Song, Oscar Salvador, David Hildenbrand, Andrew Morton,
Jason Miu, Jork Loeser
Cc: kexec, linux-mm, linux-kernel
From: "Pratyush Yadav (Google)" <pratyush@kernel.org>
The KHO radix tree constants are somewhat hard to understand. The tree
depth essentially comes from the max key width. The max key width comes
from the need to store a 52-bit PFN plus one more bit for the order.
All this is very obscure with the corrent code. The PFN width is defined
as KHO_ORDER_0_LOG2, which makes very little sense to a new reader not
already familiar with what the value means. Then the fact that an extra
bit is needed is hidden in the KHO_TREE_MAX_DEPTH calculation.
Simplify this by removing KHO_ORDER_0_LOG2 and replace it with
KHO_RADIX_KEY_WIDTH. Update the comment to explain why this value is
used. This moves the +1 from KHO_TREE_MAX_DEPTH to KHO_RADIX_KEY_WIDTH,
making things clearer.
Update kho_{encode,decode}_radix_key() to not use KHO_ORDER_0_LOG2.
Instead, refactor the code and comments to make it clearer how the
encoding and decoding is done.
In kho_encode_radix_key(), add a new variable for the shift for physical
address. Use that in calculating where the order bit goes and in
calculating the shifted PFN. Update comments to make this clearer.
In kho_radix_decode_key(), turn order_bit to 0-indexed to simplify the
eventual calculation for order. Touch up comments to make the
computation clearer.
Signed-off-by: Pratyush Yadav (Google) <pratyush@kernel.org>
---
include/linux/kho/abi/kexec_handover.h | 9 +++------
kernel/liveupdate/kexec_handover.c | 19 +++++++++++--------
2 files changed, 14 insertions(+), 14 deletions(-)
diff --git a/include/linux/kho/abi/kexec_handover.h b/include/linux/kho/abi/kexec_handover.h
index 5e2eb8519bda..2f4fb9c63942 100644
--- a/include/linux/kho/abi/kexec_handover.h
+++ b/include/linux/kho/abi/kexec_handover.h
@@ -257,11 +257,8 @@ struct kho_vmalloc {
* memory. These constants govern the indexing, sizing, and depth of the tree.
*/
enum kho_radix_consts {
- /*
- * The bit position of the order bit (and also the length of the
- * shifted physical address) for an order-0 page.
- */
- KHO_ORDER_0_LOG2 = 64 - PAGE_SHIFT,
+ /* Need to store the PFN, plus one bit for order. */
+ KHO_RADIX_KEY_WIDTH = 64 - PAGE_SHIFT + 1,
/* Size of the table in kho_radix_node, in log2 */
KHO_TABLE_SIZE_LOG2 = const_ilog2(PAGE_SIZE / sizeof(phys_addr_t)),
@@ -274,7 +271,7 @@ enum kho_radix_consts {
* and 1 bitmap level.
*/
KHO_TREE_MAX_DEPTH =
- DIV_ROUND_UP(KHO_ORDER_0_LOG2 - KHO_BITMAP_SIZE_LOG2 + 1,
+ DIV_ROUND_UP(KHO_RADIX_KEY_WIDTH - KHO_BITMAP_SIZE_LOG2,
KHO_TABLE_SIZE_LOG2) + 1,
};
diff --git a/kernel/liveupdate/kexec_handover.c b/kernel/liveupdate/kexec_handover.c
index 7349cc82f6dc..ea24f23ce292 100644
--- a/kernel/liveupdate/kexec_handover.c
+++ b/kernel/liveupdate/kexec_handover.c
@@ -97,10 +97,12 @@ static struct kho_out kho_out = {
*/
static unsigned long kho_encode_radix_key(phys_addr_t phys, unsigned int order)
{
- /* Order bits part */
- unsigned long h = 1UL << (KHO_ORDER_0_LOG2 - order);
- /* Shifted physical address part */
- unsigned long l = phys >> (PAGE_SHIFT + order);
+ /* The physical address is encoded by shifting the PFN by its order. */
+ unsigned long shift = PAGE_SHIFT + order;
+ /* Order bit goes right before the shifted PFN. */
+ unsigned long h = 1UL << (64 - shift);
+ /* Shifted PFN. */
+ unsigned long l = phys >> shift;
return h | l;
}
@@ -118,12 +120,13 @@ static unsigned long kho_encode_radix_key(phys_addr_t phys, unsigned int order)
*/
static phys_addr_t kho_decode_radix_key(unsigned long key, unsigned int *order)
{
- unsigned int order_bit = fls64(key);
+ /* fls64() indexes starting from 1. */
+ unsigned int order_bit = fls64(key) - 1;
phys_addr_t phys;
- /* order_bit is numbered starting at 1 from fls64 */
- *order = KHO_ORDER_0_LOG2 - order_bit + 1;
- /* The order is discarded by the shift */
+ /* order bit goes right before the shifted PFN. */
+ *order = 64 - (PAGE_SHIFT + order_bit);
+ /* The order bit is discarded by the shift */
phys = key << (PAGE_SHIFT + *order);
return phys;
--
2.55.0.229.g6434b31f56-goog
^ permalink raw reply related [flat|nested] 7+ messages in thread* [PATCH v4 04/21] kho: return virtual address of mem_map from kho_get_mem_map()
2026-07-25 17:21 [PATCH v4 00/21] kho: make boot time huge page allocation work nicely with KHO Pratyush Yadav
2026-07-25 17:21 ` [PATCH v4 02/21] kho: make radix max key width more obvious Pratyush Yadav
@ 2026-07-25 17:21 ` Pratyush Yadav
2026-07-25 17:21 ` [PATCH v4 12/21] kho: add kho_radix_init_tree() Pratyush Yadav
` (3 subsequent siblings)
5 siblings, 0 replies; 7+ messages in thread
From: Pratyush Yadav @ 2026-07-25 17:21 UTC (permalink / raw)
To: Mike Rapoport, Pasha Tatashin, Pratyush Yadav, Alexander Graf,
Muchun Song, Oscar Salvador, David Hildenbrand, Andrew Morton,
Jason Miu, Jork Loeser
Cc: kexec, linux-mm, linux-kernel
From: "Pratyush Yadav (Google)" <pratyush@kernel.org>
Currently the preserved memory map address is returned by
kho_get_mem_map_phys(). It is only used by kho_populate().
kho_populate() doesn't use the actual value. It only cares that the
address exists and is valid.
In coming patches, more callers will be added, all of which will need
the virtual address of the preserved memory map. Since kho_populate()
doesn't care about the actual value and only cares about validity, it
can also use the virtual address returned by kho_get_mem_map(). It only
needs to make sure the returned value is not NULL.
Rename kho_get_mem_map_phys() to kho_get_mem_map() and return the
virtual address of the preserved memory map.
Signed-off-by: Pratyush Yadav (Google) <pratyush@kernel.org>
---
kernel/liveupdate/kexec_handover.c | 25 +++++++++++++++++--------
1 file changed, 17 insertions(+), 8 deletions(-)
diff --git a/kernel/liveupdate/kexec_handover.c b/kernel/liveupdate/kexec_handover.c
index e7451743b87e..5c35c11c273b 100644
--- a/kernel/liveupdate/kexec_handover.c
+++ b/kernel/liveupdate/kexec_handover.c
@@ -512,19 +512,24 @@ static int __init kho_preserved_memory_reserve(unsigned long key)
return 0;
}
-/* Returns physical address of the preserved memory map from FDT */
-static phys_addr_t __init kho_get_mem_map_phys(const void *fdt)
+/* Returns virtual address of the preserved memory map from FDT */
+static __init void *kho_get_mem_map(const void *fdt)
{
const void *mem_ptr;
+ phys_addr_t mem_map_phys;
int len;
mem_ptr = fdt_getprop(fdt, 0, KHO_FDT_MEMORY_MAP_PROP_NAME, &len);
if (!mem_ptr || len != sizeof(u64)) {
pr_err("failed to get preserved memory map\n");
- return 0;
+ return NULL;
}
- return get_unaligned((const u64 *)mem_ptr);
+ mem_map_phys = get_unaligned((const u64 *)mem_ptr);
+ if (!mem_map_phys)
+ return NULL;
+
+ return phys_to_virt(mem_map_phys);
}
/*
@@ -1647,9 +1652,8 @@ void __init kho_populate(phys_addr_t fdt_phys, u64 fdt_len,
{
unsigned int scratch_cnt = scratch_len / sizeof(*kho_scratch);
struct kho_scratch *scratch = NULL;
- phys_addr_t mem_map_phys;
- void *fdt = NULL;
bool populated = false;
+ void *fdt = NULL;
int err;
/* Validate the input FDT */
@@ -1671,8 +1675,13 @@ void __init kho_populate(phys_addr_t fdt_phys, u64 fdt_len,
goto unmap_fdt;
}
- mem_map_phys = kho_get_mem_map_phys(fdt);
- if (!mem_map_phys)
+ /*
+ * At this point phys_to_virt() doesn't work properly and so
+ * kho_get_mem_map() can return a pre-KASLR virtual address. But here we
+ * only want to make sure the mem_map is valid so the actual value
+ * doesn't matter as long as it isn't NULL.
+ */
+ if (!kho_get_mem_map(fdt))
goto unmap_fdt;
scratch = early_memremap(scratch_phys, scratch_len);
--
2.55.0.229.g6434b31f56-goog
^ permalink raw reply related [flat|nested] 7+ messages in thread* [PATCH v4 12/21] kho: add kho_radix_init_tree()
2026-07-25 17:21 [PATCH v4 00/21] kho: make boot time huge page allocation work nicely with KHO Pratyush Yadav
2026-07-25 17:21 ` [PATCH v4 02/21] kho: make radix max key width more obvious Pratyush Yadav
2026-07-25 17:21 ` [PATCH v4 04/21] kho: return virtual address of mem_map from kho_get_mem_map() Pratyush Yadav
@ 2026-07-25 17:21 ` Pratyush Yadav
2026-07-25 17:21 ` [PATCH v4 14/21] kho: initialize kho_scratch pointer earlier in boot Pratyush Yadav
` (2 subsequent siblings)
5 siblings, 0 replies; 7+ messages in thread
From: Pratyush Yadav @ 2026-07-25 17:21 UTC (permalink / raw)
To: Mike Rapoport, Pasha Tatashin, Pratyush Yadav, Alexander Graf,
Muchun Song, Oscar Salvador, David Hildenbrand, Andrew Morton,
Jason Miu, Jork Loeser
Cc: kexec, linux-mm, linux-kernel
From: "Pratyush Yadav (Google)" <pratyush@kernel.org>
Move the initialization logic of the radix tree into
kho_radix_init_tree() instead of having users open-code it. Makes the
boundaries cleaner and reduces code duplication when a new user of the
radix tree will be added in a future commit.
Signed-off-by: Pratyush Yadav (Google) <pratyush@kernel.org>
---
include/linux/kho_radix_tree.h | 7 ++++
kernel/liveupdate/kexec_handover.c | 62 +++++++++++++++++++++---------
2 files changed, 51 insertions(+), 18 deletions(-)
diff --git a/include/linux/kho_radix_tree.h b/include/linux/kho_radix_tree.h
index 66ca936b3f06..5d6ae2893684 100644
--- a/include/linux/kho_radix_tree.h
+++ b/include/linux/kho_radix_tree.h
@@ -54,6 +54,7 @@ int kho_radix_add_key(struct kho_radix_tree *tree, unsigned long key);
void kho_radix_del_key(struct kho_radix_tree *tree, unsigned long key);
int kho_radix_walk_tree(struct kho_radix_tree *tree,
const struct kho_radix_walk_cb *cb, void *data);
+int kho_radix_init_tree(struct kho_radix_tree *tree, struct kho_radix_node *root);
void kho_radix_destroy_tree(struct kho_radix_tree *tree);
#else /* #ifdef CONFIG_KEXEC_HANDOVER */
@@ -72,6 +73,12 @@ static inline int kho_radix_walk_tree(struct kho_radix_tree *tree,
return -EOPNOTSUPP;
}
+static inline int kho_radix_init_tree(struct kho_radix_tree *tree,
+ struct kho_radix_node *root)
+{
+ return 0;
+}
+
static inline void kho_radix_destroy_tree(struct kho_radix_tree *tree) { }
#endif /* #ifdef CONFIG_KEXEC_HANDOVER */
diff --git a/kernel/liveupdate/kexec_handover.c b/kernel/liveupdate/kexec_handover.c
index 105dc0f3e691..04bfcbf9cc55 100644
--- a/kernel/liveupdate/kexec_handover.c
+++ b/kernel/liveupdate/kexec_handover.c
@@ -320,6 +320,30 @@ static void __kho_radix_destroy_tree(struct kho_radix_node *root,
kho_radix_free_node(root);
}
+/**
+ * kho_radix_init_tree - initialize the radix tree.
+ * @tree: the tree to initialize.
+ * @root: root table of the radix tree.
+ *
+ * Initialize the radix tree with the given root node. If root is %NULL, an
+ * empty root table is allocated. If root is not %NULL, it is the caller's
+ * responsibility to make sure the root is valid and in the correct format.
+ *
+ * Return: 0 on success, -errno on failure.
+ */
+int kho_radix_init_tree(struct kho_radix_tree *tree, struct kho_radix_node *root)
+{
+ if (!root)
+ root = kho_radix_alloc_node();
+ if (!root)
+ return -ENOMEM;
+
+ tree->root = root;
+ mutex_init(&tree->lock);
+ return 0;
+}
+EXPORT_SYMBOL_GPL(kho_radix_init_tree);
+
/**
* kho_radix_destroy_tree - Destroy the radix tree
* @tree: The radix tree to destroy
@@ -1499,18 +1523,23 @@ static void __init kho_mem_retrieve(void)
* catches that and never sets kho_in.scratch_phys, which stops memory
* retrieval.
*/
- kho_in.radix_tree.root = kho_get_mem_map(fdt);
- mutex_init(&kho_in.radix_tree.lock);
+ err = kho_radix_init_tree(&kho_in.radix_tree, kho_get_mem_map(fdt));
+ if (err)
+ goto err;
err = kho_radix_walk_tree(&kho_in.radix_tree, &cb, NULL);
- if (err) {
- /*
- * Failed to initialize preserved memory. Clear FDT and radix
- * so KHO users don't treat it as a KHO boot.
- */
- kho_in.fdt_phys = 0;
- kho_in.radix_tree.root = NULL;
- }
+ if (err)
+ goto err;
+
+ return;
+
+err:
+ /*
+ * Failed to initialize preserved memory. Clear FDT and radix so KHO
+ * users don't treat it as a KHO boot.
+ */
+ kho_in.fdt_phys = 0;
+ kho_in.radix_tree.root = NULL;
}
static __init int kho_out_fdt_setup(void)
@@ -1636,16 +1665,14 @@ static __init int kho_init(void)
if (!kho_enable)
return 0;
- tree->root = kzalloc(PAGE_SIZE, GFP_KERNEL);
- if (!tree->root) {
- err = -ENOMEM;
+ err = kho_radix_init_tree(tree, NULL);
+ if (err)
goto err_free_scratch;
- }
kho_out.fdt = kho_alloc_preserve(PAGE_SIZE);
if (IS_ERR(kho_out.fdt)) {
err = PTR_ERR(kho_out.fdt);
- goto err_free_kho_radix_tree_root;
+ goto err_free_kho_radix_tree;
}
err = kho_debugfs_init();
@@ -1696,9 +1723,8 @@ static __init int kho_init(void)
err_free_fdt:
kho_unpreserve_free(kho_out.fdt);
-err_free_kho_radix_tree_root:
- kfree(tree->root);
- tree->root = NULL;
+err_free_kho_radix_tree:
+ kho_radix_destroy_tree(tree);
err_free_scratch:
kho_out.fdt = NULL;
for (int i = 0; i < kho_scratch_cnt; i++) {
--
2.55.0.229.g6434b31f56-goog
^ permalink raw reply related [flat|nested] 7+ messages in thread* [PATCH v4 14/21] kho: initialize kho_scratch pointer earlier in boot
2026-07-25 17:21 [PATCH v4 00/21] kho: make boot time huge page allocation work nicely with KHO Pratyush Yadav
` (2 preceding siblings ...)
2026-07-25 17:21 ` [PATCH v4 12/21] kho: add kho_radix_init_tree() Pratyush Yadav
@ 2026-07-25 17:21 ` Pratyush Yadav
2026-07-25 17:21 ` [PATCH v4 17/21] mm/mm_init: don't rely on memblock to get KHO scratch migratetype Pratyush Yadav
2026-07-25 17:21 ` [PATCH v4 19/21] memblock: make HugeTLB bootmem allocation work with KHO Pratyush Yadav
5 siblings, 0 replies; 7+ messages in thread
From: Pratyush Yadav @ 2026-07-25 17:21 UTC (permalink / raw)
To: Mike Rapoport, Pasha Tatashin, Pratyush Yadav, Alexander Graf,
Muchun Song, Oscar Salvador, David Hildenbrand, Andrew Morton,
Jason Miu, Jork Loeser
Cc: kexec, linux-mm, linux-kernel
From: "Pratyush Yadav (Google)" <pratyush@kernel.org>
In a future patch, mm init will use kho_scratch_overlap() for deciding
the migrate type of pageblocks it initializes. The earliest user
currently is free_area_init(). kho_scratch_overlap()
relies on kho_scratch pointer being initialized. Introduce
kho_memory_init_early() to do this.
kho_populate() would normally be a good place to do this, but
unfortunately, phys_to_virt() does not work at that point on ARM64. So
we need yet another initialization function.
Signed-off-by: Pratyush Yadav (Google) <pratyush@kernel.org>
---
include/linux/kexec_handover.h | 3 +++
kernel/liveupdate/kexec_handover.c | 15 +++++++++++++--
mm/mm_init.c | 1 +
3 files changed, 17 insertions(+), 2 deletions(-)
diff --git a/include/linux/kexec_handover.h b/include/linux/kexec_handover.h
index 40cb95a80981..18ba417f1ada 100644
--- a/include/linux/kexec_handover.h
+++ b/include/linux/kexec_handover.h
@@ -33,6 +33,7 @@ void kho_remove_subtree(void *blob);
int kho_retrieve_subtree(const char *name, phys_addr_t *phys, size_t *size);
void kho_memory_init(void);
+void kho_memory_init_early(void);
void kho_populate(phys_addr_t fdt_phys, u64 fdt_len, phys_addr_t scratch_phys,
u64 scratch_len);
@@ -110,6 +111,8 @@ static inline int kho_retrieve_subtree(const char *name, phys_addr_t *phys,
static inline void kho_memory_init(void) { }
+static inline void kho_memory_init_early(void) { }
+
static inline void kho_populate(phys_addr_t fdt_phys, u64 fdt_len,
phys_addr_t scratch_phys, u64 scratch_len)
{
diff --git a/kernel/liveupdate/kexec_handover.c b/kernel/liveupdate/kexec_handover.c
index b3b97a6ff9c8..2ddfc804cdf7 100644
--- a/kernel/liveupdate/kexec_handover.c
+++ b/kernel/liveupdate/kexec_handover.c
@@ -1534,8 +1534,6 @@ static void __init kho_mem_retrieve(void)
const void *fdt = kho_get_fdt();
int err;
- kho_scratch = phys_to_virt(kho_in.scratch_phys);
-
/*
* kho_get_mem_map() should always succeed. If it fails, kho_populate()
* catches that and never sets kho_in.scratch_phys, which stops memory
@@ -1756,6 +1754,19 @@ static __init int kho_init(void)
}
fs_initcall(kho_init);
+void __init kho_memory_init_early(void)
+{
+ if (!is_kho_boot())
+ return;
+
+ /*
+ * kho_scratch_overlap() needs kho_scratch to be initialized. It
+ * is used by free_area_init() on KHO boots, so initialize it
+ * early.
+ */
+ kho_scratch = phys_to_virt(kho_in.scratch_phys);
+}
+
void __init kho_memory_init(void)
{
if (kho_in.scratch_phys)
diff --git a/mm/mm_init.c b/mm/mm_init.c
index 0f64909e8d20..84d4b1c997bc 100644
--- a/mm/mm_init.c
+++ b/mm/mm_init.c
@@ -2687,6 +2687,7 @@ void __init __weak mem_init(void)
void __init mm_core_init_early(void)
{
+ kho_memory_init_early();
hugetlb_cma_reserve();
hugetlb_bootmem_alloc();
--
2.55.0.229.g6434b31f56-goog
^ permalink raw reply related [flat|nested] 7+ messages in thread* [PATCH v4 17/21] mm/mm_init: don't rely on memblock to get KHO scratch migratetype
2026-07-25 17:21 [PATCH v4 00/21] kho: make boot time huge page allocation work nicely with KHO Pratyush Yadav
` (3 preceding siblings ...)
2026-07-25 17:21 ` [PATCH v4 14/21] kho: initialize kho_scratch pointer earlier in boot Pratyush Yadav
@ 2026-07-25 17:21 ` Pratyush Yadav
2026-07-25 17:21 ` [PATCH v4 19/21] memblock: make HugeTLB bootmem allocation work with KHO Pratyush Yadav
5 siblings, 0 replies; 7+ messages in thread
From: Pratyush Yadav @ 2026-07-25 17:21 UTC (permalink / raw)
To: Mike Rapoport, Pasha Tatashin, Pratyush Yadav, Alexander Graf,
Muchun Song, Oscar Salvador, David Hildenbrand, Andrew Morton,
Jason Miu, Jork Loeser
Cc: kexec, linux-mm, linux-kernel
From: "Pratyush Yadav (Google)" <pratyush@kernel.org>
Currently struct page init via memmap_init() or deferred_init_memmap()
only queries the migrate type from KHO for each discrete memory range.
That works currently since KHO scratch memory has a different memory
type so it is always it its own region.
An upcoming patch will add support for discovering blocks of memory with
no preservations and it will mark it as MEMBLOCK_KHO_SCRATCH to allow
allocations from them. This can lead to the bootmem KHO scratch areas to
be merged into larger free ranges. This merging breaks the selection of
migrate type.
Get rid of memblock_is_kho_scratch_memory(). Instead, use
kho_scratch_overlap() to decide the migrate type of the PFN. Since
kho_scratch_migratetype() only uses KHO functions, move it to
kexec_handover.h.
Instead of calling kho_scratch_migratetype() once for each free range,
call it once for each pageblock. Update
pageblock_migratetype_init_range() and memmap_init_range() to do so.
Since the migrate type is now evaluated for each pageblock and not each
free range, drop the migratetype arguments to deferred_free_pages() and
memmap_init_zone_range() and use MIGRATE_MOVABLE directly.
Signed-off-by: Pratyush Yadav (Google) <pratyush@kernel.org>
---
include/linux/kexec_handover.h | 15 +++++++++++++++
include/linux/memblock.h | 19 -------------------
mm/memblock.c | 10 ----------
mm/mm_init.c | 27 +++++++++++++--------------
4 files changed, 28 insertions(+), 43 deletions(-)
diff --git a/include/linux/kexec_handover.h b/include/linux/kexec_handover.h
index 18ba417f1ada..c83ec4a15fe7 100644
--- a/include/linux/kexec_handover.h
+++ b/include/linux/kexec_handover.h
@@ -5,6 +5,7 @@
#include <linux/err.h>
#include <linux/errno.h>
#include <linux/types.h>
+#include <linux/mmzone.h>
#include <asm-generic/kexec_handover.h>
struct kho_vmalloc;
@@ -39,6 +40,14 @@ void kho_populate(phys_addr_t fdt_phys, u64 fdt_len, phys_addr_t scratch_phys,
u64 scratch_len);
bool kho_scratch_overlap(phys_addr_t phys, size_t size);
+
+static inline enum migratetype kho_scratch_migratetype(unsigned long pfn,
+ enum migratetype mt)
+{
+ if (kho_scratch_overlap(PFN_PHYS(pfn), pageblock_nr_pages << PAGE_SHIFT))
+ return MIGRATE_CMA;
+ return mt;
+}
#else
static inline bool kho_is_enabled(void)
{
@@ -122,6 +131,12 @@ static inline bool kho_scratch_overlap(phys_addr_t phys, size_t size)
{
return false;
}
+
+static inline enum migratetype kho_scratch_migratetype(unsigned long pfn,
+ enum migratetype mt)
+{
+ return mt;
+}
#endif /* CONFIG_KEXEC_HANDOVER */
#endif /* LINUX_KEXEC_HANDOVER_H */
diff --git a/include/linux/memblock.h b/include/linux/memblock.h
index 5afcd99aa8c1..f142c8a8cd5b 100644
--- a/include/linux/memblock.h
+++ b/include/linux/memblock.h
@@ -613,28 +613,9 @@ static inline void memtest_report_meminfo(struct seq_file *m) { }
#ifdef CONFIG_MEMBLOCK_KHO_SCRATCH
void memblock_set_kho_scratch_only(void);
void memblock_clear_kho_scratch_only(void);
-bool memblock_is_kho_scratch_memory(phys_addr_t addr);
-
-static inline enum migratetype kho_scratch_migratetype(unsigned long pfn,
- enum migratetype mt)
-{
- if (memblock_is_kho_scratch_memory(PFN_PHYS(pfn)))
- return MIGRATE_CMA;
- return mt;
-}
#else
static inline void memblock_set_kho_scratch_only(void) { }
static inline void memblock_clear_kho_scratch_only(void) { }
-static inline bool memblock_is_kho_scratch_memory(phys_addr_t addr)
-{
- return false;
-}
-
-static inline enum migratetype kho_scratch_migratetype(unsigned long pfn,
- enum migratetype mt)
-{
- return mt;
-}
#endif
#endif /* _LINUX_MEMBLOCK_H */
diff --git a/mm/memblock.c b/mm/memblock.c
index 6349c48154f4..8b2e551435ae 100644
--- a/mm/memblock.c
+++ b/mm/memblock.c
@@ -2511,16 +2511,6 @@ __init void memblock_clear_kho_scratch_only(void)
{
kho_scratch_only = false;
}
-
-bool __init_memblock memblock_is_kho_scratch_memory(phys_addr_t addr)
-{
- int i = memblock_search(&memblock.memory, addr);
-
- if (i == -1)
- return false;
-
- return memblock_is_kho_scratch(&memblock.memory.regions[i]);
-}
#endif
#ifdef CONFIG_KEXEC_HANDOVER
diff --git a/mm/mm_init.c b/mm/mm_init.c
index 30355effe1f2..dc20d6814e48 100644
--- a/mm/mm_init.c
+++ b/mm/mm_init.c
@@ -681,7 +681,9 @@ static __meminit void pageblock_migratetype_init_range(unsigned long pfn,
const unsigned long end = pfn + nr_pages;
for (pfn = pageblock_align(pfn); pfn < end; pfn += pageblock_nr_pages) {
- init_pageblock_migratetype(pfn_to_page(pfn), migratetype, false);
+ enum migratetype mt = kho_scratch_migratetype(pfn, migratetype);
+
+ init_pageblock_migratetype(pfn_to_page(pfn), mt, false);
if (!atomic && IS_ALIGNED(pfn, PAGES_PER_SECTION))
cond_resched();
}
@@ -932,8 +934,9 @@ void __meminit memmap_init_range(unsigned long size, int nid, unsigned long zone
* over the place during system boot.
*/
if (pageblock_aligned(pfn)) {
- init_pageblock_migratetype(page, migratetype,
- isolate_pageblock);
+ enum migratetype mt = kho_scratch_migratetype(pfn, migratetype);
+
+ init_pageblock_migratetype(page, mt, isolate_pageblock);
cond_resched();
}
pfn++;
@@ -943,8 +946,7 @@ void __meminit memmap_init_range(unsigned long size, int nid, unsigned long zone
static void __init memmap_init_zone_range(struct zone *zone,
unsigned long start_pfn,
unsigned long end_pfn,
- unsigned long *hole_pfn,
- enum migratetype mt)
+ unsigned long *hole_pfn)
{
unsigned long zone_start_pfn = zone->zone_start_pfn;
unsigned long zone_end_pfn = zone_start_pfn + zone->spanned_pages;
@@ -957,7 +959,8 @@ static void __init memmap_init_zone_range(struct zone *zone,
return;
memmap_init_range(end_pfn - start_pfn, nid, zone_id, start_pfn,
- zone_end_pfn, MEMINIT_EARLY, NULL, mt, false);
+ zone_end_pfn, MEMINIT_EARLY, NULL, MIGRATE_MOVABLE,
+ false);
if (*hole_pfn < start_pfn)
init_unavailable_range(*hole_pfn, start_pfn, zone_id, nid);
@@ -973,8 +976,6 @@ static void __init memmap_init(void)
for_each_mem_pfn_range(i, MAX_NUMNODES, &start_pfn, &end_pfn, &nid) {
struct pglist_data *node = NODE_DATA(nid);
- enum migratetype mt =
- kho_scratch_migratetype(start_pfn, MIGRATE_MOVABLE);
for (j = 0; j < MAX_NR_ZONES; j++) {
struct zone *zone = node->node_zones + j;
@@ -983,7 +984,7 @@ static void __init memmap_init(void)
continue;
memmap_init_zone_range(zone, start_pfn, end_pfn,
- &hole_pfn, mt);
+ &hole_pfn);
zone_id = j;
}
}
@@ -1973,7 +1974,7 @@ unsigned long __init node_map_pfn_alignment(void)
#ifdef CONFIG_DEFERRED_STRUCT_PAGE_INIT
static void __init deferred_free_pages(unsigned long pfn,
- unsigned long nr_pages, enum migratetype mt)
+ unsigned long nr_pages)
{
struct page *page;
unsigned long i;
@@ -1981,7 +1982,7 @@ static void __init deferred_free_pages(unsigned long pfn,
if (!nr_pages)
return;
- pageblock_migratetype_init_range(pfn, nr_pages, mt, true);
+ pageblock_migratetype_init_range(pfn, nr_pages, MIGRATE_MOVABLE, true);
page = pfn_to_page(pfn);
@@ -2051,8 +2052,6 @@ deferred_init_memmap_chunk(unsigned long start_pfn, unsigned long end_pfn,
for_each_free_mem_range(i, nid, 0, &start, &end, NULL) {
unsigned long spfn = PFN_UP(start);
unsigned long epfn = PFN_DOWN(end);
- enum migratetype mt =
- kho_scratch_migratetype(spfn, MIGRATE_MOVABLE);
if (spfn >= end_pfn)
break;
@@ -2065,7 +2064,7 @@ deferred_init_memmap_chunk(unsigned long start_pfn, unsigned long end_pfn,
unsigned long chunk_end = min(mo_pfn, epfn);
nr_pages += deferred_init_pages(zone, spfn, chunk_end);
- deferred_free_pages(spfn, chunk_end - spfn, mt);
+ deferred_free_pages(spfn, chunk_end - spfn);
spfn = chunk_end;
--
2.55.0.229.g6434b31f56-goog
^ permalink raw reply related [flat|nested] 7+ messages in thread* [PATCH v4 19/21] memblock: make HugeTLB bootmem allocation work with KHO
2026-07-25 17:21 [PATCH v4 00/21] kho: make boot time huge page allocation work nicely with KHO Pratyush Yadav
` (4 preceding siblings ...)
2026-07-25 17:21 ` [PATCH v4 17/21] mm/mm_init: don't rely on memblock to get KHO scratch migratetype Pratyush Yadav
@ 2026-07-25 17:21 ` Pratyush Yadav
5 siblings, 0 replies; 7+ messages in thread
From: Pratyush Yadav @ 2026-07-25 17:21 UTC (permalink / raw)
To: Mike Rapoport, Pasha Tatashin, Pratyush Yadav, Alexander Graf,
Muchun Song, Oscar Salvador, David Hildenbrand, Andrew Morton,
Jason Miu, Jork Loeser
Cc: kexec, linux-mm, linux-kernel
From: "Pratyush Yadav (Google)" <pratyush@kernel.org>
Gigantic huge page allocation is somewhat broken currently when KHO is
used.
Firstly, they break KHO scratch size accounting. RSRV_KERN is used to
track how much memory is reserved for use by the kernel. Since
hugetlb::alloc_bootmem() calls the memblock_alloc*() APIs, the hugepages
allocated also get marked as RSRV_KERN.
Allocations marked RSRV_KERN are used by KHO to calculate how much
scratch space it should reserve to make sure the next kernel has enough
memory to boot when it is in scratch-only phase. Counting hugepages in
that blows up scratch size, and can lead to the scratch allocation
failing, making KHO unusable. This will show up when huge pages make up
more than 50% of the system, which is a fairly common use case.
Secondly, while not supported right now, huge pages are user memory and
can be preserved via KHO. The scratch spaces should not have any
preserved memory. Allocating hugepages from scratch (on a KHO boot) can
lead to them being un-preservable.
Introduce memblock_alloc_hugetlb(). This lets memblock tailor to the
needs of hugetb without exposing those details to the general allocation
routines.
First, it does not use mirrored memory for hugetlb. Mirrored memory is a
limited resource that is best saved for kernel data structures, not user
memory.
Second, if the free memory area found by memblock_find_in_range_node()
is a part of a KHO scratch area, the free area is not used. Allocation
is retried starting after the free area to ensure no hugepages come from
KHO scratch.
Third, it simplifies the argument list by baking in some hugetlb
assumptions like alignment and exact_nid. This also simplifies
allocation logic in alloc_bootmem().
Also introduce MEMBLOCK_RSRV_HUGETLB to mark reservations made for
HugeTLB. This will be used by KHO in future patches to correctly
calculate scratch sizes.
Refactor some of the preparation logic like kmemleak tracking and
accepting memory into a separate helper memblock_prep_allocation(), and
use it from both memblock_alloc_hugetlb() and the usual
memblock_alloc_range_nid().
Signed-off-by: Pratyush Yadav (Google) <pratyush@kernel.org>
---
include/linux/memblock.h | 3 ++
mm/hugetlb.c | 22 +++-----
mm/memblock.c | 114 ++++++++++++++++++++++++++++++---------
3 files changed, 100 insertions(+), 39 deletions(-)
diff --git a/include/linux/memblock.h b/include/linux/memblock.h
index f142c8a8cd5b..678fe466529a 100644
--- a/include/linux/memblock.h
+++ b/include/linux/memblock.h
@@ -51,6 +51,7 @@ extern unsigned long long max_possible_pfn;
* memory reservations yet, so we get scratch memory from the previous
* kernel that we know is good to use. It is the only memory that
* allocations may happen from in this phase.
+ * @MEMBLOCK_RSRV_HUGETLB: memory is reserved for hugetlb pages
*/
enum memblock_flags {
MEMBLOCK_NONE = 0x0, /* No special request */
@@ -61,6 +62,7 @@ enum memblock_flags {
MEMBLOCK_RSRV_NOINIT = 0x10, /* don't initialize struct pages */
MEMBLOCK_RSRV_KERN = 0x20, /* memory reserved for kernel use */
MEMBLOCK_KHO_SCRATCH = 0x40, /* scratch memory for kexec handover */
+ MEMBLOCK_RSRV_HUGETLB = 0x80, /* memory reserved for hugetlb pages */
};
/**
@@ -420,6 +422,7 @@ void *memblock_alloc_try_nid_raw(phys_addr_t size, phys_addr_t align,
void *memblock_alloc_try_nid(phys_addr_t size, phys_addr_t align,
phys_addr_t min_addr, phys_addr_t max_addr,
int nid);
+void *memblock_alloc_hugetlb(phys_addr_t size, int nid, bool exact_nid);
static __always_inline void *memblock_alloc(phys_addr_t size, phys_addr_t align)
{
diff --git a/mm/hugetlb.c b/mm/hugetlb.c
index 571212b80835..ab4afc818e8c 100644
--- a/mm/hugetlb.c
+++ b/mm/hugetlb.c
@@ -3033,29 +3033,21 @@ static __init void *alloc_bootmem(struct hstate *h, int nid, bool node_exact)
if (hugetlb_early_cma(h))
m = hugetlb_cma_alloc_bootmem(h, &listnode, node_exact);
else {
- if (node_exact)
- m = memblock_alloc_exact_nid_raw(huge_page_size(h),
- huge_page_size(h), 0,
- MEMBLOCK_ALLOC_ACCESSIBLE, nid);
- else {
- m = memblock_alloc_try_nid_raw(huge_page_size(h),
- huge_page_size(h), 0,
- MEMBLOCK_ALLOC_ACCESSIBLE, nid);
+ m = memblock_alloc_hugetlb(huge_page_size(h), nid, node_exact);
+ if (m) {
+ m->flags = 0;
+ m->cma = NULL;
+
/*
* For pre-HVO to work correctly, pages need to be on
* the list for the node they were actually allocated
* from. That node may be different in the case of
- * fallback by memblock_alloc_try_nid_raw. So,
+ * fallback by memblock_alloc_hugetlb_bootmem. So,
* extract the actual node first.
*/
- if (m)
+ if (!node_exact)
listnode = early_pfn_to_nid(PHYS_PFN(__pa(m)));
}
-
- if (m) {
- m->flags = 0;
- m->cma = NULL;
- }
}
if (m) {
diff --git a/mm/memblock.c b/mm/memblock.c
index 8b2e551435ae..ff470fd0fd10 100644
--- a/mm/memblock.c
+++ b/mm/memblock.c
@@ -19,11 +19,9 @@
#include <linux/mutex.h>
#include <linux/string_helpers.h>
-#ifdef CONFIG_KEXEC_HANDOVER
#include <linux/libfdt.h>
#include <linux/kexec_handover.h>
#include <linux/kho/abi/memblock.h>
-#endif /* CONFIG_KEXEC_HANDOVER */
#include <asm/sections.h>
#include <linux/io.h>
@@ -1506,6 +1504,32 @@ int __init_memblock memblock_set_node(phys_addr_t base, phys_addr_t size,
return 0;
}
+static void memblock_prep_allocation(phys_addr_t start, phys_addr_t size,
+ bool kmemleak_trace)
+{
+ /*
+ * Skip kmemleak for those places like kasan_init() and
+ * early_pgtable_alloc() due to high volume.
+ */
+ if (kmemleak_trace)
+ /*
+ * Memblock allocated blocks are never reported as
+ * leaks. This is because many of these blocks are
+ * only referred via the physical address which is
+ * not looked up by kmemleak.
+ */
+ kmemleak_alloc_phys(start, size, 0);
+
+ /*
+ * Some Virtual Machine platforms, such as Intel TDX or AMD SEV-SNP,
+ * require memory to be accepted before it can be used by the
+ * guest.
+ *
+ * Accept the memory of the allocated buffer.
+ */
+ accept_memory(start, size);
+}
+
/**
* memblock_alloc_range_nid - allocate boot memory block
* @size: size of memory block to be allocated in bytes
@@ -1580,28 +1604,7 @@ phys_addr_t __init memblock_alloc_range_nid(phys_addr_t size,
return 0;
done:
- /*
- * Skip kmemleak for those places like kasan_init() and
- * early_pgtable_alloc() due to high volume.
- */
- if (end != MEMBLOCK_ALLOC_NOLEAKTRACE)
- /*
- * Memblock allocated blocks are never reported as
- * leaks. This is because many of these blocks are
- * only referred via the physical address which is
- * not looked up by kmemleak.
- */
- kmemleak_alloc_phys(found, size, 0);
-
- /*
- * Some Virtual Machine platforms, such as Intel TDX or AMD SEV-SNP,
- * require memory to be accepted before it can be used by the
- * guest.
- *
- * Accept the memory of the allocated buffer.
- */
- accept_memory(found, size);
-
+ memblock_prep_allocation(found, size, end != MEMBLOCK_ALLOC_NOLEAKTRACE);
return found;
}
@@ -1756,6 +1759,69 @@ void * __init memblock_alloc_try_nid_raw(
false);
}
+/**
+ * memblock_alloc_hugetlb - allocate boot memory for HugeTLB pages
+ * @size: size of the memory to be allocated in bytes
+ * @nid: nid of the free memory to find, %NUMA_NO_NODE for any node
+ * @exact_nid: only allocate from the specified nid. If %false, the specified
+ * nid is tried first, and then all nodes are tried as fallback.
+ *
+ * HugeTLB pages are always aligned by their size, so the alignment matches
+ * @size. Since the memory is for userspace, mirrored memory is not used. The
+ * memory is not zeroed. Does not panic if request cannot be satisfied.
+ *
+ * Return:
+ * Virtual address of allocated memory block on success, %NULL on failure.
+ */
+void * __init memblock_alloc_hugetlb(phys_addr_t size, int nid, bool exact_nid)
+{
+ enum memblock_flags flags = choose_memblock_flags();
+ phys_addr_t addr, start = 0, end = MEMBLOCK_ALLOC_ACCESSIBLE;
+
+ memblock_dbg("%s: %llu bytes, nid=%d, exact_nid=%d %pS\n", __func__,
+ (u64)size, nid, exact_nid, (void *)_RET_IP_);
+
+ /* Don't waste mirrored memory on HugeTLB pages. */
+ flags &= ~MEMBLOCK_MIRROR;
+retry:
+ /* HugeTLB pages are always aligned by their size. */
+ addr = memblock_find_in_range_node(size, size, start, end, nid, flags);
+ if (addr)
+ goto found;
+
+ /* Try all nodes if allowed. */
+ if (numa_valid_node(nid) && !exact_nid) {
+ nid = NUMA_NO_NODE;
+ goto retry;
+ }
+
+ /* Found nothing... :-( */
+ return NULL;
+
+found:
+ /*
+ * HugeTLB pages can be preserved with KHO and no preserved memory can
+ * be in scratch. So retry if found address overlaps with scratch.
+ *
+ * Scratch areas are normally not very large, so this shouldn't take too
+ * many retries.
+ */
+ if (kho_scratch_overlap(addr, size)) {
+ if (memblock_bottom_up())
+ start = addr + size;
+ else
+ start = addr - size;
+
+ goto retry;
+ }
+
+ if (__memblock_reserve(addr, size, nid, MEMBLOCK_RSRV_KERN | MEMBLOCK_RSRV_HUGETLB))
+ return NULL;
+
+ memblock_prep_allocation(addr, size, true);
+ return phys_to_virt(addr);
+}
+
/**
* memblock_alloc_try_nid - allocate boot memory block
* @size: size of memory block to be allocated in bytes
--
2.55.0.229.g6434b31f56-goog
^ permalink raw reply related [flat|nested] 7+ messages in thread