[PATCH 0/5] mm: hugetlb: add CMA support for PMD-sized page

[PATCH 0/5] mm: hugetlb: add CMA support for PMD-sized page

[Qi Zheng]

From: Qi Zheng <zhengqi.arch@bytedance.com>

Hi all,

Allocating PMD-sized HugeTLB pages at runtime can become slow or fail
as physical memory becomes fragmented. Reserving HugeTLB pages at boot
avoids this problem, but keeps unused pages unavailable to ordinary
allocations.

Live upgrades are one example: they need additional HugeTLB pages only
during the upgrade. HugeTLB already supports CMA for gigantic pages,
but reserving a CMA pool for 1 GiB pages can fail when physical KASLR
and other early reservations leave too few suitable 1 GiB-aligned
ranges. At runtime, 1 GiB allocations can also fail when pages cannot
be migrated out of a suitable contiguous range.

Extend CMA support to PMD-sized pages so their backing memory can be
reserved at boot while remaining usable for movable allocations until
needed. This relaxes the reservation alignment requirement and reduces
the contiguous range required per allocation.

The series adds an optional @<page-size> suffix to hugetlb_cma and a
separate PMD-sized pool. For example, on x86-64:

	hugetlb_cma=4G hugetlb_cma=1G@2M

This reserves 4G for the existing gigantic-page pool and 1G for the
PMD-sized pool. Existing unsuffixed parameters and gigantic-page pool
sharing are preserved. The new size follows the architecture's PMD
size rather than being fixed at 2M.

The preparatory patches introduce per-pool state and prevent recursive
CMA allocation during migration before enabling PMD-sized allocation.

Thanks,
Qi

Qi Zheng (5):
  mm: hugetlb: add explicit page size to hugetlb_cma
  mm: hugetlb: group CMA pool and reservation state
  mm: hugetlb: track CMA reservations by page order
  mm: hugetlb: avoid recursive CMA allocation during migration
  mm: hugetlb: enable CMA allocation for PMD-sized pages

 .../admin-guide/kernel-parameters.txt         |  39 ++-
 include/linux/hugetlb.h                       |   4 +-
 mm/hugetlb.c                                  |  80 +++--
 mm/hugetlb_cma.c                              | 328 ++++++++++++------
 mm/hugetlb_cma.h                              |  11 +-
 mm/mempolicy.c                                |   2 +-
 mm/migrate.c                                  |   3 +-
 7 files changed, 306 insertions(+), 161 deletions(-)


base-commit: b810335a0699d1810683796b0e3ffe8827ddfe6c
-- 
2.54.0

[PATCH 1/5] mm: hugetlb: add explicit page size to hugetlb_cma

[Qi Zheng]

From: Qi Zheng <zhengqi.arch@bytedance.com>

Accept an optional @<page-size> suffix for hugetlb_cma. Initially accept
only the architecture's existing CMA page size. Preserve the behavior of
unsuffixed arguments and the existing hugepagesz/hugepages options.

Validate the suffix before applying the existing capacity, percentage and
node-list parser.

Save the arguments during early parsing and process them at reservation
time, after architecture-specific page-size information has been
initialized.

Assisted-by: LLM
Signed-off-by: Qi Zheng <zhengqi.arch@bytedance.com>
---
 mm/hugetlb_cma.c | 43 ++++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 42 insertions(+), 1 deletion(-)

diff --git a/mm/hugetlb_cma.c b/mm/hugetlb_cma.c
index 95fd2d190f0d2..a17d32fa5ebb3 100644
--- a/mm/hugetlb_cma.c
+++ b/mm/hugetlb_cma.c
@@ -25,6 +25,9 @@ static unsigned long hugetlb_cma_size __ro_after_init;
 static unsigned int hugetlb_cma_percent __initdata;
 static unsigned int hugetlb_cma_percent_in_node[MAX_NUMNODES] __initdata;
 
+static char hugetlb_cma_cmdline[COMMAND_LINE_SIZE] __initdata;
+static unsigned int hugetlb_cma_cmdline_len __initdata;
+
 #ifdef CONFIG_NUMA
 static phys_addr_t __init memblock_node_memory_size(int nid)
 {
@@ -119,11 +122,29 @@ void * __init hugetlb_cma_alloc_bootmem(struct hstate *h, int nid, bool node_exa
 	return NULL;
 }
 
-static int __init cmdline_parse_hugetlb_cma(char *p)
+static int __init hugetlb_cma_parse(char *p)
 {
 	int nid, count = 0;
 	unsigned long tmp;
 	char *s = p;
+	char *suffix = strchr(p, '@');
+
+	if (suffix) {
+		unsigned int order = arch_hugetlb_cma_order();
+		unsigned long long page_size;
+		char *end;
+
+		page_size = memparse(suffix + 1, &end);
+		if (suffix == p || !page_size || *end) {
+			pr_warn("hugetlb_cma: invalid parameter %s\n", p);
+			return -EINVAL;
+		}
+		if (!order || page_size != (PAGE_SIZE << order) ||
+		    !arch_hugetlb_valid_size(page_size)) {
+			pr_warn("hugetlb_cma: unsupported page size in %s\n", p);
+			return -EINVAL;
+		}
+	}
 
 	while (*s) {
 		if (sscanf(s, "%lu%n", &tmp, &count) != 1)
@@ -194,6 +215,20 @@ static int __init cmdline_parse_hugetlb_cma(char *p)
 	return 0;
 }
 
+static int __init cmdline_parse_hugetlb_cma(char *p)
+{
+	size_t len;
+
+	if (!p)
+		return -EINVAL;
+	len = strlen(p) + 1;
+	if (len > sizeof(hugetlb_cma_cmdline) - hugetlb_cma_cmdline_len)
+		return -EINVAL;
+	memcpy(hugetlb_cma_cmdline + hugetlb_cma_cmdline_len, p, len);
+	hugetlb_cma_cmdline_len += len;
+
+	return 0;
+}
 early_param("hugetlb_cma", cmdline_parse_hugetlb_cma);
 
 static int __init cmdline_parse_hugetlb_cma_only(char *p)
@@ -214,6 +249,12 @@ void __init hugetlb_cma_reserve(void)
 	bool node_specific_cma_alloc = false;
 	bool has_node_specific_param = false;
 	int nid;
+	char *p;
+
+	for (p = hugetlb_cma_cmdline;
+	     p < hugetlb_cma_cmdline + hugetlb_cma_cmdline_len;
+	     p += strlen(p) + 1)
+		hugetlb_cma_parse(p);
 
 	for (nid = 0; nid < MAX_NUMNODES; nid++) {
 		if (hugetlb_cma_size_in_node[nid] || hugetlb_cma_percent_in_node[nid]) {
-- 
2.54.0

[PATCH 2/5] mm: hugetlb: group CMA pool and reservation state

[Qi Zheng]

From: Qi Zheng <zhengqi.arch@bytedance.com>

The HugeTLB CMA areas, total size, per-node sizes and percentages are
stored in separate singleton variables. Group the runtime areas and
size into a pool, and the temporary reservation inputs into a separate
configuration structure.

Keep one instance of each structure and preserve the existing parsing,
reservation and allocation behavior. This separates the mechanical state
conversion from adding support for multiple page orders.

Assisted-by: LLM
Signed-off-by: Qi Zheng <zhengqi.arch@bytedance.com>
---
 mm/hugetlb_cma.c | 163 ++++++++++++++++++++++++++---------------------
 1 file changed, 89 insertions(+), 74 deletions(-)

diff --git a/mm/hugetlb_cma.c b/mm/hugetlb_cma.c
index a17d32fa5ebb3..4f27eb242dc3b 100644
--- a/mm/hugetlb_cma.c
+++ b/mm/hugetlb_cma.c
@@ -16,14 +16,21 @@
 #include "internal.h"
 #include "hugetlb_cma.h"
 
+struct hugetlb_cma_pool {
+	struct cma *cma[MAX_NUMNODES];
+	unsigned long size;
+};
 
-static struct cma *hugetlb_cma[MAX_NUMNODES] __ro_after_init;
-static unsigned long hugetlb_cma_size_in_node[MAX_NUMNODES] __initdata;
-static bool hugetlb_cma_only __ro_after_init;
-static unsigned long hugetlb_cma_size __ro_after_init;
+struct hugetlb_cma_config {
+	unsigned long size_in_node[MAX_NUMNODES];
+	unsigned int percent;
+	unsigned int percent_in_node[MAX_NUMNODES];
+};
+
+static struct hugetlb_cma_pool hugetlb_cma __ro_after_init;
+static struct hugetlb_cma_config hugetlb_cma_config __initdata;
 
-static unsigned int hugetlb_cma_percent __initdata;
-static unsigned int hugetlb_cma_percent_in_node[MAX_NUMNODES] __initdata;
+static bool hugetlb_cma_only __ro_after_init;
 
 static char hugetlb_cma_cmdline[COMMAND_LINE_SIZE] __initdata;
 static unsigned int hugetlb_cma_cmdline_len __initdata;
@@ -49,7 +56,9 @@ static phys_addr_t __init memblock_node_memory_size(int nid)
 
 void hugetlb_cma_free_frozen_folio(struct folio *folio)
 {
-	WARN_ON_ONCE(!cma_release_frozen(hugetlb_cma[folio_nid(folio)],
+	struct hugetlb_cma_pool *pool = &hugetlb_cma;
+
+	WARN_ON_ONCE(!cma_release_frozen(pool->cma[folio_nid(folio)],
 					 &folio->page, folio_nr_pages(folio)));
 }
 
@@ -61,8 +70,9 @@ struct folio *hugetlb_cma_alloc_frozen_folio(int order, gfp_t gfp_mask,
 	struct page *page = NULL;
 	const nodemask_t *nmask;
 	unsigned int cpuset_mems_cookie;
+	struct hugetlb_cma_pool *pool = &hugetlb_cma;
 
-	if (!hugetlb_cma_size)
+	if (!pool->size)
 		return NULL;
 
 retry_cpuset:
@@ -73,15 +83,15 @@ struct folio *hugetlb_cma_alloc_frozen_folio(int order, gfp_t gfp_mask,
 		nmask = nodemask;
 	}
 
-	if (hugetlb_cma[nid] && node_isset(nid, *nmask))
-		page = cma_alloc_frozen_compound(hugetlb_cma[nid], order);
+	if (pool->cma[nid] && node_isset(nid, *nmask))
+		page = cma_alloc_frozen_compound(pool->cma[nid], order);
 
 	if (!page && !(gfp_mask & __GFP_THISNODE)) {
 		for_each_node_mask(node, *nmask) {
-			if (node == nid || !hugetlb_cma[node])
+			if (node == nid || !pool->cma[node])
 				continue;
 
-			page = cma_alloc_frozen_compound(hugetlb_cma[node], order);
+			page = cma_alloc_frozen_compound(pool->cma[node], order);
 			if (page)
 				break;
 		}
@@ -101,17 +111,18 @@ struct folio *hugetlb_cma_alloc_frozen_folio(int order, gfp_t gfp_mask,
 
 void * __init hugetlb_cma_alloc_bootmem(struct hstate *h, int nid, bool node_exact)
 {
+	struct hugetlb_cma_pool *pool = &hugetlb_cma;
 	struct cma *cma;
 	void *m;
 	int node;
 
-	cma = hugetlb_cma[nid];
+	cma = pool->cma[nid];
 	m = cma_reserve_early(cma, huge_page_size(h));
 	if (m || node_exact)
 		return m;
 
 	for_each_node_mask(node, hugetlb_bootmem_nodes) {
-		cma = hugetlb_cma[node];
+		cma = pool->cma[node];
 		if (!cma || node == nid)
 			continue;
 		m = cma_reserve_early(cma, huge_page_size(h));
@@ -124,6 +135,8 @@ void * __init hugetlb_cma_alloc_bootmem(struct hstate *h, int nid, bool node_exa
 
 static int __init hugetlb_cma_parse(char *p)
 {
+	struct hugetlb_cma_pool *pool = &hugetlb_cma;
+	struct hugetlb_cma_config *config = &hugetlb_cma_config;
 	int nid, count = 0;
 	unsigned long tmp;
 	char *s = p;
@@ -157,8 +170,8 @@ static int __init hugetlb_cma_parse(char *p)
 				break;
 			nid = array_index_nospec(tmp, MAX_NUMNODES);
 
-			hugetlb_cma_size = 0;
-			hugetlb_cma_percent = 0;
+			pool->size = 0;
+			config->percent = 0;
 
 			s += count + 1;
 			tmp = memparse(s, &next);
@@ -168,12 +181,12 @@ static int __init hugetlb_cma_parse(char *p)
 						tmp, nid);
 					break;
 				}
-				hugetlb_cma_percent_in_node[nid] = tmp;
-				hugetlb_cma_size_in_node[nid] = 0;
+				config->percent_in_node[nid] = tmp;
+				config->size_in_node[nid] = 0;
 				s = next + 1;
 			} else {
-				hugetlb_cma_size_in_node[nid] = tmp;
-				hugetlb_cma_percent_in_node[nid] = 0;
+				config->size_in_node[nid] = tmp;
+				config->percent_in_node[nid] = 0;
 				s = next;
 			}
 
@@ -193,19 +206,19 @@ static int __init hugetlb_cma_parse(char *p)
 				if (tmp > 100) {
 					pr_warn("hugetlb_cma: invalid percentage %lu\n", tmp);
 				} else {
-					hugetlb_cma_percent = tmp;
-					hugetlb_cma_size = 0;
+					config->percent = tmp;
+					pool->size = 0;
 					for (nid = 0; nid < MAX_NUMNODES; nid++) {
-						hugetlb_cma_size_in_node[nid] = 0;
-						hugetlb_cma_percent_in_node[nid] = 0;
+						config->size_in_node[nid] = 0;
+						config->percent_in_node[nid] = 0;
 					}
 				}
 			} else {
-				hugetlb_cma_size = tmp;
-				hugetlb_cma_percent = 0;
+				pool->size = tmp;
+				config->percent = 0;
 				for (nid = 0; nid < MAX_NUMNODES; nid++) {
-					hugetlb_cma_size_in_node[nid] = 0;
-					hugetlb_cma_percent_in_node[nid] = 0;
+					config->size_in_node[nid] = 0;
+					config->percent_in_node[nid] = 0;
 				}
 			}
 			break;
@@ -245,6 +258,8 @@ unsigned int __weak arch_hugetlb_cma_order(void)
 
 void __init hugetlb_cma_reserve(void)
 {
+	struct hugetlb_cma_pool *pool = &hugetlb_cma;
+	struct hugetlb_cma_config *config = &hugetlb_cma_config;
 	unsigned long size, reserved, per_node, order, gigantic_page_size;
 	bool node_specific_cma_alloc = false;
 	bool has_node_specific_param = false;
@@ -257,33 +272,33 @@ void __init hugetlb_cma_reserve(void)
 		hugetlb_cma_parse(p);
 
 	for (nid = 0; nid < MAX_NUMNODES; nid++) {
-		if (hugetlb_cma_size_in_node[nid] || hugetlb_cma_percent_in_node[nid]) {
+		if (config->size_in_node[nid] || config->percent_in_node[nid]) {
 			has_node_specific_param = true;
 			break;
 		}
 	}
 
 	if (has_node_specific_param) {
-		hugetlb_cma_size = 0;
+		pool->size = 0;
 		for (nid = 0; nid < MAX_NUMNODES; nid++) {
-			if (hugetlb_cma_percent_in_node[nid]) {
+			if (config->percent_in_node[nid]) {
 				phys_addr_t node_gfp_mem = memblock_node_memory_size(nid);
 				u64 s;
 
 				s = mul_u64_u32_div((u64)node_gfp_mem,
-						    hugetlb_cma_percent_in_node[nid],
+						    config->percent_in_node[nid],
 						    100);
 
-				hugetlb_cma_size_in_node[nid] = s;
+				config->size_in_node[nid] = s;
 			}
-			hugetlb_cma_size += hugetlb_cma_size_in_node[nid];
+			pool->size += config->size_in_node[nid];
 		}
-	} else if (hugetlb_cma_percent) {
-		hugetlb_cma_size = mul_u64_u32_div((u64)memblock_phys_mem_size(),
-						   hugetlb_cma_percent, 100);
+	} else if (config->percent) {
+		pool->size = mul_u64_u32_div((u64)memblock_phys_mem_size(),
+						   config->percent, 100);
 	}
 
-	if (!hugetlb_cma_size)
+	if (!pool->size)
 		return;
 
 	order = arch_hugetlb_cma_order();
@@ -301,36 +316,36 @@ void __init hugetlb_cma_reserve(void)
 	VM_WARN_ON(order <= MAX_PAGE_ORDER);
 	gigantic_page_size = PAGE_SIZE << order;
 
-	if (hugetlb_cma_percent) {
-		unsigned long orig_size = hugetlb_cma_size;
+	if (config->percent) {
+		unsigned long orig_size = pool->size;
 
-		hugetlb_cma_size = ALIGN_DOWN(hugetlb_cma_size, PAGE_SIZE << order);
-		if (orig_size && !hugetlb_cma_size)
+		pool->size = ALIGN_DOWN(pool->size, PAGE_SIZE << order);
+		if (orig_size && !pool->size)
 			pr_warn("hugetlb_cma: reservation size rounded down to 0 from %lu MiB (%u%%)\n",
-				orig_size / SZ_1M, hugetlb_cma_percent);
+				orig_size / SZ_1M, config->percent);
 	} else if (has_node_specific_param) {
-		hugetlb_cma_size = 0;
+		pool->size = 0;
 		for (nid = 0; nid < MAX_NUMNODES; nid++) {
-			if (hugetlb_cma_percent_in_node[nid]) {
-				unsigned long orig_size = hugetlb_cma_size_in_node[nid];
+			if (config->percent_in_node[nid]) {
+				unsigned long orig_size = config->size_in_node[nid];
 
-				hugetlb_cma_size_in_node[nid] =
-					ALIGN_DOWN(hugetlb_cma_size_in_node[nid],
+				config->size_in_node[nid] =
+					ALIGN_DOWN(config->size_in_node[nid],
 						   PAGE_SIZE << order);
-				if (orig_size && !hugetlb_cma_size_in_node[nid])
+				if (orig_size && !config->size_in_node[nid])
 					pr_warn("hugetlb_cma: reservation size rounded down to 0 from %lu MiB (%u%%) on node %d\n",
 						orig_size / SZ_1M,
-						hugetlb_cma_percent_in_node[nid],
+						config->percent_in_node[nid],
 						nid);
 			}
-			hugetlb_cma_size += hugetlb_cma_size_in_node[nid];
+			pool->size += config->size_in_node[nid];
 		}
 	}
 
 	hugetlb_bootmem_set_nodes();
 
 	for (nid = 0; nid < MAX_NUMNODES; nid++) {
-		size = hugetlb_cma_size_in_node[nid];
+		size = config->size_in_node[nid];
 		if (size == 0)
 			continue;
 
@@ -344,18 +359,18 @@ void __init hugetlb_cma_reserve(void)
 			continue;
 		}
 
-		hugetlb_cma_size -= size;
-		hugetlb_cma_size_in_node[nid] = 0;
+		pool->size -= size;
+		config->size_in_node[nid] = 0;
 	}
 
 	/* Validate the CMA size again in case some invalid nodes specified. */
-	if (!hugetlb_cma_size)
+	if (!pool->size)
 		return;
 
-	if (!IS_ALIGNED(hugetlb_cma_size, gigantic_page_size)) {
+	if (!IS_ALIGNED(pool->size, gigantic_page_size)) {
 		pr_warn("hugetlb_cma: cma area must be a multiple of %lu MiB\n",
 			gigantic_page_size / SZ_1M);
-		hugetlb_cma_size = 0;
+		pool->size = 0;
 		return;
 	}
 
@@ -364,16 +379,16 @@ void __init hugetlb_cma_reserve(void)
 		 * If 3 GB area is requested on a machine with 4 numa nodes,
 		 * let's allocate 1 GB on first three nodes and ignore the last one.
 		 */
-		per_node = DIV_ROUND_UP(hugetlb_cma_size,
+		per_node = DIV_ROUND_UP(pool->size,
 					nodes_weight(hugetlb_bootmem_nodes));
 		per_node = round_up(per_node, gigantic_page_size);
-		if (hugetlb_cma_percent)
+		if (config->percent)
 			pr_info("hugetlb_cma: reserve %lu MiB (%u%%), up to %lu MiB per node\n",
-				hugetlb_cma_size / SZ_1M, hugetlb_cma_percent,
+				pool->size / SZ_1M, config->percent,
 				per_node / SZ_1M);
 		else
 			pr_info("hugetlb_cma: reserve %lu MiB, up to %lu MiB per node\n",
-				hugetlb_cma_size / SZ_1M, per_node / SZ_1M);
+				pool->size / SZ_1M, per_node / SZ_1M);
 	}
 
 	reserved = 0;
@@ -382,12 +397,12 @@ void __init hugetlb_cma_reserve(void)
 		char name[CMA_MAX_NAME];
 
 		if (node_specific_cma_alloc) {
-			if (hugetlb_cma_size_in_node[nid] == 0)
+			if (config->size_in_node[nid] == 0)
 				continue;
 
-			size = hugetlb_cma_size_in_node[nid];
+			size = config->size_in_node[nid];
 		} else {
-			size = min(per_node, hugetlb_cma_size - reserved);
+			size = min(per_node, pool->size - reserved);
 		}
 
 		snprintf(name, sizeof(name), "hugetlb%d", nid);
@@ -398,33 +413,33 @@ void __init hugetlb_cma_reserve(void)
 		 */
 		res = cma_declare_contiguous_multi(size, gigantic_page_size,
 					HUGETLB_PAGE_ORDER, name,
-					&hugetlb_cma[nid], nid);
-		if (res || !cma_validate_zones(hugetlb_cma[nid])) {
+					&pool->cma[nid], nid);
+		if (res || !cma_validate_zones(pool->cma[nid])) {
 			pr_warn("hugetlb_cma: %s: err %d, node %d\n",
 				res ? "reservation failed" : "reserved area spans zones",
 				res, nid);
-			hugetlb_cma[nid] = NULL;
+			pool->cma[nid] = NULL;
 			continue;
 		}
 
 		reserved += size;
-		if (hugetlb_cma_percent_in_node[nid])
+		if (config->percent_in_node[nid])
 			pr_info("hugetlb_cma: reserved %lu MiB (%u%%) on node %d\n",
-				size / SZ_1M, hugetlb_cma_percent_in_node[nid], nid);
+				size / SZ_1M, config->percent_in_node[nid], nid);
 		else
 			pr_info("hugetlb_cma: reserved %lu MiB on node %d\n",
 				size / SZ_1M, nid);
 
-		if (reserved >= hugetlb_cma_size)
+		if (reserved >= pool->size)
 			break;
 	}
 
 	if (!reserved)
 		/*
-		 * hugetlb_cma_size is used to determine if allocations from
+		 * pool->size is used to determine if allocations from
 		 * cma are possible.  Set to zero if no cma regions are set up.
 		 */
-		hugetlb_cma_size = 0;
+		pool->size = 0;
 }
 
 bool hugetlb_cma_exclusive_alloc(void)
@@ -434,12 +449,12 @@ bool hugetlb_cma_exclusive_alloc(void)
 
 unsigned long __init hugetlb_cma_total_size(void)
 {
-	return hugetlb_cma_size;
+	return hugetlb_cma.size;
 }
 
 void __init hugetlb_cma_validate_params(void)
 {
-	if (!hugetlb_cma_size)
+	if (!hugetlb_cma.size)
 		hugetlb_cma_only = false;
 }
 
-- 
2.54.0

[PATCH 3/5] mm: hugetlb: track CMA reservations by page order

[Qi Zheng]

From: Qi Zheng <zhengqi.arch@bytedance.com>

Prepare for a separate CMA pool for PMD-sized HugeTLB pages by organizing
CMA areas and reservation parameters into pools identified by page order.

Select the corresponding pool for allocation, CMA-only checks and
boot-time allocation decisions. Release folios through the CMA area that
owns their physical memory, including after demotion.

Record each pool's successfully reserved capacity. Remove
hugetlb_cma_validate_params(), since the per-pool CMA-only check already
handles missing or empty pools.

Continue accepting only the architecture's existing CMA page size and
preserve the default pool's use by all gigantic sizes. PMD-sized pool
support will be added in a subsequent patch.

Assisted-by: LLM
Signed-off-by: Qi Zheng <zhengqi.arch@bytedance.com>
---
 mm/hugetlb.c     |   6 +-
 mm/hugetlb_cma.c | 154 ++++++++++++++++++++++++++++++++---------------
 mm/hugetlb_cma.h |  11 ++--
 3 files changed, 112 insertions(+), 59 deletions(-)

diff --git a/mm/hugetlb.c b/mm/hugetlb.c
index 76f4d2a5e0206..9d569eae852c5 100644
--- a/mm/hugetlb.c
+++ b/mm/hugetlb.c
@@ -1368,7 +1368,7 @@ static struct folio *alloc_gigantic_frozen_folio(int order, gfp_t gfp_mask,
 	if (folio)
 		return folio;
 
-	if (hugetlb_cma_exclusive_alloc())
+	if (hugetlb_cma_exclusive_alloc(order))
 		return NULL;
 
 	folio = (struct folio *)alloc_contig_frozen_pages(1 << order, gfp_mask,
@@ -3635,7 +3635,7 @@ static void __init hugetlb_hstate_alloc_pages(struct hstate *h)
 	 * Skip gigantic hugepages allocation if early CMA
 	 * reservations are not available.
 	 */
-	if (hstate_is_gigantic(h) && hugetlb_cma_total_size() &&
+	if (hstate_is_gigantic(h) && hugetlb_cma_size(h) &&
 	    !hugetlb_early_cma(h)) {
 		pr_warn_once("hugetlb_cma is enabled, skip boot time allocation\n");
 		return;
@@ -4338,8 +4338,6 @@ static __init void hugetlb_parse_params(void)
 
 		hcp->setup(hcp->val);
 	}
-
-	hugetlb_cma_validate_params();
 }
 
 /*
diff --git a/mm/hugetlb_cma.c b/mm/hugetlb_cma.c
index 4f27eb242dc3b..9f1dfe6c2af67 100644
--- a/mm/hugetlb_cma.c
+++ b/mm/hugetlb_cma.c
@@ -19,6 +19,7 @@
 struct hugetlb_cma_pool {
 	struct cma *cma[MAX_NUMNODES];
 	unsigned long size;
+	unsigned int order;
 };
 
 struct hugetlb_cma_config {
@@ -27,14 +28,31 @@ struct hugetlb_cma_config {
 	unsigned int percent_in_node[MAX_NUMNODES];
 };
 
-static struct hugetlb_cma_pool hugetlb_cma __ro_after_init;
-static struct hugetlb_cma_config hugetlb_cma_config __initdata;
+static struct hugetlb_cma_pool hugetlb_cma[HUGE_MAX_HSTATE] __ro_after_init;
+static struct hugetlb_cma_config hugetlb_cma_config[HUGE_MAX_HSTATE] __initdata;
+static unsigned int hugetlb_cma_nr __ro_after_init;
+static unsigned int hugetlb_cma_default_order __ro_after_init;
 
 static bool hugetlb_cma_only __ro_after_init;
 
 static char hugetlb_cma_cmdline[COMMAND_LINE_SIZE] __initdata;
 static unsigned int hugetlb_cma_cmdline_len __initdata;
 
+static struct hugetlb_cma_pool *hugetlb_cma_pool(unsigned int order)
+{
+	unsigned int i;
+
+	/* Gigantic sizes share the architecture's original CMA pool. */
+	if (order_is_gigantic(order))
+		order = hugetlb_cma_default_order;
+
+	for (i = 0; i < hugetlb_cma_nr; i++) {
+		if (hugetlb_cma[i].order == order)
+			return &hugetlb_cma[i];
+	}
+	return NULL;
+}
+
 #ifdef CONFIG_NUMA
 static phys_addr_t __init memblock_node_memory_size(int nid)
 {
@@ -56,10 +74,16 @@ static phys_addr_t __init memblock_node_memory_size(int nid)
 
 void hugetlb_cma_free_frozen_folio(struct folio *folio)
 {
-	struct hugetlb_cma_pool *pool = &hugetlb_cma;
+	unsigned int i;
+	int nid = folio_nid(folio);
+	unsigned long nr_pages = folio_nr_pages(folio);
 
-	WARN_ON_ONCE(!cma_release_frozen(pool->cma[folio_nid(folio)],
-					 &folio->page, folio_nr_pages(folio)));
+	for (i = 0; i < hugetlb_cma_nr; i++) {
+		if (cma_release_frozen(hugetlb_cma[i].cma[nid], &folio->page, nr_pages))
+			return;
+	}
+
+	WARN_ON_ONCE(1);
 }
 
 struct folio *hugetlb_cma_alloc_frozen_folio(int order, gfp_t gfp_mask,
@@ -70,9 +94,9 @@ struct folio *hugetlb_cma_alloc_frozen_folio(int order, gfp_t gfp_mask,
 	struct page *page = NULL;
 	const nodemask_t *nmask;
 	unsigned int cpuset_mems_cookie;
-	struct hugetlb_cma_pool *pool = &hugetlb_cma;
+	struct hugetlb_cma_pool *pool = hugetlb_cma_pool(order);
 
-	if (!pool->size)
+	if (!pool || !pool->size)
 		return NULL;
 
 retry_cpuset:
@@ -111,7 +135,7 @@ struct folio *hugetlb_cma_alloc_frozen_folio(int order, gfp_t gfp_mask,
 
 void * __init hugetlb_cma_alloc_bootmem(struct hstate *h, int nid, bool node_exact)
 {
-	struct hugetlb_cma_pool *pool = &hugetlb_cma;
+	struct hugetlb_cma_pool *pool = hugetlb_cma_pool(huge_page_order(h));
 	struct cma *cma;
 	void *m;
 	int node;
@@ -135,15 +159,16 @@ void * __init hugetlb_cma_alloc_bootmem(struct hstate *h, int nid, bool node_exa
 
 static int __init hugetlb_cma_parse(char *p)
 {
-	struct hugetlb_cma_pool *pool = &hugetlb_cma;
-	struct hugetlb_cma_config *config = &hugetlb_cma_config;
+	struct hugetlb_cma_pool *pool;
+	struct hugetlb_cma_config *config;
+	unsigned int order = hugetlb_cma_default_order;
+	unsigned int i;
 	int nid, count = 0;
 	unsigned long tmp;
 	char *s = p;
 	char *suffix = strchr(p, '@');
 
 	if (suffix) {
-		unsigned int order = arch_hugetlb_cma_order();
 		unsigned long long page_size;
 		char *end;
 
@@ -159,6 +184,26 @@ static int __init hugetlb_cma_parse(char *p)
 		}
 	}
 
+	if (!order) {
+		pr_warn("hugetlb_cma: the option isn't supported by current arch\n");
+		return -EINVAL;
+	}
+
+	for (i = 0; i < hugetlb_cma_nr; i++) {
+		if (hugetlb_cma[i].order == order)
+			break;
+	}
+
+	if (i == ARRAY_SIZE(hugetlb_cma))
+		return -EINVAL;
+
+	pool = &hugetlb_cma[i];
+	if (i == hugetlb_cma_nr) {
+		hugetlb_cma_nr++;
+		pool->order = order;
+	}
+	config = &hugetlb_cma_config[i];
+
 	while (*s) {
 		if (sscanf(s, "%lu%n", &tmp, &count) != 1)
 			break;
@@ -256,20 +301,14 @@ unsigned int __weak arch_hugetlb_cma_order(void)
 	return 0;
 }
 
-void __init hugetlb_cma_reserve(void)
+static void __init hugetlb_cma_reserve_pool(struct hugetlb_cma_pool *pool,
+					    struct hugetlb_cma_config *config)
 {
-	struct hugetlb_cma_pool *pool = &hugetlb_cma;
-	struct hugetlb_cma_config *config = &hugetlb_cma_config;
-	unsigned long size, reserved, per_node, order, gigantic_page_size;
+	unsigned long size, reserved, per_node, hugepage_size;
+	unsigned int order = pool->order;
 	bool node_specific_cma_alloc = false;
 	bool has_node_specific_param = false;
 	int nid;
-	char *p;
-
-	for (p = hugetlb_cma_cmdline;
-	     p < hugetlb_cma_cmdline + hugetlb_cma_cmdline_len;
-	     p += strlen(p) + 1)
-		hugetlb_cma_parse(p);
 
 	for (nid = 0; nid < MAX_NUMNODES; nid++) {
 		if (config->size_in_node[nid] || config->percent_in_node[nid]) {
@@ -301,12 +340,6 @@ void __init hugetlb_cma_reserve(void)
 	if (!pool->size)
 		return;
 
-	order = arch_hugetlb_cma_order();
-	if (!order) {
-		pr_warn("hugetlb_cma: the option isn't supported by current arch\n");
-		return;
-	}
-
 	/*
 	 * HugeTLB CMA reservation is required for gigantic
 	 * huge pages which could not be allocated via the
@@ -314,12 +347,12 @@ void __init hugetlb_cma_reserve(void)
 	 * breaking this assumption.
 	 */
 	VM_WARN_ON(order <= MAX_PAGE_ORDER);
-	gigantic_page_size = PAGE_SIZE << order;
+	hugepage_size = PAGE_SIZE << order;
 
 	if (config->percent) {
 		unsigned long orig_size = pool->size;
 
-		pool->size = ALIGN_DOWN(pool->size, PAGE_SIZE << order);
+		pool->size = ALIGN_DOWN(pool->size, hugepage_size);
 		if (orig_size && !pool->size)
 			pr_warn("hugetlb_cma: reservation size rounded down to 0 from %lu MiB (%u%%)\n",
 				orig_size / SZ_1M, config->percent);
@@ -331,7 +364,7 @@ void __init hugetlb_cma_reserve(void)
 
 				config->size_in_node[nid] =
 					ALIGN_DOWN(config->size_in_node[nid],
-						   PAGE_SIZE << order);
+						   hugepage_size);
 				if (orig_size && !config->size_in_node[nid])
 					pr_warn("hugetlb_cma: reservation size rounded down to 0 from %lu MiB (%u%%) on node %d\n",
 						orig_size / SZ_1M,
@@ -351,9 +384,9 @@ void __init hugetlb_cma_reserve(void)
 
 		if (!node_isset(nid, hugetlb_bootmem_nodes)) {
 			pr_warn("hugetlb_cma: invalid node %d specified\n", nid);
-		} else if (!IS_ALIGNED(size, gigantic_page_size)) {
+		} else if (!IS_ALIGNED(size, hugepage_size)) {
 			pr_warn("hugetlb_cma: cma area of node %d must be a multiple of %lu MiB\n",
-				nid, gigantic_page_size / SZ_1M);
+				nid, hugepage_size / SZ_1M);
 		} else {
 			node_specific_cma_alloc = true;
 			continue;
@@ -367,9 +400,9 @@ void __init hugetlb_cma_reserve(void)
 	if (!pool->size)
 		return;
 
-	if (!IS_ALIGNED(pool->size, gigantic_page_size)) {
+	if (!IS_ALIGNED(pool->size, hugepage_size)) {
 		pr_warn("hugetlb_cma: cma area must be a multiple of %lu MiB\n",
-			gigantic_page_size / SZ_1M);
+			hugepage_size / SZ_1M);
 		pool->size = 0;
 		return;
 	}
@@ -381,7 +414,7 @@ void __init hugetlb_cma_reserve(void)
 		 */
 		per_node = DIV_ROUND_UP(pool->size,
 					nodes_weight(hugetlb_bootmem_nodes));
-		per_node = round_up(per_node, gigantic_page_size);
+		per_node = round_up(per_node, hugepage_size);
 		if (config->percent)
 			pr_info("hugetlb_cma: reserve %lu MiB (%u%%), up to %lu MiB per node\n",
 				pool->size / SZ_1M, config->percent,
@@ -411,7 +444,7 @@ void __init hugetlb_cma_reserve(void)
 		 * may be returned to CMA allocator in the case of
 		 * huge page demotion.
 		 */
-		res = cma_declare_contiguous_multi(size, gigantic_page_size,
+		res = cma_declare_contiguous_multi(size, hugepage_size,
 					HUGETLB_PAGE_ORDER, name,
 					&pool->cma[nid], nid);
 		if (res || !cma_validate_zones(pool->cma[nid])) {
@@ -434,28 +467,49 @@ void __init hugetlb_cma_reserve(void)
 			break;
 	}
 
-	if (!reserved)
-		/*
-		 * pool->size is used to determine if allocations from
-		 * cma are possible.  Set to zero if no cma regions are set up.
-		 */
-		pool->size = 0;
+	/* Only successfully reserved areas enable CMA allocation for this size. */
+	pool->size = reserved;
+}
+
+void __init hugetlb_cma_reserve(void)
+{
+	unsigned int i;
+	char *p;
+
+	hugetlb_cma_default_order = arch_hugetlb_cma_order();
+
+	for (p = hugetlb_cma_cmdline;
+	     p < hugetlb_cma_cmdline + hugetlb_cma_cmdline_len;
+	     p += strlen(p) + 1)
+		hugetlb_cma_parse(p);
+
+	for (i = 0; i < hugetlb_cma_nr; i++)
+		hugetlb_cma_reserve_pool(&hugetlb_cma[i], &hugetlb_cma_config[i]);
 }
 
-bool hugetlb_cma_exclusive_alloc(void)
+bool hugetlb_cma_exclusive_alloc(unsigned int order)
 {
-	return hugetlb_cma_only;
+	struct hugetlb_cma_pool *pool = hugetlb_cma_pool(order);
+
+	return hugetlb_cma_only && pool && pool->size;
 }
 
-unsigned long __init hugetlb_cma_total_size(void)
+unsigned long __init hugetlb_cma_size(struct hstate *h)
 {
-	return hugetlb_cma.size;
+	struct hugetlb_cma_pool *pool = hugetlb_cma_pool(huge_page_order(h));
+
+	return pool ? pool->size : 0;
 }
 
-void __init hugetlb_cma_validate_params(void)
+unsigned long __init hugetlb_cma_total_size(void)
 {
-	if (!hugetlb_cma.size)
-		hugetlb_cma_only = false;
+	unsigned long size = 0;
+	unsigned int i;
+
+	for (i = 0; i < hugetlb_cma_nr; i++)
+		size += hugetlb_cma[i].size;
+
+	return size;
 }
 
 bool __init hugetlb_early_cma(struct hstate *h)
@@ -463,5 +517,5 @@ bool __init hugetlb_early_cma(struct hstate *h)
 	if (arch_has_huge_bootmem_alloc())
 		return false;
 
-	return hstate_is_gigantic(h) && hugetlb_cma_only;
+	return hstate_is_gigantic(h) && hugetlb_cma_exclusive_alloc(huge_page_order(h));
 }
diff --git a/mm/hugetlb_cma.h b/mm/hugetlb_cma.h
index 730b2b4965b64..3d829258f6728 100644
--- a/mm/hugetlb_cma.h
+++ b/mm/hugetlb_cma.h
@@ -9,9 +9,9 @@ void hugetlb_cma_free_frozen_folio(struct folio *folio);
 struct folio *hugetlb_cma_alloc_frozen_folio(int order, gfp_t gfp_mask,
 				      int nid, nodemask_t *nodemask);
 void *hugetlb_cma_alloc_bootmem(struct hstate *h, int nid, bool node_exact);
-bool hugetlb_cma_exclusive_alloc(void);
+bool hugetlb_cma_exclusive_alloc(unsigned int order);
+unsigned long hugetlb_cma_size(struct hstate *h);
 unsigned long hugetlb_cma_total_size(void);
-void hugetlb_cma_validate_params(void);
 bool hugetlb_early_cma(struct hstate *h);
 #else
 static inline void hugetlb_cma_free_frozen_folio(struct folio *folio)
@@ -30,18 +30,19 @@ static inline void *hugetlb_cma_alloc_bootmem(struct hstate *h, int nid,
 	return NULL;
 }
 
-static inline bool hugetlb_cma_exclusive_alloc(void)
+static inline bool hugetlb_cma_exclusive_alloc(unsigned int order)
 {
 	return false;
 }
 
-static inline unsigned long hugetlb_cma_total_size(void)
+static inline unsigned long hugetlb_cma_size(struct hstate *h)
 {
 	return 0;
 }
 
-static inline void hugetlb_cma_validate_params(void)
+static inline unsigned long hugetlb_cma_total_size(void)
 {
+	return 0;
 }
 
 static inline bool hugetlb_early_cma(struct hstate *h)
-- 
2.54.0

[PATCH 4/5] mm: hugetlb: avoid recursive CMA allocation during migration

[Qi Zheng]

From: Qi Zheng <zhengqi.arch@bytedance.com>

Contiguous allocation can migrate allocated HugeTLB folios or replace free
HugeTLB folios while holding a CMA allocation mutex. Allocating a
replacement through CMA would recurse into that allocation path once
non-gigantic HugeTLB pages can use CMA.

Add a fresh allocation helper that bypasses CMA, and use it for free folio
replacement. Pass an explicit allow_cma argument through the migration
allocation API and disable it for MR_CONTIG_RANGE. Honor hugetlb_cma_only
by failing fresh allocation when the corresponding size requires CMA.

The existing gigantic migration restriction is unchanged.

Assisted-by: LLM
Signed-off-by: Qi Zheng <zhengqi.arch@bytedance.com>
---
 include/linux/hugetlb.h |  4 ++--
 mm/hugetlb.c            | 45 ++++++++++++++++++++++++++++++++---------
 mm/mempolicy.c          |  2 +-
 mm/migrate.c            |  3 ++-
 4 files changed, 41 insertions(+), 13 deletions(-)

diff --git a/include/linux/hugetlb.h b/include/linux/hugetlb.h
index 5029c72418631..38612f7b3478f 100644
--- a/include/linux/hugetlb.h
+++ b/include/linux/hugetlb.h
@@ -736,7 +736,7 @@ struct folio *alloc_hugetlb_folio(struct vm_area_struct *vma,
 				unsigned long addr, bool cow_from_owner);
 struct folio *alloc_hugetlb_folio_nodemask(struct hstate *h, int preferred_nid,
 				nodemask_t *nmask, gfp_t gfp_mask,
-				bool allow_alloc_fallback);
+				bool allow_alloc_fallback, bool allow_cma);
 struct folio *alloc_hugetlb_folio_reserve(struct hstate *h, int preferred_nid,
 					  nodemask_t *nmask, gfp_t gfp_mask);
 
@@ -1172,7 +1172,7 @@ alloc_hugetlb_folio_reserve(struct hstate *h, int preferred_nid,
 static inline struct folio *
 alloc_hugetlb_folio_nodemask(struct hstate *h, int preferred_nid,
 			nodemask_t *nmask, gfp_t gfp_mask,
-			bool allow_alloc_fallback)
+			bool allow_alloc_fallback, bool allow_cma)
 {
 	return NULL;
 }
diff --git a/mm/hugetlb.c b/mm/hugetlb.c
index 9d569eae852c5..218fbaee6f7e5 100644
--- a/mm/hugetlb.c
+++ b/mm/hugetlb.c
@@ -1860,8 +1860,7 @@ static struct folio *only_alloc_fresh_hugetlb_folio(struct hstate *h,
 }
 
 /*
- * Common helper to allocate a fresh hugetlb folio. All specific allocators
- * should use this function to get new hugetlb folio
+ * Common helper to allocate a fresh hugetlb folio.
  *
  * Note that returned folio is 'frozen':  ref count of head page and all tail
  * pages is zero, and the accounting must be done in the caller.
@@ -1877,6 +1876,30 @@ static struct folio *alloc_fresh_hugetlb_folio(struct hstate *h,
 	return folio;
 }
 
+/*
+ * Contiguous allocation can replace or migrate hugetlb pages while holding
+ * a CMA allocation mutex. Do not recurse into CMA to allocate replacements.
+ */
+static struct folio *alloc_fresh_hugetlb_folio_nocma(struct hstate *h, gfp_t gfp_mask,
+						     int nid, nodemask_t *nmask)
+{
+	struct folio *folio;
+	unsigned int order = huge_page_order(h);
+
+	if (hugetlb_cma_exclusive_alloc(order))
+		return NULL;
+
+	if (nid == NUMA_NO_NODE)
+		nid = numa_mem_id();
+
+	folio = alloc_buddy_frozen_folio(order, gfp_mask, nid, nmask, NULL);
+	if (folio) {
+		init_new_hugetlb_folio(folio);
+		hugetlb_vmemmap_optimize_folio(h, folio);
+	}
+	return folio;
+}
+
 void prep_and_add_allocated_folios(struct hstate *h,
 				   struct list_head *folio_list)
 {
@@ -2139,14 +2162,17 @@ static struct folio *alloc_surplus_hugetlb_folio(struct hstate *h,
 }
 
 static struct folio *alloc_migrate_hugetlb_folio(struct hstate *h, gfp_t gfp_mask,
-				     int nid, nodemask_t *nmask)
+				     int nid, nodemask_t *nmask, bool allow_cma)
 {
 	struct folio *folio;
 
 	if (hstate_is_gigantic(h))
 		return NULL;
 
-	folio = alloc_fresh_hugetlb_folio(h, gfp_mask, nid, nmask);
+	if (allow_cma)
+		folio = alloc_fresh_hugetlb_folio(h, gfp_mask, nid, nmask);
+	else
+		folio = alloc_fresh_hugetlb_folio_nocma(h, gfp_mask, nid, nmask);
 	if (!folio)
 		return NULL;
 
@@ -2219,7 +2245,8 @@ struct folio *alloc_hugetlb_folio_reserve(struct hstate *h, int preferred_nid,
 
 /* folio migration callback function */
 struct folio *alloc_hugetlb_folio_nodemask(struct hstate *h, int preferred_nid,
-		nodemask_t *nmask, gfp_t gfp_mask, bool allow_alloc_fallback)
+		nodemask_t *nmask, gfp_t gfp_mask, bool allow_alloc_fallback,
+		bool allow_cma)
 {
 	struct folio *folio = NULL;
 
@@ -2238,7 +2265,7 @@ struct folio *alloc_hugetlb_folio_nodemask(struct hstate *h, int preferred_nid,
 			gfp_mask |= __GFP_THISNODE;
 
 		folio = alloc_migrate_hugetlb_folio(h, gfp_mask, preferred_nid,
-						    nmask);
+						    nmask, allow_cma);
 	}
 
 	if (folio)
@@ -2722,8 +2749,8 @@ static int alloc_and_dissolve_hugetlb_folio(struct folio *old_folio,
 		if (!new_folio) {
 			spin_unlock_irq(&hugetlb_lock);
 			gfp_mask = htlb_alloc_mask(h) | __GFP_THISNODE;
-			new_folio = alloc_fresh_hugetlb_folio(h, gfp_mask,
-							      nid, NULL);
+			new_folio = alloc_fresh_hugetlb_folio_nocma(h, gfp_mask,
+								    nid, NULL);
 			if (!new_folio)
 				return -ENOMEM;
 			goto retry;
@@ -6333,7 +6360,7 @@ static struct folio *alloc_hugetlb_folio_vma(struct hstate *h,
 	 * consuming a reservation. Set the alloc_fallback to false to indicate
 	 * that breaking the per-node hugetlb pool is not allowed in this case.
 	 */
-	folio = alloc_hugetlb_folio_nodemask(h, node, nodemask, gfp_mask, false);
+	folio = alloc_hugetlb_folio_nodemask(h, node, nodemask, gfp_mask, false, true);
 	mpol_cond_put(mpol);
 
 	return folio;
diff --git a/mm/mempolicy.c b/mm/mempolicy.c
index 70298fded1b4a..b59b8c73b8ce3 100644
--- a/mm/mempolicy.c
+++ b/mm/mempolicy.c
@@ -1471,7 +1471,7 @@ static struct folio *alloc_migration_target_by_mpol(struct folio *src,
 		gfp = htlb_alloc_mask(h);
 		nodemask = policy_nodemask(gfp, pol, ilx, &nid);
 		return alloc_hugetlb_folio_nodemask(h, nid, nodemask, gfp,
-				htlb_allow_alloc_fallback(MR_MEMPOLICY_MBIND));
+				htlb_allow_alloc_fallback(MR_MEMPOLICY_MBIND), true);
 	}
 
 	if (folio_test_large(src))
diff --git a/mm/migrate.c b/mm/migrate.c
index 7bdcdb57652f8..670556bf5de2c 100644
--- a/mm/migrate.c
+++ b/mm/migrate.c
@@ -2213,7 +2213,8 @@ struct folio *alloc_migration_target(struct folio *src, unsigned long private)
 		gfp_mask = htlb_modify_alloc_mask(h, gfp_mask);
 		return alloc_hugetlb_folio_nodemask(h, nid,
 						mtc->nmask, gfp_mask,
-						htlb_allow_alloc_fallback(mtc->reason));
+						htlb_allow_alloc_fallback(mtc->reason),
+						mtc->reason != MR_CONTIG_RANGE);
 	}
 
 	if (folio_test_large(src)) {
-- 
2.54.0

[PATCH 5/5] mm: hugetlb: enable CMA allocation for PMD-sized pages

[Qi Zheng]

From: Qi Zheng <zhengqi.arch@bytedance.com>

Allow hugetlb_cma=<capacity>@<page-size> to select the architecture's PMD
size in addition to its existing CMA size. PMD pages are 2M on x86-64 and
2M, 32M or 512M on arm64 with 4K, 16K or 64K base pages.

Give PMD pages their own pool without changing the original pool's sharing
among other gigantic sizes. On arm64 with 4K base pages, 32M and 1G pages
continue to use the default 1G-aligned pool. Unsuffixed arguments and an
explicit default size still configure that same pool.

PMD requests use only the PMD pool, even when their order is gigantic.
Other gigantic requests use only the original pool. Apply this mapping
consistently to allocation, CMA-only policy and boot-time accounting.

Move CMA allocation into the common fresh HugeTLB allocation path so
non-gigantic pages can use CMA. Align reservations to both the selected
page size and CMA's minimum alignment, and use a bitmap granularity no
larger than the selected order. Keep the original CMA area names and give
PMD areas distinct names. Document the syntax and pool selection.

Assisted-by: LLM
Signed-off-by: Qi Zheng <zhengqi.arch@bytedance.com>
---
 .../admin-guide/kernel-parameters.txt         | 39 ++++++++++++-------
 mm/hugetlb.c                                  | 31 +++++++--------
 mm/hugetlb_cma.c                              | 26 ++++++-------
 3 files changed, 52 insertions(+), 44 deletions(-)

diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentation/admin-guide/kernel-parameters.txt
index e3f25e805fea6..71cd431f07fe2 100644
--- a/Documentation/admin-guide/kernel-parameters.txt
+++ b/Documentation/admin-guide/kernel-parameters.txt
@@ -2065,31 +2065,42 @@ Kernel parameters
 			Note that this parameter only applies to non-gigantic huge pages.
 
 	hugetlb_cma=	[HW,CMA,EARLY] The size of a CMA area used for allocation
-			of gigantic hugepages. Or using node format, the size
-			of a CMA area per node can be specified.
+			of hugepages. Or using node format, the size of a CMA area
+			per node can be specified.
 			The size can be an absolute value (e.g., 2G) or a
 			percentage of the total memory or node memory (e.g., 20%).
 			Percentage-derived sizes are rounded down to a multiple of
-			the architecture's gigantic hugepage size and may become
-			zero.
-			Format: nn[KMGTPE] or nn% or (node format)
-				<node>:nn[KMGTPE][,<node>:nn[KMGTPE]] or
-				<node>:nn%[,<node>:nn%]
-
-			The size must be a multiple of the gigantic page size.
+			the larger of the selected hugepage size and CMA's minimum
+			alignment, and may become zero.
+			Format: nn[KMGTPE][@<page-size>] or nn%[@<page-size>]
+				or (node format)
+				<node>:nn[KMGTPE][,<node>:nn[KMGTPE]][@<page-size>] or
+				<node>:nn%[,<node>:nn%][@<page-size>]
+
+			An optional @<page-size> suffix selects the architecture's
+			default CMA hugepage size or a supported PMD size. If
+			omitted, the default is used. For node format, append
+			the suffix after the entire node list. Repeat the
+			parameter to configure the two pools independently.
+
+			The size must be aligned to both the selected hugepage
+			size and CMA's minimum alignment.
 			When using node format, this applies to each per-node size.
 			Missaligned values are dropped with a warning.
 
-			Reserve a CMA area of given size and allocate gigantic
-			hugepages using the CMA allocator. If enabled, the
-			boot-time allocation of gigantic hugepages is skipped.
+			Reserve a CMA area of given size and allocate hugepages
+			using the CMA allocator. For gigantic page sizes with a
+			reserved CMA pool, normal boot-time allocation is skipped.
+			With hugetlb_cma_only=1, early CMA allocation is used where
+			supported.
+
 
 	hugetlb_cma_only=
 			[HW,CMA,EARLY] When allocating new HugeTLB pages, only
 			try to allocate from the CMA areas.
 
-			This option does nothing if hugetlb_cma= is not also
-			specified.
+			This option only affects page sizes with a CMA pool
+			reserved by hugetlb_cma=.
 
 	hugetlb_free_vmemmap=
 			[KNL] Requires CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP
diff --git a/mm/hugetlb.c b/mm/hugetlb.c
index 218fbaee6f7e5..8c00fd9d0658d 100644
--- a/mm/hugetlb.c
+++ b/mm/hugetlb.c
@@ -1362,18 +1362,8 @@ static struct folio *dequeue_hugetlb_folio(struct hstate *h, gfp_t gfp_mask,
 static struct folio *alloc_gigantic_frozen_folio(int order, gfp_t gfp_mask,
 		int nid, nodemask_t *nodemask)
 {
-	struct folio *folio;
-
-	folio = hugetlb_cma_alloc_frozen_folio(order, gfp_mask, nid, nodemask);
-	if (folio)
-		return folio;
-
-	if (hugetlb_cma_exclusive_alloc(order))
-		return NULL;
-
-	folio = (struct folio *)alloc_contig_frozen_pages(1 << order, gfp_mask,
-							  nid, nodemask);
-	return folio;
+	return (struct folio *)alloc_contig_frozen_pages(1 << order, gfp_mask,
+							 nid, nodemask);
 }
 #else /* !CONFIG_ARCH_HAS_GIGANTIC_PAGE || !CONFIG_CONTIG_ALLOC */
 static struct folio *alloc_gigantic_frozen_folio(int order, gfp_t gfp_mask, int nid,
@@ -1849,11 +1839,18 @@ static struct folio *only_alloc_fresh_hugetlb_folio(struct hstate *h,
 	if (nid == NUMA_NO_NODE)
 		nid = numa_mem_id();
 
-	if (order_is_gigantic(order))
-		folio = alloc_gigantic_frozen_folio(order, gfp_mask, nid, nmask);
-	else
-		folio = alloc_buddy_frozen_folio(order, gfp_mask, nid, nmask,
-						 node_alloc_noretry);
+	folio = hugetlb_cma_alloc_frozen_folio(order, gfp_mask, nid, nmask);
+	if (!folio) {
+		if (hugetlb_cma_exclusive_alloc(order))
+			return NULL;
+
+		if (order_is_gigantic(order))
+			folio = alloc_gigantic_frozen_folio(order, gfp_mask, nid, nmask);
+		else
+			folio = alloc_buddy_frozen_folio(order, gfp_mask, nid, nmask,
+							 node_alloc_noretry);
+	}
+
 	if (folio)
 		init_new_hugetlb_folio(folio);
 	return folio;
diff --git a/mm/hugetlb_cma.c b/mm/hugetlb_cma.c
index 9f1dfe6c2af67..9dd376979e345 100644
--- a/mm/hugetlb_cma.c
+++ b/mm/hugetlb_cma.c
@@ -42,8 +42,8 @@ static struct hugetlb_cma_pool *hugetlb_cma_pool(unsigned int order)
 {
 	unsigned int i;
 
-	/* Gigantic sizes share the architecture's original CMA pool. */
-	if (order_is_gigantic(order))
+	/* PMD pages use their own pool; other gigantic sizes share. */
+	if (order != PMD_SHIFT - PAGE_SHIFT && order_is_gigantic(order))
 		order = hugetlb_cma_default_order;
 
 	for (i = 0; i < hugetlb_cma_nr; i++) {
@@ -177,11 +177,13 @@ static int __init hugetlb_cma_parse(char *p)
 			pr_warn("hugetlb_cma: invalid parameter %s\n", p);
 			return -EINVAL;
 		}
-		if (!order || page_size != (PAGE_SIZE << order) ||
+		if ((page_size != PMD_SIZE &&
+		     page_size != (PAGE_SIZE << order)) ||
 		    !arch_hugetlb_valid_size(page_size)) {
 			pr_warn("hugetlb_cma: unsupported page size in %s\n", p);
 			return -EINVAL;
 		}
+		order = ilog2(page_size) - PAGE_SHIFT;
 	}
 
 	if (!order) {
@@ -340,14 +342,7 @@ static void __init hugetlb_cma_reserve_pool(struct hugetlb_cma_pool *pool,
 	if (!pool->size)
 		return;
 
-	/*
-	 * HugeTLB CMA reservation is required for gigantic
-	 * huge pages which could not be allocated via the
-	 * page allocator. Just warn if there is any change
-	 * breaking this assumption.
-	 */
-	VM_WARN_ON(order <= MAX_PAGE_ORDER);
-	hugepage_size = PAGE_SIZE << order;
+	hugepage_size = max(PAGE_SIZE << order, CMA_MIN_ALIGNMENT_BYTES);
 
 	if (config->percent) {
 		unsigned long orig_size = pool->size;
@@ -438,14 +433,19 @@ static void __init hugetlb_cma_reserve_pool(struct hugetlb_cma_pool *pool,
 			size = min(per_node, pool->size - reserved);
 		}
 
-		snprintf(name, sizeof(name), "hugetlb%d", nid);
+		if (order == hugetlb_cma_default_order)
+			snprintf(name, sizeof(name), "hugetlb%d", nid);
+		else
+			snprintf(name, sizeof(name), "hugetlb-%lukB-%d",
+				 (PAGE_SIZE << order) / SZ_1K, nid);
+
 		/*
 		 * Note that 'order per bit' is based on smallest size that
 		 * may be returned to CMA allocator in the case of
 		 * huge page demotion.
 		 */
 		res = cma_declare_contiguous_multi(size, hugepage_size,
-					HUGETLB_PAGE_ORDER, name,
+					min_t(unsigned int, order, HUGETLB_PAGE_ORDER), name,
 					&pool->cma[nid], nid);
 		if (res || !cma_validate_zones(pool->cma[nid])) {
 			pr_warn("hugetlb_cma: %s: err %d, node %d\n",
-- 
2.54.0

Re: [PATCH 1/5] mm: hugetlb: add explicit page size to hugetlb_cma

[Jose A. Perez de Azpillaga]

On 10/16/26 12:07 PM, Qi Zheng wrote:
> From: Qi Zheng <zhengqi.arch@bytedance.com>
> 
> Accept an optional @<page-size> suffix for hugetlb_cma. Initially accept
> only the architecture's existing CMA page size. Preserve the behavior of
> unsuffixed arguments and the existing hugepagesz/hugepages options.
> 
> Validate the suffix before applying the existing capacity, percentage and
> node-list parser.
> 
> Save the arguments during early parsing and process them at reservation
> time, after architecture-specific page-size information has been
> initialized.
> 
> Assisted-by: LLM
> Signed-off-by: Qi Zheng <zhengqi.arch@bytedance.com>

this LGTM so:

Reviewed-by: Jose A. Perez de Azpillaga <azpijr@gmail.com>

-- 
cheers,
jose a. p-a

Re: [PATCH 2/5] mm: hugetlb: group CMA pool and reservation state

[Jose A. Perez de Azpillaga]

On 10/16/26 12:07 PM, Qi Zheng wrote:
> From: Qi Zheng <zhengqi.arch@bytedance.com>
> 
> The HugeTLB CMA areas, total size, per-node sizes and percentages are
> stored in separate singleton variables. Group the runtime areas and
> size into a pool, and the temporary reservation inputs into a separate
> configuration structure.
> 
> Keep one instance of each structure and preserve the existing parsing,
> reservation and allocation behavior. This separates the mechanical state
> conversion from adding support for multiple page orders.
> 
> Assisted-by: LLM
> Signed-off-by: Qi Zheng <zhengqi.arch@bytedance.com>

this LGTM too so:

Reviewed-by: Jose A. Perez de Azpillaga <azpijr@gmail.com>

-- 
cheers,
jose a. p-a

Re: [PATCH 3/5] mm: hugetlb: track CMA reservations by page order

[Jose A. Perez de Azpillaga]

On 10/16/26 12:07 PM, Qi Zheng wrote:
> From: Qi Zheng <zhengqi.arch@bytedance.com>
> 
> Prepare for a separate CMA pool for PMD-sized HugeTLB pages by organizing
> CMA areas and reservation parameters into pools identified by page order.
> 
> Select the corresponding pool for allocation, CMA-only checks and
> boot-time allocation decisions. Release folios through the CMA area that
> owns their physical memory, including after demotion.
> 
> Record each pool's successfully reserved capacity. Remove
> hugetlb_cma_validate_params(), since the per-pool CMA-only check already
> handles missing or empty pools.
> 
> Continue accepting only the architecture's existing CMA page size and
> preserve the default pool's use by all gigantic sizes. PMD-sized pool
> support will be added in a subsequent patch.
> 
> Assisted-by: LLM
> Signed-off-by: Qi Zheng <zhengqi.arch@bytedance.com>

[snip]

> +static struct hugetlb_cma_pool hugetlb_cma[HUGE_MAX_HSTATE] __ro_after_init;
> +static struct hugetlb_cma_config hugetlb_cma_config[HUGE_MAX_HSTATE] __initdata;
> +static unsigned int hugetlb_cma_nr __ro_after_init;

the commit message prepares for one more pool, the PMD one, and
hugetlb_cma_pool() folds every gigantic order into the default pool.
would a fixed pair work instead of an order-keyed array?

-- 
cheers,
jose a. p-a

Re: [PATCH 4/5] mm: hugetlb: avoid recursive CMA allocation during migration

[Jose A. Perez de Azpillaga]

On 10/16/26 12:07 PM, Qi Zheng wrote:
> From: Qi Zheng <zhengqi.arch@bytedance.com>
> 
> Contiguous allocation can migrate allocated HugeTLB folios or replace free
> HugeTLB folios while holding a CMA allocation mutex. Allocating a
> replacement through CMA would recurse into that allocation path once
> non-gigantic HugeTLB pages can use CMA.
> 
> Add a fresh allocation helper that bypasses CMA, and use it for free folio
> replacement. Pass an explicit allow_cma argument through the migration
> allocation API and disable it for MR_CONTIG_RANGE. Honor hugetlb_cma_only
> by failing fresh allocation when the corresponding size requires CMA.
> 
> The existing gigantic migration restriction is unchanged.
> 
> Assisted-by: LLM
> Signed-off-by: Qi Zheng <zhengqi.arch@bytedance.com>

[snip]

>  /*
> - * Common helper to allocate a fresh hugetlb folio. All specific allocators
> - * should use this function to get new hugetlb folio
> + * Common helper to allocate a fresh hugetlb folio.

couldn't allow_cma be passed down to alloc_fresh_hugetlb_folio() instead
of adding a second copy of it, so this line can stay?

-- 
cheers,
jose a. p-a

Re: [PATCH 5/5] mm: hugetlb: enable CMA allocation for PMD-sized pages

[Jose A. Perez de Azpillaga]

On 10/16/26 12:07 PM, Qi Zheng wrote:
> From: Qi Zheng <zhengqi.arch@bytedance.com>
> 
> Allow hugetlb_cma=<capacity>@<page-size> to select the architecture's PMD
> size in addition to its existing CMA size. PMD pages are 2M on x86-64 and
> 2M, 32M or 512M on arm64 with 4K, 16K or 64K base pages.
> 
> Give PMD pages their own pool without changing the original pool's sharing
> among other gigantic sizes. On arm64 with 4K base pages, 32M and 1G pages
> continue to use the default 1G-aligned pool. Unsuffixed arguments and an
> explicit default size still configure that same pool.
> 
> PMD requests use only the PMD pool, even when their order is gigantic.
> Other gigantic requests use only the original pool. Apply this mapping
> consistently to allocation, CMA-only policy and boot-time accounting.
> 
> Move CMA allocation into the common fresh HugeTLB allocation path so
> non-gigantic pages can use CMA. Align reservations to both the selected
> page size and CMA's minimum alignment, and use a bitmap granularity no
> larger than the selected order. Keep the original CMA area names and give
> PMD areas distinct names. Document the syntax and pool selection.
> 
> Assisted-by: LLM
> Signed-off-by: Qi Zheng <zhengqi.arch@bytedance.com>

[snip]

> +	folio = hugetlb_cma_alloc_frozen_folio(order, gfp_mask, nid, nmask);
> +	if (!folio) {

boot-time hugepages= goes through here too, so hugepages=128 with
hugetlb_cma=512M@2M already uses half of the pool at boot. would buddy
first and cma as fallback make more sense for PMD?

[snip]

> +	if (order != PMD_SHIFT - PAGE_SHIFT && order_is_gigantic(order))
>  		order = hugetlb_cma_default_order;

doesn't this take gigantic PMD-order pages out of the default pool even
when no PMD pool is set? would trying the exact order first and then
falling back to the default pool work?

I tested it on qemu and saw no issues, but I'm hesitant about the points
above. I can give a Tested-by tag on the next version :)

-- 
cheers,
jose a. p-a