* + mm-sparse-vmemmap-allocate-shared-tail-page-array-dynamically.patch added to mm-unstable branch
@ 2026-09-27 5:51 Andrew Morton
0 siblings, 0 replies; 2+ messages in thread
From: Andrew Morton @ 2026-09-27 5:51 UTC (permalink / raw)
To: mm-commits, muchun.song, akpm
The patch titled
Subject: mm/sparse-vmemmap: allocate shared tail page array dynamically
has been added to the -mm mm-unstable branch. Its filename is
mm-sparse-vmemmap-allocate-shared-tail-page-array-dynamically.patch
This patch will shortly appear at
https://git.kernel.org/pub/scm/linux/kernel/git/akpm/25-new.git/tree/patches/mm-sparse-vmemmap-allocate-shared-tail-page-array-dynamically.patch
This patch will later appear in the mm-unstable branch at
git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm
Before you just go and hit "reply", please:
a) Consider who else should be cc'ed
b) Prefer to cc a suitable mailing list as well
c) Ideally: find the original patch on the mailing list and do a
reply-to-all to that, adding suitable additional cc's
*** Remember to use Documentation/process/submit-checklist.rst when testing your code ***
The -mm tree is included into linux-next via various
branches at git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm
and is updated there most days
------------------------------------------------------
From: Muchun Song <muchun.song@linux.dev>
Subject: mm/sparse-vmemmap: allocate shared tail page array dynamically
Date: Sun, 27 Sep 2026 10:54:31 +0800
Commit 622026e87c40 ("mm/hugetlb: remove fake head pages") added the
per-zone vmemmap_tails array. Its size depends on MAX_FOLIO_ORDER, which
had been moved to mmzone.h in preparation for the array.
PUD_ORDER is defined by linux/pgtable.h, which cannot be included from
mmzone.h without creating an include cycle. It was therefore open-coded
as PUD_SHIFT - PAGE_SHIFT.
This removed the dependency on PUD_ORDER, but not the underlying
dependency on architecture page-table definitions. PUD_SHIFT is
generally provided by architecture page-table headers, which are not
guaranteed to have been included when mmzone.h is parsed.
The dependency remained hidden because vmemmap_tails was originally
guarded by CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP. Under that condition,
MAX_FOLIO_ORDER resolves to either MAX_PAGE_ORDER or the fixed HugeTLB
limit, rather than the PUD_SHIFT-based definition.
Device DAX, however, does not require CONFIG_HUGETLB_PAGE. When it is
converted to use section-based vmemmap optimization, MAX_FOLIO_ORDER can
resolve to PUD_SHIFT - PAGE_SHIFT while it is being used to size
vmemmap_tails. This would make struct zone depend on architecture
page-table definitions being available when mmzone.h is parsed.
Replace the embedded array with a pointer and allocate it on first use.
This moves the order-count evaluation into sparse-vmemmap.c, after the
architecture page-table definitions are available, and removes the
dependency from mmzone.h.
Removing the compile-time array also removes the original reason for
keeping MAX_FOLIO_ORDER and the vmemmap optimization sizing definitions
in mmzone.h. Follow-up cleanups can place each definition in the header
owned by its respective subsystem.
Link: https://lore.kernel.org/20260927025441.741633-3-songmuchun@bytedance.com
Signed-off-by: Muchun Song <muchun.song@linux.dev>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Cc: David Hildenbrand <david@kernel.org>
Cc: Oscar Salvador <osalvador@suse.de>
Cc: Madhavan Srinivasan <maddy@linux.ibm.com>
Cc: Michael Ellerman <mpe@ellerman.id.au>
Cc: Jonathan Corbet <corbet@lwn.net>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Mike Rapoport <rppt@kernel.org>
Cc: Qi Zheng <qi.zheng@linux.dev>
Cc: Nicholas Piggin <npiggin@gmail.com>
Cc: Christophe Leroy <chleroy@kernel.org>
Cc: Randy Dunlap <rdunlap@infradead.org>
Cc: Lance Yang <lance.yang@linux.dev>
---
include/linux/mmzone.h | 7 +------
mm/sparse-vmemmap.c | 35 +++++++++++++++++++++++++++++++----
2 files changed, 32 insertions(+), 10 deletions(-)
--- a/include/linux/mmzone.h~mm-sparse-vmemmap-allocate-shared-tail-page-array-dynamically
+++ a/include/linux/mmzone.h
@@ -113,11 +113,6 @@
(VMEMMAP_OPTIMIZATION_PAGES * PAGE_SIZE / sizeof(struct page))
#define VMEMMAP_OPTIMIZATION_MIN_ORDER (ilog2(VMEMMAP_OPTIMIZATION_NR_STRUCT_PAGES) + 1)
-#define __VMEMMAP_OPTIMIZATION_NR_ORDERS \
- (MAX_FOLIO_ORDER - VMEMMAP_OPTIMIZATION_MIN_ORDER + 1)
-#define VMEMMAP_OPTIMIZATION_NR_ORDERS \
- (__VMEMMAP_OPTIMIZATION_NR_ORDERS > 0 ? __VMEMMAP_OPTIMIZATION_NR_ORDERS : 0)
-
enum migratetype {
MIGRATE_UNMOVABLE,
MIGRATE_MOVABLE,
@@ -1156,7 +1151,7 @@ struct zone {
atomic_long_t vm_stat[NR_VM_ZONE_STAT_ITEMS];
atomic_long_t vm_numa_event[NR_VM_NUMA_EVENT_ITEMS];
#ifdef CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP
- struct page *vmemmap_tails[VMEMMAP_OPTIMIZATION_NR_ORDERS];
+ struct page **vmemmap_tails;
#endif
} ____cacheline_internodealigned_in_smp;
--- a/mm/sparse-vmemmap.c~mm-sparse-vmemmap-allocate-shared-tail-page-array-dynamically
+++ a/mm/sparse-vmemmap.c
@@ -170,16 +170,43 @@ static void * __meminit vmemmap_alloc_bl
}
#ifdef CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP
+#define VMEMMAP_OPTIMIZATION_NR_ORDERS (MAX_FOLIO_ORDER - VMEMMAP_OPTIMIZATION_MIN_ORDER + 1)
+
+static __ref struct page **vmemmap_tails_alloc(struct zone *zone)
+{
+ struct page **pages;
+ const size_t size = array_size(VMEMMAP_OPTIMIZATION_NR_ORDERS, sizeof(*pages));
+
+ pages = slab_is_available() ? kzalloc_objs(*pages, VMEMMAP_OPTIMIZATION_NR_ORDERS) :
+ memblock_alloc(size, __alignof__(*pages));
+ if (!pages)
+ return NULL;
+
+ if (cmpxchg(&zone->vmemmap_tails, NULL, pages) != NULL) {
+ if (slab_is_available())
+ kfree(pages);
+ else
+ memblock_free(pages, size);
+ pages = READ_ONCE(zone->vmemmap_tails);
+ }
+
+ return pages;
+}
+
struct page __ref *vmemmap_shared_tail_page(unsigned int order, struct zone *zone)
{
void *addr;
- struct page *page;
+ struct page *page, **pages;
const unsigned int idx = order - VMEMMAP_OPTIMIZATION_MIN_ORDER;
if (WARN_ON_ONCE(idx >= VMEMMAP_OPTIMIZATION_NR_ORDERS))
return NULL;
- page = READ_ONCE(zone->vmemmap_tails[idx]);
+ pages = READ_ONCE(zone->vmemmap_tails) ? : vmemmap_tails_alloc(zone);
+ if (!pages)
+ return NULL;
+
+ page = READ_ONCE(pages[idx]);
if (likely(page))
return page;
@@ -194,12 +221,12 @@ struct page __ref *vmemmap_shared_tail_p
}
page = virt_to_page(addr);
- if (cmpxchg(&zone->vmemmap_tails[idx], NULL, page) != NULL) {
+ if (cmpxchg(&pages[idx], NULL, page) != NULL) {
if (slab_is_available())
__free_page(page);
else
memblock_free(addr, PAGE_SIZE);
- page = READ_ONCE(zone->vmemmap_tails[idx]);
+ page = READ_ONCE(pages[idx]);
}
return page;
_
Patches currently in -mm which might be from muchun.song@linux.dev are
mm-sparse-vmemmap-factor-out-shared-vmemmap-tail-page-allocation.patch
mm-sparse-vmemmap-allocate-shared-tail-page-array-dynamically.patch
mm-sparse-vmemmap-introduce-config_vmemmap_optimization.patch
mm-sparse-vmemmap-open-code-init_compound_tail.patch
mm-sparse-vmemmap-prepare-dax-vmemmap-population-for-compound-page-orders.patch
mm-sparse-vmemmap-set-compound-page-order-for-device-dax.patch
mm-sparse-vmemmap-switch-device-dax-to-shared-tail-vmemmap-pages.patch
mm-sparse-vmemmap-move-vmemmap-optimization-helpers-to-a-public-header.patch
powerpc-mm-switch-device-dax-to-shared-tail-vmemmap-pages.patch
mm-sparse-vmemmap-drop-the-extra-tail-page-from-device-dax-reservation.patch
mm-sparse-vmemmap-drop-unused-section_nr_vmemmap_pages-arguments.patch
documentation-mm-update-dax-vmemmap-deduplication-docs.patch
^ permalink raw reply [flat|nested] 2+ messages in thread
* + mm-sparse-vmemmap-allocate-shared-tail-page-array-dynamically.patch added to mm-unstable branch
@ 2026-09-30 19:01 Andrew Morton
0 siblings, 0 replies; 2+ messages in thread
From: Andrew Morton @ 2026-09-30 19:01 UTC (permalink / raw)
To: mm-commits, muchun.song, akpm
The patch titled
Subject: mm/sparse-vmemmap: allocate shared tail page array dynamically
has been added to the -mm mm-unstable branch. Its filename is
mm-sparse-vmemmap-allocate-shared-tail-page-array-dynamically.patch
This patch will shortly appear at
https://git.kernel.org/pub/scm/linux/kernel/git/akpm/25-new.git/tree/patches/mm-sparse-vmemmap-allocate-shared-tail-page-array-dynamically.patch
This patch will later appear in the mm-unstable branch at
git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm
Before you just go and hit "reply", please:
a) Consider who else should be cc'ed
b) Prefer to cc a suitable mailing list as well
c) Ideally: find the original patch on the mailing list and do a
reply-to-all to that, adding suitable additional cc's
*** Remember to use Documentation/process/submit-checklist.rst when testing your code ***
The -mm tree is included into linux-next via various
branches at git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm
and is updated there most days
------------------------------------------------------
From: Muchun Song <muchun.song@linux.dev>
Subject: mm/sparse-vmemmap: allocate shared tail page array dynamically
Date: Wed, 30 Sep 2026 22:06:17 +0800
Commit 622026e87c40 ("mm/hugetlb: remove fake head pages") added the
per-zone vmemmap_tails array. Its size depends on MAX_FOLIO_ORDER, which
had been moved to mmzone.h in preparation for the array.
PUD_ORDER is defined by linux/pgtable.h, which cannot be included from
mmzone.h without creating an include cycle. It was therefore open-coded
as PUD_SHIFT - PAGE_SHIFT.
This removed the dependency on PUD_ORDER, but not the underlying
dependency on architecture page-table definitions. PUD_SHIFT is generally
provided by architecture page-table headers, which are not guaranteed to
have been included when mmzone.h is parsed.
The dependency remained hidden because vmemmap_tails was originally
guarded by CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP. Under that condition,
MAX_FOLIO_ORDER resolves to either MAX_PAGE_ORDER or the fixed HugeTLB
limit, rather than the PUD_SHIFT-based definition.
Device DAX, however, does not require CONFIG_HUGETLB_PAGE. When it is
converted to use section-based vmemmap optimization, MAX_FOLIO_ORDER can
resolve to PUD_SHIFT - PAGE_SHIFT while it is being used to size
vmemmap_tails. This would make struct zone depend on architecture
page-table definitions being available when mmzone.h is parsed.
Replace the embedded array with a pointer and allocate it on first use.
This moves the order-count evaluation into sparse-vmemmap.c, after the
architecture page-table definitions are available, and removes the
dependency from mmzone.h.
Removing the compile-time array also removes the original reason for
keeping MAX_FOLIO_ORDER and the vmemmap optimization sizing definitions in
mmzone.h. Follow-up cleanups can place each definition in the header
owned by its respective subsystem.
Link: https://lore.kernel.org/20260930140627.57431-3-songmuchun@bytedance.com
Signed-off-by: Muchun Song <muchun.song@linux.dev>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Cc: David Hildenbrand <david@kernel.org>
Cc: Oscar Salvador <osalvador@suse.de>
Cc: Madhavan Srinivasan <maddy@linux.ibm.com>
Cc: Michael Ellerman <mpe@ellerman.id.au>
Cc: Jonathan Corbet <corbet@lwn.net>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Mike Rapoport <rppt@kernel.org>
Cc: Qi Zheng <qi.zheng@linux.dev>
Cc: Nicholas Piggin <npiggin@gmail.com>
Cc: Christophe Leroy <chleroy@kernel.org>
Cc: Ritesh Harjani (IBM) <ritesh.list@gmail.com>
Cc: Shrikanth Hegde <sshegde@linux.ibm.com>
Cc: Randy Dunlap <rdunlap@infradead.org>
Cc: Lance Yang <lance.yang@linux.dev>
---
include/linux/mmzone.h | 7 +------
mm/sparse-vmemmap.c | 39 +++++++++++++++++++++++++++++++++++----
2 files changed, 36 insertions(+), 10 deletions(-)
--- a/include/linux/mmzone.h~mm-sparse-vmemmap-allocate-shared-tail-page-array-dynamically
+++ a/include/linux/mmzone.h
@@ -113,11 +113,6 @@
(VMEMMAP_OPTIMIZATION_PAGES * PAGE_SIZE / sizeof(struct page))
#define VMEMMAP_OPTIMIZATION_MIN_ORDER (ilog2(VMEMMAP_OPTIMIZATION_NR_STRUCT_PAGES) + 1)
-#define __VMEMMAP_OPTIMIZATION_NR_ORDERS \
- (MAX_FOLIO_ORDER - VMEMMAP_OPTIMIZATION_MIN_ORDER + 1)
-#define VMEMMAP_OPTIMIZATION_NR_ORDERS \
- (__VMEMMAP_OPTIMIZATION_NR_ORDERS > 0 ? __VMEMMAP_OPTIMIZATION_NR_ORDERS : 0)
-
enum migratetype {
MIGRATE_UNMOVABLE,
MIGRATE_MOVABLE,
@@ -1156,7 +1151,7 @@ struct zone {
atomic_long_t vm_stat[NR_VM_ZONE_STAT_ITEMS];
atomic_long_t vm_numa_event[NR_VM_NUMA_EVENT_ITEMS];
#ifdef CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP
- struct page *vmemmap_tails[VMEMMAP_OPTIMIZATION_NR_ORDERS];
+ struct page **vmemmap_tails;
#endif
} ____cacheline_internodealigned_in_smp;
--- a/mm/sparse-vmemmap.c~mm-sparse-vmemmap-allocate-shared-tail-page-array-dynamically
+++ a/mm/sparse-vmemmap.c
@@ -170,16 +170,47 @@ static void * __meminit vmemmap_alloc_bl
}
#ifdef CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP
+#define VMEMMAP_OPTIMIZATION_NR_ORDERS (MAX_FOLIO_ORDER - VMEMMAP_OPTIMIZATION_MIN_ORDER + 1)
+
+static __ref struct page **vmemmap_tails(struct zone *zone)
+{
+ const size_t size = array_size(VMEMMAP_OPTIMIZATION_NR_ORDERS,
+ sizeof(*zone->vmemmap_tails));
+ struct page **pages = READ_ONCE(zone->vmemmap_tails);
+
+ if (pages)
+ return pages;
+
+ pages = slab_is_available() ? kzalloc_objs(*pages, VMEMMAP_OPTIMIZATION_NR_ORDERS) :
+ memblock_alloc(size, __alignof__(*pages));
+ if (!pages)
+ return NULL;
+
+ if (cmpxchg(&zone->vmemmap_tails, NULL, pages) != NULL) {
+ if (slab_is_available())
+ kfree(pages);
+ else
+ memblock_free(pages, size);
+ pages = READ_ONCE(zone->vmemmap_tails);
+ }
+
+ return pages;
+}
+
struct page __ref *vmemmap_shared_tail_page(unsigned int order, struct zone *zone)
{
const unsigned int idx = order - VMEMMAP_OPTIMIZATION_MIN_ORDER;
- struct page *page;
+ struct page *page, **pages;
void *addr;
if (WARN_ON_ONCE(idx >= VMEMMAP_OPTIMIZATION_NR_ORDERS))
return NULL;
- page = READ_ONCE(zone->vmemmap_tails[idx]);
+ pages = vmemmap_tails(zone);
+ if (!pages)
+ return NULL;
+
+ page = READ_ONCE(pages[idx]);
if (page)
return page;
@@ -194,12 +225,12 @@ struct page __ref *vmemmap_shared_tail_p
}
page = virt_to_page(addr);
- if (cmpxchg(&zone->vmemmap_tails[idx], NULL, page) != NULL) {
+ if (cmpxchg(&pages[idx], NULL, page) != NULL) {
if (slab_is_available())
__free_page(page);
else
memblock_free(addr, PAGE_SIZE);
- page = READ_ONCE(zone->vmemmap_tails[idx]);
+ page = READ_ONCE(pages[idx]);
}
return page;
_
Patches currently in -mm which might be from muchun.song@linux.dev are
mm-sparse-vmemmap-factor-out-shared-vmemmap-tail-page-allocation.patch
mm-sparse-vmemmap-allocate-shared-tail-page-array-dynamically.patch
mm-sparse-vmemmap-introduce-config_vmemmap_optimization.patch
mm-sparse-vmemmap-open-code-init_compound_tail.patch
mm-sparse-vmemmap-prepare-dax-vmemmap-population-for-compound-page-orders.patch
mm-sparse-vmemmap-set-compound-page-order-for-device-dax.patch
mm-sparse-vmemmap-switch-device-dax-to-shared-tail-vmemmap-pages.patch
mm-sparse-vmemmap-switch-device-dax-to-shared-tail-vmemmap-pages-fix.patch
mm-sparse-vmemmap-move-vmemmap-optimization-helpers-to-a-public-header.patch
powerpc-mm-switch-device-dax-to-shared-tail-vmemmap-pages.patch
powerpc-mm-switch-device-dax-to-shared-tail-vmemmap-pages-fix.patch
mm-sparse-vmemmap-drop-the-extra-tail-page-from-device-dax-reservation.patch
mm-sparse-vmemmap-drop-unused-section_nr_vmemmap_pages-arguments.patch
documentation-mm-update-dax-vmemmap-deduplication-docs.patch
mm-sparse-vmemmap-drop-vmemmap_populate_dax.patch
mm-sparse-vmemmap-support-device-dax-in-common-vmemmap-path.patch
mm-sparse-vmemmap-drop-device-dax-specific-population-path.patch
mm-sparse-vmemmap-remove-the-unused-ptpfn-argument.patch
mm-sparse-vmemmap-open-code-vmemmap_populate_address.patch
mm-mm_init-add-zone-mismatch-warning-during-page-init.patch
^ permalink raw reply [flat|nested] 2+ messages in thread
end of thread, other threads:[~2026-09-30 19:01 UTC | newest]
Thread overview: 2+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2026-09-30 19:01 + mm-sparse-vmemmap-allocate-shared-tail-page-array-dynamically.patch added to mm-unstable branch Andrew Morton
-- strict thread matches above, loose matches on Subject: below --
2026-09-27 5:51 Andrew Morton
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.