[PATCH 00/11] mm/hugetlb: Eliminate fake head pages from vmemmap optimization

Kiryl Shutsemau
2025-12-05T19:43:57Z

This series removes "fake head pages" from the HugeTLB vmemmap
optimization (HVO) by changing how tail pages encode their relationship
to the head page.

It simplifies compound_head() and page_ref_add_unless(). Both are in the
hot path.

Background
==========

HVO reduces memory overhead by freeing vmemmap pages for HugeTLB pages
and remapping the freed virtual addresses to a single physical page.
Previously, all tail page vmemmap entries were remapped to the first
vmemmap page (containing the head struct page), creating "fake heads" -
tail pages that appear to have PG_head set when accessed through the
deduplicated vmemmap.

This required special handling in compound_head() to detect and work
around fake heads, adding complexity and overhead to a very hot path.

New Approach
============

For architectures/configs where sizeof(struct page) is a power of 2 (the
common case), this series changes how position of the head page is encoded
in the tail pages.

Instead of storing a pointer to the head page, the ->compound_info
(renamed from ->compound_head) now stores a mask.

The mask can be applied to any tail page's virtual address to compute
the head page address. Critically, all tail pages of the same order now
have identical compound_info values, regardless of which compound page
they belong to.

This enables a key optimization: instead of remapping tail vmemmap
entries to the head page (creating fake heads), we remap them to a
shared, pre-initialized vmemmap_tail page per hstate. The head page
gets its own dedicated vmemmap page, eliminating fake heads entirely.

Benefits
========

1. Smaller generated code. On defconfig, I see ~15K reduction of text
in vmlinux:

add/remove: 6/33 grow/shrink: 54/262 up/down: 6130/-21922 (-15792)

2. Simplified compound_head(): No fake head detection needed. The
function is now branchless for power-of-2 struct page sizes.

3. Eliminated race condition: The old scheme required synchronize_rcu()
to coordinate between HVO remapping and speculative PFN walkers that
might write to fake heads. With the head page always in writable
memory, this synchronization is unnecessary.

4. Removed static key: hugetlb_optimize_vmemmap_key is no longer needed
since compound_head() no longer has HVO-specific branches.

5. Cleaner architecture: The vmemmap layout is now straightforward -
head page has its own vmemmap, tails share a read-only template.

I had hoped to see performance improvement, but my testing thus far has
shown either no change or only a slight improvement within the noise.

Series Organization
===================

Patches 1-3: Preparatory refactoring
- Change prep_compound_tail() interface to take order
- Rename compound_head field to compound_info
- Move set/clear_compound_head() near compound_head()

Patch 4: Core encoding change
- Implement mask-based encoding for power-of-2 struct page

Patches 5-6: HVO restructuring
- Refactor vmemmap_walk to support separate head/tail pages
- Introduce per-hstate vmemmap_tail, eliminate fake heads

Patches 7-9: Cleanup
- Remove fake head checks from compound_head(), PageTail(), etc.
- Remove VMEMMAP_SYNCHRONIZE_RCU and synchronize_rcu() calls
- Remove hugetlb_optimize_vmemmap_key static key

Patch 10: Optimization
- Implement branchless compound_head() for power-of-2 case

Patch 11: Documentation
- Update vmemmap_dedup.rst to reflect new architecture

Kiryl Shutsemau (11):
mm: Change the interface of prep_compound_tail()
mm: Rename the 'compound_head' field in the 'struct page' to
'compound_info'
mm: Move set/clear_compound_head() to compound_head()
mm: Rework compound_head() for power-of-2 sizeof(struct page)
mm/hugetlb: Refactor code around vmemmap_walk
mm/hugetlb: Remove fake head pages
mm: Drop fake head checks and fix a race condition
hugetlb: Remove VMEMMAP_SYNCHRONIZE_RCU
mm/hugetlb: Remove hugetlb_optimize_vmemmap_key static key
mm: Remove the branch from compound_head()
hugetlb: Update vmemmap_dedup.rst

.../admin-guide/kdump/vmcoreinfo.rst | 2 +-
Documentation/mm/vmemmap_dedup.rst | 62 ++---
include/linux/hugetlb.h | 3 +
include/linux/mm_types.h | 20 +-
include/linux/page-flags.h | 163 +++++-------
include/linux/page_ref.h | 8 +-
include/linux/types.h | 2 +-
kernel/vmcore_info.c | 2 +-
mm/hugetlb.c | 8 +-
mm/hugetlb_vmemmap.c | 245 ++++++++----------
mm/hugetlb_vmemmap.h | 4 +-
mm/internal.h | 11 +-
mm/mm_init.c | 2 +-
mm/page_alloc.c | 4 +-
mm/slab.h | 2 +-
mm/util.c | 15 +-
16 files changed, 242 insertions(+), 311 deletions(-)

--
2.51.2