Re: [LSF/MM/BPF TOPIC] 64k (or 16k) base page size on x86

[Nadia Chambers <nadia.yvette.chambers@ik.me>]

Am Fr, Feb 20, 2026 um 11:24:37 +0100, David Hildenbrand (Arm) schrieb:
> Right, see the proposal from Dev on the list.
> From user-space POV, the pagesize would be 64K for these emulated processes.
> That is, VMAs must be suitable aligned etc.
> One key thing I think is that you could run such emulated-64k process (that
> actually support it!) with 4k processes on the same machine, like Arm is
> considering.
> You would have no weird "vma crosses base pages" handling, which is just
> rather nasty and makes my head hurt.

While it had to be written, it was a straightforward extension of
converting ->vm_pgoff to being in MMUPAGE_SIZE (Kiryl's PTE_SIZE) units,
recovering offsets into pages usw. Nothing seemed terribly hard to
understand about it. Really, the code as it stands rarely approaches
what it's doing in such a way that it broadly surveys the layout of vmas
and pages and be conscious of whether pages spanned vmas or multiple
vmas were representing maps of of fragments of pages, so I'm not
convinced there's anything particularly unique to be afraid of here.


Am Fr, Feb 20, 2026 um 11:24:37 +0100, David Hildenbrand (Arm) schrieb:
> Well, yes, like Willy says, there are already similar custom solutions for
> s390x and ppc.
> Pasha talked recently about the memory waste of 16k kernel stacks and how we
> would want to reduce that to 4k. In your proposal, it would be 64k, unless
> you somehow manage to allocate multiple kernel stacks from the same 64k
> page. My head hurts thinking about whether that could work, maybe it could
> (no idea about guard pages in there, though).

Never mind wasting memory; just getting it booting in 2003 needed
adjusting things so the THREAD_SIZE stayed constant and there were
things in the core kernel that got a divide by zero or some such when
THREAD_SIZE was too large. The effort for the fix in 2003 was just
getting adequate diagnostics, not the stack allocation code itself.
The mechanically assisted forward port did something that got it booting
that, upon critical examination, I don't like and uses a lot of space.
vmalloc and guard pages don't have anything special about them either,
apart from taking work to implement, which I've done before as part of
what appears to be a now non-extant patch kit, stack_paranoia.


Am Fr, Feb 20, 2026 um 11:24:37 +0100, David Hildenbrand (Arm) schrieb:
> Let's take a look at the history of page size usage on Arm (people can feel
> free to correct me):
> (1) Most distros were using 64k on Arm.
> (2) People realized that 64k was suboptimal many use cases (memory
>     waste for stacks, pagecache, etc) and started to switch to 4k. I
>     remember that mostly HPC-centric users sticked to 64k, but there was
>     also demand from others to be able to stay on 64k.
> (3) Arm improved performance on a 4k kernel by adding cont-pte support,
>     trying to get closer to 64k native performance.
> (4) Achieving 64k native performance is hard, which is why per-process
>     page sizes are being explored to get the best out of both worlds
>     (use 64k page size only where it really matters for performance).

There are ways to deal with internal fragmentation, like tail packing
for the pagecache, switching to slab for a lot of things, potentially
even pagetables, so they don't grow with PAGE_SIZE usw. Stacks might
take more work than the average kernel data structure, but I've
implemented changes to stack allocation before at points when the
vmalloc and size varying usw. usf. stack bits weren't in mainline.
Bitblitting latencies are also issues to try to keep bounded that I've
seen little discussion of here yet.

I suppose the bad news is that I didn't get the chance to actually work
on the internal fragmentation mitigation strategies I wrote out plans
for 23 years ago. They weren't that involved, though.


Am Fr, Feb 20, 2026 um 11:24:37 +0100, David Hildenbrand (Arm) schrieb:
> Arm clearly has the added benefit of actually benefiting from hardware
> support for 64k.
> IIUC, what you are proposing feels a bit like traveling back in time when it
> comes to the memory waste problem that Arm users encountered.
> Where do you see the big difference to 64k on Arm in your proposal? Would
> you currently also be running 64k Arm in production and the memory waste etc
> is acceptable?

My original 2003 intentions for the use of Hugh's ABI compatibility
technique were to make page clustering as a solution for 32-bit large
memory „forward-looking to 64-bit“, where at the time, 64-bit meant
IA64, by using it to provide a guarantee that small superpage
allocations would never fail due to external fragmentation, which design
goal came from above. While some colouring benefits usw. were
anticipated as byproducts, this idea about how to handle small
superpages was the primary way it was conceived of as addressing 64-bit
concerns, perhaps in part by dint of IA64-centrism, or potentially a
kind of inheritance of a requirement for an equivalent of the
very-differently-implemented mechanism for the same from HP-UX. It also
seemed rather safe to set the goal of „raising the floor“ so the very
smallest TLB entries wouldn't proliferate, and the smallest page sizes
consume a lesser fraction of available TLB entries.

Proposals for trying to use the technique as some kind of fully general
method for reaching for 1 GiB if not larger pages seem ill-conceived to
me. Those kinds of issues demand changes akin to transforming the
problem so it's in an entirely lesser complexity class, not adding a
small constant to the methods already in place attempting to deal with
large-scale external fragmentation. I liked Rik's partitioning memory
into arenas of pre-constructed 1 GiB or otherwise very large superpages
as a strategy for that, for instance.

I generally care enough about architectural coverage that my recent
mechanically assisted efforts maintained an 80+ -cell regression testing
matrix of 20+ architectures with PAGE_MMUSHIFT values of 0, 2, 4 and 6.
ARM notably represented an extra architecture variant via LPAE. The arch
code I cry about was not having got the chance to use the recent MIPS
PageGrain extension to provide a 1 KiB MMUPAGE_SIZE and a PAGE_MMUSHIFT
of 8 for demonstration purposes. If I had got that done, I might have
even emailed Babaoğlu, Joy, and possibly even McKusick about it, though
Karels is reputedly the most likely suspect for who outside of Babaoğlu
and Joy might have committed CLSIZE u.a. to 3BSD's VAX VM code in 1979.

I made at least earnest attempts to cover every architecture in the
kernel. There was one I couldn't find a toolchain for that just wired
all of the MMUPAGE macros to be identical to their PAGE analogues and
hoped at least built (arc?), but otherwise, I was trying to cover them
all, even if at the moment only in QEMU. I hope I can put a personal
many-architecture testing lab like I had in the '00s together for
myself again at some point, given the chance, but they'll have to be
small things like for embedded this time with how flats in Berlin are.


-- nyc