Re: [Chapter One] THP zones: the use cases of policy zones

[Matthew Wilcox <willy@infradead.org>]

On Tue, Mar 05, 2024 at 10:51:20PM -0500, Yu Zhao wrote:
> On Thu, Feb 29, 2024 at 3:28 PM Matthew Wilcox <willy@infradead.org> wrote:
> > On Thu, Feb 29, 2024 at 11:34:33AM -0700, Yu Zhao wrote:
> > > Compared with the hugeTLB pool approach, THP zones tap into core MM
> > > features including:
> > > 1. THP allocations can fall back to the lower zones, which can have
> > >    higher latency but still succeed.
> > > 2. THPs can be either shattered (see Chapter Two) if partially
> > >    unmapped or reclaimed if becoming cold.
> > > 3. THP orders can be much smaller than the PMD/PUD orders, e.g., 64KB
> > >    contiguous PTEs on arm64 [1], which are more suitable for client
> > >    workloads.
> >
> > Can this mechanism be used to fully replace the hugetlb pool approach?
> > That would be a major selling point.  It kind of feels like it should,
> > but I am insufficiently expert to be certain.
> 
> This depends on the return value from htlb_alloc_mask(): if it's
> GFP_HIGHUSER_MOVABLE, then yes (i.e., 2MB hugeTLB folios on x86).
> Hypothetically, if users can have THPs as reliable as hugeTLB can
> offer, wouldn't most users want to go with the former since it's more
> flexible? E.g., core MM features like split (shattering) and reclaim
> in addition to HVO.

Right; the real question is what can we do to unify hugetlbfs and THPs.
The reservation ability is one feature that hugetlbfs has over THP and
removing that advantage gets us one step closer.

> > I'll read over the patches sometime soon.  There's a lot to go through.
> > Something I didn't see in the cover letter or commit messages was any
> > discussion of page->flags and how many bits we use for ZONE (particularly
> > on 32-bit).  Perhaps I'll discover the answer to that as I read.
> 
> There may be corner cases because of how different architectures use
> page->flags, but in general, this shouldn't be a big problem because
> we can have 6 zones (at most) before this series, and after this
> series, we can have 8 (at most). IOW, we need 3 bits regardless, in
> order to all existing zones.

On a 32-bit system, we'll typically only have four; DMA, NORMAL, HIGHMEM
and MOVABLE.  DMA32 will be skipped since it would match NORMAL, and
DEVICE is just not supported on 32-bit.