Re: [Chapter Two] THP shattering: the reverse of collapsing

[Yu Zhao <yuzhao@google.com>]

On Thu, Feb 29, 2024 at 4:55 PM Zi Yan <ziy@nvidia.com> wrote:
>
> On 29 Feb 2024, at 13:34, Yu Zhao wrote:
>
> > In contrast to split, shatter migrates occupied pages in a partially
> > mapped THP to a bunch of base folios. IOW, unlike split done in place,
> > shatter is the exact opposite of collapse.
> >
> > The advantage of shattering is that it keeps the original THP intact.
>
> Why keep the THP intact? To prevent the THP from fragmentation, since
> the shattered part will not be returned to buddy allocator for reuse?

There might be a confusion here: there is no "shattered part" -- the
entire THP becomes free after shattering (the occupied part is moved
to a bunch of 4KB pages).

> I agree with the idea of shattering, but keeping THP intact might
> give us trouble for 1GB THP case when PMD mapping is created after
> shattering. How to update mapcount for a PMD mapping in the middle of
> a 1GB folio? I used head[0], head[512], ... as the PMD mapping head
> page, but that is ugly. For mTHPs, there is no such problem since
> only PTE mappings are involved.

If we don't consider the copying cost during shattering, it can work
for 1GB THPs as it does for 2MB THPs.

> It might be better to just split the THP and move free pages to a
> donot-use free list until the rest are freed too

The main reason we do shattering is, using a crude analogy, a million
dollar in $10,000 bills (yes, they exist) is worth a lot more than
that in pennies. You can carry the former in your pocket but the
latter weighs at least 250 tons. So if we split, we lose money.

1GB THP is one of the important *end goals* for TAO. But I don't want
to go into details since we need to focus on the first few steps at
the current stage.

The problem with shattering for 1GB is the copying cost -- if we
shatter a 1GB THP half mapped/unmapped, we'd have to copy 512MB data,
which is unacceptable. 1GB THP requires something we call "THP
fungibility" (see the epilogue) -- we do split in place, but we also
"collapse" in place (called THP recovery, i.e., MADV_RECOVERY).

Shattering is for 2MB THPs only.


> if the zone enforces
> a minimal order that is larger than the free pages.
>
> > The cost of copying during the migration is not a side effect, but
> > rather by design, since splitting is considered a discouraged
> > behavior. In retail terms, the return of a purchase is charged with a
> > restocking fee and the original goods can be resold.
> >
> > THPs from ZONE_NOMERGE can only be shattered, since they cannot be
> > split or merged. THPs from ZONE_NOSPLIT can be shattered or split (the
> > latter requires [1]), if they are above the minimum order.
> >
> > [1] https://lore.kernel.org/20240226205534.1603748-1-zi.yan@sent.com/
> >
>
>
> --
> Best Regards,
> Yan, Zi