Re: [RFC PATCH 00/57] mm/collapse: rebuild collapse on migration primitives

[Kiryl Shutsemau <kirill@shutemov.name>]

On Tue, Aug 18, 2026 at 03:55:55PM +0200, David Hildenbrand (Arm) wrote:
> > This replaces khugepaged's anonymous collapse with an engine that
> > can collapse sub-PMD ranges. It is built around migration entries and
> > frozen folios instead of heavy locking and isolation, aiming for better
> > scalability and less disruption to the workload being collapsed.
> 
> I recall us discussing something around using some PTE/PMD markers (e.g.,
> migration entries) in the past.
> 
> One thing that needed care is handling concurrent MADV_DONTNEED + faultin after
> dropping relevant locks.

Handled at install time.

I drop the PTL after the freeze to allow allocation and copy, but sample
the PTE values (see saved_ptes) at freeze time. If something changed
under us by the time we install the new page table entries, we give up
on that candidate and roll it back; the rest of the round still
installs. We allow harmless transitions: zero page to none.

> > Which is why hugepage_vma_revalidate() demands that the VMA span the
> > whole PMD even for an mTHP order -- "we'd need to lock all VMAs in the
> > PMD range to support this", as the comment there puts it.  A PMD-granular
> > operation is only safe when one VMA owns the PMD, and that is exactly the
> > restriction in the way.  The alignment is the symptom; the PMD is the
> > design.
> 
> I disagree with "A PMD-granular operation is only safe when one VMA owns the
> PMD". It's safe when all page table walkers can be stopped (see above).

Fair, the sentence is too strong.  mmap_write_lock plus a VMA write lock on
every VMA the PMD covers, plus their rmap locks, would make it safe.

But the mechanism still clears the whole PMD, flushes, IPIs and
repopulates it to collapse each 16-page window. That's very noisy to the
workload.

And I am not sure how to deal with rmap locking here.  Nothing in mm
holds two unrelated anon_vma rwsems: vma_prepare() takes one for both VMAs
it touches, because a merge requires them to share the anon_vma, and
anon_vma_clone() takes one because "all anon_vma's share the same root".
The lock ordering in mm/rmap.c has a single anon_vma->rwsem level, so a PMD
spanning unrelated mappings would need an ordering rule that does not exist
today.

> > Between the two, nothing can reach a source, so the copy runs with no
> > lock held at all -- and the address space is left alone while it does.
> 
> Right. Concurrent MADV_DONTNEED can zap migration PTEs and other faults even
> re-fault fresh anon folios. So that must be detected before replacing migration
> entries again I guess.

Yes, that is the install-time check above.

The copy itself is safe: a zap of a migration entry only clears the slot
and adjusts rss -- see zap_nonpresent_ptes(), which neither puts the
folio nor drops its rmap -- so a frozen, locked source cannot go away
under the copy.

The rollback does the rest: it leaves a foreign slot as found and drops
the rmap the freeze kept, which is the half of the teardown the zapper
could not do.

> 
> > 
> > What that removes from every collapse path:
> > 
> >   mmap_write_lock              -> mmap_read
> >   anon_vma_lock_write()        -> nothing: an rmap walk needs the folio
> >                                   locked, and the engine holds that lock
> >                                   from freeze to putback
> >   tlb_remove_table_sync_one()  -> nothing: one ranged flush per round
> >   LRU isolation                -> nothing: sources are inert in place
> > 
> 
> Not sure how you handle PMD collapse. I recall problems with migration entries
> on the PMD level for non-folio things (we discussed something along these lines
> also in the past).

There are no PMD-level migration entries here.

The freeze is always at PTE level, so the pmd keeps pointing at the
table until the last step, and the PMD leaf goes in as the terminal
layer: verify, pmdp_collapse_flush(), deposit a fresh table, set the
leaf, all in one section under the pmd lock with the pte ptl nested
inside.

A pmd-level walker sees the old table or the leaf and never pmd_none,
and faults stay held at pte level by the migration entries throughout,
which is what lets PMD collapse run under a VMA read lock like
everything else.

> 
> > Working in windows rather than whole PMDs takes care of the other root.
> > A sub-PMD window is collapsed under the page table lock, so a collapse
> > disturbs only the window it collapses, and each candidate is validated
> 
> I recall us discussing that holding the PT lock for a longer collapse operation
> (especially on 64k) is problematic. But I don't get all the details from your
> description here.

As I mentioned above, we drop the ptl after the freeze. And take it a
second time for the install.  Allocation and the copy run in between with
no lock held -- on 64K the copy at PMD order is 512M of it, which is why
it cannot sit under either lock.

> 
> > at its own order -- a window need only fit its own VMA.  A PMD-order
> > candidate still has to own the whole PMD, which is the old rule kept
> > where it is still needed.
> > 
> > Candidates are carried through the passes a batch at a time rather than
> > one window at a time, so a round pays for its flush and its lock
> > acquisitions once.
> 
> Now I am starting to feel that there are too many changes packed in a single
> series :)

Batching is not an extra here, it is the only way collapse works at mTHP
sizes.  What a round pays once -- lru_add_drain(), the mmu_notifier
invalidate window, the two ptl acquisitions and the TLB flush -- would
otherwise be paid per collapse.  A 2M PMD is 32 collapses at order-4, and
a gigabyte collapsed into 64K mTHP is 16384 of them: the serialization
would dominate, and the workload would feel every one.

> > Reading the series
> > ==================
> > 
> > 57 patches is a lot to land on a list.  They go in blocks:
> > 
> >   1-6    helpers and shared state: pte_folio(), pte_none_or_zero(),
> >          mm/collapse.h, and the policy that replaces asking whether
> >          khugepaged started a collapse
> >   7-8    the engine's shape: entry points, a call-tree comment naming
> >          every pass, and the scan filled in
> >   9-23   the collapse half, top down: the round frame, then each pass
> >          in turn, then selection and the retry store
> 
> I fail to parse this sentence.

Bad wording.  The engine is introduced in a top-down manner:

  - patch 7 adds the external interface and a comment naming every pass
  - 8 the scan
  - 9-11 how candidates become a round and which passes the round has
  - 12-21 fill in one pass per patch in the order they run
  - 22-23 add the cursor that picks the windows and the second chance
    for the refused ones.

I'll spell that out in the next iteration.

> > So I wrote "perf bench mem usemem" for this.  It touches a region while
> > khugepaged works on it and reports the workload's own latency
> > percentiles and throughput, against per-size counters that can see
> > sub-PMD folios.  The branch is above; it is unposted and not a
> > dependency.
> 
> Something more realistic might be running some workload in a VM whereby the VM
> is getting collapsed by khugepaged.

Fair, I'll run that: a guest touching its memory while the host collapses
the backing mapping, measured from inside the guest.

> 
> [...]
> 
> > There may be a way out -- a PMD migration entry over the table during
> > the window, so the CPU never caches a walk to shoot down -- but that
> > means teaching every pmd-level walker a new kind of entry, and I have
> > not tried it.
> 
> I think we discussed that in the past and it's absolutely nasty.

Let's leave this for now. I will give it a try when it gets its turn.

> 
> [...]
> 
> > Size
> > ====
> > 
> > mm/ grows by 1915 lines net: 4475 added against 2560 deleted.
> > 
> That's quite a lot for something that reads like a cleanup at first.

About half of that is comments. :)

-- 
  Kiryl Shutsemau / Kirill A. Shutemov