Re: [RFC PATCH 19/57] mm/collapse: install a PMD leaf as the terminal layer

[Kiryl Shutsemau <kirill@shutemov.name>]

On Tue, Aug 25, 2026 at 08:23:54PM +0800, Lance Yang wrote:
> >The detached one is not: GUP-fast and RCU pte walks that read the old PMD
> >may still be inside it, and on broadcast-TLBI architectures the flush
> >expels nobody.  Quiescing it would need an IPI, which has nowhere to go
> >here -- outside the pmd lock it opens the pmd_none() window this design
> >does not have, inside it is a broadcast under a spinlock.  So the
> >detached table goes to pte_free_defer(), which holds the free until those
> >walkers finish.  One transient table page per PMD collapse is the cost.
> 
> Well, git history spells out why pmdp_get_lockless_sync() is needed here.

One more good catch, thanks!

> Could we keep pmdp_get_lockless_sync() right after pmdp_collapse_flush(),
> before map_anon_folio_pmd_nopf() (and while the locks are still held)?

Yes, I will put pmdp_get_lockless_sync() there.

Here's what I got to my tree:

      /*
       * Nothing fallible sits past here.  No anon_vma_lock_write either: rmap
       * walks on the sources are unreachable -- refcounts frozen, folio locks
       * held from freeze to putback -- non-rmap pte walkers see migration
       * entries, pmd-level observers see the old table or the leaf and never an
       * intermediate, and fork, mremap and munmap take a write lock where the
       * round holds a read lock.
       *
       * The flush inside pmdp_collapse_flush() is the round's second over this
       * range: the freeze displaced every leaf here and flushed before dropping
       * the ptl, and the verify above proved nothing has been mapped since.
       * What it covers is the paging-structure caches -- a CPU may still hold
       * the pmd-to-table link, for a table that is about to be freed -- which
       * is why the helper shoots down a pte range rather than a pmd.
       *
       * If pmd_t is too wide to load in one access, a lockless walker reads
       * it half at a time.  Such a walker holds interrupts off, so an
       * interrupt between two present values is what keeps it from assembling
       * halves of both; the flush above does not always send one.
       * pmdp_get_lockless_sync() does, and is an empty inline wherever the
       * entry loads atomically.
       */
      old_pmd = pmdp_collapse_flush(vma, cand->addr, pmd);
      pmdp_get_lockless_sync();
      old_table = pmd_pgtable(old_pmd);

      /*
       * The smp_wmb() in __folio_mark_uptodate() orders the copied data before
       * the install below publishes it.
       */
      __folio_mark_uptodate(cand->new_folio);

      /*
       * Deposit a freshly allocated table, not the one just detached: a
       * deposited table has to be quiescent, because whoever withdraws it frees
       * it immediately (zap_huge_pmd()) with nothing to hold a lockless walker
       * off first.  A table that has never been reachable is quiescent by
       * construction, which is why collapse_alloc() secured one.
       *
       * The detached table is not quiescent.  GUP-fast and RCU pte walks that
       * read the old PMD before pmdp_collapse_flush() may still be inside it,
       * and on broadcast-TLBI arches that flush expels nobody.  Quiescing it
       * would take an IPI in a pmd_none window, which this design does not
       * have.  So the table goes to pte_free_defer(), which holds the free
       * until those walkers finish, as retract_page_tables() does.  One
       * transient table page per PMD collapse is what that costs.
       */
      pgtable_trans_huge_deposit(mm, pmd, cand->deposit);
      map_anon_folio_pmd_nopf(cand->new_folio, pmd, vma, cand->addr);

Any objections?

-- 
  Kiryl Shutsemau / Kirill A. Shutemov