Linux-mm Archive on lore.kernel.org
 help / color / mirror / Atom feed
From: "David Hildenbrand (Arm)" <david@kernel.org>
To: "Lorenzo Stoakes" <ljs@kernel.org>,
	"Liam R. Howlett" <liam@infradead.org>,
	"Vlastimil Babka" <vbabka@kernel.org>,
	"Borislav Petkov (AMD)" <bp@alien8.de>,
	"Cenk Yılmaz" <cenk@deepfinresearch.com>,
	"Zhang Peng" <zippermonkey@icloud.com>,
	"Mike Rapoport" <rppt@kernel.org>,
	"Suren Baghdasaryan" <surenb@google.com>,
	"Michal Hocko" <mhocko@suse.com>,
	"Andrew Morton" <akpm@linux-foundation.org>,
	"Zi Yan" <ziy@nvidia.com>,
	"Matthew Brost" <matthew.brost@intel.com>,
	"Joshua Hahn" <joshua.hahnjy@gmail.com>,
	"Byungchul Park" <byungchul@sk.com>,
	"Gregory Price" <gourry@gourry.net>,
	"Ying Huang" <ying.huang@linux.alibaba.com>,
	"Alistair Popple" <apopple@nvidia.com>,
	"Chris Li" <chrisl@kernel.org>,
	"Kairui Song" <kasong@tencent.com>,
	"Kemeng Shi" <shikemeng@huaweicloud.com>,
	"Nhat Pham" <nphamcs@gmail.com>,
	"Baoquan He" <baoquan.he@linux.dev>,
	"Barry Song" <baohua@kernel.org>,
	"Youngjun Park" <youngjun.park@lge.com>,
	"Mel Gorman" <mgorman@suse.de>,
	"Sasha Levin" <sasha.levin@oracle.com>,
	"Rik van Riel" <riel@redhat.com>,
	"Ingo Molnar" <mingo@kernel.org>, "Peter Xu" <peterx@redhat.com>,
	"Hugh Dickins" <hughd@google.com>,
	"Axel Rasmussen" <axelrasmussen@google.com>,
	"Johannes Weiner" <hannes@cmpxchg.org>,
	"Michal Hocko" <mhocko@kernel.org>,
	"Qi Zheng" <qi.zheng@linux.dev>,
	"Shakeel Butt" <shakeel.butt@linux.dev>,
	"Yuanchu Xie" <yuanchu@google.com>, "Wei Xu" <weixugc@google.com>,
	"Baolin Wang" <baolin.wang@linux.alibaba.com>,
	"Andrea Arcangeli" <aarcange@redhat.com>,
	"Thomas Gleixner" <tglx@kernel.org>
Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org,
	 stable@vger.kernel.org,
	"David Hildenbrand (Arm)" <david@kernel.org>
Subject: [PATCH 0/4] mm: fixes for stale TLB entries due to delayed/batched TLB flushes
Date: Wed, 18 Nov 2026 10:20:35 +0100	[thread overview]
Message-ID: <20261118-file_folio_flush-v1-0-e9e739ed14e0@kernel.org> (raw)

David, what keeps you up at night? Well, the last 3 nights, TLB flushes.

We have two mechanisms that can delay/batch TLB flushes and cause
problems:

 (A) mmu_gather zap: we can delay TLB flushes until after dropping the
     PT lock, after having zapped PTEs and adjusted the mapcounts.

 (B) TTU_BATCH_FLUSH when used by reclaim: we can defer (batch) TLB
     flushes, after having unmapped folios and adjusted the mapcounts,
     and after having discarded lazyfree folios, removed folios from the
     page-cache/swap-cache or just dropped the folio lock.

Code that installs new folios (and doesn't just refault the old ones)
must be aware of that, otherwise we can end up with stale TLB entries
while new folios are already mapped into page tables.

So in general, whenever we install a new folio into the page tables,
we must make sure that no such stale TLB entries exist.

Even worse, whenever we replace a folio in the page-cache that might differ
content-wise from the old folio, or we end up modifying a new folio through
write(), we must make sure that no such stale TLB entries exist.

There are some exceptions (e.g., only once MADV_DONTNEED finished is when
a new read from the same address gives any guarantees), so the primary
focus is on folios that got unmapped transparently by the system, without
involving MADV_DONTNEED.

This series handles the following scenarios:

Patch #1: Zapping: flush the TLB before installing a CoW'ed folio.
          Handles CoW of a folio that is still in the page-cache.

Patch #2: Reclaim: flush deferred writable TLB entries before unlocking
          the folio. Handles writes through stale TLB entries getting
          lost when reclaim keeps a dirty folio and somebody else
          starts writeback.

Patch #3: Reclaim: don't defer TLB flushes when unmapping lazyfree
          folios. Handles mapping a new anon folio after reclaim
          discarded a lazyfree folio (do_anonymous_page(),
          migrate_vma_insert_page(), UFFDIO_COPY, UFFDIO_MOVE).

Patch #4: Reclaim: unlock unmapped folios only after the deferred TLB
          flush. Handles refaulting a new page-cache folio, CoW of a
          folio that is still in the page-cache, and swapping in a new
          folio (do_swap_page(), unuse_pte(), UFFDIO_CONTINUE). Also
          handles invalidation replacing a folio that reclaim kept, and
          write() modifying a new page-cache folio.

Patches #1 and #2 are small fixes that should be merged as soon as
possible. Patch #4 makes the flush added by patch #2 unnecessary by
keeping folios locked until the deferred TLB flush, and removes it
again.

Patches #3 and #4 are RFC, because #4 is rather intrusive and #3 comes
with a performance impact (see below). But there isn't really a good
alternative to fix these things for good that I see.

In theory, UFFDIO_COPY installing a new anon folio into a MAP_PRIVATE
shmem mapping is affected by concurrent zapping just like CoW faults in
patch #1. But the required setups (e.g., userfaultfd on a MAP_PRIVATE
mapping of a tmpfs file that backs a loop device) are so unusual that we
will leave it alone for now. Same applies to the hugetlb counterpart.

Remaining Problem
-----------------

There is one remaining known problem that is not handled by this
series:

Zapping (e.g., unmap_mapping_range()) removes the rmap of clean
page-cache folios before performing the deferred TLB flush. If the
mapcount drops to 0, a concurrent invalidation (e.g.,
invalidate_inode_pages2_range()) will skip the unmapped folio and can
remove it from the page-cache before the pending TLB flush.

A page fault (except a CoW fault, see patch #1) can then map a new folio,
or write() can modify it, while other CPUs still read the old folio through
stale TLB entries.

I think that this is low priority: it requires two concurrent invalidations
of the same file range while another process has the file mapped, and CPUs
can only read stale data until the zapper flushes the TLB; writes are never
lost.

It could maybe be fixed in the invalidation code? unmap_mapping_*() holds
the i_mmap_rwsem in read mode until it flushed the TLB, so
folio_unmap_invalidate() could briefly take it in write mode iff the folio
has unexpected references. But dropbehind (RWF_DONTCACHE) invalidates
folios on writeback completion while holding an additional reference, where
we must not sleep on the i_mmap_rwsem, so that would need some care.

Something for the future :)

Testing
-------

I threw a bunch of selftests at it and had AI generate some stress
tests, in particular for the benchmarks below. I also ran stress-ng to
force reclaim.

I managed to reproduce the issues in patch #3 and #4, as described in
the patch descriptions.

Stale reads out of 1024000 checked pages in 2000 iterations:

                               base    series
  lazyfree,  MGLRU            16242         0
  lazyfree,  classic LRU      16274         0
  swapcache, MGLRU            24076         0
  swapcache, classic LRU      23028         0

Performance Impact
------------------

I ran some benchmarks in an x86-64 VM with limited CPUs, RAM and zram
enabled. Running inside a VM might make TLB flushes more expensive, so
the results below might be a bit exaggerated. AI results below.

Classic LRU, 4 threads randomly touching a working set twice the size of
their memcg for 30 seconds. Pages touched per second:

                 base     series
  anon-rw        681k      1055k
  anon-ro       1013k      1027k
  file-ro        417k       419k
  file-rw        173k       183k
  lazyfree      1512k       333k

Swapping out dirty anon folios ("anon-rw") is about 55% faster, because
patch #4 now also batches TLB flushes of stale writable TLB entries.
That is what Zhang Peng's series [1] tried to achieve, and his work also
inspired how patch #4 locks folios again, but the used mechanism is
different.

Reclaiming clean anon folios ("anon-ro"), clean page-cache folios
("file-ro") and dirty page-cache folios ("file-rw") is not noticeably
affected.

Reclaiming lazyfree folios ("lazyfree") is about 4.5x slower, because
we stop batching TLB flushes when discarding them entirely, which seems
to be the right thing to do for now. We can look into optimizing this
separately later.

[1] https://lore.kernel.org/linux-mm/20261031-batch-tlb-flush-v6-0-190539839eec@tencent.com/

---
David Hildenbrand (Arm) (4):
      mm: flush stale TLB entries before installing a CoW'ed folio
      mm/vmscan: flush deferred writable TLB entries before unlocking the folio
      RFC: mm/rmap: don't defer TLB flushes when unmapping lazyfree folios
      RFC: mm/vmscan: unlock unmapped folios only after the deferred TLB flush

 mm/internal.h |   5 ++
 mm/memory.c   |   6 +++
 mm/rmap.c     |  20 +++++++-
 mm/shmem.c    |   2 +-
 mm/swap.h     |   5 +-
 mm/swapfile.c |   7 ++-
 mm/vmscan.c   | 155 ++++++++++++++++++++++++++++++++++++++++++++++++++++++----
 7 files changed, 184 insertions(+), 16 deletions(-)
---
base-commit: 0c524518413094607a6178c53a65815762bc9c45
change-id: 20261114-file_folio_flush-7d1749fab2d8

Best regards,
--  
David Hildenbrand (Arm) <david@kernel.org>



             reply	other threads:[~2026-11-18  9:21 UTC|newest]

Thread overview: 5+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-11-18  9:20 David Hildenbrand (Arm) [this message]
2026-11-18  9:20 ` [PATCH 1/4] mm: flush stale TLB entries before installing a CoW'ed folio David Hildenbrand (Arm)
2026-11-18  9:20 ` [PATCH 2/4] mm/vmscan: flush deferred writable TLB entries before unlocking the folio David Hildenbrand (Arm)
2026-11-18  9:20 ` [PATCH 3/4] RFC: mm/rmap: don't defer TLB flushes when unmapping lazyfree folios David Hildenbrand (Arm)
2026-11-18  9:20 ` [PATCH 4/4] RFC: mm/vmscan: unlock unmapped folios only after the deferred TLB flush David Hildenbrand (Arm)

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20261118-file_folio_flush-v1-0-e9e739ed14e0@kernel.org \
    --to=david@kernel.org \
    --cc=aarcange@redhat.com \
    --cc=akpm@linux-foundation.org \
    --cc=apopple@nvidia.com \
    --cc=axelrasmussen@google.com \
    --cc=baohua@kernel.org \
    --cc=baolin.wang@linux.alibaba.com \
    --cc=baoquan.he@linux.dev \
    --cc=bp@alien8.de \
    --cc=byungchul@sk.com \
    --cc=cenk@deepfinresearch.com \
    --cc=chrisl@kernel.org \
    --cc=gourry@gourry.net \
    --cc=hannes@cmpxchg.org \
    --cc=hughd@google.com \
    --cc=joshua.hahnjy@gmail.com \
    --cc=kasong@tencent.com \
    --cc=liam@infradead.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=linux-mm@kvack.org \
    --cc=ljs@kernel.org \
    --cc=matthew.brost@intel.com \
    --cc=mgorman@suse.de \
    --cc=mhocko@kernel.org \
    --cc=mhocko@suse.com \
    --cc=mingo@kernel.org \
    --cc=nphamcs@gmail.com \
    --cc=peterx@redhat.com \
    --cc=qi.zheng@linux.dev \
    --cc=riel@redhat.com \
    --cc=rppt@kernel.org \
    --cc=sasha.levin@oracle.com \
    --cc=shakeel.butt@linux.dev \
    --cc=shikemeng@huaweicloud.com \
    --cc=stable@vger.kernel.org \
    --cc=surenb@google.com \
    --cc=tglx@kernel.org \
    --cc=vbabka@kernel.org \
    --cc=weixugc@google.com \
    --cc=ying.huang@linux.alibaba.com \
    --cc=youngjun.park@lge.com \
    --cc=yuanchu@google.com \
    --cc=zippermonkey@icloud.com \
    --cc=ziy@nvidia.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox