From: "David Hildenbrand (Arm)" <david@kernel.org>
To: "Lorenzo Stoakes" <ljs@kernel.org>,
"Liam R. Howlett" <liam@infradead.org>,
"Vlastimil Babka" <vbabka@kernel.org>,
"Borislav Petkov (AMD)" <bp@alien8.de>,
"Cenk Yılmaz" <cenk@deepfinresearch.com>,
"Zhang Peng" <zippermonkey@icloud.com>,
"Mike Rapoport" <rppt@kernel.org>,
"Suren Baghdasaryan" <surenb@google.com>,
"Michal Hocko" <mhocko@suse.com>,
"Andrew Morton" <akpm@linux-foundation.org>,
"Zi Yan" <ziy@nvidia.com>,
"Matthew Brost" <matthew.brost@intel.com>,
"Joshua Hahn" <joshua.hahnjy@gmail.com>,
"Byungchul Park" <byungchul@sk.com>,
"Gregory Price" <gourry@gourry.net>,
"Ying Huang" <ying.huang@linux.alibaba.com>,
"Alistair Popple" <apopple@nvidia.com>,
"Chris Li" <chrisl@kernel.org>,
"Kairui Song" <kasong@tencent.com>,
"Kemeng Shi" <shikemeng@huaweicloud.com>,
"Nhat Pham" <nphamcs@gmail.com>,
"Baoquan He" <baoquan.he@linux.dev>,
"Barry Song" <baohua@kernel.org>,
"Youngjun Park" <youngjun.park@lge.com>,
"Mel Gorman" <mgorman@suse.de>,
"Sasha Levin" <sasha.levin@oracle.com>,
"Rik van Riel" <riel@redhat.com>,
"Ingo Molnar" <mingo@kernel.org>, "Peter Xu" <peterx@redhat.com>,
"Hugh Dickins" <hughd@google.com>,
"Axel Rasmussen" <axelrasmussen@google.com>,
"Johannes Weiner" <hannes@cmpxchg.org>,
"Michal Hocko" <mhocko@kernel.org>,
"Qi Zheng" <qi.zheng@linux.dev>,
"Shakeel Butt" <shakeel.butt@linux.dev>,
"Yuanchu Xie" <yuanchu@google.com>, "Wei Xu" <weixugc@google.com>,
"Baolin Wang" <baolin.wang@linux.alibaba.com>,
"Andrea Arcangeli" <aarcange@redhat.com>,
"Thomas Gleixner" <tglx@kernel.org>
Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org,
stable@vger.kernel.org,
"David Hildenbrand (Arm)" <david@kernel.org>
Subject: [PATCH 1/4] mm: flush stale TLB entries before installing a CoW'ed folio
Date: Wed, 18 Nov 2026 10:20:36 +0100 [thread overview]
Message-ID: <20261118-file_folio_flush-v1-1-e9e739ed14e0@kernel.org> (raw)
In-Reply-To: <20261118-file_folio_flush-v1-0-e9e739ed14e0@kernel.org>
Zapping PTEs in zap_pte_range() defers the TLB flush until after dropping
the PTL for clean page-cache folios.
A concurrent write fault in a MAP_PRIVATE mapping can then CoW the
page-cache folio and install a new anon folio without a prior TLB
flush:
CPU0 CPU1
LOAD(ADDR)
-> #PF
installs R/O PTE -> A
fuse_open()
-> invalidate_inode_pages2()
-> zap_pte_range()
clears R/O PTE
// delayed flush
// Other CPUs (incl CPU0) still cache ADDR -> A
STORE(ADDR)
-> #PF
do_cow_fault()
copies A to B, installs R/W PTE -> B
// no TLB flush: PTE was none
CPUs with stale TLB entries will still read from the old folio while writes
go to the new folio, until the TLB is flushed.
We really must process pending TLB flushes if we are not faulting the
same folio back in.
Fix it by flushing the TLB before installing an anon (CoW'ed) folio in
set_pte_range() if a TLB flush is pending. Zapping itself stays as fast
as before.
Other faults are not affected: unmap_mapping_*() users only remove
folios from the page-cache after their own TLB flush, so until then, a
refault maps the same folio again. The exception is a concurrent
invalidation that removes the already-unmapped folio from the
page-cache before the zapper flushed the TLB; that is an unlikely corner
case that we'll not handle for now.
Note that we don't care about MADV_DONTNEED followed by a new read
fault, as MADV_DONTNEED only guarantees its semantics once the operation
completed. CoW faults will flush the TLB either way.
For example, imagine that MADV_DONTNEED zaps a page-cache folio and
doesn't flush the TLB (but still holds a reference). If the page-cache
replaces the folio, a new read fault might map the new folio without
flushing the TLB. That's okay, as MADV_DONTNEED did not complete yet.
Same applies to MADV_DONTNEED of CoW'ed folios when re-faulting the
page-cache folio.
Memory reclaim, which removes folios from the page-cache before performing
its own batched TLB flush, will be handled separately.
Fixes: 1da177e4c3f4 ("Linux-2.6.12-rc2")
Reported-by: Cenk Yılmaz <cenk@deepfinresearch.com>
Debugged-by: Cenk Yılmaz <cenk@deepfinresearch.com>
Cc: stable@vger.kernel.org
Signed-off-by: David Hildenbrand (Arm) <david@kernel.org>
---
mm/memory.c | 6 ++++++
1 file changed, 6 insertions(+)
diff --git a/mm/memory.c b/mm/memory.c
index d2f48d78b0d7b..e1d9759e6fde1 100644
--- a/mm/memory.c
+++ b/mm/memory.c
@@ -5888,6 +5888,12 @@ void set_pte_range(struct vm_fault *vmf, struct folio *folio,
/* copy-on-write page */
if (write && !(vma->vm_flags & VM_SHARED)) {
VM_BUG_ON_FOLIO(nr != 1, folio);
+ /*
+ * Concurrent zapping might have dropped the PTL but not flushed
+ * the TLB yet. Flush before installing a new folio.
+ */
+ if (mm_tlb_flush_pending(vma->vm_mm))
+ flush_tlb_page(vma, addr);
folio_add_new_anon_rmap(folio, vma, addr, RMAP_EXCLUSIVE);
folio_add_lru_vma(folio, vma);
} else {
--
2.56.1
next prev parent reply other threads:[~2026-11-18 9:21 UTC|newest]
Thread overview: 5+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-11-18 9:20 [PATCH 0/4] mm: fixes for stale TLB entries due to delayed/batched TLB flushes David Hildenbrand (Arm)
2026-11-18 9:20 ` David Hildenbrand (Arm) [this message]
2026-11-18 9:20 ` [PATCH 2/4] mm/vmscan: flush deferred writable TLB entries before unlocking the folio David Hildenbrand (Arm)
2026-11-18 9:20 ` [PATCH 3/4] RFC: mm/rmap: don't defer TLB flushes when unmapping lazyfree folios David Hildenbrand (Arm)
2026-11-18 9:20 ` [PATCH 4/4] RFC: mm/vmscan: unlock unmapped folios only after the deferred TLB flush David Hildenbrand (Arm)
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20261118-file_folio_flush-v1-1-e9e739ed14e0@kernel.org \
--to=david@kernel.org \
--cc=aarcange@redhat.com \
--cc=akpm@linux-foundation.org \
--cc=apopple@nvidia.com \
--cc=axelrasmussen@google.com \
--cc=baohua@kernel.org \
--cc=baolin.wang@linux.alibaba.com \
--cc=baoquan.he@linux.dev \
--cc=bp@alien8.de \
--cc=byungchul@sk.com \
--cc=cenk@deepfinresearch.com \
--cc=chrisl@kernel.org \
--cc=gourry@gourry.net \
--cc=hannes@cmpxchg.org \
--cc=hughd@google.com \
--cc=joshua.hahnjy@gmail.com \
--cc=kasong@tencent.com \
--cc=liam@infradead.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-mm@kvack.org \
--cc=ljs@kernel.org \
--cc=matthew.brost@intel.com \
--cc=mgorman@suse.de \
--cc=mhocko@kernel.org \
--cc=mhocko@suse.com \
--cc=mingo@kernel.org \
--cc=nphamcs@gmail.com \
--cc=peterx@redhat.com \
--cc=qi.zheng@linux.dev \
--cc=riel@redhat.com \
--cc=rppt@kernel.org \
--cc=sasha.levin@oracle.com \
--cc=shakeel.butt@linux.dev \
--cc=shikemeng@huaweicloud.com \
--cc=stable@vger.kernel.org \
--cc=surenb@google.com \
--cc=tglx@kernel.org \
--cc=vbabka@kernel.org \
--cc=weixugc@google.com \
--cc=ying.huang@linux.alibaba.com \
--cc=youngjun.park@lge.com \
--cc=yuanchu@google.com \
--cc=zippermonkey@icloud.com \
--cc=ziy@nvidia.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox