The Linux Kernel Mailing List
 help / color / mirror / Atom feed
From: "Lorenzo Stoakes (ARM)" <ljs@kernel.org>
To: "David Hildenbrand (Arm)" <david@kernel.org>
Cc: Will Deacon <will@kernel.org>,
	 "Aneesh Kumar K.V" <aneesh.kumar@kernel.org>,
	Andrew Morton <akpm@linux-foundation.org>,
	 Nick Piggin <npiggin@gmail.com>,
	Peter Zijlstra <peterz@infradead.org>,
	 Arnd Bergmann <arnd@arndb.de>,
	Johannes Weiner <hannes@cmpxchg.org>,
	 Hugh Dickins <hughd@google.com>, Zi Yan <ziy@nvidia.com>,
	 Baolin Wang <baolin.wang@linux.alibaba.com>,
	"Liam R. Howlett" <liam@infradead.org>,
	 Nico Pache <nico.pache@linux.dev>,
	Ryan Roberts <ryan.roberts@arm.com>, Dev Jain <dev.jain@arm.com>,
	 Barry Song <baohua@kernel.org>,
	Lance Yang <lance.yang@linux.dev>,
	 Usama Arif <usama.arif@linux.dev>,
	Kiryl Shutsemau <kas@kernel.org>,
	 Vlastimil Babka <vbabka@kernel.org>,
	Mike Rapoport <rppt@kernel.org>,
	 Suren Baghdasaryan <surenb@google.com>,
	Michal Hocko <mhocko@suse.com>, Jan Kara <jack@suse.cz>,
	 Ross Zwisler <zwisler@kernel.org>,
	linux-arch@vger.kernel.org, linux-mm@kvack.org,
	 linux-kernel@vger.kernel.org, stable@vger.kernel.org,
	Chris Mason <mason@kernel.org>
Subject: Re: [PATCH v2 2/2] mm/huge_memory: flush dirty file PMDs before removing rmap
Date: Mon, 16 Nov 2026 15:19:01 +0000	[thread overview]
Message-ID: <avse3HEkP2TwNlVz@gremlin> (raw)
In-Reply-To: <20261116-dirty_file_pmd-v2-2-571883053174@kernel.org>

On Mon, Nov 16, 2026 at 04:06:59PM +0100, David Hildenbrand (Arm) wrote:
> Unmapping a file PMD can lose concurrent stores when it races with
> writeback. zap_huge_pmd() removes rmap and drops the PMD lock before the
> gather flushes the TLB. Writeback can then clean the folio while another
> thread can still write through a stale writable TLB entry. Those stores
> do not fault and dirty the folio again. Filesystem DAX has the same
> problem with its dirty tags.
>
> Commit 5df397dec7c4 ("mm: delay page_remove_rmap() until after the TLB
> has been flushed") fixed the same race for PTEs by adding delayed rmap
> removal to the mmu_gather infrastructure. The PMD path still removes
> rmap before flushing.
>
> Flush the TLB under the PMD lock before removing rmap for dirty file
> PMDs. Skip the early flush at address-space teardown, when no users
> remain, and for mappings without writeback, including shmem and device
> DAX.
>
> Use tlb_can_delay_rmap() to share the PTE path's architecture checks.
> On s390, clearing the entry already handles TLB synchronization, and
> __tlb_remove_page_size() immediately drops the mapping reference, so
> rmap must be removed first. Make the predicate side-effect-free and set
> the gather's delayed_rmap flag when an entry is actually queued for
> delayed rmap removal.
>
> Queue the folio before the early flush, while the gather still has a
> valid range. Keep batching references and drain a full batch only after
> dropping the PMD lock. The early flush resets the gather's TLB state,
> so draining the batch does not flush that range again.
>
> Although writing to files through mmap is generally discouraged,
> unmapping such mappings may become slower: each dirty file PMD now
> requires a TLB flush under the PMD lock. Future work should properly
> extend the delayed-rmap machinery to PMDs and hold the PMD spinlock
> across multiple entries, so their TLB invalidations and rmap removals
> can be batched.
>
> Fixes: 4b4bb46d00b3 ("dax: clear dirty entry tags on cache flush")
> Cc: stable@vger.kernel.org
> Reported-by: Kiryl Shutsemau <kas@kernel.org>
> Closes: https://lore.kernel.org/linux-mm/aoWw_iKEDLjKL2jr@thinkstation/
> Reported-by: Chris Mason <mason@kernel.org>
> Link: https://git.kernel.org/linus/5df397dec7c4c08c23bd14f162f1228836faa4ce
> Suggested-by: Lance Yang <lance.yang@linux.dev>
> Suggested-by: Usama Arif <usama.arif@linux.dev>
> Reviewed-by: Lance Yang <lance.yang@linux.dev>
> Reviewed-by: Baolin Wang <baolin.wang@linux.alibaba.com>
> Signed-off-by: David Hildenbrand (Arm) <david@kernel.org>

LGTM so:

Reviewed-by: Lorenzo Stoakes (ARM) <ljs@kernel.org>

> ---
>  include/asm-generic/tlb.h | 10 +++-------
>  mm/huge_memory.c          | 39 +++++++++++++++++++++++++++++++--------
>  mm/memory.c               |  2 +-
>  mm/mmu_gather.c           |  2 ++
>  4 files changed, 37 insertions(+), 16 deletions(-)
>
> diff --git a/include/asm-generic/tlb.h b/include/asm-generic/tlb.h
> index cb087315b4e72..b318e3e9e66a8 100644
> --- a/include/asm-generic/tlb.h
> +++ b/include/asm-generic/tlb.h
> @@ -249,11 +249,7 @@ extern bool __tlb_remove_page_size(struct mmu_gather *tlb, struct page *page, in
>  bool __tlb_remove_folio_pages(struct mmu_gather *tlb, struct page *page,
>  		unsigned int nr_pages, bool delay_rmap);
>
> -/*
> - * This both sets 'delayed_rmap', and returns true. It would be an inline
> - * function, except we define it before the 'struct mmu_gather'.
> - */
> -#define tlb_delay_rmap(tlb) (((tlb)->delayed_rmap = 1), true)
> +#define tlb_can_delay_rmap() (true)
>  extern void tlb_flush_rmaps(struct mmu_gather *tlb, struct vm_area_struct *vma);
>
>  #endif
> @@ -263,8 +259,8 @@ extern void tlb_flush_rmaps(struct mmu_gather *tlb, struct vm_area_struct *vma);
>   * delay anything. That is used on S390, which flushes remote
>   * TLBs synchronously.
>   */
> -#ifndef tlb_delay_rmap
> -#define tlb_delay_rmap(tlb) (false)
> +#ifndef tlb_can_delay_rmap
> +#define tlb_can_delay_rmap() (false)
>  static inline void tlb_flush_rmaps(struct mmu_gather *tlb, struct vm_area_struct *vma) { }
>  #endif
>
> diff --git a/mm/huge_memory.c b/mm/huge_memory.c
> index 7791872f14af7..79d400ea9553d 100644
> --- a/mm/huge_memory.c
> +++ b/mm/huge_memory.c
> @@ -2508,10 +2508,12 @@ static inline void zap_deposited_table(struct mm_struct *mm, pmd_t *pmd)
>  	mm_dec_nr_ptes(mm);
>  }
>
> -static void zap_huge_pmd_folio(struct mm_struct *mm, struct vm_area_struct *vma,
> -		pmd_t pmdval, struct folio *folio, bool is_present)
> +static void zap_huge_pmd_folio(struct mmu_gather *tlb, struct vm_area_struct *vma,
> +		pmd_t pmdval, struct folio *folio, bool is_present, bool *force_flush)
>  {
>  	const bool is_device_private = pmd_is_device_private_entry(pmdval);
> +	struct mm_struct *mm = tlb->mm;
> +	bool delay_rmap = false;
>
>  	if (folio_test_anon(folio)) {
>  		add_mm_counter(mm, MM_ANONPAGES, -HPAGE_PMD_NR);
> @@ -2519,23 +2521,42 @@ static void zap_huge_pmd_folio(struct mm_struct *mm, struct vm_area_struct *vma,
>  		add_mm_counter(mm, mm_counter_file(folio),
>  			       -HPAGE_PMD_NR);
>
> -		if (is_present && pmd_dirty(pmdval))
> +		if (is_present && pmd_dirty(pmdval)) {
>  			folio_mark_dirty(folio);
> +			/*
> +			 * Remove rmap after the TLB flush if stale TLB entries
> +			 * could still be used to write to a mapping that
> +			 * supports writeback.
> +			 */
> +			if (!tlb->fullmm && vma->vm_file &&
> +			    mapping_can_writeback(vma->vm_file->f_mapping))
> +				delay_rmap = tlb_can_delay_rmap();
> +		}
>  		if (is_present && pmd_young(pmdval) &&
>  		    likely(vma_has_recency(vma)))
>  			folio_mark_accessed(folio);
>  	}
>
> +	if (!is_present && !is_device_private)
> +		return;
> +
>  	/*
>  	 * Present and device private folios are rmappable. Remove the
>  	 * rmap after transferring young+dirty to the folio.
>  	 */
> -	if (is_present || is_device_private)
> +	if (!delay_rmap)
>  		folio_remove_rmap_pmd(folio, &folio->page, vma);
>
> -	/* Device private folios are pinned. */
> +	/* Device private folios are pinned and don't need a TLB flush. */
>  	if (is_device_private)
>  		folio_put(folio);
> +	else if (__tlb_remove_page_size(tlb, &folio->page, HPAGE_PMD_SIZE))
> +		*force_flush = true;
> +
> +	if (delay_rmap) {
> +		tlb_flush_mmu_tlbonly(tlb);
> +		folio_remove_rmap_pmd(folio, &folio->page, vma);
> +	}
>  }
>
>  static struct folio *normal_or_softleaf_folio_pmd(struct vm_area_struct *vma,
> @@ -2573,6 +2594,7 @@ bool zap_huge_pmd(struct mmu_gather *tlb, struct vm_area_struct *vma,
>  	struct mm_struct *mm = tlb->mm;
>  	struct folio *folio = NULL;
>  	bool is_present = false;
> +	bool force_flush = false;
>  	bool has_deposit;
>  	spinlock_t *ptl;
>  	pmd_t orig_pmd;
> @@ -2597,13 +2619,14 @@ bool zap_huge_pmd(struct mmu_gather *tlb, struct vm_area_struct *vma,
>  	folio = normal_or_softleaf_folio_pmd(vma, addr, orig_pmd, is_present);
>  	has_deposit = vma_has_deposited_pgtable(vma);
>  	if (folio)
> -		zap_huge_pmd_folio(mm, vma, orig_pmd, folio, is_present);
> +		zap_huge_pmd_folio(tlb, vma, orig_pmd, folio, is_present,
> +				   &force_flush);
>  	if (has_deposit)
>  		zap_deposited_table(mm, pmd);
>
>  	spin_unlock(ptl);
> -	if (is_present && folio)
> -		tlb_remove_page_size(tlb, &folio->page, HPAGE_PMD_SIZE);
> +	if (force_flush)
> +		tlb_flush_mmu(tlb);
>  	return true;
>  }
>
> diff --git a/mm/memory.c b/mm/memory.c
> index 7f019afc35ca0..cd954afb146f4 100644
> --- a/mm/memory.c
> +++ b/mm/memory.c
> @@ -1757,7 +1757,7 @@ static __always_inline void zap_present_folio_ptes(struct mmu_gather *tlb,
>  		ptent = get_and_clear_full_ptes(mm, addr, pte, nr, tlb->fullmm);
>  		if (pte_dirty(ptent)) {
>  			folio_mark_dirty(folio);
> -			if (tlb_delay_rmap(tlb)) {
> +			if (tlb_can_delay_rmap()) {
>  				delay_rmap = true;
>  				*force_flush = true;
>  			}
> diff --git a/mm/mmu_gather.c b/mm/mmu_gather.c
> index c022aeb7c84d0..fc547be33a919 100644
> --- a/mm/mmu_gather.c
> +++ b/mm/mmu_gather.c
> @@ -169,6 +169,8 @@ static bool __tlb_remove_folio_pages_size(struct mmu_gather *tlb,
>  	struct mmu_gather_batch *batch;
>
>  	VM_BUG_ON(!tlb->end);
> +	if (delay_rmap)
> +		tlb->delayed_rmap = 1;
>
>  #ifdef CONFIG_MMU_GATHER_PAGE_SIZE
>  	VM_WARN_ON(tlb->page_size != page_size);
>
> --
> 2.56.1
>

--
Cheers, Lorenzo

  reply	other threads:[~2026-11-16 15:19 UTC|newest]

Thread overview: 10+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-11-16 15:06 [PATCH v2 0/2] mm/huge_memory: flush dirty file PMDs before removing rmap David Hildenbrand (Arm)
2026-11-16 15:06 ` [PATCH v2 1/2] mm/mmu_gather: delay rmap removal on UP as well David Hildenbrand (Arm)
2026-11-16 15:16   ` Lorenzo Stoakes (ARM)
2026-11-16 15:18     ` David Hildenbrand (Arm)
2026-11-16 15:41   ` Zi Yan
2026-11-16 15:57   ` Kiryl Shutsemau (Meta)
2026-11-16 15:06 ` [PATCH v2 2/2] mm/huge_memory: flush dirty file PMDs before removing rmap David Hildenbrand (Arm)
2026-11-16 15:19   ` Lorenzo Stoakes (ARM) [this message]
2026-11-16 16:10   ` Kiryl Shutsemau (Meta)
2026-11-16 16:45   ` Zi Yan

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=avse3HEkP2TwNlVz@gremlin \
    --to=ljs@kernel.org \
    --cc=akpm@linux-foundation.org \
    --cc=aneesh.kumar@kernel.org \
    --cc=arnd@arndb.de \
    --cc=baohua@kernel.org \
    --cc=baolin.wang@linux.alibaba.com \
    --cc=david@kernel.org \
    --cc=dev.jain@arm.com \
    --cc=hannes@cmpxchg.org \
    --cc=hughd@google.com \
    --cc=jack@suse.cz \
    --cc=kas@kernel.org \
    --cc=lance.yang@linux.dev \
    --cc=liam@infradead.org \
    --cc=linux-arch@vger.kernel.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=linux-mm@kvack.org \
    --cc=mason@kernel.org \
    --cc=mhocko@suse.com \
    --cc=nico.pache@linux.dev \
    --cc=npiggin@gmail.com \
    --cc=peterz@infradead.org \
    --cc=rppt@kernel.org \
    --cc=ryan.roberts@arm.com \
    --cc=stable@vger.kernel.org \
    --cc=surenb@google.com \
    --cc=usama.arif@linux.dev \
    --cc=vbabka@kernel.org \
    --cc=will@kernel.org \
    --cc=ziy@nvidia.com \
    --cc=zwisler@kernel.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox