[PATCH v3 0/4] sparc64: D-cache alias flushing fixes

[PATCH v3 0/4] sparc64: D-cache alias flushing fixes

[Imre Kaloz]

These fix D-cache alias flushing on sparc64 CPUs with a virtually
indexed L1 D-cache (Spitfire and Cheetah).

1/4 makes xcall_deliver() bail out before it touches the trap block
when there is no other CPU to call, so an early cross call, before
init_IRQ() sets up the mondo blocks, can't end up writing to physical
address 0.

2/4 flushes only the page a PTE maps instead of its whole folio, which
removes about a million redundant page flushes from a colour-changing
mremap() of a large folio (7.3 s on a Sun Fire V240).

3/4 implements flush_cache_vmap() and flush_cache_vunmap(), which have
been no-ops. Stale vmalloc lines corrupt BPF programs and module data
on UltraSPARC III.

4/4 passes the number of PTEs to move_pte(), as Andreas suggested. It
touches mm/mremap.c and include/linux/pgtable.h, so it needs an ack
from the mm folks.

1-3 don't depend on 4 and can go in on their own. 2 and 3 are for
stable (3 needs 2), 1 and 4 aren't. The series is on top of the sparc
for-next branch.

Changes in v3:
- new 1/4 and 4/4, following Andreas' review
- 2/4: reworded the message and the move_pte() comment, marked the
  UP #endif
- 3/4: reworded the message, dropped the Suggested-by (I have no
  email address for nsafran1217, the Link credits them), noted the
  stable dependency on 2/4
- picked up Stian's and Adrian's Tested-by tags

v2: https://lore.kernel.org/r/20260928132102.1707-1-kaloz@kernel.org

Imre Kaloz (4):
  sparc64: skip the trap block in xcall_deliver() when there is no
    target
  sparc64: flush only the aliased page, not its whole folio
  sparc64: flush vmalloc ranges from the D-cache on map and unmap
  mm: pass the number of PTEs to move_pte()

 arch/sparc/include/asm/cacheflush_64.h |  9 ++--
 arch/sparc/include/asm/pgtable_64.h    | 14 ++++--
 arch/sparc/kernel/smp_64.c             | 51 +++++++++++++++-----
 arch/sparc/mm/init_64.c                | 65 ++++++++++++++++++++++++++
 arch/sparc/mm/tlb.c                    |  2 +-
 arch/sparc/mm/ultra.S                  | 25 ++++++++++
 include/linux/pgtable.h                |  2 +-
 mm/mremap.c                            |  2 +-
 8 files changed, 148 insertions(+), 22 deletions(-)


base-commit: 2bc58762a3d74e42de129d717ae3ba10ecf0ab96
-- 
2.47.3

[PATCH v3 1/4] sparc64: skip the trap block in xcall_deliver() when there is no target

[Imre Kaloz]

xcall_deliver() writes the mondo data and the target CPU list through
the trap block's physical addresses before it knows whether any CPU is
targeted. These addresses stay zero until init_IRQ() sets them up, so a
cross call made earlier, while only the boot CPU is online, would write
starting at physical address 0.

Return before touching the trap block when no other online CPU is in
the mask.

Signed-off-by: Imre Kaloz <kaloz@kernel.org>
---
 arch/sparc/kernel/smp_64.c | 9 +++++++++
 1 file changed, 9 insertions(+)

diff --git a/arch/sparc/kernel/smp_64.c b/arch/sparc/kernel/smp_64.c
index 371460e34484..4240c129c5fe 100644
--- a/arch/sparc/kernel/smp_64.c
+++ b/arch/sparc/kernel/smp_64.c
@@ -801,6 +801,15 @@ static void xcall_deliver(u64 data0, u64 data1, u64 data2, const cpumask_t *mask
 	local_irq_save(flags);
 
 	this_cpu = smp_processor_id();
+
+	/* The mondo block and cpu list are not set up until init_IRQ(),
+	 * so do not touch them unless there is a cpu to send to.
+	 */
+	if (cpumask_any_and_but(mask, cpu_online_mask, this_cpu) >= nr_cpu_ids) {
+		local_irq_restore(flags);
+		return;
+	}
+
 	tb = &trap_block[this_cpu];
 
 	mondo = __va(tb->cpu_mondo_block_pa);
-- 
2.47.3

[PATCH v3 2/4] sparc64: flush only the aliased page, not its whole folio

[Imre Kaloz]

Since the conversion to folios, move_pte() and tlb_batch_add() flushed
the D-cache of every page in the folio a PTE maps. Both are called once
per PTE, so a range backed by a large folio had each of its pages
flushed once per PTE of that folio. On SMP, each page flush on Spitfire
and Cheetah is a cross-call to all other online CPUs.

Flush only the page the PTE maps instead, as before the conversion.
Both callers see the PTEs of a range one at a time, so every page of
the range still gets its own flush. flush_dcache_folio_all() has no
other users and is turned back into flush_dcache_page_all().

On a Sun Fire V240, an mremap() that moves the 1024 PTEs of a large
folio to the other colour took 7.3 s, about a million redundant page
flushes.

Fixes: 1a10a44dfc1d ("sparc64: implement the new page table range API")
Cc: stable@vger.kernel.org
Tested-by: Stian Halseth <stian@itx.no> # Sun Fire V240, SPARC T4-1
Tested-by: John Paul Adrian Glaubitz <glaubitz@physik.fu-berlin.de>
Signed-off-by: Imre Kaloz <kaloz@kernel.org>
---
 arch/sparc/include/asm/cacheflush_64.h |  3 +--
 arch/sparc/include/asm/pgtable_64.h    |  8 ++++++--
 arch/sparc/kernel/smp_64.c             | 24 +++++++++++++-----------
 arch/sparc/mm/init_64.c                | 22 ++++++++++++++++++++++
 arch/sparc/mm/tlb.c                    |  2 +-
 5 files changed, 43 insertions(+), 16 deletions(-)

diff --git a/arch/sparc/include/asm/cacheflush_64.h b/arch/sparc/include/asm/cacheflush_64.h
index 06092572c045..02c969417e7b 100644
--- a/arch/sparc/include/asm/cacheflush_64.h
+++ b/arch/sparc/include/asm/cacheflush_64.h
@@ -38,11 +38,10 @@ void __flush_dcache_page(void *addr, int flush_icache);
 void flush_dcache_folio_impl(struct folio *folio);
 #ifdef CONFIG_SMP
 void smp_flush_dcache_folio_impl(struct folio *folio, int cpu);
-void flush_dcache_folio_all(struct mm_struct *mm, struct folio *folio);
 #else
 #define smp_flush_dcache_folio_impl(folio, cpu) flush_dcache_folio_impl(folio)
-#define flush_dcache_folio_all(mm, folio) flush_dcache_folio_impl(folio)
 #endif
+void flush_dcache_page_all(struct mm_struct *mm, struct page *page);
 
 void __flush_dcache_range(unsigned long start, unsigned long end);
 #define ARCH_IMPLEMENTS_FLUSH_DCACHE_PAGE 1
diff --git a/arch/sparc/include/asm/pgtable_64.h b/arch/sparc/include/asm/pgtable_64.h
index 0837ebbc5dce..1430489b9ac5 100644
--- a/arch/sparc/include/asm/pgtable_64.h
+++ b/arch/sparc/include/asm/pgtable_64.h
@@ -946,6 +946,10 @@ static inline void set_ptes(struct mm_struct *mm, unsigned long addr,
 	set_pte_at((mm), (addr), (ptep), __pte(0UL))
 
 #ifdef DCACHE_ALIASING_POSSIBLE
+/* This flushes only the page the PTE maps, so it relies on move_ptes()
+ * calling it for one PTE at a time, which the default pte_batch_hint()
+ * guarantees.
+ */
 #define __HAVE_ARCH_MOVE_PTE
 #define move_pte(pte, old_addr, new_addr)				\
 ({									\
@@ -955,8 +959,8 @@ static inline void set_ptes(struct mm_struct *mm, unsigned long addr,
 									\
 		if (pfn_valid(this_pfn) &&				\
 		    (((old_addr) ^ (new_addr)) & (1 << 13)))		\
-			flush_dcache_folio_all(current->mm,		\
-				page_folio(pfn_to_page(this_pfn)));	\
+			flush_dcache_page_all(current->mm,		\
+					      pfn_to_page(this_pfn));	\
 	}								\
 	newpte;								\
 })
diff --git a/arch/sparc/kernel/smp_64.c b/arch/sparc/kernel/smp_64.c
index 4240c129c5fe..6681f5263865 100644
--- a/arch/sparc/kernel/smp_64.c
+++ b/arch/sparc/kernel/smp_64.c
@@ -991,8 +991,9 @@ void smp_flush_dcache_folio_impl(struct folio *folio, int cpu)
 	put_cpu();
 }
 
-void flush_dcache_folio_all(struct mm_struct *mm, struct folio *folio)
+void flush_dcache_page_all(struct mm_struct *mm, struct page *page)
 {
+	struct folio *folio = page_folio(page);
 	void *pg_addr;
 	u64 data0;
 
@@ -1005,7 +1006,7 @@ void flush_dcache_folio_all(struct mm_struct *mm, struct folio *folio)
 	atomic_inc(&dcpage_flushes);
 #endif
 	data0 = 0;
-	pg_addr = folio_address(folio);
+	pg_addr = page_address(page);
 	if (tlb_type == spitfire) {
 		data0 = ((u64)&xcall_flush_dcache_page_spitfire);
 		if (folio_flush_mapping(folio) != NULL)
@@ -1016,18 +1017,19 @@ void flush_dcache_folio_all(struct mm_struct *mm, struct folio *folio)
 #endif
 	}
 	if (data0) {
-		unsigned int i, nr = folio_nr_pages(folio);
-
-		for (i = 0; i < nr; i++) {
-			xcall_deliver(data0, __pa(pg_addr),
-				      (u64) pg_addr, cpu_online_mask);
+		xcall_deliver(data0, __pa(pg_addr),
+			      (u64)pg_addr, cpu_online_mask);
 #ifdef CONFIG_DEBUG_DCFLUSH
-			atomic_inc(&dcpage_flushes_xcall);
+		atomic_inc(&dcpage_flushes_xcall);
 #endif
-			pg_addr += PAGE_SIZE;
-		}
 	}
-	__local_flush_dcache_folio(folio);
+#ifdef DCACHE_ALIASING_POSSIBLE
+	__flush_dcache_page(pg_addr,
+			    tlb_type == spitfire && folio_flush_mapping(folio));
+#else
+	if (tlb_type == spitfire && folio_flush_mapping(folio))
+		__flush_icache_page(__pa(pg_addr));
+#endif
 
 	preempt_enable();
 }
diff --git a/arch/sparc/mm/init_64.c b/arch/sparc/mm/init_64.c
index 103db4683b16..8d01aab43cc3 100644
--- a/arch/sparc/mm/init_64.c
+++ b/arch/sparc/mm/init_64.c
@@ -214,6 +214,28 @@ inline void flush_dcache_folio_impl(struct folio *folio)
 #endif
 }
 
+#ifndef CONFIG_SMP
+void flush_dcache_page_all(struct mm_struct *mm, struct page *page)
+{
+	struct folio *folio = page_folio(page);
+
+	if (tlb_type == hypervisor)
+		return;
+
+#ifdef CONFIG_DEBUG_DCFLUSH
+	atomic_inc(&dcpage_flushes);
+#endif
+
+#ifdef DCACHE_ALIASING_POSSIBLE
+	__flush_dcache_page(page_address(page),
+			    tlb_type == spitfire && folio_flush_mapping(folio));
+#else
+	if (tlb_type == spitfire && folio_flush_mapping(folio))
+		__flush_icache_page(page_to_phys(page));
+#endif
+}
+#endif /* !CONFIG_SMP */
+
 #define PG_dcache_dirty		PG_arch_1
 #define PG_dcache_cpu_shift	32UL
 #define PG_dcache_cpu_mask	\
diff --git a/arch/sparc/mm/tlb.c b/arch/sparc/mm/tlb.c
index 6d9dd5eb1328..1221814ca0e1 100644
--- a/arch/sparc/mm/tlb.c
+++ b/arch/sparc/mm/tlb.c
@@ -144,7 +144,7 @@ void tlb_batch_add(struct mm_struct *mm, unsigned long vaddr,
 
 		paddr = (unsigned long) page_address(page);
 		if ((paddr ^ vaddr) & (1 << 13))
-			flush_dcache_folio_all(mm, folio);
+			flush_dcache_page_all(mm, page);
 	}
 
 no_cache_flush:
-- 
2.47.3

[PATCH v3 3/4] sparc64: flush vmalloc ranges from the D-cache on map and unmap

[Imre Kaloz]

The L1 D-cache of Spitfire and Cheetah is indexed with VA bit 13, but
flush_cache_vmap() and flush_cache_vunmap() are no-ops. Lines loaded
through a vmalloc address survive the unmap, and once the page is
written through the other colour, e.g. zeroed through the linear
mapping, a later mapping of the old colour reads stale data. Anonymous
user pages arrive the same way, as tlb_batch_add() does not flush them.
On UltraSPARC III this corrupts BPF programs and module data.

Flush each page on every CPU on map and unmap, or the whole D-cache in
one cross call once the range exceeds its size.

flush_cache_vmap() also runs on the vmemmap, from sparse_init() before
init_IRQ() sets up the cross-call mondo blocks. The vmemmap has no
aliases to flush, so skip addresses outside vmalloc and module space.
Now that xcall_deliver() leaves the trap block alone when there is no
target CPU, this check only saves pointless flushes. Stable kernels
lack that change and need the check, since without it
flush_dcache_all()'s cross call from sparse_init() would write starting
at physical address 0.

Link: https://github.com/sparclinux/issues/issues/29#issuecomment-5041121375
Closes: https://github.com/sparclinux/issues/issues/29
Cc: stable@vger.kernel.org # requires "sparc64: flush only the aliased page, not its whole folio"
Tested-by: Stian Halseth <stian@itx.no> # Sun Fire V240, SPARC T4-1
Tested-by: John Paul Adrian Glaubitz <glaubitz@physik.fu-berlin.de>
Signed-off-by: Imre Kaloz <kaloz@kernel.org>
---
 arch/sparc/include/asm/cacheflush_64.h |  6 ++--
 arch/sparc/kernel/smp_64.c             | 18 +++++++++++
 arch/sparc/mm/init_64.c                | 43 ++++++++++++++++++++++++++
 arch/sparc/mm/ultra.S                  | 25 +++++++++++++++
 4 files changed, 90 insertions(+), 2 deletions(-)

diff --git a/arch/sparc/include/asm/cacheflush_64.h b/arch/sparc/include/asm/cacheflush_64.h
index 02c969417e7b..54a2d37008e6 100644
--- a/arch/sparc/include/asm/cacheflush_64.h
+++ b/arch/sparc/include/asm/cacheflush_64.h
@@ -42,6 +42,8 @@ void smp_flush_dcache_folio_impl(struct folio *folio, int cpu);
 #define smp_flush_dcache_folio_impl(folio, cpu) flush_dcache_folio_impl(folio)
 #endif
 void flush_dcache_page_all(struct mm_struct *mm, struct page *page);
+void __flush_dcache_all(unsigned long size, unsigned long line_size);
+void flush_dcache_all(void);
 
 void __flush_dcache_range(unsigned long start, unsigned long end);
 #define ARCH_IMPLEMENTS_FLUSH_DCACHE_PAGE 1
@@ -73,9 +75,9 @@ void flush_ptrace_access(struct vm_area_struct *, struct page *,
 #define flush_dcache_mmap_lock(mapping)		do { } while (0)
 #define flush_dcache_mmap_unlock(mapping)	do { } while (0)
 
-#define flush_cache_vmap(start, end)		do { } while (0)
+void flush_cache_vmap(unsigned long start, unsigned long end);
 #define flush_cache_vmap_early(start, end)	do { } while (0)
-#define flush_cache_vunmap(start, end)		do { } while (0)
+#define flush_cache_vunmap(start, end)		flush_cache_vmap(start, end)
 
 #endif /* !__ASSEMBLER__ */
 
diff --git a/arch/sparc/kernel/smp_64.c b/arch/sparc/kernel/smp_64.c
index 6681f5263865..ed6900b5d562 100644
--- a/arch/sparc/kernel/smp_64.c
+++ b/arch/sparc/kernel/smp_64.c
@@ -924,6 +924,7 @@ extern unsigned long xcall_kgdb_capture;
 
 #ifdef DCACHE_ALIASING_POSSIBLE
 extern unsigned long xcall_flush_dcache_page_cheetah;
+extern unsigned long xcall_flush_dcache_all;
 #endif
 extern unsigned long xcall_flush_dcache_page_spitfire;
 
@@ -1034,6 +1035,23 @@ void flush_dcache_page_all(struct mm_struct *mm, struct page *page)
 	preempt_enable();
 }
 
+#ifdef DCACHE_ALIASING_POSSIBLE
+void flush_dcache_all(void)
+{
+	unsigned long size, line_size;
+
+	if (tlb_type == hypervisor)
+		return;
+
+	preempt_disable();
+	size = local_cpu_data().dcache_size;
+	line_size = local_cpu_data().dcache_line_size;
+	smp_cross_call(&xcall_flush_dcache_all, 0, size, line_size);
+	__flush_dcache_all(size, line_size);
+	preempt_enable();
+}
+#endif
+
 #ifdef CONFIG_KGDB
 void kgdb_roundup_cpus(void)
 {
diff --git a/arch/sparc/mm/init_64.c b/arch/sparc/mm/init_64.c
index 8d01aab43cc3..d1f94b8d6312 100644
--- a/arch/sparc/mm/init_64.c
+++ b/arch/sparc/mm/init_64.c
@@ -27,6 +27,7 @@
 #include <linux/percpu.h>
 #include <linux/mmzone.h>
 #include <linux/gfp.h>
+#include <linux/vmalloc.h>
 
 #include <asm/head.h>
 #include <asm/page.h>
@@ -234,6 +235,17 @@ void flush_dcache_page_all(struct mm_struct *mm, struct page *page)
 		__flush_icache_page(page_to_phys(page));
 #endif
 }
+
+#ifdef DCACHE_ALIASING_POSSIBLE
+void flush_dcache_all(void)
+{
+	if (tlb_type == hypervisor)
+		return;
+
+	__flush_dcache_all(local_cpu_data().dcache_size,
+			   local_cpu_data().dcache_line_size);
+}
+#endif
 #endif /* !CONFIG_SMP */
 
 #define PG_dcache_dirty		PG_arch_1
@@ -3072,6 +3084,37 @@ arch_initcall(report_memory);
 #define do_flush_tlb_kernel_range	__flush_tlb_kernel_range
 #endif
 
+/*
+ * The L1 D-cache is indexed with VA bit 13, so lines loaded through a
+ * vmalloc address outlive the mapping.  Stores through a mapping of the
+ * other colour do not update them, and a later load through their colour
+ * returns stale data.  Flush at unmap so vmalloc's lines do not follow
+ * the page back to the allocator, and at map time since anonymous user
+ * pages are freed without a flush (see tlb_batch_add()).  Past the
+ * D-cache size, one whole-cache flush is cheaper than a cross call per
+ * page.
+ */
+void flush_cache_vmap(unsigned long start, unsigned long end)
+{
+#ifdef DCACHE_ALIASING_POSSIBLE
+	/* The vmemmap has no aliases and is mapped before cross calls work. */
+	if (tlb_type == hypervisor || !is_vmalloc_or_module_addr((void *)start))
+		return;
+
+	if (end - start > cpu_data(raw_smp_processor_id()).dcache_size) {
+		flush_dcache_all();
+		return;
+	}
+
+	for (; start < end; start += PAGE_SIZE) {
+		struct page *page = vmalloc_to_page((void *)start);
+
+		if (page)
+			flush_dcache_page_all(NULL, page);
+	}
+#endif
+}
+
 void flush_tlb_kernel_range(unsigned long start, unsigned long end)
 {
 	if (start < HI_OBP_ADDRESS && end > LOW_OBP_ADDRESS) {
diff --git a/arch/sparc/mm/ultra.S b/arch/sparc/mm/ultra.S
index 70e658d107e0..ff190670a235 100644
--- a/arch/sparc/mm/ultra.S
+++ b/arch/sparc/mm/ultra.S
@@ -242,6 +242,20 @@ __flush_dcache_page:	/* %o0=kaddr, %o1=flush_icache */
 	retl
 	 nop
 
+	/* Zeroing every D-cache tag invalidates the whole cache on
+	 * Spitfire and Cheetah alike.
+	 */
+	.align		32
+	.globl		__flush_dcache_all
+__flush_dcache_all:	/* %o0=D-cache size, %o1=D-cache line size */
+1:	subcc		%o0, %o1, %o0
+	stxa		%g0, [%o0] ASI_DCACHE_TAG
+	membar		#Sync
+	bne,pt		%icc, 1b
+	 nop
+	retl
+	 nop
+
 #endif /* DCACHE_ALIASING_POSSIBLE */
 
 	.previous
@@ -801,6 +815,17 @@ xcall_flush_dcache_page_cheetah: /* %g1 == physical page address */
 	 nop
 	retry
 	nop
+
+	.align		32
+	.globl		xcall_flush_dcache_all
+xcall_flush_dcache_all: /* %g1 == D-cache size, %g7 == D-cache line size */
+1:	subcc		%g1, %g7, %g1
+	stxa		%g0, [%g1] ASI_DCACHE_TAG
+	membar		#Sync
+	bne,pt		%icc, 1b
+	 nop
+	retry
+	nop
 #endif /* DCACHE_ALIASING_POSSIBLE */
 
 	.globl		xcall_flush_dcache_page_spitfire
-- 
2.47.3

[PATCH v3 4/4] mm: pass the number of PTEs to move_pte()

[Imre Kaloz]

Since commit f822a9a81a31 ("mm: optimize mremap() by PTE batching"),
move_ptes() moves a batch of PTEs mapping consecutive pages of a folio
at a time, but calls move_pte() once per batch with the first PTE
only. sparc64, the only architecture that overrides move_pte(), flushes
from the D-cache the page that PTE maps when the move changes its
colour, so the other pages of a batch would keep lines of the old
colour. This cannot happen today only because sparc64 has no
pte_batch_hint(), and mremap_folio_pte_batch() then never batches.

Pass nr_ptes to move_pte(), as get_and_clear_ptes() takes it. The
generic version ignores it, and sparc64 flushes every page of the
batch.

Signed-off-by: Imre Kaloz <kaloz@kernel.org>
---
 arch/sparc/include/asm/pgtable_64.h | 16 +++++++++-------
 include/linux/pgtable.h             |  2 +-
 mm/mremap.c                         |  2 +-
 3 files changed, 11 insertions(+), 9 deletions(-)

diff --git a/arch/sparc/include/asm/pgtable_64.h b/arch/sparc/include/asm/pgtable_64.h
index 1430489b9ac5..9ca787b3cbfe 100644
--- a/arch/sparc/include/asm/pgtable_64.h
+++ b/arch/sparc/include/asm/pgtable_64.h
@@ -946,21 +946,23 @@ static inline void set_ptes(struct mm_struct *mm, unsigned long addr,
 	set_pte_at((mm), (addr), (ptep), __pte(0UL))
 
 #ifdef DCACHE_ALIASING_POSSIBLE
-/* This flushes only the page the PTE maps, so it relies on move_ptes()
- * calling it for one PTE at a time, which the default pte_batch_hint()
- * guarantees.
+/* The nr PTEs map consecutive pages of one folio and all move by the
+ * same offset, so either every page of the batch aliases or none does.
  */
 #define __HAVE_ARCH_MOVE_PTE
-#define move_pte(pte, old_addr, new_addr)				\
+#define move_pte(pte, old_addr, new_addr, nr)				\
 ({									\
 	pte_t newpte = (pte);						\
 	if (tlb_type != hypervisor && pte_present(pte)) {		\
 		unsigned long this_pfn = pte_pfn(pte);			\
+		unsigned int i;						\
 									\
 		if (pfn_valid(this_pfn) &&				\
-		    (((old_addr) ^ (new_addr)) & (1 << 13)))		\
-			flush_dcache_page_all(current->mm,		\
-					      pfn_to_page(this_pfn));	\
+		    (((old_addr) ^ (new_addr)) & (1 << 13))) {		\
+			for (i = 0; i < (nr); i++)			\
+				flush_dcache_page_all(current->mm,	\
+					pfn_to_page(this_pfn + i));	\
+		}							\
 	}								\
 	newpte;								\
 })
diff --git a/include/linux/pgtable.h b/include/linux/pgtable.h
index 8c093c119e5a..b58c0e98994e 100644
--- a/include/linux/pgtable.h
+++ b/include/linux/pgtable.h
@@ -1471,7 +1471,7 @@ static inline void arch_swap_restore(swp_entry_t entry, struct folio *folio)
 #endif
 
 #ifndef __HAVE_ARCH_MOVE_PTE
-#define move_pte(pte, old_addr, new_addr)	(pte)
+#define move_pte(pte, old_addr, new_addr, nr)	(pte)
 #endif
 
 #ifndef pte_accessible
diff --git a/mm/mremap.c b/mm/mremap.c
index e8df5cdb0ac9..7fcae7f0082d 100644
--- a/mm/mremap.c
+++ b/mm/mremap.c
@@ -289,7 +289,7 @@ static int move_ptes(struct pagetable_move_control *pmc,
 			force_flush = true;
 		}
 		pte = get_and_clear_ptes(mm, old_addr, old_ptep, nr_ptes);
-		pte = move_pte(pte, old_addr, new_addr);
+		pte = move_pte(pte, old_addr, new_addr, nr_ptes);
 		pte = move_soft_dirty_pte(pte);
 
 		if (need_clear_uffd_wp && pte_is_uffd_wp_marker(pte))
-- 
2.47.3

Re: [PATCH v3 4/4] mm: pass the number of PTEs to move_pte()

[Pedro Falcato]

On Thu, Oct 22, 2026 at 01:33:40AM +0200, Imre Kaloz wrote:
> Since commit f822a9a81a31 ("mm: optimize mremap() by PTE batching"),
> move_ptes() moves a batch of PTEs mapping consecutive pages of a folio
> at a time, but calls move_pte() once per batch with the first PTE
> only. sparc64, the only architecture that overrides move_pte(), flushes
> from the D-cache the page that PTE maps when the move changes its
> colour, so the other pages of a batch would keep lines of the old
> colour. This cannot happen today only because sparc64 has no
> pte_batch_hint(), and mremap_folio_pte_batch() then never batches.
> 
> Pass nr_ptes to move_pte(), as get_and_clear_ptes() takes it. The
> generic version ignores it, and sparc64 flushes every page of the
> batch.
> 
> Signed-off-by: Imre Kaloz <kaloz@kernel.org>
> ---
>  arch/sparc/include/asm/pgtable_64.h | 16 +++++++++-------
>  include/linux/pgtable.h             |  2 +-
>  mm/mremap.c                         |  2 +-
>  3 files changed, 11 insertions(+), 9 deletions(-)
> 
> diff --git a/arch/sparc/include/asm/pgtable_64.h b/arch/sparc/include/asm/pgtable_64.h
> index 1430489b9ac5..9ca787b3cbfe 100644
> --- a/arch/sparc/include/asm/pgtable_64.h
> +++ b/arch/sparc/include/asm/pgtable_64.h
> @@ -946,21 +946,23 @@ static inline void set_ptes(struct mm_struct *mm, unsigned long addr,
>  	set_pte_at((mm), (addr), (ptep), __pte(0UL))
>  
>  #ifdef DCACHE_ALIASING_POSSIBLE
> -/* This flushes only the page the PTE maps, so it relies on move_ptes()
> - * calling it for one PTE at a time, which the default pte_batch_hint()
> - * guarantees.
> +/* The nr PTEs map consecutive pages of one folio and all move by the
> + * same offset, so either every page of the batch aliases or none does.
>   */
>  #define __HAVE_ARCH_MOVE_PTE
> -#define move_pte(pte, old_addr, new_addr)				\
> +#define move_pte(pte, old_addr, new_addr, nr)				\

The \ is now misaligned (at least according to my email client), perhaps you
can fix it up :)

The only nit I have is regarding the function's name.  move_pte with a nr is
no longer really move_pte, move_ptes() is the calling function's name... Yuck.

Also it seems a bit overengineered at this point, move_pte() returns a pte
that sparc never modifies.

>  ({									\
>  	pte_t newpte = (pte);						\
>  	if (tlb_type != hypervisor && pte_present(pte)) {		\
>  		unsigned long this_pfn = pte_pfn(pte);			\
> +		unsigned int i;						\
>  									\
>  		if (pfn_valid(this_pfn) &&				\
> -		    (((old_addr) ^ (new_addr)) & (1 << 13)))		\
> -			flush_dcache_page_all(current->mm,		\
> -					      pfn_to_page(this_pfn));	\
> +		    (((old_addr) ^ (new_addr)) & (1 << 13))) {		\
> +			for (i = 0; i < (nr); i++)			\
> +				flush_dcache_page_all(current->mm,	\
> +					pfn_to_page(this_pfn + i));	\

Is there no way to pass nr_ptes directly to flushing code?

Anyway,

Reviewed-by: Pedro Falcato <pfalcato@suse.de> 

-- 
Pedro

Re: [PATCH v3 4/4] mm: pass the number of PTEs to move_pte()

[David Hildenbrand (Arm)]

On 10/22/26 01:33, Imre Kaloz wrote:
> Since commit f822a9a81a31 ("mm: optimize mremap() by PTE batching"),
> move_ptes() moves a batch of PTEs mapping consecutive pages of a folio
> at a time, but calls move_pte() once per batch with the first PTE
> only. sparc64, the only architecture that overrides move_pte(), flushes
> from the D-cache the page that PTE maps when the move changes its
> colour, so the other pages of a batch would keep lines of the old
> colour. This cannot happen today only because sparc64 has no
> pte_batch_hint(), and mremap_folio_pte_batch() then never batches.
> 
> Pass nr_ptes to move_pte(), as get_and_clear_ptes() takes it. The
> generic version ignores it, and sparc64 flushes every page of the
> batch.
> 
> Signed-off-by: Imre Kaloz <kaloz@kernel.org>
> ---
>  arch/sparc/include/asm/pgtable_64.h | 16 +++++++++-------
>  include/linux/pgtable.h             |  2 +-
>  mm/mremap.c                         |  2 +-
>  3 files changed, 11 insertions(+), 9 deletions(-)
> 
> diff --git a/arch/sparc/include/asm/pgtable_64.h b/arch/sparc/include/asm/pgtable_64.h
> index 1430489b9ac5..9ca787b3cbfe 100644
> --- a/arch/sparc/include/asm/pgtable_64.h
> +++ b/arch/sparc/include/asm/pgtable_64.h
> @@ -946,21 +946,23 @@ static inline void set_ptes(struct mm_struct *mm, unsigned long addr,
>  	set_pte_at((mm), (addr), (ptep), __pte(0UL))
>  
>  #ifdef DCACHE_ALIASING_POSSIBLE
> -/* This flushes only the page the PTE maps, so it relies on move_ptes()
> - * calling it for one PTE at a time, which the default pte_batch_hint()
> - * guarantees.
> +/* The nr PTEs map consecutive pages of one folio and all move by the
> + * same offset, so either every page of the batch aliases or none does.
>   */
>  #define __HAVE_ARCH_MOVE_PTE
> -#define move_pte(pte, old_addr, new_addr)				\
> +#define move_pte(pte, old_addr, new_addr, nr)				\
>  ({									\
>  	pte_t newpte = (pte);						\
>  	if (tlb_type != hypervisor && pte_present(pte)) {		\
>  		unsigned long this_pfn = pte_pfn(pte);			\
> +		unsigned int i;						\
>  									\
>  		if (pfn_valid(this_pfn) &&				\
> -		    (((old_addr) ^ (new_addr)) & (1 << 13)))		\
> -			flush_dcache_page_all(current->mm,		\
> -					      pfn_to_page(this_pfn));	\
> +		    (((old_addr) ^ (new_addr)) & (1 << 13))) {		\
> +			for (i = 0; i < (nr); i++)			\
> +				flush_dcache_page_all(current->mm,	\
> +					pfn_to_page(this_pfn + i));	\
> +		}							\
>  	}								\
>  	newpte;								\
>  })
> diff --git a/include/linux/pgtable.h b/include/linux/pgtable.h
> index 8c093c119e5a..b58c0e98994e 100644
> --- a/include/linux/pgtable.h
> +++ b/include/linux/pgtable.h
> @@ -1471,7 +1471,7 @@ static inline void arch_swap_restore(swp_entry_t entry, struct folio *folio)
>  #endif
>  
>  #ifndef __HAVE_ARCH_MOVE_PTE
> -#define move_pte(pte, old_addr, new_addr)	(pte)
> +#define move_pte(pte, old_addr, new_addr, nr)	(pte)
>  #endif
>  
>  #ifndef pte_accessible
> diff --git a/mm/mremap.c b/mm/mremap.c
> index e8df5cdb0ac9..7fcae7f0082d 100644
> --- a/mm/mremap.c
> +++ b/mm/mremap.c
> @@ -289,7 +289,7 @@ static int move_ptes(struct pagetable_move_control *pmc,
>  			force_flush = true;
>  		}
>  		pte = get_and_clear_ptes(mm, old_addr, old_ptep, nr_ptes);
> -		pte = move_pte(pte, old_addr, new_addr);
> +		pte = move_pte(pte, old_addr, new_addr, nr_ptes);
>  		pte = move_soft_dirty_pte(pte);
>  
>  		if (need_clear_uffd_wp && pte_is_uffd_wp_marker(pte))


I hate move_pte() so much. And I hate sparc64 for requiring it. ;)

Is there some way we can avoid this
helper-that-shouldn't-exist-but-someone-hacked-it-in entirely?

If it makes sparc64 slightly slower #dontcare.

-- 
Cheers,

David

Re: [PATCH v3 4/4] mm: pass the number of PTEs to move_pte()

[Imre Kaloz]

Hi David,

On 10/23/26 22:53, David Hildenbrand (Arm) wrote:
> Is there some way we can avoid this
> helper-that-shouldn't-exist-but-someone-hacked-it-in entirely?
> If it makes sparc64 slightly slower #dontcare.

I tried three ways of doing that on an Ultra 45 (UltraSPARC IIIi, two
CPUs), all with move_pte() removed entirely:

 A) sparc64's flush_cache_range() flushes every present page in the
    range from the D-cache;
 B) sparc64's flush_cache_range() flushes the whole D-cache;
 C) A for ranges of up to 10 pages, B above that.

All three pass a test that moves pages between cache colors with
mremap() and checks for stale data. With no flush at all the same test
finds about 15 million stale words per run, so it does catch the
problem.

Medians of 20 runs over two boots per kernel, against this series'
move_pte() as the baseline:

                            move_pte()        A          B          C
  apt-cache gencaches           6.63 s     +0.2%      +0.3%      +0.1%
  fork+exec                                +0.7%      +3.1%      -0.6%
  mremap 8K, same color          14 us      20 us      43 us      20 us
  mremap 8K, other color         21 us      20 us      43 us      20 us
  mremap 1M, same color          30 us     524 us      60 us      59 us
  mremap 8M, same color         209 us    4152 us     229 us     227 us
  mremap 8M, other color       4176 us    4166 us     236 us     233 us
  mremap 8M THP, same color      14 us    3933 us      43 us      42 us

flush_cache_range() has more callers than mremap(), mostly in rmap,
so I also timed reclaim, migration and writeback (medians of 5):

                                  A          B          C
  MADV_PAGEOUT to swap          +0.4%     +12.9%      -1.3%
  reclaim under memory.max      +1.9%     +17.7%      +2.0%
  msync of shared file mappings -4.4%      -3.4%      -0.4%
  compaction, move_pages()      within 3% for all three

So A pays on large moves, B on reclaim, and C on neither: it costs
the same as A on small ranges and the same as B on large ones. I'd go
with C. It would only flush on Spitfire and Cheetah, which have the
aliasing D-cache, so sun4v and the SPARC64 VII in Magnus's M3000
series skip it.

If that works for you, I'd suggest dropping 4/4 and using C as its
replacement; it's below the scissors, on top of 1/4-3/4.

Pedro, that would also settle the naming and the nr question, since
move_pte() goes away.


Best,
Imre

-- >8 --
Subject: [PATCH] sparc64: replace move_pte() with a flush in flush_cache_range()

mremap() can move a page to an address of the other D-cache color,
leaving lines of the old color in the virtually indexed D-cache of
Spitfire and Cheetah.  move_pte() flushed the page for each PTE moved,
which is only correct while move_ptes() handles one PTE at a time.

Drop move_pte(), which no other architecture defines, and flush the
old range in flush_cache_range(), which mremap() calls before moving
the PTEs.  Flush each mapped page, or the whole D-cache for a range of
more than ten pages, where one whole-cache flush costs about as much
as ten page flushes.

Signed-off-by: Imre Kaloz <kaloz@kernel.org>
---
diff --git a/arch/sparc/include/asm/cacheflush_64.h b/arch/sparc/include/asm/cacheflush_64.h
index 54a2d37008e6..1a116b288816 100644
--- a/arch/sparc/include/asm/cacheflush_64.h
+++ b/arch/sparc/include/asm/cacheflush_64.h
@@ -21,8 +21,8 @@ void __flushw_user(void);
 #define flush_cache_mm(__mm) \
 	do { if ((__mm) == current->mm) flushw_user(); } while(0)
 #define flush_cache_dup_mm(mm) flush_cache_mm(mm)
-#define flush_cache_range(vma, start, end) \
-	flush_cache_mm((vma)->vm_mm)
+void flush_cache_range(struct vm_area_struct *vma, unsigned long start,
+		       unsigned long end);
 #define flush_cache_page(vma, page, pfn) \
 	flush_cache_mm((vma)->vm_mm)
 
diff --git a/arch/sparc/include/asm/pgtable_64.h b/arch/sparc/include/asm/pgtable_64.h
index 1430489b9ac5..2452d959aac1 100644
--- a/arch/sparc/include/asm/pgtable_64.h
+++ b/arch/sparc/include/asm/pgtable_64.h
@@ -945,27 +945,6 @@ static inline void set_ptes(struct mm_struct *mm, unsigned long addr,
 #define pte_clear(mm,addr,ptep)		\
 	set_pte_at((mm), (addr), (ptep), __pte(0UL))
 
-#ifdef DCACHE_ALIASING_POSSIBLE
-/* This flushes only the page the PTE maps, so it relies on move_ptes()
- * calling it for one PTE at a time, which the default pte_batch_hint()
- * guarantees.
- */
-#define __HAVE_ARCH_MOVE_PTE
-#define move_pte(pte, old_addr, new_addr)				\
-({									\
-	pte_t newpte = (pte);						\
-	if (tlb_type != hypervisor && pte_present(pte)) {		\
-		unsigned long this_pfn = pte_pfn(pte);			\
-									\
-		if (pfn_valid(this_pfn) &&				\
-		    (((old_addr) ^ (new_addr)) & (1 << 13)))		\
-			flush_dcache_page_all(current->mm,		\
-					      pfn_to_page(this_pfn));	\
-	}								\
-	newpte;								\
-})
-#endif
-
 extern pgd_t swapper_pg_dir[PTRS_PER_PGD];
 
 void paging_init(void);
diff --git a/arch/sparc/mm/init_64.c b/arch/sparc/mm/init_64.c
index d1f94b8d6312..466a4aa1be60 100644
--- a/arch/sparc/mm/init_64.c
+++ b/arch/sparc/mm/init_64.c
@@ -3115,6 +3115,95 @@ void flush_cache_vmap(unsigned long start, unsigned long end)
 #endif
 }
 
+static void flush_dcache_pfns(struct mm_struct *mm, unsigned long pfn,
+			      unsigned long nr)
+{
+	for (; nr; nr--, pfn++)
+		if (pfn_valid(pfn))
+			flush_dcache_page_all(mm, pfn_to_page(pfn));
+}
+
+/* A whole-cache flush costs about as much as this many page flushes. */
+#define FLUSH_CACHE_RANGE_ALL_PAGES	10
+
+/*
+ * mremap() moves the PTEs of [start, end) to addresses that may have the
+ * other color, and calls this on the old range first.  Flush every page
+ * mapped in the range from the D-cache, so no line of the old color
+ * outlives the move; a larger range is flushed whole without a walk.
+ * The walk takes no page table lock: mremap() holds the mmap_lock for
+ * writing, and other callers at worst miss a PTE that changes under it.
+ */
+void flush_cache_range(struct vm_area_struct *vma, unsigned long start,
+		       unsigned long end)
+{
+	struct mm_struct *mm = vma->vm_mm;
+	unsigned long addr, next;
+
+	/* flush_cache_mm() spills the user register windows. */
+	flush_cache_mm(mm);
+#ifdef DCACHE_ALIASING_POSSIBLE
+	/* The D-cache is virtually indexed on Spitfire and Cheetah. */
+	if (tlb_type != spitfire && tlb_type != cheetah &&
+	    tlb_type != cheetah_plus)
+		return;
+
+	if (((end - start) >> PAGE_SHIFT) > FLUSH_CACHE_RANGE_ALL_PAGES) {
+		flush_dcache_all();
+		return;
+	}
+
+	for (addr = start; addr < end; addr = next) {
+		pgd_t *pgd = pgd_offset(mm, addr);
+		p4d_t *p4d;
+		pud_t *pud, pudval;
+		pmd_t *pmd, pmdval;
+		pte_t *start_pte, *ptep, pte;
+
+		next = pgd_addr_end(addr, end);
+		if (pgd_none(pgdp_get(pgd)))
+			continue;
+		p4d = p4d_offset(pgd, addr);
+		next = p4d_addr_end(addr, end);
+		if (p4d_none(p4dp_get(p4d)))
+			continue;
+		pud = pud_offset(p4d, addr);
+		pudval = pudp_get(pud);
+		next = pud_addr_end(addr, end);
+		if (pud_none(pudval))
+			continue;
+		if (pud_leaf(pudval)) {
+			if (pte_present(__pte(pud_val(pudval))))
+				flush_dcache_pfns(mm, pud_pfn(pudval) +
+						  ((addr & ~PUD_MASK) >> PAGE_SHIFT),
+						  (next - addr) >> PAGE_SHIFT);
+			continue;
+		}
+		pmd = pmd_offset(pud, addr);
+		pmdval = pmdp_get_lockless(pmd);
+		next = pmd_addr_end(addr, end);
+		if (pmd_none(pmdval))
+			continue;
+		if (pmd_leaf(pmdval)) {
+			if (pte_present(__pte(pmd_val(pmdval))))
+				flush_dcache_pfns(mm, pmd_pfn(pmdval) +
+						  ((addr & ~PMD_MASK) >> PAGE_SHIFT),
+						  (next - addr) >> PAGE_SHIFT);
+			continue;
+		}
+		start_pte = pte_offset_map(pmd, addr);
+		if (!start_pte)
+			continue;
+		for (ptep = start_pte; addr < next; addr += PAGE_SIZE, ptep++) {
+			pte = ptep_get(ptep);
+			if (pte_present(pte))
+				flush_dcache_pfns(mm, pte_pfn(pte), 1);
+		}
+		pte_unmap(start_pte);
+	}
+#endif
+}
+
 void flush_tlb_kernel_range(unsigned long start, unsigned long end)
 {
 	if (start < HI_OBP_ADDRESS && end > LOW_OBP_ADDRESS) {
diff --git a/include/linux/pgtable.h b/include/linux/pgtable.h
index 8c093c119e5a..b95eea612261 100644
--- a/include/linux/pgtable.h
+++ b/include/linux/pgtable.h
@@ -1470,10 +1470,6 @@ static inline void arch_swap_restore(swp_entry_t entry, struct folio *folio)
 }
 #endif
 
-#ifndef __HAVE_ARCH_MOVE_PTE
-#define move_pte(pte, old_addr, new_addr)	(pte)
-#endif
-
 #ifndef pte_accessible
 # define pte_accessible(mm, pte)	((void)(pte), 1)
 #endif
diff --git a/mm/mremap.c b/mm/mremap.c
index e8df5cdb0ac9..2257e3010f41 100644
--- a/mm/mremap.c
+++ b/mm/mremap.c
@@ -289,7 +289,6 @@ static int move_ptes(struct pagetable_move_control *pmc,
 			force_flush = true;
 		}
 		pte = get_and_clear_ptes(mm, old_addr, old_ptep, nr_ptes);
-		pte = move_pte(pte, old_addr, new_addr);
 		pte = move_soft_dirty_pte(pte);
 
 		if (need_clear_uffd_wp && pte_is_uffd_wp_marker(pte))

Re: [PATCH v3 4/4] mm: pass the number of PTEs to move_pte()

[David Hildenbrand (Arm)]

On 10/27/26 22:26, Imre Kaloz wrote:
> Hi David,
> 
> On 10/23/26 22:53, David Hildenbrand (Arm) wrote:
>> Is there some way we can avoid this
>> helper-that-shouldn't-exist-but-someone-hacked-it-in entirely?
>> If it makes sparc64 slightly slower #dontcare.
> 
> I tried three ways of doing that on an Ultra 45 (UltraSPARC IIIi, two
> CPUs), all with move_pte() removed entirely:
> 
>  A) sparc64's flush_cache_range() flushes every present page in the
>     range from the D-cache;
>  B) sparc64's flush_cache_range() flushes the whole D-cache;
>  C) A for ranges of up to 10 pages, B above that.
> 
> All three pass a test that moves pages between cache colors with
> mremap() and checks for stale data. With no flush at all the same test
> finds about 15 million stale words per run, so it does catch the
> problem.
> 
> Medians of 20 runs over two boots per kernel, against this series'
> move_pte() as the baseline:
> 
>                             move_pte()        A          B          C
>   apt-cache gencaches           6.63 s     +0.2%      +0.3%      +0.1%
>   fork+exec                                +0.7%      +3.1%      -0.6%
>   mremap 8K, same color          14 us      20 us      43 us      20 us
>   mremap 8K, other color         21 us      20 us      43 us      20 us
>   mremap 1M, same color          30 us     524 us      60 us      59 us
>   mremap 8M, same color         209 us    4152 us     229 us     227 us
>   mremap 8M, other color       4176 us    4166 us     236 us     233 us
>   mremap 8M THP, same color      14 us    3933 us      43 us      42 us
> 
> flush_cache_range() has more callers than mremap(), mostly in rmap,
> so I also timed reclaim, migration and writeback (medians of 5):
> 
>                                   A          B          C
>   MADV_PAGEOUT to swap          +0.4%     +12.9%      -1.3%
>   reclaim under memory.max      +1.9%     +17.7%      +2.0%
>   msync of shared file mappings -4.4%      -3.4%      -0.4%
>   compaction, move_pages()      within 3% for all three
> 
> So A pays on large moves, B on reclaim, and C on neither: it costs
> the same as A on small ranges and the same as B on large ones. I'd go
> with C. It would only flush on Spitfire and Cheetah, which have the
> aliasing D-cache, so sun4v and the SPARC64 VII in Magnus's M3000
> series skip it.
> 
> If that works for you, I'd suggest dropping 4/4 and using C as its
> replacement; it's below the scissors, on top of 1/4-3/4.
But that's too complicated, requiring page table walks ....

People running museum piece hardware and complaining about performance of mremap
or memory reclaim should reconsider life choices.

So I'd say, make it as slow as you want while reducing core-MM complexity and
not adding more arch complexity.

-- 
Cheers,

David

Re: [PATCH v3 4/4] mm: pass the number of PTEs to move_pte()

[Imre Kaloz]

Hi David,

On 10/28/26 09:49, David Hildenbrand (Arm) wrote:
> But that's too complicated, requiring page table walks ....
> People running museum piece hardware and complaining about performance of mremap
> or memory reclaim should reconsider life choices.

I'm one of the museum folks myself, though more the kind who loves
running a modern kernel on old hardware than the kind who complains
about it :) A lot of these machines are still doing real work, and
these days not everyone can, or wants to, just buy a new computer.

Some of us spend our time keeping these platforms working, and I'm
not asking anyone else to carry that, which is why I tried to keep
all of it in sparc.

> So I'd say, make it as slow as you want while reducing core-MM complexity and
> not adding more arch complexity.

With C, move_pte() and __HAVE_ARCH_MOVE_PTE go away from
include/linux/pgtable.h and mm/mremap.c just like with A and B. The
walk lives in sparc's own flush_cache_range() and only runs on the
CPUs that need it.

B would be less sparc code, but every UltraSPARC I-IV machine would
pay 13-18% on reclaim for it, so I'd still lean towards C. If you feel
strongly about B, though, I'm happy to send that instead.


Best,
Imre

Re: [PATCH v3 4/4] mm: pass the number of PTEs to move_pte()

[David Hildenbrand (Arm)]

On 10/28/26 10:16, Imre Kaloz wrote:
> Hi David,
> 
> On 10/28/26 09:49, David Hildenbrand (Arm) wrote:
>> But that's too complicated, requiring page table walks ....
>> People running museum piece hardware and complaining about performance of mremap
>> or memory reclaim should reconsider life choices.
> 
> I'm one of the museum folks myself, though more the kind who loves
> running a modern kernel on old hardware than the kind who complains
> about it :) A lot of these machines are still doing real work, and
> these days not everyone can, or wants to, just buy a new computer.
> 
> Some of us spend our time keeping these platforms working, and I'm
> not asking anyone else to carry that, which is why I tried to keep
> all of it in sparc.

Okay, so mremap() and reclaim memory performance concerns make you sleep bad at
night? :)

Don't get me wrong, I know that running new kernels on old HW is fun.

Last week a got upstream Linux running on a 32bit x86 notebook from 2007 with
the snap of a finger. Everything just works (using Alpine).

I couldn't care less about performance. Nobody should.

The biggest problem all of us with supporting old hardware is the maintanance
cost. Not just core-mm, but also things like ad-hoc page table walkers you are
adding we (from core-mm) might have to touch at some point.

If there is no good reason for the complexity, out it goes.

Because otherwise people will push more and more for removing legacy support
completely, and that's not what you want, right?

> 
>> So I'd say, make it as slow as you want while reducing core-MM complexity and
>> not adding more arch complexity.
> 
> With C, move_pte() and __HAVE_ARCH_MOVE_PTE go away from
> include/linux/pgtable.h and mm/mremap.c just like with A and B. The
> walk lives in sparc's own flush_cache_range() and only runs on the
> CPUs that need it.
> 
> B would be less sparc code, but every UltraSPARC I-IV machine would
> pay 13-18% on reclaim for it, so I'd still lean towards C. If you feel
> strongly about B, though, I'm happy to send that instead.

You know my answer :)

-- 
Cheers,

David

Re: [PATCH v3 4/4] mm: pass the number of PTEs to move_pte()

[John Paul Adrian Glaubitz]

Hi David,

On Wed, 2026-10-28 at 10:22 +0100, David Hildenbrand (Arm) wrote:
> On 10/28/26 10:16, Imre Kaloz wrote:
> > Hi David,
> > 
> > On 10/28/26 09:49, David Hildenbrand (Arm) wrote:
> > > But that's too complicated, requiring page table walks ....
> > > People running museum piece hardware and complaining about performance of mremap
> > > or memory reclaim should reconsider life choices.
> > 
> > I'm one of the museum folks myself, though more the kind who loves
> > running a modern kernel on old hardware than the kind who complains
> > about it :) A lot of these machines are still doing real work, and
> > these days not everyone can, or wants to, just buy a new computer.
> > 
> > Some of us spend our time keeping these platforms working, and I'm
> > not asking anyone else to carry that, which is why I tried to keep
> > all of it in sparc.
> 
> Okay, so mremap() and reclaim memory performance concerns make you sleep bad at
> night? :)
> 
> Don't get me wrong, I know that running new kernels on old HW is fun.
> 
> Last week a got upstream Linux running on a 32bit x86 notebook from 2007 with
> the snap of a finger. Everything just works (using Alpine).

That's cool!

> I couldn't care less about performance. Nobody should.

Well, try to put it into some perspective: For some people, old hardware
is all what they have and given the current high prices, I assume many
people will be stuck with old hardware for a while now. Think of developing
countries as well.

> The biggest problem all of us with supporting old hardware is the maintanance
> cost. Not just core-mm, but also things like ad-hoc page table walkers you are
> adding we (from core-mm) might have to touch at some point.

Well, but if people are using it in the end, I think it's justified to support it.

> If there is no good reason for the complexity, out it goes.

The patch doesn't really look that complex though. It adds an additional parameter
to move_pte() which adds a few more lines of code. Not really overly complex, I think.

> Because otherwise people will push more and more for removing legacy support
> completely, and that's not what you want, right?

I don't think it's fair to make the kernel exclusive to very recent hardware only.

Adrian
> 

-- 
 .''`.  John Paul Adrian Glaubitz
: :' :  Debian Developer
`. `'   Physicist
  `-    GPG: 62FF 8A75 84E0 2956 9546  0006 7426 3B37 F5B5 F913

Re: [PATCH v3 4/4] mm: pass the number of PTEs to move_pte()

[David Hildenbrand (Arm)]

On 10/28/26 10:58, John Paul Adrian Glaubitz wrote:
> Hi David,
> 
> On Wed, 2026-10-28 at 10:22 +0100, David Hildenbrand (Arm) wrote:
>> On 10/28/26 10:16, Imre Kaloz wrote:
>>> Hi David,
>>>
>>>
>>> I'm one of the museum folks myself, though more the kind who loves
>>> running a modern kernel on old hardware than the kind who complains
>>> about it :) A lot of these machines are still doing real work, and
>>> these days not everyone can, or wants to, just buy a new computer.
>>>
>>> Some of us spend our time keeping these platforms working, and I'm
>>> not asking anyone else to carry that, which is why I tried to keep
>>> all of it in sparc.
>>
>> Okay, so mremap() and reclaim memory performance concerns make you sleep bad at
>> night? :)
>>
>> Don't get me wrong, I know that running new kernels on old HW is fun.
>>
>> Last week a got upstream Linux running on a 32bit x86 notebook from 2007 with
>> the snap of a finger. Everything just works (using Alpine).
> 
> That's cool!
> 
>> I couldn't care less about performance. Nobody should.
> 
> Well, try to put it into some perspective: For some people, old hardware
> is all what they have and given the current high prices, I assume many
> people will be stuck with old hardware for a while now. Think of developing
> countries as well.

sparc64 machines?! cmon

> 
>> The biggest problem all of us with supporting old hardware is the maintanance
>> cost. Not just core-mm, but also things like ad-hoc page table walkers you are
>> adding we (from core-mm) might have to touch at some point.
> 
> Well, but if people are using it in the end, I think it's justified to support it.

No. Run stable kernels.

> 
>> If there is no good reason for the complexity, out it goes.
> 
> The patch doesn't really look that complex though. It adds an additional parameter
> to move_pte() which adds a few more lines of code. Not really overly complex, I think.

Follow the discussion.

> 
>> Because otherwise people will push more and more for removing legacy support
>> completely, and that's not what you want, right?
> 
> I don't think it's fair to make the kernel exclusive to very recent hardware only.

"Very recent" find the mistake.

(sorry, I'm unusually grumpy today)

-- 
Cheers,

David

Re: [PATCH v3 4/4] mm: pass the number of PTEs to move_pte()

[Lorenzo Stoakes (ARM)]

On Wed, Oct 28, 2026 at 11:02:17AM +0100, David Hildenbrand (Arm) wrote:
> On 10/28/26 10:58, John Paul Adrian Glaubitz wrote:
> > I don't think it's fair to make the kernel exclusive to very recent hardware only.

Nobody's making the kernel 'exclusive' to 'recent' hardware.

It's about overburdened maintainers having to deal with museum-piece hardware at
the cost of literally _everybody else_ who uses linux.

Use an older or downstream kernel - problem solved ;)

>
> "Very recent" find the mistake.

Exactly...

>
> (sorry, I'm unusually grumpy today)

Justifiable ;)

--
Cheers, Lorenzo

Re: [PATCH v3 4/4] mm: pass the number of PTEs to move_pte()

[Imre Kaloz]

Hi David,

On 10/28/26 10:22, David Hildenbrand (Arm) wrote:
> Okay, so mremap() and reclaim memory performance concerns make you sleep bad at
> night? :)

Probably less than move_pte() kept you mm folks up at night ;)

> Don't get me wrong, I know that running new kernels on old HW is fun.
> Last week a got upstream Linux running on a 32bit x86 notebook from 2007 with
> the snap of a finger. Everything just works (using Alpine).

Nice, that's exactly the kind of thing I enjoy too.

> The biggest problem all of us with supporting old hardware is the maintanance
> cost. Not just core-mm, but also things like ad-hoc page table walkers you are
> adding we (from core-mm) might have to touch at some point.
>
> If there is no good reason for the complexity, out it goes.
>
> Because otherwise people will push more and more for removing legacy support
> completely, and that's not what you want, right?

Definitely not. Fair point, I was only looking at it from the sparc
side, and keeping these platforms cheap to carry matters more than a
few percent on reclaim. B it is, then; it's below the scissors, on top
of 1/4-3/4. Thanks for taking the time to go through this!

Andreas, if David is happy with it, would you take it in place of 4/4?


Best,
Imre

-- >8 --
Subject: [PATCH] sparc64: replace move_pte() with a D-cache flush in flush_cache_range()

mremap() can move a page to an address of the other D-cache color,
leaving lines of the old color in the virtually indexed D-cache of
Spitfire and Cheetah CPUs.  move_pte() flushed the page for each PTE
moved, which is only correct while move_ptes() handles one PTE at a
time.

Drop move_pte(), which no other architecture defines, and flush the
whole D-cache in flush_cache_range(), which mremap() calls on the old
range before moving the PTEs.  This keeps the fix in sparc64 without
a page table walk.

Signed-off-by: Imre Kaloz <kaloz@kernel.org>
---
diff --git a/arch/sparc/include/asm/cacheflush_64.h b/arch/sparc/include/asm/cacheflush_64.h
index 54a2d37008e6..1a116b288816 100644
--- a/arch/sparc/include/asm/cacheflush_64.h
+++ b/arch/sparc/include/asm/cacheflush_64.h
@@ -21,8 +21,8 @@ void __flushw_user(void);
 #define flush_cache_mm(__mm) \
 	do { if ((__mm) == current->mm) flushw_user(); } while(0)
 #define flush_cache_dup_mm(mm) flush_cache_mm(mm)
-#define flush_cache_range(vma, start, end) \
-	flush_cache_mm((vma)->vm_mm)
+void flush_cache_range(struct vm_area_struct *vma, unsigned long start,
+		       unsigned long end);
 #define flush_cache_page(vma, page, pfn) \
 	flush_cache_mm((vma)->vm_mm)
 
diff --git a/arch/sparc/include/asm/pgtable_64.h b/arch/sparc/include/asm/pgtable_64.h
index 1430489b9ac5..2452d959aac1 100644
--- a/arch/sparc/include/asm/pgtable_64.h
+++ b/arch/sparc/include/asm/pgtable_64.h
@@ -945,27 +945,6 @@ static inline void set_ptes(struct mm_struct *mm, unsigned long addr,
 #define pte_clear(mm,addr,ptep)		\
 	set_pte_at((mm), (addr), (ptep), __pte(0UL))
 
-#ifdef DCACHE_ALIASING_POSSIBLE
-/* This flushes only the page the PTE maps, so it relies on move_ptes()
- * calling it for one PTE at a time, which the default pte_batch_hint()
- * guarantees.
- */
-#define __HAVE_ARCH_MOVE_PTE
-#define move_pte(pte, old_addr, new_addr)				\
-({									\
-	pte_t newpte = (pte);						\
-	if (tlb_type != hypervisor && pte_present(pte)) {		\
-		unsigned long this_pfn = pte_pfn(pte);			\
-									\
-		if (pfn_valid(this_pfn) &&				\
-		    (((old_addr) ^ (new_addr)) & (1 << 13)))		\
-			flush_dcache_page_all(current->mm,		\
-					      pfn_to_page(this_pfn));	\
-	}								\
-	newpte;								\
-})
-#endif
-
 extern pgd_t swapper_pg_dir[PTRS_PER_PGD];
 
 void paging_init(void);
diff --git a/arch/sparc/mm/init_64.c b/arch/sparc/mm/init_64.c
index d1f94b8d6312..d870cfadba21 100644
--- a/arch/sparc/mm/init_64.c
+++ b/arch/sparc/mm/init_64.c
@@ -3115,6 +3115,24 @@ void flush_cache_vmap(unsigned long start, unsigned long end)
 #endif
 }
 
+/*
+ * mremap() moves the PTEs of [start, end) to addresses that may have the
+ * other color, and calls this on the old range first.  Flush the whole
+ * D-cache so no line of the old color outlives the move.
+ */
+void flush_cache_range(struct vm_area_struct *vma, unsigned long start,
+		       unsigned long end)
+{
+	/* flush_cache_mm() spills the user register windows. */
+	flush_cache_mm(vma->vm_mm);
+#ifdef DCACHE_ALIASING_POSSIBLE
+	/* The D-cache is virtually indexed on Spitfire and Cheetah. */
+	if (tlb_type == spitfire || tlb_type == cheetah ||
+	    tlb_type == cheetah_plus)
+		flush_dcache_all();
+#endif
+}
+
 void flush_tlb_kernel_range(unsigned long start, unsigned long end)
 {
 	if (start < HI_OBP_ADDRESS && end > LOW_OBP_ADDRESS) {
diff --git a/include/linux/pgtable.h b/include/linux/pgtable.h
index 8c093c119e5a..b95eea612261 100644
--- a/include/linux/pgtable.h
+++ b/include/linux/pgtable.h
@@ -1470,10 +1470,6 @@ static inline void arch_swap_restore(swp_entry_t entry, struct folio *folio)
 }
 #endif
 
-#ifndef __HAVE_ARCH_MOVE_PTE
-#define move_pte(pte, old_addr, new_addr)	(pte)
-#endif
-
 #ifndef pte_accessible
 # define pte_accessible(mm, pte)	((void)(pte), 1)
 #endif
diff --git a/mm/mremap.c b/mm/mremap.c
index e8df5cdb0ac9..2257e3010f41 100644
--- a/mm/mremap.c
+++ b/mm/mremap.c
@@ -289,7 +289,6 @@ static int move_ptes(struct pagetable_move_control *pmc,
 			force_flush = true;
 		}
 		pte = get_and_clear_ptes(mm, old_addr, old_ptep, nr_ptes);
-		pte = move_pte(pte, old_addr, new_addr);
 		pte = move_soft_dirty_pte(pte);
 
 		if (need_clear_uffd_wp && pte_is_uffd_wp_marker(pte))

Re: [PATCH v3 4/4] mm: pass the number of PTEs to move_pte()

[Andreas Larsson]

On 2026-10-28 11:17, Imre Kaloz wrote:
>> The biggest problem all of us with supporting old hardware is the maintanance
>> cost. Not just core-mm, but also things like ad-hoc page table walkers you are
>> adding we (from core-mm) might have to touch at some point.
>>
>> If there is no good reason for the complexity, out it goes.
>>
>> Because otherwise people will push more and more for removing legacy support
>> completely, and that's not what you want, right?
> 
> Definitely not. Fair point, I was only looking at it from the sparc
> side, and keeping these platforms cheap to carry matters more than a
> few percent on reclaim. B it is, then; it's below the scissors, on top
> of 1/4-3/4. Thanks for taking the time to go through this!
> 
> Andreas, if David is happy with it, would you take it in place of 4/4?
I'm open to a solution like that, but it'll have to wait for the next
round as we are far too far into the merge window. Patch 3 stands on its
own legs as a bugfix in the current context. So send it as a separate
patch.

Cheers,
Andreas

Re: [PATCH v3 4/4] mm: pass the number of PTEs to move_pte()

[David Hildenbrand (Arm)]

On 10/28/26 14:57, Andreas Larsson wrote:
> On 2026-10-28 11:17, Imre Kaloz wrote:
>>> The biggest problem all of us with supporting old hardware is the maintanance
>>> cost. Not just core-mm, but also things like ad-hoc page table walkers you are
>>> adding we (from core-mm) might have to touch at some point.
>>>
>>> If there is no good reason for the complexity, out it goes.
>>>
>>> Because otherwise people will push more and more for removing legacy support
>>> completely, and that's not what you want, right?
>>
>> Definitely not. Fair point, I was only looking at it from the sparc
>> side, and keeping these platforms cheap to carry matters more than a
>> few percent on reclaim. B it is, then; it's below the scissors, on top
>> of 1/4-3/4. Thanks for taking the time to go through this!
>>
>> Andreas, if David is happy with it, would you take it in place of 4/4?
> I'm open to a solution like that, but it'll have to wait for the next
> round as we are far too far into the merge window. 

Fully agreed.

-- 
Cheers,

David

Re: [PATCH v3 4/4] mm: pass the number of PTEs to move_pte()

[Imre Kaloz]

Hi Andreas, David,

On 10/28/26 14:59, David Hildenbrand (Arm) wrote:
> On 10/28/26 14:57, Andreas Larsson wrote:
>> I'm open to a solution like that, but it'll have to wait for the next
>> round as we are far too far into the merge window. Patch 3 stands on its
>> own legs as a bugfix in the current context. So send it as a separate
>> patch.
>
> Fully agreed.

Makes sense, no rush. I'll drop 4/4 and send the move_pte() removal as
a standalone patch once -rc1 is out.


Best,
Imre

Re: [PATCH v3 4/4] mm: pass the number of PTEs to move_pte()

[David Hildenbrand (Arm)]

>>
>> Because otherwise people will push more and more for removing legacy support
>> completely, and that's not what you want, right?
> 
> Definitely not. Fair point, I was only looking at it from the sparc
> side, and keeping these platforms cheap to carry matters more than a
> few percent on reclaim. B it is, then; it's below the scissors, on top
> of 1/4-3/4. Thanks for taking the time to go through this!
> 
> Andreas, if David is happy with it, would you take it in place of 4/4?

Thanks, appreciated!

-- 
Cheers,

David

Re: [PATCH v3 0/4] sparc64: D-cache alias flushing fixes

[Stian Halseth]

[-- Attachment #1: Type: text/plain, Size: 910 bytes --]

Hi Imre,

On Thu, 2026-10-22 at 01:33 +0200, Imre Kaloz wrote:
> These fix D-cache alias flushing on sparc64 CPUs with a virtually
> indexed L1 D-cache (Spitfire and Cheetah).
> 

Tested on top of Andreas' for-next on my V240 (sun4u, 2xUltraSPARC
IIIi), against the same for-next without the series, both with THP
enabled. The stale reads through a reused vmalloc address are gone, and
moving an 8 MB THP to the other colour now takes milliseconds instead
of seven seconds.

I also checked 1/4 on its own by booting with the vmemmap check in 3/4
disabled. The four cross calls from sparse_init() still reach
xcall_deliver(), but physical address 0 is now left untouched.

4/4 can't really be tested on sparc64, since move_ptes() never
batches, but it builds and nothing changes. Each patch builds
cleanly for SMP and UP.

Tested-by: Stian Halseth <stian@itx.no> # Sun Fire V240

/Stian

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 228 bytes --]

Re: [PATCH v3 0/4] sparc64: D-cache alias flushing fixes

[Andreas Larsson]

On 2026-10-22 01:33, Imre Kaloz wrote:
> These fix D-cache alias flushing on sparc64 CPUs with a virtually
> indexed L1 D-cache (Spitfire and Cheetah).
> 
> 1/4 makes xcall_deliver() bail out before it touches the trap block
> when there is no other CPU to call, so an early cross call, before
> init_IRQ() sets up the mondo blocks, can't end up writing to physical
> address 0.
> 
> 2/4 flushes only the page a PTE maps instead of its whole folio, which
> removes about a million redundant page flushes from a colour-changing
> mremap() of a large folio (7.3 s on a Sun Fire V240).
> 
> 3/4 implements flush_cache_vmap() and flush_cache_vunmap(), which have
> been no-ops. Stale vmalloc lines corrupt BPF programs and module data
> on UltraSPARC III.
> 
> 4/4 passes the number of PTEs to move_pte(), as Andreas suggested. It
> touches mm/mremap.c and include/linux/pgtable.h, so it needs an ack
> from the mm folks.
> 
> 1-3 don't depend on 4 and can go in on their own. 2 and 3 are for
> stable (3 needs 2), 1 and 4 aren't. The series is on top of the sparc
> for-next branch.
> 
> Changes in v3:
> - new 1/4 and 4/4, following Andreas' review
> - 2/4: reworded the message and the move_pte() comment, marked the
>   UP #endif
> - 3/4: reworded the message, dropped the Suggested-by (I have no
>   email address for nsafran1217, the Link credits them), noted the
>   stable dependency on 2/4
> - picked up Stian's and Adrian's Tested-by tags
> 
> v2: https://lore.kernel.org/r/20260928132102.1707-1-kaloz@kernel.org
> 
> Imre Kaloz (4):
>   sparc64: skip the trap block in xcall_deliver() when there is no
>     target
>   sparc64: flush only the aliased page, not its whole folio
>   sparc64: flush vmalloc ranges from the D-cache on map and unmap
>   mm: pass the number of PTEs to move_pte()
> 
>  arch/sparc/include/asm/cacheflush_64.h |  9 ++--
>  arch/sparc/include/asm/pgtable_64.h    | 14 ++++--
>  arch/sparc/kernel/smp_64.c             | 51 +++++++++++++++-----
>  arch/sparc/mm/init_64.c                | 65 ++++++++++++++++++++++++++
>  arch/sparc/mm/tlb.c                    |  2 +-
>  arch/sparc/mm/ultra.S                  | 25 ++++++++++
>  include/linux/pgtable.h                |  2 +-
>  mm/mremap.c                            |  2 +-
>  8 files changed, 148 insertions(+), 22 deletions(-)
> 
> 
> base-commit: 2bc58762a3d74e42de129d717ae3ba10ecf0ab96

Reviewed-by: Andreas Larsson <andreas@gaisler.com>

Picking patches 1-3 to my for-next. I won't pick number 4 up at this
point without confirmation from the mm folks.

Thanks,
Andreas