[RFC PATCH 0/6] Implement MMU_LAZY_TLB_SHOOTDOWN for alpha

[RFC PATCH 0/6] Implement MMU_LAZY_TLB_SHOOTDOWN for alpha

[Nicholas Piggin]

Hi,

I started looking at alpha TLB flushing a while back with the aim of
converting another architecture to this option. powerpc has fairly
complicated MMU and TLB code so I thought it would be good to have a
simpler reference architecture.

alpha had SMP QEMU support, TLB flushing that's fairly simple, and it
doesn't use mm_cpumask in a complicated way, so it fits.

These patches work for me, but only tested on QEMU so that's not saying
too much when it comes to race conditions, so some healthy skepticism
is recommended.

Any thoughts about the series would be interesting to hear.

Thanks,
Nick

Nicholas Piggin (6):
  alpha: remove extern inline from mmu_context
  alpha: implement simple mm_cpumask TLB flush filter
  alpha: remove TLB flushing mm_users special case
  alpha: clean mm_cpumask when flushing TLBs
  alpha: enable MMU_LAZY_TLB_SHOOTDOWN
  alpha: shoot the lazy tlb mm when flushing TLBs

 arch/alpha/Kconfig                   |   1 +
 arch/alpha/include/asm/mmu_context.h |  43 +++++-----
 arch/alpha/include/asm/tlbflush.h    |   3 +
 arch/alpha/kernel/smp.c              | 120 ++++++++++++++-------------
 arch/alpha/mm/init.c                 |   2 +
 5 files changed, 94 insertions(+), 75 deletions(-)

-- 
2.40.1

[RFC PATCH 1/6] alpha: remove extern inline from mmu_context

[Nicholas Piggin]

This caused a pain when working on alpha, and extern inlines aren't
really used these days (outside alpha), so remove them from mmu_context.

Signed-off-by: Nicholas Piggin <npiggin@gmail.com>
---
 arch/alpha/include/asm/mmu_context.h | 24 +++++-------------------
 1 file changed, 5 insertions(+), 19 deletions(-)

diff --git a/arch/alpha/include/asm/mmu_context.h b/arch/alpha/include/asm/mmu_context.h
index 4eea7c616992..8ce89350e4b3 100644
--- a/arch/alpha/include/asm/mmu_context.h
+++ b/arch/alpha/include/asm/mmu_context.h
@@ -20,11 +20,7 @@
  * table pointer or when we update the ASN of the current process.
  */
 
-/* Don't get into trouble with dueling __EXTERN_INLINEs.  */
-#ifndef __EXTERN_INLINE
 #include <asm/io.h>
-#endif
-
 
 static inline unsigned long
 __reload_thread(struct pcb_struct *pcb)
@@ -112,12 +108,7 @@ extern unsigned long last_asn;
  * run.
  */
 
-#ifndef __EXTERN_INLINE
-#define __EXTERN_INLINE extern inline
-#define __MMU_EXTERN_INLINE
-#endif
-
-extern inline unsigned long
+static inline unsigned long
 __get_new_mm_context(struct mm_struct *mm, long cpu)
 {
 	unsigned long asn = cpu_last_asn(cpu);
@@ -132,7 +123,7 @@ __get_new_mm_context(struct mm_struct *mm, long cpu)
 	return next;
 }
 
-__EXTERN_INLINE void
+static inline void
 ev5_switch_mm(struct mm_struct *prev_mm, struct mm_struct *next_mm,
 	      struct task_struct *next)
 {
@@ -162,7 +153,7 @@ ev5_switch_mm(struct mm_struct *prev_mm, struct mm_struct *next_mm,
 	task_thread_info(next)->pcb.asn = mmc & HARDWARE_ASN_MASK;
 }
 
-__EXTERN_INLINE void
+static inline void
 ev4_switch_mm(struct mm_struct *prev_mm, struct mm_struct *next_mm,
 	      struct task_struct *next)
 {
@@ -201,13 +192,13 @@ do {								\
 #define check_mmu_context()  do { } while(0)
 #endif
 
-__EXTERN_INLINE void
+static inline void
 ev5_activate_mm(struct mm_struct *prev_mm, struct mm_struct *next_mm)
 {
 	__load_new_mm_context(next_mm);
 }
 
-__EXTERN_INLINE void
+static inline void
 ev4_activate_mm(struct mm_struct *prev_mm, struct mm_struct *next_mm)
 {
 	__load_new_mm_context(next_mm);
@@ -251,9 +242,4 @@ enter_lazy_tlb(struct mm_struct *mm, struct task_struct *tsk)
 
 #include <asm-generic/mmu_context.h>
 
-#ifdef __MMU_EXTERN_INLINE
-#undef __EXTERN_INLINE
-#undef __MMU_EXTERN_INLINE
-#endif
-
 #endif /* __ALPHA_MMU_CONTEXT_H */
-- 
2.40.1

[RFC PATCH 2/6] alpha: implement simple mm_cpumask TLB flush filter

[Nicholas Piggin]

Implement a sticky mm_cpumask and use it to filter TLB flush IPIs.
Sticky meaning that when an mm runs on a CPU, it gets set in the mask
and never goes away. This is the "base" mm_cpumask implementation that
comes with the least complexity.

This reduces IPIs booting into a small rootfs by about 10-15% on 4CPU
system.

Signed-off-by: Nicholas Piggin <npiggin@gmail.com>
---
 arch/alpha/include/asm/mmu_context.h | 19 +++++++++++++++++++
 arch/alpha/kernel/smp.c              | 16 ++++++++++++++--
 arch/alpha/mm/init.c                 |  2 ++
 3 files changed, 35 insertions(+), 2 deletions(-)

diff --git a/arch/alpha/include/asm/mmu_context.h b/arch/alpha/include/asm/mmu_context.h
index 8ce89350e4b3..9c9e9a8c01a4 100644
--- a/arch/alpha/include/asm/mmu_context.h
+++ b/arch/alpha/include/asm/mmu_context.h
@@ -135,6 +135,21 @@ ev5_switch_mm(struct mm_struct *prev_mm, struct mm_struct *next_mm,
 #ifdef CONFIG_SMP
 	cpu_data[cpu].asn_lock = 1;
 	barrier();
+
+	if (!cpumask_test_cpu(cpu, mm_cpumask(next_mm))) {
+		cpumask_set_cpu(cpu, mm_cpumask(next_mm));
+		/*
+		 * Store to mm_cpumask must be visible to CPUs performing
+		 * TLB flushes before memory accesses that could bring in
+		 * new TLB entries. This orders the store above with the
+		 * load of the new context and subsequent loads of PTEs
+		 * that can then be cached in the TLB.
+		 *
+		 * The other side is in the mm_cpumask testing in TLB
+		 * flush.
+		 */
+		smp_mb();
+	}
 #endif
 	asn = cpu_last_asn(cpu);
 	mmc = next_mm->context[cpu];
@@ -151,6 +166,8 @@ ev5_switch_mm(struct mm_struct *prev_mm, struct mm_struct *next_mm,
 	   a new mm->context (via flush_tlb_mm) without the ASN serial
 	   number wrapping.  We have no way to detect when this is needed.  */
 	task_thread_info(next)->pcb.asn = mmc & HARDWARE_ASN_MASK;
+
+	WARN_ON(!cpumask_test_cpu(cpu, mm_cpumask(prev_mm)));
 }
 
 static inline void
@@ -195,12 +212,14 @@ do {								\
 static inline void
 ev5_activate_mm(struct mm_struct *prev_mm, struct mm_struct *next_mm)
 {
+	cpumask_set_cpu(smp_processor_id(), mm_cpumask(next_mm));
 	__load_new_mm_context(next_mm);
 }
 
 static inline void
 ev4_activate_mm(struct mm_struct *prev_mm, struct mm_struct *next_mm)
 {
+	cpumask_set_cpu(smp_processor_id(), mm_cpumask(next_mm));
 	__load_new_mm_context(next_mm);
 	tbiap();
 }
diff --git a/arch/alpha/kernel/smp.c b/arch/alpha/kernel/smp.c
index 7439b2377df5..b702372fbaba 100644
--- a/arch/alpha/kernel/smp.c
+++ b/arch/alpha/kernel/smp.c
@@ -145,6 +145,7 @@ smp_callin(void)
 	/* All kernel threads share the same mm context.  */
 	mmgrab(&init_mm);
 	current->active_mm = &init_mm;
+	cpumask_set_cpu(smp_processor_id(), mm_cpumask(&init_mm));
 
 	/* inform the notifiers about the new cpu */
 	notify_cpu_starting(cpuid);
@@ -655,7 +656,17 @@ flush_tlb_mm(struct mm_struct *mm)
 		}
 	}
 
-	smp_call_function(ipi_flush_tlb_mm, mm, 1);
+	/*
+	 * TLB flush IPIs will be sent to all CPUs with mm_cpumask set. The
+	 * problem of ordering the load of mm_cpumask vs a CPU switching to
+	 * the mm and caching a translation from a PTE being invalidated and
+	 * flushed here means we must have a memory barrier. This orders the
+	 * prior stores to invalidate the PTEs from the load of mm_cpumask.
+	 *
+	 * The other side is switch_mm.
+	 */
+	smp_mb();
+	smp_call_function_many(mm_cpumask(mm), ipi_flush_tlb_mm, mm, 1);
 
 	preempt_enable();
 }
@@ -706,7 +717,8 @@ flush_tlb_page(struct vm_area_struct *vma, unsigned long addr)
 	data.mm = mm;
 	data.addr = addr;
 
-	smp_call_function(ipi_flush_tlb_page, &data, 1);
+	smp_mb(); /* see flush_tlb_mm */
+	smp_call_function_many(mm_cpumask(mm), ipi_flush_tlb_page, &data, 1);
 
 	preempt_enable();
 }
diff --git a/arch/alpha/mm/init.c b/arch/alpha/mm/init.c
index a155180d7a83..33f4c0abd2c8 100644
--- a/arch/alpha/mm/init.c
+++ b/arch/alpha/mm/init.c
@@ -254,6 +254,8 @@ void __init paging_init(void)
 
 	/* Initialize the kernel's ZERO_PGE. */
 	memset(absolute_pointer(ZERO_PGE), 0, PAGE_SIZE);
+
+	cpumask_set_cpu(raw_smp_processor_id(), mm_cpumask(&init_mm));
 }
 
 #if defined(CONFIG_ALPHA_GENERIC) || defined(CONFIG_ALPHA_SRM)
-- 
2.40.1

[RFC PATCH 3/6] alpha: remove TLB flushing mm_users special case

[Nicholas Piggin]

With mm_cpumask, there is less reason to keep the mm_users special case
in the TLB flushing code, because IPIs will be filtered out by the mask.

These special cases are another complicated set of races e.g., vs
kthread_use_mm that make this code tricky to reason about.

Signed-off-by: Nicholas Piggin <npiggin@gmail.com>
---
 arch/alpha/kernel/smp.c | 68 +++++++++--------------------------------
 1 file changed, 15 insertions(+), 53 deletions(-)

diff --git a/arch/alpha/kernel/smp.c b/arch/alpha/kernel/smp.c
index b702372fbaba..e436c056267d 100644
--- a/arch/alpha/kernel/smp.c
+++ b/arch/alpha/kernel/smp.c
@@ -639,23 +639,6 @@ ipi_flush_tlb_mm(void *x)
 void
 flush_tlb_mm(struct mm_struct *mm)
 {
-	preempt_disable();
-
-	if (mm == current->active_mm) {
-		flush_tlb_current(mm);
-		if (atomic_read(&mm->mm_users) <= 1) {
-			int cpu, this_cpu = smp_processor_id();
-			for (cpu = 0; cpu < NR_CPUS; cpu++) {
-				if (!cpu_online(cpu) || cpu == this_cpu)
-					continue;
-				if (mm->context[cpu])
-					mm->context[cpu] = 0;
-			}
-			preempt_enable();
-			return;
-		}
-	}
-
 	/*
 	 * TLB flush IPIs will be sent to all CPUs with mm_cpumask set. The
 	 * problem of ordering the load of mm_cpumask vs a CPU switching to
@@ -666,8 +649,12 @@ flush_tlb_mm(struct mm_struct *mm)
 	 * The other side is switch_mm.
 	 */
 	smp_mb();
+	preempt_disable();
+	if (mm == current->active_mm)
+		flush_tlb_current(mm);
+	else
+		flush_tlb_other(mm);
 	smp_call_function_many(mm_cpumask(mm), ipi_flush_tlb_mm, mm, 1);
-
 	preempt_enable();
 }
 EXPORT_SYMBOL(flush_tlb_mm);
@@ -696,30 +683,17 @@ flush_tlb_page(struct vm_area_struct *vma, unsigned long addr)
 	struct flush_tlb_page_struct data;
 	struct mm_struct *mm = vma->vm_mm;
 
-	preempt_disable();
-
-	if (mm == current->active_mm) {
-		flush_tlb_current_page(mm, vma, addr);
-		if (atomic_read(&mm->mm_users) <= 1) {
-			int cpu, this_cpu = smp_processor_id();
-			for (cpu = 0; cpu < NR_CPUS; cpu++) {
-				if (!cpu_online(cpu) || cpu == this_cpu)
-					continue;
-				if (mm->context[cpu])
-					mm->context[cpu] = 0;
-			}
-			preempt_enable();
-			return;
-		}
-	}
-
 	data.vma = vma;
 	data.mm = mm;
 	data.addr = addr;
 
 	smp_mb(); /* see flush_tlb_mm */
+	preempt_disable();
+	if (mm == current->active_mm)
+		flush_tlb_current_page(mm, vma, addr);
+	else
+		flush_tlb_other(mm);
 	smp_call_function_many(mm_cpumask(mm), ipi_flush_tlb_page, &data, 1);
-
 	preempt_enable();
 }
 EXPORT_SYMBOL(flush_tlb_page);
@@ -751,24 +725,12 @@ flush_icache_user_page(struct vm_area_struct *vma, struct page *page,
 	if ((vma->vm_flags & VM_EXEC) == 0)
 		return;
 
+	smp_mb(); /* see flush_tlb_mm */
 	preempt_disable();
-
-	if (mm == current->active_mm) {
+	if (mm == current->active_mm)
 		__load_new_mm_context(mm);
-		if (atomic_read(&mm->mm_users) <= 1) {
-			int cpu, this_cpu = smp_processor_id();
-			for (cpu = 0; cpu < NR_CPUS; cpu++) {
-				if (!cpu_online(cpu) || cpu == this_cpu)
-					continue;
-				if (mm->context[cpu])
-					mm->context[cpu] = 0;
-			}
-			preempt_enable();
-			return;
-		}
-	}
-
-	smp_call_function(ipi_flush_icache_page, mm, 1);
-
+	else
+		flush_tlb_other(mm);
+	smp_call_function_many(mm_cpumask(mm), ipi_flush_icache_page, mm, 1);
 	preempt_enable();
 }
-- 
2.40.1

[RFC PATCH 4/6] alpha: clean mm_cpumask when flushing TLBs

[Nicholas Piggin]

mm_cpumask is a map of the CPUs which must be IPIed to flush TLBs,
and/or IPIed to shootdown lazy TLB mms at exit time.

When flushing TLBs on the CPU, trim it from mm_cpumask if the mm is not
currently active on the CPU. TLBs will have been flush, and the mm is
not active, so there is no more reason to get IPIs.

Signed-off-by: Nicholas Piggin <npiggin@gmail.com>
---
 arch/alpha/include/asm/tlbflush.h |  3 +++
 arch/alpha/kernel/smp.c           | 29 +++++++++++++++++++++++++++--
 2 files changed, 30 insertions(+), 2 deletions(-)

diff --git a/arch/alpha/include/asm/tlbflush.h b/arch/alpha/include/asm/tlbflush.h
index 94dc37cf873a..7c4e719ac9e7 100644
--- a/arch/alpha/include/asm/tlbflush.h
+++ b/arch/alpha/include/asm/tlbflush.h
@@ -12,6 +12,7 @@
 #endif
 
 extern void __load_new_mm_context(struct mm_struct *);
+extern void try_clear_mm_cpumask(struct mm_struct *);
 
 
 /* Use a few helper functions to hide the ugly broken ASN
@@ -106,6 +107,7 @@ static inline void flush_tlb_all(void)
 static inline void
 flush_tlb_mm(struct mm_struct *mm)
 {
+	try_clear_mm_cpumask(mm);
 	if (mm == current->active_mm)
 		flush_tlb_current(mm);
 	else
@@ -118,6 +120,7 @@ flush_tlb_page(struct vm_area_struct *vma, unsigned long addr)
 {
 	struct mm_struct *mm = vma->vm_mm;
 
+	try_clear_mm_cpumask(mm);
 	if (mm == current->active_mm)
 		flush_tlb_current_page(mm, vma, addr);
 	else
diff --git a/arch/alpha/kernel/smp.c b/arch/alpha/kernel/smp.c
index e436c056267d..d668b9d319af 100644
--- a/arch/alpha/kernel/smp.c
+++ b/arch/alpha/kernel/smp.c
@@ -610,6 +610,28 @@ smp_imb(void)
 }
 EXPORT_SYMBOL(smp_imb);
 
+#define asn_locked() (cpu_data[smp_processor_id()].asn_lock)
+
+/*
+ * If the mm_cpumask bit is cleared, the caller *must* flush the TLB for the
+ * mm on this CPU. It is only cleared when the mm is not active, in which
+ * case the flushing always performs flush_tlb_other that flushes everything.
+ * If that changes in callers, they will have to arrange to always do a full
+ * flush if mm_cpumask is cleared by this function.
+ */
+void
+try_clear_mm_cpumask(struct mm_struct *mm)
+{
+	int cpu;
+
+	if (current->active_mm == mm || asn_locked())
+		return;
+
+	cpu = smp_processor_id();
+	if (cpumask_test_cpu(cpu, mm_cpumask(mm)))
+		cpumask_clear_cpu(cpu, mm_cpumask(mm));
+}
+
 static void
 ipi_flush_tlb_all(void *ignored)
 {
@@ -624,12 +646,12 @@ flush_tlb_all(void)
 	on_each_cpu(ipi_flush_tlb_all, NULL, 1);
 }
 
-#define asn_locked() (cpu_data[smp_processor_id()].asn_lock)
-
 static void
 ipi_flush_tlb_mm(void *x)
 {
 	struct mm_struct *mm = x;
+
+	try_clear_mm_cpumask(mm);
 	if (mm == current->active_mm && !asn_locked())
 		flush_tlb_current(mm);
 	else
@@ -671,6 +693,7 @@ ipi_flush_tlb_page(void *x)
 	struct flush_tlb_page_struct *data = x;
 	struct mm_struct * mm = data->mm;
 
+	try_clear_mm_cpumask(mm);
 	if (mm == current->active_mm && !asn_locked())
 		flush_tlb_current_page(mm, data->vma, data->addr);
 	else
@@ -710,6 +733,8 @@ static void
 ipi_flush_icache_page(void *x)
 {
 	struct mm_struct *mm = (struct mm_struct *) x;
+
+	try_clear_mm_cpumask(mm);
 	if (mm == current->active_mm && !asn_locked())
 		__load_new_mm_context(mm);
 	else
-- 
2.40.1

Re: [RFC PATCH 4/6] alpha: clean mm_cpumask when flushing TLBs

[Matt Turner]

On Wed, May 24, 2023 at 1:18 PM Nicholas Piggin <npiggin@gmail.com> wrote:
>
> mm_cpumask is a map of the CPUs which must be IPIed to flush TLBs,
> and/or IPIed to shootdown lazy TLB mms at exit time.
>
> When flushing TLBs on the CPU, trim it from mm_cpumask if the mm is not
> currently active on the CPU. TLBs will have been flush, and the mm is
> not active, so there is no more reason to get IPIs.
>
> Signed-off-by: Nicholas Piggin <npiggin@gmail.com>
> ---
>  arch/alpha/include/asm/tlbflush.h |  3 +++
>  arch/alpha/kernel/smp.c           | 29 +++++++++++++++++++++++++++--
>  2 files changed, 30 insertions(+), 2 deletions(-)
>
> diff --git a/arch/alpha/include/asm/tlbflush.h b/arch/alpha/include/asm/tlbflush.h
> index 94dc37cf873a..7c4e719ac9e7 100644
> --- a/arch/alpha/include/asm/tlbflush.h
> +++ b/arch/alpha/include/asm/tlbflush.h
> @@ -12,6 +12,7 @@
>  #endif
>
>  extern void __load_new_mm_context(struct mm_struct *);
> +extern void try_clear_mm_cpumask(struct mm_struct *);
>
>
>  /* Use a few helper functions to hide the ugly broken ASN
> @@ -106,6 +107,7 @@ static inline void flush_tlb_all(void)
>  static inline void
>  flush_tlb_mm(struct mm_struct *mm)
>  {
> +       try_clear_mm_cpumask(mm);
>         if (mm == current->active_mm)
>                 flush_tlb_current(mm);
>         else
> @@ -118,6 +120,7 @@ flush_tlb_page(struct vm_area_struct *vma, unsigned long addr)
>  {
>         struct mm_struct *mm = vma->vm_mm;
>
> +       try_clear_mm_cpumask(mm);
>         if (mm == current->active_mm)
>                 flush_tlb_current_page(mm, vma, addr);
>         else
> diff --git a/arch/alpha/kernel/smp.c b/arch/alpha/kernel/smp.c
> index e436c056267d..d668b9d319af 100644
> --- a/arch/alpha/kernel/smp.c
> +++ b/arch/alpha/kernel/smp.c
> @@ -610,6 +610,28 @@ smp_imb(void)
>  }
>  EXPORT_SYMBOL(smp_imb);
>
> +#define asn_locked() (cpu_data[smp_processor_id()].asn_lock)
> +
> +/*
> + * If the mm_cpumask bit is cleared, the caller *must* flush the TLB for the
> + * mm on this CPU. It is only cleared when the mm is not active, in which
> + * case the flushing always performs flush_tlb_other that flushes everything.
> + * If that changes in callers, they will have to arrange to always do a full
> + * flush if mm_cpumask is cleared by this function.
> + */
> +void
> +try_clear_mm_cpumask(struct mm_struct *mm)
> +{
> +       int cpu;
> +
> +       if (current->active_mm == mm || asn_locked())
> +               return;
> +
> +       cpu = smp_processor_id();
> +       if (cpumask_test_cpu(cpu, mm_cpumask(mm)))
> +               cpumask_clear_cpu(cpu, mm_cpumask(mm));
> +}

Since this is implemented in smp.c, the function is not available in
!CONFIG_SMP:

ld: kernel/fork.o: in function `dup_mmap':
(.text+0x1734): undefined reference to `try_clear_mm_cpumask'
ld: (.text+0x1738): undefined reference to `try_clear_mm_cpumask'
ld: mm/memory.o: in function `unmap_page_range':
(.text+0x1934): undefined reference to `try_clear_mm_cpumask'
ld: (.text+0x193c): undefined reference to `try_clear_mm_cpumask'
ld: (.text+0x1a28): undefined reference to `try_clear_mm_cpumask'
ld: mm/memory.o:(.text+0x1a30): more undefined references to
`try_clear_mm_cpumask' follow
make[1]: *** [scripts/Makefile.vmlinux:35: vmlinux] Error 1
make: *** [Makefile:1249: vmlinux] Error 2

Re: [RFC PATCH 4/6] alpha: clean mm_cpumask when flushing TLBs

[Matt Turner]

On Fri, Jun 2, 2023 at 6:33 PM Matt Turner <mattst88@gmail.com> wrote:
>
> On Wed, May 24, 2023 at 1:18 PM Nicholas Piggin <npiggin@gmail.com> wrote:
> >
> > mm_cpumask is a map of the CPUs which must be IPIed to flush TLBs,
> > and/or IPIed to shootdown lazy TLB mms at exit time.
> >
> > When flushing TLBs on the CPU, trim it from mm_cpumask if the mm is not
> > currently active on the CPU. TLBs will have been flush, and the mm is
> > not active, so there is no more reason to get IPIs.
> >
> > Signed-off-by: Nicholas Piggin <npiggin@gmail.com>
> > ---
> >  arch/alpha/include/asm/tlbflush.h |  3 +++
> >  arch/alpha/kernel/smp.c           | 29 +++++++++++++++++++++++++++--
> >  2 files changed, 30 insertions(+), 2 deletions(-)
> >
> > diff --git a/arch/alpha/include/asm/tlbflush.h b/arch/alpha/include/asm/tlbflush.h
> > index 94dc37cf873a..7c4e719ac9e7 100644
> > --- a/arch/alpha/include/asm/tlbflush.h
> > +++ b/arch/alpha/include/asm/tlbflush.h
> > @@ -12,6 +12,7 @@
> >  #endif
> >
> >  extern void __load_new_mm_context(struct mm_struct *);
> > +extern void try_clear_mm_cpumask(struct mm_struct *);
> >
> >
> >  /* Use a few helper functions to hide the ugly broken ASN
> > @@ -106,6 +107,7 @@ static inline void flush_tlb_all(void)
> >  static inline void
> >  flush_tlb_mm(struct mm_struct *mm)
> >  {
> > +       try_clear_mm_cpumask(mm);
> >         if (mm == current->active_mm)
> >                 flush_tlb_current(mm);
> >         else
> > @@ -118,6 +120,7 @@ flush_tlb_page(struct vm_area_struct *vma, unsigned long addr)
> >  {
> >         struct mm_struct *mm = vma->vm_mm;
> >
> > +       try_clear_mm_cpumask(mm);
> >         if (mm == current->active_mm)
> >                 flush_tlb_current_page(mm, vma, addr);
> >         else
> > diff --git a/arch/alpha/kernel/smp.c b/arch/alpha/kernel/smp.c
> > index e436c056267d..d668b9d319af 100644
> > --- a/arch/alpha/kernel/smp.c
> > +++ b/arch/alpha/kernel/smp.c
> > @@ -610,6 +610,28 @@ smp_imb(void)
> >  }
> >  EXPORT_SYMBOL(smp_imb);
> >
> > +#define asn_locked() (cpu_data[smp_processor_id()].asn_lock)
> > +
> > +/*
> > + * If the mm_cpumask bit is cleared, the caller *must* flush the TLB for the
> > + * mm on this CPU. It is only cleared when the mm is not active, in which
> > + * case the flushing always performs flush_tlb_other that flushes everything.
> > + * If that changes in callers, they will have to arrange to always do a full
> > + * flush if mm_cpumask is cleared by this function.
> > + */
> > +void
> > +try_clear_mm_cpumask(struct mm_struct *mm)
> > +{
> > +       int cpu;
> > +
> > +       if (current->active_mm == mm || asn_locked())
> > +               return;
> > +
> > +       cpu = smp_processor_id();
> > +       if (cpumask_test_cpu(cpu, mm_cpumask(mm)))
> > +               cpumask_clear_cpu(cpu, mm_cpumask(mm));
> > +}
>
> Since this is implemented in smp.c, the function is not available in
> !CONFIG_SMP:
>
> ld: kernel/fork.o: in function `dup_mmap':
> (.text+0x1734): undefined reference to `try_clear_mm_cpumask'
> ld: (.text+0x1738): undefined reference to `try_clear_mm_cpumask'
> ld: mm/memory.o: in function `unmap_page_range':
> (.text+0x1934): undefined reference to `try_clear_mm_cpumask'
> ld: (.text+0x193c): undefined reference to `try_clear_mm_cpumask'
> ld: (.text+0x1a28): undefined reference to `try_clear_mm_cpumask'
> ld: mm/memory.o:(.text+0x1a30): more undefined references to
> `try_clear_mm_cpumask' follow
> make[1]: *** [scripts/Makefile.vmlinux:35: vmlinux] Error 1
> make: *** [Makefile:1249: vmlinux] Error 2

Anything I can do to help?

[RFC PATCH 5/6] alpha: enable MMU_LAZY_TLB_SHOOTDOWN

[Nicholas Piggin]

With the sticky mm_cpumask, and conversion to mmgrab_lazy_tlb(), alpha
meets the requirements for MMU_LAZY_TLB_SHOOTDOWN. Select it.

Signed-off-by: Nicholas Piggin <npiggin@gmail.com>
---
 arch/alpha/Kconfig      | 1 +
 arch/alpha/kernel/smp.c | 2 +-
 2 files changed, 2 insertions(+), 1 deletion(-)

diff --git a/arch/alpha/Kconfig b/arch/alpha/Kconfig
index a5c2b1aa46b0..8f9995d66345 100644
--- a/arch/alpha/Kconfig
+++ b/arch/alpha/Kconfig
@@ -35,6 +35,7 @@ config ALPHA
 	select OLD_SIGSUSPEND
 	select CPU_NO_EFFICIENT_FFS if !ALPHA_EV67
 	select MMU_GATHER_NO_RANGE
+	select MMU_LAZY_TLB_SHOOTDOWN
 	select SPARSEMEM_EXTREME if SPARSEMEM
 	select ZONE_DMA
 	help
diff --git a/arch/alpha/kernel/smp.c b/arch/alpha/kernel/smp.c
index d668b9d319af..73bbb81f336e 100644
--- a/arch/alpha/kernel/smp.c
+++ b/arch/alpha/kernel/smp.c
@@ -143,7 +143,7 @@ smp_callin(void)
 		alpha_mv.smp_callin();
 
 	/* All kernel threads share the same mm context.  */
-	mmgrab(&init_mm);
+	mmgrab_lazy_tlb(&init_mm);
 	current->active_mm = &init_mm;
 	cpumask_set_cpu(smp_processor_id(), mm_cpumask(&init_mm));
 
-- 
2.40.1

[RFC PATCH 6/6] alpha: shoot the lazy tlb mm when flushing TLBs

[Nicholas Piggin]

Since we're flushing the TLB, take the opportunity to switch away from
a lazy tlb mm if it's active, allowing the CPU to be taken out of
mm_cpumask and avoiding any further IPIs from TLB flushing or the final
lazy tlb mm shootdown.

This naturally combines lazy tlb mm shootdowns with the final exit TLB
flush IPIs, reducing the need for additional IPIs for the lazy tlb mm
shootdown.

Signed-off-by: Nicholas Piggin <npiggin@gmail.com>
---
 arch/alpha/kernel/smp.c | 11 ++++++++++-
 1 file changed, 10 insertions(+), 1 deletion(-)

diff --git a/arch/alpha/kernel/smp.c b/arch/alpha/kernel/smp.c
index 73bbb81f336e..cccd5ef721b7 100644
--- a/arch/alpha/kernel/smp.c
+++ b/arch/alpha/kernel/smp.c
@@ -624,9 +624,18 @@ try_clear_mm_cpumask(struct mm_struct *mm)
 {
 	int cpu;
 
-	if (current->active_mm == mm || asn_locked())
+	if (current->mm == mm || asn_locked())
 		return;
 
+	/*
+	 * Shoot the lazy tlb mm while we're here. This allows us to also
+	 * trim it out of the mm_cpumask if it was the active_mm, and has the
+	 * effect of avoiding final lazy shootdown IPIs in cleanup_lazy_tlbs(),
+	 * because the final TLB cleanup happens before that.
+	 */
+	if (current->active_mm == mm)
+		kthread_end_lazy_tlb_mm();
+
 	cpu = smp_processor_id();
 	if (cpumask_test_cpu(cpu, mm_cpumask(mm)))
 		cpumask_clear_cpu(cpu, mm_cpumask(mm));
-- 
2.40.1

Re: [RFC PATCH 0/6] Implement MMU_LAZY_TLB_SHOOTDOWN for alpha

[Linus Torvalds]

On Wed, May 24, 2023 at 10:18 AM Nicholas Piggin <npiggin@gmail.com> wrote:
>
> Any thoughts about the series would be interesting to hear.

Well, alpha is in a deathmatch with ia64 on "least relevant
architecture ever, and next to be removed".

There are no winners in that match.

I still have the alpha architecture manual somewhere here, but I
haven't touched any actual hardware in decades.

But I certainly don't see anything _wrong_ with your series from a
quick read-through.  It would be nice to hear that it works on real
hardware, of course, but from previous attempts, there's only a couple
of people that still occasionally run it.

               Linus

Re: [RFC PATCH 0/6] Implement MMU_LAZY_TLB_SHOOTDOWN for alpha

[Matt Turner]

On Wed, May 24, 2023 at 1:28 PM Linus Torvalds
<torvalds@linux-foundation.org> wrote:
>
> On Wed, May 24, 2023 at 10:18 AM Nicholas Piggin <npiggin@gmail.com> wrote:
> >
> > Any thoughts about the series would be interesting to hear.
>
> Well, alpha is in a deathmatch with ia64 on "least relevant
> architecture ever, and next to be removed".
>
> There are no winners in that match.
>
> I still have the alpha architecture manual somewhere here, but I
> haven't touched any actual hardware in decades.
>
> But I certainly don't see anything _wrong_ with your series from a
> quick read-through.  It would be nice to hear that it works on real
> hardware, of course, but from previous attempts, there's only a couple
> of people that still occasionally run it.

I'll be happy to test.

Re: [RFC PATCH 0/6] Implement MMU_LAZY_TLB_SHOOTDOWN for alpha

[Nicholas Piggin]

On Thu May 25, 2023 at 3:52 AM AEST, Matt Turner wrote:
> On Wed, May 24, 2023 at 1:28 PM Linus Torvalds
> <torvalds@linux-foundation.org> wrote:
> >
> > On Wed, May 24, 2023 at 10:18 AM Nicholas Piggin <npiggin@gmail.com> wrote:
> > >
> > > Any thoughts about the series would be interesting to hear.
> >
> > Well, alpha is in a deathmatch with ia64 on "least relevant
> > architecture ever, and next to be removed".
> >
> > There are no winners in that match.

Except that I don't get yelled at by so many people when things break.
Why do you think I work on powerpc? (/s, kind of).

At least it's a reference, hopefully we can get a bit more interest. If
we get a few converted then we could find some common patterns and pull
them into generic code or at least a simpler recipe as we try to convert
everything and just make this the only way it's implemented. Of course
we need the big archs on board at some point.

> > I still have the alpha architecture manual somewhere here, but I
> > haven't touched any actual hardware in decades.
> >
> > But I certainly don't see anything _wrong_ with your series from a
> > quick read-through.  It would be nice to hear that it works on real
> > hardware, of course, but from previous attempts, there's only a couple
> > of people that still occasionally run it.
>
> I'll be happy to test.

Thanks Matt, I'll put a git tree up and ping you.

Thanks,
Nick