The Linux Kernel Mailing List
 help / color / mirror / Atom feed
* [PATCH v6 RESEND 1/3] x86/mm/tlb: Make enter_lazy_tlb() always inline on x86
@ 2026-02-09 16:23 Xie Yuanbin
  2026-02-09 16:23 ` [PATCH v6 RESEND 2/3] sched: Make raw_spin_rq_unlock() inline Xie Yuanbin
                   ` (2 more replies)
  0 siblings, 3 replies; 9+ messages in thread
From: Xie Yuanbin @ 2026-02-09 16:23 UTC (permalink / raw)
  To: peterz, tglx, riel, segher, david, hpa, arnd, anna-maria, mingo,
	juri.lelli, vincent.guittot, dietmar.eggemann, rostedt, bsegall,
	mgorman, vschneid, bp, dave.hansen, luto, houwenlong.hwl,
	frederic, akpm, lorenzo.stoakes, bhe, ryan.roberts, urezki, nysal,
	max.kellermann
  Cc: linux-kernel, x86, Xie Yuanbin

enter_lazy_tlb() on x86 is short enough, and is called in context
switching, which is the hot code path.

Make enter_lazy_tlb() always inline on x86 to optimize performance.

Cc: Thomas Gleixner <tglx@kernel.org>
Cc: David Hildenbrand (Red Hat) <david@kernel.org>
Reviewed-by: Rik van Riel <riel@surriel.com>
Signed-off-by: Xie Yuanbin <qq570070308@gmail.com>
---
 arch/x86/include/asm/mmu_context.h | 23 ++++++++++++++++++++++-
 arch/x86/mm/tlb.c                  | 21 ---------------------
 2 files changed, 22 insertions(+), 22 deletions(-)

diff --git a/arch/x86/include/asm/mmu_context.h b/arch/x86/include/asm/mmu_context.h
index 1acafb1c6a93..ec3f9bebcf7b 100644
--- a/arch/x86/include/asm/mmu_context.h
+++ b/arch/x86/include/asm/mmu_context.h
@@ -136,8 +136,29 @@ static inline void mm_reset_untag_mask(struct mm_struct *mm)
 }
 #endif
 
+/*
+ * Please ignore the name of this function.  It should be called
+ * switch_to_kernel_thread().
+ *
+ * enter_lazy_tlb() is a hint from the scheduler that we are entering a
+ * kernel thread or other context without an mm.  Acceptable implementations
+ * include doing nothing whatsoever, switching to init_mm, or various clever
+ * lazy tricks to try to minimize TLB flushes.
+ *
+ * The scheduler reserves the right to call enter_lazy_tlb() several times
+ * in a row.  It will notify us that we're going back to a real mm by
+ * calling switch_mm_irqs_off().
+ */
+#ifndef MODULE
+static __always_inline void enter_lazy_tlb(struct mm_struct *mm, struct task_struct *tsk)
+{
+	if (this_cpu_read(cpu_tlbstate.loaded_mm) == &init_mm)
+		return;
+
+	this_cpu_write(cpu_tlbstate_shared.is_lazy, true);
+}
+#endif
 #define enter_lazy_tlb enter_lazy_tlb
-extern void enter_lazy_tlb(struct mm_struct *mm, struct task_struct *tsk);
 
 extern void mm_init_global_asid(struct mm_struct *mm);
 extern void mm_free_global_asid(struct mm_struct *mm);
diff --git a/arch/x86/mm/tlb.c b/arch/x86/mm/tlb.c
index 621e09d049cb..af43d177087e 100644
--- a/arch/x86/mm/tlb.c
+++ b/arch/x86/mm/tlb.c
@@ -971,27 +971,6 @@ void switch_mm_irqs_off(struct mm_struct *unused, struct mm_struct *next,
 	}
 }
 
-/*
- * Please ignore the name of this function.  It should be called
- * switch_to_kernel_thread().
- *
- * enter_lazy_tlb() is a hint from the scheduler that we are entering a
- * kernel thread or other context without an mm.  Acceptable implementations
- * include doing nothing whatsoever, switching to init_mm, or various clever
- * lazy tricks to try to minimize TLB flushes.
- *
- * The scheduler reserves the right to call enter_lazy_tlb() several times
- * in a row.  It will notify us that we're going back to a real mm by
- * calling switch_mm_irqs_off().
- */
-void enter_lazy_tlb(struct mm_struct *mm, struct task_struct *tsk)
-{
-	if (this_cpu_read(cpu_tlbstate.loaded_mm) == &init_mm)
-		return;
-
-	this_cpu_write(cpu_tlbstate_shared.is_lazy, true);
-}
-
 /*
  * Using a temporary mm allows to set temporary mappings that are not accessible
  * by other CPUs. Such mappings are needed to perform sensitive memory writes
-- 
2.51.0


^ permalink raw reply related	[flat|nested] 9+ messages in thread
end of thread, other threads:[~2026-02-12 15:20 UTC | newest]

Thread overview: 9+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2026-02-09 16:23 [PATCH v6 RESEND 1/3] x86/mm/tlb: Make enter_lazy_tlb() always inline on x86 Xie Yuanbin
2026-02-09 16:23 ` [PATCH v6 RESEND 2/3] sched: Make raw_spin_rq_unlock() inline Xie Yuanbin
2026-02-09 16:23 ` [PATCH v6 RESEND 3/3] sched/core: Make finish_task_switch() and its subfunctions always inline Xie Yuanbin
2026-02-09 19:24 ` [PATCH v6 RESEND 1/3] x86/mm/tlb: Make enter_lazy_tlb() always inline on x86 Dave Hansen
2026-02-10 15:32   ` Xie Yuanbin
2026-02-10 16:07     ` Dave Hansen
2026-02-11 15:36       ` Xie Yuanbin
2026-02-11 18:01         ` Dave Hansen
2026-02-12 15:19           ` Xie Yuanbin

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox