[PATCH 6/6] x86/mm/kaiser: Optimize __native_flush_tlb

[Peter Zijlstra <peterz@infradead.org>]

[-- Attachment #1: peterz-kaiser-opt-tlb.patch --]
[-- Type: text/plain, Size: 2069 bytes --]

Now that we have lazy user asid flushing, use that even if we have
INVPCID. Even if INVPCID would not be slower than a flushing CR3 write
(it is) this allows folding multiple user flushes.

Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
---
 arch/x86/include/asm/tlbflush.h |   38 ++++++++++++++------------------------
 1 file changed, 14 insertions(+), 24 deletions(-)

--- a/arch/x86/include/asm/tlbflush.h
+++ b/arch/x86/include/asm/tlbflush.h
@@ -377,33 +377,23 @@ static inline void flush_user_asid(u16 a
 
 static inline void __native_flush_tlb(void)
 {
-	if (!cpu_feature_enabled(X86_FEATURE_INVPCID)) {
-		flush_user_asid(this_cpu_read(cpu_tlbstate.loaded_mm_asid));
+	flush_user_asid(this_cpu_read(cpu_tlbstate.loaded_mm_asid));
 
-		/*
-		 * If current->mm == NULL then we borrow a mm
-		 * which may change during a task switch and
-		 * therefore we must not be preempted while we
-		 * write CR3 back:
-		 */
-		preempt_disable();
-		native_write_cr3(__native_read_cr3());
-		preempt_enable();
-		/*
-		 * Does not need tlb_flush_shared_nonglobals()
-		 * since the CR3 write without PCIDs flushes all
-		 * non-globals.
-		 */
-		return;
-	}
 	/*
-	 * We are no longer using globals with KAISER, so a
-	 * "nonglobals" flush would work too. But, this is more
-	 * conservative.
-	 *
-	 * Note, this works with CR4.PCIDE=0 or 1.
+	 * If current->mm == NULL then we borrow a mm
+	 * which may change during a task switch and
+	 * therefore we must not be preempted while we
+	 * write CR3 back:
 	 */
-	invpcid_flush_all();
+	preempt_disable();
+	native_write_cr3(__native_read_cr3());
+	preempt_enable();
+	/*
+	 * Does not need tlb_flush_shared_nonglobals()
+	 * since the CR3 write without PCIDs flushes all
+	 * non-globals.
+	 */
+	return;
 }
 
 static inline void __native_flush_tlb_global_irq_disabled(void)


--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>