[PATCH 2/2] ftrace: fast path for do_ftrace_mod_code()

[Lai Jiangshan <laijs@cn.fujitsu.com>]

Lai Jiangshan wrote:
> 
> Subject: [PATCH 1/2] ftrace: protect running nmi (V2)
> 


Subject: [PATCH 2/2] ftrace: fast path for do_ftrace_mod_code()

commit 90c7ac49aa819feb9433b5310089fca6399881c0
adds a fast path to prevent NMI lockup.

But the previous patch "protect executing nmi" changes
do_ftrace_mod_code()'s implementation, we still need fix to
prevent NMI lockup by adding a fast path.

A difference between this fix and 90c7ac49aa819feb9433b5310089fca6399881c0
is that: We kill any new writers in spite of probe_kernel_write()
success or fail, not only when probe_kernel_write() fail.
(When probe_kernel_write() success, new writers do not need to do
it again.)

Signed-off-by: Lai Jiangshan <laijs@cn.fujitsu.com>
---
diff --git a/arch/x86/kernel/ftrace.c b/arch/x86/kernel/ftrace.c
index 699a1c0..61cb520 100644
--- a/arch/x86/kernel/ftrace.c
+++ b/arch/x86/kernel/ftrace.c
@@ -98,6 +98,7 @@ static unsigned char *ftrace_call_replace(unsigned long ip, unsigned long addr)
 #define MOD_CODE_WRITE_FLAG (1 << 31)	/* set when NMI should do the write */
 static atomic_t nmi_running = ATOMIC_INIT(0);
 static int mod_code_status;		/* holds return value of text write */
+static int mod_code_no_write = 1;	/* set when NMI not need do the write */
 static void *mod_code_ip;		/* holds the IP to write to */
 static void *mod_code_newcode;		/* holds the text to write to the IP */
 
@@ -124,14 +125,19 @@ static void ftrace_mod_code(void)
 	 */
 	mod_code_status = probe_kernel_write(mod_code_ip, mod_code_newcode,
 					     MCOUNT_INSN_SIZE);
+
+	smb_wmb();
+	mod_code_no_write = 1;
 }
 
 void ftrace_nmi_enter(void)
 {
 	if (atomic_inc_return(&nmi_running) & MOD_CODE_WRITE_FLAG) {
 		smp_rmb();
-		ftrace_mod_code();
-		atomic_inc(&nmi_update_count);
+		if (!mod_code_no_write) {
+			ftrace_mod_code();
+			atomic_inc(&nmi_update_count);
+		}
 	}
 	/* Must have previous changes seen before executions */
 	smp_mb();
@@ -161,6 +167,7 @@ do_ftrace_mod_code(unsigned long ip, void *new_code)
 {
 	mod_code_ip = (void *)ip;
 	mod_code_newcode = new_code;
+	mod_code_no_write = 0;
 
 	/*
 	 * The previous variables need to be visible before NMIs sees
@@ -173,7 +180,8 @@ do_ftrace_mod_code(unsigned long ip, void *new_code)
 	/* Make sure all running NMIs have finished before we write the code */
 	smp_mb();
 
-	ftrace_mod_code();
+	if (!mod_code_no_write)
+		ftrace_mod_code();
 
 	/* Make sure the write happens before clearing the bit */
 	smp_mb();