[PATCH v2] x86/msr: Add write msr notrace to avoid the debug codes splash

[PATCH v2] x86/msr: Add write msr notrace to avoid the debug codes splash

[Wanpeng Li]

 ===============================
 [ INFO: suspicious RCU usage. ]
 4.8.0+ #24 Not tainted
 -------------------------------
 ./arch/x86/include/asm/msr-trace.h:47 suspicious rcu_dereference_check() usage!

 other info that might help us debug this:

 RCU used illegally from idle CPU!
 rcu_scheduler_active = 1, debug_locks = 0
 RCU used illegally from extended quiescent state!
 no locks held by swapper/1/0.

  [<ffffffff9d492b95>] do_trace_write_msr+0x135/0x140
  [<ffffffff9d06f860>] native_write_msr+0x20/0x30
  [<ffffffff9d065fad>] native_apic_msr_eoi_write+0x1d/0x30
  [<ffffffff9d05bd1d>] smp_reschedule_interrupt+0x1d/0x30
  [<ffffffff9d8daec6>] reschedule_interrupt+0x96/0xa0

As Peterz pointed out:

| The thing is, many many smp_reschedule_interrupt() invocations don't
| actually execute anything much at all and are only send to tickle the
| return to user path (which does the actual preemption).

This patch add write msr notrace to avoid the debug codes splash.

Suggested-by: Peter Zijlstra <peterz@infradead.org>
Suggested-by: Paolo Bonzini <pbonzini@redhat.com>
Cc: Ingo Molnar <mingo@kernel.org>
Cc: Mike Galbraith <efault@gmx.de>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Paolo Bonzini <pbonzini@redhat.com>
Signed-off-by: Wanpeng Li <wanpeng.li@hotmail.com>
---
v1 -> v2:
 * add write msr notrace to avoid debug codes splash instead of slowdown a very frequent interrupt

 arch/x86/include/asm/apic.h |  4 ++--
 arch/x86/include/asm/msr.h  | 15 +++++++++++++++
 arch/x86/kernel/kvm.c       |  2 +-
 3 files changed, 18 insertions(+), 3 deletions(-)

diff --git a/arch/x86/include/asm/apic.h b/arch/x86/include/asm/apic.h
index f5aaf6c..d38bbe8 100644
--- a/arch/x86/include/asm/apic.h
+++ b/arch/x86/include/asm/apic.h
@@ -174,7 +174,7 @@ static inline void disable_local_APIC(void) { }
 static inline void lapic_update_tsc_freq(void) { }
 #endif /* !CONFIG_X86_LOCAL_APIC */
 
-#ifdef CONFIG_X86_X2APIC
+#if defined CONFIG_X86_X2APIC || defined CONFIG_KVM_GUEST
 /*
  * Make previous memory operations globally visible before
  * sending the IPI through x2apic wrmsr. We need a serializing instruction or
@@ -196,7 +196,7 @@ static inline void native_apic_msr_write(u32 reg, u32 v)
 
 static inline void native_apic_msr_eoi_write(u32 reg, u32 v)
 {
-	wrmsr(APIC_BASE_MSR + (APIC_EOI >> 4), APIC_EOI_ACK, 0);
+	wrmsr_notrace(APIC_BASE_MSR + (APIC_EOI >> 4), APIC_EOI_ACK, 0);
 }
 
 static inline u32 native_apic_msr_read(u32 reg)
diff --git a/arch/x86/include/asm/msr.h b/arch/x86/include/asm/msr.h
index b5fee97..afbb221 100644
--- a/arch/x86/include/asm/msr.h
+++ b/arch/x86/include/asm/msr.h
@@ -127,6 +127,21 @@ notrace static inline void native_write_msr(unsigned int msr,
 }
 
 /* Can be uninlined because referenced by paravirt */
+notrace static inline void native_write_msr_notrace(unsigned int msr,
+					    unsigned low, unsigned high)
+{
+	asm volatile("1: wrmsr\n"
+		     "2:\n"
+		     _ASM_EXTABLE_HANDLE(1b, 2b, ex_handler_wrmsr_unsafe)
+		     : : "c" (msr), "a"(low), "d" (high) : "memory");
+}
+
+static inline void wrmsr_notrace(unsigned msr, unsigned low, unsigned high)
+{
+	native_write_msr_notrace(msr, low, high);
+}
+
+/* Can be uninlined because referenced by paravirt */
 notrace static inline int native_write_msr_safe(unsigned int msr,
 					unsigned low, unsigned high)
 {
diff --git a/arch/x86/kernel/kvm.c b/arch/x86/kernel/kvm.c
index edbbfc8..61cc6a5 100644
--- a/arch/x86/kernel/kvm.c
+++ b/arch/x86/kernel/kvm.c
@@ -319,7 +319,7 @@ static void kvm_guest_apic_eoi_write(u32 reg, u32 val)
 	 */
 	if (__test_and_clear_bit(KVM_PV_EOI_BIT, this_cpu_ptr(&kvm_apic_eoi)))
 		return;
-	apic_write(APIC_EOI, APIC_EOI_ACK);
+	native_apic_msr_eoi_write(APIC_EOI, APIC_EOI_ACK);
 }
 
 static void kvm_guest_cpu_init(void)
-- 
1.9.1

Re: [PATCH v2] x86/msr: Add write msr notrace to avoid the debug codes splash

[Ingo Molnar]

* Wanpeng Li <kernellwp@gmail.com> wrote:

>  ===============================
>  [ INFO: suspicious RCU usage. ]
>  4.8.0+ #24 Not tainted
>  -------------------------------
>  ./arch/x86/include/asm/msr-trace.h:47 suspicious rcu_dereference_check() usage!
> 
>  other info that might help us debug this:
> 
>  RCU used illegally from idle CPU!
>  rcu_scheduler_active = 1, debug_locks = 0
>  RCU used illegally from extended quiescent state!
>  no locks held by swapper/1/0.
> 
>   [<ffffffff9d492b95>] do_trace_write_msr+0x135/0x140
>   [<ffffffff9d06f860>] native_write_msr+0x20/0x30
>   [<ffffffff9d065fad>] native_apic_msr_eoi_write+0x1d/0x30
>   [<ffffffff9d05bd1d>] smp_reschedule_interrupt+0x1d/0x30
>   [<ffffffff9d8daec6>] reschedule_interrupt+0x96/0xa0
> 
> As Peterz pointed out:
> 
> | The thing is, many many smp_reschedule_interrupt() invocations don't
> | actually execute anything much at all and are only send to tickle the
> | return to user path (which does the actual preemption).
> 
> This patch add write msr notrace to avoid the debug codes splash.
> 
> Suggested-by: Peter Zijlstra <peterz@infradead.org>
> Suggested-by: Paolo Bonzini <pbonzini@redhat.com>
> Cc: Ingo Molnar <mingo@kernel.org>
> Cc: Mike Galbraith <efault@gmx.de>
> Cc: Peter Zijlstra <peterz@infradead.org>
> Cc: Thomas Gleixner <tglx@linutronix.de>
> Cc: Paolo Bonzini <pbonzini@redhat.com>
> Signed-off-by: Wanpeng Li <wanpeng.li@hotmail.com>
> ---
> v1 -> v2:
>  * add write msr notrace to avoid debug codes splash instead of slowdown a very frequent interrupt
> 
>  arch/x86/include/asm/apic.h |  4 ++--
>  arch/x86/include/asm/msr.h  | 15 +++++++++++++++
>  arch/x86/kernel/kvm.c       |  2 +-
>  3 files changed, 18 insertions(+), 3 deletions(-)

Could you please do this on top of -tip and also include the revert of:

                        #  1ec6ec14a294 x86/smp: Add irq_enter/exit() in smp_reschedule_interrupt()

in your v3 patch, because I'd rather avoid rebasing x86/urgent.

Thanks,

	Ingo

Re: [PATCH v2] x86/msr: Add write msr notrace to avoid the debug codes splash

[Wanpeng Li]

2016-10-18 17:24 GMT+08:00 Ingo Molnar <mingo@kernel.org>:
>
> * Wanpeng Li <kernellwp@gmail.com> wrote:
>
>>  ===============================
>>  [ INFO: suspicious RCU usage. ]
>>  4.8.0+ #24 Not tainted
>>  -------------------------------
>>  ./arch/x86/include/asm/msr-trace.h:47 suspicious rcu_dereference_check() usage!
>>
>>  other info that might help us debug this:
>>
>>  RCU used illegally from idle CPU!
>>  rcu_scheduler_active = 1, debug_locks = 0
>>  RCU used illegally from extended quiescent state!
>>  no locks held by swapper/1/0.
>>
>>   [<ffffffff9d492b95>] do_trace_write_msr+0x135/0x140
>>   [<ffffffff9d06f860>] native_write_msr+0x20/0x30
>>   [<ffffffff9d065fad>] native_apic_msr_eoi_write+0x1d/0x30
>>   [<ffffffff9d05bd1d>] smp_reschedule_interrupt+0x1d/0x30
>>   [<ffffffff9d8daec6>] reschedule_interrupt+0x96/0xa0
>>
>> As Peterz pointed out:
>>
>> | The thing is, many many smp_reschedule_interrupt() invocations don't
>> | actually execute anything much at all and are only send to tickle the
>> | return to user path (which does the actual preemption).
>>
>> This patch add write msr notrace to avoid the debug codes splash.
>>
>> Suggested-by: Peter Zijlstra <peterz@infradead.org>
>> Suggested-by: Paolo Bonzini <pbonzini@redhat.com>
>> Cc: Ingo Molnar <mingo@kernel.org>
>> Cc: Mike Galbraith <efault@gmx.de>
>> Cc: Peter Zijlstra <peterz@infradead.org>
>> Cc: Thomas Gleixner <tglx@linutronix.de>
>> Cc: Paolo Bonzini <pbonzini@redhat.com>
>> Signed-off-by: Wanpeng Li <wanpeng.li@hotmail.com>
>> ---
>> v1 -> v2:
>>  * add write msr notrace to avoid debug codes splash instead of slowdown a very frequent interrupt
>>
>>  arch/x86/include/asm/apic.h |  4 ++--
>>  arch/x86/include/asm/msr.h  | 15 +++++++++++++++
>>  arch/x86/kernel/kvm.c       |  2 +-
>>  3 files changed, 18 insertions(+), 3 deletions(-)
>
> Could you please do this on top of -tip and also include the revert of:
>
>                         #  1ec6ec14a294 x86/smp: Add irq_enter/exit() in smp_reschedule_interrupt()
>
> in your v3 patch, because I'd rather avoid rebasing x86/urgent.

It seems that v2 still doesn't handle paravirt part correctly, I will
revert 1ec6ec14a294 in v3 when I figure it out.

Regards,
Wanpeng Li