KVM_GUEST support breaks page fault tracing

KVM_GUEST support breaks page fault tracing

[Dave Hansen]

I noticed on some of my systems that page fault tracing doesn't work:

	cd /sys/kernel/debug/tracing
	echo 1 > events/exceptions/enable
	cat trace;
	# nothing shows up

I eventually traced it down to CONFIG_KVM_GUEST.  At least in a KVM VM,
enabling that option breaks page fault tracing, and disabling fixes it.
 I tried on some old kernels and this does not appear to be a
regression: it never worked.

Anybody have any theories about what is going on?

I'm also having problems with the page fault tracing on bare metal (same
symptom of no trace output).  I'm unsure if it's related.

Re: KVM_GUEST support breaks page fault tracing

[Thomas Gleixner]

On Thu, 8 May 2014, Dave Hansen wrote:

You forgot to cc Steven :)

> I noticed on some of my systems that page fault tracing doesn't work:
> 
> 	cd /sys/kernel/debug/tracing
> 	echo 1 > events/exceptions/enable
> 	cat trace;
> 	# nothing shows up
> 
> I eventually traced it down to CONFIG_KVM_GUEST.  At least in a KVM VM,
> enabling that option breaks page fault tracing, and disabling fixes it.
>  I tried on some old kernels and this does not appear to be a
> regression: it never worked.
> 
> Anybody have any theories about what is going on?
> 
> I'm also having problems with the page fault tracing on bare metal (same
> symptom of no trace output).  I'm unsure if it's related.
> 

Re: KVM_GUEST support breaks page fault tracing

[Dave Hansen]

[-- Attachment #1: Type: text/plain, Size: 1068 bytes --]

On 05/08/2014 03:24 PM, Thomas Gleixner wrote:
>> > I noticed on some of my systems that page fault tracing doesn't work:
>> > 
>> > 	cd /sys/kernel/debug/tracing
>> > 	echo 1 > events/exceptions/enable
>> > 	cat trace;
>> > 	# nothing shows up
>> > 
>> > I eventually traced it down to CONFIG_KVM_GUEST.  At least in a KVM VM,
>> > enabling that option breaks page fault tracing, and disabling fixes it.
>> >  I tried on some old kernels and this does not appear to be a
>> > regression: it never worked.
>> > 
>> > Anybody have any theories about what is going on?

Looks like the KVM code calls do_page_fault() directly:

> dotraplinkage void __kprobes
> do_async_page_fault(struct pt_regs *regs, unsigned long error_code)
> {
>         enum ctx_state prev_state;
> 
>         switch (kvm_read_and_reset_pf_reason()) {
>         default:
>                 do_page_fault(regs, error_code);
>                 break;
>         case KVM_PV_REASON_PAGE_NOT_PRESENT:

That seems to explain my problems in a VM.  Any objections to doing
something like the attached patch?

[-- Attachment #2: muck-with-kvm-guest-code.patch --]
[-- Type: text/x-patch, Size: 1421 bytes --]



---

 b/arch/x86/include/asm/traps.h |    5 +++++
 b/arch/x86/kernel/kvm.c        |    2 +-
 2 files changed, 6 insertions(+), 1 deletion(-)

diff -puN arch/x86/kernel/kvm.c~muck-with-kvm-guest-code arch/x86/kernel/kvm.c
--- a/arch/x86/kernel/kvm.c~muck-with-kvm-guest-code	2014-05-08 15:03:24.358110394 -0700
+++ b/arch/x86/kernel/kvm.c	2014-05-08 16:03:56.765302785 -0700
@@ -259,7 +259,7 @@ do_async_page_fault(struct pt_regs *regs
 
 	switch (kvm_read_and_reset_pf_reason()) {
 	default:
-		do_page_fault(regs, error_code);
+		trace_do_page_fault(regs, error_code);
 		break;
 	case KVM_PV_REASON_PAGE_NOT_PRESENT:
 		/* page is swapped out by the host. */
diff -puN arch/x86/include/asm/traps.h~muck-with-kvm-guest-code arch/x86/include/asm/traps.h
--- a/arch/x86/include/asm/traps.h~muck-with-kvm-guest-code	2014-05-08 16:02:14.873675048 -0700
+++ b/arch/x86/include/asm/traps.h	2014-05-08 16:03:06.519020810 -0700
@@ -74,6 +74,11 @@ dotraplinkage void do_general_protection
 dotraplinkage void do_page_fault(struct pt_regs *, unsigned long);
 #ifdef CONFIG_TRACING
 dotraplinkage void trace_do_page_fault(struct pt_regs *, unsigned long);
+#else
+static inline void trace_do_page_fault(struct pt_regs *regs, unsigned long error)
+{
+	do_page_fault(regs, error);
+}
 #endif
 dotraplinkage void do_spurious_interrupt_bug(struct pt_regs *, long);
 dotraplinkage void do_coprocessor_error(struct pt_regs *, long);
_

Re: KVM_GUEST support breaks page fault tracing

[Steven Rostedt]

On Thu, 08 May 2014 16:05:12 -0700
Dave Hansen <dave.hansen@intel.com> wrote:

Added Seiji and hpa

> On 05/08/2014 03:24 PM, Thomas Gleixner wrote:
> >> > I noticed on some of my systems that page fault tracing doesn't work:
> >> > 
> >> > 	cd /sys/kernel/debug/tracing
> >> > 	echo 1 > events/exceptions/enable
> >> > 	cat trace;
> >> > 	# nothing shows up
> >> > 
> >> > I eventually traced it down to CONFIG_KVM_GUEST.  At least in a KVM VM,
> >> > enabling that option breaks page fault tracing, and disabling fixes it.
> >> >  I tried on some old kernels and this does not appear to be a
> >> > regression: it never worked.
> >> > 
> >> > Anybody have any theories about what is going on?

Has to do with the IDT tricks done to keep page faults from getting
zero overhead when tracing is disabled. It sets up two types of traps.
One that encompasses the other. When all of the trap tracepoints are
disabled, the traps are called directly. But when any of them are
enabled, the IDT gets switched to have the traps point to exceptions
that trace the entry and exit of the exception.

see commits 25c74b10bacead867478480170083f69cfc0db48 and
d34603b07c4255b2b00a546d34f297ccd50ae4c6

> 
> Looks like the KVM code calls do_page_fault() directly:
> 
> > dotraplinkage void __kprobes
> > do_async_page_fault(struct pt_regs *regs, unsigned long error_code)
> > {
> >         enum ctx_state prev_state;
> > 
> >         switch (kvm_read_and_reset_pf_reason()) {
> >         default:
> >                 do_page_fault(regs, error_code);
> >                 break;
> >         case KVM_PV_REASON_PAGE_NOT_PRESENT:
> 
> That seems to explain my problems in a VM.  Any objections to doing
> something like the attached patch?

As your patch showed up as an attachment, I couldn't include it in my
reply. But sure, that may work. But you could also play tricks to keep
the overhead off when tracing is disabled like this one:

[ I booted this, but didn't test any kvm guests with it ]

Signed-off-by: Steven Rostedt <rostedt@goodmis.org>


diff --git a/arch/x86/include/asm/traps.h b/arch/x86/include/asm/traps.h
index 58d66fe..835ba5b 100644
--- a/arch/x86/include/asm/traps.h
+++ b/arch/x86/include/asm/traps.h
@@ -39,6 +39,8 @@ asmlinkage void simd_coprocessor_error(void);
 
 #ifdef CONFIG_TRACING
 asmlinkage void trace_page_fault(void);
+asmlinkage void trace_async_page_fault(void);
+#define trace_async_page_fault async_page_fault
 #define trace_divide_error divide_error
 #define trace_bounds bounds
 #define trace_invalid_op invalid_op
@@ -51,7 +53,6 @@ asmlinkage void trace_page_fault(void);
 #define trace_coprocessor_error coprocessor_error
 #define trace_alignment_check alignment_check
 #define trace_simd_coprocessor_error simd_coprocessor_error
-#define trace_async_page_fault async_page_fault
 #endif
 
 dotraplinkage void do_divide_error(struct pt_regs *, long);
diff --git a/arch/x86/kernel/entry_32.S b/arch/x86/kernel/entry_32.S
index a2a4f46..26e370a 100644
--- a/arch/x86/kernel/entry_32.S
+++ b/arch/x86/kernel/entry_32.S
@@ -1451,7 +1451,17 @@ ENTRY(async_page_fault)
 	jmp error_code
 	CFI_ENDPROC
 END(async_page_fault)
+
+#ifdef CONFIG_TRACING
+ENTRY(trace_async_page_fault)
+	RING0_EC_FRAME
+	ASM_CLAC
+	pushl_cfi $trace_do_async_page_fault
+	jmp error_code
+	CFI_ENDPROC
+END(async_page_fault)
 #endif
+#endif /* CONFIG_KVM_GUEST */
 
 /*
  * End of kprobes section
diff --git a/arch/x86/kernel/entry_64.S b/arch/x86/kernel/entry_64.S
index 3db806d..e1f8926 100644
--- a/arch/x86/kernel/entry_64.S
+++ b/arch/x86/kernel/entry_64.S
@@ -1289,7 +1289,7 @@ errorentry xen_stack_segment do_stack_segment
 errorentry general_protection do_general_protection
 trace_errorentry page_fault do_page_fault
 #ifdef CONFIG_KVM_GUEST
-errorentry async_page_fault do_async_page_fault
+trace_errorentry async_page_fault do_async_page_fault
 #endif
 #ifdef CONFIG_X86_MCE
 paranoidzeroentry machine_check *machine_check_vector(%rip)
diff --git a/arch/x86/kernel/kvm.c b/arch/x86/kernel/kvm.c
index 0331cb3..baef30e 100644
--- a/arch/x86/kernel/kvm.c
+++ b/arch/x86/kernel/kvm.c
@@ -252,14 +252,18 @@ u32 kvm_read_and_reset_pf_reason(void)
 }
 EXPORT_SYMBOL_GPL(kvm_read_and_reset_pf_reason);
 
-dotraplinkage void __kprobes
-do_async_page_fault(struct pt_regs *regs, unsigned long error_code)
+enum {
+	KVM_PV_DO_PAGE_FAULT,
+	KVM_PV_HANDLED,
+};
+
+static int kvm_pv_page_fault(struct pt_regs *regs, unsigned long error_code)
 {
 	enum ctx_state prev_state;
 
 	switch (kvm_read_and_reset_pf_reason()) {
 	default:
-		do_page_fault(regs, error_code);
+		return KVM_PV_DO_PAGE_FAULT;
 		break;
 	case KVM_PV_REASON_PAGE_NOT_PRESENT:
 		/* page is swapped out by the host. */
@@ -275,7 +279,34 @@ do_async_page_fault(struct pt_regs *regs, unsigned long error_code)
 		rcu_irq_exit();
 		break;
 	}
+	return KVM_PV_HANDLED;
+}
+
+dotraplinkage void __kprobes
+do_async_page_fault(struct pt_regs *regs, unsigned long error_code)
+{
+	int ret;
+
+	ret = kvm_pv_page_fault(regs, error_code);
+	if (ret == KVM_PV_HANDLED)
+		return;
+
+	do_page_fault(regs, error_code);
+}
+
+#ifdef CONFIG_TRACING
+dotraplinkage void __kprobes notrace
+trace_do_async_page_fault(struct pt_regs *regs, unsigned long error_code)
+{
+	int ret;
+
+	ret = kvm_pv_page_fault(regs, error_code);
+	if (ret == KVM_PV_HANDLED)
+		return;
+
+	trace_do_page_fault(regs, error_code);
 }
+#endif /* CONFIG_TRACING */
 
 static void __init paravirt_ops_setup(void)
 {

Re: KVM_GUEST support breaks page fault tracing

[Dave Hansen]

On 05/08/2014 04:45 PM, Steven Rostedt wrote:
> As your patch showed up as an attachment, I couldn't include it in my
> reply. But sure, that may work. But you could also play tricks to keep
> the overhead off when tracing is disabled like this one:
...

How important is it to have zero-overhead?  This bug was introduced by
trying to do just that, and I'm worried about having an explosion of so
many paths to get in to the page fault code.  The overhead in this case
is just a few noops, right?

I guess that's a question for the KVM folks.  Do you guys prefer zero
overhead or simpler code?