[PATCH] x86: Skip latched NMIs on early boot in kdump

[PATCH] x86: Skip latched NMIs on early boot in kdump

[Don Zickus]

A customer generated an external NMI using their iLO to test kdump worked.
Unfortunately, the machine hung.  Disabling the nmi_watchdog made things work.

I speculated the external NMI fired, caused the machine to panic (as expected)
and the perf NMI from the watchdog came in and was latched.  My guess was this
somehow caused the hang.

Debugging this with outb's and debug_putstr, I learned the following

- the machine hung during the first memcpy in copy_bootdata (in
arch/x86/kernel/head64.c)
- early_make_pgtable was called during this memcpy
- after early_make_pgtable, an exception vector 2 (NMI) came in
- the IP of this vector was in copy_bootdata's range
- because there was no fixup associated with this IP, the machine
  is sitting in a 'hlt' instruction (in arch/x86/kernel/head_64.S)

(copy and paste from arch/x86/kernel/head_64.S)
/* This is global to keep gas from relaxing the jumps */
ENTRY(early_idt_handler)

<snip>

        cmpl $14,72(%rsp)       # Page fault?
        jnz 10f
        GET_CR2_INTO(%rdi)      # can clobber any volatile register if pv
        call early_make_pgtable
        andl %eax,%eax
        jz 20f                  # All good

10:
        leaq 88(%rsp),%rdi      # Pointer to %rip
        call early_fixup_exception
        andl %eax,%eax
        jnz 20f                 # Found an exception entry

11:
	<snip>
1:      hlt
	^^^^^^^^^^^^ sitting here

        jmp 1b

I added the below hack, which says if the exception is an NMI just return and
things seem to work.

Now, I don't expect this to be the correct solution.  Nor do I fully understand
what this early boot code is doing, so hopefully folks wiser than me can
provide me a better patch to test. :-)

I also do not fully understand why the latched NMI is not happening immediately
after the load idt call or why it comes after a page fault (the
early_make_pgtable).  Further adding to my confusion is why the early printk
magic didn't dump a stack as I believe I had that setup on my commandline.
But I figured I would just report what I have observed.

My testing and debugging were based off a 3.10 kernel (RHEL-7) but has included
Seiji's tracepoint cleanups to arch/x86/kernel/head_64.S|head64.c.  Not much
has changed upstream here.  Also 3.14-rc4 still has the same hang.

Signed-off-by: Don Zickus <dzickus@redhat.com>
---
 arch/x86/kernel/head_64.S |    2 ++
 1 files changed, 2 insertions(+), 0 deletions(-)

diff --git a/arch/x86/kernel/head_64.S b/arch/x86/kernel/head_64.S
index 77e6d3e..05306c8 100644
--- a/arch/x86/kernel/head_64.S
+++ b/arch/x86/kernel/head_64.S
@@ -368,6 +368,8 @@ ENTRY(early_idt_handler)
 	jz 20f			# All good
 
 10:
+	cmpl $2,72(%rsp)	# NMI?
+	jz 20f			# skip NMIs
 	leaq 88(%rsp),%rdi	# Pointer to %rip
 	call early_fixup_exception
 	andl %eax,%eax
-- 
1.7.1

Re: [PATCH] x86: Skip latched NMIs on early boot in kdump

[H. Peter Anvin]

[-- Attachment #1: Type: text/plain, Size: 1519 bytes --]

On 03/07/2014 11:39 AM, Don Zickus wrote:
> A customer generated an external NMI using their iLO to test kdump worked.
> Unfortunately, the machine hung.  Disabling the nmi_watchdog made things work.
> 
> I speculated the external NMI fired, caused the machine to panic (as expected)
> and the perf NMI from the watchdog came in and was latched.  My guess was this
> somehow caused the hang.
> 

... as any other unexpected exception would.

> 
> I also do not fully understand why the latched NMI is not happening immediately
> after the load idt call or why it comes after a page fault (the
> early_make_pgtable).  Further adding to my confusion is why the early printk
> magic didn't dump a stack as I believe I had that setup on my commandline.
> But I figured I would just report what I have observed.
> 

If the kdump is initiated from NMI context, I'm wondering if it might be
possible that we haven't actually executed an IRET until this one
happens, and the IRET re-enables NMI.

> My testing and debugging were based off a 3.10 kernel (RHEL-7) but has included
> Seiji's tracepoint cleanups to arch/x86/kernel/head_64.S|head64.c.  Not much
> has changed upstream here.  Also 3.14-rc4 still has the same hang.
> 
> Signed-off-by: Don Zickus <dzickus@redhat.com>

We really shouldn't be doing the fixup lookup for NMI, either.  Probably
it makes more sense to just IRET on NMI until we have the real interrupt
vectors set up, but it needs to be done a little earlier.

How does this patch work for you?

	-hpa


[-- Attachment #2: diff --]
[-- Type: text/plain, Size: 1343 bytes --]

diff --git a/arch/x86/kernel/head_32.S b/arch/x86/kernel/head_32.S
index 81ba276..d2a2159 100644
--- a/arch/x86/kernel/head_32.S
+++ b/arch/x86/kernel/head_32.S
@@ -544,6 +544,10 @@ ENDPROC(early_idt_handlers)
 	/* This is global to keep gas from relaxing the jumps */
 ENTRY(early_idt_handler)
 	cld
+
+	cmpl $X86_TRAP_NMI,(%esp)
+	je is_nmi		# Ignore NMI
+
 	cmpl $2,%ss:early_recursion_flag
 	je hlt_loop
 	incl %ss:early_recursion_flag
@@ -594,8 +598,9 @@ ex_entry:
 	pop %edx
 	pop %ecx
 	pop %eax
-	addl $8,%esp		/* drop vector number and error code */
 	decl %ss:early_recursion_flag
+is_nmi:
+	addl $8,%esp		/* drop vector number and error code */
 	iret
 ENDPROC(early_idt_handler)
 
diff --git a/arch/x86/kernel/head_64.S b/arch/x86/kernel/head_64.S
index e1aabdb..33f36c7 100644
--- a/arch/x86/kernel/head_64.S
+++ b/arch/x86/kernel/head_64.S
@@ -343,6 +343,9 @@ early_idt_handlers:
 ENTRY(early_idt_handler)
 	cld
 
+	cmpl $X86_TRAP_NMI,(%rsp)
+	je is_nmi		# Ignore NMI
+
 	cmpl $2,early_recursion_flag(%rip)
 	jz  1f
 	incl early_recursion_flag(%rip)
@@ -405,8 +408,9 @@ ENTRY(early_idt_handler)
 	popq %rdx
 	popq %rcx
 	popq %rax
-	addq $16,%rsp		# drop vector number and error code
 	decl early_recursion_flag(%rip)
+is_nmi:
+	addq $16,%rsp		# drop vector number and error code
 	INTERRUPT_RETURN
 ENDPROC(early_idt_handler)
 

Re: [PATCH] x86: Skip latched NMIs on early boot in kdump

[Don Zickus]

On Fri, Mar 07, 2014 at 01:15:35PM -0800, H. Peter Anvin wrote:
> On 03/07/2014 11:39 AM, Don Zickus wrote:
> > A customer generated an external NMI using their iLO to test kdump worked.
> > Unfortunately, the machine hung.  Disabling the nmi_watchdog made things work.
> > 
> > I speculated the external NMI fired, caused the machine to panic (as expected)
> > and the perf NMI from the watchdog came in and was latched.  My guess was this
> > somehow caused the hang.
> > 
> 
> ... as any other unexpected exception would.
> 
> > 
> > I also do not fully understand why the latched NMI is not happening immediately
> > after the load idt call or why it comes after a page fault (the
> > early_make_pgtable).  Further adding to my confusion is why the early printk
> > magic didn't dump a stack as I believe I had that setup on my commandline.
> > But I figured I would just report what I have observed.
> > 
> 
> If the kdump is initiated from NMI context, I'm wondering if it might be
> possible that we haven't actually executed an IRET until this one
> happens, and the IRET re-enables NMI.

Ah makes sense then.

> 
> > My testing and debugging were based off a 3.10 kernel (RHEL-7) but has included
> > Seiji's tracepoint cleanups to arch/x86/kernel/head_64.S|head64.c.  Not much
> > has changed upstream here.  Also 3.14-rc4 still has the same hang.
> > 
> > Signed-off-by: Don Zickus <dzickus@redhat.com>
> 
> We really shouldn't be doing the fixup lookup for NMI, either.  Probably
> it makes more sense to just IRET on NMI until we have the real interrupt
> vectors set up, but it needs to be done a little earlier.
> 
> How does this patch work for you?

I tested it on 64 bit and it works good.  Thanks!

Cheers,
Don

> 
> 	-hpa
> 

> diff --git a/arch/x86/kernel/head_32.S b/arch/x86/kernel/head_32.S
> index 81ba276..d2a2159 100644
> --- a/arch/x86/kernel/head_32.S
> +++ b/arch/x86/kernel/head_32.S
> @@ -544,6 +544,10 @@ ENDPROC(early_idt_handlers)
>  	/* This is global to keep gas from relaxing the jumps */
>  ENTRY(early_idt_handler)
>  	cld
> +
> +	cmpl $X86_TRAP_NMI,(%esp)
> +	je is_nmi		# Ignore NMI
> +
>  	cmpl $2,%ss:early_recursion_flag
>  	je hlt_loop
>  	incl %ss:early_recursion_flag
> @@ -594,8 +598,9 @@ ex_entry:
>  	pop %edx
>  	pop %ecx
>  	pop %eax
> -	addl $8,%esp		/* drop vector number and error code */
>  	decl %ss:early_recursion_flag
> +is_nmi:
> +	addl $8,%esp		/* drop vector number and error code */
>  	iret
>  ENDPROC(early_idt_handler)
>  
> diff --git a/arch/x86/kernel/head_64.S b/arch/x86/kernel/head_64.S
> index e1aabdb..33f36c7 100644
> --- a/arch/x86/kernel/head_64.S
> +++ b/arch/x86/kernel/head_64.S
> @@ -343,6 +343,9 @@ early_idt_handlers:
>  ENTRY(early_idt_handler)
>  	cld
>  
> +	cmpl $X86_TRAP_NMI,(%rsp)
> +	je is_nmi		# Ignore NMI
> +
>  	cmpl $2,early_recursion_flag(%rip)
>  	jz  1f
>  	incl early_recursion_flag(%rip)
> @@ -405,8 +408,9 @@ ENTRY(early_idt_handler)
>  	popq %rdx
>  	popq %rcx
>  	popq %rax
> -	addq $16,%rsp		# drop vector number and error code
>  	decl early_recursion_flag(%rip)
> +is_nmi:
> +	addq $16,%rsp		# drop vector number and error code
>  	INTERRUPT_RETURN
>  ENDPROC(early_idt_handler)
>  

[tip:x86/urgent] x86: Ignore NMIs that come in during early boot

[tip-bot for H. Peter Anvin]

Commit-ID:  5fa10196bdb5f190f595ebd048490ee52dddea0f
Gitweb:     http://git.kernel.org/tip/5fa10196bdb5f190f595ebd048490ee52dddea0f
Author:     H. Peter Anvin <hpa@linux.intel.com>
AuthorDate: Fri, 7 Mar 2014 15:05:20 -0800
Committer:  H. Peter Anvin <hpa@linux.intel.com>
CommitDate: Fri, 7 Mar 2014 15:08:14 -0800

x86: Ignore NMIs that come in during early boot

Don Zickus reports:

A customer generated an external NMI using their iLO to test kdump
worked.  Unfortunately, the machine hung.  Disabling the nmi_watchdog
made things work.

I speculated the external NMI fired, caused the machine to panic (as
expected) and the perf NMI from the watchdog came in and was latched.
My guess was this somehow caused the hang.

   ----

It appears that the latched NMI stays latched until the early page
table generation on 64 bits, which causes exceptions to happen which
end in IRET, which re-enable NMI.  Therefore, ignore NMIs that come in
during early execution, until we have proper exception handling.

Reported-and-tested-by: Don Zickus <dzickus@redhat.com>
Link: http://lkml.kernel.org/r/1394221143-29713-1-git-send-email-dzickus@redhat.com
Signed-off-by: H. Peter Anvin <hpa@linux.intel.com>
Cc: <stable@vger.kernel.org> # v3.5+, older with some backport effort
---
 arch/x86/kernel/head_32.S | 7 ++++++-
 arch/x86/kernel/head_64.S | 6 +++++-
 2 files changed, 11 insertions(+), 2 deletions(-)

diff --git a/arch/x86/kernel/head_32.S b/arch/x86/kernel/head_32.S
index 81ba276..d2a2159 100644
--- a/arch/x86/kernel/head_32.S
+++ b/arch/x86/kernel/head_32.S
@@ -544,6 +544,10 @@ ENDPROC(early_idt_handlers)
 	/* This is global to keep gas from relaxing the jumps */
 ENTRY(early_idt_handler)
 	cld
+
+	cmpl $X86_TRAP_NMI,(%esp)
+	je is_nmi		# Ignore NMI
+
 	cmpl $2,%ss:early_recursion_flag
 	je hlt_loop
 	incl %ss:early_recursion_flag
@@ -594,8 +598,9 @@ ex_entry:
 	pop %edx
 	pop %ecx
 	pop %eax
-	addl $8,%esp		/* drop vector number and error code */
 	decl %ss:early_recursion_flag
+is_nmi:
+	addl $8,%esp		/* drop vector number and error code */
 	iret
 ENDPROC(early_idt_handler)
 
diff --git a/arch/x86/kernel/head_64.S b/arch/x86/kernel/head_64.S
index e1aabdb..33f36c7 100644
--- a/arch/x86/kernel/head_64.S
+++ b/arch/x86/kernel/head_64.S
@@ -343,6 +343,9 @@ early_idt_handlers:
 ENTRY(early_idt_handler)
 	cld
 
+	cmpl $X86_TRAP_NMI,(%rsp)
+	je is_nmi		# Ignore NMI
+
 	cmpl $2,early_recursion_flag(%rip)
 	jz  1f
 	incl early_recursion_flag(%rip)
@@ -405,8 +408,9 @@ ENTRY(early_idt_handler)
 	popq %rdx
 	popq %rcx
 	popq %rax
-	addq $16,%rsp		# drop vector number and error code
 	decl early_recursion_flag(%rip)
+is_nmi:
+	addq $16,%rsp		# drop vector number and error code
 	INTERRUPT_RETURN
 ENDPROC(early_idt_handler)