[RFC] arch generic way to trigger unknown NMIs

[RFC] arch generic way to trigger unknown NMIs

[Don Zickus]

Hi,

I was trying to put some automation test scripts together and was looking
for an arch neutral way to generate an unknown NMI.  I know some arches
do not support NMI.  Any way I wanted to add the code to the lkdtm module,
so I could more easily generate unknown NMIs  without hacking up the code.

For x86, I have been using:


diff --git a/drivers/misc/lkdtm.c b/drivers/misc/lkdtm.c
index ef34de7..c685242 100644
--- a/drivers/misc/lkdtm.c
+++ b/drivers/misc/lkdtm.c
@@ -43,6 +43,7 @@
 #include <linux/slab.h>
 #include <scsi/scsi_cmnd.h>
 #include <linux/debugfs.h>
+#include <asm/apic.h>
 
 #ifdef CONFIG_IDE
 #include <linux/ide.h>
@@ -78,6 +79,7 @@ enum ctype {
 	SOFTLOCKUP,
 	HARDLOCKUP,
 	HUNG_TASK,
+	NMI,
 };
 
 static char* cp_name[] = {
@@ -105,6 +107,7 @@ static char* cp_type[] = {
 	"SOFTLOCKUP",
 	"HARDLOCKUP",
 	"HUNG_TASK",
+	"NMI",
 };
 
 static struct jprobe lkdtm;
@@ -340,6 +343,9 @@ static void lkdtm_do_action(enum ctype which)
 		set_current_state(TASK_UNINTERRUPTIBLE);
 		schedule();
 		break;
+	case NMI:
+		apic->send_IPI_allbutself(NMI_VECTOR);
+		break;
 	case NONE:
 	default:
 		break;

Anyone have any thoughts?  Maybe there is an easier way?

Thanks,
Don

Re: [RFC] arch generic way to trigger unknown NMIs

[Andi Kleen]

Having a regression test for this is good, but it would 
be also good if it wasn't a private one but in some public
git repository.

>  static struct jprobe lkdtm;
> @@ -340,6 +343,9 @@ static void lkdtm_do_action(enum ctype which)
>  		set_current_state(TASK_UNINTERRUPTIBLE);
>  		schedule();
>  		break;
> +	case NMI:
> +		apic->send_IPI_allbutself(NMI_VECTOR);
> +		break;
>  	case NONE:
>  	default:
>  		break;
> 
> Anyone have any thoughts?  Maybe there is an easier way?

Do you really want the NMI on all CPUs - 1? Normally it's directed to
a single one.

-Andi

Re: [RFC] arch generic way to trigger unknown NMIs

[Don Zickus]

On Thu, Oct 07, 2010 at 09:26:41AM +0200, Andi Kleen wrote:
> 
> Having a regression test for this is good, but it would 
> be also good if it wasn't a private one but in some public
> git repository.

Yeah, I know.  It isn't meant to be private, mostly glue logic to load
this module using RedHat's internal test harness.

Is there a more public place to add a test like this?  I guess that would
be LTP.  Though last time I looked at LTP, all the tests are written in
'C' whereas I just cobbled together some shell scripts to configure kdump,
load the module, panic, process the resulting vmcore to verify it panic'd
for the right reason.

> 
> >  static struct jprobe lkdtm;
> > @@ -340,6 +343,9 @@ static void lkdtm_do_action(enum ctype which)
> >  		set_current_state(TASK_UNINTERRUPTIBLE);
> >  		schedule();
> >  		break;
> > +	case NMI:
> > +		apic->send_IPI_allbutself(NMI_VECTOR);
> > +		break;
> >  	case NONE:
> >  	default:
> >  		break;
> > 
> > Anyone have any thoughts?  Maybe there is an easier way?
> 
> Do you really want the NMI on all CPUs - 1? Normally it's directed to
> a single one.

No I prefer a single one too, but there didn't seem to be a
send_IPI_self() command, so I took the short route and sent it to
everyone. :-(

Cheers,
Don

> 
> -Andi

Re: [RFC] arch generic way to trigger unknown NMIs

[Andi Kleen]

On Thu, Oct 07, 2010 at 10:01:12AM -0400, Don Zickus wrote:
> On Thu, Oct 07, 2010 at 09:26:41AM +0200, Andi Kleen wrote:
> > 
> > Having a regression test for this is good, but it would 
> > be also good if it wasn't a private one but in some public
> > git repository.
> 
> Yeah, I know.  It isn't meant to be private, mostly glue logic to load
> this module using RedHat's internal test harness.
> 
> Is there a more public place to add a test like this?  I guess that would
> be LTP.  Though last time I looked at LTP, all the tests are written in
> 'C' whereas I just cobbled together some shell scripts to configure kdump,
> load the module, panic, process the resulting vmcore to verify it panic'd
> for the right reason

One possible place would be mce-test, but LTP would work too I guess.

> 
> > 
> > >  static struct jprobe lkdtm;
> > > @@ -340,6 +343,9 @@ static void lkdtm_do_action(enum ctype which)
> > >  		set_current_state(TASK_UNINTERRUPTIBLE);
> > >  		schedule();
> > >  		break;
> > > +	case NMI:
> > > +		apic->send_IPI_allbutself(NMI_VECTOR);
> > > +		break;
> > >  	case NONE:
> > >  	default:
> > >  		break;
> > > 
> > > Anyone have any thoughts?  Maybe there is an easier way?
> > 
> > Do you really want the NMI on all CPUs - 1? Normally it's directed to
> > a single one.
> 
> No I prefer a single one too, but there didn't seem to be a
> send_IPI_self() command, so I took the short route and sent it to
> everyone. :-(

You're not sending it to everyone, everyone but you.

Anyways for standard APIC send_IPI_cpu should be easy enough
to add. Standard NMI is usually to CPU #0 only.

-Andi

-- 
ak@linux.intel.com -- Speaking for myself only.

Re: [RFC] arch generic way to trigger unknown NMIs

[Don Zickus]

On Thu, Oct 07, 2010 at 05:11:29PM +0200, Andi Kleen wrote:
> On Thu, Oct 07, 2010 at 10:01:12AM -0400, Don Zickus wrote:
> > On Thu, Oct 07, 2010 at 09:26:41AM +0200, Andi Kleen wrote:
> > > 
> > > Having a regression test for this is good, but it would 
> > > be also good if it wasn't a private one but in some public
> > > git repository.
> > 
> > Yeah, I know.  It isn't meant to be private, mostly glue logic to load
> > this module using RedHat's internal test harness.
> > 
> > Is there a more public place to add a test like this?  I guess that would
> > be LTP.  Though last time I looked at LTP, all the tests are written in
> > 'C' whereas I just cobbled together some shell scripts to configure kdump,
> > load the module, panic, process the resulting vmcore to verify it panic'd
> > for the right reason
> 
> One possible place would be mce-test, but LTP would work too I guess.

I forgot about mce-test.  I can look at that too.

<snip>

> > 
> > No I prefer a single one too, but there didn't seem to be a
> > send_IPI_self() command, so I took the short route and sent it to
> > everyone. :-(
> 
> You're not sending it to everyone, everyone but you.
> 
> Anyways for standard APIC send_IPI_cpu should be easy enough
> to add. Standard NMI is usually to CPU #0 only.

Would that still be x86 specific though?  I could probably code that up,
though I wonder if it would be accepted just for a test case.

Cheers,
Don

Re: [RFC] arch generic way to trigger unknown NMIs

[Andi Kleen]

On Thu, Oct 07, 2010 at 11:47:58AM -0400, Don Zickus wrote:
> > > No I prefer a single one too, but there didn't seem to be a
> > > send_IPI_self() command, so I took the short route and sent it to
> > > everyone. :-(
> > 
> > You're not sending it to everyone, everyone but you.
> > 
> > Anyways for standard APIC send_IPI_cpu should be easy enough
> > to add. Standard NMI is usually to CPU #0 only.
> 
> Would that still be x86 specific though?  I could probably code that up,
> though I wonder if it would be accepted just for a test case.

The whole APIC interface is x86 specific.

"just a test case" is the wrong perspective.
Testing is important, if a feature is not tested it likely won't
work. That's especially important for anything error handling related.

-Andi
-- 
ak@linux.intel.com -- Speaking for myself only.

Re: [RFC] arch generic way to trigger unknown NMIs

[Maciej W. Rozycki]

On Thu, 7 Oct 2010, Don Zickus wrote:

> No I prefer a single one too, but there didn't seem to be a
> send_IPI_self() command, so I took the short route and sent it to
> everyone. :-(

 Odd, APIC hardware does explicitly support such a mode (there are three 
shorthands like this actually: all-but-self, all and self), so why don't 
just add a suitable wrapper?

  Maciej

Re: [RFC] arch generic way to trigger unknown NMIs

[Cyrill Gorcunov]

On Thu, Oct 07, 2010 at 11:45:59PM +0100, Maciej W. Rozycki wrote:
> On Thu, 7 Oct 2010, Don Zickus wrote:
> 
> > No I prefer a single one too, but there didn't seem to be a
> > send_IPI_self() command, so I took the short route and sent it to
> > everyone. :-(
> 
>  Odd, APIC hardware does explicitly support such a mode (there are three 
> shorthands like this actually: all-but-self, all and self), so why don't 
> just add a suitable wrapper?
> 
>   Maciej
> 

We have send_IPI_self which can't be used for NMI delivery mode (ie with
self shortland) but still can be used as say (didn't test)

	apic->send_IPI_mask(cpumask_of(cpu), NMI_VECTOR)

where cpu is a target.

  Cyrill

Re: [RFC] arch generic way to trigger unknown NMIs

[Maciej W. Rozycki]

On Mon, 11 Oct 2010, Cyrill Gorcunov wrote:

> We have send_IPI_self which can't be used for NMI delivery mode (ie with
> self shortland) but still can be used as say (didn't test)

 So what's stopping send_IPI_self() from being used for an NMI?  Can't the 
function be updated/fixed to support these if there's a need?

  Maciej

Re: [RFC] arch generic way to trigger unknown NMIs

[Cyrill Gorcunov]

On Sat, Oct 16, 2010 at 08:24:35AM +0100, Maciej W. Rozycki wrote:
> On Mon, 11 Oct 2010, Cyrill Gorcunov wrote:
> 
> > We have send_IPI_self which can't be used for NMI delivery mode (ie with
> > self shortland) but still can be used as say (didn't test)
> 
>  So what's stopping send_IPI_self() from being used for an NMI?  Can't the 
> function be updated/fixed to support these if there's a need?
> 
>   Maciej
> 

 Hi Maciej, the send_IPI_self could be modified to send NMI (at moment it
uses self shortcut with fixed delivery mode). The question is rather if
we need it without a real caller yet. When Don's patch gets merged we
will have a real caller then and could update send_IPI_self to support
NMI delivery mode. Something like that :)

  Cyrill

Re: [RFC] arch generic way to trigger unknown NMIs

[Maciej W. Rozycki]

On Sat, 16 Oct 2010, Cyrill Gorcunov wrote:

>  Hi Maciej, the send_IPI_self could be modified to send NMI (at moment it
> uses self shortcut with fixed delivery mode). The question is rather if
> we need it without a real caller yet. When Don's patch gets merged we
> will have a real caller then and could update send_IPI_self to support
> NMI delivery mode. Something like that :)

 Sounds backwards to me.  My understanding is a need has just arisen, so 
why not:

1. Update send_IPI_self().

2. Add code that makes use of the new functionality.

3. Submit all the changes as self-contained patches in a single series to 
   be applied at the same time.

?  That's what I'd imagine the most natural way of doing this would be.

  Maciej

Re: [RFC] arch generic way to trigger unknown NMIs

[Cyrill Gorcunov]

On Sat, Oct 16, 2010 at 05:15:16PM +0100, Maciej W. Rozycki wrote:
> On Sat, 16 Oct 2010, Cyrill Gorcunov wrote:
> 
> >  Hi Maciej, the send_IPI_self could be modified to send NMI (at moment it
> > uses self shortcut with fixed delivery mode). The question is rather if
> > we need it without a real caller yet. When Don's patch gets merged we
> > will have a real caller then and could update send_IPI_self to support
> > NMI delivery mode. Something like that :)
> 
>  Sounds backwards to me.  My understanding is a need has just arisen, so 
> why not:
> 
> 1. Update send_IPI_self().
> 
> 2. Add code that makes use of the new functionality.
> 
> 3. Submit all the changes as self-contained patches in a single series to 
>    be applied at the same time.
> 
> ?  That's what I'd imagine the most natural way of doing this would be.
> 
>   Maciej
> 

 Well, Maciej I believe the problem is not in modifying send_IPI_self
but rather _how_ to make it more natural and do not introduce overhead.
apic code is already weird enough :) Need to think.

(
 btw, we will have to add additional flag which would check for NMI
 being generated by "NMI-tester" and make a second apic write to
 ICR to deassert level line, ie it could be something like

  apic->send_IPI_self(NMI_VECTOR)       ; with asserts level
   default_do_nmi()                     ; check for NMI being sent for testing purpose
     apic->send_IPI_self(NMI_VECTOR)    ; with deasserts level

 iirc apic itself doesn't deassert nmi line on message with
 nmi deliver mode arrival
 )

  Cyrill

Re: [RFC] arch generic way to trigger unknown NMIs

[Maciej W. Rozycki]

On Sat, 16 Oct 2010, Cyrill Gorcunov wrote:

> > >  Hi Maciej, the send_IPI_self could be modified to send NMI (at moment it
> > > uses self shortcut with fixed delivery mode). The question is rather if
> > > we need it without a real caller yet. When Don's patch gets merged we
> > > will have a real caller then and could update send_IPI_self to support
> > > NMI delivery mode. Something like that :)
> > 
> >  Sounds backwards to me.  My understanding is a need has just arisen, so 
> > why not:
> > 
> > 1. Update send_IPI_self().
> > 
> > 2. Add code that makes use of the new functionality.
> > 
> > 3. Submit all the changes as self-contained patches in a single series to 
> >    be applied at the same time.
> > 
> > ?  That's what I'd imagine the most natural way of doing this would be.
> 
>  Well, Maciej I believe the problem is not in modifying send_IPI_self
> but rather _how_ to make it more natural and do not introduce overhead.
> apic code is already weird enough :) Need to think.
> 
> (
>  btw, we will have to add additional flag which would check for NMI
>  being generated by "NMI-tester" and make a second apic write to
>  ICR to deassert level line, ie it could be something like
> 
>   apic->send_IPI_self(NMI_VECTOR)       ; with asserts level
>    default_do_nmi()                     ; check for NMI being sent for testing purpose
>      apic->send_IPI_self(NMI_VECTOR)    ; with deasserts level
> 
>  iirc apic itself doesn't deassert nmi line on message with
>  nmi deliver mode arrival
>  )

 How different is it to the other two send_IPI shorthand calls?  Or the 
fully-fledged one?  I gather from this thread they already handle NMIs 
properly, so what is there within that cannot simply be copied over to 
this one?

  Maciej

Re: [RFC] arch generic way to trigger unknown NMIs

[Cyrill Gorcunov]

On Sun, Oct 17, 2010 at 12:59:11AM +0100, Maciej W. Rozycki wrote:
> On Sat, 16 Oct 2010, Cyrill Gorcunov wrote:
> 
> > > >  Hi Maciej, the send_IPI_self could be modified to send NMI (at moment it
> > > > uses self shortcut with fixed delivery mode). The question is rather if
> > > > we need it without a real caller yet. When Don's patch gets merged we
> > > > will have a real caller then and could update send_IPI_self to support
> > > > NMI delivery mode. Something like that :)
> > > 
> > >  Sounds backwards to me.  My understanding is a need has just arisen, so 
> > > why not:
> > > 
> > > 1. Update send_IPI_self().
> > > 
> > > 2. Add code that makes use of the new functionality.
> > > 
> > > 3. Submit all the changes as self-contained patches in a single series to 
> > >    be applied at the same time.
> > > 
> > > ?  That's what I'd imagine the most natural way of doing this would be.
> > 
> >  Well, Maciej I believe the problem is not in modifying send_IPI_self
> > but rather _how_ to make it more natural and do not introduce overhead.
> > apic code is already weird enough :) Need to think.
> > 
> > (
> >  btw, we will have to add additional flag which would check for NMI
> >  being generated by "NMI-tester" and make a second apic write to
> >  ICR to deassert level line, ie it could be something like
> > 
> >   apic->send_IPI_self(NMI_VECTOR)       ; with asserts level
> >    default_do_nmi()                     ; check for NMI being sent for testing purpose
> >      apic->send_IPI_self(NMI_VECTOR)    ; with deasserts level
> > 
> >  iirc apic itself doesn't deassert nmi line on message with
> >  nmi deliver mode arrival
> >  )
> 
>  How different is it to the other two send_IPI shorthand calls?  Or the 
> fully-fledged one?  I gather from this thread they already handle NMIs 
> properly, so what is there within that cannot simply be copied over to 
> this one?
> 
>   Maciej
> 

 Hmm, good question ;) I thought about 82489dx which required level
trigger mode for nmi delivery, new (or present day) apics always
use edge trigger mode so indeed there is no need for level assert/deassert.

  Cyrill

Re: [RFC] arch generic way to trigger unknown NMIs

[Cyrill Gorcunov]

On Sun, Oct 17, 2010 at 12:58:28PM +0400, Cyrill Gorcunov wrote:
...
> >  How different is it to the other two send_IPI shorthand calls?  Or the 
> > fully-fledged one?  I gather from this thread they already handle NMIs 
> > properly, so what is there within that cannot simply be copied over to 
> > this one?
> > 
> >   Maciej
> > 
> 
>  Hmm, good question ;) I thought about 82489dx which required level
> trigger mode for nmi delivery, new (or present day) apics always
> use edge trigger mode so indeed there is no need for level assert/deassert.
> 
>   Cyrill

I'll try to cook a patch for this.

  Cyrill