Re: [RFC PATCH] hw/arm/virt: Support NMI injection

[Gavin Shan <gshan@redhat.com>]

On 1/31/20 8:39 PM, Marc Zyngier wrote:
> On 2020-01-31 06:59, Gavin Shan wrote:
>> On 1/29/20 8:04 PM, Marc Zyngier wrote:
>>> On 2020-01-29 02:44, Alexey Kardashevskiy wrote:
>>>> On 28/01/2020 17:48, Gavin Shan wrote:
>>>>> but a NMI is injected
>>>>> through LAPIC on x86. So I'm not sure what architect (system reset on
>>>>> ppc or injecting NMI on x86) aarch64 should follow.
>>>>
>>>> I'd say whatever triggers in-kernel debugger or kdump but I am not
>>>> familiar with ARM at all :)
>>>
>>> All that is completely OS specific, and has no relation to the architecture.
>>> As I mentioned in another part of the thread, the closest thing to this
>>> would be to implement SDEI together with an IMPDEF mechanism to enter it
>>> (or even generate a RAS error).
>>>
>>> On the other hand, SDEI is pretty horrible, and means either KVM or QEMU
>>> acting like a firmware for the guest. To say that I'm not keen is a massive
>>> understatement.
>>>
>>>          M.
>>
>> Marc, could you please explain a bit about "IMPDEF mechanism"? I'm not sure if
>> it means a non-standard SDEI event should be used, corresponding to the HMP/QMP
>> "nmi" command.
> 
> SDEI doesn't quite describe *why* and *how* you enter it. You just get there by
> some mean (SError, Group-0 interrupt, or IMPlementation DEFined mechanism).
> It is then for the SDEI implementation to sort it out and enter the OS using the
> registered entry point.
> 

Thanks for the explanation, which make things much clearer.

>> Also, If I'm correct, you agree that a crash dump should be triggered on arm64
>> guest once HMP/QMP "nmi" command is issued?
> 
> No, I don't agree. QEMU and KVM are OS agnostic, and there is nothing in the ARMv8
> architecture that talks about "crash dumps".  If your "nmi" command generates
> a SDEI event and that event gets dispatched to the guest, it is entirely the guest's
> responsibility to do whatever it wants. We should stay clear of assuming behaviours.
> 

Ok. Thank you for your clarification.

>> I also dig into SDEI a bit. It seems the SDEI support in QEMU isn't upstream yet:
>>
>> https://patchew.org/QEMU/20191105091056.9541-1-guoheyi@huawei.com/
> 
> And I'm glad. SDEI, as I said, is absolutely horrible. I'm also very fortunate
> to have been CC'd on this series on an email address I cannot read.
> This would have huge impacts on both QEMU and KVM, and this needs more than
> a knee jerk reaction to support a QEMU command.
> 
> And to be honest, if what you require is the guest kernel to panic, why don't
> you just implement a QEMU-specific driver in Linux that does just that?
> Some kind of watchdog driver, maybe?
> 

Marc, sorry for the delay and didn't come to you in time because I wanted to figure
out the mechanism, which helps to get similar output as x86/ppc: NMI (or reset exception)
is received as indication to errors, possibly panic and restart the guest kernel.

The mechanism I figured out is to inject SError to guest, as below snippet shows. It
helps to get a panic and guest rebooting, which looks similar to what x86/ppc have.
I can post the patch as RFC if it's right direction to go :)

Note: I'm still investigating the code to see how SError can be injected when TCG
is used. I think we need same function when TCG is enabled, or it's something for
future.

static void do_inject_nmi_kvm(CPUState *cpu, Error **errp)
{
     struct kvm_vcpu_events events;
     int ret;

      :
     memset(&events, 0, sizeof(events));
     events.exception.serror_pending = 1;
     ret = kvm_vcpu_ioctl(CPU(cpu), KVM_SET_VCPU_EVENTS, &events);
      :
}

# echo 1 > /proc/sys/kernel/panic
# (qemu) nmi
[  812.510613] SError Interrupt on CPU0, code 0xbf000000 -- SError
[  812.510617] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 5.5.0-rc2-00187-gf72202430e30 #2
[  812.510617] Hardware name: QEMU KVM Virtual Machine, BIOS 0.0.0 02/06/2015
[  812.510618] pstate: 40400085 (nZcv daIf +PAN -UAO)
[  812.510619] pc : cpu_do_idle+0x48/0x58
[  812.510619] lr : arch_cpu_idle+0x30/0x238
[  812.510620] sp : ffff8000112c3e80
[  812.510620] pmr_save: 000000f0
[  812.510621] x29: ffff8000112c3e80 x28: 0000000040e10018
[  812.510622] x27: 000000033e50d340 x26: 0000000000000000
[  812.510623] x25: 0000000000000000 x24: ffff8000112ca21c
[  812.510624] x23: ffff800010f98cb8 x22: ffff8000112c98c8
[  812.510625] x21: ffff8000112ca1f8 x20: 0000000000000001
[  812.510626] x19: ffff800010f86018 x18: 0000000000000000
[  812.510627] x17: 0000000000000000 x16: 0000000000000000
[  812.510628] x15: 0000000000000000 x14: 0000000000000000
[  812.510629] x13: 0000000000000000 x12: ffff0002fe640100
[  812.510630] x11: ffffffffffffffff x10: 00000000000009f0
[  812.510631] x9 : ffff800010088928 x8 : ffff8000112d28d0
[  812.510632] x7 : ffff8002ed63a000 x6 : 0000002fe2092876
[  812.510633] x5 : 00ffffffffffffff x4 : ffff8002ed63a000
[  812.510634] x3 : 0000000000001bce x2 : 00000000000000f0
[  812.510635] x1 : 0000000000000000 x0 : 0000000000000060
[  812.510636] Kernel panic - not syncing: Asynchronous SError Interrupt
[  812.510637] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 5.5.0-rc2-00187-gf72202430e30 #2
[  812.510638] Hardware name: QEMU KVM Virtual Machine, BIOS 0.0.0 02/06/2015
[  812.510638] Call trace:
[  812.510639]  dump_backtrace+0x0/0x1a8
[  812.510639]  show_stack+0x1c/0x28
[  812.510640]  dump_stack+0xbc/0x100
[  812.510640]  panic+0x168/0x370
[  812.510640]  nmi_panic+0x68/0x98
[  812.510641]  arm64_serror_panic+0x84/0x90
[  812.510641]  do_serror+0x88/0x140
[  812.510642]  el1_error+0x8c/0x108
[  812.510642]  cpu_do_idle+0x48/0x58
[  812.510643]  default_idle_call+0x44/0x4c
[  812.510643]  do_idle+0x228/0x2d0
[  812.510644]  cpu_startup_entry+0x28/0x48
[  812.510644]  rest_init+0xdc/0xe8
[  812.510645]  arch_call_rest_init+0x14/0x1c
[  812.510645]  start_kernel+0x494/0x4c0
[  812.510675] SMP: stopping secondary CPUs
[  812.510676] Kernel Offset: disabled
[  812.510676] CPU features: 0x04402,22000438
[  812.510677] Memory Limit: none
        :
<guest is rebooted>

Thanks,
Gavin