* Re: [BUG REPORT] Triggering a panic in an x86 virtual machine does not wait [not found] ` <71578392-63ed-02a9-24da-2adf8cce38c7@huawei.com> @ 2023-07-05 8:59 ` Thomas Gleixner 2023-07-06 6:44 ` Baokun Li 0 siblings, 1 reply; 4+ messages in thread From: Thomas Gleixner @ 2023-07-05 8:59 UTC (permalink / raw) To: Baokun Li Cc: arjan, ashok.raj, ashok.raj, ebiederm, linux-kernel, mario.limonciello, thomas.lendacky, tony.luck, tonyb, x86, yangerkun, Baoquan He, kexec On Mon, Jul 03 2023 at 11:44, Baokun Li wrote: > When I manually trigger panic in a qume x86 VM with > > `echo c > /proc/sysrq-trigger`, > > I find that the VM will probably reboot directly, but the > PANIC_TIMEOUT is 0. > This prevents us from exporting the vmcore via panic, and even if we succeed > in panic exporting the vmcore, the processes in the vmcore are mostly > stop_this_cpu(). By dichotomizing we found the patch that introduced the > behavior change > > 45e34c8af58f ("x86/smp: Put CPUs into INIT on shutdown if possible"), Bah, I missed that this is used by crash too. So if this happens to be invoked on an AP, i.e. not on CPU 0, then the INIT will reset the machine. Fix below. Thanks, tglx --- diff --git a/arch/x86/kernel/smpboot.c b/arch/x86/kernel/smpboot.c index ed2d51960a7d..e1aa2cd7734b 100644 --- a/arch/x86/kernel/smpboot.c +++ b/arch/x86/kernel/smpboot.c @@ -1348,6 +1348,14 @@ bool smp_park_other_cpus_in_init(void) if (apic->wakeup_secondary_cpu_64 || apic->wakeup_secondary_cpu) return false; + /* + * If this is a crash stop which does not execute on the boot CPU, + * then this cannot use the INIT mechanism because INIT to the boot + * CPU will reset the machine. + */ + if (this_cpu) + return false; + for_each_present_cpu(cpu) { if (cpu == this_cpu) continue; _______________________________________________ kexec mailing list kexec@lists.infradead.org http://lists.infradead.org/mailman/listinfo/kexec ^ permalink raw reply related [flat|nested] 4+ messages in thread
* Re: [BUG REPORT] Triggering a panic in an x86 virtual machine does not wait 2023-07-05 8:59 ` [BUG REPORT] Triggering a panic in an x86 virtual machine does not wait Thomas Gleixner @ 2023-07-06 6:44 ` Baokun Li 2023-07-07 10:18 ` Thomas Gleixner 0 siblings, 1 reply; 4+ messages in thread From: Baokun Li @ 2023-07-06 6:44 UTC (permalink / raw) To: Thomas Gleixner Cc: arjan, ashok.raj, ashok.raj, ebiederm, linux-kernel, mario.limonciello, thomas.lendacky, tony.luck, tonyb, x86, yangerkun, Baoquan He, kexec, Baokun Li On 2023/7/5 16:59, Thomas Gleixner wrote: > On Mon, Jul 03 2023 at 11:44, Baokun Li wrote: > >> When I manually trigger panic in a qume x86 VM with >> >> `echo c > /proc/sysrq-trigger`, >> >> I find that the VM will probably reboot directly, but the >> PANIC_TIMEOUT is 0. >> This prevents us from exporting the vmcore via panic, and even if we succeed >> in panic exporting the vmcore, the processes in the vmcore are mostly >> stop_this_cpu(). By dichotomizing we found the patch that introduced the >> behavior change >> >> 45e34c8af58f ("x86/smp: Put CPUs into INIT on shutdown if possible"), > Bah, I missed that this is used by crash too. So if this happens to be > invoked on an AP, i.e. not on CPU 0, then the INIT will reset the > machine. Fix below. > > Thanks, > > tglx > --- > diff --git a/arch/x86/kernel/smpboot.c b/arch/x86/kernel/smpboot.c > index ed2d51960a7d..e1aa2cd7734b 100644 > --- a/arch/x86/kernel/smpboot.c > +++ b/arch/x86/kernel/smpboot.c > @@ -1348,6 +1348,14 @@ bool smp_park_other_cpus_in_init(void) > if (apic->wakeup_secondary_cpu_64 || apic->wakeup_secondary_cpu) > return false; > > + /* > + * If this is a crash stop which does not execute on the boot CPU, > + * then this cannot use the INIT mechanism because INIT to the boot > + * CPU will reset the machine. > + */ > + if (this_cpu) > + return false; > + > for_each_present_cpu(cpu) { > if (cpu == this_cpu) > continue; This patch does fix the problem of rebooting at panic, but the exported stack stays at stop_this_cpu() like below, instead of showing what the corresponding process is doing as before. PID: 681 TASK: ffff9ac2429d3080 CPU: 2 COMMAND: "fsstress" #0 [ffffb00200184fd0] stop_this_cpu at ffffffff89a4ffd8 #1 [ffffb00200184fe8] __sysvec_reboot at ffffffff89a94213 #2 [ffffb00200184ff0] sysvec_reboot at ffffffff8aee7491 --- <IRQ stack> --- RIP: 0000000000000010 RSP: 0000000000000018 RFLAGS: ffffb00200f8bd08 RAX: ffff9ac256fda9d8 RBX: 0000000009973a85 RCX: ffff9ac256fda078 RDX: ffff9ac24416e300 RSI: ffff9ac256fda9e0 RDI: ffffffffffffffff RBP: ffff9ac2443a5f88 R8: 0000000000000000 R9: ffff9ac2422eeea0 R10: ffff9ac256fda9d8 R11: 0000000000549921 R12: ffff9ac2422eeea0 R13: ffff9ac251cd23c8 R14: ffff9ac24269a800 R15: ffff9ac251cd2150 ORIG_RAX: ffffffff8a1719e4 CS: 0206 SS: ffffffff8a1719c8 bt: WARNING: possibly bogus exception frame Do you know how this happened? I would be grateful if you could fix it. Thanks! -- With Best Regards, Baokun Li . _______________________________________________ kexec mailing list kexec@lists.infradead.org http://lists.infradead.org/mailman/listinfo/kexec ^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: [BUG REPORT] Triggering a panic in an x86 virtual machine does not wait 2023-07-06 6:44 ` Baokun Li @ 2023-07-07 10:18 ` Thomas Gleixner 2023-07-07 12:40 ` Baokun Li 0 siblings, 1 reply; 4+ messages in thread From: Thomas Gleixner @ 2023-07-07 10:18 UTC (permalink / raw) To: Baokun Li Cc: arjan, ashok.raj, ashok.raj, ebiederm, linux-kernel, mario.limonciello, thomas.lendacky, tony.luck, tonyb, x86, yangerkun, Baoquan He, kexec, Baokun Li On Thu, Jul 06 2023 at 14:44, Baokun Li wrote: > On 2023/7/5 16:59, Thomas Gleixner wrote: >> + /* >> + * If this is a crash stop which does not execute on the boot CPU, >> + * then this cannot use the INIT mechanism because INIT to the boot >> + * CPU will reset the machine. >> + */ >> + if (this_cpu) >> + return false; > This patch does fix the problem of rebooting at panic, but the > exported stack stays at stop_this_cpu() like below, instead of showing > what the corresponding process is doing as before. > > PID: 681 TASK: ffff9ac2429d3080 CPU: 2 COMMAND: "fsstress" > #0 [ffffb00200184fd0] stop_this_cpu at ffffffff89a4ffd8 > #1 [ffffb00200184fe8] __sysvec_reboot at ffffffff89a94213 > #2 [ffffb00200184ff0] sysvec_reboot at ffffffff8aee7491 > --- <IRQ stack> --- > RIP: 0000000000000010 RSP: 0000000000000018 RFLAGS: ffffb00200f8bd08 > RAX: ffff9ac256fda9d8 RBX: 0000000009973a85 RCX: ffff9ac256fda078 > RDX: ffff9ac24416e300 RSI: ffff9ac256fda9e0 RDI: ffffffffffffffff > RBP: ffff9ac2443a5f88 R8: 0000000000000000 R9: ffff9ac2422eeea0 > R10: ffff9ac256fda9d8 R11: 0000000000549921 R12: ffff9ac2422eeea0 > R13: ffff9ac251cd23c8 R14: ffff9ac24269a800 R15: ffff9ac251cd2150 > ORIG_RAX: ffffffff8a1719e4 CS: 0206 SS: ffffffff8a1719c8 > bt: WARNING: possibly bogus exception frame > > Do you know how this happened? I would be grateful if you could fix it. No, I don't. But there is clearly a hint: > bt: WARNING: possibly bogus exception frame So the exception frame seems to be corrupted. I have no idea why. The question is, whether this goes away when you revert that commit or not. I can't oracle that out from your report. Can you please revert 45e34c8af58f on top of Linus tree and verify that it makes the issue go away? Thanks, tglx _______________________________________________ kexec mailing list kexec@lists.infradead.org http://lists.infradead.org/mailman/listinfo/kexec ^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: [BUG REPORT] Triggering a panic in an x86 virtual machine does not wait 2023-07-07 10:18 ` Thomas Gleixner @ 2023-07-07 12:40 ` Baokun Li 0 siblings, 0 replies; 4+ messages in thread From: Baokun Li @ 2023-07-07 12:40 UTC (permalink / raw) To: Thomas Gleixner Cc: arjan, ashok.raj, ashok.raj, ebiederm, linux-kernel, mario.limonciello, thomas.lendacky, tony.luck, tonyb, x86, yangerkun, Baoquan He, kexec, Baokun Li On 2023/7/7 18:18, Thomas Gleixner wrote: > On Thu, Jul 06 2023 at 14:44, Baokun Li wrote: >> On 2023/7/5 16:59, Thomas Gleixner wrote: >>> + /* >>> + * If this is a crash stop which does not execute on the boot CPU, >>> + * then this cannot use the INIT mechanism because INIT to the boot >>> + * CPU will reset the machine. >>> + */ >>> + if (this_cpu) >>> + return false; This does solve the problem of x86 VMs not waiting when they panic, so Reported-and-tested-by: Baokun Li <libaokun1@huawei.com> >> This patch does fix the problem of rebooting at panic, but the >> exported stack stays at stop_this_cpu() like below, instead of showing >> what the corresponding process is doing as before. >> >> PID: 681 TASK: ffff9ac2429d3080 CPU: 2 COMMAND: "fsstress" >> #0 [ffffb00200184fd0] stop_this_cpu at ffffffff89a4ffd8 >> #1 [ffffb00200184fe8] __sysvec_reboot at ffffffff89a94213 >> #2 [ffffb00200184ff0] sysvec_reboot at ffffffff8aee7491 >> --- <IRQ stack> --- >> RIP: 0000000000000010 RSP: 0000000000000018 RFLAGS: ffffb00200f8bd08 >> RAX: ffff9ac256fda9d8 RBX: 0000000009973a85 RCX: ffff9ac256fda078 >> RDX: ffff9ac24416e300 RSI: ffff9ac256fda9e0 RDI: ffffffffffffffff >> RBP: ffff9ac2443a5f88 R8: 0000000000000000 R9: ffff9ac2422eeea0 >> R10: ffff9ac256fda9d8 R11: 0000000000549921 R12: ffff9ac2422eeea0 >> R13: ffff9ac251cd23c8 R14: ffff9ac24269a800 R15: ffff9ac251cd2150 >> ORIG_RAX: ffffffff8a1719e4 CS: 0206 SS: ffffffff8a1719c8 >> bt: WARNING: possibly bogus exception frame >> >> Do you know how this happened? I would be grateful if you could fix it. > No, I don't. But there is clearly a hint: > >> bt: WARNING: possibly bogus exception frame > So the exception frame seems to be corrupted. I have no idea why. > > The question is, whether this goes away when you revert that commit or not. > I can't oracle that out from your report. > > Can you please revert 45e34c8af58f on top of Linus tree and verify that > it makes the issue go away? > > Thanks, > > tglx Yes, the stop_this_cpu() issue persisted after I reverted 45e34c8af58f and it has nothing to do with your patch, I will try to bisect to find out which patch introduced the issue. Thank you very much for helping locate and rectify the problem that the x86 VM panic does not wait! Cheers! -- With Best Regards, Baokun Li . _______________________________________________ kexec mailing list kexec@lists.infradead.org http://lists.infradead.org/mailman/listinfo/kexec ^ permalink raw reply [flat|nested] 4+ messages in thread
end of thread, other threads:[~2023-07-07 12:40 UTC | newest]
Thread overview: 4+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
[not found] <20230615193330.608657211@linutronix.de>
[not found] ` <71578392-63ed-02a9-24da-2adf8cce38c7@huawei.com>
2023-07-05 8:59 ` [BUG REPORT] Triggering a panic in an x86 virtual machine does not wait Thomas Gleixner
2023-07-06 6:44 ` Baokun Li
2023-07-07 10:18 ` Thomas Gleixner
2023-07-07 12:40 ` Baokun Li
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox