Re: kexec crash kernel running with watchdog enabled

[Hidehiro Kawai <hidehiro.kawai.ez@hitachi.com>]

Hi,

(2015/06/25 1:31), dwalker@fifo99.com wrote:
> On Wed, Jun 24, 2015 at 03:52:48PM +0900, Masami Hiramatsu wrote:
>> Hi,
>>
>> On 2015/06/23 23:05, dwalker@fifo99.com wrote:
>>>
>>> Hi,
>>>
>>> There was a commit in kernel/panic.c which altered when the kexec crash kernel is executed,
>>>
>>> commit f06e5153f4ae2e2f3b0300f0e260e40cb7fefd45
>>> Author: Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com>
>>> Date:   Fri Jun 6 14:37:07 2014 -0700
>>>
>>>     kernel/panic.c: add "crash_kexec_post_notifiers" option for kdump after panic_notifers
>>>
>>>
>>> This made it possible for smp_send_stop() to be executed prior to calling the kexec crash
>>> kernel.
>>>
>>> The issue is that smp_send_stop() offlines the cores, and other code depend on the cores being online.
>>>
>>> In my case on Octeon here's an example,
>>>
>>> panic()
>>>  crash_kexec()
>>>   machine_crash_shutdown()
>>>    octeon_generic_shutdown()
>>>
>>> Inside octeon_generic_shutdown() the Octeon watchdog is shutdown for_each_online_cpu(), but since
>>> most of the cpu's already got offlined in smp_send_stop() it means the watchdog is still alive on
>>> those cores. This results in a reboot during the crash kernel execution.
>>
>> Ah, I see.
>>
>>> Another example seem to be in default_machine_crash_shutdown() where crash_kexec_prepare_cpus() depends
>>> on an IPI for saving the registers on different cores. However, the cpu's are all offlined with
>>> interrupts disabled so they won't be running those IPI's in this case.
>>>
>>> I'm looking for any advice on how this should be fixed, or if it's already fixed. I'm not going to be
>>> submitting a patch so if anyone wants to submit one feel free to do so.
>>
>> Hmm, IMHO, when the cpu goes to offline in appropriate way(smp_send_stop), it should stop
>> watchdog timer on the offlined cpu too.
>> Or, you can also register crash handler which stops all watchdogs, but it's a bit tricky.
>>
> 
> That doesn't really fix all the issue tho. As I was explaining generic MIPS code depends on the cpu's
> effectively being online for crash data collection (with an IPI). This issue may effect other architectures also,
> because smp_send_stop() offlines the cpu on other architectures also. I haven't surveyed the other architectures
> enough to know what issue could happen from this tho.
> 
> Is it possible to move the smp_send_stop() below the notifiers ? I'm just throwing out ideas.

No, that doesn't works.  Some notifiers assume that they run in
single core mode.

Another possible solution is to add notifiers just after
machine_crash_shutdown() like this:

void panic(const char *fmt, ...)
...
-	if (!crash_kexec_post_notifiers)
-		crash_kexec(NULL);
+	crash_kexec(NULL, buf);

  and

-void crash_kexec(struct pt_regs *regs)
+void crash_kexec(struct pt_regs *regs, char *msg)
 ...
 		if (kexec_crash_image) {
 			struct pt_regs fixed_regs;
 
 			crash_setup_regs(&fixed_regs, regs);
 			crash_save_vmcoreinfo();
 			machine_crash_shutdown(&fixed_regs);
+			if (crash_kexec_post_notifiers) {
+				kmsg_dump(KMSG_DUMP_PANIC);
+				atomic_notifier_call_chain(&panic_notifier_list, 0, msg);
+			}
                        machine_kexec(kexec_crash_image);

Most of archs stop other cores in machine_crash_shutdown(),
so it will work well.  Furthermore, it simplifies the special
case where crash_kexec() is called without entering panic().

However, we need some tweaks for sh and s390 cases.  As for sh,
it seems not to stop other cores in the crash_kexec() sequence
(kdump support is incompleted?).  For s390, smp_send_stop() is
called in machine_kexec() but not machine_crash_shutdown().

Regards,

-- 
Hidehiro Kawai
Hitachi, Ltd. Research & Development Group