[RFC][Patch] Adding kmsg_dump() to reboot/halt/poweroff/emergency_restart path

[RFC][Patch] Adding kmsg_dump() to reboot/halt/poweroff/emergency_restart path

[Seiji Aguchi]

Hi,

Final messages from reboot, halt and poweroff aren't output to disk because klogd/syslogd 
has been killed and root file system has been turned read-only.

Final messages from emergency_restart aren't output to disk as well because system may
reboot before klogd/syslogd outputs messages to disk.

Therefore, it is better to put kmsg_dumper in reboot/halt/poweroff/emergency_restart path.
Any comments/advices are welcome.

Seiji

 Signed-off-by: Seiji Aguchi <seiji.aguchi@hds.com>

---
 include/linux/kmsg_dump.h |    4 ++++
 kernel/printk.c           |    4 ++++
 kernel/sys.c              |    6 ++++++
 3 files changed, 14 insertions(+), 0 deletions(-)

diff --git a/include/linux/kmsg_dump.h b/include/linux/kmsg_dump.h
index 24b4414..2a0d7d6 100644
--- a/include/linux/kmsg_dump.h
+++ b/include/linux/kmsg_dump.h
@@ -18,6 +18,10 @@ enum kmsg_dump_reason {
 	KMSG_DUMP_OOPS,
 	KMSG_DUMP_PANIC,
 	KMSG_DUMP_KEXEC,
+	KMSG_DUMP_RESTART,
+	KMSG_DUMP_HALT,
+	KMSG_DUMP_POWEROFF,
+	KMSG_DUMP_EMERG,
 };
 
 /**
diff --git a/kernel/printk.c b/kernel/printk.c
index 6d6b09f..6a63848 100644
--- a/kernel/printk.c
+++ b/kernel/printk.c
@@ -1516,6 +1516,10 @@ static const char const *kmsg_reasons[] = {
 	[KMSG_DUMP_OOPS]	= "oops",
 	[KMSG_DUMP_PANIC]	= "panic",
 	[KMSG_DUMP_KEXEC]	= "kexec",
+	[KMSG_DUMP_RESTART]	= "restart",
+	[KMSG_DUMP_HALT]	= "halt",
+	[KMSG_DUMP_POWEROFF]	= "poweroff",
+	[KMSG_DUMP_EMERG]	= "emergency_restart",
 };
 
 static const char *kmsg_to_str(enum kmsg_dump_reason reason)
diff --git a/kernel/sys.c b/kernel/sys.c
index 7f5a0cd..3eab886 100644
--- a/kernel/sys.c
+++ b/kernel/sys.c
@@ -43,6 +43,8 @@
 #include <linux/kprobes.h>
 #include <linux/user_namespace.h>
 
+#include <linux/kmsg_dump.h>
+
 #include <asm/uaccess.h>
 #include <asm/io.h>
 #include <asm/unistd.h>
@@ -285,6 +287,7 @@ out_unlock:
  */
 void emergency_restart(void)
 {
+	kmsg_dump(KMSG_DUMP_EMERG);
 	machine_emergency_restart();
 }
 EXPORT_SYMBOL_GPL(emergency_restart);
@@ -312,6 +315,7 @@ void kernel_restart(char *cmd)
 		printk(KERN_EMERG "Restarting system.\n");
 	else
 		printk(KERN_EMERG "Restarting system with command '%s'.\n", cmd);
+	kmsg_dump(KMSG_DUMP_RESTART);
 	machine_restart(cmd);
 }
 EXPORT_SYMBOL_GPL(kernel_restart);
@@ -333,6 +337,7 @@ void kernel_halt(void)
 	kernel_shutdown_prepare(SYSTEM_HALT);
 	sysdev_shutdown();
 	printk(KERN_EMERG "System halted.\n");
+	kmsg_dump(KMSG_DUMP_HALT);
 	machine_halt();
 }
 
@@ -351,6 +356,7 @@ void kernel_power_off(void)
 	disable_nonboot_cpus();
 	sysdev_shutdown();
 	printk(KERN_EMERG "Power down.\n");
+	kmsg_dump(KMSG_DUMP_POWEROFF);
 	machine_power_off();
 }
 EXPORT_SYMBOL_GPL(kernel_power_off);
-- 
1.7.2.2

Re: [RFC][Patch] Adding kmsg_dump() to reboot/halt/poweroff/emergency_restart path

[Andrew Morton]

On Mon, 18 Oct 2010 18:24:11 -0400
Seiji Aguchi <seiji.aguchi@hds.com> wrote:

> Hi,
> 
> Final messages from reboot, halt and poweroff aren't output to disk because klogd/syslogd 
> has been killed and root file system has been turned read-only.
> 
> Final messages from emergency_restart aren't output to disk as well because system may
> reboot before klogd/syslogd outputs messages to disk.
> 
> Therefore, it is better to put kmsg_dumper in reboot/halt/poweroff/emergency_restart path.
> Any comments/advices are welcome.
> 

I don't get it.

What actual problem are we solving here?  Why is the current code
inadequate?  It would help to demonstrate some use-case and to explain
how the situation improved with this patch.

RE: [RFC][Patch] Adding kmsg_dump() to reboot/halt/poweroff/emergency_restart path

[Seiji Aguchi]

Hi,

> What actual problem are we solving here?  Why is the current code
> inadequate?  It would help to demonstrate some use-case and to explain
> how the situation improved with this patch.

[Purpose]
 My purpose is developing highly reliable logging facility for enterprise use.

 I'm planning to add the following triggers of kmsg_dumper().
    - reboot/poweroff/halt/emergency_restart (this patch)
    - Machine check

 I'm also planning to add an feature outputting kernel messages to NVRAM, 
 because NVRAM is equipped with enterprise servers.
 We can realize highly reliable logging facility by outputting kernel messages to NVRAM.
 (NVRAM is commonly used on Mainframe and Commercial Unix as well.)

[Use case of reboot/poweroff/halt/emergency_restart]

 My company has often experienced the followings in our support service.
 - Customer's system suddenly reboots.
 - Customers ask us to investigate the reason of the reboot.

 We recognize the fact itself because boot messages remain in /var/log/messages.
 However, we can't investigate the reason why the system rebooted, 
 because the last messages don't remain.
 And off course we can't explain the reason.
 
 
 We can solve above problem with this patch as follows.
 Case1: reboot with command 
   - We can see "Restarting system with command:" or ""Restarting system.".

 Case2: halt with command
   - We can see "System halted.".

 Case3: poweroff with command
   - We can see " Power down.".

 Case4: emergency_restart with sysrq.
   - We can see "Sysrq:" outputted in __handle_sysrq().

 Case5: emergency_restart with softdog.
   - We can see "Initiating system reboot" in watchdog_fire().

 So, we can distinguish the reason of reboot, poweroff, halt and emergency_restart.

 If customer executed reboot command, you may think the customer should know the fact.
 However, they often claim they don't execute the command when they rebooted system by mistake.

 No evidential message remain on current Linux kernel, so we can't show the proof to the customer.
 This patch improves this situation.

Seiji

Re: [RFC][Patch] Adding kmsg_dump() to reboot/halt/poweroff/emergency_restart path

[Aaron Durbin]

On 10/27/10 12:44, Seiji Aguchi wrote:
> Hi,
>
>> What actual problem are we solving here?  Why is the current code
>> inadequate?  It would help to demonstrate some use-case and to explain
>> how the situation improved with this patch.
>
> [Purpose]
>   My purpose is developing highly reliable logging facility for
enterprise use.
>
>   I'm planning to add the following triggers of kmsg_dumper().
>      - reboot/poweroff/halt/emergency_restart (this patch)
>      - Machine check
>
>   I'm also planning to add an feature outputting kernel messages to
NVRAM,
>   because NVRAM is equipped with enterprise servers.
>   We can realize highly reliable logging facility by outputting
kernel messages to NVRAM.
>   (NVRAM is commonly used on Mainframe and Commercial Unix as well.)
>
> [Use case of reboot/poweroff/halt/emergency_restart]
>
>   My company has often experienced the followings in our support service.
>   - Customer's system suddenly reboots.
>   - Customers ask us to investigate the reason of the reboot.
>
>   We recognize the fact itself because boot messages remain in
/var/log/messages.
>   However, we can't investigate the reason why the system rebooted,
>   because the last messages don't remain.
>   And off course we can't explain the reason.
>
>
>   We can solve above problem with this patch as follows.
>   Case1: reboot with command
>     - We can see "Restarting system with command:" or ""Restarting
system.".
>
>   Case2: halt with command
>     - We can see "System halted.".
>
>   Case3: poweroff with command
>     - We can see " Power down.".
>
>   Case4: emergency_restart with sysrq.
>     - We can see "Sysrq:" outputted in __handle_sysrq().
>
>   Case5: emergency_restart with softdog.
>     - We can see "Initiating system reboot" in watchdog_fire().
>
>   So, we can distinguish the reason of reboot, poweroff, halt and
emergency_restart.
>
>   If customer executed reboot command, you may think the customer
should know the fact.
>   However, they often claim they don't execute the command when they
rebooted system by mistake.
>
>   No evidential message remain on current Linux kernel, so we can't
show the proof to the customer.
>   This patch improves this situation.
>
> Seiji

We carry patches in our kernels that do very similar things. The reason 
is essentially the same as what you have cited. On our platforms we have 
two different ways of storing events to an event log. One communicates 
with the BIOS itself; the other writes bit flags to a known area of 
non-volatile storage. That way when the machine comes back up we have a 
clear eventlog (with times) as to what happened when. Piecing these 
events together has proven to be invaluable for finding issues.

For both of the drivers that log these events they use a shared 
interface that collect various events in the kernel and present them 
through a single notifier chain for the drivers' consumption.

The things we currently track and log are the following:
- clean reboot/shutdown
- panic
- oops
- die
- NMI watchdog

An example eventlog produced by our systems looks like the following 
(63-67 are the boot numbers of the system in question):

2010-10-14 10:26:06 | System Reset | 63
2010-10-14 10:26:19 | System boot | 63
2010-10-14 11:36:43 | Kernel Shutdown | 63 | Unknown Shutdown Reason
2010-10-14 11:36:43 | System Reset | 64
2010-10-14 11:36:56 | System boot | 64
2010-10-18 14:51:54 | Kernel Shutdown | 64 | Clean
2010-10-18 14:52:38 | System Reset | 65
2010-10-18 14:52:51 | System boot | 65
2010-10-26 02:44:48 | Kernel Shutdown | 65 | Oops
2010-10-26 02:44:48 | Kernel Shutdown | 65 | Die
2010-10-26 02:44:49 | Kernel Shutdown | 65 | Panic
2010-10-26 02:45:43 | System Reset | 66
2010-10-26 02:45:56 | System boot | 66
2010-10-26 02:49:22 | Kernel Shutdown | 66 | Clean
2010-10-26 02:50:05 | System Reset | 67
2010-10-26 02:50:18 | System boot | 67
2010-10-26 11:39:20 | Kernel Shutdown | 67 | Clean

Hope that helps others know that we think such a mechansim is vital. I 
can post the patches for the common infrastructure if people are interested.

-Aaron

Re: [RFC][Patch] Adding kmsg_dump() to reboot/halt/poweroff/emergency_restart path

[Artem Bityutskiy]

On Wed, 2010-11-03 at 22:50 +0100, ext Aaron Durbin wrote:
> Hope that helps others know that we think such a mechansim is vital. I 
> can post the patches for the common infrastructure if people are interested.

I think Andrew replied that this patch is OK, it only needs some
tweaking. Just tweak it and send out.

-- 
Best Regards,
Artem Bityutskiy (Артём Битюцкий)

Re: [RFC][Patch] Adding kmsg_dump() to reboot/halt/poweroff/emergency_restart path

[KOSAKI Motohiro]

> Hi,
> 
> Final messages from reboot, halt and poweroff aren't output to disk because klogd/syslogd 
> has been killed and root file system has been turned read-only.
> 
> Final messages from emergency_restart aren't output to disk as well because system may
> reboot before klogd/syslogd outputs messages to disk.
> 
> Therefore, it is better to put kmsg_dumper in reboot/halt/poweroff/emergency_restart path.
> Any comments/advices are welcome.

I think mtdoops user don't want to dump logs if a system reboot/poweroff
safely because mtd device have write count limitation. and they are main
user of kmsg_dump().

But if they ack this one, I have no objection.

Thanks.

Re: [RFC][Patch] Adding kmsg_dump() to reboot/halt/poweroff/emergency_restart path

[Artem Bityutskiy]

On Tue, 2010-10-19 at 10:52 +0200, ext KOSAKI Motohiro wrote:
> > Hi,
> > 
> > Final messages from reboot, halt and poweroff aren't output to disk because klogd/syslogd 
> > has been killed and root file system has been turned read-only.
> > 
> > Final messages from emergency_restart aren't output to disk as well because system may
> > reboot before klogd/syslogd outputs messages to disk.
> > 
> > Therefore, it is better to put kmsg_dumper in reboot/halt/poweroff/emergency_restart path.
> > Any comments/advices are welcome.
> 
> I think mtdoops user don't want to dump logs if a system reboot/poweroff
> safely because mtd device have write count limitation. and they are main
> user of kmsg_dump().

Right, at least this is how we use mtdoops - we log crashes.

-- 
Best Regards,
Artem Bityutskiy (Артём Битюцкий)

Re: [RFC][Patch] Adding kmsg_dump() to reboot/halt/poweroff/emergency_restart path

[Andrew Morton]

On Tue, 19 Oct 2010 11:51:12 +0300
Artem Bityutskiy <Artem.Bityutskiy@nokia.com> wrote:

> On Tue, 2010-10-19 at 10:52 +0200, ext KOSAKI Motohiro wrote:
> > > Hi,
> > > 
> > > Final messages from reboot, halt and poweroff aren't output to disk because klogd/syslogd 
> > > has been killed and root file system has been turned read-only.
> > > 
> > > Final messages from emergency_restart aren't output to disk as well because system may
> > > reboot before klogd/syslogd outputs messages to disk.
> > > 
> > > Therefore, it is better to put kmsg_dumper in reboot/halt/poweroff/emergency_restart path.
> > > Any comments/advices are welcome.
> > 
> > I think mtdoops user don't want to dump logs if a system reboot/poweroff
> > safely because mtd device have write count limitation. and they are main
> > user of kmsg_dump().

mtdoops was the only user until ramoops came along.

> Right, at least this is how we use mtdoops - we log crashes.

Fair enough.  So I suppose the existing clients (mtdoops_do_dump and
ramoops_do_dump) should be altered to inspect the `reason' argument,
and return if it isn't KMSG_DUMP_OOPS, KMSG_DUMP_PANIC or
KMSG_DUMP_KEXEC (I assume?)

If that sounds suitable then I'd ask Seiji to update and resend the
patch, along with a complete changelog as earlier dicussed.

I'd suggest doing it as two patches, the first of which alters mtdoops
and ramoops to perform their actions only for the appropriate `reason'
values.

Re: [RFC][Patch] Adding kmsg_dump() to reboot/halt/poweroff/emergency_restart path

[Artem Bityutskiy]

On Wed, 2010-10-27 at 16:35 -0700, Andrew Morton wrote:
> > > I think mtdoops user don't want to dump logs if a system reboot/poweroff
> > > safely because mtd device have write count limitation. and they are main
> > > user of kmsg_dump().
> 
> mtdoops was the only user until ramoops came along.
> 
> > Right, at least this is how we use mtdoops - we log crashes.
> 
> Fair enough.  So I suppose the existing clients (mtdoops_do_dump and
> ramoops_do_dump) should be altered to inspect the `reason' argument,
> and return if it isn't KMSG_DUMP_OOPS, KMSG_DUMP_PANIC or
> KMSG_DUMP_KEXEC (I assume?)

I agree, then mtdoops/ramoops can be tweaked to handle other messages.

> If that sounds suitable then I'd ask Seiji to update and resend the
> patch, along with a complete changelog as earlier dicussed.

Yes, I think the patch makes perfect sense, just the existing
functionality should not be changed as a side effect.

> I'd suggest doing it as two patches, the first of which alters mtdoops
> and ramoops to perform their actions only for the appropriate `reason'
> values.

Right, thanks.

-- 
Best Regards,
Artem Bityutskiy (Битюцкий Артём)