[patch] teach netconsole how to do syslog

[patch] teach netconsole how to do syslog

[Jeff Moyer]

Hello,

Here's a patch which adds the option to send messages to a remote syslog,
enabled via the do_syslog= module parameter.  Currently logs everything at
info (as did the original netconsole module).  Patch is against 2.6.6,
though should apply to later.

Comments?

Jeff


--- linux-2.6.6/drivers/net/netconsole.c	2004-06-21 14:06:34.000000000 -0400
+++ linux-2.6.6-netdump/drivers/net/netconsole.c	2004-06-25 13:51:54.000000000 -0400
@@ -54,6 +54,10 @@ static char config[256];
 module_param_string(netconsole, config, 256, 0);
 MODULE_PARM_DESC(netconsole, " netconsole=[src-port]@[src-ip]/[dev],[tgt-port]@<tgt-ip>/[tgt-macaddr]\n");
 
+static int do_syslog;
+module_param(do_syslog, bool, 000);
+MODULE_PARM_DESC(do_syslog, " do_syslog=<yes|no>\n");
+
 static struct netpoll np = {
 	.name = "netconsole",
 	.dev_name = "eth0",
@@ -64,10 +68,36 @@ static struct netpoll np = {
 static int configured = 0;
 
 #define MAX_PRINT_CHUNK 1000
+#define SYSLOG_HEADER_LEN 4
+
+static int syslog_chars = SYSLOG_HEADER_LEN;
+static unsigned char syslog_line [MAX_PRINT_CHUNK + 10] = {
+	'<',
+	'5',
+	'>',
+	' ',
+	[4 ... MAX_PRINT_CHUNK+5] = '\0',
+};
+
+/*
+ * We feed kernel messages char by char, and send the UDP packet
+ * one linefeed. We buffer all characters received.
+ */
+static inline void feed_syslog_char(const unsigned char c)
+{
+	if (syslog_chars == MAX_PRINT_CHUNK)
+		syslog_chars--;
+	syslog_line[syslog_chars] = c;
+	syslog_chars++;
+	if (c == '\n') {
+		netpoll_send_udp(&np, syslog_line, syslog_chars);
+		syslog_chars = SYSLOG_HEADER_LEN;
+	}
+}
 
 static void write_msg(struct console *con, const char *msg, unsigned int len)
 {
-	int frag, left;
+	int frag, left, i;
 	unsigned long flags;
 
 	if (!np.dev)
@@ -75,11 +105,16 @@ static void write_msg(struct console *co
 
 	local_irq_save(flags);
 
-	for(left = len; left; ) {
-		frag = min(left, MAX_PRINT_CHUNK);
-		netpoll_send_udp(&np, msg, frag);
-		msg += frag;
-		left -= frag;
+	if (do_syslog) {
+		for (i = 0; i < len; i++)
+			feed_syslog_char(msg[i]);
+	} else {
+		for(left = len; left; ) {
+			frag = min(left, MAX_PRINT_CHUNK);
+			netpoll_send_udp(&np, msg, frag);
+			msg += frag;
+			left -= frag;
+		}
 	}
 
 	local_irq_restore(flags);

Re: [patch] teach netconsole how to do syslog

[Matt Mackall]

On Fri, Jun 25, 2004 at 01:57:38PM -0400, Jeff Moyer wrote:
> Hello,
> 
> Here's a patch which adds the option to send messages to a remote syslog,
> enabled via the do_syslog= module parameter.  Currently logs everything at
> info (as did the original netconsole module).  Patch is against 2.6.6,
> though should apply to later.

Well as it stands, it's already syslog compatible, as the priority level
component of the syslog protocol is optional. I've made a point of
_defining_ the netconsole protocol as syslog (note the default port)
so that this could be done at a later date. Thus, I don't think a new
command line option is necessary.

On the other hand, for this to have real value, we need to provide
real priority levels. Which means we need to plug it in higher in the
printk framework so that we a) get all messages by default and b) get
them before the levels are stripped off. I had this in an earlier
version but dropped it as being too intrusive for 2.6.0 merger.

Also, if we're going to go to the trouble of being more completely
syslog-like, it's very useful (and trivial) to throw in the hostname
as well. Timestamp is slightly more difficult, but also worth
considering.

-- 
Mathematics is the supreme nostalgia of our time.

Re: [patch] teach netconsole how to do syslog

[Jeff Moyer]

==> Regarding Re: [patch] teach netconsole how to do syslog; Matt Mackall <mpm@selenic.com> adds:

mpm> On Fri, Jun 25, 2004 at 01:57:38PM -0400, Jeff Moyer wrote:
>> Hello,
>> 
>> Here's a patch which adds the option to send messages to a remote
>> syslog, enabled via the do_syslog= module parameter.  Currently logs
>> everything at info (as did the original netconsole module).  Patch is
>> against 2.6.6, though should apply to later.

mpm> Well as it stands, it's already syslog compatible, as the priority
mpm> level component of the syslog protocol is optional. I've made a point
mpm> of _defining_ the netconsole protocol as syslog (note the default
mpm> port) so that this could be done at a later date. Thus, I don't think
mpm> a new command line option is necessary.

Well, the way it is coded currently does not usually give a full line of
output on each line in my system log:

Jun 25 14:41:18 remote-host HELP : 
Jun 25 14:41:18 remote-host SysRq : 
Jun 25 14:41:18 remote-host loglevel0-8 
Jun 25 14:41:18 remote-host reBoot 
Jun 25 14:41:18 remote-host Crash 
Jun 25 14:41:18 remote-host tErm 
Jun 25 14:41:18 remote-host kIll 
Jun 25 14:41:18 remote-host saK 
Jun 25 14:41:18 remote-host showMem 
Jun 25 14:41:18 remote-host powerOff 
Jun 25 14:41:18 remote-host showPc 
Jun 25 14:41:18 remote-host unRaw 
Jun 25 14:41:18 remote-host Sync 
Jun 25 14:41:18 remote-host showTasks 
Jun 25 14:41:18 remote-host Unmount 
Jun 25 14:41:18 remote-host  

mpm> On the other hand, for this to have real value, we need to provide
mpm> real priority levels. Which means we need to plug it in higher in the
mpm> printk framework so that we a) get all messages by default and b) get
mpm> them before the levels are stripped off. I had this in an earlier
mpm> version but dropped it as being too intrusive for 2.6.0 merger.

And are you still in favor of this?  Want to post the code?

mpm> Also, if we're going to go to the trouble of being more completely
mpm> syslog-like, it's very useful (and trivial) to throw in the hostname
mpm> as well. Timestamp is slightly more difficult, but also worth
mpm> considering.

Hostname shows up by default with my syslogd.

-Jeff

Re: [patch] teach netconsole how to do syslog

[Matt Mackall]

On Fri, Jun 25, 2004 at 02:47:18PM -0400, Jeff Moyer wrote:
> ==> Regarding Re: [patch] teach netconsole how to do syslog; Matt Mackall <mpm@selenic.com> adds:
> 
> mpm> On Fri, Jun 25, 2004 at 01:57:38PM -0400, Jeff Moyer wrote:
> >> Hello,
> >> 
> >> Here's a patch which adds the option to send messages to a remote
> >> syslog, enabled via the do_syslog= module parameter.  Currently logs
> >> everything at info (as did the original netconsole module).  Patch is
> >> against 2.6.6, though should apply to later.
> 
> mpm> Well as it stands, it's already syslog compatible, as the priority
> mpm> level component of the syslog protocol is optional. I've made a point
> mpm> of _defining_ the netconsole protocol as syslog (note the default
> mpm> port) so that this could be done at a later date. Thus, I don't think
> mpm> a new command line option is necessary.
> 
> Well, the way it is coded currently does not usually give a full line of
> output on each line in my system log:
> 
> Jun 25 14:41:18 remote-host HELP : 
> Jun 25 14:41:18 remote-host SysRq : 
> Jun 25 14:41:18 remote-host loglevel0-8 
> Jun 25 14:41:18 remote-host reBoot 
> Jun 25 14:41:18 remote-host Crash 
> Jun 25 14:41:18 remote-host tErm 
> Jun 25 14:41:18 remote-host kIll 
> Jun 25 14:41:18 remote-host saK 
> Jun 25 14:41:18 remote-host showMem 
> Jun 25 14:41:18 remote-host powerOff 
> Jun 25 14:41:18 remote-host showPc 
> Jun 25 14:41:18 remote-host unRaw 
> Jun 25 14:41:18 remote-host Sync 
> Jun 25 14:41:18 remote-host showTasks 
> Jun 25 14:41:18 remote-host Unmount 
> Jun 25 14:41:18 remote-host  

Yep, we get one UDP packet per printk currently, which works for most
things, but not everything. This could be changed to a buffered
approach, but that breaks one of my favorite debugging techniques -
adding an alphabet soup of single-character printks to trace tricky
call paths. 

So we could add a __printk that doesn't flush to outputs for stuff
like the above, or just live with it.
 
> mpm> On the other hand, for this to have real value, we need to provide
> mpm> real priority levels. Which means we need to plug it in higher in the
> mpm> printk framework so that we a) get all messages by default and b) get
> mpm> them before the levels are stripped off. I had this in an earlier
> mpm> version but dropped it as being too intrusive for 2.6.0 merger.
> 
> And are you still in favor of this?  Want to post the code?

Yep, I still think it wants doing, but I'm probably a bit too busy at
the moment to revive the old code. Also I'm undecided on the best way
to do it. Part of me wants to push it all down into the console hooks,
so they get all messages and levels and do their own filtering (eg we
might want different log levels for serial vs tty vs netconsole), and
unify with the kmesg queue while we're at it. The other part wants to
just stick in another hook.

> mpm> Also, if we're going to go to the trouble of being more completely
> mpm> syslog-like, it's very useful (and trivial) to throw in the hostname
> mpm> as well. Timestamp is slightly more difficult, but also worth
> mpm> considering.
> 
> Hostname shows up by default with my syslogd.

Yeah, hostname and timestamp are likely added by most syslogds, but
I've worked with a couple that didn't. I suppose we can ignore this
for now.

-- 
Mathematics is the supreme nostalgia of our time.

Re: [patch] teach netconsole how to do syslog

[Keith Owens]

On Fri, 25 Jun 2004 14:11:01 -0500, 
Matt Mackall <mpm@selenic.com> wrote:
>Yep, we get one UDP packet per printk currently, which works for most
>things, but not everything. This could be changed to a buffered
>approach, but that breaks one of my favorite debugging techniques -
>adding an alphabet soup of single-character printks to trace tricky
>call paths. 
>
>So we could add a __printk that doesn't flush to outputs for stuff
>like the above, or just live with it.

Other way round.  Keep printk as is and use a buffered approach for
printk over netconsole.  netconsole gets complete lines which is what
you want 99.9% of the time.  Add __printk or printk_unbuffered for the
.1% of debugging output that really wants unbuffered output.

Re: [patch] teach netconsole how to do syslog

[Matt Mackall]

On Sat, Jun 26, 2004 at 12:26:46PM +1000, Keith Owens wrote:
> On Fri, 25 Jun 2004 14:11:01 -0500, 
> Matt Mackall <mpm@selenic.com> wrote:
> >Yep, we get one UDP packet per printk currently, which works for most
> >things, but not everything. This could be changed to a buffered
> >approach, but that breaks one of my favorite debugging techniques -
> >adding an alphabet soup of single-character printks to trace tricky
> >call paths. 
> >
> >So we could add a __printk that doesn't flush to outputs for stuff
> >like the above, or just live with it.
> 
> Other way round.  Keep printk as is and use a buffered approach for
> printk over netconsole.  netconsole gets complete lines which is what
> you want 99.9% of the time. Add __printk or printk_unbuffered for the
> .1% of debugging output that really wants unbuffered output.

I think it's a bit too radical. The only user who cares is netconsole,
and then only when fed to syslogd. Using a client like netcat, the
current behavior is what you want. So while I think this might have
been the way to do it in the first place, changing the behavior of
every printk in the system in a way that might prevent information
from making it to the console in a crash seems like much more trouble
than removing the flush for the few cases that want to do multiple
printks per line and are making a minor mess with syslog. The
non-flushing __printk approach let's us choose when and where we want
to remove flushes.

But my current position is "just live with it".

-- 
Mathematics is the supreme nostalgia of our time.