Re: [patch] netconsole - log kernel messages over the network. 2.4.10.

Re: [patch] netconsole - log kernel messages over the network. 2.4.10.

[Marcelo Tosatti]

Ingo, 

Don't you think it would be useful to have some reserved memory for the
netconsole use ?

It would be nice to have a guarantee that messages are sent over network
even if the system is under real OOM. 

On Wed, 26 Sep 2001, Ingo Molnar wrote:

> 
> this is the first public release of the 'netconsole patch', a debugging
> patch that implements kernel-level network logging via UDP packets.
> 
> the special thing about this approach is the ability to send 'emergency'
> network packets even from IRQ handlers. This enables the netconsole to
> send enough info even if we crash in init or in an interrupt handler.
> 
> another property of netconsole is that it's able to share the networking
> device with other kernel subsystems, like the TCP/IP stack. So the
> networking device is not dedicated for netconsole use, it's transparently
> shared.
> 
> netconsole is also designed to be robust, it goes straight to the network
> driver, so it does not depend on the networking stack to log messages.
> 
> kernel-level netlogging is useful in a number of scenarios:
> 
>  - if remotely managed systems with no serial cable logging keep crashing
>    without any trace of an oops message in the userspace log. (the patch
>    was written to debug such a crash. Original idea of sending an
>    emergency packet from IRQ handlers comes from Daniel Veillard who's
>    system produced the crash - thanks Daniel!)
> 
>  - if for whatever reason the amount of logging is so high that a serial
>    console cannot hold it and disks can not keep up - or in cases where
>    logged messages disturb the debugged subsystem. I'm sure fellow VM
>    hackers will find this useful :-)
> 
>  - the netconsole can be used to emit crashdumps over the network, without
>    any delay between the point of crash and start of netlogging.
> 
> the kernel patch (against 2.4.10 or 2.4.9-ac), and a simple user-space
> tool to display netconsole messages can be found at:
> 
> 	http://redhat.com/~mingo/netconsole-patches/
> 
> sample startup of the netconsole on the server:
> 
>      insmod netconsole dev=eth1 target_ip=0x0a000701 \
>                   source_port=6666 \
>                   target_port=6666 \
>                   target_eth_byte0=0x00 \
>                   target_eth_byte1=0x90\
>                   target_eth_byte2=0x27 \
>                   target_eth_byte3=0x8C \
>                   target_eth_byte4=0xA0 \
>                   target_eth_byte5=0xA8
> 
> and on the client:
> 
> 	# ./netconsole-client -server 10.0.7.5 -client 10.0.7.1 -port 6666
>         [...network console startup...]
>         netconsole: network logging started up successfully!
>         SysRq : Loglevel set to 9
> 
> more about features and limitations can be found under:
> 
> 	Documentation/networking/netlogging.txt
> 
> reports, comments, suggestions welcome.
> 
> 	Ingo
> 
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
> 

[patch] netconsole - log kernel messages over the network. 2.4.10.

[Ingo Molnar]

this is the first public release of the 'netconsole patch', a debugging
patch that implements kernel-level network logging via UDP packets.

the special thing about this approach is the ability to send 'emergency'
network packets even from IRQ handlers. This enables the netconsole to
send enough info even if we crash in init or in an interrupt handler.

another property of netconsole is that it's able to share the networking
device with other kernel subsystems, like the TCP/IP stack. So the
networking device is not dedicated for netconsole use, it's transparently
shared.

netconsole is also designed to be robust, it goes straight to the network
driver, so it does not depend on the networking stack to log messages.

kernel-level netlogging is useful in a number of scenarios:

 - if remotely managed systems with no serial cable logging keep crashing
   without any trace of an oops message in the userspace log. (the patch
   was written to debug such a crash. Original idea of sending an
   emergency packet from IRQ handlers comes from Daniel Veillard who's
   system produced the crash - thanks Daniel!)

 - if for whatever reason the amount of logging is so high that a serial
   console cannot hold it and disks can not keep up - or in cases where
   logged messages disturb the debugged subsystem. I'm sure fellow VM
   hackers will find this useful :-)

 - the netconsole can be used to emit crashdumps over the network, without
   any delay between the point of crash and start of netlogging.

the kernel patch (against 2.4.10 or 2.4.9-ac), and a simple user-space
tool to display netconsole messages can be found at:

	http://redhat.com/~mingo/netconsole-patches/

sample startup of the netconsole on the server:

     insmod netconsole dev=eth1 target_ip=0x0a000701 \
                  source_port=6666 \
                  target_port=6666 \
                  target_eth_byte0=0x00 \
                  target_eth_byte1=0x90\
                  target_eth_byte2=0x27 \
                  target_eth_byte3=0x8C \
                  target_eth_byte4=0xA0 \
                  target_eth_byte5=0xA8

and on the client:

	# ./netconsole-client -server 10.0.7.5 -client 10.0.7.1 -port 6666
        [...network console startup...]
        netconsole: network logging started up successfully!
        SysRq : Loglevel set to 9

more about features and limitations can be found under:

	Documentation/networking/netlogging.txt

reports, comments, suggestions welcome.

	Ingo

Re: [patch] netconsole - log kernel messages over the network. 2.4.10.

[jamal]

Is there any reason you are not using dev_queue_xmit()?
(side benefits, you could hack this into using scatter gather schemes etc)

cheers,
jamal

Re: [patch] netconsole - log kernel messages over the network. 2.4.10.

[Andreas Dilger]

On Wed, 26 Sep 2001, Ingo Molnar wrote:
> sample startup of the netconsole on the server:
> 
>      insmod netconsole dev=eth1 target_ip=0x0a000701 \
>                   source_port=6666 \
>                   target_port=6666 \
>                   target_eth_byte0=0x00 \
>                   target_eth_byte1=0x90\
>                   target_eth_byte2=0x27 \
>                   target_eth_byte3=0x8C \
>                   target_eth_byte4=0xA0 \
>                   target_eth_byte5=0xA8

Ugh.  Maybe a wrapper script (netconsole-server) which automates this is
in order?  I imagine the eth_byteX is a MAC address (or at least that this
is in the documentation)?

> > and on the client:
> > 
> > 	# ./netconsole-client -server 10.0.7.5 -client 10.0.7.1 -port 6666
> >         [...network console startup...]
> >         netconsole: network logging started up successfully!
> >         SysRq : Loglevel set to 9

<wishlist>
Any chance the netconsole-client can be set up to listen for multiple hosts
and log to the local syslog?  This would allow you to run netconsole-client
on your normal syslog host and add timestamps, clean up old logs, etc.

I take it that this is only a one-way interface (i.e. logs sent from the
server to the client), and not a real serial-console replacement?
</wishlist>

In any case, this will be EXTREMELY useful for me, as a lot of new machines
do not have serial ports and are a PITA to debug in case of a crash.  Thanks.

Cheers, Andreas
--
Andreas Dilger  \ "If a man ate a pound of pasta and a pound of antipasto,
                 \  would they cancel out, leaving him still hungry?"
http://www-mddsp.enel.ucalgary.ca/People/adilger/               -- Dogbert

Re: [patch] netconsole - log kernel messages over the network. 2.4.10.

[Andreas Dilger]

On Sep 26, 2001  15:29 -0600, adilger wrote:
> On Wed, 26 Sep 2001, Ingo Molnar wrote:
> > sample startup of the netconsole on the server:
> > 
> >      insmod netconsole dev=eth1 target_ip=0x0a000701 \
> >                   source_port=6666 \
> >                   target_port=6666 \
> >                   target_eth_byte0=0x00 \
> >                   target_eth_byte1=0x90\
> >                   target_eth_byte2=0x27 \
> >                   target_eth_byte3=0x8C \
> >                   target_eth_byte4=0xA0 \
> >                   target_eth_byte5=0xA8
> 
> Ugh.  Maybe a wrapper script (netconsole-server) which automates this is
> in order?  I imagine the eth_byteX is a MAC address (or at least that this
> is in the documentation)?

Ok, I read the docs, and this is indeed a target MAC address.  It may still
be easier to accept a regular MAC address like target_mac=XX:XX:XX:XX:XX:XX
as the module parameter (and a target_ip=A.B.C.D).  In any case, here is a
script to automate this (ugly because of the conversions needed).

=========================================================================
#!/bin/sh
prog=netconsole-server
#
# initialize the netconsole using reasonable defaults (normally just the
# client IP address, and possibly the port.  We can determine the MAC
# address of the client system, IP address, the correct device, and verify
# that we are using an ethernet interface (required for netconsole to work).
#
# Andreas Dilger <adilger@turbolinux.com>  Sep 26, 2001

usage()
{
	cat - <<- EOF 1>&2

	Initialize a network message console over UDP.

	usage: $prog [-b] [-d dev] [-m mac] [-p port] target[:port]
		-b       - use broadcast ethernet MAC address
		-m       - specify remote system MAC address (default: detect)
		-p       - local port to use for message traffic (default: 6666)
		-d       - ethernet device to use for messages (default: detect)
		target   - hostname/IP address of remote netconsole-client
		:port    - port on target netconsole-client (default: like -p)
	EOF
	exit 1
}

PATH=$PATH:/sbin:/usr/sbin
PORT=6666

while [ $# -ge 1 ]; do case $1 in
	-b) NOMAC=1 ;;
	-d) DEV=$2; shift ;;
	-m) MAC=$2; shift ;;
	-p) PORT=$2; shift ;;
	*:*) TGT=`echo $1 | sed "s/:.*//"`; TPORT=`echo $1 | sed "s/.*://"` ;;
	*) TGT=$1 ;;
	esac
	shift
done

[ -z "$TGT" ] && usage
[ -z "$TPORT" ] && TPORT=$PORT

ping -c 1 $TGT > /dev/null 2>&1
[ $? -ne 0 ] && echo "$prog: can't ping $TGT" 1>&2 && usage

dquad_to_hex()
{
	echo $1 | sed -e "s/[()]//g" -e "s/\./ /g" | while read I0 I1 I2 I3 ; do
		printf "0x%02X%02X%02X%02X" $I0 $I1 $I2 $I3
	done
}

# output from arp -a of the form:
# good: host.domain (A.B.C.D) at 00:50:BF:06:48:C1 [ether] on eth0
# bad:  ? (A.B.C.D) at <incomplete> on eth0
arp -a | grep $TGT | { read HOSTNAME IPADDR AT MACADDR TYPE ON IFACE;
	[ "$HOSTNAME" = "?" -a -z "$MAC" -a -z "$NOMAC" ] && \
		echo "$prog: can't resolve $TGT MAC" 1>&2 && usage

	[ -z "$MAC" ] && MAC=$MACADDR
	[ -z "$DEV" ] && DEV=$IFACE
	[ "$DEV" = "$IFACE" -a "$TYPE" != "[ether]" ] && \
		echo "$prog: $DEV must be an ethernet interface" 1>&2 && usage
		
	IPHEX=`dquad_to_hex $IPADDR`
	echo $MAC | sed "s/:/ /g" | { read M0 M1 M2 M3 M4 M5;
		if [ -z "$NOMAC" ]; then
			TGTMAC="target_eth_byte0=0x$M0 target_eth_byte1=0x$M1 \
				target_eth_byte2=0x$M2 target_eth_byte3=0x$M3 \
				target_eth_byte4=0x$M4 target_eth_byte5=0x$M5"
		fi
		#insmod netconsole dev=$DEV target_ip=$IPHEX \
		echo dev=$DEV target_ip=$IPHEX \
			source_port=$PORT target_port=$TPORT $TGTMAC
	}
}

Cheers, Andreas
--
Andreas Dilger  \ "If a man ate a pound of pasta and a pound of antipasto,
                 \  would they cancel out, leaving him still hungry?"
http://www-mddsp.enel.ucalgary.ca/People/adilger/               -- Dogbert

Re: [patch] netconsole - log kernel messages over the network. 2.4.10.

[Ingo Molnar]

On Wed, 26 Sep 2001, Andreas Dilger wrote:

> Ok, I read the docs, and this is indeed a target MAC address.  It may
> still be easier to accept a regular MAC address like
> target_mac=XX:XX:XX:XX:XX:XX as the module parameter (and a
> target_ip=A.B.C.D).  In any case, here is a script to automate this
> (ugly because of the conversions needed).

thanks - i've put this script into the userspace tarball.

	Ingo

Re: [patch] netconsole - log kernel messages over the network. 2.4.10.

[Ingo Molnar]

On Wed, 26 Sep 2001, jamal wrote:

> Is there any reason you are not using dev_queue_xmit()? (side
> benefits, you could hack this into using scatter gather schemes etc)

the reason is latency of messages, and reliability of sends. If we are in
the middle of a *nasty* crash that takes down the system down 10
microseconds later, then i wanted to make 100% sure the message gets out
and the bug can be debugged. The VGA console and the serial logging
console have such properties - messages are instantaneous. Networking is
very asynchron in nature, so a new network driver extension was needed to
achieve this goal.

There are crashes where there are no new IRQs processed after the crash,
so there is no chance for dev_queue_xmit to do any useful send. We must be
able to send packets even if the output interface's hardware queues are
full - and dev_queue_xmit relies on future device interrupts to send
queued packets.

this is not just theory - about 10% of the crashes i experience (during
development...) are such, and the majority of the 'hard to fix' crashes
have such properties, especially on busy remote servers.

	Ingo

Re: [patch] netconsole - log kernel messages over the network. 2.4.10.

[Ookhoi]

Hi All,

Ingo was not aware of the sourceforge project, and suggested me to
resend my reply to lkml. Does the patch work for you guys? Do I do
something wrong? That would be more than possible. :-)


----- Forwarded message from Ookhoi <ookhoi@dds.nl> -----

To: Ingo Molnar <mingo@elte.hu>
Cc: netconsole-devel@lists.sourceforge.net

Hi Ingo,

> this is the first public release of the 'netconsole patch', a debugging
> patch that implements kernel-level network logging via UDP packets.

Is this different from the netconsole project on sourceforge? The last
message on that lists seemes to be from july, and you didn't cc it. 

> the kernel patch (against 2.4.10 or 2.4.9-ac), and a simple user-space
> tool to display netconsole messages can be found at:
>
> 	http://redhat.com/~mingo/netconsole-patches/

Can you also make older patches available for testing? The current
netconsole-2.4.10-B1 doesn't seem to work for me on plain 2.4.10 or
2.4.9-ac15 patched with Rik his latest vm patch. The patch applies fine,
but I can't load the module:

cuddle:~# uname -a
Linux cuddle 2.4.9-ac15 #1 Thu Sep 27 13:54:51 CEST 2001 i686 unknown
cuddle:~# insmod netconsole dev=eth0 target_ip=0x0a604875 source_port=6666 target_port=5555
Using /lib/modules/2.4.9-ac15/kernel/drivers/net/netconsole.o
/lib/modules/2.4.9-ac15/kernel/drivers/net/netconsole.o: init_module: Operation not permitted
Hint: insmod errors can be caused by incorrect module parameters, including invalid IO or IRQ parameters

The same message for plain 2.4.10, and the same when I use the script
netconsole-server included in netconsole-client-20010927. I also tried
it with the target_eth_byte0 thingies, with the mac address from the
local nic.

Target ip address is 10.96.72.117 (I hope 0x0a604875 is oke)

> sample startup of the netconsole on the server:
> 
>      insmod netconsole dev=eth1 target_ip=0x0a000701 \
>                   source_port=6666 \
>                   target_port=6666 \
>                   target_eth_byte0=0x00 \
>                   target_eth_byte1=0x90\
>                   target_eth_byte2=0x27 \
>                   target_eth_byte3=0x8C \
>                   target_eth_byte4=0xA0 \
>                   target_eth_byte5=0xA8
> 
> and on the client:
> 
> 	# ./netconsole-client -server 10.0.7.5 -client 10.0.7.1 -port 6666
>         [...network console startup...]
>         netconsole: network logging started up successfully!
>         SysRq : Loglevel set to 9
> 
> more about features and limitations can be found under:
> 
> 	Documentation/networking/netlogging.txt
> 
> reports, comments, suggestions welcome.

Somebody more or less asked this already, but, will things like SysRq
work in the future? That would be great. :-)  Thnx!

	Ookhoi

----- End forwarded message -----

Re: [patch] netconsole - log kernel messages over the network. 2.4.10.

[Andrew Morton]

Ookhoi wrote:
> 
> Hi All,
> 
> Ingo was not aware of the sourceforge project, and suggested me to
> resend my reply to lkml. Does the patch work for you guys? Do I do
> something wrong? That would be more than possible. :-)
> 
> ...
> cuddle:~# uname -a
> Linux cuddle 2.4.9-ac15 #1 Thu Sep 27 13:54:51 CEST 2001 i686 unknown
> cuddle:~# insmod netconsole dev=eth0 target_ip=0x0a604875 source_port=6666 target_port=5555
> Using /lib/modules/2.4.9-ac15/kernel/drivers/net/netconsole.o
> /lib/modules/2.4.9-ac15/kernel/drivers/net/netconsole.o: init_module: Operation not permitted
> Hint: insmod errors can be caused by incorrect module parameters, including invalid IO or IRQ parameters

If you're not using the eepro100 driver, then an insmod of the
netconsole driver will fail:


+       if (!ndev->poll_controller) {
+               printk(KERN_ERR "netconsole: %s's network driver does not implement netlogging yet, aborting.\n", dev);
+               return -1;
+       }

Maybe that message is in your logs somewhere?

Take a look at the poll_controller() implementation in the eepro100
part of Ingo's patch - it's dead simple.

What we need is for a bunch of people to implement poll_controller()
for *their* ethernet driver and contribute the tested diffs
back to Ingo.

-

Re: [patch] netconsole - log kernel messages over the network. 2.4.10.

[Ookhoi]

Hi Andrew,

> > Ingo was not aware of the sourceforge project, and suggested me to
> > resend my reply to lkml. Does the patch work for you guys? Do I do
> > something wrong? That would be more than possible. :-)
> > 
> > ...
> > cuddle:~# uname -a
> > Linux cuddle 2.4.9-ac15 #1 Thu Sep 27 13:54:51 CEST 2001 i686 unknown
> > cuddle:~# insmod netconsole dev=eth0 target_ip=0x0a604875 source_port=6666 target_port=5555
> > Using /lib/modules/2.4.9-ac15/kernel/drivers/net/netconsole.o
> > /lib/modules/2.4.9-ac15/kernel/drivers/net/netconsole.o: init_module: Operation not permitted
> > Hint: insmod errors can be caused by incorrect module parameters, including invalid IO or IRQ parameters
> 
> If you're not using the eepro100 driver, then an insmod of the
> netconsole driver will fail:
> 
> 
> +       if (!ndev->poll_controller) {
> +               printk(KERN_ERR "netconsole: %s's network driver does not implement netlogging yet, aborting.\n", dev);
> +               return -1;
> +       }
> 
> Maybe that message is in your logs somewhere?

Have to check that, can't reach the machine atm. But I don't use the
eepro100 driver for sure in that machine, so the insmod failure makes
sense. :-)

> Take a look at the poll_controller() implementation in the eepro100
> part of Ingo's patch - it's dead simple.

I believe you ;-)  but I'm affraid I'm just a user, no programmer at
all :-(

> What we need is for a bunch of people to implement poll_controller()
> for *their* ethernet driver and contribute the tested diffs back to
> Ingo.

Would love to contribute.. Anyway, thank you for your answer!

	Ookhoi

Re: [patch] netconsole - log kernel messages over the network. 2.4.10.

[Albert Cranford]

Great tool Ingo thanks.  Below is a tested tulip patch.
Thanks Andrew for the the inspiration.
Albert
Andrew Morton wrote:
> 
> Ookhoi wrote:
> >
> 
> What we need is for a bunch of people to implement poll_controller()
> for *their* ethernet driver and contribute the tested diffs
> back to Ingo.
> 


/etc/rc.d/rc.netconsole 192.168.1.2
dev=eth0 target_ip=0xC0A80102 source_port=6666 target_port=6666 \
        target_eth_byte0=0x00 target_eth_byte1=0x03 target_eth_byte2=0x6D \
        target_eth_byte3=0x16 target_eth_byte4=0x51 target_eth_byte5=0xAE
Using /lib/modules/2.4.10/kernel/drivers/net/netconsole.o

~~~~~~~~~~~   DMESG from server  ~~~~~~~~~~~~~~~~
Linux version 2.4.10 (root@home1) (gcc version 3.0.1) \
        #1 Thu Sep 27 22:28:20 EDT 2001
.......
Linux Tulip driver version 0.9.15-pre6 (July 2, 2001)
eth0: ADMtek Comet rev 17 at 0xdc00, 00:03:6D:16:4E:39, IRQ 10.
.........
netconsole: using network device <eth0>
netconsole: using source IP -64.-88.1.10
netconsole: using target IP -64.-88.1.2
netconsole: using source UDP port: 6666
netconsole: using target UDP port: 6666
netconsole: using target ethernet address 00:03:6d:16:51:ae.
netconsole: network logging started up successfully!
netconsole-client -server 0xc0a8010a -client 0xc0a80102 -port 6666
~~~~~~~~~ Start netconsole-client ~~~~~~~~
displaying netconsole output from server 0xc0a8010a, \
        client 0xc0a80102, UDP port 6666.
~~~~~~~~~ Alt-SysRq-s on Server output to client ~~~~~
SysRq : Emergency Sync
Syncing device 08:04 ... OK
Syncing device 08:12 ... OK
Syncing device 08:01 ... OK
Done.
~~~~~~~~~~ PATCH BEGIN ~~~~~~~~~~~~~~~
--- linux/drivers/net/tulip/tulip_core.c.orig   Thu Sep 27 21:04:14 2001
+++ linux/drivers/net/tulip/tulip_core.c        Thu Sep 27 10:10:15 2001
@@ -243,6 +243,7 @@
 static struct net_device_stats *tulip_get_stats(struct net_device *dev);
 static int private_ioctl(struct net_device *dev, struct ifreq *rq, int cmd);
 static void set_rx_mode(struct net_device *dev);
+static void poll_tulip(struct net_device *dev);
 
 
 
@@ -1671,6 +1672,9 @@
        dev->get_stats = tulip_get_stats;
        dev->do_ioctl = private_ioctl;
        dev->set_multicast_list = set_rx_mode;
+#ifdef HAVE_POLL_CONTROLLER
+       dev->poll_controller = &poll_tulip;
+#endif
 
        if (register_netdev(dev))
                goto err_out_free_ring;
@@ -1839,6 +1843,24 @@
 
        /* pci_power_off (pdev, -1); */
 }
+
+
+#ifdef HAVE_POLL_CONTROLLER
+
+/*
+ * Polling 'interrupt' - used by things like netconsole to send skbs
+ * without having to re-enable interrupts. It's not called while
+ * the interrupt routine is executing.
+ */
+
+static void poll_tulip (struct net_device *dev)
+{
+       disable_irq(dev->irq);
+       tulip_interrupt (dev->irq, dev, NULL);
+       enable_irq(dev->irq);
+}
+
+#endif
 
 
 static struct pci_driver tulip_driver = {
~~~~~~~~~~ PATCH END ~~~~~~~~~~~~~~~

-- 
Albert Cranford Deerfield Beach FL USA
ac9410@bellsouth.net

[patch] netconsole-2.4.10-C2

[Ingo Molnar]

On Thu, 27 Sep 2001, Albert Cranford wrote:

> Great tool Ingo thanks.  Below is a tested tulip patch.
> Thanks Andrew for the the inspiration.

thanks Albert - i've added it to the patch, and the latest
netconsole-2.4.10-C2 version can be downloaded from:

	http://redhat.com/~mingo/netconsole-patches/

NOTE: new client-side utilities are needed as well.

other changes:

 - netconsole-server fix from Andreas Dilger

 - introduced versioning and offsetting of output, to display messages in
   the correct order even if interim routers reorder packets. Future
   netconsole-clients should reliably detect the right protocol version.

 - small cleanups.

	Ingo

netconsole patch for 2.5?

[Barton, Christopher]

Hello,

Is there a netconsole patch for Linux 2.5?  Is netconsole on anyone's
radar for inclusion?

Thanks a lot!

On Fri, 28 Sep 2001, Ingo Molnar wrote:

> 
> On Thu, 27 Sep 2001, Albert Cranford wrote:
> 
> > Great tool Ingo thanks.  Below is a tested tulip patch.
> > Thanks Andrew for the the inspiration.
> 
> thanks Albert - i've added it to the patch, and the latest
> netconsole-2.4.10-C2 version can be downloaded from:
> 
> 	http://redhat.com/~mingo/netconsole-patches/
> 
> NOTE: new client-side utilities are needed as well.
> 
> other changes:
> 
>  - netconsole-server fix from Andreas Dilger
> 
>  - introduced versioning and offsetting of output, to display messages in
>    the correct order even if interim routers reorder packets. Future
>    netconsole-clients should reliably detect the right protocol version.
> 
>  - small cleanups.
> 
> 	Ingo
> 
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
>