Change iptables log format

Change iptables log format

[Lasse B. Jensen]

Is it possible to change the log format of iptables?

ala: 

from

Oct  2 16:39:44 charlie kernel: LOGPREFIX IN=eth0 OUT=eth1 
SRC=172.17.5.184 DST=80.60.235.54 LEN=48 TOS=0x00 PREC=0x00 TTL=123 
ID=52255 DF PROTO=TCP SPT=3240 DPT=1214 WINDOW=44032 RES=0x00 SYN URGP=0

to

Oct  2 16:39:44 charlie kernel: NEW:bred-kirstine:IN=eth0 OUT=eth1 
SRC=172.17.5.184 DST=80.60.235.54 PROTO=TCP SPT=3240 DPT=1214


--
Lasse B. Jensen

Re: Change iptables log format

[Harald Welte]

[-- Attachment #1: Type: text/plain, Size: 701 bytes --]

On Thu, Oct 02, 2003 at 04:40:05PM +0200, Lasse B. Jensen wrote:
> Is it possible to change the log format of iptables?

not without modifying the sourcecode of ipt_LOG.c

you can alternatively start using ulogd, where you only have to modify
userspace code if you want to log in a special format...

> Lasse B. Jensen

-- 
- Harald Welte <laforge@netfilter.org>             http://www.netfilter.org/
============================================================================
  "Fragmentation is like classful addressing -- an interesting early
   architectural error that shows how much experimentation was going
   on while IP was being designed."                    -- Paul Vixie

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: Change iptables log format (would be a nice feature)

[Chris Brenton]

On Thu, 2003-10-02 at 10:40, Lasse B. Jensen wrote:
>
> Is it possible to change the log format of iptables?

As others have pointed out this is not possible, however it might make a
nice feature to be added in. I've noticed that in high bandwidth
environments what chokes throughput the most is logging. This
materializes as the boxes throughput topping out quicker as well as
garbled/partial log entries being written.

I *strongly* feel that one of Netfilter's biggest strengths is the level
of detail in the logs and would hate to see that change. When your
pushing high speeds however, your choices come down to collecting
verbose info (and thus limiting throughput) or not collecting log
entries. An option that permits a terse log format (say IPs, ports &
transport only) might be a nice balance.

Just my $.02,
Chris

Re: Change iptables log format (would be a nice feature)

[Ray Leach]

[-- Attachment #1: Type: text/plain, Size: 1354 bytes --]

On Mon, 2003-10-06 at 18:12, Chris Brenton wrote:
> On Thu, 2003-10-02 at 10:40, Lasse B. Jensen wrote:
> >
> > Is it possible to change the log format of iptables?
> 
What about using something like ULOG?

AFAIR you can use ULOG (and ulogd) to 'redirect' the log to almost any
other format, including a SQL database.

> As others have pointed out this is not possible, however it might make a
> nice feature to be added in. I've noticed that in high bandwidth
> environments what chokes throughput the most is logging. This
> materializes as the boxes throughput topping out quicker as well as
> garbled/partial log entries being written.
> 
> I *strongly* feel that one of Netfilter's biggest strengths is the level
> of detail in the logs and would hate to see that change. When your
> pushing high speeds however, your choices come down to collecting
> verbose info (and thus limiting throughput) or not collecting log
> entries. An option that permits a terse log format (say IPs, ports &
> transport only) might be a nice balance.
> 
> Just my $.02,
> Chris
> 
-- 
--
Raymond Leach <raymondl@knowledgefactory.co.za>
Network Support Specialist
http://www.knowledgefactory.co.za
"lynx -source http://www.rchq.co.za/raymondl.asc | gpg --import"
Key fingerprint = 7209 A695 9EE0 E971 A9AD  00EE 8757 EE47 F06F FB28
--

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: Change iptables log format (would be a nice feature)

[Harald Welte]

[-- Attachment #1: Type: text/plain, Size: 860 bytes --]

On Mon, Oct 06, 2003 at 12:12:26PM -0400, Chris Brenton wrote:
> On Thu, 2003-10-02 at 10:40, Lasse B. Jensen wrote:
> >
> > Is it possible to change the log format of iptables?
> 
> As others have pointed out this is not possible, however it might make a
> nice feature to be added in. I've noticed that in high bandwidth
> environments what chokes throughput the most is logging. 

This is why you should use ULOG / ulogd if you log many packets.

> Just my $.02,
> Chris

-- 
- Harald Welte <laforge@netfilter.org>             http://www.netfilter.org/
============================================================================
  "Fragmentation is like classful addressing -- an interesting early
   architectural error that shows how much experimentation was going
   on while IP was being designed."                    -- Paul Vixie

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: Change iptables log format (would be a nice feature)

[Lasse B. Jensen]

On Tue, 7 Oct 2003, Harald Welte wrote:

> On Mon, Oct 06, 2003 at 12:12:26PM -0400, Chris Brenton wrote:
> > On Thu, 2003-10-02 at 10:40, Lasse B. Jensen wrote:
> > >
> > > Is it possible to change the log format of iptables?
> > 
> > As others have pointed out this is not possible, however it might make a
> > nice feature to be added in. I've noticed that in high bandwidth
> > environments what chokes throughput the most is logging. 
> 
> This is why you should use ULOG / ulogd if you log many packets.

I have now tried with ulog and ulogd and got a much better performance.... 
however i seem that the timestamp i ulog i "fucked up". My time on the 
machine i fine, but the ulog timestamp says Jan 12 00.04.02 when the 
date-output says Okt 8 11.43.22 

Can anyone give me an explanation?

/Lasse

> 
> > Just my $.02,
> > Chris
> 
> -- 
> - Harald Welte <laforge@netfilter.org>             http://www.netfilter.org/
> ============================================================================
>   "Fragmentation is like classful addressing -- an interesting early
>    architectural error that shows how much experimentation was going
>    on while IP was being designed."                    -- Paul Vixie
> 

Re: Change iptables log format (would be a nice feature)

[Harald Welte]

[-- Attachment #1: Type: text/plain, Size: 1481 bytes --]

On Wed, Oct 08, 2003 at 12:11:46PM +0200, Lasse B. Jensen wrote:
> 
> On Tue, 7 Oct 2003, Harald Welte wrote:
> 
> > On Mon, Oct 06, 2003 at 12:12:26PM -0400, Chris Brenton wrote:
> > > On Thu, 2003-10-02 at 10:40, Lasse B. Jensen wrote:
> > > >
> > > > Is it possible to change the log format of iptables?
> > > 
> > > As others have pointed out this is not possible, however it might make a
> > > nice feature to be added in. I've noticed that in high bandwidth
> > > environments what chokes throughput the most is logging. 
> > 
> > This is why you should use ULOG / ulogd if you log many packets.
> 
> I have now tried with ulog and ulogd and got a much better performance.... 
> however i seem that the timestamp i ulog i "fucked up". My time on the 
> machine i fine, but the ulog timestamp says Jan 12 00.04.02 when the 
> date-output says Okt 8 11.43.22 
> 
> Can anyone give me an explanation?

the answer is in the ulogd@lists.gnumonks.org mailinglist archive, there
is also a patch for it (in ulogd CVS).  I'm about to release ulogd-1.02
because of that bug.

> /Lasse

-- 
- Harald Welte <laforge@netfilter.org>             http://www.netfilter.org/
============================================================================
  "Fragmentation is like classful addressing -- an interesting early
   architectural error that shows how much experimentation was going
   on while IP was being designed."                    -- Paul Vixie

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: Change iptables log format (would be a nice feature)

[Lasse B. Jensen]

Thanks very much

--
Lasse B. Jensen

On Wed, 8 Oct 2003, Harald Welte wrote:

> On Wed, Oct 08, 2003 at 12:11:46PM +0200, Lasse B. Jensen wrote:
> > 
> > On Tue, 7 Oct 2003, Harald Welte wrote:
> > 
> > > On Mon, Oct 06, 2003 at 12:12:26PM -0400, Chris Brenton wrote:
> > > > On Thu, 2003-10-02 at 10:40, Lasse B. Jensen wrote:
> > > > >
> > > > > Is it possible to change the log format of iptables?
> > > > 
> > > > As others have pointed out this is not possible, however it might make a
> > > > nice feature to be added in. I've noticed that in high bandwidth
> > > > environments what chokes throughput the most is logging. 
> > > 
> > > This is why you should use ULOG / ulogd if you log many packets.
> > 
> > I have now tried with ulog and ulogd and got a much better performance.... 
> > however i seem that the timestamp i ulog i "fucked up". My time on the 
> > machine i fine, but the ulog timestamp says Jan 12 00.04.02 when the 
> > date-output says Okt 8 11.43.22 
> > 
> > Can anyone give me an explanation?
> 
> the answer is in the ulogd@lists.gnumonks.org mailinglist archive, there
> is also a patch for it (in ulogd CVS).  I'm about to release ulogd-1.02
> because of that bug.
> 
> > /Lasse
> 
> -- 
> - Harald Welte <laforge@netfilter.org>             http://www.netfilter.org/
> ============================================================================
>   "Fragmentation is like classful addressing -- an interesting early
>    architectural error that shows how much experimentation was going
>    on while IP was being designed."                    -- Paul Vixie
> 

Re: Change iptables log format (would be a nice feature)

[Peter Marshall]

I wrote a sed script to make my logs viewable ( however I had to download a
newer version of sed than the one that came with rh8 as the precompiled
version of sed would not support the ability to pipe the output of a tail -f
to sed.  See my command line command below.

----------------------------------------------------------------------------
-----------------------------
#!/bin/sh
dirc=/var/log/messages
msed="/usr/local/bin/sed -e"
tail -f $dirc |$msed 's/ kernel[^ ]*//;s/ MAC[^ ]*//;s/ LEN=.*PROTO/
PROTO/;s/ CODE=.*//;s/ WINDOW=.*//;G'
----------------------------------------------------------------------------
--------------------------------

When you run this script you will get output something like this
Oct  7 12:42:03 myserver DROP cnet-cdmz: IN=eth2 OUT=eth1 SRC=209.6.195.178
DST=192.168.10.2 PROTO=TCP SPT=3033 DPT=135

----- Original Message ----- 
From: "Chris Brenton" <cbrenton@chrisbrenton.org>
To: "Lasse B. Jensen" <gymer@odense.kollegienet.dk>
Cc: <netfilter@lists.netfilter.org>
Sent: Monday, October 06, 2003 1:12 PM
Subject: Re: Change iptables log format (would be a nice feature)


> On Thu, 2003-10-02 at 10:40, Lasse B. Jensen wrote:
> >
> > Is it possible to change the log format of iptables?
>
> As others have pointed out this is not possible, however it might make a
> nice feature to be added in. I've noticed that in high bandwidth
> environments what chokes throughput the most is logging. This
> materializes as the boxes throughput topping out quicker as well as
> garbled/partial log entries being written.
>
> I *strongly* feel that one of Netfilter's biggest strengths is the level
> of detail in the logs and would hate to see that change. When your
> pushing high speeds however, your choices come down to collecting
> verbose info (and thus limiting throughput) or not collecting log
> entries. An option that permits a terse log format (say IPs, ports &
> transport only) might be a nice balance.
>
> Just my $.02,
> Chris
>
>
>