Re: Iptables logs on High bandwidth traffic network

[Mogens Valentin <monz@danbbs.dk>]

Jozsef Kadlecsik wrote:
> On Wed, 4 May 2005, Taylor, Grant wrote:
> 
> 
>>>How about using a fifo (man mkfifo and man syslog) and let syslog pipe
>>>to that fifo. Some program can then read from the fifo, parse data, and
>>>maybe use a database for storing the parsed, now more limited, data.
>>>Might be a good ide to have the database on another system :-
>>
>>Using a FIFO to a program that parses and transmits the data to another
>>system to network might be a possibility.  Keep in mind that any
>>processing that you do on the packets has to be able to be done at least
>>as fast if not faster than the rate the packets come in.  If you ever
>>end up getting behind on the processing things will snowball on you VERY
>>quickly and more than likely end up in a very nasty mess.  This is why I
>>think it would be better to use something like TCPDump or Snort to sniff
>>the network and then post process the dumps.
> 
> 
> Why where a FIFO and a program which parses and transmit the data to
> another system any faster than syslog/syslog-ng/ulogd/etc? (Why reinvent
> the wheel?)

It might not.. AFAIK, the FIFO is implemented not as disk I/O, but is a 
memory thingy.
It *appears* as file I/O, but the filesystem is used only to create that 
named pipe.
Hence, my thought was that since logging with iptables has to go throu 
syslog, this might offload faster through a FIFO.
The app reading the FIFO would preprocess the datastream and turn it 
into chunks, exactly as Taylor put it.
Depending on what one really wants to look at / dig out of logging, the 
FIFO-reading app could also reduce data.

-- 
Kind regards,
Mogens Valentin