Iptables logs on High bandwidth traffic network

Iptables logs on High bandwidth traffic network

[bharathi]

[-- Attachment #1: Type: text/plain, Size: 707 bytes --]

Hi all,
        I am planning to implement iptables log feature on a server
machine(Dual xeon processor,Intel e100 cards,80GB SCSI and 2GB RAM)
which is running in bridge mode (On RH 7.3).The average traffic on this
machine is vary from 40-60Mbps.Hence I require some suggestion for some
my questions like,

1) On this High traffic the kernel will be stable/crash ?
2) What would be the CPU Load and the server is able to do this job
without any pain ?
3) Up to how much traffic the iptables/kernel can able to handle without
any issue and what should I do additionally if I need the
   iptable-log should handle this much traffic?


Advance thanks for your suggestion and pointers.

bharthix.   
         

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: Iptables logs on High bandwidth traffic network

[Taylor, Grant]

> Hi all,
>         I am planning to implement iptables log feature on a server
> machine(Dual xeon processor,Intel e100 cards,80GB SCSI and 2GB RAM)
> which is running in bridge mode (On RH 7.3).The average traffic on this
> machine is vary from 40-60Mbps.Hence I require some suggestion for some
> my questions like,
> 
> 1) On this High traffic the kernel will be stable/crash ?
> 2) What would be the CPU Load and the server is able to do this job
> without any pain ?
> 3) Up to how much traffic the iptables/kernel can able to handle without
> any issue and what should I do additionally if I need the
>    iptable-log should handle this much traffic?

There have been people in the past with marginal luck (at best) who tried to have the kernel log packets via the LOG target.  The problem that I think they have run in to in the pas is that the LOG target is (as far as I know and no one has refuted (If I am wrong please do so)) not meant for high volume LOGing of packets.  If you are wanting to log all traffic that passes through the box you would probably want to look at using TCPDump to sniff the network and parse it's out files or look in to something like Snort in one of it's many modes.  The reason that LOG is not meant for high volume logging is that it relies on SysLog to log it's data which in and of it's self is not meant for high volume longing.  SysLog will quite often become disk bound if you try to log such high volumes to it and thus the system will sort of flounder and snow ball in on it's self.  Further you will not see t
 he log events that this is happening b/c they themselves will not get logged b/c SysLog i
s backed up.  You may have more luck looking at ULOG but as I have not messed with it I can't say one way or the other.

(This is all speculation on my part and I have little to no hands on experience doing this.  However I have talked with many people who have been in this situation and they back up what I'm saying.  So your mileage may vary.)



Grant. . . .

Re: Iptables logs on High bandwidth traffic network

[Mogens Valentin]

Taylor, Grant wrote:
>> Hi all,
>>         I am planning to implement iptables log feature on a server
>> machine(Dual xeon processor,Intel e100 cards,80GB SCSI and 2GB RAM)
>> which is running in bridge mode (On RH 7.3).The average traffic on this
>> machine is vary from 40-60Mbps.Hence I require some suggestion for some
>> my questions like,
>>
 > The reason that
> LOG is not meant for high volume logging is that it relies on SysLog to 
> log it's data which in and of it's self is not meant for high volume 
> longing.  SysLog will quite often become disk bound if you try to log 
> such high volumes to it and thus the system will sort of flounder...

How about using a fifo (man mkfifo and man syslog) and let syslog pipe 
to that fifo. Some program can then read from the fifo, parse data, and 
maybe use a database for storing the parsed, now more limited, data.
Might be a good ide to have the database on another system :-

-- 
Kind regards,
Mogens Valentin

Re: Iptables logs on High bandwidth traffic network

[Taylor, Grant]

> How about using a fifo (man mkfifo and man syslog) and let syslog pipe 
> to that fifo. Some program can then read from the fifo, parse data, and 
> maybe use a database for storing the parsed, now more limited, data.
> Might be a good ide to have the database on another system :-

Using a FIFO to a program that parses and transmits the data to another system to network might be a possibility.  Keep in mind that any processing that you do on the packets has to be able to be done at least as fast if not faster than the rate the packets come in.  If you ever end up getting behind on the processing things will snowball on you VERY quickly and more than likely end up in a very nasty mess.  This is why I think it would be better to use something like TCPDump or Snort to sniff the network and then post process the dumps.  This post processing could probably be done as often or seldom as you would like, this is all tunable.  The reason that I like the post processing is that you have a rather large buffer before you start snowballing on your self, namely the disk to store dumps on.

(maning mkfifo...)

After reading about FIFOs and playing with them momentarily (mkfifo test; ls > test; (jump to different terminal) cat test) I would be worried about how much data could be queued in a fifo and what would happen if more data than that was dumped in to the fifo.  I personally see too many opportunities for things to break login this way.  Login this way may indeed work but I would not want to try this, especially on a higher speed link.  Seeing as I have no idea what speed this would break on I am even less likely to try it, but that is just me and my opinion.  You know what they say about opinions...



Grant. . . .

Re: Iptables logs on High bandwidth traffic network

[Jozsef Kadlecsik]

On Wed, 4 May 2005, Taylor, Grant wrote:

> > How about using a fifo (man mkfifo and man syslog) and let syslog pipe
> > to that fifo. Some program can then read from the fifo, parse data, and
> > maybe use a database for storing the parsed, now more limited, data.
> > Might be a good ide to have the database on another system :-
>
> Using a FIFO to a program that parses and transmits the data to another
> system to network might be a possibility.  Keep in mind that any
> processing that you do on the packets has to be able to be done at least
> as fast if not faster than the rate the packets come in.  If you ever
> end up getting behind on the processing things will snowball on you VERY
> quickly and more than likely end up in a very nasty mess.  This is why I
> think it would be better to use something like TCPDump or Snort to sniff
> the network and then post process the dumps.

Why where a FIFO and a program which parses and transmit the data to
another system any faster than syslog/syslog-ng/ulogd/etc? (Why reinvent
the wheel?)

If you sniff the network for logging, where do you actually want to sniff?
Before the firewall? You'll have to cope with the same traffic as the
firewall itself and won't see the dropped traffic on the other side.
Behind the firewall? You won't see again the dropped traffic coming from
outside. (Please note, I do not advise against an IDS inside which sniffs
the traffic.)

Best regards,
Jozsef
-
E-mail  : kadlec@blackhole.kfki.hu, kadlec@sunserv.kfki.hu
PGP key : http://www.kfki.hu/~kadlec/pgp_public_key.txt
Address : KFKI Research Institute for Particle and Nuclear Physics
          H-1525 Budapest 114, POB. 49, Hungary

Re: Iptables logs on High bandwidth traffic network

[Taylor, Grant]

> Why where a FIFO and a program which parses and transmit the data to
> another system any faster than syslog/syslog-ng/ulogd/etc? (Why reinvent
> the wheel?)

It is my belief that Syslog and the mechanism that it uses to log is not meant for extreme volume of login.  As I understand it Syslog will log each and every individual packet that passes through the IPTables LOG target individually, thus causing a write through the kernel in to SysLog space and possibly to disk for a VERY small amount of data.  Where as TCPDump or Snort will dump to memory for a specific amount of time or amount of traffic captured and then write a large group of packets as one write.  The former will have an I/O action for each and every packet that comes through the kernel, where as the later will have an I/O action once every x number of thousands or millions of packets (user definable).  It is much easier / efficient for a system to write a larger chunk of data at one time than it is for it to write the same amount of data in a lot of little bitty increments.  Imag
 en moving a book shelf from the basement in a house to the upstairs bedroom.  Would it be 
easier to move one book at a time up the stairs for each and every book on the shelf or would it be easier to grab as many books as you can carry and take a bunch of books at one time?  Which would you view as more work in the long run?  There is this same type of load on systems when logging via SysLog.

(Again, this is my perception of the way that SysLog writes log entries.  If someone knows this to be incorrect please tell me other wise.)

> If you sniff the network for logging, where do you actually want to sniff?
> Before the firewall? You'll have to cope with the same traffic as the
> firewall itself and won't see the dropped traffic on the other side.
> Behind the firewall? You won't see again the dropped traffic coming from
> outside. (Please note, I do not advise against an IDS inside which sniffs
> the traffic.)

If you are wanting to log for accounting reasons to know how much traffic any person behind the system moved I would sniff on the internal side of the system.  If you are wanting to log attacks and / or traffic that is dropped by the firewall I would log on the internal and external find find the difference and thus deduce what has been dropped.  Where you want to sniff will really depend on what you are wanting to log.  If you can be more specific I'd be glad to give a more specific response.  :)



Grant. . . .

Re: Iptables logs on High bandwidth traffic network

[Jozsef Kadlecsik]

On Thu, 5 May 2005, Taylor, Grant wrote:

> > Why where a FIFO and a program which parses and transmit the data to
> > another system any faster than syslog/syslog-ng/ulogd/etc? (Why reinvent
> > the wheel?)
>
> It is my belief that Syslog and the mechanism that it uses to log is not
> meant for extreme volume of login.  As I understand it Syslog will log
> each and every individual packet that passes through the IPTables LOG
> target individually, thus causing a write through the kernel in to
> SysLog space and possibly to disk for a VERY small amount of data.

That depends on how syslog is configured - you can easily disable syncing
at every log event.

> Where as TCPDump or Snort will dump to memory for a specific amount of
> time or amount of traffic captured and then write a large group of
> packets as one write.

I have looked trough the man page of tcpdump and could not find any
indication of this feature. How should one run tcpdump in order to enable
in-memory buffering?

Best regards,
Jozsef
-
E-mail  : kadlec@blackhole.kfki.hu, kadlec@sunserv.kfki.hu
PGP key : http://www.kfki.hu/~kadlec/pgp_public_key.txt
Address : KFKI Research Institute for Particle and Nuclear Physics
          H-1525 Budapest 114, POB. 49, Hungary

Re: Iptables logs on High bandwidth traffic network

[Mogens Valentin]

Jozsef Kadlecsik wrote:
> On Thu, 5 May 2005, Taylor, Grant wrote:
> 
> 
>>>Why where a FIFO and a program which parses and transmit the data to
>>>another system any faster than syslog/syslog-ng/ulogd/etc? (Why reinvent
>>>the wheel?)
>>
>>It is my belief that Syslog and the mechanism that it uses to log is not
>>meant for extreme volume of login.  As I understand it Syslog will log
>>each and every individual packet that passes through the IPTables LOG
>>target individually, thus causing a write through the kernel in to
>>SysLog space and possibly to disk for a VERY small amount of data.
> 
> 
> That depends on how syslog is configured - you can easily disable syncing
> at every log event.

Yes, but doing so may cause loss of logging, or maybe it'll just delay 
some messages due to not sync'ing at once. Lack of experience here :p
I do use the non-sync feature to some extend; doesn't seem to cause too 
much delay, though.

-- 
Kind regards,
Mogens Valentin

Re: Iptables logs on High bandwidth traffic network

[Jozsef Kadlecsik]

On Thu, 5 May 2005, Mogens Valentin wrote:

> > That depends on how syslog is configured - you can easily disable syncing
> > at every log event.
>
> Yes, but doing so may cause loss of logging, or maybe it'll just delay
> some messages due to not sync'ing at once.

A FIFO and a buffering app suffers from exactly the same problems.

Best regards,
Jozsef
-
E-mail  : kadlec@blackhole.kfki.hu, kadlec@sunserv.kfki.hu
PGP key : http://www.kfki.hu/~kadlec/pgp_public_key.txt
Address : KFKI Research Institute for Particle and Nuclear Physics
          H-1525 Budapest 114, POB. 49, Hungary

Re: Iptables logs on High bandwidth traffic network

[Mogens Valentin]

Jozsef Kadlecsik wrote:
> On Wed, 4 May 2005, Taylor, Grant wrote:
> 
> 
>>>How about using a fifo (man mkfifo and man syslog) and let syslog pipe
>>>to that fifo. Some program can then read from the fifo, parse data, and
>>>maybe use a database for storing the parsed, now more limited, data.
>>>Might be a good ide to have the database on another system :-
>>
>>Using a FIFO to a program that parses and transmits the data to another
>>system to network might be a possibility.  Keep in mind that any
>>processing that you do on the packets has to be able to be done at least
>>as fast if not faster than the rate the packets come in.  If you ever
>>end up getting behind on the processing things will snowball on you VERY
>>quickly and more than likely end up in a very nasty mess.  This is why I
>>think it would be better to use something like TCPDump or Snort to sniff
>>the network and then post process the dumps.
> 
> 
> Why where a FIFO and a program which parses and transmit the data to
> another system any faster than syslog/syslog-ng/ulogd/etc? (Why reinvent
> the wheel?)

It might not.. AFAIK, the FIFO is implemented not as disk I/O, but is a 
memory thingy.
It *appears* as file I/O, but the filesystem is used only to create that 
named pipe.
Hence, my thought was that since logging with iptables has to go throu 
syslog, this might offload faster through a FIFO.
The app reading the FIFO would preprocess the datastream and turn it 
into chunks, exactly as Taylor put it.
Depending on what one really wants to look at / dig out of logging, the 
FIFO-reading app could also reduce data.

-- 
Kind regards,
Mogens Valentin

Re: Iptables logs on High bandwidth traffic network

[Jozsef Kadlecsik]

Hi,

On Thu, 5 May 2005, Mogens Valentin wrote:

> >>>How about using a fifo (man mkfifo and man syslog) and let syslog pipe
> >>>to that fifo. Some program can then read from the fifo, parse data, and
> >>>maybe use a database for storing the parsed, now more limited, data.
> >>>Might be a good ide to have the database on another system :-
> >>
> > Why where a FIFO and a program which parses and transmit the data to
> > another system any faster than syslog/syslog-ng/ulogd/etc? (Why reinvent
> > the wheel?)
>
> It might not.. AFAIK, the FIFO is implemented not as disk I/O, but is a
> memory thingy.
> It *appears* as file I/O, but the filesystem is used only to create that
> named pipe.
> Hence, my thought was that since logging with iptables has to go throu
> syslog, this might offload faster through a FIFO.
> The app reading the FIFO would preprocess the datastream and turn it
> into chunks, exactly as Taylor put it.
> Depending on what one really wants to look at / dig out of logging, the
> FIFO-reading app could also reduce data.

At standard syslog, you can rely on its internal (not-tunable) buffering.
At syslog-ng, you can speficy explicitly the buffer size (output queue
size) per destination. At the ULOG target, you can specify how many
packets should be queued in-kernel, before transmitting a multipart
message to userspace.

And as it was mentioned by others as well, one should offload real
logging to another machine over a dedicated wire.

Best regards,
Jozsef
-
E-mail  : kadlec@blackhole.kfki.hu, kadlec@sunserv.kfki.hu
PGP key : http://www.kfki.hu/~kadlec/pgp_public_key.txt
Address : KFKI Research Institute for Particle and Nuclear Physics
          H-1525 Budapest 114, POB. 49, Hungary

Re: Iptables logs on High bandwidth traffic network

[Jason Opperisano]

On Wed, May 04, 2005 at 06:15:18PM +0530, bharathi wrote:
> Hi all,
>         I am planning to implement iptables log feature on a server
> machine(Dual xeon processor,Intel e100 cards,80GB SCSI and 2GB RAM)
> which is running in bridge mode (On RH 7.3).The average traffic on this
> machine is vary from 40-60Mbps.Hence I require some suggestion for some
> my questions like,

60Mpbs is not a ton of traffic, i'd be interested in what the expected
packets per second (pps) rate is, as that can be the
forwarding/filtering killer.  in the extreme case, think about 60 Mbps
of 1250 byte UDP packets vs. 60 Mbps of 60-300 byte TCP traffic...big
difference.

i'd also be so bold as to suggest you'd be much better off with e1000
cards over e100 cards.  in my experience, if you can get 80 Mbps
sustained out of a 100M card; you're kicking butt, and it's not
something you'd want to do indefinitely.

> 1) On this High traffic the kernel will be stable/crash ?
> 2) What would be the CPU Load and the server is able to do this job
> without any pain ?

it's possible that the NICs could peg the CPU servicing interrupts.
there are docs out there on how to nail a specific NIC to a specific
CPU, as you'll often see all your NICs being serviced by a single CPU.

i'd also note that in my experience, on both linux and the BSD's;
bridging is *much* more CPU intensive than routing; which is probably
counter-intuitive, since from a networking perspective bridging is a
simpler operation than routing.  dunno if this has to do with
implementation issues or just that it's being done in software vs.
hardware.

> 3) Up to how much traffic the iptables/kernel can able to handle without
> any issue and what should I do additionally if I need the
>    iptable-log should handle this much traffic?

if you want to do a lot of logging, add a 3rd NIC to the box; setup a
syslog server on that segment and let the syslog server deal with the
disk I/O.

-j

--
"Chris: So, what are you wearing?  Wow. I bet you can see right through
 that.
 Lois: Chris, who are you talking to? 
 Chris: Grandma."
        --Family Guy

Re: Iptables logs on High bandwidth traffic network

[Steven M Campbell]

bharathi wrote:

>Hi all,
>        I am planning to implement iptables log feature on a server
>machine(Dual xeon processor,Intel e100 cards,80GB SCSI and 2GB RAM)
>which is running in bridge mode (On RH 7.3).The average traffic on this
>machine is vary from 40-60Mbps.Hence I require some suggestion for some
>my questions like,
>
>1) On this High traffic the kernel will be stable/crash ?
>2) What would be the CPU Load and the server is able to do this job
>without any pain ?
>3) Up to how much traffic the iptables/kernel can able to handle without
>any issue and what should I do additionally if I need the
>   iptable-log should handle this much traffic?
>
>
>Advance thanks for your suggestion and pointers.
>
>bharthix.   
>         
>  
>
Along with other advice offered, you may wish to take a look at argus 
and ntop which are packages that collect nice summary information from 
ethernet traffic.  They may be appropos to your goal.

Re: Iptables logs on High bandwidth traffic network

[Jozsef Kadlecsik]

On Wed, 4 May 2005, bharathi wrote:

>         I am planning to implement iptables log feature on a server
> machine(Dual xeon processor,Intel e100 cards,80GB SCSI and 2GB RAM)
> which is running in bridge mode (On RH 7.3).The average traffic on this
> machine is vary from 40-60Mbps.Hence I require some suggestion for some
> my questions like,

That seems to be a decent machine.

> 1) On this High traffic the kernel will be stable/crash ?

We tested netfilter in Gbit environment and could pump trough 200,000pps,
when NAT-ing all the traffic. There was no stability problem whatsoever.

> 2) What would be the CPU Load and the server is able to do this job
> without any pain ?

If there will be no exotic and expensive matches like 'string', then
there shouldn't be CPU problems. However choosing a good system bus,
nice interfaces and fastest RAM you can get will serve well.

> 3) Up to how much traffic the iptables/kernel can able to handle without
> any issue and what should I do additionally if I need the
>    iptable-log should handle this much traffic?

Set up another machine connected via a third interface directly and let
that machine fight with the disk I/O issues. Use the ULOG target instead
of LOG.  (And of course disable logging on the console :-)

Best regards,
Jozsef
-
E-mail  : kadlec@blackhole.kfki.hu, kadlec@sunserv.kfki.hu
PGP key : http://www.kfki.hu/~kadlec/pgp_public_key.txt
Address : KFKI Research Institute for Particle and Nuclear Physics
          H-1525 Budapest 114, POB. 49, Hungary