RE: real-time monitor question

RE: real-time monitor question

[George Vieira]

Why not put all DROP rules into a -N DROPFILTER chain and then all DROP rules created jump to this table.
This way it's easier to get a view of any changes.

Then what you'd have to do is write a simple program which talks to your parallel port and lists the DROPFILTER chain and compares it's values to the previous set of values gathered and do something when it sees a change. easier said than done but if it's not too critical for time (every second it checks for changes) then it's possible even with scripting.

C++ code is best for this as it's smaller and faster than say running a PHP script like I do at the moment to create my MRTG graphs based on what rules have as a byte count..
eg.
Title[smtp-link]: Traffic Analysis for Incomming SMTP
Target[smtp-link]: `/bin/mysql-iptables.php --ifin eth1 --prot tcp --dport 25;/bin/mysql-iptables.php --ifout eth1 --prot tcp --dport 25`

This is getting off topic really as it's not really an iptables problem as such..

Thanks,
____________________________________________
George Vieira
Systems Manager
georgev@citadelcomputer.com.au

Citadel Computer Systems Pty Ltd
http://www.citadelcomputer.com.au
 

-----Original Message-----
From: Jeffrey D. Brower [mailto:jeff@pointhere.net]
Sent: Tuesday, August 05, 2003 11:32 AM
To: George Vieira
Cc: netfilter@lists.netfilter.org
Subject: Re: real-time monitor question


Hi George!

Right now I would be happy if I could just make the led blink when a packet
is ACCEPTed or DENYed!    ;-)

Seriously, thinking about it in those terms, if I knew once every second (or
some fraction of a second preferably) that X packets were allowed and Y
packets were denied on interface Z I could then "semi-real-time" fake it to
the interface with the same results over the period.   I am not really
interested in the number of bytes (right now) just in packets.  I was
thinking 100% real-time originally, but I am concerned about the overhead
that would introduce.  If I could get to 100% real-time, it would be great -
but your question makes me reconsider if that is even reasonable (or any
more valuable, actually).

    Jeff

----- Original Message -----
From: "George Vieira" <georgev@citadelcomputer.com.au>
To: "Jeffrey D. Brower" <jeff@pointhere.net>
Sent: Monday, August 04, 2003 5:59 PM
Subject: RE: real-time monitor question


> what kind of output did you expect? bytes dropped, packets
dropped,etc.etc..?
>
> Does it matter if it's 100% realtime or read the data every
second..etc.etc..?
>
> Thanks,
> ____________________________________________
> George Vieira
> Systems Manager
> georgev@citadelcomputer.com.au
>
> Citadel Computer Systems Pty Ltd
> http://www.citadelcomputer.com.au
>
> Phone   : +61 2 9955 2644
> HelpDesk: +61 2 9955 2698
>
>
> -----Original Message-----
> From: Jeffrey D. Brower [mailto:jeff@pointhere.net]
> Sent: Tuesday, August 05, 2003 3:31 AM
> To: netfilter@lists.netfilter.org
> Subject: real-time monitor question
>
>
> Greetings All,
>
> I need to get something that I think is probably quite simple from the
> firewall - but I don't have a clue how to exactly accomplish this.  If you
> can help, please do!
>
> I have a circuit board (hooked up to a box running netfilter/iptables)
which
> counts and displays the data sent to it via the parallel port.  The object
> is to display, in real-time, the packets on each interface that are
accepted
> and denied on a packet by packet basis.
>
> I trust netfilter and I don't want to interfere with its operation in any
> way and try to duplicate it's logic anywhere and it looks like the
userspace
> options might force me to do this to get what I need, but I don't really
> know if it will or even if this is an option.  I am not eager to actually
> queue packets myself - since I am sure to not be nearly as efficient.
>
> I found a gnumonks.org project called ulogd that seems like it _could_ be
a
> solution for me but I know nothing about it, including if I can get ACCEPT
> and DENY, by interface, by packet, buffered from it.
>
> It seemed to me that I can jump to tables for ACCEPT1 - ACCEPTn and the
same
> for DENY1 - DENYn for each of the interfaces and use the log function in
> some way - but using the log for each packet seems nightmarish to me.
>
> It occurs to me that there might be something I can do with the /proc
files.
>
> It also seems to me that any program I write that gets the info from the
> firewall might have to do a sleep to await the logic in the board to
process
> and so I might have to buffer the information from the firewall to avoid
> slowing it down or do some kind of round-robin sort of stack as long as
the
> stack is larger than the potential input flood.  It may be that I do not
> need to actually keep track of every single solitary ACCEPT but I surely
> need every DENY.
>
> I learn best by example, and I can not find any examples of this
anywhere -
> but I know people do monitoring.  I have been in the Docs and I have
looked
> in the archives of this list (but they are not searchable).  If you can
help
> me understand what I need to do - please help!
>
>
>      Jeff
>

Re: real-time monitor question

[Jeffrey D. Brower]

> Why not put all DROP rules into a -N DROPFILTER chain and then all DROP
rules
> created jump to this table.

This is kind of what I had in mind when I was saying the ACCEPT1 - ACCEPTn
table names (chains) for each interface so I could keep track of the ACCEPT
and DENY on an interface.

> Then what you'd have to do is write a simple program which talks to your
parallel port

No problem there.

> and lists the DROPFILTER chain and compares it's values to the previous
set
> of values gathered and do something when it sees a change.

Ah!  There's the rub!  I have no idea how to do that.

> C++ code is best for this as it's smaller and faster than say running a
PHP script

Agreed.

I also see your idea of using the byte count reather than the packet count.
Obviously, if the bytes denied are greater than zero, at least one packet
has been DENYed.  If I use that to drive the LEDs it looks like real time
and when I look in the actual log I can have it smaller and more usefull (as
in not every packet logged).

Re: real-time monitor question

[Jeffrey D. Brower]

Hi George!

Right now I would be happy if I could just make the led blink when a packet
is ACCEPTed or DENYed!    ;-)

Seriously, thinking about it in those terms, if I knew once every second (or
some fraction of a second preferably) that X packets were allowed and Y
packets were denied on interface Z I could then "semi-real-time" fake it to
the interface with the same results over the period.   I am not really
interested in the number of bytes (right now) just in packets.  I was
thinking 100% real-time originally, but I am concerned about the overhead
that would introduce.  If I could get to 100% real-time, it would be great -
but your question makes me reconsider if that is even reasonable (or any
more valuable, actually).

    Jeff

----- Original Message -----
From: "George Vieira" <georgev@citadelcomputer.com.au>
To: "Jeffrey D. Brower" <jeff@pointhere.net>
Sent: Monday, August 04, 2003 5:59 PM
Subject: RE: real-time monitor question


> what kind of output did you expect? bytes dropped, packets
dropped,etc.etc..?
>
> Does it matter if it's 100% realtime or read the data every
second..etc.etc..?
>
> Thanks,
> ____________________________________________
> George Vieira
> Systems Manager
> georgev@citadelcomputer.com.au
>
> Citadel Computer Systems Pty Ltd
> http://www.citadelcomputer.com.au
>
> Phone   : +61 2 9955 2644
> HelpDesk: +61 2 9955 2698
>
>
> -----Original Message-----
> From: Jeffrey D. Brower [mailto:jeff@pointhere.net]
> Sent: Tuesday, August 05, 2003 3:31 AM
> To: netfilter@lists.netfilter.org
> Subject: real-time monitor question
>
>
> Greetings All,
>
> I need to get something that I think is probably quite simple from the
> firewall - but I don't have a clue how to exactly accomplish this.  If you
> can help, please do!
>
> I have a circuit board (hooked up to a box running netfilter/iptables)
which
> counts and displays the data sent to it via the parallel port.  The object
> is to display, in real-time, the packets on each interface that are
accepted
> and denied on a packet by packet basis.
>
> I trust netfilter and I don't want to interfere with its operation in any
> way and try to duplicate it's logic anywhere and it looks like the
userspace
> options might force me to do this to get what I need, but I don't really
> know if it will or even if this is an option.  I am not eager to actually
> queue packets myself - since I am sure to not be nearly as efficient.
>
> I found a gnumonks.org project called ulogd that seems like it _could_ be
a
> solution for me but I know nothing about it, including if I can get ACCEPT
> and DENY, by interface, by packet, buffered from it.
>
> It seemed to me that I can jump to tables for ACCEPT1 - ACCEPTn and the
same
> for DENY1 - DENYn for each of the interfaces and use the log function in
> some way - but using the log for each packet seems nightmarish to me.
>
> It occurs to me that there might be something I can do with the /proc
files.
>
> It also seems to me that any program I write that gets the info from the
> firewall might have to do a sleep to await the logic in the board to
process
> and so I might have to buffer the information from the firewall to avoid
> slowing it down or do some kind of round-robin sort of stack as long as
the
> stack is larger than the potential input flood.  It may be that I do not
> need to actually keep track of every single solitary ACCEPT but I surely
> need every DENY.
>
> I learn best by example, and I can not find any examples of this
anywhere -
> but I know people do monitoring.  I have been in the Docs and I have
looked
> in the archives of this list (but they are not searchable).  If you can
help
> me understand what I need to do - please help!
>
>
>      Jeff
>

real-time monitor question

[Jeffrey D. Brower]

Greetings All,

I need to get something that I think is probably quite simple from the
firewall - but I don't have a clue how to exactly accomplish this.  If you
can help, please do!

I have a circuit board (hooked up to a box running netfilter/iptables) which
counts and displays the data sent to it via the parallel port.  The object
is to display, in real-time, the packets on each interface that are accepted
and denied on a packet by packet basis.

I trust netfilter and I don't want to interfere with its operation in any
way and try to duplicate it's logic anywhere and it looks like the userspace
options might force me to do this to get what I need, but I don't really
know if it will or even if this is an option.  I am not eager to actually
queue packets myself - since I am sure to not be nearly as efficient.

I found a gnumonks.org project called ulogd that seems like it _could_ be a
solution for me but I know nothing about it, including if I can get ACCEPT
and DENY, by interface, by packet, buffered from it.

It seemed to me that I can jump to tables for ACCEPT1 - ACCEPTn and the same
for DENY1 - DENYn for each of the interfaces and use the log function in
some way - but using the log for each packet seems nightmarish to me.

It occurs to me that there might be something I can do with the /proc files.

It also seems to me that any program I write that gets the info from the
firewall might have to do a sleep to await the logic in the board to process
and so I might have to buffer the information from the firewall to avoid
slowing it down or do some kind of round-robin sort of stack as long as the
stack is larger than the potential input flood.  It may be that I do not
need to actually keep track of every single solitary ACCEPT but I surely
need every DENY.

I learn best by example, and I can not find any examples of this anywhere -
but I know people do monitoring.  I have been in the Docs and I have looked
in the archives of this list (but they are not searchable).  If you can help
me understand what I need to do - please help!


     Jeff

Re: real-time monitor question

[Chris Wilson]

Hi Jeffrey,

> I have a circuit board (hooked up to a box running netfilter/iptables) which
> counts and displays the data sent to it via the parallel port.  The object
> is to display, in real-time, the packets on each interface that are accepted
> and denied on a packet by packet basis.
> 
> I trust netfilter and I don't want to interfere with its operation in any
> way and try to duplicate it's logic anywhere and it looks like the userspace
> options might force me to do this to get what I need, but I don't really
> know if it will or even if this is an option.  I am not eager to actually
> queue packets myself - since I am sure to not be nearly as efficient.
> 
> I found a gnumonks.org project called ulogd that seems like it _could_ be a
> solution for me but I know nothing about it, including if I can get ACCEPT
> and DENY, by interface, by packet, buffered from it.

I think that ulogd, and the ULOG target which feeds it, are the best 
solution in this case.

> It seemed to me that I can jump to tables for ACCEPT1 - ACCEPTn and the same
> for DENY1 - DENYn for each of the interfaces and use the log function in
> some way - but using the log for each packet seems nightmarish to me.

Why is that? It seems to be the best way to distinguish between packets 
being dropped in different places. If you only want a graph of ALL packets 
accepted and ALL packets dropped, then just create a couple of chains:

  iptables -N my_accept
  iptables -A my_accept -j ULOG --ulog-prefix "ACCEPT"
  iptables -A my_accept -j ACCEPT

  iptables -N my_drop
  iptables -A my_drop -j ULOG --ulog-prefix "DROP"
  iptables -A my_drop -j DROP

Then you can change ACCEPT and DROP in your firewall rules to my_accept 
and my_drop resppectively. From your user-space application, which could 
most easily be based on ulogd, you can read the log prefix of each packet 
you receive to determine whether that packet was accepted or dropped.

> It occurs to me that there might be something I can do with the /proc files.

As far as I know there are no counters of ACCEPTed/DROPped packets in 
/proc, but you could run "iptables -L -n -v" and parse the output to see 
the number of packets dropped at each rule.

> It also seems to me that any program I write that gets the info from the
> firewall might have to do a sleep to await the logic in the board to process
> and so I might have to buffer the information from the firewall to avoid
> slowing it down or do some kind of round-robin sort of stack as long as the
> stack is larger than the potential input flood.  It may be that I do not
> need to actually keep track of every single solitary ACCEPT but I surely
> need every DENY.

Maybe have a process which fork()s with either end of a pipe(). One 
process reads from Netfilter and writes to the pipe every so often (with a 
non-blocking write), while the other process reads from the pipe and 
writes to the hardware.

Hope this helps!

Cheers, Chris.
-- 
   ___ __     _
 / __// / ,__(_)_  | Chris Wilson -- UNIX Firewall Lead Developer |
/ (_ / ,\/ _/ /_ \ | NetServers.co.uk http://www.netservers.co.uk |
\ _//_/_/_//_/___/ | 21 Signet Court, Cambridge, UK. 01223 576516 |

Re: real-time monitor question

[Jeffrey D. Brower]

> I think that ulogd, and the ULOG target which feeds it, are the best
> solution in this case.

Based on that statement I got more deeply into the ULOG and it looks good.
I found an extention in the CVS called ulogd_OPRINT.c which (although I can
not find the documentation on its use) LOOKS like it is almost exactly what
I need.  Can it be that my solution could be as simple as replacing the
_output_print() function with my own function that prints to the circuit
board?


> Why is that? It seems to be the best way to distinguish between packets
> being dropped in different places.

I was really refering to the horrors of a huge syslog as each packet gets
logged and reading that log file to drive the circuit board and all the
maintenance nightmares with it.


>  If you only want a graph of ALL packets
> accepted and ALL packets dropped, then just create a couple of chains:
>
>   iptables -N my_accept
>   iptables -A my_accept -j ULOG --ulog-prefix "ACCEPT"
>   iptables -A my_accept -j ACCEPT
>
>   iptables -N my_drop
>   iptables -A my_drop -j ULOG --ulog-prefix "DROP"
>   iptables -A my_drop -j DROP

This confused me until I realized that the ULOG returns the packets for
further processing.  I had (in error) thought that I would have to deal with
the packet resolution in userspace.


> > It occurs to me that there might be something I can do with the /proc
files.
>
> As far as I know there are no counters of ACCEPTed/DROPped packets in
/proc

Another suggestion that I got kind of steered me toward lookng at bytes
rather than packets and I wonder if I can't just read one of the /proc files
to see the number of bytes accepted and denyed - the assumption being that I
could log bytes to the board and let the log files watch for connections
keeping the size of the log file down while still presenting a real-time
interface for the operator via the circuit board.  As I understand it, I can
read a /proc file 10 times a second with little impact on the system (as
bizzare as that sounds to me).


> but you could run "iptables -L -n -v" and parse the output to see  the
number of
> packets dropped at each rule.

seems like a lot of overhead.


> Maybe have a process which fork()s with either end of a pipe(). One
> process reads from Netfilter and writes to the pipe every so often (with a
> non-blocking write), while the other process reads from the pipe and
> writes to the hardware.

Never having done a fork(), nor a pipe() nor reads from Netfilter - this
sounds like a lot of fun to me (but like a heck of a learning curve).  It
sounds like what I envisioned, but I wonder if the pipe won't get stuffed
too fast and halt the input side causing lost packets, etc.  It really seems
like this exactly what I need to code in the  _output_print() function of
the ulogd_OPRINT.c program - but again, how do I keep it from over-running
the pipe?


> Hope this helps!

Yes it did.

Re: real-time monitor question

[Chris Wilson]

Hi Jeffrey,

> Based on that statement I got more deeply into the ULOG and it looks good.
> I found an extention in the CVS called ulogd_OPRINT.c which (although I can
> not find the documentation on its use) LOOKS like it is almost exactly what
> I need.  Can it be that my solution could be as simple as replacing the
> _output_print() function with my own function that prints to the circuit
> board?

It could certainly be that easy, but I'm not familiar with the code in 
question.

> Another suggestion that I got kind of steered me toward lookng at bytes
> rather than packets and I wonder if I can't just read one of the /proc files
> to see the number of bytes accepted and denyed

Unfortunately I don't think there are any counters for this either in 
/proc. You would still have to use libiptc or parse the output of 
iptables -L -n -v.

> Never having done a fork(), nor a pipe() nor reads from Netfilter - this
> sounds like a lot of fun to me (but like a heck of a learning curve).  It
> sounds like what I envisioned, but I wonder if the pipe won't get stuffed
> too fast and halt the input side causing lost packets, etc.

If you set the O_NONBLOCK option on your end of the pipe, then if the pipe 
is full the operating system will just return an EAGAIN error immediately, 
instead of waiting for the pipe to empty. In any case, it doesn't matter 
much if the input process blocks, since packet processing will continue 
anyway without interference from your process.

Cheers, Chris.
-- 
   ___ __     _
 / __// / ,__(_)_  | Chris Wilson -- UNIX Firewall Lead Developer |
/ (_ / ,\/ _/ /_ \ | NetServers.co.uk http://www.netservers.co.uk |
\ _//_/_/_//_/___/ | 21 Signet Court, Cambridge, UK. 01223 576516 |

Re: real-time monitor question

[Jeffrey D. Brower]

Hi Chris,

You are a big help again!  Thanks for your replies.

> If you set the O_NONBLOCK option on your end of the pipe, then if the pipe
> is full the operating system will just return an EAGAIN error immediately,
> instead of waiting for the pipe to empty. In any case, it doesn't matter
> much if the input process blocks, since packet processing will continue
> anyway without interference from your process.

Very cool.  I would want to continue processing packets even if my process
was falling behind so that works for me.

Let me be sure I understand all my input from the list:  (1) By setting the
O_NONBLOCK option on my end of the pipe, all I would loose would be the
information on the packets that hit while the pipe is full,  (2) but the
packets that hit while the pipe was full would process correctly anyway,
and (3) the input process can block and the output process nonblock so that
the packet filter can run full speed and I can potentially not miss anything
since the input side blocks and give me more time.

Do you know if ULOG can send an entry once a second with a summary of # of
packets that hit rules which would give me fewer reads on the pipe?

    Jeff

Re: real-time monitor question

[Chris Wilson]

Hi Jeffrey,

> Let me be sure I understand all my input from the list:  (1) By setting the
> O_NONBLOCK option on my end of the pipe

By the way, you would have both ends of this particular pipe, since its 
purpose is to isolate the part of your application which reads from 
Netfilter, from the part which writes to the display. It may not even be 
necessary since, as you say:

> all I would loose would be the information on the packets that hit while
> the pipe is full

Yes, I believe so.

> (2) but the packets that hit while the pipe was full would process
> correctly anyway

Yes, it's my understanding that with the ULOG target, unlike QUEUE, 
Netfilter will not wait for your application, or even care what it does, 
as far as the processing of the packet which hit the ULOG target is 
concerned.

> (3) the input process can block and the output process nonblock so that
> the packet filter can run full speed and I can potentially not miss
> anything since the input side blocks and give me more time.

Yes, that's right.

> Do you know if ULOG can send an entry once a second with a summary of # of
> packets that hit rules which would give me fewer reads on the pipe?

As far as I know, it cannot.

Cheers, Chris.
-- 
   ___ __     _
 / __// / ,__(_)_  | Chris Wilson -- UNIX Firewall Lead Developer |
/ (_ / ,\/ _/ /_ \ | NetServers.co.uk http://www.netservers.co.uk |
\ _//_/_/_//_/___/ | 21 Signet Court, Cambridge, UK. 01223 576516 |