netfilter stats, info and resources usage

netfilter stats, info and resources usage

[Sandro Tosi]

Hello,
we are using quite a lot iptables and we'd like to gather some 
stats/information to "what's doing" and hopefully also an idea of the 
resources used by it (in particular cpu and ram).

Probably my google search karma is very low today, given the only 
reference to CPU usage I could find was vague refs to softirq data, and 
for stats some links like:

1. http://forums.cacti.net/about36629.html
2. http://forums.cacti.net/about26714.html
3. http://people.netfilter.org/hawk/DDoS/2010-04-12__001/list.html

3 is very interesting, Jesper: how did you generate it? :)

In general: is there a place I can look for netfilter stats and also for 
the impact it generates on the loadavg of the machine?

Regards,
Sandro

PS: please CC me, I'm not subscribed

-- 
Sandro Tosi
Product Engineer
Linux based Solutions
Hosting Products
R&D | Dada.pro
sandro.tosi@register.it

Re: netfilter stats, info and resources usage

[Jan Engelhardt]

On Wednesday 2010-10-27 16:42, Sandro Tosi wrote:

> we are using quite a lot iptables and we'd like to gather some
> stats/information to "what's doing" and hopefully also an idea of the resources
> used by it (in particular cpu and ram).
>
> 1. http://forums.cacti.net/about36629.html
> 2. http://forums.cacti.net/about26714.html
> 3. http://people.netfilter.org/hawk/DDoS/2010-04-12__001/list.html
>
> 3 is very interesting, Jesper: how did you generate it? :)

JFYI, There is a lot of conntrack in there besides routing and general 
machine and interface characteristics - not much Xtables to see.

Re: netfilter stats, info and resources usage

[Sandro Tosi]

Hello, thanks for the reply.

On 10/27/2010 06:32 PM, Jan Engelhardt wrote:
> On Wednesday 2010-10-27 16:42, Sandro Tosi wrote:
>
>> we are using quite a lot iptables and we'd like to gather some
>> stats/information to "what's doing" and hopefully also an idea of the resources
>> used by it (in particular cpu and ram).
>>
>> 1. http://forums.cacti.net/about36629.html
>> 2. http://forums.cacti.net/about26714.html
>> 3. http://people.netfilter.org/hawk/DDoS/2010-04-12__001/list.html
>>
>> 3 is very interesting, Jesper: how did you generate it? :)
>
> JFYI, There is a lot of conntrack in there besides routing and general
> machine and interface characteristics - not much Xtables to see.

I'm not sure to get your reply right, but I'm actually open to any 
statistics for KPI of iptables/netfilter/conntrack/whatever - I just 
would like to retrieve meaningfull information about netfilter "stack" 
on these machines (and graph them, but that's unimportant here).

What I'm looking is cpu usage, and actually what netfilter does after I 
add a rule to it via iptables. I think of cpu usage since I have 
recently added rules that inspects the content of pkgs (using 'string' 
module) and we'd like to understand what's the impact of that. Also, 
having meaningful information of the netfilter operations can give us a 
better understanding of the machine status/usage.

I reported those 3 links because they are actually extracting 
information from what the kernel exports about NF on /proc fs, but I 
can't seem to find any info about what those values are (f.e. 
/proc/sys/net/netfilter/nf_conntrack_count reports ~7500 conns while 
'netstat -putan | wc -l' only ~3000, why that, what's the meaning of the 
values graphed and so on).

Thanks in advance,
-- 
Sandro Tosi
Product Engineer
Linux based Solutions
Hosting Products
R&D | Dada.pro
sandro.tosi@register.it

Re: netfilter stats, info and resources usage

[Jesper Dangaard Brouer]

On Wed, 2010-10-27 at 16:42 +0200, Sandro Tosi wrote:
> we are using quite a lot iptables and we'd like to gather some 
> stats/information to "what's doing" and hopefully also an idea of the 
> resources used by it (in particular cpu and ram).
> 
> Probably my google search karma is very low today, given the only 
> reference to CPU usage I could find was vague refs to softirq data, and 
> for stats some links like:
> 
> 3. http://people.netfilter.org/hawk/DDoS/2010-04-12__001/list.html
> 
> 3 is very interesting, Jesper: how did you generate it? :)

I use the tool "rrdcollect" to sample a lot of different proc values.

See the rrdcollect.conf config here:
http://people.netfilter.org/hawk/DDoS/rrdcollect_scripts/rrdcollect.conf

The RRD data files are created by a perl script.

I have put the perl script here:
http://people.netfilter.org/hawk/DDoS/rrdcollect_scripts/rrdcollect-create-datafiles.pl

I use the tool 'drraw' to show the graph data, but I have not uploaded
those files... Don't know if its the right tool for the job, but it was
very easy to change things to start with, then when the number of graphs
grew, it sort of got annoying to maintain via drraw.

Have fun!
-- 
Med venlig hilsen / Best regards
  Jesper Brouer
  ComX Networks A/S
  Linux Network Kernel Developer
  Cand. Scient Datalog / MSc.CS
  Author of http://adsl-optimizer.dk
  LinkedIn: http://www.linkedin.com/in/brouer

Re: netfilter stats, info and resources usage

[Sandro Tosi]

Hi Jesper,

On 10/28/2010 12:28 PM, Jesper Dangaard Brouer wrote:
> On Wed, 2010-10-27 at 16:42 +0200, Sandro Tosi wrote:
>> we are using quite a lot iptables and we'd like to gather some
>> stats/information to "what's doing" and hopefully also an idea of the
>> resources used by it (in particular cpu and ram).
>>
>> Probably my google search karma is very low today, given the only
>> reference to CPU usage I could find was vague refs to softirq data, and
>> for stats some links like:
>>
>> 3. http://people.netfilter.org/hawk/DDoS/2010-04-12__001/list.html
>>
>> 3 is very interesting, Jesper: how did you generate it? :)
>
> I use the tool "rrdcollect" to sample a lot of different proc values.
>
> See the rrdcollect.conf config here:
> http://people.netfilter.org/hawk/DDoS/rrdcollect_scripts/rrdcollect.conf
>
> The RRD data files are created by a perl script.
>
> I have put the perl script here:
> http://people.netfilter.org/hawk/DDoS/rrdcollect_scripts/rrdcollect-create-datafiles.pl
>
> I use the tool 'drraw' to show the graph data, but I have not uploaded
> those files... Don't know if its the right tool for the job, but it was
> very easy to change things to start with, then when the number of graphs
> grew, it sort of got annoying to maintain via drraw.

Thanks a lot for sharing, I'll definitely look at this! but... is there 
some docs about what those values are? :) If I look at kernel 
Documentation/filesystems/proc.txt there's almost no doc about /proc/net 
let alone /proc/net/stat or conntrack.

Thanks,
-- 
Sandro Tosi
Product Engineer
Linux based Solutions
Hosting Products
R&D | Dada.pro
sandro.tosi@register.it