per host accounting

per host accounting

[Yucong Sun (叶雨飞)]

Hi,

I need a way to account traffic (bytes) for ~500 ips (fixed),  and it
seems creating a plain 500 rules will affect the performance a lot.
Without implement layered rule (like a binary search?) , is there
something existing to do automatic hashing?
Things like hashlimit is great, but I don't need limit matching
function, just a way to create a hashtable and count bytes and
packets.

If there's none, I suppose it would easy enough to fork some hashlimit
code to do this.

Cheers.

Re: per host accounting

[Tom van Leeuwen]

Sounds like you want pmacct

On 07/23/2012 05:22 AM, Yucong Sun (叶雨飞) wrote:
> Hi,
>
> I need a way to account traffic (bytes) for ~500 ips (fixed),  and it
> seems creating a plain 500 rules will affect the performance a lot.
> Without implement layered rule (like a binary search?) , is there
> something existing to do automatic hashing?
> Things like hashlimit is great, but I don't need limit matching
> function, just a way to create a hashtable and count bytes and
> packets.
>
> If there's none, I suppose it would easy enough to fork some hashlimit
> code to do this.
>
> Cheers.
> --
> To unsubscribe from this list: send the line "unsubscribe netfilter" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: per host accounting

[Eric Leblond]

[-- Attachment #1: Type: text/plain, Size: 848 bytes --]

Hello,

Le dimanche 22 juillet 2012 à 20:22 -0700, Yucong Sun (叶雨飞) a écrit :
> Hi,
> 
> I need a way to account traffic (bytes) for ~500 ips (fixed),  and it
> seems creating a plain 500 rules will affect the performance a lot.
> Without implement layered rule (like a binary search?) , is there
> something existing to do automatic hashing?
> Things like hashlimit is great, but I don't need limit matching
> function, just a way to create a hashtable and count bytes and
> packets.
> 
> If there's none, I suppose it would easy enough to fork some hashlimit
> code to do this.

You can have a look at how ulogd2 and nfacct can be used for accounting:
https://home.regit.org/2012/07/flow-accounting-with-netfilter-and-ulogd2/

BR,
-- 
Eric Leblond 
Blog: http://home.regit.org/ - Portfolio: http://regit.500px.com/

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: per host accounting

[Yucong Sun (叶雨飞)]

Thanks for the reply, Yeah I'm aware all of that you have mentioned,
please allow me to elaborate my requirements a little more.

I have about 500 IPs behind a router, and I want have something on my
router to monitor the ingress bps/pps to each specific IP. And I would
like to have a cron job  that scans the result and find the top 5  IP
with most bps/pps and also do some action against it, calling a
script, sending a email etc.

So, It seems none of the existing stuff allows me to do this,  the
easiest brain-dead solution I can think of is to just create a chain
with 500 rules in it, and have a cron job to cacluate the bytes
difference every time it executes. Obviously, this  will introduce a
lot of delays, I'm hoping to have something that basically don't
affect performance too much and or something to just generates a table
of ip  / accumulative packets / accumulative bytes, and I will be able
to work with that.

On Mon, Jul 23, 2012 at 1:00 AM, Eric Leblond <eric@regit.org> wrote:
>
> Hello,
>
> Le dimanche 22 juillet 2012 à 20:22 -0700, Yucong Sun (叶雨飞) a écrit :
> > Hi,
> >
> > I need a way to account traffic (bytes) for ~500 ips (fixed),  and it
> > seems creating a plain 500 rules will affect the performance a lot.
> > Without implement layered rule (like a binary search?) , is there
> > something existing to do automatic hashing?
> > Things like hashlimit is great, but I don't need limit matching
> > function, just a way to create a hashtable and count bytes and
> > packets.
> >
> > If there's none, I suppose it would easy enough to fork some hashlimit
> > code to do this.
>
> You can have a look at how ulogd2 and nfacct can be used for accounting:
> https://home.regit.org/2012/07/flow-accounting-with-netfilter-and-ulogd2/
>
> BR,
> --
> Eric Leblond
> Blog: http://home.regit.org/ - Portfolio: http://regit.500px.com/

Re: per host accounting

[Pablo Neira Ayuso]

On Mon, Jul 23, 2012 at 03:27:08PM -0700, Yucong Sun (叶雨飞) wrote:
> Thanks for the reply, Yeah I'm aware all of that you have mentioned,
> please allow me to elaborate my requirements a little more.
> 
> I have about 500 IPs behind a router, and I want have something on my
> router to monitor the ingress bps/pps to each specific IP. And I would
> like to have a cron job  that scans the result and find the top 5  IP
> with most bps/pps and also do some action against it, calling a
> script, sending a email etc.
> 
> So, It seems none of the existing stuff allows me to do this.

You can add one nfacct rule per IP and then use the nfacct utility to
periodically dump the counters and find for top IPs. Some shell script
should allow want you need. You can also develop your own daemon with
native libnetfilter_acct interfaces to periodically pull the counters
and perform the processing you need.

> the easiest brain-dead solution I can think of is to just create a chain
> with 500 rules in it, and have a cron job to cacluate the bytes
> difference every time it executes.

Instead of this, I'd go nfacct.

> Obviously, this  will introduce a
> lot of delays, I'm hoping to have something that basically don't
> affect performance too much and or something to just generates a table
> of ip  / accumulative packets / accumulative bytes, and I will be able
> to work with that.

Well, how much is "a lot of delay". I think your performance concerns
need real numbers. I don't think that will be too much as you mention.

Re: per host accounting

[Peter Phaal]

Another alternative would be to install a Host sFlow agent
(http://host-sflow.sourceforge.net/) on your router. Traffic
monitoring is performed using the iptables statistics module and ULOG.

http://blog.sflow.com/2010/12/ulog.html

A few comments about the solution:
1. It is extremely lightweight and the data analysis can be shifted to
a different machine, further reducing the overhead on the router.
2. Sampling works well for identifying the top 5 sources
3. Host sFlow also exports cpu, memory, disk etc. statistics so you
can track router load.
4. You can perform detailed analysis on the top talkers to see what
they are doing.

Peter

On Wed, Jul 25, 2012 at 3:19 AM, Pablo Neira Ayuso <pablo@netfilter.org> wrote:
> On Mon, Jul 23, 2012 at 03:27:08PM -0700, Yucong Sun (叶雨飞) wrote:
>> Thanks for the reply, Yeah I'm aware all of that you have mentioned,
>> please allow me to elaborate my requirements a little more.
>>
>> I have about 500 IPs behind a router, and I want have something on my
>> router to monitor the ingress bps/pps to each specific IP. And I would
>> like to have a cron job  that scans the result and find the top 5  IP
>> with most bps/pps and also do some action against it, calling a
>> script, sending a email etc.
>>
>> So, It seems none of the existing stuff allows me to do this.
>
> You can add one nfacct rule per IP and then use the nfacct utility to
> periodically dump the counters and find for top IPs. Some shell script
> should allow want you need. You can also develop your own daemon with
> native libnetfilter_acct interfaces to periodically pull the counters
> and perform the processing you need.
>
>> the easiest brain-dead solution I can think of is to just create a chain
>> with 500 rules in it, and have a cron job to cacluate the bytes
>> difference every time it executes.
>
> Instead of this, I'd go nfacct.
>
>> Obviously, this  will introduce a
>> lot of delays, I'm hoping to have something that basically don't
>> affect performance too much and or something to just generates a table
>> of ip  / accumulative packets / accumulative bytes, and I will be able
>> to work with that.
>
> Well, how much is "a lot of delay". I think your performance concerns
> need real numbers. I don't think that will be too much as you mention.
> --
> To unsubscribe from this list: send the line "unsubscribe netfilter" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: per host accounting

[Bob Miller]

On Mon, 2012-07-23 at 15:27 -0700, Yucong Sun (叶雨飞) wrote:
> Thanks for the reply, Yeah I'm aware all of that you have mentioned,
> please allow me to elaborate my requirements a little more.
> 
> I have about 500 IPs behind a router, and I want have something on my
> router to monitor the ingress bps/pps to each specific IP. And I would
> like to have a cron job  that scans the result and find the top 5  IP
> with most bps/pps and also do some action against it, calling a
> script, sending a email etc.

Have you checked out the ACCOUNT target out of the xtables-addons?  You
still need to create cron jobs and a script, but it should be able to do
what you are looking for without too much load...




> 
> So, It seems none of the existing stuff allows me to do this,  the
> easiest brain-dead solution I can think of is to just create a chain
> with 500 rules in it, and have a cron job to cacluate the bytes
> difference every time it executes. Obviously, this  will introduce a
> lot of delays, I'm hoping to have something that basically don't
> affect performance too much and or something to just generates a table
> of ip  / accumulative packets / accumulative bytes, and I will be able
> to work with that.
> 
> On Mon, Jul 23, 2012 at 1:00 AM, Eric Leblond <eric@regit.org> wrote:
> >
> > Hello,
> >
> > Le dimanche 22 juillet 2012 à 20:22 -0700, Yucong Sun (叶雨飞) a écrit :
> > > Hi,
> > >
> > > I need a way to account traffic (bytes) for ~500 ips (fixed),  and it
> > > seems creating a plain 500 rules will affect the performance a lot.
> > > Without implement layered rule (like a binary search?) , is there
> > > something existing to do automatic hashing?
> > > Things like hashlimit is great, but I don't need limit matching
> > > function, just a way to create a hashtable and count bytes and
> > > packets.
> > >
> > > If there's none, I suppose it would easy enough to fork some hashlimit
> > > code to do this.
> >
> > You can have a look at how ulogd2 and nfacct can be used for accounting:
> > https://home.regit.org/2012/07/flow-accounting-with-netfilter-and-ulogd2/
> >
> > BR,
> > --
> > Eric Leblond
> > Blog: http://home.regit.org/ - Portfolio: http://regit.500px.com/
> --
> To unsubscribe from this list: send the line "unsubscribe netfilter" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

-- 
Bob Miller
867-334-7117 / 867-633-3760
http://computerisms.ca
bob@computerisms.ca
Network, Internet, Server,
and Open Source Solutions