ipt_ACCOUNT 1.15 released

ipt_ACCOUNT 1.15 released

[Thomas Jarosch]

Hello,

I'm pleased to announce the release of ipt_ACCOUNT 1.15.

Changelog:
- Support for kernel 2.6.29 / 2.6.28

Get it from here:
http://www.intra2net.com/en/developer/ipt_ACCOUNT/

Best regards,
Thomas Jarosch

Re: ipt_ACCOUNT 1.15 released

[Jan Engelhardt]

On Tuesday 2009-04-14 17:44, Thomas Jarosch wrote:
>Hello,
>
>I'm pleased to announce the release of ipt_ACCOUNT 1.15.
>Changelog:
>- Support for kernel 2.6.29 / 2.6.28
>Get it from here:
>http://www.intra2net.com/en/developer/ipt_ACCOUNT/
>Best regards,
>Thomas Jarosch

Thomas,

guess I just forward this to you, as I do not want to touch ACCOUNT
just yet.


>Date: Mon, 6 Apr 2009 15:32:24
>From: Bogdan Siara (cc'ed) <siarka2107>
>To: Jan Engelhardt
>Subject: Re: Re: New matches and targets on xtables-addons

> On Monday 2009-04-06 14:05, Bogdan Siara wrote:
> 
>>>Hi Jan,
>>>Can You add new matches and targets to xtables-addons e.g.:
>>>- ipt_ACCOUNT
>>>  http://www.intra2net.com/en/developer/ipt_ACCOUNT/index.php
>>>- ipt_account http://code.google.com/p/ipt-account
>>>- ipt_weburl http://gargoyle-router.com/weburl-code
>> 
>>ACCOUNT requires a kernel patch, and patching the kernel is outside
>>the scope of Xt-a; For only including the iptables module, I would
>>want to have their consent.
>> 
>>weburl seems redundant with respect to the existence of xt_string and
>>xt_layer7, but feel free to convince me otherwise.
>
>Can You prepare ipt_ACCOUNT and ipt_account patches/files supported for 
>kernel 2.6.29?

Re: ipt_ACCOUNT 1.15 released

[Thomas Jarosch]

Hi Jan,

On Tuesday, 14. April 2009 18:10:54 Jan Engelhardt wrote:
> guess I just forward this to you, as I do not want to touch ACCOUNT
> just yet.

As you already mentioned, I'm not sure it would be a good idea
to include it as the kernel patch extends the kernel<->user space socket 
operations in include/linux/netfilter_ipv4/ip_tables.h.

"ipt_account" is another story as it works differently.
The lastest patch seems to be for 2.6.19 / iptables 1.3.5,
though there is some recent activity on the project homepage.

I'm still surprised how many people are using ipt_ACCOUNT,
somehow it is magnetic to ISPs in central and eastern europe :-)

> >Can You prepare ipt_ACCOUNT and ipt_account patches/files supported for
> >kernel 2.6.29?

ipt_ACCOUNT is now ready for 2.6.29.

Cheers,
Thomas

Re: ipt_ACCOUNT 1.15 released

[Jan Engelhardt]

On Wednesday 2009-04-15 10:11, Thomas Jarosch wrote:
>Hi Jan,
>
>On Tuesday, 14. April 2009 18:10:54 Jan Engelhardt wrote:
>> guess I just forward this to you, as I do not want to touch ACCOUNT
>> just yet.
>
>As you already mentioned, I'm not sure it would be a good idea
>to include it as the kernel patch extends the kernel<->user space socket 
>operations in include/linux/netfilter_ipv4/ip_tables.h.
>
>"ipt_account" is another story as it works differently.
>The lastest patch seems to be for 2.6.19 / iptables 1.3.5,
>though there is some recent activity on the project homepage.

What the user wanted is moving the iptables modules 
(libipt_account.c/ACCOUNT.c) into Xt-a. The kernel patch would continue 
to be separate.

>I'm still surprised how many people are using ipt_ACCOUNT,
>somehow it is magnetic to ISPs in central and eastern europe :-)

Sometimes, ulogd is considered to big of a solution. Even I just use 
xt_quota2 for embedded devices that need to watch their kb limits 
set forth in their HSDPA link contract.

Re: ipt_ACCOUNT 1.15 released

[Thomas Jacob]

On Wed, 2009-04-15 at 10:11 +0200, Thomas Jarosch wrote:
> Hi Jan,
> 
> On Tuesday, 14. April 2009 18:10:54 Jan Engelhardt wrote:
> > guess I just forward this to you, as I do not want to touch ACCOUNT
> > just yet.
> 
> As you already mentioned, I'm not sure it would be a good idea
> to include it as the kernel patch extends the kernel<->user space socket 
> operations in include/linux/netfilter_ipv4/ip_tables.h.

I noticed this too, and the question for me is, do you really need
do things this way? Because that's really the only thing that requires
a kernel patch in your module. Ipset for instance doesn't anymore, but
I guess they've been "assigned" a permanent socket option number.... if
that could happen for your module: problem solved.

> "ipt_account" is another story as it works differently.
> The lastest patch seems to be for 2.6.19 / iptables 1.3.5,
> though there is some recent activity on the project homepage.
>
> I'm still surprised how many people are using ipt_ACCOUNT,
> somehow it is magnetic to ISPs in central and eastern europe :-)

One reason springs to mind, apart from the obvious "they were there
first" reason: 64 bit counters... your module only uses 32 bit counters
which is not really great if you all you want to do account traffic at
an ISP, because if you've got a fully loaded 100 Mbps-Port your counters
will overflow every 5 minutes, so one needs to write software
that can extract and adding up the accounting data by querying your
module very often (I just did that for a future project ;).

Also the other ipt_account allows saving and restoring the accounting
state, thereby allowing you to deal with crashes and reboots.

But as you say, ipt_account is not really supported anymore, so....

BTW, if you plan to add 64bit counters and maybe also IPv6 capability
I'd be very much willing to help ;)

Re: ipt_ACCOUNT 1.15 released

[Thomas Jarosch]

Hello Thomas,

On Wednesday, 15. April 2009 11:40:29 you wrote:
> > As you already mentioned, I'm not sure it would be a good idea
> > to include it as the kernel patch extends the kernel<->user space socket
> > operations in include/linux/netfilter_ipv4/ip_tables.h.
>
> I noticed this too, and the question for me is, do you really need
> do things this way? Because that's really the only thing that requires
> a kernel patch in your module. Ipset for instance doesn't anymore, but
> I guess they've been "assigned" a permanent socket option number.... if
> that could happen for your module: problem solved.

Interesting findings. The "IPT_BASE_CTL" socket operation usually starts
at 64 and then various operations get added to it by incrementing the value.

ipset's socket operations use absolute values like 0x201,
I'm wondering if they were registered somewhere or just "allocated".

> > I'm still surprised how many people are using ipt_ACCOUNT,
> > somehow it is magnetic to ISPs in central and eastern europe :-)
>
> One reason springs to mind, apart from the obvious "they were there
> first" reason: 64 bit counters... your module only uses 32 bit counters
> which is not really great if you all you want to do account traffic at
> an ISP, because if you've got a fully loaded 100 Mbps-Port your counters
> will overflow every 5 minutes, so one needs to write software
> that can extract and adding up the accounting data by querying your
> module very often (I just did that for a future project ;).
>
> Also the other ipt_account allows saving and restoring the accounting
> state, thereby allowing you to deal with crashes and reboots.
>
> But as you say, ipt_account is not really supported anymore, so....
>
> BTW, if you plan to add 64bit counters and maybe also IPv6 capability
> I'd be very much willing to help ;)

The 32bit counters were a design decision to store a complete class C subnet 
into one kernel page. We query the data on our own system every second to 
check the online timeout of dial up lines, so that doesn't affect us.

Transforming the internal data structure into a hash table would be the way to 
go for IPv6 support and 64 bit counters. I guess it wouldn't be slower than 
the current approach or could even be faster for large networks,
so that would need a performance benchmark.

On the other side I would look at conntrack accounting first, it wasn't ready 
for production use when I wrote ipt_ACCOUNT back then, which it is now.

Cheers,
Thomas

Re: ipt_ACCOUNT 1.15 released

[Thomas Jacob]

On Thu, 2009-04-16 at 18:34 +0200, Thomas Jarosch wrote:
> Hello Thomas,

> Interesting findings. The "IPT_BASE_CTL" socket operation usually starts
> at 64 and then various operations get added to it by incrementing the value.
> 
> ipset's socket operations use absolute values like 0x201,
> I'm wondering if they were registered somewhere or just "allocated".

include/linux/netfilter_ipv4/ip_set.h

/*
 * A sockopt of such quality has hardly ever been seen before on the
open
 * market!  This little beauty, hardly ever used: above 64, so it's
 * traditionally used for firewalling, not touched (even once!) by the
 * 2.0, 2.2 and 2.4 kernels!
 *
 * Comes with its own certificate of authenticity, valid anywhere in the
 * Free world!
 *
 * Rusty, 19.4.2000
 */
#define SO_IP_SET       83

But maybe someone on the list who remembers how this number got
to be can comment on the issue ;)

> The 32bit counters were a design decision to store a complete class C subnet 
> into one kernel page. We query the data on our own system every second to 
> check the online timeout of dial up lines, so that doesn't affect us.
>
> Transforming the internal data structure into a hash table would be the way to 
> go for IPv6 support and 64 bit counters. I guess it wouldn't be slower than 
> the current approach or could even be faster for large networks,
> so that would need a performance benchmark.

That's what I thought as well, hash tables are the way to
go. 

I am stilling thinking about implementing this, if I find the time.
Also, in order to make IPv6 accounting practical, I would probably add
some sort of subnet accounting along the following lines.

-j ACCOUNT --network 1234:4567::/32 --account_mask /64

which would only account packets in --network, and count data
for a whole subnet instead of a single IP, as assigning single
IPv6 addresses to customers is kind of silly in an ISP environment.

To collect data for single IPs just use --account_mask 128.

Another idea, possibly useful from an ISP perspective, might be
to only start the counters (i.e. add the entry to the hash bucket) if
packets with the IP/subnet have arrived from a certain direction first.

This way, your tables don't overflow with all the IP scans
from the Internet.

> On the other side I would look at conntrack accounting first, it wasn't ready 
> for production use when I wrote ipt_ACCOUNT back then, which it is now.

>From where I'm standing this looks much resource extensive than the
approaches taken by ipt_ACCOUNT or ipt_account. All I am interested in
is a byte counter per target IP/subnet that can be queried once a day
so that I know who is consuming my bandwidth. Something that passes
each connection track to user space doesn't look so good for that.

But maybe I am wrong about this, I  haven't actually tried to use it.

Re: ipt_ACCOUNT 1.15 released

[Jozsef Kadlecsik]

On Thu, 16 Apr 2009, Thomas Jacob wrote:

> On Thu, 2009-04-16 at 18:34 +0200, Thomas Jarosch wrote:
> > Hello Thomas,
> 
> > Interesting findings. The "IPT_BASE_CTL" socket operation usually starts
> > at 64 and then various operations get added to it by incrementing the value.
> > 
> > ipset's socket operations use absolute values like 0x201,
> > I'm wondering if they were registered somewhere or just "allocated".
> 
> include/linux/netfilter_ipv4/ip_set.h
> 
> /*
>  * A sockopt of such quality has hardly ever been seen before on the
> open
>  * market!  This little beauty, hardly ever used: above 64, so it's
>  * traditionally used for firewalling, not touched (even once!) by the
>  * 2.0, 2.2 and 2.4 kernels!
>  *
>  * Comes with its own certificate of authenticity, valid anywhere in the
>  * Free world!
>  *
>  * Rusty, 19.4.2000
>  */
> #define SO_IP_SET       83
> 
> But maybe someone on the list who remembers how this number got
> to be can comment on the issue ;)

Maybe Joakim Axelsson or Rusty could answer it, if they remember how that 
number was figured out.

Best regards,
Jozsef
-
E-mail  : kadlec@blackhole.kfki.hu, kadlec@mail.kfki.hu
PGP key : http://www.kfki.hu/~kadlec/pgp_public_key.txt
Address : KFKI Research Institute for Particle and Nuclear Physics
          H-1525 Budapest 114, POB. 49, Hungary

Re: ipt_ACCOUNT 1.15 released

[Thomas Jarosch]

On Thursday, 16. April 2009 20:29:46 Thomas Jacob wrote:
> Also, in order to make IPv6 accounting practical, I would probably add
> some sort of subnet accounting along the following lines.
>
> -j ACCOUNT --network 1234:4567::/32 --account_mask /64
>
> which would only account packets in --network, and count data
> for a whole subnet instead of a single IP, as assigning single
> IPv6 addresses to customers is kind of silly in an ISP environment.
>
> To collect data for single IPs just use --account_mask 128.

Well, for IPv4 you can alreay use "--src 172.16.0.0/16"
and then do "-j ACCOUNT --addr 0.0.0.0/0" to merge
the complete subnet into one single IP address.

This should be possible with IPv6, too.

> Another idea, possibly useful from an ISP perspective, might be
> to only start the counters (i.e. add the entry to the hash bucket) if
> packets with the IP/subnet have arrived from a certain direction first.
>
> This way, your tables don't overflow with all the IP scans
> from the Internet.

Maybe a conntrack "state" match can help here?

> approaches taken by ipt_ACCOUNT or ipt_account. All I am interested in
> is a byte counter per target IP/subnet that can be queried once a day
> so that I know who is consuming my bandwidth. Something that passes
> each connection track to user space doesn't look so good for that.
>
> But maybe I am wrong about this, I  haven't actually tried to use it.

Me neither :-) IIRC the problem in the past was that ulogd (v1) could lose 
packets when the userspace <-> kernel communication queue was not processed
as fast as new packets arrived. Though my memory on this is quite blurry...

Thomas

Re: ipt_ACCOUNT 1.15 released

[Thomas Jacob]

On Mon, 2009-04-20 at 12:19 +0200, Thomas Jarosch wrote:
> On Thursday, 16. April 2009 20:29:46 Thomas Jacob wrote:
> > Also, in order to make IPv6 accounting practical, I would probably add
> > some sort of subnet accounting along the following lines.
> >
> > -j ACCOUNT --network 1234:4567::/32 --account_mask /64
> >
> > which would only account packets in --network, and count data
> > for a whole subnet instead of a single IP, as assigning single
> > IPv6 addresses to customers is kind of silly in an ISP environment.
> >
> > To collect data for single IPs just use --account_mask 128.
> 
> Well, for IPv4 you can alreay use "--src 172.16.0.0/16"
> and then do "-j ACCOUNT --addr 0.0.0.0/0" to merge
> the complete subnet into one single IP address.

Hmm, then maybe haven't understood your module yet.

If I specify "--src 172.16.0.0/16 -j ACCOUNT --addr 0.0.0.0/0 --tname
X", I was under the impression that I will get entries for each single
IP that somehow appears in packets that match --src 172.16.0.0/16
in table X. Potentially a huge number (if you are getting DDOSed ;).

Then of course I could add up all these entries and come up with a
number for the subnet. And if I want to account many subnets
(1000s) this way, I will need to add many such tables and rules. 

That will probably not scale well, I think. 

Or does your module work in a different way?

Re: ipt_ACCOUNT 1.15 released

[Thomas Jarosch]

On Monday, 20. April 2009 13:31:32 Thomas Jacob wrote:
> > Well, for IPv4 you can alreay use "--src 172.16.0.0/16"
> > and then do "-j ACCOUNT --addr 0.0.0.0/0" to merge
> > the complete subnet into one single IP address.
>
> Hmm, then maybe haven't understood your module yet.
>
> If I specify "--src 172.16.0.0/16 -j ACCOUNT --addr 0.0.0.0/0 --tname
> X", I was under the impression that I will get entries for each single
> IP that somehow appears in packets that match --src 172.16.0.0/16
> in table X. Potentially a huge number (if you are getting DDOSed ;).

Yes, basically it works that way. The only exception is 0.0.0.0/0:

"A special subnet is "0.0.0.0/0": All data is stored in the src_bytes and 
src_packets structure of slot "0". This is useful if you want to account the 
overall traffic to/from your internet provider."

-> You can accumulate complete subnets on one entry if you like.

F.e. we use this to check for network activity
(=and decrase a timeout if not present).

Cheers,
Thomas