[ANNOUNCE] ipset 6.19 released

[ANNOUNCE] ipset 6.19 released

[Jozsef Kadlecsik]

Hi,

I'm happy to announce ipset 6.19. The main new feature in this release is 
the per element byte and packet counters for every set type, which is 
based on the new internal extension infrastructure. Please read the 
updated manpage on how to use counters, initialize or change them.

The management and matching of counters is added to the "set" match of 
iptables/ip6tables, which is available in the "next" branch of the 
iptables git tree.

Userspace changes:
  - Check at modules_install whether depmod ignores the extra subdir
    (reported by Husnu Demir and tian fang)
  - The utils are updated from their sources
  - Manpage typing error correction (reported by Husnu Demir)
  - Update testsuite as the trailing space was eliminated at listings
  - Add sparse checking support to userspace
  - Improve XML output: add element tag and root element (suggested by 
    Lucas Hamie)
  - Manpage updates
  - Add new testsuite entries to verify counters and the new type
    implementation
  - Introduce the new set type revisions with counter support
  - Support counters in the ipset library
  - The uapi include split in the package itself

Kernel part changes:
  - Compatibility fixes to keep the support of kernels back to 2.6.32
  - Backport nla_put_net64
  - Support package fragments for IPv4 protos without ports
    (Anders K. Pedersen)
  - Use fix sized type for timeout in the extension part
  - Make sure kernel configured properly for sparse checkings
  - Fix "may be used uninitialized" warnings (reported by Pablo Neira
    Ayuso)
  - Rename simple macro names to avoid namespace issues (reported by
    David Laight)
  - Fix sparse warnings due to missing rcu annotations (reported by
    Pablo Neira Ayuso)
  - Sparse warning about shadowed variable fixed
  - Don't call ip_nest_end needlessly in the error path (suggested by
    Pablo Neira Ayuso)
  - set match: add support to match the counters
  - The list:set type with counter support
  - The hash types with counter support
  - The bitmap types with counter support
  - Introduce the counter extension in the core
  - list:set type using the extension interface
  - Hash types using the unified code base
  - Unified hash type generation
  - Bitmap types using the unified code base
  - Unified bitmap type generation
  - Introduce extensions to elements in the core
  - Move often used IPv6 address masking function to header file
  - Make possible to test elements marked with nomatch, from userspace
  - netfilter ipset: Use ipv6_addr_equal() where appropriate.
    (YOSHIFUJI Hideaki)
  - Add a compatibility header file for easier maintenance
  - The uapi include split in the package itself
  - Reorder modules a little bit in Kbuild

You can download the source code of ipset from:
        http://ipset.netfilter.org
        ftp://ftp.netfilter.org/pub/ipset/
        git://git.netfilter.org/ipset.git

Best regards,
Jozsef
-
E-mail  : kadlec@blackhole.kfki.hu, kadlecsik.jozsef@wigner.mta.hu
PGP key : http://www.kfki.hu/~kadlec/pgp_public_key.txt
Address : Wigner Research Centre for Physics, Hungarian Academy of Sciences
          H-1525 Budapest 114, POB. 49, Hungary

Re: [ANNOUNCE] ipset 6.19 released

[Andreas Herz]

On 09/05/13 at 22:28, Jozsef Kadlecsik wrote:
> I'm happy to announce ipset 6.19. The main new feature in this release is 
> the per element byte and packet counters for every set type, which is 
> based on the new internal extension infrastructure. Please read the 
> updated manpage on how to use counters, initialize or change them.

Thanks for the release :)

Can you describe the new feature a bit more in detail?
I would like too see some use cases for this feature.
I would guess that every packet that comes through the ipset match would
increase the counter for the address (element) in the list. But what
effect does it have, since the timeout has the effect that the element
leaves the list when it goes down to zero.

-- 
Andreas Herz

Re: [ANNOUNCE] ipset 6.19 released

[Jozsef Kadlecsik]

On Fri, 10 May 2013, Andreas Herz wrote:

> On 09/05/13 at 22:28, Jozsef Kadlecsik wrote:
> > I'm happy to announce ipset 6.19. The main new feature in this release is 
> > the per element byte and packet counters for every set type, which is 
> > based on the new internal extension infrastructure. Please read the 
> > updated manpage on how to use counters, initialize or change them.
> 
> Thanks for the release :)
> 
> Can you describe the new feature a bit more in detail?
> I would like too see some use cases for this feature.
> I would guess that every packet that comes through the ipset match would
> increase the counter for the address (element) in the list. But what
> effect does it have, since the timeout has the effect that the element
> leaves the list when it goes down to zero.

If a set is defined with timeout and counters and an element times out, of 
course the counter values are lost.

Some examples:

a. Traffic counting from clients

# Create the set, fill with elements
ipset n clients hash:ip counters
ipset a clients ...
# Match in a rule
iptables -A ... -m set --match-set clients src -j ...
# List regularly the set and process the counter values
ipset l clients

b. Combined time and traffic limiting

# Create the set, fill with elements
ipset n guests hash:ip counters timeout $((2*60*60))
ipset a guests ...
# Allow traffic up to a given a limit or until timeout
iptables -A ... \
  -m set --match-set guests src --bytes-lt $((8*1024*1024)) ...

There are some subleties of course: if the set match rule is negated

   .... -m set ! --match-set foo src

that is independent from increasing the packet and byte counters for the 
elements which matched the lookup in the set - the negation happens at 
rule evaluation level.

As the updated iptables-extensions manpage shows, there are new flags to 
suppress updating the packet and byte counters at different levels if one 
matches a set from multiple rules.

Best regards,
Jozsef
-
E-mail  : kadlec@blackhole.kfki.hu, kadlecsik.jozsef@wigner.mta.hu
PGP key : http://www.kfki.hu/~kadlec/pgp_public_key.txt
Address : Wigner Research Centre for Physics, Hungarian Academy of Sciences
          H-1525 Budapest 114, POB. 49, Hungary

Re: [ANNOUNCE] ipset 6.19 released

[Dash Four]

Jozsef Kadlecsik wrote:
> Some examples:
>
> a. Traffic counting from clients
>
> # Create the set, fill with elements
> ipset n clients hash:ip counters
> ipset a clients ...
> # Match in a rule
> iptables -A ... -m set --match-set clients src -j ...
> # List regularly the set and process the counter values
> ipset l clients
>
> b. Combined time and traffic limiting
>
> # Create the set, fill with elements
> ipset n guests hash:ip counters timeout $((2*60*60))
> ipset a guests ...
> # Allow traffic up to a given a limit or until timeout
> iptables -A ... \
>   -m set --match-set guests src --bytes-lt $((8*1024*1024)) ...
>
> There are some subleties of course: if the set match rule is negated
>
>    .... -m set ! --match-set foo src
>
> that is independent from increasing the packet and byte counters for the 
> elements which matched the lookup in the set - the negation happens at 
> rule evaluation level.
>
> As the updated iptables-extensions manpage shows, there are new flags to 
> suppress updating the packet and byte counters at different levels if one 
> matches a set from multiple rules.
>   
Jozsef, where is this updated iptables-extensions manpage you speak of?

The latest released version of iptables is 1.4.18, which obviously does 
not include any of the above since it was released before ipset 6.19. 
I've just looked at the main iptables git repo and there aren't any git 
logs since that release with anything related to ipset. In the ipset 
sources, apart from the main ipset man page, there is nothing there 
which describes the above iptables match options.

Re: [ANNOUNCE] ipset 6.19 released

[Jozsef Kadlecsik]

On Sat, 11 May 2013, Dash Four wrote:

> Jozsef Kadlecsik wrote:
> > Some examples:
> > 
> > a. Traffic counting from clients
> > 
> > # Create the set, fill with elements
> > ipset n clients hash:ip counters
> > ipset a clients ...
> > # Match in a rule
> > iptables -A ... -m set --match-set clients src -j ...
> > # List regularly the set and process the counter values
> > ipset l clients
> > 
> > b. Combined time and traffic limiting
> > 
> > # Create the set, fill with elements
> > ipset n guests hash:ip counters timeout $((2*60*60))
> > ipset a guests ...
> > # Allow traffic up to a given a limit or until timeout
> > iptables -A ... \
> >   -m set --match-set guests src --bytes-lt $((8*1024*1024)) ...
> > 
> > There are some subleties of course: if the set match rule is negated
> > 
> >    .... -m set ! --match-set foo src
> > 
> > that is independent from increasing the packet and byte counters for the
> > elements which matched the lookup in the set - the negation happens at rule
> > evaluation level.
> > 
> > As the updated iptables-extensions manpage shows, there are new flags to
> > suppress updating the packet and byte counters at different levels if one
> > matches a set from multiple rules.
> >   
> Jozsef, where is this updated iptables-extensions manpage you speak of?
> 
> The latest released version of iptables is 1.4.18, which obviously does not
> include any of the above since it was released before ipset 6.19. I've just
> looked at the main iptables git repo and there aren't any git logs since that
> release with anything related to ipset. In the ipset sources, apart from the
> main ipset man page, there is nothing there which describes the above iptables
> match options.

As I wrote in the announcement, it's in the "next" branch of the iptables 
git tree yet.

Best regards,
Jozsef
-
E-mail  : kadlec@blackhole.kfki.hu, kadlecsik.jozsef@wigner.mta.hu
PGP key : http://www.kfki.hu/~kadlec/pgp_public_key.txt
Address : Wigner Research Centre for Physics, Hungarian Academy of Sciences
          H-1525 Budapest 114, POB. 49, Hungary

Re: [ANNOUNCE] ipset 6.19 released

[Dash Four]

Jozsef Kadlecsik wrote:
> On Sat, 11 May 2013, Dash Four wrote:
>   
>> Jozsef, where is this updated iptables-extensions manpage you speak of?
>>
>> The latest released version of iptables is 1.4.18, which obviously does not
>> include any of the above since it was released before ipset 6.19. I've just
>> looked at the main iptables git repo and there aren't any git logs since that
>> release with anything related to ipset. In the ipset sources, apart from the
>> main ipset man page, there is nothing there which describes the above iptables
>> match options.
>>     
>
> As I wrote in the announcement, it's in the "next" branch of the iptables 
> git tree yet.
>   
Ah, got it now. Thanks Jozsef.