[ANNOUNCE] ipset 6.6 released

[ANNOUNCE] ipset 6.6 released

[Jozsef Kadlecsik]

Hi,

Here comes ipset 6.6 with a couple of bugfixes and improvements. The terse 
list of the changes are:

Kernel part changes:
 - Use unified from/to address masking and check the usage
 - ip_set_flush returned -EPROTO instead of -IPSET_ERR_PROTOCOL, fixed
 - Take into account cidr value for the from address when creating the set
 - Adding ranges to hash types with timeout could still fail, fixed
 - Removed old, not used hashing method ip_set_chash
 - Remove variable 'ret' in type_pf_tdel(), which is set but not used
 - Use proper timeout parameter to jiffies conversion

Userspace changes:
 - Restore with bitmap:port and list:set types did not work, fixed
 - Accept "\r\n" terminated COMMIT command in restore files
 - Fix the message sequence number book-keeping
 - Protocol-level debugging support added
 - hash:net stress test in range notation added
 - ipset_mnl_query: in debug mode print the errno returned by the cb
   function
 - Accept "\r\n" terminated lines in restore files
 - Remove outdated checking of IPv6 support from configure.ac

I thank to Mr Dash Four for his tireless testings and helpful reports.

The next ipset package release will come with code simplifications, which 
mean dropping some support:

- iptree and iptreemap type support will be removed, because the automatic
  conversion to hash:ip is not fully backward compatible. The hash:ip and 
  hash:net types can be used instead.
- kernel version support below 2.6.37 will be dropped, to make code
  maintenance easier and more straightforward

You can download the source code of ipset from:
        http://ipset.netfilter.org
        ftp://ftp.netfilter.org/pub/ipset/
        git://git.netfilter.org/ipset.git

Best regards,
Jozsef
-
E-mail  : kadlec@blackhole.kfki.hu, kadlec@mail.kfki.hu
PGP key : http://www.kfki.hu/~kadlec/pgp_public_key.txt
Address : KFKI Research Institute for Particle and Nuclear Physics
          H-1525 Budapest 114, POB. 49, Hungary

Re: [ANNOUNCE] ipset 6.6 released

[Mr Dash Four]

> I thank to Mr Dash Four for his tireless testings and helpful reports.
>   
I thank you for your continuing development of this great tool which, at 
least in my case, hugely improved the stability and performance, as well 
as decreased the maintenance needed on all my systems as a direct result 
of employing this great tool!

> The next ipset package release will come with code simplifications, which 
> mean dropping some support:
>
> - iptree and iptreemap type support will be removed, because the automatic
>   conversion to hash:ip is not fully backward compatible. The hash:ip and 
>   hash:net types can be used instead.
> - kernel version support below 2.6.37 will be dropped, to make code
>   maintenance easier and more straightforward
>   
What are your plans with regards to the last point? The reason I am 
asking is because I am still on .35 (the last "stable" kernel released 
by Fedora) and I might not be able to use new releases if you introduce 
this restriction.

Also, would you consider my suggestion to include the number of members 
registered in a set to be displayed in the set headers (something like 
"Members: 15156") - this would help with managing the number of members 
in that set, which would be particularly useful for sets with large 
number of members.

Re: [ANNOUNCE] ipset 6.6 released

[Jozsef Kadlecsik]

On Tue, 24 May 2011, Mr Dash Four wrote:

> > The next ipset package release will come with code simplifications, which
> > mean dropping some support:
> > 
> > - iptree and iptreemap type support will be removed, because the automatic
> >   conversion to hash:ip is not fully backward compatible. The hash:ip and
> > hash:net types can be used instead.
> > - kernel version support below 2.6.37 will be dropped, to make code
> >   maintenance easier and more straightforward
> >   
> What are your plans with regards to the last point? The reason I am asking is
> because I am still on .35 (the last "stable" kernel released by Fedora) and I
> might not be able to use new releases if you introduce this restriction.

Hm, 2.6.35 can lessen the maintenance burden compared to the currently 
minimal supported version 2.6.34, because the main trouble comes from the 
differences between .34 and .35. So I think I can remove the older kernel
supports gradually and keep supporting 2.6.35 for a while.
 
> Also, would you consider my suggestion to include the number of members
> registered in a set to be displayed in the set headers (something like
> "Members: 15156") - this would help with managing the number of members in
> that set, which would be particularly useful for sets with large number of
> members.

The problem with it that the reported number can be inaccurate, at least 
in two cases:

- Elements can time out, so even if whatever number reported, by the
  time it's displayed, the set can even be completely empty. In the case
  of a huge set, it can even occur that the number of elements reported
  does not match the actual number of elements listed.
- Sets can be updated by the SET target

The first one is the main reason I dropped reporting the number of 
elements (the initial design of the new ipset included it).

Best regards,
Jozsef
-
E-mail  : kadlec@blackhole.kfki.hu, kadlec@mail.kfki.hu
PGP key : http://www.kfki.hu/~kadlec/pgp_public_key.txt
Address : KFKI Research Institute for Particle and Nuclear Physics
          H-1525 Budapest 114, POB. 49, Hungary

Re: [ANNOUNCE] ipset 6.6 released

[Mr Dash Four]

> Hm, 2.6.35 can lessen the maintenance burden compared to the currently 
> minimal supported version 2.6.34, because the main trouble comes from the 
> differences between .34 and .35. So I think I can remove the older kernel
> supports gradually and keep supporting 2.6.35 for a while.
>   
That would be appreciated as I think .35 is a pretty stable kernel and 
will be in use for a while yet (though I have to admit, I have patched 
this kernel extensively).

> The problem with it that the reported number can be inaccurate, at least 
> in two cases:
>
> - Elements can time out, so even if whatever number reported, by the
>   time it's displayed, the set can even be completely empty. In the case
>   of a huge set, it can even occur that the number of elements reported
>   does not match the actual number of elements listed.
> - Sets can be updated by the SET target
>
> The first one is the main reason I dropped reporting the number of 
> elements (the initial design of the new ipset included it).
>   
I see, reasonable point.

OK, would it be possible then to get this figure when I use restore - I 
am asking for this because, as you already know, ip ranges may mean 
inserting more elements than the range itself and the indicator I used 
up until now to determine how many set members I would have (the number 
of lines in my restore file) is no longer a reliable indicator, so I 
would need to know how many members I have inserted as a result of 
restore in order to determine the maxelem value, adjust it and thus save 
system resources. Do you see my point?

Re: [ANNOUNCE] ipset 6.6 released

[Jozsef Kadlecsik]

On Tue, 24 May 2011, Mr Dash Four wrote:

> > The problem with it that the reported number can be inaccurate, at least in
> > two cases:
> > 
> > - Elements can time out, so even if whatever number reported, by the
> >   time it's displayed, the set can even be completely empty. In the case
> >   of a huge set, it can even occur that the number of elements reported
> >   does not match the actual number of elements listed.
> > - Sets can be updated by the SET target
> > 
> > The first one is the main reason I dropped reporting the number of elements
> > (the initial design of the new ipset included it).
> >   
> I see, reasonable point.
> 
> OK, would it be possible then to get this figure when I use restore - I am
> asking for this because, as you already know, ip ranges may mean inserting
> more elements than the range itself and the indicator I used up until now to
> determine how many set members I would have (the number of lines in my restore
> file) is no longer a reliable indicator, so I would need to know how many
> members I have inserted as a result of restore in order to determine the
> maxelem value, adjust it and thus save system resources. Do you see my point?

Currently I can suggest you that tune both the hashsize and maxelem 
parameters: the hashsize tells the system the initial hash size (and thus 
the used memory resource) while maxelem tells up to what number of 
elements can be added to the set. You are free to enter quite high maxelem 
values, it does not waste memory. As you add more and more elements, the 
hash size is increased from the one specified by 'hashsize'. But you (or 
the SET targets) can't add more elements than "maxelem".

When you save a set, the current hashsize is saved and not the one 
specified at set creation time.

Best regards,
Jozsef
-
E-mail  : kadlec@blackhole.kfki.hu, kadlec@mail.kfki.hu
PGP key : http://www.kfki.hu/~kadlec/pgp_public_key.txt
Address : KFKI Research Institute for Particle and Nuclear Physics
          H-1525 Budapest 114, POB. 49, Hungary

Re: [ANNOUNCE] ipset 6.6 released

[Mr Dash Four]

> Currently I can suggest you that tune both the hashsize and maxelem 
> parameters: the hashsize tells the system the initial hash size (and thus 
> the used memory resource) while maxelem tells up to what number of 
> elements can be added to the set. You are free to enter quite high maxelem 
> values, it does not waste memory. As you add more and more elements, the 
> hash size is increased from the one specified by 'hashsize'. But you (or 
> the SET targets) can't add more elements than "maxelem".
>
> When you save a set, the current hashsize is saved and not the one 
> specified at set creation time.
>   
Ah, I see! So maxelem does not have an impact on system resources as 
hashsize has, is that right? If that is the case, then I may as well 
leave the default (64k) as I'd never exceed this value. Thanks for your 
input, as always!

Re: [ANNOUNCE] ipset 6.6 released

[Jozsef Kadlecsik]

On Tue, 24 May 2011, Mr Dash Four wrote:

> > Currently I can suggest you that tune both the hashsize and maxelem
> > parameters: the hashsize tells the system the initial hash size (and thus
> > the used memory resource) while maxelem tells up to what number of elements
> > can be added to the set. You are free to enter quite high maxelem values, it
> > does not waste memory. As you add more and more elements, the hash size is
> > increased from the one specified by 'hashsize'. But you (or the SET targets)
> > can't add more elements than "maxelem".
> > 
> > When you save a set, the current hashsize is saved and not the one 
> > specified at set creation time.
> >   
> Ah, I see! So maxelem does not have an impact on system resources as 
> hashsize has, is that right? If that is the case, then I may as well 
> leave the default (64k) as I'd never exceed this value. Thanks for your 
> input, as always!

Yes, exactly. With the hashsize parameter, you tell the system how much 
memory is allocated for the set when created. With maxelem it is 
instructed at what point to stop increasing the set and refuse adding more 
elements.

Best regards,
Jozsef
-
E-mail  : kadlec@blackhole.kfki.hu, kadlec@mail.kfki.hu
PGP key : http://www.kfki.hu/~kadlec/pgp_public_key.txt
Address : KFKI Research Institute for Particle and Nuclear Physics
          H-1525 Budapest 114, POB. 49, Hungary

Re: [ANNOUNCE] ipset 6.6 released

[Mr Dash Four]

> Userspace changes:
>  - Restore with bitmap:port and list:set types did not work, fixed
>   
Having had the chance to test this now, I can say that it works, and 
what's more - the loading performance is much, much better - all my sets 
now load in about 2-3 seconds, while with the 4.5 version it took in 
excess of 10 minutes, completely hogging my CPU in the process. I 
haven't had the chance yet to judge the matching performance, but this 
is what I will do in the coming days.

I have found a bug, however. :-\

When I have multiple sets of different type to restore, each restore 
file normally ends with "COMMIT" statement for ipset to commit the whole 
transaction, or so I thought. If there is a mistake (syntax or any 
other) somewhere in the restore file, which prevents the restore 
process, ipset already commits everything up to that point, which I 
think is wrong.

For example, if I have this:

n privileged-ports bitmap:port range 1-1023 timeout 0
a privileged-ports 1-1023
n test-ports bitmap:port range 12770-19999 timeout 0
a test-ports 20000-30000
a test-ports 19999
n test-port bitmap:port range 29950-29950 timeout 0
a test-port 29950
COMMIT

There is an obvious error in line 4 above ("a test-ports 20000-30000" - 
this is out of the defined range for this set) - ipset should have 
aborted the whole transaction and not committed anything, but in 
practice, privileged-ports set is already registered and its members are 
already added.

Apart from the obvious error of ipset committing before the actual 
"COMMIT" has taken place, this raises another issue when I actually try 
to reload this file - I will get an error straight away as 
privileged-ports is already registered and that shouldn't be the case. 
Thought to let you know.

Re: [ANNOUNCE] ipset 6.6 released

[Mr Dash Four]

> Yes, exactly. With the hashsize parameter, you tell the system how much 
> memory is allocated for the set when created. With maxelem it is 
> instructed at what point to stop increasing the set and refuse adding more 
> elements.
>   
It turns out that ipset is more efficient in that respect as well. Over 
15k members take about 0.6MB of RAM, which isn't that bad!

I have one suggestion though: it would be nice if you could implement a 
"check" option to accompany restore so that ipset doesn't actually do 
anything, but just checks the syntax of that file (a dummy run, if you 
like) and reports any errors or omissions (it could be more than one!). 
It would be nice to have that feature and help in avoiding errors (see 
also my other post with regards to that).

Re: [ANNOUNCE] ipset 6.6 released

[Jozsef Kadlecsik]

On Wed, 25 May 2011, Mr Dash Four wrote:

> > Userspace changes:
> >  - Restore with bitmap:port and list:set types did not work, fixed
> >   
> Having had the chance to test this now, I can say that it works, and what's
> more - the loading performance is much, much better - all my sets now load in
> about 2-3 seconds, while with the 4.5 version it took in excess of 10 minutes,
> completely hogging my CPU in the process. I haven't had the chance yet to
> judge the matching performance, but this is what I will do in the coming days.
> 
> I have found a bug, however. :-\
> 
> When I have multiple sets of different type to restore, each restore file
> normally ends with "COMMIT" statement for ipset to commit the whole
> transaction, or so I thought. If there is a mistake (syntax or any other)
> somewhere in the restore file, which prevents the restore process, ipset
> already commits everything up to that point, which I think is wrong.
> 
> For example, if I have this:
> 
> n privileged-ports bitmap:port range 1-1023 timeout 0
> a privileged-ports 1-1023
> n test-ports bitmap:port range 12770-19999 timeout 0
> a test-ports 20000-30000
> a test-ports 19999
> n test-port bitmap:port range 29950-29950 timeout 0
> a test-port 29950
> COMMIT
> 
> There is an obvious error in line 4 above ("a test-ports 20000-30000" - this
> is out of the defined range for this set) - ipset should have aborted the
> whole transaction and not committed anything, but in practice,
> privileged-ports set is already registered and its members are already added.
> 
> Apart from the obvious error of ipset committing before the actual "COMMIT"
> has taken place, this raises another issue when I actually try to reload this
> file - I will get an error straight away as privileged-ports is already
> registered and that shouldn't be the case. Thought to let you know.

If you have got a restore file with an error in it, fix the file and 
restore it again but with the '-!' flag. That way all already defined 
sets, elements are ignored (technically: not raised the error due to the 
clashing set/elements):

# ipset restore < file
<syntax error, fix the file>
# ipset -! restore < file

The "COMMIT" string is not required at all in ipset 6.x. It's just kept 
for backward compatibility reasons.

Best regards,
Jozsef
-
E-mail  : kadlec@blackhole.kfki.hu, kadlec@mail.kfki.hu
PGP key : http://www.kfki.hu/~kadlec/pgp_public_key.txt
Address : KFKI Research Institute for Particle and Nuclear Physics
          H-1525 Budapest 114, POB. 49, Hungary

Re: [ANNOUNCE] ipset 6.6 released

[Mr Dash Four]

> If you have got a restore file with an error in it, fix the file and 
> restore it again but with the '-!' flag. That way all already defined 
> sets, elements are ignored (technically: not raised the error due to the 
> clashing set/elements):
>
> # ipset restore < file
> <syntax error, fix the file>
> # ipset -! restore < file
>
> The "COMMIT" string is not required at all in ipset 6.x. It's just kept 
> for backward compatibility reasons.
>   
Didn't know that, thanks Jozsef - I'll remove all the COMMITs in the 
future as well.