[ANNOUNCE] ipset 6.6 released

[ANNOUNCE] ipset 6.6 released

[Jozsef Kadlecsik]

Hi,

Here comes ipset 6.6 with a couple of bugfixes and improvements. The terse 
list of the changes are:

Kernel part changes:
 - Use unified from/to address masking and check the usage
 - ip_set_flush returned -EPROTO instead of -IPSET_ERR_PROTOCOL, fixed
 - Take into account cidr value for the from address when creating the set
 - Adding ranges to hash types with timeout could still fail, fixed
 - Removed old, not used hashing method ip_set_chash
 - Remove variable 'ret' in type_pf_tdel(), which is set but not used
 - Use proper timeout parameter to jiffies conversion

Userspace changes:
 - Restore with bitmap:port and list:set types did not work, fixed
 - Accept "\r\n" terminated COMMIT command in restore files
 - Fix the message sequence number book-keeping
 - Protocol-level debugging support added
 - hash:net stress test in range notation added
 - ipset_mnl_query: in debug mode print the errno returned by the cb
   function
 - Accept "\r\n" terminated lines in restore files
 - Remove outdated checking of IPv6 support from configure.ac

I thank to Mr Dash Four for his tireless testings and helpful reports.

The next ipset package release will come with code simplifications, which 
mean dropping some support:

- iptree and iptreemap type support will be removed, because the automatic
  conversion to hash:ip is not fully backward compatible. The hash:ip and 
  hash:net types can be used instead.
- kernel version support below 2.6.37 will be dropped, to make code
  maintenance easier and more straightforward

You can download the source code of ipset from:
        http://ipset.netfilter.org
        ftp://ftp.netfilter.org/pub/ipset/
        git://git.netfilter.org/ipset.git

Best regards,
Jozsef
-
E-mail  : kadlec@blackhole.kfki.hu, kadlec@mail.kfki.hu
PGP key : http://www.kfki.hu/~kadlec/pgp_public_key.txt
Address : KFKI Research Institute for Particle and Nuclear Physics
          H-1525 Budapest 114, POB. 49, Hungary

Re: [ANNOUNCE] ipset 6.6 released

[Mr Dash Four]

> I thank to Mr Dash Four for his tireless testings and helpful reports.
>   
I thank you for your continuing development of this great tool which, at 
least in my case, hugely improved the stability and performance, as well 
as decreased the maintenance needed on all my systems as a direct result 
of employing this great tool!

> The next ipset package release will come with code simplifications, which 
> mean dropping some support:
>
> - iptree and iptreemap type support will be removed, because the automatic
>   conversion to hash:ip is not fully backward compatible. The hash:ip and 
>   hash:net types can be used instead.
> - kernel version support below 2.6.37 will be dropped, to make code
>   maintenance easier and more straightforward
>   
What are your plans with regards to the last point? The reason I am 
asking is because I am still on .35 (the last "stable" kernel released 
by Fedora) and I might not be able to use new releases if you introduce 
this restriction.

Also, would you consider my suggestion to include the number of members 
registered in a set to be displayed in the set headers (something like 
"Members: 15156") - this would help with managing the number of members 
in that set, which would be particularly useful for sets with large 
number of members.

Re: [ANNOUNCE] ipset 6.6 released

[Jozsef Kadlecsik]

On Tue, 24 May 2011, Mr Dash Four wrote:

> > The next ipset package release will come with code simplifications, which
> > mean dropping some support:
> > 
> > - iptree and iptreemap type support will be removed, because the automatic
> >   conversion to hash:ip is not fully backward compatible. The hash:ip and
> > hash:net types can be used instead.
> > - kernel version support below 2.6.37 will be dropped, to make code
> >   maintenance easier and more straightforward
> >   
> What are your plans with regards to the last point? The reason I am asking is
> because I am still on .35 (the last "stable" kernel released by Fedora) and I
> might not be able to use new releases if you introduce this restriction.

Hm, 2.6.35 can lessen the maintenance burden compared to the currently 
minimal supported version 2.6.34, because the main trouble comes from the 
differences between .34 and .35. So I think I can remove the older kernel
supports gradually and keep supporting 2.6.35 for a while.
 
> Also, would you consider my suggestion to include the number of members
> registered in a set to be displayed in the set headers (something like
> "Members: 15156") - this would help with managing the number of members in
> that set, which would be particularly useful for sets with large number of
> members.

The problem with it that the reported number can be inaccurate, at least 
in two cases:

- Elements can time out, so even if whatever number reported, by the
  time it's displayed, the set can even be completely empty. In the case
  of a huge set, it can even occur that the number of elements reported
  does not match the actual number of elements listed.
- Sets can be updated by the SET target

The first one is the main reason I dropped reporting the number of 
elements (the initial design of the new ipset included it).

Best regards,
Jozsef
-
E-mail  : kadlec@blackhole.kfki.hu, kadlec@mail.kfki.hu
PGP key : http://www.kfki.hu/~kadlec/pgp_public_key.txt
Address : KFKI Research Institute for Particle and Nuclear Physics
          H-1525 Budapest 114, POB. 49, Hungary

ipset 6.6 bug: subnet (mis)matching

[Mr Dash Four]

[root@test1 ~]# ipset n test hash:net family inet timeout 0
[root@test1 ~]# ipset a test 10.16.0.0-10.16.255.255
[root@test1 ~]# ipset l test
Name: test
Type: hash:net
Header: family inet hashsize 1024 maxelem 65536 timeout 0 
Size in memory: 16832
References: 0
Members:
10.16.0.0/16 timeout 0
[root@test1 ~]# ipset t test 10.16.224.0/24
10.16.224.0/24 is NOT in set test.
[root@test1 ~]# ipset t test 10.16.224.1   
10.16.224.1 is in set test.

That is plainly wrong!

Re: ipset 6.6 bug: subnet (mis)matching

[Jozsef Kadlecsik]

On Wed, 8 Jun 2011, Mr Dash Four wrote:

> [root@test1 ~]# ipset n test hash:net family inet timeout 0
> [root@test1 ~]# ipset a test 10.16.0.0-10.16.255.255
> [root@test1 ~]# ipset l test
> Name: test
> Type: hash:net
> Header: family inet hashsize 1024 maxelem 65536 timeout 0 
> Size in memory: 16832
> References: 0
> Members:
> 10.16.0.0/16 timeout 0
> [root@test1 ~]# ipset t test 10.16.224.0/24
> 10.16.224.0/24 is NOT in set test.
> [root@test1 ~]# ipset t test 10.16.224.1   
> 10.16.224.1 is in set test.
> 
> That is plainly wrong!

No, that's a feature: it makes possible to check from userspace how the 
kernel would match an IP address in the set. Maybe it's badly worded in 
the manpage: "When testing entries, if a host address is tested, then the 
kernel tries to match the host address in the networks added to the set 
and reports the result accordingly."

Best regards,
Jozsef
-
E-mail  : kadlec@blackhole.kfki.hu, kadlec@mail.kfki.hu
PGP key : http://www.kfki.hu/~kadlec/pgp_public_key.txt
Address : KFKI Research Institute for Particle and Nuclear Physics
          H-1525 Budapest 114, POB. 49, Hungary

Re: ipset 6.6 bug: subnet (mis)matching

[Mr Dash Four]

>> [root@test1 ~]# ipset n test hash:net family inet timeout 0
>> [root@test1 ~]# ipset a test 10.16.0.0-10.16.255.255
>> [root@test1 ~]# ipset l test
>> Name: test
>> Type: hash:net
>> Header: family inet hashsize 1024 maxelem 65536 timeout 0 
>> Size in memory: 16832
>> References: 0
>> Members:
>> 10.16.0.0/16 timeout 0
>> [root@test1 ~]# ipset t test 10.16.224.0/24
>> 10.16.224.0/24 is NOT in set test.
>> [root@test1 ~]# ipset t test 10.16.224.1   
>> 10.16.224.1 is in set test.
>>
>> That is plainly wrong!
>>     
>
> No, that's a feature: it makes possible to check from userspace how the 
> kernel would match an IP address in the set.
You can't be serious! How could this be a "feature"?! It is a bug, clearly!

The /24 subnet (10.16.224.0-10.16.224.255) clearly covers the single IP 
address (10.16.224.1) and the test should be positive in both cases, as 
the /16 member of the set (10.16.0.0-10.16.255.255) covers both the 
single IP address as well as the /24subnet range tested. Apparently, 
that seems not to be the case. How am I suppose to test subnet ranges then?

I presume if I have 10.16.224.0/24 as a member and do a test with 
10.16.0.0/16 that would also return a negative match, isn't that the case?

>  Maybe it's badly worded in 
> the manpage: "When testing entries, if a host address is tested, then the 
> kernel tries to match the host address in the networks added to the set 
> and reports the result accordingly."
>   
That's all well and good for hosts, but ip ranges should either be 
tested properly or testing of those should be disabled altogether 
(which, if introduced, would be a severe restriction for me - I am not 
going to run a couple of thousand "ipset t" tests just to see that all 
of the single IP addresses in the range I am interested in match - that 
simply isn't going to happen!).

Re: ipset 6.6 bug: subnet (mis)matching

[Jozsef Kadlecsik]

On Wed, 8 Jun 2011, Mr Dash Four wrote:
 
> > > [root@test1 ~]# ipset n test hash:net family inet timeout 0
> > > [root@test1 ~]# ipset a test 10.16.0.0-10.16.255.255
> > > [root@test1 ~]# ipset l test
> > > Name: test
> > > Type: hash:net
> > > Header: family inet hashsize 1024 maxelem 65536 timeout 0 Size in memory:
> > > 16832
> > > References: 0
> > > Members:
> > > 10.16.0.0/16 timeout 0
> > > [root@test1 ~]# ipset t test 10.16.224.0/24
> > > 10.16.224.0/24 is NOT in set test.
> > > [root@test1 ~]# ipset t test 10.16.224.1   10.16.224.1 is in set test.
> > > 
> > > That is plainly wrong!
> > 
> > No, that's a feature: it makes possible to check from userspace how the
> > kernel would match an IP address in the set.
> You can't be serious! How could this be a "feature"?! It is a bug, clearly!
> 
> The /24 subnet (10.16.224.0-10.16.224.255) clearly covers the single IP
> address (10.16.224.1) and the test should be positive in both cases, as the
> /16 member of the set (10.16.0.0-10.16.255.255) covers both the single IP
> address as well as the /24subnet range tested. Apparently, that seems not to
> be the case. How am I suppose to test subnet ranges then?

I can only repeat myself: for the hash:*net* types the testing with a host 
address is special and handled as the kernel would test an IP address.
However network addresses are handled as a whole and overlapping is *not* 
checked. It's documented in the manpage.

Maybe an example helps: when you add 10.6.0.0/16, you add an "apple". You 
can also add 10.6.0.0/24, a "pear". Nothing prevents you, both can be the 
member of the set, because overlapping is not handled. At deleting, 
testing they are handled as "apple" and "pear", too. If "pear" is not 
added to the set, it's not "in" the set as a member - even if we know, 
that the elements of "apple" fully cover the elements of "pear". However 
at testing 10.16.224.1[/32] you test a "seed", which is checked inside 
"apple".

Merging with already existing elements (or breaking up if a network from a 
larger network were deleted) would be very expensive. I know, you compare 
the types to iptreemap, but iptreemap was a very special case where 
merging/breaking up was a natural process and costed almost nothing.

> I presume if I have 10.16.224.0/24 as a member and do a test with 10.16.0.0/16
> that would also return a negative match, isn't that the case?

Yes, of course.
 
> >  Maybe it's badly worded in the manpage: "When testing entries, if a host
> > address is tested, then the kernel tries to match the host address in the
> > networks added to the set and reports the result accordingly."
> >   
> That's all well and good for hosts, but ip ranges should either be tested
> properly or testing of those should be disabled altogether (which, if
> introduced, would be a severe restriction for me - I am not going to run a
> couple of thousand "ipset t" tests just to see that all of the single IP
> addresses in the range I am interested in match - that simply isn't going to
> happen!).

I don't quite follow you here: you can add, delete and test the elements 
by ipset. As a special feature, you can also test the members of elements, 
too by ipset.

Probably what you expect is the following:

    add 192.168.0.0/24
    add 192.168.1.0/24
		magically aggregate into 192.168.0.0/23
    add 192.168.2.0/24
    add 192.168.3.0/24
		magically aggregate into 192.168.0.0/22
    del 192.168.1.0/24
		break up into the elements
			192.168.0.0/24
			192.168.2.0/23
		
There's no such magic there.

Best regards,
Jozsef
-
E-mail  : kadlec@blackhole.kfki.hu, kadlec@mail.kfki.hu
PGP key : http://www.kfki.hu/~kadlec/pgp_public_key.txt
Address : KFKI Research Institute for Particle and Nuclear Physics
          H-1525 Budapest 114, POB. 49, Hungary

Re: ipset 6.6 bug: subnet (mis)matching

[Mr Dash Four]

> I can only repeat myself: for the hash:*net* types the testing with a host 
> address is special and handled as the kernel would test an IP address.
> However network addresses are handled as a whole and overlapping is *not* 
> checked. It's documented in the manpage.
>   
I can understand that to a point as far as adding/deleting 
elements/subranges goes - it makes sense. Testing, however, is a 
different story.

> Maybe an example helps: when you add 10.6.0.0/16, you add an "apple". You 
> can also add 10.6.0.0/24, a "pear". Nothing prevents you, both can be the 
> member of the set, because overlapping is not handled. At deleting, 
> testing they are handled as "apple" and "pear", too. If "pear" is not 
> added to the set, it's not "in" the set as a member - even if we know, 
> that the elements of "apple" fully cover the elements of "pear". However 
> at testing 10.16.224.1[/32] you test a "seed", which is checked inside 
> "apple".
>   
Your basic presumption above is wrong - 10.16.224.1 *is* a member - i.e. 
it belongs, in network terms at least - to both 10.16.0.0/16 and 
10.16.224.0/24 ranges. Similarly, 10.16.224.0/24 also belongs to 
10.16.0.0/16 in the same way that 10.16.224.1 does to 10.16.224.0/24 and 
10.16.0.0/16.

10.16.224.1 is a member of the 10.16.224.0/24 "set", which, in turn, is 
also a member of 10.16.0.0/16 "set" (in mathematical as well as in 
networking terms). They are both in a way "subsets". Any tests for 
matching a subset belonging to a bigger set should return positive 
results. In the case of ipset they don't and that is my point.

There are no "apples" and "oranges" compared here as both ranges belong 
to each other, so they are not different from each other, apart from the 
fact that one set is bigger than the other. Basic grasp of maths would 
tell you all that.

> Merging with already existing elements (or breaking up if a network from a 
> larger network were deleted) would be very expensive. I know, you compare 
> the types to iptreemap, but iptreemap was a very special case where 
> merging/breaking up was a natural process and costed almost nothing.
>   
No, I don't make comparisons with iptreemap at all - there is no 
iptreemap set type involvement here, so there is no need for me to 
mention that type of set.

My basic (and only!) issue is that subnet ranges testing should either 
be implemented properly or not tested at all. The way I see it, that is 
not the case and I pointed out above why I think that is.

>> That's all well and good for hosts, but ip ranges should either be tested
>> properly or testing of those should be disabled altogether (which, if
>> introduced, would be a severe restriction for me - I am not going to run a
>> couple of thousand "ipset t" tests just to see that all of the single IP
>> addresses in the range I am interested in match - that simply isn't going to
>> happen!).
>>     
>
> I don't quite follow you here:
OK, if I have 10.16.0.0/16 as a member and then test a subnet range 
which is covered by the /16 address - again, lets say 10.16.1.0/24 - I 
expect to get a match as clearly, from both mathematical as well as 
networking terms, all "members" of 10.16.1.0/24 are also "members" of 
10.16.0.0/16, so the test result should return a positive. It doesn't!

I can understand a negative result if I test the other way around - not 
all "members" of 10.16.0.0/16 would belong to 10.16.1.0/24 and returning 
a negative test result in this instance is OK.

It doesn't work like that in ipset though - ipset seems to be operating 
either on "exact match" (say if I have 10.16.0.0/16 as a member and test 
with 10.16.0.0/16) or when a single ip address is tested, which I don't 
think is right and that was my point all along.

Re: ipset 6.6 bug: subnet (mis)matching

[Jozsef Kadlecsik]

On Wed, 8 Jun 2011, Mr Dash Four wrote:

> OK, if I have 10.16.0.0/16 as a member and then test a subnet range which is
> covered by the /16 address - again, lets say 10.16.1.0/24 - I expect to get a
> match as clearly, from both mathematical as well as networking terms, all
> "members" of 10.16.1.0/24 are also "members" of 10.16.0.0/16, so the test
> result should return a positive. It doesn't!
> 
> I can understand a negative result if I test the other way around - not all
> "members" of 10.16.0.0/16 would belong to 10.16.1.0/24 and returning a
> negative test result in this instance is OK.
> 
> It doesn't work like that in ipset though - ipset seems to be operating either
> on "exact match" (say if I have 10.16.0.0/16 as a member and test with
> 10.16.0.0/16) or when a single ip address is tested, which I don't think is
> right and that was my point all along.

Current behaviour cannot be modified easily. There's no overlap checking 
(and aggregation/breaking up) in ipset, so it can't be prevented that 
someone adds overlapping elements to a set:

add 10.16.0.0/16
add 10.16.1.0/24

Now if 10.16.1.0/24 is tested as you suggest, one cannot tell whether the 
explicit network 10.16.1.0/24 is added to the set or the tested elements 
are just the members of a larger network.

Currently:
    test 10.16.1.1	=> test explicit and implicit membership
    test 10.16.1.0/24	=> test explicit membership only

What you propose is to test explicit and implicit membership in all cases.

My top priority now is to get the kernel in sync with ipset 6.7. It's just 
a much higher problem, because a lot of features and fixes are missing, 
even from the newest kernel version.

Best regards,
Jozsef
-
E-mail  : kadlec@blackhole.kfki.hu, kadlec@mail.kfki.hu
PGP key : http://www.kfki.hu/~kadlec/pgp_public_key.txt
Address : KFKI Research Institute for Particle and Nuclear Physics
          H-1525 Budapest 114, POB. 49, Hungary

Re: ipset 6.6 bug: subnet (mis)matching

[Mr Dash Four]

> Current behaviour cannot be modified easily. There's no overlap checking 
> (and aggregation/breaking up) in ipset, so it can't be prevented that 
> someone adds overlapping elements to a set:
>
> add 10.16.0.0/16
> add 10.16.1.0/24
>
> Now if 10.16.1.0/24 is tested as you suggest, one cannot tell whether the 
> explicit network 10.16.1.0/24 is added to the set or the tested elements 
> are just the members of a larger network.
>
> Currently:
>     test 10.16.1.1	=> test explicit and implicit membership
>     test 10.16.1.0/24	=> test explicit membership only
>
> What you propose is to test explicit and implicit membership in all cases.
>   
The way I see it, you can take the base of a set member (10.16.0.0 in my 
case), build up a mask with the range (16), do the same with the range 
being tested and then match them with logical "and". If you get the same 
result as the set member - you have a match, if you don't there isn't a 
(complete) match.

Alternatively, you could loop through the members of the specified range 
to be tested and compare against the set members, but that would be a 
bit slower, though since this is done in userspace (i.e. interactively) 
speed wouldn't be that important compared to the kernel matching where 
good performance is needed.

> My top priority now is to get the kernel in sync with ipset 6.7. It's just 
> a much higher problem, because a lot of features and fixes are missing, 
> even from the newest kernel version.
>   
I am not asking or demanding anything to be implemented/corrected 
immediately - I am simply making you aware of a bug in ipset I found 
last night. If/When you are going to fix it it is for you to decide - 
you are the author of ipset, not me - I am just a user.