[ANNOUNCE] ipset-5.0 released

[ANNOUNCE] ipset-5.0 released

[Jozsef Kadlecsik]

Hi,

I'm happy to announce the new branch of ipset and release it's first 
element, ipset-5.0.

Just the most important improvements and changes compared to the 4.x tree:

- The hash types support both IPv4 and IPv6.
- The limitation that the elements of the ipporthash, ipportiphash and 
  ipportnethash types had to come from a max /16 netblock is removed.
- The limitation that the ipporthash and ipportnethash types could
  not store host IP addresses is removed.
- The hash set types with port sub-element are extended and the protocol
  of the port is also stored. Now you can store in a single set
  IP(v4/v6) IP address and TCP or UDP port number pairs, or even IP 
  address and ICMP(v4/v6) type/code pairs, etc.
- All set types in the 5.0 branch supports timeouts.
- Addition or deletion of multiple elements (where applicable)
  in one step is supported.
- Netlink based kernel-userspace communication protocol.
- Faster hashing method behind the hash types, which at the same time
  uses less memory.
- New, improved syntax.
- Backward compatibility layer in syntax: old 4.x commands, saved
  sets are accepted.

Please note, due to the new protocol, the ipset 5.x utility cannot talk
to the kernel modules from ipset 4.x and similarly, the ipset 4.x utility
cannot communicate with the kernel part of ipset 5.x.
Read the README file in the source tree and the new manpage of ipset.

You can download the source code from:
	http://ipset.netfilter.org
	ftp://ftp.netfilter.org/pub/ipset/
	git://git.netfilter.org/ipset.git, ipset-5 branch

Best regards,
Jozsef
-
E-mail  : kadlec@blackhole.kfki.hu, kadlec@mail.kfki.hu
PGP key : http://www.kfki.hu/~kadlec/pgp_public_key.txt
Address : KFKI Research Institute for Particle and Nuclear Physics
          H-1525 Budapest 114, POB. 49, Hungary

Re: [ANNOUNCE] ipset-5.0 released

[Jan Engelhardt]

On Friday 2010-12-17 23:26, Jozsef Kadlecsik wrote:

>Hi,
>
>I'm happy to announce the new branch of ipset and release it's first 
>element, ipset-5.0.

That is great to hear. There is a compile error with gcc 4.5
and the latest ipset git snapshot:

  CC     ipset.o
cc1: warnings being treated as errors
ipset.c: In function ‘check_allowed’:
ipset.c:344:26: error: comparison between ‘enum ipset_cmd’ and ‘enum ipset_adt’


The kernel/ directory is not compilable at all:

make -C /lib/modules/`uname -r`/build M=`pwd` all
make[1]: Entering directory `/usr/src/linux-2.6.36-rc8-34-obj/x86_64/default'
make -C /usr/src/linux-2.6.36-rc8-34 O=/usr/src/linux-2.6.36-rc8-34-obj/x86_64/default/. 
  CC [M]  /home/jengelh/code/ipset/kernel/ip_set.o
kernel/ip_set.c:31:50: error: expected expression before ‘;’ token

^ CONFIG_IP_SET_MAX not defined


kernel/ip_set.c: In function ‘start_msg’:
kernel/ip_set.c:735:40: error: ‘NFNL_SUBSYS_IPSET’ undeclared (first use in this function)

^ Neither is this, but this was to be expected. We should really
make this constant available in the Linux kernel so that people
are able to compile the out-of-tree ipset until it gets in.

(I hear this issue would not have happened would we have used genetlink!
*nudges pablo* ;-)


kernel/ip_set.c: At top level:
kernel/ip_set.c:1633:3: warning: initialization from incompatible pointer type

^ This is due to NFNL_CB_CONST not being set.
--
To unsubscribe from this list: send the line "unsubscribe netfilter-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [ANNOUNCE] ipset-5.0 released

[Jozsef Kadlecsik]

Hi Jan,

On Sat, 18 Dec 2010, Jan Engelhardt wrote:

> On Friday 2010-12-17 23:26, Jozsef Kadlecsik wrote:
> 
> >I'm happy to announce the new branch of ipset and release it's first 
> >element, ipset-5.0.
> 
> That is great to hear. There is a compile error with gcc 4.5
> and the latest ipset git snapshot:
> 
>   CC     ipset.o
> cc1: warnings being treated as errors
> ipset.c: In function ?check_allowed?:
> ipset.c:344:26: error: comparison between ?enum ipset_cmd? and ?enum ipset_adt?

I have fixed it in the git tree, thanks! 
  
> The kernel/ directory is not compilable at all:

I assume you tried to compile the source in the kernel/ directory itself. 
You should have used the toplevel Makefile, as it's stated in the README file.
 
> make -C /lib/modules/`uname -r`/build M=`pwd` all
> make[1]: Entering directory `/usr/src/linux-2.6.36-rc8-34-obj/x86_64/default'
> make -C /usr/src/linux-2.6.36-rc8-34 O=/usr/src/linux-2.6.36-rc8-34-obj/x86_64/default/. 
>   CC [M]  /home/jengelh/code/ipset/kernel/ip_set.o
> kernel/ip_set.c:31:50: error: expected expression before ?;? token
> 
> ^ CONFIG_IP_SET_MAX not defined

It's set in the toplevel Makefile and propagated into the EXTRA_CFLAGS of 
kernel/Kbuild.
  
> kernel/ip_set.c: In function ?start_msg?:
> kernel/ip_set.c:735:40: error: ?NFNL_SUBSYS_IPSET? undeclared (first use in this function)
> 
> ^ Neither is this, but this was to be expected. We should really
> make this constant available in the Linux kernel so that people
> are able to compile the out-of-tree ipset until it gets in.
> 
> (I hear this issue would not have happened would we have used genetlink!
> *nudges pablo* ;-)

It's all right, the kernel was not patched.

> kernel/ip_set.c: At top level:
> kernel/ip_set.c:1633:3: warning: initialization from incompatible pointer type
> 
> ^ This is due to NFNL_CB_CONST not being set.

Again, it's set in the toplevel directory.

After checking out the git tree and working in the toplevel directory, you 
should be able to compile the code without any error.

Best regards,
Jozsef
-
E-mail  : kadlec@blackhole.kfki.hu, kadlec@mail.kfki.hu
PGP key : http://www.kfki.hu/~kadlec/pgp_public_key.txt
Address : KFKI Research Institute for Particle and Nuclear Physics
          H-1525 Budapest 114, POB. 49, Hungary

Re: [ANNOUNCE] ipset-5.0 released

[Rob Sterenborg (lists)]

On Fri, 2010-12-17 at 23:26 +0100, Jozsef Kadlecsik wrote:
> Hi,
> 
> I'm happy to announce the new branch of ipset and release it's first 
> element, ipset-5.0.

I'm not a C programmer. I just tried to make ipset compile which seems
to have worked partially. I have no clue if I did the right thing so the
below should be reviewed.

I'm on CentOS 5.5 with a custom 2.6.36.2 kernel, gcc version 4.1.2
20080704 (Red Hat 4.1.2-48).

When running 'configure' I got this error:

./configure: line 11510: syntax error near unexpected token `[libmnl],'
./configure: line 11510: `PKG_CHECK_MODULES([libmnl], [libmnl >= 1])'

CentOS' pkg-config is installed, so, for reference: I copied
'/usr/share/aclocal/pkg.m4' into the 'm4' directory, ran 'autogen.sh'
again and after that 'configure' had no problems.

Running 'make', I got this :

cc1: error: unrecognized command line option "-Woverlength-strings"

If I remove '-Woverlength-strings' from all Makefiles then of course
there's no complaining about that anymore, but I'm not sure if that's
the way to go.

Next, I got this:

session.c: In function 'attr2data':
session.c:566: error: 'NLA_F_NET_BYTEORDER' undeclared (first use in
this function)
session.c:566: error: (Each undeclared identifier is reported only once
session.c:566: error: for each function it appears in.)
session.c: In function 'decode_errmsg':
session.c:1216: error: 'NLA_F_NET_BYTEORDER' undeclared (first use in
this function)
session.c: In function 'attr_len':
session.c:1338: error: 'NLA_F_NET_BYTEORDER' undeclared (first use in
this function)

To make it compile I did the following.
New file 'include/libipset/nla.h':

/*
* nla_type (16 bits)
* +---+---+-------------------------------+
* | N | O | Attribute Type                |
* +---+---+-------------------------------+
* N := Carries nested attributes
* O := Payload stored in network byte order
*
* Note: The N and O flag are mutually exclusive.
*/

#define NLA_F_NESTED            (1 << 15)
#define NLA_F_NET_BYTEORDER     (1 << 14)
#define NLA_TYPE_MASK           ~(NLA_F_NESTED | NLA_F_NET_BYTEORDER)

Change in 'lib/session.c':

--- session.c.orig      2010-12-18 08:00:31.000000000 +0100
+++ session.c   2010-12-18 07:59:48.000000000 +0100
@@ -23,6 +23,9 @@
 #include <libipset/utils.h>                    /* STREQ */
 #include <libipset/ui.h>                       /* IPSET_ENV_* */
 #include <libipset/session.h>                  /* prototypes */
+#ifndef NLA_TYPE_MASK
+#include <libipset/nla.h>
+#endif
 
 
 #define IPSET_NEST_MAX 4

After that 'make' runs without errors.
Running 'make modules' gives:

/usr/local/src/netfilter/ipset-5.0/kernel/ip_set.c: In function
'start_msg':
/usr/local/src/netfilter/ipset-5.0/kernel/ip_set.c:735: error:
'NFNL_SUBSYS_IPSET' undeclared (first use in this function)
/usr/local/src/netfilter/ipset-5.0/kernel/ip_set.c:735: error: (Each
undeclared identifier is reported only once
/usr/local/src/netfilter/ipset-5.0/kernel/ip_set.c:735: error: for each
function it appears in.)
/usr/local/src/netfilter/ipset-5.0/kernel/ip_set.c: At top level:
/usr/local/src/netfilter/ipset-5.0/kernel/ip_set.c:1701: error:
'NFNL_SUBSYS_IPSET' undeclared here (not in a function)
make[2]: *** [/usr/local/src/netfilter/ipset-5.0/kernel/ip_set.o] Error
1
make[1]: *** [_module_/usr/local/src/netfilter/ipset-5.0/kernel] Error 2
make[1]: Leaving directory `/usr/local/src/kernel/linux-2.6.36.2'
make: *** [modules] Error 2

I noticed there was a 'netlink.patch' file that I tried to apply to
'/usr/include/linux/netfilter/nfnetlink.h', but it wouldn't: it looks
like your nfnetlink.h is different from mine (can send a copy of the
original if you need it) so I applied manually. Still no go, same error,
probably wrong location? I copied 'nfnetlink.h' into
'kernel/include/linux/netfilter' because ip_set.c seems to look there(?)
but it wasn't there. Still no go. To get around this I dit this:

New file 'kernel/include/linux/netfilter/nfnl.h':

/* netfilter netlink message types are split in two pieces:
* 8 bit subsystem, 8bit operation.
*/

#define NFNL_SUBSYS_ID(x)       ((x & 0xff00) >> 8)
#define NFNL_MSG_TYPE(x)        (x & 0x00ff)

/* No enum here, otherwise __stringify() trick of
MODULE_ALIAS_NFNL_SUBSYS()
* won't work anymore */
#define NFNL_SUBSYS_NONE                0
#define NFNL_SUBSYS_CTNETLINK           1
#define NFNL_SUBSYS_CTNETLINK_EXP       2
#define NFNL_SUBSYS_QUEUE               3
#define NFNL_SUBSYS_ULOG                4
#define NFNL_SUBSYS_OSF                 5
#define NFNL_SUBSYS_IPSET               6
#define NFNL_SUBSYS_COUNT               7

Change in 'kernel/ip_set.c'

--- ip_set.c.orig       2010-12-16 12:26:02.000000000 +0100
+++ ip_set.c    2010-12-18 08:30:47.000000000 +0100
@@ -24,6 +24,10 @@
 #include <linux/netfilter/nfnetlink.h>
 #include <linux/netfilter/ipset/ip_set.h>
 
+#ifndef NFNL_SUBSYS_IPSET
+#include <linux/netfilter/nfnl.h>
+#endif
+
 static struct list_head ip_set_type_list;      /* all registered set
types */
 static DEFINE_MUTEX(ip_set_type_mutex);                /* protects
ip_set_type_list */
 
After that, 'make modules' for some reason warns about redefines. first
they weren't defined, now they're redefined when I use ifndef? Removing
the include, make -of course- complains again that 'NFNL_SUBSYS_IPSET'
is not defined. Well, I don't know..
Other than that everything seems to compile and install fine.

Finally, when trying the new ipset it seems that except for 'version',
every command I tried returns 'Invalid argument':

(Yes I know this is incorrect syntax, but now I know it's trying to do
something besides giving me 'Invalid argument'.)
# ipset create TEST hash   
ipset v5.0: Syntax error: typename 'hash' is unkown

(As per ipset.8 example.)
# ipset create foo bitmap:ip range 192.168.0.0/16
ipset v5.0: Kernel error received: Invalid argument

# ipset list               
ipset v5.0: Kernel error received: Invalid argument

# lsmod|grep set
ip_set                 16790  0 
nfnetlink               3179  2 ip_set,nf_conntrack_netlink

So, I guess something must have gone wrong when compiling ipset anyhow.


Thanks,
Rob

Re: [ANNOUNCE] ipset-5.0 released

[Jozsef Kadlecsik]

Hi Rob,

On Sat, 18 Dec 2010, Rob Sterenborg (lists) wrote:

> On Fri, 2010-12-17 at 23:26 +0100, Jozsef Kadlecsik wrote:
> > 
> > I'm happy to announce the new branch of ipset and release it's first 
> > element, ipset-5.0.
> 
> I'm not a C programmer. I just tried to make ipset compile which seems
> to have worked partially. I have no clue if I did the right thing so the
> below should be reviewed.
> 
> I'm on CentOS 5.5 with a custom 2.6.36.2 kernel, gcc version 4.1.2
> 20080704 (Red Hat 4.1.2-48).
> 
> When running 'configure' I got this error:
> 
> ./configure: line 11510: syntax error near unexpected token `[libmnl],'
> ./configure: line 11510: `PKG_CHECK_MODULES([libmnl], [libmnl >= 1])'
> 
> CentOS' pkg-config is installed, so, for reference: I copied
> '/usr/share/aclocal/pkg.m4' into the 'm4' directory, ran 'autogen.sh'
> again and after that 'configure' had no problems.

Autoconf has its own pitfalls... I can't reproduce it so I added
'aclocal -I m4' to autogen.sh. After checking out the git tree, could you 
give it a try whether it solves the issue?
 
> Running 'make', I got this :
> 
> cc1: error: unrecognized command line option "-Woverlength-strings"
> 
> If I remove '-Woverlength-strings' from all Makefiles then of course
> there's no complaining about that anymore, but I'm not sure if that's
> the way to go.

It's mentioned in the README file: you should simply re-run `configure'
with the additional flag '--disable-extra-flags' :-)

> Next, I got this:
> 
> session.c: In function 'attr2data':
> session.c:566: error: 'NLA_F_NET_BYTEORDER' undeclared (first use in
> this function)
> session.c:566: error: (Each undeclared identifier is reported only once
> session.c:566: error: for each function it appears in.)
> session.c: In function 'decode_errmsg':
> session.c:1216: error: 'NLA_F_NET_BYTEORDER' undeclared (first use in
> this function)
> session.c: In function 'attr_len':
> session.c:1338: error: 'NLA_F_NET_BYTEORDER' undeclared (first use in
> this function)

Your kernel header files at the default location is not recent enough.

I'm undecided yet how to solve it: maybe it should be checked by configure 
and fail immediately.
 
> To make it compile I did the following.
> New file 'include/libipset/nla.h':
> 
> /*
> * nla_type (16 bits)
> * +---+---+-------------------------------+
> * | N | O | Attribute Type                |
> * +---+---+-------------------------------+
> * N := Carries nested attributes
> * O := Payload stored in network byte order
> *
> * Note: The N and O flag are mutually exclusive.
> */
> 
> #define NLA_F_NESTED            (1 << 15)
> #define NLA_F_NET_BYTEORDER     (1 << 14)
> #define NLA_TYPE_MASK           ~(NLA_F_NESTED | NLA_F_NET_BYTEORDER)
> 
> Change in 'lib/session.c':
> 
> --- session.c.orig      2010-12-18 08:00:31.000000000 +0100
> +++ session.c   2010-12-18 07:59:48.000000000 +0100
> @@ -23,6 +23,9 @@
>  #include <libipset/utils.h>                    /* STREQ */
>  #include <libipset/ui.h>                       /* IPSET_ENV_* */
>  #include <libipset/session.h>                  /* prototypes */
> +#ifndef NLA_TYPE_MASK
> +#include <libipset/nla.h>
> +#endif

That's a possible solution but I'm uneasy about it.
  
>  #define IPSET_NEST_MAX 4
> 
> After that 'make' runs without errors.
> Running 'make modules' gives:
> 
> /usr/local/src/netfilter/ipset-5.0/kernel/ip_set.c: In function
> 'start_msg':
> /usr/local/src/netfilter/ipset-5.0/kernel/ip_set.c:735: error:
> 'NFNL_SUBSYS_IPSET' undeclared (first use in this function)
> /usr/local/src/netfilter/ipset-5.0/kernel/ip_set.c:735: error: (Each
> undeclared identifier is reported only once
> /usr/local/src/netfilter/ipset-5.0/kernel/ip_set.c:735: error: for each
> function it appears in.)
> /usr/local/src/netfilter/ipset-5.0/kernel/ip_set.c: At top level:
> /usr/local/src/netfilter/ipset-5.0/kernel/ip_set.c:1701: error:
> 'NFNL_SUBSYS_IPSET' undeclared here (not in a function)
> make[2]: *** [/usr/local/src/netfilter/ipset-5.0/kernel/ip_set.o] Error
> 1
> make[1]: *** [_module_/usr/local/src/netfilter/ipset-5.0/kernel] Error 2
> make[1]: Leaving directory `/usr/local/src/kernel/linux-2.6.36.2'
> make: *** [modules] Error 2
> 
> I noticed there was a 'netlink.patch' file that I tried to apply to
> '/usr/include/linux/netfilter/nfnetlink.h', but it wouldn't: it looks
> like your nfnetlink.h is different from mine (can send a copy of the
> original if you need it) so I applied manually. Still no go, same error,
> probably wrong location? I copied 'nfnetlink.h' into
> 'kernel/include/linux/netfilter' because ip_set.c seems to look there(?)
> but it wasn't there. Still no go. To get around this I dit this:

The netlink.patch file must be applied against the kernel tree with which 
you compile ipset. That is in your case the kernel tree at 
/usr/local/src/kernel/linux-2.6.36.2.
 
> Finally, when trying the new ipset it seems that except for 'version',
> every command I tried returns 'Invalid argument':
> 
> (Yes I know this is incorrect syntax, but now I know it's trying to do
> something besides giving me 'Invalid argument'.)
> # ipset create TEST hash   
> ipset v5.0: Syntax error: typename 'hash' is unkown
> 
> (As per ipset.8 example.)
> # ipset create foo bitmap:ip range 192.168.0.0/16
> ipset v5.0: Kernel error received: Invalid argument
> 
> # ipset list               
> ipset v5.0: Kernel error received: Invalid argument
> 
> # lsmod|grep set
> ip_set                 16790  0 
> nfnetlink               3179  2 ip_set,nf_conntrack_netlink
> 
> So, I guess something must have gone wrong when compiling ipset anyhow.

Please read the README file: you must patch your kernel source with 
netlink.patch, compile and install it. Otherwise the new nfnetlink id 
won't handled by the kernel and thus ipset can't work.

Thanks for tests and the reporting!

Best regards,
Jozsef
-
E-mail  : kadlec@blackhole.kfki.hu, kadlec@mail.kfki.hu
PGP key : http://www.kfki.hu/~kadlec/pgp_public_key.txt
Address : KFKI Research Institute for Particle and Nuclear Physics
          H-1525 Budapest 114, POB. 49, Hungary

Re: [ANNOUNCE] ipset-5.0 released

[Jan Engelhardt]

On Saturday 2010-12-18 12:13, Jozsef Kadlecsik wrote:
>On Sat, 18 Dec 2010, Rob Sterenborg (lists) wrote:
>
>> On Fri, 2010-12-17 at 23:26 +0100, Jozsef Kadlecsik wrote:
>> > 
>> > I'm happy to announce the new branch of ipset and release it's first 
>> > element, ipset-5.0.
>> 
>> I'm not a C programmer. I just tried to make ipset compile which seems
>> to have worked partially. I have no clue if I did the right thing so the
>> below should be reviewed.
>> 
>> I'm on CentOS 5.5 with a custom 2.6.36.2 kernel, gcc version 4.1.2
>> 20080704 (Red Hat 4.1.2-48).
>> 
>> When running 'configure' I got this error:
>> 
>> ./configure: line 11510: syntax error near unexpected token `[libmnl],'
>> ./configure: line 11510: `PKG_CHECK_MODULES([libmnl], [libmnl >= 1])'
>> 
>> CentOS' pkg-config is installed, so, for reference: I copied
>> '/usr/share/aclocal/pkg.m4' into the 'm4' directory, ran 'autogen.sh'
>> again and after that 'configure' had no problems.
>
>Autoconf has its own pitfalls... I can't reproduce it so I added
>'aclocal -I m4' to autogen.sh.

That is not necessary, because we already have ACLOCAL_AMFLAGS = -I m4
in Makefile.am.

>> Next, I got this:
>> 
>> session.c: In function 'attr2data':
>> session.c:566: error: 'NLA_F_NET_BYTEORDER' undeclared (first use in
>> this function)
>> session.c:566: error: (Each undeclared identifier is reported only once
>> session.c:566: error: for each function it appears in.)
>> session.c: In function 'decode_errmsg':
>> session.c:1216: error: 'NLA_F_NET_BYTEORDER' undeclared (first use in
>> this function)
>> session.c: In function 'attr_len':
>> session.c:1338: error: 'NLA_F_NET_BYTEORDER' undeclared (first use in
>> this function)
>
>Your kernel header files at the default location is not recent enough.
>
>I'm undecided yet how to solve it: maybe it should be checked by configure 
>and fail immediately.

Since this popped up in another thread: 2.6.24 is the minimal version
for this.

And somehow, it feels like it would suck to add extra #ifndef-#define-#endif
to all projects (currently that would already be libmnl and ipset5)
just for CentOS.
If one does their own kernel, surely it's just a command away
from `make headers_install`.

A few years ago, I had a similar, though more extreme, issue as the
developer of pam_mount, redhat shipped some kernel 2.6, but headers
for 2.4. Tell ya _that_ sucks.

>> # lsmod|grep set
>> ip_set                 16790  0 
>> nfnetlink               3179  2 ip_set,nf_conntrack_netlink
>> 
>> So, I guess something must have gone wrong when compiling ipset anyhow.
>
>Please read the README file: you must patch your kernel source with 
>netlink.patch, compile and install it. Otherwise the new nfnetlink id 
>won't handled by the kernel and thus ipset can't work.

ipset could try using both nfnetlink and genetlink, in case
NFNL_SUBSYS_IPSET is not defined, couldn't it?

Re: [ANNOUNCE] ipset-5.0 released

[Jozsef Kadlecsik]

On Sat, 18 Dec 2010, Jan Engelhardt wrote:

> On Saturday 2010-12-18 12:13, Jozsef Kadlecsik wrote:
> >On Sat, 18 Dec 2010, Rob Sterenborg (lists) wrote:
> >
> >> On Fri, 2010-12-17 at 23:26 +0100, Jozsef Kadlecsik wrote:
> >> > 
> >> > I'm happy to announce the new branch of ipset and release it's first 
> >> > element, ipset-5.0.
> >> 
> >> I'm not a C programmer. I just tried to make ipset compile which seems
> >> to have worked partially. I have no clue if I did the right thing so the
> >> below should be reviewed.
> >> 
> >> I'm on CentOS 5.5 with a custom 2.6.36.2 kernel, gcc version 4.1.2
> >> 20080704 (Red Hat 4.1.2-48).
> >> 
> >> When running 'configure' I got this error:
> >> 
> >> ./configure: line 11510: syntax error near unexpected token `[libmnl],'
> >> ./configure: line 11510: `PKG_CHECK_MODULES([libmnl], [libmnl >= 1])'
> >> 
> >> CentOS' pkg-config is installed, so, for reference: I copied
> >> '/usr/share/aclocal/pkg.m4' into the 'm4' directory, ran 'autogen.sh'
> >> again and after that 'configure' had no problems.
> >
> >Autoconf has its own pitfalls... I can't reproduce it so I added
> >'aclocal -I m4' to autogen.sh.
> 
> That is not necessary, because we already have ACLOCAL_AMFLAGS = -I m4
> in Makefile.am.

Yes, I know. Still I had a little hope that that'd fix the issue.
Do you have another idea? Should for example /usr/share/aclocal/pkg.m4 be 
unconditionally copied into m4/ by autogen.sh?
 
> >> Next, I got this:
> >> 
> >> session.c: In function 'attr2data':
> >> session.c:566: error: 'NLA_F_NET_BYTEORDER' undeclared (first use in
> >> this function)
> >> session.c:566: error: (Each undeclared identifier is reported only once
> >> session.c:566: error: for each function it appears in.)
> >> session.c: In function 'decode_errmsg':
> >> session.c:1216: error: 'NLA_F_NET_BYTEORDER' undeclared (first use in
> >> this function)
> >> session.c: In function 'attr_len':
> >> session.c:1338: error: 'NLA_F_NET_BYTEORDER' undeclared (first use in
> >> this function)
> >
> >Your kernel header files at the default location is not recent enough.
> >
> >I'm undecided yet how to solve it: maybe it should be checked by configure 
> >and fail immediately.
> 
> Since this popped up in another thread: 2.6.24 is the minimal version
> for this.
> 
> And somehow, it feels like it would suck to add extra 
> #ifndef-#define-#endif to all projects (currently that would already be 
> libmnl and ipset5) just for CentOS. If one does their own kernel, surely 
> it's just a command away from `make headers_install`.

Yes, exactly. I'm going to add checks to configure so it'll fail if the 
definitions are missing from the header file. Also, I'll add just the 
compiler warning flags which are supported by the actual compiler and thus
--disable-extra-flags can be eliminated.
 
> >Please read the README file: you must patch your kernel source with 
> >netlink.patch, compile and install it. Otherwise the new nfnetlink id 
> >won't handled by the kernel and thus ipset can't work.
> 
> ipset could try using both nfnetlink and genetlink, in case
> NFNL_SUBSYS_IPSET is not defined, couldn't it?

Yes, it could. It's unfortunate that old ipset modules could be compiled 
without any kernel patching while new one needs it. But we are on the 
right track: Dave accepted my jhash.h patch in net-next-2.6 and as it is 
propagated into Patrick's tree, I'll be able to submit the ipset 5.x 
kernel patches. But there's no need to rush: I hope I can get feedback 
from other architectures, like Sparc or ARM too. (On Wednesday I'll be 
able to dust off a Sparc machine and run the compilation and testsuite.)

Best regards,
Jozsef
-
E-mail  : kadlec@blackhole.kfki.hu, kadlec@mail.kfki.hu
PGP key : http://www.kfki.hu/~kadlec/pgp_public_key.txt
Address : KFKI Research Institute for Particle and Nuclear Physics
          H-1525 Budapest 114, POB. 49, Hungary

Re: [ANNOUNCE] ipset-5.0 released

[Jan Engelhardt]

On Saturday 2010-12-18 20:50, Jozsef Kadlecsik wrote:
>>>>
>>>> ./configure: line 11510: syntax error near unexpected token `[libmnl],'
>>>> ./configure: line 11510: `PKG_CHECK_MODULES([libmnl], [libmnl >= 1])'
>>>> 
>>>> CentOS' pkg-config is installed, so, for reference: I copied
>>>> '/usr/share/aclocal/pkg.m4' into the 'm4' directory, ran 'autogen.sh'
>>>> again and after that 'configure' had no problems.
>>>
>>>Autoconf has its own pitfalls... I can't reproduce it so I added
>>>'aclocal -I m4' to autogen.sh.
>> 
>>That is not necessary, because we already have ACLOCAL_AMFLAGS = -I m4
>>in Makefile.am.
>
>Yes, I know. Still I had a little hope that that'd fix the issue.
>Do you have another idea? Should for example /usr/share/aclocal/pkg.m4 be 
>unconditionally copied into m4/ by autogen.sh?

autoreconf normally has /usr/share/aclocal in its search path,
and I can say autoreconf does the job properly in CentOS 4.x.
Ima gonna setup a centos5 vm..

>> >Please read the README file: you must patch your kernel source with 
>> >netlink.patch, compile and install it. Otherwise the new nfnetlink id 
>> >won't handled by the kernel and thus ipset can't work.
>> 
>> ipset could try using both nfnetlink and genetlink, in case
>> NFNL_SUBSYS_IPSET is not defined, couldn't it?
>
>Yes, it could. It's unfortunate that old ipset modules could be compiled 
>without any kernel patching while new one needs it. But we are on the 
>right track: Dave accepted my jhash.h patch in net-next-2.6 and as it is 
>propagated into Patrick's tree, I'll be able to submit the ipset 5.x 
>kernel patches. But there's no need to rush: I hope I can get feedback 
>from other architectures, like Sparc or ARM too. (On Wednesday I'll be 
>able to dust off a Sparc machine and run the compilation and testsuite.)

I already made this year's presents: a PPC64 is here,
but it yet needs to be installed...

You can use my sparc. Pierre Chifflier already had his ulogd
pieces tested here in pair programming with shared screen(1)
sessions (great stuff that!).

== sparcv9

22:30 ares:~/code/ipset > ./configure libmnl_CFLAGS="-I$HOME/code/libmnl/include" libmnl_LIBS="-L$HOME/code/libmnl/obj32/src/.libs -Wl,-rpath,$HOME/code/libmnl/obj32/src/.libs -lmnl"
checking build system type... sparc64-suse-linux-gnu
checking host system type... sparc64-suse-linux-gnu
checking target system type... sparc64-suse-linux-gnu
checking for a BSD-compatible install... /usr/bin/install -c
checking whether build environment is sane... yes
checking for a thread-safe mkdir -p... /bin/mkdir -p
checking for gawk... gawk
checking whether make sets $(MAKE)... yes
checking for grep that handles long lines and -e... /usr/bin/grep
checking for gcc... gcc
checking whether the C compiler works... yes
checking for C compiler default output file name... a.out
checking for suffix of executables... 
checking whether we are cross compiling... no
checking for suffix of object files... o
checking whether we are using the GNU C compiler... yes
checking whether gcc accepts -g... yes
checking for gcc option to accept ISO C89... none needed
checking for style of include used by make... GNU
checking dependency style of gcc... gcc3
checking whether gcc and cc understand -c and -o together... yes
checking for a sed that does not truncate output... /usr/bin/sed
checking for egrep... /usr/bin/grep -E
checking for fgrep... /usr/bin/grep -F
checking for ld used by gcc... /usr/bin/ld
checking if the linker (/usr/bin/ld) is GNU ld... yes
checking for BSD- or MS-compatible name lister (nm)... /usr/bin/nm -B
checking the name lister (/usr/bin/nm -B) interface... BSD nm
checking whether ln -s works... yes
checking the maximum length of command line arguments... 1572864
checking whether the shell understands some XSI constructs... yes
checking whether the shell understands "+="... yes
checking for /usr/bin/ld option to reload object files... -r
checking for objdump... objdump
checking how to recognize dependent libraries... pass_all
checking for ar... ar
checking for strip... strip
checking for ranlib... ranlib
checking command to parse /usr/bin/nm -B output from gcc object... ok
checking how to run the C preprocessor... gcc -E
checking for ANSI C header files... yes
checking for sys/types.h... yes
checking for sys/stat.h... yes
checking for stdlib.h... yes
checking for string.h... yes
checking for memory.h... yes
checking for strings.h... yes
checking for inttypes.h... yes
checking for stdint.h... yes
checking for unistd.h... yes
checking for dlfcn.h... yes
checking for objdir... .libs
checking if gcc supports -fno-rtti -fno-exceptions... no
checking for gcc option to produce PIC... -fPIC -DPIC
checking if gcc PIC flag -fPIC -DPIC works... yes
checking if gcc static flag -static works... yes
checking if gcc supports -c -o file.o... yes
checking if gcc supports -c -o file.o... (cached) yes
checking whether the gcc linker (/usr/bin/ld -m elf32_sparc) supports shared libraries... yes
checking whether -lc should be explicitly linked in... no
checking dynamic linker characteristics... GNU/Linux ld.so
checking how to hardcode library paths into programs... immediate
checking whether stripping libraries is possible... yes
checking if libtool supports shared libraries... yes
checking whether to build shared libraries... yes
checking whether to build static libraries... yes
checking whether ln -s works... yes
checking for pkg-config... /usr/bin/pkg-config
checking pkg-config is at least version 0.9.0... yes
checking for libmnl... yes
checking for union nf_inet_addr... yes
configure: creating ./config.status
config.status: creating Makefile
config.status: creating lib/Makefile
config.status: creating src/Makefile
config.status: creating config.h
config.status: executing depfiles commands
config.status: executing libtool commands
22:31 ares:~/code/ipset > make -k
make  all-recursive
make[1]: Entering directory `/home/jengelh/code/ipset'
Making all in lib
make[2]: Entering directory `/home/jengelh/code/ipset/lib'
  CC     data.lo
  CC     icmp.lo
  CC     icmpv6.lo
  CC     mnl.lo
  CC     parse.lo
cc1: warnings being treated as errors
parse.c: In function ‘get_addrinfo’:
parse.c:637:11: error: cast increases required alignment of target type
parse.c:640:11: error: cast increases required alignment of target type
make[2]: *** [parse.lo] Error 1
  CC     print.lo
  CC     session.lo
  CC     types.lo
make[2]: Target `all' not remade because of errors.
make[2]: Leaving directory `/home/jengelh/code/ipset/lib'
Making all in src
make[2]: Entering directory `/home/jengelh/code/ipset/src'
  CC     ipset.o
  CC     errcode.o
  CC     ipset_bitmap_ip.o
  CC     ipset_bitmap_ipmac.o
  CC     ipset_bitmap_port.o
  CC     ipset_hash_ip.o
  CC     ipset_hash_ipport.o
  CC     ipset_hash_ipportip.o
  CC     ipset_hash_ipportnet.o
  CC     ipset_hash_net.o
  CC     ipset_hash_netport.o
  CC     ipset_list_set.o
  CC     ui.o
make[2]: *** No rule to make target `../lib/libipset.la', needed by `ipset'.
make[2]: Target `all' not remade because of errors.
make[2]: Leaving directory `/home/jengelh/code/ipset/src'
make[2]: Entering directory `/home/jengelh/code/ipset'
make[2]: Leaving directory `/home/jengelh/code/ipset'
make[1]: *** [all-recursive] Error 1
make[1]: Leaving directory `/home/jengelh/code/ipset'
make: *** [all] Error 2

22:43 ares:../ipset/kernel > make
make -C /lib/modules/`uname -r`/build M=`pwd` all
make[1]: Entering directory `/home/jengelh/code/linux'
  kernel: arch/sparc/boot/image is ready
make[1]: Leaving directory `/home/jengelh/code/linux'

That doesn't look right... hm.

The fix is:

-        $(MAKE) -C $(KERNELDIR) M=$$PWD $@
+        $(MAKE) -C $(KERNELDIR) M=$$PWD

One must not call make with 'all' as a target for modules. (And I am
surprised it happened to work on x86; normally it does not either.
Probably because on x86, I had a pure object dir, and on the sparc,
KERNELDIR refers to a source-object-dir)


=== sparc64

22:33 ares:~/code/ipset > ./configure libmnl_CFLAGS="-I$HOME/code/libmnl/include" libmnl_LIBS="-L$HOME/code/libmnl/obj64/src/.libs -Wl,-rpath,$HOME/code/libmnl/obj64/src/.libs -lmnl" CFLAGS="-m64 -O2" LDFLAGS=-m64
checking build system type... sparc64-suse-linux-gnu
checking host system type... sparc64-suse-linux-gnu
checking target system type... sparc64-suse-linux-gnu
checking for a BSD-compatible install... /usr/bin/install -c
checking whether build environment is sane... yes
checking for a thread-safe mkdir -p... /bin/mkdir -p
checking for gawk... gawk
checking whether make sets $(MAKE)... yes
checking for grep that handles long lines and -e... /usr/bin/grep
checking for gcc... gcc
checking whether the C compiler works... yes
checking for C compiler default output file name... a.out
checking for suffix of executables... 
checking whether we are cross compiling... no
checking for suffix of object files... o
checking whether we are using the GNU C compiler... yes
checking whether gcc accepts -g... yes
checking for gcc option to accept ISO C89... none needed
checking for style of include used by make... GNU
checking dependency style of gcc... gcc3
checking whether gcc and cc understand -c and -o together... yes
checking for a sed that does not truncate output... /usr/bin/sed
checking for egrep... /usr/bin/grep -E
checking for fgrep... /usr/bin/grep -F
checking for ld used by gcc... /usr/bin/ld
checking if the linker (/usr/bin/ld) is GNU ld... yes
checking for BSD- or MS-compatible name lister (nm)... /usr/bin/nm -B
checking the name lister (/usr/bin/nm -B) interface... BSD nm
checking whether ln -s works... yes
checking the maximum length of command line arguments... 1572864
checking whether the shell understands some XSI constructs... yes
checking whether the shell understands "+="... yes
checking for /usr/bin/ld option to reload object files... -r
checking for objdump... objdump
checking how to recognize dependent libraries... pass_all
checking for ar... ar
checking for strip... strip
checking for ranlib... ranlib
checking command to parse /usr/bin/nm -B output from gcc object... ok
checking how to run the C preprocessor... gcc -E
checking for ANSI C header files... yes
checking for sys/types.h... yes
checking for sys/stat.h... yes
checking for stdlib.h... yes
checking for string.h... yes
checking for memory.h... yes
checking for strings.h... yes
checking for inttypes.h... yes
checking for stdint.h... yes
checking for unistd.h... yes
checking for dlfcn.h... yes
checking for objdir... .libs
checking if gcc supports -fno-rtti -fno-exceptions... no
checking for gcc option to produce PIC... -fPIC -DPIC
checking if gcc PIC flag -fPIC -DPIC works... yes
checking if gcc static flag -static works... yes
checking if gcc supports -c -o file.o... yes
checking if gcc supports -c -o file.o... (cached) yes
checking whether the gcc linker (/usr/bin/ld -m elf64_sparc) supports shared libraries... yes
checking whether -lc should be explicitly linked in... no
checking dynamic linker characteristics... GNU/Linux ld.so
checking how to hardcode library paths into programs... immediate
checking whether stripping libraries is possible... yes
checking if libtool supports shared libraries... yes
checking whether to build shared libraries... yes
checking whether to build static libraries... yes
checking whether ln -s works... yes
checking for pkg-config... /usr/bin/pkg-config
checking pkg-config is at least version 0.9.0... yes
checking for libmnl... yes
checking for union nf_inet_addr... yes
configure: creating ./config.status
config.status: creating Makefile
config.status: creating lib/Makefile
config.status: creating src/Makefile
config.status: creating config.h
config.status: config.h is unchanged
config.status: executing depfiles commands
config.status: executing libtool commands
22:34 ares:~/code/ipset > make -k
make  all-recursive
make[1]: Entering directory `/home/jengelh/code/ipset'
Making all in lib
make[2]: Entering directory `/home/jengelh/code/ipset/lib'
  CC     data.lo
  CC     icmp.lo
  CC     icmpv6.lo
  CC     mnl.lo
  CC     parse.lo
cc1: warnings being treated as errors
parse.c: In function ‘get_addrinfo’:
parse.c:637:11: error: cast increases required alignment of target type
parse.c:640:11: error: cast increases required alignment of target type
make[2]: *** [parse.lo] Error 1
  CC     print.lo
  CC     session.lo
  CC     types.lo
make[2]: Target `all' not remade because of errors.
make[2]: Leaving directory `/home/jengelh/code/ipset/lib'
Making all in src
make[2]: Entering directory `/home/jengelh/code/ipset/src'
  CC     ipset.o
  CC     errcode.o
  CC     ipset_bitmap_ip.o
  CC     ipset_bitmap_ipmac.o
  CC     ipset_bitmap_port.o
  CC     ipset_hash_ip.o
  CC     ipset_hash_ipport.o
  CC     ipset_hash_ipportip.o
  CC     ipset_hash_ipportnet.o
  CC     ipset_hash_net.o
  CC     ipset_hash_netport.o
  CC     ipset_list_set.o
  CC     ui.o
make[2]: *** No rule to make target `../lib/libipset.la', needed by `ipset'.
make[2]: Target `all' not remade because of errors.
make[2]: Leaving directory `/home/jengelh/code/ipset/src'
make[2]: Entering directory `/home/jengelh/code/ipset'
make[2]: Leaving directory `/home/jengelh/code/ipset'
make[1]: *** [all-recursive] Error 1
make[1]: Leaving directory `/home/jengelh/code/ipset'
make: *** [all] Error 2
--
To unsubscribe from this list: send the line "unsubscribe netfilter-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [ANNOUNCE] ipset-5.0 released

[Jozsef Kadlecsik]

On Sat, 18 Dec 2010, Jan Engelhardt wrote:

> >from other architectures, like Sparc or ARM too. (On Wednesday I'll be 
> >able to dust off a Sparc machine and run the compilation and testsuite.)
> 
> I already made this year's presents: a PPC64 is here,
> but it yet needs to be installed...
> 
> You can use my sparc. Pierre Chifflier already had his ulogd
> pieces tested here in pair programming with shared screen(1)
> sessions (great stuff that!).

Let us see the fixes worked, see below
 
> == sparcv9
> 
> 22:31 ares:~/code/ipset > make -k
> make  all-recursive
> make[1]: Entering directory `/home/jengelh/code/ipset'
> Making all in lib
> make[2]: Entering directory `/home/jengelh/code/ipset/lib'
>   CC     data.lo
>   CC     icmp.lo
>   CC     icmpv6.lo
>   CC     mnl.lo
>   CC     parse.lo
> cc1: warnings being treated as errors
> parse.c: In function ?get_addrinfo?:
> parse.c:637:11: error: cast increases required alignment of target type
> parse.c:640:11: error: cast increases required alignment of target type

Oh well, the good old Sparc issue. I put back the workaround and pushed.

> 22:43 ares:../ipset/kernel > make

Nobody is supposed to run `make' in the kernel/ subdir. You should have 
typed one level above:

ares:../ipset/ > make modules

I have added a check to Makefile, to prevent it. 

Best regards,
Jozsef
-
E-mail  : kadlec@blackhole.kfki.hu, kadlec@mail.kfki.hu
PGP key : http://www.kfki.hu/~kadlec/pgp_public_key.txt
Address : KFKI Research Institute for Particle and Nuclear Physics
          H-1525 Budapest 114, POB. 49, Hungary

Re: [ANNOUNCE] ipset-5.0 released

[Jan Engelhardt]

On Sunday 2010-12-19 01:05, Jozsef Kadlecsik wrote:
>>   CC     parse.lo
>> cc1: warnings being treated as errors
>> parse.c: In function ?get_addrinfo?:
>> parse.c:637:11: error: cast increases required alignment of target type
>> parse.c:640:11: error: cast increases required alignment of target type
>
>Oh well, the good old Sparc issue. I put back the workaround and pushed.

Yeah. It seems struct sockaddr only has an alignment of 2 —
even on x86_64.

Given struct sockaddr is usually just a base of another sockaddr_XXX
object and is thus always has the same alignment of the latter,
we could avoid the memcpy:

const struct sockaddr_in *x = (void *)i->ai_addr;
err = ipset_session_data_set(session, opt, &x->sin_addr);


>> 22:43 ares:../ipset/kernel > make
>
>Nobody is supposed to run `make' in the kernel/ subdir. You should have 
>typed one level above:

Hah. So, what's the point of the Makefile? kbuild will find
Kbuild, so it uses just that.
--
To unsubscribe from this list: send the line "unsubscribe netfilter-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [ANNOUNCE] ipset-5.0 released

[Jan Engelhardt]

On Saturday 2010-12-18 22:49, Jan Engelhardt wrote:
>On Saturday 2010-12-18 20:50, Jozsef Kadlecsik wrote:
>>>>>
>>>>> ./configure: line 11510: syntax error near unexpected token `[libmnl],'
>>>>> ./configure: line 11510: `PKG_CHECK_MODULES([libmnl], [libmnl >= 1])'
>>>>> 
>>>>> CentOS' pkg-config is installed, so, for reference: I copied
>>>>> '/usr/share/aclocal/pkg.m4' into the 'm4' directory, ran 'autogen.sh'
>>>>> again and after that 'configure' had no problems.
>>>>
>>>>Autoconf has its own pitfalls... I can't reproduce it so I added
>>>>'aclocal -I m4' to autogen.sh.
>>> 
>>>That is not necessary, because we already have ACLOCAL_AMFLAGS = -I m4
>>>in Makefile.am.
>>
>>Yes, I know. Still I had a little hope that that'd fix the issue.
>>Do you have another idea? Should for example /usr/share/aclocal/pkg.m4 be 
>>unconditionally copied into m4/ by autogen.sh?
>
>autoreconf normally has /usr/share/aclocal in its search path,
>and I can say autoreconf does the job properly in CentOS 4.x.
>Ima gonna setup a centos5 vm..

I tried that now, and I see no problems relating to
PKG_CHECK_MODULES. Something looks flakey on the OP's machine.

What has been a problem in my test is that AC_PROG_GREP is not
known... I think I found a new height to RHEL obsolescence. RH5
(2007) ships autoconf 2.59 from Nov 2003.

Re: [ANNOUNCE] ipset-5.0 released

[Rob Sterenborg (lists)]

On Sat, 2010-12-18 at 12:13 +0100, Jozsef Kadlecsik wrote:
> Hi Rob,

[...]

> Your kernel header files at the default location is not recent enough.

That's entirely possible and I suppose it's the same pitfall I fell in
when compiling libmnl. :-/ I installed kernel 2.6.36.2 but didn't copy
it's header files into /usr/include, where ipset would pick them up.
Since I have no intention to go back to 2.6.18, I'm going to copy the
header files and try everything again to see what happens.

> I'm undecided yet how to solve it: maybe it should be checked by configure 
> and fail immediately.

I suppose you shouldn't do anything (yet?) because of the above.


Thanks,
Rob

Re: [ANNOUNCE] ipset-5.0 released

[Rob Sterenborg (lists)]

On Sat, 2010-12-18 at 12:13 +0100, Jozsef Kadlecsik wrote:

Hi Jozsef,

> > When running 'configure' I got this error:
> > 
> > ./configure: line 11510: syntax error near unexpected token `[libmnl],'
> > ./configure: line 11510: `PKG_CHECK_MODULES([libmnl], [libmnl >= 1])'
> > 
> > CentOS' pkg-config is installed, so, for reference: I copied
> > '/usr/share/aclocal/pkg.m4' into the 'm4' directory, ran 'autogen.sh'
> > again and after that 'configure' had no problems.
> 
> Autoconf has its own pitfalls... I can't reproduce it so I added
> 'aclocal -I m4' to autogen.sh. After checking out the git tree, could you 
> give it a try whether it solves the issue?

Call me stupid, but I couldn't figure out how to download the git branch
using git (when I did, I downloaded the 4.5 tree), but I did see it on
the web inferface. I downloaded today's snapshot from the git-5 branch;
I hope that's the same. I still get the same error however.

I've resolved the other issues I had: I should have thought of
installing the accompanying kernel header files.

> > Finally, when trying the new ipset it seems that except for 'version',
> > every command I tried returns 'Invalid argument':
> > 
> > (Yes I know this is incorrect syntax, but now I know it's trying to do
> > something besides giving me 'Invalid argument'.)
> > # ipset create TEST hash   
> > ipset v5.0: Syntax error: typename 'hash' is unkown
> > 
> > (As per ipset.8 example.)
> > # ipset create foo bitmap:ip range 192.168.0.0/16
> > ipset v5.0: Kernel error received: Invalid argument
> > 
> > # ipset list               
> > ipset v5.0: Kernel error received: Invalid argument
> > 
> > # lsmod|grep set
> > ip_set                 16790  0 
> > nfnetlink               3179  2 ip_set,nf_conntrack_netlink
> > 
> > So, I guess something must have gone wrong when compiling ipset anyhow.
> 
> Please read the README file: you must patch your kernel source with 
> netlink.patch, compile and install it. Otherwise the new nfnetlink id 
> won't handled by the kernel and thus ipset can't work.

I did this. The patch succeeded and compilation went, except for pkg.m4
which is easily resolved, without errors that I can see. I still get the
above results however.

> Thanks for tests and the reporting!

NP. Thank you..


Rob

Re: [ANNOUNCE] ipset-5.0 released

[Jozsef Kadlecsik]

Hi Rob,

On Tue, 21 Dec 2010, Rob Sterenborg (lists) wrote:

> On Sat, 2010-12-18 at 12:13 +0100, Jozsef Kadlecsik wrote:
> 
> > > When running 'configure' I got this error:
> > > 
> > > ./configure: line 11510: syntax error near unexpected token `[libmnl],'
> > > ./configure: line 11510: `PKG_CHECK_MODULES([libmnl], [libmnl >= 1])'
> > > 
> > > CentOS' pkg-config is installed, so, for reference: I copied
> > > '/usr/share/aclocal/pkg.m4' into the 'm4' directory, ran 'autogen.sh'
> > > again and after that 'configure' had no problems.
> > 
> > Autoconf has its own pitfalls... I can't reproduce it so I added
> > 'aclocal -I m4' to autogen.sh. After checking out the git tree, could you 
> > give it a try whether it solves the issue?
> 
> Call me stupid, but I couldn't figure out how to download the git branch
> using git (when I did, I downloaded the 4.5 tree), but I did see it on
> the web inferface. I downloaded today's snapshot from the git-5 branch;
> I hope that's the same. I still get the same error however.

Clone the git tree, then checkout the ipset-5 branch:

% git clone git://git.netfilter.org/ipset.git
% cd ipset
% git checkout ipset-5

> > > # ipset create TEST hash   
> > > ipset v5.0: Syntax error: typename 'hash' is unkown
> > > 
> > > (As per ipset.8 example.)
> > > # ipset create foo bitmap:ip range 192.168.0.0/16
> > > ipset v5.0: Kernel error received: Invalid argument
> > > 
> > > # ipset list               
> > > ipset v5.0: Kernel error received: Invalid argument
> > > 
> > > # lsmod|grep set
> > > ip_set                 16790  0 
> > > nfnetlink               3179  2 ip_set,nf_conntrack_netlink
> > > 
> > > So, I guess something must have gone wrong when compiling ipset anyhow.
> > 
> > Please read the README file: you must patch your kernel source with 
> > netlink.patch, compile and install it. Otherwise the new nfnetlink id 
> > won't handled by the kernel and thus ipset can't work.
> 
> I did this. The patch succeeded and compilation went, except for pkg.m4
> which is easily resolved, without errors that I can see. I still get the
> above results however.

I can think only that you still have the unpatched kernel binary running. 
Please note, netlink cannot be compiled as module, it's in the core 
kernel. So it's required to reboot using the patched kernel core.

Best regards,
Jozsef
-
E-mail  : kadlec@blackhole.kfki.hu, kadlec@mail.kfki.hu
PGP key : http://www.kfki.hu/~kadlec/pgp_public_key.txt
Address : KFKI Research Institute for Particle and Nuclear Physics
          H-1525 Budapest 114, POB. 49, Hungary

Re: [ANNOUNCE] ipset-5.0 released

[Mr Dash Four]

> I'm happy to announce the new branch of ipset and release it's first 
> element, ipset-5.0.
>   
I see that you have considered my suggestions and added port ranges to 
the hash sets. That will make my job much easier! Thank you!

Is there any difference between hash:net,ip and hash:ip,port? It seems 
as though I can specify subnets (CIDR format) of different sizes in both 
sets!

I also spotted another feature I previously missed when looked at 
5.0-pre10 - nesting of datatypes (I think the default is 4, which would 
be enough for 99% of cases). That is absolutely brilliant as up until 
now I have used multiple --match-set directives to do that job, which 
can now be done 'internally' by ipset. It also addresses the issue of 
'binding' (a feature dropped in earlier ipset releases and a feature I 
badly missed if I am being honest), but the implementation this time is 
much better. This set of features will be put to the test as I will be 
using them quite extensively!

I do have another question however: Currently the protocol part from the 
port ranges (hash sets) is not mandatory. Does that mean that if I omit 
it then the port range is matched *regardless* of the protocol (tcp or 
udp)? For example, if I have 10.1.1.0/24,80 would that match 
10.1.1.1:tcp:80 *and* 10.1.1.1:udp:80? If so, that is very good news!

I downloaded the source to look at, but won't compile it just yet as I 
am waiting for this version to be integrated in the xtables tree and 
hoping that integration is flawless and without the silly compile-time 
errors as was the case with previous xtables releases (*nudges Jan*).

As part of that process I will try and create the .spec file needed to 
build the Fedora rpm package (it would be for FC13 as I am yet to 
migrate to FC14) and will submit it with them to integrate it with FC as 
soon as possible.

Final question from me: As part of the ipset-5.0 package you provide a 
netlink patch file. I have read the README and it seems that the only 
time that patch needs to be applied is if the kernel version is >= 
2.6.31. Is that the case and are there any other 
constraints/requirements? Do I apply this patch if the kernel version is 
<= 2.6.31? It is important for me to know the answer to this question 
when I prepare the .spec file for building the rpm for Fedora.

Re: [ANNOUNCE] ipset-5.0 released

[Jozsef Kadlecsik]

On Sat, 18 Dec 2010, Mr Dash Four wrote:

> > I'm happy to announce the new branch of ipset and release it's first
> > element, ipset-5.0.
> >   
> I see that you have considered my suggestions and added port ranges to the
> hash sets. That will make my job much easier! Thank you!

That was not so hard, I hesitated just a little bit to add new features 
before the release.
 
> Is there any difference between hash:net,ip and hash:ip,port? It seems as
> though I can specify subnets (CIDR format) of different sizes in both sets!

You meant hash:net,port and hash:ip,port. Of course they are different. In 
hash:net,port you can store different sized netblock and port pairs, while 
in hash:ip,port you can store just IP address and port pairs. Let me show 
you an example:

# ipset create foo hash:net,port
# ipset add foo 192.168.0.0/30,80
# ipset list foo
Name: foo
Type: hash:net,port
Header: family inet hashsize 1024 maxelem 65536 
Size in memory: 16792
References: 0
Members:
192.168.0.0/30,tcp:80

# ipset create bar hash:ip,port
# ipset add bar 192.168.0.0/30,80
# ipset list bar
Name: bar
Type: hash:ip,port
Header: family inet hashsize 1024 maxelem 65536 
Size in memory: 16632
References: 0
Members:
192.168.0.0,tcp:80
192.168.0.1,tcp:80
192.168.0.2,tcp:80
192.168.0.3,tcp:80
 
> I also spotted another feature I previously missed when looked at 5.0-pre10 -
> nesting of datatypes (I think the default is 4, which would be enough for 99%
> of cases). That is absolutely brilliant as up until now I have used multiple
> --match-set directives to do that job, which can now be done 'internally' by
> ipset. It also addresses the issue of 'binding' (a feature dropped in earlier
> ipset releases and a feature I badly missed if I am being honest), but the
> implementation this time is much better. This set of features will be put to
> the test as I will be using them quite extensively!

Binding was unfortunately a bad ideas therefore had to be dropped. The 
multidimensional sets and the list of sets fairly well replace it.

> I do have another question however: Currently the protocol part from the port
> ranges (hash sets) is not mandatory. Does that mean that if I omit it then the
> port range is matched *regardless* of the protocol (tcp or udp)? For example,
> if I have 10.1.1.0/24,80 would that match 10.1.1.1:tcp:80 *and*
> 10.1.1.1:udp:80? If so, that is very good news!

No, that's just a shorthand notation. If the protocol is left out then TCP 
is assumed. If you need the same port with TCP and UDP, then you have to 
add both to the set. Just using the examle set bar above:

# ipset add bar 192.168.0.1,tcp:53
# ipset add bar 192.168.0.1,udp:53
# ipset list bar
Name: bar
Type: hash:ip,port
Header: family inet hashsize 1024 maxelem 65536 
Size in memory: 16696
References: 0
Members:
192.168.0.0,tcp:80
192.168.0.1,tcp:53
192.168.0.1,tcp:80
192.168.0.2,tcp:80
192.168.0.1,udp:53
192.168.0.3,tcp:80
 
> I downloaded the source to look at, but won't compile it just yet as I am
> waiting for this version to be integrated in the xtables tree and hoping that
> integration is flawless and without the silly compile-time errors as was the
> case with previous xtables releases (*nudges Jan*).

That won't be so easy for Jan, and it's up to him to decide, which ipset 
tree he wants to support in xtables-addons: the old one which does not 
need kernel patching or the new one with the burden of the netlink.patch.
 
> As part of that process I will try and create the .spec file needed to build
> the Fedora rpm package (it would be for FC13 as I am yet to migrate to FC14)
> and will submit it with them to integrate it with FC as soon as possible.
> 
> Final question from me: As part of the ipset-5.0 package you provide a netlink
> patch file. I have read the README and it seems that the only time that patch
> needs to be applied is if the kernel version is >= 2.6.31. Is that the case
> and are there any other constraints/requirements? Do I apply this patch if the
> kernel version is <= 2.6.31? It is important for me to know the answer to this
> question when I prepare the .spec file for building the rpm for Fedora.

In order to support kernel versions below 2.6.31, I had to add a lot of 
#ifdefs in xt_set.c to support the countless API changes in netfilter 
targets and matches. Hm, maybe I could support kernel releases from 
2.6.24. Below 2.6.24 there are missing netlink definitions as Jan 
mentioned.

Best regards,
Jozsef
-
E-mail  : kadlec@blackhole.kfki.hu, kadlec@mail.kfki.hu
PGP key : http://www.kfki.hu/~kadlec/pgp_public_key.txt
Address : KFKI Research Institute for Particle and Nuclear Physics
          H-1525 Budapest 114, POB. 49, Hungary

Re: [ANNOUNCE] ipset-5.0 released

[Mr Dash Four]

> # ipset create foo hash:net,port
> # ipset add foo 192.168.0.0/30,80
> # ipset list foo
> Name: foo
> Type: hash:net,port
> Header: family inet hashsize 1024 maxelem 65536 
> Size in memory: 16792
> References: 0
> Members:
> 192.168.0.0/30,tcp:80
>
> # ipset create bar hash:ip,port
> # ipset add bar 192.168.0.0/30,80
> # ipset list bar
> Name: bar
> Type: hash:ip,port
> Header: family inet hashsize 1024 maxelem 65536 
> Size in memory: 16632
> References: 0
> Members:
> 192.168.0.0,tcp:80
> 192.168.0.1,tcp:80
> 192.168.0.2,tcp:80
> 192.168.0.3,tcp:80
>   
I see!


>> I do have another question however: Currently the protocol part from the port
>> ranges (hash sets) is not mandatory. Does that mean that if I omit it then the
>> port range is matched *regardless* of the protocol (tcp or udp)? For example,
>> if I have 10.1.1.0/24,80 would that match 10.1.1.1:tcp:80 *and*
>> 10.1.1.1:udp:80? If so, that is very good news!
>>     
>
> No, that's just a shorthand notation. If the protocol is left out then TCP 
> is assumed. If you need the same port with TCP and UDP, then you have to 
> add both to the set. Just using the examle set bar above:
>
> # ipset add bar 192.168.0.1,tcp:53
> # ipset add bar 192.168.0.1,udp:53
> # ipset list bar
> Name: bar
> Type: hash:ip,port
> Header: family inet hashsize 1024 maxelem 65536 
> Size in memory: 16696
> References: 0
> Members:
> 192.168.0.0,tcp:80
> 192.168.0.1,tcp:53
> 192.168.0.1,tcp:80
> 192.168.0.2,tcp:80
> 192.168.0.1,udp:53
> 192.168.0.3,tcp:80
>   
Would it be possible to have 'something', which disregards the protocol 
on port matching?

By 'something' I mean either omission of the protocol, or 'all' to be 
specified instead of the protocol to mean no matching on protocol would 
be made (in other words the protocol to be disregarded). This will be 
especially useful for sets with quite a few number of members and will 
avoid unnecessary duplication - as things stand I have to add the same 
number of members for both tcp and udp protocols when I don't need any 
protocol matching - just the subnets and port numbers I specified. Is 
this doable?


>> I downloaded the source to look at, but won't compile it just yet as I am
>> waiting for this version to be integrated in the xtables tree and hoping that
>> integration is flawless and without the silly compile-time errors as was the
>> case with previous xtables releases (*nudges Jan*).
>>     
>
> That won't be so easy for Jan, and it's up to him to decide, which ipset 
> tree he wants to support in xtables-addons: the old one which does not 
> need kernel patching or the new one with the burden of the netlink.patch.
>   
For me it won't really make a difference as I am always building the 
kernel from source and apply quite a few (I think 11 at the last count) 
patches myself - adding one more won't make any difference. With most 
Fedora distributions the position is exactly the same - they supply one 
or two patches with xtables as well so it won't be something new.

>> Final question from me: As part of the ipset-5.0 package you provide a netlink
>> patch file. I have read the README and it seems that the only time that patch
>> needs to be applied is if the kernel version is >= 2.6.31. Is that the case
>> and are there any other constraints/requirements? Do I apply this patch if the
>> kernel version is <= 2.6.31? It is important for me to know the answer to this
>> question when I prepare the .spec file for building the rpm for Fedora.
>>     
>
> In order to support kernel versions below 2.6.31, I had to add a lot of 
> #ifdefs in xt_set.c to support the countless API changes in netfilter 
> targets and matches. Hm, maybe I could support kernel releases from 
> 2.6.24. Below 2.6.24 there are missing netlink definitions as Jan 
> mentioned.
>   
The way I see it, it is best to leave 4.x tree for versions up to 2.6.24 
and leave 5.x for newer versions and then decide whether the netlink 
patch should be applied (my understanding is that if the kernel version 
is >= 2.6.31 the patch definitely needs to be applied - is that right?).

Re: [ANNOUNCE] ipset-5.0 released

[Jan Engelhardt]

On Saturday 2010-12-18 22:51, Mr Dash Four wrote:
>> Members:
>> 192.168.0.0,tcp:80
>> 192.168.0.1,tcp:53
>> 192.168.0.1,tcp:80
>> 192.168.0.2,tcp:80
>> 192.168.0.1,udp:53
>> 192.168.0.3,tcp:80
>
>By 'something' I mean either omission of the protocol, or 'all' to
>be specified instead of the protocol to mean no matching on protocol
>would be made (in other words the protocol to be disregarded).

If you don't check the protocol, you cannot know if there even is
a port number. Not all L4 protocols have ports.

Re: [ANNOUNCE] ipset-5.0 released

[Mr Dash Four]

Jan Engelhardt wrote:
> On Saturday 2010-12-18 22:51, Mr Dash Four wrote:
>   
>>> Members:
>>> 192.168.0.0,tcp:80
>>> 192.168.0.1,tcp:53
>>> 192.168.0.1,tcp:80
>>> 192.168.0.2,tcp:80
>>> 192.168.0.1,udp:53
>>> 192.168.0.3,tcp:80
>>>       
>> By 'something' I mean either omission of the protocol, or 'all' to
>> be specified instead of the protocol to mean no matching on protocol
>> would be made (in other words the protocol to be disregarded).
>>     
>
> If you don't check the protocol, you cannot know if there even is
> a port number. Not all L4 protocols have ports.
>   
OK, let me rephrase that: I do not wish to add 2x times as many members 
in a particular set when I am not interested in the protocol match - 
whether it is tcp or udp for me is irrelevant, all I am interested in is 
the ip subnet and the port number.

Re: [ANNOUNCE] ipset-5.0 released

[Jozsef Kadlecsik]

On Sat, 18 Dec 2010, Mr Dash Four wrote:

> Would it be possible to have 'something', which disregards the protocol on
> port matching?
> 
> By 'something' I mean either omission of the protocol, or 'all' to be
> specified instead of the protocol to mean no matching on protocol would be
> made (in other words the protocol to be disregarded). This will be especially
> useful for sets with quite a few number of members and will avoid unnecessary
> duplication - as things stand I have to add the same number of members for
> both tcp and udp protocols when I don't need any protocol matching - just the
> subnets and port numbers I specified. Is this doable?

Use set types without port sub-part, like hash:net or hash:ip, etc.
I don't really see why you would want to use a type with port and then 
ignore it.

> > In order to support kernel versions below 2.6.31, I had to add a lot of
> > #ifdefs in xt_set.c to support the countless API changes in netfilter
> > targets and matches. Hm, maybe I could support kernel releases from 2.6.24.
> > Below 2.6.24 there are missing netlink definitions as Jan mentioned.
> >   
> The way I see it, it is best to leave 4.x tree for versions up to 2.6.24 and
> leave 5.x for newer versions and then decide whether the netlink patch should
> be applied (my understanding is that if the kernel version is >= 2.6.31 the
> patch definitely needs to be applied - is that right?).

The patch need to be applied below as well. And another version is 
required for kernels before the osf match was applied.

Best regards,
Jozsef
-
E-mail  : kadlec@blackhole.kfki.hu, kadlec@mail.kfki.hu
PGP key : http://www.kfki.hu/~kadlec/pgp_public_key.txt
Address : KFKI Research Institute for Particle and Nuclear Physics
          H-1525 Budapest 114, POB. 49, Hungary

Re: [ANNOUNCE] ipset-5.0 released

[Mr Dash Four]

>> By 'something' I mean either omission of the protocol, or 'all' to be
>> specified instead of the protocol to mean no matching on protocol would be
>> made (in other words the protocol to be disregarded). This will be especially
>> useful for sets with quite a few number of members and will avoid unnecessary
>> duplication - as things stand I have to add the same number of members for
>> both tcp and udp protocols when I don't need any protocol matching - just the
>> subnets and port numbers I specified. Is this doable?
>>     
>
> Use set types without port sub-part, like hash:net or hash:ip, etc.
> I don't really see why you would want to use a type with port and then 
> ignore it.
>   
I don't want to ignore the port - that stays (I need it to do the 
matching). I want to ignore the protocol, but keep the subnet and port 
number matches.

As I already mentioned, I see the need to register 2x as many members to 
a particular set just to get the match required (i.e. ignore the 
protocol) unnecessary when the alternative is to a) do not use protocol 
definition; or b) use another word (I suggested 'all') to ignore the 
protocol match and just use the subnet and port number(s) instead.

Wouldn't you agree that this is a better solution than registering twice 
as many members in a particular set in order to get the match I need?

Re: [ANNOUNCE] ipset-5.0 released

[Dennis Jacobfeuerborn]

On 12/19/2010 02:52 PM, Mr Dash Four wrote:
>
>>> By 'something' I mean either omission of the protocol, or 'all' to be
>>> specified instead of the protocol to mean no matching on protocol would be
>>> made (in other words the protocol to be disregarded). This will be
>>> especially
>>> useful for sets with quite a few number of members and will avoid
>>> unnecessary
>>> duplication - as things stand I have to add the same number of members for
>>> both tcp and udp protocols when I don't need any protocol matching -
>>> just the
>>> subnets and port numbers I specified. Is this doable?
>>
>> Use set types without port sub-part, like hash:net or hash:ip, etc.
>> I don't really see why you would want to use a type with port and then
>> ignore it.
> I don't want to ignore the port - that stays (I need it to do the
> matching). I want to ignore the protocol, but keep the subnet and port
> number matches.
>
> As I already mentioned, I see the need to register 2x as many members to a
> particular set just to get the match required (i.e. ignore the protocol)
> unnecessary when the alternative is to a) do not use protocol definition;
> or b) use another word (I suggested 'all') to ignore the protocol match and
> just use the subnet and port number(s) instead.
>
> Wouldn't you agree that this is a better solution than registering twice as
> many members in a particular set in order to get the match I need?

Given that ports are only meaningful in the context of a protocol I don't 
think that makes sense. If I want to block traffic to "port 80" in order to 
prevent access to a web server then what I'm rally saying is that I want 
top block traffic to "tcp port 80". There is no point in blocking udp port 
80 too. While there are some servers out there that use connections on the 
same port on both udp and tcp (e.g. DNS) these are the exception and should 
still be handled explicitly by defining rules for both protocols separately.
Why would you want to match a port for all protocols? Do you have a 
specific example for that?

Regards,
   Dennis

Re: [ANNOUNCE] ipset-5.0 released

[Mr Dash Four]

>>
>> Wouldn't you agree that this is a better solution than registering 
>> twice as
>> many members in a particular set in order to get the match I need?
>
> Given that ports are only meaningful in the context of a protocol I 
> don't think that makes sense.
You are mistaken! I've never actually stated, nor implied, that 
protocols are 'meaningless' when ports are specified or used - I stated 
that I am not interested in *matching* the protocol as for me this part 
of the match is irrelevant - there are plenty of examples where a 
particular service can employ either tcp or udp protocol on the same 
port - see below for examples.

> If I want to block traffic to "port 80" in order to prevent access to 
> a web server then what I'm rally saying is that I want top block 
> traffic to "tcp port 80". There is no point in blocking udp port 80 too.
True, but that is an isolated case when I want to block a traffic to 1) 
a particular well-known service (web/http), which uses 2) a well-defined 
port (80 in this case) and 3) a well-defined protocol (tcp in this case).

That, as I already mentioned is an isolated case and defining the 
protocol and port in the ipset makes perfect sense. I would like to be 
able to define a match which captures subnets and port numbers 
regardless of the protocol involved (udp or tcp) - I thought I made that 
perfectly clear.

> While there are some servers out there that use connections on the 
> same port on both udp and tcp (e.g. DNS) these are the exception and 
> should still be handled explicitly by defining rules for both 
> protocols separately.
I disagree! Why should I have to define 2x as many members in a 
particular set in order to cover tcp and udp protocols where, in 
reality, I have no interest whatsoever in matching the protocol element 
at all - all I need is a subnet match and a port number, the protocol 
part (whatever that is - udp or tcp) is completely irrelevant - I do not 
really care whether the protocol is tcp or udp.

> Why would you want to match a port for all protocols? Do you have a 
> specific example for that?
You already mentioned DNS, I'd add DHCP to that list as well as VPN, 
IPSec and Kerberos traffic, which could be either tcp or udp on the same 
port numbers. VOIP (including STUN, RTP and SIP) is another example - 
not only the port numbers vary wildly within (usually) pre-defined 
group, but could utilise tcp as well as the udp protocol on these ports. 
IRC and media streaming too, not to mention the (notoriously bad) 
Microsoft NetBIOS (ports 135, 137, 138 and 139), WINS or even LDAP. 
Various databases (MySQL, PostgreSQL and Oracle to name a few) also 
utilise both protocols on the same port numbers.

Using tcp as well as udp traffic on the same port numbers is not as 
uncommon as you might think, hence why I would like to have subnet and 
port matching regardless of what the protocol is.

Re: [ANNOUNCE] ipset-5.0 released

[Jozsef Kadlecsik]

On Sun, 19 Dec 2010, Mr Dash Four wrote:

> > > Wouldn't you agree that this is a better solution than registering 
> > > twice as many members in a particular set in order to get the match 
> > > I need?
> > 
> > Given that ports are only meaningful in the context of a protocol I don't
> > think that makes sense.
> You are mistaken! I've never actually stated, nor implied, that protocols are
> 'meaningless' when ports are specified or used - I stated that I am not
> interested in *matching* the protocol as for me this part of the match is
> irrelevant - there are plenty of examples where a particular service can
> employ either tcp or udp protocol on the same port - see below for examples.

I could not come up with a good solution. It is not possible to avoid
double lookup in the set if the elements may have (say hash:ip,port type 
of set) the forms:

ipaddr,tcp:port
ipaddr,udp:port
ipadd,tcpudp:port

But if the double-lookup is required, then better add all tcp and udp port 
variants and go with a single-lookup.

Or if you have uniform ports for all elements, you can use two set 
matches:

ipset create hash:ip ipaddrs
ipset create bitmap:port ports

... -m set --match-set ipaddrs dst -m set --match-set ports dst ...

Best regards,
Jozsef
-
E-mail  : kadlec@blackhole.kfki.hu, kadlec@mail.kfki.hu
PGP key : http://www.kfki.hu/~kadlec/pgp_public_key.txt
Address : KFKI Research Institute for Particle and Nuclear Physics
          H-1525 Budapest 114, POB. 49, Hungary

Re: [ANNOUNCE] ipset-5.0 released

[Mr Dash Four]

> I could not come up with a good solution. It is not possible to avoid
> double lookup in the set if the elements may have (say hash:ip,port type 
> of set) the forms:
>
> ipaddr,tcp:port
> ipaddr,udp:port
> ipadd,tcpudp:port
>   
I was thinking that you may interpret either the absence of the protocol 
specification ('tcp', 'udp' and 'tcpudp' in your examples above) or the 
word 'all' (even '*' though I don't know if that's possible) to mean 
that no protocol match is necessary.

> But if the double-lookup is required, then better add all tcp and udp port 
> variants and go with a single-lookup.
>   
I am not sure I understand you.

> Or if you have uniform ports for all elements, you can use two set 
> matches:
>
> ipset create hash:ip ipaddrs
> ipset create bitmap:port ports
>
> ... -m set --match-set ipaddrs dst -m set --match-set ports dst ...
>   
There is a problem with this - the 'bond' which exists with (the 
imaginary and working) hash:ip,all,port (i.e. the implicit link between 
the element's ip address/subnet and the port or port range)  is not 
there if two separate sets are used!

In order to create this 'bond' I have to use not one, but two separate 
sets (i.e. vpn_hosts/vpn_ports) and even then I cannot control this fully.

One very simple example: if I have a vpn host group which is on the 
10.1.1.0/24 subnet and operates within the 1194 (tcp/udp) port range and 
then have another vpn host group 10.1.2.0/24 which operates on a 
different port range - 80,443 (tcp and udp).

If I follow your example above I have to define *four* sets to 
accommodate the double matches, because if I put all the vpn ports into 
one ipset group (say vpn-ports) which contains ports 1194,80 and 443 and 
then use a similar ipset for the vpn hosts (say vpn-hosts) containing 
10.1.1.0/24 and 10.1.2.0/24 as its members and then do a double match 
"-m set --match-set vpn-hosts dst -m set --match-set vpn-ports dst" that 
would mean packets destined to 10.1.1.0/24 on ports 80,443 (tcp/udp) and 
10.1.2.0/24:1194 (tcp,udp) will be allowed which is not what I want.

If I want to separate this fully and make it work as it should, I have 
to define *four* sets (vpn-hosts1->10.1.1.0/24, vpn-ports1->1194, then 
vpn-hosts2->10.1.2.0/24,vpn-ports2->80,443), which makes the whole job 
rather cumbersome. I currently have this problem when I employ a similar 
approach with the existing 4.x version of ipset.

If ipset 5.x can disregard the protocol match I will have to define a 
single set and place just 3 members in it: 10.1.1.0/24,all,1194; 
10.1.2.0/24,all,80 and 10.1.2.0/24,all,443

Re: [ANNOUNCE] ipset-5.0 released

[Jozsef Kadlecsik]

On Wed, 22 Dec 2010, Mr Dash Four wrote:

> > I could not come up with a good solution. It is not possible to avoid
> > double lookup in the set if the elements may have (say hash:ip,port type of
> > set) the forms:
> > 
> > ipaddr,tcp:port
> > ipaddr,udp:port
> > ipadd,tcpudp:port
> >   
> I was thinking that you may interpret either the absence of the protocol
> specification ('tcp', 'udp' and 'tcpudp' in your examples above) or the word
> 'all' (even '*' though I don't know if that's possible) to mean that no
> protocol match is necessary.

The algorithm does not allow to ignore the protocol part. That's the 
reason why double lookup would be needed if such syntactic sugar as 
'tcpudp' were introduced. Lets assume we store in the artifical set above 
two elements:

192.168.0.1,tcpudp:53
192.168.0.1,tcp:80

Now, assuming that first 'tcpudp' is looked up, to match the first 
element, we need one lookup. To check the existence of the second element, 
we have to use two lookups: first with '192.168.0.1,tcpudp:80' (nomatch),
then with '192.168.0.1,tcp:80'. And to check the existence of any 
non-matching element (say 192.168.0.1,udp:123) needs again two lookups.

Such a hack thus would penalize the normal cases.

> > But if the double-lookup is required, then better add all tcp and udp port
> > variants and go with a single-lookup.
> >   
> I am not sure I understand you.

If you need to match the same port both with TCP and UDP, then add it 
to the set twice, with the proper protocols.
 
> > Or if you have uniform ports for all elements, you can use two set matches:
> > 
> > ipset create hash:ip ipaddrs
> > ipset create bitmap:port ports
> > 
> > ... -m set --match-set ipaddrs dst -m set --match-set ports dst ...
> >   
> There is a problem with this - the 'bond' which exists with (the imaginary and
> working) hash:ip,all,port (i.e. the implicit link between the element's ip
> address/subnet and the port or port range)  is not there if two separate sets
> are used!
> 
> In order to create this 'bond' I have to use not one, but two separate sets
> (i.e. vpn_hosts/vpn_ports) and even then I cannot control this fully.
> 
> One very simple example: if I have a vpn host group which is on the
> 10.1.1.0/24 subnet and operates within the 1194 (tcp/udp) port range and then
> have another vpn host group 10.1.2.0/24 which operates on a different port
> range - 80,443 (tcp and udp).
> 
> If I follow your example above I have to define *four* sets to accommodate the
> double matches, because if I put all the vpn ports into one ipset group (say
> vpn-ports) which contains ports 1194,80 and 443 and then use a similar ipset
> for the vpn hosts (say vpn-hosts) containing 10.1.1.0/24 and 10.1.2.0/24 as
> its members and then do a double match "-m set --match-set vpn-hosts dst -m
> set --match-set vpn-ports dst" that would mean packets destined to 10.1.1.0/24
> on ports 80,443 (tcp/udp) and 10.1.2.0/24:1194 (tcp,udp) will be allowed which
> is not what I want.
> 
> If I want to separate this fully and make it work as it should, I have to
> define *four* sets (vpn-hosts1->10.1.1.0/24, vpn-ports1->1194, then
> vpn-hosts2->10.1.2.0/24,vpn-ports2->80,443), which makes the whole job rather
> cumbersome.

Yes, therfore my assumption was that the ports are uniform, identical for 
all elements.

You wrote earlier:

> > Why would you want to match a port for all protocols? Do you have a
> > specific example for that?

> You already mentioned DNS, I'd add DHCP to that list as well as VPN, 
> IPSec and Kerberos traffic, which could be either tcp or udp on the same 
> port numbers. 

DHCP uses UDP only. VPN (e.g OpenVPN) can be configured to use UDP or TCP, 
but not both at the same time. For ipset point of view, IPSec is an 
independent (actually, two, AH and ESP) protocol. Kerberos uses a couple 
of TCP and UDP ports, but not both with the same port number (except for 
changing password under Windows, if you have to support both the old and 
new interfaces).

> VOIP (including STUN, RTP and SIP) is another example - not only the 
> port numbers vary wildly within (usually) pre-defined group, but could 
> utilise tcp as well as the udp protocol on these ports. IRC

STUN uses a single port over TCP and UDP. RTP may run over TCP but the 
majority of the applications uses UDP only. SIP is complex... IRC uses TCP 
only.

> and media streaming too, not to mention the (notoriously bad) Microsoft 
> NetBIOS (ports 135, 137, 138 and 139), WINS

>  or even LDAP. Various databases (MySQL, PostgreSQL and Oracle to name a 
> few) also utilise both protocols on the same port numbers.

LDAP and *SQL use TCP only.

Best regards,
Jozsef
-
E-mail  : kadlec@blackhole.kfki.hu, kadlec@mail.kfki.hu
PGP key : http://www.kfki.hu/~kadlec/pgp_public_key.txt
Address : KFKI Research Institute for Particle and Nuclear Physics
          H-1525 Budapest 114, POB. 49, Hungary

Re: [ANNOUNCE] ipset-5.0 released

[Mr Dash Four]

> The algorithm does not allow to ignore the protocol part. That's the 
> reason why double lookup would be needed if such syntactic sugar as 
> 'tcpudp' were introduced. Lets assume we store in the artifical set above 
> two elements:
>
> 192.168.0.1,tcpudp:53
> 192.168.0.1,tcp:80
>
> Now, assuming that first 'tcpudp' is looked up, to match the first 
> element, we need one lookup. To check the existence of the second element, 
> we have to use two lookups: first with '192.168.0.1,tcpudp:80' (nomatch),
> then with '192.168.0.1,tcp:80'. And to check the existence of any 
> non-matching element (say 192.168.0.1,udp:123) needs again two lookups.
>   
Then your matching algorithm is wrong!

Using your example above, I'd add the following change: the protocol 
specifier could be converted (internally) to a number: 1 - tcp (1 in 
binary form), 2 - udp (10 in binary form), 3 - 'tcp and udp' (11 in 
binary form). When you need to match against a real protocol (which, if 
I am not mistaken, can only be either tcp or udp), then you can use 
binary 'and' operation (&) with one simple statement to determine 
whether there is a match. Using your example above, we will have the 
following (internal) representation:

192.168.0.1,3,53
192.168.0.1,1,80

Thus, to check for 192.168.0.1:53 (tcp) - there will be a match as when 
you apply 1) ip match; and 2) protocol match - i.e. 1 & 3 (1 & 11) you 
will get >0, therefore protocol matches; and 3) port match, there is a 
match on all 3 elements, hence return a complete match. Just one lookup 
needed.

When you have 192.168.0.1:53 (udp) the situation is exactly the same as 
far as the protocol goes - 2 & 3 (10 & 11) you will get > 0, hence there 
is a match again. Just one lookup needed.

When you have 192.168.0.1:80 (tcp) - you will have a match on the ip and 
protocol, bit mismatch on the port, so no match - you also need one lookup.

You have exactly the same scenario with 192.168.0.1:123 (udp) - match on 
the ip and protocol part, but mismatch on the port, so no match!

> Such a hack thus would penalize the normal cases.
>   
There is no need for a 'hack' just sensible and efficient programming, 
that's all.

>   
>   
>> I am not sure I understand you.
>>     
>
> If you need to match the same port both with TCP and UDP, then add it 
> to the set twice, with the proper protocols.
>   
I've already dealt with this - I do not see the need to add 2x as many 
elements to a set when, in reality, I am not interested in matching the 
protocol part.

  

>> There is a problem with this - the 'bond' which exists with (the imaginary and
>> working) hash:ip,all,port (i.e. the implicit link between the element's ip
>> address/subnet and the port or port range)  is not there if two separate sets
>> are used!
>>
>> [...]
>>     
>
> Yes, therfore my assumption was that the ports are uniform, identical for 
> all elements.
>   
An isolated case and one which does not apply to a common scenario.

>> You already mentioned DNS, I'd add DHCP to that list as well as VPN, 
>> IPSec and Kerberos traffic, which could be either tcp or udp on the same 
>> port numbers. 
>>     
>
> DHCP uses UDP only.
You are mistaken - DHCPv6 (client as well as server configurations) uses 
both TCP and UDP protocols on the same port numbers.


>  VPN (e.g OpenVPN) can be configured to use UDP or TCP, 
> but not both at the same time.
You are assuming that I will have a single machine on which VPN is 
running, which, again, is an isolated case. When I have a central hub, 
which controls multiple subnets VPN packets can originate from (or be 
destined to) the same port numbers using TCP as well as UDP protocols.

>  For ipset point of view, IPSec is an 
> independent (actually, two, AH and ESP) protocol. Kerberos uses a couple 
> of TCP and UDP ports, but not both with the same port number (except for 
> changing password under Windows, if you have to support both the old and 
> new interfaces).
>   
Again, not true and you are assuming I am running a single machine - 
that is not the case at all.

Kerberos remote login, change password (except klogin which uses tcp 
only), administration and kreg can all be ran using either tcp or udp on 
the same port numbers. As for IPSec - the situation is exactly the same 
- port 1293 with either tcp or udp utilised.

>> VOIP (including STUN, RTP and SIP) is another example - not only the 
>> port numbers vary wildly within (usually) pre-defined group, but could 
>> utilise tcp as well as the udp protocol on these ports. IRC
>>     
>
> STUN uses a single port over TCP and UDP.
Yep, that's right - both protocols could be used on the same port.

>  RTP may run over TCP but the 
> majority of the applications uses UDP only.
Again, utilising TCP is not as uncommon as you might think - TCP gives 
you better line quality when you have good (optical for example) network 
and it is also the preferred option when you utilise encrypted data 
streams. Same goes for SIP.

>  SIP is complex... IRC uses TCP 
> only.
>   
IRC on Port 194, protocols could be either TCP or UDP, again, depending 
on the configuration. AOL instant messenger also utilises both tcp and 
udp protocols on the same port (531).


>>  or even LDAP. Various databases (MySQL, PostgreSQL and Oracle to name a 
>> few) also utilise both protocols on the same port numbers.
>>     
>
> LDAP and *SQL use TCP only.
>   
Wrong again! Check the documentation for the databases I mentioned 
properly and you will see that MySQL could use 3306 utilising both tcp 
and udp protocols, PostgreSQL can also utilise both protocols on its 
common port (5432) and with the case of Oracle - although the commonly 
used Oracle port is 1521 it also utilises ports 2483 (for insecure) and 
2484 (for secure) client connections replacing the 'old' 1521 port for 
insecure client connections.

Re: [ANNOUNCE] ipset-5.0 released

[David Miller]

From: Mr Dash Four <mr.dash.four@googlemail.com>
Date: Thu, 23 Dec 2010 17:50:58 +0000

>> If you need to match the same port both with TCP and UDP, then add it
>> to the set twice, with the proper protocols.
>>   
> I've already dealt with this - I do not see the need to add 2x as many
> elements to a set when, in reality, I am not interested in matching
> the protocol part.

You must, every protocol puts the ports in a different location.

We have the same issue in our flow hashing functions in the kernel.

Re: [ANNOUNCE] ipset-5.0 released

[Mr Dash Four]

>>> If you need to match the same port both with TCP and UDP, then add it
>>> to the set twice, with the proper protocols.
>>>   
>>>       
>> I've already dealt with this - I do not see the need to add 2x as many
>> elements to a set when, in reality, I am not interested in matching
>> the protocol part.
>>     
>
> You must, every protocol puts the ports in a different location.
>   
What do you mean by 'puts the ports in a different location'? Clarify 
please.

Re: [ANNOUNCE] ipset-5.0 released

[David Miller]

From: Mr Dash Four <mr.dash.four@googlemail.com>
Date: Thu, 23 Dec 2010 18:00:38 +0000

> 
>>>> If you need to match the same port both with TCP and UDP, then add it
>>>> to the set twice, with the proper protocols.
>>>>         
>>> I've already dealt with this - I do not see the need to add 2x as many
>>> elements to a set when, in reality, I am not interested in matching
>>> the protocol part.
>>>     
>>
>> You must, every protocol puts the ports in a different location.
>>   
> What do you mean by 'puts the ports in a different location'? Clarify
> please.

Look at the proto_ports_offset() function in the kernel if you don't
believe me.

static inline int proto_ports_offset(int proto)
{
	switch (proto) {
	case IPPROTO_TCP:
	case IPPROTO_UDP:
	case IPPROTO_DCCP:
	case IPPROTO_ESP:	/* SPI */
	case IPPROTO_SCTP:
	case IPPROTO_UDPLITE:
		return 0;
	case IPPROTO_AH:	/* SPI */
		return 4;
	default:
		return -EINVAL;
	}
}

Re: [ANNOUNCE] ipset-5.0 released

[Mr Dash Four]

>> What do you mean by 'puts the ports in a different location'? Clarify
>> please.
>>     
>
> Look at the proto_ports_offset() function in the kernel if you don't
> believe me.
>   
I've never stated that I do not believe you - I just asked for a 
clarification. After looking at the example below I am still unclear 
what your point is (call me daft if you like!).

> static inline int proto_ports_offset(int proto)
> {
> 	switch (proto) {
> 	case IPPROTO_TCP:
> 	case IPPROTO_UDP:
> 	case IPPROTO_DCCP:
> 	case IPPROTO_ESP:	/* SPI */
> 	case IPPROTO_SCTP:
> 	case IPPROTO_UDPLITE:
> 		return 0;
> 	case IPPROTO_AH:	/* SPI */
> 		return 4;
> 	default:
> 		return -EINVAL;
> 	}
> }
>   

Re: [ANNOUNCE] ipset-5.0 released

[Jozsef Kadlecsik]

On Thu, 23 Dec 2010, Mr Dash Four wrote:

> > The algorithm does not allow to ignore the protocol part. That's the reason
> > why double lookup would be needed if such syntactic sugar as 'tcpudp' were
> > introduced. Lets assume we store in the artifical set above two elements:
> > 
> > 192.168.0.1,tcpudp:53
> > 192.168.0.1,tcp:80
> > 
> > Now, assuming that first 'tcpudp' is looked up, to match the first element,
> > we need one lookup. To check the existence of the second element, we have to
> > use two lookups: first with '192.168.0.1,tcpudp:80' (nomatch),
> > then with '192.168.0.1,tcp:80'. And to check the existence of any
> > non-matching element (say 192.168.0.1,udp:123) needs again two lookups.
> >   
> Then your matching algorithm is wrong!

You are free to design and implement a better one.
 
> Using your example above, I'd add the following change: the protocol specifier
> could be converted (internally) to a number: 1 - tcp (1 in binary form), 2 -
> udp (10 in binary form), 3 - 'tcp and udp' (11 in binary form). When you need
> to match against a real protocol (which, if I am not mistaken, can only be
> either tcp or udp), then you can use binary 'and' operation (&) with one
> simple statement to determine whether there is a match. Using your example
> above, we will have the following (internal) representation:
> 
> 192.168.0.1,3,53
> 192.168.0.1,1,80
> 
> Thus, to check for 192.168.0.1:53 (tcp) - there will be a match as when you
> apply 1) ip match; and 2) protocol match - i.e. 1 & 3 (1 & 11) you will get
> >0, therefore protocol matches; and 3) port match, there is a match on all 3
> elements, hence return a complete match. Just one lookup needed.

Yes, assuming that first you lookup the IP address, then lookup the 
protocoll and then the port. That's three successive lookups, not one.

The implementation behind ipset looks up the  (ipaddr, proto, port) triple 
in one step. Such packing don't work there.
 
> > > You already mentioned DNS, I'd add DHCP to that list as well as VPN, IPSec
> > > and Kerberos traffic, which could be either tcp or udp on the same port
> > > numbers.     
> > 
> > DHCP uses UDP only.
> You are mistaken - DHCPv6 (client as well as server configurations) uses both
> TCP and UDP protocols on the same port numbers.

I run dibbler DHCPv6 server. It listens on UDP with IPv6 only:

# lsof -i6|grep dibbler
dibbler-s 18060   root    5u  IPv6 13581079       UDP [ff02::1:2]:547 
dibbler-s 18060   root    6u  IPv6 13581082       UDP [ff02::1:2]:547 

The clients (Linux, Windows) work flawlessly without TCP.

> >  VPN (e.g OpenVPN) can be configured to use UDP or TCP, but not both at the
> > same time.
> You are assuming that I will have a single machine on which VPN is running,
> which, again, is an isolated case. When I have a central hub, which controls
> multiple subnets VPN packets can originate from (or be destined to) the same
> port numbers using TCP as well as UDP protocols.
> 
> >  For ipset point of view, IPSec is an independent (actually, two, AH and
> > ESP) protocol. Kerberos uses a couple of TCP and UDP ports, but not both
> > with the same port number (except for changing password under Windows, if
> > you have to support both the old and new interfaces).
> >   
> Again, not true and you are assuming I am running a single machine - that is
> not the case at all.
> 
> Kerberos remote login, change password (except klogin which uses tcp only),
> administration and kreg can all be ran using either tcp or udp on the same
> port numbers. As for IPSec - the situation is exactly the same - port 1293
> with either tcp or udp utilised.

I do not assume you run a single server. The point is, you know the IP 
addresses, protocols, ports. Had you iptables rules you had to enter so 
many rules as the addresses, protocols, ports dictate.
 
> > > VOIP (including STUN, RTP and SIP) is another example - not only the port
> > > numbers vary wildly within (usually) pre-defined group, but could utilise
> > > tcp as well as the udp protocol on these ports. IRC
> > >     
> > 
> > STUN uses a single port over TCP and UDP.
> Yep, that's right - both protocols could be used on the same port.
> 
> >  RTP may run over TCP but the majority of the applications uses UDP only.
> Again, utilising TCP is not as uncommon as you might think - TCP gives you
> better line quality when you have good (optical for example) network and it is
> also the preferred option when you utilise encrypted data streams. Same goes
> for SIP.
> 
> >  SIP is complex... IRC uses TCP only.
> >   
> IRC on Port 194, protocols could be either TCP or UDP, again, depending on the
> configuration. AOL instant messenger also utilises both tcp and udp protocols
> on the same port (531). 
> 
> > >  or even LDAP. Various databases (MySQL, PostgreSQL and Oracle to name a
> > > few) also utilise both protocols on the same port numbers.
> > >     
> > 
> > LDAP and *SQL use TCP only.
> >   
> Wrong again! Check the documentation for the databases I mentioned properly
> and you will see that MySQL could use 3306 utilising both tcp and udp
> protocols, PostgreSQL can also utilise both protocols on its common port
> (5432) and with the case of Oracle - although the commonly used Oracle port is
> 1521 it also utilises ports 2483 (for insecure) and 2484 (for secure) client
> connections replacing the 'old' 1521 port for insecure client connections.

Our mySQL servers seem to listen on TCP only. Out of curiosity I checked 
the manuals and it states TCP only for listening on.

Best regards,
Jozsef
-
E-mail  : kadlec@blackhole.kfki.hu, kadlec@mail.kfki.hu
PGP key : http://www.kfki.hu/~kadlec/pgp_public_key.txt
Address : KFKI Research Institute for Particle and Nuclear Physics
          H-1525 Budapest 114, POB. 49, Hungary

Re: [ANNOUNCE] ipset-5.0 released

[Mr Dash Four]

> The implementation behind ipset looks up the  (ipaddr, proto, port) triple 
> in one step. Such packing don't work there.
>   
If that's the case how do you lookup IP address and port ranges then?

Re: [ANNOUNCE] ipset-5.0 released

[Jozsef Kadlecsik]

On Thu, 23 Dec 2010, Mr Dash Four wrote:

> > The implementation behind ipset looks up the  (ipaddr, proto, port) triple
> > in one step. Such packing don't work there.
> >   
> If that's the case how do you lookup IP address and port ranges then?

IP address and port ranges are exploded and the elements are inserted 
one-by-one. And the exploded ranges are *not* converted back to ranges 
when listing/saving the sets. At the bitmap types the ranges could be 
converted back (not done yet), at the hash types it's not possible.

Best regards,
Jozsef
-
E-mail  : kadlec@blackhole.kfki.hu, kadlec@mail.kfki.hu
PGP key : http://www.kfki.hu/~kadlec/pgp_public_key.txt
Address : KFKI Research Institute for Particle and Nuclear Physics
          H-1525 Budapest 114, POB. 49, Hungary

Re: [ANNOUNCE] ipset-5.0 released

[Jozsef Kadlecsik]

On Thu, 23 Dec 2010, Jozsef Kadlecsik wrote:

> On Thu, 23 Dec 2010, Mr Dash Four wrote:
> 
> > > The implementation behind ipset looks up the  (ipaddr, proto, port) triple
> > > in one step. Such packing don't work there.
> > >   
> > If that's the case how do you lookup IP address and port ranges then?
> 
> IP address and port ranges are exploded and the elements are inserted 
> one-by-one. And the exploded ranges are *not* converted back to ranges 
> when listing/saving the sets. At the bitmap types the ranges could be 
> converted back (not done yet), at the hash types it's not possible.

Just to illustrate:

# ipset create test hash:ip,port                          
# ipset add test 192.168.0.0/30,tcp:80-82                 
# ipset list test                                         
Name: test                                                                      
Type: hash:ip,port                                                              
Header: family inet hashsize 1024 maxelem 65536                                 
Size in memory: 16888                                                           
References: 0                                                                   
Members:                                                                        
192.168.0.3,tcp:81                                                              
192.168.0.0,tcp:82                                                              
192.168.0.1,tcp:81                                                              
192.168.0.1,tcp:82                                                              
192.168.0.3,tcp:82                                                              
192.168.0.0,tcp:80                                                              
192.168.0.2,tcp:80                                                              
192.168.0.0,tcp:81                                                              
192.168.0.1,tcp:80                                                              
192.168.0.2,tcp:82                                                              
192.168.0.2,tcp:81                                                              
192.168.0.3,tcp:80                                                              
                      
Best regards,
Jozsef
-
E-mail  : kadlec@blackhole.kfki.hu, kadlec@mail.kfki.hu
PGP key : http://www.kfki.hu/~kadlec/pgp_public_key.txt
Address : KFKI Research Institute for Particle and Nuclear Physics
          H-1525 Budapest 114, POB. 49, Hungary

Re: [ANNOUNCE] ipset-5.0 released

[Mr Dash Four]

> Just to illustrate:
>
> # ipset create test hash:ip,port                          
> # ipset add test 192.168.0.0/30,tcp:80-82                 
> # ipset list test                                         
> Name: test                                                                      
> Type: hash:ip,port                                                              
> Header: family inet hashsize 1024 maxelem 65536                                 
> Size in memory: 16888                                                           
> References: 0                                                                   
> Members:                                                                        
> 192.168.0.3,tcp:81                                                              
> 192.168.0.0,tcp:82                                                              
> 192.168.0.1,tcp:81                                                              
> 192.168.0.1,tcp:82                                                              
> 192.168.0.3,tcp:82                                                              
> 192.168.0.0,tcp:80                                                              
> 192.168.0.2,tcp:80                                                              
> 192.168.0.0,tcp:81                                                              
> 192.168.0.1,tcp:80                                                              
> 192.168.0.2,tcp:82                                                              
> 192.168.0.2,tcp:81                                                              
> 192.168.0.3,tcp:80                                                              
>   
Wow! telepathy must be my forte!!! That's just the example I emailed you 
to see if I understand you correctly!

OK, does that differ if I have hash:net,port set (I presume when listing 
with ipset -L you will show the net ranges - 
192.168.0.0-192.168.0.0,tcp:80-82), is that right?

Re: [ANNOUNCE] ipset-5.0 released

[Jozsef Kadlecsik]

On Thu, 23 Dec 2010, Mr Dash Four wrote:

> > Just to illustrate:
> > 
> > # ipset create test hash:ip,port                          # ipset add test
> > 192.168.0.0/30,tcp:80-82                 # ipset list test
> > Name: test
> > Type: hash:ip,port
> > Header: family inet hashsize 1024 maxelem 65536
> > Size in memory: 16888
> > References: 0
> > Members:
> > 192.168.0.3,tcp:81
> > 192.168.0.0,tcp:82
> > 192.168.0.1,tcp:81
> > 192.168.0.1,tcp:82
> > 192.168.0.3,tcp:82
> > 192.168.0.0,tcp:80
> > 192.168.0.2,tcp:80
> > 192.168.0.0,tcp:81
> > 192.168.0.1,tcp:80
> > 192.168.0.2,tcp:82
> > 192.168.0.2,tcp:81
> > 192.168.0.3,tcp:80                                                                
> Wow! telepathy must be my forte!!! That's just the example I emailed you to
> see if I understand you correctly!
>
> OK, does that differ if I have hash:net,port set (I presume when listing with
> ipset -L you will show the net ranges - 192.168.0.0-192.168.0.0,tcp:80-82), is
> that right?

For net types the networks are not exploded, of course:

# ipset create test hash:net,port 
# ipset add test 192.168.0.0/30,tcp:80-82
# ipset list test
Name: test                                                                      
Type: hash:net,port                                                             
Header: family inet hashsize 1024 maxelem 65536                                 
Size in memory: 16856                                                           
References: 0                                                                   
Members:                                                                        
192.168.0.0/30,tcp:80                                                           
192.168.0.0/30,tcp:82                                                           
192.168.0.0/30,tcp:81                                                           

However please note, the "net" types slow down linearly with the number of 
different network prefixes in the set.

Best regards,
Jozsef
-
E-mail  : kadlec@blackhole.kfki.hu, kadlec@mail.kfki.hu
PGP key : http://www.kfki.hu/~kadlec/pgp_public_key.txt
Address : KFKI Research Institute for Particle and Nuclear Physics
          H-1525 Budapest 114, POB. 49, Hungary

Re: [ANNOUNCE] ipset-5.0 released

[Mr Dash Four]

>> OK, does that differ if I have hash:net,port set (I presume when listing with
>> ipset -L you will show the net ranges - 192.168.0.0-192.168.0.0,tcp:80-82), is
>> that right?
>>     
>
> For net types the networks are not exploded, of course:
>
> # ipset create test hash:net,port 
> # ipset add test 192.168.0.0/30,tcp:80-82
> # ipset list test
> Name: test                                                                      
> Type: hash:net,port                                                             
> Header: family inet hashsize 1024 maxelem 65536                                 
> Size in memory: 16856                                                           
> References: 0                                                                   
> Members:                                                                        
> 192.168.0.0/30,tcp:80                                                           
> 192.168.0.0/30,tcp:82                                                           
> 192.168.0.0/30,tcp:81                                                           
>   
OK, it is much clearer to me now.

You are expanding the command line port ranges (80-82 in your example 
above) to add the underlying elements. Could you not do a similar 
arrangement for the protocol as well?

For example, if I execute "ipset add test 192.168.0.0/30,*:80-82" (or 
"all" instead of "*" if that is not possible) could you not expand that 
to represent the following elements:

192.168.0.0/30,tcp,80
..
192.168.0.0/30,tcp,82
192.168.0.0/30,udp,80
..
192.168.0.0/30,udp,82

It is not a perfect solution by any stretch, but at least it will save 
me the hassle, as with the introduction of the port ranges, of executing 
the statement twice each time I want to include net and port ranges 
without being interested in the protocol match.

As an aside note: since this is a hash set I am assuming that you use 
hashes to do the match as hash matching is very fast. I also presume 
that is the main reason you expand the port ranges so that you can 
calculate the hashes for that particular element and then match it with 
the real ip/subnet,protocol,port.

If that is the case how do you match subnets? In other words (using your 
example above) how do you match (the calculated hash for) 
192.168.0.2,tcp:80 against (the calculated hash for) 
192.168.0.0/30,tcp:80 - they, on the face of it, will differ!

> However please note, the "net" types slow down linearly with the number of 
> different network prefixes in the set.
>   
Is this slowdown more than when I use iptreemap set for example? How 
does the new set (hash) performance compares against the 'old' 
iptreemap/treemap sets used in 4.x?

Re: [ANNOUNCE] ipset-5.0 released

[Jozsef Kadlecsik]

On Sun, 26 Dec 2010, Mr Dash Four wrote:

> > > OK, does that differ if I have hash:net,port set (I presume when listing
> > > with
> > > ipset -L you will show the net ranges -
> > > 192.168.0.0-192.168.0.0,tcp:80-82), is
> > > that right?
> > 
> > For net types the networks are not exploded, of course:
> > 
> > # ipset create test hash:net,port # ipset add test 192.168.0.0/30,tcp:80-82
> > # ipset list test
> > Name: test
> > Type: hash:net,port
> > Header: family inet hashsize 1024 maxelem 65536
> > Size in memory: 16856
> > References: 0
> > Members:
> > 192.168.0.0/30,tcp:80
> > 192.168.0.0/30,tcp:82
> > 192.168.0.0/30,tcp:81                                                             
> OK, it is much clearer to me now.
> 
> You are expanding the command line port ranges (80-82 in your example above)
> to add the underlying elements. Could you not do a similar arrangement for the
> protocol as well?
> 
> For example, if I execute "ipset add test 192.168.0.0/30,*:80-82" (or "all"
> instead of "*" if that is not possible) could you not expand that to represent
> the following elements:
> 
> 192.168.0.0/30,tcp,80
> ..
> 192.168.0.0/30,tcp,82
> 192.168.0.0/30,udp,80
> ..
> 192.168.0.0/30,udp,82

That won't go: tcp and udp are not the possible protocols alone, so by any 
reasonable syntax, '*' or 'all' should have to mean all protocols from 
/etc/protocols.
 
> It is not a perfect solution by any stretch, but at least it will save me the
> hassle, as with the introduction of the port ranges, of executing the
> statement twice each time I want to include net and port ranges without being
> interested in the protocol match.

What I could come up as something similar in effect is to make possible to 
use a copy of /etc/services maintained by the administrator of the given 
ipset installation and whatever protocols and ports a service name is
resolved to from that file, use those.

However such a feature won't be added tomorrow. Nor this year (:-) or the 
very near future.
 
> As an aside note: since this is a hash set I am assuming that you use hashes
> to do the match as hash matching is very fast. I also presume that is the main
> reason you expand the port ranges so that you can calculate the hashes for
> that particular element and then match it with the real
> ip/subnet,protocol,port.

Yes, that's the reason for the expansion.
 
> If that is the case how do you match subnets? In other words (using your
> example above) how do you match (the calculated hash for) 192.168.0.2,tcp:80
> against (the calculated hash for) 192.168.0.0/30,tcp:80 - they, on the face of
> it, will differ!

For the "net" (sub)type of sets the different prefixes in the set is 
recorded. When testing (adding/deleting) entries, the IP part of the entry 
is masked with the prefixes one by one and looked up until a match is 
found or the list is exhausted - that's why the speed is linearly slows 
down with the different number of prefixes added to the set. An example:

# ipset create test hash:net
# ipset add test 192.168.0.0/30
# ipset add test 192.168.1.0/24

There are two different prefixes in the set, 30, 24. So when the command 
is issued

# ipset test test 192.168.0.1

the address 192.168.0.1 is masked with /30 and the result is looked up 
and the match is reported:

192.168.0.1 is in set test.

If we want to test 192.168.1.1, then first it's masked with /30. Because 
there's no match, it's masked again by /24 then and now it's found:

root@teg:~/ipset/src# ./ipset test test 192.168.1.1                             
192.168.1.1 is in set test.

> > However please note, the "net" types slow down linearly with the number of
> > different network prefixes in the set.
> >   
> Is this slowdown more than when I use iptreemap set for example? How does the
> new set (hash) performance compares against the 'old' iptreemap/treemap sets
> used in 4.x?

I never measured the speed difference between the two types. A good guess 
is that iptreemap is faster. However, the problem with iptree(map) is that 
it's susceptible when used as a container to block attackers dynamically 
by the SET target: it's trivial to mount an attack by which the worst tree 
structure is forced to be created. Therefore iptree(map) was dropped at
ipset-5.

Best regards,
Jozsef
-
E-mail  : kadlec@blackhole.kfki.hu, kadlec@mail.kfki.hu
PGP key : http://www.kfki.hu/~kadlec/pgp_public_key.txt
Address : KFKI Research Institute for Particle and Nuclear Physics
          H-1525 Budapest 114, POB. 49, Hungary

Re: [ANNOUNCE] ipset-5.0 released

[Mr Dash Four]

>> You are expanding the command line port ranges (80-82 in your example above)
>> to add the underlying elements. Could you not do a similar arrangement for the
>> protocol as well?
>>     
> That won't go: tcp and udp are not the possible protocols alone, so by any 
> reasonable syntax, '*' or 'all' should have to mean all protocols from 
> /etc/protocols.
>   
Whatever the appropriate place holder is - 'all', '*', '@', '$$$$$$' 
etc. etc - as long as it does the job at the end it won't matter.

>> It is not a perfect solution by any stretch, but at least it will save me the
>> hassle, as with the introduction of the port ranges, of executing the
>> statement twice each time I want to include net and port ranges without being
>> interested in the protocol match.
>>     
>
> What I could come up as something similar in effect is to make possible to 
> use a copy of /etc/services maintained by the administrator of the given 
> ipset installation and whatever protocols and ports a service name is
> resolved to from that file, use those.
>   
I wasn't aware that there are more than 2 (tcp and udp) protocols on 
which I could define/use ports!

> However such a feature won't be added tomorrow. Nor this year (:-) or the 
> very near future.
>   
What are your plans?

> For the "net" (sub)type of sets the different prefixes in the set is 
> recorded. When testing (adding/deleting) entries, the IP part of the entry 
> is masked with the prefixes one by one and looked up until a match is 
> found or the list is exhausted - that's why the speed is linearly slows 
> down with the different number of prefixes added to the set. An example:
>
> # ipset create test hash:net
> # ipset add test 192.168.0.0/30
> # ipset add test 192.168.1.0/24
>
> There are two different prefixes in the set, 30, 24. So when the command 
> is issued
>
> # ipset test test 192.168.0.1
>
> the address 192.168.0.1 is masked with /30 and the result is looked up 
> and the match is reported:
>   
So, in other words 'internally' you store something like (using your 
example above) 192.168.0.0 (the IP part), 30 (the prefix), tcp, 80 in 
the structure, is that right?
Do you then use the mask of each member of that set (30 and 24 in your 
example above) until you get a match? In other words, you 'walk through' 
every internally stored element and try their mask and then lookup for a 
match, is that how you do it?


>> Is this slowdown more than when I use iptreemap set for example? How does the
>> new set (hash) performance compares against the 'old' iptreemap/treemap sets
>> used in 4.x?
>>     
>
> I never measured the speed difference between the two types. A good guess 
> is that iptreemap is faster.
Thought so!

> However, the problem with iptree(map) is that 
> it's susceptible when used as a container to block attackers dynamically 
> by the SET target: it's trivial to mount an attack by which the worst tree 
> structure is forced to be created.
Could you expand on that bit please? What do you mean?

Re: [ANNOUNCE] ipset-5.0 released

[Jozsef Kadlecsik]

On Sun, 26 Dec 2010, Mr Dash Four wrote:

> > > You are expanding the command line port ranges (80-82 in your example
> > > above)
> > > to add the underlying elements. Could you not do a similar arrangement for
> > > the
> > > protocol as well?
> > >     
> > That won't go: tcp and udp are not the possible protocols alone, so by any
> > reasonable syntax, '*' or 'all' should have to mean all protocols from
> > /etc/protocols.
> >   
> Whatever the appropriate place holder is - 'all', '*', '@', '$$$$$$' etc. etc
> - as long as it does the job at the end it won't matter.

It does matter. Non-intuitive syntax and notation are bad. Moreover, the 
notation should be open to handle other protocols too (see below).
 
> > What I could come up as something similar in effect is to make possible to
> > use a copy of /etc/services maintained by the administrator of the given
> > ipset installation and whatever protocols and ports a service name is
> > resolved to from that file, use those.
> >   
> I wasn't aware that there are more than 2 (tcp and udp) protocols on which I
> could define/use ports!

sctp, udplite.

ICMP and ICMPv6 already interpreted as "proto":"port" by ipset, regardless 
that "port" is actually type and code.

> > However such a feature won't be added tomorrow. Nor this year (:-) or the
> > very near future.
> >   
> What are your plans?

Double check and triple check both the kernel and userspace of ipset 5. 
Get more feedback, especially from uncommon architecture. Submit for 
kernel inclusion. After that new features can be added.
 
> > For the "net" (sub)type of sets the different prefixes in the set is
> > recorded. When testing (adding/deleting) entries, the IP part of the entry
> > is masked with the prefixes one by one and looked up until a match is found
> > or the list is exhausted - that's why the speed is linearly slows down with
> > the different number of prefixes added to the set. An example:
> > 
> > # ipset create test hash:net
> > # ipset add test 192.168.0.0/30
> > # ipset add test 192.168.1.0/24
> > 
> > There are two different prefixes in the set, 30, 24. So when the command is
> > issued
> > 
> > # ipset test test 192.168.0.1
> > 
> > the address 192.168.0.1 is masked with /30 and the result is looked up and
> > the match is reported:
> >   
> So, in other words 'internally' you store something like (using your example
> above) 192.168.0.0 (the IP part), 30 (the prefix), tcp, 80 in the structure,
> is that right?

Yes, exactly.

> Do you then use the mask of each member of that set (30 and 24 in your example
> above) until you get a match? In other words, you 'walk through' every
> internally stored element and try their mask and then lookup for a match, is
> that how you do it?

Besides the elements, the different prefixes are also stored in the set. 
So the test set in the example "knows" that the prefixes 30 and 24 occur 
only and use only those to find a match, starting from the most speficic 
prefix to the least one.

> > However, the problem with iptree(map) is that it's susceptible when used as
> > a container to block attackers dynamically by the SET target: it's trivial
> > to mount an attack by which the worst tree structure is forced to be
> > created.
> Could you expand on that bit please? What do you mean?

The iptree(map) sets store the elements in a tree structure which follows 
the dotted decimal notation of the IPv4 addresses. Let's assume, someone 
catches attackers by a given match and stores their IP addresses in the 
iptree type of set called "attackers":

# If some matches are true, it's an attacker, store the address
...... -j SET attackers src
# Drop known attackers
.... -m set attackers src -j DROP

What I referred to was that it's not hard to send forged packets, which 
create the widest and least filled in (i.e. worst) tree structure. It does 
not slow down the set but pushes it to the worst memory usage case.

I regard it as a weakness of the type: an attacker must not be able to 
trigger any worst case situation of a set type intentionally.

Best regards,
Jozsef
-
E-mail  : kadlec@blackhole.kfki.hu, kadlec@mail.kfki.hu
PGP key : http://www.kfki.hu/~kadlec/pgp_public_key.txt
Address : KFKI Research Institute for Particle and Nuclear Physics
          H-1525 Budapest 114, POB. 49, Hungary

Re: [ANNOUNCE] ipset-5.0 released

[Mr Dash Four]

>> Whatever the appropriate place holder is - 'all', '*', '@', '$$$$$$' etc. etc
>> - as long as it does the job at the end it won't matter.
>>     
>
> It does matter. Non-intuitive syntax and notation are bad. Moreover, the 
> notation should be open to handle other protocols too (see below).
>   
What I meant was that whatever the prefix *you* decide to put in, for 
*me* it won't matter as long as it does its job.

>> I wasn't aware that there are more than 2 (tcp and udp) protocols on which I
>> could define/use ports!
>>     
>
> sctp, udplite.
>   
I learn something new every day. Then it makes even more sense to store 
the protocol as a number, not as string for easier matching - useful if 
you one day decide to apply masking (or other convenient form of 
searching) against the captured protocol (which could also be a number). 
;-)

>> What are your plans?
>>     
>
> Double check and triple check both the kernel and userspace of ipset 5. 
> Get more feedback, especially from uncommon architecture. Submit for 
> kernel inclusion. After that new features can be added.
>   
Fair enough. Any idea when ipset 5.2 will be incorporated into xtables 
so that I could download it and start testing it? *nudges Jan*

As it stands, I do not have enough information to build the rpm spec 
file in order to create a complete xtables rpm package to incorporate 
your version of ipset (without this I cannot distribute it across my 
test machines easily), so unless I am able to build this rpm the whole 
thing is a no go from me.

>> So, in other words 'internally' you store something like (using your example
>> above) 192.168.0.0 (the IP part), 30 (the prefix), tcp, 80 in the structure,
>> is that right?
>>     
>
> Yes, exactly.
>
>   
>> Do you then use the mask of each member of that set (30 and 24 in your example
>> above) until you get a match? In other words, you 'walk through' every
>> internally stored element and try their mask and then lookup for a match, is
>> that how you do it?
>>     
>
> Besides the elements, the different prefixes are also stored in the set. 
> So the test set in the example "knows" that the prefixes 30 and 24 occur 
> only and use only those to find a match, starting from the most speficic 
> prefix to the least one.
>   
If I understand you correctly, you keep a separate structure which 
stores all prefixes used in the target set, walk through that structure 
and for each element you build start-end range based on the IP address 
captured and the current prefix. You then create a hash on the 
calculated start-end range, protocol and port triple and then test that 
hash against the hash value of the target set triple (IP range, 
protocol, port). Is that how you do it?

I understand that things are a bit easier with hash:ip,protocol,port as 
you don't need to store prefixes - you just calculate the hashes and 
match them against the corresponding hashes of the set triple, right?

>>> However, the problem with iptree(map) is that it's susceptible when used as
>>> a container to block attackers dynamically by the SET target: it's trivial
>>> to mount an attack by which the worst tree structure is forced to be
>>> created.
>>>       
>> Could you expand on that bit please? What do you mean?
>>     
>
> The iptree(map) sets store the elements in a tree structure which follows 
> the dotted decimal notation of the IPv4 addresses. Let's assume, someone 
> catches attackers by a given match and stores their IP addresses in the 
> iptree type of set called "attackers":
>
> # If some matches are true, it's an attacker, store the address
> ...... -j SET attackers src
> # Drop known attackers
> .... -m set attackers src -j DROP
>
> What I referred to was that it's not hard to send forged packets, which 
> create the widest and least filled in (i.e. worst) tree structure. It does 
> not slow down the set but pushes it to the worst memory usage case.
>   
Couldn't the same be said for the hash (or any other?) ipsets as well? I 
am not using that 'feature' anyway, as I include the ip addresses/ranges 
manually from script files (though some of them are automatically 
generated and are based on the geoip database which is downloaded at 
regular intervals), so that 'vulnerability' does not apply to me.

> I regard it as a weakness of the type: an attacker must not be able to 
> trigger any worst case situation of a set type intentionally.
>   
How is that changed in the hash (or bitmap even) ipsets - would that not 
be the same then: an attacker crafts specifically designed packets to 
include certain ip/port ranges in a set (*a* set - it could be any set)?

I think the culprit is not the set itself, but the SET target as I find 
it rather foolish that someone would let an outsider to manipulate a set 
using this SET target. For example, if I am an attacker and I know that 
the target machine relies on access to certain IP address, then I could 
craft a packet in such a way that it triggers this SET target which then 
includes the IP address/range in the 'attackers' set and therefore 
disable access to that address on the target machine - job done!

Re: [ANNOUNCE] ipset-5.0 released

[Jozsef Kadlecsik]

On Mon, 27 Dec 2010, Mr Dash Four wrote:

> I learn something new every day. Then it makes even more sense to store the
> protocol as a number, not as string for easier matching - useful if you one
> day decide to apply masking (or other convenient form of searching) against
> the captured protocol (which could also be a number). ;-)

Of course the protocol number is stored and not the string.
 
> > Double check and triple check both the kernel and userspace of ipset 5. Get
> > more feedback, especially from uncommon architecture. Submit for kernel
> > inclusion. After that new features can be added.
> >   
> Fair enough. Any idea when ipset 5.2 will be incorporated into xtables so that
> I could download it and start testing it? *nudges Jan*

You constantly ignore the difference between xtables and xtables-addons...
 
> As it stands, I do not have enough information to build the rpm spec file in
> order to create a complete xtables rpm package to incorporate your version of
> ipset (without this I cannot distribute it across my test machines easily), so
> unless I am able to build this rpm the whole thing is a no go from me.
> 
> > Besides the elements, the different prefixes are also stored in the set. So
> > the test set in the example "knows" that the prefixes 30 and 24 occur only
> > and use only those to find a match, starting from the most speficic prefix
> > to the least one.
> >   
> If I understand you correctly, you keep a separate structure which stores all
> prefixes used in the target set, walk through that structure and for each
> element you build start-end range based on the IP address captured and the
> current prefix. You then create a hash on the calculated start-end range,
> protocol and port triple and then test that hash against the hash value of the
> target set triple (IP range, protocol, port). Is that how you do it?

No, there's no point to store start-end ranges: the network address and 
the prefix is stored, which thus require less memory compared to the 
range.

> I understand that things are a bit easier with hash:ip,protocol,port as you
> don't need to store prefixes - you just calculate the hashes and match them
> against the corresponding hashes of the set triple, right?

Yes, that's it.
 
> > What I referred to was that it's not hard to send forged packets, which
> > create the widest and least filled in (i.e. worst) tree structure. It does
> > not slow down the set but pushes it to the worst memory usage case.
> >   
> Couldn't the same be said for the hash (or any other?) ipsets as well? I am
> not using that 'feature' anyway, as I include the ip addresses/ranges manually
> from script files (though some of them are automatically generated and are
> based on the geoip database which is downloaded at regular intervals), so that
> 'vulnerability' does not apply to me.

No, there's no such weakness in the hash or bitmap types.
 
> > I regard it as a weakness of the type: an attacker must not be able to
> > trigger any worst case situation of a set type intentionally.
> >   
> How is that changed in the hash (or bitmap even) ipsets - would that not be
> the same then: an attacker crafts specifically designed packets to include
> certain ip/port ranges in a set (*a* set - it could be any set)?

The attacker cannot guess the initial parameter of the hash types, so even 
if he/she knew the hash size, cannot figure out what addresses to send in 
order to create clashing entries in the hash. The initial parameter is 
even not saved when saving a set, so if you save a hash type of set, 
destroy and then restore it, the set won't be expressed by the same hash 
structure as the original one because the initial parameter is internal 
and created random.
 
> I think the culprit is not the set itself, but the SET target as I find it
> rather foolish that someone would let an outsider to manipulate a set using
> this SET target. For example, if I am an attacker and I know that the target
> machine relies on access to certain IP address, then I could craft a packet in
> such a way that it triggers this SET target which then includes the IP
> address/range in the 'attackers' set and therefore disable access to that
> address on the target machine - job done!

Of course, it's absolutely possible. However in order to slow down for 
example ssh scanning, even short time blocking of the addresses from 
which scanning is detected can be quite useful.

Best regards,
Jozsef
-
E-mail  : kadlec@blackhole.kfki.hu, kadlec@mail.kfki.hu
PGP key : http://www.kfki.hu/~kadlec/pgp_public_key.txt
Address : KFKI Research Institute for Particle and Nuclear Physics
          H-1525 Budapest 114, POB. 49, Hungary

Re: [ANNOUNCE] ipset-5.0 released

[Mr Dash Four]

>> Fair enough. Any idea when ipset 5.2 will be incorporated into xtables so that
>> I could download it and start testing it? *nudges Jan*
>>     
>
> You constantly ignore the difference between xtables and xtables-addons...
>   
You know exactly what I meant.

>> If I understand you correctly, you keep a separate structure which stores all
>> prefixes used in the target set, walk through that structure and for each
>> element you build start-end range based on the IP address captured and the
>> current prefix. You then create a hash on the calculated start-end range,
>> protocol and port triple and then test that hash against the hash value of the
>> target set triple (IP range, protocol, port). Is that how you do it?
>>     
>
> No, there's no point to store start-end ranges: the network address and 
> the prefix is stored, which thus require less memory compared to the 
> range.
>   
OK, so you store IP address and then the range. Do you then perform the 
process as I described above in order to find a match?

>> I think the culprit is not the set itself, but the SET target as I find it
>> rather foolish that someone would let an outsider to manipulate a set using
>> this SET target. For example, if I am an attacker and I know that the target
>> machine relies on access to certain IP address, then I could craft a packet in
>> such a way that it triggers this SET target which then includes the IP
>> address/range in the 'attackers' set and therefore disable access to that
>> address on the target machine - job done!
>>     
>
> Of course, it's absolutely possible. However in order to slow down for 
> example ssh scanning, even short time blocking of the addresses from 
> which scanning is detected can be quite useful.
>   
Anything which manipulates the internal structure from outside is, in my 
opinion, not a good idea for the reason I mentioned above.

On a separate note, once I've installed ipset 5.2 I am very keen on 
testing both type of sets (iptreemap and the new hash set) to see what 
is the performance penalty with the new sets (there will probably be one 
as your search algorithm is more complex in 5.x and would require more 
iterations compared with the 4.x type sets).

Re: [ANNOUNCE] ipset-5.0 released

[Jozsef Kadlecsik]

On Mon, 27 Dec 2010, Mr Dash Four wrote:

> > > Fair enough. Any idea when ipset 5.2 will be incorporated into 
> > > xtables so that I could download it and start testing it? *nudges 
> > > Jan*
> > 
> > You constantly ignore the difference between xtables and xtables-addons...
> >   
> You know exactly what I meant.

I hope, but still one cannot be sure.

We call the unified part of iptables and ip6tables to xtables and it's in 
the "iptables" package which has never included ipset (except the 
userspace match/target interfaces).

The "xtables-addons" package is the successor of patch-o-matic-ng for 
extensions which are not (yet) accepted in netfilter and Jan included 
ipset into it.

So please take the trouble and use the correct term.
 
> > > If I understand you correctly, you keep a separate structure which 
> > > stores all prefixes used in the target set, walk through that 
> > > structure and for each element you build start-end range based on 
> > > the IP address captured and the current prefix. You then create a 
> > > hash on the calculated start-end range, protocol and port triple and 
> > > then test that hash against the hash value of the target set triple 
> > > (IP range, protocol, port). Is that how you do it?
> > 
> > No, there's no point to store start-end ranges: the network address and the
> > prefix is stored, which thus require less memory compared to the range.
> >   
> OK, so you store IP address and then the range.

The IP address and the prefix is stored. Which can easily be converted 
into a range anytime.

> Do you then perform the process as I described above in order to find a 
> match?

Leaving out the range stuff, more-or-less yes. In order to test an 
element:

1. The IP address is masked with the first prefix from the list of
   prefixes in the set.
2. The resulted address and prefix are hashed; or if it's a hash:net,port 
   type, then the resulted address, prefix, protocol number and port 
   number are hashed
3. The hash slot is looked up and searched for a matching entry
4. If there's no match, then (virtually) shift off the first prefix from 
   the list of the prefixes and goto 1.
5. If there's a match then report success. If the list of prefixes is
   empty, report failure.
 
> On a separate note, once I've installed ipset 5.2 I am very keen on testing
> both type of sets (iptreemap and the new hash set) to see what is the
> performance penalty with the new sets (there will probably be one as your
> search algorithm is more complex in 5.x and would require more iterations
> compared with the 4.x type sets).

Different methods give different results depending on what you measure: 
insert element, delete, lookup, lookup non-matching entry, memory usage. 
Have a look at Rusty Russell's hash comparisons:
http://rusty.ozlabs.org/?p=94. The measuring of the previous hashing 
method in ipset 5.x can be found here: 
http://www.kfki.hu/~kadlec/sw/netfilter/hash/. The current one was also 
compared to other algorithms but I haven't had time yet to write the 
summary.

I'm looking forward to see your results.

Best regards,
Jozsef
-
E-mail  : kadlec@blackhole.kfki.hu, kadlec@mail.kfki.hu
PGP key : http://www.kfki.hu/~kadlec/pgp_public_key.txt
Address : KFKI Research Institute for Particle and Nuclear Physics
          H-1525 Budapest 114, POB. 49, Hungary

Re: [ANNOUNCE] ipset-5.0 released

[Mr Dash Four]

>>> The implementation behind ipset looks up the  (ipaddr, proto, port) triple
>>> in one step. Such packing don't work there.
>>>   
>>>       
>> If that's the case how do you lookup IP address and port ranges then?
>>     
>
> IP address and port ranges are exploded and the elements are inserted 
> one-by-one. And the exploded ranges are *not* converted back to ranges 
> when listing/saving the sets. At the bitmap types the ranges could be 
> converted back (not done yet), at the hash types it's not possible.
>   
If I understand you correctly, if I define hash:net,proto,port ipset and 
add a single element to it - 10.1.1.0/30,udp,80-83 - that translates (in 
primitive terms) to:

10.1.1.0,udp,80
10.1.1.0,udp,81
...
10.1.1.0,udp,83
10.1.1.1,udp,80
...
10.1.1.1,udp,83
...
...
10.1.1.3,udp,83

In other words, the set actually consist of 4 (subnet size) * 1 
(protocol) * 4 (port ranges) =16 'internal' elements, is that right?

One other question - if I insert the above element in the set what is 
shown when I execute ipset -L: "10.1.1.0-10.1.1.3,udp,80-83" or the 
various permutations I listed above?

Re: [ANNOUNCE] ipset-5.0 released

[Jozsef Kadlecsik]

On Thu, 23 Dec 2010, Mr Dash Four wrote:

> > > > The implementation behind ipset looks up the  (ipaddr, proto, port)
> > > > triple
> > > > in one step. Such packing don't work there.
> > > >         
> > > If that's the case how do you lookup IP address and port ranges then?
> > >     
> > 
> > IP address and port ranges are exploded and the elements are inserted
> > one-by-one. And the exploded ranges are *not* converted back to ranges when
> > listing/saving the sets. At the bitmap types the ranges could be converted
> > back (not done yet), at the hash types it's not possible.
> >   
> If I understand you correctly, if I define hash:net,proto,port ipset and add a
> single element to it - 10.1.1.0/30,udp,80-83 - that translates (in primitive
> terms) to:
> 
> 10.1.1.0,udp,80
> 10.1.1.0,udp,81
> ...
> 10.1.1.0,udp,83
> 10.1.1.1,udp,80
> ...
> 10.1.1.1,udp,83
> ...
> ...
> 10.1.1.3,udp,83

No, "net" types are not exploded in the terms of networks.
 
> One other question - if I insert the above element in the set what is shown
> when I execute ipset -L: "10.1.1.0-10.1.1.3,udp,80-83" or the various
> permutations I listed above?

The protocol does not allow to list a subset of the elements in a set. 
Just the whole set can be listed.

Best regards,
Jozsef
-
E-mail  : kadlec@blackhole.kfki.hu, kadlec@mail.kfki.hu
PGP key : http://www.kfki.hu/~kadlec/pgp_public_key.txt
Address : KFKI Research Institute for Particle and Nuclear Physics
          H-1525 Budapest 114, POB. 49, Hungary

Re: [ANNOUNCE] ipset-5.0 released

[Jan Engelhardt]

On Thursday 2010-12-23 18:50, Mr Dash Four wrote:
>
>> SIP is complex... IRC uses TCP only.
>>  
> IRC on Port 194

Maybe 20 years back in time, these days it is practically 666[6-9]
or something in that range.