ipporthash, ipportiphash, ipportnethash problems

ipporthash, ipportiphash, ipportnethash problems

[Mr Dash Four]

I am not able to get *any* matches when I have the following combinations:

1. ipporthash: x.x.x.x,y [src,dst] or [dst,src] (i.e. source IP address 
and destination port and vice versa);
2. ipportiphash: x.x.x.x,y,z.z.z.z [src,dst,dst] or [dst,src,src] (i.e. 
source IP address, destination port and destination IP address and vice 
versa);
3. ipportnethash x.x.x.x,y,z.z.z.z/c [src,dst,dst] or [dst,src,src] 
(i.e. source IP address, destination port and destination subnet and 
vice versa);

When I split up the above sets like:

For case 1: match-set single-set src match-set single-port dst (and vice 
versa with src and dst reversed) - all in one line - i.e. creating two 
separate sets containing the appropriate src IP address and and dst 
ports respectively;
For case 2: match-set single-set src match-set double-set dst,dst (and 
vice versa with src and dst reversed) - all in one line - i.e. creating 
two separate sets containing the appropriate src IP address and another 
set containing the destination IP addresses *and* ports respectively;
For case 3: match-set single-set src match-set double-net-set dst,dst 
(and vice versa with src and dst reversed) - all in one line - i.e. 
creating two separate sets containing the appropriate src IP address and 
another set containing the destination IP subnet addresses *and* ports 
respectively;

I was able to get a match! This leads me to believe that either xtables 
has a bug and can't handle mixed src,dst designations in the same set, 
or, I am doing something wrong. Which is it?

Re: ipporthash, ipportiphash, ipportnethash problems

[Jozsef Kadlecsik]

On Thu, 30 Sep 2010, Mr Dash Four wrote:

> I am not able to get *any* matches when I have the following combinations:
> 
> 1. ipporthash: x.x.x.x,y [src,dst] or [dst,src] (i.e. source IP address and
> destination port and vice versa);
> 2. ipportiphash: x.x.x.x,y,z.z.z.z [src,dst,dst] or [dst,src,src] (i.e. source
> IP address, destination port and destination IP address and vice versa);
> 3. ipportnethash x.x.x.x,y,z.z.z.z/c [src,dst,dst] or [dst,src,src] (i.e.
> source IP address, destination port and destination subnet and vice versa);
> 
> When I split up the above sets like:
> 
> For case 1: match-set single-set src match-set single-port dst (and vice versa
> with src and dst reversed) - all in one line - i.e. creating two separate sets
> containing the appropriate src IP address and and dst ports respectively;
> For case 2: match-set single-set src match-set double-set dst,dst (and vice
> versa with src and dst reversed) - all in one line - i.e. creating two
> separate sets containing the appropriate src IP address and another set
> containing the destination IP addresses *and* ports respectively;
> For case 3: match-set single-set src match-set double-net-set dst,dst (and
> vice versa with src and dst reversed) - all in one line - i.e. creating two
> separate sets containing the appropriate src IP address and another set
> containing the destination IP subnet addresses *and* ports respectively;
> 
> I was able to get a match! This leads me to believe that either xtables has a
> bug and can't handle mixed src,dst designations in the same set, or, I am
> doing something wrong. Which is it?

You should provide at least the following: kernel, iptables, ipset version 
numbers, the iptables rules and the (non)matching sets with the elements.
Without exact details I cannot help.

Best regards,
Jozsef
-
E-mail  : kadlec@blackhole.kfki.hu, kadlec@mail.kfki.hu
PGP key : http://www.kfki.hu/~kadlec/pgp_public_key.txt
Address : KFKI Research Institute for Particle and Nuclear Physics
          H-1525 Budapest 114, POB. 49, Hungary

Re: ipporthash, ipportiphash, ipportnethash problems

[Mr Dash Four]

> You should provide at least the following: kernel, iptables, ipset version 
> numbers, the iptables rules and the (non)matching sets with the elements.
> Without exact details I cannot help.
>   
kernel-2.6.34.7-56.fc13.x86_64
iptables-1.4.7-2.fc13.x86_64
ipset 4.3
xtables-addons 1.29

For the purpose of this small test, I am initiating a connection from 
the local machine (10.1.2.7, network 10.1.2.0/24) to my other machine 
(10.1.1.3, network 10.1.1.0/24) over ssh (port 22) and ask iptables to 
match that and log it to the syslog.

1. IP,port hash (non)match (trying to match source IP 10.1.2.7 with 
destination port 22)

ipset -N double-set ipporthash --network 10.1.2.0/24
ipset -A double-set 10.1.2.7,22
ipset -L double-set
==================
Name: double-set
Type: ipporthash
References: 0
Header: from: 10.1.2.0 to: 10.1.2.255 hashsize: 1024 probes: 8 resize: 50
Members:
10.1.2.7,22
==================

iptables -A fw2net -p 6 -m set --match-set double-set src,dst -d 
10.1.1.3 -g log0

==================
Chain fw2net (1 references)
 pkts bytes target     prot opt in     out     source               
destination        
    0     0 blackout   all  --  *      *       0.0.0.0/0            
0.0.0.0/0           ctstate INVALID,NEW
    0     0 ACCEPT     all  --  *      *       0.0.0.0/0            
0.0.0.0/0           ctstate RELATED,ESTABLISHED
    0     0 log0       tcp  --  *      *       0.0.0.0/0            
10.1.1.3            [goto] match-set double-set src,dst
    0     0 ACCEPT     all  --  *      *       0.0.0.0/0            
0.0.0.0/0          
==================

After initiating a connection to 10.1.1.3:22 (check for matches):

==================
Chain fw2net (1 references)
 pkts bytes target     prot opt in     out     source               
destination        
    7   520 blackout   all  --  *      *       0.0.0.0/0            
0.0.0.0/0           ctstate INVALID,NEW
   11  1473 ACCEPT     all  --  *      *       0.0.0.0/0            
0.0.0.0/0           ctstate RELATED,ESTABLISHED
    0     0 log0       tcp  --  *      *       0.0.0.0/0            
10.1.1.3            [goto] match-set double-set src,dst
    7   520 ACCEPT     all  --  *      *       0.0.0.0/0            
0.0.0.0/0          
==================

As you can see - there are NO matches!


2. IP,port,IP hash (non)match (trying to match source IP 10.1.2.7 with 
destination port 22 and destination IP 10.1.1.3)

ipset -N tripple-set ipportiphash --network 10.1.2.0/24
ipset -A tripple-set 10.1.2.7,22,10.1.1.3
ipset -L tripple-set
==================
Name: tripple-set
Type: ipportiphash
References: 0
Header: from: 10.1.2.0 to: 10.1.2.255 hashsize: 1024 probes: 8 resize: 50
Members:
10.1.2.7,22,10.1.1.3
==================

iptables -A fw2net -p 6 -m set --match-set tripple-set src,dst,dst -g log0

==================
Chain fw2net (1 references)
 pkts bytes target     prot opt in     out     source               
destination         
    0     0 blackout   all  --  *      *       0.0.0.0/0            
0.0.0.0/0           ctstate INVALID,NEW
    0     0 ACCEPT     all  --  *      *       0.0.0.0/0            
0.0.0.0/0           ctstate RELATED,ESTABLISHED
    0     0 log0       tcp  --  *      *       0.0.0.0/0            
0.0.0.0/0           [goto] match-set tripple-set src,dst,dst
    0     0 ACCEPT     all  --  *      *       0.0.0.0/0            
0.0.0.0/0           
==================

After initiating a connection to 10.1.1.3:22 (check for matches):

==================
Chain fw2net (1 references)
 pkts bytes target     prot opt in     out     source               
destination         
    6   412 blackout   all  --  *      *       0.0.0.0/0            
0.0.0.0/0           ctstate INVALID,NEW
   10  1397 ACCEPT     all  --  *      *       0.0.0.0/0            
0.0.0.0/0           ctstate RELATED,ESTABLISHED
    0     0 log0       tcp  --  *      *       0.0.0.0/0            
0.0.0.0/0           [goto] match-set tripple-set src,dst,dst
    6   412 ACCEPT     all  --  *      *       0.0.0.0/0            
0.0.0.0/0           
==================

As you can see - there are NO matches!

3. IP,port,IP/cidr (non)match (trying to match source IP 10.1.2.7 with 
destination port 22 and destination subnet 10.1.1.0/24)

ipset -N tripple-set ipportnethash --network 10.1.2.0/24
ipset -A tripple-set 10.1.2.7,22,10.1.1.0/24
ipset -L tripple-set
==================
Name: tripple-set
Type: ipportnethash
References: 0
Header: from: 10.1.2.0 to: 10.1.2.255 hashsize: 1024 probes: 8 resize: 50
Members:
10.1.2.7,22,10.1.1.0/24
==================

iptables -A fw2net -p 6 -m set --match-set tripple-set src,dst,dst -g log0

==================
Chain fw2net (1 references)
 pkts bytes target     prot opt in     out     source               
destination        
    0     0 blackout   all  --  *      *       0.0.0.0/0            
0.0.0.0/0           ctstate INVALID,NEW
    0     0 ACCEPT     all  --  *      *       0.0.0.0/0            
0.0.0.0/0           ctstate RELATED,ESTABLISHED
    0     0 log0       tcp  --  *      *       0.0.0.0/0            
0.0.0.0/0           [goto] match-set tripple-set src,dst,dst
    0     0 ACCEPT     all  --  *      *       0.0.0.0/0            
0.0.0.0/0          
==================

After initiating a connection to 10.1.1.3:22 (check for matches):

==================
Chain fw2net (1 references)
 pkts bytes target     prot opt in     out     source               
destination        
    6   412 blackout   all  --  *      *       0.0.0.0/0            
0.0.0.0/0           ctstate INVALID,NEW
   11  1473 ACCEPT     all  --  *      *       0.0.0.0/0            
0.0.0.0/0           ctstate RELATED,ESTABLISHED
    0     0 log0       tcp  --  *      *       0.0.0.0/0            
0.0.0.0/0           [goto] match-set tripple-set src,dst,dst
    6   412 ACCEPT     all  --  *      *       0.0.0.0/0            
0.0.0.0/0          
==================

As you can see - there are NO matches!

So, in all three scenarios above there are NO matches! As I already 
indicated - 1) I assume when I reverse the src and dst combinations in 
all three scenarios above the result would be the same; and 2) When I 
split up the above sets in the way I already indicated in my initial 
post on this thread I am getting the correct matches, which leads me to 
believe that either xtables has a bug and can't handle mixed src,dst 
designations in the same set, or, I am doing something wrong. Which is it?

Re: ipporthash, ipportiphash, ipportnethash problems

[Jozsef Kadlecsik]

On Fri, 1 Oct 2010, Mr Dash Four wrote:

> > You should provide at least the following: kernel, iptables, ipset version
> > numbers, the iptables rules and the (non)matching sets with the elements.
> > Without exact details I cannot help.
> >   
> kernel-2.6.34.7-56.fc13.x86_64
> iptables-1.4.7-2.fc13.x86_64
> ipset 4.3
> xtables-addons 1.29
> 
> For the purpose of this small test, I am initiating a connection from the
> local machine (10.1.2.7, network 10.1.2.0/24) to my other machine (10.1.1.3,
> network 10.1.1.0/24) over ssh (port 22) and ask iptables to match that and log
> it to the syslog.
[...]

Thanks for the complete description - I have just released ipset 4.4 which 
fixes this nasty bug.

Best regards,
Jozsef
-
E-mail  : kadlec@blackhole.kfki.hu, kadlec@mail.kfki.hu
PGP key : http://www.kfki.hu/~kadlec/pgp_public_key.txt
Address : KFKI Research Institute for Particle and Nuclear Physics
          H-1525 Budapest 114, POB. 49, Hungary

Re: ipporthash, ipportiphash, ipportnethash problems

[Mr Dash Four]

> Thanks for the complete description - I have just released ipset 4.4 which 
> fixes this nasty bug.
>   
No worries - I was surprised to find it as ipporthash in particular 
should have been widely used in practice. As for ipportiphash and 
ipportnethash - there were recently included in the distribution due to 
another bug I found when compiling xtables (even though ipportiphash and 
ipportnethash were compiled the .ko files were never included in the 
final release making it impossible to use these two construct types).

A couple of questions and ideas for future releases:

1. Normally I used to get ipset as part of the xtables source. Is that 
still the case or do I have to download the ipset source and compile it 
separately to get the latest changes you made?
2. Related to that question - I used to get the xtables .src.rpm from 
which to compile the code and make an installation rpm. I don't seem to 
be able to do that any longer as all fedora repos are too slow to update 
the latest changes. Do you have a link from which I could get the latest 
changes in ipset/xtables in src.rpm which allows me to build an 
installation rpm. The reason I am asking this is because I am building 
an update images on all my machines (using kickstart) and need the 
installation rpms with the latest changes you made. Compiling this from 
source won't be possible as there is now way I could install this as 
part of the image-building process.

I think you mentioned that in future releases the restrictions on all 
*hash constructs (with regards to using /16 - B class - IP addresses) 
will be removed. Could you confirm that is still the case?

Would it be possible to include a new construct - IP,port,IP,port 
(without IP address restriction) - and how difficult is this to implement?

Similar question - would it be possible to include protocol (either by 
name or its number) as part of all available constructs for matching? 
The reason for this is that, as it stands, in order for me to match, 
say, "IP,port" hosts form my personal VPN I need to use two separate 
sets - one for UDP and one for TCP and I also to create two separate 
statements for iptables (one for tcp and one for udp), which is a bit of 
a waste and not very easy to administer. It would be nice if I could use 
"IP,protocol", "port,protocol", "IP,port,protocol", 
"IP,port,IP,protocol", "IP,port,IP/cidr,protocol" and so forth.

Re: ipporthash, ipportiphash, ipportnethash problems

[Jozsef Kadlecsik]

On Sat, 2 Oct 2010, Mr Dash Four wrote:

> > Thanks for the complete description - I have just released ipset 4.4 which
> > fixes this nasty bug.
> >   
> No worries - I was surprised to find it as ipporthash in particular should
> have been widely used in practice. 

I'm baffled too for not discovering the bug much much earlier, by myself 
or by somebody else.

> As for ipportiphash and ipportnethash - there were recently included in 
> the distribution due to another bug I found when compiling xtables (even 
> though ipportiphash and ipportnethash were compiled the .ko files were 
> never included in the final release making it impossible to use these 
> two construct types).
> 
> A couple of questions and ideas for future releases:
> 
> 1. Normally I used to get ipset as part of the xtables source. Is that still
> the case or do I have to download the ipset source and compile it separately
> to get the latest changes you made?

It's absolutely up to you: currently ipset can be installed alone, or as a 
part of xtables-addons. Jan updates xtables-addons quite reqularly.

> 2. Related to that question - I used to get the xtables .src.rpm from which to
> compile the code and make an installation rpm. I don't seem to be able to do
> that any longer as all fedora repos are too slow to update the latest changes.
> Do you have a link from which I could get the latest changes in ipset/xtables
> in src.rpm which allows me to build an installation rpm. The reason I am
> asking this is because I am building an update images on all my machines
> (using kickstart) and need the installation rpms with the latest changes you
> made. Compiling this from source won't be possible as there is now way I could
> install this as part of the image-building process.

No, I'm not aware of the source rpm for ipset. That's bad if there's one
out there without refreshed. 
 
> I think you mentioned that in future releases the restrictions on all *hash
> constructs (with regards to using /16 - B class - IP addresses) will be
> removed. Could you confirm that is still the case?

Yes, definitely.

> Would it be possible to include a new construct - IP,port,IP,port (without IP
> address restriction) - and how difficult is this to implement?

It's not difficult, but I see little value in such a type. The client 
source port (except for a few protocols in some cases) is random, so how 
could it be useful to mach for that? Or do you know a case where it can 
still be useful?

> Similar question - would it be possible to include protocol (either by name or
> its number) as part of all available constructs for matching? The reason for
> this is that, as it stands, in order for me to match, say, "IP,port" hosts
> form my personal VPN I need to use two separate sets - one for UDP and one for
> TCP and I also to create two separate statements for iptables (one for tcp and
> one for udp), which is a bit of a waste and not very easy to administer. It
> would be nice if I could use "IP,protocol", "port,protocol",
> "IP,port,protocol", "IP,port,IP,protocol", "IP,port,IP/cidr,protocol" and so
> forth.

The next branch includes exactly that, i.e. the possibility to specify 
proto:port for the ipport, ipportip and ipportnet types. And IPv6 support 
too.

Best regards,
Jozsef
-
E-mail  : kadlec@blackhole.kfki.hu, kadlec@mail.kfki.hu
PGP key : http://www.kfki.hu/~kadlec/pgp_public_key.txt
Address : KFKI Research Institute for Particle and Nuclear Physics
          H-1525 Budapest 114, POB. 49, Hungary

Re: ipporthash, ipportiphash, ipportnethash problems

[Mr Dash Four]

>> As for ipportiphash and ipportnethash - there were recently included in 
>> the distribution due to another bug I found when compiling xtables (even 
>> though ipportiphash and ipportnethash were compiled the .ko files were 
>> never included in the final release making it impossible to use these 
>> two construct types).
>>
>> A couple of questions and ideas for future releases:
>>
>> 1. Normally I used to get ipset as part of the xtables source. Is that still
>> the case or do I have to download the ipset source and compile it separately
>> to get the latest changes you made?
>>     
>
> It's absolutely up to you: currently ipset can be installed alone, or as a 
> part of xtables-addons. Jan updates xtables-addons quite reqularly.
>   
I just taken the latest git from xtables-addons and it is not there.

>> 2. Related to that question - I used to get the xtables .src.rpm from which to
>> compile the code and make an installation rpm. I don't seem to be able to do
>> that any longer as all fedora repos are too slow to update the latest changes.
>> Do you have a link from which I could get the latest changes in ipset/xtables
>> in src.rpm which allows me to build an installation rpm. The reason I am
>> asking this is because I am building an update images on all my machines
>> (using kickstart) and need the installation rpms with the latest changes you
>> made. Compiling this from source won't be possible as there is now way I could
>> install this as part of the image-building process.
>>     
>
> No, I'm not aware of the source rpm for ipset. That's bad if there's one
> out there without refreshed. 
>   
This is a major headache for me for 2 reasons:

1. I use 'rpmbuild' to compile from source as it allows me to build rpms 
for different architectures (I compile on x86_64 for i686 with 'rpmbuild 
--target=xxx'). If I use 'ordinary' make there is no way (at least I 
don't know of any) I could compile for different architecture.

2. rpmbuild also allow me to pack the necessary binaries (executables & 
modules/.ko files) for the target architecture to be transferred to the 
target machine - conveniently in rpm which could be installed (together 
with the necessary dependencies) as part of the target image building 
process.

Even if I solve problem No 2 (i.e. by tar-ing the necessary modules and 
executables) I can't see a way to get past compiling for a different 
architecture (and resolve all the dependencies)! Any ideas?

On a side note, I just managed to built v4.4. in the last hour or so for 
x86_64 (haven't tested it yet) and had to readjust the prefix & lib 
directories in the Makefile (I use prefix=/ and libdir=/lib64) and 
installed the new version, but I can't do anything for my other machines 
(which are mostly i686-based and their image is built using kickstart).


>> Would it be possible to include a new construct - IP,port,IP,port (without IP
>> address restriction) - and how difficult is this to implement?
>>     
>
> It's not difficult, but I see little value in such a type. The client 
> source port (except for a few protocols in some cases) is random, so how 
> could it be useful to mach for that? Or do you know a case where it can 
> still be useful?
>   
I can give you of at least 2 uses based on my experience:

1. voip - normally I can restrict the range of source ports and ip 
address when a voip connection is initialed (usually ports 8000-8050 and 
I have to also define the right ip address to point to the outside 
network), but the only way I can do this at present is if I base this on 
UID in iptables (i.e. the id of the process this is run under) - I 
cannot use sets at all because I do not have the appropriate construct. 
If I had "IP,port,IP,port (and protocol!)" as a construct to work with I 
can just built the sets (as I use 4-5 different servers outside the 
network and they work on different ports too) and that would be it!

2. VPN - for the same reason: to increase security I normally bind the 
source port to a particular range and then if I had a "IP,port,IP,port 
(and protocol!)" construct to work with I will not be using UID in 
iptables to match, but will use this construct directly without any 
further need to do anything!

>> Similar question - would it be possible to include protocol (either by name or
>> its number) as part of all available constructs for matching? The reason for
>> this is that, as it stands, in order for me to match, say, "IP,port" hosts
>> form my personal VPN I need to use two separate sets - one for UDP and one for
>> TCP and I also to create two separate statements for iptables (one for tcp and
>> one for udp), which is a bit of a waste and not very easy to administer. It
>> would be nice if I could use "IP,protocol", "port,protocol",
>> "IP,port,protocol", "IP,port,IP,protocol", "IP,port,IP/cidr,protocol" and so
>> forth.
>>     
>
> The next branch includes exactly that, i.e. the possibility to specify 
> proto:port for the ipport, ipportip and ipportnet types. And IPv6 support 
> too.
>   
That's brilliant news! I take it you will be introducing protocol 
support for all the constructs, is that right? How long would it take 
before you release this?

Re: ipporthash, ipportiphash, ipportnethash problems

[Jan Engelhardt]

On Saturday 2010-10-02 22:08, Mr Dash Four wrote:
>>
>>It's absolutely up to you: currently ipset can be installed alone,
>>or as a part of xtables-addons. Jan updates xtables-addons quite
>>reqularly.
>
>I just taken the latest git from xtables-addons and it is not there.

Well give me a little leeway. It hasn't even been 24 hours since 4.4.

>>>2. I used to get the xtables .src.rpm from which to compile the
>>>code and make an installation rpm. I don't seem to be able to do
>>>that any longer as all fedora repos are too slow to update the
>>>latest changes. Do you have a link from which I could get the
>>>latest changes in ipset/xtables in src.rpm.

What is sad too is that Fedora has had so far no interest (or perhaps
just no maintainer) to take it up into the distro.

>>No, I'm not aware of the source rpm for ipset. That's bad if there's one
>>out there without refreshed.   
>
>This is a major headache for me for 2 reasons:
>
>1. I use 'rpmbuild' to compile from source as it allows me to build
>rpms for different architectures (I compile on x86_64 for i686 with
>'rpmbuild --target=xxx'). If I use 'ordinary' make there is no way
>(at least I don't know of any) I could compile for different
>architecture.

>Even if I solve problem No 2 (i.e. by tar-ing the necessary modules and
>executables) I can't see a way to get past compiling for a different
>architecture (and resolve all the dependencies)! Any ideas?

You could use the openSUSE Build Service to produce packages for
various distributions (including Fedora) and the two archs. It should
be possible to include Fedora-specific blocks in the preexisting
xtables-addons.spec too.

>2. rpmbuild also allow me to pack the necessary binaries
>(executables & modules/.ko files) for the target architecture to be
>transferred to the target machine - conveniently in rpm which could
>be installed (together with the necessary dependencies) as part of
>the target image building process.

Yeah but rpmbuild is often run in a jailed environment (for various
definitions of "jail"; ranging from simple chroot to Xen or KVM
instances), and network operations don't exactly contribute to the
idempotency of the build process that is sought.

But that should not really matter, because after a repo is done
building, you can use the regular system tools (zypper, yum) to
update, so that rpmbuild itself needs no network activity.

Re: ipporthash, ipportiphash, ipportnethash problems

[Mr Dash Four]

> Well give me a little leeway. It hasn't even been 24 hours since 4.4.
>   
Well, I know you are busy working on the latest nf_conntrack patches 
so... (-;


> What is sad too is that Fedora has had so far no interest (or perhaps
> just no maintainer) to take it up into the distro.
>   
I won't miss it that as much if I could prepare the rpms myself from 
source (via make with dist-rpm target for example) - I can live with that.


>> Even if I solve problem No 2 (i.e. by tar-ing the necessary modules and
>> executables) I can't see a way to get past compiling for a different
>> architecture (and resolve all the dependencies)! Any ideas?
>>     
>
> You could use the openSUSE Build Service to produce packages for
> various distributions (including Fedora) and the two archs. It should
> be possible to include Fedora-specific blocks in the preexisting
> xtables-addons.spec too.
>   
I have a policy that I built all system-sensitive packages from source 
on one or two of my machines locally, so doing that externally isn't 
really an option for me.


> But that should not really matter, because after a repo is done
> building, you can use the regular system tools (zypper, yum) to
> update, so that rpmbuild itself needs no network activity.
>   
If I could compile and, at least, build the binaries in one single 
package (be it rpm, tar.gz or similar) so that they can be installed 
during my kickstart I would be happy with that.

As a last option I could compile the source on i686 environment (for 
i686 machine to be installed on), but I definitely need a complete 
package (in rpm, tar.gz etc - as I already pointed out) so that I could 
transfer it for my kickstart to unpack and install when building the 
update image. That's my dilemma at present.

Re: ipporthash, ipportiphash, ipportnethash problems

[Jan Engelhardt]

On Saturday 2010-10-02 22:54, Mr Dash Four wrote:
>>Well give me a little leeway. It hasn't even been 24 hours since 4.4.
>
>Well, I know you are busy working on the latest nf_conntrack patches
>so... (-;

Am I?

>>What is sad too is that Fedora has had so far no interest (or
>>perhaps just no maintainer) to take it up into the distro.
>
>I won't miss it that as much if I could prepare the rpms myself from
>source (via make with dist-rpm target for example) - I can live with
>that.

Talk Fedora into it and become a maintainer :p

>>You could use the openSUSE Build Service to produce packages for
>>various distributions (including Fedora) and the two archs. It
>>should be possible to include Fedora-specific blocks in the
>>preexisting xtables-addons.spec too.
>
>I have a policy that I built all system-sensitive packages from
>source on one or two of my machines locally, so doing that
>externally isn't really an option for me.

You can put up a local BS instance. Of course, the same prerequisite
as for raw building applies - you need some x86_64 box to begin
building x86_64.

Re: ipporthash, ipportiphash, ipportnethash problems

[Jozsef Kadlecsik]

On Sat, 2 Oct 2010, Mr Dash Four wrote:

> > No, I'm not aware of the source rpm for ipset. That's bad if there's one
> > out there without refreshed.   
> This is a major headache for me for 2 reasons:
[...]

Sorry, what I provide is a generic, distribution-independent package. I'm 
aware that this can create a maintenance problem in a 
distribution-dependent environment, but I cannot help at that.

> > > Would it be possible to include a new construct - IP,port,IP,port (without
> > > IP
> > > address restriction) - and how difficult is this to implement?
> > 
> > It's not difficult, but I see little value in such a type. The client source
> > port (except for a few protocols in some cases) is random, so how could it
> > be useful to mach for that? Or do you know a case where it can still be
> > useful?
> >   
> I can give you of at least 2 uses based on my experience:
> 
> 1. voip - normally I can restrict the range of source ports and ip address
> when a voip connection is initialed (usually ports 8000-8050 and I have to
> also define the right ip address to point to the outside network), but the
> only way I can do this at present is if I base this on UID in iptables (i.e.
> the id of the process this is run under) - I cannot use sets at all because I
> do not have the appropriate construct. If I had "IP,port,IP,port (and
> protocol!)" as a construct to work with I can just built the sets (as I use
> 4-5 different servers outside the network and they work on different ports
> too) and that would be it!
> 
> 2. VPN - for the same reason: to increase security I normally bind the source
> port to a particular range and then if I had a "IP,port,IP,port (and
> protocol!)" construct to work with I will not be using UID in iptables to
> match, but will use this construct directly without any further need to do
> anything!

The present 4.x branch is in "maintenance" mode for me. I'll think on 
adding such a type to 5.x.

> > The next branch includes exactly that, i.e. the possibility to specify
> > proto:port for the ipport, ipportip and ipportnet types. And IPv6 support
> > too.
> >   
> That's brilliant news! I take it you will be introducing protocol support for
> all the constructs, is that right? How long would it take before you release
> this?

I'm going to release ipset 5.0 around the netfilter developer workshop 
this month.

Best regards,
Jozsef
-
E-mail  : kadlec@blackhole.kfki.hu, kadlec@mail.kfki.hu
PGP key : http://www.kfki.hu/~kadlec/pgp_public_key.txt
Address : KFKI Research Institute for Particle and Nuclear Physics
          H-1525 Budapest 114, POB. 49, Hungary

Re: ipporthash, ipportiphash, ipportnethash problems

[Mr Dash Four]

>> This is a major headache for me for 2 reasons:
>>     
>
> Sorry, what I provide is a generic, distribution-independent package. I'm 
> aware that this can create a maintenance problem in a 
> distribution-dependent environment, but I cannot help at that.
>   
I have managed to find a solution, but it is pretty ugly! I can now 
package the compiled files (from BUILDROOT) into rpm, though what I will 
work on when I next have the time for it is to get the compilation 
process to execute in arch-independent environment. I will also 
fine-tune the rpm spec file and post it here so that whoever is 
interested in packaging xtables+ipset into rpm can use this file to 
prepare rpms instead of relying on the people from fedora who 'maintain' 
the repos to do it (I am still waiting for the 1.29 rpms to show up on 
fedora updates which is a disgrace really)!


>> I can give you of at least 2 uses based on my experience:
>>
>>     
> The present 4.x branch is in "maintenance" mode for me. I'll think on 
> adding such a type to 5.x.
>   
If I can help you out with some testing I would gladly do it.

Another feature you may add to your list is support for port ranges in a 
single set element, like "IP,port-port" for example. You already have 
similar support for multiple IP addresses (when subnets are used) - port 
ranges is another useful feature to have. One example where I can use 
this is when defining 'high-' (or unprivileged) ports - currently I 
'solve' this particular problem with enrolling a set consisting of 1024 
elements containing ports 0-1023 and then specifying a negative match 
(i.e. not privileged) on that set, which is not very convenient.

>> That's brilliant news! I take it you will be introducing protocol support for
>> all the constructs, is that right? How long would it take before you release
>> this?
>>     
>
> I'm going to release ipset 5.0 around the netfilter developer workshop 
> this month.
>   
Superb news!

Re: ipporthash, ipportiphash, ipportnethash problems

[Mr Dash Four]

OK, I quickly tested the new version with my previous test on the ssh 
port and it all works now, though I found some other (minor) issues:

When I do 'make IP_NF_SET_MAX=xx IP_NF_SET_HASHSIZE=xx' and then do 
'make install' (as root) both values I have given during my first 'make' 
were completely disregarded.

If I do 'make IP_NF_SET_MAX=xx IP_NF_SET_HASHSIZE=xx' and 'make 
IP_NF_SET_MAX=xx IP_NF_SET_HASHSIZE=xx install' I can see that at least 
the default hash size has also been disregarded (built a set without 
specifying --hashsize and when I executed 'ipset -L' the hashsize was 
1024 and not the number I have given during make). I cannot check 
whether IP_NF_SET_MAX has also been disregarded (don't know how to do it!).


Is this a known issue or have I done something wrong?

Re: ipporthash, ipportiphash, ipportnethash problems

[Jozsef Kadlecsik]

On Sat, 2 Oct 2010, Mr Dash Four wrote:

> OK, I quickly tested the new version with my previous test on the ssh port and
> it all works now, though I found some other (minor) issues:
> 
> When I do 'make IP_NF_SET_MAX=xx IP_NF_SET_HASHSIZE=xx' and then do 'make
> install' (as root) both values I have given during my first 'make' were
> completely disregarded.
> 
> If I do 'make IP_NF_SET_MAX=xx IP_NF_SET_HASHSIZE=xx' and 'make
> IP_NF_SET_MAX=xx IP_NF_SET_HASHSIZE=xx install' I can see that at least the
> default hash size has also been disregarded (built a set without specifying
> --hashsize and when I executed 'ipset -L' the hashsize was 1024 and not the
> number I have given during make). I cannot check whether IP_NF_SET_MAX has
> also been disregarded (don't know how to do it!).

IP_NF_SET_MAX is honored. Run "make ... V=1" and you see that the value is 
passed to CONFIG_IP_NF_SET_MAX which is used up directly in 
kernel/ip_set.c. However, IP_NF_SET_HASHSIZE is forgotten and not 
referenced in the userspace code, so setting the default hashsize at 
compiling time does not work yet.

Best regards,
Jozsef
-
E-mail  : kadlec@blackhole.kfki.hu, kadlec@mail.kfki.hu
PGP key : http://www.kfki.hu/~kadlec/pgp_public_key.txt
Address : KFKI Research Institute for Particle and Nuclear Physics
          H-1525 Budapest 114, POB. 49, Hungary

Re: ipporthash, ipportiphash, ipportnethash problems

[Mr Dash Four]

> IP_NF_SET_MAX is honored. Run "make ... V=1" and you see that the value is 
> passed to CONFIG_IP_NF_SET_MAX which is used up directly in 
> kernel/ip_set.c. However, IP_NF_SET_HASHSIZE is forgotten and not 
> referenced in the userspace code, so setting the default hashsize at 
> compiling time does not work yet.
>   
So I should not use IP_NF_SET_HASHSIZE for the time being until it is 
fixed, right?

Re: ipporthash, ipportiphash, ipportnethash problems

[Jozsef Kadlecsik]

On Sun, 3 Oct 2010, Mr Dash Four wrote:

> > IP_NF_SET_MAX is honored. Run "make ... V=1" and you see that the value is
> > passed to CONFIG_IP_NF_SET_MAX which is used up directly in kernel/ip_set.c.
> > However, IP_NF_SET_HASHSIZE is forgotten and not referenced in the userspace
> > code, so setting the default hashsize at compiling time does not work yet.
> >   
> So I should not use IP_NF_SET_HASHSIZE for the time being until it is fixed,
> right?

Yes, because currently it's ignored. In this week I'm going to fix it in 
the git repository but won't release a new version just for this.

Best regards,
Jozsef
-
E-mail  : kadlec@blackhole.kfki.hu, kadlec@mail.kfki.hu
PGP key : http://www.kfki.hu/~kadlec/pgp_public_key.txt
Address : KFKI Research Institute for Particle and Nuclear Physics
          H-1525 Budapest 114, POB. 49, Hungary

Re: ipporthash, ipportiphash, ipportnethash problems

[Mr Dash Four]

>> So I should not use IP_NF_SET_HASHSIZE for the time being until it is fixed,
>> right?
>>     
>
> Yes, because currently it's ignored. In this week I'm going to fix it in 
> the git repository but won't release a new version just for this.
>   
There is another issue I found when using ipmap sets: when I execute, 
for example, "ipset -N port-map ipmap --from 10 --to 30000" and then add 
an element "ipset -A port-map 20" the two statements are accepted 
without any error given (they shouldn't be as the map defined is an IP 
map, not a port map and the values submitted are numbers, not IP 
addresses)!

When I issue "ipset -L port-map" I get:

Header: from: 0.0.0.10 to: 0.0.117.48
members:
0.0.0.20

Is this deliberate or a bug?

Re: ipporthash, ipportiphash, ipportnethash problems

[Mike Wright]

Mr Dash Four wrote:
> 
>>> So I should not use IP_NF_SET_HASHSIZE for the time being until it is 
>>> fixed,
>>> right?
>>>     
>>
>> Yes, because currently it's ignored. In this week I'm going to fix it 
>> in the git repository but won't release a new version just for this.
>>   
> There is another issue I found when using ipmap sets: when I execute, 
> for example, "ipset -N port-map ipmap --from 10 --to 30000" and then add 
> an element "ipset -A port-map 20" the two statements are accepted 
> without any error given (they shouldn't be as the map defined is an IP 
> map, not a port map and the values submitted are numbers, not IP 
> addresses)!
> 
> When I issue "ipset -L port-map" I get:
> 
> Header: from: 0.0.0.10 to: 0.0.117.48
> members:
> 0.0.0.20
> 
> Is this deliberate or a bug?

It's deliberate.

The numbers are treated as modulo 256.

30,000 / 256 = 117.1875
.1875 * 256 = 48

Thus the whole number parts become segments of the quad: 117.48

Re: ipporthash, ipportiphash, ipportnethash problems

[Mr Dash Four]

>>
>> Is this deliberate or a bug?
>
> It's deliberate.
>
> The numbers are treated as modulo 256.
>
> 30,000 / 256 = 117.1875
> .1875 * 256 = 48
>
> Thus the whole number parts become segments of the quad: 117.48
Right, thanks for clearing that out - I now know how to convert my 
ip-to-country numbers to IP addresses ;-)

Re: ipporthash, ipportiphash, ipportnethash problems

[Jozsef Kadlecsik]

On Wed, 6 Oct 2010, Mr Dash Four wrote:

> There is another issue I found when using ipmap sets: when I execute, for
> example, "ipset -N port-map ipmap --from 10 --to 30000" and then add an
> element "ipset -A port-map 20" the two statements are accepted without any
> error given (they shouldn't be as the map defined is an IP map, not a port map
> and the values submitted are numbers, not IP addresses)!
> 
> When I issue "ipset -L port-map" I get:
> 
> Header: from: 0.0.0.10 to: 0.0.117.48
> members:
> 0.0.0.20
> 
> Is this deliberate or a bug?

Whatever name has got the set, the type what counts. So if it is defined 
as an ipmap type, then the system tries to interpret the input parameters 
as IP addresses. "10" and "20", etc are strange-looking, but valid IP 
addresses :-)

Best regards,
Jozsef
-
E-mail  : kadlec@blackhole.kfki.hu, kadlec@mail.kfki.hu
PGP key : http://www.kfki.hu/~kadlec/pgp_public_key.txt
Address : KFKI Research Institute for Particle and Nuclear Physics
          H-1525 Budapest 114, POB. 49, Hungary