ipset: stops working after a while

ipset: stops working after a while

[Aidas Kasparas]

Hi,

I have a strange problem with iptables/ipset. I'm under 80Mbps+ SYN
flood (if to believe hoster's monitoring). Attack is not sofisticated --
SYN packet's source address is [pseudo]random, spread more or less
evenly over all available IPv4 address space (even block E). Idea was to
drop first SYN packet from every IP address I haven't seen in past 10
secs. So, I set up the following setup:


iptables -N firstdrop

iptables -N fd_0; iptables -A firstdrop -s 0.0.0.0/5 -j fd_0
iptables -N fd_8; iptables -A firstdrop -s 8.0.0.0/5 -j fd_8
...
iptables -N fd_216; iptables -A firstdrop -s 216.0.0.0/5 -j fd_216

ipset create fd_1 hash:ip timeout 10
ipset create fd_2 hash:ip timeout 10
...
ipset create fd_223 hash:ip timeout 10

iptables -A fd_0 -s 1.0.0.0/8 -m set --match-set fd_1 src -j RETURN;
iptables -A fd_0 -s 1.0.0.0/8 -j SET --add-set fd_1 src;
iptables -A fd_0 -s 2.0.0.0/8 -m set --match-set fd_2 src -j RETURN;
iptables -A fd_0 -s 2.0.0.0/8 -j SET --add-set fd_2 src;
...
iptables -A fd_0 -s 7.0.0.0/8 -m set --match-set fd_7 src -j RETURN;
iptables -A fd_0 -s 7.0.0.0/8 -j SET --add-set fd_7 src;
iptables -A fd_8 -s 8.0.0.0/8 -m set --match-set fd_8 src -j RETURN;
iptables -A fd_8 -s 8.0.0.0/8 -j SET --add-set fd_8 src;
...
iptables -A fd_216 -s 223.0.0.0/8 -m set --match-set fd_223 src -j RETURN;
iptables -A fd_216 -s 223.0.0.0/8 -j SET --add-set fd_223 src;

iptables -A fd_0 -j DROP
iptables -A fd_8 -j DROP
...
iptables -A fd_216 -j DROP

iptables -A INPUT -s admin's_IP -j ACCEPT
iptables -A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
iptables -A INPUT -j firstdrop


That works. Legitimate connections goes through. Outgoing traffic drops
to few megabits, as required for normal operations.


But, after a while (sometimes day, sometimes few hours) no IP addresses
are added to fd_$i's by these iptables rules. And all the packets become
DROP'ed. But if I try to add address by ipset command, or by different
iptables rule -- then these addresses are added to ipsets (not sure, if
that would work for any and all of them).

Where should I search for the problem?

kernel 3.2.0-24.39 from Ubuntu 12.04, x86_64
ipset 6.11-2 (Ubuntu)
when not working sets looks like this:

# ipset list fd_88
Name: fd_88
Type: hash:ip
Header: family inet hashsize 1024 maxelem 65536 timeout 10
Size in memory: 82040
References: 3
Members:
#

Third reference is for testing iptables rules, which adds test ip
address to every fd_i, to be checked if adding works.

-- 
Aidas Kasparas
IT administrator
GM Consult Group, UAB

+370 686 08473
http://www.gmc.lt

Re: ipset: stops working after a while

[Jozsef Kadlecsik]

On Thu, 7 Jun 2012, Aidas Kasparas wrote:

> I have a strange problem with iptables/ipset. I'm under 80Mbps+ SYN
> flood (if to believe hoster's monitoring). Attack is not sofisticated --
> SYN packet's source address is [pseudo]random, spread more or less
> evenly over all available IPv4 address space (even block E). Idea was to
> drop first SYN packet from every IP address I haven't seen in past 10
> secs. So, I set up the following setup:
> 
> iptables -N firstdrop
> 
> iptables -N fd_0; iptables -A firstdrop -s 0.0.0.0/5 -j fd_0
> iptables -N fd_8; iptables -A firstdrop -s 8.0.0.0/5 -j fd_8
> ...
> iptables -N fd_216; iptables -A firstdrop -s 216.0.0.0/5 -j fd_216
> 
> ipset create fd_1 hash:ip timeout 10
> ipset create fd_2 hash:ip timeout 10
> ...
> ipset create fd_223 hash:ip timeout 10
> 
> iptables -A fd_0 -s 1.0.0.0/8 -m set --match-set fd_1 src -j RETURN;
> iptables -A fd_0 -s 1.0.0.0/8 -j SET --add-set fd_1 src;
> iptables -A fd_0 -s 2.0.0.0/8 -m set --match-set fd_2 src -j RETURN;
> iptables -A fd_0 -s 2.0.0.0/8 -j SET --add-set fd_2 src;
> ...
> iptables -A fd_0 -s 7.0.0.0/8 -m set --match-set fd_7 src -j RETURN;
> iptables -A fd_0 -s 7.0.0.0/8 -j SET --add-set fd_7 src;
> iptables -A fd_8 -s 8.0.0.0/8 -m set --match-set fd_8 src -j RETURN;
> iptables -A fd_8 -s 8.0.0.0/8 -j SET --add-set fd_8 src;
> ...
> iptables -A fd_216 -s 223.0.0.0/8 -m set --match-set fd_223 src -j RETURN;
> iptables -A fd_216 -s 223.0.0.0/8 -j SET --add-set fd_223 src;
> 
> iptables -A fd_0 -j DROP
> iptables -A fd_8 -j DROP
> ...
> iptables -A fd_216 -j DROP
> 
> iptables -A INPUT -s admin's_IP -j ACCEPT
> iptables -A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
> iptables -A INPUT -j firstdrop
> 
> That works. Legitimate connections goes through. Outgoing traffic drops
> to few megabits, as required for normal operations.
> 
> But, after a while (sometimes day, sometimes few hours) no IP addresses
> are added to fd_$i's by these iptables rules. And all the packets become
> DROP'ed. But if I try to add address by ipset command, or by different
> iptables rule -- then these addresses are added to ipsets (not sure, if
> that would work for any and all of them).

Maybe your given set gets full. From the manpage:

"When  entries  added  by the SET target of iptables/ip6tables, then the
hash size is fixed and the set won't be duplicated,  even  if  the  new
entry cannot be added to the set."

Best regards,
Jozsef
-
E-mail  : kadlec@blackhole.kfki.hu, kadlecsik.jozsef@wigner.mta.hu
PGP key : http://www.kfki.hu/~kadlec/pgp_public_key.txt
Address : Wigner Research Centre for Physics, Hungarian Academy of Sciences
          H-1525 Budapest 114, POB. 49, Hungary

Re: ipset: stops working after a while

[Aidas Kasparas]

On 2012.06.07 09:59, Jozsef Kadlecsik wrote:
> On Thu, 7 Jun 2012, Aidas Kasparas wrote:
> 
>> I have a strange problem with iptables/ipset. I'm under 80Mbps+ SYN
>> flood (if to believe hoster's monitoring). Attack is not sofisticated --
>> SYN packet's source address is [pseudo]random, spread more or less
>> evenly over all available IPv4 address space (even block E). Idea was to
>> drop first SYN packet from every IP address I haven't seen in past 10
>> secs. So, I set up the following setup:
>>
>> iptables -N firstdrop
>>
>> iptables -N fd_0; iptables -A firstdrop -s 0.0.0.0/5 -j fd_0
>> iptables -N fd_8; iptables -A firstdrop -s 8.0.0.0/5 -j fd_8
>> ...
>> iptables -N fd_216; iptables -A firstdrop -s 216.0.0.0/5 -j fd_216
>>
>> ipset create fd_1 hash:ip timeout 10
>> ipset create fd_2 hash:ip timeout 10
>> ...
>> ipset create fd_223 hash:ip timeout 10
>>
>> iptables -A fd_0 -s 1.0.0.0/8 -m set --match-set fd_1 src -j RETURN;
>> iptables -A fd_0 -s 1.0.0.0/8 -j SET --add-set fd_1 src;
>> iptables -A fd_0 -s 2.0.0.0/8 -m set --match-set fd_2 src -j RETURN;
>> iptables -A fd_0 -s 2.0.0.0/8 -j SET --add-set fd_2 src;
>> ...
>> iptables -A fd_0 -s 7.0.0.0/8 -m set --match-set fd_7 src -j RETURN;
>> iptables -A fd_0 -s 7.0.0.0/8 -j SET --add-set fd_7 src;
>> iptables -A fd_8 -s 8.0.0.0/8 -m set --match-set fd_8 src -j RETURN;
>> iptables -A fd_8 -s 8.0.0.0/8 -j SET --add-set fd_8 src;
>> ...
>> iptables -A fd_216 -s 223.0.0.0/8 -m set --match-set fd_223 src -j RETURN;
>> iptables -A fd_216 -s 223.0.0.0/8 -j SET --add-set fd_223 src;
>>
>> iptables -A fd_0 -j DROP
>> iptables -A fd_8 -j DROP
>> ...
>> iptables -A fd_216 -j DROP
>>
>> iptables -A INPUT -s admin's_IP -j ACCEPT
>> iptables -A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
>> iptables -A INPUT -j firstdrop
>>
>> That works. Legitimate connections goes through. Outgoing traffic drops
>> to few megabits, as required for normal operations.
>>
>> But, after a while (sometimes day, sometimes few hours) no IP addresses
>> are added to fd_$i's by these iptables rules. And all the packets become
>> DROP'ed. But if I try to add address by ipset command, or by different
>> iptables rule -- then these addresses are added to ipsets (not sure, if
>> that would work for any and all of them).
> 
> Maybe your given set gets full. From the manpage:
> 
> "When  entries  added  by the SET target of iptables/ip6tables, then the
> hash size is fixed and the set won't be duplicated,  even  if  the  new
> entry cannot be added to the set."
> 

Ok. But if set is full, and I list it, it should show at least some
members present. When it stops working, it shows no members at all.

On the other hand, I create sets with timeout 10. So, every 3 secs there
should be expiration process which trows ~ 1/3 of entries from each
chain. And this should make place for some new entries.


-- 
Aidas Kasparas
IT administrator
GM Consult Group, UAB

+370 686 08473
http://www.gmc.lt

Re: ipset: stops working after a while

[Neal Murphy]

On Thursday 07 June 2012 13:43:25 Aidas Kasparas wrote:
> On 2012.06.07 09:59, Jozsef Kadlecsik wrote:
> > Maybe your given set gets full. From the manpage:
> > 
> > "When  entries  added  by the SET target of iptables/ip6tables, then the
> > hash size is fixed and the set won't be duplicated,  even  if  the  new
> > entry cannot be added to the set."
> 
> Ok. But if set is full, and I list it, it should show at least some
> members present. When it stops working, it shows no members at all.
> 
> On the other hand, I create sets with timeout 10. So, every 3 secs there
> should be expiration process which trows ~ 1/3 of entries from each
> chain. And this should make place for some new entries.

I'll address *your* problem, not the problem you observed with the ipset code 
(which may be a real problem).

How many entres are in the set when it is 'full'? Have you tried setting the 
initial size of the hash to the maximum (64ki?)?

Utilization of a set ought to be non-deterministic because you can't know how 
many 'new' addresses will arrive in any given time interval. In other words (a 
contrived example), suppose at time T you had 64ki addresses in the hash with 
20,000 set to expire in 3â…“ seconds (the rest later). In the next 3333ms, 2000 
new addresses arrive but can't fitin the set. At T+3333ms, 20k addresses 
expire. In the next 2000ms, 21k new addresses arrive; 1000 of them won't fit 
in the set. 80Mb/s allows for a lot of packets; even slow consumer-grade 
switches can exceed 64ki addresses in 10 seconds via SYN packets.

Given all that, can you redesign your sets so you don't (maybe can't) fill 
them?

Re: ipset: stops working after a while

[Jozsef Kadlecsik]

On Thu, 7 Jun 2012, Neal Murphy wrote:

> On Thursday 07 June 2012 13:43:25 Aidas Kasparas wrote:
> > On 2012.06.07 09:59, Jozsef Kadlecsik wrote:
> > > Maybe your given set gets full. From the manpage:
> > > 
> > > "When  entries  added  by the SET target of iptables/ip6tables, then the
> > > hash size is fixed and the set won't be duplicated,  even  if  the  new
> > > entry cannot be added to the set."
> > 
> > Ok. But if set is full, and I list it, it should show at least some
> > members present. When it stops working, it shows no members at all.
> > 
> > On the other hand, I create sets with timeout 10. So, every 3 secs there
> > should be expiration process which trows ~ 1/3 of entries from each
> > chain. And this should make place for some new entries.
> 
> I'll address *your* problem, not the problem you observed with the ipset code 
> (which may be a real problem).
> 
> How many entres are in the set when it is 'full'? Have you tried setting the 
> initial size of the hash to the maximum (64ki?)?

According to the listing of the set:

# ipset list fd_88
Name: fd_88
Type: hash:ip
Header: family inet hashsize 1024 maxelem 65536 timeout 10
                             ^^^^
Size in memory: 82040
References: 3
Members:

Because the SET target won't trigger increasing the hash size and the 
max collision is limited to 12, that means there can be at the maximum 
12*1024 elements in that set. And that's the theoretical maximum.

By the way the hash size is not limited in ipset 6.x when creating a hash 
type of set.

Best regards,
Jozsef
-
E-mail  : kadlec@blackhole.kfki.hu, kadlecsik.jozsef@wigner.mta.hu
PGP key : http://www.kfki.hu/~kadlec/pgp_public_key.txt
Address : Wigner Research Centre for Physics, Hungarian Academy of Sciences
          H-1525 Budapest 114, POB. 49, Hungary

Re: ipset: stops working after a while

[Aidas Kasparas]

On 2012.06.08 00:22, Neal Murphy wrote:
> On Thursday 07 June 2012 13:43:25 Aidas Kasparas wrote:
>> On 2012.06.07 09:59, Jozsef Kadlecsik wrote:
>>> Maybe your given set gets full. From the manpage:
>>>
>>> "When  entries  added  by the SET target of iptables/ip6tables, then the
>>> hash size is fixed and the set won't be duplicated,  even  if  the  new
>>> entry cannot be added to the set."
>>
>> Ok. But if set is full, and I list it, it should show at least some
>> members present. When it stops working, it shows no members at all.
>>
>> On the other hand, I create sets with timeout 10. So, every 3 secs there
>> should be expiration process which trows ~ 1/3 of entries from each
>> chain. And this should make place for some new entries.
> 
> I'll address *your* problem, not the problem you observed with the ipset code 
> (which may be a real problem).
> 

Thanks. I tried to increase hash sizes, reenable DROPs and found out
that there was an error in the destination IP address of ACCEPT rule,
which had to pass good packets through. System corrected works for
several days now, so I believe the ipset code had no problems which
affected my setup.

> How many entres are in the set when it is 'full'? Have you tried setting the 
> initial size of the hash to the maximum (64ki?)?
> 
> Utilization of a set ought to be non-deterministic because you can't know how 
> many 'new' addresses will arrive in any given time interval. In other words (a 
> contrived example), suppose at time T you had 64ki addresses in the hash with 
> 20,000 set to expire in 3â…“ seconds (the rest later). In the next 3333ms, 2000 
> new addresses arrive but can't fitin the set. At T+3333ms, 20k addresses 
> expire. In the next 2000ms, 21k new addresses arrive; 1000 of them won't fit 
> in the set. 80Mb/s allows for a lot of packets; even slow consumer-grade 
> switches can exceed 64ki addresses in 10 seconds via SYN packets.
> 
> Given all that, can you redesign your sets so you don't (maybe can't) fill 
> them?

Well, lets do the math.
80Mbps is 10MBps.
60 bytes on wire for SYN packet makes 167k SYN packets/s.
I have 222 sets, so, if I'm lucky and attack spreads evenly over all of
them, I have 750 inserts/ set * sec.
In practice, I never seen a set filled with more than 9000 entries.
Maybe monitoring lies and attack is not so big. Maybe small fraction of
packets overflow collision bucket and are not inserted.

Anyway, I upped hash size to 8k and 16k on two different boxes. As in
latter case this can use up to 2.5GB of RAM just for ipsets, and
hardware I'm renting has 4 and 8GB of RAM, I was not able to go higher.

On the other hand I still need to check, how good jhash2 is for my case,
as all the IPs in one set have the same first byte. And how difficult
would be to construct addresses to collide with given IP address of good
guys.

Thanks again.

-- 
Aidas Kasparas
IT administrator
GM Consult Group, UAB

http://www.gmc.lt