* ipset: stops working after a while @ 2012-06-07 5:23 Aidas Kasparas 2012-06-07 6:59 ` Jozsef Kadlecsik 0 siblings, 1 reply; 6+ messages in thread From: Aidas Kasparas @ 2012-06-07 5:23 UTC (permalink / raw) To: netfilter Hi, I have a strange problem with iptables/ipset. I'm under 80Mbps+ SYN flood (if to believe hoster's monitoring). Attack is not sofisticated -- SYN packet's source address is [pseudo]random, spread more or less evenly over all available IPv4 address space (even block E). Idea was to drop first SYN packet from every IP address I haven't seen in past 10 secs. So, I set up the following setup: iptables -N firstdrop iptables -N fd_0; iptables -A firstdrop -s 0.0.0.0/5 -j fd_0 iptables -N fd_8; iptables -A firstdrop -s 8.0.0.0/5 -j fd_8 ... iptables -N fd_216; iptables -A firstdrop -s 216.0.0.0/5 -j fd_216 ipset create fd_1 hash:ip timeout 10 ipset create fd_2 hash:ip timeout 10 ... ipset create fd_223 hash:ip timeout 10 iptables -A fd_0 -s 1.0.0.0/8 -m set --match-set fd_1 src -j RETURN; iptables -A fd_0 -s 1.0.0.0/8 -j SET --add-set fd_1 src; iptables -A fd_0 -s 2.0.0.0/8 -m set --match-set fd_2 src -j RETURN; iptables -A fd_0 -s 2.0.0.0/8 -j SET --add-set fd_2 src; ... iptables -A fd_0 -s 7.0.0.0/8 -m set --match-set fd_7 src -j RETURN; iptables -A fd_0 -s 7.0.0.0/8 -j SET --add-set fd_7 src; iptables -A fd_8 -s 8.0.0.0/8 -m set --match-set fd_8 src -j RETURN; iptables -A fd_8 -s 8.0.0.0/8 -j SET --add-set fd_8 src; ... iptables -A fd_216 -s 223.0.0.0/8 -m set --match-set fd_223 src -j RETURN; iptables -A fd_216 -s 223.0.0.0/8 -j SET --add-set fd_223 src; iptables -A fd_0 -j DROP iptables -A fd_8 -j DROP ... iptables -A fd_216 -j DROP iptables -A INPUT -s admin's_IP -j ACCEPT iptables -A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT iptables -A INPUT -j firstdrop That works. Legitimate connections goes through. Outgoing traffic drops to few megabits, as required for normal operations. But, after a while (sometimes day, sometimes few hours) no IP addresses are added to fd_$i's by these iptables rules. And all the packets become DROP'ed. But if I try to add address by ipset command, or by different iptables rule -- then these addresses are added to ipsets (not sure, if that would work for any and all of them). Where should I search for the problem? kernel 3.2.0-24.39 from Ubuntu 12.04, x86_64 ipset 6.11-2 (Ubuntu) when not working sets looks like this: # ipset list fd_88 Name: fd_88 Type: hash:ip Header: family inet hashsize 1024 maxelem 65536 timeout 10 Size in memory: 82040 References: 3 Members: # Third reference is for testing iptables rules, which adds test ip address to every fd_i, to be checked if adding works. -- Aidas Kasparas IT administrator GM Consult Group, UAB +370 686 08473 http://www.gmc.lt ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: ipset: stops working after a while 2012-06-07 5:23 ipset: stops working after a while Aidas Kasparas @ 2012-06-07 6:59 ` Jozsef Kadlecsik 2012-06-07 17:43 ` Aidas Kasparas 0 siblings, 1 reply; 6+ messages in thread From: Jozsef Kadlecsik @ 2012-06-07 6:59 UTC (permalink / raw) To: Aidas Kasparas; +Cc: netfilter On Thu, 7 Jun 2012, Aidas Kasparas wrote: > I have a strange problem with iptables/ipset. I'm under 80Mbps+ SYN > flood (if to believe hoster's monitoring). Attack is not sofisticated -- > SYN packet's source address is [pseudo]random, spread more or less > evenly over all available IPv4 address space (even block E). Idea was to > drop first SYN packet from every IP address I haven't seen in past 10 > secs. So, I set up the following setup: > > iptables -N firstdrop > > iptables -N fd_0; iptables -A firstdrop -s 0.0.0.0/5 -j fd_0 > iptables -N fd_8; iptables -A firstdrop -s 8.0.0.0/5 -j fd_8 > ... > iptables -N fd_216; iptables -A firstdrop -s 216.0.0.0/5 -j fd_216 > > ipset create fd_1 hash:ip timeout 10 > ipset create fd_2 hash:ip timeout 10 > ... > ipset create fd_223 hash:ip timeout 10 > > iptables -A fd_0 -s 1.0.0.0/8 -m set --match-set fd_1 src -j RETURN; > iptables -A fd_0 -s 1.0.0.0/8 -j SET --add-set fd_1 src; > iptables -A fd_0 -s 2.0.0.0/8 -m set --match-set fd_2 src -j RETURN; > iptables -A fd_0 -s 2.0.0.0/8 -j SET --add-set fd_2 src; > ... > iptables -A fd_0 -s 7.0.0.0/8 -m set --match-set fd_7 src -j RETURN; > iptables -A fd_0 -s 7.0.0.0/8 -j SET --add-set fd_7 src; > iptables -A fd_8 -s 8.0.0.0/8 -m set --match-set fd_8 src -j RETURN; > iptables -A fd_8 -s 8.0.0.0/8 -j SET --add-set fd_8 src; > ... > iptables -A fd_216 -s 223.0.0.0/8 -m set --match-set fd_223 src -j RETURN; > iptables -A fd_216 -s 223.0.0.0/8 -j SET --add-set fd_223 src; > > iptables -A fd_0 -j DROP > iptables -A fd_8 -j DROP > ... > iptables -A fd_216 -j DROP > > iptables -A INPUT -s admin's_IP -j ACCEPT > iptables -A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT > iptables -A INPUT -j firstdrop > > That works. Legitimate connections goes through. Outgoing traffic drops > to few megabits, as required for normal operations. > > But, after a while (sometimes day, sometimes few hours) no IP addresses > are added to fd_$i's by these iptables rules. And all the packets become > DROP'ed. But if I try to add address by ipset command, or by different > iptables rule -- then these addresses are added to ipsets (not sure, if > that would work for any and all of them). Maybe your given set gets full. From the manpage: "When entries added by the SET target of iptables/ip6tables, then the hash size is fixed and the set won't be duplicated, even if the new entry cannot be added to the set." Best regards, Jozsef - E-mail : kadlec@blackhole.kfki.hu, kadlecsik.jozsef@wigner.mta.hu PGP key : http://www.kfki.hu/~kadlec/pgp_public_key.txt Address : Wigner Research Centre for Physics, Hungarian Academy of Sciences H-1525 Budapest 114, POB. 49, Hungary ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: ipset: stops working after a while 2012-06-07 6:59 ` Jozsef Kadlecsik @ 2012-06-07 17:43 ` Aidas Kasparas 2012-06-07 21:22 ` Neal Murphy 0 siblings, 1 reply; 6+ messages in thread From: Aidas Kasparas @ 2012-06-07 17:43 UTC (permalink / raw) To: Jozsef Kadlecsik; +Cc: netfilter On 2012.06.07 09:59, Jozsef Kadlecsik wrote: > On Thu, 7 Jun 2012, Aidas Kasparas wrote: > >> I have a strange problem with iptables/ipset. I'm under 80Mbps+ SYN >> flood (if to believe hoster's monitoring). Attack is not sofisticated -- >> SYN packet's source address is [pseudo]random, spread more or less >> evenly over all available IPv4 address space (even block E). Idea was to >> drop first SYN packet from every IP address I haven't seen in past 10 >> secs. So, I set up the following setup: >> >> iptables -N firstdrop >> >> iptables -N fd_0; iptables -A firstdrop -s 0.0.0.0/5 -j fd_0 >> iptables -N fd_8; iptables -A firstdrop -s 8.0.0.0/5 -j fd_8 >> ... >> iptables -N fd_216; iptables -A firstdrop -s 216.0.0.0/5 -j fd_216 >> >> ipset create fd_1 hash:ip timeout 10 >> ipset create fd_2 hash:ip timeout 10 >> ... >> ipset create fd_223 hash:ip timeout 10 >> >> iptables -A fd_0 -s 1.0.0.0/8 -m set --match-set fd_1 src -j RETURN; >> iptables -A fd_0 -s 1.0.0.0/8 -j SET --add-set fd_1 src; >> iptables -A fd_0 -s 2.0.0.0/8 -m set --match-set fd_2 src -j RETURN; >> iptables -A fd_0 -s 2.0.0.0/8 -j SET --add-set fd_2 src; >> ... >> iptables -A fd_0 -s 7.0.0.0/8 -m set --match-set fd_7 src -j RETURN; >> iptables -A fd_0 -s 7.0.0.0/8 -j SET --add-set fd_7 src; >> iptables -A fd_8 -s 8.0.0.0/8 -m set --match-set fd_8 src -j RETURN; >> iptables -A fd_8 -s 8.0.0.0/8 -j SET --add-set fd_8 src; >> ... >> iptables -A fd_216 -s 223.0.0.0/8 -m set --match-set fd_223 src -j RETURN; >> iptables -A fd_216 -s 223.0.0.0/8 -j SET --add-set fd_223 src; >> >> iptables -A fd_0 -j DROP >> iptables -A fd_8 -j DROP >> ... >> iptables -A fd_216 -j DROP >> >> iptables -A INPUT -s admin's_IP -j ACCEPT >> iptables -A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT >> iptables -A INPUT -j firstdrop >> >> That works. Legitimate connections goes through. Outgoing traffic drops >> to few megabits, as required for normal operations. >> >> But, after a while (sometimes day, sometimes few hours) no IP addresses >> are added to fd_$i's by these iptables rules. And all the packets become >> DROP'ed. But if I try to add address by ipset command, or by different >> iptables rule -- then these addresses are added to ipsets (not sure, if >> that would work for any and all of them). > > Maybe your given set gets full. From the manpage: > > "When entries added by the SET target of iptables/ip6tables, then the > hash size is fixed and the set won't be duplicated, even if the new > entry cannot be added to the set." > Ok. But if set is full, and I list it, it should show at least some members present. When it stops working, it shows no members at all. On the other hand, I create sets with timeout 10. So, every 3 secs there should be expiration process which trows ~ 1/3 of entries from each chain. And this should make place for some new entries. -- Aidas Kasparas IT administrator GM Consult Group, UAB +370 686 08473 http://www.gmc.lt ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: ipset: stops working after a while 2012-06-07 17:43 ` Aidas Kasparas @ 2012-06-07 21:22 ` Neal Murphy 2012-06-08 7:29 ` Jozsef Kadlecsik 2012-06-12 5:24 ` Aidas Kasparas 0 siblings, 2 replies; 6+ messages in thread From: Neal Murphy @ 2012-06-07 21:22 UTC (permalink / raw) To: netfilter On Thursday 07 June 2012 13:43:25 Aidas Kasparas wrote: > On 2012.06.07 09:59, Jozsef Kadlecsik wrote: > > Maybe your given set gets full. From the manpage: > > > > "When entries added by the SET target of iptables/ip6tables, then the > > hash size is fixed and the set won't be duplicated, even if the new > > entry cannot be added to the set." > > Ok. But if set is full, and I list it, it should show at least some > members present. When it stops working, it shows no members at all. > > On the other hand, I create sets with timeout 10. So, every 3 secs there > should be expiration process which trows ~ 1/3 of entries from each > chain. And this should make place for some new entries. I'll address *your* problem, not the problem you observed with the ipset code (which may be a real problem). How many entres are in the set when it is 'full'? Have you tried setting the initial size of the hash to the maximum (64ki?)? Utilization of a set ought to be non-deterministic because you can't know how many 'new' addresses will arrive in any given time interval. In other words (a contrived example), suppose at time T you had 64ki addresses in the hash with 20,000 set to expire in 3â…“ seconds (the rest later). In the next 3333ms, 2000 new addresses arrive but can't fitin the set. At T+3333ms, 20k addresses expire. In the next 2000ms, 21k new addresses arrive; 1000 of them won't fit in the set. 80Mb/s allows for a lot of packets; even slow consumer-grade switches can exceed 64ki addresses in 10 seconds via SYN packets. Given all that, can you redesign your sets so you don't (maybe can't) fill them? ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: ipset: stops working after a while 2012-06-07 21:22 ` Neal Murphy @ 2012-06-08 7:29 ` Jozsef Kadlecsik 2012-06-12 5:24 ` Aidas Kasparas 1 sibling, 0 replies; 6+ messages in thread From: Jozsef Kadlecsik @ 2012-06-08 7:29 UTC (permalink / raw) To: Neal Murphy; +Cc: netfilter On Thu, 7 Jun 2012, Neal Murphy wrote: > On Thursday 07 June 2012 13:43:25 Aidas Kasparas wrote: > > On 2012.06.07 09:59, Jozsef Kadlecsik wrote: > > > Maybe your given set gets full. From the manpage: > > > > > > "When entries added by the SET target of iptables/ip6tables, then the > > > hash size is fixed and the set won't be duplicated, even if the new > > > entry cannot be added to the set." > > > > Ok. But if set is full, and I list it, it should show at least some > > members present. When it stops working, it shows no members at all. > > > > On the other hand, I create sets with timeout 10. So, every 3 secs there > > should be expiration process which trows ~ 1/3 of entries from each > > chain. And this should make place for some new entries. > > I'll address *your* problem, not the problem you observed with the ipset code > (which may be a real problem). > > How many entres are in the set when it is 'full'? Have you tried setting the > initial size of the hash to the maximum (64ki?)? According to the listing of the set: # ipset list fd_88 Name: fd_88 Type: hash:ip Header: family inet hashsize 1024 maxelem 65536 timeout 10 ^^^^ Size in memory: 82040 References: 3 Members: Because the SET target won't trigger increasing the hash size and the max collision is limited to 12, that means there can be at the maximum 12*1024 elements in that set. And that's the theoretical maximum. By the way the hash size is not limited in ipset 6.x when creating a hash type of set. Best regards, Jozsef - E-mail : kadlec@blackhole.kfki.hu, kadlecsik.jozsef@wigner.mta.hu PGP key : http://www.kfki.hu/~kadlec/pgp_public_key.txt Address : Wigner Research Centre for Physics, Hungarian Academy of Sciences H-1525 Budapest 114, POB. 49, Hungary ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: ipset: stops working after a while 2012-06-07 21:22 ` Neal Murphy 2012-06-08 7:29 ` Jozsef Kadlecsik @ 2012-06-12 5:24 ` Aidas Kasparas 1 sibling, 0 replies; 6+ messages in thread From: Aidas Kasparas @ 2012-06-12 5:24 UTC (permalink / raw) To: neal.p.murphy; +Cc: netfilter On 2012.06.08 00:22, Neal Murphy wrote: > On Thursday 07 June 2012 13:43:25 Aidas Kasparas wrote: >> On 2012.06.07 09:59, Jozsef Kadlecsik wrote: >>> Maybe your given set gets full. From the manpage: >>> >>> "When entries added by the SET target of iptables/ip6tables, then the >>> hash size is fixed and the set won't be duplicated, even if the new >>> entry cannot be added to the set." >> >> Ok. But if set is full, and I list it, it should show at least some >> members present. When it stops working, it shows no members at all. >> >> On the other hand, I create sets with timeout 10. So, every 3 secs there >> should be expiration process which trows ~ 1/3 of entries from each >> chain. And this should make place for some new entries. > > I'll address *your* problem, not the problem you observed with the ipset code > (which may be a real problem). > Thanks. I tried to increase hash sizes, reenable DROPs and found out that there was an error in the destination IP address of ACCEPT rule, which had to pass good packets through. System corrected works for several days now, so I believe the ipset code had no problems which affected my setup. > How many entres are in the set when it is 'full'? Have you tried setting the > initial size of the hash to the maximum (64ki?)? > > Utilization of a set ought to be non-deterministic because you can't know how > many 'new' addresses will arrive in any given time interval. In other words (a > contrived example), suppose at time T you had 64ki addresses in the hash with > 20,000 set to expire in 3â…“ seconds (the rest later). In the next 3333ms, 2000 > new addresses arrive but can't fitin the set. At T+3333ms, 20k addresses > expire. In the next 2000ms, 21k new addresses arrive; 1000 of them won't fit > in the set. 80Mb/s allows for a lot of packets; even slow consumer-grade > switches can exceed 64ki addresses in 10 seconds via SYN packets. > > Given all that, can you redesign your sets so you don't (maybe can't) fill > them? Well, lets do the math. 80Mbps is 10MBps. 60 bytes on wire for SYN packet makes 167k SYN packets/s. I have 222 sets, so, if I'm lucky and attack spreads evenly over all of them, I have 750 inserts/ set * sec. In practice, I never seen a set filled with more than 9000 entries. Maybe monitoring lies and attack is not so big. Maybe small fraction of packets overflow collision bucket and are not inserted. Anyway, I upped hash size to 8k and 16k on two different boxes. As in latter case this can use up to 2.5GB of RAM just for ipsets, and hardware I'm renting has 4 and 8GB of RAM, I was not able to go higher. On the other hand I still need to check, how good jhash2 is for my case, as all the IPs in one set have the same first byte. And how difficult would be to construct addresses to collide with given IP address of good guys. Thanks again. -- Aidas Kasparas IT administrator GM Consult Group, UAB http://www.gmc.lt ^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2012-06-12 5:24 UTC | newest] Thread overview: 6+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2012-06-07 5:23 ipset: stops working after a while Aidas Kasparas 2012-06-07 6:59 ` Jozsef Kadlecsik 2012-06-07 17:43 ` Aidas Kasparas 2012-06-07 21:22 ` Neal Murphy 2012-06-08 7:29 ` Jozsef Kadlecsik 2012-06-12 5:24 ` Aidas Kasparas
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.