banning bot ips with ipset

banning bot ips with ipset

[Nigel Heron]

Hi list,
We're using iptables (with shorewall for config) and lvs on our 
firewall/load balancer. We're being attacked by a botnet alternating 
between http request floods and syn floods. We have a way to identify 
the ips that are http'ing and started dropping them in iptables, once we 
got to ~1700 banned ips the server stopped nat'ing completely (not sure 
why..) and we were forced to remove the blacklist. We're now banning non 
north-american class-As to drop half the bots, but it's obviously not a 
good long term solution. We just came across ipset, but the lack of any 
feedback on the net (besides on *.netfilter.org) has us a bit worried 
about real world deployment. Is ipset stable enough to be deployed on 
live environments? iphash seems like the best set type for us, how many 
ips can the set handle before there's a noticeable slowdown? any 
feedback would be appreciated.

obviously, we don't expect ipset to help us with the syn flood, at 
~30Mb/s of syn traffic, syn cookies aren't helping either, is there a 
syn-proxy implementation for linux?

also, if it helps anyone else .. while trying ipset 2.4.5 i had to add:
#include <stdbool.h>
to "kernel/ip_set_setlist.c" to get it to compile.

thanks,

-nigel.

Re: banning bot ips with ipset

[G.W. Haywood]

Hi there,

On Tue, 25 Nov 2008, Nigel Heron wrote:

> We're being attacked by a botnet ... started dropping them in
> iptables, once we got to ~1700 banned ips the server stopped nat'ing
> completely (not sure why..)

It probably just ran out of steam.  The performance of iptables with
thousands of rules can be poor if you don't structure them carefully.

> Is ipset stable enough to be deployed on live environments?

I've been using it for years with absolutely zero problems.

> iphash seems like the best set type for us, how many
> ips can the set handle before there's a noticeable slowdown?

I currently have about 50,000 ipset (iphash) rules on modest hardware,
with no noticeable performance impact.  There's a good report here:

http://people.netfilter.org/kadlec/nftest.pdf

One of the authors also wrote ipset.  He's on this list.

> obviously, we don't expect ipset to help us with the syn flood, at
> ~30Mb/s of syn traffic, syn cookies aren't helping either, is there
> a syn-proxy implementation for linux?

Sorry, never looked at it, but a quick Google for /"syn proxy" linux/
gave me over 1,000 hits.

--

73,
Ged.

Re: banning bot ips with ipset

[Adem]

"G.W. Haywood" wrote:
> On Tue, 25 Nov 2008, Nigel Heron wrote:
> 
> > We're being attacked by a botnet ... started dropping them in
> > iptables, once we got to ~1700 banned ips the server stopped nat'ing
> > completely (not sure why..)
> 
> It probably just ran out of steam.  The performance of iptables with
> thousands of rules can be poor if you don't structure them carefully.
> 
> > Is ipset stable enough to be deployed on live environments?
> 
> I've been using it for years with absolutely zero problems.
> 
> > iphash seems like the best set type for us, how many
> > ips can the set handle before there's a noticeable slowdown?
> 
> I currently have about 50,000 ipset (iphash) rules on modest hardware,
> with no noticeable performance impact.  There's a good report here:
> http://people.netfilter.org/kadlec/nftest.pdf

Do you understand what the authors means with this statement in section 4.2:

  "As the graph displays, the system handled almost 
   3,500,000 concurrent connections at the peak."

I wonder how this is possible... :-)
I think one would need a machine with 54 NIC's (real and/or virtual) attached to it, isn't it? :-)

Re: banning bot ips with ipset

[Jozsef Kadlecsik]

On Tue, 25 Nov 2008, Adem wrote:

> "G.W. Haywood" wrote:
> > On Tue, 25 Nov 2008, Nigel Heron wrote:
> > 
> > > We're being attacked by a botnet ... started dropping them in
> > > iptables, once we got to ~1700 banned ips the server stopped nat'ing
> > > completely (not sure why..)
> > 
> > It probably just ran out of steam.  The performance of iptables with
> > thousands of rules can be poor if you don't structure them carefully.
> > 
> > > Is ipset stable enough to be deployed on live environments?
> > 
> > I've been using it for years with absolutely zero problems.
> > 
> > > iphash seems like the best set type for us, how many
> > > ips can the set handle before there's a noticeable slowdown?
> > 
> > I currently have about 50,000 ipset (iphash) rules on modest hardware,
> > with no noticeable performance impact.  There's a good report here:
> > http://people.netfilter.org/kadlec/nftest.pdf
> 
> Do you understand what the authors means with this statement in section 4.2:
> 
>   "As the graph displays, the system handled almost 
>    3,500,000 concurrent connections at the peak."
> 
> I wonder how this is possible... :-)

It means the number of conntrack entries at the maximum.

> I think one would need a machine with 54 NIC's (real and/or virtual) 
> attached to it, isn't it? :-)

No, it does not imply there was so many packets flowing through at the 
same time. The pps graphs show the maximum number of packets per 
second handled by the system during the tests.

Best regards,
Jozsef
-
E-mail  : kadlec@blackhole.kfki.hu, kadlec@mail.kfki.hu
PGP key : http://www.kfki.hu/~kadlec/pgp_public_key.txt
Address : KFKI Research Institute for Particle and Nuclear Physics
          H-1525 Budapest 114, POB. 49, Hungary

Re: banning bot ips with ipset

[Marek Kierdelewicz]

>Hi list,

Hi,

>obviously not a good long term solution. We just came across ipset, but
>the lack of any feedback on the net (besides on *.netfilter.org) has us
>a bit worried about real world deployment.
> Is ipset stable enough to be deployed on live environments? 

I'm using ipset for customer access control at linux boxes (routers at
ISP company). It's mature, stable and as Mr Haywood said - also
scalable.

Cheers,
Marek Kierdelewicz

Re: banning bot ips with ipset

[Jozsef Kadlecsik]

On Tue, 25 Nov 2008, Nigel Heron wrote:

> Is ipset stable enough to be deployed on live environments? iphash seems 
> like the best set type for us, how many ips can the set handle before 
> there's a noticeable slowdown? any feedback would be appreciated.

ipset is stable enough for in production usage. The speed of the iphash 
set is actually constant by nature and independent of the number of actual 
entries (your network card and bus will slow you down and not iphash).
 
> also, if it helps anyone else .. while trying ipset 2.4.5 i had to add:
> #include <stdbool.h>
> to "kernel/ip_set_setlist.c" to get it to compile.

What is your kernel version?

Best regards,
Jozsef
-
E-mail  : kadlec@blackhole.kfki.hu, kadlec@mail.kfki.hu
PGP key : http://www.kfki.hu/~kadlec/pgp_public_key.txt
Address : KFKI Research Institute for Particle and Nuclear Physics
          H-1525 Budapest 114, POB. 49, Hungary

Re: banning bot ips with ipset

[Nigel Heron]

Jozsef Kadlecsik wrote:

> ipset is stable enough for in production usage. The speed of the iphash 
> set is actually constant by nature and independent of the number of actual 
> entries (your network card and bus will slow you down and not iphash).
>  

thanks for everyone's feedback, we tried it on a test box with all our 
rules and it seems to work really well. we'll try it live during the week.

>> also, if it helps anyone else .. while trying ipset 2.4.5 i had to add:
>> #include <stdbool.h>
>> to "kernel/ip_set_setlist.c" to get it to compile.
> 
> What is your kernel version?
> 

2.6.18.2 which comes with opensuse 10.2
gcc "4.1.2 20061115 (prerelease) (SUSE Linux)" .. also bundled with 
opensuse 10.2


thanks,
-nigel.