chance to impress the suits

chance to impress the suits

[Jason]

Hello,  I have been given a task that I think netfilter is ideal for, but
need a little help.  I need to be able to limit the number of connections
going through a router running netfilter to a max of 500. When I hit 500,
I want to reject any new connections. I know that the iplimit match does
this, but I don't seem to be having any luck getting it to work.  Here is
the scenario..

----                ----                ----
=A =  -> Port 80 -> +NF+  -> Port 80 -> =C =
----                ----                ----

Simple enough.  NF is my netfilter router, A is source, C is destination.
C is an application that when it gets overworked, pretty much tarpits
connecting clients and never lets go.  I have tried many permutations of

iptables -A FORWARD -p tcp -i lan -m state --state NEW -m iplimit
--iplimit-above 1 -j REJECT

with out much luck.  Does anyone have any idea on how to make iplimit in a
router situation work?  Should I apply this to the output chain?

--
Jason Baker
baker@cyborgworkshop.com
www.cyborgworkshop.com

Re: chance to impress the suits

[Joel Newkirk]

On Wednesday 26 February 2003 06:57 pm, Jason wrote:
> Hello,  I have been given a task that I think netfilter is ideal for,
> but need a little help.  I need to be able to limit the number of
> connections going through a router running netfilter to a max of 500.
> When I hit 500, I want to reject any new connections. I know that the
> iplimit match does this, but I don't seem to be having any luck
> getting it to work.  Here is the scenario..
>
> ----                ----                ----
> =A =  -> Port 80 -> +NF+  -> Port 80 -> =C =
> ----                ----                ----
>
> Simple enough.  NF is my netfilter router, A is source, C is
> destination. C is an application that when it gets overworked, pretty
> much tarpits connecting clients and never lets go.  I have tried many
> permutations of
>
> iptables -A FORWARD -p tcp -i lan -m state --state NEW -m iplimit
> --iplimit-above 1 -j REJECT
>
> with out much luck.  Does anyone have any idea on how to make iplimit
> in a router situation work?  Should I apply this to the output chain?

Definitely the FORWARD chain, OUTPUT is for connections from the 
firewalling box itself.  Make sure this appears before any ACCEPT rules 
in your FORWARD chain, too.  Have you tried:

iptables -A FORWARD -i eth0 -p tcp --syn --dport 80 -m iplimit 
--iplimit-above 500 -j REJECT

This is almost precisely the format of the example rule for iplimit...  I 
noticed you used "-i lan" above - is that a typo?  You have to specify a 
valid interface name, which my version presumes is eth0 for traffic from 
the LAN.

When you say "without much luck" what do you mean?  Everything still gets 
through, nothing gets through at all, or what?  Does "iptables -L -v -n" 
show any matches to this rule?

j

Re: chance to impress the suits

[Arnt Karlsen]

On Thu, 27 Feb 2003 02:01:04 -0500, 
Joel Newkirk <netfilter@newkirk.us> wrote in message 
<200302270201.04762.netfilter@newkirk.us>:

> On Wednesday 26 February 2003 06:57 pm, Jason wrote:
> >
> > iptables -A FORWARD -p tcp -i lan -m state --state NEW -m iplimit
> > --iplimit-above 1 -j REJECT

..<snip>

> Definitely the FORWARD chain, OUTPUT is for connections from the 
> firewalling box itself.  Make sure this appears before any ACCEPT
> rules in your FORWARD chain, too.  Have you tried:
> 
> iptables -A FORWARD -i eth0 -p tcp --syn --dport 80 -m iplimit 
> --iplimit-above 500 -j REJECT
> 
> This is almost precisely the format of the example rule for iplimit...
> I noticed you used "-i lan" above - is that a typo?  You have to
> specify a valid interface name, which my version presumes is eth0 for
> traffic from the LAN.
> 

.." -i $lan "?  AFAICT, above rule is valid if "lan" is a substitute
for an ip address etc, for a variable, you will want to declare it, 
some people like CAPS, for, say, " -i $LAN ", YMMV.

..you don't mention _why_ your suits want this.  Using un-throttled
802.11 links with some _cheap_ routers that rebooted! every time 
they got 256 simultaneous connections for a client, I started with 
throttling, then capping connections, and ended up wrapping all his
isp clients in vpn/poptop tunnels.  Poptop, because some people 
still runs Wintendo 95, and my client like this business too.  

..yup, my first client is an isp.  ;-)

-- 
..med vennlig hilsen = with Kind Regards from Arnt... ;-)
...with a number of polar bear hunters in his ancestry...
  Scenarios always come in sets of three: 
  best case, worst case, and just in case.

Re: chance to impress the suits

[Jason]

The firewall I am using allows me to rename connections to lan or internet
or dmz using ifname to make things a little easier to understand, so yes
lan is eth0.  What has happened thus far is that connections that the rule
never seems to be triggered.  I didnt think about another rule being hit
before this one,  I will tear this guy down and see what I have going on.
Thank you.

-- 
Jason Baker
baker@cyborgworkshop.com
www.cyborgworkshop.com

On Thu, 27 Feb 2003, Joel Newkirk wrote:

> On Wednesday 26 February 2003 06:57 pm, Jason wrote:
> > Hello,  I have been given a task that I think netfilter is ideal for,
> > but need a little help.  I need to be able to limit the number of
> > connections going through a router running netfilter to a max of 500.
> > When I hit 500, I want to reject any new connections. I know that the
> > iplimit match does this, but I don't seem to be having any luck
> > getting it to work.  Here is the scenario..
> >
> > ----                ----                ----
> > =A =  -> Port 80 -> +NF+  -> Port 80 -> =C =
> > ----                ----                ----
> >
> > Simple enough.  NF is my netfilter router, A is source, C is
> > destination. C is an application that when it gets overworked, pretty
> > much tarpits connecting clients and never lets go.  I have tried many
> > permutations of
> >
> > iptables -A FORWARD -p tcp -i lan -m state --state NEW -m iplimit
> > --iplimit-above 1 -j REJECT
> >
> > with out much luck.  Does anyone have any idea on how to make iplimit
> > in a router situation work?  Should I apply this to the output chain?
>
> Definitely the FORWARD chain, OUTPUT is for connections from the
> firewalling box itself.  Make sure this appears before any ACCEPT rules
> in your FORWARD chain, too.  Have you tried:
>
> iptables -A FORWARD -i eth0 -p tcp --syn --dport 80 -m iplimit
> --iplimit-above 500 -j REJECT
>
> This is almost precisely the format of the example rule for iplimit...  I
> noticed you used "-i lan" above - is that a typo?  You have to specify a
> valid interface name, which my version presumes is eth0 for traffic from
> the LAN.
>
> When you say "without much luck" what do you mean?  Everything still gets
> through, nothing gets through at all, or what?  Does "iptables -L -v -n"
> show any matches to this rule?
>
> j
>
>

Re: chance to impress the suits

[Jason]

What we have is a server that makes as many connections to an application
as it can.   Its supposed to be a realtime app, so this is desired
behaviour.  Unfortunatly, the app is owned by a different group that can't
seem to get a grip on how much hardware they need. So we max them out, and
their solution when they hit too many connections is to allow the port to
be opened by the client (us)  but never send any data or a RST or
anything! So my server ends up with tens of thousands of connections in
wait and I end up running out of threads pretty quickly. So my thought
was by putting an iptables box in the stream with iplimit and either
redirecting connections that go over a max count to a "sorry we're busy
page" or denying the connection all together, I can save my machine until
they get the hardware they need. Is their perhaps a better method?  Right
now I have to babysit my servers from 8pm to 3am and kill the route to
their application when things get ugly.  Pretty nasty solution.

-- 
Jason Baker
baker@cyborgworkshop.com
www.cyborgworkshop.com

On Thu, 27 Feb 2003, Arnt Karlsen wrote:

> On Thu, 27 Feb 2003 02:01:04 -0500,
> Joel Newkirk <netfilter@newkirk.us> wrote in message
> <200302270201.04762.netfilter@newkirk.us>:
>
> > On Wednesday 26 February 2003 06:57 pm, Jason wrote:
> > >
> > > iptables -A FORWARD -p tcp -i lan -m state --state NEW -m iplimit
> > > --iplimit-above 1 -j REJECT
>
> ..<snip>
>
> > Definitely the FORWARD chain, OUTPUT is for connections from the
> > firewalling box itself.  Make sure this appears before any ACCEPT
> > rules in your FORWARD chain, too.  Have you tried:
> >
> > iptables -A FORWARD -i eth0 -p tcp --syn --dport 80 -m iplimit
> > --iplimit-above 500 -j REJECT
> >
> > This is almost precisely the format of the example rule for iplimit...
> > I noticed you used "-i lan" above - is that a typo?  You have to
> > specify a valid interface name, which my version presumes is eth0 for
> > traffic from the LAN.
> >
>
> .." -i $lan "?  AFAICT, above rule is valid if "lan" is a substitute
> for an ip address etc, for a variable, you will want to declare it,
> some people like CAPS, for, say, " -i $LAN ", YMMV.
>
> ..you don't mention _why_ your suits want this.  Using un-throttled
> 802.11 links with some _cheap_ routers that rebooted! every time
> they got 256 simultaneous connections for a client, I started with
> throttling, then capping connections, and ended up wrapping all his
> isp clients in vpn/poptop tunnels.  Poptop, because some people
> still runs Wintendo 95, and my client like this business too.
>
> ..yup, my first client is an isp.  ;-)
>
>

Re: chance to impress the suits

[Jason]

They need a bigger box.   The whole app on their end is 6 boxes,  two
front end web servers, two application servers and then the database.
They upgraded the database, which used to be the bottleneck, and we then
maxed out their web servers. They upgraded their web servers and now we
max out their app servers. They have more hardware on order, but Sun is
claiming a month lead time for Sunfires.  But I figure in the meantime,
this should work out pretty well.

-- 
Jason Baker
baker@cyborgworkshop.com
www.cyborgworkshop.com

On Fri, 28 Feb 2003, Arnt Karlsen wrote:

> On Thu, 27 Feb 2003 08:17:33 -0600 (CST),
> Jason <baker@cyborgworkshop.com> wrote in message
> <Pine.LNX.4.50.0302270811550.19694-100000@alfred.home.cyborgworkshop.co
> m>:
>
> > What we have is a server that makes as many connections to an
> > application as it can.   Its supposed to be a realtime app, so this is
> > desired behaviour.
>
> ...says the guys who can't get that grip???
>
> > Unfortunatly, the app is owned by a different group that can't
> > seem to get a grip on how much hardware they need. So we max them out,
> > and their solution when they hit too many connections is to allow the
> > port to be opened by the client (us)  but never send any data or a RST
> > or anything! So my server ends up with tens of thousands of
> > connections in wait and I end up running out of threads pretty
> > quickly.
>
> ..ah, an _authorized_ dos attack.  ;-)
>
> > So my thought was by putting an iptables box in the stream
> > with iplimit and either redirecting connections that go over a max
> > count to a "sorry we're busy page" or denying the connection all
> > together, I can save my machine until they get the hardware they need.
>
> ..this is where I fall off:  They need a smaller box to not dos attack
> you, or a bigger box to not dos attack you???
>
> > Is their perhaps a better method?  Right now I have to babysit my
> > servers from 8pm to 3am and kill the route to their application when
> > things get ugly.  Pretty nasty solution.
> >
>
> ..played with the kernel settings in /proc/sys/net ?  You have
> checked the Patch-o-matic stuff for ideas?
>
> ..I get the feeling the "real time" autorized dos attack application
> should _re-use_ its own established connections, and make new
> connections _only_ when needed, and, _destroy_ the old connections
> as soon as they are no longer needed.  Pretty basic.  Does it?
>
> ..I don't see how an autorized dos attack application spraying
> new network connections like crazy, can _ever_be_ "real time".
>
>

Re: chance to impress the suits

[Arnt Karlsen]

On Thu, 27 Feb 2003 08:17:33 -0600 (CST), 
Jason <baker@cyborgworkshop.com> wrote in message 
<Pine.LNX.4.50.0302270811550.19694-100000@alfred.home.cyborgworkshop.co
m>:

> What we have is a server that makes as many connections to an
> application as it can.   Its supposed to be a realtime app, so this is
> desired behaviour. 

...says the guys who can't get that grip???

> Unfortunatly, the app is owned by a different group that can't
> seem to get a grip on how much hardware they need. So we max them out,
> and their solution when they hit too many connections is to allow the
> port to be opened by the client (us)  but never send any data or a RST
> or anything! So my server ends up with tens of thousands of
> connections in wait and I end up running out of threads pretty
> quickly.  

..ah, an _authorized_ dos attack.  ;-)

> So my thought was by putting an iptables box in the stream
> with iplimit and either redirecting connections that go over a max
> count to a "sorry we're busy page" or denying the connection all
> together, I can save my machine until they get the hardware they need.

..this is where I fall off:  They need a smaller box to not dos attack
you, or a bigger box to not dos attack you???

> Is their perhaps a better method?  Right now I have to babysit my
> servers from 8pm to 3am and kill the route to their application when
> things get ugly.  Pretty nasty solution.
> 

..played with the kernel settings in /proc/sys/net ?  You have 
checked the Patch-o-matic stuff for ideas?

..I get the feeling the "real time" autorized dos attack application
should _re-use_ its own established connections, and make new
connections _only_ when needed, and, _destroy_ the old connections 
as soon as they are no longer needed.  Pretty basic.  Does it?

..I don't see how an autorized dos attack application spraying 
new network connections like crazy, can _ever_be_ "real time".

-- 
..med vennlig hilsen = with Kind Regards from Arnt... ;-)
...with a number of polar bear hunters in his ancestry...
  Scenarios always come in sets of three: 
  best case, worst case, and just in case.

Re: chance to impress the suits

[Jason]

well remember, them getting bigger boxes are to handle the number of
connections that I am giving to them.  Its not a matter of TCP slowing
down, its us waiting on their application to finish and their app just
handles waits very poorly.

-- 
Jason Baker
baker@cyborgworkshop.com
www.cyborgworkshop.com

On Fri, 28 Feb 2003, Arnt Karlsen wrote:

> On Thu, 27 Feb 2003 15:21:02 -0600 (CST),
> Jason <baker@cyborgworkshop.com> wrote in message
> <Pine.LNX.4.50.0302271518100.19962-100000@alfred.home.cyborgworkshop.co
> m>:
>
> > They need a bigger box.   The whole app on their end is 6 boxes,  two
> > front end web servers, two application servers and then the database.
> > They upgraded the database, which used to be the bottleneck, and we
> > then maxed out their web servers. They upgraded their web servers and
> > now we max out their app servers. They have more hardware on order,
> > but Sun is claiming a month lead time for Sunfires.  But I figure in
> > the meantime, this should work out pretty well.
> >
>
> ..ok, I'm still deep in the dark:  How is a bigger set of iron, that can
> spew _more_ connections, gonna help avoid the authorized dos attack???
>
> ..the 6-too-small-boxes understands ipcm, and will back off when you
> drop packages or connection?
>
> ..I _think_ a _short!_ timeout on these connections, could work.
>
>

Re: chance to impress the suits

[Arnt Karlsen]

On Thu, 27 Feb 2003 15:21:02 -0600 (CST), 
Jason <baker@cyborgworkshop.com> wrote in message 
<Pine.LNX.4.50.0302271518100.19962-100000@alfred.home.cyborgworkshop.co
m>:

> They need a bigger box.   The whole app on their end is 6 boxes,  two
> front end web servers, two application servers and then the database.
> They upgraded the database, which used to be the bottleneck, and we
> then maxed out their web servers. They upgraded their web servers and
> now we max out their app servers. They have more hardware on order,
> but Sun is claiming a month lead time for Sunfires.  But I figure in
> the meantime, this should work out pretty well.
> 

..ok, I'm still deep in the dark:  How is a bigger set of iron, that can
spew _more_ connections, gonna help avoid the authorized dos attack???

..the 6-too-small-boxes understands ipcm, and will back off when you
drop packages or connection?  

..I _think_ a _short!_ timeout on these connections, could work.

-- 
..med vennlig hilsen = with Kind Regards from Arnt... ;-)
...with a number of polar bear hunters in his ancestry...
  Scenarios always come in sets of three: 
  best case, worst case, and just in case.

Re: chance to impress the suits

[Arnt Karlsen]

On Fri, 28 Feb 2003 06:53:45 -0600 (CST), 
Jason <baker@cyborgworkshop.com> wrote in message 
<Pine.LNX.4.50.0302280652440.20514-100000@alfred.home.cyborgworkshop.co
m>:

> well remember, them getting bigger boxes are to handle the number of
> connections that I am giving to them.  Its not a matter of TCP slowing
> down, its us waiting on their application to finish and their app just
> handles waits very poorly.

...effectively dos attacking itself, and with authority?  How is 
it controlled?  There must have been a control philosophy somewhere 
between the design stage and the upgrade of this application?   

..an idea: limit the number of connections to a level the authorized 
dos attack app _can_ handle, and to reject or reset rather than 
dropping connections?

-- 
..med vennlig hilsen = with Kind Regards from Arnt... ;-)
...with a number of polar bear hunters in his ancestry...
  Scenarios always come in sets of three: 
  best case, worst case, and just in case.

Re: chance to impress the suits

[Jason]

> ..an idea: limit the number of connections to a level the authorized
> dos attack app _can_ handle, and to reject or reset rather than
> dropping connections?

That is what I hope to do with the iptables box running iplimit.  Except
instead of rejecting the connections Im going to redirect users to a
static page that just pretty much says "we're busy try again"

Re: chance to impress the suits

[Arnt Karlsen]

On Fri, 28 Feb 2003 20:04:44 -0600 (CST), 
Jason <baker@cyborgworkshop.com> wrote in message 
<Pine.LNX.4.50.0302282003460.20817-100000@alfred.home.cyborgworkshop.co
m>:

> > ..an idea: limit the number of connections to a level the authorized
> > dos attack app _can_ handle, and to reject or reset rather than
> > dropping connections?
> 
> That is what I hope to do with the iptables box running iplimit. 
> Except instead of rejecting the connections Im going to redirect users
> to a static page that just pretty much says "we're busy try again"
> 

..can the authorized dos attack application use this to back off?
Failing that, try kill off its old no-longer-used connections.

-- 
..med vennlig hilsen = with Kind Regards from Arnt... ;-)
...with a number of polar bear hunters in his ancestry...
  Scenarios always come in sets of three: 
  best case, worst case, and just in case.