Tux and netfilter

Tux and netfilter

[Marek Habersack]

[-- Attachment #1: Type: text/plain, Size: 4480 bytes --]

Hello,

  One of the servers at our colo is under a heavy attack consisting of
massive synfloods followed by the connections to port 80 issuing bogus HTTP
requestss like

  ET xxx.yyy.zzz.vvv HTTP/X.Y
  Host: /search.php
  Referer: http://google.com/

  ^^^^^^^^^note the referer, an impossible one.

Where xx etc is an IP number, there are numerous other variations of the
requests. The number of packets amounts to around 200M in the course of
around 5h (latest data). I have tuned the kernel so that it survives the
number of packets, but still netstat shows around 20k sockets in FIN2_WAIT
state (most of them), the machine load is ~3. So far so good. I've put tux
in front of apache (which is running vBulletin :>) to stop apache from
bogging down the machine and that helped a bit. I've modified tux so that it
zaps HTTP/1.0 and bogus Referer connections immediately, that also helped a
bit but it wasn't enough. So I resorted to using the TARPIT target and the
string matcher with iptables. The rules used are:

  33M 6284M TARPIT tcp  --  *      *       0.0.0.0/0 0.0.0.0/0          tcp dpt:80 STRING match HTTP/1.0^M 
  31M 6505M TARPIT tcp  --  *      *       0.0.0.0/0 0.0.0.0/0          tcp dpt:80 STRING match ET 66.28. 
 573K  119M mightbevalid  tcp  --  *      *       0.0.0.0/0 0.0.0.0/0          tcp dpt:80 STRING match HTTP/1.1^M 
  928  185K TARPIT tcp  --  *      *       0.0.0.0/0 0.0.0.0/0          tcp dpt:80 STRING match Referer: http://google.com/^M 
    0     0 TARPIT tcp  --  *      *       0.0.0.0/0 0.0.0.0/0          tcp dpt:80 STRING match Referer: http://www.google.com/^M 
    0     0 TARPIT tcp  --  *      *       0.0.0.0/0 0.0.0.0/0          tcp dpt:80 STRING match Referer: http://www.google.com^M 
    0     0 TARPIT tcp  --  *      *       0.0.0.0/0 0.0.0.0/0          tcp dpt:80 STRING match Referer: http://google.com^M 
  70M 2879M ACCEPT tcp  --  *      *       0.0.0.0/0 0.0.0.0/0          tcp flags:!0x16/0x02 
Chain mightbevalid (1 references)
 pkts bytes target     prot opt in     out     source
destination         
23245 4892K ACCEPT     tcp  --  *      *       0.0.0.0/0 0.0.0.0/0          STRING match .php 
    0     0 ACCEPT     tcp  --  *      *       0.0.0.0/0 0.0.0.0/0          STRING match GET /index.html HTTP/1.1 
 386K   78M ACCEPT     tcp  --  *      *       0.0.0.0/0 0.0.0.0/0          STRING match GET / HTTP/1.1^M 
 164K   36M TARPIT     tcp  --  *      *       0.0.0.0/0 0.0.0.0/0          


The counter values are the real ones. I would use the iplimit matcher, but I
don't want to use connection tracking since that would hose the machine
pretty quick. All of the above actions have the effect that the machine is
reachable, interaction is good, but tux is practically clogged (it's
configured to accept 50k connections, keep alives are at 1000 since setting
them to 0 makes tux close any connection immediately, no logging etc,). Apache sits
on port 81 and when accessed directly it works fine, that's good enough, but
I'd like to do more. And here I come to the real question I want to ask to
the list. Is it possible and if yes, then how, to redirect the offending
packets from within tux to the TARPIT chain? It seems a bit backwards to me
and I'm not quite sure whether it's possible at all, but maybe?
I'm not asking for ready code, just some helping hand and some advice 
where to start.

If any of the above sounds chaotic and unclear, don't hesitate to ask me...
:)

TIA,

marek

p.s. The machine is PIII/800Mhz with 0.5GB of RAM, kernel 2.4.21-aa1rc8,
syncookies are on and here are the relevant sysctl values I'm using (I know,
some of them have dangerous values, but it works pretty well atm):

net/ipv4/icmp_echo_ignore_broadcasts=1
fs/file-max=70000
fs/dir-notify-enable=0
net/ipv4/tcp_keepalive_time=30
net/core/rmem_max=262143
net/core/rmem_default=262143
net/core/wmem_max=262143
net/core/wmem_default=262143
net/ipv4/tcp_sack=0
net/ipv4/tcp_timestamps=0
net/ipv4/tcp_syncookies=1
net/ipv4/icmp_echo_ignore_all =1
net/ipv4/icmp_ignore_bogus_error_responses = 1
net/ipv4/tcp_syn_retries = 1
net/ipv4/tcp_synack_retries = 1
net/ipv4/tcp_keepalive_probes = 1
net/ipv4/tcp_keepalive_intvl = 10
net/ipv4/tcp_max_syn_backlog = 64
net/ipv4/tcp_low_latency = 1
net/ipv4/tcp_abort_on_overflow = 1
net/ipv4/ipfrag_time = 30
net/ipv4/tcp_fin_timeout = 10
net/ipv4/tcp_max_orphans = 2048
net/ipv4/tcp_tw_reuse = 1

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: Tux and netfilter

[Arnt Karlsen]

On Sun, 6 Jul 2003 23:14:03 +0200, 
Marek Habersack <grendel@caudium.net> wrote in message 
<20030706211403.GA9350@thanes.org>:


> 
> The counter values are the real ones. I would use the iplimit matcher,
> but I don't want to use connection tracking since that would hose the
> machine pretty quick. All of the above actions have the effect that
> the machine is reachable, interaction is good, but tux is practically
> clogged (it's configured to accept 50k connections, keep alives are at
> 1000 since setting

..1000 seconds???  Shave off a zero or two, you should be able to serve
any valid traffic within 5 seconds.

> them to 0 makes tux close any connection immediately, no logging
> etc,). Apache sits on port 81 and when accessed directly it works
> fine, that's good enough, but I'd like to do more. And here I come to
> the real question I want to ask to the list. Is it possible and if
> yes, then how, to redirect the offending packets from within tux to
> the TARPIT chain? 

..does your TARPIT traffic cost _you_ anything?

> net/ipv4/icmp_echo_ignore_broadcasts=1

..also possible to lie and say the box is a crashing, 
or hung dead wintendo.

> fs/file-max=70000
> fs/dir-notify-enable=0
> net/ipv4/tcp_keepalive_time=30
> net/core/rmem_max=262143
> net/core/rmem_default=262143
> net/core/wmem_max=262143
> net/core/wmem_default=262143
> net/ipv4/tcp_sack=0
> net/ipv4/tcp_timestamps=0
> net/ipv4/tcp_syncookies=1
> net/ipv4/icmp_echo_ignore_all =1
> net/ipv4/icmp_ignore_bogus_error_responses = 1
> net/ipv4/tcp_syn_retries = 1
> net/ipv4/tcp_synack_retries = 1
> net/ipv4/tcp_keepalive_probes = 1
> net/ipv4/tcp_keepalive_intvl = 10
> net/ipv4/tcp_max_syn_backlog = 64
> net/ipv4/tcp_low_latency = 1
> net/ipv4/tcp_abort_on_overflow = 1
> net/ipv4/ipfrag_time = 30
> net/ipv4/tcp_fin_timeout = 10
> net/ipv4/tcp_max_orphans = 2048

..why so many?  Most of these would come from "google", no?

> net/ipv4/tcp_tw_reuse = 1
> 


-- 
..med vennlig hilsen = with Kind Regards from Arnt... ;-)
...with a number of polar bear hunters in his ancestry...
  Scenarios always come in sets of three: 
  best case, worst case, and just in case.

Re: Tux and netfilter

[Marek Habersack]

[-- Attachment #1: Type: text/plain, Size: 2168 bytes --]

On Mon, Jul 07, 2003 at 01:47:51AM +0200, Arnt Karlsen scribbled:
[snip]
> > clogged (it's configured to accept 50k connections, keep alives are at
> > 1000 since setting
> 
> ..1000 seconds???  Shave off a zero or two, you should be able to serve
> any valid traffic within 5 seconds.
They are not 1000 seconds - 1000 keepalive slots tux keeps active around.
After there are more than that, it starts killing connections in the LRU
order.

> > them to 0 makes tux close any connection immediately, no logging
> > etc,). Apache sits on port 81 and when accessed directly it works
> > fine, that's good enough, but I'd like to do more. And here I come to
> > the real question I want to ask to the list. Is it possible and if
> > yes, then how, to redirect the offending packets from within tux to
> > the TARPIT chain? 
> 
> ..does your TARPIT traffic cost _you_ anything?
A little bit, a tiny bit. Works better than a DROP, actually. And it has the
added advantage that it ties resources of the attacker somewhat.

> > net/ipv4/icmp_echo_ignore_broadcasts=1
> 
> ..also possible to lie and say the box is a crashing, 
> or hung dead wintendo.
> 
> > fs/file-max=70000
> > fs/dir-notify-enable=0
> > net/ipv4/tcp_keepalive_time=30
> > net/core/rmem_max=262143
> > net/core/rmem_default=262143
> > net/core/wmem_max=262143
> > net/core/wmem_default=262143
> > net/ipv4/tcp_sack=0
> > net/ipv4/tcp_timestamps=0
> > net/ipv4/tcp_syncookies=1
> > net/ipv4/icmp_echo_ignore_all =1
> > net/ipv4/icmp_ignore_bogus_error_responses = 1
> > net/ipv4/tcp_syn_retries = 1
> > net/ipv4/tcp_synack_retries = 1
> > net/ipv4/tcp_keepalive_probes = 1
> > net/ipv4/tcp_keepalive_intvl = 10
> > net/ipv4/tcp_max_syn_backlog = 64
> > net/ipv4/tcp_low_latency = 1
> > net/ipv4/tcp_abort_on_overflow = 1
> > net/ipv4/ipfrag_time = 30
> > net/ipv4/tcp_fin_timeout = 10
> > net/ipv4/tcp_max_orphans = 2048
> 
> ..why so many?  Most of these would come from "google", no?
That's not a lot... Well, the ones that come from "google" constitute
perhaps 1/3 of the total. And I still have to think about the legit clients.

regards,

marek

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]