Re: Protecting against DoS

Re: Protecting against DoS

[Geffrey Velásquez]

Hi,

> Hello,
>
> 	First make sure you are using tcpsyn_cookies:
>
> echo 1 > /proc/sys/net/ipv4/tcp_syncookies -- if you have not compiled


Is that valid for forwarded packets? or only destinated to the firewall?



Regards,
Geffrey.

Re: Protecting against DoS

[John A. Sullivan III]

On Tue, 2003-12-09 at 14:11, Geffrey Velásquez wrote:
> Hi,
> 
> > Hello,
> >
> > 	First make sure you are using tcpsyn_cookies:
> >
> > echo 1 > /proc/sys/net/ipv4/tcp_syncookies -- if you have not compiled
> 
> 
> Is that valid for forwarded packets? or only destinated to the firewall?
> 
<snip>
I'm glad you brought that up.  In fact, I'm delighted at this entire
discussion since we are developing the final self-protection rules for
use in the ISCS project.

We have avoided using these /proc settings for just that concern - that
they are mostly for the gateway itself and not for the devices being
protected by it whether it is anti-spoofing with rp_filter or protecting
against syn_floods.  Is this assumption of ours true? Thanks, all - John
-- 
John A. Sullivan III
Chief Technology Officer
Nexus Management
+1 207-985-7880
john.sullivan@nexusmgmt.com
---
If you are interested in helping to develop a GPL enterprise class
VPN/Firewall/Security device management console, please visit
http://iscs.sourceforge.net 

Re: Protecting against DoS

[Ralf Spenneberg]

Hi,

Am Die, 2003-12-09 um 19.01 schrieb John A. Sullivan III:
> > > echo 1 > /proc/sys/net/ipv4/tcp_syncookies -- if you have not compiled
> > 
> > 
> > Is that valid for forwarded packets? or only destinated to the firewall?
This is valid only for local packets.

> We have avoided using these /proc settings for just that concern - that
> they are mostly for the gateway itself and not for the devices being
> protected by it whether it is anti-spoofing with rp_filter or protecting
> against syn_floods.  Is this assumption of ours true? Thanks, all - John
Actually it depends. Most just concern local packets, but
rp_filter and accept_source_route for example tests for all packets.


Cheers,

Ralf
-- 
Ralf Spenneberg
RHCE, RHCX

Book: VPN mit Linux
Book: Intrusion Detection für Linux Server   http://www.spenneberg.com
IPsec-Howto				     http://www.ipsec-howto.org
Honeynet Project Mirror:                     http://honeynet.spenneberg.org

Re: Protecting against DoS

[John A. Sullivan III]

On Tue, 2003-12-09 at 13:16, Ralf Spenneberg wrote:
> Hi,
> 
> Am Die, 2003-12-09 um 19.01 schrieb John A. Sullivan III:
> > > > echo 1 > /proc/sys/net/ipv4/tcp_syncookies -- if you have not compiled
> > > 
> > > 
> > > Is that valid for forwarded packets? or only destinated to the firewall?
> This is valid only for local packets.
> 
> > We have avoided using these /proc settings for just that concern - that
> > they are mostly for the gateway itself and not for the devices being
> > protected by it whether it is anti-spoofing with rp_filter or protecting
> > against syn_floods.  Is this assumption of ours true? Thanks, all - John
> Actually it depends. Most just concern local packets, but
> rp_filter and accept_source_route for example tests for all packets.
> 
> 
> Cheers,
> 
> Ralf
Ah, very interesting.  Yes, we do disable source route and we disable
redirects through /proc on our gateways but I thought (in my ignorance)
that rp_filter only affected traffic to the gateway.  So you are saying
that if rp_filter=1 and a packet comes into the gateway on eth0 bound
for an address on the other side of eth1 but not the address bound to
eth1 and has a source address of some device on eth0, that rp_filter
will drop the packet before it ever gets to netfilter?
To take it a step further, if rp_filter=2 and a packet come into the
gateway on eth0 bound for an address on the other side of eth1 but not
on a network directly connected to eth1 and has a source address that is
from a network that is either directly or indirectly connected to eth1,
rp_filter will drop the packet before it gets to netfilter? This
actually seems a bit strange since the network on the other side of eth0
is indirectly connected to eth1 but I'm not quite sure what rp_filter=2
means.
Thanks - John
-- 
John A. Sullivan III
Chief Technology Officer
Nexus Management
+1 207-985-7880
john.sullivan@nexusmgmt.com
---
If you are interested in helping to develop a GPL enterprise class
VPN/Firewall/Security device management console, please visit
http://iscs.sourceforge.net 

Protecting against DoS

[Pasi Kärkkäinen]

Hello!

I was thinking about the correct or best way to protect my Linux/netfilter 
box againts DoS-attacks. 

Some time ago one of the windows users in my LAN managed to get nimda (or
some other) worm to his computer. The worm started scanning the internet
for other vulnerable boxes, opening big amount of tcp-connections all the
time without closing them. 

So after a while I hit the limit of max. open connections
(/proc/sys/net/ipv4/ip_conntrack_max), and the firewall-box is basicly
DoS:ed. With the default settings, open tcp-connections stay in the state
table for 5 days, so it takes a looong time to get things running again if
you don't reload the modules or reboot the box..

Now I have a couple of questions to be sure about the facts while setting 
up the correct limits to prevent this kind of DoS-attacks..


1) Is the correct formula to calculate the maximum number of connections
(for /proc/sys/net/ipv4/ip_conntrack_max) free_memory_in_bytes / 350 ? This
is what I got from the Netfilter FAQ: "You can easily increase the number of
maximal tracked connections, but be aware that each tracked connection eats
about 350 bytes of non-swappable kernel memory!"

2) Netfilter FAQ: "To optimize performance, please also raise the number of
hash buckets by using the hashsize module loadtime parameter of the
ip_conntrack.o module." What's the correct formula to calculate good value
for hashsize?

3) Is there some problem other than the idle tcp-connections dying sooner if I
lower the the value of TCP_CONNTRACK_ESTABLISHED in
/usr/src/linux/net/ipv4/netfilter/ip_conntrack_proto_tcp.c from 5 days to 1
day or even less (to get the possible non-closed tcp-connections out from the state
table sooner) ?

4) What's the correct place to set up limits for new connections (to prevent
the state table being filled up in DoS) ? Is it better to do in the 
mangle-table/PREROUTING-chain something like "-m state --state NEW -m limit 
--limit 5/sec -j RETURN && -j DROP" than later in the filter-table/FORWARD-chain?
I'm thinking about performance here..

5) I'm thinking about measuring average "new connections per second"-rate
and setting up limits to obey that.. is this good way?

6) Do you have some other tips? What are the biggest problems in addition to
getting the state table filled up..


Thanks for your replies!

-- Pasi Kärkkäinen
       
                                   ^
                                .     .
                                 Linux
                              /    -    \
                             Choice.of.the
                           .Next.Generation.

Re: Protecting against DoS

[Michael Gale]

Hello,

	First make sure you are using tcpsyn_cookies:

echo 1 > /proc/sys/net/ipv4/tcp_syncookies -- if you have not compiled it into the kernel. This will help prevent DOS by assigning each incoming syn packet a cookie instead of a actually connection state. A connection state will be created once the three way hand shake is completed.

Second -- you should be dropping all packets on all interfaces and then only allow connections you have to pass.

Michael.


On Tue, 9 Dec 2003 17:43:34 +0200
Pasi Kärkkäinen <pasik@iki.fi> wrote:

> Hello!
> 
> I was thinking about the correct or best way to protect my Linux/netfilter 
> box againts DoS-attacks. 
> 
> Some time ago one of the windows users in my LAN managed to get nimda (or
> some other) worm to his computer. The worm started scanning the internet
> for other vulnerable boxes, opening big amount of tcp-connections all the
> time without closing them. 
> 
> So after a while I hit the limit of max. open connections
> (/proc/sys/net/ipv4/ip_conntrack_max), and the firewall-box is basicly
> DoS:ed. With the default settings, open tcp-connections stay in the state
> table for 5 days, so it takes a looong time to get things running again if
> you don't reload the modules or reboot the box..
> 
> Now I have a couple of questions to be sure about the facts while setting 
> up the correct limits to prevent this kind of DoS-attacks..
> 
> 
> 1) Is the correct formula to calculate the maximum number of connections
> (for /proc/sys/net/ipv4/ip_conntrack_max) free_memory_in_bytes / 350 ? This
> is what I got from the Netfilter FAQ: "You can easily increase the number of
> maximal tracked connections, but be aware that each tracked connection eats
> about 350 bytes of non-swappable kernel memory!"
> 
> 2) Netfilter FAQ: "To optimize performance, please also raise the number of
> hash buckets by using the hashsize module loadtime parameter of the
> ip_conntrack.o module." What's the correct formula to calculate good value
> for hashsize?
> 
> 3) Is there some problem other than the idle tcp-connections dying sooner if I
> lower the the value of TCP_CONNTRACK_ESTABLISHED in
> /usr/src/linux/net/ipv4/netfilter/ip_conntrack_proto_tcp.c from 5 days to 1
> day or even less (to get the possible non-closed tcp-connections out from the state
> table sooner) ?
> 
> 4) What's the correct place to set up limits for new connections (to prevent
> the state table being filled up in DoS) ? Is it better to do in the 
> mangle-table/PREROUTING-chain something like "-m state --state NEW -m limit 
> --limit 5/sec -j RETURN && -j DROP" than later in the filter-table/FORWARD-chain?
> I'm thinking about performance here..
> 
> 5) I'm thinking about measuring average "new connections per second"-rate
> and setting up limits to obey that.. is this good way?
> 
> 6) Do you have some other tips? What are the biggest problems in addition to
> getting the state table filled up..
> 
> 
> Thanks for your replies!
> 
> -- Pasi Kärkkäinen
>        
>                                    ^
>                                 .     .
>                                  Linux
>                               /    -    \
>                              Choice.of.the
>                            .Next.Generation.
> 


-- 
Michael Gale
Network Administrator
Utilitran Corporation

Re: Protecting against DoS

[Pasi Kärkkäinen]

On Tue, Dec 09, 2003 at 09:02:21AM -0700, Michael Gale wrote:
> 
> Hello,
> 
> 	First make sure you are using tcpsyn_cookies:
> 
> echo 1 > /proc/sys/net/ipv4/tcp_syncookies -- if you have not compiled it into the kernel. 
> This will help prevent DOS by assigning each incoming syn packet a cookie instead of a actually 
> connection state. A connection state will be created once the three way hand shake is completed.
> 
> Second -- you should be dropping all packets on all interfaces and then only allow connections 
> you have to pass.
> 

Yes.. I'm already doing both of these things. I was thinking of doing some
extra in addition of these.. Sorry I didn't mention about these already.

There are always some connections allowed that can be used to fill up the
state table..

Thanks anyway!

> Michael.
> 

-- Pasi Kärkkäinen
       
                                   ^
                                .     .
                                 Linux
                              /    -    \
                             Choice.of.the
                           .Next.Generation.

Re: Protecting against DoS

[Michael Gale]

Hello,

	Can you provide more detail on the type of traffic that caused the DOS -- this may help people in the list with suggestions on how to block it :)

Michael.



On Tue, 9 Dec 2003 18:28:20 +0200
Pasi Kärkkäinen <pasik@iki.fi> wrote:

> On Tue, Dec 09, 2003 at 09:02:21AM -0700, Michael Gale wrote:
> > 
> > Hello,
> > 
> > 	First make sure you are using tcpsyn_cookies:
> > 
> > echo 1 > /proc/sys/net/ipv4/tcp_syncookies -- if you have not compiled it into the kernel. 
> > This will help prevent DOS by assigning each incoming syn packet a cookie instead of a actually 
> > connection state. A connection state will be created once the three way hand shake is completed.
> > 
> > Second -- you should be dropping all packets on all interfaces and then only allow connections 
> > you have to pass.
> > 
> 
> Yes.. I'm already doing both of these things. I was thinking of doing some
> extra in addition of these.. Sorry I didn't mention about these already.
> 
> There are always some connections allowed that can be used to fill up the
> state table..
> 
> Thanks anyway!
> 
> > Michael.
> > 
> 
> -- Pasi Kärkkäinen
>        
>                                    ^
>                                 .     .
>                                  Linux
>                               /    -    \
>                              Choice.of.the
>                            .Next.Generation.
> 


-- 
Michael Gale
Network Administrator
Utilitran Corporation

Re: Protecting against DoS

[Pasi Kärkkäinen]

On Tue, Dec 09, 2003 at 09:40:47AM -0700, Michael Gale wrote:
> Hello,
> 
> 	Can you provide more detail on the type of traffic that caused the DOS -- this may help people in the list with suggestions on how to block it :)
> 

Yep. It was tcp-connections from the windows box (infected by the worm) to
some network-ranges on the internet. source-port was pretty much random, but
the destination was always 80. So the normal 'allow web browsing' rules
allowed the worm to DoS the linux-firewall.  

It just opened the connections all the time, but didn't close them. 

> Michael.
> 

-- Pasi Kärkkäinen
       
                                   ^
                                .     .
                                 Linux
                              /    -    \
                             Choice.of.the
                           .Next.Generation.

Re: Protecting against DoS

[Michael Gale]

Hello,

	You could try using a rate limit -- you could allow a machine to make lets say 10 outbound connections a second and then ...

Depending on your network policy you could drop or log all other outbound request.

Michael.



On Tue, 9 Dec 2003 18:51:46 +0200
Pasi Kärkkäinen <pasik@iki.fi> wrote:

> On Tue, Dec 09, 2003 at 09:40:47AM -0700, Michael Gale wrote:
> > Hello,
> > 
> > 	Can you provide more detail on the type of traffic that caused the DOS -- this may help people in the list with suggestions on how to block it :)
> > 
> 
> Yep. It was tcp-connections from the windows box (infected by the worm) to
> some network-ranges on the internet. source-port was pretty much random, but
> the destination was always 80. So the normal 'allow web browsing' rules
> allowed the worm to DoS the linux-firewall.  
> 
> It just opened the connections all the time, but didn't close them. 
> 
> > Michael.
> > 
> 
> -- Pasi Kärkkäinen
>        
>                                    ^
>                                 .     .
>                                  Linux
>                               /    -    \
>                              Choice.of.the
>                            .Next.Generation.
> 


-- 
Michael Gale
Network Administrator
Utilitran Corporation

Re: Protecting against DoS

[Pasi Kärkkäinen]

On Tue, Dec 09, 2003 at 10:06:50AM -0700, Michael Gale wrote:
> Hello,
> 
> 	You could try using a rate limit -- you could allow a machine to make lets say 10 outbound 
> connections a second and then ...
> 
> Depending on your network policy you could drop or log all other outbound request.
> 

This is what I'm planning to do.. and this is also the reason I was asking
the questions in the original mail :-)

-- Pasi Kärkkäinen
       
                                   ^
                                .     .
                                 Linux
                              /    -    \
                             Choice.of.the
                           .Next.Generation.

Re: Protecting against DoS

[Geffrey Velásquez]

Hum.. but what happen if the workstation is infected? the virus will do a
DoS to the user of the workstation, the firewall will block valid
connections.
> On Tue, Dec 09, 2003 at 10:06:50AM -0700, Michael Gale wrote:
>> Hello,
>>
>> 	You could try using a rate limit -- you could allow a machine to make
>> 	lets say 10 outbound
>> connections a second and then ...
>>
>> Depending on your network policy you could drop or log all other
>> outbound request.
>>
>
> This is what I'm planning to do.. and this is also the reason I was
> asking the questions in the original mail :-)
>
> -- Pasi Kärkkäinen
>
>                                   ^
>                                .     .
>                                 Linux
>                              /    -    \
>                             Choice.of.the
>                           .Next.Generation.

Re: Protecting against DoS

[Arnt Karlsen]

On Tue, 9 Dec 2003 14:20:09 -0500 (PET), 
"Geffrey Velásquez" <g_netfilter@netfids.com> wrote in message 
<36917.200.48.142.50.1070997609.squirrel@www.netfids.com>:

> Hum.. but what happen if the workstation is infected? the virus will
> do a DoS to the user of the workstation, the firewall will block valid
> connections.

.. ;-)  A further refinement could be force a reinstall on the infected
wintendo, or reboot it as a thin client off a boot server, so the user
can do the work he is paid to do.  ;-)

-- 
..med vennlig hilsen = with Kind Regards from Arnt... ;-)
...with a number of polar bear hunters in his ancestry...
  Scenarios always come in sets of three: 
  best case, worst case, and just in case.

Re: Protecting against DoS

[Pasi Kärkkäinen]

On Tue, Dec 09, 2003 at 07:13:22PM +0200, Pasi Kärkkäinen wrote:
> On Tue, Dec 09, 2003 at 10:06:50AM -0700, Michael Gale wrote:
> > Hello,
> > 
> > 	You could try using a rate limit -- you could allow a machine to make lets say 10 outbound 
> > connections a second and then ...
> > 
> > Depending on your network policy you could drop or log all other outbound request.
> > 
> 
> This is what I'm planning to do.. and this is also the reason I was asking
> the questions in the original mail :-)
> 

If you have some facts/suggestions, please comment my original questions.. 

Thanks!

-- Pasi Kärkkäinen
       
                                   ^
                                .     .
                                 Linux
                              /    -    \
                             Choice.of.the
                           .Next.Generation.

Re: Protecting against DoS

[Peter Frischknecht]

Hello,

I wanted to let you know that you are not alone.
We manage networks in apartment complexes.  Hundreds of students use
their computers simultaneously and we have always been able to deal with
DoS attacks.

The attack you described is identical to the one we have suffered since
sometime in September.  In short: it cannot be stopped.  But there are
many things you can do.

As you may have found in personal research, the worm has the following
characteristics:
1 - It is a Zombie.  Computers do NOT automatically start attacking web
sites.  They wait for an instruction from outside.
2 - The connection to the outside "master server" is done via IRC.
3 - The infected computer attacks 1 to a few web servers at a time. (we
have never seen more than 3).
4 - Completely random spoofed src addresses.
5 - The MAC address is (Thank You GOD) not being spoofed.
6 - The intensity of the attack deems the offending computer almost
useless during the attack.

It seemed from your description that your network was fairly small.
In order to save bandwidth, we implement a transparent caching
strategy.  So we redirect port 80 to the cache.  Guess what!!!  Our
caching server ends up the target of the attack!!!

I can tell you that I tried every pertinent module in the patch-o-matic
volume.  It was a 3 month ordeal with very frequent lockups. "connlimit"
does not work because once you enable SYN cookies, there are no
connections to limit.  "limit" draws too much CPU power, eventually
helping lock up the box.

Blocking the foreign IP sources at the FORWARD level (or INPUT in my
case) stopped the packets, but still kept my server with a very high
system use.

I have hundreds of rules in the MANGLE and NAT tables, so the bad
packets were still traveling and being matched against all those rules. 
I placed the drops for the foreign IPs in the MANGLE table.  If you look
at the docs, it is the first table the packet hits.

BTW, to stop foreign IP packets, I created one ACCEP rule for every
known IP class inside my network.  The last line simply DROPs everything
coming from the internal network.

My networks are basically functional now.  But the HIGH CPU use
persists.  It is likely caused by the interrupt handling or the packet
counters/rules checking.  I recently ran accross this page on lowering
the interrupt handling:
ftp://robur.slu.se/pub/Linux/net-development/NAPI/NAPI_HOWTO.txt

Feel free to correspond with me.  We can trade ideas on the topic.

Peter Frischknecht
Empowering Solutions, Inc.

Re: Protecting against DoS

[Pasi Kärkkäinen]

On Sat, Jan 10, 2004 at 08:50:50PM -0500, Peter Frischknecht wrote:
> Hello,
> 
> I wanted to let you know that you are not alone.
> We manage networks in apartment complexes.  Hundreds of students use
> their computers simultaneously and we have always been able to deal with
> DoS attacks.
> 

Hello!


> The attack you described is identical to the one we have suffered since
> sometime in September.  In short: it cannot be stopped.  But there are
> many things you can do.
> 
> As you may have found in personal research, the worm has the following
> characteristics:
> 1 - It is a Zombie.  Computers do NOT automatically start attacking web
> sites.  They wait for an instruction from outside.
> 2 - The connection to the outside "master server" is done via IRC.
> 3 - The infected computer attacks 1 to a few web servers at a time. (we
> have never seen more than 3).
> 4 - Completely random spoofed src addresses.
> 5 - The MAC address is (Thank You GOD) not being spoofed.
> 6 - The intensity of the attack deems the offending computer almost
> useless during the attack.
> 

Yes, sounds like the same thing.

> It seemed from your description that your network was fairly small.
> In order to save bandwidth, we implement a transparent caching
> strategy.  So we redirect port 80 to the cache.  Guess what!!!  Our
> caching server ends up the target of the attack!!!
> 

Hehe.


> I can tell you that I tried every pertinent module in the patch-o-matic
> volume.  It was a 3 month ordeal with very frequent lockups. "connlimit"
> does not work because once you enable SYN cookies, there are no
> connections to limit.  "limit" draws too much CPU power, eventually
> helping lock up the box.
> 

Hmm.. how did you set up the limits? I wrote small application that
calculates the average new connections/sec rates and I then set up the
limits according to that..


> Blocking the foreign IP sources at the FORWARD level (or INPUT in my
> case) stopped the packets, but still kept my server with a very high
> system use.
> 
> I have hundreds of rules in the MANGLE and NAT tables, so the bad
> packets were still traveling and being matched against all those rules. 
> I placed the drops for the foreign IPs in the MANGLE table.  If you look
> at the docs, it is the first table the packet hits.
> 

Yep. It seems to be very difficult to protect against these kind of
attacks.. 

> BTW, to stop foreign IP packets, I created one ACCEP rule for every
> known IP class inside my network.  The last line simply DROPs everything
> coming from the internal network.
> 

I'm doing this also.

> My networks are basically functional now.  But the HIGH CPU use
> persists.  It is likely caused by the interrupt handling or the packet
> counters/rules checking.  I recently ran accross this page on lowering
> the interrupt handling:
> ftp://robur.slu.se/pub/Linux/net-development/NAPI/NAPI_HOWTO.txt
> 
> Feel free to correspond with me.  We can trade ideas on the topic.
> 

Sorry for the long delay in the answer :)

-- Pasi Kärkkäinen
       
                                   ^
                                .     .
                                 Linux
                              /    -    \
                             Choice.of.the
                           .Next.Generation.