ip contrack problem, not strictly followed RFC, DoS very much possible

ip contrack problem, not strictly followed RFC, DoS very much possible

[Grzegorz Piotr Jaskiewicz]

Hi list

There is little bug, eversince, no author would agree to correct it 
(dunno why) in ip_conntrack_proto_tcp.c:91:
unsigned long ip_ct_tcp_timeout_established =   5 DAYS;

Making it 5 days, makes linux router vournable to (D)DoS attacks. You 
can fill out conntrack hash tables very quickly, making it virtually 
dead. This computer will only respond to direct action, from keyboard, 
com port. This is insane, it just blocks it self, and does nothing, no 
fallback scenario, nothing.
As far as I remember ( I have to look and find exact place where it's 
writen ), RFC specifies this timeout as max 100s. 5 days is insane, and 
no argumentation will explain it. I would suggest changing it to 100 
SECS and remove line:
#define DAYS * 24 HOURS

as it won't be used anymore.

If someone has argumentation for 5 days timeout, please speak out. In 
everyday life, router, desktop, server usage 100s is enough there, and 
makes my life easier, as many other linux admins.

Thanks.

-- 
GJ

Re: ip contrack problem, not strictly followed RFC, DoS very much possible

[Baruch Even]

Grzegorz Piotr Jaskiewicz wrote:
> If someone has argumentation for 5 days timeout, please speak out. In 
> everyday life, router, desktop, server usage 100s is enough there, and 
> makes my life easier, as many other linux admins.

1. The correct value is 5 days for the case of long idle connections, 
there are such things, and there is no reason to force such connections 
to do keepalive every 100 seconds to stay alive.

2. You can change it in your firewall in 
/proc/sys/net/ipv4/netfilter/ip_conntrack_tcp_timeout_established

Baruch

Re: ip contrack problem, not strictly followed RFC, DoS very much possible

[Jose Luis Domingo Lopez]

[-- Attachment #1: Type: text/plain, Size: 1267 bytes --]

On Monday, 06 December 2004, at 14:54:59 +0100,
Grzegorz Piotr Jaskiewicz wrote:

> If someone has argumentation for 5 days timeout, please speak out. In 
> everyday life, router, desktop, server usage 100s is enough there, and 
> makes my life easier, as many other linux admins.
> 
Maybe five days is a bit high, but there are definitely many (maybe badly
designed) applications that expect a TCP connection to remain open (and
traffic not dropped) for much longer than your proposed 100 seconds.

It is not unusual the need to tweak the settings for several commercial
firewalls I work with in several customers to raise the default timeouts
for TCP connection tracking, because some application breaks if the
connection gets put out of the firewalls' connection tables and the
traffic dropped.

Many times is just "my users are too lazy to double click the 'start
connection' icon again when they come from their breakfast, and want to be
able to enter commands on the remote host again". But at least, the
parameter is tunable in recent kernel versions, and not hardcoded in the
kernel sources like it was some time ago.

Greetings.

-- 
Jose Luis Domingo Lopez
Linux Registered User #189436     Debian Linux Sid (Linux 2.6.10-rc3)


[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: ip contrack problem, not strictly followed RFC, DoS very much possible

[Lee Revell]

On Mon, 2004-12-06 at 20:11 +0100, Jose Luis Domingo Lopez wrote:
> It is not unusual the need to tweak the settings for several commercial
> firewalls I work with in several customers to raise the default timeouts
> for TCP connection tracking, because some application breaks if the
> connection gets put out of the firewalls' connection tables and the
> traffic dropped.
> 
> Many times is just "my users are too lazy to double click the 'start
> connection' icon again when they come from their breakfast, and want to be
> able to enter commands on the remote host again". But at least, the
> parameter is tunable in recent kernel versions, and not hardcoded in the
> kernel sources like it was some time ago.

This is not an application problem.  TCP is a reliable, connection
oriented protocol.  Users are right to expect that their TCP connections
do not get timed out by an overzealous firewall because the user did not
type anything in an SSH window for 20 minutes.

The problem here is that your firewall is too dumb or your firewall
rules too aggressive to distinguish a valid connection from a security
threat.  Please don't blame the users.

Lee

Re: ip contrack problem, not strictly followed RFC, DoS very much possible

[Valdis.Kletnieks]

[-- Attachment #1: Type: text/plain, Size: 860 bytes --]

On Mon, 06 Dec 2004 14:54:59 +0100, Grzegorz Piotr Jaskiewicz said:

> There is little bug, eversince, no author would agree to correct it 
> (dunno why) in ip_conntrack_proto_tcp.c:91:
> unsigned long ip_ct_tcp_timeout_established =   5 DAYS;

If you so desire, you can probably workaround this by doing:

echo 100 > /proc/sys/net/ipv4/netfilter/ip_conntrack_tcp_timeout_established

Of course, then if you don't type in an SSH window for 5 minutes, it evaporates
on you - and even SSH keepalives don't help if a router takes a nose dive and
it takes 2 minutes for our NOC to slap it upside the head.  This is a case
*against* keepalives there - if a router hiccups and drops a keepalive on an
otherwise idle session, you nuke a perfectly good idle session for reasons
totally contrary to the original purpose of TCP, namely to *survive* such a
router burp.


[-- Attachment #2: Type: application/pgp-signature, Size: 226 bytes --]

Re: ip contrack problem, not strictly followed RFC, DoS very much possible

[gj]

On Mon, 6 Dec 2004 at 20:48:45, Valdis.Kletnieks@vt.edu wrote:

> On Mon, 06 Dec 2004 14:54:59 +0100, Grzegorz Piotr Jaskiewicz said:
> 
> > There is little bug, eversince, no author would agree to correct it 
> > (dunno why) in ip_conntrack_proto_tcp.c:91:
> > unsigned long ip_ct_tcp_timeout_established =   5 DAYS;
> 
> If you so desire, you can probably workaround this by doing:
> 
> echo 100 >
> /proc/sys/net/ipv4/netfilter/ip_conntrack_tcp_timeout_established
> 
> Of course, then if you don't type in an SSH window for 5 minutes, it
> evaporates
> on you - and even SSH keepalives don't help if a router takes a nose dive
> and
> it takes 2 minutes for our NOC to slap it upside the head.  This is a case
> *against* keepalives there - if a router hiccups and drops a keepalive on
> an
> otherwise idle session, you nuke a perfectly good idle session for reasons
> totally contrary to the original purpose of TCP, namely to *survive* such a
> router burp.

Shouldn't it be protocols thingie to take care about connections ?
Ussualy some protocols are sending ping packet to peer. 
This value as it is now, keeps too many connections in memory, which often leads
to conntrack overflow, that blocks litteraly whole machine up. That is nothing
more than DoS, and besides, there is no fallback routine, something that uppon
error would react. Like, flush very likely to be dead connections, etc.



-- 
Grzegorz Jaskiewicz
K4 Labs

Re: ip contrack problem, not strictly followed RFC, DoS very much possible

[Willy Tarreau]

On Mon, Dec 06, 2004 at 02:54:59PM +0100, Grzegorz Piotr Jaskiewicz wrote:
 
> If someone has argumentation for 5 days timeout, please speak out. In 
> everyday life, router, desktop, server usage 100s is enough there, and 
> makes my life easier, as many other linux admins.

Uhh ?

please try to start 4 xterms through your firewall, then work for 100s in
one of them, then only start to use anyone of the other ones. If it's hung,
maybe you'll remember how low you put the timeout, and think about all those
users coming to you complaining that they only have to stop typing to talk
to someone before they find their xemacs, xterms, etc... dead.

I've seen several firewalls which caused trouble by breaking connections
after 2 hours of idle, while applications used a default keep-alive of 4
hours. Eventhough 5 days may seem a bit long for some usages (eg: frontal
web servers), several hours is not that much for internal firewalls.

Willy

Re: [netfilter-core] ip contrack problem, not strictly followed RFC, DoS very much possible

[Jozsef Kadlecsik]

Hi,

On Mon, 6 Dec 2004, Grzegorz Piotr Jaskiewicz wrote:

> Making it 5 days, makes linux router vournable to (D)DoS attacks. You
> can fill out conntrack hash tables very quickly, making it virtually
> dead. This computer will only respond to direct action, from keyboard,
> com port. This is insane, it just blocks it self, and does nothing, no
> fallback scenario, nothing.

The default hash size is computed very modestly from the size of the
available physical RAM of the machine. If the table fills up too fast,
then increase the hashsize when loading in the ip_conntrack module.
And/or add more RAM to the machine.  And/or setup rate-limiting in the raw
table, before packets hit conntrack itself.

But you are right, we should give more help to overcome disasters. For
example we could search more aggressively for unreplied (i.e droppable)
connections when the table is full, so releasing resources for new
connections (jenkins hash is too good :-). We could also introduce
"reserved conntrack entries", which could be occupied by specific
connections only, to make possible the remote management even when
otherwise the table is full. Just some coding is required :-).

Best regards,
Jozsef
-
E-mail  : kadlec@blackhole.kfki.hu, kadlec@sunserv.kfki.hu
PGP key : http://www.kfki.hu/~kadlec/pgp_public_key.txt
Address : KFKI Research Institute for Particle and Nuclear Physics
          H-1525 Budapest 114, POB. 49, Hungary