defense against conntrack attacks

defense against conntrack attacks

[Don Cohen]

I noticed in
http://www.linuxvirtualserver.org/Joseph.Mack/HOWTO/LVS-HOWTO-3.html#conntrack
reports that conntrack is a bottleneck.

==== section 1 ====
Here's a summary of some experiments that show this is true and
further suggest that the real expense is in creating new conntrack
records.  If this is a well known phenomenon, then I apologize for the
waste of bandwidth and invite you to skip to section 2.

These were done with 200MHz machines, one generating packets and
the other forwarding.  They're connected by a 100Mbit link.

First test:
 on router
 iptables -A PREROUTING -t mangle -j MARK --set-mark 2
 traffic generator sends 7000 syn's in 2.4 sec (to an address that
  won't reply) all packets the same except randomized destination ports
 result: all marked and forwarded

Second test:
 on router
 iptables -A PREROUTING -t mangle -m state --state new -j MARK --set-mark 1
 traffic generator sends 6131 syn's in 2.06 sec, just like before
 result: 1766 marked new (and I assume forwarded, the total number sent
  out that interface was a little more, I suppose arp, etc.)

This shows that conntrack is a bottleneck, at least dealing with new
packets.

Third test:
 router as in second test
 traffic generator now sends identical syn packets,
  8382 in 2.08 sec
 result: all marked new (and I presume forwarded, again the total
  number sent out that interface is a bit higher)

This shows, I think (please correct me if I'm wrong) that it's
creating new conntrack records (and perhaps garbage collecting old
ones) that is the real bottleneck.

Fourth and fifth tests:
 router as in first and second test
 traffic generator now creates a real connection, just does rcp of
  a large file to a third machine on the other side of the router
 result: no difference, in each case 25MBytes copied in ~4 sec.

Which shows that conntrack is not a bottleneck for packets in
established connections.

==== section 2 ====

So what?  You might argue that 
- in normal usage new connections are not generated very fast 
- if you're syn flooded then you're dead in any case

I argue that you're not necessarily dead in any case.
If the flood really fills up your bandwidth then you are.
But we see above that the amount of syn bandwidth needed to attack a
conntracking machine is far less than it can otherwise handle.
If, in the case above, the attacker could send "only" 1000 syn's/sec
then the attack would work.  But if conntrack could be improved, e.g.,
to make connection creation faster, the same attack would not work.
Also, even if the attacker really can fill the bandwidth between
himself and the firewall, an improved conntrack could maintain
communication between other interfaces. 

I conclude that if the code for creating new connections could be made
faster (it would have to be MUCH faster) there would be some benefit
in doing so.  I've not looked at that.  I presume whoever wrote it was
at least trying to make it efficient.  If anyone thinks this is a
promising approach, please feel free to say so.

What I'm really interested in is a defense that, e.g., rate limits
syns.  The problem I see in the current implementation (again, correct
me if I'm wrong) is that the very first thing that happens when a
packet arrives is that conntrack processes it, and this includes the
code for creating a conntrack record in the case where there is none.

Below is my proposal to fix this.  I hope for replies that address
the following:
- Does this seem like a good approach?
- What would go wrong?
- Any better ideas?

I propose that the code for creating the conntrack record be postponed
until after the point where a defense would have a chance to drop the
packet.

If you recall my posting in April, you won't be surprised that I want
to put the connection creation code well after the current postrouting
hook.  After that hook the packet is enqueued for traffic control, and
it can be dropped between then and when it is dequeued to be sent.  So
I'd like to create a new hook after that dequeue, and this is where I
propose to create the connection record.

It occurs to me that there may be code now that assumes that the
record will exist, e.g., when you try to do something like the
 -m state --state new
above.  If so, that code would have to be fixed, e.g., to use a
default conntrack record if none exists.  But I'd expect that
conntrack records might be garbage collected at arbitrary times,
including between the time of arrival of the packet that created them
and the time that packet goes through some later hook.  So perhaps
this is not a problem.

Another possible problem I see is that creating the connection record
might require data that is no longer available at the later point,
such as the IP addresses in the packet when it arrived (which might by
now be altered by NAT).  I recall this also arose in the April
exchange.  I proposed then that it would be useful to save a little
"historical" data per skbuff for use by later netfilter hooks.  I
propose now that this should specifically include source/dest IP
addresses and ports.  I think I mentioned before that it should
include the device where the packet arrived.

I look forward to your replies.

Re: defense against conntrack attacks

[Harald Welte]

[-- Attachment #1: Type: text/plain, Size: 5245 bytes --]

On Sat, Jun 15, 2002 at 03:38:22PM -0700, Don Cohen wrote:

> Here's a summary of some experiments that show this is true and
> further suggest that the real expense is in creating new conntrack
> records.  If this is a well known phenomenon, then I apologize for the
> waste of bandwidth and invite you to skip to section 2.

thanks, this is the kind of testing/benchmarkin which is appreciated.

> These were done with 200MHz machines, one generating packets and
> the other forwarding.  They're connected by a 100Mbit link.
> 
> First test:
>  on router
>  iptables -A PREROUTING -t mangle -j MARK --set-mark 2
>  traffic generator sends 7000 syn's in 2.4 sec (to an address that
>   won't reply) all packets the same except randomized destination ports
>  result: all marked and forwarded
> 
> Second test:
>  on router
>  iptables -A PREROUTING -t mangle -m state --state new -j MARK --set-mark 1
>  traffic generator sends 6131 syn's in 2.06 sec, just like before
>  result: 1766 marked new (and I assume forwarded, the total number sent
>   out that interface was a little more, I suppose arp, etc.)
> 
> This shows that conntrack is a bottleneck, at least dealing with new
> packets.

there are a couple of issues related to this:

1 the number of hash buckets in your conntrack hash table (this is a module
  loadtime parameter and the default depends on your systems total RAM size)
2 the conntrack hash function.  If all (or more/most) connections would end
  up in the same couple of buckets, performance would suffer significantly.
3 did you see any messages in your kernel log ?  any OOM messages?
  I could imagine that the kernel has difficulties allocating the needed
  memory for conntrack entries

> Third test:
>  router as in second test
>  traffic generator now sends identical syn packets,
>   8382 in 2.08 sec
>  result: all marked new (and I presume forwarded, again the total
>   number sent out that interface is a bit higher)
> 
> This shows, I think (please correct me if I'm wrong) that it's
> creating new conntrack records (and perhaps garbage collecting old
> ones) that is the real bottleneck.

yes, the results seem to 

> Fourth and fifth tests:
>  router as in first and second test
>  traffic generator now creates a real connection, just does rcp of
>   a large file to a third machine on the other side of the router
>  result: no difference, in each case 25MBytes copied in ~4 sec.
> 
> Which shows that conntrack is not a bottleneck for packets in
> established connections.

it's only a very slight decrease in performance in the case of established
connections, yes.  This decrease tends to be outweighed by the performance
gain through significantly reduced complexity of the packet filtering ruleset
with stateful firewalling.

> So what?  You might argue that 
> - in normal usage new connections are not generated very fast 

depends on your usage.  7000 syn's in 2 sec is not so unrealistic in front
of huge webservers.  Imagine somebody flooding you at wirespeed 100mbit
ethernet... 147k packets [which is a problem only for routing, not even
talking about firewalling/conntracking/...].

> I conclude that if the code for creating new connections could be made
> faster (it would have to be MUCH faster) there would be some benefit
> in doing so.  I've not looked at that.  I presume whoever wrote it was
> at least trying to make it efficient.  If anyone thinks this is a
> promising approach, please feel free to say so.

the question would be:  what exactly is so slow? why is it slow?

> What I'm really interested in is a defense that, e.g., rate limits
> syns.  The problem I see in the current implementation (again, correct
> me if I'm wrong) is that the very first thing that happens when a
> packet arrives is that conntrack processes it, and this includes the
> code for creating a conntrack record in the case where there is none.

yes, I think Jozsef is working on a 'conntrack exemption table' where you
could tell it that certain [maybe above a certain limit] packets are 
not conntracked.

> If you recall my posting in April, you won't be surprised that I want
> to put the connection creation code well after the current postrouting
> hook.  After that hook the packet is enqueued for traffic control, and
> it can be dropped between then and when it is dequeued to be sent.  So
> I'd like to create a new hook after that dequeue, and this is where I
> propose to create the connection record.

This sounds odd.  First, this is too revolutionary to the overall concept.

Second, you need conntrack information before postrouting:

- for doing DNAT [which _needs_ to happen before the routing decision]
- for any stateful rules in the FORWARD chain of the filter table
- for any stateful rules in the INPUT chain of the filter table

-- 
Live long and prosper
- Harald Welte / laforge@gnumonks.org               http://www.gnumonks.org/
============================================================================
GCS/E/IT d- s-: a-- C+++ UL++++$ P+++ L++++$ E--- W- N++ o? K- w--- O- M+ 
V-- PS++ PE-- Y++ PGP++ t+ 5-- !X !R tv-- b+++ !DI !D G+ e* h--- r++ y+(*)

[-- Attachment #2: Type: application/pgp-signature, Size: 232 bytes --]

Re: defense against conntrack attacks

[Henrik Nordstrom]

On Sunday 16 June 2002 22.16, Harald Welte wrote:

> > If you recall my posting in April, you won't be surprised that I
> > want to put the connection creation code well after the current
> > postrouting hook.  After that hook the packet is enqueued for
> > traffic control, and it can be dropped between then and when it
> > is dequeued to be sent.  So I'd like to create a new hook after
> > that dequeue, and this is where I propose to create the
> > connection record.
>
> This sounds odd.  First, this is too revolutionary to the overall
> concept.
>
> Second, you need conntrack information before postrouting:
>
> - for doing DNAT [which _needs_ to happen before the routing
> decision] - for any stateful rules in the FORWARD chain of the
> filter table - for any stateful rules in the INPUT chain of the
> filter table


This proposal reminds me a lot of a similar proposal I made some 
(many) months ago.

Instead of creating the conntrack entry upon entry, create it "just in 
time", i.e. when first code requiring a conntrack entry is executed. 
Simply put any call to ip_conntrack_get().

This would be

  Any NAT target
  Any state/conntrack match
  etc..
  with a counter-measure in the nat table to not cause it to 
automatically initiate tracking (today calls ip_conntrack_get to tell 
if the packet should enter the table or not).

This approach makes the "conntrack exception" quite natural in most 
ruleset by my opinion; if you don't have any state or NAT rules 
touching the connection then it won't be tracked, but adds a bit of 
complexity to the design and collides with the sub-goal of being able 
to later add NAT rules without the risk of disturbing existing 
connections.

After our discussions the last time manual excepmtion is probably 
preferable I think. Less risk of confusion over what is being tracked 
and what not, and considerably less of a design change. But I have 
not yet fully given up the "just in time" idea.

Regards
Henrik

Re: defense against conntrack attacks

[Don Cohen]

Harald Welte writes:
 > On Sat, Jun 15, 2002 at 03:38:22PM -0700, Don Cohen wrote:
 > 
 > > Here's a summary of some experiments that show this is true ...
 > 
 > thanks, this is the kind of testing/benchmarkin which is appreciated.
 > 
 > > These were done with 200MHz machines, one generating packets and
 > > the other forwarding.  They're connected by a 100Mbit link.
I should also have mentioned, 2.4.3 kernel.
The code might have changed a fair amount since then.

 ...
 > > This shows that conntrack is a bottleneck, at least dealing with new
 > > packets.
 > 
 > there are a couple of issues related to this:
 > 
 > 1 the number of hash buckets in your conntrack hash table (this is a module
 >   loadtime parameter and the default depends on your systems total RAM size)
 > 2 the conntrack hash function.  If all (or more/most) connections would end
 >   up in the same couple of buckets, performance would suffer significantly.
 > 3 did you see any messages in your kernel log ?  any OOM messages?
 >   I could imagine that the kernel has difficulties allocating the needed
 >   memory for conntrack entries

The only log messages I see that appear to be related are:
Jun 14 14:52:25 router1 kernel: ip_tables: (c)2000 Netfilter core team
and
Jun 14 15:01:10 router1 kernel: ip_conntrack (1024 buckets, 8192 max)
(I see several of those since I rmmod'ed and reloaded conntrack
several times).
I assumed without looking that the hash function would be mostly
different when only the destination port changed, which is what I was
doing.  Attacking the hash function is yet another approach that I was
planning to worry about, but I haven't gotten there yet.
(Anyone have ideas about defense against such an attack?  I have a few
that seem pretty obvious.) 

 > > I conclude that if the code for creating new connections could be made
 > > faster (it would have to be MUCH faster) there would be some benefit
 > > in doing so.  I've not looked at that.  I presume whoever wrote it was
 > > at least trying to make it efficient.  If anyone thinks this is a
 > > promising approach, please feel free to say so.
 > 
 > the question would be:  what exactly is so slow? why is it slow?

Of course, anyone who knows how to improve the code should please
speak up.  I still hope to read and understand it, but if/when I get
there I expect I'll then understand why it has to take as long as it
does as opposed to seeing how to fix it.

 > > What I'm really interested in is a defense that, e.g., rate limits
 > > syns.  The problem I see in the current implementation (again, correct
 > > me if I'm wrong) is that the very first thing that happens when a
 > > packet arrives is that conntrack processes it, and this includes the
 > > code for creating a conntrack record in the case where there is none.
 > 
 > yes, I think Jozsef is working on a 'conntrack exemption table' where you
 > could tell it that certain [maybe above a certain limit] packets are 
 > not conntracked.

Isn't that disastrous?  For instance, NAT can't work in that case,
right?  And any rules that refer to conntrack state.  Unless you just
drop the packets, of course.  And yes, that's what I want to do, but
of course I have some ideas about how to decide WHICH packets to drop.

 > Second, you need conntrack information before postrouting:
 > 
 > - for doing DNAT [which _needs_ to happen before the routing decision]
 > - for any stateful rules in the FORWARD chain of the filter table
 > - for any stateful rules in the INPUT chain of the filter table

The question is how much data is needed for those things.  A lot
depends on where the actual expense is, which I don't yet know.

I'll mention my other plan, which I was beginning to like better anyway.
It's a version of IMQ that runs before conntrack and rate limits the
"expensive" packets.  This could even read conntrack state in order to
decide which packets were expensive.