Asymmetric routing and connection tracking

Asymmetric routing and connection tracking

[Tore Anderson]

  Hi.  I'm experiencing performance problems I think is related to
 netfilter (the prime suspect is connection tracking) when I have
 asymmetric routing.  My network looks something like this, if
 simplified enough:

    /------------------------+---(virutal router IP)--- servers
    |                        |
   eth2                     eth2
    |                        |
    R1 eth1------------eth1 R2
    |                        |
   eth0                     eth0
    |                        |
    \------------------------+--- transit provider

  R1 and R2 are a redundant router pair, which both get full BGP feeds
 from my transit providers on eth0.  On eth2 there's an access LAN
 (actually there's a lot of these) with servers and so on, and the
 default router address for those servers are present on either R1 and
 R2 (only one at a time).  On eth1 they speak OSPF so that the router
 that does not have the virtual address on eth2 still have a route to
 that subnet (because traffic bound to/from eth2 use connection
 tracking, only the active virtual router have a link-local route to the
 access LAN).

  My prefix is announced to my transit provider using a lower metric
 from R1, so normally inbound traffic is routed to it.  R1 is also the
 default virtual router, so normally R2 rarely see any traffic at all.
 However, if R2 reboots for some reason, R2 will take over the virtual
 router address on eth2, and my transit provider will reroute inbound
 traffic to it.  So far so good.  However, when R1 comes back online,
 I end up in a situation where inbound traffic is sent first to R1, then
 on to R2, out to the servers on the access LAN and then back to R2,
 which then routes the traffic directly out to the transit provider.
 Thus R1 only sees the inbound traffic.

  This worked fine...  until the inbound traffic level exceeds an
 insignificant amount (normally I have around 50-100Mbps, over 50% of
 which is HTTP GET requests, so mostly NEW connections).  I see severe
 packet loss when this happens, which doesn't stop until I either move
 the virtual address back to R1 (or simply shut it down completely).

  My conntrack table size is 0,5M (1 connection pr bucket) - normally
 the table has around 0,2M entries.  But for traffic that pass from
 eth0 to eth1 and vice verca there's no rules that match statefully
 (only simple filtering on src/dest net).

  Has anybody experienced similar problems, or can offer any insight as
 to how to solve it?

Kind regards
-- 
Tore Anderson

Re: Asymmetric routing and connection tracking

[Thomas Jacob]

On Tue, 2007-08-07 at 09:06 +0200, Tore Anderson wrote:
>
>  traffic to it.  So far so good.  However, when R1 comes back online,
>  I end up in a situation where inbound traffic is sent first to R1, then
>  on to R2, out to the servers on the access LAN and then back to R2,
>  which then routes the traffic directly out to the transit provider.
>  Thus R1 only sees the inbound traffic.

Assuming that you do some sort of stateful firewalling
on both R1 and R2, I wonder why this works at all, as
at least netfilter in kernel 2.6 isn't too happy
about only seeing one direction of a particular connection's traffic, 
(this used to work in 2.4 stock kernels).

Apart from that, I'd be interested to know why you
set up this system with BGP and OSPF? If you have
just on upstream transit provider?

Couldn't you simply use a another virtual IP on the
transit provider side of R1 and R2, then ask
your upstream provider to route to
that virtual IP and use some sort of generic
IP-Failover-System that switches internal and external
V-IPs only as a block. This way, you also avoid
your asymmetric routing problem. 

    Thomas

Re: Asymmetric routing and connection tracking

[Tore Anderson]

* Thomas Jacob

> Assuming that you do some sort of stateful firewalling on both R1 and
> R2, I wonder why this works at all, as at least netfilter in kernel
> 2.6 isn't too happy about only seeing one direction of a particular
> connection's traffic, (this used to work in 2.4 stock kernels).

  There are no rules that does stateful matching for traffic that is
 forwarded between two routers.  Such rules are only applied for traffic
 that is forwarded to an access LAN (like eth2).  Traffic that comes in
 from a transit provider and are forwarded directly to another router is
 only filtered based on simple source/destination matches in the IP.

  If such rules do not match, and the router is the last hop before a
 packet reaches its destination, it will apply stateful matching.  But
 this happens only for packets to/from access VLANs like eth2 in my
 drawing.

> Apart from that, I'd be interested to know why you set up this system
> with BGP and OSPF? If you have just on upstream transit provider?

  I have several transit providers, but since it's not really relevant
 for the problem (at least I don't think so) I didn't bother to draw
 them.  You're right that using some form of VRRP could have been a
 possible solution though if I had only one, so I should have mentioned
 it.  Apologies.

  My network is a bit larger than my drawing anyway, and it's not that
 easy (or desireable, for that matter) to make sure packets between two
 hosts take the same route in both directions.  But I would like to be
 able to use stateful firewalling on the routers that have the role as
 access routers too.

Regards
-- 
Tore Anderson

Re: Asymmetric routing and connection tracking

[Thomas Jacob]

>   There are no rules that does stateful matching for traffic that is
>  forwarded between two routers.  Such rules are only applied for traffic
>  that is forwarded to an access LAN (like eth2).  Traffic that comes in
>  from a transit provider and are forwarded directly to another router is
>  only filtered based on simple source/destination matches in the IP.
> 
>   If such rules do not match, and the router is the last hop before a
>  packet reaches its destination, it will apply stateful matching.  But
>  this happens only for packets to/from access VLANs like eth2 in my
>  drawing.

So in the scenario described, R1 wouldn't do any stateful packet
filtering for packets to and from the internal server network? But
the connections will be added to the connection tracking table of
R1 nonetheless (unless you use the NOTRACK target
in raw), only not with an ESTABLISHED state which probably
means they timeout more often than needed and you have
more insert/remove actions over the connection tracking hash
table, maybe that's the source of your problem.

You could try the NOTRACK/raw thing on the (internal-)standby-router,
or you could route the packets of connections that have incoming data
from
R1 back out over R1 with a policy routing setup (+connmark) on R2.

>   I have several transit providers, but since it's not really relevant
>  for the problem (at least I don't think so) I didn't bother to draw
>  them.  You're right that using some form of VRRP could have been a
>  possible solution though if I had only one, so I should have mentioned
>  it.  Apologies.

OK that makes sense, otherwise this would be high
up on my list for "most complicated solution for a simple problem" :-)

  Thomas

Re: Asymmetric routing and connection tracking

[Tore Anderson]

* Thomas Jacob

> So in the scenario described, R1 wouldn't do any stateful packet 
> filtering for packets to and from the internal server network? But 
> the connections will be added to the connection tracking table of R1
> nonetheless (unless you use the NOTRACK target in raw), only not with
> an ESTABLISHED state which probably means they timeout more often
> than needed and you have more insert/remove actions over the
> connection tracking hash table, maybe that's the source of your
> problem.
> 
> You could try the NOTRACK/raw thing on the (internal-)standby-router,

  Hmm.  I was not aware of NOTRACK/raw.  This is very interesting and
 on first look it seems like exactly the thing I needed.  Thank you very
 much for the pointer!  I'll dig into it now.  :-)

Regards
-- 
Tore Anderson