Problem with ipconntrack+snat and changing routes

Problem with ipconntrack+snat and changing routes

[Joshua Snyder]

I have ran into this more than once and I just wanted to know if anyone 
else has ever seen this; I know everything I am talking about it valid 
for 2.4 kernels.  I don't think this is different in 2.6 but it might be.

What seems to be going on is the following.  Say we have a machine with 
two connections to the Internet and we are doing Nat on both of the 
external interfaces.  Something like...

                                          Eth2
                    Eth0  *-----------*--------> Cable 12.23.32.15
Internal network --------| Linux box |
  192.168.0.0/24          *-----------*--------> Dsl 34.36.23.89
                                          Eth1

Then we setup some Source Nat like...

# Snat for Cable
iptables -t nat -A POSTROUTING -o eth2 -j SNAT --to 12.23.32.15
# Snat for Dsl
iptables -t nat -A POSTROUTING -o eth1 -j SNAT --to 34.36.23.89

This works just fine... now here is where things start breaking.  If you 
use routing tables to statically put traffic onto one or the other 
external links things work as you would expect.  But you try to do route 
load balancing or you change which link your transmitting packets on you 
can end-up with very odd results.

So, if we do this to load balance traffic...

ip route add default scope global \
	nexthop via 12.23.32.1 dev eth2 weight 1 \
	nexthop via 34.36.23.1 dev eth1 weight 1

Now we are doing route based load balancing.  And everything will work 
fine until the route cache for a tracked connection expires.  For 
instance if you ssh into a machine but you go idle for an hour or so... 
Then what happens is the following, the Linux box will do a new route 
lookup if the new route lookup matches what you had before everything 
will still work fine.  But if the machine picks the other interface as 
the new route, packets will get sent out on the wrong interface still 
snated with the ip address of the other interface.

For example what you see is this... In the example above if you had a 
connection that was originally using eth2 so it was getting nated to 
12.23.32.15.  But if later on the route changes so your now going out 
eth1 you will still be transmitting packets with the Ip address of 
12.23.32.15 on your DSL link which has an ip address of 34.36.23.89. 
Most of the time this traffic will be dropped by your Isp.

Now, I understand that you can't just start sending packets for this 
connection with a new ip address.  But it doesn't sound like we should 
be sending these packets down the wrong interface.  I am not sure what 
the best thing to do in this case is... The above example is just one of 
a few ways you can see this problem(manually changing routing and 
Ospf/Bgp for example).  The worst case is if you have a routing change 
but your conntrack doesn't expire and routing doesn't change again after 
the first time... the only way to get stuff to work again is to unload 
ip_conntrack from the kernel.  I have ran into these issues a few times 
now and I was just wondering if anyone else has seen this or how SNAT 
should handle these cases.

			Josh

Re: Problem with ipconntrack+snat and changing routes

[Alexander Samad]

[-- Attachment #1: Type: text/plain, Size: 4375 bytes --]

On Thu, Nov 03, 2005 at 05:42:52PM -0500, Joshua Snyder wrote:
> I have ran into this more than once and I just wanted to know if anyone 
> else has ever seen this; I know everything I am talking about it valid 
> for 2.4 kernels.  I don't think this is different in 2.6 but it might be.
> 
> What seems to be going on is the following.  Say we have a machine with 
> two connections to the Internet and we are doing Nat on both of the 
> external interfaces.  Something like...
> 
>                                          Eth2
>                    Eth0  *-----------*--------> Cable 12.23.32.15
> Internal network --------| Linux box |
>  192.168.0.0/24          *-----------*--------> Dsl 34.36.23.89
>                                          Eth1
> 
> Then we setup some Source Nat like...
> 
> # Snat for Cable
> iptables -t nat -A POSTROUTING -o eth2 -j SNAT --to 12.23.32.15
> # Snat for Dsl
> iptables -t nat -A POSTROUTING -o eth1 -j SNAT --to 34.36.23.89
> 
> This works just fine... now here is where things start breaking.  If you 
> use routing tables to statically put traffic onto one or the other 
> external links things work as you would expect.  But you try to do route 
> load balancing or you change which link your transmitting packets on you 
> can end-up with very odd results.
> 
> So, if we do this to load balance traffic...
> 
> ip route add default scope global \
> 	nexthop via 12.23.32.1 dev eth2 weight 1 \
> 	nexthop via 34.36.23.1 dev eth1 weight 1

Hi

I have this setup and don;t experience the problem you have outline, I
think because you are missing a bit of the puzzle, you also have to
setup ip rules as well in your case something like

ip ru add from 12.23.32.15 lookup 200
ip ru add from 34.36.23.89 lookup 201

and then placing routing info in those route tables for each specific
link for example for the cable

ip route add defalt via 12.23.32.1 dev eth 2 tab 200

The issue is that conntrack should keep a record of the open connection
and will presumably do the SNAT'ing before it hits the SNAT entries in
the nat table as it is an already open connection.

The rules above state that if the packet has a src address of
12.23.32.15 only use the route table 200 which only routes out the cable
modem.

For normal ip asymetrical routing would not be a problem, but most ISP
block these packets, so in your example below the packet is still going
out but with the cable modem address but out the adsl network and being
blocked by the adsl ISP.


Alex

> 
> Now we are doing route based load balancing.  And everything will work 
> fine until the route cache for a tracked connection expires.  For 
> instance if you ssh into a machine but you go idle for an hour or so... 
> Then what happens is the following, the Linux box will do a new route 
> lookup if the new route lookup matches what you had before everything 
> will still work fine.  But if the machine picks the other interface as 
> the new route, packets will get sent out on the wrong interface still 
> snated with the ip address of the other interface.
> 
> For example what you see is this... In the example above if you had a 
> connection that was originally using eth2 so it was getting nated to 
> 12.23.32.15.  But if later on the route changes so your now going out 
> eth1 you will still be transmitting packets with the Ip address of 
> 12.23.32.15 on your DSL link which has an ip address of 34.36.23.89. 
> Most of the time this traffic will be dropped by your Isp.
> 
> Now, I understand that you can't just start sending packets for this 
> connection with a new ip address.  But it doesn't sound like we should 
> be sending these packets down the wrong interface.  I am not sure what 
> the best thing to do in this case is... The above example is just one of 
> a few ways you can see this problem(manually changing routing and 
> Ospf/Bgp for example).  The worst case is if you have a routing change 
> but your conntrack doesn't expire and routing doesn't change again after 
> the first time... the only way to get stuff to work again is to unload 
> ip_conntrack from the kernel.  I have ran into these issues a few times 
> now and I was just wondering if anyone else has seen this or how SNAT 
> should handle these cases.
> 
> 			Josh
> 
> 

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]