SNAT and multiple ISP problem

SNAT and multiple ISP problem

[Maxim Koshelev]

Hi all,
I have a problem that I can't find working way to do some kind of load
balance beetween two internet channels using SNAT. I've tried various
schemes to do this but in all of them kernel makes wrong routing.
For example if I try to add such rules in nat:

iptables -t nat -A POSTROUTING -o eth0 -j SNAT --to $ETH0IP
iptables -t nat -A POSTROUTING -o ppp0 -j SNAT --to $PPP0IP

kernel will do SNAT only according default route in table main. It is
ok. But if I try to change default routing selection (e.g. by adding
in 'mangle' table some marks and adding 'ip rules' with this marks)
kernel begins to route some packets nated to ppp0 into eth0 and
vice-versa! I can simple monitor this ugly behavior using tcpdump. It
brings to lost packets and connections drops ( because one of IP's is
192.168... but second one is real )

Is there any way to make right routing?

Re: SNAT and multiple ISP problem

[Vlado Drz(ík]

Maxim Koshelev  wrote / napísal(a):
> Hi all,
> I have a problem that I can't find working way to do some kind of load
> balance beetween two internet channels using SNAT. I've tried various
> schemes to do this but in all of them kernel makes wrong routing.
> For example if I try to add such rules in nat:
> 
> iptables -t nat -A POSTROUTING -o eth0 -j SNAT --to $ETH0IP
> iptables -t nat -A POSTROUTING -o ppp0 -j SNAT --to $PPP0IP
> 
> kernel will do SNAT only according default route in table main. It is
> ok. But if I try to change default routing selection (e.g. by adding
> in 'mangle' table some marks and adding 'ip rules' with this marks)
> kernel begins to route some packets nated to ppp0 into eth0 and
> vice-versa! I can simple monitor this ugly behavior using tcpdump. It
> brings to lost packets and connections drops ( because one of IP's is
> 192.168... but second one is real )
> 
> Is there any way to make right routing?

Please post your routing table and mark setup. For me it seems that you
don't have routing tables setup up correctly (or is not doing what you
want). For example imcomming packets that are marked are using
alternative routing table and that doesn't include your internal network
routes (and so is sent outside to default route after un-NATing).

Regards,
Vlado.

Re: SNAT and multiple ISP problem

[Vlado Držík]

Покотиленко Костик  wrote / napísal(a):
> В Вск, 26/04/2009 в 17:45 +0400, Maxim Koshelev пишет:
>> Hi all,
>> I have a problem that I can't find working way to do some kind of load
>> balance beetween two internet channels using SNAT. I've tried various
>> schemes to do this but in all of them kernel makes wrong routing.
>> For example if I try to add such rules in nat:
>>
>> iptables -t nat -A POSTROUTING -o eth0 -j SNAT --to $ETH0IP
>> iptables -t nat -A POSTROUTING -o ppp0 -j SNAT --to $PPP0IP
>>
>> kernel will do SNAT only according default route in table main. It is
>> ok. But if I try to change default routing selection (e.g. by adding
>> in 'mangle' table some marks and adding 'ip rules' with this marks)
>> kernel begins to route some packets nated to ppp0 into eth0 and
>> vice-versa! I can simple monitor this ugly behavior using tcpdump. It
>> brings to lost packets and connections drops ( because one of IP's is
>> 192.168... but second one is real )
>>
>> Is there any way to make right routing?
> 
> One hint: when dialing with connctrack you should remember that seen
> connections get remembered, and when you play with routing some
> connections had already been natted to different address. So, when you
> dinamically change routing just to make sure do "conntrack -F conntrack
> && conntrack -F expect".
> 

More precisely you have to flush route cache also after such change (if
this is problem).
# ip route flush cache

Example of route cache.
# ip route show cache
192.43.172.30 from 62.200.200.10 via 62.200.200.1 dev eth1
    cache  mtu 1500 advmss 1460 metric10 64
10.1.1.53 from 66.152.85.206 dev eth0  src 62.200.200.10
    cache  mtu 1500 advmss 1460 metric10 64 iif eth1
67.63.48.89 from 62.200.200.10 via 62.200.200.1 dev eth1
    cache  ipid 0xbb81 mtu 1500 advmss 1460 metric10 64
193.100.100.2 from 192.100.100.50 dev eth2
    cache  mtu 1500 advmss 1460 metric10 64
local 10.1.1.2 from 10.1.1.53 dev lo  src 10.1.1.2
    cache <local,src-direct>  iif eth0