From mboxrd@z Thu Jan  1 00:00:00 1970
From: "Serguei G. Poltorak" <poltorak@alsenet.com>
Date: Thu, 02 Dec 2010 11:37:13 +0000
Subject: [LARTC] default route with two nexthops and MASQUERADE problem
Message-Id: <2005242609.354.1291289833887.JavaMail.root@z2.alsenet.com>
MIME-Version: 1
Content-Type: multipart/mixed; boundary="===============1858866294=="
List-Id: <lartc.vger.kernel.org>
To: lartc@vger.kernel.org

--===============1858866294==
Content-Type: multipart/alternative; 
	boundary="----=_Part_353_567304406.1291289833886"

------=_Part_353_567304406.1291289833886
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit

Dear all, 

I've the following problem with routing + NAT: 
If I've two ISP and I'm using two nexthop in default route with MASQUERADE on both ISP links, I see routing cache regenerated, but sometimes packets sent to a new link (after cache regeneration) uses wrong source address for masquerading. 

Here is the config. 

I've two links to outside via two different providers: eth1 and eth2 
eth0 is the LAN 

# ip a (part of output, since we have 3 more interfaces disabled) 
2: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast qlen 1000 
link/ether 00:1a:92:9e:66:e8 brd ff:ff:ff:ff:ff:ff 
inet 192.168.1.254/24 brd 192.168.1.255 scope global eth1 
valid_lft forever preferred_lft forever 
3: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast qlen 1000 
link/ether d8:5d:4c:80:6b:2b brd ff:ff:ff:ff:ff:ff 
inet 192.168.2.254/24 brd 192.168.2.255 scope global eth2 
valid_lft forever preferred_lft forever 
6: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast qlen 1000 
link/ether 00:1a:92:9e:76:82 brd ff:ff:ff:ff:ff:ff 
inet 192.168.5.1/24 brd 192.168.5.255 scope global eth0 
valid_lft forever preferred_lft forever 

# ip r (main table) 
192.168.5.0/24 dev eth0 proto kernel scope link src 192.168.5.1 
192.168.2.0/24 dev eth2 proto kernel scope link src 192.168.2.254 
192.168.1.0/24 dev eth1 proto kernel scope link src 192.168.1.254 
default 
nexthop via 192.168.1.1 dev eth1 weight 1 
nexthop via 192.168.2.1 dev eth2 weight 1 

# ip r s t eth1 
default via 192.168.1.1 dev eth1 

# ip r s t eth2 
default via 192.168.2.1 dev eth2 

# ip ru 
0: from all lookup local 
32450: from 192.168.2.254 lookup eth2 
32717: from 192.168.5.124 lookup eth1 
32766: from all lookup main 
32767: from all lookup default 

Q1: if I do pings from two PC in LAN: 5.137 and 5.147, to the same IP how can they go via different links (ping 195.60.x.x is run on both computers)? 

# ip r g 195.60.x.x from 192.168.5.137 iif eth0 
195.60.169.6 from 192.168.5.137 via 192.168.1.1 dev eth1 src 192.168.5.1 
cache <src-direct> mtu 1500 advmss 1460 hoplimit 128 iif eth0 

# ip r g 195.60.x.x from 192.168.5.147 iif eth0 
195.60.169.6 from 192.168.5.147 via 192.168.2.1 dev eth2 src 192.168.5.1 
cache <src-direct> mtu 1500 advmss 1460 hoplimit 128 iif eth0 

The routing in my case should be the same for all users. it shoul send packets to the same destination via the same link always (even if the source IP is different). isn't it? 

Q2: Sometimes I see in tcpdump on external ifaces that the routing cache was regenerated. This can be forced by "ip r f t cache". This sometimes results in change of the link for my pings. But one of two machines suddenly looses connection. After the tcpdump it is because the routing has decided to use another link, but the MASQUERADE was not updated at that time: 

# tcpdump -i eth1 
IP 192.168.2.254 > 195.60.x.x: ICMP echo request, id 10677, seq 242, length 64 (request from .5.147 with wrong source address due to MASQUERADE not updated according to the routing cache purge - hence, no reply, since the source address of the MASQUERADEd packet is wrong) 
IP 192.168.1.254 > 195.60.x.x: ICMP echo request, id 37387, seq 244, length 64 (request from .5.137) 
IP 195.60.x.x > 192.168.1.254: ICMP echo reply, id 37387, seq 244, length 64 

Here is my MASQUERADE setting 
# iptables -L -t nat 
Chain POSTROUTING (policy ACCEPT 752K packets, 48M bytes) 
pkts bytes target prot opt in out source destination 
2840K 256M MASQUERADE all -- any eth1 192.168.5.0/24 anywhere 
2491K 229M MASQUERADE all -- any eth2 192.168.5.0/24 anywhere 


I understand that I can use conntrack to mark packets, but it is a little bit more complicated. I would prefer to use destination IP as the key for routing. What is wrong in this scenario? why routing cache purges does not notify NAT-engine about changes in routing? 

PoltoS 

------=_Part_353_567304406.1291289833886
Content-Type: text/html; charset=utf-8
Content-Transfer-Encoding: quoted-printable

<html><head><style type=3D'text/css'>p { margin: 0; }</style></head><body><=
div style=3D'font-family: Times New Roman; font-size: 12pt; color: #000000'=
>Dear all,<br><br>I've the following problem with routing + NAT:<br>If I've=
 two ISP and I'm using two nexthop in default route with MASQUERADE on both=
 ISP links, I see routing cache regenerated, but sometimes packets sent to =
a new link (after cache regeneration) uses wrong source address for masquer=
ading.<br><br>Here is the config.<br><br>I've two links to outside via two =
different providers: eth1 and eth2<br>eth0 is the LAN<br><br># ip a (part o=
f output, since we have 3 more interfaces disabled)<br>2: eth1: &lt;BROADCA=
ST,MULTICAST,UP,LOWER_UP&gt; mtu 1500 qdisc pfifo_fast qlen 1000<br>&nbsp;&=
nbsp;&nbsp; link/ether 00:1a:92:9e:66:e8 brd ff:ff:ff:ff:ff:ff<br>&nbsp;&nb=
sp;&nbsp; inet 192.168.1.254/24 brd 192.168.1.255 scope global eth1<br>&nbs=
p;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; valid_lft forever preferred_lft forever<br=
>3: eth2: &lt;BROADCAST,MULTICAST,UP,LOWER_UP&gt; mtu 1500 qdisc pfifo_fast=
 qlen 1000<br>&nbsp;&nbsp;&nbsp; link/ether d8:5d:4c:80:6b:2b brd ff:ff:ff:=
ff:ff:ff<br>&nbsp;&nbsp;&nbsp; inet 192.168.2.254/24 brd 192.168.2.255 scop=
e global eth2<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; valid_lft forever pre=
ferred_lft forever<br>6: eth0: &lt;BROADCAST,MULTICAST,UP,LOWER_UP&gt; mtu =
1500 qdisc pfifo_fast qlen 1000<br>&nbsp;&nbsp;&nbsp; link/ether 00:1a:92:9=
e:76:82 brd ff:ff:ff:ff:ff:ff<br>&nbsp;&nbsp;&nbsp; inet 192.168.5.1/24 brd=
 192.168.5.255 scope global eth0<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; va=
lid_lft forever preferred_lft forever<br><br># ip r (main table)<br>192.168=
.5.0/24 dev eth0&nbsp; proto kernel&nbsp; scope link&nbsp; src 192.168.5.1 =
<br>192.168.2.0/24 dev eth2&nbsp; proto kernel&nbsp; scope link&nbsp; src 1=
92.168.2.254 <br>192.168.1.0/24 dev eth1&nbsp; proto kernel&nbsp; scope lin=
k&nbsp; src 192.168.1.254 <br>default <br>&nbsp;&nbsp;&nbsp; nexthop via 19=
2.168.1.1&nbsp; dev eth1 weight 1<br>&nbsp;&nbsp;&nbsp; nexthop via 192.168=
.2.1&nbsp; dev eth2 weight 1<br><br># ip r s t eth1<br>default via 192.168.=
1.1 dev eth1 <br><br># ip r s t eth2<br>default via 192.168.2.1 dev eth2 <b=
r><br># ip ru<br>0:&nbsp;&nbsp;&nbsp; from all lookup local <br>32450:&nbsp=
;&nbsp;&nbsp; from 192.168.2.254 lookup eth2 <br>32717:&nbsp;&nbsp;&nbsp; f=
rom 192.168.5.124 lookup eth1 <br>32766:&nbsp;&nbsp;&nbsp; from all lookup =
main <br>32767:&nbsp;&nbsp;&nbsp; from all lookup default <br><br>Q1: if I =
do pings from two PC in LAN: 5.137 and 5.147, to the same IP how can they g=
o via different links (ping 195.60.x.x is run on both computers)?<br><br># =
ip r g 195.60.x.x from 192.168.5.137 iif eth0<br>195.60.169.6 from 192.168.=
5.137 via 192.168.1.1 dev eth1&nbsp; src 192.168.5.1 <br>&nbsp;&nbsp;&nbsp;=
 cache &lt;src-direct&gt;&nbsp; mtu 1500 advmss 1460 hoplimit 128 iif eth0<=
br><br># ip r g 195.60.x.x from 192.168.5.147 iif eth0<br>195.60.169.6 from=
 192.168.5.147 via 192.168.2.1 dev eth2&nbsp; src 192.168.5.1 <br>&nbsp;&nb=
sp;&nbsp; cache &lt;src-direct&gt;&nbsp; mtu 1500 advmss 1460 hoplimit 128 =
iif eth0<br><br>The routing in my case should be the same for all users. it=
 shoul send packets to the same destination via the same link always (even =
if the source IP is different). isn't it?<br><br>Q2: Sometimes I see in tcp=
dump on external ifaces that the routing cache was regenerated. This can be=
 forced by "ip r f t cache". This sometimes results in change of the link f=
or my pings. But one of two machines suddenly looses connection. After the =
tcpdump it is because the routing has decided to use another link, but the =
MASQUERADE was not updated at that time:<br>&nbsp;<br># tcpdump -i eth1<br>=
IP 192.168.2.254 &gt; 195.60.x.x: ICMP echo request, id 10677, seq 242, len=
gth 64 (request from .5.147 with wrong source address due to MASQUERADE not=
 updated according to the routing cache purge - hence, no reply, since the =
source address of the MASQUERADEd packet is wrong)<br>IP 192.168.1.254 &gt;=
 195.60.x.x: ICMP echo request, id 37387, seq 244, length 64 (request from =
.5.137)<br>IP 195.60.x.x &gt; 192.168.1.254: ICMP echo reply, id 37387, seq=
 244, length 64<br><br>Here is my MASQUERADE setting<br># iptables -L -t na=
t<br>Chain POSTROUTING (policy ACCEPT 752K packets, 48M bytes)<br>&nbsp;pkt=
s bytes target&nbsp;&nbsp;&nbsp;&nbsp; prot opt in&nbsp;&nbsp;&nbsp;&nbsp; =
out&nbsp;&nbsp;&nbsp;&nbsp; source&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp=
;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; destination&nbsp;&nbsp;&nbsp;&n=
bsp;&nbsp;&nbsp;&nbsp;&nbsp; <br>2840K&nbsp; 256M MASQUERADE&nbsp; all&nbsp=
; --&nbsp; any&nbsp;&nbsp;&nbsp; eth1&nbsp;&nbsp;&nbsp; 192.168.5.0/24&nbsp=
;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; anywhere&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp=
;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; <br>2491K&nbsp; 229M MASQUERADE&nbsp; all&n=
bsp; --&nbsp; any&nbsp;&nbsp;&nbsp; eth2&nbsp;&nbsp;&nbsp; 192.168.5.0/24&n=
bsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; anywhere <br><br><br>I understand that I=
 can use conntrack to mark packets, but it is a little bit more complicated=
. I would prefer to use destination IP as the key for routing. What is wron=
g in this scenario? why routing cache purges does not notify NAT-engine abo=
ut changes in routing?<br><br>PoltoS<br></div></body></html>
------=_Part_353_567304406.1291289833886--

--===============1858866294==
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: inline

_______________________________________________
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

--===============1858866294==--