Re: Wrong routing when combining ip rule with SNAT

[Nikolaus Rath <Nikolaus@rath.org>]

Nikolaus Rath <Nikolaus@rath.org> writes:
> Vigneswaran R <vignesh@atc.tcs.com> writes:
>> On 09/13/2013 10:40 AM, Nikolaus Rath wrote:
>>> I've enabled packet forwarding and SNAT on the "ebox" computer as
>>> follows:
>>>
>>> root@ebox:~# ip route
>>> default via 23.92.25.1 dev eth0
>>> 23.92.25.0/24 dev eth0  proto kernel  scope link  src 23.92.25.96
>>> 192.168.12.0/24 dev rath  proto kernel  scope link  src 192.168.12.1
>>>
>>> root@ebox:~# iptables  -L -n -v
>>> Chain INPUT (policy ACCEPT 1314 packets, 1736K bytes)
>>>   pkts bytes target     prot opt in     out     source               destination
>>>
>>> Chain FORWARD (policy DROP 0 packets, 0 bytes)
>>>   pkts bytes target     prot opt in     out     source               destination
>>>   150K   62M ACCEPT     all  --  rath   eth0    0.0.0.0/0            0.0.0.0/0
>>> 86746  200M ACCEPT     all  --  eth0   rath    0.0.0.0/0            0.0.0.0/0            state RELATED,ESTABLISHED
>>>    319 22076 LOG        all  --  *      *       0.0.0.0/0            0.0.0.0/0            limit: avg 1/min burst 30 LOG flags 0 level 4 prefix "Rejected forwarding: "
>>>    393 26172 REJECT     all  --  *      *       0.0.0.0/0            0.0.0.0/0            reject-with icmp-net-prohibited
>>>
>>> Chain OUTPUT (policy ACCEPT 1142 packets, 2412K bytes)
>>>   pkts bytes target     prot opt in     out     source destination
>>>   root@ebox:~# iptables -t nat -L -n -v
>>> Chain PREROUTING (policy ACCEPT 36378 packets, 2383K bytes)
>>>
>>> Chain INPUT (policy ACCEPT 19982 packets, 1334K bytes)
>>>   pkts bytes target     prot opt in     out     source               destination
>>>
>>> Chain OUTPUT (policy ACCEPT 61430 packets, 4601K bytes)
>>>   pkts bytes target     prot opt in     out     source               destination
>>>
>>> Chain POSTROUTING (policy ACCEPT 8333 packets, 564K bytes)
>>>   pkts bytes target     prot opt in     out     source               destination
>>> 69488 5081K SNAT       all  --  *      eth0    0.0.0.0/0            0.0.0.0/0            to:23.92.25.96
>>>
>>>     >From a second computer "vostro", I can now use ebox as a
>>> gateway:
>>>
>>> root@vostro:~# ip route add 190.93.249.164 via 192.168.12.1
>>>
>>> This works fine, now connections to whatismyip.com (190.93.249.164) go
>>> through ebox.
>>>
>>> However, when I try to be a bit more selective on vostro and use a
>>> special routing table, things don't work anymore:
>>>
>>> root@vostro:~# iptables -t mangle -L -n
>>> Chain PREROUTING (policy ACCEPT)
>>> target     prot opt source               destination
>>>
>>> Chain INPUT (policy ACCEPT)
>>> target     prot opt source               destination
>>>
>>> Chain FORWARD (policy ACCEPT)
>>> target     prot opt source               destination
>>>
>>> Chain OUTPUT (policy ACCEPT)
>>> target     prot opt source               destination
>>> MARK       tcp  --  0.0.0.0/0            190.93.249.164       tcp dpt:80 MARK set 0x1
>>> LOG        tcp  --  0.0.0.0/0            190.93.249.164       tcp dpt:80 LOG flags 0 level 4 prefix "marked: "
>>>
>>> Chain POSTROUTING (policy ACCEPT)
>>> target     prot opt source               destination
>>>
>>> root@vostro:~# ip route del 190.93.249.164 via 192.168.12.1
>>> root@vostro:~# ip route add default via 192.168.12.1 table tovpn
>>> root@vostro:~# ip rule add fwmark 0x1 table tovpn
>>>
>>> Now connections from vostro to 190.93.249.164 still make it to ebox, and
>>> from ebox to 190.93.249.164, but the answers get stuck on ebox:
>>>
>>> Sep 13 04:47:53 ebox kernel: Rejected forwarding: IN=eth0 OUT=eth0 MAC=f2:3c:91:69:db:07:84:78:ac:0d:79:c1:08:00 SRC=190.93.249.164 DST=192.168.17.47 LEN=60 TOS=0x00 PREC=0x00 TTL=58 ID=0 DF PROTO=TCP SPT=80 DPT=39024 WINDOW=14480 RES=0x00 ACK SYN URGP=0
>>>
>>> It seems that ebox tries to send the packet destined to go trough the
>>> rath to eth0 instead, and consequency rejects them because forwarding is
>>> only enabled from eth0 to rath.
>>>
>>> However, this only happens when vostro has the gateway route set in a
>>> special routing table rather than the default table -- but how does ebox
>>> even know about that?
>>>
>>> Can someone explain to me what is happening here and why?
>>
>> I have a doubt. It seems, rath of ebox is assigned with IP address in
>> the range 192.168.12.0/24.
>> However, IP address of vostro seems to be
>> 192.168.17.47 (assuming /24). Ebox doesn't have any route to this
>> range. So it try to use default route via eth0.
>>
>> What I assume is, 'vostro' has IP addresses in (atleast) two ranges
>> (192.168.12.0/24, 192.168.17.0/24).
>
> That's correct.
>
> nikratio@vostro:~$ ip addr
> 5: br0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP 
>     link/ether c8:60:00:bf:a2:7f brd ff:ff:ff:ff:ff:ff
>     inet 192.168.17.47/24 brd 192.168.17.255 scope global br0
>        valid_lft forever preferred_lft forever
>     inet6 fe80::ca60:ff:febf:a27f/64 scope link 
>        valid_lft forever preferred_lft forever
> 6: rath: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN qlen 500
>     link/none 
>     inet 192.168.12.4/24 scope global rath
>        valid_lft forever preferred_lft forever
>
>> In the default routing table, the src IP is set to 192.168.12.x (for
>> the packets originating from vostro). However, the 'tovpn' table
>> didn't specify the src IP. So, when the 'tovpn' table is being used,
>> the packets may have got the src IP as 192.168.17.x.
>
> Hmm. This would make sense, but looking at the default table, the source
> address for the route via 192.168.12.x is actually also not set:
>
> nikratio@vostro:~$ ip route
> default via 192.168.17.1 dev br0 
> 190.93.249.164 via 192.168.12.1 dev rath 
> 192.168.12.0/24 dev rath  proto kernel  scope link  src 192.168.12.4 
> 192.168.17.0/24 dev br0  proto kernel  scope link  src 192.168.17.47 
>
> This works just fine, despite the entry having no source address. So why
> is it working in the default table, but not in the tovpn table?
>
>
>> I think, you can avoid this by explicitly specifying the src IP when
>> adding the route to 'tovpn' table,
>>
>>     ip route add default via 192.168.12.1 src 192.168.12.x table tovpn
>
>
> I'll of course try this nevertheless, thanks!

Nope, this doesn't help. It's still going out with the wrong src:

root@vostro:~# ip route list table tovpn
190.93.249.164 via 192.168.12.1 dev rath  src 192.168.12.2

root@ebox:~# tail /var/log/kern.log
Sep 13 21:28:57 ebox kernel: Rejected forwarding: IN=eth0 OUT=eth0 MAC=f2:3c:91:69:db:07:84:78:ac:0d:79:c1:08:00 SRC=91.189.89.199 DST=192.168.17.47 LEN=76 TOS=0x00 PREC=0x00 TTL=47 ID=0 DF PROTO=UDP SPT=123 DPT=123 LEN=56 


Any ideas?

Best,
Nikolaus

-- 
Encrypted emails preferred.
PGP fingerprint: 5B93 61F8 4EA2 E279 ABF6  02CF A9AD B7F8 AE4E 425C

             »Time flies like an arrow, fruit flies like a Banana.«