Re: ECMP routing: problematic selection of outgoing interface

[Andrew Lunn <andrew@lunn.ch>]

On Wed, May 16, 2018 at 01:51:36AM +0000, Hirotaka Yamamoto wrote:
> Hi,
> 
> Recently I have built a highly-available network using an ECMP
> route connected to two isolated L2 switches as follows.
> 
> Router-- ToR switch 1  ---- Linux
>      |   192.168.11.1/24     |  eth0: 192.168.11.2/24
>      |                       |  eth1: 192.168.12.2/24
>      +-- ToR switch 2  ------+
>          192.168.12.1/24
> 
> The (default) route has been configured with:
> 
>     $ sudo ip route add default \
>            nexthop via 192.168.11.1 \
>            nexthop via 192.168.12.1
> 
> Then I found that Linux chooses a wrong outgoing device for some
> destination/source address pairs like this:
> 
>     $ ip route get 12.34.56.78 from 192.168.12.2:
>     12.34.56.78 from 192.168.12.2 via 192.168.11.1 dev eth0 uid 0
>                                                  # dev should be "eth1"
> 
> As a consequence, programs like SSH or curl do not work for such
> destinations because routers drop packets having strange source
> addresses.

Hi Hirotaka

I assume you add the 192.168.11.1 and 192.168.12.1 to the interfaces
using global scope? Global scope means the IP addresses are valid
everywhere. All routers should know how to route packets to these IP
addresses. So a host is free to pick any of its global scope IP
addresses and use them. The outgoing interface should not matter,
since all routers downstream of it should have routes for the global
scope IP addresses.

It sounds like your router is doing reverse path filtering. It is
checking its routing table for the source address, and throwing the
packets away if they don't come in the interface the route points out
of. If you don't trust your network, this makes sense. It helps to
stop a host spoofing another host, by sending packets with a spoofed
IP address. But you probably want to do reverse path filtering on the
gateway which borders between the networks you do trust and those you
don't.

	Andrew