Re: ip_route_output_key returns wrong gateway info with specific ip rules

["Maciej Żenczykowski" <zenczykowski@gmail.com>]

(guesswork, read it with a grain of salt)

In my experience stuff like this usually ends up being caused by the
order in which src ip selection (for locally generated packets without
an explicit bind(ip) call, but thus also src ip selection for auto
nat), route lookup and nat happen.
Easiest fix has been to use iptables mangle prerouting to mark
packets, thus also forcing a reroute, and then using fwmark ip rules
instead of (or maybe in addition to) from ip rules.

In your particular case I'm guessing the route lookup is happening
post-source-nat src ip substitution (or even with a 0 during nat sec
ip selection).  While theoretically SNAT happens in POSTROUTING, and
thus after routing, I think this is only truly the case for the first
packet of a flow.

To be fair my info may be long obsolete, since I most recently (1-2
weeks ago) ran into something like this on some 2.4 kernel (wrt54gl
openwrt 8.09.2) while trying to use a different src ip for SNAT to
port 80/443 then for all other ports.
In my case iptables mangle prerouting (ip rule fwmark lookup) was used
to mark SYNs from local client ip to dest port 80/443, and then src ip
routing (ip rule from X) worked for everything else (ie. the rest of
the tcp connection).
Although that was with dst port based SNAT rules and not MASQUERADE.

- Maciej