policy routing vs dnat replies

policy routing vs dnat replies

[Andy Lutomirski]

I hit an annoying policy routing corner case today.  I have a router
with two WAN interfaces (and no BGP).  I have policy routing set up so
that, if a source address matches either of my public networks, then
outgoing packets use the correct interface.  If neither rule matches
(e.g. the source is 0.0.0.0 for source address selection), then the
default route is whichever one I prefer at the moment.  It looks like
this:

$ ip rule
0:	from all lookup local
32766:	from all lookup main
32767:	from all lookup default
40000:	from <net2> lookup isp2
40001:	from <net1> lookup isp1
40010:	from all lookup real_default

The relevant routes are:

default via <gw1> dev eth0.2  table isp1  src <src1>
default via <gw2> dev eth0.3  table isp2  src <src2>
default via <gw2> dev eth0.3  table real_default  src <src2>  metric 101
default via <gw1> dev eth0.2  table real_default  src <src1>  metric 102

(Yes, this is a bit verbose, but I don't know a more concise way to do this.)

This works nicely: if I specifically bind to one of my public
addresses, the corresponding WAN link is used, and if not or if I'm
coming from a private address, then the metrics determine which link
to use.

DNAT breaks it.  I have a rule:
-A PREROUTING -i eth0.2 -d <ip1> -p tcp --dport <port> -j DNAT --to
<internal host>

<ip1> lives on isp1.  Someone sends a SYN.  It gets routed to the
internal host, and that host sends a SYN/ACK back.  The SYN/ACK has a
source ip that isn't on net1 or net2, so it matches the 'lookup
real_default' rule and gets routed to *gw2*.  iptables rewrites the
source address after the routing decision, and my router sends a
packet with a source address belonging to isp1 to isp2's gateway.  The
packet is then dropped.

Is there any way I can either convince iptables to rewrite the source
address in the prerouting hook or to query the conntrack source
address from the policy rule?  Is there a better solution?  I'm
currently using a somewhat gross combination of MARK and fwmark
matches to work around this problem.  One possibility would be:

Thanks,
Andy

P.S.  Linux 3.2 (at least) appears to have a bug: the SYN/ACK has
ctdir ORIGINAL as seen from the the mangle PREROUTING chain.  I'll
send a real bug report for that if I can reproduce it cleanly on a
newer kernel.

Re: policy routing vs dnat replies

[Stephen Clark]

On 10/03/2012 08:44 PM, Andy Lutomirski wrote:
> I hit an annoying policy routing corner case today.  I have a router
> with two WAN interfaces (and no BGP).  I have policy routing set up so
> that, if a source address matches either of my public networks, then
> outgoing packets use the correct interface.  If neither rule matches
> (e.g. the source is 0.0.0.0 for source address selection), then the
> default route is whichever one I prefer at the moment.  It looks like
> this:
>
> $ ip rule
> 0:	from all lookup local
> 32766:	from all lookup main
> 32767:	from all lookup default
> 40000:	from<net2>  lookup isp2
> 40001:	from<net1>  lookup isp1
> 40010:	from all lookup real_default
>
> The relevant routes are:
>
> default via<gw1>  dev eth0.2  table isp1  src<src1>
> default via<gw2>  dev eth0.3  table isp2  src<src2>
> default via<gw2>  dev eth0.3  table real_default  src<src2>   metric 101
> default via<gw1>  dev eth0.2  table real_default  src<src1>   metric 102
>
> (Yes, this is a bit verbose, but I don't know a more concise way to do this.)
>
> This works nicely: if I specifically bind to one of my public
> addresses, the corresponding WAN link is used, and if not or if I'm
> coming from a private address, then the metrics determine which link
> to use.
>
> DNAT breaks it.  I have a rule:
> -A PREROUTING -i eth0.2 -d<ip1>  -p tcp --dport<port>  -j DNAT --to
> <internal host>
>
> <ip1>  lives on isp1.  Someone sends a SYN.  It gets routed to the
> internal host, and that host sends a SYN/ACK back.  The SYN/ACK has a
> source ip that isn't on net1 or net2, so it matches the 'lookup
> real_default' rule and gets routed to *gw2*.  iptables rewrites the
> source address after the routing decision, and my router sends a
> packet with a source address belonging to isp1 to isp2's gateway.  The
> packet is then dropped.
>
> Is there any way I can either convince iptables to rewrite the source
> address in the prerouting hook or to query the conntrack source
> address from the policy rule?  Is there a better solution?  I'm
> currently using a somewhat gross combination of MARK and fwmark
> matches to work around this problem.  One possibility would be:
>
> Thanks,
> Andy
>
> P.S.  Linux 3.2 (at least) appears to have a bug: the SYN/ACK has
> ctdir ORIGINAL as seen from the the mangle PREROUTING chain.  I'll
> send a real bug report for that if I can reproduce it cleanly on a
> newer kernel.
> --
> To unsubscribe from this list: send the line "unsubscribe netdev" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
>    
Well what I did faced with a similar problem was add a higher prio rule
that said if from ip1 lookup isp1.

-- 

"They that give up essential liberty to obtain temporary safety,
deserve neither liberty nor safety."  (Ben Franklin)

"The course of history shows that as a government grows, liberty
decreases."  (Thomas Jefferson)