Routing for multiple uplinks/providers

Routing for multiple uplinks/providers

[Lloyd Standish]

Hello Friends,

I have have coded a load-balancing netfilters router, which seems to work fine, but I have a question about sending a router's response to queries out over the right interface.

Suppose a router has 2 outward-facing interfaces (uplinks) and a LAN (3 interfaces).  The LAN addresses are SNAT'd over the 2 outward interface addresses (WANs).

Suppose the router also runs a few services (for example a web server), and might get incoming requests from either of its 2 uplinks.  (I realize that running services on more than one IP might be uncommon.)

How to ensure that answers to incoming requests are routed out over the correct interface?

This lartc page (http://lartc.org/howto/lartc.rpdb.multiple-links.html) appears to indicate that all that is necessary are rules like these (there is a diagram in that page):

ip rule add from $IP1 table T1
ip rule add from $IP2 table T2

where $IP1 and $IP2 are the WAN addresses of each of 2 outward-facing interfaces.  The page says, "It will work for all processes running on the router itself, and for the local network, if it is masqueraded."

I don't understand this.  In the first place, how does SNAT know about what interface the packet we are replying to came from? That was a *previous* packet.

Furthermore, I don't see how the policy routing rules above will work to for processes running on the router itself.

It appears to me that the only way to choose the right outgoing interface for a reply packet in this situation is to use state information, as follows:

1) mark the incoming packets with a number associated with each interface
2) do a "... -m state --state ESTABLISHED,RELATED -j CONNMARK --restore-mark" on packets sent as replies (ESTABLISHED,RELATED), and
3) choose the correct interface with a rule like "ip rule add fwmark <n> table <table>"

I must be missing something important.  Please explain.

-- 
Lloyd

Re: Routing for multiple uplinks/providers

[Lloyd Standish]

On Thu, 29 Dec 2011 11:21:52 -0600, Lloyd Standish <lloyd@crnatural.net> wrote:

> Suppose a router has 2 outward-facing interfaces (uplinks) and a LAN (3 interfaces).  The LAN addresses are SNAT'd over the 2 outward interface addresses (WANs).
>

<snip>
> How to ensure that answers to incoming requests are routed out over the correct interface?
> This lartc page (http://lartc.org/howto/lartc.rpdb.multiple-links.html) appears to indicate that all that is necessary are rules like these (there is a diagram in that page):
> ip rule add from $IP1 table T1
> ip rule add from $IP2 table T2
> where $IP1 and $IP2 are the WAN addresses of each of 2 outward-facing interfaces.  The page says, "It will work for all processes running on the router itself, and for the local network, if it is masqueraded."
> I don't understand this.  In the first place, how does SNAT know about what interface the packet we are replying to came from? That was a *previous* packet.

I think I can answer my own question, "for the record."

It turns out that SNAT *is* stateful, according to the following tutorial (http://bec.at/support/iptables-tutorial/x4679.html):

"Only the first packet in a connection is mangled by SNAT, and after that all future packets using the same connection will also be SNATted. Furthermore, the initial rules in the POSTROUTING chain will be applied to all the packets in the same stream."

So SNAT keeps connection-level state information, allowing it to write in the correct source address for all ESTABLISHED,RELATED packets belonging to a connection.  That explains why a rule like

ip rule add from $IP1 table T1  (where $IP1 is our interface address)

would be routed via the right table, in a situation where there are multiple uplinks (multiple routing tables).

-- 
Lloyd

Re: Routing for multiple uplinks/providers

[Ed W]

On 01/01/2012 16:30, Lloyd Standish wrote:
> On Thu, 29 Dec 2011 11:21:52 -0600, Lloyd Standish 
> <lloyd@crnatural.net> wrote:
>
>> Suppose a router has 2 outward-facing interfaces (uplinks) and a LAN 
>> (3 interfaces).  The LAN addresses are SNAT'd over the 2 outward 
>> interface addresses (WANs).
>>
>
> <snip>
>> How to ensure that answers to incoming requests are routed out over 
>> the correct interface?
>> This lartc page 
>> (http://lartc.org/howto/lartc.rpdb.multiple-links.html) appears to 
>> indicate that all that is necessary are rules like these (there is a 
>> diagram in that page):
>> ip rule add from $IP1 table T1
>> ip rule add from $IP2 table T2
>> where $IP1 and $IP2 are the WAN addresses of each of 2 outward-facing 
>> interfaces.  The page says, "It will work for all processes running 
>> on the router itself, and for the local network, if it is masqueraded."
>> I don't understand this.  In the first place, how does SNAT know 
>> about what interface the packet we are replying to came from? That 
>> was a *previous* packet.
>
> I think I can answer my own question, "for the record."
>
> It turns out that SNAT *is* stateful, according to the following 
> tutorial (http://bec.at/support/iptables-tutorial/x4679.html):
>
> "Only the first packet in a connection is mangled by SNAT, and after 
> that all future packets using the same connection will also be 
> SNATted. Furthermore, the initial rules in the POSTROUTING chain will 
> be applied to all the packets in the same stream."
>
> So SNAT keeps connection-level state information, allowing it to write 
> in the correct source address for all ESTABLISHED,RELATED packets 
> belonging to a connection.  That explains why a rule like
>
> ip rule add from $IP1 table T1  (where $IP1 is our interface address)
>
> would be routed via the right table, in a situation where there are 
> multiple uplinks (multiple routing tables).
>

I believe also routes are cached per IP, so I guess it might 
accidentally persist beyond even individual streams (assuming to/from 
same IPs)

I think this is why for "perfect" load balancing and routing robustness 
it's considered "best" to mark connections and use some randomness to 
route connections to outbound interfaces? I'm out of my depth here 
though so ask a grown-up to confirm that...

Ed W

Re: Routing for multiple uplinks/providers

[Lloyd Standish]

On Mon, 02 Jan 2012 06:43:37 -0600, Ed W <lists@wildgooses.com> wrote:

> I believe also routes are cached per IP, so I guess it might accidentally persist beyond even individual streams (assuming to/from same IPs)

Thanks, I didn't think of that.  If no policy routing rules are matched, the cached route should be used.  Of course, this doesn't explain why connections were dropped when the outgoing interface had a non-routable (LAN) IP number, but were never dropped from the one interface that has a public IP.

I'm hoping someone will comment on the use of "-p tcp" in the SNAT invocation:

(example rule)
iptables -t nat -A POSTROUTING -p tcp -o eth0 -j SNAT --to-source 194.236.50.155

I have always seen this rule with "-p tcp", and yet I think it is best to mark ALL my outgoing connections, in case the one interface associated with the default route happens to be down.  For example, DNS queries are usually UDP, I believe.

-- 
Lloyd Standish