problem with multi-level SNAT? (multiple uplinks/providers)

problem with multi-level SNAT? (multiple uplinks/providers)

[Lloyd Standish]

Greetings,

I programmed a load-balancing router for my ISP friend.  It balanced connections from a LAN with about 80 users over 3 to 5 uplinks.

There were problems with breaking download connections after about 5-200 MB.  The connections would simply drop, with no indication of why in the server logs.  I believe (but cannot confirm) the problem was caused by reply packets sent out the wrong interface.

However, there was an interface that never dropped connections.  This interface was different from the others in that it had a public IP assigned to it.  The other interfaces were connected to routers, and used LAN IP interface addresses that were SNATted in the router to a public IP.  In other words, the other interfaces had "upstream NAT."

This failing router used the following load-balancing method:

ip route add default scope global nexthop via $P1 dev $IF1 weight 15 nexthop via $P2 dev $IF2 weight 15...

SNAT was used to "bind" connections to each interface's source address, and a policy routing rule like this was used to choose the correct routing table (each table simply routed to a different interface:

ip rule add from ${!wan} table $table

The problem with dropping connections was finally fixed in router version #2.  It uses fwmark to mark all new connections, and a rule like this to send the packet to the correct routing table:

ip rule add fwmark $((cardnum+1)) table $table

Instead of relying on SNAT to assign the correct source address to subsequent packets in established/related packets, these packets are fwmarked like this:

iptables -t mangle -A PREROUTING -i ${lan_if} -m state --state ESTABLISHED,RELATED -j CONNMARK --restore-mark

Connected to the same uplinks/providers, THIS router does not drop connections.

I would like to know why SNAT without packet marking drops the connections for interfaces that have "upstream NAT," while the fwmark method succeeds.  I'd appreciate any comments.  I can of course supply more information if necessary.

As a related aside, I'd appreciate comments on whether or not the "-p tcp" in the following example rule is advisable (I omitted "-p tcp" from my SNAT rules.):
iptables -t nat -A POSTROUTING -p tcp -o eth0 -j SNAT --to-source 194.236.50.155

Regards,
Lloyd

Re: problem with multi-level SNAT? (multiple uplinks/providers)

[Andrew Beverley]

On Sun, 2012-01-01 at 11:12 -0600, Lloyd Standish wrote:
> I would like to know why SNAT without packet marking drops the
>  connections for interfaces that have "upstream NAT," while the fwmark
>  method succeeds.

I can't answer this, but on your other comment...

> As a related aside, I'd appreciate comments on whether or not the "-p
>  tcp" in the following example rule is advisable (I omitted "-p tcp"
>  from my SNAT rules.): iptables -t nat -A POSTROUTING -p tcp -o eth0 -j
>  SNAT --to-source 194.236.50.155

Well, I never put a -p tcp in SNAT for any of my configurations. I'm not
sure why you would want to limit SNAT to just TCP?

Andy

Re: problem with multi-level SNAT? (multiple uplinks/providers)

[Lloyd Standish]

On Mon, 02 Jan 2012 12:25:33 -0600, Andrew Beverley <andy@andybev.com> wrote:

>> As a related aside, I'd appreciate comments on whether or not the "-p
>>  tcp" in the following example rule is advisable (I omitted "-p tcp"
>>  from my SNAT rules.): iptables -t nat -A POSTROUTING -p tcp -o eth0 -j
>>  SNAT --to-source 194.236.50.155
> Well, I never put a -p tcp in SNAT for any of my configurations. I'm not
> sure why you would want to limit SNAT to just TCP?

Thanks for the comment.  I asked because I have seen "-p tcp" in other people's load-balancing scripts, and I couldn't understand why they were limiting to tcp.  I thought there must be something here I'm missing.  (Maybe they used it in order to be able to specify a port range for the SNAT).

I think that since SNAT applies to entire connections, other protocols will be SNATted with '-p tcp' if they are RELATED to an initial (NEW) tcp packet.  Maybe that is why the 'p tcp' specifier appears not to give trouble.

Anyway, I'm glad my logic is the same as yours.  I'll leave '-p tcp' out of my SNAT rules.  I want all traffic balanced over my interfaces - except of course packets that match certain policy routing rules.

(By the way, my new load-balancing router is working great, and my ISP and I are very relieved, after weeks of mysteriously-broken connections and disgruntled users.)

--
Lloyd