Re: Load balancing multiple ADSL connections

[cronolog+lartc <cronolog+lartc@googlemail.com>]

Hi there,
> I was thinking, if I rent some rack space in a server room with a 10 
> MBit connection, could I have my Linux gateway set up with 6 NIC 
> cards, 1 LAN and 5xADSL with VPN to my server? The server could feed 
> data across all 5 lines evenly and possibly meter the data based on 
> the ping times. That way I would have a lot more bandwidth for the 
> dollar. 7 MBit down and 3.5MBit up for $275 a month, assuming the rack 
> space costs $100/mo
>
> However I have been searching around and have not found any answer 
> that the standard Linux kernel / OpenVPN can do something like this. 
> Particularly with the ADSL you don't get a fixed bandwidth so you need 
> to dynamically adjust your throttling based on what you get.
If I were to set this up, I'd first create 5 (OpenVPN) tunnels between 
the local linux gateway/server, and the remote linux server. Ensuring 
each tunnel is forced over a different ADSL link could be fun. You could 
set up 5 local routing tables, each with only one default route, one 
table for each ADSL link, then use the port number to select the 
appropriate routing table, and therefore ADSL link. My personal 
preference of doing this is marking the packets in the iptables mangle 
table, and use the "ip rule" command to select the routing table based 
on firewall mark. For example (with 2 uplinks):

ip route add default via 192.168.1.1 table 101
ip route add default via 192.168.2.1 table 102
iptables -t mangle -A OUTPUT -d ${remoteServerIP} -p udp --dport 
${OpenVPNTunnel1Port} -j MARK --set-mark 201
iptables -t mangle -A OUTPUT -d ${remoteServerIP} -p udp --dport 
${OpenVPNTunnel2Port} -j MARK --set-mark 202
ip rule add fwmark 201 pref 11101 table 101
ip rule add fwmark 202 pref 11102 table 102

(I assume there's an ADSL router per link and the public IPs are not on 
the local box, otherwise I think it'd get messy handling multiple ppp 
client sessions on the same box, especially if you don't have static IPs 
on all the links. Well, there's always running each ppp session in a 
dedicated virtual machine and set up your virtual networks appropriately 
to tie everything together, but let's not complicate things more than 
they already are.)

Now that the tunnels are in place, you need to somehow distribute 
traffic over them. I'd install quagga on both the local and remote 
servers, and run OSPF between them over all 5 tunnels, with the remote 
server pushing the default route back to the local one. Hopefully this 
should install 5 default routes on the local server each pointing over a 
different tunnel, and the kernel should load balance between them now 
(you may need to tweak path costs and such in quagga). Also, the same 
works the other way - you advertise your local subnet to the remote 
server over all 5 tunnels with quagga, and the remote server should load 
balance the traffic back to you.

I'm no quagga expert though (only using it to run RIP over OpenVPN at 
the moment since I don't need the added benefits OSPF provides), so 
would need to read the manual to work out the config to set this up.

But by running a routing protocol, especially OSPF, in this scenario 
would mean that if some ADSL links go down and up temporarily, or you 
add/remove links permanently, OSPF will take care of modifying the 
available routes on either server transparently for you.

> I see load balancing routers but they are connection based - i.e. any 
> one file transfer would still operate at 1.5 MBit, you can just have 
> multiple at the same time. This causes problems too, in my office we 
> have load balancing proxy servers and since your IP address changes 
> all the time many secure websites do not work.
I think even with this set-up, each connection flow would only go over 
one tunnel, I don't think linux routes on a per packet basis? Nor would 
I want it to in most cases, you'd get more issues with packets arriving 
in the wrong order when sending data if you do this, which will hurt tcp 
throughput.

However, since you are routing everything via your remote server in this 
case, which will almost certainly need to be NATting traffic as it 
finally goes out via it's single uplink, you won't have the issue which 
you see with your office proxy servers which are basically load 
balancing straight onto the Internet via multiple uplinks hence the 
multiple IP issue you see.

As such, this may partially negate the fact that you're still limited to 
1 ADSL-link worth of bandwidth per flow, since you'll probably have 
several flows going on simultaneously, especially for web browsing, so 
I'd still expect an overall increase in performance for such types of tasks.

> I have a home office so I need to make sure that the ping times stay 
> low. This is pretty easy with the T1 because of the fixed bandwidth.
Since you're planning on sending all data via a remote server first, 
you're already building in latency to your solution and so increasing 
ping times. Short of testing and knowing the exact setup, it's hard to 
say exactly how much added latency you'll see.

But this is a fairly complex design you're trying to come up with here, 
and there are a few details missing to fully configure it, such as how 
the ADSL links are presented to the local server, so it would take quite 
some fiddling to get it all working. Good luck if you try to go through 
with it though, I don't think it's impossible to configure. 5 links does 
feel a bit excessive to me, but you know your own requirements.


-- Muntasir