Load balancing multiple ADSL connections

Load balancing multiple ADSL connections

[Bas]

I currently have a T1 connection which is 1.5M both ways and costs $350 
a month - and the only internet access available besides dialup.
I'm moving into a new area where ADSL is offered, 1.5M/768k for $35 a month.

I was thinking, if I rent some rack space in a server room with a 10 
MBit connection, could I have my Linux gateway set up with 6 NIC cards, 
1 LAN and 5xADSL with VPN to my server? The server could feed data 
across all 5 lines evenly and possibly meter the data based on the ping 
times. That way I would have a lot more bandwidth for the dollar. 7 MBit 
down and 3.5MBit up for $275 a month, assuming the rack space costs $100/mo

However I have been searching around and have not found any answer that 
the standard Linux kernel / OpenVPN can do something like this. 
Particularly with the ADSL you don't get a fixed bandwidth so you need 
to dynamically adjust your throttling based on what you get.

I see load balancing routers but they are connection based - i.e. any 
one file transfer would still operate at 1.5 MBit, you can just have 
multiple at the same time. This causes problems too, in my office we 
have load balancing proxy servers and since your IP address changes all 
the time many secure websites do not work.

I have a home office so I need to make sure that the ping times stay 
low. This is pretty easy with the T1 because of the fixed bandwidth.

The ADSL provider does not support bonding - that would be an easy 
option but unfortunately not available. T1's can be bonded but that's 
too expensive.

Bas

Re: Load balancing multiple ADSL connections

[cronolog+lartc]

Hi there,
> I was thinking, if I rent some rack space in a server room with a 10 
> MBit connection, could I have my Linux gateway set up with 6 NIC 
> cards, 1 LAN and 5xADSL with VPN to my server? The server could feed 
> data across all 5 lines evenly and possibly meter the data based on 
> the ping times. That way I would have a lot more bandwidth for the 
> dollar. 7 MBit down and 3.5MBit up for $275 a month, assuming the rack 
> space costs $100/mo
>
> However I have been searching around and have not found any answer 
> that the standard Linux kernel / OpenVPN can do something like this. 
> Particularly with the ADSL you don't get a fixed bandwidth so you need 
> to dynamically adjust your throttling based on what you get.
If I were to set this up, I'd first create 5 (OpenVPN) tunnels between 
the local linux gateway/server, and the remote linux server. Ensuring 
each tunnel is forced over a different ADSL link could be fun. You could 
set up 5 local routing tables, each with only one default route, one 
table for each ADSL link, then use the port number to select the 
appropriate routing table, and therefore ADSL link. My personal 
preference of doing this is marking the packets in the iptables mangle 
table, and use the "ip rule" command to select the routing table based 
on firewall mark. For example (with 2 uplinks):

ip route add default via 192.168.1.1 table 101
ip route add default via 192.168.2.1 table 102
iptables -t mangle -A OUTPUT -d ${remoteServerIP} -p udp --dport 
${OpenVPNTunnel1Port} -j MARK --set-mark 201
iptables -t mangle -A OUTPUT -d ${remoteServerIP} -p udp --dport 
${OpenVPNTunnel2Port} -j MARK --set-mark 202
ip rule add fwmark 201 pref 11101 table 101
ip rule add fwmark 202 pref 11102 table 102

(I assume there's an ADSL router per link and the public IPs are not on 
the local box, otherwise I think it'd get messy handling multiple ppp 
client sessions on the same box, especially if you don't have static IPs 
on all the links. Well, there's always running each ppp session in a 
dedicated virtual machine and set up your virtual networks appropriately 
to tie everything together, but let's not complicate things more than 
they already are.)

Now that the tunnels are in place, you need to somehow distribute 
traffic over them. I'd install quagga on both the local and remote 
servers, and run OSPF between them over all 5 tunnels, with the remote 
server pushing the default route back to the local one. Hopefully this 
should install 5 default routes on the local server each pointing over a 
different tunnel, and the kernel should load balance between them now 
(you may need to tweak path costs and such in quagga). Also, the same 
works the other way - you advertise your local subnet to the remote 
server over all 5 tunnels with quagga, and the remote server should load 
balance the traffic back to you.

I'm no quagga expert though (only using it to run RIP over OpenVPN at 
the moment since I don't need the added benefits OSPF provides), so 
would need to read the manual to work out the config to set this up.

But by running a routing protocol, especially OSPF, in this scenario 
would mean that if some ADSL links go down and up temporarily, or you 
add/remove links permanently, OSPF will take care of modifying the 
available routes on either server transparently for you.

> I see load balancing routers but they are connection based - i.e. any 
> one file transfer would still operate at 1.5 MBit, you can just have 
> multiple at the same time. This causes problems too, in my office we 
> have load balancing proxy servers and since your IP address changes 
> all the time many secure websites do not work.
I think even with this set-up, each connection flow would only go over 
one tunnel, I don't think linux routes on a per packet basis? Nor would 
I want it to in most cases, you'd get more issues with packets arriving 
in the wrong order when sending data if you do this, which will hurt tcp 
throughput.

However, since you are routing everything via your remote server in this 
case, which will almost certainly need to be NATting traffic as it 
finally goes out via it's single uplink, you won't have the issue which 
you see with your office proxy servers which are basically load 
balancing straight onto the Internet via multiple uplinks hence the 
multiple IP issue you see.

As such, this may partially negate the fact that you're still limited to 
1 ADSL-link worth of bandwidth per flow, since you'll probably have 
several flows going on simultaneously, especially for web browsing, so 
I'd still expect an overall increase in performance for such types of tasks.

> I have a home office so I need to make sure that the ping times stay 
> low. This is pretty easy with the T1 because of the fixed bandwidth.
Since you're planning on sending all data via a remote server first, 
you're already building in latency to your solution and so increasing 
ping times. Short of testing and knowing the exact setup, it's hard to 
say exactly how much added latency you'll see.

But this is a fairly complex design you're trying to come up with here, 
and there are a few details missing to fully configure it, such as how 
the ADSL links are presented to the local server, so it would take quite 
some fiddling to get it all working. Good luck if you try to go through 
with it though, I don't think it's impossible to configure. 5 links does 
feel a bit excessive to me, but you know your own requirements.


-- Muntasir

Re: Load balancing multiple ADSL connections

[Bas]

Hi Muntasir,

Thanks for your comprehensive response. I figured out earlier today that 
the key word I was missing is "bonding". If you search for VPN bonding 
then a number of solutions surface. (I was previously searching for 
"load balancing", etc)

I think I found the answer. The key package is "ifenslave". This allows 
you to bond multiple interfaces.
Each interface needs to have a unique IP on both sides, so even if your 
server (in the data center) has a single 100M line you still need a 
public IP for each ADSL line - as you say, so you can tell Linux to 
route each IP address through a different modem.

Then you set up a SSH or VPN bridge for each modem, so you get tun0, 
tun1, etc .. Then in your interfaces file you put something like this:

iface bond0 inet static
     address 172.26.0.1
     netmask 255.255.255.252
     bond-slaves tun0 tun1
     bond_mode balance-rr

That last line tells it to balance with a round robin, i.e. it sends one 
packet for each interface.


This is a good writeup:
http://www.linuxfoundation.org/collaborate/workgroups/networking/bonding

There are a number of inefficiencies in the system:
- There is significant overhead in re-ordering the packets. They say 
that in the case of 4 parallel connections, TCP connections will only 
get 2.3x the performance. UDP, if the application tolerates packet 
ordering issues, can scale almost linear
- The network is balanced by # of packets, not necessarily throughput. 
So with mixed traffic the modems may not be utilized evenly
- And of course DSL had no "guaranteed" bandwidth so if one leg falls 
behind, a large number of packets from other connections may be 
discarded because they are too far out of order.

I'm also concerned how well the system deals with interference. The DSL 
has no SLA. If one DSL leg times out the servers may get confused and 
not communicate anymore until I come in and reset the connections. The 
data center is 6 hours drive from here so I would have to build in back 
doors for remote control. It almost would take some special software to 
constantly monitor the network and re-negotiate / reconfigure the link 
as conditions change.

This seems like something a student could use for his thesis :) but for 
me it's probably better to just keep the T1 and be done with it.


On 09/13/2013 07:47 PM, cronolog+lartc wrote:
> Hi there,
>> I was thinking, if I rent some rack space in a server room with a 10
>> MBit connection, could I have my Linux gateway set up with 6 NIC
>> cards, 1 LAN and 5xADSL with VPN to my server? The server could feed
>> data across all 5 lines evenly and possibly meter the data based on
>> the ping times. That way I would have a lot more bandwidth for the
>> dollar. 7 MBit down and 3.5MBit up for $275 a month, assuming the rack
>> space costs $100/mo
>>
>> However I have been searching around and have not found any answer
>> that the standard Linux kernel / OpenVPN can do something like this.
>> Particularly with the ADSL you don't get a fixed bandwidth so you need
>> to dynamically adjust your throttling based on what you get.
> If I were to set this up, I'd first create 5 (OpenVPN) tunnels between
> the local linux gateway/server, and the remote linux server. Ensuring
> each tunnel is forced over a different ADSL link could be fun. You could
> set up 5 local routing tables, each with only one default route, one
> table for each ADSL link, then use the port number to select the
> appropriate routing table, and therefore ADSL link. My personal
> preference of doing this is marking the packets in the iptables mangle
> table, and use the "ip rule" command to select the routing table based
> on firewall mark. For example (with 2 uplinks):
>
> ip route add default via 192.168.1.1 table 101
> ip route add default via 192.168.2.1 table 102
> iptables -t mangle -A OUTPUT -d ${remoteServerIP} -p udp --dport
> ${OpenVPNTunnel1Port} -j MARK --set-mark 201
> iptables -t mangle -A OUTPUT -d ${remoteServerIP} -p udp --dport
> ${OpenVPNTunnel2Port} -j MARK --set-mark 202
> ip rule add fwmark 201 pref 11101 table 101
> ip rule add fwmark 202 pref 11102 table 102
>
> (I assume there's an ADSL router per link and the public IPs are not on
> the local box, otherwise I think it'd get messy handling multiple ppp
> client sessions on the same box, especially if you don't have static IPs
> on all the links. Well, there's always running each ppp session in a
> dedicated virtual machine and set up your virtual networks appropriately
> to tie everything together, but let's not complicate things more than
> they already are.)
>
> Now that the tunnels are in place, you need to somehow distribute
> traffic over them. I'd install quagga on both the local and remote
> servers, and run OSPF between them over all 5 tunnels, with the remote
> server pushing the default route back to the local one. Hopefully this
> should install 5 default routes on the local server each pointing over a
> different tunnel, and the kernel should load balance between them now
> (you may need to tweak path costs and such in quagga). Also, the same
> works the other way - you advertise your local subnet to the remote
> server over all 5 tunnels with quagga, and the remote server should load
> balance the traffic back to you.
>
> I'm no quagga expert though (only using it to run RIP over OpenVPN at
> the moment since I don't need the added benefits OSPF provides), so
> would need to read the manual to work out the config to set this up.
>
> But by running a routing protocol, especially OSPF, in this scenario
> would mean that if some ADSL links go down and up temporarily, or you
> add/remove links permanently, OSPF will take care of modifying the
> available routes on either server transparently for you.
>
>> I see load balancing routers but they are connection based - i.e. any
>> one file transfer would still operate at 1.5 MBit, you can just have
>> multiple at the same time. This causes problems too, in my office we
>> have load balancing proxy servers and since your IP address changes
>> all the time many secure websites do not work.
> I think even with this set-up, each connection flow would only go over
> one tunnel, I don't think linux routes on a per packet basis? Nor would
> I want it to in most cases, you'd get more issues with packets arriving
> in the wrong order when sending data if you do this, which will hurt tcp
> throughput.
>
> However, since you are routing everything via your remote server in this
> case, which will almost certainly need to be NATting traffic as it
> finally goes out via it's single uplink, you won't have the issue which
> you see with your office proxy servers which are basically load
> balancing straight onto the Internet via multiple uplinks hence the
> multiple IP issue you see.
>
> As such, this may partially negate the fact that you're still limited to
> 1 ADSL-link worth of bandwidth per flow, since you'll probably have
> several flows going on simultaneously, especially for web browsing, so
> I'd still expect an overall increase in performance for such types of tasks.
>
>> I have a home office so I need to make sure that the ping times stay
>> low. This is pretty easy with the T1 because of the fixed bandwidth.
> Since you're planning on sending all data via a remote server first,
> you're already building in latency to your solution and so increasing
> ping times. Short of testing and knowing the exact setup, it's hard to
> say exactly how much added latency you'll see.
>
> But this is a fairly complex design you're trying to come up with here,
> and there are a few details missing to fully configure it, such as how
> the ADSL links are presented to the local server, so it would take quite
> some fiddling to get it all working. Good luck if you try to go through
> with it though, I don't think it's impossible to configure. 5 links does
> feel a bit excessive to me, but you know your own requirements.
>
>
> -- Muntasir
>