[LARTC] ADSL channel boding or Load balancing

[LARTC] ADSL channel boding or Load balancing

[the sew]

Hi There,

We currently using iproute2 for load balancing. However we need more
upload speed as we load balance over 3 dsl lines.

I've been looking for a way to combine the upload speed to more
faster. Found a site called www.upstreaminter.net where these guys
bond the adsl channels to improve uploads, Since downloading is
problem as its need to know the ip address of the downloader they use
VPN to their existing infrastruture to improve download speeds to a
single IP.  Im not interested in doing that though, Our company uses
alot of mail to our smarthost and the upload is only 256k on the 4Mbit
Line

These guys talking about using USB modems , some researching says your
ISP must support MLPPP ,

Anyone with any expierencing bonding adsl modems with a standard
kernel and distro?

Most nix's come with RP-PPPOE, will this work?

Thanks

Sew
_______________________________________________
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

Re: [LARTC] ADSL channel boding or Load balancing

[Grant Taylor]

On 10/11/07 10:17, the sew wrote:
> We currently using iproute2 for load balancing. However we need more 
> upload speed as we load balance over 3 dsl lines.

*nod*

> I've been looking for a way to combine the upload speed to more 
> faster. Found a site called www.upstreaminter.net where these guys 
> bond the adsl channels to improve uploads, Since downloading is 
> problem as its need to know the ip address of the downloader they use 
> VPN to their existing infrastruture to improve download speeds to a 
> single IP.  Im not interested in doing that though, Our company uses 
> alot of mail to our smarthost and the upload is only 256k on the 
> 4Mbit Line

Now you are entering some more complex and interesting territory.  The 
problem you have with traditional load balancing that the LARTC suggests 
for SOHO users is that you have multiple external IP addresses.  Thus 
any single stream is limited to the speed of the link that it is using.

If you are using multiple links from a single provider who is willing to 
work with you, you could do something like I did with the two 1.1 Mbps 
SDSL links that I have aggregated in to one 2.2 Mbps link.  I can 
easily, and routinely do, push and / or pull 1.9+ Mbps on a single ftp 
stream.  I have a small globally routable DMZ behind the two globally 
routable IPs for my SDSL links.  My ISP has a router here on my premise 
and is using OSPF to load balance across the SDSL links.  (A little 
ASCII drawing comes to mind.)

                :
                :        +--------+     +--------+
+-----+     +-----+     | SDSL 1 +-----+ SDSL 1 |     +------+
| LAN +-----| NAT +-----+ CPERTR |     | ISPRTR +-----+ INet |
+-----+     +-----+     | SDSL 2 +-----+ SDSL 2 |     +------+
                :        +--------+     +--------+
  Your domain   :   ISP domain
                :

To recap, the following interfaces all have globally routable IP addresses:
  - NAT outside
  - CPE Router inside
  - CPE Router SDSL 1 outside
  - CPE Router SDSL 2 outside
  - ISP Router SDSL 1 inside
  - ISP Router SDSL 2 inside
  - ISP Router INET outside

The only interfaces that have private IPs are the LAN and the NAT inside.

As such the CPE Router and ISP Router load balance across the two SDSL 
links.  The world sees my office as being from the NAT outside and does 
not give a darn about any thing in between.  We even have OSPF set up to 
realize if one of the SDSL links goes down to re-route across the 
remaining link.

> These guys talking about using USB modems , some researching says 
> your ISP must support MLPPP ,

I don't think the technology of the modem really matters all that much. 
  However you will require support from your provider, be it PPP 
Multi-Link, or load balancing like I'm doing, or something similar.

> Anyone with any expierencing bonding adsl modems with a standard 
> kernel and distro?

Not speciffically.  Though, given a scenario like mine, I don't see why 
it would matter as long as you can get Equal Cost Multi-Path to work 
correctly.  My provider required that their own equipment be used on 
customer premise because they were using OSPF an interior routing 
protocol to my premise.  So my ISP extended their premise to be located 
at my office so that the IRP stays with in their control.  Heh, it works 
and I get aggregated speeds and redundancy, so I'm not going to 
complain.  Seeing as how the CPE Router above is a Cisco 3640 I can't 
say any thing for sure about Linux, but it should be able to do the job 
just fine.

> Most nix's come with RP-PPPOE, will this work?

I don't know if RP-PPPoE will do PPP Multi-Linking or not.  Though I 
don't think that it is an aspect of RP-PPPoE, but rather the underlying 
PPP implementation on your system.  It is my understanding that RP-PPPoE 
just passes traffic from "over Ethernet" to your PPP daemon.  So it is 
probably really up to your PPP daemon.  Granted you will have to have 
your providers support to be able to do this too.

Another option you could look in to if you have access to a system with 
high speed (read OC-1 or better) internet access somewhere is tunnels. 
That is create an IP in IP or GRE tunnel from your external IP to a 
co-located system and then load balance across the tunnels and NAT at 
the co-located system.  Your router and the co-located system would need 
to know how to spread the bandwidth across the tunnels in to you.

                 :                            :
             +--------+     +--------+        :
+-----+     | ADSL 1 +-----+ ADSL 1 |     +------+     +------+
| LAN +-----+ CPENAT |     | ISPRTR +-----+ COLO |-----+ INet |
+-----+     | ADSL 2 +-----+ ADSL 2 |     +------+     +------+
             +--------+     +--------+        :
                 :                            :
   Your Domain   :         Shared Domain      :   Public Domain
                 :                            :

In this scenario, you would set up a tunnel from the CPE NAT Router ADSL 
1 outside to the COLO inside and from the CPE NAT Router ADSL 2 outside 
to the COLO inside.  You would add a route to the COLO box via the ISPs 
upstream gateway to each ADSL connection and use the COLO box as your 
default gateways.  I.e. the CPD NAT Router would have a routing table 
similar to this:

0.0.0.0/0	<Tunnel 1 internal IP>
0.0.0.0/0	<Tunnel 2 internal IP>
<COLO/32>	<ADSL 1 GW>
<COLO/32>	<ADSL 2 GW>

This way computers on your LAN will use the CPE NAT Router as their 
default gateway.  The CPE NAT Router will really have two default 
gateways, the far end of the tunnels.  The Co-Located box is what will 
really be doing the NATing for the LAN.  As far as subnets are 
concerned, this is what the network will look like.

                +--------+     +--------+
+--------+     | ...1.2 +-----| ...1.1 |     +------+
| ...0.x +-----+ ...0.1 |     | Global +-----+ INet |
+--------+     + ...2.2 +-----| ...2.1 |     +------+
                |--------+     +--------+

The CPE Router (that is not NATing) will use the Co-Located box tunnels 
as its two default gateways and know about the LAN and tunnel subnets. 
The CPE Router will have routes to the globally routable IP of the 
Co-Located box via the ISPs upstream gateways but not use the ISPs 
upstream gateways as default routes.

The Co-Located box will use the tunnels as multiple routes to get to the 
LAN subnet and know about the tunnels and LAN subnet.

I know that this type of set up is extremely convoluted but will work 
and it will work with out the support of the ISP(s).  Heck, you could 
even use multiple different ISPs each with different layer 1 technology. 
  Thus you would really have redundancy in case one of the links goes 
down.  You could even expand this modle to include a second different 
co-located box to allow for even more redundancy.  (I'm going to shut up 
now.)



Grant. . . .
_______________________________________________
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

Re: [LARTC] ADSL channel boding or Load balancing

[Grant Taylor]

On 10/11/07 11:18, Grant Taylor wrote:
> Another option you could look in to if you have access to a system 
> with high speed (read OC-1 or better) internet access somewhere is 
> tunnels. That is create an IP in IP or GRE tunnel from your external 
> IP to a co-located system and then load balance across the tunnels 
> and NAT at the co-located system.

I would recommend that you not use an encrypting tunnel to reach the 
Co-Located box as you are dealing with traffic that is going to go 
across the internet at large any way, just let it go out.  However if 
you want to take this idea to connect multiple sites together, you 
obviously would want to consider encrypting VPNs.



Grant. . . .
_______________________________________________
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

Re: [LARTC] ADSL channel boding or Load balancing

[the sew]

Hi Grant,

Thanks for all the info, I like your SDSL option with the ospf,
exactly what I would like


Our situation is quite simple, Our ISP is telkom, one main Provider
for our Country, they will not touch anything other than the standard
services they provide. No SDSL, only ADSL. with standard pppoe. I
guess i'm bit stuck there. I could do it to our co-location like u
suggest, but that will add more costs to line rental where lines here
is quite pricey. That would be a last resort for me

I've seen the bonding in action,but havent seen the configs as they
kept it quite secret as its a "new" thing. Looks like I'm gonna have
to try this route on my own. I could not find much info on this. I
will give it a try with 2 x USB modems and tell pppd to use
multilinking and see if I can attach the device as one, I'm sure
downloads will be speed to a single dsl only, but upload should be
double. I think I might get a bit stuck as they might do the bonding
to a co-location like you said and just resell it, will give it a
#!/bin/bash anyway

Thanks again for your info

Sew
.




On 10/11/07, Grant Taylor <gtaylor@riverviewtech.net> wrote:
> On 10/11/07 11:18, Grant Taylor wrote:
> > Another option you could look in to if you have access to a system
> > with high speed (read OC-1 or better) internet access somewhere is
> > tunnels. That is create an IP in IP or GRE tunnel from your external
> > IP to a co-located system and then load balance across the tunnels
> > and NAT at the co-located system.
>
> I would recommend that you not use an encrypting tunnel to reach the
> Co-Located box as you are dealing with traffic that is going to go
> across the internet at large any way, just let it go out.  However if
> you want to take this idea to connect multiple sites together, you
> obviously would want to consider encrypting VPNs.
>
>
>
> Grant. . . .
> _______________________________________________
> LARTC mailing list
> LARTC@mailman.ds9a.nl
> http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc
>
_______________________________________________
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

Re: [LARTC] ADSL channel boding or Load balancing

[Grant Taylor]

On 10/11/07 13:12, the sew wrote:
> Thanks for all the info, I like your SDSL option with the ospf, 
> exactly what I would like

*nod*  I like it too.  :)

> Our situation is quite simple, Our ISP is telkom, one main Provider 
> for our Country, they will not touch anything other than the standard 
> services they provide. No SDSL, only ADSL. with standard pppoe. I 
> guess i'm bit stuck there. I could do it to our co-location like u 
> suggest, but that will add more costs to line rental where lines here 
> is quite pricey. That would be a last resort for me

You can do the tunneling like I was referring to across any thing that 
will carry IP, it does not have to be SDSL or any other form of 
connection in particular.

What do you mean by "... that will add more costs to line rental ..."? 
Are you trying to say that you pay for bandwidth, so you would be 
doubling what passes through your co-located box?

> I've seen the bonding in action,but havent seen the configs as they 
> kept it quite secret as its a "new" thing. Looks like I'm gonna have 
> to try this route on my own. I could not find much info on this. I 
> will give it a try with 2 x USB modems and tell pppd to use 
> multilinking and see if I can attach the device as one, I'm sure 
> downloads will be speed to a single dsl only, but upload should be 
> double. I think I might get a bit stuck as they might do the bonding 
> to a co-location like you said and just resell it, will give it a 
> #!/bin/bash anyway

I don't think you need to worry about using a USB modem persay, if the 
provider is using standard RFC 1483 (2684) (Multiprotocol Encapsulation 
over ATM Adaptation Layer 5) encapsulation to carry the ethernet frames 
back to the router.  You could easily use an external ADSL ATM to 
Ethernet bridging modem with out a problem.  Now if the provider is 
wanting to do something more special such that there are two different 
ATM connections with the ATM stack on the Linux system using the USB 
ADSL ATM modems, then yes you would need to use the USB modems.  However 
I think it would be much more complex to try to multiplex across two 
different ATM connections with out an intermediary IP layer.

With regards to PPP Multi-Linking, things have been standardized for a 
long time.  Well as standardized as things can be when there is little 
call for it.  Basically what is done is two separate PPP connections are 
made between the PPP daemons on each end where the PPP daemon knows that 
some traffic for a given logical interface will pass down both 
connections.  The PPP daemon splits the traffic that it sends and 
combines the traffic that it receives.  In short this is much like 
striping data (RAID 0) across two drives except that it is used for 
networking.  I think you will need to find out some more information to 
know what direction to go to get this to work.  Ideally if your provider 
does support PPP Multi-Linking there is no reason (that I know of) why 
you could not use this across PPPoE.  If more providers supported this, 
this would be a very good thing.  Heck I think you can even get the PPP 
daemon to realize that one of the links is down and to fall back to the 
single link, thus you have some redundancy.

> Thanks again for your info

You are welcome.



Grant. . . .
_______________________________________________
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

Re: [LARTC] ADSL channel boding or Load balancing

[the sew]

Hi Grant,

You were right, the usb modem mehtod was just a way for me to see how
usb modems works in linux, u basicly add new firmware and load the
driver and it sees your usb modem also as a ethernet device and u use
pppd with that :-)

Im trying your option below to get increased upload, but no luck yet


from your ascii , mine looks similar

                 +--------+     +--------+
 +--------+     | ...245.18 +-----| ...77.1 |     +------+
 | ...0.x +-----+ ...0.1 |     | COLO +-----+  101.x |
 +--------+     + ...245.19 +-----| ...78.1 |     +------+
                 |--------+     +--------+


the background how I got this setup is that 245.18 and 19 has pppoe
into my ISP, then I made a pptp tunnel to COLO on 2 diffrent ips and
got my new ips which is 245.18 Ptp 77.1 and 245.19 Ptp 78.1.

All this works fine, I've even add multiple routes to 101.x via dev
ppp0 nexthop dev ppp1, but this just load balances the links, not
increase uploads

Is it possible at all to route all upload via both links at the same
time, maybe disable the cached routes, let it just change routes the
whole time?

Thanks agian for your help

Sew
_______________________________________________
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

Re: [LARTC] ADSL channel boding or Load balancing

[Grant Taylor]

On 10/25/07 00:49, the sew wrote:
> You were right, the usb modem mehtod was just a way for me to see how 
> usb modems works in linux, u basicly add new firmware and load the 
> driver and it sees your usb modem also as a ethernet device and u use 
> pppd with that :-)

*nod*  That's the experience that I had too.  I will say that I 
perceived performance to be a bit better with the USB ADSL modem than I 
did with external bridging modems.  I attributed this perceived 
performance improvement to be related to more intelligent caching of 
data on the system with the IP stack with comparison to external modems 
that did not know if there would be any more packets coming or not thus 
the external would have to wait longer before transmitting packets.  I 
have no numbers to back this up, but I did perceive a performance 
improvement both under Linux and Windows with my USB ADSL modem and 
under Windows with my PCI ADSL modem.

> Im trying your option below to get increased upload, but no luck yet
> 
> from your ascii , mine looks similar

                 +-----------+     +---------+
  +--------+     | ...245.18 +-----+ ...77.1 |     +-------+
  | ...0.x +-----+ ...0.1    |     |    COLO +-----+ 101.x |
  +--------+     | ...245.19 +-----+ ...78.1 |     +-------+
                 +-----------+     +---------+

(Slight clean up.)

> the background how I got this setup is that 245.18 and 19 has pppoe 
> into my ISP, then I made a pptp tunnel to COLO on 2 diffrent ips and 
> got my new ips which is 245.18 Ptp 77.1 and 245.19 Ptp 78.1.
> 
> All this works fine, I've even add multiple routes to 101.x via dev 
> ppp0 nexthop dev ppp1, but this just load balances the links, not 
> increase uploads

Where are you doing your NATing to the world?  Are you NATing on your 
PPtP tunnels or on the COLO system?

> Is it possible at all to route all upload via both links at the same 
> time, maybe disable the cached routes, let it just change routes the 
> whole time?

If you are doing your NATing on the COLO system and you add two routes 
to your internal network via the two PPtP tunnels, you should be able to 
equal cost multipath route across both PPtP tunnels to achieve increased 
bandwidth.  The key part is that both tunnels have to appear to the 
world as a single external IP.

> Thanks agian for your help

You are welcome.



Grant. . . .
_______________________________________________
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

Re: [LARTC] ADSL channel boding or Load balancing

[the sew]

                            +-----------+     +---------+
          +--------+     | ...245.18 +-----+ ...77.1 |     +-------+
          | ...0.x +-----+ ...0.1    |     |    COLO +-----+ 101.x |
          +--------+     | ...245.19 +-----+ ...78.1 |     +-------+
                            +-----------+     +---------+
>
> (Slight clean up.)
>
>
> Where are you doing your NATing to the world?  Are you NATing on your
> PPtP tunnels or on the COLO system?

Currently I'm natting on 0.1 ( -o ppp+ -j MASQUERADE ) , This was
setup by default as I did not want 0.x to be routed. I've however
taken off the natting, and added a route for 0.20/32 dev ppp62 nexthop
dev ppp32 ( the 2 vpn interfaces) at COLO and obviously same nexthop
routes at 0.1 for 101.x

testing from 0.20, I scp a tar file over to 101.20 , still goes via
one line at time, the route cache which I disabled, just reroute it
the whole time( about every 5 sec) via the diffrent uplink, but not to
our result we want

I use sysstat to check the speeds and tcpdump verified I its from 0.20
-> 101.20 ssh


>
> If you are doing your NATing on the COLO system and you add two routes
> to your internal network via the two PPtP tunnels, you should be able to
> equal cost multipath route across both PPtP tunnels to achieve increased
> bandwidth.  The key part is that both tunnels have to appear to the
> world as a single external IP.

I understand now very clearly the key part.

My problem must be the tunnel, im sure im messing up,  the equal cost
multipath routing , am I using the right utitily? , still iproute2
right, or is iptables gonna play big part here as well?



S
_______________________________________________
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

Re: [LARTC] ADSL channel boding or Load balancing

[Grant Taylor]

On 10/25/2007 1:06 PM, the sew wrote
>                +-----------+     +---------+
> +--------+     | ...245.18 +-----+ ...77.1 |     +-------+
> | ...0.x +-----+ ...0.1    |     |    COLO +-----+ 101.x |
> +--------+     | ...245.19 +-----+ ...78.1 |     +-------+
>                +-----------+     +---------+
> 
> Currently I'm natting on 0.1 ( -o ppp+ -j MASQUERADE ) , This was 
> setup by default as I did not want 0.x to be routed. I've however 
> taken off the natting, and added a route for 0.20/32 dev ppp62 
> nexthop dev ppp32 ( the 2 vpn interfaces) at COLO and obviously same 
> nexthop routes at 0.1 for 101.x

Ok, good.  The NATing at 0.1 was part of the problem.

> testing from 0.20, I scp a tar file over to 101.20 , still goes via 
> one line at time, the route cache which I disabled, just reroute it 
> the whole time( about every 5 sec) via the diffrent uplink, but not 
> to our result we want

Hum.  I would question the routes then.

> I use sysstat to check the speeds and tcpdump verified I its from 
> 0.20 -> 101.20 ssh

Ok.

> I understand now very clearly the key part.
> 
> My problem must be the tunnel, im sure im messing up,  the equal cost 
> multipath routing , am I using the right utitily? , still iproute2 
> right, or is iptables gonna play big part here as well?

To the best of my knowledge you will still be using the "ip" command 
from iproute2.  Will you please show us your routing table and / or the 
ip route command you are using to set up your ecmp route?



Grant. . . .
_______________________________________________
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc