[LARTC] Problem with Load Balancing

[LARTC] Problem with Load Balancing

[Vladimir Burciaga Aguilar]

Hi everybody!

I'm trying to implement the load balancing for a LAN with two ISPs. I've 
installed a Suse Linux Enterpise Server 9 with iproute2 for that porpouse.

The server have two NICs, one of them is for both the LAN and ISP 1. I've 
setup both NICs with YAST (if I use ip for this, then the whole thing 
doesn't work!) and execute the following commands to setup the routing 
tables:

ip route flush cache
ip route flush default
ip route flush table 1
ip route flush table 2

ip route add 10.1.254.0/24 dev eth0 src 10.1.254.251
ip route add 10.1.1.0/24 dev eth1 src 10.1.1.200

ip route show table main | while read ROUTE ; do ip route add table 1 $ROUTE 
; done
ip route show table main | while read ROUTE ; do ip route add table 2 $ROUTE 
; done

ip route add table 1 default via 10.1.254.254
ip route add table 2 default via 10.1.1.254

ip rule add from 10.1.254.251 table 1
ip rule add from 10.1.1.200 table 2

ip route add default equalize nexthop via 10.1.254.254 dev eth0 weight 1 \
nexthop via 10.1.1.254 dev eth1 weight 1


(All of this came from 
http://linux.lcampino.cl/wiki/index.php/Balanceo_de_Carga#Configurando_la_red 
(in Spanish)).

At this point, things seems to work fine. I browse in the server and watch 
with iptraf that both NICs have traffic, but not at 50%-50% rate.


Now I need to implement the following:

- Make load balancing the nearest to 50%-50% rate (Both Internet connections 
have equal bandwith).
- When one of the links goes down, all the traffic redirects to the other 
inmediately and automatically. Also, when the link is up again, the load 
balancing restart too.
- If posible, when downloading a single big file (i.e. *.exe, *.iso), the 
packets come from both connections.

I've read a lot about this, but I'm still confuse and very very lost...! 
:-)))

I'm not sure if I need to adjust some kernel paramters and rebuit it, 
execute another sequence of commands, apply a kernel patch, etc.

I really apreciate if someone could bring me some light in all this, or tell 
me what documentation, web page or patch do I need to use.


Thanks in advanced for all your time and recomendations!


P.D. Sorry for my English!

_________________________________________________________________
Platica con tus amigos en linea con MSN Messenger 
http://messenger.t1msn.com.mx/

_______________________________________________
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

RE: [LARTC] Problem with Load Balancing

[William T Mullaney]

[-- Attachment #1.1: Type: text/plain, Size: 3829 bytes --]

Vlad,

We have also set up a somewhat similar method of load balancing.  Our
traffic is never a 50-50 split (well 3:2 is how we have it set, but it
doesn't always get close to that), but as the load picks up, it tends to be
closer to the actual amount.

Dead gateway detection has never worked for us, and one day I'll probably
bother other members of the LARTC group to get some help, but the method
that we use is to check the output of the ip neighbor command.  Basically,
if our two ISPs are 10.1.1.254 and 10.2.2.254, we run a bash script via cron
every minute that does a call something like:

ETH1 = ip neigh 10.1.1.254 | egrep "REACHABLE|DELAY|PROBE|STALE" -c 
ETH2 = ip neigh 10.2.2.254 | egrep "REACHABLE|DELAY|PROBE|STALE" -c

The neighbor system basically monitors ARP and if it sees a message leave an
interface without a reply after something like 3-5 seconds, it moves the
interface to DELAY, after another few seconds it moves to PROBE and does an
active arp request, and if that fails to work in a few seconds, it becomes
INCOMPLETE or FAILED or just simply isn't listed.  If no data is sent either
way for a while, the entry can be marked STALE or removed.

With the above lines, we get a 1 in the ETH1 or ETH2 variables if the next
neighbor is up, and a 0 if not.  From there you can use some if scripts to
detect if both are up, or if only one is up, which one.  In our case, if
both are up we clear the default route and then make it something like

ip route add default nexthop via 10.1.1.254 dev eth1 weight 1 \
nexthop via 10.2.2.254 dev eth2 weight 1

and if only one is up we clear it and make it :

ip route add default nexthop via 10.1.1.254 dev eth1 
or 
ip route add default nexthop via 10.2.2.254 dev eth2 


With some additional scripting we can allow this to be overridden, we can
set the link to prefer using only one line, but switch to the other if the
preferred line fails, and to take input from programs like Nagios to
auto-prefer one line or another if ping times get high, etc.  In addition,
the script remembers the state it was in (so that it only changes the
routing table when needed), controls DNS, can flush the DNS cache, and
reports status back to Nagios.  Once I get all the bugs out and some
documentation, I'd be happy to post it to the news group, though you or
anyone else can send me an email if you would like to take a look at it
before then.

In practice, this method usually detects and adjusts outbound connections
quickly without user intervention; DNS changes with short TTLS take care of
inbound connections.  Just be careful... if you don't have something sending
traffic out to your upstream routers (and back) every few minutes, the entry
in your ARP table can potentially be removed and thus cause your system to
think an unused gateway has failed, or that a recovered gateway is still
down.  This could be checked with a quick "if ip neigh test fails, ping
neighbor 5 times, then test again before making decisions".  Running an
uptime monitor that pings or does something else to/through the gateway
(regardless of default route) also takes care of this.


-Will

-----Original Message-----
From: Vladimir Burciaga Aguilar [mailto:anakinv7@hotmail.com] 
Sent: Thursday, September 14, 2006 10:25 PM
To: lartc@mailman.ds9a.nl
Subject: [LARTC] Problem with Load Balancing

Hi everybody!

I'm trying to implement the load balancing for a LAN with two ISPs. I've 
installed a Suse Linux Enterpise Server 9 with iproute2 for that porpouse.

The server have two NICs, one of them is for both the LAN and ISP 1. I've 
setup both NICs with YAST (if I use ip for this, then the whole thing 
doesn't work!) and execute the following commands to setup the routing 
tables:

ip route flush cache
ip route flush default
ip route flush table 1
ip route flush table 2

[snip]

[-- Attachment #1.2: Type: text/html, Size: 5187 bytes --]

[-- Attachment #2: Type: text/plain, Size: 143 bytes --]

_______________________________________________
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

RE: [LARTC] Problem with Load Balancing

[Vladimir Burciaga Aguilar]

>We have also set up a somewhat similar method of load balancing.  Our
>traffic is never a 50-50 split (well 3:2 is how we have it set, but it
>doesn't always get close to that), but as the load picks up, it tends to be
>closer to the actual amount.

Well, then there is not much to do about this.

>Dead gateway detection has never worked for us, and one day I'll probably
>bother other members of the LARTC group to get some help, but the method
>that we use is to check the output of the ip neighbor command.  Basically,
>if our two ISPs are 10.1.1.254 and 10.2.2.254, we run a bash script via 
>cron
>every minute that does a call something like:
>
>ETH1 = ip neigh 10.1.1.254 | egrep "REACHABLE|DELAY|PROBE|STALE" -c
>ETH2 = ip neigh 10.2.2.254 | egrep "REACHABLE|DELAY|PROBE|STALE" -c
>
>The neighbor system basically monitors ARP and if it sees a message leave 
>an
>interface without a reply after something like 3-5 seconds, it moves the
>interface to DELAY, after another few seconds it moves to PROBE and does an
>active arp request, and if that fails to work in a few seconds, it becomes
>INCOMPLETE or FAILED or just simply isn't listed.  If no data is sent 
>either
>way for a while, the entry can be marked STALE or removed.
>
>With the above lines, we get a 1 in the ETH1 or ETH2 variables if the next
>neighbor is up, and a 0 if not.  From there you can use some if scripts to
>detect if both are up, or if only one is up, which one.  In our case, if
>both are up we clear the default route and then make it something like
>
>ip route add default nexthop via 10.1.1.254 dev eth1 weight 1 \
>nexthop via 10.2.2.254 dev eth2 weight 1
>
>and if only one is up we clear it and make it :
>
>ip route add default nexthop via 10.1.1.254 dev eth1
>or
>ip route add default nexthop via 10.2.2.254 dev eth2


Ok, William, this looks like what I'm looking for. I'm going to test it and 
tell you how it works for us. By the way, about the download of a single 
file between the two conections, do you know if there is a way to do it?

Thanks for your help and time and sorry for the delay!

_________________________________________________________________
Prodigy/MSN Spaces: Tu espacio en la red http://spaces.msn.com/

_______________________________________________
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

RE: [LARTC] Problem with Load Balancing

[William T Mullaney]

[-- Attachment #1.1: Type: text/plain, Size: 1691 bytes --]

To my knowledge, there is no way to download one file from two different
connections connected to two different ISPs at the same time.  If you are
running BGP then you might be able to load balance across the two links, but
that would require your upstream providers to allow you to use it, and
possibly the purchase of a public AS number an IP address space depending on
the setup.  If you are doing NAT past this link (IE both of your lines go
two the same ISP and same address blocks, but they want to give you 2x 10mb
links for 20mb total), then you can look at doing load balancing on layer 2
(Fast EtherChannel, bonding, Link Aggregate Groups, whatever), or creating 2
PPP style links between the computers and using a routing protocol like
OSPF, EIGRP (but not on Linux) or something.  I believe OSPF does equal cost
load balancing, BGP and EIGRP can, I think, do unequal cost load balancing.
But either way, I don't think that's the solution in your case.

The only other option I can think of would be some sort of software that
sends every other packet to a different IP or something, which would need to
run at the end you are downloading at or maybe at your ISPs, but I can't
think of anything like that.

-Will

-----Original Message-----
From: Vladimir Burciaga Aguilar [mailto:anakinv7@hotmail.com] 
Sent: Monday, September 18, 2006 12:09 PM
To: lartc@mailman.ds9a.nl
Subject: RE: [LARTC] Problem with Load Balancing

>We have also set up a somewhat similar method of load balancing.  Our
>traffic is never a 50-50 split (well 3:2 is how we have it set, but it
>doesn't always get close to that), but as the load picks up, it tends to be
>closer to the actual amount.

[snip]

[-- Attachment #1.2: Type: text/html, Size: 2433 bytes --]

[-- Attachment #2: Type: text/plain, Size: 143 bytes --]

_______________________________________________
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

RE: [LARTC] Problem with Load Balancing

[Raj Mathur]

>>>>> "William" = William T Mullaney <William> writes:

    William> To my knowledge, there is no way to download one file
    William> from two different connections connected to two different
    William> ISPs at the same time.  If you are running BGP then you
    William> might be able to load balance across the two links, but
    William> that would require your upstream providers to allow you
    William> to use it, and possibly the purchase of a public AS
    William> number an IP address space depending on the setup.  If
    William> you are doing NAT past this link (IE both of your lines
    William> go two the same ISP and same address blocks, but they
    William> want to give you 2x 10mb links for 20mb total), then you
    William> can look at doing load balancing on layer 2 (Fast
    William> EtherChannel, bonding, Link Aggregate Groups, whatever),
    William> or creating 2 PPP style links between the computers and
    William> using a routing protocol like OSPF, EIGRP (but not on
    William> Linux) or something.  I believe OSPF does equal cost load
    William> balancing, BGP and EIGRP can, I think, do unequal cost
    William> load balancing.  But either way, I don't think that's the
    William> solution in your case.

    William> The only other option I can think of would be some sort
    William> of software that sends every other packet to a different
    William> IP or something, which would need to run at the end you
    William> are downloading at or maybe at your ISPs, but I can't
    William> think of anything like that.

Wouldn't some download manager software that splits the file up into
multiple simultaneous downloads do the trick?  Agreed, not a single
download across multiple ISPs, but definitely a single file across
multiple ISPs.

Regards,

-- Raju
-- 
Raj Mathur                raju@kandalaya.org      http://kandalaya.org/
       GPG: 78D4 FC67 367F 40E2 0DD5  0FEF C968 D0EF CC68 D17F
                      It is the mind that moves
_______________________________________________
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

RE: [LARTC] Problem with Load Balancing

[William T Mullaney]

[-- Attachment #1.1: Type: text/plain, Size: 3931 bytes --]

Well, if you had a download manager and the system at the other side allowed
you to start your transfers in the middle of the file (which isn't out of
the question) that could potentially work.  The problem is that as far as I
see, there's nothing to force the second connection onto the second line.
It's been kind of a crap shoot of what line gets more information.  In
theory you could start the first download stream (and it's routed to ISP A),
then perhaps your email client goes out to check your POP account, so that
goes over ISP B.  The next connection, the second stream, now goes out over
ISP B again.  Honestly I don't know exactly how the equalize command for ip
route works, though I would think it says to always use the "less used"
connection (perhaps on PPS, BPS, % use, whatever, on a per second, 30
second, minute average?), but in my experience that and the weight options
don't ever get you exactly 50/50 (or whatever you specify) traffic.  

Things like bit torrent would probably perform better because there are
(possibly) many streams for each file, as would having 50 people downloading
files vs one.  It seems to be just like rolling dice, if you only roll twice
you might get two evens or two odds, but if you roll tons of times, you
should tend to get a more even distribution. 

-Will

-----Original Message-----
From: Raj Mathur [mailto:raju@linux-delhi.org] 
Sent: Sunday, September 24, 2006 2:49 PM
To: lartc@mailman.ds9a.nl
Subject: RE: [LARTC] Problem with Load Balancing

>>>>> "William" == William T Mullaney <William> writes:

    William> To my knowledge, there is no way to download one file
    William> from two different connections connected to two different
    William> ISPs at the same time.  If you are running BGP then you
    William> might be able to load balance across the two links, but
    William> that would require your upstream providers to allow you
    William> to use it, and possibly the purchase of a public AS
    William> number an IP address space depending on the setup.  If
    William> you are doing NAT past this link (IE both of your lines
    William> go two the same ISP and same address blocks, but they
    William> want to give you 2x 10mb links for 20mb total), then you
    William> can look at doing load balancing on layer 2 (Fast
    William> EtherChannel, bonding, Link Aggregate Groups, whatever),
    William> or creating 2 PPP style links between the computers and
    William> using a routing protocol like OSPF, EIGRP (but not on
    William> Linux) or something.  I believe OSPF does equal cost load
    William> balancing, BGP and EIGRP can, I think, do unequal cost
    William> load balancing.  But either way, I don't think that's the
    William> solution in your case.

    William> The only other option I can think of would be some sort
    William> of software that sends every other packet to a different
    William> IP or something, which would need to run at the end you
    William> are downloading at or maybe at your ISPs, but I can't
    William> think of anything like that.

Wouldn't some download manager software that splits the file up into
multiple simultaneous downloads do the trick?  Agreed, not a single download
across multiple ISPs, but definitely a single file across multiple ISPs.

Regards,

-- Raju
-- 
Raj Mathur                raju@kandalaya.org      http://kandalaya.org/
       GPG: 78D4 FC67 367F 40E2 0DD5  0FEF C968 D0EF CC68 D17F
                      It is the mind that moves
_______________________________________________
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

______________________________________________________________________
This email has been scanned by the MessageLabs Email Security System.
For more information please visit http://www.messagelabs.com/email
______________________________________________________________________

[-- Attachment #1.2: Type: text/html, Size: 6416 bytes --]

[-- Attachment #2: Type: text/plain, Size: 143 bytes --]

_______________________________________________
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

Re: [LARTC] Problem with Load Balancing

[Alessandro Ren]

    The second connections will problably gets routed though the same 
link because of route cache I think.

    []s.

William T Mullaney wrote:
>
> Well, if you had a download manager and the system at the other side 
> allowed you to start your transfers in the middle of the file (which 
> isn't out of the question) that could potentially work.  The problem 
> is that as far as I see, there's nothing to force the second 
> connection onto the second line.  It's been kind of a crap shoot of 
> what line gets more information.  In theory you could start the first 
> download stream (and it's routed to ISP A), then perhaps your email 
> client goes out to check your POP account, so that goes over ISP B.  
> The next connection, the second stream, now goes out over ISP B 
> again.  Honestly I don't know exactly how the equalize command for ip 
> route works, though I would think it says to always use the "less 
> used" connection (perhaps on PPS, BPS, % use, whatever, on a per 
> second, 30 second, minute average?), but in my experience that and the 
> weight options don't ever get you exactly 50/50 (or whatever you 
> specify) traffic. 
>
> Things like bit torrent would probably perform better because there 
> are (possibly) many streams for each file, as would having 50 people 
> downloading files vs one.  It seems to be just like rolling dice, if 
> you only roll twice you might get two evens or two odds, but if you 
> roll tons of times, you should tend to get a more even distribution.
>
> -Will
>
> -----Original Message-----
> From: Raj Mathur [mailto:raju@linux-delhi.org]
> Sent: Sunday, September 24, 2006 2:49 PM
> To: lartc@mailman.ds9a.nl
> Subject: RE: [LARTC] Problem with Load Balancing
>
> >>>>> "William" = William T Mullaney <William> writes:
>
>     William> To my knowledge, there is no way to download one file
>     William> from two different connections connected to two different
>     William> ISPs at the same time.  If you are running BGP then you
>     William> might be able to load balance across the two links, but
>     William> that would require your upstream providers to allow you
>     William> to use it, and possibly the purchase of a public AS
>     William> number an IP address space depending on the setup.  If
>     William> you are doing NAT past this link (IE both of your lines
>     William> go two the same ISP and same address blocks, but they
>     William> want to give you 2x 10mb links for 20mb total), then you
>     William> can look at doing load balancing on layer 2 (Fast
>     William> EtherChannel, bonding, Link Aggregate Groups, whatever),
>     William> or creating 2 PPP style links between the computers and
>     William> using a routing protocol like OSPF, EIGRP (but not on
>     William> Linux) or something.  I believe OSPF does equal cost load
>     William> balancing, BGP and EIGRP can, I think, do unequal cost
>     William> load balancing.  But either way, I don't think that's the
>     William> solution in your case.
>
>     William> The only other option I can think of would be some sort
>     William> of software that sends every other packet to a different
>     William> IP or something, which would need to run at the end you
>     William> are downloading at or maybe at your ISPs, but I can't
>     William> think of anything like that.
>
> Wouldn't some download manager software that splits the file up into 
> multiple simultaneous downloads do the trick?  Agreed, not a single 
> download across multiple ISPs, but definitely a single file across 
> multiple ISPs.
>
> Regards,
>
> -- Raju
> -- 
> Raj Mathur                raju@kandalaya.org      http://kandalaya.org/
>        GPG: 78D4 FC67 367F 40E2 0DD5  0FEF C968 D0EF CC68 D17F
>                       It is the mind that moves 
> _______________________________________________
> LARTC mailing list
> LARTC@mailman.ds9a.nl
> http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc
>
> ______________________________________________________________________
> This email has been scanned by the MessageLabs Email Security System.
> For more information please visit http://www.messagelabs.com/email 
> ______________________________________________________________________
>
> ------------------------------------------------------------------------
>
> _______________________________________________
> LARTC mailing list
> LARTC@mailman.ds9a.nl
> http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc
>   
_______________________________________________
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc