[LARTC] Problems in Dead Gateway Detection / Failover - Multiple

[LARTC] Problems in Dead Gateway Detection / Failover - Multiple

[Manish Kathuria]

Hello,

I have configured a load balancing router using Julian's patches and as 
described in "nano.txt" for two ISP links as shown below.



              ISP 1                   ISP 2
                .                       .
                |                       |
                |                       |
                |                       |
                | WAN              WAN  |
              +-\-+                   +-\-+
              |   |                   |   |
              |R1 | GW1           GW2 |R2 |
              |   |------.    --------|   |
              |   |      |    |       |   |
              +---+      |    |       +---+
                    EXT1 |    | EXT2
                        +\----\-+
                        |       |
                        | LINUX |
                        | ROUTER|
                        |       |
                        |       |
                        |       |
                        +---/---+
                            | INT IF
                            |
                            |
                            |
                    /----------------\
                    |      LAN       |
                    |                |
                    \----------------/


LAN NETWORK = 192.168.100.0/24
INT IF = 192.168.100.1

ISP1 NETWORK = 10.20.30.128/29
R1 - ROUTER1
GW1  = 10.20.30.129
EXT1 = 10.20.30.130

ISP2 NETWORK = 172.16.32.128/29
R2 - ROUTER2
GW2  = 172.16.32.129
EXT2 = 172.16.32.130

Both the ISPs have provided /29 subnets of Public IPs. The above 
mentioned addresses are just for example.

The gateways for both the ISPs are routers placed at the same location 
which are further connected through Radio Link and Leased Line.

Things work fine as long as both the ISP links are alive. While testing 
the dead gateway detection and failover functionality we observed that 
if we make the first hop gateway (i.e Router R1 or R2) of one of the 
ISPs dead by either disconnecting the ethernet cable between Linux 
Router and R1/R2 or by switching off the gateway (R1/R2) itself, dead 
gateway detection takes place and failover to the other ISP takes place. 
  However, if there is a problem in the ISP connectivity at any of the 
subsequent hops, there is no dead gateway detection and failover also 
does not take place. I have tested this on various linux kernels from 
2.4 as well as 2.6 series.

Somehow I have never faced a similar problem before and things have been 
working perfectly. In real life situation here, the first hop gateway is 
rarely going to be down so dead gateway detection and failover is going 
to be required whenever there is some connectivity problem at any of the 
later hops. So that's where dead gateway detection needs to work.

What could be the reason ? How can this be resolved ? I would appreciate 
any pointers or suggestions.

Thanks,

Manish Kathuria
_______________________________________________
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

Re: [LARTC] Problems in Dead Gateway Detection / Failover -

[gypsy]

Manish Kathuria wrote:
--= snip =--
>   However, if there is a problem in the ISP connectivity at any of the
> subsequent hops, there is no dead gateway detection and failover also
> does not take place. I have tested this on various linux kernels from
> 2.4 as well as 2.6 series.
> 
> Somehow I have never faced a similar problem before and things have been
> working perfectly. In real life situation here, the first hop gateway is
> rarely going to be down so dead gateway detection and failover is going
> to be required whenever there is some connectivity problem at any of the
> later hops. So that's where dead gateway detection needs to work.
> 
> What could be the reason ? How can this be resolved ? I would appreciate
> any pointers or suggestions.
> 
> Thanks,
> 
> Manish Kathuria

Manish,

Same here (a long time ago.  I no longer have multiple ISPs).

I don't have any answers for you, but here are a few pointers:

Use arping in a script, pinging the farthest hop that arping can reach
that is of interest.  Whenever arping returns a bad status, run 'ip
route flush cache'.  Put a nice long sleep in the script and run it all
the time.

Perhaps in that same script, 'ping -n1 -I' each WAN interface in turn to
some destination that must always be up but reachable only by/on that
interface.  Run 'ip route flush cache' whenever that ping fails.

You are just trying to detect the up or down status of the link, so
don't flood the connection with arping and ping packets.  Using sleep,
space those pings apart to something sensible.

Although Julian has never confirmed (or denied) this, it was my
experience that only the **__FIRST__** nexhop affected the up or down
status of the connection.  If that succeeded, nothing would flag the
connection as dead.  If you know C, perhaps you can examine Julian's
kernel patch to see if there is any useful information there.  In my
opinion, Julian should document exactly how DGD works.  Perhaps he has
and I just can't find it on his web site, but (when I cared), I was not
able to find anything useful there.

Have you tried to engage Julian in a conversation to resolve this?  He
posts here occasionally but I do not know if he answers questions about
DGD off this list.
--
gypsy
_______________________________________________
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

Re: [LARTC] Problems in Dead Gateway Detection / Failover -

[Manish Kathuria]

gypsy wrote:
> Manish Kathuria wrote:
> --= snip =--
> 
>>  However, if there is a problem in the ISP connectivity at any of the
>>subsequent hops, there is no dead gateway detection and failover also
>>does not take place. I have tested this on various linux kernels from
>>2.4 as well as 2.6 series.
>>
>>Somehow I have never faced a similar problem before and things have been
>>working perfectly. In real life situation here, the first hop gateway is
>>rarely going to be down so dead gateway detection and failover is going
>>to be required whenever there is some connectivity problem at any of the
>>later hops. So that's where dead gateway detection needs to work.
>>
>>What could be the reason ? How can this be resolved ? I would appreciate
>>any pointers or suggestions.
>>
>>Thanks,
>>
>>Manish Kathuria
> 
> 
> Manish,
> 
> Same here (a long time ago.  I no longer have multiple ISPs).
> 
> I don't have any answers for you, but here are a few pointers:

Thanks for your mail. I wil try out the suggestions given by you.

> 
> Use arping in a script, pinging the farthest hop that arping can reach
> that is of interest.  Whenever arping returns a bad status, run 'ip
> route flush cache'.  Put a nice long sleep in the script and run it all
> the time. >
> Perhaps in that same script, 'ping -n1 -I' each WAN interface in turn to
> some destination that must always be up but reachable only by/on that
> interface.  Run 'ip route flush cache' whenever that ping fails.

The only thing is whether by doing this the kernel would be able to mark 
the gateway having bad status as down or not. If it does not any other 
intervention, then its really superb.

> 
> You are just trying to detect the up or down status of the link, so
> don't flood the connection with arping and ping packets.  Using sleep,
> space those pings apart to something sensible.

I was thinking of writing a daemon which will ping a remote host through 
each of the WAN interfaces every 5 seconds. If one of them gives a bad 
status response continuosly for 8-10 times, the default route will be 
changed to the other ISP's gateway and if the status changes again, it 
will be restored back to the load balanced multipath state.

Will have to actually try and see which method fits in better here and 
is more elegant. If your suggestion works, its perhaps the best way out.

> 
> Although Julian has never confirmed (or denied) this, it was my
> experience that only the **__FIRST__** nexhop affected the up or down
> status of the connection.  If that succeeded, nothing would flag the
> connection as dead.  If you know C, perhaps you can examine Julian's
> kernel patch to see if there is any useful information there.  In my
> opinion, Julian should document exactly how DGD works.  Perhaps he has
> and I just can't find it on his web site, but (when I cared), I was not
> able to find anything useful there.

There are excellent documents at http://www.ssi.bg/~ja/dgd-usage.txt and 
http://www.ssi.bg/~ja/nano.txt which have explained it very well. 
Quoting from the dgd-usage.txt document here ...


---Begin Quote---

* the alternative routes check the neighbour state not only for gateways
but  for hosts, i.e. for any kind of neighbours. Note that in some cases
the  neighbour  can remain  in reachable  state  while its  nexthops are
failed.   For example, it is even possible the gateway to be a proxy ARP
server  and the gateway IP to remain  always in reachable state. In such
case we can not notice the real state of the gateway's IP.

* the alternative routes can be a list from unipath or multipath routes,
using  NOARP  and  ARP devices.  As  result,  the first  alive  or first
suspected  (but not dead)  route is selected by  inspecting the state of
the gateways in each path or the neighbours through the used device from
the path.

* as  result we take care of the state of each path in a multipath route
and  we  try to  use  only the  alive  paths considering  their relative
weights

---End Quote---

In the current situaion I am dealing with, the firsthop gateway is 
always reachable. It is only the subsequent hops which can go down. And 
when that happens, the dead gateway detection doesnt work, the outgoing 
traffic keeps on going out through the dead ISP's WAN interface. But 
what confuses me is that DGD does work for one of the ISPs which is also 
identically connected.

Could running routed / gated play a role here in resolving this problem ?

> 
> Have you tried to engage Julian in a conversation to resolve this?  He
> posts here occasionally but I do not know if he answers questions about
> DGD off this list.

I have not done it so far.

> --
> gypsy
> 

Thanks once again for your suggestions.

--
Manish Kathuria
_______________________________________________
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

Re: [LARTC] Problems in Dead Gateway Detection / Failover - Multiple

[Eduardo Fernández]

Hi!

Did you finally write a script for dead gateway detection beyond first
hop? Did you find any other solution to this problem? I'm quite
interested and I bet other multipath users here are interested too.

My linux router has 10 dsl links (adding 15 more in short), when one
of the dsl routers goes down the kernel does not always notice. Don't
know why. Also, if a dsl route is up but the internet link is down
dead gateway detection doesn't work either.

Thanks!

Edu


On 1/26/06, Manish Kathuria <manish@tuxspace.com> wrote:
> Hello,
>
> I have configured a load balancing router using Julian's patches and as
> described in "nano.txt" for two ISP links as shown below.
>
>
>
>               ISP 1                   ISP 2
>                 .                       .
>                 |                       |
>                 |                       |
>                 |                       |
>                 | WAN              WAN  |
>               +-\-+                   +-\-+
>               |   |                   |   |
>               |R1 | GW1           GW2 |R2 |
>               |   |------.    --------|   |
>               |   |      |    |       |   |
>               +---+      |    |       +---+
>                     EXT1 |    | EXT2
>                         +\----\-+
>                         |       |
>                         | LINUX |
>                         | ROUTER|
>                         |       |
>                         |       |
>                         |       |
>                         +---/---+
>                             | INT IF
>                             |
>                             |
>                             |
>                     /----------------\
>                     |      LAN       |
>                     |                |
>                     \----------------/
>
>
> LAN NETWORK = 192.168.100.0/24
> INT IF = 192.168.100.1
>
> ISP1 NETWORK = 10.20.30.128/29
> R1 - ROUTER1
> GW1  = 10.20.30.129
> EXT1 = 10.20.30.130
>
> ISP2 NETWORK = 172.16.32.128/29
> R2 - ROUTER2
> GW2  = 172.16.32.129
> EXT2 = 172.16.32.130
>
> Both the ISPs have provided /29 subnets of Public IPs. The above
> mentioned addresses are just for example.
>
> The gateways for both the ISPs are routers placed at the same location
> which are further connected through Radio Link and Leased Line.
>
> Things work fine as long as both the ISP links are alive. While testing
> the dead gateway detection and failover functionality we observed that
> if we make the first hop gateway (i.e Router R1 or R2) of one of the
> ISPs dead by either disconnecting the ethernet cable between Linux
> Router and R1/R2 or by switching off the gateway (R1/R2) itself, dead
> gateway detection takes place and failover to the other ISP takes place.
>   However, if there is a problem in the ISP connectivity at any of the
> subsequent hops, there is no dead gateway detection and failover also
> does not take place. I have tested this on various linux kernels from
> 2.4 as well as 2.6 series.
>
> Somehow I have never faced a similar problem before and things have been
> working perfectly. In real life situation here, the first hop gateway is
> rarely going to be down so dead gateway detection and failover is going
> to be required whenever there is some connectivity problem at any of the
> later hops. So that's where dead gateway detection needs to work.
>
> What could be the reason ? How can this be resolved ? I would appreciate
> any pointers or suggestions.
>
> Thanks,
>
> Manish Kathuria
> _______________________________________________
> LARTC mailing list
> LARTC@mailman.ds9a.nl
> http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc
>
_______________________________________________
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

Re:[LARTC] Problems in Dead Gateway Detection / Failover -

[Shashikant Mundlik]

[-- Attachment #1.1: Type: text/plain, Size: 1119 bytes --]

Hi There,
 
I am also trying to do the same for my network. 
I have two links from different ISPs and I want to configure a failover and
load balancing Linux router.
 
I am facing same problem here, that how to detect link failure and let Linux
box switch the gateway.
 
I know it works when the first gateway is physically down and not reachable.
But what to do if my link is up but there is problem at nexthop level and
its not routing packets to destination. 
 
Please tell me if this can be overcome by setting multipath routing. 
 
Another way I can think of doing this is to use a script which will check if
the default route is alive every 15 mins and if not it will make changes in
routing table and route the packets through different link.
 
I don't know if this is the best way to do this. If any one know how to do
this better please share. 
 
If you guys thinks this can work, lets help each other to write such scrip.
 
I am new to LARTC and just now started learning it to solve my network
problems.
 
Please help me to achieve this.
 
Thanks in advance.
 
Regards,
 

Shashikant Mundlik 

Pune, India.


[-- Attachment #1.2: Type: text/html, Size: 5917 bytes --]

[-- Attachment #2: Type: text/plain, Size: 143 bytes --]

_______________________________________________
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

Re: [LARTC] Problems in Dead Gateway Detection / Failover

[Alessandro Ren]

[-- Attachment #1.1: Type: text/plain, Size: 2527 bytes --]


    I have a script that connects to 20 diferent sites on the port 80 
coming from each link interface a have on my linux router.
    If I reach less than 20% of my sites, I assume the link is down and 
do all the routing and firewall adjustments to make the traffic goes to 
other routes, removing the problematic link out, setting ip rules, 
routes in tables and the main multipath default route and commenting in 
the firewall the MARKs the would go via the link thats down and it also 
sets QoS and tries to bring the link that is down back UP.
    Althought I've tested with only 3 links, it supports any number of them.
    It's works very nice so far.

    []s.


Shashikant Mundlik wrote:
> Hi There,
>  
> I am also trying to do the same for my network.
> I have two links from different ISPs and I want to configure a 
> failover and load balancing Linux router.
>  
> I am facing same problem here, that how to detect link failure and let 
> Linux box switch the gateway.
>  
> I know it works when the first gateway is physically down and not 
> reachable. But what to do if my link is up but there is problem at 
> nexthop level and its not routing packets to destination.
>  
> Please tell me if this can be overcome by setting multipath routing.
>  
> Another way I can think of doing this is to use a script which will 
> check if the default route is alive every 15 mins and if not it will 
> make changes in routing table and route the packets through different 
> link.
>  
> I don't know if this is the best way to do this. If any one know how 
> to do this better please share.
>  
> If you guys thinks this can work, lets help each other to write such 
> scrip.
>  
> I am new to LARTC and just now started learning it to solve my network 
> problems.
>  
> Please help me to achieve this.
>  
> Thanks in advance.
>  
> Regards,
>  
>
> Shashikant Mundlik
>
> Pune, India.
>
> ------------------------------------------------------------------------
>
> _______________________________________________
> LARTC mailing list
> LARTC@mailman.ds9a.nl
> http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc
>   

-- 
__________________________________________________
*Alessandro Ren*
	/*OpServices*/
/*Luciana de Abreu, 471 - Sala 403*/
/*Porto Alegre, RS - CEP 90570-060*/

*(*   phone 55(51)3061-3588
*4*    fax 55(51)3061-3588
	*Q*   mobile 55(51)8151-8212
*:*   email alessandro.ren@opservices.com.br 
<mailto:%22alessandro.ren@opservices.com.br%22>

__________________________________________________

[-- Attachment #1.2: Type: text/html, Size: 9290 bytes --]

[-- Attachment #2: Type: text/plain, Size: 143 bytes --]

_______________________________________________
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

Re: [LARTC] Problems in Dead Gateway Detection / Failover

[Alessandro Ren]

[-- Attachment #1.1: Type: text/plain, Size: 5078 bytes --]


    I bind to the interface IP and connect to 20 different sites or 
more, the sites are listed in a text file, using the TCP connect in perl.
    Off course, the ip rule tables the the marks in the firewall must be 
set correcly so you know that the connections are going through the 
right interface.
    I can share de script, it's a litle complex in its structus, as it 
depends on some external scripts, but I will try the share and problably 
get more and better ideas to do the fail over / multi path routing.
    I will prepare and sent a email with it shortly.

    []s.

Shashikant Mundlik wrote:
> Hi Ren,
>  
> Thanks for your help. But how do you check that you reach less than 20 
> of your sites. (do you mean 20 websites?).
> Will you able to share the script?
> That will be great help.
>  
> Thanks and regards,
>  
>
> *Shashikant** Mundlik *
>
>  
>
> System Administrator
>
> UBICS, Pune                                        
> Phone:  91 20 2729 1004  x 138  
> Mobile : 91 9372 044015
>
>  
>
> www.ubics.com <blocked::http://www.ubics.com/>
>
> The UB Group
>
> DISCLAIMER AND PRIVILEGE NOTICE:
> This e-mail message contains confidential, copyright, proprietary and 
> legally privileged information. It should not be used by anyone who is 
> not the original intended recipient. If you have erroneously received 
> this message, please delete it immediately and notify the sender. The 
> recipient must note and understand that any views expressed in this 
> message are those of the individual sender and no binding nature of 
> the message shall be implied or assumed unless the sender does so 
> expressly with due authority of UBICS, Inc.
>
>  
>
>  
>
> ------------------------------------------------------------------------
> *From:* Alessandro Ren [mailto:alessandro.ren@opservices.com.br]
> *Sent:* Monday, April 17, 2006 7:31 PM
> *To:* smundlik@ubicsindia.com
> *Cc:* manish@tuxspace.com; lartc@mailman.ds9a.nl
> *Subject:* Re: [LARTC] Problems in Dead Gateway Detection / Failover - 
> MultipleISP Links
>
>
>     I have a script that connects to 20 diferent sites on the port 80 
> coming from each link interface a have on my linux router.
>     If I reach less than 20% of my sites, I assume the link is down 
> and do all the routing and firewall adjustments to make the traffic 
> goes to other routes, removing the problematic link out, setting ip 
> rules, routes in tables and the main multipath default route and 
> commenting in the firewall the MARKs the would go via the link thats 
> down and it also sets QoS and tries to bring the link that is down 
> back UP.
>     Althought I've tested with only 3 links, it supports any number of 
> them.
>     It's works very nice so far.
>
>     []s.
>
>
> Shashikant Mundlik wrote:
>> Hi There,
>>  
>> I am also trying to do the same for my network.
>> I have two links from different ISPs and I want to configure a 
>> failover and load balancing Linux router.
>>  
>> I am facing same problem here, that how to detect link failure and 
>> let Linux box switch the gateway.
>>  
>> I know it works when the first gateway is physically down and not 
>> reachable. But what to do if my link is up but there is problem at 
>> nexthop level and its not routing packets to destination.
>>  
>> Please tell me if this can be overcome by setting multipath routing.
>>  
>> Another way I can think of doing this is to use a script which will 
>> check if the default route is alive every 15 mins and if not it will 
>> make changes in routing table and route the packets through different 
>> link.
>>  
>> I don't know if this is the best way to do this. If any one know how 
>> to do this better please share.
>>  
>> If you guys thinks this can work, lets help each other to write such 
>> scrip.
>>  
>> I am new to LARTC and just now started learning it to solve my 
>> network problems.
>>  
>> Please help me to achieve this.
>>  
>> Thanks in advance.
>>  
>> Regards,
>>  
>>
>> Shashikant Mundlik
>>
>> Pune, India.
>>
>> ------------------------------------------------------------------------
>>
>> _______________________________________________
>> LARTC mailing list
>> LARTC@mailman.ds9a.nl
>> http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc
>>   
>
> -- 
> __________________________________________________
> *Alessandro Ren*
> 	/*OpServices*/
> /*Luciana de Abreu, 471 - Sala 403*/
> /*Porto Alegre, RS - CEP 90570-060*/
>
> *(*   phone 55(51)3061-3588
> *4*    fax 55(51)3061-3588
> 	*Q*   mobile 55(51)8151-8212
> *:*   email alessandro.ren@opservices.com.br 
> <mailto:%22alessandro.ren@opservices.com.br%22>
>
> __________________________________________________

-- 
__________________________________________________
*Alessandro Ren*
	/*OpServices*/
/*Luciana de Abreu, 471 - Sala 403*/
/*Porto Alegre, RS - CEP 90570-060*/

*(*   phone 55(51)3061-3588
*4*    fax 55(51)3061-3588
	*Q*   mobile 55(51)8151-8212
*:*   email alessandro.ren@opservices.com.br 
<mailto:%22alessandro.ren@opservices.com.br%22>

__________________________________________________

[-- Attachment #1.2: Type: text/html, Size: 19012 bytes --]

[-- Attachment #2: Type: text/plain, Size: 143 bytes --]

_______________________________________________
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

RE: [LARTC] Problems in Dead Gateway Detection / Failover

[Shashikant Mundlik]

[-- Attachment #1.1: Type: text/plain, Size: 3634 bytes --]

Hi Ren,
 
Thanks for your help. But how do you check that you reach less than 20 of
your sites. (do you mean 20 websites?).
Will you able to share the script? 
That will be great help. 
 
Thanks and regards,
 

Shashikant Mundlik 

 

System Administrator

UBICS, Pune                                        
Phone:  91 20 2729 1004  x 138  
Mobile : 91 9372 044015 

 

www.ubics.com <blocked::http://www.ubics.com/> 

The UB Group

DISCLAIMER AND PRIVILEGE NOTICE:
This e-mail message contains confidential, copyright, proprietary and
legally privileged information. It should not be used by anyone who is not
the original intended recipient. If you have erroneously received this
message, please delete it immediately and notify the sender. The recipient
must note and understand that any views expressed in this message are those
of the individual sender and no binding nature of the message shall be
implied or assumed unless the sender does so expressly with due authority of
UBICS, Inc.

 

 

  _____  

From: Alessandro Ren [mailto:alessandro.ren@opservices.com.br] 
Sent: Monday, April 17, 2006 7:31 PM
To: smundlik@ubicsindia.com
Cc: manish@tuxspace.com; lartc@mailman.ds9a.nl
Subject: Re: [LARTC] Problems in Dead Gateway Detection / Failover -
MultipleISP Links



    I have a script that connects to 20 diferent sites on the port 80 coming
from each link interface a have on my linux router.
    If I reach less than 20% of my sites, I assume the link is down and do
all the routing and firewall adjustments to make the traffic goes to other
routes, removing the problematic link out, setting ip rules, routes in
tables and the main multipath default route and commenting in the firewall
the MARKs the would go via the link thats down and it also sets QoS and
tries to bring the link that is down back UP.
    Althought I've tested with only 3 links, it supports any number of them.
    It's works very nice so far.

    []s.


Shashikant Mundlik wrote: 

Hi There,
 
I am also trying to do the same for my network. 
I have two links from different ISPs and I want to configure a failover and
load balancing Linux router.
 
I am facing same problem here, that how to detect link failure and let Linux
box switch the gateway.
 
I know it works when the first gateway is physically down and not reachable.
But what to do if my link is up but there is problem at nexthop level and
its not routing packets to destination. 
 
Please tell me if this can be overcome by setting multipath routing. 
 
Another way I can think of doing this is to use a script which will check if
the default route is alive every 15 mins and if not it will make changes in
routing table and route the packets through different link.
 
I don't know if this is the best way to do this. If any one know how to do
this better please share. 
 
If you guys thinks this can work, lets help each other to write such scrip.
 
I am new to LARTC and just now started learning it to solve my network
problems.
 
Please help me to achieve this.
 
Thanks in advance.
 
Regards,
 

Shashikant Mundlik 



Pune, India.



  _____  


_______________________________________________

LARTC mailing list

LARTC@mailman.ds9a.nl

http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

  


-- 
__________________________________________________

Alessandro Ren
	OpServices
Luciana de Abreu, 471 - Sala 403
Porto Alegre, RS - CEP 90570-060
	
*   phone 55(51)3061-3588
*    fax 55(51)3061-3588
	*   mobile 55(51)8151-8212
*   email alessandro.ren@opservices.com.br
<mailto:%22alessandro.ren@opservices.com.br%22> 
	
__________________________________________________


[-- Attachment #1.2: Type: text/html, Size: 16001 bytes --]

[-- Attachment #2: Type: text/plain, Size: 143 bytes --]

_______________________________________________
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

RE: [LARTC] Problems in Dead Gateway Detection / Failover

[Shashikant Mundlik]

[-- Attachment #1.1: Type: text/plain, Size: 4978 bytes --]

Thanks a lot Ren!
That will be a great help. 
 
Thanks,
 

Shashikant Mundlik 

 



  _____  

From: Alessandro Ren [mailto:alessandro.ren@opservices.com.br] 
Sent: Monday, April 17, 2006 8:47 PM
To: smundlik@ubicsindia.com
Cc: manish@tuxspace.com; lartc@mailman.ds9a.nl
Subject: Re: [LARTC] Problems in Dead Gateway Detection / Failover -
MultipleISP Links




    I bind to the interface IP and connect to 20 different sites or more,
the sites are listed in a text file, using the TCP connect in perl.
    Off course, the ip rule tables the the marks in the firewall must be set
correcly so you know that the connections are going through the right
interface.
    I can share de script, it's a litle complex in its structus, as it
depends on some external scripts, but I will try the share and problably get
more and better ideas to do the fail over / multi path routing.
    I will prepare and sent a email with it shortly.

    []s.

Shashikant Mundlik wrote: 

Hi Ren,
 
Thanks for your help. But how do you check that you reach less than 20 of
your sites. (do you mean 20 websites?).
Will you able to share the script? 
That will be great help. 
 
Thanks and regards,
 

Shashikant Mundlik 



System Administrator

UBICS, Pune                                        
Phone:  91 20 2729 1004  x 138  
Mobile : 91 9372 044015 





www.ubics.com <blocked::http://www.ubics.com/> 

The UB Group

DISCLAIMER AND PRIVILEGE NOTICE:
This e-mail message contains confidential, copyright, proprietary and
legally privileged information. It should not be used by anyone who is not
the original intended recipient. If you have erroneously received this
message, please delete it immediately and notify the sender. The recipient
must note and understand that any views expressed in this message are those
of the individual sender and no binding nature of the message shall be
implied or assumed unless the sender does so expressly with due authority of
UBICS, Inc.



 

  _____  

From: Alessandro Ren [mailto:alessandro.ren@opservices.com.br] 
Sent: Monday, April 17, 2006 7:31 PM
To: smundlik@ubicsindia.com
Cc: manish@tuxspace.com; lartc@mailman.ds9a.nl
Subject: Re: [LARTC] Problems in Dead Gateway Detection / Failover -
MultipleISP Links



    I have a script that connects to 20 diferent sites on the port 80 coming
from each link interface a have on my linux router.
    If I reach less than 20% of my sites, I assume the link is down and do
all the routing and firewall adjustments to make the traffic goes to other
routes, removing the problematic link out, setting ip rules, routes in
tables and the main multipath default route and commenting in the firewall
the MARKs the would go via the link thats down and it also sets QoS and
tries to bring the link that is down back UP.
    Althought I've tested with only 3 links, it supports any number of them.
    It's works very nice so far.

    []s.


Shashikant Mundlik wrote: 

Hi There,
 
I am also trying to do the same for my network. 
I have two links from different ISPs and I want to configure a failover and
load balancing Linux router.
 
I am facing same problem here, that how to detect link failure and let Linux
box switch the gateway.
 
I know it works when the first gateway is physically down and not reachable.
But what to do if my link is up but there is problem at nexthop level and
its not routing packets to destination. 
 
Please tell me if this can be overcome by setting multipath routing. 
 
Another way I can think of doing this is to use a script which will check if
the default route is alive every 15 mins and if not it will make changes in
routing table and route the packets through different link.
 
I don't know if this is the best way to do this. If any one know how to do
this better please share. 
 
If you guys thinks this can work, lets help each other to write such scrip.
 
I am new to LARTC and just now started learning it to solve my network
problems.
 
Please help me to achieve this.
 
Thanks in advance.
 
Regards,
 

Shashikant Mundlik 



Pune, India.



  _____  


_______________________________________________

LARTC mailing list

LARTC@mailman.ds9a.nl

http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

  


-- 
__________________________________________________

Alessandro Ren
	OpServices
Luciana de Abreu, 471 - Sala 403
Porto Alegre, RS - CEP 90570-060
	
*   phone 55(51)3061-3588
*    fax 55(51)3061-3588
	*   mobile 55(51)8151-8212
*   email alessandro.ren@opservices.com.br
<mailto:%22alessandro.ren@opservices.com.br%22> 
	
__________________________________________________



-- 
__________________________________________________

Alessandro Ren
	OpServices
Luciana de Abreu, 471 - Sala 403
Porto Alegre, RS - CEP 90570-060
	
*   phone 55(51)3061-3588
*    fax 55(51)3061-3588
	*   mobile 55(51)8151-8212
*   email alessandro.ren@opservices.com.br
<mailto:%22alessandro.ren@opservices.com.br%22> 
	
__________________________________________________


[-- Attachment #1.2: Type: text/html, Size: 22660 bytes --]

[-- Attachment #2: Type: text/plain, Size: 143 bytes --]

_______________________________________________
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

RE: [LARTC] Problems in Dead Gateway Detection /

[LinuXKiD]

Hi,

I've some similar:

I croned a perl script that every 2 minutes check via ICMP
some referential host ( for each "default route").

If some route is down , I take off it from "default routes table".

But I think that make it by TCP connect at 80 port is better.

bests.

andres



-----Mensaje original-----
De: lartc-bounces@mailman.ds9a.nl [mailto:lartc-bounces@mailman.ds9a.nl]En
nombre de Alessandro Ren
Enviado el: Lunes, 17 de Abril de 2006 12:17 p.m.
Para: smundlik@ubicsindia.com
CC: lartc@mailman.ds9a.nl
Asunto: Re: [LARTC] Problems in Dead Gateway Detection / Failover-
MultipleISP Links



    I bind to the interface IP and connect to 20 different sites or more,
the sites are listed in a text file, using the TCP connect in perl.
    Off course, the ip rule tables the the marks in the firewall must be set
correcly so you know that the connections are going through the right
interface.
    I can share de script, it's a litle complex in its structus, as it
depends on some external scripts, but I will try the share and problably get
more and better ideas to do the fail over / multi path routing.
    I will prepare and sent a email with it shortly.

    []s.

Shashikant Mundlik wrote:
Hi Ren,

Thanks for your help. But how do you check that you reach less than 20 of
your sites. (do you mean 20 websites?).
Will you able to share the script?
That will be great help.

Thanks and regards,

Shashikant Mundlik

System Administrator
UBICS, Pune
Phone:  91 20 2729 1004  x 138
Mobile : 91 9372 044015

www.ubics.com
The UB Group
DISCLAIMER AND PRIVILEGE NOTICE:
This e-mail message contains confidential, copyright, proprietary and
legally privileged information. It should not be used by anyone who is not
the original intended recipient. If you have erroneously received this
message, please delete it immediately and notify the sender. The recipient
must note and understand that any views expressed in this message are those
of the individual sender and no binding nature of the message shall be
implied or assumed unless the sender does so expressly with due authority of
UBICS, Inc.






From: Alessandro Ren [mailto:alessandro.ren@opservices.com.br]
Sent: Monday, April 17, 2006 7:31 PM
To: smundlik@ubicsindia.com
Cc: manish@tuxspace.com; lartc@mailman.ds9a.nl
Subject: Re: [LARTC] Problems in Dead Gateway Detection / Failover -
MultipleISP Links



    I have a script that connects to 20 diferent sites on the port 80 coming
from each link interface a have on my linux router.
    If I reach less than 20% of my sites, I assume the link is down and do
all the routing and firewall adjustments to make the traffic goes to other
routes, removing the problematic link out, setting ip rules, routes in
tables and the main multipath default route and commenting in the firewall
the MARKs the would go via the link thats down and it also sets QoS and
tries to bring the link that is down back UP.
    Althought I've tested with only 3 links, it supports any number of them.
    It's works very nice so far.

    []s.


Shashikant Mundlik wrote:
Hi There,

I am also trying to do the same for my network.
I have two links from different ISPs and I want to configure a failover and
load balancing Linux router.

I am facing same problem here, that how to detect link failure and let Linux
box switch the gateway.

I know it works when the first gateway is physically down and not reachable.
But what to do if my link is up but there is problem at nexthop level and
its not routing packets to destination.

Please tell me if this can be overcome by setting multipath routing.

Another way I can think of doing this is to use a script which will check if
the default route is alive every 15 mins and if not it will make changes in
routing table and route the packets through different link.

I don't know if this is the best way to do this. If any one know how to do
this better please share.

If you guys thinks this can work, lets help each other to write such scrip.

I am new to LARTC and just now started learning it to solve my network
problems.

Please help me to achieve this.

Thanks in advance.

Regards,

Shashikant Mundlik
Pune, India.

_______________________________________________
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc



--
__________________________________________________
Alessandro Ren
OpServices
Luciana de Abreu, 471 - Sala 403
Porto Alegre, RS - CEP 90570-060

(   phone 55(51)3061-3588
4    fax 55(51)3061-3588
Q   mobile 55(51)8151-8212
:   email alessandro.ren@opservices.com.br

__________________________________________________



--
__________________________________________________
Alessandro Ren
OpServices
Luciana de Abreu, 471 - Sala 403
Porto Alegre, RS - CEP 90570-060

(   phone 55(51)3061-3588
4    fax 55(51)3061-3588
Q   mobile 55(51)8151-8212
:   email alessandro.ren@opservices.com.br

__________________________________________________

_______________________________________________
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

Re: [LARTC] Problems in Dead Gateway Detection / Failover

[Alessandro Ren]

[-- Attachment #1.1: Type: text/plain, Size: 6333 bytes --]


    So, I will try to explain how all the parts get together but in any 
doubt, just ask me:

    The main script is check_links_balanced.pl and it runs on the 
crontab in my case each minute or 2 minutes. In the beginning of the 
script there are some setups:
    $OPNET_CONF="/usr/local/scripts/opnet.conf";
     We have a service the we call OpNet, that's why the OPNET thing, 
so, this is where the configurations for the links are, I will attach my 
configuration so you can base yours, very simple.
$RCFIREWALL="/etc/rc.d/rc.firewall";
    Where your firewall script is, the main script need to check if the 
firewall is ok and change it if a link goes DOWN ou UP.

# hosts file
$HOSTS_FILE="/usr/local/scripts/hosts.txt";
    The lists of hosts, can be IPs ou names.

# logfile
$LOGFILE="/var/log/check_links_balanced.log";
    Well, the log ifle to see how things are going

# mininal % os hosts that must be UP to consider a link UP
$CRITICAL=30;

    So, you have to create an entry for each link and the 
/etc/iproute2/rt_tables using LINK1 , LINK2 and so on for the table name 
for each link that you have. This is important, because everything in 
connected to the link number, like, LINK1, the firewall mark 1 will send 
packets to the LINK1, will use the configurations of the rc.LINK1, will 
set the wshaper.LINK1 script and so on.
   
   Ok, so you will have a /etc/rc.d/rc.LINKx and /etc/rc.d/wshaper.LINKx 
for each link, these rc.LINKx will set the routing table LINKx properly 
and put the link UP, whether its a ethernet or ADSL with a PPP interface.
    For PPP interfaces, we will have some extra configurations in 
/etc/ppp, like /etc/ppp/ip-up that will have to set some routes when the 
ADSL goes UP, based on th interface, it will set default route for the 
table LINKx and set up rules, removing old rules if the IP is dynamic 
and setting the new one for the new IP interface. In /etc/ppp/peers you 
must create one configuraion for each PPP interface you have and each 
one gets an fixed name, using unit x, so I know the PPP0 will always be 
the same ADSL, otherwise linux will choose the number of the PPP 
interface dynamicly, and everything would be lost. I also  have one 
configuration for each PPPOE interface.
    The only thing that I can not do yet is work widh DHCP interfaces, I 
have still to see show dhclient can be used to to the same thing a I do 
with the PPP interfaces.

    The firewall has to have the following in mangle:

# here, one for each link wiht a MARK, in this case
# LINK1 - eth1 - is a cable with fixed IP. and LINK2 is and ADSL
$iptables -A OUTPUT -t mangle -o eth1 -j MARK --set-mark 1
$iptables -A OUTPUT -t mangle -o ppp0 -j MARK --set-mark 2

# CONNMARK PREROUTING
# pakets with state invalid can not be used with CONNMARK
$iptables -t mangle -A PREROUTING -j MARK --set-mark 10 -m state --state 
INVALID
$iptables -t mangle -A PREROUTING -j RETURN -m state --state INVALID

# if the paket belongs to an already known an "tagged" connection
#   then copy conmark -> mark and go ahead with routing
$iptables -t mangle -A PREROUTING -j CONNMARK --restore-mark
$iptables -t mangle -A PREROUTING -j RETURN -m mark ! --mark 0

# if it is a "untagged" connection and coming from an outside inteface
#   then save this as connmark and copy connmark -> mark
$iptables -t mangle -A PREROUTING -j CONNMARK --set-mark 1 -i eth1
$iptables -t mangle -A PREROUTING -j CONNMARK --set-mark 2 -i ppp0
$iptables -t mangle -A PREROUTING -j CONNMARK --restore-mark

# CONNMARK POSTROUTING
$iptables -A POSTROUTING -t mangle -m mark ! --mark 0 -j RETURN
$iptables -A POSTROUTING -t mangle -j MARK --set-mark 1 -m state --state 
NEW -o eth1
$iptables -A POSTROUTING -t mangle -j MARK --set-mark 2 -m state --state 
NEW -o ppp0
$iptables -A POSTROUTING -t mangle -j CONNMARK --save-mark -m state 
--state NEW

   This will balanced the internet access and you can set some 
connections to go a specific link

# Secure sites always via the same link, to keep integrity
$iptables -A PREROUTING -p tcp -t mangle -s 192.168.0.0/16 --dport 5000 
-j MARK --set-mark 1
   So here LAN access to port TCP 5000 will always get out via LINK1, 
when LINK1 is DOWN, the main scripts will comment this line OUT and run 
rc.firewall, so this packets will the go though the other links.
    See if you have tree links, you culd do that
$iptables -A PREROUTING -p tcp -t mangle -s 192.168.0.0/16 --dport 5000 
-j MARK --set-mark 3
$iptables -A PREROUTING -p tcp -t mangle -s 192.168.0.0/16 --dport 5000 
-j MARK --set-mark 2

    I will mark the same packts three time, CPU waste, but the packet 
would via LINK2, if LINK2 goes down, they would go via LINK3, if LINK3 
and LINK2 goes down, the lines get commented, the packets go via the 
remaing link or links.

    In the end of the scripts you have to have the NAT part
# NAT eth1
IP=`/usr/local/scripts/get_ip_interface.pl eth1`
$iptables -A POSTROUTING -t nat -m mark --mark 1 -j SNAT --to-source $IP

# NAT ppp0
IP=`/usr/local/scripts/get_ip_interface.pl ppp0`
$iptables -A POSTROUTING -t nat -m mark --mark 2 -j SNAT --to-source $IP
    You see that I first get the interface IP, that because the IP can 
change for dynamic links and the NAT must be reset to the new IP.

    Well, attached are the main script, the main configuration, the 
rc.LINKx and wshaper.LINKx that I use for my links as the ADSL 
configuration that I use here.
    I know this setup is complex and it took me a long time to get to 
it. I will answer any questions regarding it to try and help.
    I am using kernel 2.6.x and it also works for kernel 2.4.x with the 
CONNMAK patch.
    So, I am also attaching configure.pl script that generates all these 
configurations, yes, I've made it easy even for me.
    You can download the scripts and examples from here
    http://www.opservices.com.br/check_links_balanced.tgz

    Any help or improvements, let me now.

    []s.

-- 
__________________________________________________
*Alessandro Ren*
	/*OpServices*/
/*Luciana de Abreu, 471 - Sala 403*/
/*Porto Alegre, RS - CEP 90570-060*/

*(*   phone 55(51)3061-3588
*4*    fax 55(51)3061-3588
	*Q*   mobile 55(51)8151-8212
*:*   email alessandro.ren@opservices.com.br 
<mailto:%22alessandro.ren@opservices.com.br%22>

__________________________________________________

[-- Attachment #1.2: Type: text/html, Size: 9045 bytes --]

[-- Attachment #2: Type: text/plain, Size: 143 bytes --]

_______________________________________________
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

Re: [LARTC] Problems in Dead Gateway Detection / Failover - Multiple

[Manish Kathuria]

Eduardo Fernández wrote:
> Hi!
> 
> Did you finally write a script for dead gateway detection beyond first
> hop? Did you find any other solution to this problem? I'm quite
> interested and I bet other multipath users here are interested too.
> 
> My linux router has 10 dsl links (adding 15 more in short), when one
> of the dsl routers goes down the kernel does not always notice. Don't
> know why. Also, if a dsl route is up but the internet link is down
> dead gateway detection doesn't work either.
> 
> Thanks!
> 
> Edu
> 
> 
If you follow the nano.txt procedure and apply the patches, it works 
perfectly as long as the first hop is dead. But to ensure failover, when 
  connectivity goes down at any of the hops, you can use the nano.txt 
for configuring the interfaces and multipath routes (call it default 
configuration) and also run a script in the background to modify the 
routes as described below.

1. Periodically keep on checking if a remote host is reachable from each 
of the gateways by pinging it after every n seconds.

2. If the remote host is not reachable after a number of tries (which 
you can decide according to your own specific situation) from a 
particular gateway, remove that route. If you have just two internet 
links, there would be only one gateway left. But if you have more than 
two links alive you can again define multipath routes with appropriate 
weights for the active gateways. The possible combinations will increase 
exponentially with the increase in number of internet links so you will 
have to factor is all the cases in the script.

3. Restore the default configuration when the remote host is reachable 
from all the gateways.

I am not too sure how its going to behave with 10 links because if the 
links are not so stable it will result in very frequent changes.

--
Manish Kathuria
http://www.tuxspace.com /

_______________________________________________
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc