A smart router for more than one default routes

A smart router for more than one default routes

[Han Changzhe]

Hello experts,

I'm setting up a routing server on Linux with following links

 1. An Ethernet link (eth0) to the 1st internet link (fast, but can't
    access some sites);
 2. A VPN link (tun0) to provide services to local users;
 3. A VPN link (tun1) to a proxy server as the 2nd internet link (slow,
    free).

My target is:
   * for common internet access, routing the packets through eth0;
   * for the sites can't be accessed through eth0, routing them through 
tun1.

By now, I set the routing table manually for serveral sites and it works 
fine. Because there are thousands of them and the sites change with 
time, so I want a better solution.

My idea is like this: setting up more than one default routes for 
internet access, then dynamically change the route table (or route table 
cache) with some software according to the internet access results.

For example, if we get a timeout from https://www.google.com through 
eth0, the software should try it through tun1 link and, when succeed, 
adding the later route to current route table.

I don't know if any routing software on Linux work as I expected. I 
tried quagga with zebra + ospf but not successful.

FYI, it's not a common case for link based fail-over/load balance.


Please give me suggestions!

Thanks in advance,

Changzhe

Re: A smart router for more than one default routes

[Dave Taht]

This is not really the best place for a routing question. If you raise
your question on the quagga list, you might be better off.



On Wed, Feb 25, 2015 at 10:39 PM, Han Changzhe <hcz@nebulat.com> wrote:
> Hello experts,
>
> I'm setting up a routing server on Linux with following links
>
> 1. An Ethernet link (eth0) to the 1st internet link (fast, but can't
>    access some sites);
> 2. A VPN link (tun0) to provide services to local users;
> 3. A VPN link (tun1) to a proxy server as the 2nd internet link (slow,
>    free).
>
> My target is:
>   * for common internet access, routing the packets through eth0;
>   * for the sites can't be accessed through eth0, routing them through tun1.

Well, one of the things we have been working on in the homenet working group is
source specific routing, which could possibly help here, but it is
non-deterministic.

> By now, I set the routing table manually for serveral sites and it works
> fine. Because there are thousands of them and the sites change with time, so
> I want a better solution.
>
> My idea is like this: setting up more than one default routes for internet
> access, then dynamically change the route table (or route table cache) with
> some software according to the internet access results.
>
> For example, if we get a timeout from https://www.google.com through eth0,
> the software should try it through tun1 link and, when succeed, adding the
> later route to current route table.

Well you are conflating several layers of the protocol here.

It is hard to recognise a timeout, for example, without sniffing for
syns/syn_acks
on the gateway. That sniffer could simultaneously try a syn out one of
the vpn interfaces and if a syn/ack is not received from the main
interface, and one IS received from the vpn, insert a route for it.
You would still need to clean out that table periodically.

Then you would to insert and delete rules for each ip (or more likely
network) you wish to reroute
based on your measurements of what is working or not, and to otherwise
fall back to the default ethernet route.

Say for example you could not get dns from 8.8.8.8 locally.

ip route add 8.8.8.8 dev tun0

This doesnt help you on any protocols except tcp. udp apps are
different. so is quic, etc.

a bulk method would be to go through the alexa top 1 million to see
what you could and could not access, and set up routes for each (but
this does not handle your desire for 2 tunnels)

> I don't know if any routing software on Linux work as I expected. I tried
> quagga with zebra + ospf but not successful.

ospf? oy, no....

>
> FYI, it's not a common case for link based fail-over/load balance.
>
>
> Please give me suggestions!

Well, my way would probably involve a squid or polipo web proxy to
make the failover case easier. A lot of users would not dig that...

> Thanks in advance,
>
> Changzhe
>
> --
> To unsubscribe from this list: send the line "unsubscribe lartc" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html



-- 
Dave Täht
Let's make wifi fast, less jittery and reliable again!

https://plus.google.com/u/0/107942175615993706558/posts/TVX3o84jjmb

Re: A smart router for more than one default routes

[Erik Auerswald]

Hello Changzhe,

On Thu, Feb 26, 2015 at 02:39:03PM +0800, Han Changzhe wrote:
> I'm setting up a routing server on Linux with following links
> 
> 1. An Ethernet link (eth0) to the 1st internet link (fast, but can't
>    access some sites);
> 2. A VPN link (tun0) to provide services to local users;
> 3. A VPN link (tun1) to a proxy server as the 2nd internet link (slow,
>    free).
> 
> My target is:
>   * for common internet access, routing the packets through eth0;
>   * for the sites can't be accessed through eth0, routing them
> through tun1.
> 
> By now, I set the routing table manually for serveral sites and it
> works fine. Because there are thousands of them and the sites change
> with time, so I want a better solution.
> 
> My idea is like this: setting up more than one default routes for
> internet access, then dynamically change the route table (or route
> table cache) with some software according to the internet access
> results.
> 
> For example, if we get a timeout from https://www.google.com through
> eth0, the software should try it through tun1 link and, when
> succeed, adding the later route to current route table.
> 
> I don't know if any routing software on Linux work as I expected. I
> tried quagga with zebra + ospf but not successful.

As I understand it the list of networks inaccessible via eth0 is
maintained manually and needs to be synced to every site. The sites
are all configured identically, with eth0 as primary Internet access
interface, and tun1 as secondary.

The problem is not a good fit to traditional IP routing protocols (if
I understand it correctly).

I would advise to use some configuration management tool (puppet, chef,
cfengine, ...).

Alternatively, you could roll your own configuration update using
e.g. git or rsync to maintain one config file describing the routing
table, and a program (e.g. script called via cron) periodically checking
for changes in the config file, applying them if needed.

Cheers,
Erik
-- 
But heck, system administration is hard, what's a little more rope?
Here, hold this gun while I position your foot...
                        -- Valerie Aurora

Re: A smart router for more than one default routes

[Han Changzhe]

Thanks Dave! Please see below.

Changzhe

On 2015-02-26 15:30, Dave Taht wrote:
> This is not really the best place for a routing question. If you raise
> your question on the quagga list, you might be better off.
>

I couldn't find a better place for the question and thought guys in LARTC
should be professional enough to help. And thanks for your suggestion on
the quagga list! I'll try it.

>
> On Wed, Feb 25, 2015 at 10:39 PM, Han Changzhe <hcz@nebulat.com> wrote:
>> Hello experts,
>>
>> I'm setting up a routing server on Linux with following links
>>
>> 1. An Ethernet link (eth0) to the 1st internet link (fast, but can't
>>     access some sites);
>> 2. A VPN link (tun0) to provide services to local users;
>> 3. A VPN link (tun1) to a proxy server as the 2nd internet link (slow,
>>     free).
>>
>> My target is:
>>    * for common internet access, routing the packets through eth0;
>>    * for the sites can't be accessed through eth0, routing them through tun1.
> Well, one of the things we have been working on in the homenet working group is
> source specific routing, which could possibly help here, but it is
> non-deterministic.

As for the project you mentioned, are there any public materials for me 
to follow up?


>> By now, I set the routing table manually for serveral sites and it works
>> fine. Because there are thousands of them and the sites change with time, so
>> I want a better solution.
>>
>> My idea is like this: setting up more than one default routes for internet
>> access, then dynamically change the route table (or route table cache) with
>> some software according to the internet access results.
>>
>> For example, if we get a timeout from https://www.google.com through eth0,
>> the software should try it through tun1 link and, when succeed, adding the
>> later route to current route table.
> Well you are conflating several layers of the protocol here.
>
> It is hard to recognise a timeout, for example, without sniffing for
> syns/syn_acks
> on the gateway. That sniffer could simultaneously try a syn out one of
> the vpn interfaces and if a syn/ack is not received from the main
> interface, and one IS received from the vpn, insert a route for it.
Yes, a sniffer like that should work. Is it possible to allow the kernel
to raise an exception when TCP connection time-out happens so users can
handle the exception with the hock to try the vpn interface and manipulate
the route table?

> You would still need to clean out that table periodically.
Yes. With the old Linux routing cache, it might be easier. Then we only
add routes to the cache.

> Then you would to insert and delete rules for each ip (or more likely
> network) you wish to reroute
> based on your measurements of what is working or not, and to otherwise
> fall back to the default ethernet route.
>
> Say for example you could not get dns from 8.8.8.8 locally.
>
> ip route add 8.8.8.8 dev tun0
>
> This doesnt help you on any protocols except tcp. udp apps are
> different. so is quic, etc.
>
> a bulk method would be to go through the alexa top 1 million to see
> what you could and could not access, and set up routes for each (but
> this does not handle your desire for 2 tunnels)
Actually I prepared a routing list of around 3K items which should
work by now. It's just not beautiful.


>> I don't know if any routing software on Linux work as I expected. I tried
>> quagga with zebra + ospf but not successful.
> ospf? oy, no....
>
>> FYI, it's not a common case for link based fail-over/load balance.
>>
>>
>> Please give me suggestions!
> Well, my way would probably involve a squid or polipo web proxy to
> make the failover case easier. A lot of users would not dig that...
It should work for web accessing. But we still need to support ports
for email/ftp and other applications.

>> Thanks in advance,
>>
>> Changzhe
>>
>> --
>> To unsubscribe from this list: send the line "unsubscribe lartc" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
>

Re: A smart router for more than one default routes

[Han Changzhe]

Hi Erik,

Please see below.

Thanks,

Changzhe

On 2015-02-26 17:31, Erik Auerswald wrote:
> Hello Changzhe,
>
> On Thu, Feb 26, 2015 at 02:39:03PM +0800, Han Changzhe wrote:
>> I'm setting up a routing server on Linux with following links
>>
>> 1. An Ethernet link (eth0) to the 1st internet link (fast, but can't
>>     access some sites);
>> 2. A VPN link (tun0) to provide services to local users;
>> 3. A VPN link (tun1) to a proxy server as the 2nd internet link (slow,
>>     free).
>>
>> My target is:
>>    * for common internet access, routing the packets through eth0;
>>    * for the sites can't be accessed through eth0, routing them
>> through tun1.
>>
>> By now, I set the routing table manually for serveral sites and it
>> works fine. Because there are thousands of them and the sites change
>> with time, so I want a better solution.
>>
>> My idea is like this: setting up more than one default routes for
>> internet access, then dynamically change the route table (or route
>> table cache) with some software according to the internet access
>> results.
>>
>> For example, if we get a timeout from https://www.google.com through
>> eth0, the software should try it through tun1 link and, when
>> succeed, adding the later route to current route table.
>>
>> I don't know if any routing software on Linux work as I expected. I
>> tried quagga with zebra + ospf but not successful.
> As I understand it the list of networks inaccessible via eth0 is
> maintained manually and needs to be synced to every site. The sites
> are all configured identically, with eth0 as primary Internet access
> interface, and tun1 as secondary.
By now, the list is maintained manually while I wish the process to be 
automatic.
We may sync the routing list or not  because different sites may face 
different access
limitations.

In the ideal case, each site should maintain a small common routing list
which should be synced with a central server while  at the same time 
maintain its
local routing list which changes dynamically according to user requests 
and local
networking conditions. So syncing the routing table isn't the most tough 
problem.


> The problem is not a good fit to traditional IP routing protocols (if
> I understand it correctly).
I supposed it a simple and common case easily handled by available open 
source
software. Apparently it's not that easy.

> I would advise to use some configuration management tool (puppet, chef,
> cfengine, ...).
>
> Alternatively, you could roll your own configuration update using
> e.g. git or rsync to maintain one config file describing the routing
> table, and a program (e.g. script called via cron) periodically checking
> for changes in the config file, applying them if needed.
>
> Cheers,
> Erik

Re: A smart router for more than one default routes

[Dennis Jacobfeuerborn]

On 27.02.2015 06:58, Han Changzhe wrote:
> Thanks Dave! Please see below.
> 
> Changzhe
> 
> On 2015-02-26 15:30, Dave Taht wrote:
>> This is not really the best place for a routing question. If you raise
>> your question on the quagga list, you might be better off.
>>
> 
> I couldn't find a better place for the question and thought guys in LARTC
> should be professional enough to help. And thanks for your suggestion on
> the quagga list! I'll try it.
> 
>>
>> On Wed, Feb 25, 2015 at 10:39 PM, Han Changzhe <hcz@nebulat.com> wrote:
>>> Hello experts,
>>>
>>> I'm setting up a routing server on Linux with following links
>>>
>>> 1. An Ethernet link (eth0) to the 1st internet link (fast, but can't
>>>     access some sites);
>>> 2. A VPN link (tun0) to provide services to local users;
>>> 3. A VPN link (tun1) to a proxy server as the 2nd internet link (slow,
>>>     free).
>>>
>>> My target is:
>>>    * for common internet access, routing the packets through eth0;
>>>    * for the sites can't be accessed through eth0, routing them
>>> through tun1.
>> Well, one of the things we have been working on in the homenet working
>> group is
>> source specific routing, which could possibly help here, but it is
>> non-deterministic.
> 
> As for the project you mentioned, are there any public materials for me
> to follow up?
> 
> 
>>> By now, I set the routing table manually for serveral sites and it works
>>> fine. Because there are thousands of them and the sites change with
>>> time, so
>>> I want a better solution.
>>>
>>> My idea is like this: setting up more than one default routes for
>>> internet
>>> access, then dynamically change the route table (or route table
>>> cache) with
>>> some software according to the internet access results.
>>>
>>> For example, if we get a timeout from https://www.google.com through
>>> eth0,
>>> the software should try it through tun1 link and, when succeed,
>>> adding the
>>> later route to current route table.
>> Well you are conflating several layers of the protocol here.
>>
>> It is hard to recognise a timeout, for example, without sniffing for
>> syns/syn_acks
>> on the gateway. That sniffer could simultaneously try a syn out one of
>> the vpn interfaces and if a syn/ack is not received from the main
>> interface, and one IS received from the vpn, insert a route for it.
> Yes, a sniffer like that should work. Is it possible to allow the kernel
> to raise an exception when TCP connection time-out happens so users can
> handle the exception with the hock to try the vpn interface and manipulate
> the route table?

You could potentially use the -E option of the conntrack-tools to listen
to these kinds of events:
http://conntrack-tools.netfilter.org/conntrack.html

Regards,
  Dennis

Re: A smart router for more than one default routes

[Dave Taht]

On Thu, Feb 26, 2015 at 9:58 PM, Han Changzhe <hcz@nebulat.com> wrote:
> Thanks Dave! Please see below.
>
> Changzhe
>
> On 2015-02-26 15:30, Dave Taht wrote:
>>
>> This is not really the best place for a routing question. If you raise
>> your question on the quagga list, you might be better off.
>>
>
> I couldn't find a better place for the question and thought guys in LARTC
> should be professional enough to help. And thanks for your suggestion on
> the quagga list! I'll try it.
>
>>
>> On Wed, Feb 25, 2015 at 10:39 PM, Han Changzhe <hcz@nebulat.com> wrote:
>>>
>>> Hello experts,
>>>
>>> I'm setting up a routing server on Linux with following links
>>>
>>> 1. An Ethernet link (eth0) to the 1st internet link (fast, but can't
>>>     access some sites);
>>> 2. A VPN link (tun0) to provide services to local users;
>>> 3. A VPN link (tun1) to a proxy server as the 2nd internet link (slow,
>>>     free).
>>>
>>> My target is:
>>>    * for common internet access, routing the packets through eth0;
>>>    * for the sites can't be accessed through eth0, routing them through
>>> tun1.
>>
>> Well, one of the things we have been working on in the homenet working
>> group is
>> source specific routing, which could possibly help here, but it is
>> non-deterministic.
>
>
> As for the project you mentioned, are there any public materials for me to
> follow up?

It is one of the products of the ietf homenet working group and mostly
oriented around IPv6, but can be used with ipv4.

https://plus.google.com/107942175615993706558/posts/PtnPVES2KRd

The one, published, RFC

https://tools.ietf.org/html/rfc7368

More stuff from homenet that is in progress:

https://tools.ietf.org/wg/homenet/

The idea behind source specific routing is, you have multiple uplinks
to the internet with distinct IP addresses and delegated subnets, and
hosts get more than one IP address - so that when a host sources a
query from a different IP address it goes out via a different
interface on the router(s). This works well with what is known as
"happy eyeballs" - where a browser tries multiple addresses to get out
to the web site on the internet - helps on failover, etc.

There is some working code - already in openwrt - and available as
open source in mainline linux - but it could use some love.
"babels" is quite mature - (referenced above) - and hnetd is coming
along but not quite baked yet.

A lovely use for the idea - that would work in your case were it in
more protocols, is in multipath mosh. That version of mosh attempts
connections via every way possible, and measures the path
with the shortest RTT and uses that. So, in your case, it would
probably choose the non-vpn path if not blocked, and the vpn path if
it was.

https://github.com/boutier/mosh

Incidentally most of my vpns are routed over ipv6, over tinc. IPv4 got
to be a PITA, now that so many random 10 networks exist in
double/triple natted situations.

>
>>> By now, I set the routing table manually for serveral sites and it works
>>> fine. Because there are thousands of them and the sites change with time,
>>> so
>>> I want a better solution.
>>>
>>> My idea is like this: setting up more than one default routes for
>>> internet
>>> access, then dynamically change the route table (or route table cache)
>>> with
>>> some software according to the internet access results.
>>>
>>> For example, if we get a timeout from https://www.google.com through
>>> eth0,
>>> the software should try it through tun1 link and, when succeed, adding
>>> the
>>> later route to current route table.
>>
>> Well you are conflating several layers of the protocol here.
>>
>> It is hard to recognise a timeout, for example, without sniffing for
>> syns/syn_acks
>> on the gateway. That sniffer could simultaneously try a syn out one of
>> the vpn interfaces and if a syn/ack is not received from the main
>> interface, and one IS received from the vpn, insert a route for it.
>
> Yes, a sniffer like that should work. Is it possible to allow the kernel
> to raise an exception when TCP connection time-out happens so users can
> handle the exception with the hock to try the vpn interface and manipulate
> the route table?
>
>> You would still need to clean out that table periodically.
>
> Yes. With the old Linux routing cache, it might be easier. Then we only
> add routes to the cache.
>
>> Then you would to insert and delete rules for each ip (or more likely
>> network) you wish to reroute
>> based on your measurements of what is working or not, and to otherwise
>> fall back to the default ethernet route.
>>
>> Say for example you could not get dns from 8.8.8.8 locally.
>>
>> ip route add 8.8.8.8 dev tun0
>>
>> This doesnt help you on any protocols except tcp. udp apps are
>> different. so is quic, etc.
>>
>> a bulk method would be to go through the alexa top 1 million to see
>> what you could and could not access, and set up routes for each (but
>> this does not handle your desire for 2 tunnels)
>
> Actually I prepared a routing list of around 3K items which should
> work by now. It's just not beautiful.
>
>
>>> I don't know if any routing software on Linux work as I expected. I tried
>>> quagga with zebra + ospf but not successful.
>>
>> ospf? oy, no....
>>
>>> FYI, it's not a common case for link based fail-over/load balance.
>>>
>>>
>>> Please give me suggestions!
>>
>> Well, my way would probably involve a squid or polipo web proxy to
>> make the failover case easier. A lot of users would not dig that...
>
> It should work for web accessing. But we still need to support ports
> for email/ftp and other applications.
>
>
>>> Thanks in advance,
>>>
>>> Changzhe
>>>
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe lartc" in
>>> the body of a message to majordomo@vger.kernel.org
>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>
>>
>>
>
> --
> To unsubscribe from this list: send the line "unsubscribe lartc" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html



-- 
Dave Täht
Let's make wifi fast, less jittery and reliable again!

https://plus.google.com/u/0/107942175615993706558/posts/TVX3o84jjmb

Re: A smart router for more than one default routes

[Han Changzhe]

On 2015-02-28 20:53, Dennis Jacobfeuerborn wrote:
> On 27.02.2015 06:58, Han Changzhe wrote:
>> Thanks Dave! Please see below.
>>
>> Changzhe
>>
>> On 2015-02-26 15:30, Dave Taht wrote:
>>> This is not really the best place for a routing question. If you raise
>>> your question on the quagga list, you might be better off.
>>>
>> I couldn't find a better place for the question and thought guys in LARTC
>> should be professional enough to help. And thanks for your suggestion on
>> the quagga list! I'll try it.
>>
>>> On Wed, Feb 25, 2015 at 10:39 PM, Han Changzhe <hcz@nebulat.com> wrote:
>>>> Hello experts,
>>>>
>>>> I'm setting up a routing server on Linux with following links
>>>>
>>>> 1. An Ethernet link (eth0) to the 1st internet link (fast, but can't
>>>>      access some sites);
>>>> 2. A VPN link (tun0) to provide services to local users;
>>>> 3. A VPN link (tun1) to a proxy server as the 2nd internet link (slow,
>>>>      free).
>>>>
>>>> My target is:
>>>>     * for common internet access, routing the packets through eth0;
>>>>     * for the sites can't be accessed through eth0, routing them
>>>> through tun1.
>>> Well, one of the things we have been working on in the homenet working
>>> group is
>>> source specific routing, which could possibly help here, but it is
>>> non-deterministic.
>> As for the project you mentioned, are there any public materials for me
>> to follow up?
>>
>>
>>>> By now, I set the routing table manually for serveral sites and it works
>>>> fine. Because there are thousands of them and the sites change with
>>>> time, so
>>>> I want a better solution.
>>>>
>>>> My idea is like this: setting up more than one default routes for
>>>> internet
>>>> access, then dynamically change the route table (or route table
>>>> cache) with
>>>> some software according to the internet access results.
>>>>
>>>> For example, if we get a timeout from https://www.google.com through
>>>> eth0,
>>>> the software should try it through tun1 link and, when succeed,
>>>> adding the
>>>> later route to current route table.
>>> Well you are conflating several layers of the protocol here.
>>>
>>> It is hard to recognise a timeout, for example, without sniffing for
>>> syns/syn_acks
>>> on the gateway. That sniffer could simultaneously try a syn out one of
>>> the vpn interfaces and if a syn/ack is not received from the main
>>> interface, and one IS received from the vpn, insert a route for it.
>> Yes, a sniffer like that should work. Is it possible to allow the kernel
>> to raise an exception when TCP connection time-out happens so users can
>> handle the exception with the hock to try the vpn interface and manipulate
>> the route table?
> You could potentially use the -E option of the conntrack-tools to listen
> to these kinds of events:
> http://conntrack-tools.netfilter.org/conntrack.html
>
> Regards,
>    Dennis
>
Thanks Dennis! It seems a good tool. I'll try it!

Regards,

Changzhe