[LARTC] Solved: Using more than 1 Internet Line

* [LARTC] Solved: Using more than 1 Internet Line
@ 2001-12-02 19:24 Christoph Simon
  2001-12-02 19:41 ` Christoph Simon
                   ` (13 more replies)
  0 siblings, 14 replies; 15+ messages in thread
From: Christoph Simon @ 2001-12-02 19:24 UTC (permalink / raw)
  To: lartc

[-- Attachment #1: Type: text/plain, Size: 722 bytes --]

I've been looking for about a year to find a way to use more than one
independent link to the Internet from a single computer. But I only
found many others who tried to do the same, without success.

Then I found out about Julian Anastasov's patches, and well, now it's
working. Unfortunately, this is not a five minutes solution, nor will
it help in all kinds of situations. So I decided to write a document
which tells step by step how this can be done and how it works, for
which I received lots of support from Julian. Thanks.

I've Attached the text to this message. I hope it is useful to
others. Here it is already working.

--
Christoph Simon
ciccio@kiosknet.com.br
---
^X^C
q
quit
:q
^C
end
x
exit
ZZ
^D
?
help
.

[-- Attachment #2: nano.txt --]
[-- Type: text/plain, Size: 34350 bytes --]

Nano-Howto to use more than one independent Internet connection.
----------------------------------------------------------------

1. Intro
========

1.0 Thanks
----------

The solution presented here is based on the patches by Julian
Anastasov. Like several other users on the net struggling with this, I
believe they fill in yet another gap for using Linux as a very
advanced router. It was hard for me not only to get this running for
the first time, but also to understand most of the principles. Also in
this Julian has proofed to have unlimited patience. Thanks.

He also agreed to expose this file on his site:

	http://www.linuxvirtualserver.org/~julian/nano.txt

1.1 Initial situation
----------------------

I've been looking for a solution to use more than one connection to
Internet at the same time for a long time. I've found out, that I'm
not alone; there seem to be many having this very problem. The reasons
are generally just to cheaply increase total throughput, or to provide
for a fail-over in case of cheap lines. This is not the situation of a
cluster of web-servers, where main traffic is input, but for networks
which are mainly dedicated to any kind of download.

1.2. Bad news
-------------

The setup of all this is not a question of 5 minutes, so I'll tell at
the beginning where are the limits.

First, this will not work for a single computer with two modem
connections. Load balancing does work reasonably well, but only with a
high number of simultaneous connections to different remote sites.

The tests I'm performing are on a network with 17 client computers
with an daily download (12h) of some 600MB of HTTP traffic in several
hundreds of thousands of connections.

When one connection fails while being used, this connection will be
lost and can't be continued on another line. The user will have to
establish this connection anew. If it was a big download and the
server doesn't support continuation, he'll be out of luck.

The lines I'm using have external modems which react as any other
device with Ethernet. There is no PPP, PPPoE, etc. I can't tell if
this works or would need modification for instance to use two
telephone lines. Maybe someone else can fill in this.

1.3 Good news
-------------

What does work is:

- there is no restriction to 2 lines, though I didn't test more.
- using 2.4 kernels
- interaction with ip(8) and iptables(8)
- usage of both lines at the same time
- if any line fails, new connections will use any other which working.
- if a line fails, no reconfiguration of the network is required.

Right now, I can't tell how long it takes in case of a fail over, but
there where several cases I found in the logs and no user realized
that there was a problem.

2. Setup
========

2.1. Preparing the kernel
-------------------------

Stock kernels are not able to provide this functionality. Julian
Anastasov wrote a patch for 2.2 and 2.4 Linux kernels. I only tried
with several 2.4 kernels, now running 2.4.15pre5. The patches have not
been written for this version, but apply cleanly. The patches are
available on Julian's web-page:

	http://www.linuxvirtualserver.org/~julian/#routes

Choose the patches for your kernel, download them, and apply all of
them.

The patches will not offer any additional configuration options.  The
kernel needs to have "equal cost multi path" enabled. This is done in
Network Options. After choosing "IP: advanced router" there will be an
option for it. Only an indirect requirement is, that we will need NAT:
Any host on the local network needs to be able to appear on the
Internet with any of the external IPs; this is the main purpose of
NAT. To get NAT, of course we have to enable connection tracking. But
this would be done anyway. No further options are required, but any
networking option may be used. Particularly, I've enabled almost all
options for netfilter and QoS and it's working fine.

Nothing special is required to run this kernel (no boot options,
etc.).

2.2 Netfilter issues
--------------------

As I stated at the beginning, to make use of the patches and the
configuration described here, we need a relatively high number of
connections and remote servers. I've tried bitterly, but wasn't able
to achieve this effect alone. I started several simultaneous
downloads, ran Mozilla on several computers at the same time, opening
different big sites, but no avail. When the regular users came, things
suddenly showed to work correctly. So this is certainly not a setup
for a single box, and then, we'll need to do some address
translation. Beside this, a firewall will almost always be
unavoidable, but it's setup is out of scope here. A minimal setup
would enable forwarding in the kernel, establish a default policy of
ALLOW, and create at least two SNAT rules:

	iptables -t nat -A POSTROUTING -o IFE1 -s NWE1/NME1 -j SNAT --to IPE1
	iptables -t nat -A POSTROUTING -o IFE2 -s NWE2/NME2 -j SNAT --to IPE2

If the ISP didn't give us a fixed IP, for instance using PPPoE, but it
seems that it must be an ARP type device, we would say:

	iptables -t nat -A POSTROUTING -o IFE1 -s NWE1/NME1 -j MASQUERADE
	iptables -t nat -A POSTROUTING -o IFE2 -s NWE2/NME2 -j MASQUERADE

I didn't try this, but it will work if the analog thing works with
just one line. This is not really new, as we would have to do
something similar anyway, even with only one line to the Internet.

At least for netfilter (not sure for ipfwadm/ipchains), the firewall
must be stateful. This can be done by:

	iptables -t filter -N keep_state
	iptables -t filter -A keep_state -m state --state RELATED,ESTABLISHED \
		-j ACCEPT
	iptables -t filter -A keep_state -j RETURN

	iptables -t nat -N keep_state
	iptables -t nat -A keep_state -m state --state RELATED,ESTABLISHED \
		-j ACCEPT
	iptables -t nat -A keep_state -j RETURN

and calling this at the beginning of the script:

	iptables -t nat -A PREROUTING -j keep_state
	iptables -t nat -A POSTROUTING -j keep_state
	iptables -t nat -A OUTPUT -j keep_state
	iptables -t filter -A INPUT -j keep_state
	iptables -t filter -A FORWARD -j keep_state
	iptables -t filter -A OUTPUT -j keep_state

2.3 Routing
-----------

Here I'll assume that the computer has three network cards: one to the
local network, and two for each of the external links (DSL modems in
my case). This needs not to be so, as there could be more DSL modems,
and, at least in theory, it's possible to use one card for more than
one modems, using a hub and two or more IPs on the same device. This
is not a recommended setup because it may introduce security problems,
but might be the only one if we had more lines than PCI slots for
additional network cards.

Abbreviations:
IFI		internal interface
IPI		IP address of internal interface
NMI		netmask for the internal interface
IFE1, IFE2	external interfaces
IPE1, IPE2	external IP address
NWE1, NWE2	external network address
NME1, NME2	mask for the external network (number of bits, like in /24)
BRD1, BRD2	broadcast address for external network
GWE1, GWE2	gateway for external interface

2.3.1 The local loopback and the internal interface
---------------------------------------------------

The local loopback must be up and running. There is nothing unusual
about this.

	ip link set lo up
	ip addr add 127.0.0.1/8 brd + dev lo

At this point, we should be able to ping localhost.

We also set up the internal interface IFI with the internal address
IPI and the netmask for the local network NMI. Assigning an address
will cause the kernel to automatically insert an appropriate route
into table main. This is just fine. Later, there will also be a route
for each external interface, but none of the routes in table main will
have a gateway.

We'll want to give table main a priority of 50 to make sure it is
looked at first.

But also we want to make sure that there is no default route in table
main. The commands so far didn't insert any, but if this is a
reconfiguration, there might remain some from the previous setup. If
there is none, the last command will fail, but that's OK: we just want
to be sure.

	ip link set IFI up
	ip addr add IPI/NMI brd + dev IFI
	ip rule add prio 50 table main
	ip route del default table main

At this point, we should be able to ping any host on the local
network.

2.3.2 Setup of external interfaces
----------------------------------

The configuration of the external interfaces isn't really different
from that of the internal, but for now, we don't specify a gateway nor
do we insert a default route.

	ip link set IFE1 up
	ip addr flush dev IFE1
	ip addr add IPE1/NME1 brd BRD1 dev IFE1

	ip link set IFE2 up
	ip addr flush dev IFE2
	ip addr add IPE2/NME2 brd BRD2 dev IFE2

As ip(8) allows to assign more than one address to an interface, it's
important that there doesn't remain any configuration from a previous
setup; failing to do so might cause the same address to be specified
twice. That's what the "addr flush" commands are for. When we actually
specify the address (the last line of each block), the kernel, again,
will insert automatically a route into table main, very much like in
the case of the internal interface.

Now, we should be able to ping the external IPs (IPE1 and IPE2), as
well as the gateways or any other host or device which belongs to
those external network addresses. In my case there is none beside the
gateways. For this reason it would be correct to consider such an
interface like a point-to-point device, but technically it is not.

2.3.3 Setup of the default routes
---------------------------------

This is the complicated part of the setup. During this discussion,
I'll mention the setup commands step by step. In the end, I'll repeat
them all together.

Before really starting, let's make a small experiment: Using any kind
of packet tracer (logs from iptables, tcpdump, etc), we will observe
that if we ping 127.0.0.1, the source address of that packet will be
127.0.0.1; if we ping any address of the local network, the source
address will be that of the internal interface. And if we ping any
host or device outside, the source address will be that of the device
where the packet is going out. Who is being that smart?  It might be
the application (e.g., ping) itself, using a bind(2) call, but this is
not normally done for a client. It's the routing system. In other
words, until a packet actually reaches the routing system, it doesn't
have a source address.

Well, in practice, some programs try to be smarter than the routing
system, use the bind(2) call, but actually choose a less than optimal
route. This would easily be the case under our setup, as a program
can't know which of the two external interfaces it should use. Also,
there are some versions of ping which allow the user to specify the
source address. While being an interesting tool also in our setup, the
`being smart' is then really up to the user.

As I stated in the requirements, NAT is one of them and so is
connection tracking. This consists of a set of information for each
known connection, including specific NAT details.

We see, even if the packet originates on the same computer, in our
case, a packet can have one of four addresses: 127.0.0.1, IPI, IPE1,
or IPE2. This is called the local source address, which plays a major
role when doing NAT. Here, this was one of the key concepts I had to
understand to be able to understand the rest.

I'll try to `deduce' the routing commands by following a packet from a
host on the local network. While it might also originate from the
computer we are configuring, this will be the most typical path. I'll
call that host w1 (the first workstation).

The setup of w1 is as usual: It has a loopback interface, a network
card with an IP address compatible with the local network address, and
a default route which uses as a gateway the computer we are
configuring here (IPI). As soon as the user on w1 does anything which
requires access to Internet, her program will set up a socket and call
connect(2). Then, an IP packet will be formed. The routing system on
w1 will set the source address to w1's local IP, the destination
address to that of the remote server on the Internet.  It will use the
default route to send this packet to us.

The packet will come in by the local interface IFI, being welcome, if
present, by the prerouting chain of the mangle table, but certainly by
the prerouting chain of the nat table.

The nat table will search the already existing connection structures
and find out, that this is a new connection. It will create an
uninitialized info-block, and pass the packet to the routing system.

At this time, we have the following information: The packet came in by
the local interface IFI, has the source address of w1, but we don't
know yet the local source address.

Being a new connection, there will be nothing matching in the
cache. Also, our configuration so far doesn't have any default route
yet. So we need one which tells what to do with such a packet:

	ip rule add prio 222 table 222
	ip route add default table 222 proto static \
		nexthop via GWE1 dev IFE1 \
		nexthop via GWE2 dev IFE2

This is a multipath default route, causing the kernel to extract each
time another alternative; there could be more than these two.

Now, we get two additional informations: the output device (either
IFE1 or IFE2) and the IP address of the gateway (either GWE1 or
GWE2). Actually we also may get a third information, which is our
local IP address, which is not contained in this route, but the system
will deduce it by interface and gateway if it is not a dynamic
IP. That's it, routing is doing for us right now, beside sending the
packet to the forward chain of the filter table.

Of course, forwarding must be enabled in the kernel, and there must
not be any netfilter rule disallowing this packet. After that, the
packet is passed along to the postrouting chain of the nat table.

Using the information about the packet we gathered so far, the NAT
system will look for a rule, and find one of the two we gave in the
previous section. If our IP is fixed, now the local source address
will became known, but if it is dynamic, i.e., we used the MASQUERADE
target, we will need to go back to routing, as there is no other way
to find out the right local source address. The answer will come from
table 222, the multipath route, but this time, also interface and
gateway need to match. The NAT system will insert this into the, now
initialized, NAT information in the connection structure, replace the
source address in the packet by this one (which was still the IP of
w1), and get this packet on it's way to the gateway, going out by the
proper output device.

Note here, that when asking the multipath route, table 222, with a
known output device and gateway, the kernel still has to deduce the IP
address. It will do so, looking at the first address attached to the
interface which, by netmask, is compatible with the gateway
address. If there were more than one possible answers, only the first
will be used. This might be the case if we have both lines on the same
network card, which will work only if we use the MASQUERADE target in
the iptables rules. If we didn't use the gateway (stock kernels don't)
to make this match, we surely would be in troubles, because the output
device is the same for both. This means: To be able to use more than
one Internet connection on one and the same network card, the gateways
must be different, the IP addresses must be public IPs and must not
belong to the same subnet. Additionally, we should avoid SNAT in the
iptables rules and take special measures to protect against IP
spoofing.

The gateway, and potentially lots of routers in the Internet, are
supposed to do the right thing to make this packet reach it's
destination. There, the answer packet will use the source address
(which is now either IPE1 or IPE2) to come back into our computer by
the same interface where it left. Essentially, the procedure is
reversed; our routing table will find the way to w1 already in table
main.

Most user actions on the Internet aren't happy with just one packet of
data; let's have a look what happens if a second packet is sent from
w1 to the same destination:

Again, the packet will arrive at the local interface of this computer
and pass through the prerouting chain of the mangle table if that
exists. The prerouting chain of the nat table will start looking for
an existing, matching connection, and this time, it will find one. As
there is no DNAT rule in the netfilter setup for this packet, NAT will
just hand over the packet to the routing system.

The routing system will look at the connection, and the patches will
find that there is NAT information present, and that after routing
some mangling will be done. Now, the routing question will give the
input device (IFI), the source address (the IP of w1), the destination
address (that of the server on the Internet), and the local source
address (either IPE1 or IPE2). In this situation, we do not want to
match the multipath route, because the multipath route doesn't look at
local source address, we couldn't make sure that we get this time the
same line as before. So we add a table for each interface:

	ip rule add prio 201 from NWE1/NME1 table 201
	ip route add default via GWE1 dev IFE1 src IPE1 proto static table 201
	ip route append prohibit default table 201 metric 1 proto static

	ip rule add prio 202 from NWE2/NME2 table 202
	ip route add default via GWE2 dev IFE2 src IPE2 proto static table 202
	ip route append prohibit default table 202 metric 1 proto static

This can't be matched when table 222 should be matched, as we now
require the local source address to be known. We choose a priority
that makes sure these routes are found after the main table and before
table 222, which is more general than these.

These are still default route, because they must match any Internet
address.

The third line of each block is similar to a REJECT target in iptables
in case the corresponding interface is not working: If the client on
the local network sends a packet on an established connection, but in
the meanwhile the interface stopped operating, we will send this
client an ICMP controll message `PKT_FILTERED', hoping to cause it to
stop sending packets, and the user might wish to open a new
connection, which will succeed if at least one other line is still
working.

Stepping through all routing tables, either table 201 or table 202
will match, according to the output device selected when we matched
the multipath route for the first packet. The tricky part is, what the
patches are doing here: They look at the local source address. If it
is zero, the old behavior would just match the multipath route in
table 222, but as we do know already the local source address (i.e.,
it is not zero) from what we found in the connection structure,
particularly what concerns NAT, we will try to match the rules against
the local source address rather than the packet's source address,
which is still the IP of w1. This is why either table 201 or table 202
will provide the match, telling the gateway address and the output
device.

Having done it's job, the routing system will send the packet to the
forward chain of the filter table, to continue in the postrouting
chain of the nat table. That has an easy game now, because, no matter
if the found target is MASQUERADE or SNAT, all necessary information
is already known. The packet's source address will be replaced by our
local source address and the packet can continue it's way going out by
the chosen output device, and reach it's destination through the
Internet. The same procedure is repeated for all further packets,
because the connection information is kept alive during a certain
time, even after closing the connection (FIN or RST).

Now let's have a look at a packet which originates on this computer
with a destination on the Internet. As we found out in our experiment,
the source address is not known before routing, and we won't need any
NAT. Here we will assume that the program which generates this packet
does not use the bind() call. If it would, either table 201 or table
202, according to the IP this program binds to, would be used and no
(further) load balancing can take place. So, the packet will start
crossing the output chain of the mangle table, if that is present, and
go to the output chain of nat. As there is no rule for a destination
nat, it will continue being evaluated in the routing system. No local
source address is known yet, so the multipath route in table 222 will
match. This will provide the output device, the gateway and also the
local source IP. Now the connection information can be completed. With
that, the packet will cross the output chain of the filter table and
reach the postrouting chain of the nat table. From there the packet
can be sent to the gateway. From now on, the connection is known and
all further packets will use this path.

Now the full sequence of commands, all together. For convenience, I'll
repeat the meaning of the abbreviations:

Abbreviations:
IFI		internal interface
IPI		IP address of internal interface
NMI		netmask for the internal interface
IFE1, IFE2	external interfaces
IPE1, IPE2	external IP address
NWE1, NWE2	external network address
NME1, NME2	mask for the external network (number of bits, like in /24)
BRD1, BRD2	broadcast address for external network
GWE1, GWE2	gateway for external interface

	ip link set IFI up
	ip addr add IPI/NMI brd + dev IFI
	ip rule add prio 50 table main
	ip route del default table main

	ip link set IFE1 up
	ip addr flush dev IFE1
	ip addr add IPE1/NME1 brd BRD1 dev IFE1

	ip link set IFE2 up
	ip addr flush dev IFE2
	ip addr add IPE2/NME2 brd BRD2 dev IFE2

	ip rule add prio 201 from NWE1/NME1 table 201
	ip route add default via GWE1 dev IFE1 src IPE1 proto static table 201
	ip route append prohibit default table 201 metric 1 proto static

	ip rule add prio 202 from NWE2/NME2 table 202
	ip route add default via GWE2 dev IFE2 src IPE2 proto static table 202
	ip route append prohibit default table 202 metric 1 proto static

	ip rule add prio 222 table 222
	ip route add default table 222 proto static \
		nexthop via GWE1 dev IFE1 \
		nexthop via GWE2 dev IFE2

Let's check it out:

	ip address

This should print on the terminal one entry for the local loopback,
IFI, IFE1 and IFE2, and maybe some other things, if we have it
configured (like my GRE tunnels).

	ip rule

This should look like this:

	0:      from all lookup local
	50:     from all lookup main
	201:    from NWE1/NME1 lookup 201
	202:    from NWE2/NME2 lookup 202
	222:    from all lookup 222
	32766:  from all lookup main
	32767:  from all lookup default

Table local is used for the local loopback, table main has the network
routes to the internal network and for the external interfaces, which
only give access to our gateways. Tables 201 and 202 (which also might
have the same priority), will provide a default route if the local
source address is known (because they have to match NWE1 or NWE2). And
table 222 will provide the multipath route. The tables with priority
32766 and 32767 will not be used.

	  ip route list table main

Giving:

	NWI/NMI    dev IFI   proto kernel  scope link  src IPI
	NWE1/NME1  dev IFE1  proto kernel  scope link  src IPE1
	NWE2/NME2  dev IFE2  proto kernel  scope link  src IPE2

These are only routes to the corresponding networks without using a
gateway.

	ip route list table 201

Giving:

	default via GWE1 dev IFE1  proto static  src IPE1
	prohibit default  proto static  metric 1

And:

	ip route list table 201

Giving:

	default via GWE1 dev IFE1  proto static  src IPE1
	prohibit default  proto static  metric 1

These are the default routes requiring the local source address to be
known. And finally:

	ip route list table 222

Giving:

	default  proto static
	        nexthop via GWE1  dev IFE1 weight 1
	        nexthop via GWE2  dev IFE2 weight 1

which of course is our multipath route. The weights where inserted by
the kernel and could be changed. Actually, they need to be changed
specially if the two lines have a very different capacity. With this
configuration, both lines are going to be used the same amount of
times (at least in the long run), even in case the first line has 4
times the capacity of the second. To compensate for that, we would add
"weight 4" after IFE1, like in:

	ip route add default table 222 proto static \
		nexthop via GWE1 dev IFE1 weight 4\
		nexthop via GWE2 dev IFE2 weight 1

The numbers should be as small as possible, because in this case, the
kernel will create 5 alternative routes, four of them being the
same. For this reason, it wouldn't really be an equivalent to say
"weight 400" and "weight 100", nor would it be a good idea.

Now, let's see how the kernel chooses certain routes:

Asking, how to reach the gateway:

	ip route get GW1

We get the route from table main:

	GW1 dev IFE1  src IPE1
	    cache  mtu 1500

Now, let's ask for the route to a host on the internet. Note, that the
ip(8) command does no DNS lookups; we need to provide the IP. This one
goes to www.kernel.org:

	ip route get 204.152.189.113

My system answered:

	204.152.189.113 via GWE1 dev IFE1  src IPE1
	    cache  mtu 1500

i.e., having chosen the first interface. As it got this from the
cache, it won't be easy to get the answer for the second interface
until the cache expires, or we flush it explicitly. If not found in
the cache, this answer must come from table 222. To test tables 201
and 202, we can use:

	ip route get from IPE1 to 204.152.189.113
	ip route get from IPE2 to 204.152.189.113

and would get respectively:

	204.152.189.113 from IPE1 via GWE1 dev IFE1
	    cache  mtu 1500
	204.152.189.113 from IPE2 via GWE2 dev IFE2
	    cache  mtu 1500

Up to here, we know how all external lines are being used while they
are working. It's also easy to guess what happens if all external
lines fail. But let's see what happens if only one external line
fails:

At a first glance, nothing special happens. If one of the routes in
the multipath statement is unavailable, it just stops being eligible
and we get always the other one (one of the others, if more than
two). But it is less evident, what happen during the transition.

When we try to send a packet to another device, and that fails, like
in real live, we'll blame the other side. But if we can't reach one
neighbor, it doesn't mean necessarily that we can't reach any
neighbor. This depends on the reason for the failure. If the problem
is in our network card or in the cable attached to it, we won't reach
any other, but it's also possible that only this remote device is
down. Well, in our particular case, this is the same, because, from
the functional point of view, this is a point to point connection, and
beside the gateway and ourselves, there is no other device. But the
lower level transport system can't know that and hence will mark this
link just as suspected to fail. Neither the routing cache nor the
connection information have yet knowledge of the problem, and the
user's connection will block.

After a certain time, Ethernet will consider the gateway dead and mark
it as unreachable. If this is because the link was brought down, the
routing cache will be flushed immediately, forcing the next packet to
choose a new route, which will get only a working one. Otherwise, the
packet could use a route in the cache which is actually unreachable,
causing it to fail: All connections which are in the cache and which
use this route will just go nowhere. After a certain interval of time,
the entries in the routing cache expire, and than the route have to be
looked for again, offering only working routes, and things will work
again.

From the user's point of view: An established connection using a line
which stops working will be lost. She'll need to establish a new
connection. Until the cache expires, a new connection will work only
if the route isn't yet in the cache, or if the cached route uses a
working line. But if the route is cached and uses the failing line,
the connection will not work.

The time until the cache expires is controlled by the kernel variable
/proc/sys/net/ipv4/route/gc_interval and defaults to 60 seconds. The
bigger this number, the longer it will take until failing routes are
not being used again, but the shorter this time, the smaller the
chances to find a connection in the cache, loosing time to look up the
connection in the routing tables.

The inverse is analog: If the line comes back up, cached routes won't
use it. But routes found by matching the multipath route in table 222
will consider this route as eligible again, and possibly use it. But
more on this in the next section.

2.4 Keeping them alive
----------------------

We're almost done. There is a subtle issue about the kernel detecting
the state of an interface. We can not find out whether an interface is
working without just trying to use it. As long as the kernel doesn't
use the interface, it can't know if it is working or not. When we try
to use it, the kernel will mark the interface according to the result
of that operation. If this means, the interface is down, and we
continue trying to access Internet, the kernel will simply use the
other interface. How then can the kernel find out, that the failing
interface is working again? Well, it can't, because it wouldn't even
try. This is good, because users we wouldn't like to have to wait for
a timeout with each packet. But then, there needs to be a way to
notice when the interface comes back up. The only way is to use the
interface explicitly with non critical packets. Before using these
patches I had written a daemon which would set up explicit routes to
each gateway and send pings in regular intervals (once per
minute). Then the default route was set either to the preferred
interface or to the only one working. Now, this daemon doesn't need to
change routes anymore, but it still has to send the pings.

But there isn't really a daemon needed for this. A simple script:

	while : ; do
	  ping -c 1 GWE1 > /dev/null 2>&1
	  ping -c 1 GWE2 > /dev/null 2>&1
	  sleep 60
	done
can do this.

At the beginning, I was worried because I've a link where the provider
doesn't reply pings. But this is really not necessary. The trick is,
that ARP requests must be answered for anything to work.  One
consequence of this is, that for now I'm only sure that this works for
devices which are based on Ethernet, i.e., which use the ARP. Other
devices, like dialup links, can only work if the underlying network
protocol provides a correct status change. But I'm not aware if or
which Linux protocols are implemented this way.

What I've learned from this is, that it's not enough to just set a
device up or down, or to add or remove a route. Somehow, the kernel
must also be told, that this device or route is working or stopped
doing so.

Running the pings once every minute all the time isn't what seems to
be an elegant solution, but it works. Hopefully, someday we'll get
this done behind the scene.

During development of this setup, I've considered using arping rather
than ICMP echo replies. Both can be problematic. The gateway can drop
echo messages, so ping wouldn't work. On the other hand, I've noticed
in two situations that the provider apparently had installed some kind
of load balancing making the MAC address of the gateway toggling from
one to another. This renders arping useless. So I stayed with
ping. This will work anyway, because my daemon doesn't need to know
the result: Even if I don't get an ICMP echo reply, the kernel will
fill in the right MAC address for this gateway and know that it is
reachable or not.

It seems that newer versions of arping use a broadcast mode, which
would solve the problem with a provider that changes the MAC address
of the gateway every few minutes. I didn't try it yet, but I will.

3. Comments
3.2 About the patches
---------------------

Some of us are curious and wish to know what the Julian's patches
actually do. Here is a summary:

- With the patches enabled, we can use "proto static" for all
  routes. Doing so, the route for a link will survive even if that
  device stops working. This is essential, specially if the reason for
  using two lines is redundancy: If a line fails, all attributes of
  the device remain, beside the fact that it can't be used until it
  comes up again. We can continue pinging the gateway for to detect
  the moment the link comes up again. Equal cost multipath exists also
  without these patches, but wouldn't work in such a situation.

- If the device continues existing even if it doesn't work, the kernel
  needs a way to decide whether this is a route might be chosen,
  specially if several are available. This is called the ``dead
  gateway detection''; it works for any kind of unicast routes
  (multipath/unipath, input/output) and provided by these patches.

- A multipath route consists of several routes with the same source
  and destination, but a different path, which is specified by the
  device and the gateway. It is because of the patches that the kernel
  knows how to distinguish the route not only by source and
  destination, but also by device and gateway. Also, at least when
  each line has a different device, the kernel's reverse path checking
  (rp_filter) will work correctly.

- The NAT system has no idea how to deal with alternative routes, this
  is something the routing system has to do. And one of the additions
  of these patches is make the routing system prepare the route of an
  individual packet for the nat system, such that it looks as if it
  where the only possible route.

3.1 Practical experience
-----------------------

It's very little time that I use this, so obviously I can't give any
long term results. But after the first few days of working, I can say
that load-balancing is working reasonably well. I could imagine a more
fine grained behavior which would benefit single users with two DSL
lines, but in my particular situation I've seen a difference on total
load of only 15%.

Fail-over happened already several times and went by without being
noticed by the users. In the time I'm using this, I didn't experience
any problem. It just works :-).

ciccio@kiosknet.com.br

^ permalink raw reply	[flat|nested] 15+ messages in thread