[LARTC] Can't use two links on a linux box - correction/addition

[Mike Fedyk mfedyk@matchmail.com]

<PRE>I've just subscribed to this list, and found this article.  It solved my problem
perfectly, but he described the solution I was trying as possible, which I've
found is not.

I have one machine running several services including firewall, masq, smtp,
pop3, http, etc.  This is the first Linux computer for the company.  I'm working
on diversifying their server setup later...

I was reading some old docs that policy routing didn't work on packets from
local processes.  I'm sure glad it was wrong. :)

Here's what my routes looked like:
10.0.0.0 dev eth0  scope link  src 10.0.0.2
63.194.293.210 dev eth1  scope link  src 63.194.293.219
10.1.1.1 dev eth2  scope link  src 10.1.1.2
10.0.0.0/24 dev eth0  scope link
127.0.0.0/8 dev lo  scope link
default
        nexthop via 63.194.293.210  dev eth1 weight 1
        nexthop via 10.1.1.1  dev eth2 weight 1

here's my ip rule:
0:      from all lookup local
32766:  from all lookup main
32767:  from all lookup default

The default.  Now here's my problem.  If I connect from the internet to one of
my tcp services on 10.1.1.2 (which is behind another masq-ing firewall from the
isp for dsl) I would get responses from 63.194.239.202.  As you can see, this is
not good, and kills any tcp traffic.

I added two tables, 40 and 50.
Each has a route to the ISP's gateway and a default route.

Here's the new ip rule:
0:      from all lookup local
32764:  from 10.1.1.2 lookup 50
32765:  from 63.194.293.219 lookup 40
32766:  from all lookup main
32767:  from all lookup default

Everything is fine now, thanks to Van's help.

I just don't want anyone else to waste their time on what I tried...

HTH

van Leeuwen Wrote:

I've quoted your entire e-mail, and my response is at the far bottom.

On Wed, 25 Oct 2000, Gustavo Gibson da Silva wrote:

&gt;<i> Hi,
</I>
&gt;<i> I have a linux box connected to two leased lines as follows:
</I>
&gt;<i>         _/\__/\_
</I>&gt;<i>        /        \     +--------+        +------------+
</I>&gt;<i>       ( Internet )----+ Router |        | SMTP server|
</I>&gt;<i>        \_  __  _/     +----+---+        +------+-----+
</I>&gt;<i>          \/  \/            | 200.200.200.254   |
</I>&gt;<i>    100.100.100.2\   -+-------+----- ... ---------+--
</I>&gt;<i>     100.100.100.1\   | 200.200.200.1
</I>&gt;<i>               +-+--+--+
</I>&gt;<i>               | Linux |
</I>&gt;<i>               +---+---+    _/\__/\_
</I>&gt;<i>       192.168.1.1 |       /        \
</I>&gt;<i>                  -+------( Intranet )
</I>&gt;<i>                           \_  __  _/
</I>&gt;<i>                             \/  \/
</I>
Nice set up. Not all that unfamiliar to me. ;)

&gt;<i> The router came first when there was a small intranet. Now the network
</I>&gt;<i> has grown very much and we were unable to upgrade our link speed because
</I>&gt;<i> the local telco is unable to provide a 128Kbps link. Then we decided to
</I>&gt;<i> install a linux box to act as a proxy server (squid) with a wanpipe card
</I>&gt;<i> and lease a frame-relay link. This new link would have a different IP
</I>&gt;<i> address and there would be two squids running. One would use the old
</I>&gt;<i> link and another the new one. In order to achieve that I first attached
</I>&gt;<i> the outgoing IP address for each squid to the propper network interface.
</I>&gt;<i> Then I followed Adv-Routing HowTo and tried the following:
</I>
&gt;<i> # ip rule add from 100.100.100.1 table 200
</I>&gt;<i> # ip route add default via 100.100.100.2 table 200
</I>&gt;<i> # ip route add default via 200.200.200.1 table main
</I>&gt;<i> # ip route flush cache
</I>
That looks too darn short.

&gt;<i> I tried some traceroutes with the -s option to check whether this was
</I>&gt;<i> working but apparently it was not. I also tried to create the default
</I>&gt;<i> route via 200.200.200.1 on table 253 but it behaved just the same way.
</I>&gt;<i> Then I created it on a custom table (201) and added a rule for it. Then
</I>&gt;<i> the network was unreachable.
</I>
&gt;<i> I tried the same things on kernel 2.2.14 (suse 6.4) and kernel 2.2.16
</I>&gt;<i> (on a modified redhat). I tried also on a machine with two network
</I>&gt;<i> cards. I also played with /proc/sys/net/ipv4/ip_forward.
</I>
&gt;<i> The ip command version was ss991023 (suse 6.4) and ss001007 on redhat.
</I>
&gt;<i> What am I doing wrong?
</I>
The routing. I have set up more than one linux router for the situation
you're describing.

Let me assume that you want to do equal cost loadbalancing on the
two uplinks to the internet. Let me also assume that the interfaces are
called eth1 and eth2 for the 100.100.100.1 and 200.200.200.1 links
respectively.

First, you want to set up interface-specific routing tables. These are going
to pretend there's only one single uplink to the internet.

   ip route add 200.200.200.0/24 dev eth2 table 200
   ip route add default via 200.200.200.254 dev eth2 table 200

   ip route add 100.100.100.2 dev eth1 table 100
   ip route add default via 100.100.100.1 dev eth1 table 100

Then you want to set up your ordinary routing. Routing to the gateways is
totally unambiguous, as the 'trouble' starts at the gateways (multiple paths to
get to hosts beyond them, after all). So, there's host- or network routes to
the gateways in the main table. Note the src hints, so that the kernel will
insert the correct IP address in packets with an unspecified source address
leaving through that interface. This means eventual replies will be sent back
to that interface.

   ip route add 100.100.100.2 src 100.100.100.1 dev eth1
   ip route add 200.200.200.0/24 src 200.200.200.1 dev eth2

Okay, the ordinary static routing is done. However, we don't have a default
route yet. Let's make that a multipath route. That is done as follows:

   ip route add default scope global nexthop via 100.100.100.2 \
                                     nexthop via 200.200.200.254

This is an equal cost multipath default route onto the internet, and it
should provide you with full connectivity and some load balancing. Note that
there is a weight option to the nexthop subcommand of ip route. You might
want check the ip-cref document for that.

Finally, we want to redirect to the specific tables for the case we do have
a source address:

   ip rule add from 100.100.100.1 table 100
   ip rule add from 200.200.200.1 table 200

(Never mind the tables I've chosen, they're there for mnemonic value only)

Anyway, this should provide you with a routing setup that will function,
will keep the SMTP server you mentioned reachable, and will do some load
balancing over the uplinks. The specific tables (100 and 200) are not 
strictly necessary, but they provide you with the option to do a static
choice of the uplink route to take for certain traffic. This is done by
binding the proxy for that traffic to the corresponding IP address.

Doei, Arthur.

-- 
  /\    /  |             Fight Scientology, See URL: <A HREF="http://xenu.xtdnet.nl/">http://xenu.xtdnet.nl/</A> |
 /__\  /   | Buttons. Lotsa buttons. I like buttons. [Big Dog]              |
/    \/__  | A friend is someone with whom you can dare to Be yourself.     |
 Just Be   +-Arthur van Leeuwen, <A HREF="mailto:arthurvl@sci.kun.nl------------------------+">arthurvl@sci.kun.nl------------------------+</A>


</PRE>