[LARTC] multiple gateway problem

[LARTC] multiple gateway problem

[Ross Simpson]

Hello,

I'm trying to implement a multiple gateway solution on my redat 2.2.16 box,
and am running into troubles.

I have IDSL service from two providers, and have both routers doing port
forwarding of selected ports to my linux box.
Currently, only requests coming from router #1 (the router specified as
GATEWAY in /etc/sysconfig/network) are being fulfilled -- I assume that
requests from router #2 are being sent to router #1 as a gateway, and
therefore never getting to the client.

I have installed iproute-2.2.4-14, and followed the instructions from the
advanced routing howto:

echo 199 lucent >> /etc/iproute2/rt_tables (router #1)
echo 200 speed >> /etc/iproute2/rt_tables (router #2)
ip rule add from 10.4.44.1 table lucent
ip rule add from 10.4.44.2 table speed

ip rule ls:
0:      from all lookup local
32764:  from 10.4.44.2 lookup speed
32765:  from 10.4.44.1 lookup lucent
32766:  from all lookup main
32767:  from all lookup 253

ip route add default via 10.4.44.1 dev eth0 table speed
ip route add default via 10.4.44.2 dev eth0 table lucent
ip route flush cache


I can access forwarded ports on the external ip of the 'lucent' router, but
forwarded ports on the 'speed' router are not answering.
I've verified that port forwarding on the speed router works.


Any ideas?  Am I missing a step here?

thx

Ross


_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://ds9a.nl/2.4Routing/

Re: [LARTC] multiple gateway problem

[Jorge Castellet]

How many ethernet adaptors do you have on linuxbox ?
Can you draw a squeme ?

Jorge ;)

----- Original Message -----
From: "Ross Simpson" <simpsora@usa.net>
To: <lartc@mailman.ds9a.nl>
Sent: Monday, December 03, 2001 8:15 PM
Subject: [LARTC] multiple gateway problem


> Hello,
>
> I'm trying to implement a multiple gateway solution on my redat 2.2.16
box,
> and am running into troubles.
>
> I have IDSL service from two providers, and have both routers doing port
> forwarding of selected ports to my linux box.
> Currently, only requests coming from router #1 (the router specified as
> GATEWAY in /etc/sysconfig/network) are being fulfilled -- I assume that
> requests from router #2 are being sent to router #1 as a gateway, and
> therefore never getting to the client.
>
> I have installed iproute-2.2.4-14, and followed the instructions from the
> advanced routing howto:
>
> echo 199 lucent >> /etc/iproute2/rt_tables (router #1)
> echo 200 speed >> /etc/iproute2/rt_tables (router #2)
> ip rule add from 10.4.44.1 table lucent
> ip rule add from 10.4.44.2 table speed
>
> ip rule ls:
> 0:      from all lookup local
> 32764:  from 10.4.44.2 lookup speed
> 32765:  from 10.4.44.1 lookup lucent
> 32766:  from all lookup main
> 32767:  from all lookup 253
>
> ip route add default via 10.4.44.1 dev eth0 table speed
> ip route add default via 10.4.44.2 dev eth0 table lucent
> ip route flush cache
>
>
> I can access forwarded ports on the external ip of the 'lucent' router,
but
> forwarded ports on the 'speed' router are not answering.
> I've verified that port forwarding on the speed router works.
>
>
> Any ideas?  Am I missing a step here?
>
> thx
>
> Ross
>
>
> _______________________________________________
> LARTC mailing list / LARTC@mailman.ds9a.nl
> http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO:
http://ds9a.nl/2.4Routing/
>


_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://ds9a.nl/2.4Routing/

Re: [LARTC] multiple gateway problem

[Whit Blauvelt]

Ross,

I'm new to this stuff myself, but do you have any default routes set up for
packets that are neither from 10.4.44.1 or 10.4.44.2? Are you testing from
inside or outside? In my own trials what you list below will allow outside
connections through either line, but inside-to-outside stuff isn't caught by
either rule and depends on general defaults.

I'm unclear though on just when a packet generated on the router takes on
which IP as its identity (for instance, when it presents with the
'localhost' IP). It does seem that when a request comes in on a certain
external IP that IP is preserved in the packets of a responding daemon, if
the daemon's on the router, so rules like yours apply to it. 

Obviously I should spend an afternoon with a packet sniffer testing this
stuff out - I haven't run across a full explanation of it yet.

Whit

On Mon, Dec 03, 2001 at 11:15:42AM -0800, Ross Simpson wrote:

> echo 199 lucent >> /etc/iproute2/rt_tables (router #1)
> echo 200 speed >> /etc/iproute2/rt_tables (router #2)
> ip rule add from 10.4.44.1 table lucent
> ip rule add from 10.4.44.2 table speed
> 
> ip rule ls:
> 0:      from all lookup local
> 32764:  from 10.4.44.2 lookup speed
> 32765:  from 10.4.44.1 lookup lucent
> 32766:  from all lookup main
> 32767:  from all lookup 253
> 
> ip route add default via 10.4.44.1 dev eth0 table speed
> ip route add default via 10.4.44.2 dev eth0 table lucent
> ip route flush cache
> 
> I can access forwarded ports on the external ip of the 'lucent' router, but
> forwarded ports on the 'speed' router are not answering.
> I've verified that port forwarding on the speed router works.

_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://ds9a.nl/2.4Routing/

RE: [LARTC] multiple gateway problem

[Ross Simpson]

First off, here's a diagram:


               I N T E R N E T
               /              \
   -------------             -------------
   | 10.4.44.1 |             | 10.4.44.2 |
   |  lucent   |             |   speed   |
   -------------             -------------
    port-fw 80                 port-fw 80
              \               /
               \             /
                \           /
                 -----------
                 |   hub   |
                 -----------
                      |
                      |
                      |
                      |
                     eth0
                --------------
                | 10.4.44.11 |
                |   linux    |
                --------------

I have a default gateway as specified in /etc/sysconfig/network:
	GATEWAYDEV=eth0
	GATEWAY\x10.4.44.1

I ran the below commands to use multiple default gateways.

So here's what I would _like_ to see:
	Traffic coming to the box from the internal network uses the default route
from /etc/sysconfig/network.
	Traffic coming from the internet (from the 10.4.44.1 router, then
port-forwarded 10.4.44.11) should use 10.4.44.1 as the gateway to return the
packets to the client.
	10.4.44.2 should work identically to 10.4.44.1.

Right now, traffic coming from the system default gateway works great.
Traffic coming from 10.4.44.2 gets to the system, however I would guess that
it's being sent back to 10.4.44.1 as it is the default gateway.

As I'm watching a tcpdump, I see that packets are coming in with their
original (external) IP addresses, instead of the address of the router (I
was thinking that port forwarding temporarily changed the source IP of the
packet; apparently not).  So the setup is not working because external IPs
don't match 10.4.44.1 or 10.4.44.2, and the system's default gateway is
used.

So, I guess my question becomes: is there any way for linux to tell which
router the packet came from?  Could it tell maybe by mac address?

Thanks for the help!
Ross


-----Original Message-----
From: Whit Blauvelt [mailto:whit@transpect.com]
Sent: Tuesday, December 04, 2001 9:54 AM
To: Ross Simpson
Cc: lartc@mailman.ds9a.nl
Subject: Re: [LARTC] multiple gateway problem


Ross,

I'm new to this stuff myself, but do you have any default routes set up for
packets that are neither from 10.4.44.1 or 10.4.44.2? Are you testing from
inside or outside? In my own trials what you list below will allow outside
connections through either line, but inside-to-outside stuff isn't caught by
either rule and depends on general defaults.

I'm unclear though on just when a packet generated on the router takes on
which IP as its identity (for instance, when it presents with the
'localhost' IP). It does seem that when a request comes in on a certain
external IP that IP is preserved in the packets of a responding daemon, if
the daemon's on the router, so rules like yours apply to it.

Obviously I should spend an afternoon with a packet sniffer testing this
stuff out - I haven't run across a full explanation of it yet.

Whit

On Mon, Dec 03, 2001 at 11:15:42AM -0800, Ross Simpson wrote:

> echo 199 lucent >> /etc/iproute2/rt_tables (router #1)
> echo 200 speed >> /etc/iproute2/rt_tables (router #2)
> ip rule add from 10.4.44.1 table lucent
> ip rule add from 10.4.44.2 table speed
>
> ip rule ls:
> 0:      from all lookup local
> 32764:  from 10.4.44.2 lookup speed
> 32765:  from 10.4.44.1 lookup lucent
> 32766:  from all lookup main
> 32767:  from all lookup 253
>
> ip route add default via 10.4.44.1 dev eth0 table speed
> ip route add default via 10.4.44.2 dev eth0 table lucent
> ip route flush cache
>
> I can access forwarded ports on the external ip of the 'lucent' router,
but
> forwarded ports on the 'speed' router are not answering.
> I've verified that port forwarding on the speed router works.


_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://ds9a.nl/2.4Routing/

Re: [LARTC] multiple gateway problem

[Whit Blauvelt]

Ross,

The diagram really helps. My guess is what you need to do is put a second
address on eth0, and then differentiate your packets according to which
address they have arrived at on the Linux box - for example, 10.4.44.11 or
10.4.44.12.

And yeah, there should be a way to do it by MAC address, but this should be
simpler.

Whit

On Tue, Dec 04, 2001 at 02:11:37PM -0800, Ross Simpson wrote:
> First off, here's a diagram:
> 
> 
>                I N T E R N E T
>                /              \
>    -------------             -------------
>    | 10.4.44.1 |             | 10.4.44.2 |
>    |  lucent   |             |   speed   |
>    -------------             -------------
>     port-fw 80                 port-fw 80
>               \               /
>                \             /
>                 \           /
>                  -----------
>                  |   hub   |
>                  -----------
>                       |
>                       |
>                       |
>                       |
>                      eth0
>                 --------------
>                 | 10.4.44.11 |
>                 |   linux    |
>                 --------------
> 
> I have a default gateway as specified in /etc/sysconfig/network:
> 	GATEWAYDEV=eth0
> 	GATEWAY\x10.4.44.1
> 
> I ran the below commands to use multiple default gateways.
> 
> So here's what I would _like_ to see:
> 	Traffic coming to the box from the internal network uses the default route
> from /etc/sysconfig/network.
> 	Traffic coming from the internet (from the 10.4.44.1 router, then
> port-forwarded 10.4.44.11) should use 10.4.44.1 as the gateway to return the
> packets to the client.
> 	10.4.44.2 should work identically to 10.4.44.1.
> 
> Right now, traffic coming from the system default gateway works great.
> Traffic coming from 10.4.44.2 gets to the system, however I would guess that
> it's being sent back to 10.4.44.1 as it is the default gateway.
> 
> As I'm watching a tcpdump, I see that packets are coming in with their
> original (external) IP addresses, instead of the address of the router (I
> was thinking that port forwarding temporarily changed the source IP of the
> packet; apparently not).  So the setup is not working because external IPs
> don't match 10.4.44.1 or 10.4.44.2, and the system's default gateway is
> used.
> 
> So, I guess my question becomes: is there any way for linux to tell which
> router the packet came from?  Could it tell maybe by mac address?
> 
> Thanks for the help!
> Ross

_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://ds9a.nl/2.4Routing/

Re: [LARTC] multiple gateway problem

[Kristian Hoffmann]

I think the second addess on the inner-most machine would be
necessary.  The inner machine could even have two private addresses and
have your two routers do NAT.  Then setup multiple routing tables with
different default gateways (one 10.4.44.1, the other 10.4.44.2) and policy
routing rules on your inner machine.  Then outgoing traffic can be sent
through either internet connection.

Ex.

ip rule add from 10.4.44.11 table 100
ip route add default table 100 via 10.4.44.1

ip rule add from 10.4.44.12 table 101
ip route add default table 101 via 10.4.44.2

It also helps sometimes to add the contents of your main routing table to
any others you setup that have default routes since nothing will get past
the default route in that table.

-Kristian


On Tue, 4 Dec 2001, Whit Blauvelt wrote:

> Ross,
> 
> The diagram really helps. My guess is what you need to do is put a second
> address on eth0, and then differentiate your packets according to which
> address they have arrived at on the Linux box - for example, 10.4.44.11 or
> 10.4.44.12.
> 
> And yeah, there should be a way to do it by MAC address, but this should be
> simpler.
> 
> Whit
> 
> On Tue, Dec 04, 2001 at 02:11:37PM -0800, Ross Simpson wrote:
> > First off, here's a diagram:
> > 
> > 
> >                I N T E R N E T
> >                /              \
> >    -------------             -------------
> >    | 10.4.44.1 |             | 10.4.44.2 |
> >    |  lucent   |             |   speed   |
> >    -------------             -------------
> >     port-fw 80                 port-fw 80
> >               \               /
> >                \             /
> >                 \           /
> >                  -----------
> >                  |   hub   |
> >                  -----------
> >                       |
> >                       |
> >                       |
> >                       |
> >                      eth0
> >                 --------------
> >                 | 10.4.44.11 |
> >                 |   linux    |
> >                 --------------
> > 
> > I have a default gateway as specified in /etc/sysconfig/network:
> > 	GATEWAYDEV=eth0
> > 	GATEWAY\x10.4.44.1
> > 
> > I ran the below commands to use multiple default gateways.
> > 
> > So here's what I would _like_ to see:
> > 	Traffic coming to the box from the internal network uses the default route
> > from /etc/sysconfig/network.
> > 	Traffic coming from the internet (from the 10.4.44.1 router, then
> > port-forwarded 10.4.44.11) should use 10.4.44.1 as the gateway to return the
> > packets to the client.
> > 	10.4.44.2 should work identically to 10.4.44.1.
> > 
> > Right now, traffic coming from the system default gateway works great.
> > Traffic coming from 10.4.44.2 gets to the system, however I would guess that
> > it's being sent back to 10.4.44.1 as it is the default gateway.
> > 
> > As I'm watching a tcpdump, I see that packets are coming in with their
> > original (external) IP addresses, instead of the address of the router (I
> > was thinking that port forwarding temporarily changed the source IP of the
> > packet; apparently not).  So the setup is not working because external IPs
> > don't match 10.4.44.1 or 10.4.44.2, and the system's default gateway is
> > used.
> > 
> > So, I guess my question becomes: is there any way for linux to tell which
> > router the packet came from?  Could it tell maybe by mac address?
> > 
> > Thanks for the help!
> > Ross
> 
> _______________________________________________
> LARTC mailing list / LARTC@mailman.ds9a.nl
> http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://ds9a.nl/2.4Routing/
> 


_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://ds9a.nl/2.4Routing/

RE: [LARTC] multiple gateway problem

[Ross Simpson]

Well, I've done some more testing and am still having trouble.

I added another IP to eth0:
	ip addr add 10.4.44.1224 brd 10.4.44.255 dev eth0

It pings fine from either router.

I changed the 10.4.44.1 router to forward port 80 to 10.4.44.11, and
  changed 10.4.44.2 to forward to 10.4.44.12.

When I execute the commands below, I get the same old thing -- connections
to the 10.4.44.1 router work fine, but connections to 10.4.44.2 are never
established.

I also tried changing the 'from' keyword below to 'to', with the same
result.

Here's a tcpdump snippet of a request I made externally to the (external ip
of) the 10.4.44.2 router:
23:15:33.694184 eth0 < 199.182.120.202.33921 > 10.4.44.12.http: S
1955898960:1955898960(0) win 8760 <mss 1460> (DF)
23:15:33.694281 eth0 > 10.4.44.12.http > 199.182.120.202.33921: S
2999353690:2999353690(0) ack 1955898961 win 32120 <mss 1460> (DF)
23:15:33.724810 eth0 < 199.182.120.202.33921 > 10.4.44.12.http: . 1:1(0) ack
1 win 8760 (DF)
23:15:35.347623 eth0 < 199.182.120.202.33921 > 10.4.44.12.http: P 1:8(7) ack
1 win 8760 (DF)
23:15:35.347696 eth0 > 10.4.44.12.http > 199.182.120.202.33921: . 1:1(0) ack
8 win 32120 (DF)
23:15:35.355858 eth0 > 10.4.44.12.http > 199.182.120.202.33921: P
1:1461(1460) ack 8 win 32120 (DF)
23:15:35.355913 eth0 > 10.4.44.12.http > 199.182.120.202.33921: P
1461:2921(1460) ack 8 win 32120 (DF)
23:15:35.571995 eth0 < 199.182.120.202.33921 > 10.4.44.12.http: . 8:8(0) ack
1461 win 8760 (DF)
23:15:35.572119 eth0 > 10.4.44.12.http > 199.182.120.202.33921: P
2921:4381(1460) ack 8 win 32120 (DF)
23:15:35.572144 eth0 > 10.4.44.12.http > 199.182.120.202.33921: P
4381:5841(1460) ack 8 win 32120 (DF)
23:15:35.572157 eth0 > 10.4.44.12.http > 199.182.120.202.33921: FP
5841:7223(1382) ack 8 win 32120 (DF)
23:15:35.984004 eth0 < 199.182.120.202.33921 > 10.4.44.12.http: . 8:8(0) ack
2921 win 8760 (DF)
23:15:36.203976 eth0 < 199.182.120.202.33921 > 10.4.44.12.http: . 8:8(0) ack
4381 win 8760 (DF)
23:15:36.210403 eth0 < 199.182.120.202.33921 > 10.4.44.12.http: . 8:8(0) ack
5841 win 8760 (DF)
23:15:36.330731 eth0 < 199.182.120.202.33921 > 10.4.44.12.http: . 8:8(0) ack
7224 win 8760 (DF)
23:15:36.336468 eth0 < 199.182.120.202.33921 > 10.4.44.12.http: F 8:8(0) ack
7224 win 8760 (DF)
23:15:36.336519 eth0 > 10.4.44.12.http > 199.182.120.202.33921: .
7224:7224(0) ack 9 win 32120 (DF)

The packets seem to hit the correct ip on the webserver, but I can't tell
what happens next.  I'm assuming that the packets are routed back through
the system default gw (10.4.44.1), and therefore never received by the
client.


One thing that I've noticed is strange -- when I issue an 'ip route' command
after setting up the two rules and routes, I get:
	10.4.44.0/24 dev eth0  proto kernel  scope link  src 10.4.44.11
	127.0.0.0/8 dev lo  scope link
	default via 10.4.44.1 dev eth0

It looks like I'm missing one of the routes I've added, but I can only ever
get one route to appear there.


Last point: I'm currently using a 2.2.16-22 kernel, with the iproute2 tools
installed.
Here's the appropriate section of my .config:
	#
	# Networking options
	#
	CONFIG_PACKET=y
	CONFIG_NETLINK=y
	CONFIG_RTNETLINK=y
	CONFIG_NETLINK_DEV=y
	CONFIG_FIREWALL=y
	CONFIG_FILTER=y
	CONFIG_UNIX=y
	CONFIG_INET=y
	CONFIG_IP_MULTICAST=y
	CONFIG_IP_ADVANCED_ROUTER=y
	CONFIG_IP_MULTIPLE_TABLES=y
	CONFIG_IP_ROUTE_MULTIPATH=y
	CONFIG_IP_ROUTE_TOS=y
	CONFIG_IP_ROUTE_VERBOSE=y
	CONFIG_IP_ROUTE_LARGE_TABLES=y
	CONFIG_IP_ROUTE_NAT=y
	# CONFIG_IP_PNP is not set
	CONFIG_IP_FIREWALL=y
	CONFIG_IP_FIREWALL_NETLINK=y
	CONFIG_NETLINK_DEV=y
	CONFIG_IP_ROUTE_FWMARK=y
	CONFIG_IP_TRANSPARENT_PROXY=y
	CONFIG_IP_MASQUERADE=y

Any further ideas?
I seem to be close, but missing some crucial step ;)

Thanks for the help,
Ross



-----Original Message-----
From: lartc-admin@mailman.ds9a.nl [mailto:lartc-admin@mailman.ds9a.nl]On
Behalf Of Kristian Hoffmann
Sent: Tuesday, December 04, 2001 11:15 PM
To: lartc@mailman.ds9a.nl
Subject: Re: [LARTC] multiple gateway problem


I think the second addess on the inner-most machine would be
necessary.  The inner machine could even have two private addresses and
have your two routers do NAT.  Then setup multiple routing tables with
different default gateways (one 10.4.44.1, the other 10.4.44.2) and policy
routing rules on your inner machine.  Then outgoing traffic can be sent
through either internet connection.

Ex.

ip rule add from 10.4.44.11 table 100
ip route add default table 100 via 10.4.44.1

ip rule add from 10.4.44.12 table 101
ip route add default table 101 via 10.4.44.2

It also helps sometimes to add the contents of your main routing table to
any others you setup that have default routes since nothing will get past
the default route in that table.

-Kristian


_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://ds9a.nl/2.4Routing/

Re: [LARTC] multiple gateway problem

[Whit Blauvelt]

On Thu, Dec 06, 2001 at 10:12:23AM -0800, Ross Simpson wrote:
> Well, I've done some more testing and am still having trouble.
> 
> I added another IP to eth0:
> 	ip addr add 10.4.44.1224 brd 10.4.44.255 dev eth0

typo, right? ------------------^

> The packets seem to hit the correct ip on the webserver, but I can't tell
> what happens next.  I'm assuming that the packets are routed back through
> the system default gw (10.4.44.1), and therefore never received by the
> client.
> 
> One thing that I've noticed is strange -- when I issue an 'ip route' command
> after setting up the two rules and routes, I get:
> 	10.4.44.0/24 dev eth0  proto kernel  scope link  src 10.4.44.11
> 	127.0.0.0/8 dev lo  scope link
> 	default via 10.4.44.1 dev eth0

It might be conceptually clearer to set the rules to each IP to point to
different tables. For 10.4.44.0/24 have it lookup table whatever, where that
table has the default route back through the router those would come in on,
and for the other IP block do the same back to its router. This has worked
in my testing here for a very similar setup. It doesn't cover stuff that
originates on the server, but does seem to be working for stuff that comes
in on whatever interface. This should also be doable in a single table, but
the logic is harder to follow. (For stuff originating locally, I understand
the rules will see it as "dev lo" - haven't tested that out.)

Whit


_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://ds9a.nl/2.4Routing/