Multiple inet gw and multipath

Multiple inet gw and multipath

[Alpt]

[-- Attachment #1: Type: text/plain, Size: 1654 bytes --]

Hi there,
this is the situation:

We have a multipath default route. Each nexthop is an ipip tunnel to different
inet gateways.

	default equalize 
        	nexthop via 10.10.1.1  dev tunl0 weight 1
	        nexthop via 10.10.1.2  dev tunl1 weight 1
	        nexthop via 10.10.1.3  dev tunl2 weight 1

Each inet gw has a different IP on the Internet, therefore the tcp connection
breaks if the packets go first from one gw and then after another one.
What we need is to use the same gw for all the packets of a connection.

We can use "-o, --out-interface [!] name" to mark the outgoing packets,
especially the packets which initialize a new connection, and associate each
tunX interface to a routing rule.

What's your advises?

You can understand better the the whole situation here:                                                                        
http://archives.free.net.ph/message/20060207.030317.37885ea1.en.html
http://marc.theaimsgroup.com/?l=linux-net&m=113550638110682&w=2                                                                
and here:                                                                                                                      
http://marc.theaimsgroup.com/?l=linux-net&m=113636640615375&w=2     

If we can resolve this problem, then it we'll be possible to use at the same
time multiple inet gw in Netsukuku:

http://lab.dyne.org/Ntk_IGS

Thanks ^_^
-- 
:wq!
"I don't know nothing" The One Who reached the Thinking Matter   '.'

[ Alpt --- Freaknet Medialab ]
[ GPG Key ID 441CF0EE ]
[ Key fingerprint = 8B02 26E8 831A 7BB9 81A9  5277 BFF8 037E 441C F0EE ]

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

RE: Multiple inet gw and multipath

[Gary W. Smith]

Well, what we do is to mark a packet coming in an interface and use a
special table based on mark to decide which route to take.  Ours is a
little more complicated as we are nat'ing addresses internal from each
external interface.  We end up having servers with two internal IP's
which are nat'ed to an external IP.  But the logic should still hold
true for you.

Iptables sets the mark in the mangle table.
-A PREROUTING -i eth0 -j MARK --set-mark 0x4 
-A PREROUTING -i eth1 -j MARK --set-mark 0x8 

Here is the script to handle the route.
#!/bin/sh

# LAN
LAN_IF=eth2
LAN_IP=10.94.64.254
LAN_NET=10.94.64.0/24

# NET 1
INET1_IF=eth0
INET1_IP=80.90.100.129
INET1_NET=80.90.100.138/29
INET1_GW=80.90.100.134

# NET 2
INET2_IF=eth1
INET2_IP=80.90.100.241
INET2_NET=80.90.100.240/29
INET2_GW=80.90.100.246


/sbin/ip rule del prio 50 table main
/sbin/ip rule add prio 50 table main
/sbin/ip route del default table main

/sbin/ip rule del prio 201 from $INET1_NET table 201
/sbin/ip rule add prio 201 from $INET1_NET table 201
/sbin/ip route add default via $INET1_GW dev $INET1_IF \
            src $INET1_IP proto static table 201
/sbin/ip route append prohibit default table 201 metric 1 proto static

/sbin/ip rule del prio 202 from $INET2_NET table 202
/sbin/ip rule add prio 202 from $INET2_NET table 202
/sbin/ip route add default via $INET2_GW dev $INET2_IF \
            src $INET2_IP proto static table 202
/sbin/ip route append prohibit default table 202 metric 1 proto static


# Match outgoing packets to their source interface
/sbin/ip rule del prio 221 fwmark 4 table 201
/sbin/ip rule add prio 221 fwmark 4 table 201
/sbin/ip rule del prio 222 fwmark 8 table 202
/sbin/ip rule add prio 222 fwmark 8 table 202

/sbin/ip rule del prio 250 table 250
/sbin/ip rule add prio 250 table 250
# Net 2 will be the default route, Net 1 is for incoming DSL
/sbin/ip route add default table 250 proto static \
            nexthop via $INET1_GW dev $INET1_IF \
            nexthop via $INET2_GW dev $INET2_IF




> -----Original Message-----
> From: netfilter-bounces@lists.netfilter.org [mailto:netfilter-
> bounces@lists.netfilter.org] On Behalf Of Alpt
> Sent: Tuesday, February 28, 2006 1:45 PM
> To: netfilter@lists.netfilter.org
> Cc: netsukuku@freaknet.org
> Subject: Multiple inet gw and multipath
> 
> Hi there,
> this is the situation:
> 
> We have a multipath default route. Each nexthop is an ipip tunnel to
> different
> inet gateways.
> 
> 	default equalize
>         	nexthop via 10.10.1.1  dev tunl0 weight 1
> 	        nexthop via 10.10.1.2  dev tunl1 weight 1
> 	        nexthop via 10.10.1.3  dev tunl2 weight 1
> 
> Each inet gw has a different IP on the Internet, therefore the tcp
> connection
> breaks if the packets go first from one gw and then after another one.
> What we need is to use the same gw for all the packets of a
connection.
> 
> We can use "-o, --out-interface [!] name" to mark the outgoing
packets,
> especially the packets which initialize a new connection, and
associate
> each
> tunX interface to a routing rule.
> 
> What's your advises?
> 
> You can understand better the the whole situation here:
> http://archives.free.net.ph/message/20060207.030317.37885ea1.en.html
> http://marc.theaimsgroup.com/?l=linux-net&m=113550638110682&w=2
> and here:
> http://marc.theaimsgroup.com/?l=linux-net&m=113636640615375&w=2
> 
> If we can resolve this problem, then it we'll be possible to use at
the
> same
> time multiple inet gw in Netsukuku:
> 
> http://lab.dyne.org/Ntk_IGS
> 
> Thanks ^_^
> --
> :wq!
> "I don't know nothing" The One Who reached the Thinking Matter   '.'
> 
> [ Alpt --- Freaknet Medialab ]
> [ GPG Key ID 441CF0EE ]
> [ Key fingerprint = 8B02 26E8 831A 7BB9 81A9  5277 BFF8 037E 441C F0EE
]

Re: Multiple inet gw and multipath

[Alpt]

On Tue, Feb 28, 2006 at 03:06:57PM -0800, <Gary W. Smith>:
~> Well, what we do is to mark a packet coming in an interface and use a
~> special table based on mark to decide which route to take.  Ours is a
~> little more complicated as we are nat'ing addresses internal from each
~> external interface.  We end up having servers with two internal IP's
~> which are nat'ed to an external IP.  But the logic should still hold
~> true for you.
~> 
~> Iptables sets the mark in the mangle table.
~> -A PREROUTING -i eth0 -j MARK --set-mark 0x4 
~> -A PREROUTING -i eth1 -j MARK --set-mark 0x8


Unfortunately we have to use only one IP per node because we are in an adhoc
net. If we assign different IPs to the same node conflicts will arise, 
therefore each interface and tunnel has to have the same IP.
If it hadn't been like this we would have used the source routing method
described in the lartc HOWTO.

Let's recap:
We have multiple gw. When a new connection is established through a gw,
all the packets belonging to the same connection must be sent through the
same gw. 
We cannot use the source routing method since all the IFs use the same IP,
thus in order to accomplish this we have to:
mark with the same id all the packets which belong to the same
connection.
Each connection has to have a different mark in order to go through
different gateways.

A simple idea is to assign a mark to each tunnel (outgoing IF), and
when a new connection is created through a specific tunnel, all the outgoing
packets of the connection are marked with the same id. But how?

Another idea is to conntrack the connection and marking the packets with a
4bit number which is the hash of the destination IP. Probably this requires a
new netfilter extension.

That's all,
regards and thanks
-- 
:wq!
"I don't know nothing" The One Who reached the Thinking Matter   '.'

[ Alpt --- Freaknet Medialab ]
[ GPG Key ID 441CF0EE ]
[ Key fingerprint = 8B02 26E8 831A 7BB9 81A9  5277 BFF8 037E 441C F0EE ]

Re: Multiple inet gw and multipath

[Philip Craig]

On 03/01/2006 02:55 PM, Alpt wrote:
> We have multiple gw. When a new connection is established through a gw,
> all the packets belonging to the same connection must be sent through the
> same gw. 
> We cannot use the source routing method since all the IFs use the same IP,
> thus in order to accomplish this we have to:
> mark with the same id all the packets which belong to the same
> connection.
> Each connection has to have a different mark in order to go through
> different gateways.
> 
> A simple idea is to assign a mark to each tunnel (outgoing IF), and
> when a new connection is created through a specific tunnel, all the outgoing
> packets of the connection are marked with the same id. But how?
> 
> Another idea is to conntrack the connection and marking the packets with a
> 4bit number which is the hash of the destination IP. Probably this requires a
> new netfilter extension.

Why the destination IP?
It should work if you just mark the connection with the same mark
you use for the route tables.  Some untested rules:

# Save the gateway in the connection mark for new outgoing connections
iptables -t mangle -A POSTROUTING -o eth0 -m conntrack --ctstate NEW -j CONNMARK --set-mark 0x4
iptables -t mangle -A POSTROUTING -o eth1 -m conntrack --ctstate NEW -j CONNMARK --set-mark 0x8
# Save the gateway in the connection mark for new incoming connections
iptables -t mangle -A PREROUTING -i eth0 -m conntrack --ctstate NEW -j CONNMARK --set-mark 0x4
iptables -t mangle -A PREROUTING -i eth1 -m conntrack --ctstate NEW -j CONNMARK --set-mark 0x8
# Use the correct gateway for reply packets from the LAN
iptables -t mangle -A PREROUTING -i eth2 -m conntrack --ctstate ESTABLISHED,RELATED -j CONNMARK --restore-mark
# Use the correct gateway for reply packets from local connections
iptables -t mangle -A OUTPUT -m conntrack --ctstate ESTABLISHED,RELATED -j CONNMARK --restore-mark

Re: Multiple inet gw and multipath

[Alpt]

On Wed, Mar 01, 2006 at 03:35:54PM +1000, <Philip Craig>:
~> On 03/01/2006 02:55 PM, Alpt wrote:
~> > We have multiple gw. When a new connection is established through a gw,
~> > all the packets belonging to the same connection must be sent through the
~> > same gw. 
~> > We cannot use the source routing method since all the IFs use the same IP,
~> > thus in order to accomplish this we have to:
~> > mark with the same id all the packets which belong to the same
~> > connection.
~> > Each connection has to have a different mark in order to go through
~> > different gateways.

~> It should work if you just mark the connection with the same mark
~> you use for the route tables.  Some untested rules:
~> # Save the gateway in the connection mark for new outgoing connections
~> iptables -t mangle -A POSTROUTING -o eth0 -m conntrack --ctstate NEW -j CONNMARK --set-mark 0x4
~> iptables -t mangle -A POSTROUTING -o eth1 -m conntrack --ctstate NEW -j CONNMARK --set-mark 0x8
~> # Use the correct gateway for reply packets from local connections
~> iptables -t mangle -A OUTPUT -m conntrack --ctstate ESTABLISHED,RELATED -j CONNMARK --restore-mark

Thanks for your advice, it works.
Here below there are the rules and routes we've used in the tests.
(I'm including them here because they might be useful to someone who'll read
the archive of the ml in search of the solution for the same problem ;)

pc1:~/src# ip rule
0:      from all lookup local 
32764:  from 10.198.117.159 fwmark 0x8 lookup 202 
32765:  from 10.198.117.159 fwmark 0x4 lookup 201 
32766:  from all lookup main 
32767:  from all lookup default 
pc1:~/src# 

pc1:~/src# ip route show table 201
default via 10.198.117.95 dev tunl0 

pc1:~/src# ip route show table 202
default via 10.198.117.3 dev tunl1 
pc1:~/src# 

pc1:~/src# ip route
10.198.117.3 via 10.198.117.159 dev eth0  proto 15  scope link 
10.198.117.95 via 10.198.117.159 dev eth1  proto 15  scope link 
10.0.0.0/8 dev eth0  proto kernel  scope link  src 10.198.117.159 
10.0.0.0/8 dev eth1  proto kernel  scope link  src 10.198.117.159 
10.0.0.0/8 dev tunl0  proto kernel  scope link  src 10.198.117.159 
10.0.0.0/8 dev tunl1  proto kernel  scope link  src 10.198.117.159 
default equalize 
        nexthop via 10.198.117.95  dev tunl0 weight 1
        nexthop via 10.198.117.3  dev tunl1 weight 1
pc1:~/src# 

iptables -t mangle -A OUTPUT -m conntrack --ctstate ESTABLISHED,RELATED  -j CONNMARK --restore-mark
iptables -t mangle -A POSTROUTING -o tunl0 -m conntrack --ctstate NEW -j CONNMARK --set-mark 0x4
iptables -t mangle -A POSTROUTING -o tunl1 -m conntrack --ctstate NEW -j CONNMARK --set-mark 0x8

ip rule add from  10.198.117.159 fwmark 0x4 lookup 201
ip rule add from  10.198.117.159 fwmark 0x8 lookup 202

ip route add table 201 default via 10.198.117.95  dev tunl0
ip route add table 202 default via 10.198.117.3  dev tunl1

-- 
:wq!
"I don't know nothing" The One Who reached the Thinking Matter   '.'

[ Alpt --- Freaknet Medialab ]
[ GPG Key ID 441CF0EE ]
[ Key fingerprint = 8B02 26E8 831A 7BB9 81A9  5277 BFF8 037E 441C F0EE ]