[LARTC] Re: Routing packets over multiple links (NICS) all on the

[LARTC] Re: Routing packets over multiple links (NICS) all on the

[Jarek Poplawski]

On 14-08-2006 17:42, Jacques Rompen wrote:
...
> What we want to do is the following:
> I live in a student complex with 7 other people. Every room has its own 
> internet connection from the same ISP.
> Ip, gateway, subnet are asigned through dhcp on mac-adres basis. Every 
> internet connection is capped at 20mbit
> up/down.
> We want to get all computers on an internal network. So we need some 
> sort of router that accepts all 8 internet
> connections and routes it out over a gbit nic -> switch -> internal network
> So we have a computer with 10 nics and "hopefully" enough internal bandwith.

Maybe you have some other reasons to make one gateway but to have 
internal network you only need to set a second address from local 
range (not colliding with that used by ISP) on each box.

But if you really need this together and with multipath, setting 
CONFIG_IP_ROUTE_MULTIPATH = "y" and 
CONFIG_IP_ROUTE_MULTIPATH_CACHED = "n" (!) should change 
multipath randomly.

Jarek P.


_______________________________________________
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

Re: [LARTC] Re: Routing packets over multiple links (NICS) all on the

[Jacques Rompen]

[-- Attachment #1.1: Type: text/plain, Size: 2056 bytes --]

On 8/16/06, Jarek Poplawski <jarkao2@o2.pl> wrote:
>
> On 14-08-2006 17:42, Jacques Rompen wrote:
> ...
> > What we want to do is the following:
> > I live in a student complex with 7 other people. Every room has its own
> > internet connection from the same ISP.
> > Ip, gateway, subnet are asigned through dhcp on mac-adres basis. Every
> > internet connection is capped at 20mbit
> > up/down.
> > We want to get all computers on an internal network. So we need some
> > sort of router that accepts all 8 internet
> > connections and routes it out over a gbit nic -> switch -> internal
> network
> > So we have a computer with 10 nics and "hopefully" enough internal
> bandwith.
>
> Maybe you have some other reasons to make one gateway but to have
> internal network you only need to set a second address from local
> range (not colliding with that used by ISP) on each box.
>
> But if you really need this together and with multipath, setting
> CONFIG_IP_ROUTE_MULTIPATH = "y" and
> CONFIG_IP_ROUTE_MULTIPATH_CACHED = "n" (!) should change
> multipath randomly.
>
> Jarek P.
>
>
> _______________________________________________
> LARTC mailing list
> LARTC@mailman.ds9a.nl
> http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc
>

The 2 ips on every box wouldnt work because: Most people here are complete
network newbies, we want to connect more then 8 computers (most people have
a laptop and pc), we would only have a 20mbit internal network.

Im a bit confused. In the 2.4 kernel there issent even an option
ip_route_multipath_cached, only ip_route_multipath. Or am i mistaking here?
If there issent, then I couldnt have put it to 'Y' so why didnt the random
function work?

In the 2.6 kernel there is a option ip_route_multipath and
ip_route_multipath_cached. But if i dont say 'Y' to
ip_route_multipatch_cached i cant say yes to round_robin, random_match etc.
I thought i needed that option random_match set to 'y'. (im not sure if its
exactly spelled like that, but i'm talking about the four or five sub
options to ip_route_multipath_cached)

[-- Attachment #1.2: Type: text/html, Size: 2681 bytes --]

[-- Attachment #2: Type: text/plain, Size: 143 bytes --]

_______________________________________________
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

[LARTC] Re: Routing packets over multiple links (NICS) all on the

[Jarek Poplawski]

On 16-08-2006 14:50, Jacques Rompen wrote:
...
> The 2 ips on every box wouldnt work because: Most people here are 
> complete network newbies, we want to connect more then 8 computers (most 
> people have a laptop and pc), we would only have a 20mbit internal network.
>  
> Im a bit confused. In the 2.4 kernel there issent even an option 
> ip_route_multipath_cached, only ip_route_multipath. Or am i mistaking here?
> If there issent, then I couldnt have put it to 'Y' so why didnt the 
> random function work?
>  
> In the 2.6 kernel there is a option ip_route_multipath and 
> ip_route_multipath_cached. But if i dont say 'Y' to 
> ip_route_multipatch_cached i cant say yes to round_robin, random_match 
> etc. I thought i needed that option random_match set to 'y'. (im not 
> sure if its exactly spelled like that, but i'm talking about the four or 
> five sub options to ip_route_multipath_cached)

As far as I know multipath cached should work for routing output 
of a current box but it's not working for router (if something 
has not changed lately). Multipath without "cached" should do 
simple randomization, but maybe for the begining it would be 
better to assign fixed paths by source ip and save experimenting 
with multipaths for later.

Jarek P.

_______________________________________________
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

Re: [LARTC] Re: Routing packets over multiple links (NICS) all on the

[Jacques Rompen]

[-- Attachment #1.1: Type: text/plain, Size: 838 bytes --]

On 8/16/06, Jarek Poplawski <jarkao2@o2.pl> wrote:
>
> As far as I know multipath cached should work for routing output
> of a current box but it's not working for router (if something
> has not changed lately). Multipath without "cached" should do
> simple randomization, but maybe for the begining it would be
> better to assign fixed paths by source ip and save experimenting
> with multipaths for later.
>
> Jarek P.


Hi Jarek,

I will try with only multipath.

Ive tried the static routes. But still ive had trouble getting this code to
work
ip ro add default  nexthop via x.x.x.x dev eth1 weight 1 nexthop via
y.y.y.y dev eth2
ip route add default table provider1 via x.x.x.x dev eth1
ip route add default table provider2 via y.y.y.y dev eth2
because x.x.x.x = y.y.y.y in my case. Somehow it doesnt look at the eth
device anymore.

[-- Attachment #1.2: Type: text/html, Size: 1254 bytes --]

[-- Attachment #2: Type: text/plain, Size: 143 bytes --]

_______________________________________________
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

[LARTC] Re: Routing packets over multiple links (NICS) all on the

[Jarek Poplawski]

On 16-08-2006 16:18, Jacques Rompen wrote:
> 
> 
> On 8/16/06, *Jarek Poplawski* <jarkao2@o2.pl <mailto:jarkao2@o2.pl>> wrote:
> 
>     As far as I know multipath cached should work for routing output
>     of a current box but it's not working for router (if something
>     has not changed lately). Multipath without "cached" should do
>     simple randomization, but maybe for the begining it would be
>     better to assign fixed paths by source ip and save experimenting
>     with multipaths for later.
> 
>     Jarek P.
> 
>  
> Hi Jarek,
>  
> I will try with only multipath.
>  
> Ive tried the static routes. But still ive had trouble getting this code 
> to work
> ip ro add default  nexthop via x.x.x.x dev eth1 weight 1 nexthop via
> y.y.y.y dev eth2
> ip route add default table provider1 via x.x.x.x dev eth1
> ip route add default table provider2 via y.y.y.y dev eth2
> because x.x.x.x = y.y.y.y in my case. Somehow it doesnt look at the eth 
> device anymore.

Anyway in my opinion it should work.
Try something like this:

#IP_ROUTE_MULTIPATH = "y"
#(if 2.6.x  IP_ROUTE_MULTIPATH_CACHED = "n" !)
#Here with CONNMARK method (so without Anastasov's patch!).
#Change 192.168.0.0/24 and eth0 to your local network
#address/mask and interface.

ip route add default nexthop via x.x.x.x dev eth1 \
   nexthop via x.x.x.x dev eth2

ip route add 192.168.0.0/24 dev eth0 table 101
ip route add default via x.x.x.x dev eth1 table 101

ip route add 192.168.0.0/24 dev eth0 table 102
ip route add default via x.x.x.x dev eth2 table 102

ip rule add fwmark 1 table 101
ip rule add fwmark 2 table 102

iptables -t mangle -A POSTROUTING -o eth1 -j MARK --set-mark 1
iptables -t mangle -A POSTROUTING -o eth2 -j MARK --set-mark 2

iptables -t mangle -A POSTROUTING -j CONNMARK --save-mark
iptables -t mangle -A PREROUTING -j CONNMARK --restore-mark

echo 0 > /proc/sys/net/ipv4/conf/all/rp_filter

You should also add NAT (MASQERADE) rules.

Nexthop devices will be changed randomly per flow (with some 
caching) so try with destinations from different networks.

Jarek P.

_______________________________________________
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

Re: [LARTC] Re: Routing packets over multiple links (NICS) all on the

[Luciano Ruete]

On Thursday 17 August 2006 03:54, Jarek Poplawski wrote:
> On 16-08-2006 16:18, Jacques Rompen wrote:
> > On 8/16/06, *Jarek Poplawski* <jarkao2@o2.pl <mailto:jarkao2@o2.pl>>
> > wrote:
> >
> >     As far as I know multipath cached should work for routing output
> >     of a current box but it's not working for router (if something
> >     has not changed lately). Multipath without "cached" should do
> >     simple randomization, but maybe for the begining it would be
> >     better to assign fixed paths by source ip and save experimenting
> >     with multipaths for later.
> >
> >     Jarek P.
> >
> >
> > Hi Jarek,
> >
> > I will try with only multipath.
> >
> > Ive tried the static routes. But still ive had trouble getting this code
> > to work
> > ip ro add default  nexthop via x.x.x.x dev eth1 weight 1 nexthop via
> > y.y.y.y dev eth2
> > ip route add default table provider1 via x.x.x.x dev eth1
> > ip route add default table provider2 via y.y.y.y dev eth2
> > because x.x.x.x = y.y.y.y in my case. Somehow it doesnt look at the eth
> > device anymore.
>
> Anyway in my opinion it should work.
> Try something like this:
>
> #IP_ROUTE_MULTIPATH = "y"
> #(if 2.6.x  IP_ROUTE_MULTIPATH_CACHED = "n" !)
> #Here with CONNMARK method (so without Anastasov's patch!).
> #Change 192.168.0.0/24 and eth0 to your local network
> #address/mask and interface.
>
> ip route add default nexthop via x.x.x.x dev eth1 \
>    nexthop via x.x.x.x dev eth2
>
> ip route add 192.168.0.0/24 dev eth0 table 101
> ip route add default via x.x.x.x dev eth1 table 101
>
> ip route add 192.168.0.0/24 dev eth0 table 102
> ip route add default via x.x.x.x dev eth2 table 102
>
> ip rule add fwmark 1 table 101
> ip rule add fwmark 2 table 102
>
> iptables -t mangle -A POSTROUTING -o eth1 -j MARK --set-mark 1
> iptables -t mangle -A POSTROUTING -o eth2 -j MARK --set-mark 2
>
> iptables -t mangle -A POSTROUTING -j CONNMARK --save-mark
> iptables -t mangle -A PREROUTING -j CONNMARK --restore-mark
>
> echo 0 > /proc/sys/net/ipv4/conf/all/rp_filter


if all ethx are on the same switch, you will need:
echo 1 > /proc/sys/net/ipv4/conf/all/arp_ignore
#you can also do it in a per iface basis

by default, if you recive an arp request(whohas) on a iface(iface1), asking 
for an ip that another iface(iface2) has, then the arp request is ansered 
with the MAC address of the first iface(iface1). If all devices are in the 
same fisical segment this can lead to a single device answer for all IPs, or 
to a total caos. 

This one is very hard to catch and i forgot to mention, now solved :-)

--
Luciano
_______________________________________________
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

Re: [LARTC] Re: Routing packets over multiple links (NICS) all on the

[Jacques Rompen]

[-- Attachment #1.1: Type: text/plain, Size: 1363 bytes --]

On 8/17/06, Luciano Ruete <luciano@lugmen.org.ar> wrote:
>
>
>
> if all ethx are on the same switch, you will need:
> echo 1 > /proc/sys/net/ipv4/conf/all/arp_ignore
> #you can also do it in a per iface basis
>
> by default, if you recive an arp request(whohas) on a iface(iface1),
> asking
> for an ip that another iface(iface2) has, then the arp request is ansered
> with the MAC address of the first iface(iface1). If all devices are in the
> same fisical segment this can lead to a single device answer for all IPs,
> or
> to a total caos.
>
> This one is very hard to catch and i forgot to mention, now solved :-)
>
> --
> Luciano


I will try all this tommorow, don't have acces to the box right now.

About turning of arp. If you turn them all of, wouldnt the nics (ip's) be
unfindable from the outside world? Or does the switch they connect to
respond to such an arp request aswell?
But what you are saying is that if i had a box with 2 nics each connected to
a different ISP, so each nic with a different gateway and ip. That if nic1
would recieve a arp request for the ip from nic2, it would respond with
mac-adres from nic1. So that could mean that all packets would come in over
nic1, even if they have destination ip(nic2)? Seems to me that this is never
realy desirable. Especialy if nic2 would have an static ip and i would
unplug nic2 on purpose.

[-- Attachment #1.2: Type: text/html, Size: 1735 bytes --]

[-- Attachment #2: Type: text/plain, Size: 143 bytes --]

_______________________________________________
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

Re: [LARTC] Re: Routing packets over multiple links (NICS) all on the

[Luciano Ruete]

On Thursday 17 August 2006 14:48, Jacques Rompen wrote:
> On 8/17/06, Luciano Ruete <luciano@lugmen.org.ar> wrote:
> > if all ethx are on the same switch, you will need:
> > echo 1 > /proc/sys/net/ipv4/conf/all/arp_ignore
> > #you can also do it in a per iface basis
> >
> > by default, if you recive an arp request(whohas) on a iface(iface1),
> > asking
> > for an ip that another iface(iface2) has, then the arp request is ansered
> > with the MAC address of the first iface(iface1). If all devices are in
> > the same fisical segment this can lead to a single device answer for all
> > IPs, or
> > to a total caos.
> >
> > This one is very hard to catch and i forgot to mention, now solved :-)
> >
> > --
> > Luciano
>
> I will try all this tommorow, don't have acces to the box right now.
>
> About turning of arp. If you turn them all of, wouldnt the nics (ip's) be
> unfindable from the outside world? Or does the switch they connect to
> respond to such an arp request aswell?

what you're turning off is that the device answers arp(whohas) requests for 
ips that are not from that specific device itself. The ips assigned to the 
device will cotinue answer as normal.

> But what you are saying is that if i had a box with 2 nics each connected
> to a different ISP, so each nic with a different gateway and ip. That if
> nic1 would recieve a arp request for the ip from nic2, it would respond
> with mac-adres from nic1. 

exactly!

> So that could mean that all packets would come in 
> over nic1, even if they have destination ip(nic2)? 

exactly, and will arrive destiny anyway, but they are incoming for the wrong 
iface.

> Seems to me that this is 
> never realy desirable.  

It is a really cuestionable default, but for shure that there are reasons for 
this(maybe a thread in lkml archives could answer this cuestion), reasons 
that i do not know.

> Especialy if nic2 would have an static ip and i  
> would unplug nic2 on purpose.

well thats the other scenario where i face the same problem, first guess is 
obvious MAC cache, and you tend to think that is fucked up, and no, it is 
this crossed arp answer.

--
Luciano
_______________________________________________
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

Re: [LARTC] Re: Routing packets over multiple links (NICS) all on the

[Jacques Rompen]

[-- Attachment #1.1: Type: text/plain, Size: 3234 bytes --]

On 8/17/06, Luciano Ruete <luciano@lugmen.org.ar> wrote:
>
> >
> > I will try all this tommorow, don't have acces to the box right now.
> >
> > About turning of arp. If you turn them all of, wouldnt the nics (ip's)
> be
> > unfindable from the outside world? Or does the switch they connect to
> > respond to such an arp request aswell?
>
> what you're turning off is that the device answers arp(whohas) requests
> for
> ips that are not from that specific device itself. The ips assigned to the
> device will cotinue answer as normal.
>
> > But what you are saying is that if i had a box with 2 nics each
> connected
> > to a different ISP, so each nic with a different gateway and ip. That if
> > nic1 would recieve a arp request for the ip from nic2, it would respond
> > with mac-adres from nic1.
>
> exactly!
>
> > So that could mean that all packets would come in
> > over nic1, even if they have destination ip(nic2)?
>
> exactly, and will arrive destiny anyway, but they are incoming for the
> wrong
> iface.
>
> > Seems to me that this is
> > never realy desirable.
>
> It is a really cuestionable default, but for shure that there are reasons
> for
> this(maybe a thread in lkml archives could answer this cuestion), reasons
> that i do not know.
>
> > Especialy if nic2 would have an static ip and i
> > would unplug nic2 on purpose.
>
> well thats the other scenario where i face the same problem, first guess
> is
> obvious MAC cache, and you tend to think that is fucked up, and no, it is
> this crossed arp answer.
>
> --
> Luciano


Hi Luciano and Jarek,

Thanks for all your help so far.

Ive had some try on the box this weekend. Unfortunatly its not in the
student complex anymore, but it will be very soon. So for now ive simulated
the situation by connecting 2 nics to a switch and that switch to a router.
This is the script i used so far, no NAT rules yet because i have no extra
computer to connect to the box. Its Jareks script completed with some hints
from Luciano


ip route add default nexthop via x.x.x.x dev eth1 onlink\
   nexthop via x.x.x.x dev eth2 onlink

ip route add 192.168.0.0/24 dev eth0 table 101
ip route add default via x.x.x.x dev eth1 table 101

ip route add 192.168.0.0/24 dev eth0 table 102
ip route add default via x.x.x.x dev eth2 table 102

ip rule add fwmark 1 table 101
ip rule add fwmark 2 table 102

iptables -t mangle -A POSTROUTING -o eth1 -j MARK --set-mark 1
iptables -t mangle -A POSTROUTING -o eth2 -j MARK --set-mark 2

iptables -t mangle -A POSTROUTING -j CONNMARK --save-mark
iptables -t mangle -A PREROUTING -j CONNMARK --restore-mark

echo 0 > /proc/sys/net/ipv4/conf/all/rp_filter

echo 1 > /proc/sys/net/ipv4/conf/all/arp_ignore

I am thinking if it wouldn't be better to include a src = ipe1 to the "route
add default" entries.
ofcourse this would go wrong if i would recive a new ip from the isp on one
of the nics. So if its not needed then i would rather leave it out.
ive tested the script with

ip route get from IPE1 to 204.152.189.113
ip route get from IPE2 to 204.152.189.113

it does indeed answer wit eth1 for the first and eth2 for the seccond. It
didnt do this with my other attempts. Im not sure if this is a real vallid
tes though.

Regards,

Jacques

[-- Attachment #1.2: Type: text/html, Size: 4067 bytes --]

[-- Attachment #2: Type: text/plain, Size: 143 bytes --]

_______________________________________________
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc