Problem with routing decisions, and multihop

Problem with routing decisions, and multihop

[Lluís Batlle]

[-- Attachment #1: Type: text/plain, Size: 3340 bytes --]

Hi!

I have many problems getting this thing to work. There's a host with
two network interfaces, where there are two routers to Internet in two
separated networks. The host uses multihop routing for deciding to
which router send the packets... but the routing decision is wrong
made. Some packets with source address of one NIC, go to other
network.

I have a host with three NICs in it:
eth0 - LAN, 192.168.0.0/20
eth1 192.168.16.1 - subnetwork 192.168.16.0/28, with a router
(192.168.16.2) to internet
eth2 192.168.17.1 - subnetwork 192.168.17.0/28, with another router
(192.168.17.2) to internet

The routing rules are:
0:      from all lookup local 
50:     from all lookup main 
201:    from 192.168.17.0/28 iif eth2 lookup 201 
202:    from 192.168.16.0/28 iif eth1 lookup 202 
222:    from all lookup 222 
32766:  from all lookup main 
32767:  from all lookup default 

The table 'main':
192.168.17.0/28 dev eth2  proto kernel  scope link  src 192.168.17.1 
192.168.16.0/28 dev eth1  proto kernel  scope link  src 192.168.16.1 
192.168.0.0/20 dev eth0  proto kernel  scope link  src 192.168.1.2 

The table '201':
default via 192.168.17.2 dev eth2  proto static  src 192.168.17.1 
prohibit default  proto static  metric 1 

The table '202':
default via 192.168.16.2 dev eth1  proto static  src 192.168.16.1 
prohibit default  proto static  metric 1 

The table '222', where there is the multihop gateway specification:

default equalize 
        nexthop via 192.168.16.2  dev eth1 weight 1
        nexthop via 192.168.17.2  dev eth2 weight 1

I've added the following packet LOG lines into 'mangle' table, for
knowing when the "WRONG INTERFACE" decision is being made:
Chain POSTROUTING (policy ACCEPT 329K packets, 93M bytes)
 pkts bytes target     prot opt in     out     source              
destination
    2    80 LOG        all  --  any    eth1    192.168.17.1        
anywhere            LOG level warning ip-options prefix `WRONG IFACE:
'
    0     0 LOG        all  --  any    eth2    192.168.16.1        
anywhere            LOG level warning ip-options prefix `WRONG IFACE:
'

(Don't look at counters; right now, for getting good internet access,
I'm not using multihop)

So, often appears in the kernel log, specially with 'ftp' and 'ssh'
connections (and rarely with www connections):
Jul  4 15:50:14 thecrow WRONG IFACE: IN= OUT=eth2 SRC=192.168.16.1
DST=216.165.191.52 LE
N=72 TOS=0x00 PREC=0x00 TTL=64 ID=9582 DF PROTO=TCP SPT=56528 DPT=6667
WINDOW=18824 RES=
0x00 ACK PSH URGP=0 
Jul  4 16:01:29 thecrow WRONG IFACE: IN= OUT=eth1 SRC=192.168.17.1
DST=130.206.1.5 LEN=40 TOS=0x00 PREC=0x00 TTL=64 ID=66 DF PROTO=TCP
SPT=33820 DPT=21 WINDOW=0 RES=0x00 RST URGP=0

Even though, when I use 'tcpdump' for catching the wrong packets (that
is: tcpdump -i eth1 host 192.168.17.1  _or_ tcpdump -i eth2 host
192.168.16.1 ) results that _A LOT MORE PACKETS_ are BADLY ROUTED,
than sent to the LOG target.

My conclusion: iptables 'matching' doesn't work; also does the route
decision part.

I absolutely don't know what more to do... I'm running iptables
v1.2.11, and kernel 2.6.11-gentoo-r11. Exactly same happened with
kernel 2.4.28-gentoo. :(

I even attach the scripts I use for doing the routing and the NAT for the LAN.

Please, help!

[-- Attachment #2: masquerading.multi-eth --]
[-- Type: application/octet-stream, Size: 3880 bytes --]

#!/bin/bash

GW1=192.168.16.2
GW2=192.168.17.2
NE1=192.168.16.0/28
NE2=192.168.17.0/28
NLOCAL=192.168.0.0/20
IPE1=192.168.16.1
IPE2=192.168.17.1

IPTABLES="/sbin/iptables"

$IPTABLES -t nat -F PREROUTING
$IPTABLES -t nat -F POSTROUTING
$IPTABLES -t nat -F OUTPUT
$IPTABLES -t filter -F INPUT
$IPTABLES -t filter -F FORWARD
$IPTABLES -t filter -F OUTPUT
$IPTABLES -t filter -F keep_state >&/dev/null
$IPTABLES -t filter -X keep_state >&/dev/null
$IPTABLES -t nat -F keep_state >&/dev/null
$IPTABLES -t nat -X keep_state >&/dev/null


$IPTABLES -t filter -N keep_state
$IPTABLES -t filter -A keep_state -m state --state RELATED,ESTABLISHED -j ACCEPT
$IPTABLES -t filter -A keep_state -j RETURN

$IPTABLES -t nat -N keep_state
$IPTABLES -t nat -A keep_state -m state --state RELATED,ESTABLISHED -j ACCEPT
$IPTABLES -t nat -A keep_state -j RETURN

$IPTABLES -t nat -A PREROUTING -j keep_state
$IPTABLES -t nat -A POSTROUTING -j keep_state
$IPTABLES -t nat -A OUTPUT -j keep_state
$IPTABLES -t filter -A INPUT -j keep_state
$IPTABLES -t filter -A FORWARD -j keep_state
$IPTABLES -t filter -A OUTPUT -j keep_state

$IPTABLES -t nat -A POSTROUTING -o eth1 -s $NLOCAL -j SNAT --to $IPE1
$IPTABLES -t nat -A POSTROUTING -o eth2 -s $NLOCAL -j SNAT --to $IPE2
#$IPTABLES -t nat -A POSTROUTING -o eth1 -j MASQUERADE
#$IPTABLES -t nat -A POSTROUTING -o eth2 -j MASQUERADE

# Accés web a en filemon per thecrow:2000
$IPTABLES -t nat -A PREROUTING --protocol tcp \
	--destination-port 2000 -s 192.168.0.0/20 -j DNAT\
	--to-destination 192.168.17.2:80

# Accés telnet a en filemon per thecrow:2001
$IPTABLES -t nat -A PREROUTING --protocol tcp \
	--destination-port 2001 -s 192.168.0.0/20 -j DNAT\
	--to-destination 192.168.17.2:23

# Accés telnet a en mortadelo per thecrow:2002
$IPTABLES -t nat -A PREROUTING --protocol tcp \
	--destination-port 2002 -s 192.168.0.0/20 -j DNAT\
	--to-destination 192.168.16.2:23

# Accés a l'AS400 per telnet, des d'ordinadors concrets
$IPTABLES -t nat -A PREROUTING --protocol tcp \
	--destination-port 23 -s 80.32.29.234 \
	-d 80.24.26.25 -j DNAT --to-destination 192.168.1.7:23

# Accés al DNS secundari (per filemon, a Anubis)
$IPTABLES -t nat -A PREROUTING --protocol udp \
	--destination-port 53 -s 0.0.0.0/0 -d 192.168.17.1 \
	-j DNAT --to-destination 192.168.1.4:53

# Accés al DNS primari (per mortadelo, a Anubis)
$IPTABLES -t nat -A PREROUTING --protocol udp \
	--destination-port 53 -s 0.0.0.0/0 -d 192.168.16.1 \
	-j DNAT --to-destination 192.168.1.3:53

# Accés al SMTP (per mortadelo, a Spawn)
$IPTABLES -t nat -A PREROUTING --protocol tcp \
	--destination-port 25 -s 0.0.0.0/0 -d 192.168.16.1 \
	-j DNAT --to-destination 192.168.1.1:25

# Accés al POP3 (per mortadelo, a Spawn)
$IPTABLES -t nat -A PREROUTING --protocol tcp \
	--destination-port 110 -s 0.0.0.0/0 -d 192.168.16.1 \
	-j DNAT --to-destination 192.168.1.1:110

# Accés al WEB (per mortadelo, a Spawn)
$IPTABLES -t nat -A PREROUTING --protocol tcp \
	--destination-port 80 -s 0.0.0.0/0 -d 192.168.16.1 \
	-j DNAT --to-destination 192.168.1.1:80

# Accés al WEB HTTPS (per mortadelo, a Spawn)
$IPTABLES -t nat -A PREROUTING --protocol tcp \
	--destination-port 443 -s 0.0.0.0/0 -d 192.168.16.1 \
	-j DNAT --to-destination 192.168.1.1:443

# Accés al SSH (per mortadelo, a dns)
$IPTABLES -t nat -A PREROUTING --protocol tcp \
	--destination-port 22 -s 0.0.0.0/0 -d 192.168.16.1 \
	-j DNAT --to-destination 192.168.1.3:22

# Accés al SSH (per mortadelo, a dns)
$IPTABLES -t nat -A PREROUTING --protocol tcp \
	--destination-port 50 -s 0.0.0.0/0 -d 192.168.16.1 \
	-j DNAT --to-destination 192.168.1.3:22

# LOG de paquets erronis
iptables -t mangle -F POSTROUTING
iptables -t mangle -A POSTROUTING -j LOG --log-prefix "WRONG IFACE: " --log-ip-options -s 192.168.17.1 -o eth1
iptables -t mangle -A POSTROUTING -j LOG --log-prefix "WRONG IFACE: " --log-ip-options -s 192.168.16.1 -o eth2

[-- Attachment #3: routing.multi-eth --]
[-- Type: application/octet-stream, Size: 1538 bytes --]

#!/bin/bash

echo 1 >/proc/sys/net/ipv4/conf/all/forwarding

/sbin/modprobe ip_conntrack_ftp
/sbin/modprobe ip_nat_ftp

IP="/sbin/ip"

ip route flush all
$IP link set eth0 up
$IP addr flush dev eth0
$IP addr add 192.168.1.2/20 brd + dev eth0
$IP addr flush dev eth1
$IP addr add 192.168.16.1/28 brd + dev eth1
$IP addr flush dev eth2
$IP addr add 192.168.17.1/28 brd + dev eth2

ifconfig eth0 up
ifconfig eth1 up
ifconfig eth2 up
$IP rule del prio 50 >&/dev/null
$IP rule add prio 50 table main
$IP route del default table main >&/dev/null

# a mortadelo i filemón
#GW1=192.168.1.10
#GW2=192.168.1.13
GW1=192.168.16.2
GW2=192.168.17.2
NE1=192.168.16.0/28
NE2=192.168.17.0/28
IPE1=192.168.16.1
IPE2=192.168.17.1

# proto static
#$IP route del table 222 >&/dev/null
#$IP route add default table 222 equalize\
#	nexthop via $GW1 dev eth1 weight 1 nexthop via $GW2 dev eth2 weight 1
$IP route add default table 222 via $GW1 dev eth1

$IP rule del prio 222 >&/dev/null
$IP rule add prio 222 table 222

$IP rule del prio 202 from $NE1 >&/dev/null
$IP route del table 202 >&/dev/null
$IP route add default via $GW1 dev eth1 src $IPE1 proto static table 202
$IP route append prohibit default table 202 metric 1 proto static
$IP rule add prio 202 from $NE1 dev eth1 table 202

$IP rule del prio 201 from $NE2 >&/dev/null
$IP route del table 201 >&/dev/null
$IP route add default via $GW2 dev eth2 src $IPE2 proto static table 201
$IP route append prohibit default table 201 metric 1 proto static
$IP rule add prio 201 from $NE2 dev eth2 table 201

Problem with routing decisions, and multihop

[Lluis Batle]

[-- Attachment #1: Type: text/plain, Size: 3294 bytes --]

Hi!

I have many problems getting this thing to work. There's a host with
two network interfaces, where there are two routers to Internet in two
separated networks. The host uses multihop routing for deciding to
which router send the packets... but the routing decision is wrong
made. Some packets with source address of one NIC, go to other
network.

I have a host with three NICs in it:
eth0 - LAN, 192.168.0.0/20
eth1 192.168.16.1 - subnetwork 192.168.16.0/28, with a router
(192.168.16.2) to internet
eth2 192.168.17.1 - subnetwork 192.168.17.0/28, with another router
(192.168.17.2) to internet

The routing rules are:
0:      from all lookup local
50:     from all lookup main
201:    from 192.168.17.0/28 iif eth2 lookup 201
202:    from 192.168.16.0/28 iif eth1 lookup 202
222:    from all lookup 222
32766:  from all lookup main
32767:  from all lookup default

The table 'main':
192.168.17.0/28 dev eth2  proto kernel  scope link  src 192.168.17.1
192.168.16.0/28 dev eth1  proto kernel  scope link  src 192.168.16.1
192.168.0.0/20 dev eth0  proto kernel  scope link  src 192.168.1.2

The table '201':
default via 192.168.17.2 dev eth2  proto static  src 192.168.17.1
prohibit default  proto static  metric 1

The table '202':
default via 192.168.16.2 dev eth1  proto static  src 192.168.16.1
prohibit default  proto static  metric 1

The table '222', where there is the multihop gateway specification:

default equalize
        nexthop via 192.168.16.2  dev eth1 weight 1
        nexthop via 192.168.17.2  dev eth2 weight 1

I've added the following packet LOG lines into 'mangle' table, for
knowing when the "WRONG INTERFACE" decision is being made:
Chain POSTROUTING (policy ACCEPT 329K packets, 93M bytes)
 pkts bytes target     prot opt in     out     source
destination
    2    80 LOG        all  --  any    eth1    192.168.17.1
anywhere            LOG level warning ip-options prefix `WRONG IFACE:
'
    0     0 LOG        all  --  any    eth2    192.168.16.1
anywhere            LOG level warning ip-options prefix `WRONG IFACE:
'

(Don't look at counters; right now, for getting good internet access,
I'm not using multihop)

So, often appears in the kernel log, specially with 'ftp' and 'ssh'
connections (and rarely with www connections):
Jul  4 15:50:14 thecrow WRONG IFACE: IN= OUT=eth2 SRC=192.168.16.1
DST=216.165.191.52 LE
N=72 TOS=0x00 PREC=0x00 TTL=64 ID=9582 DF PROTO=TCP SPT=56528 DPT=6667
WINDOW=18824 RES=
0x00 ACK PSH URGP=0
Jul  4 16:01:29 thecrow WRONG IFACE: IN= OUT=eth1 SRC=192.168.17.1
DST=130.206.1.5 LEN=40 TOS=0x00 PREC=0x00 TTL=64 ID=66 DF PROTO=TCP
SPT=33820 DPT=21 WINDOW=0 RES=0x00 RST URGP=0

Even though, when I use 'tcpdump' for catching the wrong packets (that
is: tcpdump -i eth1 host 192.168.17.1  _or_ tcpdump -i eth2 host
192.168.16.1 ) results that _A LOT MORE PACKETS_ are BADLY ROUTED,
than sent to the LOG target.

My conclusion: iptables 'matching' doesn't work; also does the route
decision part.

I absolutely don't know what more to do... I'm running iptables
v1.2.11, and kernel 2.6.11-gentoo-r11. Exactly same happened with
kernel 2.4.28-gentoo. :(

I even attach the scripts I use for doing the routing and the NAT for the LAN.

Please, help!

[-- Attachment #2: masquerading.multi-eth --]
[-- Type: application/octet-stream, Size: 3880 bytes --]

#!/bin/bash

GW1=192.168.16.2
GW2=192.168.17.2
NE1=192.168.16.0/28
NE2=192.168.17.0/28
NLOCAL=192.168.0.0/20
IPE1=192.168.16.1
IPE2=192.168.17.1

IPTABLES="/sbin/iptables"

$IPTABLES -t nat -F PREROUTING
$IPTABLES -t nat -F POSTROUTING
$IPTABLES -t nat -F OUTPUT
$IPTABLES -t filter -F INPUT
$IPTABLES -t filter -F FORWARD
$IPTABLES -t filter -F OUTPUT
$IPTABLES -t filter -F keep_state >&/dev/null
$IPTABLES -t filter -X keep_state >&/dev/null
$IPTABLES -t nat -F keep_state >&/dev/null
$IPTABLES -t nat -X keep_state >&/dev/null


$IPTABLES -t filter -N keep_state
$IPTABLES -t filter -A keep_state -m state --state RELATED,ESTABLISHED -j ACCEPT
$IPTABLES -t filter -A keep_state -j RETURN

$IPTABLES -t nat -N keep_state
$IPTABLES -t nat -A keep_state -m state --state RELATED,ESTABLISHED -j ACCEPT
$IPTABLES -t nat -A keep_state -j RETURN

$IPTABLES -t nat -A PREROUTING -j keep_state
$IPTABLES -t nat -A POSTROUTING -j keep_state
$IPTABLES -t nat -A OUTPUT -j keep_state
$IPTABLES -t filter -A INPUT -j keep_state
$IPTABLES -t filter -A FORWARD -j keep_state
$IPTABLES -t filter -A OUTPUT -j keep_state

$IPTABLES -t nat -A POSTROUTING -o eth1 -s $NLOCAL -j SNAT --to $IPE1
$IPTABLES -t nat -A POSTROUTING -o eth2 -s $NLOCAL -j SNAT --to $IPE2
#$IPTABLES -t nat -A POSTROUTING -o eth1 -j MASQUERADE
#$IPTABLES -t nat -A POSTROUTING -o eth2 -j MASQUERADE

# Accés web a en filemon per thecrow:2000
$IPTABLES -t nat -A PREROUTING --protocol tcp \
	--destination-port 2000 -s 192.168.0.0/20 -j DNAT\
	--to-destination 192.168.17.2:80

# Accés telnet a en filemon per thecrow:2001
$IPTABLES -t nat -A PREROUTING --protocol tcp \
	--destination-port 2001 -s 192.168.0.0/20 -j DNAT\
	--to-destination 192.168.17.2:23

# Accés telnet a en mortadelo per thecrow:2002
$IPTABLES -t nat -A PREROUTING --protocol tcp \
	--destination-port 2002 -s 192.168.0.0/20 -j DNAT\
	--to-destination 192.168.16.2:23

# Accés a l'AS400 per telnet, des d'ordinadors concrets
$IPTABLES -t nat -A PREROUTING --protocol tcp \
	--destination-port 23 -s 80.32.29.234 \
	-d 80.24.26.25 -j DNAT --to-destination 192.168.1.7:23

# Accés al DNS secundari (per filemon, a Anubis)
$IPTABLES -t nat -A PREROUTING --protocol udp \
	--destination-port 53 -s 0.0.0.0/0 -d 192.168.17.1 \
	-j DNAT --to-destination 192.168.1.4:53

# Accés al DNS primari (per mortadelo, a Anubis)
$IPTABLES -t nat -A PREROUTING --protocol udp \
	--destination-port 53 -s 0.0.0.0/0 -d 192.168.16.1 \
	-j DNAT --to-destination 192.168.1.3:53

# Accés al SMTP (per mortadelo, a Spawn)
$IPTABLES -t nat -A PREROUTING --protocol tcp \
	--destination-port 25 -s 0.0.0.0/0 -d 192.168.16.1 \
	-j DNAT --to-destination 192.168.1.1:25

# Accés al POP3 (per mortadelo, a Spawn)
$IPTABLES -t nat -A PREROUTING --protocol tcp \
	--destination-port 110 -s 0.0.0.0/0 -d 192.168.16.1 \
	-j DNAT --to-destination 192.168.1.1:110

# Accés al WEB (per mortadelo, a Spawn)
$IPTABLES -t nat -A PREROUTING --protocol tcp \
	--destination-port 80 -s 0.0.0.0/0 -d 192.168.16.1 \
	-j DNAT --to-destination 192.168.1.1:80

# Accés al WEB HTTPS (per mortadelo, a Spawn)
$IPTABLES -t nat -A PREROUTING --protocol tcp \
	--destination-port 443 -s 0.0.0.0/0 -d 192.168.16.1 \
	-j DNAT --to-destination 192.168.1.1:443

# Accés al SSH (per mortadelo, a dns)
$IPTABLES -t nat -A PREROUTING --protocol tcp \
	--destination-port 22 -s 0.0.0.0/0 -d 192.168.16.1 \
	-j DNAT --to-destination 192.168.1.3:22

# Accés al SSH (per mortadelo, a dns)
$IPTABLES -t nat -A PREROUTING --protocol tcp \
	--destination-port 50 -s 0.0.0.0/0 -d 192.168.16.1 \
	-j DNAT --to-destination 192.168.1.3:22

# LOG de paquets erronis
iptables -t mangle -F POSTROUTING
iptables -t mangle -A POSTROUTING -j LOG --log-prefix "WRONG IFACE: " --log-ip-options -s 192.168.17.1 -o eth1
iptables -t mangle -A POSTROUTING -j LOG --log-prefix "WRONG IFACE: " --log-ip-options -s 192.168.16.1 -o eth2

[-- Attachment #3: routing.multi-eth --]
[-- Type: application/octet-stream, Size: 1538 bytes --]

#!/bin/bash

echo 1 >/proc/sys/net/ipv4/conf/all/forwarding

/sbin/modprobe ip_conntrack_ftp
/sbin/modprobe ip_nat_ftp

IP="/sbin/ip"

ip route flush all
$IP link set eth0 up
$IP addr flush dev eth0
$IP addr add 192.168.1.2/20 brd + dev eth0
$IP addr flush dev eth1
$IP addr add 192.168.16.1/28 brd + dev eth1
$IP addr flush dev eth2
$IP addr add 192.168.17.1/28 brd + dev eth2

ifconfig eth0 up
ifconfig eth1 up
ifconfig eth2 up
$IP rule del prio 50 >&/dev/null
$IP rule add prio 50 table main
$IP route del default table main >&/dev/null

# a mortadelo i filemón
#GW1=192.168.1.10
#GW2=192.168.1.13
GW1=192.168.16.2
GW2=192.168.17.2
NE1=192.168.16.0/28
NE2=192.168.17.0/28
IPE1=192.168.16.1
IPE2=192.168.17.1

# proto static
#$IP route del table 222 >&/dev/null
#$IP route add default table 222 equalize\
#	nexthop via $GW1 dev eth1 weight 1 nexthop via $GW2 dev eth2 weight 1
$IP route add default table 222 via $GW1 dev eth1

$IP rule del prio 222 >&/dev/null
$IP rule add prio 222 table 222

$IP rule del prio 202 from $NE1 >&/dev/null
$IP route del table 202 >&/dev/null
$IP route add default via $GW1 dev eth1 src $IPE1 proto static table 202
$IP route append prohibit default table 202 metric 1 proto static
$IP rule add prio 202 from $NE1 dev eth1 table 202

$IP rule del prio 201 from $NE2 >&/dev/null
$IP route del table 201 >&/dev/null
$IP route add default via $GW2 dev eth2 src $IPE2 proto static table 201
$IP route append prohibit default table 201 metric 1 proto static
$IP rule add prio 201 from $NE2 dev eth2 table 201

Problem with routing decisions, and multihop

[Lluis Batle]

Hi!

I have many problems getting this thing to work. There's a host with
two network interfaces, where there are two routers to Internet in two
separated networks. The host uses multihop routing for deciding to
which router send the packets... but the routing decision is wrong
made. Some packets with source address of one NIC, go to other
network.

I have a host with three NICs in it:
eth0 - LAN, 192.168.0.0/20
eth1 192.168.16.1 - subnetwork 192.168.16.0/28, with a router
(192.168.16.2) to internet
eth2 192.168.17.1 - subnetwork 192.168.17.0/28, with another router
(192.168.17.2) to internet

The routing rules are:
0:      from all lookup local
50:     from all lookup main
201:    from 192.168.17.0/28 iif eth2 lookup 201
202:    from 192.168.16.0/28 iif eth1 lookup 202
222:    from all lookup 222
32766:  from all lookup main
32767:  from all lookup default

The table 'main':
192.168.17.0/28 dev eth2  proto kernel  scope link  src 192.168.17.1
192.168.16.0/28 dev eth1  proto kernel  scope link  src 192.168.16.1
192.168.0.0/20 dev eth0  proto kernel  scope link  src 192.168.1.2

The table '201':
default via 192.168.17.2 dev eth2  proto static  src 192.168.17.1
prohibit default  proto static  metric 1

The table '202':
default via 192.168.16.2 dev eth1  proto static  src 192.168.16.1
prohibit default  proto static  metric 1

The table '222', where there is the multihop gateway specification:

default equalize
        nexthop via 192.168.16.2  dev eth1 weight 1
        nexthop via 192.168.17.2  dev eth2 weight 1

I've added the following packet LOG lines into 'mangle' table, for
knowing when the "WRONG INTERFACE" decision is being made:
Chain POSTROUTING (policy ACCEPT 329K packets, 93M bytes)
 pkts bytes target     prot opt in     out     source
destination
    2    80 LOG        all  --  any    eth1    192.168.17.1
anywhere            LOG level warning ip-options prefix `WRONG IFACE:
'
    0     0 LOG        all  --  any    eth2    192.168.16.1
anywhere            LOG level warning ip-options prefix `WRONG IFACE:
'

(Don't look at counters; right now, for getting good internet access,
I'm not using multihop)

So, often appears in the kernel log, specially with 'ftp' and 'ssh'
connections (and rarely with www connections):
Jul  4 15:50:14 thecrow WRONG IFACE: IN= OUT=eth2 SRC=192.168.16.1
DST=216.165.191.52 LE
N=72 TOS=0x00 PREC=0x00 TTL=64 ID=9582 DF PROTO=TCP SPT=56528 DPT=6667
WINDOW=18824 RES=
0x00 ACK PSH URGP=0
Jul  4 16:01:29 thecrow WRONG IFACE: IN= OUT=eth1 SRC=192.168.17.1
DST=130.206.1.5 LEN=40 TOS=0x00 PREC=0x00 TTL=64 ID=66 DF PROTO=TCP
SPT=33820 DPT=21 WINDOW=0 RES=0x00 RST URGP=0

Even though, when I use 'tcpdump' for catching the wrong packets (that
is: tcpdump -i eth1 host 192.168.17.1  _or_ tcpdump -i eth2 host
192.168.16.1 ) results that _A LOT MORE PACKETS_ are BADLY ROUTED,
than sent to the LOG target.

My conclusion: iptables 'matching' doesn't work; also does the route
decision part.

I absolutely don't know what more to do... I'm running iptables
v1.2.11, and kernel 2.6.11-gentoo-r11. Exactly same happened with
kernel 2.4.28-gentoo. :(

For example, here is a test. I want to ftp to "ftp.rediris.es". I look
which would be the route:
# ip route get 130.206.1.5
130.206.1.5 via 192.168.17.2 dev eth2  src 192.168.17.1 
    cache  mtu 1500 advmss 1460 metric10 64
I try the ftp:
# ftp ftp.rediris.es
And in the kernel log appears:
Jul  4 16:19:25 thecrow WRONG IFACE: IN= OUT=eth1 SRC=192.168.17.1
DST=130.206.1.5 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=43245 DF PROTO=TCP
SPT=49828 DPT=21 WINDOW=5840 RES=0x00 SYN URGP=0


Please, help... I'm desperate.

Re: Problem with routing decisions, and multihop

[/dev/rob0]

On Monday 04 July 2005 09:32, Lluis Batle wrote:
> I have many problems getting this thing to work. There's a host with
> two network interfaces, where there are two routers to Internet in
> two separated networks. The host uses multihop routing for deciding
> to which router send the packets... but the routing decision is wrong

Does your kernel have the routes patch by Julian Anastasov?
http://www.ssi.bg/~ja/

> Please, help... I'm desperate.

You'll be more likely to get help on the LARTC mailing list.

Why RFC 1918 addresses on these interfaces? Do you have extra router 
equipment between you and the Internet?

Are multiple "ip route get ip.add.re.ss" commands in sequence showing 
routes out the same interface?
-- 
    mail to this address is discarded unless "/dev/rob0"
    or "not-spam" is in Subject: header

Re: Problem with routing decisions, and multihop

[/dev/rob0]

On Monday 04 July 2005 10:07, I wrote:
> Are multiple "ip route get ip.add.re.ss" commands in sequence showing
> routes out the same interface?

for X in `seq 10` ; do /sbin/ip route get $X.$X.$X.$X ; done
-- 
    mail to this address is discarded unless "/dev/rob0"
    or "not-spam" is in Subject: header

Problem with routing decisions, and multihop

[Lluis Batle]

No, everything is right in its output:
1.1.1.1 via 192.168.16.2 dev eth1  src 192.168.16.1
    cache  mtu 1500 advmss 1460 metric10 64
2.2.2.2 via 192.168.17.2 dev eth2  src 192.168.17.1
    cache  mtu 1500 advmss 1460 metric10 64
3.3.3.3 via 192.168.16.2 dev eth1  src 192.168.16.1
    cache  mtu 1500 advmss 1460 metric10 64
4.4.4.4 via 192.168.17.2 dev eth2  src 192.168.17.1
    cache  mtu 1500 advmss 1460 metric10 64
5.5.5.5 via 192.168.16.2 dev eth1  src 192.168.16.1
    cache  mtu 1500 advmss 1460 metric10 64
6.6.6.6 via 192.168.17.2 dev eth2  src 192.168.17.1
    cache  mtu 1500 advmss 1460 metric10 64
7.7.7.7 via 192.168.16.2 dev eth1  src 192.168.16.1
    cache  mtu 1500 advmss 1460 metric10 64
8.8.8.8 via 192.168.17.2 dev eth2  src 192.168.17.1
    cache  mtu 1500 advmss 1460 metric10 64
9.9.9.9 via 192.168.16.2 dev eth1  src 192.168.16.1
    cache  mtu 1500 advmss 1460 metric10 64
10.10.10.10 via 192.168.17.2 dev eth2  src 192.168.17.1
    cache  mtu 1500 advmss 1460 metric10 64

About the 16.x and 17.x addresses... yes, there are other routers,
which make NAT (192.168.16.2 and 192.168.17.2) to internet.

On 7/4/05, /dev/rob0 <rob0@gmx.co.uk> wrote:
> On Monday 04 July 2005 10:07, I wrote:
> > Are multiple "ip route get ip.add.re.ss" commands in sequence showing
> > routes out the same interface?
>
> for X in `seq 10` ; do /sbin/ip route get $X.$X.$X.$X ; done
> --
>     mail to this address is discarded unless "/dev/rob0"
>     or "not-spam" is in Subject: header
>
>

Re: Problem with routing decisions, and multihop

[/dev/rob0]

> On 7/4/05, /dev/rob0 <rob0@gmx.co.uk> wrote:
> > On Monday 04 July 2005 10:07, I wrote:
> > > Are multiple "ip route get ip.add.re.ss" commands in sequence
> > > showing routes out the same interface?
> >
> > for X in `seq 10` ; do /sbin/ip route get $X.$X.$X.$X ; done

On Monday 04 July 2005 10:32, Lluis Batle wrote:
> No, everything is right in its output:
> 1.1.1.1 via 192.168.16.2 dev eth1  src 192.168.16.1
>     cache  mtu 1500 advmss 1460 metric10 64
> 2.2.2.2 via 192.168.17.2 dev eth2  src 192.168.17.1
>     cache  mtu 1500 advmss 1460 metric10 64

>>="masquerading.multi-eth" (misnamed: it does no masquerading)

>>NE1=192.168.16.0/28
>>NE2=192.168.17.0/28

Let's see, those are .0-.15 on the last quad.

>>NLOCAL=192.168.0.0/20

And this is 0.0 through 15.255 ... IOW, wrong, excluding both $NE1 and 
$NE2. Try 192.168.16.0/23. It would not hurt for you to brush up on 
TCP/IP and subnetting basics.

>> $IPTABLES -t nat -F PREROUTING
>> $IPTABLES -t nat -F POSTROUTING
>> $IPTABLES -t nat -F OUTPUT
>> $IPTABLES -t filter -F INPUT
>> $IPTABLES -t filter -F FORWARD
>> $IPTABLES -t filter -F OUTPUT
>> $IPTABLES -t filter -F keep_state >&/dev/null
>> $IPTABLES -t filter -X keep_state >&/dev/null
>> $IPTABLES -t nat -F keep_state >&/dev/null
>> $IPTABLES -t nat -X keep_state >&/dev/null

Could be rewritten as:
iptables -F ; iptables -X ; iptables -t nat -F ; iptables -t nat -X

>> $IPTABLES -t filter -N keep_state
>> $IPTABLES -t filter -A keep_state -m state \
>> --state RELATED,ESTABLISHED -j ACCEPT  
>> $IPTABLES -t filter -A keep_state -j RETURN
>> 
>> $IPTABLES -t nat -N keep_state
>> $IPTABLES -t nat -A keep_state -m state \
>> --state RELATED,ESTABLISHED -j ACCEPT 
>> $IPTABLES -t nat -A keep_state -j RETURN

1. IMO it's confusing to give chains the same name in different tables.
2. The RETURN rules are pointless. That's what happens at the end of a 
chain, anyway.
3. --state in -t nat? Is that possible? Does it work? Does it break 
anything?

> About the 16.x and 17.x addresses... yes, there are other routers,
> which make NAT (192.168.16.2 and 192.168.17.2) to internet.

This seems odd to me. I prefer to use external IP directly, for many 
reasons. It also eliminates other potential points of failure.

It's even more odd considering that you're doing DNAT on the already- 
NAT'ed Linux machine. Why not do the DNAT in the external routers? 
Also, those DNAT rules refer to other RFC 1918 netblocks.
-- 
    mail to this address is discarded unless "/dev/rob0"
    or "not-spam" is in Subject: header

Re: Problem with routing decisions, and multihop

[Lluís Batlle]

Thanks :) I answer between lines...

On 7/4/05, /dev/rob0 <rob0@gmx.co.uk> wrote:
 > >>="masquerading.multi-eth" (misnamed: it does no masquerading)
Ok. I tried with MASQUERADE, but by now I use SNAT.
> 
> >>NE1=192.168.16.0/28
> >>NE2=192.168.17.0/28
> 
> Let's see, those are .0-.15 on the last quad.
> 
> >>NLOCAL=192.168.0.0/20
> 
> And this is 0.0 through 15.255 ... IOW, wrong, excluding both $NE1 and
> $NE2. Try 192.168.16.0/23. It would not hurt for you to brush up on
> TCP/IP and subnetting basics.

Oh. Is it wrong? I don't understand what's "IOW". Where should I try
your proposed subnet? why?

> 
> >> $IPTABLES -t nat -F PREROUTING
> >> $IPTABLES -t nat -F POSTROUTING
> >> $IPTABLES -t nat -F OUTPUT
> >> $IPTABLES -t filter -F INPUT
> >> $IPTABLES -t filter -F FORWARD
> >> $IPTABLES -t filter -F OUTPUT
> >> $IPTABLES -t filter -F keep_state >&/dev/null
> >> $IPTABLES -t filter -X keep_state >&/dev/null
> >> $IPTABLES -t nat -F keep_state >&/dev/null
> >> $IPTABLES -t nat -X keep_state >&/dev/null
> 
> Could be rewritten as:
> iptables -F ; iptables -X ; iptables -t nat -F ; iptables -t nat -X
Ok :)

> 
> >> $IPTABLES -t filter -N keep_state
> >> $IPTABLES -t filter -A keep_state -m state \
> >> --state RELATED,ESTABLISHED -j ACCEPT
> >> $IPTABLES -t filter -A keep_state -j RETURN
> >>
> >> $IPTABLES -t nat -N keep_state
> >> $IPTABLES -t nat -A keep_state -m state \
> >> --state RELATED,ESTABLISHED -j ACCEPT
> >> $IPTABLES -t nat -A keep_state -j RETURN
> 
> 1. IMO it's confusing to give chains the same name in different tables.
I agree... but by now does that matter?
> 2. The RETURN rules are pointless. That's what happens at the end of a
> chain, anyway.
Ok. That's what I thought
> 3. --state in -t nat? Is that possible? Does it work? Does it break
> anything?

It seems it's possible. I get no error from those commands. Anyway,
I've thought that happens double application of that rule, through
filter and nat tables. I've removed everything about 'keep_state' in
the nat table. Everything is still working bad. Even from the computer
itself (local processes). Routing doesn't work, even without any
"NAT"-related chain/rule.

> 
> > About the 16.x and 17.x addresses... yes, there are other routers,
> > which make NAT (192.168.16.2 and 192.168.17.2) to internet.
> 
> This seems odd to me. I prefer to use external IP directly, for many
> reasons. It also eliminates other potential points of failure.
I agree :) but it's hard for me to configure the routers, so they give
to me the public addresses. I think that double-NAT should not break
anything. :)
> 
> It's even more odd considering that you're doing DNAT on the already-
> NAT'ed Linux machine. Why not do the DNAT in the external routers?
> Also, those DNAT rules refer to other RFC 1918 netblocks.
mmm I've never read RFC 1918. :) I'll take a look at it.
Thanks!

- I finish the working time in few minutes. Tomorrow morning I'll keep
on with this.

Re: Problem with routing decisions, and multihop

[/dev/rob0]

On Monday 04 July 2005 11:54, Lluís Batlle wrote:
> Thanks :) I answer between lines...

Thank you.

> On 7/4/05, /dev/rob0 <rob0@gmx.co.uk> wrote:
>  > >>="masquerading.multi-eth" (misnamed: it does no masquerading)
>
> Ok. I tried with MASQUERADE, but by now I use SNAT.

Right. MASQUERADE will not work with multiple routing.

> > >>NE1=192.168.16.0/28
> > >>NE2=192.168.17.0/28
> >
> > Let's see, those are .0-.15 on the last quad.
> >
> > >>NLOCAL=192.168.0.0/20
> >
> > And this is 0.0 through 15.255 ... IOW, wrong, excluding both $NE1
> > and $NE2. Try 192.168.16.0/23. It would not hurt for you to brush
> > up on TCP/IP and subnetting basics.
>
> Oh. Is it wrong? I don't understand what's "IOW". Where should I try
> your proposed subnet? why?

IOW="in other words", a common Internet shorthand.

192.168.0.0/20, set as $NLOCAL in your iptables script, excludes your
IP addresses and networks. No packet hitting the rules which refer to 
that value will match, so the rules are ignored.

The rules to which I am referring:
$IPTABLES -t nat -A POSTROUTING -o eth1 -s $NLOCAL -j SNAT --to $IPE1
$IPTABLES -t nat -A POSTROUTING -o eth2 -s $NLOCAL -j SNAT --to $IPE2
Your SNAT rules.

Change "NLOCAL=192.168.0.0/20" to "NLOCAL=192.168.0.0/16", or as 
previously suggested, "NLOCAL=192.168.16.0/23". I suppose you could 
even omit the source specification altogether:
$IPTABLES -t nat -A POSTROUTING -o eth1 -j SNAT --to $IPE1
$IPTABLES -t nat -A POSTROUTING -o eth2 -j SNAT --to $IPE2

> > 1. IMO it's confusing to give chains the same name in different
> > tables.
>
> I agree... but by now does that matter?

Simply a point of style. You can give chains any names you wish, no 
matter how confusing they might be in context:

###  Kids, don't try this at home. Professional stunt driver on a
###  closed track.
iptables -N InputLogDrop
iptables -N ForwardAllow
iptables -A InputLogDrop -j ACCEPT
iptables -A FORWARD -j InputLogDrop
iptables -A ForwardAllow -j LOG
iptables -A ForwardAllow -p tcp -j REJECT
iptables -A ForwardAllow -j DROP
iptables -A INPUT -j ForwardAllow
###  For my next trick, I will campaign to be elected Prime Minister.
###  Thank you for your support in the polls.

> > 3. --state in -t nat? Is that possible? Does it work? Does it break
> > anything?
>
> It seems it's possible. I get no error from those commands. Anyway,

Perhaps it doesn't break anything, but I have read here that only 
packets of --state NEW hit the -t nat PREROUTING chain. I don't know 
about the relationship between connection tracking and NAT.

> I've thought that happens double application of that rule, through
> filter and nat tables. I've removed everything about 'keep_state' in
> the nat table. Everything is still working bad. Even from the

Likely because of NLOCAL in your script. If that's not the case it's 
beyond my limited understanding, and once again I'll suggest you take 
it to LARTC. Some people in LARTC know more about this than I do.

> > already-NAT'ed Linux machine. Why not do the DNAT in the external
> > routers? Also, those DNAT rules refer to other RFC 1918 netblocks.
>
> mmm I've never read RFC 1918. :) I'll take a look at it.

"RFC 1918 netblocks" is simply another form of shorthand to refer to 
IPv4 ranges which are reserved for private use, namely 10.0.0.0/8, 
172.16.0.0/12, and 192.168.0.0/16. I rarely read RFC's myself (but I 
must confess to a fondness for RFC 1149. :) )
-- 
    mail to this address is discarded unless "/dev/rob0"
    or "not-spam" is in Subject: header

Re: Problem with routing decisions, and multihop

[Lluís Batlle]

Again between lines...
On 7/4/05, /dev/rob0 <rob0@gmx.co.uk> wrote:
> On Monday 04 July 2005 11:54, Lluís Batlle wrote:
> > > >>NE1=192.168.16.0/28
> > > >>NE2=192.168.17.0/28
> > >
> > > Let's see, those are .0-.15 on the last quad.
> > >
> > > >>NLOCAL=192.168.0.0/20
> > >
> > > And this is 0.0 through 15.255 ... IOW, wrong, excluding both $NE1
> > > and $NE2. Try 192.168.16.0/23. It would not hurt for you to brush
> > > up on TCP/IP and subnetting basics.
> >
> > Oh. Is it wrong? I don't understand what's "IOW". Where should I try
> > your proposed subnet? why?
> 
> IOW="in other words", a common Internet shorthand.
> 
> 192.168.0.0/20, set as $NLOCAL in your iptables script, excludes your
> IP addresses and networks. No packet hitting the rules which refer to
> that value will match, so the rules are ignored.
Why? in the LAN (eth0, 192.168.0.0/20) there are many computers... if
I change it to 192.168.0.0/16, eth1 and eth2 _won't_ be appart
subnetworks! It's important to them to be excluded.
IOW, there must be no intersection between the networks of the different NICs.
> 
> The rules to which I am referring:
> $IPTABLES -t nat -A POSTROUTING -o eth1 -s $NLOCAL -j SNAT --to $IPE1
> $IPTABLES -t nat -A POSTROUTING -o eth2 -s $NLOCAL -j SNAT --to $IPE2
> Your SNAT rules.
> 
> Change "NLOCAL=192.168.0.0/20" to "NLOCAL=192.168.0.0/16", or as
> previously suggested, "NLOCAL=192.168.16.0/23". I suppose you could
> even omit the source specification altogether:
> $IPTABLES -t nat -A POSTROUTING -o eth1 -j SNAT --to $IPE1
> $IPTABLES -t nat -A POSTROUTING -o eth2 -j SNAT --to $IPE2
Will, that way, the kernel maintain connection-tables for SNAT even
for local connections?
> ###  Kids, don't try this at home. Professional stunt driver on a
> ###  closed track.
> iptables -N InputLogDrop
> iptables -N ForwardAllow
> iptables -A InputLogDrop -j ACCEPT
> iptables -A FORWARD -j InputLogDrop
> iptables -A ForwardAllow -j LOG
> iptables -A ForwardAllow -p tcp -j REJECT
> iptables -A ForwardAllow -j DROP
> iptables -A INPUT -j ForwardAllow
> ###  For my next trick, I will campaign to be elected Prime Minister.
> ###  Thank you for your support in the polls.
:)))
> Perhaps it doesn't break anything, but I have read here that only
> packets of --state NEW hit the -t nat PREROUTING chain. I don't know
> about the relationship between connection tracking and NAT.
Can you give a link about that?
> 
> "RFC 1918 netblocks" is simply another form of shorthand to refer to
> IPv4 ranges which are reserved for private use, namely 10.0.0.0/8,
> 172.16.0.0/12, and 192.168.0.0/16. I rarely read RFC's myself (but I
> must confess to a fondness for RFC 1149. :) )
Hahaha :)

Re: Problem with routing decisions, and multihop (solved)

[Lluís Batlle]

Argh. Stupid me..............
The tables 201 and 202, who decide the route for packets with already
defined source addres were in the rules that way:
The routing rules are:
0:      from all lookup local
50:     from all lookup main
201:    from 192.168.17.0/28 iif eth2 lookup 201
202:    from 192.168.16.0/28 iif eth1 lookup 202
222:    from all lookup 222
32766:  from all lookup main
32767:  from all lookup default

Look at the terrible "iif ethx"!!!!!!!!! The packets were routed _at
random_ by the 'equalizing routing decision', even if with decided
address. Argh.
Solution: there should not be "if ethx" in the routing rule matching

Now it works.

Even though, I'd prefer to get the Julian's patches working. When I
try them, all packets go to the same interface (the first hop). I'll
keep on working...

Always, always, there is the deadly "user fault". :)))

Thanks for your support!