Changing default route causes packet drop

Changing default route causes packet drop

[John Meissen]

I'm not sure if this is the right place to ask, or if it's even the right
question. Hopefully someone can point me in the right direction.

I had a traditional setup with two ethernet interfaces on my Linux box 
(WAN=eth0/LAN=eth1), and NATing the traffic that was forwarded between them.

I added another interface (eth2), and simply want to change the default
routing to go through it. I'm leaving various services listening on all
interfaces.

If I change the default route to use eth2, I can route from the internal
network to the outside just fine, and I can connect from the internal net
to services on the system fine. But incoming connections on the original
WAN (eth0) don't complete. They hang at SYN_RECV, as if I had a DROP rule.

I.e., what used to be

  internal <-> (eth1) gateway forward (eth0) <-> WAN
  internal <-> (eth1) gateway local service
                gateway local service (eth0) <-> WAN
is now

  internal <-> (eth1) gateway forward (eth2) <-> WAN
  internal <-> (eth1) gateway local service

but
                gateway local service (eth0) <-> WAN

now drops connection attempts.

I don't see what difference there should be between eth0 and eth1, except
that eth0 isn't forwarded. That shouldn't affect connections to processes
listening on that interface.

I've tried to keep the iptables config simple for this. The only change I'm
making is changing the default route with the 'route' command.

# iptables -L -v -n
Chain INPUT (policy ACCEPT 63555 packets, 73M bytes)
 pkts bytes target     prot opt in     out     source               destination 

   11  3626 ACCEPT     udp  --  eth1   *       0.0.0.0/0            0.0.0.0/0   
        udp spt:68 dpt:67
    0     0 ACCEPT     tcp  --  eth1   *       0.0.0.0/0            0.0.0.0/0   
        tcp spt:68 dpt:67
    0     0 ACCEPT     udp  --  eth1   *       0.0.0.0/0            0.0.0.0/0   
        udp spt:67 dpt:68
    0     0 ACCEPT     tcp  --  eth1   *       0.0.0.0/0            0.0.0.0/0   
        tcp spt:67 dpt:68
 1937  127K ACCEPT     udp  --  eth1   *       0.0.0.0/0            0.0.0.0/0   
        udp dpt:53
    0     0 ACCEPT     tcp  --  eth1   *       0.0.0.0/0            0.0.0.0/0   
        tcp dpt:53

Chain FORWARD (policy ACCEPT 39362 packets, 42M bytes)
 pkts bytes target     prot opt in     out     source               destination 

31533 2844K ACCEPT     all  --  *      *       192.168.10.0/24      0.0.0.0/0   


Chain OUTPUT (policy ACCEPT 42150 packets, 5745K bytes)
 pkts bytes target     prot opt in     out     source               destination 


and

# iptables -t nat -L -v -n
Chain PREROUTING (policy ACCEPT 859K packets, 57M bytes)
 pkts bytes target     prot opt in     out     source               destination 


Chain POSTROUTING (policy ACCEPT 584K packets, 46M bytes)
 pkts bytes target     prot opt in     out     source               destination 

 755K   72M MASQUERADE  all  --  *      *       192.168.10.0/24      0.0.0.0/0  


Chain OUTPUT (policy ACCEPT 1015K packets, 100M bytes)
 pkts bytes target     prot opt in     out     source               destination 

Re: Changing default route causes packet drop

[Gáspár Lajos]

Hi John,

1. Set up multiple routing tables.

a.) I have the following in my /etc/iproute2/rt_tables: [cat 
/etc/iproute/rt_tables]

#
# reserved values
#
255    local
254    main
253    default
0    unspec
#
# local
#
#1    inr.ruhep
201    PPP2
200    PPP1


b.) I have a route setup script: [cat /etc/network/routes]

#!/bin/bash

WAN1_IF='ppp1'
WAN1_TB='PPP1'
WAN1_MARK='1'
WAN1_IP=`ip addr show dev $WAN1_IF | grep 'inet ' | awk '{print $2}' | 
awk 'BEGIN{FS="/"}{print $1}'`
WAN1_GW=`ip addr show dev $WAN1_IF | grep 'inet ' | awk '{print $4}' | 
awk 'BEGIN{FS="/"}{print $1}'`

WAN2_IF='ppp2'
WAN2_TB='PPP2'
WAN2_MARK='2'
WAN2_IP=`ip addr show dev $WAN2_IF | grep 'inet ' | awk '{print $2}' | 
awk 'BEGIN{FS="/"}{print $1}'`
WAN2_GW=`ip addr show dev $WAN2_IF | grep 'inet ' | awk '{print $4}' | 
awk 'BEGIN{FS="/"}{print $1}'`

ip route flush table $WAN1_TB
ip route flush table $WAN2_TB

test ! "$WAN1_IP" == "" && ip route add table $WAN1_TB dev $WAN1_IF 
default via $WAN1_GW src $WAN1_IP
test ! "$WAN2_IP" == "" && ip route add table $WAN2_TB dev $WAN2_IF 
default via $WAN2_GW src $WAN2_IP

for prio in `ip rule show | grep $WAN1_TB | awk 'BEGIN{FS=":"}{print $1}'`
  do
  ip rule del prio $prio
  done
for prio in `ip rule show | grep $WAN2_TB | awk 'BEGIN{FS=":"}{print $1}'`
  do
  ip rule del prio $prio
  done

test ! "$WAN2_IP" == "" && ip rule add fwmark $WAN1_MARK table $WAN1_TB
test ! "$WAN2_IP" == "" && ip rule add fwmark $WAN2_MARK table $WAN2_TB

test ! "$WAN1_IP" == "" && ip rule add from $WAN1_IP table $WAN1_TB
test ! "$WAN2_IP" == "" && ip rule add from $WAN2_IP table $WAN2_TB

test -e /proc/sys/net/ipv4/conf/$WAN1_IF/rp_filter && echo '0' 
 >/proc/sys/net/ipv4/conf/$WAN1_IF/rp_filter
test -e /proc/sys/net/ipv4/conf/$WAN2_IF/rp_filter && echo '0' 
 >/proc/sys/net/ipv4/conf/$WAN2_IF/rp_filter

ip route del default
ip route add default dev $WAN1_IF scope link

ip route flush cache

exit 0

c.) Call this script whenever a WAN interface is coming up.

In my /etc/interfaces:

auto adsl1
iface adsl1 inet ppp
    provider PPP1
    up /bin/sleep 10
    up /etc/network/routes

auto adsl2
iface adsl2 inet ppp
    provider PPP2
    up /bin/sleep 10
    up /etc/network/routes

2. Do the Netfilter/Iptables part:

Mark the outgoing packets in the mangle table's POSTROUTING chain with 
WAN1_MARK or WAN2_MARK:
iptables -t mangle -A POSTROUTING -j MARK --set-mark 1 .... (your 
matching criteria for WAN1....)
iptables -t mangle -A POSTROUTING -j MARK --set-mark 2 .... (your 
matching criteria for WAN2....)


Hope I could help:

  Swifty

2010-07-05 11:03 keltezéssel, John Meissen írta:
> I'm not sure if this is the right place to ask, or if it's even the right
> question. Hopefully someone can point me in the right direction.
>
> I had a traditional setup with two ethernet interfaces on my Linux box
> (WAN=eth0/LAN=eth1), and NATing the traffic that was forwarded between them.
>
> I added another interface (eth2), and simply want to change the default
> routing to go through it. I'm leaving various services listening on all
> interfaces.
>
> If I change the default route to use eth2, I can route from the internal
> network to the outside just fine, and I can connect from the internal net
> to services on the system fine. But incoming connections on the original
> WAN (eth0) don't complete. They hang at SYN_RECV, as if I had a DROP rule.
>
> I.e., what used to be
>
>    internal<->  (eth1) gateway forward (eth0)<->  WAN
>    internal<->  (eth1) gateway local service
>                  gateway local service (eth0)<->  WAN
> is now
>
>    internal<->  (eth1) gateway forward (eth2)<->  WAN
>    internal<->  (eth1) gateway local service
>
> but
>                  gateway local service (eth0)<->  WAN
>
> now drops connection attempts.
>
> I don't see what difference there should be between eth0 and eth1, except
> that eth0 isn't forwarded. That shouldn't affect connections to processes
> listening on that interface.
>
> I've tried to keep the iptables config simple for this. The only change I'm
> making is changing the default route with the 'route' command.
>
> # iptables -L -v -n
> Chain INPUT (policy ACCEPT 63555 packets, 73M bytes)
>   pkts bytes target     prot opt in     out     source               destination
>
>     11  3626 ACCEPT     udp  --  eth1   *       0.0.0.0/0            0.0.0.0/0
>          udp spt:68 dpt:67
>      0     0 ACCEPT     tcp  --  eth1   *       0.0.0.0/0            0.0.0.0/0
>          tcp spt:68 dpt:67
>      0     0 ACCEPT     udp  --  eth1   *       0.0.0.0/0            0.0.0.0/0
>          udp spt:67 dpt:68
>      0     0 ACCEPT     tcp  --  eth1   *       0.0.0.0/0            0.0.0.0/0
>          tcp spt:67 dpt:68
>   1937  127K ACCEPT     udp  --  eth1   *       0.0.0.0/0            0.0.0.0/0
>          udp dpt:53
>      0     0 ACCEPT     tcp  --  eth1   *       0.0.0.0/0            0.0.0.0/0
>          tcp dpt:53
>
> Chain FORWARD (policy ACCEPT 39362 packets, 42M bytes)
>   pkts bytes target     prot opt in     out     source               destination
>
> 31533 2844K ACCEPT     all  --  *      *       192.168.10.0/24      0.0.0.0/0
>
>
> Chain OUTPUT (policy ACCEPT 42150 packets, 5745K bytes)
>   pkts bytes target     prot opt in     out     source               destination
>
>
> and
>
> # iptables -t nat -L -v -n
> Chain PREROUTING (policy ACCEPT 859K packets, 57M bytes)
>   pkts bytes target     prot opt in     out     source               destination
>
>
> Chain POSTROUTING (policy ACCEPT 584K packets, 46M bytes)
>   pkts bytes target     prot opt in     out     source               destination
>
>   755K   72M MASQUERADE  all  --  *      *       192.168.10.0/24      0.0.0.0/0
>
>
> Chain OUTPUT (policy ACCEPT 1015K packets, 100M bytes)
>   pkts bytes target     prot opt in     out     source               destination
>
>
>
> --
> To unsubscribe from this list: send the line "unsubscribe netfilter" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
>    

Re: Changing default route causes packet drop

[Pascal Hambourg]

Hello,

John Meissen a écrit :
> 
> I had a traditional setup with two ethernet interfaces on my Linux box 
> (WAN=eth0/LAN=eth1), and NATing the traffic that was forwarded between them.
> 
> I added another interface (eth2), and simply want to change the default
> routing to go through it. I'm leaving various services listening on all
> interfaces.
> 
> If I change the default route to use eth2, I can route from the internal
> network to the outside just fine, and I can connect from the internal net
> to services on the system fine. But incoming connections on the original
> WAN (eth0) don't complete. They hang at SYN_RECV, as if I had a DROP rule.

1) Check that source validation by reverse path is disabled for eth0
(sysctl net.ipv4.conf.{all,eth0}.rp_filter=0).

2) If you don't setup some routing policy (such as source address based
routing), packets sent in reply to packets received on eth0 will now be
sent through eth2 by default because of the new default route, but still
with the source address of eth0. Such traffic may be considered as
spoofing and discarded by the ISP eth2 is connected to.

Re: Changing default route causes packet drop

[John Meissen]

> Hello,
> 
> John Meissen a écrit :
> > 
> > I had a traditional setup with two ethernet interfaces on my Linux box 
> > (WAN=eth0/LAN=eth1), and NATing the traffic that was forwarded between them.
> > 
> > I added another interface (eth2), and simply want to change the default
> > routing to go through it. I'm leaving various services listening on all
> > interfaces.
> > 
> > If I change the default route to use eth2, I can route from the internal
> > network to the outside just fine, and I can connect from the internal net
> > to services on the system fine. But incoming connections on the original
> > WAN (eth0) don't complete. They hang at SYN_RECV, as if I had a DROP rule.
> 
> 1) Check that source validation by reverse path is disabled for eth0
> (sysctl net.ipv4.conf.{all,eth0}.rp_filter=0).
> 
> 2) If you don't setup some routing policy (such as source address based
> routing), packets sent in reply to packets received on eth0 will now be
> sent through eth2 by default because of the new default route, but still
> with the source address of eth0. Such traffic may be considered as
> spoofing and discarded by the ISP eth2 is connected to.

Yes, I should learn to not post to mailing lists at 3AM, that a good night's
sleep is generally better for solving problems. :-P

I realized I was thinking of the problem in terms of interfaces, not routing.
Once I slept on it I realized the problem was 2), and that I couldn't really
do what I was proposing. Relocating the new connection and making a minor
change to the DHCP server to specify the new default route for the rest of
the network solved the problem.

Thanks.

john-