load balancing between ppp0 and ppp1, private IPs

load balancing between ppp0 and ppp1, private IPs

[Lloyd Standish]

Hello,

I want to use load balancing to allow simultaneous use of 2 relatively slow GPRS (ppp over GSM cellular phone) modems.  I have a custom-compiled Ubuntu kernel 2.6.20.3 with Debian Etch, iptables v1.3.8.

The modems are seen by the system as regular serial modems, ppp0 and ppp1.  I also have a LAN (eth0). 

My interfaces, ppp0 and ppp1 have dynamically assigned *private* IP numbers.  For example:

ppp0: 10.60.32.71
ppp1: 10.40.89.42

My strategy for load balancing is to use conntrack to save the mark of the *first* packet, and mark all subsequent packets for that connection the same way.  Two custom routing tables are set up, each set to push traffic to one of the 2 interfaces, according to the packet mark:

TABLE    INTERFACE    MARK
rt_link1   ppp0                1
rt_link2   ppp1                2

My efforts to do load balancing between the 2 interfaces have not been successful.  Before I go into details, a major question:  Will load balancing work without public, routable IP numbers for my outward-facing interfaces?

--
Lloyd

Re: load balancing between ppp0 and ppp1, private IPs

[Brian Austin - Standard Universal]

yes, it will.

b


Lloyd Standish wrote:
> Hello,
>
> I want to use load balancing to allow simultaneous use of 2 relatively slow GPRS (ppp over GSM cellular phone) modems.  I have a custom-compiled Ubuntu kernel 2.6.20.3 with Debian Etch, iptables v1.3.8.
>
> The modems are seen by the system as regular serial modems, ppp0 and ppp1.  I also have a LAN (eth0). 
>
> My interfaces, ppp0 and ppp1 have dynamically assigned *private* IP numbers.  For example:
>
> ppp0: 10.60.32.71
> ppp1: 10.40.89.42
>
> My strategy for load balancing is to use conntrack to save the mark of the *first* packet, and mark all subsequent packets for that connection the same way.  Two custom routing tables are set up, each set to push traffic to one of the 2 interfaces, according to the packet mark:
>
> TABLE    INTERFACE    MARK
> rt_link1   ppp0                1
> rt_link2   ppp1                2
>
> My efforts to do load balancing between the 2 interfaces have not been successful.  Before I go into details, a major question:  Will load balancing work without public, routable IP numbers for my outward-facing interfaces?
>
> --
> Lloyd
>
>
> --
> To unsubscribe from this list: send the line "unsubscribe netfilter" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>   

Re: load balancing between ppp0 and ppp1, private IPs

[Lloyd Standish]

Hello,

To review, in my previous post I wrote:

>> I want to use load balancing to allow simultaneous use of 2 relatively slow GPRS (ppp over GSM cellular phone) modems.  I have a custom-compiled Ubuntu kernel 2.6.20.3 with Debian Etch, iptables v1.3.8.
>>
>> The modems are seen by the system as regular serial modems, ppp0 and ppp1.  I also have a LAN (eth0).
>>
>> My interfaces, ppp0 and ppp1 have dynamically assigned *private* IP numbers.  For example:
>>
>> ppp0: 10.60.32.71
>> ppp1: 10.40.89.42
>>
>> My strategy for load balancing is to use conntrack to save the mark of the *first* packet, and mark all subsequent packets for that connection the same way.  Two custom routing tables are set up, each set to push traffic to one of the 2 interfaces, according to the packet mark:
>>
>> TABLE    INTERFACE    MARK
>> rt_link1   ppp0                1
>> rt_link2   ppp1                2
>>

Using cat /proc/net/ip_conntrack, my tcp connections show up duly marked with 1 and 2:

tcp      6 25 TIME_WAIT src=10.60.32.71 dst=192.16.204.139 sport=36753 dport=80 packets=5 bytes=928 src=192.16.204.139 dst=10.60.32.71 sport=80 dport=36753 packets=5 bytes=400 [ASSURED] mark=1 secmark=0 use=1
tcp      6 431999 ESTABLISHED src=10.60.32.71 dst=72.249.38.123 sport=33089 dport=80 packets=235 bytes=15019 src=72.249.38.123 dst=10.60.32.71 sport=80 dport=33089 packets=238 bytes=354112 [ASSURED] mark=2 secmark=0 use=1
tcp      6 431929 ESTABLISHED src=10.60.32.71 dst=207.171.166.176 sport=60476 dport=80 packets=15 bytes=1697 src=207.171.166.176 dst=10.60.32.71 sport=80 dport=60476 packets=14 bytes=14607 [ASSURED] mark=2 secmark=0 use=1

PROBLEM: as you can see, all the connections are being sent to 10.60.32.71, which is ppp0 (my default route).  The connections with the "2" mark should be sent through ppp1 (10.40.89.42)  (Note that these IP numbers are examples only and are dynamically assigned upon dialup.

This is confirmed by netstat:
debiandesk:/home/lloyd#  netstat -i
Kernel Interface table
Iface   MTU Met   RX-OK RX-ERR RX-DRP RX-OVR    TX-OK TX-ERR TX-DRP TX-OVR Flg
eth0   1500 0         0      0      0      0     7221      0      0      0 BMRU
lo    16436 0      1674      0      0      0     1674      0      0      0 LRU
ppp0   1500 0     19505      0      0      0    19830      0      0      0 MOPRU
ppp1   1500 0         3      0      0      0        3      0      0      0 MOPRU

I have suspected kernel misconfiguration.  (Wouldn't iptables complain if I try to create a rule that names a destination module that is missing?)  I have tried messing with the nat rules, loading modules such as ip_nat, and much digging and studying of posts on load balancing with iptables.

My kernel is configured with CONFIG_IP_MULTIPLE_TABLES=y, and all the iptables stuff is either compiled in or available as modules.

The script for my load balancing attempt is below.  I'm hoping someone will spot the problem so I can get this working.  I can post output from lsmod, etc, if needed.

#!/bin/bash
modprobe xt_connmark
modprobe xt_statistic
modprobe xt_mark

# I read that the following is required for state RELATED to be set - ?
modprobe nf_conntrack_ftp

echo 1 >| /proc/sys/net/ipv4/ip_forward
echo 0 >| /proc/sys/net/ipv4/conf/all/rp_filter

# extract IP for each ppp interface
src0=`ip route show dev ppp0 | head -n1 | cut --delimiter=" " --fields=10`
src1=`ip route show dev ppp1 | head -n1 | cut --delimiter=" " --fields=10`

if echo "$src1" | grep 'ppp1'
then
	echo "ppp1 is not up"
	exit 1;
fi
if echo "$src0" | grep 'ppp0'
then
	echo "ppp0 is not up"
	exit 1;
fi

# flush iptables entries (temporarily set policies to ACCEPT)
iptables -t filter -F
iptables -t filter -X
iptables -t nat -F
iptables -t nat -X
iptables -t mangle -F
iptables -t mangle -X
iptables -t filter -P INPUT ACCEPT
iptables -t filter -P OUTPUT ACCEPT
iptables -t filter -P FORWARD ACCEPT
#iptables -t mangle -A PREROUTING -d 192.168.1.0/24 -j ACCEPT

# restore the fwmark on packets that belong to an existing connection
iptables -A PREROUTING -t mangle -m state --state ESTABLISHED,RELATED -j CONNMARK --restore-mark

# any packets with mark can be accepted at this point
iptables -A PREROUTING -t mangle -m mark ! --mark 0 -j ACCEPT

# set ALL packets to mark 1 (ppp0)
iptables -A PREROUTING -p tcp -t mangle -j MARK --set-mark 1

# set every other packet (50%) to mark 2 (ppp1)
iptables -A PREROUTING -p tcp -t mangle -m statistic --mode nth --every 2 --packet 0 -j MARK --set-mark 2

# save mark in connection table
iptables -A PREROUTING -t mangle -j CONNMARK --save-mark

# I am unclear on the following.   Do nat on packets to secondary interface ppp1
iptables -t nat -A POSTROUTING -o ppp1 -m state --state NEW,RELATED -j MASQUERADE

# other non-working ideas
#iptables -t nat -A POSTROUTING -o ppp0 -j SNAT --to-source $src0
#iptables -t nat -A POSTROUTING -o ppp1 -j SNAT --to-source $src1

# add rt_link1 as new routing table, if not already
if ! cat /etc/iproute2/rt_tables | grep -q '^251'
then
	echo '251     rt_link1' >> /etc/iproute2/rt_tables
fi

# add rt_link2 as new routing table, if not already
if ! cat /etc/iproute2/rt_tables | grep -q '^252'
then
	echo '252     rt_link2' >> /etc/iproute2/rt_tables
fi

# add default routes for new tables
ip route flush table rt_link1 2>/dev/null
ip route add table rt_link1 default dev ppp0
#ip route add default via 10.60.255.254 table rt_link1
ip route flush table rt_link2 2>/dev/null
ip route add table rt_link2 default dev ppp1
#ip route add default via 10.60.255.254 table rt_link2

#add fwmark rules
if ! ip rule show | grep -q 'rt_link2'
then
	ip rule add fwmark 1 table rt_link1
	ip rule add fwmark 2 table rt_link2
fi
ip route flush cache
<end of script>


Here's the output of 'ip rule show':

debiandesk:/home/lloyd# ip rule show
0:      from all lookup 255
32764:  from all fwmark 0x2 lookup rt_link2
32765:  from all fwmark 0x1 lookup rt_link1
32766:  from all lookup main
32767:  from all lookup default