SNAT for two interfaces not working

[Sheldon Hearn <sheldonh@clue.co.za>]

[-- Attachment #1: Type: text/plain, Size: 3366 bytes --]


Hi folks,

I approach you in desperation, after more than 8 hours of hacking and 4 
hours of Googling.

I have a Netfilter firewall (2.6.12-gentoo-r9 which is Linux-2.6.12.6 
with some Gentoo patches) that protects a number of privately addressed 
hosts.  It has a leased line, with a configuration that works perfectly 
and now I am trying to offload HTTP traffic over a new ADSL line.

My approach is to fwmark HTTP traffic, and route that via the ADSL 
router using iproute2.  It's "almost" working.  The initial SYN leaves 
(correctly SNAT'd) and the SYN+ACK arrives, but the packet goes missing 
at this point.  It doesn't hit INPUT or FORWARD.

I'm doing SNAT for both the leased line and the ADSL line.  SNAT is 
working fine for the leased line, and I'm worried that there's a 
problem with using SNAT (or MASQUERADE, which also doesn't work) on 
multiple interfaces.  The Google results don't look good.  The only 
thread I found with a firm conclusion was "gave up, using OpenBSD PF".  
I've just migrated this box _from_ FreeBSD so I can use Netfilter, so I 
_really_ don't want to go there.

Here's the layout:

eth0: 10.0.1.1/24				[LAN]
eth1: 172.16.1.2/24	(peer 172.16.1.1)	[leased line]
eth2: 172.16.2.2/24	(peer 172.16.2.1)	[ADSL]

Both peers NAT for us.  The default gateway is 172.16.1.1, which means 
that we route over the leased line by default.

I started out using FireHOL, but have trimmed down my firewall script to 
the bare essentials to maximise my chance of getting help on this list:

iptables -t mangle -A PREROUTING -i eth0 \
    -p tcp -m multiport --dports 80,443,8080 -j MARK --set-mark 2
iptables -t nat -A POSTROUTING -o eth2 -j SNAT --to-source 172.16.2.2
iptables -t nat -A POSTROUTING -o eth1 \
    -j SNAT --to-source 172.16.1.2
iptables -t filter -A FORWARD -j ACCEPT
iptables -t filter -A INPUT -j ACCEPT
iptables -t filter -A OUTPUT -j ACCEPT
ip rule add prio 100 fwmark 2 table 2
ip route add table 2 default via 172.16.2.1 dev eth2
ip route add default via 196.15.213.254 dev eth1
ip route 10.0.1.0/24 dev eth0

So, I hop onto 10.0.1.4 and try to establish an outbound HTTP 
connection.  I sniff eth2 on the firewall and see this:

13:40:37.936255 IP 172.16.2.2.62206 > 209.47.140.1.80: S 
838943189:838943189(0) win 65535 <mss 1460,nop,nop,sackOK,nop,wscale 
1,nop,nop,timestamp 127711 0>

13:40:37.976520 IP 209.47.140.1.80 > 172.16.2.2.62206: S 
2257786260:2257786260(0) ack 838943190 win 5840 <mss 
1432,nop,nop,sackOK>

Obviously they keep on trying.  On eth0, I see this:

13:40:37.936232 IP 10.0.1.4.62206 > 209.47.140.1.80: S 
838943189:838943189(0) win 65535 <mss 1460,nop,nop,sackOK,nop,wscale 
1,nop,nop,timestamp 127711 0>

13:40:40.931238 IP 10.0.1.4.62206 > 209.47.140.1.80: S 
838943189:838943189(0) win 65535 <mss 1460,nop,nop,sackOK,nop,wscale 
1,nop,nop,timestamp 128011 0>

So basically, the SYN+ACK is arriving back at the firewall, but the 
firewall then ignores it.  If I add logging, I see the packet hit 
PREROUTING, but that's it.

I've implemented HTTP-over-ADSL offload on FreeBSD IPFW+Squid and Linux 
IPFW, and thought I'd have absolutely no problem with Linux Netfilter.

Any help (even "it's known to not work, give up now" or "works for me, 
keep trying") would be GREATLY appreciated.

Thanks,
Sheldon.

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]