SNAT for two interfaces not working

SNAT for two interfaces not working

[Sheldon Hearn]

[-- Attachment #1: Type: text/plain, Size: 3366 bytes --]


Hi folks,

I approach you in desperation, after more than 8 hours of hacking and 4 
hours of Googling.

I have a Netfilter firewall (2.6.12-gentoo-r9 which is Linux-2.6.12.6 
with some Gentoo patches) that protects a number of privately addressed 
hosts.  It has a leased line, with a configuration that works perfectly 
and now I am trying to offload HTTP traffic over a new ADSL line.

My approach is to fwmark HTTP traffic, and route that via the ADSL 
router using iproute2.  It's "almost" working.  The initial SYN leaves 
(correctly SNAT'd) and the SYN+ACK arrives, but the packet goes missing 
at this point.  It doesn't hit INPUT or FORWARD.

I'm doing SNAT for both the leased line and the ADSL line.  SNAT is 
working fine for the leased line, and I'm worried that there's a 
problem with using SNAT (or MASQUERADE, which also doesn't work) on 
multiple interfaces.  The Google results don't look good.  The only 
thread I found with a firm conclusion was "gave up, using OpenBSD PF".  
I've just migrated this box _from_ FreeBSD so I can use Netfilter, so I 
_really_ don't want to go there.

Here's the layout:

eth0: 10.0.1.1/24				[LAN]
eth1: 172.16.1.2/24	(peer 172.16.1.1)	[leased line]
eth2: 172.16.2.2/24	(peer 172.16.2.1)	[ADSL]

Both peers NAT for us.  The default gateway is 172.16.1.1, which means 
that we route over the leased line by default.

I started out using FireHOL, but have trimmed down my firewall script to 
the bare essentials to maximise my chance of getting help on this list:

iptables -t mangle -A PREROUTING -i eth0 \
    -p tcp -m multiport --dports 80,443,8080 -j MARK --set-mark 2
iptables -t nat -A POSTROUTING -o eth2 -j SNAT --to-source 172.16.2.2
iptables -t nat -A POSTROUTING -o eth1 \
    -j SNAT --to-source 172.16.1.2
iptables -t filter -A FORWARD -j ACCEPT
iptables -t filter -A INPUT -j ACCEPT
iptables -t filter -A OUTPUT -j ACCEPT
ip rule add prio 100 fwmark 2 table 2
ip route add table 2 default via 172.16.2.1 dev eth2
ip route add default via 196.15.213.254 dev eth1
ip route 10.0.1.0/24 dev eth0

So, I hop onto 10.0.1.4 and try to establish an outbound HTTP 
connection.  I sniff eth2 on the firewall and see this:

13:40:37.936255 IP 172.16.2.2.62206 > 209.47.140.1.80: S 
838943189:838943189(0) win 65535 <mss 1460,nop,nop,sackOK,nop,wscale 
1,nop,nop,timestamp 127711 0>

13:40:37.976520 IP 209.47.140.1.80 > 172.16.2.2.62206: S 
2257786260:2257786260(0) ack 838943190 win 5840 <mss 
1432,nop,nop,sackOK>

Obviously they keep on trying.  On eth0, I see this:

13:40:37.936232 IP 10.0.1.4.62206 > 209.47.140.1.80: S 
838943189:838943189(0) win 65535 <mss 1460,nop,nop,sackOK,nop,wscale 
1,nop,nop,timestamp 127711 0>

13:40:40.931238 IP 10.0.1.4.62206 > 209.47.140.1.80: S 
838943189:838943189(0) win 65535 <mss 1460,nop,nop,sackOK,nop,wscale 
1,nop,nop,timestamp 128011 0>

So basically, the SYN+ACK is arriving back at the firewall, but the 
firewall then ignores it.  If I add logging, I see the packet hit 
PREROUTING, but that's it.

I've implemented HTTP-over-ADSL offload on FreeBSD IPFW+Squid and Linux 
IPFW, and thought I'd have absolutely no problem with Linux Netfilter.

Any help (even "it's known to not work, give up now" or "works for me, 
keep trying") would be GREATLY appreciated.

Thanks,
Sheldon.

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: SNAT for two interfaces not working

[David Coulson]

Sheldon Hearn wrote:
> So basically, the SYN+ACK is arriving back at the firewall, but the 
> firewall then ignores it.  If I add logging, I see the packet hit 
> PREROUTING, but that's it.

Disable return path filtering on the interfaces.

echo 0 > /proc/sys/net/ipv4/conf/eth0/rp_filter

David

Re: SNAT for two interfaces not working

[Sheldon Hearn]

[-- Attachment #1: Type: text/plain, Size: 670 bytes --]

On Wednesday 28 September 2005 15:51, David Coulson wrote:
> Sheldon Hearn wrote:
> > So basically, the SYN+ACK is arriving back at the firewall, but the
> > firewall then ignores it.  If I add logging, I see the packet hit
> > PREROUTING, but that's it.
>
> Disable return path filtering on the interfaces.
>
> echo 0 > /proc/sys/net/ipv4/conf/eth0/rp_filter

Thank you.  Thank you, thank you, thank you.

Grovel.  Grovel, grovel, grovel.

Ciao,
Sheldon.
-- 
Sheldon Hearn
IT Director
Clue Technologies (PTY) Ltd

Web:		http://www.clue.co.za/
Mail:		sheldonh@clue.co.za
Office:		+27-21-434-8034
Mobile:		+27-83-564-3276
Timezone:	SAST (+0200)

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: SNAT for two interfaces not working

[Sheldon Hearn]

[-- Attachment #1: Type: text/plain, Size: 638 bytes --]

On Wednesday 28 September 2005 15:51, David Coulson wrote:
> Sheldon Hearn wrote:
> > So basically, the SYN+ACK is arriving back at the firewall, but the
> > firewall then ignores it.  If I add logging, I see the packet hit
> > PREROUTING, but that's it.
>
> Disable return path filtering on the interfaces.
>
> echo 0 > /proc/sys/net/ipv4/conf/eth0/rp_filter
>
> David

For the integrity of the archives, it was actually 
net.ipv4.conf.eth2.rp_filter; I forgot to mention that, and I'd hate 
for someone else to struggle any more than necessary with the same 
problem.

But again, thank you so much.

Ciao,
Sheldon.

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]