NAT & Multihoming Problem

NAT & Multihoming Problem

[Matt]

I hope someone can help - or at least tell me whether what I'm trying to do is possible!

I have set up a linux box with 3 NICs, two external public IPs and one internal private IP. I set up the multihoming as per section 4.2 of the Advanced Routing HowTo. This seems to work OK without trying to do any NAT. (I've successfully pinged each public IP from a dial up connection on a laptop whilst the cable for the other NIC is disconnected)

Things don't work when I try to add NAT to the picture. I want to do port forwarding on both public IPs to a single private IP. Is this possible at all?

The current set up is as follows:

linux router:
eth0: 192.168.0.230/24
eth1: 100.100.251.218/29
eth2: 200.200.108.140/29

internal server:
eth0: 192.168.0.5/24

================ IPCHAINS STUFF ===============

#!/bin/sh

IPTABLES="/sbin/iptables"

#Time to clean house

#Clear out any existing firewall rules, and any chains that might have
#been created
$IPTABLES -F
$IPTABLES -F INPUT
$IPTABLES -F OUTPUT
$IPTABLES -F FORWARD
$IPTABLES -F -t mangle
$IPTABLES -F -t nat
$IPTABLES -X

#Setup our policies
$IPTABLES -P INPUT DROP
$IPTABLES -P OUTPUT ACCEPT
$IPTABLES -P FORWARD ACCEPT

#This enables ip forwarding, and thus by extension, NAT
#Turn this on if you're going to be doing NAT or Masquerading
echo 1 > /proc/sys/net/ipv4/ip_forward

/sbin/modprobe ip_conntrack
/sbin/modprobe ip_conntrack_ftp
/sbin/modprobe ip_nat_ftp
/sbin/modprobe iptable_nat


#Source NAT everything heading out the external interfaces
$IPTABLES -t nat -A POSTROUTING -o eth1 -j SNAT --to 100.100.251.218
$IPTABLES -t nat -A POSTROUTING -o eth2 -j SNAT --to 200.200.64.140

#This is the rule for publishing the Internal Server
$IPTABLES -t nat -A PREROUTING -i eth1 -p tcp --dport 56100 -j DNAT --to 192.168.0.5
$IPTABLES -t nat -A PREROUTING -i eth2 -p tcp --dport 56100 -j DNAT --to 192.168.0.5

#Now, our firewall chain
#We use the limit commands to cap the rate at which it alerts to 15
#log messages per minute
$IPTABLES -N firewall
$IPTABLES -A firewall -m limit --limit 1000/minute -j LOG --log-prefix Firewall:
$IPTABLES -A firewall -j DROP

#Now, our dropwall chain, for the final catchall filter
$IPTABLES -N dropwall
$IPTABLES -A dropwall -m limit --limit 1000/minute -j LOG --log-prefix Dropwall:
$IPTABLES -A dropwall -j DROP

#Our "hey, them's some bad tcp flags!" chain
$IPTABLES -N badflags
$IPTABLES -A badflags -m limit --limit 1000/minute -j LOG --log-prefix Badflags:
$IPTABLES -A badflags -j DROP

#And our silent logging chain
$IPTABLES -N silent
$IPTABLES -A silent -j DROP

#Accept ourselves (loopback interface), 'cause we're all warm and friendly
$IPTABLES -A INPUT -i lo -j ACCEPT

#basic state-matching
#This allows us to accept related and established connections, so
#client-side things like ftp work properly, for example.
$IPTABLES -A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT

#Our final trap. Everything on INPUT goes to the dropwall so we don't get silent drops
$IPTABLES -A INPUT -j dropwall

/sbin/ip route flush cache



=============== ROUTING STUFF ================

# Set up supplementary routing tables

ip route add 100.100.251.216/29 dev eth1 src 100.100.251.218 table T1
ip route add default via 100.100.251.217 table T1
ip route add 200.200.64.136/29 dev eth2 src 200.200.64.140 table T2
ip route add default via 200.200.64.137 table T2

# set default route for traffic originating from this machine to go via a third router
ip route add default via 192.168.0.252

ip rule add from 100.100.251.218 table T1
ip rule add from 200.200.64.140 table T2

ip route add 192.168.0.0/24 dev eth0 table T1
ip route add 200.200.64.136/29 dev eth2 table T1
ip route add 127.0.0.0/8 dev lo table T1
ip route add 192.168.0.0/24 dev eth0 table T2
ip route add 100.100.251.216/29 dev eth1 table T2
ip route add 127.0.0.0/8 dev lo table T2




When type "ip rule add from 192.168.0.5 lookup T2" at the server, it all starts working from my laptop and dial up account but this means, of course, that all traffic from the internal server is routed through 200.200.64.140 regardless of which interface it arrived at.

What am I doing wrong? Or am I trying to do something impossible?

Hope you can help,

Matt.

RE: NAT & Multihoming Problem

[Matt]

Thanks for your replies, Grant & Luis

Grant - sorry, I attempted to make the rules list shorter so I must have missed out some important rules.

The aim is that I'm trying to firstly get all traffic arriving at eth1 on the router to be NAT'd to an internal server and then replies sent back out on the same interface. Secondly - likewise for eth2 - all traffic arriving on eth2 to be replied on eth2, including any NAT'd traffic to the same internal server.

Luis suggested that I mark the packets... So after checking out chapter 11 of the adv routing howto, I added the following lines:

$IPTABLES -t mangle -A PREROUTING -i eth1 -j MARK --set-mark 11
$IPTABLES -t mangle -A PREROUTING -i eth2 -j MARK --set-mark 12

These lines occur after the SNAT lines, though I've tried above and below and it seems to make no difference.

I also added the following routing rules:

ip rule add fwmark 11 table T1
ip rule add fwmark 12 table T2

It still doesn't work.

There is a broken link in sect 11 of the adv routing howto that mentions that marking "collides" with SNAT and that I must turn off the reverse path filter. Could someone tell me whether this is relevant here?

I would also like to mention that when I tcpdump on eth0, I can see the SYN packets arriving from my dial up account and being sent to the internal server, I can see the reply being sent from the internal server to the router and also a packet that duplicates the reply with the correct public IP of the dial up account... but for some reason this packet does not get sent out on eth2.

So I'm not sure whether this is a question for this list or a routing list... sorry.

Any help would be appreciated.

Thanks.

-----Original Message-----
From: Grant Taylor
Sent: 05 December 2006 02:58
To: Matt
Subject: Re: NAT & Multihoming Problem


On 12/04/06 11:15, Matt wrote:
> I hope someone can help - or at least tell me whether what I'm trying to do is possible!
> 
> I have set up a linux box with 3 NICs, two external public IPs and one internal private IP. I set up the multihoming as per section 4.2 of the Advanced Routing HowTo. This seems to work OK without trying to do any NAT. (I've successfully pinged each public IP from a dial up connection on a laptop whilst the cable for the other NIC is disconnected)
> 
> Things don't work when I try to add NAT to the picture. I want to do port forwarding on both public IPs to a single private IP. Is this possible at all?

I don't know for sure that this is all of your problem or not.  However one 
thing that I do see is that you have a policy of DROP for your filter:INPUT 
chain.  You do have a statefull rule to allow related / established traffic 
in.  However, you do not appear to have any rules to allow new inbound traffic.

> What am I doing wrong? Or am I trying to do something impossible?



Grant. . . .

Re: NAT & Multihoming Problem

[Taylor, Grant]

On 12/04/06 11:15, Matt wrote:
> I hope someone can help - or at least tell me whether what I'm trying to do is possible!
> 
> I have set up a linux box with 3 NICs, two external public IPs and one internal private IP. I set up the multihoming as per section 4.2 of the Advanced Routing HowTo. This seems to work OK without trying to do any NAT. (I've successfully pinged each public IP from a dial up connection on a laptop whilst the cable for the other NIC is disconnected)
> 
> Things don't work when I try to add NAT to the picture. I want to do port forwarding on both public IPs to a single private IP. Is this possible at all?

I don't know for sure that this is all of your problem or not.  However one
thing that I do see is that you have a policy of DROP for your filter:INPUT
chain.  You do have a statefull rule to allow related / established traffic
in.  However, you do not appear to have any rules to allow new inbound traffic.

> What am I doing wrong? Or am I trying to do something impossible?



Grant. . . .

P.S.  Message was resent to the mailing list for all to see.

Re: NAT & Multihoming Problem

[Taylor, Grant]

Matt wrote:
> Thanks for your replies, Grant & Luis

You are welcome.

> Grant - sorry, I attempted to make the rules list shorter so I must have missed out some important rules.

That is ok.

> The aim is that I'm trying to firstly get all traffic arriving at eth1 on the router to be NAT'd to an internal server and then replies sent back out on the same interface. Secondly - likewise for eth2 - all traffic arriving on eth2 to be replied on eth2, including any NAT'd traffic to the same internal server.

This seems fairly reasonable.

> Luis suggested that I mark the packets... So after checking out chapter 11 of the adv routing howto, I added the following lines:
> 
> $IPTABLES -t mangle -A PREROUTING -i eth1 -j MARK --set-mark 11
> $IPTABLES -t mangle -A PREROUTING -i eth2 -j MARK --set-mark 12

It has been my (unsuccessfully) experience that MARKing inbound packets as 
they come in to a system does not necessarily ensure that returning outbound 
packets will be MARKed.  I believe this is where CONNMARK comes in to play 
to help maintain state of packet MARKs.  Rather that is to say that MARK is 
only maintained while packets are passing through the kernel.  Once a packet 
leaves the system any replies do not have the MARK associated with them.

Thus far in my career, I have only barely skimmed the surface of MARK and / 
or CONNMARK as I was able to do everything I needed to with IP Route 2 rules 
or plain routing.  As such, I'm probably not the best qualified to give you 
a complete answer on this topic.

I believe that you have to MARK the traffic as it passes through your router 
inbound to your server and save the MARK with CONNMARK.  Then have CONNMARK 
restore the MARK on the returning outbound traffic so that you can apply the 
usual rules / filters with the MARK.

> These lines occur after the SNAT lines, though I've tried above and below and it seems to make no difference.

Indeed.

> I also added the following routing rules:
> 
> ip rule add fwmark 11 table T1
> ip rule add fwmark 12 table T2

I have had very similar experience with a recent problem.  What I found was 
that outbound packets did have the correct IP SNATed to them.  However they 
were using the wrong interface to send the packets.  So, I wrote an ip rule 
to match based on the source IP rather than a MARK and things worked correctly.

> It still doesn't work.
> 
> There is a broken link in sect 11 of the adv routing howto that mentions that marking "collides" with SNAT and that I must turn off the reverse path filter. Could someone tell me whether this is relevant here?

A quick Google for "IPTables connmark" returns these top two links which may 
help shed some light on things for you.

http://home.regit.org/?page_id=7
http://home.regit.org/?page_id=20

> I would also like to mention that when I tcpdump on eth0, I can see the SYN packets arriving from my dial up account and being sent to the internal server, I can see the reply being sent from the internal server to the router and also a packet that duplicates the reply with the correct public IP of the dial up account... but for some reason this packet does not get sent out on eth2.

(*nod*  See above.)

> So I'm not sure whether this is a question for this list or a routing list... sorry.



Grant. . . .

RE: NAT & Multihoming Problem

[Matt]

Grant, Luis

With your help, I have found the magic commands to make this work!

Thanks again!

For your, and anyone else's interest, the answer is:

====================== IPTABLES commands =================================

#!/bin/sh

IPTABLES="/sbin/iptables"

# prevent incoming packets on masqueraded connections from being dropped
# as "martians" due to the destination address being translated before the
# rp_filter check is performed
# MATT NOTES: this does not seem to be necessary...
#echo 0 > /proc/sys/net/ipv4/conf/eth1/rp_filter
#echo 0 > /proc/sys/net/ipv4/conf/eth2/rp_filter

#Time to clean house

#Clear out any existing firewall rules, and any chains that might have
#been created
$IPTABLES -F
$IPTABLES -F INPUT
$IPTABLES -F OUTPUT
$IPTABLES -F FORWARD
$IPTABLES -F -t mangle
$IPTABLES -F -t nat
$IPTABLES -X

#Setup our policies
$IPTABLES -P INPUT DROP
$IPTABLES -P OUTPUT ACCEPT
$IPTABLES -P FORWARD ACCEPT

#This enables ip forwarding, and thus by extension, NAT
echo 1 > /proc/sys/net/ipv4/ip_forward

/sbin/modprobe ip_conntrack
/sbin/modprobe ip_conntrack_ftp
/sbin/modprobe ip_nat_ftp
/sbin/modprobe iptable_nat

#Our actual rules

#Our NAT stuff

#Source NAT everything heading out the external interface to be the
#given IP. 
$IPTABLES -t nat -A POSTROUTING -o eth1 -j SNAT --to 100.100.251.218
$IPTABLES -t nat -A POSTROUTING -o eth2 -j SNAT --to 200.200.64.140

#Mark incoming packets for later routing
$IPTABLES -t mangle -A PREROUTING -j CONNMARK --restore-mark
$IPTABLES -t mangle -A PREROUTING -i eth1 -j MARK --set-mark 11
$IPTABLES -t mangle -A PREROUTING -i eth2 -j MARK --set-mark 12

#save mark on outgoing packets
$IPTABLES -t mangle -A POSTROUTING -j CONNMARK --save-mark

#These are the rules for publishing the internal server
$IPTABLES -t nat -A PREROUTING -i eth1 -p tcp --dport 56100 -j DNAT --to 192.168.0.5
$IPTABLES -t nat -A PREROUTING -i eth2 -p tcp --dport 56100 -j DNAT --to 192.168.0.5


================== IPROUTE2 STUFF ===========================

# Set up supplementary routing tables

ip route add 100.100.251.216/29 dev eth1 src 100.100.251.218 table T1
ip route add default via 100.100.251.217 table T1
ip route add 200.200.64.136/29 dev eth2 src 200.200.64.140 table T2
ip route add default via 200.200.64.137 table T2

# Not necessary as these routes are added by configuring network cards
#ip route add 100.100.251.216/29 dev eth1 src 100.100.251.218
#ip route add 200.200.64.136/29 dev eth2 src 200.200.64.140

# set default route for traffic originating from this machine to go via third router
# not necessarily what other people may want to do, as you may want outgoing traffic
# going through eth1/eth2 in some load balanced way
ip route add default via 192.168.0.252

ip rule add from 100.100.251.218 table T1
ip rule add from 200.200.64.140 table T2

#not sure what this does, but is recommended in Advanced Routing HOWTO
ip route add 192.168.0.0/24 dev eth0 table T1
ip route add 200.200.64.136/29 dev eth2 table T1
ip route add 127.0.0.0/8 dev lo table T1
ip route add 192.168.0.0/24 dev eth0 table T2
ip route add 100.100.251.216/29 dev eth1 table T2
ip route add 127.0.0.0/8 dev lo table T2

#route based on mark
ip rule add fwmark 11 table T1
ip rule add fwmark 12 table T2


================================================


Regards,

Matt.