How to make a mutli-homed host use one IP for a NAT'ed host

How to make a mutli-homed host use one IP for a NAT'ed host

[Greg Cope]

Hi All,

I have a mutli-homed router/firewall.

One IP address is the default router gateway address

Three other (virtual) IP's are for applications (one of which is SMTP).

My mailserver is on a NAT'ed DMZ.

However when it sends mail the router uses the default gateway address
as opposed to the SMTP IP address.

This upsets reverse DNS lookups as it now appears to come from the
router as opposed to the mail IP (that has the correct reverse IP
address lookup).

Any ideas?

Greg

Re: How to make a mutli-homed host use one IP for a NAT'ed host

[Taylor Grant]

Greg, if I understand you correctly you have a system that is multi homed with one of it's IP addresses being port forwarded to a DMZ server?  What happens is that your DMZ server replies back to your router / firewall which then in turn send the traffic out it's default route with a source IP of something other than what you want your SMTP traffic to use?  If this is the case you need to so a few tings to get your traffic to go out the interface that you want with the IP that you want.

For starters you will need to use IPTables to mark the traffic that is coming in to your router / firewall from the DMZ mail server.
Second you will need to create a new routeing table for this traffic via something like the following:

iptables -t nat -A OUTPUT -s $DMZ_Server_IP -p tcp --sport 25 -j MARK --set-mark $SMTP_Mark
ip route add table $IPRoute2_SMTP_Table dev $DEV_of_SMTP_network_interface_or_alias src $IP_of_DEV_of_SMTP_network_interface_or_alias
ip route add table $IPRoute2_SMTP_Table default via $Default_INet_Gateway
ip rule add fwmark $SMTP_Mark table $IPRoute2_SMTP_Table

Where:
$DMZ_Server_IP is the internal LAN ip of your SMTP server in the DMZ.
$SMTP_Mark is the value you want to use to mark the packets that need to use the alternant route.
$IPRoute2_SMTP_Table is the name as it appears in /etc/iproute2/rt_tables or the number of the table that you want to use.
$DEV_of_SMTP_network_interface_or_alias is the device name or alias of your network interface that you want all SMTP traffic to use.
$IP_of_DEV_of_SMTP_network_interface_or_alias is the IP address of the device name or alias of your network interface that you want all SMTP traffic to use.
$Default_INet_Gateway is the IP address of your internet gateway.

This should cause any SMPT traffic (source port 25 from the SMTP server) to go out a different interface / alias and thus use a different source IP when the traffic is sent out to the world.  If you have any questions or need more help let me know and I'll see what I can do.


Greg Cope wrote:
> Hi All,
> 
> I have a mutli-homed router/firewall.
> 
> One IP address is the default router gateway address
> 
> Three other (virtual) IP's are for applications (one of which is SMTP).
> 
> My mailserver is on a NAT'ed DMZ.
> 
> However when it sends mail the router uses the default gateway address
> as opposed to the SMTP IP address.
> 
> This upsets reverse DNS lookups as it now appears to come from the
> router as opposed to the mail IP (that has the correct reverse IP
> address lookup).
> 
> Any ideas?
> 
> Greg

Re: How to make a mutli-homed host use one IP for a NAT'ed host

[Greg Cope]

Hi all,

Hum... not quite working for me yet, nearlt there but I get the error:

"MARK: can only be called from "mangle" table, not "nat""

So I used:

iptables -A PREROUTING -i eth0 -t mangle -s $DMZ_HOST_IP -p tcp
--dport 25 -j MARK --set-mark 2

Q: Is eth0 correct as this is the red/ INET IFACE and not the DMZ dev
IFACE (that would be eth1)

And then:

ip route add table $IPROUTE2_SMTP_TABLE dev $INET_IFACE src $MAIL_INET_ALIAS
ip route add table $IPROUTE2_SMTP_TABLE default via $INET_IP
ip rule add fwmark $SMTP_MARK table $IPROUTE2_SMTP_TABLE

Where $SMTP_MARK=2 and IPROUTE2_SMTP_TABLE=smtp.out

I have "echo 25 smtp.out >> /etc/iproute2/rt_tables"

Packets still come from the "wrong" ip address

Any suggestions.

Thanks.

Greg

On 4/21/05, Greg Cope <gregcope@gmail.com> wrote:
> Wow - Thanks!
> 
> Your synopsis is correct.
> 
> Here is my spin on your excellent advice:
> 
> I've assumed that SMTP_MARK is just an integer label
> SMTP_MARK=999
> 
> $IPTABLES -t nat -A OUTPUT -s $DMZ_MAIL_IP -p tcp --sport 25 -j MARK
> --set-mark $SMTP_MARK
> 
> Quick question - this marks only packets from the DMZ_MAIL_IP that use
> TCP and are from sport 25, whereas my mailserver picks high ports to
> connect to exterior SMTP servers, so I might change this to:
> 
> $IPTABLES -t nat -A OUTPUT -s $DMZ_MAIL_IP -p tcp --dport 25 -j MARK
> --set-mark $SMTP_MARK
> 
> Ie if dest port (ie SMTP server) get marked
> 
> ip route add table $IPRoute2_SMTP_Table dev $MAIL_INET_ALIAS_IFACE src
> $MAIL_INET_ALIAS
> ip route add table $IPRoute2_SMTP_Table default via $INET_IP
> ip rule add fwmark $SMTP_MARK table $IPRoute2_SMTP_Table
> 
> Can you elaborate on $IPRoute2_SMTP_Table - I assume I need to add a
> line to /etc/iproute2/rt_tables like:
> 
> 250     smtp
> 
> Any good concise how to's on this?
> 
> Again many thanks for the excellent and prompt reply!
> 
> Greg
> 
> On 4/21/05, Taylor Grant <gtaylor@riverviewtech.net> wrote:
> > Greg, if I understand you correctly you have a system that is multi homed with one of it's IP addresses being port forwarded to a DMZ server?  What happens is that your DMZ server replies back to your router / firewall which then in turn send the traffic out it's default route with a source IP of something other than what you want your SMTP traffic to use?  If this is the case you need to so a few tings to get your traffic to go out the interface that you want with the IP that you want.
> >
> > For starters you will need to use IPTables to mark the traffic that is coming in to your router / firewall from the DMZ mail server.
> > Second you will need to create a new routeing table for this traffic via something like the following:
> >
> > iptables -t nat -A OUTPUT -s $DMZ_Server_IP -p tcp --sport 25 -j MARK --set-mark $SMTP_Mark
> > ip route add table $IPRoute2_SMTP_Table dev $DEV_of_SMTP_network_interface_or_alias src $IP_of_DEV_of_SMTP_network_interface_or_alias
> > ip route add table $IPRoute2_SMTP_Table default via $Default_INet_Gateway
> > ip rule add fwmark $SMTP_Mark table $IPRoute2_SMTP_Table
> >
> > Where:
> > $DMZ_Server_IP is the internal LAN ip of your SMTP server in the DMZ.
> > $SMTP_Mark is the value you want to use to mark the packets that need to use the alternant route.
> > $IPRoute2_SMTP_Table is the name as it appears in /etc/iproute2/rt_tables or the number of the table that you want to use.
> > $DEV_of_SMTP_network_interface_or_alias is the device name or alias of your network interface that you want all SMTP traffic to use.
> > $IP_of_DEV_of_SMTP_network_interface_or_alias is the IP address of the device name or alias of your network interface that you want all SMTP traffic to use.
> > $Default_INet_Gateway is the IP address of your internet gateway.
> >
> > This should cause any SMPT traffic (source port 25 from the SMTP server) to go out a different interface / alias and thus use a different source IP when the traffic is sent out to the world.  If you have any questions or need more help let me know and I'll see what I can do.
> >
> >
> > Greg Cope wrote:
> > > Hi All,
> > >
> > > I have a mutli-homed router/firewall.
> > >
> > > One IP address is the default router gateway address
> > >
> > > Three other (virtual) IP's are for applications (one of which is SMTP).
> > >
> > > My mailserver is on a NAT'ed DMZ.
> > >
> > > However when it sends mail the router uses the default gateway address
> > > as opposed to the SMTP IP address.
> > >
> > > This upsets reverse DNS lookups as it now appears to come from the
> > > router as opposed to the mail IP (that has the correct reverse IP
> > > address lookup).
> > >
> > > Any ideas?
> > >
> > > Greg
> >
>

Re: How to make a mutli-homed host use one IP for a NAT'ed host

[Taylor Grant]

> Hi all,
> 
> Hum... not quite working for me yet, nearlt there but I get the error:
> 
> "MARK: can only be called from "mangle" table, not "nat""
> 
> So I used:
> 
> iptables -A PREROUTING -i eth0 -t mangle -s $DMZ_HOST_IP -p tcp
> --dport 25 -j MARK --set-mark 2

Sorry, my mistake.  It was late at night after a long day.  :(

> Q: Is eth0 correct as this is the red/ INET IFACE and not the DMZ dev
> IFACE (that would be eth1)

No.  I think you should use eth1 in your IPTables rule as you are looking to mark the traffic that is coming back to the router / firewall from the DMZ/SMTP server that is outbound to the world.  Basically you want to mark the SMTP server's returning traffic as a control handle that you can look for with an IPRoute2 rule so that the routing core can decide what routing table to use to send the traffic back out to the world.

> And then:
> 
> ip route add table $IPROUTE2_SMTP_TABLE dev $INET_IFACE src $MAIL_INET_ALIAS
> ip route add table $IPROUTE2_SMTP_TABLE default via $INET_IP
> ip rule add fwmark $SMTP_MARK table $IPROUTE2_SMTP_TABLE
> 
> Where $SMTP_MARK=2 and IPROUTE2_SMTP_TABLE=smtp.out
> 
> I have "echo 25 smtp.out >> /etc/iproute2/rt_tables"
> 
> Packets still come from the "wrong" ip address
> 
> Any suggestions.
> 
> Thanks.
> 
> Greg

Try changing your eth0 to eth1 in your IPTables mark rule.  Other than that (and my snafu about the wrong table) I think your set up should work just fine.  I feel like you are very close to having what you want set up and working.  :)



Grant. . . .

Re: How to make a mutli-homed host use one IP for a NAT'ed host

[Greg Cope]

> Sorry, my mistake.  It was late at night after a long day.  :(

Many Thanks Grant.

No need to appologies - and your up already!

So I swapped to eth1 - and it would appear that packets are getting
marked as if I add in the ip route add command I can no longer connect
to an SMTP server from the mailserver (using telnet HOST 25)

So the last few hurdles...

I need to use the same router - an ADSL thing as everything else, I
just want the packets to be sent from a different $MAIL_INET_ALIAS IP
(as opposed to the usual INET_IP

$ ip rule add fwmark $SMTP_MARK table $IPROUTE2_SMTP_TABLE

Works fine and makes sense.

However I think these are the sticking points:

ip route add table $IPROUTE2_SMTP_TABLE dev $INET_IFACE src $MAIL_INET_ALIAS
ip route add table $IPROUTE2_SMTP_TABLE default via $INET_IP

I get 

$ ip route add table smtp.out dev eth0 src 217.154.55.250
$ ip route add table smtp.out default via 217.154.55.241
RTNETLINK answers: File exists


.241 is my internet gateway router, not the default external IP of my
firewall which is eth0=217.154.55.249. eth0:1=217.154.55.250 which is
the $MAIL_INET_ALIAS

Combinations of either of the above mean the mail server cannot
connect to an exteriror mail server.

Hum....

Greg

Re: How to make a mutli-homed host use one IP for a NAT'ed host

[Taylor, Grant]

> No need to appologies - and your up already!

Heh, ya the glorious life of a computer consultant. :)

> So I swapped to eth1 - and it would appear that packets are getting
> marked...

*nod*

> So the last few hurdles...
> 
> I need to use the same router - an ADSL thing as everything else, I
> just want the packets to be sent from a different $MAIL_INET_ALIAS IP
> (as opposed to the usual INET_IP

This makes perfect sense to me and is what I understood you to be wanting.  Now to just get the syntax correct so that it does work.

> However I think these are the sticking points:
> 
> ip route add table $IPROUTE2_SMTP_TABLE dev $INET_IFACE src $MAIL_INET_ALIAS
> ip route add table $IPROUTE2_SMTP_TABLE default via $INET_IP
> 
> I get 
> 
> $ ip route add table smtp.out dev eth0 src 217.154.55.250
> $ ip route add table smtp.out default via 217.154.55.241
> RTNETLINK answers: File exists

You may need to change your device entire in the above rule to be eth0:1 not eth0.  Some testing should tell you this for sure.

> .241 is my internet gateway router, not the default external IP of my
> firewall which is eth0=217.154.55.249. eth0:1=217.154.55.250 which is
> the $MAIL_INET_ALIAS
> 
> Combinations of either of the above mean the mail server cannot
> connect to an exteriror mail server.

Hmm, would it be possible for me to see the full output of your ip route tables and rules prior to what we have done here?  I'd also like to see the output of your iptables rules for the filter, nat, and mangle tables.  This way I can look at the entire picture and hopefully give you the correct info with out having to work with pieces of the information.



Grant. . . .

Re: How to make a mutli-homed host use one IP for a NAT'ed host

[Greg Cope]

> > So the last few hurdles...
> >
> > I need to use the same router - an ADSL thing as everything else, I
> > just want the packets to be sent from a different $MAIL_INET_ALIAS IP
> > (as opposed to the usual INET_IP
> 
> This makes perfect sense to me and is what I understood you to be wanting.  Now to just get the syntax correct so that it does work.

ok.
 
> > However I think these are the sticking points:
> >
> > ip route add table $IPROUTE2_SMTP_TABLE dev $INET_IFACE src $MAIL_INET_ALIAS
> > ip route add table $IPROUTE2_SMTP_TABLE default via $INET_IP
> >
> > I get
> >
> > $ ip route add table smtp.out dev eth0 src 217.154.55.250
> > $ ip route add table smtp.out default via 217.154.55.241
> > RTNETLINK answers: File exists
> 
> You may need to change your device entire in the above rule to be eth0:1 not eth0.  Some testing should tell you this for sure.

Tried that :-)

> Hmm, would it be possible for me to see the full output of your ip route tables and rules prior to what we have done here?  I'd also like to see the output of your iptables rules for the filter, nat, and mangle tables.  This way I can look at the entire picture and hopefully give you the correct info with out having to work with pieces of the information.

The IP tables are quite long as we have quite a few VPN rules.

The rest is quite straight forward.

What would be easiest to start with.

Greg

PS its night time here in the UK ......

Re: How to make a mutli-homed host use one IP for a NAT'ed host

[Taylor, Grant]

> The IP tables are quite long as we have quite a few VPN rules.

Ok.  I understand and will take that in to advisement.

> The rest is quite straight forward.
> 
> What would be easiest to start with.

Well if you are not doing any thing weird as far as filtering in your nat or mangle tables then I'd like to see an output of your filter tables FORWARD chain.  If you want to scrub IPs for security reasons I understand perfectly.  Can I also see an output of "ip route list table main"?  Can I get a list of what devices / aliases and their IPs are (or scrubbed for conversation)?  That way what I send back to you could be close to production information.

> PS its night time here in the UK ......

*nod*  I'm presently at -5 UTC (day light savings time, we are really -6).



Grant. . . .