"DNAT" w/o changing source address?

"DNAT" w/o changing source address?

[John Madden]

I've got the typical DNAT configuration working fine, but I'm wondering
if there's a way to "port forward" without changing the source address
of the packets so that the destination sees the actual client's IP?

I've got a home-brewed load balancer running Pound for load balancing
HTTPS traffic to a cluster of web mail servers but I'd like to have
SMTP/POP/IMAP redirected to the single mail server without changing the
source address so things like RBL's still work.  This is to replace an
existing LVS installation, if that provides some idea of the workalike
I'm trying to build.  I've Googled for hours in vain...

Thanks,
  John



-- 
John Madden
Sr. UNIX Systems Engineer
Ivy Tech Community College of Indiana
jmadden@ivytech.edu

Re: "DNAT" w/o changing source address?

[Grant Taylor]

On 10/3/2007 10:21 AM, John Madden wrote:
> I've got the typical DNAT configuration working fine, but I'm 
> wondering if there's a way to "port forward" without changing the 
> source address of the packets so that the destination sees the actual 
> client's IP?

Um, correct me if I'm wrong, but Destination NATing should not alter the 
source IP address of the packet that is being NATed.

Honestly, I wonder how you are doing your DNATing and if you are not 
also possibly unknowingly SNATing as well.



Grant. . . .

Re: "DNAT" w/o changing source address?

[Pascal Hambourg]

Hello,

Grant Taylor a écrit :
> 
> Um, correct me if I'm wrong, but Destination NATing should not alter the 
> source IP address of the packet that is being NATed.

To be exhaustive, the only exception is in the OUTPUT chain on kernel 
versions less than 2.6.11, when DNAT changes the output interface the 
source address is also changed in order to match the new interface. 
However DNAT in the PREROUTING chain never changes the source address.

Re: "DNAT" w/o changing source address?

[Grant Taylor]

On 10/3/2007 6:50 PM, Pascal Hambourg wrote:
> To be exhaustive, the only exception is in the OUTPUT chain on kernel 
> versions less than 2.6.11, when DNAT changes the output interface the 
> source address is also changed in order to match the new interface. 
> However DNAT in the PREROUTING chain never changes the source address.

Ah, thank you for being exhaustive.  That makes more sense and supports 
what I have seen.

However this begs the question of what the original poster's 
configuration is that s/he is seeing this type of behavior.



Grant. . . .

Re: "DNAT" w/o changing source address?

[John Madden]

> Um, correct me if I'm wrong, but Destination NATing should not alter the 
> source IP address of the packet that is being NATed.
> 
> Honestly, I wonder how you are doing your DNATing and if you are not 
> also possibly unknowingly SNATing as well.

Hmm, well here are the rules I'm running.  The port forward:

echo "1" > /proc/sys/net/ipv4/ip_forward
iptables -t nat -A PREROUTING -d $EXTIP -p tcp --dport 25 -j DNAT --to
$MAILSERVER:25

And the SNAT for return traffic:

iptables -t nat -A POSTROUTING -d $MAILSERVER -j SNAT --to $EXTIP

...At least, I found that traffic wouldn't flow without this additional
rule.  Have I gotten something else fundamentally wrong here?

John



-- 
John Madden
Sr. UNIX Systems Engineer
Ivy Tech Community College of Indiana
jmadden@ivytech.edu

Re: "DNAT" w/o changing source address?

[John Madden]

> To be exhaustive, the only exception is in the OUTPUT chain on kernel 
> versions less than 2.6.11, when DNAT changes the output interface the 
> source address is also changed in order to match the new interface. 
> However DNAT in the PREROUTING chain never changes the source address.

This is on RHEL 4, which runs some variation of 2.6.9...

John




-- 
John Madden
Sr. UNIX Systems Engineer
Ivy Tech Community College of Indiana
jmadden@ivytech.edu

Re: "DNAT" w/o changing source address?

[Grant Taylor]

On 10/04/07 08:14, John Madden wrote:
> Hmm, well here are the rules I'm running.  The port forward:
> 
> echo "1" > /proc/sys/net/ipv4/ip_forward

*nod*

> iptables -t nat -A PREROUTING -d $EXTIP -p tcp --dport 25 -j DNAT 
> --to $MAILSERVER:25

*nod*

> And the SNAT for return traffic:
> 
> iptables -t nat -A POSTROUTING -d $MAILSERVER -j SNAT --to $EXTIP

Um, here in lies the rub.

> ...At least, I found that traffic wouldn't flow without this 
> additional rule.  Have I gotten something else fundamentally wrong 
> here?

That is very odd.  Do you have other rules in place that could be 
interfering with what you are doing?

Normally with a server behind a NAT all I need to do is DNAT the traffic 
and allow the returning traffic to pass back out through the same NATing 
system and allow it's outbound MASQUERADEing / SNAT to hide the internal 
source IP address.

If you do not have this type of scenario but rather both the redirecting 
IP and the real mail server's IP are both globally routable, then you 
may need to do something else.  Is this possibly the case?



Grant. . . .

Re: "DNAT" w/o changing source address?

[Pascal Hambourg]

John Madden a écrit :
> 
> Hmm, well here are the rules I'm running.  The port forward:
> 
> echo "1" > /proc/sys/net/ipv4/ip_forward
> iptables -t nat -A PREROUTING -d $EXTIP -p tcp --dport 25 -j DNAT --to
> $MAILSERVER:25
> 
> And the SNAT for return traffic:
> 
> iptables -t nat -A POSTROUTING -d $MAILSERVER -j SNAT --to $EXTIP

Ok, this is the rule that changes the source address. The DNAT rule in 
the PREROUTING chain could not do it, even with a kernel 2.6.9.

> ...At least, I found that traffic wouldn't flow without this additional
> rule.  Have I gotten something else fundamentally wrong here?

The above SNAT rule itself is not for return traffic. First, it matches 
packets destined to the mail server, i.e. original traffic. Second, 
return traffic skips the nat table chains.

If traffic does not flow without it, it could mean that the mail server 
does not send the reply traffic back to the NAT box. This is a routing 
problem. Does the mail server use the NAT box as its default gateway ?

Re: "DNAT" w/o changing source address?

[John Madden]

> That is very odd.  Do you have other rules in place that could be 
> interfering with what you are doing?

I have a dozen or so other rules that do the same thing for different
IP's (this is a load balancer).  

> Normally with a server behind a NAT all I need to do is DNAT the traffic 
> and allow the returning traffic to pass back out through the same NATing 
> system and allow it's outbound MASQUERADEing / SNAT to hide the internal 
> source IP address.

Well I thought that's what I was doing with that SNAT rule. =)

> If you do not have this type of scenario but rather both the redirecting 
> IP and the real mail server's IP are both globally routable, then you 
> may need to do something else.  Is this possibly the case?

Yeah, both machines have globally routable IP's.

John




-- 
John Madden
Sr. UNIX Systems Engineer
Ivy Tech Community College of Indiana
jmadden@ivytech.edu

Re: "DNAT" w/o changing source address?

[John Madden]

> If traffic does not flow without it, it could mean that the mail server 
> does not send the reply traffic back to the NAT box. This is a routing 
> problem. Does the mail server use the NAT box as its default gateway ?

Ah, now we're getting somewhere.  No, the mail server doesn't use the
NAT box as it's default gateway, it's using a general default route
somewhere else in the network for it.  The NAT box and the mail server
are on different VLAN's, but that's about all that separates them --
both have globally routable IP's.  

I'm literally just trying to emulate the functionality of LVS here,
where port 80 on an IP goes to one machine and port 25 goes somewhere
else.

John




-- 
John Madden
Sr. UNIX Systems Engineer
Ivy Tech Community College of Indiana
jmadden@ivytech.edu

Re: "DNAT" w/o changing source address?

[Pascal Hambourg]

John Madden a écrit :
>>If traffic does not flow without it, it could mean that the mail server 
>>does not send the reply traffic back to the NAT box. This is a routing 
>>problem. Does the mail server use the NAT box as its default gateway ?
> 
> Ah, now we're getting somewhere.  No, the mail server doesn't use the
> NAT box as it's default gateway, it's using a general default route
> somewhere else in the network for it.  The NAT box and the mail server
> are on different VLAN's, but that's about all that separates them --

Do you mean that they are in different subnets ?

> both have globally routable IP's.  

Private/public addressing does not matter here. You can have public 
addresses behind a NAT box, although it may sound unusual (NAT is mostly 
used to hide private addressing when you don't have enough public 
addresses). The important word is "behind", meaning that traffic in both 
directions flows through the NAT box. This is important because the NAT 
box changed the source and/or destination address on the original 
traffic, so it must put it back on the reply traffic in order for the 
client to accept it as a reply. It's not the SNAT rule which puts the 
original address back, it only makes the server see the NAT box as the 
client and send the reply traffic back to it. But the drawback is that 
the server does not see the real client source address.

Without SNAT, the mail server could use the NAT box as a gateway at 
least for SMTP reply traffic (this could be done with advanced routing 
if the mail server runs Linux) if they are in the same subnet or if a 
tunnel can be established directly between them.

> I'm literally just trying to emulate the functionality of LVS here,
> where port 80 on an IP goes to one machine and port 25 goes somewhere
> else.

Sorry, I do not know how LVS works. I just know how Netfilter NAT works.

Re: "DNAT" w/o changing source address?

[Grant Taylor]

On 10/04/07 09:19, John Madden wrote:
> I have a dozen or so other rules that do the same thing for different 
> IP's (this is a load balancer).

Ah, ok.

> Well I thought that's what I was doing with that SNAT rule. =)

No, with the SNAT rule you really are making the traffic appear as if it 
is from the NATing box its self.  Here in lies your rub that you are 
trying to avoid.

> Yeah, both machines have globally routable IP's.

Look for a more detailed discussion as a follow up to Pascal's post.



Grant. . . .

Re: "DNAT" w/o changing source address?

[John Madden]

> > Ah, now we're getting somewhere.  No, the mail server doesn't use the
> > NAT box as it's default gateway, it's using a general default route
> > somewhere else in the network for it.  The NAT box and the mail server
> > are on different VLAN's, but that's about all that separates them --
> 
> Do you mean that they are in different subnets ?

Sure.  But they could easily be on the same subnet.  

> Private/public addressing does not matter here. You can have public 
> addresses behind a NAT box, although it may sound unusual (NAT is mostly 
> used to hide private addressing when you don't have enough public 
> addresses). The important word is "behind", meaning that traffic in both 
> directions flows through the NAT box. This is important because the NAT 
> box changed the source and/or destination address on the original 
> traffic, so it must put it back on the reply traffic in order for the 
> client to accept it as a reply. It's not the SNAT rule which puts the 
> original address back, it only makes the server see the NAT box as the 
> client and send the reply traffic back to it. But the drawback is that 
> the server does not see the real client source address.

Right.  What I want instead is for the NAT box to change the destination
IP to direct the flow to the mail server.  I don't care where the reply
traffic goes (back through the NAT box is fine), I just need to maintain
the source IP's (which implies not going back through the NAT, but
rather directly back to the real client) to avoid confusion, make proper
use of RBL's, etc.

Imagine troubleshooting Outlook POP3 clients when everyone's coming from
the same IP.... *shudder*... 

> Without SNAT, the mail server could use the NAT box as a gateway at 
> least for SMTP reply traffic (this could be done with advanced routing 
> if the mail server runs Linux) if they are in the same subnet or if a 
> tunnel can be established directly between them.

The box does run Linux, but let's assume it doesn't.  I really don't
want to be horking with that machine in this manner.

> Sorry, I do not know how LVS works. I just know how Netfilter NAT works.

The idea is that when users hit "mail.ivytech.edu" in their browsers,
they get the web mail client.  When they hit that same address with
their SMTP clients, they'll talk to the MTA.  LVS allows you to do this
transparently and I assumed the same could be done with iptables --
that's all I'm trying to accomplish here.  

If the box could just modify the headers to change the destination IP
and drop the packets back on the wire without any change to the source
IP happening, I think I'd be happy.

John



-- 
John Madden
Sr. UNIX Systems Engineer
Ivy Tech Community College of Indiana
jmadden@ivytech.edu

Re: "DNAT" w/o changing source address?

[Grant Taylor]

On 10/04/07 09:59, Pascal Hambourg wrote:
> Do you mean that they are in different subnets ?

I doubt that the OPs systems are on different subnets, nor do I think 
that it really matters for what s/he is wanting to do.

> Private/public addressing does not matter here. You can have public 
> addresses behind a NAT box, although it may sound unusual (NAT is mostly 
> used to hide private addressing when you don't have enough public 
> addresses). The important word is "behind", meaning that traffic in both 
> directions flows through the NAT box. This is important because the NAT 
> box changed the source and/or destination address on the original 
> traffic, so it must put it back on the reply traffic in order for the 
> client to accept it as a reply. It's not the SNAT rule which puts the 
> original address back, it only makes the server see the NAT box as the 
> client and send the reply traffic back to it. But the drawback is that 
> the server does not see the real client source address.
> 
> Without SNAT, the mail server could use the NAT box as a gateway at 
> least for SMTP reply traffic (this could be done with advanced routing 
> if the mail server runs Linux) if they are in the same subnet or if a 
> tunnel can be established directly between them.

Ah yes, the old triangle of systems is being avoided.  System A talks to 
system B who talks to system C who talks back to system A.  System A 
knows that it talked to system B who for some strange reason is not 
replying while there is this other character system C that is striking 
up a conversation very improperly and as such being told where to RST to!

> Sorry, I do not know how LVS works. I just know how Netfilter NAT works.

As I understand it there are basically three different modes of 
operation for Linux Virtual Server (a.k.a. LVS) load balancers / 
directors:  NATing, Direct Routing, and Tunnel (a variant of DR).

I think DR and or Tunnel would be the better of the modes for this 
situation, seeing as how NAT will not work because the reverse traffic 
does not flow back through the LVS Director.

As I understand it (which could very well be flawed) in DR and Tunnel 
mode, your upstream router says that the virtual server is available via 
the next hop of the LVS director.  The LVS director in turn load 
balances and forwards (routes) the traffic on to the real servers which 
have the target IP bound to an interface.  Thus when the traffic does 
come in to the real server it comes in and goes directly in to the bound 
IP and associated services.  Thus when the services reply to the traffic 
they use the targeted IP as the source IP for reply traffic.  Thus we 
end up with system A talking to system C by way of system B with system 
C replying directly through routing.

I think DR and Tunneling are more of a switching / bridging technology 
in such as they alter the target ethernet mac addresses between the 
various systems to spread the load.

I'm not quite sure how you could emulate this with out using LVS, except 
possibly with bridging and EBTables rules to alter the target MAC of the 
inbound frames so that they are re-forwarded on to the real server. 
However for this to work your systems will have to be in the same 
broadcast domain.  If you want help with this, drop me a line and I'll 
see what I can 'switch' up.  ;)



Grant. . . .

Re: "DNAT" w/o changing source address?

[Grant Taylor]

On 10/04/07 10:13, John Madden wrote:
> Sure.  But they could easily be on the same subnet.

Ok.  So long as the NATing system can be connected to both subnets / 
VLANs I don't think this will be a problem.  If it is a problem, you may 
have to put both systems on the same subnet.

> Right.  What I want instead is for the NAT box to change the 
> destination IP to direct the flow to the mail server.  I don't care 
> where the reply traffic goes (back through the NAT box is fine), I 
> just need to maintain the source IP's (which implies not going back 
> through the NAT, but rather directly back to the real client) to 
> avoid confusion, make proper use of RBL's, etc.

This is why SNATing will not work.  You will have to do something 
fancier.  Like I eluded to in my previous message, I think you could do 
this with bridging and EBTables.

> Imagine troubleshooting Outlook POP3 clients when everyone's coming 
> from the same IP.... *shudder*...

*NO*, I will not think about such horror, shame on you for even 
suggesting it!

> The box does run Linux, but let's assume it doesn't.  I really don't 
> want to be horking with that machine in this manner.

Understood.

> The idea is that when users hit "mail.ivytech.edu" in their browsers, 
> they get the web mail client.  When they hit that same address with 
> their SMTP clients, they'll talk to the MTA.  LVS allows you to do 
> this transparently and I assumed the same could be done with iptables 
> -- that's all I'm trying to accomplish here.

LVS is not using traditional routing and as such you need to use 
something beyond that.

> If the box could just modify the headers to change the destination IP 
> and drop the packets back on the wire without any change to the 
> source IP happening, I think I'd be happy.

Do you need to even change the destination IP if you can somehow get the 
traffic over to the mail server?  I'm still thinking bridging and 
EBTables.  I'll think about this and get back to you with a proposed 
solution.



Grant. . . .

Re: "DNAT" w/o changing source address?

[Pascal Hambourg]

Grant Taylor a écrit :
> On 10/04/07 09:59, Pascal Hambourg wrote:
> 
>> Do you mean that they are in different subnets ?
> 
> I doubt that the OPs systems are on different subnets, nor do I think 
> that it really matters for what s/he is wanting to do.

It does matter. Granted, maybe should I say "broadcast domain" instead 
of "subnet" but they usually overlap. A router can be used as a gateway 
in a route only if it is directly reachable, which implies it is in the 
same subnet/broadcast domain. You mentionned bridging, which also 
implies the same broadcast domain.

PS : thanks for the explanation about LVS.

Re: "DNAT" w/o changing source address?

[Pascal Hambourg]

John Madden a écrit :
>>>The NAT box and the mail server
>>>are on different VLAN's, but that's about all that separates them --
>>
>>Do you mean that they are in different subnets ?
> 
> Sure.  But they could easily be on the same subnet.  

Ok. That may be useful.

> What I want instead is for the NAT box to change the destination
> IP to direct the flow to the mail server.  I don't care where the reply
> traffic goes (back through the NAT box is fine), I just need to maintain
> the source IP's

Then do not use SNAT.

> (which implies not going back through the NAT, but
> rather directly back to the real client)

On the contrary, it implies going back to the NAT, else the reply 
traffic arrives at the client with the wrong source address. See Grant's 
reply about the triangle ABC.

> Imagine troubleshooting Outlook POP3 clients when everyone's coming from
> the same IP.... *shudder*... 

I'd rather not. :-s

>>Without SNAT, the mail server could use the NAT box as a gateway at 
>>least for SMTP reply traffic (this could be done with advanced routing 
>>if the mail server runs Linux) if they are in the same subnet or if a 
>>tunnel can be established directly between them.
> 
> The box does run Linux, but let's assume it doesn't.  I really don't
> want to be horking with that machine in this manner.

Ok, then the easiest solution is to put the NAT box and the server in 
the same subnet and use the NAT box as the default gateway on the 
server. You may have trouble with ICMP "Redirect" messages sent by the 
NAT box if its own default gateway is also in the same subnet, but you 
can disable them on the NAT box or ignore them on the server.

> If the box could just modify the headers to change the destination IP
> and drop the packets back on the wire without any change to the source
> IP happening, I think I'd be happy.

That's just what DNAT does. The rest is about routing.

Re: "DNAT" w/o changing source address?

[Grant Taylor]

On 10/04/07 10:52, Pascal Hambourg wrote:
> It does matter. Granted, maybe should I say "broadcast domain" instead 
> of "subnet" but they usually overlap. A router can be used as a gateway 
> in a route only if it is directly reachable, which implies it is in the 
> same subnet/broadcast domain. You mentionned bridging, which also 
> implies the same broadcast domain.

You are correct.  However I should have been a bit more specific in that 
I don't think that it will matter either way as I think a solution for 
either config can be developed.  Thus it does not matter what it is 
because both can probably be solved.  As far as what the solution is, 
yes it does matter.

I have done more and more with bridging and VLANs to provide very custom 
solutions for a lot of my clients.  I have spanned a single subnet 
across 25+ broadcast domains using bridging and EBTables.  As such the 
lines tend to bluer a lot.  ;)

> PS : thanks for the explanation about LVS.

You're welcome.  I hope that I did an adequate job at explaining it 
based on the fact that I have never used it my self (no call for it 
/yet/), just done a lot of reading.



Grant. . . .

Re: "DNAT" w/o changing source address?

[John Madden]

On Thu, 2007-10-04 at 14:12 -0500, Grant Taylor wrote:
> On 10/04/07 10:52, Pascal Hambourg wrote:
> > It does matter. Granted, maybe should I say "broadcast domain" instead 
> > of "subnet" but they usually overlap. A router can be used as a gateway 
> > in a route only if it is directly reachable, which implies it is in the 
> > same subnet/broadcast domain. You mentionned bridging, which also 
> > implies the same broadcast domain.
> 
> You are correct.  However I should have been a bit more specific in that 
> I don't think that it will matter either way as I think a solution for 
> either config can be developed.  Thus it does not matter what it is 
> because both can probably be solved.  As far as what the solution is, 
> yes it does matter.

Well thanks all for the tips and pointing out what I was doing wrong
here.  I've decided to throw the whole thing out the window -- I'll be
pulling the separate load balancer box out of the picture, moving the
installation of Pound to the mail server itself, and just changing
$EXTIP's hostname to be an alias for $MAILSERVER.  Things will be much
simpler, but not nearly as sexy.  I suppose it'll do. :)

Thanks again,
  John


-- 
John Madden
Sr. UNIX Systems Engineer
Ivy Tech Community College of Indiana
jmadden@ivytech.edu

Re: "DNAT" w/o changing source address?

[Grant Taylor]

On 10/04/07 10:29, Grant Taylor wrote:
> Do you need to even change the destination IP if you can somehow get the 
> traffic over to the mail server?  I'm still thinking bridging and 
> EBTables.  I'll think about this and get back to you with a proposed 
> solution.

After some food and some thought, I am convinced that this can be done 
with bridging and EBTables.

For the sake of conversation I'm going to assume that the NATing host 
has or can have access to both VLANs.  Let vlan0 be the vlan of the 
router and NATing host and let vlan1 be the vlan of the real host that 
you want to redirect the traffic to.

  - Create a bridge bridge 'bri0' that has vlan0 and vlan1 as ports in it.
  - Assign the NATing hosts IP address(s) to the bri0 interface.
  - Create an EBTables rule in the BROUTING chain of the broute table 
that looks for the following conditions:
     - Proper ethernet protocol - IP
     - Proper IP protocol - TCP and / or UDP
     - Proper destination port
  - Have said EBTables rule DNAT the traffic to the MAC address of the 
real host that you want to redirect the traffic to.
  - Have the IP address bound to an interface on the real host that you 
want to redirect the traffic to.
  - Create a second EBTables rule in the BROUTING chain of the broute 
table that causes all other traffic to be routed like normal.

This will cause the NATing system to handle normal traffic while 
redirecting the traffic to the real host on the MAC layer (2).  Thus the 
real host will receive the traffic in with the proper destination IP, 
which it will know how to use.  Thus the real host will have the IP in 
question thus allowing traffic to originate from said IP back to the 
original client with the proper source IP.

Heck, if you wanted to you could even do this before the traffic gets to 
the NATing host.  This way, you don't even have to have any thing 
special on the NATing host.  Thus both your NATing host and real host 
could be any OS with an IP stack that you want them to be.  The Linux 
bridge with EBTables could take care of this before the traffic reaches 
any system.  (More on this later.)

Note:  Some people would rather assign IP addresses to the physical 
bridge port, but I prefer to use the logical bridge interface.  It 
really is up to you.



Grant. . . .