Problem with DNAT of UDP packets getting undone

Problem with DNAT of UDP packets getting undone

[Mongiovi, Roy]

I've got a DNS server running Redhat EL 3.0 AS on a private network behind a load balancer.  UDP queries are supposed to come into the load balancer on an internet routable VIP and be sent to the DNS servers on the private network.  The DNS servers are supposed to send their responses directly to the client via an internet connection that is not behind the load balancer.

Unfortunately, the load balancer is sending the query packets to the DNS server with the destination IP address set to its private network address.  In order to get the responses to appear to come from the load balancer's VIP, I have configured the VIP address on the DNS server's loopback device as a /32 address.  I put a rule in the nat table PREROUTING chain to do a DNAT on the incoming packets so that they would appear to be destined for the VIP rather than the private network.  That way, when DNS sent the response packet it would automatically come from the VIP address.

Unfortunately, this just doesn't quite work.  I've added logging rules to the mangle PREROUTING and the INPUT and FORWARD chains so that I can make sure the DNAT is happening.  It is.  As far as packets going into the server everything is according to plan.  I also added logging rules to all the OUTPUT and POSTROUTING chains, and here's where things get strange.  I see the response packet generated by the DNS software in the mangle and filter OUTPUT chains as well as the mangle POSTROUTING chain.  In all cases it has the desired VIP source IP.  But for some reason, I don't see it in the nat OUTPUT or POSTROUTING chains.  I don't really understand that at all.

But the real problem is that when the packet exits the server (as verified by tcpdump), the source IP has been changed back to the private network IP.  How and why is that happening, and is there any way to accomplish what I'm attempting? Thanks.
Roy
---
Roy Mongiovi     Senior Member Technical Staff    AT&T Labs, Inc.
       roy.mongiovi@att.com                            4A70
Tough are the souls that tread the knife's edge     575 Morosgo Drive
       Jethro Tull - "Passion Play"                     Atlanta, GA 30324 

Re: Problem with DNAT of UDP packets getting undone

[Martijn Lievaart]

<citaat van="Mongiovi, Roy">
> I've got a DNS server running Redhat EL 3.0 AS on a private network behind
> a load balancer.  UDP queries are supposed to come into the load balancer
> on an internet routable VIP and be sent to the DNS servers on the private
> network.  The DNS servers are supposed to send their responses directly to
> the client via an internet connection that is not behind the load
> balancer.
>
> Unfortunately, the load balancer is sending the query packets to the DNS
> server with the destination IP address set to its private network
> address.  In order to get the responses to appear to come from the load
> balancer's VIP, I have configured the VIP address on the DNS server's
> loopback device as a /32 address.  I put a rule in the nat table
> PREROUTING chain to do a DNAT on the incoming packets so that they would
> appear to be destined for the VIP rather than the private network.  That
> way, when DNS sent the response packet it would automatically come from
> the VIP address.
>
> Unfortunately, this just doesn't quite work.  I've added logging rules to
> the mangle PREROUTING and the INPUT and FORWARD chains so that I can make
> sure the DNAT is happening.  It is.  As far as packets going into the
> server everything is according to plan.  I also added logging rules to all
> the OUTPUT and POSTROUTING chains, and here's where things get strange.  I
> see the response packet generated by the DNS software in the mangle and
> filter OUTPUT chains as well as the mangle POSTROUTING chain.  In all
> cases it has the desired VIP source IP.  But for some reason, I don't see
> it in the nat OUTPUT or POSTROUTING chains.  I don't really understand
> that at all.
>
> But the real problem is that when the packet exits the server (as verified
> by tcpdump), the source IP has been changed back to the private network
> IP.  How and why is that happening, and is there any way to accomplish
> what I'm attempting? Thanks.

This is expected. A nat mapping is set up on the first packet of a
"connection" and a reverse NAT is done automagically on all return
packets. Exactly what you are seeing.

This cannot easily be solved with current Linux kernels. Current kernels
only do connection oriented NAT. You could insert a Cisco device or
something similar to do the kind of NAT you require.

But......, the loadbalancer does DNAT, so it shoud do the corresponding
SNAT on the return packets. If it does not do this, it is seriously
broken, imnsho.

HTH,
M4

Re: Problem with DNAT of UDP packets getting undone

[Grant Taylor]

On 11/06/07 11:03, Martijn Lievaart wrote:
> This is expected. A nat mapping is set up on the first packet of a 
> "connection" and a reverse NAT is done automagically on all return 
> packets. Exactly what you are seeing.

I'll agree to the NATing part.  However the fact that the OP is 
successfully using the loopback interface surprises me.  It was my 
(mis)understanding that the loopback interface was holly and would not 
talk to traffic that did not originate or terminate on the loopback 
interface as well.  Or is the a side effect that the NATing code is 
sending the traffic out the loopback interface destined to the loopback 
interface as well, thus NATing is bridging the security barrier?  I am 
almost positive that the same could not be done with routing.

> This cannot easily be solved with current Linux kernels. Current 
> kernels only do connection oriented NAT. You could insert a Cisco 
> device or something similar to do the kind of NAT you require.

Would it be possible to use stateless NATing via IP Route 2 rather than 
IPTables to achieve this?

> But......, the loadbalancer does DNAT, so it shoud do the 
> corresponding SNAT on the return packets. If it does not do this, it 
> is seriously broken, imnsho.

Remember that the OP said "... The DNS servers are supposed to send 
their responses directly to the client via an internet connection that 
is not behind the load balancer. ...".  Thus the reply traffic is not 
passing back out through the load balancer and as such the load balancer 
never sees the traffic to undo the NATing in place.

My recommendation to the OP is to set up the LVS in 'Direct Routing' 
mode with the VIP bound to an additional non-ARPing interface other than 
the loopback interface on all the real nodes.  This way the LVS 
Director(s) will initially receive the traffic to the VIP and then 
re-route the traffic on to the real server node.  The real server node 
will then receive the traffic destined to the VIP with the DNS server 
running on it.  This way all the traffic is really to and from the VIP 
address(s).  This scenario will allow the real server(s) to use a 
different route for the returning traffic than the original traffic used.



Grant. . . .

Re: Problem with DNAT of UDP packets getting undone

[Pascal Hambourg]

Hello,

Grant Taylor a écrit :
> On 11/06/07 11:03, Martijn Lievaart wrote:
> 
>> This is expected. A nat mapping is set up on the first packet of a 
>> "connection" and a reverse NAT is done automagically on all return 
>> packets. Exactly what you are seeing.
> 
> I'll agree to the NATing part.

So do I. Otherwise, stateful NAT would not work very well.

> However the fact that the OP is 
> successfully using the loopback interface surprises me.  It was my 
> (mis)understanding that the loopback interface was holly and would not 
> talk to traffic that did not originate or terminate on the loopback 
> interface as well.

You may confuse with the restriction from some RFCs stating that 
127.0.0.0/8 addresses are reserved for internal host use, i.e. the 
loopback interface. There is no such restriction for other addresses 
that may be configured on the loopback interface. Also, the Linux IP 
stack follows the "weak" model by default, so any unicast address 
(except 127.0.0.0/8) configured on any interface  can be used for 
communications on any other interface. So any non-127.0.0.0/8 address 
configured on the loopback interface can be used for communications on 
any other interface.

> Or is the a side effect that the NATing code is 
> sending the traffic out the loopback interface destined to the loopback 
> interface as well, thus NATing is bridging the security barrier?  I am 
> almost positive that the same could not be done with routing.

Nope, NAT has nothing to do with this, and the loopback interface is not 
involved.

>> This cannot easily be solved with current Linux kernels. Current 
>> kernels only do connection oriented NAT. You could insert a Cisco 
>> device or something similar to do the kind of NAT you require.
> 
> Would it be possible to use stateless NATing via IP Route 2 rather than 
> IPTables to achieve this?

The old stateless NAT in the routing code controlled with iproute2 is 
considered broken and all references to it were removed from kernel 
2.6.9. But a new stateless NAT is coming with the next kernel release 
2.6.24.

For now, an ugly workaround may be to use the NOTRACK target in the 
'raw' table on the (supposedly) return packets, to skip the connection 
tracking and the automagic reverse DNAT. I think this will work for DNS 
over UDP, maybe not so well for TCP.

Re: Problem with DNAT of UDP packets getting undone

[Grant Taylor]

On 11/06/07 14:24, Pascal Hambourg wrote:
> You may confuse with the restriction from some RFCs stating that 
> 127.0.0.0/8 addresses are reserved for internal host use, i.e. the 
> loopback interface. There is no such restriction for other addresses 
> that may be configured on the loopback interface. Also, the Linux IP 
> stack follows the "weak" model by default, so any unicast address 
> (except 127.0.0.0/8) configured on any interface  can be used for 
> communications on any other interface. So any non-127.0.0.0/8 address 
> configured on the loopback interface can be used for communications on 
> any other interface.

Ok.  I did not know for sure as I have not tried this my self and can't 
say for sure one way or another.

> Nope, NAT has nothing to do with this, and the loopback interface is not 
> involved.

In light of the above, agreed.

> The old stateless NAT in the routing code controlled with iproute2 is 
> considered broken and all references to it were removed from kernel 
> 2.6.9. But a new stateless NAT is coming with the next kernel release 
> 2.6.24.

...

> For now, an ugly workaround may be to use the NOTRACK target in the 
> 'raw' table on the (supposedly) return packets, to skip the connection 
> tracking and the automagic reverse DNAT. I think this will work for DNS 
> over UDP, maybe not so well for TCP.

Yes, "Ugly!".



Grant. . . .