NAT and DNS/NTP servers

NAT and DNS/NTP servers

[Andrew E. Mileski]

I'm observing lost packets when a gateway doing NAT also locally hosts
a server for DNS or NTP.  I believe this the result of the ambiguous
conditions that can exist when routing server-to-server packets coming
into the gateway.

I can avoid it by specifying --to-source ports in a SNAT rule for NTP,
or with DNAT and --to-destination port to force usage of the gateway's
server (which is likely the best solution).  I've not tried this on DNS
yet, but I expect it will solve the problem as well.

Is this a known situation for these protocols (and similar)?  Or am I
barking up the wrong tree?

Thanks.

-- 
Andrew E. Mileski

Re: NAT and DNS/NTP servers

[Antony Stone]

On Saturday 01 May 2004 3:52 am, Andrew E. Mileski wrote:

> I'm observing lost packets when a gateway doing NAT also locally hosts
> a server for DNS or NTP.  I believe this the result of the ambiguous
> conditions that can exist when routing server-to-server packets coming
> into the gateway.

Show us your rules and explain what is ambiguous about them?

> I can avoid it by specifying --to-source ports in a SNAT rule for NTP,

If you are hosting NTP on the machine running netfilter, why would you want a 
SNAT rule?   Show us an example?

> or with DNAT and --to-destination port to force usage of the gateway's
> server (which is likely the best solution).

Why?   What address are the clients sending the packets to (such that they 
need DNATting in order to be handled by the server you're talking about)?

> Is this a known situation for these protocols (and similar)?  Or am I
> barking up the wrong tree?

This does not sound like a familiar problem (either which I have seen myself, 
or heard other people discussing).   I think if we saw a picture of your tree 
we could tell you if it was the wrong one to bark up (ie: show us your 
ruleset and explain what goes wrong with it).

Regards,

Antony.

-- 
Success is a lousy teacher.  It seduces smart people into thinking they can't 
lose.

 - William H Gates III

                                                     Please reply to the list;
                                                           please don't CC me.

Re: NAT and DNS/NTP servers

[Andrew E. Mileski]

Antony Stone wrote:
> On Saturday 01 May 2004 3:52 am, Andrew E. Mileski wrote:
> 
>>I'm observing lost packets when a gateway doing NAT also locally hosts
>>a server for DNS or NTP.  I believe this the result of the ambiguous
>>conditions that can exist when routing server-to-server packets coming
>>into the gateway.
> 
> Show us your rules and explain what is ambiguous about them?

There is only one rule needed if all other defaults are ACCEPT:

This one is on my network:
iptables -t nat -A POSTROUTING -s ! $WAN_IP -o $WAN_IFC -j SNAT 
--to-source $WAN_IP

This one is on another network with the same problem:
iptables -t nat -A POSTROUTING -s $INTERNAL_NET -j MASQUERADE

Both DNS and NTP server-to-server UDP packets have the same source
and destination ports.  What I am seeing is a failure to remap the
source port on outgoing packets when a local server is running on
the gateway.

I speculate this is because an idle local server without an active
connection doesn't have an entry in the connection table.  Though
I've also seen failures with an active local server, but I haven't
got a good theory on the mechanics behind it yet.

I have seen port remapping in some situations, so I know that
feature works.  It just isn't being triggered in some situations,
which I can't yet reason why.

As an aside, I think (S?)NTP pays attention to the source port, so
a port remap may have side effects for this protocol.  This isn't
a concern of mine though.

-- 
Andrew E. Mileski

Re: NAT and DNS/NTP servers

[Andrew E. Mileski]

Andrew E. Mileski wrote:
> I speculate this is because an idle local server without an active
> connection doesn't have an entry in the connection table.  Though
> I've also seen failures with an active local server, but I haven't
> got a good theory on the mechanics behind it yet.

I just realized this theory is bogus too, as it should be easy to
tell there is a listener on the local port.  I assume this is
probably checked.

-- 
Andrew E. Mileski

Re: NAT and DNS/NTP servers

[Antony Stone]

On Saturday 01 May 2004 6:42 pm, Andrew E. Mileski wrote:

> Antony Stone wrote:
> > On Saturday 01 May 2004 3:52 am, Andrew E. Mileski wrote:
> >>I'm observing lost packets when a gateway doing NAT also locally hosts
> >>a server for DNS or NTP.  I believe this the result of the ambiguous
> >>conditions that can exist when routing server-to-server packets coming
> >>into the gateway.
> >
> > Show us your rules and explain what is ambiguous about them?
>
> There is only one rule needed if all other defaults are ACCEPT:
>
> This one is on my network:
> iptables -t nat -A POSTROUTING -s ! $WAN_IP -o $WAN_IFC -j SNAT
> --to-source $WAN_IP

What is $WAN_IP, and what are the IP addresses on the machine on which this 
rule is running?

I still don't really understand why you need a SNAT rule on a machine which is 
running the service locally?

Please give a clear explanation of where the client is, where the server is, 
and why you want a SNAT rule in the system.   Also, please explain what is 
"ambiguous" about all this, as you described it - I don't understand what you 
mean by that in this context.

Regards,

Antony.

-- 
Never write it in Perl if you can do it in Awk.
Never do it in Awk if sed can handle it.
Never use sed when tr can do the job.
Never invoke tr when cat is sufficient.
Avoid using cat whenever possible.

                                                     Please reply to the list;
                                                           please don't CC me.

Re: NAT and DNS/NTP servers

[Andrew E. Mileski]

Antony Stone wrote:
> On Saturday 01 May 2004 6:42 pm, Andrew E. Mileski wrote:
>>There is only one rule needed if all other defaults are ACCEPT:
>>
>>This one is on my network:
>>iptables -t nat -A POSTROUTING -s ! $WAN_IP -o $WAN_IFC -j SNAT
>>--to-source $WAN_IP
> 
> What is $WAN_IP, and what are the IP addresses on the machine on which this 
> rule is running?

LAN is a private network:
   LAN_IFC=eth0
   LAN_NET=192.168.0/24
   LAN_IP=192.168.1.1

WAN is the internet:
   WAN_IFC=eth1
   WAN_NET=209.217.118.0/29
   WAN_IP=209.217.118.226

> I still don't really understand why you need a SNAT rule on a machine which is 
> running the service locally?

The machine is a gateway to the internet as well.  The gateway has to
use SNAT to allow private hosts to reach the internet for other
services, like say HTTP.  Private hosts are not restricted to the
services provided by the gateway.

> Please give a clear explanation of where the client is, where the server is, 
> and why you want a SNAT rule in the system.   Also, please explain what is 
> "ambiguous" about all this, as you described it - I don't understand what you 
> mean by that in this context.

A private host (192.168.1.2) on a private network (192.168.1.0/24) that
is contacting a remote server on the internet.

I used the term "ambiguous" to attempt to describe the situation I'm
seeing where there are two possible destinations for a reply packet
coming from the internet.  The packet can be routed to the server
process on the gateway machine, or to a private host.

As I've stated I'm unclear on exactly what is happening, so don't
attribute too much weight to it or my phraseology.  I'm attempting to
put words to a process I can not yet fully understand or explain.  My
appologies for this.

For example:

The NTP server process on the gateway sends a UDP packet to a NTP server
on internet, let's say time.nist.gov.  The packet sent out WAN_IFC has:
   source address: WAN_IP = 209.217.118.226
   source port: 123
   destination address: time.nist.gov = 192.43.244.18
   destination port: 123
A reply packet from time.nist.gov arrives on WAN_IFC with:
   source address: time.nist.gov = 192.43.244.18
   source port: 123
   destination address: WAN_IP = 209.217.118.226
   destination port: 123

When a private client (192.168.1.2) sends a packet to the same place,
the packet arriving on the LAN_IFC has:
   source address: private client = 192.168.1.2
   source port: 123
   destination address: time.nist.gov = 192.43.244.18
   destination port: 123
This gets SNAT'ed by the gateway and sent out the WAN_IFC to the
internet looking like this:
   source address: WAN_IP = 209.217.118.226
   source port: 123
   destination address: time.nist.gov = 192.43.244.18
   destination port: 123
A reply packet from time.nist.gov arrives on WAN_IFC with:
   source address: time.nist.gov = 192.43.244.18
   source port: 123
   destination address: WAN_IP = 66.11.173.24
   destination port: 123
Because the port wasn't remapped, there appears to be ambiguity
in whether the reply packet should be routed to the private host
(192.168.1.2) or the NTP server process on the router.

-- 
Andrew E. Mileski

Re: NAT and DNS/NTP servers

[Andrew E. Mileski]

Andrew E. Mileski wrote:
> When a private client (192.168.1.2) sends a packet to the same place,
> the packet arriving on the LAN_IFC has:
>   source address: private client = 192.168.1.2
>   source port: 123
>   destination address: time.nist.gov = 192.43.244.18
>   destination port: 123
> This gets SNAT'ed by the gateway and sent out the WAN_IFC to the
> internet looking like this:
>   source address: WAN_IP = 209.217.118.226
>   source port: 123
>   destination address: time.nist.gov = 192.43.244.18
>   destination port: 123
> A reply packet from time.nist.gov arrives on WAN_IFC with:
>   source address: time.nist.gov = 192.43.244.18
>   source port: 123
>   destination address: WAN_IP = 66.11.173.24
>   destination port: 123
> Because the port wasn't remapped, there appears to be ambiguity
> in whether the reply packet should be routed to the private host
> (192.168.1.2) or the NTP server process on the router.

Doh! made a mistake the WAN_IP in the reply above should
be 209.217.118.226 not 66.11.173.24.  Sorry.  [That's why
I tried to avoid typing numbers originally; to avoid mistakes.]

-- 
Andrew E. Mileski

Re: NAT and DNS/NTP servers

[Antony Stone]

On Saturday 01 May 2004 8:51 pm, Andrew E. Mileski wrote:

> Antony Stone wrote:
> > On Saturday 01 May 2004 6:42 pm, Andrew E. Mileski wrote:
> >>There is only one rule needed if all other defaults are ACCEPT:
> >>
> >>This one is on my network:
> >>iptables -t nat -A POSTROUTING -s ! $WAN_IP -o $WAN_IFC -j SNAT
> >>--to-source $WAN_IP
> >
> > What is $WAN_IP, and what are the IP addresses on the machine on which
> > this rule is running?
>
>    WAN_IP=209.217.118.226
>
> > I still don't really understand why you need a SNAT rule on a machine
> > which is running the service locally?
>
> For example:
>
> The NTP server process on the gateway sends a UDP packet to a NTP server
> on internet, let's say time.nist.gov.

Okay, I think you mean that the NTP *client* process on the gateway sends a 
packet to an NTP server on the Internet.   I was getting confused because you 
kept talking about a server, and I couldn't see where SNAT would come into 
that, however for a client process I can understand it.

> The packet sent out WAN_IFC has:
>    source address: WAN_IP = 209.217.118.226
>    source port: 123
>    destination address: time.nist.gov = 192.43.244.18
>    destination port: 123
> A reply packet from time.nist.gov arrives on WAN_IFC with:
>    source address: time.nist.gov = 192.43.244.18
>    source port: 123
>    destination address: WAN_IP = 209.217.118.226
>    destination port: 123

That all seems perfectly clear and reasonable.

> When a private client (192.168.1.2) sends a packet to the same place,
> the packet arriving on the LAN_IFC ... gets SNAT'ed by the gateway and sent
> out the WAN_IFC to the internet looking like this:
>    source address: WAN_IP = 209.217.118.226
>    source port: 123
>    destination address: time.nist.gov = 192.43.244.18
>    destination port: 123

Yup - I agree with that as well.   The netfilter code will not change the 
source port of a packet unless that source port is already bound to an 
existing connection.

> Because the port wasn't remapped, there appears to be ambiguity
> in whether the reply packet should be routed to the private host
> (192.168.1.2) or the NTP server process on the router.

If there is a connection tracking table entry (because the packet came from a 
LAN client) then the reply packet will be routed back to the client.   If 
there isn't, then it won't be.   I see no opportunity for ambiguity because 
netfilter is stateful.

I do not think that netfilter will ever get "confused" about whether a reply 
packet is in response to a locally-generated request, or a NATted connection 
from a LAN client machine.   The gateway which you are running netfilter on 
will always keep an accurate record of (a) locally-bound ports, and (b) 
connection tracking table entries for packets originating elsewhere.

What makes you think that netfilter is getting anything wrong?   Are you 
seeing packets in LOG entries which do not seem to correspond to current 
connections?   If you are simply seeing connections not succeeding from time 
to time, do keep in mind that this is UDP we're talking about, which although 
is strictly known as User Datagram Protocol, can be fairly accurately 
interpreted as Unreliable Datagram Protocol (unlike TCP, which is a reliable 
service).

Regards,

Antony.

-- 
You can spend the whole of your life trying to be popular,
but at the end of the day the size of the crowd at your funeral
will be largely dictated by the weather.

 - Frank Skinner

                                                     Please reply to the list;
                                                           please don't CC me.

Re: NAT and DNS/NTP servers

[Andrew E. Mileski]

Antony Stone wrote:
> 
> I do not think that netfilter will ever get "confused" about whether a reply 
> packet is in response to a locally-generated request, or a NATted connection 
> from a LAN client machine.   The gateway which you are running netfilter on 
> will always keep an accurate record of (a) locally-bound ports, and (b) 
> connection tracking table entries for packets originating elsewhere.
> 
> What makes you think that netfilter is getting anything wrong?   Are you 
> seeing packets in LOG entries which do not seem to correspond to current 
> connections?   If you are simply seeing connections not succeeding from time 
> to time, do keep in mind that this is UDP we're talking about, which although 
> is strictly known as User Datagram Protocol, can be fairly accurately 
> interpreted as Unreliable Datagram Protocol (unlike TCP, which is a reliable 
> service).

This is from a different machine (sorry), but my point is the same...

1) Make sure there are no existing entries
    # grep '123' /proc/net/ip_conntrack
    #

2) Start up a NTP server on the gateway with a poll period long enough
    to allow an entry in /proc/net/ip_conntrack to expire (I used
    minpoll 9 maxpoll 9)
    # grep '123' /proc/net/ip_conntrack
udp      17 166 src=66.11.173.24 dst=192.43.244.18 sport=123 dport=123 
src=192.43.244.18 dst=66.11.173.24 sport=123 dport=123 [ASSURED] use=1

3) Start up a NTP server on a private client with a short poll period
    (I used minpoll 4 maxpoll 4)
    # grep '123' /proc/net/ip_conntrack
udp      17 177 src=66.11.173.24 dst=192.43.244.18 sport=123 dport=123 
src=192.43.244.18 dst=66.11.173.24 sport=123 dport=123 [ASSURED] use=1
udp      17 162 src=192.168.2.244 dst=192.43.244.18 sport=123 dport=123 
src=192.43.244.18 dst=66.11.173.24 sport=123 dport=12 [ASSURED] use=1

The port was remapped to 12.  All is okay.

4) Stop the private host NTP server, keeping the gateway NTP server
    listening (long poll time remember), and make sure there are no
    existing entries (wait for any to expire)
    # grep '123' /proc/net/ip_conntrack
    #

5) Start a NTP server on a private client again with the short poll
    period
    # grep '123' /proc/net/ip_conntrack
udp      17 167 src=192.168.2.244 dst=192.43.244.18 sport=123 dport=123 
src=192.43.244.18 dst=66.11.173.24 sport=123 dport=123 [ASSURED] use=1

The port wasn't remapped... incoming packets will ALWAYS be routed
to this machine.  The NTP server process on the gateway will never
receive packets while this connecection exists.

It seems netfilter doesn't check for the existence of a local listener
on a port before deciding whether or not to remap it.  Only the
connection table seems to be referenced.

Bug or feature, I don't know.  I just wanted confirmation that this
exists.  I stated earlier that there are two ways around it.

-- 
Andrew E. Mileski

Re: NAT and DNS/NTP servers

[Antony Stone]

On Saturday 01 May 2004 9:48 pm, Andrew E. Mileski wrote:

> It seems netfilter doesn't check for the existence of a local listener
> on a port before deciding whether or not to remap it.  Only the
> connection table seems to be referenced.

This is correct, because remapping is only relevant to packets leaving the 
machine (this machine acting as client), and a local listener is only 
relevant to packets arriving at the machine (this machine acting as server).

I remain confused from your description as to whether you are talking about 
the machine running the netfilter rules being an NTP server (servicing 
requests from other clients), or an NTP client (sending requests to other 
servers).

It may be that the problem you are experiencing is simply due to the default 
timeout on UDP "connections" (which is an artificial concept anyway, built 
into netfilter simply to try and make UDP conform to the stateful packet 
monitoring mechanism), and you need to adjust the timings by fiddling about 
with /proc/sys/net/ipv4/netfilter/ip_conntrack_udp_timeout ?

Regards,

Antony.
-- 
If you want to be happy for an hour, get drunk.
If you want to be happy for a year, get married.
If you want to be happy for a lifetime, get a garden.

                                                     Please reply to the list;
                                                           please don't CC me.

Re: NAT and DNS/NTP servers

[Andrew E. Mileski]

Antony Stone wrote:
> On Saturday 01 May 2004 9:48 pm, Andrew E. Mileski wrote:
>>It seems netfilter doesn't check for the existence of a local listener
>>on a port before deciding whether or not to remap it.  Only the
>>connection table seems to be referenced.
> 
> This is correct, because remapping is only relevant to packets leaving the 
> machine (this machine acting as client), and a local listener is only 
> relevant to packets arriving at the machine (this machine acting as server).
> 
> I remain confused from your description as to whether you are talking about 
> the machine running the netfilter rules being an NTP server (servicing 
> requests from other clients), or an NTP client (sending requests to other 
> servers).

The machine running the netfilter rules is an NTP and DNS server
(servicing requests from other clients), _and_ a NTP and DNS client
(sending requests to other servers).  This is the nature of both
of these protocols (peering).

The machine running the netfilter rules is also a gateway, passing
requests from a private network to servers on the internet.

> It may be that the problem you are experiencing is simply due to the default 
> timeout on UDP "connections" (which is an artificial concept anyway, built 
> into netfilter simply to try and make UDP conform to the stateful packet 
> monitoring mechanism), and you need to adjust the timings by fiddling about 
> with /proc/sys/net/ipv4/netfilter/ip_conntrack_udp_timeout ?

That will work on one condition:  a guarantee can be made that the
server on the gateway will get a packet out before forwarding one
for a private host.  The gateway server must own the first entry
in the conntrack table for the port.

Otherwise the time workaround can easily be done with NTP, by setting
ip_conntrack_udp_timeout to an amount longer than the poll period of the
NTP server on the gateway.

However, with DNS one never knows when the server will go out to
the internet.  It can be quite a long time if all serviced requests
reference cached entries.

Hence my solutions (both tested) of an explicit rule to either:
  A) Force all private hosts to use the DNS/NTP servers on the gateway.
  B) Force a port remap on all DNS/NTP packets from private hosts
     passing through the gateway.

-- 
Andrew E. Mileski

Re: NAT and DNS/NTP servers

[Antony Stone]

On Saturday 01 May 2004 10:42 pm, Andrew E. Mileski wrote:

> Antony Stone wrote:
> >
> > I remain confused from your description as to whether you are talking
> > about the machine running the netfilter rules being an NTP server
> > (servicing requests from other clients), or an NTP client (sending
> > requests to other servers).
>
> The machine running the netfilter rules is an NTP and DNS server
> (servicing requests from other clients), _and_ a NTP and DNS client
> (sending requests to other servers).
>
> Hence my solutions (both tested) of an explicit rule to either:
>   A) Force all private hosts to use the DNS/NTP servers on the gateway.

This sounds like a very good idea, for both protocols, since:

a) if you have a caching DNS server on your network, you may as well make 
maximum use of it, instead of sending requests out to the Internet direct 
from individual clients

b) if you have an NTP server on your network, it is more important that all 
your local machines are synchronised to it than that some of them are 
correct, but different from others.

Therefore I think pointing all local clients at the DNS / NTP servers on the 
gateway, and then pointing the single DNS / NTP clients running on the 
gateway to the outside world, is the correct solution.

Regards,

Antony.

-- 
I want to build a machine that will be proud of me.

 - Danny Hillis, creator of The Connection Machine

                                                     Please reply to the list;
                                                           please don't CC me.

RE: NAT and DNS/NTP servers

[Steve Jones]

If you didn't just make a typo in the transcription of this, you may
want to check your LAN side address.  Your LAN_NET in this is not the
subnet that your LAN_IP is on..  I wonder if this is confusing your
rules?
-Steve

-----Original Message-----

LAN is a private network:
   LAN_IFC=eth0
   LAN_NET=192.168.0/24
   LAN_IP=192.168.1.1

WAN is the internet:
   WAN_IFC=eth1
   WAN_NET=209.217.118.0/29
   WAN_IP=209.217.118.226

Re: NAT and DNS/NTP servers

[Andrew E. Mileski]

Steve Jones wrote:
> If you didn't just make a typo in the transcription of this, you may
> want to check your LAN side address.  Your LAN_NET in this is not the
> subnet that your LAN_IP is on..  I wonder if this is confusing your
> rules?

Keen eye, but it was indeed a typo.  Sorry.

-- 
Andrew E. Mileski