Re: conntrack NAT problem w/ UDP

Re: conntrack NAT problem w/ UDP

[Patrick McHardy]

chip@innovates.com wrote:
> The packet rewriting is not happening for the outgoing packets, 
> therefore the other end can never reply because it has a private IP 
> address being sent as the source address.  
> 
> The timeout issue has been thoroughly debunked by numerous people.  
> Asterisk sends a packet every 30 seconds to keep NAT connections alive.  
> 
> 
> The problem is definitely something getting lost in netfilter.  When the 
> problem occurs, DNS and other UDP traffic with different source ports 
> than destination ports are still correctly rewritten.  The last time the 
> problem occurred for me, I completely reload the all the tables and 
> tried using SNAT vs. MASQUERADE.  Nothing made any packets get rewritten 
> for Asterisk connections except rebooting the router.  Next time, I will 
> try unloading all the netfilter modules and reloading them before 
> rebooting.  
> 
> When this happens I don't have much time for debugging, because of the 
> urgency to get a production phone system back in service.  That is why 
> I'm asking what should I examine to find out where netfilter is missing 
> this connection?

You could monitor the conntrack events with the conntrack tool.
A table dump while the problem occurs might also help.

RE: conntrack NAT problem w/ UDP

[chip]

[-- Attachment #1: BDY.TXT --]
[-- Type: text/plain, Size: 2566 bytes --]

The packet rewriting is not happening for the outgoing packets, 
therefore the other end can never reply because it has a private IP 
address being sent as the source address.  

The timeout issue has been thoroughly debunked by numerous people.  
Asterisk sends a packet every 30 seconds to keep NAT connections alive.  


The problem is definitely something getting lost in netfilter.  When the 
problem occurs, DNS and other UDP traffic with different source ports 
than destination ports are still correctly rewritten.  The last time the 
problem occurred for me, I completely reload the all the tables and 
tried using SNAT vs. MASQUERADE.  Nothing made any packets get rewritten 
for Asterisk connections except rebooting the router.  Next time, I will 
try unloading all the netfilter modules and reloading them before 
rebooting.  

When this happens I don't have much time for debugging, because of the 
urgency to get a production phone system back in service.  That is why 
I'm asking what should I examine to find out where netfilter is missing 
this connection?

Chip Schweiss



-----Original Message-----
From: bof [mailto:bof@bof.de] 
Sent: Wednesday, December 28, 2005 1:16 AM
To: Chip Schweiss
Cc: netfilter-devel
Subject: Re: conntrack NAT problem w/ UDP

> There seems to be an intermittent problem w/ NAT when the source port 
> and destination port are the same on UDP traffic.
[...]
> I diagnosed the problem by sniffing the traffic between the router and 

> internet device (DSL or cable modem).

How much time was there between the last correctly rewritten packet,
and the first bad packet?

I bet it was more than 120 seconds.

UDP is connectionless. Thus, conntracking must use some form of
idle (no packets flowing) timeout for its NAT entries. The timeout
is 120 seconds for replied (packets seen in both directions)
UDP connections.

With the setup you described, having fixed "real" ports, maybe
you can just set up static nat (use the SNAT or DNAT target after
matching client IP address), instead of dynamic (random mangled port)
NAT like MASQUERADE, and your problem will go away - because the 
decision
will be the same each time a new conntrack is created after a pause.

You could also try to work around the problem by making the application
send data in the correct direction more often than each 120 seconds.

Or you could manipulate the UDP timeout on the NAT box. Look/change
/proc/sys/net/ipv4/netfilter/ip_conntrack_udp_stream. However, this
will affect all UDP traffic (traceroute, DNS lookups).

best regards
  Patrick

Re: conntrack NAT problem w/ UDP

[Patrick Schaaf]

> There seems to be an intermittent problem w/ NAT when the source port 
> and destination port are the same on UDP traffic.
[...]
> I diagnosed the problem by sniffing the traffic between the router and 
> internet device (DSL or cable modem).

How much time was there between the last correctly rewritten packet,
and the first bad packet?

I bet it was more than 120 seconds.

UDP is connectionless. Thus, conntracking must use some form of
idle (no packets flowing) timeout for its NAT entries. The timeout
is 120 seconds for replied (packets seen in both directions)
UDP connections.

With the setup you described, having fixed "real" ports, maybe
you can just set up static nat (use the SNAT or DNAT target after
matching client IP address), instead of dynamic (random mangled port)
NAT like MASQUERADE, and your problem will go away - because the decision
will be the same each time a new conntrack is created after a pause.

You could also try to work around the problem by making the application
send data in the correct direction more often than each 120 seconds.

Or you could manipulate the UDP timeout on the NAT box. Look/change
/proc/sys/net/ipv4/netfilter/ip_conntrack_udp_stream. However, this
will affect all UDP traffic (traceroute, DNS lookups).

best regards
  Patrick

conntrack NAT problem w/ UDP

[chip]

[-- Attachment #1: BDY.TXT --]
[-- Type: text/plain, Size: 1223 bytes --]

There seems to be an intermittent problem w/ NAT when the source port 
and destination port are the same on UDP traffic.

I've found this on two routers, one with OpenWRT whiterussian RC3 
(kernel 2.4.30) and on SuSE 10 (kernel 2.6.13-15).  At random times NAT 
fails to operate on the UDP traffic for IAX2 from Asterisk.  IAX2 is UDP 
port 4569 to port 4569.  The kernel will stop rewriting the source port 
when using either SNAT or MASQUERADE.  

Rebooting the kernel is the only solution I have found to fixing the 
problem.  I have found numerous reports of this problem.  It's usually 
reported as Asterisk failing to register.  

I diagnosed the problem by sniffing the traffic between the router and 
internet device (DSL or cable modem).  This revealed the source address 
is not being rewritten.  

Any suggestions on how to further trace where the problem in netfilter 
is occurring would be of great benefit.  Unfortunately, I only see this 
problem once every couple months.  So I need to know where to look for 
more information on the problem.  

I've posted this bug in bugzilla ID #421.  


Chip Schweiss

Innovates, Inc.
4038 Avery Ln.
Bridgeton, MO 63044
(314) 298-3708 - (618) 207-3442 - (877) 444-2003