* Intermittent NAT failure when multiple hosts send UDP packets @ 2005-09-17 3:01 Bernardo Innocenti 2005-09-18 2:40 ` Bernardo Innocenti 0 siblings, 1 reply; 6+ messages in thread From: Bernardo Innocenti @ 2005-09-17 3:01 UTC (permalink / raw) To: lkml; +Cc: netfilter-devel This smells like a bug in UDP ip_nat_proto_udp.c or nearby. I'm seeing this on 2.6.12-1.1447_FC4, but code in 2.6.13 is still the same. I've setup SNAT the usual way: iptables -A POSTROUTING -t nat -o ppp0 -j SNAT --to-source 151.38.19.110 When multiple clients in the LAN send UDP packets to the same port of the same remote host, I see something like this in my /proc/net/ip_conntrack: udp 17 170 src=10.3.3.2 dst=194.185.88.60 sport=5060 dport=5060 src=194.185.88.60 dst=151.38.19.110 sport=5060 dport=5060 [ASSURED] use=1 udp 17 29 src=10.3.3.2 dst=212.97.59.76 sport=5060 dport=5060 [UNREPLIED] src=212.97.59.76 dst=151.38.19.110 sport=5060 dport=5060 use=1 udp 17 177 src=10.3.3.250 dst=194.185.88.60 sport=5060 dport=5060 src=194.185.88.60 dst=151.38.19.110 sport=5060 dport=1024 [ASSURED] use=1 In the last line, the destination port has been properly remapped from 5060 to 1024 to distingish between incoming packets. However, I see packets going out over ppp0 without the source address properly rewritten to 151.38.19.110: 04:38:28.739514 IP 10.3.3.2.5060 > 194.185.88.60.5060: UDP, length 536 This doesn't happen when there's just a single host sending to port 5060. Sometimes I must restart the interface to trigger this bug. -- // Bernardo Innocenti - Develer S.r.l., R&D dept. \X/ http://www.develer.com/ ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Intermittent NAT failure when multiple hosts send UDP packets 2005-09-17 3:01 Intermittent NAT failure when multiple hosts send UDP packets Bernardo Innocenti @ 2005-09-18 2:40 ` Bernardo Innocenti 2005-09-20 19:35 ` Bernardo Innocenti 0 siblings, 1 reply; 6+ messages in thread From: Bernardo Innocenti @ 2005-09-18 2:40 UTC (permalink / raw) To: Bernardo Innocenti; +Cc: lkml, netfilter-devel Never mind, it was fixed in 2.6.13, probably by this patch: https://lists.netfilter.org/pipermail/netfilter-devel/2004-March/014412.html Bernardo Innocenti wrote: > This smells like a bug in UDP ip_nat_proto_udp.c or nearby. > I'm seeing this on 2.6.12-1.1447_FC4, but code in 2.6.13 is > still the same. > > I've setup SNAT the usual way: > > iptables -A POSTROUTING -t nat -o ppp0 -j SNAT --to-source 151.38.19.110 > > When multiple clients in the LAN send UDP packets to the same port of > the same remote host, I see something like this in my /proc/net/ip_conntrack: > > udp 17 170 src=10.3.3.2 dst=194.185.88.60 sport=5060 dport=5060 src=194.185.88.60 dst=151.38.19.110 sport=5060 dport=5060 [ASSURED] use=1 > udp 17 29 src=10.3.3.2 dst=212.97.59.76 sport=5060 dport=5060 [UNREPLIED] src=212.97.59.76 dst=151.38.19.110 sport=5060 dport=5060 use=1 > udp 17 177 src=10.3.3.250 dst=194.185.88.60 sport=5060 dport=5060 src=194.185.88.60 dst=151.38.19.110 sport=5060 dport=1024 [ASSURED] use=1 > > In the last line, the destination port has been properly remapped from > 5060 to 1024 to distingish between incoming packets. > > However, I see packets going out over ppp0 without the source > address properly rewritten to 151.38.19.110: > > 04:38:28.739514 IP 10.3.3.2.5060 > 194.185.88.60.5060: UDP, length 536 > > This doesn't happen when there's just a single host sending to port 5060. > Sometimes I must restart the interface to trigger this bug. -- // Bernardo Innocenti - Develer S.r.l., R&D dept. \X/ http://www.develer.com/ ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Intermittent NAT failure when multiple hosts send UDP packets 2005-09-18 2:40 ` Bernardo Innocenti @ 2005-09-20 19:35 ` Bernardo Innocenti 2005-09-20 21:15 ` Patrick McHardy 0 siblings, 1 reply; 6+ messages in thread From: Bernardo Innocenti @ 2005-09-20 19:35 UTC (permalink / raw) To: Bernardo Innocenti; +Cc: lkml, netfilter-devel I'm sorry to say that this bug has shown up again on 2.6.13 too, so it's not fixed at all. It's quite hard to trigger, but after it does, packets are consistently routed with the source IP untranslated. Bernardo Innocenti wrote: > Never mind, it was fixed in 2.6.13, probably by this patch: > > https://lists.netfilter.org/pipermail/netfilter-devel/2004-March/014412.html > > > Bernardo Innocenti wrote: > >>This smells like a bug in UDP ip_nat_proto_udp.c or nearby. >>I'm seeing this on 2.6.12-1.1447_FC4, but code in 2.6.13 is >>still the same. >> >>I've setup SNAT the usual way: >> >> iptables -A POSTROUTING -t nat -o ppp0 -j SNAT --to-source 151.38.19.110 >> >>When multiple clients in the LAN send UDP packets to the same port of >>the same remote host, I see something like this in my /proc/net/ip_conntrack: >> >> udp 17 170 src=10.3.3.2 dst=194.185.88.60 sport=5060 dport=5060 src=194.185.88.60 dst=151.38.19.110 sport=5060 dport=5060 [ASSURED] use=1 >> udp 17 29 src=10.3.3.2 dst=212.97.59.76 sport=5060 dport=5060 [UNREPLIED] src=212.97.59.76 dst=151.38.19.110 sport=5060 dport=5060 use=1 >> udp 17 177 src=10.3.3.250 dst=194.185.88.60 sport=5060 dport=5060 src=194.185.88.60 dst=151.38.19.110 sport=5060 dport=1024 [ASSURED] use=1 >> >>In the last line, the destination port has been properly remapped from >>5060 to 1024 to distingish between incoming packets. >> >>However, I see packets going out over ppp0 without the source >>address properly rewritten to 151.38.19.110: >> >> 04:38:28.739514 IP 10.3.3.2.5060 > 194.185.88.60.5060: UDP, length 536 >> >>This doesn't happen when there's just a single host sending to port 5060. >>Sometimes I must restart the interface to trigger this bug. > > -- // Bernardo Innocenti - Develer S.r.l., R&D dept. \X/ http://www.develer.com/ ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Intermittent NAT failure when multiple hosts send UDP packets 2005-09-20 19:35 ` Bernardo Innocenti @ 2005-09-20 21:15 ` Patrick McHardy 2005-09-21 0:11 ` Bernardo Innocenti 0 siblings, 1 reply; 6+ messages in thread From: Patrick McHardy @ 2005-09-20 21:15 UTC (permalink / raw) To: Bernardo Innocenti; +Cc: lkml, netfilter-devel Bernardo Innocenti wrote: > I'm sorry to say that this bug has shown up again on > 2.6.13 too, so it's not fixed at all. > > It's quite hard to trigger, but after it does, packets > are consistently routed with the source IP untranslated. Please try "echo 255 > /proc/sys/net/ipv4/netfilter/ip_conntrack_log_invalid" and modprobe ipt_LOG to see if conntrack ignores them because of invalid checksums or something. ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Intermittent NAT failure when multiple hosts send UDP packets 2005-09-20 21:15 ` Patrick McHardy @ 2005-09-21 0:11 ` Bernardo Innocenti 2005-09-27 21:00 ` Bernardo Innocenti 0 siblings, 1 reply; 6+ messages in thread From: Bernardo Innocenti @ 2005-09-21 0:11 UTC (permalink / raw) To: Patrick McHardy; +Cc: lkml, netfilter-devel Patrick McHardy wrote: > Bernardo Innocenti wrote: > >>I'm sorry to say that this bug has shown up again on >>2.6.13 too, so it's not fixed at all. >> >>It's quite hard to trigger, but after it does, packets >>are consistently routed with the source IP untranslated. > > > Please try "echo 255 > > /proc/sys/net/ipv4/netfilter/ip_conntrack_log_invalid" > and modprobe ipt_LOG to see if conntrack ignores them because > of invalid checksums or something. It doesn't seem to be the case. I only see a few occasional errors, probably caused by miserable hosts crawling with worms: ip_ct_tcp: invalid packet ignored IN= OUT= SRC=83.40.242.177 DST=151.38.19.110 LEN=48 TOS=0x18 PREC=0x20 TTL=114 ID=42553 DF PROTO=TCP SPT=3017 DPT=4662 SEQ=3667342556 ACK=0 WINDOW=65535 RES=0x00 SYN URGP=0 OPT (0204055001010402) device eth1 left promiscuous mode ip_ct_tcp: SEQ is under the lower bound (already ACKed data retransmitted) IN= OUT= SRC=62.241.4.1 DST=151.38.19.110 LEN=40 TOS=0x18 PREC=0x20 TTL=39 ID=62928 PROTO=TCP SPT=995 DPT=41567 SEQ=0 ACK=1313396353 WINDOW=0 RES=0x00 ACK RST URGP=0 ip_ct_icmp: bad ICMP checksum IN= OUT= SRC=68.192.221.135 DST=151.38.19.110 LEN=81 TOS=0x18 PREC=0x20 TTL=49 ID=30671 PROTO=ICMP TYPE=3 CODE=1 [SRC=151.38.19.110 DST=68.192.221.135 LEN=53 TOS=0x00 PREC=0x00 TTL=52 ID=32600 DF PROTO=UDP SPT=13049 DPT=12464 LEN=33 ] ip_ct_tcp: invalid state IN= OUT= SRC=168.226.95.26 DST=151.38.19.110 LEN=52 TOS=0x18 PREC=0x20 TTL=106 ID=26190 DF PROTO=TCP SPT=4662 DPT=40358 SEQ=2466472710 ACK=1425847173 WINDOW=65535 RES=0x00 ACK URGP=0 OPT (0101080A00048B3304B5C54C) ip_ct_tcp: invalid packet ignored IN= OUT= SRC=82.224.25.130 DST=151.38.19.110 LEN=48 TOS=0x18 PREC=0x20 TTL=116 ID=17411 DF PROTO=TCP SPT=3565 DPT=4662 SEQ=1051168262 ACK=0 WINDOW=16384 RES=0x00 SYN URGP=0 OPT (0204058401010402) By the way, all the rest of my TCP and UDP traffic is still being translated and routed correctly: # tcpdump -n -i ppp0 host sip.squillo.it -v -v -v 01:52:38.020005 IP (tos 0x0, ttl 63, id 0, offset 0, flags [DF], proto 17, length: 454) 151.38.19.110.1024 > 194.185.88.60.5060: UDP, length 426 01:52:38.096282 IP (tos 0x38, ttl 55, id 23343, offset 0, flags [DF], proto 17, length: 475) 194.185.88.60.5060 > 151.38.19.110.1024: UDP, length 447 01:52:38.100324 IP (tos 0x38, ttl 55, id 23344, offset 0, flags [DF], proto 17, length: 560) 194.185.88.60.5060 > 151.38.19.110.1024: UDP, length 532 01:52:38.110865 IP (tos 0x0, ttl 63, id 1, offset 0, flags [DF], proto 17, length: 597) 151.38.19.110.1024 > 194.185.88.60.5060: UDP, length 569 01:52:38.216213 IP (tos 0x38, ttl 55, id 23353, offset 0, flags [DF], proto 17, length: 475) 194.185.88.60.5060 > 151.38.19.110.1024: UDP, length 447 01:52:38.238967 IP (tos 0x38, ttl 55, id 23354, offset 0, flags [DF], proto 17, length: 553) 194.185.88.60.5060 > 151.38.19.110.1024: UDP, length 525 It's just this one connection that doesn't: # tcpdump -n -i ppp0 host sip.squillo.it -v -v -v 01:44:04.936695 IP (tos 0x38, ttl 63, id 8416, offset 0, flags [DF], proto 17, length: 564) 10.3.3.2.5060 > 194.185.88.60.5060: UDP, length 536 01:44:06.937433 IP (tos 0x38, ttl 63, id 8418, offset 0, flags [DF], proto 17, length: 564) 10.3.3.2.5060 > 194.185.88.60.5060: UDP, length 536 01:44:10.936911 IP (tos 0x38, ttl 63, id 8420, offset 0, flags [DF], proto 17, length: 564) 10.3.3.2.5060 > 194.185.88.60.5060: UDP, length 536 If I stop transmitting packets until the conntrack timer expires, then everything works again normally for a few hours. If I read ip_nat_core.c correctly, manip_pkt() is responsible for doing the IP address translation, while udp_manip_pkt() takes care of the port in the UDP packet. I'm staring at this code without a clue how it could selectively skip one of the tuples. -- // Bernardo Innocenti - Develer S.r.l., R&D dept. \X/ http://www.develer.com/ ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Intermittent NAT failure when multiple hosts send UDP packets 2005-09-21 0:11 ` Bernardo Innocenti @ 2005-09-27 21:00 ` Bernardo Innocenti 0 siblings, 0 replies; 6+ messages in thread From: Bernardo Innocenti @ 2005-09-27 21:00 UTC (permalink / raw) To: Bernardo Innocenti; +Cc: Patrick McHardy, lkml, netfilter-devel Bernardo Innocenti wrote: > Patrick McHardy wrote: >>Bernardo Innocenti wrote: >> >>>It's quite hard to trigger, but after it does, packets >>>are consistently routed with the source IP untranslated. >> >>Please try "echo 255 > >>/proc/sys/net/ipv4/netfilter/ip_conntrack_log_invalid" >>and modprobe ipt_LOG to see if conntrack ignores them because >>of invalid checksums or something. > > It doesn't seem to be the case. I only see a few occasional > errors, probably caused by miserable hosts crawling with worms: PROBLEM SOLVED! I'm glad to say It's *almost* not a kernel bug, more like a missing feature. In my ip-up.local script, I do: echo "Loading NAT modules:" /sbin/modprobe ip_conntrack /sbin/modprobe ip_conntrack_ftp /sbin/modprobe ip_conntrack_irc /sbin/modprobe iptable_nat /sbin/modprobe ip_nat_ftp /sbin/modprobe ip_nat_irc [...several filtering and QoS rules...] $iptab -A POSTROUTING -t nat -o $IFNAME -j SNAT --to-source $IPLOCAL My ip-down.local attempts to do the opposite: echo "Flushing all current rules:" $iptab -F $iptab -F -t nat echo "Clearing all chains:" $iptab -X $iptab -Z $iptab -X -t nat $iptab -Z -t nat echo "Removing NAT modules:" /sbin/rmmod ip_nat_ftp ip_nat_irc iptable_nat ip_conntrack ip_conntrack_ftp ip_conntrack_irc Note the order of the modules in the last line: ip_conntrack cannot be unloaded because it's still being used by ip_conntrack_ftp and ip_conntrack_irc. So, whenever the PPP link goes down, ip_conntrack remains loaded with all connections still being tracked until the timer expires! If ppp0 goes up again soon enough, the script reloads the ip_nat modules, but the existing connections are no longer being translated. Would it be possible to do something at module initialization time to recover those connections? Meanwhile, I've fixed my rmmod to make sure ip_conntrack really gets unloaded. -- // Bernardo Innocenti - Develer S.r.l., R&D dept. \X/ http://www.develer.com/ ^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2005-09-27 21:00 UTC | newest] Thread overview: 6+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2005-09-17 3:01 Intermittent NAT failure when multiple hosts send UDP packets Bernardo Innocenti 2005-09-18 2:40 ` Bernardo Innocenti 2005-09-20 19:35 ` Bernardo Innocenti 2005-09-20 21:15 ` Patrick McHardy 2005-09-21 0:11 ` Bernardo Innocenti 2005-09-27 21:00 ` Bernardo Innocenti
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox