* 2.6.11-rc2 TCP ignores PMTU ICMP (Re: Linux 2.6.11-rc2) [not found] ` <200501251054.37053.david-b@pacbell.net> @ 2005-01-27 9:02 ` Janos Farkas 2005-01-27 19:28 ` David Brownell 0 siblings, 1 reply; 9+ messages in thread From: Janos Farkas @ 2005-01-27 9:02 UTC (permalink / raw) To: David Brownell; +Cc: David Ford, linux-kernel, netdev On 2005-01-25 at 10:54:36, David Brownell wrote: > On Tuesday 25 January 2005 10:35 am, David Ford wrote: > > PMTU bug -- or better said, bad firewall admin who blocks all ICMP. > > PMTU bug, sure -- but one that came late in RC2. Remember: same firewall > in both cases, but only RC2 breaks. The ICMP packet has landed in > the RC2 system, which ignores it. 2.6.10 handled it correctly... I > suspect one of the TCP cleanups borked this. > > My current workaround is "ifconfig eth0 mtu 1492" but that's not > something I'd expect to keep. Indeed, I had to shuffle my machines around a bit to get a proof that something is broken, but now I can confirm the above with a connection to cvs.sourceforge.net: 192.168.59.10 is the 2.6.11-rc2 host, 192.168.59.1 is the (now pre-rc2) gateway with a PPPoE 1492 mtu/pmtu host. 59.10 does not run any firewall/packet filter. (almost full-fledged perfect three-way handshake) 08:05:43.046546 IP (tos 0x0, ttl 64, id 21526, offset 0, flags [DF], length: 60) 192.168.59.10.29612 > 66.35.250.207.2401: SWE [tcp sum ok] 933042757:933042757(0) win 5840 <mss 1460,sackOK,timestamp 2707027 0,nop,wscale 2> 08:05:43.249819 IP (tos 0x0, ttl 45, id 0, offset 0, flags [DF], length: 60) 66.35.250.207.2401 > 192.168.59.10.29612: S [tcp sum ok] 2956308271:2956308271(0) ack 933042758 win 5792 <mss 1460,sackOK,timestamp 312382981 2707027,nop,wscale 0> 08:05:43.249881 IP (tos 0x0, ttl 64, id 21528, offset 0, flags [DF], length: 52) 192.168.59.10.29612 > 66.35.250.207.2401: . [tcp sum ok] ack 1 win 1460 <nop,nop,timestamp 2707230 312382981> (CVS protocol starts) 08:05:43.363697 IP (tos 0x0, ttl 64, id 21530, offset 0, flags [DF], length: 116) 192.168.59.10.29612 > 66.35.250.207.2401: P [tcp sum ok] 1:65(64) ack 1 win 1460 <nop,nop,timestamp 2707344 312382981> 08:05:43.568504 IP (tos 0x0, ttl 45, id 14633, offset 0, flags [DF], length: 52) 66.35.250.207.2401 > 192.168.59.10.29612: . [tcp sum ok] ack 65 win 5792 <nop,nop,timestamp 312383013 2707344> 08:05:43.964094 IP (tos 0x0, ttl 45, id 14634, offset 0, flags [DF], length: 63) 66.35.250.207.2401 > 192.168.59.10.29612: P [tcp sum ok] 1:12(11) ack 65 win 5792 <nop,nop,timestamp 312383053 2707344> 08:05:43.964121 IP (tos 0x0, ttl 64, id 21532, offset 0, flags [DF], length: 52) 192.168.59.10.29612 > 66.35.250.207.2401: . [tcp sum ok] ack 12 win 1460 <nop,nop,timestamp 2707945 312383053> 08:05:43.964258 IP (tos 0x0, ttl 64, id 21534, offset 0, flags [DF], length: 440) 192.168.59.10.29612 > 66.35.250.207.2401: P [tcp sum ok] 65:453(388) ack 12 win 1460 <nop,nop,timestamp 2707945 312383053> 08:05:44.184893 IP (tos 0x0, ttl 45, id 14635, offset 0, flags [DF], length: 52) 66.35.250.207.2401 > 192.168.59.10.29612: . [tcp sum ok] ack 453 win 6432 <nop,nop,timestamp 312383075 2707945> 08:05:44.192050 IP (tos 0x0, ttl 45, id 14636, offset 0, flags [DF], length: 566) 66.35.250.207.2401 > 192.168.59.10.29612: P [tcp sum ok] 12:526(514) ack 453 win 6432 <nop,nop,timestamp 312383075 2707945> 08:05:44.192274 IP (tos 0x0, ttl 64, id 21536, offset 0, flags [DF], length: 65) 192.168.59.10.29612 > 66.35.250.207.2401: P [tcp sum ok] 453:466(13) ack 526 win 1728 <nop,nop,timestamp 2708174 312383075> 08:05:44.432709 IP (tos 0x0, ttl 45, id 14637, offset 0, flags [DF], length: 52) 66.35.250.207.2401 > 192.168.59.10.29612: . [tcp sum ok] ack 466 win 6432 <nop,nop,timestamp 312383100 2708174> 08:05:44.893674 IP (tos 0x0, ttl 64, id 21538, offset 0, flags [DF], length: 85) 192.168.59.10.29612 > 66.35.250.207.2401: P [tcp sum ok] 466:499(33) ack 526 win 1728 <nop,nop,timestamp 2708876 312383100> 08:05:45.094985 IP (tos 0x0, ttl 45, id 14638, offset 0, flags [DF], length: 52) 66.35.250.207.2401 > 192.168.59.10.29612: . [tcp sum ok] ack 499 win 6432 <nop,nop,timestamp 312383166 2708876> (first ethernet frame-sized segment is about to be sent) 08:05:45.264633 IP (tos 0x0, ttl 64, id 21540, offset 0, flags [DF], length: 1500) 192.168.59.10.29612 > 66.35.250.207.2401: . [tcp sum ok] 499:1947(1448) ack 526 win 1728 <nop,nop,timestamp 2709247 312383166> 08:05:45.264694 IP (tos 0x0, ttl 64, id 21542, offset 0, flags [DF], length: 1500) 192.168.59.10.29612 > 66.35.250.207.2401: . [tcp sum ok] 1947:3395(1448) ack 526 win 1728 <nop,nop,timestamp 2709247 312383166> (the PPPoE gateway notifies the host about the PMTU issue, one for each segment) 08:05:45.265330 IP (tos 0xc0, ttl 128, id 17970, offset 0, flags [none], length: 576) 192.168.59.1 > 192.168.59.10: icmp 556: 66.35.250.207 unreachable - need to frag (mtu 1492) 08:05:45.265453 IP (tos 0xc0, ttl 128, id 17971, offset 0, flags [none], length: 576) 192.168.59.1 > 192.168.59.10: icmp 556: 66.35.250.207 unreachable - need to frag (mtu 1492) (the source host sends the full frame again...?) 08:05:45.770331 IP (tos 0x0, ttl 64, id 21544, offset 0, flags [DF], length: 1500) 192.168.59.10.29612 > 66.35.250.207.2401: . [tcp sum ok] 499:1947(1448) ack 526 win 1728 <nop,nop,timestamp 2709754 312383166> 08:05:45.770900 IP (tos 0xc0, ttl 128, id 17972, offset 0, flags [none], length: 576) 192.168.59.1 > 192.168.59.10: icmp 556: 66.35.250.207 unreachable - need to frag (mtu 1492) (and looping goes on..) 08:05:46.783148 IP (tos 0x0, ttl 64, id 21546, offset 0, flags [DF], length: 1500) 192.168.59.10.29612 > 66.35.250.207.2401: . [tcp sum ok] 499:1947(1448) ack 526 win 1728 <nop,nop,timestamp 2710768 312383166> 08:05:46.783752 IP (tos 0xc0, ttl 128, id 17973, offset 0, flags [none], length: 576) 192.168.59.1 > 192.168.59.10: icmp 556: 66.35.250.207 unreachable - need to frag (mtu 1492) 08:05:48.808861 IP (tos 0x0, ttl 64, id 21548, offset 0, flags [DF], length: 1500) 192.168.59.10.29612 > 66.35.250.207.2401: . [tcp sum ok] 499:1947(1448) ack 526 win 1728 <nop,nop,timestamp 2712796 312383166> 08:05:48.809882 IP (tos 0xc0, ttl 128, id 17974, offset 0, flags [none], length: 576) 192.168.59.1 > 192.168.59.10: icmp 556: 66.35.250.207 unreachable - need to frag (mtu 1492) ... I don't have now a pre-rc2 host to try this on, but I regularly use sourceforge anoncvs to at least confess honestly that it did work :) -- Janos | romfs is at http://romfs.sourceforge.net/ | Don't talk about silence. ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: 2.6.11-rc2 TCP ignores PMTU ICMP (Re: Linux 2.6.11-rc2) 2005-01-27 9:02 ` 2.6.11-rc2 TCP ignores PMTU ICMP (Re: Linux 2.6.11-rc2) Janos Farkas @ 2005-01-27 19:28 ` David Brownell 2005-01-27 20:00 ` Janos Farkas 2005-01-27 23:11 ` David Brownell 0 siblings, 2 replies; 9+ messages in thread From: David Brownell @ 2005-01-27 19:28 UTC (permalink / raw) To: Janos Farkas; +Cc: David Ford, linux-kernel, netdev On Thursday 27 January 2005 1:02 am, Janos Farkas wrote: > On 2005-01-25 at 10:54:36, David Brownell wrote: > > On Tuesday 25 January 2005 10:35 am, David Ford wrote: > > > PMTU bug -- or better said, bad firewall admin who blocks all ICMP. > > > > PMTU bug, sure -- but one that came late in RC2. Remember: same firewall > > in both cases, but only RC2 breaks. The ICMP packet has landed in > > the RC2 system, which ignores it. 2.6.10 handled it correctly... I > > suspect one of the TCP cleanups borked this. And in fact, I'm pretty sure RC1 handled it fine too. This failure appeared in a BK pull I did a couple days before RC2, and installing it clobbered a "more virgin" RC1 install; so it's hard to verify. It took a while to track the failure mode down to being PMTU related, but the only TCP-related changes I noticed in what I pulled sure seemed like cleanups (removing some "duplicated" state). > > My current workaround is "ifconfig eth0 mtu 1492" but that's not > > something I'd expect to keep. > > Indeed, I had to shuffle my machines around a bit to get a proof that > something is broken, but now I can confirm the above with a connection > to cvs.sourceforge.net: Thanks for confirming it wasn't just me ... I confess I'm a bit surprised more folk haven't reported this yet! Your symptoms are exactly like those I saw, just with a different mission-critical application: CVS, not SMTP. Did you happen to notice whether CVS pulls worked, when pushes (like this) failed? - Dave > ... > > (the PPPoE gateway notifies the host about the PMTU issue, one for each > segment) > > 08:05:45.265330 IP (tos 0xc0, ttl 128, id 17970, offset 0, flags [none], length: 576) 192.168.59.1 > 192.168.59.10: icmp 556: 66.35.250.207 unreachable - need to frag (mtu 1492) > 08:05:45.265453 IP (tos 0xc0, ttl 128, id 17971, offset 0, flags [none], length: 576) 192.168.59.1 > 192.168.59.10: icmp 556: 66.35.250.207 unreachable - need to frag (mtu 1492) > > (the source host sends the full frame again...?) > > 08:05:45.770331 IP (tos 0x0, ttl 64, id 21544, offset 0, flags [DF], length: 1500) 192.168.59.10.29612 > 66.35.250.207.2401: . [tcp sum ok] 499:1947(1448) ack 526 win 1728 <nop,nop,timestamp 2709754 312383166> > 08:05:45.770900 IP (tos 0xc0, ttl 128, id 17972, offset 0, flags [none], length: 576) 192.168.59.1 > 192.168.59.10: icmp 556: 66.35.250.207 unreachable - need to frag (mtu 1492) > > (and looping goes on..) > > 08:05:46.783148 IP (tos 0x0, ttl 64, id 21546, offset 0, flags [DF], length: 1500) 192.168.59.10.29612 > 66.35.250.207.2401: . [tcp sum ok] 499:1947(1448) ack 526 win 1728 <nop,nop,timestamp 2710768 312383166> > 08:05:46.783752 IP (tos 0xc0, ttl 128, id 17973, offset 0, flags [none], length: 576) 192.168.59.1 > 192.168.59.10: icmp 556: 66.35.250.207 unreachable - need to frag (mtu 1492) > 08:05:48.808861 IP (tos 0x0, ttl 64, id 21548, offset 0, flags [DF], length: 1500) 192.168.59.10.29612 > 66.35.250.207.2401: . [tcp sum ok] 499:1947(1448) ack 526 win 1728 <nop,nop,timestamp 2712796 312383166> > 08:05:48.809882 IP (tos 0xc0, ttl 128, id 17974, offset 0, flags [none], length: 576) 192.168.59.1 > 192.168.59.10: icmp 556: 66.35.250.207 unreachable - need to frag (mtu 1492) > > ... > ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: 2.6.11-rc2 TCP ignores PMTU ICMP (Re: Linux 2.6.11-rc2) 2005-01-27 19:28 ` David Brownell @ 2005-01-27 20:00 ` Janos Farkas 2005-01-27 23:11 ` David Brownell 1 sibling, 0 replies; 9+ messages in thread From: Janos Farkas @ 2005-01-27 20:00 UTC (permalink / raw) To: David Brownell; +Cc: David Ford, linux-kernel, netdev On 2005-01-27 at 11:28:48, David Brownell wrote: > > Indeed, I had to shuffle my machines around a bit to get a proof that > > something is broken, but now I can confirm the above with a connection > > to cvs.sourceforge.net: > > Thanks for confirming it wasn't just me ... I confess I'm a bit > surprised more folk haven't reported this yet! > > Your symptoms are exactly like those I saw, just with a different > mission-critical application: CVS, not SMTP. Did you happen to > notice whether CVS pulls worked, when pushes (like this) failed? I'm familiar with symptoms arising from various stochastic/assymetric networking problems with larger frames, I had cable :) This was a pull (update), BTW, CVS can send more data in this situation too. But it turned painfully obvious when trying to send mail to LK via SMTP. Oh, but while plain rc2 may have this problem, this particular endpoint had rc2-bk3 running. I also chose to set the ethernet MTU to 1492 as an easy workaround... -- Janos | romfs is at http://romfs.sourceforge.net/ | Don't talk about silence. ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: 2.6.11-rc2 TCP ignores PMTU ICMP (Re: Linux 2.6.11-rc2) 2005-01-27 19:28 ` David Brownell 2005-01-27 20:00 ` Janos Farkas @ 2005-01-27 23:11 ` David Brownell 2005-01-27 23:41 ` David S. Miller 1 sibling, 1 reply; 9+ messages in thread From: David Brownell @ 2005-01-27 23:11 UTC (permalink / raw) To: Janos Farkas; +Cc: David Ford, linux-kernel, netdev, Art Haas I just got an interesting "I see these problems too" report. It may provide a useful clue. According to "Art Haas" <ahaas@airmail.net>: > I'm running the current BK kernel now, and I'm not seeing the problems > right now because, I found, I do not have some of the IP masquerading > modules installed on my machine. When these modules get installed then > the cvs/rsync problems appear. I do have CONFIG_IP_NF_TARGET_MASQUERADE=y on the system where I'm seeing this, though it's not doing anything just now. Haven't yet made time to see if disabling it improves things ... but if that's a factor, it could explain why more people aren't suffering with this problem. - Dave ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: 2.6.11-rc2 TCP ignores PMTU ICMP (Re: Linux 2.6.11-rc2) 2005-01-27 23:11 ` David Brownell @ 2005-01-27 23:41 ` David S. Miller 2005-01-28 1:33 ` Patrick McHardy 0 siblings, 1 reply; 9+ messages in thread From: David S. Miller @ 2005-01-27 23:41 UTC (permalink / raw) To: David Brownell Cc: jf-ml-k1-1087813225, david+challenge-response, linux-kernel, netdev, ahaas I've forwarded this to netfilter-devel for inspection. Thanks for collecting all the data points so well. ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: 2.6.11-rc2 TCP ignores PMTU ICMP (Re: Linux 2.6.11-rc2) 2005-01-27 23:41 ` David S. Miller @ 2005-01-28 1:33 ` Patrick McHardy 2005-01-28 3:00 ` Rusty Russell ` (2 more replies) 0 siblings, 3 replies; 9+ messages in thread From: Patrick McHardy @ 2005-01-28 1:33 UTC (permalink / raw) To: David S. Miller Cc: David Brownell, jf-ml-k1-1087813225, david+challenge-response, linux-kernel, netdev, ahaas [-- Attachment #1: Type: text/plain, Size: 226 bytes --] David S. Miller wrote: >I've forwarded this to netfilter-devel for inspection. >Thanks for collecting all the data points so well. > Here is the fix for everyone. Please report back if it doesn't solve the problem. Thanks. [-- Attachment #2: x --] [-- Type: text/plain, Size: 487 bytes --] ===== net/ipv4/netfilter/ip_nat_proto_tcp.c 1.10 vs edited ===== --- 1.10/net/ipv4/netfilter/ip_nat_proto_tcp.c 2005-01-17 23:00:55 +01:00 +++ edited/net/ipv4/netfilter/ip_nat_proto_tcp.c 2005-01-28 02:13:06 +01:00 @@ -105,7 +105,7 @@ return 0; iph = (struct iphdr *)((*pskb)->data + iphdroff); - hdr = (struct tcphdr *)((*pskb)->data + iph->ihl*4); + hdr = (struct tcphdr *)((*pskb)->data + hdroff); if (maniptype == IP_NAT_MANIP_SRC) { /* Get rid of src ip and src pt */ ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: 2.6.11-rc2 TCP ignores PMTU ICMP (Re: Linux 2.6.11-rc2) 2005-01-28 1:33 ` Patrick McHardy @ 2005-01-28 3:00 ` Rusty Russell 2005-01-28 14:04 ` Art Haas 2005-01-31 22:13 ` Bill Davidsen 2 siblings, 0 replies; 9+ messages in thread From: Rusty Russell @ 2005-01-28 3:00 UTC (permalink / raw) To: Patrick McHardy Cc: David S. Miller, David Brownell, jf-ml-k1-1087813225, david+challenge-response, lkml - Kernel Mailing List, netdev, ahaas On Fri, 2005-01-28 at 02:33 +0100, Patrick McHardy wrote: > David S. Miller wrote: > > >I've forwarded this to netfilter-devel for inspection. > >Thanks for collecting all the data points so well. > > > Here is the fix for everyone. Please report back if it doesn't > solve the problem. Thanks. Verified by nfsim (the ICMP tests used UDP, I've just added TCP and ICMP, and will do SCTP). Thanks, Rusty. -- A bad analogy is like a leaky screwdriver -- Richard Braakman ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: 2.6.11-rc2 TCP ignores PMTU ICMP (Re: Linux 2.6.11-rc2) 2005-01-28 1:33 ` Patrick McHardy 2005-01-28 3:00 ` Rusty Russell @ 2005-01-28 14:04 ` Art Haas 2005-01-31 22:13 ` Bill Davidsen 2 siblings, 0 replies; 9+ messages in thread From: Art Haas @ 2005-01-28 14:04 UTC (permalink / raw) To: Patrick McHardy Cc: David S. Miller, David Brownell, jf-ml-k1-1087813225, david+challenge-response, linux-kernel, netdev On Fri, Jan 28, 2005 at 02:33:10AM +0100, Patrick McHardy wrote: > Here is the fix for everyone. Please report back if it doesn't > solve the problem. Thanks. > > [ ... snip ... ] Success!!! Art Haas -- Man once surrendering his reason, has no remaining guard against absurdities the most monstrous, and like a ship without rudder, is the sport of every wind. -Thomas Jefferson to James Smith, 1822 ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: 2.6.11-rc2 TCP ignores PMTU ICMP (Re: Linux 2.6.11-rc2) 2005-01-28 1:33 ` Patrick McHardy 2005-01-28 3:00 ` Rusty Russell 2005-01-28 14:04 ` Art Haas @ 2005-01-31 22:13 ` Bill Davidsen 2 siblings, 0 replies; 9+ messages in thread From: Bill Davidsen @ 2005-01-31 22:13 UTC (permalink / raw) To: Patrick McHardy Cc: David S. Miller, David Brownell, jf-ml-k1-1087813225, david+challenge-response, linux-kernel, netdev, ahaas Patrick McHardy wrote: > David S. Miller wrote: > >> I've forwarded this to netfilter-devel for inspection. >> Thanks for collecting all the data points so well. >> > Here is the fix for everyone. Please report back if it doesn't > solve the problem. Thanks. Worked here. > > > > ------------------------------------------------------------------------ > > ===== net/ipv4/netfilter/ip_nat_proto_tcp.c 1.10 vs edited ===== > --- 1.10/net/ipv4/netfilter/ip_nat_proto_tcp.c 2005-01-17 23:00:55 +01:00 > +++ edited/net/ipv4/netfilter/ip_nat_proto_tcp.c 2005-01-28 02:13:06 +01:00 > @@ -105,7 +105,7 @@ > return 0; > > iph = (struct iphdr *)((*pskb)->data + iphdroff); > - hdr = (struct tcphdr *)((*pskb)->data + iph->ihl*4); > + hdr = (struct tcphdr *)((*pskb)->data + hdroff); > > if (maniptype == IP_NAT_MANIP_SRC) { > /* Get rid of src ip and src pt */ -- -bill davidsen (davidsen@tmr.com) "The secret to procrastination is to put things off until the last possible moment - but no longer" -me ^ permalink raw reply [flat|nested] 9+ messages in thread
end of thread, other threads:[~2005-01-31 22:13 UTC | newest]
Thread overview: 9+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
[not found] <200501232251.42394.david-b@pacbell.net>
[not found] ` <41F6916F.7060000@blue-labs.org>
[not found] ` <200501251054.37053.david-b@pacbell.net>
2005-01-27 9:02 ` 2.6.11-rc2 TCP ignores PMTU ICMP (Re: Linux 2.6.11-rc2) Janos Farkas
2005-01-27 19:28 ` David Brownell
2005-01-27 20:00 ` Janos Farkas
2005-01-27 23:11 ` David Brownell
2005-01-27 23:41 ` David S. Miller
2005-01-28 1:33 ` Patrick McHardy
2005-01-28 3:00 ` Rusty Russell
2005-01-28 14:04 ` Art Haas
2005-01-31 22:13 ` Bill Davidsen
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).