Re: Linux 2.6.11-rc2

Re: Linux 2.6.11-rc2

[David Brownell]

I'm seeing a problem with TCP as accessed through KMail (SuSE 9.2, x86_64).
But oddly enough, only for sending mail, not reading it; and not through
other (reading) applications... it's a regression with respect to rc1 and
earlier kernels.  Basically, it can only send REALLY TINY emails...

What ethereal shows me is roughly:

 - SMTP connect, initial handshake, ok (ACKed later)
 - Send two 1500 byte Ethernet packets
 - Each gets an ICMP destination unreachable, frag needed, next hop MTU 1492
 - ... all retransmits are 1500 bytes not 1492, triggering ICMPs ...

Naturally the connection goes nowhere.  

- Dave

Re: Linux 2.6.11-rc2

[Martin Josefsson]

[-- Attachment #1: Type: text/plain, Size: 800 bytes --]

On Sun, 2005-01-23 at 22:51 -0800, David Brownell wrote:
> I'm seeing a problem with TCP as accessed through KMail (SuSE 9.2, x86_64).
> But oddly enough, only for sending mail, not reading it; and not through
> other (reading) applications... it's a regression with respect to rc1 and
> earlier kernels.  Basically, it can only send REALLY TINY emails...
> 
> What ethereal shows me is roughly:
> 
>  - SMTP connect, initial handshake, ok (ACKed later)
>  - Send two 1500 byte Ethernet packets
>  - Each gets an ICMP destination unreachable, frag needed, next hop MTU 1492
>  - ... all retransmits are 1500 bytes not 1492, triggering ICMPs ...
> 
> Naturally the connection goes nowhere.  

Is there a firewall on this machine? And if so, do you allow inbound
icmp?

-- 
/Martin

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: Linux 2.6.11-rc2

[David Brownell]

On Monday 24 January 2005 10:05 am, Martin Josefsson wrote:
> On Sun, 2005-01-23 at 22:51 -0800, David Brownell wrote:
> > ...
> >  - Each gets an ICMP destination unreachable, frag needed, next hop MTU 1492
> >  - ... all retransmits are 1500 bytes not 1492, triggering ICMPs ...
> > 
> > Naturally the connection goes nowhere.  
> 
> Is there a firewall on this machine? And if so, do you allow inbound
> icmp?

It's behind a firewall; no firewall on that box though.  The only
difference between a working RC1 and non-working RC2 was the kernel.

- Dave

Re: Linux 2.6.11-rc2

[David Ford]

PMTU bug -- or better said, bad firewall admin who blocks all ICMP.

http://blue-labs.org/clue/mtu-mss.php

-david

David Brownell wrote:

>I'm seeing a problem with TCP as accessed through KMail (SuSE 9.2, x86_64).
>But oddly enough, only for sending mail, not reading it; and not through
>other (reading) applications... it's a regression with respect to rc1 and
>earlier kernels.  Basically, it can only send REALLY TINY emails...
>
>  
>

Re: Linux 2.6.11-rc2

[David Brownell]

On Tuesday 25 January 2005 10:35 am, David Ford wrote:
> PMTU bug -- or better said, bad firewall admin who blocks all ICMP.

PMTU bug, sure -- but one that came late in RC2.  Remember:  same firewall
in both cases, but only RC2 breaks.  The ICMP packet has landed in
the RC2 system, which ignores it.  2.6.10 handled it correctly... I
suspect one of the TCP cleanups borked this.

My current workaround is "ifconfig eth0 mtu 1492" but that's not
something I'd expect to keep.

- Dave


> http://blue-labs.org/clue/mtu-mss.php
> 
> -david
> 
> David Brownell wrote:
> 
> >I'm seeing a problem with TCP as accessed through KMail (SuSE 9.2, x86_64).
> >But oddly enough, only for sending mail, not reading it; and not through
> >other (reading) applications... it's a regression with respect to rc1 and
> >earlier kernels.  Basically, it can only send REALLY TINY emails...
> >
> >  
> >
> 

2.6.11-rc2 TCP ignores PMTU ICMP (Re: Linux 2.6.11-rc2)

[Janos Farkas]

On 2005-01-25 at 10:54:36, David Brownell wrote:
> On Tuesday 25 January 2005 10:35 am, David Ford wrote:
> > PMTU bug -- or better said, bad firewall admin who blocks all ICMP.
> 
> PMTU bug, sure -- but one that came late in RC2.  Remember:  same firewall
> in both cases, but only RC2 breaks.  The ICMP packet has landed in
> the RC2 system, which ignores it.  2.6.10 handled it correctly... I
> suspect one of the TCP cleanups borked this.
> 
> My current workaround is "ifconfig eth0 mtu 1492" but that's not
> something I'd expect to keep.

Indeed, I had to shuffle my machines around a bit to get a proof that
something is broken, but now I can confirm the above with a connection
to cvs.sourceforge.net:

192.168.59.10 is the 2.6.11-rc2 host, 192.168.59.1 is the (now pre-rc2)
gateway with a PPPoE 1492 mtu/pmtu host.  59.10 does not run any
firewall/packet filter.

(almost full-fledged perfect three-way handshake)

08:05:43.046546 IP (tos 0x0, ttl  64, id 21526, offset 0, flags [DF], length: 60) 192.168.59.10.29612 > 66.35.250.207.2401: SWE [tcp sum ok] 933042757:933042757(0) win 5840 <mss 1460,sackOK,timestamp 2707027 0,nop,wscale 2>
08:05:43.249819 IP (tos 0x0, ttl  45, id 0, offset 0, flags [DF], length: 60) 66.35.250.207.2401 > 192.168.59.10.29612: S [tcp sum ok] 2956308271:2956308271(0) ack 933042758 win 5792 <mss 1460,sackOK,timestamp 312382981 2707027,nop,wscale 0>
08:05:43.249881 IP (tos 0x0, ttl  64, id 21528, offset 0, flags [DF], length: 52) 192.168.59.10.29612 > 66.35.250.207.2401: . [tcp sum ok] ack 1 win 1460 <nop,nop,timestamp 2707230 312382981>

(CVS protocol starts)

08:05:43.363697 IP (tos 0x0, ttl  64, id 21530, offset 0, flags [DF], length: 116) 192.168.59.10.29612 > 66.35.250.207.2401: P [tcp sum ok] 1:65(64) ack 1 win 1460 <nop,nop,timestamp 2707344 312382981>
08:05:43.568504 IP (tos 0x0, ttl  45, id 14633, offset 0, flags [DF], length: 52) 66.35.250.207.2401 > 192.168.59.10.29612: . [tcp sum ok] ack 65 win 5792 <nop,nop,timestamp 312383013 2707344>
08:05:43.964094 IP (tos 0x0, ttl  45, id 14634, offset 0, flags [DF], length: 63) 66.35.250.207.2401 > 192.168.59.10.29612: P [tcp sum ok] 1:12(11) ack 65 win 5792 <nop,nop,timestamp 312383053 2707344>
08:05:43.964121 IP (tos 0x0, ttl  64, id 21532, offset 0, flags [DF], length: 52) 192.168.59.10.29612 > 66.35.250.207.2401: . [tcp sum ok] ack 12 win 1460 <nop,nop,timestamp 2707945 312383053>
08:05:43.964258 IP (tos 0x0, ttl  64, id 21534, offset 0, flags [DF], length: 440) 192.168.59.10.29612 > 66.35.250.207.2401: P [tcp sum ok] 65:453(388) ack 12 win 1460 <nop,nop,timestamp 2707945 312383053>
08:05:44.184893 IP (tos 0x0, ttl  45, id 14635, offset 0, flags [DF], length: 52) 66.35.250.207.2401 > 192.168.59.10.29612: . [tcp sum ok] ack 453 win 6432 <nop,nop,timestamp 312383075 2707945>
08:05:44.192050 IP (tos 0x0, ttl  45, id 14636, offset 0, flags [DF], length: 566) 66.35.250.207.2401 > 192.168.59.10.29612: P [tcp sum ok] 12:526(514) ack 453 win 6432 <nop,nop,timestamp 312383075 2707945>
08:05:44.192274 IP (tos 0x0, ttl  64, id 21536, offset 0, flags [DF], length: 65) 192.168.59.10.29612 > 66.35.250.207.2401: P [tcp sum ok] 453:466(13) ack 526 win 1728 <nop,nop,timestamp 2708174 312383075>
08:05:44.432709 IP (tos 0x0, ttl  45, id 14637, offset 0, flags [DF], length: 52) 66.35.250.207.2401 > 192.168.59.10.29612: . [tcp sum ok] ack 466 win 6432 <nop,nop,timestamp 312383100 2708174>
08:05:44.893674 IP (tos 0x0, ttl  64, id 21538, offset 0, flags [DF], length: 85) 192.168.59.10.29612 > 66.35.250.207.2401: P [tcp sum ok] 466:499(33) ack 526 win 1728 <nop,nop,timestamp 2708876 312383100>
08:05:45.094985 IP (tos 0x0, ttl  45, id 14638, offset 0, flags [DF], length: 52) 66.35.250.207.2401 > 192.168.59.10.29612: . [tcp sum ok] ack 499 win 6432 <nop,nop,timestamp 312383166 2708876>

(first ethernet frame-sized segment is about to be sent)

08:05:45.264633 IP (tos 0x0, ttl  64, id 21540, offset 0, flags [DF], length: 1500) 192.168.59.10.29612 > 66.35.250.207.2401: . [tcp sum ok] 499:1947(1448) ack 526 win 1728 <nop,nop,timestamp 2709247 312383166>
08:05:45.264694 IP (tos 0x0, ttl  64, id 21542, offset 0, flags [DF], length: 1500) 192.168.59.10.29612 > 66.35.250.207.2401: . [tcp sum ok] 1947:3395(1448) ack 526 win 1728 <nop,nop,timestamp 2709247 312383166>

(the PPPoE gateway notifies the host about the PMTU issue, one for each
segment)

08:05:45.265330 IP (tos 0xc0, ttl 128, id 17970, offset 0, flags [none], length: 576) 192.168.59.1 > 192.168.59.10: icmp 556: 66.35.250.207 unreachable - need to frag (mtu 1492)
08:05:45.265453 IP (tos 0xc0, ttl 128, id 17971, offset 0, flags [none], length: 576) 192.168.59.1 > 192.168.59.10: icmp 556: 66.35.250.207 unreachable - need to frag (mtu 1492)

(the source host sends the full frame again...?)

08:05:45.770331 IP (tos 0x0, ttl  64, id 21544, offset 0, flags [DF], length: 1500) 192.168.59.10.29612 > 66.35.250.207.2401: . [tcp sum ok] 499:1947(1448) ack 526 win 1728 <nop,nop,timestamp 2709754 312383166>
08:05:45.770900 IP (tos 0xc0, ttl 128, id 17972, offset 0, flags [none], length: 576) 192.168.59.1 > 192.168.59.10: icmp 556: 66.35.250.207 unreachable - need to frag (mtu 1492)

(and looping goes on..)

08:05:46.783148 IP (tos 0x0, ttl  64, id 21546, offset 0, flags [DF], length: 1500) 192.168.59.10.29612 > 66.35.250.207.2401: . [tcp sum ok] 499:1947(1448) ack 526 win 1728 <nop,nop,timestamp 2710768 312383166>
08:05:46.783752 IP (tos 0xc0, ttl 128, id 17973, offset 0, flags [none], length: 576) 192.168.59.1 > 192.168.59.10: icmp 556: 66.35.250.207 unreachable - need to frag (mtu 1492)
08:05:48.808861 IP (tos 0x0, ttl  64, id 21548, offset 0, flags [DF], length: 1500) 192.168.59.10.29612 > 66.35.250.207.2401: . [tcp sum ok] 499:1947(1448) ack 526 win 1728 <nop,nop,timestamp 2712796 312383166>
08:05:48.809882 IP (tos 0xc0, ttl 128, id 17974, offset 0, flags [none], length: 576) 192.168.59.1 > 192.168.59.10: icmp 556: 66.35.250.207 unreachable - need to frag (mtu 1492)

...

I don't have now a pre-rc2 host to try this on, but I regularly use
sourceforge anoncvs to at least confess honestly that it did work :)

-- 
Janos | romfs is at http://romfs.sourceforge.net/ | Don't talk about silence.

Re: 2.6.11-rc2 TCP ignores PMTU ICMP (Re: Linux 2.6.11-rc2)

[David Brownell]

On Thursday 27 January 2005 1:02 am, Janos Farkas wrote:
> On 2005-01-25 at 10:54:36, David Brownell wrote:
> > On Tuesday 25 January 2005 10:35 am, David Ford wrote:
> > > PMTU bug -- or better said, bad firewall admin who blocks all ICMP.
> > 
> > PMTU bug, sure -- but one that came late in RC2.  Remember:  same firewall
> > in both cases, but only RC2 breaks.  The ICMP packet has landed in
> > the RC2 system, which ignores it.  2.6.10 handled it correctly... I
> > suspect one of the TCP cleanups borked this.

And in fact, I'm pretty sure RC1 handled it fine too.  This failure
appeared in a BK pull I did a couple days before RC2, and installing
it clobbered a "more virgin" RC1 install; so it's hard to verify.

It took a while to track the failure mode down to being PMTU related,
but the only TCP-related changes I noticed in what I pulled sure
seemed like cleanups (removing some "duplicated" state).


> > My current workaround is "ifconfig eth0 mtu 1492" but that's not
> > something I'd expect to keep.
> 
> Indeed, I had to shuffle my machines around a bit to get a proof that
> something is broken, but now I can confirm the above with a connection
> to cvs.sourceforge.net:

Thanks for confirming it wasn't just me ... I confess I'm a bit
surprised more folk haven't reported this yet! 

Your symptoms are exactly like those I saw, just with a different
mission-critical application:  CVS, not SMTP.  Did you happen to
notice whether CVS pulls worked, when pushes (like this) failed?

- Dave


> ...
> 
> (the PPPoE gateway notifies the host about the PMTU issue, one for each
> segment)
> 
> 08:05:45.265330 IP (tos 0xc0, ttl 128, id 17970, offset 0, flags [none], length: 576) 192.168.59.1 > 192.168.59.10: icmp 556: 66.35.250.207 unreachable - need to frag (mtu 1492)
> 08:05:45.265453 IP (tos 0xc0, ttl 128, id 17971, offset 0, flags [none], length: 576) 192.168.59.1 > 192.168.59.10: icmp 556: 66.35.250.207 unreachable - need to frag (mtu 1492)
> 
> (the source host sends the full frame again...?)
> 
> 08:05:45.770331 IP (tos 0x0, ttl  64, id 21544, offset 0, flags [DF], length: 1500) 192.168.59.10.29612 > 66.35.250.207.2401: . [tcp sum ok] 499:1947(1448) ack 526 win 1728 <nop,nop,timestamp 2709754 312383166>
> 08:05:45.770900 IP (tos 0xc0, ttl 128, id 17972, offset 0, flags [none], length: 576) 192.168.59.1 > 192.168.59.10: icmp 556: 66.35.250.207 unreachable - need to frag (mtu 1492)
> 
> (and looping goes on..)
> 
> 08:05:46.783148 IP (tos 0x0, ttl  64, id 21546, offset 0, flags [DF], length: 1500) 192.168.59.10.29612 > 66.35.250.207.2401: . [tcp sum ok] 499:1947(1448) ack 526 win 1728 <nop,nop,timestamp 2710768 312383166>
> 08:05:46.783752 IP (tos 0xc0, ttl 128, id 17973, offset 0, flags [none], length: 576) 192.168.59.1 > 192.168.59.10: icmp 556: 66.35.250.207 unreachable - need to frag (mtu 1492)
> 08:05:48.808861 IP (tos 0x0, ttl  64, id 21548, offset 0, flags [DF], length: 1500) 192.168.59.10.29612 > 66.35.250.207.2401: . [tcp sum ok] 499:1947(1448) ack 526 win 1728 <nop,nop,timestamp 2712796 312383166>
> 08:05:48.809882 IP (tos 0xc0, ttl 128, id 17974, offset 0, flags [none], length: 576) 192.168.59.1 > 192.168.59.10: icmp 556: 66.35.250.207 unreachable - need to frag (mtu 1492)
> 
> ...
> 

Re: 2.6.11-rc2 TCP ignores PMTU ICMP (Re: Linux 2.6.11-rc2)

[Janos Farkas]

On 2005-01-27 at 11:28:48, David Brownell wrote:
> > Indeed, I had to shuffle my machines around a bit to get a proof that
> > something is broken, but now I can confirm the above with a connection
> > to cvs.sourceforge.net:
> 
> Thanks for confirming it wasn't just me ... I confess I'm a bit
> surprised more folk haven't reported this yet! 
> 
> Your symptoms are exactly like those I saw, just with a different
> mission-critical application:  CVS, not SMTP.  Did you happen to
> notice whether CVS pulls worked, when pushes (like this) failed?

I'm familiar with symptoms arising from various stochastic/assymetric
networking problems with larger frames, I had cable :)  This was a pull
(update), BTW, CVS can send more data in this situation too.  But it
turned painfully obvious when trying to send mail to LK via SMTP.  Oh,
but while plain rc2 may have this problem, this particular endpoint had
rc2-bk3 running.  I also chose to set the ethernet MTU to 1492 as an
easy workaround...

-- 
Janos | romfs is at http://romfs.sourceforge.net/ | Don't talk about silence.

Re: 2.6.11-rc2 TCP ignores PMTU ICMP (Re: Linux 2.6.11-rc2)

[David Brownell]

I just got an interesting "I see these problems too" report.  It
may provide a useful clue.  According to "Art Haas" <ahaas@airmail.net>:

> I'm running the current BK kernel now, and I'm not seeing the problems
> right now because, I found, I do not have some of the IP masquerading
> modules installed on my machine. When these modules get installed then
> the cvs/rsync problems appear. 
 
I do have CONFIG_IP_NF_TARGET_MASQUERADE=y on the system where I'm
seeing this, though it's not doing anything just now.  Haven't yet
made time to see if disabling it improves things ... but if that's
a factor, it could explain why more people aren't suffering with
this problem.

- Dave

Re: 2.6.11-rc2 TCP ignores PMTU ICMP (Re: Linux 2.6.11-rc2)

[David S. Miller]

I've forwarded this to netfilter-devel for inspection.
Thanks for collecting all the data points so well.

Re: 2.6.11-rc2 TCP ignores PMTU ICMP (Re: Linux 2.6.11-rc2)

[Patrick McHardy]

[-- Attachment #1: Type: text/plain, Size: 226 bytes --]

David S. Miller wrote:

>I've forwarded this to netfilter-devel for inspection.
>Thanks for collecting all the data points so well.
>
Here is the fix for everyone. Please report back if it doesn't
solve the problem. Thanks.



[-- Attachment #2: x --]
[-- Type: text/plain, Size: 487 bytes --]

===== net/ipv4/netfilter/ip_nat_proto_tcp.c 1.10 vs edited =====
--- 1.10/net/ipv4/netfilter/ip_nat_proto_tcp.c	2005-01-17 23:00:55 +01:00
+++ edited/net/ipv4/netfilter/ip_nat_proto_tcp.c	2005-01-28 02:13:06 +01:00
@@ -105,7 +105,7 @@
 		return 0;
 
 	iph = (struct iphdr *)((*pskb)->data + iphdroff);
-	hdr = (struct tcphdr *)((*pskb)->data + iph->ihl*4);
+	hdr = (struct tcphdr *)((*pskb)->data + hdroff);
 
 	if (maniptype == IP_NAT_MANIP_SRC) {
 		/* Get rid of src ip and src pt */

Re: 2.6.11-rc2 TCP ignores PMTU ICMP (Re: Linux 2.6.11-rc2)

[Rusty Russell]

On Fri, 2005-01-28 at 02:33 +0100, Patrick McHardy wrote:
> David S. Miller wrote:
> 
> >I've forwarded this to netfilter-devel for inspection.
> >Thanks for collecting all the data points so well.
> >
> Here is the fix for everyone. Please report back if it doesn't
> solve the problem. Thanks.

Verified by nfsim (the ICMP tests used UDP, I've just added TCP and
ICMP, and will do SCTP).

Thanks,
Rusty.
-- 
A bad analogy is like a leaky screwdriver -- Richard Braakman

Re: 2.6.11-rc2 TCP ignores PMTU ICMP (Re: Linux 2.6.11-rc2)

[Art Haas]

On Fri, Jan 28, 2005 at 02:33:10AM +0100, Patrick McHardy wrote:
> Here is the fix for everyone. Please report back if it doesn't
> solve the problem. Thanks.
> 
> [ ... snip ... ]

Success!!!

Art Haas
-- 
Man once surrendering his reason, has no remaining guard against absurdities
the most monstrous, and like a ship without rudder, is the sport of every wind.

-Thomas Jefferson to James Smith, 1822

Re: 2.6.11-rc2 TCP ignores PMTU ICMP (Re: Linux 2.6.11-rc2)

[Bill Davidsen]

Patrick McHardy wrote:
> David S. Miller wrote:
> 
>> I've forwarded this to netfilter-devel for inspection.
>> Thanks for collecting all the data points so well.
>>
> Here is the fix for everyone. Please report back if it doesn't
> solve the problem. Thanks.

Worked here.
> 
> 
> 
> ------------------------------------------------------------------------
> 
> ===== net/ipv4/netfilter/ip_nat_proto_tcp.c 1.10 vs edited =====
> --- 1.10/net/ipv4/netfilter/ip_nat_proto_tcp.c	2005-01-17 23:00:55 +01:00
> +++ edited/net/ipv4/netfilter/ip_nat_proto_tcp.c	2005-01-28 02:13:06 +01:00
> @@ -105,7 +105,7 @@
>  		return 0;
>  
>  	iph = (struct iphdr *)((*pskb)->data + iphdroff);
> -	hdr = (struct tcphdr *)((*pskb)->data + iph->ihl*4);
> +	hdr = (struct tcphdr *)((*pskb)->data + hdroff);
>  
>  	if (maniptype == IP_NAT_MANIP_SRC) {
>  		/* Get rid of src ip and src pt */


-- 
    -bill davidsen (davidsen@tmr.com)
"The secret to procrastination is to put things off until the
  last possible moment - but no longer"  -me