two other cases Re: [Bug 11721] after upgrade to 2.6.27 i cannot navigate

two other cases Re: [Bug 11721] after upgrade to 2.6.27 i cannot navigate

[Aldo Maggi]

just as matter of information, two other cases similar to mine were
reported in ubuntu bug pages:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/272896
https://bugs.launchpad.net/ubuntu/+bug/285430
i originated the first one and gave Jarek's first solution:
tcp_sack=0
which worked for the two other users.

maybe they could be contacted in order to perform further tests in a
different environment.

aldo  

Re: two other cases Re: [Bug 11721] after upgrade to 2.6.27 i cannot navigate

[Jarek Poplawski]

On Tue, Oct 21, 2008 at 09:00:02AM +0200, Aldo Maggi wrote:
> just as matter of information, two other cases similar to mine were
> reported in ubuntu bug pages:
> https://bugs.launchpad.net/ubuntu/+source/linux/+bug/272896
> https://bugs.launchpad.net/ubuntu/+bug/285430
> i originated the first one and gave Jarek's first solution:

Hmm... let's say you gave my 50% of our first solution...

Jarek P.

Re: two other cases Re: [Bug 11721] after upgrade to 2.6.27 i cannot navigate

[Aldo Maggi]

Il giorno Tue, 21 Oct 2008 07:20:16 +0000
Jarek Poplawski <jarkao2@gmail.com> ha scritto:

> On Tue, Oct 21, 2008 at 09:00:02AM +0200, Aldo Maggi wrote:
> > just as matter of information, two other cases similar to mine were
> > reported in ubuntu bug pages:
> > https://bugs.launchpad.net/ubuntu/+source/linux/+bug/272896
> > https://bugs.launchpad.net/ubuntu/+bug/285430
> > i originated the first one and gave Jarek's first solution:
> 
> Hmm... let's say you gave my 50% of our first solution...
> 
> Jarek P.

ehi jarek! do not be so modest! :-D
i've acted just like a bot! :-D
i didn't have even a clue of the problem! if i gave you ssh access to my
lan you would have had the possibility to settle the matter with no my
intervention at all!

good day to you all! :-)
aldo

Re: two other cases Re: [Bug 11721] after upgrade to 2.6.27 i cannot navigate

[Jarek Poplawski]

On Tue, Oct 21, 2008 at 09:38:27AM +0200, Aldo Maggi wrote:
> Il giorno Tue, 21 Oct 2008 07:20:16 +0000
> Jarek Poplawski <jarkao2@gmail.com> ha scritto:
> 
> > On Tue, Oct 21, 2008 at 09:00:02AM +0200, Aldo Maggi wrote:
> > > just as matter of information, two other cases similar to mine were
> > > reported in ubuntu bug pages:
> > > https://bugs.launchpad.net/ubuntu/+source/linux/+bug/272896
> > > https://bugs.launchpad.net/ubuntu/+bug/285430
> > > i originated the first one and gave Jarek's first solution:
> > 
> > Hmm... let's say you gave my 50% of our first solution...
> > 
> > Jarek P.
> 
> ehi jarek! do not be so modest! :-D
> i've acted just like a bot! :-D
> i didn't have even a clue of the problem!

You too?...

> if i gave you ssh access to my
> lan you would have had the possibility to settle the matter with no my
> intervention at all!

Why would you want to destroy your lan?

Cheers,
Jarek P.

Re: two other cases Re: [Bug 11721] after upgrade to 2.6.27 i cannot navigate

[Ilpo Järvinen]

Ah, I forgot to add bugzilla back last time, so added it there now.

On Tue, 21 Oct 2008, Aldo Maggi wrote:

> just as matter of information, two other cases similar to mine were
> reported in ubuntu bug pages:
> https://bugs.launchpad.net/ubuntu/+source/linux/+bug/272896
> https://bugs.launchpad.net/ubuntu/+bug/285430
> i originated the first one and gave Jarek's first solution:
> tcp_sack=0
> which worked for the two other users.
> 
> maybe they could be contacted in order to perform further tests in a
> different environment.

It's hardly surprising that I couldn't reproduce this, the non-compliance 
here is most probably in isp's device or in some end-user sold embedded 
device.

Can you try this another debug patch below (on 2.6.27.2 is fine). It moves 
the mss to the last position but should keep timestamps in place by making 
wscale as first option. It is well possible that you won't get it working 
at all except with all ts,sack and wscale set to 0 (the most likely 
result). Please try with all wscale,sack,ts combinations (no need to 
provide dumps, just working/not working per case)... This should 
tell us for quite high certaintity what is the actual option which is 
causing this (would it not be the mss-at-beginning which is the most 
likely cause), however, your finding may well be specific to your network 
while the other people might a bit different results.

In order to provide maximal compatibility, I think we just restore the 
previous ordering of the fields (basically the first patch you tested).
It has no additional cost, so it won't hurt any, but it's quite ridiculous 
still that some devices care so little about basic tcp spec which has 
devastating effect on interoperatibility here, they should really fix
the devices instead but knowing how little most of the isp & etc. care
(or even understand) I'm not expecting too much to happen on that
front, and those who care probably run some semi-sane stuff already
anyway... :-). ...Sadly, it's much easier and cheaper to blame the 
end-user's equipment or Linux (if/once it becomes known that it's in use) 
and do nothing in case one is fool enough to complain to them.


-- 
 i.


--
diff --git a/net/ipv4/tcp_output.c b/net/ipv4/tcp_output.c
index de54f02..63b0a3f 100644
--- a/net/ipv4/tcp_output.c
+++ b/net/ipv4/tcp_output.c
@@ -376,6 +376,13 @@ static void tcp_options_write(__be32 *ptr, struct tcp_sock *tp,
 		*md5_hash = NULL;
 	}
 
+	if (unlikely(opts->ws)) {
+		*ptr++ = htonl((TCPOPT_NOP << 24) |
+			       (TCPOPT_WINDOW << 16) |
+			       (TCPOLEN_WINDOW << 8) |
+			       opts->ws);
+	}
+
 	if (likely(OPTION_TS & opts->options)) {
 		if (unlikely(OPTION_SACK_ADVERTISE & opts->options)) {
 			*ptr++ = htonl((TCPOPT_SACK_PERM << 24) |
@@ -392,12 +399,6 @@ static void tcp_options_write(__be32 *ptr, struct tcp_sock *tp,
 		*ptr++ = htonl(opts->tsecr);
 	}
 
-	if (unlikely(opts->mss)) {
-		*ptr++ = htonl((TCPOPT_MSS << 24) |
-			       (TCPOLEN_MSS << 16) |
-			       opts->mss);
-	}
-
 	if (unlikely(OPTION_SACK_ADVERTISE & opts->options &&
 		     !(OPTION_TS & opts->options))) {
 		*ptr++ = htonl((TCPOPT_NOP << 24) |
@@ -406,11 +407,10 @@ static void tcp_options_write(__be32 *ptr, struct tcp_sock *tp,
 			       TCPOLEN_SACK_PERM);
 	}
 
-	if (unlikely(opts->ws)) {
-		*ptr++ = htonl((TCPOPT_NOP << 24) |
-			       (TCPOPT_WINDOW << 16) |
-			       (TCPOLEN_WINDOW << 8) |
-			       opts->ws);
+	if (unlikely(opts->mss)) {
+		*ptr++ = htonl((TCPOPT_MSS << 24) |
+			       (TCPOLEN_MSS << 16) |
+			       opts->mss);
 	}
 
 	if (unlikely(opts->num_sack_blocks)) {

Re: two other cases Re: [Bug 11721] after upgrade to 2.6.27 i cannot navigate

[Jarek Poplawski]

On Tue, Oct 21, 2008 at 12:36:33PM +0300, Ilpo Järvinen wrote:
...
> Can you try this another debug patch below (on 2.6.27.2 is fine). It moves 
> the mss to the last position but should keep timestamps in place by making 
> wscale as first option. It is well possible that you won't get it working 
> at all except with all ts,sack and wscale set to 0 (the most likely 
> result). Please try with all wscale,sack,ts combinations (no need to 
> provide dumps, just working/not working per case)... This should 
> tell us for quite high certaintity what is the actual option which is 
> causing this (would it not be the mss-at-beginning which is the most 
> likely cause), [...]

I'd like to remind it seems to work with only sack off, so mss after
ts. If so, my suspicion would be around rfc's/options' dating?

Jarek P.

Re: two other cases Re: [Bug 11721] after upgrade to 2.6.27 i cannot navigate

[Ilpo Järvinen]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 1322 bytes --]

On Tue, 21 Oct 2008, Jarek Poplawski wrote:

> On Tue, Oct 21, 2008 at 12:36:33PM +0300, Ilpo Järvinen wrote:
> ...
> > Can you try this another debug patch below (on 2.6.27.2 is fine). It moves 
> > the mss to the last position but should keep timestamps in place by making 
> > wscale as first option. It is well possible that you won't get it working 
> > at all except with all ts,sack and wscale set to 0 (the most likely 
> > result). Please try with all wscale,sack,ts combinations (no need to 
> > provide dumps, just working/not working per case)... This should 
> > tell us for quite high certaintity what is the actual option which is 
> > causing this (would it not be the mss-at-beginning which is the most 
> > likely cause), [...]
> 
> I'd like to remind it seems to work with only sack off, so mss after
> ts. If so, my suspicion would be around rfc's/options' dating?

Thanks, I missed that case... there are so many cases already... :-)
...I'm not sure we'll find some very obvious reason for this, rfc dating 
probably has very little to do here.

I still think that putting an alternative option in front is useful test 
(though my initial mss-at-beginning thought wasn't bullet-proof) but 
maybe we could skip all those alternative settings of wscale,sack,ts and 
have them just all enabled.


-- 
 i.

Re: two other cases Re: [Bug 11721] after upgrade to 2.6.27 i cannot navigate

[Jarek Poplawski]

On Tue, Oct 21, 2008 at 01:51:10PM +0300, Ilpo Järvinen wrote:
> On Tue, 21 Oct 2008, Jarek Poplawski wrote:
> 
> > On Tue, Oct 21, 2008 at 12:36:33PM +0300, Ilpo Järvinen wrote:
> > ...
> > > Can you try this another debug patch below (on 2.6.27.2 is fine). It moves 
> > > the mss to the last position but should keep timestamps in place by making 
> > > wscale as first option. It is well possible that you won't get it working 
> > > at all except with all ts,sack and wscale set to 0 (the most likely 
> > > result). Please try with all wscale,sack,ts combinations (no need to 
> > > provide dumps, just working/not working per case)... This should 
> > > tell us for quite high certaintity what is the actual option which is 
> > > causing this (would it not be the mss-at-beginning which is the most 
> > > likely cause), [...]
> > 
> > I'd like to remind it seems to work with only sack off, so mss after
> > ts. If so, my suspicion would be around rfc's/options' dating?
> 
> Thanks, I missed that case... there are so many cases already... :-)
> ...I'm not sure we'll find some very obvious reason for this, rfc dating 
> probably has very little to do here.
> 
> I still think that putting an alternative option in front is useful test 
> (though my initial mss-at-beginning thought wasn't bullet-proof) but 
> maybe we could skip all those alternative settings of wscale,sack,ts and 
> have them just all enabled.

I agree with you, but when this test with all options enabled fails
(?!), IMHO disabling sack only could be enough to get it working
again.

Jarek P.

Re: two other cases Re: [Bug 11721] after upgrade to 2.6.27 i cannot navigate

[Ilpo Järvinen]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 1768 bytes --]

On Tue, 21 Oct 2008, Jarek Poplawski wrote:

> On Tue, Oct 21, 2008 at 01:51:10PM +0300, Ilpo Järvinen wrote:
> > On Tue, 21 Oct 2008, Jarek Poplawski wrote:
> > 
> > > On Tue, Oct 21, 2008 at 12:36:33PM +0300, Ilpo Järvinen wrote:
> > > ...
> > > > Can you try this another debug patch below (on 2.6.27.2 is fine). It moves 
> > > > the mss to the last position but should keep timestamps in place by making 
> > > > wscale as first option. It is well possible that you won't get it working 
> > > > at all except with all ts,sack and wscale set to 0 (the most likely 
> > > > result). Please try with all wscale,sack,ts combinations (no need to 
> > > > provide dumps, just working/not working per case)... This should 
> > > > tell us for quite high certaintity what is the actual option which is 
> > > > causing this (would it not be the mss-at-beginning which is the most 
> > > > likely cause), [...]
> > > 
> > > I'd like to remind it seems to work with only sack off, so mss after
> > > ts. If so, my suspicion would be around rfc's/options' dating?
> > 
> > Thanks, I missed that case... there are so many cases already... :-)
> > ...I'm not sure we'll find some very obvious reason for this, rfc dating 
> > probably has very little to do here.
> > 
> > I still think that putting an alternative option in front is useful test 
> > (though my initial mss-at-beginning thought wasn't bullet-proof) but 
> > maybe we could skip all those alternative settings of wscale,sack,ts and 
> > have them just all enabled.
> 
> I agree with you, but when this test with all options enabled fails
> (?!), IMHO disabling sack only could be enough to get it working
> again.

...Yes it should, but we could be all wrong and it just works with them 
all too :-).


-- 
 i.

Re: two other cases Re: [Bug 11721] after upgrade to 2.6.27 i cannot navigate

[Jarek Poplawski]

On Tue, Oct 21, 2008 at 03:18:57PM +0300, Ilpo Järvinen wrote:
...
> ...Yes it should, but we could be all wrong and it just works with them 
> all too :-).

All execept you! I guess you want it tested because you assume it's
possible? (Anyway it would be really funny :-)

Jarek P.

Re: two other cases Re: [Bug 11721] after upgrade to 2.6.27 i cannot navigate

[Ilpo Järvinen]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 948 bytes --]

On Tue, 21 Oct 2008, Jarek Poplawski wrote:

> On Tue, Oct 21, 2008 at 03:18:57PM +0300, Ilpo Järvinen wrote:
> ...
> > ...Yes it should, but we could be all wrong and it just works with them 
> > all too :-).
> 
> All execept you! I guess you want it tested because you assume it's
> possible? (Anyway it would be really funny :-)

We're dealing with a case with in itself is really funny, yes, I assume 
it's very much possible though not too likely :-). I'd have very much 
wanted to test it myself too but sadly my network does not exhibit this 
corner-case behavior (ie., bug). Whether I'm right or wrong is not that 
significant for me :-), just understanding what the black-box does (or
is fine with) instead of just knowing how to workaround that. Or did I 
miss another case where the significance of mss specifically being there 
was decided? :-)

But whatever, if Aldo feels it's waste of his time he can just ignore 
me... :-)


-- 
 i.

Re: two other cases Re: [Bug 11721] after upgrade to 2.6.27 i cannot navigate

[Aldo Maggi]

Il giorno Tue, 21 Oct 2008 17:16:43 +0300 (EEST)
"Ilpo Järvinen" <ilpo.jarvinen@helsinki.fi> ha scritto:

[...]

> 
> But whatever, if Aldo feels it's waste of his time he can just ignore 
> me... :-)
> 
> 

ARE YOU JOKIIIIING? :-)
i'm only too honoured to cooperate from my very low level with you!

you do not imagine how much i value your commitment for linux!
my only dream is to see bloody ms in the dust ... and linux as a winner!

many thanks for your and your collegues work!
aldo
 

Re: two other cases Re: [Bug 11721] after upgrade to 2.6.27 i cannot navigate

[Jarek Poplawski]

On Tue, Oct 21, 2008 at 05:16:43PM +0300, Ilpo Järvinen wrote:
...
> We're dealing with a case with in itself is really funny, yes, I assume 
> it's very much possible though not too likely :-). I'd have very much 
> wanted to test it myself too but sadly my network does not exhibit this 
> corner-case behavior (ie., bug). Whether I'm right or wrong is not that 
> significant for me :-), just understanding what the black-box does (or
> is fine with) instead of just knowing how to workaround that. Or did I 
> miss another case where the significance of mss specifically being there 
> was decided? :-)
> 
> But whatever, if Aldo feels it's waste of his time he can just ignore 
> me... :-)

Hmm... If by any chance my previous messages caused you think like
this, I'm really sorry. I completely agree with the way you deal with
this, and I don't pretend I know the result. I still agree there seems
to be not much logic in this, so all results are probable, and worth
checking - especially when we have such helpful assistant as Aldo.

Thanks,
Jarek P.

Re: two other cases Re: [Bug 11721] after upgrade to 2.6.27 i cannot navigate

[Aldo Maggi]

[-- Attachment #1: Type: text/plain, Size: 1180 bytes --]

Il giorno Tue, 21 Oct 2008 12:36:33 +0300 (EEST)
"Ilpo Järvinen" <ilpo.jarvinen@helsinki.fi> ha scritto:

[...]

> 
> Can you try this another debug patch below (on 2.6.27.2 is fine). It
> moves the mss to the last position but should keep timestamps in
> place by making wscale as first option. It is well possible that you
> won't get it working at all except with all ts,sack and wscale set to
> 0 (the most likely result). Please try with all wscale,sack,ts
> combinations (no need to provide dumps, just working/not working per
> case)... This should tell us for quite high certaintity what is the
> actual option which is causing this (would it not be the
> mss-at-beginning which is the most likely cause), however, your
> finding may well be specific to your network while the other people
> might a bit different results.

i've compiled 2.6.27.2 source after having patched it with your today's
patch.

it works! i.e. i can navigate (w3m kernel.org) and update (apt-get
update)

i send you anyway the usual tcpdump as in previous msgs, should it be
of any help.

do you want me to provide some more commands output?

waiting...

aldo



[-- Attachment #2: tcpdump_2-6-27-2-ilpo-2110-patch --]
[-- Type: application/octet-stream, Size: 1454 bytes --]

20:14:15.771710 IP 192.168.254.1.41264 > 204.152.191.5.80: S 2671126883:2671126883(0) win 5840 <nop,wscale 6,sackOK,timestamp 34138 0,mss 1460>
	0x0000:  00a0 c543 5feb 0010 5ac5 681c 0800 4500  ...C_...Z.h...E.
	0x0010:  003c ac39 4000 3f06 453a c0a8 fe01 cc98  .<.9@.?.E:......
	0x0020:  bf05 a130 0050 9f36 2963 0000 0000 a002  ...0.P.6)c......
	0x0030:  16d0 f773 0000 0103 0306 0402 080a 0000  ...s............
	0x0040:  855a 0000 0000 0204 05b4                 .Z........
20:14:15.992523 IP 192.168.254.1.41264 > 204.152.191.5.80: . ack 4239991373 win 92 <nop,nop,timestamp 34193 1103211035>
	0x0000:  00a0 c543 5feb 0010 5ac5 681c 0800 4500  ...C_...Z.h...E.
	0x0010:  0034 ac3a 4000 3f06 4541 c0a8 fe01 cc98  .4.:@.?.EA......
	0x0020:  bf05 a130 0050 9f36 2964 fcb9 224d 8010  ...0.P.6)d.."M..
	0x0030:  005c 3188 0000 0101 080a 0000 8591 41c1  .\1...........A.
	0x0040:  aa1b                                     ..
20:14:15.992787 IP 192.168.254.1.41264 > 204.152.191.5.80: P 0:240(240) ack 1 win 92 <nop,nop,timestamp 34193 1103211035>
	0x0000:  00a0 c543 5feb 0010 5ac5 681c 0800 4500  ...C_...Z.h...E.
	0x0010:  0124 ac3b 4000 3f06 4450 c0a8 fe01 cc98  .$.;@.?.DP......
	0x0020:  bf05 a130 0050 9f36 2964 fcb9 224d 8018  ...0.P.6)d.."M..
	0x0030:  005c 376a 0000 0101 080a 0000 8591 41c1  .\7j..........A.
	0x0040:  aa1b 4745 5420 2f20 4854 5450 2f31 2e30  ..GET./.HTTP/1.0
	0x0050:  0d0a 5573 6572 2d41 6765 6e74 3a20 7733  ..User-Agent:.w3

Re: two other cases Re: [Bug 11721] after upgrade to 2.6.27 i cannot navigate

[Jarek Poplawski]

On Tue, Oct 21, 2008 at 08:18:57PM +0200, Aldo Maggi wrote:
...
> i've compiled 2.6.27.2 source after having patched it with your today's
> patch.
> 
> it works! i.e. i can navigate (w3m kernel.org) and update (apt-get
> update)

Ilpo, I should say you're incredible! ...But, since I've promised to
myself not to disturb you anymore (again), I can't do this, sorry :-(

Jarek P.

Re: two other cases Re: [Bug 11721] after upgrade to 2.6.27 i cannot navigate

[David Miller]

From: Jarek Poplawski <jarkao2@gmail.com>
Date: Tue, 21 Oct 2008 20:45:39 +0200

> On Tue, Oct 21, 2008 at 08:18:57PM +0200, Aldo Maggi wrote:
> ...
> > i've compiled 2.6.27.2 source after having patched it with your today's
> > patch.
> > 
> > it works! i.e. i can navigate (w3m kernel.org) and update (apt-get
> > update)
> 
> Ilpo, I should say you're incredible! ...But, since I've promised to
> myself not to disturb you anymore (again), I can't do this, sorry :-(

Indeed, excellent work Ilpo.

Ilpo, now that we know this fixes things for sure, could you submit
this formally with a proper signoff?

I'll queue it up for -stable too.

Thanks!

Re: two other cases Re: [Bug 11721] after upgrade to 2.6.27 i cannot navigate

[Ilpo Järvinen]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 4369 bytes --]

On Tue, 21 Oct 2008, David Miller wrote:

> From: Jarek Poplawski <jarkao2@gmail.com>
> Date: Tue, 21 Oct 2008 20:45:39 +0200
> 
> > On Tue, Oct 21, 2008 at 08:18:57PM +0200, Aldo Maggi wrote:
> > ...
> > > i've compiled 2.6.27.2 source after having patched it with your today's
> > > patch.
> > > 
> > > it works! i.e. i can navigate (w3m kernel.org) and update (apt-get
> > > update)
> > 
> > Ilpo, I should say you're incredible! ...But, since I've promised to
> > myself not to disturb you anymore (again), I can't do this, sorry :-(
> 
> Indeed, excellent work Ilpo.
> 
> Ilpo, now that we know this fixes things for sure, could you submit
> this formally with a proper signoff?
> 
> I'll queue it up for -stable too.
> 
> Thanks!

Sure, here below is one with a warning (it's the first patch + comment).

Olon, can you please check this as well if it affect to your case too 
(though the symptoms were not that clear in your case).

It would be nice for Aldo to check what the result will be with my second 
patch (only) using sack=1,ts=0,wscale=0. I guess it does but it's a bit 
unclear if nop's in front help or not (having the patch below should 
anyway help also in that case as the mss option gets moved before it 
anyway).

--
[PATCH] tcp: Restore ordering of TCP options for the sake of inter-operability

This is not our bug! Sadly some devices cannot cope with the change
of TCP option ordering which was a result of the recent rewrite of
the option code (not that there was some particular reason steming
from the rewrite for the reordering) though any ordering of TCP
options is perfectly legal. Thus we restore the original ordering
to allow interoperability with/through such broken devices and add
some warning about this trap. Since the reordering just happened
without any particular reason, this change shouldn't cost us
anything.

There are already couple of known failure reports (within close
proximity of the last release), so the problem might be more
wide-spread than a single device. And other reports which may
be due to the same problem though the symptoms were less obvious.
Analysis of one of the case revealed (with very high probability)
that sack capability cannot be negotiated as the first option
(SYN never got a response).

Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Reported-by: Aldo Maggi <sentiniate@tiscali.it>
Tested-by: Aldo Maggi <sentiniate@tiscali.it>
---
 net/ipv4/tcp_output.c |   23 +++++++++++++++++------
 1 files changed, 17 insertions(+), 6 deletions(-)

diff --git a/net/ipv4/tcp_output.c b/net/ipv4/tcp_output.c
index de54f02..e4c5ac9 100644
--- a/net/ipv4/tcp_output.c
+++ b/net/ipv4/tcp_output.c
@@ -362,6 +362,17 @@ struct tcp_out_options {
 	__u32 tsval, tsecr;	/* need to include OPTION_TS */
 };
 
+/* Beware: Something in the Internet is very sensitive to the ordering of
+ * TCP options, we learned this through the hard way, so be careful here.
+ * Luckily we can at least blame others for their non-compliance but from
+ * inter-operatibility perspective it seems that we're somewhat stuck with
+ * the ordering which we have been using if we want to keep working with
+ * those broken things (not that it currently hurts anybody as there isn't
+ * particular reason why the ordering would need to be changed).
+ *
+ * At least SACK_PERM as the first option is known to lead to a disaster
+ * (but it may well be that other scenarios fail similarly).
+ */
 static void tcp_options_write(__be32 *ptr, struct tcp_sock *tp,
 			      const struct tcp_out_options *opts,
 			      __u8 **md5_hash) {
@@ -376,6 +387,12 @@ static void tcp_options_write(__be32 *ptr, struct tcp_sock *tp,
 		*md5_hash = NULL;
 	}
 
+	if (unlikely(opts->mss)) {
+		*ptr++ = htonl((TCPOPT_MSS << 24) |
+			       (TCPOLEN_MSS << 16) |
+			       opts->mss);
+	}
+
 	if (likely(OPTION_TS & opts->options)) {
 		if (unlikely(OPTION_SACK_ADVERTISE & opts->options)) {
 			*ptr++ = htonl((TCPOPT_SACK_PERM << 24) |
@@ -392,12 +409,6 @@ static void tcp_options_write(__be32 *ptr, struct tcp_sock *tp,
 		*ptr++ = htonl(opts->tsecr);
 	}
 
-	if (unlikely(opts->mss)) {
-		*ptr++ = htonl((TCPOPT_MSS << 24) |
-			       (TCPOLEN_MSS << 16) |
-			       opts->mss);
-	}
-
 	if (unlikely(OPTION_SACK_ADVERTISE & opts->options &&
 		     !(OPTION_TS & opts->options))) {
 		*ptr++ = htonl((TCPOPT_NOP << 24) |
-- 
1.5.2.2

Re: two other cases Re: [Bug 11721] after upgrade to 2.6.27 i cannot navigate

[Aldo Maggi]

[-- Attachment #1: Type: text/plain, Size: 1129 bytes --]

Il giorno Wed, 22 Oct 2008 13:00:01 +0300 (EEST)
"Ilpo Järvinen" <ilpo.jarvinen@helsinki.fi> ha scritto:

[...]

> 
> It would be nice for Aldo to check what the result will be with my
> second patch (only) using sack=1,ts=0,wscale=0. I guess it does but
> it's a bit unclear if nop's in front help or not (having the patch
> below should anyway help also in that case as the mss option gets
> moved before it anyway).

in order to avoid misunderstandings, i list herebelow my actions:

i've used on paperino the kernel 2.6.27.2 patched with ilpo's
21.10.2008 patch (see please comment 47 in
http://bugzilla.kernel.org/show_bug.cgi?id=11721, and NOT with the patch
contained in comment 58, right?)
i've modified the following files:
echo 0 > /proc/sys/net/ipv4/tcp_window_scaling
echo 0 > /proc/sys/net/ipv4/tcp_timestamps
echo 1 > /proc/sys/net/ipv4/tcp_sack

the results are that i CAN navigate (w3m kernel.org) and update my
system (apt-get update in debian).

should it be of any help to ilpo, i attach the usual tcpdump on the wan
eth of topolino (my home server/gw)

ciao!
aldo



[...]

[-- Attachment #2: 2210-tcpdump_2-6-27-2-ilpo-2110-patch --]
[-- Type: application/octet-stream, Size: 1233 bytes --]

13:33:40.892753 IP 192.168.254.1.44529 > 204.152.191.5.80: S 2225948920:2225948920(0) win 5840 <nop,nop,sackOK,mss 1460>
	0x0000:  00a0 c543 5feb 0010 5ac5 681c 0800 4500  ...C_...Z.h...E.
	0x0010:  0030 8fac 4000 3f06 61d3 c0a8 fe01 cc98  .0..@.?.a.......
	0x0020:  bf05 adf1 0050 84ad 48f8 0000 0000 7002  .....P..H.....p.
	0x0030:  16d0 a61f 0000 0101 0402 0204 05b4       ..............
13:33:41.112954 IP 192.168.254.1.44529 > 204.152.191.5.80: . ack 2927042638 win 5840
	0x0000:  00a0 c543 5feb 0010 5ac5 681c 0800 4500  ...C_...Z.h...E.
	0x0010:  0028 8fad 4000 3f06 61da c0a8 fe01 cc98  .(..@.?.a.......
	0x0020:  bf05 adf1 0050 84ad 48f9 ae77 204e 5010  .....P..H..w.NP.
	0x0030:  16d0 040e 0000                           ......
13:33:41.113227 IP 192.168.254.1.44529 > 204.152.191.5.80: P 0:240(240) ack 1 win 5840
	0x0000:  00a0 c543 5feb 0010 5ac5 681c 0800 4500  ...C_...Z.h...E.
	0x0010:  0118 8fae 4000 3f06 60e9 c0a8 fe01 cc98  ....@.?.`.......
	0x0020:  bf05 adf1 0050 84ad 48f9 ae77 204e 5018  .....P..H..w.NP.
	0x0030:  16d0 09f0 0000 4745 5420 2f20 4854 5450  ......GET./.HTTP
	0x0040:  2f31 2e30 0d0a 5573 6572 2d41 6765 6e74  /1.0..User-Agent
	0x0050:  3a20 7733 6d2f 302e 352e 320d 0a41 6363  :.w3m/0.5.2..Acc

Re: two other cases Re: [Bug 11721] after upgrade to 2.6.27 i cannot navigate

[Ilpo Järvinen]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: TEXT/PLAIN; charset=utf-8, Size: 2022 bytes --]

On Wed, 22 Oct 2008, Aldo Maggi wrote:

> Il giorno Wed, 22 Oct 2008 13:00:01 +0300 (EEST)
> "Ilpo Järvinen" <ilpo.jarvinen@helsinki.fi> ha scritto:
> 
> [...]
> 
> > 
> > It would be nice for Aldo to check what the result will be with my
> > second patch (only) using sack=1,ts=0,wscale=0. I guess it does but
> > it's a bit unclear if nop's in front help or not (having the patch
> > below should anyway help also in that case as the mss option gets
> > moved before it anyway).
> 
> in order to avoid misunderstandings, i list herebelow my actions:
> 
> i've used on paperino the kernel 2.6.27.2 patched with ilpo's
> 21.10.2008 patch (see please comment 47 in
> http://bugzilla.kernel.org/show_bug.cgi?id=11721, and NOT with the patch
> contained in comment 58, right?)

Yes, you understood me correctly. This was more or less for completeleness 
test only, to see if the nop's make any difference or not. You found out 
that they do which seemed the more sensible alternative anyway.

> i've modified the following files:
> echo 0 > /proc/sys/net/ipv4/tcp_window_scaling
> echo 0 > /proc/sys/net/ipv4/tcp_timestamps
> echo 1 > /proc/sys/net/ipv4/tcp_sack
> 
> the results are that i CAN navigate (w3m kernel.org) and update my
> system (apt-get update in debian).
>
> should it be of any help to ilpo, i attach the usual tcpdump on the wan
> eth of topolino (my home server/gw)

...It confirms that you made the right test. :-)

So only thing that was problem in your case is that sackOK option (or 
SACK_PERM in code notation) couldn't begin TCP options. With any other 
option as the first one, including nop, it does work. Thanks for all the 
testing you made and reporting it in the first place.

We'll put the latest patch (from comment #58) and it soon makes to 
2.6.28-rcsomething and hopefully with a semi-short delay to some
2.6.27.y, whichever y is appropriate at that time. I hope that the
ubuntu kernel people will then include that as well but it's beoynd
my control what they do.

Thanks again.

-- 
 i.

Re: two other cases Re: [Bug 11721] after upgrade to 2.6.27 i cannot navigate

[Aldo Maggi]

Il giorno Wed, 22 Oct 2008 15:09:40 +0300 (EEST)
"Ilpo Järvinen" <ilpo.jarvinen@helsinki.fi> ha scritto:

[...]

> So only thing that was problem in your case is that sackOK option (or 
> SACK_PERM in code notation) couldn't begin TCP options. With any
> other option as the first one, including nop, it does work. Thanks
> for all the testing you made and reporting it in the first place.

i thank you for your efforts in solving this problem!
it has been a real pleasere for me to cooperate with you!

all the best to you all

[...]
 
> Thanks again.
> 

i thank you! :-)

aldo

Re: tcp_sack problem Re: [Bug 11721] after upgrade to 2.6.27 i cannot navigate

[Jarek Poplawski]

On Sat, Oct 25, 2008 at 10:15:15PM +0200, Aldo Maggi wrote:
> Il giorno Tue, 21 Oct 2008 07:49:54 +0000
> Jarek Poplawski <jarkao2@gmail.com> ha scritto:
> 
> > On Tue, Oct 21, 2008 at 09:27:21AM +0200, Aldo Maggi wrote:
> > ...
> > > as soon as i've time i'll replace the modem and run some tests.
> 
> just to let you know!
> i have changed my modem with a new one.
> the bug has disappeared with no modification of the tcp files.
> this shows that the problem was due to my old zyxel.
> 
> ciao!
> aldo
> 

Aldo, I think it's a very useful information, so I forward this to the
people.

Thanks again,
Jarek P.

Re: two other cases Re: [Bug 11721] after upgrade to 2.6.27 i cannot navigate

[David Miller]

From: "Ilpo Järvinen" <ilpo.jarvinen@helsinki.fi>
Date: Wed, 22 Oct 2008 13:00:01 +0300 (EEST)

> [PATCH] tcp: Restore ordering of TCP options for the sake of inter-operability
> 
> This is not our bug! Sadly some devices cannot cope with the change
> of TCP option ordering which was a result of the recent rewrite of
> the option code (not that there was some particular reason steming
> from the rewrite for the reordering) though any ordering of TCP
> options is perfectly legal. Thus we restore the original ordering
> to allow interoperability with/through such broken devices and add
> some warning about this trap. Since the reordering just happened
> without any particular reason, this change shouldn't cost us
> anything.
> 
> There are already couple of known failure reports (within close
> proximity of the last release), so the problem might be more
> wide-spread than a single device. And other reports which may
> be due to the same problem though the symptoms were less obvious.
> Analysis of one of the case revealed (with very high probability)
> that sack capability cannot be negotiated as the first option
> (SYN never got a response).
> 
> Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
> Reported-by: Aldo Maggi <sentiniate@tiscali.it>
> Tested-by: Aldo Maggi <sentiniate@tiscali.it>

Applied and I'll queue up for -stable, thanks!

tcp_sack problem Re: [Bug 11721] after upgrade to 2.6.27 i cannot navigate

[Jarek Poplawski]

Nice job Aldo!

I forward this message to netdev and Cc our best tcp expert.
Any new replies should be rather "Reply All" (not bugzilla only).

Thanks,
Jarek P.

On Sat, Oct 18, 2008 at 01:20:56PM -0700, bugme-daemon@bugzilla.kernel.org wrote:
> http://bugzilla.kernel.org/show_bug.cgi?id=11721
> 
...
> ------- Comment #30 from sentiniate@tiscali.it  2008-10-18 13:20 -------
> absolutely i'm not fed up! and actually i thank you for your patience.
> 
> i set paperino as you wrote and added ifconfig eth0 mtu 1400 as well
> i set the same value of mtu in topolino as well, i hope i was not wrong doing
> so!
> 
> i attach the files you ask for, but now i can navigate with 2.6.27-rc1-git1 as
> well!
> 
> afterwards i've run some more tests and have found out that the culprit is
> tcp_sack, if using kernel 2.6.27-rc1-gt1 it is set to "1" i cannot navigate if
> to "0" i can.
> so i enclose, should it be of any help, also the tcpdump on eth0 of topolino
> when on paperino tcp_sack is set to "1".
> 
> thanks
> aldo
> 
> 
> -- 
> Configure bugmail: http://bugzilla.kernel.org/userprefs.cgi?tab=email
> ------- You are receiving this mail because: -------
> You are on the CC list for the bug, or are watching someone who is.

Re: tcp_sack problem Re: [Bug 11721] after upgrade to 2.6.27 i cannot navigate

[Jarek Poplawski]

[RESEND] I forgot to add Aldo's email before - sorry!

On Sat, Oct 18, 2008 at 11:02:52PM +0200, Jarek Poplawski wrote:
> Nice job Aldo!
> 
> I forward this message to netdev and Cc our best tcp expert.
> Any new replies should be rather "Reply All" (not bugzilla only).
> 
> Thanks,
> Jarek P.
> 
> On Sat, Oct 18, 2008 at 01:20:56PM -0700, bugme-daemon@bugzilla.kernel.org wrote:
> > http://bugzilla.kernel.org/show_bug.cgi?id=11721
> > 
> ...
> > ------- Comment #30 from sentiniate@tiscali.it  2008-10-18 13:20 -------
> > absolutely i'm not fed up! and actually i thank you for your patience.
> > 
> > i set paperino as you wrote and added ifconfig eth0 mtu 1400 as well
> > i set the same value of mtu in topolino as well, i hope i was not wrong doing
> > so!
> > 
> > i attach the files you ask for, but now i can navigate with 2.6.27-rc1-git1 as
> > well!
> > 
> > afterwards i've run some more tests and have found out that the culprit is
> > tcp_sack, if using kernel 2.6.27-rc1-gt1 it is set to "1" i cannot navigate if
> > to "0" i can.
> > so i enclose, should it be of any help, also the tcpdump on eth0 of topolino
> > when on paperino tcp_sack is set to "1".
> > 
> > thanks
> > aldo
> > 
> > 
> > -- 
> > Configure bugmail: http://bugzilla.kernel.org/userprefs.cgi?tab=email
> > ------- You are receiving this mail because: -------
> > You are on the CC list for the bug, or are watching someone who is.

Re: tcp_sack problem Re: [Bug 11721] after upgrade to 2.6.27 i cannot navigate

[Jarek Poplawski]

>> On Sat, Oct 18, 2008 at 01:20:56PM -0700, bugme-daemon@bugzilla.kernel.org wrote:
>>> http://bugzilla.kernel.org/show_bug.cgi?id=11721
...
>>> ------- Comment #30 from sentiniate@tiscali.it  2008-10-18 13:20 -------
...
>>> afterwards i've run some more tests and have found out that the culprit is
>>> tcp_sack, if using kernel 2.6.27-rc1-gt1 it is set to "1" i cannot navigate if
>>> to "0" i can.
...

Aldo, for curiosity, could you also try if this change only
(instead of tcp_sack etc.) can't do similar effect?:

echo 0 > /proc/sys/net/ipv4/tcp_timestamps

Thanks,
Jarek P.

Re: tcp_sack problem Re: [Bug 11721] after upgrade to 2.6.27 i cannot navigate

[Ilpo Järvinen]

On Sat, 18 Oct 2008, Jarek Poplawski wrote:

> [RESEND] I forgot to add Aldo's email before - sorry!
> 
> On Sat, Oct 18, 2008 at 11:02:52PM +0200, Jarek Poplawski wrote:
> > Nice job Aldo!
> > 
> > I forward this message to netdev and Cc our best tcp expert.
> > Any new replies should be rather "Reply All" (not bugzilla only).
> > 
> > Thanks,
> > Jarek P.
> > 
> > On Sat, Oct 18, 2008 at 01:20:56PM -0700, bugme-daemon@bugzilla.kernel.org wrote:
> > > http://bugzilla.kernel.org/show_bug.cgi?id=11721
> > > 
> > ...
> > > ------- Comment #30 from sentiniate@tiscali.it  2008-10-18 13:20 -------
> > > absolutely i'm not fed up! and actually i thank you for your patience.
> > > 
> > > i set paperino as you wrote and added ifconfig eth0 mtu 1400 as well
> > > i set the same value of mtu in topolino as well, i hope i was not wrong doing
> > > so!
> > > 
> > > i attach the files you ask for, but now i can navigate with 2.6.27-rc1-git1 as
> > > well!
> > > 
> > > afterwards i've run some more tests and have found out that the culprit is
> > > tcp_sack, if using kernel 2.6.27-rc1-gt1 it is set to "1" i cannot navigate if
> > > to "0" i can.
> > > so i enclose, should it be of any help, also the tcpdump on eth0 of topolino
> > > when on paperino tcp_sack is set to "1".

So this ended up into tcp domain after all (I took earlier a brief 
anyway and found out that there are not that many changes 2.6.26..2.6.27 
-- net/ipv4/tcp*.c include/net/tcp.h)...

I compared your packet against a good one from elsewhere.. I couldn't 
compare your latest dumps fully because attachments 18366 and 18367 are 
with different TCP options (you forgot zeros to sysctls in them?)... 
Anyway, only thing that seemed to be different to that case from elsewhere 
were those extra bytes in the beginning (some below ip protocol that 
gets captured by tcpdump?) which are equal in both working and broken case 
of yours and the different ordering of the tcp options as noted by Jarek 
earlier. I tried to go through the fields one by one but nothing seemed to 
be wrong...

...Might be something crazy in the way that is too picky on tcp option 
ordering which wouldn't surprise me that much... :-) Please try if
the patch below does any difference (on paperino is enough, the gw seems 
innocent here).

If that didn't help, can you please restore the sysctls to 1 and redo
2.6.26.6 dump (like in attachment 18366) so that I get a fully comparable 
sample.

-- 
 i.


diff --git a/net/ipv4/tcp_output.c b/net/ipv4/tcp_output.c
index 990a584..850a4e9 100644
--- a/net/ipv4/tcp_output.c
+++ b/net/ipv4/tcp_output.c
@@ -376,6 +376,12 @@ static void tcp_options_write(__be32 *ptr, struct tcp_sock *tp,
 		*md5_hash = NULL;
 	}
 
+	if (unlikely(opts->mss)) {
+		*ptr++ = htonl((TCPOPT_MSS << 24) |
+			       (TCPOLEN_MSS << 16) |
+			       opts->mss);
+	}
+
 	if (likely(OPTION_TS & opts->options)) {
 		if (unlikely(OPTION_SACK_ADVERTISE & opts->options)) {
 			*ptr++ = htonl((TCPOPT_SACK_PERM << 24) |
@@ -392,12 +398,6 @@ static void tcp_options_write(__be32 *ptr, struct tcp_sock *tp,
 		*ptr++ = htonl(opts->tsecr);
 	}
 
-	if (unlikely(opts->mss)) {
-		*ptr++ = htonl((TCPOPT_MSS << 24) |
-			       (TCPOLEN_MSS << 16) |
-			       opts->mss);
-	}
-
 	if (unlikely(OPTION_SACK_ADVERTISE & opts->options &&
 		     !(OPTION_TS & opts->options))) {
 		*ptr++ = htonl((TCPOPT_NOP << 24) |

Re: tcp_sack problem Re: [Bug 11721] after upgrade to 2.6.27 i cannot navigate

[Jarek Poplawski]

On Mon, Oct 20, 2008 at 12:38:51PM +0300, Ilpo Järvinen wrote:
...
> ...Might be something crazy in the way that is too picky on tcp option 
> ordering which wouldn't surprise me that much... :-) Please try if
> the patch below does any difference (on paperino is enough, the gw seems 
> innocent here).

I've just thought about this too(!), and it seems that turning of
tcp_timestamps should do similar reorder. But of course, I'll try
not to disturb anymore...

Thanks,
Jarek P.

Re: tcp_sack problem Re: [Bug 11721] after upgrade to 2.6.27 i cannot navigate

[Ilpo Järvinen]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 853 bytes --]

On Mon, 20 Oct 2008, Jarek Poplawski wrote:

> On Mon, Oct 20, 2008 at 12:38:51PM +0300, Ilpo Järvinen wrote:
> ...
> > ...Might be something crazy in the way that is too picky on tcp option 
> > ordering which wouldn't surprise me that much... :-) Please try if
> > the patch below does any difference (on paperino is enough, the gw seems 
> > innocent here).
> 
> I've just thought about this too(!), and it seems that turning of
> tcp_timestamps should do similar reorder. But of course, I'll try
> not to disturb anymore...

Yeah, you should be right. No need to patch+compile+boot hassle him, 
turning timestamps should be "enough" to move mss into first slot... :-)

If that works we probably need to play around a bit to figure out what
is the actual cause (since it could involve timestamps or mss which the 
sysctl alone cannot prove).

-- 
 i.

Re: tcp_sack problem Re: [Bug 11721] after upgrade to 2.6.27 i cannot navigate

[Jarek Poplawski]

On Mon, Oct 20, 2008 at 08:47:37AM -0700, bugme-daemon@bugzilla.kernel.org wrote:
> http://bugzilla.kernel.org/show_bug.cgi?id=11721
...
> ------- Comment #40 from sentiniate@tiscali.it  2008-10-20 08:47 -------
> ok setting echo 0 > /proc/sys/net/ipv4/tcp_timestamps solves the problem as
> well.
> 
> anyway i'm willing to do whatever test you deem necessary if this can help you
> solve the issue, so as soon as i can i'll apply the patch provided by  Ilpo
> Järvinen and compile (do not worry i've been a linux user for the last 10
> years and up to 4 or 5 yrs ago i used to compile even three times a week!)
> 
> thanks
> aldo

Aldo, I forward this message to Ilpo and netdev, because "they" aren't
on the Cc list of this report. (With next replies try to use the mail
with their addresses.)

I think, you could test Ilpo's patch as well: this would make slightly
different test/header. (Of course, all these /proc tcp values and mtus
should be back to normal.)

Thanks,
Jarek P.

Re: tcp_sack problem Re: [Bug 11721] after upgrade to 2.6.27 i cannot navigate

[Aldo Maggi]

[-- Attachment #1: Type: text/plain, Size: 918 bytes --]

Jarek,
sorry i did not read your last msg before, so i have used the bug page
for posting.

i repeat here what i wrote there:

Ilpo,
i'm sending herewith attached the output of
tcpdump -i eth0 -nXX -c3 'dst port 80 and tcp-syn != 0' on topolino when:
1) kernel 2.6.26.6 is running on paperino and tcp_window_scaling,
tcp_timestamps, tcp_sack are set to "1"
2) kernel 2.6.26.6 is running on paperino and tcp_window_scaling,
tcp_timestamps, tcp_sack are set to "0"
3) kernel 2.6.27-rc1-git1 is running on paperino and tcp_window_scaling,
tcp_timestamps, tcp_sack are set to "1"
4) kernel 2.6.27-rc1-git1 is running on paperino and tcp_window_scaling,
tcp_timestamps, tcp_sack are set to "0"

moreover, i read more carefully what you and Jarek wrote above, please let me
know if you deem still necessary that i apply the patch you provided and
compile or if you like i compile the latest kernel or whatever.

thanks
aldo

[-- Attachment #2: tcpdump_2-6-26_tcp_0 --]
[-- Type: application/octet-stream, Size: 1217 bytes --]

21:27:18.619016 IP 192.168.254.1.60630 > 204.152.191.37.80: S 1427374161:1427374161(0) win 5840 <mss 1460>
	0x0000:  00a0 c543 5feb 0010 5ac5 681c 0800 4500  ...C_...Z.h...E.
	0x0010:  002c 4c04 4000 3f06 a55f c0a8 fe01 cc98  .,L.@.?.._......
	0x0020:  bf25 ecd6 0050 5514 0051 0000 0000 6002  .%...PU..Q....`.
	0x0030:  16d0 f461 0000 0204 05b4                 ...a......
21:27:18.839051 IP 192.168.254.1.60630 > 204.152.191.37.80: . ack 2813695036 win 5840
	0x0000:  00a0 c543 5feb 0010 5ac5 681c 0800 4500  ...C_...Z.h...E.
	0x0010:  0028 4c05 4000 3f06 a562 c0a8 fe01 cc98  .(L.@.?..b......
	0x0020:  bf25 ecd6 0050 5514 0052 a7b5 943c 5010  .%...PU..R...<P.
	0x0030:  16d0 d01c 0000                           ......
21:27:18.839293 IP 192.168.254.1.60630 > 204.152.191.37.80: P 0:240(240) ack 1 win 5840
	0x0000:  00a0 c543 5feb 0010 5ac5 681c 0800 4500  ...C_...Z.h...E.
	0x0010:  0118 4c06 4000 3f06 a471 c0a8 fe01 cc98  ..L.@.?..q......
	0x0020:  bf25 ecd6 0050 5514 0052 a7b5 943c 5018  .%...PU..R...<P.
	0x0030:  16d0 d5fe 0000 4745 5420 2f20 4854 5450  ......GET./.HTTP
	0x0040:  2f31 2e30 0d0a 5573 6572 2d41 6765 6e74  /1.0..User-Agent
	0x0050:  3a20 7733 6d2f 302e 352e 320d 0a41 6363  :.w3m/0.5.2..Acc

[-- Attachment #3: tcpdump_2-6-26_tcp_1 --]
[-- Type: application/octet-stream, Size: 1459 bytes --]

21:25:22.757937 IP 192.168.254.1.57412 > 204.152.191.5.80: S 3914166537:3914166537(0) win 5840 <mss 1460,sackOK,timestamp 1694512 0,nop,wscale 6>
	0x0000:  00a0 c543 5feb 0010 5ac5 681c 0800 4500  ...C_...Z.h...E.
	0x0010:  003c aa39 4000 3f06 473a c0a8 fe01 cc98  .<.9@.?.G:......
	0x0020:  bf05 e044 0050 e94d 7109 0000 0000 a002  ...D.P.Mq.......
	0x0030:  16d0 d0b2 0000 0204 05b4 0402 080a 0019  ................
	0x0040:  db30 0000 0000 0103 0306                 .0........
21:25:22.978410 IP 192.168.254.1.57412 > 204.152.191.5.80: . ack 506544671 win 92 <nop,nop,timestamp 1694567 1021078027>
	0x0000:  00a0 c543 5feb 0010 5ac5 681c 0800 4500  ...C_...Z.h...E.
	0x0010:  0034 aa3a 4000 3f06 4741 c0a8 fe01 cc98  .4.:@.?.GA......
	0x0020:  bf05 e044 0050 e94d 710a 1e31 421f 8010  ...D.P.Mq..1B...
	0x0030:  005c 0e73 0000 0101 080a 0019 db67 3cdc  .\.s.........g<.
	0x0040:  6a0b                                     j.
21:25:22.978644 IP 192.168.254.1.57412 > 204.152.191.5.80: P 0:240(240) ack 1 win 92 <nop,nop,timestamp 1694567 1021078027>
	0x0000:  00a0 c543 5feb 0010 5ac5 681c 0800 4500  ...C_...Z.h...E.
	0x0010:  0124 aa3b 4000 3f06 4650 c0a8 fe01 cc98  .$.;@.?.FP......
	0x0020:  bf05 e044 0050 e94d 710a 1e31 421f 8018  ...D.P.Mq..1B...
	0x0030:  005c 1455 0000 0101 080a 0019 db67 3cdc  .\.U.........g<.
	0x0040:  6a0b 4745 5420 2f20 4854 5450 2f31 2e30  j.GET./.HTTP/1.0
	0x0050:  0d0a 5573 6572 2d41 6765 6e74 3a20 7733  ..User-Agent:.w3

[-- Attachment #4: tcpdump_2-6-27-rc1-git1_tcp_0 --]
[-- Type: application/octet-stream, Size: 1214 bytes --]

21:33:43.584625 IP 192.168.254.1.56735 > 204.152.191.5.80: S 2776174833:2776174833(0) win 5840 <mss 1460>
	0x0000:  00a0 c543 5feb 0010 5ac5 681c 0800 4500  ...C_...Z.h...E.
	0x0010:  002c 1487 4000 3f06 dcfc c0a8 fe01 cc98  .,..@.?.........
	0x0020:  bf05 dd9f 0050 a579 10f1 0000 0000 6002  .....P.y......`.
	0x0030:  16d0 a2b3 0000 0204 05b4                 ..........
21:33:43.805846 IP 192.168.254.1.56735 > 204.152.191.5.80: . ack 4067528717 win 5840
	0x0000:  00a0 c543 5feb 0010 5ac5 681c 0800 4500  ...C_...Z.h...E.
	0x0010:  0028 1488 4000 3f06 dcff c0a8 fe01 cc98  .(..@.?.........
	0x0020:  bf05 dd9f 0050 a579 10f2 f271 900d 5010  .....P.y...q..P.
	0x0030:  16d0 37e1 0000                           ..7...
21:33:43.806085 IP 192.168.254.1.56735 > 204.152.191.5.80: P 0:240(240) ack 1 win 5840
	0x0000:  00a0 c543 5feb 0010 5ac5 681c 0800 4500  ...C_...Z.h...E.
	0x0010:  0118 1489 4000 3f06 dc0e c0a8 fe01 cc98  ....@.?.........
	0x0020:  bf05 dd9f 0050 a579 10f2 f271 900d 5018  .....P.y...q..P.
	0x0030:  16d0 3dc3 0000 4745 5420 2f20 4854 5450  ..=...GET./.HTTP
	0x0040:  2f31 2e30 0d0a 5573 6572 2d41 6765 6e74  /1.0..User-Agent
	0x0050:  3a20 7733 6d2f 302e 352e 320d 0a41 6363  :.w3m/0.5.2..Acc

[-- Attachment #5: tcpdump_2-6-27-rc1-git1_tcp_1 --]
[-- Type: application/octet-stream, Size: 1446 bytes --]

21:31:29.392675 IP 192.168.254.1.35711 > 204.152.191.37.80: S 654709271:654709271(0) win 5840 <sackOK,timestamp 4294934889 0,mss 1460,nop,wscale 6>
	0x0000:  00a0 c543 5feb 0010 5ac5 681c 0800 4500  ...C_...Z.h...E.
	0x0010:  003c 937e 4000 3f06 5dd5 c0a8 fe01 cc98  .<.~@.?.].......
	0x0020:  bf25 8b7f 0050 2706 1217 0000 0000 a002  .%...P'.........
	0x0030:  16d0 a072 0000 0402 080a ffff 8169 0000  ...r.........i..
	0x0040:  0000 0204 05b4 0103 0306                 ..........
21:31:32.390480 IP 192.168.254.1.35711 > 204.152.191.37.80: S 654709271:654709271(0) win 5840 <sackOK,timestamp 4294935639 0,mss 1460,nop,wscale 6>
	0x0000:  00a0 c543 5feb 0010 5ac5 681c 0800 4500  ...C_...Z.h...E.
	0x0010:  003c 937f 4000 3f06 5dd4 c0a8 fe01 cc98  .<..@.?.].......
	0x0020:  bf25 8b7f 0050 2706 1217 0000 0000 a002  .%...P'.........
	0x0030:  16d0 9d84 0000 0402 080a ffff 8457 0000  .............W..
	0x0040:  0000 0204 05b4 0103 0306                 ..........
21:31:38.390344 IP 192.168.254.1.35711 > 204.152.191.37.80: S 654709271:654709271(0) win 5840 <sackOK,timestamp 4294937139 0,mss 1460,nop,wscale 6>
	0x0000:  00a0 c543 5feb 0010 5ac5 681c 0800 4500  ...C_...Z.h...E.
	0x0010:  003c 9380 4000 3f06 5dd3 c0a8 fe01 cc98  .<..@.?.].......
	0x0020:  bf25 8b7f 0050 2706 1217 0000 0000 a002  .%...P'.........
	0x0030:  16d0 97a8 0000 0402 080a ffff 8a33 0000  .............3..
	0x0040:  0000 0204 05b4 0103 0306                 ..........

Re: tcp_sack problem Re: [Bug 11721] after upgrade to 2.6.27 i cannot navigate

[Jarek Poplawski]

On Mon, Oct 20, 2008 at 09:48:18PM +0200, Aldo Maggi wrote:
...
> moreover, i read more carefully what you and Jarek wrote above, please let me
> know if you deem still necessary that i apply the patch you provided and
> compile or if you like i compile the latest kernel or whatever.

Aldo, my advice, in case Ilpo is busy now:

Since it looks like the problem/bug isn't in the kernel, but rather
somewhere in between (router, modem etc.) which is mislead by changed
(compared to 2.6.26) order of tcp options, I doubt Ilpo needs now
these dumps so much: it should be enough to say works or doesn't work.

You have checked it works in 2 cases (without tcp_sack, and without
tcp_timestamp) when mss is the first option again. With Ilpo's patch
these two options could be still present, and mss at the beginning.
It should work because it's like 2.6.26 did this, but I think, it
would be nice to check this anyway, if it's not a problem. Then
trying the latest kernel (at least 2.6.27) with this patch should be
most useful.

Thanks,
Jarek P.

Re: tcp_sack problem Re: [Bug 11721] after upgrade to 2.6.27 i cannot navigate

[Ilpo Järvinen]

On Mon, 20 Oct 2008, Aldo Maggi wrote:

> i repeat here what i wrote there:
> 
> Ilpo,
> i'm sending herewith attached the output of
> tcpdump -i eth0 -nXX -c3 'dst port 80 and tcp-syn != 0' on topolino when:
> 1) kernel 2.6.26.6 is running on paperino and tcp_window_scaling,
> tcp_timestamps, tcp_sack are set to "1"
> 2) kernel 2.6.26.6 is running on paperino and tcp_window_scaling,
> tcp_timestamps, tcp_sack are set to "0"
> 3) kernel 2.6.27-rc1-git1 is running on paperino and tcp_window_scaling,
> tcp_timestamps, tcp_sack are set to "1"
> 4) kernel 2.6.27-rc1-git1 is running on paperino and tcp_window_scaling,
> tcp_timestamps, tcp_sack are set to "0"

Thanks, very unambiquously being told what each case is, helps a lot... :-)

Difference is: ip checksum, ipid, lsb of dstaddr, sport, seqno, tcp 
checksum, timestamp. Plus the option reordering as discussed before. Here 
are the options:

    28  0204  <
    29  05b4  <
    30  0402    0402
    31  080a    080a
    32  0019  | ffff
    33  db30  | 8169
    34  0000    0000
    35  0000    0000
    36        > 0204
    37        > 05b4
    38  0103    0103
    39  0306    0306

Could you try if tcp_window_scaling=1, tcp_sack=1, tcp_timestamps=0 also 
works, if you didn't already (it's not that obvious from the earlier 
acknowledgement you gave that these were the exact options you used or 
not)...

> moreover, i read more carefully what you and Jarek wrote above, please let me
> know if you deem still necessary that i apply the patch you provided and
> compile or if you like i compile the latest kernel or whatever.

If that ws=1,s=1,ts=0 test succeeds, then testing my patch would be useful 
too. Anyway it seems that we're dealing with some violation of spec here
(by either the peer or some middle node, 2.6.27-whatever is not doing 
something outside of spec), just trying to narrow down which of the 
options is the actual cause. We might need to try later some other 
ordering of options too to rule out possibilities by trial-and-error
(ie., unless you have a better knowledge about the devices on the path, 
information which I sort of expect to be out of (y)our reach :-)).


-- 
 i.

Re: tcp_sack problem Re: [Bug 11721] after upgrade to 2.6.27 i cannot navigate

[Ilpo Järvinen]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 2018 bytes --]

Reply to all please... ...I restored them again for you... :-)

On Mon, 20 Oct 2008, Aldo Maggi wrote:

> Il giorno Mon, 20 Oct 2008 23:55:48 +0300 (EEST)
> "Ilpo Järvinen" <ilpo.jarvinen@helsinki.fi> ha scritto:
> 
> [...]
> 
> > Could you try if tcp_window_scaling=1, tcp_sack=1, tcp_timestamps=0
> > also works, if you didn't already (it's not that obvious from the
> > earlier acknowledgement you gave that these were the exact options
> > you used or not)...
> 
> sorry! i was not precise!
> those are exactly the options i tried in my acknowledgment no. 40,
> i.e. tcp_window_scaling and tcp_sack equal to "1" and tcp_timestamps=0

Np, thanks for the confirmation. I sort of assumed this already since 
the alternative interpretation of Jarek's mails didn't make too much 
sense after all (and I jumped into this thread in between, so I just 
scanned them through with haste on the first time concentrating more
on your responses and which of the logs I should download :-)).

> > > moreover, i read more carefully what you and Jarek wrote above,
> > > please let me know if you deem still necessary that i apply the
> > > patch you provided and compile or if you like i compile the latest
> > > kernel or whatever.
> > 
> > If that ws=1,s=1,ts=0 test succeeds, then testing my patch would be
> > useful too. 
> 
> i'm right now compiling the -27-rc1 kernel, on my pc it will take at
> least 3 hrs (that is, it will finish 02.00 rome time :-D )

Ok, I'll check that tomorrow then as mine trimmed down 2.6.28-rc (ah, 
it's still -g something, no rc1 just yet) did complete already :-), just 
going to try to bombard google.it from here too and then it's the bedtime 
for me...

> should it work i'll apply your patch t0 2.6.27.2 and compile again. 

Ok, it's quite likely that both will work with the patch. Anyway, it's 
enough that you do the future tests with 2.6.27.y if we decide to try some 
other ordering (I'll try to think a bit of that tomorrow to see what 
would be to most sensible ordering).

-- 
 i.

Re: tcp_sack problem Re: [Bug 11721] after upgrade to 2.6.27 i cannot navigate

[Aldo Maggi]

Il giorno Tue, 21 Oct 2008 00:51:26 +0300 (EEST)
"Ilpo Järvinen" <ilpo.jarvinen@helsinki.fi> ha scritto:

> Reply to all please... ...I restored them again for you... :-)

sorry! i'll be more careful in the future!


[...]

thanks :-)
aldo

Re: tcp_sack problem Re: [Bug 11721] after upgrade to 2.6.27 i cannot navigate

[Aldo Maggi]

[-- Attachment #1: Type: text/plain, Size: 475 bytes --]


[...]

> > 
> > i'm right now compiling the -27-rc1 kernel, on my pc it will take at
> > least 3 hrs (that is, it will finish 02.00 rome time :-D )
> 

ok! the new kernel with ilpo's patch works!!
please find herewith attached the tcpdump output on eth0 on topolino 
while on paperino kernel 2.6.27-rc1-git1 with ilpo patch was running ,
the three tcp files all set to 1

[...]

as soon as i come back home i will start compilation of 2.6.27.2 with
ilpo patch

thanks 
aldo

[-- Attachment #2: tcpdump_2-6-27-rc1-git1-ilpo-tcp_1 --]
[-- Type: application/octet-stream, Size: 1457 bytes --]

07:15:43.733574 IP 192.168.254.1.57604 > 204.152.191.5.80: S 2965188623:2965188623(0) win 5840 <mss 1460,sackOK,timestamp 145214 0,nop,wscale 6>
	0x0000:  00a0 c543 5feb 0010 5ac5 681c 0800 4500  ...C_...Z.h...E.
	0x0010:  003c 0731 4000 3f06 ea42 c0a8 fe01 cc98  .<.1@.?..B......
	0x0020:  bf05 e104 0050 b0bd 300f 0000 0000 a002  .....P..0.......
	0x0030:  16d0 ed86 0000 0204 05b4 0402 080a 0002  ................
	0x0040:  373e 0000 0000 0103 0306                 7>........
07:15:43.943869 IP 192.168.254.1.57604 > 204.152.191.5.80: . ack 1887470968 win 92 <nop,nop,timestamp 145267 1056499011>
	0x0000:  00a0 c543 5feb 0010 5ac5 681c 0800 4500  ...C_...Z.h...E.
	0x0010:  0034 0732 4000 3f06 ea49 c0a8 fe01 cc98  .4.2@.?..I......
	0x0020:  bf05 e104 0050 b0bd 3010 7080 8578 8010  .....P..0.p..x..
	0x0030:  005c 184c 0000 0101 080a 0002 3773 3ef8  .\.L........7s>.
	0x0040:  e543                                     .C
07:15:43.944110 IP 192.168.254.1.57604 > 204.152.191.5.80: P 0:240(240) ack 1 win 92 <nop,nop,timestamp 145267 1056499011>
	0x0000:  00a0 c543 5feb 0010 5ac5 681c 0800 4500  ...C_...Z.h...E.
	0x0010:  0124 0733 4000 3f06 e958 c0a8 fe01 cc98  .$.3@.?..X......
	0x0020:  bf05 e104 0050 b0bd 3010 7080 8578 8018  .....P..0.p..x..
	0x0030:  005c 1e2e 0000 0101 080a 0002 3773 3ef8  .\..........7s>.
	0x0040:  e543 4745 5420 2f20 4854 5450 2f31 2e30  .CGET./.HTTP/1.0
	0x0050:  0d0a 5573 6572 2d41 6765 6e74 3a20 7733  ..User-Agent:.w3

Re: tcp_sack problem Re: [Bug 11721] after upgrade to 2.6.27 i cannot navigate

[Ilpo Järvinen]

In general, please use reply to all when dealing with linux developers (so 
that related people & lists end up being cc'ed), we just work that way :-).
I did restore CCs for you this time.

On Mon, 20 Oct 2008, Aldo Maggi wrote:

> bear with me but i do not wish to create problems to you all, therefore
> i'd like to be sure to be doing the correct procedure before going on
> with the compilation:
> 
> i've used your patch with the sources of kernel 2.6.27-rc1 which i had
> already patched vith git1 because of problems with the compilation
> (problems which had been solved with git1 patch).
> 
> what follows is the output of patch, is it correct?
> 
> paperino:/usr/src/linux-2.6.27-rc1-git1# patch -p1 < ../patch_iarno
> patching file net/ipv4/tcp_output.c
> Hunk #1 succeeded at 371 (offset -5 lines).
> Hunk #2 succeeded at 393 (offset -5 lines).

-5 offset might well happen and patch was able to apply it right 
regardless of that (I might have had some unrelated modifications 
or so in my local dev tree which causes the line numbers to shift
a bit).

> i'll anyway go on with the compilation, but i don't want that you
> expect i do something and i do something else! :-D

I'm fine with the kernel and your procedure... :-) And also 2.6.27 won't 
make much difference to 2.6.27-rc-something I think, so you can ignore 
Jarek's suggestion to use 2.6.27+patch if you feel -rc is easier for you 
now that you've started with it. I'm fine with either kernel 
version+patch.

-- 
 i.