preliminary conclusions regarding window size issues

preliminary conclusions regarding window size issues

[bert hubert]

Two things:

1) packages.gentoo.org is currently unreachable by 2.6.7-recent.
This has been confirmed from several places, very easy to reproduce. Bug has
been filed with gentoo to fix their firewall.

2) What Alessandro Suardi sees is highly similar, except that he has it with
*all* remotes, except for google.it and a very small number of other
servers.

This is what Alessandro saw going out. We artificially lowered the MTU
because of possible tunelling loss:

01:03:36.323132 192.168.1.3.33992 > 213.244.168.210.10000: S [tcp sum ok] 3497585848:3497585848(0) 
	win 5440 <mss 1360,sackOK,timestamp 2311996 0,nop,wscale 7> 
	(DF) (ttl 64, id 43908, len 60)
01:03:36.396660 213.244.168.210.10000 > 192.168.1.3.33992: S [tcp sum ok] 3030562636:3030562636(0) 
	ack 3497585849 win 5792 <mss 1452,sackOK,timestamp  2142457957 2311996,nop,wscale 0> 
	(DF) (ttl 53, id 0, len 60)
01:03:36.396719 192.168.1.3.33992 > 213.244.168.210.10000: . [tcp sum ok]
	ack 1 win 42 <nop,nop,timestamp 2312084 2142457957> 
	(DF) (ttl 64, id 43909, len 52)

Perfect SYN, SYN|ACK, ACK.

01:03:36.397362 192.168.1.3.33992 > 213.244.168.210.10000: P 1:463(462) ack 1 win 42 
	<nop,nop,timestamp 2312085 2142457957> 
	(DF) (ttl 64, id 43910, len 514)

The GET request.

01:03:36.497588 213.244.168.210.10000 > 192.168.1.3.33992: . [tcp sum ok] ack 463 
	win 6432 <nop,nop,timestamp 2142457967 2312085> 
	(DF) (ttl 53, id 59171, len 52) 

And acked by my server. This trace is identical to what I see on the
receiving end:

29.84 62.211.168.xx.33992 > 213.244.168.210.10000: S [tcp sum ok] 3497585848:3497585848(0) 
	win 5440 <mss 1360,sackOK,timestamp 2311996 0,nop,wscale 7> 
	(DF) (ttl 50, id 43908, len 60)
29.84 213.244.168.210.10000 > 62.211.168.xx.33992: S [tcp sum ok] 3030562636:3030562636(0) 
	ack 3497585849 win 5792 <mss 1460,sackOK,timestamp 2142457957 2311996,nop,wscale 0>  
	(DF) (ttl 64, id 0, len 60)
29.93 62.211.168.xx.33992 > 213.244.168.210.10000: . [tcp sum ok] 1:1(0) 
	ack 1 win 42 <nop,nop,timestamp 2312084 2142457957> 
	(DF) (ttl 50, id 43909, len 52)

29.95 62.211.168.xx.33992 > 213.244.168.210.10000: P [tcp sum ok] 1:463(462)
	ack 1 win 42 <nop,nop,timestamp 2312085 2142457957> 
	(DF) (ttl 50, id 43910, len 514)
29.95 213.244.168.210.10000 > 62.211.168.xx.33992: . [tcp sum ok] 1:1(0) 
	ack 463 win 6432 <nop,nop,timestamp 2142457967 2312085> 
	(DF) (ttl 64, id 59171, len 52)

Except for TTL and NAT, this is identical.

>From here, things start to differ. I measure that I send out:

29.95 213.244.168.210.10000 > 62.211.168.xx.33992: . [tcp sum ok] 1:1349(1348) 
	ack 463 win 6432 <nop,nop,timestamp 2142457967 2312085> 
	(DF) (ttl 64, id 59172, len 1400)
29.95 213.244.168.210.10000 > 62.211.168.xx.33992: P [tcp sum ok] 1349:2697(1348) 
	ack 463 win 6432 <nop,nop,timestamp 2142457967 2312085> 
	(DF) (ttl 64, id 59173, len 1400)

This next packet is a repeat, because no ACK:

30.23 213.244.168.210.10000 > 62.211.168.xx.33992: . [tcp sum ok] 1:1349(1348) 
	ack 463 win 6432 <nop,nop,timestamp 2142457996 2312085> 
	(DF) (ttl 64, id 59174, len 1400)

ad nauseam. Alessandro never sees these packets! After a while, he
disconnects, which happens pretty normally. From another trace (NOTE!):

00:38:21.326397 192.168.1.3.33285 > 213.244.168.210.10000: F 420:420(0) 
	ack 1 win 45 <nop,nop,timestamp 796784 2142304361> 
	(DF)
00:38:21.410353 213.244.168.210.10000 > 192.168.1.3.33285: . 
	ack 421 win 6432 <nop,nop,timestamp 2142306461 796784> 
	(DF)
	
We've tried with wscale=0,1,2 and these all work. Things go wrong for
wscale>=3. My current feeling is that some kind of QoS device is
interfering, and that the 'wscale gets stuffed' theory is wrong in this
case.

I recall that 'Packeteer' QoS devices try to mess with windows.

Alessandro has this DSL modem, which crashed once during testing.
http://www.usr.com/support/product-template.asp?prod=9003

So we're not done debugging.

-- 
http://www.PowerDNS.com      Open source, database driven DNS Software 
http://lartc.org           Linux Advanced Routing & Traffic Control HOWTO

Re: preliminary conclusions regarding window size issues

[Jamie Lokier]

bert hubert wrote:
> Alessandro never sees these packets!
...
> My current feeling is that some kind of QoS device is interfering,
> and that the 'wscale gets stuffed' theory is wrong in this case.
> 
> I recall that 'Packeteer' QoS devices try to mess with windows.

It's a bit fiddly to arrange, but can you repeat the test and
artificially lower the TTL for these packets which disappear?

An iptable mangle rule would do the trick -- mangle the TTL only on
packets which match this point in the trace.

The idea is to reduce the TTL like traceroute does, so you can see
which hop is causing these packets to disappear -- perhaps it'll stand
out proudly as a QoS device which can be named, blamed and shamed.

-- Jamie

Re: preliminary conclusions regarding window size issues

[bert hubert]

On Thu, Jul 08, 2004 at 02:44:43AM +0100, Jamie Lokier wrote:

> An iptable mangle rule would do the trick -- mangle the TTL only on
> packets which match this point in the trace.

Indeed fiddly - not only does the packet have to disappear, we need an ICMP
to confirm that. This is because the packet currently disappears anyhow. 

Another thought that ocurred to me is that this might be a window tracking
firewall that says, based on the scaled window size which it misinterprets
because it does not understand window scaling: "I'm not going to let this
packet pass, I've seen that the intended recipient announced a 43 byte
window size".

The idea such a silly firewall would have is that it 'protects' a host from
traffic it can't handle.

This jives with the observed fact that things work up to and including
wscale=2, but breaks with wscale=3. With wscale=3, the scaled window size is
730. With wscale=2, the observed window of 1460 is big enough to let a
packet pass.

We could verify this assumption by checking if lowering the MTU to say 700
allows wscale=3 to work. 

Looking at the traceroute to Alessandro, my current suspect is this machine:

(The 1655 ports scanned but not shown below are in state: closed)
PORT    STATE    SERVICE
81/tcp  filtered hosts2-ns
135/tcp filtered msrpc
445/tcp filtered microsoft-ds
514/tcp open     shell
No exact OS matches for host (If you know what OS is running on it, see
http://www.insecure.org/cgi-bin/nmap-submit.cgi).
TCP/IP fingerprint:
SInfo(V=3.50%P=i686-pc-linux-gnu%D=7/8%Time=40ECDF49%O=514%C=1)
TSeq(Class=TR%IPID=Z%TS=U)
T1(Resp=Y%DF=Y%W=1020%ACK=S++%Flags=AS%Ops=ME)
T2(Resp=N)
T3(Resp=Y%DF=Y%W=1020%ACK=S++%Flags=AS%Ops=ME)
T4(Resp=Y%DF=N%W=0%ACK=O%Flags=R%Ops=)
T5(Resp=Y%DF=N%W=0%ACK=S++%Flags=AR%Ops=)
T6(Resp=Y%DF=N%W=0%ACK=O%Flags=R%Ops=)
T7(Resp=Y%DF=N%W=0%ACK=S%Flags=AR%Ops=)
PU(Resp=N)


TCP Sequence Prediction: Class=truly random
                         Difficulty=9999999 (Good luck!)
TCP ISN Seq. Numbers: 9D217EAD 78BBFA4A 6C815E49 191A3C0A 2A07597F 8B869DAA
IPID Sequence Generation: All zeros

Nmap run completed -- 1 IP address (1 host up) scanned in 25.593 seconds

TCP port 514 is rsh, but when I try rsh on that port it doesn't work.

-- 
http://www.PowerDNS.com      Open source, database driven DNS Software 
http://lartc.org           Linux Advanced Routing & Traffic Control HOWTO

window tracking firewall involved, was: Re: preliminary conclusions regarding window size issues

[bert hubert]

On Thu, Jul 08, 2004 at 08:03:26AM +0200, bert hubert wrote:

[ theory that a window tracking firewall drops packets for which it thinks
  the intended recipient has no room, as they are larger than the window size
  it sees, where it neglects to scale that window size ]

> We could verify this assumption by checking if lowering the MTU to say 700
> allows wscale=3 to work. 

This has now been confirmed with the packages.gentoo.org firewall!

Setting MTU does not help because the kernel adjusts the window size too.
However, fiddling with MSS helps.

Setting wscale to 3 breaks connectivity to packages.gentoo.org, but one can
restore it with:

# iptables -t mangle -A OUTPUT -p tcp -o wlan0 -d 198.63.211.232 -j TCPMSS \
	--set-mss=742 --tcp-flags SYN,RST SYN

742 is the limit value, note how this is pretty close to the scaled window
size of 730.

Trace of succesful connection:

08:29:54.799158 192.168.1.4.33116 > 198.63.211.232.80: S 258776237:258776237(0) 
	win 5840 <mss 742,sackOK,timestamp 5528023 0,nop,wscale 3> 
	(DF)
08:29:54.935830 198.63.211.232.80 > 192.168.1.4.33116: S 3729725382:3729725382(0) 
	ack 258776238 win 5792 <mss 1460,sackOK,timestamp 824466134 5528023,nop,wscale 0> 
	(DF)
08:29:54.935997 192.168.1.4.33116 > 198.63.211.232.80: 
	. ack 1 win 730 <nop,nop,timestamp 5528160 824466134> 
	(DF)
08:29:54.936116 192.168.1.4.33116 > 198.63.211.232.80: P 1:107(106) 
	ack 1 win 730 <nop,nop,timestamp 5528160 824466134> 
	(DF)
08:29:55.058090 198.63.211.232.80 > 192.168.1.4.33116: . 
	ack 107 win 5792 <nop,nop,timestamp 824466159 5528160> (DF)
08:29:55.058325 198.63.211.232.80 > 192.168.1.4.33116: . 1:731(730) 
	ack 107 win 5792 <nop,nop,timestamp 824466160 5528160> 
	(DF)
08:29:55.058425 192.168.1.4.33116 > 198.63.211.232.80: . 
	ack 731 win 912 <nop,nop,timestamp 5528283 824466160> 
	(DF)
08:29:55.181172 198.63.211.232.80 > 192.168.1.4.33116: . 731:1461(730) 
	ack 107 win 5792 <nop,nop,timestamp 824466184 5528283> 
	(DF)

Packages.gentoo.org is behind a
Running: Cisco IOS 12.X
OS details: Cisco 7200 router running IOS 12.1(14)E6
OS Fingerprint:

Not sure if that would be the culprit though.

-- 
http://www.PowerDNS.com      Open source, database driven DNS Software 
http://lartc.org           Linux Advanced Routing & Traffic Control HOWTO

Re: window tracking firewall involved, was: Re: preliminary conclusions regarding window size issues

[David S. Miller]

On Thu, 8 Jul 2004 08:37:00 +0200
bert hubert <ahu@ds9a.nl> wrote:

> On Thu, Jul 08, 2004 at 08:03:26AM +0200, bert hubert wrote:
> 
> [ theory that a window tracking firewall drops packets for which it thinks
>   the intended recipient has no room, as they are larger than the window size
>   it sees, where it neglects to scale that window size ]
> 
> > We could verify this assumption by checking if lowering the MTU to say 700
> > allows wscale=3 to work. 
> 
> This has now been confirmed with the packages.gentoo.org firewall!

It's the netfilter patches added to the gentoo WOLK kernel running
on packages.gentoo.org

Specifically, it's the tcp-window-tracking patch from netfilter's
patch-o-matic.  There's some bug in there wrt. it's window scaling
support.

I bet if the tcp-window-scaling diff is removed from the kernel running
there, the problem will totally go away.

I note that it is using a very old version of the tcp-window-tracking
patch, the current version is 2.2 and probably fixes this bug.  The
gentoo linux-2.4.20-wolk-4.14 kernel is using version 1.7

Re: window tracking firewall involved, was: Re: preliminary conclusions regarding window size issues

[Martin Josefsson]

[-- Attachment #1: Type: text/plain, Size: 1130 bytes --]

On Thu, 2004-07-08 at 17:37, David S. Miller wrote:

> > This has now been confirmed with the packages.gentoo.org firewall!
> 
> It's the netfilter patches added to the gentoo WOLK kernel running
> on packages.gentoo.org
> 
> Specifically, it's the tcp-window-tracking patch from netfilter's
> patch-o-matic.  There's some bug in there wrt. it's window scaling
> support.
> 
> I bet if the tcp-window-scaling diff is removed from the kernel running
> there, the problem will totally go away.
> 
> I note that it is using a very old version of the tcp-window-tracking
> patch, the current version is 2.2 and probably fixes this bug.  The
> gentoo linux-2.4.20-wolk-4.14 kernel is using version 1.7

That bug was probably fixed May 21 2003 according to cvs history.
"Patch updated: window scaling bug fixed, improved, etc. (JK)."
It updates the version to 1.9

As reference, I'm using v2.2 with -bk from 040626 which does use
wscale=7 and I don't see any problems connecting to/from machines with
lower or equal wscale. I drop and log all packets tcp-window-tracking
classifies as INVALID.

-- 
/Martin

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: preliminary conclusions regarding window size issues

[Redeeman]

hey bert, a little update on things.
for 2 days ago, when we chatted on irc first, i could reach lkml.org,
however, i had played abit with various settings.. i cant get that
working now, neither can i get tcpdump to give me info....

i replaced my ugly speedstream router with a linux box, and speed has
rised, as you predicted :D, it must have been the host mapping that
doing some bad things, thanks for suggesting that :D

theres a new site, which i cant reach either.. with 2.6.7, but 2.6.5
can. its my ipv6 tunnel broker, netgroup, the url is:
http://noodle.ngdc.net/~hroi/tb/
however, when i bring the ipv6 tunnel up, i can reach it, but thats
probably via ipv6.
with ipv6 i cant connect to lkml.org either..

if you help me with the tcpdump paramters i will gladly provide all info
i can..

thanks for all the help you did. it seems very close now :D

On Thu, 2004-07-08 at 01:27 +0200, bert hubert wrote:
> Two things:
> 
> 1) packages.gentoo.org is currently unreachable by 2.6.7-recent.
> This has been confirmed from several places, very easy to reproduce. Bug has
> been filed with gentoo to fix their firewall.
> 
> 2) What Alessandro Suardi sees is highly similar, except that he has it with
> *all* remotes, except for google.it and a very small number of other
> servers.
> 
> This is what Alessandro saw going out. We artificially lowered the MTU
> because of possible tunelling loss:
> 
> 01:03:36.323132 192.168.1.3.33992 > 213.244.168.210.10000: S [tcp sum ok] 3497585848:3497585848(0) 
> 	win 5440 <mss 1360,sackOK,timestamp 2311996 0,nop,wscale 7> 
> 	(DF) (ttl 64, id 43908, len 60)
> 01:03:36.396660 213.244.168.210.10000 > 192.168.1.3.33992: S [tcp sum ok] 3030562636:3030562636(0) 
> 	ack 3497585849 win 5792 <mss 1452,sackOK,timestamp  2142457957 2311996,nop,wscale 0> 
> 	(DF) (ttl 53, id 0, len 60)
> 01:03:36.396719 192.168.1.3.33992 > 213.244.168.210.10000: . [tcp sum ok]
> 	ack 1 win 42 <nop,nop,timestamp 2312084 2142457957> 
> 	(DF) (ttl 64, id 43909, len 52)
> 
> Perfect SYN, SYN|ACK, ACK.
> 
> 01:03:36.397362 192.168.1.3.33992 > 213.244.168.210.10000: P 1:463(462) ack 1 win 42 
> 	<nop,nop,timestamp 2312085 2142457957> 
> 	(DF) (ttl 64, id 43910, len 514)
> 
> The GET request.
> 
> 01:03:36.497588 213.244.168.210.10000 > 192.168.1.3.33992: . [tcp sum ok] ack 463 
> 	win 6432 <nop,nop,timestamp 2142457967 2312085> 
> 	(DF) (ttl 53, id 59171, len 52) 
> 
> And acked by my server. This trace is identical to what I see on the
> receiving end:
> 
> 29.84 62.211.168.xx.33992 > 213.244.168.210.10000: S [tcp sum ok] 3497585848:3497585848(0) 
> 	win 5440 <mss 1360,sackOK,timestamp 2311996 0,nop,wscale 7> 
> 	(DF) (ttl 50, id 43908, len 60)
> 29.84 213.244.168.210.10000 > 62.211.168.xx.33992: S [tcp sum ok] 3030562636:3030562636(0) 
> 	ack 3497585849 win 5792 <mss 1460,sackOK,timestamp 2142457957 2311996,nop,wscale 0>  
> 	(DF) (ttl 64, id 0, len 60)
> 29.93 62.211.168.xx.33992 > 213.244.168.210.10000: . [tcp sum ok] 1:1(0) 
> 	ack 1 win 42 <nop,nop,timestamp 2312084 2142457957> 
> 	(DF) (ttl 50, id 43909, len 52)
> 
> 29.95 62.211.168.xx.33992 > 213.244.168.210.10000: P [tcp sum ok] 1:463(462)
> 	ack 1 win 42 <nop,nop,timestamp 2312085 2142457957> 
> 	(DF) (ttl 50, id 43910, len 514)
> 29.95 213.244.168.210.10000 > 62.211.168.xx.33992: . [tcp sum ok] 1:1(0) 
> 	ack 463 win 6432 <nop,nop,timestamp 2142457967 2312085> 
> 	(DF) (ttl 64, id 59171, len 52)
> 
> Except for TTL and NAT, this is identical.
> 
> From here, things start to differ. I measure that I send out:
> 
> 29.95 213.244.168.210.10000 > 62.211.168.xx.33992: . [tcp sum ok] 1:1349(1348) 
> 	ack 463 win 6432 <nop,nop,timestamp 2142457967 2312085> 
> 	(DF) (ttl 64, id 59172, len 1400)
> 29.95 213.244.168.210.10000 > 62.211.168.xx.33992: P [tcp sum ok] 1349:2697(1348) 
> 	ack 463 win 6432 <nop,nop,timestamp 2142457967 2312085> 
> 	(DF) (ttl 64, id 59173, len 1400)
> 
> This next packet is a repeat, because no ACK:
> 
> 30.23 213.244.168.210.10000 > 62.211.168.xx.33992: . [tcp sum ok] 1:1349(1348) 
> 	ack 463 win 6432 <nop,nop,timestamp 2142457996 2312085> 
> 	(DF) (ttl 64, id 59174, len 1400)
> 
> ad nauseam. Alessandro never sees these packets! After a while, he
> disconnects, which happens pretty normally. From another trace (NOTE!):
> 
> 00:38:21.326397 192.168.1.3.33285 > 213.244.168.210.10000: F 420:420(0) 
> 	ack 1 win 45 <nop,nop,timestamp 796784 2142304361> 
> 	(DF)
> 00:38:21.410353 213.244.168.210.10000 > 192.168.1.3.33285: . 
> 	ack 421 win 6432 <nop,nop,timestamp 2142306461 796784> 
> 	(DF)
> 	
> We've tried with wscale=0,1,2 and these all work. Things go wrong for
> wscale>=3. My current feeling is that some kind of QoS device is
> interfering, and that the 'wscale gets stuffed' theory is wrong in this
> case.
> 
> I recall that 'Packeteer' QoS devices try to mess with windows.
> 
> Alessandro has this DSL modem, which crashed once during testing.
> http://www.usr.com/support/product-template.asp?prod=9003
> 
> So we're not done debugging.
> 
> -- 
> http://www.PowerDNS.com      Open source, database driven DNS Software 
> http://lartc.org           Linux Advanced Routing & Traffic Control HOWTO
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
> 

Re: preliminary conclusions regarding window size issues

[bert hubert]

On Thu, Jul 08, 2004 at 11:57:04PM +0200, Redeeman wrote:
> hey bert, a little update on things.
> for 2 days ago, when we chatted on irc first, i could reach lkml.org,
> however, i had played abit with various settings.. i cant get that
> working now, neither can i get tcpdump to give me info....

This has been resolved as an IPv6 routing issue - lkml.org also has an AAAA
record but Redeeman's IPv6 is not fully functional.

Regarding the packages.gentoo.org kernel, 'NQWOLK' has been coined :-)

Alessandro, shall we try the MSS clamp route to further determine what is
going on? With packages.gentoo.org the TCP behaviour confirmed that a 'smart
firewall' was to blame and not something that stamped over the wscale,
perhaps we can narrow down your problem as well.

Thanks.

-- 
http://www.PowerDNS.com      Open source, database driven DNS Software 
http://lartc.org           Linux Advanced Routing & Traffic Control HOWTO