Re: analysis of TCP window size issues still around - several reports / SACK involved?

Re: analysis of TCP window size issues still around - several reports / SACK involved?

[ALESSANDRO.SUARDI]

Hi all,

 sorry about the silly formatting, but the webmail
 client isn't really nice in this respect :(

Further data points: -bk18 built with GCC 3.4.1
 still hangs talking to most of the world except
 google.it (redirected from google.com) from
 my home DSL (US Robotics USR9003 router).

It _works_ without changing anything when in
 my office (unsure about hardware there).

It _also_ works from my home _if_ I fire up the
 Cisco VPN client (4.0.4.B) to an Oracle VPN
 server and access the 'net (kernel.org and
 all) via our proxies through the VPN.

Hope that helps, and sorry for top-posting.

Bert Hubert wrote:

> On Thu, Jul 01, 2004 at 10:32:25PM -0300, 
> Arnaldo Carvalho de Melo wrote:
> 
> > > > Rather than using 
> *tcp_prot.memory_pressure, just go back to 
> looking at
> > > > tcp_memory_pressure.
> > > 
> > > Hehe, applied thanks Stephen.
> 
> People,
> 
> There are still persistent reports of TCP 
> problems, even after patching away
> the memory_pressure pointer problem. From one 
> trace I've seen, by Alessandro
> Suardi, but I lack the SACK knowledge to fully 
> interpret these traces:
> 
> 22:42:40.890025 192.168.1.6.32843 > 
> 204.152.189.116.http: S 1994994484:1994994484(0) 
> win 5840 <mss 1460,sackOK,timestamp 4294940315 
> 0,nop,wscale 7> (DF)
> 22:42:41.143063 204.152.189.116.http > 
> 192.168.1.6.32843: S 1404108869:1404108869(0) 
> ack 1994994485 win 5792 <mss 
> 1452,sackOK,timestamp 3383469176 
> 4294940315,nop,wscale 0> (DF)
> 22:42:41.143123 192.168.1.6.32843 > 
> 204.152.189.116.http: . ack 1 win 45 
> <nop,nop,timestamp 4294940568 3383469176> (DF)
> 
> Alessandro's machine does perform window 
> scaling, tcpdump however does not
> understand that and neglects to multiply 45 by 
> 2^7 (=5760). Kernel.org does do
> wscale, but defaults to 2^0.
> 
> 22:42:41.143362 192.168.1.6.32843 > 
> 204.152.189.116.http: P 1:421(420) ack 1 win 45 
> <nop,nop,timestamp 4294940568 3383469176> (DF)
> 
> Alessandro's machine sends a GET request.
> 
> 22:42:41.147669 204.152.189.116.http > 
> 192.168.1.6.32843: S 1404108869:1404108869(0) 
> ack 1994994485 win 5792 <mss 
> 1452,sackOK,timestamp 3383469180 
> 4294940315,nop,wscale 0> (DF)
> 
> www.kernel.org acts like it did not see our ACK.
> 
> 
> 22:42:41.147723 192.168.1.6.32843 > 
> 204.152.189.116.http: . ack 1 win 45 
> <nop,nop,timestamp 4294940572 
> 3383469180,nop,nop,sack sack 1 {0:1} > (DF)
> 
> Allessandro's machine sends a selective ACK - 
> could have gotten away with a
> regular one I'd think?
> 
> 22:42:41.408763 204.152.189.116.http > 
> 192.168.1.6.32843: . ack 421 win 6432 
> <nop,nop,timestamp 3383469440 4294940568> (DF)
> 
> www.kernel.org acks the GET, but from then on 
> does not send anything.
> 
> After a minute, Alessandro gets bored and 
> presses STOP in Mozilla:
> 
> 22:43:41.051537 192.168.1.6.32843 > 
> 204.152.189.116.http: F 421:421(0) ack 1 win 45 
> <nop,nop,timestamp 33189 3383469440> (DF)
> 
> k.o acks this FIN, but also sends a selective 
> ack:
> 
> 22:43:41.304371 204.152.189.116.http > 
> 192.168.1.6.32843: . ack 422 win 6432 
> <nop,nop,timestamp 3383529343 33189,nop,nop,sack 
> sack 1 {421:422} > (DF)
> 
> Here are the underlying reports:
> 
> http://lkml.org/lkml/2004/7/4/116 (Alessandro 
> Suardi)
> 	The _only_ site I found I can browse without 
> disabling TCP
> 	  window scaling is http://www.google.it.
> 
> 	tcpdump at: 
> http://xoomer.virgilio.it/incident/tcpdump.out 
> 
> http://lkml.org/lkml/2004/7/5/105 (Phy Prabab)
> 	Concerning my issue with ftp'ing from remote 
> sites,
> 	using these sysctl's I was able to get the 
> performance
> 	back:
> 
> 	net.ipv4.tcp_default_win_scale=0
> 	net.ipv4.tcp_moderate_rcvbuf=0
> 
> 	2.6.7-bk18 w/out sysctls:
> 	2573621 bytes received in 2e+02 seconds (12 
> Kbytes/s)
> 
> 	2.6.7-bk18 w/sysctls:
> 	2573621 bytes received in 0.69 seconds (3.6e+03
> 
> 	Kbytes/s)
> 
> These both refer to bk after the 
> *tcp_prot.memory_pressure patch was
> applied.
> 
> Thanks for your attention.
> 
> -- 
> http://www.PowerDNS.com      Open source, 
> database driven DNS Software 
> http://lartc.org           Linux Advanced 
> Routing & Traffic Control HOWTO
> 


--alessandro

Re: analysis of TCP window size issues still around - several reports / SACK involved?

[bert hubert]

On Tue, Jul 06, 2004 at 06:30:51AM -0600, ALESSANDRO.SUARDI@ORACLE.COM wrote:

> Further data points: -bk18 built with GCC 3.4.1 still hangs talking to
>  most of the world except google.it (redirected from google.com) from my
>  home DSL (US Robotics USR9003 router).

I juist built 2.6.7-mm6 in an attempt to reproduce your problems but I can't
see them, can you check with that version too?

> It _also_ works from my home _if_ I fire up the Cisco VPN client (4.0.4.B)
>  to an Oracle VPN server and access the 'net (kernel.org and all) via our
>  proxies through the VPN.

Is this the 5mb binary kernel module blob version? Can you reproduce the bug
on a system that has never touched this module?

Furthermore, can you email me your IP address and try to connect to
http://ds9a.nl/hi-ahu afterwards I confirm receipt? I want to see the server
side of the story.

Thanks.

-- 
http://www.PowerDNS.com      Open source, database driven DNS Software 
http://lartc.org           Linux Advanced Routing & Traffic Control HOWTO

[PATCH] TCP acts like it is always out of memory.

[Stephen Hemminger]

Current 2.6.7 tree acts as if it is alway under memory pressure because
a recent change did a s/tcp_memory_pressure/tcp_prot.memory_pressure/.
The problem is tcp_prot.memory_pressure is a pointer, so it is always non-zero!

Rather than using *tcp_prot.memory_pressure, just go back to looking at
tcp_memory_pressure.

Signed-off-by: Stephen Hemminger <shemminger@osdl.org>

diff -Nru a/net/ipv4/tcp_input.c b/net/ipv4/tcp_input.c
--- a/net/ipv4/tcp_input.c	2004-07-01 13:36:58 -07:00
+++ b/net/ipv4/tcp_input.c	2004-07-01 13:36:58 -07:00
@@ -259,7 +259,7 @@
 	/* Check #1 */
 	if (tp->rcv_ssthresh < tp->window_clamp &&
 	    (int)tp->rcv_ssthresh < tcp_space(sk) &&
-	    !tcp_prot.memory_pressure) {
+	    !tcp_memory_pressure) {
 		int incr;
 
 		/* Check #2. Increase window, if skb with such overhead
@@ -349,7 +349,7 @@
 	if (ofo_win) {
 		if (sk->sk_rcvbuf < sysctl_tcp_rmem[2] &&
 		    !(sk->sk_userlocks & SOCK_RCVBUF_LOCK) &&
-		    !tcp_prot.memory_pressure &&
+		    !tcp_memory_pressure &&
 		    atomic_read(&tcp_memory_allocated) < sysctl_tcp_mem[0])
 			sk->sk_rcvbuf = min(atomic_read(&sk->sk_rmem_alloc),
 					    sysctl_tcp_rmem[2]);
@@ -3764,7 +3764,7 @@
 
 	if (atomic_read(&sk->sk_rmem_alloc) >= sk->sk_rcvbuf)
 		tcp_clamp_window(sk, tp);
-	else if (tcp_prot.memory_pressure)
+	else if (tcp_memory_pressure)
 		tp->rcv_ssthresh = min(tp->rcv_ssthresh, 4U * tp->advmss);
 
 	tcp_collapse_ofo_queue(sk);
@@ -3844,7 +3844,7 @@
 
 	if (tp->packets_out < tp->snd_cwnd &&
 	    !(sk->sk_userlocks & SOCK_SNDBUF_LOCK) &&
-	    !tcp_prot.memory_pressure &&
+	    !tcp_memory_pressure &&
 	    atomic_read(&tcp_memory_allocated) < sysctl_tcp_mem[0]) {
  		int sndmem = max_t(u32, tp->mss_clamp, tp->mss_cache) +
 			MAX_TCP_HEADER + 16 + sizeof(struct sk_buff),
diff -Nru a/net/ipv4/tcp_output.c b/net/ipv4/tcp_output.c
--- a/net/ipv4/tcp_output.c	2004-07-01 13:36:58 -07:00
+++ b/net/ipv4/tcp_output.c	2004-07-01 13:36:58 -07:00
@@ -672,7 +672,7 @@
 	if (free_space < full_space/2) {
 		tp->ack.quick = 0;
 
-		if (tcp_prot.memory_pressure)
+		if (tcp_memory_pressure)
 			tp->rcv_ssthresh = min(tp->rcv_ssthresh, 4U*tp->advmss);
 
 		if (free_space < mss)
diff -Nru a/net/ipv4/tcp_timer.c b/net/ipv4/tcp_timer.c
--- a/net/ipv4/tcp_timer.c	2004-07-01 13:36:58 -07:00
+++ b/net/ipv4/tcp_timer.c	2004-07-01 13:36:58 -07:00
@@ -257,7 +257,7 @@
 	TCP_CHECK_TIMER(sk);
 
 out:
-	if (tcp_prot.memory_pressure)
+	if (tcp_memory_pressure)
 		sk_stream_mem_reclaim(sk);
 out_unlock:
 	bh_unlock_sock(sk);

Re: [PATCH] TCP acts like it is always out of memory.

[David S. Miller]

On Thu, 1 Jul 2004 13:37:38 -0700
Stephen Hemminger <shemminger@osdl.org> wrote:

> Current 2.6.7 tree acts as if it is alway under memory pressure because
> a recent change did a s/tcp_memory_pressure/tcp_prot.memory_pressure/.
> The problem is tcp_prot.memory_pressure is a pointer, so it is always non-zero!
> 
> Rather than using *tcp_prot.memory_pressure, just go back to looking at
> tcp_memory_pressure.

Hehe, applied thanks Stephen.

Re: [PATCH] TCP acts like it is always out of memory.

[Arnaldo Carvalho de Melo]

Em Thu, Jul 01, 2004 at 02:04:06PM -0700, David S. Miller escreveu:
> On Thu, 1 Jul 2004 13:37:38 -0700
> Stephen Hemminger <shemminger@osdl.org> wrote:
> 
> > Current 2.6.7 tree acts as if it is alway under memory pressure because
> > a recent change did a s/tcp_memory_pressure/tcp_prot.memory_pressure/.
> > The problem is tcp_prot.memory_pressure is a pointer, so it is always non-zero!
> > 
> > Rather than using *tcp_prot.memory_pressure, just go back to looking at
> > tcp_memory_pressure.
> 
> Hehe, applied thanks Stephen.

:-) Thanks Stephen for the fix, this was a leftover of the conversion of
the memory pressure members in struct proto to pointers, to cover the case
pointed out by David related to the ipv6_mapped functionality in the
1.1722.122.23 changeset, (i.e.  tcp_prot and tcpv6_prot having to share the
same accounting variables), I forgot to convert all places where the
tcp_prot.memory_pressure memory is used, the fix is exactly what I should
have done. Due to family health problems I was unable to promply fix this
thinko, so, again, thank you very much.

Best Regards,

- Arnaldo

analysis of TCP window size issues still around - several reports / SACK involved?

[bert hubert]

On Thu, Jul 01, 2004 at 10:32:25PM -0300, Arnaldo Carvalho de Melo wrote:

> > > Rather than using *tcp_prot.memory_pressure, just go back to looking at
> > > tcp_memory_pressure.
> > 
> > Hehe, applied thanks Stephen.

People,

There are still persistent reports of TCP problems, even after patching away
the memory_pressure pointer problem. From one trace I've seen, by Alessandro
Suardi, but I lack the SACK knowledge to fully interpret these traces:

22:42:40.890025 192.168.1.6.32843 > 204.152.189.116.http: S 1994994484:1994994484(0) win 5840 <mss 1460,sackOK,timestamp 4294940315 0,nop,wscale 7> (DF)
22:42:41.143063 204.152.189.116.http > 192.168.1.6.32843: S 1404108869:1404108869(0) ack 1994994485 win 5792 <mss 1452,sackOK,timestamp 3383469176 4294940315,nop,wscale 0> (DF)
22:42:41.143123 192.168.1.6.32843 > 204.152.189.116.http: . ack 1 win 45 <nop,nop,timestamp 4294940568 3383469176> (DF)

Alessandro's machine does perform window scaling, tcpdump however does not
understand that and neglects to multiply 45 by 2^7 (=5760). Kernel.org does do
wscale, but defaults to 2^0.

22:42:41.143362 192.168.1.6.32843 > 204.152.189.116.http: P 1:421(420) ack 1 win 45 <nop,nop,timestamp 4294940568 3383469176> (DF)

Alessandro's machine sends a GET request.

22:42:41.147669 204.152.189.116.http > 192.168.1.6.32843: S 1404108869:1404108869(0) ack 1994994485 win 5792 <mss 1452,sackOK,timestamp 3383469180 4294940315,nop,wscale 0> (DF)

www.kernel.org acts like it did not see our ACK.

22:42:41.147723 192.168.1.6.32843 > 204.152.189.116.http: . ack 1 win 45 <nop,nop,timestamp 4294940572 3383469180,nop,nop,sack sack 1 {0:1} > (DF)

Allessandro's machine sends a selective ACK - could have gotten away with a
regular one I'd think?

22:42:41.408763 204.152.189.116.http > 192.168.1.6.32843: . ack 421 win 6432 <nop,nop,timestamp 3383469440 4294940568> (DF)

www.kernel.org acks the GET, but from then on does not send anything.

After a minute, Alessandro gets bored and presses STOP in Mozilla:

22:43:41.051537 192.168.1.6.32843 > 204.152.189.116.http: F 421:421(0) ack 1 win 45 <nop,nop,timestamp 33189 3383469440> (DF)

k.o acks this FIN, but also sends a selective ack:

22:43:41.304371 204.152.189.116.http > 192.168.1.6.32843: . ack 422 win 6432 <nop,nop,timestamp 3383529343 33189,nop,nop,sack sack 1 {421:422} > (DF)

Here are the underlying reports:

http://lkml.org/lkml/2004/7/4/116 (Alessandro Suardi)
	The _only_ site I found I can browse without disabling TCP
	  window scaling is http://www.google.it.

	tcpdump at: http://xoomer.virgilio.it/incident/tcpdump.out 

http://lkml.org/lkml/2004/7/5/105 (Phy Prabab)
	Concerning my issue with ftp'ing from remote sites,
	using these sysctl's I was able to get the performance
	back:

	net.ipv4.tcp_default_win_scale=0
	net.ipv4.tcp_moderate_rcvbuf=0

	2.6.7-bk18 w/out sysctls:
	2573621 bytes received in 2e+02 seconds (12 Kbytes/s)

	2.6.7-bk18 w/sysctls:
	2573621 bytes received in 0.69 seconds (3.6e+03
	Kbytes/s)

These both refer to bk after the *tcp_prot.memory_pressure patch was
applied.

Thanks for your attention.

-- 
http://www.PowerDNS.com      Open source, database driven DNS Software 
http://lartc.org           Linux Advanced Routing & Traffic Control HOWTO

Re: analysis of TCP window size issues still around - several reports / SACK involved?

[David S. Miller]

On Tue, 6 Jul 2004 11:35:03 +0200
bert hubert <ahu@ds9a.nl> wrote:

> 22:42:40.890025 192.168.1.6.32843 > 204.152.189.116.http: S 1994994484:1994994484(0) win 5840 <mss 1460,sackOK,timestamp 4294940315 0,nop,wscale 7> (DF)
> 22:42:41.143063 204.152.189.116.http > 192.168.1.6.32843: S 1404108869:1404108869(0) ack 1994994485 win 5792 <mss 1452,sackOK,timestamp 3383469176 4294940315,nop,wscale 0> (DF)
> 22:42:41.143123 192.168.1.6.32843 > 204.152.189.116.http: . ack 1 win 45 <nop,nop,timestamp 4294940568 3383469176> (DF)
> 
> Alessandro's machine does perform window scaling, tcpdump however does not
> understand that and neglects to multiply 45 by 2^7 (=5760). Kernel.org does do
> wscale, but defaults to 2^0.

tcpdump's behavior is correct, it's just reporting the raw window
field in the TCP header, unscaled, and that is fine.  In fact I'd
rather it do this, so that diagnosing dumps are easier.  If tcpdump
tries to be too clever, scaling the window, then I might end up
chasing down a tcpdump bug rather than a TCP one :-)

What would be more interesting is to get the tcpdump trace from the
other side of this connection.  This is crucial, as it will show how
and in what way exactly the window scale options and/or window fields
are being edited by a firewall or other device and thus causing
the problems.

Re: analysis of TCP window size issues still around - several reports / SACK involved?

[bert hubert]

On Tue, Jul 06, 2004 at 01:19:55PM -0700, David S. Miller wrote:

> rather it do this, so that diagnosing dumps are easier.  If tcpdump
> tries to be too clever, scaling the window, then I might end up
> chasing down a tcpdump bug rather than a TCP one :-)

True - it might want to print '43 (*128=5706)' or something like that.

> What would be more interesting is to get the tcpdump trace from the
> other side of this connection.  This is crucial, as it will show how
> and in what way exactly the window scale options and/or window fields
> are being edited by a firewall or other device and thus causing
> the problems.

I have an appointment with Alessandro tomorrow evening at 11PM CEST to do
just that.

It sounds like window scaling may become yet another ECN...

-- 
http://www.PowerDNS.com      Open source, database driven DNS Software 
http://lartc.org           Linux Advanced Routing & Traffic Control HOWTO

Re: analysis of TCP window size issues still around - several reports / SACK involved?

[David S. Miller]

On Tue, 6 Jul 2004 22:27:08 +0200
bert hubert <ahu@ds9a.nl> wrote:

> On Tue, Jul 06, 2004 at 01:19:55PM -0700, David S. Miller wrote:
> 
> > What would be more interesting is to get the tcpdump trace from the
> > other side of this connection.  This is crucial, as it will show how
> > and in what way exactly the window scale options and/or window fields
> > are being edited by a firewall or other device and thus causing
> > the problems.
> 
> I have an appointment with Alessandro tomorrow evening at 11PM CEST to do
> just that.

That's great, thanks a lot.

Re: analysis of TCP window size issues still around - several reports / SACK involved?

[Alessandro Suardi]

bert hubert wrote:
> On Tue, Jul 06, 2004 at 01:19:55PM -0700, David S. Miller wrote:
> 
> 
>>rather it do this, so that diagnosing dumps are easier.  If tcpdump
>>tries to be too clever, scaling the window, then I might end up
>>chasing down a tcpdump bug rather than a TCP one :-)
> 
> 
> True - it might want to print '43 (*128=5706)' or something like that.
> 
> 
>>What would be more interesting is to get the tcpdump trace from the
>>other side of this connection.  This is crucial, as it will show how
>>and in what way exactly the window scale options and/or window fields
>>are being edited by a firewall or other device and thus causing
>>the problems.
> 
> 
> I have an appointment with Alessandro tomorrow evening at 11PM CEST to do
> just that.

Sorry about being slightly late - I read the thread from my VPN
  link (which does work), now I'm turning it off and will email
  Bert with my actual IP address and my connection.

Thanks,

--alessandro

  "Practice is more important than theory. A _lot_ more important."
     (Linus Torvalds on lkml, 1 June 2004)