tcp_window_scaling degrades performance

tcp_window_scaling degrades performance

[Maciej Soltysiak]

Hi

I have been experiencied weird problems with network throughput
lately and I after experimenting with /proc/sys/net/ipv4 knobs
I found that when I have tcp_window_scaling 0 I can
get throughput from a distant server of about 600kB/s (well, 200kB/s
is fast enough)

   # links ftp://ftp.task.gda.pl/ls-lR

When I turn tcp_window_scaling to 1 the transfer starts with 600kB,
then drops nicely to 20kB/s then starts growing again and reaches
about 400kB/s

Anyway It works, but there are some hosts, with which I can't talk to
faster than 800B/s !

This is a Debian SID with vanilla-2.6.7-bk20, but I am having these
problems for a few weeks now.

It might be a TCP kernel issue, but I know that between me and the
other hosts is a checkpoint fw-1 fp3, it may be the cause of problems,
because, well, checkpoint is crazy.

Anyway, I can provide tcpdump, and tcptrace analysis of it and the
state of sysctls if anyone would be willing to help me figure out
what's going on.

Regards,
Maciej

Re: tcp_window_scaling degrades performance

[Kasper Sandberg]

hey, are you having some kind of hardware router?
bert hubert, helped me, and we found out my router was causing me the
problems, however, my problems were more severe, as my limit was 50kb/s

On Thu, 2004-07-15 at 17:09 +0200, Maciej Soltysiak wrote:
> Hi
> 
> I have been experiencied weird problems with network throughput
> lately and I after experimenting with /proc/sys/net/ipv4 knobs
> I found that when I have tcp_window_scaling 0 I can
> get throughput from a distant server of about 600kB/s (well, 200kB/s
> is fast enough)
> 
>    # links ftp://ftp.task.gda.pl/ls-lR
> 
> When I turn tcp_window_scaling to 1 the transfer starts with 600kB,
> then drops nicely to 20kB/s then starts growing again and reaches
> about 400kB/s
> 
> Anyway It works, but there are some hosts, with which I can't talk to
> faster than 800B/s !
> 
> This is a Debian SID with vanilla-2.6.7-bk20, but I am having these
> problems for a few weeks now.
> 
> It might be a TCP kernel issue, but I know that between me and the
> other hosts is a checkpoint fw-1 fp3, it may be the cause of problems,
> because, well, checkpoint is crazy.
> 
> Anyway, I can provide tcpdump, and tcptrace analysis of it and the
> state of sysctls if anyone would be willing to help me figure out
> what's going on.
> 
> Regards,
> Maciej
> 
> 
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
> 

Re: tcp_window_scaling degrades performance

[Andi Kleen]

Maciej Soltysiak <solt@dns.toxicfilms.tv> writes:


> I have been experiencied weird problems with network throughput
> lately and I after experimenting with /proc/sys/net/ipv4 knobs
> I found that when I have tcp_window_scaling 0 I can
> get throughput from a distant server of about 600kB/s (well, 200kB/s
> is fast enough)

It's pretty easy for you to find out. Do a tcpdump -v or ethereal -v 
from both the side of a host you download from and from the linux side.
Then compare all packets. If they don't match the firewall is 
doing something bad. Especially check window values and TCP options
in the SYN packets

It is very very likely the firewall, window scaling works for a lot
of people.

-Andi

Re[2]: tcp_window_scaling degrades performance

[Maciej Soltysiak]

AK> It's pretty easy for you to find out. Do a tcpdump -v or ethereal -v
AK> from both the side of a host you download from and from the linux side.
AK> Then compare all packets. If they don't match the firewall is 
AK> doing something bad. Especially check window values and TCP options
AK> in the SYN packets
I will do that for sure, but preliminary investigation shows that
this behaviour does not show with 2.6.7 and earlier, but appears for sure in
2.6.7-bk13 (Haven't tried earlier bk snapshots)

AK> It is very very likely the firewall, window scaling works for a lot
AK> of people.
It is probable, but here: only 2.6.7+ machines behave like this.
I also noticed, that turning tcp_window_scaling off does not always
fix the problem, turning tcp_bic to 0 too helps even more.

Regards,
Maciej

Re[3]: tcp_window_scaling degrades performance

[Maciej Soltysiak]

AK>> It's pretty easy for you to find out. Do a tcpdump -v or ethereal -v
AK>> from both the side of a host you download from and from the linux side.
AK>> Then compare all packets. If they don't match the firewall is 
AK>> doing something bad. Especially check window values and TCP options
AK>> in the SYN packets
MS> I will do that for sure, but preliminary investigation shows that
MS> this behaviour does not show with 2.6.7 and earlier, but appears for sure in
MS> 2.6.7-bk13 (Haven't tried earlier bk snapshots)
I seem to have isolated the changeset that causes my machines to have
very slow throughput, even as low as 2kB/s when tcp_window_scaling is
enabled.

http://linux.bkbits.net:8080/linux-2.6/cset@1.1757.1.25?nav=index.html|ChangeSet@-3w
[TCP]: TCP acts like it is always out of memory.

Signed-off-by: Stephen Hemminger <shemminger@osdl.org>
Signed-off-by: David S. Miller <davem@redhat.com>

Stephen, David, could you review this changeset, figure out possible
causes or suggest more methods for me to analyse this problem ?

Note that I am not saying that window_scaling is badly implemented,
but it just seems related.

Regards,
Maciej

[partially solved] tcp_window_scaling degrades performance

[Maciej Soltysiak]

> AK> It is very very likely the firewall, window scaling works for a lot
> AK> of people.
> It is probable, but here: only 2.6.7+ machines behave like this.
> I also noticed, that turning tcp_window_scaling off does not always
> fix the problem, turning tcp_bic to 0 too helps even more.
It appears that checkpoint fw-1 that is here spoils everything so that
current linux boxes need to disable tcp_windows_scaling to get
reasonable throughput. When I switched a server to a different link
that bypasses checkpoint it worked well.

Although it most propably is checkpoint's fault (This one is based on
linux 2.4.9-39cpsmp) I belive that the change in the kernel that started
producing these problems is this one:
http://linux.bkbits.net:8080/linux-2.6/cset@1.1784.10.7?nav=index.html|ChangeSet@-3w

When I removed this patch from a 2.6.8-rc1 kernel it started to work
good again. But of course it may be a blind shot.

I am planning on investigating this issue from the checkpoint's
perspective.

Anyway if there is anyone willing to investigate the tcpdumps and
tcptraces of the slow/fast throughput, here it is:
http://soltysiak.com/tcp.php

I am no guru, but it shows:
1) slower throughput
2) twice as much packets exchanged
3) different advertised windows

Best regards,
Maciek

Re: Re[3]: tcp_window_scaling degrades performance

[David S. Miller]

On Fri, 16 Jul 2004 16:01:57 +0200
Maciej Soltysiak <solt@dns.toxicfilms.tv> wrote:

> I seem to have isolated the changeset that causes my machines to have
> very slow throughput, even as low as 2kB/s when tcp_window_scaling is
> enabled.

The change is valid, firewalls are just corrupting the connection
when window scaling is in use which is a valid RFC standard
TCP feature.

Please, get the firewall fixed.