TCP-Protection is really a pain...

TCP-Protection is really a pain...

[Christian Schmid]

Hi.

Your new dynamically adjusted socket-buffer in 2.6.10 is really a pain for big servers. PLEASE tell 
me a way how to disable it.

Thank you.

Best regards,
Christian Schmid

Re: TCP-Protection is really a pain...

[Stephen Hemminger]

On Thu, 03 Feb 2005 05:07:30 +0100
Christian Schmid <webmaster@rapidforum.com> wrote:

> Hi.
> 
> Your new dynamically adjusted socket-buffer in 2.6.10 is really a pain for big servers. PLEASE tell 
> me a way how to disable it.

sysctl -w net.ipv4.tcp_wmem="4096 8192 16384"

Your single stream will be slower, but the memory footprint will be smaller.

Re: TCP-Protection is really a pain...

[Christian Schmid]

Actually, thats my problem. Single streams are too slow! Before I had buffers up to 500 KB. This was 
very nice to CPU because I only needed to "push" more data once in 5 seconds. I am doing this every 
second now... *sigh* well maybe you might just want to add a /proc file in order to configure this 
behaviour.

btw: Another problem I am experiencing is that downloads suddenly break in speed from 360 kb/sec to 
8-12 kb/sec. 5 seconds later they stall completely. But the interesting part is, that the send-queue 
is completely full (checked with a grep in netstat). This looks like as if the receiver is just too 
slow. But this is not the case. That makes it rather funny. The receiver is waiting with an empty 
pipe but linux doesn't send. What could this be?


Stephen Hemminger wrote:
> On Thu, 03 Feb 2005 05:07:30 +0100
> Christian Schmid <webmaster@rapidforum.com> wrote:
> 
> 
>>Hi.
>>
>>Your new dynamically adjusted socket-buffer in 2.6.10 is really a pain for big servers. PLEASE tell 
>>me a way how to disable it.
> 
> 
> sysctl -w net.ipv4.tcp_wmem="4096 8192 16384"
> 
> Your single stream will be slower, but the memory footprint will be smaller.
> 
> 

Re: TCP-Protection is really a pain...

[Jeff Garzik]

Christian Schmid wrote:
> btw: Another problem I am experiencing is that downloads suddenly break 
> in speed from 360 kb/sec to 8-12 kb/sec. 5 seconds later they stall 
> completely. But the interesting part is, that the send-queue is 
> completely full (checked with a grep in netstat). This looks like as if 
> the receiver is just too slow. But this is not the case. That makes it 
> rather funny. The receiver is waiting with an empty pipe but linux 
> doesn't send. What could this be?


With recent kernels, I've noticed that my browser progress bar will 
download about 98% of a page, then stall for several seconds, then complete.

Very strange.  Haven't investigated it at all, though.

	Jeff

Re: TCP-Protection is really a pain...

[Christian Schmid]

Several users complain about downloads stalling at 99% as well... I was unable to solve this issue.

Jeff Garzik wrote:
> Christian Schmid wrote:
> 
>> btw: Another problem I am experiencing is that downloads suddenly 
>> break in speed from 360 kb/sec to 8-12 kb/sec. 5 seconds later they 
>> stall completely. But the interesting part is, that the send-queue is 
>> completely full (checked with a grep in netstat). This looks like as 
>> if the receiver is just too slow. But this is not the case. That makes 
>> it rather funny. The receiver is waiting with an empty pipe but linux 
>> doesn't send. What could this be?
> 
> 
> 
> With recent kernels, I've noticed that my browser progress bar will 
> download about 98% of a page, then stall for several seconds, then 
> complete.
> 
> Very strange.  Haven't investigated it at all, though.
> 
>     Jeff
> 
> 
> 
> 

Re: TCP-Protection is really a pain...

[Stephen Hemminger]

On Thu, 03 Feb 2005 06:21:46 +0100
Christian Schmid <webmaster@rapidforum.com> wrote:

> Actually, thats my problem. Single streams are too slow! Before I had buffers up to 500 KB. This was 
> very nice to CPU because I only needed to "push" more data once in 5 seconds. I am doing this every 
> second now... *sigh* well maybe you might just want to add a /proc file in order to configure this 
> behaviour.
> 
> btw: Another problem I am experiencing is that downloads suddenly break in speed from 360 kb/sec to 
> 8-12 kb/sec. 5 seconds later they stall completely. But the interesting part is, that the send-queue 
> is completely full (checked with a grep in netstat). This looks like as if the receiver is just too 
> slow. But this is not the case. That makes it rather funny. The receiver is waiting with an empty 
> pipe but linux doesn't send. What could this be?
>

Are you using a board that support TCP Segmentation Offload.  The problem may well be that
before we were not doing congestion control properly with TSO.  A pre-2.6.8 host with TSO was violating
all sorts of RFC's and unfairly monopolizing bandwidth.

-- 
Stephen Hemminger	<shemminger@osdl.org>

Re: TCP-Protection is really a pain...

[Christian Schmid]

Hm how can I check that? and what do you mean with "board"? Mainboard? NIC? its an onboard-nic on 
this mainboard: http://www.supermicro.com/products/motherboard/Xeon/E7501/X5DP8-G2.cfm

The worst thing is the stalled downloads. It drops to 8-12 kb-sec. sometimes after a few seconds it 
goes up to full speed again, but sometimes it suddenly stops completely. After a few seconds it 
continues with full speed or it stalls forever. send-queue is always full.....

Thank you for your help in this matter.

Chris


Stephen Hemminger wrote:
> On Thu, 03 Feb 2005 06:21:46 +0100
> Christian Schmid <webmaster@rapidforum.com> wrote:
> 
> 
>>Actually, thats my problem. Single streams are too slow! Before I had buffers up to 500 KB. This was 
>>very nice to CPU because I only needed to "push" more data once in 5 seconds. I am doing this every 
>>second now... *sigh* well maybe you might just want to add a /proc file in order to configure this 
>>behaviour.
>>
>>btw: Another problem I am experiencing is that downloads suddenly break in speed from 360 kb/sec to 
>>8-12 kb/sec. 5 seconds later they stall completely. But the interesting part is, that the send-queue 
>>is completely full (checked with a grep in netstat). This looks like as if the receiver is just too 
>>slow. But this is not the case. That makes it rather funny. The receiver is waiting with an empty 
>>pipe but linux doesn't send. What could this be?
>>
> 
> 
> Are you using a board that support TCP Segmentation Offload.  The problem may well be that
> before we were not doing congestion control properly with TSO.  A pre-2.6.8 host with TSO was violating
> all sorts of RFC's and unfairly monopolizing bandwidth.
> 

Re: TCP-Protection is really a pain...

[Stephen Hemminger]

On Thu, 03 Feb 2005 20:58:48 +0100
Christian Schmid <webmaster@rapidforum.com> wrote:

E1000 support TSO. To check if it is enabled do:
	ethtool -k eth0
To turn it off use.
	ethtool -K eth0 tso off

You get the idea..

-- 
Stephen Hemminger	<shemminger@osdl.org>

Re: TCP-Protection is really a pain...

[Christian Schmid]

Offload parameters for eth0:
rx-checksumming: on
tx-checksumming: on
scatter-gather: on
tcp segmentation offload: on

What exactly is tcp segmentation offload? Where can I read more about it? Should I disable it or is 
this not a good idea?

Stephen Hemminger wrote:
> On Thu, 03 Feb 2005 20:58:48 +0100
> Christian Schmid <webmaster@rapidforum.com> wrote:
> 
> E1000 support TSO. To check if it is enabled do:
> 	ethtool -k eth0
> To turn it off use.
> 	ethtool -K eth0 tso off
> 
> You get the idea..
> 

Re: TCP-Protection is really a pain...

[Rick Jones]

Christian Schmid wrote:
> Offload parameters for eth0:
> rx-checksumming: on
> tx-checksumming: on
> scatter-gather: on
> tcp segmentation offload: on
> 
> What exactly is tcp segmentation offload? 

TCP Segmentation Offload, aka TSO or in other contexts "large send" can be 
thought of as a "poor man's" Jumbo Frame.  The NIC advertises to the host that 
it can resegment larger TCP segments into sizes apropriate for the network.

This allows the host CPU stack to make fewer trips down the protocol stack to 
transfer a given quantity of data, thus reducing the service demand (quantity of 
CPU consumed per quantity of work).  The reduction in service demand can result 
in an increase in throughput if the transfer was previously CPU-bound.

It has the advantage over JumboFrame in that it does not require support in the 
rest of the infrastructure (switches, routers, receivers etc). I call it "poor 
man's" JumboFrame because it does not reduce the overheads on the receiver - the 
receiver still sees just as many "normally" sized segments as before, and sends 
just as many ACKs per KB transferred as before.

If a NIC is implemented with a microprocessor (bletch - personal opinion) the 
additional demands of TSO resegmenation may actually reduce throughput even as 
it greatly reduces server CPU utilization.  That was particularly evident in old 
Tigon2 NICs.  I am not sure if that is noticable in current generation BMC570X's 
or not - most of the systems I have at my disposal have BCM5701's in them and 
I'm not sure the drivers allow TSO to be enabled on that rev?  I've not seen a 
drop in maximum throughput on the "e1000" NICs I've played with, which have been 
some dual-port cards HP sells. YMMV.

 > Where can I read more about it?

In addition to whatever there may be in Linux documentation...
TSO or large send is also a long-standing feature of TCP  in AIX and (dare I say 
it here?-) Windows.  As such IBM and Microsoft may have writeups - I'm pretty 
sure that Microsoft had a write-up about it on their website - talking about the 
NDIS interface at least.  It is also a fairly recent addition to HP-UX, but I am 
not sure if there are any writeups about it on HP's websites yet.

>Should I disable it or is this not a good idea?

That depends.  If you are concerned about proper slow-start behaviour at the 
beginning of a connection you should disable it until you are on a later rev.

I've always wondered if it wouldn't be an interesting experiment with a "real" 
server to see if dishonouring slow-start at connection establishment (and there 
only, keep it after RTO) made all that much a difference in the host's 
retransmission rates...

rick jones

Re: TCP-Protection is really a pain...

[Jeff Garzik]

Christian Schmid wrote:
> What exactly is tcp segmentation offload? Where can I read more about 
> it? Should I disable it or is this not a good idea?

It's an optimization designed to reduce CPU usage for zero-copy apps 
(those that use sendfile(2)).

It would be a useful datapoint to disable TSO, and see if that improves 
things for you.

	Jeff

Re: TCP-Protection is really a pain...

[Christian Schmid]

I turned it off and without it, I was unable to reproduce any stalls anymore.

Jeff Garzik wrote:
> Christian Schmid wrote:
> 
>> What exactly is tcp segmentation offload? Where can I read more about 
>> it? Should I disable it or is this not a good idea?
> 
> 
> It's an optimization designed to reduce CPU usage for zero-copy apps 
> (those that use sendfile(2)).
> 
> It would be a useful datapoint to disable TSO, and see if that improves 
> things for you.
> 
>     Jeff
> 
> 
> 
>