Question on TSO maximum segment sizes.

Question on TSO maximum segment sizes.

[Waskiewicz Jr, Peter P]

I'm having an issue with TSO right vs. hardware that can't take the
maximum segment size sent from the stack.  I've been told that the
maximum packet size that can be sent to the hardware today is 64k, but
my hardware can only take 32k in certain modes per queue due to hardware
limitations.  I have two questions regarding this: 1) where is this
value set in the TCP code, and 2) Is this something that can be
configured on the fly?  If the answer to 2 is no, I will try and put
something together to allow this to happen.

Thanks in advance,
 
PJ Waskiewicz
Intel Corporation
peter.p.waskiewicz.jr@intel.com

Re: Question on TSO maximum segment sizes.

[David Miller]

From: "Waskiewicz Jr, Peter P" <peter.p.waskiewicz.jr@intel.com>
Date: Thu, 11 Oct 2007 16:27:14 -0700

> I'm having an issue with TSO right vs. hardware that can't take the
> maximum segment size sent from the stack.  I've been told that the
> maximum packet size that can be sent to the hardware today is 64k, but
> my hardware can only take 32k in certain modes per queue due to hardware
> limitations.  I have two questions regarding this: 1) where is this
> value set in the TCP code, and 2) Is this something that can be
> configured on the fly?  If the answer to 2 is no, I will try and put
> something together to allow this to happen.

The TCP code just builds the maximum possible for the underlying
protocol, be it ipv4 or ipv6.  It takes the underlying protocol
maximum packet length, subtracts the amount of header space it
knows will be used, and uses that.

You'll need to use GSO sw segmentation to split the TSO packets
which are too big for your HW to handle.

Re: Question on TSO maximum segment sizes.

[Rick Jones]

David Miller wrote:
> From: "Waskiewicz Jr, Peter P" <peter.p.waskiewicz.jr@intel.com>
> Date: Thu, 11 Oct 2007 16:27:14 -0700
> 
> 
>>I'm having an issue with TSO right vs. hardware that can't take the
>>maximum segment size sent from the stack.  I've been told that the
>>maximum packet size that can be sent to the hardware today is 64k, but
>>my hardware can only take 32k in certain modes per queue due to hardware
>>limitations. 

Bletch.

>> I have two questions regarding this: 1) where is this
>>value set in the TCP code, and 2) Is this something that can be
>>configured on the fly?  If the answer to 2 is no, I will try and put
>>something together to allow this to happen.
> 
> 
> The TCP code just builds the maximum possible for the underlying
> protocol, be it ipv4 or ipv6.  It takes the underlying protocol
> maximum packet length, subtracts the amount of header space it
> knows will be used, and uses that.
> 
> You'll need to use GSO sw segmentation to split the TSO packets
> which are too big for your HW to handle.

For just messing about, might it be possible to tweak the socket buffer sizes 
and tcp_tso_win_divisor to kludge things for a short while?  Couldn't ship that 
way certainly, but assuming Peter's going to get his broken hardware fixed it 
might let him limp along until then.

rick jones

Re: Question on TSO maximum segment sizes.

[David Miller]

From: Rick Jones <rick.jones2@hp.com>
Date: Thu, 11 Oct 2007 16:50:46 -0700

> For just messing about, might it be possible to tweak the socket buffer sizes 
> and tcp_tso_win_divisor to kludge things for a short while?  Couldn't ship that 
> way certainly, but assuming Peter's going to get his broken hardware fixed it 
> might let him limp along until then.

TCP dynamically grows the socket buffer sizes unless the application
explicitly sets them via setsockopt() and the limits imposed in those
cases are controlled by tcp_{,r,w}mem[] sysctls.  Decreasing those
will kill performance exactly for the cases this person cares about.

No, the only way to deal with this is to GSO segment incoming frames
inside of the driver when they exceed the HW limits.

RE: Question on TSO maximum segment sizes.

[Waskiewicz Jr, Peter P]

> From: Rick Jones <rick.jones2@hp.com>
> Date: Thu, 11 Oct 2007 16:50:46 -0700
> 
> > For just messing about, might it be possible to tweak the socket 
> > buffer sizes and tcp_tso_win_divisor to kludge things for a short 
> > while?  Couldn't ship that way certainly, but assuming 
> Peter's going 
> > to get his broken hardware fixed it might let him limp 
> along until then.
> 
> TCP dynamically grows the socket buffer sizes unless the 
> application explicitly sets them via setsockopt() and the 
> limits imposed in those cases are controlled by 
> tcp_{,r,w}mem[] sysctls.  Decreasing those will kill 
> performance exactly for the cases this person cares about.
> 
> No, the only way to deal with this is to GSO segment incoming 
> frames inside of the driver when they exceed the HW limits.

Thanks Dave and Rick.  I'll mess with this and make my driver happy
again.

Cheers,
-PJ Waskiewicz

Re: Question on TSO maximum segment sizes.

[Ben Greear]

Waskiewicz Jr, Peter P wrote:
>> From: Rick Jones <rick.jones2@hp.com>
>> Date: Thu, 11 Oct 2007 16:50:46 -0700
>>
>>> For just messing about, might it be possible to tweak the socket 
>>> buffer sizes and tcp_tso_win_divisor to kludge things for a short 
>>> while?  Couldn't ship that way certainly, but assuming 
>> Peter's going 
>>> to get his broken hardware fixed it might let him limp 
>> along until then.
>>
>> TCP dynamically grows the socket buffer sizes unless the 
>> application explicitly sets them via setsockopt() and the 
>> limits imposed in those cases are controlled by 
>> tcp_{,r,w}mem[] sysctls.  Decreasing those will kill 
>> performance exactly for the cases this person cares about.

I just tried turning off my explicit SO_SNDBUF/SO_RCVBUG settings in my app,
and the connection ran very poorly through a link with even a small
bit of latency (~2-4ms I believe).

It ran near gige line speed through a cross-over cable.

I have the sysctl max values set very generous, though the min
and default are fairly small.

This was with kernel 2.6.20.

Was the auto-tuning put in after 2.6.20?  If not, has this
been tested through a higher latency link?  Or, am I confused
and you are talking about some other setsockopt?

Thanks,
Ben


-- 
Ben Greear <greearb@candelatech.com>
Candela Technologies Inc  http://www.candelatech.com

Re: Question on TSO maximum segment sizes.

[David Miller]

From: Ben Greear <greearb@candelatech.com>
Date: Thu, 11 Oct 2007 17:17:48 -0700

> Was the auto-tuning put in after 2.6.20?  If not, has this
> been tested through a higher latency link?  Or, am I confused
> and you are talking about some other setsockopt?

It's been there for quite some time.

Re: Question on TSO maximum segment sizes.

[John Heffner]

Ben Greear wrote:
> I just tried turning off my explicit SO_SNDBUF/SO_RCVBUG settings in my 
> app,
> and the connection ran very poorly through a link with even a small
> bit of latency (~2-4ms I believe).

I often run at full gigabit or faster with latencies of 100+ ms.  Can 
you give a bit more detail?

   -John

Re: Question on TSO maximum segment sizes.

[Ben Greear]

John Heffner wrote:
> Ben Greear wrote:
>> I just tried turning off my explicit SO_SNDBUF/SO_RCVBUG settings in 
>> my app,
>> and the connection ran very poorly through a link with even a small
>> bit of latency (~2-4ms I believe).
> 
> I often run at full gigabit or faster with latencies of 100+ ms.  Can 
> you give a bit more detail?

I'm doing non-blocking TCP sends, write sizes of around 64k, and I'm
doing this to/from the same system with the send-to-self patch.
I am running full-duplex (930+Mbps tx + rx on both ports)
(this is about 980Mbps on the wire).

If I don't set the tx-queue % rx-queue sizes large (~4MB), it does not
perform well when running through my latency emulator.

If I do set them, then it runs just fine (barring an occassional
TCP throughput deadlock & ACK storm that is being worked on separately.)

If it works for other people, then it could be something else
weird about my setup.  I have an easy work-around (been in the
code for years and years now), so I'm not too worried either way.

Thanks,
Ben

> 
>   -John


-- 
Ben Greear <greearb@candelatech.com>
Candela Technologies Inc  http://www.candelatech.com