Assertions when lowering tcp_tso_win_divisor

Assertions when lowering tcp_tso_win_divisor

[Sridhar Samudrala]

We are seeing following assertions when running specweb99 with
tcp_tso_win_divisor lowered to 4, 2 or 1 on linux 2.6.9 based kernels.

KERNEL: assertion (!sk->sk_forward_alloc) failed at net/core/stream.c (279)
KERNEL: assertion (!sk->sk_forward_alloc) failed at net/ipv4/af_inet.c (152)

looks like an issue with the accounting of sk_forward_alloc with non-default
values of tcp_tso_win_divisor.

Thanks
Sridhar

Re: Assertions when lowering tcp_tso_win_divisor

[David S. Miller]

On Tue, 16 Nov 2004 09:47:14 -0800 (PST)
Sridhar Samudrala <sri@us.ibm.com> wrote:

> We are seeing following assertions when running specweb99 with
> tcp_tso_win_divisor lowered to 4, 2 or 1 on linux 2.6.9 based kernels.
> 
> KERNEL: assertion (!sk->sk_forward_alloc) failed at net/core/stream.c (279)
> KERNEL: assertion (!sk->sk_forward_alloc) failed at net/ipv4/af_inet.c (152)
> 
> looks like an issue with the accounting of sk_forward_alloc with non-default
> values of tcp_tso_win_divisor.

The issue is if you change the value while there are still existing
sockets which used a previous value at least once.

Looks like we'll have to cache this inside of the per-socket structure
to fix this bug, but realize that once I do that the new sysctl value
will not apply to any existing TCP sockets and this will effect your
testing.  So for example, if the web server was started with one particular
value of the sysctl, no matter how many times you change the sysctl after-
wards, the old original sysctl value will be used for every web server
connection made to the machine.

Re: Assertions when lowering tcp_tso_win_divisor

[Herbert Xu]

David S. Miller <davem@davemloft.net> wrote:
>
>> KERNEL: assertion (!sk->sk_forward_alloc) failed at net/core/stream.c (279)
>> KERNEL: assertion (!sk->sk_forward_alloc) failed at net/ipv4/af_inet.c (152)
>> 
>> looks like an issue with the accounting of sk_forward_alloc with non-default
>> values of tcp_tso_win_divisor.
> 
> The issue is if you change the value while there are still existing
> sockets which used a previous value at least once.

Could you elaborate the code path that you've spotted that causes this
when the divisor is changed?

Perhaps we're not caching the MSS value everywhere where we should be?

Cheers,
-- 
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

Re: Assertions when lowering tcp_tso_win_divisor

[David S. Miller]

On Wed, 17 Nov 2004 07:39:42 +1100
Herbert Xu <herbert@gondor.apana.org.au> wrote:

> Could you elaborate the code path that you've spotted that causes this
> when the divisor is changed?
> 
> Perhaps we're not caching the MSS value everywhere where we should be?

Currently I think your theory is better than mine, that
2.6.9 TCP is buggy wrt. TSO. :-)

Re: Assertions when lowering tcp_tso_win_divisor

[Herbert Xu]

Sridhar Samudrala <sri@us.ibm.com> wrote:
> We are seeing following assertions when running specweb99 with
> tcp_tso_win_divisor lowered to 4, 2 or 1 on linux 2.6.9 based kernels.

2.6.9 is known to have bugs in TCP book-keeping.  Please try 2.6.10-rc2
instead.

Cheers,
-- 
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

Re: Assertions when lowering tcp_tso_win_divisor

[David S. Miller]

On Wed, 17 Nov 2004 07:59:18 +1100
Herbert Xu <herbert@gondor.apana.org.au> wrote:

> Sridhar Samudrala <sri@us.ibm.com> wrote:
> > We are seeing following assertions when running specweb99 with
> > tcp_tso_win_divisor lowered to 4, 2 or 1 on linux 2.6.9 based kernels.
> 
> 2.6.9 is known to have bugs in TCP book-keeping.  Please try 2.6.10-rc2
> instead.

Actually, that's true.  I missed that this report was against
2.6.9 vanilla, TCP is buggy as can be wrt TSO in that
release.

Re: Assertions when lowering tcp_tso_win_divisor

[Sridhar Samudrala]

On Tue, 16 Nov 2004, David S. Miller wrote:

> On Wed, 17 Nov 2004 07:59:18 +1100
> Herbert Xu <herbert@gondor.apana.org.au> wrote:
>
> > Sridhar Samudrala <sri@us.ibm.com> wrote:
> > > We are seeing following assertions when running specweb99 with
> > > tcp_tso_win_divisor lowered to 4, 2 or 1 on linux 2.6.9 based kernels.
> >
> > 2.6.9 is known to have bugs in TCP book-keeping.  Please try 2.6.10-rc2
> > instead.
>
> Actually, that's true.  I missed that this report was against
> 2.6.9 vanilla, TCP is buggy as can be wrt TSO in that
> release.

Unfortunately as we cannot upgrade to 2.6.10-rc2, i tried to locate all the
TSO accounting related bugfixes that went in after 2.6.9 and came up with
the following 4 patches.

[TCP]: Fix new packet len calc in tcp_fragment()
http://linux.bkbits.net:8080/linux-2.5/cset@4175f423T1X-UctOUzkYgMxq16ODAw?nav=index.html|src/|src/net|src/net/ipv4|related/net/ipv4/tcp_output.c

[TCP]: Fix tcp_trim_head() calculations.
http://linux.bkbits.net:8080/linux-2.5/cset@41789ca8a4gfwR352SuIgAl6GbQmbQ?nav=index.html|src/|src/net|src/net/ipv4|related/net/ipv4/tcp_output.c

[TCP]: Only re-set TSO size for packet which was TSO to begin with.
http://linux.bkbits.net:8080/linux-2.5/cset@417982139wJw8yO-JClu818UQuoa_g?nav=index.html|src/|src/net|src/net/ipv4|related/net/ipv4/tcp_output.c

[TCP]: Handle real partial-ACKs of TSO frames correctly.
http://linux.bkbits.net:8080/linux-2.5/cset@417dcb39pNYZ1fRBQTQES0JVcUji2g?nav=index.html|src/|src/net|src/net/ipv4|related/net/ipv4/tcp_output.c

Even with these 4 patches applied, we are still seeing the assertions.

Also, the webserver is started only after lowering tcp_tso_win_divisor and
the value is is not changed again.

Are there any other TSO related patches apart from the above 4 that went in
after 2.6.9? Or could it be that this bug is still there?

Thanks
Sridhar

Re: Assertions when lowering tcp_tso_win_divisor

[David S. Miller]

On Fri, 19 Nov 2004 18:00:21 -0800 (PST)
Sridhar Samudrala <sri@us.ibm.com> wrote:

> Unfortunately as we cannot upgrade to 2.6.10-rc2

This doesn't work.  You cannot expect upstream maintainers
to work with you if you're not willing to or able to
try things out with the latest kernels.

I have an inkling as to the reason why you're restricted
in this way, which is that you're limited to some vendor's
kernel.  Am I right?

In that case, work with the vendor unless you can find some
way to remove the lack of flexibility you have in using newer
kernels.

Re: Assertions when lowering tcp_tso_win_divisor

[Sridhar Samudrala]

On Fri, 19 Nov 2004, David S. Miller wrote:

> On Fri, 19 Nov 2004 18:00:21 -0800 (PST)
> Sridhar Samudrala <sri@us.ibm.com> wrote:
>
> > Unfortunately as we cannot upgrade to 2.6.10-rc2
>
> This doesn't work.  You cannot expect upstream maintainers
> to work with you if you're not willing to or able to
> try things out with the latest kernels.
>
> I have an inkling as to the reason why you're restricted
> in this way, which is that you're limited to some vendor's
> kernel.  Am I right?

Yes. You are right. It is a vendor kernel based on 2.6.9 along
with some vendor specific kernel patches.
I agree that it is not always possible to support older kernels
and i was not expecting you to do that.
I tried to do this myself by identifying the tso related patches
that went in after 2.6.9 and just wanted to find out if i missed
any other patches that may have fixed the tso related assertions.

We will try to see if this problem can be reproduced with
2.6.10-rc2.

Thanks
Sridhar

Re: Assertions when lowering tcp_tso_win_divisor

[Arnaldo Carvalho de Melo]

Sridhar Samudrala wrote:
> On Fri, 19 Nov 2004, David S. Miller wrote:
> 
> 
>>On Fri, 19 Nov 2004 18:00:21 -0800 (PST)
>>Sridhar Samudrala <sri@us.ibm.com> wrote:
>>
>>
>>>Unfortunately as we cannot upgrade to 2.6.10-rc2
>>
>>This doesn't work.  You cannot expect upstream maintainers
>>to work with you if you're not willing to or able to
>>try things out with the latest kernels.
>>
>>I have an inkling as to the reason why you're restricted
>>in this way, which is that you're limited to some vendor's
>>kernel.  Am I right?
> 
> 
> Yes. You are right. It is a vendor kernel based on 2.6.9 along
> with some vendor specific kernel patches.
> I agree that it is not always possible to support older kernels
> and i was not expecting you to do that.
> I tried to do this myself by identifying the tso related patches
> that went in after 2.6.9 and just wanted to find out if i missed
> any other patches that may have fixed the tso related assertions.
> 
> We will try to see if this problem can be reproduced with
> 2.6.10-rc2.

That will help figuring out the problem, even if afterwards you
continue using 2.6.9, by determining if the problem was solved already
8) i.e. it may be another bug, not the tso related ones solved in
2.6.10-rc2.

- Arnaldo

Re: Assertions when lowering tcp_tso_win_divisor

[David S. Miller]

On Sat, 20 Nov 2004 15:19:54 -0800 (PST)
Sridhar Samudrala <sri@us.ibm.com> wrote:

> I tried to do this myself by identifying the tso related patches
> that went in after 2.6.9 and just wanted to find out if i missed
> any other patches that may have fixed the tso related assertions.

At a minimum you missed the net/core/skbuff.c patch Herbert Xu
made to fix missing of copying around tso bits when copying/cloning
SKBs.