possible problem with TSO and ip_queue

possible problem with TSO and ip_queue

[Chris Caputo]

Environment is 2.6.16.9 with e1000 NICs.

Paul and I (as part of the Quagga project) are working on a user mode 
method for doing BGP MD5 checksums using ip_queue.  All is working except 
when TSO is enabled I am seeing some problems.

It appears that when TSO is enabled, ip_queue can be used to manipulate 
packets that have a size higher than the particular MSS (and MTU) for a 
TCP stream.

This results in a larger than MSS packet going out to the NIC with a TCP 
MD5 checksum, getting split by the NIC to match the MTU of the LAN, and 
then being received as multiple packets with incorrect MD5 checksums.

Then a retransmit with a smaller/correct MSS happens and the packets are 
received with valid MD5 checksums.

So in effect, the data eventually makes it across the link, but it is sent 
twice, once with invalid checksums and once with valid checksums, and 
throughput is slow.

When TSO is disabled, things happen as they should.

Does this sound like a bug or by design?

Does it make sense that ip_queue mangled packets be subjected to TSO, 
given that the TCP header can be messed with by the user mode code?

Chris

Re: possible problem with TSO and ip_queue

[David S. Miller]

From: Chris Caputo <ccaputo@alt.net>
Date: Wed, 10 May 2006 18:44:30 +0000 (GMT)

> Does this sound like a bug or by design?
> 
> Does it make sense that ip_queue mangled packets be subjected to TSO, 
> given that the TCP header can be messed with by the user mode code?

This is by design.  Netfilter looks at full TSO frames, but the
TSO hardware in the cards do not interpret the TCP options in any
way whatsoever when splitting the TSO packet up into MSS frames,
they just copy the options as-is into each sub-frame.

Once you add MD5 checksums to the TCP packet, TSO can no longer be
used on that path, so you'll have to disable TSO either in the route
or via some other means.

Re: possible problem with TSO and ip_queue

[Paul Jakma]

On Wed, 10 May 2006, David S. Miller wrote:

> This is by design.  Netfilter looks at full TSO frames,

That explains it.

> Once you add MD5 checksums to the TCP packet, TSO can no longer be 
> used on that path, so you'll have to disable TSO either in the 
> route or via some other means.

Ok.

Is there a better way to deal with TSO besides documenting:

"disable TSO on all interfaces which /ever/ potentially could be used
  to reach TCP-MD5 authenticated BGP peers."

?

regards,
-- 
Paul Jakma	paul@clubi.ie	paul@jakma.org	Key ID: 64A2FF6A
Fortune:
Nothing ever becomes real until it is experienced.
- John Keats

Re: possible problem with TSO and ip_queue

[David S. Miller]

From: Paul Jakma <paul@clubi.ie>
Date: Wed, 10 May 2006 21:07:31 +0100 (IST)

> Is there a better way to deal with TSO besides documenting:
> 
> "disable TSO on all interfaces which /ever/ potentially could be used
>   to reach TCP-MD5 authenticated BGP peers."
> 
> ?

When you have a rule installed that will add MD5, just mark
the route as not being TSO capable.

What's the problem?

Re: possible problem with TSO and ip_queue

[Paul Jakma]

On Wed, 10 May 2006, David S. Miller wrote:

> When you have a rule installed that will add MD5, just mark the 
> route as not being TSO capable.

Ah, didn't realise this could be done with netfilter. What's the 
magic incantation? :)

> What's the problem?

None, that's perfect - we just didn't know!

regards,
-- 
Paul Jakma	paul@clubi.ie	paul@jakma.org	Key ID: 64A2FF6A
Fortune:
You will be run over by a beer truck.

Re: possible problem with TSO and ip_queue

[David S. Miller]

From: Paul Jakma <paul@clubi.ie>
Date: Wed, 10 May 2006 21:17:33 +0100 (IST)

> On Wed, 10 May 2006, David S. Miller wrote:
> 
> > When you have a rule installed that will add MD5, just mark the 
> > route as not being TSO capable.
> 
> Ah, didn't realise this could be done with netfilter. What's the 
> magic incantation? :)

Unfortunately, you can't disable this per-route like I thought
was possible, you can only disable it per-device or per-socket.