Re: TCP_MAXSEG vs TCP/generic segmentation offload (tso/gso)

Re: TCP_MAXSEG vs TCP/generic segmentation offload (tso/gso)

[Eric Dumazet]

Le jeudi 25 novembre 2010 à 14:44 +0100, Niels Möller a écrit :
> [ This is a slightly updated repost of a an October 21 mail to the
>   linux-net list. Any hints or advice appreciated. /Niels ]
> 

CC netdev

> I have been observing large ethernet packets when generating TCP traffic
> over a local ethernet, up to a bit over 20000 bytes, even though the
> interface MTU is 1500 bytes.
> 
> Furthermore, I tried to use setsockopt with TCP_MAXSEG to limit the TCP
> segment size further, to 1000 bytes, and that didn't have any effect.
> 
> When bugreporting a related problem to the debian kernel maintainers, I
> was told that the behaviour may be linked to the use of TCP segmentation
> offload (see http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=600286).
> 
> Disabling TSO and GSO using ethtool solves both problems: Generated
> packets now are limited in size by both the interface MTU and the
> segment size set with setsockopt. (Except the atl1c driver, where
> ethtool -K eth0 tso off only results in a "Cannot set device tcp
> segmentation offload settings: Operation not supported").
> 
> Before I try to write proper bug reports on specific network drivers (I
> have seen problems with several network drivers on different machines,
> unfortunately using different linux versions), I would like to know:
> 
> 1. Is TCP_MAXSEG supposed to work at all with network drivers that do
>    tcp segmentation offload?
> 
> 2. If it is supposed to work, can someone give a rough sketch on how the
>    per-socket segment size, set with setsockopt(... TCP_MAXSEG,...), is
>    passed down to the driver and to the network hardware? I suspect it
>    ought to be passed with each "pseudo-packet" to be transmitted.
> 
> I have spent some time searching the documentation and the net for
> answers, without result, hence I'm posting to this list. I'm not
> subscribed, so please cc any replies.
> 
> (Regarding packets larger than the interface MTU, that seems clearly
> buggy to me, and I think I already know enough to be able to file proper
> bug reports. And in the atl1c driver, it appears to have been fixed
> between 1.0.0.1-NAPI and 1.0.1.0-NAPI).

GSO is a software technique. Same for GRO.

Physical frames are indeed 1500 bytes (on regular ethernet links)

tcpdump gives you the high level view, before segmentation done in lower
levels (by NIC itself or in linux stack) in Transmit path.

We also have GRO in receive path, able to coalesce several 1500 bytes
frames into a single one (if same tcp flow), so that overhead in stacks
is lowered (netfilter, IP stack, tcp stack, bridge, routing ...)

So... there is no 'bug', unless you trust too much tcpdump output.

Re: TCP_MAXSEG vs TCP/generic segmentation offload (tso/gso)

[Niels Möller]

Eric Dumazet <eric.dumazet@gmail.com> writes:

> So... there is no 'bug', unless you trust too much tcpdump output.

I really expected tcpdump -e to display the actual values in the link
layer header, including the correct frame size. It's more than a bit
confusing if that is not the case...

In the future, I will try to remember to always run tcpdump on a network
node which (i) is different from the sending one, and (ii) has GRO
disabled (and hence will discard packets if it has trouble processing
them all, rather than coalesce them).

What about the TCP_MAXSEG socket option, should that work? From a quick
look at driver source code, I could only see the handling of the
per-interface MTU, no per-socket segment size.

Thanks for the quick reply,
/Niels

-- 
Niels Möller. PGP-encrypted email is preferred. Keyid C0B98E26.
Internet email is subject to wholesale government surveillance.

Re: TCP_MAXSEG vs TCP/generic segmentation offload (tso/gso)

[Eric Dumazet]

Le jeudi 25 novembre 2010 à 16:09 +0100, Niels Möller a écrit :
> Eric Dumazet <eric.dumazet@gmail.com> writes:
> 
> > So... there is no 'bug', unless you trust too much tcpdump output.
> 
> I really expected tcpdump -e to display the actual values in the link
> layer header, including the correct frame size. It's more than a bit
> confusing if that is not the case...
> 
> In the future, I will try to remember to always run tcpdump on a network
> node which (i) is different from the sending one, and (ii) has GRO
> disabled (and hence will discard packets if it has trouble processing
> them all, rather than coalesce them).
> 

Just disable GSO and TSO on sending machine, then tcpdump will show you
individual frames.


> What about the TCP_MAXSEG socket option, should that work? From a quick
> look at driver source code, I could only see the handling of the
> per-interface MTU, no per-socket segment size.

TCP_MAXSEG is certainly not handled in driver layer, but TCP layer.

/* If user gave his TCP_MAXSEG, record it to clamp */
if (tp->rx_opt.user_mss)
	tp->rx_opt.mss_clamp = tp->rx_opt.user_mss;

I believe TCP_MAXSEG is working fine, but GRO/GSO dont care at all :
They coalesce frames whatever their size is.

Re: TCP_MAXSEG vs TCP/generic segmentation offload (tso/gso)

[Niels Möller]

Eric Dumazet <eric.dumazet@gmail.com> writes:

> I believe TCP_MAXSEG is working fine, but GRO/GSO dont care at all :
> They coalesce frames whatever their size is.

I was under the impression that TSO (and maybe GSO) implied more
cleverness in the network card; that the network card more or less gets
to decide by itself how to divide a tcp stream into segments. And for
example in the atl1c driver which I looked a bit into, this was what the
REG_MTU register was for. Seems I have gotten this totally wrong.

Maybe Documentation/networking/netdevices.txt could clarify how it
works. Currently, it says

: Segmentation Offload (GSO, TSO) is an exception to this rule.  The
: upper layer protocol may pass a large socket buffer to the device
: transmit routine, and the device will break that up into separate
: packets based on the current MTU.

Regards, and thanks for your patience,
/Niels

-- 
Niels Möller. PGP-encrypted email is preferred. Keyid C0B98E26.
Internet email is subject to wholesale government surveillance.

Re: TCP_MAXSEG vs TCP/generic segmentation offload (tso/gso)

[Eric Dumazet]

Le jeudi 25 novembre 2010 à 17:25 +0100, Niels Möller a écrit :


> 
> I was under the impression that TSO (and maybe GSO) implied more
> cleverness in the network card; that the network card more or less gets
> to decide by itself how to divide a tcp stream into segments. And for
> example in the atl1c driver which I looked a bit into, this was what the
> REG_MTU register was for. Seems I have gotten this totally wrong.
> 

You were not totally wrong, but device does not use its own MTU to
perform the split : We give it the MSS of the flow.

You can have multiple flows in parallel, each with its own MSS, while
device has a single MTU.

> Maybe Documentation/networking/netdevices.txt could clarify how it
> works. Currently, it says
> 
> : Segmentation Offload (GSO, TSO) is an exception to this rule.  The
> : upper layer protocol may pass a large socket buffer to the device
> : transmit routine, and the device will break that up into separate
> : packets based on the current MTU.


MTU means : maximum transmission unit. But each layer has its own :)

In this context, TCP protocol, so MSS should be taken into account.

By default, MSS derives from device MTU  (ipv4 without options case :
MSS = MTU - 40), but user can change it with TCP_MAXSEG.