some TSO analysis

some TSO analysis

[Anton Blanchard]

Hi,

Some web server benchmarking on recent 2.6 shows TSO not to be the gain
it was in earlier kernels. I took a quick look at varying
tcp_tso_win_divisor, below are the results. 

Each line is one packet, with the size of each component of the packet
logged. Note the 4 byte packets at the end are a workaround for an e1000
bug. The MTU is 1500 bytes. The kernel is BK from today. The test does
64kB writes or sendfiles.

I noticed a few things:

- In the normal copy case, it looks like we split the first page between
  the skb->data and the first frag. eg:

  data: 1634 frags: 1408 4096 4096 4096 2108 4

  Is there a reason why the remaining 1408 bytes cant be stuffed into
  skb->data? Or is that an issue with MTU sized buffers?

- TSO really doesnt come into play until we set tcp_tso_win_divisor to
  2. BTW In this setup I am talking to a 2.4 machine on the same gigabit
  switch. At 4 the normal copy case sees some gains, but interestingly
  the zero copy does not.

Anton


tcp_tso_win_divisor = 8 (default)
normal copy:
data: 442 frags: 
data: 1634 frags: 2772 4 
data: 1634 frags: 896 1876 4 
data: 1634 frags: 2176 596 4 
data: 1634 frags: 2772 4 
data: 1634 frags: 640 2132 4 
data: 1634 frags: 1920 852 4 
data: 1634 frags: 2772 4 
data: 1634 frags: 384 2388 4 
data: 1634 frags: 1664 1108 4 

zero copy:
data: 66 frags: 1448 
data: 66 frags: 1200 248 
data: 66 frags: 1448 
data: 66 frags: 1448 
data: 66 frags: 952 
data: 66 frags: 1448 
data: 66 frags: 1448 
data: 66 frags: 1200 248 
data: 66 frags: 1448 
data: 66 frags: 1448 


tcp_tso_win_divisor = 4
normal copy:
data: 1634 frags: 2176 4096 844 4 
data: 1634 frags: 3200 3916 4 
data: 1634 frags: 128 3020 4 
data: 1634 frags: 1024 4096 1996 4 
data: 1634 frags: 2048 4096 972 4 
data: 1634 frags: 3072 4044 4 
data: 1634 frags: 4096 3020 4 
data: 1634 frags: 1024 4096 1996 4 
data: 1634 frags: 2048 4096 972 4 
data: 1634 frags: 3072 4044 4 

zero copy:
data: 66 frags: 1448 
data: 66 frags: 1448 
data: 66 frags: 1200 248 
data: 66 frags: 1448 
data: 66 frags: 1448 
data: 66 frags: 952 
data: 66 frags: 1448 
data: 66 frags: 1448 
data: 66 frags: 1200 248 
data: 66 frags: 1448 


tcp_tso_win_divisor = 2
normal copy:
data: 1634 frags: 1408 4096 4096 4096 2108 4 
data: 1634 frags: 1920 4096 4096 4096 1596 4 
data: 1634 frags: 2432 4096 4096 4096 1084 4 
data: 1634 frags: 2944 4096 4096 4096 572 4 
data: 1634 frags: 3456 4096 4096 4096 60 4 
data: 1634 frags: 3968 4096 4096 3644 4 
data: 1634 frags: 384 4096 4096 4096 3132 4 
data: 1634 frags: 896 4096 4096 4096 2620 4 
data: 1634 frags: 2432 4096 4096 4096 1084 4 

zero copy:
data: 66 frags: 2304 4096 4096 4096 2780 4 
data: 66 frags: 1312 4096 4096 4096 3772 4 
data: 66 frags: 320 4096 4096 4096 4096 668 4 
data: 66 frags: 3424 4096 4096 4096 1660 4 
data: 66 frags: 2432 4096 4096 4096 2652 4 
data: 66 frags: 1440 4096 4096 4096 3644 4 
data: 66 frags: 448 4096 4096 4096 4096 540 4 
data: 66 frags: 3552 4096 4096 4096 1532 4 
data: 66 frags: 2560 4096 4096 4096 2524 4 
data: 66 frags: 1568 4096 4096 4096 3516 4 

Re: some TSO analysis

[Herbert Xu]

Anton Blanchard <anton@samba.org> wrote:
> 
> Some web server benchmarking on recent 2.6 shows TSO not to be the gain
> it was in earlier kernels. I took a quick look at varying
> tcp_tso_win_divisor, below are the results. 

Thanks for doing the testing.  It's very interesting.
 
> I noticed a few things:
> 
> - In the normal copy case, it looks like we split the first page between
>  the skb->data and the first frag. eg:
> 
>  data: 1634 frags: 1408 4096 4096 4096 2108 4
> 
>  Is there a reason why the remaining 1408 bytes cant be stuffed into
>  skb->data? Or is that an issue with MTU sized buffers?

The reason is that the skb has to accomodate other things like
headers and skb_shared_info.  It seems to have added up to a
quite considerable value in your case.

Perhaps sendmsg should check whether the data can fit in the space
available in skb->data, and if not just go for a fragment right away.

> - TSO really doesnt come into play until we set tcp_tso_win_divisor to
>  2. BTW In this setup I am talking to a 2.4 machine on the same gigabit
>  switch. At 4 the normal copy case sees some gains, but interestingly
>  the zero copy does not.

This is understandable.  You really need to have a rcv window that's
much bigger than 64K to have any chance of filling up the TSO packet
with tcp_tso_win_divisor set to 8.

BTW, how did you measure the gains?

Cheers,
-- 
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

Re: some TSO analysis

[Anton Blanchard]

> >  data: 1634 frags: 1408 4096 4096 4096 2108 4
> > 
> >  Is there a reason why the remaining 1408 bytes cant be stuffed into
> >  skb->data? Or is that an issue with MTU sized buffers?
> 
> The reason is that the skb has to accomodate other things like
> headers and skb_shared_info.  It seems to have added up to a
> quite considerable value in your case.
> 
> Perhaps sendmsg should check whether the data can fit in the space
> available in skb->data, and if not just go for a fragment right away.

Originally I thought we could save ourselves a fragment. However
thinking about it more we will always have at least the headers in 
skb->data. If so there isnt much to gain by spilling into the fragment
straight away. Or am I wrong?

> This is understandable.  You really need to have a rcv window that's
> much bigger than 64K to have any chance of filling up the TSO packet
> with tcp_tso_win_divisor set to 8.

That makes sense.

> BTW, how did you measure the gains?

The TSO gains? On older 2.6 the TSO gain was easy to see on a web
benchmark. On recent kernels it isnt, but we havent tested a divisor
of 4 or 2 yet.

Im interested if the divisor of 8 was based on some testing, or if we
can tune the default a bit. It would be nice for TSO to kick in on
normal workloads (like web serving).

Anton

Re: some TSO analysis

[David S. Miller]

On Thu, 11 Nov 2004 19:36:47 +1100
Anton Blanchard <anton@samba.org> wrote:

> Im interested if the divisor of 8 was based on some testing, or if we
> can tune the default a bit. It would be nice for TSO to kick in on
> normal workloads (like web serving).

It was the best compromise found between performance and burstyness.

If you set it too small, you get large bursts of TSO created
frames which is very undesirable.