low vxlan throughput with tso enabled

low vxlan throughput with tso enabled

[Reiner Herrmann]

Hi!

I have a vxlan tunnel established between two network interfaces, which
both have a MTU of 1500. The vxlan interfaces have the same MTU.
With TSO enabled, I observe low throughput with TCP connections (<100 kB/s).
Disabling TSO works around this issue and throughput is as expected.
Can someone please explain how TSO is influencing the tunnel to cause
such a difference?

Kind regards,
Reiner

Re: low vxlan throughput with tso enabled

[Tom Herbert]

On Tue, Feb 3, 2015 at 7:50 AM, Reiner Herrmann
<reiner.herrmann@sophos.com> wrote:
> Hi!
>
> I have a vxlan tunnel established between two network interfaces, which
> both have a MTU of 1500. The vxlan interfaces have the same MTU.
> With TSO enabled, I observe low throughput with TCP connections (<100 kB/s).
> Disabling TSO works around this issue and throughput is as expected.
> Can someone please explain how TSO is influencing the tunnel to cause
> such a difference?
>
Please provide more information. What NIC? What is the link rate? What
are you running to test. Also report 'ethtool -k' of both tunnel and
Ethernet interfaces, and tcpdump on the interfaces (to see GSO
packets).

Thanks

> Kind regards,
> Reiner
>
> --
> To unsubscribe from this list: send the line "unsubscribe netdev" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: low vxlan throughput with tso enabled

[Rick Jones]

On 02/03/2015 07:50 AM, Reiner Herrmann wrote:
> I have a vxlan tunnel established between two network interfaces, which
> both have a MTU of 1500. The vxlan interfaces have the same MTU.
> With TSO enabled, I observe low throughput with TCP connections (<100
> kB/s).
> Disabling TSO works around this issue and throughput is as expected.
> Can someone please explain how TSO is influencing the tunnel to cause
> such a difference?

I've been under the impression that one generally wants the MTU of a 
tunnel interface to be no more than the MTU of the physical interface 
over which it runs, less the size of the encapsulation headers used by 
the tunnel.  What happens when you make the MTU of the tunnel interface 
1400 bytes instead of 1500?

rick jones

Re: low vxlan throughput with tso enabled

[Reiner Herrmann]

[-- Attachment #1: Type: text/plain, Size: 6220 bytes --]

On 02/03/2015 05:26 PM, Tom Herbert wrote:
> On Tue, Feb 3, 2015 at 7:50 AM, Reiner Herrmann
> <reiner.herrmann@sophos.com> wrote:
>> I have a vxlan tunnel established between two network interfaces, which
>> both have a MTU of 1500. The vxlan interfaces have the same MTU.
>> With TSO enabled, I observe low throughput with TCP connections (<100 kB/s).
>> Disabling TSO works around this issue and throughput is as expected.
>> Can someone please explain how TSO is influencing the tunnel to cause
>> such a difference?
> Please provide more information. What NIC? What is the link rate? What
> are you running to test. Also report 'ethtool -k' of both tunnel and
> Ethernet interfaces, and tcpdump on the interfaces (to see GSO
> packets).

Both are GBit NICs (it can be reproduced with different GBit NICs).
One has a link rate of 100 Mbps, the other 1000 Mbps.

To test the throughput, I'm using iperf (in TCP mode) between the two vxlan
interfaces.
I attached captures of the Ethernet interfaces from both devices, one with TSO
enabled on both vxlan interfaces (low throughput), and one with TSO disabled
on the vxlan interface of the iperf client (high throughput).

And below is the ethtool information you asked for.
Thank you for having a look.


Endpoint 1:
===========

Features for eth0:
rx-checksumming: on
tx-checksumming: on
     tx-checksum-ipv4: off [fixed]
     tx-checksum-ip-generic: on
     tx-checksum-ipv6: off [fixed]
     tx-checksum-fcoe-crc: off [fixed]
     tx-checksum-sctp: off [fixed]
scatter-gather: on
     tx-scatter-gather: on
     tx-scatter-gather-fraglist: off [fixed]
tcp-segmentation-offload: on
     tx-tcp-segmentation: on
     tx-tcp-ecn-segmentation: off [fixed]
     tx-tcp6-segmentation: on
udp-fragmentation-offload: off [fixed]
generic-segmentation-offload: on
generic-receive-offload: on
large-receive-offload: off [fixed]
rx-vlan-offload: on
tx-vlan-offload: on
ntuple-filters: off [fixed]
receive-hashing: on
highdma: on [fixed]
rx-vlan-filter: on [fixed]
vlan-challenged: off [fixed]
tx-lockless: off [fixed]
netns-local: off [fixed]
tx-gso-robust: off [fixed]
tx-fcoe-segmentation: off [fixed]
tx-gre-segmentation: off [fixed]
tx-udp_tnl-segmentation: off [fixed]
tx-mpls-segmentation: off [fixed]
fcoe-mtu: off [fixed]
tx-nocache-copy: off
loopback: off [fixed]
rx-fcs: off
rx-all: off
tx-vlan-stag-hw-insert: off [fixed]
rx-vlan-stag-hw-parse: off [fixed]
rx-vlan-stag-filter: off [fixed]

Features for vxlan0:
rx-checksumming: on
tx-checksumming: on
     tx-checksum-ipv4: off [fixed]
     tx-checksum-ip-generic: on
     tx-checksum-ipv6: off [fixed]
     tx-checksum-fcoe-crc: off [fixed]
     tx-checksum-sctp: off [fixed]
scatter-gather: on
     tx-scatter-gather: on
     tx-scatter-gather-fraglist: off [fixed]
tcp-segmentation-offload: on
     tx-tcp-segmentation: on
     tx-tcp-ecn-segmentation: on
     tx-tcp6-segmentation: on
udp-fragmentation-offload: on
generic-segmentation-offload: on
generic-receive-offload: on
large-receive-offload: off [fixed]
rx-vlan-offload: off [fixed]
tx-vlan-offload: on
ntuple-filters: off [fixed]
receive-hashing: off [fixed]
highdma: off [fixed]
rx-vlan-filter: off [fixed]
vlan-challenged: off [fixed]
tx-lockless: on [fixed]
netns-local: on [fixed]
tx-gso-robust: off [fixed]
tx-fcoe-segmentation: off [fixed]
tx-gre-segmentation: off [fixed]
tx-udp_tnl-segmentation: off [fixed]
tx-mpls-segmentation: off [fixed]
fcoe-mtu: off [fixed]
tx-nocache-copy: off
loopback: off [fixed]
rx-fcs: off [fixed]
rx-all: off [fixed]
tx-vlan-stag-hw-insert: on
rx-vlan-stag-hw-parse: off [fixed]
rx-vlan-stag-filter: off [fixed]


Endpoint 2:
===========

Features for eth0:
rx-checksumming: on
tx-checksumming: on
     tx-checksum-ipv4: off [fixed]
     tx-checksum-ip-generic: on
     tx-checksum-ipv6: off [fixed]
     tx-checksum-fcoe-crc: off [fixed]
     tx-checksum-sctp: off [fixed]
scatter-gather: on
     tx-scatter-gather: on
     tx-scatter-gather-fraglist: off [fixed]
tcp-segmentation-offload: on
     tx-tcp-segmentation: on
     tx-tcp-ecn-segmentation: off [fixed]
     tx-tcp6-segmentation: on
udp-fragmentation-offload: off [fixed]
generic-segmentation-offload: on
generic-receive-offload: on
large-receive-offload: off [fixed]
rx-vlan-offload: on
tx-vlan-offload: on
ntuple-filters: off [fixed]
receive-hashing: on
highdma: on [fixed]
rx-vlan-filter: off [fixed]
vlan-challenged: off [fixed]
tx-lockless: off [fixed]
netns-local: off [fixed]
tx-gso-robust: off [fixed]
tx-fcoe-segmentation: off [fixed]
tx-gre-segmentation: off [fixed]
tx-ipip-segmentation: off [fixed]
tx-sit-segmentation: off [fixed]
tx-udp_tnl-segmentation: off [fixed]
tx-mpls-segmentation: off [fixed]
fcoe-mtu: off [fixed]
tx-nocache-copy: off
loopback: off [fixed]
rx-fcs: off
rx-all: off
tx-vlan-stag-hw-insert: off [fixed]
rx-vlan-stag-hw-parse: off [fixed]
rx-vlan-stag-filter: off [fixed]
l2-fwd-offload: off [fixed]
busy-poll: off [fixed]

Features for vxlan0:
rx-checksumming: on
tx-checksumming: on
     tx-checksum-ipv4: off [fixed]
     tx-checksum-ip-generic: on
     tx-checksum-ipv6: off [fixed]
     tx-checksum-fcoe-crc: off [fixed]
     tx-checksum-sctp: off [fixed]
scatter-gather: on
     tx-scatter-gather: on
     tx-scatter-gather-fraglist: off [fixed]
tcp-segmentation-offload: on
     tx-tcp-segmentation: on
     tx-tcp-ecn-segmentation: on
     tx-tcp6-segmentation: on
udp-fragmentation-offload: on
generic-segmentation-offload: on
generic-receive-offload: on
large-receive-offload: off [fixed]
rx-vlan-offload: off [fixed]
tx-vlan-offload: on
ntuple-filters: off [fixed]
receive-hashing: off [fixed]
highdma: off [fixed]
rx-vlan-filter: off [fixed]
vlan-challenged: off [fixed]
tx-lockless: on [fixed]
netns-local: off [fixed]
tx-gso-robust: off [fixed]
tx-fcoe-segmentation: off [fixed]
tx-gre-segmentation: off [fixed]
tx-ipip-segmentation: off [fixed]
tx-sit-segmentation: off [fixed]
tx-udp_tnl-segmentation: off [fixed]
tx-mpls-segmentation: off [fixed]
fcoe-mtu: off [fixed]
tx-nocache-copy: off
loopback: off [fixed]
rx-fcs: off [fixed]
rx-all: off [fixed]
tx-vlan-stag-hw-insert: on
rx-vlan-stag-hw-parse: off [fixed]
rx-vlan-stag-filter: off [fixed]
l2-fwd-offload: off [fixed]
busy-poll: off [fixed]



[-- Attachment #2: device1_eth0_tso_off.pcap.xz --]
[-- Type: application/x-xz, Size: 338104 bytes --]

[-- Attachment #3: device1_eth0_tso_on.pcap.xz --]
[-- Type: application/x-xz, Size: 31604 bytes --]

[-- Attachment #4: device2_eth0_tso_off.pcap.xz --]
[-- Type: application/x-xz, Size: 334164 bytes --]

[-- Attachment #5: device2_eth0_tso_on.pcap.xz --]
[-- Type: application/x-xz, Size: 35744 bytes --]

Re: low vxlan throughput with tso enabled

[Reiner Herrmann]

On 02/03/2015 05:36 PM, Rick Jones wrote:
> I've been under the impression that one generally wants the MTU of a tunnel interface to be no more than the MTU of the physical interface over which it runs, less the 
> size of the encapsulation headers used by the tunnel.  What happens when you make the MTU of the tunnel interface 1400 bytes instead of 1500?
Lowering the MTU also brings throughput back to normal, but I'm interested in
why disabling TSO also seems to help (even with the same MTU).