low vxlan throughput with tso enabled

netdev.vger.kernel.org archive mirror
 help / color / mirror / Atom feed

* low vxlan throughput with tso enabled
@ 2015-02-03 15:50 Reiner Herrmann
  2015-02-03 16:26 ` Tom Herbert
  2015-02-03 16:36 ` Rick Jones
  0 siblings, 2 replies; 5+ messages in thread
From: Reiner Herrmann @ 2015-02-03 15:50 UTC (permalink / raw)
  To: netdev; +Cc: svens, Vittorio Curcio

Hi!

I have a vxlan tunnel established between two network interfaces, which
both have a MTU of 1500. The vxlan interfaces have the same MTU.
With TSO enabled, I observe low throughput with TCP connections (<100 kB/s).
Disabling TSO works around this issue and throughput is as expected.
Can someone please explain how TSO is influencing the tunnel to cause
such a difference?

Kind regards,
Reiner

^ permalink raw reply	[flat|nested] 5+ messages in thread

* Re: low vxlan throughput with tso enabled
  2015-02-03 15:50 low vxlan throughput with tso enabled Reiner Herrmann
@ 2015-02-03 16:26 ` Tom Herbert
  2015-02-04 12:34   ` Reiner Herrmann
  2015-02-03 16:36 ` Rick Jones
  1 sibling, 1 reply; 5+ messages in thread
From: Tom Herbert @ 2015-02-03 16:26 UTC (permalink / raw)
  To: Reiner Herrmann; +Cc: Linux Netdev List, svens, Vittorio Curcio

On Tue, Feb 3, 2015 at 7:50 AM, Reiner Herrmann
<reiner.herrmann@sophos.com> wrote:
> Hi!
>
> I have a vxlan tunnel established between two network interfaces, which
> both have a MTU of 1500. The vxlan interfaces have the same MTU.
> With TSO enabled, I observe low throughput with TCP connections (<100 kB/s).
> Disabling TSO works around this issue and throughput is as expected.
> Can someone please explain how TSO is influencing the tunnel to cause
> such a difference?
>
Please provide more information. What NIC? What is the link rate? What
are you running to test. Also report 'ethtool -k' of both tunnel and
Ethernet interfaces, and tcpdump on the interfaces (to see GSO
packets).

Thanks

> Kind regards,
> Reiner
>
> --
> To unsubscribe from this list: send the line "unsubscribe netdev" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

^ permalink raw reply	[flat|nested] 5+ messages in thread

* Re: low vxlan throughput with tso enabled
  2015-02-03 16:26 ` Tom Herbert
@ 2015-02-04 12:34   ` Reiner Herrmann
  0 siblings, 0 replies; 5+ messages in thread
From: Reiner Herrmann @ 2015-02-04 12:34 UTC (permalink / raw)
  To: Tom Herbert; +Cc: Linux Netdev List, svens, Vittorio Curcio

[-- Attachment #1: Type: text/plain, Size: 6220 bytes --]

On 02/03/2015 05:26 PM, Tom Herbert wrote:
> On Tue, Feb 3, 2015 at 7:50 AM, Reiner Herrmann
> <reiner.herrmann@sophos.com> wrote:
>> I have a vxlan tunnel established between two network interfaces, which
>> both have a MTU of 1500. The vxlan interfaces have the same MTU.
>> With TSO enabled, I observe low throughput with TCP connections (<100 kB/s).
>> Disabling TSO works around this issue and throughput is as expected.
>> Can someone please explain how TSO is influencing the tunnel to cause
>> such a difference?
> Please provide more information. What NIC? What is the link rate? What
> are you running to test. Also report 'ethtool -k' of both tunnel and
> Ethernet interfaces, and tcpdump on the interfaces (to see GSO
> packets).

Both are GBit NICs (it can be reproduced with different GBit NICs).
One has a link rate of 100 Mbps, the other 1000 Mbps.

To test the throughput, I'm using iperf (in TCP mode) between the two vxlan
interfaces.
I attached captures of the Ethernet interfaces from both devices, one with TSO
enabled on both vxlan interfaces (low throughput), and one with TSO disabled
on the vxlan interface of the iperf client (high throughput).

And below is the ethtool information you asked for.
Thank you for having a look.


Endpoint 1:
===========

Features for eth0:
rx-checksumming: on
tx-checksumming: on
     tx-checksum-ipv4: off [fixed]
     tx-checksum-ip-generic: on
     tx-checksum-ipv6: off [fixed]
     tx-checksum-fcoe-crc: off [fixed]
     tx-checksum-sctp: off [fixed]
scatter-gather: on
     tx-scatter-gather: on
     tx-scatter-gather-fraglist: off [fixed]
tcp-segmentation-offload: on
     tx-tcp-segmentation: on
     tx-tcp-ecn-segmentation: off [fixed]
     tx-tcp6-segmentation: on
udp-fragmentation-offload: off [fixed]
generic-segmentation-offload: on
generic-receive-offload: on
large-receive-offload: off [fixed]
rx-vlan-offload: on
tx-vlan-offload: on
ntuple-filters: off [fixed]
receive-hashing: on
highdma: on [fixed]
rx-vlan-filter: on [fixed]
vlan-challenged: off [fixed]
tx-lockless: off [fixed]
netns-local: off [fixed]
tx-gso-robust: off [fixed]
tx-fcoe-segmentation: off [fixed]
tx-gre-segmentation: off [fixed]
tx-udp_tnl-segmentation: off [fixed]
tx-mpls-segmentation: off [fixed]
fcoe-mtu: off [fixed]
tx-nocache-copy: off
loopback: off [fixed]
rx-fcs: off
rx-all: off
tx-vlan-stag-hw-insert: off [fixed]
rx-vlan-stag-hw-parse: off [fixed]
rx-vlan-stag-filter: off [fixed]

Features for vxlan0:
rx-checksumming: on
tx-checksumming: on
     tx-checksum-ipv4: off [fixed]
     tx-checksum-ip-generic: on
     tx-checksum-ipv6: off [fixed]
     tx-checksum-fcoe-crc: off [fixed]
     tx-checksum-sctp: off [fixed]
scatter-gather: on
     tx-scatter-gather: on
     tx-scatter-gather-fraglist: off [fixed]
tcp-segmentation-offload: on
     tx-tcp-segmentation: on
     tx-tcp-ecn-segmentation: on
     tx-tcp6-segmentation: on
udp-fragmentation-offload: on
generic-segmentation-offload: on
generic-receive-offload: on
large-receive-offload: off [fixed]
rx-vlan-offload: off [fixed]
tx-vlan-offload: on
ntuple-filters: off [fixed]
receive-hashing: off [fixed]
highdma: off [fixed]
rx-vlan-filter: off [fixed]
vlan-challenged: off [fixed]
tx-lockless: on [fixed]
netns-local: on [fixed]
tx-gso-robust: off [fixed]
tx-fcoe-segmentation: off [fixed]
tx-gre-segmentation: off [fixed]
tx-udp_tnl-segmentation: off [fixed]
tx-mpls-segmentation: off [fixed]
fcoe-mtu: off [fixed]
tx-nocache-copy: off
loopback: off [fixed]
rx-fcs: off [fixed]
rx-all: off [fixed]
tx-vlan-stag-hw-insert: on
rx-vlan-stag-hw-parse: off [fixed]
rx-vlan-stag-filter: off [fixed]


Endpoint 2:
===========

Features for eth0:
rx-checksumming: on
tx-checksumming: on
     tx-checksum-ipv4: off [fixed]
     tx-checksum-ip-generic: on
     tx-checksum-ipv6: off [fixed]
     tx-checksum-fcoe-crc: off [fixed]
     tx-checksum-sctp: off [fixed]
scatter-gather: on
     tx-scatter-gather: on
     tx-scatter-gather-fraglist: off [fixed]
tcp-segmentation-offload: on
     tx-tcp-segmentation: on
     tx-tcp-ecn-segmentation: off [fixed]
     tx-tcp6-segmentation: on
udp-fragmentation-offload: off [fixed]
generic-segmentation-offload: on
generic-receive-offload: on
large-receive-offload: off [fixed]
rx-vlan-offload: on
tx-vlan-offload: on
ntuple-filters: off [fixed]
receive-hashing: on
highdma: on [fixed]
rx-vlan-filter: off [fixed]
vlan-challenged: off [fixed]
tx-lockless: off [fixed]
netns-local: off [fixed]
tx-gso-robust: off [fixed]
tx-fcoe-segmentation: off [fixed]
tx-gre-segmentation: off [fixed]
tx-ipip-segmentation: off [fixed]
tx-sit-segmentation: off [fixed]
tx-udp_tnl-segmentation: off [fixed]
tx-mpls-segmentation: off [fixed]
fcoe-mtu: off [fixed]
tx-nocache-copy: off
loopback: off [fixed]
rx-fcs: off
rx-all: off
tx-vlan-stag-hw-insert: off [fixed]
rx-vlan-stag-hw-parse: off [fixed]
rx-vlan-stag-filter: off [fixed]
l2-fwd-offload: off [fixed]
busy-poll: off [fixed]

Features for vxlan0:
rx-checksumming: on
tx-checksumming: on
     tx-checksum-ipv4: off [fixed]
     tx-checksum-ip-generic: on
     tx-checksum-ipv6: off [fixed]
     tx-checksum-fcoe-crc: off [fixed]
     tx-checksum-sctp: off [fixed]
scatter-gather: on
     tx-scatter-gather: on
     tx-scatter-gather-fraglist: off [fixed]
tcp-segmentation-offload: on
     tx-tcp-segmentation: on
     tx-tcp-ecn-segmentation: on
     tx-tcp6-segmentation: on
udp-fragmentation-offload: on
generic-segmentation-offload: on
generic-receive-offload: on
large-receive-offload: off [fixed]
rx-vlan-offload: off [fixed]
tx-vlan-offload: on
ntuple-filters: off [fixed]
receive-hashing: off [fixed]
highdma: off [fixed]
rx-vlan-filter: off [fixed]
vlan-challenged: off [fixed]
tx-lockless: on [fixed]
netns-local: off [fixed]
tx-gso-robust: off [fixed]
tx-fcoe-segmentation: off [fixed]
tx-gre-segmentation: off [fixed]
tx-ipip-segmentation: off [fixed]
tx-sit-segmentation: off [fixed]
tx-udp_tnl-segmentation: off [fixed]
tx-mpls-segmentation: off [fixed]
fcoe-mtu: off [fixed]
tx-nocache-copy: off
loopback: off [fixed]
rx-fcs: off [fixed]
rx-all: off [fixed]
tx-vlan-stag-hw-insert: on
rx-vlan-stag-hw-parse: off [fixed]
rx-vlan-stag-filter: off [fixed]
l2-fwd-offload: off [fixed]
busy-poll: off [fixed]



[-- Attachment #2: device1_eth0_tso_off.pcap.xz --]
[-- Type: application/x-xz, Size: 338104 bytes --]

[-- Attachment #3: device1_eth0_tso_on.pcap.xz --]
[-- Type: application/x-xz, Size: 31604 bytes --]

[-- Attachment #4: device2_eth0_tso_off.pcap.xz --]
[-- Type: application/x-xz, Size: 334164 bytes --]

[-- Attachment #5: device2_eth0_tso_on.pcap.xz --]
[-- Type: application/x-xz, Size: 35744 bytes --]

^ permalink raw reply	[flat|nested] 5+ messages in thread

* Re: low vxlan throughput with tso enabled
  2015-02-03 15:50 low vxlan throughput with tso enabled Reiner Herrmann
  2015-02-03 16:26 ` Tom Herbert
@ 2015-02-03 16:36 ` Rick Jones
  2015-02-04 12:42   ` Reiner Herrmann
  1 sibling, 1 reply; 5+ messages in thread
From: Rick Jones @ 2015-02-03 16:36 UTC (permalink / raw)
  To: Reiner Herrmann, netdev; +Cc: svens, Vittorio Curcio

On 02/03/2015 07:50 AM, Reiner Herrmann wrote:
> I have a vxlan tunnel established between two network interfaces, which
> both have a MTU of 1500. The vxlan interfaces have the same MTU.
> With TSO enabled, I observe low throughput with TCP connections (<100
> kB/s).
> Disabling TSO works around this issue and throughput is as expected.
> Can someone please explain how TSO is influencing the tunnel to cause
> such a difference?

I've been under the impression that one generally wants the MTU of a 
tunnel interface to be no more than the MTU of the physical interface 
over which it runs, less the size of the encapsulation headers used by 
the tunnel.  What happens when you make the MTU of the tunnel interface 
1400 bytes instead of 1500?

rick jones

^ permalink raw reply	[flat|nested] 5+ messages in thread

* Re: low vxlan throughput with tso enabled
  2015-02-03 16:36 ` Rick Jones
@ 2015-02-04 12:42   ` Reiner Herrmann
  0 siblings, 0 replies; 5+ messages in thread
From: Reiner Herrmann @ 2015-02-04 12:42 UTC (permalink / raw)
  To: Rick Jones, netdev; +Cc: svens, Vittorio Curcio

On 02/03/2015 05:36 PM, Rick Jones wrote:
> I've been under the impression that one generally wants the MTU of a tunnel interface to be no more than the MTU of the physical interface over which it runs, less the 
> size of the encapsulation headers used by the tunnel.  What happens when you make the MTU of the tunnel interface 1400 bytes instead of 1500?
Lowering the MTU also brings throughput back to normal, but I'm interested in
why disabling TSO also seems to help (even with the same MTU).

^ permalink raw reply	[flat|nested] 5+ messages in thread

end of thread, other threads:[~2015-02-04 12:42 UTC | newest]

Thread overview: 5+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2015-02-03 15:50 low vxlan throughput with tso enabled Reiner Herrmann
2015-02-03 16:26 ` Tom Herbert
2015-02-04 12:34   ` Reiner Herrmann
2015-02-03 16:36 ` Rick Jones
2015-02-04 12:42   ` Reiner Herrmann

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).