From: "Eren Türkay" <erent@skyatlas.com>
To: Eric Dumazet <eric.dumazet@gmail.com>
Cc: Network Development <netdev@vger.kernel.org>
Subject: Re: PROBLEM: Bnx2x Checksum/Length Error Over GRE Tunnel
Date: Wed, 04 Feb 2015 12:00:46 +0200 [thread overview]
Message-ID: <54D1EDCE.3030602@skyatlas.com> (raw)
In-Reply-To: <1422989196.907.34.camel@edumazet-glaptop2.roam.corp.google.com>
[-- Attachment #1: Type: text/plain, Size: 3498 bytes --]
On 03-02-2015 20:46, Eric Dumazet wrote:
> Can you please send : ifconfig -a, as I suspect an MTU issue
You are right. Previously I set gre interface MTU to 1500 and NIC interface had 9000 MTU.
I was able to get 3Gbit in a single flow, and 5-6Gbit in parallel. When I recreated the interfaces,
the MTU of gre was set 8976 and I can get 9.5Gbit in a single flow. The reason why I set 1500 on
gre interface was to mimic openvswitch setup as the main problem I have been investigating is VM
network performance problem (they have 3Gbit throughput). I have OpenStack installation using OVS with GRE
and those interfaces are set 1500 MTU by default. I tried to test iproute2 implementation with 1500.
With correct MTU setting in gre interface, it seems to work without checksumming and offloading
but I haven't tested my original OVS setting. I will tweak OVS bridge MTU settings and try to get
it work. If baremetal-to-baremetal works, I suspect there is an issue with OVS setup.
However, when I enable tx on, netperf cannot even connect.
"ifconfig -a" and "ovs-vsctl show" result is attached.
I am not sure if tx offloading will affect the VM performance. Regardless of OVS problem, there seems to be a
bug with segmentation offloading as it's only possible to use the interface with "tx off". I have compiled the
kernel with debugging on and I can enable debugging on related parts in the kernel for further information.
> Here I easily reach line rate with a single flow.
>
> ip tunnel add gre1 mode gre remote 10.246.11.52 local 10.246.11.51
> ip link set gre1 up
> ip addr add 7.7.8.51/24 dev gre1
ip tunnel add gre1 mode gre remote 10.20.0.162 local 10.20.0.161
ip l set gre1 up
ip addr add 7.7.8.161/24 dev gre1
root@compute1vx:~# ip r get 7.7.8.162
7.7.8.162 dev gre1 src 7.7.8.161
cache expires 539sec mtu 8976
root@compute1vx:~# ethtool -K em1 tx off
Actual changes:
tx-checksumming: off
tx-checksum-ipv4: off
tx-checksum-ipv6: off
tcp-segmentation-offload: off
tx-tcp-segmentation: off [requested on]
tx-tcp-ecn-segmentation: off [requested on]
tx-tcp6-segmentation: off [requested on]
root@compute1vx:~# netperf -H 7.7.8.162 -Cc
MIGRATED TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 7.7.8.162 () port 0 AF_INET : demo
Recv Send Send Utilization Service Demand
Socket Socket Message Elapsed Send Recv Send Recv
Size Size Size Time Throughput local remote local remote
bytes bytes bytes secs. 10^6bits/s % S % S us/KB us/KB
87380 16384 16384 10.00 9534.00 4.05 4.39 0.835 0.906
root@compute1vx:~# ethtool -K em1 tx on
Actual changes:
tx-checksumming: on
tx-checksum-ipv4: on
tx-checksum-ipv6: on
tcp-segmentation-offload: on
tx-tcp-segmentation: on
tx-tcp-ecn-segmentation: on
tx-tcp6-segmentation: on
root@compute1vx:~# netperf -H 7.7.8.162 -Cc
establish control: are you sure there is a netserver listening on 7.7.8.162 at port 12865?
establish_control could not establish the control connection from 0.0.0.0 port 0 address family AF_UNSPEC to 7.7.8.162 port 12865 address family AF_INET
Regards,
Eren
--
System Administrator
https://skyatlas.com/
[-- Attachment #2: ifconfig-and-ovs-result.txt --]
[-- Type: text/plain, Size: 6306 bytes --]
root@compute1vx:~# ifconfig -a
br-int Link encap:Ethernet HWaddr 62:6e:a4:45:76:4d
inet6 addr: fe80::e897:f0ff:fee3:5a0d/64 Scope:Link
UP BROADCAST RUNNING MTU:1500 Metric:1
RX packets:50 errors:0 dropped:0 overruns:0 frame:0
TX packets:8 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:3856 (3.8 KB) TX bytes:648 (648.0 B)
br-tun Link encap:Ethernet HWaddr 72:52:f7:60:bc:4a
inet6 addr: fe80::70d8:27ff:fe57:36fd/64 Scope:Link
UP BROADCAST RUNNING MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:8 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:0 (0.0 B) TX bytes:648 (648.0 B)
em1 Link encap:Ethernet HWaddr 00:26:55:1f:9d:a0
inet addr:10.20.0.161 Bcast:10.20.0.255 Mask:255.255.255.0
inet6 addr: fe80::226:55ff:fe1f:9da0/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:9000 Metric:1
RX packets:2775967 errors:1204 dropped:0 overruns:1204 frame:0
TX packets:19544604 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:309879554 (309.8 MB) TX bytes:80709251505 (80.7 GB)
Interrupt:42 Memory:fb000000-fb7fffff
em2 Link encap:Ethernet HWaddr 00:26:55:1f:9d:a4
inet addr:192.168.88.61 Bcast:192.168.88.255 Mask:255.255.255.0
inet6 addr: fe80::226:55ff:fe1f:9da4/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:981996 errors:0 dropped:0 overruns:0 frame:0
TX packets:504463 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:1423897061 (1.4 GB) TX bytes:38829176 (38.8 MB)
Interrupt:53 Memory:fa000000-fa7fffff
gre0 Link encap:UNSPEC HWaddr 00-00-00-00-32-36-3A-35-00-00-00-00-00-00-00-00
NOARP MTU:1476 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)
gre1 Link encap:UNSPEC HWaddr 0A-14-00-A1-00-00-00-00-00-00-00-00-00-00-00-00
inet addr:7.7.8.161 P-t-P:7.7.8.161 Mask:255.255.255.0
inet6 addr: fe80::5efe:a14:a1/64 Scope:Link
UP POINTOPOINT RUNNING NOARP MTU:8976 Metric:1
RX packets:2541984 errors:0 dropped:0 overruns:0 frame:0
TX packets:1704256 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:133877496 (133.8 MB) TX bytes:78944172512 (78.9 GB)
gretap0 Link encap:Ethernet HWaddr 00:00:00:00:00:00
BROADCAST MULTICAST MTU:1462 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)
lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:65536 Metric:1
RX packets:15 errors:0 dropped:0 overruns:0 frame:0
TX packets:15 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:1878 (1.8 KB) TX bytes:1878 (1.8 KB)
ovs-system Link encap:Ethernet HWaddr fa:69:1a:53:4f:73
BROADCAST MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)
qbre90aa6bf-36 Link encap:Ethernet HWaddr 3a:d6:2c:ba:09:57
inet6 addr: fe80::387c:4bff:fe67:904f/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:44 errors:0 dropped:0 overruns:0 frame:0
TX packets:8 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:2760 (2.7 KB) TX bytes:648 (648.0 B)
qvbe90aa6bf-36 Link encap:Ethernet HWaddr 3a:d6:2c:ba:09:57
inet6 addr: fe80::38d6:2cff:feba:957/64 Scope:Link
UP BROADCAST RUNNING PROMISC MULTICAST MTU:1500 Metric:1
RX packets:45 errors:0 dropped:0 overruns:0 frame:0
TX packets:15 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:3466 (3.4 KB) TX bytes:1206 (1.2 KB)
qvoe90aa6bf-36 Link encap:Ethernet HWaddr d2:d7:77:9a:d9:1e
inet6 addr: fe80::d0d7:77ff:fe9a:d91e/64 Scope:Link
UP BROADCAST RUNNING PROMISC MULTICAST MTU:1500 Metric:1
RX packets:15 errors:0 dropped:0 overruns:0 frame:0
TX packets:45 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:1206 (1.2 KB) TX bytes:3466 (3.4 KB)
virbr0 Link encap:Ethernet HWaddr 8e:54:70:cb:8a:fc
inet addr:192.168.122.1 Bcast:192.168.122.255 Mask:255.255.255.0
UP BROADCAST MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)
root@compute1vx:~# ovs-vsctl show
db3856c4-3af5-45af-9676-debc379a3e9a
Bridge br-int
fail_mode: secure
Port patch-tun
Interface patch-tun
type: patch
options: {peer=patch-int}
Port br-int
Interface br-int
type: internal
Port "qvoe90aa6bf-36"
tag: 1
Interface "qvoe90aa6bf-36"
Bridge br-tun
Port br-tun
Interface br-tun
type: internal
Port patch-int
Interface patch-int
type: patch
options: {peer=patch-tun}
Port "gre-0a1400a2"
Interface "gre-0a1400a2"
type: gre
options: {df_default="true", in_key=flow, local_ip="10.20.0.161", out_key=flow, remote_ip="10.20.0.162"}
ovs_version: "2.0.2"
next prev parent reply other threads:[~2015-02-04 10:00 UTC|newest]
Thread overview: 7+ messages / expand[flat|nested] mbox.gz Atom feed top
[not found] <54D0A345.2030108@skyatlas.com>
2015-02-03 18:05 ` PROBLEM: Bnx2x Checksum/Length Error Over GRE Tunnel Eren Türkay
2015-02-03 18:46 ` Eric Dumazet
2015-02-03 19:29 ` Rick Jones
2015-02-04 10:00 ` Eren Türkay [this message]
2015-02-05 16:13 ` Yuval Mintz
2015-02-06 7:51 ` Eren Türkay
2015-02-08 5:51 ` Yuval Mintz
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=54D1EDCE.3030602@skyatlas.com \
--to=erent@skyatlas.com \
--cc=eric.dumazet@gmail.com \
--cc=netdev@vger.kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).