From: "Eren Türkay" <erent@skyatlas.com>
To: Eric Dumazet <eric.dumazet@gmail.com>
Cc: Network Development <netdev@vger.kernel.org>
Subject: Re: PROBLEM: Bnx2x Checksum/Length Error Over GRE Tunnel
Date: Wed, 04 Feb 2015 12:00:46 +0200 [thread overview]
Message-ID: <54D1EDCE.3030602@skyatlas.com> (raw)
In-Reply-To: <1422989196.907.34.camel@edumazet-glaptop2.roam.corp.google.com>
[-- Attachment #1: Type: text/plain, Size: 3498 bytes --]
On 03-02-2015 20:46, Eric Dumazet wrote:
> Can you please send : ifconfig -a, as I suspect an MTU issue
You are right. Previously I set gre interface MTU to 1500 and NIC interface had 9000 MTU.
I was able to get 3Gbit in a single flow, and 5-6Gbit in parallel. When I recreated the interfaces,
the MTU of gre was set 8976 and I can get 9.5Gbit in a single flow. The reason why I set 1500 on
gre interface was to mimic openvswitch setup as the main problem I have been investigating is VM
network performance problem (they have 3Gbit throughput). I have OpenStack installation using OVS with GRE
and those interfaces are set 1500 MTU by default. I tried to test iproute2 implementation with 1500.
With correct MTU setting in gre interface, it seems to work without checksumming and offloading
but I haven't tested my original OVS setting. I will tweak OVS bridge MTU settings and try to get
it work. If baremetal-to-baremetal works, I suspect there is an issue with OVS setup.
However, when I enable tx on, netperf cannot even connect.
"ifconfig -a" and "ovs-vsctl show" result is attached.
I am not sure if tx offloading will affect the VM performance. Regardless of OVS problem, there seems to be a
bug with segmentation offloading as it's only possible to use the interface with "tx off". I have compiled the
kernel with debugging on and I can enable debugging on related parts in the kernel for further information.
> Here I easily reach line rate with a single flow.
>
> ip tunnel add gre1 mode gre remote 10.246.11.52 local 10.246.11.51
> ip link set gre1 up
> ip addr add 7.7.8.51/24 dev gre1
ip tunnel add gre1 mode gre remote 10.20.0.162 local 10.20.0.161
ip l set gre1 up
ip addr add 7.7.8.161/24 dev gre1
root@compute1vx:~# ip r get 7.7.8.162
7.7.8.162 dev gre1 src 7.7.8.161
cache expires 539sec mtu 8976
root@compute1vx:~# ethtool -K em1 tx off
Actual changes:
tx-checksumming: off
tx-checksum-ipv4: off
tx-checksum-ipv6: off
tcp-segmentation-offload: off
tx-tcp-segmentation: off [requested on]
tx-tcp-ecn-segmentation: off [requested on]
tx-tcp6-segmentation: off [requested on]
root@compute1vx:~# netperf -H 7.7.8.162 -Cc
MIGRATED TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 7.7.8.162 () port 0 AF_INET : demo
Recv Send Send Utilization Service Demand
Socket Socket Message Elapsed Send Recv Send Recv
Size Size Size Time Throughput local remote local remote
bytes bytes bytes secs. 10^6bits/s % S % S us/KB us/KB
87380 16384 16384 10.00 9534.00 4.05 4.39 0.835 0.906
root@compute1vx:~# ethtool -K em1 tx on
Actual changes:
tx-checksumming: on
tx-checksum-ipv4: on
tx-checksum-ipv6: on
tcp-segmentation-offload: on
tx-tcp-segmentation: on
tx-tcp-ecn-segmentation: on
tx-tcp6-segmentation: on
root@compute1vx:~# netperf -H 7.7.8.162 -Cc
establish control: are you sure there is a netserver listening on 7.7.8.162 at port 12865?
establish_control could not establish the control connection from 0.0.0.0 port 0 address family AF_UNSPEC to 7.7.8.162 port 12865 address family AF_INET
Regards,
Eren
--
System Administrator
https://skyatlas.com/
[-- Attachment #2: ifconfig-and-ovs-result.txt --]
[-- Type: text/plain, Size: 6306 bytes --]
root@compute1vx:~# ifconfig -a
br-int Link encap:Ethernet HWaddr 62:6e:a4:45:76:4d
inet6 addr: fe80::e897:f0ff:fee3:5a0d/64 Scope:Link
UP BROADCAST RUNNING MTU:1500 Metric:1
RX packets:50 errors:0 dropped:0 overruns:0 frame:0
TX packets:8 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:3856 (3.8 KB) TX bytes:648 (648.0 B)
br-tun Link encap:Ethernet HWaddr 72:52:f7:60:bc:4a
inet6 addr: fe80::70d8:27ff:fe57:36fd/64 Scope:Link
UP BROADCAST RUNNING MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:8 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:0 (0.0 B) TX bytes:648 (648.0 B)
em1 Link encap:Ethernet HWaddr 00:26:55:1f:9d:a0
inet addr:10.20.0.161 Bcast:10.20.0.255 Mask:255.255.255.0
inet6 addr: fe80::226:55ff:fe1f:9da0/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:9000 Metric:1
RX packets:2775967 errors:1204 dropped:0 overruns:1204 frame:0
TX packets:19544604 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:309879554 (309.8 MB) TX bytes:80709251505 (80.7 GB)
Interrupt:42 Memory:fb000000-fb7fffff
em2 Link encap:Ethernet HWaddr 00:26:55:1f:9d:a4
inet addr:192.168.88.61 Bcast:192.168.88.255 Mask:255.255.255.0
inet6 addr: fe80::226:55ff:fe1f:9da4/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:981996 errors:0 dropped:0 overruns:0 frame:0
TX packets:504463 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:1423897061 (1.4 GB) TX bytes:38829176 (38.8 MB)
Interrupt:53 Memory:fa000000-fa7fffff
gre0 Link encap:UNSPEC HWaddr 00-00-00-00-32-36-3A-35-00-00-00-00-00-00-00-00
NOARP MTU:1476 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)
gre1 Link encap:UNSPEC HWaddr 0A-14-00-A1-00-00-00-00-00-00-00-00-00-00-00-00
inet addr:7.7.8.161 P-t-P:7.7.8.161 Mask:255.255.255.0
inet6 addr: fe80::5efe:a14:a1/64 Scope:Link
UP POINTOPOINT RUNNING NOARP MTU:8976 Metric:1
RX packets:2541984 errors:0 dropped:0 overruns:0 frame:0
TX packets:1704256 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:133877496 (133.8 MB) TX bytes:78944172512 (78.9 GB)
gretap0 Link encap:Ethernet HWaddr 00:00:00:00:00:00
BROADCAST MULTICAST MTU:1462 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)
lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:65536 Metric:1
RX packets:15 errors:0 dropped:0 overruns:0 frame:0
TX packets:15 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:1878 (1.8 KB) TX bytes:1878 (1.8 KB)
ovs-system Link encap:Ethernet HWaddr fa:69:1a:53:4f:73
BROADCAST MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)
qbre90aa6bf-36 Link encap:Ethernet HWaddr 3a:d6:2c:ba:09:57
inet6 addr: fe80::387c:4bff:fe67:904f/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:44 errors:0 dropped:0 overruns:0 frame:0
TX packets:8 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:2760 (2.7 KB) TX bytes:648 (648.0 B)
qvbe90aa6bf-36 Link encap:Ethernet HWaddr 3a:d6:2c:ba:09:57
inet6 addr: fe80::38d6:2cff:feba:957/64 Scope:Link
UP BROADCAST RUNNING PROMISC MULTICAST MTU:1500 Metric:1
RX packets:45 errors:0 dropped:0 overruns:0 frame:0
TX packets:15 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:3466 (3.4 KB) TX bytes:1206 (1.2 KB)
qvoe90aa6bf-36 Link encap:Ethernet HWaddr d2:d7:77:9a:d9:1e
inet6 addr: fe80::d0d7:77ff:fe9a:d91e/64 Scope:Link
UP BROADCAST RUNNING PROMISC MULTICAST MTU:1500 Metric:1
RX packets:15 errors:0 dropped:0 overruns:0 frame:0
TX packets:45 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:1206 (1.2 KB) TX bytes:3466 (3.4 KB)
virbr0 Link encap:Ethernet HWaddr 8e:54:70:cb:8a:fc
inet addr:192.168.122.1 Bcast:192.168.122.255 Mask:255.255.255.0
UP BROADCAST MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)
root@compute1vx:~# ovs-vsctl show
db3856c4-3af5-45af-9676-debc379a3e9a
Bridge br-int
fail_mode: secure
Port patch-tun
Interface patch-tun
type: patch
options: {peer=patch-int}
Port br-int
Interface br-int
type: internal
Port "qvoe90aa6bf-36"
tag: 1
Interface "qvoe90aa6bf-36"
Bridge br-tun
Port br-tun
Interface br-tun
type: internal
Port patch-int
Interface patch-int
type: patch
options: {peer=patch-tun}
Port "gre-0a1400a2"
Interface "gre-0a1400a2"
type: gre
options: {df_default="true", in_key=flow, local_ip="10.20.0.161", out_key=flow, remote_ip="10.20.0.162"}
ovs_version: "2.0.2"
next prev parent reply other threads:[~2015-02-04 10:00 UTC|newest]
Thread overview: 7+ messages / expand[flat|nested] mbox.gz Atom feed top
[not found] <54D0A345.2030108@skyatlas.com>
2015-02-03 18:05 ` PROBLEM: Bnx2x Checksum/Length Error Over GRE Tunnel Eren Türkay
2015-02-03 18:46 ` Eric Dumazet
2015-02-03 19:29 ` Rick Jones
2015-02-04 10:00 ` Eren Türkay [this message]
2015-02-05 16:13 ` Yuval Mintz
2015-02-06 7:51 ` Eren Türkay
2015-02-08 5:51 ` Yuval Mintz
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=54D1EDCE.3030602@skyatlas.com \
--to=erent@skyatlas.com \
--cc=eric.dumazet@gmail.com \
--cc=netdev@vger.kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.