PROBLEM: Bnx2x Checksum/Length Error Over GRE Tunnel

PROBLEM: Bnx2x Checksum/Length Error Over GRE Tunnel

[Eren Türkay]

[-- Attachment #1: Type: text/plain, Size: 11634 bytes --]

Hello,

I am having incorrect checksum and length calculation error when using
GRE tunnel and this leads to unstable/unreliable connections where even
simple nc connection cannot be made correctly due to a lot of TCP
retransmissions. The reported iperf output becomes 23Kbit/s.

When I disable tx checksumming with "ethtool -K eth0 tx off", the
problem seems to be solved but then I lose tcp segmentation offloading
support, without which adds an additional overhead on 10Gbit network
interface. With tx off, I get 4 to 7Gbit/s with parallel 40 connections
using iperf.

In tcpdump output (tcpdump -i em1 proto gre) when tx checksumming is
enabled, I see "checksum error" in every packet + "IP truncated-ip -
63631 bytes missing!" messages, and a lot of TCP
retransmissions. When tx is off, checksums are correct but I get 4-7Gbit
output. With GSO generalization code in [0] it is reported that
9.3Gbit/s is possible over GRE tunnel but I'm nowhere near these results.

Pcap file is attached when tx is on. I tried to contact the original
maintainers (broadcom) but I thought it would help to send it to netdev
mailing list as well.

Thank you for your time.

- Eren

[0] http://thread.gmane.org/gmane.linux.network/332194

Additional Information
======================
em1 is a 10G interface. Jumbo frames are enabled and MTU is set to 9000.
My GRE tunnel has 1500 MTU, and it is tunneled using em1.

Host OS: Ubuntu 14.04
Kernel: 3.18.0 (compiled from ubuntu-vivid repository)
Physical host: HP ProLiant BL460c G6
NIC: Broadcom Corporation NetXtreme II BCM57711E 10-Gigabit PCIe
Ethtool info:

driver: bnx2x
version: 1.710.51-0
firmware-version: bc 6.2.28 phy baa0.105
bus-info: 0000:02:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: yes


ethtook -k em1:
Features for em1:
rx-checksumming: on
tx-checksumming: on
        tx-checksum-ipv4: on
        tx-checksum-ip-generic: off [fixed]
        tx-checksum-ipv6: on
        tx-checksum-fcoe-crc: off [fixed]
        tx-checksum-sctp: off [fixed]
scatter-gather: on
        tx-scatter-gather: on
        tx-scatter-gather-fraglist: off [fixed]
tcp-segmentation-offload: on
        tx-tcp-segmentation: on
        tx-tcp-ecn-segmentation: on
        tx-tcp6-segmentation: on
udp-fragmentation-offload: off [fixed]
generic-segmentation-offload: on
generic-receive-offload: on
large-receive-offload: off
rx-vlan-offload: on [fixed]
tx-vlan-offload: on
ntuple-filters: off [fixed]
receive-hashing: on
highdma: on [fixed]
rx-vlan-filter: off [fixed]
vlan-challenged: off [fixed]
tx-lockless: off [fixed]
netns-local: off [fixed]
tx-gso-robust: off [fixed]
tx-fcoe-segmentation: off [fixed]
tx-gre-segmentation: on
tx-ipip-segmentation: on
tx-sit-segmentation: on
tx-udp_tnl-segmentation: on
tx-mpls-segmentation: off [fixed]
fcoe-mtu: off [fixed]
tx-nocache-copy: off
loopback: off
rx-fcs: off [fixed]
rx-all: off [fixed]
tx-vlan-stag-hw-insert: off [fixed]
rx-vlan-stag-hw-parse: off [fixed]
rx-vlan-stag-filter: off [fixed]
l2-fwd-offload: off [fixed]
busy-poll: on [fixed]



./ver-linux:
Linux compute1vx 3.18.0-11-generic #12-Ubuntu SMP Wed Jan 28 18:16:44
EET 2015 x86_64 x86_64 x86_64 GNU/Linux

Gnu C                  4.8
Gnu make               3.81
binutils               2.24
util-linux             2.20.1
mount                  support
module-init-tools      15
e2fsprogs              1.42.9
PPP                    2.4.5
Linux C Library        2.19
Dynamic linker (ldd)   2.19
Procps                 3.3.9
Net-tools              1.60
Kbd                    1.15.5
Sh-utils               8.21
wireless-tools         30
Modules Loaded         ip_gre ip_tunnel vhost_net vhost macvtap macvlan
nf_conntrack_ipv6 nf_defrag_ipv6 xt_mac xt_physdev br_netfilter xt_set
iptable_raw ip_set_hash_ip ip_set nfnetlink veth ipt_MASQUERADE
nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4
nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4
xt_CHECKSUM iptable_mangle xt_tcpudp bridge stp llc ip6table_filter
ip6_tables iptable_filter ip_tables ebtable_nat ebtables x_tables nbd
openvswitch geneve gre ib_iser rdma_cm iw_cm ib_cm ib_sa ib_mad ib_core
ib_addr iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi gpio_ich
dm_multipath scsi_dh radeon intel_powerclamp coretemp kvm_intel kvm
crct10dif_pclmul crc32_pclmul ttm ghash_clmulni_intel drm_kms_helper
aesni_intel aes_x86_64 drm lrw gf128mul glue_helper ablk_helper
i7core_edac ipmi_si edac_core lpc_ich 8250_fintek cryptd hpilo hpwdt
i2c_algo_bit serio_raw ipmi_msghandler acpi_power_meter mac_hid lp
parport mlx4_en vxlan ip6_udp_tunnel udp_tunnel hid_generic bnx2x
psmouse usbhid ptp pps_core hid mdio mlx4_core hpsa libcrc32c


/proc/cpuinfo:
processor       : 23
vendor_id       : GenuineIntel
cpu family      : 6
model           : 44
model name      : Intel(R) Xeon(R) CPU           L5640  @ 2.27GHz
stepping        : 2
microcode       : 0x1a
cpu MHz         : 1600.000
cache size      : 12288 KB
physical id     : 0
siblings        : 12
core id         : 1
cpu cores       : 6
apicid          : 3
initial apicid  : 3
fpu             : yes
fpu_exception   : yes
cpuid level     : 11
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge
mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe
syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good
nopl xtopology nonstop_tsc aperfmperf pni pclmulqdq dtes64 monitor
ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid dca sse4_1 sse4_2
popcnt aes lahf_lm ida arat epb dtherm tpr_shadow vnmi flexpriority ept vpid
bugs            :
bogomips        : 4533.56
clflush size    : 64
cache_alignment : 64
address sizes   : 40 bits physical, 48 bits virtual
power management:




/proc/modules:
ip_gre 18295 0 - Live 0xffffffffc05e7000
ip_tunnel 24254 1 ip_gre, Live 0xffffffffc05f3000
vhost_net 18104 1 - Live 0xffffffffc05ed000
vhost 29047 1 vhost_net, Live 0xffffffffc05de000
macvtap 18255 1 vhost_net, Live 0xffffffffc05d8000
macvlan 23656 1 macvtap, Live 0xffffffffc05cd000
nf_conntrack_ipv6 18894 4 - Live 0xffffffffc05c3000
nf_defrag_ipv6 34841 1 nf_conntrack_ipv6, Live 0xffffffffc05b5000
xt_mac 12492 1 - Live 0xffffffffc05b0000
xt_physdev 12587 10 - Live 0xffffffffc05ab000
br_netfilter 17878 1 xt_physdev, Live 0xffffffffc05a5000
xt_set 13314 1 - Live 0xffffffffc0596000
iptable_raw 12678 1 - Live 0xffffffffc0585000
ip_set_hash_ip 31394 1 - Live 0xffffffffc059c000
ip_set 41059 2 xt_set,ip_set_hash_ip, Live 0xffffffffc058a000
nfnetlink 14725 1 ip_set, Live 0xffffffffc0580000
veth 13376 0 - Live 0xffffffffc057b000
ipt_MASQUERADE 12678 3 - Live 0xffffffffc0576000
nf_nat_masquerade_ipv4 13412 1 ipt_MASQUERADE, Live 0xffffffffc0571000
iptable_nat 12875 1 - Live 0xffffffffc0567000
nf_nat_ipv4 14115 1 iptable_nat, Live 0xffffffffc056c000
nf_nat 22050 2 nf_nat_masquerade_ipv4,nf_nat_ipv4, Live 0xffffffffc0560000
nf_conntrack_ipv4 14806 6 - Live 0xffffffffc054f000
nf_defrag_ipv4 12758 1 nf_conntrack_ipv4, Live 0xffffffffc054a000
xt_conntrack 12760 9 - Live 0xffffffffc0557000
nf_conntrack 105074 6
nf_conntrack_ipv6,nf_nat_masquerade_ipv4,nf_nat_ipv4,nf_nat,nf_conntrack_ipv4,xt_conntrack,
Live 0xffffffffc052f000
ipt_REJECT 12541 2 - Live 0xffffffffc052a000
nf_reject_ipv4 13183 1 ipt_REJECT, Live 0xffffffffc0525000
xt_CHECKSUM 12549 1 - Live 0xffffffffc051b000
iptable_mangle 12695 1 - Live 0xffffffffc0516000
xt_tcpudp 12884 11 - Live 0xffffffffc0520000
bridge 108463 1 br_netfilter, Live 0xffffffffc04fa000
stp 12976 1 bridge, Live 0xffffffffc04f5000
llc 14396 2 bridge,stp, Live 0xffffffffc045d000
ip6table_filter 12815 1 - Live 0xffffffffc0430000
ip6_tables 27026 1 ip6table_filter, Live 0xffffffffc04e9000
iptable_filter 12810 1 - Live 0xffffffffc0334000
ip_tables 27240 4 iptable_raw,iptable_nat,iptable_mangle,iptable_filter,
Live 0xffffffffc044d000
ebtable_nat 12807 0 - Live 0xffffffffc024c000
ebtables 35009 1 ebtable_nat, Live 0xffffffffc0426000
x_tables 34059 15
xt_mac,xt_physdev,xt_set,iptable_raw,ipt_MASQUERADE,xt_conntrack,ipt_REJECT,xt_CHECKSUM,iptable_mangle,xt_tcpudp,ip6table_filter,ip6_tables,iptable_filter,ip_tables,ebtables,
Live 0xffffffffc0306000
nbd 17642 0 - Live 0xffffffffc0252000
openvswitch 79201 0 - Live 0xffffffffc04d4000
geneve 13338 1 openvswitch, Live 0xffffffffc0247000


             [20/1876]
gre 13796 2 ip_gre,openvswitch, Live 0xffffffffc0293000
ib_iser 51896 0 - Live 0xffffffffc0418000
rdma_cm 43465 1 ib_iser, Live 0xffffffffc040c000
iw_cm 36940 1 rdma_cm, Live 0xffffffffc035c000
ib_cm 42689 1 rdma_cm, Live 0xffffffffc0350000
ib_sa 33950 2 rdma_cm,ib_cm, Live 0xffffffffc032a000
ib_mad 47486 2 ib_cm,ib_sa, Live 0xffffffffc02df000
ib_core 88311 6 ib_iser,rdma_cm,iw_cm,ib_cm,ib_sa,ib_mad, Live
0xffffffffc0313000
ib_addr 18923 2 rdma_cm,ib_core, Live 0xffffffffc02d9000
iscsi_tcp 18333 0 - Live 0xffffffffc0280000
libiscsi_tcp 25146 1 iscsi_tcp, Live 0xffffffffc02a4000
libiscsi 57233 3 ib_iser,iscsi_tcp,libiscsi_tcp, Live 0xffffffffc02f3000
scsi_transport_iscsi 99909 4 ib_iser,iscsi_tcp,libiscsi, Live
0xffffffffc02bf000
gpio_ich 13586 0 - Live 0xffffffffc036a000
dm_multipath 22843 0 - Live 0xffffffffc02ec000
scsi_dh 14882 1 dm_multipath, Live 0xffffffffc028e000
radeon 1552388 1 - Live 0xffffffffc067b000
intel_powerclamp 18823 0 - Live 0xffffffffc0675000
coretemp 13441 0 - Live 0xffffffffc0173000
kvm_intel 148362 4 - Live 0xffffffffc0a54000
kvm 462696 1 kvm_intel, Live 0xffffffffc0462000
crct10dif_pclmul 14307 0 - Live 0xffffffffc0178000
crc32_pclmul 13133 0 - Live 0xffffffffc0458000
ttm 85166 1 radeon, Live 0xffffffffc0437000
ghash_clmulni_intel 13230 0 - Live 0xffffffffc011a000
drm_kms_helper 98431 1 radeon, Live 0xffffffffc03be000
aesni_intel 169590 0 - Live 0xffffffffc03e1000
aes_x86_64 17131 1 aesni_intel, Live 0xffffffffc03db000
drm 317626 4 radeon,ttm,drm_kms_helper, Live 0xffffffffc036f000
lrw 13286 1 aesni_intel, Live 0xffffffffc02ba000
gf128mul 14951 1 lrw, Live 0xffffffffc033b000
glue_helper 13990 1 aesni_intel, Live 0xffffffffc0289000
ablk_helper 13597 1 aesni_intel, Live 0xffffffffc02b5000
i7core_edac 24139 0 - Live 0xffffffffc029d000
ipmi_si 53386 0 - Live 0xffffffffc0341000
edac_core 51908 2 i7core_edac, Live 0xffffffffc025b000
lpc_ich 21093 0 - Live 0xffffffffc0269000
8250_fintek 12925 0 - Live 0xffffffffc011f000
cryptd 20359 3 ghash_clmulni_intel,aesni_intel,ablk_helper, Live
0xffffffffc02af000
hpilo 17394 0 - Live 0xffffffffc027a000
hpwdt 14257 0 - Live 0xffffffffc0275000
i2c_algo_bit 13413 1 radeon, Live 0xffffffffc0270000
serio_raw 13483 0 - Live 0xffffffffc0298000
ipmi_msghandler 45318 1 ipmi_si, Live 0xffffffffc0147000
acpi_power_meter 18075 0 - Live 0xffffffffc00f0000
mac_hid 13227 0 - Live 0xffffffffc0081000
lp 17759 0 - Live 0xffffffffc0087000
parport 42348 1 lp, Live 0xffffffffc0075000
mlx4_en 90408 0 - Live 0xffffffffc022f000
vxlan 37128 2 openvswitch,mlx4_en, Live 0xffffffffc013c000
ip6_udp_tunnel 12755 2 geneve,vxlan, Live 0xffffffffc00e8000
udp_tunnel 13187 2 geneve,vxlan, Live 0xffffffffc00a5000
hid_generic 12559 0 - Live 0xffffffffc00f9000
bnx2x 722154 0 - Live 0xffffffffc017d000
psmouse 111586 0 - Live 0xffffffffc0156000
usbhid 52657 0 - Live 0xffffffffc012e000
ptp 19395 2 mlx4_en,bnx2x, Live 0xffffffffc0124000
pps_core 19382 1 ptp, Live 0xffffffffc006f000
hid 110426 2 hid_generic,usbhid, Live 0xffffffffc00fe000
mdio 13561 1 bnx2x, Live 0xffffffffc006a000
mlx4_core 245682 1 mlx4_en, Live 0xffffffffc00ab000
hpsa 81092 2 - Live 0xffffffffc0090000
libcrc32c 12644 2 openvswitch,bnx2x, Live 0xffffffffc0065000
-- 
System Administrator
https://skyatlas.com/




[-- Attachment #2: bnx2x-tx-on-netcat-connection.pcap --]
[-- Type: application/vnd.tcpdump.pcap, Size: 822 bytes --]

Re: PROBLEM: Bnx2x Checksum/Length Error Over GRE Tunnel

[Eric Dumazet]

On Tue, 2015-02-03 at 20:05 +0200, Eren Türkay wrote:
> Hello,
> 
> I am having incorrect checksum and length calculation error when using
> GRE tunnel and this leads to unstable/unreliable connections where even
> simple nc connection cannot be made correctly due to a lot of TCP
> retransmissions. The reported iperf output becomes 23Kbit/s.
> 
> When I disable tx checksumming with "ethtool -K eth0 tx off", the
> problem seems to be solved but then I lose tcp segmentation offloading
> support, without which adds an additional overhead on 10Gbit network
> interface. With tx off, I get 4 to 7Gbit/s with parallel 40 connections
> using iperf.
> 
> In tcpdump output (tcpdump -i em1 proto gre) when tx checksumming is
> enabled, I see "checksum error" in every packet + "IP truncated-ip -
> 63631 bytes missing!" messages, and a lot of TCP
> retransmissions. When tx is off, checksums are correct but I get 4-7Gbit
> output. With GSO generalization code in [0] it is reported that
> 9.3Gbit/s is possible over GRE tunnel but I'm nowhere near these results.

Strange, I am using bnx2x with no such problem.

Please note that tcpdump can display "checksum error" because it doesn't
always know the NIC will generate the checksums.

Can you please send : ifconfig -a, as I suspect an MTU issue

Here I easily reach line rate with a single flow.

ip tunnel add gre1 mode gre remote 10.246.11.52 local 10.246.11.51
ip link set gre1 up
ip addr add 7.7.8.51/24 dev gre1


lpk51:~# ip ro get 7.7.8.52
7.7.8.52 dev gre1  src 7.7.8.51 
    cache  expires 557sec mtu 1476

lpk51:~# ./netperf -H 7.7.8.52 -Cc
MIGRATED TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 7.7.8.52 () port 0 AF_INET
Recv   Send    Send                          Utilization       Service Demand
Socket Socket  Message  Elapsed              Send     Recv     Send    Recv
Size   Size    Size     Time     Throughput  local    remote   local   remote
bytes  bytes   bytes    secs.    10^6bits/s  % S      % S      us/KB   us/KB

 87380  16384  16384    10.00      9227.94   2.52     3.96     0.715   1.125  

Re: PROBLEM: Bnx2x Checksum/Length Error Over GRE Tunnel

[Rick Jones]

On 02/03/2015 10:46 AM, Eric Dumazet wrote:
> On Tue, 2015-02-03 at 20:05 +0200, Eren Türkay wrote:
>> Hello,
>>
>> I am having incorrect checksum and length calculation error when using
>> GRE tunnel and this leads to unstable/unreliable connections where even
>> simple nc connection cannot be made correctly due to a lot of TCP
>> retransmissions. The reported iperf output becomes 23Kbit/s.
>>
>> When I disable tx checksumming with "ethtool -K eth0 tx off", the
>> problem seems to be solved but then I lose tcp segmentation offloading
>> support, without which adds an additional overhead on 10Gbit network
>> interface. With tx off, I get 4 to 7Gbit/s with parallel 40 connections
>> using iperf.
>>
>> In tcpdump output (tcpdump -i em1 proto gre) when tx checksumming is
>> enabled, I see "checksum error" in every packet + "IP truncated-ip -
>> 63631 bytes missing!" messages, and a lot of TCP
>> retransmissions. When tx is off, checksums are correct but I get 4-7Gbit
>> output. With GSO generalization code in [0] it is reported that
>> 9.3Gbit/s is possible over GRE tunnel but I'm nowhere near these results.
>
> Strange, I am using bnx2x with no such problem.

A pedantic question, but do the driver and firmware versions match 
between your setup and his?

rick

Re: PROBLEM: Bnx2x Checksum/Length Error Over GRE Tunnel

[Eren Türkay]

[-- Attachment #1: Type: text/plain, Size: 3498 bytes --]

On 03-02-2015 20:46, Eric Dumazet wrote:
> Can you please send : ifconfig -a, as I suspect an MTU issue

You are right. Previously I set gre interface MTU to 1500 and NIC interface had 9000 MTU.
I was able to get 3Gbit in a single flow, and 5-6Gbit in parallel. When I recreated the interfaces,
the MTU of gre was set 8976 and I can get 9.5Gbit in a single flow. The reason why I set 1500 on
gre interface was to mimic openvswitch setup as the main problem I have been investigating is VM
network performance problem (they have 3Gbit throughput). I have OpenStack installation using OVS with GRE
and those interfaces are set 1500 MTU by default. I tried to test iproute2 implementation with 1500.

With correct MTU setting in gre interface, it seems to work without checksumming and offloading
but I haven't tested my original OVS setting. I will tweak OVS bridge MTU settings and try to get
it work. If baremetal-to-baremetal works, I suspect there is an issue with OVS setup.
However, when I enable tx on, netperf cannot even connect.

"ifconfig -a" and "ovs-vsctl show" result is attached.

I am not sure if tx offloading will affect the VM performance. Regardless of OVS problem, there seems to be a
bug with segmentation offloading as it's only possible to use the interface with "tx off". I have compiled the
kernel with debugging on and I can enable debugging on related parts in the kernel for further information.

> Here I easily reach line rate with a single flow.
> 
> ip tunnel add gre1 mode gre remote 10.246.11.52 local 10.246.11.51
> ip link set gre1 up
> ip addr add 7.7.8.51/24 dev gre1

ip tunnel add gre1 mode gre remote 10.20.0.162 local 10.20.0.161
ip l set gre1 up
ip addr add 7.7.8.161/24 dev gre1

root@compute1vx:~# ip r get 7.7.8.162
7.7.8.162 dev gre1  src 7.7.8.161 
    cache  expires 539sec mtu 8976


root@compute1vx:~# ethtool -K em1 tx off
Actual changes:
tx-checksumming: off
        tx-checksum-ipv4: off
        tx-checksum-ipv6: off
tcp-segmentation-offload: off
        tx-tcp-segmentation: off [requested on]
        tx-tcp-ecn-segmentation: off [requested on]
        tx-tcp6-segmentation: off [requested on]

root@compute1vx:~# netperf -H 7.7.8.162 -Cc                                                                                                                                                                                                   
MIGRATED TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 7.7.8.162 () port 0 AF_INET : demo
Recv   Send    Send                          Utilization       Service Demand
Socket Socket  Message  Elapsed              Send     Recv     Send    Recv
Size   Size    Size     Time     Throughput  local    remote   local   remote
bytes  bytes   bytes    secs.    10^6bits/s  % S      % S      us/KB   us/KB

 87380  16384  16384    10.00      9534.00   4.05     4.39     0.835   0.906  


root@compute1vx:~# ethtool -K em1 tx on
Actual changes:
tx-checksumming: on
        tx-checksum-ipv4: on
        tx-checksum-ipv6: on
tcp-segmentation-offload: on
        tx-tcp-segmentation: on
        tx-tcp-ecn-segmentation: on
        tx-tcp6-segmentation: on

root@compute1vx:~# netperf -H 7.7.8.162 -Cc
establish control: are you sure there is a netserver listening on 7.7.8.162 at port 12865?
establish_control could not establish the control connection from 0.0.0.0 port 0 address family AF_UNSPEC to 7.7.8.162 port 12865 address family AF_INET

Regards,
Eren

-- 
System Administrator
https://skyatlas.com/

[-- Attachment #2: ifconfig-and-ovs-result.txt --]
[-- Type: text/plain, Size: 6306 bytes --]

root@compute1vx:~# ifconfig -a
br-int    Link encap:Ethernet  HWaddr 62:6e:a4:45:76:4d  
          inet6 addr: fe80::e897:f0ff:fee3:5a0d/64 Scope:Link
          UP BROADCAST RUNNING  MTU:1500  Metric:1
          RX packets:50 errors:0 dropped:0 overruns:0 frame:0
          TX packets:8 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:3856 (3.8 KB)  TX bytes:648 (648.0 B)

br-tun    Link encap:Ethernet  HWaddr 72:52:f7:60:bc:4a  
          inet6 addr: fe80::70d8:27ff:fe57:36fd/64 Scope:Link
          UP BROADCAST RUNNING  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:8 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:0 (0.0 B)  TX bytes:648 (648.0 B)

em1       Link encap:Ethernet  HWaddr 00:26:55:1f:9d:a0  
          inet addr:10.20.0.161  Bcast:10.20.0.255  Mask:255.255.255.0
          inet6 addr: fe80::226:55ff:fe1f:9da0/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:9000  Metric:1
          RX packets:2775967 errors:1204 dropped:0 overruns:1204 frame:0
          TX packets:19544604 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:309879554 (309.8 MB)  TX bytes:80709251505 (80.7 GB)
          Interrupt:42 Memory:fb000000-fb7fffff 

em2       Link encap:Ethernet  HWaddr 00:26:55:1f:9d:a4  
          inet addr:192.168.88.61  Bcast:192.168.88.255  Mask:255.255.255.0
          inet6 addr: fe80::226:55ff:fe1f:9da4/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:981996 errors:0 dropped:0 overruns:0 frame:0
          TX packets:504463 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:1423897061 (1.4 GB)  TX bytes:38829176 (38.8 MB)
          Interrupt:53 Memory:fa000000-fa7fffff 

gre0      Link encap:UNSPEC  HWaddr 00-00-00-00-32-36-3A-35-00-00-00-00-00-00-00-00  
          NOARP  MTU:1476  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

gre1      Link encap:UNSPEC  HWaddr 0A-14-00-A1-00-00-00-00-00-00-00-00-00-00-00-00  
          inet addr:7.7.8.161  P-t-P:7.7.8.161  Mask:255.255.255.0
          inet6 addr: fe80::5efe:a14:a1/64 Scope:Link
          UP POINTOPOINT RUNNING NOARP  MTU:8976  Metric:1
          RX packets:2541984 errors:0 dropped:0 overruns:0 frame:0
          TX packets:1704256 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:133877496 (133.8 MB)  TX bytes:78944172512 (78.9 GB)

gretap0   Link encap:Ethernet  HWaddr 00:00:00:00:00:00  
          BROADCAST MULTICAST  MTU:1462  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

lo        Link encap:Local Loopback  
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:15 errors:0 dropped:0 overruns:0 frame:0
          TX packets:15 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:1878 (1.8 KB)  TX bytes:1878 (1.8 KB)

ovs-system Link encap:Ethernet  HWaddr fa:69:1a:53:4f:73  
          BROADCAST MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

qbre90aa6bf-36 Link encap:Ethernet  HWaddr 3a:d6:2c:ba:09:57  
          inet6 addr: fe80::387c:4bff:fe67:904f/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:44 errors:0 dropped:0 overruns:0 frame:0
          TX packets:8 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:2760 (2.7 KB)  TX bytes:648 (648.0 B)

qvbe90aa6bf-36 Link encap:Ethernet  HWaddr 3a:d6:2c:ba:09:57  
          inet6 addr: fe80::38d6:2cff:feba:957/64 Scope:Link
          UP BROADCAST RUNNING PROMISC MULTICAST  MTU:1500  Metric:1
          RX packets:45 errors:0 dropped:0 overruns:0 frame:0
          TX packets:15 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:3466 (3.4 KB)  TX bytes:1206 (1.2 KB)

qvoe90aa6bf-36 Link encap:Ethernet  HWaddr d2:d7:77:9a:d9:1e  
          inet6 addr: fe80::d0d7:77ff:fe9a:d91e/64 Scope:Link
          UP BROADCAST RUNNING PROMISC MULTICAST  MTU:1500  Metric:1
          RX packets:15 errors:0 dropped:0 overruns:0 frame:0
          TX packets:45 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:1206 (1.2 KB)  TX bytes:3466 (3.4 KB)

virbr0    Link encap:Ethernet  HWaddr 8e:54:70:cb:8a:fc  
          inet addr:192.168.122.1  Bcast:192.168.122.255  Mask:255.255.255.0
          UP BROADCAST MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)


root@compute1vx:~# ovs-vsctl show
db3856c4-3af5-45af-9676-debc379a3e9a
    Bridge br-int
        fail_mode: secure
        Port patch-tun
            Interface patch-tun
                type: patch
                options: {peer=patch-int}
        Port br-int
            Interface br-int
                type: internal
        Port "qvoe90aa6bf-36"
            tag: 1
            Interface "qvoe90aa6bf-36"
    Bridge br-tun
        Port br-tun
            Interface br-tun
                type: internal
        Port patch-int
            Interface patch-int
                type: patch
                options: {peer=patch-tun}
        Port "gre-0a1400a2"
            Interface "gre-0a1400a2"
                type: gre
                options: {df_default="true", in_key=flow, local_ip="10.20.0.161", out_key=flow, remote_ip="10.20.0.162"}
    ovs_version: "2.0.2"

RE: PROBLEM: Bnx2x Checksum/Length Error Over GRE Tunnel

[Yuval Mintz]

> I am not sure if tx offloading will affect the VM performance. Regardless of OVS
> problem, there seems to be a bug with segmentation offloading as it's only
> possible to use the interface with "tx off". I have compiled the kernel with
> debugging on and I can enable debugging on related parts in the kernel for
> further information.
> 
> > Here I easily reach line rate with a single flow.
> >
> > ip tunnel add gre1 mode gre remote 10.246.11.52 local 10.246.11.51 ip
> > link set gre1 up ip addr add 7.7.8.51/24 dev gre1
> 
> ip tunnel add gre1 mode gre remote 10.20.0.162 local 10.20.0.161 ip l set gre1
> up ip addr add 7.7.8.161/24 dev gre1

Tried this on latest net-next [using iproute2] and I didn't manage to hit the
issue you've indicated.
I.e., traffic seemed to work fine on gre tunnels using both mtu sizes and
regardless of whether Tx-checksum-offload was enabled or disabled.

Is there anything else which is worth mentioning in your setup?

Thanks,
Yuval

Re: PROBLEM: Bnx2x Checksum/Length Error Over GRE Tunnel

[Eren Türkay]

On 05-02-2015 18:13, Yuval Mintz wrote:
> Tried this on latest net-next [using iproute2] and I didn't manage to hit the
> issue you've indicated.
> I.e., traffic seemed to work fine on gre tunnels using both mtu sizes and
> regardless of whether Tx-checksum-offload was enabled or disabled.

Interesting. Could it be a firmware issue?

To summarize my current situation, I cannot get the network work with tx on so I
need to disable tx. Using iproute2, tunnels with MTU 8976 works OK (getting
9.7Gbit/s in a single flow). When MTU is decreased to 1500, I can only get
3-4Gbit/s.

Being unable to use tx offloading is strange.

> Is there anything else which is worth mentioning in your setup?

For iproute2, I don't have anything special. I just create a tunnel bridge and
test it. For OVS part, I have openstack installation using neutron openvswitch
plugin, which is irrelevant for now. Openvswitch kernel module is loaded but
does it have an effect on iproute2? Please inform me if you need anything else.

Just to inform other people having the same issue, by increasing the MTU of OVS
bridges (br-int, tap, qvo, qbr, etc.) to 8950, setting br-tun ovs bridge to
8976, and instructing VMs to use 8950 MTU, I can now get 7.5Gbit from
VM-to-network node using GRE tunnel in a single flow, and 9.5Gbit using 2 flows.
I should add that I needed to enable MQ in libvirt.xml (using 4 queues).

> Thanks,
> Yuval

Regards,
Eren

-- 
System Administrator
https://skyatlas.com/

RE: PROBLEM: Bnx2x Checksum/Length Error Over GRE Tunnel

[Yuval Mintz]

> > Tried this on latest net-next [using iproute2] and I didn't manage to
> > hit the issue you've indicated.
> > I.e., traffic seemed to work fine on gre tunnels using both mtu sizes
> > and regardless of whether Tx-checksum-offload was enabled or disabled.
> 
> Interesting. Could it be a firmware issue?

All latest drivers use the same FW;  The difference is usually in management
FW, which should have no relevance to a feature such as this.

Care to share a dump [ethtool -d] and lspci [-vvv] outputs for the device?

Thanks,
Yuval