* Extreme slowness in IPIP tunnel when routing through kernel 3.18 and later
@ 2016-03-17 14:02 Patrick Boutilier
2016-03-17 16:47 ` Tom Herbert
2016-03-17 18:27 ` Jesse Gross
0 siblings, 2 replies; 6+ messages in thread
From: Patrick Boutilier @ 2016-03-17 14:02 UTC (permalink / raw)
To: netdev
[-- Attachment #1: Type: text/plain, Size: 3623 bytes --]
I have an IPIP tunnel setup between two hosts in different buildings.
The Linux router they route through causes extreme slowness in the
tunnel when running kernels from 3.18 on . tcpdump shows many cksum
errors which don't show up in the 3.17 and earlier kernels. Speed goes
back to normal when rx-checksumming and tx-checksumming are turned off
using ethtool on the Linux router . Would this be an effect of bulk
network packet transmission that was introduced in 3.18 or some other issue?
eth2
driver: igb
version: 5.2.15-k
firmware-version: 1.5.1
bus-info: 0000:06:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: no
eth6
driver: ixgbe
version: 3.19.1-k
firmware-version: 0x80000389
bus-info: 0000:41:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: no
Example of cksum from tcpdump:
Flags [.], cksum 0x2237 (incorrect -> 0xbb02), seq 424119200:424120648,
ack 1, win 115, options [nop,nop,TS val 303173784 ecr 3971495454],
length 1448
Results of nuttcp test with rx-checksumming and tx-checksumming on:
Testing upload inside tunnel
0.0625 MB / 5.00 sec = 0.1048 Mbps 79 retrans
0.0625 MB / 5.00 sec = 0.1048 Mbps 59 retrans
0.0625 MB / 5.00 sec = 0.1049 Mbps 27 retrans
0.2096 MB / 16.11 sec = 0.1092 Mbps 0 %TX 0 %RX 186 retrans
0.47 msRTT
Testing download inside tunnel
0.5000 MB / 5.00 sec = 0.8389 Mbps 112 retrans
2.6250 MB / 5.00 sec = 4.4040 Mbps 412 retrans
0.8750 MB / 5.00 sec = 1.4680 Mbps 156 retrans
4.0243 MB / 15.48 sec = 2.1809 Mbps 0 %TX 0 %RX 705 retrans
0.47 msRTT
Testing upload outside tunnel
423.0000 MB / 5.00 sec = 709.4949 Mbps 275 retrans
442.0625 MB / 5.00 sec = 741.7008 Mbps 53 retrans
418.5625 MB / 5.00 sec = 702.1329 Mbps 133 retrans
1287.0080 MB / 15.08 sec = 715.9252 Mbps 1 %TX 30 %RX 461 retrans
0.43 msRTT
Testing download outside tunnel
552.4375 MB / 5.00 sec = 926.8347 Mbps 26 retrans
555.1875 MB / 5.00 sec = 931.4499 Mbps 17 retrans
553.0000 MB / 5.00 sec = 927.7802 Mbps 16 retrans
1664.2149 MB / 15.03 sec = 928.6737 Mbps 9 %TX 5 %RX 60 retrans 0.46
msRTT
Results of nuttcp test with rx-checksumming and tx-checksumming off:
Testing upload inside tunnel
440.0000 MB / 5.00 sec = 738.0112 Mbps 117 retrans
433.0000 MB / 5.00 sec = 726.4892 Mbps 135 retrans
447.9375 MB / 5.00 sec = 751.5714 Mbps 112 retrans
1321.0069 MB / 15.04 sec = 736.7213 Mbps 3 %TX 21 %RX 364 retrans
0.51 msRTT
Testing download inside tunnel
494.0625 MB / 5.00 sec = 828.8973 Mbps 335 retrans
471.0000 MB / 5.00 sec = 790.2050 Mbps 219 retrans
469.8750 MB / 5.00 sec = 788.3174 Mbps 220 retrans
1435.1875 MB / 15.00 sec = 802.4203 Mbps 89 %TX 12 %RX 774 retrans
0.46 msRTT
Testing upload outside tunnel
431.2500 MB / 5.00 sec = 723.2808 Mbps 33 retrans
446.3750 MB / 5.00 sec = 748.9268 Mbps 10 retrans
475.4375 MB / 5.00 sec = 797.7044 Mbps 0 retrans
1355.0834 MB / 15.06 sec = 754.6438 Mbps 1 %TX 33 %RX 43 retrans
0.53 msRTT
Testing download outside tunnel
424.8125 MB / 5.00 sec = 712.7061 Mbps 102 retrans
456.5625 MB / 5.00 sec = 765.9872 Mbps 80 retrans
454.0000 MB / 5.00 sec = 761.6935 Mbps 116 retrans
1335.7823 MB / 15.01 sec = 746.6959 Mbps 4 %TX 5 %RX 298 retrans
0.55 msRTT
[-- Attachment #2: boutilpj.vcf --]
[-- Type: text/x-vcard, Size: 286 bytes --]
begin:vcard
fn:Patrick Boutilier
n:Boutilier;Patrick
org:;Nova Scotia Department of Education
adr:;;2021 Brunswick Street;Halifax;NS;B3K 2Y5;Canada
email;internet:boutilpj@ednet.ns.ca
title:WAN Communications Specialist
tel;work:902-424-6800
tel;fax:902-424-0874
version:2.1
end:vcard
^ permalink raw reply [flat|nested] 6+ messages in thread* Re: Extreme slowness in IPIP tunnel when routing through kernel 3.18 and later
2016-03-17 14:02 Extreme slowness in IPIP tunnel when routing through kernel 3.18 and later Patrick Boutilier
@ 2016-03-17 16:47 ` Tom Herbert
2016-03-17 20:30 ` Patrick Boutilier
2016-03-17 18:27 ` Jesse Gross
1 sibling, 1 reply; 6+ messages in thread
From: Tom Herbert @ 2016-03-17 16:47 UTC (permalink / raw)
To: Patrick Boutilier; +Cc: Linux Kernel Network Developers
On Thu, Mar 17, 2016 at 7:02 AM, Patrick Boutilier <boutilpj@ednet.ns.ca> wrote:
> I have an IPIP tunnel setup between two hosts in different buildings. The
> Linux router they route through causes extreme slowness in the tunnel when
> running kernels from 3.18 on . tcpdump shows many cksum errors which don't
> show up in the 3.17 and earlier kernels. Speed goes back to normal when
> rx-checksumming and tx-checksumming are turned off using ethtool on the
> Linux router . Would this be an effect of bulk network packet transmission
> that was introduced in 3.18 or some other issue?
>
If this is happening on a router then the only case where checksums
are pertinent should be if GSO/GRO is enabled. Please try disabling
these. Also, are these showing up as checksum errors on the receiver
(netstat -s).
>
> eth2
> driver: igb
> version: 5.2.15-k
> firmware-version: 1.5.1
> bus-info: 0000:06:00.0
> supports-statistics: yes
> supports-test: yes
> supports-eeprom-access: yes
> supports-register-dump: yes
> supports-priv-flags: no
>
>
> eth6
> driver: ixgbe
> version: 3.19.1-k
> firmware-version: 0x80000389
> bus-info: 0000:41:00.0
> supports-statistics: yes
> supports-test: yes
> supports-eeprom-access: yes
> supports-register-dump: yes
> supports-priv-flags: no
>
>
>
>
>
> Example of cksum from tcpdump:
>
> Flags [.], cksum 0x2237 (incorrect -> 0xbb02), seq 424119200:424120648, ack
> 1, win 115, options [nop,nop,TS val 303173784 ecr 3971495454], length 1448
>
>
>
>
>
> Results of nuttcp test with rx-checksumming and tx-checksumming on:
>
>
> Testing upload inside tunnel
>
>
>
> 0.0625 MB / 5.00 sec = 0.1048 Mbps 79 retrans
> 0.0625 MB / 5.00 sec = 0.1048 Mbps 59 retrans
> 0.0625 MB / 5.00 sec = 0.1049 Mbps 27 retrans
>
> 0.2096 MB / 16.11 sec = 0.1092 Mbps 0 %TX 0 %RX 186 retrans 0.47
> msRTT
>
>
>
> Testing download inside tunnel
>
>
>
> 0.5000 MB / 5.00 sec = 0.8389 Mbps 112 retrans
> 2.6250 MB / 5.00 sec = 4.4040 Mbps 412 retrans
> 0.8750 MB / 5.00 sec = 1.4680 Mbps 156 retrans
>
> 4.0243 MB / 15.48 sec = 2.1809 Mbps 0 %TX 0 %RX 705 retrans 0.47
> msRTT
>
>
> Testing upload outside tunnel
>
>
>
> 423.0000 MB / 5.00 sec = 709.4949 Mbps 275 retrans
> 442.0625 MB / 5.00 sec = 741.7008 Mbps 53 retrans
> 418.5625 MB / 5.00 sec = 702.1329 Mbps 133 retrans
>
> 1287.0080 MB / 15.08 sec = 715.9252 Mbps 1 %TX 30 %RX 461 retrans 0.43
> msRTT
>
>
>
> Testing download outside tunnel
>
>
>
> 552.4375 MB / 5.00 sec = 926.8347 Mbps 26 retrans
> 555.1875 MB / 5.00 sec = 931.4499 Mbps 17 retrans
> 553.0000 MB / 5.00 sec = 927.7802 Mbps 16 retrans
>
> 1664.2149 MB / 15.03 sec = 928.6737 Mbps 9 %TX 5 %RX 60 retrans 0.46
> msRTT
>
>
>
>
>
>
> Results of nuttcp test with rx-checksumming and tx-checksumming off:
>
>
>
> Testing upload inside tunnel
>
>
>
> 440.0000 MB / 5.00 sec = 738.0112 Mbps 117 retrans
> 433.0000 MB / 5.00 sec = 726.4892 Mbps 135 retrans
> 447.9375 MB / 5.00 sec = 751.5714 Mbps 112 retrans
>
> 1321.0069 MB / 15.04 sec = 736.7213 Mbps 3 %TX 21 %RX 364 retrans 0.51
> msRTT
>
>
>
> Testing download inside tunnel
>
>
>
> 494.0625 MB / 5.00 sec = 828.8973 Mbps 335 retrans
> 471.0000 MB / 5.00 sec = 790.2050 Mbps 219 retrans
> 469.8750 MB / 5.00 sec = 788.3174 Mbps 220 retrans
>
> 1435.1875 MB / 15.00 sec = 802.4203 Mbps 89 %TX 12 %RX 774 retrans 0.46
> msRTT
>
>
> Testing upload outside tunnel
>
>
>
> 431.2500 MB / 5.00 sec = 723.2808 Mbps 33 retrans
> 446.3750 MB / 5.00 sec = 748.9268 Mbps 10 retrans
> 475.4375 MB / 5.00 sec = 797.7044 Mbps 0 retrans
>
> 1355.0834 MB / 15.06 sec = 754.6438 Mbps 1 %TX 33 %RX 43 retrans 0.53
> msRTT
>
>
>
> Testing download outside tunnel
>
>
>
> 424.8125 MB / 5.00 sec = 712.7061 Mbps 102 retrans
> 456.5625 MB / 5.00 sec = 765.9872 Mbps 80 retrans
> 454.0000 MB / 5.00 sec = 761.6935 Mbps 116 retrans
>
> 1335.7823 MB / 15.01 sec = 746.6959 Mbps 4 %TX 5 %RX 298 retrans 0.55
> msRTT
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Extreme slowness in IPIP tunnel when routing through kernel 3.18 and later
2016-03-17 16:47 ` Tom Herbert
@ 2016-03-17 20:30 ` Patrick Boutilier
0 siblings, 0 replies; 6+ messages in thread
From: Patrick Boutilier @ 2016-03-17 20:30 UTC (permalink / raw)
To: Tom Herbert; +Cc: Linux Kernel Network Developers
[-- Attachment #1: Type: text/plain, Size: 4540 bytes --]
On 03/17/2016 01:47 PM, Tom Herbert wrote:
> On Thu, Mar 17, 2016 at 7:02 AM, Patrick Boutilier <boutilpj@ednet.ns.ca> wrote:
>> I have an IPIP tunnel setup between two hosts in different buildings. The
>> Linux router they route through causes extreme slowness in the tunnel when
>> running kernels from 3.18 on . tcpdump shows many cksum errors which don't
>> show up in the 3.17 and earlier kernels. Speed goes back to normal when
>> rx-checksumming and tx-checksumming are turned off using ethtool on the
>> Linux router . Would this be an effect of bulk network packet transmission
>> that was introduced in 3.18 or some other issue?
>>
> If this is happening on a router then the only case where checksums
> are pertinent should be if GSO/GRO is enabled. Please try disabling
> these. Also, are these showing up as checksum errors on the receiver
> (netstat -s).
Yes, turning off GSO/GRO also fixes the issue.
Not sure which value of netstat I am looking for.
Compiling a new kernel now with Jesse's patch to test. Will report back
in a bit.
>
>
>>
>> eth2
>> driver: igb
>> version: 5.2.15-k
>> firmware-version: 1.5.1
>> bus-info: 0000:06:00.0
>> supports-statistics: yes
>> supports-test: yes
>> supports-eeprom-access: yes
>> supports-register-dump: yes
>> supports-priv-flags: no
>>
>>
>> eth6
>> driver: ixgbe
>> version: 3.19.1-k
>> firmware-version: 0x80000389
>> bus-info: 0000:41:00.0
>> supports-statistics: yes
>> supports-test: yes
>> supports-eeprom-access: yes
>> supports-register-dump: yes
>> supports-priv-flags: no
>>
>>
>>
>>
>>
>> Example of cksum from tcpdump:
>>
>> Flags [.], cksum 0x2237 (incorrect -> 0xbb02), seq 424119200:424120648, ack
>> 1, win 115, options [nop,nop,TS val 303173784 ecr 3971495454], length 1448
>>
>>
>>
>>
>>
>> Results of nuttcp test with rx-checksumming and tx-checksumming on:
>>
>>
>> Testing upload inside tunnel
>>
>>
>>
>> 0.0625 MB / 5.00 sec = 0.1048 Mbps 79 retrans
>> 0.0625 MB / 5.00 sec = 0.1048 Mbps 59 retrans
>> 0.0625 MB / 5.00 sec = 0.1049 Mbps 27 retrans
>>
>> 0.2096 MB / 16.11 sec = 0.1092 Mbps 0 %TX 0 %RX 186 retrans 0.47
>> msRTT
>>
>>
>>
>> Testing download inside tunnel
>>
>>
>>
>> 0.5000 MB / 5.00 sec = 0.8389 Mbps 112 retrans
>> 2.6250 MB / 5.00 sec = 4.4040 Mbps 412 retrans
>> 0.8750 MB / 5.00 sec = 1.4680 Mbps 156 retrans
>>
>> 4.0243 MB / 15.48 sec = 2.1809 Mbps 0 %TX 0 %RX 705 retrans 0.47
>> msRTT
>>
>>
>> Testing upload outside tunnel
>>
>>
>>
>> 423.0000 MB / 5.00 sec = 709.4949 Mbps 275 retrans
>> 442.0625 MB / 5.00 sec = 741.7008 Mbps 53 retrans
>> 418.5625 MB / 5.00 sec = 702.1329 Mbps 133 retrans
>>
>> 1287.0080 MB / 15.08 sec = 715.9252 Mbps 1 %TX 30 %RX 461 retrans 0.43
>> msRTT
>>
>>
>>
>> Testing download outside tunnel
>>
>>
>>
>> 552.4375 MB / 5.00 sec = 926.8347 Mbps 26 retrans
>> 555.1875 MB / 5.00 sec = 931.4499 Mbps 17 retrans
>> 553.0000 MB / 5.00 sec = 927.7802 Mbps 16 retrans
>>
>> 1664.2149 MB / 15.03 sec = 928.6737 Mbps 9 %TX 5 %RX 60 retrans 0.46
>> msRTT
>>
>>
>>
>>
>>
>>
>> Results of nuttcp test with rx-checksumming and tx-checksumming off:
>>
>>
>>
>> Testing upload inside tunnel
>>
>>
>>
>> 440.0000 MB / 5.00 sec = 738.0112 Mbps 117 retrans
>> 433.0000 MB / 5.00 sec = 726.4892 Mbps 135 retrans
>> 447.9375 MB / 5.00 sec = 751.5714 Mbps 112 retrans
>>
>> 1321.0069 MB / 15.04 sec = 736.7213 Mbps 3 %TX 21 %RX 364 retrans 0.51
>> msRTT
>>
>>
>>
>> Testing download inside tunnel
>>
>>
>>
>> 494.0625 MB / 5.00 sec = 828.8973 Mbps 335 retrans
>> 471.0000 MB / 5.00 sec = 790.2050 Mbps 219 retrans
>> 469.8750 MB / 5.00 sec = 788.3174 Mbps 220 retrans
>>
>> 1435.1875 MB / 15.00 sec = 802.4203 Mbps 89 %TX 12 %RX 774 retrans 0.46
>> msRTT
>>
>>
>> Testing upload outside tunnel
>>
>>
>>
>> 431.2500 MB / 5.00 sec = 723.2808 Mbps 33 retrans
>> 446.3750 MB / 5.00 sec = 748.9268 Mbps 10 retrans
>> 475.4375 MB / 5.00 sec = 797.7044 Mbps 0 retrans
>>
>> 1355.0834 MB / 15.06 sec = 754.6438 Mbps 1 %TX 33 %RX 43 retrans 0.53
>> msRTT
>>
>>
>>
>> Testing download outside tunnel
>>
>>
>>
>> 424.8125 MB / 5.00 sec = 712.7061 Mbps 102 retrans
>> 456.5625 MB / 5.00 sec = 765.9872 Mbps 80 retrans
>> 454.0000 MB / 5.00 sec = 761.6935 Mbps 116 retrans
>>
>> 1335.7823 MB / 15.01 sec = 746.6959 Mbps 4 %TX 5 %RX 298 retrans 0.55
>> msRTT
[-- Attachment #2: boutilpj.vcf --]
[-- Type: text/x-vcard, Size: 286 bytes --]
begin:vcard
fn:Patrick Boutilier
n:Boutilier;Patrick
org:;Nova Scotia Department of Education
adr:;;2021 Brunswick Street;Halifax;NS;B3K 2Y5;Canada
email;internet:boutilpj@ednet.ns.ca
title:WAN Communications Specialist
tel;work:902-424-6800
tel;fax:902-424-0874
version:2.1
end:vcard
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Extreme slowness in IPIP tunnel when routing through kernel 3.18 and later
2016-03-17 14:02 Extreme slowness in IPIP tunnel when routing through kernel 3.18 and later Patrick Boutilier
2016-03-17 16:47 ` Tom Herbert
@ 2016-03-17 18:27 ` Jesse Gross
2016-03-17 21:13 ` Patrick Boutilier
1 sibling, 1 reply; 6+ messages in thread
From: Jesse Gross @ 2016-03-17 18:27 UTC (permalink / raw)
To: Patrick Boutilier; +Cc: Linux Kernel Network Developers
On Thu, Mar 17, 2016 at 7:02 AM, Patrick Boutilier <boutilpj@ednet.ns.ca> wrote:
> I have an IPIP tunnel setup between two hosts in different buildings. The
> Linux router they route through causes extreme slowness in the tunnel when
> running kernels from 3.18 on . tcpdump shows many cksum errors which don't
> show up in the 3.17 and earlier kernels. Speed goes back to normal when
> rx-checksumming and tx-checksumming are turned off using ethtool on the
> Linux router . Would this be an effect of bulk network packet transmission
> that was introduced in 3.18 or some other issue?
I just sent out a likely fix, would you mind testing it?
https://patchwork.ozlabs.org/patch/599213/
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Extreme slowness in IPIP tunnel when routing through kernel 3.18 and later
2016-03-17 18:27 ` Jesse Gross
@ 2016-03-17 21:13 ` Patrick Boutilier
2016-03-17 21:18 ` Jesse Gross
0 siblings, 1 reply; 6+ messages in thread
From: Patrick Boutilier @ 2016-03-17 21:13 UTC (permalink / raw)
To: Jesse Gross; +Cc: Linux Kernel Network Developers
[-- Attachment #1: Type: text/plain, Size: 792 bytes --]
On 03/17/2016 03:27 PM, Jesse Gross wrote:
> On Thu, Mar 17, 2016 at 7:02 AM, Patrick Boutilier <boutilpj@ednet.ns.ca> wrote:
>> I have an IPIP tunnel setup between two hosts in different buildings. The
>> Linux router they route through causes extreme slowness in the tunnel when
>> running kernels from 3.18 on . tcpdump shows many cksum errors which don't
>> show up in the 3.17 and earlier kernels. Speed goes back to normal when
>> rx-checksumming and tx-checksumming are turned off using ethtool on the
>> Linux router . Would this be an effect of bulk network packet transmission
>> that was introduced in 3.18 or some other issue?
>
> I just sent out a likely fix, would you mind testing it?
> https://patchwork.ozlabs.org/patch/599213/
>
Yes, that patch fixes the problem.
Thanks.
[-- Attachment #2: boutilpj.vcf --]
[-- Type: text/x-vcard, Size: 286 bytes --]
begin:vcard
fn:Patrick Boutilier
n:Boutilier;Patrick
org:;Nova Scotia Department of Education
adr:;;2021 Brunswick Street;Halifax;NS;B3K 2Y5;Canada
email;internet:boutilpj@ednet.ns.ca
title:WAN Communications Specialist
tel;work:902-424-6800
tel;fax:902-424-0874
version:2.1
end:vcard
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Extreme slowness in IPIP tunnel when routing through kernel 3.18 and later
2016-03-17 21:13 ` Patrick Boutilier
@ 2016-03-17 21:18 ` Jesse Gross
0 siblings, 0 replies; 6+ messages in thread
From: Jesse Gross @ 2016-03-17 21:18 UTC (permalink / raw)
To: Patrick Boutilier; +Cc: Linux Kernel Network Developers
On Thu, Mar 17, 2016 at 2:13 PM, Patrick Boutilier <boutilpj@ednet.ns.ca> wrote:
> On 03/17/2016 03:27 PM, Jesse Gross wrote:
>>
>> On Thu, Mar 17, 2016 at 7:02 AM, Patrick Boutilier <boutilpj@ednet.ns.ca>
>> wrote:
>>>
>>> I have an IPIP tunnel setup between two hosts in different buildings. The
>>> Linux router they route through causes extreme slowness in the tunnel
>>> when
>>> running kernels from 3.18 on . tcpdump shows many cksum errors which
>>> don't
>>> show up in the 3.17 and earlier kernels. Speed goes back to normal when
>>> rx-checksumming and tx-checksumming are turned off using ethtool on the
>>> Linux router . Would this be an effect of bulk network packet
>>> transmission
>>> that was introduced in 3.18 or some other issue?
>>
>>
>> I just sent out a likely fix, would you mind testing it?
>> https://patchwork.ozlabs.org/patch/599213/
>>
>
> Yes, that patch fixes the problem.
Great, thanks a lot for testing.
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2016-03-17 21:18 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2016-03-17 14:02 Extreme slowness in IPIP tunnel when routing through kernel 3.18 and later Patrick Boutilier
2016-03-17 16:47 ` Tom Herbert
2016-03-17 20:30 ` Patrick Boutilier
2016-03-17 18:27 ` Jesse Gross
2016-03-17 21:13 ` Patrick Boutilier
2016-03-17 21:18 ` Jesse Gross
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).