From: Eric Dumazet <edumazet@google.com>
To: Stefan Wahren <wahrenst@gmx.net>
Cc: Jakub Kicinski <kuba@kernel.org>,
Neal Cardwell <ncardwell@google.com>,
Fabio Estevam <festevam@gmail.com>,
linux-imx@nxp.com, Stefan Wahren <stefan.wahren@chargebyte.com>,
Michael Heimpold <mhei@heimpold.de>,
netdev@vger.kernel.org
Subject: Re: iperf performance regression since Linux 5.18
Date: Mon, 9 Oct 2023 21:10:55 +0200 [thread overview]
Message-ID: <CANn89iKY58YSknzOzkEHxFu=C=1_p=pXGAHGo9ZkAfAGon9ayw@mail.gmail.com> (raw)
In-Reply-To: <7f31ddc8-9971-495e-a1f6-819df542e0af@gmx.net>
On Mon, Oct 9, 2023 at 8:58 PM Stefan Wahren <wahrenst@gmx.net> wrote:
>
> Hi,
> we recently switched on our ARM NXP i.MX6ULL based embedded device
> (Tarragon Master [1]) from an older kernel version to Linux 6.1. After
> that we noticed a measurable performance regression on the Ethernet
> interface (driver: fec, 100 Mbit link) while running iperf client on the
> device:
>
> BAD
>
> # iperf -t 10 -i 1 -c 192.168.1.129
> ------------------------------------------------------------
> Client connecting to 192.168.1.129, TCP port 5001
> TCP window size: 96.2 KByte (default)
> ------------------------------------------------------------
> [ 3] local 192.168.1.12 port 56022 connected with 192.168.1.129 port 5001
> [ ID] Interval Transfer Bandwidth
> [ 3] 0.0- 1.0 sec 9.88 MBytes 82.8 Mbits/sec
> [ 3] 1.0- 2.0 sec 9.62 MBytes 80.7 Mbits/sec
> [ 3] 2.0- 3.0 sec 9.75 MBytes 81.8 Mbits/sec
> [ 3] 3.0- 4.0 sec 9.62 MBytes 80.7 Mbits/sec
> [ 3] 4.0- 5.0 sec 9.62 MBytes 80.7 Mbits/sec
> [ 3] 5.0- 6.0 sec 9.62 MBytes 80.7 Mbits/sec
> [ 3] 6.0- 7.0 sec 9.50 MBytes 79.7 Mbits/sec
> [ 3] 7.0- 8.0 sec 9.75 MBytes 81.8 Mbits/sec
> [ 3] 8.0- 9.0 sec 9.62 MBytes 80.7 Mbits/sec
> [ 3] 9.0-10.0 sec 9.50 MBytes 79.7 Mbits/sec
> [ 3] 0.0-10.0 sec 96.5 MBytes 80.9 Mbits/sec
>
> GOOD
>
> # iperf -t 10 -i 1 -c 192.168.1.129
> ------------------------------------------------------------
> Client connecting to 192.168.1.129, TCP port 5001
> TCP window size: 96.2 KByte (default)
> ------------------------------------------------------------
> [ 3] local 192.168.1.12 port 54898 connected with 192.168.1.129 port 5001
> [ ID] Interval Transfer Bandwidth
> [ 3] 0.0- 1.0 sec 11.2 MBytes 94.4 Mbits/sec
> [ 3] 1.0- 2.0 sec 11.0 MBytes 92.3 Mbits/sec
> [ 3] 2.0- 3.0 sec 10.8 MBytes 90.2 Mbits/sec
> [ 3] 3.0- 4.0 sec 11.0 MBytes 92.3 Mbits/sec
> [ 3] 4.0- 5.0 sec 10.9 MBytes 91.2 Mbits/sec
> [ 3] 5.0- 6.0 sec 10.9 MBytes 91.2 Mbits/sec
> [ 3] 6.0- 7.0 sec 10.8 MBytes 90.2 Mbits/sec
> [ 3] 7.0- 8.0 sec 10.9 MBytes 91.2 Mbits/sec
> [ 3] 8.0- 9.0 sec 10.9 MBytes 91.2 Mbits/sec
> [ 3] 9.0-10.0 sec 10.9 MBytes 91.2 Mbits/sec
> [ 3] 0.0-10.0 sec 109 MBytes 91.4 Mbits/sec
>
> We were able to bisect this down to this commit:
>
> first bad commit: [65466904b015f6eeb9225b51aeb29b01a1d4b59c] tcp: adjust
> TSO packet sizes based on min_rtt
>
> Disabling this new setting via:
>
> echo 0 > /proc/sys/net/ipv4/tcp_tso_rtt_log
>
> confirm that this was the cause of the performance regression.
>
> Is it expected that the new default setting has such a performance impact?
Thanks for the report
Normally no. I guess you need to give us more details.
qdisc in use, MTU in use, congestion control in use, "ss -temoi dst
192.168.1.129 " output from sender side while the flow is running.
Note that reaching line rate on a TCP flow is always tricky,
regardless of what 'line rate' is.
I suspect an issue on the receiving side with larger GRO packets perhaps ?
You could try to limit GRO or TSO packet sizes to determine if this is
a driver issue.
(ip link set dev ethX gro_max_size XXXXX gso_max_size YYYYY)
>
> More information of the platform ...
>
> # ethtool -k eth0
> Features for eth0:
> rx-checksumming: on
> tx-checksumming: on
> tx-checksum-ipv4: on
> tx-checksum-ip-generic: off [fixed]
> tx-checksum-ipv6: on
> tx-checksum-fcoe-crc: off [fixed]
> tx-checksum-sctp: off [fixed]
> scatter-gather: on
> tx-scatter-gather: on
> tx-scatter-gather-fraglist: off [fixed]
> tcp-segmentation-offload: on
> tx-tcp-segmentation: on
> tx-tcp-ecn-segmentation: off [fixed]
> tx-tcp-mangleid-segmentation: off
> tx-tcp6-segmentation: off [fixed]
> generic-segmentation-offload: on
> generic-receive-offload: on
> large-receive-offload: off [fixed]
> rx-vlan-offload: on
> tx-vlan-offload: off [fixed]
> ntuple-filters: off [fixed]
> receive-hashing: off [fixed]
> highdma: off [fixed]
> rx-vlan-filter: off [fixed]
> vlan-challenged: off [fixed]
> tx-lockless: off [fixed]
> netns-local: off [fixed]
> tx-gso-robust: off [fixed]
> tx-fcoe-segmentation: off [fixed]
> tx-gre-segmentation: off [fixed]
> tx-gre-csum-segmentation: off [fixed]
> tx-ipxip4-segmentation: off [fixed]
> tx-ipxip6-segmentation: off [fixed]
> tx-udp_tnl-segmentation: off [fixed]
> tx-udp_tnl-csum-segmentation: off [fixed]
> tx-gso-partial: off [fixed]
> tx-tunnel-remcsum-segmentation: off [fixed]
> tx-sctp-segmentation: off [fixed]
> tx-esp-segmentation: off [fixed]
> tx-udp-segmentation: off [fixed]
> tx-gso-list: off [fixed]
> fcoe-mtu: off [fixed]
> tx-nocache-copy: off
> loopback: off [fixed]
> rx-fcs: off [fixed]
> rx-all: off [fixed]
> tx-vlan-stag-hw-insert: off [fixed]
> rx-vlan-stag-hw-parse: off [fixed]
> rx-vlan-stag-filter: off [fixed]
> l2-fwd-offload: off [fixed]
> hw-tc-offload: off [fixed]
> esp-hw-offload: off [fixed]
> esp-tx-csum-hw-offload: off [fixed]
> rx-udp_tunnel-port-offload: off [fixed]
> tls-hw-tx-offload: off [fixed]
> tls-hw-rx-offload: off [fixed]
> rx-gro-hw: off [fixed]
> tls-hw-record: off [fixed]
> rx-gro-list: off
> macsec-hw-offload: off [fixed]
> rx-udp-gro-forwarding: off
> hsr-tag-ins-offload: off [fixed]
> hsr-tag-rm-offload: off [fixed]
> hsr-fwd-offload: off [fixed]
> hsr-dup-offload: off [fixed]
>
> [1] -
> https://elixir.bootlin.com/linux/latest/source/arch/arm/boot/dts/nxp/imx/imx6ull-tarragon-master.dts
next prev parent reply other threads:[~2023-10-09 19:11 UTC|newest]
Thread overview: 20+ messages / expand[flat|nested] mbox.gz Atom feed top
2023-10-09 18:58 iperf performance regression since Linux 5.18 Stefan Wahren
2023-10-09 19:10 ` Eric Dumazet [this message]
2023-10-09 19:19 ` Neal Cardwell
2023-10-13 13:37 ` Stefan Wahren
2023-10-14 19:40 ` Neal Cardwell
2023-10-14 21:16 ` Dave Taht
2023-10-14 22:51 ` Eric Dumazet
2023-10-14 23:24 ` Neal Cardwell
2023-10-14 23:26 ` Stefan Wahren
2023-10-15 10:23 ` Stefan Wahren
2023-10-16 9:49 ` Eric Dumazet
2023-10-16 10:35 ` Eric Dumazet
2023-10-16 18:25 ` Stefan Wahren
2023-10-16 18:47 ` Eric Dumazet
2023-10-17 9:53 ` Stefan Wahren
2023-10-17 12:08 ` Eric Dumazet
2023-10-17 12:17 ` Stefan Wahren
2023-10-16 18:21 ` Stefan Wahren
2023-10-15 0:06 ` Stefan Wahren
2023-10-11 12:58 ` Stefan Wahren
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to='CANn89iKY58YSknzOzkEHxFu=C=1_p=pXGAHGo9ZkAfAGon9ayw@mail.gmail.com' \
--to=edumazet@google.com \
--cc=festevam@gmail.com \
--cc=kuba@kernel.org \
--cc=linux-imx@nxp.com \
--cc=mhei@heimpold.de \
--cc=ncardwell@google.com \
--cc=netdev@vger.kernel.org \
--cc=stefan.wahren@chargebyte.com \
--cc=wahrenst@gmx.net \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).