* [QUESTION] poor TX performance on new GbE driver
@ 2017-10-22 19:14 Ard Biesheuvel
2017-10-22 19:27 ` Florian Fainelli
2017-10-22 19:38 ` Eric Dumazet
0 siblings, 2 replies; 5+ messages in thread
From: Ard Biesheuvel @ 2017-10-22 19:14 UTC (permalink / raw)
To: <netdev@vger.kernel.org>
Hello all,
I am working on upstreaming a network driver for a Socionext SoC, and
I am having some trouble figuring out why my TX performance is
horrible when booting a Debian Stretch rootfs, while booting a Ubuntu
17.04 rootfs works absolutely fine. Note that this is using the exact
same kernel image, booted off the network.
Under Ubuntu, I get the following iperf results from the box to my AMD
Seattle based devbox with a 1 Gbit switch in between. (The NIC in
question is also 1 Gbit)
$ sudo iperf -c dogfood.local -r
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 85.3 KByte (default)
------------------------------------------------------------
------------------------------------------------------------
Client connecting to dogfood.local, TCP port 5001
TCP window size: 748 KByte (default)
------------------------------------------------------------
[ 5] local 192.168.1.112 port 51666 connected with 192.168.1.106 port 5001
[ ID] Interval Transfer Bandwidth
[ 5] 0.0-10.0 sec 1.07 GBytes 920 Mbits/sec
[ 4] local 192.168.1.112 port 5001 connected with 192.168.1.106 port 33048
[ 4] 0.0-10.0 sec 1.10 GBytes 940 Mbits/sec
Booting the *exact* same kernel into a Debian based rootfs results in
the following numbers
$ sudo iperf -c dogfood.local -r
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 85.3 KByte (default)
------------------------------------------------------------
------------------------------------------------------------
Client connecting to dogfood.local, TCP port 5001
TCP window size: 85.0 KByte (default)
------------------------------------------------------------
[ 5] local 192.168.1.112 port 40132 connected with 192.168.1.106 port 5001
[ ID] Interval Transfer Bandwidth
[ 5] 0.0-10.1 sec 4.12 MBytes 3.43 Mbits/sec
[ 4] local 192.168.1.112 port 5001 connected with 192.168.1.106 port 33068
[ 4] 0.0-10.0 sec 1.10 GBytes 939 Mbits/sec
The ifconfig stats look perfectly fine to me (TX errors 0 dropped 0
overruns 0 carrier 0 collisions 0). During the TX test, the CPUs are
almost completely idle. (This system has 24 cores, but not
particularly powerful ones.)
This test is based on v4.14-rc4, but v4.13 gives the same results.
Could anyone please shed a light on this? What tuning parameters
and/or stats should I be looking at? I am a seasoned kernel developer
but a newbie when it comes to networking, so hopefully I just need a
nudge to go looking in the right place.
Thanks,
Ard.
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [QUESTION] poor TX performance on new GbE driver
2017-10-22 19:14 [QUESTION] poor TX performance on new GbE driver Ard Biesheuvel
@ 2017-10-22 19:27 ` Florian Fainelli
2017-10-22 19:34 ` Ard Biesheuvel
2017-10-22 19:38 ` Eric Dumazet
1 sibling, 1 reply; 5+ messages in thread
From: Florian Fainelli @ 2017-10-22 19:27 UTC (permalink / raw)
To: Ard Biesheuvel, <netdev@vger.kernel.org>
On 10/22/2017 12:14 PM, Ard Biesheuvel wrote:
> Hello all,
>
> I am working on upstreaming a network driver for a Socionext SoC, and
> I am having some trouble figuring out why my TX performance is
> horrible when booting a Debian Stretch rootfs, while booting a Ubuntu
> 17.04 rootfs works absolutely fine. Note that this is using the exact
> same kernel image, booted off the network.
>
> Under Ubuntu, I get the following iperf results from the box to my AMD
> Seattle based devbox with a 1 Gbit switch in between. (The NIC in
> question is also 1 Gbit)
>
>
> $ sudo iperf -c dogfood.local -r
> ------------------------------------------------------------
> Server listening on TCP port 5001
> TCP window size: 85.3 KByte (default)
> ------------------------------------------------------------
> ------------------------------------------------------------
> Client connecting to dogfood.local, TCP port 5001
> TCP window size: 748 KByte (default)
> ------------------------------------------------------------
> [ 5] local 192.168.1.112 port 51666 connected with 192.168.1.106 port 5001
> [ ID] Interval Transfer Bandwidth
> [ 5] 0.0-10.0 sec 1.07 GBytes 920 Mbits/sec
> [ 4] local 192.168.1.112 port 5001 connected with 192.168.1.106 port 33048
> [ 4] 0.0-10.0 sec 1.10 GBytes 940 Mbits/sec
>
> Booting the *exact* same kernel into a Debian based rootfs results in
> the following numbers
> $ sudo iperf -c dogfood.local -r
> ------------------------------------------------------------
> Server listening on TCP port 5001
> TCP window size: 85.3 KByte (default)
> ------------------------------------------------------------
> ------------------------------------------------------------
> Client connecting to dogfood.local, TCP port 5001
> TCP window size: 85.0 KByte (default)
> ------------------------------------------------------------
> [ 5] local 192.168.1.112 port 40132 connected with 192.168.1.106 port 5001
> [ ID] Interval Transfer Bandwidth
> [ 5] 0.0-10.1 sec 4.12 MBytes 3.43 Mbits/sec
> [ 4] local 192.168.1.112 port 5001 connected with 192.168.1.106 port 33068
> [ 4] 0.0-10.0 sec 1.10 GBytes 939 Mbits/sec
>
> The ifconfig stats look perfectly fine to me (TX errors 0 dropped 0
> overruns 0 carrier 0 collisions 0). During the TX test, the CPUs are
> almost completely idle. (This system has 24 cores, but not
> particularly powerful ones.)
>
> This test is based on v4.14-rc4, but v4.13 gives the same results.
>
> Could anyone please shed a light on this? What tuning parameters
> and/or stats should I be looking at? I am a seasoned kernel developer
> but a newbie when it comes to networking, so hopefully I just need a
> nudge to go looking in the right place.
You could look at /proc/net/snmp and see if you get higher level TCP/IP
drops. The second run appears to be fine, is it possible that somehow
your TX ring starts in a invalid state of some sort, TX activity cleans
it up during the first run and the second run operates under normal
condition? At first glance I can't think of any sensible difference
between the two rootfs that would explain what happens but it might be
worth comparing /proc/sys/net between the two and spot possible TCP
parameters differences.
How is UDP doing in your test cases? Once your image is loaded
everything should be in the page cache already so there should not be
any heavy NFS activity while you run your tests right? You might also
want to try to take a perf capture of the first run and see where and
how packets may be dropped: perf record -g -e skb:kfree_skb iperf -c ..
may help here.
--
Florian
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [QUESTION] poor TX performance on new GbE driver
2017-10-22 19:27 ` Florian Fainelli
@ 2017-10-22 19:34 ` Ard Biesheuvel
0 siblings, 0 replies; 5+ messages in thread
From: Ard Biesheuvel @ 2017-10-22 19:34 UTC (permalink / raw)
To: Florian Fainelli; +Cc: <netdev@vger.kernel.org>
On 22 October 2017 at 20:27, Florian Fainelli <f.fainelli@gmail.com> wrote:
> On 10/22/2017 12:14 PM, Ard Biesheuvel wrote:
>> Hello all,
>>
>> I am working on upstreaming a network driver for a Socionext SoC, and
>> I am having some trouble figuring out why my TX performance is
>> horrible when booting a Debian Stretch rootfs, while booting a Ubuntu
>> 17.04 rootfs works absolutely fine. Note that this is using the exact
>> same kernel image, booted off the network.
>>
>> Under Ubuntu, I get the following iperf results from the box to my AMD
>> Seattle based devbox with a 1 Gbit switch in between. (The NIC in
>> question is also 1 Gbit)
>>
>>
>> $ sudo iperf -c dogfood.local -r
>> ------------------------------------------------------------
>> Server listening on TCP port 5001
>> TCP window size: 85.3 KByte (default)
>> ------------------------------------------------------------
>> ------------------------------------------------------------
>> Client connecting to dogfood.local, TCP port 5001
>> TCP window size: 748 KByte (default)
>> ------------------------------------------------------------
>> [ 5] local 192.168.1.112 port 51666 connected with 192.168.1.106 port 5001
>> [ ID] Interval Transfer Bandwidth
>> [ 5] 0.0-10.0 sec 1.07 GBytes 920 Mbits/sec
>> [ 4] local 192.168.1.112 port 5001 connected with 192.168.1.106 port 33048
>> [ 4] 0.0-10.0 sec 1.10 GBytes 940 Mbits/sec
>>
>> Booting the *exact* same kernel into a Debian based rootfs results in
>> the following numbers
>> $ sudo iperf -c dogfood.local -r
>> ------------------------------------------------------------
>> Server listening on TCP port 5001
>> TCP window size: 85.3 KByte (default)
>> ------------------------------------------------------------
>> ------------------------------------------------------------
>> Client connecting to dogfood.local, TCP port 5001
>> TCP window size: 85.0 KByte (default)
>> ------------------------------------------------------------
>> [ 5] local 192.168.1.112 port 40132 connected with 192.168.1.106 port 5001
>> [ ID] Interval Transfer Bandwidth
>> [ 5] 0.0-10.1 sec 4.12 MBytes 3.43 Mbits/sec
>> [ 4] local 192.168.1.112 port 5001 connected with 192.168.1.106 port 33068
>> [ 4] 0.0-10.0 sec 1.10 GBytes 939 Mbits/sec
>>
>> The ifconfig stats look perfectly fine to me (TX errors 0 dropped 0
>> overruns 0 carrier 0 collisions 0). During the TX test, the CPUs are
>> almost completely idle. (This system has 24 cores, but not
>> particularly powerful ones.)
>>
>> This test is based on v4.14-rc4, but v4.13 gives the same results.
>>
>> Could anyone please shed a light on this? What tuning parameters
>> and/or stats should I be looking at? I am a seasoned kernel developer
>> but a newbie when it comes to networking, so hopefully I just need a
>> nudge to go looking in the right place.
Hi Florian,
Thanks for your response.
>
> You could look at /proc/net/snmp and see if you get higher level TCP/IP
> drops. The second run appears to be fine, is it possible that somehow
> your TX ring starts in a invalid state of some sort, TX activity cleans
> it up during the first run and the second run operates under normal
> condition?
The 'second' run is the opposite direction, due to the '-r' parameter.
I only included it for reference, since it works fine in both cases.
> At first glance I can't think of any sensible difference
> between the two rootfs that would explain what happens but it might be
> worth comparing /proc/sys/net between the two and spot possible TCP
> parameters differences.
>
Right, I can check that.
> How is UDP doing in your test cases? Once your image is loaded
> everything should be in the page cache already so there should not be
> any heavy NFS activity while you run your tests right?
I don't use NFS, only the kernel image is booted off the network, but
the rootfses are actually on SSDs
> You might also
> want to try to take a perf capture of the first run and see where and
> how packets may be dropped: perf record -g -e skb:kfree_skb iperf -c ..
> may help here.
OK, that looks interesting - let me try that.
Thanks a lot!
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [QUESTION] poor TX performance on new GbE driver
2017-10-22 19:14 [QUESTION] poor TX performance on new GbE driver Ard Biesheuvel
2017-10-22 19:27 ` Florian Fainelli
@ 2017-10-22 19:38 ` Eric Dumazet
2017-10-22 19:54 ` Ard Biesheuvel
1 sibling, 1 reply; 5+ messages in thread
From: Eric Dumazet @ 2017-10-22 19:38 UTC (permalink / raw)
To: Ard Biesheuvel; +Cc: <netdev@vger.kernel.org>
On Sun, 2017-10-22 at 20:14 +0100, Ard Biesheuvel wrote:
> Hello all,
>
> I am working on upstreaming a network driver for a Socionext SoC, and
> I am having some trouble figuring out why my TX performance is
> horrible when booting a Debian Stretch rootfs, while booting a Ubuntu
> 17.04 rootfs works absolutely fine. Note that this is using the exact
> same kernel image, booted off the network.
>
> Under Ubuntu, I get the following iperf results from the box to my AMD
> Seattle based devbox with a 1 Gbit switch in between. (The NIC in
> question is also 1 Gbit)
>
>
> $ sudo iperf -c dogfood.local -r
> ------------------------------------------------------------
> Server listening on TCP port 5001
> TCP window size: 85.3 KByte (default)
> ------------------------------------------------------------
> ------------------------------------------------------------
> Client connecting to dogfood.local, TCP port 5001
> TCP window size: 748 KByte (default)
> ------------------------------------------------------------
> [ 5] local 192.168.1.112 port 51666 connected with 192.168.1.106 port 5001
> [ ID] Interval Transfer Bandwidth
> [ 5] 0.0-10.0 sec 1.07 GBytes 920 Mbits/sec
> [ 4] local 192.168.1.112 port 5001 connected with 192.168.1.106 port 33048
> [ 4] 0.0-10.0 sec 1.10 GBytes 940 Mbits/sec
>
> Booting the *exact* same kernel into a Debian based rootfs results in
> the following numbers
> $ sudo iperf -c dogfood.local -r
> ------------------------------------------------------------
> Server listening on TCP port 5001
> TCP window size: 85.3 KByte (default)
> ------------------------------------------------------------
> ------------------------------------------------------------
> Client connecting to dogfood.local, TCP port 5001
> TCP window size: 85.0 KByte (default)
> ------------------------------------------------------------
> [ 5] local 192.168.1.112 port 40132 connected with 192.168.1.106 port 5001
> [ ID] Interval Transfer Bandwidth
> [ 5] 0.0-10.1 sec 4.12 MBytes 3.43 Mbits/sec
> [ 4] local 192.168.1.112 port 5001 connected with 192.168.1.106 port 33068
> [ 4] 0.0-10.0 sec 1.10 GBytes 939 Mbits/sec
>
> The ifconfig stats look perfectly fine to me (TX errors 0 dropped 0
> overruns 0 carrier 0 collisions 0). During the TX test, the CPUs are
> almost completely idle. (This system has 24 cores, but not
> particularly powerful ones.)
>
> This test is based on v4.14-rc4, but v4.13 gives the same results.
>
> Could anyone please shed a light on this? What tuning parameters
> and/or stats should I be looking at? I am a seasoned kernel developer
> but a newbie when it comes to networking, so hopefully I just need a
> nudge to go looking in the right place.
This description smells a problem with TX completions being deferred.
TX interrupts being lost or deferred too much.
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [QUESTION] poor TX performance on new GbE driver
2017-10-22 19:38 ` Eric Dumazet
@ 2017-10-22 19:54 ` Ard Biesheuvel
0 siblings, 0 replies; 5+ messages in thread
From: Ard Biesheuvel @ 2017-10-22 19:54 UTC (permalink / raw)
To: Eric Dumazet; +Cc: <netdev@vger.kernel.org>
On 22 October 2017 at 20:38, Eric Dumazet <eric.dumazet@gmail.com> wrote:
> On Sun, 2017-10-22 at 20:14 +0100, Ard Biesheuvel wrote:
>> Hello all,
>>
>> I am working on upstreaming a network driver for a Socionext SoC, and
>> I am having some trouble figuring out why my TX performance is
>> horrible when booting a Debian Stretch rootfs, while booting a Ubuntu
>> 17.04 rootfs works absolutely fine. Note that this is using the exact
>> same kernel image, booted off the network.
>>
>> Under Ubuntu, I get the following iperf results from the box to my AMD
>> Seattle based devbox with a 1 Gbit switch in between. (The NIC in
>> question is also 1 Gbit)
>>
>>
>> $ sudo iperf -c dogfood.local -r
>> ------------------------------------------------------------
>> Server listening on TCP port 5001
>> TCP window size: 85.3 KByte (default)
>> ------------------------------------------------------------
>> ------------------------------------------------------------
>> Client connecting to dogfood.local, TCP port 5001
>> TCP window size: 748 KByte (default)
>> ------------------------------------------------------------
>> [ 5] local 192.168.1.112 port 51666 connected with 192.168.1.106 port 5001
>> [ ID] Interval Transfer Bandwidth
>> [ 5] 0.0-10.0 sec 1.07 GBytes 920 Mbits/sec
>> [ 4] local 192.168.1.112 port 5001 connected with 192.168.1.106 port 33048
>> [ 4] 0.0-10.0 sec 1.10 GBytes 940 Mbits/sec
>>
>> Booting the *exact* same kernel into a Debian based rootfs results in
>> the following numbers
>> $ sudo iperf -c dogfood.local -r
>> ------------------------------------------------------------
>> Server listening on TCP port 5001
>> TCP window size: 85.3 KByte (default)
>> ------------------------------------------------------------
>> ------------------------------------------------------------
>> Client connecting to dogfood.local, TCP port 5001
>> TCP window size: 85.0 KByte (default)
>> ------------------------------------------------------------
>> [ 5] local 192.168.1.112 port 40132 connected with 192.168.1.106 port 5001
>> [ ID] Interval Transfer Bandwidth
>> [ 5] 0.0-10.1 sec 4.12 MBytes 3.43 Mbits/sec
>> [ 4] local 192.168.1.112 port 5001 connected with 192.168.1.106 port 33068
>> [ 4] 0.0-10.0 sec 1.10 GBytes 939 Mbits/sec
>>
>> The ifconfig stats look perfectly fine to me (TX errors 0 dropped 0
>> overruns 0 carrier 0 collisions 0). During the TX test, the CPUs are
>> almost completely idle. (This system has 24 cores, but not
>> particularly powerful ones.)
>>
>> This test is based on v4.14-rc4, but v4.13 gives the same results.
>>
>> Could anyone please shed a light on this? What tuning parameters
>> and/or stats should I be looking at? I am a seasoned kernel developer
>> but a newbie when it comes to networking, so hopefully I just need a
>> nudge to go looking in the right place.
>
>
> This description smells a problem with TX completions being deferred.
>
> TX interrupts being lost or deferred too much.
>
Right. Do you have any if there are any userland-accessible tunables
that may affect this? I'm still rather stumped that the issue only
appears when running the Debian Stretch rootfs ...
Thanks,
Ard.
^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2017-10-22 19:54 UTC | newest]
Thread overview: 5+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2017-10-22 19:14 [QUESTION] poor TX performance on new GbE driver Ard Biesheuvel
2017-10-22 19:27 ` Florian Fainelli
2017-10-22 19:34 ` Ard Biesheuvel
2017-10-22 19:38 ` Eric Dumazet
2017-10-22 19:54 ` Ard Biesheuvel
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.