* [PATCH net-next] net: microchip: lan743x : bidirectional throughput improvement
@ 2023-11-16 5:43 Vishvambar Panth S
2023-11-17 18:16 ` Simon Horman
` (2 more replies)
0 siblings, 3 replies; 5+ messages in thread
From: Vishvambar Panth S @ 2023-11-16 5:43 UTC (permalink / raw)
To: netdev, linux-kernel, kuba, jacob.e.keller
Cc: bryan.whitehead, UNGLinuxDriver, davem, edumazet, pabeni
The LAN743x/PCI11xxx DMA descriptors are always 4 dwords long, but the
device supports placing the descriptors in memory back to back or
reserving space in between them using its DMA_DESCRIPTOR_SPACE (DSPACE)
configurable hardware setting. Currently DSPACE is unnecessarily set to
match the host's L1 cache line size, resulting in space reserved in
between descriptors in most platforms and causing a suboptimal behavior
(single PCIe Mem transaction per descriptor). By changing the setting
to DSPACE=16 many descriptors can be packed in a single PCIe Mem
transaction resulting in a massive performance improvement in
bidirectional tests without any negative effects.
Tested and verified improvements on x64 PC and several ARM platforms
(typical data below)
Test setup 1: x64 PC with LAN7430 ---> x64 PC
iperf3 UDP bidirectional with DSPACE set to L1 CACHE Size:
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID][Role] Interval Transfer Bitrate
[ 5][TX-C] 0.00-10.00 sec 170 MBytes 143 Mbits/sec sender
[ 5][TX-C] 0.00-10.04 sec 169 MBytes 141 Mbits/sec receiver
[ 7][RX-C] 0.00-10.00 sec 1.02 GBytes 876 Mbits/sec sender
[ 7][RX-C] 0.00-10.04 sec 1.02 GBytes 870 Mbits/sec receiver
iperf3 UDP bidirectional with DSPACE set to 16 Bytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID][Role] Interval Transfer Bitrate
[ 5][TX-C] 0.00-10.00 sec 1.11 GBytes 956 Mbits/sec sender
[ 5][TX-C] 0.00-10.04 sec 1.11 GBytes 951 Mbits/sec receiver
[ 7][RX-C] 0.00-10.00 sec 1.10 GBytes 948 Mbits/sec sender
[ 7][RX-C] 0.00-10.04 sec 1.10 GBytes 942 Mbits/sec receiver
Test setup 2 : RK3399 with LAN7430 ---> x64 PC
RK3399 Spec:
The SOM-RK3399 is ARM module designed and developed by FriendlyElec.
Cores: 64-bit Dual Core Cortex-A72 + Quad Core Cortex-A53
Frequency: Cortex-A72(up to 2.0GHz), Cortex-A53(up to 1.5GHz)
PCIe: PCIe x4, compatible with PCIe 2.1, Dual operation mode
iperf3 UDP bidirectional with DSPACE set to L1 CACHE Size:
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID][Role] Interval Transfer Bitrate
[ 5][TX-C] 0.00-10.00 sec 534 MBytes 448 Mbits/sec sender
[ 5][TX-C] 0.00-10.05 sec 534 MBytes 446 Mbits/sec receiver
[ 7][RX-C] 0.00-10.00 sec 1.12 GBytes 961 Mbits/sec sender
[ 7][RX-C] 0.00-10.05 sec 1.11 GBytes 946 Mbits/sec receiver
iperf3 UDP bidirectional with DSPACE set to 16 Bytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID][Role] Interval Transfer Bitrate
[ 5][TX-C] 0.00-10.00 sec 966 MBytes 810 Mbits/sec sender
[ 5][TX-C] 0.00-10.04 sec 965 MBytes 806 Mbits/sec receiver
[ 7][RX-C] 0.00-10.00 sec 1.11 GBytes 956 Mbits/sec sender
[ 7][RX-C] 0.00-10.04 sec 1.07 GBytes 919 Mbits/sec receiver
Signed-off-by: Vishvambar Panth S <vishvambarpanth.s@microchip.com>
---
drivers/net/ethernet/microchip/lan743x_main.h | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/drivers/net/ethernet/microchip/lan743x_main.h b/drivers/net/ethernet/microchip/lan743x_main.h
index b648461787d2..be79cb0ae5af 100644
--- a/drivers/net/ethernet/microchip/lan743x_main.h
+++ b/drivers/net/ethernet/microchip/lan743x_main.h
@@ -1075,7 +1075,7 @@ struct lan743x_adapter {
#define DMA_DESCRIPTOR_SPACING_32 (32)
#define DMA_DESCRIPTOR_SPACING_64 (64)
#define DMA_DESCRIPTOR_SPACING_128 (128)
-#define DEFAULT_DMA_DESCRIPTOR_SPACING (L1_CACHE_BYTES)
+#define DEFAULT_DMA_DESCRIPTOR_SPACING (DMA_DESCRIPTOR_SPACING_16)
#define DMAC_CHANNEL_STATE_SET(start_bit, stop_bit) \
(((start_bit) ? 2 : 0) | ((stop_bit) ? 1 : 0))
--
2.25.1
^ permalink raw reply related [flat|nested] 5+ messages in thread
* Re: [PATCH net-next] net: microchip: lan743x : bidirectional throughput improvement
2023-11-16 5:43 [PATCH net-next] net: microchip: lan743x : bidirectional throughput improvement Vishvambar Panth S
@ 2023-11-17 18:16 ` Simon Horman
2023-11-17 23:51 ` Jacob Keller
2023-11-17 18:22 ` Florian Fainelli
2023-11-21 2:00 ` patchwork-bot+netdevbpf
2 siblings, 1 reply; 5+ messages in thread
From: Simon Horman @ 2023-11-17 18:16 UTC (permalink / raw)
To: Vishvambar Panth S
Cc: netdev, linux-kernel, kuba, jacob.e.keller, bryan.whitehead,
UNGLinuxDriver, davem, edumazet, pabeni
On Thu, Nov 16, 2023 at 11:13:50AM +0530, Vishvambar Panth S wrote:
> The LAN743x/PCI11xxx DMA descriptors are always 4 dwords long, but the
> device supports placing the descriptors in memory back to back or
> reserving space in between them using its DMA_DESCRIPTOR_SPACE (DSPACE)
> configurable hardware setting. Currently DSPACE is unnecessarily set to
> match the host's L1 cache line size, resulting in space reserved in
> between descriptors in most platforms and causing a suboptimal behavior
> (single PCIe Mem transaction per descriptor). By changing the setting
> to DSPACE=16 many descriptors can be packed in a single PCIe Mem
> transaction resulting in a massive performance improvement in
> bidirectional tests without any negative effects.
> Tested and verified improvements on x64 PC and several ARM platforms
> (typical data below)
>
> Test setup 1: x64 PC with LAN7430 ---> x64 PC
>
> iperf3 UDP bidirectional with DSPACE set to L1 CACHE Size:
> - - - - - - - - - - - - - - - - - - - - - - - - -
> [ ID][Role] Interval Transfer Bitrate
> [ 5][TX-C] 0.00-10.00 sec 170 MBytes 143 Mbits/sec sender
> [ 5][TX-C] 0.00-10.04 sec 169 MBytes 141 Mbits/sec receiver
> [ 7][RX-C] 0.00-10.00 sec 1.02 GBytes 876 Mbits/sec sender
> [ 7][RX-C] 0.00-10.04 sec 1.02 GBytes 870 Mbits/sec receiver
>
> iperf3 UDP bidirectional with DSPACE set to 16 Bytes
> - - - - - - - - - - - - - - - - - - - - - - - - -
> [ ID][Role] Interval Transfer Bitrate
> [ 5][TX-C] 0.00-10.00 sec 1.11 GBytes 956 Mbits/sec sender
> [ 5][TX-C] 0.00-10.04 sec 1.11 GBytes 951 Mbits/sec receiver
> [ 7][RX-C] 0.00-10.00 sec 1.10 GBytes 948 Mbits/sec sender
> [ 7][RX-C] 0.00-10.04 sec 1.10 GBytes 942 Mbits/sec receiver
>
> Test setup 2 : RK3399 with LAN7430 ---> x64 PC
>
> RK3399 Spec:
> The SOM-RK3399 is ARM module designed and developed by FriendlyElec.
> Cores: 64-bit Dual Core Cortex-A72 + Quad Core Cortex-A53
> Frequency: Cortex-A72(up to 2.0GHz), Cortex-A53(up to 1.5GHz)
> PCIe: PCIe x4, compatible with PCIe 2.1, Dual operation mode
>
> iperf3 UDP bidirectional with DSPACE set to L1 CACHE Size:
> - - - - - - - - - - - - - - - - - - - - - - - - -
> [ ID][Role] Interval Transfer Bitrate
> [ 5][TX-C] 0.00-10.00 sec 534 MBytes 448 Mbits/sec sender
> [ 5][TX-C] 0.00-10.05 sec 534 MBytes 446 Mbits/sec receiver
> [ 7][RX-C] 0.00-10.00 sec 1.12 GBytes 961 Mbits/sec sender
> [ 7][RX-C] 0.00-10.05 sec 1.11 GBytes 946 Mbits/sec receiver
>
> iperf3 UDP bidirectional with DSPACE set to 16 Bytes
> - - - - - - - - - - - - - - - - - - - - - - - - -
> [ ID][Role] Interval Transfer Bitrate
> [ 5][TX-C] 0.00-10.00 sec 966 MBytes 810 Mbits/sec sender
> [ 5][TX-C] 0.00-10.04 sec 965 MBytes 806 Mbits/sec receiver
> [ 7][RX-C] 0.00-10.00 sec 1.11 GBytes 956 Mbits/sec sender
> [ 7][RX-C] 0.00-10.04 sec 1.07 GBytes 919 Mbits/sec receiver
>
> Signed-off-by: Vishvambar Panth S <vishvambarpanth.s@microchip.com>
Thanks,
I think you should have included Jacob's Reviewed-by tag from
the previous posting of this patch [1].
And echoing his comments there, a very nice performance boost :)
Reviewed-by: Simon Horman <horms@kernel.org>
[1] https://lore.kernel.org/netdev/e5ffec56-5512-1acc-b85c-ac0771634c22@intel.com/
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [PATCH net-next] net: microchip: lan743x : bidirectional throughput improvement
2023-11-16 5:43 [PATCH net-next] net: microchip: lan743x : bidirectional throughput improvement Vishvambar Panth S
2023-11-17 18:16 ` Simon Horman
@ 2023-11-17 18:22 ` Florian Fainelli
2023-11-21 2:00 ` patchwork-bot+netdevbpf
2 siblings, 0 replies; 5+ messages in thread
From: Florian Fainelli @ 2023-11-17 18:22 UTC (permalink / raw)
To: Vishvambar Panth S, netdev, linux-kernel, kuba, jacob.e.keller
Cc: bryan.whitehead, UNGLinuxDriver, davem, edumazet, pabeni
On 11/15/23 21:43, Vishvambar Panth S wrote:
> The LAN743x/PCI11xxx DMA descriptors are always 4 dwords long, but the
> device supports placing the descriptors in memory back to back or
> reserving space in between them using its DMA_DESCRIPTOR_SPACE (DSPACE)
> configurable hardware setting. Currently DSPACE is unnecessarily set to
> match the host's L1 cache line size, resulting in space reserved in
> between descriptors in most platforms and causing a suboptimal behavior
> (single PCIe Mem transaction per descriptor). By changing the setting
> to DSPACE=16 many descriptors can be packed in a single PCIe Mem
> transaction resulting in a massive performance improvement in
> bidirectional tests without any negative effects.
> Tested and verified improvements on x64 PC and several ARM platforms
> (typical data below)
>
> Test setup 1: x64 PC with LAN7430 ---> x64 PC
>
> iperf3 UDP bidirectional with DSPACE set to L1 CACHE Size:
> - - - - - - - - - - - - - - - - - - - - - - - - -
> [ ID][Role] Interval Transfer Bitrate
> [ 5][TX-C] 0.00-10.00 sec 170 MBytes 143 Mbits/sec sender
> [ 5][TX-C] 0.00-10.04 sec 169 MBytes 141 Mbits/sec receiver
> [ 7][RX-C] 0.00-10.00 sec 1.02 GBytes 876 Mbits/sec sender
> [ 7][RX-C] 0.00-10.04 sec 1.02 GBytes 870 Mbits/sec receiver
>
> iperf3 UDP bidirectional with DSPACE set to 16 Bytes
> - - - - - - - - - - - - - - - - - - - - - - - - -
> [ ID][Role] Interval Transfer Bitrate
> [ 5][TX-C] 0.00-10.00 sec 1.11 GBytes 956 Mbits/sec sender
> [ 5][TX-C] 0.00-10.04 sec 1.11 GBytes 951 Mbits/sec receiver
> [ 7][RX-C] 0.00-10.00 sec 1.10 GBytes 948 Mbits/sec sender
> [ 7][RX-C] 0.00-10.04 sec 1.10 GBytes 942 Mbits/sec receiver
>
> Test setup 2 : RK3399 with LAN7430 ---> x64 PC
>
> RK3399 Spec:
> The SOM-RK3399 is ARM module designed and developed by FriendlyElec.
> Cores: 64-bit Dual Core Cortex-A72 + Quad Core Cortex-A53
> Frequency: Cortex-A72(up to 2.0GHz), Cortex-A53(up to 1.5GHz)
> PCIe: PCIe x4, compatible with PCIe 2.1, Dual operation mode
>
> iperf3 UDP bidirectional with DSPACE set to L1 CACHE Size:
> - - - - - - - - - - - - - - - - - - - - - - - - -
> [ ID][Role] Interval Transfer Bitrate
> [ 5][TX-C] 0.00-10.00 sec 534 MBytes 448 Mbits/sec sender
> [ 5][TX-C] 0.00-10.05 sec 534 MBytes 446 Mbits/sec receiver
> [ 7][RX-C] 0.00-10.00 sec 1.12 GBytes 961 Mbits/sec sender
> [ 7][RX-C] 0.00-10.05 sec 1.11 GBytes 946 Mbits/sec receiver
>
> iperf3 UDP bidirectional with DSPACE set to 16 Bytes
> - - - - - - - - - - - - - - - - - - - - - - - - -
> [ ID][Role] Interval Transfer Bitrate
> [ 5][TX-C] 0.00-10.00 sec 966 MBytes 810 Mbits/sec sender
> [ 5][TX-C] 0.00-10.04 sec 965 MBytes 806 Mbits/sec receiver
> [ 7][RX-C] 0.00-10.00 sec 1.11 GBytes 956 Mbits/sec sender
> [ 7][RX-C] 0.00-10.04 sec 1.07 GBytes 919 Mbits/sec receiver
>
> Signed-off-by: Vishvambar Panth S <vishvambarpanth.s@microchip.com>
Reviewed-by: Florian Fainelli <florian.fainelli@broadcom.com>
--
Florian
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [PATCH net-next] net: microchip: lan743x : bidirectional throughput improvement
2023-11-17 18:16 ` Simon Horman
@ 2023-11-17 23:51 ` Jacob Keller
0 siblings, 0 replies; 5+ messages in thread
From: Jacob Keller @ 2023-11-17 23:51 UTC (permalink / raw)
To: Simon Horman, Vishvambar Panth S
Cc: netdev, linux-kernel, kuba, bryan.whitehead, UNGLinuxDriver,
davem, edumazet, pabeni
On 11/17/2023 10:16 AM, Simon Horman wrote:
> On Thu, Nov 16, 2023 at 11:13:50AM +0530, Vishvambar Panth S wrote:
>> The LAN743x/PCI11xxx DMA descriptors are always 4 dwords long, but the
>> device supports placing the descriptors in memory back to back or
>> reserving space in between them using its DMA_DESCRIPTOR_SPACE (DSPACE)
>> configurable hardware setting. Currently DSPACE is unnecessarily set to
>> match the host's L1 cache line size, resulting in space reserved in
>> between descriptors in most platforms and causing a suboptimal behavior
>> (single PCIe Mem transaction per descriptor). By changing the setting
>> to DSPACE=16 many descriptors can be packed in a single PCIe Mem
>> transaction resulting in a massive performance improvement in
>> bidirectional tests without any negative effects.
>> Tested and verified improvements on x64 PC and several ARM platforms
>> (typical data below)
>>
>> Test setup 1: x64 PC with LAN7430 ---> x64 PC
>>
>> iperf3 UDP bidirectional with DSPACE set to L1 CACHE Size:
>> - - - - - - - - - - - - - - - - - - - - - - - - -
>> [ ID][Role] Interval Transfer Bitrate
>> [ 5][TX-C] 0.00-10.00 sec 170 MBytes 143 Mbits/sec sender
>> [ 5][TX-C] 0.00-10.04 sec 169 MBytes 141 Mbits/sec receiver
>> [ 7][RX-C] 0.00-10.00 sec 1.02 GBytes 876 Mbits/sec sender
>> [ 7][RX-C] 0.00-10.04 sec 1.02 GBytes 870 Mbits/sec receiver
>>
>> iperf3 UDP bidirectional with DSPACE set to 16 Bytes
>> - - - - - - - - - - - - - - - - - - - - - - - - -
>> [ ID][Role] Interval Transfer Bitrate
>> [ 5][TX-C] 0.00-10.00 sec 1.11 GBytes 956 Mbits/sec sender
>> [ 5][TX-C] 0.00-10.04 sec 1.11 GBytes 951 Mbits/sec receiver
>> [ 7][RX-C] 0.00-10.00 sec 1.10 GBytes 948 Mbits/sec sender
>> [ 7][RX-C] 0.00-10.04 sec 1.10 GBytes 942 Mbits/sec receiver
>>
>> Test setup 2 : RK3399 with LAN7430 ---> x64 PC
>>
>> RK3399 Spec:
>> The SOM-RK3399 is ARM module designed and developed by FriendlyElec.
>> Cores: 64-bit Dual Core Cortex-A72 + Quad Core Cortex-A53
>> Frequency: Cortex-A72(up to 2.0GHz), Cortex-A53(up to 1.5GHz)
>> PCIe: PCIe x4, compatible with PCIe 2.1, Dual operation mode
>>
>> iperf3 UDP bidirectional with DSPACE set to L1 CACHE Size:
>> - - - - - - - - - - - - - - - - - - - - - - - - -
>> [ ID][Role] Interval Transfer Bitrate
>> [ 5][TX-C] 0.00-10.00 sec 534 MBytes 448 Mbits/sec sender
>> [ 5][TX-C] 0.00-10.05 sec 534 MBytes 446 Mbits/sec receiver
>> [ 7][RX-C] 0.00-10.00 sec 1.12 GBytes 961 Mbits/sec sender
>> [ 7][RX-C] 0.00-10.05 sec 1.11 GBytes 946 Mbits/sec receiver
>>
>> iperf3 UDP bidirectional with DSPACE set to 16 Bytes
>> - - - - - - - - - - - - - - - - - - - - - - - - -
>> [ ID][Role] Interval Transfer Bitrate
>> [ 5][TX-C] 0.00-10.00 sec 966 MBytes 810 Mbits/sec sender
>> [ 5][TX-C] 0.00-10.04 sec 965 MBytes 806 Mbits/sec receiver
>> [ 7][RX-C] 0.00-10.00 sec 1.11 GBytes 956 Mbits/sec sender
>> [ 7][RX-C] 0.00-10.04 sec 1.07 GBytes 919 Mbits/sec receiver
>>
>> Signed-off-by: Vishvambar Panth S <vishvambarpanth.s@microchip.com>
>
> Thanks,
>
> I think you should have included Jacob's Reviewed-by tag from
> the previous posting of this patch [1].
>
> And echoing his comments there, a very nice performance boost :)
>
> Reviewed-by: Simon Horman <horms@kernel.org>
>
> [1] https://lore.kernel.org/netdev/e5ffec56-5512-1acc-b85c-ac0771634c22@intel.com/
For the record, this version also looks good to me!
Reviewed-by: Jacob Keller <jacob.e.keller@intel.com>
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [PATCH net-next] net: microchip: lan743x : bidirectional throughput improvement
2023-11-16 5:43 [PATCH net-next] net: microchip: lan743x : bidirectional throughput improvement Vishvambar Panth S
2023-11-17 18:16 ` Simon Horman
2023-11-17 18:22 ` Florian Fainelli
@ 2023-11-21 2:00 ` patchwork-bot+netdevbpf
2 siblings, 0 replies; 5+ messages in thread
From: patchwork-bot+netdevbpf @ 2023-11-21 2:00 UTC (permalink / raw)
To: Vishvambar Panth S
Cc: netdev, linux-kernel, kuba, jacob.e.keller, bryan.whitehead,
UNGLinuxDriver, davem, edumazet, pabeni
Hello:
This patch was applied to netdev/net-next.git (main)
by Jakub Kicinski <kuba@kernel.org>:
On Thu, 16 Nov 2023 11:13:50 +0530 you wrote:
> The LAN743x/PCI11xxx DMA descriptors are always 4 dwords long, but the
> device supports placing the descriptors in memory back to back or
> reserving space in between them using its DMA_DESCRIPTOR_SPACE (DSPACE)
> configurable hardware setting. Currently DSPACE is unnecessarily set to
> match the host's L1 cache line size, resulting in space reserved in
> between descriptors in most platforms and causing a suboptimal behavior
> (single PCIe Mem transaction per descriptor). By changing the setting
> to DSPACE=16 many descriptors can be packed in a single PCIe Mem
> transaction resulting in a massive performance improvement in
> bidirectional tests without any negative effects.
> Tested and verified improvements on x64 PC and several ARM platforms
> (typical data below)
>
> [...]
Here is the summary with links:
- [net-next] net: microchip: lan743x : bidirectional throughput improvement
https://git.kernel.org/netdev/net-next/c/45933b2db91b
You are awesome, thank you!
--
Deet-doot-dot, I am a bot.
https://korg.docs.kernel.org/patchwork/pwbot.html
^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2023-11-21 2:00 UTC | newest]
Thread overview: 5+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2023-11-16 5:43 [PATCH net-next] net: microchip: lan743x : bidirectional throughput improvement Vishvambar Panth S
2023-11-17 18:16 ` Simon Horman
2023-11-17 23:51 ` Jacob Keller
2023-11-17 18:22 ` Florian Fainelli
2023-11-21 2:00 ` patchwork-bot+netdevbpf
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).