Re: [PATCH net-next] net: xilinx: axienet: Configure and report coalesce parameters in DMAengine flow

[Sean Anderson <sean.anderson@linux.dev>]

On 6/3/25 07:07, Gupta, Suraj wrote:
> [Public]
> 
>> -----Original Message-----
>> From: Sean Anderson <sean.anderson@linux.dev>
>> Sent: Saturday, May 31, 2025 2:15 AM
>> To: Gupta, Suraj <Suraj.Gupta2@amd.com>; andrew+netdev@lunn.ch;
>> davem@davemloft.net; edumazet@google.com; kuba@kernel.org;
>> pabeni@redhat.com; vkoul@kernel.org; Simek, Michal <michal.simek@amd.com>;
>> Pandey, Radhey Shyam <radhey.shyam.pandey@amd.com>; horms@kernel.org
>> Cc: netdev@vger.kernel.org; linux-arm-kernel@lists.infradead.org; linux-
>> kernel@vger.kernel.org; git (AMD-Xilinx) <git@amd.com>; Katakam, Harini
>> <harini.katakam@amd.com>
>> Subject: Re: [PATCH net-next] net: xilinx: axienet: Configure and report coalesce
>> parameters in DMAengine flow
>>
>> Caution: This message originated from an External Source. Use proper caution
>> when opening attachments, clicking links, or responding.
>>
>>
>> On 5/30/25 06:18, Gupta, Suraj wrote:
>> > [AMD Official Use Only - AMD Internal Distribution Only]
>> >
>> >> -----Original Message-----
>> >> From: Sean Anderson <sean.anderson@linux.dev>
>> >> Sent: Thursday, May 29, 2025 9:48 PM
>> >> To: Gupta, Suraj <Suraj.Gupta2@amd.com>; andrew+netdev@lunn.ch;
>> >> davem@davemloft.net; edumazet@google.com; kuba@kernel.org;
>> >> pabeni@redhat.com; vkoul@kernel.org; Simek, Michal
>> >> <michal.simek@amd.com>; Pandey, Radhey Shyam
>> >> <radhey.shyam.pandey@amd.com>; horms@kernel.org
>> >> Cc: netdev@vger.kernel.org; linux-arm-kernel@lists.infradead.org;
>> >> linux- kernel@vger.kernel.org; git (AMD-Xilinx) <git@amd.com>;
>> >> Katakam, Harini <harini.katakam@amd.com>
>> >> Subject: Re: [PATCH net-next] net: xilinx: axienet: Configure and
>> >> report coalesce parameters in DMAengine flow
>> >>
>> >> Caution: This message originated from an External Source. Use proper
>> >> caution when opening attachments, clicking links, or responding.
>> >>
>> >>
>> >> On 5/28/25 08:00, Gupta, Suraj wrote:
>> >> > [AMD Official Use Only - AMD Internal Distribution Only]
>> >> >
>> >> >> -----Original Message-----
>> >> >> From: Sean Anderson <sean.anderson@linux.dev>
>> >> >> Sent: Tuesday, May 27, 2025 9:47 PM
>> >> >> To: Gupta, Suraj <Suraj.Gupta2@amd.com>; andrew+netdev@lunn.ch;
>> >> >> davem@davemloft.net; edumazet@google.com; kuba@kernel.org;
>> >> >> pabeni@redhat.com; vkoul@kernel.org; Simek, Michal
>> >> >> <michal.simek@amd.com>; Pandey, Radhey Shyam
>> >> >> <radhey.shyam.pandey@amd.com>; horms@kernel.org
>> >> >> Cc: netdev@vger.kernel.org; linux-arm-kernel@lists.infradead.org;
>> >> >> linux- kernel@vger.kernel.org; git (AMD-Xilinx) <git@amd.com>;
>> >> >> Katakam, Harini <harini.katakam@amd.com>
>> >> >> Subject: Re: [PATCH net-next] net: xilinx: axienet: Configure and
>> >> >> report coalesce parameters in DMAengine flow
>> >> >>
>> >> >> Caution: This message originated from an External Source. Use
>> >> >> proper caution when opening attachments, clicking links, or responding.
>> >> >>
>> >> >>
>> >> >> On 5/25/25 06:22, Suraj Gupta wrote:
>> >> >> > Add support to configure / report interrupt coalesce count and
>> >> >> > delay via ethtool in DMAEngine flow.
>> >> >> > Netperf numbers are not good when using non-dmaengine default
>> >> >> > values, so tuned coalesce count and delay and defined separate
>> >> >> > default values in dmaengine flow.
>> >> >> >
>> >> >> > Netperf numbers and CPU utilisation change in DMAengine flow
>> >> >> > after introducing coalescing with default parameters:
>> >> >> > coalesce parameters:
>> >> >> >    Transfer type        Before(w/o coalescing)  After(with coalescing)
>> >> >> > TCP Tx, CPU utilisation%      925, 27                 941, 22
>> >> >> > TCP Rx, CPU utilisation%      607, 32                 741, 36
>> >> >> > UDP Tx, CPU utilisation%      857, 31                 960, 28
>> >> >> > UDP Rx, CPU utilisation%      762, 26                 783, 18
>> >> >> >
>> >> >> > Above numbers are observed with 4x Cortex-a53.
>> >> >>
>> >> >> How does this affect latency? I would expect these RX settings to
>> >> >> increase latency around 5-10x. I only use these settings with DIM
>> >> >> since it will disable coalescing during periods of light load for better latency.
>> >> >>
>> >> >> (of course the way to fix this in general is RSS or some other
>> >> >> method involving multiple queues).
>> >> >>
>> >> >
>> >> > I took values before NAPI addition in legacy flow (rx_threshold:
>> >> > 24, rx_usec: 50) as
>> >> reference. But netperf numbers were low with them, so tried tuning
>> >> both and selected the pair which gives good numbers.
>> >>
>> >> Yeah, but the reason is that you are trading latency for throughput.
>> >> There is only one queue, so when the interface is saturated you will
>> >> not get good latency anyway (since latency-sensitive packets will get
>> >> head-of-line blocked). But when activity is sparse you can good
>> >> latency if there is no coalescing. So I think coalescing should only
>> >> be used when there is a lot of traffic. Hence why I only adjusted the
>> >> settings once I implemented DIM. I think you should be able to
>> >> implement it by calling net_dim from axienet_dma_rx_cb, but it will not be as
>> efficient without NAPI.
>> >>
>> >
>> > Ok, got it. I'll keep default values used before NAPI in legacy flow (coalesce count:
>> 24, delay: 50) for both Tx and Rx and remove perf comparisons.
>>
>> Those settings are actually probably even worse for latency. I'd leave the settings at
>> 0/0 (coalescing disabled) to match the existing behavior. I think the perf comparisons
>> are helpful, especially for people who know they are going to be throughput-limited.
>>
>> My main point is that I think extending the dmaengine API to allow for DIM will have
>> practical benefits in reduced latency.
>>
> Sure, will implement DIM for both Tx and Rx in next version. However, I noticed it's implemented for Rx only in legacy flow. Is there any specific reason for that?

There's no latency issue with sending packets. It doesn't matter when we
process Tx completions as long as we refill the ring in time to send
more packets. So we can aggressively set the Tx coalescing for maximum
throughput.

--Sean