Re: [PATCH net] sfc: use budget for TX completions

[Jakub Kicinski <kuba@kernel.org>]

On Wed, 14 Jun 2023 09:03:05 +0100 Martin Habets wrote:
> On Mon, Jun 12, 2023 at 04:42:54PM +0200, Íñigo Huguet wrote:
> > When running workloads heavy unbalanced towards TX (high TX, low RX
> > traffic), sfc driver can retain the CPU during too long times. Although
> > in many cases this is not enough to be visible, it can affect
> > performance and system responsiveness.
> > 
> > A way to reproduce it is to use a debug kernel and run some parallel
> > netperf TX tests. In some systems, this will lead to this message being
> > logged:
> >   kernel:watchdog: BUG: soft lockup - CPU#12 stuck for 22s!
> > 
> > The reason is that sfc driver doesn't account any NAPI budget for the TX
> > completion events work. With high-TX/low-RX traffic, this makes that the
> > CPU is held for long time for NAPI poll.
> > 
> > Documentations says "drivers can process completions for any number of Tx
> > packets but should only process up to budget number of Rx packets".
> > However, many drivers do limit the amount of TX completions that they
> > process in a single NAPI poll.  
> 
> I think your work and what other drivers do shows that the documentation is
> no longer correct. I haven't checked when that was written, but maybe it
> was years ago when link speeds were lower.
> Clearly for drivers that support higher link speeds this is an issue, so we
> should update the documentation. Not sure what constitutes a high link speed,
> with current CPUs for me it's anything >= 50G.

The documentation is pretty recent. I haven't seen this lockup once 
in production or testing. Do multiple queues complete on the same CPU
for SFC or something weird like that?