Re: [PATCH net] net/mlx5e: Fix stalled channels with threaded NAPI

[Josef Bacik <josef@toxicpanda.com>]

On Thu, Oct 22, 2026 at 02:38:41PM +0300, Tariq Toukan wrote:
> The arm doorbell carries the consumer counter, and the device tracks the
> last index it reported an event for. Arming while CQEs posted after the
> last event are still unconsumed generates an event right away.
>
> The affinity path depends on exactly that, since commit a8c2eb15797a
> ("net/mlx5e: Stop NAPI when irq balancer changes affinity").
>
> Event moderation can delay that event, but not drop it.
>
> So the root analysis doesn't match the mlx5 device behavior.

The CQ has a backlog much deeper than one poll, about 6900 CQEs in the
run in the changelog.  The thread polls 64, is off the IRQ's CPU,
completes and arms.  There are CQEs posted after the last event, so the
event comes right away, and the device now remembers the current
producer index.  The thread polls another 64, completes and arms again.
If nothing has arrived in between there is no CQE posted after the last
event, only older ones that are still unconsumed, so there is no event.
The rest of the backlog waits for the next arrival.

That matches the counters of the stalled channel.  It had 0 polls and 0
events for 150 seconds.  New connections then gave it 16 events, 16
polls and 1018 packets, so one poll of 64 per event.  It was armed the
whole time, it just took a new completion to get an event.  Then nothing
for 14 seconds, then 93 polls and 5917 packets, and it was empty.

Is that how the device behaves, no event on arm when everything that is
unconsumed was posted before the last event?  That part is my reading of
your description, I haven't got a way to check it.

> Is it a degradation? Did you have any "good"/working version?

I don't know.  I've only run threaded NAPI on these machines on the
7.3-rc5 based kernel, I have no older kernel that I know to be good.

> If arming really doesn't raise an event here, softirq NAPI would stall
> the same way when the IRQ affinity changes under load, for example by
> irqbalance.
> Was it seen there too? I believe not, so it must be something else
> causing the stall...

I haven't seen it there, and I haven't tried changing the affinity under
load.  If the above is right I wouldn't expect it.  With softirq the
path is taken once, that arm raises the event, the interrupt lands on
the new CPU, and from then on the check passes and the poll goes on
until the CQ is empty without arming again.  A NAPI thread takes the
path at every poll, so it needs a second event for a backlog that the
first one already covered.

I'll fix the changelog and the comment in v2 once I know I've got the
device's side of it right, and move the check into
mlx5e_channel_no_affinity_change() as Jakub asked.

Thanks,
Josef