Re: [PATCH net] net/mlx5e: Fix stalled channels with threaded NAPI

[Tariq Toukan <tariqt@nvidia.com>]

On 22/10/2026 15:56, Josef Bacik wrote:
> On Thu, Oct 22, 2026 at 02:38:41PM +0300, Tariq Toukan wrote:
>> The arm doorbell carries the consumer counter, and the device tracks the
>> last index it reported an event for. Arming while CQEs posted after the
>> last event are still unconsumed generates an event right away.
>>
>> The affinity path depends on exactly that, since commit a8c2eb15797a
>> ("net/mlx5e: Stop NAPI when irq balancer changes affinity").
>>
>> Event moderation can delay that event, but not drop it.
>>
>> So the root analysis doesn't match the mlx5 device behavior.
> 
> The CQ has a backlog much deeper than one poll, about 6900 CQEs in the
> run in the changelog.  The thread polls 64, is off the IRQ's CPU,
> completes and arms.  There are CQEs posted after the last event, so the
> event comes right away, and the device now remembers the current
> producer index.  The thread polls another 64, completes and arms again.
> If nothing has arrived in between there is no CQE posted after the last
> event, only older ones that are still unconsumed, so there is no event.
> The rest of the backlog waits for the next arrival.
> 
> That matches the counters of the stalled channel.  It had 0 polls and 0
> events for 150 seconds.  New connections then gave it 16 events, 16
> polls and 1018 packets, so one poll of 64 per event.  It was armed the
> whole time, it just took a new completion to get an event.  Then nothing
> for 14 seconds, then 93 polls and 5917 packets, and it was empty.
> 
> Is that how the device behaves, no event on arm when everything that is
> unconsumed was posted before the last event?  That part is my reading of
> your description, I haven't got a way to check it.
> 

I have to soften my previous reply: the immediate-event guarantee I 
quoted covers CQEs posted after the last event, so it does not rule out 
your scenario, where the leftover backlog was all reported by that 
event. Your reading is consistent with the documentation and with your 
counters.

I'm checking the exact behavior with our HW architects and will update 
here, hopefully within a few days.

>> Is it a degradation? Did you have any "good"/working version?
> 
> I don't know.  I've only run threaded NAPI on these machines on the
> 7.3-rc5 based kernel, I have no older kernel that I know to be good.
> 
>> If arming really doesn't raise an event here, softirq NAPI would stall
>> the same way when the IRQ affinity changes under load, for example by
>> irqbalance.
>> Was it seen there too? I believe not, so it must be something else
>> causing the stall...
> 
> I haven't seen it there, and I haven't tried changing the affinity under
> load.  If the above is right I wouldn't expect it.  With softirq the
> path is taken once, that arm raises the event, the interrupt lands on
> the new CPU, and from then on the check passes and the poll goes on
> until the CQ is empty without arming again.  A NAPI thread takes the
> path at every poll, so it needs a second event for a backlog that the
> first one already covered.
> 
> I'll fix the changelog and the comment in v2 once I know I've got the
> device's side of it right, and move the check into
> mlx5e_channel_no_affinity_change() as Jakub asked.
> 
> Thanks,
> Josef