[PATCH net] net/mlx5e: Fix stalled channels with threaded NAPI

[PATCH net] net/mlx5e: Fix stalled channels with threaded NAPI

[Josef Bacik]

With threaded NAPI an mlx5e channel can stop receiving when a burst of
traffic ends. It stays dead for minutes on an idle machine, while the
connections hashed to it time out.

When mlx5e_napi_poll() has more to do, it checks that it runs on a CPU of
its interrupt's affinity mask. If it does not, it takes the affinity to
have been changed: it completes the NAPI after at most budget - 1 packets
and arms the CQs, so that the next interrupt moves the poll to the new CPU.

A NAPI thread runs wherever the scheduler puts it, so with threaded NAPI a
busy queue takes this path at nearly every poll. That costs an interrupt
for every 64 packets. Worse, it counts on an interrupt for the CQEs that
are still in the CQ when it is armed, and none comes: the next event is
raised by the next new completion. While packets keep arriving this goes
unnoticed. When a flood stops, what is in the CQ stays there, and is worked
off one poll of 64 packets per new arrival. If the RQ has run dry as well,
arrivals are dropped (rx_out_of_buffer) and bring no completion either.

Seen on a 50 Gbit/s port (firmware 22.47.1088) with 63 channels, threaded
NAPI and rx-usecs 64: a CPU-bound task bound to each of the 128 CPUs and
32 Gbit/s of UDP in 8 flows for 7 seconds, then both stopped. After the
second round one channel had 0 polls and 0 events for 150 seconds on an
idle machine, while TCP connections hashed to it timed out and
rx_out_of_buffer counted their SYNs. Later arrivals took it 16 polls and
1018 packets further, it stood still again for 14 seconds, and another 93
polls and 5917 packets emptied it. In a longer test 22 of 48 iperf3 runs
failed with "unable to connect stream: Connection timed out", with no ping
to the same address lost.

i40e and iavf force an interrupt at this point, and mlx5e does for XSK.
Doing that here would cure the stall and keep the interrupt per 64 packets.

Fix by going on polling when the NAPI is threaded. With that, 8 of 8
rounds leave every channel alive, ch_aff_change stays 0, rx_out_of_buffer
does not move once the flood is over, none of the 48 runs fails, and the
interrupts taken during the flood halve.

Fixes: 29863d41bb6e ("net: implement threaded-able napi poll loop support")
Assisted-by: LLM
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
---
I hit this while comparing threaded NAPI with softirq on two bare metal
machines.  The runs in the changelog are from a 7.3-rc5 based kernel, with
and without this change.  The patch applies to net unchanged, there I've
only compiled it.

I don't have the PRM, so "arming the CQ raises no event for the CQEs that
are already there" is what the counters showed and not something I checked
against the documentation.
---
 drivers/net/ethernet/mellanox/mlx5/core/en_txrx.c | 9 ++++++++-
 1 file changed, 8 insertions(+), 1 deletion(-)

diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_txrx.c b/drivers/net/ethernet/mellanox/mlx5/core/en_txrx.c
index 185105606469..3142e90e68ce 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_txrx.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_txrx.c
@@ -207,7 +207,14 @@ int mlx5e_napi_poll(struct napi_struct *napi, int budget)
 				rq);
 
 	if (busy) {
-		if (likely(mlx5e_channel_no_affinity_change(c))) {
+		/* A NAPI thread runs where the scheduler puts it, which says
+		 * nothing about the affinity of the interrupt. Completing here
+		 * would leave the CQEs that are pending to the next interrupt,
+		 * and arming the CQs raises none for those, only for a new
+		 * completion.
+		 */
+		if (likely(mlx5e_channel_no_affinity_change(c)) ||
+		    test_bit(NAPI_STATE_THREADED, &napi->state)) {
 			work_done = budget;
 			goto out;
 		}

---
base-commit: abeacca61eea4f6803df4a595b656ea45c2d3249
change-id: 20261021-b4-mlx5e-threaded-napi-stall-7d0a14c7674d

Re: [PATCH net] net/mlx5e: Fix stalled channels with threaded NAPI

[sashiko-bot]

> With threaded NAPI an mlx5e channel can stop receiving when a burst of
> traffic ends. It stays dead for minutes on an idle machine, while the
> connections hashed to it time out.
> 
> When mlx5e_napi_poll() has more to do, it checks that it runs on a CPU of
> its interrupt's affinity mask. If it does not, it takes the affinity to
> have been changed: it completes the NAPI after at most budget - 1 packets
> and arms the CQs, so that the next interrupt moves the poll to the new CPU.
> 
> A NAPI thread runs wherever the scheduler puts it, so with threaded NAPI a
> busy queue takes this path at nearly every poll. That costs an interrupt
> for every 64 packets. Worse, it counts on an interrupt for the CQEs that
> are still in the CQ when it is armed, and none comes: the next event is
> raised by the next new completion. While packets keep arriving this goes
> unnoticed. When a flood stops, what is in the CQ stays there, and is worked
> [ ... ]
> interrupts taken during the flood halve.
> 
> Fixes: 29863d41bb6e ("net: implement threaded-able napi poll loop support")
> Assisted-by: LLM
> Signed-off-by: Josef Bacik <josef@toxicpanda.com>

Sashiko has reviewed this patch and found no issues.

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20261021-b4-mlx5e-threaded-napi-stall-v1-1-d719c17b3ec4@toxicpanda.com?part=1

Re: [PATCH net] net/mlx5e: Fix stalled channels with threaded NAPI

[Björn Töpel]

Josef Bacik <josef@toxicpanda.com> writes:

> With threaded NAPI an mlx5e channel can stop receiving when a burst of
> traffic ends. It stays dead for minutes on an idle machine, while the
> connections hashed to it time out.
>
> When mlx5e_napi_poll() has more to do, it checks that it runs on a CPU of
> its interrupt's affinity mask. If it does not, it takes the affinity to
> have been changed: it completes the NAPI after at most budget - 1 packets
> and arms the CQs, so that the next interrupt moves the poll to the new CPU.
>
> A NAPI thread runs wherever the scheduler puts it, so with threaded NAPI a
> busy queue takes this path at nearly every poll. That costs an interrupt
> for every 64 packets. Worse, it counts on an interrupt for the CQEs that
> are still in the CQ when it is armed, and none comes: the next event is
> raised by the next new completion. While packets keep arriving this goes
> unnoticed. When a flood stops, what is in the CQ stays there, and is worked
> off one poll of 64 packets per new arrival. If the RQ has run dry as well,
> arrivals are dropped (rx_out_of_buffer) and bring no completion either.
>
> Seen on a 50 Gbit/s port (firmware 22.47.1088) with 63 channels, threaded
> NAPI and rx-usecs 64: a CPU-bound task bound to each of the 128 CPUs and
> 32 Gbit/s of UDP in 8 flows for 7 seconds, then both stopped. After the
> second round one channel had 0 polls and 0 events for 150 seconds on an
> idle machine, while TCP connections hashed to it timed out and
> rx_out_of_buffer counted their SYNs. Later arrivals took it 16 polls and
> 1018 packets further, it stood still again for 14 seconds, and another 93
> polls and 5917 packets emptied it. In a longer test 22 of 48 iperf3 runs
> failed with "unable to connect stream: Connection timed out", with no ping
> to the same address lost.
>
> i40e and iavf force an interrupt at this point, and mlx5e does for XSK.
> Doing that here would cure the stall and keep the interrupt per 64 packets.
>
> Fix by going on polling when the NAPI is threaded. With that, 8 of 8
> rounds leave every channel alive, ch_aff_change stays 0, rx_out_of_buffer
> does not move once the flood is over, none of the 48 runs fails, and the
> interrupts taken during the flood halve.
>
> Fixes: 29863d41bb6e ("net: implement threaded-able napi poll loop support")
> Assisted-by: LLM
> Signed-off-by: Josef Bacik <josef@toxicpanda.com>
> ---
> I hit this while comparing threaded NAPI with softirq on two bare metal
> machines.  The runs in the changelog are from a 7.3-rc5 based kernel, with
> and without this change.  The patch applies to net unchanged, there I've
> only compiled it.
>
> I don't have the PRM, so "arming the CQ raises no event for the CQEs that
> are already there" is what the counters showed and not something I checked
> against the documentation.
> ---
>  drivers/net/ethernet/mellanox/mlx5/core/en_txrx.c | 9 ++++++++-
>  1 file changed, 8 insertions(+), 1 deletion(-)
>
> diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_txrx.c b/drivers/net/ethernet/mellanox/mlx5/core/en_txrx.c
> index 185105606469..3142e90e68ce 100644
> --- a/drivers/net/ethernet/mellanox/mlx5/core/en_txrx.c
> +++ b/drivers/net/ethernet/mellanox/mlx5/core/en_txrx.c
> @@ -207,7 +207,14 @@ int mlx5e_napi_poll(struct napi_struct *napi, int budget)
>  				rq);
>  
>  	if (busy) {
> -		if (likely(mlx5e_channel_no_affinity_change(c))) {
> +		/* A NAPI thread runs where the scheduler puts it, which says
> +		 * nothing about the affinity of the interrupt. Completing here
> +		 * would leave the CQEs that are pending to the next interrupt,
> +		 * and arming the CQs raises none for those, only for a new
> +		 * completion.
> +		 */
> +		if (likely(mlx5e_channel_no_affinity_change(c)) ||
> +		    test_bit(NAPI_STATE_THREADED, &napi->state)) {

Interesting. I wonder if more drivers hit this -- I think I've seen
similar affinity checks elsewhere.

I'd move this into a helper to hide these NAPI details. Regardless,

Reviewed-by: Björn Töpel <bjorn@kernel.org>

Re: [PATCH net] net/mlx5e: Fix stalled channels with threaded NAPI

[Jakub Kicinski]

On Wed, 21 Oct 2026 17:37:24 +0000 Josef Bacik wrote:
> +		if (likely(mlx5e_channel_no_affinity_change(c)) ||
> +		    test_bit(NAPI_STATE_THREADED, &napi->state)) {

I'd say this chek belongs in mlx5e_channel_no_affinity_change()
-- 
pw-bot: cr

Re: [PATCH net] net/mlx5e: Fix stalled channels with threaded NAPI

[Tariq Toukan]

On 21/10/2026 20:37, Josef Bacik wrote:
> With threaded NAPI an mlx5e channel can stop receiving when a burst of
> traffic ends. It stays dead for minutes on an idle machine, while the
> connections hashed to it time out.
> 

Hi,

Thanks for this interesting report and patch.

> When mlx5e_napi_poll() has more to do, it checks that it runs on a CPU of
> its interrupt's affinity mask. If it does not, it takes the affinity to
> have been changed: it completes the NAPI after at most budget - 1 packets
> and arms the CQs, so that the next interrupt moves the poll to the new CPU.
> 
> A NAPI thread runs wherever the scheduler puts it, so with threaded NAPI a
> busy queue takes this path at nearly every poll. That costs an interrupt
> for every 64 packets. Worse, it counts on an interrupt for the CQEs that
> are still in the CQ when it is armed, and none comes: the next event is
> raised by the next new completion. While packets keep arriving this goes
> unnoticed. When a flood stops, what is in the CQ stays there, and is worked
> off one poll of 64 packets per new arrival. If the RQ has run dry as well,
> arrivals are dropped (rx_out_of_buffer) and bring no completion either.
> 
> Seen on a 50 Gbit/s port (firmware 22.47.1088) with 63 channels, threaded
> NAPI and rx-usecs 64: a CPU-bound task bound to each of the 128 CPUs and
> 32 Gbit/s of UDP in 8 flows for 7 seconds, then both stopped. After the
> second round one channel had 0 polls and 0 events for 150 seconds on an
> idle machine, while TCP connections hashed to it timed out and
> rx_out_of_buffer counted their SYNs. Later arrivals took it 16 polls and
> 1018 packets further, it stood still again for 14 seconds, and another 93
> polls and 5917 packets emptied it. In a longer test 22 of 48 iperf3 runs
> failed with "unable to connect stream: Connection timed out", with no ping
> to the same address lost.
> 
> i40e and iavf force an interrupt at this point, and mlx5e does for XSK.
> Doing that here would cure the stall and keep the interrupt per 64 packets.
> 

The arm doorbell carries the consumer counter, and the device tracks the 
last index it reported an event for. Arming while CQEs posted after the 
last event are still unconsumed generates an event right away.

The affinity path depends on exactly that, since commit a8c2eb15797a 
("net/mlx5e: Stop NAPI when irq balancer changes affinity").

Event moderation can delay that event, but not drop it.

So the root analysis doesn't match the mlx5 device behavior.

> Fix by going on polling when the NAPI is threaded. With that, 8 of 8
> rounds leave every channel alive, ch_aff_change stays 0, rx_out_of_buffer
> does not move once the flood is over, none of the 48 runs fails, and the
> interrupts taken during the flood halve.
> 
> Fixes: 29863d41bb6e ("net: implement threaded-able napi poll loop support")
> Assisted-by: LLM
> Signed-off-by: Josef Bacik <josef@toxicpanda.com>
> ---
> I hit this while comparing threaded NAPI with softirq on two bare metal
> machines.  The runs in the changelog are from a 7.3-rc5 based kernel, with
> and without this change.  The patch applies to net unchanged, there I've
> only compiled it.

Is it a degradation? Did you have any "good"/working version?

> 
> I don't have the PRM, so "arming the CQ raises no event for the CQEs that
> are already there" is what the counters showed and not something I checked
> against the documentation.
> ---
>   drivers/net/ethernet/mellanox/mlx5/core/en_txrx.c | 9 ++++++++-
>   1 file changed, 8 insertions(+), 1 deletion(-)
> 
> diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_txrx.c b/drivers/net/ethernet/mellanox/mlx5/core/en_txrx.c
> index 185105606469..3142e90e68ce 100644
> --- a/drivers/net/ethernet/mellanox/mlx5/core/en_txrx.c
> +++ b/drivers/net/ethernet/mellanox/mlx5/core/en_txrx.c
> @@ -207,7 +207,14 @@ int mlx5e_napi_poll(struct napi_struct *napi, int budget)
>   				rq);
>   
>   	if (busy) {
> -		if (likely(mlx5e_channel_no_affinity_change(c))) {
> +		/* A NAPI thread runs where the scheduler puts it, which says
> +		 * nothing about the affinity of the interrupt. Completing here
> +		 * would leave the CQEs that are pending to the next interrupt,
> +		 * and arming the CQs raises none for those, only for a new
> +		 * completion.
> +		 */
> +		if (likely(mlx5e_channel_no_affinity_change(c)) ||
> +		    test_bit(NAPI_STATE_THREADED, &napi->state)) {
>   			work_done = budget;
>   			goto out;
>   		}
> 

If arming really doesn't raise an event here, softirq NAPI would stall 
the same way when the IRQ affinity changes under load, for example by 
irqbalance.
Was it seen there too? I believe not, so it must be something else 
causing the stall...

> ---
> base-commit: abeacca61eea4f6803df4a595b656ea45c2d3249
> change-id: 20261021-b4-mlx5e-threaded-napi-stall-7d0a14c7674d

Re: [PATCH net] net/mlx5e: Fix stalled channels with threaded NAPI

[Josef Bacik]

On Thu, Oct 22, 2026 at 02:38:41PM +0300, Tariq Toukan wrote:
> The arm doorbell carries the consumer counter, and the device tracks the
> last index it reported an event for. Arming while CQEs posted after the
> last event are still unconsumed generates an event right away.
>
> The affinity path depends on exactly that, since commit a8c2eb15797a
> ("net/mlx5e: Stop NAPI when irq balancer changes affinity").
>
> Event moderation can delay that event, but not drop it.
>
> So the root analysis doesn't match the mlx5 device behavior.

The CQ has a backlog much deeper than one poll, about 6900 CQEs in the
run in the changelog.  The thread polls 64, is off the IRQ's CPU,
completes and arms.  There are CQEs posted after the last event, so the
event comes right away, and the device now remembers the current
producer index.  The thread polls another 64, completes and arms again.
If nothing has arrived in between there is no CQE posted after the last
event, only older ones that are still unconsumed, so there is no event.
The rest of the backlog waits for the next arrival.

That matches the counters of the stalled channel.  It had 0 polls and 0
events for 150 seconds.  New connections then gave it 16 events, 16
polls and 1018 packets, so one poll of 64 per event.  It was armed the
whole time, it just took a new completion to get an event.  Then nothing
for 14 seconds, then 93 polls and 5917 packets, and it was empty.

Is that how the device behaves, no event on arm when everything that is
unconsumed was posted before the last event?  That part is my reading of
your description, I haven't got a way to check it.

> Is it a degradation? Did you have any "good"/working version?

I don't know.  I've only run threaded NAPI on these machines on the
7.3-rc5 based kernel, I have no older kernel that I know to be good.

> If arming really doesn't raise an event here, softirq NAPI would stall
> the same way when the IRQ affinity changes under load, for example by
> irqbalance.
> Was it seen there too? I believe not, so it must be something else
> causing the stall...

I haven't seen it there, and I haven't tried changing the affinity under
load.  If the above is right I wouldn't expect it.  With softirq the
path is taken once, that arm raises the event, the interrupt lands on
the new CPU, and from then on the check passes and the poll goes on
until the CQ is empty without arming again.  A NAPI thread takes the
path at every poll, so it needs a second event for a backlog that the
first one already covered.

I'll fix the changelog and the comment in v2 once I know I've got the
device's side of it right, and move the check into
mlx5e_channel_no_affinity_change() as Jakub asked.

Thanks,
Josef

Re: [PATCH net] net/mlx5e: Fix stalled channels with threaded NAPI

[Tariq Toukan]

On 22/10/2026 15:56, Josef Bacik wrote:
> On Thu, Oct 22, 2026 at 02:38:41PM +0300, Tariq Toukan wrote:
>> The arm doorbell carries the consumer counter, and the device tracks the
>> last index it reported an event for. Arming while CQEs posted after the
>> last event are still unconsumed generates an event right away.
>>
>> The affinity path depends on exactly that, since commit a8c2eb15797a
>> ("net/mlx5e: Stop NAPI when irq balancer changes affinity").
>>
>> Event moderation can delay that event, but not drop it.
>>
>> So the root analysis doesn't match the mlx5 device behavior.
> 
> The CQ has a backlog much deeper than one poll, about 6900 CQEs in the
> run in the changelog.  The thread polls 64, is off the IRQ's CPU,
> completes and arms.  There are CQEs posted after the last event, so the
> event comes right away, and the device now remembers the current
> producer index.  The thread polls another 64, completes and arms again.
> If nothing has arrived in between there is no CQE posted after the last
> event, only older ones that are still unconsumed, so there is no event.
> The rest of the backlog waits for the next arrival.
> 
> That matches the counters of the stalled channel.  It had 0 polls and 0
> events for 150 seconds.  New connections then gave it 16 events, 16
> polls and 1018 packets, so one poll of 64 per event.  It was armed the
> whole time, it just took a new completion to get an event.  Then nothing
> for 14 seconds, then 93 polls and 5917 packets, and it was empty.
> 
> Is that how the device behaves, no event on arm when everything that is
> unconsumed was posted before the last event?  That part is my reading of
> your description, I haven't got a way to check it.
> 

I have to soften my previous reply: the immediate-event guarantee I 
quoted covers CQEs posted after the last event, so it does not rule out 
your scenario, where the leftover backlog was all reported by that 
event. Your reading is consistent with the documentation and with your 
counters.

I'm checking the exact behavior with our HW architects and will update 
here, hopefully within a few days.

>> Is it a degradation? Did you have any "good"/working version?
> 
> I don't know.  I've only run threaded NAPI on these machines on the
> 7.3-rc5 based kernel, I have no older kernel that I know to be good.
> 
>> If arming really doesn't raise an event here, softirq NAPI would stall
>> the same way when the IRQ affinity changes under load, for example by
>> irqbalance.
>> Was it seen there too? I believe not, so it must be something else
>> causing the stall...
> 
> I haven't seen it there, and I haven't tried changing the affinity under
> load.  If the above is right I wouldn't expect it.  With softirq the
> path is taken once, that arm raises the event, the interrupt lands on
> the new CPU, and from then on the check passes and the poll goes on
> until the CQ is empty without arming again.  A NAPI thread takes the
> path at every poll, so it needs a second event for a backlog that the
> first one already covered.
> 
> I'll fix the changelog and the comment in v2 once I know I've got the
> device's side of it right, and move the check into
> mlx5e_channel_no_affinity_change() as Jakub asked.
> 
> Thanks,
> Josef