[PATCH net-next v3] net: napi: Skip last poll when arming gro timer in busy poll

[PATCH net-next v3] net: napi: Skip last poll when arming gro timer in busy poll

[Martin Karsten]

Skip the extra call to napi->poll(), if the gro timer is armed at the
end of busy polling. This removes the need for having a separate
__busy_poll_stop() routine and its code is moved directly into the
relevant places in busy_poll_stop(). Remove obsolete comment about
ndo_busy_poll_stop().

This is a follow-up to commit 58e2330bd455 ("net: napi: Avoid gro timer
misfiring at end of busypoll"), which has deferred arming the gro timer
to the end of __busy_poll_stop() to eliminate a race condition between
a short timer and long poll that could leave the queue stuck with
interrupts disabled and no timer armed.

Co-developed-by: Dragos Tatulea <dtatulea@nvidia.com>
Signed-off-by: Dragos Tatulea <dtatulea@nvidia.com>
Signed-off-by: Martin Karsten <mkarsten@uwaterloo.ca>
---
Changes since v2 [1]:
- Fix patch formatting problems.
- Leave STATE_PREFER_BUSY_POLL bit set when timer is armed.
  Don't want interrupts anyway until timer clears STATE_PREFER_BUSY_POLL.

Changes since RFC [2]:
- Sending only the cleanup/improvement patch to net-next.
- Expand commit message to summarize original issue.
- Streamline control flow.
- Improve comments.

[1] https://lore.kernel.org/all/f0ecdde3-de16-47ba-b795-fe0042114e96@uwaterloo.c/
[2] https://lore.kernel.org/all/20260428175134.1197036-4-dtatulea@nvidia.com/
---
 net/core/dev.c | 63 ++++++++++++++++++++++----------------------------
 1 file changed, 28 insertions(+), 35 deletions(-)

diff --git a/net/core/dev.c b/net/core/dev.c
index 267cf3854395..a7b8f9a3588a 100644
--- a/net/core/dev.c
+++ b/net/core/dev.c
@@ -6862,22 +6862,6 @@ static void skb_defer_free_flush(void)
 
 #if defined(CONFIG_NET_RX_BUSY_POLL)
 
-static void __busy_poll_stop(struct napi_struct *napi, unsigned long timeout)
-{
-	if (!timeout) {
-		gro_normal_list(&napi->gro);
-		__napi_schedule(napi);
-		return;
-	}
-
-	/* Flush too old packets. If HZ < 1000, flush all packets */
-	gro_flush_normal(&napi->gro, HZ >= 1000);
-
-	clear_bit(NAPI_STATE_SCHED, &napi->state);
-	hrtimer_start(&napi->timer, ns_to_ktime(timeout),
-		      HRTIMER_MODE_REL_PINNED);
-}
-
 enum {
 	NAPI_F_PREFER_BUSY_POLL	= 1,
 	NAPI_F_END_ON_RESCHED	= 2,
@@ -6893,8 +6877,8 @@ static void busy_poll_stop(struct napi_struct *napi, void *have_poll_lock,
 	/* Busy polling means there is a high chance device driver hard irq
 	 * could not grab NAPI_STATE_SCHED, and that NAPI_STATE_MISSED was
 	 * set in napi_schedule_prep().
-	 * Since we are about to call napi->poll() once more, we can safely
-	 * clear NAPI_STATE_MISSED.
+	 * Since we either call napi->poll() once more or start the timer,
+	 * we can safely clear NAPI_STATE_MISSED.
 	 *
 	 * Note: x86 could use a single "lock and ..." instruction
 	 * to perform these two clear_bit()
@@ -6907,27 +6891,36 @@ static void busy_poll_stop(struct napi_struct *napi, void *have_poll_lock,
 
 	if (flags & NAPI_F_PREFER_BUSY_POLL) {
 		napi->defer_hard_irqs_count = napi_get_defer_hard_irqs(napi);
-		if (napi->defer_hard_irqs_count) {
-			/* A short enough gro flush timeout and long enough
-			 * poll can result in timer firing too early.
-			 * Timer will be armed later if necessary.
-			 */
+		if (napi->defer_hard_irqs_count)
 			timeout = napi_get_gro_flush_timeout(napi);
+	}
+	if (timeout) {
+		netpoll_poll_unlock(have_poll_lock);
+
+		/* Drain aged GRO packets before clearing SCHED since the NAPI
+		 * won't run again until after the timer fires. When HZ < 1000,
+		 * GRO age comparison is too coarse, so flush everything.
+		 */
+		gro_flush_normal(&napi->gro, HZ >= 1000);
+
+		clear_bit(NAPI_STATE_SCHED, &napi->state);
+		hrtimer_start(&napi->timer, ns_to_ktime(timeout),
+			      HRTIMER_MODE_REL_PINNED);
+	} else {
+		/* All we really want here is to re-enable device interrupts. */
+		rc = napi->poll(napi, budget);
+		/* We can't gro_normal_list() here, because napi->poll() might
+		 * have rearmed the napi (napi_complete_done()) in which case
+		 * it could already be running on another CPU.
+		 */
+		trace_napi_poll(napi, rc, budget);
+		netpoll_poll_unlock(have_poll_lock);
+		if (rc == budget) {
+			gro_normal_list(&napi->gro);
+			__napi_schedule(napi);
 		}
 	}
 
-	/* All we really want here is to re-enable device interrupts.
-	 * Ideally, a new ndo_busy_poll_stop() could avoid another round.
-	 */
-	rc = napi->poll(napi, budget);
-	/* We can't gro_normal_list() here, because napi->poll() might have
-	 * rearmed the napi (napi_complete_done()) in which case it could
-	 * already be running on another CPU.
-	 */
-	trace_napi_poll(napi, rc, budget);
-	netpoll_poll_unlock(have_poll_lock);
-	if (rc == budget)
-		__busy_poll_stop(napi, timeout);
 	bpf_net_ctx_clear(bpf_net_ctx);
 	local_bh_enable();
 }

base-commit: 627ac78f2741e2ebd2225e2e953b6964a8a9182f
-- 
2.54.0

Re: [PATCH net-next v3] net: napi: Skip last poll when arming gro timer in busy poll

[Joe Damato]

On Mon, May 18, 2026 at 02:13:32PM -0400, Martin Karsten wrote:
> Skip the extra call to napi->poll(), if the gro timer is armed at the
> end of busy polling. This removes the need for having a separate
> __busy_poll_stop() routine and its code is moved directly into the
> relevant places in busy_poll_stop(). Remove obsolete comment about
> ndo_busy_poll_stop().
> 
> This is a follow-up to commit 58e2330bd455 ("net: napi: Avoid gro timer
> misfiring at end of busypoll"), which has deferred arming the gro timer
> to the end of __busy_poll_stop() to eliminate a race condition between
> a short timer and long poll that could leave the queue stuck with
> interrupts disabled and no timer armed.
> 
> Co-developed-by: Dragos Tatulea <dtatulea@nvidia.com>
> Signed-off-by: Dragos Tatulea <dtatulea@nvidia.com>
> Signed-off-by: Martin Karsten <mkarsten@uwaterloo.ca>
> ---
> Changes since v2 [1]:
> - Fix patch formatting problems.
> - Leave STATE_PREFER_BUSY_POLL bit set when timer is armed.
>   Don't want interrupts anyway until timer clears STATE_PREFER_BUSY_POLL.
> 
> Changes since RFC [2]:
> - Sending only the cleanup/improvement patch to net-next.
> - Expand commit message to summarize original issue.
> - Streamline control flow.
> - Improve comments.
> 
> [1] https://lore.kernel.org/all/f0ecdde3-de16-47ba-b795-fe0042114e96@uwaterloo.c/
> [2] https://lore.kernel.org/all/20260428175134.1197036-4-dtatulea@nvidia.com/
> ---
>  net/core/dev.c | 63 ++++++++++++++++++++++----------------------------
>  1 file changed, 28 insertions(+), 35 deletions(-)

Reviewed-by: Joe Damato <joe@dama.to>

Re: [PATCH net-next v3] net: napi: Skip last poll when arming gro timer in busy poll

[Paolo Abeni]

On 5/18/26 8:13 PM, Martin Karsten wrote:
> +	} else {
> +		/* All we really want here is to re-enable device interrupts. */
> +		rc = napi->poll(napi, budget);
> +		/* We can't gro_normal_list() here, because napi->poll() might
> +		 * have rearmed the napi (napi_complete_done()) in which case
> +		 * it could already be running on another CPU.
> +		 */
> +		trace_napi_poll(napi, rc, budget);
> +		netpoll_poll_unlock(have_poll_lock);
> +		if (rc == budget) {
> +			gro_normal_list(&napi->gro);

Not blocking, but I find the above slightly confusing after this
refactor. Perhaps it should be re-worded?

/P

Re: [PATCH net-next v3] net: napi: Skip last poll when arming gro timer in busy poll

[Martin Karsten]

On 2026-05-21 10:13, Paolo Abeni wrote:
> On 5/18/26 8:13 PM, Martin Karsten wrote:
>> +	} else {
>> +		/* All we really want here is to re-enable device interrupts. */
>> +		rc = napi->poll(napi, budget);
>> +		/* We can't gro_normal_list() here, because napi->poll() might
>> +		 * have rearmed the napi (napi_complete_done()) in which case
>> +		 * it could already be running on another CPU.
>> +		 */
>> +		trace_napi_poll(napi, rc, budget);
>> +		netpoll_poll_unlock(have_poll_lock);
>> +		if (rc == budget) {
>> +			gro_normal_list(&napi->gro);
> 
> Not blocking, but I find the above slightly confusing after this
> refactor. Perhaps it should be re-worded?

I'd be happy to reword, but I am not familiar with the internals of gro 
processing, so would appreciate any help I can get?

Should I try-and-error via subsequent patch submission(s) or can you 
comment on this version:

		/* Use the driver poll to re-enable device interrupts. */
		rc = napi->poll(napi, budget);
		trace_napi_poll(napi, rc, budget);
		netpoll_poll_unlock(have_poll_lock);
		if (rc == budget) {
			/* If napi_complete_done has not been called by the
			 * driver poll, the napi needs to be rescheduled.
			 * Use the opportunity to pass up completed gro skbs.
			 */
			gro_normal_list(&napi->gro);

Thanks,
Martin

Re: [PATCH net-next v3] net: napi: Skip last poll when arming gro timer in busy poll

[Paolo Abeni]

On 5/21/26 5:42 PM, Martin Karsten wrote:
> On 2026-05-21 10:13, Paolo Abeni wrote:
>> On 5/18/26 8:13 PM, Martin Karsten wrote:
>>> +	} else {
>>> +		/* All we really want here is to re-enable device interrupts. */
>>> +		rc = napi->poll(napi, budget);
>>> +		/* We can't gro_normal_list() here, because napi->poll() might
>>> +		 * have rearmed the napi (napi_complete_done()) in which case
>>> +		 * it could already be running on another CPU.
>>> +		 */
>>> +		trace_napi_poll(napi, rc, budget);
>>> +		netpoll_poll_unlock(have_poll_lock);
>>> +		if (rc == budget) {
>>> +			gro_normal_list(&napi->gro);
>>
>> Not blocking, but I find the above slightly confusing after this
>> refactor. Perhaps it should be re-worded?
> 
> I'd be happy to reword, but I am not familiar with the internals of gro 
> processing, so would appreciate any help I can get?
> 
> Should I try-and-error via subsequent patch submission(s) or can you 
> comment on this version:
> 
> 		/* Use the driver poll to re-enable device interrupts. */
> 		rc = napi->poll(napi, budget);
> 		trace_napi_poll(napi, rc, budget);
> 		netpoll_poll_unlock(have_poll_lock);
> 		if (rc == budget) {
> 			/* If napi_complete_done has not been called by the
> 			 * driver poll, the napi needs to be rescheduled.
> 			 * Use the opportunity to pass up completed gro skbs.

Possibly something alike the following:

			/* More work to do: no concurrent napi-related 	
			 * irq is possible and the current thread can
			 * pass up the pending skb batch.
			 */

?

Re: [PATCH net-next v3] net: napi: Skip last poll when arming gro timer in busy poll

[Jakub Kicinski]

On Thu, 21 May 2026 18:19:21 +0200 Paolo Abeni wrote:
> >> Not blocking, but I find the above slightly confusing after this
> >> refactor. Perhaps it should be re-worded?  
> > 
> > I'd be happy to reword, but I am not familiar with the internals of gro 
> > processing, so would appreciate any help I can get?
> > 
> > Should I try-and-error via subsequent patch submission(s) or can you 
> > comment on this version:
> > 
> > 		/* Use the driver poll to re-enable device interrupts. */
> > 		rc = napi->poll(napi, budget);
> > 		trace_napi_poll(napi, rc, budget);
> > 		netpoll_poll_unlock(have_poll_lock);
> > 		if (rc == budget) {
> > 			/* If napi_complete_done has not been called by the
> > 			 * driver poll, the napi needs to be rescheduled.
> > 			 * Use the opportunity to pass up completed gro skbs.  
> 
> Possibly something alike the following:
> 
> 			/* More work to do: no concurrent napi-related 	
> 			 * irq is possible and the current thread can
> 			 * pass up the pending skb batch.
> 			 */

I'd stay focused on the ownership FWIW:

		/* Unless rc == budget we no longer own the NAPI instance,
		 * IRQ may fire on another CPU, poll this NAPI and enter GRO.
		 */

(in the current location of the comment i.e. _before_ the if (rc == budget))