[PATCH net-next] net: add a fast path in __netif_schedule()

[PATCH net-next] net: add a fast path in __netif_schedule()

[Eric Dumazet]

Cpus serving NIC interrupts and specifically TX completions are often
trapped in also restarting a busy qdisc (because qdisc was stopped by BQL
or the driver's own flow control).

When they call netdev_tx_completed_queue() or netif_tx_wake_queue(),
they call __netif_schedule() so that the queue can be run
later from net_tx_action() (involving NET_TX_SOFTIRQ)

Quite often, by the time the cpu reaches net_tx_action(), another cpu
grabbed the qdisc spinlock from __dev_xmit_skb(), and we spend too much
time spinning on this lock.

We can detect in __netif_schedule() if a cpu is already at a specific
point in __dev_xmit_skb() where we have the guarantee the queue will
be run.

This patch gives a 13 % increase of throughput on an IDPF NIC (200Gbit),
32 TX qeues, sending UDP packets of 120 bytes.

This also helps __qdisc_run() to not force a NET_TX_SOFTIRQ
if another thread is waiting in __dev_xmit_skb()

Before:

sar -n DEV 5 5|grep eth1|grep Average
Average:         eth1   1496.44 52191462.56    210.00 13369396.90      0.00      0.00      0.00     54.76

After:

sar -n DEV 5 5|grep eth1|grep Average
Average:         eth1   1457.88 59363099.96    205.08 15206384.35      0.00      0.00      0.00     62.29

Signed-off-by: Eric Dumazet <edumazet@google.com>
---
 net/core/dev.c | 7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/net/core/dev.c b/net/core/dev.c
index 821e7c718924405d0a7c10e41f677b98aa2d070b..9482b905c66a53501ad3b737ad4461533b9e7a4e 100644
--- a/net/core/dev.c
+++ b/net/core/dev.c
@@ -3373,6 +3373,13 @@ static void __netif_reschedule(struct Qdisc *q)
 
 void __netif_schedule(struct Qdisc *q)
 {
+	/* If q->defer_list is not empty, at least one thread is
+	 * in __dev_xmit_skb() before llist_del_all(&q->defer_list).
+	 * This thread will attempt to run the queue.
+	 */
+	if (!llist_empty(&q->defer_list))
+		return;
+
 	if (!test_and_set_bit(__QDISC_STATE_SCHED, &q->state))
 		__netif_reschedule(q);
 }
-- 
2.51.0.858.gf9c4a03a3a-goog

Re: [PATCH net-next] net: add a fast path in __netif_schedule()

[Kuniyuki Iwashima]

On Fri, Oct 17, 2025 at 7:53 AM Eric Dumazet <edumazet@google.com> wrote:
>
> Cpus serving NIC interrupts and specifically TX completions are often
> trapped in also restarting a busy qdisc (because qdisc was stopped by BQL
> or the driver's own flow control).
>
> When they call netdev_tx_completed_queue() or netif_tx_wake_queue(),
> they call __netif_schedule() so that the queue can be run
> later from net_tx_action() (involving NET_TX_SOFTIRQ)
>
> Quite often, by the time the cpu reaches net_tx_action(), another cpu
> grabbed the qdisc spinlock from __dev_xmit_skb(), and we spend too much
> time spinning on this lock.
>
> We can detect in __netif_schedule() if a cpu is already at a specific
> point in __dev_xmit_skb() where we have the guarantee the queue will
> be run.
>
> This patch gives a 13 % increase of throughput on an IDPF NIC (200Gbit),
> 32 TX qeues, sending UDP packets of 120 bytes.
>
> This also helps __qdisc_run() to not force a NET_TX_SOFTIRQ
> if another thread is waiting in __dev_xmit_skb()
>
> Before:
>
> sar -n DEV 5 5|grep eth1|grep Average
> Average:         eth1   1496.44 52191462.56    210.00 13369396.90      0.00      0.00      0.00     54.76
>
> After:
>
> sar -n DEV 5 5|grep eth1|grep Average
> Average:         eth1   1457.88 59363099.96    205.08 15206384.35      0.00      0.00      0.00     62.29
>
> Signed-off-by: Eric Dumazet <edumazet@google.com>

Reviewed-by: Kuniyuki Iwashima <kuniyu@google.com>

Thanks!

Re: [PATCH net-next] net: add a fast path in __netif_schedule()

[patchwork-bot+netdevbpf]

Hello:

This patch was applied to netdev/net-next.git (main)
by Jakub Kicinski <kuba@kernel.org>:

On Fri, 17 Oct 2025 14:53:34 +0000 you wrote:
> Cpus serving NIC interrupts and specifically TX completions are often
> trapped in also restarting a busy qdisc (because qdisc was stopped by BQL
> or the driver's own flow control).
> 
> When they call netdev_tx_completed_queue() or netif_tx_wake_queue(),
> they call __netif_schedule() so that the queue can be run
> later from net_tx_action() (involving NET_TX_SOFTIRQ)
> 
> [...]

Here is the summary with links:
  - [net-next] net: add a fast path in __netif_schedule()
    https://git.kernel.org/netdev/net-next/c/f8a55d5e71e6

You are awesome, thank you!
-- 
Deet-doot-dot, I am a bot.
https://korg.docs.kernel.org/patchwork/pwbot.html