From: Kuniyuki Iwashima <kuniyu@google.com>
To: Ray Zhang <sgzhang@google.com>
Cc: netdev@vger.kernel.org, Eric Dumazet <edumazet@kernel.org>,
Paolo Abeni <pabeni@redhat.com>,
Willem de Bruijn <willemb@google.com>,
"David S . Miller" <davem@davemloft.net>,
Jakub Kicinski <kuba@kernel.org>, Simon Horman <horms@kernel.org>,
David Decotigny <decot@google.com>,
linux-kernel@vger.kernel.org, stable@vger.kernel.org
Subject: Re: [PATCH net] net: count last skb of each sk_backlog pass in __release_sock()
Date: Sun, 1 Nov 2026 17:15:55 +0000 [thread overview]
Message-ID: <audwphXNeZhVItS0@google.com> (raw)
In-Reply-To: <20261101075145.698399-1-sgzhang@google.com>
On Sun, 01 Nov 2026 00:51:45 -0700, Ray Zhang wrote:
> Running 500 concurrent TCPv6 streams (neper "tcp_stream -6 -T 1 -F 500")
> on a 2-core Cortex-A53 (CONFIG_PREEMPT_NONE=y, HZ=1000, LATENCY_WARN)
I guess you tested it on 6.18, but note that CONFIG_PREEMPT_NONE
is no longer selectable on arm64 since 7dadeaa6e851.
> triggers resched_latency_warn() in __release_sock():
>
> [ 3655.311821] sched: CPU 0 need_resched set for > 100006892 ns (100 ticks) without schedule
> [ 3655.311889] Call trace:
> ...
> [ 3655.311944] resched_latency_warn+0x7c/0x88
> [ 3655.311958] sched_tick+0x1c0/0x418
> ...
> [ 3655.312578] net_rx_action+0x238/0x438
> ...
> [ 3655.312641] do_softirq+0x4c/0x90
> [ 3655.312651] __local_bh_enable_ip+0x84/0x90
> [ 3655.312663] local_bh_enable+0x1c/0x28
> [ 3655.312674] __dev_queue_xmit+0x4a8/0xb00
> ...
> [ 3655.312782] tcp_v6_do_rcv+0x240/0x5f8
> [ 3655.312793] __release_sock+0x9c/0x188
> [ 3655.312806] release_sock+0x150/0x168
> [ 3655.312817] tcp_sendmsg+0x4c/0x70
>
> __release_sock() can loop past 100 ms without calling cond_resched():
>
> - Each outer-loop pass drains sk_backlog, but "if (!skb) break;" runs
> before "++nb", so a pass of B skbs only adds B - 1 to nb (0 for 1 skb).
> - With 500 TCPv6 streams and NAPI sharing two Cortex-A53 cores, runnable
> tasks keep TIF_NEED_RESCHED set on CPU 0.
> - During spin_unlock_bh() and TX softirqs (__dev_queue_xmit() ->
> local_bh_enable()), tcp_add_backlog() coalesces arriving pure ACKs into
> a single tail skb.
> - Repeated 1-skb passes never advance nb or call cond_resched(),
> triggering resched_latency_warn() after 100 ticks.
>
> Restore the do-while loop and drop "if (!skb) break;" so every skb
> increments nb across passes and calls cond_resched() every 16 skbs.
>
> Reproduced the issue and verified the fix on two 2-core Cortex-A53 hosts
> (CONFIG_PREEMPT_NONE=y, HZ=1000, LATENCY_WARN, latency_warn_once=0) with
> a userspace reproducer over /dev/net/tun:
>
> - CPU 0 runs a TCPv6 sender and a second runnable task while CPU 1 writes
> pure ACKs to /dev/net/tun (~30k TCPBacklogCoalesce/run), sustaining
> 1-skb sk_backlog refills in __release_sock() for ~400 ms (ftrace).
> - Before: CPU 0 stayed unscheduled in __release_sock(), hitting
> resched_latency_warn() 293 and 275 times on the two hosts (> 100.9 ms).
> - After: cond_resched() yielded CPU 0 inside __release_sock(), with
> 0 resched_latency_warn() hits on both hosts.
> - Also observed no regression with the workload from
> commit 16c610162d1f ("net: call cond_resched() less often in
> __release_sock()"), "super_netperf 4 -H srv -T,10 -l 3000 &".
>
> Fixes: 16c610162d1f ("net: call cond_resched() less often in __release_sock()")
> Cc: stable@vger.kernel.org
> Signed-off-by: Ray Zhang <sgzhang@google.com>
Reviewed-by: Kuniyuki Iwashima <kuniyu@google.com>
next prev parent reply other threads:[~2026-11-01 17:15 UTC|newest]
Thread overview: 5+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-11-01 7:51 [PATCH net] net: count last skb of each sk_backlog pass in __release_sock() Ray Zhang
2026-11-01 17:15 ` Kuniyuki Iwashima [this message]
2026-11-02 5:35 ` Ray Zhang
2026-11-02 8:15 ` Eric Dumazet
2026-11-05 10:20 ` patchwork-bot+netdevbpf
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=audwphXNeZhVItS0@google.com \
--to=kuniyu@google.com \
--cc=davem@davemloft.net \
--cc=decot@google.com \
--cc=edumazet@kernel.org \
--cc=horms@kernel.org \
--cc=kuba@kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=netdev@vger.kernel.org \
--cc=pabeni@redhat.com \
--cc=sgzhang@google.com \
--cc=stable@vger.kernel.org \
--cc=willemb@google.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.