From: Kuniyuki Iwashima <kuniyu@google.com>
To: Ray Zhang <sgzhang@google.com>
Cc: netdev@vger.kernel.org, Eric Dumazet <edumazet@kernel.org>,
Paolo Abeni <pabeni@redhat.com>,
Willem de Bruijn <willemb@google.com>,
"David S . Miller" <davem@davemloft.net>,
Jakub Kicinski <kuba@kernel.org>, Simon Horman <horms@kernel.org>,
David Decotigny <decot@google.com>,
linux-kernel@vger.kernel.org, stable@vger.kernel.org
Subject: Re: [PATCH net] net: count last skb of each sk_backlog pass in __release_sock()
Date: Sun, 1 Nov 2026 17:15:55 +0000 [thread overview]
Message-ID: <audwphXNeZhVItS0@google.com> (raw)
In-Reply-To: <20261101075145.698399-1-sgzhang@google.com>
On Sun, 01 Nov 2026 00:51:45 -0700, Ray Zhang wrote:
> Running 500 concurrent TCPv6 streams (neper "tcp_stream -6 -T 1 -F 500")
> on a 2-core Cortex-A53 (CONFIG_PREEMPT_NONE=y, HZ=1000, LATENCY_WARN)
I guess you tested it on 6.18, but note that CONFIG_PREEMPT_NONE
is no longer selectable on arm64 since 7dadeaa6e851.
> triggers resched_latency_warn() in __release_sock():
>
> [ 3655.311821] sched: CPU 0 need_resched set for > 100006892 ns (100 ticks) without schedule
> [ 3655.311889] Call trace:
> ...
> [ 3655.311944] resched_latency_warn+0x7c/0x88
> [ 3655.311958] sched_tick+0x1c0/0x418
> ...
> [ 3655.312578] net_rx_action+0x238/0x438
> ...
> [ 3655.312641] do_softirq+0x4c/0x90
> [ 3655.312651] __local_bh_enable_ip+0x84/0x90
> [ 3655.312663] local_bh_enable+0x1c/0x28
> [ 3655.312674] __dev_queue_xmit+0x4a8/0xb00
> ...
> [ 3655.312782] tcp_v6_do_rcv+0x240/0x5f8
> [ 3655.312793] __release_sock+0x9c/0x188
> [ 3655.312806] release_sock+0x150/0x168
> [ 3655.312817] tcp_sendmsg+0x4c/0x70
>
> __release_sock() can loop past 100 ms without calling cond_resched():
>
> - Each outer-loop pass drains sk_backlog, but "if (!skb) break;" runs
> before "++nb", so a pass of B skbs only adds B - 1 to nb (0 for 1 skb).
> - With 500 TCPv6 streams and NAPI sharing two Cortex-A53 cores, runnable
> tasks keep TIF_NEED_RESCHED set on CPU 0.
> - During spin_unlock_bh() and TX softirqs (__dev_queue_xmit() ->
> local_bh_enable()), tcp_add_backlog() coalesces arriving pure ACKs into
> a single tail skb.
> - Repeated 1-skb passes never advance nb or call cond_resched(),
> triggering resched_latency_warn() after 100 ticks.
>
> Restore the do-while loop and drop "if (!skb) break;" so every skb
> increments nb across passes and calls cond_resched() every 16 skbs.
>
> Reproduced the issue and verified the fix on two 2-core Cortex-A53 hosts
> (CONFIG_PREEMPT_NONE=y, HZ=1000, LATENCY_WARN, latency_warn_once=0) with
> a userspace reproducer over /dev/net/tun:
>
> - CPU 0 runs a TCPv6 sender and a second runnable task while CPU 1 writes
> pure ACKs to /dev/net/tun (~30k TCPBacklogCoalesce/run), sustaining
> 1-skb sk_backlog refills in __release_sock() for ~400 ms (ftrace).
> - Before: CPU 0 stayed unscheduled in __release_sock(), hitting
> resched_latency_warn() 293 and 275 times on the two hosts (> 100.9 ms).
> - After: cond_resched() yielded CPU 0 inside __release_sock(), with
> 0 resched_latency_warn() hits on both hosts.
> - Also observed no regression with the workload from
> commit 16c610162d1f ("net: call cond_resched() less often in
> __release_sock()"), "super_netperf 4 -H srv -T,10 -l 3000 &".
>
> Fixes: 16c610162d1f ("net: call cond_resched() less often in __release_sock()")
> Cc: stable@vger.kernel.org
> Signed-off-by: Ray Zhang <sgzhang@google.com>
Reviewed-by: Kuniyuki Iwashima <kuniyu@google.com>
next prev parent reply other threads:[~2026-11-01 17:15 UTC|newest]
Thread overview: 3+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-11-01 7:51 [PATCH net] net: count last skb of each sk_backlog pass in __release_sock() Ray Zhang
2026-11-01 17:15 ` Kuniyuki Iwashima [this message]
2026-11-02 5:35 ` Ray Zhang
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=audwphXNeZhVItS0@google.com \
--to=kuniyu@google.com \
--cc=davem@davemloft.net \
--cc=decot@google.com \
--cc=edumazet@kernel.org \
--cc=horms@kernel.org \
--cc=kuba@kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=netdev@vger.kernel.org \
--cc=pabeni@redhat.com \
--cc=sgzhang@google.com \
--cc=stable@vger.kernel.org \
--cc=willemb@google.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox