Netdev List
 help / color / mirror / Atom feed
From: Kuniyuki Iwashima <kuniyu@google.com>
To: Ray Zhang <sgzhang@google.com>
Cc: netdev@vger.kernel.org, Eric Dumazet <edumazet@kernel.org>,
	 Paolo Abeni <pabeni@redhat.com>,
	Willem de Bruijn <willemb@google.com>,
	 "David S . Miller" <davem@davemloft.net>,
	Jakub Kicinski <kuba@kernel.org>, Simon Horman <horms@kernel.org>,
	 David Decotigny <decot@google.com>,
	linux-kernel@vger.kernel.org, stable@vger.kernel.org
Subject: Re: [PATCH net] net: count last skb of each sk_backlog pass in __release_sock()
Date: Sun, 1 Nov 2026 17:15:55 +0000	[thread overview]
Message-ID: <audwphXNeZhVItS0@google.com> (raw)
In-Reply-To: <20261101075145.698399-1-sgzhang@google.com>

On Sun, 01 Nov 2026 00:51:45 -0700, Ray Zhang wrote:
> Running 500 concurrent TCPv6 streams (neper "tcp_stream -6 -T 1 -F 500")
> on a 2-core Cortex-A53 (CONFIG_PREEMPT_NONE=y, HZ=1000, LATENCY_WARN)

I guess you tested it on 6.18, but note that CONFIG_PREEMPT_NONE
is no longer selectable on arm64 since 7dadeaa6e851.


> triggers resched_latency_warn() in __release_sock():
> 
>   [ 3655.311821] sched: CPU 0 need_resched set for > 100006892 ns (100 ticks) without schedule
>   [ 3655.311889] Call trace:
>   ...
>   [ 3655.311944]  resched_latency_warn+0x7c/0x88
>   [ 3655.311958]  sched_tick+0x1c0/0x418
>   ...
>   [ 3655.312578]  net_rx_action+0x238/0x438
>   ...
>   [ 3655.312641]  do_softirq+0x4c/0x90
>   [ 3655.312651]  __local_bh_enable_ip+0x84/0x90
>   [ 3655.312663]  local_bh_enable+0x1c/0x28
>   [ 3655.312674]  __dev_queue_xmit+0x4a8/0xb00
>   ...
>   [ 3655.312782]  tcp_v6_do_rcv+0x240/0x5f8
>   [ 3655.312793]  __release_sock+0x9c/0x188
>   [ 3655.312806]  release_sock+0x150/0x168
>   [ 3655.312817]  tcp_sendmsg+0x4c/0x70
> 
> __release_sock() can loop past 100 ms without calling cond_resched():
> 
> - Each outer-loop pass drains sk_backlog, but "if (!skb) break;" runs
>   before "++nb", so a pass of B skbs only adds B - 1 to nb (0 for 1 skb).
> - With 500 TCPv6 streams and NAPI sharing two Cortex-A53 cores, runnable
>   tasks keep TIF_NEED_RESCHED set on CPU 0.
> - During spin_unlock_bh() and TX softirqs (__dev_queue_xmit() ->
>   local_bh_enable()), tcp_add_backlog() coalesces arriving pure ACKs into
>   a single tail skb.
> - Repeated 1-skb passes never advance nb or call cond_resched(),
>   triggering resched_latency_warn() after 100 ticks.
> 
> Restore the do-while loop and drop "if (!skb) break;" so every skb
> increments nb across passes and calls cond_resched() every 16 skbs.
> 
> Reproduced the issue and verified the fix on two 2-core Cortex-A53 hosts
> (CONFIG_PREEMPT_NONE=y, HZ=1000, LATENCY_WARN, latency_warn_once=0) with
> a userspace reproducer over /dev/net/tun:
> 
> - CPU 0 runs a TCPv6 sender and a second runnable task while CPU 1 writes
>   pure ACKs to /dev/net/tun (~30k TCPBacklogCoalesce/run), sustaining
>   1-skb sk_backlog refills in __release_sock() for ~400 ms (ftrace).
> - Before: CPU 0 stayed unscheduled in __release_sock(), hitting
>   resched_latency_warn() 293 and 275 times on the two hosts (> 100.9 ms).
> - After: cond_resched() yielded CPU 0 inside __release_sock(), with
>   0 resched_latency_warn() hits on both hosts.
> - Also observed no regression with the workload from
>   commit 16c610162d1f ("net: call cond_resched() less often in
>   __release_sock()"), "super_netperf 4 -H srv -T,10 -l 3000 &".
> 
> Fixes: 16c610162d1f ("net: call cond_resched() less often in __release_sock()")
> Cc: stable@vger.kernel.org
> Signed-off-by: Ray Zhang <sgzhang@google.com>

Reviewed-by: Kuniyuki Iwashima <kuniyu@google.com>

  reply	other threads:[~2026-11-01 17:15 UTC|newest]

Thread overview: 3+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-11-01  7:51 [PATCH net] net: count last skb of each sk_backlog pass in __release_sock() Ray Zhang
2026-11-01 17:15 ` Kuniyuki Iwashima [this message]
2026-11-02  5:35   ` Ray Zhang

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=audwphXNeZhVItS0@google.com \
    --to=kuniyu@google.com \
    --cc=davem@davemloft.net \
    --cc=decot@google.com \
    --cc=edumazet@kernel.org \
    --cc=horms@kernel.org \
    --cc=kuba@kernel.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=netdev@vger.kernel.org \
    --cc=pabeni@redhat.com \
    --cc=sgzhang@google.com \
    --cc=stable@vger.kernel.org \
    --cc=willemb@google.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox