From: Usama Arif <usama.arif@linux.dev>
To: Breno Leitao <leitao@debian.org>
Cc: davem@davemloft.net, edumazet@google.com, horms@kernel.org,
kuba@kernel.org, kuniyu@google.com, linux-kernel@vger.kernel.org,
netdev@vger.kernel.org, pabeni@redhat.com, willemb@google.com,
shakeel.butt@linux.dev, hannes@cmpxchg.org, riel@surriel.com,
kernel-team@meta.com
Subject: Re: [PATCH] net: use sync wakeups for socket error reports
Date: Wed, 8 Jul 2026 17:08:59 +0100 [thread overview]
Message-ID: <bc16e45e-32ef-45d8-99af-14f088733ab3@linux.dev> (raw)
In-Reply-To: <ak5rH-SWjiqR6MVe@gmail.com>
On 08/07/2026 16:25, Breno Leitao wrote:
> On Wed, Jul 08, 2026 at 06:38:15AM -0700, Usama Arif wrote:
>> Measured on a 176-core EPYC 9D64 host running a Meta production
>> workload, bpftrace on tracepoint:ipi:ipi_send_cpu with a kstack filter
>> attributed the sock_def_error_report -> ep_poll_callback ->
>> try_to_wake_up -> ttwu_queue_wakelist -> __smp_call_single_queue
>> chain to 16,326 IPIs/min.
>
> I am interested in why so many sock_def_error_report().
>
> That's seems a lot for genuine socket errors (RST/ICMP) on a healthy
> host, so I suspect these aren't errors at all?
>
> Can you share the full stack above sock_def_error_report()?
I ran this bpftrace script the host now (results added at the end):
sudo bpftrace -e '
kprobe:sock_def_error_report
{
@wake_src[kstack()] = count();
}
interval:s:60
{
print(@wake_src, 5);
exit();
}'
The biggest source is tcp_sendmsg -> __skb_tstamp_tx, which as you said
is not an actual error. __skb_tstamp_tx clones the outgoing skb, tags it
with ee_origin = SO_EE_ORIGIN_TIMESTAMPING and ee_errno = ENOMSG, enqueues
it on sk->sk_error_queue via sock_queue_err_skb, and calls sk_error_report
so epoll raises EPOLLERR. Userspace then reads it with recvmsg(MSG_ERRQUEUE)
to get the SND/ACK timestamp.
So the workload has SO_TIMESTAMPING enabled on its TCP sockets, and every
packet completion and every ACK triggers a timestamp delivery through the
error-queue path, which is why sock_def_error_report fires.
Attached 2 probes
@wake_src[
sock_def_error_report+1
sk_error_report+17
sock_queue_err_skb+285
__skb_tstamp_tx+903
tcp_ack+3399
tcp_rcv_established+1630
tcp_v6_do_rcv+372
tcp_v6_rcv+4748
ip6_protocol_deliver_rcu+653
ip6_input_finish+79
ip6_input+43
ipv6_list_rcv+4339
__netif_receive_skb_list_core+244
netif_receive_skb_list_internal+433
napi_complete_done+149
bnxt_poll_p5+499
net_rx_action+513
irq_exit_rcu+312
common_interrupt+62
asm_common_interrupt+34
]: 1812
@wake_src[
sock_def_error_report+1
sk_error_report+17
sock_dequeue_err_skb+194
ipv6_recv_error+74
bpf_trampoline_6442598004+73
____sys_recvmsg.llvm.18251018526254450710+168
___sys_recvmsg+312
__x64_sys_recvmsg+95
do_syscall_64+316
entry_SYSCALL_64_after_hwframe+75
]: 8045
@wake_src[
sock_def_error_report+1
sk_error_report+17
sock_queue_err_skb+285
__skb_tstamp_tx+903
tcp_ack+3399
tcp_rcv_established+1258
tcp_v6_do_rcv+372
tcp_v6_rcv+4748
ip6_protocol_deliver_rcu+653
ip6_input_finish+79
ip6_input+43
ipv6_list_rcv+4339
__netif_receive_skb_list_core+244
netif_receive_skb_list_internal+433
napi_complete_done+149
bnxt_poll_p5+499
net_rx_action+513
irq_exit_rcu+312
common_interrupt+125
asm_common_interrupt+34
cpuidle_enter_state+202
cpuidle_enter+40
cpu_startup_entry+497
ap_starting+0
common_startup_64+318
]: 12603
@wake_src[
sock_def_error_report+1
sk_error_report+17
sock_queue_err_skb+285
__skb_tstamp_tx+903
tcp_ack+3399
tcp_rcv_established+1258
tcp_v6_do_rcv+372
tcp_v6_rcv+4748
ip6_protocol_deliver_rcu+653
ip6_input_finish+79
ip6_input+43
ipv6_list_rcv+4339
__netif_receive_skb_list_core+244
netif_receive_skb_list_internal+433
napi_complete_done+149
bnxt_poll_p5+499
net_rx_action+513
irq_exit_rcu+312
common_interrupt+62
asm_common_interrupt+34
]: 19314
@wake_src[
sock_def_error_report+1
sk_error_report+17
sock_queue_err_skb+285
__skb_tstamp_tx+903
bnxt_start_xmit+1769
dev_hard_start_xmit+160
sch_direct_xmit+165
__qdisc_run+714
__dev_queue_xmit+2052
skb_do_redirect+2531
netkit_xmit+715
dev_hard_start_xmit+160
__dev_queue_xmit+1049
ip6_finish_output2+848
ip6_finish_output+213
ip6_output+86
ip6_xmit+933
inet6_csk_xmit+163
__tcp_transmit_skb+2733
tcp_write_xmit+2948
__tcp_push_pending_frames+46
tcp_sendmsg_locked+4187
tcp_sendmsg+40
__x64_sys_sendmsg+567
do_syscall_64+316
entry_SYSCALL_64_after_hwframe+75
]: 33514
next prev parent reply other threads:[~2026-07-08 16:09 UTC|newest]
Thread overview: 4+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-07-08 13:38 [PATCH] net: use sync wakeups for socket error reports Usama Arif
2026-07-08 15:25 ` Breno Leitao
2026-07-08 16:08 ` Usama Arif [this message]
2026-07-08 16:32 ` Eric Dumazet
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=bc16e45e-32ef-45d8-99af-14f088733ab3@linux.dev \
--to=usama.arif@linux.dev \
--cc=davem@davemloft.net \
--cc=edumazet@google.com \
--cc=hannes@cmpxchg.org \
--cc=horms@kernel.org \
--cc=kernel-team@meta.com \
--cc=kuba@kernel.org \
--cc=kuniyu@google.com \
--cc=leitao@debian.org \
--cc=linux-kernel@vger.kernel.org \
--cc=netdev@vger.kernel.org \
--cc=pabeni@redhat.com \
--cc=riel@surriel.com \
--cc=shakeel.butt@linux.dev \
--cc=willemb@google.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox