From: Usama Arif <usama.arif@linux.dev>
To: Breno Leitao <leitao@debian.org>
Cc: davem@davemloft.net, edumazet@google.com, horms@kernel.org,
kuba@kernel.org, kuniyu@google.com, linux-kernel@vger.kernel.org,
netdev@vger.kernel.org, pabeni@redhat.com, willemb@google.com,
shakeel.butt@linux.dev, hannes@cmpxchg.org, riel@surriel.com,
kernel-team@meta.com
Subject: Re: [PATCH] net: use sync wakeups for socket error reports
Date: Wed, 8 Jul 2026 17:08:59 +0100 [thread overview]
Message-ID: <bc16e45e-32ef-45d8-99af-14f088733ab3@linux.dev> (raw)
In-Reply-To: <ak5rH-SWjiqR6MVe@gmail.com>
On 08/07/2026 16:25, Breno Leitao wrote:
> On Wed, Jul 08, 2026 at 06:38:15AM -0700, Usama Arif wrote:
>> Measured on a 176-core EPYC 9D64 host running a Meta production
>> workload, bpftrace on tracepoint:ipi:ipi_send_cpu with a kstack filter
>> attributed the sock_def_error_report -> ep_poll_callback ->
>> try_to_wake_up -> ttwu_queue_wakelist -> __smp_call_single_queue
>> chain to 16,326 IPIs/min.
>
> I am interested in why so many sock_def_error_report().
>
> That's seems a lot for genuine socket errors (RST/ICMP) on a healthy
> host, so I suspect these aren't errors at all?
>
> Can you share the full stack above sock_def_error_report()?
I ran this bpftrace script the host now (results added at the end):
sudo bpftrace -e '
kprobe:sock_def_error_report
{
@wake_src[kstack()] = count();
}
interval:s:60
{
print(@wake_src, 5);
exit();
}'
The biggest source is tcp_sendmsg -> __skb_tstamp_tx, which as you said
is not an actual error. __skb_tstamp_tx clones the outgoing skb, tags it
with ee_origin = SO_EE_ORIGIN_TIMESTAMPING and ee_errno = ENOMSG, enqueues
it on sk->sk_error_queue via sock_queue_err_skb, and calls sk_error_report
so epoll raises EPOLLERR. Userspace then reads it with recvmsg(MSG_ERRQUEUE)
to get the SND/ACK timestamp.
So the workload has SO_TIMESTAMPING enabled on its TCP sockets, and every
packet completion and every ACK triggers a timestamp delivery through the
error-queue path, which is why sock_def_error_report fires.
Attached 2 probes
@wake_src[
sock_def_error_report+1
sk_error_report+17
sock_queue_err_skb+285
__skb_tstamp_tx+903
tcp_ack+3399
tcp_rcv_established+1630
tcp_v6_do_rcv+372
tcp_v6_rcv+4748
ip6_protocol_deliver_rcu+653
ip6_input_finish+79
ip6_input+43
ipv6_list_rcv+4339
__netif_receive_skb_list_core+244
netif_receive_skb_list_internal+433
napi_complete_done+149
bnxt_poll_p5+499
net_rx_action+513
irq_exit_rcu+312
common_interrupt+62
asm_common_interrupt+34
]: 1812
@wake_src[
sock_def_error_report+1
sk_error_report+17
sock_dequeue_err_skb+194
ipv6_recv_error+74
bpf_trampoline_6442598004+73
____sys_recvmsg.llvm.18251018526254450710+168
___sys_recvmsg+312
__x64_sys_recvmsg+95
do_syscall_64+316
entry_SYSCALL_64_after_hwframe+75
]: 8045
@wake_src[
sock_def_error_report+1
sk_error_report+17
sock_queue_err_skb+285
__skb_tstamp_tx+903
tcp_ack+3399
tcp_rcv_established+1258
tcp_v6_do_rcv+372
tcp_v6_rcv+4748
ip6_protocol_deliver_rcu+653
ip6_input_finish+79
ip6_input+43
ipv6_list_rcv+4339
__netif_receive_skb_list_core+244
netif_receive_skb_list_internal+433
napi_complete_done+149
bnxt_poll_p5+499
net_rx_action+513
irq_exit_rcu+312
common_interrupt+125
asm_common_interrupt+34
cpuidle_enter_state+202
cpuidle_enter+40
cpu_startup_entry+497
ap_starting+0
common_startup_64+318
]: 12603
@wake_src[
sock_def_error_report+1
sk_error_report+17
sock_queue_err_skb+285
__skb_tstamp_tx+903
tcp_ack+3399
tcp_rcv_established+1258
tcp_v6_do_rcv+372
tcp_v6_rcv+4748
ip6_protocol_deliver_rcu+653
ip6_input_finish+79
ip6_input+43
ipv6_list_rcv+4339
__netif_receive_skb_list_core+244
netif_receive_skb_list_internal+433
napi_complete_done+149
bnxt_poll_p5+499
net_rx_action+513
irq_exit_rcu+312
common_interrupt+62
asm_common_interrupt+34
]: 19314
@wake_src[
sock_def_error_report+1
sk_error_report+17
sock_queue_err_skb+285
__skb_tstamp_tx+903
bnxt_start_xmit+1769
dev_hard_start_xmit+160
sch_direct_xmit+165
__qdisc_run+714
__dev_queue_xmit+2052
skb_do_redirect+2531
netkit_xmit+715
dev_hard_start_xmit+160
__dev_queue_xmit+1049
ip6_finish_output2+848
ip6_finish_output+213
ip6_output+86
ip6_xmit+933
inet6_csk_xmit+163
__tcp_transmit_skb+2733
tcp_write_xmit+2948
__tcp_push_pending_frames+46
tcp_sendmsg_locked+4187
tcp_sendmsg+40
__x64_sys_sendmsg+567
do_syscall_64+316
entry_SYSCALL_64_after_hwframe+75
]: 33514
next prev parent reply other threads:[~2026-07-08 16:09 UTC|newest]
Thread overview: 4+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-07-08 13:38 [PATCH] net: use sync wakeups for socket error reports Usama Arif
2026-07-08 15:25 ` Breno Leitao
2026-07-08 16:08 ` Usama Arif [this message]
2026-07-08 16:32 ` Eric Dumazet
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=bc16e45e-32ef-45d8-99af-14f088733ab3@linux.dev \
--to=usama.arif@linux.dev \
--cc=davem@davemloft.net \
--cc=edumazet@google.com \
--cc=hannes@cmpxchg.org \
--cc=horms@kernel.org \
--cc=kernel-team@meta.com \
--cc=kuba@kernel.org \
--cc=kuniyu@google.com \
--cc=leitao@debian.org \
--cc=linux-kernel@vger.kernel.org \
--cc=netdev@vger.kernel.org \
--cc=pabeni@redhat.com \
--cc=riel@surriel.com \
--cc=shakeel.butt@linux.dev \
--cc=willemb@google.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.