* [PATCH] net: use sync wakeups for socket error reports
@ 2026-07-08 13:38 Usama Arif
2026-07-08 15:25 ` Breno Leitao
0 siblings, 1 reply; 4+ messages in thread
From: Usama Arif @ 2026-07-08 13:38 UTC (permalink / raw)
To: davem, edumazet, horms, kuba, kuniyu, linux-kernel, netdev,
pabeni, willemb, shakeel.butt, hannes, riel, kernel-team
Cc: Usama Arif
sock_def_readable() and sock_def_write_space() pass WF_SYNC to their
waitqueue wakeups, switch sock_def_error_report() to
wake_up_interruptible_sync_poll() so EPOLLERR waiters get the same hint.
ep_poll_callback() forwards it through to
try_to_wake_up() / select_task_rq_fair(), where wake_affine() can
prefer the waker CPU and skip a cross-CPU wakelist IPI when its
heuristics agree.
WF_SYNC matches the choice already made for readable/write_space. Some
error reports have the same producer/consumer shape: the waker has queued
an skb on sk->sk_error_queue, and the wakee is about to dequeue and copy
it out. Other reports publish sk_err/socket state directly, but still wake
a task likely to consume that socket state immediately.
Measured on a 176-core EPYC 9D64 host running a Meta production
workload, bpftrace on tracepoint:ipi:ipi_send_cpu with a kstack filter
attributed the sock_def_error_report -> ep_poll_callback ->
try_to_wake_up -> ttwu_queue_wakelist -> __smp_call_single_queue
chain to 16,326 IPIs/min. Switching to wake_up_interruptible_sync_poll()
will help reduce those IPIs.
Signed-off-by: Usama Arif <usama.arif@linux.dev>
---
net/core/sock.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/net/core/sock.c b/net/core/sock.c
index 8a59bfaa8096..c724f1442987 100644
--- a/net/core/sock.c
+++ b/net/core/sock.c
@@ -3639,7 +3639,7 @@ static void sock_def_error_report(struct sock *sk)
rcu_read_lock();
wq = rcu_dereference(sk->sk_wq);
if (skwq_has_sleeper(wq))
- wake_up_interruptible_poll(&wq->wait, EPOLLERR);
+ wake_up_interruptible_sync_poll(&wq->wait, EPOLLERR);
sk_wake_async_rcu(sk, SOCK_WAKE_IO, POLL_ERR);
rcu_read_unlock();
}
--
2.53.0-Meta
^ permalink raw reply related [flat|nested] 4+ messages in thread
* Re: [PATCH] net: use sync wakeups for socket error reports
2026-07-08 13:38 [PATCH] net: use sync wakeups for socket error reports Usama Arif
@ 2026-07-08 15:25 ` Breno Leitao
2026-07-08 16:08 ` Usama Arif
0 siblings, 1 reply; 4+ messages in thread
From: Breno Leitao @ 2026-07-08 15:25 UTC (permalink / raw)
To: Usama Arif
Cc: davem, edumazet, horms, kuba, kuniyu, linux-kernel, netdev,
pabeni, willemb, shakeel.butt, hannes, riel, kernel-team
On Wed, Jul 08, 2026 at 06:38:15AM -0700, Usama Arif wrote:
> Measured on a 176-core EPYC 9D64 host running a Meta production
> workload, bpftrace on tracepoint:ipi:ipi_send_cpu with a kstack filter
> attributed the sock_def_error_report -> ep_poll_callback ->
> try_to_wake_up -> ttwu_queue_wakelist -> __smp_call_single_queue
> chain to 16,326 IPIs/min.
I am interested in why so many sock_def_error_report().
That's seems a lot for genuine socket errors (RST/ICMP) on a healthy
host, so I suspect these aren't errors at all?
Can you share the full stack above sock_def_error_report()?
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: [PATCH] net: use sync wakeups for socket error reports
2026-07-08 15:25 ` Breno Leitao
@ 2026-07-08 16:08 ` Usama Arif
2026-07-08 16:32 ` Eric Dumazet
0 siblings, 1 reply; 4+ messages in thread
From: Usama Arif @ 2026-07-08 16:08 UTC (permalink / raw)
To: Breno Leitao
Cc: davem, edumazet, horms, kuba, kuniyu, linux-kernel, netdev,
pabeni, willemb, shakeel.butt, hannes, riel, kernel-team
On 08/07/2026 16:25, Breno Leitao wrote:
> On Wed, Jul 08, 2026 at 06:38:15AM -0700, Usama Arif wrote:
>> Measured on a 176-core EPYC 9D64 host running a Meta production
>> workload, bpftrace on tracepoint:ipi:ipi_send_cpu with a kstack filter
>> attributed the sock_def_error_report -> ep_poll_callback ->
>> try_to_wake_up -> ttwu_queue_wakelist -> __smp_call_single_queue
>> chain to 16,326 IPIs/min.
>
> I am interested in why so many sock_def_error_report().
>
> That's seems a lot for genuine socket errors (RST/ICMP) on a healthy
> host, so I suspect these aren't errors at all?
>
> Can you share the full stack above sock_def_error_report()?
I ran this bpftrace script the host now (results added at the end):
sudo bpftrace -e '
kprobe:sock_def_error_report
{
@wake_src[kstack()] = count();
}
interval:s:60
{
print(@wake_src, 5);
exit();
}'
The biggest source is tcp_sendmsg -> __skb_tstamp_tx, which as you said
is not an actual error. __skb_tstamp_tx clones the outgoing skb, tags it
with ee_origin = SO_EE_ORIGIN_TIMESTAMPING and ee_errno = ENOMSG, enqueues
it on sk->sk_error_queue via sock_queue_err_skb, and calls sk_error_report
so epoll raises EPOLLERR. Userspace then reads it with recvmsg(MSG_ERRQUEUE)
to get the SND/ACK timestamp.
So the workload has SO_TIMESTAMPING enabled on its TCP sockets, and every
packet completion and every ACK triggers a timestamp delivery through the
error-queue path, which is why sock_def_error_report fires.
Attached 2 probes
@wake_src[
sock_def_error_report+1
sk_error_report+17
sock_queue_err_skb+285
__skb_tstamp_tx+903
tcp_ack+3399
tcp_rcv_established+1630
tcp_v6_do_rcv+372
tcp_v6_rcv+4748
ip6_protocol_deliver_rcu+653
ip6_input_finish+79
ip6_input+43
ipv6_list_rcv+4339
__netif_receive_skb_list_core+244
netif_receive_skb_list_internal+433
napi_complete_done+149
bnxt_poll_p5+499
net_rx_action+513
irq_exit_rcu+312
common_interrupt+62
asm_common_interrupt+34
]: 1812
@wake_src[
sock_def_error_report+1
sk_error_report+17
sock_dequeue_err_skb+194
ipv6_recv_error+74
bpf_trampoline_6442598004+73
____sys_recvmsg.llvm.18251018526254450710+168
___sys_recvmsg+312
__x64_sys_recvmsg+95
do_syscall_64+316
entry_SYSCALL_64_after_hwframe+75
]: 8045
@wake_src[
sock_def_error_report+1
sk_error_report+17
sock_queue_err_skb+285
__skb_tstamp_tx+903
tcp_ack+3399
tcp_rcv_established+1258
tcp_v6_do_rcv+372
tcp_v6_rcv+4748
ip6_protocol_deliver_rcu+653
ip6_input_finish+79
ip6_input+43
ipv6_list_rcv+4339
__netif_receive_skb_list_core+244
netif_receive_skb_list_internal+433
napi_complete_done+149
bnxt_poll_p5+499
net_rx_action+513
irq_exit_rcu+312
common_interrupt+125
asm_common_interrupt+34
cpuidle_enter_state+202
cpuidle_enter+40
cpu_startup_entry+497
ap_starting+0
common_startup_64+318
]: 12603
@wake_src[
sock_def_error_report+1
sk_error_report+17
sock_queue_err_skb+285
__skb_tstamp_tx+903
tcp_ack+3399
tcp_rcv_established+1258
tcp_v6_do_rcv+372
tcp_v6_rcv+4748
ip6_protocol_deliver_rcu+653
ip6_input_finish+79
ip6_input+43
ipv6_list_rcv+4339
__netif_receive_skb_list_core+244
netif_receive_skb_list_internal+433
napi_complete_done+149
bnxt_poll_p5+499
net_rx_action+513
irq_exit_rcu+312
common_interrupt+62
asm_common_interrupt+34
]: 19314
@wake_src[
sock_def_error_report+1
sk_error_report+17
sock_queue_err_skb+285
__skb_tstamp_tx+903
bnxt_start_xmit+1769
dev_hard_start_xmit+160
sch_direct_xmit+165
__qdisc_run+714
__dev_queue_xmit+2052
skb_do_redirect+2531
netkit_xmit+715
dev_hard_start_xmit+160
__dev_queue_xmit+1049
ip6_finish_output2+848
ip6_finish_output+213
ip6_output+86
ip6_xmit+933
inet6_csk_xmit+163
__tcp_transmit_skb+2733
tcp_write_xmit+2948
__tcp_push_pending_frames+46
tcp_sendmsg_locked+4187
tcp_sendmsg+40
__x64_sys_sendmsg+567
do_syscall_64+316
entry_SYSCALL_64_after_hwframe+75
]: 33514
^ permalink raw reply [flat|nested] 4+ messages in thread* Re: [PATCH] net: use sync wakeups for socket error reports
2026-07-08 16:08 ` Usama Arif
@ 2026-07-08 16:32 ` Eric Dumazet
0 siblings, 0 replies; 4+ messages in thread
From: Eric Dumazet @ 2026-07-08 16:32 UTC (permalink / raw)
To: Usama Arif
Cc: Breno Leitao, davem, horms, kuba, kuniyu, linux-kernel, netdev,
pabeni, willemb, shakeel.butt, hannes, riel, kernel-team
On Wed, Jul 8, 2026 at 6:09 PM Usama Arif <usama.arif@linux.dev> wrote:
>
>
>
> On 08/07/2026 16:25, Breno Leitao wrote:
> > On Wed, Jul 08, 2026 at 06:38:15AM -0700, Usama Arif wrote:
> >> Measured on a 176-core EPYC 9D64 host running a Meta production
> >> workload, bpftrace on tracepoint:ipi:ipi_send_cpu with a kstack filter
> >> attributed the sock_def_error_report -> ep_poll_callback ->
> >> try_to_wake_up -> ttwu_queue_wakelist -> __smp_call_single_queue
> >> chain to 16,326 IPIs/min.
> >
> > I am interested in why so many sock_def_error_report().
> >
> > That's seems a lot for genuine socket errors (RST/ICMP) on a healthy
> > host, so I suspect these aren't errors at all?
> >
> > Can you share the full stack above sock_def_error_report()?
>
>
> I ran this bpftrace script the host now (results added at the end):
>
> sudo bpftrace -e '
> kprobe:sock_def_error_report
> {
> @wake_src[kstack()] = count();
> }
> interval:s:60
> {
> print(@wake_src, 5);
> exit();
> }'
>
>
> The biggest source is tcp_sendmsg -> __skb_tstamp_tx, which as you said
> is not an actual error. __skb_tstamp_tx clones the outgoing skb, tags it
> with ee_origin = SO_EE_ORIGIN_TIMESTAMPING and ee_errno = ENOMSG, enqueues
> it on sk->sk_error_queue via sock_queue_err_skb, and calls sk_error_report
> so epoll raises EPOLLERR. Userspace then reads it with recvmsg(MSG_ERRQUEUE)
> to get the SND/ACK timestamp.
>
> So the workload has SO_TIMESTAMPING enabled on its TCP sockets, and every
> packet completion and every ACK triggers a timestamp delivery through the
> error-queue path, which is why sock_def_error_report fires.
>
It seems we can not please everyone.
https://lore.kernel.org/netdev/20260526063650.952-1-xuewen.yan@unisoc.com/
Perhaps this SYNC heuristic should be a per-socket choice so that
applications can decide what is best for them.
^ permalink raw reply [flat|nested] 4+ messages in thread
end of thread, other threads:[~2026-07-08 16:32 UTC | newest]
Thread overview: 4+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2026-07-08 13:38 [PATCH] net: use sync wakeups for socket error reports Usama Arif
2026-07-08 15:25 ` Breno Leitao
2026-07-08 16:08 ` Usama Arif
2026-07-08 16:32 ` Eric Dumazet
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox