From: Peter Zijlstra <peterz@infradead.org>
To: Eric Dumazet <edumazet@google.com>
Cc: Thomas Gleixner <tglx@linutronix.de>,
Ingo Molnar <mingo@redhat.com>, Borislav Petkov <bp@alien8.de>,
Dave Hansen <dave.hansen@linux.intel.com>,
"H . Peter Anvin" <hpa@zytor.com>,
Steven Rostedt <rostedt@goodmis.org>,
linux-kernel <linux-kernel@vger.kernel.org>,
Alexei Starovoitov <ast@kernel.org>,
Daniel Borkmann <daniel@iogearbox.net>,
Masami Hiramatsu <mhiramat@kernel.org>,
x86@kernel.org, bpf@vger.kernel.org,
Eric Dumazet <eric.dumazet@gmail.com>,
Greg Thelen <gthelen@google.com>,
Stephane Eranian <eranian@google.com>
Subject: Re: [PATCH v2] x86/alternatives: remove false sharing in poke_int3_handler()
Date: Mon, 24 Mar 2025 12:18:17 +0100 [thread overview]
Message-ID: <20250324111817.GA14944@noisy.programming.kicks-ass.net> (raw)
In-Reply-To: <20250324083051.3938815-1-edumazet@google.com>
On Mon, Mar 24, 2025 at 08:30:51AM +0000, Eric Dumazet wrote:
> eBPF programs can be run 50,000,000 times per second on busy servers.
>
> Whenever /proc/sys/kernel/bpf_stats_enabled is turned off,
> hundreds of calls sites are patched from text_poke_bp_batch()
> and we see a huge loss of performance due to false sharing
> on bp_desc.refs lasting up to three seconds.
>
> 51.30% server_bin [kernel.kallsyms] [k] poke_int3_handler
> |
> |--46.45%--poke_int3_handler
> | exc_int3
> | asm_exc_int3
> | |
> | |--24.26%--cls_bpf_classify
> | | tcf_classify
> | | __dev_queue_xmit
> | | ip6_finish_output2
> | | ip6_output
> | | ip6_xmit
> | | inet6_csk_xmit
> | | __tcp_transmit_skb
> | | |
> | | |--9.00%--tcp_v6_do_rcv
> | | | tcp_v6_rcv
> | | | ip6_protocol_deliver_rcu
> | | | ip6_rcv_finish
> | | | ipv6_rcv
> | | | __netif_receive_skb
> | | | process_backlog
> | | | __napi_poll
> | | | net_rx_action
> | | | __softirqentry_text_start
> | | | asm_call_sysvec_on_stack
> | | | do_softirq_own_stack
>
> Fix this by replacing bp_desc.refs with a per-cpu bp_refs.
>
> Before the patch, on a host with 240 cores (480 threads):
>
> $ bpftool prog | grep run_time_ns
> ...
> 105: sched_cls name hn_egress tag 699fc5eea64144e3 gpl run_time_ns
> 3009063719 run_cnt 82757845
>
> -> average cost is 36 nsec per call
>
> echo 0 >/proc/sys/kernel/bpf_stats_enabled
> text_poke_bp_batch(nr_entries=2)
> text_poke_bp_batch+1
> text_poke_finish+27
> arch_jump_label_transform_apply+22
> jump_label_update+98
> __static_key_slow_dec_cpuslocked+64
> static_key_slow_dec+31
> bpf_stats_handler+236
> proc_sys_call_handler+396
> vfs_write+761
> ksys_write+102
> do_syscall_64+107
> entry_SYSCALL_64_after_hwframe+103
> Took 324 usec
>
> text_poke_bp_batch(nr_entries=164)
> text_poke_bp_batch+1
> text_poke_finish+27
> arch_jump_label_transform_apply+22
> jump_label_update+98
> __static_key_slow_dec_cpuslocked+64
> static_key_slow_dec+31
> bpf_stats_handler+236
> proc_sys_call_handler+396
> vfs_write+761
> ksys_write+102
> do_syscall_64+107
> entry_SYSCALL_64_after_hwframe+103
> Took 2655300 usec
>
> After this patch:
>
> $ bpftool prog | grep run_time_ns
> ...
> 105: sched_cls name hn_egress tag 699fc5eea64144e3 gpl run_time_ns
> 1928223019 run_cnt 67682728
>
> -> average cost is 28 nsec per call
>
> echo 0 >/proc/sys/kernel/bpf_stats_enabled
> text_poke_bp_batch(nr_entries=2)
> text_poke_bp_batch+1
> text_poke_finish+27
> arch_jump_label_transform_apply+22
> jump_label_update+98
> __static_key_slow_dec_cpuslocked+64
> static_key_slow_dec+31
> bpf_stats_handler+236
> proc_sys_call_handler+396
> vfs_write+761
> ksys_write+102
> do_syscall_64+107
> entry_SYSCALL_64_after_hwframe+103
> Took 519 usec
>
> text_poke_bp_batch(nr_entries=164)
> text_poke_bp_batch+1
> text_poke_finish+27
> arch_jump_label_transform_apply+22
> jump_label_update+98
> __static_key_slow_dec_cpuslocked+64
> static_key_slow_dec+31
> bpf_stats_handler+236
> proc_sys_call_handler+396
> vfs_write+761
> ksys_write+102
> do_syscall_64+107
> entry_SYSCALL_64_after_hwframe+103
> Took 702 usec
This is unreadable due to the amount of pointless repetition.
Also, urgh.
next prev parent reply other threads:[~2025-03-24 11:18 UTC|newest]
Thread overview: 3+ messages / expand[flat|nested] mbox.gz Atom feed top
2025-03-24 8:30 [PATCH v2] x86/alternatives: remove false sharing in poke_int3_handler() Eric Dumazet
2025-03-24 11:18 ` Peter Zijlstra [this message]
2025-03-25 4:28 ` Eric Dumazet
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20250324111817.GA14944@noisy.programming.kicks-ass.net \
--to=peterz@infradead.org \
--cc=ast@kernel.org \
--cc=bp@alien8.de \
--cc=bpf@vger.kernel.org \
--cc=daniel@iogearbox.net \
--cc=dave.hansen@linux.intel.com \
--cc=edumazet@google.com \
--cc=eranian@google.com \
--cc=eric.dumazet@gmail.com \
--cc=gthelen@google.com \
--cc=hpa@zytor.com \
--cc=linux-kernel@vger.kernel.org \
--cc=mhiramat@kernel.org \
--cc=mingo@redhat.com \
--cc=rostedt@goodmis.org \
--cc=tglx@linutronix.de \
--cc=x86@kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox