From: Jiri Olsa <jolsa@kernel.org>
To: Alexei Starovoitov <ast@kernel.org>,
Daniel Borkmann <daniel@iogearbox.net>,
Andrii Nakryiko <andrii@kernel.org>
Cc: bpf@vger.kernel.org, Martin KaFai Lau <kafai@fb.com>,
Song Liu <songliubraving@fb.com>, Yonghong Song <yhs@fb.com>,
John Fastabend <john.fastabend@gmail.com>,
KP Singh <kpsingh@chromium.org>,
Stanislav Fomichev <sdf@google.com>, Hao Luo <haoluo@google.com>
Subject: [PATCHv2 bpf 0/2] bpf: Disable preemption in perf_event_output helpers code
Date: Thu, 20 Jul 2023 10:57:02 +0200 [thread overview]
Message-ID: <20230720085704.190592-1-jolsa@kernel.org> (raw)
hi,
we got report of kernel crash [1][3] within bpf_event_output helper.
The reason is the nesting protection code in bpf_event_output that expects
disabled preemption, which is not guaranteed for programs executed by
bpf_prog_run_array_cg.
I managed to reproduce on tracing side where we have the same problem
in bpf_perf_event_output. The reproducer [2] just creates busy uprobe
and call bpf_perf_event_output helper a lot.
v2 changes:
- I changed 'Fixes' commits to where I saw we switched from preempt_disable
to migrate_disable, but I'm not completely sure about the patch 2, because
it was tricky to find, would be nice if somebody could check on that
thanks,
jirka
[1] https://github.com/cilium/cilium/issues/26756
[2] https://git.kernel.org/pub/scm/linux/kernel/git/jolsa/perf.git/commit/?h=bpf_output_fix_reproducer&id=8054dcc634121b884c7c331329d61d93351d03b5
[3] slack:
[66194.378161] BUG: kernel NULL pointer dereference, address: 0000000000000001
[66194.378324] #PF: supervisor instruction fetch in kernel mode
[66194.378447] #PF: error_code(0x0010) - not-present page
...
[66194.378692] Oops: 0010 [#1] PREEMPT SMP NOPTI
...
[66194.380666] <TASK>
[66194.380775] ? perf_output_sample+0x12a/0x9a0
[66194.380902] ? finish_task_switch.isra.0+0x81/0x280
[66194.381024] ? perf_event_output+0x66/0xa0
[66194.381148] ? bpf_event_output+0x13a/0x190
[66194.381270] ? bpf_event_output_data+0x22/0x40
[66194.381391] ? bpf_prog_dfc84bbde731b257_cil_sock4_connect+0x40a/0xacb
[66194.381519] ? xa_load+0x87/0xe0
[66194.381635] ? __cgroup_bpf_run_filter_sock_addr+0xc1/0x1a0
[66194.381759] ? release_sock+0x3e/0x90
[66194.381876] ? sk_setsockopt+0x1a1/0x12f0
[66194.381996] ? udp_pre_connect+0x36/0x50
[66194.382114] ? inet_dgram_connect+0x93/0xa0
[66194.382233] ? __sys_connect+0xb4/0xe0
[66194.382353] ? udp_setsockopt+0x27/0x40
[66194.382470] ? __pfx_udp_push_pending_frames+0x10/0x10
[66194.382593] ? __sys_setsockopt+0xdf/0x1a0
[66194.382713] ? __x64_sys_connect+0xf/0x20
[66194.382832] ? do_syscall_64+0x3a/0x90
[66194.382949] ? entry_SYSCALL_64_after_hwframe+0x72/0xdc
[66194.383077] </TASK>
---
Jiri Olsa (2):
bpf: Disable preemption in bpf_perf_event_output
bpf: Disable preemption in bpf_event_output
kernel/trace/bpf_trace.c | 18 ++++++++++++++----
1 file changed, 14 insertions(+), 4 deletions(-)
next reply other threads:[~2023-07-20 8:57 UTC|newest]
Thread overview: 11+ messages / expand[flat|nested] mbox.gz Atom feed top
2023-07-20 8:57 Jiri Olsa [this message]
2023-07-20 8:57 ` [PATCHv2 bpf 1/2] bpf: Disable preemption in bpf_perf_event_output Jiri Olsa
2023-07-21 11:40 ` Hou Tao
2023-07-20 8:57 ` [PATCHv2 bpf 2/2] bpf: Disable preemption in bpf_event_output Jiri Olsa
2023-07-21 12:16 ` Hou Tao
2023-07-21 12:45 ` Jiri Olsa
2023-07-21 13:45 ` Hou Tao
2023-07-24 18:01 ` Alexei Starovoitov
2023-07-21 12:12 ` [PATCHv2 bpf 0/2] bpf: Disable preemption in perf_event_output helpers code Hou Tao
2023-07-21 12:44 ` Jiri Olsa
2023-07-21 13:42 ` Hou Tao
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20230720085704.190592-1-jolsa@kernel.org \
--to=jolsa@kernel.org \
--cc=andrii@kernel.org \
--cc=ast@kernel.org \
--cc=bpf@vger.kernel.org \
--cc=daniel@iogearbox.net \
--cc=haoluo@google.com \
--cc=john.fastabend@gmail.com \
--cc=kafai@fb.com \
--cc=kpsingh@chromium.org \
--cc=sdf@google.com \
--cc=songliubraving@fb.com \
--cc=yhs@fb.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.