From: Leon Hwang <leon.hwang@linux.dev>
To: bpf@vger.kernel.org
Cc: Alexei Starovoitov <ast@kernel.org>,
Andrii Nakryiko <andrii@kernel.org>,
Daniel Borkmann <daniel@iogearbox.net>,
Jiri Olsa <jolsa@kernel.org>
Subject: [BUG] Deadlock triggered by bpfsnoop funcgraph feature
Date: Wed, 27 Aug 2025 10:13:26 +0800 [thread overview]
Message-ID: <a08c7c19-1831-481f-9160-0583d850347a@linux.dev> (raw)
Hi,
I’ve encountered a reproducible deadlock while developing the funcgraph
feature for bpfsnoop [0].
Even on the latest bpf-next_base commit
2465bb83e0b4 ("Merge branch
's390-bpf-add-s390-jit-support-for-timed-may_goto'"),
the issue still persists.
Reproduction:
1. Build bpfsnoop with Go 1.24 and LLVM 20.
2. Start a VM using vmtest [1].
3. Trigger the deadlock with:
'./bpfsnoop -k "htab_*_elem" --output-fgraph --fgraph-debug'
Logs:
[ 126.934205] watchdog: CPU1: Watchdog detected hard LOCKUP on cpu 1
[ 126.934406] Modules linked in:
[ 126.934713] irq event stamp: 283284
[ 126.934806] hardirqs last enabled at (283283): [<ffffffffa7fa89f8>]
default_idle_call+0xb8/0x1d0
[ 126.934925] hardirqs last disabled at (283284): [<ffffffffa73ac21f>]
tick_nohz_idle_exit+0x8f/0x110
[ 126.935026] softirqs last enabled at (283262): [<ffffffffa72a4a06>]
__irq_exit_rcu+0xa6/0xd0
[ 126.935124] softirqs last disabled at (283255): [<ffffffffa72a4a06>]
__irq_exit_rcu+0xa6/0xd0
[ 126.935518] CPU: 1 UID: 0 PID: 0 Comm: swapper/1 Not tainted
6.17.0-rc1-gcb708c11617a #23 PREEMPT(full)
[ 126.935662] Hardware name: QEMU Ubuntu 24.04 PC (i440FX + PIIX,
1996), BIOS 1.16.3-debian-1.16.3-2 04/01/2014
[ 126.935865] RIP: 0010:__lock_acquire+0x30f/0x2590
[ 126.935973] Code: 89 f8 45 89 f7 49 89 de 4c 89 e3 41 89 cc 48 89 c1
eb 3e 48 8d 04 80 48 8d 04 80 48 8d 34 c5 40 78 87 a9 0f b6 86 c4 00 00
00 <84> c0 74 12 41 38 c0 44 0f 47 c0 80 be c6 00 00 00 02 44 0f 44 c0
[ 126.936062] RSP: 0018:ffffad20800ab008 EFLAGS: 00000007
[ 126.936219] RAX: 0000000000000003 RBX: ffff97af803f2d18 RCX:
0000000000000000
[ 126.936308] RDX: 0000000000000001 RSI: ffffffffa9877c28 RDI:
0000000000000007
[ 126.936394] RBP: ffffad20800ab080 R08: 0000000000000003 R09:
0000000000000005
[ 126.936480] R10: 0000000000000000 R11: 0000000000000007 R12:
0000000000000003
[ 126.936566] R13: ffff97af803f2240 R14: ffff97af803f2db8 R15:
0000000000000000
[ 126.936655] FS: 0000000000000000(0000) GS:ffff97b0126d8000(0000)
knlGS:0000000000000000
[ 126.936744] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 126.936830] CR2: 000000c0285a3000 CR3: 0000000102c62004 CR4:
0000000000770ef0
[ 126.936918] PKRU: 55555554
[ 126.937038] Call Trace:
[ 126.937133] <TASK>
[ 126.937222] ? __lock_acquire+0x43d/0x2590
[ 126.937620] lock_acquire+0xb1/0x2c0
[ 126.937706] ? __bpf_prog_enter_recur+0x2a/0x110
[ 126.937826] ? lock_release+0xc6/0x280
[ 126.937910] ? lock_release+0xc6/0x280
[ 126.938006] __bpf_prog_enter_recur+0x3e/0x110
[ 126.938090] ? __bpf_prog_enter_recur+0x2a/0x110
[ 126.938204] bpf_trampoline_6442539790+0x88/0x110
[ 126.938301] rcu_lockdep_current_cpu_online+0x9/0x70
[ 126.938392] ? rcu_read_lock_held+0x31/0x60
[ 126.938501] bpf_trampoline_6442539812+0x66/0x110
[ 126.938594] rcu_read_lock_held+0x9/0x60
[ 126.938678] ? __htab_map_lookup_elem+0x25/0xf0
[ 126.938798] bpf_trampoline_6442491246+0x79/0x123
[ 126.938894] __htab_map_lookup_elem+0x9/0xf0
[ 126.938991] ?
bpf_prog_243665d136749c2c_bpfsnoop_fgraph_tailcallee+0x129/0x14a
[ 126.939080] ? __htab_map_lookup_elem+0x9/0xf0
[ 126.939182] bpf_prog_1d471894f1fc624c_bpfsnoop_fgraph+0x12e/0x3e8
[ 126.939285] ? lock_release+0xc6/0x280
[ 126.939381] ? __bpf_prog_enter_recur+0x43/0x110
[ 126.939473] bpf_trampoline_6442539790+0x4b/0x110
[ 126.939566] rcu_lockdep_current_cpu_online+0x9/0x70
[ 126.939649] ? rcu_read_lock_held+0x31/0x60
[ 126.939737] bpf_trampoline_6442539812+0x66/0x110
[ 126.939829] rcu_read_lock_held+0x9/0x60
[ 126.939913] ? __htab_map_lookup_elem+0x25/0xf0
[ 126.940010] bpf_trampoline_6442491246+0x79/0x123
[ 126.940105] __htab_map_lookup_elem+0x9/0xf0
[ 126.940212] ?
bpf_prog_243665d136749c2c_bpfsnoop_fgraph_tailcallee+0x129/0x14a
[ 126.940300] ? rcu_lockdep_current_cpu_online+0x9/0x70
[ 126.940402] bpf_prog_1ed83077283e3ded_bpfsnoop_fgraph+0x12e/0x423
[ 126.940517] ? __bpf_prog_enter_recur+0x43/0x110
[ 126.940609] bpf_trampoline_6442491246+0xac/0x123
[ 126.940705] __htab_map_lookup_elem+0x9/0xf0
[ 126.940796] ?
bpf_prog_243665d136749c2c_bpfsnoop_fgraph_tailcallee+0x95/0x14a
[ 126.940895] ? bpf_prog_1d471894f1fc624c_bpfsnoop_fgraph+0x12e/0x3e8
[ 126.940980] ? bpf_prog_1d471894f1fc624c_bpfsnoop_fgraph+0x12e/0x3e8
[ 126.941080] bpf_prog_8c9f4824b35e5d8e_bpfsnoop_fgraph+0x12e/0x423
[ 126.941181] ? lock_release+0xc6/0x280
[ 126.941265] ? lock_release+0xc6/0x280
[ 126.941360] ? __bpf_prog_enter_recur+0x43/0x110
[ 126.941452] bpf_trampoline_6442539790+0x99/0x110
[ 126.941544] rcu_lockdep_current_cpu_online+0x9/0x70
[ 126.941627] ? rcu_read_lock_held+0x31/0x60
[ 126.941715] bpf_trampoline_6442539812+0x66/0x110
[ 126.941813] rcu_read_lock_held+0x9/0x60
[ 126.941896] ? __htab_map_lookup_elem+0x25/0xf0
[ 126.941993] bpf_trampoline_6442491246+0x79/0x123
[ 126.942089] __htab_map_lookup_elem+0x9/0xf0
[ 126.942186] ?
bpf_prog_243665d136749c2c_bpfsnoop_fgraph_tailcallee+0x95/0x14a
[ 126.942276] ? bpf_trampoline_6442539812+0x4b/0x110
[ 126.942360] ? bpf_trampoline_6442539812+0x4b/0x110
[ 126.942446] bpf_prog_8c9f4824b35e5d8e_bpfsnoop_fgraph+0x12e/0x423
[ 126.942552] ? __bpf_tramp_exit+0x72/0x130
[ 126.942647] ? __bpf_prog_enter_recur+0x43/0x110
[ 126.942739] bpf_trampoline_6442539812+0x99/0x110
[ 126.942832] rcu_read_lock_held+0x9/0x60
[ 126.942915] ? __htab_map_lookup_elem+0x25/0xf0
[ 126.943012] bpf_trampoline_6442491246+0x79/0x123
[ 126.943108] __htab_map_lookup_elem+0x9/0xf0
[ 126.943209] ?
bpf_prog_243665d136749c2c_bpfsnoop_fgraph_tailcallee+0x95/0x14a
[ 126.943299] ? bpf_trampoline_6442491246+0x79/0x123
[ 126.943383] ? bpf_trampoline_6442491246+0x79/0x123
[ 126.943469] bpf_prog_1d471894f1fc624c_bpfsnoop_fgraph+0x12e/0x3e8
[ 126.943571] ? lock_release+0xc6/0x280
[ 126.943666] ? __bpf_prog_enter_recur+0x43/0x110
[ 126.943758] bpf_trampoline_6442539812+0x4b/0x110
[ 126.943851] rcu_read_lock_held+0x9/0x60
[ 126.943934] ? __htab_map_lookup_elem+0x25/0xf0
[ 126.944031] bpf_trampoline_6442491246+0x79/0x123
[ 126.944126] __htab_map_lookup_elem+0x9/0xf0
[ 126.944235] ?
bpf_prog_243665d136749c2c_bpfsnoop_fgraph_tailcallee+0x129/0x14a
[ 126.944345] ? bpf_prog_5c5e9b8ca18045f2_bpfsnoop_fgraph+0x12e/0x3f2
[ 126.944442] bpf_prog_5c5b59f2388bb72a_bpfsnoop_fgraph+0x12e/0x3f2
[ 126.944545] ? lock_release+0xc6/0x280
[ 126.944640] ? __bpf_prog_enter_recur+0x43/0x110
[ 126.944732] bpf_trampoline_6442491246+0x56/0x123
[ 126.944828] __htab_map_lookup_elem+0x9/0xf0
[ 126.944931] ?
bpf_prog_243665d136749c2c_bpfsnoop_fgraph_tailcallee+0x129/0x14a
[ 126.945019] ? tick_nohz_idle_exit+0xc9/0x110
[ 126.945108] bpf_prog_5c5e9b8ca18045f2_bpfsnoop_fgraph+0x12e/0x3f2
[ 126.945210] ? lock_release+0xc6/0x280
[ 126.945305] ? __bpf_prog_enter_recur+0x43/0x110
[ 126.945411] bpf_trampoline_6442519845+0x5e/0x133
[ 126.945510] hrtimer_start_range_ns+0x9/0x4b0
[ 126.945603] ? tick_nohz_restart_sched_tick+0x89/0xe0
[ 126.945694] tick_nohz_idle_exit+0xc9/0x110
[ 126.945789] do_idle+0x150/0x250
[ 126.945890] cpu_startup_entry+0x2d/0x30
[ 126.945976] start_secondary+0xfc/0x100
[ 126.946069] common_startup_64+0x12c/0x138
[ 126.946197] </TASK>
Full log: [2].
Additional information:
* Kernel version: 6.17.0-rc1-gcb708c11617a
* Config: [3].
Links:
[0] https://github.com/bpfsnoop/bpfsnoop
[1] https://github.com/danobi/vmtest
[2]
https://gist.githubusercontent.com/Asphaltt/88d11c49e62485f4d4f4a7664089c3cd/raw/f26c123c0ec5f3e5ac588844db51bbec0bb0f9c7/deadlock-crash.log
[3]
https://gist.githubusercontent.com/Asphaltt/88d11c49e62485f4d4f4a7664089c3cd/raw/f26c123c0ec5f3e5ac588844db51bbec0bb0f9c7/config
Thanks,
Leon
next reply other threads:[~2025-08-27 2:13 UTC|newest]
Thread overview: 12+ messages / expand[flat|nested] mbox.gz Atom feed top
2025-08-27 2:13 Leon Hwang [this message]
2025-08-27 2:23 ` [BUG] Deadlock triggered by bpfsnoop funcgraph feature Alexei Starovoitov
2025-08-27 2:58 ` Leon Hwang
2025-08-28 0:42 ` Alexei Starovoitov
2025-08-28 2:40 ` Leon Hwang
2025-08-28 11:50 ` Paul E. McKenney
2025-08-28 13:39 ` Leon Hwang
2025-08-28 16:43 ` Alexei Starovoitov
2025-08-28 17:24 ` Paul E. McKenney
2025-08-29 2:21 ` Leon Hwang
2025-08-29 18:08 ` Alexei Starovoitov
2025-09-01 2:38 ` Leon Hwang
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=a08c7c19-1831-481f-9160-0583d850347a@linux.dev \
--to=leon.hwang@linux.dev \
--cc=andrii@kernel.org \
--cc=ast@kernel.org \
--cc=bpf@vger.kernel.org \
--cc=daniel@iogearbox.net \
--cc=jolsa@kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.