From: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
To: linux-kernel@vger.kernel.org
Cc: Greg Kroah-Hartman <gregkh@linuxfoundation.org>,
stable@vger.kernel.org, Kuniyuki Iwashima <kuniyu@amazon.com>,
Ayushman Dutta <ayudutta@amazon.com>,
"Naveen N. Rao" <naveen.n.rao@linux.ibm.com>,
Anil S Keshavamurthy <anil.s.keshavamurthy@intel.com>,
"David S. Miller" <davem@davemloft.net>,
Masami Hiramatsu <mhiramat@kernel.org>,
Wang Nan <wangnan0@huawei.com>,
Kuniyuki Iwashima <kuni1840@gmail.com>,
Andrew Morton <akpm@linux-foundation.org>
Subject: [PATCH 4.14 41/42] kprobes: dont call disarm_kprobe() for disabled kprobes
Date: Fri, 2 Sep 2022 14:19:05 +0200 [thread overview]
Message-ID: <20220902121400.172910355@linuxfoundation.org> (raw)
In-Reply-To: <20220902121358.773776406@linuxfoundation.org>
From: Kuniyuki Iwashima <kuniyu@amazon.com>
commit 9c80e79906b4ca440d09e7f116609262bb747909 upstream.
The assumption in __disable_kprobe() is wrong, and it could try to disarm
an already disarmed kprobe and fire the WARN_ONCE() below. [0] We can
easily reproduce this issue.
1. Write 0 to /sys/kernel/debug/kprobes/enabled.
# echo 0 > /sys/kernel/debug/kprobes/enabled
2. Run execsnoop. At this time, one kprobe is disabled.
# /usr/share/bcc/tools/execsnoop &
[1] 2460
PCOMM PID PPID RET ARGS
# cat /sys/kernel/debug/kprobes/list
ffffffff91345650 r __x64_sys_execve+0x0 [FTRACE]
ffffffff91345650 k __x64_sys_execve+0x0 [DISABLED][FTRACE]
3. Write 1 to /sys/kernel/debug/kprobes/enabled, which changes
kprobes_all_disarmed to false but does not arm the disabled kprobe.
# echo 1 > /sys/kernel/debug/kprobes/enabled
# cat /sys/kernel/debug/kprobes/list
ffffffff91345650 r __x64_sys_execve+0x0 [FTRACE]
ffffffff91345650 k __x64_sys_execve+0x0 [DISABLED][FTRACE]
4. Kill execsnoop, when __disable_kprobe() calls disarm_kprobe() for the
disabled kprobe and hits the WARN_ONCE() in __disarm_kprobe_ftrace().
# fg
/usr/share/bcc/tools/execsnoop
^C
Actually, WARN_ONCE() is fired twice, and __unregister_kprobe_top() misses
some cleanups and leaves the aggregated kprobe in the hash table. Then,
__unregister_trace_kprobe() initialises tk->rp.kp.list and creates an
infinite loop like this.
aggregated kprobe.list -> kprobe.list -.
^ |
'.__.'
In this situation, these commands fall into the infinite loop and result
in RCU stall or soft lockup.
cat /sys/kernel/debug/kprobes/list : show_kprobe_addr() enters into the
infinite loop with RCU.
/usr/share/bcc/tools/execsnoop : warn_kprobe_rereg() holds kprobe_mutex,
and __get_valid_kprobe() is stuck in
the loop.
To avoid the issue, make sure we don't call disarm_kprobe() for disabled
kprobes.
[0]
Failed to disarm kprobe-ftrace at __x64_sys_execve+0x0/0x40 (error -2)
WARNING: CPU: 6 PID: 2460 at kernel/kprobes.c:1130 __disarm_kprobe_ftrace.isra.19 (kernel/kprobes.c:1129)
Modules linked in: ena
CPU: 6 PID: 2460 Comm: execsnoop Not tainted 5.19.0+ #28
Hardware name: Amazon EC2 c5.2xlarge/, BIOS 1.0 10/16/2017
RIP: 0010:__disarm_kprobe_ftrace.isra.19 (kernel/kprobes.c:1129)
Code: 24 8b 02 eb c1 80 3d c4 83 f2 01 00 75 d4 48 8b 75 00 89 c2 48 c7 c7 90 fa 0f 92 89 04 24 c6 05 ab 83 01 e8 e4 94 f0 ff <0f> 0b 8b 04 24 eb b1 89 c6 48 c7 c7 60 fa 0f 92 89 04 24 e8 cc 94
RSP: 0018:ffff9e6ec154bd98 EFLAGS: 00010282
RAX: 0000000000000000 RBX: ffffffff930f7b00 RCX: 0000000000000001
RDX: 0000000080000001 RSI: ffffffff921461c5 RDI: 00000000ffffffff
RBP: ffff89c504286da8 R08: 0000000000000000 R09: c0000000fffeffff
R10: 0000000000000000 R11: ffff9e6ec154bc28 R12: ffff89c502394e40
R13: ffff89c502394c00 R14: ffff9e6ec154bc00 R15: 0000000000000000
FS: 00007fe800398740(0000) GS:ffff89c812d80000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 000000c00057f010 CR3: 0000000103b54006 CR4: 00000000007706e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
PKRU: 55555554
Call Trace:
<TASK>
__disable_kprobe (kernel/kprobes.c:1716)
disable_kprobe (kernel/kprobes.c:2392)
__disable_trace_kprobe (kernel/trace/trace_kprobe.c:340)
disable_trace_kprobe (kernel/trace/trace_kprobe.c:429)
perf_trace_event_unreg.isra.2 (./include/linux/tracepoint.h:93 kernel/trace/trace_event_perf.c:168)
perf_kprobe_destroy (kernel/trace/trace_event_perf.c:295)
_free_event (kernel/events/core.c:4971)
perf_event_release_kernel (kernel/events/core.c:5176)
perf_release (kernel/events/core.c:5186)
__fput (fs/file_table.c:321)
task_work_run (./include/linux/sched.h:2056 (discriminator 1) kernel/task_work.c:179 (discriminator 1))
exit_to_user_mode_prepare (./include/linux/resume_user_mode.h:49 kernel/entry/common.c:169 kernel/entry/common.c:201)
syscall_exit_to_user_mode (./arch/x86/include/asm/jump_label.h:55 ./arch/x86/include/asm/nospec-branch.h:384 ./arch/x86/include/asm/entry-common.h:94 kernel/entry/common.c:133 kernel/entry/common.c:296)
do_syscall_64 (arch/x86/entry/common.c:87)
entry_SYSCALL_64_after_hwframe (arch/x86/entry/entry_64.S:120)
RIP: 0033:0x7fe7ff210654
Code: 15 79 89 20 00 f7 d8 64 89 02 48 c7 c0 ff ff ff ff eb be 0f 1f 00 8b 05 9a cd 20 00 48 63 ff 85 c0 75 11 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 3a f3 c3 48 83 ec 18 48 89 7c 24 08 e8 34 fc
RSP: 002b:00007ffdbd1d3538 EFLAGS: 00000246 ORIG_RAX: 0000000000000003
RAX: 0000000000000000 RBX: 0000000000000008 RCX: 00007fe7ff210654
RDX: 0000000000000000 RSI: 0000000000002401 RDI: 0000000000000008
RBP: 0000000000000000 R08: 94ae31d6fda838a4 R0900007fe8001c9d30
R10: 00007ffdbd1d34b0 R11: 0000000000000246 R12: 00007ffdbd1d3600
R13: 0000000000000000 R14: fffffffffffffffc R15: 00007ffdbd1d3560
</TASK>
Link: https://lkml.kernel.org/r/20220813020509.90805-1-kuniyu@amazon.com
Fixes: 69d54b916d83 ("kprobes: makes kprobes/enabled works correctly for optimized kprobes.")
Signed-off-by: Kuniyuki Iwashima <kuniyu@amazon.com>
Reported-by: Ayushman Dutta <ayudutta@amazon.com>
Cc: "Naveen N. Rao" <naveen.n.rao@linux.ibm.com>
Cc: Anil S Keshavamurthy <anil.s.keshavamurthy@intel.com>
Cc: "David S. Miller" <davem@davemloft.net>
Cc: Masami Hiramatsu <mhiramat@kernel.org>
Cc: Wang Nan <wangnan0@huawei.com>
Cc: Kuniyuki Iwashima <kuniyu@amazon.com>
Cc: Kuniyuki Iwashima <kuni1840@gmail.com>
Cc: Ayushman Dutta <ayudutta@amazon.com>
Cc: <stable@vger.kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
---
kernel/kprobes.c | 10 ++++++----
1 file changed, 6 insertions(+), 4 deletions(-)
--- a/kernel/kprobes.c
+++ b/kernel/kprobes.c
@@ -1687,12 +1687,14 @@ static struct kprobe *__disable_kprobe(s
/* Try to disarm and disable this/parent probe */
if (p == orig_p || aggr_kprobe_disabled(orig_p)) {
/*
- * If kprobes_all_disarmed is set, orig_p
- * should have already been disarmed, so
- * skip unneed disarming process.
+ * Don't be lazy here. Even if 'kprobes_all_disarmed'
+ * is false, 'orig_p' might not have been armed yet.
+ * Note arm_all_kprobes() __tries__ to arm all kprobes
+ * on the best effort basis.
*/
- if (!kprobes_all_disarmed)
+ if (!kprobes_all_disarmed && !kprobe_disabled(orig_p))
disarm_kprobe(orig_p, true);
+
orig_p->flags |= KPROBE_FLAG_DISABLED;
}
}
next prev parent reply other threads:[~2022-09-02 12:29 UTC|newest]
Thread overview: 46+ messages / expand[flat|nested] mbox.gz Atom feed top
2022-09-02 12:18 [PATCH 4.14 00/42] 4.14.292-rc1 review Greg Kroah-Hartman
2022-09-02 12:18 ` [PATCH 4.14 01/42] audit: fix potential double free on error path from fsnotify_add_inode_mark Greg Kroah-Hartman
2022-09-02 12:18 ` [PATCH 4.14 02/42] parisc: Fix exception handler for fldw and fstw instructions Greg Kroah-Hartman
2022-09-02 12:18 ` [PATCH 4.14 03/42] pinctrl: amd: Dont save/restore interrupt status and wake status bits Greg Kroah-Hartman
2022-09-02 12:18 ` [PATCH 4.14 04/42] xfrm: fix refcount leak in __xfrm_policy_check() Greg Kroah-Hartman
2022-09-02 12:18 ` [PATCH 4.14 05/42] af_key: Do not call xfrm_probe_algs in parallel Greg Kroah-Hartman
2022-09-02 12:18 ` [PATCH 4.14 06/42] rose: check NULL rose_loopback_neigh->loopback Greg Kroah-Hartman
2022-09-02 12:18 ` [PATCH 4.14 07/42] bonding: 802.3ad: fix no transmission of LACPDUs Greg Kroah-Hartman
2022-09-02 12:18 ` [PATCH 4.14 08/42] net: ipvtap - add __init/__exit annotations to module init/exit funcs Greg Kroah-Hartman
2022-09-02 12:18 ` [PATCH 4.14 09/42] netfilter: ebtables: reject blobs that dont provide all entry points Greg Kroah-Hartman
2022-09-02 12:18 ` [PATCH 4.14 10/42] netfilter: nft_payload: report ERANGE for too long offset and length Greg Kroah-Hartman
2022-09-02 12:18 ` [PATCH 4.14 11/42] netfilter: nft_payload: do not truncate csum_offset and csum_type Greg Kroah-Hartman
2022-09-02 12:18 ` [PATCH 4.14 12/42] net: Fix data-races around weight_p and dev_weight_[rt]x_bias Greg Kroah-Hartman
2022-09-02 12:18 ` [PATCH 4.14 13/42] ratelimit: Fix data-races in ___ratelimit() Greg Kroah-Hartman
2022-09-02 12:18 ` [PATCH 4.14 14/42] net: Fix a data-race around sysctl_tstamp_allow_data Greg Kroah-Hartman
2022-09-02 12:18 ` [PATCH 4.14 15/42] net: Fix a data-race around sysctl_net_busy_poll Greg Kroah-Hartman
2022-09-02 12:18 ` [PATCH 4.14 16/42] net: Fix a data-race around sysctl_net_busy_read Greg Kroah-Hartman
2022-09-02 12:18 ` [PATCH 4.14 17/42] net: Fix a data-race around netdev_budget Greg Kroah-Hartman
2022-09-02 12:18 ` [PATCH 4.14 18/42] net: Fix a data-race around netdev_budget_usecs Greg Kroah-Hartman
2022-09-02 12:18 ` [PATCH 4.14 19/42] net: Fix a data-race around sysctl_somaxconn Greg Kroah-Hartman
2022-09-02 12:18 ` [PATCH 4.14 20/42] ixgbe: stop resetting SYSTIME in ixgbe_ptp_start_cyclecounter Greg Kroah-Hartman
2022-09-02 12:18 ` [PATCH 4.14 21/42] btrfs: check if root is readonly while setting security xattr Greg Kroah-Hartman
2022-09-02 12:18 ` [PATCH 4.14 22/42] loop: Check for overflow while configuring loop Greg Kroah-Hartman
2022-09-02 12:18 ` [PATCH 4.14 23/42] asm-generic: sections: refactor memory_intersects Greg Kroah-Hartman
2022-09-02 12:18 ` [PATCH 4.14 24/42] mm/hugetlb: fix hugetlb not supporting softdirty tracking Greg Kroah-Hartman
2022-09-02 12:18 ` [PATCH 4.14 25/42] md: call __md_stop_writes in md_stop Greg Kroah-Hartman
2022-09-02 12:18 ` [PATCH 4.14 26/42] mm: Force TLB flush for PFNMAP mappings before unlink_file_vma() Greg Kroah-Hartman
2022-09-02 12:18 ` [PATCH 4.14 27/42] arm64: map FDT as RW for early_init_dt_scan() Greg Kroah-Hartman
2022-09-02 12:18 ` [PATCH 4.14 28/42] s390/mm: do not trigger write fault when vma does not allow VM_WRITE Greg Kroah-Hartman
2022-09-02 12:18 ` [PATCH 4.14 29/42] x86/cpu: Add Tiger Lake to Intel family Greg Kroah-Hartman
2022-09-02 12:18 ` [PATCH 4.14 30/42] x86/bugs: Add "unknown" reporting for MMIO Stale Data Greg Kroah-Hartman
2022-09-02 12:18 ` [PATCH 4.14 31/42] kbuild: Fix include path in scripts/Makefile.modpost Greg Kroah-Hartman
2022-09-02 12:18 ` [PATCH 4.14 32/42] Bluetooth: L2CAP: Fix build errors in some archs Greg Kroah-Hartman
2022-09-02 12:18 ` [PATCH 4.14 33/42] media: pvrusb2: fix memory leak in pvr_probe Greg Kroah-Hartman
2022-09-02 12:18 ` [PATCH 4.14 34/42] HID: hidraw: fix memory leak in hidraw_release() Greg Kroah-Hartman
2022-09-02 12:18 ` [PATCH 4.14 35/42] fbdev: fb_pm2fb: Avoid potential divide by zero error Greg Kroah-Hartman
2022-09-02 12:19 ` [PATCH 4.14 36/42] ftrace: Fix NULL pointer dereference in is_ftrace_trampoline when ftrace is dead Greg Kroah-Hartman
2022-09-02 12:19 ` [PATCH 4.14 37/42] mm/rmap: Fix anon_vma->degree ambiguity leading to double-reuse Greg Kroah-Hartman
2022-09-02 12:19 ` [PATCH 4.14 38/42] neigh: fix possible DoS due to net iface start/stop loop Greg Kroah-Hartman
2022-09-02 12:19 ` [PATCH 4.14 39/42] s390/hypfs: avoid error message under KVM Greg Kroah-Hartman
2022-09-02 12:19 ` [PATCH 4.14 40/42] netfilter: conntrack: NF_CONNTRACK_PROCFS should no longer default to y Greg Kroah-Hartman
2022-09-02 12:19 ` Greg Kroah-Hartman [this message]
2022-09-02 12:19 ` [PATCH 4.14 42/42] net: neigh: dont call kfree_skb() under spin_lock_irqsave() Greg Kroah-Hartman
2022-09-02 16:35 ` [PATCH 4.14 00/42] 4.14.292-rc1 review Jon Hunter
2022-09-03 0:34 ` Guenter Roeck
2022-09-03 13:14 ` Naresh Kamboju
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20220902121400.172910355@linuxfoundation.org \
--to=gregkh@linuxfoundation.org \
--cc=akpm@linux-foundation.org \
--cc=anil.s.keshavamurthy@intel.com \
--cc=ayudutta@amazon.com \
--cc=davem@davemloft.net \
--cc=kuni1840@gmail.com \
--cc=kuniyu@amazon.com \
--cc=linux-kernel@vger.kernel.org \
--cc=mhiramat@kernel.org \
--cc=naveen.n.rao@linux.ibm.com \
--cc=stable@vger.kernel.org \
--cc=wangnan0@huawei.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.