public inbox for linux-trace-kernel@vger.kernel.org
 help / color / mirror / Atom feed
* [BUG] kprobes: WARNING in __arm_kprobe_ftrace when kprobe-ftrace arming fails with -ENOMEM under fault injection
@ 2026-03-02  9:00 Zw Tang
  2026-03-02 17:35 ` Sasha Levin
  0 siblings, 1 reply; 3+ messages in thread
From: Zw Tang @ 2026-03-02  9:00 UTC (permalink / raw)
  To: Naveen N Rao, Masami Hiramatsu, Steven Rostedt
  Cc: linux-kernel, linux-trace-kernel, linux-perf-users,
	Arnaldo Carvalho de Melo

Hi,

I am reporting a WARNING triggered by a syzkaller reproducer on Linux 7.0.0-rc1.

The kernel hits a WARN in kprobes while trying to arm a kprobe via ftrace:

Failed to arm kprobe-ftrace at __split_text_end+0x4/0x11 (error -12)
WARNING: kernel/kprobes.c:1147 at __arm_kprobe_ftrace()

This seems to be triggered through perf_event_open() -> trace_kprobe
-> kprobes. The reproducer enables systematic fault injection and
injects a failure (nth=7), and the arming path returns -ENOMEM (-12).
Instead of cleanly failing, kprobes emits a WARNING.

This is reproducible only with fault injection enabled.

Reproducer:
C reproducer: https://pastebin.com/raw/casZvuLe
console output: https://pastebin.com/raw/1xkwRUmc
kernel config: https://pastebin.com/raw/8Er8SZz0

Kernel:
git tree: torvalds/linux
commit: 4d349ee5c7782f8b27f6cb550f112c5e26fff38d
kernel version: 7.0.0-rc1-00301-g4d349ee5c778 #5 PREEMPT_RT (lazy)
hardware: QEMU Ubuntu 24.10



[   92.516728] WARNING: kernel/kprobes.c:1147 at
arm_kprobe+0x563/0x620, CPU#0: syz.1.94/783
[   92.516766] Modules linked in:
[   92.516809] CPU: 0 UID: 0 PID: 783 Comm: syz.1.94 Not tainted
7.0.0-rc1-00301-g4d349ee5c778 #5 PREEMPT_{RT,(lazy)}
0b4dbcd6f14740930e77a74387d10aec6dbca841
[   92.516842] Hardware name: QEMU Ubuntu 24.10 PC (i440FX + PIIX,
1996), BIOS 1.16.3-debian-1.16.3-2 04/01/2014
[   92.516855] RIP: 0010:arm_kprobe+0x56a/0x620
[   92.516885] Code: ff 4c 89 fa 48 b8 00 00 00 00 00 fc ff df 48 c1
ea 03 80 3c 02 00 0f 85 a8 00 00 00 48 8d 3d cd d3 8d 06 48 8b 75 28
44 89 e2 <67> 48 0f b9 3a e9 81 fc ff ff e8 87 8c ff ff 48 8d 3d c0 d3
8d 06
[   92.516905] RSP: 0018:ffff88800faf7a48 EFLAGS: 00010246
[   92.516924] RAX: dffffc0000000000 RBX: ffffffff89f46b40 RCX: 0000000000000000
[   92.516939] RDX: 00000000fffffff4 RSI: ffffffff81200004 RDI: ffffffff88481300
[   92.516955] RBP: ffff88800c566a18 R08: 0000000000000000 R09: fffffbfff108bacb
[   92.516969] R10: fffffbfff108baca R11: ffffffff8845d657 R12: 00000000fffffff4
[   92.516984] R13: ffffffff8845dc20 R14: ffff88800c566a90 R15: ffff88800c566a40
[   92.517002] FS:  00007f42ed38f6c0(0000) GS:ffff8880e224e000(0000)
knlGS:0000000000000000
[   92.517023] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[   92.517041] CR2: 00007fe44aa68710 CR3: 000000000e340000 CR4: 0000000000350ef0
[   92.517057] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[   92.517073] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000600
[   92.517090] Call Trace:
[   92.517099]  <TASK>
[   92.517126]  enable_kprobe+0x1fc/0x2c0
[   92.517173]  enable_trace_kprobe+0x227/0x4b0
[   92.517240]  kprobe_register+0x84/0xc0
[   92.517279]  perf_trace_event_init+0x527/0xa20
[   92.517329]  perf_kprobe_init+0x156/0x200
[   92.517367]  perf_kprobe_event_init+0x101/0x1c0
[   92.517406]  perf_try_init_event+0x145/0xa10
[   92.517458]  perf_event_alloc+0x1f91/0x5390
[   92.517509]  ? perf_event_alloc+0x1e4d/0x5390
[   92.517586]  ? perf_event_mmap_output+0xf00/0xf00
[   92.517709]  __do_sys_perf_event_open+0x557/0x2d50
[   92.517762]  ? write_comp_data+0x29/0x80
[   92.517788]  ? irqentry_exit+0x157/0xb20
[   92.517822]  ? perf_release+0x50/0x50
[   92.517848]  ? irqentry_exit+0x157/0xb20
[   92.517897]  ? __split_text_end+0x4/0x11
[   92.517956]  ? tracer_hardirqs_on+0x80/0x3b0
[   92.517986]  ? do_syscall_64+0x94/0x1160
[   92.518022]  ? __sanitizer_cov_trace_pc+0x20/0x50
[   92.518072]  do_syscall_64+0x129/0x1160
[   92.518118]  entry_SYSCALL_64_after_hwframe+0x4b/0x53
[   92.518142] RIP: 0033:0x7f42ee92ebe9
[   92.518164] Code: ff ff c3 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40
00 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24
08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 c7 c1 a8 ff ff ff f7 d8 64 89
01 48
[   92.518184] RSP: 002b:00007f42ed38f038 EFLAGS: 00000246 ORIG_RAX:
000000000000012a
[   92.518207] RAX: ffffffffffffffda RBX: 00007f42eeb65fa0 RCX: 00007f42ee92ebe9
[   92.518222] RDX: 0000000000000000 RSI: ffffffffffffffff RDI: 0000200000000140
[   92.518248] RBP: 00007f42ed38f090 R08: 0000000000000008 R09: 0000000000000000
[   92.518262] R10: ffffffffffffffff R11: 0000000000000246 R12: 0000000000000001
[   92.518277] R13: 00007f42eeb66038 R14: 00007f42eeb65fa0 R15: 00007fff172e7218
[   92.518358]  </TASK>



Notes:

The reproducer sets up fault injection (/proc/thread-self/fail-nth,
failslab/fail_page_alloc knobs) and injects nth=7 before calling
perf_event_open().

The failure is reported as -ENOMEM when arming kprobe-ftrace, and the
WARN is triggered in __arm_kprobe_ftrace().


Thanks,
Zw Tang

^ permalink raw reply	[flat|nested] 3+ messages in thread

* Re: [BUG] kprobes: WARNING in __arm_kprobe_ftrace when kprobe-ftrace arming fails with -ENOMEM under fault injection
  2026-03-02  9:00 [BUG] kprobes: WARNING in __arm_kprobe_ftrace when kprobe-ftrace arming fails with -ENOMEM under fault injection Zw Tang
@ 2026-03-02 17:35 ` Sasha Levin
  2026-03-04  1:44   ` Masami Hiramatsu
  0 siblings, 1 reply; 3+ messages in thread
From: Sasha Levin @ 2026-03-02 17:35 UTC (permalink / raw)
  To: Zw Tang, Naveen N Rao, Masami Hiramatsu, Steven Rostedt
  Cc: Sasha Levin, linux-kernel, linux-trace-kernel, linux-perf-users,
	Arnaldo Carvalho de Melo, David S . Miller

https://lore.kernel.org/all/CAPHJ_V+J6YDb_wX2nhXU6kh466Dt_nyDSas-1i_Y8s7tqY-Mzw@mail.gmail.com/

## 1. Bug Summary

A WARNING is triggered in `__arm_kprobe_ftrace()` at `kernel/kprobes.c:1147` when fault injection causes `ftrace_set_filter_ip()` to return `-ENOMEM` during kprobe arming via `perf_event_open()`. This is a false-positive warning — the error path itself is correct and the error propagates cleanly to userspace, but the `WARN_ONCE()` macro fires a kernel warning splat that is inappropriate for a recoverable allocation failure. The affected subsystem is kprobes/ftrace. Severity: warning only (no crash, hang, or data corruption).

## 2. Stack Trace Analysis

```
WARNING: kernel/kprobes.c:1147 at arm_kprobe+0x563/0x620, CPU#0
Call Trace:
 <TASK>
 enable_kprobe+0x1fc/0x2c0
 enable_trace_kprobe+0x227/0x4b0
 kprobe_register+0x84/0xc0
 perf_trace_event_init+0x527/0xa20
 perf_kprobe_init+0x156/0x200
 perf_kprobe_event_init+0x101/0x1c0
 perf_try_init_event+0x145/0xa10
 perf_event_alloc+0x1f91/0x5390
 __do_sys_perf_event_open+0x557/0x2d50
 do_syscall_64+0x129/0x1160
 entry_SYSCALL_64_after_hwframe+0x4b/0x53
 </TASK>
```

The crash point is `__arm_kprobe_ftrace()` at `kernel/kprobes.c:1147`, inlined into `arm_kprobe()`. The calling chain is process context: `perf_event_open()` syscall -> `perf_kprobe_event_init()` -> `enable_trace_kprobe()` -> `enable_kprobe()` -> `arm_kprobe()` -> `__arm_kprobe_ftrace()`. R12 holds `0xfffffff4` which is `-12` (`-ENOMEM`), confirming the allocation failure injected by fault injection.

## 3. Root Cause Analysis

The root cause is an overly aggressive `WARN_ONCE()` in `__arm_kprobe_ftrace()` at `kernel/kprobes.c:1147`:

```c
ret = ftrace_set_filter_ip(ops, (unsigned long)p->addr, 0, 0);
if (WARN_ONCE(ret < 0, "Failed to arm kprobe-ftrace at %pS (error %d)\n", p->addr, ret))
    return ret;
```

Prior to commit 9c89bb8e3272 ("kprobes: treewide: Cleanup the error messages for kprobes"), this was a simple `pr_debug()`. That commit promoted it to `WARN_ONCE()` as part of a treewide message cleanup, under the rationale that failures here indicate unexpected conditions. However, `ftrace_set_filter_ip()` calls into memory allocation paths (e.g., `ftrace_hash_move_and_update_ops()` -> `__ftrace_hash_update_ipmodify()` or `ftrace_hash` allocation), and those allocations can legitimately fail under memory pressure or fault injection.

The error handling is actually correct — the `-ENOMEM` propagates back through `arm_kprobe()` -> `enable_kprobe()` and ultimately causes the `perf_event_open()` syscall to return an error to userspace. The only problem is the spurious `WARN_ONCE()` which triggers a kernel warning splat and stack trace for what is a recoverable, non-buggy situation.

The same issue also applies to the `WARN()` on line 1152 for `register_ftrace_function()`, which can also fail with `-ENOMEM`.

## 4. Affected Versions

This issue was introduced by commit 9c89bb8e3272 ("kprobes: treewide: Cleanup the error messages for kprobes"), which first appeared in v5.16-rc1. All kernel versions from v5.16 onward are affected, including the reporter's v7.0.0-rc1.

This is a regression from v5.15, where the same failure path used `pr_debug()` and did not emit any warning.

## 5. Relevant Commits and Fixes

Introducing commit:
  9c89bb8e3272 ("kprobes: treewide: Cleanup the error messages for kprobes")
  Author: Masami Hiramatsu <mhiramat@kernel.org>
  Merged in v5.16-rc1

This commit changed `pr_debug()` to `WARN_ONCE()` in `__arm_kprobe_ftrace()` for the `ftrace_set_filter_ip()` failure path, and changed a `pr_debug()` to `WARN()` for the `register_ftrace_function()` failure path.

No fix for this issue exists in mainline or stable as of the reporter's kernel version.

The suggested fix is to downgrade both `WARN_ONCE()` (line 1147) and `WARN()` (line 1152) in `__arm_kprobe_ftrace()` back to `pr_warn_once()` / `pr_warn()` respectively. This preserves the improved error messages from 9c89bb8e3272 while avoiding spurious warning splats on recoverable failures. The error code is already propagated correctly to the caller.

## 6. Prior Discussions

No prior reports of this specific issue were found on lore.kernel.org. No related mailing list discussions or proposed patches addressing the WARN severity in `__arm_kprobe_ftrace()` were found.

## 7. Suggested Actions

1. The fix is to downgrade the WARN_ONCE/WARN in __arm_kprobe_ftrace()
   (kernel/kprobes.c lines 1147 and 1152) to pr_warn_once/pr_warn.
   Specifically:

   - Line 1147: Change WARN_ONCE(ret < 0, ...) to a simple
     if (ret < 0) { pr_warn_once(...); return ret; }

   - Line 1152: Change WARN(ret < 0, ...) to a simple
     if (ret < 0) { pr_warn(...); goto err_ftrace; }

2. This is a low-severity issue that only manifests with fault injection
   enabled. No data corruption or crash occurs — the error is correctly
   propagated to userspace. The warning is cosmetic but noisy and can
   cause false-positive syzbot/syzkaller reports.


^ permalink raw reply	[flat|nested] 3+ messages in thread

* Re: [BUG] kprobes: WARNING in __arm_kprobe_ftrace when kprobe-ftrace arming fails with -ENOMEM under fault injection
  2026-03-02 17:35 ` Sasha Levin
@ 2026-03-04  1:44   ` Masami Hiramatsu
  0 siblings, 0 replies; 3+ messages in thread
From: Masami Hiramatsu @ 2026-03-04  1:44 UTC (permalink / raw)
  To: Sasha Levin
  Cc: Zw Tang, Naveen N Rao, Steven Rostedt, linux-kernel,
	linux-trace-kernel, linux-perf-users, Arnaldo Carvalho de Melo,
	David S . Miller

On Mon,  2 Mar 2026 12:35:49 -0500
Sasha Levin <sashal@kernel.org> wrote:

> https://lore.kernel.org/all/CAPHJ_V+J6YDb_wX2nhXU6kh466Dt_nyDSas-1i_Y8s7tqY-Mzw@mail.gmail.com/
> 
> ## 1. Bug Summary
> 
> A WARNING is triggered in `__arm_kprobe_ftrace()` at `kernel/kprobes.c:1147` when fault injection causes `ftrace_set_filter_ip()` to return `-ENOMEM` during kprobe arming via `perf_event_open()`. This is a false-positive warning — the error path itself is correct and the error propagates cleanly to userspace, but the `WARN_ONCE()` macro fires a kernel warning splat that is inappropriate for a recoverable allocation failure. The affected subsystem is kprobes/ftrace. Severity: warning only (no crash, hang, or data corruption).

Thanks for analysis and Nice catch Zw Tang!
This looks good analysis to me.

> 
> ## 2. Stack Trace Analysis
> 
> ```
> WARNING: kernel/kprobes.c:1147 at arm_kprobe+0x563/0x620, CPU#0
> Call Trace:
>  <TASK>
>  enable_kprobe+0x1fc/0x2c0
>  enable_trace_kprobe+0x227/0x4b0
>  kprobe_register+0x84/0xc0
>  perf_trace_event_init+0x527/0xa20
>  perf_kprobe_init+0x156/0x200
>  perf_kprobe_event_init+0x101/0x1c0
>  perf_try_init_event+0x145/0xa10
>  perf_event_alloc+0x1f91/0x5390
>  __do_sys_perf_event_open+0x557/0x2d50
>  do_syscall_64+0x129/0x1160
>  entry_SYSCALL_64_after_hwframe+0x4b/0x53
>  </TASK>
> ```
> 
> The crash point is `__arm_kprobe_ftrace()` at `kernel/kprobes.c:1147`, inlined into `arm_kprobe()`. The calling chain is process context: `perf_event_open()` syscall -> `perf_kprobe_event_init()` -> `enable_trace_kprobe()` -> `enable_kprobe()` -> `arm_kprobe()` -> `__arm_kprobe_ftrace()`. R12 holds `0xfffffff4` which is `-12` (`-ENOMEM`), confirming the allocation failure injected by fault injection.
> 
> ## 3. Root Cause Analysis
> 
> The root cause is an overly aggressive `WARN_ONCE()` in `__arm_kprobe_ftrace()` at `kernel/kprobes.c:1147`:
> 
> ```c
> ret = ftrace_set_filter_ip(ops, (unsigned long)p->addr, 0, 0);
> if (WARN_ONCE(ret < 0, "Failed to arm kprobe-ftrace at %pS (error %d)\n", p->addr, ret))
>     return ret;
> ```
> 
> Prior to commit 9c89bb8e3272 ("kprobes: treewide: Cleanup the error messages for kprobes"), this was a simple `pr_debug()`. That commit promoted it to `WARN_ONCE()` as part of a treewide message cleanup, under the rationale that failures here indicate unexpected conditions. However, `ftrace_set_filter_ip()` calls into memory allocation paths (e.g., `ftrace_hash_move_and_update_ops()` -> `__ftrace_hash_update_ipmodify()` or `ftrace_hash` allocation), and those allocations can legitimately fail under memory pressure or fault injection.
> 
> The error handling is actually correct — the `-ENOMEM` propagates back through `arm_kprobe()` -> `enable_kprobe()` and ultimately causes the `perf_event_open()` syscall to return an error to userspace. The only problem is the spurious `WARN_ONCE()` which triggers a kernel warning splat and stack trace for what is a recoverable, non-buggy situation.
> 
> The same issue also applies to the `WARN()` on line 1152 for `register_ftrace_function()`, which can also fail with `-ENOMEM`.
> 
> ## 4. Affected Versions
> 
> This issue was introduced by commit 9c89bb8e3272 ("kprobes: treewide: Cleanup the error messages for kprobes"), which first appeared in v5.16-rc1. All kernel versions from v5.16 onward are affected, including the reporter's v7.0.0-rc1.
> 
> This is a regression from v5.15, where the same failure path used `pr_debug()` and did not emit any warning.
> 
> ## 5. Relevant Commits and Fixes
> 
> Introducing commit:
>   9c89bb8e3272 ("kprobes: treewide: Cleanup the error messages for kprobes")
>   Author: Masami Hiramatsu <mhiramat@kernel.org>
>   Merged in v5.16-rc1
> 
> This commit changed `pr_debug()` to `WARN_ONCE()` in `__arm_kprobe_ftrace()` for the `ftrace_set_filter_ip()` failure path, and changed a `pr_debug()` to `WARN()` for the `register_ftrace_function()` failure path.
> 
> No fix for this issue exists in mainline or stable as of the reporter's kernel version.
> 
> The suggested fix is to downgrade both `WARN_ONCE()` (line 1147) and `WARN()` (line 1152) in `__arm_kprobe_ftrace()` back to `pr_warn_once()` / `pr_warn()` respectively. This preserves the improved error messages from 9c89bb8e3272 while avoiding spurious warning splats on recoverable failures. The error code is already propagated correctly to the caller.
> 
> ## 6. Prior Discussions
> 
> No prior reports of this specific issue were found on lore.kernel.org. No related mailing list discussions or proposed patches addressing the WARN severity in `__arm_kprobe_ftrace()` were found.
> 
> ## 7. Suggested Actions
> 
> 1. The fix is to downgrade the WARN_ONCE/WARN in __arm_kprobe_ftrace()
>    (kernel/kprobes.c lines 1147 and 1152) to pr_warn_once/pr_warn.
>    Specifically:
> 
>    - Line 1147: Change WARN_ONCE(ret < 0, ...) to a simple
>      if (ret < 0) { pr_warn_once(...); return ret; }
> 
>    - Line 1152: Change WARN(ret < 0, ...) to a simple
>      if (ret < 0) { pr_warn(...); goto err_ftrace; }

Sounds reasonable. Let's fix it.

> 
> 2. This is a low-severity issue that only manifests with fault injection
>    enabled. No data corruption or crash occurs — the error is correctly
>    propagated to userspace. The warning is cosmetic but noisy and can
>    cause false-positive syzbot/syzkaller reports.
> 

Yeah, it is just too aggressive warnings. Let me remove if the error
can be handled correctly.

Thank you,

-- 
Masami Hiramatsu (Google) <mhiramat@kernel.org>

^ permalink raw reply	[flat|nested] 3+ messages in thread

end of thread, other threads:[~2026-03-04  1:44 UTC | newest]

Thread overview: 3+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2026-03-02  9:00 [BUG] kprobes: WARNING in __arm_kprobe_ftrace when kprobe-ftrace arming fails with -ENOMEM under fault injection Zw Tang
2026-03-02 17:35 ` Sasha Levin
2026-03-04  1:44   ` Masami Hiramatsu

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox