* Re: [BUG] tracing: Too many tries to read user space
2026-07-08 12:37 ` [BUG] tracing: Too many tries to read user space Jeongho Choi
@ 2026-07-08 13:18 ` Steven Rostedt
2026-07-09 0:04 ` Masami Hiramatsu
2026-07-10 3:22 ` Masami Hiramatsu
1 sibling, 1 reply; 6+ messages in thread
From: Steven Rostedt @ 2026-07-08 13:18 UTC (permalink / raw)
To: Jeongho Choi
Cc: linux-trace-kernel, linux-kernel, mhiramat, ji2yoon.jo,
minki.jang, hajun.sung
On Wed, 8 Jul 2026 21:37:53 +0900
Jeongho Choi <jh1012.choi@samsung.com> wrote:
> The code at the WARN location mentioned in the log above is as follows.
>
> 7374 if (WARN_ONCE(trys++ > 100, "Error: Too many
> tries to read user space"))
> 7375 return NULL;
>
This happens when something forces a schedule.
>
> Our current analysis is as follows:
>
> In the Gmail process, during a low memory situation, LMKD writes strings
> to /sys/kernel/tracing/trace_marker for systrace recording. At the same
> time, it broadcasts a sigkill due to low memory, which is causing the
> LMKD trace marker operation to stall.
>
Can you see what is being scheduled in? Perhaps use the persistent ring
buffer (if you can) and enable sched_switch tracepoint in it.
The loop is this:
do {
/*
* It is possible that something is trying to migrate this
* task. What happens then, is when preemption is enabled,
* the migration thread will preempt this task, try to
* migrate it, fail, then let it run again. That will
* cause this to loop again and never succeed.
* On failures, enabled and disable preemption with
* migration enabled, to allow the migration thread to
* migrate this task.
*/
if (trys) {
preempt_enable_notrace();
preempt_disable_notrace();
cpu = smp_processor_id();
buffer = per_cpu_ptr(tinfo->tbuf, cpu)->buf;
}
/*
* If for some reason, copy_from_user() always causes a context
* switch, this would then cause an infinite loop.
* If this task is preempted by another user space task, it
* will cause this task to try again. But just in case something
* changes where the copying from user space causes another task
* to run, prevent this from going into an infinite loop.
* 100 tries should be plenty.
*/
if (WARN_ONCE(trys++ > 100, "Error: Too many tries to read user space"))
return NULL;
/* Read the current CPU context switch counter */
cnt = nr_context_switches_cpu(cpu);
/*
* Preemption is going to be enabled, but this task must
* remain on this CPU.
*/
migrate_disable();
/*
* Now preemption is being enabled and another task can come in
* and use the same buffer and corrupt our data.
*/
preempt_enable_notrace();
/* Make sure preemption is enabled here */
lockdep_assert_preemption_enabled();
if (copy_func) {
ret = copy_func(buffer, ptr, size, data);
} else {
ret = __copy_from_user(buffer, ptr, size);
}
preempt_disable_notrace();
migrate_enable();
/* if it faulted, no need to test if the buffer was corrupted */
if (ret)
return NULL;
/*
* Preemption is disabled again, now check the per CPU context
* switch counter. If it doesn't match, then another user space
* process may have schedule in and corrupted our buffer. In that
* case the copying must be retried.
*/
} while (nr_context_switches_cpu(cpu) != cnt);
What it does is to write into a per CPU buffer from user space. If it
schedules out while trying this, it will try again as another task could
have come in and corrupted the buffer.
For some reason, when preemption is enabled to copy from user, something
is forcing a schedule of the process. If we can figure out what is doing
that, perhaps we can fix this. A similar thing happened with the migration
thread that commit edca33a56297d ("tracing: Fix failure to read user space
from system call trace events") fixed.
-- Steve
^ permalink raw reply [flat|nested] 6+ messages in thread* Re: [BUG] tracing: Too many tries to read user space
2026-07-08 12:37 ` [BUG] tracing: Too many tries to read user space Jeongho Choi
2026-07-08 13:18 ` Steven Rostedt
@ 2026-07-10 3:22 ` Masami Hiramatsu
2026-07-10 11:46 ` Steven Rostedt
2026-07-10 12:33 ` Steven Rostedt
1 sibling, 2 replies; 6+ messages in thread
From: Masami Hiramatsu @ 2026-07-10 3:22 UTC (permalink / raw)
To: Jeongho Choi
Cc: linux-trace-kernel, linux-kernel, rostedt, ji2yoon.jo, minki.jang,
hajun.sung
On Wed, 8 Jul 2026 21:37:53 +0900
Jeongho Choi <jh1012.choi@samsung.com> wrote:
> Hello,
>
> We are seeing a reproducible kernel panic related to the tracing code
> when it fails to read user-space memory.
>
> The issue was originally reported through the Android/Google Issue
> Tracker, and we were advised to report it to the upstream trace mailing
> list because the affected code is upstream.
>
> Environment:
>
> Architecture: arm64
> Kernel: Linux 6.18.21
> Base: Android Common Kernel (android17-6.18)
> Affected area: kernel/trace/
>
> The relevant error/panic log is:
> [48916.569148] [9: lmkd: 536] Error: Too many tries to read
> user space
> [48916.569156] [9: lmkd: 536] WARNING: CPU: 9 PID: 536 at
> kernel/trace/trace.c:7374 trace_user_fault_read+0x334/0x360
> [48916.569443] [9: lmkd: 536] CPU: 9 UID: 1069 PID: 536 Comm:
> lmkd Tainted: G OE 6.18.21-android17-5-ga1a8e8cab9ec-4k
> #1 PREEMPT 25372cd4750dcac3c0fe86b57d47c665f97a6046
>
> [48916.569450] [9: lmkd: 536] pstate: 63402005 (nZCv daif
> +PAN -UAO +TCO +DIT -SSBS BTYPE=--)
> [48916.569452] [9: lmkd: 536] pc :
> trace_user_fault_read+0x334/0x360
> [48916.569454] [9: lmkd: 536] lr :
> trace_user_fault_read+0x330/0x360
> [48916.569456] [9: lmkd: 536] sp : ffffffc096993cc0
> [48916.569457] [9: lmkd: 536] x29: ffffffc096993cd0 x28:
> 0000000000000065 x27: ffffff8817e45200
> [48916.569461] [9: lmkd: 536] x26: 0000000000000000 x25:
> 000000011616c082 x24: 0000000000000000
> [48916.569463] [9: lmkd: 536] x23: 0000000000000009 x22:
> 000000000000002b x21: ffffff880597f740
> [48916.569466] [9: lmkd: 536] x20: 0000007fc42dd940 x19:
> ffffff8817e45770 x18: ffffffd5947ce240
> [48916.569468] [9: lmkd: 536] x17: 2073656972742079 x16:
> 6e616d206f6f5420 x15: 3a726f727245205d
> [48916.569471] [9: lmkd: 536] x14: 36333520203a646b x13:
> 6563617073207265 x12: 0000000000000001
> [48916.569473] [9: lmkd: 536] x11: 6f74207365697274 x10:
> 0000000000000001 x9 : 4bd9ad4516d75100
> [48916.569476] [9: lmkd: 536] x8 : 4bd9ad4516d75100 x7 :
> 205d383431393635 x6 : 2e36313938345b0a
> [48916.569479] [9: lmkd: 536] x5 : ffffffc080fa5998 x4 :
> ffffffd591707202 x3 : 0001360a00000000
> [48916.569481] [9: lmkd: 536] x2 : ffffffc096993af4 x1 :
> 00000000000000c0 x0 : 0000000000000028
> [48916.569484] [9: lmkd: 536] Call trace:
> [48916.569486] [9: lmkd: 536]
> trace_user_fault_read+0x334/0x360 (P)
> [48916.569488] [9: lmkd: 536] tracing_mark_write+0x84/0x174
> [48916.569491] [9: lmkd: 536] __arm64_sys_write+0x2a0/0x5c0
> [48916.569494] [9: lmkd: 536] invoke_syscall+0x58/0xe4
> [48916.569498] [9: lmkd: 536] do_el0_svc+0x48/0xdc
> [48916.569500] [9: lmkd: 536] el0_svc+0x3c/0x98
> [48916.569503] [9: lmkd: 536]
> el0t_64_sync_handler+0x20/0x130
> [48916.569505] [9: lmkd: 536] el0t_64_sync+0x1c4/0x1c8
> [48916.569508] [9: lmkd: 536] Kernel panic - not syncing:
> kernel: panic_on_warn set ...
>
>
> The code at the WARN location mentioned in the log above is as follows.
>
> 7374 if (WARN_ONCE(trys++ > 100, "Error: Too many
> tries to read user space"))
> 7375 return NULL;
>
>
> Our current analysis is as follows:
>
> In the Gmail process, during a low memory situation, LMKD writes strings
> to /sys/kernel/tracing/trace_marker for systrace recording. At the same
> time, it broadcasts a sigkill due to low memory, which is causing the
> LMKD trace marker operation to stall.
Hm, in my view, this warning indicates that the circuit breaker has
triggered correctly, so that is not a bug. Under the heavy memory
pressure and low-memory situation, the page can be reclaimed soon
after it is copied.
However, this seems a bit strange that we only checks the CPU-wide context
switching in the loop. Instead, can we introduce a per-cpu sequence counter
to per-cpu buffer, and check it?
Could you try this ?
From f76d8e4400a5961725d17899f4290c9334987e2b Mon Sep 17 00:00:00 2001
From: Masami Hiramatsu <mhiramat@kernel.org>
Date: Fri, 10 Jul 2026 12:11:00 +0900
Subject: [PATCH] tracing: Use per-CPU sequence counter in
trace_user_fault_read
trace_user_fault_read() copies trace data from user space to the
per-CPU trace buffer. When preemption is enabled during the copy, it
checks if any context switches occurred on the current CPU via
nr_context_switches_cpu() to detect whether the buffer may have been
corrupted by another trace writer.
However, under heavy memory pressure, copying from user space can trigger
page faults (e.g., for swapped-out BSS or anonymous pages) that block and
cause a context switch. Because nr_context_switches_cpu() detects any
context switch (even unrelated ones), it mistakenly assumes the buffer was
corrupted. This leads to repeated retries (up to the 100-try limit), which
causes a WARN_ONCE backtrace and returns -EFAULT to user space, even if
no other task ever accessed the trace buffer.
To mitigate this issue, replace the CPU-wide context switch check with
a dedicated per-CPU sequence counter in struct trace_user_buf.
Since only other tasks invoking trace_user_fault_read() on the same CPU
will increment this counter, unrelated context switches (including those
from page fault sleep) will no longer trigger retries.
Signed-off-by: Masami Hiramatsu (Google) <mhiramat@kernel.org>
---
kernel/trace/trace.c | 29 ++++++++++++++++++-----------
1 file changed, 18 insertions(+), 11 deletions(-)
diff --git a/kernel/trace/trace.c b/kernel/trace/trace.c
index 1bc27c0ad029..46cec63f5798 100644
--- a/kernel/trace/trace.c
+++ b/kernel/trace/trace.c
@@ -5989,6 +5989,7 @@ static ssize_t write_marker_to_buffer(struct trace_array *tr, const char *buf,
struct trace_user_buf {
char *buf;
+ unsigned int sequence;
};
static DEFINE_MUTEX(trace_user_buffer_mutex);
@@ -6031,7 +6032,10 @@ static int user_fault_buffer_enable(struct trace_user_buf_info *tinfo, size_t si
/* Clear each buffer in case of error */
for_each_possible_cpu(cpu) {
- per_cpu_ptr(tinfo->tbuf, cpu)->buf = NULL;
+ struct trace_user_buf *tbuf = per_cpu_ptr(tinfo->tbuf, cpu);
+
+ tbuf->buf = NULL;
+ tbuf->sequence = 0;
}
for_each_possible_cpu(cpu) {
@@ -6196,8 +6200,9 @@ char *trace_user_fault_read(struct trace_user_buf_info *tinfo,
trace_user_buf_copy copy_func, void *data)
{
int cpu = smp_processor_id();
- char *buffer = per_cpu_ptr(tinfo->tbuf, cpu)->buf;
- unsigned int cnt;
+ struct trace_user_buf *tbuf = per_cpu_ptr(tinfo->tbuf, cpu);
+ char *buffer = tbuf->buf;
+ unsigned int seq;
int trys = 0;
int ret;
@@ -6211,10 +6216,10 @@ char *trace_user_fault_read(struct trace_user_buf_info *tinfo,
return NULL;
/*
- * This acts similar to a seqcount. The per CPU context switches are
+ * This acts similar to a seqcount. The per CPU sequence counters are
* recorded, migration is disabled and preemption is enabled. The
* read of the user space memory is copied into the per CPU buffer.
- * Preemption is disabled again, and if the per CPU context switches count
+ * Preemption is disabled again, and if the per CPU sequence count
* is still the same, it means the buffer has not been corrupted.
* If the count is different, it is assumed the buffer is corrupted
* and reading must be tried again.
@@ -6235,7 +6240,8 @@ char *trace_user_fault_read(struct trace_user_buf_info *tinfo,
preempt_enable_notrace();
preempt_disable_notrace();
cpu = smp_processor_id();
- buffer = per_cpu_ptr(tinfo->tbuf, cpu)->buf;
+ tbuf = per_cpu_ptr(tinfo->tbuf, cpu);
+ buffer = tbuf->buf;
}
/*
@@ -6250,8 +6256,9 @@ char *trace_user_fault_read(struct trace_user_buf_info *tinfo,
if (WARN_ONCE(trys++ > 100, "Error: Too many tries to read user space"))
return NULL;
- /* Read the current CPU context switch counter */
- cnt = nr_context_switches_cpu(cpu);
+ /* Increment the per-CPU buffer sequence counter */
+ tbuf->sequence++;
+ seq = tbuf->sequence;
/*
* Preemption is going to be enabled, but this task must
@@ -6282,12 +6289,12 @@ char *trace_user_fault_read(struct trace_user_buf_info *tinfo,
return NULL;
/*
- * Preemption is disabled again, now check the per CPU context
- * switch counter. If it doesn't match, then another user space
+ * Preemption is disabled again, now check the per CPU sequence
+ * counter. If it doesn't match, then another user space
* process may have schedule in and corrupted our buffer. In that
* case the copying must be retried.
*/
- } while (nr_context_switches_cpu(cpu) != cnt);
+ } while (tbuf->sequence != seq);
return buffer;
}
--
2.43.0
--
Masami Hiramatsu (Google) <mhiramat@kernel.org>
^ permalink raw reply related [flat|nested] 6+ messages in thread