Linux Trace Kernel
 help / color / mirror / Atom feed
* [BUG] tracing: Too many tries to read user space
       [not found] <CGME20260708123754epcas2p1f15cc305ddb09f97164491d750769ef7@epcas2p1.samsung.com>
@ 2026-07-08 12:37 ` Jeongho Choi
  2026-07-08 13:18   ` Steven Rostedt
  2026-07-10  3:22   ` Masami Hiramatsu
  0 siblings, 2 replies; 6+ messages in thread
From: Jeongho Choi @ 2026-07-08 12:37 UTC (permalink / raw)
  To: linux-trace-kernel, linux-kernel, rostedt, mhiramat
  Cc: ji2yoon.jo, minki.jang, hajun.sung

[-- Attachment #1: Type: text/plain, Size: 3535 bytes --]

Hello,

We are seeing a reproducible kernel panic related to the tracing code
when it fails to read user-space memory.

The issue was originally reported through the Android/Google Issue
Tracker, and we were advised to report it to the upstream trace mailing
list because the affected code is upstream.

Environment:

Architecture: arm64
Kernel: Linux 6.18.21
Base: Android Common Kernel (android17-6.18)
Affected area: kernel/trace/

The relevant error/panic log is:
[48916.569148] [9:           lmkd:  536] Error: Too many tries to read
user space
[48916.569156] [9:           lmkd:  536] WARNING: CPU: 9 PID: 536 at
kernel/trace/trace.c:7374 trace_user_fault_read+0x334/0x360
[48916.569443] [9:           lmkd:  536] CPU: 9 UID: 1069 PID: 536 Comm:
lmkd Tainted: G           OE       6.18.21-android17-5-ga1a8e8cab9ec-4k
#1 PREEMPT  25372cd4750dcac3c0fe86b57d47c665f97a6046

[48916.569450] [9:           lmkd:  536] pstate: 63402005 (nZCv daif
+PAN -UAO +TCO +DIT -SSBS BTYPE=--)
[48916.569452] [9:           lmkd:  536] pc :
trace_user_fault_read+0x334/0x360
[48916.569454] [9:           lmkd:  536] lr :
trace_user_fault_read+0x330/0x360
[48916.569456] [9:           lmkd:  536] sp : ffffffc096993cc0
[48916.569457] [9:           lmkd:  536] x29: ffffffc096993cd0 x28:
0000000000000065 x27: ffffff8817e45200
[48916.569461] [9:           lmkd:  536] x26: 0000000000000000 x25:
000000011616c082 x24: 0000000000000000
[48916.569463] [9:           lmkd:  536] x23: 0000000000000009 x22:
000000000000002b x21: ffffff880597f740
[48916.569466] [9:           lmkd:  536] x20: 0000007fc42dd940 x19:
ffffff8817e45770 x18: ffffffd5947ce240
[48916.569468] [9:           lmkd:  536] x17: 2073656972742079 x16:
6e616d206f6f5420 x15: 3a726f727245205d
[48916.569471] [9:           lmkd:  536] x14: 36333520203a646b x13:
6563617073207265 x12: 0000000000000001
[48916.569473] [9:           lmkd:  536] x11: 6f74207365697274 x10:
0000000000000001 x9 : 4bd9ad4516d75100
[48916.569476] [9:           lmkd:  536] x8 : 4bd9ad4516d75100 x7 :
205d383431393635 x6 : 2e36313938345b0a
[48916.569479] [9:           lmkd:  536] x5 : ffffffc080fa5998 x4 :
ffffffd591707202 x3 : 0001360a00000000
[48916.569481] [9:           lmkd:  536] x2 : ffffffc096993af4 x1 :
00000000000000c0 x0 : 0000000000000028
[48916.569484] [9:           lmkd:  536] Call trace:
[48916.569486] [9:           lmkd:  536]
trace_user_fault_read+0x334/0x360 (P)
[48916.569488] [9:           lmkd:  536]  tracing_mark_write+0x84/0x174
[48916.569491] [9:           lmkd:  536]  __arm64_sys_write+0x2a0/0x5c0
[48916.569494] [9:           lmkd:  536]  invoke_syscall+0x58/0xe4
[48916.569498] [9:           lmkd:  536]  do_el0_svc+0x48/0xdc
[48916.569500] [9:           lmkd:  536]  el0_svc+0x3c/0x98
[48916.569503] [9:           lmkd:  536]
el0t_64_sync_handler+0x20/0x130
[48916.569505] [9:           lmkd:  536]  el0t_64_sync+0x1c4/0x1c8
[48916.569508] [9:           lmkd:  536] Kernel panic - not syncing:
kernel: panic_on_warn set ...


The code at the WARN location mentioned in the log above is as follows.

7374                 if (WARN_ONCE(trys++ > 100, "Error: Too many
  tries to read user space"))
7375                         return NULL;


Our current analysis is as follows:

In the Gmail process, during a low memory situation, LMKD writes strings
to /sys/kernel/tracing/trace_marker for systrace recording. At the same
time, it broadcasts a sigkill due to low memory, which is causing the
LMKD trace marker operation to stall.


Please review this issue.
Thanks,
Jeongho Choi

[-- Attachment #2: Type: text/plain, Size: 0 bytes --]



^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: [BUG] tracing: Too many tries to read user space
  2026-07-08 12:37 ` [BUG] tracing: Too many tries to read user space Jeongho Choi
@ 2026-07-08 13:18   ` Steven Rostedt
  2026-07-09  0:04     ` Masami Hiramatsu
  2026-07-10  3:22   ` Masami Hiramatsu
  1 sibling, 1 reply; 6+ messages in thread
From: Steven Rostedt @ 2026-07-08 13:18 UTC (permalink / raw)
  To: Jeongho Choi
  Cc: linux-trace-kernel, linux-kernel, mhiramat, ji2yoon.jo,
	minki.jang, hajun.sung

On Wed, 8 Jul 2026 21:37:53 +0900
Jeongho Choi <jh1012.choi@samsung.com> wrote:

> The code at the WARN location mentioned in the log above is as follows.
> 
> 7374                 if (WARN_ONCE(trys++ > 100, "Error: Too many
>   tries to read user space"))
> 7375                         return NULL;
> 

This happens when something forces a schedule.

> 
> Our current analysis is as follows:
> 
> In the Gmail process, during a low memory situation, LMKD writes strings
> to /sys/kernel/tracing/trace_marker for systrace recording. At the same
> time, it broadcasts a sigkill due to low memory, which is causing the
> LMKD trace marker operation to stall.
> 

Can you see what is being scheduled in? Perhaps use the persistent ring
buffer (if you can) and enable sched_switch tracepoint in it.

The loop is this:

	do {
		/*
		 * It is possible that something is trying to migrate this
		 * task. What happens then, is when preemption is enabled,
		 * the migration thread will preempt this task, try to
		 * migrate it, fail, then let it run again. That will
		 * cause this to loop again and never succeed.
		 * On failures, enabled and disable preemption with
		 * migration enabled, to allow the migration thread to
		 * migrate this task.
		 */
		if (trys) {
			preempt_enable_notrace();
			preempt_disable_notrace();
			cpu = smp_processor_id();
			buffer = per_cpu_ptr(tinfo->tbuf, cpu)->buf;
		}

		/*
		 * If for some reason, copy_from_user() always causes a context
		 * switch, this would then cause an infinite loop.
		 * If this task is preempted by another user space task, it
		 * will cause this task to try again. But just in case something
		 * changes where the copying from user space causes another task
		 * to run, prevent this from going into an infinite loop.
		 * 100 tries should be plenty.
		 */
		if (WARN_ONCE(trys++ > 100, "Error: Too many tries to read user space"))
			return NULL;

		/* Read the current CPU context switch counter */
		cnt = nr_context_switches_cpu(cpu);

		/*
		 * Preemption is going to be enabled, but this task must
		 * remain on this CPU.
		 */
		migrate_disable();

		/*
		 * Now preemption is being enabled and another task can come in
		 * and use the same buffer and corrupt our data.
		 */
		preempt_enable_notrace();

		/* Make sure preemption is enabled here */
		lockdep_assert_preemption_enabled();

		if (copy_func) {
			ret = copy_func(buffer, ptr, size, data);
		} else {
			ret = __copy_from_user(buffer, ptr, size);
		}

		preempt_disable_notrace();
		migrate_enable();

		/* if it faulted, no need to test if the buffer was corrupted */
		if (ret)
			return NULL;

		/*
		 * Preemption is disabled again, now check the per CPU context
		 * switch counter. If it doesn't match, then another user space
		 * process may have schedule in and corrupted our buffer. In that
		 * case the copying must be retried.
		 */
	} while (nr_context_switches_cpu(cpu) != cnt);

What it does is to write into a per CPU buffer from user space. If it
schedules out while trying this, it will try again as another task could
have come in and corrupted the buffer.

For some reason, when preemption is enabled to copy from user, something
is forcing a schedule of the process. If we can figure out what is doing
that, perhaps we can fix this. A similar thing happened with the migration
thread that commit edca33a56297d ("tracing: Fix failure to read user space
from system call trace events") fixed.

-- Steve

^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: [BUG] tracing: Too many tries to read user space
  2026-07-08 13:18   ` Steven Rostedt
@ 2026-07-09  0:04     ` Masami Hiramatsu
  0 siblings, 0 replies; 6+ messages in thread
From: Masami Hiramatsu @ 2026-07-09  0:04 UTC (permalink / raw)
  To: Steven Rostedt
  Cc: Jeongho Choi, linux-trace-kernel, linux-kernel, mhiramat,
	ji2yoon.jo, minki.jang, hajun.sung

On Wed, 8 Jul 2026 09:18:04 -0400
Steven Rostedt <rostedt@goodmis.org> wrote:

> On Wed, 8 Jul 2026 21:37:53 +0900
> Jeongho Choi <jh1012.choi@samsung.com> wrote:
> 
> > The code at the WARN location mentioned in the log above is as follows.
> > 
> > 7374                 if (WARN_ONCE(trys++ > 100, "Error: Too many
> >   tries to read user space"))
> > 7375                         return NULL;
> > 
> 
> This happens when something forces a schedule.
> 
> > 
> > Our current analysis is as follows:
> > 
> > In the Gmail process, during a low memory situation, LMKD writes strings
> > to /sys/kernel/tracing/trace_marker for systrace recording. At the same
> > time, it broadcasts a sigkill due to low memory, which is causing the
> > LMKD trace marker operation to stall.
> > 
> 
> Can you see what is being scheduled in? Perhaps use the persistent ring
> buffer (if you can) and enable sched_switch tracepoint in it.

Nit: If the device just caused oom and not crash/reboot the kernel, you can
just use normal ring buffer (or perfetto).

Thanks, 

-- 
Masami Hiramatsu (Google) <mhiramat@kernel.org>

^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: [BUG] tracing: Too many tries to read user space
  2026-07-08 12:37 ` [BUG] tracing: Too many tries to read user space Jeongho Choi
  2026-07-08 13:18   ` Steven Rostedt
@ 2026-07-10  3:22   ` Masami Hiramatsu
  2026-07-10 11:46     ` Steven Rostedt
  2026-07-10 12:33     ` Steven Rostedt
  1 sibling, 2 replies; 6+ messages in thread
From: Masami Hiramatsu @ 2026-07-10  3:22 UTC (permalink / raw)
  To: Jeongho Choi
  Cc: linux-trace-kernel, linux-kernel, rostedt, ji2yoon.jo, minki.jang,
	hajun.sung

On Wed, 8 Jul 2026 21:37:53 +0900
Jeongho Choi <jh1012.choi@samsung.com> wrote:

> Hello,
> 
> We are seeing a reproducible kernel panic related to the tracing code
> when it fails to read user-space memory.
> 
> The issue was originally reported through the Android/Google Issue
> Tracker, and we were advised to report it to the upstream trace mailing
> list because the affected code is upstream.
> 
> Environment:
> 
> Architecture: arm64
> Kernel: Linux 6.18.21
> Base: Android Common Kernel (android17-6.18)
> Affected area: kernel/trace/
> 
> The relevant error/panic log is:
> [48916.569148] [9:           lmkd:  536] Error: Too many tries to read
> user space
> [48916.569156] [9:           lmkd:  536] WARNING: CPU: 9 PID: 536 at
> kernel/trace/trace.c:7374 trace_user_fault_read+0x334/0x360
> [48916.569443] [9:           lmkd:  536] CPU: 9 UID: 1069 PID: 536 Comm:
> lmkd Tainted: G           OE       6.18.21-android17-5-ga1a8e8cab9ec-4k
> #1 PREEMPT  25372cd4750dcac3c0fe86b57d47c665f97a6046
> 
> [48916.569450] [9:           lmkd:  536] pstate: 63402005 (nZCv daif
> +PAN -UAO +TCO +DIT -SSBS BTYPE=--)
> [48916.569452] [9:           lmkd:  536] pc :
> trace_user_fault_read+0x334/0x360
> [48916.569454] [9:           lmkd:  536] lr :
> trace_user_fault_read+0x330/0x360
> [48916.569456] [9:           lmkd:  536] sp : ffffffc096993cc0
> [48916.569457] [9:           lmkd:  536] x29: ffffffc096993cd0 x28:
> 0000000000000065 x27: ffffff8817e45200
> [48916.569461] [9:           lmkd:  536] x26: 0000000000000000 x25:
> 000000011616c082 x24: 0000000000000000
> [48916.569463] [9:           lmkd:  536] x23: 0000000000000009 x22:
> 000000000000002b x21: ffffff880597f740
> [48916.569466] [9:           lmkd:  536] x20: 0000007fc42dd940 x19:
> ffffff8817e45770 x18: ffffffd5947ce240
> [48916.569468] [9:           lmkd:  536] x17: 2073656972742079 x16:
> 6e616d206f6f5420 x15: 3a726f727245205d
> [48916.569471] [9:           lmkd:  536] x14: 36333520203a646b x13:
> 6563617073207265 x12: 0000000000000001
> [48916.569473] [9:           lmkd:  536] x11: 6f74207365697274 x10:
> 0000000000000001 x9 : 4bd9ad4516d75100
> [48916.569476] [9:           lmkd:  536] x8 : 4bd9ad4516d75100 x7 :
> 205d383431393635 x6 : 2e36313938345b0a
> [48916.569479] [9:           lmkd:  536] x5 : ffffffc080fa5998 x4 :
> ffffffd591707202 x3 : 0001360a00000000
> [48916.569481] [9:           lmkd:  536] x2 : ffffffc096993af4 x1 :
> 00000000000000c0 x0 : 0000000000000028
> [48916.569484] [9:           lmkd:  536] Call trace:
> [48916.569486] [9:           lmkd:  536]
> trace_user_fault_read+0x334/0x360 (P)
> [48916.569488] [9:           lmkd:  536]  tracing_mark_write+0x84/0x174
> [48916.569491] [9:           lmkd:  536]  __arm64_sys_write+0x2a0/0x5c0
> [48916.569494] [9:           lmkd:  536]  invoke_syscall+0x58/0xe4
> [48916.569498] [9:           lmkd:  536]  do_el0_svc+0x48/0xdc
> [48916.569500] [9:           lmkd:  536]  el0_svc+0x3c/0x98
> [48916.569503] [9:           lmkd:  536]
> el0t_64_sync_handler+0x20/0x130
> [48916.569505] [9:           lmkd:  536]  el0t_64_sync+0x1c4/0x1c8
> [48916.569508] [9:           lmkd:  536] Kernel panic - not syncing:
> kernel: panic_on_warn set ...
> 
> 
> The code at the WARN location mentioned in the log above is as follows.
> 
> 7374                 if (WARN_ONCE(trys++ > 100, "Error: Too many
>   tries to read user space"))
> 7375                         return NULL;
> 
> 
> Our current analysis is as follows:
> 
> In the Gmail process, during a low memory situation, LMKD writes strings
> to /sys/kernel/tracing/trace_marker for systrace recording. At the same
> time, it broadcasts a sigkill due to low memory, which is causing the
> LMKD trace marker operation to stall.

Hm, in my view, this warning indicates that the circuit breaker has
triggered correctly, so that is not a bug. Under the heavy memory
pressure and low-memory situation, the page can be reclaimed soon
after it is copied.

However, this seems a bit strange that we only checks the CPU-wide context
switching in the loop. Instead, can we introduce a per-cpu sequence counter
to per-cpu buffer, and check it? 

Could you try this ?

From f76d8e4400a5961725d17899f4290c9334987e2b Mon Sep 17 00:00:00 2001
From: Masami Hiramatsu <mhiramat@kernel.org>
Date: Fri, 10 Jul 2026 12:11:00 +0900
Subject: [PATCH] tracing: Use per-CPU sequence counter in
 trace_user_fault_read

trace_user_fault_read() copies trace data from user space to the
per-CPU trace buffer. When preemption is enabled during the copy, it
checks if any context switches occurred on the current CPU via
nr_context_switches_cpu() to detect whether the buffer may have been
corrupted by another trace writer.

However, under heavy memory pressure, copying from user space can trigger
page faults (e.g., for swapped-out BSS or anonymous pages) that block and
cause a context switch. Because nr_context_switches_cpu() detects any
context switch (even unrelated ones), it mistakenly assumes the buffer was
corrupted. This leads to repeated retries (up to the 100-try limit), which
causes a WARN_ONCE backtrace and returns -EFAULT to user space, even if
no other task ever accessed the trace buffer.

To mitigate this issue, replace the CPU-wide context switch check with
a dedicated per-CPU sequence counter in struct trace_user_buf.

Since only other tasks invoking trace_user_fault_read() on the same CPU
will increment this counter, unrelated context switches (including those
from page fault sleep) will no longer trigger retries.

Signed-off-by: Masami Hiramatsu (Google) <mhiramat@kernel.org>
---
 kernel/trace/trace.c | 29 ++++++++++++++++++-----------
 1 file changed, 18 insertions(+), 11 deletions(-)

diff --git a/kernel/trace/trace.c b/kernel/trace/trace.c
index 1bc27c0ad029..46cec63f5798 100644
--- a/kernel/trace/trace.c
+++ b/kernel/trace/trace.c
@@ -5989,6 +5989,7 @@ static ssize_t write_marker_to_buffer(struct trace_array *tr, const char *buf,
 
 struct trace_user_buf {
 	char		*buf;
+	unsigned int	sequence;
 };
 
 static DEFINE_MUTEX(trace_user_buffer_mutex);
@@ -6031,7 +6032,10 @@ static int user_fault_buffer_enable(struct trace_user_buf_info *tinfo, size_t si
 
 	/* Clear each buffer in case of error */
 	for_each_possible_cpu(cpu) {
-		per_cpu_ptr(tinfo->tbuf, cpu)->buf = NULL;
+		struct trace_user_buf *tbuf = per_cpu_ptr(tinfo->tbuf, cpu);
+
+		tbuf->buf = NULL;
+		tbuf->sequence = 0;
 	}
 
 	for_each_possible_cpu(cpu) {
@@ -6196,8 +6200,9 @@ char *trace_user_fault_read(struct trace_user_buf_info *tinfo,
 			     trace_user_buf_copy copy_func, void *data)
 {
 	int cpu = smp_processor_id();
-	char *buffer = per_cpu_ptr(tinfo->tbuf, cpu)->buf;
-	unsigned int cnt;
+	struct trace_user_buf *tbuf = per_cpu_ptr(tinfo->tbuf, cpu);
+	char *buffer = tbuf->buf;
+	unsigned int seq;
 	int trys = 0;
 	int ret;
 
@@ -6211,10 +6216,10 @@ char *trace_user_fault_read(struct trace_user_buf_info *tinfo,
 		return NULL;
 
 	/*
-	 * This acts similar to a seqcount. The per CPU context switches are
+	 * This acts similar to a seqcount. The per CPU sequence counters are
 	 * recorded, migration is disabled and preemption is enabled. The
 	 * read of the user space memory is copied into the per CPU buffer.
-	 * Preemption is disabled again, and if the per CPU context switches count
+	 * Preemption is disabled again, and if the per CPU sequence count
 	 * is still the same, it means the buffer has not been corrupted.
 	 * If the count is different, it is assumed the buffer is corrupted
 	 * and reading must be tried again.
@@ -6235,7 +6240,8 @@ char *trace_user_fault_read(struct trace_user_buf_info *tinfo,
 			preempt_enable_notrace();
 			preempt_disable_notrace();
 			cpu = smp_processor_id();
-			buffer = per_cpu_ptr(tinfo->tbuf, cpu)->buf;
+			tbuf = per_cpu_ptr(tinfo->tbuf, cpu);
+			buffer = tbuf->buf;
 		}
 
 		/*
@@ -6250,8 +6256,9 @@ char *trace_user_fault_read(struct trace_user_buf_info *tinfo,
 		if (WARN_ONCE(trys++ > 100, "Error: Too many tries to read user space"))
 			return NULL;
 
-		/* Read the current CPU context switch counter */
-		cnt = nr_context_switches_cpu(cpu);
+		/* Increment the per-CPU buffer sequence counter */
+		tbuf->sequence++;
+		seq = tbuf->sequence;
 
 		/*
 		 * Preemption is going to be enabled, but this task must
@@ -6282,12 +6289,12 @@ char *trace_user_fault_read(struct trace_user_buf_info *tinfo,
 			return NULL;
 
 		/*
-		 * Preemption is disabled again, now check the per CPU context
-		 * switch counter. If it doesn't match, then another user space
+		 * Preemption is disabled again, now check the per CPU sequence
+		 * counter. If it doesn't match, then another user space
 		 * process may have schedule in and corrupted our buffer. In that
 		 * case the copying must be retried.
 		 */
-	} while (nr_context_switches_cpu(cpu) != cnt);
+	} while (tbuf->sequence != seq);
 
 	return buffer;
 }
-- 
2.43.0



-- 
Masami Hiramatsu (Google) <mhiramat@kernel.org>

^ permalink raw reply related	[flat|nested] 6+ messages in thread

* Re: [BUG] tracing: Too many tries to read user space
  2026-07-10  3:22   ` Masami Hiramatsu
@ 2026-07-10 11:46     ` Steven Rostedt
  2026-07-10 12:33     ` Steven Rostedt
  1 sibling, 0 replies; 6+ messages in thread
From: Steven Rostedt @ 2026-07-10 11:46 UTC (permalink / raw)
  To: Masami Hiramatsu (Google)
  Cc: Jeongho Choi, linux-trace-kernel, linux-kernel, ji2yoon.jo,
	minki.jang, hajun.sung

On Fri, 10 Jul 2026 12:22:31 +0900
Masami Hiramatsu (Google) <mhiramat@kernel.org> wrote:

> However, this seems a bit strange that we only checks the CPU-wide context
> switching in the loop. Instead, can we introduce a per-cpu sequence counter
> to per-cpu buffer, and check it? 

I originally tried this but found a situation that it fails:

  tbuf->sequence = 0;

	Task 1				Task 2
	------				------

   tbuf->sequence++;
   seq = tbuf->sequence; (seq = 1)

   preempt_enable();

   [schedule] ---------------------->

				  tbuf->sequence++;
				  seq = tbuf->sequence; (seq = 2);

				  preempt_enable();

				  copy_from_user(buffer);

	     <--------------------[schedule]

   copy_from_user(buffer);

   *** BUFFER NOW CORRUPTED ***

   [schedule] ---------------------->

				  preempt_disable();

				} while (tubf->sequence != seq); // tbuf->sequence == seq !!!!


This is why we use a CPU wide counter.

-- Steve
  


^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: [BUG] tracing: Too many tries to read user space
  2026-07-10  3:22   ` Masami Hiramatsu
  2026-07-10 11:46     ` Steven Rostedt
@ 2026-07-10 12:33     ` Steven Rostedt
  1 sibling, 0 replies; 6+ messages in thread
From: Steven Rostedt @ 2026-07-10 12:33 UTC (permalink / raw)
  To: Masami Hiramatsu (Google)
  Cc: Jeongho Choi, linux-trace-kernel, linux-kernel, ji2yoon.jo,
	minki.jang, hajun.sung

On Fri, 10 Jul 2026 12:22:31 +0900
Masami Hiramatsu (Google) <mhiramat@kernel.org> wrote:

> Hm, in my view, this warning indicates that the circuit breaker has
> triggered correctly, so that is not a bug. Under the heavy memory
> pressure and low-memory situation, the page can be reclaimed soon
> after it is copied.

So you are saying that every time the copy_from_user() is executed, the
page is reclaimed? And this causes a schedule?

Now, I did have a version that used sched_switch and only incremented the
counter when a non-kernel thread was scheduled in. Then the test would
check if the counter increased by 2 or more. As an increase by 1 meant that
only kernel threads scheduled in which would not corrupt the buffer. The 1
increment was the current task scheduling back.

This is based on that work (I'm glad I save old versions in my git tree :-)

Funny, the comments were from the original change I did back in August of
2025, which mentions kernel threads scheduling in to handle the fault.

I also kept this around in case it was needed. Looks like it may be needed.

-- Steve

diff --git a/kernel/trace/trace.c b/kernel/trace/trace.c
index 18710c190c92..19354fe2fca1 100644
--- a/kernel/trace/trace.c
+++ b/kernel/trace/trace.c
@@ -53,6 +53,8 @@
 #include <linux/io.h> /* vmap_page_range() */
 #include <linux/fs_context.h>
 
+#include <trace/events/sched.h>
+
 #include <asm/setup.h> /* COMMAND_LINE_SIZE */
 
 #include "trace.h"
@@ -5984,6 +5986,32 @@ struct trace_user_buf {
 static DEFINE_MUTEX(trace_user_buffer_mutex);
 static struct trace_user_buf_info *trace_user_buffer;
 
+static DEFINE_PER_CPU(unsigned long, sched_switch_cnt);
+
+/*
+ * The per CPU buffer trace_user_buffer is written to optimstically.
+ * The counter sched_switch_cnt is taken, preemption is enabled,
+ * the copying of the user space memory is placed into the trace_user_buffer,
+ * Preeption is re-enabled and the count is read again. If the count is greater
+ * than one from its previous reading, it means that another user space
+ * task scheduled in and the buffer is unreliable for use.
+ */
+static void
+probe_sched_switch(void *ignore, bool preempt,
+		   struct task_struct *prev, struct task_struct *next,
+		   unsigned int prev_state)
+{
+	/*
+	 * The buffer can only be corrupted by another user space task.
+	 * Ignore kernel tasks that may be scheduled in order to process
+	 * the faulting memory.
+	 */
+	if (!is_user_task(next))
+		return;
+
+	this_cpu_inc(sched_switch_cnt);
+}
+
 /**
  * trace_user_fault_destroy - free up allocated memory of a trace user buffer
  * @tinfo: The descriptor to free up
@@ -6003,6 +6031,8 @@ void trace_user_fault_destroy(struct trace_user_buf_info *tinfo)
 		kfree(buf);
 	}
 	free_percpu(tinfo->tbuf);
+
+	unregister_trace_sched_switch(probe_sched_switch, NULL);
 }
 
 static int user_fault_buffer_enable(struct trace_user_buf_info *tinfo, size_t size)
@@ -6053,11 +6083,17 @@ static int user_buffer_init(struct trace_user_buf_info **tinfo, size_t size)
 
 	lockdep_assert_held(&trace_user_buffer_mutex);
 
+	ret = register_trace_sched_switch(probe_sched_switch, NULL);
+	if (ret < 0)
+		return ret;
+
 	if (!*tinfo) {
 		alloc = true;
 		*tinfo = kzalloc_obj(**tinfo);
-		if (!*tinfo)
+		if (!*tinfo) {
+			unregister_trace_sched_switch(probe_sched_switch, NULL);
 			return -ENOMEM;
+		}
 	}
 
 	ret = user_fault_buffer_enable(*tinfo, size);
@@ -6241,7 +6277,7 @@ char *trace_user_fault_read(struct trace_user_buf_info *tinfo,
 			return NULL;
 
 		/* Read the current CPU context switch counter */
-		cnt = nr_context_switches_cpu(cpu);
+		cnt = this_cpu_read(sched_switch_cnt);
 
 		/*
 		 * Preemption is going to be enabled, but this task must
@@ -6272,12 +6308,19 @@ char *trace_user_fault_read(struct trace_user_buf_info *tinfo,
 			return NULL;
 
 		/*
-		 * Preemption is disabled again, now check the per CPU context
-		 * switch counter. If it doesn't match, then another user space
-		 * process may have schedule in and corrupted our buffer. In that
-		 * case the copying must be retried.
+		 * Preemption is disabled again, now check the sched_switch_cnt.
+		 * If it increased by two or more, then another user space process
+		 * may have schedule in and corrupted our buffer. In that case
+		 * the copying must be retried.
+		 *
+		 * Note, if this task was scheduled out and only kernel threads
+		 * were scheduled in (maybe to process the fault), then the
+		 * counter would increment again when this task scheduled in.
+		 * If this task scheduled out and another user task scheduled
+		 * in, this task would still need to be scheduled back in and
+		 * the counter would increment by at least two.
 		 */
-	} while (nr_context_switches_cpu(cpu) != cnt);
+	} while (this_cpu_read(sched_switch_cnt) > cnt + 1);
 
 	return buffer;
 }

^ permalink raw reply related	[flat|nested] 6+ messages in thread

end of thread, other threads:[~2026-07-10 12:33 UTC | newest]

Thread overview: 6+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
     [not found] <CGME20260708123754epcas2p1f15cc305ddb09f97164491d750769ef7@epcas2p1.samsung.com>
2026-07-08 12:37 ` [BUG] tracing: Too many tries to read user space Jeongho Choi
2026-07-08 13:18   ` Steven Rostedt
2026-07-09  0:04     ` Masami Hiramatsu
2026-07-10  3:22   ` Masami Hiramatsu
2026-07-10 11:46     ` Steven Rostedt
2026-07-10 12:33     ` Steven Rostedt

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox