Re: [BUG] tracing: Too many tries to read user space

[Masami Hiramatsu (Google) <mhiramat@kernel.org>]

On Fri, 10 Jul 2026 08:33:57 -0400
Steven Rostedt <rostedt@goodmis.org> wrote:

> On Fri, 10 Jul 2026 12:22:31 +0900
> Masami Hiramatsu (Google) <mhiramat@kernel.org> wrote:
> 
> > Hm, in my view, this warning indicates that the circuit breaker has
> > triggered correctly, so that is not a bug. Under the heavy memory
> > pressure and low-memory situation, the page can be reclaimed soon
> > after it is copied.
> 
> So you are saying that every time the copy_from_user() is executed, the
> page is reclaimed? And this causes a schedule?
> 
> Now, I did have a version that used sched_switch and only incremented the
> counter when a non-kernel thread was scheduled in. Then the test would
> check if the counter increased by 2 or more. As an increase by 1 meant that
> only kernel threads scheduled in which would not corrupt the buffer. The 1
> increment was the current task scheduling back.
> 
> This is based on that work (I'm glad I save old versions in my git tree :-)
> 
> Funny, the comments were from the original change I did back in August of
> 2025, which mentions kernel threads scheduling in to handle the fault.
> 
> I also kept this around in case it was needed. Looks like it may be needed.

Hmm, it's not clear whether this is the result of a scheduling storm
caused by kernel threads. But maybe worth to try.

Jeongho, can you try this patch? Or/and you maybe also try using
mlock the LMKD process memory which is passed to trace_marker.

Thank you, 

> 
> -- Steve
> 
> diff --git a/kernel/trace/trace.c b/kernel/trace/trace.c
> index 18710c190c92..19354fe2fca1 100644
> --- a/kernel/trace/trace.c
> +++ b/kernel/trace/trace.c
> @@ -53,6 +53,8 @@
>  #include <linux/io.h> /* vmap_page_range() */
>  #include <linux/fs_context.h>
>  
> +#include <trace/events/sched.h>
> +
>  #include <asm/setup.h> /* COMMAND_LINE_SIZE */
>  
>  #include "trace.h"
> @@ -5984,6 +5986,32 @@ struct trace_user_buf {
>  static DEFINE_MUTEX(trace_user_buffer_mutex);
>  static struct trace_user_buf_info *trace_user_buffer;
>  
> +static DEFINE_PER_CPU(unsigned long, sched_switch_cnt);
> +
> +/*
> + * The per CPU buffer trace_user_buffer is written to optimstically.
> + * The counter sched_switch_cnt is taken, preemption is enabled,
> + * the copying of the user space memory is placed into the trace_user_buffer,
> + * Preeption is re-enabled and the count is read again. If the count is greater
> + * than one from its previous reading, it means that another user space
> + * task scheduled in and the buffer is unreliable for use.
> + */
> +static void
> +probe_sched_switch(void *ignore, bool preempt,
> +		   struct task_struct *prev, struct task_struct *next,
> +		   unsigned int prev_state)
> +{
> +	/*
> +	 * The buffer can only be corrupted by another user space task.
> +	 * Ignore kernel tasks that may be scheduled in order to process
> +	 * the faulting memory.
> +	 */
> +	if (!is_user_task(next))
> +		return;
> +
> +	this_cpu_inc(sched_switch_cnt);
> +}
> +
>  /**
>   * trace_user_fault_destroy - free up allocated memory of a trace user buffer
>   * @tinfo: The descriptor to free up
> @@ -6003,6 +6031,8 @@ void trace_user_fault_destroy(struct trace_user_buf_info *tinfo)
>  		kfree(buf);
>  	}
>  	free_percpu(tinfo->tbuf);
> +
> +	unregister_trace_sched_switch(probe_sched_switch, NULL);
>  }
>  
>  static int user_fault_buffer_enable(struct trace_user_buf_info *tinfo, size_t size)
> @@ -6053,11 +6083,17 @@ static int user_buffer_init(struct trace_user_buf_info **tinfo, size_t size)
>  
>  	lockdep_assert_held(&trace_user_buffer_mutex);
>  
> +	ret = register_trace_sched_switch(probe_sched_switch, NULL);
> +	if (ret < 0)
> +		return ret;
> +
>  	if (!*tinfo) {
>  		alloc = true;
>  		*tinfo = kzalloc_obj(**tinfo);
> -		if (!*tinfo)
> +		if (!*tinfo) {
> +			unregister_trace_sched_switch(probe_sched_switch, NULL);
>  			return -ENOMEM;
> +		}
>  	}
>  
>  	ret = user_fault_buffer_enable(*tinfo, size);
> @@ -6241,7 +6277,7 @@ char *trace_user_fault_read(struct trace_user_buf_info *tinfo,
>  			return NULL;
>  
>  		/* Read the current CPU context switch counter */
> -		cnt = nr_context_switches_cpu(cpu);
> +		cnt = this_cpu_read(sched_switch_cnt);
>  
>  		/*
>  		 * Preemption is going to be enabled, but this task must
> @@ -6272,12 +6308,19 @@ char *trace_user_fault_read(struct trace_user_buf_info *tinfo,
>  			return NULL;
>  
>  		/*
> -		 * Preemption is disabled again, now check the per CPU context
> -		 * switch counter. If it doesn't match, then another user space
> -		 * process may have schedule in and corrupted our buffer. In that
> -		 * case the copying must be retried.
> +		 * Preemption is disabled again, now check the sched_switch_cnt.
> +		 * If it increased by two or more, then another user space process
> +		 * may have schedule in and corrupted our buffer. In that case
> +		 * the copying must be retried.
> +		 *
> +		 * Note, if this task was scheduled out and only kernel threads
> +		 * were scheduled in (maybe to process the fault), then the
> +		 * counter would increment again when this task scheduled in.
> +		 * If this task scheduled out and another user task scheduled
> +		 * in, this task would still need to be scheduled back in and
> +		 * the counter would increment by at least two.
>  		 */
> -	} while (nr_context_switches_cpu(cpu) != cnt);
> +	} while (this_cpu_read(sched_switch_cnt) > cnt + 1);
>  
>  	return buffer;
>  }


-- 
Masami Hiramatsu (Google) <mhiramat@kernel.org>