Re: [RFC] tracing: Try user copies with page faults disabled first

[Steven Rostedt <rostedt@goodmis.org>]

On Wed, 15 Jul 2026 08:54:54 -0700
Usama Arif <usama.arif@linux.dev> wrote:

> trace_user_fault_read() is called with preemption disabled to copy user
> memory into a per-cpu scratch buffer. The existing implementation enables
> preemption around the copy because faulting user memory can sleep. That
> opens a window where another task can run on the same CPU and clobber the
> per-cpu buffer, so the copy is wrapped in a retry loop: sample
> nr_context_switches_cpu(), do the preempt-enabled copy, and retry if the
> counter changed. If this fails to complete 100 times, the function gives up
> with a warning.
> 
> nr_context_switches_cpu() reads rq->nr_switches. That counter increments
> for every context switch on the CPU, not only for switches to tasks that
> use this tracing scratch buffer. On a heavily loaded system, unrelated
> scheduler activity can move the counter during every preempt-enabled copy
> attempt, exhaust the retry guard, and trigger the warning.
> 
> This is showing up across the Meta fleet around 100 times a day since the
> kernel began upgrading to 7.1, mostly on arm servers:
> 
>   Error: Too many tries to read user space
>   WARNING: kernel/trace/trace.c:6244 at trace_user_fault_read+0x284/0x2c8, CPU#28: Collection-18/677527
>   CPU: 28 UID: 0 PID: 677527 Comm: Collection-18 Kdump: loaded Not tainted 7.1.0-.... #1 PREEMPTLAZY
>   Hardware name: Quanta Java Island MP 29F0EMA08CH/Java Island, BIOS F0EJ3A16 03/12/2026
>   Call trace:
>    trace_user_fault_read+0x284/0x2c8 (P)
>    syscall_get_data+0x144/0x2c0
>    perf_syscall_enter+0xc0/0x2d8
>    syscall_trace_enter+0x1a0/0x270
>    do_el0_svc+0x54/0xb8
>    el0_svc+0x44/0x268
>    el0t_64_sync_handler+0x7c/0x120
>    el0t_64_sync+0x17c/0x180
>   ---[ end trace 0000000000000000 ]---
> 
> The retry loop is only needed when preemption must be enabled for the user
> copy. If the user pages are already resident, the copy can complete without
> fault handling that sleeps, and preemption can stay disabled throughout.
> 
> Add a fast path that first tries the copy with page faults disabled. For
> the plain copy_from_user case, use __copy_from_user_inatomic(). If the
> probe faults, the architecture exception-table fixup returns a non-zero
> not-copied count and trace_user_fault_read() falls back to the existing
> preempt-enabled slow path.
> 
> Custom copy callbacks need the same behavior. Update the syscall argument
> copy callbacks to report a non-zero return only when the pagefault-disabled
> probe faults. With page faults enabled, keep their previous behavior:
> record the syscall event and omit only the individual user argument that
> still cannot be copied.
> 
> The slow path remains in place for nonresident pages and permanent copy
> failures. nr_context_switches_cpu() still overcounts, but the retry loop is
> now avoided for the common resident-page case that does not need fault
> handling.

This was reported also under memory load.

I have a patch that will only do a retry if another *user* task schedules
in, and will not be bothered by kernel tasks (which may be scheduled in due
to the copy from user to begin with).

Can you see if that works too?

  https://lore.kernel.org/all/20260710083357.49e05ff6@gandalf.local.home/

I'm also thinking it may be backwards to try with page_fault disabled
first. Because if the text is not present in memory (which is the case for
a lot of syscalls), it will fail there and then go into the "slow path" to
try with enabling preemption. If you bug hits then, it could cause the
warning to show up.

If you first try with preemption disabled, it will likely pull the page
into memory, so if it does schedule, doing it with pagefault_disabled() a
second time is more likely to succeed (the page has been pulled in via the
first attempt). That is, the first attempt pulls in the page but fails due
to being scheduled out, the second attempt with pagefault_disabled() may be
more likely to succeed. Thus, on option is to simply alternate between
re-enabling preemption, and calling with page_fault() disabled.

-- Steve


> 
> Reported-by: Breno Leitao <leitao@debian.org>
> Signed-off-by: Usama Arif <usama.arif@linux.dev>
> ---
> This warning is very likely occuring when the fleetwide profiler runs
> and something else seems to load the server. (As we have perf_syscall_enter()
> in the stack and I see the profiler process active when the warning prints).
> It can occur several days after boot, so its a bit difficult to verify
> if the warning will go away with this patch deployed.
> If the patch looks good, we can deploy it in the fleet and report back.
> ---