Re: [RFC] tracing: Try user copies with page faults disabled first

[Usama Arif <usama.arif@linux.dev>]

On 15/07/2026 20:11, Steven Rostedt wrote:
> On Wed, 15 Jul 2026 08:54:54 -0700
> Usama Arif <usama.arif@linux.dev> wrote:
> 
>> trace_user_fault_read() is called with preemption disabled to copy user
>> memory into a per-cpu scratch buffer. The existing implementation enables
>> preemption around the copy because faulting user memory can sleep. That
>> opens a window where another task can run on the same CPU and clobber the
>> per-cpu buffer, so the copy is wrapped in a retry loop: sample
>> nr_context_switches_cpu(), do the preempt-enabled copy, and retry if the
>> counter changed. If this fails to complete 100 times, the function gives up
>> with a warning.
>>
>> nr_context_switches_cpu() reads rq->nr_switches. That counter increments
>> for every context switch on the CPU, not only for switches to tasks that
>> use this tracing scratch buffer. On a heavily loaded system, unrelated
>> scheduler activity can move the counter during every preempt-enabled copy
>> attempt, exhaust the retry guard, and trigger the warning.
>>
>> This is showing up across the Meta fleet around 100 times a day since the
>> kernel began upgrading to 7.1, mostly on arm servers:
>>
>>   Error: Too many tries to read user space
>>   WARNING: kernel/trace/trace.c:6244 at trace_user_fault_read+0x284/0x2c8, CPU#28: Collection-18/677527
>>   CPU: 28 UID: 0 PID: 677527 Comm: Collection-18 Kdump: loaded Not tainted 7.1.0-.... #1 PREEMPTLAZY
>>   Hardware name: Quanta Java Island MP 29F0EMA08CH/Java Island, BIOS F0EJ3A16 03/12/2026
>>   Call trace:
>>    trace_user_fault_read+0x284/0x2c8 (P)
>>    syscall_get_data+0x144/0x2c0
>>    perf_syscall_enter+0xc0/0x2d8
>>    syscall_trace_enter+0x1a0/0x270
>>    do_el0_svc+0x54/0xb8
>>    el0_svc+0x44/0x268
>>    el0t_64_sync_handler+0x7c/0x120
>>    el0t_64_sync+0x17c/0x180
>>   ---[ end trace 0000000000000000 ]---
>>
>> The retry loop is only needed when preemption must be enabled for the user
>> copy. If the user pages are already resident, the copy can complete without
>> fault handling that sleeps, and preemption can stay disabled throughout.
>>
>> Add a fast path that first tries the copy with page faults disabled. For
>> the plain copy_from_user case, use __copy_from_user_inatomic(). If the
>> probe faults, the architecture exception-table fixup returns a non-zero
>> not-copied count and trace_user_fault_read() falls back to the existing
>> preempt-enabled slow path.
>>
>> Custom copy callbacks need the same behavior. Update the syscall argument
>> copy callbacks to report a non-zero return only when the pagefault-disabled
>> probe faults. With page faults enabled, keep their previous behavior:
>> record the syscall event and omit only the individual user argument that
>> still cannot be copied.
>>
>> The slow path remains in place for nonresident pages and permanent copy
>> failures. nr_context_switches_cpu() still overcounts, but the retry loop is
>> now avoided for the common resident-page case that does not need fault
>> handling.
> 
> This was reported also under memory load.
> 
> I have a patch that will only do a retry if another *user* task schedules
> in, and will not be bothered by kernel tasks (which may be scheduled in due
> to the copy from user to begin with).
> 
> Can you see if that works too?
> 
>   https://lore.kernel.org/all/20260710083357.49e05ff6@gandalf.local.home/
> 
Thanks, I will try and put this in the next kernel, although it will take sometime
for the new kernel to roll in the fleet and to see if the warnings disappear.