Linux Trace Kernel
 help / color / mirror / Atom feed
From: Usama Arif <usama.arif@linux.dev>
To: Usama Arif <usama.arif@linux.dev>
Cc: linux-kernel@vger.kernel.org, linux-trace-kernel@vger.kernel.org,
	mathieu.desnoyers@efficios.com, mhiramat@kernel.org,
	rostedt@goodmis.org, leitao@debian.org
Subject: Re: [RFC] tracing: Try user copies with page faults disabled first
Date: Fri, 17 Jul 2026 10:37:57 -0700	[thread overview]
Message-ID: <20260717173758.3644427-1-usama.arif@linux.dev> (raw)
In-Reply-To: <20260715155454.4127988-1-usama.arif@linux.dev>

On Wed, 15 Jul 2026 08:54:54 -0700 Usama Arif <usama.arif@linux.dev> wrote:

> trace_user_fault_read() is called with preemption disabled to copy user
> memory into a per-cpu scratch buffer. The existing implementation enables
> preemption around the copy because faulting user memory can sleep. That
> opens a window where another task can run on the same CPU and clobber the
> per-cpu buffer, so the copy is wrapped in a retry loop: sample
> nr_context_switches_cpu(), do the preempt-enabled copy, and retry if the
> counter changed. If this fails to complete 100 times, the function gives up
> with a warning.
> 
> nr_context_switches_cpu() reads rq->nr_switches. That counter increments
> for every context switch on the CPU, not only for switches to tasks that
> use this tracing scratch buffer. On a heavily loaded system, unrelated
> scheduler activity can move the counter during every preempt-enabled copy
> attempt, exhaust the retry guard, and trigger the warning.
> 
> This is showing up across the Meta fleet around 100 times a day since the
> kernel began upgrading to 7.1, mostly on arm servers:
> 
>   Error: Too many tries to read user space
>   WARNING: kernel/trace/trace.c:6244 at trace_user_fault_read+0x284/0x2c8, CPU#28: Collection-18/677527
>   CPU: 28 UID: 0 PID: 677527 Comm: Collection-18 Kdump: loaded Not tainted 7.1.0-.... #1 PREEMPTLAZY
>   Hardware name: Quanta Java Island MP 29F0EMA08CH/Java Island, BIOS F0EJ3A16 03/12/2026
>   Call trace:
>    trace_user_fault_read+0x284/0x2c8 (P)
>    syscall_get_data+0x144/0x2c0
>    perf_syscall_enter+0xc0/0x2d8
>    syscall_trace_enter+0x1a0/0x270
>    do_el0_svc+0x54/0xb8
>    el0_svc+0x44/0x268
>    el0t_64_sync_handler+0x7c/0x120
>    el0t_64_sync+0x17c/0x180
>   ---[ end trace 0000000000000000 ]---
> 

I think the actual problem might be:
https://lore.kernel.org/all/20260717173252.3431565-1-usama.arif@linux.dev/
 

      parent reply	other threads:[~2026-07-17 17:38 UTC|newest]

Thread overview: 4+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-07-15 15:54 [RFC] tracing: Try user copies with page faults disabled first Usama Arif
2026-07-15 19:11 ` Steven Rostedt
2026-07-16 12:05   ` Usama Arif
2026-07-17 17:37 ` Usama Arif [this message]

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20260717173758.3644427-1-usama.arif@linux.dev \
    --to=usama.arif@linux.dev \
    --cc=leitao@debian.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=linux-trace-kernel@vger.kernel.org \
    --cc=mathieu.desnoyers@efficios.com \
    --cc=mhiramat@kernel.org \
    --cc=rostedt@goodmis.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox