All of lore.kernel.org
 help / color / mirror / Atom feed
From: Usama Arif <usama.arif@linux.dev>
To: Usama Arif <usama.arif@linux.dev>
Cc: linux-kernel@vger.kernel.org, linux-trace-kernel@vger.kernel.org,
	mathieu.desnoyers@efficios.com, mhiramat@kernel.org,
	rostedt@goodmis.org, leitao@debian.org
Subject: Re: [RFC] tracing: Try user copies with page faults disabled first
Date: Fri, 17 Jul 2026 10:37:57 -0700	[thread overview]
Message-ID: <20260717173758.3644427-1-usama.arif@linux.dev> (raw)
In-Reply-To: <20260715155454.4127988-1-usama.arif@linux.dev>

On Wed, 15 Jul 2026 08:54:54 -0700 Usama Arif <usama.arif@linux.dev> wrote:

> trace_user_fault_read() is called with preemption disabled to copy user
> memory into a per-cpu scratch buffer. The existing implementation enables
> preemption around the copy because faulting user memory can sleep. That
> opens a window where another task can run on the same CPU and clobber the
> per-cpu buffer, so the copy is wrapped in a retry loop: sample
> nr_context_switches_cpu(), do the preempt-enabled copy, and retry if the
> counter changed. If this fails to complete 100 times, the function gives up
> with a warning.
> 
> nr_context_switches_cpu() reads rq->nr_switches. That counter increments
> for every context switch on the CPU, not only for switches to tasks that
> use this tracing scratch buffer. On a heavily loaded system, unrelated
> scheduler activity can move the counter during every preempt-enabled copy
> attempt, exhaust the retry guard, and trigger the warning.
> 
> This is showing up across the Meta fleet around 100 times a day since the
> kernel began upgrading to 7.1, mostly on arm servers:
> 
>   Error: Too many tries to read user space
>   WARNING: kernel/trace/trace.c:6244 at trace_user_fault_read+0x284/0x2c8, CPU#28: Collection-18/677527
>   CPU: 28 UID: 0 PID: 677527 Comm: Collection-18 Kdump: loaded Not tainted 7.1.0-.... #1 PREEMPTLAZY
>   Hardware name: Quanta Java Island MP 29F0EMA08CH/Java Island, BIOS F0EJ3A16 03/12/2026
>   Call trace:
>    trace_user_fault_read+0x284/0x2c8 (P)
>    syscall_get_data+0x144/0x2c0
>    perf_syscall_enter+0xc0/0x2d8
>    syscall_trace_enter+0x1a0/0x270
>    do_el0_svc+0x54/0xb8
>    el0_svc+0x44/0x268
>    el0t_64_sync_handler+0x7c/0x120
>    el0t_64_sync+0x17c/0x180
>   ---[ end trace 0000000000000000 ]---
> 

I think the actual problem might be:
https://lore.kernel.org/all/20260717173252.3431565-1-usama.arif@linux.dev/
 

      parent reply	other threads:[~2026-07-17 17:38 UTC|newest]

Thread overview: 4+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-07-15 15:54 [RFC] tracing: Try user copies with page faults disabled first Usama Arif
2026-07-15 19:11 ` Steven Rostedt
2026-07-16 12:05   ` Usama Arif
2026-07-17 17:37 ` Usama Arif [this message]

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20260717173758.3644427-1-usama.arif@linux.dev \
    --to=usama.arif@linux.dev \
    --cc=leitao@debian.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=linux-trace-kernel@vger.kernel.org \
    --cc=mathieu.desnoyers@efficios.com \
    --cc=mhiramat@kernel.org \
    --cc=rostedt@goodmis.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.