Re: [PATCH v5 13/17] perf: Support deferred user callchains

[Namhyung Kim <namhyung@kernel.org>]

On Mon, Apr 28, 2025 at 06:02:53PM -0400, Steven Rostedt wrote:
> On Mon, 28 Apr 2025 13:42:15 -0700
> Namhyung Kim <namhyung@kernel.org> wrote:
> 
> 
> > > 
> > > Could we just not use deferred when running with "-a" for now? Or could we
> > > possibly just make the deferred stacktrace its own event? Have it be
> > > possible that perf just registers a signal instance with the deferred
> > > unwinding logic, and then perf can handle where to write the information. I
> > > don't know perf well enough to implement that.  
> > 
> > Even if it excludes per-CPU events, per-task events also can attach to a
> > CPU and that's the default behavior of the perf record IIRC.  In that
> > case, it needs to be careful when it accesses the event since the task
> > can migrate to another CPU.  So I'm not sure if it's a good idea to
> > track event that requested the deferred callchains.
> 
> Wait? Even for per task, it uses per cpu?

Yep, it has per-task + per-CPU events.

> 
> > 
> > Also it doesn't need to emit duplicate deferred callchains if a task
> > has multiple events and they are requesting callchains.  Unfortunately,
> > the kernel cannot know which events are related or profiled together.
> > 
> > Hmm.. maybe we can add a cookie to the event itself (by ioctl or
> > something) in order to group events in a profiling session and then use
> > that for deferred callchains?  Task should maintain a list of active
> > cookies (or sessions) somehow but then perf can check if the current CPU
> > has events with matching cookies and emit a deferred callchain.
> 
> Could we add a callchain event? It gets woken at any request but not
> triggered until the task returns. A way that there would be only a single
> event for every perf instance, but it can trace any task.

Thing is that the kernel doesn't know the relationship between events.
For example, if I run this command on a machine with 100 CPUs:

  $ perf record -e cycles,instructions -- $MYPROG

it would open 200 events and they don't know each other.  Later other
process can start a new perf profiling for the same task.  IIUC there's
no way to identify which one is related in the kernel.

So I think we need a way to share some informaiton for those 200 events
and then emits deferred callchain records with the shared info.

> 
> It could use the cookie method that ftrace uses, where the request gets a
> cookie, and can be recorded to the perf event in the interrupt. Then the
> callchain would record the cookie along with the stack trace, and then perf
> tool could just match up the kernel stacks with their cookies to the user
> stack with its cookie.

Yep, but the kernel should know which events (or ring buffer) it should
emit the deferred callchains.  I don't think it needs to include the
cookie in the perf data, but it can be used to find which event or ring
buffer for the session is related to this request.

Thanks,
Namhyung