Re: Tracing sched_switch events for client application when process is switched back in

[Arnaldo Carvalho de Melo <acme@kernel.org>]

Em Mon, Nov 14, 2016 at 07:17:01PM +0100, Milian Wolff escreveu:
> On Monday, November 14, 2016 12:36:11 PM CET Arnaldo Carvalho de Melo wrote:
> > Em Mon, Nov 14, 2016 at 03:10:26PM +0100, Milian Wolff escreveu:
> > > features:

> > > - sample CPU events for a given application with backtraces

> > > 	perf record --call-graph dwarf ./foo

> > > - record switch-out events with backtraces (`-e
> > > sched:sched_switch/fp=dwarf/`) - somehow record the switch-in events,
> > > which are associated with a different process and thus currently
> > > discarded

> > You want the backtraces with that, and since this is a meta-event... Humm,
> > for the switch in would the above be sufficient? What value would be in
> > knowing the backtrace in that case?

> Knowing the backtrace would ensure I get the real location that triggered a 
> switch, i.e. often times that comes from a syscall like futex or the like. Or 

Sure, sure, a backtrace for a sched switch _out_ of a thread is useful,
and that you get from the sched:sched_switch tracepoint, what I'm
alluding to is the lack of a backtrace with the PERF_RECORD_SWITCH _IN_,
meta-event i.e. when the kernel switchs back to the thread being
monitored.

> would I always get a CPU event (e.g. cycles) with a backtrace before the 

Is there a need for that? With sched:sched_switch + backtrace you get a
backtrace right at the moment of the sched_switch _out_, then, with
PERF_RECORD_SWITCH (--switch-events) you'll get the missing piece, the
sched switch _in_, that gets a CPU back to your thread. You'll not know
from where it came (the thread using the CPU right before this switch),
unless you're root, then PERF_RECORD_SWITCH _will_ have that info.

> switch happens, pointing to the same location?
> 
> > So something like would do? Try not looking at the "PERF_RECORD_SWITCH OUT"
> > lines, they are generated by the kernel _after_ the corresponding
> > sched:sched_switch is registered:
> > 
> > [root@jouet ~]# perf record --call-graph=dwarf --switch-events -e
> 
> <snip>
> 
> Yes, that seems to work just fine:
> 
> perf record -e cycles -e sched:sched_switch \
> 	--call-graph dwarf --switch-events \
> 	./a.out
> 
> That gives me all information. With some post-processing one could then turn 
> this into a really useful profiling tool, as one will only need a single perf 
> record step to allow both, on-CPU as well as off-CPU profiling for application 
> developers.

right, off-cpu info in a 'perf trace' session is something we should
have, for instance. It should appear like a syscall, i.e. another way
out of the thread that will take some time to complete (sched_out ...
sched_in).

- Arnaldo
 
> Many thanks for the help, Arnaldo, and sorry again for the confusion on my 
> side.
> 
> Cheers
> -- 
> Milian Wolff | milian.wolff@kdab.com | Software Engineer
> KDAB (Deutschland) GmbH&Co KG, a KDAB Group company
> Tel: +49-30-521325470
> KDAB - The Qt Experts