Re: Tracing sched_switch events for client application when process is switched back in

[Milian Wolff <milian.wolff@kdab.com>]

[-- Attachment #1: Type: text/plain, Size: 3357 bytes --]

On Monday, November 14, 2016 12:36:11 PM CET Arnaldo Carvalho de Melo wrote:
> Em Mon, Nov 14, 2016 at 03:10:26PM +0100, Milian Wolff escreveu:
> > On Monday, October 10, 2016 9:36:55 PM CET Arnaldo Carvalho de Melo wrote:
> > > Em Thu, Oct 06, 2016 at 06:38:28PM +0200, Milian Wolff escreveu:
> > > > only find 7, whereas tracing sched_switch finds all 100 that actually
> > > > take
> > > > place?
> > 
> > Hey there,
> > 
> > long delay but I looked at this again today.
> > 
> > > This is looking only for the sched switches for the monitored workload,
> > > which in this case is 'sleep 1', this wasn't a system wide session.
> > > 
> > > add -a and you'll get those other switches, if I got what you described.
> > 
> > I've used the sources of the simple C application I showed in the first
> > email of this thread (still quoted above). Then I try:
> > 
> > perf record --switch-events -a ./a.out
> > 
> > And indeed, I get all switch events. But I now also profile all
> > applications on the machine, which is not what I want. I want to combine
> > the following
>
> So you remove that '-a' from that command line and gets just the context
> switches for that ./a.out app, no?
>
> Lets see:

<snip>

> 5 out, 5 in, in a lightly loaded machine, plenty of CPUs, looks about right,
> no?

Hm this is odd. Before writing the last mail, and the one from one month ago, 
I tried this and it did not work. But now it does seem to work as intended, 
even with my simple application code...

perf record --switch-events ./a.out
perf script --show-switch-events | grep -c SWITCH
200

So this clearly works as intended - sorry for the noise.

> But...
> 
> > features:
> > 
> > - sample CPU events for a given application with backtraces
> > 
> > 	perf record --call-graph dwarf ./foo
> > 
> > - record switch-out events with backtraces (`-e
> > sched:sched_switch/fp=dwarf/`) - somehow record the switch-in events,
> > which are associated with a different process and thus currently
> > discarded
> 
> You want the backtraces with that, and since this is a meta-event... Humm,
> for the switch in would the above be sufficient? What value would be in
> knowing the backtrace in that case?

Knowing the backtrace would ensure I get the real location that triggered a 
switch, i.e. often times that comes from a syscall like futex or the like. Or 
would I always get a CPU event (e.g. cycles) with a backtrace before the 
switch happens, pointing to the same location?

> So something like would do? Try not looking at the "PERF_RECORD_SWITCH OUT"
> lines, they are generated by the kernel _after_ the corresponding
> sched:sched_switch is registered:
> 
> [root@jouet ~]# perf record --call-graph=dwarf --switch-events -e

<snip>

Yes, that seems to work just fine:

perf record -e cycles -e sched:sched_switch \
	--call-graph dwarf --switch-events \
	./a.out

That gives me all information. With some post-processing one could then turn 
this into a really useful profiling tool, as one will only need a single perf 
record step to allow both, on-CPU as well as off-CPU profiling for application 
developers.

Many thanks for the help, Arnaldo, and sorry again for the confusion on my 
side.

Cheers
-- 
Milian Wolff | milian.wolff@kdab.com | Software Engineer
KDAB (Deutschland) GmbH&Co KG, a KDAB Group company
Tel: +49-30-521325470
KDAB - The Qt Experts

[-- Attachment #2: smime.p7s --]
[-- Type: application/pkcs7-signature, Size: 5903 bytes --]