Accessing sample data (like period) for trace events from python scripts

Accessing sample data (like period) for trace events from python scripts

[Milian Wolff]

[-- Attachment #1: Type: text/plain, Size: 887 bytes --]

Hey all,

It doesn't seem to be possible to access the period, tid and various other 
sample fields from a python (and perl?) script using `perf script`. My use-
case would be analyzing a perf data file created by perf inject --sched-stat.

I have hacked something together locally in python_process_tracepoint, but 
wonder what an upstreamable version should look like:

- I simply added the sample data to the dict, like is done in 
python_process_general_event

- Alternatively, one could think of more `common_period(context)` functions. 
If that is preferable, I could add the sample to the scripting_context and 
make that work.

Please advise the path forward, I'm willing to send a patch upstream to make 
it work.

Cheers
-- 
Milian Wolff | milian.wolff@kdab.com | Software Engineer
KDAB (Deutschland) GmbH&Co KG, a KDAB Group company
Tel: +49-30-521325470
KDAB - The Qt Experts

[-- Attachment #2: smime.p7s --]
[-- Type: application/pkcs7-signature, Size: 5903 bytes --]

Re: Accessing sample data (like period) for trace events from python scripts

[Namhyung Kim]

Hi Milian,

On Tue, Aug 16, 2016 at 08:42:01PM +0200, Milian Wolff wrote:
> Hey all,
> 
> It doesn't seem to be possible to access the period, tid and various other 
> sample fields from a python (and perl?) script using `perf script`. My use-
> case would be analyzing a perf data file created by perf inject
> --sched-stat.

I think you can access param_dict["sample"]["period"].

Thanks,
Namhyung


> 
> I have hacked something together locally in python_process_tracepoint, but 
> wonder what an upstreamable version should look like:
> 
> - I simply added the sample data to the dict, like is done in 
> python_process_general_event
> 
> - Alternatively, one could think of more `common_period(context)` functions. 
> If that is preferable, I could add the sample to the scripting_context and 
> make that work.
> 
> Please advise the path forward, I'm willing to send a patch upstream to make 
> it work.
> 
> Cheers
> -- 
> Milian Wolff | milian.wolff@kdab.com | Software Engineer
> KDAB (Deutschland) GmbH&Co KG, a KDAB Group company
> Tel: +49-30-521325470
> KDAB - The Qt Experts

Re: Accessing sample data (like period) for trace events from python scripts

[Milian Wolff]

[-- Attachment #1: Type: text/plain, Size: 1338 bytes --]

On Wednesday, August 17, 2016 12:05:53 PM CEST Namhyung Kim wrote:
> Hi Milian,
> 
> On Tue, Aug 16, 2016 at 08:42:01PM +0200, Milian Wolff wrote:
> > Hey all,
> > 
> > It doesn't seem to be possible to access the period, tid and various other
> > sample fields from a python (and perl?) script using `perf script`. My
> > use-
> > case would be analyzing a perf data file created by perf inject
> > --sched-stat.
> 
> I think you can access param_dict["sample"]["period"].

No, that is not possible, but would be a viable solution as well.

~~~~~~~~~~~~~~~~~~
/tmp$ cat perf-script.py

import os
import sys

sys.path.append(os.environ['PERF_EXEC_PATH'] + \
        '/scripts/python/Perf-Trace-Util/lib/Perf/Trace')

from perf_trace_context import *
from Core import *

def trace_unhandled(event_name, context, event_fields_dict):
        print event_fields_dict['sample']['period']

/tmp$ perf script -s perf-script.py
Traceback (most recent call last):
  File "perf-script.py", line 11, in trace_unhandled
    print event_fields_dict['sample']['period']
KeyError: 'sample'
Fatal Python error: problem in Python trace event handler
Aborted (core dumped)
~~~~~~~~~~~~~~~~~~

Cheers
-- 
Milian Wolff | milian.wolff@kdab.com | Software Engineer
KDAB (Deutschland) GmbH&Co KG, a KDAB Group company
Tel: +49-30-521325470
KDAB - The Qt Experts

[-- Attachment #2: smime.p7s --]
[-- Type: application/pkcs7-signature, Size: 5903 bytes --]

Re: Accessing sample data (like period) for trace events from python scripts

[Namhyung Kim]

On Wed, Aug 17, 2016 at 10:48:31AM +0200, Milian Wolff wrote:
> On Wednesday, August 17, 2016 12:05:53 PM CEST Namhyung Kim wrote:
> > Hi Milian,
> > 
> > On Tue, Aug 16, 2016 at 08:42:01PM +0200, Milian Wolff wrote:
> > > Hey all,
> > > 
> > > It doesn't seem to be possible to access the period, tid and various other
> > > sample fields from a python (and perl?) script using `perf script`. My
> > > use-
> > > case would be analyzing a perf data file created by perf inject
> > > --sched-stat.
> > 
> > I think you can access param_dict["sample"]["period"].
> 
> No, that is not possible, but would be a viable solution as well.

Sorry for late reply.  Oh, you wanted to access period of tracepoint
but I guess it's always 1, no?

Thanks,
Namhyung


> 
> ~~~~~~~~~~~~~~~~~~
> /tmp$ cat perf-script.py
> 
> import os
> import sys
> 
> sys.path.append(os.environ['PERF_EXEC_PATH'] + \
>         '/scripts/python/Perf-Trace-Util/lib/Perf/Trace')
> 
> from perf_trace_context import *
> from Core import *
> 
> def trace_unhandled(event_name, context, event_fields_dict):
>         print event_fields_dict['sample']['period']
> 
> /tmp$ perf script -s perf-script.py
> Traceback (most recent call last):
>   File "perf-script.py", line 11, in trace_unhandled
>     print event_fields_dict['sample']['period']
> KeyError: 'sample'
> Fatal Python error: problem in Python trace event handler
> Aborted (core dumped)
> ~~~~~~~~~~~~~~~~~~
> 
> Cheers
> -- 
> Milian Wolff | milian.wolff@kdab.com | Software Engineer
> KDAB (Deutschland) GmbH&Co KG, a KDAB Group company
> Tel: +49-30-521325470
> KDAB - The Qt Experts

Re: Accessing sample data (like period) for trace events from python scripts

[Namhyung Kim]

On Fri, Aug 19, 2016 at 10:35:49AM +0900, Namhyung Kim wrote:
> On Wed, Aug 17, 2016 at 10:48:31AM +0200, Milian Wolff wrote:
> > On Wednesday, August 17, 2016 12:05:53 PM CEST Namhyung Kim wrote:
> > > Hi Milian,
> > > 
> > > On Tue, Aug 16, 2016 at 08:42:01PM +0200, Milian Wolff wrote:
> > > > Hey all,
> > > > 
> > > > It doesn't seem to be possible to access the period, tid and various other
> > > > sample fields from a python (and perl?) script using `perf script`. My
> > > > use-
> > > > case would be analyzing a perf data file created by perf inject
> > > > --sched-stat.
> > > 
> > > I think you can access param_dict["sample"]["period"].
> > 
> > No, that is not possible, but would be a viable solution as well.
> 
> Sorry for late reply.  Oh, you wanted to access period of tracepoint
> but I guess it's always 1, no?

OK, it seems sched_stat events have periods bigger than 1.  But you
can access it using 'runtime' or 'delay' field in the event too.  I
think sample data of tracepoint event is mostly meaningless and you
should access to the event fields for useful information.  That's why
it doesn't provide the info for tracepoint events, I guess.

What do you specifically want to know from sample data?

Thanks,
Namhyung


> 
> > 
> > ~~~~~~~~~~~~~~~~~~
> > /tmp$ cat perf-script.py
> > 
> > import os
> > import sys
> > 
> > sys.path.append(os.environ['PERF_EXEC_PATH'] + \
> >         '/scripts/python/Perf-Trace-Util/lib/Perf/Trace')
> > 
> > from perf_trace_context import *
> > from Core import *
> > 
> > def trace_unhandled(event_name, context, event_fields_dict):
> >         print event_fields_dict['sample']['period']
> > 
> > /tmp$ perf script -s perf-script.py
> > Traceback (most recent call last):
> >   File "perf-script.py", line 11, in trace_unhandled
> >     print event_fields_dict['sample']['period']
> > KeyError: 'sample'
> > Fatal Python error: problem in Python trace event handler
> > Aborted (core dumped)
> > ~~~~~~~~~~~~~~~~~~
> > 
> > Cheers
> > -- 
> > Milian Wolff | milian.wolff@kdab.com | Software Engineer
> > KDAB (Deutschland) GmbH&Co KG, a KDAB Group company
> > Tel: +49-30-521325470
> > KDAB - The Qt Experts
> 
> 

Re: Accessing sample data (like period) for trace events from python scripts

[Milian Wolff]

[-- Attachment #1: Type: text/plain, Size: 2020 bytes --]

On Freitag, 19. August 2016 10:51:01 CEST Namhyung Kim wrote:
> On Fri, Aug 19, 2016 at 10:35:49AM +0900, Namhyung Kim wrote:
> > On Wed, Aug 17, 2016 at 10:48:31AM +0200, Milian Wolff wrote:
> > > On Wednesday, August 17, 2016 12:05:53 PM CEST Namhyung Kim wrote:
> > > > Hi Milian,
> > > > 
> > > > On Tue, Aug 16, 2016 at 08:42:01PM +0200, Milian Wolff wrote:
> > > > > Hey all,
> > > > > 
> > > > > It doesn't seem to be possible to access the period, tid and various
> > > > > other
> > > > > sample fields from a python (and perl?) script using `perf script`.
> > > > > My
> > > > > use-
> > > > > case would be analyzing a perf data file created by perf inject
> > > > > --sched-stat.
> > > > 
> > > > I think you can access param_dict["sample"]["period"].
> > > 
> > > No, that is not possible, but would be a viable solution as well.
> > 
> > Sorry for late reply.  Oh, you wanted to access period of tracepoint
> > but I guess it's always 1, no?
> 
> OK, it seems sched_stat events have periods bigger than 1.  But you
> can access it using 'runtime' or 'delay' field in the event too.  I
> think sample data of tracepoint event is mostly meaningless and you
> should access to the event fields for useful information.  That's why
> it doesn't provide the info for tracepoint events, I guess.
>
> What do you specifically want to know from sample data?

Using `perf inject --sched-stat`, I can get the sched_stat event period to the 
sched_switch event (which has the proper backtrace). That is the data I want 
to access. This is doable when I run:

perf script -F event,period

But it is not accessible from Perl or Python.

Note: the runtime/delay fields are not translated by perf inject, it simply 
sets the period which is then used by the other perf tools.

I really think it should be made possible to access that data from Python/
Perl.

Cheers
-- 
Milian Wolff | milian.wolff@kdab.com | Software Engineer
KDAB (Deutschland) GmbH&Co KG, a KDAB Group company
Tel: +49-30-521325470
KDAB - The Qt Experts

[-- Attachment #2: smime.p7s --]
[-- Type: application/pkcs7-signature, Size: 5903 bytes --]

Re: Accessing sample data (like period) for trace events from python scripts

[Namhyung Kim]

Hello Milian,

On Sun, Aug 21, 2016 at 5:30 AM, Milian Wolff <milian.wolff@kdab.com> wrote:
> On Freitag, 19. August 2016 10:51:01 CEST Namhyung Kim wrote:
>> On Fri, Aug 19, 2016 at 10:35:49AM +0900, Namhyung Kim wrote:
>> > On Wed, Aug 17, 2016 at 10:48:31AM +0200, Milian Wolff wrote:
>> > > On Wednesday, August 17, 2016 12:05:53 PM CEST Namhyung Kim wrote:
>> > > > Hi Milian,
>> > > >
>> > > > On Tue, Aug 16, 2016 at 08:42:01PM +0200, Milian Wolff wrote:
>> > > > > Hey all,
>> > > > >
>> > > > > It doesn't seem to be possible to access the period, tid and various
>> > > > > other
>> > > > > sample fields from a python (and perl?) script using `perf script`.
>> > > > > My
>> > > > > use-
>> > > > > case would be analyzing a perf data file created by perf inject
>> > > > > --sched-stat.
>> > > >
>> > > > I think you can access param_dict["sample"]["period"].
>> > >
>> > > No, that is not possible, but would be a viable solution as well.
>> >
>> > Sorry for late reply.  Oh, you wanted to access period of tracepoint
>> > but I guess it's always 1, no?
>>
>> OK, it seems sched_stat events have periods bigger than 1.  But you
>> can access it using 'runtime' or 'delay' field in the event too.  I
>> think sample data of tracepoint event is mostly meaningless and you
>> should access to the event fields for useful information.  That's why
>> it doesn't provide the info for tracepoint events, I guess.
>>
>> What do you specifically want to know from sample data?
>
> Using `perf inject --sched-stat`, I can get the sched_stat event period to the
> sched_switch event (which has the proper backtrace). That is the data I want
> to access. This is doable when I run:
>
> perf script -F event,period
>
> But it is not accessible from Perl or Python.
>
> Note: the runtime/delay fields are not translated by perf inject, it simply
> sets the period which is then used by the other perf tools.
>
> I really think it should be made possible to access that data from Python/
> Perl.

As I said, what you really want is 'runtime' or 'delay' and they are
already accessible from scripts.  The perf inject uses the period
directly since it knows it's same as the value of the fields but I'm
not sure it's the right way to do.  Anyway, if you want to make the
period accessible from scripts, you can add it in
util/scripting-engines/trace-event-python.c:python_process_tracepoint()
and the perl counterpart.

Thanks,
Namhyung

Re: Accessing sample data (like period) for trace events from python scripts

[Milian Wolff]

[-- Attachment #1: Type: text/plain, Size: 3184 bytes --]

On Dienstag, 23. August 2016 11:42:54 CEST Namhyung Kim wrote:
> Hello Milian,

Hey Namhyung,

> On Sun, Aug 21, 2016 at 5:30 AM, Milian Wolff <milian.wolff@kdab.com> wrote:
> > On Freitag, 19. August 2016 10:51:01 CEST Namhyung Kim wrote:
> >> On Fri, Aug 19, 2016 at 10:35:49AM +0900, Namhyung Kim wrote:
> >> > On Wed, Aug 17, 2016 at 10:48:31AM +0200, Milian Wolff wrote:
> >> > > On Wednesday, August 17, 2016 12:05:53 PM CEST Namhyung Kim wrote:
> >> > > > Hi Milian,
> >> > > > 
> >> > > > On Tue, Aug 16, 2016 at 08:42:01PM +0200, Milian Wolff wrote:
> >> > > > > Hey all,
> >> > > > > 
> >> > > > > It doesn't seem to be possible to access the period, tid and
> >> > > > > various
> >> > > > > other
> >> > > > > sample fields from a python (and perl?) script using `perf
> >> > > > > script`.
> >> > > > > My
> >> > > > > use-
> >> > > > > case would be analyzing a perf data file created by perf inject
> >> > > > > --sched-stat.
> >> > > > 
> >> > > > I think you can access param_dict["sample"]["period"].
> >> > > 
> >> > > No, that is not possible, but would be a viable solution as well.
> >> > 
> >> > Sorry for late reply.  Oh, you wanted to access period of tracepoint
> >> > but I guess it's always 1, no?
> >> 
> >> OK, it seems sched_stat events have periods bigger than 1.  But you
> >> can access it using 'runtime' or 'delay' field in the event too.  I
> >> think sample data of tracepoint event is mostly meaningless and you
> >> should access to the event fields for useful information.  That's why
> >> it doesn't provide the info for tracepoint events, I guess.
> >> 
> >> What do you specifically want to know from sample data?
> > 
> > Using `perf inject --sched-stat`, I can get the sched_stat event period to
> > the sched_switch event (which has the proper backtrace). That is the data
> > I want to access. This is doable when I run:
> > 
> > perf script -F event,period
> > 
> > But it is not accessible from Perl or Python.
> > 
> > Note: the runtime/delay fields are not translated by perf inject, it
> > simply
> > sets the period which is then used by the other perf tools.
> > 
> > I really think it should be made possible to access that data from Python/
> > Perl.
> 
> As I said, what you really want is 'runtime' or 'delay' and they are
> already accessible from scripts. The perf inject uses the period
> directly since it knows it's same as the value of the fields but I'm
> not sure it's the right way to do. 

If we don't store it in the event's period field, then we would not be able to 
use either of the following in `perf report`:

  --show-total-period
  -g period

I think this would make this feature less intuitive to use, no?

> Anyway, if you want to make the period accessible from scripts, you can add 
it in util/scripting-engines/trace-event-python.c:python_process_tracepoint()
> and the perl counterpart.

OK, I will provide a patch for that. Would it be OK to simply make the whole 
sample accessible, just like it already is for non-tracepoint events?

Cheers
-- 
Milian Wolff | milian.wolff@kdab.com | Software Engineer
KDAB (Deutschland) GmbH&Co KG, a KDAB Group company
Tel: +49-30-521325470
KDAB - The Qt Experts

[-- Attachment #2: smime.p7s --]
[-- Type: application/pkcs7-signature, Size: 5903 bytes --]