perf: prctl(PR_TASK_PERF_EVENTS_DISABLE) has no effect

perf: prctl(PR_TASK_PERF_EVENTS_DISABLE) has no effect

[Andrew Steets]

prctl(PR_TASK_PERF_EVENTS_DISABLE) doesn't appear to disable perf event
counters.  Here is a demonstration program:

#include <linux/prctl.h>

void loop_events_disabled() {
     volatile int x;
     int i;
     for (i = 0; i < 1000000000; i++)
         x = i;
     return;
}

void loop_events_enabled() {
     volatile int x;
     int i;
     for (i = 0; i < 1000000000; i++)
         x = i;
     return;
}

int main(int argc, char **argv) {
     prctl(PR_TASK_PERF_EVENTS_ENABLE);
     loop_events_enabled();
     prctl(PR_TASK_PERF_EVENTS_DISABLE);
     loop_events_disabled();
}

I would not expect to see loop_events_disabled() show up in the profile
as reported by perf report, but it does.

$ perf record ./a.out
$ perf report -n --stdio
# Events: 3K cycles
#
# Overhead  Samples    Command      Shared Object                Symbol
# ........ ..........  .......  .................  ....................
#
     51.80%       1679    a.out  a.out              [.] loop_events_enabled
     48.07%       1578    a.out  a.out              [.] loop_events_disabled
      0.03%          5    a.out  [kernel.kallsyms]  [k] intel_pmu_enable_all
      0.03%          1    a.out  [kernel.kallsyms]  [k] sched_clock
      0.03%          1    a.out  [kernel.kallsyms]  [k] ktime_get
      0.03%          1    a.out  [kernel.kallsyms]  [k] update_cpu_load
      0.01%          1    a.out  [sunrpc]           [k] generic_match

I have tested this on several kernels including 3.3rc1.  Can anyone tell
me if I'm using this wrong or if this is a bug?

-Andrew

-- 

---------------------------------------------------------------
This email, along with any attachments, is confidential. If you 
believe you received this message in error, please contact the 
sender immediately and delete all copies of the message.  
Thank you.

Re: perf: prctl(PR_TASK_PERF_EVENTS_DISABLE) has no effect

[Peter Zijlstra]

On Fri, 2012-01-27 at 11:03 -0600, Andrew Steets wrote:
> Can anyone tell
> me if I'm using this wrong or if this is a bug? 

You're using it wrong, it will disable events you own (created) not
events that monitor you.

Re: perf: prctl(PR_TASK_PERF_EVENTS_DISABLE) has no effect

[Andrew Steets]

On 1/27/12 11:12 AM, Peter Zijlstra wrote:
> On Fri, 2012-01-27 at 11:03 -0600, Andrew Steets wrote:
>> Can anyone tell
>> me if I'm using this wrong or if this is a bug?
>
> You're using it wrong, it will disable events you own (created) not
> events that monitor you.

Is there an alternate way of disabling events that monitor the current
process?  I ask because I came across the following description in
tools/perf/design.txt:

     A process can enable or disable all the counter groups that are
     attached to it, using prctl:

         prctl(PR_TASK_PERF_EVENTS_ENABLE);

         prctl(PR_TASK_PERF_EVENTS_DISABLE);

     This applies to all counters on the current process, whether created
     by this process or by another, and doesn't affect any counters that
     this process has created on other processes.  It only enables or
     disables the group leaders, not any other members in the groups.

-Andrew

-- 

---------------------------------------------------------------
This email, along with any attachments, is confidential. If you 
believe you received this message in error, please contact the 
sender immediately and delete all copies of the message.  
Thank you.

Re: perf: prctl(PR_TASK_PERF_EVENTS_DISABLE) has no effect

[Peter Zijlstra]

On Fri, 2012-01-27 at 14:06 -0600, Andrew Steets wrote:
> On 1/27/12 11:12 AM, Peter Zijlstra wrote:
> > On Fri, 2012-01-27 at 11:03 -0600, Andrew Steets wrote:
> >> Can anyone tell
> >> me if I'm using this wrong or if this is a bug?
> >
> > You're using it wrong, it will disable events you own (created) not
> > events that monitor you.
> 
> Is there an alternate way of disabling events that monitor the current
> process? 

Nope, nor will there ever be.

>  I ask because I came across the following description in
> tools/perf/design.txt:
> 
>      A process can enable or disable all the counter groups that are
>      attached to it, using prctl:
> 
>          prctl(PR_TASK_PERF_EVENTS_ENABLE);
> 
>          prctl(PR_TASK_PERF_EVENTS_DISABLE);
> 
>      This applies to all counters on the current process, whether created
>      by this process or by another, and doesn't affect any counters that
>      this process has created on other processes.  It only enables or
>      disables the group leaders, not any other members in the groups.


That's wrong.. also I wouldn't ever allow such a 'feature', that's just
asking for trouble.

Re: perf: prctl(PR_TASK_PERF_EVENTS_DISABLE) has no effect

[Ingo Molnar]

* Andrew Steets <asteets@rgmadvisors.com> wrote:

> prctl(PR_TASK_PERF_EVENTS_DISABLE) doesn't appear to disable 
> perf event counters.  Here is a demonstration program:

btw., what's your usecase?

Thanks,

	Ingo

Re: perf: prctl(PR_TASK_PERF_EVENTS_DISABLE) has no effect

[Andrew Steets]

On 1/28/12 6:01 AM, Ingo Molnar wrote:

>> prctl(PR_TASK_PERF_EVENTS_DISABLE) doesn't appear to disable
>> perf event counters.  Here is a demonstration program:
>
> btw., what's your usecase?
>

I'm trying to profile a small section of a long-running program.  I ran
into trouble using call-graph recording and I thought this might be an
alternative way of getting what I was after.

-Andrew

-- 

---------------------------------------------------------------
This email, along with any attachments, is confidential. If you 
believe you received this message in error, please contact the 
sender immediately and delete all copies of the message.  
Thank you.

Re: perf: prctl(PR_TASK_PERF_EVENTS_DISABLE) has no effect

[Ingo Molnar]

* Andrew Steets <asteets@rgmadvisors.com> wrote:

> On 1/28/12 6:01 AM, Ingo Molnar wrote:
> 
> >> prctl(PR_TASK_PERF_EVENTS_DISABLE) doesn't appear to 
> >> disable perf event counters.  Here is a demonstration 
> >> program:
> >
> > btw., what's your usecase?
> 
> I'm trying to profile a small section of a long-running 
> program.  I ran into trouble using call-graph recording and I 
> thought this might be an alternative way of getting what I was 
> after.

That usecase indeed makes sense. Peter, could we allow this for 
privileged tasks, depending on the perf_paranoia settings or 
such?

Thanks,

	Ingo

Re: perf: prctl(PR_TASK_PERF_EVENTS_DISABLE) has no effect

[Gleb Natapov]

On Sun, Jan 29, 2012 at 05:32:35PM +0100, Ingo Molnar wrote:
> 
> * Andrew Steets <asteets@rgmadvisors.com> wrote:
> 
> > On 1/28/12 6:01 AM, Ingo Molnar wrote:
> > 
> > >> prctl(PR_TASK_PERF_EVENTS_DISABLE) doesn't appear to 
> > >> disable perf event counters.  Here is a demonstration 
> > >> program:
> > >
> > > btw., what's your usecase?
> > 
> > I'm trying to profile a small section of a long-running 
> > program.  I ran into trouble using call-graph recording and I 
> > thought this might be an alternative way of getting what I was 
> > after.
> 
> That usecase indeed makes sense. Peter, could we allow this for 
> privileged tasks, depending on the perf_paranoia settings or 
> such?
> 
This sounds useful not only for privileged tasks. Why not make
it event attribute? If user wants PR_TASK_PERF_EVENTS_DISABLE be
respected for an event it can specify special argument like:

perf record -e cycles:n ./a.out

--
			Gleb.

Re: perf: prctl(PR_TASK_PERF_EVENTS_DISABLE) has no effect

[Peter Zijlstra]

On Sun, 2012-01-29 at 17:32 +0100, Ingo Molnar wrote:
> * Andrew Steets <asteets@rgmadvisors.com> wrote:
> 
> > On 1/28/12 6:01 AM, Ingo Molnar wrote:
> > 
> > >> prctl(PR_TASK_PERF_EVENTS_DISABLE) doesn't appear to 
> > >> disable perf event counters.  Here is a demonstration 
> > >> program:
> > >
> > > btw., what's your usecase?
> > 
> > I'm trying to profile a small section of a long-running 
> > program.  I ran into trouble using call-graph recording and I 
> > thought this might be an alternative way of getting what I was 
> > after.
> 
> That usecase indeed makes sense. Peter, could we allow this for 
> privileged tasks, depending on the perf_paranoia settings or 
> such?

I really dislike it. The sane way around this would be to allow easy
self-profiling instead of doing things arse about face like that.

Re: perf: prctl(PR_TASK_PERF_EVENTS_DISABLE) has no effect

[Ingo Molnar]

* Peter Zijlstra <a.p.zijlstra@chello.nl> wrote:

> On Sun, 2012-01-29 at 17:32 +0100, Ingo Molnar wrote:
> > * Andrew Steets <asteets@rgmadvisors.com> wrote:
> > 
> > > On 1/28/12 6:01 AM, Ingo Molnar wrote:
> > > 
> > > >> prctl(PR_TASK_PERF_EVENTS_DISABLE) doesn't appear to 
> > > >> disable perf event counters.  Here is a demonstration 
> > > >> program:
> > > >
> > > > btw., what's your usecase?
> > > 
> > > I'm trying to profile a small section of a long-running 
> > > program.  I ran into trouble using call-graph recording 
> > > and I thought this might be an alternative way of getting 
> > > what I was after.
> > 
> > That usecase indeed makes sense. Peter, could we allow this 
> > for privileged tasks, depending on the perf_paranoia 
> > settings or such?
> 
> I really dislike it. The sane way around this would be to 
> allow easy self-profiling instead of doing things arse about 
> face like that.

So, what workflow are you suggesting to Andrew?

I guess we are also hurting from the lack of dwarf stack 
backtrace decoding - that would allow the filtering by parent 
function without modifying the code. I think Frederic had a 
prototype working for 32-bit - any update on that?

Andrew could work that problem around right now by adding:

   -fno-omit-frame-pointer

to the build of the utility - that should activate -g and 
perf-report's --parent filter should also work fine.

Thanks,

	Ingo

Re: perf: prctl(PR_TASK_PERF_EVENTS_DISABLE) has no effect

[Peter Zijlstra]

On Mon, 2012-01-30 at 11:11 +0100, Ingo Molnar wrote:
> So, what workflow are you suggesting to Andrew?

Librarize perf record, then in your code do something like:

#include "perf_record.h"

  handle = perf_record_init(); /* creates perf events and creates
                                  a record thread that writes samples
                                  to perf.data, consumes env(PERF_*)
                                  for configuration, registers with
                                  at_exit() for cleanup */
  if (!handle)
    /* burn */

  /* do you other code */

  perf_record_start(handle);

  /* do the bit you want profiled */

  perf_record_stop(handle);

Then build with -lperfrecord or so. Not too hard, right?

Re: perf: prctl(PR_TASK_PERF_EVENTS_DISABLE) has no effect

[Ingo Molnar]

* Peter Zijlstra <a.p.zijlstra@chello.nl> wrote:

> On Mon, 2012-01-30 at 11:11 +0100, Ingo Molnar wrote:
> > So, what workflow are you suggesting to Andrew?
> 
> Librarize perf record, then in your code do something like:
> 
> #include "perf_record.h"

Maybe. (and then it shouldnt be limited to perf_record.h but 
should be events.h plus libevents.so or such)

> 
>   handle = perf_record_init(); /* creates perf events and creates
>                                   a record thread that writes samples
>                                   to perf.data, consumes env(PERF_*)
>                                   for configuration, registers with
>                                   at_exit() for cleanup */
>   if (!handle)
>     /* burn */
> 
>   /* do you other code */
> 
>   perf_record_start(handle);
> 
>   /* do the bit you want profiled */
> 
>   perf_record_stop(handle);
> 
> Then build with -lperfrecord or so. Not too hard, right?

Isnt a simple prctl() so much easier and faster?

What's your concern with the prctl()? This would arguably be the 
right kind of usage for prctl(): it's an established API/ABI for 
process/task-wide settings.

Thanks,

	Ingo

Re: perf: prctl(PR_TASK_PERF_EVENTS_DISABLE) has no effect

[Peter Zijlstra]

On Mon, 2012-01-30 at 12:31 +0100, Ingo Molnar wrote:
> * Peter Zijlstra <a.p.zijlstra@chello.nl> wrote:
> 
> > On Mon, 2012-01-30 at 11:11 +0100, Ingo Molnar wrote:
> > > So, what workflow are you suggesting to Andrew?
> > 
> > Librarize perf record, then in your code do something like:
> > 
> > #include "perf_record.h"
> 
> Maybe. (and then it shouldnt be limited to perf_record.h but 
> should be events.h plus libevents.so or such)

Yes it should be, you want to reserve the more generic name for less
narrow interfaces.

> > 
> >   handle = perf_record_init(); /* creates perf events and creates
> >                                   a record thread that writes samples
> >                                   to perf.data, consumes env(PERF_*)
> >                                   for configuration, registers with
> >                                   at_exit() for cleanup */
> >   if (!handle)
> >     /* burn */
> > 
> >   /* do you other code */
> > 
> >   perf_record_start(handle);
> > 
> >   /* do the bit you want profiled */
> > 
> >   perf_record_stop(handle);
> > 
> > Then build with -lperfrecord or so. Not too hard, right?
> 
> Isnt a simple prctl() so much easier and faster?

I really don't want to add another two prctl()s for this, ideally I'd
remove the ones we have now, but I've never done due to maintaining
backwards blah..

> What's your concern with the prctl()? This would arguably be the 
> right kind of usage for prctl(): it's an established API/ABI for 
> process/task-wide settings.

Its doing things backwards, also the whole concept of allowing people to
hide things from a profiler is so rotten I'm not willing to even
consider the notion.

Re: perf: prctl(PR_TASK_PERF_EVENTS_DISABLE) has no effect

[Ingo Molnar]

* Peter Zijlstra <a.p.zijlstra@chello.nl> wrote:

> > What's your concern with the prctl()? This would arguably be 
> > the right kind of usage for prctl(): it's an established 
> > API/ABI for process/task-wide settings.
> 
> Its doing things backwards, [...]

What does that mean?

> [...] also the whole concept of allowing people to hide things 
> from a profiler is so rotten I'm not willing to even consider 
> the notion.

But what we want here is not to hide things from the profiler, 
what we want to be able to is to *ask* the profiler to hide 
things for us, and we do that for a good reason.

This distinction (which I agree is important) could be 
implemented by adding a "allow self-profiling" bit (default 
disabled) in the perf_attr. That way such code would not be able 
to 'hide' from a simple:

  perf top
  perf record -a

session, but would be able to self-profile from such a session:

  perf record --allow-self-profile ...

where there could be some easy shortcut for 
--allow-self-profile, such as:

  perf record -S

that way no code could ever hide from a profiler, only if the 
profiler is specifically allowing self-profiling. (opt in)

The librarization you suggested might make sense too - but i 
think people will gravitate towards the easier to use variant, 
and prctl() is as easy and straightforward as it gets.

We can Cc: it to Linus with an explicit [RFC] and he can shoot 
it down if the API is ugly - but I don't think it's ugly.

Hm?

Thanks,

	Ingo

Re: perf: prctl(PR_TASK_PERF_EVENTS_DISABLE) has no effect

[Arnaldo Carvalho de Melo]

Em Mon, Jan 30, 2012 at 12:01:33PM +0100, Peter Zijlstra escreveu:
> On Mon, 2012-01-30 at 11:11 +0100, Ingo Molnar wrote:
> > So, what workflow are you suggesting to Andrew?
> 
> Librarize perf record, then in your code do something like:
> 
> #include "perf_record.h"
> 
>   handle = perf_record_init(); /* creates perf events and creates
>                                   a record thread that writes samples
>                                   to perf.data, consumes env(PERF_*)
>                                   for configuration, registers with
>                                   at_exit() for cleanup */
>   if (!handle)
>     /* burn */
> 
>   /* do you other code */
> 
>   perf_record_start(handle);
> 
>   /* do the bit you want profiled */
> 
>   perf_record_stop(handle);
> 
> Then build with -lperfrecord or so. Not too hard, right?

This looks like the test__PERF_RECORD function in
tools/perf/builtin-test.c, copying here just the equivalent parts to the
above:

	#include "util/evlist.h"

        struct perf_record_opts opts = {
                .target_pid = -1,
                .target_tid = -1,
                .no_delay   = true,
                .freq       = 10,
                .mmap_pages = 256,
                .sample_id_all_avail = true,
        };

        struct perf_evlist *handle = perf_evlist__new(NULL, NULL);

        /* 
         * We need at least one evsel in the evlist, use the default 
         * one: "cycles". 
         */ 
        err = perf_evlist__add_default(evlist);

	perf_evlist__config_attrs(evlist, &opts);

	err = perf_evlist__open(evlist, opts.group);

        perf_evlist__enable(evlist);

	/* do the bit you want profiled */

        perf_evlist__disable(evlist);


----


perf_evlist__config_attrs will, among other chores, do:

        if (opts->target_pid == -1 && opts->target_tid == -1 &&
            !opts->system_wide) {
                attr->disabled = 1;
                attr->enable_on_exec = 1;
        }

And you can fudge it a bit more to your liking before calling
perf_evlist__open().

	This 'perf test' use case was more for preparing for monitoring
a workload that a program would start, setting up the events, etc,
leaving all disabled and then letting the workload begin which would, by
means of enable_on_exec start the counters/events.

	But I guess you can just setup evlist->workload.pid to getpid()
and use perf_evlist__enable()/ perf_evlist_disable() pairs to your
liking.

	Writing a test case that would do exactly what you want in 'perf
test' would be awesome and would allow us to figure out if the
abstractions in place are enough for your use case. Then we could
librarize just this subset and go from there.

	Take a look as well at test__basic_mmap, it may be useful as
another example of using these evsel/evlist classes.

	What is needed to link you can find in tools/perf/util/setup.py,
remove util/python.c and you should be set.

- Arnaldo

Re: perf: prctl(PR_TASK_PERF_EVENTS_DISABLE) has no effect

[Arnaldo Carvalho de Melo]

Em Mon, Jan 30, 2012 at 02:45:07PM +0100, Peter Zijlstra escreveu:
> On Mon, 2012-01-30 at 12:31 +0100, Ingo Molnar wrote:
> > * Peter Zijlstra <a.p.zijlstra@chello.nl> wrote:
> > > On Mon, 2012-01-30 at 11:11 +0100, Ingo Molnar wrote:
> > > > So, what workflow are you suggesting to Andrew?
> > > Librarize perf record, then in your code do something like:
> > > #include "perf_record.h"

> > Maybe. (and then it shouldnt be limited to perf_record.h but 
> > should be events.h plus libevents.so or such)

> Yes it should be, you want to reserve the more generic name for less
> narrow interfaces.

Agreed.
 
- Arnaldo

Re: perf: prctl(PR_TASK_PERF_EVENTS_DISABLE) has no effect

[Frederic Weisbecker]

On Mon, Jan 30, 2012 at 11:11:21AM +0100, Ingo Molnar wrote:
> 
> * Peter Zijlstra <a.p.zijlstra@chello.nl> wrote:
> 
> > On Sun, 2012-01-29 at 17:32 +0100, Ingo Molnar wrote:
> > > * Andrew Steets <asteets@rgmadvisors.com> wrote:
> > > 
> > > > On 1/28/12 6:01 AM, Ingo Molnar wrote:
> > > > 
> > > > >> prctl(PR_TASK_PERF_EVENTS_DISABLE) doesn't appear to 
> > > > >> disable perf event counters.  Here is a demonstration 
> > > > >> program:
> > > > >
> > > > > btw., what's your usecase?
> > > > 
> > > > I'm trying to profile a small section of a long-running 
> > > > program.  I ran into trouble using call-graph recording 
> > > > and I thought this might be an alternative way of getting 
> > > > what I was after.
> > > 
> > > That usecase indeed makes sense. Peter, could we allow this 
> > > for privileged tasks, depending on the perf_paranoia 
> > > settings or such?
> > 
> > I really dislike it. The sane way around this would be to 
> > allow easy self-profiling instead of doing things arse about 
> > face like that.
> 
> So, what workflow are you suggesting to Andrew?
> 
> I guess we are also hurting from the lack of dwarf stack 
> backtrace decoding - that would allow the filtering by parent 
> function without modifying the code. I think Frederic had a 
> prototype working for 32-bit - any update on that?

I haven't touched that for a while. I would be glad if
somebody else could relay me on this work.

I think Jiri Olsa has been working on something about Dwarf cfi
unwinding with a different approach. Mine was about dumping chunks
of stack and regs and do the unwinding on post processing. I think
Jiri is doing the unwinding from the event overflow fast path.

> 
> Andrew could work that problem around right now by adding:
> 
>    -fno-omit-frame-pointer
> 
> to the build of the utility - that should activate -g and 
> perf-report's --parent filter should also work fine.
> 
> Thanks,
> 
> 	Ingo

perf: prctl(PR_TASK_PERF_EVENTS_DISABLE) has no effect

[Iegorov Oleg]

as there was no proposed solution that helped me in response to the same 
post by Andrew Steets, I would like to know if it is possible to 
disable/enable perf event counters from the source code?

calling prctl(PR_TASK_PERF_EVENTS_DISABLE) has no effect, nor does 
compiling with -fno-omit-frame-pointer option.

It would be extremely useful to disable perf event counters for some 
parts of code and re-enable them for other parts of code, like:

prctl(PR_TASK_PERF_EVENTS_DISABLE);
// not important for performance analysis code
prctl(PR_TASK_PERF_EVENTS_ENABLE);
// code that needs to be analysed

and then, run perf:

$ perf record ./program
$ perf report

Can anyone tell me how can I enable such functionality?
Thank you

--Oleg

Re: perf: prctl(PR_TASK_PERF_EVENTS_DISABLE) has no effect

[Ingo Molnar]

* Iegorov Oleg <oleg_iegorov@mentor.com> wrote:

> as there was no proposed solution that helped me in response to the
> same post by Andrew Steets, I would like to know if it is possible
> to disable/enable perf event counters from the source code?
> 
> calling prctl(PR_TASK_PERF_EVENTS_DISABLE) has no effect, nor does
> compiling with -fno-omit-frame-pointer option.
> 
> It would be extremely useful to disable perf event counters for some
> parts of code and re-enable them for other parts of code, like:
> 
> prctl(PR_TASK_PERF_EVENTS_DISABLE);
> // not important for performance analysis code
> prctl(PR_TASK_PERF_EVENTS_ENABLE);
> // code that needs to be analysed
> 
> and then, run perf:
> 
> $ perf record ./program
> $ perf report
> 
> Can anyone tell me how can I enable such functionality?

So, the kernel bits to do this in a pretty quirky way are there, 
see:

    https://lkml.org/lkml/2012/1/30/99

but the librarization bits are definitely non-obvious to do and 
it's no surprise that it has not been done yet.

Regular 'perf record' in itself is not self-profiling - it's 
another task profiling you, so we cannot blanket allow 
PR_TASK_PERF_EVENTS_DISABLE to disable profiling.

What we might want to do instead on the kernel side to offer the 
functionality you are asking for is to enable 'public/weak' 
events be created by the profiler on an opt-in basis, which can 
be turned off by child tasks as well via 
PR_TASK_PERF_EVENTS_DISABLE.

On the profiling workflow side it would work in a very simple 
way, like this:

    perf record --self-profiling ./my-app

In your app you stick in appropriately placed 
PR_TASK_PERF_EVENTS_DISABLE/ENABLE calls.

On the technical side perf record creates events with that 
struct perf_event_attr::self_profiling flag set to 1. (the flag 
is disabled by default)

The PR_TASK_PERF_EVENTS_DISABLE code in the modified kernel then 
iterates through all events and disables those that have this 
flag set, not just the ones owned by this task.

Maybe someone on Cc: would be interested in implementing this 
new perf events feature?

Thanks,

	Ingo

Re: perf: prctl(PR_TASK_PERF_EVENTS_DISABLE) has no effect

[Peter Zijlstra]

On Fri, 2012-07-27 at 09:26 +0200, Ingo Molnar wrote:
> 
> Maybe someone on Cc: would be interested in implementing this 
> new perf events feature? 

Why would we go build new kernel interfaces because userspace is silly?

It really isn't that hard to make userspace do what is needed, it just
takes a bit of work.

Re: perf: prctl(PR_TASK_PERF_EVENTS_DISABLE) has no effect

[Ingo Molnar]

* Peter Zijlstra <a.p.zijlstra@chello.nl> wrote:

> On Fri, 2012-07-27 at 09:26 +0200, Ingo Molnar wrote:
> > 
> > Maybe someone on Cc: would be interested in implementing this 
> > new perf events feature? 
> 
> Why would we go build new kernel interfaces because userspace 
> is silly?

Because it's (much) easier to use the existing perf tools almost 
as-is instead of librarizing your own.

It would also allow other usecases, like self-profiling a 
library and then profiling it within the context of a larger app 
that you don't want to rebuild and which dynamically links this 
library.

It also allows system-wide profiling after you've modified a 
library to self-profile, while your suggestion does not allow 
that.

> It really isn't that hard to make userspace do what is needed, 
> it just takes a bit of work.

Even if your suggested solution was available (it isn't), my 
suggested approach is easier to use and covers more usecases.

User-space expecting the kernel to provide usable and minimal 
interfaces is not 'being silly'. It's the fundamental task of a 
kernel to provide them.

Thanks,

	Ingo

Re: perf: prctl(PR_TASK_PERF_EVENTS_DISABLE) has no effect

[Peter Zijlstra]

On Fri, 2012-07-27 at 10:18 +0200, Ingo Molnar wrote:
> * Peter Zijlstra <a.p.zijlstra@chello.nl> wrote:
> 
> > On Fri, 2012-07-27 at 09:26 +0200, Ingo Molnar wrote:
> > > 
> > > Maybe someone on Cc: would be interested in implementing this 
> > > new perf events feature? 
> > 
> > Why would we go build new kernel interfaces because userspace 
> > is silly?
> 
> Because it's (much) easier to use the existing perf tools almost 
> as-is instead of librarizing your own.
> 
> It would also allow other usecases, like self-profiling a 
> library and then profiling it within the context of a larger app 
> that you don't want to rebuild and which dynamically links this 
> library.

Uhm.. why not? For the first use proper self profiling, for the second
do a regular 3rd party profile.

> It also allows system-wide profiling after you've modified a 
> library to self-profile, while your suggestion does not allow 
> that.

But its no long self-profiling when some other process is involved. And
system wide is definitely not self.

> > It really isn't that hard to make userspace do what is needed, 
> > it just takes a bit of work.
> 
> Even if your suggested solution was available (it isn't), my 
> suggested approach is easier to use and covers more usecases.
> 
> User-space expecting the kernel to provide usable and minimal 
> interfaces is not 'being silly'. It's the fundamental task of a 
> kernel to provide them.

Bloating the interface for something that is already well possible is.

I really don't see the problem, other than that people simply don't want
to do work.

Re: perf: prctl(PR_TASK_PERF_EVENTS_DISABLE) has no effect

[Iegorov Oleg]

Le 2012-07-27 10:29, Peter Zijlstra a écrit :
> On Fri, 2012-07-27 at 10:18 +0200, Ingo Molnar wrote:
>> * Peter Zijlstra<a.p.zijlstra@chello.nl>  wrote:
>>
>>> On Fri, 2012-07-27 at 09:26 +0200, Ingo Molnar wrote:
>>>> Maybe someone on Cc: would be interested in implementing this
>>>> new perf events feature?
>>> Why would we go build new kernel interfaces because userspace
>>> is silly?
>> Because it's (much) easier to use the existing perf tools almost
>> as-is instead of librarizing your own.
>>
>> It would also allow other usecases, like self-profiling a
>> library and then profiling it within the context of a larger app
>> that you don't want to rebuild and which dynamically links this
>> library.
> Uhm.. why not? For the first use proper self profiling, for the second
> do a regular 3rd party profile.
>
>> It also allows system-wide profiling after you've modified a
>> library to self-profile, while your suggestion does not allow
>> that.
> But its no long self-profiling when some other process is involved. And
> system wide is definitely not self.
>
>>> It really isn't that hard to make userspace do what is needed,
>>> it just takes a bit of work.
>> Even if your suggested solution was available (it isn't), my
>> suggested approach is easier to use and covers more usecases.
>>
>> User-space expecting the kernel to provide usable and minimal
>> interfaces is not 'being silly'. It's the fundamental task of a
>> kernel to provide them.
> Bloating the interface for something that is already well possible is.
>
> I really don't see the problem, other than that people simply don't want
> to do work.
I totally agree with Ingo. The key words here are "much easier to use".

Moreover, in my use case, it would add a lot of complexity to rebuild 
the application each time a new performance event is added/removed from 
the event set (if I understood correctly Peter's approach).

Regards,
Oleg

Re: perf: prctl(PR_TASK_PERF_EVENTS_DISABLE) has no effect

[Ingo Molnar]

* Peter Zijlstra <a.p.zijlstra@chello.nl> wrote:

> > It also allows system-wide profiling after you've modified a 
> > library to self-profile, while your suggestion does not 
> > allow that.
> 
> But its no long self-profiling when some other process is 
> involved. And system wide is definitely not self.

If I'm a library developer and want to self-profile my 
modifications, then it's entirely appropriate to do a 
system-wide profile to see the system-wide use of this library!

System-wide and self-profiling is not exclusive. Think of it as 
a narrow, precise filter applied to a given area of 
functionality only.

AFAICS the alternative, under your method, would be to recompile 
every single app in the system - that's cumbersome beyond 
imagination, I wouldn't even call it a solution, let alone a 
quality implementation.

> > > It really isn't that hard to make userspace do what is 
> > > needed, it just takes a bit of work.
> > 
> > Even if your suggested solution was available (it isn't), my 
> > suggested approach is easier to use and covers more 
> > usecases.
> > 
> > User-space expecting the kernel to provide usable and 
> > minimal interfaces is not 'being silly'. It's the 
> > fundamental task of a kernel to provide them.
> 
> Bloating the interface for something that is already well 
> possible is.

There's no 'bloat' worth speaking off: a single bit out of an 
already allocated bitmap, plus a single check in an already 
existing loop, plus a single new command-line flag to the 
tooling side and minimal glue.

There's no runtime overhead to any other perf functionality, at 
all.

Thanks,

	Ingo

[RFD] perf: events defined contexts (was Re: perf: prctl(PR_TASK_PERF_EVENTS_DISABLE) has no effect)

[Frederic Weisbecker]

On Fri, Jul 27, 2012 at 09:26:47AM +0200, Ingo Molnar wrote:
> 
> * Iegorov Oleg <oleg_iegorov@mentor.com> wrote:
> 
> > as there was no proposed solution that helped me in response to the
> > same post by Andrew Steets, I would like to know if it is possible
> > to disable/enable perf event counters from the source code?
> > 
> > calling prctl(PR_TASK_PERF_EVENTS_DISABLE) has no effect, nor does
> > compiling with -fno-omit-frame-pointer option.
> > 
> > It would be extremely useful to disable perf event counters for some
> > parts of code and re-enable them for other parts of code, like:
> > 
> > prctl(PR_TASK_PERF_EVENTS_DISABLE);
> > // not important for performance analysis code
> > prctl(PR_TASK_PERF_EVENTS_ENABLE);
> > // code that needs to be analysed
> > 
> > and then, run perf:
> > 
> > $ perf record ./program
> > $ perf report
> > 
> > Can anyone tell me how can I enable such functionality?
> 
> So, the kernel bits to do this in a pretty quirky way are there, 
> see:
> 
>     https://lkml.org/lkml/2012/1/30/99
> 
> but the librarization bits are definitely non-obvious to do and 
> it's no surprise that it has not been done yet.
> 
> Regular 'perf record' in itself is not self-profiling - it's 
> another task profiling you, so we cannot blanket allow 
> PR_TASK_PERF_EVENTS_DISABLE to disable profiling.
> 
> What we might want to do instead on the kernel side to offer the 
> functionality you are asking for is to enable 'public/weak' 
> events be created by the profiler on an opt-in basis, which can 
> be turned off by child tasks as well via 
> PR_TASK_PERF_EVENTS_DISABLE.
> 
> On the profiling workflow side it would work in a very simple 
> way, like this:
> 
>     perf record --self-profiling ./my-app
> 
> In your app you stick in appropriately placed 
> PR_TASK_PERF_EVENTS_DISABLE/ENABLE calls.
> 
> On the technical side perf record creates events with that 
> struct perf_event_attr::self_profiling flag set to 1. (the flag 
> is disabled by default)
> 
> The PR_TASK_PERF_EVENTS_DISABLE code in the modified kernel then 
> iterates through all events and disables those that have this 
> flag set, not just the ones owned by this task.
> 
> Maybe someone on Cc: would be interested in implementing this 
> new perf events feature?

The problem is more general than that I think.
We need to be able to define finer grained contexts than just
"task" and/or "CPU".

And reusing events themselves would be a nice interface.

For example create 3 events:

A = irq entry tracepoint
B = irq exit tracepoint
C = cpu-cycles

And say: I want to count cpu-cycles when event A fires and stop counting
when B fires.

With that you can count cpu cycles on irqs.

You could use any event you want to define your contexts: lock, functions, etc...

And even uprobes to define areas in userspace to profile. Would that solve
the initial problem in the thread? Like hook on function library entry/exit?

I talked about that to Jiri Olsa several times. May be he would be interested
in implementing this. I posted some patchets one year ago but got sidetracked.

This can give you an idea from where we can start: https://lkml.org/lkml/2011/3/14/346

Jiri, would you be interested in working on this?

Re: [RFD] perf: events defined contexts (was Re: perf: prctl(PR_TASK_PERF_EVENTS_DISABLE) has no effect)

[Jiri Olsa]

On Fri, Jul 27, 2012 at 01:56:36PM +0200, Frederic Weisbecker wrote:

SNIP

> 
> The problem is more general than that I think.
> We need to be able to define finer grained contexts than just
> "task" and/or "CPU".
> 
> And reusing events themselves would be a nice interface.
> 
> For example create 3 events:
> 
> A = irq entry tracepoint
> B = irq exit tracepoint
> C = cpu-cycles
> 
> And say: I want to count cpu-cycles when event A fires and stop counting
> when B fires.
> 
> With that you can count cpu cycles on irqs.
> 
> You could use any event you want to define your contexts: lock, functions, etc...
> 
> And even uprobes to define areas in userspace to profile. Would that solve
> the initial problem in the thread? Like hook on function library entry/exit?
> 
> I talked about that to Jiri Olsa several times. May be he would be interested
> in implementing this. I posted some patchets one year ago but got sidetracked.
> 
> This can give you an idea from where we can start: https://lkml.org/lkml/2011/3/14/346
> 
> Jiri, would you be interested in working on this?

yep, I remember also talking about A/B being functions entry/exit

I think that was also the initial push for having ftrace perf events
support, which already got in.

I'll check ;)

jirka

Re: perf: prctl(PR_TASK_PERF_EVENTS_DISABLE) has no effect

[Andi Kleen]

Ingo Molnar <mingo@kernel.org> writes:

> * Peter Zijlstra <a.p.zijlstra@chello.nl> wrote:
>
>> > It also allows system-wide profiling after you've modified a 
>> > library to self-profile, while your suggestion does not 
>> > allow that.
>> 
>> But its no long self-profiling when some other process is 
>> involved. And system wide is definitely not self.
>
> If I'm a library developer and want to self-profile my 
> modifications, then it's entirely appropriate to do a 
> system-wide profile to see the system-wide use of this library!
>
> System-wide and self-profiling is not exclusive. Think of it as 
> a narrow, precise filter applied to a given area of 
> functionality only.
>
> AFAICS the alternative, under your method, would be to recompile 
> every single app in the system - that's cumbersome beyond 
> imagination, I wouldn't even call it a solution, let alone a 
> quality implementation.
>
>> > > It really isn't that hard to make userspace do what is 
>> > > needed, it just takes a bit of work.
>> > 
>> > Even if your suggested solution was available (it isn't), my 
>> > suggested approach is easier to use and covers more 
>> > usecases.
>> > 
>> > User-space expecting the kernel to provide usable and 
>> > minimal interfaces is not 'being silly'. It's the 
>> > fundamental task of a kernel to provide them.
>> 
>> Bloating the interface for something that is already well 
>> possible is.
>
> There's no 'bloat' worth speaking off: a single bit out of an 
> already allocated bitmap, plus a single check in an already 
> existing loop, plus a single new command-line flag to the 
> tooling side and minimal glue.

I tried it some time ago because I had a user who was very interested in
getting rid of initialization overhead in measurements.
I have a little library too to self instrument, but it's 
still a lot more work than adding a ioctl or two.

The patch is actually broken (causes occasional crashes)
and does not implement the opt in bit (but that would be easy 
to add). And it's against a quite old kernel.

Also had to do the disabling only after exec to avoid some issues.

But it worked well enough for some tests.

There are likely better ways to implement this
(or least ways that don't crash :-)
Frederic's generic contexts also sound nice.

The nice thing is that combined with some LD_PRELOAD 
tricks you can even use it with unmodified binaries.

(not a submission or anything, just FYI)

-Andi

http://firstfloor.org/~andi/perf-disabled-1


-- 
ak@linux.intel.com -- Speaking for myself only

Re: perf: prctl(PR_TASK_PERF_EVENTS_DISABLE) has no effect

[Peter Zijlstra]

On Fri, 2012-07-27 at 13:53 +0200, Ingo Molnar wrote:
> > Bloating the interface for something that is already well 
> > possible is.
> 
> There's no 'bloat' worth speaking off: a single bit out of an 
> already allocated bitmap, plus a single check in an already 
> existing loop,

Uhm, no. The existing prctl() loop is over the fd's the task owns, you
want a loop over the fd's that monitor you. This needs new prctl()s at
the very least.

Re: perf: prctl(PR_TASK_PERF_EVENTS_DISABLE) has no effect

[Ingo Molnar]

* Peter Zijlstra <a.p.zijlstra@chello.nl> wrote:

> On Fri, 2012-07-27 at 13:53 +0200, Ingo Molnar wrote:
> > > Bloating the interface for something that is already well 
> > > possible is.
> > 
> > There's no 'bloat' worth speaking off: a single bit out of 
> > an already allocated bitmap, plus a single check in an 
> > already existing loop,
> 
> Uhm, no. The existing prctl() loop is over the fd's the task 
> owns, you want a loop over the fd's that monitor you. This 
> needs new prctl()s at the very least.

We could add a new prctl if you think, but I thought to not 
complicate it and offer it as a simple extension of the 
semantics to loop over active events. No existing binary's 
behavior will change.

Thanks,

	Ingo

Re: perf: prctl(PR_TASK_PERF_EVENTS_DISABLE) has no effect

[Peter Zijlstra]

On Tue, 2012-07-31 at 09:16 +0200, Ingo Molnar wrote:
> * Peter Zijlstra <a.p.zijlstra@chello.nl> wrote:
> 
> > On Fri, 2012-07-27 at 13:53 +0200, Ingo Molnar wrote:
> > > > Bloating the interface for something that is already well 
> > > > possible is.
> > > 
> > > There's no 'bloat' worth speaking off: a single bit out of 
> > > an already allocated bitmap, plus a single check in an 
> > > already existing loop,
> > 
> > Uhm, no. The existing prctl() loop is over the fd's the task 
> > owns, you want a loop over the fd's that monitor you. This 
> > needs new prctl()s at the very least.
> 
> We could add a new prctl if you think, but I thought to not 
> complicate it and offer it as a simple extension of the 
> semantics to loop over active events. No existing binary's 
> behavior will change.

It muddles up the semantics of the existing prctl()s though.

Ideally we'd simply remove the current ones though, I don't think
anybody actually uses them and as an owner you actually have all the fds
to call ioctl() on.

This would let us get rid of the entire event->owner, event->owner_entry
and task->perf_event_list mess. See perf_event_exit_task() and
perf_release() for why I'd love that crap to go away.

Re: [RFD] perf: events defined contexts (was Re: perf: prctl(PR_TASK_PERF_EVENTS_DISABLE) has no effect)

[Namhyung Kim]

Hi,

On Fri, 27 Jul 2012 14:45:38 +0200, Jiri Olsa wrote:
> On Fri, Jul 27, 2012 at 01:56:36PM +0200, Frederic Weisbecker wrote:
>> The problem is more general than that I think.
>> We need to be able to define finer grained contexts than just
>> "task" and/or "CPU".
>> 
>> And reusing events themselves would be a nice interface.
>> 
>> For example create 3 events:
>> 
>> A = irq entry tracepoint
>> B = irq exit tracepoint
>> C = cpu-cycles
>> 
>> And say: I want to count cpu-cycles when event A fires and stop counting
>> when B fires.
>> 
>> With that you can count cpu cycles on irqs.
>> 
>> You could use any event you want to define your contexts: lock, functions, etc...
>> 
>> And even uprobes to define areas in userspace to profile. Would that solve
>> the initial problem in the thread? Like hook on function library entry/exit?
>> 
>> I talked about that to Jiri Olsa several times. May be he would be interested
>> in implementing this. I posted some patchets one year ago but got sidetracked.
>> 
>> This can give you an idea from where we can start: https://lkml.org/lkml/2011/3/14/346
>> 
>> Jiri, would you be interested in working on this?
>
> yep, I remember also talking about A/B being functions entry/exit
>
> I think that was also the initial push for having ftrace perf events
> support, which already got in.
>
> I'll check ;)
>

Very cool. :) I'm also interested in the feature although my background
knowledge is not firm enough yet. Please let me know if there's anything
I can help you on this.

Thanks,
Namhyung