perf: prctl(PR_TASK_PERF_EVENTS_DISABLE) has no effect

perf: prctl(PR_TASK_PERF_EVENTS_DISABLE) has no effect

[Iegorov Oleg]

as there was no proposed solution that helped me in response to the same 
post by Andrew Steets, I would like to know if it is possible to 
disable/enable perf event counters from the source code?

calling prctl(PR_TASK_PERF_EVENTS_DISABLE) has no effect, nor does 
compiling with -fno-omit-frame-pointer option.

It would be extremely useful to disable perf event counters for some 
parts of code and re-enable them for other parts of code, like:

prctl(PR_TASK_PERF_EVENTS_DISABLE);
// not important for performance analysis code
prctl(PR_TASK_PERF_EVENTS_ENABLE);
// code that needs to be analysed

and then, run perf:

$ perf record ./program
$ perf report

Can anyone tell me how can I enable such functionality?
Thank you

--Oleg

Re: perf: prctl(PR_TASK_PERF_EVENTS_DISABLE) has no effect

[Ingo Molnar]

* Iegorov Oleg <oleg_iegorov@mentor.com> wrote:

> as there was no proposed solution that helped me in response to the
> same post by Andrew Steets, I would like to know if it is possible
> to disable/enable perf event counters from the source code?
> 
> calling prctl(PR_TASK_PERF_EVENTS_DISABLE) has no effect, nor does
> compiling with -fno-omit-frame-pointer option.
> 
> It would be extremely useful to disable perf event counters for some
> parts of code and re-enable them for other parts of code, like:
> 
> prctl(PR_TASK_PERF_EVENTS_DISABLE);
> // not important for performance analysis code
> prctl(PR_TASK_PERF_EVENTS_ENABLE);
> // code that needs to be analysed
> 
> and then, run perf:
> 
> $ perf record ./program
> $ perf report
> 
> Can anyone tell me how can I enable such functionality?

So, the kernel bits to do this in a pretty quirky way are there, 
see:

    https://lkml.org/lkml/2012/1/30/99

but the librarization bits are definitely non-obvious to do and 
it's no surprise that it has not been done yet.

Regular 'perf record' in itself is not self-profiling - it's 
another task profiling you, so we cannot blanket allow 
PR_TASK_PERF_EVENTS_DISABLE to disable profiling.

What we might want to do instead on the kernel side to offer the 
functionality you are asking for is to enable 'public/weak' 
events be created by the profiler on an opt-in basis, which can 
be turned off by child tasks as well via 
PR_TASK_PERF_EVENTS_DISABLE.

On the profiling workflow side it would work in a very simple 
way, like this:

    perf record --self-profiling ./my-app

In your app you stick in appropriately placed 
PR_TASK_PERF_EVENTS_DISABLE/ENABLE calls.

On the technical side perf record creates events with that 
struct perf_event_attr::self_profiling flag set to 1. (the flag 
is disabled by default)

The PR_TASK_PERF_EVENTS_DISABLE code in the modified kernel then 
iterates through all events and disables those that have this 
flag set, not just the ones owned by this task.

Maybe someone on Cc: would be interested in implementing this 
new perf events feature?

Thanks,

	Ingo

Re: perf: prctl(PR_TASK_PERF_EVENTS_DISABLE) has no effect

[Peter Zijlstra]

On Fri, 2012-07-27 at 09:26 +0200, Ingo Molnar wrote:
> 
> Maybe someone on Cc: would be interested in implementing this 
> new perf events feature? 

Why would we go build new kernel interfaces because userspace is silly?

It really isn't that hard to make userspace do what is needed, it just
takes a bit of work.

Re: perf: prctl(PR_TASK_PERF_EVENTS_DISABLE) has no effect

[Ingo Molnar]

* Peter Zijlstra <a.p.zijlstra@chello.nl> wrote:

> On Fri, 2012-07-27 at 09:26 +0200, Ingo Molnar wrote:
> > 
> > Maybe someone on Cc: would be interested in implementing this 
> > new perf events feature? 
> 
> Why would we go build new kernel interfaces because userspace 
> is silly?

Because it's (much) easier to use the existing perf tools almost 
as-is instead of librarizing your own.

It would also allow other usecases, like self-profiling a 
library and then profiling it within the context of a larger app 
that you don't want to rebuild and which dynamically links this 
library.

It also allows system-wide profiling after you've modified a 
library to self-profile, while your suggestion does not allow 
that.

> It really isn't that hard to make userspace do what is needed, 
> it just takes a bit of work.

Even if your suggested solution was available (it isn't), my 
suggested approach is easier to use and covers more usecases.

User-space expecting the kernel to provide usable and minimal 
interfaces is not 'being silly'. It's the fundamental task of a 
kernel to provide them.

Thanks,

	Ingo

Re: perf: prctl(PR_TASK_PERF_EVENTS_DISABLE) has no effect

[Peter Zijlstra]

On Fri, 2012-07-27 at 10:18 +0200, Ingo Molnar wrote:
> * Peter Zijlstra <a.p.zijlstra@chello.nl> wrote:
> 
> > On Fri, 2012-07-27 at 09:26 +0200, Ingo Molnar wrote:
> > > 
> > > Maybe someone on Cc: would be interested in implementing this 
> > > new perf events feature? 
> > 
> > Why would we go build new kernel interfaces because userspace 
> > is silly?
> 
> Because it's (much) easier to use the existing perf tools almost 
> as-is instead of librarizing your own.
> 
> It would also allow other usecases, like self-profiling a 
> library and then profiling it within the context of a larger app 
> that you don't want to rebuild and which dynamically links this 
> library.

Uhm.. why not? For the first use proper self profiling, for the second
do a regular 3rd party profile.

> It also allows system-wide profiling after you've modified a 
> library to self-profile, while your suggestion does not allow 
> that.

But its no long self-profiling when some other process is involved. And
system wide is definitely not self.

> > It really isn't that hard to make userspace do what is needed, 
> > it just takes a bit of work.
> 
> Even if your suggested solution was available (it isn't), my 
> suggested approach is easier to use and covers more usecases.
> 
> User-space expecting the kernel to provide usable and minimal 
> interfaces is not 'being silly'. It's the fundamental task of a 
> kernel to provide them.

Bloating the interface for something that is already well possible is.

I really don't see the problem, other than that people simply don't want
to do work.

Re: perf: prctl(PR_TASK_PERF_EVENTS_DISABLE) has no effect

[Iegorov Oleg]

Le 2012-07-27 10:29, Peter Zijlstra a écrit :
> On Fri, 2012-07-27 at 10:18 +0200, Ingo Molnar wrote:
>> * Peter Zijlstra<a.p.zijlstra@chello.nl>  wrote:
>>
>>> On Fri, 2012-07-27 at 09:26 +0200, Ingo Molnar wrote:
>>>> Maybe someone on Cc: would be interested in implementing this
>>>> new perf events feature?
>>> Why would we go build new kernel interfaces because userspace
>>> is silly?
>> Because it's (much) easier to use the existing perf tools almost
>> as-is instead of librarizing your own.
>>
>> It would also allow other usecases, like self-profiling a
>> library and then profiling it within the context of a larger app
>> that you don't want to rebuild and which dynamically links this
>> library.
> Uhm.. why not? For the first use proper self profiling, for the second
> do a regular 3rd party profile.
>
>> It also allows system-wide profiling after you've modified a
>> library to self-profile, while your suggestion does not allow
>> that.
> But its no long self-profiling when some other process is involved. And
> system wide is definitely not self.
>
>>> It really isn't that hard to make userspace do what is needed,
>>> it just takes a bit of work.
>> Even if your suggested solution was available (it isn't), my
>> suggested approach is easier to use and covers more usecases.
>>
>> User-space expecting the kernel to provide usable and minimal
>> interfaces is not 'being silly'. It's the fundamental task of a
>> kernel to provide them.
> Bloating the interface for something that is already well possible is.
>
> I really don't see the problem, other than that people simply don't want
> to do work.
I totally agree with Ingo. The key words here are "much easier to use".

Moreover, in my use case, it would add a lot of complexity to rebuild 
the application each time a new performance event is added/removed from 
the event set (if I understood correctly Peter's approach).

Regards,
Oleg

Re: perf: prctl(PR_TASK_PERF_EVENTS_DISABLE) has no effect

[Ingo Molnar]

* Peter Zijlstra <a.p.zijlstra@chello.nl> wrote:

> > It also allows system-wide profiling after you've modified a 
> > library to self-profile, while your suggestion does not 
> > allow that.
> 
> But its no long self-profiling when some other process is 
> involved. And system wide is definitely not self.

If I'm a library developer and want to self-profile my 
modifications, then it's entirely appropriate to do a 
system-wide profile to see the system-wide use of this library!

System-wide and self-profiling is not exclusive. Think of it as 
a narrow, precise filter applied to a given area of 
functionality only.

AFAICS the alternative, under your method, would be to recompile 
every single app in the system - that's cumbersome beyond 
imagination, I wouldn't even call it a solution, let alone a 
quality implementation.

> > > It really isn't that hard to make userspace do what is 
> > > needed, it just takes a bit of work.
> > 
> > Even if your suggested solution was available (it isn't), my 
> > suggested approach is easier to use and covers more 
> > usecases.
> > 
> > User-space expecting the kernel to provide usable and 
> > minimal interfaces is not 'being silly'. It's the 
> > fundamental task of a kernel to provide them.
> 
> Bloating the interface for something that is already well 
> possible is.

There's no 'bloat' worth speaking off: a single bit out of an 
already allocated bitmap, plus a single check in an already 
existing loop, plus a single new command-line flag to the 
tooling side and minimal glue.

There's no runtime overhead to any other perf functionality, at 
all.

Thanks,

	Ingo

Re: perf: prctl(PR_TASK_PERF_EVENTS_DISABLE) has no effect

[Andi Kleen]

Ingo Molnar <mingo@kernel.org> writes:

> * Peter Zijlstra <a.p.zijlstra@chello.nl> wrote:
>
>> > It also allows system-wide profiling after you've modified a 
>> > library to self-profile, while your suggestion does not 
>> > allow that.
>> 
>> But its no long self-profiling when some other process is 
>> involved. And system wide is definitely not self.
>
> If I'm a library developer and want to self-profile my 
> modifications, then it's entirely appropriate to do a 
> system-wide profile to see the system-wide use of this library!
>
> System-wide and self-profiling is not exclusive. Think of it as 
> a narrow, precise filter applied to a given area of 
> functionality only.
>
> AFAICS the alternative, under your method, would be to recompile 
> every single app in the system - that's cumbersome beyond 
> imagination, I wouldn't even call it a solution, let alone a 
> quality implementation.
>
>> > > It really isn't that hard to make userspace do what is 
>> > > needed, it just takes a bit of work.
>> > 
>> > Even if your suggested solution was available (it isn't), my 
>> > suggested approach is easier to use and covers more 
>> > usecases.
>> > 
>> > User-space expecting the kernel to provide usable and 
>> > minimal interfaces is not 'being silly'. It's the 
>> > fundamental task of a kernel to provide them.
>> 
>> Bloating the interface for something that is already well 
>> possible is.
>
> There's no 'bloat' worth speaking off: a single bit out of an 
> already allocated bitmap, plus a single check in an already 
> existing loop, plus a single new command-line flag to the 
> tooling side and minimal glue.

I tried it some time ago because I had a user who was very interested in
getting rid of initialization overhead in measurements.
I have a little library too to self instrument, but it's 
still a lot more work than adding a ioctl or two.

The patch is actually broken (causes occasional crashes)
and does not implement the opt in bit (but that would be easy 
to add). And it's against a quite old kernel.

Also had to do the disabling only after exec to avoid some issues.

But it worked well enough for some tests.

There are likely better ways to implement this
(or least ways that don't crash :-)
Frederic's generic contexts also sound nice.

The nice thing is that combined with some LD_PRELOAD 
tricks you can even use it with unmodified binaries.

(not a submission or anything, just FYI)

-Andi

http://firstfloor.org/~andi/perf-disabled-1


-- 
ak@linux.intel.com -- Speaking for myself only

Re: perf: prctl(PR_TASK_PERF_EVENTS_DISABLE) has no effect

[Peter Zijlstra]

On Fri, 2012-07-27 at 13:53 +0200, Ingo Molnar wrote:
> > Bloating the interface for something that is already well 
> > possible is.
> 
> There's no 'bloat' worth speaking off: a single bit out of an 
> already allocated bitmap, plus a single check in an already 
> existing loop,

Uhm, no. The existing prctl() loop is over the fd's the task owns, you
want a loop over the fd's that monitor you. This needs new prctl()s at
the very least.

Re: perf: prctl(PR_TASK_PERF_EVENTS_DISABLE) has no effect

[Ingo Molnar]

* Peter Zijlstra <a.p.zijlstra@chello.nl> wrote:

> On Fri, 2012-07-27 at 13:53 +0200, Ingo Molnar wrote:
> > > Bloating the interface for something that is already well 
> > > possible is.
> > 
> > There's no 'bloat' worth speaking off: a single bit out of 
> > an already allocated bitmap, plus a single check in an 
> > already existing loop,
> 
> Uhm, no. The existing prctl() loop is over the fd's the task 
> owns, you want a loop over the fd's that monitor you. This 
> needs new prctl()s at the very least.

We could add a new prctl if you think, but I thought to not 
complicate it and offer it as a simple extension of the 
semantics to loop over active events. No existing binary's 
behavior will change.

Thanks,

	Ingo

Re: perf: prctl(PR_TASK_PERF_EVENTS_DISABLE) has no effect

[Peter Zijlstra]

On Tue, 2012-07-31 at 09:16 +0200, Ingo Molnar wrote:
> * Peter Zijlstra <a.p.zijlstra@chello.nl> wrote:
> 
> > On Fri, 2012-07-27 at 13:53 +0200, Ingo Molnar wrote:
> > > > Bloating the interface for something that is already well 
> > > > possible is.
> > > 
> > > There's no 'bloat' worth speaking off: a single bit out of 
> > > an already allocated bitmap, plus a single check in an 
> > > already existing loop,
> > 
> > Uhm, no. The existing prctl() loop is over the fd's the task 
> > owns, you want a loop over the fd's that monitor you. This 
> > needs new prctl()s at the very least.
> 
> We could add a new prctl if you think, but I thought to not 
> complicate it and offer it as a simple extension of the 
> semantics to loop over active events. No existing binary's 
> behavior will change.

It muddles up the semantics of the existing prctl()s though.

Ideally we'd simply remove the current ones though, I don't think
anybody actually uses them and as an owner you actually have all the fds
to call ioctl() on.

This would let us get rid of the entire event->owner, event->owner_entry
and task->perf_event_list mess. See perf_event_exit_task() and
perf_release() for why I'd love that crap to go away.

[RFD] perf: events defined contexts (was Re: perf: prctl(PR_TASK_PERF_EVENTS_DISABLE) has no effect)

[Frederic Weisbecker]

On Fri, Jul 27, 2012 at 09:26:47AM +0200, Ingo Molnar wrote:
> 
> * Iegorov Oleg <oleg_iegorov@mentor.com> wrote:
> 
> > as there was no proposed solution that helped me in response to the
> > same post by Andrew Steets, I would like to know if it is possible
> > to disable/enable perf event counters from the source code?
> > 
> > calling prctl(PR_TASK_PERF_EVENTS_DISABLE) has no effect, nor does
> > compiling with -fno-omit-frame-pointer option.
> > 
> > It would be extremely useful to disable perf event counters for some
> > parts of code and re-enable them for other parts of code, like:
> > 
> > prctl(PR_TASK_PERF_EVENTS_DISABLE);
> > // not important for performance analysis code
> > prctl(PR_TASK_PERF_EVENTS_ENABLE);
> > // code that needs to be analysed
> > 
> > and then, run perf:
> > 
> > $ perf record ./program
> > $ perf report
> > 
> > Can anyone tell me how can I enable such functionality?
> 
> So, the kernel bits to do this in a pretty quirky way are there, 
> see:
> 
>     https://lkml.org/lkml/2012/1/30/99
> 
> but the librarization bits are definitely non-obvious to do and 
> it's no surprise that it has not been done yet.
> 
> Regular 'perf record' in itself is not self-profiling - it's 
> another task profiling you, so we cannot blanket allow 
> PR_TASK_PERF_EVENTS_DISABLE to disable profiling.
> 
> What we might want to do instead on the kernel side to offer the 
> functionality you are asking for is to enable 'public/weak' 
> events be created by the profiler on an opt-in basis, which can 
> be turned off by child tasks as well via 
> PR_TASK_PERF_EVENTS_DISABLE.
> 
> On the profiling workflow side it would work in a very simple 
> way, like this:
> 
>     perf record --self-profiling ./my-app
> 
> In your app you stick in appropriately placed 
> PR_TASK_PERF_EVENTS_DISABLE/ENABLE calls.
> 
> On the technical side perf record creates events with that 
> struct perf_event_attr::self_profiling flag set to 1. (the flag 
> is disabled by default)
> 
> The PR_TASK_PERF_EVENTS_DISABLE code in the modified kernel then 
> iterates through all events and disables those that have this 
> flag set, not just the ones owned by this task.
> 
> Maybe someone on Cc: would be interested in implementing this 
> new perf events feature?

The problem is more general than that I think.
We need to be able to define finer grained contexts than just
"task" and/or "CPU".

And reusing events themselves would be a nice interface.

For example create 3 events:

A = irq entry tracepoint
B = irq exit tracepoint
C = cpu-cycles

And say: I want to count cpu-cycles when event A fires and stop counting
when B fires.

With that you can count cpu cycles on irqs.

You could use any event you want to define your contexts: lock, functions, etc...

And even uprobes to define areas in userspace to profile. Would that solve
the initial problem in the thread? Like hook on function library entry/exit?

I talked about that to Jiri Olsa several times. May be he would be interested
in implementing this. I posted some patchets one year ago but got sidetracked.

This can give you an idea from where we can start: https://lkml.org/lkml/2011/3/14/346

Jiri, would you be interested in working on this?

Re: [RFD] perf: events defined contexts (was Re: perf: prctl(PR_TASK_PERF_EVENTS_DISABLE) has no effect)

[Jiri Olsa]

On Fri, Jul 27, 2012 at 01:56:36PM +0200, Frederic Weisbecker wrote:

SNIP

> 
> The problem is more general than that I think.
> We need to be able to define finer grained contexts than just
> "task" and/or "CPU".
> 
> And reusing events themselves would be a nice interface.
> 
> For example create 3 events:
> 
> A = irq entry tracepoint
> B = irq exit tracepoint
> C = cpu-cycles
> 
> And say: I want to count cpu-cycles when event A fires and stop counting
> when B fires.
> 
> With that you can count cpu cycles on irqs.
> 
> You could use any event you want to define your contexts: lock, functions, etc...
> 
> And even uprobes to define areas in userspace to profile. Would that solve
> the initial problem in the thread? Like hook on function library entry/exit?
> 
> I talked about that to Jiri Olsa several times. May be he would be interested
> in implementing this. I posted some patchets one year ago but got sidetracked.
> 
> This can give you an idea from where we can start: https://lkml.org/lkml/2011/3/14/346
> 
> Jiri, would you be interested in working on this?

yep, I remember also talking about A/B being functions entry/exit

I think that was also the initial push for having ftrace perf events
support, which already got in.

I'll check ;)

jirka

Re: [RFD] perf: events defined contexts (was Re: perf: prctl(PR_TASK_PERF_EVENTS_DISABLE) has no effect)

[Namhyung Kim]

Hi,

On Fri, 27 Jul 2012 14:45:38 +0200, Jiri Olsa wrote:
> On Fri, Jul 27, 2012 at 01:56:36PM +0200, Frederic Weisbecker wrote:
>> The problem is more general than that I think.
>> We need to be able to define finer grained contexts than just
>> "task" and/or "CPU".
>> 
>> And reusing events themselves would be a nice interface.
>> 
>> For example create 3 events:
>> 
>> A = irq entry tracepoint
>> B = irq exit tracepoint
>> C = cpu-cycles
>> 
>> And say: I want to count cpu-cycles when event A fires and stop counting
>> when B fires.
>> 
>> With that you can count cpu cycles on irqs.
>> 
>> You could use any event you want to define your contexts: lock, functions, etc...
>> 
>> And even uprobes to define areas in userspace to profile. Would that solve
>> the initial problem in the thread? Like hook on function library entry/exit?
>> 
>> I talked about that to Jiri Olsa several times. May be he would be interested
>> in implementing this. I posted some patchets one year ago but got sidetracked.
>> 
>> This can give you an idea from where we can start: https://lkml.org/lkml/2011/3/14/346
>> 
>> Jiri, would you be interested in working on this?
>
> yep, I remember also talking about A/B being functions entry/exit
>
> I think that was also the initial push for having ftrace perf events
> support, which already got in.
>
> I'll check ;)
>

Very cool. :) I'm also interested in the feature although my background
knowledge is not firm enough yet. Please let me know if there's anything
I can help you on this.

Thanks,
Namhyung