RFC: Performance Monitor Counters device

RFC: Performance Monitor Counters device

[Segher Boessenkool]

Hi people,

I'm currently implementing a Linux driver to use the PowerPC
performance monitor counters (PMC's).  I thought I'd ask here
about some design/implementation issues, before I do a lot
of stupid/timewasting things ;)

The goal of this driver is threefold:

1) provide an easy interface for programs to be able to monitor
   themselves -> need to change your programs code, very
   fine-grained profiling.
2) provide a little more capable interface to use with something
   like gmon.
3) and you can do all that on the kernel, too.

For 1), the program-to-be-monitored just needs to be able to
select a set of PMC's, and read the counters.  Pretty easy.

For 2), you need the same as for 1), plus have the kernel
generate a signal on every Nth occurence of some event.  The
PowerPC generates an exception for that, so this is not all
that hard either.  Maybe the kernel can communicate which PMC
set off the signal; if not, it's trivial for the userland to
figure that out from the current counters.

For 3), it's just the same as above, except the niggly details
I left out (that are needed there, too): you need to select
whether the counters count event in supervisor and/or in
problem state, and/or only on marked tasks.


Now the questions ;)

1) What's the best interface for this kind of thing?  A char
   device?  With ioctl()'s?  a sysctl?  something in /proc?
   I'm not interested in ease of implementation (I'll have to
   hack some on gprof too, for this -- so I'm not afraid of
   the kernel ;) ), but in what's philosophically/technically/
   procatically the best interface.
2) Do we want to be able to profile several tasks (independently
   from each other) at once?  This will require some bookkeeping
   and task switch overhead, and is probably not necessary in
   practice; also, it will degrade the quality of the results some.
3) [I'm ashamed I have to ask this...]  Is there a good tutorial
   to kernel locks somewhere?
4) Security: I want to generate most of the settings in userland,
   for maximum ease of use and ease of implementation; but that
   brings up some security issues.  Only allowing root to
   profile code isn't ideal, either.  So:
   a) Don't automagically load the module; if root loads it, let's
      hope he knows what he's doing;
   b) Have the pmc device be accessible only to a 'trusted' group;
   c) A setuid driver program to start profiling;
   d) Something much more clever?


Thanks to anyone who read this far,

Segher


** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: RFC: Performance Monitor Counters device

[Benjamin Herrenschmidt]

>For 2), you need the same as for 1), plus have the kernel
>generate a signal on every Nth occurence of some event.  The
>PowerPC generates an exception for that, so this is not all
>that hard either.  Maybe the kernel can communicate which PMC
>set off the signal; if not, it's trivial for the userland to
>figure that out from the current counters.

Hrm... you should check the various CPU erratas, I think
the perf. counter exception suffers from the same bugs
the thermal exceptions has regarding clashing with DEC
and thus losing exception context information on some
G4s. If this is the case, that would basically prevent
using it.

Ben.


** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: RFC: Performance Monitor Counters device

[Segher Boessenkool]

Benjamin Herrenschmidt wrote:
> Hrm... you should check the various CPU erratas, I think
> the perf. counter exception suffers from the same bugs
> the thermal exceptions has regarding clashing with DEC
> and thus losing exception context information on some
> G4s. If this is the case, that would basically prevent
> using it.

Ooh, that sounds bad!  Thanks for the hint.


Segher

** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: RFC: Performance Monitor Counters device

[Benjamin Herrenschmidt]

>Benjamin Herrenschmidt wrote:
>> Hrm... you should check the various CPU erratas, I think
>> the perf. counter exception suffers from the same bugs
>> the thermal exceptions has regarding clashing with DEC
>> and thus losing exception context information on some
>> G4s. If this is the case, that would basically prevent
>> using it.
>
>Ooh, that sounds bad!  Thanks for the hint.

Yah, and we have to thank MOL for helping us find out
what was going on with the TAU exception easily ;)

The way MOL plays with the DEC made the clash happen
more often, and since it then happened within MOL emulation
context, the MOL kernel module could figure out something
wrong happened, kill the emulator, and stay alive while
printing nice debug informations ;)
If that happened during normal kernel operations, I suspect
we would have just died hard.

Ben.


** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: RFC: Performance Monitor Counters device

[Segher Boessenkool]

Benjamin Herrenschmidt wrote:
>
> >Benjamin Herrenschmidt wrote:
> >> Hrm... you should check the various CPU erratas, I think
> >> the perf. counter exception suffers from the same bugs
> >> the thermal exceptions has regarding clashing with DEC
> >> and thus losing exception context information on some
> >> G4s. If this is the case, that would basically prevent
> >> using it.
> >
> >Ooh, that sounds bad!  Thanks for the hint.
>
> Yah, and we have to thank MOL for helping us find out
> what was going on with the TAU exception easily ;)
>
> The way MOL plays with the DEC made the clash happen
> more often, and since it then happened within MOL emulation
> context, the MOL kernel module could figure out something
> wrong happened, kill the emulator, and stay alive while
> printing nice debug informations ;)
> If that happened during normal kernel operations, I suspect
> we would have just died hard.
>

Okay, here's the data I found on that errata:

It does not exist on 7450 etc.
It exists on 7410 before version 1.3 .
I have no data on 7400.
It does not exist on 750 etc.

[As my only G4 is a 7410 version 1.3, I won't be affected by this.  Hurray.]

The problem itself: if two of thermal assist, decrementer, performance
monitor interrupts happen within 1 cycle of each other, evil things
happen with SRR0 and SRR1, so that the return address becomes
unrecoverable.

Suggested solution:

We can forget about thermal assist, as the TAU on all 74xx is broken
and unsupported.  (Says those same errata sheets).
If necessary, it's possible to disable the decrementer interrupt and have
the performance monitor perform its function.

But I won't run into this, so I'm happy for now :)


Segher


** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: RFC: Performance Monitor Counters device

[samuel]

On Sat, Sep 14, 2002 at 01:28:26PM +0200, Segher Boessenkool wrote:
> Okay, here's the data I found on that errata:
>
> It does not exist on 7450 etc.
> It exists on 7410 before version 1.3 .
> I have no data on 7400.
> It does not exist on 750 etc.

Actually, I have experienced this bug on one of my 750-based machines
(possibly, only one of the revs were affected). If I were you, I would be
extremely careful using the TAU/performance interrupt.

/Samuel

** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: RFC: Performance Monitor Counters device

[Benjamin Herrenschmidt]

>It does not exist on 7450 etc.
>It exists on 7410 before version 1.3 .
>I have no data on 7400.
>It does not exist on 750 etc.
>
>[As my only G4 is a 7410 version 1.3, I won't be affected by this.  Hurray.]

I had it happening on 7400.

>The problem itself: if two of thermal assist, decrementer, performance
>monitor interrupts happen within 1 cycle of each other, evil things
>happen with SRR0 and SRR1, so that the return address becomes
>unrecoverable.
>
>Suggested solution:
>
>We can forget about thermal assist, as the TAU on all 74xx is broken
>and unsupported.  (Says those same errata sheets).
>If necessary, it's possible to disable the decrementer interrupt and have
>the performance monitor perform its function.
>
>But I won't run into this, so I'm happy for now :)

We can't disable the DEC that easily as it's the primary timer
source of the kernel ;)

Ben.


** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: RFC: Performance Monitor Counters device

[Segher Boessenkool]

> We can't disable the DEC that easily as it's the primary timer
> source of the kernel ;)

The performance timers can fully take its place if necessary, I think :)


Segher


** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: RFC: Performance Monitor Counters device

[Paul Mackerras]

Segher Boessenkool writes:

> I'm currently implementing a Linux driver to use the PowerPC
> performance monitor counters (PMC's).  I thought I'd ask here

Cool. :)

> For 1), the program-to-be-monitored just needs to be able to
> select a set of PMC's, and read the counters.  Pretty easy.

I'm curious - how are you going to cope with the fact that different
PPC implementations have different numbers of PMCs and different
sets of events (and numbering of the events) that the PMCs will count?

Paul.

** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: RFC: Performance Monitor Counters device

[Segher Boessenkool]

Paul Mackerras wrote:
> I'm curious - how are you going to cope with the fact that different
> PPC implementations have different numbers of PMCs and different
> sets of events (and numbering of the events) that the PMCs will count?

The kernel module doesn't care, except for not allowing illegal values;
userland doesn't care all that much either, really, as a number is just
a number :)  It's mostly the user that cares.

Ah well, an example might clarify:


ppcprofile --pmc1=6 --pmc2=6 ./hello_world

ppcprofile will look for existence of /proc/sys/debug/pmc/pmc[12], and
do some sysctls on it.  i.e., the kernel module will only make entries
for available hardware.

Note that the kernel module doesn't need to check if a certain event
number is defined; it only needs to check if it fits into the number
of bits allocated to it.


Segher


** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: RFC: Performance Monitor Counters device

[Anton Blanchard]

> I'm currently implementing a Linux driver to use the PowerPC
> performance monitor counters (PMC's).  I thought I'd ask here
> about some design/implementation issues, before I do a lot
> of stupid/timewasting things ;)

It would be great if we could plug into something like oprofile.
Linus was even interested in merging some sort of simple profiling
infrastructure.

We have a similar issue on ppc64, if we could share at least part
of it between ppc32 and ppc64 that would be nice.

Anton

** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: RFC: Performance Monitor Counters device

[Segher Boessenkool]

Anton Blanchard wrote:
> It would be great if we could plug into something like oprofile.
> Linus was even interested in merging some sort of simple profiling
> infrastructure.
>
> We have a similar issue on ppc64, if we could share at least part
> of it between ppc32 and ppc64 that would be nice.

Some parts of the interface, like the raw/cooked counter reading and
the simplest forms of profiling interrupts, can probably be shared
across all architectures; ppc32 and ppc64 can probably share about
everything.


Segher


** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: RFC: Performance Monitor Counters device

[David Engebretsen]

Segher Boessenkool wrote:
>
> Hi people,
>
> I'm currently implementing a Linux driver to use the PowerPC
> performance monitor counters (PMC's).  I thought I'd ask here

> The goal of this driver is threefold:
>
> 1) provide an easy interface for programs to be able to monitor
>    themselves -> need to change your programs code, very
>    fine-grained profiling.
> 2) provide a little more capable interface to use with something
>    like gmon.
> 3) and you can do all that on the kernel, too.
>

In ppc64, we have implemented a generic interface via a syscall (which is akin
to what IA64 has done).  Though this syscall interface, a user level program can
confure the counters to collect pretty much any desired data.  The kernel puts
the data in the PMCs, collects a trace of SIAR & SDAR values, maintians a
cumulative count in 64b data structures, etc.  The traces are nice as they
include data from the entire kernel, inlcluding area where runing hard
disabled.  I think most of this is in our trees (see perfmon.[ch], although
there are some recent additions I am still cleaning up which should be ready in
a week or so.

The present code only allows the kernel to be profiled and for a mode which
collects slices of data by rotating through the counters during a run
(collecting CPI, cache miss rates, branch mispredicts, etc).  We have not yet
added any user space collection, other than a quick proof of concept hack.


> 1) What's the best interface for this kind of thing?  A char
>    device?  With ioctl()'s?  a sysctl?  something in /proc?
>    I'm not interested in ease of implementation (I'll have to
>    hack some on gprof too, for this -- so I'm not afraid of
>    the kernel ;) ), but in what's philosophically/technically/
>    procatically the best interface.

The interface question is one I have been concerned with too.  We chose a
syscall as it was quickest to implement & is efficient.  A driver with ioctls or
/proc are also good candidates for discussion.

> 4) Security: I want to generate most of the settings in userland,
>    for maximum ease of use and ease of implementation; but that
>    brings up some security issues.  Only allowing root to
>    profile code isn't ideal, either.  So:
>    a) Don't automagically load the module; if root loads it, let's
>       hope he knows what he's doing;
>    b) Have the pmc device be accessible only to a 'trusted' group;
>    c) A setuid driver program to start profiling;
>    d) Something much more clever?

What we have implemented to date does raise security issues as by exposing this
hardware to a user it will allow them to severely affect the system.  Only idea
we have had thus far is assume root knows what they are doing :)

Anton had mentioned in a followup about integration into oprofile or other
tools.  This would be nice to consider as well.  Presently the data generated by
our tools is in one of two very simple formats: one is just a kernel profile
(just like the standard kernel profiler), the other is a trace containing pairs
of SIAR and SDAR registers.

Dave.

** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: RFC: Performance Monitor Counters device

[Segher Boessenkool]

David Engebretsen wrote:
> In ppc64, we have implemented a generic interface via a syscall (which is akin
> to what IA64 has done).

I'd rather not introduce a syscall for this -- but maybe it'll prove to
be beneficial to do so.

> Though this syscall interface, a user level program can
> confure the counters to collect pretty much any desired data.  The kernel puts
> the data in the PMCs, collects a trace of SIAR & SDAR values,

No SDAR on any ppc32 machine I'm aware of :(  (Maybe some old 6xx has it, though).
Btw., the 750 manual and binutils use the name SIA instead of SIAR.  Sigh.

> maintians a
> cumulative count in 64b data structures, etc.  The traces are nice as they

The 64-bit idea is very very nice :)  That'll allow a program to just set
the counters at the start of eexecution, and only read them again when it's
finished.  Can't get much simpler ;)

> include data from the entire kernel, inlcluding area where runing hard
> disabled.  I think most of this is in our trees (see perfmon.[ch], although
> there are some recent additions I am still cleaning up which should be ready in
> a week or so.

I don't want to have the profiling buffers in kernel space, as they can very well
be bigger than physical memory...  On the other hand, it might be needed for
performance (or to avoid races) if doing very fine-grained profiling.  I think
I'll just sneak out of this by not allowing so fine-grained stuff ;)

> The present code only allows the kernel to be profiled and for a mode which
> collects slices of data by rotating through the counters during a run
> (collecting CPI, cache miss rates, branch mispredicts, etc).  We have not yet
> added any user space collection, other than a quick proof of concept hack.

I'm doing user space profiling first; with that done, adding kernel profiling
will be pretty much trivial (just setting a few different bits.  Oh, and user
space will have to find out the text segment addresses to profile in a different
way, obviously).

>
> > 1) What's the best interface for this kind of thing?  A char
> >    device?  With ioctl()'s?  a sysctl?  something in /proc?
> >    I'm not interested in ease of implementation (I'll have to
> >    hack some on gprof too, for this -- so I'm not afraid of
> >    the kernel ;) ), but in what's philosophically/technically/
> >    procatically the best interface.
>
> The interface question is one I have been concerned with too.  We chose a
> syscall as it was quickest to implement & is efficient.  A driver with ioctls or
> /proc are also good candidates for discussion.

I think I'll do a sysctl interface first.  It would be great if ppc32, ppc64, ia64
etc. can use the same interface, but this might very well not be practical.
Sharing some infrastructure will probably be perfectly well possible, though.
For now, I'll just get it working first...

>
> > 4) Security: I want to generate most of the settings in userland,
> >    for maximum ease of use and ease of implementation; but that
> >    brings up some security issues.  Only allowing root to
> >    profile code isn't ideal, either.  So:
> >    a) Don't automagically load the module; if root loads it, let's
> >       hope he knows what he's doing;
> >    b) Have the pmc device be accessible only to a 'trusted' group;
> >    c) A setuid driver program to start profiling;
> >    d) Something much more clever?
>
> What we have implemented to date does raise security issues as by exposing this
> hardware to a user it will allow them to severely affect the system.  Only idea
> we have had thus far is assume root knows what they are doing :)

On ppc32, userland can *always* read all pmc's.  You might consider that a
security risk already (consider timing attacks on some crypto algo, for
example).
The worse issue imho is that it's pretty easy to lock up/severely slow down
a machine by setting some idiotic values to the exception stuff.

>
> Anton had mentioned in a followup about integration into oprofile or other
> tools.  This would be nice to consider as well.

I'm generating a gmon.out :)
gprof will need some changes to properly cope, though.  And maybe the gmon
format will need a revision to have it store the names of the shared libraries
profiled...  I think the ia64 people already discussed something like this?

> Presently the data generated by
> our tools is in one of two very simple formats: one is just a kernel profile
> (just like the standard kernel profiler), the other is a trace containing pairs
> of SIAR and SDAR registers.

How do you use that second trace?  Just curious, I'm always happy to learn some
new techniques ;)


Cheers,

Segher


** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: RFC: Performance Monitor Counters device

[Dave Engebretsen]

Segher Boessenkool wrote:
>
> David Engebretsen wrote:

> The 64-bit idea is very very nice :)  That'll allow a program to just set
> the counters at the start of eexecution, and only read them again when it's
> finished.  Can't get much simpler ;)

Of course this depends on what you are trying to achive.  For profiling
or when timeslicing the counters, this is not needed generally. If you
want to just collect a fixed set of values over a long run, it is
usefull.  This however is not the mode I have generally been finding
useful.

> I don't want to have the profiling buffers in kernel space, as they can very well
> be bigger than physical memory...  On the other hand, it might be needed for
> performance (or to avoid races) if doing very fine-grained profiling.  I think
> I'll just sneak out of this by not allowing so fine-grained stuff ;)

Not sure I follow here - allowing a profiling buffer which is even a
fraction of total system memory will significantly perturb the data.
What we have been planning here (but have not implemented) is a
mechanism to freeze the generation of trace events while the user space
application drains & processes the data into a more compact form, such
as a profile.

> > What we have implemented to date does raise security issues as by exposing this
> > hardware to a user it will allow them to severely affect the system.  Only idea
> > we have had thus far is assume root knows what they are doing :)
>
> On ppc32, userland can *always* read all pmc's.  You might consider that a
> security risk already (consider timing attacks on some crypto algo, for
> example).
> The worse issue imho is that it's pretty easy to lock up/severely slow down
> a machine by setting some idiotic values to the exception stuff.

Reading isn't really the problem :)  It is the interrupt rate
implications that are of concern.

> > Presently the data generated by
> > our tools is in one of two very simple formats: one is just a kernel profile
> > (just like the standard kernel profiler), the other is a trace containing pairs
> > of SIAR and SDAR registers.
>
> How do you use that second trace?  Just curious, I'm always happy to learn some
> new techniques ;)

As the input for a tool which generates a profile based on the trace
points.  For example, if you are collecting L2 miss addresses on a 32b
application and want a resolution of 1 word, the possible input values
would require a rather large buffer to use a simple profile scheme.  A
trace is a more compact representation of the data and allows fine grain
resolution without requiring a huge buffer.

Dave.

** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: RFC: Performance Monitor Counters device

[Segher Boessenkool]

Dave Engebretsen wrote:
>
> Segher Boessenkool wrote:
> >
> > David Engebretsen wrote:
>
> > The 64-bit idea is very very nice :)  That'll allow a program to just set
> > the counters at the start of eexecution, and only read them again when it's
> > finished.  Can't get much simpler ;)
>
> Of course this depends on what you are trying to achive.  For profiling
> or when timeslicing the counters, this is not needed generally. If you
> want to just collect a fixed set of values over a long run, it is
> usefull.  This however is not the mode I have generally been finding
> useful.

Well, it's an easy way to see if a change in a program actually did help ;)

Note that the 32-bit (31, actually) counters can already overflow in about
one or two seconds.

> > I don't want to have the profiling buffers in kernel space, as they can very well
> > be bigger than physical memory...  On the other hand, it might be needed for
> > performance (or to avoid races) if doing very fine-grained profiling.  I think
> > I'll just sneak out of this by not allowing so fine-grained stuff ;)
>
> Not sure I follow here - allowing a profiling buffer which is even a
> fraction of total system memory will significantly perturb the data.

Not necessarily, like when counting number of insns completed, or something
like that.  But for most things, yes of course cache effects will be huge
with a profiling buffer >= 10% of L1 cache or so.  Even more so with bigger
buffers.


Cheers,

Segher


** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: RFC: Performance Monitor Counters device

[Ethan Benson]

On Thu, Sep 12, 2002 at 10:03:02PM +0200, Segher Boessenkool wrote:
>
> Now the questions ;)
>
> 1) What's the best interface for this kind of thing?  A char
>    device?  With ioctl()'s?  a sysctl?  something in /proc?
>    I'm not interested in ease of implementation (I'll have to
>    hack some on gprof too, for this -- so I'm not afraid of
>    the kernel ;) ), but in what's philosophically/technically/
>    procatically the best interface.


just my humble opinions but:

ioctl() is the sewer of unix, its satanic don't use it.

/proc is a linux dumping ground for endless random cruft that belongs
somewhere else.  stop the pollution.

sysctl doesn't seem applicable since its more to flip flags and
switchs in the kernel at runtime. (it sounds like your developing a
more involved interface then that).

that leaves a /dev node which i think is probably most appropriate
(and gives you simple access control via standard permissions for
free)

> 4) Security: I want to generate most of the settings in userland,
>    for maximum ease of use and ease of implementation; but that
>    brings up some security issues.  Only allowing root to
>    profile code isn't ideal, either.  So:
>    a) Don't automagically load the module; if root loads it, let's
>       hope he knows what he's doing;

just loading the module shouldn't remove all security IMO.

>    b) Have the pmc device be accessible only to a 'trusted' group;

this i think is the best way, just implement the interface solely
though a /dev device node, the admin can set the mode to 660 and group
to whatever and thus control access through standard unix permissions.

>    c) A setuid driver program to start profiling;

ugh, don't do setuid, its a massive ammount of work to try and make it
safe.

--
Ethan Benson
http://www.alaska.net/~erbenson/

** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: RFC: Performance Monitor Counters device

[Rob Latham]

On Thu, Sep 12, 2002 at 10:03:02PM +0200, Segher Boessenkool wrote:

> Now the questions ;)
>
> 1) What's the best interface for this kind of thing?  A char
>    device?  With ioctl()'s?  a sysctl?  something in /proc?
>    I'm not interested in ease of implementation (I'll have to
>    hack some on gprof too, for this -- so I'm not afraid of
>    the kernel ;) ), but in what's philosophically/technically/
>    procatically the best interface.
> 2) Do we want to be able to profile several tasks (independently
>    from each other) at once?  This will require some bookkeeping
>    and task switch overhead, and is probably not necessary in
>    practice; also, it will degrade the quality of the results some.
> 3) [I'm ashamed I have to ask this...]  Is there a good tutorial
>    to kernel locks somewhere?
> 4) Security: I want to generate most of the settings in userland,
>    for maximum ease of use and ease of implementation; but that
>    brings up some security issues.  Only allowing root to
>    profile code isn't ideal, either.  So:
>    a) Don't automagically load the module; if root loads it, let's
>       hope he knows what he's doing;
>    b) Have the pmc device be accessible only to a 'trusted' group;
>    c) A setuid driver program to start profiling;
>    d) Something much more clever?

it'd be really cool if the performance counter stuff looked like
perfctr  (http://user.it.uu.se/~mikpe/linux/perfctr/). it uses a /dev/
entry.

The nice thing about making it perfctr-like is that it can tie into
oprofile (as mentioned in another reply) and also tie into PAPI
(http://icl.cs.utk.edu/projects/papi/).

PAPI has support for all kinds of performance counters on all sorts of
platforms.  PAPI can be ported to whatever interface you end up
deciding upon: it's just that if it's perfctr-like, the porting
becomes a lot eaiser :>

just my non-hacker two cents.
==rob

--
Rob Latham                                        Woodridge, IL USA

** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/