Performance counters and profiling on MIPS

Performance counters and profiling on MIPS

[Jonathan Day]

hi,

Two quick and semi-related questions for the Gurus of
the MIPS. First off, it would appear that profiling on
any of the Broadcom MIPS processors is broken. I get
the following warnings when compiling the
platform-specific irq.c file:

  CC      arch/mips/sibyte/sb1250/irq.o
arch/mips/sibyte/sb1250/irq.c: In function
'plat_irq_dispatch':
arch/mips/sibyte/sb1250/irq.c:462: warning: implicit
declaration of function 'sbprof_cpu_intr'
arch/mips/sibyte/sb1250/irq.c:467: warning: implicit
declaration of function 'sb1250_timer_interrupt'
arch/mips/sibyte/sb1250/irq.c:471: warning: implicit
declaration of function 'sb1250_mailbox_interrupt'

On linking, it's revealed why the declarations are
implicit:

arch/mips/sibyte/sb1250/built-in.o: In function
`plat_irq_dispatch':
arch/mips/sibyte/sb1250/irq.c:462: undefined reference
to `sbprof_cpu_intr'
arch/mips/sibyte/sb1250/irq.c:462: relocation
truncated to fit: R_MIPS_26 against `sbprof_cpu_intr'
arch/mips/sibyte/sb1250/irq.c:462: undefined reference
to `sbprof_cpu_intr'
arch/mips/sibyte/sb1250/irq.c:462: relocation
truncated to fit: R_MIPS_26 against `sbprof_cpu_intr'

Actually, with the code as it is in the git
repository, you will also get:

arch/mips/sibyte/sb1250/irq.c:461: undefined reference
to `exception_epc'

But this can be fixed by adding the following line to
irq.c in the asm block of includes:

#include <asm/branch.h>

The primary function, sbprof_cpu_intr(), seems to be
missing. It is called in the bcm1480 and sb1250
versions of irq.c. I looked but couldn't see anything
comparable in any other Sibyte directories, any other
MIPS architectures in general, or indeed in any other
architecture in general.

The ZBus profiling is also broken, showing some signs
of being a little stale. This one's not quite so
important to me, but it would still be very useful:

arch/mips/sibyte/sb1250/bcm1250_tbprof.c: In function
'sbprof_tb_ioctl':
arch/mips/sibyte/sb1250/bcm1250_tbprof.c:362: error:
expected expression before
'wait_queue_t'
arch/mips/sibyte/sb1250/bcm1250_tbprof.c:363: error:
'wait' undeclared (first use in this function)
arch/mips/sibyte/sb1250/bcm1250_tbprof.c:363: error:
(Each undeclared identifier is reported only once
arch/mips/sibyte/sb1250/bcm1250_tbprof.c:363: error:
for each function it appears in.)
arch/mips/sibyte/sb1250/bcm1250_tbprof.c: In function
'sbprof_tb_init':
arch/mips/sibyte/sb1250/bcm1250_tbprof.c:396: warning:
format '%lld' expects type 'long long int', but
argument 2 has type 'u_int64_t'

Ok, so my first question is: who (if anyone) is
working on the profiling code and are there any
patches - regardless of how experimental - that will
get this part of the code working?

My second question is with regards to accessing the
performance counters and timestamp counters from
userspace. On some architectures, this is as simple as
using a single macro.

In the case of the ix86 architecture (yuk!), the
timestamp counters can be read with nothing more than
an rdtsc() call, as follows:

asm volatile ("rdtsc" : "=a"(*(elg_ui4
*)&clock_value),
                "=d"(*(((elg_ui4 *)&clock_value)+1)));

What is the closest equiv. for the MIPS processors?

Jonathan


__________________________________________________
Do You Yahoo!?
Tired of spam?  Yahoo! Mail has the best spam protection around 
http://mail.yahoo.com 

Re: Performance counters and profiling on MIPS

[Ralf Baechle]

On Wed, Jun 07, 2006 at 10:22:52AM -0700, Jonathan Day wrote:

> Two quick and semi-related questions for the Gurus of
> the MIPS. First off, it would appear that profiling on
> any of the Broadcom MIPS processors is broken. I get
> the following warnings when compiling the
> platform-specific irq.c file:

This is ZBus-based profiling which also isn't supported by the standard
profiling tool oprofile.  Oprofile is using the performance counters of
the processor itself.

> My second question is with regards to accessing the
> performance counters and timestamp counters from
> userspace. On some architectures, this is as simple as
> using a single macro.
> 
> In the case of the ix86 architecture (yuk!), the
> timestamp counters can be read with nothing more than
> an rdtsc() call, as follows:
> 
> asm volatile ("rdtsc" : "=a"(*(elg_ui4
> *)&clock_value),
>                 "=d"(*(((elg_ui4 *)&clock_value)+1)));
> 
> What is the closest equiv. for the MIPS processors?

On most R4000-style processors (that includes the SB1 core of the Sibyte
chips) applications can access the cycle counter through an
mfc0 $reg, $c0_count instruction.  However mfc0 is a priviledged
instruction, so that doesn't work for code that doesn't have kernel
priviledges.

For release 2 of the MISP32 / MIPS64 architecture there is a new
instruction, rdhwr which an application - so the OS permits it - can
use to read c0_count.

Now there are two problems with that approach in your case:

 o SB1 implements release 0.95 of the MIPS64 architecture, SB1A release 1.
   Iow these cores don't have rdhwr.
 o In general on a multiprocessor system you don't have a guarantee that
   the count registers of all processors are running at the same speed or
   were set to the same value at any time.
      This is more of a general problem; in case of the BCM1250 the cores
   are actually running at the same speed and afair are synchronized by
   the reset.

  Ralf

Re: Performance counters and profiling on MIPS

[Jonathan Day]

Thank you for the valuable information.

One thing I'd like to throw open to the list: there's
one way to access the counters on the R4000-type
processors, another on the version 2 MIPS64, yet
another on the ix86, and so on.

Would it make sense to place some standardised
interface in, say, the assembly header files and hide
the implementation-specific details? In the case of
the R4000-type cores, this would need to involve some
sort of counter device in the kernel which the macro
would call to perform the priviledged instruction. (It
feels a little bit of a hack, but it's the simplest
way to provide access to resources that aren't made
public.)

What I'm thinking is that this generic interface would
then be used on all other architectures, where such
counters exist. That way, implementation-specific
stuff can be abstracted out and programs that need
access to performance counters can all be coded to a
generic interface, rather than one interface for each
version of every CPU API, which is inevitably going to
be far more prone to error.

Jonathan

--- Ralf Baechle <ralf@linux-mips.org> wrote:

> On Wed, Jun 07, 2006 at 10:22:52AM -0700, Jonathan
> Day wrote:
> 
> > Two quick and semi-related questions for the Gurus
> of
> > the MIPS. First off, it would appear that
> profiling on
> > any of the Broadcom MIPS processors is broken. I
> get
> > the following warnings when compiling the
> > platform-specific irq.c file:
> 
> This is ZBus-based profiling which also isn't
> supported by the standard
> profiling tool oprofile.  Oprofile is using the
> performance counters of
> the processor itself.
> 
> > My second question is with regards to accessing
> the
> > performance counters and timestamp counters from
> > userspace. On some architectures, this is as
> simple as
> > using a single macro.
> > 
> > In the case of the ix86 architecture (yuk!), the
> > timestamp counters can be read with nothing more
> than
> > an rdtsc() call, as follows:
> > 
> > asm volatile ("rdtsc" : "=a"(*(elg_ui4
> > *)&clock_value),
> >                 "=d"(*(((elg_ui4
> *)&clock_value)+1)));
> > 
> > What is the closest equiv. for the MIPS
> processors?
> 
> On most R4000-style processors (that includes the
> SB1 core of the Sibyte
> chips) applications can access the cycle counter
> through an
> mfc0 $reg, $c0_count instruction.  However mfc0 is a
> priviledged
> instruction, so that doesn't work for code that
> doesn't have kernel
> priviledges.
> 
> For release 2 of the MISP32 / MIPS64 architecture
> there is a new
> instruction, rdhwr which an application - so the OS
> permits it - can
> use to read c0_count.
> 
> Now there are two problems with that approach in
> your case:
> 
>  o SB1 implements release 0.95 of the MIPS64
> architecture, SB1A release 1.
>    Iow these cores don't have rdhwr.
>  o In general on a multiprocessor system you don't
> have a guarantee that
>    the count registers of all processors are running
> at the same speed or
>    were set to the same value at any time.
>       This is more of a general problem; in case of
> the BCM1250 the cores
>    are actually running at the same speed and afair
> are synchronized by
>    the reset.
> 
>   Ralf
> 
> 


__________________________________________________
Do You Yahoo!?
Tired of spam?  Yahoo! Mail has the best spam protection around 
http://mail.yahoo.com 

Re: Performance counters and profiling on MIPS

[Prasad Boddupalli]

Perfctr (http://user.it.uu.se/~mikpe/linux/perfctr/) and PAPI
(http://icl.cs.utk.edu/papi/) are precisely such attempts. Except that
MIPS ports of them do not seem to be available.

regards,
Prasad.

On 6/13/06, Jonathan Day <imipak@yahoo.com> wrote:
> Thank you for the valuable information.
>
> One thing I'd like to throw open to the list: there's
> one way to access the counters on the R4000-type
> processors, another on the version 2 MIPS64, yet
> another on the ix86, and so on.
>
> Would it make sense to place some standardised
> interface in, say, the assembly header files and hide
> the implementation-specific details? In the case of
> the R4000-type cores, this would need to involve some
> sort of counter device in the kernel which the macro
> would call to perform the priviledged instruction. (It
> feels a little bit of a hack, but it's the simplest
> way to provide access to resources that aren't made
> public.)
>
> What I'm thinking is that this generic interface would
> then be used on all other architectures, where such
> counters exist. That way, implementation-specific
> stuff can be abstracted out and programs that need
> access to performance counters can all be coded to a
> generic interface, rather than one interface for each
> version of every CPU API, which is inevitably going to
> be far more prone to error.
>
> Jonathan

Re: Performance counters and profiling on MIPS

[Nigel Stephens]

Prasad Boddupalli wrote:
> Perfctr (http://user.it.uu.se/~mikpe/linux/perfctr/) and PAPI
> (http://icl.cs.utk.edu/papi/) are precisely such attempts. Except that
> MIPS ports of them do not seem to be available.

There's also perfmon2, for which a MIPS patch is available - though no 
idea how up-to-date it is. See http://www.linux-mips.org/wiki/Perfmon2

Nigel

Re: Performance counters and profiling on MIPS

[Jonathan Day]

Ok, the kernel version number listed is current to
2.6.17-rc6, and the MIPS patches -almost- go in
cleanly.

In the syscalls in arch/mips/kernel, there is a new
syscall (sys_tee) that throws the patches off as it is
not in the context. This is very easy to massage.

The same is true of include/asm-mips/unistd.h, except
there the count of syscalls is also off by one. Again,
a very easy fix.

Other than that, it looks current and looks good. I'm
going to be doing some testing on it, to see whether
it works as well as it looks, or whether it causes the
CPU to leap three feet in the air, discharging the
magic blue smoke.

If other people have had success with it, though, I
would definitely suggest considering it for inclusion
in the linux-mips GIT tree. Those who don't need
performance counters won't be adversely affected, and
those of us who do would likely benefit.

If the linux-mips tree would not be appropriate, then
could someone take up hypnosis and get it included in
the main tree?

Jonathan

--- Nigel Stephens <nigel@mips.com> wrote:

> Prasad Boddupalli wrote:
> > Perfctr
> (http://user.it.uu.se/~mikpe/linux/perfctr/) and
> PAPI
> > (http://icl.cs.utk.edu/papi/) are precisely such
> attempts. Except that
> > MIPS ports of them do not seem to be available.
> 
> There's also perfmon2, for which a MIPS patch is
> available - though no 
> idea how up-to-date it is. See
> http://www.linux-mips.org/wiki/Perfmon2
> 
> Nigel
> 


__________________________________________________
Do You Yahoo!?
Tired of spam?  Yahoo! Mail has the best spam protection around 
http://mail.yahoo.com 

Re: Performance counters and profiling on MIPS

[Philip Mucci]

Hi folks,

I just got around to checking this thread...about 4 months late. Stefane
will release another version of the perfmon2 kernel patch in a few
days, he's been working with LKML on cleaning things up. The current
patch should go reasonably cleanly into the latest tree. The problem is
that he's maintaining the core patch against linux mainline, so there is
a bit of skew...be prepared to fix up some sys call numbers and some
includes, that's about it. There have been some issues with some
versions of linux mips not registering a cpu in /sys/devices, but that
may be fixed in the latest snapshots. 

A part of this patch is the libpfm support library, which helps in
counter allocation amongst other things. It isn't necessary to use the
interface. It can also support the scenario where 

The new release of PAPI will include support for this subsystem on
Linux/MIPS, but only for some MIPS64 processors. To add a new MIPS
processors should be fairly easy, as the performance hardware on these
chips isn't so exciting. 

Regards,

Philip

On Wed, 2006-06-14 at 11:14 -0700, Jonathan Day wrote:
> Ok, the kernel version number listed is current to
> 2.6.17-rc6, and the MIPS patches -almost- go in
> cleanly.
> 
> In the syscalls in arch/mips/kernel, there is a new
> syscall (sys_tee) that throws the patches off as it is
> not in the context. This is very easy to massage.
> 
> The same is true of include/asm-mips/unistd.h, except
> there the count of syscalls is also off by one. Again,
> a very easy fix.
> 
> Other than that, it looks current and looks good. I'm
> going to be doing some testing on it, to see whether
> it works as well as it looks, or whether it causes the
> CPU to leap three feet in the air, discharging the
> magic blue smoke.
> 
> If other people have had success with it, though, I
> would definitely suggest considering it for inclusion
> in the linux-mips GIT tree. Those who don't need
> performance counters won't be adversely affected, and
> those of us who do would likely benefit.
> 
> If the linux-mips tree would not be appropriate, then
> could someone take up hypnosis and get it included in
> the main tree?
> 
> Jonathan
> 
> --- Nigel Stephens <nigel@mips.com> wrote:
> 
> > Prasad Boddupalli wrote:
> > > Perfctr
> > (http://user.it.uu.se/~mikpe/linux/perfctr/) and
> > PAPI
> > > (http://icl.cs.utk.edu/papi/) are precisely such
> > attempts. Except that
> > > MIPS ports of them do not seem to be available.
> > 
> > There's also perfmon2, for which a MIPS patch is
> > available - though no 
> > idea how up-to-date it is. See
> > http://www.linux-mips.org/wiki/Perfmon2
> > 
> > Nigel
> > 
> 
> 
> __________________________________________________
> Do You Yahoo!?
> Tired of spam?  Yahoo! Mail has the best spam protection around 
> http://mail.yahoo.com 
>