2.6.13-rc2 perfmon2 new code base with MIPS5K/20K support + libpfm available

2.6.13-rc2 perfmon2 new code base with MIPS5K/20K support + libpfm available

[Philip Mucci]

Hello Linux-MIPSers,

Below is an announcement of the kernel perfmon/libpfm support and user
library for hardware performance monitoring on MIPS (and other
platforms). This support mirrors that which has been available on IA64
for quite some time and is an effort by Stefane Eranian, David Gibson
myself and others to establish a fully functional kernel substrate for
hardware performance analysis. This is based on a large body of
work/experience with PM kernel support both on Linux and many other
systems.

Note that the MIPS support is based on a 2.6.13-rc2 snapshot of the LM
code base, soon to be updated to 2.6.15. If you want to patch against
the head, you'll probably have to fix up the syscall numbers.

I have tested the code on a 20K system + 64bit kernel (that has 1 entire
PMC register) and soon will be testing on a 5K system when it comes back
from being serviced. n32/n64 ABI's have not yet been tested but support
is there.

PAPI is also in the works for these systems. Feel free to give it a spin
and tell us where it breaks or otherwise offends you. 

Regards,

Philip

P.S. The code should be relatively easy to extend to other members of
the product line...currently I've only got kernel code in for
5K/20K/25K, which are nearly identical as far as the PMC goes. 

-------- Forwarded Message --------
From: Stephane Eranian <eranian@hpl.hp.com>
Reply-To: eranian@hpl.hp.com
To: linux-kernel@vger.kernel.org
Cc: linux-ia64@vger.kernel.org, perfmon@napali.hpl.hp.com,
perfctr-devel@lists.sourceforge.net
Subject: [Perfctr-devel] 2.6.16-rc1 perfmon2 new code base with
Montecito support + libpfm available
Date: Wed, 18 Jan 2006 12:56:30 -0800

Hello,

I have released a new version of the perfmon base package.
This release is relative to 2.6.16-rc1.

Due to the addition of the migrate_pages() system call, all perfmon
system calls have been shifted by one. As a consequence, you must
relink your applications using the new version of libpfm which
is also released today: libpfm-3.2-060118.

This new kernel patch includes several important changes:

	- added support for the Dual-Core Itanium 2 (Montecito) processor

	- working support for MIPS5K and MIPS20k (by Phil Mucci)

	- lots of cleanups in the PMU description modules. Some common
	  functionalities moved into core (masking of reserved fields)

	- PFM_REGFL_NEMUL64 managed from perfmon core

The new version of the library, libpfm, includes the following changes:

	- added support for MIPS5K, MIPS20K

	- added pfm_get_event_counter() interface with man page

	- added support for compiling using a 32-bit ABI on a 64-bit OS
	  (not available on all platforms, see config.mk)

	- reworked all standalone examples

	- cleanups and fixes in the generic examples

The two packages can be downloaded from the SourceForge website at:

	http://www.sf.net/projects/perfmon2

I will be posting the patches directly to LKML for
review very shortly.

Enjoy,

Re: 2.6.13-rc2 perfmon2 new code base with MIPS5K/20K support + libpfm available

[Ralf Baechle]

On Thu, Jan 19, 2006 at 11:30:02AM +0100, Philip Mucci wrote:

> Below is an announcement of the kernel perfmon/libpfm support and user
> library for hardware performance monitoring on MIPS (and other
> platforms). This support mirrors that which has been available on IA64
> for quite some time and is an effort by Stefane Eranian, David Gibson
> myself and others to establish a fully functional kernel substrate for
> hardware performance analysis. This is based on a large body of
> work/experience with PM kernel support both on Linux and many other
> systems.
> 
> Note that the MIPS support is based on a 2.6.13-rc2 snapshot of the LM
> code base, soon to be updated to 2.6.15. If you want to patch against
> the head, you'll probably have to fix up the syscall numbers.
> 
> I have tested the code on a 20K system + 64bit kernel (that has 1 entire
> PMC register) and soon will be testing on a 5K system when it comes back
> from being serviced. n32/n64 ABI's have not yet been tested but support
> is there.
> 
> PAPI is also in the works for these systems. Feel free to give it a spin
> and tell us where it breaks or otherwise offends you. 
> 
> Regards,
> 
> Philip
> 
> P.S. The code should be relatively easy to extend to other members of
> the product line...currently I've only got kernel code in for
> 5K/20K/25K, which are nearly identical as far as the PMC goes. 

More recently myself and others have verified that Oprofile is working
on the 5K, 20K, 24K, 25K cores and the SB1 and SB1A cores in the Sibyte
SOCs - and a few more in the queue.  So now I wonder how perfmon2 is
going to interfear with oprofile which already is in the kernel?

I've put a quick page into the wiki at http://www.linux-mips.org/wiki/Perfmon2
It's really just a starting point but people should know what's there.

  Ralf

Re: 2.6.13-rc2 perfmon2 new code base with MIPS5K/20K support + libpfm available

[Philip Mucci]

Hi Ralf,

Thanks for the link on the wiki...

I'm sure Stefane can answer this question more accurately, but I do know
that oprofile and perfmon have coexisted quite happily for a while in
the IA64 line..Of course, you can't use both at the same time. oprofile
takes over the counters and uses them from a 'global' context, and you
need to be root to set them. 

perfmon is intended up be used for performance tuning in production
multiprogrammed environments, although it also has system-wide and
per-cpu counting modes. So you can have multiple people using the
counters inside their processes and threads and all the counts are
preserved as the state and the full 64 bit values are part of the
process context, for the per-thread monitoring modes.

Again, this is a laymans answer and I'll let Stefane fill in the gaps of
how oprofile and perfmon2 are integrated on the IA64 platform and how we
make that work similarly on the MIPS platform. 

As I haven't updated my tree quite yet to 2.6.15, I haven't yet seen the
oprofile code for the below...but it integration should be quite simple.
(My tree only has the rm9000). As soon as I reintegrate to 2.6.15, I'll
be sure to make sure oprofile and perfmon2 play nice.

Anyways, glad to hear other folks are as interested in performance
analysis!

Regards,

Philip

> More recently myself and others have verified that Oprofile is working
> on the 5K, 20K, 24K, 25K cores and the SB1 and SB1A cores in the Sibyte
> SOCs - and a few more in the queue.  So now I wonder how perfmon2 is
> going to interfear with oprofile which already is in the kernel?
> 
> I've put a quick page into the wiki at http://www.linux-mips.org/wiki/Perfmon2
> It's really just a starting point but people should know what's there.
> 
>   Ralf

Re: 2.6.13-rc2 perfmon2 new code base with MIPS5K/20K support + libpfm available

[Nigel Stephens]

Philip Mucci wrote:

>perfmon is intended up be used for performance tuning in production
>multiprogrammed environments, although it also has system-wide and
>per-cpu counting modes. So you can have multiple people using the
>counters inside their processes and threads and all the counts are
>preserved as the state and the full 64 bit values are part of the
>process context, for the per-thread monitoring modes.
>  
>

How does perfmon differ from the perfctr project, which seems to be 
doing something very similar? See http://user.it.uu.se/~mikpe/linux/perfctr/

>
>
>Anyways, glad to hear other folks are as interested in performance
>analysis!
>
>  
>

We most definitely are, in particular we are looking for good tools with 
which to analyse threaded applications running on multi-threading 
hardware. Does this version of perfmon support threaded code?

Nigel

Re: 2.6.13-rc2 perfmon2 new code base with MIPS5K/20K support + libpfm available

[Philip Mucci]

Hi Nigel,

There is a much more detailed discussion to be found on the perfctr
mailing list...we all worked through that when deciding on perfmon. I
have been a big advocate in the past for perfctr especially since it
played such an important part of the PAPI work. But in a nutshell:

- kernel level event multiplexing
- buffered IP sampling and profiling
- remote (third party) monitoring
- support for event address features through sampling like IA64, PIV and
PPC64 support
- support for randomization

To name the most important ones...

Stefane has worked with us, and Mikael P, to make sure that perfmon
provides everything that Perfctr did...one notable addition was support
for mmaped counters where you can read the full 64 bit quantity in user
mode...Not possible on MIPS64 though...darn privileged mfc
instructions...that's avail on the 10/12/14K and probably others of the
MIPS line...

Check this link:
http://sourceforge.net/mailarchive/message.php?msg_id=12829209

Regards,

Phil


On Thu, 2006-01-19 at 15:33 +0000, Nigel Stephens wrote:
> 
> Philip Mucci wrote:
> 
> >perfmon is intended up be used for performance tuning in production
> >multiprogrammed environments, although it also has system-wide and
> >per-cpu counting modes. So you can have multiple people using the
> >counters inside their processes and threads and all the counts are
> >preserved as the state and the full 64 bit values are part of the
> >process context, for the per-thread monitoring modes.
> >  
> >
> 
> How does perfmon differ from the perfctr project, which seems to be 
> doing something very similar? See http://user.it.uu.se/~mikpe/linux/perfctr/
> 
> >
> >
> >Anyways, glad to hear other folks are as interested in performance
> >analysis!
> >
> >  
> >
> 
> We most definitely are, in particular we are looking for good tools with 
> which to analyse threaded applications running on multi-threading 
> hardware. Does this version of perfmon support threaded code?
> 
> Nigel

Re: 2.6.13-rc2 perfmon2 new code base with MIPS5K/20K support + libpfm available

[Stephane Eranian]

Ralf,

On Thu, Jan 19, 2006 at 01:36:09PM +0000, Ralf Baechle wrote:
> > PAPI is also in the works for these systems. Feel free to give it a spin
> > and tell us where it breaks or otherwise offends you. 
> > 
> > P.S. The code should be relatively easy to extend to other members of
> > the product line...currently I've only got kernel code in for
> > 5K/20K/25K, which are nearly identical as far as the PMC goes. 
> 
> More recently myself and others have verified that Oprofile is working
> on the 5K, 20K, 24K, 25K cores and the SB1 and SB1A cores in the Sibyte
> SOCs - and a few more in the queue.  So now I wonder how perfmon2 is
> going to interfear with oprofile which already is in the kernel?
> 
On Itaniun, where perfmon was orginally designed, we have Oprofile and
perfmon2 working together. You can use Oprofile or perfmon2 native tools.

Oprofile user level tools are using the perfmon2 interface to program the
counters. But the sampling buffer format and buffer export interface remain
the same as with "native" Oprofile. As Phil mentioned, perfmon2 implements
a kernel evel sampling buffer. But the format of the buffer, i.e., what gets
recorded, how it is recorded in the buffer and how the buffer gets exported
to user is implemented by a simple kernel module called a "custom sampling
format". The core of each format is a handler function called on counter
overflow. We use this mechanism to connect the Oprofile PMU interrupt handler
to perfmon, it is just 100 lines of C and we reuse the full Oprofile sample
recording infrastructure.

There is nothing specific to Itanium is thr way Oprofile is connecte to perfmon2.
As such, I believe that we could implement the same trick on all the other
architectures supported by perfmon2. I think some people would like to try this
on i386 first. But you are certainly welcome to try this on MIPS. The Oprofile
tools are already aware of perfmon on Itanium. It may be that they need to be
made ware when compiling for i386 or MIPS.

The goal of perfmon2 is not to get rid of OProfile but to allow the two
to work side by side but avoid duplication of kernel code as much as possible.
Some people are satisfied with the functionalities of Oprofile others are not,
especially because Oprofile does not provide per-thread monitoring. The perfmon2
interface is designed to be generic and flexible, it was designed with a particular
tool or measurement in mind.

> I've put a quick page into the wiki at http://www.linux-mips.org/wiki/Perfmon2
> It's really just a starting point but people should know what's there.
> 

-- 

-Stephane

Re: [perfmon] Re: 2.6.13-rc2 perfmon2 new code base with MIPS5K/20K support + libpfm available

[Philip Mucci]

Stefane,

Are you saying that oprofile user level tools don't even use the
oprofile driver but rather just set up a system-wide IP-sampling context
for perfmon2?

Think of all the work that Ralf just did. ;-)

Phil

On Thu, 2006-01-19 at 10:16 -0800, Stephane Eranian wrote:
> Ralf,
> 
> On Thu, Jan 19, 2006 at 01:36:09PM +0000, Ralf Baechle wrote:
> > > PAPI is also in the works for these systems. Feel free to give it a spin
> > > and tell us where it breaks or otherwise offends you. 
> > > 
> > > P.S. The code should be relatively easy to extend to other members of
> > > the product line...currently I've only got kernel code in for
> > > 5K/20K/25K, which are nearly identical as far as the PMC goes. 
> > 
> > More recently myself and others have verified that Oprofile is working
> > on the 5K, 20K, 24K, 25K cores and the SB1 and SB1A cores in the Sibyte
> > SOCs - and a few more in the queue.  So now I wonder how perfmon2 is
> > going to interfear with oprofile which already is in the kernel?
> > 
> On Itaniun, where perfmon was orginally designed, we have Oprofile and
> perfmon2 working together. You can use Oprofile or perfmon2 native tools.
> 
> Oprofile user level tools are using the perfmon2 interface to program the
> counters. But the sampling buffer format and buffer export interface remain
> the same as with "native" Oprofile. As Phil mentioned, perfmon2 implements
> a kernel evel sampling buffer. But the format of the buffer, i.e., what gets
> recorded, how it is recorded in the buffer and how the buffer gets exported
> to user is implemented by a simple kernel module called a "custom sampling
> format". The core of each format is a handler function called on counter
> overflow. We use this mechanism to connect the Oprofile PMU interrupt handler
> to perfmon, it is just 100 lines of C and we reuse the full Oprofile sample
> recording infrastructure.
> 
> There is nothing specific to Itanium is thr way Oprofile is connecte to perfmon2.
> As such, I believe that we could implement the same trick on all the other
> architectures supported by perfmon2. I think some people would like to try this
> on i386 first. But you are certainly welcome to try this on MIPS. The Oprofile
> tools are already aware of perfmon on Itanium. It may be that they need to be
> made ware when compiling for i386 or MIPS.
> 
> The goal of perfmon2 is not to get rid of OProfile but to allow the two
> to work side by side but avoid duplication of kernel code as much as possible.
> Some people are satisfied with the functionalities of Oprofile others are not,
> especially because Oprofile does not provide per-thread monitoring. The perfmon2
> interface is designed to be generic and flexible, it was designed with a particular
> tool or measurement in mind.
> 
> > I've put a quick page into the wiki at http://www.linux-mips.org/wiki/Perfmon2
> > It's really just a starting point but people should know what's there.
> > 
> 

Re: [perfmon] Re: 2.6.13-rc2 perfmon2 new code base with MIPS5K/20K support + libpfm available

[Stephane Eranian]

Phil,

On Thu, Jan 19, 2006 at 07:38:56PM +0100, Philip Mucci wrote:
> Stefane,
> 
> Are you saying that oprofile user level tools don't even use the
> oprofile driver but rather just set up a system-wide IP-sampling context
> for perfmon2?
> 
Yes, I think that is the way this works. I will verify this.

> Think of all the work that Ralf just did. ;-)
> 
If you don't have perfmon2, then you have to provide you arch/mips/oprofile
files for sure.

Re: [perfmon] Re: 2.6.13-rc2 perfmon2 new code base with MIPS5K/20K support + libpfm available

[Philip Mucci]

Yup...just looked at the code...should be simple to adapt...I'll give it
a spin on perfmon2...the only thing that seems to be obviously wrong are
the missing arguments to PFM_CREATE_CONTEXT.

Phil

On Thu, 2006-01-19 at 11:14 -0800, Stephane Eranian wrote:
> Phil,
> 
> On Thu, Jan 19, 2006 at 07:38:56PM +0100, Philip Mucci wrote:
> > Stefane,
> > 
> > Are you saying that oprofile user level tools don't even use the
> > oprofile driver but rather just set up a system-wide IP-sampling context
> > for perfmon2?
> > 
> Yes, I think that is the way this works. I will verify this.
> 
> > Think of all the work that Ralf just did. ;-)
> > 
> If you don't have perfmon2, then you have to provide you arch/mips/oprofile
> files for sure.
> _______________________________________________
> perfmon mailing list
> perfmon@linux.hpl.hp.com
> http://www.hpl.hp.com/hosted/linux/mail-archives/perfmon/

Re: [perfmon] Re: 2.6.13-rc2 perfmon2 new code base with MIPS5K/20K support + libpfm available

[Stephane Eranian]

On Thu, Jan 19, 2006 at 08:30:33PM +0100, Philip Mucci wrote:
> Yup...just looked at the code...should be simple to adapt...I'll give it
> a spin on perfmon2...the only thing that seems to be obviously wrong are
> the missing arguments to PFM_CREATE_CONTEXT.
> 
Yes, they are using the old perfmon2 interface available just
on IA-64. This perfmonctl() call is still available for backward
compatibility reason but ONLY on Ia-64. For all other
platforms you need to use the pfm_*() system calls. In fact, I think
that even on IA-64 they should eventaully migrate to this model.