2.6.16-rc5 perfmon2 new code base + libpfm with Montecito support

2.6.16-rc5 perfmon2 new code base + libpfm with Montecito support

[Stephane Eranian]

Hello,

I have released another version of the perfmon new base package.
This release is relative to 2.6.16-rc5

Due to the addition of a few system calls since 2.6.16-rc1, the
system call base number for all perfmon calls has changed yet again
on all architectures. As such you need to use the new version of
libpfm: libpfm-3.2-060308. 

This new kernel patch includes several important changes:

	- migrated all of /proc perfmon interface to /sys. The information
	  is split between general and per-cpu. As such it is located in
	  /sys/kernel/perfmon and /sys/devices/system/cpu*/perfmon/

	- all data structures exchanged with the kernel now use fixed size
	  data types. As such they can be used without any changes by a 32-bit
	  application running on top on a 64-bit OS. The following changes took
	  place:
		* all bitvectors use u64 and have fixed number of bits
		* size_t is now u64
		* struct timespec (set timeout) replaced by u32 (unit in micro-seconds)
		* default format sampling buffer IP (instruction pointer) is u64

	- added 32-bit ABI support for x86-64 arch. As a consequence, it is now
	  possible to run a 32-bit P4 tool on a 64-bit EM64T processor, assuming the
	  tool know the specifics of EM64T PMU. That goes also for PEBS support.

	- Simplified the PMU description data structures. The structure pfm_pmu_config
	  is now split between a perfmon internal and PMU description version providing
	  further isolation between the core and module-provided support. There is
	  a simplification also in the struct pfm_reg_desc tables describing the mappings.
	  Tables are much smaller because they only contain the actual register mappings
	  for a particular PMU model as opposed to a fixed size table of 320 elements. The
	  perfmon core now builds its internal representation based on this information as
	  opposed to using it directly.

	- removed extraneous locking and interrupt masking for the PMU interrupt handler

	- split kernel perfmon headers files based on functionality

	- added experimental support for smp_call_single() on i386. If the implementation
	  is satisfactory, I'll push it as a separate patch to lkml.

There was not update to the MIPS specific patch. As such it is currently broken and
cannot be used with the new libpfm.

The new version of the library, libpfm, includes the following changes:

	- preliminary support for Dual-Core Itanium 2 (Montecito) based on
	  publicly available documentation

	- update to match 2.6.16-rc5 kernel patch

	- renamed all *gen_x86_64* files to *amd_x86_64* to avoid confusion
	  with EM64T.

	- incorporated AMD provided event table for Opteron (Ray Bryant)


The two packages can be downloaded from the SourceForge website at:

	http://www.sf.net/projects/perfmon2

Enjoy,

-- 
-Stephane

Re: 2.6.16-rc5 perfmon2 new code base + libpfm with Montecito support

[William Cohen]

Stephane Eranian wrote:
> Hello,
> 
> I have released another version of the perfmon new base package.
> This release is relative to 2.6.16-rc5

Hello Stephane,

Is there any thoughts on how perfmon2 is going to work with xen enabled 
kernels or processors that support virtualization?

-Will

Re: 2.6.16-rc5 perfmon2 new code base + libpfm with Montecito support

[Stephane Eranian]

Will,

On Wed, Mar 08, 2006 at 03:40:16PM -0500, William Cohen wrote:
> >
> >I have released another version of the perfmon new base package.
> >This release is relative to 2.6.16-rc5
> 
> Hello Stephane,
> 
> Is there any thoughts on how perfmon2 is going to work with xen enabled 
> kernels or processors that support virtualization?
> 

AFAIK, there is currently no Xen support for PMU on any platforms. By "support"
I mean suport for guests using the PMU. As such, I do not think this works.
I am planning on looking at this next because this is becoming a pressing matter
and not just on IA-64.

My first goal is to ensure that a guest using perfmon2 works once it is virtualized.
That implies that the Xen VMM does save/restore PMU state on guest switch. That's a bare
minimum.

As for HW virutlization support, I think it helps a little bit but there still needs to
be some additional code in the VMM to make this work correctly. This is also something
I want to look at.

-- 
-Stephane

Re: [Perfctr-devel] 2.6.16-rc5 perfmon2 new code base + libpfm with Montecito support

[William Cohen]

[-- Attachment #1: Type: text/plain, Size: 493 bytes --]

Hi Stephane,

I have been looking through the perfmon2 code to see how it is going to 
work with OProfile. It looks like the ia64 oprofile support has not been 
modified to work with the changes in perfmon2. Has the ia64 kernel been 
built with perfmon2 and oprofile support? I don't have easy access to an 
ia64, so I haven't been able to verify that the attached patch works. 
However, I expect that the changes in the patch will be required for 
OProfile to function with perfmon2.

-Will


[-- Attachment #2: perfmon2ia64_oprof.diff --]
[-- Type: text/x-patch, Size: 2754 bytes --]

--- linux-2.6.16-rc5-perfop/arch/ia64/oprofile/perfmon.c.ia64chg	2006-03-13 10:56:30.000000000 -0500
+++ linux-2.6.16-rc5-perfop/arch/ia64/oprofile/perfmon.c	2006-03-13 12:31:48.000000000 -0500
@@ -8,22 +8,27 @@
  */
 
 #include <linux/kernel.h>
+#include <linux/module.h>
 #include <linux/config.h>
 #include <linux/oprofile.h>
 #include <linux/sched.h>
-#include <asm/perfmon.h>
+#include <linux/perfmon.h>
 #include <asm/ptrace.h>
 #include <asm/errno.h>
 
 static int allow_ints;
 
 static int
-perfmon_handler(struct task_struct *task, void *buf, pfm_ovfl_arg_t *arg,
-                struct pt_regs *regs, unsigned long stamp)
+perfmon_handler(void *buf, struct pfm_ovfl_arg *arg,
+                unsigned long ip, u64 stamp, void *data)
 {
 	int event = arg->pmd_eventid;
  
-	arg->ovfl_ctrl.bits.reset_ovfl_pmds = 1;
+	/* FIXME: oprofile_add_sample expect to have
+	   struct pt_regs * const regs */
+	struct pt_regs * const regs = (struct pt_regs *) data;
+
+	arg->ovfl_ctrl = PFM_OVFL_CTRL_RESET;
 
 	/* the owner of the oprofile event buffer may have exited
 	 * without perfmon being shutdown (e.g. SIGSEGV)
@@ -54,6 +59,7 @@
  	.fmt_name 	    = "oprofile_format",
  	.fmt_uuid	    = OPROFILE_FMT_UUID,
  	.fmt_handler	    = perfmon_handler,
+	.owner		    = THIS_MODULE,
 };
 
 
@@ -76,9 +82,9 @@
 
 static int using_perfmon;
 
-int perfmon_init(struct oprofile_operations * ops)
+int __init op_perfmon_init(struct oprofile_operations * ops)
 {
-	int ret = pfm_register_buffer_fmt(&oprofile_fmt);
+	int ret = pfm_register_smpl_fmt(&oprofile_fmt);
 	if (ret)
 		return -ENODEV;
 
@@ -91,10 +97,10 @@
 }
 
 
-void perfmon_exit(void)
+void __exit op_perfmon_exit(void)
 {
 	if (!using_perfmon)
 		return;
 
-	pfm_unregister_buffer_fmt(oprofile_fmt.fmt_uuid);
+	pfm_unregister_smpl_fmt(oprofile_fmt.fmt_uuid);
 }
--- linux-2.6.16-rc5-perfop/arch/ia64/oprofile/init.c.ia64chg	2006-03-13 11:02:19.000000000 -0500
+++ linux-2.6.16-rc5-perfop/arch/ia64/oprofile/init.c	2006-03-13 11:03:09.000000000 -0500
@@ -12,8 +12,8 @@
 #include <linux/init.h>
 #include <linux/errno.h>
  
-extern int perfmon_init(struct oprofile_operations * ops);
-extern void perfmon_exit(void);
+extern int op_perfmon_init(struct oprofile_operations * ops);
+extern void op_perfmon_exit(void);
 extern void ia64_backtrace(struct pt_regs * const regs, unsigned int depth);
 
 int __init oprofile_arch_init(struct oprofile_operations * ops)
@@ -22,7 +22,7 @@
 
 #ifdef CONFIG_PERFMON
 	/* perfmon_init() can fail, but we have no way to report it */
-	ret = perfmon_init(ops);
+	ret = op_perfmon_init(ops);
 #endif
 	ops->backtrace = ia64_backtrace;
 
@@ -33,6 +33,6 @@
 void oprofile_arch_exit(void)
 {
 #ifdef CONFIG_PERFMON
-	perfmon_exit();
+	op_perfmon_exit();
 #endif
 }

Re: [Perfctr-devel] 2.6.16-rc5 perfmon2 new code base + libpfm with Montecito support

[Stephane Eranian]

Will,

On Mon, Mar 13, 2006 at 01:39:01PM -0500, William Cohen wrote:
> Hi Stephane,
> 
> I have been looking through the perfmon2 code to see how it is going to 
> work with OProfile. It looks like the ia64 oprofile support has not been 
> modified to work with the changes in perfmon2. Has the ia64 kernel been 
> built with perfmon2 and oprofile support? I don't have easy access to an 
> ia64, so I haven't been able to verify that the attached patch works. 
> However, I expect that the changes in the patch will be required for 
> OProfile to function with perfmon2.
> 
Good timing. I just fixed this today. Now it compiles fine on
IA64.  I also started looking into using the same technique on
i386. It is very easy. It looks like opcontrol or ophelp
would need to be updated. I think the trick is to make
sure that ophelp knows the PMU mapping used by perfmon2,
i.e., knows that PERFEVTSEL0 is PMC0 for instance.

-- 
-Stephane

Re: [Perfctr-devel] 2.6.16-rc5 perfmon2 new code base + libpfm with Montecito support

[William Cohen]

Stephane Eranian wrote:
> Will,
> 
> On Mon, Mar 13, 2006 at 01:39:01PM -0500, William Cohen wrote:
> 
>>Hi Stephane,
>>
>>I have been looking through the perfmon2 code to see how it is going to 
>>work with OProfile. It looks like the ia64 oprofile support has not been 
>>modified to work with the changes in perfmon2. Has the ia64 kernel been 
>>built with perfmon2 and oprofile support? I don't have easy access to an 
>>ia64, so I haven't been able to verify that the attached patch works. 
>>However, I expect that the changes in the patch will be required for 
>>OProfile to function with perfmon2.
>>
> 
> Good timing. I just fixed this today. Now it compiles fine on
> IA64.  I also started looking into using the same technique on
> i386. It is very easy. It looks like opcontrol or ophelp
> would need to be updated. I think the trick is to make
> sure that ophelp knows the PMU mapping used by perfmon2,
> i.e., knows that PERFEVTSEL0 is PMC0 for instance.
> 

Yes, I have a similar patch for i386 in the kernel. I don't yet have 
modifications for opcontrol or ophelp.

One question would be identifying the processor when using the perfmon2 
support for i386/* processors? There is prior support in the oprofile 
driver for i386 processors. Identify the processor differently depending 
on whether perfmon2 is being used to distinguish between the different 
interfaces? The way that OProfile has the events each name processor 
requires a different directory in /usr/share/oprofile. Would prefer to 
keep down the proliferation of new directories.

-Will

Re: [Perfctr-devel] 2.6.16-rc5 perfmon2 new code base + libpfm with Montecito support

[Stephane Eranian]

Will,

On Mon, Mar 13, 2006 at 02:15:53PM -0500, William Cohen wrote:
> 
> Yes, I have a similar patch for i386 in the kernel. I don't yet have 
> modifications for opcontrol or ophelp.

Your patch is very close to mine. I'll merge tthe two and will include
it in my next version.

My understand of opcontrol is that is passes the information from ophelp
to the kernel via /dev/oprofile. I don't know how it passes it to the oprofiled
daemon.

The daemon should not be difficult to change. I hacked something quickly
and got it up on pentium M. The only remaining problem is ophelp, I think.


> One question would be identifying the processor when using the perfmon2 
> support for i386/* processors? There is prior support in the oprofile 
> driver for i386 processors. Identify the processor differently depending 
> on whether perfmon2 is being used to distinguish between the different 
> interfaces? The way that OProfile has the events each name processor 
> requires a different directory in /usr/share/oprofile. Would prefer to 
> keep down the proliferation of new directories.

I think it would be easier to check in /sys/kernel/perfmon to detect
that it is running on perfmon2. Then opcontrol can pass the inormation
around to ophelp, oprofiled is necessary. Ophelp then just needs
to know the perfmon2 register mapping for each CPU. I don't know
how this information is represented today.

-- 
-Stephane

Re: [Perfctr-devel] 2.6.16-rc5 perfmon2 new code base + libpfm with Montecito support

[William Cohen]

Stephane Eranian wrote:
> Will,
> 
> On Mon, Mar 13, 2006 at 02:15:53PM -0500, William Cohen wrote:
> 
>>Yes, I have a similar patch for i386 in the kernel. I don't yet have 
>>modifications for opcontrol or ophelp.
> 
> 
> Your patch is very close to mine. I'll merge tthe two and will include
> it in my next version.
> 
> My understand of opcontrol is that is passes the information from ophelp
> to the kernel via /dev/oprofile. I don't know how it passes it to the oprofiled
> daemon.
> 
> The daemon should not be difficult to change. I hacked something quickly
> and got it up on pentium M. The only remaining problem is ophelp, I think.

Is new identifier cpu_type being created or is i386/pentium_m being used 
also for perfmon2? It could be inferred that perfmon2 is being used if 
the procesor is identified, but there are no number directories for the 
counters (/dev/oprofile/[0-9]+).

I looked over the code this afternoon to refresh my memory how it actual 
work (as opposed to how I thought it works).

ophelp uses oprofile/libop/op_cpu_type.c:op_get_cpu_type to read 
/dev/oprofile/cpu_type. Once ophelp obtains the cpu type the appropriate 
directory from /usr/share/oprofile for the events. This information is 
passed to oprofiled; you can see it with "opcontrol --start --verbose".

oprofile/daemon/opd_perfmon.c and opd_perfmon.h are the place that the 
performance monitoring hardware is set for perfmon. 
opd_perfmon.c:write_pmu() is currently hard coded for ia64. As you have 
seen, it doesn't use the libpfm to set up the performance monitoring 
hardware.

Looking over the code in the linux/arch/i386/oprofile the code for 
typing the processor is tied to the initialization of the performance 
monitoring hardware. Probably want to factor out cpu idenification code 
from oprofile native handling of the performance monitoring code. So the 
pseudo code is something like in 
linux/arch/i386/oprofile/init.c:oprofile_arch_init():

char *processor_id = identify_processer();
ret = op_perfmon_support(ops, processor_id);
if (ret < 0)
	ret = op_nmi_init(ops, processor_id);
if (ret < 0)
	ret = op_nmi_timer(ops, processor_id);
return ret;

-Will

Re: [Perfctr-devel] 2.6.16-rc5 perfmon2 new code base + libpfm with Montecito support

[Stephane Eranian]

Will,

On Mon, Mar 13, 2006 at 03:58:06PM -0500, William Cohen wrote:
> 
> Is new identifier cpu_type being created or is i386/pentium_m being used 
> also for perfmon2? It could be inferred that perfmon2 is being used if 
> the procesor is identified, but there are no number directories for the 
> counters (/dev/oprofile/[0-9]+).
> 
Existing OProfile codes treats pentium M as i386/piii as it should more or less.

> I looked over the code this afternoon to refresh my memory how it actual 
> work (as opposed to how I thought it works).
> 
> ophelp uses oprofile/libop/op_cpu_type.c:op_get_cpu_type to read 
> /dev/oprofile/cpu_type. Once ophelp obtains the cpu type the appropriate 
> directory from /usr/share/oprofile for the events. This information is 
> passed to oprofiled; you can see it with "opcontrol --start --verbose".
> 
In the quick hack I did over the week-end, I followed what John had done
for IA-64. In other words, I also have a get_cpu() routine for i386 and
I basically did a cut&paste of what was in nmi_int.c. As such,
/dev/oprofile/cpu_type is preserved to avoid confusing ophelp.

> oprofile/daemon/opd_perfmon.c and opd_perfmon.h are the place that the 
> performance monitoring hardware is set for perfmon. 
> opd_perfmon.c:write_pmu() is currently hard coded for ia64. As you have 
> seen, it doesn't use the libpfm to set up the performance monitoring 
> hardware.

Libpfm is really a helper library and is not required to use the kernel
interface. However, there is a bit of an anomaly in libpfm at the moment.
The library also includes the kernel header files and perfmon2 syscall stubs.
Both would eventually be found in libc.  As such, for the time being, some
applications may choose to link against libpfm, just for the stubs. The current
oprofiled does not do this and has every perfmon.h definitions it needs hardcoded
(syscalls stubs are also recreated).

> 
> Looking over the code in the linux/arch/i386/oprofile the code for 
> typing the processor is tied to the initialization of the performance 
> monitoring hardware. Probably want to factor out cpu idenification code 
> from oprofile native handling of the performance monitoring code. So the 
> pseudo code is something like in 
> linux/arch/i386/oprofile/init.c:oprofile_arch_init():
> 
> char *processor_id = identify_processer();
> ret = op_perfmon_support(ops, processor_id);
> if (ret < 0)
> 	ret = op_nmi_init(ops, processor_id);
> if (ret < 0)
> 	ret = op_nmi_timer(ops, processor_id);
> return ret;
> 
Yes that could be a possibility. That implies that there may be
situations where perfmon does not support a processor that is
somehow supported by OProfile. I suspect this could be the case
for very old CPUs.

-- 
-Stephane

Re: [Perfctr-devel] 2.6.16-rc5 perfmon2 new code base + libpfm with Montecito support

[Stephane Eranian]

John,

On Mon, Mar 13, 2006 at 11:20:57PM +0000, John Levon wrote:
> On Mon, Mar 13, 2006 at 01:03:54PM -0800, Stephane Eranian wrote:
> 
> > > 1) the event names are synchronised so we don't need massive duplication
> > 
> > The kernel perfmon2 interface does not know anything about event names.
> 
> Well, it sounded like Will was proposing extra event directories.

I think Will was trying to solve the register naming differences.
I do not know how you deal with this in ophelp.

> > > 2) that "start a thread on each CPU" API is fixed to be sane
> >
> > Please develop on this point some more.
> 
> The kernel interface should just let me say "I want this setup on all
> CPUs" and do the IPIs for me.
> 
That's is because you are assuming a model were you always want to monitor
all CPUs each time and measure the same thing everywhere.  This does not
always makes sense in large configurations. You may want to only monitor
a subset of CPUs or you may want to monitor different things on different
CPUs at the same time, for instance because they handle different workloads
or interrupts.

When it comes to sampling, I think you will agree with me that the kernel level 
sampling buffer must be per-cpu. I think this is also how you manage
it in OProfile. I think it also makes sense to process the buffer locally for
cache affinity reasons for instance. Keep monitoring overhead minimum by exploiting
locality. I think (correct me if I am wrong) in Oprofile you somehow merge the
per-cpu buffers into a single buffer which is then read via read() by user level
applications. For some measurements merging is not necessarily what is needed.

In your model, I would have to pass a bitmap of CPUs to monitor, then internally
the kernel would have propagate the setup via IPI and maintain a context per-cpu.
Then upon return, it would have to pass information as to how to mmap the per-cpu
buffers. You have a choice of doing one mmap() per buffer or to do
a single large mmap() covering the possibly discountiguous physical pages 
backing each per-cpu buffer. In either case, it would make sense to ensure
that the thread processing each buffer runs on the CPU where the samples
have been collected to minimize cache traffic which is very important on NUMA
machines. Typically on those machines, every effort is made to keep all memory
accesses local, I do not see why this would not also apply to profiling.

Note that in the new perfmon code base, you do not have to create one thread
per monitored CPU. All you need to do is to ensure that the thread runs on
the CPU it needs to access when issuing perfmon2 calls causing actual PMU 
HW accesses. A single thread can very well control lots of context bound to
different threads or CPU.

But again, I am always open to discussions/proposals on this.

-- 
-Stephane