perf counters: problem with perf record

perf counters: problem with perf record

[Simon Kämpflein]

Hello,

I have a problem with the new kernel perf tools in kernel 2.6.31.6 (very
nice tools by the way). I like to monitor the cache-misses of a program.
"perf stat -e cache-misses program" works without problems, but I can't
get "perf record -e cache-misses program" working (even running as root
and CONFIG_PERF_COUNTERS=y):

Error: perfcounter syscall returned with -1 (Operation not supported)
Fatal: No CONFIG_PERF_COUNTERS=y kernel support configured?

strace of "perf stat":
SYS_336(0x8508da0, 0x1074, 0xffffffff, 0xffffffff, 0) = 3

strace of "perf record":
SYS_336(0x8508da0, 0xf85, 0xffffffff, 0xffffffff, 0) = -1 EOPNOTSUPP
(Operation not supported)

My CPU is a "Intel(R) Pentium(R) M processor 1.80GHz":
[    0.004337] p6 PMU driver.
[    0.004343] ... version:                 0
[    0.004345] ... bit width:               32
[    0.004347] ... generic counters:        2
[    0.004349] ... value mask:              00000000ffffffff
[    0.004351] ... max period:              000000007fffffff
[    0.004353] ... fixed-purpose counters:  0
[    0.004355] ... counter mask:            0000000000000003

Best regards,
Simon

Re: perf counters: problem with perf record

[Peter Zijlstra]

On Sun, 2009-11-15 at 03:14 +1000, Simon Kämpflein wrote:
> Hello,
> 
> I have a problem with the new kernel perf tools in kernel 2.6.31.6 (very
> nice tools by the way). I like to monitor the cache-misses of a program.
> "perf stat -e cache-misses program" works without problems, but I can't
> get "perf record -e cache-misses program" working (even running as root
> and CONFIG_PERF_COUNTERS=y):
> 
> Error: perfcounter syscall returned with -1 (Operation not supported)
> Fatal: No CONFIG_PERF_COUNTERS=y kernel support configured?
> 
> strace of "perf stat":
> SYS_336(0x8508da0, 0x1074, 0xffffffff, 0xffffffff, 0) = 3
> 
> strace of "perf record":
> SYS_336(0x8508da0, 0xf85, 0xffffffff, 0xffffffff, 0) = -1 EOPNOTSUPP
> (Operation not supported)
> 
> My CPU is a "Intel(R) Pentium(R) M processor 1.80GHz":
> [    0.004337] p6 PMU driver.
> [    0.004343] ... version:                 0
> [    0.004345] ... bit width:               32
> [    0.004347] ... generic counters:        2
> [    0.004349] ... value mask:              00000000ffffffff
> [    0.004351] ... max period:              000000007fffffff
> [    0.004353] ... fixed-purpose counters:  0
> [    0.004355] ... counter mask:            0000000000000003

Does the bootlog perchance contain the output of?

        if (!cpu_has_apic) {
                pr_info("no APIC, boot with the \"lapic\" boot parameter
to force-enable it.\n");
                pr_info("no hardware sampling interrupt available.\n");
                x86_pmu.apic = 0;
        }

And does booting with the suggested lapic parameter cure your problems?

Re: perf counters: problem with perf record

[Simon Kämpflein]

Peter Zijlstra schrieb:
> On Sun, 2009-11-15 at 03:14 +1000, Simon Kämpflein wrote:
>> Hello,
>>
>> I have a problem with the new kernel perf tools in kernel 2.6.31.6 (very
>> nice tools by the way). I like to monitor the cache-misses of a program.
>> "perf stat -e cache-misses program" works without problems, but I can't
>> get "perf record -e cache-misses program" working (even running as root
>> and CONFIG_PERF_COUNTERS=y):
>>
>> Error: perfcounter syscall returned with -1 (Operation not supported)
>> Fatal: No CONFIG_PERF_COUNTERS=y kernel support configured?
>>
>> My CPU is a "Intel(R) Pentium(R) M processor 1.80GHz":
> 
> Does the bootlog perchance contain the output of?
> 
>         if (!cpu_has_apic) {
>                 pr_info("no APIC, boot with the \"lapic\" boot parameter
> to force-enable it.\n");
>                 pr_info("no hardware sampling interrupt available.\n");
>                 x86_pmu.apic = 0;
>         }

Yes. I didn't realize that it has something to do with the Performance
Counters because I didn't see that it belongs to the Performance Counter
output.
Maybe it's a good idea to add another comment making it clear that the
performance counters are not fully working in this case. Or detect this
case in "perf-record" and output a message relating to that.

> And does booting with the suggested lapic parameter cure your problems?
> 

Yes, thank you!

Regars,
Simon

Re: perf counters: problem with perf record

[Ingo Molnar]

* Simon K??mpflein <s.kaempflein@gmx.de> wrote:

> Peter Zijlstra schrieb:
> > On Sun, 2009-11-15 at 03:14 +1000, Simon K??mpflein wrote:
> >> Hello,
> >>
> >> I have a problem with the new kernel perf tools in kernel 2.6.31.6 (very
> >> nice tools by the way). I like to monitor the cache-misses of a program.
> >> "perf stat -e cache-misses program" works without problems, but I can't
> >> get "perf record -e cache-misses program" working (even running as root
> >> and CONFIG_PERF_COUNTERS=y):
> >>
> >> Error: perfcounter syscall returned with -1 (Operation not supported)
> >> Fatal: No CONFIG_PERF_COUNTERS=y kernel support configured?
> >>
> >> My CPU is a "Intel(R) Pentium(R) M processor 1.80GHz":
> > 
> > Does the bootlog perchance contain the output of?
> > 
> >         if (!cpu_has_apic) {
> >                 pr_info("no APIC, boot with the \"lapic\" boot parameter
> > to force-enable it.\n");
> >                 pr_info("no hardware sampling interrupt available.\n");
> >                 x86_pmu.apic = 0;
> >         }
> 
> Yes. I didn't realize that it has something to do with the Performance
> Counters because I didn't see that it belongs to the Performance Counter
> output.
> Maybe it's a good idea to add another comment making it clear that the
> performance counters are not fully working in this case. Or detect this
> case in "perf-record" and output a message relating to that.

Ok. Mind sending a patch that changes the message so that it explains it 
all to you more clearly?

> > And does booting with the suggested lapic parameter cure your problems?
> > 
> 
> Yes, thank you!

Great.

You might want to send another patch that allows the .config enabling of 
that lapic boot parameter. It sucks to carry around boot parameters.

	Ingo

Re: perf counters: problem with perf record

[Simon Kämpflein]

Ingo Molnar wrote
>> Maybe it's a good idea to add another comment making it clear that the
>> performance counters are not fully working in this case. Or detect this
>> case in "perf-record" and output a message relating to that.
> 
> Ok. Mind sending a patch that changes the message so that it explains it 
> all to you more clearly?

Here's a patch for this:

More accurate error message when "perf record" fails because there is no
APIC support:
---
 tools/perf/builtin-record.c |    4 ++++
 1 files changed, 4 insertions(+), 0 deletions(-)

diff --git a/tools/perf/builtin-record.c b/tools/perf/builtin-record.c
index 04f335e..77b21f1 100644
--- a/tools/perf/builtin-record.c
+++ b/tools/perf/builtin-record.c
@@ -303,6 +303,10 @@ try_again:
                printf("\n");
                error("perfcounter syscall returned with %d (%s)\n",
                        fd[nr_cpu][counter], strerror(err));
+
+               if (attr->type == PERF_TYPE_HARDWARE && err == EOPNOTSUPP)
+                        die("No hardware sampling interrupt available.
No APIC? Boot the kernel with the \"lapic\" boot parameter to
force-enable it.\n");
+
                die("No CONFIG_PERF_EVENTS=y kernel support configured?\n");
                exit(-1);
        }
--
1.5.4.3


> 
>>> And does booting with the suggested lapic parameter cure your problems?
>>>
>> Yes, thank you!
> 
> Great.
> 
> You might want to send another patch that allows the .config enabling of 
> that lapic boot parameter. It sucks to carry around boot parameters.
> 

I'm not sure if this is a good idea. Some systems may crash if you
force-enable it:
http://lkml.indiana.edu/hypermail/linux/kernel/0307.3/1635.html

And I'm also not familiar with the kernel internals on that topic ;)

Simon