Getting PEBS to work

Getting PEBS to work

[Brendan Gregg]

G'Day perf-users,

Different Q than my previous email... This is a physical system (Linux
3.13, Sandy Bridge) that's I think is supposed to have PEBS, but it
doesn't work. Eg:

# perf record -e r412e:p -c 10000 -a -- sleep 1
Error:
'precise' request may not be supported. Try removing 'p' modifier.
sleep: Terminated

Maybe I'm doing it wrong? Here's -vv:

# perf record -vv -e r412e:p -c 10000 -a -- sleep 1
------------------------------------------------------------
perf_event_attr:
  type                4
  size                96
  config              0x412e
  sample_period       10000
  sample_freq         10000
  sample_type         0x87
  read_format         0
  disabled            1    inherit             1
  pinned              0    exclusive           0
  exclude_user        0    exclude_kernel      0
  exclude_hv          0    exclude_idle        0
  mmap                1    comm                1
  freq                0    inherit_stat        0
  enable_on_exec      0    task                0
  watermark           0    precise_ip          1
  mmap_data           0    sample_id_all       1
  exclude_host        0    exclude_guest       1
  excl.callchain_kern 0    excl.callchain_user 0
  mmap2               0
  wakeup_events       0
  wakeup_watermark    0
  bp_type             0
  bp_addr             0
  config1             0
  bp_len              0
  config2             0
  branch_sample_type  0
  sample_regs_user    0
  sample_stack_user   0
------------------------------------------------------------
perf_event_open: pid -1  cpu 0  group_fd -1  flags 0
perf_event_open failed, error -95
Error:
'precise' request may not be supported. Try removing 'p' modifier.
sleep: Terminated

This CPU does have DS:

# cpuid -1 | gerp 'debug store'
      debug store                            = true
      64-bit debug store                      = true
      CPL-qualified debug store               = true

And bit 12 of IA32_MISC_ENABLE is 0, which should mean PEBS is supported:

# rdmsr 0x1a0 -f 12:12
0

Any other way to debug?

I see x86_pmu_hw_config() has been updated since 3.13, so maybe I
should upgrade the kernel... Thanks,

Brendan

Re: Getting PEBS to work

[Brendan Gregg]

G'Day Jim,

Thanks, but no luck here:

# perf record -vv -e r412e:pp -c 10000 -a -- sleep 1
------------------------------------------------------------
perf_event_attr:
  type                4
  size                96
  config              0x412e
  sample_period       10000
  sample_freq         10000
  sample_type         0x87
  read_format         0
  disabled            1    inherit             1
  pinned              0    exclusive           0
  exclude_user        0    exclude_kernel      0
  exclude_hv          0    exclude_idle        0
  mmap                1    comm                1
  freq                0    inherit_stat        0
  enable_on_exec      0    task                0
  watermark           0    precise_ip          2
  mmap_data           0    sample_id_all       1
  exclude_host        0    exclude_guest       1
  excl.callchain_kern 0    excl.callchain_user 0
  mmap2               0
  wakeup_events       0
  wakeup_watermark    0
  bp_type             0
  bp_addr             0
  config1             0
  bp_len              0
  config2             0
  branch_sample_type  0
  sample_regs_user    0
  sample_stack_user   0
------------------------------------------------------------
perf_event_open: pid -1  cpu 0  group_fd -1  flags 0
perf_event_open failed, error -95
Error:
'precise' request may not be supported. Try removing 'p' modifier.
sleep: Terminated

It does have "precise_ip 2", so it parsed.

Oh, I should have said, r412e should be LLC misses. Pretty standard.
Manual says it is PEBS supported.

So you have PEBS working? What's your kernel version and CPU type? Thanks,

Brendan

On Thu, May 19, 2016 at 4:09 PM, jim mauro <jim.mauro@gmail.com> wrote:
> [resending without the link, as it was rejected because of the html]....
>
> Hey Brendan - Looks like you need to append "pp" to the event encoding.
>
> Does "perf record -vv -e r412e:pp -c 10000 -a -- sleep 1" work?
>
>
>
> On Thu, May 19, 2016 at 3:29 PM, Brendan Gregg <brendan.d.gregg@gmail.com>
> wrote:
>>
>> G'Day perf-users,
>>
>> Different Q than my previous email... This is a physical system (Linux
>> 3.13, Sandy Bridge) that's I think is supposed to have PEBS, but it
>> doesn't work. Eg:
>>
>> # perf record -e r412e:p -c 10000 -a -- sleep 1
>> Error:
>> 'precise' request may not be supported. Try removing 'p' modifier.
>> sleep: Terminated
>>
>> Maybe I'm doing it wrong? Here's -vv:
>>
>> # perf record -vv -e r412e:p -c 10000 -a -- sleep 1
>> ------------------------------------------------------------
>> perf_event_attr:
>>   type                4
>>   size                96
>>   config              0x412e
>>   sample_period       10000
>>   sample_freq         10000
>>   sample_type         0x87
>>   read_format         0
>>   disabled            1    inherit             1
>>   pinned              0    exclusive           0
>>   exclude_user        0    exclude_kernel      0
>>   exclude_hv          0    exclude_idle        0
>>   mmap                1    comm                1
>>   freq                0    inherit_stat        0
>>   enable_on_exec      0    task                0
>>   watermark           0    precise_ip          1
>>   mmap_data           0    sample_id_all       1
>>   exclude_host        0    exclude_guest       1
>>   excl.callchain_kern 0    excl.callchain_user 0
>>   mmap2               0
>>   wakeup_events       0
>>   wakeup_watermark    0
>>   bp_type             0
>>   bp_addr             0
>>   config1             0
>>   bp_len              0
>>   config2             0
>>   branch_sample_type  0
>>   sample_regs_user    0
>>   sample_stack_user   0
>> ------------------------------------------------------------
>> perf_event_open: pid -1  cpu 0  group_fd -1  flags 0
>> perf_event_open failed, error -95
>> Error:
>> 'precise' request may not be supported. Try removing 'p' modifier.
>> sleep: Terminated
>>
>> This CPU does have DS:
>>
>> # cpuid -1 | gerp 'debug store'
>>       debug store                            = true
>>       64-bit debug store                      = true
>>       CPL-qualified debug store               = true
>>
>> And bit 12 of IA32_MISC_ENABLE is 0, which should mean PEBS is supported:
>>
>> # rdmsr 0x1a0 -f 12:12
>> 0
>>
>> Any other way to debug?
>>
>> I see x86_pmu_hw_config() has been updated since 3.13, so maybe I
>> should upgrade the kernel... Thanks,
>>
>> Brendan
>> --
>> To unsubscribe from this list: send the line "unsubscribe
>> linux-perf-users" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
>

Re: Getting PEBS to work

[Brendan Gregg]

On Thu, May 19, 2016 at 4:53 PM, jim mauro <jim.mauro@gmail.com> wrote:
> Hey Brendan -
>
> D'oh! No mate, I do not have PEBS working.
>
> # perf record -e r412e:pp -c 10000 -a -- sleep 1
> Error:
> The sys_perf_event_open() syscall returned with 22 (Invalid argument) for
> event (r412e:pp).
>
> This is an Ivy Bridge system, 3.8.13 kernel. Not yet sure why it's not
> working, digging in now....
>
> As an aside, I'm looking at Table 18-27 PEBS Performance Events for Intel
> Microarchitecture Code Name Sandy Bridge in the
> Intel manual, and I do not see LLC misses as a supported PEBS event.

Ah, thanks, hm, wonder where I saw LLC miss as PEBS then.

Well, INST_RETIRED should be a safer test, as that's in table 18-27:

# perf record -e r01c0:p -c 100000000 -a -- sleep 1
Error:
'precise' request may not be supported. Try removing 'p' modifier.
sleep: Terminated

Brendan

Re: Getting PEBS to work

[Brendan Gregg]

On Thu, May 19, 2016 at 5:09 PM, jim mauro <jim.mauro@gmail.com> wrote:
> Hey Brendan - This seems to be working:
>
> # perf stat -e r4301c0:pp -a -- sleep 10
>
>  Performance counter stats for 'system wide':
>
>           80845274      r4301c0:pp
>
>       10.001588491 seconds time elapsed
>
> It's this event:
>
> INST_RETIRED.PREC_DIST:Precise instruction retired event with HW to reduce
> effect of PEBS shadow in IP distribution
> cpu/umask=0x01,event=0xC0,name=INST_RETIRED.PREC_DIST/

Woah, you typed r4301c0. :) Dunno what that is. Doesn't work for me anyway:

# perf stat -e r4301c0:pp -a -- sleep 10
^Csleep: Interrupt

 Performance counter stats for 'system wide':

   <not supported> r4301c0:pp

       2.297987255 seconds time elapsed

Nor did r01c0. But pretty interesting you got PEBS output, and on
Linux 3.8. So maybe my 3.13 isn't the problem...

Brendan

Re: Getting PEBS to work

[Vince Weaver]

On Thu, 19 May 2016, Brendan Gregg wrote:

> On Thu, May 19, 2016 at 5:09 PM, jim mauro <jim.mauro@gmail.com> wrote:
>
> Nor did r01c0. But pretty interesting you got PEBS output, and on
> Linux 3.8. So maybe my 3.13 isn't the problem...

does your dmesg say anything?  If I recall there are sandybridge machines 
with PEBS disabled due to errata.  Usually you need to update your 
CPU firmware to get it working.

Vince

Re: Getting PEBS to work

[Brendan Gregg]

On Thu, May 19, 2016 at 5:47 PM, Vince Weaver <vincent.weaver@maine.edu> wrote:
> On Thu, 19 May 2016, Brendan Gregg wrote:
>
>> On Thu, May 19, 2016 at 5:09 PM, jim mauro <jim.mauro@gmail.com> wrote:
>>
>> Nor did r01c0. But pretty interesting you got PEBS output, and on
>> Linux 3.8. So maybe my 3.13 isn't the problem...
>
> does your dmesg say anything?  If I recall there are sandybridge machines
> with PEBS disabled due to errata.  Usually you need to update your
> CPU firmware to get it working.

# dmesg | egrep -i 'pmu|pmc|pebs'
[    0.381095] Performance Events: PEBS fmt1+, 16-deep LBR,
SandyBridge events, full-width counters, Intel PMU driver.
[    0.381101] perf_event_intel: PEBS disabled due to CPU errata,
please upgrade microcode
[    0.395348] NMI watchdog: enabled on all CPUs, permanently consumes
one hw-PMU counter.
[    1.231836] nouveau  [     PMC][0000:01:00.0] MSI interrupts enabled

Gah, it was there all along!

Thanks Vince. I hope future PEBS explorers find the commands in this
thread useful!

Now I have to update some microcode...

Brendan

Re: Getting PEBS to work

[Arnaldo Carvalho de Melo]

Em Thu, May 19, 2016 at 06:06:36PM -0700, Brendan Gregg escreveu:
> On Thu, May 19, 2016 at 5:47 PM, Vince Weaver <vincent.weaver@maine.edu> wrote:
> > On Thu, 19 May 2016, Brendan Gregg wrote:
> >> On Thu, May 19, 2016 at 5:09 PM, jim mauro <jim.mauro@gmail.com> wrote:

> >> Nor did r01c0. But pretty interesting you got PEBS output, and on
> >> Linux 3.8. So maybe my 3.13 isn't the problem...

> > does your dmesg say anything?  If I recall there are sandybridge machines
> > with PEBS disabled due to errata.  Usually you need to update your
> > CPU firmware to get it working.
> 
> # dmesg | egrep -i 'pmu|pmc|pebs'
> [    0.381095] Performance Events: PEBS fmt1+, 16-deep LBR,
> SandyBridge events, full-width counters, Intel PMU driver.
> [    0.381101] perf_event_intel: PEBS disabled due to CPU errata,
> please upgrade microcode
> [    0.395348] NMI watchdog: enabled on all CPUs, permanently consumes
> one hw-PMU counter.
> [    1.231836] nouveau  [     PMC][0000:01:00.0] MSI interrupts enabled
> 
> Gah, it was there all along!

Wish this was exported by the kernel somehow, then tooling could emit a
sensible message :-\

- Arnaldo
 
> Thanks Vince. I hope future PEBS explorers find the commands in this
> thread useful!
> 
> Now I have to update some microcode...

Re: Getting PEBS to work

[Peter Zijlstra]

On Mon, May 23, 2016 at 11:05:39AM -0300, Arnaldo Carvalho de Melo wrote:
> > # dmesg | egrep -i 'pmu|pmc|pebs'
> > [    0.381095] Performance Events: PEBS fmt1+, 16-deep LBR,
> > SandyBridge events, full-width counters, Intel PMU driver.
> > [    0.381101] perf_event_intel: PEBS disabled due to CPU errata,
> > please upgrade microcode
> > [    0.395348] NMI watchdog: enabled on all CPUs, permanently consumes
> > one hw-PMU counter.
> > [    1.231836] nouveau  [     PMC][0000:01:00.0] MSI interrupts enabled
> > 
> > Gah, it was there all along!
> 
> Wish this was exported by the kernel somehow, then tooling could emit a
> sensible message :-\

You mean something like so?

--- a/arch/x86/events/core.c
+++ b/arch/x86/events/core.c
@@ -1716,6 +1716,8 @@ ssize_t x86_event_sysfs_show(char *page,
 	return ret;
 }
 
+static struct attribute *x86_pmu_attrs[];
+
 static int __init init_hw_perf_events(void)
 {
 	struct x86_pmu_quirk *quirk;
@@ -1779,6 +1781,9 @@ static int __init init_hw_perf_events(vo
 			x86_pmu_events_group.attrs = tmp;
 	}
 
+	if (!x86_pmu.pebs)
+		x86_pmu_attrs[1] = NULL;
+
 	pr_info("... version:                %d\n",     x86_pmu.version);
 	pr_info("... bit width:              %d\n",     x86_pmu.cntval_bits);
 	pr_info("... generic registers:      %d\n",     x86_pmu.num_counters);
@@ -2120,10 +2125,20 @@ static ssize_t set_attr_rdpmc(struct dev
 	return count;
 }
 
+static ssize_t pebs_show(struct device *cdev,
+			 struct device_attribute *attr,
+			 char *buf)
+{
+	return snprintf(buf, 40, "%d\n",
+			!!(x86_pmu.pebs_active && !x86_pmu.pebs_broken));
+}
+
 static DEVICE_ATTR(rdpmc, S_IRUSR | S_IWUSR, get_attr_rdpmc, set_attr_rdpmc);
+static DEVICE_ATTR_RO(pebs);
 
 static struct attribute *x86_pmu_attrs[] = {
 	&dev_attr_rdpmc.attr,
+	&dev_attr_pebs.attr,
 	NULL,
 };
 

Re: Getting PEBS to work

[Arnaldo Carvalho de Melo]

Em Tue, May 24, 2016 at 12:17:26PM +0200, Peter Zijlstra escreveu:
> On Mon, May 23, 2016 at 11:05:39AM -0300, Arnaldo Carvalho de Melo wrote:
> > > # dmesg | egrep -i 'pmu|pmc|pebs'
> > > [    0.381095] Performance Events: PEBS fmt1+, 16-deep LBR,
> > > SandyBridge events, full-width counters, Intel PMU driver.
> > > [    0.381101] perf_event_intel: PEBS disabled due to CPU errata,
> > > please upgrade microcode
> > > [    0.395348] NMI watchdog: enabled on all CPUs, permanently consumes
> > > one hw-PMU counter.
> > > [    1.231836] nouveau  [     PMC][0000:01:00.0] MSI interrupts enabled
> > > 
> > > Gah, it was there all along!
> > 
> > Wish this was exported by the kernel somehow, then tooling could emit a
> > sensible message :-\
> 
> You mean something like so?

Probably, I'll try prototyping the tooling part later.

Thanks!

- Arnaldo
 
> --- a/arch/x86/events/core.c
> +++ b/arch/x86/events/core.c
> @@ -1716,6 +1716,8 @@ ssize_t x86_event_sysfs_show(char *page,
>  	return ret;
>  }
>  
> +static struct attribute *x86_pmu_attrs[];
> +
>  static int __init init_hw_perf_events(void)
>  {
>  	struct x86_pmu_quirk *quirk;
> @@ -1779,6 +1781,9 @@ static int __init init_hw_perf_events(vo
>  			x86_pmu_events_group.attrs = tmp;
>  	}
>  
> +	if (!x86_pmu.pebs)
> +		x86_pmu_attrs[1] = NULL;
> +
>  	pr_info("... version:                %d\n",     x86_pmu.version);
>  	pr_info("... bit width:              %d\n",     x86_pmu.cntval_bits);
>  	pr_info("... generic registers:      %d\n",     x86_pmu.num_counters);
> @@ -2120,10 +2125,20 @@ static ssize_t set_attr_rdpmc(struct dev
>  	return count;
>  }
>  
> +static ssize_t pebs_show(struct device *cdev,
> +			 struct device_attribute *attr,
> +			 char *buf)
> +{
> +	return snprintf(buf, 40, "%d\n",
> +			!!(x86_pmu.pebs_active && !x86_pmu.pebs_broken));
> +}
> +
>  static DEVICE_ATTR(rdpmc, S_IRUSR | S_IWUSR, get_attr_rdpmc, set_attr_rdpmc);
> +static DEVICE_ATTR_RO(pebs);
>  
>  static struct attribute *x86_pmu_attrs[] = {
>  	&dev_attr_rdpmc.attr,
> +	&dev_attr_pebs.attr,
>  	NULL,
>  };
>  

Re: Getting PEBS to work

[Andi Kleen]

Peter Zijlstra <peterz@infradead.org> writes:

> On Mon, May 23, 2016 at 11:05:39AM -0300, Arnaldo Carvalho de Melo wrote:
>> > # dmesg | egrep -i 'pmu|pmc|pebs'
>> > [    0.381095] Performance Events: PEBS fmt1+, 16-deep LBR,
>> > SandyBridge events, full-width counters, Intel PMU driver.
>> > [    0.381101] perf_event_intel: PEBS disabled due to CPU errata,
>> > please upgrade microcode
>> > [    0.395348] NMI watchdog: enabled on all CPUs, permanently consumes
>> > one hw-PMU counter.
>> > [    1.231836] nouveau  [     PMC][0000:01:00.0] MSI interrupts enabled
>> > 
>> > Gah, it was there all along!
>> 
>> Wish this was exported by the kernel somehow, then tooling could emit a
>> sensible message :-\
>
> You mean something like so?

Also the disable/microcode fix is really only needed for PEBS events
with cmask != 0, such as cycles:pp. Everything else in PEBS works fine
on Sandy Bridge. 

-Andi

-- 
ak@linux.intel.com -- Speaking for myself only