Re: perf software events broken in containers

[William Cohen <wcohen@redhat.com>]

On 03/22/2017 03:59 PM, Brendan Gregg wrote:
> G'Day Will,
> 
> On Wed, Mar 22, 2017 at 12:15 PM, William Cohen <wcohen@redhat.com> wrote:
>> On 03/22/2017 02:24 PM, Brendan Gregg wrote:
>>> G'Day,
> [...]
>>>
>>> The kernel is returning errno 1 to the sys_perf_event_open() call in
>>> __perf_evsel__open(). I'm trying to find out which kernel function
>>> throws the EPERM, but almost nothing is tracable via ftrace/kprobes.
>>> It's pretty annoying... Thanks for any ideas,
>>
>> Hi Brendan,
>>
>> Several years ago I had a problem with the perf_event_open not working on arm machines.  I looked at the kernel source code and just kept putting systemtap one-liners like the following on the various functions used by the perf_event_open to see which function was the one returning the error code:
>>
>> stap -v -e 'probe kernel.function("idr_find").return {printf("%s %s 0x%x\n", pn(), $$parms$, $return)}'
>>
> 
> Right, I've been doing that with ftrace/kprobes/bcc/BPF... Many of the
> functions aren't visible, though, I suspect inlined.
> 

Hi Brendan,

Yes, the inlined functions are going to make it hard to get some of the return values.  However, you can probably narrow down the boundary between executed functions and not executed functions. Would the Intel Processor Trace (http://halobates.de/blog/) be usable to help track down this problem?

-Will