Re: perf software events broken in containers

[Brendan Gregg <brendan.d.gregg@gmail.com>]

On Wed, Mar 22, 2017 at 1:29 PM, William Cohen <wcohen@redhat.com> wrote:
> On 03/22/2017 03:59 PM, Brendan Gregg wrote:
>> G'Day Will,
>>
>> On Wed, Mar 22, 2017 at 12:15 PM, William Cohen <wcohen@redhat.com> wrote:
>>> On 03/22/2017 02:24 PM, Brendan Gregg wrote:
>>>> G'Day,
>> [...]
>>>>
>>>> The kernel is returning errno 1 to the sys_perf_event_open() call in
>>>> __perf_evsel__open(). I'm trying to find out which kernel function
>>>> throws the EPERM, but almost nothing is tracable via ftrace/kprobes.
>>>> It's pretty annoying... Thanks for any ideas,
>>>
>>> Hi Brendan,
>>>
>>> Several years ago I had a problem with the perf_event_open not working on arm machines.  I looked at the kernel source code and just kept putting systemtap one-liners like the following on the various functions used by the perf_event_open to see which function was the one returning the error code:
>>>
>>> stap -v -e 'probe kernel.function("idr_find").return {printf("%s %s 0x%x\n", pn(), $$parms$, $return)}'
>>>
>>
>> Right, I've been doing that with ftrace/kprobes/bcc/BPF... Many of the
>> functions aren't visible, though, I suspect inlined.
>>
>
> Hi Brendan,
>
> Yes, the inlined functions are going to make it hard to get some of the return values.  However, you can probably narrow down the boundary between executed functions and not executed functions. Would the Intel Processor Trace (http://halobates.de/blog/) be usable to help track down this problem?
>

I'm in the cloud, so I usually don't even have PMCs. :)

Looks like the issue is a change with docker, I'd guess related to
this syscall whitelisting. Doing a "docker run --cap-add cap_sys_admin
..." results in a container that can run perf (but has sysadmin
privilege always on), although we still need the --privileged for it
to work fully. Previously we could run normal containers and run some
ad hoc privileged shells only with "docker exec -it --privileged ...
bash" for running perf.

Weirdly, the container claims it has cap_sys_admin either way, but one
way it doesn't work and the other it does. I filed
https://github.com/docker/docker/issues/32018

We also found that docker explicitly blocks perf_event_open() by default[1]:

"perf_event_open    Tracing/profiling syscall, which could leak a lot
of information on the host."

[1] https://docs.docker.com/engine/security/seccomp/#significant-syscalls-blocked-by-the-default-profile

Brendan