Using perf with cgroups and containers

[William Cohen <wcohen@redhat.com>]

Hi,

I have been looking at how perf supports cgroups and containers.  The
"-G" option allows limiting the data collected to a particular cgroup.
Thus, one can use the option to collect some information about a
particular cgroup with something like:

$ sudo perf stat -a -e cycles  -G machine.slice/machine-qemu\\x2drhel7\\x2dx86_64.scope -e instructions -G machine.slice/machine-qemu\\x2drhel7\\x2dx86_64.scope -- sleep 1

 Performance counter stats for 'system wide':

         9,668,237      cycles                    machine.slice/machine-qemu\x2drhel7\x2dx86_64.scope [82.28%]
         4,685,886      instructions              machine.slice/machine-qemu\x2drhel7\x2dx86_64.scope #    0.48  insns per cycle         [82.28%]

       1.001359839 seconds time elapsed

However, this approach seems to be awkward.  It requires specifying
the cgroup for each event.  It also requires the system-wide option
("-a") to get information for all the tasks in the cgroup and
superuser privileges.  Thus, even if all the tasks are owned by the
user running the perf command, the command still needs superuser
privileges.

Another limitation is when within a container there doesn't seem to be
a way of doing the equivalent to a "perf record -a ..." to collect
related to that container.  When running within a container going to
get something like the following:

# perf stat -a ls
Error:
You may not have permission to collect system-wide stats.
Consider tweaking /proc/sys/kernel/perf_event_paranoid:
 -1 - Not paranoid at all
  0 - Disallow raw tracepoint access for unpriv
  1 - Disallow cpu events for unpriv
  2 - Disallow kernel profiling for unpriv


There is a middle ground between monitoring a process/set of processes
and monitoring the entire machine that perf could do a better job of.
Have three broad categories to scope the data collection: pid, cgroup,
and container.

For pid perf can be monitoring itself or a set of processes.  perf has
checks to make sure that the monitoring process has permission to
monitor the other processes. 

For cgroup monitoring perf implementation only works for systemwide
monitoring "-a".  Why should someone need to specify "-a" when they
specify the cgroup to monitor?  It seems like this should operater
much more like the selection of processes to monitor.

For containers one might want to either monitor a container from
outside for system health or from the inside while doing development.
Currently perf can monitor from outside but isn't able to monitor from
inside.

Can something be done to improve the usability of perf for cgroups and
containers?

-Will