[perf tool] cgroup support broken on Debian?

[perf tool] cgroup support broken on Debian?

[Vince Weaver]

Hello

has anyone tested the perf tool cgroup support recently?

I was trying to get it working with a command like
	sudo perf stat -a -e cycles:u,cycles:u,cycles:u -G systemd -- sleep 1

and it just failed by unhelfully dumping the "-G" help text.
Once I added a lot of extra debug printfs to tools/perf/util/cgroup.c 
things became a little clearer.

First, you apparently need "perf_event" passed as a mount option to the 
cgroup or you cannot attach perf to it (should perf be modified to 
print a warning in this case rather than just printing the unhelpful
helf text?)

Secondly, the cgroup mount point detection completely fails on my debian 
box.  On my machine /proc/mounts has this:

  ...
  none /sys/fs/cgroup tmpfs rw,relatime,size=4k,mode=755 0 0
  systemd /sys/fs/cgroup/systemd cgroup rw,nosuid,nodev,noexec,relatime,perf_event,name=systemd 0 0

The current perf code looks for "cgroup" in the type field to find the 
root cgroupfs tree.  This fails because as seen above on my machine the
cgroup mount has type tmpfs.  And when it finds
/sys/fs/cgroup/systemd as the root it tacks the name onto the end
(/sys/fs/cgroup/systemd/systemd) which obviously doesn't exist.

Once I hack the code to avoid that I do finally get some cgroup readings.

I was checking if this was a known problem, a Debian issue, or what...

Vince

Re: [perf tool] cgroup support broken on Debian?

[Arnaldo Carvalho de Melo]

Em Mon, Dec 15, 2014 at 04:07:46PM -0500, Vince Weaver escreveu:
> Hello
> 
> has anyone tested the perf tool cgroup support recently?
> 
> I was trying to get it working with a command like
> 	sudo perf stat -a -e cycles:u,cycles:u,cycles:u -G systemd -- sleep 1
> 
> and it just failed by unhelfully dumping the "-G" help text.
> Once I added a lot of extra debug printfs to tools/perf/util/cgroup.c 
> things became a little clearer.
> 
> First, you apparently need "perf_event" passed as a mount option to the 
> cgroup or you cannot attach perf to it (should perf be modified to 
> print a warning in this case rather than just printing the unhelpful
> helf text?)

yes, please
 
> Secondly, the cgroup mount point detection completely fails on my debian 
> box.  On my machine /proc/mounts has this:
> 
>   ...
>   none /sys/fs/cgroup tmpfs rw,relatime,size=4k,mode=755 0 0
>   systemd /sys/fs/cgroup/systemd cgroup rw,nosuid,nodev,noexec,relatime,perf_event,name=systemd 0 0
> 
> The current perf code looks for "cgroup" in the type field to find the 
> root cgroupfs tree.  This fails because as seen above on my machine the
> cgroup mount has type tmpfs.  And when it finds
> /sys/fs/cgroup/systemd as the root it tacks the name onto the end
> (/sys/fs/cgroup/systemd/systemd) which obviously doesn't exist.
> 
> Once I hack the code to avoid that I do finally get some cgroup readings.
> 
> I was checking if this was a known problem, a Debian issue, or what...

Its just that the cgroup support is rough, patches are welcome to
improve the situation.

BTW, here its fedora20, same problem:

[root@zoo ~]# mount | grep cgroup
tmpfs on /sys/fs/cgroup type tmpfs
(rw,nosuid,nodev,noexec,seclabel,mode=755)
cgroup on /sys/fs/cgroup/systemd type cgroup
(rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd)

- Arnaldo

Re: [perf tool] cgroup support broken on Debian?

[Stephane Eranian]

Vince,

On Mon, Dec 15, 2014 at 11:01 PM, Arnaldo Carvalho de Melo
<arnaldo.melo@gmail.com> wrote:
> Em Mon, Dec 15, 2014 at 04:07:46PM -0500, Vince Weaver escreveu:
>> Hello
>>
>> has anyone tested the perf tool cgroup support recently?
>>
>> I was trying to get it working with a command like
>>       sudo perf stat -a -e cycles:u,cycles:u,cycles:u -G systemd -- sleep 1
>>
It fails for me on Ubuntu Trusty as well. I think they have changed
the way cgroup
fs is visible. The cgroup file system type is not there anymore. They are using
tmpfs which is not ideal to detect just cgroup. Looks like now, we have to look
at the mount point which is flaky.


>> and it just failed by unhelfully dumping the "-G" help text.
>> Once I added a lot of extra debug printfs to tools/perf/util/cgroup.c
>> things became a little clearer.
>>
>> First, you apparently need "perf_event" passed as a mount option to the
>> cgroup or you cannot attach perf to it (should perf be modified to
>> print a warning in this case rather than just printing the unhelpful
>> helf text?)
>
> yes, please
>
Yes, the error needs to be improved or even added. The following common
errors are encountered:

- -G option requires -a (system-wide), otherwise it does not work at all
- -G option must always be specified AFTER the ALL event list
- -G and explicit event groups is not well supported by the tool yet.
An event group must have the same cgroup.
- Cgroups are specified per event
- -G cgroup order follows the event order: -e e1, e2, e3 -G g1,g2,g3:
g1->e1, g2->e2, g3->e3
- It is possible to indicate no cgroup with -G: -e e1,e2,e3 -G g1,,g3
using empty group (,,)


>> Secondly, the cgroup mount point detection completely fails on my debian
>> box.  On my machine /proc/mounts has this:
>>
>>   ...
>>   none /sys/fs/cgroup tmpfs rw,relatime,size=4k,mode=755 0 0

Yes, my Ubuntu system too.

>>   systemd /sys/fs/cgroup/systemd cgroup rw,nosuid,nodev,noexec,relatime,perf_event,name=systemd 0 0
>>
>> The current perf code looks for "cgroup" in the type field to find the
>> root cgroupfs tree.  This fails because as seen above on my machine the
>> cgroup mount has type tmpfs.  And when it finds
>> /sys/fs/cgroup/systemd as the root it tacks the name onto the end
>> (/sys/fs/cgroup/systemd/systemd) which obviously doesn't exist.
>>
>> Once I hack the code to avoid that I do finally get some cgroup readings.
>>
>> I was checking if this was a known problem, a Debian issue, or what...
>
Obviously, the way cgroups are exposed has changed since I wrote the code
an it needs to update. Thanks for looking into this.

> Its just that the cgroup support is rough, patches are welcome to
> improve the situation.
>
> BTW, here its fedora20, same problem:
>
> [root@zoo ~]# mount | grep cgroup
> tmpfs on /sys/fs/cgroup type tmpfs
> (rw,nosuid,nodev,noexec,seclabel,mode=755)
> cgroup on /sys/fs/cgroup/systemd type cgroup
> (rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd)
>
> - Arnaldo

Re: [perf tool] cgroup support broken on Debian?

[Vince Weaver]

On Mon, 15 Dec 2014, Stephane Eranian wrote:
> On Mon, Dec 15, 2014 at 11:01 PM, Arnaldo Carvalho de Melo <arnaldo.melo@gmail.com> wrote:

> fs is visible. The cgroup file system type is not there anymore. They are using
> tmpfs which is not ideal to detect just cgroup. Looks like now, we have to look
> at the mount point which is flaky.

The trivial fix is to just always assume things will be under
	/sys/fs/cgroup
which looks like to be the new official mount point.

This will break on older systems though, or systems that mount cgroupfs in 
multiple locations.

Another alternative is to change the interface to require the full 
cgroupfs pathname as an argument to -G

> Yes, the error needs to be improved or even added. The following common
> errors are encountered:
> 
> - -G option requires -a (system-wide), otherwise it does not work at all
> - -G option must always be specified AFTER the ALL event list
> - -G and explicit event groups is not well supported by the tool yet.
> An event group must have the same cgroup.
> - Cgroups are specified per event
> - -G cgroup order follows the event order: -e e1, e2, e3 -G g1,g2,g3:
> g1->e1, g2->e2, g3->e3
> - It is possible to indicate no cgroup with -G: -e e1,e2,e3 -G g1,,g3
> using empty group (,,)

I'll see if I can get a patch together that impements this.

Vince

Re: [perf tool] cgroup support broken on Debian?

[Stephane Eranian]

Vince,

On Tue, Dec 16, 2014 at 12:17 PM, Vince Weaver <vincent.weaver@maine.edu> wrote:
> On Mon, 15 Dec 2014, Stephane Eranian wrote:
>> On Mon, Dec 15, 2014 at 11:01 PM, Arnaldo Carvalho de Melo <arnaldo.melo@gmail.com> wrote:
>
>> fs is visible. The cgroup file system type is not there anymore. They are using
>> tmpfs which is not ideal to detect just cgroup. Looks like now, we have to look
>> at the mount point which is flaky.
>
> The trivial fix is to just always assume things will be under
>         /sys/fs/cgroup
> which looks like to be the new official mount point.
>
> This will break on older systems though, or systems that mount cgroupfs in
> multiple locations.
>
> Another alternative is to change the interface to require the full
> cgroupfs pathname as an argument to -G
>
That's not user friendly!
We need to detect where this is mounted to.
I think we can try the tmpfs mount points. Just need to then check if
a file entry
that can only exist for cgroups is there, such as cgroups.procs.

>> Yes, the error needs to be improved or even added. The following common
>> errors are encountered:
>>
>> - -G option requires -a (system-wide), otherwise it does not work at all
>> - -G option must always be specified AFTER the ALL event list
>> - -G and explicit event groups is not well supported by the tool yet.
>> An event group must have the same cgroup.
>> - Cgroups are specified per event
>> - -G cgroup order follows the event order: -e e1, e2, e3 -G g1,g2,g3:
>> g1->e1, g2->e2, g3->e3
>> - It is possible to indicate no cgroup with -G: -e e1,e2,e3 -G g1,,g3
>> using empty group (,,)
>
> I'll see if I can get a patch together that impements this.
>
> Vince

Re: [perf tool] cgroup support broken on Debian?

[Zefan Li]

On 2014/12/17 1:17, Vince Weaver wrote:
> On Mon, 15 Dec 2014, Stephane Eranian wrote:
>> On Mon, Dec 15, 2014 at 11:01 PM, Arnaldo Carvalho de Melo <arnaldo.melo@gmail.com> wrote:
> 
>> fs is visible. The cgroup file system type is not there anymore. They are using
>> tmpfs which is not ideal to detect just cgroup. Looks like now, we have to look
>> at the mount point which is flaky.
> 
> The trivial fix is to just always assume things will be under
> 	/sys/fs/cgroup
> which looks like to be the new official mount point.
> 
> This will break on older systems though, or systems that mount cgroupfs in 
> multiple locations.
> 
> Another alternative is to change the interface to require the full 
> cgroupfs pathname as an argument to -G
> 

What's the problem here?

none /sys/fs/cgroup tmpfs rw,relatime,size=4k,mode=755 0 0
systemd /sys/fs/cgroup/systemd cgroup rw,nosuid,nodev,noexec,relatime,perf_event,name=systemd 0 0

cgroup is mounted in /sys/fs/cgroup/systemd, so you should pass '/' to the -G argument:

	sudo perf stat -a -e cycles:u,cycles:u,cycles:u -G / -- sleep 1

Re: [perf tool] cgroup support broken on Debian?

[Stephane Eranian]

On Wed, Dec 17, 2014 at 3:20 AM, Zefan Li <lizefan@huawei.com> wrote:
> On 2014/12/17 1:17, Vince Weaver wrote:
>> On Mon, 15 Dec 2014, Stephane Eranian wrote:
>>> On Mon, Dec 15, 2014 at 11:01 PM, Arnaldo Carvalho de Melo <arnaldo.melo@gmail.com> wrote:
>>
>>> fs is visible. The cgroup file system type is not there anymore. They are using
>>> tmpfs which is not ideal to detect just cgroup. Looks like now, we have to look
>>> at the mount point which is flaky.
>>
>> The trivial fix is to just always assume things will be under
>>       /sys/fs/cgroup
>> which looks like to be the new official mount point.
>>
>> This will break on older systems though, or systems that mount cgroupfs in
>> multiple locations.
>>
>> Another alternative is to change the interface to require the full
>> cgroupfs pathname as an argument to -G
>>
>
> What's the problem here?
>
> none /sys/fs/cgroup tmpfs rw,relatime,size=4k,mode=755 0 0
> systemd /sys/fs/cgroup/systemd cgroup rw,nosuid,nodev,noexec,relatime,perf_event,name=systemd 0 0
>
> cgroup is mounted in /sys/fs/cgroup/systemd, so you should pass '/' to the -G argument:
>
Is that the only mountpoint possible?
The tool needs to detect a valid mount point to locate the named cgroup.
That's assuming that if I create cgroup foo, then it appears under
//sys/fs/cgroup/systemd/foo

>         sudo perf stat -a -e cycles:u,cycles:u,cycles:u -G / -- sleep 1
>

Re: [perf tool] cgroup support broken on Debian?

[Zefan Li]

On 2014/12/17 10:29, Stephane Eranian wrote:
> On Wed, Dec 17, 2014 at 3:20 AM, Zefan Li <lizefan@huawei.com> wrote:
>> On 2014/12/17 1:17, Vince Weaver wrote:
>>> On Mon, 15 Dec 2014, Stephane Eranian wrote:
>>>> On Mon, Dec 15, 2014 at 11:01 PM, Arnaldo Carvalho de Melo <arnaldo.melo@gmail.com> wrote:
>>>
>>>> fs is visible. The cgroup file system type is not there anymore. They are using
>>>> tmpfs which is not ideal to detect just cgroup. Looks like now, we have to look
>>>> at the mount point which is flaky.
>>>
>>> The trivial fix is to just always assume things will be under
>>>       /sys/fs/cgroup
>>> which looks like to be the new official mount point.
>>>
>>> This will break on older systems though, or systems that mount cgroupfs in
>>> multiple locations.
>>>
>>> Another alternative is to change the interface to require the full
>>> cgroupfs pathname as an argument to -G
>>>
>>
>> What's the problem here?
>>
>> none /sys/fs/cgroup tmpfs rw,relatime,size=4k,mode=755 0 0
>> systemd /sys/fs/cgroup/systemd cgroup rw,nosuid,nodev,noexec,relatime,perf_event,name=systemd 0 0
>>
>> cgroup is mounted in /sys/fs/cgroup/systemd, so you should pass '/' to the -G argument:
>>
> Is that the only mountpoint possible?
> The tool needs to detect a valid mount point to locate the named cgroup.
> That's assuming that if I create cgroup foo, then it appears under
> //sys/fs/cgroup/systemd/foo
> 

There can be only one cgroupfs mountpoint which has perf_event subsystem
attached to it.

So for this setup:

mount -t tmpfs /sys/fs/cgroup
mkdir /sys/fs/cgroup/memory
mkdir /sys/fs/cgroup/perf
mount -t cgroup -o memory memcg /sys/fs/cgroup/memory
mount -t cgroup -o perf_event perf /sys/fs/cgroup/perf

The perf tool will locate the mountpoint as /sys/fs/cgroup/perf.

Re: [perf tool] cgroup support broken on Debian?

[Stephane Eranian]

On Tue, Dec 16, 2014 at 10:02 PM, Zefan Li <lizefan@huawei.com> wrote:
> On 2014/12/17 10:29, Stephane Eranian wrote:
>> On Wed, Dec 17, 2014 at 3:20 AM, Zefan Li <lizefan@huawei.com> wrote:
>>> On 2014/12/17 1:17, Vince Weaver wrote:
>>>> On Mon, 15 Dec 2014, Stephane Eranian wrote:
>>>>> On Mon, Dec 15, 2014 at 11:01 PM, Arnaldo Carvalho de Melo <arnaldo.melo@gmail.com> wrote:
>>>>
>>>>> fs is visible. The cgroup file system type is not there anymore. They are using
>>>>> tmpfs which is not ideal to detect just cgroup. Looks like now, we have to look
>>>>> at the mount point which is flaky.
>>>>
>>>> The trivial fix is to just always assume things will be under
>>>>       /sys/fs/cgroup
>>>> which looks like to be the new official mount point.
>>>>
>>>> This will break on older systems though, or systems that mount cgroupfs in
>>>> multiple locations.
>>>>
>>>> Another alternative is to change the interface to require the full
>>>> cgroupfs pathname as an argument to -G
>>>>
>>>
>>> What's the problem here?
>>>
>>> none /sys/fs/cgroup tmpfs rw,relatime,size=4k,mode=755 0 0
>>> systemd /sys/fs/cgroup/systemd cgroup rw,nosuid,nodev,noexec,relatime,perf_event,name=systemd 0 0
>>>
>>> cgroup is mounted in /sys/fs/cgroup/systemd, so you should pass '/' to the -G argument:
>>>
>> Is that the only mountpoint possible?
>> The tool needs to detect a valid mount point to locate the named cgroup.
>> That's assuming that if I create cgroup foo, then it appears under
>> //sys/fs/cgroup/systemd/foo
>>
>
> There can be only one cgroupfs mountpoint which has perf_event subsystem
> attached to it.
>
> So for this setup:
>
> mount -t tmpfs /sys/fs/cgroup
> mkdir /sys/fs/cgroup/memory
> mkdir /sys/fs/cgroup/perf
> mount -t cgroup -o memory memcg /sys/fs/cgroup/memory
> mount -t cgroup -o perf_event perf /sys/fs/cgroup/perf
>
> The perf tool will locate the mountpoint as /sys/fs/cgroup/perf.
>
You can mount the cgroup fs for perf_event multiple times. I
tried and it works, though it is useless.

Vince, I think the correct way to detect which entry is for perf_event
is to look for filesystem type cgroup and option perf_event. It cannot
be anything else. First match is good enough.

Re: [perf tool] cgroup support broken on Debian?

[Zefan Li]

>>>> What's the problem here?
>>>>
>>>> none /sys/fs/cgroup tmpfs rw,relatime,size=4k,mode=755 0 0
>>>> systemd /sys/fs/cgroup/systemd cgroup rw,nosuid,nodev,noexec,relatime,perf_event,name=systemd 0 0
>>>>
>>>> cgroup is mounted in /sys/fs/cgroup/systemd, so you should pass '/' to the -G argument:
>>>>
>>> Is that the only mountpoint possible?
>>> The tool needs to detect a valid mount point to locate the named cgroup.
>>> That's assuming that if I create cgroup foo, then it appears under
>>> //sys/fs/cgroup/systemd/foo
>>>
>>
>> There can be only one cgroupfs mountpoint which has perf_event subsystem
>> attached to it.
>>
>> So for this setup:
>>
>> mount -t tmpfs /sys/fs/cgroup
>> mkdir /sys/fs/cgroup/memory
>> mkdir /sys/fs/cgroup/perf
>> mount -t cgroup -o memory memcg /sys/fs/cgroup/memory
>> mount -t cgroup -o perf_event perf /sys/fs/cgroup/perf
>>
>> The perf tool will locate the mountpoint as /sys/fs/cgroup/perf.
>>
> You can mount the cgroup fs for perf_event multiple times. I
> tried and it works, though it is useless.
> 

Yes, but they are the same.

> Vince, I think the correct way to detect which entry is for perf_event
> is to look for filesystem type cgroup and option perf_event. It cannot
> be anything else. First match is good enough.
> .

+1