Problems with perf "event grouping" on Java apps

Problems with perf "event grouping" on Java apps

[Jen-Cheng(Tommy) Huang]

Hi

I am trying to use perf with event grouping to profile Java apps.
My kernel is 3.16.1-1.el6.elrepo.x86_64
perf version 3.16.1-1.el6.elrepo.x86_64
The Java app I use is hadoop TaskTracker, which spawns multiple child JVMs.
There seems to be something wrong with event group, since very few
samples were collected if I use more than one event in a group.
The following shows an example

I used
perf record -e '{instructions}:S' -c 100000 [TaskTracker command]
This one works fine and it collects a reasonable amount of samples and
several threads are sampled.

However, the following command does not work.
perf record -e '{instructions,cycles}:S' -c 100000 [TaskTracker command]
Less than 1000 samples were collected and only one thread was sampled

If I don't use :S to read samples, the sample counts are reasonable,
but the event grouping is gone. (both events were sampled, no leader
event.)
perf record -e '{instructions,cycles}' -c 100000 [TaskTracker command]

I tested single-threaded app, such as SPEC CPU.
Everything works fine. The event grouping works fine, so as the :S flag.
However, the Java app that I tested does not.
Please let me know if any suggestions on this issue.

Thanks.

Re: Problems with perf

[Tommy]

Jen-Cheng(Tommy) Huang <tommy24 <at> gatech.edu> writes:
> 
> Hi
> 
> I am trying to use perf with event grouping to profile Java apps.
> My kernel is 3.16.1-1.el6.elrepo.x86_64
> perf version 3.16.1-1.el6.elrepo.x86_64
> The Java app I use is hadoop TaskTracker, which spawns multiple child 
JVMs.
> There seems to be something wrong with event group, since very few
> samples were collected if I use more than one event in a group.
> The following shows an example
> 
> I used
> perf record -e '{instructions}:S' -c 100000 [TaskTracker command]
> This one works fine and it collects a reasonable amount of samples and
> several threads are sampled.
> 
> However, the following command does not work.
> perf record -e '{instructions,cycles}:S' -c 100000 [TaskTracker command]
> Less than 1000 samples were collected and only one thread was sampled
> 
> If I don't use :S to read samples, the sample counts are reasonable,
> but the event grouping is gone. (both events were sampled, no leader
> event.)
> perf record -e '{instructions,cycles}' -c 100000 [TaskTracker command]
> 
> I tested single-threaded app, such as SPEC CPU.
> Everything works fine. The event grouping works fine, so as the :S flag.
> However, the Java app that I tested does not.
> Please let me know if any suggestions on this issue.
> 
> Thanks.
> 
To make it clear about the sample count difference, this is the stats 
collected from a few seconds run of the same app: With only one event in a 
group, it has ~362339 samples while 2 events in a group, it has only 938 
samples (only 1 thread was profiled). I think the sample recording is 
somehow stopped.
Thanks.

Re: Problems with perf

[Tommy]

Tommy <tommy24 <at> gatech.edu> writes:

> 
> Jen-Cheng(Tommy) Huang <tommy24 <at> gatech.edu> writes:
> > 
> > Hi
> > 
> > I am trying to use perf with event grouping to profile Java apps.
> > My kernel is 3.16.1-1.el6.elrepo.x86_64
> > perf version 3.16.1-1.el6.elrepo.x86_64
> > The Java app I use is hadoop TaskTracker, which spawns multiple child 
> JVMs.
> > There seems to be something wrong with event group, since very few
> > samples were collected if I use more than one event in a group.
> > The following shows an example
> > 
> > I used
> > perf record -e '{instructions}:S' -c 100000 [TaskTracker command]
> > This one works fine and it collects a reasonable amount of samples and
> > several threads are sampled.
> > 
> > However, the following command does not work.
> > perf record -e '{instructions,cycles}:S' -c 100000 [TaskTracker command]
> > Less than 1000 samples were collected and only one thread was sampled
> > 
> > If I don't use :S to read samples, the sample counts are reasonable,
> > but the event grouping is gone. (both events were sampled, no leader
> > event.)
> > perf record -e '{instructions,cycles}' -c 100000 [TaskTracker command]
> > 
> > I tested single-threaded app, such as SPEC CPU.
> > Everything works fine. The event grouping works fine, so as the :S flag.
> > However, the Java app that I tested does not.
> > Please let me know if any suggestions on this issue.
> > 
> > Thanks.
> > 
> To make it clear about the sample count difference, this is the stats 
> collected from a few seconds run of the same app: With only one event in a 
> group, it has ~362339 samples while 2 events in a group, it has only 938 
> samples (only 1 thread was profiled). I think the sample recording is 
> somehow stopped.
> Thanks.

I found that even the woken counts are different
1 event in a group:
[ perf record: Woken up 27 times to write data ]
[ perf record: Captured and wrote 7.165 MB perf.data (~313046 samples) ]

2 events in a group:
[ perf record: Woken up 1 times to write data ]
[ perf record: Captured and wrote 0.051 MB perf.data (~2223 samples) ]

It seems that something has been disabled, but no error/warning messages 
were shown.
Thanks.

Re: Problems with perf

[Jiri Olsa]

On Mon, Aug 18, 2014 at 08:21:22PM +0000, Tommy wrote:
> Tommy <tommy24 <at> gatech.edu> writes:
> 
> > 
> > Jen-Cheng(Tommy) Huang <tommy24 <at> gatech.edu> writes:
> > > 
> > > Hi
> > > 
> > > I am trying to use perf with event grouping to profile Java apps.
> > > My kernel is 3.16.1-1.el6.elrepo.x86_64
> > > perf version 3.16.1-1.el6.elrepo.x86_64
> > > The Java app I use is hadoop TaskTracker, which spawns multiple child 
> > JVMs.
> > > There seems to be something wrong with event group, since very few
> > > samples were collected if I use more than one event in a group.
> > > The following shows an example
> > > 
> > > I used
> > > perf record -e '{instructions}:S' -c 100000 [TaskTracker command]
> > > This one works fine and it collects a reasonable amount of samples and
> > > several threads are sampled.
> > > 
> > > However, the following command does not work.
> > > perf record -e '{instructions,cycles}:S' -c 100000 [TaskTracker command]
> > > Less than 1000 samples were collected and only one thread was sampled
> > > 
> > > If I don't use :S to read samples, the sample counts are reasonable,
> > > but the event grouping is gone. (both events were sampled, no leader
> > > event.)
> > > perf record -e '{instructions,cycles}' -c 100000 [TaskTracker command]
> > > 
> > > I tested single-threaded app, such as SPEC CPU.
> > > Everything works fine. The event grouping works fine, so as the :S flag.
> > > However, the Java app that I tested does not.
> > > Please let me know if any suggestions on this issue.
> > > 
> > > Thanks.
> > > 
> > To make it clear about the sample count difference, this is the stats 
> > collected from a few seconds run of the same app: With only one event in a 
> > group, it has ~362339 samples while 2 events in a group, it has only 938 
> > samples (only 1 thread was profiled). I think the sample recording is 
> > somehow stopped.
> > Thanks.
> 
> I found that even the woken counts are different
> 1 event in a group:
> [ perf record: Woken up 27 times to write data ]
> [ perf record: Captured and wrote 7.165 MB perf.data (~313046 samples) ]
> 
> 2 events in a group:
> [ perf record: Woken up 1 times to write data ]
> [ perf record: Captured and wrote 0.051 MB perf.data (~2223 samples) ]

hi,
we switch off event inheritance for multiple events in group
with leader sampling. That's probably why you see less samples
and only one thread. 

The reason for that is the way we read event values, which
does not support inherited events yet.

I'll check and see what can be done about that.. at least
display a warning ;-)

thanks,
jirka

Re: Problems with perf

[Andi Kleen]

Jiri Olsa <jolsa@redhat.com> writes:
> hi,
> we switch off event inheritance for multiple events in group
> with leader sampling. That's probably why you see less samples
> and only one thread. 

If that's the problem then a short term workaround would
be using -a

-Andi

Re: Problems with perf

[Jen-Cheng(Tommy) Huang]

Thank you guys.
So far the "-a" full system mode seems working.
Thanks for the workaround.
I will update you guys for any issues.
Thanks.


On Tue, Aug 19, 2014 at 9:09 AM, Andi Kleen <andi@firstfloor.org> wrote:
> Jiri Olsa <jolsa@redhat.com> writes:
>> hi,
>> we switch off event inheritance for multiple events in group
>> with leader sampling. That's probably why you see less samples
>> and only one thread.
>
> If that's the problem then a short term workaround would
> be using -a
>
> -Andi