git bisect and perf

git bisect and perf

[David Ahern]

I am trying find out when (ie. which commit) the pebs feature actually 
started working on a westmere system and hitting a problem with git-bisect:

# git bisect start arch/x86/kernel/cpu/perf_event*

# git bisect bad v2.6.35

# git bisect good v2.6.38
Some good revs are not ancestor of the bad rev.
git bisect cannot work properly in this case.
Maybe you mistake good and bad revs?

Any ideas on why git bisect would not work?

David

Re: git bisect and perf

[Roland Dreier]

On Tue, Jun 5, 2012 at 8:29 AM, David Ahern <dsahern@gmail.com> wrote:
> # git bisect good v2.6.38
> Some good revs are not ancestor of the bad rev.
> git bisect cannot work properly in this case.
> Maybe you mistake good and bad revs?

git bisect is telling you what is wrong -- as the man page says:

    This command uses git rev-list --bisect to help drive the binary
    search process to find which change introduced a bug, given an old
    "good" commit object name and a later "bad" commit object name.

so it assumes the good commit is older than the bad commit.

You can actually use git bisect in your case, although it gets very confusing
unless you write yourself a little wrapper alias: just swap the meaning of
good and bad, ie do

# git bisect start arch/x86/kernel/cpu/perf_event*
# git bisect good v2.6.35
# git bisect bad v2.6.38

and try somehow to remember that when you do a test, "bad" means
PEBS works and "good" means PEBS doesn't work.

 - R.

Re: git bisect and perf

[David Ahern]

On 6/5/12 9:51 AM, Roland Dreier wrote:
> On Tue, Jun 5, 2012 at 8:29 AM, David Ahern<dsahern@gmail.com>  wrote:
>> # git bisect good v2.6.38
>> Some good revs are not ancestor of the bad rev.
>> git bisect cannot work properly in this case.
>> Maybe you mistake good and bad revs?
>
> git bisect is telling you what is wrong -- as the man page says:
>
>      This command uses git rev-list --bisect to help drive the binary
>      search process to find which change introduced a bug, given an old
>      "good" commit object name and a later "bad" commit object name.
>
> so it assumes the good commit is older than the bad commit.
>
> You can actually use git bisect in your case, although it gets very confusing
> unless you write yourself a little wrapper alias: just swap the meaning of
> good and bad, ie do
>
> # git bisect start arch/x86/kernel/cpu/perf_event*
> # git bisect good v2.6.35
> # git bisect bad v2.6.38
>
> and try somehow to remember that when you do a test, "bad" means
> PEBS works and "good" means PEBS doesn't work.
>
>   - R.

Thanks, Roland. I'll reverse the logic.

David

Re: git bisect and perf

[Peter Zijlstra]

On Tue, 2012-06-05 at 09:29 -0600, David Ahern wrote:
> I am trying find out when (ie. which commit) the pebs feature
> actually 
> started working on a westmere system

What do you mean with working? The whole cycles:pp thing is magic and
unrelated to 'regular' PEBS stuff. .35 would indeed be the version we
introduced PEBS support and it should actually work.

But if you're looking for the cycles:pp stuff (commit 7639dae0ca1)
that's .38-rc1-ish.

Re: git bisect and perf

[Peter Zijlstra]

On Tue, 2012-06-05 at 18:00 +0200, Peter Zijlstra wrote:
> On Tue, 2012-06-05 at 09:29 -0600, David Ahern wrote:
> > I am trying find out when (ie. which commit) the pebs feature
> > actually 
> > started working on a westmere system
> 
> What do you mean with working? The whole cycles:pp thing is magic and
> unrelated to 'regular' PEBS stuff. .35 would indeed be the version we
> introduced PEBS support and it should actually work.
> 
> But if you're looking for the cycles:pp stuff (commit 7639dae0ca1)
> that's .38-rc1-ish.

So on kernels pre that commit you could still do: 

  perf record -e r108000c0:pp

and get the same effect. Its just that after that commit r3c:pp will be
magically translated to the above.

And just to be complete (as Stephane always needs to point out) event 3c
and 108000c0 aren't fully identical, they differ in that one counts
while idle and the other doesn't (but I forgot which way around).

Luckily not a lot of people care about profiling idle time.

Re: git bisect and perf

[David Ahern]

On 6/5/12 10:05 AM, Peter Zijlstra wrote:
> On Tue, 2012-06-05 at 18:00 +0200, Peter Zijlstra wrote:
>> On Tue, 2012-06-05 at 09:29 -0600, David Ahern wrote:
>>> I am trying find out when (ie. which commit) the pebs feature
>>> actually
>>> started working on a westmere system
>>
>> What do you mean with working? The whole cycles:pp thing is magic and
>> unrelated to 'regular' PEBS stuff.

:p requires PEBS?

> .35 would indeed be the version we
>> introduced PEBS support and it should actually work.

I'm referring to cycles:p

model name	: Intel(R) Xeon(R) CPU           E5620  @ 2.40GHz
Fedora 14, 2.6.35.14-106.fc14.x86_64 (few perf_event* differences from 
2.6.35.14).

# perf record -e cycles:p -v -- sleep 1

   Warning:  ... trying to fall back to cpu-clock-ticks

...

Cycles is supported; it's the :p that is not. perf userspace only 
recently gained the knowledge for discriminating the two. Using perf 
from acme/perf/urgent:

# /tmp/pbuild/perf record -e cycles:p -v -- sleep 1

   Error: sys_perf_event_open() syscall returned with 28 (No space left 
on device).  /bin/dmesg may provide additional information.

I chased this to intel_pebs_constraints() returning &emptyconstraint 
which has a weight of 0 (validate_event -> intel_get_event_constraints 
-> intel_pebs_constraints).


>>
>> But if you're looking for the cycles:pp stuff (commit 7639dae0ca1)
>> that's .38-rc1-ish.
>
> So on kernels pre that commit you could still do:
>
>    perf record -e r108000c0:pp

Indeed that works on the F14 kernel.

David

Re: git bisect and perf

[Peter Zijlstra]

On Tue, 2012-06-05 at 11:12 -0600, David Ahern wrote:
> On 6/5/12 10:05 AM, Peter Zijlstra wrote:
> > On Tue, 2012-06-05 at 18:00 +0200, Peter Zijlstra wrote:
> >> On Tue, 2012-06-05 at 09:29 -0600, David Ahern wrote:
> >>> I am trying find out when (ie. which commit) the pebs feature
> >>> actually
> >>> started working on a westmere system
> >>
> >> What do you mean with working? The whole cycles:pp thing is magic and
> >> unrelated to 'regular' PEBS stuff.
> 
> :p requires PEBS?

On Intel it does.. On AMD you want IBS for that.

Note that on Intel there's a difference between :p and :pp.

>From perf_event_attr:

        /*
         * precise_ip:
         *
         *  0 - SAMPLE_IP can have arbitrary skid
         *  1 - SAMPLE_IP must have constant skid
         *  2 - SAMPLE_IP requested to have 0 skid
         *  3 - SAMPLE_IP must have 0 skid
         *
         *  See also PERF_RECORD_MISC_EXACT_IP
         */
        precise_ip     :  2, /* skid constraint       */

So 0 being the default (no :p) we use the regular PMI and provide the IP
from the spot that served the exception. This can be a few instructions
_after_ the instruction that triggered the event overflow.

1 (:p) means constant skid, for Intel this maps to regular PEBS and
returns the IP as provided by the PEBS record. This is one instruction
after the instruction that caused the event overflow (because the PEBS
hardware assist triggers when we're executing the instruction that
triggered it, it needs to have already fetched the instruction and
incremented the IP etc..).

2 (:pp) requests 0 skid, since PEBS provides IP + 1 instruction, we need
to rewind the instruction stream. In order to do this we use the LBR in
conjunction with a FREEZE_LBR_ON_PMI feature from DEBUG_CTL and limit
the PEBS buffer to a single entry. So when we get the PEBS buffer full
PMI, we read the LBR, find the start of the most recent basic block (to
address of the last branch) and decode the instruction stream until we
find the current IP. Then we know the IP that caused it is the previous
one. Tada!

We set the PERF_RECORD_MISC_EXACT_IP bit when the rewind worked and we
can guarantee 0 skid (+- hardware bugs, of which there are a few).

3 (:ppp) would require 0 skid, but since the unwind using LBR isn't a
guaranteed thing we don't implement this for Intel (or anybody else for
that matter afaik).


Now not all events the Intel PMU can do are PEBS capable; in fact most
aren't. So the whole cycles thing is about mapping the regular cycles
event to something that is PEBS capable.

Re: git bisect and perf

[David Ahern]

Thanks for the explanation and patience.

On 6/5/12 11:35 AM, Peter Zijlstra wrote:
> Now not all events the Intel PMU can do are PEBS capable; in fact most
> aren't. So the whole cycles thing is about mapping the regular cycles
> event to something that is PEBS capable.
>


I think I get it now. 7639dae0 is the commit I needed. With it 2.6.35.14 
can do cycles:pp profiling.

In general only the events listed in intel_XXXXX_pebs_events can be used 
with :p.

David

Re: git bisect and perf

[Stephane Eranian]

On Wed, Jun 6, 2012 at 1:48 AM, David Ahern <dsahern@gmail.com> wrote:
> Thanks for the explanation and patience.
>
>
> On 6/5/12 11:35 AM, Peter Zijlstra wrote:
>>
>> Now not all events the Intel PMU can do are PEBS capable; in fact most
>> aren't. So the whole cycles thing is about mapping the regular cycles
>> event to something that is PEBS capable.
>>
>
>
> I think I get it now. 7639dae0 is the commit I needed. With it 2.6.35.14 can
> do cycles:pp profiling.
>
> In general only the events listed in intel_XXXXX_pebs_events can be used
> with :p.
>
Yes.

> David
>

Re: git bisect and perf

[Stephane Eranian]

On Tue, Jun 5, 2012 at 7:12 PM, David Ahern <dsahern@gmail.com> wrote:
> On 6/5/12 10:05 AM, Peter Zijlstra wrote:
>>
>> On Tue, 2012-06-05 at 18:00 +0200, Peter Zijlstra wrote:
>>>
>>> On Tue, 2012-06-05 at 09:29 -0600, David Ahern wrote:
>>>>
>>>> I am trying find out when (ie. which commit) the pebs feature
>>>> actually
>>>> started working on a westmere system
>>>
>>>
>>> What do you mean with working? The whole cycles:pp thing is magic and
>>> unrelated to 'regular' PEBS stuff.
>
>
> :p requires PEBS?
>
>
>> .35 would indeed be the version we
>>>
>>> introduced PEBS support and it should actually work.
>
>
> I'm referring to cycles:p
>
> model name      : Intel(R) Xeon(R) CPU           E5620  @ 2.40GHz
That's not as helpful as /proc/cpuinfo. I suspect this is a Westmere
system, isn't it?

> Fedora 14, 2.6.35.14-106.fc14.x86_64 (few perf_event* differences from
> 2.6.35.14).
>
> # perf record -e cycles:p -v -- sleep 1
>
>  Warning:  ... trying to fall back to cpu-clock-ticks
>
> ...
>
> Cycles is supported; it's the :p that is not. perf userspace only recently
> gained the knowledge for discriminating the two. Using perf from
> acme/perf/urgent:
>
> # /tmp/pbuild/perf record -e cycles:p -v -- sleep 1
>
>  Error: sys_perf_event_open() syscall returned with 28 (No space left on
> device).  /bin/dmesg may provide additional information.
>
> I chased this to intel_pebs_constraints() returning &emptyconstraint which
> has a weight of 0 (validate_event -> intel_get_event_constraints ->
> intel_pebs_constraints).
>
>
>
>>>
>>> But if you're looking for the cycles:pp stuff (commit 7639dae0ca1)
>>> that's .38-rc1-ish.
>>
>>
>> So on kernels pre that commit you could still do:
>>
>>   perf record -e r108000c0:pp
>
>
> Indeed that works on the F14 kernel.
>
> David

Re: git bisect and perf

[David Ahern]

On 6/5/12 1:04 PM, Stephane Eranian wrote:
>> I'm referring to cycles:p
>>
>> model name      : Intel(R) Xeon(R) CPU           E5620  @ 2.40GHz
> That's not as helpful as /proc/cpuinfo. I suspect this is a Westmere
> system, isn't it?
>


Yes:
processor	: 15
vendor_id	: GenuineIntel
cpu family	: 6
model		: 44
model name	: Intel(R) Xeon(R) CPU           E5620  @ 2.40GHz
stepping	: 2
cpu MHz		: 2394.133
cache size	: 12288 KB

[    0.146055] Performance Events: PEBS fmt1+, Westmere events, Intel 
PMU driver.
[    0.146645] ... version:                3
[    0.146897] ... bit width:              48
[    0.147150] ... generic registers:      4
[    0.147404] ... value mask:             0000ffffffffffff
[    0.147657] ... max period:             000000007fffffff
[    0.147907] ... fixed-purpose events:   3
[    0.148158] ... event mask:             000000070000000f