Performance Events hangs with Intel P4 system

Performance Events hangs with Intel P4 system

[Jaswinder Singh Rajput]

Hello,

I am testing Performance Events on P4 with HT:

[    0.002243] Performance Events: Netburst events, Netburst P4/Xeon PMU driver.
[    0.002432] ... version:                0
[    0.002545] ... bit width:              40
[    0.002659] ... generic registers:      18
[    0.002772] ... value mask:             000000ffffffffff
[    0.002887] ... max period:             0000007fffffffff
[    0.003004] ... fixed-purpose events:   0
[    0.003118] ... event mask:             000000000003ffff

dmesg : http://userweb.kernel.org/~jaswinder/P4_HT/dmesg-2634-rc7-tip.txt
config : http://userweb.kernel.org/~jaswinder/P4_HT/config-2634-rc7-tip.txt

[jaswinder@ht perf]$ ./perf stat -e cycles ls > /dev/null

 Performance counter stats for 'ls':

  <not counted>  cycles

    0.003447053  seconds time elapsed

[jaswinder@ht perf]$ ./perf stat -e cycles,instructions ls > /dev/null

 Performance counter stats for 'ls':

  <not counted>  cycles
        1862188  instructions             #      0.000 IPC    (scaled
from 54.91%)

    0.003419230  seconds time elapsed

[jaswinder@ht perf]$ ./perf stat -e
cycles,instructions,cache-references ls > /dev/null

<<this dumps on screen and hangs the systems>>

How can I fix this problem.

Thanks,
--
Jaswinder Singh.

Re: Performance Events hangs with Intel P4 system

[Cyrill Gorcunov]

On Friday, May 14, 2010, Jaswinder Singh Rajput
<jaswinderlinux@gmail.com> wrote:
> Hello,
>
> I am testing Performance Events on P4 with HT:
>
> [    0.002243] Performance Events: Netburst events, Netburst P4/Xeon PMU driver.
> [    0.002432] ... version:                0
> [    0.002545] ... bit width:              40
> [    0.002659] ... generic registers:      18
> [    0.002772] ... value mask:             000000ffffffffff
> [    0.002887] ... max period:             0000007fffffffff
> [    0.003004] ... fixed-purpose events:   0
> [    0.003118] ... event mask:             000000000003ffff
>
> dmesg : http://userweb.kernel.org/~jaswinder/P4_HT/dmesg-2634-rc7-tip.txt
> config : http://userweb.kernel.org/~jaswinder/P4_HT/config-2634-rc7-tip.txt
>
> [jaswinder@ht perf]$ ./perf stat -e cycles ls > /dev/null
>
>  Performance counter stats for 'ls':
>
>   <not counted>  cycles
>
>     0.003447053  seconds time elapsed
>
> [jaswinder@ht perf]$ ./perf stat -e cycles,instructions ls > /dev/null
>
>  Performance counter stats for 'ls':
>
>   <not counted>  cycles
>         1862188  instructions             #      0.000 IPC    (scaled
> from 54.91%)
>
>     0.003419230  seconds time elapsed
>
> [jaswinder@ht perf]$ ./perf stat -e
> cycles,instructions,cache-references ls > /dev/null
>
> <<this dumps on screen and hangs the systems>>
>
> How can I fix this problem.
>
> Thanks,
> --
> Jaswinder Singh.
>

thanks for testing. Is there any issue in dmesg for first two perf
calls before it hangs? And is there a chance for screen dump (photo or
netconsole)?

Re: Performance Events hangs with Intel P4 system

[Jaswinder Singh Rajput]

Hello Cyrill,

On Fri, May 14, 2010 at 8:49 AM, Cyrill Gorcunov <gorcunov@gmail.com> wrote:
> On Friday, May 14, 2010, Jaswinder Singh Rajput
> <jaswinderlinux@gmail.com> wrote:
>> Hello,
>>
>> I am testing Performance Events on P4 with HT:
>>
>> [    0.002243] Performance Events: Netburst events, Netburst P4/Xeon PMU driver.
>> [    0.002432] ... version:                0
>> [    0.002545] ... bit width:              40
>> [    0.002659] ... generic registers:      18
>> [    0.002772] ... value mask:             000000ffffffffff
>> [    0.002887] ... max period:             0000007fffffffff
>> [    0.003004] ... fixed-purpose events:   0
>> [    0.003118] ... event mask:             000000000003ffff
>>
>> dmesg : http://userweb.kernel.org/~jaswinder/P4_HT/dmesg-2634-rc7-tip.txt
>> config : http://userweb.kernel.org/~jaswinder/P4_HT/config-2634-rc7-tip.txt
>>
>> [jaswinder@ht perf]$ ./perf stat -e cycles ls > /dev/null
>>
>>  Performance counter stats for 'ls':
>>
>>   <not counted>  cycles
>>
>>     0.003447053  seconds time elapsed
>>
>> [jaswinder@ht perf]$ ./perf stat -e cycles,instructions ls > /dev/null
>>
>>  Performance counter stats for 'ls':
>>
>>   <not counted>  cycles
>>         1862188  instructions             #      0.000 IPC    (scaled
>> from 54.91%)
>>
>>     0.003419230  seconds time elapsed
>>
>> [jaswinder@ht perf]$ ./perf stat -e
>> cycles,instructions,cache-references ls > /dev/null
>>
>> <<this dumps on screen and hangs the systems>>
>>
>> How can I fix this problem.
>>
>> Thanks,
>> --
>> Jaswinder Singh.
>>
>
> thanks for testing. Is there any issue in dmesg for first two perf
> calls before it hangs?

There is no issue in dmesg for first two perf calls.

> And is there a chance for screen dump (photo or
> netconsole)?
>

Here is netconsole :

[jaswinder@ht perf]$ ./perf stat -e
cycles,instructions,cache-references ls > /dev/null

Message from syslogd@ht at May 14 09:39:32 ...
 kernel:[  314.908284] general protection fault: 0000 [#1] PREEMPT SMP

Message from syslogd@ht at May 14 09:39:32 ...
 kernel:[  314.908296] last sysfs file:
/sys/class/net/eth1/statistics/collisions

Message from syslogd@ht at May 14 09:39:32 ...
 kernel:[  314.908346] Process ls (pid: 2726, ti=edac2000
task=ed9eb240 task.ti=edac2000)

Message from syslogd@ht at May 14 09:39:32 ...
 kernel:[  314.908349] Stack:

Message from syslogd@ht at May 14 09:39:32 ...
 kernel:[  314.908387] Call Trace:

Message from syslogd@ht at May 14 09:39:32 ...
 kernel:[  314.908549] Code: cb 89 d1 e8 89 c7 13 00 8b 75 dc 8b 45 e4
8b 55 d0 23 15 68 37 58 c1 8b b6 c4 00 00 00 01 75 f0 f7 d8 8b 4d f0
23 05 64 37 58 c1 <0f> 30 8b 45 dc e8 a4 63 06 00 8b 45 ec 83 c4 30 5b
5e 5f 5d c3

Message from syslogd@ht at May 14 09:39:32 ...
 kernel:[  314.908612] EIP: [<c100ccca>]
x86_perf_event_set_period+0x19d/0x1b2 SS:ESP 0068:edac3d70

If you need more information, please let me know.

Thanks,
--

Jaswinder Singh.

Re: Performance Events hangs with Intel P4 system

[Cyrill Gorcunov]

On Friday, May 14, 2010, Jaswinder Singh Rajput
<jaswinderlinux@gmail.com> wrote:
> Hello Cyrill,
>
> On Fri, May 14, 2010 at 8:49 AM, Cyrill Gorcunov <gorcunov@gmail.com> wrote:
>> On Friday, May 14, 2010, Jaswinder Singh Rajput
>> <jaswinderlinux@gmail.com> wrote:
>>> Hello,
>>>
>>> I am testing Performance Events on P4 with HT:
>>>
>>> [    0.002243] Performance Events: Netburst events, Netburst P4/Xeon PMU driver.
>>> [    0.002432] ... version:                0
>>> [    0.002545] ... bit width:              40
>>> [    0.002659] ... generic registers:      18
>>> [    0.002772] ... value mask:             000000ffffffffff
>>> [    0.002887] ... max period:             0000007fffffffff
>>> [    0.003004] ... fixed-purpose events:   0
>>> [    0.003118] ... event mask:             000000000003ffff
>>>
>>> dmesg : http://userweb.kernel.org/~jaswinder/P4_HT/dmesg-2634-rc7-tip.txt
>>> config : http://userweb.kernel.org/~jaswinder/P4_HT/config-2634-rc7-tip.txt
>>>
>>> [jaswinder@ht perf]$ ./perf stat -e cycles ls > /dev/null
>>>
>>>  Performance counter stats for 'ls':
>>>
>>>   <not counted>  cycles
>>>
>>>     0.003447053  seconds time elapsed
>>>
>>> [jaswinder@ht perf]$ ./perf stat -e cycles,instructions ls > /dev/null
>>>
>>>  Performance counter stats for 'ls':
>>>
>>>   <not counted>  cycles
>>>         1862188  instructions             #      0.000 IPC    (scaled
>>> from 54.91%)
>>>
>>>     0.003419230  seconds time elapsed
>>>
>>> [jaswinder@ht perf]$ ./perf stat -e
>>> cycles,instructions,cache-references ls > /dev/null
>>>
>>> <<this dumps on screen and hangs the systems>>
>>>
>>> How can I fix this problem.
>>>
>>> Thanks,
>>> --
>>> Jaswinder Singh.
>>>
>>
>> thanks for testing. Is there any issue in dmesg for first two perf
>> calls before it hangs?
>
> There is no issue in dmesg for first two perf calls.
>
>> And is there a chance for screen dump (photo or
>> netconsole)?
>>
>
> Here is netconsole :
>
> [jaswinder@ht perf]$ ./perf stat -e
> cycles,instructions,cache-references ls > /dev/null
>
> Message from syslogd@ht at May 14 09:39:32 ...
>  kernel:[  314.908284] general protection fault: 0000 [#1] PREEMPT SMP
>
> Message from syslogd@ht at May 14 09:39:32 ...
>  kernel:[  314.908296] last sysfs file:
> /sys/class/net/eth1/statistics/collisions
>
> Message from syslogd@ht at May 14 09:39:32 ...
>  kernel:[  314.908346] Process ls (pid: 2726, ti=edac2000
> task=ed9eb240 task.ti=edac2000)
>
> Message from syslogd@ht at May 14 09:39:32 ...
>  kernel:[  314.908349] Stack:
>
> Message from syslogd@ht at May 14 09:39:32 ...
>  kernel:[  314.908387] Call Trace:
>
> Message from syslogd@ht at May 14 09:39:32 ...
>  kernel:[  314.908549] Code: cb 89 d1 e8 89 c7 13 00 8b 75 dc 8b 45 e4
> 8b 55 d0 23 15 68 37 58 c1 8b b6 c4 00 00 00 01 75 f0 f7 d8 8b 4d f0
> 23 05 64 37 58 c1 <0f> 30 8b 45 dc e8 a4 63 06 00 8b 45 ec 83 c4 30 5b
> 5e 5f 5d c3
>
> Message from syslogd@ht at May 14 09:39:32 ...
>  kernel:[  314.908612] EIP: [<c100ccca>]
> x86_perf_event_set_period+0x19d/0x1b2 SS:ESP 0068:edac3d70
>
> If you need more information, please let me know.
>
> Thanks,
> --
>
> Jaswinder Singh.
>

thanks Jaswinder, i'll take a look, meanwhile if you get a chance to
test latest -tip/master it would be great

Re: Performance Events hangs with Intel P4 system

[Cyrill Gorcunov]

On Friday, May 14, 2010, Cyrill Gorcunov <gorcunov@gmail.com> wrote:
> On Friday, May 14, 2010, Jaswinder Singh Rajput
> <jaswinderlinux@gmail.com> wrote:
>> Hello Cyrill,
>>
>> On Fri, May 14, 2010 at 8:49 AM, Cyrill Gorcunov <gorcunov@gmail.com> wrote:
>>> On Friday, May 14, 2010, Jaswinder Singh Rajput
>>> <jaswinderlinux@gmail.com> wrote:
>>>> Hello,
>>>>
>>>> I am testing Performance Events on P4 with HT:
>>>>
>>>> [    0.002243] Performance Events: Netburst events, Netburst P4/Xeon PMU driver.
>>>> [    0.002432] ... version:                0
>>>> [    0.002545] ... bit width:              40
>>>> [    0.002659] ... generic registers:      18
>>>> [    0.002772] ... value mask:             000000ffffffffff
>>>> [    0.002887] ... max period:             0000007fffffffff
>>>> [    0.003004] ... fixed-purpose events:   0
>>>> [    0.003118] ... event mask:             000000000003ffff
>>>>
>>>> dmesg : http://userweb.kernel.org/~jaswinder/P4_HT/dmesg-2634-rc7-tip.txt
>>>> config : http://userweb.kernel.org/~jaswinder/P4_HT/config-2634-rc7-tip.txt
>>>>
>>>> [jaswinder@ht perf]$ ./perf stat -e cycles ls > /dev/null
>>>>
>>>>  Performance counter stats for 'ls':
>>>>
>>>>   <not counted>  cycles
>>>>
>>>>     0.003447053  seconds time elapsed
>>>>
>>>> [jaswinder@ht perf]$ ./perf stat -e cycles,instructions ls > /dev/null
>>>>
>>>>  Performance counter stats for 'ls':
>>>>
>>>>   <not counted>  cycles
>>>>         1862188  instructions             #      0.000 IPC    (scaled
>>>> from 54.91%)
>>>>
>>>>     0.003419230  seconds time elapsed
>>>>
>>>> [jaswinder@ht perf]$ ./perf stat -e
>>>> cycles,instructions,cache-references ls > /dev/null
>>>>
>>>> <<this dumps on screen and hangs the systems>>
>>>>
>>>> How can I fix this problem.
>>>>
>>>> Thanks,
>>>> --
>>>> Jaswinder Singh.
>>>>
>>>
>>> thanks for testing. Is there any issue in dmesg for first two perf
>>> calls before it hangs?
>>
>> There is no issue in dmesg for first two perf calls.
>>
>>> And is there a chance for screen dump (photo or
>>> netconsole)?
>>>
>>
>> Here is netconsole :
>>
>> [jaswinder@ht perf]$ ./perf stat -e
>> cycles,instructions,cache-references ls > /dev/null
>>
>> Message from syslogd@ht at May 14 09:39:32 ...
>>  kernel:[  314.908284] general protection fault: 0000 [#1] PREEMPT SMP
>>
>> Message from syslogd@ht at May 14 09:39:32 ...
>>  kernel:[  314.908296] last sysfs file:
>> /sys/class/net/eth1/statistics/collisions
>>
>> Message from syslogd@ht at May 14 09:39:32 ...
>>  kernel:[  314.908346] Process ls (pid: 2726, ti=edac2000
>> task=ed9eb240 task.ti=edac2000)
>>
>> Message from syslogd@ht at May 14 09:39:32 ...
>>  kernel:[  314.908349] Stack:
>>
>> Message from syslogd@ht at May 14 09:39:32 ...
>>  kernel:[  314.908387] Call Trace:
>>
>> Message from syslogd@ht at May 14 09:39:32 ...
>>  kernel:[  314.908549] Code: cb 89 d1 e8 89 c7 13 00 8b 75 dc 8b 45 e4
>> 8b 55 d0 23 15 68 37 58 c1 8b b6 c4 00 00 00 01 75 f0 f7 d8 8b 4d f0
>> 23 05 64 37 58 c1 <0f> 30 8b 45 dc e8 a4 63 06 00 8b 45 ec 83 c4 30 5b
>> 5e 5f 5d c3
>>
>> Message from syslogd@ht at May 14 09:39:32 ...
>>  kernel:[  thanks Jaswinder, i'll take a look, meanwhile if you get a chance to
> test latest -tip/master it would be great
>

it's found that we have problem in cache events, we're working on
that, thanks for report!

Re: Performance Events hangs with Intel P4 system

[Jaswinder Singh Rajput]

Hello Cyrill,

On Fri, May 14, 2010 at 11:53 AM, Cyrill Gorcunov <gorcunov@gmail.com> wrote:
> On Friday, May 14, 2010, Cyrill Gorcunov <gorcunov@gmail.com> wrote:
>>> Message from syslogd@ht at May 14 09:39:32 ...
>>>  kernel:[  thanks Jaswinder, i'll take a look, meanwhile if you get a chance to
>> test latest -tip/master it would be great
>>
>
> it's found that we have problem in cache events, we're working on
> that, thanks for report!
>

Great !!

If you need any help from my side, please let me know. Even though it
is very hot here, I will try my best ;-)

Thanks,
--
Jaswinder Singh.

Re: Performance Events hangs with Intel P4 system

[Cyrill Gorcunov]

On Friday, May 14, 2010, Jaswinder Singh Rajput
<jaswinderlinux@gmail.com> wrote:
> Hello Cyrill,
>
> On Fri, May 14, 2010 at 11:53 AM, Cyrill Gorcunov <gorcunov@gmail.com> wrote:
>> On Friday, May 14, 2010, Cyrill Gorcunov <gorcunov@gmail.com> wrote:
>>>> Message from syslogd@ht at May 14 09:39:32 ...
>>>>  kernel:[  thanks Jaswinder, i'll take a look, meanwhile if you get a chance to
>>> test latest -tip/master it would be great
>>>
>>
>> it's found that we have problem in cache events, we're working on
>> that, thanks for report!
>>
>
> Great !!
>
> If you need any help from my side, please let me know. Even though it
> is very hot here, I will try my best ;-)
>
> Thanks,
> --
> Jaswinder Singh.
>

Ming is narrowing down the guilty commit. I thought about my last
patch related to escr hashing, but it shouldn't bring such effect.
Hmm...

Re: Performance Events hangs with Intel P4 system

[Cyrill Gorcunov]

On Friday, May 14, 2010, Cyrill Gorcunov <gorcunov@gmail.com> wrote:
> On Friday, May 14, 2010, Jaswinder Singh Rajput
> <jaswinderlinux@gmail.com> wrote:
>> Hello Cyrill,
>>
>> On Fri, May 14, 2010 at 11:53 AM, Cyrill Gorcunov <gorcunov@gmail.com> wrote:
>>> On Friday, May 14, 2010, Cyrill Gorcunov <gorcunov@gmail.com> wrote:
>>>>> Message from syslogd@ht at May 14 09:39:32 ...
>>>>>  kernel:[  thanks Jaswinder, i'll take a look, meanwhile if you get a chance to
>>>> test latest -tip/master it would be great
>>>>
>>>
>>> it's found that we have problem in cache events, we're working on
>>> that, thanks for report!
>>>
>>
>> Great !!
>>
>> If you need any help from my side, please let me know. Even though it
>> is very hot here, I will try my best ;-)
>>
>> Thanks,
>> --
>> Jaswinder Singh.
>>
>
> Ming is narrowing down the guilty commit. I thought about my last
> patch related to escr hashing, but it shouldn't bring such effect.
> Hmm...
>

Jaswander, if you manage to bisect it -- this would be just great.

Re: Performance Events hangs with Intel P4 system

[Cyrill Gorcunov]

On Friday, May 14, 2010, Cyrill Gorcunov <gorcunov@gmail.com> wrote:
> On Friday, May 14, 2010, Cyrill Gorcunov <gorcunov@gmail.com> wrote:
>> On Friday, May 14, 2010, Jaswinder Singh Rajput
>> <jaswinderlinux@gmail.com> wrote:
>>> Hello Cyrill,
>>>
>>> On Fri, May 14, 2010 at 11:53 AM, Cyrill Gorcunov <gorcunov@gmail.com> wrote:
>>>> On Friday, May 14, 2010, Cyrill Gorcunov <gorcunov@gmail.com> wrote:
>>>>>> Message from syslogd@ht at May 14 09:39:32 ...
>>>>>>  kernel:[  thanks Jaswinder, i'll take a look, meanwhile if you get a chance to
>>>>> test latest -tip/master it would be great
>>>>>
>>>>
>>>> it's found that we have problem in cache events, we're working on
>>>> that, thanks for report!
>>>>
>>>
>>> Great !!
>>>
>>> If you need any help from my side, please let me know. Even though it
>>> is very hot here, I will try my best ;-)
>>>
>>> Thanks,
>>> --
>>> Jaswinder Singh.
>>>
>>
>> Ming is narrowing down the guilty commit. I thought about my last
>> patch related to escr hashing, but it shouldn't bring such effect.
>> Hmm...
>>
>
> Jaswander, if you manage to bisect it -- this would be just great.
>

ok, Ming has spotted the problem. The patch will be there soon.

Re: Performance Events hangs with Intel P4 system

[Lin Ming]

On Fri, 2010-05-14 at 18:04 +0800, Cyrill Gorcunov wrote:
> On Friday, May 14, 2010, Cyrill Gorcunov <gorcunov@gmail.com> wrote:
> > On Friday, May 14, 2010, Jaswinder Singh Rajput
> > <jaswinderlinux@gmail.com> wrote:
> >> Hello Cyrill,
> >>
> >> On Fri, May 14, 2010 at 11:53 AM, Cyrill Gorcunov <gorcunov@gmail.com> wrote:
> >>> On Friday, May 14, 2010, Cyrill Gorcunov <gorcunov@gmail.com> wrote:
> >>>>> Message from syslogd@ht at May 14 09:39:32 ...
> >>>>>  kernel:[  thanks Jaswinder, i'll take a look, meanwhile if you get a chance to
> >>>> test latest -tip/master it would be great
> >>>>
> >>>
> >>> it's found that we have problem in cache events, we're working on
> >>> that, thanks for report!
> >>>
> >>
> >> Great !!
> >>
> >> If you need any help from my side, please let me know. Even though it
> >> is very hot here, I will try my best ;-)
> >>
> >> Thanks,
> >> --
> >> Jaswinder Singh.
> >>
> >
> > Ming is narrowing down the guilty commit. I thought about my last
> > patch related to escr hashing, but it shouldn't bring such effect.
> > Hmm...
> >
> 
> Jaswander, if you manage to bisect it -- this would be just great.

Hi, Jaswinder 

Below patch fixes the regression on my P4 machine.
Would you please have a try it?

Thanks.
---

Subject: [PATCH] x86, perf: P4 PMU -- fix wrong compare

p4_event_bind::cntr is "unsigned char".
But p4_next_cntr has return type of "int".
So the explicit conversion is needed to get the correct result. 

Signed-off-by: Lin Ming <ming.m.lin@intel.com>
---
 arch/x86/kernel/cpu/perf_event_p4.c |    4 ++--
 1 files changed, 2 insertions(+), 2 deletions(-)

diff --git a/arch/x86/kernel/cpu/perf_event_p4.c b/arch/x86/kernel/cpu/perf_event_p4.c
index cb875b1..9358793 100644
--- a/arch/x86/kernel/cpu/perf_event_p4.c
+++ b/arch/x86/kernel/cpu/perf_event_p4.c
@@ -780,7 +780,7 @@ static int p4_pmu_schedule_events(struct cpu_hw_events *cpuc, int n, int *assign
 		if (unlikely(escr_idx == -1))
 			goto done;
 
-		if (hwc->idx != -1 && !p4_should_swap_ts(hwc->config, cpu)) {
+		if (hwc->idx != (unsigned char)-1 && !p4_should_swap_ts(hwc->config, cpu)) {
 			cntr_idx = hwc->idx;
 			if (assign)
 				assign[i] = hwc->idx;
@@ -788,7 +788,7 @@ static int p4_pmu_schedule_events(struct cpu_hw_events *cpuc, int n, int *assign
 		}
 
 		cntr_idx = p4_next_cntr(thread, used_mask, bind);
-		if (cntr_idx == -1 || test_bit(escr_idx, escr_mask))
+		if (cntr_idx == (unsigned char)-1 || test_bit(escr_idx, escr_mask))
 			goto done;
 
 		p4_pmu_swap_config_ts(hwc, cpu);

Re: Performance Events hangs with Intel P4 system

[Ingo Molnar]

* Lin Ming <ming.m.lin@intel.com> wrote:

> p4_event_bind::cntr is "unsigned char".
> But p4_next_cntr has return type of "int".
> So the explicit conversion is needed to get the correct result. 

> @@ -780,7 +780,7 @@ static int p4_pmu_schedule_events(struct cpu_hw_events *cpuc, int n, int *assign
>  		if (unlikely(escr_idx == -1))
>  			goto done;
>  
> -		if (hwc->idx != -1 && !p4_should_swap_ts(hwc->config, cpu)) {
> +		if (hwc->idx != (unsigned char)-1 && !p4_should_swap_ts(hwc->config, cpu)) {

That cast is _extremely_ ugly. Please fix the signedness of the types instead.

	Ingo

Re: Performance Events hangs with Intel P4 system

[Cyrill Gorcunov]

On Friday, May 14, 2010, Ingo Molnar <mingo@elte.hu> wrote:
>
> * Lin Ming <ming.m.lin@intel.com> wrote:
>
>> p4_event_bind::cntr is "unsigned char".
>> But p4_next_cntr has return type of "int".
>> So the explicit conversion is needed to get the correct result.
>
>> @@ -780,7 +780,7 @@ static int p4_pmu_schedule_events(struct cpu_hw_events *cpuc, int n, int *assign
>>               if (unlikely(escr_idx == -1))
>>                       goto done;
>>
>> -             if (hwc->idx != -1 && !p4_should_swap_ts(hwc->config, cpu)) {
>> +             if (hwc->idx != (unsigned char)-1 && !p4_should_swap_ts(hwc->config, cpu)) {
>
> That cast is _extremely_ ugly. Please fix the signedness of the types instead.
>
>         Ingo
>
it's completely my fault. I'll check all occurences of this silly and
shame on me bug in a couple of hours. Sorry for this.

Re: Performance Events hangs with Intel P4 system

[Cyrill Gorcunov]

On Fri, May 14, 2010 at 01:56:55PM +0200, Ingo Molnar wrote:
> 
> * Lin Ming <ming.m.lin@intel.com> wrote:
> 
> > p4_event_bind::cntr is "unsigned char".
> > But p4_next_cntr has return type of "int".
> > So the explicit conversion is needed to get the correct result. 
> 
> > @@ -780,7 +780,7 @@ static int p4_pmu_schedule_events(struct cpu_hw_events *cpuc, int n, int *assign
> >  		if (unlikely(escr_idx == -1))
> >  			goto done;
> >  
> > -		if (hwc->idx != -1 && !p4_should_swap_ts(hwc->config, cpu)) {
> > +		if (hwc->idx != (unsigned char)-1 && !p4_should_swap_ts(hwc->config, cpu)) {
> 
> That cast is _extremely_ ugly. Please fix the signedness of the types instead.
> 
> 	Ingo
> 

Ingo, what about this one? Jaswinder could you give it a shot (untested)?

	-- Cyrill
---
[PATCH -tip/master] x86,perf: P4 PMU - fix counters allocation logic and sign issue

Jaswinder reported GP:
|
| Message from syslogd@ht at May 14 09:39:32 ...
| kernel:[  314.908612] EIP: [<c100ccca>]
| x86_perf_event_set_period+0x19d/0x1b2 SS:ESP 0068:edac3d70
|

Ming has narrowed it down to comparation issue between signed/unsigned values.
As result event index reaches value 255 which in turn leads to GP fault.

Also it was found that p4_next_cntr has a broken logic and should return
counter index if only it was not yet borrowed for another event.

Reported-by: Jaswinder Singh Rajput <jaswinderlinux@gmail.com>
Reported-by: Lin Ming <ming.m.lin@intel.com>
Bisected-by: Lin Ming <ming.m.lin@intel.com>
CC: Peter Zijlstra <a.p.zijlstra@chello.nl>
CC: Ingo Molnar <mingo@elte.hu>
CC: Frederic Weisbecker <fweisbec@gmail.com>
Signed-off-by: Cyrill Gorcunov <gorcunov@openvz.org>
---
 arch/x86/kernel/cpu/perf_event_p4.c |    8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)

Index: linux-2.6.git/arch/x86/kernel/cpu/perf_event_p4.c
=====================================================================
--- linux-2.6.git.orig/arch/x86/kernel/cpu/perf_event_p4.c
+++ linux-2.6.git/arch/x86/kernel/cpu/perf_event_p4.c
@@ -18,7 +18,7 @@
 struct p4_event_bind {
 	unsigned int opcode;			/* Event code and ESCR selector */
 	unsigned int escr_msr[2];		/* ESCR MSR for this event */
-	unsigned char cntr[2][P4_CNTR_LIMIT];	/* counter index (offset), -1 on abscence */
+	char cntr[2][P4_CNTR_LIMIT];		/* counter index (offset), -1 on abscence */
 };
 
 struct p4_cache_event_bind {
@@ -747,11 +747,11 @@ static int p4_get_escr_idx(unsigned int 
 static int p4_next_cntr(int thread, unsigned long *used_mask,
 			struct p4_event_bind *bind)
 {
-	int i = 0, j;
+	int i, j;
 
 	for (i = 0; i < P4_CNTR_LIMIT; i++) {
-		j = bind->cntr[thread][i++];
-		if (j == -1 || !test_bit(j, used_mask))
+		j = (int)bind->cntr[thread][i];
+		if (j != -1 && !test_bit(j, used_mask))
 			return j;
 	}
 

Re: Performance Events hangs with Intel P4 system

[Cyrill Gorcunov]

On Fri, May 14, 2010 at 05:52:40PM +0400, Cyrill Gorcunov wrote:
> On Fri, May 14, 2010 at 01:56:55PM +0200, Ingo Molnar wrote:
> > 
> > * Lin Ming <ming.m.lin@intel.com> wrote:
> > 
> > > p4_event_bind::cntr is "unsigned char".
> > > But p4_next_cntr has return type of "int".
> > > So the explicit conversion is needed to get the correct result. 
> > 
> > > @@ -780,7 +780,7 @@ static int p4_pmu_schedule_events(struct cpu_hw_events *cpuc, int n, int *assign
> > >  		if (unlikely(escr_idx == -1))
> > >  			goto done;
> > >  
> > > -		if (hwc->idx != -1 && !p4_should_swap_ts(hwc->config, cpu)) {
> > > +		if (hwc->idx != (unsigned char)-1 && !p4_should_swap_ts(hwc->config, cpu)) {
> > 
> > That cast is _extremely_ ugly. Please fix the signedness of the types instead.
> > 
> > 	Ingo
> > 
> 
> Ingo, what about this one? Jaswinder could you give it a shot (untested)?
> 
> 	-- Cyrill
> ---
>

This one should be much better
---
[PATCH -tip/master] x86,perf: P4 PMU - fix counters allocation logic and sign issue

Jaswinder reported GP:
|
| Message from syslogd@ht at May 14 09:39:32 ...
| kernel:[  314.908612] EIP: [<c100ccca>]
| x86_perf_event_set_period+0x19d/0x1b2 SS:ESP 0068:edac3d70
|

Ming has narrowed it down to comparision issue between arguments with
different sizes. As result event index reaches value 255 which in turn
leads to GP fault.

Also it was found that p4_next_cntr has a broken logic and should return
counter index if only it was not yet borrowed for another event.

Reported-by: Jaswinder Singh Rajput <jaswinderlinux@gmail.com>
Reported-by: Lin Ming <ming.m.lin@intel.com>
Bisected-by: Lin Ming <ming.m.lin@intel.com>
CC: Peter Zijlstra <a.p.zijlstra@chello.nl>
CC: Ingo Molnar <mingo@elte.hu>
CC: Frederic Weisbecker <fweisbec@gmail.com>
Signed-off-by: Cyrill Gorcunov <gorcunov@openvz.org>
---
 arch/x86/kernel/cpu/perf_event_p4.c |    6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

Index: linux-2.6.git/arch/x86/kernel/cpu/perf_event_p4.c
=====================================================================
--- linux-2.6.git.orig/arch/x86/kernel/cpu/perf_event_p4.c
+++ linux-2.6.git/arch/x86/kernel/cpu/perf_event_p4.c
@@ -747,11 +747,11 @@ static int p4_get_escr_idx(unsigned int 
 static int p4_next_cntr(int thread, unsigned long *used_mask,
 			struct p4_event_bind *bind)
 {
-	int i = 0, j;
+	int i, j;
 
 	for (i = 0; i < P4_CNTR_LIMIT; i++) {
-		j = bind->cntr[thread][i++];
-		if (j == -1 || !test_bit(j, used_mask))
+		j = bind->cntr[thread][i];
+		if (j != (unsigned char)-1 && !test_bit(j, used_mask))
 			return j;
 	}
 

Re: Performance Events hangs with Intel P4 system

[Jaswinder Singh Rajput]

Hello Cyrill,

On Fri, May 14, 2010 at 8:22 PM, Cyrill Gorcunov <gorcunov@gmail.com> wrote:
> [PATCH -tip/master] x86,perf: P4 PMU - fix counters allocation logic and sign issue
>
> Jaswinder reported GP:
> |
> | Message from syslogd@ht at May 14 09:39:32 ...
> | kernel:[  314.908612] EIP: [<c100ccca>]
> | x86_perf_event_set_period+0x19d/0x1b2 SS:ESP 0068:edac3d70
> |
>
> Ming has narrowed it down to comparision issue between arguments with
> different sizes. As result event index reaches value 255 which in turn
> leads to GP fault.
>
> Also it was found that p4_next_cntr has a broken logic and should return
> counter index if only it was not yet borrowed for another event.
>
> Reported-by: Jaswinder Singh Rajput <jaswinderlinux@gmail.com>
> Reported-by: Lin Ming <ming.m.lin@intel.com>
> Bisected-by: Lin Ming <ming.m.lin@intel.com>
> CC: Peter Zijlstra <a.p.zijlstra@chello.nl>
> CC: Ingo Molnar <mingo@elte.hu>
> CC: Frederic Weisbecker <fweisbec@gmail.com>
> Signed-off-by: Cyrill Gorcunov <gorcunov@openvz.org>
> ---

Yes, this works for me. Now I am not getting general protection fault.

It seems hardware events are not supported for P4 yet.

$ ./perf stat -e
cycles,instructions,cache-references,cache-misses,branches,branch-misses,bus-cycles
ls > /dev/null

 Performance counter stats for 'ls':

  <not counted>  cycles
         601636  instructions             #      0.000 IPC    (scaled
from 91.80%)
  <not counted>  cache-references
  <not counted>  cache-misses
  <not counted>  branches
  <not counted>  branch-misses
  <not counted>  bus-cycles

    0.003364910  seconds time elapsed


Thanks,
--
Jaswinder Singh.



>  arch/x86/kernel/cpu/perf_event_p4.c |    6 +++---
>  1 file changed, 3 insertions(+), 3 deletions(-)
>
> Index: linux-2.6.git/arch/x86/kernel/cpu/perf_event_p4.c
> =====================================================================
> --- linux-2.6.git.orig/arch/x86/kernel/cpu/perf_event_p4.c
> +++ linux-2.6.git/arch/x86/kernel/cpu/perf_event_p4.c
> @@ -747,11 +747,11 @@ static int p4_get_escr_idx(unsigned int
>  static int p4_next_cntr(int thread, unsigned long *used_mask,
>                        struct p4_event_bind *bind)
>  {
> -       int i = 0, j;
> +       int i, j;
>
>        for (i = 0; i < P4_CNTR_LIMIT; i++) {
> -               j = bind->cntr[thread][i++];
> -               if (j == -1 || !test_bit(j, used_mask))
> +               j = bind->cntr[thread][i];
> +               if (j != (unsigned char)-1 && !test_bit(j, used_mask))
>                        return j;
>        }
>
>

Re: Performance Events hangs with Intel P4 system

[Cyrill Gorcunov]

On Fri, May 14, 2010 at 09:52:28PM +0530, Jaswinder Singh Rajput wrote:
... 
> Yes, this works for me. Now I am not getting general protection fault.
> 
> It seems hardware events are not supported for P4 yet.
> 
> $ ./perf stat -e
> cycles,instructions,cache-references,cache-misses,branches,branch-misses,bus-cycles
> ls > /dev/null
> 
>  Performance counter stats for 'ls':
> 
>   <not counted>  cycles
>          601636  instructions             #      0.000 IPC    (scaled
> from 91.80%)
>   <not counted>  cache-references
>   <not counted>  cache-misses
>   <not counted>  branches
>   <not counted>  branch-misses
>   <not counted>  bus-cycles
> 
>     0.003364910  seconds time elapsed

Thanks Jaswinder,

it means counters management somehow screwed at moment (it was working before).
I'm working on it.

	-- Cyrill

Re: Performance Events hangs with Intel P4 system

[Jaswinder Singh Rajput]

Hello Cyrill,

On Fri, May 14, 2010 at 9:58 PM, Cyrill Gorcunov <gorcunov@gmail.com> wrote:
> On Fri, May 14, 2010 at 09:52:28PM +0530, Jaswinder Singh Rajput wrote:
> ...
>> Yes, this works for me. Now I am not getting general protection fault.
>>
>> It seems hardware events are not supported for P4 yet.
>>
>> $ ./perf stat -e
>> cycles,instructions,cache-references,cache-misses,branches,branch-misses,bus-cycles
>> ls > /dev/null
>>
>>  Performance counter stats for 'ls':
>>
>>   <not counted>  cycles
>>          601636  instructions             #      0.000 IPC    (scaled
>> from 91.80%)
>>   <not counted>  cache-references
>>   <not counted>  cache-misses
>>   <not counted>  branches
>>   <not counted>  branch-misses
>>   <not counted>  bus-cycles
>>
>>     0.003364910  seconds time elapsed
>
> Thanks Jaswinder,
>
> it means counters management somehow screwed at moment (it was working before).
> I'm working on it.
>

Ok, I added few more events and now I am able see few hardware events :

$ ./perf stat -e
cycles,instructions,cache-references,cache-misses,branches,branch-misses,bus-cycles,L1-dcache-loads,L1-dcache-load-misses,L1-dcache-stores,L1-dcache-store-misses,L1-dcache-prefetches,L1-dcache-prefetch-misses,L1-icache-loads,L1-icache-load-misses,L1-icache-prefetches,L1-icache-prefetch-misses,LLC-loads,LLC-load-misses,LLC-stores,LLC-store-misses,LLC-prefetches,LLC-prefetch-misses,dTLB-loads,dTLB-load-misses,dTLB-stores,dTLB-store-misses,dTLB-prefetches,dTLB-prefetch-misses,iTLB-loads,iTLB-load-misses
ls -lR /dev > /dev/null

 Performance counter stats for 'ls -lR /dev':

  <not counted>  cycles
       10159428  instructions             #      0.000 IPC    (scaled
from 11.71%)
  <not counted>  cache-references
  <not counted>  cache-misses
        2160905  branches                  (scaled from 9.71%)
          80630  branch-misses            #      3.731 %      (scaled
from 15.68%)
  <not counted>  bus-cycles
  <not counted>  L1-dcache-loads
          90289  L1-dcache-load-misses     (scaled from 14.23%)
  <not counted>  L1-dcache-stores
  <not counted>  L1-dcache-store-misses
  <not counted>  L1-dcache-prefetches
  <not counted>  L1-dcache-prefetch-misses
  <not counted>  L1-icache-loads
  <not counted>  L1-icache-load-misses
  <not counted>  L1-icache-prefetches
  <not counted>  L1-icache-prefetch-misses
  <not counted>  LLC-loads
           3162  LLC-load-misses           (scaled from 12.40%)
  <not counted>  LLC-stores
  <not counted>  LLC-store-misses
  <not counted>  LLC-prefetches
  <not counted>  LLC-prefetch-misses
  <not counted>  dTLB-loads
          21825  dTLB-load-misses          (scaled from 11.04%)
  <not counted>  dTLB-stores
           1386  dTLB-store-misses         (scaled from 9.96%)
  <not counted>  dTLB-prefetches
  <not counted>  dTLB-prefetch-misses
  <not counted>  iTLB-loads
  <not counted>  iTLB-load-misses

    0.019403422  seconds time elapsed


Thanks, good work :-)

--
Jaswinder Singh.

Re: Performance Events hangs with Intel P4 system

[Cyrill Gorcunov]

On Fri, May 14, 2010 at 10:06:44PM +0530, Jaswinder Singh Rajput wrote:
...
> >
> > Thanks Jaswinder,
> >
> > it means counters management somehow screwed at moment (it was working before).
> > I'm working on it.
> >
> 
> Ok, I added few more events and now I am able see few hardware events :
> 
...
Thanks, but I need to figure out where they hide in first place. Will keep
you in touch.

	-- Cyrill

Re: Performance Events hangs with Intel P4 system

[Jaswinder Singh Rajput]

Hello Cyrill,

On Fri, May 14, 2010 at 10:10 PM, Cyrill Gorcunov <gorcunov@gmail.com> wrote:
> On Fri, May 14, 2010 at 10:06:44PM +0530, Jaswinder Singh Rajput wrote:
> ...
>> >
>> > Thanks Jaswinder,
>> >
>> > it means counters management somehow screwed at moment (it was working before).
>> > I'm working on it.
>> >
>>
>> Ok, I added few more events and now I am able see few hardware events :
>>
> ...
> Thanks, but I need to figure out where they hide in first place. Will keep
> you in touch.
>

Earlier I given the command 'ls' which was very small only 601636
instructions so branches
and branch-misses came up with <not counted>

When I given a bigger command 'ls -lR /dev' with 10159428
instructions, so I get branches and branch-misses count.

Thanks,
--
Jaswinder Singh.

Re: Performance Events hangs with Intel P4 system

[Cyrill Gorcunov]

On Fri, May 14, 2010 at 11:18:52PM +0530, Jaswinder Singh Rajput wrote:
... 
> Earlier I given the command 'ls' which was very small only 601636
> instructions so branches
> and branch-misses came up with <not counted>
> 
> When I given a bigger command 'ls -lR /dev' with 10159428
> instructions, so I get branches and branch-misses count.
> 
> Thanks,
> --
> Jaswinder Singh.
> 

Jaswinder, could you give this patch a shot so that I will be
able to put your tested-by tag and send it upstream? (it's slightly
tuned from previous but should be same on bin level). This GP
issue need to be fixed before any other things.

	-- Cyrill
---
 arch/x86/kernel/cpu/perf_event_p4.c |    8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)

Index: linux-2.6.git/arch/x86/kernel/cpu/perf_event_p4.c
=====================================================================
--- linux-2.6.git.orig/arch/x86/kernel/cpu/perf_event_p4.c
+++ linux-2.6.git/arch/x86/kernel/cpu/perf_event_p4.c
@@ -18,7 +18,7 @@
 struct p4_event_bind {
 	unsigned int opcode;			/* Event code and ESCR selector */
 	unsigned int escr_msr[2];		/* ESCR MSR for this event */
-	unsigned char cntr[2][P4_CNTR_LIMIT];	/* counter index (offset), -1 on abscence */
+	char cntr[2][P4_CNTR_LIMIT];		/* counter index (offset), -1 on abscence */
 };
 
 struct p4_cache_event_bind {
@@ -747,11 +747,11 @@ static int p4_get_escr_idx(unsigned int 
 static int p4_next_cntr(int thread, unsigned long *used_mask,
 			struct p4_event_bind *bind)
 {
-	int i = 0, j;
+	int i, j;
 
 	for (i = 0; i < P4_CNTR_LIMIT; i++) {
-		j = bind->cntr[thread][i++];
-		if (j == -1 || !test_bit(j, used_mask))
+		j = bind->cntr[thread][i];
+		if (j != -1 && !test_bit(j, used_mask))
 			return j;
 	}
 

Re: Performance Events hangs with Intel P4 system

[Jaswinder Singh Rajput]

Hello Cyrill,

On Fri, May 14, 2010 at 11:31 PM, Cyrill Gorcunov <gorcunov@gmail.com> wrote:
>
>
> Jaswinder, could you give this patch a shot so that I will be
> able to put your tested-by tag and send it upstream? (it's slightly
> tuned from previous but should be same on bin level). This GP
> issue need to be fixed before any other things.
>
>        -- Cyrill
> ---
>  arch/x86/kernel/cpu/perf_event_p4.c |    8 ++++----
>  1 file changed, 4 insertions(+), 4 deletions(-)
>
> Index: linux-2.6.git/arch/x86/kernel/cpu/perf_event_p4.c
> =====================================================================
> --- linux-2.6.git.orig/arch/x86/kernel/cpu/perf_event_p4.c
> +++ linux-2.6.git/arch/x86/kernel/cpu/perf_event_p4.c
> @@ -18,7 +18,7 @@
>  struct p4_event_bind {
>        unsigned int opcode;                    /* Event code and ESCR selector */
>        unsigned int escr_msr[2];               /* ESCR MSR for this event */
> -       unsigned char cntr[2][P4_CNTR_LIMIT];   /* counter index (offset), -1 on abscence */
> +       char cntr[2][P4_CNTR_LIMIT];            /* counter index (offset), -1 on abscence */
>  };
>
>  struct p4_cache_event_bind {
> @@ -747,11 +747,11 @@ static int p4_get_escr_idx(unsigned int
>  static int p4_next_cntr(int thread, unsigned long *used_mask,
>                        struct p4_event_bind *bind)
>  {
> -       int i = 0, j;
> +       int i, j;
>
>        for (i = 0; i < P4_CNTR_LIMIT; i++) {
> -               j = bind->cntr[thread][i++];
> -               if (j == -1 || !test_bit(j, used_mask))
> +               j = bind->cntr[thread][i];
> +               if (j != -1 && !test_bit(j, used_mask))
>                        return j;
>        }
>
>

Yes, this patch fixes the GP issue.

Thanks,
--
Jaswinder Singh.

Re: Performance Events hangs with Intel P4 system

[Cyrill Gorcunov]

On Fri, May 14, 2010 at 10:06:44PM +0530, Jaswinder Singh Rajput wrote:
... 
> Ok, I added few more events and now I am able see few hardware events :
> 
> $ ./perf stat -e
> cycles,instructions,cache-references,cache-misses,branches,branch-misses,bus-cycles,L1-dcache-loads,L1-dcache-load-misses,L1-dcache-stores,L1-dcache-store-misses,L1-dcache-prefetches,L1-dcache-prefetch-misses,L1-icache-loads,L1-icache-load-misses,L1-icache-prefetches,L1-icache-prefetch-misses,LLC-loads,LLC-load-misses,LLC-stores,LLC-store-misses,LLC-prefetches,LLC-prefetch-misses,dTLB-loads,dTLB-load-misses,dTLB-stores,dTLB-store-misses,dTLB-prefetches,dTLB-prefetch-misses,iTLB-loads,iTLB-load-misses
> ls -lR /dev > /dev/null
> 

ok, they work as expected. Could you try only "cycles"? The things are
that different events may attempt to borrow same resources already allocated
for another event, ie they can't run simultaneously. And iirc we encode
only a subset of chache events. Though all "general" events should work.
(by "general" events I mean "cycles", "instructions", "cache-references",
 "cache-misses", "branch-instructions", "branch-misses", "bus-cycles").
Could you check them one-by-one?

	-- Cyrill