Re: [PATCH] tools/perf/test: Check for perf stat return code in perf all PMU test

["Mi, Dapeng" <dapeng1.mi@linux.intel.com>]

On 4/7/2026 8:48 AM, Mi, Dapeng wrote:
> On 4/3/2026 11:39 PM, Ian Rogers wrote:
>> On Fri, Apr 3, 2026 at 12:36 AM Mi, Dapeng <dapeng1.mi@linux.intel.com> wrote:
>>> On 4/3/2026 1:32 AM, Falcon, Thomas wrote:
>>>> On Wed, 2026-04-01 at 13:40 -0700, Ian Rogers wrote:
>>>>> On Mon, Mar 23, 2026 at 3:40 AM Venkat <venkat88@linux.ibm.com>
>>>>> wrote:
>>>>>>> On 15 Mar 2026, at 4:27 PM, Athira Rajeev
>>>>>>> <atrajeev@linux.ibm.com> wrote:
>>>>>>>
>>>>>>> Currently in "perf all PMU test", for "perf stat -e <event>
>>>>>>> true",
>>>>>>> below checks are done:
>>>>>>> - if return code is zero, look for "not supported" to decide pass
>>>>>>>  scenario
>>>>>>> - check for "not supported" to ignore the event
>>>>>>> - looks for "No permission to enable" to skip the event.
>>>>>>> - If output has "Bad event name", fail the test.
>>>>>>> - Use "Access to performance monitoring and observability
>>>>>>> operations is
>>>>>>>  limited." to ignore fail due to access limitations
>>>>>>>
>>>>>>> If we failed to see event and it is supported, retries with
>>>>>>> longer
>>>>>>> workload "perf bench internals synthesize".
>>>>>>> - Here if output has <event>, the test is a pass.
>>>>>>>
>>>>>>> Snippet of code check:
>>>>>>>  ```
>>>>>>>  output=$(perf stat -e "$p" perf bench internals synthesize 2>&1)
>>>>>>>  if echo "$output" | grep -q "$p"
>>>>>>>  ```
>>>>>>> - if output doesn't have event printed in logs, considers it
>>>>>>> fail.
>>>>>>>
>>>>>>> But this results in false pass for events in some cases.
>>>>>>> Example, if perf stat fails as below:
>>>>>>>
>>>>>>> # ./perf stat -e pmu/event/  true
>>>>>>> event syntax error: 'pmu/event/'
>>>>>>>                     \___ Bad event or PMU
>>>>>>>
>>>>>>> Unable to find PMU or event on a PMU of 'pmu'
>>>>>>> Run 'perf list' for a list of valid events
>>>>>>>
>>>>>>>  Usage: perf stat [<options>] [<command>]
>>>>>>>
>>>>>>>    -e, --event <event>   event selector. use 'perf list' to list
>>>>>>> available events
>>>>>>> # echo $?
>>>>>>> 129
>>>>>>>
>>>>>>> Since this has non-zero return code and doesn't have the
>>>>>>> fail strings being checked in the test, it will enter check using
>>>>>>> longer workload. and since the output fail log has event, it
>>>>>>> declares test as "supported".
>>>>>>>
>>>>>>> Since all the fail strings can't be added in the check, update
>>>>>>> the testcase to check return code before proceeding to longer
>>>>>>> workload run.
>>>>>>>
>>>>>>> Another missing scenario is when system wide monitoring is
>>>>>>> supported
>>>>>>> example:
>>>>>>> # ./perf stat -e pmu/event/ true
>>>>>>> Error:
>>>>>>> No supported events found.
>>>>>>>  Unsupported event (pmu/event/H) in per-thread mode, enable
>>>>>>> system wide with '-a'.
>>>>>>>
>>>>>>> Update testcase to check with "perf stat -a -e $p" as well
>>>>>>>
>>>>>>> Signed-off-by: Athira Rajeev <atrajeev@linux.ibm.com>
>>>>>>> ---
>>>>>> Tested this patch.
>>>>>>
>>>>>>
>>>>>> With this patch:
>>>>>>
>>>>>> Testing hv_24x7/CPM_ADJUNCT_INST/ -- perf stat failed with non-zero
>>>>>> return code
>>>>>> Testing hv_24x7/CPM_ADJUNCT_PCYC/ -- perf stat failed with non-zero
>>>>>> return code
>>>>>>
>>>>>>
>>>>>>
>>>>>> Tested-by: Venkat Rao Bagalkote <venkat88@linux.ibm.com>
>>>>> Testing on an Intel Alderlake the test is now failing:
>>>>> ```
>>>>> ...
>>>>> Testing offcore_requests_outstanding.l3_miss_demand_data_rd --
>>>>> supported
>>>>> Testing ocr.full_streaming_wr.any_response -- perf stat failed with
>>>>> non-zero return code
>>>>> Testing ocr.partial_streaming_wr.any_response -- perf stat failed
>>>>> with
>>>>> non-zero return code
>>>>> Testing ocr.streaming_wr.any_response -- supported
>>>>> ...
>>>>> ```
>>>>>
>>>>> Running `perf stat` manually reveals an issue with the event:
>>>>> ```
>>>>> $ sudo perf stat -vv -e ocr.full_streaming_wr.any_response -a sleep
>>>>> 1
>>>>> Using CPUID GenuineIntel-6-B7-1
>>>>> Attempt to add: cpu_atom/ocr.full_streaming_wr.any_response/
>>>>> ..after resolving event:
>>>>> cpu_atom/event=0xb7,period=0x186a3,umask=0x1,offcore_rsp=0x8000000100
>>>>> 00/
>>>>> ocr.full_streaming_wr.any_response ->
>>>>> cpu_atom/ocr.full_streaming_wr.any_response/
>>>>> Control descriptor is not initialized
>>>>> ------------------------------------------------------------
>>>>> perf_event_attr:
>>>>>  type                             10 (cpu_atom)
>>>>>  size                             144
>>>>> ------------------------------------------------------------
>>>>> perf_event_attr:
>>>>>  type                             0 (PERF_TYPE_HARDWARE)
>>>>>  config                           0xa00000000
>>>>> (cpu_atom/PERF_COUNT_HW_CPU_CYCLES/)
>>>>>  disabled                         1
>>>>> ------------------------------------------------------------
>>>>> sys_perf_event_open: pid 0  cpu -1  group_fd -1  flags 0x8 = 3
>>>>> ------------------------------------------------------------
>>>>> perf_event_attr:
>>>>>  type                             0 (PERF_TYPE_HARDWARE)
>>>>>  config                           0x400000000
>>>>> (cpu_core/PERF_COUNT_HW_CPU_CYCLES/)
>>>>>  disabled                         1
>>>>> ------------------------------------------------------------
>>>>> sys_perf_event_open: pid 0  cpu -1  group_fd -1  flags 0x8 = 3
>>>>>  config                           0x1b7
>>>>> (ocr.demand_data_rd.l3_hit.snoop_hit_no_fwd)
>>>>>  sample_type                      IDENTIFIER
>>>>>  read_format
>>>>> TOTAL_TIME_ENABLED|TOTAL_TIME_RUNNING
>>>>>  disabled                         1
>>>>>  inherit                          1
>>>>>  { bp_addr, config1 }             0x800000010000
>>>>> ------------------------------------------------------------
>>>>> sys_perf_event_open: pid -1  cpu 16  group_fd -1  flags 0x8
>>>>> sys_perf_event_open failed, error -22
>>>>> switching off deferred callchain support
>>>>> Warning:
>>>>> ocr.full_streaming_wr.any_response event is not supported by the
>>>>> kernel.
>>>>> The sys_perf_event_open() syscall failed for event
>>>>> (ocr.full_streaming_wr.any_response): Invalid argument
>>>>> "dmesg | grep -i perf" may provide additional information.
>>>>>
>>>>> Error:
>>>>> No supported events found.
>>>>> The sys_perf_event_open() syscall failed for event
>>>>> (ocr.full_streaming_wr.any_response): Invalid argument
>>>>> "dmesg | grep -i perf" may provide additional information.
>>>>> ```
>>>>>
>>>>> This looks like a latent Intel cpu_atom PMU bug. Thomas, wdyt?
>>> Hmm, it looks the error is caused by the invalid bitmask of OFFCORE_RSP_x
>>> MSRs. Currently the valid bitmask of OFFCORE_RSP_x MSR is set to
>>> 0x3fffffffff in intel_grt_extra_regs[], while the msr value is set
>>> 0x800000010000 for the ocr.full_streaming_wr.any_response event. The bit 47
>>> is recognized an invalid bit and then abort the event creation.
>>>
>>> Base on the description "Table 21-56. MSR_OFFCORE_RSPx Request Type
>>> Definition" in SDM, bit 47 should be a valid bit now. Suppose bit 47 should
>>> not be a valid bit when adding the ADL PMU support, but it's updated and
>>> becomes valid later.
>>>
>>> Along with the constant updates of perf event lists
>>> (https://github.com/intel/perfmon), we have noticed there are mismatches
>>> more or less between the driver hardcoded events and perfmon event list.
>>> Currently we are summarizing the mismatches. Once these mismatches are
>>> finalized. we would submit a patchset to fix these mismatches.
>> That's great, if it takes too long perhaps we could just remove the
>> events for now.
> Suppose it won't be too long. I plan to post the patchset in next release
> cycle. The code changes are simple but need much time to verify on all
> kinds of platforms. Thanks.

The patch
(https://lore.kernel.org/all/20260515061143.338553-5-dapeng1.mi@linux.intel.com/)
would fix this issue. Thanks.


>
>
>> Thanks,
>> Ian
>>
>>> Thanks.
>>>
>>>> +Dapeng, Zide, Andi
>>>>
>>>> Thanks,
>>>> Tom
>>>>
>>>>> Thanks,
>>>>> Ian
>>>>>
>>>>>> Regards,
>>>>>> Venkat.
>>>>>>
>>>>>>
>>>>>>
>>>>>>> tools/perf/tests/shell/stat_all_pmu.sh | 20 ++++++++++++++++++++
>>>>>>> 1 file changed, 20 insertions(+)
>>>>>>>
>>>>>>> diff --git a/tools/perf/tests/shell/stat_all_pmu.sh
>>>>>>> b/tools/perf/tests/shell/stat_all_pmu.sh
>>>>>>> index 9c466c0efa85..6c4d59cbfa5f 100755
>>>>>>> --- a/tools/perf/tests/shell/stat_all_pmu.sh
>>>>>>> +++ b/tools/perf/tests/shell/stat_all_pmu.sh
>>>>>>> @@ -53,6 +53,26 @@ do
>>>>>>>     continue
>>>>>>>   fi
>>>>>>>
>>>>>>> +  # check with system wide if it is supported.
>>>>>>> +  output=$(perf stat -a -e "$p" true 2>&1)
>>>>>>> +  stat_result=$?
>>>>>>> +  if echo "$output" | grep -q "not supported"
>>>>>>> +  then
>>>>>>> +    # Event not supported, so ignore.
>>>>>>> +    echo "not supported"
>>>>>>> +    continue
>>>>>>> +  fi
>>>>>>> +
>>>>>>> +  # checked through possible access limitations and permissions.
>>>>>>> +  # At this step, non-zero return code from "perf stat" needs to
>>>>>>> +  # reported as fail for the user to investigate
>>>>>>> +  if [ $stat_result -ne 0 ]
>>>>>>> +  then
>>>>>>> +    echo "perf stat failed with non-zero return code"
>>>>>>> +    err=1
>>>>>>> +    continue
>>>>>>> +  fi
>>>>>>> +
>>>>>>>   # We failed to see the event and it is supported. Possibly the
>>>>>>> workload was
>>>>>>>   # too small so retry with something longer.
>>>>>>>   output=$(perf stat -e "$p" perf bench internals synthesize
>>>>>>> 2>&1)
>>>>>>> --
>>>>>>> 2.47.3
>>>>>>>