[PATCH 0/3] perf fix test case 14

[PATCH 0/3] perf fix test case 14

[Thomas Richter]

I have taken the following commits from Jiri Olsa's
branch perf/attr_test to work on a fix.

Commit-id
070b9644981e perf tests attr: Do not store failed events
c9666c26ead0 perf tests attr: Make compare_data global
10eb9496d1c3 perf tests attr: Fix compare logic
4ba31b633fab perf tests attr: Add 1s for exclude_kernel ..
ede0b0a2e007 perf tests attr: Fix no-delay test

I have then added 3 new fixes

Thomas Richter (3):
  perf: fix incorrect sample_type value for perf stat tests
  perf: fix exit code check in test case execution
  perf: fix perf test case 14

I can test these changes only on s390.
With these changes some test-stat-xxx tests run fine.

How to handle test cases currently unsupported on a platform?
For example  test-stat-default issues a setup for 
PERF_TYPE_HARDWARE / PERF_COUNT_HW_STALLED_CYCLES_BACKEND
which is not supported on s390. The perf_event_open()
system call fails and no event-00-07--1 file is created.

The compare of the test result then fails because attr.py
has nothing to check against.
We could test if an event file exists and if not report an
unsupported test. Is this a good idea?
Any other proposals?

Right now I have commented out these subtests in my environment.

-- 
2.9.3

[PATCH 1/3] perf: fix incorrect sample_type value for perf stat tests

[Thomas Richter]

Test case 14 runs 'perf stat' command to execute all
tests/attr/test-stat* test cases. They all fail with
a mismatch in sample_type attribute, for example:

running './tests/attr//test-stat-basic'
...
compare
  matchng [event:base-stat]
      to [event-0-0-4]
      ...
      [sample_type] 0 65536
  ->FAIL

The reason for the failure is in create_perf_stat_counter()
in file builtin-stat.c. When command 'perf stat record -o -'
is used the value of perf_event_stat.sample_type is not changed.
All other invocations of 'perf stat' change this
member to PERF_SAMPLE_IDENTIFIER (2^16).

None of the test-stat-* tests are invoked with
'perf stat record -o -' and the sample_type comparion fails.

Fix this by changing the expected value for attribute sample_type
in the base_stat file.

Signed-off-by: Thomas Richter <tmricht@linux.vnet.ibm.com>
---
 tools/perf/tests/attr/base-stat | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/tools/perf/tests/attr/base-stat b/tools/perf/tests/attr/base-stat
index f4cf148..4a48d16 100644
--- a/tools/perf/tests/attr/base-stat
+++ b/tools/perf/tests/attr/base-stat
@@ -8,7 +8,7 @@ type=0
 size=112
 config=0
 sample_period=0
-sample_type=0
+sample_type=65536
 read_format=3
 disabled=1
 inherit=1
-- 
2.9.3

[PATCH 2/3] perf: fix exit code check in test case execution

[Thomas Richter]

Jiri Olsa's
Commit c9666c26ead0 ("perf tests attr: Make compare_data global")
introduced a wrong check on the command's return code exit status test:

  -        if ret != int(self.ret):
  +        if compare_data(str(ret), str(self.ret)):
             raise Unsup(self)

This check succeeds when the expected return code matches
the actual return code and raises the Unsup exceptions.
This is not correct.

Revert to the orignal check to raise the exception on
mismatch.

Signed-off-by: Thomas Richter <tmricht@linux.vnet.ibm.com>
---
 tools/perf/tests/attr.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/tools/perf/tests/attr.py b/tools/perf/tests/attr.py
index 5787879..d0242d8 100644
--- a/tools/perf/tests/attr.py
+++ b/tools/perf/tests/attr.py
@@ -174,7 +174,7 @@ class Test(object):
 
         log.info("  '%s' ret %d, expected %s" % (cmd, ret, str(self.ret)))
 
-        if compare_data(str(ret), str(self.ret)):
+        if ret != int(self.ret):
             raise Unsup(self)
 
     def compare(self, expect, result):
-- 
2.9.3

[PATCH 3/3] perf: fix perf test case 14

[Thomas Richter]

Command perf test -v 14 (Setup struct perf_event_attr test)
always reports success even if the test case fails.
It works correctly if you also specify -F (for don't fork).

   root@s35lp76 perf]# ./perf test -v 14
   14: Setup struct perf_event_attr               :
   --- start ---
   running './tests/attr/test-record-no-delay'
   [ perf record: Woken up 1 times to write data ]
   [ perf record: Captured and wrote 0.002 MB /tmp/tmp4E1h7R/perf.data
     (1 samples) ]
   expected task=0, got 1
   expected precise_ip=0, got 3
   expected wakeup_events=1, got 0
   FAILED './tests/attr/test-record-no-delay' - match failure
   test child finished with 0
   ---- end ----
   Setup struct perf_event_attr: Ok

The reason for the wrong error reporting is the return value of the
system() library call. It is called in run_dir() file tests/attr.c
and returns the exit status, in above case 0xff00.
This value is given as parameter to the exit() function which
can only handle values 0-0xff.
The child process terminates with exit value of 0 and the parent
does not detect any error.

This patch corrects the error reporting and prints the
correct test result.

Signed-off-by: Thomas Richter <tmricht@linux.vnet.ibm.com>
---
 tools/perf/tests/attr.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/tools/perf/tests/attr.c b/tools/perf/tests/attr.c
index 88dc51f..131b510 100644
--- a/tools/perf/tests/attr.c
+++ b/tools/perf/tests/attr.c
@@ -150,7 +150,7 @@ static int run_dir(const char *d, const char *perf)
 	snprintf(cmd, 3*PATH_MAX, PYTHON " %s/attr.py -d %s/attr/ -p %s %.*s",
 		 d, d, perf, vcnt, v);
 
-	return system(cmd);
+	return system(cmd) ? TEST_FAIL : TEST_OK;
 }
 
 int test__attr(int subtest __maybe_unused)
-- 
2.9.3

Re: [PATCH 0/3] perf fix test case 14

[Jiri Olsa]

On Tue, Jun 06, 2017 at 04:31:53PM +0200, Thomas Richter wrote:
> I have taken the following commits from Jiri Olsa's
> branch perf/attr_test to work on a fix.
> 
> Commit-id
> 070b9644981e perf tests attr: Do not store failed events
> c9666c26ead0 perf tests attr: Make compare_data global
> 10eb9496d1c3 perf tests attr: Fix compare logic
> 4ba31b633fab perf tests attr: Add 1s for exclude_kernel ..
> ede0b0a2e007 perf tests attr: Fix no-delay test
> 
> I have then added 3 new fixes
> 
> Thomas Richter (3):
>   perf: fix incorrect sample_type value for perf stat tests
>   perf: fix exit code check in test case execution
>   perf: fix perf test case 14

those looks fine.. any chance you could take over
those other 5 from me and post all together?

> 
> I can test these changes only on s390.
> With these changes some test-stat-xxx tests run fine.
> 
> How to handle test cases currently unsupported on a platform?
> For example  test-stat-default issues a setup for 
> PERF_TYPE_HARDWARE / PERF_COUNT_HW_STALLED_CYCLES_BACKEND
> which is not supported on s390. The perf_event_open()
> system call fails and no event-00-07--1 file is created.
> 
> The compare of the test result then fails because attr.py
> has nothing to check against.
> We could test if an event file exists and if not report an
> unsupported test. Is this a good idea?
> Any other proposals?

or maybe add list of supported or not supported archs
to test's [config] section?

thanks,
jirka

Re: [PATCH 0/3] perf fix test case 14

[Thomas-Mich Richter]

On 06/07/2017 01:17 PM, Jiri Olsa wrote:
> On Tue, Jun 06, 2017 at 04:31:53PM +0200, Thomas Richter wrote:
>> I have taken the following commits from Jiri Olsa's
>> branch perf/attr_test to work on a fix.
>>
>> Commit-id
>> 070b9644981e perf tests attr: Do not store failed events
>> c9666c26ead0 perf tests attr: Make compare_data global
>> 10eb9496d1c3 perf tests attr: Fix compare logic
>> 4ba31b633fab perf tests attr: Add 1s for exclude_kernel ..
>> ede0b0a2e007 perf tests attr: Fix no-delay test
>>
>> I have then added 3 new fixes
>>
>> Thomas Richter (3):
>>   perf: fix incorrect sample_type value for perf stat tests
>>   perf: fix exit code check in test case execution
>>   perf: fix perf test case 14
> 
> those looks fine.. any chance you could take over
> those other 5 from me and post all together?

I would prefer that you post your 5 patches yourself.
I have tested them as well, so you can add me
as Tested-by.

Then I post my 3 patches on top and Arnaldo can
collect them and bundle them together.

> 
>>
>> I can test these changes only on s390.
>> With these changes some test-stat-xxx tests run fine.
>>
>> How to handle test cases currently unsupported on a platform?
>> For example  test-stat-default issues a setup for 
>> PERF_TYPE_HARDWARE / PERF_COUNT_HW_STALLED_CYCLES_BACKEND
>> which is not supported on s390. The perf_event_open()
>> system call fails and no event-00-07--1 file is created.
>>
>> The compare of the test result then fails because attr.py
>> has nothing to check against.
>> We could test if an event file exists and if not report an
>> unsupported test. Is this a good idea?
>> Any other proposals?
> 
> or maybe add list of supported or not supported archs
> to test's [config] section?

Good idea, I will think of something and come back to you.

-- 
Thomas Richter, Dept 3303, IBM LTC Boeblingen Germany
--
Vorsitzende des Aufsichtsrats: Martina Koederitz 
Geschäftsführung: Dirk Wittkopp
Sitz der Gesellschaft: Böblingen / Registergericht: Amtsgericht Stuttgart, HRB 243294