* Re: [PATCH 0/9] perf test: the testsuite again
2016-03-16 13:50 [PATCH 0/9] perf test: the testsuite again Michael Petlan
@ 2016-03-27 12:24 ` Jiri Olsa
2016-03-27 12:24 ` Jiri Olsa
2016-03-27 12:25 ` Jiri Olsa
2 siblings, 0 replies; 5+ messages in thread
From: Jiri Olsa @ 2016-03-27 12:24 UTC (permalink / raw)
To: Michael Petlan; +Cc: linux-perf-users, acme
On Wed, Mar 16, 2016 at 02:50:06PM +0100, Michael Petlan wrote:
> Hi all,
>
> after some time, I am sending the proposed testsuite again. I have fixed
> couple of things there since last time.
>
> * three verbosity levels are now supported (perf test suite -vv)
> * logs and temp files are stored in /tmp/, so the main tree is not
> spoiled anymore
> * more things can be parametrized from the outside
> * log parsing should be more robust now
>
> The testsuite is roughly tested on x86_64, i386, ppc64, s390x and aarch64.
> The following patches should be applicable upon the Arnaldo's tree, to the
> 'perf/core' branch.
>
> Some issues to be discussed:
>
> 1) Packaging
>
> While the other tests are packed within perf, the character of this suite
> does not fit that model very well. Nowadays, it is run only when the dir
> 'testsuite' is found in the current dir. Otherwise the test is skipped.
> I still haven't found any better solution for this.
>
> 2) In case it gets into the kernel tree, where the 'upstream' will be?
>
> I have been developing the suite in my github repository [1]. I am planning
> to continue with that, if the suite gets or not into the kernel tree. There
> are still many testcases missing. I also backport the suite for RHEL testing
> purposes.
>
> It is easier for me to continue the development there and merge bigger
> chunks into the kernel tree, than move all the development here. Is that
> an acceptable model?
>
> Is there anything else that you would like to point at?
also I think you could erase the ==== quoted lines, like:
======== base_annotate ========
-- [ PASS ] -- perf_annotate :: setup :: building the example code
-- [ PASS ] -- perf_annotate :: setup :: record data
you have the 'perf_annotate' info in each line, which I assume
is connected to the base_annotate header
jirka
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [PATCH 0/9] perf test: the testsuite again
2016-03-16 13:50 [PATCH 0/9] perf test: the testsuite again Michael Petlan
2016-03-27 12:24 ` Jiri Olsa
@ 2016-03-27 12:24 ` Jiri Olsa
2016-03-29 17:28 ` Michael Petlan
2016-03-27 12:25 ` Jiri Olsa
2 siblings, 1 reply; 5+ messages in thread
From: Jiri Olsa @ 2016-03-27 12:24 UTC (permalink / raw)
To: Michael Petlan; +Cc: linux-perf-users, acme
On Wed, Mar 16, 2016 at 02:50:06PM +0100, Michael Petlan wrote:
> Hi all,
>
> after some time, I am sending the proposed testsuite again. I have fixed
> couple of things there since last time.
>
> * three verbosity levels are now supported (perf test suite -vv)
> * logs and temp files are stored in /tmp/, so the main tree is not
> spoiled anymore
> * more things can be parametrized from the outside
> * log parsing should be more robust now
>
> The testsuite is roughly tested on x86_64, i386, ppc64, s390x and aarch64.
> The following patches should be applicable upon the Arnaldo's tree, to the
> 'perf/core' branch.
>
> Some issues to be discussed:
>
> 1) Packaging
>
> While the other tests are packed within perf, the character of this suite
> does not fit that model very well. Nowadays, it is run only when the dir
> 'testsuite' is found in the current dir. Otherwise the test is skipped.
> I still haven't found any better solution for this.
>
> 2) In case it gets into the kernel tree, where the 'upstream' will be?
>
> I have been developing the suite in my github repository [1]. I am planning
> to continue with that, if the suite gets or not into the kernel tree. There
> are still many testcases missing. I also backport the suite for RHEL testing
> purposes.
>
> It is easier for me to continue the development there and merge bigger
> chunks into the kernel tree, than move all the development here. Is that
> an acceptable model?
>
> Is there anything else that you would like to point at?
ctrl-c does not stop the suite.. I checked and you run several
command on background.. is there a reason for that?
[jolsa@krava perf]$ sudo ./perf test suite -v
44: Testsuite :
--- start ---
test child forked, pid 25786
======== base_stat ========
-- [ PASS ] -- perf_stat :: test_basic :: basic execution
...
-- [ PASS ] -- perf_annotate :: test_basic :: redirected input
-- [ PASS ] -- perf_annotate :: test_basic :: execution without data
## [ PASS ] ## perf_annotate :: test_basic SUMMARY
## [ SKIP ] ## perf_annotate :: cleanup :: testcase skipped
======== base_report ========
^C
[jolsa@krava perf]$ ^C
[jolsa@krava perf]$ ^C
[jolsa@krava perf]$ -- [ PASS ] -- perf_report :: setup :: prepare the perf.data file
-- [ SKIP ] -- perf_report :: test_basic :: help message :: testcase skipped
-- [ FAIL ] -- perf_report :: test_basic :: basic execution (output regexp parsing)
-- [ FAIL ] -- perf_report :: test_basic :: number of samples (output regexp parsing)
-- [ PASS ] -- perf_report :: test_basic :: header
jirka
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [PATCH 0/9] perf test: the testsuite again
2016-03-27 12:24 ` Jiri Olsa
@ 2016-03-29 17:28 ` Michael Petlan
0 siblings, 0 replies; 5+ messages in thread
From: Michael Petlan @ 2016-03-29 17:28 UTC (permalink / raw)
To: Jiri Olsa; +Cc: linux-perf-users, acme
On Sun, 27 Mar 2016, Jiri Olsa wrote:
> ctrl-c does not stop the suite.. I checked and you run several
> command on background.. is there a reason for that?
>
>
> [jolsa@krava perf]$ sudo ./perf test suite -v
> 44: Testsuite :
> --- start ---
> test child forked, pid 25786
> ======== base_stat ========
> -- [ PASS ] -- perf_stat :: test_basic :: basic execution
>
> ...
>
> -- [ PASS ] -- perf_annotate :: test_basic :: redirected input
> -- [ PASS ] -- perf_annotate :: test_basic :: execution without data
> ## [ PASS ] ## perf_annotate :: test_basic SUMMARY
> ## [ SKIP ] ## perf_annotate :: cleanup :: testcase skipped
>
> ======== base_report ========
> ^C
> [jolsa@krava perf]$ ^C
> [jolsa@krava perf]$ ^C
> [jolsa@krava perf]$ -- [ PASS ] -- perf_report :: setup :: prepare the perf.data file
> -- [ SKIP ] -- perf_report :: test_basic :: help message :: testcase skipped
> -- [ FAIL ] -- perf_report :: test_basic :: basic execution (output regexp parsing)
> -- [ FAIL ] -- perf_report :: test_basic :: number of samples (output regexp parsing)
> -- [ PASS ] -- perf_report :: test_basic :: header
>
>
I think this is a generic behaviour of perf-test itself. In the case of the
testsuite it just has a bigger "race window" you can hit with the C-c.
Every test forks:
---- end ----
roundtrip evsel->name check: Ok
12: Check parsing of sched tracepoints fields :
--- start ---
test child forked, pid 8304
test child finished with 0
---- end ----
Check parsing of sched tracepoints fields: Ok
13: Generate and check syscalls:sys_enter_openat event fields:
--- start ---
test child forked, pid 8305
mmap size 528384B
test child finished with 0
---- end ----
And I managed to C-c the test 14 hitting the "race window", so it continued
printing something:
14: struct perf_event_attr setup :
--- start ---
test child forked, pid 8306
running './tests/attr/test-record-C0'
unsupp './tests/attr/test-record-C0'
running './tests/attr/test-record-basic'
unsupp './tests/attr/test-record-basic'
running './tests/attr/test-record-branch-any'
unsupp './tests/attr/test-record-branch-any'
running './tests/attr/test-record-branch-filter-any'
unsupp './tests/attr/test-record-branch-filter-any'
running './tests/attr/test-record-branch-filter-any_call'
unsupp './tests/attr/test-record-branch-filter-any_call'
running './tests/attr/test-record-branch-filter-any_ret'
unsupp './tests/attr/test-record-branch-filter-any_ret'
running './tests/attr/test-record-branch-filter-hv'
unsupp './tests/attr/test-record-branch-filter-hv'
running './tests/attr/test-record-branch-filter-ind_call'
unsupp './tests/attr/test-record-branch-filter-ind_call'
running './tests/attr/test-record-branch-filter-k'
unsupp './tests/attr/test-record-branch-filter-k'
running './tests/attr/test-record-branch-filter-u'
unsupp './tests/attr/test-record-branch-filter-u'
running './tests/attr/test-record-count'
unsupp './tests/attr/test-record-count'
running './tests/attr/test-record-data'
unsupp './tests/attr/test-record-data'
running './tests/attr/test-record-freq'
unsupp './tests/attr/test-record-freq'
running './tests/attr/test-record-graph-default'
unsupp './tests/attr/test-record-graph-default'
running './tests/attr/test-record-graph-dwarf'
unsupp './tests/attr/test-record-graph-dwarf'
running './tests/attr/test-record-graph-fp'
^C
[root@muflon perf]# sh: line 1: 8357 Terminated PERF_TEST_ATTR=/tmp/tmpzPprWC ./perf record -o /tmp/tmpzPprWC/perf.data --call-graph fp kill > /dev/null 2>&1
unsupp './tests/attr/test-record-graph-fp'
running './tests/attr/test-record-group'
unsupp './tests/attr/test-record-group'
running './tests/attr/test-record-group-sampling'
unsupp './tests/attr/test-record-group-sampling'
running './tests/attr/test-record-group1'
unsupp './tests/attr/test-record-group1'
When looking at `ps aux | grep perf | grep -v grep`, you can see the following:
# in this case you won't hit the "race window", C-c would work
root 8286 0.0 0.1 76944 5692 pts/0 S+ 18:16 0:00 perf test -v
root 8292 0.0 0.0 81316 3640 pts/0 S+ 18:16 0:00 perf test -v
# in this case you hit the "race window" (run ps aux exactly when a sub-test
# was running), so C-c would kill the perf test itself, but not the PID 8307
# (which will continue printing stuff)
root 8286 0.0 0.1 76944 5692 pts/0 S+ 18:16 0:00 perf test -v
root 8306 0.0 0.0 76944 936 pts/0 S+ 18:16 0:00 perf test -v
root 8307 1.6 0.3 192668 12340 pts/0 D+ 18:16 0:00 /usr/bin/python2 ./tests/attr.py -d ./tests/attr/ -p ./perf -v
When the testsuite runs, it usually runs some /bin/bash instead of the python2
here. Since the testsuite spends most of the time in running the subtests outside
of the C code (known issue), your chance that you hit that moment by C-c is pretty
high. I don't think it can be fixed without re-designing the perf-test itself.
Regards,
Michael
>
> jirka
> --
> To unsubscribe from this list: send the line "unsubscribe linux-perf-users" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [PATCH 0/9] perf test: the testsuite again
2016-03-16 13:50 [PATCH 0/9] perf test: the testsuite again Michael Petlan
2016-03-27 12:24 ` Jiri Olsa
2016-03-27 12:24 ` Jiri Olsa
@ 2016-03-27 12:25 ` Jiri Olsa
2 siblings, 0 replies; 5+ messages in thread
From: Jiri Olsa @ 2016-03-27 12:25 UTC (permalink / raw)
To: Michael Petlan; +Cc: linux-perf-users, acme
On Wed, Mar 16, 2016 at 02:50:06PM +0100, Michael Petlan wrote:
> Hi all,
>
> after some time, I am sending the proposed testsuite again. I have fixed
> couple of things there since last time.
>
> * three verbosity levels are now supported (perf test suite -vv)
> * logs and temp files are stored in /tmp/, so the main tree is not
> spoiled anymore
> * more things can be parametrized from the outside
> * log parsing should be more robust now
>
> The testsuite is roughly tested on x86_64, i386, ppc64, s390x and aarch64.
> The following patches should be applicable upon the Arnaldo's tree, to the
> 'perf/core' branch.
>
> Some issues to be discussed:
>
> 1) Packaging
>
> While the other tests are packed within perf, the character of this suite
> does not fit that model very well. Nowadays, it is run only when the dir
> 'testsuite' is found in the current dir. Otherwise the test is skipped.
> I still haven't found any better solution for this.
>
> 2) In case it gets into the kernel tree, where the 'upstream' will be?
>
> I have been developing the suite in my github repository [1]. I am planning
> to continue with that, if the suite gets or not into the kernel tree. There
> are still many testcases missing. I also backport the suite for RHEL testing
> purposes.
>
> It is easier for me to continue the development there and merge bigger
> chunks into the kernel tree, than move all the development here. Is that
> an acceptable model?
>
> Is there anything else that you would like to point at?
>
> Thanks.
> Michael
so when I run 'sudo perf test suite -v' I assume I'm getting
one test pre line output with status at the begning:
======== base_report ========
-- [ PASS ] -- perf_report :: setup :: prepare the perf.data file
-- [ SKIP ] -- perf_report :: test_basic :: help message :: testcase skipped
-- [ FAIL ] -- perf_report :: test_basic :: basic execution (output regexp parsing)
-- [ FAIL ] -- perf_report :: test_basic :: number of samples (output regexp parsing)
-- [ PASS ] -- perf_report :: test_basic :: header
-- [ FAIL ] -- perf_report :: test_basic :: show CPU utilization (output regexp parsing)
-- [ FAIL ] -- perf_report :: test_basic :: pid (output regexp parsing)
-- [ FAIL ] -- perf_report :: test_basic :: non-existing symbol (output regexp parsing)
-- [ FAIL ] -- perf_report :: test_basic :: symbol filter (output regexp parsing)
## [ FAIL ] ## perf_report :: test_basic SUMMARY :: 6 failures found
## [ SKIP ] ## perf_report :: cleanup :: testcase skipped
that seems great, and it'd be nice if you could frce this 1 line per
test even if some test fails and spits out some error message, like:
-- [ PASS ] -- perf_probe :: test_listing :: userspace functions list
Failed to find the path for kernel: Invalid ELF file
Error: Failed to show vars.
-- [ FAIL ] -- perf_probe :: test_listing :: kernel variables list (command exitcode + output regexp parsing)
-- [ PASS ] -- perf_probe :: test_listing :: userspace variables list
Failed to find the path for kernel: Invalid ELF file
Error: Failed to show lines.
-- [ FAIL ] -- perf_probe :: test_listing :: kernel lines list (command exitcode + output regexp parsing)
Failed to find the path for kernel: Invalid ELF file
Error: Failed to show lines.
-- [ FAIL ] -- perf_probe :: test_listing :: kernel source lines list (command exitcode + output regexp parsing)
-- [ SKIP ] -- perf_probe :: test_listing :: userspace lines list :: testcase skipped
which breaks otherwise nice output ;-)
jirka
^ permalink raw reply [flat|nested] 5+ messages in thread