Re: [PATCH 00/11] perftool-testsuite 2nd batch

[Namhyung Kim <namhyung@kernel.org>]

Hello,

On Mon, Jun 24, 2024 at 03:07:58PM +0200, Michael Petlan wrote:
> Hello,
> 
> posting a second batch of tests from perftool-testsuite [1], as a part
> of the testsuite upstreaming effort.
> 
> This batch brings several perf-probe tests ideated by Masami Hiramatsu 
> and a perf-report test.
> 
> We have also refactored settings files and fixed shellcheck warnings.
> 
> Finally, I have noticed that the scripts are run twice, once run by the
> driver (e.g. perftool-testsuite_*.sh) and once by perf-test's testsuite
> construction feature (that adds all executable shell scripts in all
> sub directories automatically). This is undesirable, so I have fixed
> it for now by skipping "base_*" dirs in the search. I see that this
> approach is not ideal, because the aim of the search seems to be to get
> rid of various test drivers, unify the way the tests are detected and
> run, while allowing them to be in subdirectories, like the Carsten
> Haitzler's coresight tests. However, I still think this is better
> solution that just having a pile of scripts without hierarchy that
> are grabbed and ordered into one-level queue of tests. Apart from
> allowing the tests to share some setup procedure, etc., is helps to
> keep the structure more clear.
> 
> A better design for future might be to have shell tests ordered in
> directories, such as
> 
>   # cd tests/shell/
>   # find . -name \*.sh
>   coresight/asm_pure_loop.sh
>   coresight/memcpy_thread_16k_10.sh
>   coresight/thread_loop_check_tid_10.sh
>   coresight/thread_loop_check_tid_2.sh
>   coresight/unroll_loop_thread_10.sh
>   base_probe/test_adding_kernel.sh
>   base_probe/test_adding_blacklisted.sh
>   base_probe/test_basic.sh
>   base_probe/test_invalid_options.sh
>   base_probe/test_line_semantics.sh
>   base_report/setup.sh
>   base_report/test_basic.sh
>   [...]  
> 
> The directories might also contain some "header" file that would
> bear the name of the parent testcase for all the tests inside it
> (e.g. "perf probe shell tests").
> 
> Then, no separate drivers would be needed, perf-test testsuite-
> creation mechanism would search for the shell scripts as it does
> now, but it would create multilevel structure with subtests,
> such as:
> 
>   97: perf probe shell tests
>   97:1: perf probe basic functionality
>   97:2: perf probe tests with arguments
>   97:3: perf probe invalid options handling
>   98: CoreSight tests
>   98:1: CoreSight / ASM Pure loop
>   98:2: CoreSight / Thread Loop ...
>   [...]
> 
> ... similar to how C-tests can be organized. In case there is
> a setup.sh file in such a subdirectory, it would be run too.
> With properly designed names, also tag-like (substring matching)
> approach could be used to select desired tests (i.e. to override
> the tree structure and pick tests related to some topic, no matter
> if they are in base_stat or base_record subdirectory).
> 
> This plan also counts with processing the current test scripts and
> possibly putting them to best matching place in the structure.
> 
> There is already a bunch of shell tests and I thinkg it might be
> good to organize them into a structure.
> 
> Thoughts and ideas are welcome.
> 
> Thanks and regards,
> 
> Michael

Thanks for the work, but it doesn't apply to the current
perf-tools-next.  Can you please rebase?

Thanks,
Namhyung

> 
> 
> [1] https://github.com/rfmvh/perftool-testsuite
> 
> 
> Michael Petlan (2):
>   perf tests shell: Skip base_* dirs in test script search
>   perf testsuite: Install perf-report tests
> 
> Veronika Molnarova (9):
>   perf testsuite: Merge settings files for shell tests
>   perf testsuite: Fix shellcheck warnings
>   perf testsuite probe: Add test for blacklisted kprobes handling
>   perf testsuite probe: Add test for basic perf-probe options
>   perf testsuite probe: Add test for invalid options
>   perf testsuite probe: Add test for line semantics
>   perf testsuite: Add common output checking helper
>   perf testsuite report: Add test for perf-report basic functionality
>   perf testsuite report: Add test case for perf report
> 
>  tools/perf/Makefile.perf                      |   2 +
>  tools/perf/tests/shell/base_probe/settings.sh |  48 -----
>  .../base_probe/test_adding_blacklisted.sh     |  65 ++++++
>  .../shell/base_probe/test_adding_kernel.sh    |   3 -
>  .../perf/tests/shell/base_probe/test_basic.sh |  78 ++++++++
>  .../shell/base_probe/test_invalid_options.sh  |  77 +++++++
>  .../shell/base_probe/test_line_semantics.sh   |  53 +++++
>  tools/perf/tests/shell/base_report/setup.sh   |  30 +++
>  .../shell/base_report/stderr-whitelist.txt    |   5 +
>  .../tests/shell/base_report/test_basic.sh     | 188 ++++++++++++++++++
>  .../shell/common/check_errors_whitelisted.pl  |  50 +++++
>  tools/perf/tests/shell/common/init.sh         |  31 ++-
>  tools/perf/tests/shell/common/settings.sh     |  28 ++-
>  .../tests/shell/perftool-testsuite_report.sh  |  23 +++
>  tools/perf/tests/tests-scripts.c              |   2 +
>  15 files changed, 627 insertions(+), 56 deletions(-)
>  delete mode 100644 tools/perf/tests/shell/base_probe/settings.sh
>  create mode 100755 tools/perf/tests/shell/base_probe/test_adding_blacklisted.sh
>  create mode 100755 tools/perf/tests/shell/base_probe/test_basic.sh
>  create mode 100755 tools/perf/tests/shell/base_probe/test_invalid_options.sh
>  create mode 100755 tools/perf/tests/shell/base_probe/test_line_semantics.sh
>  create mode 100755 tools/perf/tests/shell/base_report/setup.sh
>  create mode 100644 tools/perf/tests/shell/base_report/stderr-whitelist.txt
>  create mode 100755 tools/perf/tests/shell/base_report/test_basic.sh
>  create mode 100755 tools/perf/tests/shell/common/check_errors_whitelisted.pl
>  create mode 100755 tools/perf/tests/shell/perftool-testsuite_report.sh
> 
> -- 
> 2.43.0
>