Re: [PATCH 00/10] Introduce structure for shell tests

[Arnaldo Carvalho de Melo <acme@kernel.org>]

On Fri, Dec 20, 2024 at 11:03:24PM +0100, vmolnaro@redhat.com wrote:
> From: Veronika Molnarova <vmolnaro@redhat.com>
> 
> Hello,
> 
> Sending the third patchset from perftool-testsuite upstreaming effort,
> which also contains new possibilities for shell tests, such as a two-level
> structured test hierarchy, a setup file for the test suite and ability to
> store logs.
> 
> The patches do not add any new test cases but instead provide the needed
> environment that was temporarily replaced by the perftool test drivers.
> 
> We wanted to provide the possibility to have a shell test consisting 
> of multiple subtests, as is already done for the C tests. The logical 
> structuring of the test cases was a part of the perftool test suite,
> and we saw this as an opportunity to introduce a structured approach for 
> the perf shell tests.
> 
> A directory in the shell directory will be represented as a test suite 
> if it contains at least one executable shell test. In case of two and 
> more tests, the subtests are are visibly differentiated from the test 
> suite by having a subtest index. All deeper levels of subdirectories 
> are still searched for tests but do not create additional levels of 
> hierarchy.
> 
> Some test suites require setup to be done before they are run, such 
> recording samples or setting up test probes. This can be done by 
> adding a "setup.sh" executable file in the test suite directory, which 
> will be run before all of the tests. If the setup fails, all of 
> the tests are skipped, as it is assumed that the setup is required 
> for their execution. The setup file also gives us the possibility to 
> name the test suite. If there is no setup file, the name is derived 
> from the name of the directory.
> 
> Lastly, we wanted to provide a way to store the test logs after execution 
> for debugging purposes, if necessary. The test logs for perftool tests 
> are stored in a '/tmp/perf_test_*' temporary directory. By default,
> these logs are cleared after the test finishes. However, if the env
> variable PERFTEST_KEEP_LOGS is set to "y", the test logs are retained
> for debugging.
> 
> For now, all of the perftool tests are marked as exclusive, preventing 
> from running parallel. This may change in the future if we ensure that 
> they will not interfere with other tests being run simultaneously.

I tried to apply it now to perf-tools-next and didn't manage to :-\

Can you take a look please? I was going to test it.

- Arnaldo
 
> Thoughts and ideas are welcome.
> 
> Thanks and regards,
> 
> Veronika
> 
> Michael Petlan (1):
>   perf testsuite: Fix perf-report tests installation
> 
> Veronika Molnarova (9):
>   perf test perftool_testsuite: Add missing description
>   perf test perftool_testsuite: Return correct value for skipping
>   perf test perftool_testsuite: Use absolute paths
>   perf tests: Create a structure for shell tests
>   perf test: Provide setup for the shell test suite
>   perftool-testsuite: Add empty setup for base_probe
>   perf test: Introduce storing logs for shell tests
>   perf test: Format log directories for shell tests
>   perf test: Remove perftool drivers
> 
>  tools/perf/Makefile.perf                      |   3 +-
>  tools/perf/tests/builtin-test.c               | 151 +++++++++-
>  tools/perf/tests/shell/base_probe/setup.sh    |  13 +
>  .../base_probe/test_adding_blacklisted.sh     |  17 +-
>  .../shell/base_probe/test_adding_kernel.sh    |  57 ++--
>  .../perf/tests/shell/base_probe/test_basic.sh |  23 +-
>  .../shell/base_probe/test_invalid_options.sh  |  15 +-
>  .../shell/base_probe/test_line_semantics.sh   |  11 +-
>  tools/perf/tests/shell/base_report/setup.sh   |   8 +-
>  .../tests/shell/base_report/test_basic.sh     |  49 ++--
>  tools/perf/tests/shell/common/init.sh         |   6 +-
>  .../tests/shell/perftool-testsuite_probe.sh   |  23 --
>  .../tests/shell/perftool-testsuite_report.sh  |  23 --
>  tools/perf/tests/tests-scripts.c              | 258 +++++++++++++++---
>  tools/perf/tests/tests-scripts.h              |  15 +
>  tools/perf/tests/tests.h                      |   8 +-
>  16 files changed, 489 insertions(+), 191 deletions(-)
>  create mode 100755 tools/perf/tests/shell/base_probe/setup.sh
>  delete mode 100755 tools/perf/tests/shell/perftool-testsuite_probe.sh
>  delete mode 100755 tools/perf/tests/shell/perftool-testsuite_report.sh
> 
> -- 
> 2.43.0