Re: [PATCH v3 0/9] Run tests in parallel showing number of tests running

[Namhyung Kim <namhyung@kernel.org>]

On Thu, Oct 17, 2024 at 10:47:10PM -0700, Ian Rogers wrote:
> Avoid waitpid so that stdout/stderr aren't destroyed prior to wanting
> to read them for display. When running on a color terminal, display
> the number of running tests (1 if sequential). To avoid previous
> flicker, only delete and refresh the display line when it changes. An
> earlier version of this code is here:
> https://lore.kernel.org/lkml/20240701044236.475098-1-irogers@google.com/
> 
> Add a signal handler for perf tests so that unexpected signals are
> displayed and test clean up is possible.
> 
> In perf test add an "exclusive" flag that causes a test to be run with
> no other test. Set this flag manually for C tests and via a
> "(exclusive)" in the test description for shell tests. Add the flag to
> shell tests that may fail when run with other tests.
> 
> Change the perf test loop to run in two passes. For parallel
> execution, the first pass runs all tests that can be run in parallel
> then the 2nd runs remaining tests sequentially. This causes the
> "exclusive" tests to be run last and with test numbers moderately out
> of alignment.
> 
> Change the default to be to run tests in parallel. Running tests in
> parallel brings the execution time down to less than half.

Can you renumber the tests so that the numbers can be continuous?

Also, it doesn't apply to the current perf-tools-next, please rebase.

Thanks,
Namhyung

> 
> v3: Mark additional shell tests as "(exclusive)" to avoid issues with
>     shared resources suggested by Namhyung. Add dependent signal
>     handler change so that kill/ctrl-C don't leave lots of processes,
>     previously sent here:
>     https://lore.kernel.org/lkml/20241017052137.225514-1-irogers@google.com/
> 
> v2: Fix inaccurate remaining counts when running specific
>     tests. Rename "remaining" to "active" to better reflect the
>     testing behavior. Move the exclusive flag to test cases and not
>     entire suites. Add more "(exclusive)" flags to test as
>     suggested-by James Clark. Remove "(exclusive)" flag from test
>     descriptions to keep the command line output more concise. Add
>     James Clark's tested-by.
> 
> Ian Rogers (9):
>   tools subcmd: Add non-waitpid check_if_command_finished()
>   perf test: Display number of active running tests
>   perf test: Reduce scope of parallel variable
>   perf test: Avoid list test blocking on writing to stdout
>   perf test: Tag parallel failing shell tests with "(exclusive)"
>   perf test: Add a signal handler around running a test
>   perf test: Run parallel tests in two passes
>   perf test: Make parallel testing the default
>   perf test: Add a signal handler to kill forked child processes
> 
>  tools/lib/subcmd/run-command.c                |  33 ++
>  tools/perf/tests/builtin-test.c               | 304 ++++++++++++------
>  .../tests/shell/coresight/asm_pure_loop.sh    |   2 +-
>  .../shell/coresight/memcpy_thread_16k_10.sh   |   2 +-
>  .../coresight/thread_loop_check_tid_10.sh     |   2 +-
>  .../coresight/thread_loop_check_tid_2.sh      |   2 +-
>  .../shell/coresight/unroll_loop_thread_10.sh  |   2 +-
>  tools/perf/tests/shell/list.sh                |   5 +-
>  .../tests/shell/perftool-testsuite_report.sh  |   2 +-
>  tools/perf/tests/shell/probe_vfs_getname.sh   |   2 +-
>  .../shell/record+script_probe_vfs_getname.sh  |   2 +-
>  tools/perf/tests/shell/record.sh              |   2 +-
>  tools/perf/tests/shell/record_lbr.sh          |   2 +-
>  tools/perf/tests/shell/record_offcpu.sh       |   2 +-
>  tools/perf/tests/shell/stat_all_pmu.sh        |   2 +-
>  tools/perf/tests/shell/stat_bpf_counters.sh   |   2 +-
>  tools/perf/tests/shell/test_arm_coresight.sh  |   2 +-
>  .../tests/shell/test_arm_coresight_disasm.sh  |   2 +-
>  tools/perf/tests/shell/test_arm_spe.sh        |   2 +-
>  tools/perf/tests/shell/test_data_symbol.sh    |   2 +-
>  tools/perf/tests/shell/test_intel_pt.sh       |   2 +-
>  .../perf/tests/shell/test_stat_intel_tpebs.sh |   2 +-
>  .../tests/shell/trace+probe_vfs_getname.sh    |   2 +-
>  tools/perf/tests/task-exit.c                  |   9 +-
>  tools/perf/tests/tests-scripts.c              |   7 +-
>  tools/perf/tests/tests.h                      |   9 +
>  tools/perf/util/color.h                       |   1 +
>  27 files changed, 293 insertions(+), 115 deletions(-)
> 
> -- 
> 2.47.0.105.g07ac214952-goog
>