[PATCH v3 0/9] KVM Selftest Runner

Vipin Sharma
2025-09-30T16:36:49Z

Hello,

This is v3 of KVM selftest runner. After making changes on feedback
given in v2, this series has reduced from 15 patches to 9 patches. I
have tried to address all of the comments from v2. There are none left
open and are incorporated to best of my understanding.

To recap (copied from v2), KVM Selftest Runner allows running KVM
selftests with added features not present in default selftest runner
provided by selftests framework.

This Runner has two broad goals:
1. Make it easier for contributors and maintainers to run various
configuration of tests with features like preserving output,
controlling output verbosity, parallelism, different combinations of
command line arguments.
2. Provide common place to write interesting and useful combinations of
tests command line arguments to improve KVM test coverage. Default
selftests runner provide little to no control over this.

Future patches will add features like:
- Print process id of the test in execution.
- CTRL+C currently spits out lots of warning (depending on --job value).
This will be fixed in the next version.
- Add more tests configurations.
- Provide a way to set the environment in which runner will start tests. For
example, setting huge pages, stress testing based on resources
available on host.

This series is also available on github at:

https://github.com/shvipin/linux kvm/sefltests/runner-v3

v3:
- Created "tests_install" rule in Makefile.kvm to auto generate default
testcases, which will be ignored in .gitignore.
- Changed command line option names to pass testcase files, directories,
executable paths, print based on test status, and what to print.
Removed certain other options based on feedback in v2.
- Merged command.py into selftest.py
- Fixed issue where timed out test's stdout and stderr were not printed.
- Reduced python version from 3.7 to 3.6.
- Fixed issue where test status numerical value was printed instead of
text like PASSED, FAILED, SKIPPED, etc.
- Added README.rst.

v2: https://lore.kernel.org/kvm/20250606235619.1841595-1-vipinsh@google.com/
- Automatic default test generation.
- Command line flag to provide executables location
- Dump output to filesystem with timestamp
- Accept absolute path of *.test files/directory location
- Sticky status at bottom for the current state of runner.
- Knobs to control output verbosity
- Colored output for terminals.

v1: https://lore.kernel.org/kvm/20250222005943.3348627-1-vipinsh@google.com/
- Parallel test execution.
- Dumping separate output for each test.
- Timeout for test execution
- Specify single test or a test directory.

RFC: https://lore.kernel.org/kvm/20240821223012.3757828-1-vipinsh@google.com/

Vipin Sharma (9):
KVM: selftest: Create KVM selftest runner
KVM: selftests: Provide executables path option to the KVM selftest
runner
KVM: selftests: Add timeout option in selftests runner
KVM: selftests: Add option to save selftest runner output to a
directory
KVM: selftests: Run tests concurrently in KVM selftests runner
KVM: selftests: Add various print flags to KVM selftest runner
KVM: selftests: Print sticky KVM selftests runner status at bottom
KVM: selftests: Add rule to generate default tests for KVM selftests
runner
KVM: selftests: Provide README.rst for KVM selftests runner

tools/testing/selftests/kvm/.gitignore | 6 +-
tools/testing/selftests/kvm/Makefile.kvm | 20 ++
tools/testing/selftests/kvm/runner/README.rst | 54 +++++
.../testing/selftests/kvm/runner/__main__.py | 184 ++++++++++++++++++
.../testing/selftests/kvm/runner/selftest.py | 105 ++++++++++
.../selftests/kvm/runner/test_runner.py | 79 ++++++++
.../2slot_5vcpu_10iter.test | 1 +
.../no_dirty_log_protect.test | 1 +
8 files changed, 449 insertions(+), 1 deletion(-)
create mode 100644 tools/testing/selftests/kvm/runner/README.rst
create mode 100644 tools/testing/selftests/kvm/runner/__main__.py
create mode 100644 tools/testing/selftests/kvm/runner/selftest.py
create mode 100644 tools/testing/selftests/kvm/runner/test_runner.py
create mode 100644 tools/testing/selftests/kvm/tests/dirty_log_perf_test/2slot_5vcpu_10iter.test
create mode 100644 tools/testing/selftests/kvm/tests/dirty_log_perf_test/no_dirty_log_protect.test

--
2.51.0.618.g983fd99d29-goog