Re: [PATCH v3 9/9] KVM: selftests: Provide README.rst for KVM selftests runner

[Marc Zyngier <maz@kernel.org>]

On Wed, 01 Oct 2025 18:32:25 +0100,
Vipin Sharma <vipinsh@google.com> wrote:
> 
> On 2025-10-01 09:44:22, Marc Zyngier wrote:
> > On Tue, 30 Sep 2025 17:36:35 +0100,
> > Vipin Sharma <vipinsh@google.com> wrote:
> > > 
> > > +KVM selftest runner is highly configurable test executor that allows to run
> > > +tests with different configurations (not just the default), parallely, save
> > 
> > s/parallely/in parallel/
> > 
> 
> Thanks, I will fix it.
> 
> > > +output to disk hierarchically, control what gets printed on console, provide
> > > +execution status.
> > > +
> > > +To generate default tests use::
> > > +
> > > +  # make tests_install
> > > +
> > > +This will create ``testcases_default_gen`` directory which will have testcases
> > 
> > I don't think using the future tense is correct here. I'd rather see
> > something written in the present tense, possibly imperative. For
> > example:
> > 
> > "Create 'blah' directory containing 'foo' files, one per test-case.
> > 
> 
> Thanks, I will fix it.
> 
> > > +in `default.test` files. Each KVM selftest will have a directory in  which
> > > +`default.test` file will be created with executable path relative to KVM
> > > +selftest root directory i.e. `/tools/testing/selftests/kvm`.
> > 
> > Shouldn't this honor the existing build output directives? If it
> > actually does, then you want to call this out.
> > 
> 
> To generate default test files in a specific directory one can use
> "OUTPUT" in the make command

The standard way to do this is documented in the top level Makefile:

<quote>
# This does not need to match to the root of the kernel source tree.
#
# For example, you can do this:
#
#  cd /dir/to/store/output/files; make -f /dir/to/kernel/source/Makefile
#
# If you want to save output files in a different location, there are
# two syntaxes to specify it.
#
# 1) O=
# Use "make O=dir/to/store/output/files/"
#
# 2) Set KBUILD_OUTPUT
# Set the environment variable KBUILD_OUTPUT to point to the output directory.
# export KBUILD_OUTPUT=dir/to/store/output/files/; make
#
# The O= assignment takes precedence over the KBUILD_OUTPUT environment
# variable.
</quote>

Your new infrastructure should support the existing mechanism (and
avoid introducing a new one).

> 
>   make OUTPUT="~/test/directory/path" tests_install
> 
> This allows to generate testcases_default_gen in the given output
> directory. default.test files will still have test binary path relative
> kvm selftest root directory. 
> 
> $OUTPUT
> └── testcases_default_gen
>     ├── access_tracking_perf_test
>     │   └── default.test
>     ├── arch_timer
>     │   └── default.test
>     ├── arm64
>     │   ├── aarch32_id_regs
>     │   │   └── default.test
>     │   ├── arch_timer_edge_cases
>     │   │   └── default.test
>     │   ├── debug-exceptions
>     │   │   └── default.test
>     │   ├── external_aborts
>     │   │   └── default.test
>     │   │   └── default.test
>     │   └── ...
>     ├── coalesced_io_test
>     │   └── default.test
>     ├── demand_paging_test
>     │   └── default.test
>     ├── ...
> 
> So, arm64/aarch32_id_regs/default.test will have 'arm64/aarch32_id_regs'
> 
> User can then supply -p/--path with the path of build output directory
> to runner. 
> 
>   python3 runner -p ~/path/to/selftest/binaries -d $OUTPUT/testcases_default_gen
> 
> If -p not given then current directory is considered for test
> executables.
>
> > > For example, the
> > > +`dirty_log_perf_test` will have::
> > > +
> > > +  # cat testcase_default_gen/dirty_log_perf_test/default.test
> > > +  dirty_log_perf_test
> > > +
> > > +Runner will execute `dirty_log_perf_test`. Testcases files can also provide
> > > +extra arguments to the test::
> > > +
> > > +  # cat tests/dirty_log_perf_test/2slot_5vcpu_10iter.test
> > > +  dirty_log_perf_test -x 2 -v 5 -i 10
> > > +
> > > +In this case runner will execute the `dirty_log_perf_test` with the options.
> > > +
> > 
> > The beginning of the text talks about "non-default' configurations,
> > but you only seem to talk about the default stuff. How does one deals
> > with a non-default config?
> > 
> 
> In the patch 1, I created two sample tests files,
> 2slot_5vcpu_10iter.test and no_dirty_log_protect.test, in the directory
> tools/testing/selftests/kvm/tests/dirty_log_perf_test.
> 
> Contents of those files provide non-default arguments to test, for example,
> 2slot_5vcpu10iter.test has the command:
> 
>   dirty_log_perf_test -x 2 -v 5  -i 10
> 
> One can run these non-default tests as (assuming current directory is
> kvm selftests):
> 
>   python3 runner -d ./tests
> 
> Over the time we will add more of these non-default interesting
> testcases. One can then run:
> 
>   python3 runner -d ./tests ./testcases_default_gen

That's not what I am complaining about. What you call "configuration"
seems to just be "random set of parameters for a random test".

In practice, your runner does not seem configurable at all. You just
treat all possible configurations of a single test as different tests.

My (admittedly very personal) view of what a configuration should be
is "run this single test with these parameters varying in these
ranges, for this long".

Thanks,

	M.

-- 
Without deviation from the norm, progress is not possible.