Re: [PATCH v4 1/9] KVM: selftest: Create KVM selftest runner

[Vipin Sharma <vipinsh@google.com>]

On Tue, Jun 16, 2026 at 09:15:24AM -0700, Ackerley Tng wrote:
> Sean Christopherson <seanjc@google.com> writes:
> 
> > On Fri, Jun 12, 2026, Ackerley Tng wrote:
> >> Sean Christopherson <seanjc@google.com> writes:
> >>
> >> > On Thu, Jun 11, 2026, Ackerley Tng wrote:
> >> >> Sean Christopherson <seanjc@google.com> writes:
> >> >>
> >> >> > On Wed, Jun 10, 2026, Ackerley Tng wrote:
> >> >> >> Vipin Sharma <vipinsh@google.com> writes:
> >> >> >> My (future) use case is that with hugepages, I want to run something
> >> >> >> like
> >> >> >>
> >> >> >>   ./guest_memfd_test --order=0
> >> >> >>   ./guest_memfd_test --order=9
> >> >> >>   ./guest_memfd_test --order=18
> >> >> >>
> >> >> >> And 0, 9 and 18 are the supported HugeTLB orders on the machine being
> >> >> >> tested. I'd like to iterate over supported HugeTLB orders at runner
> >> >> >> runtime instead of at build time.
> >> >> >
> >> >> > No.  The right way to handle this is to define testcases for the "interesting"
> >> >> > sizes, and then rely on the test itself to SKIP if the size is unsupported.  This
> >> >> > is no different than a test that requires EPT, or nested VMX, or nested SVM, etc.
> >> >>
> >> >> That should work too. So at build time I'd make it define all the
> >> >> possible HugeTLB sizes on every arch, and then skip as necessary.
> >> >
> >> > Not necessarily at "build time", the testcases can also come from your local
> >> > environment.
> >> >
> >> >> Why though, why not find the supported sizes at runtime?
> >> >
> >> > You can find the supported sizes at runtime, just not in the test runner.  I want
> >> > the runner itself to be largely oblivious to what's its running.  Disallowing
> >> > more or less _any_ test specific configuration/setup in the runner is the only
> >> > way I see of keeping the runner strictly focused on running tests/testcases.
> >>
> >> The runner should 100% focus on running tests, I think it's hard for the
> >> runner to avoid the process of test discovery though.
> >>
> >> The current test discovery process is to go down the tree of directories
> >> and find files, 1 file == 1 testcase. Each file should (statically)
> >> contain a single command.
> >>
> >> Since you're not opposed to runtime discovery of test cases, how about
> >> something like:
> >>
> >>     if the test case file has executable permissions
> >>         execute it and get back a list of test cases to be run.
> >
> > I don't hate it, but I still dislike the idea.  I am all in favor of runtime
> > *discovery* of testcases, but I am staunchly opposed to runtime *definition* of
> > testcases.

I don't think we should support this kind of usecase. Our goal is to
keep test runner as simple as possible. This will add one extra parsing
logic in the runner and also add more wrapper scripts in kvm selftests
for tests to create their own dynamic variants.

If dynamic variation is needed, then there is one option in this runner.
As testcases are executed by runner in a shell environment. This
basically means the test will run as a shell script and provide variable
expansion or call other terminal commands if present. One can define
interesting configuration which can be dynamic based on the host in a
testcase file. For example if one writes a test command as:

  dirty_log_perf_test -x 1 -v $(nproc) -i $(( (RANDOM % 5) + 2 ))

This will run above dirty_log_perf_test with the number of vCPUs equal
to the host CPU count and some randomly selected iteration count between
2 to 6.

This way one can provide interesting configurations which can test max
of a system in a separate test directory which can be invoked when
needed.

For predefined input values of a test, either one can write multiple
*.test files or use kselftest harness and add kselftest API based test
cases. Both option will still require that test should SKIP if something
needed is not present on the host as mentioned below.

> >
> > If I define guest_memfd testcases for 4KiB, 2MiB, and 1GiB pages, but the
> > underlying system only support 4KiB and 2MiB, then I want to see a SKIP for the
> > 1GiB testcase.  If the definition is dynamic, then the 1GiB testcase simply won't
> > exist.
> >
> 
> Seeing the SKIP is a huge advantage over dynamic definition. Thanks for
> explaining. Makes a lot of sense now to have everything defined statically.
>