Re: [PATCH v2 11/15] KVM: selftests: Auto generate default tests for KVM Selftests Runner

[Sean Christopherson <seanjc@google.com>]

On Wed, Jul 09, 2025, Oliver Upton wrote:
> On Fri, Jun 06, 2025 at 04:56:15PM -0700, Vipin Sharma wrote:
> > Add logic in Makefile.kvm to auto generate default test files for KVM
> > Selftests Runner. Preserve the hierarchy of test executables for
> > autogenerated files.
> > 
> > Autogeneration of default test files guards against missing these files
> > for new tests cases or rename of existing ones. These autogenerated
> > files will be checked in so that during git-diff one can easily identify
> > if the existing test files for the same test binary also needs an
> > update. It also add new tests automatically in the default coverage.
> 
> I'm not sure I'm sold on the merits of committing these to the tree. My
> preference would be to prioritize signal to noise and just commit the
> 'interesting' test configurations and generate the defaults from
> $(TEST_PROGS) somewhere they're gitignored.

Yeah, I don't love it either, even though I suggested it.

> There's no amount of foolproofing that'll prevent folks from renaming a
> test name w/o updating the 'interesting' test configurations that depend
> upon it. Seriously -- I'm sure I'll manage to break it at least once :)

I was more worried about clobbering someone's (poorly named) default.test, e.g.
if the user modifies or creates their own default.test and the building selftests
overwrites their file.

The main reason I pushed for committing the default testcases is to avoid having
to figure out when and where to generate the testcases.  I don't like the idea
of generating testcase on *every* build; it's not slow, but ugh it seem beyond
wasteful, and it feels weird to dump files into the output directory that the
user might not want.

The obvious alternative would be to add a dedicated make command.  The more I
think about it, the more I think that's probably the way to go.  We already have
to `make headers_install`, so having to do `make testcases_install` or whatever
doesn't seem too onerous.  The biggest conundrum is probably what to put in a
.gitignore for folks that do in-tree builds.  I guess maybe we could add
default.test to the .gitignore?  That'd "document" that that name is reserved,
i.e. would help prevent people from trying to create their own default.test
files.

And since the runner supports listing multiple testcase directories, it'd be
quite easy to "install" the defaults to a dedicated directory while also pulling
testcases that are commited to the repo.

> On top of that, there's a lot of selftests that take no arguments. Not
> sure what we gain having a duplicated definition for these ones outside
> of $(TEST_PROGS).

The defaults are purely a way to communicate the existence of the tests to the
runner.  My hacky bash scripts for running selftests copies all binaries into a
directly and then runs everything in the "output" directory, but that has obvious
flaws :-)

-- 
kvm-riscv mailing list
kvm-riscv@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kvm-riscv