Re: [PATCH v4 1/9] KVM: selftest: Create KVM selftest runner

From: Sean Christopherson <seanjc@google.com>
To: Ackerley Tng <ackerleytng@google.com>
Cc: Vipin Sharma <vipinsh@google.com>,
	kvm@vger.kernel.org, kvmarm@lists.linux.dev,
	 kvm-riscv@lists.infradead.org, pbonzini@redhat.com,
	borntraeger@linux.ibm.com,  frankja@linux.ibm.com,
	imbrenda@linux.ibm.com, anup@brainfault.org,
	 atish.patra@linux.dev, zhaotianrui@loongson.cn,
	maobibo@loongson.cn,  chenhuacai@kernel.org, maz@kernel.org,
	oliver.upton@linux.dev,  ajones@ventanamicro.com
Subject: Re: [PATCH v4 1/9] KVM: selftest: Create KVM selftest runner
Date: Fri, 12 Jun 2026 16:56:11 -0700	[thread overview]
Message-ID: <aiycmwMbhWuYghJh@google.com> (raw)
In-Reply-To: <CAEvNRgFLnGkqvvks3joAwPjqLz=ATmqiS0UWk71g+YEsxNsfsg@mail.gmail.com>

On Fri, Jun 12, 2026, Ackerley Tng wrote:
> Sean Christopherson <seanjc@google.com> writes:
> 
> > On Thu, Jun 11, 2026, Ackerley Tng wrote:
> >> Sean Christopherson <seanjc@google.com> writes:
> >>
> >> > On Wed, Jun 10, 2026, Ackerley Tng wrote:
> >> >> Vipin Sharma <vipinsh@google.com> writes:
> >> >> My (future) use case is that with hugepages, I want to run something
> >> >> like
> >> >>
> >> >>   ./guest_memfd_test --order=0
> >> >>   ./guest_memfd_test --order=9
> >> >>   ./guest_memfd_test --order=18
> >> >>
> >> >> And 0, 9 and 18 are the supported HugeTLB orders on the machine being
> >> >> tested. I'd like to iterate over supported HugeTLB orders at runner
> >> >> runtime instead of at build time.
> >> >
> >> > No.  The right way to handle this is to define testcases for the "interesting"
> >> > sizes, and then rely on the test itself to SKIP if the size is unsupported.  This
> >> > is no different than a test that requires EPT, or nested VMX, or nested SVM, etc.
> >>
> >> That should work too. So at build time I'd make it define all the
> >> possible HugeTLB sizes on every arch, and then skip as necessary.
> >
> > Not necessarily at "build time", the testcases can also come from your local
> > environment.
> >
> >> Why though, why not find the supported sizes at runtime?
> >
> > You can find the supported sizes at runtime, just not in the test runner.  I want
> > the runner itself to be largely oblivious to what's its running.  Disallowing
> > more or less _any_ test specific configuration/setup in the runner is the only
> > way I see of keeping the runner strictly focused on running tests/testcases.
> 
> The runner should 100% focus on running tests, I think it's hard for the
> runner to avoid the process of test discovery though.
>
> The current test discovery process is to go down the tree of directories
> and find files, 1 file == 1 testcase. Each file should (statically)
> contain a single command.
> 
> Since you're not opposed to runtime discovery of test cases, how about
> something like:
> 
>     if the test case file has executable permissions
>         execute it and get back a list of test cases to be run.

I don't hate it, but I still dislike the idea.  I am all in favor of runtime
*discovery* of testcases, but I am staunchly opposed to runtime *definition* of
testcases.

If I define guest_memfd testcases for 4KiB, 2MiB, and 1GiB pages, but the
underlying system only support 4KiB and 2MiB, then I want to see a SKIP for the
1GiB testcase.  If the definition is dynamic, then the 1GiB testcase simply won't
exist.

>     else
>         read it as a single test case.
> 
> I'd then use the executable version, check HugeTLB setup on the machine
> under test and return bunch of separate commands to be run (each with a
> different HugeTLB size).
> 
> The runner still doesn't need to deal test-specific config, that's part
> of the executable that tells the runner what to run.