From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 2FC3B2C027F; Thu, 2 Oct 2025 14:41:30 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1759416090; cv=none; b=NMD6cysleQDQO/v/H4dMy1FuzlYcELjCAB/J3o/AkatoRMggvOmZRXSpDi9CSUWhqj7epeyp6gA7X+s7+7vlBI4nAypaTRCSseloVh6DKNHp1uze4ozWTRwoxMywdt89uDcgbiPrS6/3UhgwMvmIFzI/WPmCdT7g5TAGaNMyYbA= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1759416090; c=relaxed/simple; bh=HcWa8+/7HpfIO4HCatnbcdT8Qggyf0k+gQ/HNIeL38o=; h=Date:Message-ID:From:To:Cc:Subject:In-Reply-To:References: MIME-Version:Content-Type; b=S//G4SQjmTMMzbHGGz3Rc5P6SQRLH3LCxF3cAB6+r6tLwPerYBVEgGGu9lZfqzfmOfxkCQdUemShOETOM+9ZBHbWWc6OKO3Rs55encWp/b2bPQggb0L2EbVqrf6+3Vf1d8tsCbKy3GXtZY3q0jpgbDGYFLbozLytoKdiLFc3gCQ= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=NVugEe9f; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="NVugEe9f" Received: by smtp.kernel.org (Postfix) with ESMTPSA id DF784C4CEF4; Thu, 2 Oct 2025 14:41:29 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1759416090; bh=HcWa8+/7HpfIO4HCatnbcdT8Qggyf0k+gQ/HNIeL38o=; h=Date:From:To:Cc:Subject:In-Reply-To:References:From; b=NVugEe9ft5Jkhp5UvmLvmY87/yzeX0fnBUxalpGV6FkHWWeUMCfELz+u2e5HgaDeS hJAsl25RRJxePXYtCzVuZvAjLjaNEidW1JfBlSyLMQXVfcKR6euXzePe94qsNjDr97 mjH++vBvOR2p1LAP+SWbIk1KcyPe7wmgbJbpgqG4BpiZ2KcwlN7whWk0C7eunBTN62 vzZZR5YScFwOvy4dqe8ABIErGg3V1fnpUX7NUY8rH6VcgJGkdKxuad4I2JakrYIaFd 4O+eT+0mOolKRflSJ6Z2V3PKycTlJnulYUiNqg5WP5EYGa2QEmCeRQmyF/68pThyyk MMSkPRfpzC1vw== Received: from sofa.misterjones.org ([185.219.108.64] helo=goblin-girl.misterjones.org) by disco-boy.misterjones.org with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.98.2) (envelope-from ) id 1v4KUp-0000000B6Jf-0vxs; Thu, 02 Oct 2025 14:41:27 +0000 Date: Thu, 02 Oct 2025 15:41:26 +0100 Message-ID: <86a529z7qh.wl-maz@kernel.org> From: Marc Zyngier To: Vipin Sharma Cc: kvm@vger.kernel.org, kvmarm@lists.linux.dev, kvm-riscv@lists.infradead.org, seanjc@google.com, pbonzini@redhat.com, borntraeger@linux.ibm.com, frankja@linux.ibm.com, imbrenda@linux.ibm.com, anup@brainfault.org, atish.patra@linux.dev, zhaotianrui@loongson.cn, maobibo@loongson.cn, chenhuacai@kernel.org, oliver.upton@linux.dev, ajones@ventanamicro.com Subject: Re: [PATCH v3 9/9] KVM: selftests: Provide README.rst for KVM selftests runner In-Reply-To: <20251001173225.GA420255.vipinsh@google.com> References: <20250930163635.4035866-1-vipinsh@google.com> <20250930163635.4035866-10-vipinsh@google.com> <86qzvnypsp.wl-maz@kernel.org> <20251001173225.GA420255.vipinsh@google.com> User-Agent: Wanderlust/2.15.9 (Almost Unreal) SEMI-EPG/1.14.7 (Harue) FLIM-LB/1.14.9 (=?UTF-8?B?R29qxY0=?=) APEL-LB/10.8 EasyPG/1.0.0 Emacs/30.1 (aarch64-unknown-linux-gnu) MULE/6.0 (HANACHIRUSATO) Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 (generated by SEMI-EPG 1.14.7 - "Harue") Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-SA-Exim-Connect-IP: 185.219.108.64 X-SA-Exim-Rcpt-To: vipinsh@google.com, kvm@vger.kernel.org, kvmarm@lists.linux.dev, kvm-riscv@lists.infradead.org, seanjc@google.com, pbonzini@redhat.com, borntraeger@linux.ibm.com, frankja@linux.ibm.com, imbrenda@linux.ibm.com, anup@brainfault.org, atish.patra@linux.dev, zhaotianrui@loongson.cn, maobibo@loongson.cn, chenhuacai@kernel.org, oliver.upton@linux.dev, ajones@ventanamicro.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false On Wed, 01 Oct 2025 18:32:25 +0100, Vipin Sharma wrote: >=20 > On 2025-10-01 09:44:22, Marc Zyngier wrote: > > On Tue, 30 Sep 2025 17:36:35 +0100, > > Vipin Sharma wrote: > > >=20 > > > +KVM selftest runner is highly configurable test executor that allows= to run > > > +tests with different configurations (not just the default), parallel= y, save > >=20 > > s/parallely/in parallel/ > >=20 >=20 > Thanks, I will fix it. >=20 > > > +output to disk hierarchically, control what gets printed on console,= provide > > > +execution status. > > > + > > > +To generate default tests use:: > > > + > > > + # make tests_install > > > + > > > +This will create ``testcases_default_gen`` directory which will have= testcases > >=20 > > I don't think using the future tense is correct here. I'd rather see > > something written in the present tense, possibly imperative. For > > example: > >=20 > > "Create 'blah' directory containing 'foo' files, one per test-case. > >=20 >=20 > Thanks, I will fix it. >=20 > > > +in `default.test` files. Each KVM selftest will have a directory in = which > > > +`default.test` file will be created with executable path relative to= KVM > > > +selftest root directory i.e. `/tools/testing/selftests/kvm`. > >=20 > > Shouldn't this honor the existing build output directives? If it > > actually does, then you want to call this out. > >=20 >=20 > To generate default test files in a specific directory one can use > "OUTPUT" in the make command The standard way to do this is documented in the top level Makefile: # This does not need to match to the root of the kernel source tree. # # For example, you can do this: # # cd /dir/to/store/output/files; make -f /dir/to/kernel/source/Makefile # # If you want to save output files in a different location, there are # two syntaxes to specify it. # # 1) O=3D # Use "make O=3Ddir/to/store/output/files/" # # 2) Set KBUILD_OUTPUT # Set the environment variable KBUILD_OUTPUT to point to the output directo= ry. # export KBUILD_OUTPUT=3Ddir/to/store/output/files/; make # # The O=3D assignment takes precedence over the KBUILD_OUTPUT environment # variable. Your new infrastructure should support the existing mechanism (and avoid introducing a new one). >=20 > make OUTPUT=3D"~/test/directory/path" tests_install >=20 > This allows to generate testcases_default_gen in the given output > directory. default.test files will still have test binary path relative > kvm selftest root directory.=20 >=20 > $OUTPUT > =E2=94=94=E2=94=80=E2=94=80 testcases_default_gen > =E2=94=9C=E2=94=80=E2=94=80 access_tracking_perf_test > =E2=94=82 =E2=94=94=E2=94=80=E2=94=80 default.test > =E2=94=9C=E2=94=80=E2=94=80 arch_timer > =E2=94=82 =E2=94=94=E2=94=80=E2=94=80 default.test > =E2=94=9C=E2=94=80=E2=94=80 arm64 > =E2=94=82 =E2=94=9C=E2=94=80=E2=94=80 aarch32_id_regs > =E2=94=82 =E2=94=82 =E2=94=94=E2=94=80=E2=94=80 default.test > =E2=94=82 =E2=94=9C=E2=94=80=E2=94=80 arch_timer_edge_cases > =E2=94=82 =E2=94=82 =E2=94=94=E2=94=80=E2=94=80 default.test > =E2=94=82 =E2=94=9C=E2=94=80=E2=94=80 debug-exceptions > =E2=94=82 =E2=94=82 =E2=94=94=E2=94=80=E2=94=80 default.test > =E2=94=82 =E2=94=9C=E2=94=80=E2=94=80 external_aborts > =E2=94=82 =E2=94=82 =E2=94=94=E2=94=80=E2=94=80 default.test > =E2=94=82 =E2=94=82 =E2=94=94=E2=94=80=E2=94=80 default.test > =E2=94=82 =E2=94=94=E2=94=80=E2=94=80 ... > =E2=94=9C=E2=94=80=E2=94=80 coalesced_io_test > =E2=94=82 =E2=94=94=E2=94=80=E2=94=80 default.test > =E2=94=9C=E2=94=80=E2=94=80 demand_paging_test > =E2=94=82 =E2=94=94=E2=94=80=E2=94=80 default.test > =E2=94=9C=E2=94=80=E2=94=80 ... >=20 > So, arm64/aarch32_id_regs/default.test will have 'arm64/aarch32_id_regs' >=20 > User can then supply -p/--path with the path of build output directory > to runner.=20 >=20 > python3 runner -p ~/path/to/selftest/binaries -d $OUTPUT/testcases_defa= ult_gen >=20 > If -p not given then current directory is considered for test > executables. > > > > For example, the > > > +`dirty_log_perf_test` will have:: > > > + > > > + # cat testcase_default_gen/dirty_log_perf_test/default.test > > > + dirty_log_perf_test > > > + > > > +Runner will execute `dirty_log_perf_test`. Testcases files can also = provide > > > +extra arguments to the test:: > > > + > > > + # cat tests/dirty_log_perf_test/2slot_5vcpu_10iter.test > > > + dirty_log_perf_test -x 2 -v 5 -i 10 > > > + > > > +In this case runner will execute the `dirty_log_perf_test` with the = options. > > > + > >=20 > > The beginning of the text talks about "non-default' configurations, > > but you only seem to talk about the default stuff. How does one deals > > with a non-default config? > >=20 >=20 > In the patch 1, I created two sample tests files, > 2slot_5vcpu_10iter.test and no_dirty_log_protect.test, in the directory > tools/testing/selftests/kvm/tests/dirty_log_perf_test. >=20 > Contents of those files provide non-default arguments to test, for exampl= e, > 2slot_5vcpu10iter.test has the command: >=20 > dirty_log_perf_test -x 2 -v 5 -i 10 >=20 > One can run these non-default tests as (assuming current directory is > kvm selftests): >=20 > python3 runner -d ./tests >=20 > Over the time we will add more of these non-default interesting > testcases. One can then run: >=20 > python3 runner -d ./tests ./testcases_default_gen That's not what I am complaining about. What you call "configuration" seems to just be "random set of parameters for a random test". In practice, your runner does not seem configurable at all. You just treat all possible configurations of a single test as different tests. My (admittedly very personal) view of what a configuration should be is "run this single test with these parameters varying in these ranges, for this long". Thanks, M. --=20 Without deviation from the norm, progress is not possible.