From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-pl1-f201.google.com (mail-pl1-f201.google.com [209.85.214.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 2DEE019FA93 for ; Tue, 30 Sep 2025 16:36:50 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.201 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1759250213; cv=none; b=a4W6KGJhoupp0l8AgnqK6Qb0FOVDOFHGsnT60rrUB4ti93tR5cxfbRTPEMxk3AfpEzZWm0Sv5u02cBUYXv3sw/Q0dmen+0OOkUaK53mA4R9+gX+jMZBT+TZI/DMzuExs+cnuqt51Q/98EqNm1b2VUNsqYAvwshwko/IrMum674M= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1759250213; c=relaxed/simple; bh=nycwqP6AhjxaXuzHDgYWDB/IbLSgAIDiRc2tUarsNhw=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=rX14SubiV4J/u92DDPc9CzHJr7JH0HfuEywS53eIIHVMoDhGYxZT4+5YD0lIWGruHhO8kop9u01n1okKGiioHEeQtnC2WmI+D0Y65XRP33p9rzMZ02cD8Syx75aNCSbdq4j3Fe/YH3082iZPlig61lIAOBo0bq9ZuzOmApuD3ZU= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--vipinsh.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=a2++wGh6; arc=none smtp.client-ip=209.85.214.201 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--vipinsh.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="a2++wGh6" Received: by mail-pl1-f201.google.com with SMTP id d9443c01a7336-27c62320f16so69613895ad.1 for ; Tue, 30 Sep 2025 09:36:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1759250210; x=1759855010; darn=lists.linux.dev; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=rc4ucMrZZLQgysJDwujeADN63tAlAVtHcaQvgLoGAb0=; b=a2++wGh69gUigJFxKnQJldo53NETqm6V1cg7yBLNeGaqlXM/dAImf/AzzkQ9Iqc4uv 8BuQIgwZszZaB7iv/cnsBOEvMcKdJxjSRoI3u1A2tu5on8/75D6RC9xl5Er4wv9tDFNj RzD0cSCAviqBnHvYhJVDHSrKyJx3KuWN53CK2M08My42fggzD8hy9bw7BGnA3qKbPvZ9 pLvqVgDEWkbzBgm71Yjl5Zt50UI7KN5NLrQju6qIbgsU715sJl9KSqrvrAizyFOLNyeK uK+9EJ9JBa6jptdHLuT26hD8WPH4XYK/IwUTT0qjcd2D6fsDd+uNHma7ysAY9oBTksU1 Hvbw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1759250210; x=1759855010; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=rc4ucMrZZLQgysJDwujeADN63tAlAVtHcaQvgLoGAb0=; b=oGCR+w6xXv0zwdto1Hf6lJWcd7g3a+ZIOJh58or7tMwVA5VuMb3tZJgGBnTn9GuJJ8 JqPmOZVYkGfrVioNUB4SOFFLz+7f33rFIqFmfsn8CpSLY+0Zh2ohhYMIGKeGi21g9tcU G6KXrjO+w3z7US489Q0WuI8wCgbGmVeh+gSeocXDwcsbVnsDaS+Ud+ijbjG51J3vSxNo ySCZfA525cCkwxz69XcL90YeMXiZjs71X20k6g2Jtlq/1jkka30h5YLdyFgLWpQ2kTa1 38lQ4N9t7CG2ixHL/tcxItMF5pKKCYWH8hFKZpsO+rfR4vhqJjS4rI8xAxqh+SaNfYSU Q4lQ== X-Forwarded-Encrypted: i=1; AJvYcCUtyOsPFZFmosnWXQPxoMR6xEV6v0U/eU++syW/UeBgBEyFcmejAxE2U5oxtOZwVwJpgVWNK5A=@lists.linux.dev X-Gm-Message-State: AOJu0YzH7Zub8aZwhiaF9byiZvknxREadNxq7fISILIMsG6sTRt6U3z4 5XvpnlTwG8/WM7BfFOBAh6NRHYsiTL8r0WVVUDeO4otQH5PTOfDQQjMKm2VsYMe7SFtr52GXa+a zLHfSaN4GKg== X-Google-Smtp-Source: AGHT+IHLVHVPK97xGLpcLMiansWk7hFaSR+T9o/MXlCUi7oz8YlG+P3ooNANbdPhcTUhvNECCSwntR5R1+dy X-Received: from pjbgm21.prod.google.com ([2002:a17:90b:1015:b0:330:88c4:627]) (user=vipinsh job=prod-delivery.src-stubby-dispatcher) by 2002:a17:902:ebc8:b0:267:da75:e0f with SMTP id d9443c01a7336-28e7f276e18mr4968975ad.11.1759250210348; Tue, 30 Sep 2025 09:36:50 -0700 (PDT) Date: Tue, 30 Sep 2025 09:36:27 -0700 In-Reply-To: <20250930163635.4035866-1-vipinsh@google.com> Precedence: bulk X-Mailing-List: kvmarm@lists.linux.dev List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20250930163635.4035866-1-vipinsh@google.com> X-Mailer: git-send-email 2.51.0.618.g983fd99d29-goog Message-ID: <20250930163635.4035866-2-vipinsh@google.com> Subject: [PATCH v3 1/9] KVM: selftest: Create KVM selftest runner From: Vipin Sharma To: kvm@vger.kernel.org, kvmarm@lists.linux.dev, kvm-riscv@lists.infradead.org Cc: seanjc@google.com, pbonzini@redhat.com, borntraeger@linux.ibm.com, frankja@linux.ibm.com, imbrenda@linux.ibm.com, anup@brainfault.org, atish.patra@linux.dev, zhaotianrui@loongson.cn, maobibo@loongson.cn, chenhuacai@kernel.org, maz@kernel.org, oliver.upton@linux.dev, ajones@ventanamicro.com, Vipin Sharma Content-Type: text/plain; charset="UTF-8" Implement a basic KVM selftest runner in Python to run selftests. Add command line options to select individual testcase file or a directory containing multiple testcase files. After selecting the tests to run, start their execution and print their final execution status (passed, failed, skipped, no run), stdout and stderr on terminal. Print execution status in colors on the terminals where it is supported to easily distinguish different statuses of the tests execution. If a test fails or times out, then return with a non-zero exit code after all of the tests execution have completed. If none of the tests fails or times out then exit with status 0 Provide some sample test configuration files to demonstrate the execution of the runner. Runner can be started from tools/testing/selftests/kvm directory as: python3 runner --dirs tests OR python3 runner --testcases \ tests/dirty_log_perf_test/no_dirty_log_protect.test This is a very basic implementation of the runner. Next patches will enhance the runner by adding more features like parallelization, dumping output to file system, time limit, out-of-tree builds run, etc. Signed-off-by: Vipin Sharma --- tools/testing/selftests/kvm/.gitignore | 4 +- .../testing/selftests/kvm/runner/__main__.py | 94 +++++++++++++++++++ .../testing/selftests/kvm/runner/selftest.py | 64 +++++++++++++ .../selftests/kvm/runner/test_runner.py | 37 ++++++++ .../2slot_5vcpu_10iter.test | 1 + .../no_dirty_log_protect.test | 1 + 6 files changed, 200 insertions(+), 1 deletion(-) create mode 100644 tools/testing/selftests/kvm/runner/__main__.py create mode 100644 tools/testing/selftests/kvm/runner/selftest.py create mode 100644 tools/testing/selftests/kvm/runner/test_runner.py create mode 100644 tools/testing/selftests/kvm/tests/dirty_log_perf_test/2slot_5vcpu_10iter.test create mode 100644 tools/testing/selftests/kvm/tests/dirty_log_perf_test/no_dirty_log_protect.test diff --git a/tools/testing/selftests/kvm/.gitignore b/tools/testing/selftests/kvm/.gitignore index 1d41a046a7bf..95af97b1ff9e 100644 --- a/tools/testing/selftests/kvm/.gitignore +++ b/tools/testing/selftests/kvm/.gitignore @@ -3,10 +3,12 @@ !/**/ !*.c !*.h +!*.py !*.S !*.sh +!*.test !.gitignore !config !settings !Makefile -!Makefile.kvm \ No newline at end of file +!Makefile.kvm diff --git a/tools/testing/selftests/kvm/runner/__main__.py b/tools/testing/selftests/kvm/runner/__main__.py new file mode 100644 index 000000000000..8d1a78450e41 --- /dev/null +++ b/tools/testing/selftests/kvm/runner/__main__.py @@ -0,0 +1,94 @@ +# SPDX-License-Identifier: GPL-2.0 +# Copyright 2025 Google LLC +# +# Author: vipinsh@google.com (Vipin Sharma) + +import argparse +import logging +import os +import sys + +from test_runner import TestRunner +from selftest import SelftestStatus + + +def cli(): + parser = argparse.ArgumentParser( + prog="KVM Selftests Runner", + formatter_class=argparse.RawTextHelpFormatter, + allow_abbrev=False + ) + + parser.add_argument("-t", + "--testcases", + nargs="*", + default=[], + help="Testcases to run. Provide the space separated testcases paths") + + parser.add_argument("-d", + "--dirs", + nargs="*", + default=[], + help="Run the testcases present in the given directory and all of its sub directories. Provide the space separated paths to add multiple directories.") + + return parser.parse_args() + + +def setup_logging(): + class TerminalColorFormatter(logging.Formatter): + reset = "\033[0m" + red_bold = "\033[31;1m" + green = "\033[32m" + yellow = "\033[33m" + blue = "\033[34m" + + COLORS = { + SelftestStatus.PASSED: green, + SelftestStatus.NO_RUN: blue, + SelftestStatus.SKIPPED: yellow, + SelftestStatus.FAILED: red_bold + } + + def __init__(self, fmt=None, datefmt=None): + super().__init__(fmt, datefmt) + + def format(self, record): + return (self.COLORS.get(record.levelno, "") + + super().format(record) + self.reset) + + logger = logging.getLogger("runner") + logger.setLevel(logging.INFO) + + ch = logging.StreamHandler() + ch_formatter = TerminalColorFormatter(fmt="%(asctime)s | %(message)s", + datefmt="%H:%M:%S") + ch.setFormatter(ch_formatter) + logger.addHandler(ch) + + +def fetch_testcases_in_dirs(dirs): + testcases = [] + for dir in dirs: + for root, child_dirs, files in os.walk(dir): + for file in files: + testcases.append(os.path.join(root, file)) + return testcases + + +def fetch_testcases(args): + testcases = args.testcases + testcases.extend(fetch_testcases_in_dirs(args.dirs)) + # Remove duplicates + testcases = list(dict.fromkeys(testcases)) + return testcases + + +def main(): + args = cli() + setup_logging() + testcases = fetch_testcases(args) + return TestRunner(testcases).start() + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/tools/testing/selftests/kvm/runner/selftest.py b/tools/testing/selftests/kvm/runner/selftest.py new file mode 100644 index 000000000000..34005f83f0c3 --- /dev/null +++ b/tools/testing/selftests/kvm/runner/selftest.py @@ -0,0 +1,64 @@ +# SPDX-License-Identifier: GPL-2.0 +# Copyright 2025 Google LLC +# +# Author: vipinsh@google.com (Vipin Sharma) + +import pathlib +import enum +import os +import subprocess + +class SelftestStatus(enum.IntEnum): + """ + Selftest Status. Integer values are just +1 to the logging.INFO level. + """ + + PASSED = 21 + NO_RUN = 22 + SKIPPED = 23 + FAILED = 24 + + def __str__(self): + return str.__str__(self.name) + +class Selftest: + """ + Represents a single selftest. + + Extract the test execution command from test file and executes it. + """ + + def __init__(self, test_path): + test_command = pathlib.Path(test_path).read_text().strip() + if not test_command: + raise ValueError("Empty test command in " + test_path) + + test_command = os.path.join(".", test_command) + self.exists = os.path.isfile(test_command.split(maxsplit=1)[0]) + self.test_path = test_path + self.command = test_command + self.status = SelftestStatus.NO_RUN + self.stdout = "" + self.stderr = "" + + def run(self): + if not self.exists: + self.stderr = "File doesn't exists." + return + + run_args = { + "universal_newlines": True, + "shell": True, + "stdout": subprocess.PIPE, + "stderr": subprocess.PIPE + } + proc = subprocess.run(self.command, **run_args) + self.stdout = proc.stdout + self.stderr = proc.stderr + + if proc.returncode == 0: + self.status = SelftestStatus.PASSED + elif proc.returncode == 4: + self.status = SelftestStatus.SKIPPED + else: + self.status = SelftestStatus.FAILED diff --git a/tools/testing/selftests/kvm/runner/test_runner.py b/tools/testing/selftests/kvm/runner/test_runner.py new file mode 100644 index 000000000000..4418777d75e3 --- /dev/null +++ b/tools/testing/selftests/kvm/runner/test_runner.py @@ -0,0 +1,37 @@ +# SPDX-License-Identifier: GPL-2.0 +# Copyright 2025 Google LLC +# +# Author: vipinsh@google.com (Vipin Sharma) + +import logging +from selftest import Selftest +from selftest import SelftestStatus + +logger = logging.getLogger("runner") + + +class TestRunner: + def __init__(self, testcases): + self.tests = [] + + for testcase in testcases: + self.tests.append(Selftest(testcase)) + + def _log_result(self, test_result): + logger.info("*** stdout ***\n" + test_result.stdout) + logger.info("*** stderr ***\n" + test_result.stderr) + logger.log(test_result.status, + f"[{test_result.status.name}] {test_result.test_path}") + + def start(self): + ret = 0 + + for test in self.tests: + test.run() + self._log_result(test) + + if (test.status not in [SelftestStatus.PASSED, + SelftestStatus.NO_RUN, + SelftestStatus.SKIPPED]): + ret = 1 + return ret diff --git a/tools/testing/selftests/kvm/tests/dirty_log_perf_test/2slot_5vcpu_10iter.test b/tools/testing/selftests/kvm/tests/dirty_log_perf_test/2slot_5vcpu_10iter.test new file mode 100644 index 000000000000..5b8d56b44a75 --- /dev/null +++ b/tools/testing/selftests/kvm/tests/dirty_log_perf_test/2slot_5vcpu_10iter.test @@ -0,0 +1 @@ +dirty_log_perf_test -x 2 -v 5 -i 10 diff --git a/tools/testing/selftests/kvm/tests/dirty_log_perf_test/no_dirty_log_protect.test b/tools/testing/selftests/kvm/tests/dirty_log_perf_test/no_dirty_log_protect.test new file mode 100644 index 000000000000..ed3490b1d1a1 --- /dev/null +++ b/tools/testing/selftests/kvm/tests/dirty_log_perf_test/no_dirty_log_protect.test @@ -0,0 +1 @@ +dirty_log_perf_test -g -- 2.51.0.618.g983fd99d29-goog