From: Luis Chamberlain <mcgrof@kernel.org>
To: Chuck Lever <cel@kernel.org>, Daniel Gomez <da.gomez@kruces.com>,
kdevops@lists.linux.dev
Cc: Luis Chamberlain <mcgrof@kernel.org>
Subject: [PATCH v2 7/9] all: run black
Date: Tue, 29 Jul 2025 23:01:43 -0700 [thread overview]
Message-ID: <20250730060147.182140-8-mcgrof@kernel.org> (raw)
In-Reply-To: <20250730060147.182140-1-mcgrof@kernel.org>
Run black to fix tons of styling issues with tons of Python scripts.
In order to help bots ensure they don't add odd styling we need a
convention.
Signed-off-by: Luis Chamberlain <mcgrof@kernel.org>
---
.../blktests/augment_expunge_list.py | 95 ++--
.../workflows/blktests/gen-expunge-args.py | 46 +-
.../workflows/blktests/gen-results-dir.py | 39 +-
.../blktests/get_new_expunge_files.py | 16 +-
.../dynamic-kconfig/gen-dynamic-pci.py | 89 ++--
.../workflows/fstests/augment_expunge_list.py | 163 +++++--
.../workflows/fstests/bad_files_summary.py | 46 +-
.../fstests/fstests-checktime-distribution.py | 46 +-
.../workflows/fstests/gen_results_summary.py | 132 +++---
.../fstests/get_new_expunge_files.py | 24 +-
playbooks/python/workflows/fstests/lib/git.py | 21 +-
.../workflows/fstests/xunit_merge_all.py | 33 +-
.../sysbench/sysbench-tps-compare.py | 84 +++-
.../workflows/sysbench/sysbench-tps-plot.py | 36 +-
.../sysbench/sysbench-tps-variance.py | 435 +++++++++++++-----
.../gen_pcie_passthrough_guestfs_xml.py | 49 +-
.../linux-mirror/python/gen-mirror-files.py | 131 +++---
.../linux-mirror/python/start-mirroring.py | 116 +++--
scripts/check_commit_format.py | 28 +-
.../generation/check_for_atomic_calls.py | 71 +--
.../generation/check_for_sleepy_calls.py | 202 +++++---
scripts/detect_whitespace_issues.py | 38 +-
scripts/fix_whitespace_issues.py | 44 +-
scripts/generate_refs.py | 6 +-
scripts/honey-badger.py | 103 +++--
scripts/spdxcheck.py | 201 ++++----
scripts/update_ssh_config_guestfs.py | 49 +-
.../workflows/blktests/blktests_watchdog.py | 75 ++-
scripts/workflows/cxl/gen_qemu_cxl.py | 235 +++++++---
scripts/workflows/fstests/fstests_watchdog.py | 99 ++--
scripts/workflows/generic/crash_report.py | 4 +-
scripts/workflows/generic/crash_watchdog.py | 78 +++-
scripts/workflows/lib/blktests.py | 47 +-
scripts/workflows/lib/crash.py | 12 +-
scripts/workflows/lib/fstests.py | 155 ++++---
scripts/workflows/lib/kssh.py | 178 ++++---
scripts/workflows/lib/systemd_remote.py | 101 ++--
.../workflows/pynfs/check_pynfs_results.py | 17 +-
38 files changed, 2250 insertions(+), 1094 deletions(-)
diff --git a/playbooks/python/workflows/blktests/augment_expunge_list.py b/playbooks/python/workflows/blktests/augment_expunge_list.py
index 4fe856a3..a7b8eb42 100755
--- a/playbooks/python/workflows/blktests/augment_expunge_list.py
+++ b/playbooks/python/workflows/blktests/augment_expunge_list.py
@@ -15,58 +15,81 @@ import configparser
from itertools import chain
oscheck_ansible_python_dir = os.path.dirname(os.path.abspath(__file__))
-oscheck_sort_expunge = oscheck_ansible_python_dir + "/../../../scripts/workflows/fstests/sort-expunges.sh"
+oscheck_sort_expunge = (
+ oscheck_ansible_python_dir + "/../../../scripts/workflows/fstests/sort-expunges.sh"
+)
top_dir = oscheck_ansible_python_dir + "/../../../../"
-blktests_last_kernel = top_dir + 'workflows/blktests/results/last-kernel.txt'
+blktests_last_kernel = top_dir + "workflows/blktests/results/last-kernel.txt"
expunge_name = "failures.txt"
+
def append_line(output_file, test_failure_line):
# We want to now add entries like block/xxx where xxx are digits
output = open(output_file, "a+")
output.write("%s\n" % test_failure_line)
output.close()
+
def is_config_bool_true(config, name):
- if name in config and config[name].strip('\"') == "y":
+ if name in config and config[name].strip('"') == "y":
return True
return False
+
def config_string(config, name):
if name in config:
- return config[name].strip('\"')
+ return config[name].strip('"')
return None
+
def get_config(dotconfig):
- config = configparser.ConfigParser(allow_no_value=True, strict=False, interpolation=None)
+ config = configparser.ConfigParser(
+ allow_no_value=True, strict=False, interpolation=None
+ )
with open(dotconfig) as lines:
lines = chain(("[top]",), lines)
config.read_file(lines)
return config["top"]
return None
+
def read_blktest_last_kernel():
if not os.path.isfile(blktests_last_kernel):
return None
- kfile = open(blktests_last_kernel, 'r')
+ kfile = open(blktests_last_kernel, "r")
all_lines = kfile.readlines()
kfile.close()
for line in all_lines:
return line.strip()
return None
+
def main():
- parser = argparse.ArgumentParser(description='Augments expunge list for blktest')
- parser.add_argument('results', metavar='<directory with results>', type=str,
- help='directory with results file')
- parser.add_argument('outputdir', metavar='<output directory>', type=str,
- help='The directory where to generate the expunge failure.txt ')
- parser.add_argument('--verbose', const=True, default=False, action="store_const",
- help='Print more verbose information')
+ parser = argparse.ArgumentParser(description="Augments expunge list for blktest")
+ parser.add_argument(
+ "results",
+ metavar="<directory with results>",
+ type=str,
+ help="directory with results file",
+ )
+ parser.add_argument(
+ "outputdir",
+ metavar="<output directory>",
+ type=str,
+ help="The directory where to generate the expunge failure.txt ",
+ )
+ parser.add_argument(
+ "--verbose",
+ const=True,
+ default=False,
+ action="store_const",
+ help="Print more verbose information",
+ )
args = parser.parse_args()
expunge_kernel_dir = ""
- dotconfig = top_dir + '/.config'
+ dotconfig = top_dir + "/.config"
config = get_config(dotconfig)
if not config:
sys.stdout.write("%s does not exist\n" % (dotconfig))
@@ -85,7 +108,7 @@ def main():
continue
if not os.path.isfile(f):
continue
- if not f.endswith('.bad') and not f.endswith('.dmesg'):
+ if not f.endswith(".bad") and not f.endswith(".dmesg"):
continue
bad_files.append(f)
@@ -97,13 +120,13 @@ def main():
# f may be results/last-run/nodev/meta/009.dmesg
bad_file_list = f.split("/")
bad_file_list_len = len(bad_file_list) - 1
- bad_file = bad_file_list[bad_file_list_len]
- test_group = bad_file_list[bad_file_list_len-1]
- bdev = bad_file_list[bad_file_list_len-2]
+ bad_file = bad_file_list[bad_file_list_len]
+ test_group = bad_file_list[bad_file_list_len - 1]
+ bdev = bad_file_list[bad_file_list_len - 2]
if args.verbose:
sys.stdout.write("%s\n" % bad_file_list)
- sys.stdout.write("\tbad_file: %s\n" %bad_file)
+ sys.stdout.write("\tbad_file: %s\n" % bad_file)
sys.stdout.write("\ttest_group: %s\n" % test_group)
sys.stdout.write("\tkernel: %s\n" % kernel)
@@ -126,11 +149,11 @@ def main():
sys.exit(1)
# This is like for example block/xxx where xxx are digits
- test_failure_line = test_group + '/' + bad_file_test_number
+ test_failure_line = test_group + "/" + bad_file_test_number
# now to stuff this into expunge files such as:
# expunges/sles/15.3/failures.txt
- expunge_kernel_dir = args.outputdir + '/' + kernel + '/'
+ expunge_kernel_dir = args.outputdir + "/" + kernel + "/"
output_dir = expunge_kernel_dir
output_file = output_dir + expunge_name
shortcut_kernel_dir = None
@@ -145,19 +168,23 @@ def main():
sles_release_name = config_string(config, "CONFIG_KDEVOPS_HOSTS_PREFIX")
sles_release_parts = sles_release_name.split("sp")
if len(sles_release_parts) <= 1:
- sys.stderr.write("Unexpected sles_release_name: %s\n" % sles_release_name)
+ sys.stderr.write(
+ "Unexpected sles_release_name: %s\n" % sles_release_name
+ )
sys.exit(1)
- sles_point_release = sles_release_parts[0].split("sles")[1] + "." + sles_release_parts[1]
+ sles_point_release = (
+ sles_release_parts[0].split("sles")[1] + "." + sles_release_parts[1]
+ )
# This becomes generic release directory, not specific to any
# kernel.
- shortcut_dir = args.outputdir + '/' + "sles/" + sles_point_release + '/'
+ shortcut_dir = args.outputdir + "/" + "sles/" + sles_point_release + "/"
shortcut_kernel_dir = shortcut_dir
shortcut_file = shortcut_dir + expunge_name
else:
ksplit = kernel.split(".")
shortcut_kernel = ksplit[0] + "." + ksplit[1] + "." + ksplit[2]
- shortcut_kernel_dir = args.outputdir + '/' + shortcut_kernel + '/'
+ shortcut_kernel_dir = args.outputdir + "/" + shortcut_kernel + "/"
shortcut_dir = shortcut_kernel_dir
shortcut_file = shortcut_dir + expunge_name
@@ -170,10 +197,13 @@ def main():
os.makedirs(output_dir)
if not os.path.isfile(output_file):
- sys.stdout.write("====%s/%s new failure found file was empty\n" % (test_group, test_failure_line))
+ sys.stdout.write(
+ "====%s/%s new failure found file was empty\n"
+ % (test_group, test_failure_line)
+ )
append_line(output_file, test_failure_line)
else:
- existing_file = open(output_file, 'r')
+ existing_file = open(output_file, "r")
all_lines = existing_file.readlines()
existing_file.close()
found = False
@@ -182,13 +212,18 @@ def main():
found = True
break
if not found:
- sys.stdout.write("%s %s new failure found\n" % (test_group, test_failure_line))
+ sys.stdout.write(
+ "%s %s new failure found\n" % (test_group, test_failure_line)
+ )
append_line(output_file, test_failure_line)
if expunge_kernel_dir != "":
sys.stdout.write("Sorting %s ...\n" % (expunge_kernel_dir))
- sys.stdout.write("Running %s %s...\n" % (oscheck_sort_expunge, expunge_kernel_dir))
+ sys.stdout.write(
+ "Running %s %s...\n" % (oscheck_sort_expunge, expunge_kernel_dir)
+ )
subprocess.call([oscheck_sort_expunge, expunge_kernel_dir])
-if __name__ == '__main__':
+
+if __name__ == "__main__":
main()
diff --git a/playbooks/python/workflows/blktests/gen-expunge-args.py b/playbooks/python/workflows/blktests/gen-expunge-args.py
index e73713f8..4d7f6d4c 100755
--- a/playbooks/python/workflows/blktests/gen-expunge-args.py
+++ b/playbooks/python/workflows/blktests/gen-expunge-args.py
@@ -14,16 +14,37 @@ import os
import sys
import subprocess
+
def main():
- parser = argparse.ArgumentParser(description='Generates expunge arguments to run blktests check based on results directory')
- parser.add_argument('--test-group', metavar='<group>', type=str,
- help='group of tests to focus on otherwise all groups are considered')
- parser.add_argument('results', metavar='<directory with blktests results>', type=str,
- help='directory with blktests results')
- parser.add_argument('--gen-exclude-args', const=True, default=False, action="store_const",
- help='Generate exclude arguments so to be passed to blktests check')
- parser.add_argument('--verbose', const=True, default=False, action="store_const",
- help='Print more verbose information')
+ parser = argparse.ArgumentParser(
+ description="Generates expunge arguments to run blktests check based on results directory"
+ )
+ parser.add_argument(
+ "--test-group",
+ metavar="<group>",
+ type=str,
+ help="group of tests to focus on otherwise all groups are considered",
+ )
+ parser.add_argument(
+ "results",
+ metavar="<directory with blktests results>",
+ type=str,
+ help="directory with blktests results",
+ )
+ parser.add_argument(
+ "--gen-exclude-args",
+ const=True,
+ default=False,
+ action="store_const",
+ help="Generate exclude arguments so to be passed to blktests check",
+ )
+ parser.add_argument(
+ "--verbose",
+ const=True,
+ default=False,
+ action="store_const",
+ help="Print more verbose information",
+ )
args = parser.parse_args()
bad_files = []
@@ -34,7 +55,7 @@ def main():
continue
if not os.path.isfile(f):
continue
- if f.endswith('.bad') or f.endswith('.dmesg'):
+ if f.endswith(".bad") or f.endswith(".dmesg"):
bad_files.append(f)
continue
exclude_args = ""
@@ -58,12 +79,13 @@ def main():
if args.test_group and args.test_group != group:
continue
if args.gen_exclude_args:
- exclude_args += (" -x %s/%s" % (group, fail))
+ exclude_args += " -x %s/%s" % (group, fail)
else:
sys.stdout.write("%s/%s\n" % (group, fail))
if args.gen_exclude_args:
sys.stdout.write("%s\n" % (exclude_args))
-if __name__ == '__main__':
+
+if __name__ == "__main__":
main()
diff --git a/playbooks/python/workflows/blktests/gen-results-dir.py b/playbooks/python/workflows/blktests/gen-results-dir.py
index cb3e76aa..e18504bc 100755
--- a/playbooks/python/workflows/blktests/gen-results-dir.py
+++ b/playbooks/python/workflows/blktests/gen-results-dir.py
@@ -18,7 +18,8 @@ oscheck_ansible_python_dir = os.path.dirname(os.path.abspath(__file__))
top_dir = oscheck_ansible_python_dir + "/../../../../"
results_dir = top_dir + "workflows/blktests/results/"
last_run_dir = results_dir + "last-run/"
-blktests_last_kernel = top_dir + 'workflows/blktests/results/last-kernel.txt'
+blktests_last_kernel = top_dir + "workflows/blktests/results/last-kernel.txt"
+
def clean_empty_dir(target_results):
for i in range(1, 3):
@@ -31,12 +32,23 @@ def clean_empty_dir(target_results):
else:
clean_empty_dir(f)
+
def main():
- parser = argparse.ArgumentParser(description='Get list of expunge files not yet committed in git')
- parser.add_argument('--clean-dir-only', metavar='<clean_dir_only>', type=str, default='none',
- help='Do not perform an evaluation, just clean empty directories on the specified directory')
- parser.add_argument('--copy-all', action='store_true',
- help='Copy all test results without filtering')
+ parser = argparse.ArgumentParser(
+ description="Get list of expunge files not yet committed in git"
+ )
+ parser.add_argument(
+ "--clean-dir-only",
+ metavar="<clean_dir_only>",
+ type=str,
+ default="none",
+ help="Do not perform an evaluation, just clean empty directories on the specified directory",
+ )
+ parser.add_argument(
+ "--copy-all",
+ action="store_true",
+ help="Copy all test results without filtering",
+ )
args = parser.parse_args()
if not os.path.isfile(blktests_last_kernel):
@@ -44,7 +56,7 @@ def main():
sys.exit(1)
kernel = None
- f = open(blktests_last_kernel, 'r')
+ f = open(blktests_last_kernel, "r")
for line in f:
kernel = line.strip()
if not line:
@@ -56,9 +68,11 @@ def main():
clean_empty_dir(args.clean_dir_only)
sys.exit(0)
- target_results = results_dir + kernel + '/'
+ target_results = results_dir + kernel + "/"
if not os.path.isdir(last_run_dir):
- sys.stdout.write("Ignoring last-run directory %s as it is empty ...\n" % (last_run_dir))
+ sys.stdout.write(
+ "Ignoring last-run directory %s as it is empty ...\n" % (last_run_dir)
+ )
sys.exit(0)
sys.stdout.write("Copying %s to %s ...\n" % (last_run_dir, target_results))
copytree(last_run_dir, target_results, dirs_exist_ok=True)
@@ -89,8 +103,8 @@ def main():
test_name = test_name_file_list[0]
test_dir = os.path.dirname(f)
- name_lookup_base = test_dir + '/' + test_name + '*'
- name_lookup = test_dir + '/' + test_name + '.*'
+ name_lookup_base = test_dir + "/" + test_name + "*"
+ name_lookup = test_dir + "/" + test_name + ".*"
listing = glob.glob(name_lookup)
bad_ext_found = False
if len(listing) > 0:
@@ -102,5 +116,6 @@ def main():
os.unlink(r)
clean_empty_dir(target_results)
-if __name__ == '__main__':
+
+if __name__ == "__main__":
main()
diff --git a/playbooks/python/workflows/blktests/get_new_expunge_files.py b/playbooks/python/workflows/blktests/get_new_expunge_files.py
index 45f0e1a3..f1f1241b 100755
--- a/playbooks/python/workflows/blktests/get_new_expunge_files.py
+++ b/playbooks/python/workflows/blktests/get_new_expunge_files.py
@@ -14,10 +14,17 @@ import sys
import subprocess
from lib import git
+
def main():
- parser = argparse.ArgumentParser(description='Get list of expunge files not yet committed in git')
- parser.add_argument('expunge_dir', metavar='<directory with expunge files>', type=str,
- help='directory with expunge files')
+ parser = argparse.ArgumentParser(
+ description="Get list of expunge files not yet committed in git"
+ )
+ parser.add_argument(
+ "expunge_dir",
+ metavar="<directory with expunge files>",
+ type=str,
+ help="directory with expunge files",
+ )
args = parser.parse_args()
block_expunge_dir = args.expunge_dir
@@ -36,5 +43,6 @@ def main():
short_file = f.split("../")[1]
sys.stdout.write("%s\n" % (short_file))
-if __name__ == '__main__':
+
+if __name__ == "__main__":
main()
diff --git a/playbooks/python/workflows/dynamic-kconfig/gen-dynamic-pci.py b/playbooks/python/workflows/dynamic-kconfig/gen-dynamic-pci.py
index ed984ae1..107bb25b 100755
--- a/playbooks/python/workflows/dynamic-kconfig/gen-dynamic-pci.py
+++ b/playbooks/python/workflows/dynamic-kconfig/gen-dynamic-pci.py
@@ -15,27 +15,31 @@ sys_bus_prefix = "/sys/bus/pci/devices/"
debug = 0
+
def get_first_dir(path):
if len(os.listdir(path)) > 0:
return os.listdir(path)[0]
return None
+
def get_sysname(sys_path, entry):
sys_entry_path = sys_path + entry
if not os.path.isfile(sys_entry_path):
return None
- entry_fd = open(sys_entry_path, 'r')
+ entry_fd = open(sys_entry_path, "r")
line = entry_fd.readlines()[0]
line = line.strip()
entry_fd.close()
return line
+
# kconfig does not like some characters
def strip_kconfig_name(name):
- fixed_name = name.replace("\"", "")
+ fixed_name = name.replace('"', "")
fixed_name = fixed_name.replace("'", "")
return fixed_name
+
def get_special_device_nvme(pci_id, IOMMUGroup):
pci_id_name = strip_kconfig_name(pci_id)
sys_path = sys_bus_prefix + pci_id + "/nvme/"
@@ -51,7 +55,13 @@ def get_special_device_nvme(pci_id, IOMMUGroup):
fw = get_sysname(block_sys_path, "firmware_rev")
if not fw:
return None
- return "%s IOMMU group %s - /dev/%s - %s with FW %s" % (pci_id_name, IOMMUGroup, block_device_name, model, fw)
+ return "%s IOMMU group %s - /dev/%s - %s with FW %s" % (
+ pci_id_name,
+ IOMMUGroup,
+ block_device_name,
+ model,
+ fw,
+ )
def get_kconfig_device_name(pci_id, sdevice, IOMMUGroup):
@@ -63,26 +73,31 @@ def get_kconfig_device_name(pci_id, sdevice, IOMMUGroup):
return strip_kconfig_name(default_name)
return strip_kconfig_name(special_name)
+
def add_pcie_kconfig_string(prefix, val, name):
config_name = prefix + "_" + name.upper()
sys.stdout.write("config %s\n" % (config_name))
sys.stdout.write("\tstring\n")
- sys.stdout.write("\tdefault \"%s\"\n" % (strip_kconfig_name(str(val))))
+ sys.stdout.write('\tdefault "%s"\n' % (strip_kconfig_name(str(val))))
sys.stdout.write("\n")
+
def add_pcie_kconfig_name(config_name, sdevice):
sys.stdout.write("config %s\n" % (config_name))
- sys.stdout.write("\tbool \"%s\"\n" % (sdevice))
+ sys.stdout.write('\tbool "%s"\n' % (sdevice))
sys.stdout.write("\tdefault n\n")
sys.stdout.write("\thelp\n")
sys.stdout.write("\t Enabling this will PCI-E passthrough this device onto the\n")
sys.stdout.write("\t target guest.\n")
sys.stdout.write("\n")
+
def add_pcie_kconfig_target(config_name, sdevice):
sys.stdout.write("config %s_TARGET_GUEST\n" % (config_name))
- sys.stdout.write("\tstring \"Taret guest to offload %s\"\n" % (strip_kconfig_name(sdevice)))
- sys.stdout.write("\tdefault \"\"\n")
+ sys.stdout.write(
+ '\tstring "Taret guest to offload %s"\n' % (strip_kconfig_name(sdevice))
+ )
+ sys.stdout.write('\tdefault ""\n')
sys.stdout.write("\tdepends on %s\n" % config_name)
sys.stdout.write("\tdepends on KDEVOPS_LIBVIRT_PCIE_PASSTHROUGH_TYPE_EACH\n")
sys.stdout.write("\thelp\n")
@@ -90,7 +105,10 @@ def add_pcie_kconfig_target(config_name, sdevice):
sys.stdout.write("\t target guest.\n")
sys.stdout.write("\n")
-def add_pcie_kconfig_entry(pci_id, sdevice, domain, bus, slot, function, IOMMUGroup, config_id):
+
+def add_pcie_kconfig_entry(
+ pci_id, sdevice, domain, bus, slot, function, IOMMUGroup, config_id
+):
prefix = passthrough_prefix + "_%04d" % config_id
name = get_kconfig_device_name(pci_id, sdevice, IOMMUGroup)
add_pcie_kconfig_name(prefix, name)
@@ -104,22 +122,23 @@ def add_pcie_kconfig_entry(pci_id, sdevice, domain, bus, slot, function, IOMMUGr
add_pcie_kconfig_string(prefix, slot, "slot")
add_pcie_kconfig_string(prefix, function, "function")
+
def add_new_device(slot, sdevice, IOMMUGroup, possible_id):
# Example expeced format 0000:2d:00.0
- m = re.match(r"^(?P<DOMAIN>\w+):"
- "(?P<BUS>\w+):"
- "(?P<MSLOT>\w+)\."
- "(?P<FUNCTION>\w+)$", slot)
+ m = re.match(
+ r"^(?P<DOMAIN>\w+):" "(?P<BUS>\w+):" "(?P<MSLOT>\w+)\." "(?P<FUNCTION>\w+)$",
+ slot,
+ )
if not m:
return possible_id
possible_id += 1
slot_dict = m.groupdict()
- domain = "0x" + slot_dict['DOMAIN']
- bus = "0x" + slot_dict['BUS']
- mslot = "0x" + slot_dict['MSLOT']
- function = "0x" + slot_dict['FUNCTION']
+ domain = "0x" + slot_dict["DOMAIN"]
+ bus = "0x" + slot_dict["BUS"]
+ mslot = "0x" + slot_dict["MSLOT"]
+ function = "0x" + slot_dict["FUNCTION"]
if debug:
sys.stdout.write("\tslot: %s\n" % (slot))
@@ -130,17 +149,26 @@ def add_new_device(slot, sdevice, IOMMUGroup, possible_id):
sys.stdout.write("\tIOMMUGroup: %s\n" % (IOMMUGroup))
if possible_id == 1:
- sys.stdout.write("# Automatically generated PCI-E passthrough Kconfig by kdevops\n\n")
+ sys.stdout.write(
+ "# Automatically generated PCI-E passthrough Kconfig by kdevops\n\n"
+ )
- add_pcie_kconfig_entry(slot, sdevice, domain, bus, mslot, function, IOMMUGroup, possible_id)
+ add_pcie_kconfig_entry(
+ slot, sdevice, domain, bus, mslot, function, IOMMUGroup, possible_id
+ )
return possible_id
+
def main():
num_candidate_devices = 0
- parser = argparse.ArgumentParser(description='Creates a Kconfig file lspci output')
- parser.add_argument('input', metavar='<input file with lspci -Dvmmm output>', type=str,
- help='input file wth lspci -Dvmmm output')
+ parser = argparse.ArgumentParser(description="Creates a Kconfig file lspci output")
+ parser.add_argument(
+ "input",
+ metavar="<input file with lspci -Dvmmm output>",
+ type=str,
+ help="input file wth lspci -Dvmmm output",
+ )
args = parser.parse_args()
lspci_output = args.input
@@ -149,7 +177,7 @@ def main():
sys.stdout.write("input file did not exist: %s\n" % (lspci_output))
sys.exit(1)
- lspci = open(lspci_output, 'r')
+ lspci = open(lspci_output, "r")
all_lines = lspci.readlines()
lspci.close()
@@ -159,17 +187,18 @@ def main():
for line in all_lines:
line = line.strip()
- m = re.match(r"^(?P<TAG>\w+):"
- "(?P<STRING>.*)$", line)
+ m = re.match(r"^(?P<TAG>\w+):" "(?P<STRING>.*)$", line)
if not m:
continue
eval_line = m.groupdict()
- tag = eval_line['TAG']
- data = eval_line['STRING']
+ tag = eval_line["TAG"]
+ data = eval_line["STRING"]
data = data.strip()
if tag == "Slot":
if sdevice:
- num_candidate_devices = add_new_device(slot, sdevice, IOMMUGroup, num_candidate_devices)
+ num_candidate_devices = add_new_device(
+ slot, sdevice, IOMMUGroup, num_candidate_devices
+ )
slot = data
sdevice = None
IOMMUGroup = None
@@ -180,11 +209,13 @@ def main():
# Handle the last device
if sdevice and slot:
- num_candidate_devices = add_new_device(slot, sdevice, IOMMUGroup, num_candidate_devices)
+ num_candidate_devices = add_new_device(
+ slot, sdevice, IOMMUGroup, num_candidate_devices
+ )
add_pcie_kconfig_string(passthrough_prefix, num_candidate_devices, "NUM_DEVICES")
os.unlink(lspci_output)
-if __name__ == '__main__':
+if __name__ == "__main__":
main()
diff --git a/playbooks/python/workflows/fstests/augment_expunge_list.py b/playbooks/python/workflows/fstests/augment_expunge_list.py
index 9265cd8b..7f31401e 100755
--- a/playbooks/python/workflows/fstests/augment_expunge_list.py
+++ b/playbooks/python/workflows/fstests/augment_expunge_list.py
@@ -16,44 +16,69 @@ import configparser
from itertools import chain
oscheck_ansible_python_dir = os.path.dirname(os.path.abspath(__file__))
-oscheck_sort_expunge = oscheck_ansible_python_dir + "/../../../scripts/workflows/fstests/sort-expunges.sh"
+oscheck_sort_expunge = (
+ oscheck_ansible_python_dir + "/../../../scripts/workflows/fstests/sort-expunges.sh"
+)
top_dir = oscheck_ansible_python_dir + "/../../../../"
+
def append_line(output_file, test_failure_line):
# We want to now add entries like generic/xxx where xxx are digits
output = open(output_file, "a+")
output.write("%s\n" % test_failure_line)
output.close()
+
def is_config_bool_true(config, name):
- if name in config and config[name].strip('\"') == "y":
+ if name in config and config[name].strip('"') == "y":
return True
return False
+
def get_config(dotconfig):
- config = configparser.ConfigParser(allow_no_value=True, strict=False, interpolation=None)
+ config = configparser.ConfigParser(
+ allow_no_value=True, strict=False, interpolation=None
+ )
with open(dotconfig) as lines:
lines = chain(("[top]",), lines)
config.read_file(lines)
return config["top"]
return None
+
def main():
- parser = argparse.ArgumentParser(description='Augments expunge list for oscheck')
- parser.add_argument('filesystem', metavar='<filesystem name>', type=str,
- help='filesystem which was tested')
- parser.add_argument('results', metavar='<directory with results>', type=str,
- help='directory with results file')
- parser.add_argument('outputdir', metavar='<output directory>', type=str,
- help='The directory where to generate the expunge lists to')
- parser.add_argument('--verbose', const=True, default=False, action="store_const",
- help='Print more verbose information')
+ parser = argparse.ArgumentParser(description="Augments expunge list for oscheck")
+ parser.add_argument(
+ "filesystem",
+ metavar="<filesystem name>",
+ type=str,
+ help="filesystem which was tested",
+ )
+ parser.add_argument(
+ "results",
+ metavar="<directory with results>",
+ type=str,
+ help="directory with results file",
+ )
+ parser.add_argument(
+ "outputdir",
+ metavar="<output directory>",
+ type=str,
+ help="The directory where to generate the expunge lists to",
+ )
+ parser.add_argument(
+ "--verbose",
+ const=True,
+ default=False,
+ action="store_const",
+ help="Print more verbose information",
+ )
args = parser.parse_args()
expunge_kernel_dir = ""
-# all_files = os.listdir(args.results)
- dotconfig = top_dir + '/.config'
+ # all_files = os.listdir(args.results)
+ dotconfig = top_dir + "/.config"
config = get_config(dotconfig)
if not config:
sys.stdout.write("%s does not exist\n" % (dotconfig))
@@ -67,7 +92,7 @@ def main():
continue
if not os.path.isfile(f):
continue
- if f.endswith('.bad') or f.endswith('.dmesg'):
+ if f.endswith(".bad") or f.endswith(".dmesg"):
bad_files.append(f)
for f in bad_files:
if args.verbose:
@@ -78,15 +103,15 @@ def main():
# where xxx are digits
bad_file_list = f.split("/")
bad_file_list_len = len(bad_file_list) - 1
- bad_file = bad_file_list[bad_file_list_len]
- test_group = bad_file_list[bad_file_list_len-1]
- section = bad_file_list[bad_file_list_len-2]
- kernel = bad_file_list[bad_file_list_len-3]
- hostname = bad_file_list[bad_file_list_len-4]
+ bad_file = bad_file_list[bad_file_list_len]
+ test_group = bad_file_list[bad_file_list_len - 1]
+ section = bad_file_list[bad_file_list_len - 2]
+ kernel = bad_file_list[bad_file_list_len - 3]
+ hostname = bad_file_list[bad_file_list_len - 4]
if args.verbose:
sys.stdout.write("%s\n" % bad_file_list)
- sys.stdout.write("\tbad_file: %s\n" %bad_file)
+ sys.stdout.write("\tbad_file: %s\n" % bad_file)
sys.stdout.write("\ttest_group: %s\n" % test_group)
sys.stdout.write("\tsection: %s\n" % section)
sys.stdout.write("\thostname: %s\n" % hostname)
@@ -94,20 +119,22 @@ def main():
bad_file_parts = bad_file.split(".")
bad_file_test_number = bad_file_parts[0]
# This is like for example generic/xxx where xxx are digits
- test_failure_line = test_group + '/' + bad_file_test_number
+ test_failure_line = test_group + "/" + bad_file_test_number
# now to stuff this into expunge files such as:
# path/4.19.17/xfs/unassigned/xfs_nocrc.txt
- expunge_kernel_dir = args.outputdir + '/' + kernel + '/' + args.filesystem + '/'
- output_dir = expunge_kernel_dir + 'unassigned/'
- output_file = output_dir + section + '.txt'
+ expunge_kernel_dir = args.outputdir + "/" + kernel + "/" + args.filesystem + "/"
+ output_dir = expunge_kernel_dir + "unassigned/"
+ output_file = output_dir + section + ".txt"
base_kernel = kernel
if base_kernel.endswith("+"):
base_kernel = kernel.replace("+", "")
- base_expunge_kernel_dir = args.outputdir + '/' + base_kernel + '/' + args.filesystem + '/'
- base_output_dir = base_expunge_kernel_dir + 'unassigned/'
- base_output_file = base_output_dir + section + '.txt'
+ base_expunge_kernel_dir = (
+ args.outputdir + "/" + base_kernel + "/" + args.filesystem + "/"
+ )
+ base_output_dir = base_expunge_kernel_dir + "unassigned/"
+ base_output_file = base_output_dir + section + ".txt"
shortcut_kernel_dir = None
shortcut_dir = None
@@ -124,24 +151,39 @@ def main():
sles_release_name = sles_release_parts[0]
sles_release_parts = sles_release_name.split("sp")
if len(sles_release_parts) <= 1:
- sys.stderr.write("Unexpected sles_release_name: %s\n" % sles_release_name)
+ sys.stderr.write(
+ "Unexpected sles_release_name: %s\n" % sles_release_name
+ )
sys.exit(1)
sles_point_release = sles_release_parts[0] + "." + sles_release_parts[1]
# This becomes generic release directory, not specific to any
# kernel.
- shortcut_kernel_dir = args.outputdir + '/' + "sles/" + sles_point_release + '/' + args.filesystem + '/'
+ shortcut_kernel_dir = (
+ args.outputdir
+ + "/"
+ + "sles/"
+ + sles_point_release
+ + "/"
+ + args.filesystem
+ + "/"
+ )
- shortcut_dir = shortcut_kernel_dir + 'unassigned/'
- shortcut_file = shortcut_dir + section + '.txt'
+ shortcut_dir = shortcut_kernel_dir + "unassigned/"
+ shortcut_file = shortcut_dir + section + ".txt"
else:
ksplit = kernel.split(".")
shortcut_kernel = ksplit[0] + "." + ksplit[1] + "." + ksplit[2]
- shortcut_kernel_dir = args.outputdir + '/' + shortcut_kernel + '/' + args.filesystem + '/'
- shortcut_dir = shortcut_kernel_dir + 'unassigned/'
- shortcut_file = shortcut_dir + section + '.txt'
+ shortcut_kernel_dir = (
+ args.outputdir + "/" + shortcut_kernel + "/" + args.filesystem + "/"
+ )
+ shortcut_dir = shortcut_kernel_dir + "unassigned/"
+ shortcut_file = shortcut_dir + section + ".txt"
elif is_config_bool_true(config, "CONFIG_LIBVIRT_OPENSUSE"):
- if is_config_bool_true(config, "CONFIG_WORKFLOW_KOTD_ENABLE") and "leap" in hostname:
+ if (
+ is_config_bool_true(config, "CONFIG_WORKFLOW_KOTD_ENABLE")
+ and "leap" in hostname
+ ):
leap_host_parts = hostname.split("leap")
if len(leap_host_parts) <= 1:
sys.stderr.write("Invalid hostname: %s\n" % hostname)
@@ -150,22 +192,34 @@ def main():
leap_release_name = leap_release_parts[0]
leap_release_parts = leap_release_name.split("sp")
if len(leap_release_parts) <= 1:
- sys.stderr.write("Unexpected sles_release_name: %s\n" % leap_release_name)
+ sys.stderr.write(
+ "Unexpected sles_release_name: %s\n" % leap_release_name
+ )
sys.exit(1)
leap_point_release = leap_release_parts[0] + "." + leap_release_parts[1]
# This becomes generic release directory, not specific to any
# kernel.
- shortcut_kernel_dir = args.outputdir + '/' + "opensuse-leap/" + leap_point_release + '/' + args.filesystem + '/'
+ shortcut_kernel_dir = (
+ args.outputdir
+ + "/"
+ + "opensuse-leap/"
+ + leap_point_release
+ + "/"
+ + args.filesystem
+ + "/"
+ )
- shortcut_dir = shortcut_kernel_dir + 'unassigned/'
- shortcut_file = shortcut_dir + section + '.txt'
+ shortcut_dir = shortcut_kernel_dir + "unassigned/"
+ shortcut_file = shortcut_dir + section + ".txt"
else:
ksplit = kernel.split(".")
shortcut_kernel = ksplit[0] + "." + ksplit[1] + "." + ksplit[2]
- shortcut_kernel_dir = args.outputdir + '/' + shortcut_kernel + '/' + args.filesystem + '/'
- shortcut_dir = shortcut_kernel_dir + 'unassigned/'
- shortcut_file = shortcut_dir + section + '.txt'
+ shortcut_kernel_dir = (
+ args.outputdir + "/" + shortcut_kernel + "/" + args.filesystem + "/"
+ )
+ shortcut_dir = shortcut_kernel_dir + "unassigned/"
+ shortcut_file = shortcut_dir + section + ".txt"
if not os.path.isdir(output_dir):
if shortcut_dir and os.path.isdir(shortcut_dir):
@@ -173,7 +227,10 @@ def main():
output_file = shortcut_file
expunge_kernel_dir = shortcut_kernel_dir
elif base_kernel != kernel and os.path.isdir(base_output_dir):
- sys.stdout.write("<== expunges for %s not found but found base kernel %s expunge directory ==>\n" % (kernel, base_kernel))
+ sys.stdout.write(
+ "<== expunges for %s not found but found base kernel %s expunge directory ==>\n"
+ % (kernel, base_kernel)
+ )
expunge_kernel_dir = base_expunge_kernel_dir
output_dir = base_output_dir
output_file = base_output_file
@@ -182,10 +239,13 @@ def main():
os.makedirs(output_dir)
if not os.path.isfile(output_file):
- sys.stdout.write("====%s %s new failure found file was empty\n" % (section, test_failure_line))
+ sys.stdout.write(
+ "====%s %s new failure found file was empty\n"
+ % (section, test_failure_line)
+ )
append_line(output_file, test_failure_line)
else:
- existing_file = open(output_file, 'r')
+ existing_file = open(output_file, "r")
all_lines = existing_file.readlines()
existing_file.close()
found = False
@@ -194,13 +254,18 @@ def main():
found = True
break
if not found:
- sys.stdout.write("%s %s new failure found\n" % (section, test_failure_line))
+ sys.stdout.write(
+ "%s %s new failure found\n" % (section, test_failure_line)
+ )
append_line(output_file, test_failure_line)
if expunge_kernel_dir != "":
sys.stdout.write("Sorting %s ...\n" % (expunge_kernel_dir))
- sys.stdout.write("Running %s %s...\n" % (oscheck_sort_expunge, expunge_kernel_dir))
+ sys.stdout.write(
+ "Running %s %s...\n" % (oscheck_sort_expunge, expunge_kernel_dir)
+ )
subprocess.call([oscheck_sort_expunge, expunge_kernel_dir])
-if __name__ == '__main__':
+
+if __name__ == "__main__":
main()
diff --git a/playbooks/python/workflows/fstests/bad_files_summary.py b/playbooks/python/workflows/fstests/bad_files_summary.py
index 164409b5..d0cf4058 100755
--- a/playbooks/python/workflows/fstests/bad_files_summary.py
+++ b/playbooks/python/workflows/fstests/bad_files_summary.py
@@ -11,6 +11,7 @@ import argparse
import os
import sys
+
def parse_results_ascii(sections, results, kernel, filesystem):
sys.stdout.write("%s on %s\n" % (filesystem, kernel))
for section in sections:
@@ -18,6 +19,7 @@ def parse_results_ascii(sections, results, kernel, filesystem):
for test in results[section]:
sys.stdout.write("\t%s\n" % test)
+
def parse_results_html(sections, results, kernel, filesystem):
sys.stdout.write("<html><title>%s on %s</title><body>" % (filesystem, kernel))
sys.stdout.write("<h1>%s on %s</h1>\n" % (filesystem, kernel))
@@ -33,15 +35,28 @@ def parse_results_html(sections, results, kernel, filesystem):
sys.stdout.write("</tr>\n")
sys.stdout.write("</table></body></html>")
+
def main():
- parser = argparse.ArgumentParser(description='generate html file from results')
- parser.add_argument('filesystem', metavar='<filesystem name>', type=str,
- help='filesystem which was tested')
- parser.add_argument('results', metavar='<directory with results>', type=str,
- help='directory with results file')
- parser.add_argument('--format', metavar='<output format>', type=str,
- help='Output format: ascii html, the default is ascii',
- default='txt')
+ parser = argparse.ArgumentParser(description="generate html file from results")
+ parser.add_argument(
+ "filesystem",
+ metavar="<filesystem name>",
+ type=str,
+ help="filesystem which was tested",
+ )
+ parser.add_argument(
+ "results",
+ metavar="<directory with results>",
+ type=str,
+ help="directory with results file",
+ )
+ parser.add_argument(
+ "--format",
+ metavar="<output format>",
+ type=str,
+ help="Output format: ascii html, the default is ascii",
+ default="txt",
+ )
args = parser.parse_args()
results = dict()
sections = list()
@@ -51,27 +66,27 @@ def main():
for root, dirs, all_files in os.walk(args.results):
for fname in all_files:
f = os.path.join(root, fname)
- #sys.stdout.write("%s\n" % f)
+ # sys.stdout.write("%s\n" % f)
if os.path.isdir(f):
continue
if not os.path.isfile(f):
continue
- if not f.endswith('.bad'):
+ if not f.endswith(".bad"):
continue
# f may be results/oscheck-xfs/4.19.0-4-amd64/xfs/generic/091.out.bad
bad_file_list = f.split("/")
bad_file_list_len = len(bad_file_list) - 1
bad_file = bad_file_list[bad_file_list_len]
- test_type = bad_file_list[bad_file_list_len-1]
- section = bad_file_list[bad_file_list_len-2]
- kernel = bad_file_list[bad_file_list_len-3]
+ test_type = bad_file_list[bad_file_list_len - 1]
+ section = bad_file_list[bad_file_list_len - 2]
+ kernel = bad_file_list[bad_file_list_len - 3]
bad_file_parts = bad_file.split(".")
bad_file_part_len = len(bad_file_parts) - 1
bad_file_test_number = bad_file_parts[bad_file_part_len - 2]
# This is like for example generic/091
- test_failure_line = test_type + '/' + bad_file_test_number
+ test_failure_line = test_type + "/" + bad_file_test_number
test_section = results.get(section)
if not test_section:
@@ -86,5 +101,6 @@ def main():
else:
parse_results_ascii(sections, results, kernel, args.filesystem)
-if __name__ == '__main__':
+
+if __name__ == "__main__":
main()
diff --git a/playbooks/python/workflows/fstests/fstests-checktime-distribution.py b/playbooks/python/workflows/fstests/fstests-checktime-distribution.py
index b84e0a29..75025fa2 100755
--- a/playbooks/python/workflows/fstests/fstests-checktime-distribution.py
+++ b/playbooks/python/workflows/fstests/fstests-checktime-distribution.py
@@ -16,12 +16,21 @@ import subprocess
import collections
oscheck_ansible_python_dir = os.path.dirname(os.path.abspath(__file__))
-oscheck_sort_expunge = oscheck_ansible_python_dir + "/../../../scripts/workflows/fstests/sort-expunges.sh"
+oscheck_sort_expunge = (
+ oscheck_ansible_python_dir + "/../../../scripts/workflows/fstests/sort-expunges.sh"
+)
+
def main():
- parser = argparse.ArgumentParser(description='Creates check.time.distribution files for all found check.time files')
- parser.add_argument('results', metavar='<directory with check.time files>', type=str,
- help='directory with check.time files')
+ parser = argparse.ArgumentParser(
+ description="Creates check.time.distribution files for all found check.time files"
+ )
+ parser.add_argument(
+ "results",
+ metavar="<directory with check.time files>",
+ type=str,
+ help="directory with check.time files",
+ )
args = parser.parse_args()
expunge_kernel_dir = ""
@@ -31,22 +40,22 @@ def main():
for root, dirs, all_files in os.walk(args.results):
for fname in all_files:
f = os.path.join(root, fname)
- #sys.stdout.write("%s\n" % f)
+ # sys.stdout.write("%s\n" % f)
if os.path.isdir(f):
continue
if not os.path.isfile(f):
continue
- if not f.endswith('check.time'):
+ if not f.endswith("check.time"):
continue
# f may be results/oscheck-xfs/4.19.0-4-amd64/check.time
- time_distribution = f + '.distribution'
+ time_distribution = f + ".distribution"
if os.path.isfile(time_distribution):
os.unlink(time_distribution)
- checktime = open(f, 'r')
- distribution = open(time_distribution, 'w')
+ checktime = open(f, "r")
+ distribution = open(time_distribution, "w")
sys.stdout.write("checktime: %s\n" % f)
@@ -57,17 +66,17 @@ def main():
num_tests = 0
for line in all_lines:
line = line.strip()
- m = re.match(r"^(?P<GROUP>\w+)/"
- "(?P<NUMBER>\d+)\s+"
- "(?P<TIME>\d+)$", line)
+ m = re.match(
+ r"^(?P<GROUP>\w+)/" "(?P<NUMBER>\d+)\s+" "(?P<TIME>\d+)$", line
+ )
if not m:
continue
testline = m.groupdict()
num_tests += 1
- if int(testline['TIME']) in results:
- results[int(testline['TIME'])] += 1
+ if int(testline["TIME"]) in results:
+ results[int(testline["TIME"])] += 1
else:
- results[int(testline['TIME'])] = 1
+ results[int(testline["TIME"])] = 1
od = collections.OrderedDict(sorted(results.items()))
v_total = 0
@@ -76,8 +85,11 @@ def main():
v_total += v
if num_tests != v_total:
- sys.stdout.write("Unexpected error, total tests: %d but computed sum test: %d\n" % (num_tests, v_total))
+ sys.stdout.write(
+ "Unexpected error, total tests: %d but computed sum test: %d\n"
+ % (num_tests, v_total)
+ )
-if __name__ == '__main__':
+if __name__ == "__main__":
main()
diff --git a/playbooks/python/workflows/fstests/gen_results_summary.py b/playbooks/python/workflows/fstests/gen_results_summary.py
index 28dc064c..c0d0702b 100644
--- a/playbooks/python/workflows/fstests/gen_results_summary.py
+++ b/playbooks/python/workflows/fstests/gen_results_summary.py
@@ -20,22 +20,25 @@ import time
from datetime import datetime
from junitparser import JUnitXml, Property, Properties, Failure, Error, Skipped
+
def get_results(dirroot, results_file):
"""Return a list of files named results_file in a directory hierarchy"""
for dirpath, _dirs, filenames in os.walk(dirroot):
if results_file in filenames:
- yield dirpath + '/' + results_file
+ yield dirpath + "/" + results_file
+
def parse_timestamp(timestamp):
"""Parse an ISO-8601-like timestamp as found in an xUnit file."""
if timestamp == "":
return 0
- for fmt in ('%Y-%m-%dT%H:%M:%S%z', '%Y-%m-%dT%H:%M:%S'):
+ for fmt in ("%Y-%m-%dT%H:%M:%S%z", "%Y-%m-%dT%H:%M:%S"):
try:
return time.mktime(datetime.strptime(timestamp, fmt).timetuple())
except ValueError:
pass
- raise ValueError('no valid timestamp format found')
+ raise ValueError("no valid timestamp format found")
+
def failed_tests(testsuite):
"""This iterator the failed tests from the testsuite."""
@@ -43,6 +46,7 @@ def failed_tests(testsuite):
if isinstance(testcase.result, Failure):
yield testcase
+
def get_property(props, key):
"""Return the value of the first property with the given name"""
if props is None:
@@ -52,6 +56,7 @@ def get_property(props, key):
return prop.value
return None
+
def get_properties(props, key):
"""An interator which returns values of properties with a given name."""
if props is None:
@@ -60,6 +65,7 @@ def get_properties(props, key):
if prop.name == key:
yield prop.value
+
def remove_properties(props, key):
"""Remove properties with a given name."""
if props is None:
@@ -68,6 +74,7 @@ def remove_properties(props, key):
if prop.name == key:
props.remove(prop)
+
def print_tests(out_f, testsuite, result_type, type_label):
"""Print all of the tests which match a particular result_type"""
found = False
@@ -81,17 +88,18 @@ def print_tests(out_f, testsuite, result_type, type_label):
if result is None:
continue
if not found:
- out_f.write(' %s: ' % type_label)
+ out_f.write(" %s: " % type_label)
pos = len(type_label) + 4
found = True
name_len = len(testcase.name) + 1
pos += name_len + 1
if pos > 76:
- out_f.write('\n ')
+ out_f.write("\n ")
pos = name_len + 5
- out_f.write(testcase.name + ' ')
+ out_f.write(testcase.name + " ")
if found:
- out_f.write('\n')
+ out_f.write("\n")
+
def total_tests(testsuites):
"""Print the total number of tests in an array of testsuites"""
@@ -101,6 +109,7 @@ def total_tests(testsuites):
total += testsuite.tests
return total
+
def sum_testsuites(testsuites):
"""Summarize all of the test suite statistics"""
runtime = 0
@@ -116,6 +125,7 @@ def sum_testsuites(testsuites):
errors += testsuite.errors
return (tests, skipped, failures, errors, runtime)
+
def print_summary(out_f, testsuite, verbose, print_section):
"""Print a summary for a particular test suite
@@ -126,9 +136,9 @@ def print_summary(out_f, testsuite, verbose, print_section):
ext4/bigalloc 244 tests, 25 skipped, 5 errors, 880 seconds
generic/219 generic/235 generic/422 generic/451 generic/456
"""
- cfg = get_property(testsuite.properties(), 'TESTCFG')
+ cfg = get_property(testsuite.properties(), "TESTCFG")
if cfg is None:
- cfg = get_property(testsuite.properties(), 'FSTESTCFG')
+ cfg = get_property(testsuite.properties(), "FSTESTCFG")
runtime = testsuite.time
tests = testsuite.tests
@@ -139,70 +149,74 @@ def print_summary(out_f, testsuite, verbose, print_section):
for test_case in testsuite:
classname = test_case.classname
class_list = classname.split(".")
- section = class_list[len(class_list)-1]
+ section = class_list[len(class_list) - 1]
break
- out_f.write('%s: %d tests, ' % (section, tests))
+ out_f.write("%s: %d tests, " % (section, tests))
else:
- out_f.write('%s: %d tests, ' % (cfg, tests))
+ out_f.write("%s: %d tests, " % (cfg, tests))
if failures > 0:
- out_f.write('%d failures, ' % failures)
+ out_f.write("%d failures, " % failures)
if errors > 0:
- out_f.write('%d errors, ' % errors)
+ out_f.write("%d errors, " % errors)
if skipped > 0:
- out_f.write('%d skipped, ' % skipped)
+ out_f.write("%d skipped, " % skipped)
if runtime is None:
runtime = 0
- out_f.write('%d seconds\n' % runtime)
+ out_f.write("%d seconds\n" % runtime)
if verbose:
for test_case in testsuite:
- status = 'Pass'
+ status = "Pass"
for result in test_case.result:
if isinstance(result, Failure):
- status = 'Failed'
+ status = "Failed"
if isinstance(result, Skipped):
- status = 'Skipped'
+ status = "Skipped"
if isinstance(result, Error):
- status = 'Error'
- out_f.write(" %-12s %-8s %ds\n" %
- (test_case.name, status, test_case.time))
+ status = "Error"
+ out_f.write(" %-12s %-8s %ds\n" % (test_case.name, status, test_case.time))
else:
if failures > 0:
- print_tests(out_f, testsuite, Failure, 'Failures')
+ print_tests(out_f, testsuite, Failure, "Failures")
if errors > 0:
- print_tests(out_f, testsuite, Error, 'Errors')
+ print_tests(out_f, testsuite, Error, "Errors")
+
def print_property_line(out_f, props, key):
"""Print a line containing the given property."""
value = get_property(props, key)
if value is not None and value != "":
- out_f.write('%-10s %s\n' % (key + ':', value))
+ out_f.write("%-10s %s\n" % (key + ":", value))
+
def print_properties(out_f, props, key):
"""Print multiple property lines."""
for value in get_properties(props, key):
- out_f.write('%-10s %s\n' % (key + ':', value))
+ out_f.write("%-10s %s\n" % (key + ":", value))
+
def print_header(out_f, props):
"""Print the header of the report."""
- print_property_line(out_f, props, 'TESTRUNID')
- print_property_line(out_f, props, 'KERNEL')
- print_property_line(out_f, props, 'CMDLINE')
- print_property_line(out_f, props, 'CPUS')
- print_property_line(out_f, props, 'MEM')
- print_property_line(out_f, props, 'MNTOPTS')
- out_f.write('\n')
+ print_property_line(out_f, props, "TESTRUNID")
+ print_property_line(out_f, props, "KERNEL")
+ print_property_line(out_f, props, "CMDLINE")
+ print_property_line(out_f, props, "CPUS")
+ print_property_line(out_f, props, "MEM")
+ print_property_line(out_f, props, "MNTOPTS")
+ out_f.write("\n")
+
def print_trailer(out_f, props):
"""Print the trailer of the report."""
- out_f.write('\n')
- print_property_line(out_f, props, 'FSTESTIMG')
- print_property_line(out_f, props, 'FSTESTPRJ')
- print_properties(out_f, props, 'FSTESTVER')
- print_property_line(out_f, props, 'FSTESTCFG')
- print_property_line(out_f, props, 'FSTESTSET')
- print_property_line(out_f, props, 'FSTESTEXC')
- print_property_line(out_f, props, 'FSTESTOPT')
- print_property_line(out_f, props, 'GCE ID')
+ out_f.write("\n")
+ print_property_line(out_f, props, "FSTESTIMG")
+ print_property_line(out_f, props, "FSTESTPRJ")
+ print_properties(out_f, props, "FSTESTVER")
+ print_property_line(out_f, props, "FSTESTCFG")
+ print_property_line(out_f, props, "FSTESTSET")
+ print_property_line(out_f, props, "FSTESTEXC")
+ print_property_line(out_f, props, "FSTESTOPT")
+ print_property_line(out_f, props, "GCE ID")
+
def check_for_ltm(results_dir, props):
"""Check to see if the results directory was created by the LTM and
@@ -210,15 +224,15 @@ def check_for_ltm(results_dir, props):
mode.
"""
try:
- out_f = open(os.path.join(results_dir, 'ltm-run-stats'))
+ out_f = open(os.path.join(results_dir, "ltm-run-stats"))
for line in out_f:
- key, value = line.split(': ', 1)
- value = value.rstrip('\n').strip('"')
+ key, value = line.split(": ", 1)
+ value = value.rstrip("\n").strip('"')
remove_properties(props, key)
props.add_property(Property(key, value))
out_f.close()
- remove_properties(props, 'GCE ID')
- remove_properties(props, 'FSTESTCFG')
+ remove_properties(props, "GCE ID")
+ remove_properties(props, "FSTESTCFG")
return True
except IOError:
try:
@@ -227,9 +241,15 @@ def check_for_ltm(results_dir, props):
pass
return False
-def gen_results_summary(results_dir, output_fn=None, merge_fn=None,
- verbose=False, print_section=False,
- results_file='results.xml'):
+
+def gen_results_summary(
+ results_dir,
+ output_fn=None,
+ merge_fn=None,
+ verbose=False,
+ print_section=False,
+ results_file="results.xml",
+):
"""Scan a results directory and generate a summary file"""
reports = []
combined = JUnitXml()
@@ -263,16 +283,18 @@ def gen_results_summary(results_dir, output_fn=None, merge_fn=None,
combined.add_testsuite(testsuite)
nr_files += 1
- out_f.write('Totals: %d tests, %d skipped, %d failures, %d errors, %ds\n' \
- % sum_testsuites(reports))
+ out_f.write(
+ "Totals: %d tests, %d skipped, %d failures, %d errors, %ds\n"
+ % sum_testsuites(reports)
+ )
print_trailer(out_f, props)
if merge_fn is not None:
combined.update_statistics()
- combined.write(merge_fn + '.new')
+ combined.write(merge_fn + ".new")
if os.path.exists(merge_fn):
- os.rename(merge_fn, merge_fn + '.bak')
- os.rename(merge_fn + '.new', merge_fn)
+ os.rename(merge_fn, merge_fn + ".bak")
+ os.rename(merge_fn + ".new", merge_fn)
return nr_files
diff --git a/playbooks/python/workflows/fstests/get_new_expunge_files.py b/playbooks/python/workflows/fstests/get_new_expunge_files.py
index a01330a2..5b164044 100755
--- a/playbooks/python/workflows/fstests/get_new_expunge_files.py
+++ b/playbooks/python/workflows/fstests/get_new_expunge_files.py
@@ -14,12 +14,23 @@ import sys
import subprocess
from lib import git
+
def main():
- parser = argparse.ArgumentParser(description='Get list of expunge files not yet committed in git')
- parser.add_argument('filesystem', metavar='<filesystem name>', type=str,
- help='filesystem which was tested')
- parser.add_argument('expunge_dir', metavar='<directory with expunge files>', type=str,
- help='directory with expunge files')
+ parser = argparse.ArgumentParser(
+ description="Get list of expunge files not yet committed in git"
+ )
+ parser.add_argument(
+ "filesystem",
+ metavar="<filesystem name>",
+ type=str,
+ help="filesystem which was tested",
+ )
+ parser.add_argument(
+ "expunge_dir",
+ metavar="<directory with expunge files>",
+ type=str,
+ help="directory with expunge files",
+ )
args = parser.parse_args()
fs_expunge_dir = args.expunge_dir
@@ -41,5 +52,6 @@ def main():
short_file = f.split("../")[1]
sys.stdout.write("%s\n" % (short_file))
-if __name__ == '__main__':
+
+if __name__ == "__main__":
main()
diff --git a/playbooks/python/workflows/fstests/lib/git.py b/playbooks/python/workflows/fstests/lib/git.py
index 1884f565..fe8cf857 100644
--- a/playbooks/python/workflows/fstests/lib/git.py
+++ b/playbooks/python/workflows/fstests/lib/git.py
@@ -2,25 +2,36 @@
import subprocess, os
+
class GitError(Exception):
pass
+
+
class ExecutionError(GitError):
def __init__(self, errcode):
self.error_code = errcode
+
+
class TimeoutExpired(GitError):
def __init__(self, errcode):
self.error_code = errcode
return "timeout"
+
def _check(process):
if process.returncode != 0:
raise ExecutionError(process.returncode)
+
def is_new_file(file):
- cmd = ['git', 'status', '-s', file ]
- process = subprocess.Popen(cmd,
- stdout=subprocess.PIPE, stderr=subprocess.STDOUT,
- close_fds=True, universal_newlines=True)
+ cmd = ["git", "status", "-s", file]
+ process = subprocess.Popen(
+ cmd,
+ stdout=subprocess.PIPE,
+ stderr=subprocess.STDOUT,
+ close_fds=True,
+ universal_newlines=True,
+ )
data = None
try:
data = process.communicate(timeout=1)
@@ -31,6 +42,6 @@ def is_new_file(file):
process.wait()
if process.returncode != 0:
return False
- if stdout.startswith('??'):
+ if stdout.startswith("??"):
return True
return False
diff --git a/playbooks/python/workflows/fstests/xunit_merge_all.py b/playbooks/python/workflows/fstests/xunit_merge_all.py
index 6af6e981..d2d2b64c 100755
--- a/playbooks/python/workflows/fstests/xunit_merge_all.py
+++ b/playbooks/python/workflows/fstests/xunit_merge_all.py
@@ -11,6 +11,7 @@ import os
import sys
from junitparser import JUnitXml, TestSuite
+
def get_test_suite(filename):
try:
ts = JUnitXml.fromfile(filename)
@@ -18,21 +19,31 @@ def get_test_suite(filename):
sys.exit("Couldn't open %s: %s" % (filename, e[1]))
if type(ts) != TestSuite:
- sys.exit('%s is not a xUnit report file' % filename)
+ sys.exit("%s is not a xUnit report file" % filename)
return ts
+
def merge_ts(old_ts, new_ts):
for tc in new_ts:
old_ts.add_testcase(tc)
old_ts.update_statistics()
return old_ts
+
def main():
- parser = argparse.ArgumentParser(description='Merge all xunit files into one')
- parser.add_argument('results', metavar='<directory with results>', type=str,
- help='directory with results file')
- parser.add_argument('outputfile', metavar='<output file>', type=str,
- help='The file to generate output to')
+ parser = argparse.ArgumentParser(description="Merge all xunit files into one")
+ parser.add_argument(
+ "results",
+ metavar="<directory with results>",
+ type=str,
+ help="directory with results file",
+ )
+ parser.add_argument(
+ "outputfile",
+ metavar="<output file>",
+ type=str,
+ help="The file to generate output to",
+ )
args = parser.parse_args()
all_xunit_ts = None
@@ -46,7 +57,7 @@ def main():
continue
if not os.path.isfile(f):
continue
- if not f.endswith('.xml'):
+ if not f.endswith(".xml"):
continue
sys.stdout.write("Processing %s ...\n" % f)
@@ -60,7 +71,11 @@ def main():
if all_xunit_ts:
all_xunit_ts.write(args.outputfile)
- sys.stdout.write("%s generated by merging all the above %d xunit files successfully\n" % (args.outputfile, num))
+ sys.stdout.write(
+ "%s generated by merging all the above %d xunit files successfully\n"
+ % (args.outputfile, num)
+ )
+
-if __name__ == '__main__':
+if __name__ == "__main__":
main()
diff --git a/playbooks/python/workflows/sysbench/sysbench-tps-compare.py b/playbooks/python/workflows/sysbench/sysbench-tps-compare.py
index 8faecdcf..dafeb119 100755
--- a/playbooks/python/workflows/sysbench/sysbench-tps-compare.py
+++ b/playbooks/python/workflows/sysbench/sysbench-tps-compare.py
@@ -8,18 +8,20 @@ import re
import argparse
from concurrent.futures import ThreadPoolExecutor
+
# Function to parse a line and extract time and TPS
def parse_line(line):
- match = re.search(r'\[\s*(\d+)s\s*\].*?tps:\s*([\d.]+)', line)
+ match = re.search(r"\[\s*(\d+)s\s*\].*?tps:\s*([\d.]+)", line)
if match:
time_in_seconds = int(match.group(1))
tps = float(match.group(2))
return time_in_seconds, tps
return None
+
# Function to read and parse sysbench output file
def read_sysbench_output(file_path):
- with open(file_path, 'r') as file:
+ with open(file_path, "r") as file:
lines = file.readlines()
with ThreadPoolExecutor() as executor:
@@ -27,23 +29,58 @@ def read_sysbench_output(file_path):
return [result for result in results if result is not None]
+
# Function to list available matplotlib themes
def list_themes():
print("Available matplotlib themes:")
for style in plt.style.available:
print(style)
+
# Main function
def main():
- parser = argparse.ArgumentParser(description='Compare sysbench outputs.')
- parser.add_argument('file1', type=str, nargs='?', default='sysbench_output_doublewrite.txt', help='First sysbench output file')
- parser.add_argument('file2', type=str, nargs='?', default='sysbench_output_nodoublewrite.txt', help='Second sysbench output file')
- parser.add_argument('--legend1', type=str, default='innodb_doublewrite=ON', help='Legend for the first file')
- parser.add_argument('--legend2', type=str, default='innodb_doublewrite=OFF', help='Legend for the second file')
- parser.add_argument('--theme', type=str, default='dark_background', help='Matplotlib theme to use')
- parser.add_argument('--list-themes', action='store_true', help='List available matplotlib themes')
- parser.add_argument('--output', type=str, default='a_vs_b.png', help='Path of file to save')
- parser.add_argument('--report-interval', type=int, default=1, help='Time interval in seconds for reporting')
+ parser = argparse.ArgumentParser(description="Compare sysbench outputs.")
+ parser.add_argument(
+ "file1",
+ type=str,
+ nargs="?",
+ default="sysbench_output_doublewrite.txt",
+ help="First sysbench output file",
+ )
+ parser.add_argument(
+ "file2",
+ type=str,
+ nargs="?",
+ default="sysbench_output_nodoublewrite.txt",
+ help="Second sysbench output file",
+ )
+ parser.add_argument(
+ "--legend1",
+ type=str,
+ default="innodb_doublewrite=ON",
+ help="Legend for the first file",
+ )
+ parser.add_argument(
+ "--legend2",
+ type=str,
+ default="innodb_doublewrite=OFF",
+ help="Legend for the second file",
+ )
+ parser.add_argument(
+ "--theme", type=str, default="dark_background", help="Matplotlib theme to use"
+ )
+ parser.add_argument(
+ "--list-themes", action="store_true", help="List available matplotlib themes"
+ )
+ parser.add_argument(
+ "--output", type=str, default="a_vs_b.png", help="Path of file to save"
+ )
+ parser.add_argument(
+ "--report-interval",
+ type=int,
+ default=1,
+ help="Time interval in seconds for reporting",
+ )
args = parser.parse_args()
@@ -62,37 +99,40 @@ def main():
tps_data_2 = [(time * args.report_interval, tps) for time, tps in tps_data_2]
# Determine the maximum time value to decide if we need to use hours or seconds
- max_time_in_seconds = max(max(tps_data_1, key=lambda x: x[0])[0], max(tps_data_2, key=lambda x: x[0])[0])
+ max_time_in_seconds = max(
+ max(tps_data_1, key=lambda x: x[0])[0], max(tps_data_2, key=lambda x: x[0])[0]
+ )
use_hours = max_time_in_seconds > 2 * 3600
# Convert times if necessary
if use_hours:
tps_data_1 = [(time / 3600, tps) for time, tps in tps_data_1]
tps_data_2 = [(time / 3600, tps) for time, tps in tps_data_2]
- time_label = 'Time (hours)'
+ time_label = "Time (hours)"
else:
- time_label = 'Time (seconds)'
+ time_label = "Time (seconds)"
# Create pandas DataFrames
- df1 = pd.DataFrame(tps_data_1, columns=[time_label, 'TPS'])
- df2 = pd.DataFrame(tps_data_2, columns=[time_label, 'TPS'])
+ df1 = pd.DataFrame(tps_data_1, columns=[time_label, "TPS"])
+ df2 = pd.DataFrame(tps_data_2, columns=[time_label, "TPS"])
# Plot the TPS values
plt.figure(figsize=(30, 12))
- plt.plot(df1[time_label], df1['TPS'], 'ro', markersize=2, label=args.legend1)
- plt.plot(df2[time_label], df2['TPS'], 'go', markersize=2, label=args.legend2)
+ plt.plot(df1[time_label], df1["TPS"], "ro", markersize=2, label=args.legend1)
+ plt.plot(df2[time_label], df2["TPS"], "go", markersize=2, label=args.legend2)
- plt.title('Transactions Per Second (TPS) Over Time')
+ plt.title("Transactions Per Second (TPS) Over Time")
plt.xlabel(time_label)
- plt.ylabel('TPS')
+ plt.ylabel("TPS")
plt.grid(True)
# Try plotting without this to zoom in
plt.ylim(0)
plt.legend()
plt.tight_layout()
plt.savefig(args.output)
- #plt.show()
+ # plt.show()
+
-if __name__ == '__main__':
+if __name__ == "__main__":
main()
diff --git a/playbooks/python/workflows/sysbench/sysbench-tps-plot.py b/playbooks/python/workflows/sysbench/sysbench-tps-plot.py
index 5505ced8..ee56b1f5 100755
--- a/playbooks/python/workflows/sysbench/sysbench-tps-plot.py
+++ b/playbooks/python/workflows/sysbench/sysbench-tps-plot.py
@@ -9,27 +9,38 @@ import re
import argparse
from concurrent.futures import ThreadPoolExecutor
+
# Function to parse a line and extract time and TPS from text
def parse_line(line):
- match = re.search(r'\[\s*(\d+)s\s*\].*?tps:\s*([\d.]+)', line)
+ match = re.search(r"\[\s*(\d+)s\s*\].*?tps:\s*([\d.]+)", line)
if match:
time_in_seconds = int(match.group(1))
tps = float(match.group(2))
return time_in_seconds, tps
return None
+
def main():
# Setup argument parser
- parser = argparse.ArgumentParser(description="Generate TPS plot from sysbench output")
- parser.add_argument('input_file', type=str, help="Path to the input file (text or JSON)")
- parser.add_argument('--output', type=str, default='tps_over_time.png', help="Output image file (default: tps_over_time.png)")
+ parser = argparse.ArgumentParser(
+ description="Generate TPS plot from sysbench output"
+ )
+ parser.add_argument(
+ "input_file", type=str, help="Path to the input file (text or JSON)"
+ )
+ parser.add_argument(
+ "--output",
+ type=str,
+ default="tps_over_time.png",
+ help="Output image file (default: tps_over_time.png)",
+ )
# Parse arguments
args = parser.parse_args()
# Read the input file
try:
- with open(args.input_file, 'r') as file:
+ with open(args.input_file, "r") as file:
# Read text lines and parse them concurrently
lines = file.readlines()
with ThreadPoolExecutor() as executor:
@@ -51,19 +62,19 @@ def main():
# Convert times if necessary
if use_hours:
tps_data = [(time / 3600, tps) for time, tps in tps_data]
- time_label = 'Time (hours)'
+ time_label = "Time (hours)"
else:
- time_label = 'Time (seconds)'
+ time_label = "Time (seconds)"
# Create a pandas DataFrame
- df = pd.DataFrame(tps_data, columns=[time_label, 'TPS'])
+ df = pd.DataFrame(tps_data, columns=[time_label, "TPS"])
# Plot the TPS values
plt.figure(figsize=(30, 12))
- plt.plot(df[time_label], df['TPS'], 'o', markersize=2)
- plt.title('Transactions Per Second (TPS) Over Time')
+ plt.plot(df[time_label], df["TPS"], "o", markersize=2)
+ plt.title("Transactions Per Second (TPS) Over Time")
plt.xlabel(time_label)
- plt.ylabel('TPS')
+ plt.ylabel("TPS")
plt.grid(True)
plt.ylim(0)
plt.tight_layout()
@@ -72,5 +83,6 @@ def main():
plt.savefig(args.output)
print(f"TPS plot saved to {args.output}")
-if __name__ == '__main__':
+
+if __name__ == "__main__":
main()
diff --git a/playbooks/python/workflows/sysbench/sysbench-tps-variance.py b/playbooks/python/workflows/sysbench/sysbench-tps-variance.py
index 96971458..aee083b9 100755
--- a/playbooks/python/workflows/sysbench/sysbench-tps-variance.py
+++ b/playbooks/python/workflows/sysbench/sysbench-tps-variance.py
@@ -9,15 +9,17 @@ import seaborn as sns
import argparse
from scipy.stats import norm
+
def extract_tps(filename):
tps_values = []
- with open(filename, 'r') as file:
+ with open(filename, "r") as file:
for line in file:
- match = re.search(r'tps: (\d+\.\d+)', line)
+ match = re.search(r"tps: (\d+\.\d+)", line)
if match:
tps_values.append(float(match.group(1)))
return tps_values
+
def analyze_tps(tps_values):
mean_tps = np.mean(tps_values)
median_tps = np.median(tps_values)
@@ -25,28 +27,49 @@ def analyze_tps(tps_values):
variance_tps = np.var(tps_values)
return mean_tps, median_tps, std_tps, variance_tps
+
def print_statistics(label, tps_values):
mean_tps, median_tps, std_tps, variance_tps = analyze_tps(tps_values)
- print(f'{label} Statistics:')
- print(f'Mean TPS: {mean_tps:.2f}')
- print(f'Median TPS: {median_tps:.2f}')
- print(f'Standard Deviation of TPS: {std_tps:.2f}')
- print(f'Variance of TPS: {variance_tps:.2f}\n')
+ print(f"{label} Statistics:")
+ print(f"Mean TPS: {mean_tps:.2f}")
+ print(f"Median TPS: {median_tps:.2f}")
+ print(f"Standard Deviation of TPS: {std_tps:.2f}")
+ print(f"Variance of TPS: {variance_tps:.2f}\n")
+
def plot_histograms(tps_values1, tps_values2, legend1, legend2, color1, color2, outdir):
plt.figure(figsize=(20, 12))
- bins = np.linspace(min(min(tps_values1), min(tps_values2)), max(max(tps_values1), max(tps_values2)), 30)
- plt.hist(tps_values1, bins=bins, alpha=0.5, color=color1, edgecolor='black', label=legend1)
+ bins = np.linspace(
+ min(min(tps_values1), min(tps_values2)),
+ max(max(tps_values1), max(tps_values2)),
+ 30,
+ )
+ plt.hist(
+ tps_values1,
+ bins=bins,
+ alpha=0.5,
+ color=color1,
+ edgecolor="black",
+ label=legend1,
+ )
if tps_values2:
- plt.hist(tps_values2, bins=bins, alpha=0.5, color=color2, edgecolor='black', label=legend2)
- plt.title('Distribution of TPS Values')
- plt.xlabel('Transactions Per Second (TPS)')
- plt.ylabel('Frequency')
- plt.legend(loc='best')
+ plt.hist(
+ tps_values2,
+ bins=bins,
+ alpha=0.5,
+ color=color2,
+ edgecolor="black",
+ label=legend2,
+ )
+ plt.title("Distribution of TPS Values")
+ plt.xlabel("Transactions Per Second (TPS)")
+ plt.ylabel("Frequency")
+ plt.legend(loc="best")
plt.grid(True)
- plt.savefig(outdir + 'histogram.png')
+ plt.savefig(outdir + "histogram.png")
plt.show()
+
def plot_box_plots(tps_values1, tps_values2, legend1, legend2, color1, color2, outdir):
data = []
labels = []
@@ -58,112 +81,180 @@ def plot_box_plots(tps_values1, tps_values2, legend1, legend2, color1, color2, o
plt.figure(figsize=(20, 12))
box = plt.boxplot(data, labels=labels, patch_artist=True)
colors = [color1, color2]
- for patch, color in zip(box['boxes'], colors):
+ for patch, color in zip(box["boxes"], colors):
patch.set_facecolor(color)
- plt.title('Box Plot of TPS Values')
- plt.ylabel('Transactions Per Second (TPS)')
+ plt.title("Box Plot of TPS Values")
+ plt.ylabel("Transactions Per Second (TPS)")
plt.grid(True)
- plt.savefig(outdir + 'box_plot.png')
+ plt.savefig(outdir + "box_plot.png")
plt.show()
-def plot_density_plots(tps_values1, tps_values2, legend1, legend2, color1, color2, outdir):
+
+def plot_density_plots(
+ tps_values1, tps_values2, legend1, legend2, color1, color2, outdir
+):
plt.figure(figsize=(20, 12))
sns.kdeplot(tps_values1, fill=True, label=legend1, color=color1)
if tps_values2:
sns.kdeplot(tps_values2, fill=True, label=legend2, color=color2)
- plt.title('Density Plot of TPS Values')
- plt.xlabel('Transactions Per Second (TPS)')
- plt.ylabel('Density')
- plt.legend(loc='best')
+ plt.title("Density Plot of TPS Values")
+ plt.xlabel("Transactions Per Second (TPS)")
+ plt.ylabel("Density")
+ plt.legend(loc="best")
plt.grid(True)
- plt.savefig(outdir + 'density_plot.png')
+ plt.savefig(outdir + "density_plot.png")
plt.show()
-def plot_combined_hist_density(tps_values1, tps_values2, legend1, legend2, color1, color2, outdir):
+
+def plot_combined_hist_density(
+ tps_values1, tps_values2, legend1, legend2, color1, color2, outdir
+):
plt.figure(figsize=(20, 12))
- bins = np.linspace(min(min(tps_values1), min(tps_values2)), max(max(tps_values1), max(tps_values2)), 30)
- plt.hist(tps_values1, bins=bins, alpha=0.3, color=color1, edgecolor='black', label=f'Histogram {legend1}', density=True)
+ bins = np.linspace(
+ min(min(tps_values1), min(tps_values2)),
+ max(max(tps_values1), max(tps_values2)),
+ 30,
+ )
+ plt.hist(
+ tps_values1,
+ bins=bins,
+ alpha=0.3,
+ color=color1,
+ edgecolor="black",
+ label=f"Histogram {legend1}",
+ density=True,
+ )
if tps_values2:
- plt.hist(tps_values2, bins=bins, alpha=0.3, color=color2, edgecolor='black', label=f'Histogram {legend2}', density=True)
- sns.kdeplot(tps_values1, fill=False, label=f'Density {legend1}', color=color1)
+ plt.hist(
+ tps_values2,
+ bins=bins,
+ alpha=0.3,
+ color=color2,
+ edgecolor="black",
+ label=f"Histogram {legend2}",
+ density=True,
+ )
+ sns.kdeplot(tps_values1, fill=False, label=f"Density {legend1}", color=color1)
if tps_values2:
- sns.kdeplot(tps_values2, fill=False, label=f'Density {legend2}', color=color2)
+ sns.kdeplot(tps_values2, fill=False, label=f"Density {legend2}", color=color2)
mean1, std1 = np.mean(tps_values1), np.std(tps_values1)
ax2 = plt.gca().twinx()
- ax2.set_ylabel('Density')
- ax2.axvline(mean1, color=color1, linestyle='dotted', linewidth=2)
- ax2.axvline(mean1 - std1, color=color1, linestyle='dotted', linewidth=1)
- ax2.axvline(mean1 + std1, color=color1, linestyle='dotted', linewidth=1)
+ ax2.set_ylabel("Density")
+ ax2.axvline(mean1, color=color1, linestyle="dotted", linewidth=2)
+ ax2.axvline(mean1 - std1, color=color1, linestyle="dotted", linewidth=1)
+ ax2.axvline(mean1 + std1, color=color1, linestyle="dotted", linewidth=1)
if tps_values2:
mean2, std2 = np.mean(tps_values2), np.std(tps_values2)
- ax2.axvline(mean2, color=color2, linestyle='dotted', linewidth=2)
- ax2.axvline(mean2 - std2, color=color2, linestyle='dotted', linewidth=1)
- ax2.axvline(mean2 + std2, color=color2, linestyle='dotted', linewidth=1)
-
- plt.title('Combined Histogram and Density Plot of TPS Values')
- plt.xlabel('Transactions Per Second (TPS)')
- plt.ylabel('Frequency/Density')
- plt.legend(loc='best')
+ ax2.axvline(mean2, color=color2, linestyle="dotted", linewidth=2)
+ ax2.axvline(mean2 - std2, color=color2, linestyle="dotted", linewidth=1)
+ ax2.axvline(mean2 + std2, color=color2, linestyle="dotted", linewidth=1)
+
+ plt.title("Combined Histogram and Density Plot of TPS Values")
+ plt.xlabel("Transactions Per Second (TPS)")
+ plt.ylabel("Frequency/Density")
+ plt.legend(loc="best")
plt.grid(True)
- plt.savefig(outdir + 'combined_hist_density.png')
+ plt.savefig(outdir + "combined_hist_density.png")
plt.show()
+
def plot_bell_curve(tps_values1, tps_values2, legend1, legend2, color1, color2, outdir):
plt.figure(figsize=(20, 12))
mean1, std1 = np.mean(tps_values1), np.std(tps_values1)
- x1 = np.linspace(mean1 - 3*std1, mean1 + 3*std1, 100)
- plt.plot(x1, norm.pdf(x1, mean1, std1) * 100, label=f'Bell Curve {legend1}', color=color1) # Multiplying by 100 for percentage
+ x1 = np.linspace(mean1 - 3 * std1, mean1 + 3 * std1, 100)
+ plt.plot(
+ x1, norm.pdf(x1, mean1, std1) * 100, label=f"Bell Curve {legend1}", color=color1
+ ) # Multiplying by 100 for percentage
if tps_values2:
mean2, std2 = np.mean(tps_values2), np.std(tps_values2)
- x2 = np.linspace(mean2 - 3*std2, mean2 + 3*std2, 100)
- plt.plot(x2, norm.pdf(x2, mean2, std2) * 100, label=f'Bell Curve {legend2}', color=color2) # Multiplying by 100 for percentage
-
- plt.title('Bell Curve (Normal Distribution) of TPS Values')
- plt.xlabel('Transactions Per Second (TPS)')
- plt.ylabel('Probability Density (%)')
- plt.legend(loc='best')
+ x2 = np.linspace(mean2 - 3 * std2, mean2 + 3 * std2, 100)
+ plt.plot(
+ x2,
+ norm.pdf(x2, mean2, std2) * 100,
+ label=f"Bell Curve {legend2}",
+ color=color2,
+ ) # Multiplying by 100 for percentage
+
+ plt.title("Bell Curve (Normal Distribution) of TPS Values")
+ plt.xlabel("Transactions Per Second (TPS)")
+ plt.ylabel("Probability Density (%)")
+ plt.legend(loc="best")
plt.grid(True)
- plt.savefig(outdir + 'bell_curve.png')
+ plt.savefig(outdir + "bell_curve.png")
plt.show()
-def plot_combined_hist_bell_curve(tps_values1, tps_values2, legend1, legend2, color1, color2, outdir):
+
+def plot_combined_hist_bell_curve(
+ tps_values1, tps_values2, legend1, legend2, color1, color2, outdir
+):
fig, ax1 = plt.subplots(figsize=(20, 12))
- bins = np.linspace(min(min(tps_values1), min(tps_values2)), max(max(tps_values1), max(tps_values2)), 30)
- ax1.hist(tps_values1, bins=bins, alpha=0.5, color=color1, edgecolor='black', label=legend1)
+ bins = np.linspace(
+ min(min(tps_values1), min(tps_values2)),
+ max(max(tps_values1), max(tps_values2)),
+ 30,
+ )
+ ax1.hist(
+ tps_values1,
+ bins=bins,
+ alpha=0.5,
+ color=color1,
+ edgecolor="black",
+ label=legend1,
+ )
if tps_values2:
- ax1.hist(tps_values2, bins=bins, alpha=0.5, color=color2, edgecolor='black', label=legend2)
-
- ax1.set_xlabel('Transactions Per Second (TPS)')
- ax1.set_ylabel('Frequency')
- ax1.legend(loc='upper left')
+ ax1.hist(
+ tps_values2,
+ bins=bins,
+ alpha=0.5,
+ color=color2,
+ edgecolor="black",
+ label=legend2,
+ )
+
+ ax1.set_xlabel("Transactions Per Second (TPS)")
+ ax1.set_ylabel("Frequency")
+ ax1.legend(loc="upper left")
ax1.grid(True)
ax2 = ax1.twinx()
mean1, std1 = np.mean(tps_values1), np.std(tps_values1)
- x1 = np.linspace(mean1 - 3*std1, mean1 + 3*std1, 100)
- ax2.plot(x1, norm.pdf(x1, mean1, std1) * 100, label=f'Bell Curve {legend1}', color=color1, linestyle='dashed')
- ax2.axvline(mean1, color=color1, linestyle='dotted', linewidth=2)
- ax2.axvline(mean1 - std1, color=color1, linestyle='dotted', linewidth=1)
- ax2.axvline(mean1 + std1, color=color1, linestyle='dotted', linewidth=1)
+ x1 = np.linspace(mean1 - 3 * std1, mean1 + 3 * std1, 100)
+ ax2.plot(
+ x1,
+ norm.pdf(x1, mean1, std1) * 100,
+ label=f"Bell Curve {legend1}",
+ color=color1,
+ linestyle="dashed",
+ )
+ ax2.axvline(mean1, color=color1, linestyle="dotted", linewidth=2)
+ ax2.axvline(mean1 - std1, color=color1, linestyle="dotted", linewidth=1)
+ ax2.axvline(mean1 + std1, color=color1, linestyle="dotted", linewidth=1)
if tps_values2:
mean2, std2 = np.mean(tps_values2), np.std(tps_values2)
- x2 = np.linspace(mean2 - 3*std2, mean2 + 3*std2, 100)
- ax2.plot(x2, norm.pdf(x2, mean2, std2) * 100, label=f'Bell Curve {legend2}', color=color2, linestyle='dashed')
- ax2.axvline(mean2, color=color2, linestyle='dotted', linewidth=2)
- ax2.axvline(mean2 - std2, color=color2, linestyle='dotted', linewidth=1)
- ax2.axvline(mean2 + std2, color=color2, linestyle='dotted', linewidth=1)
-
- ax2.set_ylabel('Probability Density (%)')
- ax2.legend(loc='upper center')
-
- plt.title('Combined Histogram and Bell Curve of TPS Values')
- plt.savefig(outdir + 'combined_hist_bell_curve.png')
+ x2 = np.linspace(mean2 - 3 * std2, mean2 + 3 * std2, 100)
+ ax2.plot(
+ x2,
+ norm.pdf(x2, mean2, std2) * 100,
+ label=f"Bell Curve {legend2}",
+ color=color2,
+ linestyle="dashed",
+ )
+ ax2.axvline(mean2, color=color2, linestyle="dotted", linewidth=2)
+ ax2.axvline(mean2 - std2, color=color2, linestyle="dotted", linewidth=1)
+ ax2.axvline(mean2 + std2, color=color2, linestyle="dotted", linewidth=1)
+
+ ax2.set_ylabel("Probability Density (%)")
+ ax2.legend(loc="upper center")
+
+ plt.title("Combined Histogram and Bell Curve of TPS Values")
+ plt.savefig(outdir + "combined_hist_bell_curve.png")
plt.show()
+
def plot_variance_bars(variance1, variance2, legend1, legend2, color1, color2, outdir):
fig, ax1 = plt.subplots(figsize=(20, 12))
@@ -173,24 +264,39 @@ def plot_variance_bars(variance1, variance2, legend1, legend2, color1, color2, o
bars = plt.bar(labels, variances, color=colors)
for bar, variance in zip(bars, variances):
- plt.text(bar.get_x() + bar.get_width() / 2, bar.get_height(), f'{variance:.2f}', ha='center', va='bottom')
+ plt.text(
+ bar.get_x() + bar.get_width() / 2,
+ bar.get_height(),
+ f"{variance:.2f}",
+ ha="center",
+ va="bottom",
+ )
# Calculate the factor by which the larger variance is greater than the smaller variance
if variance1 != 0 and variance2 != 0:
factor = max(variance1, variance2) / min(variance1, variance2)
- factor_text = f'Variance Factor: {factor:.2f}'
- plt.text(1, max(variances) * 1.05, factor_text, ha='center', va='bottom', fontsize=12, color='white')
-
- plt.title('Variance of TPS Values')
- plt.ylabel('Variance')
+ factor_text = f"Variance Factor: {factor:.2f}"
+ plt.text(
+ 1,
+ max(variances) * 1.05,
+ factor_text,
+ ha="center",
+ va="bottom",
+ fontsize=12,
+ color="white",
+ )
+
+ plt.title("Variance of TPS Values")
+ plt.ylabel("Variance")
# Add lollipop marker
for bar, variance in zip(bars, variances):
- plt.plot(bar.get_x() + bar.get_width() / 2, variance, 'o', color='black')
+ plt.plot(bar.get_x() + bar.get_width() / 2, variance, "o", color="black")
- plt.savefig(outdir + 'variance_bar.png')
+ plt.savefig(outdir + "variance_bar.png")
plt.show()
+
def plot_outliers(tps_values1, tps_values2, legend1, legend2, color1, color2, outdir):
data = [tps_values1]
labels = [legend1]
@@ -202,41 +308,69 @@ def plot_outliers(tps_values1, tps_values2, legend1, legend2, color1, color2, ou
colors.append(color2)
fig, ax = plt.subplots(figsize=(20, 12))
- box = ax.boxplot(data, labels=labels, patch_artist=True, showfliers=True,
- whiskerprops=dict(color='white', linewidth=2),
- capprops=dict(color='white', linewidth=2),
- medianprops=dict(color='yellow', linewidth=2))
+ box = ax.boxplot(
+ data,
+ labels=labels,
+ patch_artist=True,
+ showfliers=True,
+ whiskerprops=dict(color="white", linewidth=2),
+ capprops=dict(color="white", linewidth=2),
+ medianprops=dict(color="yellow", linewidth=2),
+ )
# Color the boxes
- for patch, color in zip(box['boxes'], colors):
+ for patch, color in zip(box["boxes"], colors):
patch.set_facecolor(color)
# Scatter plot for the actual points
for i, (d, color) in enumerate(zip(data, colors)):
y = d
# Adding jitter to the x-axis for better visibility
- x = np.random.normal(i + 1, 0.04, size=len(y)) # Adding some jitter to the x-axis
- ax.scatter(x, y, alpha=0.6, color=color, edgecolor='black')
+ x = np.random.normal(
+ i + 1, 0.04, size=len(y)
+ ) # Adding some jitter to the x-axis
+ ax.scatter(x, y, alpha=0.6, color=color, edgecolor="black")
- plt.title('Outliers in TPS Values')
- plt.ylabel('Transactions Per Second (TPS)')
+ plt.title("Outliers in TPS Values")
+ plt.ylabel("Transactions Per Second (TPS)")
plt.grid(True)
- plt.savefig(outdir + 'outliers_plot.png')
+ plt.savefig(outdir + "outliers_plot.png")
plt.show()
+
def main():
- parser = argparse.ArgumentParser(description='Analyze and compare TPS values from sysbench output files.')
- parser.add_argument('file1', help='First TPS file')
- parser.add_argument('--legend1', type=str, default='innodb_doublewrite=ON', help='Legend for the first file')
- parser.add_argument('file2', nargs='?', default=None, help='Second TPS file (optional)')
- parser.add_argument('--legend2', type=str, default='innodb_doublewrite=OFF', help='Legend for the second file')
- parser.add_argument('--dir', type=str, default='./', help='Path to place images')
- parser.add_argument('--color1', default='cyan', help='Color for the first dataset (default: cyan)')
- parser.add_argument('--color2', default='orange', help='Color for the second dataset (default: orange)')
+ parser = argparse.ArgumentParser(
+ description="Analyze and compare TPS values from sysbench output files."
+ )
+ parser.add_argument("file1", help="First TPS file")
+ parser.add_argument(
+ "--legend1",
+ type=str,
+ default="innodb_doublewrite=ON",
+ help="Legend for the first file",
+ )
+ parser.add_argument(
+ "file2", nargs="?", default=None, help="Second TPS file (optional)"
+ )
+ parser.add_argument(
+ "--legend2",
+ type=str,
+ default="innodb_doublewrite=OFF",
+ help="Legend for the second file",
+ )
+ parser.add_argument("--dir", type=str, default="./", help="Path to place images")
+ parser.add_argument(
+ "--color1", default="cyan", help="Color for the first dataset (default: cyan)"
+ )
+ parser.add_argument(
+ "--color2",
+ default="orange",
+ help="Color for the second dataset (default: orange)",
+ )
args = parser.parse_args()
- plt.style.use('dark_background') # Set the dark theme
+ plt.style.use("dark_background") # Set the dark theme
tps_values1 = extract_tps(args.file1)
tps_values2 = extract_tps(args.file2) if args.file2 else None
@@ -245,22 +379,89 @@ def main():
if tps_values2:
print_statistics(args.legend2, tps_values2)
- plot_histograms(tps_values1, tps_values2, args.legend1, args.legend2 if args.legend2 else '', args.color1, args.color2, args.dir)
- plot_box_plots(tps_values1, tps_values2, args.legend1, args.legend2 if args.legend2 else '', args.color1, args.color2, args.dir)
- plot_density_plots(tps_values1, tps_values2, args.legend1, args.legend2 if args.legend2 else '', args.color1, args.color2, args.dir)
- plot_combined_hist_density(tps_values1, tps_values2, args.legend1, args.legend2 if args.legend2 else '', args.color1, args.color2, args.dir)
- plot_bell_curve(tps_values1, tps_values2, args.legend1, args.legend2 if args.legend2 else '', args.color1, args.color2, args.dir)
- plot_combined_hist_bell_curve(tps_values1, tps_values2, args.legend1, args.legend2 if args.legend2 else '', args.color1, args.color2, args.dir)
+ plot_histograms(
+ tps_values1,
+ tps_values2,
+ args.legend1,
+ args.legend2 if args.legend2 else "",
+ args.color1,
+ args.color2,
+ args.dir,
+ )
+ plot_box_plots(
+ tps_values1,
+ tps_values2,
+ args.legend1,
+ args.legend2 if args.legend2 else "",
+ args.color1,
+ args.color2,
+ args.dir,
+ )
+ plot_density_plots(
+ tps_values1,
+ tps_values2,
+ args.legend1,
+ args.legend2 if args.legend2 else "",
+ args.color1,
+ args.color2,
+ args.dir,
+ )
+ plot_combined_hist_density(
+ tps_values1,
+ tps_values2,
+ args.legend1,
+ args.legend2 if args.legend2 else "",
+ args.color1,
+ args.color2,
+ args.dir,
+ )
+ plot_bell_curve(
+ tps_values1,
+ tps_values2,
+ args.legend1,
+ args.legend2 if args.legend2 else "",
+ args.color1,
+ args.color2,
+ args.dir,
+ )
+ plot_combined_hist_bell_curve(
+ tps_values1,
+ tps_values2,
+ args.legend1,
+ args.legend2 if args.legend2 else "",
+ args.color1,
+ args.color2,
+ args.dir,
+ )
# Plot variance bars
_, _, _, variance1 = analyze_tps(tps_values1)
if tps_values2:
_, _, _, variance2 = analyze_tps(tps_values2)
- plot_variance_bars(variance1, variance2, args.legend1, args.legend2, args.color1, args.color2, args.dir)
+ plot_variance_bars(
+ variance1,
+ variance2,
+ args.legend1,
+ args.legend2,
+ args.color1,
+ args.color2,
+ args.dir,
+ )
else:
- plot_variance_bars(variance1, 0, args.legend1, '', args.color1, 'black', args.dir) # Use black for the second bar if there's only one dataset
-
- plot_outliers(tps_values1, tps_values2, args.legend1, args.legend2 if args.legend2 else '', args.color1, args.color2, args.dir)
-
-if __name__ == '__main__':
+ plot_variance_bars(
+ variance1, 0, args.legend1, "", args.color1, "black", args.dir
+ ) # Use black for the second bar if there's only one dataset
+
+ plot_outliers(
+ tps_values1,
+ tps_values2,
+ args.legend1,
+ args.legend2 if args.legend2 else "",
+ args.color1,
+ args.color2,
+ args.dir,
+ )
+
+
+if __name__ == "__main__":
main()
diff --git a/playbooks/roles/gen_nodes/python/gen_pcie_passthrough_guestfs_xml.py b/playbooks/roles/gen_nodes/python/gen_pcie_passthrough_guestfs_xml.py
index a5abdf0f..076fe583 100755
--- a/playbooks/roles/gen_nodes/python/gen_pcie_passthrough_guestfs_xml.py
+++ b/playbooks/roles/gen_nodes/python/gen_pcie_passthrough_guestfs_xml.py
@@ -24,11 +24,12 @@ pcie_hotplug_template = """<!-- PCIE passthrough device -->
<!-- End of PCIE passthrough device -->
"""
+
def main():
- topdir = os.environ.get('TOPDIR', '.')
+ topdir = os.environ.get("TOPDIR", ".")
# load extra_vars
- with open(f'{topdir}/extra_vars.yaml') as stream:
+ with open(f"{topdir}/extra_vars.yaml") as stream:
extra_vars = yaml.safe_load(stream)
yaml_nodes_file = f'{topdir}/{extra_vars["kdevops_nodes"]}'
@@ -38,36 +39,48 @@ def main():
nodes = yaml.safe_load(stream)
# add pcie devices
- for node in nodes['guestfs_nodes']:
- name = node['name']
- pcipassthrough = node.get('pcipassthrough')
+ for node in nodes["guestfs_nodes"]:
+ name = node["name"]
+ pcipassthrough = node.get("pcipassthrough")
if not pcipassthrough:
continue
for dev_key_name in pcipassthrough:
dev = pcipassthrough.get(dev_key_name)
dev_keys = list(dev.keys())
- if 'domain' not in dev_keys or 'bus' not in dev_keys or 'slot' not in dev_keys or 'function' not in dev_keys:
- raise Exception(f"Missing pcie attributes for device %s in %s" %
- (dev_key_name, yaml_nodes_file))
- domain = hex(dev.get('domain'))
- bus = hex(dev.get('bus'))
- slot = hex(dev.get('slot'))
- function = hex(dev.get('function'))
+ if (
+ "domain" not in dev_keys
+ or "bus" not in dev_keys
+ or "slot" not in dev_keys
+ or "function" not in dev_keys
+ ):
+ raise Exception(
+ f"Missing pcie attributes for device %s in %s"
+ % (dev_key_name, yaml_nodes_file)
+ )
+ domain = hex(dev.get("domain"))
+ bus = hex(dev.get("bus"))
+ slot = hex(dev.get("slot"))
+ function = hex(dev.get("function"))
- pcie_xml = f"{extra_vars['guestfs_path']}/{name}/pcie_passthrough_" + dev_key_name + ".xml"
+ pcie_xml = (
+ f"{extra_vars['guestfs_path']}/{name}/pcie_passthrough_"
+ + dev_key_name
+ + ".xml"
+ )
if os.path.exists(pcie_xml):
os.remove(pcie_xml)
- device_xml = open(pcie_xml, 'w')
+ device_xml = open(pcie_xml, "w")
context = {
- "domain" : domain,
- "bus" : bus,
- "slot" : slot,
- "function" : function,
+ "domain": domain,
+ "bus": bus,
+ "slot": slot,
+ "function": function,
}
device_xml.write(pcie_hotplug_template.format(**context))
device_xml.close()
+
if __name__ == "__main__":
main()
diff --git a/playbooks/roles/linux-mirror/python/gen-mirror-files.py b/playbooks/roles/linux-mirror/python/gen-mirror-files.py
index 65fc909d..c43b34a8 100755
--- a/playbooks/roles/linux-mirror/python/gen-mirror-files.py
+++ b/playbooks/roles/linux-mirror/python/gen-mirror-files.py
@@ -13,9 +13,9 @@ import time
import os
from pathlib import Path
-topdir = os.environ.get('TOPDIR', '.')
+topdir = os.environ.get("TOPDIR", ".")
yaml_dir = topdir + "/playbooks/roles/linux-mirror/linux-mirror-systemd/"
-default_mirrors_yaml = yaml_dir + 'mirrors.yaml'
+default_mirrors_yaml = yaml_dir + "mirrors.yaml"
service_template = """[Unit]
Description={short_name} mirror [{target}]
@@ -44,19 +44,37 @@ OnUnitInactiveSec={refresh}
WantedBy=default.target
"""
+
def main():
- parser = argparse.ArgumentParser(description='gen-mirror-files')
- parser.add_argument('--yaml-mirror', metavar='<yaml_mirror>', type=str,
- default=default_mirrors_yaml,
- help='The yaml mirror input file.')
- parser.add_argument('--verbose', const=True, default=False, action="store_const",
- help='Be verbose on otput.')
- parser.add_argument('--refresh', metavar='<refresh>', type=str,
- default='360m',
- help='How often to update the git tree.')
- parser.add_argument('--refresh-on-boot', metavar='<refresh>', type=str,
- default='10m',
- help='How long to wait on boot to update the git tree.')
+ parser = argparse.ArgumentParser(description="gen-mirror-files")
+ parser.add_argument(
+ "--yaml-mirror",
+ metavar="<yaml_mirror>",
+ type=str,
+ default=default_mirrors_yaml,
+ help="The yaml mirror input file.",
+ )
+ parser.add_argument(
+ "--verbose",
+ const=True,
+ default=False,
+ action="store_const",
+ help="Be verbose on otput.",
+ )
+ parser.add_argument(
+ "--refresh",
+ metavar="<refresh>",
+ type=str,
+ default="360m",
+ help="How often to update the git tree.",
+ )
+ parser.add_argument(
+ "--refresh-on-boot",
+ metavar="<refresh>",
+ type=str,
+ default="10m",
+ help="How long to wait on boot to update the git tree.",
+ )
args = parser.parse_args()
if not os.path.isfile(args.yaml_mirror):
@@ -64,86 +82,95 @@ def main():
sys.exit(1)
# load the yaml input file
- with open(f'{args.yaml_mirror}') as stream:
+ with open(f"{args.yaml_mirror}") as stream:
yaml_vars = yaml.safe_load(stream)
- if yaml_vars.get('mirrors') is None:
- raise Exception(f"Missing mirrors descriptions on %s" %
- (args.yaml_mirror))
+ if yaml_vars.get("mirrors") is None:
+ raise Exception(f"Missing mirrors descriptions on %s" % (args.yaml_mirror))
- if (args.verbose):
+ if args.verbose:
sys.stdout.write("Yaml mirror input: %s\n\n" % args.yaml_mirror)
total = 0
- for mirror in yaml_vars['mirrors']:
+ for mirror in yaml_vars["mirrors"]:
total = total + 1
- if mirror.get('short_name') is None:
- raise Exception(f"Missing required short_name on mirror item #%d on file: %s" % (total, args.yaml_mirror))
- if mirror.get('url') is None:
- raise Exception(f"Missing required url on mirror item #%d on file: %s" % (total, args.yaml_mirror))
- if mirror.get('target') is None:
- raise Exception(f"Missing required target on mirror item #%d on file: %s" % (total, args.yaml_mirror))
-
- short_name = mirror['short_name'].replace("/", "-")
- url = mirror['url']
- target = mirror['target']
+ if mirror.get("short_name") is None:
+ raise Exception(
+ f"Missing required short_name on mirror item #%d on file: %s"
+ % (total, args.yaml_mirror)
+ )
+ if mirror.get("url") is None:
+ raise Exception(
+ f"Missing required url on mirror item #%d on file: %s"
+ % (total, args.yaml_mirror)
+ )
+ if mirror.get("target") is None:
+ raise Exception(
+ f"Missing required target on mirror item #%d on file: %s"
+ % (total, args.yaml_mirror)
+ )
+
+ short_name = mirror["short_name"].replace("/", "-")
+ url = mirror["url"]
+ target = mirror["target"]
service_file = f"{yaml_dir}" + short_name + "-mirror" + ".service"
timer_file = f"{yaml_dir}" + short_name + "-mirror" + ".timer"
refresh = args.refresh
- if mirror.get('refresh'):
- refresh = mirror.get('refresh')
+ if mirror.get("refresh"):
+ refresh = mirror.get("refresh")
refresh_on_boot = args.refresh_on_boot
- if mirror.get('refresh_on_boot'):
- refresh = mirror.get('refresh_on_boot')
+ if mirror.get("refresh_on_boot"):
+ refresh = mirror.get("refresh_on_boot")
- if (args.verbose):
+ if args.verbose:
sys.stdout.write("Mirror #%d\n" % total)
- sys.stdout.write("\tshort_name: %s\n" % (mirror['short_name']))
- sys.stdout.write("\turl: %s\n" % (mirror['short_name']))
- sys.stdout.write("\ttarget: %s\n" % (mirror['short_name']))
+ sys.stdout.write("\tshort_name: %s\n" % (mirror["short_name"]))
+ sys.stdout.write("\turl: %s\n" % (mirror["short_name"]))
+ sys.stdout.write("\ttarget: %s\n" % (mirror["short_name"]))
sys.stdout.write("\tservice: %s\n" % (service_file))
sys.stdout.write("\ttimer: %s\n" % (timer_file))
sys.stdout.write("\trefresh: %s\n" % (refresh))
sys.stdout.write("\trefresh_on_boot: %s\n" % (refresh_on_boot))
if os.path.exists(service_file):
- if (args.verbose):
+ if args.verbose:
sys.stdout.write("\toverwrite_service: True\n")
os.remove(service_file)
else:
- if (args.verbose):
+ if args.verbose:
sys.stdout.write("\toverwrite_service: False\n")
- output_service = open(service_file, 'w')
+ output_service = open(service_file, "w")
context = {
- "short_name" : short_name,
- "url" : url,
- "target" : target,
+ "short_name": short_name,
+ "url": url,
+ "target": target,
}
output_service.write(service_template.format(**context))
output_service.close()
if os.path.exists(timer_file):
- if (args.verbose):
+ if args.verbose:
sys.stdout.write("\toverwrite_timer: True\n")
os.remove(timer_file)
else:
- if (args.verbose):
+ if args.verbose:
sys.stdout.write("\toverwrite_timer: False\n")
- output_timer = open(timer_file, 'w')
+ output_timer = open(timer_file, "w")
context = {
- "short_name" : short_name,
- "url" : url,
- "target" : target,
- "refresh" : refresh,
- "refresh_on_boot" : refresh_on_boot,
+ "short_name": short_name,
+ "url": url,
+ "target": target,
+ "refresh": refresh,
+ "refresh_on_boot": refresh_on_boot,
}
output_timer.write(timer_template.format(**context))
output_timer.close()
+
if __name__ == "__main__":
main()
diff --git a/playbooks/roles/linux-mirror/python/start-mirroring.py b/playbooks/roles/linux-mirror/python/start-mirroring.py
index 4e6b9ec2..03ede449 100755
--- a/playbooks/roles/linux-mirror/python/start-mirroring.py
+++ b/playbooks/roles/linux-mirror/python/start-mirroring.py
@@ -14,24 +14,25 @@ import os
from pathlib import Path
import subprocess
-topdir = os.environ.get('TOPDIR', '.')
+topdir = os.environ.get("TOPDIR", ".")
yaml_dir = topdir + "/playbooks/roles/linux-mirror/linux-mirror-systemd/"
-default_mirrors_yaml = yaml_dir + 'mirrors.yaml'
+default_mirrors_yaml = yaml_dir + "mirrors.yaml"
+
+mirror_path = "/mirror/"
-mirror_path = '/mirror/'
def mirror_entry(mirror, args):
- short_name = mirror['short_name']
- url = mirror['url']
- target = mirror['target']
+ short_name = mirror["short_name"]
+ url = mirror["url"]
+ target = mirror["target"]
reference = None
reference_args = []
- if mirror.get('reference'):
- reference = mirror_path + mirror.get('reference')
- reference_args = [ '--reference', reference ]
+ if mirror.get("reference"):
+ reference = mirror_path + mirror.get("reference")
+ reference_args = ["--reference", reference]
- if (args.verbose):
+ if args.verbose:
sys.stdout.write("\tshort_name: %s\n" % (short_name))
sys.stdout.write("\turl: %s\n" % (url))
sys.stdout.write("\ttarget: %s\n" % (url))
@@ -40,28 +41,31 @@ def mirror_entry(mirror, args):
else:
sys.stdout.write("\treference: %s\n" % (reference))
cmd = [
- 'git',
- '-C',
- mirror_path,
- 'clone',
- '--verbose',
- '--progress',
- '--mirror',
- url,
- target ]
+ "git",
+ "-C",
+ mirror_path,
+ "clone",
+ "--verbose",
+ "--progress",
+ "--mirror",
+ url,
+ target,
+ ]
cmd = cmd + reference_args
mirror_target = mirror_path + target
if os.path.isdir(mirror_target):
return
sys.stdout.write("Mirroring: %s onto %s\n" % (short_name, mirror_target))
- if (args.verbose):
+ if args.verbose:
sys.stdout.write("%s\n" % (cmd))
sys.stdout.write("%s\n" % (" ".join(cmd)))
- process = subprocess.Popen(cmd,
- stdout=subprocess.PIPE,
- stderr=subprocess.STDOUT,
- close_fds=True,
- universal_newlines=True)
+ process = subprocess.Popen(
+ cmd,
+ stdout=subprocess.PIPE,
+ stderr=subprocess.STDOUT,
+ close_fds=True,
+ universal_newlines=True,
+ )
try:
data = process.communicate(timeout=12000)
except subprocess.TimeoutExpired:
@@ -73,12 +77,21 @@ def mirror_entry(mirror, args):
def main():
- parser = argparse.ArgumentParser(description='start-mirroring')
- parser.add_argument('--yaml-mirror', metavar='<yaml_mirror>', type=str,
- default=default_mirrors_yaml,
- help='The yaml mirror input file.')
- parser.add_argument('--verbose', const=True, default=False, action="store_const",
- help='Be verbose on otput.')
+ parser = argparse.ArgumentParser(description="start-mirroring")
+ parser.add_argument(
+ "--yaml-mirror",
+ metavar="<yaml_mirror>",
+ type=str,
+ default=default_mirrors_yaml,
+ help="The yaml mirror input file.",
+ )
+ parser.add_argument(
+ "--verbose",
+ const=True,
+ default=False,
+ action="store_const",
+ help="Be verbose on otput.",
+ )
args = parser.parse_args()
if not os.path.isfile(args.yaml_mirror):
@@ -86,14 +99,13 @@ def main():
sys.exit(1)
# load the yaml input file
- with open(f'{args.yaml_mirror}') as stream:
+ with open(f"{args.yaml_mirror}") as stream:
yaml_vars = yaml.safe_load(stream)
- if yaml_vars.get('mirrors') is None:
- raise Exception(f"Missing mirrors descriptions on %s" %
- (args.yaml_mirror))
+ if yaml_vars.get("mirrors") is None:
+ raise Exception(f"Missing mirrors descriptions on %s" % (args.yaml_mirror))
- if (args.verbose):
+ if args.verbose:
sys.stdout.write("Yaml mirror input: %s\n\n" % args.yaml_mirror)
# We do 3 passes, first to check the file has all requirements
@@ -103,25 +115,35 @@ def main():
# The second pass is for mirrors which do not have a reference, the
# third and final pass is for mirrors which do have a reference.
total = 0
- for mirror in yaml_vars['mirrors']:
+ for mirror in yaml_vars["mirrors"]:
total = total + 1
- if mirror.get('short_name') is None:
- raise Exception(f"Missing required short_name on mirror item #%d on file: %s" % (total, args.yaml_mirror))
- if mirror.get('url') is None:
- raise Exception(f"Missing required url on mirror item #%d on file: %s" % (total, args.yaml_mirror))
- if mirror.get('target') is None:
- raise Exception(f"Missing required target for mirror %s on yaml file %s on item #%d" % (mirror.get('short_name'), args.yaml_mirror, total))
+ if mirror.get("short_name") is None:
+ raise Exception(
+ f"Missing required short_name on mirror item #%d on file: %s"
+ % (total, args.yaml_mirror)
+ )
+ if mirror.get("url") is None:
+ raise Exception(
+ f"Missing required url on mirror item #%d on file: %s"
+ % (total, args.yaml_mirror)
+ )
+ if mirror.get("target") is None:
+ raise Exception(
+ f"Missing required target for mirror %s on yaml file %s on item #%d"
+ % (mirror.get("short_name"), args.yaml_mirror, total)
+ )
# Mirror trees without a reference first
- for mirror in yaml_vars['mirrors']:
- if not mirror.get('reference'):
+ for mirror in yaml_vars["mirrors"]:
+ if not mirror.get("reference"):
mirror_entry(mirror, args)
# Mirror trees which need a reference last
- for mirror in yaml_vars['mirrors']:
- if mirror.get('reference'):
+ for mirror in yaml_vars["mirrors"]:
+ if mirror.get("reference"):
mirror_entry(mirror, args)
+
if __name__ == "__main__":
main()
diff --git a/scripts/check_commit_format.py b/scripts/check_commit_format.py
index f72f9d1a..3ac37944 100755
--- a/scripts/check_commit_format.py
+++ b/scripts/check_commit_format.py
@@ -11,11 +11,16 @@ import subprocess
import sys
import re
+
def get_latest_commit_message():
"""Get the latest commit message"""
try:
- result = subprocess.run(['git', 'log', '-1', '--pretty=format:%B'],
- capture_output=True, text=True, check=True)
+ result = subprocess.run(
+ ["git", "log", "-1", "--pretty=format:%B"],
+ capture_output=True,
+ text=True,
+ check=True,
+ )
return result.stdout
except subprocess.CalledProcessError:
print("Error: Failed to get commit message")
@@ -24,30 +29,35 @@ def get_latest_commit_message():
print("Error: git command not found")
return None
+
def check_commit_format(commit_msg):
"""Check commit message formatting"""
issues = []
if not commit_msg:
return ["No commit message found"]
- lines = commit_msg.strip().split('\n')
+ lines = commit_msg.strip().split("\n")
# Find Generated-by line
generated_by_idx = None
signed_off_by_idx = None
for i, line in enumerate(lines):
- if line.startswith('Generated-by: Claude AI'):
+ if line.startswith("Generated-by: Claude AI"):
generated_by_idx = i
- elif line.startswith('Signed-off-by:'):
+ elif line.startswith("Signed-off-by:"):
signed_off_by_idx = i
# If Generated-by is present, check formatting
if generated_by_idx is not None:
if signed_off_by_idx is None:
- issues.append("Generated-by: Claude AI found but no Signed-off-by tag present")
+ issues.append(
+ "Generated-by: Claude AI found but no Signed-off-by tag present"
+ )
else:
# Check if Generated-by is immediately followed by Signed-off-by (no lines in between)
if signed_off_by_idx != generated_by_idx + 1:
lines_between = signed_off_by_idx - generated_by_idx - 1
if lines_between > 0:
- issues.append(f"Generated-by: Claude AI must be immediately followed by Signed-off-by (found {lines_between} lines between them)")
+ issues.append(
+ f"Generated-by: Claude AI must be immediately followed by Signed-off-by (found {lines_between} lines between them)"
+ )
for i in range(generated_by_idx + 1, signed_off_by_idx):
if lines[i].strip():
issues.append(f" - Non-empty line at {i+1}: '{lines[i]}'")
@@ -55,6 +65,7 @@ def check_commit_format(commit_msg):
issues.append(f" - Empty line at {i+1}")
return issues
+
def main():
"""Main function to check commit message format"""
commit_msg = get_latest_commit_message()
@@ -81,5 +92,6 @@ def main():
print("✅ Commit message formatting is correct!")
return 0
-if __name__ == '__main__':
+
+if __name__ == "__main__":
sys.exit(main())
diff --git a/scripts/coccinelle/generation/check_for_atomic_calls.py b/scripts/coccinelle/generation/check_for_atomic_calls.py
index 5849f964..9fb06ef2 100755
--- a/scripts/coccinelle/generation/check_for_atomic_calls.py
+++ b/scripts/coccinelle/generation/check_for_atomic_calls.py
@@ -28,27 +28,27 @@ parser = argparse.ArgumentParser(
description="Generate a Coccinelle checker for atomic context in transitive callers of a target function."
)
parser.add_argument(
- "--levels", "-l",
+ "--levels",
+ "-l",
type=int,
required=True,
- help="Maximum number of transitive caller levels to follow (e.g., 5)"
+ help="Maximum number of transitive caller levels to follow (e.g., 5)",
)
parser.add_argument(
- "--target", "-t",
+ "--target",
+ "-t",
type=str,
required=True,
- help="Target function to trace (e.g., __find_get_block_slow)"
+ help="Target function to trace (e.g., __find_get_block_slow)",
)
parser.add_argument(
- "--output", "-o",
- type=str,
- required=True,
- help="Output .cocci file to generate"
+ "--output", "-o", type=str, required=True, help="Output .cocci file to generate"
)
args = parser.parse_args()
max_depth = args.levels
target_func = args.target
+
# Add a function to get the number of processors for parallel jobs
def get_nprocs():
try:
@@ -56,6 +56,7 @@ def get_nprocs():
except:
return 1 # Default to 1 if can't determine
+
outfile = args.output
header = f"""// SPDX-License-Identifier: GPL-2.0
/// Autogenerated by gen_atomic_context_chain.py
@@ -141,10 +142,11 @@ register_caller(fn, None)
"""
with open(outfile, "w") as f:
f.write(header)
-
+
# Generate all the caller chain rules
for level in range(1, max_depth + 1):
- f.write(f"""
+ f.write(
+ f"""
// Level {level} caller discovery
@caller{level} depends on after_start exists@
identifier virtual.transitive_caller;
@@ -162,12 +164,14 @@ transitive_caller << virtual.transitive_caller;
@@
print(f"🔄 Chain level {level}: {{fn}} calls {{transitive_caller}} at {{p[0].file}}:{{p[0].line}}")
register_caller(fn, p[0].file)
-""")
+"""
+ )
# Check for atomic context in each caller in our chain
for level in range(1, max_depth + 1):
# First, check for common atomic primitives
- f.write(f"""
+ f.write(
+ f"""
// Level {level} atomic context check - Common atomic primitives
@atomiccheck{level} depends on after_start exists@
identifier virtual.transitive_caller;
@@ -246,10 +250,12 @@ key = (p1[0].file, p1[0].line, transitive_caller)
if key not in seen_atomic:
seen_atomic.add(key)
print(f"⚠️ WARNING: atomic context at level {level}: {{p1[0].current_element}} at {{p1[0].file}}:{{p1[0].line}} may reach {{transitive_caller}}() → eventually {target_func}()")
-""")
+"""
+ )
# Check for lock-related functions directly calling our chain
- f.write(f"""
+ f.write(
+ f"""
// Level {level} atomic context check - Lock-related functions
@atomic_fn_check{level} depends on after_start exists@
identifier virtual.transitive_caller;
@@ -275,10 +281,12 @@ transitive_caller << virtual.transitive_caller;
atomic_keywords = ['lock', 'irq', 'atomic', 'bh', 'intr', 'preempt', 'disable', 'napi', 'rcu']
if any(kw in lock_fn.lower() for kw in atomic_keywords):
print(f"⚠️ WARNING: potential atomic function at level {level}: {{lock_fn}} (name suggests lock handling) contains call to {{transitive_caller}}() at {{p[0].file}}:{{p[0].line}} → eventually {target_func}()")
-""")
+"""
+ )
# Check for spinlock regions
- f.write(f"""
+ f.write(
+ f"""
// Level {level} spinlock region check
@spinlock_region{level} depends on after_start exists@
identifier virtual.transitive_caller;
@@ -319,10 +327,12 @@ key = (p1[0].file, p1[0].line, p3[0].line, transitive_caller)
if key not in seen_spinlock_regions:
seen_spinlock_regions.add(key)
print(f"⚠️ WARNING: spinlock region at level {level}: {{p1[0].current_element}} at {{p1[0].file}}:{{p1[0].line}} contains call to {{transitive_caller}}() at line {{p3[0].line}} → eventually {target_func}()")
-""")
+"""
+ )
# Look for functions that can't sleep
- f.write(f"""
+ f.write(
+ f"""
// Level {level} check - Can't sleep contexts
@cant_sleep{level} depends on after_start exists@
identifier virtual.transitive_caller;
@@ -346,10 +356,12 @@ p2 << cant_sleep{level}.p2;
transitive_caller << virtual.transitive_caller;
@@
print(f"⚠️ WARNING: Non-sleeping context at {{p1[0].file}}:{{p1[0].line}} but calls {{transitive_caller}}() at line {{p2[0].line}} → eventually {target_func}()")
-""")
+"""
+ )
# Check for network driver contexts
- f.write(f"""
+ f.write(
+ f"""
// Level {level} check - Network driver contexts (commonly atomic)
@netdriver{level} depends on after_start exists@
identifier virtual.transitive_caller;
@@ -389,10 +401,12 @@ p2 << netdriver{level}.p2;
transitive_caller << virtual.transitive_caller;
@@
print(f"⚠️ WARNING: Network driver context at {{p1[0].file}}:{{p1[0].line}} but calls {{transitive_caller}}() at line {{p2[0].line}} → eventually {target_func}()")
-""")
+"""
+ )
# Check for functions that might call from atomic context by name
- f.write(f"""
+ f.write(
+ f"""
// Level {level} check - Function with name suggesting atomic context
@atomic_name{level} depends on after_start exists@
identifier virtual.transitive_caller;
@@ -411,10 +425,12 @@ p << atomic_name{level}.p;
transitive_caller << virtual.transitive_caller;
@@
print(f"⚠️ WARNING: Function with atomic-suggesting name {{atomic_fn}} calls {{transitive_caller}}() at {{p[0].file}}:{{p[0].line}} → eventually {target_func}()")
-""")
+"""
+ )
# Check for sleep-incompatible contexts but target function might sleep
- f.write(f"""
+ f.write(
+ f"""
// Level {level} check - Target function called in context where might_sleep is used
@might_sleep_check{level} depends on after_start exists@
identifier virtual.transitive_caller;
@@ -436,8 +452,11 @@ p2 << might_sleep_check{level}.p2;
transitive_caller << virtual.transitive_caller;
@@
print(f"⚠️ WARNING: Function has might_sleep() at {{p1[0].file}}:{{p1[0].line}} but also calls {{transitive_caller}}() at line {{p2[0].line}} → eventually {target_func}()")
-""")
+"""
+ )
f.write("\n")
-print(f"✅ Generated {outfile} with enhanced atomic checks for `{target_func}` up to {max_depth} levels. Run with: make coccicheck MODE=report COCCI={outfile} J={get_nprocs()}")
+print(
+ f"✅ Generated {outfile} with enhanced atomic checks for `{target_func}` up to {max_depth} levels. Run with: make coccicheck MODE=report COCCI={outfile} J={get_nprocs()}"
+)
diff --git a/scripts/coccinelle/generation/check_for_sleepy_calls.py b/scripts/coccinelle/generation/check_for_sleepy_calls.py
index 87bd5b26..a32c9ee2 100755
--- a/scripts/coccinelle/generation/check_for_sleepy_calls.py
+++ b/scripts/coccinelle/generation/check_for_sleepy_calls.py
@@ -34,34 +34,35 @@ parser = argparse.ArgumentParser(
description="Generate a Coccinelle checker to find sleeping functions called by a target function."
)
parser.add_argument(
- "--function", "-f",
+ "--function",
+ "-f",
type=str,
required=True,
- help="Target function to analyze (e.g., netif_rx_ni)"
+ help="Target function to analyze (e.g., netif_rx_ni)",
)
parser.add_argument(
- "--max-depth", "-d",
+ "--max-depth",
+ "-d",
type=int,
default=3,
- help="Maximum depth of function call chain to analyze (default: 3)"
+ help="Maximum depth of function call chain to analyze (default: 3)",
)
parser.add_argument(
- "--output", "-o",
- type=str,
- required=True,
- help="Output .cocci file to generate"
+ "--output", "-o", type=str, required=True, help="Output .cocci file to generate"
)
parser.add_argument(
- "--sleepy-function", "-s",
+ "--sleepy-function",
+ "-s",
type=str,
default=None,
- help="Specific function to check for that may cause sleeping (e.g., folio_wait_locked)"
+ help="Specific function to check for that may cause sleeping (e.g., folio_wait_locked)",
)
parser.add_argument(
- "--expected", "-e",
+ "--expected",
+ "-e",
action="store_true",
default=False,
- help="Indicate that the function is expected to have a sleep path (verified by manual inspection)"
+ help="Indicate that the function is expected to have a sleep path (verified by manual inspection)",
)
args = parser.parse_args()
target_func = args.function
@@ -70,6 +71,7 @@ outfile = args.output
sleepy_func = args.sleepy_function
expected_to_sleep = args.expected
+
# Add a function to get the number of processors for parallel jobs
def get_nprocs():
try:
@@ -77,17 +79,49 @@ def get_nprocs():
except:
return 1 # Default to 1 if can't determine
+
# List of common functions known to sleep
known_sleepy_functions = [
- "msleep", "ssleep", "usleep_range", "schedule", "schedule_timeout",
- "wait_event", "wait_for_completion", "mutex_lock", "down_read", "down_write",
- "kthread_create", "kthread_run", "kmalloc", "__kmalloc", "kmem_cache_alloc",
- "vmalloc", "vzalloc", "kvmalloc", "kzalloc", "__vmalloc", "kvzalloc",
- "sock_create", "sock_create_kern", "sock_create_lite", "sock_socket",
- "filp_open", "open_bdev_exclusive", "create_workqueue",
- "alloc_workqueue", "__alloc_workqueue_key", "request_threaded_irq",
- "request_module", "try_module_get", "module_put", "printk", "GFP_KERNEL",
- "copy_from_user", "copy_to_user", "__copy_from_user", "__copy_to_user"
+ "msleep",
+ "ssleep",
+ "usleep_range",
+ "schedule",
+ "schedule_timeout",
+ "wait_event",
+ "wait_for_completion",
+ "mutex_lock",
+ "down_read",
+ "down_write",
+ "kthread_create",
+ "kthread_run",
+ "kmalloc",
+ "__kmalloc",
+ "kmem_cache_alloc",
+ "vmalloc",
+ "vzalloc",
+ "kvmalloc",
+ "kzalloc",
+ "__vmalloc",
+ "kvzalloc",
+ "sock_create",
+ "sock_create_kern",
+ "sock_create_lite",
+ "sock_socket",
+ "filp_open",
+ "open_bdev_exclusive",
+ "create_workqueue",
+ "alloc_workqueue",
+ "__alloc_workqueue_key",
+ "request_threaded_irq",
+ "request_module",
+ "try_module_get",
+ "module_put",
+ "printk",
+ "GFP_KERNEL",
+ "copy_from_user",
+ "copy_to_user",
+ "__copy_from_user",
+ "__copy_to_user",
]
# If a specific sleepy function is provided, only check for that one
@@ -97,9 +131,17 @@ if sleepy_func:
# List of common GFP flags that indicate sleeping is allowed
sleepy_gfp_flags = [
- "GFP_KERNEL", "GFP_USER", "GFP_HIGHUSER", "GFP_DMA",
- "GFP_DMA32", "GFP_NOWAIT", "GFP_NOIO", "GFP_NOFS",
- "__GFP_WAIT", "__GFP_IO", "__GFP_FS"
+ "GFP_KERNEL",
+ "GFP_USER",
+ "GFP_HIGHUSER",
+ "GFP_DMA",
+ "GFP_DMA32",
+ "GFP_NOWAIT",
+ "GFP_NOIO",
+ "GFP_NOFS",
+ "__GFP_WAIT",
+ "__GFP_IO",
+ "__GFP_FS",
]
# Create a stats directory
@@ -111,8 +153,9 @@ with open(outfile, "w") as f:
title = f"Detect if function '{target_func}' calls any functions that might sleep"
if sleepy_func:
title = f"Detect if function '{target_func}' calls '{sleepy_func}'"
-
- f.write(f"""// SPDX-License-Identifier: GPL-2.0
+
+ f.write(
+ f"""// SPDX-License-Identifier: GPL-2.0
/// Autogenerated by check_for_sleepy_calls.py
/// {title}
// Options: --no-includes --include-headers
@@ -281,10 +324,12 @@ def save_stats():
with open(stats_file, "w") as f:
json.dump(stats, f, indent=2)
-""")
+"""
+ )
# Define the rule to find direct calls to the target function
- f.write(f"""
+ f.write(
+ f"""
// Find direct function calls made by target function
@find_calls@
identifier fn;
@@ -305,11 +350,13 @@ total_calls_checked += 1
register_call(target_func, fn, p[0].file, p[0].line)
register_func_for_analysis(fn)
save_stats()
-""")
+"""
+ )
# Add direct checking for specific sleepy function if provided
if sleepy_func:
- f.write(f"""
+ f.write(
+ f"""
// Direct check: Does target function call sleepy function directly?
@direct_sleepy_call@
position p;
@@ -327,12 +374,16 @@ global total_sleep_routines_checked
total_sleep_routines_checked += 1
register_sleep_point(target_func, sleepy_func, p[0].file, p[0].line, "directly calls target sleep function")
save_stats()
-""")
-
+"""
+ )
+
# Generate rules for checking nested function calls
- for depth in range(2, max_depth + 1): # Start from 2 as level 1 is the direct call we already checked
+ for depth in range(
+ 2, max_depth + 1
+ ): # Start from 2 as level 1 is the direct call we already checked
# Find functions called by functions at the previous level
- f.write(f"""
+ f.write(
+ f"""
// Level {depth} - Find functions called by level {depth-1} functions
@find_calls_l{depth}@
identifier fn1;
@@ -357,11 +408,13 @@ if fn1 in seen_funcs and fn1 != fn2: # Avoid self-recursion
register_call(fn1, fn2, p[0].file, p[0].line)
register_func_for_analysis(fn2)
save_stats()
-""")
+"""
+ )
if sleepy_func:
# If looking for a specific sleepy function, check at this level
- f.write(f"""
+ f.write(
+ f"""
// Level {depth} - Find calls to sleepy function
@sleepy_call_l{depth}@
identifier fn;
@@ -383,10 +436,12 @@ total_sleep_routines_checked += 1
if fn in seen_funcs:
register_sleep_point(fn, sleepy_func, p[0].file, p[0].line, f"level {depth} call to target sleep function")
save_stats()
-""")
+"""
+ )
else:
# If doing general sleep checking, check for known sleepy functions at this level
- f.write(f"""
+ f.write(
+ f"""
// Level {depth} - Check for known sleepy functions
@known_sleepers_l{depth}@
identifier fn;
@@ -394,16 +449,22 @@ position p;
@@
fn(...) {{
<...
- (""")
+ ("""
+ )
# Add all known sleepy functions to the pattern
for i, sleepy_func_name in enumerate(known_sleepy_functions):
if i > 0:
- f.write(f"""
- |""")
- f.write(f"""
- {sleepy_func_name}@p(...)""")
-
- f.write(f"""
+ f.write(
+ f"""
+ |"""
+ )
+ f.write(
+ f"""
+ {sleepy_func_name}@p(...)"""
+ )
+
+ f.write(
+ f"""
)
...>
}}
@@ -419,12 +480,14 @@ if fn in seen_funcs:
sleep_func = p[0].current_element
register_sleep_point(fn, sleep_func, p[0].file, p[0].line, f"level {depth} call to known sleeping function")
save_stats()
-""")
+"""
+ )
# Only add the other sleep detection rules if we're not constraining to a specific function
if not sleepy_func:
# Check for GFP_KERNEL and other sleepy allocation flags
- f.write(f"""
+ f.write(
+ f"""
// Check for sleepy memory allocation flags
@check_sleepy_alloc@
position p;
@@ -432,15 +495,21 @@ identifier fn;
@@
fn(...) {{
<...
- (""")
+ ("""
+ )
# Add patterns for all sleepy GFP flags
for i, flag in enumerate(sleepy_gfp_flags):
if i > 0:
- f.write(f"""
- |""")
- f.write(f"""
- {flag}@p""")
- f.write(f"""
+ f.write(
+ f"""
+ |"""
+ )
+ f.write(
+ f"""
+ {flag}@p"""
+ )
+ f.write(
+ f"""
)
...>
}}
@@ -456,10 +525,12 @@ if fn in seen_funcs:
flag = p[0].current_element
register_sleep_point(fn, flag, p[0].file, p[0].line, "uses allocation flag that may sleep")
save_stats()
-""")
+"""
+ )
# Check for mutex locks
- f.write(f"""
+ f.write(
+ f"""
// Check for mutex locks
@check_mutex@
position p;
@@ -510,10 +581,12 @@ if fn in seen_funcs:
lock_func = p[0].current_element
register_sleep_point(fn, lock_func, p[0].file, p[0].line, "uses mutex or completion that may sleep")
save_stats()
-""")
+"""
+ )
# Check for might_sleep calls
- f.write(f"""
+ f.write(
+ f"""
// Check for explicit might_sleep calls
@check_might_sleep@
position p;
@@ -542,10 +615,12 @@ if fn in seen_funcs:
sleep_func = p[0].current_element
register_sleep_point(fn, sleep_func, p[0].file, p[0].line, "contains explicit might_sleep() call")
save_stats()
-""")
+"""
+ )
# Check for functions with names suggesting they might sleep
- f.write(f"""
+ f.write(
+ f"""
// Check for functions with sleep-suggesting names
@check_sleep_names@
position p;
@@ -574,10 +649,12 @@ if fn in seen_funcs:
sleep_fn.startswith("local_")):
register_sleep_point(fn, sleep_fn, p[0].file, p[0].line, "calls function with name suggesting it might sleep")
save_stats()
-""")
+"""
+ )
# Add a finalization rule that summarizes the findings
- f.write(f"""
+ f.write(
+ f"""
@finalize:python@
@@
# Save any final stats before finishing
@@ -669,7 +746,8 @@ try:
shutil.rmtree(stats_dir)
except Exception as e:
print(f"Note: Could not clean up stats directory: {{e}}")
-""")
+"""
+ )
msg = f"✅ Generated {outfile} to check if '{target_func}' might sleep"
if sleepy_func:
diff --git a/scripts/detect_whitespace_issues.py b/scripts/detect_whitespace_issues.py
index 165a33e2..de5fef70 100755
--- a/scripts/detect_whitespace_issues.py
+++ b/scripts/detect_whitespace_issues.py
@@ -12,37 +12,40 @@ import os
import sys
from pathlib import Path
+
def check_file_whitespace(file_path):
"""Check a single file for whitespace issues"""
issues = []
try:
- with open(file_path, 'rb') as f:
+ with open(file_path, "rb") as f:
content = f.read()
# Skip binary files
- if b'\0' in content:
+ if b"\0" in content:
return issues
- lines = content.decode('utf-8', errors='ignore').splitlines(keepends=True)
+ lines = content.decode("utf-8", errors="ignore").splitlines(keepends=True)
# Check trailing whitespace
for line_num, line in enumerate(lines, 1):
- if line.rstrip('\n\r').endswith(' ') or line.rstrip('\n\r').endswith('\t'):
+ if line.rstrip("\n\r").endswith(" ") or line.rstrip("\n\r").endswith("\t"):
issues.append(f"Line {line_num}: Trailing whitespace")
# Check missing newline at end of file
- if content and not content.endswith(b'\n'):
+ if content and not content.endswith(b"\n"):
issues.append("Missing newline at end of file")
# Check for excessive blank lines (more than 2 consecutive)
blank_count = 0
for line_num, line in enumerate(lines, 1):
- if line.strip() == '':
+ if line.strip() == "":
blank_count += 1
else:
if blank_count > 2:
- issues.append(f"Line {line_num - blank_count}: {blank_count} consecutive blank lines")
+ issues.append(
+ f"Line {line_num - blank_count}: {blank_count} consecutive blank lines"
+ )
blank_count = 0
except Exception as e:
@@ -50,6 +53,7 @@ def check_file_whitespace(file_path):
return issues
+
def main():
"""Main function to scan for whitespace issues"""
if len(sys.argv) > 1:
@@ -57,10 +61,15 @@ def main():
else:
# Default to git tracked files with modifications
import subprocess
+
try:
- result = subprocess.run(['git', 'diff', '--name-only'],
- capture_output=True, text=True, check=True)
- paths = result.stdout.strip().split('\n') if result.stdout.strip() else []
+ result = subprocess.run(
+ ["git", "diff", "--name-only"],
+ capture_output=True,
+ text=True,
+ check=True,
+ )
+ paths = result.stdout.strip().split("\n") if result.stdout.strip() else []
if not paths:
print("No modified files found in git")
return
@@ -82,7 +91,7 @@ def main():
if path.is_file():
# Skip certain file types
- if path.suffix in ['.pyc', '.so', '.o', '.bin', '.jpg', '.png', '.gif']:
+ if path.suffix in [".pyc", ".so", ".o", ".bin", ".jpg", ".png", ".gif"]:
continue
issues = check_file_whitespace(path)
@@ -93,7 +102,9 @@ def main():
for issue in issues:
print(f" ⚠️ {issue}")
- print(f"\nSummary: {total_issues} whitespace issues found in {files_with_issues} files")
+ print(
+ f"\nSummary: {total_issues} whitespace issues found in {files_with_issues} files"
+ )
if total_issues > 0:
print("\nTo fix these issues:")
@@ -105,5 +116,6 @@ def main():
print("✅ No whitespace issues found!")
return 0
-if __name__ == '__main__':
+
+if __name__ == "__main__":
sys.exit(main())
diff --git a/scripts/fix_whitespace_issues.py b/scripts/fix_whitespace_issues.py
index 3e69ea50..585ff8b9 100755
--- a/scripts/fix_whitespace_issues.py
+++ b/scripts/fix_whitespace_issues.py
@@ -12,19 +12,20 @@ import os
import sys
from pathlib import Path
+
def fix_file_whitespace(file_path):
"""Fix whitespace issues in a single file"""
issues_fixed = []
try:
- with open(file_path, 'rb') as f:
+ with open(file_path, "rb") as f:
content = f.read()
# Skip binary files
- if b'\0' in content:
+ if b"\0" in content:
return issues_fixed
- original_content = content.decode('utf-8', errors='ignore')
+ original_content = content.decode("utf-8", errors="ignore")
lines = original_content.splitlines(keepends=True)
modified = False
@@ -33,12 +34,12 @@ def fix_file_whitespace(file_path):
for line_num, line in enumerate(lines, 1):
original_line = line
# Remove trailing whitespace but preserve line endings
- if line.endswith('\r\n'):
- cleaned_line = line.rstrip(' \t\r\n') + '\r\n'
- elif line.endswith('\n'):
- cleaned_line = line.rstrip(' \t\n') + '\n'
+ if line.endswith("\r\n"):
+ cleaned_line = line.rstrip(" \t\r\n") + "\r\n"
+ elif line.endswith("\n"):
+ cleaned_line = line.rstrip(" \t\n") + "\n"
else:
- cleaned_line = line.rstrip(' \t')
+ cleaned_line = line.rstrip(" \t")
if original_line != cleaned_line:
issues_fixed.append(f"Line {line_num}: Removed trailing whitespace")
@@ -52,7 +53,7 @@ def fix_file_whitespace(file_path):
i = 0
while i < len(new_lines):
line = new_lines[i]
- if line.strip() == '':
+ if line.strip() == "":
blank_count += 1
if blank_count <= 2:
final_lines.append(line)
@@ -65,15 +66,15 @@ def fix_file_whitespace(file_path):
i += 1
# Fix missing newline at end of file
- new_content = ''.join(final_lines)
- if new_content and not new_content.endswith('\n'):
- new_content += '\n'
+ new_content = "".join(final_lines)
+ if new_content and not new_content.endswith("\n"):
+ new_content += "\n"
issues_fixed.append("Added missing newline at end of file")
modified = True
# Write back if modified
if modified:
- with open(file_path, 'w', encoding='utf-8') as f:
+ with open(file_path, "w", encoding="utf-8") as f:
f.write(new_content)
except Exception as e:
@@ -81,6 +82,7 @@ def fix_file_whitespace(file_path):
return issues_fixed
+
def main():
"""Main function to fix whitespace issues"""
if len(sys.argv) > 1:
@@ -88,10 +90,15 @@ def main():
else:
# Default to git tracked files with modifications
import subprocess
+
try:
- result = subprocess.run(['git', 'diff', '--name-only'],
- capture_output=True, text=True, check=True)
- paths = result.stdout.strip().split('\n') if result.stdout.strip() else []
+ result = subprocess.run(
+ ["git", "diff", "--name-only"],
+ capture_output=True,
+ text=True,
+ check=True,
+ )
+ paths = result.stdout.strip().split("\n") if result.stdout.strip() else []
if not paths:
print("No modified files found in git")
return 0
@@ -113,7 +120,7 @@ def main():
if path.is_file():
# Skip certain file types
- if path.suffix in ['.pyc', '.so', '.o', '.bin', '.jpg', '.png', '.gif']:
+ if path.suffix in [".pyc", ".so", ".o", ".bin", ".jpg", ".png", ".gif"]:
continue
fixes = fix_file_whitespace(path)
@@ -133,5 +140,6 @@ def main():
print("✅ No whitespace issues found to fix!")
return 0
-if __name__ == '__main__':
+
+if __name__ == "__main__":
sys.exit(main())
diff --git a/scripts/generate_refs.py b/scripts/generate_refs.py
index 4bd179b9..45b74348 100755
--- a/scripts/generate_refs.py
+++ b/scripts/generate_refs.py
@@ -120,7 +120,7 @@ def _ref_generator_choices_static(args, conf_name, ref_name, ref_help):
# Add "_USER_REF" suffix to avoid static duplicates when both user
# and default Kconfig files exists. Fixes 'warning: choice value used
# outside its choice group'
- if 'refs' in args and args.refs != 0:
+ if "refs" in args and args.refs != 0:
conf_name = conf_name + "_USER_REF"
refs.update({ref_name: conf_name})
f.write("config {}\n".format(conf_name))
@@ -328,7 +328,9 @@ def kreleases(args) -> None:
for release in data["releases"]:
if release["moniker"] == args.moniker:
# Check if release.json is aa.bb.cc type
- if re.compile(r'^\d+\.\d+(\.\d+|-rc\d+)?$').match(release["version"]):
+ if re.compile(r"^\d+\.\d+(\.\d+|-rc\d+)?$").match(
+ release["version"]
+ ):
reflist.append("v" + release["version"])
else:
reflist.append(release["version"])
diff --git a/scripts/honey-badger.py b/scripts/honey-badger.py
index 29c3966b..43dd45c2 100755
--- a/scripts/honey-badger.py
+++ b/scripts/honey-badger.py
@@ -21,16 +21,18 @@ KERNEL_PPA_URL = "https://kernel.ubuntu.com/mainline/"
ARCH = "amd64"
KERNEL_DIR = "/tmp/kernels"
+
def is_dpkg_installed():
- return shutil.which('dpkg') is not None
+ return shutil.which("dpkg") is not None
+
def extract_deb(deb_file, tempdir, verbose=False, dest="/"):
try:
if verbose:
print(f"Extracting {deb_file} onto {tempdir}")
# Extract the ar archive
- subprocess.run(['ar', 'x', deb_file], check=True, cwd=tempdir)
- data_tarball = next(f for f in os.listdir(tempdir) if f.startswith('data.tar'))
+ subprocess.run(["ar", "x", deb_file], check=True, cwd=tempdir)
+ data_tarball = next(f for f in os.listdir(tempdir) if f.startswith("data.tar"))
# Extract the data tarball to the correct locations
with tarfile.open(os.path.join(tempdir, data_tarball)) as tar:
tar.extractall(path=dest)
@@ -43,7 +45,8 @@ def extract_deb(deb_file, tempdir, verbose=False, dest="/"):
if os.path.exists(tempdir):
if verbose:
print(f"Removing temporary directory {tempdir}")
- subprocess.run(['rm', '-rf', tempdir])
+ subprocess.run(["rm", "-rf", tempdir])
+
def install_kernel_packages(package_files, verbose=False, use_ar=False, dest="/"):
if use_ar:
@@ -56,49 +59,66 @@ def install_kernel_packages(package_files, verbose=False, use_ar=False, dest="/"
for package in package_files:
if verbose:
print("Running: dpkg %s" % package)
- subprocess.run(['sudo', 'dpkg', '-i', package], check=True)
+ subprocess.run(["sudo", "dpkg", "-i", package], check=True)
+
def parse_version(version):
- match = re.match(r'v(\d+)\.(\d+)(?:\.(\d+))?(?:-(rc\d+))?', version)
+ match = re.match(r"v(\d+)\.(\d+)(?:\.(\d+))?(?:-(rc\d+))?", version)
if match:
major, minor, patch, rc = match.groups()
- return (int(major), int(minor), int(patch) if patch else 0, rc if rc else '')
- return (0, 0, 0, '')
+ return (int(major), int(minor), int(patch) if patch else 0, rc if rc else "")
+ return (0, 0, 0, "")
+
def get_kernel_versions():
response = requests.get(KERNEL_PPA_URL)
response.raise_for_status()
- soup = BeautifulSoup(response.text, 'html.parser')
- versions = [href.strip('/') for link in soup.find_all('a') if (href := link.get('href')).startswith('v') and href.endswith('/')]
+ soup = BeautifulSoup(response.text, "html.parser")
+ versions = [
+ href.strip("/")
+ for link in soup.find_all("a")
+ if (href := link.get("href")).startswith("v") and href.endswith("/")
+ ]
versions = sorted(versions, key=parse_version, reverse=True)
return group_versions(versions)
+
def group_versions(versions):
grouped = []
- for key, group in groupby(versions, lambda x: (parse_version(x)[0], parse_version(x)[1])):
+ for key, group in groupby(
+ versions, lambda x: (parse_version(x)[0], parse_version(x)[1])
+ ):
group = sorted(group, key=parse_version, reverse=True)
grouped.append(list(group))
return grouped
+
def verify_kernel_files(version):
url = f"{KERNEL_PPA_URL}{version}/{ARCH}/"
response = requests.get(url)
if response.status_code != 200:
return False
- soup = BeautifulSoup(response.text, 'html.parser')
- files = [a['href'] for a in soup.find_all('a') if a['href'].endswith('.deb')]
- if any('linux-image-unsigned' in f for f in files) and any('linux-modules' in f for f in files):
+ soup = BeautifulSoup(response.text, "html.parser")
+ files = [a["href"] for a in soup.find_all("a") if a["href"].endswith(".deb")]
+ if any("linux-image-unsigned" in f for f in files) and any(
+ "linux-modules" in f for f in files
+ ):
return True
return False
+
def download_and_install(file_type, version, verbose=False, use_ar=False, dest="/"):
url = f"{KERNEL_PPA_URL}{version}/{ARCH}/"
response = requests.get(url)
response.raise_for_status() # Ensure we raise an exception for failed requests
- soup = BeautifulSoup(response.text, 'html.parser')
+ soup = BeautifulSoup(response.text, "html.parser")
- deb_files = [a['href'] for a in soup.find_all('a') if a['href'].endswith('.deb') and file_type in a['href']]
+ deb_files = [
+ a["href"]
+ for a in soup.find_all("a")
+ if a["href"].endswith(".deb") and file_type in a["href"]
+ ]
local_deb_files = []
if not os.path.exists(KERNEL_DIR):
@@ -111,7 +131,7 @@ def download_and_install(file_type, version, verbose=False, use_ar=False, dest="
print(f"Attempting to download from {full_url}...")
r = requests.get(full_url, stream=True)
if r.status_code == 200:
- with open(local_file, 'wb') as f:
+ with open(local_file, "wb") as f:
f.write(r.content)
if verbose:
print(f"Downloaded {local_file}.")
@@ -122,16 +142,38 @@ def download_and_install(file_type, version, verbose=False, use_ar=False, dest="
install_kernel_packages(local_deb_files, verbose, use_ar, dest)
+
def main():
dpkg_installed = is_dpkg_installed()
parser = argparse.ArgumentParser(description="Linux stable kernel honey badger")
- parser.add_argument('--list', action='store_true', help='List available kernels')
- parser.add_argument('--install', action='store_true', help='Install specified number of latest kernels')
- parser.add_argument('--dest', type=str, help='Install the packages into the specified directory')
- parser.add_argument('--verbose', action='store_true', help='Enable verbose output')
- parser.add_argument('--use-ar', action='store_true', default=dpkg_installed, help='Do not use dpkg even if present')
- parser.add_argument('--use-file', type=str, help='Skip download and just install this debian package file')
- parser.add_argument('-c', '--count', type=int, default=1, help='Number of kernels to list or install')
+ parser.add_argument("--list", action="store_true", help="List available kernels")
+ parser.add_argument(
+ "--install",
+ action="store_true",
+ help="Install specified number of latest kernels",
+ )
+ parser.add_argument(
+ "--dest", type=str, help="Install the packages into the specified directory"
+ )
+ parser.add_argument("--verbose", action="store_true", help="Enable verbose output")
+ parser.add_argument(
+ "--use-ar",
+ action="store_true",
+ default=dpkg_installed,
+ help="Do not use dpkg even if present",
+ )
+ parser.add_argument(
+ "--use-file",
+ type=str,
+ help="Skip download and just install this debian package file",
+ )
+ parser.add_argument(
+ "-c",
+ "--count",
+ type=int,
+ default=1,
+ help="Number of kernels to list or install",
+ )
args = parser.parse_args()
kernel_versions_grouped = get_kernel_versions()
@@ -145,7 +187,9 @@ def main():
break
version = group[0] # Pick the latest version in the group
if args.verbose:
- print(f"Verifying files for {version} at {KERNEL_PPA_URL}{version}/{ARCH}/...")
+ print(
+ f"Verifying files for {version} at {KERNEL_PPA_URL}{version}/{ARCH}/..."
+ )
if verify_kernel_files(version):
valid_versions.append(version)
else:
@@ -157,7 +201,7 @@ def main():
return
if args.use_file and args.install:
- pkgs = [ args.use_file ]
+ pkgs = [args.use_file]
install_kernel_packages(pkgs, args.verbose, args.use_ar, args.dest)
return
@@ -165,9 +209,12 @@ def main():
for version in valid_versions:
if args.verbose:
print(f"Installing kernel version {version}...")
- files = ['linux-modules', 'linux-image-unsigned', 'linux-headers']
+ files = ["linux-modules", "linux-image-unsigned", "linux-headers"]
for file_type in files:
- download_and_install(file_type, version, args.verbose, args.use_ar, args.dest)
+ download_and_install(
+ file_type, version, args.verbose, args.use_ar, args.dest
+ )
+
if __name__ == "__main__":
main()
diff --git a/scripts/spdxcheck.py b/scripts/spdxcheck.py
index d6fb62e2..3a1f3a95 100755
--- a/scripts/spdxcheck.py
+++ b/scripts/spdxcheck.py
@@ -11,31 +11,35 @@ import git
import re
import os
+
class ParserException(Exception):
def __init__(self, tok, txt):
self.tok = tok
self.txt = txt
+
class SPDXException(Exception):
def __init__(self, el, txt):
self.el = el
self.txt = txt
+
class SPDXdata(object):
def __init__(self):
self.license_files = 0
self.exception_files = 0
- self.licenses = [ ]
- self.exceptions = { }
+ self.licenses = []
+ self.exceptions = {}
+
# Read the spdx data from the LICENSES directory
def read_spdxdata(repo):
# The subdirectories of LICENSES in the kernel source
# Note: exceptions needs to be parsed as last directory.
- #license_dirs = [ "preferred", "dual", "deprecated", "exceptions" ]
- license_dirs = [ "preferred" ]
- lictree = repo.head.commit.tree['LICENSES']
+ # license_dirs = [ "preferred", "dual", "deprecated", "exceptions" ]
+ license_dirs = ["preferred"]
+ lictree = repo.head.commit.tree["LICENSES"]
spdx = SPDXdata()
@@ -46,50 +50,65 @@ def read_spdxdata(repo):
exception = None
for l in open(el.path).readlines():
- if l.startswith('Valid-License-Identifier:'):
- lid = l.split(':')[1].strip().upper()
+ if l.startswith("Valid-License-Identifier:"):
+ lid = l.split(":")[1].strip().upper()
if lid in spdx.licenses:
- raise SPDXException(el, 'Duplicate License Identifier: %s' %lid)
+ raise SPDXException(
+ el, "Duplicate License Identifier: %s" % lid
+ )
else:
spdx.licenses.append(lid)
- elif l.startswith('SPDX-Exception-Identifier:'):
- exception = l.split(':')[1].strip().upper()
+ elif l.startswith("SPDX-Exception-Identifier:"):
+ exception = l.split(":")[1].strip().upper()
spdx.exceptions[exception] = []
- elif l.startswith('SPDX-Licenses:'):
- for lic in l.split(':')[1].upper().strip().replace(' ', '').replace('\t', '').split(','):
+ elif l.startswith("SPDX-Licenses:"):
+ for lic in (
+ l.split(":")[1]
+ .upper()
+ .strip()
+ .replace(" ", "")
+ .replace("\t", "")
+ .split(",")
+ ):
if not lic in spdx.licenses:
- raise SPDXException(None, 'Exception %s missing license %s' %(exception, lic))
+ raise SPDXException(
+ None,
+ "Exception %s missing license %s" % (exception, lic),
+ )
spdx.exceptions[exception].append(lic)
elif l.startswith("License-Text:"):
if exception:
if not len(spdx.exceptions[exception]):
- raise SPDXException(el, 'Exception %s is missing SPDX-Licenses' %exception)
+ raise SPDXException(
+ el, "Exception %s is missing SPDX-Licenses" % exception
+ )
spdx.exception_files += 1
else:
spdx.license_files += 1
break
return spdx
+
class id_parser(object):
- reserved = [ 'AND', 'OR', 'WITH' ]
- tokens = [ 'LPAR', 'RPAR', 'ID', 'EXC' ] + reserved
+ reserved = ["AND", "OR", "WITH"]
+ tokens = ["LPAR", "RPAR", "ID", "EXC"] + reserved
- precedence = ( ('nonassoc', 'AND', 'OR'), )
+ precedence = (("nonassoc", "AND", "OR"),)
- t_ignore = ' \t'
+ t_ignore = " \t"
def __init__(self, spdx):
self.spdx = spdx
self.lasttok = None
self.lastid = None
- self.lexer = lex.lex(module = self, reflags = re.UNICODE)
+ self.lexer = lex.lex(module=self, reflags=re.UNICODE)
# Initialize the parser. No debug file and no parser rules stored on disk
# The rules are small enough to be generated on the fly
- self.parser = yacc.yacc(module = self, write_tables = False, debug = False)
+ self.parser = yacc.yacc(module=self, write_tables=False, debug=False)
self.lines_checked = 0
self.checked = 0
self.spdx_valid = 0
@@ -100,93 +119,95 @@ class id_parser(object):
# Validate License and Exception IDs
def validate(self, tok):
id = tok.value.upper()
- if tok.type == 'ID':
+ if tok.type == "ID":
if not id in self.spdx.licenses:
- raise ParserException(tok, 'Invalid License ID')
+ raise ParserException(tok, "Invalid License ID")
self.lastid = id
- elif tok.type == 'EXC':
+ elif tok.type == "EXC":
if id not in self.spdx.exceptions:
- raise ParserException(tok, 'Invalid Exception ID')
+ raise ParserException(tok, "Invalid Exception ID")
if self.lastid not in self.spdx.exceptions[id]:
- raise ParserException(tok, 'Exception not valid for license %s' %self.lastid)
+ raise ParserException(
+ tok, "Exception not valid for license %s" % self.lastid
+ )
self.lastid = None
- elif tok.type != 'WITH':
+ elif tok.type != "WITH":
self.lastid = None
# Lexer functions
def t_RPAR(self, tok):
- r'\)'
+ r"\)"
self.lasttok = tok.type
return tok
def t_LPAR(self, tok):
- r'\('
+ r"\("
self.lasttok = tok.type
return tok
def t_ID(self, tok):
- r'[A-Za-z.0-9\-+]+'
+ r"[A-Za-z.0-9\-+]+"
- if self.lasttok == 'EXC':
+ if self.lasttok == "EXC":
print(tok)
- raise ParserException(tok, 'Missing parentheses')
+ raise ParserException(tok, "Missing parentheses")
tok.value = tok.value.strip()
val = tok.value.upper()
if val in self.reserved:
tok.type = val
- elif self.lasttok == 'WITH':
- tok.type = 'EXC'
+ elif self.lasttok == "WITH":
+ tok.type = "EXC"
self.lasttok = tok.type
self.validate(tok)
return tok
def t_error(self, tok):
- raise ParserException(tok, 'Invalid token')
+ raise ParserException(tok, "Invalid token")
def p_expr(self, p):
- '''expr : ID
- | ID WITH EXC
- | expr AND expr
- | expr OR expr
- | LPAR expr RPAR'''
+ """expr : ID
+ | ID WITH EXC
+ | expr AND expr
+ | expr OR expr
+ | LPAR expr RPAR"""
pass
def p_error(self, p):
if not p:
- raise ParserException(None, 'Unfinished license expression')
+ raise ParserException(None, "Unfinished license expression")
else:
- raise ParserException(p, 'Syntax error')
+ raise ParserException(p, "Syntax error")
def parse(self, expr):
self.lasttok = None
self.lastid = None
- self.parser.parse(expr, lexer = self.lexer)
+ self.parser.parse(expr, lexer=self.lexer)
def parse_lines(self, fd, maxlines, fname):
self.checked += 1
self.curline = 0
try:
for line in fd:
- line = line.decode(locale.getpreferredencoding(False), errors='ignore')
+ line = line.decode(locale.getpreferredencoding(False), errors="ignore")
self.curline += 1
if self.curline > maxlines:
break
self.lines_checked += 1
if line.find("SPDX-License-Identifier:") < 0:
continue
- expr = line.split(':')[1].strip()
+ expr = line.split(":")[1].strip()
# Remove trailing comment closure
- if line.strip().endswith('*/'):
- expr = expr.rstrip('*/').strip()
+ if line.strip().endswith("*/"):
+ expr = expr.rstrip("*/").strip()
# Remove trailing xml comment closure
- if line.strip().endswith('-->'):
- expr = expr.rstrip('-->').strip()
+ if line.strip().endswith("-->"):
+ expr = expr.rstrip("-->").strip()
# Special case for SH magic boot code files
- if line.startswith('LIST \"'):
- expr = expr.rstrip('\"').strip()
+ if line.startswith('LIST "'):
+ expr = expr.rstrip('"').strip()
self.parse(expr)
self.spdx_valid += 1
#
@@ -199,11 +220,14 @@ class id_parser(object):
if pe.tok:
col = line.find(expr) + pe.tok.lexpos
tok = pe.tok.value
- sys.stdout.write('%s: %d:%d %s: %s\n' %(fname, self.curline, col, pe.txt, tok))
+ sys.stdout.write(
+ "%s: %d:%d %s: %s\n" % (fname, self.curline, col, pe.txt, tok)
+ )
else:
- sys.stdout.write('%s: %d:0 %s\n' %(fname, self.curline, col, pe.txt))
+ sys.stdout.write("%s: %d:0 %s\n" % (fname, self.curline, col, pe.txt))
self.spdx_errors += 1
+
def scan_git_tree(tree):
for el in tree.traverse():
# Exclude stuff which would make pointless noise
@@ -214,25 +238,38 @@ def scan_git_tree(tree):
continue
if not os.path.isfile(el.path):
continue
- with open(el.path, 'rb') as fd:
+ with open(el.path, "rb") as fd:
parser.parse_lines(fd, args.maxlines, el.path)
+
def scan_git_subtree(tree, path):
- for p in path.strip('/').split('/'):
+ for p in path.strip("/").split("/"):
tree = tree[p]
scan_git_tree(tree)
-if __name__ == '__main__':
- ap = ArgumentParser(description='SPDX expression checker')
- ap.add_argument('path', nargs='*', help='Check path or file. If not given full git tree scan. For stdin use "-"')
- ap.add_argument('-m', '--maxlines', type=int, default=15,
- help='Maximum number of lines to scan in a file. Default 15')
- ap.add_argument('-v', '--verbose', action='store_true', help='Verbose statistics output')
+if __name__ == "__main__":
+
+ ap = ArgumentParser(description="SPDX expression checker")
+ ap.add_argument(
+ "path",
+ nargs="*",
+ help='Check path or file. If not given full git tree scan. For stdin use "-"',
+ )
+ ap.add_argument(
+ "-m",
+ "--maxlines",
+ type=int,
+ default=15,
+ help="Maximum number of lines to scan in a file. Default 15",
+ )
+ ap.add_argument(
+ "-v", "--verbose", action="store_true", help="Verbose statistics output"
+ )
args = ap.parse_args()
# Sanity check path arguments
- if '-' in args.path and len(args.path) > 1:
+ if "-" in args.path and len(args.path) > 1:
sys.stderr.write('stdin input "-" must be the only path argument\n')
sys.exit(1)
@@ -249,49 +286,49 @@ if __name__ == '__main__':
except SPDXException as se:
if se.el:
- sys.stderr.write('%s: %s\n' %(se.el.path, se.txt))
+ sys.stderr.write("%s: %s\n" % (se.el.path, se.txt))
else:
- sys.stderr.write('%s\n' %se.txt)
+ sys.stderr.write("%s\n" % se.txt)
sys.exit(1)
except Exception as ex:
- sys.stderr.write('FAIL: %s\n' %ex)
- sys.stderr.write('%s\n' %traceback.format_exc())
+ sys.stderr.write("FAIL: %s\n" % ex)
+ sys.stderr.write("%s\n" % traceback.format_exc())
sys.exit(1)
try:
- if len(args.path) and args.path[0] == '-':
- stdin = os.fdopen(sys.stdin.fileno(), 'rb')
- parser.parse_lines(stdin, args.maxlines, '-')
+ if len(args.path) and args.path[0] == "-":
+ stdin = os.fdopen(sys.stdin.fileno(), "rb")
+ parser.parse_lines(stdin, args.maxlines, "-")
else:
if args.path:
for p in args.path:
if os.path.isfile(p):
- parser.parse_lines(open(p, 'rb'), args.maxlines, p)
+ parser.parse_lines(open(p, "rb"), args.maxlines, p)
elif os.path.isdir(p):
scan_git_subtree(repo.head.reference.commit.tree, p)
else:
- sys.stderr.write('path %s does not exist\n' %p)
+ sys.stderr.write("path %s does not exist\n" % p)
sys.exit(1)
else:
# Full git tree scan
scan_git_tree(repo.head.commit.tree)
if args.verbose:
- sys.stderr.write('\n')
- sys.stderr.write('License files: %12d\n' %spdx.license_files)
- sys.stderr.write('Exception files: %12d\n' %spdx.exception_files)
- sys.stderr.write('License IDs %12d\n' %len(spdx.licenses))
- sys.stderr.write('Exception IDs %12d\n' %len(spdx.exceptions))
- sys.stderr.write('\n')
- sys.stderr.write('Files checked: %12d\n' %parser.checked)
- sys.stderr.write('Lines checked: %12d\n' %parser.lines_checked)
- sys.stderr.write('Files with SPDX: %12d\n' %parser.spdx_valid)
- sys.stderr.write('Files with errors: %12d\n' %parser.spdx_errors)
+ sys.stderr.write("\n")
+ sys.stderr.write("License files: %12d\n" % spdx.license_files)
+ sys.stderr.write("Exception files: %12d\n" % spdx.exception_files)
+ sys.stderr.write("License IDs %12d\n" % len(spdx.licenses))
+ sys.stderr.write("Exception IDs %12d\n" % len(spdx.exceptions))
+ sys.stderr.write("\n")
+ sys.stderr.write("Files checked: %12d\n" % parser.checked)
+ sys.stderr.write("Lines checked: %12d\n" % parser.lines_checked)
+ sys.stderr.write("Files with SPDX: %12d\n" % parser.spdx_valid)
+ sys.stderr.write("Files with errors: %12d\n" % parser.spdx_errors)
sys.exit(0)
except Exception as ex:
- sys.stderr.write('FAIL: %s\n' %ex)
- sys.stderr.write('%s\n' %traceback.format_exc())
+ sys.stderr.write("FAIL: %s\n" % ex)
+ sys.stderr.write("%s\n" % traceback.format_exc())
sys.exit(1)
diff --git a/scripts/update_ssh_config_guestfs.py b/scripts/update_ssh_config_guestfs.py
index 4d178d49..8b212a9c 100755
--- a/scripts/update_ssh_config_guestfs.py
+++ b/scripts/update_ssh_config_guestfs.py
@@ -30,6 +30,7 @@ ssh_template = """Host {name} {addr}
LogLevel FATAL
"""
+
# We take the first IPv4 address on the first non-loopback interface.
def get_addr(name):
attempt = 0
@@ -38,7 +39,15 @@ def get_addr(name):
if attempt > 60:
raise Exception(f"Unable to get an address for {name} after 60s")
- result = subprocess.run(['/usr/bin/virsh','qemu-agent-command',name,'{"execute":"guest-network-get-interfaces"}'], capture_output=True)
+ result = subprocess.run(
+ [
+ "/usr/bin/virsh",
+ "qemu-agent-command",
+ name,
+ '{"execute":"guest-network-get-interfaces"}',
+ ],
+ capture_output=True,
+ )
# Did it error out? Sleep and try again.
if result.returncode != 0:
time.sleep(1)
@@ -48,15 +57,15 @@ def get_addr(name):
netinfo = json.loads(result.stdout)
ret = None
- for iface in netinfo['return']:
- if iface['name'] == 'lo':
+ for iface in netinfo["return"]:
+ if iface["name"] == "lo":
continue
- if 'ip-addresses' not in iface:
+ if "ip-addresses" not in iface:
continue
- for addr in iface['ip-addresses']:
- if addr['ip-address-type'] != 'ipv4':
+ for addr in iface["ip-addresses"]:
+ if addr["ip-address-type"] != "ipv4":
continue
- ret = addr['ip-address']
+ ret = addr["ip-address"]
break
# If we didn't get an address, try again
@@ -64,11 +73,12 @@ def get_addr(name):
return ret
time.sleep(1)
+
def main():
- topdir = os.environ.get('TOPDIR', '.')
+ topdir = os.environ.get("TOPDIR", ".")
# load extra_vars
- with open(f'{topdir}/extra_vars.yaml') as stream:
+ with open(f"{topdir}/extra_vars.yaml") as stream:
extra_vars = yaml.safe_load(stream)
# slurp in the guestfs_nodes list
@@ -76,23 +86,28 @@ def main():
nodes = yaml.safe_load(stream)
if extra_vars.get("topdir_path_has_sha256sum", False):
- ssh_config = f'{Path.home()}/.ssh/config_kdevops_{extra_vars["topdir_path_sha256sum"]}'
+ ssh_config = (
+ f'{Path.home()}/.ssh/config_kdevops_{extra_vars["topdir_path_sha256sum"]}'
+ )
else:
- ssh_config = f'{Path.home()}/.ssh/config_kdevops_{extra_vars["kdevops_host_prefix"]}'
+ ssh_config = (
+ f'{Path.home()}/.ssh/config_kdevops_{extra_vars["kdevops_host_prefix"]}'
+ )
# make a stanza for each node
- sshconf = open(ssh_config, 'w')
- for node in nodes['guestfs_nodes']:
- name = node['name']
+ sshconf = open(ssh_config, "w")
+ for node in nodes["guestfs_nodes"]:
+ name = node["name"]
addr = get_addr(name)
context = {
- "name" : name,
- "addr" : addr,
- "sshkey" : f"{extra_vars['guestfs_path']}/{name}/ssh/id_ed25519"
+ "name": name,
+ "addr": addr,
+ "sshkey": f"{extra_vars['guestfs_path']}/{name}/ssh/id_ed25519",
}
sshconf.write(ssh_template.format(**context))
sshconf.close()
os.chmod(ssh_config, 0o600)
+
if __name__ == "__main__":
main()
diff --git a/scripts/workflows/blktests/blktests_watchdog.py b/scripts/workflows/blktests/blktests_watchdog.py
index 0cf7af98..2dd4444b 100755
--- a/scripts/workflows/blktests/blktests_watchdog.py
+++ b/scripts/workflows/blktests/blktests_watchdog.py
@@ -15,11 +15,14 @@ import configparser
import argparse
from itertools import chain
+
def print_blktest_host_status(host, verbose, basedir, config):
kernel = kssh.get_uname(host).rstrip()
section = blktests.get_section(host, config)
- (last_test, last_test_time, current_time_str, delta_seconds, stall_suspect) = blktests.get_blktest_host(host, basedir, kernel, section, config)
- checktime = blktests.get_last_run_time(host, basedir, kernel, section, last_test)
+ (last_test, last_test_time, current_time_str, delta_seconds, stall_suspect) = (
+ blktests.get_blktest_host(host, basedir, kernel, section, config)
+ )
+ checktime = blktests.get_last_run_time(host, basedir, kernel, section, last_test)
percent_done = 0
if checktime > 0:
@@ -38,12 +41,26 @@ def print_blktest_host_status(host, verbose, basedir, config):
sys.stdout.write("Last test : None\n")
else:
percent_done_str = "%.0f%%" % (0)
- sys.stdout.write("%35s%20s%20s%20s%20s%15s%30s\n" % (host, "None", percent_done_str, 0, 0, stall_str, kernel))
+ sys.stdout.write(
+ "%35s%20s%20s%20s%20s%15s%30s\n"
+ % (host, "None", percent_done_str, 0, 0, stall_str, kernel)
+ )
return
if not verbose:
percent_done_str = "%.0f%%" % (percent_done)
- sys.stdout.write("%35s%20s%20s%20s%20s%15s%30s\n" % (host, last_test, percent_done_str, str(delta_seconds), str(checktime), stall_str, kernel))
+ sys.stdout.write(
+ "%35s%20s%20s%20s%20s%15s%30s\n"
+ % (
+ host,
+ last_test,
+ percent_done_str,
+ str(delta_seconds),
+ str(checktime),
+ stall_str,
+ kernel,
+ )
+ )
return
sys.stdout.write("Host : %s\n" % (host))
@@ -62,23 +79,37 @@ def print_blktest_host_status(host, verbose, basedir, config):
sys.stdout.write("OK")
sys.stdout.write("\n")
+
def _main():
- parser = argparse.ArgumentParser(description='blktest-watchdog')
- parser.add_argument('hostfile', metavar='<ansible hostfile>', type=str,
- default='hosts',
- help='Ansible hostfile to use')
- parser.add_argument('hostsection', metavar='<ansible hostsection>', type=str,
- default='baseline',
- help='The name of the section to read hosts from')
- parser.add_argument('--verbose', const=True, default=False, action="store_const",
- help='Be verbose on otput.')
+ parser = argparse.ArgumentParser(description="blktest-watchdog")
+ parser.add_argument(
+ "hostfile",
+ metavar="<ansible hostfile>",
+ type=str,
+ default="hosts",
+ help="Ansible hostfile to use",
+ )
+ parser.add_argument(
+ "hostsection",
+ metavar="<ansible hostsection>",
+ type=str,
+ default="baseline",
+ help="The name of the section to read hosts from",
+ )
+ parser.add_argument(
+ "--verbose",
+ const=True,
+ default=False,
+ action="store_const",
+ help="Be verbose on otput.",
+ )
args = parser.parse_args()
if not os.path.isfile(args.hostfile):
sys.stdout.write("%s does not exist\n" % (args.hostfile))
sys.exit(1)
- dotconfig = os.path.dirname(os.path.abspath(args.hostfile)) + '/.config'
+ dotconfig = os.path.dirname(os.path.abspath(args.hostfile)) + "/.config"
config = blktests.get_config(dotconfig)
if not config:
sys.stdout.write("%s does not exist\n" % (dotconfig))
@@ -86,9 +117,21 @@ def _main():
basedir = os.path.dirname(dotconfig)
hosts = blktests.get_hosts(args.hostfile, args.hostsection)
- sys.stdout.write("%35s%20s%20s%20s%20s%15s%30s\n" % ("Hostname", "Test-name", "Completion %", "runtime(s)", "last-runtime(s)", "Stall-status", "Kernel"))
+ sys.stdout.write(
+ "%35s%20s%20s%20s%20s%15s%30s\n"
+ % (
+ "Hostname",
+ "Test-name",
+ "Completion %",
+ "runtime(s)",
+ "last-runtime(s)",
+ "Stall-status",
+ "Kernel",
+ )
+ )
for h in hosts:
print_blktest_host_status(h, args.verbose, basedir, config)
-if __name__ == '__main__':
+
+if __name__ == "__main__":
ret = _main()
diff --git a/scripts/workflows/cxl/gen_qemu_cxl.py b/scripts/workflows/cxl/gen_qemu_cxl.py
index 778567ea..92755f12 100755
--- a/scripts/workflows/cxl/gen_qemu_cxl.py
+++ b/scripts/workflows/cxl/gen_qemu_cxl.py
@@ -3,114 +3,213 @@
import argparse
import os
+
def qemu_print(kind, value, last=False):
global args
- if args.format == 'xml':
- print('<qemu:arg value=\'%s\'/>' % kind)
- print('<qemu:arg value=\'%s\'/>' % value)
+ if args.format == "xml":
+ print("<qemu:arg value='%s'/>" % kind)
+ print("<qemu:arg value='%s'/>" % value)
else:
- print('%s %s %s' % (kind, value, '' if last else '\\'))
+ print("%s %s %s" % (kind, value, "" if last else "\\"))
+
def host_bridge(hb_id, bus, addr):
- return 'pxb-cxl,bus=pcie.0,id=cxl.%d,bus_nr=0x%x,addr=0x%x' % (hb_id, bus, addr)
+ return "pxb-cxl,bus=pcie.0,id=cxl.%d,bus_nr=0x%x,addr=0x%x" % (hb_id, bus, addr)
+
def root_port(rp_id, hb_id, port, slot):
- return 'cxl-rp,port=%d,bus=cxl.%d,id=cxl_rp%d,chassis=0,slot=%d' % (port, hb_id, rp_id, slot)
+ return "cxl-rp,port=%d,bus=cxl.%d,id=cxl_rp%d,chassis=0,slot=%d" % (
+ port,
+ hb_id,
+ rp_id,
+ slot,
+ )
+
def switch(rp_id):
- return 'cxl-upstream,bus=cxl_rp%d,id=cxl_switch%d,addr=0.0,multifunction=on' % (rp_id, rp_id)
+ return "cxl-upstream,bus=cxl_rp%d,id=cxl_switch%d,addr=0.0,multifunction=on" % (
+ rp_id,
+ rp_id,
+ )
+
def mailbox(rp_id):
- return 'cxl-switch-mailbox-cci,bus=cxl_rp%d,addr=0.1,target=cxl_switch%d' % (rp_id, rp_id)
+ return "cxl-switch-mailbox-cci,bus=cxl_rp%d,addr=0.1,target=cxl_switch%d" % (
+ rp_id,
+ rp_id,
+ )
+
def downstream_port(dport_id, dport, rp_id, slot):
- return 'cxl-downstream,port=%d,bus=cxl_switch%d,id=cxl_dport%d,chassis=0,slot=%d' % (dport, rp_id, dport_id, slot)
+ return (
+ "cxl-downstream,port=%d,bus=cxl_switch%d,id=cxl_dport%d,chassis=0,slot=%d"
+ % (dport, rp_id, dport_id, slot)
+ )
+
def memdev(dport_id, path, sizestr, sizeval, create):
- filename = '%s/cxl_mem%d.raw' % (path, dport_id)
- if not(os.path.exists(filename)) and create:
- if not(os.path.exists(path)):
- print('ERROR: Tried to create memdev file but directory %s does not exist.' % path)
+ filename = "%s/cxl_mem%d.raw" % (path, dport_id)
+ if not (os.path.exists(filename)) and create:
+ if not (os.path.exists(path)):
+ print(
+ "ERROR: Tried to create memdev file but directory %s does not exist."
+ % path
+ )
exit(1)
os.umask(0)
- with open(filename, 'wb') as file:
+ with open(filename, "wb") as file:
file.truncate(sizeval)
- return 'memory-backend-file,id=cxl_memdev%d,share=on,mem-path=%s,size=%s' % (dport_id, filename, sizestr)
+ return "memory-backend-file,id=cxl_memdev%d,share=on,mem-path=%s,size=%s" % (
+ dport_id,
+ filename,
+ sizestr,
+ )
+
def lsa(path, create):
- filename = '%s/cxl_lsa.raw' % path
- if not(os.path.exists(filename)) and create:
- if not(os.path.exists(path)):
- print('ERROR: Tried to create lsa file but directory %s does not exist.' % path)
+ filename = "%s/cxl_lsa.raw" % path
+ if not (os.path.exists(filename)) and create:
+ if not (os.path.exists(path)):
+ print(
+ "ERROR: Tried to create lsa file but directory %s does not exist."
+ % path
+ )
exit(1)
os.umask(0)
- with open(filename, 'wb') as file:
+ with open(filename, "wb") as file:
file.truncate(256 * 1024 * 1024)
- return 'memory-backend-file,id=cxl_lsa,share=on,mem-path=%s,size=256M' % filename
+ return "memory-backend-file,id=cxl_lsa,share=on,mem-path=%s,size=256M" % filename
def type3(dport_id):
- return 'cxl-type3,bus=cxl_dport%d,memdev=cxl_memdev%d,lsa=cxl_lsa,id=cxl_mem%d' % (dport_id, dport_id, dport_id)
+ return "cxl-type3,bus=cxl_dport%d,memdev=cxl_memdev%d,lsa=cxl_lsa,id=cxl_mem%d" % (
+ dport_id,
+ dport_id,
+ dport_id,
+ )
+
def fmw(num_hb):
- s = ''
+ s = ""
for hb in range(num_hb):
- s += 'cxl-fmw.0.targets.%d=cxl.%d,' % (hb, hb)
- return s + 'cxl-fmw.0.size=8G,cxl-fmw.0.interleave-granularity=256'
-
-parser = argparse.ArgumentParser(description='QEMU CXL configuration generator', usage='%(prog)s [options]')
-parser.add_argument('-m', '--memdev-path', dest='memdev_path',
- help='Path to location of backing memdev files', required=True)
-parser.add_argument('-s', '--size', dest='size',
- help='Size of each memory device in bytes (i.e. 512M, 16G)', required=True)
-parser.add_argument('-c', '--create-memdev-files', dest='create_memdevs',
- help='Create memdev file if not found', action='store_true', default=False)
-parser.add_argument('-f', '--format', dest='format',
- help='Format of QEMU args',
- default='cmdline', choices=['cmdline', 'xml'])
-parser.add_argument('-b', '--host-bridges', dest='num_hb',
- help='Number of host bridges',
- type=int, default=1, choices=range(1,5))
-parser.add_argument('-r', '--root-ports', dest='num_rp',
- help='Number of root ports per host bridge',
- type=int, default=1, choices=range(1,5))
-parser.add_argument('-d', '--downstream-ports', dest='num_dport',
- help='Number of downstream ports per switch',
- type=int, default=1, choices=range(1,9))
-parser.add_argument('-p', '--pci-bus-number', dest='bus_nr',
- help='PCI bus number for first host bridge (default: 0x38)',
- type=int, default=0x38)
-parser.add_argument('--bus-alloc-per-host-bridge', dest='bus_per',
- help='Number of PCI buses to allocate per host bridge (default: 16)',
- type=int, default=16)
-parser.add_argument('--pci-function-number', dest='func_nr',
- help='Starting PCI function number for host bridges on root PCI bus 0 (default: 9)',
- type=int, default=9)
+ s += "cxl-fmw.0.targets.%d=cxl.%d," % (hb, hb)
+ return s + "cxl-fmw.0.size=8G,cxl-fmw.0.interleave-granularity=256"
+
+
+parser = argparse.ArgumentParser(
+ description="QEMU CXL configuration generator", usage="%(prog)s [options]"
+)
+parser.add_argument(
+ "-m",
+ "--memdev-path",
+ dest="memdev_path",
+ help="Path to location of backing memdev files",
+ required=True,
+)
+parser.add_argument(
+ "-s",
+ "--size",
+ dest="size",
+ help="Size of each memory device in bytes (i.e. 512M, 16G)",
+ required=True,
+)
+parser.add_argument(
+ "-c",
+ "--create-memdev-files",
+ dest="create_memdevs",
+ help="Create memdev file if not found",
+ action="store_true",
+ default=False,
+)
+parser.add_argument(
+ "-f",
+ "--format",
+ dest="format",
+ help="Format of QEMU args",
+ default="cmdline",
+ choices=["cmdline", "xml"],
+)
+parser.add_argument(
+ "-b",
+ "--host-bridges",
+ dest="num_hb",
+ help="Number of host bridges",
+ type=int,
+ default=1,
+ choices=range(1, 5),
+)
+parser.add_argument(
+ "-r",
+ "--root-ports",
+ dest="num_rp",
+ help="Number of root ports per host bridge",
+ type=int,
+ default=1,
+ choices=range(1, 5),
+)
+parser.add_argument(
+ "-d",
+ "--downstream-ports",
+ dest="num_dport",
+ help="Number of downstream ports per switch",
+ type=int,
+ default=1,
+ choices=range(1, 9),
+)
+parser.add_argument(
+ "-p",
+ "--pci-bus-number",
+ dest="bus_nr",
+ help="PCI bus number for first host bridge (default: 0x38)",
+ type=int,
+ default=0x38,
+)
+parser.add_argument(
+ "--bus-alloc-per-host-bridge",
+ dest="bus_per",
+ help="Number of PCI buses to allocate per host bridge (default: 16)",
+ type=int,
+ default=16,
+)
+parser.add_argument(
+ "--pci-function-number",
+ dest="func_nr",
+ help="Starting PCI function number for host bridges on root PCI bus 0 (default: 9)",
+ type=int,
+ default=9,
+)
args = parser.parse_args()
-suffix_dict = {'M': 1024 ** 2, 'G': 1024 ** 3}
+suffix_dict = {"M": 1024**2, "G": 1024**3}
suffix = args.size[-1].upper()
-if not(suffix in suffix_dict):
- print('ERROR: size must end in M (for MiB) or G (for GiB)')
+if not (suffix in suffix_dict):
+ print("ERROR: size must end in M (for MiB) or G (for GiB)")
exit(1)
size = int(args.size[:-1]) * suffix_dict[suffix]
slot = 0
-qemu_print('-machine', 'cxl=on')
-qemu_print('-object', lsa(args.memdev_path, args.create_memdevs))
+qemu_print("-machine", "cxl=on")
+qemu_print("-object", lsa(args.memdev_path, args.create_memdevs))
for hb in range(args.num_hb):
- qemu_print('-device', host_bridge(hb, args.bus_nr + hb * args.bus_per, hb + args.func_nr))
+ qemu_print(
+ "-device", host_bridge(hb, args.bus_nr + hb * args.bus_per, hb + args.func_nr)
+ )
for rp in range(args.num_rp):
rp_id = hb * args.num_rp + rp
- qemu_print('-device', root_port(rp_id, hb, rp, slot))
+ qemu_print("-device", root_port(rp_id, hb, rp, slot))
slot += 1
- qemu_print('-device', switch(rp_id))
- qemu_print('-device', mailbox(rp_id))
+ qemu_print("-device", switch(rp_id))
+ qemu_print("-device", mailbox(rp_id))
for dport in range(args.num_dport):
- dport_id = rp_id * args.num_dport + dport;
- qemu_print('-device', downstream_port(dport_id, dport, rp_id, slot))
+ dport_id = rp_id * args.num_dport + dport
+ qemu_print("-device", downstream_port(dport_id, dport, rp_id, slot))
slot += 1
- qemu_print('-object', memdev(dport_id, args.memdev_path, args.size, size, args.create_memdevs))
- qemu_print('-device', type3(dport_id))
-qemu_print('-M', fmw(args.num_hb), last=True)
+ qemu_print(
+ "-object",
+ memdev(
+ dport_id, args.memdev_path, args.size, size, args.create_memdevs
+ ),
+ )
+ qemu_print("-device", type3(dport_id))
+qemu_print("-M", fmw(args.num_hb), last=True)
diff --git a/scripts/workflows/fstests/fstests_watchdog.py b/scripts/workflows/fstests/fstests_watchdog.py
index 3fef5484..beb229e6 100755
--- a/scripts/workflows/fstests/fstests_watchdog.py
+++ b/scripts/workflows/fstests/fstests_watchdog.py
@@ -17,6 +17,7 @@ import configparser
import argparse
from itertools import chain
+
def print_fstest_host_status(host, verbose, use_remote, use_ssh, basedir, config):
if "CONFIG_DEVCONFIG_ENABLE_SYSTEMD_JOURNAL_REMOTE" in config and not use_ssh:
configured_kernel = None
@@ -35,8 +36,11 @@ def print_fstest_host_status(host, verbose, use_remote, use_ssh, basedir, config
kernel = kssh.get_uname(host).rstrip()
section = fstests.get_section(host, config)
- (last_test, last_test_time, current_time_str, delta_seconds, stall_suspect) = \
- fstests.get_fstest_host(use_remote, use_ssh, host, basedir, kernel, section, config)
+ (last_test, last_test_time, current_time_str, delta_seconds, stall_suspect) = (
+ fstests.get_fstest_host(
+ use_remote, use_ssh, host, basedir, kernel, section, config
+ )
+ )
checktime = fstests.get_checktime(host, basedir, kernel, section, last_test)
percent_done = (delta_seconds * 100 / checktime) if checktime > 0 else 0
@@ -49,10 +53,9 @@ def print_fstest_host_status(host, verbose, use_remote, use_ssh, basedir, config
stall_str = "Hung-Stalled"
crash_state = "OK"
- watchdog = KernelCrashWatchdog(host_name=host,
- decode_crash=True,
- reset_host=True,
- save_warnings=True)
+ watchdog = KernelCrashWatchdog(
+ host_name=host, decode_crash=True, reset_host=True, save_warnings=True
+ )
crash_file, warning_file = watchdog.check_and_reset_host()
if crash_file:
crash_state = "CRASH"
@@ -60,7 +63,9 @@ def print_fstest_host_status(host, verbose, use_remote, use_ssh, basedir, config
crash_state = "WARNING"
if not verbose:
- soak_duration_seconds = int(config.get("CONFIG_FSTESTS_SOAK_DURATION", '0').strip('"'))
+ soak_duration_seconds = int(
+ config.get("CONFIG_FSTESTS_SOAK_DURATION", "0").strip('"')
+ )
uses_soak = fstests.fstests_test_uses_soak_duration(last_test or "")
is_soaking = uses_soak and soak_duration_seconds != 0
soaking_str = "(soak)" if is_soaking else ""
@@ -83,30 +88,56 @@ def print_fstest_host_status(host, verbose, use_remote, use_ssh, basedir, config
sys.stdout.write("Delta: %d total second\n" % (delta_seconds))
sys.stdout.write("\t%d minutes\n" % (delta_seconds / 60))
sys.stdout.write("\t%d seconds\n" % (delta_seconds % 60))
- sys.stdout.write("Timeout-status: %s\n" % ("POSSIBLE-STALL" if stall_suspect else "OK"))
+ sys.stdout.write(
+ "Timeout-status: %s\n" % ("POSSIBLE-STALL" if stall_suspect else "OK")
+ )
sys.stdout.write("Crash-status : %s\n" % crash_state)
+
def _main():
- parser = argparse.ArgumentParser(description='fstest-watchdog')
- parser.add_argument('hostfile', metavar='<ansible hostfile>', type=str,
- default='hosts',
- help='Ansible hostfile to use')
- parser.add_argument('hostsection', metavar='<ansible hostsection>', type=str,
- default='baseline',
- help='The name of the section to read hosts from')
- parser.add_argument('--verbose', const=True, default=False, action="store_const",
- help='Be verbose on output.')
- parser.add_argument('--use-systemd-remote', const=True, default=True, action="store_const",
- help='Use systemd-remote uploaded journals if available')
- parser.add_argument('--use-ssh', const=True, default=False, action="store_const",
- help='Force to only use ssh for journals.')
+ parser = argparse.ArgumentParser(description="fstest-watchdog")
+ parser.add_argument(
+ "hostfile",
+ metavar="<ansible hostfile>",
+ type=str,
+ default="hosts",
+ help="Ansible hostfile to use",
+ )
+ parser.add_argument(
+ "hostsection",
+ metavar="<ansible hostsection>",
+ type=str,
+ default="baseline",
+ help="The name of the section to read hosts from",
+ )
+ parser.add_argument(
+ "--verbose",
+ const=True,
+ default=False,
+ action="store_const",
+ help="Be verbose on output.",
+ )
+ parser.add_argument(
+ "--use-systemd-remote",
+ const=True,
+ default=True,
+ action="store_const",
+ help="Use systemd-remote uploaded journals if available",
+ )
+ parser.add_argument(
+ "--use-ssh",
+ const=True,
+ default=False,
+ action="store_const",
+ help="Force to only use ssh for journals.",
+ )
args = parser.parse_args()
if not os.path.isfile(args.hostfile):
sys.stdout.write("%s does not exist\n" % (args.hostfile))
sys.exit(1)
- dotconfig = os.path.dirname(os.path.abspath(args.hostfile)) + '/.config'
+ dotconfig = os.path.dirname(os.path.abspath(args.hostfile)) + "/.config"
config = fstests.get_config(dotconfig)
if not config:
sys.stdout.write("%s does not exist\n" % (dotconfig))
@@ -119,11 +150,16 @@ def _main():
if group is not None:
remote_gid = group[2]
if remote_gid not in os.getgrouplist(os.getlogin(), os.getgid()):
- sys.stderr.write("Your username is not part of the group %s\n" % remote_group)
+ sys.stderr.write(
+ "Your username is not part of the group %s\n" % remote_group
+ )
sys.stderr.write("Fix this and try again")
sys.exit(1)
else:
- sys.stderr.write("The group %s was not found, add Kconfig support for the systemd-remote-journal group used" % remote_group)
+ sys.stderr.write(
+ "The group %s was not found, add Kconfig support for the systemd-remote-journal group used"
+ % remote_group
+ )
sys.exit(1)
hosts = fstests.get_hosts(args.hostfile, args.hostsection)
@@ -133,13 +169,13 @@ def _main():
f"{'Kernel':<38} {'Crash-status':<10}\n"
)
for h in hosts:
- print_fstest_host_status(h, args.verbose,
- args.use_systemd_remote,
- args.use_ssh,
- basedir,
- config)
+ print_fstest_host_status(
+ h, args.verbose, args.use_systemd_remote, args.use_ssh, basedir, config
+ )
- soak_duration_seconds = int(config.get("CONFIG_FSTESTS_SOAK_DURATION", '0').strip('"'))
+ soak_duration_seconds = int(
+ config.get("CONFIG_FSTESTS_SOAK_DURATION", "0").strip('"')
+ )
journal_method = "ssh"
if "CONFIG_DEVCONFIG_ENABLE_SYSTEMD_JOURNAL_REMOTE" in config and not args.use_ssh:
journal_method = "systemd-journal-remote"
@@ -147,5 +183,6 @@ def _main():
sys.stdout.write("\n%25s%20s\n" % ("Journal-method", "Soak-duration(s)"))
sys.stdout.write("%25s%20d\n" % (journal_method, soak_duration_seconds))
-if __name__ == '__main__':
+
+if __name__ == "__main__":
ret = _main()
diff --git a/scripts/workflows/generic/crash_report.py b/scripts/workflows/generic/crash_report.py
index 10b4f958..65a05343 100755
--- a/scripts/workflows/generic/crash_report.py
+++ b/scripts/workflows/generic/crash_report.py
@@ -104,6 +104,8 @@ def generate_commit_log():
if __name__ == "__main__":
if not CRASH_DIR.exists():
- print(f"No crashes, filesystem corruption isues, or kernel warnings were detected on this run.")
+ print(
+ f"No crashes, filesystem corruption isues, or kernel warnings were detected on this run."
+ )
exit(0)
generate_commit_log()
diff --git a/scripts/workflows/generic/crash_watchdog.py b/scripts/workflows/generic/crash_watchdog.py
index f271cf9c..15e92513 100755
--- a/scripts/workflows/generic/crash_watchdog.py
+++ b/scripts/workflows/generic/crash_watchdog.py
@@ -24,13 +24,16 @@ logging.basicConfig(
)
logger = logging.getLogger("crash_watchdog")
+
def get_active_hosts():
"""Get the list of active hosts from kdevops configuration."""
try:
# First try to get the hosts from the ansible inventory
result = subprocess.run(
["ansible-inventory", "-i", "hosts", "--list"],
- capture_output=True, text=True, check=True
+ capture_output=True,
+ text=True,
+ check=True,
)
inventory = yaml.safe_load(result.stdout)
hosts = inventory.get("baseline", {}).get("hosts", [])
@@ -39,6 +42,7 @@ def get_active_hosts():
logger.error(f"Error getting active hosts: {e}")
return []
+
def run_crash_watchdog_on_host(args, this_host_name):
watchdog = KernelCrashWatchdog(
host_name=this_host_name,
@@ -46,13 +50,15 @@ def run_crash_watchdog_on_host(args, this_host_name):
full_log=args.full_log,
decode_crash=not args.no_decode,
reset_host=not args.no_reset,
- save_warnings = args.save_warnings,
+ save_warnings=args.save_warnings,
)
crashed = False
warnings_found = False
- crash_file, warning_file = watchdog.check_and_reset_host(method=args.method, get_fstests_log=args.fstests_log)
+ crash_file, warning_file = watchdog.check_and_reset_host(
+ method=args.method, get_fstests_log=args.fstests_log
+ )
if warning_file:
logger.warning(f"Kernel warning and logged to {warning_file}")
@@ -66,6 +72,7 @@ def run_crash_watchdog_on_host(args, this_host_name):
logger.debug(f"No crash detected for host {this_host_name}")
return crashed, [crash_file], warnings_found, warning_file
+
def run_crash_watchdog_all_hosts(args):
"""Check all active hosts for kernel crashes."""
hosts = get_active_hosts()
@@ -74,12 +81,12 @@ def run_crash_watchdog_all_hosts(args):
warnings_detected = False
warning_files = []
- logger.info(
- f"Checking {len(hosts)} hosts for kernel crashes: {', '.join(hosts)}"
- )
+ logger.info(f"Checking {len(hosts)} hosts for kernel crashes: {', '.join(hosts)}")
for host in hosts:
- host_crash_detected, crash_file, host_warnings_detected, warnings_file = run_crash_watchdog_on_host(args, host)
+ host_crash_detected, crash_file, host_warnings_detected, warnings_file = (
+ run_crash_watchdog_on_host(args, host)
+ )
if host_crash_detected and crash_file:
crash_detected = True
crash_files.append(crash_file)
@@ -87,10 +94,13 @@ def run_crash_watchdog_all_hosts(args):
if host_warnings_detected and warnings_file:
warnings_detected = True
warning_files.append(warning_file)
- logger.warning(f"Kernel warning found on host {host}, logs saved to {warning_file}")
+ logger.warning(
+ f"Kernel warning found on host {host}, logs saved to {warning_file}"
+ )
return crash_detected, crash_files, warnings_detected, warning_files
+
def write_log_section(f, title, files, label):
f.write(f"# {title}\n\n")
for path in files:
@@ -102,6 +112,7 @@ def write_log_section(f, title, files, label):
except Exception as e:
f.write(f"\nError reading {label.lower()} file: {e}\n\n")
+
def main():
parser = argparse.ArgumentParser(
description="Detect and handle kernel crashes or kernel warnings in hosts.",
@@ -138,22 +149,45 @@ Examples:
Get all kernel warnings only:
./crash_watchdog.py e3-ext4-2k --method remote --save-warnings sad.warn
""",
- formatter_class=argparse.RawTextHelpFormatter
+ formatter_class=argparse.RawTextHelpFormatter,
)
- parser.add_argument("--host-name", help="Optional name of the host to check", default="all")
- parser.add_argument("--output-dir", help="Directory to store crash logs", default="crashes")
+ parser.add_argument(
+ "--host-name", help="Optional name of the host to check", default="all"
+ )
+ parser.add_argument(
+ "--output-dir", help="Directory to store crash logs", default="crashes"
+ )
parser.add_argument(
"--method",
choices=["auto", "remote", "console", "ssh"],
default="auto",
- help="Choose method to collect logs: auto, remote, console, or ssh"
+ help="Choose method to collect logs: auto, remote, console, or ssh",
+ )
+ parser.add_argument(
+ "--full-log",
+ action="store_true",
+ help="Get full kernel log instead of only crash context",
+ )
+ parser.add_argument(
+ "--no-decode",
+ action="store_true",
+ help="Disable decoding crash logs with decode_stacktrace.sh",
+ )
+ parser.add_argument(
+ "--no-reset",
+ action="store_true",
+ help="Do not reset the guest even if a crash is detected",
+ )
+ parser.add_argument(
+ "--fstests-log",
+ help="Show all kernel log lines for a specific fstests test ID (e.g., generic/750)",
+ )
+ parser.add_argument(
+ "--save-warnings",
+ help="Do you want detected and save kernel warnings",
+ default=True,
)
- parser.add_argument("--full-log", action="store_true", help="Get full kernel log instead of only crash context")
- parser.add_argument("--no-decode", action="store_true", help="Disable decoding crash logs with decode_stacktrace.sh")
- parser.add_argument("--no-reset", action="store_true", help="Do not reset the guest even if a crash is detected")
- parser.add_argument("--fstests-log", help="Show all kernel log lines for a specific fstests test ID (e.g., generic/750)")
- parser.add_argument("--save-warnings", help="Do you want detected and save kernel warnings", default=True)
args = parser.parse_args()
crash_files = []
warnings_files = []
@@ -164,10 +198,14 @@ Examples:
args.save_warnings = False
args.full_log_mode = True
- if (args.host_name != "all"):
- crash_detected, crash_files, warnings_detected, warnings_files = run_crash_watchdog_on_host(args, args.host_name)
+ if args.host_name != "all":
+ crash_detected, crash_files, warnings_detected, warnings_files = (
+ run_crash_watchdog_on_host(args, args.host_name)
+ )
else:
- crash_detected, crash_files, warnings_detected, warnings_files = run_crash_watchdog_all_hosts(args)
+ crash_detected, crash_files, warnings_detected, warnings_files = (
+ run_crash_watchdog_all_hosts(args)
+ )
if warnings_detected:
logger.warning("Kernel warnings detected in one or more hosts")
diff --git a/scripts/workflows/lib/blktests.py b/scripts/workflows/lib/blktests.py
index 6f156854..7945c750 100644
--- a/scripts/workflows/lib/blktests.py
+++ b/scripts/workflows/lib/blktests.py
@@ -7,18 +7,21 @@ import argparse
import re
from itertools import chain
+
class BlktestsError(Exception):
pass
+
def blktests_check_pid(host):
pid = kssh.first_process_name_pid(host, "check")
if pid <= 0:
return pid
- dir = "/proc/" + str(pid) + "/cwd/tests"
+ dir = "/proc/" + str(pid) + "/cwd/tests"
if kssh.dir_exists(host, dir):
return pid
return 0
+
def get_blktest_host(host, basedir, kernel, section, config):
stall_suspect = False
if kernel == "Uname-issue":
@@ -39,7 +42,7 @@ def get_blktest_host(host, basedir, kernel, section, config):
last_test_time = latest_dmesg_blktest_line.split("at ")[1].rstrip()
current_time_str = kssh.get_current_time(host).rstrip()
- blktests_date_str_format = '%Y-%m-%d %H:%M:%S'
+ blktests_date_str_format = "%Y-%m-%d %H:%M:%S"
d1 = datetime.strptime(last_test_time, blktests_date_str_format)
d2 = datetime.strptime(current_time_str, blktests_date_str_format)
@@ -49,20 +52,26 @@ def get_blktest_host(host, basedir, kernel, section, config):
if "CONFIG_BLKTESTS_WATCHDOG" not in config:
enable_watchdog = False
else:
- enable_watchdog = config["CONFIG_BLKTESTS_WATCHDOG"].strip('\"')
+ enable_watchdog = config["CONFIG_BLKTESTS_WATCHDOG"].strip('"')
if enable_watchdog:
- max_new_test_time = config["CONFIG_BLKTESTS_WATCHDOG_MAX_NEW_TEST_TIME"].strip('\"')
+ max_new_test_time = config["CONFIG_BLKTESTS_WATCHDOG_MAX_NEW_TEST_TIME"].strip(
+ '"'
+ )
max_new_test_time = int(max_new_test_time)
if not max_new_test_time:
max_new_test_time = 60
- hung_multiplier_long_tests = config["CONFIG_BLKTESTS_WATCHDOG_HUNG_MULTIPLIER_LONG_TESTS"].strip('\"')
+ hung_multiplier_long_tests = config[
+ "CONFIG_BLKTESTS_WATCHDOG_HUNG_MULTIPLIER_LONG_TESTS"
+ ].strip('"')
hung_multiplier_long_tests = int(hung_multiplier_long_tests)
if not hung_multiplier_long_tests:
hung_multiplier_long_tests = 10
- hung_fast_test_max_time = config["CONFIG_BLKTESTS_WATCHDOG_HUNG_FAST_TEST_MAX_TIME"].strip('\"')
+ hung_fast_test_max_time = config[
+ "CONFIG_BLKTESTS_WATCHDOG_HUNG_FAST_TEST_MAX_TIME"
+ ].strip('"')
hung_fast_test_max_time = int(hung_fast_test_max_time)
if not hung_fast_test_max_time:
hung_fast_test_max_time = 5
@@ -83,16 +92,21 @@ def get_blktest_host(host, basedir, kernel, section, config):
# If a test typically takes between 1 second to 30 seconds we can likely
# safely assume the system has crashed after hung_fast_test_max_time
# minutes
- elif last_run_time_s > 0:
+ elif last_run_time_s > 0:
suspect_crash_time_seconds = 60 * hung_fast_test_max_time
- if delta_seconds >= suspect_crash_time_seconds and 'blktestsstart/000' not in last_test and 'blktestsend/000' not in last_test:
+ if (
+ delta_seconds >= suspect_crash_time_seconds
+ and "blktestsstart/000" not in last_test
+ and "blktestsend/000" not in last_test
+ ):
stall_suspect = True
return (last_test, last_test_time, current_time_str, delta_seconds, stall_suspect)
+
def get_last_run_time(host, basedir, kernel, section, last_test):
- results_dir = basedir + '/workflows/blktests/results/last-run/'
+ results_dir = basedir + "/workflows/blktests/results/last-run/"
if not last_test:
return 0
if not os.path.isdir(results_dir):
@@ -115,7 +129,7 @@ def get_last_run_time(host, basedir, kernel, section, last_test):
break
if not ok_file:
return 0
- f = open(ok_file, 'r')
+ f = open(ok_file, "r")
for line in f:
if not "runtime" in line:
continue
@@ -129,21 +143,28 @@ def get_last_run_time(host, basedir, kernel, section, last_test):
return float(time_string_elems[0])
return 0
+
def get_config(dotconfig):
- config = configparser.ConfigParser(allow_no_value=True, strict=False, interpolation=None)
+ config = configparser.ConfigParser(
+ allow_no_value=True, strict=False, interpolation=None
+ )
with open(dotconfig) as lines:
lines = chain(("[top]",), lines)
config.read_file(lines)
return config["top"]
return None
+
def get_section(host, config):
- hostprefix = config["CONFIG_KDEVOPS_HOSTS_PREFIX"].strip('\"')
+ hostprefix = config["CONFIG_KDEVOPS_HOSTS_PREFIX"].strip('"')
return host.split(hostprefix + "-")[1].replace("-", "_")
+
def get_hosts(hostfile, hostsection):
hosts = []
- config = configparser.ConfigParser(allow_no_value=True, strict=False, interpolation=None)
+ config = configparser.ConfigParser(
+ allow_no_value=True, strict=False, interpolation=None
+ )
config.read(hostfile)
if hostsection not in config:
return hosts
diff --git a/scripts/workflows/lib/crash.py b/scripts/workflows/lib/crash.py
index d9d121d1..5397d784 100755
--- a/scripts/workflows/lib/crash.py
+++ b/scripts/workflows/lib/crash.py
@@ -262,7 +262,7 @@ class KernelCrashWatchdog:
save_warnings=False,
context_prefix=0,
context_postfix=35,
- ssh_timeout = 180,
+ ssh_timeout=180,
):
self.host_name = host_name
self.output_dir = os.path.join(output_dir, host_name)
@@ -411,7 +411,7 @@ class KernelCrashWatchdog:
if not key_log_line:
logger.warning(f"Error getting key log line for {file_path}")
- continue;
+ continue
# Use the first relevant line for any context
log_hash = hashlib.md5(key_log_line.encode()).hexdigest()
self.known_crashes.append(log_hash)
@@ -551,9 +551,7 @@ class KernelCrashWatchdog:
seconds = float(match.group(1))
wall_time = boot_time + timedelta(seconds=seconds)
timestamp = wall_time.strftime("%b %d %H:%M:%S")
- converted_lines.append(
- f"{timestamp} {self.host_name} {match.group(2)}"
- )
+ converted_lines.append(f"{timestamp} {self.host_name} {match.group(2)}")
else:
converted_lines.append(line)
@@ -780,7 +778,9 @@ class KernelCrashWatchdog:
)
logger.info(f"{self.host_name} is now reachable.")
except subprocess.TimeoutExpired:
- logger.error(f"Timeout: SSH connection to {self.host_name} did not succeed within {self.ssh_timeout} seconds. This kernel is probably seriously broken.")
+ logger.error(
+ f"Timeout: SSH connection to {self.host_name} did not succeed within {self.ssh_timeout} seconds. This kernel is probably seriously broken."
+ )
sys.exit(1)
except subprocess.CalledProcessError as e:
logger.warning(f"Failed to wait for SSH on {self.host_name}: {e}")
diff --git a/scripts/workflows/lib/fstests.py b/scripts/workflows/lib/fstests.py
index 328277a9..ee3fbe45 100644
--- a/scripts/workflows/lib/fstests.py
+++ b/scripts/workflows/lib/fstests.py
@@ -7,71 +7,75 @@ import configparser
import argparse
from itertools import chain
+
class FstestsError(Exception):
pass
+
def fstests_check_pid(host):
pid = kssh.first_process_name_pid(host, "check")
if pid <= 0:
return pid
- dir = "/proc/" + str(pid) + "/cwd/tests"
+ dir = "/proc/" + str(pid) + "/cwd/tests"
if kssh.dir_exists(host, dir):
return pid
return 0
+
# Later on we can automate this list with a git grep on the fstests
# tests/ directory, and we inject this here.
def fstests_test_uses_soak_duration(testname):
- USES_SOAK_DURATION = [ "generic/019" ]
- USES_SOAK_DURATION += [ "generic/388" ]
- USES_SOAK_DURATION += [ "generic/475" ]
- USES_SOAK_DURATION += [ "generic/476" ]
- USES_SOAK_DURATION += [ "generic/521" ]
- USES_SOAK_DURATION += [ "generic/522" ]
- USES_SOAK_DURATION += [ "generic/616" ]
- USES_SOAK_DURATION += [ "generic/617" ]
- USES_SOAK_DURATION += [ "generic/642" ]
- USES_SOAK_DURATION += [ "generic/650" ]
- USES_SOAK_DURATION += [ "generic/648" ]
- USES_SOAK_DURATION += [ "xfs/285" ]
- USES_SOAK_DURATION += [ "xfs/517" ]
- USES_SOAK_DURATION += [ "xfs/560" ]
- USES_SOAK_DURATION += [ "xfs/561" ]
- USES_SOAK_DURATION += [ "xfs/562" ]
- USES_SOAK_DURATION += [ "xfs/565" ]
- USES_SOAK_DURATION += [ "xfs/570" ]
- USES_SOAK_DURATION += [ "xfs/571" ]
- USES_SOAK_DURATION += [ "xfs/572" ]
- USES_SOAK_DURATION += [ "xfs/573" ]
- USES_SOAK_DURATION += [ "xfs/574" ]
- USES_SOAK_DURATION += [ "xfs/575" ]
- USES_SOAK_DURATION += [ "xfs/576" ]
- USES_SOAK_DURATION += [ "xfs/577" ]
- USES_SOAK_DURATION += [ "xfs/578" ]
- USES_SOAK_DURATION += [ "xfs/579" ]
- USES_SOAK_DURATION += [ "xfs/580" ]
- USES_SOAK_DURATION += [ "xfs/581" ]
- USES_SOAK_DURATION += [ "xfs/582" ]
- USES_SOAK_DURATION += [ "xfs/583" ]
- USES_SOAK_DURATION += [ "xfs/584" ]
- USES_SOAK_DURATION += [ "xfs/585" ]
- USES_SOAK_DURATION += [ "xfs/586" ]
- USES_SOAK_DURATION += [ "xfs/587" ]
- USES_SOAK_DURATION += [ "xfs/588" ]
- USES_SOAK_DURATION += [ "xfs/589" ]
- USES_SOAK_DURATION += [ "xfs/590" ]
- USES_SOAK_DURATION += [ "xfs/591" ]
- USES_SOAK_DURATION += [ "xfs/592" ]
- USES_SOAK_DURATION += [ "xfs/593" ]
- USES_SOAK_DURATION += [ "xfs/594" ]
- USES_SOAK_DURATION += [ "xfs/595" ]
- USES_SOAK_DURATION += [ "xfs/727" ]
- USES_SOAK_DURATION += [ "xfs/729" ]
- USES_SOAK_DURATION += [ "xfs/800" ]
+ USES_SOAK_DURATION = ["generic/019"]
+ USES_SOAK_DURATION += ["generic/388"]
+ USES_SOAK_DURATION += ["generic/475"]
+ USES_SOAK_DURATION += ["generic/476"]
+ USES_SOAK_DURATION += ["generic/521"]
+ USES_SOAK_DURATION += ["generic/522"]
+ USES_SOAK_DURATION += ["generic/616"]
+ USES_SOAK_DURATION += ["generic/617"]
+ USES_SOAK_DURATION += ["generic/642"]
+ USES_SOAK_DURATION += ["generic/650"]
+ USES_SOAK_DURATION += ["generic/648"]
+ USES_SOAK_DURATION += ["xfs/285"]
+ USES_SOAK_DURATION += ["xfs/517"]
+ USES_SOAK_DURATION += ["xfs/560"]
+ USES_SOAK_DURATION += ["xfs/561"]
+ USES_SOAK_DURATION += ["xfs/562"]
+ USES_SOAK_DURATION += ["xfs/565"]
+ USES_SOAK_DURATION += ["xfs/570"]
+ USES_SOAK_DURATION += ["xfs/571"]
+ USES_SOAK_DURATION += ["xfs/572"]
+ USES_SOAK_DURATION += ["xfs/573"]
+ USES_SOAK_DURATION += ["xfs/574"]
+ USES_SOAK_DURATION += ["xfs/575"]
+ USES_SOAK_DURATION += ["xfs/576"]
+ USES_SOAK_DURATION += ["xfs/577"]
+ USES_SOAK_DURATION += ["xfs/578"]
+ USES_SOAK_DURATION += ["xfs/579"]
+ USES_SOAK_DURATION += ["xfs/580"]
+ USES_SOAK_DURATION += ["xfs/581"]
+ USES_SOAK_DURATION += ["xfs/582"]
+ USES_SOAK_DURATION += ["xfs/583"]
+ USES_SOAK_DURATION += ["xfs/584"]
+ USES_SOAK_DURATION += ["xfs/585"]
+ USES_SOAK_DURATION += ["xfs/586"]
+ USES_SOAK_DURATION += ["xfs/587"]
+ USES_SOAK_DURATION += ["xfs/588"]
+ USES_SOAK_DURATION += ["xfs/589"]
+ USES_SOAK_DURATION += ["xfs/590"]
+ USES_SOAK_DURATION += ["xfs/591"]
+ USES_SOAK_DURATION += ["xfs/592"]
+ USES_SOAK_DURATION += ["xfs/593"]
+ USES_SOAK_DURATION += ["xfs/594"]
+ USES_SOAK_DURATION += ["xfs/595"]
+ USES_SOAK_DURATION += ["xfs/727"]
+ USES_SOAK_DURATION += ["xfs/729"]
+ USES_SOAK_DURATION += ["xfs/800"]
if testname in USES_SOAK_DURATION:
return True
return False
+
def get_fstest_host(use_remote, use_ssh, host, basedir, kernel, section, config):
stall_suspect = False
force_ssh = False
@@ -111,7 +115,7 @@ def get_fstest_host(use_remote, use_ssh, host, basedir, kernel, section, config)
else:
current_time_str = systemd_remote.get_current_time(host).rstrip()
- fstests_date_str_format = '%Y-%m-%d %H:%M:%S'
+ fstests_date_str_format = "%Y-%m-%d %H:%M:%S"
d1 = datetime.strptime(last_test_time, fstests_date_str_format)
d2 = datetime.strptime(current_time_str, fstests_date_str_format)
@@ -120,31 +124,37 @@ def get_fstest_host(use_remote, use_ssh, host, basedir, kernel, section, config)
soak_duration_seconds = 0
if "CONFIG_FSTESTS_SOAK_DURATION" in config:
- soak_duration_seconds = config["CONFIG_FSTESTS_SOAK_DURATION"].strip('\"')
+ soak_duration_seconds = config["CONFIG_FSTESTS_SOAK_DURATION"].strip('"')
soak_duration_seconds = int(soak_duration_seconds)
if "CONFIG_FSTESTS_WATCHDOG" not in config:
enable_watchdog = False
else:
- enable_watchdog = config["CONFIG_FSTESTS_WATCHDOG"].strip('\"')
+ enable_watchdog = config["CONFIG_FSTESTS_WATCHDOG"].strip('"')
if enable_watchdog:
- max_new_test_time = config["CONFIG_FSTESTS_WATCHDOG_MAX_NEW_TEST_TIME"].strip('\"')
+ max_new_test_time = config["CONFIG_FSTESTS_WATCHDOG_MAX_NEW_TEST_TIME"].strip(
+ '"'
+ )
max_new_test_time = int(max_new_test_time)
if not max_new_test_time:
max_new_test_time = 60
- hung_multiplier_long_tests = config["CONFIG_FSTESTS_WATCHDOG_HUNG_MULTIPLIER_LONG_TESTS"].strip('\"')
+ hung_multiplier_long_tests = config[
+ "CONFIG_FSTESTS_WATCHDOG_HUNG_MULTIPLIER_LONG_TESTS"
+ ].strip('"')
hung_multiplier_long_tests = int(hung_multiplier_long_tests)
if not hung_multiplier_long_tests:
hung_multiplier_long_tests = 10
- hung_fast_test_max_time = config["CONFIG_FSTESTS_WATCHDOG_HUNG_FAST_TEST_MAX_TIME"].strip('\"')
+ hung_fast_test_max_time = config[
+ "CONFIG_FSTESTS_WATCHDOG_HUNG_FAST_TEST_MAX_TIME"
+ ].strip('"')
hung_fast_test_max_time = int(hung_fast_test_max_time)
if not hung_fast_test_max_time:
hung_fast_test_max_time = 5
- checktime = get_checktime(host, basedir, kernel, section, last_test)
+ checktime = get_checktime(host, basedir, kernel, section, last_test)
# If no known prior run time test is known we use a max. This only
# applies to the first run.
@@ -160,23 +170,37 @@ def get_fstest_host(use_remote, use_ssh, host, basedir, kernel, section, config)
# If a test typically takes between 1 second to 30 seconds we can likely
# safely assume the system has crashed after hung_fast_test_max_time
# minutes
- elif checktime > 0:
+ elif checktime > 0:
suspect_crash_time_seconds = 60 * hung_fast_test_max_time
if fstests_test_uses_soak_duration(last_test):
suspect_crash_time_seconds += soak_duration_seconds
- if delta_seconds >= suspect_crash_time_seconds and 'fstestsstart/000' not in last_test and 'fstestsend/000' not in last_test:
+ if (
+ delta_seconds >= suspect_crash_time_seconds
+ and "fstestsstart/000" not in last_test
+ and "fstestsend/000" not in last_test
+ ):
stall_suspect = True
return (last_test, last_test_time, current_time_str, delta_seconds, stall_suspect)
+
def get_checktime(host, basedir, kernel, section, last_test):
- checktime_dir = basedir + '/workflows/fstests/results/' + host + '/' + kernel + '/' + section + '/'
- checktime_file = checktime_dir + 'check.time'
+ checktime_dir = (
+ basedir
+ + "/workflows/fstests/results/"
+ + host
+ + "/"
+ + kernel
+ + "/"
+ + section
+ + "/"
+ )
+ checktime_file = checktime_dir + "check.time"
if not os.path.isfile(checktime_file):
return 0
- cp = open(checktime_file, 'r')
+ cp = open(checktime_file, "r")
for line in cp:
elems = line.rstrip().split(" ")
this_test = elems[0].rstrip().replace(" ", "")
@@ -184,21 +208,28 @@ def get_checktime(host, basedir, kernel, section, last_test):
return int(elems[1])
return 0
+
def get_config(dotconfig):
- config = configparser.ConfigParser(allow_no_value=True, strict=False, interpolation=None)
+ config = configparser.ConfigParser(
+ allow_no_value=True, strict=False, interpolation=None
+ )
with open(dotconfig) as lines:
lines = chain(("[top]",), lines)
config.read_file(lines)
return config["top"]
return None
+
def get_section(host, config):
- hostprefix = config["CONFIG_KDEVOPS_HOSTS_PREFIX"].strip('\"')
+ hostprefix = config["CONFIG_KDEVOPS_HOSTS_PREFIX"].strip('"')
return host.split(hostprefix + "-")[1].replace("-", "_")
+
def get_hosts(hostfile, hostsection):
hosts = []
- config = configparser.ConfigParser(allow_no_value=True, strict=False, interpolation=None)
+ config = configparser.ConfigParser(
+ allow_no_value=True, strict=False, interpolation=None
+ )
config.read(hostfile)
if hostsection not in config:
return hosts
diff --git a/scripts/workflows/lib/kssh.py b/scripts/workflows/lib/kssh.py
index 11472767..a9b4b29d 100644
--- a/scripts/workflows/lib/kssh.py
+++ b/scripts/workflows/lib/kssh.py
@@ -2,28 +2,36 @@
import subprocess, os
+
class KsshError(Exception):
pass
+
+
class ExecutionError(KsshError):
def __init__(self, errcode):
self.error_code = errcode
+
+
class TimeoutExpired(KsshError):
def __init__(self, errcode):
self.error_code = errcode
return "timeout"
+
def _check(process):
if process.returncode != 0:
raise ExecutionError(process.returncode)
+
def dir_exists(host, dirname):
- cmd = ['ssh', host,
- 'sudo',
- 'ls', '-ld',
- dirname ]
- process = subprocess.Popen(cmd,
- stdout=subprocess.PIPE, stderr=subprocess.STDOUT,
- close_fds=True, universal_newlines=True)
+ cmd = ["ssh", host, "sudo", "ls", "-ld", dirname]
+ process = subprocess.Popen(
+ cmd,
+ stdout=subprocess.PIPE,
+ stderr=subprocess.STDOUT,
+ close_fds=True,
+ universal_newlines=True,
+ )
data = None
try:
data = process.communicate(timeout=120)
@@ -37,17 +45,35 @@ def dir_exists(host, dirname):
else:
return False
+
def first_process_name_pid(host, process_name):
- cmd = ['ssh', host,
- 'sudo',
- 'ps', '-ef',
- '|', 'grep', '-v', 'grep',
- '|', 'grep', process_name,
- '|', 'awk', '\'{print $2}\'',
- '|', 'tail', '-1' ]
- process = subprocess.Popen(cmd,
- stdout=subprocess.PIPE, stderr=subprocess.STDOUT,
- close_fds=True, universal_newlines=True)
+ cmd = [
+ "ssh",
+ host,
+ "sudo",
+ "ps",
+ "-ef",
+ "|",
+ "grep",
+ "-v",
+ "grep",
+ "|",
+ "grep",
+ process_name,
+ "|",
+ "awk",
+ "'{print $2}'",
+ "|",
+ "tail",
+ "-1",
+ ]
+ process = subprocess.Popen(
+ cmd,
+ stdout=subprocess.PIPE,
+ stderr=subprocess.STDOUT,
+ close_fds=True,
+ universal_newlines=True,
+ )
data = None
try:
data = process.communicate(timeout=120)
@@ -62,14 +88,16 @@ def first_process_name_pid(host, process_name):
return 0
return int(stdout)
+
def prog_exists(host, prog):
- cmd = ['ssh', host,
- 'sudo',
- 'which',
- prog ]
- process = subprocess.Popen(cmd,
- stdout=subprocess.PIPE, stderr=subprocess.STDOUT,
- close_fds=True, universal_newlines=True)
+ cmd = ["ssh", host, "sudo", "which", prog]
+ process = subprocess.Popen(
+ cmd,
+ stdout=subprocess.PIPE,
+ stderr=subprocess.STDOUT,
+ close_fds=True,
+ universal_newlines=True,
+ )
data = None
try:
data = process.communicate(timeout=120)
@@ -82,11 +110,16 @@ def prog_exists(host, prog):
return False
return True
+
def get_uname(host):
- cmd = ['ssh', host, 'uname', '-r' ]
- process = subprocess.Popen(cmd,
- stdout=subprocess.PIPE, stderr=subprocess.STDOUT,
- close_fds=True, universal_newlines=True)
+ cmd = ["ssh", host, "uname", "-r"]
+ process = subprocess.Popen(
+ cmd,
+ stdout=subprocess.PIPE,
+ stderr=subprocess.STDOUT,
+ close_fds=True,
+ universal_newlines=True,
+ )
data = None
try:
data = process.communicate(timeout=120)
@@ -99,28 +132,49 @@ def get_uname(host):
return "Uname-issue"
return stdout
+
def get_test(host, suite):
- if suite not in [ 'fstests', 'blktests']:
+ if suite not in ["fstests", "blktests"]:
return None
run_string = "run " + suite
- cmd = ['ssh', host,
- 'sudo',
- 'dmesg',
- '|', 'grep', '"' + run_string + '"',
- '|', 'awk', '-F"' + run_string + ' "', '\'{print $2}\'',
- '|', 'tail', '-1' ]
- if prog_exists(host, 'journalctl'):
- cmd = ['ssh', host,
- 'sudo',
- 'journalctl',
- '-k',
- '-g'
- '"' + run_string + '"'
- '|', 'awk', '-F"' + run_string + ' "', '\'{print $2}\'',
- '|', 'tail', '-1' ]
- process = subprocess.Popen(cmd,
- stdout=subprocess.PIPE, stderr=subprocess.STDOUT,
- close_fds=True, universal_newlines=True)
+ cmd = [
+ "ssh",
+ host,
+ "sudo",
+ "dmesg",
+ "|",
+ "grep",
+ '"' + run_string + '"',
+ "|",
+ "awk",
+ '-F"' + run_string + ' "',
+ "'{print $2}'",
+ "|",
+ "tail",
+ "-1",
+ ]
+ if prog_exists(host, "journalctl"):
+ cmd = [
+ "ssh",
+ host,
+ "sudo",
+ "journalctl",
+ "-k",
+ "-g" '"' + run_string + '"' "|",
+ "awk",
+ '-F"' + run_string + ' "',
+ "'{print $2}'",
+ "|",
+ "tail",
+ "-1",
+ ]
+ process = subprocess.Popen(
+ cmd,
+ stdout=subprocess.PIPE,
+ stderr=subprocess.STDOUT,
+ close_fds=True,
+ universal_newlines=True,
+ )
data = None
try:
data = process.communicate(timeout=120)
@@ -139,19 +193,33 @@ def get_test(host, suite):
return stdout
+
def get_last_fstest(host):
- return get_test(host, 'fstests')
+ return get_test(host, "fstests")
+
def get_last_blktest(host):
- return get_test(host, 'blktests')
+ return get_test(host, "blktests")
+
def get_current_time(host):
- cmd = ['ssh', host,
- 'date', '--rfc-3339=\'seconds\'',
- '|', 'awk', '-F"+"', '\'{print $1}\'' ]
- process = subprocess.Popen(cmd,
- stdout=subprocess.PIPE, stderr=subprocess.STDOUT,
- close_fds=True, universal_newlines=True)
+ cmd = [
+ "ssh",
+ host,
+ "date",
+ "--rfc-3339='seconds'",
+ "|",
+ "awk",
+ '-F"+"',
+ "'{print $1}'",
+ ]
+ process = subprocess.Popen(
+ cmd,
+ stdout=subprocess.PIPE,
+ stderr=subprocess.STDOUT,
+ close_fds=True,
+ universal_newlines=True,
+ )
data = None
try:
data = process.communicate(timeout=120)
diff --git a/scripts/workflows/lib/systemd_remote.py b/scripts/workflows/lib/systemd_remote.py
index c95ca7e5..1a65de9f 100644
--- a/scripts/workflows/lib/systemd_remote.py
+++ b/scripts/workflows/lib/systemd_remote.py
@@ -3,19 +3,27 @@
import subprocess, os, sys
from datetime import datetime
+
class SystemdError(Exception):
pass
+
+
class ExecutionError(SystemdError):
def __init__(self, errcode):
self.error_code = errcode
+
+
class TimeoutExpired(SystemdError):
def __init__(self, errcode):
self.error_code = errcode
return "timeout"
+
def get_host_ip(host):
try:
- result = subprocess.run(["ssh", "-G", host], capture_output=True, text=True, check=True)
+ result = subprocess.run(
+ ["ssh", "-G", host], capture_output=True, text=True, check=True
+ )
for line in result.stdout.splitlines():
if line.startswith("hostname "):
return line.split()[1]
@@ -23,15 +31,17 @@ def get_host_ip(host):
logger.warning(f"Failed to resolve IP for {host}: {e}")
return None
+
def get_current_time(host):
- format = '%Y-%m-%d %H:%M:%S'
+ format = "%Y-%m-%d %H:%M:%S"
today = datetime.today()
today_str = today.strftime(format)
return today_str
+
def get_extra_journals(remote_path, host):
ip = get_host_ip(host)
- extra_journals_path = "remote-" + ip + '@'
+ extra_journals_path = "remote-" + ip + "@"
extra_journals = []
for file in os.listdir(remote_path):
if extra_journals_path in file:
@@ -39,37 +49,33 @@ def get_extra_journals(remote_path, host):
extra_journals.append(remote_path + file)
return extra_journals
+
def get_uname(remote_path, host, configured_kernel):
ip = get_host_ip(host)
extra_journals = get_extra_journals(remote_path, host)
- fpath = remote_path + "remote-" + ip + '.journal'
+ fpath = remote_path + "remote-" + ip + ".journal"
grep = "Linux version"
- grep_str = "\"Linux version\""
- cmd = [
- 'journalctl',
- '--no-pager',
- '-n 1',
- '-k',
- '-g',
- grep,
- '--file',
- fpath ]
+ grep_str = '"Linux version"'
+ cmd = ["journalctl", "--no-pager", "-n 1", "-k", "-g", grep, "--file", fpath]
cmd = cmd + extra_journals
cmd_verbose = [
- 'journalctl',
- '--no-pager',
- '-n 1',
- '-k',
- '-g',
- grep_str,
- '--file',
- fpath ]
+ "journalctl",
+ "--no-pager",
+ "-n 1",
+ "-k",
+ "-g",
+ grep_str,
+ "--file",
+ fpath,
+ ]
cmd_verbose = cmd_verbose + extra_journals
- process = subprocess.Popen(cmd,
- stdout=subprocess.PIPE,
- stderr=subprocess.STDOUT,
- close_fds=True,
- universal_newlines=True)
+ process = subprocess.Popen(
+ cmd,
+ stdout=subprocess.PIPE,
+ stderr=subprocess.STDOUT,
+ close_fds=True,
+ universal_newlines=True,
+ )
data = None
try:
data = process.communicate(timeout=120)
@@ -88,42 +94,41 @@ def get_uname(remote_path, host, configured_kernel):
sys.stderr.write("\nCommand used:\n%s\n\n" % " ".join(cmd_verbose))
return None
if len(last_line.split(grep)) <= 1:
- sys.stderr.write("\nThe string %s could not be used to split the line." % grep_str)
+ sys.stderr.write(
+ "\nThe string %s could not be used to split the line." % grep_str
+ )
sys.stderr.write("\nCommand used:\n%s\n\n" % " ".join(cmd_verbose))
return None
kernel_line = last_line.split(grep)[1].strip()
if len(last_line.split()) <= 1:
- sys.stderr.write("\nThe string %s was used but could not find kernel version." % grep_str)
+ sys.stderr.write(
+ "\nThe string %s was used but could not find kernel version." % grep_str
+ )
sys.stderr.write("\nCommand used:\n%s\n\n" % " ".join(cmd_verbose))
return None
kernel = kernel_line.split()[0].strip()
return kernel
+
# Returns something like "xfs/040 at 2023-12-17 23:52:14"
def get_test(remote_path, host, suite):
ip = get_host_ip(host)
- if suite not in [ 'fstests', 'blktests']:
+ if suite not in ["fstests", "blktests"]:
return None
# Example: /var/log/journal/remote/remote-line-xfs-reflink.journal
- fpath = remote_path + "remote-" + ip + '.journal'
+ fpath = remote_path + "remote-" + ip + ".journal"
extra_journals = get_extra_journals(remote_path, host)
run_string = "run " + suite
- cmd = [
- 'journalctl',
- '--no-pager',
- '-n 1',
- '-k',
- '-g',
- run_string,
- '--file',
- fpath ]
+ cmd = ["journalctl", "--no-pager", "-n 1", "-k", "-g", run_string, "--file", fpath]
cmd = cmd + extra_journals
- process = subprocess.Popen(cmd,
- stdout=subprocess.PIPE,
- stderr=subprocess.STDOUT,
- close_fds=True,
- universal_newlines=True)
+ process = subprocess.Popen(
+ cmd,
+ stdout=subprocess.PIPE,
+ stderr=subprocess.STDOUT,
+ close_fds=True,
+ universal_newlines=True,
+ )
data = None
try:
data = process.communicate(timeout=120)
@@ -142,8 +147,10 @@ def get_test(remote_path, host, suite):
return test_line
+
def get_last_fstest(remote_path, host):
- return get_test(remote_path, host, 'fstests')
+ return get_test(remote_path, host, "fstests")
+
def get_last_blktest(remote_path, host):
- return get_test(remote_path, host, 'blktests')
+ return get_test(remote_path, host, "blktests")
diff --git a/scripts/workflows/pynfs/check_pynfs_results.py b/scripts/workflows/pynfs/check_pynfs_results.py
index 724d2ace..753d177f 100755
--- a/scripts/workflows/pynfs/check_pynfs_results.py
+++ b/scripts/workflows/pynfs/check_pynfs_results.py
@@ -12,20 +12,21 @@ import json
import sys
import pprint
+
def main():
base = json.load(open(sys.argv[1]))
result = json.load(open(sys.argv[2]))
failures = {}
- for case in result['testcase']:
- if 'failure' in case:
- failures[case['code']] = case
+ for case in result["testcase"]:
+ if "failure" in case:
+ failures[case["code"]] = case
- for case in base['testcase']:
- if 'failure' in case:
- if case['code'] in failures:
- del failures[case['code']]
+ for case in base["testcase"]:
+ if "failure" in case:
+ if case["code"] in failures:
+ del failures[case["code"]]
if len(failures) != 0:
pprint.pprint(failures)
@@ -33,6 +34,6 @@ def main():
else:
sys.exit(0)
+
if __name__ == "__main__":
main()
-
--
2.47.2
next prev parent reply other threads:[~2025-07-30 6:01 UTC|newest]
Thread overview: 19+ messages / expand[flat|nested] mbox.gz Atom feed top
2025-07-30 6:01 [PATCH v2 0/9] kdevops: add support for A/B testing Luis Chamberlain
2025-07-30 6:01 ` [PATCH v2 1/9] roles/guestfs: add missing bootlinux_9p: False Luis Chamberlain
2025-07-30 14:17 ` Chuck Lever
2025-07-30 6:01 ` [PATCH v2 2/9] Makefile: suppress Ansible warnings during configuration generation Luis Chamberlain
2025-07-30 6:22 ` Daniel Gomez
2025-07-30 6:01 ` [PATCH v2 3/9] playbooks: few space cleanups Luis Chamberlain
2025-07-30 6:01 ` [PATCH v2 4/9] style: add extensive code formatting checks to make style Luis Chamberlain
2025-07-30 6:01 ` [PATCH v2 5/9] Makefile: move styling to scripts/style.Makefile Luis Chamberlain
2025-07-30 6:01 ` [PATCH v2 6/9] CLAUDE.md: add instrucitons to verify commit Luis Chamberlain
2025-07-30 6:01 ` Luis Chamberlain [this message]
2025-07-31 12:57 ` [PATCH v2 7/9] all: run black Daniel Gomez
2025-08-01 8:12 ` Daniel Gomez
2025-08-01 12:55 ` Chuck Lever
2025-08-01 16:29 ` Daniel Gomez
2025-08-01 16:55 ` Chuck Lever
2025-07-30 6:01 ` [PATCH v2 8/9] devconfig: add automatic APT mirror fallback for Debian testing Luis Chamberlain
2025-07-30 6:41 ` Daniel Gomez
2025-08-01 17:39 ` Luis Chamberlain
2025-07-30 6:01 ` [PATCH v2 9/9] bootlinux: add support for A/B kernel testing Luis Chamberlain
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20250730060147.182140-8-mcgrof@kernel.org \
--to=mcgrof@kernel.org \
--cc=cel@kernel.org \
--cc=da.gomez@kruces.com \
--cc=kdevops@lists.linux.dev \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.