* [igt-dev] [PATCH i-g-t v3 1/1] scripts:igt-doc.py: add a parser to document tests inlined
2023-02-14 12:35 [igt-dev] [PATCH i-g-t v3 0/1] Add a script to allow inlined test documentation Mauro Carvalho Chehab
@ 2023-02-14 12:35 ` Mauro Carvalho Chehab
[not found] ` <20230221181517.pgt2apt7mczaikmx@ldmartin-desk2.jf.intel.com>
1 sibling, 0 replies; 3+ messages in thread
From: Mauro Carvalho Chehab @ 2023-02-14 12:35 UTC (permalink / raw)
To: igt-dev
From: Mauro Carvalho Chehab <mchehab@kernel.org>
Tests need to be documentation, as otherwise its goal will be
lost with time. Keeping documentation out of the sources is also
not such a good idea, as they tend to bitrot.
So, add a script to allow keeping the documentation inlined, and
add tools to verify if the documentation has gaps.
Signed-off-by: Mauro Carvalho Chehab <mchehab@kernel.org>
---
v3:
- Added MIT license;
- Some cleanups on its comments;
- Use enumerate() and remove fileinput import;
- Print an error if the field is invalid;
- Kept just one form (either singular or plural) to fields;
- Added a more comprehensive python documentation for the class,
explaining how wildcards are processed;
- Some cosmetic changes;
- Fixed some typos at fields array.
v2:
- Convert to Python
---
scripts/igt_doc.py | 575 +++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 575 insertions(+)
create mode 100755 scripts/igt_doc.py
diff --git a/scripts/igt_doc.py b/scripts/igt_doc.py
new file mode 100755
index 000000000000..b7e3129d3f54
--- /dev/null
+++ b/scripts/igt_doc.py
@@ -0,0 +1,575 @@
+#!/usr/bin/env python3
+# pylint: disable=C0301,R0914,R0912,R0915
+# SPDX-License-Identifier: (GPL-2.0 OR MIT)
+
+## Copyright (C) 2023 Intel Corporation ##
+## Author: Mauro Carvalho Chehab <mchehab@kernel.org> ##
+## ##
+## Allow keeping inlined test documentation and validate ##
+## if the documentation is kept updated. ##
+
+"""Maintain test plan and test implementation documentation on IGT."""
+
+import argparse
+import re
+import subprocess
+import sys
+
+IGT_BUILD_PATH = 'build/'
+IGT_RUNNER = 'runner/igt_runner'
+
+# Fields that mat be inside TEST and SUBTEST macros
+fields = [
+ 'Category', # Hardware building block / Software building block / ...
+ 'Sub-category', # waitfence / dmabuf/ sysfs / debugfs / ...
+ 'Functionality', # basic test / ...
+ 'Test category', # functionality test / pereformance / stress
+ 'Run type', # BAT / workarouds / stress / developer-specific / ...
+ 'Issue', # Bug tracker issue(s)
+ 'GPU excluded platforms', # none / DG1 / DG2 / TGL / MTL / PVC / ATS-M / ...
+ 'GPU requirements', # Any other specific platform requirements
+ 'Depends on', # some other IGT test, like igt@test@subtes
+ 'Test requirements', # other non-platform requirements
+ 'Description'] # Description of the test
+
+#
+# TestList class definition
+#
+class TestList:
+
+ """
+ Parse and handle test lists with test/subtest documentation, in the
+ form of C comments, with two meta-tags (TEST and SUBTEST), and a set of
+ `field: value` items:
+
+ /**
+ * TEST: Check if new IGT test documentation logic functionality is working
+ * Category: Software build block
+ * Sub-category: documentation
+ * Functionality: test documentation
+ * Test category: ReST generation
+ * Run type: IGT kunit test
+ * Issue: none
+ * GPU excluded platforms: none
+ * GPU requirements: none
+ * Depends on: @igt@deadbeef@basic
+ * Test requirements: Need python3 to run it
+ * Description: Complete description of this test
+ *
+ * SUBTEST: foo
+ * Description: do foo things
+ * description continuing on another line
+ *
+ * SUBTEST: bar
+ * Description: do bar things
+ * description continuing on another line
+ */
+
+ /**
+ * SUBTEST: test-%s-binds-%s-with-%ld-size
+ * Description: Test arg[2] arg[1] binds with arg[3] size
+ *
+ * SUBTEST: test-%s-%ld-size
+ * Description: Test arg[1] with %arg[2] size
+ *
+ * arg[1]:
+ *
+ * @large: large
+ * something
+ * @mixed: mixed
+ * something
+ * @small: small
+ * something
+ *
+ * arg[2]: buffer size
+ * of some type
+ *
+ * arg[3]:
+ *
+ * @binds: foo
+ * @misaligned-binds: misaligned
+ * @userptr-binds: userptr
+ * @userptr-misaligned-binds: userptr misaligned
+ */
+
+ The valid `field` values are defined on a JSON configuration file.
+
+ The TEST documents the common fields present on all tests. Typically,
+ each IGT C test file should contain just one such field.
+
+ The SUBTEST contains the fields that are specific to each subtest.
+
+ Some IGT tests may have strings or integer wildcards, like:
+ test-%s-%ld-size
+
+ Those are declared using igt_subtest_f() and similar macros.
+
+ The wildcard arguments there need to be expanded. This is done by
+ defining arg[1] to arg[n] at the same code comment that contains the
+ SUBTEST as those variables are locally processed on each comment line.
+ """
+
+
+ def __init__(self):
+ self.doc = {}
+ self.test_number = 0
+ self.min_test_prefix = ''
+
+ #
+ # ancillary methods
+ #
+
+ def expand_subtest(self, fname, test_name, test):
+
+ """Expand subtest wildcards providing an array with subtests"""
+
+ subtest_array = []
+
+ for subtest in self.doc[test]["subtest"].keys():
+ summary = test_name + '@' + self.doc[test]["subtest"][subtest]["Summary"]
+
+ if not summary:
+ continue
+
+ num_vars = summary.count('%')
+
+ # Handle trivial case: no wildcards
+ if num_vars == 0:
+ subtest_dict = {}
+
+ subtest_dict["Summary"] = summary
+
+ for k in sorted(self.doc[test]["subtest"][subtest].keys()):
+ if k == 'Summary':
+ continue
+ if k == 'arg':
+ continue
+
+ if self.doc[test]["subtest"][subtest][k] == self.doc[test][k]:
+ continue
+
+ subtest_dict[k] = self.doc[test]["subtest"][subtest][k]
+ subtest_array.append(subtest_dict)
+
+ continue
+
+ # Handle summaries with wildcards
+
+ # Convert subtest arguments into an array
+ arg_array = {}
+ arg_ref = self.doc[test]["subtest"][subtest]["arg"]
+
+ for arg_k in self.doc[test]["arg"][arg_ref].keys():
+ arg_array[arg_k] = []
+ if int(arg_k) > num_vars:
+ continue
+
+ for arg_el in sorted(self.doc[test]["arg"][arg_ref][arg_k].keys()):
+ arg_array[arg_k].append(arg_el)
+
+ size = len(arg_array)
+
+ if size < num_vars:
+ sys.exit(f"{fname}:subtest {summary} needs {num_vars} arguments but only {size} are defined.")
+
+ for j in range(0, num_vars):
+ if arg_array[j] is None:
+ sys.exit(f"{fname}:subtest{summary} needs arg[{j}], but this is not defined.")
+
+
+ # convert numeric wildcards to string ones
+ summary = re.sub(r'%(d|ld|lld|i|u|lu|llu)','%s', summary)
+
+ # Expand subtests
+ pos = [ 0 ] * num_vars
+ args = [ 0 ] * num_vars
+ arg_map = [ 0 ] * num_vars
+ while 1:
+ for j in range(0, num_vars):
+ arg_val = arg_array[j][pos[j]]
+ args[j] = arg_val
+
+ if arg_val in self.doc[test]["arg"][arg_ref][j]:
+ arg_map[j] = self.doc[test]["arg"][arg_ref][j][arg_val]
+ if re.match(r"\<.*\>", self.doc[test]["arg"][arg_ref][j][arg_val]):
+ args[j] = "<" + arg_val + ">"
+ else:
+ arg_map[j] = arg_val
+
+ arg_summary = summary % tuple(args)
+
+ # Store the element
+ subtest_dict = {}
+ subtest_dict["Summary"] = arg_summary
+
+ for field in sorted(self.doc[test]["subtest"][subtest].keys()):
+ if field == 'Summary':
+ continue
+ if field == 'arg':
+ continue
+
+ sub_field = self.doc[test]["subtest"][subtest][field]
+ sub_field = re.sub(r"%?\barg\[(\d+)\]", lambda m: arg_map[int(m.group(1)) - 1], sub_field) # pylint: disable=W0640
+ if sub_field == self.doc[test][field]:
+ continue
+
+ subtest_dict[field] = sub_field
+
+ subtest_array.append(subtest_dict)
+
+ # Increment variable inside the array
+ arr_pos = 0
+ while pos[arr_pos] + 1 >= len(arg_array[arr_pos]):
+ pos[arr_pos] = 0
+ arr_pos += 1
+ if arr_pos >= num_vars:
+ break
+
+ if arr_pos >= num_vars:
+ break
+
+ pos[arr_pos] += 1
+
+ return subtest_array
+
+ #
+ # Output methods
+ #
+
+ def print_test(self):
+
+ """Print tests and subtests ordered by tests"""
+
+ for test in sorted(self.doc.keys()):
+ fname = self.doc[test]["File"]
+
+ name = re.sub(r'.*tests/', '', fname)
+ name = re.sub(r'\.[ch]', '', name)
+ name = "igt@" + name
+
+ print(len(name) * '=')
+ print(name)
+ print(len(name) * '=')
+ print()
+
+ for field in sorted(self.doc[test].keys()):
+ if field == "subtest":
+ continue
+ if field == "arg":
+ continue
+
+ print(f":{field}: {self.doc[test][field]}")
+
+ subtest_array = self.expand_subtest(fname, name, test)
+
+ for subtest in subtest_array:
+ print()
+ print(subtest["Summary"])
+ print(len(subtest["Summary"]) * '=')
+ print("")
+
+ for field in sorted(subtest.keys()):
+ if field == 'Summary':
+ continue
+ if field == 'arg':
+ continue
+
+ print(f":{field}:", subtest[field])
+
+ print()
+
+ print()
+ print()
+
+ #
+ # Subtest list methods
+ #
+
+ def get_subtests(self):
+
+ """Return an array with all subtests"""
+
+ subtests = []
+
+ for test in sorted(self.doc.keys()):
+ fname = self.doc[test]["File"]
+
+ test_name = re.sub(r'.*tests/', '', fname)
+ test_name = re.sub(r'\.[ch]', '', test_name)
+ test_name = "igt@" + test_name
+
+ subtest_array = self.expand_subtest(fname, test_name, test)
+
+ for subtest in subtest_array:
+ subtests.append(subtest["Summary"])
+
+ return subtests
+
+ #
+ # Validation methods
+ #
+ def check_tests(self):
+
+ """Compare documented subtests with the IGT test list"""
+
+ doc_subtests = sorted(self.get_subtests())
+
+ for i in range(0, len(doc_subtests)): # pylint: disable=C0200
+ doc_subtests[i] = re.sub(r'\<[^\>]+\>', r'\\d+', doc_subtests[i])
+
+ # Get a list of tests from
+ result = subprocess.run([ f"{IGT_BUILD_PATH}/{IGT_RUNNER}", # pylint: disable=W1510
+ "-L", "-t", self.min_test_prefix,
+ f"{IGT_BUILD_PATH}/tests"],
+ capture_output = True, text = True)
+ if result.returncode:
+ print( result.stdout)
+ print("Error:", result.stderr)
+ sys.exit(result.returncode)
+
+ run_subtests = sorted(result.stdout.splitlines())
+
+ # Compare arrays
+
+ run_missing = []
+ doc_uneeded = []
+
+ for doc_test in doc_subtests:
+ found = False
+ for run_test in run_subtests:
+ if re.match(r'^' + doc_test + r'$', run_test):
+ found = True
+ break
+ if not found:
+ doc_uneeded.append(doc_test)
+
+ for run_test in run_subtests:
+ found = False
+ for doc_test in doc_subtests:
+ if re.match(r'^' + doc_test + r'$', run_test):
+ found = True
+ break
+ if not found:
+ run_missing.append(run_test)
+
+ if doc_uneeded:
+ print("Unused documentation")
+ for test_name in doc_uneeded:
+ print(test_name)
+
+ if run_missing:
+ if doc_uneeded:
+ print()
+ print("Missing documentation")
+ for test_name in run_missing:
+ print(test_name)
+
+ #
+ # File handling methods
+ #
+
+ def add_file_documentation(self, fname, field_re):
+
+ """Adds the contents of test/subtest documentation form a file"""
+
+ current_test = None
+ current_subtest = None
+
+ handle_section = ''
+ current_field = ''
+ arg_number = 1
+ cur_arg = -1
+ cur_arg_element = 0
+
+ prefix = re.sub(r'.*tests/', '', fname)
+ prefix = r'igt\@' + re.sub(r'(.*/).*', r'\1', prefix)
+
+ if self.min_test_prefix == '':
+ self.min_test_prefix = prefix
+ elif len(prefix) < len(self.min_test_prefix):
+ self.min_test_prefix = prefix
+
+ with open(fname, 'r', encoding='utf8') as handle:
+ arg_ref = None
+ current_test = ''
+ subtest_number = 0
+
+ for file_ln,file_line in enumerate(handle):
+ if re.match(r'^\s*\*$', file_line):
+ continue
+
+ if re.match(r'^\s*\*/$', file_line):
+ handle_section = ''
+ current_subtest = None
+ arg_ref = None
+ cur_arg = -1
+
+ continue
+
+ if re.match(r'^\s*/\*\*$', file_line):
+ handle_section = '1'
+ continue
+
+ if not handle_section:
+ continue
+
+ file_line = re.sub(r'^\s*\*\s*', '', file_line)
+
+ # Handle only known sections
+ if handle_section == '1':
+ current_field = ''
+
+ # Check if it is a new TEST section
+ if (match := re.match(r'^TEST:\s*(.*)', file_line)):
+ current_test = self.test_number
+ self.test_number += 1
+
+ handle_section = 'test'
+
+ self.doc[current_test] = {}
+ self.doc[current_test]["arg"] = {}
+ self.doc[current_test]["Summary"] = match.group(1)
+ self.doc[current_test]["File"] = fname
+ self.doc[current_test]["subtest"] = {}
+ current_subtest = None
+
+ continue
+
+ # Check if it is a new SUBTEST section
+ if (match := re.match(r'^SUBTESTS?:\s*(.*)', file_line)):
+ current_subtest = subtest_number
+ subtest_number += 1
+
+ current_field = ''
+ handle_section = 'subtest'
+
+ self.doc[current_test]["subtest"][current_subtest] = {}
+
+ self.doc[current_test]["subtest"][current_subtest]["Summary"] = match.group(1)
+ self.doc[current_test]["subtest"][current_subtest]["Description"] = ''
+
+ if not arg_ref:
+ arg_ref = arg_number
+ arg_number += 1
+ self.doc[current_test]["arg"][arg_ref] = {}
+
+ self.doc[current_test]["subtest"][current_subtest]["arg"] = arg_ref
+
+ continue
+
+ # It is a known section. Parse its contents
+ if (match := re.match(field_re, file_line)):
+ current_field = match.group(1).lower().capitalize()
+ match_val = match.group(2)
+
+ if handle_section == 'test':
+ self.doc[current_test][current_field] = match_val
+ else:
+ self.doc[current_test]["subtest"][current_subtest][current_field] = match_val
+
+ cur_arg = -1
+
+ continue
+
+ # Use hashes for arguments to avoid duplication
+ if (match := re.match(r'arg\[(\d+)\]:\s*(.*)', file_line)):
+ current_field = ''
+ if arg_ref is None:
+ sys.exit(f"{fname}:{file_ln + 1}: arguments should be defined after one or more subtests, at the same comment")
+
+ cur_arg = int(match.group(1)) - 1
+ if cur_arg not in self.doc[current_test]["arg"][arg_ref]:
+ self.doc[current_test]["arg"][arg_ref][cur_arg] = {}
+
+ cur_arg_element = match.group(2)
+
+ if match.group(2):
+ # Should be used only for numeric values
+ self.doc[current_test]["arg"][arg_ref][cur_arg][cur_arg_element] = "<" + match.group(2) + ">"
+
+ continue
+
+ if (match := re.match(r'\@(\S+):\s*(.*)', file_line)):
+ if cur_arg >= 0:
+ current_field = ''
+ if arg_ref is None:
+ sys.exit(f"{fname}:{file_ln + 1}: arguments should be defined after one or more subtests, at the same comment")
+
+ cur_arg_element = match.group(1)
+ self.doc[current_test]["arg"][arg_ref][cur_arg][cur_arg_element] = match.group(2)
+
+ else:
+ print(f"{fname}:{file_ln + 1}: Warning: invalid argument: @%s: %s" %
+ (match.group(1), match.group(2)))
+
+ continue
+
+ if (match := re.match(r'^(.*):', file_line)):
+ sys.exit(f"{fname}:{file_ln + 1}: Error: unrecognized field '%s'. Need to add at %s" %
+ (match.group(1), fname))
+
+ # Handle multi-line field contents
+ if current_field:
+ if (match := re.match(r'\s*(.*)', file_line)):
+ if handle_section == 'test':
+ self.doc[current_test][current_field] += " " + \
+ match.group(1)
+ else:
+ self.doc[current_test]["subtest"][current_subtest][current_field] += " " + \
+ match.group(1)
+
+ continue
+
+ # Handle multi-line argument contents
+ if cur_arg >= 0 and arg_ref is not None:
+ if (match := re.match(r'\s*\*?\s*(.*)', file_line)):
+ match_val = match.group(1)
+
+ match = re.match(r'^(\<.*)\>$',self.doc[current_test]["arg"][arg_ref][cur_arg][cur_arg_element])
+ if match:
+ self.doc[current_test]["arg"][arg_ref][cur_arg][cur_arg_element] = match.group(1) + ' ' + match_val + ">"
+ else:
+ self.doc[current_test]["arg"][arg_ref][cur_arg][cur_arg_element] += ' ' + match_val
+
+ continue
+
+#
+# Main
+#
+
+parser = argparse.ArgumentParser(description = "Print formatted kernel documentation to stdout.",
+ formatter_class = argparse.ArgumentDefaultsHelpFormatter,
+ epilog = 'If no action specified, assume --rest.')
+parser.add_argument("--rest", action="store_true",
+ help="Generate documentation from the source files, in ReST file format.")
+parser.add_argument("--show-subtests", action="store_true",
+ help="Shows the name of the documented subtests in alphabetical order.")
+parser.add_argument("--check-testlist", action="store_true",
+ help="Compare documentation against IGT runner testlist.")
+parser.add_argument("--igt-build-path",
+ help="Path where the IGT runner is sitting. Used by --check-testlist.",
+ default=IGT_BUILD_PATH)
+parser.add_argument('--files', nargs='+', required=True,
+ help="File name(s) to be processed")
+
+parse_args = parser.parse_args()
+
+field_regex = re.compile(r"(" + '|'.join(fields) + r'):\s*(.*)', re.I)
+
+tests = TestList()
+
+for filename in parse_args.files:
+ tests.add_file_documentation(filename, field_regex)
+
+RUN = 0
+if parse_args.show_subtests:
+ RUN = 1
+ for sub in tests.get_subtests():
+ print (sub)
+
+if parse_args.check_testlist:
+ RUN = 1
+ tests.check_tests()
+
+if not RUN or parse_args.rest:
+ tests.print_test()
--
2.39.0
^ permalink raw reply related [flat|nested] 3+ messages in thread