[igt-dev] [PATCH i-g-t 0/7] Improve scripts/igt_doc.py script

[igt-dev] [PATCH i-g-t 0/7] Improve scripts/igt_doc.py script

[Mauro Carvalho Chehab]

From: Mauro Carvalho Chehab <mchehab@kernel.org>

Make the script more generic by allowing it to get the actual fields
from a config file, produce hierarchical output and sort/filter
fields when showing subtests.

Mauro Carvalho Chehab (7):
  scripts/igt_doc.py: beautify its code
  scripts/igt_doc.py: add JSON file output
  scripts/igt_doc.py: dynamically create fields array from a JSON file
  scripts/igt_doc.py: add support to specify numeric values
  scripts/igt_doc.py: add support to produce hierarchical output
  scripts/igt_doc.py: improve --show-subtests logic
  scripts/igt_doc.py: add error handler for subprocess

 scripts/igt_doc.py | 567 +++++++++++++++++++++++++++++++++++++++------
 1 file changed, 502 insertions(+), 65 deletions(-)

-- 
2.39.2

[igt-dev] [PATCH i-g-t 1/7] scripts/igt_doc.py: beautify its code

[Mauro Carvalho Chehab]

From: Mauro Carvalho Chehab <mchehab@kernel.org>

Change some comments and add blank lines before docstring, in
order to better document its code.

No functional changes.

Signed-off-by: Mauro Carvalho Chehab <mchehab@kernel.org>
Reviewed-by: Zbigniew Kempczyński <zbigniew.kempczynski@intel.com>
---
 scripts/igt_doc.py | 5 ++---
 1 file changed, 2 insertions(+), 3 deletions(-)
 mode change 100755 => 100644 scripts/igt_doc.py

diff --git a/scripts/igt_doc.py b/scripts/igt_doc.py
old mode 100755
new mode 100644
index b7e3129d3f54..c8bd4054f638
--- a/scripts/igt_doc.py
+++ b/scripts/igt_doc.py
@@ -109,7 +109,6 @@ class TestList:
     SUBTEST as those variables are locally processed on each comment line.
     """
 
-
     def __init__(self):
         self.doc = {}
         self.test_number = 0
@@ -238,7 +237,7 @@ class TestList:
 
     def print_test(self):
 
-        """Print tests and subtests ordered by tests"""
+        """Print tests and subtests"""
 
         for test in sorted(self.doc.keys()):
             fname = self.doc[test]["File"]
@@ -368,7 +367,7 @@ class TestList:
     # File handling methods
     #
 
-    def add_file_documentation(self, fname, field_re):
+    def add_file_documentation(self, fname):
 
         """Adds the contents of test/subtest documentation form a file"""
 
-- 
2.39.2

[igt-dev] [PATCH i-g-t 2/7] scripts/igt_doc.py: add JSON file output

[Mauro Carvalho Chehab]

From: Mauro Carvalho Chehab <mchehab@kernel.org>

At least for debugging purposes, it is interesting to see how the
script is parsing the input files and expanding wildcards on a more
condensed way. So, add support to output the dictionary contents
into a json file.

Signed-off-by: Mauro Carvalho Chehab <mchehab@kernel.org>
Reviewed-by: Zbigniew Kempczyński <zbigniew.kempczynski@intel.com>
Signed-off-by: Mauro Carvalho Chehab <mchehab@kernel.org>
---
 scripts/igt_doc.py | 51 ++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 51 insertions(+)

diff --git a/scripts/igt_doc.py b/scripts/igt_doc.py
index c8bd4054f638..d21f87dc7183 100644
--- a/scripts/igt_doc.py
+++ b/scripts/igt_doc.py
@@ -11,6 +11,7 @@
 """Maintain test plan and test implementation documentation on IGT."""
 
 import argparse
+import json
 import re
 import subprocess
 import sys
@@ -231,6 +232,42 @@ class TestList:
 
         return subtest_array
 
+    def expand_dictionary(self):
+
+        """ prepares a dictonary with subtest arguments expanded """
+
+        test_dict = {}
+
+        for test in self.doc:                   # pylint: disable=C0206
+            fname = self.doc[test]["File"]
+
+            name = re.sub(r'.*tests/', '', fname)
+            name = re.sub(r'\.[ch]', '', name)
+            name = "igt@" + name
+
+            test_dict[name] = {}
+
+            for field in self.doc[test]:
+                if field == "subtest":
+                    continue
+                if field == "arg":
+                    continue
+
+                test_dict[name][field] = self.doc[test][field]
+
+            subtest_array = self.expand_subtest(fname, name, test)
+            for subtest in subtest_array:
+                summary = subtest["Summary"]
+                test_dict[name][summary] = {}
+                for field in sorted(subtest.keys()):
+                    if field == 'Summary':
+                        continue
+                    if field == 'arg':
+                        continue
+                    test_dict[name][summary][field] = subtest[field]
+
+        return test_dict
+
     #
     # Output methods
     #
@@ -280,6 +317,14 @@ class TestList:
             print()
             print()
 
+    def print_json(self, out_fname):
+
+        """Adds the contents of test/subtest documentation form a file"""
+        test_dict = self.expand_dictionary()
+
+        with open(out_fname, "w", encoding='utf8') as write_file:
+            json.dump(test_dict, write_file, indent=4)
+
     #
     # Subtest list methods
     #
@@ -541,6 +586,8 @@ parser = argparse.ArgumentParser(description = "Print formatted kernel documenta
                                  epilog = 'If no action specified, assume --rest.')
 parser.add_argument("--rest", action="store_true",
                     help="Generate documentation from the source files, in ReST file format.")
+parser.add_argument("--to-json",
+                    help="Output test documentation in JSON format as TO_JSON file")
 parser.add_argument("--show-subtests", action="store_true",
                     help="Shows the name of the documented subtests in alphabetical order.")
 parser.add_argument("--check-testlist", action="store_true",
@@ -570,5 +617,9 @@ if parse_args.check_testlist:
     RUN = 1
     tests.check_tests()
 
+if parse_args.to_json:
+    RUN = 1
+    tests.print_json(parse_args.to_json)
+
 if not RUN or parse_args.rest:
     tests.print_test()
-- 
2.39.2

[igt-dev] [PATCH i-g-t 3/7] scripts/igt_doc.py: dynamically create fields array from a JSON file

[Mauro Carvalho Chehab]

From: Mauro Carvalho Chehab <mchehab@kernel.org>

Defining TEST/SUBTEST fields is not an easy task, and may change with
time. So, place them into a JSON file, together with the files that
are implements the test scope.

Signed-off-by: Mauro Carvalho Chehab <mchehab@kernel.org>
Reviewed-by: Zbigniew Kempczyński <zbigniew.kempczynski@intel.com>
---
 scripts/igt_doc.py | 239 +++++++++++++++++++++++++++++++++++++--------
 1 file changed, 197 insertions(+), 42 deletions(-)
 mode change 100644 => 100755 scripts/igt_doc.py

diff --git a/scripts/igt_doc.py b/scripts/igt_doc.py
old mode 100644
new mode 100755
index d21f87dc7183..3f061a5fcfd3
--- a/scripts/igt_doc.py
+++ b/scripts/igt_doc.py
@@ -1,5 +1,5 @@
 #!/usr/bin/env python3
-# pylint: disable=C0301,R0914,R0912,R0915
+# pylint: disable=C0301,R0902,R0914,R0912,R0915
 # SPDX-License-Identifier: (GPL-2.0 OR MIT)
 
 ## Copyright (C) 2023    Intel Corporation                 ##
@@ -11,27 +11,66 @@
 """Maintain test plan and test implementation documentation on IGT."""
 
 import argparse
+import glob
 import json
 import re
 import subprocess
 import sys
 
-IGT_BUILD_PATH = 'build/'
+IGT_BUILD_PATH = 'build'
 IGT_RUNNER = 'runner/igt_runner'
 
-# Fields that mat be inside TEST and SUBTEST macros
-fields = [
-    'Category',       # Hardware building block / Software building block / ...
-    'Sub-category',   # waitfence / dmabuf/ sysfs / debugfs / ...
-    'Functionality',  # basic test / ...
-    'Test category',  # functionality test / pereformance / stress
-    'Run type',       # BAT / workarouds / stress / developer-specific / ...
-    'Issue',          # Bug tracker issue(s)
-    'GPU excluded platforms',     # none / DG1 / DG2 / TGL / MTL / PVC / ATS-M / ...
-    'GPU requirements',  # Any other specific platform requirements
-    'Depends on',     # some other IGT test, like igt@test@subtes
-    'Test requirements',  # other non-platform requirements
-    'Description']    # Description of the test
+#
+# Ancillary logic to allow plurals on fields
+#
+# As suggested at https://stackoverflow.com/questions/18902608/generating-the-plural-form-of-a-noun/19018986
+#
+
+def plural(field):
+
+    """
+    Poor man's conversion to plural.
+
+    It should cover usual English plural rules, although it is not meant
+    to cover exceptions (except for a few ones that could be useful on actual
+    fields).
+
+    """
+
+    if (match := re.match(r"(.*)\b(\S+)", field)):
+        ret_str = match.group(1)
+        word = match.group(2)
+
+        if word.isupper():
+            ret_str += word
+        elif word in ["of", "off", "on", "description", "todo"]:
+            ret_str += word
+        elif word.endswith('ed'):
+            ret_str += word
+        elif word[-1:] in ['s', 'x', 'z']:
+            ret_str += word + 'es'
+        elif word[-2:] in ['sh', 'ch']:
+            ret_str += word + 'es'
+        elif word.endswith('fe'):
+            ret_str += word[:-2] + 'ves'
+        elif word.endswith('f'):
+            ret_str += word[:-1] + 'ves'
+        elif word.endswith('y'):
+            ret_str += word[:-1] + 'ies'
+        elif word.endswith('o'):
+            ret_str += word + 'es'
+        elif word.endswith('us'):
+            ret_str += word[:-2] + 'i'
+        elif word.endswith('on'):
+            ret_str += word[:-2] + 'a'
+        elif word.endswith('an'):
+            ret_str += word[:-2] + 'en'
+        else:
+            ret_str += word + 's'
+
+        return ret_str
+
+    return field
 
 #
 # TestList class definition
@@ -48,13 +87,7 @@ class TestList:
          * Category: Software build block
          * Sub-category: documentation
          * Functionality: test documentation
-         * Test category: ReST generation
-         * Run type: IGT kunit test
          * Issue: none
-         * GPU excluded platforms: none
-         * GPU requirements: none
-         * Depends on: @igt@deadbeef@basic
-         * Test requirements: Need python3 to run it
          * Description: Complete description of this test
          *
          * SUBTEST: foo
@@ -108,17 +141,138 @@ class TestList:
     The wildcard arguments there need to be expanded. This is done by
     defining arg[1] to arg[n] at the same code comment that contains the
     SUBTEST as those variables are locally processed on each comment line.
+
+    This script needs a configuration file, in JSON format, describing the
+    fields which will be parsed for TEST and/or SUBTEST tags.
+
+    An example of such file is:
+
+    {
+        "files": [ "tests/driver/*.c" ],
+        "fields": {
+            "Category": {
+                "Sub-category": {
+                    "Functionality": {
+                    }
+                }
+            },
+            "Issue": {
+                "_properties_": {
+                    "description": "If the test is used to solve an issue, point to the URL containing the issue."
+                }
+            },
+            "Description" : {
+                "_properties_": {
+                    "description": "Provides a description for the test/subtest."
+                }
+            }
+        }
+    }
+
+    So, the above JSON config file expects tags like those:
+
+    TEST: foo
+    Description: foo
+
+    SUBTEST: bar
+    Category: Hardware
+    Sub-category: EU
+    Description: test bar on EU
+
+    SUBTEST: foobar
+    Category: Software
+    Type: ioctl
+    Description: test ioctls
     """
 
-    def __init__(self):
+    def __init__(self, config_fname, file_list):
         self.doc = {}
         self.test_number = 0
         self.min_test_prefix = ''
+        self.config = None
+        self.filenames = file_list
+        self.props = {}
+        self.config_fname = config_fname
+        self.level_count = 0
+        self.field_list = {}
+
+        with open(config_fname, 'r', encoding='utf8') as handle:
+            self.config = json.load(handle)
+
+            self.__add_field(None, 0, 0, self.config["fields"])
+
+            sublevel_count = [ 0 ] * self.level_count
+
+            for field, item in self.props.items():
+                if "sublevel" in item["_properties_"]:
+                    level = item["_properties_"]["level"]
+                    sublevel_count[level - 1] += 1
+                field_lc = field.lower()
+                self.field_list[field_lc] = field
+                field_plural = plural(field_lc)
+                if field_lc != field_plural:
+                    self.field_list[field_plural] = field
+
+            # Remove non-multilevel items, as we're only interested on
+            # hierarchical item levels here
+            for field, item in self.props.items():
+                if "sublevel" in item["_properties_"]:
+                    level = item["_properties_"]["level"]
+                    if sublevel_count[level - 1] == 1:
+                        del item["_properties_"]["level"]
+                        del item["_properties_"]["sublevel"]
+
+            if not self.filenames:
+                self.filenames = []
+                files = self.config["files"]
+                for cfg_file in files:
+                    cfg_file = os.path.realpath(os.path.dirname(config_fname)) + "/" + cfg_file
+                    for fname in glob.glob(cfg_file):
+                        self.filenames.append(fname)
+
+        if not self.filenames:
+            sys.exit("Need file names to be processed")
+
+        # Parse files, expanding wildcards
+        field_re = re.compile(r"(" + '|'.join(self.field_list.keys()) + r'):\s*(.*)', re.I)
+
+        for fname in self.filenames:
+            self.__add_file_documentation(fname, field_re)
 
     #
     # ancillary methods
     #
 
+    def __add_field(self, name, sublevel, hierarchy_level, field):
+
+        """ Flatten config fields into a non-hierarchical dictionary """
+
+        for key in field:
+            if key not in self.props:
+                self.props[key] = {}
+                self.props[key]["_properties_"] = {}
+
+            if name:
+                if key == "_properties_":
+                    if key not in self.props:
+                        self.props[key] = {}
+                    self.props[name][key].update(field[key])
+
+                    sublevel += 1
+                    hierarchy_level += 1
+                    if "sublevel" in self.props[name][key]:
+                        if self.props[name][key]["sublevel"] != sublevel:
+                            sys.exit(f"Error: config defined {name} as sublevel {self.props[key]['sublevel']}, but wants to redefine as sublevel {sublevel}")
+
+                    self.props[name][key]["level"] = self.level_count
+                    self.props[name][key]["sublevel"] = sublevel
+
+                    continue
+            else:
+                self.level_count += 1
+
+            self.__add_field(key, sublevel, hierarchy_level, field[key])
+
     def expand_subtest(self, fname, test_name, test):
 
         """Expand subtest wildcards providing an array with subtests"""
@@ -145,7 +299,7 @@ class TestList:
                     if k == 'arg':
                         continue
 
-                    if self.doc[test]["subtest"][subtest][k] == self.doc[test][k]:
+                    if k in self.doc[test] and self.doc[test]["subtest"][subtest][k] == self.doc[test][k]:
                         continue
 
                     subtest_dict[k] = self.doc[test]["subtest"][subtest][k]
@@ -210,8 +364,9 @@ class TestList:
 
                     sub_field = self.doc[test]["subtest"][subtest][field]
                     sub_field = re.sub(r"%?\barg\[(\d+)\]", lambda m: arg_map[int(m.group(1)) - 1], sub_field) # pylint: disable=W0640
-                    if sub_field == self.doc[test][field]:
-                        continue
+                    if field in self.doc[test]:
+                        if sub_field in self.doc[test][field] and sub_field == self.doc[test][field]:
+                            continue
 
                     subtest_dict[field] = sub_field
 
@@ -323,7 +478,7 @@ class TestList:
         test_dict = self.expand_dictionary()
 
         with open(out_fname, "w", encoding='utf8') as write_file:
-            json.dump(test_dict, write_file, indent=4)
+            json.dump(test_dict, write_file, indent = 4)
 
     #
     # Subtest list methods
@@ -412,7 +567,7 @@ class TestList:
     # File handling methods
     #
 
-    def add_file_documentation(self, fname):
+    def __add_file_documentation(self, fname, field_re):
 
         """Adds the contents of test/subtest documentation form a file"""
 
@@ -503,7 +658,7 @@ class TestList:
 
                 # It is a known section. Parse its contents
                 if (match := re.match(field_re, file_line)):
-                    current_field = match.group(1).lower().capitalize()
+                    current_field = self.field_list[match.group(1).lower()]
                     match_val = match.group(2)
 
                     if handle_section == 'test':
@@ -550,17 +705,19 @@ class TestList:
 
                 if (match := re.match(r'^(.*):', file_line)):
                     sys.exit(f"{fname}:{file_ln + 1}: Error: unrecognized field '%s'. Need to add at %s" %
-                            (match.group(1), fname))
+                            (match.group(1), self.config_fname))
 
                 # Handle multi-line field contents
                 if current_field:
                     if (match := re.match(r'\s*(.*)', file_line)):
                         if handle_section == 'test':
-                            self.doc[current_test][current_field] += " " + \
-                                match.group(1)
+                            dic = self.doc[current_test]
                         else:
-                            self.doc[current_test]["subtest"][current_subtest][current_field] += " " + \
-                                match.group(1)
+                            dic = self.doc[current_test]["subtest"][current_subtest]
+
+                        if dic[current_field] != '':
+                            dic[current_field] += " "
+                        dic[current_field] += match.group(1)
 
                     continue
 
@@ -573,8 +730,9 @@ class TestList:
                         if match:
                             self.doc[current_test]["arg"][arg_ref][cur_arg][cur_arg_element] = match.group(1) + ' ' + match_val + ">"
                         else:
-                            self.doc[current_test]["arg"][arg_ref][cur_arg][cur_arg_element] += ' ' + match_val
-
+                            if self.doc[current_test]["arg"][arg_ref][cur_arg][cur_arg_element] != '':
+                                self.doc[current_test]["arg"][arg_ref][cur_arg][cur_arg_element] += ' '
+                            self.doc[current_test]["arg"][arg_ref][cur_arg][cur_arg_element] += match_val
                     continue
 
 #
@@ -584,6 +742,8 @@ class TestList:
 parser = argparse.ArgumentParser(description = "Print formatted kernel documentation to stdout.",
                                  formatter_class = argparse.ArgumentDefaultsHelpFormatter,
                                  epilog = 'If no action specified, assume --rest.')
+parser.add_argument("--config", required = True,
+                    help="JSON file describing the test plan template")
 parser.add_argument("--rest", action="store_true",
                     help="Generate documentation from the source files, in ReST file format.")
 parser.add_argument("--to-json",
@@ -595,17 +755,12 @@ parser.add_argument("--check-testlist", action="store_true",
 parser.add_argument("--igt-build-path",
                     help="Path where the IGT runner is sitting. Used by --check-testlist.",
                     default=IGT_BUILD_PATH)
-parser.add_argument('--files', nargs='+', required=True,
+parser.add_argument('--files', nargs='+',
                     help="File name(s) to be processed")
 
 parse_args = parser.parse_args()
 
-field_regex = re.compile(r"(" + '|'.join(fields) + r'):\s*(.*)', re.I)
-
-tests = TestList()
-
-for filename in parse_args.files:
-    tests.add_file_documentation(filename, field_regex)
+tests = TestList(parse_args.config, parse_args.files)
 
 RUN = 0
 if parse_args.show_subtests:
-- 
2.39.2

[igt-dev] [PATCH i-g-t 4/7] scripts/igt_doc.py: add support to specify numeric values

[Mauro Carvalho Chehab]

From: Mauro Carvalho Chehab <mchehab@kernel.org>

Add a way to specify a set of default values to be used when numeric
wildcards like "%d" are present at the igt subtest.

Signed-off-by: Mauro Carvalho Chehab <mchehab@kernel.org>
---
 scripts/igt_doc.py | 36 ++++++++++++++++++++++++++++++++++++
 1 file changed, 36 insertions(+)

diff --git a/scripts/igt_doc.py b/scripts/igt_doc.py
index 3f061a5fcfd3..a1d3a0350499 100755
--- a/scripts/igt_doc.py
+++ b/scripts/igt_doc.py
@@ -126,6 +126,27 @@ class TestList:
          * @userptr-misaligned-binds:	userptr misaligned
          */
 
+    It is also possible to define a list of values that will actually be
+    used by the test and no string replacement is needed.
+    This is done by using one or multiple arg[n].values lines:
+
+        /**
+         * SUBTEST: unbind-all-%d-vmas
+         * Description: unbind all with %arg[1] VMAs
+         *
+         * arg[1].values: 2, 8
+         * arg[1].values: 16, 32
+         */
+
+        /**
+         * SUBTEST: unbind-all-%d-vmas
+         * Description: unbind all with %arg[1] VMAs
+         *
+         * arg[1].values: 64, 128
+         */
+
+    Such feature is specially useful for numeric parameters
+
     The valid `field` values are defined on a JSON configuration file.
 
     The TEST documents the common fields present on all tests. Typically,
@@ -703,6 +724,21 @@ class TestList:
 
                     continue
 
+                # We don't need a multi-lined version here
+                if (match := re.match(r'arg\[(\d+)\]\.values:\s*(.*)', file_line)):
+                    cur_arg = int(match.group(1)) - 1
+
+                    if cur_arg not in self.doc[current_test]["arg"][arg_ref]:
+                        self.doc[current_test]["arg"][arg_ref][cur_arg] = {}
+
+                    values = match.group(2).split(",")
+                    for split_val in values:
+                        if split_val == "":
+                            continue
+                        self.doc[current_test]["arg"][arg_ref][cur_arg][split_val.strip()] = split_val.strip()
+
+                    continue
+
                 if (match := re.match(r'^(.*):', file_line)):
                     sys.exit(f"{fname}:{file_ln + 1}: Error: unrecognized field '%s'. Need to add at %s" %
                             (match.group(1), self.config_fname))
-- 
2.39.2

[igt-dev] [PATCH i-g-t 5/7] scripts/igt_doc.py: add support to produce hierarchical output

[Mauro Carvalho Chehab]

From: Mauro Carvalho Chehab <mchehab@kernel.org>

Internally, the JSON dictionary allows specifying fields on
a hierarchical way. Add support to also print the documentation
using it.

While here, also allow specifying a file name for the ReST output.

Signed-off-by: Mauro Carvalho Chehab <mchehab@kernel.org>
---
 scripts/igt_doc.py | 199 +++++++++++++++++++++++++++++++++++++++------
 1 file changed, 173 insertions(+), 26 deletions(-)

diff --git a/scripts/igt_doc.py b/scripts/igt_doc.py
index a1d3a0350499..12a9ebfc0348 100755
--- a/scripts/igt_doc.py
+++ b/scripts/igt_doc.py
@@ -20,6 +20,26 @@ import sys
 IGT_BUILD_PATH = 'build'
 IGT_RUNNER = 'runner/igt_runner'
 
+#
+# ancillary functions to sort dictionary hierarchy
+#
+def _sort_per_level(item):
+    if "level" not in item[1]["_properties_"]:
+        return item[0]
+
+    return "%05d_%05d_%s" % (item[1]["_properties_"]["level"], item[1]["_properties_"]["sublevel"], item[0])   # pylint: disable=C0209
+
+def _sort_using_array(item, array):
+    ret_str = ''
+    for field in array:
+        if field in item[1]:
+            ret_str += '_' + field + '_' + item[1][field]
+
+    if ret_str == '':
+        ret_str="________"
+
+    return ret_str
+
 #
 # Ancillary logic to allow plurals on fields
 #
@@ -242,6 +262,7 @@ class TestList:
                     if sublevel_count[level - 1] == 1:
                         del item["_properties_"]["level"]
                         del item["_properties_"]["sublevel"]
+            del self.props["_properties_"]
 
             if not self.filenames:
                 self.filenames = []
@@ -294,7 +315,7 @@ class TestList:
 
             self.__add_field(key, sublevel, hierarchy_level, field[key])
 
-    def expand_subtest(self, fname, test_name, test):
+    def expand_subtest(self, fname, test_name, test, allow_inherit):
 
         """Expand subtest wildcards providing an array with subtests"""
 
@@ -320,11 +341,13 @@ class TestList:
                     if k == 'arg':
                         continue
 
-                    if k in self.doc[test] and self.doc[test]["subtest"][subtest][k] == self.doc[test][k]:
-                        continue
+                    if not allow_inherit:
+                        if k in self.doc[test] and self.doc[test]["subtest"][subtest][k] == self.doc[test][k]:
+                            continue
 
                     subtest_dict[k] = self.doc[test]["subtest"][subtest][k]
-                    subtest_array.append(subtest_dict)
+
+                subtest_array.append(subtest_dict)
 
                 continue
 
@@ -385,9 +408,11 @@ class TestList:
 
                     sub_field = self.doc[test]["subtest"][subtest][field]
                     sub_field = re.sub(r"%?\barg\[(\d+)\]", lambda m: arg_map[int(m.group(1)) - 1], sub_field) # pylint: disable=W0640
-                    if field in self.doc[test]:
-                        if sub_field in self.doc[test][field] and sub_field == self.doc[test][field]:
-                            continue
+
+                    if not allow_inherit:
+                        if field in self.doc[test]:
+                            if sub_field in self.doc[test][field] and sub_field == self.doc[test][field]:
+                                continue
 
                     subtest_dict[field] = sub_field
 
@@ -408,9 +433,9 @@ class TestList:
 
         return subtest_array
 
-    def expand_dictionary(self):
+    def expand_dictionary(self, subtest_only):
 
-        """ prepares a dictonary with subtest arguments expanded """
+        """ prepares a dictionary with subtest arguments expanded """
 
         test_dict = {}
 
@@ -421,26 +446,31 @@ class TestList:
             name = re.sub(r'\.[ch]', '', name)
             name = "igt@" + name
 
-            test_dict[name] = {}
+            if not subtest_only:
+                test_dict[name] = {}
 
-            for field in self.doc[test]:
-                if field == "subtest":
-                    continue
-                if field == "arg":
-                    continue
+                for field in self.doc[test]:
+                    if field == "subtest":
+                        continue
+                    if field == "arg":
+                        continue
 
-                test_dict[name][field] = self.doc[test][field]
+                    test_dict[name][field] = self.doc[test][field]
+                dic = test_dict[name]
+            else:
+                dic = test_dict
 
-            subtest_array = self.expand_subtest(fname, name, test)
+            subtest_array = self.expand_subtest(fname, name, test, subtest_only)
             for subtest in subtest_array:
                 summary = subtest["Summary"]
-                test_dict[name][summary] = {}
+
+                dic[summary] = {}
                 for field in sorted(subtest.keys()):
                     if field == 'Summary':
                         continue
                     if field == 'arg':
                         continue
-                    test_dict[name][summary][field] = subtest[field]
+                    dic[summary][field] = subtest[field]
 
         return test_dict
 
@@ -448,9 +478,15 @@ class TestList:
     # Output methods
     #
 
-    def print_test(self):
+    def print_rest_flat(self, filename):
 
-        """Print tests and subtests"""
+        """Print tests and subtests ordered by tests"""
+
+        handler = None
+        if filename:
+            original_stdout = sys.stdout
+            handler = open(filename, "w", encoding='utf8') # pylint: disable=R1732
+            sys.stdout = handler
 
         for test in sorted(self.doc.keys()):
             fname = self.doc[test]["File"]
@@ -472,7 +508,7 @@ class TestList:
 
                 print(f":{field}: {self.doc[test][field]}")
 
-            subtest_array = self.expand_subtest(fname, name, test)
+            subtest_array = self.expand_subtest(fname, name, test, False)
 
             for subtest in subtest_array:
                 print()
@@ -493,10 +529,101 @@ class TestList:
             print()
             print()
 
+        if handler:
+            handler.close()
+            sys.stdout = original_stdout
+
+    def print_nested_rest(self, filename):
+
+        """Print tests and subtests ordered by tests"""
+
+        handler = None
+        if filename:
+            original_stdout = sys.stdout
+            handler = open(filename, "w", encoding='utf8') # pylint: disable=R1732
+            sys.stdout = handler
+
+        # Identify the sort order for the fields
+        fields_order = []
+        fields = sorted(self.props.items(), key = _sort_per_level)
+        for item in fields:
+            fields_order.append(item[0])
+
+        # Receives a flat subtest dictionary, with wildcards expanded
+        subtest_dict = self.expand_dictionary(True)
+
+        subtests = sorted(subtest_dict.items(),
+                          key = lambda x: _sort_using_array(x, fields_order))
+
+        # Use the level markers below
+        level_markers='=-^_~:.`"*+#'
+
+        # Print the data
+        old_fields = [ '' ] * len(fields_order)
+
+        for subtest, fields in subtests:
+            # Check what level has different message
+            marker = 0
+            for cur_level in range(0, len(fields_order)):  # pylint: disable=C0200
+                field = fields_order[cur_level]
+                if not "level" in self.props[field]["_properties_"]:
+                    continue
+                if field in fields:
+                    if old_fields[cur_level] != fields[field]:
+                        break
+                    marker += 1
+
+            # print hierarchy
+            for i in range(cur_level, len(fields_order)):
+                if not "level" in self.props[fields_order[i]]["_properties_"]:
+                    continue
+                if not fields_order[i] in fields:
+                    continue
+
+                if marker >= len(level_markers):
+                    sys.exit(f"Too many levels: {marker}, maximum limit is {len(level_markers):}")
+
+                title_str = fields_order[i] + ": " + fields[fields_order[i]]
+
+                print(title_str)
+                print(level_markers[marker] * len(title_str))
+                print()
+                marker += 1
+
+            print()
+            print("``" + subtest + "``")
+            print()
+
+            # print non-hierarchy fields
+            for field in fields_order:
+                if "level" in self.props[field]["_properties_"]:
+                    continue
+
+                if field in fields:
+                    print(f":{field}: {fields[field]}")
+
+            # Store current values
+            for i in range(cur_level, len(fields_order)):
+                field = fields_order[i]
+                if not "level" in self.props[field]["_properties_"]:
+                    continue
+                if field in fields:
+                    old_fields[i] = fields[field]
+                else:
+                    old_fields[i] = ''
+
+            print()
+
+        if handler:
+            handler.close()
+            sys.stdout = original_stdout
+
     def print_json(self, out_fname):
 
         """Adds the contents of test/subtest documentation form a file"""
-        test_dict = self.expand_dictionary()
+
+        # Receives a dictionary with tests->subtests with expanded subtests
+        test_dict = self.expand_dictionary(False)
 
         with open(out_fname, "w", encoding='utf8') as write_file:
             json.dump(test_dict, write_file, indent = 4)
@@ -663,7 +790,22 @@ class TestList:
                     current_field = ''
                     handle_section = 'subtest'
 
+                    # subtests inherit properties from the tests
                     self.doc[current_test]["subtest"][current_subtest] = {}
+                    for field in self.doc[current_test].keys():
+                        if field == "arg":
+                            continue
+                        if field == "summary":
+                            continue
+                        if field == "File":
+                            continue
+                        if field == "subtest":
+                            continue
+                        if field == "_properties_":
+                            continue
+                        if field == "Description":
+                            continue
+                        self.doc[current_test]["subtest"][current_subtest][field] = self.doc[current_test][field]
 
                     self.doc[current_test]["subtest"][current_subtest]["Summary"] = match.group(1)
                     self.doc[current_test]["subtest"][current_subtest]["Description"] = ''
@@ -780,8 +922,10 @@ parser = argparse.ArgumentParser(description = "Print formatted kernel documenta
                                  epilog = 'If no action specified, assume --rest.')
 parser.add_argument("--config", required = True,
                     help="JSON file describing the test plan template")
-parser.add_argument("--rest", action="store_true",
-                    help="Generate documentation from the source files, in ReST file format.")
+parser.add_argument("--rest",
+                    help="Output documentation from the source files in REST file.")
+parser.add_argument("--per-test", action="store_true",
+                    help="Modifies ReST output to print subtests per test.")
 parser.add_argument("--to-json",
                     help="Output test documentation in JSON format as TO_JSON file")
 parser.add_argument("--show-subtests", action="store_true",
@@ -813,4 +957,7 @@ if parse_args.to_json:
     tests.print_json(parse_args.to_json)
 
 if not RUN or parse_args.rest:
-    tests.print_test()
+    if parse_args.per_test:
+        tests.print_rest_flat(parse_args.rest)
+    else:
+        tests.print_nested_rest(parse_args.rest)
-- 
2.39.2

[igt-dev] [PATCH i-g-t 6/7] scripts/igt_doc.py: improve --show-subtests logic

[Mauro Carvalho Chehab]

From: Mauro Carvalho Chehab <mchehab@kernel.org>

Add command line parameters to improve --show-subtests:
1) to sort results of --show-subtest based on the contents of a field.
   This is useful, for instance, to check if all subtests have
   proper values for a given field.
2) to filter for an specific field value. This is interesting to be
   able to compare testlists with the expected documented test list.

Signed-off-by: Mauro Carvalho Chehab <mchehab@kernel.org>
---
 scripts/igt_doc.py | 64 ++++++++++++++++++++++++++++++++++++++++------
 1 file changed, 56 insertions(+), 8 deletions(-)

diff --git a/scripts/igt_doc.py b/scripts/igt_doc.py
index 12a9ebfc0348..9280be743524 100755
--- a/scripts/igt_doc.py
+++ b/scripts/igt_doc.py
@@ -1,5 +1,5 @@
 #!/usr/bin/env python3
-# pylint: disable=C0301,R0902,R0914,R0912,R0915
+# pylint: disable=C0301,R0902,R0914,R0912,R0915,R1702,C0302
 # SPDX-License-Identifier: (GPL-2.0 OR MIT)
 
 ## Copyright (C) 2023    Intel Corporation                 ##
@@ -632,11 +632,27 @@ class TestList:
     # Subtest list methods
     #
 
-    def get_subtests(self):
+    def get_subtests(self, sort_field = None, filter_field_expr = None):
 
         """Return an array with all subtests"""
 
-        subtests = []
+        subtests = {}
+        subtests[""] = []
+
+        if sort_field:
+            if sort_field.lower() not in self.field_list:
+                sys.exit(f"Field '{sort_field}' is not defined")
+            sort_field = self.field_list[sort_field.lower()]
+
+        if filter_field_expr:
+            if not (match := re.match(r"(.*)=~\s*(.*)", filter_field_expr)):
+                sys.exit(f"Filter field {filter_field_expr} is not at <field> =~ <regex> syntax")
+
+            field = match.group(1).strip().lower()
+            if field not in self.field_list:
+                sys.exit(f"Field '{field}' is not defined")
+            filter_field = self.field_list[field]
+            regex = re.compile("{0}".format(match.group(2).strip()), re.I) # pylint: disable=C0209
 
         for test in sorted(self.doc.keys()):
             fname = self.doc[test]["File"]
@@ -645,10 +661,26 @@ class TestList:
             test_name = re.sub(r'\.[ch]', '', test_name)
             test_name = "igt@" + test_name
 
-            subtest_array = self.expand_subtest(fname, test_name, test)
+            subtest_array = self.expand_subtest(fname, test_name, test, True)
 
             for subtest in subtest_array:
-                subtests.append(subtest["Summary"])
+                if filter_field_expr:
+                    if filter_field not in subtest:
+                        continue
+                    if not re.match(regex, subtest[filter_field]):
+                        continue
+
+                if sort_field:
+                    if sort_field in subtest:
+                        if subtest[sort_field] not in subtests:
+                            subtests[subtest[sort_field]] = []
+
+                        subtests[subtest[sort_field]].append(subtest["Summary"])
+                    else:
+                        subtests[""].append(subtest["Summary"])
+
+                else:
+                    subtests[""].append(subtest["Summary"])
 
         return subtests
 
@@ -659,7 +691,7 @@ class TestList:
 
         """Compare documented subtests with the IGT test list"""
 
-        doc_subtests = sorted(self.get_subtests())
+        doc_subtests = sorted(self.get_subtests()[""])
 
         for i in range(0, len(doc_subtests)): # pylint: disable=C0200
             doc_subtests[i] = re.sub(r'\<[^\>]+\>', r'\\d+', doc_subtests[i])
@@ -930,6 +962,10 @@ parser.add_argument("--to-json",
                     help="Output test documentation in JSON format as TO_JSON file")
 parser.add_argument("--show-subtests", action="store_true",
                     help="Shows the name of the documented subtests in alphabetical order.")
+parser.add_argument("--sort-field",
+                    help="modify --show-tests to sort output based on SORT_FIELD value")
+parser.add_argument("--filter-field",
+                    help="modify --show-tests to filter output based a regex given by FILTER_FIELD=~'regex'")
 parser.add_argument("--check-testlist", action="store_true",
                     help="Compare documentation against IGT runner testlist.")
 parser.add_argument("--igt-build-path",
@@ -945,8 +981,20 @@ tests = TestList(parse_args.config, parse_args.files)
 RUN = 0
 if parse_args.show_subtests:
     RUN = 1
-    for sub in tests.get_subtests():
-        print (sub)
+    if parse_args.sort_field:
+        test_subtests = tests.get_subtests(parse_args.sort_field, parse_args.filter_field)
+        for val_key in sorted(test_subtests.keys()):
+            if not test_subtests[val_key]:
+                continue
+            if val_key == "":
+                print("not defined:")
+            else:
+                print(f"{val_key}:")
+            for sub in test_subtests[val_key]:
+                print (f"  {sub}")
+    else:
+        for sub in tests.get_subtests(parse_args.sort_field, parse_args.filter_field)[""]:
+            print (sub)
 
 if parse_args.check_testlist:
     RUN = 1
-- 
2.39.2

[igt-dev] [PATCH i-g-t 7/7] scripts/igt_doc.py: add error handler for subprocess

[Mauro Carvalho Chehab]

From: Mauro Carvalho Chehab <mchehab@kernel.org>

Handle exceptions when trying to excecute IGT runner.

Signed-off-by: Mauro Carvalho Chehab <mchehab@kernel.org>
---
 scripts/igt_doc.py | 17 +++++++++--------
 1 file changed, 9 insertions(+), 8 deletions(-)

diff --git a/scripts/igt_doc.py b/scripts/igt_doc.py
index 9280be743524..c6b0ed30fcbc 100755
--- a/scripts/igt_doc.py
+++ b/scripts/igt_doc.py
@@ -697,14 +697,15 @@ class TestList:
             doc_subtests[i] = re.sub(r'\<[^\>]+\>', r'\\d+', doc_subtests[i])
 
         # Get a list of tests from
-        result = subprocess.run([ f"{IGT_BUILD_PATH}/{IGT_RUNNER}",  # pylint: disable=W1510
-                                "-L", "-t",  self.min_test_prefix,
-                                f"{IGT_BUILD_PATH}/tests"],
-                                capture_output = True, text = True)
-        if result.returncode:
-            print( result.stdout)
-            print("Error:", result.stderr)
-            sys.exit(result.returncode)
+        try:
+            result = subprocess.run([ f"{IGT_BUILD_PATH}/{IGT_RUNNER}",
+                                    "-L", "-t",  self.min_test_prefix,
+                                    f"{IGT_BUILD_PATH}/tests"], check = True,
+                                    capture_output = True, text = True)
+        except subprocess.CalledProcessError as sub_err:
+            print(sub_err.stderr)
+            print("Error:", sub_err)
+            sys.exit(1)
 
         run_subtests = sorted(result.stdout.splitlines())
 
-- 
2.39.2