[igt-dev] [PATCH i-g-t v2 0/3] ./scripts/igt_doc.py: speedup check logic

[igt-dev] [PATCH i-g-t v2 0/3] ./scripts/igt_doc.py: speedup check logic

[Mauro Carvalho Chehab]

From: Mauro Carvalho Chehab <mchehab@kernel.org>

The logic which checks for missing/obsolete documentation is too
slow. Currently, on my notebook it takes 12 seconds:

	$ time ./scripts/igt_doc.py --check --config tests/xe/xe_*json

	real	0m12.085s
	user	0m10.468s
	sys	0m1.457s

Implent it on a different way, in order to speedup the build process.

After the change, it now takes less than 500ms:

	$ time ./scripts/igt_doc.py --check --config tests/xe/xe_*json

	real	0m0.460s
	user	0m0.357s
	sys	0m0.102s

It should be noticed that it is possible to speed it up even
further by running:

	 build/tests/* --list-subtests

tasks in parallel. I did some tests: the improvements weren't so
relevant on my notebook with 8 CPUs, and, on my tests with python 3.11
it seemed a little too fragile when using multithreading, as it was not
hard to mess with it, not having threads or subprocess forks not
happening, and without triggering an error.

As the benefit of such optimization was marginal, let's avoid the
additional complexity of multithreading.

---

v2:
  - remove a now uneeded loop

Mauro Carvalho Chehab (3):
  scripts/igt_doc.py: cleanup some pylint warnings
  scripts/igt_doc.py: don't depend on igt_runner anymore
  scripts/test_list.py: use a compiled regex for check

Mauro Carvalho Chehab (3):
  scripts/igt_doc.py: cleanup some pylint warnings
  scripts/igt_doc.py: don't depend on igt_runner anymore
  scripts/test_list.py: use a compiled regex for check

 docs/testplan/meson.build |  7 ++--
 scripts/igt_doc.py        | 11 +++---
 scripts/test_list.py      | 73 ++++++++++++++++++++++++++-------------
 3 files changed, 55 insertions(+), 36 deletions(-)

-- 
2.39.2

[igt-dev] [PATCH i-g-t v2 1/3] scripts/igt_doc.py: cleanup some pylint warnings

[Mauro Carvalho Chehab]

From: Mauro Carvalho Chehab <mchehab@kernel.org>

Ensure that pylint won't report any issues on igt_doc.py.

Signed-off-by: Mauro Carvalho Chehab <mchehab@kernel.org>
---
 scripts/igt_doc.py   | 1 +
 scripts/test_list.py | 4 ++--
 2 files changed, 3 insertions(+), 2 deletions(-)

diff --git a/scripts/igt_doc.py b/scripts/igt_doc.py
index 547cb81bce02..8fa5af15033e 100755
--- a/scripts/igt_doc.py
+++ b/scripts/igt_doc.py
@@ -11,6 +11,7 @@
 """Maintain test plan and test implementation documentation on IGT."""
 
 import argparse
+import sys
 
 from test_list import TestList
 
diff --git a/scripts/test_list.py b/scripts/test_list.py
index 6af1914822a3..d1b9a2794967 100755
--- a/scripts/test_list.py
+++ b/scripts/test_list.py
@@ -1,5 +1,5 @@
 #!/usr/bin/env python3
-# pylint: disable=C0301,R0902,R0914,R0912,R0915,R1702,C0302
+# pylint: disable=C0301,R0902,R0914,R0912,R0913,R0915,R1702,C0302
 # SPDX-License-Identifier: (GPL-2.0 OR MIT)
 
 ## Copyright (C) 2023    Intel Corporation                 ##
@@ -394,7 +394,7 @@ class TestList:
         for subtest in self.doc[test]["subtest"].keys():
             summary = test_name
             if self.doc[test]["subtest"][subtest]["Summary"] != '':
-                 summary += '@' + self.doc[test]["subtest"][subtest]["Summary"]
+                summary += '@' + self.doc[test]["subtest"][subtest]["Summary"]
             if not summary:
                 continue
 
-- 
2.39.2

[igt-dev] [PATCH i-g-t v2 2/3] scripts/igt_doc.py: don't depend on igt_runner anymore

[Mauro Carvalho Chehab]

From: Mauro Carvalho Chehab <mchehab@kernel.org>

There's no need to actually call IGT runner to get test lists.

Remove such dependency, in order to speedup --check.

Signed-off-by: Mauro Carvalho Chehab <mchehab@kernel.org>
---
 docs/testplan/meson.build |  7 ++----
 scripts/igt_doc.py        | 10 +++-----
 scripts/test_list.py      | 53 ++++++++++++++++++++++++++-------------
 3 files changed, 41 insertions(+), 29 deletions(-)

diff --git a/docs/testplan/meson.build b/docs/testplan/meson.build
index ea9f6eeb7038..3347f61876ef 100644
--- a/docs/testplan/meson.build
+++ b/docs/testplan/meson.build
@@ -11,11 +11,8 @@ xe_test_config = join_paths(source_root, 'tests', 'xe', 'xe_test_config.json')
 check_testlist = []
 if build_tests
 	doc_dependencies = test_executables
-	if jsonc.found()
-		# Check if documentation matches the actual tests
-		check_testlist = [ '--check-testlist', '--igt-build-path', build_root ]
-		doc_dependencies += runner
-	endif
+	# Check if documentation matches the actual tests
+	check_testlist = [ '--check-testlist', '--igt-build-path', build_root ]
 else
 	doc_dependencies = []
 endif
diff --git a/scripts/igt_doc.py b/scripts/igt_doc.py
index 8fa5af15033e..01df35f98ace 100755
--- a/scripts/igt_doc.py
+++ b/scripts/igt_doc.py
@@ -16,7 +16,6 @@ import sys
 from test_list import TestList
 
 IGT_BUILD_PATH = 'build'
-IGT_RUNNER = 'runner/igt_runner'
 
 parser = argparse.ArgumentParser(description = "Print formatted kernel documentation to stdout.",
                                  formatter_class = argparse.ArgumentDefaultsHelpFormatter,
@@ -36,24 +35,21 @@ parser.add_argument("--sort-field",
 parser.add_argument("--filter-field",
                     help="modify --show-subtests to filter output based a regex given by FILTER_FIELD=~'regex'")
 parser.add_argument("--check-testlist", action="store_true",
-                    help="Compare documentation against IGT runner testlist.")
+                    help="Compare documentation against IGT built tests.")
 parser.add_argument("--include-plan", action="store_true",
                     help="Include test plans, if any.")
 parser.add_argument("--igt-build-path",
-                    help="Path where the IGT runner is sitting. Used by --check-testlist.",
+                    help="Path to the IGT build directory. Used by --check-testlist.",
                     default=IGT_BUILD_PATH)
 parser.add_argument("--gen-testlist",
                     help="Generate documentation at the GEN_TESTLIST directory, using SORT_FIELD to split the tests. Requires --sort-field.")
-parser.add_argument("--igt-runner",
-                    help="Path where the IGT runner is sitting. Used by --check-testlist.",
-                    default=IGT_RUNNER)
 parser.add_argument('--files', nargs='+',
                     help="File name(s) to be processed")
 
 parse_args = parser.parse_args()
 
 tests = TestList(parse_args.config, parse_args.include_plan, parse_args.files,
-                 parse_args.igt_build_path, parse_args.igt_runner)
+                 parse_args.igt_build_path)
 
 RUN = 0
 if parse_args.show_subtests:
diff --git a/scripts/test_list.py b/scripts/test_list.py
index d1b9a2794967..287351e717b3 100755
--- a/scripts/test_list.py
+++ b/scripts/test_list.py
@@ -245,7 +245,7 @@ class TestList:
     """
 
     def __init__(self, config_fname, include_plan = False, file_list = False,
-                 igt_build_path = None, igt_runner = None):
+                 igt_build_path = None):
         self.doc = {}
         self.test_number = 0
         self.config = None
@@ -254,7 +254,6 @@ class TestList:
         self.props = {}
         self.config_fname = config_fname
         self.igt_build_path = igt_build_path
-        self.igt_runner = igt_runner
         self.level_count = 0
         self.field_list = {}
         self.title = None
@@ -766,6 +765,38 @@ class TestList:
 
         return subtests
 
+    def __get_testlist(self, name):
+        match = re.match(r"(.*/)?(.*)\.c$", name)
+        if not match:
+            return []
+
+        basename = "igt@" + match.group(2)
+
+        fname = os.path.join(self.igt_build_path, "tests", match.group(2))
+        if not os.path.isfile(fname):
+            print(f"Error: file {fname} doesn't exist.")
+            sys.exit(1)
+        try:
+            result = subprocess.run([ fname, "--list-subtests" ],
+                                    check = True,
+                                    stdout = subprocess.PIPE,
+                                    universal_newlines=True)
+            subtests = result.stdout.splitlines()
+
+            return [basename  + "@" + i for i in subtests]
+        except subprocess.CalledProcessError:
+            # Handle it as a test using igt_simple_main
+            return [basename]
+
+    def get_testlist(self):
+
+        """ Return a list of tests as reported by --list-subtests """
+        tests = []
+        for name in self.filenames:
+            tests += self.__get_testlist(name)
+
+        return sorted(tests)
+
     #
     # Validation methods
     #
@@ -773,28 +804,16 @@ class TestList:
 
         """Compare documented subtests with the IGT test list"""
 
-        if not self.igt_build_path or not self.igt_runner:
-            sys.exit("Need the IGT build path and igt_runner executable file name")
+        if not self.igt_build_path:
+            sys.exit("Need the IGT build path")
 
         doc_subtests = sorted(self.get_subtests()[""])
 
         for i in range(0, len(doc_subtests)): # pylint: disable=C0200
             doc_subtests[i] = re.sub(r'\<[^\>]+\>', r'\\d+', doc_subtests[i])
 
-        test_prefix = os.path.commonprefix(doc_subtests)
-
         # Get a list of tests from
-        try:
-            result = subprocess.run([ f"{self.igt_build_path}/{self.igt_runner}",
-                                    "-L", "-t",  test_prefix,
-                                    f"{self.igt_build_path}/tests"], check = True,
-                                    stdout=subprocess.PIPE, universal_newlines=True)
-        except subprocess.CalledProcessError as sub_err:
-            print(sub_err.stderr)
-            print("Error:", sub_err)
-            sys.exit(1)
-
-        run_subtests = sorted(result.stdout.splitlines())
+        run_subtests = self.get_testlist()
 
         # Compare arrays
 
-- 
2.39.2

[igt-dev] [PATCH i-g-t v2 3/3] scripts/test_list.py: use a compiled regex for check

[Mauro Carvalho Chehab]

From: Mauro Carvalho Chehab <mchehab@kernel.org>

Speed up even further the logic for --check-testlist by
pre-compiling the regex that will be used to validate the
results.

Signed-off-by: Mauro Carvalho Chehab <mchehab@kernel.org>
---
 scripts/test_list.py | 16 +++++++++++-----
 1 file changed, 11 insertions(+), 5 deletions(-)

diff --git a/scripts/test_list.py b/scripts/test_list.py
index 287351e717b3..96d8883db122 100755
--- a/scripts/test_list.py
+++ b/scripts/test_list.py
@@ -820,10 +820,18 @@ class TestList:
         run_missing = []
         doc_uneeded = []
 
+        test_regex = r""
+        for doc_test in doc_subtests:
+            if test_regex != r"":
+                test_regex += r"|"
+            test_regex += r'^' + doc_test + r'$'
+
+        test_regex = re.compile(test_regex)
+
         for doc_test in doc_subtests:
             found = False
             for run_test in run_subtests:
-                if re.match(r'^' + doc_test + r'$', run_test):
+                if re.match(test_regex, run_test):
                     found = True
                     break
             if not found:
@@ -831,10 +839,8 @@ class TestList:
 
         for run_test in run_subtests:
             found = False
-            for doc_test in doc_subtests:
-                if re.match(r'^' + doc_test + r'$', run_test):
-                    found = True
-                    break
+            if re.match(test_regex, run_test):
+                found = True
             if not found:
                 run_missing.append(run_test)
 
-- 
2.39.2

[igt-dev] ✗ Fi.CI.BAT: failure for ./scripts/igt_doc.py: speedup check logic (rev2)

[Patchwork]

[-- Attachment #1: Type: text/plain, Size: 7931 bytes --]

== Series Details ==

Series: ./scripts/igt_doc.py: speedup check logic (rev2)
URL   : https://patchwork.freedesktop.org/series/116379/
State : failure

== Summary ==

CI Bug Log - changes from CI_DRM_12995 -> IGTPW_8790
====================================================

Summary
-------

  **FAILURE**

  Serious unknown changes coming with IGTPW_8790 absolutely need to be
  verified manually.
  
  If you think the reported changes have nothing to do with the changes
  introduced in IGTPW_8790, please notify your bug team to allow them
  to document this new failure mode, which will reduce false positives in CI.

  External URL: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_8790/index.html

Participating hosts (37 -> 35)
------------------------------

  Missing    (2): fi-kbl-soraka fi-snb-2520m 

Possible new issues
-------------------

  Here are the unknown changes that may have been introduced in IGTPW_8790:

### IGT changes ###

#### Possible regressions ####

  * igt@i915_selftest@live@hangcheck:
    - fi-hsw-4770:        [PASS][1] -> [DMESG-WARN][2]
   [1]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_12995/fi-hsw-4770/igt@i915_selftest@live@hangcheck.html
   [2]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_8790/fi-hsw-4770/igt@i915_selftest@live@hangcheck.html

  
Known issues
------------

  Here are the changes found in IGTPW_8790 that come from known issues:

### IGT changes ###

#### Issues hit ####

  * igt@i915_pm_rps@basic-api:
    - bat-dg2-11:         [PASS][3] -> [FAIL][4] ([i915#8308])
   [3]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_12995/bat-dg2-11/igt@i915_pm_rps@basic-api.html
   [4]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_8790/bat-dg2-11/igt@i915_pm_rps@basic-api.html

  * igt@i915_selftest@live@gt_heartbeat:
    - fi-glk-j4005:       [PASS][5] -> [DMESG-FAIL][6] ([i915#5334])
   [5]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_12995/fi-glk-j4005/igt@i915_selftest@live@gt_heartbeat.html
   [6]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_8790/fi-glk-j4005/igt@i915_selftest@live@gt_heartbeat.html
    - fi-skl-guc:         [PASS][7] -> [DMESG-FAIL][8] ([i915#5334])
   [7]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_12995/fi-skl-guc/igt@i915_selftest@live@gt_heartbeat.html
   [8]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_8790/fi-skl-guc/igt@i915_selftest@live@gt_heartbeat.html

  * igt@i915_selftest@live@hangcheck:
    - bat-dg2-11:         [PASS][9] -> [ABORT][10] ([i915#7913] / [i915#7979])
   [9]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_12995/bat-dg2-11/igt@i915_selftest@live@hangcheck.html
   [10]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_8790/bat-dg2-11/igt@i915_selftest@live@hangcheck.html

  * igt@i915_selftest@live@migrate:
    - bat-dg2-11:         [PASS][11] -> [DMESG-WARN][12] ([i915#7699])
   [11]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_12995/bat-dg2-11/igt@i915_selftest@live@migrate.html
   [12]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_8790/bat-dg2-11/igt@i915_selftest@live@migrate.html

  * igt@i915_selftest@live@reset:
    - bat-rpls-2:         [PASS][13] -> [ABORT][14] ([i915#4983] / [i915#7913] / [i915#7981])
   [13]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_12995/bat-rpls-2/igt@i915_selftest@live@reset.html
   [14]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_8790/bat-rpls-2/igt@i915_selftest@live@reset.html

  * igt@i915_selftest@live@slpc:
    - bat-rpls-1:         [PASS][15] -> [DMESG-FAIL][16] ([i915#6367] / [i915#7996])
   [15]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_12995/bat-rpls-1/igt@i915_selftest@live@slpc.html
   [16]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_8790/bat-rpls-1/igt@i915_selftest@live@slpc.html
    - bat-adln-1:         NOTRUN -> [DMESG-FAIL][17] ([i915#6997])
   [17]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_8790/bat-adln-1/igt@i915_selftest@live@slpc.html

  * igt@kms_chamelium_hpd@common-hpd-after-suspend:
    - bat-adln-1:         NOTRUN -> [SKIP][18] ([i915#7828])
   [18]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_8790/bat-adln-1/igt@kms_chamelium_hpd@common-hpd-after-suspend.html
    - bat-rpls-1:         NOTRUN -> [SKIP][19] ([i915#7828])
   [19]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_8790/bat-rpls-1/igt@kms_chamelium_hpd@common-hpd-after-suspend.html

  * igt@kms_pipe_crc_basic@read-crc:
    - bat-adlp-9:         NOTRUN -> [SKIP][20] ([i915#3546]) +1 similar issue
   [20]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_8790/bat-adlp-9/igt@kms_pipe_crc_basic@read-crc.html

  * igt@kms_pipe_crc_basic@suspend-read-crc:
    - bat-rpls-1:         NOTRUN -> [SKIP][21] ([i915#1845])
   [21]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_8790/bat-rpls-1/igt@kms_pipe_crc_basic@suspend-read-crc.html

  
#### Possible fixes ####

  * igt@gem_exec_suspend@basic-s3@smem:
    - bat-rpls-1:         [ABORT][22] ([i915#6687] / [i915#7978]) -> [PASS][23]
   [22]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_12995/bat-rpls-1/igt@gem_exec_suspend@basic-s3@smem.html
   [23]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_8790/bat-rpls-1/igt@gem_exec_suspend@basic-s3@smem.html

  * igt@i915_module_load@load:
    - fi-elk-e7500:       [DMESG-WARN][24] ([i915#4391]) -> [PASS][25]
   [24]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_12995/fi-elk-e7500/igt@i915_module_load@load.html
   [25]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_8790/fi-elk-e7500/igt@i915_module_load@load.html

  * igt@i915_selftest@live@gt_lrc:
    - bat-adln-1:         [INCOMPLETE][26] ([i915#7609]) -> [PASS][27]
   [26]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_12995/bat-adln-1/igt@i915_selftest@live@gt_lrc.html
   [27]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_8790/bat-adln-1/igt@i915_selftest@live@gt_lrc.html

  * igt@kms_pipe_crc_basic@nonblocking-crc-frame-sequence@pipe-d-dp-1:
    - bat-dg2-8:          [FAIL][28] ([i915#7932]) -> [PASS][29]
   [28]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_12995/bat-dg2-8/igt@kms_pipe_crc_basic@nonblocking-crc-frame-sequence@pipe-d-dp-1.html
   [29]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_8790/bat-dg2-8/igt@kms_pipe_crc_basic@nonblocking-crc-frame-sequence@pipe-d-dp-1.html

  
  [i915#1845]: https://gitlab.freedesktop.org/drm/intel/issues/1845
  [i915#3546]: https://gitlab.freedesktop.org/drm/intel/issues/3546
  [i915#4391]: https://gitlab.freedesktop.org/drm/intel/issues/4391
  [i915#4983]: https://gitlab.freedesktop.org/drm/intel/issues/4983
  [i915#5334]: https://gitlab.freedesktop.org/drm/intel/issues/5334
  [i915#6367]: https://gitlab.freedesktop.org/drm/intel/issues/6367
  [i915#6687]: https://gitlab.freedesktop.org/drm/intel/issues/6687
  [i915#6997]: https://gitlab.freedesktop.org/drm/intel/issues/6997
  [i915#7609]: https://gitlab.freedesktop.org/drm/intel/issues/7609
  [i915#7699]: https://gitlab.freedesktop.org/drm/intel/issues/7699
  [i915#7828]: https://gitlab.freedesktop.org/drm/intel/issues/7828
  [i915#7913]: https://gitlab.freedesktop.org/drm/intel/issues/7913
  [i915#7932]: https://gitlab.freedesktop.org/drm/intel/issues/7932
  [i915#7978]: https://gitlab.freedesktop.org/drm/intel/issues/7978
  [i915#7979]: https://gitlab.freedesktop.org/drm/intel/issues/7979
  [i915#7981]: https://gitlab.freedesktop.org/drm/intel/issues/7981
  [i915#7996]: https://gitlab.freedesktop.org/drm/intel/issues/7996
  [i915#8308]: https://gitlab.freedesktop.org/drm/intel/issues/8308


Build changes
-------------

  * CI: CI-20190529 -> None
  * IGT: IGT_7251 -> IGTPW_8790

  CI-20190529: 20190529
  CI_DRM_12995: 0c5be3027d7ac4a8b107bf1113867744aa3bb440 @ git://anongit.freedesktop.org/gfx-ci/linux
  IGTPW_8790: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_8790/index.html
  IGT_7251: 55fa959aad79b3771350a801c1c2dbd4e5034102 @ https://gitlab.freedesktop.org/drm/igt-gpu-tools.git

== Logs ==

For more details see: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_8790/index.html

[-- Attachment #2: Type: text/html, Size: 9134 bytes --]