[igt-dev] [PATCH i-g-t 0/3] generate an ordered testlist for Xe driver

[igt-dev] [PATCH i-g-t 0/3] generate an ordered testlist for Xe driver

[Mauro Carvalho Chehab]

From: Mauro Carvalho Chehab <mchehab@kernel.org>

Add support for ordering tests at the documentation tool, and produce
Xe fast feedback testlists on an ordered way.

Mauro Carvalho Chehab (3):
  scripts/test_list.py: add support for ordering testlists
  xe_test_config.json: add an order for run type
  tests/xe: add run type fields to order testlists

 scripts/test_list.py         | 55 +++++++++++++++++++++++++++++++-----
 tests/xe/xe_live_ktest.c     |  2 +-
 tests/xe/xe_module_load.c    |  2 +-
 tests/xe/xe_test_config.json |  7 ++++-
 4 files changed, 56 insertions(+), 10 deletions(-)

-- 
2.40.1

[igt-dev] [PATCH i-g-t 1/3] scripts/test_list.py: add support for ordering testlists

[Mauro Carvalho Chehab]

From: Mauro Carvalho Chehab <mchehab@kernel.org>

Testlists usually require an special order, like:
- an initial test which loads the driver;
- normal tests;
- dangerous tests (like KUnit ones).

Add support to handle order, by adding an "order" property at
the config file.

Signed-off-by: Mauro Carvalho Chehab <mchehab@kernel.org>
---
 scripts/test_list.py | 55 ++++++++++++++++++++++++++++++++++++++------
 1 file changed, 48 insertions(+), 7 deletions(-)

diff --git a/scripts/test_list.py b/scripts/test_list.py
index f676024c9571..76fdcf3dbb4f 100755
--- a/scripts/test_list.py
+++ b/scripts/test_list.py
@@ -819,18 +819,25 @@ class TestList:
     # Subtest list methods
     #
 
-    def get_subtests(self, sort_field = None, expand = None):
+    def get_subtests(self, sort_field = None, expand = None, with_order = False):
 
         """Return an array with all subtests"""
 
         subtests = {}
         subtests[""] = []
 
+        order = None
+
         if sort_field:
             if sort_field.lower() not in self.field_list:
                 sys.exit(f"Field '{sort_field}' is not defined")
             sort_field = self.field_list[sort_field.lower()]
 
+            if with_order:
+                if "_properties_" in self.props[sort_field]:
+                    if "order" in self.props[sort_field]["_properties_"]:
+                        order = self.props[sort_field]["_properties_"]["order"]
+
         for test in sorted(self.doc.keys()):
             fname = self.doc[test]["File"]
 
@@ -848,20 +855,54 @@ class TestList:
                     if sort_field in subtest:
                         if expand:
                             test_list = subtest[sort_field].split(expand)
+                            test_list = [s.strip() for s in test_list]
+
                             for test_elem in test_list:
-                                test_elem = test_elem.strip()
                                 if test_elem not in subtests:
                                     subtests[test_elem] = []
-                                subtests[test_elem].append(subtest["Summary"])
+                                if order:
+                                    subtests[test_elem].append((subtest["Summary"], test_list))
+                                else:
+                                    subtests[test_elem].append(subtest["Summary"])
                         else:
                             if subtest[sort_field] not in subtests:
                                 subtests[subtest[sort_field]] = []
-                            subtests[subtest[sort_field]].append(subtest["Summary"])
+                                if order:
+                                    subtests[test_elem].append((subtest["Summary"], [subtest[sort_field]]))
+                                else:
+                                    subtests[subtest[sort_field]].append(subtest["Summary"])
+                    else:
+                        if order:
+                            subtests[test_elem].append((subtest["Summary"], [subtest[sort_field]]))
+                        else:
+                            subtests[""].append(subtest["Summary"])
+
+                else:
+                    if order:
+                        subtests[test_elem].append((subtest["Summary"], [subtest[sort_field]]))
                     else:
                         subtests[""].append(subtest["Summary"])
 
-                else:
-                    subtests[""].append(subtest["Summary"])
+        if order:
+            for group, tests in subtests.items():
+                prefix_tests = []
+                suffix_tests = []
+                middle_tests = []
+                is_prefix = True
+                for k in order:
+                    if k == "__all__":
+                        is_prefix = False
+                        continue
+                    for test in tests:
+                        if k in test[1]:
+                            if is_prefix:
+                                prefix_tests.append(test[0])
+                            else:
+                                suffix_tests.append(test[0])
+                for test in tests:
+                    if test[0] not in prefix_tests and test[0] not in suffix_tests:
+                        middle_tests.append(test[0])
+                subtests[group] = prefix_tests + middle_tests + suffix_tests
 
         return subtests
 
@@ -1203,7 +1244,7 @@ class TestList:
         test_prefix = re.sub(r'^igt@', '', test_prefix)
 
         # NOTE: currently, it uses a comma for multi-value delimitter
-        test_subtests = self.get_subtests(sort_field, ",")
+        test_subtests = self.get_subtests(sort_field, ",", with_order = True)
 
         if not os.path.exists(directory):
             os.makedirs(directory)
-- 
2.40.1

[igt-dev] [PATCH i-g-t 2/3] xe_test_config.json: add an order for run type

[Mauro Carvalho Chehab]

From: Mauro Carvalho Chehab <mchehab@kernel.org>

When generating test lists based on run type for the Xe driver,
order them in a way that the first test will be the one called
after booting the machine (e. g. module load), then normal tests
and finally KUnit ones.

Signed-off-by: Mauro Carvalho Chehab <mchehab@kernel.org>
---
 tests/xe/xe_test_config.json | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)

diff --git a/tests/xe/xe_test_config.json b/tests/xe/xe_test_config.json
index 334d372d1330..e89f998bc66b 100644
--- a/tests/xe/xe_test_config.json
+++ b/tests/xe/xe_test_config.json
@@ -22,7 +22,12 @@
                     },
                     "Run type": {
                         "_properties_": {
-                            "description": "Defines what category of testlist it belongs"
+                            "description": "Defines what category of testlist it belongs",
+                            "order": [
+                                "boot",
+                                "__all__",
+                                "kunit"
+                            ]
                         }
                     }
                 }
-- 
2.40.1

[igt-dev] [PATCH i-g-t 3/3] tests/xe: add run type fields to order testlists

[Mauro Carvalho Chehab]

From: Mauro Carvalho Chehab <mchehab@kernel.org>

Order tests at the fast feedback testlist, when generating
testlists based on Run type field:

- add boot to xe_module_load, to run such test first when producing
  a fast feedback testlist;
- place kunit tests at the end of the fast feedback testlist, as they
  have a higher chance of causing GPU hangups.

Signed-off-by: Mauro Carvalho Chehab <mchehab@kernel.org>
---
 tests/xe/xe_live_ktest.c  | 2 +-
 tests/xe/xe_module_load.c | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)

diff --git a/tests/xe/xe_live_ktest.c b/tests/xe/xe_live_ktest.c
index 7dcf679069b8..f0cbf4f623bc 100644
--- a/tests/xe/xe_live_ktest.c
+++ b/tests/xe/xe_live_ktest.c
@@ -8,7 +8,7 @@
  * Sub-category: kunit
  * Functionality: kunit
  * Test category: functionality test
- * Run type: BAT
+ * Run type: BAT, kunit
  *
  * SUBTEST: bo
  * Functionality: bo
diff --git a/tests/xe/xe_module_load.c b/tests/xe/xe_module_load.c
index 1c2d4a2e2ea3..3b2dc1fdc90f 100644
--- a/tests/xe/xe_module_load.c
+++ b/tests/xe/xe_module_load.c
@@ -99,7 +99,7 @@ static const char * const unwanted_drivers[] = {
 /**
  * SUBTEST: force-load
  * Description: Load the Xe driver passing ``force_probe=*`` parameter
- * Run type: BAT
+ * Run type: BAT, boot
  *
  * SUBTEST: load
  * Description: Load the Xe driver
-- 
2.40.1

[igt-dev] ✗ Fi.CI.BAT: failure for generate an ordered testlist for Xe driver

[Patchwork]

[-- Attachment #1: Type: text/plain, Size: 7198 bytes --]

== Series Details ==

Series: generate an ordered testlist for Xe driver
URL   : https://patchwork.freedesktop.org/series/119570/
State : failure

== Summary ==

CI Bug Log - changes from IGT_7339 -> IGTPW_9214
====================================================

Summary
-------

  **FAILURE**

  Serious unknown changes coming with IGTPW_9214 absolutely need to be
  verified manually.
  
  If you think the reported changes have nothing to do with the changes
  introduced in IGTPW_9214, please notify your bug team to allow them
  to document this new failure mode, which will reduce false positives in CI.

  External URL: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_9214/index.html

Participating hosts (44 -> 43)
------------------------------

  Missing    (1): fi-snb-2520m 

Possible new issues
-------------------

  Here are the unknown changes that may have been introduced in IGTPW_9214:

### IGT changes ###

#### Possible regressions ####

  * igt@i915_suspend@basic-s2idle-without-i915:
    - bat-mtlp-6:         [PASS][1] -> [INCOMPLETE][2]
   [1]: https://intel-gfx-ci.01.org/tree/drm-tip/IGT_7339/bat-mtlp-6/igt@i915_suspend@basic-s2idle-without-i915.html
   [2]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_9214/bat-mtlp-6/igt@i915_suspend@basic-s2idle-without-i915.html

  
Known issues
------------

  Here are the changes found in IGTPW_9214 that come from known issues:

### IGT changes ###

#### Issues hit ####

  * igt@i915_selftest@live@gt_heartbeat:
    - fi-kbl-soraka:      [PASS][3] -> [DMESG-FAIL][4] ([i915#5334] / [i915#7872])
   [3]: https://intel-gfx-ci.01.org/tree/drm-tip/IGT_7339/fi-kbl-soraka/igt@i915_selftest@live@gt_heartbeat.html
   [4]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_9214/fi-kbl-soraka/igt@i915_selftest@live@gt_heartbeat.html

  * igt@i915_selftest@live@gt_mocs:
    - bat-mtlp-8:         [PASS][5] -> [DMESG-FAIL][6] ([i915#7059])
   [5]: https://intel-gfx-ci.01.org/tree/drm-tip/IGT_7339/bat-mtlp-8/igt@i915_selftest@live@gt_mocs.html
   [6]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_9214/bat-mtlp-8/igt@i915_selftest@live@gt_mocs.html

  * igt@i915_selftest@live@reset:
    - bat-rpls-1:         [PASS][7] -> [ABORT][8] ([i915#4983] / [i915#7461] / [i915#8347] / [i915#8384])
   [7]: https://intel-gfx-ci.01.org/tree/drm-tip/IGT_7339/bat-rpls-1/igt@i915_selftest@live@reset.html
   [8]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_9214/bat-rpls-1/igt@i915_selftest@live@reset.html

  * igt@i915_selftest@live@slpc:
    - bat-rpls-2:         NOTRUN -> [DMESG-WARN][9] ([i915#6367])
   [9]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_9214/bat-rpls-2/igt@i915_selftest@live@slpc.html

  * igt@i915_suspend@basic-s2idle-without-i915:
    - bat-rpls-2:         NOTRUN -> [ABORT][10] ([i915#6687] / [i915#8668])
   [10]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_9214/bat-rpls-2/igt@i915_suspend@basic-s2idle-without-i915.html

  * igt@kms_chamelium_hpd@common-hpd-after-suspend:
    - bat-dg2-11:         NOTRUN -> [SKIP][11] ([i915#7828])
   [11]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_9214/bat-dg2-11/igt@kms_chamelium_hpd@common-hpd-after-suspend.html

  * igt@kms_pipe_crc_basic@nonblocking-crc-frame-sequence:
    - bat-dg2-11:         NOTRUN -> [SKIP][12] ([i915#1845] / [i915#5354]) +3 similar issues
   [12]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_9214/bat-dg2-11/igt@kms_pipe_crc_basic@nonblocking-crc-frame-sequence.html

  
#### Possible fixes ####

  * igt@gem_exec_suspend@basic-s3@smem:
    - fi-rkl-11600:       [FAIL][13] ([fdo#103375]) -> [PASS][14]
   [13]: https://intel-gfx-ci.01.org/tree/drm-tip/IGT_7339/fi-rkl-11600/igt@gem_exec_suspend@basic-s3@smem.html
   [14]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_9214/fi-rkl-11600/igt@gem_exec_suspend@basic-s3@smem.html

  * igt@i915_selftest@live@gt_lrc:
    - bat-dg2-11:         [INCOMPLETE][15] ([i915#7609] / [i915#7913]) -> [PASS][16]
   [15]: https://intel-gfx-ci.01.org/tree/drm-tip/IGT_7339/bat-dg2-11/igt@i915_selftest@live@gt_lrc.html
   [16]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_9214/bat-dg2-11/igt@i915_selftest@live@gt_lrc.html

  * igt@i915_selftest@live@requests:
    - bat-mtlp-8:         [DMESG-FAIL][17] ([i915#8497]) -> [PASS][18]
   [17]: https://intel-gfx-ci.01.org/tree/drm-tip/IGT_7339/bat-mtlp-8/igt@i915_selftest@live@requests.html
   [18]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_9214/bat-mtlp-8/igt@i915_selftest@live@requests.html

  * igt@i915_selftest@live@reset:
    - bat-rpls-2:         [ABORT][19] ([i915#4983] / [i915#7461] / [i915#7913] / [i915#8347]) -> [PASS][20]
   [19]: https://intel-gfx-ci.01.org/tree/drm-tip/IGT_7339/bat-rpls-2/igt@i915_selftest@live@reset.html
   [20]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_9214/bat-rpls-2/igt@i915_selftest@live@reset.html

  * igt@i915_selftest@live@slpc:
    - bat-mtlp-6:         [DMESG-WARN][21] ([i915#6367]) -> [PASS][22]
   [21]: https://intel-gfx-ci.01.org/tree/drm-tip/IGT_7339/bat-mtlp-6/igt@i915_selftest@live@slpc.html
   [22]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_9214/bat-mtlp-6/igt@i915_selftest@live@slpc.html

  
#### Warnings ####

  * igt@i915_module_load@load:
    - bat-adlp-11:        [DMESG-WARN][23] ([i915#4423]) -> [ABORT][24] ([i915#4423])
   [23]: https://intel-gfx-ci.01.org/tree/drm-tip/IGT_7339/bat-adlp-11/igt@i915_module_load@load.html
   [24]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_9214/bat-adlp-11/igt@i915_module_load@load.html

  
  [fdo#103375]: https://bugs.freedesktop.org/show_bug.cgi?id=103375
  [i915#1845]: https://gitlab.freedesktop.org/drm/intel/issues/1845
  [i915#4423]: https://gitlab.freedesktop.org/drm/intel/issues/4423
  [i915#4983]: https://gitlab.freedesktop.org/drm/intel/issues/4983
  [i915#5334]: https://gitlab.freedesktop.org/drm/intel/issues/5334
  [i915#5354]: https://gitlab.freedesktop.org/drm/intel/issues/5354
  [i915#6367]: https://gitlab.freedesktop.org/drm/intel/issues/6367
  [i915#6687]: https://gitlab.freedesktop.org/drm/intel/issues/6687
  [i915#7059]: https://gitlab.freedesktop.org/drm/intel/issues/7059
  [i915#7461]: https://gitlab.freedesktop.org/drm/intel/issues/7461
  [i915#7609]: https://gitlab.freedesktop.org/drm/intel/issues/7609
  [i915#7828]: https://gitlab.freedesktop.org/drm/intel/issues/7828
  [i915#7872]: https://gitlab.freedesktop.org/drm/intel/issues/7872
  [i915#7913]: https://gitlab.freedesktop.org/drm/intel/issues/7913
  [i915#8347]: https://gitlab.freedesktop.org/drm/intel/issues/8347
  [i915#8384]: https://gitlab.freedesktop.org/drm/intel/issues/8384
  [i915#8497]: https://gitlab.freedesktop.org/drm/intel/issues/8497
  [i915#8668]: https://gitlab.freedesktop.org/drm/intel/issues/8668


Build changes
-------------

  * CI: CI-20190529 -> None
  * IGT: IGT_7339 -> IGTPW_9214

  CI-20190529: 20190529
  CI_DRM_13290: d8109039969700315ed327adfab8732c3c882bd6 @ git://anongit.freedesktop.org/gfx-ci/linux
  IGTPW_9214: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_9214/index.html
  IGT_7339: 4476edb0fb09b886f7e667d1174d13299d98c23f @ https://gitlab.freedesktop.org/drm/igt-gpu-tools.git

== Logs ==

For more details see: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_9214/index.html

[-- Attachment #2: Type: text/html, Size: 8525 bytes --]