[igt-dev] [PATCH i-g-t v2 0/7] better sync test_list.py with igt_runner

[igt-dev] [PATCH i-g-t v2 0/7] better sync test_list.py with igt_runner

[Mauro Carvalho Chehab]

From: Mauro Carvalho Chehab <mchehab@kernel.org>

This patch series contains the patches from:
	https://patchwork.freedesktop.org/series/125401/

It contains some additional changes meant to sync how IGT
runner, documentation tools and CI works with regular expressions:

- the IGT runner regular expressions are now case-insensitive.
  As there are several KMS tests have upper case letters to indicate
  port entries. There's no sense of having tests named:
  igt@foo-port-A being different than igt@foo-port-a. One of
  the runner tests were changed to only match if case is not
  sensitive on regular expressions;

- when igt_runner is in dry-run mode, it doesn't need root
  permissions;

- currently, test_list.py is considering testlists as regular
  expressions, but they're really a normal list of tests.
  Change the logic to handle it accordingly.

---

v2:
  - instead of modifying a test for case-insensitive regex, add a
    new subtest.

Mauro Carvalho Chehab (4):
  igt_runner: don't require root for dry_run
  scripts/test_list.py: better handle list of tests
  runner/job_list: use case-insensitive regular expressions
  runner/runner_tests: add a test to check case-insensitive match

Petri Latvala (3):
  runner: Normalize testlist entries that don't list subtests
  runner/runner_tests: Unit tests for binary-name-only testlist entries
  runner/runner_tests: Fix name of no-subtests in unit tests

 runner/job_list.c                | 22 ++++++-
 runner/runner_tests.c            | 99 +++++++++++++++++++++++++++++++-
 runner/settings.c                |  3 +-
 scripts/test_list.py             | 54 ++++++++++++-----
 tests/intel/kms_test_config.json |  2 +-
 tests/intel/xe_test_config.json  |  1 +
 6 files changed, 161 insertions(+), 20 deletions(-)

-- 
2.41.0

[igt-dev] [PATCH i-g-t v2 1/7] runner: Normalize testlist entries that don't list subtests

[Mauro Carvalho Chehab]

From: Petri Latvala <adrinael@adrinael.net>

As the syntax for "all subtests" and "test that doesn't have subtests"
is the same, check the subtest listing when building the execution
plan. Doing that makes it possible to still have "all subtests" in
testlists (albeit not originally designed to support that) and enables
blocklisting particular subtests.

Signed-off-by: Petri Latvala <adrinael@adrinael.net>
Reviewed-by: Mauro Carvalho Chehab <mchehab@kernel.org>
Cc: Arkadiusz Hiler <arek@hiler.eu>
Cc: Kamil Konieczny <kamil.konieczny@linux.intel.com>
Cc: Juha-Pekka Heikkila <juhapekka.heikkila@gmail.com>
Cc: Bhanuprakash Modem <bhanuprakash.modem@intel.com>
Cc: Ashutosh Dixit <ashutosh.dixit@intel.com>
Closes: https://gitlab.freedesktop.org/drm/igt-gpu-tools/-/issues/149
---
 runner/job_list.c | 22 +++++++++++++++++++++-
 1 file changed, 21 insertions(+), 1 deletion(-)

diff --git a/runner/job_list.c b/runner/job_list.c
index e6ea836310fd..27cbb10bce56 100644
--- a/runner/job_list.c
+++ b/runner/job_list.c
@@ -230,8 +230,28 @@ static bool job_list_from_test_list(struct job_list *job_list,
 			continue;
 
 		if (sscanf(line, "igt@%ms", &binary) == 1) {
-			if ((delim = strchr(binary, '@')) != NULL)
+			if ((delim = strchr(binary, '@')) != NULL) {
 				*delim++ = '\0';
+			} else {
+				/*
+				 * No subtests specified. Check
+				 * whether the user means "all
+				 * subtests" or if the test doesn't
+				 * have any.
+				 */
+				if (entry.binary) {
+					/* First flush the entry we're building for multiple-mode */
+					add_job_list_entry(job_list, entry.binary, entry.subtests, entry.subtest_count);
+					memset(&entry, 0, sizeof(entry));
+					any = true;
+				}
+
+				add_subtests(job_list, settings, binary,
+					     &settings->include_regexes,
+					     &settings->exclude_regexes);
+				any = true;
+				continue;
+			}
 
 			if (!settings->multiple_mode) {
 				char **subtests = NULL;
-- 
2.41.0

[igt-dev] [PATCH i-g-t v2 2/7] runner/runner_tests: Unit tests for binary-name-only testlist entries

[Mauro Carvalho Chehab]

From: Petri Latvala <adrinael@adrinael.net>

Acked-by: Mauro Carvalho Chehab <mchehab@kernel.org>
Signed-off-by: Petri Latvala <adrinael@adrinael.net>
---
 runner/runner_tests.c | 70 +++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 70 insertions(+)

diff --git a/runner/runner_tests.c b/runner/runner_tests.c
index a7e968f85b91..e6024de11a60 100644
--- a/runner/runner_tests.c
+++ b/runner/runner_tests.c
@@ -1810,6 +1810,76 @@ igt_main
 		}
 	}
 
+	igt_subtest_group {
+		const char testlisttext[] = "igt@successtest";
+		const char blocktext[] = "igt@successtest@first";
+		struct job_list *list = malloc(sizeof(*list));
+		volatile int dirfd = -1;
+		char dirname[] = "tmpdirXXXXXX";
+		volatile int listfd;
+		volatile int blockfd;
+		char listfilename[] = "tmplistXXXXXX";
+		char blockfilename[] = "tmpblockXXXXXX";
+
+		igt_fixture {
+			igt_require(mkdtemp(dirname) != NULL);
+			rmdir(dirname);
+
+			igt_require((listfd = mkstemp(listfilename)) >= 0);
+			igt_require(write(listfd, testlisttext, strlen(testlisttext)) == strlen(testlisttext));
+			igt_require((blockfd = mkstemp(blockfilename)) >= 0);
+			igt_require(write(blockfd, blocktext, strlen(blocktext)) == strlen(blocktext));
+			close(listfd);
+			close(blockfd);
+
+			init_job_list(list);
+		}
+
+		igt_subtest("only-binary-name-in-testlist") {
+			const char *argv[] = { "runner",
+				"--allow-non-root",
+				"--test-list", listfilename,
+				testdatadir,
+				dirname,
+			};
+
+			igt_assert(parse_options(ARRAY_SIZE(argv), (char**)argv, settings));
+
+			igt_assert(create_job_list(list, settings));
+			/* No multi-mode, binary has two subtests, should be normalized to have individual subtests */
+			igt_assert_eq(list->size, 2);
+			igt_assert_eq(list->entries[0].subtest_count, 1);
+			igt_assert_eq(list->entries[1].subtest_count, 1);
+		}
+
+		igt_subtest("only-binary-name-in-testlist-with-blocks") {
+			const char *argv[] = { "runner",
+				"--allow-non-root",
+				"--test-list", listfilename,
+				"-b", blockfilename,
+				testdatadir,
+				dirname,
+			};
+
+			igt_assert(parse_options(ARRAY_SIZE(argv), (char**)argv, settings));
+
+			igt_assert(create_job_list(list, settings));
+			/* No multi-mode, binary has two subtests, one of them blocked */
+			igt_assert_eq(list->size, 1);
+			igt_assert_eq(list->entries[0].subtest_count, 1);
+			igt_assert_eqstr(list->entries[0].subtests[0], "second-subtest");
+		}
+
+		igt_fixture {
+			unlink(listfilename);
+			unlink(blockfilename);
+			close(dirfd);
+			clear_directory(dirname);
+			free_job_list(list);
+			free(list);
+		}
+	}
+
 	igt_subtest_group {
 		struct job_list *list = malloc(sizeof(*list));
 		volatile int dirfd = -1;
-- 
2.41.0

[igt-dev] [PATCH i-g-t v2 3/7] runner/runner_tests: Fix name of no-subtests in unit tests

[Mauro Carvalho Chehab]

From: Petri Latvala <adrinael@adrinael.net>

Now that runner normalizes no-subtest entries in testlists, make sure
the name matches what's available.

Acked-by: Mauro Carvalho Chehab <mchehab@kernel.org>
Signed-off-by: Petri Latvala <adrinael@adrinael.net>
---
 runner/runner_tests.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/runner/runner_tests.c b/runner/runner_tests.c
index e6024de11a60..11ff05bc0e7e 100644
--- a/runner/runner_tests.c
+++ b/runner/runner_tests.c
@@ -802,7 +802,7 @@ igt_main
 		char filename[] = "tmplistXXXXXX";
 		const char testlisttext[] = "igt@successtest@first-subtest\n"
 			"igt@successtest@second-subtest\n"
-			"igt@nosubtests\n";
+			"igt@no-subtests\n";
 		int multiple;
 		struct job_list *list = malloc(sizeof(*list));
 
@@ -830,7 +830,7 @@ igt_main
 
 				igt_assert_eqstr(list->entries[0].binary, "successtest");
 				if (!multiple) igt_assert_eqstr(list->entries[1].binary, "successtest");
-				igt_assert_eqstr(list->entries[multiple ? 1 : 2].binary, "nosubtests");
+				igt_assert_eqstr(list->entries[multiple ? 1 : 2].binary, "no-subtests");
 
 				igt_assert_eq(list->entries[0].subtest_count, multiple ? 2 : 1);
 				igt_assert_eq(list->entries[1].subtest_count, multiple ? 0 : 1);
-- 
2.41.0

[igt-dev] [PATCH i-g-t v2 4/7] igt_runner: don't require root for dry_run

[Mauro Carvalho Chehab]

From: Mauro Carvalho Chehab <mchehab@kernel.org>

When --dry-run option is used, there's no need to require root,
as no tests will actually be executed.

Reviewed-by: Kamil Konieczny <kamil.konieczny@linux.intel.com>
Signed-off-by: Mauro Carvalho Chehab <mchehab@kernel.org>
---
 runner/settings.c | 1 +
 1 file changed, 1 insertion(+)

diff --git a/runner/settings.c b/runner/settings.c
index 23aa82963344..788957948b4f 100644
--- a/runner/settings.c
+++ b/runner/settings.c
@@ -687,6 +687,7 @@ bool parse_options(int argc, char **argv,
 			break;
 		case OPT_DRY_RUN:
 			settings->dry_run = true;
+			settings->allow_non_root = true;
 			break;
 		case OPT_ALLOW_NON_ROOT:
 			settings->allow_non_root = true;
-- 
2.41.0

[igt-dev] [PATCH i-g-t v2 5/7] scripts/test_list.py: better handle list of tests

[Mauro Carvalho Chehab]

From: Mauro Carvalho Chehab <mchehab@kernel.org>

Not all test lists are regular expressions. Some are just
names. That's the case of IGT runner testlist files.

Add support for that.

In practice, this change will produce the same testlists
as before, but using a logic closer to what igt_runner()
does.

Reviewed-by: Kamil Konieczny <kamil.konieczny@linux.intel.com>
Signed-off-by: Mauro Carvalho Chehab <mchehab@kernel.org>
---
 scripts/test_list.py             | 54 +++++++++++++++++++++++---------
 tests/intel/kms_test_config.json |  2 +-
 tests/intel/xe_test_config.json  |  1 +
 3 files changed, 41 insertions(+), 16 deletions(-)

diff --git a/scripts/test_list.py b/scripts/test_list.py
index e54873f07e20..06215242107a 100644
--- a/scripts/test_list.py
+++ b/scripts/test_list.py
@@ -319,10 +319,6 @@ class TestList:
             self.props["Class"]["_properties_"]["level"] = 1
             self.props["Class"]["_properties_"]["sublevel"] = sublevel_count[0] + 1
 
-        flags = 0
-        if self.config.get("case_insensitive_testlist", False):
-            flags |= re.IGNORECASE
-
         # Remove non-multilevel items, as we're only interested on
         # hierarchical item levels here
         for field, item in self.props.items():
@@ -337,7 +333,7 @@ class TestList:
                 testlist = {}
                 for value in item["_properties_"]["include"]:
                     for name in value.keys():
-                        self.read_testlist(testlist, name, cfg_path + value[name], flags)
+                        self.read_testlist(field, item, testlist, name, cfg_path + value[name])
 
                 item["_properties_"]["include"] = testlist
 
@@ -346,7 +342,7 @@ class TestList:
                 testlist = {}
                 for value in item["_properties_"]["exclude"]:
                     for name in value.keys():
-                        self.read_testlist(testlist, name, cfg_path + value[name], flags)
+                        self.read_testlist(field, item, testlist, name, cfg_path + value[name])
 
                 item["_properties_"]["exclude"] = testlist
 
@@ -450,7 +446,22 @@ class TestList:
 
             self.__add_field(key, sublevel, hierarchy_level, field[key])
 
-    def read_testlist(self, testlist, name, filename, flags):
+    def read_testlist(self, field, item, testlist, name, filename):
+
+        match_type = item["_properties_"].get("match-type", "subtest-match")
+
+        match_type_regex = set(["regex", "regex-ignorecase"])
+        match_type_str = set(["subtest-match"])
+        match_types = match_type_regex | match_type_str
+
+        if match_type not in match_types:
+            sys.exit(f"Error: invalid match type '{match_type}' for {field}")
+
+        if match_type == "regex":
+            flags = 0
+        elif match_type == "regex-ignorecase":
+            flags = re.IGNORECASE
+
         base = r"^\s*({}[^\s\{}]+)(\S*)\s*(\#.*)?$"
         regex = re.compile(base.format(self.main_name, self.subtest_separator))
 
@@ -459,12 +470,25 @@ class TestList:
         with open(filename, 'r', newline = '', encoding = 'utf8') as fp:
             for line in fp:
                 match = regex.match(line)
-                if match:
-                    test = match.group(1)
-                    subtest = match.group(2)
-                    if not subtest.endswith("$"):
-                        subtest += r"(\@.*)?$"
-                    testlist[name].append(re.compile(f"{test}{subtest}", flags))
+                if not match:
+                    continue
+
+                test = match.group(1)
+                subtest = match.group(2)
+                test_name = f"{test}{subtest}"
+
+                if match_type in match_type_regex:
+                    testlist[name].append(re.compile(test_name, flags))
+                    continue
+
+                subtests = test_name.split(self.subtest_separator)[:3]
+
+                prefix=""
+                for subtest in subtests:
+                    subtest = prefix + subtest
+                    prefix = subtest + self.subtest_separator
+
+                testlist[name].append(re.compile(f"{subtest}(@.*)?"))
 
     def __filter_subtest(self, test, subtest, field_not_found_value):
 
@@ -504,7 +528,7 @@ class TestList:
             for names, regex_array in self.props[field]["_properties_"]["include"].items():
                 name = set(re.split(",\s*", names))
                 for regex in regex_array:
-                    if regex.match(testname):
+                    if regex.fullmatch(testname):
                         values.update(name)
                         break
 
@@ -514,7 +538,7 @@ class TestList:
                 for names, regex_array in self.props[field]["_properties_"]["exclude"].items():
                     deleted_names = set(re.split(",\s*", names))
                     for regex in regex_array:
-                        if regex.match(testname):
+                        if regex.fullmatch(testname):
                             if sorted(deleted_names) == sorted(values):
                                 set_full_if_empty = False
                             values.discard(deleted_names)
diff --git a/tests/intel/kms_test_config.json b/tests/intel/kms_test_config.json
index 40cf69dd0904..837380ee7fae 100644
--- a/tests/intel/kms_test_config.json
+++ b/tests/intel/kms_test_config.json
@@ -4,7 +4,6 @@
     "files": [ "../chamelium/kms_*.c", "../kms_*.c", "../testdisplay.c", "kms_*.c" ],
     "exclude_files": [ "../chamelium/kms_chamelium_helper.c", "../kms_color_helper.c",
                        "kms_dsc_helper.c" ],
-    "case_insensitive_testlist": true,
     "fields": {
         "Category": {
             "_properties_": {
@@ -24,6 +23,7 @@
                 "_properties_": {
                     "description": "Defines what category of testlist it belongs",
                     "default-testlist": "FULL",
+                    "match-type": "subtest-match",
                     "include": [
                         { "i915 BAT": "../intel-ci/fast-feedback.testlist" },
                         { "i915 BAT chamelium": "../intel-ci/fast-feedback-chamelium-only.testlist" },
diff --git a/tests/intel/xe_test_config.json b/tests/intel/xe_test_config.json
index 20faf73b7270..5fd7c888f274 100644
--- a/tests/intel/xe_test_config.json
+++ b/tests/intel/xe_test_config.json
@@ -33,6 +33,7 @@
                             "mandatory": true,
                             "description": "Defines what category of testlist it belongs",
                             "default-testlist": "FULL",
+                            "match-type": "subtest-match",
                             "include": [
                                 { "Xe BAT": "../intel-ci/xe-fast-feedback.testlist" }
                             ],
-- 
2.41.0

[igt-dev] [PATCH i-g-t v2 6/7] runner/job_list: use case-insensitive regular expressions

[Mauro Carvalho Chehab]

From: Mauro Carvalho Chehab <mchehab@kernel.org>

There are some KMS tests that are case-sensitive. Eventually,
those could be blocklisted with a different case, causing troubles
for CI runs.

As it makes no sense to have the same test name with different
cases, handle regular expressions in a case-insensitive way.

Reviewed-by: Kamil Konieczny <kamil.konieczny@linux.intel.com>
Signed-off-by: Mauro Carvalho Chehab <mchehab@kernel.org>
---
 runner/settings.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/runner/settings.c b/runner/settings.c
index 788957948b4f..42d8137f18e9 100644
--- a/runner/settings.c
+++ b/runner/settings.c
@@ -323,7 +323,7 @@ static bool add_regex(struct regex_list *list, char *new)
 	GRegex *regex;
 	GError *error = NULL;
 
-	regex = g_regex_new(new, G_REGEX_OPTIMIZE, 0, &error);
+	regex = g_regex_new(new, G_REGEX_CASELESS | G_REGEX_OPTIMIZE, 0, &error);
 	if (error) {
 		usage(stderr, "Invalid regex '%s': %s", new, error->message);
 		g_error_free(error);
-- 
2.41.0

[igt-dev] [PATCH i-g-t v2 7/7] runner/runner_tests: add a test to check case-insensitive match

[Mauro Carvalho Chehab]

From: Mauro Carvalho Chehab <mchehab@kernel.org>

As regular expressions are now case-insensitive, add ab
unit test to verify that case-insensitive logic is working
as expected.

CC: Petri Latvala <adrinael@adrinael.net>
Signed-off-by: Mauro Carvalho Chehab <mchehab@kernel.org>
---
 runner/runner_tests.c | 25 +++++++++++++++++++++++++
 1 file changed, 25 insertions(+)

diff --git a/runner/runner_tests.c b/runner/runner_tests.c
index 11ff05bc0e7e..0aa7dd6626b7 100644
--- a/runner/runner_tests.c
+++ b/runner/runner_tests.c
@@ -1813,13 +1813,16 @@ igt_main
 	igt_subtest_group {
 		const char testlisttext[] = "igt@successtest";
 		const char blocktext[] = "igt@successtest@first";
+		const char blocktext_upper[] = "igt@successTEST@first";
 		struct job_list *list = malloc(sizeof(*list));
 		volatile int dirfd = -1;
 		char dirname[] = "tmpdirXXXXXX";
 		volatile int listfd;
 		volatile int blockfd;
+		volatile int blockfd_upper;
 		char listfilename[] = "tmplistXXXXXX";
 		char blockfilename[] = "tmpblockXXXXXX";
+		char blockfilename_upper[] = "tmpBLOCKXXXXXX";
 
 		igt_fixture {
 			igt_require(mkdtemp(dirname) != NULL);
@@ -1829,8 +1832,11 @@ igt_main
 			igt_require(write(listfd, testlisttext, strlen(testlisttext)) == strlen(testlisttext));
 			igt_require((blockfd = mkstemp(blockfilename)) >= 0);
 			igt_require(write(blockfd, blocktext, strlen(blocktext)) == strlen(blocktext));
+			igt_require((blockfd_upper = mkstemp(blockfilename_upper)) >= 0);
+			igt_require(write(blockfd_upper, blocktext_upper, strlen(blocktext_upper)) == strlen(blocktext_upper));
 			close(listfd);
 			close(blockfd);
+			close(blockfd_upper);
 
 			init_job_list(list);
 		}
@@ -1870,9 +1876,28 @@ igt_main
 			igt_assert_eqstr(list->entries[0].subtests[0], "second-subtest");
 		}
 
+		igt_subtest("only-binary-name-in-testlist-with-case-insensitive-blocklist") {
+			const char *argv[] = { "runner",
+				"--allow-non-root",
+				"--test-list", listfilename,
+				"-b", blockfilename_upper,
+				testdatadir,
+				dirname,
+			};
+
+			igt_assert(parse_options(ARRAY_SIZE(argv), (char**)argv, settings));
+
+			igt_assert(create_job_list(list, settings));
+			/* No multi-mode, binary has two subtests, one of them blocked */
+			igt_assert_eq(list->size, 1);
+			igt_assert_eq(list->entries[0].subtest_count, 1);
+			igt_assert_eqstr(list->entries[0].subtests[0], "second-subtest");
+		}
+
 		igt_fixture {
 			unlink(listfilename);
 			unlink(blockfilename);
+			unlink(blockfilename_upper);
 			close(dirfd);
 			clear_directory(dirname);
 			free_job_list(list);
-- 
2.41.0

Re: [igt-dev] [PATCH i-g-t v2 7/7] runner/runner_tests: add a test to check case-insensitive match

[Kamil Konieczny]

Hi Mauro,
On 2023-10-30 at 10:50:02 +0100, Mauro Carvalho Chehab wrote:
> From: Mauro Carvalho Chehab <mchehab@kernel.org>
> 
> As regular expressions are now case-insensitive, add ab
> unit test to verify that case-insensitive logic is working
> as expected.
> 
> CC: Petri Latvala <adrinael@adrinael.net>
> Signed-off-by: Mauro Carvalho Chehab <mchehab@kernel.org>

Reviewed-by: Kamil Konieczny <kamil.konieczny@linux.intel.com>

> ---
>  runner/runner_tests.c | 25 +++++++++++++++++++++++++
>  1 file changed, 25 insertions(+)
> 
> diff --git a/runner/runner_tests.c b/runner/runner_tests.c
> index 11ff05bc0e7e..0aa7dd6626b7 100644
> --- a/runner/runner_tests.c
> +++ b/runner/runner_tests.c
> @@ -1813,13 +1813,16 @@ igt_main
>  	igt_subtest_group {
>  		const char testlisttext[] = "igt@successtest";
>  		const char blocktext[] = "igt@successtest@first";
> +		const char blocktext_upper[] = "igt@successTEST@first";
>  		struct job_list *list = malloc(sizeof(*list));
>  		volatile int dirfd = -1;
>  		char dirname[] = "tmpdirXXXXXX";
>  		volatile int listfd;
>  		volatile int blockfd;
> +		volatile int blockfd_upper;
>  		char listfilename[] = "tmplistXXXXXX";
>  		char blockfilename[] = "tmpblockXXXXXX";
> +		char blockfilename_upper[] = "tmpBLOCKXXXXXX";
>  
>  		igt_fixture {
>  			igt_require(mkdtemp(dirname) != NULL);
> @@ -1829,8 +1832,11 @@ igt_main
>  			igt_require(write(listfd, testlisttext, strlen(testlisttext)) == strlen(testlisttext));
>  			igt_require((blockfd = mkstemp(blockfilename)) >= 0);
>  			igt_require(write(blockfd, blocktext, strlen(blocktext)) == strlen(blocktext));
> +			igt_require((blockfd_upper = mkstemp(blockfilename_upper)) >= 0);
> +			igt_require(write(blockfd_upper, blocktext_upper, strlen(blocktext_upper)) == strlen(blocktext_upper));
>  			close(listfd);
>  			close(blockfd);
> +			close(blockfd_upper);
>  
>  			init_job_list(list);
>  		}
> @@ -1870,9 +1876,28 @@ igt_main
>  			igt_assert_eqstr(list->entries[0].subtests[0], "second-subtest");
>  		}
>  
> +		igt_subtest("only-binary-name-in-testlist-with-case-insensitive-blocklist") {
> +			const char *argv[] = { "runner",
> +				"--allow-non-root",
> +				"--test-list", listfilename,
> +				"-b", blockfilename_upper,
> +				testdatadir,
> +				dirname,
> +			};
> +
> +			igt_assert(parse_options(ARRAY_SIZE(argv), (char**)argv, settings));
> +
> +			igt_assert(create_job_list(list, settings));
> +			/* No multi-mode, binary has two subtests, one of them blocked */
> +			igt_assert_eq(list->size, 1);
> +			igt_assert_eq(list->entries[0].subtest_count, 1);
> +			igt_assert_eqstr(list->entries[0].subtests[0], "second-subtest");
> +		}
> +
>  		igt_fixture {
>  			unlink(listfilename);
>  			unlink(blockfilename);
> +			unlink(blockfilename_upper);
>  			close(dirfd);
>  			clear_directory(dirname);
>  			free_job_list(list);
> -- 
> 2.41.0
> 

Re: [igt-dev] [PATCH i-g-t v2 2/7] runner/runner_tests: Unit tests for binary-name-only testlist entries

[Kamil Konieczny]

Hi Mauro,
On 2023-10-30 at 10:49:57 +0100, Mauro Carvalho Chehab wrote:
> From: Petri Latvala <adrinael@adrinael.net>

While it is ok to ignore here checkpatch.pl warnings/errors
(it keeps original code style and errors are are for 'char**' only)
imho here it would help to add at least one sentance with desciption.

With or without expanding description:
Reviewed-by: Kamil Konieczny <kamil.konieczny@linux.intel.com>

> 
> Acked-by: Mauro Carvalho Chehab <mchehab@kernel.org>
> Signed-off-by: Petri Latvala <adrinael@adrinael.net>
> ---
>  runner/runner_tests.c | 70 +++++++++++++++++++++++++++++++++++++++++++
>  1 file changed, 70 insertions(+)
> 
> diff --git a/runner/runner_tests.c b/runner/runner_tests.c
> index a7e968f85b91..e6024de11a60 100644
> --- a/runner/runner_tests.c
> +++ b/runner/runner_tests.c
> @@ -1810,6 +1810,76 @@ igt_main
>  		}
>  	}
>  
> +	igt_subtest_group {
> +		const char testlisttext[] = "igt@successtest";
> +		const char blocktext[] = "igt@successtest@first";
> +		struct job_list *list = malloc(sizeof(*list));
> +		volatile int dirfd = -1;
> +		char dirname[] = "tmpdirXXXXXX";
> +		volatile int listfd;
> +		volatile int blockfd;
> +		char listfilename[] = "tmplistXXXXXX";
> +		char blockfilename[] = "tmpblockXXXXXX";
> +
> +		igt_fixture {
> +			igt_require(mkdtemp(dirname) != NULL);
> +			rmdir(dirname);
> +
> +			igt_require((listfd = mkstemp(listfilename)) >= 0);
> +			igt_require(write(listfd, testlisttext, strlen(testlisttext)) == strlen(testlisttext));
> +			igt_require((blockfd = mkstemp(blockfilename)) >= 0);
> +			igt_require(write(blockfd, blocktext, strlen(blocktext)) == strlen(blocktext));
> +			close(listfd);
> +			close(blockfd);
> +
> +			init_job_list(list);
> +		}
> +
> +		igt_subtest("only-binary-name-in-testlist") {
> +			const char *argv[] = { "runner",
> +				"--allow-non-root",
> +				"--test-list", listfilename,
> +				testdatadir,
> +				dirname,
> +			};
> +
> +			igt_assert(parse_options(ARRAY_SIZE(argv), (char**)argv, settings));
> +
> +			igt_assert(create_job_list(list, settings));
> +			/* No multi-mode, binary has two subtests, should be normalized to have individual subtests */
> +			igt_assert_eq(list->size, 2);
> +			igt_assert_eq(list->entries[0].subtest_count, 1);
> +			igt_assert_eq(list->entries[1].subtest_count, 1);
> +		}
> +
> +		igt_subtest("only-binary-name-in-testlist-with-blocks") {
> +			const char *argv[] = { "runner",
> +				"--allow-non-root",
> +				"--test-list", listfilename,
> +				"-b", blockfilename,
> +				testdatadir,
> +				dirname,
> +			};
> +
> +			igt_assert(parse_options(ARRAY_SIZE(argv), (char**)argv, settings));
> +
> +			igt_assert(create_job_list(list, settings));
> +			/* No multi-mode, binary has two subtests, one of them blocked */
> +			igt_assert_eq(list->size, 1);
> +			igt_assert_eq(list->entries[0].subtest_count, 1);
> +			igt_assert_eqstr(list->entries[0].subtests[0], "second-subtest");
> +		}
> +
> +		igt_fixture {
> +			unlink(listfilename);
> +			unlink(blockfilename);
> +			close(dirfd);
> +			clear_directory(dirname);
> +			free_job_list(list);
> +			free(list);
> +		}
> +	}
> +
>  	igt_subtest_group {
>  		struct job_list *list = malloc(sizeof(*list));
>  		volatile int dirfd = -1;
> -- 
> 2.41.0
> 

Re: [igt-dev] [PATCH i-g-t v2 3/7] runner/runner_tests: Fix name of no-subtests in unit tests

[Kamil Konieczny]

Hi Mauro,
On 2023-10-30 at 10:49:58 +0100, Mauro Carvalho Chehab wrote:
> From: Petri Latvala <adrinael@adrinael.net>
> 
> Now that runner normalizes no-subtest entries in testlists, make sure
> the name matches what's available.
> 
> Acked-by: Mauro Carvalho Chehab <mchehab@kernel.org>
> Signed-off-by: Petri Latvala <adrinael@adrinael.net>

Reviewed-by: Kamil Konieczny <kamil.konieczny@linux.intel.com>

> ---
>  runner/runner_tests.c | 4 ++--
>  1 file changed, 2 insertions(+), 2 deletions(-)
> 
> diff --git a/runner/runner_tests.c b/runner/runner_tests.c
> index e6024de11a60..11ff05bc0e7e 100644
> --- a/runner/runner_tests.c
> +++ b/runner/runner_tests.c
> @@ -802,7 +802,7 @@ igt_main
>  		char filename[] = "tmplistXXXXXX";
>  		const char testlisttext[] = "igt@successtest@first-subtest\n"
>  			"igt@successtest@second-subtest\n"
> -			"igt@nosubtests\n";
> +			"igt@no-subtests\n";
>  		int multiple;
>  		struct job_list *list = malloc(sizeof(*list));
>  
> @@ -830,7 +830,7 @@ igt_main
>  
>  				igt_assert_eqstr(list->entries[0].binary, "successtest");
>  				if (!multiple) igt_assert_eqstr(list->entries[1].binary, "successtest");
> -				igt_assert_eqstr(list->entries[multiple ? 1 : 2].binary, "nosubtests");
> +				igt_assert_eqstr(list->entries[multiple ? 1 : 2].binary, "no-subtests");
>  
>  				igt_assert_eq(list->entries[0].subtest_count, multiple ? 2 : 1);
>  				igt_assert_eq(list->entries[1].subtest_count, multiple ? 0 : 1);
> -- 
> 2.41.0
> 

[igt-dev] ✓ CI.xeBAT: success for better sync test_list.py with igt_runner (rev2)

[Patchwork]

[-- Attachment #1: Type: text/plain, Size: 2897 bytes --]

== Series Details ==

Series: better sync test_list.py with igt_runner (rev2)
URL   : https://patchwork.freedesktop.org/series/125512/
State : success

== Summary ==

CI Bug Log - changes from XEIGT_7566_BAT -> XEIGTPW_10083_BAT
====================================================

Summary
-------

  **SUCCESS**

  No regressions found.

  

Participating hosts (4 -> 4)
------------------------------

  No changes in participating hosts

Possible new issues
-------------------

  Here are the unknown changes that may have been introduced in XEIGTPW_10083_BAT:

### IGT changes ###

#### Suppressed ####

  The following results come from untrusted machines, tests, or statuses.
  They do not affect the overall result.

  * {igt@xe_evict_ccs@evict-ccs-overcommit-parallel-nofree-samefd}:
    - bat-atsm-2:         [PASS][1] -> [FAIL][2]
   [1]: https://intel-gfx-ci.01.org/tree/intel-xe/IGT_7566/bat-atsm-2/igt@xe_evict_ccs@evict-ccs-overcommit-parallel-nofree-samefd.html
   [2]: https://intel-gfx-ci.01.org/tree/intel-xe/IGTPW_10083/bat-atsm-2/igt@xe_evict_ccs@evict-ccs-overcommit-parallel-nofree-samefd.html

  
Known issues
------------

  Here are the changes found in XEIGTPW_10083_BAT that come from known issues:

### IGT changes ###

#### Possible fixes ####

  * {igt@xe_create@create-execqueues-noleak}:
    - bat-adlp-7:         [FAIL][3] ([Intel XE#524]) -> [PASS][4]
   [3]: https://intel-gfx-ci.01.org/tree/intel-xe/IGT_7566/bat-adlp-7/igt@xe_create@create-execqueues-noleak.html
   [4]: https://intel-gfx-ci.01.org/tree/intel-xe/IGTPW_10083/bat-adlp-7/igt@xe_create@create-execqueues-noleak.html

  
#### Warnings ####

  * igt@kms_frontbuffer_tracking@basic:
    - bat-adlp-7:         [DMESG-FAIL][5] ([Intel XE#282] / [i915#2017]) -> [FAIL][6] ([Intel XE#616] / [Intel XE#750])
   [5]: https://intel-gfx-ci.01.org/tree/intel-xe/IGT_7566/bat-adlp-7/igt@kms_frontbuffer_tracking@basic.html
   [6]: https://intel-gfx-ci.01.org/tree/intel-xe/IGTPW_10083/bat-adlp-7/igt@kms_frontbuffer_tracking@basic.html

  
  {name}: This element is suppressed. This means it is ignored when computing
          the status of the difference (SUCCESS, WARNING, or FAILURE).

  [Intel XE#282]: https://gitlab.freedesktop.org/drm/xe/kernel/issues/282
  [Intel XE#524]: https://gitlab.freedesktop.org/drm/xe/kernel/issues/524
  [Intel XE#616]: https://gitlab.freedesktop.org/drm/xe/kernel/issues/616
  [Intel XE#750]: https://gitlab.freedesktop.org/drm/xe/kernel/issues/750
  [i915#2017]: https://gitlab.freedesktop.org/drm/intel/issues/2017


Build changes
-------------

  * IGT: IGT_7566 -> IGTPW_10083

  IGTPW_10083: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_10083/index.html
  IGT_7566: 7566
  xe-456-d7eb9a5c3acda8d87699bc82add7e8d88f5f4700: d7eb9a5c3acda8d87699bc82add7e8d88f5f4700

== Logs ==

For more details see: https://intel-gfx-ci.01.org/tree/intel-xe/IGTPW_10083/index.html

[-- Attachment #2: Type: text/html, Size: 3509 bytes --]

[igt-dev] ✗ Fi.CI.BAT: failure for better sync test_list.py with igt_runner (rev2)

[Patchwork]

[-- Attachment #1: Type: text/plain, Size: 4568 bytes --]

== Series Details ==

Series: better sync test_list.py with igt_runner (rev2)
URL   : https://patchwork.freedesktop.org/series/125512/
State : failure

== Summary ==

CI Bug Log - changes from CI_DRM_13812 -> IGTPW_10083
====================================================

Summary
-------

  **FAILURE**

  Serious unknown changes coming with IGTPW_10083 absolutely need to be
  verified manually.
  
  If you think the reported changes have nothing to do with the changes
  introduced in IGTPW_10083, please notify your bug team (lgci.bug.filing@intel.com) to allow them
  to document this new failure mode, which will reduce false positives in CI.

  External URL: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_10083/index.html

Participating hosts (40 -> 37)
------------------------------

  Missing    (3): bat-adlp-11 fi-snb-2520m fi-bsw-n3050 

Possible new issues
-------------------

  Here are the unknown changes that may have been introduced in IGTPW_10083:

### IGT changes ###

#### Possible regressions ####

  * igt@gem_exec_fence@basic-await@ccs0:
    - bat-dg2-9:          [PASS][1] -> [FAIL][2]
   [1]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_13812/bat-dg2-9/igt@gem_exec_fence@basic-await@ccs0.html
   [2]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_10083/bat-dg2-9/igt@gem_exec_fence@basic-await@ccs0.html

  
Known issues
------------

  Here are the changes found in IGTPW_10083 that come from known issues:

### IGT changes ###

#### Issues hit ####

  * igt@i915_selftest@live@hangcheck:
    - bat-dg2-11:         [PASS][3] -> [ABORT][4] ([i915#9523])
   [3]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_13812/bat-dg2-11/igt@i915_selftest@live@hangcheck.html
   [4]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_10083/bat-dg2-11/igt@i915_selftest@live@hangcheck.html

  * igt@kms_addfb_basic@addfb25-y-tiled-small-legacy:
    - fi-hsw-4770:        NOTRUN -> [SKIP][5] ([fdo#109271] / [i915#5190])
   [5]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_10083/fi-hsw-4770/igt@kms_addfb_basic@addfb25-y-tiled-small-legacy.html

  * igt@kms_pipe_crc_basic@compare-crc-sanitycheck-nv12@pipe-a-vga-1:
    - fi-hsw-4770:        NOTRUN -> [SKIP][6] ([fdo#109271]) +12 other tests skip
   [6]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_10083/fi-hsw-4770/igt@kms_pipe_crc_basic@compare-crc-sanitycheck-nv12@pipe-a-vga-1.html

  * igt@kms_pipe_crc_basic@nonblocking-crc-frame-sequence:
    - bat-adlp-9:         NOTRUN -> [SKIP][7] ([i915#3546]) +2 other tests skip
   [7]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_10083/bat-adlp-9/igt@kms_pipe_crc_basic@nonblocking-crc-frame-sequence.html

  * igt@kms_pipe_crc_basic@suspend-read-crc@pipe-c-vga-1:
    - fi-hsw-4770:        NOTRUN -> [DMESG-WARN][8] ([i915#8841]) +6 other tests dmesg-warn
   [8]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_10083/fi-hsw-4770/igt@kms_pipe_crc_basic@suspend-read-crc@pipe-c-vga-1.html

  * igt@kms_psr@sprite_plane_onoff:
    - fi-hsw-4770:        NOTRUN -> [SKIP][9] ([fdo#109271] / [i915#1072]) +3 other tests skip
   [9]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_10083/fi-hsw-4770/igt@kms_psr@sprite_plane_onoff.html

  
#### Possible fixes ####

  * igt@kms_frontbuffer_tracking@basic:
    - fi-bsw-nick:        [FAIL][10] ([i915#9276]) -> [PASS][11]
   [10]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_13812/fi-bsw-nick/igt@kms_frontbuffer_tracking@basic.html
   [11]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_10083/fi-bsw-nick/igt@kms_frontbuffer_tracking@basic.html

  
  {name}: This element is suppressed. This means it is ignored when computing
          the status of the difference (SUCCESS, WARNING, or FAILURE).

  [fdo#109271]: https://bugs.freedesktop.org/show_bug.cgi?id=109271
  [i915#1072]: https://gitlab.freedesktop.org/drm/intel/issues/1072
  [i915#3546]: https://gitlab.freedesktop.org/drm/intel/issues/3546
  [i915#5190]: https://gitlab.freedesktop.org/drm/intel/issues/5190
  [i915#8841]: https://gitlab.freedesktop.org/drm/intel/issues/8841
  [i915#9276]: https://gitlab.freedesktop.org/drm/intel/issues/9276
  [i915#9523]: https://gitlab.freedesktop.org/drm/intel/issues/9523


Build changes
-------------

  * CI: CI-20190529 -> None
  * IGT: IGT_7566 -> IGTPW_10083

  CI-20190529: 20190529
  CI_DRM_13812: a946b5bdf710ff270c68769fabe8b26587a54054 @ git://anongit.freedesktop.org/gfx-ci/linux
  IGTPW_10083: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_10083/index.html
  IGT_7566: 7566

== Logs ==

For more details see: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_10083/index.html

[-- Attachment #2: Type: text/html, Size: 5504 bytes --]