* [igt-dev] [PATCH i-g-t 0/5] Fix a series of issues while handling testlist
@ 2023-11-02 13:06 Mauro Carvalho Chehab
2023-11-02 13:06 ` [igt-dev] [PATCH i-g-t 1/5] scripts/test_list.py: better parse list of tests Mauro Carvalho Chehab
` (4 more replies)
0 siblings, 5 replies; 11+ messages in thread
From: Mauro Carvalho Chehab @ 2023-11-02 13:06 UTC (permalink / raw)
To: igt-dev
From: Mauro Carvalho Chehab <mchehab@kernel.org>
There is still a huge gap between what igt_runner and test_list.py
generates. Those are related to several issues:
- The properties related to input file test list parsing are not grouped
altogether and have weird names. A better namespace for JSON config
for such options needs to be defined.
- FULL testlist is currently just the tests that don't belong to other
testlists;
- i915 and Xe has a different set of FULL testlist;
- case-insensitive regex were not applied the right way, as we need
to be able to define what kind of match applies for include and
exclude files.
Mauro Carvalho Chehab (5):
scripts/test_list.py: better parse list of tests
scripts/test_list.py: speedup update testlist logic
intel/*.json: better handle FULL testlist
scripts/test_list.py: fix regex filtering logic
scripts/test_list.py: use different types for include/exclude
scripts/test_list.py | 84 ++++++++++++++++++--------------
tests/intel/kms_test_config.json | 28 ++++++-----
tests/intel/xe_test_config.json | 17 ++++---
3 files changed, 73 insertions(+), 56 deletions(-)
--
2.41.0
^ permalink raw reply [flat|nested] 11+ messages in thread
* [igt-dev] [PATCH i-g-t 1/5] scripts/test_list.py: better parse list of tests
2023-11-02 13:06 [igt-dev] [PATCH i-g-t 0/5] Fix a series of issues while handling testlist Mauro Carvalho Chehab
@ 2023-11-02 13:06 ` Mauro Carvalho Chehab
2023-11-02 14:57 ` Kamil Konieczny
2023-11-02 13:06 ` [igt-dev] [PATCH i-g-t 2/5] scripts/test_list.py: speedup update testlist logic Mauro Carvalho Chehab
` (3 subsequent siblings)
4 siblings, 1 reply; 11+ messages in thread
From: Mauro Carvalho Chehab @ 2023-11-02 13:06 UTC (permalink / raw)
To: igt-dev
From: Mauro Carvalho Chehab <mchehab@kernel.org>
The JSON config items for using a list of tests to fill a
field is confusing:
- there are several fields defined under __properties__, but
they're not grouped altogether;
- the names of such fields are badly defined;
- the "FULL" testlist is not really a list of all tests, as
it currently exclude tests used on other platforms.
Rewrite the logic to group them into a dict inside a field,
like:
{ "Field": {
"_properties_": {
"update-from-file": {
"type": "subtest-match", # or "regex", "regex-ignorecase"
"default-if-not-excluded": "not on testlists",
"append-value-if-not-excluded": "FULL",
"include": [
{ "BAT": "../ci-dir/driver-bat.testlist" },
{ "func1": "../ci-dir/driver-func1.testlist" },
],
"exclude": [
{ "Xe BAT, FULL": "../ci-dir/driver.blocklist" }
{ "func1, FULL": "../ci-dir/driver-func1.blocklist" }
]
}
}
}
This will update all tests from ../ci-dir/driver-bat.testlist setting
"Field" with "BAT", excluding the ones at ../ci-dir/driver.blocklist.
Similarly, the "func1" value will be parsed.
The "FULL" testlist will contain all tests but the ones inside the
two exclude lists: ../ci-dir/driver.blocklist and ../ci-dir/driver-func1.blocklist.
Signed-off-by: Mauro Carvalho Chehab <mchehab@kernel.org>
---
scripts/test_list.py | 70 ++++++++++++++++++--------------
tests/intel/kms_test_config.json | 28 +++++++------
tests/intel/xe_test_config.json | 16 ++++----
3 files changed, 63 insertions(+), 51 deletions(-)
diff --git a/scripts/test_list.py b/scripts/test_list.py
index 741ec5f4b5b5..bd609feeb1f1 100644
--- a/scripts/test_list.py
+++ b/scripts/test_list.py
@@ -328,23 +328,25 @@ class TestList:
del item["_properties_"]["level"]
del item["_properties_"]["sublevel"]
- # Read testlist files if any
- if "include" in item["_properties_"]:
- testlist = {}
- for value in item["_properties_"]["include"]:
- for name in value.keys():
- self.read_testlist(field, item, testlist, name, cfg_path + value[name])
+ update = self.props[field]["_properties_"].get("update-from-file")
+ if update:
+ # Read testlist files if any
+ if "include" in update:
+ testlist = {}
+ for value in update["include"]:
+ for name in value.keys():
+ self.read_testlist(update, field, item, testlist, name, cfg_path + value[name])
- item["_properties_"]["include"] = testlist
+ update["include"] = testlist
- # Read blocklist files if any
- if "exclude" in item["_properties_"]:
- testlist = {}
- for value in item["_properties_"]["exclude"]:
- for name in value.keys():
- self.read_testlist(field, item, testlist, name, cfg_path + value[name])
+ # Read blocklist files if any
+ if "exclude" in update:
+ testlist = {}
+ for value in update["exclude"]:
+ for name in value.keys():
+ self.read_testlist(update, field, item, testlist, name, cfg_path + value[name])
- item["_properties_"]["exclude"] = testlist
+ update["exclude"] = testlist
if "_properties_" in self.props:
del self.props["_properties_"]
@@ -446,9 +448,9 @@ class TestList:
self.__add_field(key, sublevel, hierarchy_level, field[key])
- def read_testlist(self, field, item, testlist, name, filename):
+ def read_testlist(self, update, field, item, testlist, name, filename):
- match_type = item["_properties_"].get("match-type", "subtest-match")
+ match_type = update.get("type", "subtest-match")
match_type_regex = set(["regex", "regex-ignorecase"])
match_type_str = set(["subtest-match"])
@@ -514,10 +516,13 @@ class TestList:
if "_properties_" not in self.props[field]:
continue
- if "include" not in self.props[field]["_properties_"]:
+ update = self.props[field]["_properties_"].get("update-from-file")
+ if not update:
continue
- default_value = self.props[field]["_properties_"].get("default-testlist")
+ match_type = update.get("type", "subtest-match")
+ default_value = update.get("default--if-not-excluded")
+ append_value = update.get("append-value-if-not-excluded")
testname = subtest_dict["_summary_"]
@@ -527,7 +532,11 @@ class TestList:
else:
values = set()
- for names, regex_array in self.props[field]["_properties_"]["include"].items():
+ if append_value:
+ include_names = set(re.split(",\s*", append_value))
+ values.update(include_names)
+
+ for names, regex_array in update.get("include", {}).items():
name = set(re.split(",\s*", names))
for regex in regex_array:
if regex.fullmatch(testname):
@@ -535,20 +544,19 @@ class TestList:
break
# If test is at a global blocklist, ignore it
- set_full_if_empty = True
- if "exclude" in self.props[field]["_properties_"]:
- for names, regex_array in self.props[field]["_properties_"]["exclude"].items():
- deleted_names = set(re.split(",\s*", names))
- for regex in regex_array:
- if regex.fullmatch(testname):
- if sorted(deleted_names) == sorted(values):
- set_full_if_empty = False
- values.discard(deleted_names)
+ set_default = True
+ for names, regex_array in update.get("exclude", {}).items():
+ deleted_names = set(re.split(",\s*", names))
+ for regex in regex_array:
+ if regex.fullmatch(testname):
+ if sorted(deleted_names) == sorted(values):
+ set_default = False
+ values -= deleted_names
- if default_value and set_full_if_empty and not values:
- values = set([default_value])
+ if default_value and set_default and not values:
+ values.update([default_value])
- if values:
+ if values or self.props[field]["_properties_"].get("mandatory"):
subtest_dict[field] = ", ".join(sorted(values))
def expand_subtest(self, fname, test_name, test, allow_inherit, with_lines = False, with_subtest_nr = False):
diff --git a/tests/intel/kms_test_config.json b/tests/intel/kms_test_config.json
index 837380ee7fae..53465c534bdc 100644
--- a/tests/intel/kms_test_config.json
+++ b/tests/intel/kms_test_config.json
@@ -22,20 +22,22 @@
"Run type": {
"_properties_": {
"description": "Defines what category of testlist it belongs",
- "default-testlist": "FULL",
- "match-type": "subtest-match",
- "include": [
- { "i915 BAT": "../intel-ci/fast-feedback.testlist" },
- { "i915 BAT chamelium": "../intel-ci/fast-feedback-chamelium-only.testlist" },
- { "i915 chamelium": "../intel-ci/chamelium-only.testlist" },
+ "update-from-file": {
+ "append-value-if-not-excluded": "FULL",
+ "match-type": "subtest-match",
+ "include": [
+ { "i915 BAT": "../intel-ci/fast-feedback.testlist" },
+ { "i915 BAT chamelium": "../intel-ci/fast-feedback-chamelium-only.testlist" },
+ { "i915 chamelium": "../intel-ci/chamelium-only.testlist" },
- { "Xe BAT": "../intel-ci/xe-fast-feedback.testlist" },
- { "Xe BAT chamelium": "../intel-ci/xe-fast-feedback-chamelium-only.testlist" }
- ],
- "exclude": [
- { "i915 BAT, i915 BAT chamelium, i915 chamelium": "../intel-ci/blacklist.txt" },
- { "Xe BAT, Xe BAT chamelium": "../intel-ci/xe.blocklist.txt" }
- ]
+ { "Xe BAT": "../intel-ci/xe-fast-feedback.testlist" },
+ { "Xe BAT chamelium": "../intel-ci/xe-fast-feedback-chamelium-only.testlist" }
+ ],
+ "exclude": [
+ { "i915 BAT, i915 BAT chamelium, i915 chamelium, FULL": "../intel-ci/blacklist.txt" },
+ { "Xe BAT, Xe BAT chamelium, FULL": "../intel-ci/xe.blocklist.txt" }
+ ]
+ }
}
}
},
diff --git a/tests/intel/xe_test_config.json b/tests/intel/xe_test_config.json
index dd7aa4776ec6..6c8059b6f21f 100644
--- a/tests/intel/xe_test_config.json
+++ b/tests/intel/xe_test_config.json
@@ -33,13 +33,15 @@
"mandatory": true,
"description": "Defines what category of testlist it belongs",
"default-testlist": "FULL",
- "match-type": "subtest-match",
- "include": [
- { "Xe BAT": "../intel-ci/xe-fast-feedback.testlist" }
- ],
- "exclude": [
- { "Xe BAT": "../intel-ci/xe.blocklist.txt" }
- ],
+ "update-from-file": {
+ "type": "subtest-match",
+ "include": [
+ { "Xe BAT": "../intel-ci/xe-fast-feedback.testlist" }
+ ],
+ "exclude": [
+ { "Xe BAT": "../intel-ci/xe.blocklist.txt" }
+ ],
+ },
"order": [
"boot",
"__all__",
--
2.41.0
^ permalink raw reply related [flat|nested] 11+ messages in thread
* [igt-dev] [PATCH i-g-t 2/5] scripts/test_list.py: speedup update testlist logic
2023-11-02 13:06 [igt-dev] [PATCH i-g-t 0/5] Fix a series of issues while handling testlist Mauro Carvalho Chehab
2023-11-02 13:06 ` [igt-dev] [PATCH i-g-t 1/5] scripts/test_list.py: better parse list of tests Mauro Carvalho Chehab
@ 2023-11-02 13:06 ` Mauro Carvalho Chehab
2023-11-02 14:37 ` Kamil Konieczny
2023-11-02 13:06 ` [igt-dev] [PATCH i-g-t 3/5] intel/*.json: better handle FULL testlist Mauro Carvalho Chehab
` (2 subsequent siblings)
4 siblings, 1 reply; 11+ messages in thread
From: Mauro Carvalho Chehab @ 2023-11-02 13:06 UTC (permalink / raw)
To: igt-dev
From: Mauro Carvalho Chehab <mchehab@kernel.org>
Compile the expand regex to speed it up the parsing of
test lists.
Signed-off-by: Mauro Carvalho Chehab <mchehab@kernel.org>
---
scripts/test_list.py | 10 ++++++----
1 file changed, 6 insertions(+), 4 deletions(-)
diff --git a/scripts/test_list.py b/scripts/test_list.py
index bd609feeb1f1..252fda576c92 100644
--- a/scripts/test_list.py
+++ b/scripts/test_list.py
@@ -512,6 +512,8 @@ class TestList:
return False
def update_testlist_field(self, subtest_dict):
+ expand = re.compile(",\s*")
+
for field in self.props.keys():
if "_properties_" not in self.props[field]:
continue
@@ -528,16 +530,16 @@ class TestList:
value = subtest_dict.get(field)
if value:
- values = set(re.split(",\s*", value))
+ values = set(expand.split(value))
else:
values = set()
if append_value:
- include_names = set(re.split(",\s*", append_value))
+ include_names = set(expand.split(append_value))
values.update(include_names)
for names, regex_array in update.get("include", {}).items():
- name = set(re.split(",\s*", names))
+ name = set(expand.split(names))
for regex in regex_array:
if regex.fullmatch(testname):
values.update(name)
@@ -546,7 +548,7 @@ class TestList:
# If test is at a global blocklist, ignore it
set_default = True
for names, regex_array in update.get("exclude", {}).items():
- deleted_names = set(re.split(",\s*", names))
+ deleted_names = set(expand.split(names))
for regex in regex_array:
if regex.fullmatch(testname):
if sorted(deleted_names) == sorted(values):
--
2.41.0
^ permalink raw reply related [flat|nested] 11+ messages in thread
* [igt-dev] [PATCH i-g-t 3/5] intel/*.json: better handle FULL testlist
2023-11-02 13:06 [igt-dev] [PATCH i-g-t 0/5] Fix a series of issues while handling testlist Mauro Carvalho Chehab
2023-11-02 13:06 ` [igt-dev] [PATCH i-g-t 1/5] scripts/test_list.py: better parse list of tests Mauro Carvalho Chehab
2023-11-02 13:06 ` [igt-dev] [PATCH i-g-t 2/5] scripts/test_list.py: speedup update testlist logic Mauro Carvalho Chehab
@ 2023-11-02 13:06 ` Mauro Carvalho Chehab
2023-11-02 14:39 ` Kamil Konieczny
2023-11-02 13:06 ` [igt-dev] [PATCH i-g-t 4/5] scripts/test_list.py: fix regex filtering logic Mauro Carvalho Chehab
2023-11-02 13:06 ` [igt-dev] [PATCH i-g-t 5/5] scripts/test_list.py: use different types for include/exclude Mauro Carvalho Chehab
4 siblings, 1 reply; 11+ messages in thread
From: Mauro Carvalho Chehab @ 2023-11-02 13:06 UTC (permalink / raw)
To: igt-dev
From: Mauro Carvalho Chehab <mchehab@kernel.org>
On KMS, the FULL testlist for i915 and Xe may be different, as they
have different blocklists.
Rename FULL to Xe FULL or i915 FULL, accordingly.
Signed-off-by: Mauro Carvalho Chehab <mchehab@kernel.org>
---
tests/intel/kms_test_config.json | 6 +++---
tests/intel/xe_test_config.json | 5 +++--
2 files changed, 6 insertions(+), 5 deletions(-)
diff --git a/tests/intel/kms_test_config.json b/tests/intel/kms_test_config.json
index 53465c534bdc..e94737981ab7 100644
--- a/tests/intel/kms_test_config.json
+++ b/tests/intel/kms_test_config.json
@@ -23,7 +23,7 @@
"_properties_": {
"description": "Defines what category of testlist it belongs",
"update-from-file": {
- "append-value-if-not-excluded": "FULL",
+ "append-value-if-not-excluded": "Xe FULL, i915 FULL",
"match-type": "subtest-match",
"include": [
{ "i915 BAT": "../intel-ci/fast-feedback.testlist" },
@@ -34,8 +34,8 @@
{ "Xe BAT chamelium": "../intel-ci/xe-fast-feedback-chamelium-only.testlist" }
],
"exclude": [
- { "i915 BAT, i915 BAT chamelium, i915 chamelium, FULL": "../intel-ci/blacklist.txt" },
- { "Xe BAT, Xe BAT chamelium, FULL": "../intel-ci/xe.blocklist.txt" }
+ { "i915 BAT, i915 BAT chamelium, i915 chamelium, i915 FULL": "../intel-ci/blacklist.txt" },
+ { "Xe BAT, Xe BAT chamelium, Xe FULL": "../intel-ci/xe.blocklist.txt" }
]
}
}
diff --git a/tests/intel/xe_test_config.json b/tests/intel/xe_test_config.json
index 6c8059b6f21f..44c3dd4b4fee 100644
--- a/tests/intel/xe_test_config.json
+++ b/tests/intel/xe_test_config.json
@@ -35,12 +35,13 @@
"default-testlist": "FULL",
"update-from-file": {
"type": "subtest-match",
+ "append-value-if-not-excluded": "Xe FULL",
"include": [
{ "Xe BAT": "../intel-ci/xe-fast-feedback.testlist" }
],
"exclude": [
- { "Xe BAT": "../intel-ci/xe.blocklist.txt" }
- ],
+ { "Xe BAT, Xe FULL": "../intel-ci/xe.blocklist.txt" }
+ ]
},
"order": [
"boot",
--
2.41.0
^ permalink raw reply related [flat|nested] 11+ messages in thread
* [igt-dev] [PATCH i-g-t 4/5] scripts/test_list.py: fix regex filtering logic
2023-11-02 13:06 [igt-dev] [PATCH i-g-t 0/5] Fix a series of issues while handling testlist Mauro Carvalho Chehab
` (2 preceding siblings ...)
2023-11-02 13:06 ` [igt-dev] [PATCH i-g-t 3/5] intel/*.json: better handle FULL testlist Mauro Carvalho Chehab
@ 2023-11-02 13:06 ` Mauro Carvalho Chehab
2023-11-02 14:49 ` Kamil Konieczny
2023-11-02 13:06 ` [igt-dev] [PATCH i-g-t 5/5] scripts/test_list.py: use different types for include/exclude Mauro Carvalho Chehab
4 siblings, 1 reply; 11+ messages in thread
From: Mauro Carvalho Chehab @ 2023-11-02 13:06 UTC (permalink / raw)
To: igt-dev
From: Mauro Carvalho Chehab <mchehab@kernel.org>
Normal regular expressions don't seek from the beginning.
However, Python re.match is actually an alias for:
/^<regex/
Seeking from the beginning. Fix it by using, instead,
re.search(), which handles regular expressions the standard
way.
Signed-off-by: Mauro Carvalho Chehab <mchehab@kernel.org>
---
scripts/test_list.py | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/scripts/test_list.py b/scripts/test_list.py
index 252fda576c92..a7758d5ecb91 100644
--- a/scripts/test_list.py
+++ b/scripts/test_list.py
@@ -500,10 +500,10 @@ class TestList:
for filter_field, regex in self.filters.items():
if filter_field in subtest:
- if not regex.match(subtest[filter_field]):
+ if not regex.search(subtest[filter_field]):
return True
elif filter_field in test:
- if not regex.match(test[filter_field]):
+ if not regex.search(test[filter_field]):
return True
else:
return field_not_found_value
--
2.41.0
^ permalink raw reply related [flat|nested] 11+ messages in thread
* [igt-dev] [PATCH i-g-t 5/5] scripts/test_list.py: use different types for include/exclude
2023-11-02 13:06 [igt-dev] [PATCH i-g-t 0/5] Fix a series of issues while handling testlist Mauro Carvalho Chehab
` (3 preceding siblings ...)
2023-11-02 13:06 ` [igt-dev] [PATCH i-g-t 4/5] scripts/test_list.py: fix regex filtering logic Mauro Carvalho Chehab
@ 2023-11-02 13:06 ` Mauro Carvalho Chehab
2023-11-02 14:52 ` Kamil Konieczny
4 siblings, 1 reply; 11+ messages in thread
From: Mauro Carvalho Chehab @ 2023-11-02 13:06 UTC (permalink / raw)
To: igt-dev
From: Mauro Carvalho Chehab <mchehab@kernel.org>
The type of match for include/exclude may be different. So,
split it into two separate arguments.
Signed-off-by: Mauro Carvalho Chehab <mchehab@kernel.org>
---
scripts/test_list.py | 12 +++++++-----
tests/intel/kms_test_config.json | 2 +-
tests/intel/xe_test_config.json | 2 +-
3 files changed, 9 insertions(+), 7 deletions(-)
diff --git a/scripts/test_list.py b/scripts/test_list.py
index a7758d5ecb91..13f788783483 100644
--- a/scripts/test_list.py
+++ b/scripts/test_list.py
@@ -335,7 +335,9 @@ class TestList:
testlist = {}
for value in update["include"]:
for name in value.keys():
- self.read_testlist(update, field, item, testlist, name, cfg_path + value[name])
+ match_type = update.get("include-type", "subtest-match")
+
+ self.read_testlist(update, match_type, field, item, testlist, name, cfg_path + value[name])
update["include"] = testlist
@@ -344,7 +346,9 @@ class TestList:
testlist = {}
for value in update["exclude"]:
for name in value.keys():
- self.read_testlist(update, field, item, testlist, name, cfg_path + value[name])
+ match_type = update.get("exclude-type", "subtest-match")
+
+ self.read_testlist(update, match_type, field, item, testlist, name, cfg_path + value[name])
update["exclude"] = testlist
@@ -448,9 +452,7 @@ class TestList:
self.__add_field(key, sublevel, hierarchy_level, field[key])
- def read_testlist(self, update, field, item, testlist, name, filename):
-
- match_type = update.get("type", "subtest-match")
+ def read_testlist(self, update, match_type, field, item, testlist, name, filename):
match_type_regex = set(["regex", "regex-ignorecase"])
match_type_str = set(["subtest-match"])
diff --git a/tests/intel/kms_test_config.json b/tests/intel/kms_test_config.json
index e94737981ab7..d38562c02ed6 100644
--- a/tests/intel/kms_test_config.json
+++ b/tests/intel/kms_test_config.json
@@ -24,7 +24,7 @@
"description": "Defines what category of testlist it belongs",
"update-from-file": {
"append-value-if-not-excluded": "Xe FULL, i915 FULL",
- "match-type": "subtest-match",
+ "exclude-type": "regex-ignorecase",
"include": [
{ "i915 BAT": "../intel-ci/fast-feedback.testlist" },
{ "i915 BAT chamelium": "../intel-ci/fast-feedback-chamelium-only.testlist" },
diff --git a/tests/intel/xe_test_config.json b/tests/intel/xe_test_config.json
index 44c3dd4b4fee..a2d764307831 100644
--- a/tests/intel/xe_test_config.json
+++ b/tests/intel/xe_test_config.json
@@ -34,7 +34,7 @@
"description": "Defines what category of testlist it belongs",
"default-testlist": "FULL",
"update-from-file": {
- "type": "subtest-match",
+ "exclude-type": "regex-ignorecase",
"append-value-if-not-excluded": "Xe FULL",
"include": [
{ "Xe BAT": "../intel-ci/xe-fast-feedback.testlist" }
--
2.41.0
^ permalink raw reply related [flat|nested] 11+ messages in thread
* Re: [igt-dev] [PATCH i-g-t 2/5] scripts/test_list.py: speedup update testlist logic
2023-11-02 13:06 ` [igt-dev] [PATCH i-g-t 2/5] scripts/test_list.py: speedup update testlist logic Mauro Carvalho Chehab
@ 2023-11-02 14:37 ` Kamil Konieczny
0 siblings, 0 replies; 11+ messages in thread
From: Kamil Konieczny @ 2023-11-02 14:37 UTC (permalink / raw)
To: igt-dev
Hi Mauro,
On 2023-11-02 at 14:06:24 +0100, Mauro Carvalho Chehab wrote:
> From: Mauro Carvalho Chehab <mchehab@kernel.org>
>
> Compile the expand regex to speed it up the parsing of
> test lists.
>
> Signed-off-by: Mauro Carvalho Chehab <mchehab@kernel.org>
Reviewed-by: Kamil Konieczny <kamil.konieczny@linux.intel.com>
> ---
> scripts/test_list.py | 10 ++++++----
> 1 file changed, 6 insertions(+), 4 deletions(-)
>
> diff --git a/scripts/test_list.py b/scripts/test_list.py
> index bd609feeb1f1..252fda576c92 100644
> --- a/scripts/test_list.py
> +++ b/scripts/test_list.py
> @@ -512,6 +512,8 @@ class TestList:
> return False
>
> def update_testlist_field(self, subtest_dict):
> + expand = re.compile(",\s*")
> +
> for field in self.props.keys():
> if "_properties_" not in self.props[field]:
> continue
> @@ -528,16 +530,16 @@ class TestList:
>
> value = subtest_dict.get(field)
> if value:
> - values = set(re.split(",\s*", value))
> + values = set(expand.split(value))
> else:
> values = set()
>
> if append_value:
> - include_names = set(re.split(",\s*", append_value))
> + include_names = set(expand.split(append_value))
> values.update(include_names)
>
> for names, regex_array in update.get("include", {}).items():
> - name = set(re.split(",\s*", names))
> + name = set(expand.split(names))
> for regex in regex_array:
> if regex.fullmatch(testname):
> values.update(name)
> @@ -546,7 +548,7 @@ class TestList:
> # If test is at a global blocklist, ignore it
> set_default = True
> for names, regex_array in update.get("exclude", {}).items():
> - deleted_names = set(re.split(",\s*", names))
> + deleted_names = set(expand.split(names))
> for regex in regex_array:
> if regex.fullmatch(testname):
> if sorted(deleted_names) == sorted(values):
> --
> 2.41.0
>
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [igt-dev] [PATCH i-g-t 3/5] intel/*.json: better handle FULL testlist
2023-11-02 13:06 ` [igt-dev] [PATCH i-g-t 3/5] intel/*.json: better handle FULL testlist Mauro Carvalho Chehab
@ 2023-11-02 14:39 ` Kamil Konieczny
0 siblings, 0 replies; 11+ messages in thread
From: Kamil Konieczny @ 2023-11-02 14:39 UTC (permalink / raw)
To: igt-dev
Hi Mauro,
On 2023-11-02 at 14:06:25 +0100, Mauro Carvalho Chehab wrote:
> From: Mauro Carvalho Chehab <mchehab@kernel.org>
>
> On KMS, the FULL testlist for i915 and Xe may be different, as they
> have different blocklists.
>
> Rename FULL to Xe FULL or i915 FULL, accordingly.
>
> Signed-off-by: Mauro Carvalho Chehab <mchehab@kernel.org>
Reviewed-by: Kamil Konieczny <kamil.konieczny@linux.intel.com>
> ---
> tests/intel/kms_test_config.json | 6 +++---
> tests/intel/xe_test_config.json | 5 +++--
> 2 files changed, 6 insertions(+), 5 deletions(-)
>
> diff --git a/tests/intel/kms_test_config.json b/tests/intel/kms_test_config.json
> index 53465c534bdc..e94737981ab7 100644
> --- a/tests/intel/kms_test_config.json
> +++ b/tests/intel/kms_test_config.json
> @@ -23,7 +23,7 @@
> "_properties_": {
> "description": "Defines what category of testlist it belongs",
> "update-from-file": {
> - "append-value-if-not-excluded": "FULL",
> + "append-value-if-not-excluded": "Xe FULL, i915 FULL",
> "match-type": "subtest-match",
> "include": [
> { "i915 BAT": "../intel-ci/fast-feedback.testlist" },
> @@ -34,8 +34,8 @@
> { "Xe BAT chamelium": "../intel-ci/xe-fast-feedback-chamelium-only.testlist" }
> ],
> "exclude": [
> - { "i915 BAT, i915 BAT chamelium, i915 chamelium, FULL": "../intel-ci/blacklist.txt" },
> - { "Xe BAT, Xe BAT chamelium, FULL": "../intel-ci/xe.blocklist.txt" }
> + { "i915 BAT, i915 BAT chamelium, i915 chamelium, i915 FULL": "../intel-ci/blacklist.txt" },
> + { "Xe BAT, Xe BAT chamelium, Xe FULL": "../intel-ci/xe.blocklist.txt" }
> ]
> }
> }
> diff --git a/tests/intel/xe_test_config.json b/tests/intel/xe_test_config.json
> index 6c8059b6f21f..44c3dd4b4fee 100644
> --- a/tests/intel/xe_test_config.json
> +++ b/tests/intel/xe_test_config.json
> @@ -35,12 +35,13 @@
> "default-testlist": "FULL",
> "update-from-file": {
> "type": "subtest-match",
> + "append-value-if-not-excluded": "Xe FULL",
> "include": [
> { "Xe BAT": "../intel-ci/xe-fast-feedback.testlist" }
> ],
> "exclude": [
> - { "Xe BAT": "../intel-ci/xe.blocklist.txt" }
> - ],
> + { "Xe BAT, Xe FULL": "../intel-ci/xe.blocklist.txt" }
> + ]
> },
> "order": [
> "boot",
> --
> 2.41.0
>
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [igt-dev] [PATCH i-g-t 4/5] scripts/test_list.py: fix regex filtering logic
2023-11-02 13:06 ` [igt-dev] [PATCH i-g-t 4/5] scripts/test_list.py: fix regex filtering logic Mauro Carvalho Chehab
@ 2023-11-02 14:49 ` Kamil Konieczny
0 siblings, 0 replies; 11+ messages in thread
From: Kamil Konieczny @ 2023-11-02 14:49 UTC (permalink / raw)
To: igt-dev
Hi Mauro,
On 2023-11-02 at 14:06:26 +0100, Mauro Carvalho Chehab wrote:
> From: Mauro Carvalho Chehab <mchehab@kernel.org>
>
> Normal regular expressions don't seek from the beginning.
> However, Python re.match is actually an alias for:
>
> /^<regex/
>
> Seeking from the beginning. Fix it by using, instead,
> re.search(), which handles regular expressions the standard
> way.
>
> Signed-off-by: Mauro Carvalho Chehab <mchehab@kernel.org>
Reviewed-by: Kamil Konieczny <kamil.konieczny@linux.intel.com>
> ---
> scripts/test_list.py | 4 ++--
> 1 file changed, 2 insertions(+), 2 deletions(-)
>
> diff --git a/scripts/test_list.py b/scripts/test_list.py
> index 252fda576c92..a7758d5ecb91 100644
> --- a/scripts/test_list.py
> +++ b/scripts/test_list.py
> @@ -500,10 +500,10 @@ class TestList:
>
> for filter_field, regex in self.filters.items():
> if filter_field in subtest:
> - if not regex.match(subtest[filter_field]):
> + if not regex.search(subtest[filter_field]):
> return True
> elif filter_field in test:
> - if not regex.match(test[filter_field]):
> + if not regex.search(test[filter_field]):
> return True
> else:
> return field_not_found_value
> --
> 2.41.0
>
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [igt-dev] [PATCH i-g-t 5/5] scripts/test_list.py: use different types for include/exclude
2023-11-02 13:06 ` [igt-dev] [PATCH i-g-t 5/5] scripts/test_list.py: use different types for include/exclude Mauro Carvalho Chehab
@ 2023-11-02 14:52 ` Kamil Konieczny
0 siblings, 0 replies; 11+ messages in thread
From: Kamil Konieczny @ 2023-11-02 14:52 UTC (permalink / raw)
To: igt-dev
Hi Mauro,
On 2023-11-02 at 14:06:27 +0100, Mauro Carvalho Chehab wrote:
> From: Mauro Carvalho Chehab <mchehab@kernel.org>
>
> The type of match for include/exclude may be different. So,
> split it into two separate arguments.
>
> Signed-off-by: Mauro Carvalho Chehab <mchehab@kernel.org>
Reviewed-by: Kamil Konieczny <kamil.konieczny@linux.intel.com>
> ---
> scripts/test_list.py | 12 +++++++-----
> tests/intel/kms_test_config.json | 2 +-
> tests/intel/xe_test_config.json | 2 +-
> 3 files changed, 9 insertions(+), 7 deletions(-)
>
> diff --git a/scripts/test_list.py b/scripts/test_list.py
> index a7758d5ecb91..13f788783483 100644
> --- a/scripts/test_list.py
> +++ b/scripts/test_list.py
> @@ -335,7 +335,9 @@ class TestList:
> testlist = {}
> for value in update["include"]:
> for name in value.keys():
> - self.read_testlist(update, field, item, testlist, name, cfg_path + value[name])
> + match_type = update.get("include-type", "subtest-match")
> +
> + self.read_testlist(update, match_type, field, item, testlist, name, cfg_path + value[name])
>
> update["include"] = testlist
>
> @@ -344,7 +346,9 @@ class TestList:
> testlist = {}
> for value in update["exclude"]:
> for name in value.keys():
> - self.read_testlist(update, field, item, testlist, name, cfg_path + value[name])
> + match_type = update.get("exclude-type", "subtest-match")
> +
> + self.read_testlist(update, match_type, field, item, testlist, name, cfg_path + value[name])
>
> update["exclude"] = testlist
>
> @@ -448,9 +452,7 @@ class TestList:
>
> self.__add_field(key, sublevel, hierarchy_level, field[key])
>
> - def read_testlist(self, update, field, item, testlist, name, filename):
> -
> - match_type = update.get("type", "subtest-match")
> + def read_testlist(self, update, match_type, field, item, testlist, name, filename):
>
> match_type_regex = set(["regex", "regex-ignorecase"])
> match_type_str = set(["subtest-match"])
> diff --git a/tests/intel/kms_test_config.json b/tests/intel/kms_test_config.json
> index e94737981ab7..d38562c02ed6 100644
> --- a/tests/intel/kms_test_config.json
> +++ b/tests/intel/kms_test_config.json
> @@ -24,7 +24,7 @@
> "description": "Defines what category of testlist it belongs",
> "update-from-file": {
> "append-value-if-not-excluded": "Xe FULL, i915 FULL",
> - "match-type": "subtest-match",
> + "exclude-type": "regex-ignorecase",
> "include": [
> { "i915 BAT": "../intel-ci/fast-feedback.testlist" },
> { "i915 BAT chamelium": "../intel-ci/fast-feedback-chamelium-only.testlist" },
> diff --git a/tests/intel/xe_test_config.json b/tests/intel/xe_test_config.json
> index 44c3dd4b4fee..a2d764307831 100644
> --- a/tests/intel/xe_test_config.json
> +++ b/tests/intel/xe_test_config.json
> @@ -34,7 +34,7 @@
> "description": "Defines what category of testlist it belongs",
> "default-testlist": "FULL",
> "update-from-file": {
> - "type": "subtest-match",
> + "exclude-type": "regex-ignorecase",
> "append-value-if-not-excluded": "Xe FULL",
> "include": [
> { "Xe BAT": "../intel-ci/xe-fast-feedback.testlist" }
> --
> 2.41.0
>
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [igt-dev] [PATCH i-g-t 1/5] scripts/test_list.py: better parse list of tests
2023-11-02 13:06 ` [igt-dev] [PATCH i-g-t 1/5] scripts/test_list.py: better parse list of tests Mauro Carvalho Chehab
@ 2023-11-02 14:57 ` Kamil Konieczny
0 siblings, 0 replies; 11+ messages in thread
From: Kamil Konieczny @ 2023-11-02 14:57 UTC (permalink / raw)
To: igt-dev
Hi Mauro,
On 2023-11-02 at 14:06:23 +0100, Mauro Carvalho Chehab wrote:
> From: Mauro Carvalho Chehab <mchehab@kernel.org>
>
> The JSON config items for using a list of tests to fill a
> field is confusing:
> - there are several fields defined under __properties__, but
> they're not grouped altogether;
> - the names of such fields are badly defined;
> - the "FULL" testlist is not really a list of all tests, as
> it currently exclude tests used on other platforms.
>
> Rewrite the logic to group them into a dict inside a field,
> like:
>
> { "Field": {
> "_properties_": {
> "update-from-file": {
> "type": "subtest-match", # or "regex", "regex-ignorecase"
> "default-if-not-excluded": "not on testlists",
> "append-value-if-not-excluded": "FULL",
> "include": [
> { "BAT": "../ci-dir/driver-bat.testlist" },
> { "func1": "../ci-dir/driver-func1.testlist" },
> ],
> "exclude": [
> { "Xe BAT, FULL": "../ci-dir/driver.blocklist" }
> { "func1, FULL": "../ci-dir/driver-func1.blocklist" }
> ]
> }
> }
> }
>
> This will update all tests from ../ci-dir/driver-bat.testlist setting
> "Field" with "BAT", excluding the ones at ../ci-dir/driver.blocklist.
> Similarly, the "func1" value will be parsed.
>
> The "FULL" testlist will contain all tests but the ones inside the
> two exclude lists: ../ci-dir/driver.blocklist and ../ci-dir/driver-func1.blocklist.
>
> Signed-off-by: Mauro Carvalho Chehab <mchehab@kernel.org>
Acked-by: Kamil Konieczny <kamil.konieczny@linux.intel.com>
> ---
> scripts/test_list.py | 70 ++++++++++++++++++--------------
> tests/intel/kms_test_config.json | 28 +++++++------
> tests/intel/xe_test_config.json | 16 ++++----
> 3 files changed, 63 insertions(+), 51 deletions(-)
>
> diff --git a/scripts/test_list.py b/scripts/test_list.py
> index 741ec5f4b5b5..bd609feeb1f1 100644
> --- a/scripts/test_list.py
> +++ b/scripts/test_list.py
> @@ -328,23 +328,25 @@ class TestList:
> del item["_properties_"]["level"]
> del item["_properties_"]["sublevel"]
>
> - # Read testlist files if any
> - if "include" in item["_properties_"]:
> - testlist = {}
> - for value in item["_properties_"]["include"]:
> - for name in value.keys():
> - self.read_testlist(field, item, testlist, name, cfg_path + value[name])
> + update = self.props[field]["_properties_"].get("update-from-file")
> + if update:
> + # Read testlist files if any
> + if "include" in update:
> + testlist = {}
> + for value in update["include"]:
> + for name in value.keys():
> + self.read_testlist(update, field, item, testlist, name, cfg_path + value[name])
>
> - item["_properties_"]["include"] = testlist
> + update["include"] = testlist
>
> - # Read blocklist files if any
> - if "exclude" in item["_properties_"]:
> - testlist = {}
> - for value in item["_properties_"]["exclude"]:
> - for name in value.keys():
> - self.read_testlist(field, item, testlist, name, cfg_path + value[name])
> + # Read blocklist files if any
> + if "exclude" in update:
> + testlist = {}
> + for value in update["exclude"]:
> + for name in value.keys():
> + self.read_testlist(update, field, item, testlist, name, cfg_path + value[name])
>
> - item["_properties_"]["exclude"] = testlist
> + update["exclude"] = testlist
>
> if "_properties_" in self.props:
> del self.props["_properties_"]
> @@ -446,9 +448,9 @@ class TestList:
>
> self.__add_field(key, sublevel, hierarchy_level, field[key])
>
> - def read_testlist(self, field, item, testlist, name, filename):
> + def read_testlist(self, update, field, item, testlist, name, filename):
>
> - match_type = item["_properties_"].get("match-type", "subtest-match")
> + match_type = update.get("type", "subtest-match")
>
> match_type_regex = set(["regex", "regex-ignorecase"])
> match_type_str = set(["subtest-match"])
> @@ -514,10 +516,13 @@ class TestList:
> if "_properties_" not in self.props[field]:
> continue
>
> - if "include" not in self.props[field]["_properties_"]:
> + update = self.props[field]["_properties_"].get("update-from-file")
> + if not update:
> continue
>
> - default_value = self.props[field]["_properties_"].get("default-testlist")
> + match_type = update.get("type", "subtest-match")
> + default_value = update.get("default--if-not-excluded")
> + append_value = update.get("append-value-if-not-excluded")
>
> testname = subtest_dict["_summary_"]
>
> @@ -527,7 +532,11 @@ class TestList:
> else:
> values = set()
>
> - for names, regex_array in self.props[field]["_properties_"]["include"].items():
> + if append_value:
> + include_names = set(re.split(",\s*", append_value))
> + values.update(include_names)
> +
> + for names, regex_array in update.get("include", {}).items():
> name = set(re.split(",\s*", names))
> for regex in regex_array:
> if regex.fullmatch(testname):
> @@ -535,20 +544,19 @@ class TestList:
> break
>
> # If test is at a global blocklist, ignore it
> - set_full_if_empty = True
> - if "exclude" in self.props[field]["_properties_"]:
> - for names, regex_array in self.props[field]["_properties_"]["exclude"].items():
> - deleted_names = set(re.split(",\s*", names))
> - for regex in regex_array:
> - if regex.fullmatch(testname):
> - if sorted(deleted_names) == sorted(values):
> - set_full_if_empty = False
> - values.discard(deleted_names)
> + set_default = True
> + for names, regex_array in update.get("exclude", {}).items():
> + deleted_names = set(re.split(",\s*", names))
> + for regex in regex_array:
> + if regex.fullmatch(testname):
> + if sorted(deleted_names) == sorted(values):
> + set_default = False
> + values -= deleted_names
>
> - if default_value and set_full_if_empty and not values:
> - values = set([default_value])
> + if default_value and set_default and not values:
> + values.update([default_value])
>
> - if values:
> + if values or self.props[field]["_properties_"].get("mandatory"):
> subtest_dict[field] = ", ".join(sorted(values))
>
> def expand_subtest(self, fname, test_name, test, allow_inherit, with_lines = False, with_subtest_nr = False):
> diff --git a/tests/intel/kms_test_config.json b/tests/intel/kms_test_config.json
> index 837380ee7fae..53465c534bdc 100644
> --- a/tests/intel/kms_test_config.json
> +++ b/tests/intel/kms_test_config.json
> @@ -22,20 +22,22 @@
> "Run type": {
> "_properties_": {
> "description": "Defines what category of testlist it belongs",
> - "default-testlist": "FULL",
> - "match-type": "subtest-match",
> - "include": [
> - { "i915 BAT": "../intel-ci/fast-feedback.testlist" },
> - { "i915 BAT chamelium": "../intel-ci/fast-feedback-chamelium-only.testlist" },
> - { "i915 chamelium": "../intel-ci/chamelium-only.testlist" },
> + "update-from-file": {
> + "append-value-if-not-excluded": "FULL",
> + "match-type": "subtest-match",
> + "include": [
> + { "i915 BAT": "../intel-ci/fast-feedback.testlist" },
> + { "i915 BAT chamelium": "../intel-ci/fast-feedback-chamelium-only.testlist" },
> + { "i915 chamelium": "../intel-ci/chamelium-only.testlist" },
>
> - { "Xe BAT": "../intel-ci/xe-fast-feedback.testlist" },
> - { "Xe BAT chamelium": "../intel-ci/xe-fast-feedback-chamelium-only.testlist" }
> - ],
> - "exclude": [
> - { "i915 BAT, i915 BAT chamelium, i915 chamelium": "../intel-ci/blacklist.txt" },
> - { "Xe BAT, Xe BAT chamelium": "../intel-ci/xe.blocklist.txt" }
> - ]
> + { "Xe BAT": "../intel-ci/xe-fast-feedback.testlist" },
> + { "Xe BAT chamelium": "../intel-ci/xe-fast-feedback-chamelium-only.testlist" }
> + ],
> + "exclude": [
> + { "i915 BAT, i915 BAT chamelium, i915 chamelium, FULL": "../intel-ci/blacklist.txt" },
> + { "Xe BAT, Xe BAT chamelium, FULL": "../intel-ci/xe.blocklist.txt" }
> + ]
> + }
> }
> }
> },
> diff --git a/tests/intel/xe_test_config.json b/tests/intel/xe_test_config.json
> index dd7aa4776ec6..6c8059b6f21f 100644
> --- a/tests/intel/xe_test_config.json
> +++ b/tests/intel/xe_test_config.json
> @@ -33,13 +33,15 @@
> "mandatory": true,
> "description": "Defines what category of testlist it belongs",
> "default-testlist": "FULL",
> - "match-type": "subtest-match",
> - "include": [
> - { "Xe BAT": "../intel-ci/xe-fast-feedback.testlist" }
> - ],
> - "exclude": [
> - { "Xe BAT": "../intel-ci/xe.blocklist.txt" }
> - ],
> + "update-from-file": {
> + "type": "subtest-match",
> + "include": [
> + { "Xe BAT": "../intel-ci/xe-fast-feedback.testlist" }
> + ],
> + "exclude": [
> + { "Xe BAT": "../intel-ci/xe.blocklist.txt" }
> + ],
> + },
> "order": [
> "boot",
> "__all__",
> --
> 2.41.0
>
^ permalink raw reply [flat|nested] 11+ messages in thread
end of thread, other threads:[~2023-11-02 14:57 UTC | newest]
Thread overview: 11+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2023-11-02 13:06 [igt-dev] [PATCH i-g-t 0/5] Fix a series of issues while handling testlist Mauro Carvalho Chehab
2023-11-02 13:06 ` [igt-dev] [PATCH i-g-t 1/5] scripts/test_list.py: better parse list of tests Mauro Carvalho Chehab
2023-11-02 14:57 ` Kamil Konieczny
2023-11-02 13:06 ` [igt-dev] [PATCH i-g-t 2/5] scripts/test_list.py: speedup update testlist logic Mauro Carvalho Chehab
2023-11-02 14:37 ` Kamil Konieczny
2023-11-02 13:06 ` [igt-dev] [PATCH i-g-t 3/5] intel/*.json: better handle FULL testlist Mauro Carvalho Chehab
2023-11-02 14:39 ` Kamil Konieczny
2023-11-02 13:06 ` [igt-dev] [PATCH i-g-t 4/5] scripts/test_list.py: fix regex filtering logic Mauro Carvalho Chehab
2023-11-02 14:49 ` Kamil Konieczny
2023-11-02 13:06 ` [igt-dev] [PATCH i-g-t 5/5] scripts/test_list.py: use different types for include/exclude Mauro Carvalho Chehab
2023-11-02 14:52 ` Kamil Konieczny
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox