From: Petri Latvala <petri.latvala@intel.com>
To: igt-dev@lists.freedesktop.org
Cc: Petri Latvala <petri.latvala@intel.com>
Subject: [igt-dev] [PATCH i-g-t 11/11] runner/json_tests: Add test for parsing dynamic subtests with same name
Date: Tue, 17 Dec 2019 11:48:00 +0200 [thread overview]
Message-ID: <20191217094800.30687-11-petri.latvala@intel.com> (raw)
In-Reply-To: <20191217094800.30687-1-petri.latvala@intel.com>
Multiple different subtests can have a dynamic subtest with the same
name. Add a test to make sure we correctly delimit the output parsing.
Signed-off-by: Petri Latvala <petri.latvala@intel.com>
Reviewed-by: Arkadiusz Hiler <arkadiusz.hiler@intel.com>
---
.../0/dmesg.txt | 10 +++
.../0/err.txt | 10 +++
.../0/journal.txt | 3 +
.../0/out.txt | 11 +++
.../README.txt | 2 +
.../endtime.txt | 1 +
.../joblist.txt | 1 +
.../metadata.txt | 12 +++
.../reference.json | 84 +++++++++++++++++++
.../starttime.txt | 1 +
.../uname.txt | 1 +
runner/runner_json_tests.c | 1 +
12 files changed, 137 insertions(+)
create mode 100644 runner/json_tests_data/dynamic-subtest-name-in-multiple-subtests/0/dmesg.txt
create mode 100644 runner/json_tests_data/dynamic-subtest-name-in-multiple-subtests/0/err.txt
create mode 100644 runner/json_tests_data/dynamic-subtest-name-in-multiple-subtests/0/journal.txt
create mode 100644 runner/json_tests_data/dynamic-subtest-name-in-multiple-subtests/0/out.txt
create mode 100644 runner/json_tests_data/dynamic-subtest-name-in-multiple-subtests/README.txt
create mode 100644 runner/json_tests_data/dynamic-subtest-name-in-multiple-subtests/endtime.txt
create mode 100644 runner/json_tests_data/dynamic-subtest-name-in-multiple-subtests/joblist.txt
create mode 100644 runner/json_tests_data/dynamic-subtest-name-in-multiple-subtests/metadata.txt
create mode 100644 runner/json_tests_data/dynamic-subtest-name-in-multiple-subtests/reference.json
create mode 100644 runner/json_tests_data/dynamic-subtest-name-in-multiple-subtests/starttime.txt
create mode 100644 runner/json_tests_data/dynamic-subtest-name-in-multiple-subtests/uname.txt
diff --git a/runner/json_tests_data/dynamic-subtest-name-in-multiple-subtests/0/dmesg.txt b/runner/json_tests_data/dynamic-subtest-name-in-multiple-subtests/0/dmesg.txt
new file mode 100644
index 00000000..59741e5f
--- /dev/null
+++ b/runner/json_tests_data/dynamic-subtest-name-in-multiple-subtests/0/dmesg.txt
@@ -0,0 +1,10 @@
+6,1157,23426155175691,-;Console: switching to colour dummy device 80x25
+14,1158,23426155175708,-;[IGT] dynamic: executing
+14,1159,23426155184875,-;[IGT] dynamic: starting subtest subtest-one
+14,1160,23426155184895,-;[IGT] dynamic: starting dynamic subtest this-name-is-shared
+14,1160,23426155184895,-;This dmesg output is from subtest-one/this-name-is-shared
+14,1159,23426155184875,-;[IGT] dynamic: starting subtest subtest-two
+14,1160,23426155184895,-;[IGT] dynamic: starting dynamic subtest this-name-is-shared
+14,1160,23426155184895,-;This dmesg output is from subtest-two/this-name-is-shared
+14,1162,23426155293846,-;[IGT] dynamic: exiting, ret=0
+6,1163,23426155294003,-;Console: switching to colour frame buffer device 240x75
diff --git a/runner/json_tests_data/dynamic-subtest-name-in-multiple-subtests/0/err.txt b/runner/json_tests_data/dynamic-subtest-name-in-multiple-subtests/0/err.txt
new file mode 100644
index 00000000..3d551819
--- /dev/null
+++ b/runner/json_tests_data/dynamic-subtest-name-in-multiple-subtests/0/err.txt
@@ -0,0 +1,10 @@
+Starting subtest: subtest-one
+Starting dynamic subtest: this-name-is-shared
+This stderr output is from subtest-one/this-name-is-shared
+Dynamic subtest this-name-is-shared: SUCCESS (0.050s)
+Subtest subtest-one: SUCCESS (0.050s)
+Starting subtest: subtest-two
+Starting dynamic subtest: this-name-is-shared
+This stderr output is from subtest-two/this-name-is-shared
+Dynamic subtest this-name-is-shared: SUCCESS (0.050s)
+Subtest subtest-two: SUCCESS (0.050s)
diff --git a/runner/json_tests_data/dynamic-subtest-name-in-multiple-subtests/0/journal.txt b/runner/json_tests_data/dynamic-subtest-name-in-multiple-subtests/0/journal.txt
new file mode 100644
index 00000000..3af2b007
--- /dev/null
+++ b/runner/json_tests_data/dynamic-subtest-name-in-multiple-subtests/0/journal.txt
@@ -0,0 +1,3 @@
+subtest-one
+subtest-two
+exit:0 (0.100s)
diff --git a/runner/json_tests_data/dynamic-subtest-name-in-multiple-subtests/0/out.txt b/runner/json_tests_data/dynamic-subtest-name-in-multiple-subtests/0/out.txt
new file mode 100644
index 00000000..e025c096
--- /dev/null
+++ b/runner/json_tests_data/dynamic-subtest-name-in-multiple-subtests/0/out.txt
@@ -0,0 +1,11 @@
+IGT-Version: 1.23-g9e957acd (x86_64) (Linux: 4.18.0-1-amd64 x86_64)
+Starting subtest: subtest-one
+Starting dynamic subtest: this-name-is-shared
+This stdout output is from subtest-one/this-name-is-shared
+Dynamic subtest this-name-is-shared: SUCCESS (0.050s)
+Subtest subtest-one: SUCCESS (0.050s)
+Starting subtest: subtest-two
+Starting dynamic subtest: this-name-is-shared
+This stdout output is from subtest-two/this-name-is-shared
+Dynamic subtest this-name-is-shared: SUCCESS (0.050s)
+Subtest subtest-two: SUCCESS (0.050s)
diff --git a/runner/json_tests_data/dynamic-subtest-name-in-multiple-subtests/README.txt b/runner/json_tests_data/dynamic-subtest-name-in-multiple-subtests/README.txt
new file mode 100644
index 00000000..e141077d
--- /dev/null
+++ b/runner/json_tests_data/dynamic-subtest-name-in-multiple-subtests/README.txt
@@ -0,0 +1,2 @@
+Parsing of dynamic subtest output is tricky if multiple subtests use
+the same name for a dynamic subtest.
diff --git a/runner/json_tests_data/dynamic-subtest-name-in-multiple-subtests/endtime.txt b/runner/json_tests_data/dynamic-subtest-name-in-multiple-subtests/endtime.txt
new file mode 100644
index 00000000..5c7608b5
--- /dev/null
+++ b/runner/json_tests_data/dynamic-subtest-name-in-multiple-subtests/endtime.txt
@@ -0,0 +1 @@
+1560163492.410489
diff --git a/runner/json_tests_data/dynamic-subtest-name-in-multiple-subtests/joblist.txt b/runner/json_tests_data/dynamic-subtest-name-in-multiple-subtests/joblist.txt
new file mode 100644
index 00000000..9296b9ed
--- /dev/null
+++ b/runner/json_tests_data/dynamic-subtest-name-in-multiple-subtests/joblist.txt
@@ -0,0 +1 @@
+dynamic subtest-one,subtest-two
diff --git a/runner/json_tests_data/dynamic-subtest-name-in-multiple-subtests/metadata.txt b/runner/json_tests_data/dynamic-subtest-name-in-multiple-subtests/metadata.txt
new file mode 100644
index 00000000..8da1aee2
--- /dev/null
+++ b/runner/json_tests_data/dynamic-subtest-name-in-multiple-subtests/metadata.txt
@@ -0,0 +1,12 @@
+abort_mask : 0
+name : dynamic-subtest-name-in-multiple-subtests
+dry_run : 0
+sync : 0
+log_level : 0
+overwrite : 0
+multiple_mode : 1
+inactivity_timeout : 0
+use_watchdog : 0
+piglit_style_dmesg : 0
+test_root : /path/does/not/exist
+results_path : /path/does/not/exist
diff --git a/runner/json_tests_data/dynamic-subtest-name-in-multiple-subtests/reference.json b/runner/json_tests_data/dynamic-subtest-name-in-multiple-subtests/reference.json
new file mode 100644
index 00000000..370fce4d
--- /dev/null
+++ b/runner/json_tests_data/dynamic-subtest-name-in-multiple-subtests/reference.json
@@ -0,0 +1,84 @@
+{
+ "__type__":"TestrunResult",
+ "results_version":10,
+ "name":"dynamic-subtest-name-in-multiple-subtests",
+ "uname":"Linux hostname 4.18.0-1-amd64 #1 SMP Debian 4.18.6-1 (2018-09-06) x86_64",
+ "time_elapsed":{
+ "__type__":"TimeAttribute",
+ "start":1560163492.266377,
+ "end":1560163492.4104891
+ },
+ "tests":{
+ "igt@dynamic@subtest-one@this-name-is-shared":{
+ "out":"IGT-Version: 1.23-g9e957acd (x86_64) (Linux: 4.18.0-1-amd64 x86_64)\nStarting subtest: subtest-one\nStarting dynamic subtest: this-name-is-shared\nThis stdout output is from subtest-one\/this-name-is-shared\nDynamic subtest this-name-is-shared: SUCCESS (0.050s)\nSubtest subtest-one: SUCCESS (0.050s)\n",
+ "igt-version":"IGT-Version: 1.23-g9e957acd (x86_64) (Linux: 4.18.0-1-amd64 x86_64)",
+ "result":"warn",
+ "time":{
+ "__type__":"TimeAttribute",
+ "start":0,
+ "end":0.050000000000000003
+ },
+ "err":"Starting subtest: subtest-one\nStarting dynamic subtest: this-name-is-shared\nThis stderr output is from subtest-one\/this-name-is-shared\nDynamic subtest this-name-is-shared: SUCCESS (0.050s)\nSubtest subtest-one: SUCCESS (0.050s)\n",
+ "dmesg":"<6> [23426155.175691] Console: switching to colour dummy device 80x25\n<6> [23426155.175708] [IGT] dynamic: executing\n<6> [23426155.184875] [IGT] dynamic: starting subtest subtest-one\n<6> [23426155.184895] [IGT] dynamic: starting dynamic subtest this-name-is-shared\n<6> [23426155.184895] This dmesg output is from subtest-one\/this-name-is-shared\n"
+ },
+ "igt@dynamic@subtest-two@this-name-is-shared":{
+ "out":"Starting subtest: subtest-two\nStarting dynamic subtest: this-name-is-shared\nThis stdout output is from subtest-two\/this-name-is-shared\nDynamic subtest this-name-is-shared: SUCCESS (0.050s)\nSubtest subtest-two: SUCCESS (0.050s)\n",
+ "igt-version":"IGT-Version: 1.23-g9e957acd (x86_64) (Linux: 4.18.0-1-amd64 x86_64)",
+ "result":"warn",
+ "time":{
+ "__type__":"TimeAttribute",
+ "start":0,
+ "end":0.050000000000000003
+ },
+ "err":"Starting subtest: subtest-two\nStarting dynamic subtest: this-name-is-shared\nThis stderr output is from subtest-two\/this-name-is-shared\nDynamic subtest this-name-is-shared: SUCCESS (0.050s)\nSubtest subtest-two: SUCCESS (0.050s)\n",
+ "dmesg":"<6> [23426155.184875] [IGT] dynamic: starting subtest subtest-two\n<6> [23426155.184895] [IGT] dynamic: starting dynamic subtest this-name-is-shared\n<6> [23426155.184895] This dmesg output is from subtest-two\/this-name-is-shared\n<6> [23426155.293846] [IGT] dynamic: exiting, ret=0\n<6> [23426155.294003] Console: switching to colour frame buffer device 240x75\n"
+ }
+ },
+ "totals":{
+ "":{
+ "crash":0,
+ "pass":0,
+ "dmesg-fail":0,
+ "dmesg-warn":0,
+ "skip":0,
+ "incomplete":0,
+ "timeout":0,
+ "notrun":0,
+ "fail":0,
+ "warn":2
+ },
+ "root":{
+ "crash":0,
+ "pass":0,
+ "dmesg-fail":0,
+ "dmesg-warn":0,
+ "skip":0,
+ "incomplete":0,
+ "timeout":0,
+ "notrun":0,
+ "fail":0,
+ "warn":2
+ },
+ "igt@dynamic":{
+ "crash":0,
+ "pass":0,
+ "dmesg-fail":0,
+ "dmesg-warn":0,
+ "skip":0,
+ "incomplete":0,
+ "timeout":0,
+ "notrun":0,
+ "fail":0,
+ "warn":2
+ }
+ },
+ "runtimes":{
+ "igt@dynamic":{
+ "time":{
+ "__type__":"TimeAttribute",
+ "start":0,
+ "end":0.10000000000000001
+ }
+ }
+ }
+}
diff --git a/runner/json_tests_data/dynamic-subtest-name-in-multiple-subtests/starttime.txt b/runner/json_tests_data/dynamic-subtest-name-in-multiple-subtests/starttime.txt
new file mode 100644
index 00000000..b30b32ee
--- /dev/null
+++ b/runner/json_tests_data/dynamic-subtest-name-in-multiple-subtests/starttime.txt
@@ -0,0 +1 @@
+1560163492.266377
diff --git a/runner/json_tests_data/dynamic-subtest-name-in-multiple-subtests/uname.txt b/runner/json_tests_data/dynamic-subtest-name-in-multiple-subtests/uname.txt
new file mode 100644
index 00000000..a7aef6f7
--- /dev/null
+++ b/runner/json_tests_data/dynamic-subtest-name-in-multiple-subtests/uname.txt
@@ -0,0 +1 @@
+Linux hostname 4.18.0-1-amd64 #1 SMP Debian 4.18.6-1 (2018-09-06) x86_64
diff --git a/runner/runner_json_tests.c b/runner/runner_json_tests.c
index 3443bd2b..21e87a02 100644
--- a/runner/runner_json_tests.c
+++ b/runner/runner_json_tests.c
@@ -163,6 +163,7 @@ static const char *dirnames[] = {
"dmesg-warn-level-piglit-style",
"dmesg-warn-level-one-piglit-style",
"dynamic-subtests",
+ "dynamic-subtest-name-in-multiple-subtests",
};
igt_main
--
2.19.1
_______________________________________________
igt-dev mailing list
igt-dev@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/igt-dev
next prev parent reply other threads:[~2019-12-17 9:48 UTC|newest]
Thread overview: 13+ messages / expand[flat|nested] mbox.gz Atom feed top
2019-12-17 9:47 [igt-dev] [PATCH i-g-t 01/11] runner/resultgen: Extract igt-version field handling to a helper Petri Latvala
2019-12-17 9:47 ` [igt-dev] [PATCH i-g-t v2 02/11] runner/resultgen: Extract finding begin/end lines for a subtest " Petri Latvala
2019-12-27 8:45 ` Arkadiusz Hiler
2019-12-17 9:47 ` [igt-dev] [PATCH i-g-t v2 03/11] runner/resultgen: Extract finding begin/end pointers for test output to helpers Petri Latvala
2019-12-17 9:47 ` [igt-dev] [PATCH i-g-t v2 04/11] runner/resultgen: Hoist handling of dynamic subtest output to a helper Petri Latvala
2019-12-17 9:47 ` [igt-dev] [PATCH i-g-t v2 05/11] runner/resultgen: Extrude dynamic subtest result texts Petri Latvala
2019-12-17 9:47 ` [igt-dev] [PATCH i-g-t 06/11] runner/resultgen: Add support for extra validation hook in find_matches() Petri Latvala
2019-12-17 9:47 ` [igt-dev] [PATCH i-g-t 07/11] runner/resultgen: Make subtest result line finding more robust Petri Latvala
2019-12-17 9:47 ` [igt-dev] [PATCH i-g-t 08/11] runner/json_tests: Adapt to dynamic subtest result parsing Petri Latvala
2019-12-17 9:47 ` [igt-dev] [PATCH i-g-t 09/11] runner/resultgen: Don't report subtest result if it has dynamic subtests Petri Latvala
2019-12-17 9:47 ` [igt-dev] [PATCH i-g-t 10/11] runner/json_tests: Adapt to no longer reporting subtests with " Petri Latvala
2019-12-17 9:48 ` Petri Latvala [this message]
-- strict thread matches above, loose matches on Subject: below --
2019-12-02 13:00 [igt-dev] [PATCH i-g-t 00/11] runner: Don't report subtests that have " Petri Latvala
2019-12-02 13:01 ` [igt-dev] [PATCH i-g-t 11/11] runner/json_tests: Add test for parsing dynamic subtests with same name Petri Latvala
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20191217094800.30687-11-petri.latvala@intel.com \
--to=petri.latvala@intel.com \
--cc=igt-dev@lists.freedesktop.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox