public inbox for igt-dev@lists.freedesktop.org
 help / color / mirror / Atom feed
* [igt-dev] [PATCH i-g-t v2] runner: Add explicit "notrun" results for tests that were not executed
@ 2018-12-17 13:07 Petri Latvala
  2018-12-17 13:29 ` Petri Latvala
                   ` (4 more replies)
  0 siblings, 5 replies; 7+ messages in thread
From: Petri Latvala @ 2018-12-17 13:07 UTC (permalink / raw)
  To: igt-dev; +Cc: Petri Latvala, Martin Peres

When possible, all tests we know we were going to attempt to execute
now appear in the results as "notrun". The only known case where it's
not possible to add an explicit "notrun" is when running in
multiple-mode, because "no subtests" and "run all subtests, we didn't
list them beforehand" are represented the same.

Note: A function call is commented out in resultgen.c because it needs
the function to exist, and that's in a patch that is still in flight.

v2: Rebase and adjust to already landed json changes

Signed-off-by: Petri Latvala <petri.latvala@intel.com>
Cc: Martin Peres <martin.peres@linux.intel.com>
Cc: Arkadiusz Hiler <arkadiusz.hiler@intel.com>
Acked-by: Martin Peres <martin.peres@linux.intel.com>
---
 .../aborted-after-a-test/reference.json       |  54 +++++++-
 .../aborted-on-boot/reference.json            |  70 +++++++++-
 .../reference.json                            |  56 +++++++-
 .../notrun-results-multiple-mode/0/dmesg.txt  |   5 +
 .../notrun-results-multiple-mode/0/err.txt    |   2 +
 .../0/journal.txt                             |   2 +
 .../notrun-results-multiple-mode/0/out.txt    |   3 +
 .../notrun-results-multiple-mode/README.txt   |   4 +
 .../notrun-results-multiple-mode/endtime.txt  |   1 +
 .../notrun-results-multiple-mode/joblist.txt  |   5 +
 .../notrun-results-multiple-mode/metadata.txt |  12 ++
 .../reference.json                            | 102 +++++++++++++++
 .../starttime.txt                             |   1 +
 .../notrun-results-multiple-mode/uname.txt    |   1 +
 .../notrun-results/0/dmesg.txt                |   5 +
 .../json_tests_data/notrun-results/0/err.txt  |   2 +
 .../notrun-results/0/journal.txt              |   2 +
 .../json_tests_data/notrun-results/0/out.txt  |   3 +
 .../json_tests_data/notrun-results/README.txt |   4 +
 .../notrun-results/endtime.txt                |   1 +
 .../notrun-results/joblist.txt                |   5 +
 .../notrun-results/metadata.txt               |  12 ++
 .../notrun-results/reference.json             | 120 ++++++++++++++++++
 .../notrun-results/starttime.txt              |   1 +
 .../json_tests_data/notrun-results/uname.txt  |   1 +
 runner/resultgen.c                            |  42 +++++-
 runner/runner_json_tests.c                    |   2 +
 27 files changed, 507 insertions(+), 11 deletions(-)
 create mode 100644 runner/json_tests_data/notrun-results-multiple-mode/0/dmesg.txt
 create mode 100644 runner/json_tests_data/notrun-results-multiple-mode/0/err.txt
 create mode 100644 runner/json_tests_data/notrun-results-multiple-mode/0/journal.txt
 create mode 100644 runner/json_tests_data/notrun-results-multiple-mode/0/out.txt
 create mode 100644 runner/json_tests_data/notrun-results-multiple-mode/README.txt
 create mode 100644 runner/json_tests_data/notrun-results-multiple-mode/endtime.txt
 create mode 100644 runner/json_tests_data/notrun-results-multiple-mode/joblist.txt
 create mode 100644 runner/json_tests_data/notrun-results-multiple-mode/metadata.txt
 create mode 100644 runner/json_tests_data/notrun-results-multiple-mode/reference.json
 create mode 100644 runner/json_tests_data/notrun-results-multiple-mode/starttime.txt
 create mode 100644 runner/json_tests_data/notrun-results-multiple-mode/uname.txt
 create mode 100644 runner/json_tests_data/notrun-results/0/dmesg.txt
 create mode 100644 runner/json_tests_data/notrun-results/0/err.txt
 create mode 100644 runner/json_tests_data/notrun-results/0/journal.txt
 create mode 100644 runner/json_tests_data/notrun-results/0/out.txt
 create mode 100644 runner/json_tests_data/notrun-results/README.txt
 create mode 100644 runner/json_tests_data/notrun-results/endtime.txt
 create mode 100644 runner/json_tests_data/notrun-results/joblist.txt
 create mode 100644 runner/json_tests_data/notrun-results/metadata.txt
 create mode 100644 runner/json_tests_data/notrun-results/reference.json
 create mode 100644 runner/json_tests_data/notrun-results/starttime.txt
 create mode 100644 runner/json_tests_data/notrun-results/uname.txt

diff --git a/runner/json_tests_data/aborted-after-a-test/reference.json b/runner/json_tests_data/aborted-after-a-test/reference.json
index 19ba9285..06a8bff6 100644
--- a/runner/json_tests_data/aborted-after-a-test/reference.json
+++ b/runner/json_tests_data/aborted-after-a-test/reference.json
@@ -21,6 +21,30 @@
       "err":"Starting subtest: first-subtest\nSubtest first-subtest: SUCCESS (0.000s)\n",
       "dmesg":"<6> [3216186.095083] Console: switching to colour dummy device 80x25\n<6> [3216186.095097] [IGT] successtest: executing\n<6> [3216186.101115] [IGT] successtest: starting subtest first-subtest\n<6> [3216186.101160] [IGT] successtest: exiting, ret=0\n<6> [3216186.101299] Console: switching to colour frame buffer device 240x75\n"
     },
+    "igt@successtest@second-subtest":{
+	"out":"",
+	"err":"",
+	"dmesg":"",
+	"result":"notrun"
+    },
+    "igt@no-subtests":{
+	"out":"",
+	"err":"",
+	"dmesg":"",
+	"result":"notrun"
+    },
+      "igt@skippers@skip-one":{
+	"out":"",
+	"err":"",
+	"dmesg":"",
+	"result":"notrun"
+    },
+      "igt@skippers@skip-two":{
+	"out":"",
+	"err":"",
+	"dmesg":"",
+	"result":"notrun"
+    },
     "igt@runner@aborted":{
 	"out":"Aborted after: successtest (first-subtest)\n\nKernel tainted (0x200)\n",
 	"result":"fail",
@@ -37,7 +61,7 @@
       "skip":0,
       "incomplete":0,
       "timeout":0,
-      "notrun":0,
+      "notrun":4,
       "fail":1,
       "warn":0
     },
@@ -49,7 +73,7 @@
       "skip":0,
       "incomplete":0,
       "timeout":0,
-      "notrun":0,
+      "notrun":4,
       "fail":1,
       "warn":0
     },
@@ -61,7 +85,31 @@
       "skip":0,
       "incomplete":0,
       "timeout":0,
-      "notrun":0,
+      "notrun":1,
+      "fail":0,
+      "warn":0
+    },
+    "igt@no-subtests":{
+      "crash":0,
+      "pass":0,
+      "dmesg-fail":0,
+      "dmesg-warn":0,
+      "skip":0,
+      "incomplete":0,
+      "timeout":0,
+      "notrun":1,
+      "fail":0,
+      "warn":0
+    },
+    "igt@skippers":{
+      "crash":0,
+      "pass":0,
+      "dmesg-fail":0,
+      "dmesg-warn":0,
+      "skip":0,
+      "incomplete":0,
+      "timeout":0,
+      "notrun":2,
       "fail":0,
       "warn":0
     },
diff --git a/runner/json_tests_data/aborted-on-boot/reference.json b/runner/json_tests_data/aborted-on-boot/reference.json
index 0a8723eb..d354fbac 100644
--- a/runner/json_tests_data/aborted-on-boot/reference.json
+++ b/runner/json_tests_data/aborted-on-boot/reference.json
@@ -9,6 +9,36 @@
     "end":1539953735.1723731
   },
   "tests":{
+    "igt@successtest@first-subtest":{
+	"out":"",
+	"err":"",
+	"dmesg":"",
+	"result":"notrun"
+    },
+    "igt@successtest@second-subtest":{
+	"out":"",
+	"err":"",
+	"dmesg":"",
+	"result":"notrun"
+    },
+    "igt@no-subtests":{
+	"out":"",
+	"err":"",
+	"dmesg":"",
+	"result":"notrun"
+    },
+      "igt@skippers@skip-one":{
+	"out":"",
+	"err":"",
+	"dmesg":"",
+	"result":"notrun"
+    },
+      "igt@skippers@skip-two":{
+	"out":"",
+	"err":"",
+	"dmesg":"",
+	"result":"notrun"
+    },
     "igt@runner@aborted":{
 	"out":"Aborted after: startup\n\nKernel tainted (0x200)\n",
 	"result":"fail",
@@ -25,7 +55,7 @@
       "skip":0,
       "incomplete":0,
       "timeout":0,
-      "notrun":0,
+      "notrun":5,
       "fail":1,
       "warn":0
     },
@@ -37,10 +67,46 @@
       "skip":0,
       "incomplete":0,
       "timeout":0,
-      "notrun":0,
+      "notrun":5,
       "fail":1,
       "warn":0
     },
+    "igt@successtest":{
+      "crash":0,
+      "pass":0,
+      "dmesg-fail":0,
+      "dmesg-warn":0,
+      "skip":0,
+      "incomplete":0,
+      "timeout":0,
+      "notrun":2,
+      "fail":0,
+      "warn":0
+    },
+    "igt@no-subtests":{
+      "crash":0,
+      "pass":0,
+      "dmesg-fail":0,
+      "dmesg-warn":0,
+      "skip":0,
+      "incomplete":0,
+      "timeout":0,
+      "notrun":1,
+      "fail":0,
+      "warn":0
+    },
+    "igt@skippers":{
+      "crash":0,
+      "pass":0,
+      "dmesg-fail":0,
+      "dmesg-warn":0,
+      "skip":0,
+      "incomplete":0,
+      "timeout":0,
+      "notrun":2,
+      "fail":0,
+      "warn":0
+    },
     "igt@runner":{
       "crash":0,
       "pass":0,
diff --git a/runner/json_tests_data/incomplete-before-any-subtests/reference.json b/runner/json_tests_data/incomplete-before-any-subtests/reference.json
index 8dc684c8..8404bb66 100644
--- a/runner/json_tests_data/incomplete-before-any-subtests/reference.json
+++ b/runner/json_tests_data/incomplete-before-any-subtests/reference.json
@@ -15,7 +15,31 @@
       "result":"incomplete",
       "err":"",
       "dmesg":"<6> [3216186.095083] Console: switching to colour dummy device 80x25\n<6> [3216186.095097] [IGT] successtest: executing\n"
-    }
+    },
+    "igt@successtest@second-subtest":{
+	"out":"",
+	"err":"",
+	"dmesg":"",
+	"result":"notrun"
+    },
+    "igt@no-subtests":{
+	"out":"",
+	"err":"",
+	"dmesg":"",
+	"result":"notrun"
+    },
+      "igt@skippers@skip-one":{
+	"out":"",
+	"err":"",
+	"dmesg":"",
+	"result":"notrun"
+    },
+      "igt@skippers@skip-two":{
+	"out":"",
+	"err":"",
+	"dmesg":"",
+	"result":"notrun"
+    },
   },
   "totals":{
     "":{
@@ -26,7 +50,7 @@
       "skip":0,
       "incomplete":1,
       "timeout":0,
-      "notrun":0,
+      "notrun":4,
       "fail":0,
       "warn":0
     },
@@ -38,7 +62,7 @@
       "skip":0,
       "incomplete":1,
       "timeout":0,
-      "notrun":0,
+      "notrun":4,
       "fail":0,
       "warn":0
     },
@@ -50,10 +74,34 @@
       "skip":0,
       "incomplete":1,
       "timeout":0,
-      "notrun":0,
+      "notrun":1,
+      "fail":0,
+      "warn":0
+    },
+    "igt@no-subtests":{
+      "crash":0,
+      "pass":0,
+      "dmesg-fail":0,
+      "dmesg-warn":0,
+      "skip":0,
+      "incomplete":0,
+      "timeout":0,
+      "notrun":1,
       "fail":0,
       "warn":0
     },
+    "igt@skippers":{
+      "crash":0,
+      "pass":0,
+      "dmesg-fail":0,
+      "dmesg-warn":0,
+      "skip":0,
+      "incomplete":0,
+      "timeout":0,
+      "notrun":2,
+      "fail":0,
+      "warn":0
+    }
   },
   "runtimes":{
   }
diff --git a/runner/json_tests_data/notrun-results-multiple-mode/0/dmesg.txt b/runner/json_tests_data/notrun-results-multiple-mode/0/dmesg.txt
new file mode 100644
index 00000000..a189e704
--- /dev/null
+++ b/runner/json_tests_data/notrun-results-multiple-mode/0/dmesg.txt
@@ -0,0 +1,5 @@
+6,951,3216186095083,-;Console: switching to colour dummy device 80x25
+14,952,3216186095097,-;[IGT] successtest: executing
+14,953,3216186101115,-;[IGT] successtest: starting subtest first-subtest
+14,954,3216186101160,-;[IGT] successtest: exiting, ret=0
+6,955,3216186101299,-;Console: switching to colour frame buffer device 240x75
diff --git a/runner/json_tests_data/notrun-results-multiple-mode/0/err.txt b/runner/json_tests_data/notrun-results-multiple-mode/0/err.txt
new file mode 100644
index 00000000..5dc78057
--- /dev/null
+++ b/runner/json_tests_data/notrun-results-multiple-mode/0/err.txt
@@ -0,0 +1,2 @@
+Starting subtest: first-subtest
+Subtest first-subtest: SUCCESS (0.000s)
diff --git a/runner/json_tests_data/notrun-results-multiple-mode/0/journal.txt b/runner/json_tests_data/notrun-results-multiple-mode/0/journal.txt
new file mode 100644
index 00000000..86a30e07
--- /dev/null
+++ b/runner/json_tests_data/notrun-results-multiple-mode/0/journal.txt
@@ -0,0 +1,2 @@
+first-subtest
+exit:0 (0.014s)
diff --git a/runner/json_tests_data/notrun-results-multiple-mode/0/out.txt b/runner/json_tests_data/notrun-results-multiple-mode/0/out.txt
new file mode 100644
index 00000000..5946bf31
--- /dev/null
+++ b/runner/json_tests_data/notrun-results-multiple-mode/0/out.txt
@@ -0,0 +1,3 @@
+IGT-Version: 1.23-g0c763bfd (x86_64) (Linux: 4.18.0-1-amd64 x86_64)
+Starting subtest: first-subtest
+Subtest first-subtest: SUCCESS (0.000s)
diff --git a/runner/json_tests_data/notrun-results-multiple-mode/README.txt b/runner/json_tests_data/notrun-results-multiple-mode/README.txt
new file mode 100644
index 00000000..fedb6cf5
--- /dev/null
+++ b/runner/json_tests_data/notrun-results-multiple-mode/README.txt
@@ -0,0 +1,4 @@
+Tests that are not executed are expected to be present in results.json
+when possible. If run with --multiple-mode, we can't distinguish
+tests without subtests from tests where we attempt to execute all
+subtests.
diff --git a/runner/json_tests_data/notrun-results-multiple-mode/endtime.txt b/runner/json_tests_data/notrun-results-multiple-mode/endtime.txt
new file mode 100644
index 00000000..635f6ae9
--- /dev/null
+++ b/runner/json_tests_data/notrun-results-multiple-mode/endtime.txt
@@ -0,0 +1 @@
+1539953735.172373
diff --git a/runner/json_tests_data/notrun-results-multiple-mode/joblist.txt b/runner/json_tests_data/notrun-results-multiple-mode/joblist.txt
new file mode 100644
index 00000000..31ef8413
--- /dev/null
+++ b/runner/json_tests_data/notrun-results-multiple-mode/joblist.txt
@@ -0,0 +1,5 @@
+successtest first-subtest
+successtest second-subtest
+no-subtests
+skippers skip-one
+skippers skip-two
diff --git a/runner/json_tests_data/notrun-results-multiple-mode/metadata.txt b/runner/json_tests_data/notrun-results-multiple-mode/metadata.txt
new file mode 100644
index 00000000..e387d8ed
--- /dev/null
+++ b/runner/json_tests_data/notrun-results-multiple-mode/metadata.txt
@@ -0,0 +1,12 @@
+abort_mask : 0
+name : normal-run
+dry_run : 0
+sync : 0
+log_level : 0
+overwrite : 0
+multiple_mode : 1
+inactivity_timeout : 0
+use_watchdog : 0
+piglit_style_dmesg : 0
+test_root : /path/does/not/exist
+results_path : /path/does/not/exist
diff --git a/runner/json_tests_data/notrun-results-multiple-mode/reference.json b/runner/json_tests_data/notrun-results-multiple-mode/reference.json
new file mode 100644
index 00000000..de1c3c31
--- /dev/null
+++ b/runner/json_tests_data/notrun-results-multiple-mode/reference.json
@@ -0,0 +1,102 @@
+{
+  "__type__":"TestrunResult",
+  "results_version":10,
+  "name":"normal-run",
+  "uname":"Linux hostname 4.18.0-1-amd64 #1 SMP Debian 4.18.6-1 (2018-09-06) x86_64",
+  "time_elapsed":{
+    "__type__":"TimeAttribute",
+    "start":1539953735.1110389,
+    "end":1539953735.1723731
+  },
+  "tests":{
+    "igt@successtest@first-subtest":{
+      "out":"Starting subtest: first-subtest\nSubtest first-subtest: SUCCESS (0.000s)\n",
+      "igt-version":"IGT-Version: 1.23-g0c763bfd (x86_64) (Linux: 4.18.0-1-amd64 x86_64)",
+      "result":"pass",
+      "time":{
+        "__type__":"TimeAttribute",
+        "start":0,
+        "end":0
+      },
+      "err":"Starting subtest: first-subtest\nSubtest first-subtest: SUCCESS (0.000s)\n",
+      "dmesg":"<6> [3216186.095083] Console: switching to colour dummy device 80x25\n<6> [3216186.095097] [IGT] successtest: executing\n<6> [3216186.101115] [IGT] successtest: starting subtest first-subtest\n<6> [3216186.101160] [IGT] successtest: exiting, ret=0\n<6> [3216186.101299] Console: switching to colour frame buffer device 240x75\n"
+    },
+    "igt@successtest@second-subtest":{
+      "result":"notrun",
+	"out":"",
+	"err":"",
+      "dmesg":""
+    },
+    "igt@skippers@skip-one":{
+      "result":"notrun",
+	"out":"",
+	"err":"",
+      "dmesg":""
+      },
+    "igt@skippers@skip-two":{
+      "result":"notrun",
+	"out":"",
+	"err":"",
+      "dmesg":""
+    }
+  },
+  "totals":{
+    "":{
+      "crash":0,
+      "pass":1,
+      "dmesg-fail":0,
+      "dmesg-warn":0,
+      "skip":0,
+      "incomplete":0,
+      "timeout":0,
+      "notrun":3,
+      "fail":0,
+      "warn":0
+    },
+    "root":{
+      "crash":0,
+      "pass":1,
+      "dmesg-fail":0,
+      "dmesg-warn":0,
+      "skip":0,
+      "incomplete":0,
+      "timeout":0,
+      "notrun":3,
+      "fail":0,
+      "warn":0
+    },
+    "igt@successtest":{
+      "crash":0,
+      "pass":1,
+      "dmesg-fail":0,
+      "dmesg-warn":0,
+      "skip":0,
+      "incomplete":0,
+      "timeout":0,
+      "notrun":1,
+      "fail":0,
+      "warn":0
+    },
+    "igt@skippers":{
+      "crash":0,
+      "pass":0,
+      "dmesg-fail":0,
+      "dmesg-warn":0,
+      "skip":0,
+      "incomplete":0,
+      "timeout":0,
+      "notrun":2,
+      "fail":0,
+      "warn":0
+    }
+  },
+  "runtimes":{
+    "igt@successtest":{
+      "time":{
+        "__type__":"TimeAttribute",
+        "start":0,
+        "end":0.014
+      }
+    },
+  }
+}
diff --git a/runner/json_tests_data/notrun-results-multiple-mode/starttime.txt b/runner/json_tests_data/notrun-results-multiple-mode/starttime.txt
new file mode 100644
index 00000000..ae038f18
--- /dev/null
+++ b/runner/json_tests_data/notrun-results-multiple-mode/starttime.txt
@@ -0,0 +1 @@
+1539953735.111039
diff --git a/runner/json_tests_data/notrun-results-multiple-mode/uname.txt b/runner/json_tests_data/notrun-results-multiple-mode/uname.txt
new file mode 100644
index 00000000..a7aef6f7
--- /dev/null
+++ b/runner/json_tests_data/notrun-results-multiple-mode/uname.txt
@@ -0,0 +1 @@
+Linux hostname 4.18.0-1-amd64 #1 SMP Debian 4.18.6-1 (2018-09-06) x86_64
diff --git a/runner/json_tests_data/notrun-results/0/dmesg.txt b/runner/json_tests_data/notrun-results/0/dmesg.txt
new file mode 100644
index 00000000..a189e704
--- /dev/null
+++ b/runner/json_tests_data/notrun-results/0/dmesg.txt
@@ -0,0 +1,5 @@
+6,951,3216186095083,-;Console: switching to colour dummy device 80x25
+14,952,3216186095097,-;[IGT] successtest: executing
+14,953,3216186101115,-;[IGT] successtest: starting subtest first-subtest
+14,954,3216186101160,-;[IGT] successtest: exiting, ret=0
+6,955,3216186101299,-;Console: switching to colour frame buffer device 240x75
diff --git a/runner/json_tests_data/notrun-results/0/err.txt b/runner/json_tests_data/notrun-results/0/err.txt
new file mode 100644
index 00000000..5dc78057
--- /dev/null
+++ b/runner/json_tests_data/notrun-results/0/err.txt
@@ -0,0 +1,2 @@
+Starting subtest: first-subtest
+Subtest first-subtest: SUCCESS (0.000s)
diff --git a/runner/json_tests_data/notrun-results/0/journal.txt b/runner/json_tests_data/notrun-results/0/journal.txt
new file mode 100644
index 00000000..86a30e07
--- /dev/null
+++ b/runner/json_tests_data/notrun-results/0/journal.txt
@@ -0,0 +1,2 @@
+first-subtest
+exit:0 (0.014s)
diff --git a/runner/json_tests_data/notrun-results/0/out.txt b/runner/json_tests_data/notrun-results/0/out.txt
new file mode 100644
index 00000000..5946bf31
--- /dev/null
+++ b/runner/json_tests_data/notrun-results/0/out.txt
@@ -0,0 +1,3 @@
+IGT-Version: 1.23-g0c763bfd (x86_64) (Linux: 4.18.0-1-amd64 x86_64)
+Starting subtest: first-subtest
+Subtest first-subtest: SUCCESS (0.000s)
diff --git a/runner/json_tests_data/notrun-results/README.txt b/runner/json_tests_data/notrun-results/README.txt
new file mode 100644
index 00000000..4355aebd
--- /dev/null
+++ b/runner/json_tests_data/notrun-results/README.txt
@@ -0,0 +1,4 @@
+Tests that are not executed are expected to be present in results.json
+when possible. If run without --multiple-mode, we can distinguish
+tests without subtests from tests where we attempt to execute all
+subtests.
diff --git a/runner/json_tests_data/notrun-results/endtime.txt b/runner/json_tests_data/notrun-results/endtime.txt
new file mode 100644
index 00000000..635f6ae9
--- /dev/null
+++ b/runner/json_tests_data/notrun-results/endtime.txt
@@ -0,0 +1 @@
+1539953735.172373
diff --git a/runner/json_tests_data/notrun-results/joblist.txt b/runner/json_tests_data/notrun-results/joblist.txt
new file mode 100644
index 00000000..31ef8413
--- /dev/null
+++ b/runner/json_tests_data/notrun-results/joblist.txt
@@ -0,0 +1,5 @@
+successtest first-subtest
+successtest second-subtest
+no-subtests
+skippers skip-one
+skippers skip-two
diff --git a/runner/json_tests_data/notrun-results/metadata.txt b/runner/json_tests_data/notrun-results/metadata.txt
new file mode 100644
index 00000000..c501ae0e
--- /dev/null
+++ b/runner/json_tests_data/notrun-results/metadata.txt
@@ -0,0 +1,12 @@
+abort_mask : 0
+name : normal-run
+dry_run : 0
+sync : 0
+log_level : 0
+overwrite : 0
+multiple_mode : 0
+inactivity_timeout : 0
+use_watchdog : 0
+piglit_style_dmesg : 0
+test_root : /path/does/not/exist
+results_path : /path/does/not/exist
diff --git a/runner/json_tests_data/notrun-results/reference.json b/runner/json_tests_data/notrun-results/reference.json
new file mode 100644
index 00000000..6b5ff69b
--- /dev/null
+++ b/runner/json_tests_data/notrun-results/reference.json
@@ -0,0 +1,120 @@
+{
+  "__type__":"TestrunResult",
+  "results_version":10,
+  "name":"normal-run",
+  "uname":"Linux hostname 4.18.0-1-amd64 #1 SMP Debian 4.18.6-1 (2018-09-06) x86_64",
+  "time_elapsed":{
+    "__type__":"TimeAttribute",
+    "start":1539953735.1110389,
+    "end":1539953735.1723731
+  },
+  "tests":{
+    "igt@successtest@first-subtest":{
+      "out":"Starting subtest: first-subtest\nSubtest first-subtest: SUCCESS (0.000s)\n",
+      "igt-version":"IGT-Version: 1.23-g0c763bfd (x86_64) (Linux: 4.18.0-1-amd64 x86_64)",
+      "result":"pass",
+      "time":{
+        "__type__":"TimeAttribute",
+        "start":0,
+        "end":0
+      },
+      "err":"Starting subtest: first-subtest\nSubtest first-subtest: SUCCESS (0.000s)\n",
+      "dmesg":"<6> [3216186.095083] Console: switching to colour dummy device 80x25\n<6> [3216186.095097] [IGT] successtest: executing\n<6> [3216186.101115] [IGT] successtest: starting subtest first-subtest\n<6> [3216186.101160] [IGT] successtest: exiting, ret=0\n<6> [3216186.101299] Console: switching to colour frame buffer device 240x75\n"
+    },
+    "igt@successtest@second-subtest":{
+      "result":"notrun",
+	"out":"",
+	"err":"",
+      "dmesg":""
+    },
+    "igt@no-subtests":{
+      "result":"notrun",
+	"out":"",
+	"err":"",
+      "dmesg":""
+    },
+    "igt@skippers@skip-one":{
+      "result":"notrun",
+	"out":"",
+	"err":"",
+      "dmesg":""
+      },
+    "igt@skippers@skip-two":{
+      "result":"notrun",
+	"out":"",
+	"err":"",
+      "dmesg":""
+    }
+  },
+  "totals":{
+    "":{
+      "crash":0,
+      "pass":1,
+      "dmesg-fail":0,
+      "dmesg-warn":0,
+      "skip":0,
+      "incomplete":0,
+      "timeout":0,
+      "notrun":4,
+      "fail":0,
+      "warn":0
+    },
+    "root":{
+      "crash":0,
+      "pass":1,
+      "dmesg-fail":0,
+      "dmesg-warn":0,
+      "skip":0,
+      "incomplete":0,
+      "timeout":0,
+      "notrun":4,
+      "fail":0,
+      "warn":0
+    },
+    "igt@successtest":{
+      "crash":0,
+      "pass":1,
+      "dmesg-fail":0,
+      "dmesg-warn":0,
+      "skip":0,
+      "incomplete":0,
+      "timeout":0,
+      "notrun":1,
+      "fail":0,
+      "warn":0
+    },
+    "igt@no-subtests":{
+      "crash":0,
+      "pass":0,
+      "dmesg-fail":0,
+      "dmesg-warn":0,
+      "skip":0,
+      "incomplete":0,
+      "timeout":0,
+      "notrun":1,
+      "fail":0,
+      "warn":0
+    },
+    "igt@skippers":{
+      "crash":0,
+      "pass":0,
+      "dmesg-fail":0,
+      "dmesg-warn":0,
+      "skip":0,
+      "incomplete":0,
+      "timeout":0,
+      "notrun":2,
+      "fail":0,
+      "warn":0
+    }
+  },
+  "runtimes":{
+    "igt@successtest":{
+      "time":{
+        "__type__":"TimeAttribute",
+        "start":0,
+        "end":0.014
+      }
+    },
+  }
+}
diff --git a/runner/json_tests_data/notrun-results/starttime.txt b/runner/json_tests_data/notrun-results/starttime.txt
new file mode 100644
index 00000000..ae038f18
--- /dev/null
+++ b/runner/json_tests_data/notrun-results/starttime.txt
@@ -0,0 +1 @@
+1539953735.111039
diff --git a/runner/json_tests_data/notrun-results/uname.txt b/runner/json_tests_data/notrun-results/uname.txt
new file mode 100644
index 00000000..a7aef6f7
--- /dev/null
+++ b/runner/json_tests_data/notrun-results/uname.txt
@@ -0,0 +1 @@
+Linux hostname 4.18.0-1-amd64 #1 SMP Debian 4.18.6-1 (2018-09-06) x86_64
diff --git a/runner/resultgen.c b/runner/resultgen.c
index a1a91cd0..be884955 100644
--- a/runner/resultgen.c
+++ b/runner/resultgen.c
@@ -1034,6 +1034,44 @@ static bool parse_test_directory(int dirfd,
 	return status;
 }
 
+static void try_add_notrun_results(const struct job_list_entry *entry,
+				   const struct settings *settings,
+				   struct results *results)
+{
+	struct subtests subtests = {};
+	struct json_object *current_test;
+	size_t i;
+
+	if (entry->subtest_count == 0) {
+		char piglit_name[256];
+
+		/* We cannot distinguish no-subtests from run-all-subtests in multiple-mode */
+		if (settings->multiple_mode)
+			return;
+		generate_piglit_name(entry->binary, NULL, piglit_name, sizeof(piglit_name));
+		current_test = get_or_create_json_object(results->tests, piglit_name);
+		json_object_object_add(current_test, "out", json_object_new_string(""));
+		json_object_object_add(current_test, "err", json_object_new_string(""));
+		json_object_object_add(current_test, "dmesg", json_object_new_string(""));
+		json_object_object_add(current_test, "result", json_object_new_string("notrun"));
+	}
+
+	for (i = 0; i < entry->subtest_count; i++) {
+		char piglit_name[256];
+
+		generate_piglit_name(entry->binary, entry->subtests[i], piglit_name, sizeof(piglit_name));
+		current_test = get_or_create_json_object(results->tests, piglit_name);
+		json_object_object_add(current_test, "out", json_object_new_string(""));
+		json_object_object_add(current_test, "err", json_object_new_string(""));
+		json_object_object_add(current_test, "dmesg", json_object_new_string(""));
+		json_object_object_add(current_test, "result", json_object_new_string("notrun"));
+		add_subtest(&subtests, strdup(entry->subtests[i]));
+	}
+
+	add_to_totals(entry->binary, &subtests, results);
+	free_subtests(&subtests);
+}
+
 static void create_result_root_nodes(struct json_object *root,
 				     struct results *results)
 {
@@ -1123,8 +1161,8 @@ struct json_object *generate_results_json(int dirfd)
 
 		snprintf(name, 16, "%zd", i);
 		if ((testdirfd = openat(dirfd, name, O_DIRECTORY | O_RDONLY)) < 0) {
-			fprintf(stderr, "Warning: Cannot open result directory %s\n", name);
-			break;
+			try_add_notrun_results(&job_list.entries[i], &settings, &results);
+			continue;
 		}
 
 		if (!parse_test_directory(testdirfd, &job_list.entries[i], &settings, &results)) {
diff --git a/runner/runner_json_tests.c b/runner/runner_json_tests.c
index 17b81524..09a93189 100644
--- a/runner/runner_json_tests.c
+++ b/runner/runner_json_tests.c
@@ -157,6 +157,8 @@ static const char *dirnames[] = {
 	"aborted-on-boot",
 	"aborted-after-a-test",
 	"dmesg-escapes",
+	"notrun-results",
+	"notrun-results-multiple-mode",
 };
 
 igt_main
-- 
2.19.1

_______________________________________________
igt-dev mailing list
igt-dev@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/igt-dev

^ permalink raw reply related	[flat|nested] 7+ messages in thread
* Re: [igt-dev] [PATCH i-g-t v2] runner: Add explicit "notrun" results for tests that were not executed
  2018-12-17 13:07 [igt-dev] [PATCH i-g-t v2] runner: Add explicit "notrun" results for tests that were not executed Petri Latvala
@ 2018-12-17 13:29 ` Petri Latvala
  2018-12-17 14:11 ` [igt-dev] ✓ Fi.CI.BAT: success for " Patchwork
                   ` (3 subsequent siblings)
  4 siblings, 0 replies; 7+ messages in thread
From: Petri Latvala @ 2018-12-17 13:29 UTC (permalink / raw)
  To: igt-dev; +Cc: Martin Peres

On Mon, Dec 17, 2018 at 03:07:40PM +0200, Petri Latvala wrote:
> Note: A function call is commented out in resultgen.c because it needs
> the function to exist, and that's in a patch that is still in flight.

Imagine that I remembered to remove this...


-- 
Petri Latvala
_______________________________________________
igt-dev mailing list
igt-dev@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/igt-dev

^ permalink raw reply	[flat|nested] 7+ messages in thread
* [igt-dev] ✓ Fi.CI.BAT: success for runner: Add explicit "notrun" results for tests that were not executed
  2018-12-17 13:07 [igt-dev] [PATCH i-g-t v2] runner: Add explicit "notrun" results for tests that were not executed Petri Latvala
  2018-12-17 13:29 ` Petri Latvala
@ 2018-12-17 14:11 ` Patchwork
  2018-12-17 15:07 ` [igt-dev] ✗ Fi.CI.IGT: failure " Patchwork
                   ` (2 subsequent siblings)
  4 siblings, 0 replies; 7+ messages in thread
From: Patchwork @ 2018-12-17 14:11 UTC (permalink / raw)
  To: Petri Latvala; +Cc: igt-dev

== Series Details ==

Series: runner: Add explicit "notrun" results for tests that were not executed
URL   : https://patchwork.freedesktop.org/series/54136/
State : success

== Summary ==

CI Bug Log - changes from CI_DRM_5324 -> IGTPW_2163
====================================================

Summary
-------

  **SUCCESS**

  No regressions found.

  External URL: https://patchwork.freedesktop.org/api/1.0/series/54136/revisions/1/mbox/

Known issues
------------

  Here are the changes found in IGTPW_2163 that come from known issues:

### IGT changes ###

#### Issues hit ####

  * igt@gem_exec_suspend@basic-s4-devices:
    - fi-kbl-7500u:       PASS -> DMESG-WARN [fdo#105128] / [fdo#107139]

  * igt@i915_selftest@live_hangcheck:
    - fi-bwr-2160:        PASS -> DMESG-FAIL [fdo#108735]
    - fi-cfl-8109u:       NOTRUN -> INCOMPLETE [fdo#106070]

  * igt@kms_pipe_crc_basic@nonblocking-crc-pipe-a:
    - fi-byt-clapper:     PASS -> FAIL [fdo#107362]

  * igt@kms_pipe_crc_basic@read-crc-pipe-b-frame-sequence:
    - fi-byt-clapper:     PASS -> FAIL [fdo#103191] / [fdo#107362]

  
#### Possible fixes ####

  * igt@kms_chamelium@hdmi-hpd-fast:
    - fi-kbl-7500u:       FAIL [fdo#108767] -> PASS

  * igt@kms_flip@basic-flip-vs-dpms:
    - fi-skl-6700hq:      DMESG-WARN [fdo#105998] -> PASS

  * igt@kms_pipe_crc_basic@nonblocking-crc-pipe-b:
    - fi-byt-clapper:     FAIL [fdo#107362] -> PASS

  
  [fdo#103191]: https://bugs.freedesktop.org/show_bug.cgi?id=103191
  [fdo#105128]: https://bugs.freedesktop.org/show_bug.cgi?id=105128
  [fdo#105998]: https://bugs.freedesktop.org/show_bug.cgi?id=105998
  [fdo#106070]: https://bugs.freedesktop.org/show_bug.cgi?id=106070
  [fdo#107139]: https://bugs.freedesktop.org/show_bug.cgi?id=107139
  [fdo#107362]: https://bugs.freedesktop.org/show_bug.cgi?id=107362
  [fdo#108735]: https://bugs.freedesktop.org/show_bug.cgi?id=108735
  [fdo#108767]: https://bugs.freedesktop.org/show_bug.cgi?id=108767


Participating hosts (49 -> 46)
------------------------------

  Additional (1): fi-cfl-8109u 
  Missing    (4): fi-ilk-m540 fi-byt-squawks fi-bsw-cyan fi-hsw-4200u 


Build changes
-------------

    * IGT: IGT_4748 -> IGTPW_2163

  CI_DRM_5324: 93009153b9bde1d65ca49bb1729cd1111591144b @ git://anongit.freedesktop.org/gfx-ci/linux
  IGTPW_2163: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_2163/
  IGT_4748: 90c76cb9bb47a5a3ebb34ad6b1a557bc02d39713 @ git://anongit.freedesktop.org/xorg/app/intel-gpu-tools

== Logs ==

For more details see: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_2163/
_______________________________________________
igt-dev mailing list
igt-dev@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/igt-dev

^ permalink raw reply	[flat|nested] 7+ messages in thread
* [igt-dev] ✗ Fi.CI.IGT: failure for runner: Add explicit "notrun" results for tests that were not executed
  2018-12-17 13:07 [igt-dev] [PATCH i-g-t v2] runner: Add explicit "notrun" results for tests that were not executed Petri Latvala
  2018-12-17 13:29 ` Petri Latvala
  2018-12-17 14:11 ` [igt-dev] ✓ Fi.CI.BAT: success for " Patchwork
@ 2018-12-17 15:07 ` Patchwork
  2018-12-20 12:02 ` [igt-dev] [PATCH i-g-t v2] " Arkadiusz Hiler
  2018-12-20 12:02 ` [igt-dev] ✗ Fi.CI.IGT: failure for " Patchwork
  4 siblings, 0 replies; 7+ messages in thread
From: Patchwork @ 2018-12-17 15:07 UTC (permalink / raw)
  To: Petri Latvala; +Cc: igt-dev

== Series Details ==

Series: runner: Add explicit "notrun" results for tests that were not executed
URL   : https://patchwork.freedesktop.org/series/54136/
State : failure

== Summary ==

CI Bug Log - changes from CI_DRM_5324_full -> IGTPW_2163_full
====================================================

Summary
-------

  **FAILURE**

  Serious unknown changes coming with IGTPW_2163_full absolutely need to be
  verified manually.
  
  If you think the reported changes have nothing to do with the changes
  introduced in IGTPW_2163_full, please notify your bug team to allow them
  to document this new failure mode, which will reduce false positives in CI.

  External URL: https://patchwork.freedesktop.org/api/1.0/series/54136/revisions/1/mbox/

Possible new issues
-------------------

  Here are the unknown changes that may have been introduced in IGTPW_2163_full:

### IGT changes ###

#### Possible regressions ####

  * igt@gem_busy@extended-semaphore-blt:
    - shard-hsw:          PASS -> FAIL

  
#### Warnings ####

  * igt@kms_frontbuffer_tracking@fbc-1p-primscrn-spr-indfb-draw-render:
    - shard-snb:          SKIP -> PASS

  * igt@pm_rc6_residency@rc6-accuracy:
    - shard-kbl:          SKIP -> PASS

  
Known issues
------------

  Here are the changes found in IGTPW_2163_full that come from known issues:

### IGT changes ###

#### Issues hit ####

  * igt@gem_exec_fence@basic-await-default:
    - shard-hsw:          PASS -> FAIL [fdo#108888]

  * igt@kms_busy@extended-pageflip-modeset-hang-oldfb-render-b:
    - shard-glk:          PASS -> DMESG-WARN [fdo#107956]

  * igt@kms_ccs@pipe-b-crc-sprite-planes-basic:
    - shard-glk:          PASS -> FAIL [fdo#108145]

  * igt@kms_cursor_crc@cursor-256x256-random:
    - shard-apl:          PASS -> FAIL [fdo#103232] +4

  * igt@kms_cursor_crc@cursor-64x64-sliding:
    - shard-glk:          PASS -> FAIL [fdo#103232] +4

  * igt@kms_cursor_crc@cursor-size-change:
    - shard-kbl:          PASS -> FAIL [fdo#103232] +1

  * igt@kms_cursor_legacy@pipe-a-single-bo:
    - shard-hsw:          PASS -> INCOMPLETE [fdo#103540]

  * igt@kms_flip@2x-flip-vs-expired-vblank-interruptible:
    - shard-glk:          PASS -> FAIL [fdo#105363]

  * igt@kms_frontbuffer_tracking@fbc-1p-primscrn-spr-indfb-draw-pwrite:
    - shard-apl:          NOTRUN -> FAIL [fdo#103167]

  * igt@kms_frontbuffer_tracking@fbc-1p-primscrn-spr-indfb-fullscreen:
    - shard-glk:          PASS -> FAIL [fdo#103167] +1

  * igt@kms_frontbuffer_tracking@fbc-1p-primscrn-spr-indfb-onoff:
    - shard-apl:          PASS -> FAIL [fdo#103167]
    - shard-kbl:          PASS -> FAIL [fdo#103167]

  * igt@kms_plane@pixel-format-pipe-b-planes-source-clamping:
    - shard-glk:          PASS -> FAIL [fdo#108948]

  * igt@kms_plane@plane-position-covered-pipe-a-planes:
    - shard-glk:          PASS -> FAIL [fdo#103166] +3
    - shard-apl:          PASS -> FAIL [fdo#103166] +1
    - shard-kbl:          PASS -> FAIL [fdo#103166] +1

  * igt@kms_plane_multiple@atomic-pipe-b-tiling-y:
    - shard-apl:          NOTRUN -> FAIL [fdo#103166]

  * igt@kms_rotation_crc@multiplane-rotation-cropping-top:
    - shard-kbl:          PASS -> DMESG-FAIL [fdo#108950]

  * igt@kms_setmode@basic:
    - shard-hsw:          PASS -> FAIL [fdo#99912]

  
#### Possible fixes ####

  * igt@kms_ccs@pipe-a-crc-sprite-planes-basic:
    - shard-apl:          FAIL [fdo#106510] / [fdo#108145] -> PASS
    - shard-kbl:          FAIL [fdo#107725] / [fdo#108145] -> PASS

  * igt@kms_color@pipe-c-legacy-gamma:
    - shard-kbl:          FAIL [fdo#104782] -> PASS
    - shard-apl:          FAIL [fdo#104782] -> PASS

  * igt@kms_cursor_crc@cursor-128x128-onscreen:
    - shard-apl:          FAIL [fdo#103232] -> PASS +2

  * igt@kms_cursor_crc@cursor-256x256-dpms:
    - shard-kbl:          FAIL [fdo#103232] -> PASS

  * igt@kms_cursor_crc@cursor-256x256-suspend:
    - shard-apl:          FAIL [fdo#103191] / [fdo#103232] -> PASS

  * igt@kms_cursor_crc@cursor-64x64-suspend:
    - shard-kbl:          FAIL [fdo#103191] / [fdo#103232] -> PASS

  * igt@kms_flip@plain-flip-fb-recreate:
    - shard-apl:          INCOMPLETE [fdo#103927] -> PASS

  * igt@kms_frontbuffer_tracking@fbc-1p-primscrn-cur-indfb-onoff:
    - shard-apl:          FAIL [fdo#103167] -> PASS +2
    - shard-kbl:          FAIL [fdo#103167] -> PASS +1

  * igt@kms_frontbuffer_tracking@fbc-2p-primscrn-cur-indfb-draw-blt:
    - shard-glk:          FAIL [fdo#103167] -> PASS +4

  * igt@kms_pipe_crc_basic@hang-read-crc-pipe-c:
    - shard-apl:          DMESG-WARN [fdo#103558] / [fdo#105602] -> PASS +20

  * igt@kms_plane_multiple@atomic-pipe-a-tiling-x:
    - shard-apl:          FAIL [fdo#103166] -> PASS +4

  * igt@kms_plane_multiple@atomic-pipe-a-tiling-y:
    - shard-kbl:          FAIL [fdo#103166] -> PASS

  * igt@kms_plane_multiple@atomic-pipe-b-tiling-y:
    - shard-glk:          FAIL [fdo#103166] -> PASS +3

  * igt@kms_rotation_crc@multiplane-rotation-cropping-top:
    - shard-glk:          DMESG-FAIL [fdo#105763] / [fdo#106538] -> PASS

  
#### Warnings ####

  * igt@kms_plane_multiple@atomic-pipe-b-tiling-yf:
    - shard-apl:          DMESG-FAIL [fdo#103166] / [fdo#103558] / [fdo#105602] -> FAIL [fdo#103166]

  
  [fdo#103166]: https://bugs.freedesktop.org/show_bug.cgi?id=103166
  [fdo#103167]: https://bugs.freedesktop.org/show_bug.cgi?id=103167
  [fdo#103191]: https://bugs.freedesktop.org/show_bug.cgi?id=103191
  [fdo#103232]: https://bugs.freedesktop.org/show_bug.cgi?id=103232
  [fdo#103540]: https://bugs.freedesktop.org/show_bug.cgi?id=103540
  [fdo#103558]: https://bugs.freedesktop.org/show_bug.cgi?id=103558
  [fdo#103927]: https://bugs.freedesktop.org/show_bug.cgi?id=103927
  [fdo#104782]: https://bugs.freedesktop.org/show_bug.cgi?id=104782
  [fdo#105363]: https://bugs.freedesktop.org/show_bug.cgi?id=105363
  [fdo#105602]: https://bugs.freedesktop.org/show_bug.cgi?id=105602
  [fdo#105763]: https://bugs.freedesktop.org/show_bug.cgi?id=105763
  [fdo#106510]: https://bugs.freedesktop.org/show_bug.cgi?id=106510
  [fdo#106538]: https://bugs.freedesktop.org/show_bug.cgi?id=106538
  [fdo#107725]: https://bugs.freedesktop.org/show_bug.cgi?id=107725
  [fdo#107956]: https://bugs.freedesktop.org/show_bug.cgi?id=107956
  [fdo#108145]: https://bugs.freedesktop.org/show_bug.cgi?id=108145
  [fdo#108888]: https://bugs.freedesktop.org/show_bug.cgi?id=108888
  [fdo#108948]: https://bugs.freedesktop.org/show_bug.cgi?id=108948
  [fdo#108950]: https://bugs.freedesktop.org/show_bug.cgi?id=108950
  [fdo#99912]: https://bugs.freedesktop.org/show_bug.cgi?id=99912


Participating hosts (7 -> 5)
------------------------------

  Missing    (2): shard-skl shard-iclb 


Build changes
-------------

    * IGT: IGT_4748 -> IGTPW_2163
    * Piglit: piglit_4509 -> None

  CI_DRM_5324: 93009153b9bde1d65ca49bb1729cd1111591144b @ git://anongit.freedesktop.org/gfx-ci/linux
  IGTPW_2163: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_2163/
  IGT_4748: 90c76cb9bb47a5a3ebb34ad6b1a557bc02d39713 @ git://anongit.freedesktop.org/xorg/app/intel-gpu-tools
  piglit_4509: fdc5a4ca11124ab8413c7988896eec4c97336694 @ git://anongit.freedesktop.org/piglit

== Logs ==

For more details see: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_2163/
_______________________________________________
igt-dev mailing list
igt-dev@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/igt-dev

^ permalink raw reply	[flat|nested] 7+ messages in thread
* Re: [igt-dev] [PATCH i-g-t v2] runner: Add explicit "notrun" results for tests that were not executed
  2018-12-17 13:07 [igt-dev] [PATCH i-g-t v2] runner: Add explicit "notrun" results for tests that were not executed Petri Latvala
                   ` (2 preceding siblings ...)
  2018-12-17 15:07 ` [igt-dev] ✗ Fi.CI.IGT: failure " Patchwork
@ 2018-12-20 12:02 ` Arkadiusz Hiler
  2018-12-20 12:04   ` Martin Peres
  2018-12-20 12:02 ` [igt-dev] ✗ Fi.CI.IGT: failure for " Patchwork
  4 siblings, 1 reply; 7+ messages in thread
From: Arkadiusz Hiler @ 2018-12-20 12:02 UTC (permalink / raw)
  To: Petri Latvala; +Cc: igt-dev, Martin Peres

On Mon, Dec 17, 2018 at 03:07:40PM +0200, Petri Latvala wrote:
> When possible, all tests we know we were going to attempt to execute
> now appear in the results as "notrun". The only known case where it's
> not possible to add an explicit "notrun" is when running in
> multiple-mode, because "no subtests" and "run all subtests, we didn't
> list them beforehand" are represented the same.
> 
> Note: A function call is commented out in resultgen.c because it needs
> the function to exist, and that's in a patch that is still in flight.
> 
> v2: Rebase and adjust to already landed json changes
> 
> Signed-off-by: Petri Latvala <petri.latvala@intel.com>
> Cc: Martin Peres <martin.peres@linux.intel.com>
> Cc: Arkadiusz Hiler <arkadiusz.hiler@intel.com>
> Acked-by: Martin Peres <martin.peres@linux.intel.com>
Reviewed-by: Arkadiusz Hiler <arkadiusz.hiler@intel.com>
_______________________________________________
igt-dev mailing list
igt-dev@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/igt-dev

^ permalink raw reply	[flat|nested] 7+ messages in thread
* [igt-dev] ✗ Fi.CI.IGT: failure for runner: Add explicit "notrun" results for tests that were not executed
  2018-12-17 13:07 [igt-dev] [PATCH i-g-t v2] runner: Add explicit "notrun" results for tests that were not executed Petri Latvala
                   ` (3 preceding siblings ...)
  2018-12-20 12:02 ` [igt-dev] [PATCH i-g-t v2] " Arkadiusz Hiler
@ 2018-12-20 12:02 ` Patchwork
  4 siblings, 0 replies; 7+ messages in thread
From: Patchwork @ 2018-12-20 12:02 UTC (permalink / raw)
  To: Petri Latvala; +Cc: igt-dev

== Series Details ==

Series: runner: Add explicit "notrun" results for tests that were not executed
URL   : https://patchwork.freedesktop.org/series/54136/
State : failure

== Summary ==

CI Bug Log - changes from CI_DRM_5324_full -> IGTPW_2163_full
====================================================

Summary
-------

  **FAILURE**

  Serious unknown changes coming with IGTPW_2163_full absolutely need to be
  verified manually.
  
  If you think the reported changes have nothing to do with the changes
  introduced in IGTPW_2163_full, please notify your bug team to allow them
  to document this new failure mode, which will reduce false positives in CI.

  External URL: https://patchwork.freedesktop.org/api/1.0/series/54136/revisions/1/mbox/

Possible new issues
-------------------

  Here are the unknown changes that may have been introduced in IGTPW_2163_full:

### IGT changes ###

#### Possible regressions ####

  * igt@gem_busy@extended-semaphore-blt:
    - shard-hsw:          PASS -> FAIL

  
#### Warnings ####

  * igt@kms_frontbuffer_tracking@fbc-1p-primscrn-spr-indfb-draw-render:
    - shard-snb:          SKIP -> PASS

  * igt@pm_rc6_residency@rc6-accuracy:
    - shard-kbl:          SKIP -> PASS

  
Known issues
------------

  Here are the changes found in IGTPW_2163_full that come from known issues:

### IGT changes ###

#### Issues hit ####

  * igt@gem_exec_fence@basic-await-default:
    - shard-hsw:          PASS -> FAIL [fdo#108888]

  * igt@kms_busy@extended-pageflip-modeset-hang-oldfb-render-b:
    - shard-glk:          PASS -> DMESG-WARN [fdo#107956]

  * igt@kms_ccs@pipe-b-crc-sprite-planes-basic:
    - shard-glk:          PASS -> FAIL [fdo#108145]

  * igt@kms_cursor_crc@cursor-256x256-random:
    - shard-apl:          PASS -> FAIL [fdo#103232] +4

  * igt@kms_cursor_crc@cursor-64x64-sliding:
    - shard-glk:          PASS -> FAIL [fdo#103232] +4

  * igt@kms_cursor_crc@cursor-size-change:
    - shard-kbl:          PASS -> FAIL [fdo#103232] +1

  * igt@kms_cursor_legacy@pipe-a-single-bo:
    - shard-hsw:          PASS -> INCOMPLETE [fdo#103540]

  * igt@kms_flip@2x-flip-vs-expired-vblank-interruptible:
    - shard-glk:          PASS -> FAIL [fdo#105363]

  * igt@kms_frontbuffer_tracking@fbc-1p-primscrn-spr-indfb-draw-pwrite:
    - shard-apl:          NOTRUN -> FAIL [fdo#103167]

  * igt@kms_frontbuffer_tracking@fbc-1p-primscrn-spr-indfb-fullscreen:
    - shard-glk:          PASS -> FAIL [fdo#103167] +1

  * igt@kms_frontbuffer_tracking@fbc-1p-primscrn-spr-indfb-onoff:
    - shard-apl:          PASS -> FAIL [fdo#103167]
    - shard-kbl:          PASS -> FAIL [fdo#103167]

  * igt@kms_plane@pixel-format-pipe-b-planes-source-clamping:
    - shard-glk:          PASS -> FAIL [fdo#108948]

  * igt@kms_plane@plane-position-covered-pipe-a-planes:
    - shard-glk:          PASS -> FAIL [fdo#103166] +3
    - shard-apl:          PASS -> FAIL [fdo#103166] +1
    - shard-kbl:          PASS -> FAIL [fdo#103166] +1

  * igt@kms_plane_multiple@atomic-pipe-b-tiling-y:
    - shard-apl:          NOTRUN -> FAIL [fdo#103166]

  * igt@kms_rotation_crc@multiplane-rotation-cropping-top:
    - shard-kbl:          PASS -> DMESG-FAIL [fdo#108950]

  * igt@kms_setmode@basic:
    - shard-hsw:          PASS -> FAIL [fdo#99912]

  
#### Possible fixes ####

  * igt@kms_ccs@pipe-a-crc-sprite-planes-basic:
    - shard-apl:          FAIL [fdo#106510] / [fdo#108145] -> PASS
    - shard-kbl:          FAIL [fdo#107725] / [fdo#108145] -> PASS

  * igt@kms_color@pipe-c-legacy-gamma:
    - shard-kbl:          FAIL [fdo#104782] -> PASS
    - shard-apl:          FAIL [fdo#104782] -> PASS

  * igt@kms_cursor_crc@cursor-128x128-onscreen:
    - shard-apl:          FAIL [fdo#103232] -> PASS +2

  * igt@kms_cursor_crc@cursor-256x256-dpms:
    - shard-kbl:          FAIL [fdo#103232] -> PASS

  * igt@kms_cursor_crc@cursor-256x256-suspend:
    - shard-apl:          FAIL [fdo#103191] / [fdo#103232] -> PASS

  * igt@kms_cursor_crc@cursor-64x64-suspend:
    - shard-kbl:          FAIL [fdo#103191] / [fdo#103232] -> PASS

  * igt@kms_flip@plain-flip-fb-recreate:
    - shard-apl:          INCOMPLETE [fdo#103927] -> PASS

  * igt@kms_frontbuffer_tracking@fbc-1p-primscrn-cur-indfb-onoff:
    - shard-apl:          FAIL [fdo#103167] -> PASS +2
    - shard-kbl:          FAIL [fdo#103167] -> PASS +1

  * igt@kms_frontbuffer_tracking@fbc-2p-primscrn-cur-indfb-draw-blt:
    - shard-glk:          FAIL [fdo#103167] -> PASS +4

  * igt@kms_pipe_crc_basic@hang-read-crc-pipe-c:
    - shard-apl:          DMESG-WARN [fdo#103558] / [fdo#105602] -> PASS +20

  * igt@kms_plane_multiple@atomic-pipe-a-tiling-x:
    - shard-apl:          FAIL [fdo#103166] -> PASS +4

  * igt@kms_plane_multiple@atomic-pipe-a-tiling-y:
    - shard-kbl:          FAIL [fdo#103166] -> PASS

  * igt@kms_plane_multiple@atomic-pipe-b-tiling-y:
    - shard-glk:          FAIL [fdo#103166] -> PASS +3

  * igt@kms_rotation_crc@multiplane-rotation-cropping-top:
    - shard-glk:          DMESG-FAIL [fdo#105763] / [fdo#106538] -> PASS

  
#### Warnings ####

  * igt@kms_plane_multiple@atomic-pipe-b-tiling-yf:
    - shard-apl:          DMESG-FAIL [fdo#103166] / [fdo#103558] / [fdo#105602] -> FAIL [fdo#103166]

  
  [fdo#103166]: https://bugs.freedesktop.org/show_bug.cgi?id=103166
  [fdo#103167]: https://bugs.freedesktop.org/show_bug.cgi?id=103167
  [fdo#103191]: https://bugs.freedesktop.org/show_bug.cgi?id=103191
  [fdo#103232]: https://bugs.freedesktop.org/show_bug.cgi?id=103232
  [fdo#103540]: https://bugs.freedesktop.org/show_bug.cgi?id=103540
  [fdo#103558]: https://bugs.freedesktop.org/show_bug.cgi?id=103558
  [fdo#103927]: https://bugs.freedesktop.org/show_bug.cgi?id=103927
  [fdo#104782]: https://bugs.freedesktop.org/show_bug.cgi?id=104782
  [fdo#105363]: https://bugs.freedesktop.org/show_bug.cgi?id=105363
  [fdo#105602]: https://bugs.freedesktop.org/show_bug.cgi?id=105602
  [fdo#105763]: https://bugs.freedesktop.org/show_bug.cgi?id=105763
  [fdo#106510]: https://bugs.freedesktop.org/show_bug.cgi?id=106510
  [fdo#106538]: https://bugs.freedesktop.org/show_bug.cgi?id=106538
  [fdo#107725]: https://bugs.freedesktop.org/show_bug.cgi?id=107725
  [fdo#107956]: https://bugs.freedesktop.org/show_bug.cgi?id=107956
  [fdo#108145]: https://bugs.freedesktop.org/show_bug.cgi?id=108145
  [fdo#108888]: https://bugs.freedesktop.org/show_bug.cgi?id=108888
  [fdo#108948]: https://bugs.freedesktop.org/show_bug.cgi?id=108948
  [fdo#108950]: https://bugs.freedesktop.org/show_bug.cgi?id=108950
  [fdo#99912]: https://bugs.freedesktop.org/show_bug.cgi?id=99912


Participating hosts (7 -> 5)
------------------------------

  Missing    (2): shard-skl shard-iclb 


Build changes
-------------

    * IGT: IGT_4748 -> IGTPW_2163
    * Piglit: piglit_4509 -> None

  CI_DRM_5324: 93009153b9bde1d65ca49bb1729cd1111591144b @ git://anongit.freedesktop.org/gfx-ci/linux
  IGTPW_2163: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_2163/
  IGT_4748: 90c76cb9bb47a5a3ebb34ad6b1a557bc02d39713 @ git://anongit.freedesktop.org/xorg/app/intel-gpu-tools
  piglit_4509: fdc5a4ca11124ab8413c7988896eec4c97336694 @ git://anongit.freedesktop.org/piglit

== Logs ==

For more details see: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_2163/
_______________________________________________
igt-dev mailing list
igt-dev@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/igt-dev

^ permalink raw reply	[flat|nested] 7+ messages in thread
* Re: [igt-dev] [PATCH i-g-t v2] runner: Add explicit "notrun" results for tests that were not executed
  2018-12-20 12:02 ` [igt-dev] [PATCH i-g-t v2] " Arkadiusz Hiler
@ 2018-12-20 12:04   ` Martin Peres
  0 siblings, 0 replies; 7+ messages in thread
From: Martin Peres @ 2018-12-20 12:04 UTC (permalink / raw)
  To: Arkadiusz Hiler, Petri Latvala; +Cc: igt-dev

On 20/12/2018 14:02, Arkadiusz Hiler wrote:
> On Mon, Dec 17, 2018 at 03:07:40PM +0200, Petri Latvala wrote:
>> When possible, all tests we know we were going to attempt to execute
>> now appear in the results as "notrun". The only known case where it's
>> not possible to add an explicit "notrun" is when running in
>> multiple-mode, because "no subtests" and "run all subtests, we didn't
>> list them beforehand" are represented the same.
>>
>> Note: A function call is commented out in resultgen.c because it needs
>> the function to exist, and that's in a patch that is still in flight.
>>
>> v2: Rebase and adjust to already landed json changes
>>
>> Signed-off-by: Petri Latvala <petri.latvala@intel.com>
>> Cc: Martin Peres <martin.peres@linux.intel.com>
>> Cc: Arkadiusz Hiler <arkadiusz.hiler@intel.com>
>> Acked-by: Martin Peres <martin.peres@linux.intel.com>
> Reviewed-by: Arkadiusz Hiler <arkadiusz.hiler@intel.com>
> 

And the failure caught in CI is actually from this known bug:
https://bugs.freedesktop.org/show_bug.cgi?id=109119

So, good for pushing!
_______________________________________________
igt-dev mailing list
igt-dev@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/igt-dev

^ permalink raw reply	[flat|nested] 7+ messages in thread
end of thread, other threads:[~2018-12-20 12:04 UTC | newest]

Thread overview: 7+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2018-12-17 13:07 [igt-dev] [PATCH i-g-t v2] runner: Add explicit "notrun" results for tests that were not executed Petri Latvala
2018-12-17 13:29 ` Petri Latvala
2018-12-17 14:11 ` [igt-dev] ✓ Fi.CI.BAT: success for " Patchwork
2018-12-17 15:07 ` [igt-dev] ✗ Fi.CI.IGT: failure " Patchwork
2018-12-20 12:02 ` [igt-dev] [PATCH i-g-t v2] " Arkadiusz Hiler
2018-12-20 12:04   ` Martin Peres
2018-12-20 12:02 ` [igt-dev] ✗ Fi.CI.IGT: failure for " Patchwork

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox