* [igt-dev] [PATCH i-g-t 0/5] New test runner to rule them all, v3
@ 2018-08-08 11:06 Petri Latvala
2018-08-08 11:06 ` [igt-dev] [PATCH i-g-t 1/5] lib: Print subtest starting/ending line to stderr too Petri Latvala
` (7 more replies)
0 siblings, 8 replies; 11+ messages in thread
From: Petri Latvala @ 2018-08-08 11:06 UTC (permalink / raw)
To: igt-dev
Once more, with feeling.
This version addresses Arek's review feedback, adds autotools
building, and adds a couple of TODO comments. Hopefully this version
is good to land so work can begin on implementing those TODO items
without changes getting lost where only interdiff can reveal them.
Petri Latvala (5):
lib: Print subtest starting/ending line to stderr too
lib: Export igt_gettime and igt_time_elapsed
uwildmat: Case-insensitive test selection
runner: New test runner
runner: Unit tests for the runner
Makefile.am | 4 +
configure.ac | 15 +
lib/igt_core.c | 38 +-
lib/igt_core.h | 20 +
lib/uwildmat/uwildmat.c | 11 +-
meson.build | 6 +
meson_options.txt | 6 +
runner/.gitignore | 4 +
runner/Makefile.am | 43 ++
runner/executor.c | 1022 +++++++++++++++++++++++++++++++++++++++++
runner/executor.h | 49 ++
runner/job_list.c | 484 +++++++++++++++++++
runner/job_list.h | 37 ++
runner/meson.build | 49 ++
runner/output_strings.h | 55 +++
runner/resultgen.c | 962 ++++++++++++++++++++++++++++++++++++++
runner/resultgen.h | 9 +
runner/results.c | 26 ++
runner/resume.c | 47 ++
runner/runner.c | 40 ++
runner/runner_tests.c | 973 +++++++++++++++++++++++++++++++++++++++
runner/settings.c | 502 ++++++++++++++++++++
runner/settings.h | 111 +++++
runner/testdata/Makefile.am | 25 +
runner/testdata/meson.build | 20 +
runner/testdata/no-subtests.c | 6 +
runner/testdata/skippers.c | 14 +
runner/testdata/successtest.c | 10 +
28 files changed, 4573 insertions(+), 15 deletions(-)
create mode 100644 runner/.gitignore
create mode 100644 runner/Makefile.am
create mode 100644 runner/executor.c
create mode 100644 runner/executor.h
create mode 100644 runner/job_list.c
create mode 100644 runner/job_list.h
create mode 100644 runner/meson.build
create mode 100644 runner/output_strings.h
create mode 100644 runner/resultgen.c
create mode 100644 runner/resultgen.h
create mode 100644 runner/results.c
create mode 100644 runner/resume.c
create mode 100644 runner/runner.c
create mode 100644 runner/runner_tests.c
create mode 100644 runner/settings.c
create mode 100644 runner/settings.h
create mode 100644 runner/testdata/Makefile.am
create mode 100644 runner/testdata/meson.build
create mode 100644 runner/testdata/no-subtests.c
create mode 100644 runner/testdata/skippers.c
create mode 100644 runner/testdata/successtest.c
--
2.14.1
_______________________________________________
igt-dev mailing list
igt-dev@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/igt-dev
^ permalink raw reply [flat|nested] 11+ messages in thread
* [igt-dev] [PATCH i-g-t 1/5] lib: Print subtest starting/ending line to stderr too
2018-08-08 11:06 [igt-dev] [PATCH i-g-t 0/5] New test runner to rule them all, v3 Petri Latvala
@ 2018-08-08 11:06 ` Petri Latvala
2018-08-08 11:17 ` Chris Wilson
2018-08-08 11:06 ` [igt-dev] [PATCH i-g-t 2/5] lib: Export igt_gettime and igt_time_elapsed Petri Latvala
` (6 subsequent siblings)
7 siblings, 1 reply; 11+ messages in thread
From: Petri Latvala @ 2018-08-08 11:06 UTC (permalink / raw)
To: igt-dev
when instructed via the environment. This is needed for the new test
runner to properly assign stderr output to the correct subtest.
v2:
Print the subtest result from skip_subtests_henceforth handling also
to stderr.
Signed-off-by: Petri Latvala <petri.latvala@intel.com>
---
lib/igt_core.c | 17 ++++++++++++++++-
1 file changed, 16 insertions(+), 1 deletion(-)
diff --git a/lib/igt_core.c b/lib/igt_core.c
index 5e6fb7ad..d3385756 100644
--- a/lib/igt_core.c
+++ b/lib/igt_core.c
@@ -301,6 +301,8 @@ GKeyFile *igt_key_file;
char *igt_frame_dump_path;
+static bool stderr_needs_sentinel = false;
+
const char *igt_test_name(void)
{
return command_str;
@@ -643,6 +645,8 @@ static void common_init_env(void)
}
igt_frame_dump_path = getenv("IGT_FRAME_DUMP_PATH");
+
+ stderr_needs_sentinel = getenv("IGT_SENTINEL_ON_STDERR") != NULL;
}
static int common_init(int *argc, char **argv,
@@ -919,12 +923,20 @@ bool __igt_run_subtest(const char *subtest_name)
(!__igt_plain_output) ? "\x1b[1m" : "", subtest_name,
skip_subtests_henceforth == SKIP ?
"SKIP" : "FAIL", (!__igt_plain_output) ? "\x1b[0m" : "");
+ fflush(stdout);
+ if (stderr_needs_sentinel)
+ fprintf(stderr, "Subtest %s: %s\n", subtest_name,
+ skip_subtests_henceforth == SKIP ?
+ "SKIP" : "FAIL");
return false;
}
igt_kmsg(KMSG_INFO "%s: starting subtest %s\n",
command_str, subtest_name);
- igt_debug("Starting subtest: %s\n", subtest_name);
+ igt_info("Starting subtest: %s\n", subtest_name);
+ fflush(stdout);
+ if (stderr_needs_sentinel)
+ fprintf(stderr, "Starting subtest: %s\n", subtest_name);
_igt_log_buffer_reset();
@@ -979,6 +991,9 @@ static void exit_subtest(const char *result)
in_subtest, result, time_elapsed(&subtest_time, &now),
(!__igt_plain_output) ? "\x1b[0m" : "");
fflush(stdout);
+ if (stderr_needs_sentinel)
+ fprintf(stderr, "Subtest %s: %s (%.3fs)\n",
+ in_subtest, result, time_elapsed(&subtest_time, &now));
igt_terminate_spin_batches();
--
2.14.1
_______________________________________________
igt-dev mailing list
igt-dev@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/igt-dev
^ permalink raw reply related [flat|nested] 11+ messages in thread
* [igt-dev] [PATCH i-g-t 2/5] lib: Export igt_gettime and igt_time_elapsed
2018-08-08 11:06 [igt-dev] [PATCH i-g-t 0/5] New test runner to rule them all, v3 Petri Latvala
2018-08-08 11:06 ` [igt-dev] [PATCH i-g-t 1/5] lib: Print subtest starting/ending line to stderr too Petri Latvala
@ 2018-08-08 11:06 ` Petri Latvala
2018-08-08 11:06 ` [igt-dev] [PATCH i-g-t 3/5] uwildmat: Case-insensitive test selection Petri Latvala
` (5 subsequent siblings)
7 siblings, 0 replies; 11+ messages in thread
From: Petri Latvala @ 2018-08-08 11:06 UTC (permalink / raw)
To: igt-dev
Signed-off-by: Petri Latvala <petri.latvala@intel.com>
---
lib/igt_core.c | 23 +++++++++++------------
lib/igt_core.h | 20 ++++++++++++++++++++
2 files changed, 31 insertions(+), 12 deletions(-)
diff --git a/lib/igt_core.c b/lib/igt_core.c
index d3385756..c52c0818 100644
--- a/lib/igt_core.c
+++ b/lib/igt_core.c
@@ -403,9 +403,8 @@ void igt_kmsg(const char *format, ...)
#define time_valid(ts) ((ts)->tv_sec || (ts)->tv_nsec)
-static double
-time_elapsed(struct timespec *then,
- struct timespec* now)
+double igt_time_elapsed(struct timespec *then,
+ struct timespec *now)
{
double elapsed = -1.;
@@ -417,7 +416,7 @@ time_elapsed(struct timespec *then,
return elapsed;
}
-static int gettime(struct timespec *ts)
+int igt_gettime(struct timespec *ts)
{
memset(ts, 0, sizeof(*ts));
errno = 0;
@@ -450,7 +449,7 @@ uint64_t igt_nsec_elapsed(struct timespec *start)
{
struct timespec now;
- gettime(&now);
+ igt_gettime(&now);
if ((start->tv_sec | start->tv_nsec) == 0) {
*start = now;
return 0;
@@ -811,7 +810,7 @@ out:
igt_install_exit_handler(common_exit_handler);
if (!test_with_subtests)
- gettime(&subtest_time);
+ igt_gettime(&subtest_time);
for (i = 0; (optind + i) < *argc; i++)
argv[i + 1] = argv[optind + i];
@@ -940,7 +939,7 @@ bool __igt_run_subtest(const char *subtest_name)
_igt_log_buffer_reset();
- gettime(&subtest_time);
+ igt_gettime(&subtest_time);
return (in_subtest = subtest_name);
}
@@ -985,15 +984,15 @@ static void exit_subtest(const char *result)
{
struct timespec now;
- gettime(&now);
+ igt_gettime(&now);
igt_info("%sSubtest %s: %s (%.3fs)%s\n",
(!__igt_plain_output) ? "\x1b[1m" : "",
- in_subtest, result, time_elapsed(&subtest_time, &now),
+ in_subtest, result, igt_time_elapsed(&subtest_time, &now),
(!__igt_plain_output) ? "\x1b[0m" : "");
fflush(stdout);
if (stderr_needs_sentinel)
fprintf(stderr, "Subtest %s: %s (%.3fs)\n",
- in_subtest, result, time_elapsed(&subtest_time, &now));
+ in_subtest, result, igt_time_elapsed(&subtest_time, &now));
igt_terminate_spin_batches();
@@ -1484,7 +1483,7 @@ void igt_exit(void)
struct timespec now;
const char *result;
- gettime(&now);
+ igt_gettime(&now);
switch (igt_exitcode) {
case IGT_EXIT_SUCCESS:
@@ -1501,7 +1500,7 @@ void igt_exit(void)
}
printf("%s (%.3fs)\n",
- result, time_elapsed(&subtest_time, &now));
+ result, igt_time_elapsed(&subtest_time, &now));
}
exit(igt_exitcode);
diff --git a/lib/igt_core.h b/lib/igt_core.h
index aaf1b626..b80e1702 100644
--- a/lib/igt_core.h
+++ b/lib/igt_core.h
@@ -928,6 +928,26 @@ extern enum igt_log_level igt_log_level;
void igt_set_timeout(unsigned int seconds,
const char *op);
+/**
+ * igt_gettime:
+ * @ts: current monotonic clock reading
+ *
+ * Reports the current time in the monotonic clock.
+ * Returns: 0 on success, -errno on failure.
+ */
+int igt_gettime(struct timespec *ts);
+
+/**
+ * igt_time_elapsed:
+ * @then: Earlier timestamp
+ * @now: Later timestamp
+ *
+ * Returns: Time between two timestamps in seconds, as a floating
+ * point number.
+ */
+double igt_time_elapsed(struct timespec *then,
+ struct timespec *now);
+
/**
* igt_nsec_elapsed:
* @start: measure from this point in time
--
2.14.1
_______________________________________________
igt-dev mailing list
igt-dev@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/igt-dev
^ permalink raw reply related [flat|nested] 11+ messages in thread
* [igt-dev] [PATCH i-g-t 3/5] uwildmat: Case-insensitive test selection
2018-08-08 11:06 [igt-dev] [PATCH i-g-t 0/5] New test runner to rule them all, v3 Petri Latvala
2018-08-08 11:06 ` [igt-dev] [PATCH i-g-t 1/5] lib: Print subtest starting/ending line to stderr too Petri Latvala
2018-08-08 11:06 ` [igt-dev] [PATCH i-g-t 2/5] lib: Export igt_gettime and igt_time_elapsed Petri Latvala
@ 2018-08-08 11:06 ` Petri Latvala
2018-08-08 11:07 ` [igt-dev] [PATCH i-g-t v3 4/5] runner: New test runner Petri Latvala
` (4 subsequent siblings)
7 siblings, 0 replies; 11+ messages in thread
From: Petri Latvala @ 2018-08-08 11:06 UTC (permalink / raw)
To: igt-dev
Since we only use plain ascii in subtest names, using non-locale-aware
tolower() to compare case-insensitively works.
Doing this within uwildmat instead of tolowering the subtest name and
then calling uwildmat() is required, because of selection strings
like:
foo,bar,!Foo
The above line will select subtest Bar, and not select Foo.
Signed-off-by: Petri Latvala <petri.latvala@intel.com>
---
lib/uwildmat/uwildmat.c | 11 ++++++++---
1 file changed, 8 insertions(+), 3 deletions(-)
diff --git a/lib/uwildmat/uwildmat.c b/lib/uwildmat/uwildmat.c
index 09155865..3beafce2 100644
--- a/lib/uwildmat/uwildmat.c
+++ b/lib/uwildmat/uwildmat.c
@@ -93,6 +93,7 @@
** accompanying test suite achieves 100% coverage of this file.
*/
+#include <ctype.h>
#include <string.h>
#include <stdint.h>
#include "uwildmat/uwildmat.h"
@@ -222,6 +223,7 @@ match_class(uint32_t text, const unsigned char *start,
const unsigned char *p = start;
uint32_t first = 0;
uint32_t last;
+ uint32_t lc = tolower(text);
/* Check for an inverted character class (starting with ^). If the
character matches the character class, we return !reversed; that way,
@@ -244,12 +246,13 @@ match_class(uint32_t text, const unsigned char *start,
if (allowrange && *p == '-' && p < end) {
p++;
p += utf8_decode(p, end, &last);
- if (text >= first && text <= last)
+ if ((text >= first && text <= last) ||
+ (lc >= first && lc <= last))
return !reversed;
allowrange = false;
} else {
p += utf8_decode(p, end, &first);
- if (text == first)
+ if (text == first || lc == first)
return !reversed;
allowrange = true;
}
@@ -272,6 +275,7 @@ match_pattern(const unsigned char *text, const unsigned char *start,
bool ismeta;
int matched, width;
uint32_t c;
+ unsigned char lc;
for (; p <= end; p++) {
if (!*text && *p != '*')
@@ -284,7 +288,8 @@ match_pattern(const unsigned char *text, const unsigned char *start,
/* Fall through. */
default:
- if (*text++ != *p)
+ lc = tolower(*text);
+ if (*text++ != *p && lc != *p)
return false;
break;
--
2.14.1
_______________________________________________
igt-dev mailing list
igt-dev@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/igt-dev
^ permalink raw reply related [flat|nested] 11+ messages in thread
* [igt-dev] [PATCH i-g-t v3 4/5] runner: New test runner
2018-08-08 11:06 [igt-dev] [PATCH i-g-t 0/5] New test runner to rule them all, v3 Petri Latvala
` (2 preceding siblings ...)
2018-08-08 11:06 ` [igt-dev] [PATCH i-g-t 3/5] uwildmat: Case-insensitive test selection Petri Latvala
@ 2018-08-08 11:07 ` Petri Latvala
2018-08-08 11:07 ` [igt-dev] [PATCH i-g-t v4 5/5] runner: Unit tests for the runner Petri Latvala
` (3 subsequent siblings)
7 siblings, 0 replies; 11+ messages in thread
From: Petri Latvala @ 2018-08-08 11:07 UTC (permalink / raw)
To: igt-dev
This is a new test runner to replace piglit. Piglit has been very
useful as a test runner, but certain improvements have been very
difficult if possible at all in a generic test running framework.
Important improvements over piglit:
- Faster to launch. Being able to make assumptions about what we're
executing makes it possible to save significant amounts of time. For
example, a testlist file's line "igt@somebinary@somesubtest" already
has all the information we need to construct the correct command
line to execute that particular subtest, instead of listing all
subtests of all test binaries and mapping them to command
lines. Same goes for the regexp filters command line flags -t and
-x; If we use -x somebinaryname, we don't need to list subtests from
somebinaryname, we already know none of them will get executed.
- Logs of incomplete tests. Piglit collects test output to memory and
dumps them to a file when the test is complete. The new runner
writes all output to disk immediately.
- Ability to execute multiple subtests in one binary execution. This
was possible with piglit, but its semantics made it very hard to
implement in practice. For example, having a testlist file not only
selected a subset of tests to run, but also mandated that they be
executed in the same order.
- Flexible timeout support. Instead of mandating a time tests cannot
exceed, the new runner has a timeout on inactivity. Activity is
any output on the test's stdout or stderr, or kernel activity via
/dev/kmsg.
The runner is fairly piglit compatible. The command line is very
similar, with a few additions. IGT_TEST_ROOT environment flag is still
supported, but can also be set via command line (in place of igt.py in
piglit command line).
The results are a set of log files, processed into a piglit-compatible
results.json file (BZ2 compression TODO). There are some new fields in
the json for extra information:
- "igt-version" contains the IGT version line. In
multiple-subtests-mode the version information is only printed once,
so it needs to be duplicated to all subtest results this way.
- "dmesg-warnings" contains the dmesg lines that triggered a
dmesg-warn/dmesg-fail state.
- Runtime information will be different. Piglit takes a timestamp at
the beginning and at the end of execution for runtime. The new
runner uses the subtest output text. The binary execution time will
also be included; The key "igt@somebinary" will have the runtime of
the binary "somebinary", whereas "igt@somebinary@a" etc will have
the runtime of the subtests. Substracting the subtest runtimes from
the binary runtime yields the total time spent doing setup in
igt_fixture blocks.
v2:
- use clock handling from igt_core instead of copypaste
- install results binary
- less magic numbers
- scanf doesn't give empty strings after all
- use designated array initialization with _F_JOURNAL and pals
- add more comments to dump_dmesg
- use signal in kill_child instead of bool
- use more 'usual' return values for execute_entry
- use signal number instead of magic integers
- use IGT_EXIT_INVALID instead of magic 79
- properly remove files in clear_test_result_directory()
- remove magic numbers
- warn if results directory contains extra files
- fix naming in matches_any
- construct command line in a cleaner way in add_subtests()
- clarify error in filtered_job_list
- replace single string fprintfs with fputs
- use getline() more sanely
- refactor string constants to a shared header
- explain non-nul-terminated string handling in resultgen
- saner line parsing
- rename gen_igt_name to generate_piglit_name
- clean up parse_result_string
- explain what we're parsing in resultgen
- explain the runtime accumulation in add_runtime
- refactor result overriding
- stop passing needle sizes to find_line functions
- refactor stdout/stderr parsing
- fix regex whitelist compiling
- add TODO for suppressions.txt
- refactor dmesg parsing
- fill_from_journal returns void
- explain missing result fields with TODO comments
- log_level parsing with typeof
- pass stdout/stderr to usage() instead of a bool
- fix absolute_path overflow
- refactor settings serialization
- remove maybe_strdup function
- refactor job list serialization
- refactor resuming, add new resume binary
- catch mmap failure correctly
v3:
- rename runner to igt_runner, etc
- add meson option for building the runner
- use UPPER_CASE names for string constants
- add TODO comments for future refactoring
- add a midding close()
- const correctness where applicable
- also build with autotools
---
Makefile.am | 4 +
configure.ac | 14 +
meson.build | 6 +
meson_options.txt | 6 +
runner/.gitignore | 3 +
runner/Makefile.am | 31 ++
runner/executor.c | 1022 +++++++++++++++++++++++++++++++++++++++++++++++
runner/executor.h | 49 +++
runner/job_list.c | 484 ++++++++++++++++++++++
runner/job_list.h | 37 ++
runner/meson.build | 41 ++
runner/output_strings.h | 55 +++
runner/resultgen.c | 962 ++++++++++++++++++++++++++++++++++++++++++++
runner/resultgen.h | 9 +
runner/results.c | 26 ++
runner/resume.c | 47 +++
runner/runner.c | 40 ++
runner/settings.c | 502 +++++++++++++++++++++++
runner/settings.h | 111 +++++
19 files changed, 3449 insertions(+)
create mode 100644 runner/.gitignore
create mode 100644 runner/Makefile.am
create mode 100644 runner/executor.c
create mode 100644 runner/executor.h
create mode 100644 runner/job_list.c
create mode 100644 runner/job_list.h
create mode 100644 runner/meson.build
create mode 100644 runner/output_strings.h
create mode 100644 runner/resultgen.c
create mode 100644 runner/resultgen.h
create mode 100644 runner/results.c
create mode 100644 runner/resume.c
create mode 100644 runner/runner.c
create mode 100644 runner/settings.c
create mode 100644 runner/settings.h
diff --git a/Makefile.am b/Makefile.am
index 5bc486b9..044c0a07 100644
--- a/Makefile.am
+++ b/Makefile.am
@@ -27,6 +27,10 @@ if BUILD_TESTS
SUBDIRS += tests
endif
+if BUILD_RUNNER
+SUBDIRS += runner
+endif
+
if BUILD_X86
if BUILD_ASSEMBLER
SUBDIRS += assembler
diff --git a/configure.ac b/configure.ac
index 416a3240..b73cc9f0 100644
--- a/configure.ac
+++ b/configure.ac
@@ -374,6 +374,18 @@ fi
AM_CONDITIONAL(BUILD_TESTS, [test "x$BUILD_TESTS" = xyes])
AC_DEFINE_UNQUOTED(TARGET_CPU_PLATFORM, ["$host_cpu"], [Target platform])
+AC_ARG_ENABLE(runner,
+ AS_HELP_STRING([--disable-runner],
+ [Enable building test runner (default: auto)]),
+ [BUILD_RUNNER=$enableval], [BUILD_RUNNER="auto"])
+if test "x$BUILD_RUNNER" = xauto; then
+ PKG_CHECK_EXISTS([json-c], [BUILD_RUNNER=yes], [BUILD_RUNNER=no])
+fi
+if test "x$BUILD_RUNNER" = xyes; then
+ PKG_CHECK_MODULES(JSONC, [json-c])
+fi
+AM_CONDITIONAL(BUILD_RUNNER, [test "x$BUILD_RUNNER" = xyes])
+
files="broadwell cherryview haswell ivybridge sandybridge valleyview skylake"
for file in $files; do
REGISTER_FILES="$REGISTER_FILES $file `cat $srcdir/tools/registers/$file`"
@@ -408,6 +420,7 @@ AC_CONFIG_FILES([
assembler/test/Makefile
assembler/intel-gen4asm.pc
overlay/Makefile
+ runner/Makefile
])
AC_CONFIG_FILES([tools/intel_aubdump], [chmod +x tools/intel_aubdump])
@@ -431,6 +444,7 @@ echo " Assembler : ${enable_assembler}"
echo " Debugger : ${enable_debugger}"
echo " Overlay : X: ${enable_overlay_xlib}, Xv: ${enable_overlay_xvlib}"
echo " x86-specific tools : ${build_x86}"
+echo " Test runner : ${BUILD_RUNNER}"
echo ""
echo " • API-Documentation : ${enable_gtk_doc}"
echo " • Fail on warnings : ${enable_werror}"
diff --git a/meson.build b/meson.build
index 682f44da..d45db9ac 100644
--- a/meson.build
+++ b/meson.build
@@ -38,6 +38,8 @@ _build_docs = false
_docs_required = false
_build_tests = false
_tests_required = false
+_build_runner = false
+_runner_required = false
build_overlay = get_option('build_overlay')
overlay_backends = get_option('overlay_backends')
@@ -48,6 +50,7 @@ build_chamelium = get_option('build_chamelium')
build_docs = get_option('build_docs')
build_tests = get_option('build_tests')
with_libdrm = get_option('with_libdrm')
+build_runner = get_option('build_runner')
_build_overlay = build_overlay != 'false'
_overlay_required = build_overlay == 'true'
@@ -61,6 +64,8 @@ _build_docs = build_docs != 'false'
_docs_required = build_docs == 'true'
_build_tests = build_tests != 'false'
_tests_required = build_tests == 'true'
+_build_runner = build_runner != 'false'
+_runner_required = build_runner == 'true'
build_info = []
@@ -228,6 +233,7 @@ else
endif
subdir('benchmarks')
subdir('tools')
+subdir('runner')
if libdrm_intel.found()
subdir('assembler')
endif
diff --git a/meson_options.txt b/meson_options.txt
index 05e63463..89a3731c 100644
--- a/meson_options.txt
+++ b/meson_options.txt
@@ -52,6 +52,12 @@ option('with_libdrm',
choices : ['', 'auto', 'intel', 'nouveau', 'amdgpu'],
description : 'libdrm libraries to be used')
+option('build_runner',
+ type : 'combo',
+ value : 'auto',
+ choices : ['auto', 'true', 'false'],
+ description : 'Build test runner')
+
option('use_rpath',
type : 'boolean',
value : false,
diff --git a/runner/.gitignore b/runner/.gitignore
new file mode 100644
index 00000000..e3919fa7
--- /dev/null
+++ b/runner/.gitignore
@@ -0,0 +1,3 @@
+igt_runner
+igt_resume
+igt_results
diff --git a/runner/Makefile.am b/runner/Makefile.am
new file mode 100644
index 00000000..9c13a83c
--- /dev/null
+++ b/runner/Makefile.am
@@ -0,0 +1,31 @@
+
+if BUILD_RUNNER
+
+runnerlib = librunner.la
+noinst_LTLIBRARIES = $(runnerlib)
+librunner_la_SOURCES = \
+ settings.c \
+ job_list.c \
+ executor.c \
+ resultgen.c \
+ $(NULL)
+
+bin_PROGRAMS = \
+ igt_runner \
+ igt_resume \
+ igt_results \
+ $(NULL)
+
+LDADD = $(runnerlib) $(JSONC_LIBS) ../lib/libintel_tools.la
+
+igt_runner_SOURCES = runner.c
+igt_resume_SOURCES = resume.c
+igt_results_SOURCES = results.c
+
+AM_CFLAGS = $(JSONC_CFLAGS) \
+ $(CWARNFLAGS) -Wno-unused-result $(DEBUG_CFLAGS) \
+ -I$(srcdir)/.. \
+ -I$(srcdir)/../lib \
+ -D_GNU_SOURCE
+
+endif
diff --git a/runner/executor.c b/runner/executor.c
new file mode 100644
index 00000000..60cf4da5
--- /dev/null
+++ b/runner/executor.c
@@ -0,0 +1,1022 @@
+#include <errno.h>
+#include <fcntl.h>
+#include <linux/watchdog.h>
+#include <signal.h>
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+#include <sys/ioctl.h>
+#include <sys/select.h>
+#include <sys/signalfd.h>
+#include <sys/stat.h>
+#include <sys/types.h>
+#include <sys/utsname.h>
+#include <sys/wait.h>
+#include <time.h>
+#include <unistd.h>
+
+#include "igt_core.h"
+#include "executor.h"
+#include "output_strings.h"
+
+static struct {
+ int *fds;
+ size_t num_dogs;
+} watchdogs;
+
+static void close_watchdogs(struct settings *settings)
+{
+ size_t i;
+
+ if (settings && settings->log_level >= LOG_LEVEL_VERBOSE)
+ printf("Closing watchdogs\n");
+
+ for (i = 0; i < watchdogs.num_dogs; i++) {
+ write(watchdogs.fds[i], "V", 1);
+ close(watchdogs.fds[i]);
+ }
+}
+
+static void close_watchdogs_atexit(void)
+{
+ close_watchdogs(NULL);
+}
+
+static void init_watchdogs(struct settings *settings)
+{
+ int i;
+ char name[32];
+ int fd;
+
+ memset(&watchdogs, 0, sizeof(watchdogs));
+
+ if (!settings->use_watchdog || settings->inactivity_timeout <= 0)
+ return;
+
+ if (settings->log_level >= LOG_LEVEL_VERBOSE) {
+ printf("Initializing watchdogs\n");
+ }
+
+ atexit(close_watchdogs_atexit);
+
+ for (i = 0; ; i++) {
+ snprintf(name, sizeof(name), "/dev/watchdog%d", i);
+ if ((fd = open(name, O_RDWR | O_CLOEXEC)) < 0)
+ break;
+
+ watchdogs.num_dogs++;
+ watchdogs.fds = realloc(watchdogs.fds, watchdogs.num_dogs * sizeof(int));
+ watchdogs.fds[i] = fd;
+
+ if (settings->log_level >= LOG_LEVEL_VERBOSE)
+ printf(" %s\n", name);
+ }
+}
+
+static int watchdogs_set_timeout(int timeout)
+{
+ size_t i;
+ int orig_timeout = timeout;
+
+ for (i = 0; i < watchdogs.num_dogs; i++) {
+ if (ioctl(watchdogs.fds[i], WDIOC_SETTIMEOUT, &timeout)) {
+ write(watchdogs.fds[i], "V", 1);
+ close(watchdogs.fds[i]);
+ watchdogs.fds[i] = -1;
+ continue;
+ }
+
+ if (timeout < orig_timeout) {
+ /*
+ * Timeout of this caliber refused. We want to
+ * use the same timeout for all devices.
+ */
+ return watchdogs_set_timeout(timeout);
+ }
+ }
+
+ return timeout;
+}
+
+static void ping_watchdogs(void)
+{
+ size_t i;
+
+ for (i = 0; i < watchdogs.num_dogs; i++) {
+ ioctl(watchdogs.fds[i], WDIOC_KEEPALIVE, 0);
+ }
+}
+
+static void prune_subtest(struct job_list_entry *entry, char *subtest)
+{
+ char *excl;
+
+ /*
+ * Subtest pruning is done by adding exclusion strings to the
+ * subtest list. The last matching item on the subtest
+ * selection command line flag decides whether to run a
+ * subtest, see igt_core.c for details. If the list is empty,
+ * the expected subtest set is unknown, so we need to add '*'
+ * first so we can start excluding.
+ */
+
+ if (entry->subtest_count == 0) {
+ entry->subtest_count++;
+ entry->subtests = realloc(entry->subtests, entry->subtest_count * sizeof(*entry->subtests));
+ entry->subtests[0] = strdup("*");
+ }
+
+ excl = malloc(strlen(subtest) + 2);
+ excl[0] = '!';
+ strcpy(excl + 1, subtest);
+
+ entry->subtest_count++;
+ entry->subtests = realloc(entry->subtests, entry->subtest_count * sizeof(*entry->subtests));
+ entry->subtests[entry->subtest_count - 1] = excl;
+}
+
+static bool prune_from_journal(struct job_list_entry *entry, int fd)
+{
+ char *subtest;
+ FILE *f;
+ bool any_pruned = false;
+
+ /*
+ * Each journal line is a subtest that has been started, or
+ * the line 'exit:$exitcode (time)', or 'timeout:$exitcode (time)'.
+ */
+
+ f = fdopen(fd, "r");
+ if (!f)
+ return false;
+
+ while (fscanf(f, "%ms", &subtest) == 1) {
+ if (!strncmp(subtest, EXECUTOR_EXIT, strlen(EXECUTOR_EXIT))) {
+ /* Fully done. Mark that by making the binary name invalid. */
+ fscanf(f, " (%*fs)");
+ entry->binary[0] = '\0';
+ free(subtest);
+ continue;
+ }
+
+ if (!strncmp(subtest, EXECUTOR_TIMEOUT, strlen(EXECUTOR_TIMEOUT))) {
+ fscanf(f, " (%*fs)");
+ free(subtest);
+ continue;
+ }
+
+ prune_subtest(entry, subtest);
+
+ free(subtest);
+ any_pruned = true;
+ }
+
+ fclose(f);
+ return any_pruned;
+}
+
+static const char *filenames[_F_LAST] = {
+ [_F_JOURNAL] = "journal.txt",
+ [_F_OUT] = "out.txt",
+ [_F_ERR] = "err.txt",
+ [_F_DMESG] = "dmesg.txt",
+};
+
+static int open_at_end(int dirfd, const char *name)
+{
+ int fd = openat(dirfd, name, O_RDWR | O_CREAT | O_CLOEXEC, 0666);
+ char last;
+
+ if (fd >= 0) {
+ if (lseek(fd, -1, SEEK_END) >= 0 &&
+ read(fd, &last, 1) == 1 &&
+ last != '\n') {
+ write(fd, "\n", 1);
+ }
+ lseek(fd, 0, SEEK_END);
+ }
+
+ return fd;
+}
+
+static int open_for_reading(int dirfd, const char *name)
+{
+ return openat(dirfd, name, O_RDONLY);
+}
+
+bool open_output_files(int dirfd, int *fds, bool write)
+{
+ int i;
+ int (*openfunc)(int, const char*) = write ? open_at_end : open_for_reading;
+
+ for (i = 0; i < _F_LAST; i++) {
+ if ((fds[i] = openfunc(dirfd, filenames[i])) < 0) {
+ while (--i >= 0)
+ close(fds[i]);
+ return false;
+ }
+ }
+
+ return true;
+}
+
+void close_outputs(int *fds)
+{
+ int i;
+
+ for (i = 0; i < _F_LAST; i++) {
+ close(fds[i]);
+ }
+}
+
+static void dump_dmesg(int kmsgfd, int outfd)
+{
+ /*
+ * Write kernel messages to the log file until we reach
+ * 'now'. Unfortunately, /dev/kmsg doesn't support seeking to
+ * -1 from SEEK_END so we need to use a second fd to read a
+ * message to match against, or stop when we reach EAGAIN.
+ */
+
+ int comparefd = open("/dev/kmsg", O_RDONLY | O_NONBLOCK);
+ unsigned flags;
+ unsigned long long seq, cmpseq, usec;
+ char cont;
+ char buf[256];
+ ssize_t r;
+
+ if (comparefd < 0)
+ return;
+
+ if (fcntl(kmsgfd, F_SETFL, O_NONBLOCK))
+ return;
+
+ while (1) {
+ if (comparefd >= 0) {
+ r = read(comparefd, buf, sizeof(buf) - 1);
+ if (r < 0) {
+ if (errno != EAGAIN && errno != EPIPE)
+ return;
+ } else {
+ buf[r] = '\0';
+ if (sscanf(buf, "%u,%llu,%llu,%c;",
+ &flags, &cmpseq, &usec, &cont) == 4) {
+ /* Reading comparison record done. */
+ close(comparefd);
+ comparefd = -1;
+ }
+ }
+ }
+
+ r = read(kmsgfd, buf, sizeof(buf));
+ if (r <= 0) {
+ if (errno == EPIPE)
+ continue;
+
+ /*
+ * If EAGAIN, we're done. If some other error,
+ * we can't do anything anyway.
+ */
+ return;
+ }
+
+ write(outfd, buf, r);
+
+ if (comparefd < 0 && sscanf(buf, "%u,%llu,%llu,%c;",
+ &flags, &seq, &usec, &cont) == 4) {
+ /*
+ * Comparison record has been read, compare
+ * the sequence number to see if we have read
+ * enough.
+ */
+ if (seq >= cmpseq)
+ return;
+ }
+ }
+}
+
+static bool kill_child(int sig, pid_t child)
+{
+ /*
+ * Send the signal to the child directly, and to the child's
+ * process group.
+ */
+ kill(-child, sig);
+ if (kill(child, sig) && errno == ESRCH) {
+ fprintf(stderr, "Child process does not exist. This shouldn't happen.\n");
+ return false;
+ }
+
+ return true;
+}
+
+/*
+ * Returns:
+ * =0 - Success
+ * <0 - Failure executing
+ * >0 - Timeout happened, need to recreate from journal
+ */
+static int monitor_output(pid_t child,
+ int outfd, int errfd, int kmsgfd, int sigfd,
+ int *outputs,
+ struct settings *settings)
+{
+ fd_set set;
+ char buf[256];
+ char *outbuf = NULL;
+ size_t outbufsize = 0;
+ char current_subtest[256] = {};
+ struct signalfd_siginfo siginfo;
+ ssize_t s;
+ int n, status;
+ int nfds = outfd;
+ int timeout = settings->inactivity_timeout;
+ int timeout_intervals = 1, intervals_left;
+ int wd_extra = 10;
+ int killed = 0; /* 0 if not killed, signal number otherwise */
+ struct timespec time_beg, time_end;
+ bool aborting = false;
+
+ igt_gettime(&time_beg);
+
+ if (errfd > nfds)
+ nfds = errfd;
+ if (kmsgfd > nfds)
+ nfds = kmsgfd;
+ if (sigfd > nfds)
+ nfds = sigfd;
+ nfds++;
+
+ if (timeout > 0) {
+ /*
+ * Use original timeout plus some leeway. If we're still
+ * alive, we want to kill the test process instead of cutting
+ * power.
+ */
+ int wd_timeout = watchdogs_set_timeout(timeout + wd_extra);
+
+ if (wd_timeout < timeout + wd_extra) {
+ /* Watchdog timeout smaller, so ping it more often */
+ if (wd_timeout - wd_extra < 0)
+ wd_extra = wd_timeout / 2;
+ timeout_intervals = timeout / (wd_timeout - wd_extra);
+ intervals_left = timeout_intervals;
+ timeout /= timeout_intervals;
+
+ if (settings->log_level >= LOG_LEVEL_VERBOSE) {
+ printf("Watchdog doesn't support the timeout we requested (shortened to %d seconds).\n"
+ "Using %d intervals of %d seconds.\n",
+ wd_timeout, timeout_intervals, timeout);
+ }
+ }
+ }
+
+ while (outfd >= 0 || errfd >= 0 || sigfd >= 0) {
+ struct timeval tv = { .tv_sec = timeout };
+
+ FD_ZERO(&set);
+ if (outfd >= 0)
+ FD_SET(outfd, &set);
+ if (errfd >= 0)
+ FD_SET(errfd, &set);
+ if (kmsgfd >= 0)
+ FD_SET(kmsgfd, &set);
+ if (sigfd >= 0)
+ FD_SET(sigfd, &set);
+
+ n = select(nfds, &set, NULL, NULL, timeout == 0 ? NULL : &tv);
+ if (n < 0) {
+ /* TODO */
+ return -1;
+ }
+
+ if (n == 0) {
+ intervals_left--;
+ if (intervals_left) {
+ continue;
+ }
+
+ ping_watchdogs();
+
+ switch (killed) {
+ case 0:
+ if (settings->log_level >= LOG_LEVEL_NORMAL) {
+ printf("Timeout. Killing the current test with SIGTERM.\n");
+ }
+
+ killed = SIGTERM;
+ if (!kill_child(killed, child))
+ return -1;
+
+ /*
+ * Now continue the loop and let the
+ * dying child be handled normally.
+ */
+ timeout = 2; /* Timeout for waiting selected by fair dice roll. */
+ watchdogs_set_timeout(20);
+ intervals_left = timeout_intervals = 1;
+ break;
+ case SIGTERM:
+ if (settings->log_level >= LOG_LEVEL_NORMAL) {
+ printf("Timeout. Killing the current test with SIGKILL.\n");
+ }
+
+ killed = SIGKILL;
+ if (!kill_child(killed, child))
+ return -1;
+
+ intervals_left = timeout_intervals = 1;
+ break;
+ case SIGKILL:
+ /* Nothing that can be done, really. Let's tell the caller we want to abort. */
+ if (settings->log_level >= LOG_LEVEL_NORMAL) {
+ fprintf(stderr, "Child refuses to die. Aborting.\n");
+ }
+ close_watchdogs(settings);
+ free(outbuf);
+ close(outfd);
+ close(errfd);
+ close(kmsgfd);
+ close(sigfd);
+ return -1;
+ }
+
+ continue;
+ }
+
+ intervals_left = timeout_intervals;
+ ping_watchdogs();
+
+ /* TODO: Refactor these handlers to their own functions */
+ if (outfd >= 0 && FD_ISSET(outfd, &set)) {
+ char *newline;
+
+ s = read(outfd, buf, sizeof(buf));
+ if (s <= 0) {
+ if (s < 0) {
+ fprintf(stderr, "Error reading test's stdout: %s\n",
+ strerror(errno));
+ }
+
+ close(outfd);
+ outfd = -1;
+ goto out_end;
+ }
+
+ write(outputs[_F_OUT], buf, s);
+ if (settings->sync) {
+ fdatasync(outputs[_F_OUT]);
+ }
+
+ outbuf = realloc(outbuf, outbufsize + s);
+ memcpy(outbuf + outbufsize, buf, s);
+ outbufsize += s;
+
+ while ((newline = memchr(outbuf, '\n', outbufsize)) != NULL) {
+ size_t linelen = newline - outbuf + 1;
+
+ if (linelen > strlen(STARTING_SUBTEST) &&
+ !memcmp(outbuf, STARTING_SUBTEST, strlen(STARTING_SUBTEST))) {
+ write(outputs[_F_JOURNAL], outbuf + strlen(STARTING_SUBTEST),
+ linelen - strlen(STARTING_SUBTEST));
+ memcpy(current_subtest, outbuf + strlen(STARTING_SUBTEST),
+ linelen - strlen(STARTING_SUBTEST));
+ current_subtest[linelen - strlen(STARTING_SUBTEST)] = '\0';
+
+ if (settings->log_level >= LOG_LEVEL_VERBOSE) {
+ fwrite(outbuf, 1, linelen, stdout);
+ }
+ }
+ if (linelen > strlen(SUBTEST_RESULT) &&
+ !memcmp(outbuf, SUBTEST_RESULT, strlen(SUBTEST_RESULT))) {
+ char *delim = memchr(outbuf, ':', linelen);
+
+ if (delim != NULL) {
+ size_t subtestlen = delim - outbuf - strlen(SUBTEST_RESULT);
+ if (memcmp(current_subtest, outbuf + strlen(SUBTEST_RESULT),
+ subtestlen)) {
+ /* Result for a test that didn't ever start */
+ write(outputs[_F_JOURNAL],
+ outbuf + strlen(SUBTEST_RESULT),
+ subtestlen);
+ write(outputs[_F_JOURNAL], "\n", 1);
+ if (settings->sync) {
+ fdatasync(outputs[_F_JOURNAL]);
+ }
+ current_subtest[0] = '\0';
+ }
+
+ if (settings->log_level >= LOG_LEVEL_VERBOSE) {
+ fwrite(outbuf, 1, linelen, stdout);
+ }
+ }
+ }
+
+ memmove(outbuf, newline + 1, outbufsize - linelen);
+ outbufsize -= linelen;
+ }
+ }
+ out_end:
+
+ if (errfd >= 0 && FD_ISSET(errfd, &set)) {
+ s = read(errfd, buf, sizeof(buf));
+ if (s <= 0) {
+ if (s < 0) {
+ fprintf(stderr, "Error reading test's stderr: %s\n",
+ strerror(errno));
+ }
+ close(errfd);
+ errfd = -1;
+ } else {
+ write(outputs[_F_ERR], buf, s);
+ if (settings->sync) {
+ fdatasync(outputs[_F_ERR]);
+ }
+ }
+ }
+
+ if (kmsgfd >= 0 && FD_ISSET(kmsgfd, &set)) {
+ s = read(kmsgfd, buf, sizeof(buf));
+ if (s < 0) {
+ if (errno != EPIPE) {
+ fprintf(stderr, "Error reading from kmsg, stopping monitoring: %s\n",
+ strerror(errno));
+ close(kmsgfd);
+ kmsgfd = -1;
+ }
+ } else {
+ write(outputs[_F_DMESG], buf, s);
+ if (settings->sync) {
+ fdatasync(outputs[_F_DMESG]);
+ }
+ }
+ }
+
+ if (sigfd >= 0 && FD_ISSET(sigfd, &set)) {
+ double time;
+
+ s = read(sigfd, &siginfo, sizeof(siginfo));
+ if (s < 0) {
+ fprintf(stderr, "Error reading from signalfd: %s\n",
+ strerror(errno));
+ continue;
+ } else if (siginfo.ssi_signo == SIGCHLD) {
+ if (child != waitpid(child, &status, WNOHANG)) {
+ fprintf(stderr, "Failed to reap child\n");
+ status = 9999;
+ } else if (WIFEXITED(status)) {
+ status = WEXITSTATUS(status);
+ if (status >= 128) {
+ status = 128 - status;
+ }
+ } else if (WIFSIGNALED(status)) {
+ status = -WTERMSIG(status);
+ } else {
+ status = 9999;
+ }
+ } else {
+ /* We're dying, so we're taking them with us */
+ if (settings->log_level >= LOG_LEVEL_NORMAL)
+ printf("Abort requested, terminating children\n");
+
+ aborting = true;
+ timeout = 2;
+ killed = SIGTERM;
+ if (!kill_child(killed, child))
+ return -1;
+
+ continue;
+ }
+
+ igt_gettime(&time_end);
+
+ time = igt_time_elapsed(&time_beg, &time_end);
+ if (time < 0.0)
+ time = 0.0;
+
+ if (!aborting) {
+ dprintf(outputs[_F_JOURNAL], "%s%d (%.3fs)\n",
+ killed ? EXECUTOR_TIMEOUT : EXECUTOR_EXIT,
+ status, time);
+ if (settings->sync) {
+ fdatasync(outputs[_F_JOURNAL]);
+ }
+ }
+
+ close(sigfd);
+ sigfd = -1;
+ child = 0;
+ }
+ }
+
+ dump_dmesg(kmsgfd, outputs[_F_DMESG]);
+ if (settings->sync)
+ fdatasync(outputs[_F_DMESG]);
+
+ free(outbuf);
+ close(outfd);
+ close(errfd);
+ close(kmsgfd);
+ close(sigfd);
+
+ if (aborting)
+ return -1;
+
+ return killed;
+}
+
+static void execute_test_process(int outfd, int errfd,
+ struct settings *settings,
+ struct job_list_entry *entry)
+{
+ char *argv[4] = {};
+ size_t rootlen;
+
+ dup2(outfd, STDOUT_FILENO);
+ dup2(errfd, STDERR_FILENO);
+
+ setpgid(0, 0);
+
+ rootlen = strlen(settings->test_root);
+ argv[0] = malloc(rootlen + strlen(entry->binary) + 2);
+ strcpy(argv[0], settings->test_root);
+ argv[0][rootlen] = '/';
+ strcpy(argv[0] + rootlen + 1, entry->binary);
+
+ if (entry->subtest_count) {
+ size_t argsize;
+ size_t i;
+
+ argv[1] = strdup("--run-subtest");
+ argsize = strlen(entry->subtests[0]);
+ argv[2] = malloc(argsize + 1);
+ strcpy(argv[2], entry->subtests[0]);
+
+ for (i = 1; i < entry->subtest_count; i++) {
+ char *sub = entry->subtests[i];
+ size_t sublen = strlen(sub);
+
+ argv[2] = realloc(argv[2], argsize + sublen + 2);
+ argv[2][argsize] = ',';
+ strcpy(argv[2] + argsize + 1, sub);
+ argsize += sublen + 1;
+ }
+ }
+
+ execv(argv[0], argv);
+ fprintf(stderr, "Cannot execute %s\n", argv[0]);
+ exit(IGT_EXIT_INVALID);
+}
+
+static int digits(size_t num)
+{
+ int ret = 0;
+ while (num) {
+ num /= 10;
+ ret++;
+ }
+
+ if (ret == 0) ret++;
+ return ret;
+}
+
+/*
+ * Returns:
+ * =0 - Success
+ * <0 - Failure executing
+ * >0 - Timeout happened, need to recreate from journal
+ */
+static int execute_entry(size_t idx,
+ size_t total,
+ struct settings *settings,
+ struct job_list_entry *entry,
+ int testdirfd, int resdirfd)
+{
+ int dirfd;
+ int outputs[_F_LAST];
+ int kmsgfd;
+ int sigfd;
+ sigset_t mask;
+ int outpipe[2] = { -1, -1 };
+ int errpipe[2] = { -1, -1 };
+ char name[32];
+ pid_t child;
+ int result;
+
+ snprintf(name, sizeof(name), "%zd", idx);
+ mkdirat(resdirfd, name, 0777);
+ if ((dirfd = openat(resdirfd, name, O_DIRECTORY | O_RDONLY | O_CLOEXEC)) < 0) {
+ fprintf(stderr, "Error accessing individual test result directory\n");
+ return -1;
+ }
+
+ if (!open_output_files(dirfd, outputs, true)) {
+ close(dirfd);
+ fprintf(stderr, "Error opening output files\n");
+ return -1;
+ }
+
+ if (settings->sync) {
+ fsync(dirfd);
+ fsync(resdirfd);
+ }
+
+ if (pipe(outpipe) || pipe(errpipe)) {
+ close_outputs(outputs);
+ close(dirfd);
+ close(outpipe[0]);
+ close(outpipe[1]);
+ close(errpipe[0]);
+ close(errpipe[1]);
+ fprintf(stderr, "Error creating pipes: %s\n", strerror(errno));
+ return -1;
+ }
+
+ if ((kmsgfd = open("/dev/kmsg", O_RDONLY | O_CLOEXEC)) < 0) {
+ fprintf(stderr, "Warning: Cannot open /dev/kmsg\n");
+ } else {
+ /* TODO: Checking of abort conditions in pre-execute dmesg */
+ lseek(kmsgfd, 0, SEEK_END);
+ }
+
+ sigemptyset(&mask);
+ sigaddset(&mask, SIGCHLD);
+ sigaddset(&mask, SIGINT);
+ sigaddset(&mask, SIGTERM);
+ sigaddset(&mask, SIGQUIT);
+ sigprocmask(SIG_BLOCK, &mask, NULL);
+ sigfd = signalfd(-1, &mask, O_CLOEXEC);
+
+ if (sigfd < 0) {
+ /* TODO: Handle better */
+ fprintf(stderr, "Cannot monitor child process with signalfd\n");
+ close(outpipe[0]);
+ close(errpipe[0]);
+ close(outpipe[1]);
+ close(errpipe[1]);
+ close(kmsgfd);
+ close_outputs(outputs);
+ close(dirfd);
+ return -1;
+ }
+
+ if (settings->log_level >= LOG_LEVEL_NORMAL) {
+ int width = digits(total);
+ printf("[%0*zd/%0*zd] %s", width, idx + 1, width, total, entry->binary);
+ if (entry->subtest_count > 0) {
+ size_t i;
+ const char *delim = "";
+
+ printf(" (");
+ for (i = 0; i < entry->subtest_count; i++) {
+ printf("%s%s", delim, entry->subtests[i]);
+ delim = ", ";
+ }
+ printf(")");
+ }
+ printf("\n");
+ }
+
+ if ((child = fork())) {
+ int outfd = outpipe[0];
+ int errfd = errpipe[0];
+ close(outpipe[1]);
+ close(errpipe[1]);
+
+ result = monitor_output(child, outfd, errfd, kmsgfd, sigfd,
+ outputs, settings);
+ } else {
+ int outfd = outpipe[1];
+ int errfd = errpipe[1];
+ close(outpipe[0]);
+ close(errpipe[0]);
+
+ sigprocmask(SIG_UNBLOCK, &mask, NULL);
+
+ setenv("IGT_SENTINEL_ON_STDERR", "1", 1);
+
+ execute_test_process(outfd, errfd, settings, entry);
+ }
+
+ /* TODO: Refactor this whole function to use onion teardown */
+ close(outpipe[1]);
+ close(errpipe[1]);
+ close(kmsgfd);
+ close_outputs(outputs);
+ close(dirfd);
+
+ return result;
+}
+
+static int remove_file(int dirfd, const char *name)
+{
+ return unlinkat(dirfd, name, 0) && errno != ENOENT;
+}
+
+static bool clear_test_result_directory(int dirfd)
+{
+ int i;
+
+ for (i = 0; i < _F_LAST; i++) {
+ if (remove_file(dirfd, filenames[i])) {
+ fprintf(stderr, "Error deleting %s from test result directory: %s\n",
+ filenames[i],
+ strerror(errno));
+ return false;
+ }
+ }
+
+ return true;
+}
+
+static bool clear_old_results(char *path)
+{
+ int dirfd;
+ size_t i;
+
+ if ((dirfd = open(path, O_DIRECTORY | O_RDONLY)) < 0) {
+ if (errno == ENOENT) {
+ /* Successfully cleared if it doesn't even exist */
+ return true;
+ }
+
+ fprintf(stderr, "Error clearing old results: %s\n", strerror(errno));
+ return false;
+ }
+
+ if (unlinkat(dirfd, "uname.txt", 0) && errno != ENOENT) {
+ close(dirfd);
+ fprintf(stderr, "Error clearing old results: %s\n", strerror(errno));
+ return false;
+ }
+
+ for (i = 0; true; i++) {
+ char name[32];
+ int resdirfd;
+
+ snprintf(name, sizeof(name), "%zd", i);
+ if ((resdirfd = openat(dirfd, name, O_DIRECTORY | O_RDONLY)) < 0)
+ break;
+
+ if (!clear_test_result_directory(resdirfd)) {
+ close(resdirfd);
+ close(dirfd);
+ return false;
+ }
+ close(resdirfd);
+ if (unlinkat(dirfd, name, AT_REMOVEDIR)) {
+ fprintf(stderr,
+ "Warning: Result directory %s contains extra files\n",
+ name);
+ }
+ }
+
+ close(dirfd);
+
+ return true;
+}
+
+bool initialize_execute_state_from_resume(int dirfd,
+ struct execute_state *state,
+ struct settings *settings,
+ struct job_list *list)
+{
+ struct job_list_entry *entry;
+ int resdirfd, fd, i;
+
+ free_settings(settings);
+ free_job_list(list);
+ memset(state, 0, sizeof(*state));
+
+ if (!read_settings(settings, dirfd) ||
+ !read_job_list(list, dirfd)) {
+ close(dirfd);
+ return false;
+ }
+
+ for (i = list->size; i >= 0; i--) {
+ char name[32];
+
+ snprintf(name, sizeof(name), "%d", i);
+ if ((resdirfd = openat(dirfd, name, O_DIRECTORY | O_RDONLY)) >= 0)
+ break;
+ }
+
+ if (i < 0)
+ /* Nothing has been executed yet, state is fine as is */
+ goto success;
+
+ entry = &list->entries[i];
+ state->next = i;
+ if ((fd = openat(resdirfd, filenames[_F_JOURNAL], O_RDONLY)) >= 0) {
+ if (!prune_from_journal(entry, fd)) {
+ /*
+ * The test does not have subtests, or
+ * incompleted before the first subtest
+ * began. Either way, not suitable to
+ * re-run.
+ */
+ state->next = i + 1;
+ } else if (entry->binary[0] == '\0') {
+ /* This test is fully completed */
+ state->next = i + 1;
+ }
+
+ close(fd);
+ }
+
+ success:
+ close(resdirfd);
+ close(dirfd);
+
+ return true;
+}
+
+bool initialize_execute_state(struct execute_state *state,
+ struct settings *settings,
+ struct job_list *job_list)
+{
+ memset(state, 0, sizeof(*state));
+
+ if (!validate_settings(settings))
+ return false;
+
+ if (!serialize_settings(settings) ||
+ !serialize_job_list(job_list, settings))
+ return false;
+
+ if (settings->overwrite &&
+ !clear_old_results(settings->results_path))
+ return false;
+
+ return true;
+}
+
+bool execute(struct execute_state *state,
+ struct settings *settings,
+ struct job_list *job_list)
+{
+ struct utsname unamebuf;
+ int resdirfd, testdirfd, unamefd;
+
+ if ((resdirfd = open(settings->results_path, O_DIRECTORY | O_RDONLY)) < 0) {
+ /* Initialize state should have done this */
+ fprintf(stderr, "Error: Failure opening results path %s\n",
+ settings->results_path);
+ return false;
+ }
+
+ if ((testdirfd = open(settings->test_root, O_DIRECTORY | O_RDONLY)) < 0) {
+ fprintf(stderr, "Error: Failure opening test root %s\n",
+ settings->test_root);
+ close(resdirfd);
+ return false;
+ }
+
+ /* TODO: On resume, don't rewrite, verify that content matches current instead */
+ if ((unamefd = openat(resdirfd, "uname.txt", O_CREAT | O_WRONLY | O_TRUNC, 0666)) < 0) {
+ fprintf(stderr, "Error: Failure creating opening uname.txt: %s\n",
+ strerror(errno));
+ close(testdirfd);
+ close(resdirfd);
+ return false;
+ }
+
+ init_watchdogs(settings);
+
+ if (!uname(&unamebuf)) {
+ dprintf(unamefd, "%s %s %s %s %s\n",
+ unamebuf.sysname,
+ unamebuf.nodename,
+ unamebuf.release,
+ unamebuf.version,
+ unamebuf.machine);
+ } else {
+ dprintf(unamefd, "uname() failed\n");
+ }
+ close(unamefd);
+
+ for (; state->next < job_list->size;
+ state->next++) {
+ int result = execute_entry(state->next,
+ job_list->size,
+ settings,
+ &job_list->entries[state->next],
+ testdirfd, resdirfd);
+ if (result != 0) {
+ close(testdirfd);
+ close_watchdogs(settings);
+ if (result > 0) {
+ initialize_execute_state_from_resume(resdirfd, state, settings, job_list);
+ return execute(state, settings, job_list);
+ }
+ close(resdirfd);
+ return false;
+ }
+ }
+
+ close(testdirfd);
+ close(resdirfd);
+ close_watchdogs(settings);
+ return true;
+}
diff --git a/runner/executor.h b/runner/executor.h
new file mode 100644
index 00000000..8fe1605b
--- /dev/null
+++ b/runner/executor.h
@@ -0,0 +1,49 @@
+#ifndef RUNNER_EXECUTOR_H
+#define RUNNER_EXECUTOR_H
+
+#include "job_list.h"
+#include "settings.h"
+
+struct execute_state
+{
+ size_t next;
+};
+
+enum {
+ _F_JOURNAL,
+ _F_OUT,
+ _F_ERR,
+ _F_DMESG,
+ _F_LAST,
+};
+
+bool open_output_files(int dirfd, int *fds, bool write);
+void close_outputs(int *fds);
+
+/*
+ * Initialize execute_state object to a state where it's ready to
+ * execute. Will validate the settings and serialize both settings and
+ * the job_list into the result directory, overwriting old files if
+ * settings set to do so.
+ */
+bool initialize_execute_state(struct execute_state *state,
+ struct settings *settings,
+ struct job_list *job_list);
+
+/*
+ * Initialize execute_state object to a state where it's ready to
+ * resume an already existing run. settings and job_list must have
+ * been initialized with init_settings et al, and will be read from
+ * the result directory pointed to by dirfd.
+ */
+bool initialize_execute_state_from_resume(int dirfd,
+ struct execute_state *state,
+ struct settings *settings,
+ struct job_list *job_list);
+
+bool execute(struct execute_state *state,
+ struct settings *settings,
+ struct job_list *job_list);
+
+
+#endif
diff --git a/runner/job_list.c b/runner/job_list.c
new file mode 100644
index 00000000..e3f820c3
--- /dev/null
+++ b/runner/job_list.c
@@ -0,0 +1,484 @@
+#include <errno.h>
+#include <fcntl.h>
+#include <linux/limits.h>
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+#include <sys/stat.h>
+#include <sys/types.h>
+#include <unistd.h>
+
+#include "job_list.h"
+#include "igt_core.h"
+
+static bool matches_any(const char *str, struct regex_list *list)
+{
+ size_t i;
+
+ for (i = 0; i < list->size; i++) {
+ if (regexec(list->regexes[i], str,
+ (size_t)0, NULL, 0) == 0) {
+ return true;
+ }
+ }
+
+ return false;
+}
+
+static void add_job_list_entry(struct job_list *job_list,
+ char *binary,
+ char **subtests,
+ size_t subtest_count)
+{
+ struct job_list_entry *entry;
+
+ job_list->size++;
+ job_list->entries = realloc(job_list->entries, job_list->size * sizeof(*job_list->entries));
+ entry = &job_list->entries[job_list->size - 1];
+
+ entry->binary = binary;
+ entry->subtests = subtests;
+ entry->subtest_count = subtest_count;
+}
+
+static void add_subtests(struct job_list *job_list, struct settings *settings,
+ char *binary,
+ struct regex_list *include, struct regex_list *exclude)
+{
+ FILE *p;
+ char cmd[256] = {};
+ char *subtestname;
+ char **subtests = NULL;
+ size_t num_subtests = 0;
+ int s;
+
+ s = snprintf(cmd, sizeof(cmd), "%s/%s --list-subtests",
+ settings->test_root, binary);
+ if (s < 0) {
+ fprintf(stderr, "Failure generating command string, this shouldn't happen.\n");
+ return;
+ }
+
+ if (s >= sizeof(cmd)) {
+ fprintf(stderr, "Path to binary too long, ignoring: %s/%s\n",
+ settings->test_root, binary);
+ return;
+ }
+
+ p = popen(cmd, "r");
+ if (!p) {
+ fprintf(stderr, "popen failed when executing %s: %s\n",
+ cmd,
+ strerror(errno));
+ return;
+ }
+
+ while (fscanf(p, "%ms", &subtestname) == 1) {
+ if (exclude && exclude->size && matches_any(subtestname, exclude)) {
+ free(subtestname);
+ continue;
+ }
+
+ if (include && include->size && !matches_any(subtestname, include)) {
+ free(subtestname);
+ continue;
+ }
+
+ if (settings->multiple_mode) {
+ num_subtests++;
+ subtests = realloc(subtests, num_subtests * sizeof(*subtests));
+ subtests[num_subtests - 1] = strdup(subtestname);
+ } else {
+ subtests = malloc(sizeof(*subtests));
+ *subtests = strdup(subtestname);
+ add_job_list_entry(job_list, strdup(binary), subtests, 1);
+ subtests = NULL;
+ }
+
+ free(subtestname);
+ }
+
+ if (num_subtests)
+ add_job_list_entry(job_list, strdup(binary), subtests, num_subtests);
+
+ s = pclose(p);
+ if (s == 0) {
+ return;
+ } else if (s == -1) {
+ fprintf(stderr, "popen error when executing %s: %s\n", binary, strerror(errno));
+ } else if (WIFEXITED(s)) {
+ if (WEXITSTATUS(s) == IGT_EXIT_INVALID) {
+ /* No subtests on this one */
+ if (exclude && exclude->size && matches_any(binary, exclude)) {
+ return;
+ }
+ if (!include || !include->size || matches_any(binary, include)) {
+ add_job_list_entry(job_list, strdup(binary), NULL, 0);
+ return;
+ }
+ }
+ } else {
+ fprintf(stderr, "Test binary %s died unexpectedly\n", binary);
+ }
+}
+
+static bool filtered_job_list(struct job_list *job_list,
+ struct settings *settings,
+ int fd)
+{
+ FILE *f;
+ char buf[128];
+
+ if (job_list->entries != NULL) {
+ fprintf(stderr, "Caller didn't clear the job list, this shouldn't happen\n");
+ exit(1);
+ }
+
+ f = fdopen(fd, "r");
+
+ while (fscanf(f, "%127s", buf) == 1) {
+ if (!strcmp(buf, "TESTLIST") || !(strcmp(buf, "END")))
+ continue;
+
+ /*
+ * If the binary name matches exclude filters, no
+ * subtests are added.
+ */
+ if (settings->exclude_regexes.size && matches_any(buf, &settings->exclude_regexes))
+ continue;
+
+ /*
+ * If the binary name matches include filters (or include filters not present),
+ * all subtests except those matching exclude filters are added.
+ */
+ if (!settings->include_regexes.size || matches_any(buf, &settings->include_regexes)) {
+ if (settings->multiple_mode && !settings->exclude_regexes.size)
+ /*
+ * Optimization; we know that all
+ * subtests will be included, so we
+ * get to omit executing
+ * --list-subtests.
+ */
+ add_job_list_entry(job_list, strdup(buf), NULL, 0);
+ else
+ add_subtests(job_list, settings, buf,
+ NULL, &settings->exclude_regexes);
+ continue;
+ }
+
+ /*
+ * Binary name doesn't match exclude or include filters.
+ */
+ add_subtests(job_list, settings, buf,
+ &settings->include_regexes,
+ &settings->exclude_regexes);
+ }
+
+ return job_list->size != 0;
+}
+
+static bool job_list_from_test_list(struct job_list *job_list,
+ struct settings *settings)
+{
+ FILE *f;
+ char *line = NULL;
+ size_t line_len = 0;
+ struct job_list_entry entry = {};
+ bool any = false;
+
+ if ((f = fopen(settings->test_list, "r")) == NULL) {
+ fprintf(stderr, "Cannot open test list file %s\n", settings->test_list);
+ return false;
+ }
+
+ while (1) {
+ char *binary;
+ char *delim;
+
+ if (getline(&line, &line_len, f) == -1) {
+ if (errno == EINTR)
+ continue;
+ else
+ break;
+ }
+
+ /* # starts a comment */
+ if ((delim = strchr(line, '#')) != NULL)
+ *delim = '\0';
+
+ if (sscanf(line, "igt@%ms", &binary) == 1) {
+ if ((delim = strchr(binary, '@')) != NULL)
+ *delim++ = '\0';
+
+ if (!settings->multiple_mode) {
+ char **subtests = NULL;
+ if (delim) {
+ subtests = malloc(sizeof(char*));
+ subtests[0] = strdup(delim);
+ }
+ add_job_list_entry(job_list, strdup(binary),
+ subtests, (size_t)(subtests != NULL));
+ any = true;
+ free(binary);
+ binary = NULL;
+ continue;
+ }
+
+ /*
+ * If the currently built entry has the same
+ * binary, add a subtest. Otherwise submit
+ * what's already built and start a new one.
+ */
+ if (entry.binary && !strcmp(entry.binary, binary)) {
+ if (!delim) {
+ /* ... except we didn't get a subtest */
+ fprintf(stderr,
+ "Error: Unexpected test without subtests "
+ "after same test had subtests\n");
+ free(binary);
+ fclose(f);
+ return false;
+ }
+ entry.subtest_count++;
+ entry.subtests = realloc(entry.subtests,
+ entry.subtest_count *
+ sizeof(*entry.subtests));
+ entry.subtests[entry.subtest_count - 1] = strdup(delim);
+ free(binary);
+ binary = NULL;
+ continue;
+ }
+
+ if (entry.binary) {
+ add_job_list_entry(job_list, entry.binary, entry.subtests, entry.subtest_count);
+ any = true;
+ }
+
+ memset(&entry, 0, sizeof(entry));
+ entry.binary = strdup(binary);
+ if (delim) {
+ entry.subtests = malloc(sizeof(*entry.subtests));
+ entry.subtests[0] = strdup(delim);
+ entry.subtest_count = 1;
+ }
+
+ free(binary);
+ binary = NULL;
+ }
+ }
+
+ if (entry.binary) {
+ add_job_list_entry(job_list, entry.binary, entry.subtests, entry.subtest_count);
+ any = true;
+ }
+
+ free(line);
+ fclose(f);
+ return any;
+}
+
+void init_job_list(struct job_list *job_list)
+{
+ memset(job_list, 0, sizeof(*job_list));
+}
+
+void free_job_list(struct job_list *job_list)
+{
+ int i, k;
+
+ for (i = 0; i < job_list->size; i++) {
+ struct job_list_entry *entry = &job_list->entries[i];
+
+ free(entry->binary);
+ for (k = 0; k < entry->subtest_count; k++) {
+ free(entry->subtests[k]);
+ }
+ free(entry->subtests);
+ }
+ free(job_list->entries);
+ init_job_list(job_list);
+}
+
+bool create_job_list(struct job_list *job_list,
+ struct settings *settings)
+{
+ int dirfd, fd;
+ bool result;
+
+ if (!settings->test_root) {
+ fprintf(stderr, "No test root set; this shouldn't happen\n");
+ return false;
+ }
+
+ free_job_list(job_list);
+
+ dirfd = open(settings->test_root, O_DIRECTORY | O_RDONLY);
+ if (dirfd < 0) {
+ fprintf(stderr, "Test directory %s cannot be opened\n", settings->test_root);
+ return false;
+ }
+
+ fd = openat(dirfd, "test-list.txt", O_RDONLY);
+ if (fd < 0) {
+ fprintf(stderr, "Cannot open %s/test-list.txt\n", settings->test_root);
+ close(dirfd);
+ return false;
+ }
+
+ /*
+ * If a test_list is given (not to be confused with
+ * test-list.txt), we use it directly without making tests
+ * list their subtests. If include/exclude filters are given
+ * we filter them directly from the test_list.
+ */
+ if (settings->test_list)
+ result = job_list_from_test_list(job_list, settings);
+ else
+ result = filtered_job_list(job_list, settings, fd);
+
+ close(fd);
+ close(dirfd);
+
+ return result;
+}
+
+static char joblist_filename[] = "joblist.txt";
+bool serialize_job_list(struct job_list *job_list, struct settings *settings)
+{
+ int dirfd, fd;
+ size_t i, k;
+ FILE *f;
+
+ if (!settings->results_path) {
+ fprintf(stderr, "No results-path set; this shouldn't happen\n");
+ return false;
+ }
+
+ if ((dirfd = open(settings->results_path, O_DIRECTORY | O_RDONLY)) < 0) {
+ mkdir(settings->results_path, 0777);
+ if ((dirfd = open(settings->results_path, O_DIRECTORY | O_RDONLY)) < 0) {
+ fprintf(stderr, "Creating results-path failed\n");
+ return false;
+ }
+ }
+
+ if (!settings->overwrite &&
+ faccessat(dirfd, joblist_filename, F_OK, 0) == 0) {
+ fprintf(stderr, "Job list file already exists and not overwriting\n");
+ close(dirfd);
+ return false;
+ }
+
+ if (settings->overwrite &&
+ unlinkat(dirfd, joblist_filename, 0) != 0 &&
+ errno != ENOENT) {
+ fprintf(stderr, "Error removing old job list\n");
+ close(dirfd);
+ return false;
+ }
+
+ if ((fd = openat(dirfd, joblist_filename, O_CREAT | O_EXCL | O_WRONLY, 0666)) < 0) {
+ fprintf(stderr, "Creating job list serialization file failed: %s\n", strerror(errno));
+ close(dirfd);
+ return false;
+ }
+
+ f = fdopen(fd, "w");
+ if (!f) {
+ close(fd);
+ close(dirfd);
+ return false;
+ }
+
+ for (i = 0; i < job_list->size; i++) {
+ struct job_list_entry *entry = &job_list->entries[i];
+ fputs(entry->binary, f);
+
+ if (entry->subtest_count) {
+ const char *delim = "";
+
+ fprintf(f, " ");
+
+ for (k = 0; k < entry->subtest_count; k++) {
+ fprintf(f, "%s%s", delim, entry->subtests[k]);
+ delim = ",";
+ }
+ }
+
+ fprintf(f, "\n");
+ }
+
+ if (settings->sync) {
+ fsync(fd);
+ fsync(dirfd);
+ }
+
+ fclose(f);
+ close(dirfd);
+ return true;
+}
+
+bool read_job_list(struct job_list *job_list, int dirfd)
+{
+ int fd;
+ FILE *f;
+ ssize_t read;
+ char *line = NULL;
+ size_t line_len = 0;
+
+ free_job_list(job_list);
+
+ if ((fd = openat(dirfd, joblist_filename, O_RDONLY)) < 0)
+ return false;
+
+ f = fdopen(fd, "r");
+ if (!f) {
+ close(fd);
+ return false;
+ }
+
+ while ((read = getline(&line, &line_len, f))) {
+ char *binary, *sublist, *comma;
+ char **subtests = NULL;
+ size_t num_subtests = 0, len;
+
+ if (read < 0) {
+ if (errno == EINTR)
+ continue;
+ else
+ break;
+ }
+
+ len = strlen(line);
+ if (len > 0 && line[len - 1] == '\n')
+ line[len - 1] = '\0';
+
+ sublist = strchr(line, ' ');
+ if (!sublist) {
+ add_job_list_entry(job_list, strdup(line), NULL, 0);
+ continue;
+ }
+
+ *sublist++ = '\0';
+ binary = strdup(line);
+
+ do {
+ comma = strchr(sublist, ',');
+ if (comma) {
+ *comma++ = '\0';
+ }
+
+ ++num_subtests;
+ subtests = realloc(subtests, num_subtests * sizeof(*subtests));
+ subtests[num_subtests - 1] = strdup(sublist);
+ sublist = comma;
+ } while (comma != NULL);
+
+ add_job_list_entry(job_list, binary, subtests, num_subtests);
+ }
+
+ free(line);
+ fclose(f);
+
+ return true;
+}
diff --git a/runner/job_list.h b/runner/job_list.h
new file mode 100644
index 00000000..c726ab09
--- /dev/null
+++ b/runner/job_list.h
@@ -0,0 +1,37 @@
+#ifndef RUNNER_JOB_LIST_H
+#define RUNNER_JOB_LIST_H
+
+#include <stdbool.h>
+
+#include "settings.h"
+
+struct job_list_entry {
+ char *binary;
+ char **subtests;
+ /*
+ * 0 = all, or test has no subtests.
+ *
+ * If the original job_list was to run all subtests of a
+ * binary and such a run was incomplete, resuming from the
+ * execution journal will fill the subtest array with already
+ * started subtests prepended with '!' so the test binary will
+ * not run them. subtest_count will still reflect the size of
+ * the above array.
+ */
+ size_t subtest_count;
+};
+
+struct job_list
+{
+ struct job_list_entry *entries;
+ size_t size;
+};
+
+void init_job_list(struct job_list *job_list);
+void free_job_list(struct job_list *job_list);
+bool create_job_list(struct job_list *job_list, struct settings *settings);
+
+bool serialize_job_list(struct job_list *job_list, struct settings *settings);
+bool read_job_list(struct job_list *job_list, int dirfd);
+
+#endif
diff --git a/runner/meson.build b/runner/meson.build
new file mode 100644
index 00000000..9dafb312
--- /dev/null
+++ b/runner/meson.build
@@ -0,0 +1,41 @@
+jsonc = dependency('json-c', required: _runner_required)
+
+runnerlib_sources = [ 'settings.c',
+ 'job_list.c',
+ 'executor.c',
+ 'resultgen.c',
+ ]
+
+runner_sources = [ 'runner.c' ]
+resume_sources = [ 'resume.c' ]
+results_sources = [ 'results.c' ]
+
+if _build_runner and jsonc.found()
+ subdir('testdata')
+
+ runnerlib = static_library('igt_runner', runnerlib_sources,
+ include_directories : inc,
+ dependencies : jsonc)
+
+ runner = executable('igt_runner', runner_sources,
+ link_with : runnerlib,
+ install : true,
+ install_dir : bindir,
+ dependencies : igt_deps)
+
+ resume = executable('igt_resume', resume_sources,
+ link_with : runnerlib,
+ install : true,
+ install_dir : bindir,
+ dependencies : igt_deps)
+
+ results = executable('igt_results', results_sources,
+ link_with : runnerlib,
+ install : true,
+ install_dir : bindir,
+ dependencies : igt_deps)
+
+ build_info += 'Build test runner: Yes'
+else
+ build_info += 'Build test runner: No'
+endif
diff --git a/runner/output_strings.h b/runner/output_strings.h
new file mode 100644
index 00000000..1e52c2ce
--- /dev/null
+++ b/runner/output_strings.h
@@ -0,0 +1,55 @@
+#ifndef RUNNER_OUTPUT_STRINGS_H
+#define RUNNER_OUTPUT_STRINGS_H
+
+/*
+ * Output when a subtest has begun. Is followed by the subtest name.
+ *
+ * Example:
+ * Starting subtest: subtestname
+ */
+static const char STARTING_SUBTEST[] = "Starting subtest: ";
+
+/*
+ * Output when a subtest has ended. Is followed by the subtest name
+ * and optionally its runtime.
+ *
+ * Examples:
+ * Subtest subtestname: SKIP
+ * Subtest subtestname: PASS (0.003s)
+ */
+static const char SUBTEST_RESULT[] = "Subtest ";
+
+/*
+ * Output in dmesg when a subtest has begin. Is followed by the subtest name.
+ *
+ * Example:
+ * [IGT] test-binary-name: starting subtest subtestname
+ */
+static const char STARTING_SUBTEST_DMESG[] = ": starting subtest ";
+
+/*
+ * Output when a test process is executed.
+ *
+ * Example:
+ * IGT-Version: 1.22-gde9af343 (x86_64) (Linux: 4.12.0-1-amd64 x86_64)
+ */
+static const char IGT_VERSIONSTRING[] = "IGT-Version: ";
+
+/*
+ * Output by the executor to mark the test's exit code.
+ *
+ * Example:
+ * exit:77 (0.003s)
+ */
+static const char EXECUTOR_EXIT[] = "exit:";
+
+/*
+ * Output by the executor to mark the test as timeouted, with an exit
+ * code.
+ *
+ * Example:
+ * timeout:-15 (360.000s)
+ */
+static const char EXECUTOR_TIMEOUT[] = "timeout:";
+
+#endif
diff --git a/runner/resultgen.c b/runner/resultgen.c
new file mode 100644
index 00000000..347b52a4
--- /dev/null
+++ b/runner/resultgen.c
@@ -0,0 +1,962 @@
+#include <ctype.h>
+#include <fcntl.h>
+#include <stdio.h>
+#include <string.h>
+#include <sys/mman.h>
+#include <sys/stat.h>
+#include <sys/types.h>
+#include <unistd.h>
+
+#include <json.h>
+
+#include "igt_core.h"
+#include "resultgen.h"
+#include "settings.h"
+#include "executor.h"
+#include "output_strings.h"
+
+struct subtests
+{
+ char **names;
+ size_t size;
+};
+
+/*
+ * A lot of string handling here operates on an mmapped buffer, and
+ * thus we can't assume null-terminated strings. Buffers will be
+ * passed around as pointer+size, or pointer+pointer-past-the-end, the
+ * mem*() family of functions is used instead of str*().
+ */
+
+static char *find_line_starting_with(char *haystack, const char *needle, char *end)
+{
+ while (haystack < end) {
+ char *line_end = memchr(haystack, '\n', end - haystack);
+
+ if (end - haystack < strlen(needle))
+ return NULL;
+ if (!memcmp(haystack, needle, strlen(needle)))
+ return haystack;
+ if (line_end == NULL)
+ return NULL;
+ haystack = line_end + 1;
+ }
+
+ return NULL;
+}
+
+static char *find_line_starting_with_either(char *haystack,
+ const char *needle1,
+ const char *needle2,
+ char *end)
+{
+ while (haystack < end) {
+ char *line_end = memchr(haystack, '\n', end - haystack);
+ size_t linelen = line_end != NULL ? line_end - haystack : end - haystack;
+ if ((linelen >= strlen(needle1) && !memcmp(haystack, needle1, strlen(needle1))) ||
+ (linelen >= strlen(needle2) && !memcmp(haystack, needle2, strlen(needle2))))
+ return haystack;
+
+ if (line_end == NULL)
+ return NULL;
+
+ haystack = line_end + 1;
+ }
+
+ return NULL;
+}
+
+static char *next_line(char *line, char *bufend)
+{
+ char *ret;
+
+ if (!line)
+ return NULL;
+
+ ret = memchr(line, '\n', bufend - line);
+ if (ret)
+ ret++;
+
+ if (ret < bufend)
+ return ret;
+ else
+ return NULL;
+}
+
+static char *find_line_after_last(char *begin,
+ const char *needle1,
+ const char *needle2,
+ char *end)
+{
+ char *one, *two;
+ char *current_pos = begin;
+ char *needle1_newline = malloc(strlen(needle1) + 2);
+ char *needle2_newline = malloc(strlen(needle2) + 2);
+
+ needle1_newline[0] = needle2_newline[0] = '\n';
+ strcpy(needle1_newline + 1, needle1);
+ strcpy(needle2_newline + 1, needle2);
+
+ while (true) {
+ one = memmem(current_pos, end - current_pos, needle1_newline, strlen(needle1_newline));
+ two = memmem(current_pos, end - current_pos, needle2_newline, strlen(needle2_newline));
+ if (one == NULL && two == NULL)
+ break;
+
+ if (one != NULL && current_pos < one)
+ current_pos = one;
+ if (two != NULL && current_pos < two)
+ current_pos = two;
+
+ one = next_line(current_pos, end);
+ if (one != NULL)
+ current_pos = one;
+ }
+ free(needle1_newline);
+ free(needle2_newline);
+
+ one = memchr(current_pos, '\n', end - current_pos);
+ if (one != NULL)
+ return ++one;
+
+ return current_pos;
+}
+
+static size_t count_lines(const char *buf, const char *bufend)
+{
+ size_t ret = 0;
+ while (buf < bufend && (buf = memchr(buf, '\n', bufend - buf)) != NULL) {
+ ret++;
+ buf++;
+ }
+
+ return ret;
+}
+
+static char *lowercase(const char *str)
+{
+ char *ret = malloc(strlen(str) + 1);
+ char *q = ret;
+
+ while (*str) {
+ if (isspace(*str))
+ break;
+
+ *q++ = tolower(*str++);
+ }
+ *q = '\0';
+
+ return ret;
+}
+
+static void append_line(char **buf, size_t *buflen, char *line)
+{
+ size_t linelen = strlen(line);
+
+ *buf = realloc(*buf, *buflen + linelen + 1);
+ strcpy(*buf + *buflen, line);
+ *buflen += linelen;
+}
+
+static void generate_piglit_name(const char *binary, const char *subtest,
+ char *namebuf, size_t namebuf_size)
+{
+ char *lc_binary = lowercase(binary);
+ char *lc_subtest = NULL;
+
+ if (!subtest) {
+ snprintf(namebuf, namebuf_size, "igt@%s", lc_binary);
+ free(lc_binary);
+ return;
+ }
+
+ lc_subtest = lowercase(subtest);
+
+ snprintf(namebuf, namebuf_size, "igt@%s@%s", lc_binary, lc_subtest);
+
+ free(lc_binary);
+ free(lc_subtest);
+}
+
+static const struct {
+ const char *output_str;
+ const char *result_str;
+} resultmap[] = {
+ { "SUCCESS", "pass" },
+ { "SKIP", "skip" },
+ { "FAIL", "fail" },
+ { "CRASH", "crash" },
+ { "TIMEOUT", "timeout" },
+};
+static void parse_result_string(char *resultstring, size_t len, const char **result, double *time)
+{
+ size_t i;
+ size_t wordlen = 0;
+
+ while (wordlen < len && !isspace(resultstring[wordlen])) {
+ wordlen++;
+ }
+
+ *result = NULL;
+ for (i = 0; i < (sizeof(resultmap) / sizeof(resultmap[0])); i++) {
+ if (!strncmp(resultstring, resultmap[i].output_str, wordlen)) {
+ *result = resultmap[i].result_str;
+ break;
+ }
+ }
+
+ /* If the result string is unknown, use incomplete */
+ if (!*result)
+ *result = "incomplete";
+
+ /*
+ * Check for subtest runtime after the result. The string is
+ * '(' followed by the runtime in seconds as floating point,
+ * followed by 's)'.
+ */
+ wordlen++;
+ if (wordlen < len && resultstring[wordlen] == '(') {
+ char *dup;
+
+ wordlen++;
+ dup = malloc(len - wordlen + 1);
+ memcpy(dup, resultstring + wordlen, len - wordlen);
+ dup[len - wordlen] = '\0';
+ *time = strtod(dup, NULL);
+
+ free(dup);
+ }
+}
+
+static void parse_subtest_result(char *subtest, const char **result, double *time, char *buf, char *bufend)
+{
+ char *line;
+ char *line_end;
+ char *resultstring;
+ size_t linelen;
+ size_t subtestlen = strlen(subtest);
+
+ *result = NULL;
+ *time = 0.0;
+
+ if (!buf) return;
+
+ /*
+ * The result line structure is:
+ *
+ * - The string "Subtest " (`SUBTEST_RESULT` from output_strings.h)
+ * - The subtest name
+ * - The characters ':' and ' '
+ * - Subtest result string
+ * - Optional:
+ * -- The characters ' ' and '('
+ * -- Subtest runtime in seconds as floating point
+ * -- The characters 's' and ')'
+ *
+ * Example:
+ * Subtest subtestname: PASS (0.003s)
+ */
+
+ line = find_line_starting_with(buf, SUBTEST_RESULT, bufend);
+ if (!line) {
+ *result = "incomplete";
+ return;
+ }
+
+ line_end = memchr(line, '\n', bufend - line);
+ linelen = line_end != NULL ? line_end - line : bufend - line;
+
+ if (strlen(SUBTEST_RESULT) + subtestlen + strlen(": ") > linelen ||
+ strncmp(line + strlen(SUBTEST_RESULT), subtest, subtestlen))
+ return parse_subtest_result(subtest, result, time, line + linelen, bufend);
+
+ resultstring = line + strlen(SUBTEST_RESULT) + subtestlen + strlen(": ");
+ parse_result_string(resultstring, linelen - (resultstring - line), result, time);
+}
+
+static struct json_object *get_or_create_json_object(struct json_object *base,
+ const char *key)
+{
+ struct json_object *ret;
+
+ if (json_object_object_get_ex(base, key, &ret))
+ return ret;
+
+ ret = json_object_new_object();
+ json_object_object_add(base, key, ret);
+
+ return ret;
+}
+
+static void set_result(struct json_object *obj, const char *result)
+{
+ json_object_object_add(obj, "result",
+ json_object_new_string(result));
+}
+
+static void add_runtime(struct json_object *obj, double time)
+{
+ double oldtime;
+ struct json_object *timeobj = get_or_create_json_object(obj, "time");
+ struct json_object *oldend;
+
+ json_object_object_add(timeobj, "__type__",
+ json_object_new_string("TimeAttribute"));
+ json_object_object_add(timeobj, "start",
+ json_object_new_double(0.0));
+
+ if (!json_object_object_get_ex(timeobj, "end", &oldend)) {
+ json_object_object_add(timeobj, "end",
+ json_object_new_double(time));
+ return;
+ }
+
+ /* Add the runtime to the existing runtime. */
+ oldtime = json_object_get_double(oldend);
+ time += oldtime;
+ json_object_object_add(timeobj, "end",
+ json_object_new_double(time));
+}
+
+static void set_runtime(struct json_object *obj, double time)
+{
+ struct json_object *timeobj = get_or_create_json_object(obj, "time");
+
+ json_object_object_add(timeobj, "__type__",
+ json_object_new_string("TimeAttribute"));
+ json_object_object_add(timeobj, "start",
+ json_object_new_double(0.0));
+ json_object_object_add(timeobj, "end",
+ json_object_new_double(time));
+}
+
+static bool fill_from_output(int fd, const char *binary, const char *key,
+ struct subtests *subtests,
+ struct json_object *tests)
+{
+ char *buf, *bufend;
+ struct stat statbuf;
+ char piglit_name[256];
+ char *igt_version = NULL;
+ size_t igt_version_len = 0;
+ struct json_object *current_test = NULL;
+ size_t i;
+
+ if (fstat(fd, &statbuf))
+ return false;
+
+ if (statbuf.st_size != 0) {
+ buf = mmap(NULL, statbuf.st_size, PROT_READ, MAP_SHARED, fd, 0);
+ if (buf == MAP_FAILED)
+ return false;
+ } else {
+ buf = NULL;
+ }
+
+ bufend = buf + statbuf.st_size;
+
+ igt_version = find_line_starting_with(buf, IGT_VERSIONSTRING, bufend);
+ if (igt_version) {
+ char *newline = memchr(igt_version, '\n', bufend - igt_version);
+ igt_version_len = newline - igt_version;
+ }
+
+ /* TODO: Refactor to helper functions */
+ if (subtests->size == 0) {
+ /* No subtests */
+ generate_piglit_name(binary, NULL, piglit_name, sizeof(piglit_name));
+ current_test = get_or_create_json_object(tests, piglit_name);
+
+ json_object_object_add(current_test, key,
+ json_object_new_string_len(buf, statbuf.st_size));
+ if (igt_version)
+ json_object_object_add(current_test, "igt-version",
+ json_object_new_string_len(igt_version,
+ igt_version_len));
+
+ return true;
+ }
+
+ for (i = 0; i < subtests->size; i++) {
+ char *this_sub_begin, *this_sub_result;
+ const char *resulttext;
+ char *beg, *end, *startline;
+ double time;
+ int begin_len;
+ int result_len;
+
+ generate_piglit_name(binary, subtests->names[i], piglit_name, sizeof(piglit_name));
+ current_test = get_or_create_json_object(tests, piglit_name);
+
+ begin_len = asprintf(&this_sub_begin, "%s%s\n", STARTING_SUBTEST, subtests->names[i]);
+ result_len = asprintf(&this_sub_result, "%s%s: ", SUBTEST_RESULT, subtests->names[i]);
+
+ if (begin_len < 0 || result_len < 0) {
+ fprintf(stderr, "Failure generating strings\n");
+ return false;
+ }
+
+ beg = find_line_starting_with(buf, this_sub_begin, bufend);
+ end = find_line_starting_with(buf, this_sub_result, bufend);
+ startline = beg;
+
+ free(this_sub_begin);
+ free(this_sub_result);
+
+ if (beg == NULL && end == NULL) {
+ /* No output at all */
+ beg = bufend;
+ end = bufend;
+ }
+
+ if (beg == NULL) {
+ /*
+ * Subtest didn't start, probably skipped from
+ * fixture already. Start from the result
+ * line, it gets adjusted below.
+ */
+ beg = end;
+ }
+
+ /* Include the output after the previous subtest output */
+ beg = find_line_after_last(buf,
+ STARTING_SUBTEST,
+ SUBTEST_RESULT,
+ beg);
+
+ if (end == NULL) {
+ /* Incomplete result. Find the next starting subtest or result. */
+ end = next_line(startline, bufend);
+ if (end != NULL) {
+ end = find_line_starting_with_either(end,
+ STARTING_SUBTEST,
+ SUBTEST_RESULT,
+ bufend);
+ }
+ if (end == NULL) {
+ end = bufend;
+ }
+ } else {
+ /*
+ * Now pointing to the line where this sub's
+ * result is. We need to include that of
+ * course.
+ */
+ char *nexttest = next_line(end, bufend);
+
+ /* Stretch onwards until the next subtest begins or ends */
+ if (nexttest != NULL) {
+ nexttest = find_line_starting_with_either(nexttest,
+ STARTING_SUBTEST,
+ SUBTEST_RESULT,
+ bufend);
+ }
+ if (nexttest != NULL) {
+ end = nexttest;
+ } else {
+ end = bufend;
+ }
+ }
+
+ json_object_object_add(current_test, key,
+ json_object_new_string_len(beg, end - beg));
+
+ if (igt_version) {
+ json_object_object_add(current_test, "igt-version",
+ json_object_new_string_len(igt_version,
+ igt_version_len));
+ }
+
+ if (!json_object_object_get_ex(current_test, "result", NULL)) {
+ parse_subtest_result(subtests->names[i], &resulttext, &time, beg, end);
+ set_result(current_test, resulttext);
+ set_runtime(current_test, time);
+ }
+ }
+
+ return true;
+}
+
+/*
+ * This regexp controls the kmsg handling. All kernel log records that
+ * have log level of warning or higher convert the result to
+ * dmesg-warn/dmesg-fail unless they match this regexp.
+ *
+ * TODO: Move this to external files, i915-suppressions.txt,
+ * general-suppressions.txt et al.
+ */
+
+#define _ "|"
+static const char igt_dmesg_whitelist[] =
+ "ACPI: button: The lid device is not compliant to SW_LID" _
+ "ACPI: .*: Unable to dock!" _
+ "IRQ [0-9]+: no longer affine to CPU[0-9]+" _
+ "IRQ fixup: irq [0-9]+ move in progress, old vector [0-9]+" _
+ /* i915 tests set module options, expected message */
+ "Setting dangerous option [a-z_]+ - tainting kernel" _
+ /* Raw printk() call, uses default log level (warn) */
+ "Suspending console\\(s\\) \\(use no_console_suspend to debug\\)" _
+ "atkbd serio[0-9]+: Failed to (deactivate|enable) keyboard on isa[0-9]+/serio[0-9]+" _
+ "cache: parent cpu[0-9]+ should not be sleeping" _
+ "hpet[0-9]+: lost [0-9]+ rtc interrupts" _
+ /* i915 selftests terminate normally with ENODEV from the
+ * module load after the testing finishes, which produces this
+ * message.
+ */
+ "i915: probe of [0-9:.]+ failed with error -25" _
+ /* swiotbl warns even when asked not to */
+ "mock: DMA: Out of SW-IOMMU space for [0-9]+ bytes" _
+ "usb usb[0-9]+: root hub lost power or was reset"
+ ;
+#undef _
+
+static regex_t re;
+
+static int init_regex_whitelist(void)
+{
+ static int status = -1;
+
+ if (status == -1) {
+ if (regcomp(&re, igt_dmesg_whitelist, REG_EXTENDED | REG_NOSUB) != 0) {
+ fprintf(stderr, "Cannot compile dmesg whitelist regexp\n");
+ status = 1;
+ return false;
+ }
+
+ status = 0;
+ }
+
+ return status;
+}
+
+static bool parse_dmesg_line(char* line,
+ unsigned *flags, unsigned long long *ts_usec,
+ char *continuation, char **message)
+{
+ unsigned long long seq;
+ int s;
+
+ s = sscanf(line, "%u,%llu,%llu,%c;", flags, &seq, ts_usec, continuation);
+ if (s != 4) {
+ /*
+ * Machine readable key/value pairs begin with
+ * a space. We ignore them.
+ */
+ if (line[0] != ' ') {
+ fprintf(stderr, "Cannot parse kmsg record: %s\n", line);
+ }
+ return false;
+ }
+
+ *message = strchr(line, ';');
+ if (!message) {
+ fprintf(stderr, "No ; found in kmsg record, this shouldn't happen\n");
+ return false;
+ }
+ (*message)++;
+
+ return true;
+}
+
+static void add_dmesg(struct json_object *obj,
+ const char *dmesg, size_t dmesglen,
+ const char *warnings, size_t warningslen)
+{
+ json_object_object_add(obj, "dmesg",
+ json_object_new_string_len(dmesg, dmesglen));
+
+ if (warnings) {
+ json_object_object_add(obj, "dmesg-warnings",
+ json_object_new_string_len(warnings, warningslen));
+ }
+}
+
+static void add_empty_dmesgs_where_missing(struct json_object *tests,
+ char *binary,
+ struct subtests *subtests)
+{
+ struct json_object *current_test;
+ char piglit_name[256];
+ size_t i;
+
+ for (i = 0; i < subtests->size; i++) {
+ generate_piglit_name(binary, subtests->names[i], piglit_name, sizeof(piglit_name));
+ current_test = get_or_create_json_object(tests, piglit_name);
+ if (!json_object_object_get_ex(current_test, "dmesg", NULL)) {
+ add_dmesg(current_test, "", 0, NULL, 0);
+ }
+ }
+
+}
+
+static bool fill_from_dmesg(int fd, char *binary,
+ struct subtests *subtests,
+ struct json_object *tests)
+{
+ char *line = NULL, *warnings = NULL, *dmesg = NULL;
+ size_t linelen = 0, warningslen = 0, dmesglen = 0;
+ struct json_object *current_test = NULL;
+ FILE *f = fdopen(fd, "r");
+ char piglit_name[256];
+ ssize_t read;
+ size_t i;
+
+ if (!f) {
+ return false;
+ }
+
+ if (init_regex_whitelist()) {
+ fclose(f);
+ return false;
+ }
+
+ while ((read = getline(&line, &linelen, f)) > 0) {
+ char *formatted;
+ unsigned flags;
+ unsigned long long ts_usec;
+ char continuation;
+ char *message, *subtest;
+
+ if (!parse_dmesg_line(line, &flags, &ts_usec, &continuation, &message))
+ continue;
+
+ asprintf(&formatted, "<%u> [%llu.%06llu] %s",
+ flags & 0x07, ts_usec / 1000000, ts_usec % 1000000, message);
+
+ if ((subtest = strstr(message, STARTING_SUBTEST_DMESG)) != NULL) {
+ if (current_test != NULL) {
+ /* Done with the previous subtest, file up */
+ add_dmesg(current_test, dmesg, dmesglen, warnings, warningslen);
+
+ free(dmesg);
+ free(warnings);
+ dmesg = warnings = NULL;
+ dmesglen = warningslen = 0;
+ }
+
+ subtest += strlen(STARTING_SUBTEST_DMESG);
+ generate_piglit_name(binary, subtest, piglit_name, sizeof(piglit_name));
+ current_test = get_or_create_json_object(tests, piglit_name);
+ }
+
+ if ((flags & 0x07) <= 4 && continuation != 'c' &&
+ regexec(&re, message, (size_t)0, NULL, 0) == REG_NOMATCH) {
+ append_line(&warnings, &warningslen, formatted);
+ }
+ append_line(&dmesg, &dmesglen, formatted);
+ free(formatted);
+ }
+
+ if (current_test != NULL) {
+ add_dmesg(current_test, dmesg, dmesglen, warnings, warningslen);
+ } else {
+ /*
+ * Didn't get any subtest messages at all. If there
+ * are subtests, add all of the dmesg gotten to all of
+ * them.
+ */
+ for (i = 0; i < subtests->size; i++) {
+ generate_piglit_name(binary, subtests->names[i], piglit_name, sizeof(piglit_name));
+ current_test = get_or_create_json_object(tests, piglit_name);
+ /*
+ * Don't bother with warnings, any subtests
+ * there are would have skip as their result
+ * anyway.
+ */
+ add_dmesg(current_test, dmesg, dmesglen, NULL, 0);
+ }
+
+ if (subtests->size == 0) {
+ generate_piglit_name(binary, NULL, piglit_name, sizeof(piglit_name));
+ current_test = get_or_create_json_object(tests, piglit_name);
+ add_dmesg(current_test, dmesg, dmesglen, warnings, warningslen);
+ }
+ }
+
+ add_empty_dmesgs_where_missing(tests, binary, subtests);
+
+ free(dmesg);
+ free(warnings);
+ fclose(f);
+ return true;
+}
+
+static const char *result_from_exitcode(int exitcode)
+{
+ switch (exitcode) {
+ case IGT_EXIT_TIMEOUT:
+ return "timeout";
+ case IGT_EXIT_SKIP:
+ return "skip";
+ case IGT_EXIT_SUCCESS:
+ return "pass";
+ case IGT_EXIT_INVALID:
+ return "notrun";
+ default:
+ return "fail";
+ }
+}
+
+static void add_subtest(struct subtests *subtests, char *subtest)
+{
+ size_t len = strlen(subtest);
+
+ if (len == 0)
+ return;
+
+ if (subtest[len - 1] == '\n')
+ subtest[len - 1] = '\0';
+
+ subtests->size++;
+ subtests->names = realloc(subtests->names, sizeof(*subtests->names) * subtests->size);
+ subtests->names[subtests->size - 1] = subtest;
+}
+
+static void fill_from_journal(int fd, char *binary,
+ struct subtests *subtests,
+ struct json_object *tests)
+{
+ FILE *f = fdopen(fd, "r");
+ char *line = NULL;
+ size_t linelen = 0;
+ ssize_t read;
+ char exitline[] = "exit:";
+ char timeoutline[] = "timeout:";
+ int exitcode = -1;
+ bool has_timeout = false;
+
+ while ((read = getline(&line, &linelen, f)) > 0) {
+ if (read >= strlen(exitline) && !memcmp(line, exitline, strlen(exitline))) {
+ char *p = strchr(line, '(');
+ char piglit_name[256];
+ double time = 0.0;
+ struct json_object *obj;
+
+ generate_piglit_name(binary, NULL, piglit_name, sizeof(piglit_name));
+ obj = get_or_create_json_object(tests, piglit_name);
+
+ exitcode = atoi(line + strlen(exitline));
+
+ if (p)
+ time = strtod(p + 1, NULL);
+
+ add_runtime(obj, time);
+ } else if (read >= strlen(timeoutline) && !memcmp(line, timeoutline, strlen(timeoutline))) {
+ has_timeout = true;
+
+ if (subtests->size) {
+ /* Assign the timeout to the previously appeared subtest */
+ char *last_subtest = subtests->names[subtests->size - 1];
+ char piglit_name[256];
+ char *p = strchr(line, '(');
+ double time = 0.0;
+ struct json_object *obj;
+
+ generate_piglit_name(binary, last_subtest, piglit_name, sizeof(piglit_name));
+ obj = get_or_create_json_object(tests, piglit_name);
+
+ set_result(obj, "timeout");
+
+ if (p)
+ time = strtod(p + 1, NULL);
+
+ /* Add runtime for the subtest... */
+ add_runtime(obj, time);
+
+ /* ... and also for the binary */
+ generate_piglit_name(binary, NULL, piglit_name, sizeof(piglit_name));
+ obj = get_or_create_json_object(tests, piglit_name);
+ add_runtime(obj, time);
+ }
+ } else {
+ add_subtest(subtests, strdup(line));
+ }
+ }
+
+ if (subtests->size == 0) {
+ char piglit_name[256];
+ struct json_object *obj;
+ const char *result = has_timeout ? "timeout" : result_from_exitcode(exitcode);
+
+ generate_piglit_name(binary, NULL, piglit_name, sizeof(piglit_name));
+ obj = get_or_create_json_object(tests, piglit_name);
+ set_result(obj, result);
+ }
+
+ free(line);
+}
+
+static void override_result_single(struct json_object *obj)
+{
+ const char *errtext = NULL, *result = NULL;
+ struct json_object *textobj;
+ bool dmesgwarns = false;
+
+ if (json_object_object_get_ex(obj, "err", &textobj))
+ errtext = json_object_get_string(textobj);
+ if (json_object_object_get_ex(obj, "result", &textobj))
+ result = json_object_get_string(textobj);
+ if (json_object_object_get_ex(obj, "dmesg-warnings", &textobj))
+ dmesgwarns = true;
+
+ if (!strcmp(result, "pass") &&
+ count_lines(errtext, errtext + strlen(errtext)) > 2) {
+ set_result(obj, "warn");
+ }
+
+ if (dmesgwarns) {
+ if (!strcmp(result, "pass") || !strcmp(result, "warn")) {
+ set_result(obj, "dmesg-warn");
+ } else if (!strcmp(result, "fail")) {
+ set_result(obj, "dmesg-fail");
+ }
+ }
+}
+
+static void override_results(char *binary,
+ struct subtests *subtests,
+ struct json_object *tests)
+{
+ struct json_object *obj;
+ char piglit_name[256];
+ size_t i;
+
+ if (subtests->size == 0) {
+ generate_piglit_name(binary, NULL, piglit_name, sizeof(piglit_name));
+ obj = get_or_create_json_object(tests, piglit_name);
+ override_result_single(obj);
+ return;
+ }
+
+ for (i = 0; i < subtests->size; i++) {
+ generate_piglit_name(binary, subtests->names[i], piglit_name, sizeof(piglit_name));
+ obj = get_or_create_json_object(tests, piglit_name);
+ override_result_single(obj);
+ }
+}
+
+static bool parse_test_directory(int dirfd, char *binary, struct json_object *tests)
+{
+ int fds[_F_LAST];
+ struct subtests subtests = {};
+
+ if (!open_output_files(dirfd, fds, false)) {
+ fprintf(stderr, "Error opening output files\n");
+ return false;
+ }
+
+ /*
+ * fill_from_journal fills the subtests struct and adds
+ * timeout results where applicable.
+ */
+ fill_from_journal(fds[_F_JOURNAL], binary, &subtests, tests);
+
+ if (!fill_from_output(fds[_F_OUT], binary, "out", &subtests, tests) ||
+ !fill_from_output(fds[_F_ERR], binary, "err", &subtests, tests) ||
+ !fill_from_dmesg(fds[_F_DMESG], binary, &subtests, tests)) {
+ fprintf(stderr, "Error parsing output files\n");
+ return false;
+ }
+
+ override_results(binary, &subtests, tests);
+
+ close_outputs(fds);
+
+ return true;
+}
+
+bool generate_results(int dirfd)
+{
+ struct settings settings;
+ struct job_list job_list;
+ struct json_object *obj, *tests;
+ int resultsfd, testdirfd, unamefd;
+ const char *json_string;
+ size_t i;
+
+ init_settings(&settings);
+ init_job_list(&job_list);
+
+ if (!read_settings(&settings, dirfd)) {
+ fprintf(stderr, "resultgen: Cannot parse settings\n");
+ return false;
+ }
+
+ if (!read_job_list(&job_list, dirfd)) {
+ fprintf(stderr, "resultgen: Cannot parse job list\n");
+ return false;
+ }
+
+ /* TODO: settings.overwrite */
+ if ((resultsfd = openat(dirfd, "results.json", O_WRONLY | O_CREAT | O_TRUNC, 0666)) < 0) {
+ fprintf(stderr, "resultgen: Cannot create results file\n");
+ return false;
+ }
+
+ obj = json_object_new_object();
+ json_object_object_add(obj, "__type__", json_object_new_string("TestrunResult"));
+ json_object_object_add(obj, "results_version", json_object_new_int(9));
+ json_object_object_add(obj, "name",
+ settings.name ?
+ json_object_new_string(settings.name) :
+ json_object_new_string(""));
+
+ if ((unamefd = openat(dirfd, "uname.txt", O_RDONLY)) >= 0) {
+ char buf[128];
+ ssize_t r = read(unamefd, buf, 128);
+
+ if (r > 0 && buf[r - 1] == '\n')
+ r--;
+
+ json_object_object_add(obj, "uname",
+ json_object_new_string_len(buf, r));
+ close(unamefd);
+ }
+
+ /*
+ * Result fields that won't be added:
+ *
+ * - glxinfo
+ * - wglinfo
+ * - clinfo
+ *
+ * Result fields that are TODO:
+ *
+ * - lspci
+ * - options
+ * - time_elapsed
+ * - totals
+ */
+
+ tests = json_object_new_object();
+ json_object_object_add(obj, "tests", tests);
+
+ for (i = 0; i < job_list.size; i++) {
+ char name[16];
+
+ snprintf(name, 16, "%zd", i);
+ if ((testdirfd = openat(dirfd, name, O_DIRECTORY | O_RDONLY)) < 0) {
+ fprintf(stderr, "Warning: Cannot open result directory %s\n", name);
+ break;
+ }
+
+ if (!parse_test_directory(testdirfd, job_list.entries[i].binary, tests)) {
+ close(resultsfd);
+ return false;
+ }
+ }
+
+ json_string = json_object_to_json_string_ext(obj, JSON_C_TO_STRING_PRETTY);
+ write(resultsfd, json_string, strlen(json_string));
+ return true;
+}
+
+bool generate_results_path(char *resultspath)
+{
+ int dirfd = open(resultspath, O_DIRECTORY | O_RDONLY);
+
+ if (dirfd < 0)
+ return false;
+
+ return generate_results(dirfd);
+}
diff --git a/runner/resultgen.h b/runner/resultgen.h
new file mode 100644
index 00000000..83a0876b
--- /dev/null
+++ b/runner/resultgen.h
@@ -0,0 +1,9 @@
+#ifndef RUNNER_RESULTGEN_H
+#define RUNNER_RESULTGEN_H
+
+#include <stdbool.h>
+
+bool generate_results(int dirfd);
+bool generate_results_path(char *resultspath);
+
+#endif
diff --git a/runner/results.c b/runner/results.c
new file mode 100644
index 00000000..3eb7cb15
--- /dev/null
+++ b/runner/results.c
@@ -0,0 +1,26 @@
+#include <fcntl.h>
+#include <stdio.h>
+#include <stdlib.h>
+#include <sys/stat.h>
+#include <sys/types.h>
+
+#include "resultgen.h"
+
+int main(int argc, char **argv)
+{
+ int dirfd;
+
+ if (argc < 2)
+ exit(1);
+
+ dirfd = open(argv[1], O_DIRECTORY | O_RDONLY);
+ if (dirfd < 0)
+ exit(1);
+
+ if (generate_results(dirfd)) {
+ printf("Results generated\n");
+ exit(0);
+ }
+
+ exit(1);
+}
diff --git a/runner/resume.c b/runner/resume.c
new file mode 100644
index 00000000..b3a2a71e
--- /dev/null
+++ b/runner/resume.c
@@ -0,0 +1,47 @@
+#include <errno.h>
+#include <fcntl.h>
+#include <stdio.h>
+#include <string.h>
+#include <sys/stat.h>
+#include <sys/types.h>
+
+#include "settings.h"
+#include "job_list.h"
+#include "executor.h"
+#include "resultgen.h"
+
+int main(int argc, char **argv)
+{
+ struct settings settings;
+ struct job_list job_list;
+ struct execute_state state;
+ int dirfd;
+
+ init_settings(&settings);
+ init_job_list(&job_list);
+
+ if (argc < 2) {
+ fprintf(stderr, "Usage: %s results-directory\n", argv[0]);
+ return 1;
+ }
+
+ if ((dirfd = open(argv[1], O_RDONLY | O_DIRECTORY)) < 0) {
+ fprintf(stderr, "Failure opening %s: %s\n", argv[1], strerror(errno));
+ return 1;
+ }
+
+ if (!initialize_execute_state_from_resume(dirfd, &state, &settings, &job_list)) {
+ return 1;
+ }
+
+ if (!execute(&state, &settings, &job_list)) {
+ return 1;
+ }
+
+ if (!generate_results_path(settings.results_path)) {
+ return 1;
+ }
+
+ printf("Done.\n");
+ return 0;
+}
diff --git a/runner/runner.c b/runner/runner.c
new file mode 100644
index 00000000..b685786a
--- /dev/null
+++ b/runner/runner.c
@@ -0,0 +1,40 @@
+#include <stdio.h>
+#include <string.h>
+
+#include "settings.h"
+#include "job_list.h"
+#include "executor.h"
+#include "resultgen.h"
+
+int main(int argc, char **argv)
+{
+ struct settings settings;
+ struct job_list job_list;
+ struct execute_state state;
+
+ init_settings(&settings);
+ init_job_list(&job_list);
+
+ if (!parse_options(argc, argv, &settings)) {
+ return 1;
+ }
+
+ if (!create_job_list(&job_list, &settings)) {
+ return 1;
+ }
+
+ if (!initialize_execute_state(&state, &settings, &job_list)) {
+ return 1;
+ }
+
+ if (!execute(&state, &settings, &job_list)) {
+ return 1;
+ }
+
+ if (!generate_results_path(settings.results_path)) {
+ return 1;
+ }
+
+ printf("Done.\n");
+ return 0;
+}
diff --git a/runner/settings.c b/runner/settings.c
new file mode 100644
index 00000000..31754a12
--- /dev/null
+++ b/runner/settings.c
@@ -0,0 +1,502 @@
+#include "settings.h"
+
+#include <errno.h>
+#include <fcntl.h>
+#include <getopt.h>
+#include <libgen.h>
+#include <limits.h>
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+#include <sys/types.h>
+#include <sys/stat.h>
+#include <unistd.h>
+
+enum {
+ OPT_ABORT_ON_ERROR,
+ OPT_TEST_LIST,
+ OPT_IGNORE_MISSING,
+ OPT_HELP = 'h',
+ OPT_NAME = 'n',
+ OPT_DRY_RUN = 'd',
+ OPT_INCLUDE = 't',
+ OPT_EXCLUDE = 'x',
+ OPT_SYNC = 's',
+ OPT_LOG_LEVEL = 'l',
+ OPT_OVERWRITE = 'o',
+ OPT_MULTIPLE = 'm',
+ OPT_TIMEOUT = 'c',
+ OPT_WATCHDOG = 'g',
+};
+
+static struct {
+ int level;
+ const char *name;
+} log_levels[] = {
+ { LOG_LEVEL_NORMAL, "normal" },
+ { LOG_LEVEL_QUIET, "quiet" },
+ { LOG_LEVEL_VERBOSE, "verbose" },
+ { 0, 0 },
+};
+
+static bool set_log_level(struct settings* settings, const char *level)
+{
+ typeof(*log_levels) *it;
+
+ for (it = log_levels; it->name; it++) {
+ if (!strcmp(level, it->name)) {
+ settings->log_level = it->level;
+ return true;
+ }
+ }
+
+ return false;
+}
+
+static const char *usage_str =
+ "usage: runner [options] [test_root] results-path\n\n"
+ "Options:\n"
+ " Piglit compatible:\n"
+ " -h, --help Show this help message and exit\n"
+ " -n <test name>, --name <test name>\n"
+ " Name of this test run\n"
+ " -d, --dry-run Do not execute the tests\n"
+ " -t <regex>, --include-tests <regex>\n"
+ " Run only matching tests (can be used more than once)\n"
+ " -x <regex>, --exclude-tests <regex>\n"
+ " Exclude matching tests (can be used more than once)\n"
+ " --abort-on-monitored-error\n"
+ " Abort execution when a fatal condition is detected.\n"
+ " <TODO>\n"
+ " -s, --sync Sync results to disk after every test\n"
+ " -l {quiet,verbose,dummy}, --log-level {quiet,verbose,dummy}\n"
+ " Set the logger verbosity level\n"
+ " --test-list TEST_LIST\n"
+ " A file containing a list of tests to run\n"
+ " -o, --overwrite If the results-path already exists, delete it\n"
+ " --ignore-missing Ignored but accepted, for piglit compatibility\n"
+ "\n"
+ " Incompatible options:\n"
+ " -m, --multiple-mode Run multiple subtests in the same binary execution.\n"
+ " If a testlist file is given, consecutive subtests are\n"
+ " run in the same execution if they are from the same\n"
+ " binary. Note that in that case relative ordering of the\n"
+ " subtest execution is dictated by the test binary, not\n"
+ " the testlist\n"
+ " --inactivity-timeout <seconds>\n"
+ " Kill the running test after <seconds> of inactivity in\n"
+ " the test's stdout, stderr, or dmesg\n"
+ " --use-watchdog Use hardware watchdog for lethal enforcement of the\n"
+ " above timeout. Killing the test process is still\n"
+ " attempted at timeout trigger.\n"
+ " [test_root] Directory that contains the IGT tests. The environment\n"
+ " variable IGT_TEST_ROOT will be used if set, overriding\n"
+ " this option if given.\n"
+ ;
+
+static void usage(const char *extra_message, FILE *f)
+{
+ if (extra_message)
+ fprintf(f, "%s\n\n", extra_message);
+
+ fputs(usage_str, f);
+}
+
+static bool add_regex(struct regex_list *list, char *new)
+{
+ regex_t *regex;
+ size_t buflen;
+ char *buf;
+ int s;
+
+ regex = malloc(sizeof(*regex));
+
+ if ((s = regcomp(regex, new,
+ REG_EXTENDED | REG_NOSUB)) != 0) {
+ buflen = regerror(s, regex, NULL, 0);
+ buf = malloc(buflen);
+ regerror(s, regex, buf, buflen);
+ usage(buf, stderr);
+
+ free(buf);
+ regfree(regex);
+ free(regex);
+ return false;
+ }
+
+ list->regexes = realloc(list->regexes,
+ (list->size + 1) * sizeof(*list->regexes));
+ list->regex_strings = realloc(list->regex_strings,
+ (list->size + 1) * sizeof(*list->regex_strings));
+ list->regexes[list->size] = regex;
+ list->regex_strings[list->size] = new;
+ list->size++;
+
+ return true;
+}
+
+static void free_regexes(struct regex_list *regexes)
+{
+ size_t i;
+
+ for (i = 0; i < regexes->size; i++) {
+ free(regexes->regex_strings[i]);
+ regfree(regexes->regexes[i]);
+ free(regexes->regexes[i]);
+ }
+ free(regexes->regex_strings);
+ free(regexes->regexes);
+}
+
+static bool readable_file(char *filename)
+{
+ return !access(filename, R_OK);
+}
+
+void init_settings(struct settings *settings)
+{
+ memset(settings, 0, sizeof(*settings));
+}
+
+void free_settings(struct settings *settings)
+{
+ free(settings->test_list);
+ free(settings->name);
+ free(settings->test_root);
+ free(settings->results_path);
+
+ free_regexes(&settings->include_regexes);
+ free_regexes(&settings->exclude_regexes);
+
+ init_settings(settings);
+}
+
+bool parse_options(int argc, char **argv,
+ struct settings *settings)
+{
+ int c;
+ char *env_test_root;
+
+ static struct option long_options[] = {
+ {"help", no_argument, NULL, OPT_HELP},
+ {"name", required_argument, NULL, OPT_NAME},
+ {"dry-run", no_argument, NULL, OPT_DRY_RUN},
+ {"include-tests", required_argument, NULL, OPT_INCLUDE},
+ {"exclude-tests", required_argument, NULL, OPT_EXCLUDE},
+ {"abort-on-monitored-error", no_argument, NULL, OPT_ABORT_ON_ERROR},
+ {"sync", no_argument, NULL, OPT_SYNC},
+ {"log-level", required_argument, NULL, OPT_LOG_LEVEL},
+ {"test-list", required_argument, NULL, OPT_TEST_LIST},
+ {"overwrite", no_argument, NULL, OPT_OVERWRITE},
+ {"ignore-missing", no_argument, NULL, OPT_IGNORE_MISSING},
+ {"multiple-mode", no_argument, NULL, OPT_MULTIPLE},
+ {"inactivity-timeout", required_argument, NULL, OPT_TIMEOUT},
+ {"use-watchdog", no_argument, NULL, OPT_WATCHDOG},
+ { 0, 0, 0, 0},
+ };
+
+ free_settings(settings);
+
+ optind = 1;
+
+ while ((c = getopt_long(argc, argv, "hn:dt:x:sl:om", long_options, NULL)) != -1) {
+ switch (c) {
+ case OPT_HELP:
+ usage(NULL, stdout);
+ goto error;
+ case OPT_NAME:
+ settings->name = strdup(optarg);
+ break;
+ case OPT_DRY_RUN:
+ settings->dry_run = true;
+ break;
+ case OPT_INCLUDE:
+ if (!add_regex(&settings->include_regexes, strdup(optarg)))
+ goto error;
+ break;
+ case OPT_EXCLUDE:
+ if (!add_regex(&settings->exclude_regexes, strdup(optarg)))
+ goto error;
+ break;
+ case OPT_ABORT_ON_ERROR:
+ settings->abort_on_error = true;
+ break;
+ case OPT_SYNC:
+ settings->sync = true;
+ break;
+ case OPT_LOG_LEVEL:
+ if (!set_log_level(settings, optarg)) {
+ usage("Cannot parse log level", stderr);
+ goto error;
+ }
+ break;
+ case OPT_TEST_LIST:
+ settings->test_list = absolute_path(optarg);
+ break;
+ case OPT_OVERWRITE:
+ settings->overwrite = true;
+ break;
+ case OPT_IGNORE_MISSING:
+ /* Ignored, piglit compatibility */
+ break;
+ case OPT_MULTIPLE:
+ settings->multiple_mode = true;
+ break;
+ case OPT_TIMEOUT:
+ settings->inactivity_timeout = atoi(optarg);
+ break;
+ case OPT_WATCHDOG:
+ settings->use_watchdog = true;
+ break;
+ case '?':
+ usage(NULL, stderr);
+ goto error;
+ default:
+ usage("Cannot parse options", stderr);
+ goto error;
+ }
+ }
+
+ switch (argc - optind) {
+ case 2:
+ settings->test_root = absolute_path(argv[optind]);
+ ++optind;
+ /* fallthrough */
+ case 1:
+ settings->results_path = absolute_path(argv[optind]);
+ break;
+ case 0:
+ usage("Results-path missing", stderr);
+ goto error;
+ default:
+ usage("Extra arguments after results-path", stderr);
+ goto error;
+ }
+
+ if ((env_test_root = getenv("IGT_TEST_ROOT")) != NULL) {
+ free(settings->test_root);
+ settings->test_root = absolute_path(env_test_root);
+ }
+
+ if (!settings->test_root) {
+ usage("Test root not set", stderr);
+ goto error;
+ }
+
+ if (!settings->name) {
+ char *name = strdup(settings->results_path);
+ settings->name = strdup(basename(name));
+ free(name);
+ }
+
+ return true;
+
+ error:
+ free_settings(settings);
+ return false;
+}
+
+bool validate_settings(struct settings *settings)
+{
+ int dirfd, fd;
+
+ if (settings->test_list && !readable_file(settings->test_list)) {
+ usage("Cannot open test-list file", stderr);
+ return false;
+ }
+
+ if (!settings->results_path) {
+ usage("No results-path set; this shouldn't happen", stderr);
+ return false;
+ }
+
+ if (!settings->test_root) {
+ usage("No test root set; this shouldn't happen", stderr);
+ return false;
+ }
+
+ dirfd = open(settings->test_root, O_DIRECTORY | O_RDONLY);
+ if (dirfd < 0) {
+ fprintf(stderr, "Test directory %s cannot be opened\n", settings->test_root);
+ return false;
+ }
+
+ fd = openat(dirfd, "test-list.txt", O_RDONLY);
+ if (fd < 0) {
+ fprintf(stderr, "Cannot open %s/test-list.txt\n", settings->test_root);
+ close(dirfd);
+ return false;
+ }
+
+ close(fd);
+ close(dirfd);
+
+ return true;
+}
+
+char *absolute_path(char *path)
+{
+ char *result = NULL;
+ char *tmppath, *tmpname;
+
+ result = realpath(path, NULL);
+ if (result != NULL)
+ return result;
+
+ tmppath = strdup(path);
+ tmpname = dirname(tmppath);
+ free(result);
+ result = realpath(tmpname, NULL);
+ free(tmppath);
+
+ if (result != NULL) {
+ char *ret;
+
+ tmppath = strdup(path);
+ tmpname = basename(tmppath);
+
+ asprintf(&ret, "%s/%s", result, tmpname);
+ free(result);
+ free(tmppath);
+ return ret;
+ }
+
+ free(result);
+ return NULL;
+}
+
+static char settings_filename[] = "metadata.txt";
+bool serialize_settings(struct settings *settings)
+{
+#define SERIALIZE_LINE(f, s, name, format) fprintf(f, "%s : " format "\n", #name, s->name)
+
+ int dirfd, fd;
+ FILE *f;
+
+ if (!settings->results_path) {
+ usage("No results-path set; this shouldn't happen", stderr);
+ return false;
+ }
+
+ if ((dirfd = open(settings->results_path, O_DIRECTORY | O_RDONLY)) < 0) {
+ mkdir(settings->results_path, 0755);
+ if ((dirfd = open(settings->results_path, O_DIRECTORY | O_RDONLY)) < 0) {
+ usage("Creating results-path failed", stderr);
+ return false;
+ }
+ }
+
+ if (!settings->overwrite &&
+ faccessat(dirfd, settings_filename, F_OK, 0) == 0) {
+ usage("Settings metadata already exists and not overwriting", stderr);
+ return false;
+ }
+
+ if (settings->overwrite &&
+ unlinkat(dirfd, settings_filename, 0) != 0 &&
+ errno != ENOENT) {
+ usage("Error removing old settings metadata", stderr);
+ return false;
+ }
+
+ if ((fd = openat(dirfd, settings_filename, O_CREAT | O_EXCL | O_WRONLY, 0666)) < 0) {
+ char *msg;
+
+ asprintf(&msg, "Creating settings serialization file failed: %s", strerror(errno));
+ usage(msg, stderr);
+
+ free(msg);
+ close(dirfd);
+ return false;
+ }
+
+ f = fdopen(fd, "w");
+ if (!f) {
+ close(fd);
+ close(dirfd);
+ return false;
+ }
+
+ SERIALIZE_LINE(f, settings, abort_on_error, "%d");
+ if (settings->test_list)
+ SERIALIZE_LINE(f, settings, test_list, "%s");
+ if (settings->name)
+ SERIALIZE_LINE(f, settings, name, "%s");
+ SERIALIZE_LINE(f, settings, dry_run, "%d");
+ SERIALIZE_LINE(f, settings, sync, "%d");
+ SERIALIZE_LINE(f, settings, log_level, "%d");
+ SERIALIZE_LINE(f, settings, overwrite, "%d");
+ SERIALIZE_LINE(f, settings, multiple_mode, "%d");
+ SERIALIZE_LINE(f, settings, inactivity_timeout, "%d");
+ SERIALIZE_LINE(f, settings, use_watchdog, "%d");
+ SERIALIZE_LINE(f, settings, test_root, "%s");
+ SERIALIZE_LINE(f, settings, results_path, "%s");
+
+ if (settings->sync) {
+ fsync(fd);
+ fsync(dirfd);
+ }
+
+ fclose(f);
+ close(dirfd);
+ return true;
+
+#undef SERIALIZE_LINE
+}
+
+bool read_settings(struct settings *settings, int dirfd)
+{
+#define PARSE_LINE(s, name, val, field, write) \
+ if (!strcmp(name, #field)) { \
+ s->field = write; \
+ free(name); \
+ free(val); \
+ name = val = NULL; \
+ continue; \
+ }
+
+ int fd;
+ FILE *f;
+ char *name = NULL, *val = NULL;
+
+ free_settings(settings);
+
+ if ((fd = openat(dirfd, settings_filename, O_RDONLY)) < 0)
+ return false;
+
+ f = fdopen(fd, "r");
+ if (!f) {
+ close(fd);
+ return false;
+ }
+
+ while (fscanf(f, "%ms : %ms", &name, &val) == 2) {
+ int numval = atoi(val);
+ PARSE_LINE(settings, name, val, abort_on_error, numval);
+ PARSE_LINE(settings, name, val, test_list, val ? strdup(val) : NULL);
+ PARSE_LINE(settings, name, val, name, val ? strdup(val) : NULL);
+ PARSE_LINE(settings, name, val, dry_run, numval);
+ PARSE_LINE(settings, name, val, sync, numval);
+ PARSE_LINE(settings, name, val, log_level, numval);
+ PARSE_LINE(settings, name, val, overwrite, numval);
+ PARSE_LINE(settings, name, val, multiple_mode, numval);
+ PARSE_LINE(settings, name, val, inactivity_timeout, numval);
+ PARSE_LINE(settings, name, val, use_watchdog, numval);
+ PARSE_LINE(settings, name, val, test_root, val ? strdup(val) : NULL);
+ PARSE_LINE(settings, name, val, results_path, val ? strdup(val) : NULL);
+
+ printf("Warning: Unknown field in settings file: %s = %s\n",
+ name, val);
+ free(name);
+ free(val);
+ name = val = NULL;
+ }
+
+ free(name);
+ free(val);
+ fclose(f);
+
+ return true;
+
+#undef PARSE_LINE
+}
diff --git a/runner/settings.h b/runner/settings.h
new file mode 100644
index 00000000..9d1f03fb
--- /dev/null
+++ b/runner/settings.h
@@ -0,0 +1,111 @@
+#ifndef RUNNER_SETTINGS_H
+#define RUNNER_SETTINGS_H
+
+#include <stdbool.h>
+#include <stddef.h>
+#include <sys/types.h>
+#include <regex.h>
+
+enum {
+ LOG_LEVEL_NORMAL = 0,
+ LOG_LEVEL_QUIET = -1,
+ LOG_LEVEL_VERBOSE = 1,
+};
+
+struct regex_list {
+ char **regex_strings;
+ regex_t** regexes;
+ size_t size;
+};
+
+struct settings {
+ bool abort_on_error;
+ char *test_list;
+ char *name;
+ bool dry_run;
+ struct regex_list include_regexes;
+ struct regex_list exclude_regexes;
+ bool sync;
+ int log_level;
+ bool overwrite;
+ bool multiple_mode;
+ int inactivity_timeout;
+ bool use_watchdog;
+ char *test_root;
+ char *results_path;
+};
+
+/**
+ * init_settings:
+ *
+ * Initializes a settings object to an empty state (all values NULL, 0
+ * or false).
+ *
+ * @settings: Object to initialize. Storage for it must exist.
+ */
+void init_settings(struct settings *settings);
+
+/**
+ * free_settings:
+ *
+ * Releases all allocated resources for a settings object and
+ * initializes it to an empty state (see #init_settings).
+ *
+ * @settings: Object to release and initialize.
+ */
+void free_settings(struct settings *settings);
+
+/**
+ * parse_options:
+ *
+ * Parses command line options and sets the settings object to
+ * designated values.
+ *
+ * The function can be called again on the same settings object. The
+ * old values will be properly released and cleared. On a parse
+ * failure, the settings object will be in an empty state (see
+ * #init_settings) and usage instructions will be printed with an
+ * error message.
+ *
+ * @argc: Argument count
+ * @argv: Argument array. First element is the program name.
+ * @settings: Settings object to fill with values. Must have proper
+ * storage.
+ *
+ * Returns: True on successful parse, false on error.
+ */
+bool parse_options(int argc, char **argv,
+ struct settings *settings);
+
+/**
+ * validate_settings:
+ *
+ * Checks the settings object against the system to see if executing
+ * on it can be done. Checks pathnames for existence and access
+ * rights. Note that this function will not check that the designated
+ * job listing (through a test-list file or the -t/-x flags) yields a
+ * non-zero amount of testing to be done. On errors, usage
+ * instructions will be printed with an error message.
+ *
+ * @settings: Settings object to check.
+ *
+ * Returns: True on valid settings, false on any error.
+ */
+bool validate_settings(struct settings *settings);
+
+/* TODO: Better place for this */
+char *absolute_path(char *path);
+
+/**
+ * serialize_settings:
+ *
+ * Serializes the settings object to a file in the results_path
+ * directory.
+ *
+ * @settings: Settings object to serialize.
+ */
+bool serialize_settings(struct settings *settings);
+
+bool read_settings(struct settings *settings, int dirfd);
+
+#endif
--
2.14.1
_______________________________________________
igt-dev mailing list
igt-dev@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/igt-dev
^ permalink raw reply related [flat|nested] 11+ messages in thread
* [igt-dev] [PATCH i-g-t v4 5/5] runner: Unit tests for the runner
2018-08-08 11:06 [igt-dev] [PATCH i-g-t 0/5] New test runner to rule them all, v3 Petri Latvala
` (3 preceding siblings ...)
2018-08-08 11:07 ` [igt-dev] [PATCH i-g-t v3 4/5] runner: New test runner Petri Latvala
@ 2018-08-08 11:07 ` Petri Latvala
2018-08-08 11:18 ` [igt-dev] [PATCH i-g-t 0/5] New test runner to rule them all, v3 Arkadiusz Hiler
` (2 subsequent siblings)
7 siblings, 0 replies; 11+ messages in thread
From: Petri Latvala @ 2018-08-08 11:07 UTC (permalink / raw)
To: igt-dev
TODO: Unit tests for the results.json file contents.
v2:
- Avoid writing the nul character to mock files
- Properly set up tmpdirs
v3:
- Restore the resume-related changes that were lost in rebase
v4:
- Better teardown for temporary directories
- Build with autotools
Signed-off-by: Petri Latvala <petri.latvala@intel.com>
---
configure.ac | 1 +
runner/.gitignore | 1 +
runner/Makefile.am | 12 +
runner/meson.build | 8 +
runner/runner_tests.c | 973 ++++++++++++++++++++++++++++++++++++++++++
runner/testdata/Makefile.am | 25 ++
runner/testdata/meson.build | 20 +
runner/testdata/no-subtests.c | 6 +
runner/testdata/skippers.c | 14 +
runner/testdata/successtest.c | 10 +
10 files changed, 1070 insertions(+)
create mode 100644 runner/runner_tests.c
create mode 100644 runner/testdata/Makefile.am
create mode 100644 runner/testdata/meson.build
create mode 100644 runner/testdata/no-subtests.c
create mode 100644 runner/testdata/skippers.c
create mode 100644 runner/testdata/successtest.c
diff --git a/configure.ac b/configure.ac
index b73cc9f0..72f35994 100644
--- a/configure.ac
+++ b/configure.ac
@@ -421,6 +421,7 @@ AC_CONFIG_FILES([
assembler/intel-gen4asm.pc
overlay/Makefile
runner/Makefile
+ runner/testdata/Makefile
])
AC_CONFIG_FILES([tools/intel_aubdump], [chmod +x tools/intel_aubdump])
diff --git a/runner/.gitignore b/runner/.gitignore
index e3919fa7..c5030e66 100644
--- a/runner/.gitignore
+++ b/runner/.gitignore
@@ -1,3 +1,4 @@
igt_runner
igt_resume
igt_results
+runner_test
diff --git a/runner/Makefile.am b/runner/Makefile.am
index 9c13a83c..ddcbf86a 100644
--- a/runner/Makefile.am
+++ b/runner/Makefile.am
@@ -1,6 +1,8 @@
if BUILD_RUNNER
+SUBDIRS = testdata
+
runnerlib = librunner.la
noinst_LTLIBRARIES = $(runnerlib)
librunner_la_SOURCES = \
@@ -28,4 +30,14 @@ AM_CFLAGS = $(JSONC_CFLAGS) \
-I$(srcdir)/../lib \
-D_GNU_SOURCE
+TESTS = runner_test
+check_PROGRAMS = runner_test
+runner_test_SOURCES = runner_tests.c
+runner_test_CFLAGS = -DTESTDATA_DIRECTORY=\"$(abs_builddir)/testdata\" \
+ -I$(top_srcdir)/include/drm-uapi \
+ $(DRM_CFLAGS) $(CAIRO_CFLAGS) $(LIBUDEV_CFLAGS) \
+ -I$(srcdir)/.. \
+ -I$(srcdir)/../lib \
+ -D_GNU_SOURCE
+
endif
diff --git a/runner/meson.build b/runner/meson.build
index 9dafb312..bea1d897 100644
--- a/runner/meson.build
+++ b/runner/meson.build
@@ -9,6 +9,7 @@ runnerlib_sources = [ 'settings.c',
runner_sources = [ 'runner.c' ]
resume_sources = [ 'resume.c' ]
results_sources = [ 'results.c' ]
+runner_test_sources = [ 'runner_tests.c' ]
if _build_runner and jsonc.found()
subdir('testdata')
@@ -35,6 +36,13 @@ if _build_runner and jsonc.found()
install_dir : bindir,
dependencies : igt_deps)
+ runner_test = executable('runner_test', runner_test_sources,
+ c_args : '-DTESTDATA_DIRECTORY="@0@"'.format(testdata_dir),
+ link_with : runnerlib,
+ install : false,
+ dependencies : igt_deps)
+ test('runner', runner_test)
+
build_info += 'Build test runner: Yes'
else
build_info += 'Build test runner: No'
diff --git a/runner/runner_tests.c b/runner/runner_tests.c
new file mode 100644
index 00000000..7c662acc
--- /dev/null
+++ b/runner/runner_tests.c
@@ -0,0 +1,973 @@
+#include <dirent.h>
+#include <fcntl.h>
+#include <sys/stat.h>
+#include <sys/types.h>
+#include <unistd.h>
+
+#include "igt.h"
+
+#include "settings.h"
+#include "job_list.h"
+#include "executor.h"
+
+static char testdatadir[] = TESTDATA_DIRECTORY;
+
+static void igt_assert_eqstr(char *one, char *two)
+{
+ if (one == NULL && two == NULL)
+ return;
+
+ igt_assert_f(one != NULL && two != NULL, "Strings differ (one is NULL): %s vs %s\n", one, two);
+
+ igt_assert_f(!strcmp(one, two), "Strings differ: '%s' vs '%s'\n", one, two);
+}
+
+static void debug_print_executions(struct job_list *list)
+{
+ size_t i;
+ int k;
+
+ igt_debug("Executions:\n");
+ for (i = 0; i < list->size; i++) {
+ struct job_list_entry *entry = &list->entries[i];
+ igt_debug(" %s\n", entry->binary);
+ for (k = 0; k < entry->subtest_count; ++k) {
+ igt_debug(" %s\n", entry->subtests[k]);
+ }
+ }
+
+}
+
+static char *dump_file(int dirfd, char *name)
+{
+ int fd = openat(dirfd, name, O_RDONLY);
+ ssize_t s;
+ char *buf = malloc(256);
+
+ if (fd < 0) {
+ free(buf);
+ return NULL;
+ }
+
+ s = read(fd, buf, 255);
+ if (s < 0) {
+ free(buf);
+ return NULL;
+ }
+
+ buf[s] = '\0';
+ return buf;
+}
+
+static void job_list_filter_test(char *name, char *filterarg1, char *filterarg2,
+ size_t expected_normal, size_t expected_multiple)
+{
+ int multiple;
+ struct settings settings;
+
+ igt_fixture
+ init_settings(&settings);
+
+ for (multiple = 0; multiple < 2; multiple++) {
+ igt_subtest_f("job-list-filters-%s-%s", name, multiple ? "multiple" : "normal") {
+ struct job_list list;
+ char *argv[] = { "runner",
+ /* Ugly but does the trick */
+ multiple ? "--multiple-mode" : "--sync",
+ filterarg1, filterarg2,
+ testdatadir,
+ "path-to-results",
+ };
+ bool success = false;
+ size_t size;
+
+ init_job_list(&list);
+ igt_assert(parse_options(ARRAY_SIZE(argv), argv, &settings));
+
+ success = create_job_list(&list, &settings);
+ size = list.size;
+
+ if (success)
+ debug_print_executions(&list);
+
+ free_job_list(&list);
+
+ igt_assert_f(success, "Job list creation failed\n");
+ igt_assert_eq(size, multiple ? expected_multiple : expected_normal);
+ }
+ }
+
+ igt_fixture
+ free_settings(&settings);
+}
+
+static void clear_directory_fd(int dirfd)
+{
+ DIR* d;
+ struct dirent *dirent;
+
+ d = fdopendir(dirfd);
+
+ if (dirfd < 0 || d == NULL) {
+ return;
+ }
+
+ while ((dirent = readdir(d)) != NULL) {
+ if (strcmp(dirent->d_name, ".") &&
+ strcmp(dirent->d_name, "..")) {
+ if (dirent->d_type == DT_REG) {
+ unlinkat(dirfd, dirent->d_name, 0);
+ } else if (dirent->d_type == DT_DIR) {
+ clear_directory_fd(openat(dirfd, dirent->d_name, O_DIRECTORY | O_RDONLY));
+ unlinkat(dirfd, dirent->d_name, AT_REMOVEDIR);
+ }
+ }
+ }
+
+ closedir(d);
+}
+
+static void clear_directory(char *name)
+{
+ int dirfd = open(name, O_DIRECTORY | O_RDONLY);
+ clear_directory_fd(dirfd);
+ rmdir(name);
+}
+
+static void assert_settings_equal(struct settings *one, struct settings *two)
+{
+ /*
+ * Regex lists are not serialized, and thus won't be compared
+ * here.
+ */
+ igt_assert_eq(one->abort_on_error, two->abort_on_error);
+ igt_assert_eqstr(one->test_list, two->test_list);
+ igt_assert_eqstr(one->name, two->name);
+ igt_assert_eq(one->dry_run, two->dry_run);
+ igt_assert_eq(one->sync, two->sync);
+ igt_assert_eq(one->log_level, two->log_level);
+ igt_assert_eq(one->overwrite, two->overwrite);
+ igt_assert_eq(one->multiple_mode, two->multiple_mode);
+ igt_assert_eq(one->inactivity_timeout, two->inactivity_timeout);
+ igt_assert_eq(one->use_watchdog, two->use_watchdog);
+ igt_assert_eqstr(one->test_root, two->test_root);
+ igt_assert_eqstr(one->results_path, two->results_path);
+}
+
+static void assert_job_list_equal(struct job_list *one, struct job_list *two)
+{
+ size_t i, k;
+
+ igt_assert_eq(one->size, two->size);
+
+ for (i = 0; i < one->size; i++) {
+ struct job_list_entry *eone = &one->entries[i];
+ struct job_list_entry *etwo = &two->entries[i];
+
+ igt_assert_eqstr(eone->binary, etwo->binary);
+ igt_assert_eq(eone->subtest_count, etwo->subtest_count);
+
+ for (k = 0; k < eone->subtest_count; k++) {
+ igt_assert_eqstr(eone->subtests[k], etwo->subtests[k]);
+ }
+ }
+}
+
+static void assert_execution_created(int dirfd, char *name)
+{
+ int fd;
+
+ igt_assert_f((fd = openat(dirfd, name, O_RDONLY)) >= 0,
+ "Execute didn't create %s\n", name);
+ close(fd);
+}
+
+static void assert_execution_results_exist(int dirfd)
+{
+ assert_execution_created(dirfd, "journal.txt");
+ assert_execution_created(dirfd, "out.txt");
+ assert_execution_created(dirfd, "err.txt");
+ assert_execution_created(dirfd, "dmesg.txt");
+}
+
+igt_main
+{
+ struct settings settings;
+
+ igt_fixture
+ init_settings(&settings);
+
+ igt_subtest("default-settings") {
+ char *argv[] = { "runner",
+ "test-root-dir",
+ "path-to-results",
+ };
+
+ igt_assert(parse_options(ARRAY_SIZE(argv), argv, &settings));
+
+ igt_assert(!settings.abort_on_error);
+ igt_assert(!settings.test_list);
+ igt_assert_eqstr(settings.name, "path-to-results");
+ igt_assert(!settings.dry_run);
+ igt_assert_eq(settings.include_regexes.size, 0);
+ igt_assert_eq(settings.exclude_regexes.size, 0);
+ igt_assert(!settings.sync);
+ igt_assert_eq(settings.log_level, LOG_LEVEL_NORMAL);
+ igt_assert(!settings.overwrite);
+ igt_assert(!settings.multiple_mode);
+ igt_assert_eq(settings.inactivity_timeout, 0);
+ igt_assert(!settings.use_watchdog);
+ igt_assert(strstr(settings.test_root, "test-root-dir") != NULL);
+ igt_assert(strstr(settings.results_path, "path-to-results") != NULL);
+ }
+
+ igt_subtest_group {
+ char *cwd;
+ char *path;
+
+ igt_fixture {
+ igt_require((cwd = realpath(".", NULL)) != NULL);
+ path = NULL;
+ }
+
+ igt_subtest("absolute-path-converter") {
+ struct {
+ char *path;
+ bool null;
+ } data[] = { { "simple-name", false },
+ { "foo/bar", true },
+ { ".", false },
+ };
+ size_t i;
+
+ for (i = 0; i < ARRAY_SIZE(data); i++) {
+ free(path);
+ path = absolute_path(data[i].path);
+ if (data[i].null) {
+ igt_assert(path == NULL);
+ continue;
+ }
+
+ igt_assert(path[0] == '/');
+ igt_debug("Got path %s for %s\n", path, data[i].path);
+ igt_assert(strstr(path, cwd) == path);
+ if (strcmp(data[i].path, ".")) {
+ igt_assert(strstr(path, data[i].path) != NULL);
+ }
+ }
+ }
+
+ igt_fixture {
+ free(cwd);
+ free(path);
+ }
+ }
+
+ igt_subtest_group {
+ char dirname[] = "tmpdirXXXXXX";
+ char tmptestlist[] = "tmp.testlist";
+ char pathtotestlist[64];
+ char *path;
+
+ igt_fixture {
+ int dirfd, fd;
+
+ path = NULL;
+
+ igt_require(mkdtemp(dirname) != NULL);
+ igt_require((dirfd = open(dirname, O_DIRECTORY | O_RDONLY)) >= 0);
+ igt_require((fd = openat(dirfd, tmptestlist, O_CREAT | O_EXCL | O_WRONLY, 0660)) >= 0);
+ close(fd);
+ close(dirfd);
+
+ strcpy(pathtotestlist, dirname);
+ strcat(pathtotestlist, "/");
+ strcat(pathtotestlist, tmptestlist);
+ }
+
+ igt_subtest("absolute-path-usage") {
+ char *argv[] = { "runner",
+ "--test-list", pathtotestlist,
+ testdatadir,
+ dirname,
+ };
+
+ igt_assert(parse_options(ARRAY_SIZE(argv), argv, &settings));
+
+ path = realpath(testdatadir, NULL);
+ igt_assert(path != NULL);
+ igt_assert_eqstr(settings.test_root, path);
+ free(path);
+ path = realpath(dirname, NULL);
+ igt_assert(path != NULL);
+ igt_assert_eqstr(settings.results_path, path);
+ free(path);
+ path = realpath(pathtotestlist, NULL);
+ igt_assert(path != NULL);
+ igt_assert_eqstr(settings.test_list, path);
+ }
+
+ igt_fixture {
+ int dirfd;
+
+ igt_require((dirfd = open(dirname, O_DIRECTORY | O_RDONLY)) >= 0);
+ unlinkat(dirfd, tmptestlist, 0);
+ close(dirfd);
+ rmdir(dirname);
+
+ free(path);
+ }
+ }
+
+ igt_subtest("environment-overrides-test-root-flag") {
+ char *argv[] = { "runner",
+ "test-root-dir",
+ "path-to-results",
+ };
+
+ setenv("IGT_TEST_ROOT", testdatadir, 1);
+ igt_assert(parse_options(ARRAY_SIZE(argv), argv, &settings));
+
+ igt_assert(!settings.abort_on_error);
+ igt_assert(!settings.test_list);
+ igt_assert_eqstr(settings.name, "path-to-results");
+ igt_assert(!settings.dry_run);
+ igt_assert_eq(settings.include_regexes.size, 0);
+ igt_assert_eq(settings.exclude_regexes.size, 0);
+ igt_assert(!settings.sync);
+ igt_assert_eq(settings.log_level, LOG_LEVEL_NORMAL);
+ igt_assert(!settings.overwrite);
+ igt_assert(!settings.multiple_mode);
+ igt_assert_eq(settings.inactivity_timeout, 0);
+ igt_assert(!settings.use_watchdog);
+ igt_assert(strstr(settings.test_root, testdatadir) != NULL);
+ igt_assert(strstr(settings.results_path, "path-to-results") != NULL);
+ }
+
+ igt_fixture {
+ unsetenv("IGT_TEST_ROOT");
+ }
+
+ igt_subtest("parse-all-settings") {
+ char *argv[] = { "runner",
+ "-n", "foo",
+ "--abort-on-monitored-error",
+ "--test-list", "path-to-test-list",
+ "--ignore-missing",
+ "--dry-run",
+ "-t", "pattern1",
+ "-t", "pattern2",
+ "-x", "xpattern1",
+ "-x", "xpattern2",
+ "-s",
+ "-l", "verbose",
+ "--overwrite",
+ "--multiple-mode",
+ "--inactivity-timeout", "27",
+ "--use-watchdog",
+ "test-root-dir",
+ "path-to-results",
+ };
+
+ igt_assert(parse_options(ARRAY_SIZE(argv), argv, &settings));
+
+ igt_assert(settings.abort_on_error);
+ igt_assert(strstr(settings.test_list, "path-to-test-list") != NULL);
+ igt_assert_eqstr(settings.name, "foo");
+ igt_assert(settings.dry_run);
+ igt_assert_eq(settings.include_regexes.size, 2);
+ igt_assert_eqstr(settings.include_regexes.regex_strings[0], "pattern1");
+ igt_assert_eqstr(settings.include_regexes.regex_strings[1], "pattern2");
+ igt_assert_eq(settings.exclude_regexes.size, 2);
+ igt_assert_eqstr(settings.exclude_regexes.regex_strings[0], "xpattern1");
+ igt_assert_eqstr(settings.exclude_regexes.regex_strings[1], "xpattern2");
+ igt_assert(settings.sync);
+ igt_assert_eq(settings.log_level, LOG_LEVEL_VERBOSE);
+ igt_assert(settings.overwrite);
+ igt_assert(settings.multiple_mode);
+ igt_assert_eq(settings.inactivity_timeout, 27);
+ igt_assert(settings.use_watchdog);
+ igt_assert(strstr(settings.test_root, "test-root-dir") != NULL);
+ igt_assert(strstr(settings.results_path, "path-to-results") != NULL);
+ }
+
+ igt_subtest("invalid-option") {
+ char *argv[] = { "runner",
+ "--no-such-option",
+ "test-root-dir",
+ "results-path",
+ };
+
+ igt_assert(!parse_options(ARRAY_SIZE(argv), argv, &settings));
+ }
+
+ igt_subtest("paths-missing") {
+ char *argv[] = { "runner",
+ "-o",
+ };
+ igt_assert(!parse_options(ARRAY_SIZE(argv), argv, &settings));
+ }
+
+ igt_subtest("log-levels") {
+ char *argv[] = { "runner",
+ "-l", "normal",
+ "test-root-dir",
+ "results-path",
+ };
+
+ igt_assert(parse_options(ARRAY_SIZE(argv), argv, &settings));
+ igt_assert_eq(settings.log_level, LOG_LEVEL_NORMAL);
+
+ argv[2] = "quiet";
+ igt_assert(parse_options(ARRAY_SIZE(argv), argv, &settings));
+ igt_assert_eq(settings.log_level, LOG_LEVEL_QUIET);
+
+ argv[2] = "verbose";
+ igt_assert(parse_options(ARRAY_SIZE(argv), argv, &settings));
+ igt_assert_eq(settings.log_level, LOG_LEVEL_VERBOSE);
+ }
+
+ igt_subtest("parse-clears-old-data") {
+ char *argv[] = { "runner",
+ "-n", "foo",
+ "--dry-run",
+ "test-root-dir",
+ "results-path",
+ };
+
+ igt_assert(parse_options(ARRAY_SIZE(argv), argv, &settings));
+
+ igt_assert_eqstr(settings.name, "foo");
+ igt_assert(settings.dry_run);
+ igt_assert(!settings.test_list);
+ igt_assert(!settings.sync);
+
+ argv[1] = "--test-list";
+ argv[3] = "--sync";
+
+ igt_assert(parse_options(ARRAY_SIZE(argv), argv, &settings));
+
+ igt_assert_eqstr(settings.name, "results-path");
+ igt_assert(!settings.dry_run);
+ igt_assert(strstr(settings.test_list, "foo") != NULL);
+ igt_assert(settings.sync);
+ }
+
+ igt_subtest_group {
+ char filename[] = "tmplistXXXXXX";
+ int fd = -1;
+
+ igt_fixture {
+ igt_require((fd = mkstemp(filename)) >= 0);
+ close(fd);
+ }
+
+ igt_subtest("validate-ok") {
+ char *argv[] = { "runner",
+ "--test-list", filename,
+ testdatadir,
+ "path-to-results",
+ };
+
+ igt_assert(parse_options(ARRAY_SIZE(argv), argv, &settings));
+
+ igt_assert(validate_settings(&settings));
+ }
+
+ igt_fixture {
+ unlink(filename);
+ }
+ }
+
+ igt_subtest("validate-no-test-list") {
+ char *nosuchfile = "no-such-file";
+ char *argv[] = { "runner",
+ "--test-list", nosuchfile,
+ testdatadir,
+ "path-to-results",
+ };
+
+ igt_assert_lt(open(nosuchfile, O_RDONLY), 0);
+ igt_assert(parse_options(ARRAY_SIZE(argv), argv, &settings));
+
+ igt_assert(!validate_settings(&settings));
+ }
+
+ igt_subtest_group {
+ char dirname[] = "tmpdirXXXXXX";
+ struct job_list list;
+
+ igt_fixture {
+ igt_require(mkdtemp(dirname) != NULL);
+ init_job_list(&list);
+ }
+
+ igt_subtest("job-list-no-test-list-txt") {
+ char *argv[] = { "runner",
+ dirname,
+ "path-to-results",
+ };
+
+ igt_assert(parse_options(ARRAY_SIZE(argv), argv, &settings));
+
+ igt_assert(!create_job_list(&list, &settings));
+ }
+
+ igt_fixture {
+ rmdir(dirname);
+ free_job_list(&list);
+ }
+ }
+
+ job_list_filter_test("nofilters", "-n", "placeholderargs", 5, 3);
+ job_list_filter_test("binary-include", "-t", "successtest", 2, 1);
+ job_list_filter_test("binary-exclude", "-x", "successtest", 3, 2);
+ job_list_filter_test("subtest-include", "-t", "first-subtest", 1, 1);
+ job_list_filter_test("subtest-exclude", "-x", "second-subtest", 4, 3);
+
+ igt_subtest_group {
+ char filename[] = "tmplistXXXXXX";
+ char testlisttext[] = "igt@successtest@first-subtest\n"
+ "igt@successtest@second-subtest\n"
+ "igt@nosubtests\n";
+ int fd = -1, multiple;
+ struct job_list list;
+
+ igt_fixture {
+ igt_require((fd = mkstemp(filename)) >= 0);
+ igt_require(write(fd, testlisttext, strlen(testlisttext)) == strlen(testlisttext));
+ close(fd);
+ init_job_list(&list);
+ }
+
+ for (multiple = 0; multiple < 2; multiple++) {
+ igt_subtest_f("job-list-testlist-%s", multiple ? "multiple" : "normal") {
+ char *argv[] = { "runner",
+ "--test-list", filename,
+ multiple ? "--multiple-mode" : "--sync",
+ testdatadir,
+ "path-to-results",
+ };
+
+ igt_assert(parse_options(ARRAY_SIZE(argv), argv, &settings));
+ igt_assert(create_job_list(&list, &settings));
+
+ igt_assert_eq(list.size, multiple ? 2 : 3);
+
+ igt_assert_eqstr(list.entries[0].binary, "successtest");
+ if (!multiple) igt_assert_eqstr(list.entries[1].binary, "successtest");
+ igt_assert_eqstr(list.entries[multiple ? 1 : 2].binary, "nosubtests");
+
+ igt_assert_eq(list.entries[0].subtest_count, multiple ? 2 : 1);
+ igt_assert_eq(list.entries[1].subtest_count, multiple ? 0 : 1);
+ if (!multiple) igt_assert_eq(list.entries[2].subtest_count, 0);
+
+ igt_assert_eqstr(list.entries[0].subtests[0], "first-subtest");
+ igt_assert_eqstr(list.entries[multiple ? 0 : 1].subtests[multiple ? 1 : 0], "second-subtest");
+ }
+ }
+
+ igt_fixture {
+ unlink(filename);
+ free_job_list(&list);
+ }
+ }
+
+ igt_subtest_group {
+ char dirname[] = "tmpdirXXXXXX";
+ int dirfd = -1, fd = -1;
+ struct settings cmp_settings;
+
+ igt_fixture {
+ igt_require(mkdtemp(dirname) != NULL);
+ rmdir(dirname);
+ init_settings(&cmp_settings);
+ }
+
+ igt_subtest("settings-serialize") {
+ char *argv[] = { "runner",
+ testdatadir,
+ dirname,
+ };
+
+ igt_assert(parse_options(ARRAY_SIZE(argv), argv, &settings));
+
+ igt_assert(serialize_settings(&settings));
+
+ dirfd = open(dirname, O_DIRECTORY, O_RDONLY);
+ igt_assert_f(dirfd >= 0, "Serialization did not create the results directory\n");
+
+ igt_assert_f((fd = openat(dirfd, "metadata.txt", O_RDONLY)),
+ "Opening %s/metadata.txt failed\n", dirname);
+ close(fd);
+
+ igt_assert_f(read_settings(&cmp_settings, dirfd), "Reading settings failed\n");
+ assert_settings_equal(&settings, &cmp_settings);
+ }
+
+ igt_fixture {
+ close(fd);
+ close(dirfd);
+ clear_directory(dirname);
+ free_settings(&cmp_settings);
+ }
+ }
+
+ igt_subtest_group {
+ char dirname[] = "tmpdirXXXXXX";
+ int dirfd = -1, fd = -1;
+ struct job_list list, cmp_list;
+ int multiple;
+
+ igt_fixture {
+ init_job_list(&list);
+ init_job_list(&cmp_list);
+ igt_require(mkdtemp(dirname) != NULL);
+ rmdir(dirname);
+ }
+
+ for (multiple = 0; multiple < 2; multiple++) {
+ igt_subtest_f("job-list-serialize-%s", multiple ? "multiple" : "normal") {
+ char *argv[] = { "runner",
+ /* Ugly */
+ multiple ? "--multiple-mode" : "--sync",
+ testdatadir,
+ dirname,
+ };
+
+ igt_assert(parse_options(ARRAY_SIZE(argv), argv, &settings));
+ igt_assert(create_job_list(&list, &settings));
+
+ igt_assert(serialize_settings(&settings));
+ igt_assert(serialize_job_list(&list, &settings));
+
+ dirfd = open(dirname, O_DIRECTORY, O_RDONLY);
+ igt_assert_f(dirfd >= 0, "Serialization did not create the results directory\n");
+
+ igt_assert_f((fd = openat(dirfd, "joblist.txt", O_RDONLY)) >= 0,
+ "Opening %s/joblist.txt failed\n", dirname);
+ close(fd);
+ fd = -1;
+
+ igt_assert_f(read_job_list(&cmp_list, dirfd), "Reading job list failed\n");
+ assert_job_list_equal(&list, &cmp_list);
+ }
+
+ igt_fixture {
+ close(fd);
+ close(dirfd);
+ clear_directory(dirname);
+ free_job_list(&cmp_list);
+ free_job_list(&list);
+ }
+ }
+ }
+
+ igt_subtest_group {
+ char dirname[] = "tmpdirXXXXXX";
+ struct job_list list;
+ int dirfd = -1, fd = -1;
+
+ igt_fixture {
+ init_job_list(&list);
+ igt_require(mkdtemp(dirname) != NULL);
+ rmdir(dirname);
+ }
+
+ igt_subtest("execute-initialize-new-run") {
+ struct execute_state state;
+ char *argv[] = { "runner",
+ testdatadir,
+ dirname,
+ };
+
+ igt_assert(parse_options(ARRAY_SIZE(argv), argv, &settings));
+ igt_assert(create_job_list(&list, &settings));
+
+ igt_assert(initialize_execute_state(&state, &settings, &list));
+
+ igt_assert_eq(state.next, 0);
+ igt_assert_eq(list.size, 5);
+ igt_assert_f((dirfd = open(dirname, O_DIRECTORY | O_RDONLY)) >= 0,
+ "Execute state initialization didn't create the results directory.\n");
+ igt_assert_f((fd = openat(dirfd, "metadata.txt", O_RDONLY)) >= 0,
+ "Execute state initialization didn't serialize settings.\n");
+ close(fd);
+ igt_assert_f((fd = openat(dirfd, "joblist.txt", O_RDONLY)) >= 0,
+ "Execute state initialization didn't serialize the job list.\n");
+ close(fd);
+ igt_assert_f((fd = openat(dirfd, "journal.txt", O_RDONLY)) < 0,
+ "Execute state initialization created a journal.\n");
+ igt_assert_f((fd = openat(dirfd, "uname.txt", O_RDONLY)) < 0,
+ "Execute state initialization created uname.txt.\n");
+ }
+
+ igt_fixture {
+ close(fd);
+ close(dirfd);
+ clear_directory(dirname);
+ free_job_list(&list);
+ }
+ }
+
+ igt_subtest_group {
+ char dirname[] = "tmpdirXXXXXX";
+ struct job_list list;
+ int dirfd = -1, subdirfd = -1, fd = -1;
+
+ igt_fixture {
+ init_job_list(&list);
+ igt_require(mkdtemp(dirname) != NULL);
+ }
+
+ igt_subtest("execute-initialize-subtest-started") {
+ struct execute_state state;
+ char *argv[] = { "runner",
+ "--multiple-mode",
+ "-t", "successtest",
+ testdatadir,
+ dirname,
+ };
+ char journaltext[] = "first-subtest\n";
+ char excludestring[] = "!first-subtest";
+
+ igt_assert(parse_options(ARRAY_SIZE(argv), argv, &settings));
+ igt_assert(create_job_list(&list, &settings));
+ igt_assert(list.size == 1);
+ igt_assert(list.entries[0].subtest_count == 0);
+
+ igt_assert(serialize_settings(&settings));
+ igt_assert(serialize_job_list(&list, &settings));
+
+ igt_assert((dirfd = open(dirname, O_DIRECTORY | O_RDONLY)) >= 0);
+ igt_assert(mkdirat(dirfd, "0", 0770) == 0);
+ igt_assert((subdirfd = openat(dirfd, "0", O_DIRECTORY | O_RDONLY)) >= 0);
+ igt_assert((fd = openat(subdirfd, "journal.txt", O_CREAT | O_WRONLY | O_EXCL, 0660)) >= 0);
+ igt_assert(write(fd, journaltext, strlen(journaltext)) == strlen(journaltext));
+
+ free_job_list(&list);
+ free_settings(&settings);
+ igt_assert(initialize_execute_state_from_resume(dirfd, &state, &settings, &list));
+
+ igt_assert_eq(state.next, 0);
+ igt_assert_eq(list.size, 1);
+ igt_assert_eq(list.entries[0].subtest_count, 2);
+ igt_assert_eqstr(list.entries[0].subtests[0], "*");
+ igt_assert_eqstr(list.entries[0].subtests[1], excludestring);
+ }
+
+ igt_fixture {
+ close(fd);
+ close(subdirfd);
+ close(dirfd);
+ clear_directory(dirname);
+ free_job_list(&list);
+ }
+ }
+
+ igt_subtest_group {
+ char dirname[] = "tmpdirXXXXXX";
+ struct job_list list;
+ int dirfd = -1, subdirfd = -1, fd = -1;
+
+ igt_fixture {
+ init_job_list(&list);
+ igt_require(mkdtemp(dirname) != NULL);
+ }
+
+ igt_subtest("execute-initialize-subtests-complete") {
+ struct execute_state state;
+ char *argv[] = { "runner",
+ "--multiple-mode",
+ testdatadir,
+ dirname,
+ };
+ char journaltext[] = "first-subtest\nsecond-subtest\nexit:0\n";
+
+ igt_assert(parse_options(ARRAY_SIZE(argv), argv, &settings));
+ igt_assert(create_job_list(&list, &settings));
+ igt_assert(list.size == 3);
+
+ if (!strcmp(list.entries[0].binary, "no-subtests")) {
+ struct job_list_entry tmp = list.entries[0];
+ list.entries[0] = list.entries[1];
+ list.entries[1] = tmp;
+ }
+
+ igt_assert(list.entries[0].subtest_count == 0);
+
+ igt_assert(serialize_settings(&settings));
+ igt_assert(serialize_job_list(&list, &settings));
+
+ igt_assert_lte(0, dirfd = open(dirname, O_DIRECTORY | O_RDONLY));
+ igt_assert_eq(mkdirat(dirfd, "0", 0770), 0);
+ igt_assert((subdirfd = openat(dirfd, "0", O_DIRECTORY | O_RDONLY)) >= 0);
+ igt_assert_lte(0, fd = openat(subdirfd, "journal.txt", O_CREAT | O_WRONLY | O_EXCL, 0660));
+ igt_assert_eq(write(fd, journaltext, sizeof(journaltext)), sizeof(journaltext));
+
+ free_job_list(&list);
+ free_settings(&settings);
+ igt_assert(initialize_execute_state_from_resume(dirfd, &state, &settings, &list));
+
+ igt_assert_eq(state.next, 1);
+ igt_assert_eq(list.size, 3);
+ }
+
+ igt_fixture {
+ close(fd);
+ close(subdirfd);
+ close(dirfd);
+ clear_directory(dirname);
+ free_job_list(&list);
+ }
+ }
+
+ igt_subtest_group {
+ struct job_list list;
+ int dirfd = -1, subdirfd = -1, fd = -1;
+ int multiple;
+
+ igt_fixture {
+ init_job_list(&list);
+ }
+
+ for (multiple = 0; multiple < 2; multiple++) {
+ char dirname[] = "tmpdirXXXXXX";
+
+ igt_fixture {
+ igt_require(mkdtemp(dirname) != NULL);
+ rmdir(dirname);
+ }
+
+ igt_subtest_f("execute-subtests-%s", multiple ? "multiple" : "normal") {
+ struct execute_state state;
+ char *argv[] = { "runner",
+ multiple ? "--multiple-mode" : "--sync",
+ "-t", "-subtest",
+ testdatadir,
+ dirname,
+ };
+ char testdirname[16];
+ size_t expected_tests = multiple ? 2 : 3;
+ size_t i;
+
+ igt_assert(parse_options(ARRAY_SIZE(argv), argv, &settings));
+ igt_assert(create_job_list(&list, &settings));
+ igt_assert(initialize_execute_state(&state, &settings, &list));
+
+ igt_assert(execute(&state, &settings, &list));
+ igt_assert_f((dirfd = open(dirname, O_DIRECTORY | O_RDONLY)) >= 0,
+ "Execute didn't create the results directory\n");
+
+ igt_assert_f((fd = openat(dirfd, "uname.txt", O_RDONLY)) >= 0,
+ "Execute didn't create uname.txt\n");
+ close(fd);
+ fd = -1;
+
+ for (i = 0; i < expected_tests; i++) {
+ snprintf(testdirname, 16, "%zd", i);
+
+ igt_assert_f((subdirfd = openat(dirfd, testdirname, O_DIRECTORY | O_RDONLY)) >= 0,
+ "Execute didn't create result directory '%s'\n", testdirname);
+ assert_execution_results_exist(subdirfd);
+ close(subdirfd);
+ }
+
+ snprintf(testdirname, 16, "%zd", expected_tests);
+ igt_assert_f((subdirfd = openat(dirfd, testdirname, O_DIRECTORY | O_RDONLY)) < 0,
+ "Execute created too many directories\n");
+ }
+
+ igt_fixture {
+ close(fd);
+ close(subdirfd);
+ close(dirfd);
+ clear_directory(dirname);
+ free_job_list(&list);
+ }
+ }
+ }
+
+ igt_subtest_group {
+ struct job_list list;
+ int dirfd = -1, subdirfd = -1, fd = -1;
+ int multiple;
+
+ igt_fixture {
+ init_job_list(&list);
+ }
+
+ for (multiple = 0; multiple < 2; multiple++) {
+ char dirname[] = "tmpdirXXXXXX";
+
+ igt_fixture {
+ igt_require(mkdtemp(dirname) != NULL);
+ rmdir(dirname);
+ }
+
+ igt_subtest_f("execute-skipper-journal-%s", multiple ? "multiple" : "normal") {
+ struct execute_state state;
+ char *argv[] = { "runner",
+ multiple ? "--multiple-mode" : "--sync",
+ "-t", "skippers",
+ testdatadir,
+ dirname,
+ };
+ char *dump;
+ char *expected_0 = multiple ?
+ "skip-one\nskip-two\nexit:77 (" :
+ "skip-one\nexit:77 (";
+ char *expected_1 = "skip-two\nexit:77 (";
+
+ igt_assert(parse_options(ARRAY_SIZE(argv), argv, &settings));
+ igt_assert(create_job_list(&list, &settings));
+ igt_assert(initialize_execute_state(&state, &settings, &list));
+
+ igt_assert(execute(&state, &settings, &list));
+ igt_assert_f((dirfd = open(dirname, O_DIRECTORY | O_RDONLY)) >= 0,
+ "Execute didn't create the results directory\n");
+
+ igt_assert_f((fd = openat(dirfd, "uname.txt", O_RDONLY)) >= 0,
+ "Execute didn't create uname.txt\n");
+ close(fd);
+ fd = -1;
+
+ igt_assert_f((subdirfd = openat(dirfd, "0", O_DIRECTORY | O_RDONLY)) >= 0,
+ "Execute didn't create result directory '0'\n");
+ dump = dump_file(subdirfd, "journal.txt");
+ igt_assert_f(dump != NULL,
+ "Execute didn't create the journal\n");
+ /* Trim out the runtime */
+ dump[strlen(expected_0)] = '\0';
+ igt_assert_eqstr(dump, expected_0);
+ free(dump);
+ close(subdirfd);
+ subdirfd = -1;
+
+ if (!multiple) {
+ igt_assert_f((subdirfd = openat(dirfd, "1", O_DIRECTORY | O_RDONLY)) >= 0,
+ "Execute didn't create result directory '1'\n");
+ dump = dump_file(subdirfd, "journal.txt");
+ igt_assert_f(dump != NULL,
+ "Execute didn't create the journal\n");
+ dump[strlen(expected_1)] = '\0';
+ igt_assert_eqstr(dump, expected_1);
+ free(dump);
+ close(subdirfd);
+ subdirfd = -1;
+ }
+ }
+
+ igt_fixture {
+ close(fd);
+ close(subdirfd);
+ close(dirfd);
+ clear_directory(dirname);
+ free_job_list(&list);
+ }
+ }
+ }
+
+ igt_fixture
+ free_settings(&settings);
+}
diff --git a/runner/testdata/Makefile.am b/runner/testdata/Makefile.am
new file mode 100644
index 00000000..fe225d8d
--- /dev/null
+++ b/runner/testdata/Makefile.am
@@ -0,0 +1,25 @@
+testdata_progs = no-subtests skippers successtest
+
+noinst_PROGRAMS = $(testdata_progs)
+
+test-list.txt: Makefile
+ @echo TESTLIST > $@
+ @echo ${testdata_progs} >> $@
+ @echo END TESTLIST >> $@
+
+noinst_DATA = test-list.txt
+
+all-local: .gitignore
+.gitignore: Makefile.am
+ @echo "$(testdata_progs) test-list.txt /.gitignore" | sed 's/\s\+/\n/g' | sort > $@
+
+CLEANFILES = test-list.txt .gitignore
+
+AM_CFLAGS = $(CWARNFLAGS) -Wno-unused-result $(DEBUG_CFLAGS) \
+ -I$(top_srcdir)/include/drm-uapi \
+ -I$(srcdir)/../.. \
+ -I$(srcdir)/../../lib \
+ $(DRM_CFLAGS) $(CAIRO_CFLAGS) $(LIBUDEV_CFLAGS) \
+ -D_GNU_SOURCE
+
+LDADD = $(top_builddir)/lib/libintel_tools.la
diff --git a/runner/testdata/meson.build b/runner/testdata/meson.build
new file mode 100644
index 00000000..011eff8e
--- /dev/null
+++ b/runner/testdata/meson.build
@@ -0,0 +1,20 @@
+
+testdata_progs = [ 'successtest',
+ 'no-subtests',
+ 'skippers',
+ ]
+
+testdata_executables = []
+
+foreach prog : testdata_progs
+ testdata_executables += executable(prog, prog + '.c',
+ dependencies : igt_deps,
+ install : false)
+endforeach
+
+testdata_list = custom_target('testdata_testlist',
+ output : 'test-list.txt',
+ command : [ gen_testlist, '@OUTPUT@', testdata_progs ],
+ build_by_default : true)
+
+testdata_dir = meson.current_build_dir()
diff --git a/runner/testdata/no-subtests.c b/runner/testdata/no-subtests.c
new file mode 100644
index 00000000..00c92e25
--- /dev/null
+++ b/runner/testdata/no-subtests.c
@@ -0,0 +1,6 @@
+#include "igt.h"
+
+igt_simple_main
+{
+
+}
diff --git a/runner/testdata/skippers.c b/runner/testdata/skippers.c
new file mode 100644
index 00000000..be4a31b4
--- /dev/null
+++ b/runner/testdata/skippers.c
@@ -0,0 +1,14 @@
+#include "igt.h"
+
+igt_main
+{
+ igt_fixture {
+ igt_require_f(false, "Skipping from fixture\n");
+ }
+
+ igt_subtest("skip-one")
+ igt_debug("Should be skipped\n");
+
+ igt_subtest("skip-two")
+ igt_debug("Should be skipped\n");
+}
diff --git a/runner/testdata/successtest.c b/runner/testdata/successtest.c
new file mode 100644
index 00000000..bb411eb4
--- /dev/null
+++ b/runner/testdata/successtest.c
@@ -0,0 +1,10 @@
+#include "igt.h"
+
+igt_main
+{
+ igt_subtest("first-subtest")
+ igt_debug("Running first subtest\n");
+
+ igt_subtest("second-subtest")
+ igt_debug("Running second subtest\n");
+}
--
2.14.1
_______________________________________________
igt-dev mailing list
igt-dev@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/igt-dev
^ permalink raw reply related [flat|nested] 11+ messages in thread
* Re: [igt-dev] [PATCH i-g-t 1/5] lib: Print subtest starting/ending line to stderr too
2018-08-08 11:06 ` [igt-dev] [PATCH i-g-t 1/5] lib: Print subtest starting/ending line to stderr too Petri Latvala
@ 2018-08-08 11:17 ` Chris Wilson
2018-08-08 11:28 ` Petri Latvala
0 siblings, 1 reply; 11+ messages in thread
From: Chris Wilson @ 2018-08-08 11:17 UTC (permalink / raw)
To: Petri Latvala, igt-dev
Quoting Petri Latvala (2018-08-08 12:06:57)
> when instructed via the environment. This is needed for the new test
> runner to properly assign stderr output to the correct subtest.
My assumption was that this would all be integrated into the igt
framework, so this information would be provided by the runner to igt as
part of its protocol -- I was expecting much more directness and that
when we run the tests we get the same results as if run indirectly (kmsg
capturing, warn diagnosing, /dev/watchdog integration, etc)
-Chris
_______________________________________________
igt-dev mailing list
igt-dev@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/igt-dev
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [igt-dev] [PATCH i-g-t 0/5] New test runner to rule them all, v3
2018-08-08 11:06 [igt-dev] [PATCH i-g-t 0/5] New test runner to rule them all, v3 Petri Latvala
` (4 preceding siblings ...)
2018-08-08 11:07 ` [igt-dev] [PATCH i-g-t v4 5/5] runner: Unit tests for the runner Petri Latvala
@ 2018-08-08 11:18 ` Arkadiusz Hiler
2018-08-08 12:41 ` [igt-dev] ✓ Fi.CI.BAT: success for " Patchwork
2018-08-08 17:35 ` [igt-dev] ✗ Fi.CI.IGT: failure " Patchwork
7 siblings, 0 replies; 11+ messages in thread
From: Arkadiusz Hiler @ 2018-08-08 11:18 UTC (permalink / raw)
To: Petri Latvala; +Cc: igt-dev
On Wed, Aug 08, 2018 at 02:06:56PM +0300, Petri Latvala wrote:
> Once more, with feeling.
>
> This version addresses Arek's review feedback, adds autotools
> building, and adds a couple of TODO comments. Hopefully this version
> is good to land so work can begin on implementing those TODO items
> without changes getting lost where only interdiff can reveal them.
>
> Petri Latvala (5):
> lib: Print subtest starting/ending line to stderr too
> lib: Export igt_gettime and igt_time_elapsed
> uwildmat: Case-insensitive test selection
> runner: New test runner
> runner: Unit tests for the runner
This looked pretty good in the last iteration already, so have my
Reviewed-by: Arkadiusz Hiler <arkadiusz.hiler@intel.com>
on the whole thing.
Merging this will make any future updates more reviewable :P
Thanks for the hard work!
--
Cheers,
Arek
_______________________________________________
igt-dev mailing list
igt-dev@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/igt-dev
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [igt-dev] [PATCH i-g-t 1/5] lib: Print subtest starting/ending line to stderr too
2018-08-08 11:17 ` Chris Wilson
@ 2018-08-08 11:28 ` Petri Latvala
0 siblings, 0 replies; 11+ messages in thread
From: Petri Latvala @ 2018-08-08 11:28 UTC (permalink / raw)
To: Chris Wilson; +Cc: igt-dev
On Wed, Aug 08, 2018 at 12:17:48PM +0100, Chris Wilson wrote:
> Quoting Petri Latvala (2018-08-08 12:06:57)
> > when instructed via the environment. This is needed for the new test
> > runner to properly assign stderr output to the correct subtest.
>
> My assumption was that this would all be integrated into the igt
> framework, so this information would be provided by the runner to igt as
> part of its protocol -- I was expecting much more directness and that
> when we run the tests we get the same results as if run indirectly (kmsg
> capturing, warn diagnosing, /dev/watchdog integration, etc)
You can consider this a temporary implementation detail. I'm thinking
of making the tests print their logs via a single fd if the runner is
executing them, making all that sentinel business unnecessary.
Having the runner consolidate the logs and handle the log buffering
would also make it possible to grab the debug prints from
incompletes...
--
Petri Latvala
_______________________________________________
igt-dev mailing list
igt-dev@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/igt-dev
^ permalink raw reply [flat|nested] 11+ messages in thread
* [igt-dev] ✓ Fi.CI.BAT: success for New test runner to rule them all, v3
2018-08-08 11:06 [igt-dev] [PATCH i-g-t 0/5] New test runner to rule them all, v3 Petri Latvala
` (5 preceding siblings ...)
2018-08-08 11:18 ` [igt-dev] [PATCH i-g-t 0/5] New test runner to rule them all, v3 Arkadiusz Hiler
@ 2018-08-08 12:41 ` Patchwork
2018-08-08 17:35 ` [igt-dev] ✗ Fi.CI.IGT: failure " Patchwork
7 siblings, 0 replies; 11+ messages in thread
From: Patchwork @ 2018-08-08 12:41 UTC (permalink / raw)
To: Petri Latvala; +Cc: igt-dev
== Series Details ==
Series: New test runner to rule them all, v3
URL : https://patchwork.freedesktop.org/series/47878/
State : success
== Summary ==
= CI Bug Log - changes from CI_DRM_4633 -> IGTPW_1691 =
== Summary - SUCCESS ==
No regressions found.
External URL: https://patchwork.freedesktop.org/api/1.0/series/47878/revisions/1/mbox/
== Known issues ==
Here are the changes found in IGTPW_1691 that come from known issues:
=== IGT changes ===
==== Issues hit ====
igt@drv_selftest@live_hangcheck:
{fi-icl-u}: NOTRUN -> INCOMPLETE (fdo#107399)
igt@drv_selftest@live_workarounds:
fi-cnl-psr: PASS -> DMESG-FAIL (fdo#107292)
igt@kms_pipe_crc_basic@suspend-read-crc-pipe-c:
{fi-icl-u}: NOTRUN -> DMESG-WARN (fdo#107382) +4
{igt@kms_psr@primary_page_flip}:
{fi-icl-u}: NOTRUN -> FAIL (fdo#107383) +3
==== Possible fixes ====
igt@drv_selftest@live_hangcheck:
fi-skl-guc: DMESG-FAIL (fdo#107174) -> PASS
igt@drv_selftest@live_workarounds:
fi-whl-u: DMESG-FAIL (fdo#107292) -> PASS
fi-kbl-x1275: DMESG-FAIL (fdo#107292) -> PASS
{igt@kms_psr@primary_mmap_gtt}:
fi-cnl-psr: DMESG-WARN (fdo#107372) -> PASS
{name}: This element is suppressed. This means it is ignored when computing
the status of the difference (SUCCESS, WARNING, or FAILURE).
fdo#107174 https://bugs.freedesktop.org/show_bug.cgi?id=107174
fdo#107292 https://bugs.freedesktop.org/show_bug.cgi?id=107292
fdo#107372 https://bugs.freedesktop.org/show_bug.cgi?id=107372
fdo#107382 https://bugs.freedesktop.org/show_bug.cgi?id=107382
fdo#107383 https://bugs.freedesktop.org/show_bug.cgi?id=107383
fdo#107399 https://bugs.freedesktop.org/show_bug.cgi?id=107399
== Participating hosts (51 -> 48) ==
Additional (2): fi-icl-u fi-bxt-dsi
Missing (5): fi-ctg-p8600 fi-ilk-m540 fi-byt-squawks fi-bsw-cyan fi-hsw-4200u
== Build changes ==
* IGT: IGT_4588 -> IGTPW_1691
CI_DRM_4633: ea6e3f703e4d234c9c8eaec6c533355c7454ecb6 @ git://anongit.freedesktop.org/gfx-ci/linux
IGTPW_1691: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_1691/
IGT_4588: 7e5abbe4d9b2129bbbf02be77a70cad3da2ab941 @ git://anongit.freedesktop.org/xorg/app/intel-gpu-tools
== Logs ==
For more details see: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_1691/issues.html
_______________________________________________
igt-dev mailing list
igt-dev@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/igt-dev
^ permalink raw reply [flat|nested] 11+ messages in thread
* [igt-dev] ✗ Fi.CI.IGT: failure for New test runner to rule them all, v3
2018-08-08 11:06 [igt-dev] [PATCH i-g-t 0/5] New test runner to rule them all, v3 Petri Latvala
` (6 preceding siblings ...)
2018-08-08 12:41 ` [igt-dev] ✓ Fi.CI.BAT: success for " Patchwork
@ 2018-08-08 17:35 ` Patchwork
7 siblings, 0 replies; 11+ messages in thread
From: Patchwork @ 2018-08-08 17:35 UTC (permalink / raw)
To: Petri Latvala; +Cc: igt-dev
== Series Details ==
Series: New test runner to rule them all, v3
URL : https://patchwork.freedesktop.org/series/47878/
State : failure
== Summary ==
= CI Bug Log - changes from IGT_4588_full -> IGTPW_1691_full =
== Summary - FAILURE ==
Serious unknown changes coming with IGTPW_1691_full absolutely need to be
verified manually.
If you think the reported changes have nothing to do with the changes
introduced in IGTPW_1691_full, please notify your bug team to allow them
to document this new failure mode, which will reduce false positives in CI.
External URL: https://patchwork.freedesktop.org/api/1.0/series/47878/revisions/1/mbox/
== Possible new issues ==
Here are the unknown changes that may have been introduced in IGTPW_1691_full:
=== IGT changes ===
==== Possible regressions ====
igt@gem_eio@reset-stress:
shard-snb: PASS -> FAIL
igt@gem_eio@wait-10ms:
shard-hsw: PASS -> FAIL
== Known issues ==
Here are the changes found in IGTPW_1691_full that come from known issues:
=== IGT changes ===
==== Issues hit ====
igt@gem_ctx_isolation@vcs1-s3:
shard-kbl: PASS -> INCOMPLETE (fdo#103665)
igt@gem_wait@write-busy-render:
shard-snb: PASS -> INCOMPLETE (fdo#105411) +1
igt@kms_cursor_legacy@2x-long-cursor-vs-flip-legacy:
shard-hsw: PASS -> FAIL (fdo#105767)
igt@kms_rotation_crc@primary-rotation-180:
shard-snb: PASS -> FAIL (fdo#103925)
igt@kms_setmode@basic:
shard-kbl: PASS -> FAIL (fdo#99912)
igt@pm_rpm@basic-pci-d3-state:
shard-glk: PASS -> FAIL (fdo#106539) +1
igt@testdisplay:
shard-glk: PASS -> INCOMPLETE (fdo#107093, k.org#198133, fdo#103359)
==== Possible fixes ====
igt@drv_suspend@shrink:
shard-snb: INCOMPLETE (fdo#105411, fdo#106886) -> PASS
igt@gem_softpin@evict-snoop-interruptible:
shard-snb: INCOMPLETE (fdo#105411) -> SKIP
igt@kms_setmode@basic:
shard-apl: FAIL (fdo#99912) -> PASS
igt@kms_vblank@pipe-a-ts-continuation-modeset-rpm:
shard-apl: FAIL (fdo#106539) -> PASS +1
igt@pm_rpm@gem-pread:
shard-glk: WARN -> PASS
igt@pm_rpm@modeset-non-lpsp-stress:
shard-kbl: FAIL (fdo#106539) -> PASS +1
shard-hsw: FAIL (fdo#106539) -> PASS +1
shard-glk: FAIL (fdo#106539) -> PASS +1
fdo#103359 https://bugs.freedesktop.org/show_bug.cgi?id=103359
fdo#103665 https://bugs.freedesktop.org/show_bug.cgi?id=103665
fdo#103925 https://bugs.freedesktop.org/show_bug.cgi?id=103925
fdo#105411 https://bugs.freedesktop.org/show_bug.cgi?id=105411
fdo#105767 https://bugs.freedesktop.org/show_bug.cgi?id=105767
fdo#106539 https://bugs.freedesktop.org/show_bug.cgi?id=106539
fdo#106886 https://bugs.freedesktop.org/show_bug.cgi?id=106886
fdo#107093 https://bugs.freedesktop.org/show_bug.cgi?id=107093
fdo#99912 https://bugs.freedesktop.org/show_bug.cgi?id=99912
k.org#198133 https://bugzilla.kernel.org/show_bug.cgi?id=198133
== Participating hosts (5 -> 5) ==
No changes in participating hosts
== Build changes ==
* IGT: IGT_4588 -> IGTPW_1691
* Linux: CI_DRM_4632 -> CI_DRM_4633
CI_DRM_4632: 648e2ff1094eabf43613f41d4d719c1a1f555dbb @ git://anongit.freedesktop.org/gfx-ci/linux
CI_DRM_4633: ea6e3f703e4d234c9c8eaec6c533355c7454ecb6 @ git://anongit.freedesktop.org/gfx-ci/linux
IGTPW_1691: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_1691/
IGT_4588: 7e5abbe4d9b2129bbbf02be77a70cad3da2ab941 @ git://anongit.freedesktop.org/xorg/app/intel-gpu-tools
== Logs ==
For more details see: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_1691/shards.html
_______________________________________________
igt-dev mailing list
igt-dev@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/igt-dev
^ permalink raw reply [flat|nested] 11+ messages in thread
end of thread, other threads:[~2018-08-08 17:35 UTC | newest]
Thread overview: 11+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2018-08-08 11:06 [igt-dev] [PATCH i-g-t 0/5] New test runner to rule them all, v3 Petri Latvala
2018-08-08 11:06 ` [igt-dev] [PATCH i-g-t 1/5] lib: Print subtest starting/ending line to stderr too Petri Latvala
2018-08-08 11:17 ` Chris Wilson
2018-08-08 11:28 ` Petri Latvala
2018-08-08 11:06 ` [igt-dev] [PATCH i-g-t 2/5] lib: Export igt_gettime and igt_time_elapsed Petri Latvala
2018-08-08 11:06 ` [igt-dev] [PATCH i-g-t 3/5] uwildmat: Case-insensitive test selection Petri Latvala
2018-08-08 11:07 ` [igt-dev] [PATCH i-g-t v3 4/5] runner: New test runner Petri Latvala
2018-08-08 11:07 ` [igt-dev] [PATCH i-g-t v4 5/5] runner: Unit tests for the runner Petri Latvala
2018-08-08 11:18 ` [igt-dev] [PATCH i-g-t 0/5] New test runner to rule them all, v3 Arkadiusz Hiler
2018-08-08 12:41 ` [igt-dev] ✓ Fi.CI.BAT: success for " Patchwork
2018-08-08 17:35 ` [igt-dev] ✗ Fi.CI.IGT: failure " Patchwork
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).