[igt-dev] [PATCH i-g-t 0/4] New runner to rule them all

[igt-dev] [PATCH i-g-t 0/4] New runner to rule them all

[Petri Latvala]

A new test running framework to replace piglit; Refer to patch 3/4 for
a more thorough explanation.

Note, building it with autotools not done. Only meson hooked up atm.

New dependency: libjson-c
Future new dependency: libbz2

Cc: Maarten Lankhorst <maarten.lankhorst@linux.intel.com>
Cc: Arkadiusz Hiler <arkadiusz.hiler@intel.com>
Cc: Tomi Sarvela <tomi.p.sarvela@intel.com>
Cc: Martin Peres <martin.peres@linux.intel.com>

Petri Latvala (4):
  lib: Print subtest starting/ending line to stderr too
  uwildmat: Case-insensitive test selection
  runner: New test runner
  runner: Unit tests for the runner

 lib/igt_core.c                |   12 +-
 lib/uwildmat/uwildmat.c       |   11 +-
 meson.build                   |    1 +
 runner/executor.c             | 1076 +++++++++++++++++++++++++++++++++++++++++
 runner/executor.h             |   42 ++
 runner/job_list.c             |  492 +++++++++++++++++++
 runner/job_list.h             |   37 ++
 runner/meson.build            |   34 ++
 runner/resultgen.c            |  864 +++++++++++++++++++++++++++++++++
 runner/resultgen.h            |    9 +
 runner/results.c              |   26 +
 runner/runner.c               |   40 ++
 runner/runner_tests.c         |  967 ++++++++++++++++++++++++++++++++++++
 runner/settings.c             |  506 +++++++++++++++++++
 runner/settings.h             |  111 +++++
 runner/testdata/meson.build   |   20 +
 runner/testdata/no-subtests.c |    6 +
 runner/testdata/skippers.c    |   14 +
 runner/testdata/successtest.c |   10 +
 19 files changed, 4274 insertions(+), 4 deletions(-)
 create mode 100644 runner/executor.c
 create mode 100644 runner/executor.h
 create mode 100644 runner/job_list.c
 create mode 100644 runner/job_list.h
 create mode 100644 runner/meson.build
 create mode 100644 runner/resultgen.c
 create mode 100644 runner/resultgen.h
 create mode 100644 runner/results.c
 create mode 100644 runner/runner.c
 create mode 100644 runner/runner_tests.c
 create mode 100644 runner/settings.c
 create mode 100644 runner/settings.h
 create mode 100644 runner/testdata/meson.build
 create mode 100644 runner/testdata/no-subtests.c
 create mode 100644 runner/testdata/skippers.c
 create mode 100644 runner/testdata/successtest.c

-- 
2.14.1

_______________________________________________
igt-dev mailing list
igt-dev@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/igt-dev

[igt-dev] [PATCH i-g-t 1/4] lib: Print subtest starting/ending line to stderr too

[Petri Latvala]

when instructed via the environment. This is needed for the new test
runner to properly assign stderr output to the correct subtest.

Signed-off-by: Petri Latvala <petri.latvala@intel.com>
Cc: Maarten Lankhorst <maarten.lankhorst@linux.intel.com>
Cc: Arkadiusz Hiler <arkadiusz.hiler@intel.com>
Cc: Tomi Sarvela <tomi.p.sarvela@intel.com>
Cc: Martin Peres <martin.peres@linux.intel.com>
---
 lib/igt_core.c | 12 +++++++++++-
 1 file changed, 11 insertions(+), 1 deletion(-)

diff --git a/lib/igt_core.c b/lib/igt_core.c
index 5092a3f0..da149c10 100644
--- a/lib/igt_core.c
+++ b/lib/igt_core.c
@@ -305,6 +305,8 @@ GKeyFile *igt_key_file;
 
 char *igt_frame_dump_path;
 
+static bool stderr_needs_sentinel = false;
+
 const char *igt_test_name(void)
 {
 	return command_str;
@@ -649,6 +651,8 @@ static void common_init_env(void)
 	}
 
 	igt_frame_dump_path = getenv("IGT_FRAME_DUMP_PATH");
+
+	stderr_needs_sentinel = getenv("IGT_SENTINEL_ON_STDERR") != NULL;
 }
 
 static int common_init(int *argc, char **argv,
@@ -932,7 +936,10 @@ bool __igt_run_subtest(const char *subtest_name)
 
 	igt_kmsg(KMSG_INFO "%s: starting subtest %s\n",
 		 command_str, subtest_name);
-	igt_debug("Starting subtest: %s\n", subtest_name);
+	igt_info("Starting subtest: %s\n", subtest_name);
+	fflush(stdout);
+	if (stderr_needs_sentinel)
+		fprintf(stderr, "Starting subtest: %s\n", subtest_name);
 
 	_igt_log_buffer_reset();
 
@@ -987,6 +994,9 @@ static void exit_subtest(const char *result)
 		 in_subtest, result, time_elapsed(&subtest_time, &now),
 		 (!__igt_plain_output) ? "\x1b[0m" : "");
 	fflush(stdout);
+	if (stderr_needs_sentinel)
+		fprintf(stderr, "Subtest %s: %s (%.3fs)\n",
+			in_subtest, result, time_elapsed(&subtest_time, &now));
 
 	igt_terminate_spin_batches();
 
-- 
2.14.1

_______________________________________________
igt-dev mailing list
igt-dev@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/igt-dev

[igt-dev] [PATCH i-g-t 2/4] uwildmat: Case-insensitive test selection

[Petri Latvala]

Since we only use plain ascii in subtest names, using non-locale-aware
tolower() to compare case-insensitively works.

Doing this within uwildmat instead of tolowering the subtest name and
then calling uwildmat() is required, because of selection strings
like:

foo,bar,!Foo

The above line will select subtest Bar, and not select Foo.

Signed-off-by: Petri Latvala <petri.latvala@intel.com>
Cc: Maarten Lankhorst <maarten.lankhorst@linux.intel.com>
Cc: Arkadiusz Hiler <arkadiusz.hiler@intel.com>
Cc: Tomi Sarvela <tomi.p.sarvela@intel.com>
Cc: Martin Peres <martin.peres@linux.intel.com>
---
 lib/uwildmat/uwildmat.c | 11 ++++++++---
 1 file changed, 8 insertions(+), 3 deletions(-)

diff --git a/lib/uwildmat/uwildmat.c b/lib/uwildmat/uwildmat.c
index 09155865..3beafce2 100644
--- a/lib/uwildmat/uwildmat.c
+++ b/lib/uwildmat/uwildmat.c
@@ -93,6 +93,7 @@
  **  accompanying test suite achieves 100% coverage of this file.
  */
 
+#include <ctype.h>
 #include <string.h>
 #include <stdint.h>
 #include "uwildmat/uwildmat.h"
@@ -222,6 +223,7 @@ match_class(uint32_t text, const unsigned char *start,
 	const unsigned char *p = start;
 	uint32_t first = 0;
 	uint32_t last;
+	uint32_t lc = tolower(text);
 
 	/* Check for an inverted character class (starting with ^).  If the
 	   character matches the character class, we return !reversed; that way,
@@ -244,12 +246,13 @@ match_class(uint32_t text, const unsigned char *start,
 		if (allowrange && *p == '-' && p < end) {
 			p++;
 			p += utf8_decode(p, end, &last);
-			if (text >= first && text <= last)
+			if ((text >= first && text <= last) ||
+			    (lc >= first && lc <= last))
 				return !reversed;
 			allowrange = false;
 		} else {
 			p += utf8_decode(p, end, &first);
-			if (text == first)
+			if (text == first || lc == first)
 				return !reversed;
 			allowrange = true;
 		}
@@ -272,6 +275,7 @@ match_pattern(const unsigned char *text, const unsigned char *start,
 	bool ismeta;
 	int matched, width;
 	uint32_t c;
+	unsigned char lc;
 
 	for (; p <= end; p++) {
 		if (!*text && *p != '*')
@@ -284,7 +288,8 @@ match_pattern(const unsigned char *text, const unsigned char *start,
 			/* Fall through. */
 
 		default:
-			if (*text++ != *p)
+			lc = tolower(*text);
+			if (*text++ != *p && lc != *p)
 				return false;
 			break;
 
-- 
2.14.1

_______________________________________________
igt-dev mailing list
igt-dev@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/igt-dev

[igt-dev] [PATCH i-g-t 3/4] runner: New test runner

[Petri Latvala]

This is a new test runner to replace piglit. Piglit has been very
useful as a test runner, but certain improvements have been very
difficult if possible at all in a generic test running framework.

Important improvements over piglit:

- Faster to launch. Being able to make assumptions about what we're
  executing makes it possible to save significant amounts of time. For
  example, a testlist file's line "igt@somebinary@somesubtest" already
  has all the information we need to construct the correct command
  line to execute that particular subtest, instead of listing all
  subtests of all test binaries and mapping them to command
  lines. Same goes for the regexp filters command line flags -t and
  -x; If we use -x somebinaryname, we don't need to list subtests from
  somebinaryname, we already know none of them will get executed.

- Logs of incomplete tests. Piglit collects test output to memory and
  dumps them to a file when the test is complete. The new runner
  writes all output to disk immediately.

- Ability to execute multiple subtests in one binary execution. This
  was possible with piglit, but its semantics made it very hard to
  implement in practice. For example, having a testlist file not only
  selected a subset of tests to run, but also mandated that they be
  executed in the same order.

- Flexible timeout support. Instead of mandating a time tests cannot
  exceed, the new runner has a timeout on inactivity. Activity is
  any output on the test's stdout or stderr, or kernel activity via
  /dev/kmsg.

The runner is fairly piglit compatible. The command line is very
similar, with a few additions. IGT_TEST_ROOT environment flag is still
supported, but can also be set via command line (in place of igt.py in
piglit command line).

The results are a set of log files, processed into a piglit-compatible
results.json file (BZ2 compression TODO). There are some new fields in
the json for extra information:

- "igt-version" contains the IGT version line. In
  multiple-subtests-mode the version information is only printed once,
  so it needs to be duplicated to all subtest results this way.
- "dmesg-warnings" contains the dmesg lines that triggered a
  dmesg-warn/dmesg-fail state.
- Runtime information will be different. Piglit takes a timestamp at
  the beginning and at the end of execution for runtime. The new
  runner uses the subtest output text. The binary execution time will
  also be included; The key "igt@somebinary" will have the runtime of
  the binary "somebinary", whereas "igt@somebinary@a" etc will have
  the runtime of the subtests. Substracting the subtest runtimes from
  the binary runtime yields the total time spent doing setup in
  igt_fixture blocks.

Signed-off-by: Petri Latvala <petri.latvala@intel.com>
Cc: Maarten Lankhorst <maarten.lankhorst@linux.intel.com>
Cc: Arkadiusz Hiler <arkadiusz.hiler@intel.com>
Cc: Tomi Sarvela <tomi.p.sarvela@intel.com>
Cc: Martin Peres <martin.peres@linux.intel.com>
---
 meson.build        |    1 +
 runner/executor.c  | 1076 ++++++++++++++++++++++++++++++++++++++++++++++++++++
 runner/executor.h  |   42 ++
 runner/job_list.c  |  492 ++++++++++++++++++++++++
 runner/job_list.h  |   37 ++
 runner/meson.build |   24 ++
 runner/resultgen.c |  864 +++++++++++++++++++++++++++++++++++++++++
 runner/resultgen.h |    9 +
 runner/results.c   |   26 ++
 runner/runner.c    |   40 ++
 runner/settings.c  |  506 ++++++++++++++++++++++++
 runner/settings.h  |  111 ++++++
 12 files changed, 3228 insertions(+)
 create mode 100644 runner/executor.c
 create mode 100644 runner/executor.h
 create mode 100644 runner/job_list.c
 create mode 100644 runner/job_list.h
 create mode 100644 runner/meson.build
 create mode 100644 runner/resultgen.c
 create mode 100644 runner/resultgen.h
 create mode 100644 runner/results.c
 create mode 100644 runner/runner.c
 create mode 100644 runner/settings.c
 create mode 100644 runner/settings.h

diff --git a/meson.build b/meson.build
index 5b783e5d..35e42631 100644
--- a/meson.build
+++ b/meson.build
@@ -136,6 +136,7 @@ subdir('lib')
 subdir('tests')
 subdir('benchmarks')
 subdir('tools')
+subdir('runner')
 if libdrm_intel.found()
 	subdir('assembler')
 	if ['x86', 'x86_64'].contains(host_machine.cpu_family())
diff --git a/runner/executor.c b/runner/executor.c
new file mode 100644
index 00000000..2754f5a2
--- /dev/null
+++ b/runner/executor.c
@@ -0,0 +1,1076 @@
+#include <errno.h>
+#include <fcntl.h>
+#include <linux/watchdog.h>
+#include <signal.h>
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+#include <sys/ioctl.h>
+#include <sys/select.h>
+#include <sys/signalfd.h>
+#include <sys/stat.h>
+#include <sys/types.h>
+#include <sys/utsname.h>
+#include <sys/wait.h>
+#include <time.h>
+#include <unistd.h>
+
+#include "executor.h"
+
+/* Clock handling copied from igt_core.c */
+
+static clockid_t igt_clock;
+
+#define time_valid(ts) ((ts)->tv_sec || (ts)->tv_nsec)
+
+static double
+time_elapsed(struct timespec *then,
+	     struct timespec *now)
+{
+	double elapsed = -1.;
+
+	if (time_valid(then) && time_valid(now)) {
+		elapsed = now->tv_sec - then->tv_sec;
+		elapsed += (now->tv_nsec - then->tv_nsec) * 1e-9;
+	}
+
+	return elapsed;
+}
+
+static int gettime(struct timespec *ts)
+{
+	memset(ts, 0, sizeof(*ts));
+	errno = 0;
+
+	/* Stay on the same clock for consistency. */
+	if (igt_clock != (clockid_t)-1) {
+		if (clock_gettime(igt_clock, ts))
+			goto error;
+		return 0;
+	}
+
+#ifdef CLOCK_MONOTONIC_RAW
+	if (!clock_gettime(igt_clock = CLOCK_MONOTONIC_RAW, ts))
+		return 0;
+#endif
+#ifdef CLOCK_MONOTONIC_COARSE
+	if (!clock_gettime(igt_clock = CLOCK_MONOTONIC_COARSE, ts))
+		return 0;
+#endif
+	if (!clock_gettime(igt_clock = CLOCK_MONOTONIC, ts))
+		return 0;
+error:
+	fprintf(stderr, "Warning: Could not read monotonic time: %s\n",
+		strerror(errno));
+
+	return -errno;
+}
+
+struct watchdogs
+{
+	int *fds;
+	size_t num_dogs;
+} watchdogs;
+
+static void close_watchdogs(struct settings *settings)
+{
+	size_t i;
+
+	if (settings && settings->log_level >= LOG_LEVEL_VERBOSE)
+		printf("Closing watchdogs\n");
+
+	for (i = 0; i < watchdogs.num_dogs; i++) {
+		write(watchdogs.fds[i], "V", 1);
+		close(watchdogs.fds[i]);
+	}
+}
+
+static void close_watchdogs_atexit()
+{
+	close_watchdogs(NULL);
+}
+
+static void init_watchdogs(struct settings *settings)
+{
+	int i;
+	char name[32];
+	int fd;
+
+	memset(&watchdogs, 0, sizeof(watchdogs));
+
+	if (!settings->use_watchdog || settings->inactivity_timeout <= 0)
+		return;
+
+	if (settings->log_level >= LOG_LEVEL_VERBOSE) {
+		printf("Initializing watchdogs\n");
+	}
+
+	atexit(close_watchdogs_atexit);
+
+	for (i = 0; ; i++) {
+		snprintf(name, 32, "/dev/watchdog%d", i);
+		if ((fd = open(name, O_RDWR | O_CLOEXEC)) < 0)
+			break;
+
+		watchdogs.num_dogs++;
+		watchdogs.fds = realloc(watchdogs.fds, watchdogs.num_dogs * sizeof(int));
+		watchdogs.fds[i] = fd;
+
+		if (settings->log_level >= LOG_LEVEL_VERBOSE)
+			printf(" %s\n", name);
+	}
+}
+
+static int watchdogs_set_timeout(int timeout)
+{
+	size_t i;
+	int orig_timeout = timeout;
+
+	for (i = 0; i < watchdogs.num_dogs; i++) {
+		if (ioctl(watchdogs.fds[i], WDIOC_SETTIMEOUT, &timeout)) {
+			write(watchdogs.fds[i], "V", 1);
+			close(watchdogs.fds[i]);
+			watchdogs.fds[i] = -1;
+			continue;
+		}
+
+		if (timeout < orig_timeout) {
+			/*
+			 * Timeout of this caliber refused. We want to
+			 * use the same timeout for all devices.
+			 */
+			return watchdogs_set_timeout(timeout);
+		}
+	}
+
+	return timeout;
+}
+
+static void ping_watchdogs()
+{
+	size_t i;
+
+	for (i = 0; i < watchdogs.num_dogs; i++) {
+		ioctl(watchdogs.fds[i], WDIOC_KEEPALIVE, 0);
+	}
+}
+
+static void prune_subtest(struct job_list_entry *entry, char *subtest)
+{
+	char *excl;
+
+	/*
+	 * Subtest pruning is done by adding exclusion strings to the
+	 * subtest list. The last matching item on the subtest
+	 * selection command line flag decides whether to run a
+	 * subtest, see igt_core.c for details.  If the list is empty,
+	 * the expected subtest set is unknown, so we need to add '*'
+	 * first so we can start excluding.
+	 */
+
+	if (entry->subtest_count == 0) {
+		entry->subtest_count++;
+		entry->subtests = realloc(entry->subtests, entry->subtest_count * sizeof(*entry->subtests));
+		entry->subtests[0] = strdup("*");
+	}
+
+	excl = malloc(strlen(subtest) + 2);
+	excl[0] = '!';
+	strcpy(excl + 1, subtest);
+
+	entry->subtest_count++;
+	entry->subtests = realloc(entry->subtests, entry->subtest_count * sizeof(*entry->subtests));
+	entry->subtests[entry->subtest_count - 1] = excl;
+}
+
+static bool prune_from_journal(struct job_list_entry *entry, int fd)
+{
+	char *subtest;
+	FILE *f;
+	bool any_pruned = false;
+
+	/*
+	 * Each journal line is a subtest that has been started, or
+	 * the line 'exit:$exitcode (time)', or 'timeout:$exitcode (time)'.
+	 */
+
+	f = fdopen(fd, "r");
+	if (!f)
+		return false;
+
+	while (fscanf(f, "%ms", &subtest) == 1) {
+		if (subtest[0] == '\0') {
+			/* EOF */
+			free(subtest);
+			break;
+		}
+
+		if (!strncmp(subtest, "exit:", 5)) {
+			/* Fully done. Mark that by making the binary name invalid. */
+			fscanf(f, " (%*fs)");
+			entry->binary[0] = '\0';
+			free(subtest);
+			continue;
+		}
+
+		if (!strncmp(subtest, "timeout:", 8)) {
+			fscanf(f, " (%*fs)");
+			free(subtest);
+			continue;
+		}
+
+		prune_subtest(entry, subtest);
+
+		free(subtest);
+		any_pruned = true;
+	}
+
+	fclose(f);
+	return any_pruned;
+}
+
+static char *filenames[] = {
+	"journal.txt",
+	"out.txt",
+	"err.txt",
+	"dmesg.txt",
+};
+
+static int open_at_end(int dirfd, char *name)
+{
+	int fd = openat(dirfd, name, O_RDWR | O_CREAT | O_CLOEXEC, 0666);
+	char last;
+
+	if (fd >= 0) {
+		if (lseek(fd, -1, SEEK_END) >= 0 &&
+		    read(fd, &last, 1) == 1 &&
+		    last != '\n') {
+			write(fd, "\n", 1);
+		}
+		lseek(fd, 0, SEEK_END);
+	}
+
+	return fd;
+}
+
+static int open_for_reading(int dirfd, char *name)
+{
+	return openat(dirfd, name, O_RDONLY);
+}
+
+bool open_output_files(int dirfd, int *fds, bool write)
+{
+	int i;
+	int (*openfunc)(int, char*) = write ? open_at_end : open_for_reading;
+
+	for (i = 0; i < _F_LAST; i++) {
+		if ((fds[i] = openfunc(dirfd, filenames[i])) < 0) {
+			while (--i >= 0)
+				close(fds[i]);
+			return false;
+		}
+	}
+
+	return true;
+}
+
+void close_outputs(int *fds)
+{
+	int i;
+
+	for (i = 0; i < _F_LAST; i++) {
+		close(fds[i]);
+	}
+}
+
+static void dump_dmesg(int kmsgfd, int outfd)
+{
+	/*
+	 * Write kernel messages to the log file until we reach
+	 * 'now'. Unfortunately, /dev/kmsg doesn't support seeking to
+	 * -1 from SEEK_END so we need to use a second fd to read a
+	 * message to match against, or stop when we reach EAGAIN.
+	 */
+
+	int comparefd = open("/dev/kmsg", O_RDONLY | O_NONBLOCK);
+	unsigned flags;
+	unsigned long long seq, cmpseq, usec;
+	char cont;
+	char buf[256];
+	ssize_t r;
+
+	if (comparefd < 0)
+		return;
+
+	if (fcntl(kmsgfd, F_SETFL, O_NONBLOCK))
+		return;
+
+	while (1) {
+		if (comparefd >= 0) {
+			r = read(comparefd, buf, sizeof(buf) - 1);
+			if (r < 0) {
+				if (errno != EAGAIN && errno != EPIPE)
+					return;
+			} else {
+				buf[r] = '\0';
+				if (sscanf(buf, "%u,%llu,%llu,%c;",
+					   &flags, &cmpseq, &usec, &cont) == 4) {
+					close(comparefd);
+					comparefd = -1;
+				}
+			}
+		}
+
+		r = read(kmsgfd, buf, sizeof(buf));
+		if (r <= 0) {
+			if (errno == EPIPE)
+				continue;
+
+			/*
+			 * If EAGAIN, we're done. If some other error,
+			 * we can't do anything anyway.
+			 */
+			return;
+		}
+
+		write(outfd, buf, r);
+
+		if (comparefd < 0 && sscanf(buf, "%u,%llu,%llu,%c;",
+					    &flags, &seq, &usec, &cont) == 4) {
+			if (seq >= cmpseq)
+				return;
+		}
+	}
+}
+
+static bool kill_child(bool use_sigkill,
+		       struct settings *settings,
+		       pid_t child)
+{
+	int sig = use_sigkill ? SIGKILL : SIGTERM;
+
+	/*
+	 * Send the signal to the child directly, and to the child's
+	 * process group.
+	 */
+	kill(-child, sig);
+	if (kill(child, sig) && errno == ESRCH) {
+		fprintf(stderr, "Child process does not exist. This shouldn't happen.\n");
+		return false;
+	}
+
+	return true;
+}
+
+static char starting_subtest[] = "Starting subtest: ";
+static size_t starting_len = sizeof(starting_subtest) - 1;
+static char subtest_result_beg[] = "Subtest ";
+static size_t subtest_result_len = sizeof(subtest_result_beg) - 1;
+/*
+ * Returns:
+ *  >0 - Success
+ *  =0 - Failure executing
+ *  <0 - Timeout happened, need to recreate from journal
+ */
+static int monitor_output(pid_t child,
+			   int outfd, int errfd, int kmsgfd, int sigfd,
+			   int *outputs,
+			   struct settings *settings)
+{
+	fd_set set;
+	char buf[256];
+	char *outbuf = NULL;
+	size_t outbufsize = 0;
+	char current_subtest[256] = {};
+	struct signalfd_siginfo siginfo;
+	ssize_t s;
+	int n, status;
+	int nfds = outfd;
+	int timeout = settings->inactivity_timeout;
+	int timeout_intervals = 1, intervals_left;
+	int wd_extra = 10;
+	int killed = 0; /* 1 = sigterm sent, 2 = sigkill sent */
+	struct timespec time_beg, time_end;
+	bool aborting = false;
+
+	gettime(&time_beg);
+
+	if (errfd > nfds)
+		nfds = errfd;
+	if (kmsgfd > nfds)
+		nfds = kmsgfd;
+	if (sigfd > nfds)
+		nfds = sigfd;
+	nfds++;
+
+	if (timeout > 0) {
+		/*
+		 * Use original timeout plus some leeway. If we're still
+		 * alive, we want to kill the test process instead of cutting
+		 * power.
+		 */
+		int wd_timeout = watchdogs_set_timeout(timeout + wd_extra);
+
+		if (wd_timeout < timeout + wd_extra) {
+			/* Watchdog timeout smaller, so ping it more often */
+			if (wd_timeout - wd_extra < 0)
+				wd_extra = wd_timeout / 2;
+			timeout_intervals = timeout / (wd_timeout - wd_extra);
+			intervals_left = timeout_intervals;
+			timeout /= timeout_intervals;
+
+			if (settings->log_level >= LOG_LEVEL_VERBOSE) {
+				printf("Watchdog doesn't support the timeout we requested (shortened to %d seconds).\n"
+				       "Using %d intervals of %d seconds.\n",
+				       wd_timeout, timeout_intervals, timeout);
+			}
+		}
+	}
+
+	while (outfd >= 0 || errfd >= 0 || sigfd >= 0) {
+		struct timeval tv = { .tv_sec = timeout };
+
+		FD_ZERO(&set);
+		if (outfd >= 0)
+			FD_SET(outfd, &set);
+		if (errfd >= 0)
+			FD_SET(errfd, &set);
+		if (kmsgfd >= 0)
+			FD_SET(kmsgfd, &set);
+		if (sigfd >= 0)
+			FD_SET(sigfd, &set);
+
+		n = select(nfds, &set, NULL, NULL, timeout == 0 ? NULL : &tv);
+		if (n < 0) {
+			/* TODO */
+			return 0;
+		}
+
+		if (n == 0) {
+			intervals_left--;
+			if (intervals_left) {
+				continue;
+			}
+
+			ping_watchdogs();
+
+			switch (killed) {
+			case 0:
+				if (settings->log_level >= LOG_LEVEL_NORMAL) {
+					printf("Timeout. Killing the current test with SIGTERM.\n");
+				}
+
+				if (!kill_child(false, settings, child))
+					return 0;
+				killed = 1;
+
+				/*
+				 * Now continue the loop and let the
+				 * dying child be handled normally.
+				 */
+				timeout = 2; /* Timeout for waiting selected by fair dice roll. */
+				watchdogs_set_timeout(20);
+				intervals_left = timeout_intervals = 1;
+				break;
+			case 1:
+				if (settings->log_level >= LOG_LEVEL_NORMAL) {
+					printf("Timeout. Killing the current test with SIGKILL.\n");
+				}
+
+				if (!kill_child(true, settings, child))
+					return 0;
+
+				killed = 2;
+				break;
+			case 2:
+				/* Nothing that can be done, really. Let's tell the caller we want to abort. */
+				if (settings->log_level >= LOG_LEVEL_NORMAL) {
+					fprintf(stderr, "Child refuses to die. Aborting.\n");
+				}
+				close_watchdogs(settings);
+				free(outbuf);
+				close(outfd);
+				close(errfd);
+				close(kmsgfd);
+				close(sigfd);
+				return 0;
+			}
+
+			continue;
+		}
+
+		intervals_left = timeout_intervals;
+		ping_watchdogs();
+
+		if (outfd >= 0 && FD_ISSET(outfd, &set)) {
+			char *newline;
+
+			s = read(outfd, buf, sizeof(buf));
+			if (s <= 0) {
+				if (s < 0) {
+					fprintf(stderr, "Error reading test's stdout: %s\n",
+						strerror(errno));
+				}
+
+				close(outfd);
+				outfd = -1;
+				goto out_end;
+			}
+
+			write(outputs[_F_OUT], buf, s);
+			if (settings->sync) {
+				fdatasync(outputs[_F_OUT]);
+			}
+
+			outbuf = realloc(outbuf, outbufsize + s);
+			memcpy(outbuf + outbufsize, buf, s);
+			outbufsize += s;
+
+			while ((newline = memchr(outbuf, '\n', outbufsize)) != NULL) {
+				size_t linelen = newline - outbuf + 1;
+
+				if (linelen > starting_len &&
+				    !memcmp(outbuf, starting_subtest, starting_len)) {
+					write(outputs[_F_JOURNAL], outbuf + starting_len,
+					      linelen - starting_len);
+					memcpy(current_subtest, outbuf + starting_len,
+					       linelen - starting_len);
+					current_subtest[linelen - starting_len] = '\0';
+
+					if (settings->log_level >= LOG_LEVEL_VERBOSE) {
+						fwrite(outbuf, 1, linelen, stdout);
+					}
+				}
+				if (linelen > subtest_result_len &&
+				    !memcmp(outbuf, subtest_result_beg, subtest_result_len)) {
+					char *delim = memchr(outbuf, ':', linelen);
+
+					if (delim != NULL) {
+						size_t subtestlen = delim - outbuf - subtest_result_len;
+						if (memcmp(current_subtest, outbuf + subtest_result_len,
+							   subtestlen)) {
+							/* Result for a test that didn't ever start */
+							write(outputs[_F_JOURNAL],
+							      outbuf + subtest_result_len,
+							      subtestlen);
+							write(outputs[_F_JOURNAL], "\n", 1);
+							if (settings->sync) {
+								fdatasync(outputs[_F_JOURNAL]);
+							}
+							current_subtest[0] = '\0';
+						}
+
+						if (settings->log_level >= LOG_LEVEL_VERBOSE) {
+							fwrite(outbuf, 1, linelen, stdout);
+						}
+					}
+				}
+
+				memmove(outbuf, newline + 1, outbufsize - linelen);
+				outbufsize -= linelen;
+			}
+		}
+	out_end:
+
+		if (errfd >= 0 && FD_ISSET(errfd, &set)) {
+			s = read(errfd, buf, sizeof(buf));
+			if (s <= 0) {
+				if (s < 0) {
+					fprintf(stderr, "Error reading test's stderr: %s\n",
+						strerror(errno));
+				}
+				close(errfd);
+				errfd = -1;
+			} else {
+				write(outputs[_F_ERR], buf, s);
+				if (settings->sync) {
+					fdatasync(outputs[_F_ERR]);
+				}
+			}
+		}
+
+		if (kmsgfd >= 0 && FD_ISSET(kmsgfd, &set)) {
+			s = read(kmsgfd, buf, sizeof(buf));
+			if (s < 0) {
+				if (errno != EPIPE) {
+					fprintf(stderr, "Error reading from kmsg, stopping monitoring: %s\n",
+						strerror(errno));
+					close(kmsgfd);
+					kmsgfd = -1;
+				}
+			} else {
+				write(outputs[_F_DMESG], buf, s);
+				if (settings->sync) {
+					fdatasync(outputs[_F_DMESG]);
+				}
+			}
+		}
+
+		if (sigfd >= 0 && FD_ISSET(sigfd, &set)) {
+			double time;
+
+			s = read(sigfd, &siginfo, sizeof(siginfo));
+			if (s < 0) {
+				fprintf(stderr, "Error reading from signalfd: %s\n",
+					strerror(errno));
+				continue;
+			} else if (siginfo.ssi_signo == SIGCHLD) {
+				if (child != waitpid(child, &status, WNOHANG)) {
+					fprintf(stderr, "Failed to reap child\n");
+					status = 9999;
+				} else if (WIFEXITED(status)) {
+					status = WEXITSTATUS(status);
+					if (status >= 128) {
+						status = 128 - status;
+					}
+				} else if (WIFSIGNALED(status)) {
+					status = -WTERMSIG(status);
+				} else {
+					status = 9999;
+				}
+			} else {
+				/* We're dying, so we're taking them with us */
+				if (settings->log_level >= LOG_LEVEL_NORMAL)
+					printf("Abort requested, terminating children\n");
+
+				if (!kill_child(false, settings, child))
+					return 0;
+				aborting = true;
+				timeout = 2;
+				killed = 1;
+
+				continue;
+			}
+
+			gettime(&time_end);
+
+			time = time_elapsed(&time_beg, &time_end);
+			if (time < 0.0)
+				time = 0.0;
+
+			if (!aborting) {
+				dprintf(outputs[_F_JOURNAL], "%s:%d (%.3fs)\n",
+					killed ? "timeout" : "exit",
+					status, time);
+				if (settings->sync) {
+					fdatasync(outputs[_F_JOURNAL]);
+				}
+			}
+
+			close(sigfd);
+			sigfd = -1;
+			child = 0;
+		}
+	}
+
+	dump_dmesg(kmsgfd, outputs[_F_DMESG]);
+	if (settings->sync)
+		fdatasync(outputs[_F_DMESG]);
+
+	free(outbuf);
+	close(outfd);
+	close(errfd);
+	close(kmsgfd);
+	close(sigfd);
+
+	if (aborting)
+		return 0;
+
+	return killed ? -1 : 1;
+}
+
+static void execute_test_process(int outfd, int errfd,
+				 struct settings *settings,
+				 struct job_list_entry *entry)
+{
+	char *argv[4] = {};
+	size_t rootlen;
+
+	dup2(outfd, STDOUT_FILENO);
+	dup2(errfd, STDERR_FILENO);
+
+	setpgid(0, 0);
+
+	rootlen = strlen(settings->test_root);
+	argv[0] = malloc(rootlen + strlen(entry->binary) + 2);
+	strcpy(argv[0], settings->test_root);
+	argv[0][rootlen] = '/';
+	strcpy(argv[0] + rootlen + 1, entry->binary);
+
+	if (entry->subtest_count) {
+		size_t argsize;
+		size_t i;
+
+		argv[1] = "--run-subtest";
+		argsize = strlen(entry->subtests[0]);
+		argv[2] = malloc(argsize + 1);
+		strcpy(argv[2], entry->subtests[0]);
+
+		for (i = 1; i < entry->subtest_count; i++) {
+			char *sub = entry->subtests[i];
+			size_t sublen = strlen(sub);
+
+			argv[2] = realloc(argv[2], argsize + sublen + 2);
+			argv[2][argsize] = ',';
+			strcpy(argv[2] + argsize + 1, sub);
+			argsize += sublen + 1;
+		}
+	}
+
+	execv(argv[0], argv);
+	fprintf(stderr, "Cannot execute %s\n", argv[0]);
+	exit(79);
+}
+
+static int digits(size_t num)
+{
+	int ret = 0;
+	while (num) {
+		num /= 10;
+		ret++;
+	}
+
+	if (ret == 0) ret++;
+	return ret;
+}
+
+/*
+ * Returns:
+ *  >0 - Success
+ *  =0 - Failure executing
+ *  <0 - Timeout happened, need to recreate from journal
+ */
+static int execute_entry(size_t idx,
+			  size_t total,
+			  struct settings *settings,
+			  struct job_list_entry *entry,
+			  int testdirfd, int resdirfd)
+{
+	int dirfd;
+	int outputs[_F_LAST];
+	int kmsgfd;
+	int sigfd;
+	sigset_t mask;
+	int outpipe[2] = { -1, -1 };
+	int errpipe[2] = { -1, -1 };
+	char name[32];
+	pid_t child;
+	int result;
+
+	snprintf(name, 32, "%zd", idx);
+	mkdirat(resdirfd, name, 0777);
+	if ((dirfd = openat(resdirfd, name, O_DIRECTORY | O_RDONLY | O_CLOEXEC)) < 0) {
+		fprintf(stderr, "Error accessing individual test result directory\n");
+		return 0;
+	}
+
+	if (!open_output_files(dirfd, outputs, true)) {
+		close(dirfd);
+		fprintf(stderr, "Error opening output files\n");
+		return 0;
+	}
+
+	if (settings->sync) {
+		fsync(dirfd);
+		fsync(resdirfd);
+	}
+
+	if (pipe(outpipe) || pipe(errpipe)) {
+		close_outputs(outputs);
+		close(dirfd);
+		close(outpipe[0]);
+		close(outpipe[1]);
+		close(errpipe[0]);
+		close(errpipe[1]);
+		fprintf(stderr, "Error creating pipes: %s\n", strerror(errno));
+		return 0;
+	}
+
+	if ((kmsgfd = open("/dev/kmsg", O_RDONLY | O_CLOEXEC)) < 0) {
+		fprintf(stderr, "Warning: Cannot open /dev/kmsg\n");
+	} else {
+		/* TODO: Checking of abort conditions in pre-execute dmesg */
+		lseek(kmsgfd, 0, SEEK_END);
+	}
+
+	sigemptyset(&mask);
+	sigaddset(&mask, SIGCHLD);
+	sigaddset(&mask, SIGINT);
+	sigaddset(&mask, SIGTERM);
+	sigaddset(&mask, SIGQUIT);
+	sigprocmask(SIG_BLOCK, &mask, NULL);
+	sigfd = signalfd(-1, &mask, O_CLOEXEC);
+
+	if (sigfd < 0) {
+		/* TODO: Handle better */
+		fprintf(stderr, "Cannot monitor child process with signalfd\n");
+		close(outpipe[0]);
+		close(errpipe[0]);
+		close(outpipe[1]);
+		close(errpipe[1]);
+		close(kmsgfd);
+		close_outputs(outputs);
+		close(dirfd);
+		return 0;
+	}
+
+	if (settings->log_level >= LOG_LEVEL_NORMAL) {
+		int width = digits(total);
+		printf("[%0*zd/%0*zd] %s", width, idx + 1, width, total, entry->binary);
+		if (entry->subtest_count > 0) {
+			size_t i;
+			char *delim = "";
+
+			printf(" (");
+			for (i = 0; i < entry->subtest_count; i++) {
+				printf("%s%s", delim, entry->subtests[i]);
+				delim = ", ";
+			}
+			printf(")");
+		}
+		printf("\n");
+	}
+
+	if ((child = fork())) {
+		int outfd = outpipe[0];
+		int errfd = errpipe[0];
+		close(outpipe[1]);
+		close(errpipe[1]);
+
+		result = monitor_output(child, outfd, errfd, kmsgfd, sigfd,
+					outputs, settings);
+	} else {
+		int outfd = outpipe[1];
+		int errfd = errpipe[1];
+		close(outpipe[0]);
+		close(errpipe[0]);
+
+		sigprocmask(SIG_UNBLOCK, &mask, NULL);
+
+		setenv("IGT_SENTINEL_ON_STDERR", "1", 1);
+
+		execute_test_process(outfd, errfd, settings, entry);
+	}
+
+	close(outpipe[1]);
+	close(errpipe[1]);
+	close(kmsgfd);
+	close_outputs(outputs);
+	close(dirfd);
+
+	return result;
+}
+
+static bool clear_test_result_directory(int dirfd)
+{
+	if (unlinkat(dirfd, "out.txt", 0) ||
+	    unlinkat(dirfd, "err.txt", 0) ||
+	    unlinkat(dirfd, "dmesg.txt", 0) ||
+	    unlinkat(dirfd, "journal.txt", 0)) {
+		if (errno != ENOENT) {
+			fprintf(stderr, "Error clearing test result directories: %s\n",
+				strerror(errno));
+			return false;
+		}
+	}
+
+	return true;
+}
+
+static bool clear_old_results(char *path)
+{
+	int dirfd;
+	size_t i;
+
+	if ((dirfd = open(path, O_DIRECTORY | O_RDONLY)) < 0) {
+		if (errno == ENOENT) {
+			/* Successfully cleared if it doesn't even exist */
+			return true;
+		}
+
+		fprintf(stderr, "Error clearing old results: %s\n", strerror(errno));
+		return false;
+	}
+
+	if (unlinkat(dirfd, "uname.txt", 0) && errno != ENOENT) {
+		close(dirfd);
+		fprintf(stderr, "Error clearing old results: %s\n", strerror(errno));
+		return false;
+	}
+
+	for (i = 0; true; i++) {
+		char name[32];
+		int resdirfd;
+
+		snprintf(name, 32, "%zd", i);
+		if ((resdirfd = openat(dirfd, name, O_DIRECTORY | O_RDONLY)) < 0)
+			break;
+
+		if (!clear_test_result_directory(resdirfd)) {
+			close(resdirfd);
+			close(dirfd);
+			return false;
+		}
+		close(resdirfd);
+		unlinkat(dirfd, name, AT_REMOVEDIR);
+	}
+
+	close(dirfd);
+
+	return true;
+}
+
+static bool initialize_execute_from_resume(struct execute_state *state,
+					   struct settings *orig_settings,
+					   struct job_list *orig_job_list)
+{
+	struct settings settings;
+	struct job_list list;
+	struct job_list_entry *entry;
+	int dirfd, resdirfd, fd, i;
+
+	init_settings(&settings);
+	init_job_list(&list);
+
+	if ((dirfd = open(orig_settings->results_path, O_DIRECTORY | O_RDONLY)) < 0)
+		return false;
+
+	if (!read_settings(&settings, dirfd) ||
+	    !read_job_list(&list, dirfd)) {
+		close(dirfd);
+		return false;
+	}
+
+	for (i = list.size; i >= 0; i--) {
+		char name[32];
+
+		snprintf(name, 32, "%d", i);
+		if ((resdirfd = openat(dirfd, name, O_DIRECTORY | O_RDONLY)) >= 0) {
+			break;
+		}
+	}
+
+	if (i < 0) {
+		/* Nothing has been executed yet, state is fine as is */
+		goto success;
+	}
+
+	entry = &list.entries[i];
+	state->next = i;
+	if ((fd = openat(resdirfd, filenames[_F_JOURNAL], O_RDONLY)) >= 0) {
+		if (!prune_from_journal(entry, fd)) {
+			/*
+			 * The test does not have subtests, or
+			 * incompleted before the first subtest
+			 * began. Either way, not suitable to
+			 * re-run.
+			 */
+			state->next = i + 1;
+		} else if (entry->binary[0] == '\0') {
+			/* This test is fully completed */
+			state->next = i + 1;
+		}
+
+		close(fd);
+	}
+
+ success:
+	free_settings(orig_settings);
+	free_job_list(orig_job_list);
+	*orig_settings = settings;
+	*orig_job_list = list;
+	close(resdirfd);
+	close(dirfd);
+
+	return true;
+}
+
+bool initialize_execute_state(struct execute_state *state,
+			      struct settings *settings,
+			      struct job_list *job_list)
+{
+	memset(state, 0, sizeof(*state));
+
+	if (!validate_settings(settings))
+		return false;
+
+	if (settings->overwrite &&
+	    !clear_old_results(settings->results_path))
+		return false;
+
+	if (!serialize_settings(settings) ||
+	    !serialize_job_list(job_list, settings))
+		return initialize_execute_from_resume(state, settings, job_list);
+
+	return true;
+}
+
+bool execute(struct execute_state *state,
+	     struct settings *settings,
+	     struct job_list *job_list)
+{
+	struct utsname unamebuf;
+	int resdirfd, testdirfd, unamefd;
+
+	if ((resdirfd = open(settings->results_path, O_DIRECTORY | O_RDONLY)) < 0) {
+		/* Initialize state should have done this */
+		fprintf(stderr, "Error: Failure opening results path %s\n",
+			settings->results_path);
+		return false;
+	}
+
+	if ((testdirfd = open(settings->test_root, O_DIRECTORY | O_RDONLY)) < 0) {
+		fprintf(stderr, "Error: Failure opening test root %s\n",
+			settings->test_root);
+		close(resdirfd);
+		return false;
+	}
+
+	/* TODO: On resume, don't rewrite, verify that content matches current instead */
+	if ((unamefd = openat(resdirfd, "uname.txt", O_CREAT | O_WRONLY | O_TRUNC, 0666)) < 0) {
+		fprintf(stderr, "Error: Failure creating opening uname.txt: %s\n",
+			strerror(errno));
+		close(testdirfd);
+		close(resdirfd);
+		return false;
+	}
+
+	init_watchdogs(settings);
+
+	if (!uname(&unamebuf)) {
+		dprintf(unamefd, "%s %s %s %s %s\n",
+			unamebuf.sysname,
+			unamebuf.nodename,
+			unamebuf.release,
+			unamebuf.version,
+			unamebuf.machine);
+	} else {
+		dprintf(unamefd, "uname() failed\n");
+	}
+	close(unamefd);
+
+	for (; state->next < job_list->size;
+	     state->next++) {
+		int result = execute_entry(state->next,
+					   job_list->size,
+					   settings,
+					   &job_list->entries[state->next],
+					   testdirfd, resdirfd);
+		if (result <= 0) {
+			close(testdirfd);
+			close(resdirfd);
+			close_watchdogs(settings);
+			if (result < 0) {
+				memset(state, 0, sizeof(*state));
+				initialize_execute_from_resume(state, settings, job_list);
+				return execute(state, settings, job_list);
+			}
+			return false;
+		}
+	}
+
+	close(testdirfd);
+	close(resdirfd);
+	close_watchdogs(settings);
+	return true;
+}
diff --git a/runner/executor.h b/runner/executor.h
new file mode 100644
index 00000000..bdda983a
--- /dev/null
+++ b/runner/executor.h
@@ -0,0 +1,42 @@
+#ifndef RUNNER_EXECUTOR_H
+#define RUNNER_EXECUTOR_H
+
+#include "job_list.h"
+#include "settings.h"
+
+struct execute_state
+{
+	size_t next;
+};
+
+enum {
+	_F_JOURNAL,
+	_F_OUT,
+	_F_ERR,
+	_F_DMESG,
+	_F_LAST,
+};
+
+bool open_output_files(int dirfd, int *fds, bool write);
+void close_outputs(int *fds);
+
+/*
+ * Initialize execute_state object to a state where it's ready to
+ * resume. Will validate the settings and serialize both settings and
+ * the job_list into the result directory if they are not yet written
+ * there.
+ *
+ * If executions have already started, will communicate the resume
+ * point by setting the appropriate next_job_list_entry value, and
+ * possibly modifying the job_list object.
+ */
+bool initialize_execute_state(struct execute_state *state,
+			      struct settings *settings,
+			      struct job_list *job_list);
+
+bool execute(struct execute_state *state,
+	     struct settings *settings,
+	     struct job_list *job_list);
+
+
+#endif
diff --git a/runner/job_list.c b/runner/job_list.c
new file mode 100644
index 00000000..ec942481
--- /dev/null
+++ b/runner/job_list.c
@@ -0,0 +1,492 @@
+#include <errno.h>
+#include <fcntl.h>
+#include <linux/limits.h>
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+#include <sys/stat.h>
+#include <sys/types.h>
+#include <unistd.h>
+
+#include "job_list.h"
+#include "igt_core.h"
+
+static bool matches_any(const char *needle, struct regex_list *haystacks)
+{
+	size_t i;
+
+	for (i = 0; i < haystacks->size; i++) {
+		if (regexec(haystacks->regexes[i], needle,
+			    (size_t)0, NULL, 0) == 0) {
+			return true;
+		}
+	}
+
+	return false;
+}
+
+static void add_job_list_entry(struct job_list *job_list,
+			       char *binary,
+			       char **subtests,
+			       size_t subtest_count)
+{
+	struct job_list_entry *entry;
+
+	job_list->size++;
+	job_list->entries = realloc(job_list->entries, job_list->size * sizeof(*job_list->entries));
+	entry = &job_list->entries[job_list->size - 1];
+
+	entry->binary = binary;
+	entry->subtests = subtests;
+	entry->subtest_count = subtest_count;
+}
+
+static void add_subtests(struct job_list *job_list, struct settings *settings,
+			 char *binary,
+			 struct regex_list *include, struct regex_list *exclude)
+{
+	FILE *p;
+	char cmd[256] = {};
+	char *subtestname;
+	char **subtests = NULL;
+	size_t num_subtests = 0;
+	size_t rootlen = strlen(settings->test_root);
+	size_t binarylen = strlen(binary);
+	int idx, s;
+
+	if (rootlen + binarylen + strlen(" --list-subtests") + 1 > 256) {
+		/* This shouldn't happen */
+		fprintf(stderr, "Path to binary too long, ignoring: %s/%s\n",
+			settings->test_root, binary);
+		return;
+	}
+
+	strcpy(cmd, settings->test_root);
+	idx = rootlen;
+	cmd[idx++] = '/';
+	strcpy(cmd + idx, binary);
+	idx += binarylen;
+	strcpy(cmd + idx, " --list-subtests");
+
+	p = popen(cmd, "r");
+	if (!p) {
+		fprintf(stderr, "popen failed when executing %s: %s\n",
+			cmd,
+			strerror(errno));
+		return;
+	}
+
+	while (fscanf(p, "%ms", &subtestname) == 1) {
+		if (exclude && exclude->size && matches_any(subtestname, exclude)) {
+			free(subtestname);
+			continue;
+		}
+
+		if (include && include->size && !matches_any(subtestname, include)) {
+			free(subtestname);
+			continue;
+		}
+
+		if (settings->multiple_mode) {
+			num_subtests++;
+			subtests = realloc(subtests, num_subtests * sizeof(*subtests));
+			subtests[num_subtests - 1] = strdup(subtestname);
+		} else {
+			subtests = malloc(sizeof(*subtests));
+			*subtests = strdup(subtestname);
+			add_job_list_entry(job_list, strdup(binary), subtests, 1);
+			subtests = NULL;
+		}
+
+		free(subtestname);
+	}
+
+	if (num_subtests) {
+		add_job_list_entry(job_list, strdup(binary), subtests, num_subtests);
+	}
+
+	s = pclose(p);
+	if (s == 0) {
+		return;
+	} else if (s == -1) {
+		fprintf(stderr, "popen error when executing %s: %s\n", binary, strerror(errno));
+	} else if (WIFEXITED(s)) {
+		if (WEXITSTATUS(s) == IGT_EXIT_INVALID) {
+			/* No subtests on this one */
+			if (exclude && exclude->size && matches_any(binary, exclude)) {
+				return;
+			}
+			if (!include || !include->size || matches_any(binary, include)) {
+				add_job_list_entry(job_list, strdup(binary), NULL, 0);
+				return;
+			}
+		}
+	} else {
+		fprintf(stderr, "Test binary %s died unexpectedly\n", binary);
+	}
+}
+
+static bool filtered_job_list(struct job_list *job_list,
+			      struct settings *settings,
+			      int fd)
+{
+	FILE *f;
+	char buf[128];
+
+	if (job_list->entries != NULL) {
+		fprintf(stderr, "Job list not cleared, this shouldn't happen\n");
+		exit(1);
+	}
+
+	f = fdopen(fd, "r");
+
+	while (fscanf(f, "%127s", buf) == 1) {
+		if (!strcmp(buf, "TESTLIST") || !(strcmp(buf, "END")))
+			continue;
+
+		/*
+		 * If the binary name matches exclude filters, no
+		 * subtests are added.
+		 */
+		if (settings->exclude_regexes.size && matches_any(buf, &settings->exclude_regexes))
+			continue;
+
+		/*
+		 * If the binary name matches include filters (or include filters not present),
+		 * all subtests except those matching exclude filters are added.
+		 */
+		if (!settings->include_regexes.size || matches_any(buf, &settings->include_regexes)) {
+			if (settings->multiple_mode && !settings->exclude_regexes.size)
+				/*
+				 * Optimization; we know that all
+				 * subtests will be included, so we
+				 * get to omit executing
+				 * --list-subtests.
+				 */
+				add_job_list_entry(job_list, strdup(buf), NULL, 0);
+			else
+				add_subtests(job_list, settings, buf,
+					     NULL, &settings->exclude_regexes);
+			continue;
+		}
+
+		/*
+		 * Binary name doesn't match exclude or include filters.
+		 */
+		add_subtests(job_list, settings, buf,
+			     &settings->include_regexes,
+			     &settings->exclude_regexes);
+	}
+
+	return job_list->size != 0;
+}
+
+static bool job_list_from_test_list(struct job_list *job_list,
+				    struct settings *settings)
+{
+	FILE *f;
+	ssize_t read;
+	char *line = NULL;
+	size_t line_len = 0;
+	struct job_list_entry entry = {};
+	bool any = false;
+
+	if ((f = fopen(settings->test_list, "r")) == NULL) {
+		fprintf(stderr, "Cannot open test list file %s\n", settings->test_list);
+		return false;
+	}
+
+	while ((read = getline(&line, &line_len, f))) {
+		char *binary;
+		char *delim;
+
+		if (read < 0) {
+			if (errno == EINTR)
+				continue;
+			else
+				break;
+		}
+
+		/* # starts a comment */
+		if ((delim = strchr(line, '#')) != NULL) {
+			*delim = '\0';
+		}
+
+		if (sscanf(line, "igt@%ms", &binary) == 1) {
+			if ((delim = strchr(binary, '@')) != NULL) {
+				*delim++ = '\0';
+			}
+
+			if (!settings->multiple_mode) {
+				char **subtests = NULL;
+				if (delim) {
+					subtests = malloc(sizeof(char*));
+					subtests[0] = strdup(delim);
+				}
+				add_job_list_entry(job_list, strdup(binary),
+						   subtests, (size_t)(subtests != NULL));
+				any = true;
+				free(binary);
+				binary = NULL;
+				continue;
+			}
+
+			/*
+			 * If the currently built entry has the same
+			 * binary, add a subtest. Otherwise submit
+			 * what's already built and start a new one.
+			 */
+			if (entry.binary && !strcmp(entry.binary, binary)) {
+				if (!delim) {
+					/* ... except we didn't get a subtest */
+					fprintf(stderr,
+						"Error: Unexpected test without subtests "
+						"after same test had subtests\n");
+					free(binary);
+					fclose(f);
+					return false;
+				}
+				entry.subtest_count++;
+				entry.subtests = realloc(entry.subtests,
+							 entry.subtest_count *
+							 sizeof(*entry.subtests));
+				entry.subtests[entry.subtest_count - 1] = strdup(delim);
+				free(binary);
+				binary = NULL;
+				continue;
+			}
+
+			if (entry.binary) {
+				add_job_list_entry(job_list, entry.binary, entry.subtests, entry.subtest_count);
+				any = true;
+			}
+
+			memset(&entry, 0, sizeof(entry));
+			entry.binary = strdup(binary);
+			if (delim) {
+				entry.subtests = malloc(sizeof(*entry.subtests));
+				entry.subtests[0] = strdup(delim);
+				entry.subtest_count = 1;
+			}
+
+			free(binary);
+			binary = NULL;
+		}
+	}
+
+	if (entry.binary) {
+		add_job_list_entry(job_list, entry.binary, entry.subtests, entry.subtest_count);
+		any = true;
+	}
+
+	free(line);
+	fclose(f);
+	return any;
+}
+
+void init_job_list(struct job_list *job_list)
+{
+	memset(job_list, 0, sizeof(*job_list));
+}
+
+void free_job_list(struct job_list *job_list)
+{
+	int i, k;
+
+	for (i = 0; i < job_list->size; i++) {
+		struct job_list_entry *entry = &job_list->entries[i];
+
+		free(entry->binary);
+		for (k = 0; k < entry->subtest_count; k++) {
+			free(entry->subtests[k]);
+		}
+		free(entry->subtests);
+	}
+	free(job_list->entries);
+	init_job_list(job_list);
+}
+
+bool create_job_list(struct job_list *job_list,
+		     struct settings *settings)
+{
+	int dirfd, fd;
+	bool result;
+
+	if (!settings->test_root) {
+		fprintf(stderr, "No test root set; this shouldn't happen\n");
+		return false;
+	}
+
+	free_job_list(job_list);
+
+	dirfd = open(settings->test_root, O_DIRECTORY | O_RDONLY);
+	if (dirfd < 0) {
+		fprintf(stderr, "Test directory %s cannot be opened\n", settings->test_root);
+		return false;
+	}
+
+	fd = openat(dirfd, "test-list.txt", O_RDONLY);
+	if (fd < 0) {
+		fprintf(stderr, "Cannot open %s/test-list.txt\n", settings->test_root);
+		close(dirfd);
+		return false;
+	}
+
+	/*
+	 * If a test_list is given (not to be confused with
+	 * test-list.txt), we use it directly without making tests
+	 * list their subtests. If include/exclude filters are given
+	 * we filter them directly from the test_list.
+	 */
+	if (settings->test_list)
+		result = job_list_from_test_list(job_list, settings);
+	else
+		result = filtered_job_list(job_list, settings, fd);
+
+	close(fd);
+	close(dirfd);
+
+	return result;
+}
+
+static char joblist_filename[] = "joblist.txt";
+bool serialize_job_list(struct job_list *job_list, struct settings *settings)
+{
+	int dirfd, fd;
+	size_t i, k;
+	FILE *f;
+
+	if (!settings->results_path) {
+		fprintf(stderr, "No results-path set; this shouldn't happen\n");
+		return false;
+	}
+
+	if ((dirfd = open(settings->results_path, O_DIRECTORY | O_RDONLY)) < 0) {
+		mkdir(settings->results_path, 0777);
+		if ((dirfd = open(settings->results_path, O_DIRECTORY | O_RDONLY)) < 0) {
+			fprintf(stderr, "Creating results-path failed\n");
+			return false;
+		}
+	}
+
+	if ((fd = openat(dirfd, joblist_filename, O_RDONLY)) >= 0) {
+		close(fd);
+
+		if (!settings->overwrite) {
+			/* Serialization data already exists, not overwriting. */
+			close(fd);
+			close(dirfd);
+			return false;
+		}
+
+		if (unlinkat(dirfd, joblist_filename, 0) != 0) {
+			fprintf(stderr, "Error overwriting old job list\n");
+			close(dirfd);
+			return false;
+		}
+	}
+
+	if ((fd = openat(dirfd, joblist_filename, O_CREAT | O_EXCL | O_WRONLY, 0666)) < 0) {
+		fprintf(stderr, "Creating job list serialization file failed\n");
+		return false;
+	}
+
+	f = fdopen(fd, "w");
+	if (!f) {
+		close(fd);
+		close(dirfd);
+		return false;
+	}
+
+	for (i = 0; i < job_list->size; i++) {
+		struct job_list_entry *entry = &job_list->entries[i];
+		fprintf(f, "%s", entry->binary);
+
+		if (entry->subtest_count) {
+			char *delim = "";
+
+			fprintf(f, " ");
+
+			for (k = 0; k < entry->subtest_count; k++) {
+				fprintf(f, "%s%s", delim, entry->subtests[k]);
+				delim = ",";
+			}
+		}
+
+		fprintf(f, "\n");
+	}
+
+	if (settings->sync) {
+		fsync(fd);
+		fsync(dirfd);
+	}
+
+	fclose(f);
+	close(dirfd);
+	return true;
+}
+
+bool read_job_list(struct job_list *job_list, int dirfd)
+{
+	int fd;
+	FILE *f;
+	ssize_t read;
+	char *line = NULL;
+	size_t line_len = 0;
+
+	free_job_list(job_list);
+
+	if ((fd = openat(dirfd, joblist_filename, O_RDONLY)) < 0)
+		return false;
+
+	f = fdopen(fd, "r");
+	if (!f) {
+		close(fd);
+		return false;
+	}
+
+	while ((read = getline(&line, &line_len, f))) {
+		char *binary, *sublist, *comma;
+		char **subtests = NULL;
+		size_t num_subtests = 0, len;
+
+		if (read < 0) {
+			if (errno == EINTR)
+				continue;
+			else
+				break;
+		}
+
+		len = strlen(line);
+		if (len > 0 && line[len - 1] == '\n')
+			line[len - 1] = '\0';
+
+		sublist = strchr(line, ' ');
+		if (!sublist) {
+			add_job_list_entry(job_list, strdup(line), NULL, 0);
+			continue;
+		}
+
+		*sublist++ = '\0';
+		binary = strdup(line);
+
+		do {
+			comma = strchr(sublist, ',');
+			if (comma) {
+				*comma++ = '\0';
+			}
+
+			++num_subtests;
+			subtests = realloc(subtests, num_subtests * sizeof(*subtests));
+			subtests[num_subtests - 1] = strdup(sublist);
+			sublist = comma;
+		} while (comma != NULL);
+
+		add_job_list_entry(job_list, binary, subtests, num_subtests);
+	}
+
+	free(line);
+	fclose(f);
+
+	return true;
+}
diff --git a/runner/job_list.h b/runner/job_list.h
new file mode 100644
index 00000000..c726ab09
--- /dev/null
+++ b/runner/job_list.h
@@ -0,0 +1,37 @@
+#ifndef RUNNER_JOB_LIST_H
+#define RUNNER_JOB_LIST_H
+
+#include <stdbool.h>
+
+#include "settings.h"
+
+struct job_list_entry {
+	char *binary;
+	char **subtests;
+	/*
+	 * 0 = all, or test has no subtests.
+	 *
+	 * If the original job_list was to run all subtests of a
+	 * binary and such a run was incomplete, resuming from the
+	 * execution journal will fill the subtest array with already
+	 * started subtests prepended with '!' so the test binary will
+	 * not run them. subtest_count will still reflect the size of
+	 * the above array.
+	 */
+	size_t subtest_count;
+};
+
+struct job_list
+{
+	struct job_list_entry *entries;
+	size_t size;
+};
+
+void init_job_list(struct job_list *job_list);
+void free_job_list(struct job_list *job_list);
+bool create_job_list(struct job_list *job_list, struct settings *settings);
+
+bool serialize_job_list(struct job_list *job_list, struct settings *settings);
+bool read_job_list(struct job_list *job_list, int dirfd);
+
+#endif
diff --git a/runner/meson.build b/runner/meson.build
new file mode 100644
index 00000000..773b4f3c
--- /dev/null
+++ b/runner/meson.build
@@ -0,0 +1,24 @@
+jsonc = dependency('json-c', required: true)
+
+runnerlib_sources = [ 'settings.c',
+		      'job_list.c',
+		      'executor.c',
+		      'resultgen.c',
+		    ]
+
+runner_sources = [ 'runner.c',
+		 ]
+
+runnerlib = static_library('igt_runner', runnerlib_sources,
+			   include_directories : inc,
+			   dependencies : jsonc)
+
+runner = executable('runner', runner_sources,
+		    link_with : runnerlib,
+		    install : true,
+		    install_dir : bindir,
+		    dependencies : igt_deps)
+
+results = executable('results', 'results.c',
+		     link_with : runnerlib,
+)
diff --git a/runner/resultgen.c b/runner/resultgen.c
new file mode 100644
index 00000000..978c07be
--- /dev/null
+++ b/runner/resultgen.c
@@ -0,0 +1,864 @@
+#include <ctype.h>
+#include <fcntl.h>
+#include <stdio.h>
+#include <string.h>
+#include <sys/mman.h>
+#include <sys/stat.h>
+#include <sys/types.h>
+#include <unistd.h>
+
+#include <json.h>
+
+#include "resultgen.h"
+#include "settings.h"
+#include "executor.h"
+#include "igt_core.h"
+
+static char subtest_result_beg[] = "Subtest ";
+static size_t subtest_result_len = sizeof(subtest_result_beg) - 1;
+static char starting_subtest[] = "Starting subtest: ";
+static size_t starting_len = sizeof(starting_subtest) - 1;
+static char starting_subtest_dmesg[] = ": starting subtest ";
+static size_t starting_subtest_dmesg_len = sizeof(starting_subtest_dmesg) - 1;
+
+struct subtests
+{
+	char **names;
+	size_t size;
+};
+
+static char *find_line(char *haystack, char *needle, size_t needle_size, char *end)
+{
+	char *line = haystack;
+
+	while (line < end) {
+		char *line_end = memchr(line, '\n', end - line);
+
+		if (end - line < needle_size)
+			return NULL;
+		if (!memcmp(line, needle, needle_size))
+			return line;
+		if (line_end == NULL)
+			return NULL;
+		line = line_end + 1;
+	}
+
+	return NULL;
+}
+
+static char *find_either_line(char *haystack,
+			      char *needle1, size_t needle1_size,
+			      char *needle2, size_t needle2_size,
+			      char *end)
+{
+	char *line = haystack;
+
+	while (line < end) {
+		char *line_end = memchr(line, '\n', end - line);
+		size_t linelen = line_end != NULL ? line_end - line : end - line;
+		if ((linelen >= needle1_size && !memcmp(line, needle1, needle1_size)) ||
+		    (linelen >= needle2_size && !memcmp(line, needle2, needle2_size)))
+			return line;
+
+		if (line_end == NULL)
+			return NULL;
+
+		line = line_end + 1;
+	}
+
+	return NULL;
+}
+
+static char *next_line(char *line, char *bufend)
+{
+	char *ret;
+
+	if (!line)
+		return NULL;
+
+	ret = memchr(line, '\n', bufend - line);
+	if (ret)
+		ret++;
+
+	return ret;
+}
+
+static size_t count_lines(char *buf, char *bufend)
+{
+	size_t ret = 0;
+	while ((buf = next_line(buf, bufend)) != NULL)
+		ret++;
+
+	return ret;
+}
+
+static char *lowercase(char *str)
+{
+	char *ret = malloc(strlen(str) + 1);
+	char *q = ret;
+
+	while (*str) {
+		if (isspace(*str))
+			break;
+
+		*q++ = tolower(*str++);
+	}
+	*q = '\0';
+
+	return ret;
+}
+
+static void append_line(char **buf, size_t *buflen, char *line)
+{
+	size_t linelen = strlen(line);
+
+	*buf = realloc(*buf, *buflen + linelen + 1);
+	strcpy(*buf + *buflen, line);
+	*buflen += linelen;
+}
+
+static char *gen_igt_name(char *binary, char *subtest)
+{
+	static char namebuf[256];
+
+	char *lc_binary = lowercase(binary);
+	char *lc_subtest = NULL;
+
+	if (!subtest) {
+		snprintf(namebuf, 256, "igt@%s", lc_binary);
+		free(lc_binary);
+		return namebuf;
+	}
+
+	lc_subtest = lowercase(subtest);
+
+	snprintf(namebuf, 256, "igt@%s@%s", lc_binary, lc_subtest);
+
+	free(lc_binary);
+	free(lc_subtest);
+	return namebuf;
+}
+
+static struct {
+	char *output_str;
+	char *result_str;
+} resultmap[] = {
+	{ "SUCCESS", "pass" },
+	{ "SKIP", "skip" },
+	{ "FAIL", "fail" },
+	{ "CRASH", "crash" },
+	{ "TIMEOUT", "timeout" },
+};
+static void parse_result_string(char *resultstring, size_t len, char **result, double *time)
+{
+	size_t i;
+	size_t wordlen = 0;
+
+	while (wordlen < len) {
+		if (isspace(resultstring[wordlen]))
+			break;
+		wordlen++;
+	}
+
+	for (i = 0; i < (sizeof(resultmap) / sizeof(resultmap[0])); i++) {
+		if (!strncmp(resultstring, resultmap[i].output_str, wordlen)) {
+			*result = resultmap[i].result_str;
+			break;
+		}
+	}
+
+	/* If we don't find a result line, the test didn't finish. */
+	if (!*result)
+		*result = "incomplete";
+
+	wordlen++;
+	if (wordlen < len && resultstring[wordlen] == '(') {
+		wordlen++;
+		char *dup = malloc(len - wordlen);
+		memcpy(dup, resultstring + wordlen, len - wordlen);
+		dup[len - wordlen] = '\0';
+		*time = strtod(dup, NULL);
+
+		free(dup);
+	}
+}
+
+static void parse_subtest_result(char *subtest, char **result, double *time, char *buf, char *bufend)
+{
+	char *line;
+	char *line_end;
+	char *resultstring;
+	size_t linelen;
+	size_t subtestlen = strlen(subtest);
+
+	*result = NULL;
+	*time = 0.0;
+
+	if (!buf) return;
+
+	line = find_line(buf, subtest_result_beg, subtest_result_len, bufend);
+	if (!line) {
+		*result = "incomplete";
+		return;
+	}
+
+	line_end = memchr(line, '\n', bufend - line);
+	linelen = line_end != NULL ? line_end - line : bufend - line;
+
+	if (subtest_result_len + subtestlen + 2 > linelen ||
+	    strncmp(line + subtest_result_len, subtest, subtestlen))
+		return parse_subtest_result(subtest, result, time, line + linelen, bufend);
+
+	resultstring = line + subtest_result_len + subtestlen + 2;
+	parse_result_string(resultstring, linelen - (resultstring - line), result, time);
+}
+
+static struct json_object *get_or_create_json_object(struct json_object *base,
+						     char *key)
+{
+	struct json_object *ret;
+
+	if (json_object_object_get_ex(base, key, &ret))
+		return ret;
+
+	ret = json_object_new_object();
+	json_object_object_add(base, key, ret);
+
+	return ret;
+}
+
+static void add_or_override_result(struct json_object *obj, char *from, char *to)
+{
+	struct json_object *current;
+	const char *oldresult;
+
+	if (!json_object_object_get_ex(obj, "result", &current)) {
+		json_object_object_add(obj, "result",
+				       json_object_new_string(to));
+		return;
+	}
+
+	oldresult = json_object_get_string(current);
+	if (from != NULL && !strcmp(oldresult, from))
+		json_object_object_add(obj, "result",
+				       json_object_new_string(to));
+}
+
+static void add_runtime(struct json_object *obj, double time)
+{
+	double oldtime;
+	struct json_object *timeobj = get_or_create_json_object(obj, "time");
+	struct json_object *oldend;
+
+	json_object_object_add(timeobj, "__type__",
+			       json_object_new_string("TimeAttribute"));
+	json_object_object_add(timeobj, "start",
+			       json_object_new_double(0.0));
+
+	if (!json_object_object_get_ex(timeobj, "end", &oldend)) {
+		json_object_object_add(timeobj, "end",
+				       json_object_new_double(time));
+		return;
+	}
+
+	oldtime = json_object_get_double(oldend);
+	time += oldtime;
+	json_object_object_add(timeobj, "end",
+			       json_object_new_double(time));
+}
+
+static char versionstring[] = "IGT-Version: ";
+static size_t versionlen = sizeof(versionstring) - 1;
+static bool fill_from_output(int fd, char *binary, bool is_stdout,
+			     struct subtests *subtests,
+			     struct json_object *tests)
+{
+	char *buf, *bufend;
+	struct stat statbuf;
+	char *igt_name = NULL;
+	char *igt_version = NULL;
+	size_t igt_version_len = 0;
+	struct json_object *current_test = NULL;
+	size_t i;
+
+	if (fstat(fd, &statbuf))
+		return false;
+
+	buf = mmap(NULL, statbuf.st_size, PROT_READ, MAP_SHARED, fd, 0);
+	if (!buf)
+		return false;
+
+	bufend = buf + statbuf.st_size;
+
+	if (is_stdout) {
+		char *newline;
+
+		igt_version = find_line(buf, versionstring, versionlen, bufend);
+		if (igt_version) {
+			newline = memchr(igt_version, '\n', bufend - igt_version);
+			igt_version_len = newline - igt_version;
+		} else {
+			igt_version = NULL;
+		}
+	}
+
+	if (subtests->size == 0) {
+		/* No subtests */
+		igt_name = gen_igt_name(binary, NULL);
+		current_test = get_or_create_json_object(tests, igt_name);
+
+		json_object_object_add(current_test, is_stdout ? "out" : "err",
+				       json_object_new_string_len(buf, statbuf.st_size));
+		if (!is_stdout && count_lines(buf, buf + statbuf.st_size) > 2) {
+			add_or_override_result(current_test, "pass", "warn");
+		}
+		if (igt_version) {
+			json_object_object_add(current_test, "igt-version",
+					       json_object_new_string_len(igt_version,
+									  igt_version_len));
+		}
+		return true;
+	}
+
+	for (i = 0; i < subtests->size; i++) {
+		char *begin_subtest;
+		char *subtest_result;
+		char *beg, *end, *startline;
+		int begin_len;
+		int result_len;
+
+		igt_name = gen_igt_name(binary, subtests->names[i]);
+		current_test = get_or_create_json_object(tests, igt_name);
+
+		begin_len = asprintf(&begin_subtest, "%s%s\n", starting_subtest, subtests->names[i]);
+		result_len = asprintf(&subtest_result, "%s%s: ", subtest_result_beg, subtests->names[i]);
+
+		if (begin_len < 0 || result_len < 0) {
+			fprintf(stderr, "Failure generating strings\n");
+			return false;
+		}
+
+		beg = find_line(buf, begin_subtest, begin_len, bufend);
+		end = find_line(buf, subtest_result, result_len, bufend);
+		startline = beg;
+
+		free(begin_subtest);
+		free(subtest_result);
+
+		if (beg == NULL && end == NULL) {
+			/* No output at all */
+			beg = bufend;
+			end = bufend;
+		}
+
+		if (beg == NULL) {
+			/*
+			 * Subtest didn't start, probably skipped from
+			 * fixture already. Find the previous subtest
+			 * output.
+			 */
+			beg = find_either_line(buf,
+					       starting_subtest, starting_len,
+					       subtest_result_beg, subtest_result_len,
+					       end);
+			if (beg == NULL) {
+				beg = buf;
+			} else {
+				while (beg < end) {
+					char *result;
+					beg = next_line(beg, bufend);
+					result = find_either_line(beg,
+								  starting_subtest, starting_len,
+								  subtest_result_beg, subtest_result_len,
+								  bufend);
+					if (result == NULL || result > end) {
+						break;
+					}
+					beg = result;
+				}
+			}
+		} else {
+			/* Include the output after the previous subtest output */
+			char *prevtest = find_either_line(buf,
+							  starting_subtest, starting_len,
+							  subtest_result_beg, subtest_result_len,
+							  beg);
+
+			if (prevtest == NULL) {
+				beg = buf;
+			} else {
+				while (prevtest != NULL && prevtest < beg) {
+					char *result;
+					prevtest = next_line(prevtest, beg);
+					result = find_either_line(prevtest,
+								  starting_subtest, starting_len,
+								  subtest_result_beg, subtest_result_len,
+								  beg);
+					if (result == NULL) {
+						beg = prevtest;
+						break;
+					}
+					prevtest = result;
+				}
+			}
+		}
+
+		if (end == NULL) {
+			/* Incomplete result. Find the next starting subtest or result. */
+			end = next_line(startline, bufend);
+			if (end != NULL) {
+				end = find_either_line(end,
+						       starting_subtest, starting_len,
+						       subtest_result_beg, subtest_result_len,
+						       bufend);
+			}
+			if (end == NULL) {
+				end = bufend;
+			}
+		} else {
+			/* Stretch onwards until the next subtest begins or ends */
+			char *nexttest = next_line(end, bufend);
+			if (nexttest != NULL) {
+				nexttest = find_either_line(nexttest,
+							    starting_subtest, starting_len,
+							    subtest_result_beg, subtest_result_len,
+							    bufend);
+			}
+			if (nexttest != NULL) {
+				end = nexttest;
+			} else {
+				end = bufend;
+			}
+		}
+
+		json_object_object_add(current_test, is_stdout ? "out" : "err",
+				       json_object_new_string_len(beg, end - beg));
+
+		if (is_stdout) {
+			char *result;
+			double time;
+			parse_subtest_result(subtests->names[i], &result, &time, beg, end);
+			add_or_override_result(current_test, NULL, result);
+			add_runtime(current_test, time);
+			if (igt_version) {
+				json_object_object_add(current_test, "igt-version",
+						       json_object_new_string_len(igt_version,
+										  igt_version_len));
+			}
+		} else if (count_lines(startline, end) > 2) {
+			add_or_override_result(current_test, "pass", "warn");
+		}
+	}
+
+	return true;
+}
+
+/*
+ * This regexp controls the kmsg handling. All kernel log records that
+ * have log level of warning or higher convert the result to
+ * dmesg-warn/dmesg-fail unless they match this regexp.
+ */
+
+#define _ "|"
+static const char igt_dmesg_whitelist[] =
+	"ACPI: button: The lid device is not compliant to SW_LID" _
+	"ACPI: .*: Unable to dock!" _
+	"IRQ [0-9]+: no longer affine to CPU[0-9]+" _
+	"IRQ fixup: irq [0-9]+ move in progress, old vector [0-9]+" _
+	/* i915 tests set module options, expected message */
+	"Setting dangerous option [a-z_]+ - tainting kernel" _
+	/* Raw printk() call, uses default log level (warn) */
+	"Suspending console\\(s\\) \\(use no_console_suspend to debug\\)" _
+	"atkbd serio[0-9]+: Failed to (deactivate|enable) keyboard on isa[0-9]+/serio[0-9]+" _
+	"cache: parent cpu[0-9]+ should not be sleeping" _
+	"hpet[0-9]+: lost [0-9]+ rtc interrupts" _
+	/* i915 selftests terminate normally with ENODEV from the
+	 * module load after the testing finishes, which produces this
+	 * message.
+	 */
+	"i915: probe of [0-9:.]+ failed with error -25" _
+	/* swiotbl warns even when asked not to */
+	"mock: DMA: Out of SW-IOMMU space for [0-9]+ bytes" _
+	"usb usb[0-9]+: root hub lost power or was reset"
+	;
+#undef _
+
+static regex_t re;
+
+static bool init_regex_whitelist()
+{
+	int status = -1;
+
+	if (status == -1) {
+		if (regcomp(&re, igt_dmesg_whitelist, REG_EXTENDED | REG_NOSUB) != 0) {
+			fprintf(stderr, "Cannot compile dmesg whitelist regexp\n");
+			status = 0;
+			return false;
+		}
+
+		status = 1;
+	}
+
+	return status;
+}
+
+static bool fill_from_dmesg(int fd, char *binary,
+			    struct subtests *subtests,
+			    struct json_object *tests)
+{
+	char *line = NULL, *warnings = NULL, *dmesg = NULL;
+	size_t linelen = 0, warningslen = 0, dmesglen = 0;
+	struct json_object *current_test = NULL;
+	FILE *f = fdopen(fd, "r");
+	char *igt_name = NULL;
+	ssize_t read;
+	size_t i;
+
+	if (!f) {
+		return false;
+	}
+
+	if (init_regex_whitelist() != 1) {
+		fclose(f);
+		return false;
+	}
+
+	while ((read = getline(&line, &linelen, f)) > 0) {
+		char formatted[256];
+		unsigned flags;
+		unsigned long long seq, ts_usec;
+		char continuation;
+		char *message, *subtest;
+		int s;
+
+		s = sscanf(line, "%u,%llu,%llu,%c;", &flags, &seq, &ts_usec, &continuation);
+
+		if (s != 4) {
+			/*
+			 * Machine readable key/value pairs begin with
+			 * a space. We ignore them.
+			 */
+			if (line[0] != ' ') {
+				fprintf(stderr, "Cannot parse kmsg record: %s\n", line);
+			}
+
+			continue;
+		}
+
+		message = strchr(line, ';');
+		if (!message) {
+			fprintf(stderr, "No ; found, this shouldn't happen\n");
+			return false;
+		}
+		message++;
+		snprintf(formatted, sizeof(formatted), "<%u> [%llu.%06llu] %s",
+			 flags & 0x07, ts_usec / 1000000, ts_usec % 1000000, message);
+
+		if ((subtest = strstr(message, starting_subtest_dmesg)) != NULL) {
+			if (current_test != NULL) {
+				/* Done with the previous subtest, file up */
+				json_object_object_add(current_test, "dmesg",
+						       json_object_new_string_len(dmesg, dmesglen));
+				if (warnings) {
+					json_object_object_add(current_test, "dmesg-warnings",
+							       json_object_new_string_len(warnings, warningslen));
+					add_or_override_result(current_test, "pass", "dmesg-warn");
+					add_or_override_result(current_test, "fail", "dmesg-fail");
+					add_or_override_result(current_test, "warn", "dmesg-warn");
+				}
+				free(dmesg);
+				free(warnings);
+				dmesg = warnings = NULL;
+				dmesglen = warningslen = 0;
+			}
+
+			subtest += starting_subtest_dmesg_len;
+			igt_name = gen_igt_name(binary, subtest);
+			current_test = get_or_create_json_object(tests, igt_name);
+		}
+
+		if ((flags & 0x07) <= 4 && continuation != 'c' &&
+		    regexec(&re, message, (size_t)0, NULL, 0) == REG_NOMATCH) {
+			append_line(&warnings, &warningslen, formatted);
+		}
+		append_line(&dmesg, &dmesglen, formatted);
+	}
+
+	if (current_test != NULL) {
+		json_object_object_add(current_test, "dmesg",
+				       json_object_new_string_len(dmesg, dmesglen));
+		if (warnings) {
+			json_object_object_add(current_test, "dmesg-warnings",
+					       json_object_new_string_len(warnings, warningslen));
+			add_or_override_result(current_test, "pass", "dmesg-warn");
+			add_or_override_result(current_test, "fail", "dmesg-fail");
+			add_or_override_result(current_test, "warn", "dmesg-warn");
+		}
+	} else {
+		/*
+		 * Didn't get any subtest messages at all. If there
+		 * are subtests, add all of the dmesg gotten to all of
+		 * them.
+		 */
+		for (i = 0; i < subtests->size; i++) {
+			igt_name = gen_igt_name(binary, subtests->names[i]);
+			current_test = get_or_create_json_object(tests, igt_name);
+			json_object_object_add(current_test, "dmesg",
+					       json_object_new_string_len(dmesg, dmesglen));
+			/*
+			 * Don't bother with warnings, any subtests
+			 * there are would have skip as their result
+			 * anyway.
+			 */
+		}
+
+		if (i == 0) {
+			/* There were no subtests */
+			igt_name = gen_igt_name(binary, NULL);
+			current_test = get_or_create_json_object(tests, igt_name);
+			json_object_object_add(current_test, "dmesg",
+					       json_object_new_string_len(dmesg, dmesglen));
+			if (warnings) {
+				json_object_object_add(current_test, "dmesg-warnings",
+						       json_object_new_string_len(warnings, warningslen));
+				add_or_override_result(current_test, "pass", "dmesg-warn");
+				add_or_override_result(current_test, "fail", "dmesg-fail");
+				add_or_override_result(current_test, "warn", "dmesg-warn");
+			}
+		}
+	}
+
+	/*
+	 * Add an empty string as the dmesg of all subtests that
+	 * didn't get any dmesg yet.
+	 */
+	for (i = 0; i < subtests->size; i++) {
+		igt_name = gen_igt_name(binary, subtests->names[i]);
+		current_test = get_or_create_json_object(tests, igt_name);
+		if (!json_object_object_get_ex(current_test, "dmesg", NULL)) {
+			json_object_object_add(current_test, "dmesg",
+					       json_object_new_string(""));
+		}
+	}
+
+	free(dmesg);
+	free(warnings);
+	fclose(f);
+	return true;
+}
+
+static char *result_from_exitcode(int exitcode)
+{
+	switch (exitcode) {
+	case IGT_EXIT_TIMEOUT:
+		return "timeout";
+	case IGT_EXIT_SKIP:
+		return "skip";
+	case IGT_EXIT_SUCCESS:
+		return "pass";
+	case IGT_EXIT_INVALID:
+		return "notrun";
+	default:
+		return "fail";
+	}
+}
+
+static void add_subtest(struct subtests *subtests, char *subtest)
+{
+	size_t len = strlen(subtest);
+
+	if (len == 0)
+		return;
+
+	if (subtest[len - 1] == '\n')
+		subtest[len - 1] = '\0';
+
+	subtests->size++;
+	subtests->names = realloc(subtests->names, sizeof(*subtests->names) * subtests->size);
+	subtests->names[subtests->size - 1] = subtest;
+}
+
+static bool fill_from_journal(int fd, char *binary,
+				  struct subtests *subtests,
+				  struct json_object *tests)
+{
+	FILE *f = fdopen(fd, "r");
+	char *line = NULL;
+	size_t linelen = 0;
+	ssize_t read;
+	char exitline[] = "exit:";
+	size_t exitlen = sizeof(exitline) - 1;
+	char timeoutline[] = "timeout:";
+	size_t timeoutlen = sizeof(timeoutline) - 1;
+	int exitcode = 10000;
+
+	while ((read = getline(&line, &linelen, f)) > 0) {
+		if (read >= exitlen && !memcmp(line, exitline, exitlen)) {
+			char *p = strchr(line, '(');
+			char *igt_name = gen_igt_name(binary, NULL);
+			double time = 0.0;
+			struct json_object *obj = get_or_create_json_object(tests, igt_name);
+
+			exitcode = atoi(line + exitlen);
+
+			if (p) {
+				time = strtod(p + 1, NULL);
+			}
+
+			add_runtime(obj, time);
+		} else if (read >= timeoutlen && !memcmp(line, timeoutline, timeoutlen)) {
+			if (subtests->size) {
+				char *last_subtest = subtests->names[subtests->size - 1];
+				char *igt_name = gen_igt_name(binary, last_subtest);
+				char *p = strchr(line, '(');
+				double time = 0.0;
+				struct json_object *obj = get_or_create_json_object(tests, igt_name);
+
+				json_object_object_add(obj, "result",
+						       json_object_new_string("timeout"));
+
+				if (p) {
+					time = strtod(p + 1, NULL);
+				}
+
+				add_runtime(obj, time);
+
+				igt_name = gen_igt_name(binary, NULL);
+				obj = get_or_create_json_object(tests, igt_name);
+				add_runtime(obj, time);
+			}
+		} else {
+			add_subtest(subtests, strdup(line));
+		}
+	}
+
+	if (subtests->size == 0 && exitcode != 10000) {
+		char *igt_name = gen_igt_name(binary, NULL);
+		struct json_object *obj = get_or_create_json_object(tests, igt_name);
+		char *result = result_from_exitcode(exitcode);
+		json_object_object_add(obj, "result",
+				       json_object_new_string(result));
+	}
+
+	free(line);
+	return true;
+}
+
+static bool parse_test_directory(int dirfd, char *binary, struct json_object *tests)
+{
+	int fds[_F_LAST];
+	struct subtests subtests = {};
+
+	if (!open_output_files(dirfd, fds, false)) {
+		fprintf(stderr, "Error opening output files\n");
+		return false;
+	}
+
+	/* fill_from_journal also fills the subtests struct */
+	if (!fill_from_journal(fds[_F_JOURNAL], binary, &subtests, tests)) {
+		fprintf(stderr, "Error reading from journal\n");
+		return false;
+	}
+
+	/* Order of these is important */
+	if (!fill_from_output(fds[_F_OUT], binary, true, &subtests, tests) ||
+	    !fill_from_output(fds[_F_ERR], binary, false, &subtests, tests) ||
+	    !fill_from_dmesg(fds[_F_DMESG], binary, &subtests, tests)) {
+		fprintf(stderr, "Error parsing output files\n");
+		return false;
+	}
+
+	close_outputs(fds);
+
+	return true;
+}
+
+bool generate_results(int dirfd)
+{
+	struct settings settings;
+	struct job_list job_list;
+	struct json_object *obj, *tests;
+	int resultsfd, testdirfd, unamefd;
+	const char *json_string;
+	size_t i;
+
+	init_settings(&settings);
+	init_job_list(&job_list);
+
+	if (!read_settings(&settings, dirfd)) {
+		fprintf(stderr, "resultgen: Cannot parse settings\n");
+		return false;
+	}
+
+	if (!read_job_list(&job_list, dirfd)) {
+		fprintf(stderr, "resultgen: Cannot parse job list\n");
+		return false;
+	}
+
+	/* TODO: settings.overwrite */
+	if ((resultsfd = openat(dirfd, "results.json", O_WRONLY | O_CREAT | O_TRUNC, 0666)) < 0) {
+		fprintf(stderr, "resultgen: Cannot create results file\n");
+		return false;
+	}
+
+	obj = json_object_new_object();
+	json_object_object_add(obj, "__type__", json_object_new_string("TestrunResult"));
+	json_object_object_add(obj, "results_version", json_object_new_int(9));
+	json_object_object_add(obj, "name",
+			       settings.name ?
+			       json_object_new_string(settings.name) :
+			       json_object_new_string(""));
+
+	if ((unamefd = openat(dirfd, "uname.txt", O_RDONLY)) >= 0) {
+		char buf[128];
+		ssize_t r = read(unamefd, buf, 128);
+
+		if (r > 0 && buf[r - 1] == '\n')
+			r--;
+
+		json_object_object_add(obj, "uname",
+				       json_object_new_string_len(buf, r));
+		close(unamefd);
+	}
+
+	/* lspci */
+	/* results_version */
+	/* glxinfo */
+	/* wglinfo */
+	/* clinfo */
+	/* options */
+	/* time_elapsed */
+	/* totals */
+
+	tests = json_object_new_object();
+	json_object_object_add(obj, "tests", tests);
+
+	for (i = 0; i < job_list.size; i++) {
+		char name[16];
+
+		snprintf(name, 16, "%zd", i);
+		if ((testdirfd = openat(dirfd, name, O_DIRECTORY | O_RDONLY)) < 0) {
+			fprintf(stderr, "Warning: Cannot open result directory %s\n", name);
+			break;
+		}
+
+		if (!parse_test_directory(testdirfd, job_list.entries[i].binary, tests)) {
+			close(resultsfd);
+			return false;
+		}
+	}
+
+	json_string = json_object_to_json_string_ext(obj, JSON_C_TO_STRING_PRETTY);
+	write(resultsfd, json_string, strlen(json_string));
+	return true;
+}
+
+bool generate_results_path(char *resultspath)
+{
+	int dirfd = open(resultspath, O_DIRECTORY | O_RDONLY);
+
+	if (dirfd < 0)
+		return false;
+
+	return generate_results(dirfd);
+}
diff --git a/runner/resultgen.h b/runner/resultgen.h
new file mode 100644
index 00000000..83a0876b
--- /dev/null
+++ b/runner/resultgen.h
@@ -0,0 +1,9 @@
+#ifndef RUNNER_RESULTGEN_H
+#define RUNNER_RESULTGEN_H
+
+#include <stdbool.h>
+
+bool generate_results(int dirfd);
+bool generate_results_path(char *resultspath);
+
+#endif
diff --git a/runner/results.c b/runner/results.c
new file mode 100644
index 00000000..3eb7cb15
--- /dev/null
+++ b/runner/results.c
@@ -0,0 +1,26 @@
+#include <fcntl.h>
+#include <stdio.h>
+#include <stdlib.h>
+#include <sys/stat.h>
+#include <sys/types.h>
+
+#include "resultgen.h"
+
+int main(int argc, char **argv)
+{
+	int dirfd;
+
+	if (argc < 2)
+		exit(1);
+
+	dirfd = open(argv[1], O_DIRECTORY | O_RDONLY);
+	if (dirfd < 0)
+		exit(1);
+
+	if (generate_results(dirfd)) {
+		printf("Results generated\n");
+		exit(0);
+	}
+
+	exit(1);
+}
diff --git a/runner/runner.c b/runner/runner.c
new file mode 100644
index 00000000..b685786a
--- /dev/null
+++ b/runner/runner.c
@@ -0,0 +1,40 @@
+#include <stdio.h>
+#include <string.h>
+
+#include "settings.h"
+#include "job_list.h"
+#include "executor.h"
+#include "resultgen.h"
+
+int main(int argc, char **argv)
+{
+	struct settings settings;
+	struct job_list job_list;
+	struct execute_state state;
+
+	init_settings(&settings);
+	init_job_list(&job_list);
+
+	if (!parse_options(argc, argv, &settings)) {
+		return 1;
+	}
+
+	if (!create_job_list(&job_list, &settings)) {
+		return 1;
+	}
+
+	if (!initialize_execute_state(&state, &settings, &job_list)) {
+		return 1;
+	}
+
+	if (!execute(&state, &settings, &job_list)) {
+		return 1;
+	}
+
+	if (!generate_results_path(settings.results_path)) {
+		return 1;
+	}
+
+	printf("Done.\n");
+	return 0;
+}
diff --git a/runner/settings.c b/runner/settings.c
new file mode 100644
index 00000000..50e4ea10
--- /dev/null
+++ b/runner/settings.c
@@ -0,0 +1,506 @@
+#include "settings.h"
+
+#include <fcntl.h>
+#include <getopt.h>
+#include <libgen.h>
+#include <limits.h>
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+#include <sys/types.h>
+#include <sys/stat.h>
+#include <unistd.h>
+
+enum {
+	OPT_ABORT_ON_ERROR,
+	OPT_TEST_LIST,
+	OPT_IGNORE_MISSING,
+	OPT_HELP = 'h',
+	OPT_NAME = 'n',
+	OPT_DRY_RUN = 'd',
+	OPT_INCLUDE = 't',
+	OPT_EXCLUDE = 'x',
+	OPT_SYNC = 's',
+	OPT_LOG_LEVEL = 'l',
+	OPT_OVERWRITE = 'o',
+	OPT_MULTIPLE = 'm',
+	OPT_TIMEOUT = 'c',
+	OPT_WATCHDOG = 'g',
+};
+
+static struct {
+	int level;
+	const char *name;
+} log_levels[] = {
+	{ LOG_LEVEL_NORMAL, "normal" },
+	{ LOG_LEVEL_QUIET, "quiet" },
+	{ LOG_LEVEL_VERBOSE, "verbose" },
+	{ 0, 0 },
+};
+
+static bool set_log_level(struct settings* settings, const char *level)
+{
+	int c;
+
+	for (c = 0; log_levels[c].name != NULL; c++) {
+		if (!strcmp(level, log_levels[c].name)) {
+			settings->log_level = log_levels[c].level;
+			return true;
+		}
+	}
+
+	return false;
+}
+
+static const char *usage_str =
+	"usage: runner [options] [test_root] results-path\n\n"
+	"Options:\n"
+	" Piglit compatible:\n"
+	"  -h, --help            Show this help message and exit\n"
+	"  -n <test name>, --name <test name>\n"
+	"                        Name of this test run\n"
+	"  -d, --dry-run         Do not execute the tests\n"
+	"  -t <regex>, --include-tests <regex>\n"
+	"                        Run only matching tests (can be used more than once)\n"
+	"  -x <regex>, --exclude-tests <regex>\n"
+	"                        Exclude matching tests (can be used more than once)\n"
+	"  --abort-on-monitored-error\n"
+	"                        Abort execution when a fatal condition is detected.\n"
+	"                        <TODO>\n"
+	"  -s, --sync            Sync results to disk after every test\n"
+	"  -l {quiet,verbose,dummy}, --log-level {quiet,verbose,dummy}\n"
+	"                        Set the logger verbosity level\n"
+	"  --test-list TEST_LIST\n"
+	"                        A file containing a list of tests to run\n"
+	"  -o, --overwrite       If the results-path already exists, delete it\n"
+	"  --ignore-missing      Ignored but accepted, for piglit compatibility\n"
+	"\n"
+	" Incompatible options:\n"
+	"  -m, --multiple-mode   Run multiple subtests in the same binary execution.\n"
+	"                        If a testlist file is given, consecutive subtests are\n"
+	"                        run in the same execution if they are from the same\n"
+	"                        binary. Note that in that case relative ordering of the\n"
+	"                        subtest execution is dictated by the test binary, not\n"
+	"                        the testlist\n"
+	"  --inactivity-timeout <seconds>\n"
+	"                        Kill the running test after <seconds> of inactivity in\n"
+	"                        the test's stdout, stderr, or dmesg\n"
+	"  --use-watchdog        Use hardware watchdog for lethal enforcement of the\n"
+	"                        above timeout. Killing the test process is still\n"
+	"                        attempted at timeout trigger.\n"
+	"  [test_root]           Directory that contains the IGT tests. The environment\n"
+	"                        variable IGT_TEST_ROOT will be used if set, overriding\n"
+	"                        this option if given.\n"
+	;
+
+static void usage(const char *extra_message, bool use_stderr)
+{
+	FILE *f = use_stderr ? stderr : stdout;
+
+	if (extra_message)
+		fprintf(f, "%s\n\n", extra_message);
+
+	fprintf(f, "%s", usage_str);
+}
+
+static bool add_regex(struct regex_list *list, char *new)
+{
+	regex_t *regex;
+	size_t buflen;
+	char *buf;
+	int s;
+
+	regex = malloc(sizeof(*regex));
+
+	if ((s = regcomp(regex, new,
+			 REG_EXTENDED | REG_NOSUB)) != 0) {
+		buflen = regerror(s, regex, NULL, 0);
+		buf = malloc(buflen);
+		regerror(s, regex, buf, buflen);
+		usage(buf, true);
+
+		free(buf);
+		regfree(regex);
+		free(regex);
+		return false;
+	}
+
+	list->regexes = realloc(list->regexes,
+				(list->size + 1) * sizeof(*list->regexes));
+	list->regex_strings = realloc(list->regex_strings,
+				      (list->size + 1) * sizeof(*list->regex_strings));
+	list->regexes[list->size] = regex;
+	list->regex_strings[list->size] = new;
+	list->size++;
+
+	return true;
+}
+
+static void free_regexes(struct regex_list *regexes)
+{
+	size_t i;
+
+	for (i = 0; i < regexes->size; i++) {
+		free(regexes->regex_strings[i]);
+		regfree(regexes->regexes[i]);
+		free(regexes->regexes[i]);
+	}
+	free(regexes->regex_strings);
+	free(regexes->regexes);
+}
+
+static bool readable_file(char *filename)
+{
+	return !access(filename, R_OK);
+}
+
+void init_settings(struct settings *settings)
+{
+	memset(settings, 0, sizeof(*settings));
+}
+
+void free_settings(struct settings *settings)
+{
+	free(settings->test_list);
+	free(settings->name);
+	free(settings->test_root);
+	free(settings->results_path);
+
+	free_regexes(&settings->include_regexes);
+	free_regexes(&settings->exclude_regexes);
+
+	init_settings(settings);
+}
+
+bool parse_options(int argc, char **argv,
+		   struct settings *settings)
+{
+	int c;
+	char *env_test_root;
+
+	static struct option long_options[] = {
+		{"help", no_argument, NULL, OPT_HELP},
+		{"name", required_argument, NULL, OPT_NAME},
+		{"dry-run", no_argument, NULL, OPT_DRY_RUN},
+		{"include-tests", required_argument, NULL, OPT_INCLUDE},
+		{"exclude-tests", required_argument, NULL, OPT_EXCLUDE},
+		{"abort-on-monitored-error", no_argument, NULL, OPT_ABORT_ON_ERROR},
+		{"sync", no_argument, NULL, OPT_SYNC},
+		{"log-level", required_argument, NULL, OPT_LOG_LEVEL},
+		{"test-list", required_argument, NULL, OPT_TEST_LIST},
+		{"overwrite", no_argument, NULL, OPT_OVERWRITE},
+		{"ignore-missing", no_argument, NULL, OPT_IGNORE_MISSING},
+		{"multiple-mode", no_argument, NULL, OPT_MULTIPLE},
+		{"inactivity-timeout", required_argument, NULL, OPT_TIMEOUT},
+		{"use-watchdog", no_argument, NULL, OPT_WATCHDOG},
+		{ 0, 0, 0, 0},
+	};
+
+	free_settings(settings);
+
+	optind = 1;
+
+	while ((c = getopt_long(argc, argv, "hn:dt:x:sl:om", long_options, NULL)) != -1) {
+		switch (c) {
+		case OPT_HELP:
+			usage(NULL, false);
+			goto error;
+		case OPT_NAME:
+			settings->name = strdup(optarg);
+			break;
+		case OPT_DRY_RUN:
+			settings->dry_run = true;
+			break;
+		case OPT_INCLUDE:
+			if (!add_regex(&settings->include_regexes, strdup(optarg)))
+				goto error;
+			break;
+		case OPT_EXCLUDE:
+			if (!add_regex(&settings->exclude_regexes, strdup(optarg)))
+				goto error;
+			break;
+		case OPT_ABORT_ON_ERROR:
+			settings->abort_on_error = true;
+			break;
+		case OPT_SYNC:
+			settings->sync = true;
+			break;
+		case OPT_LOG_LEVEL:
+			if (!set_log_level(settings, optarg)) {
+				usage("Cannot parse log level", true);
+				goto error;
+			}
+			break;
+		case OPT_TEST_LIST:
+			settings->test_list = absolute_path(optarg);
+			break;
+		case OPT_OVERWRITE:
+			settings->overwrite = true;
+			break;
+		case OPT_IGNORE_MISSING:
+			/* Ignored, piglit compatibility */
+			break;
+		case OPT_MULTIPLE:
+			settings->multiple_mode = true;
+			break;
+		case OPT_TIMEOUT:
+			settings->inactivity_timeout = atoi(optarg);
+			break;
+		case OPT_WATCHDOG:
+			settings->use_watchdog = true;
+			break;
+		case '?':
+			usage(NULL, true);
+			goto error;
+		default:
+			usage("Cannot parse options", true);
+			goto error;
+		}
+	}
+
+	switch (argc - optind) {
+	case 2:
+		settings->test_root = absolute_path(argv[optind]);
+		++optind;
+		/* fallthrough */
+	case 1:
+		settings->results_path = absolute_path(argv[optind]);
+		break;
+	case 0:
+		usage("Results-path missing", true);
+		goto error;
+	default:
+		usage("Extra arguments after results-path", true);
+		goto error;
+	}
+
+	if ((env_test_root = getenv("IGT_TEST_ROOT")) != NULL) {
+		free(settings->test_root);
+		settings->test_root = absolute_path(env_test_root);
+	}
+
+	if (!settings->test_root) {
+		usage("Test root not set", true);
+		goto error;
+	}
+
+	if (!settings->name) {
+		char *name = strdup(settings->results_path);
+		settings->name = strdup(basename(name));
+		free(name);
+	}
+
+	return true;
+
+ error:
+	free_settings(settings);
+	return false;
+}
+
+bool validate_settings(struct settings *settings)
+{
+	int dirfd, fd;
+
+	if (settings->test_list && !readable_file(settings->test_list)) {
+		usage("Cannot open test-list file", true);
+		return false;
+	}
+
+	if (!settings->results_path) {
+		usage("No results-path set; this shouldn't happen", true);
+		return false;
+	}
+
+	if (!settings->test_root) {
+		usage("No test root set; this shouldn't happen", true);
+		return false;
+	}
+
+	dirfd = open(settings->test_root, O_DIRECTORY | O_RDONLY);
+	if (dirfd < 0) {
+		fprintf(stderr, "Test directory %s cannot be opened\n", settings->test_root);
+		return false;
+	}
+
+	fd = openat(dirfd, "test-list.txt", O_RDONLY);
+	if (fd < 0) {
+		fprintf(stderr, "Cannot open %s/test-list.txt\n", settings->test_root);
+		close(dirfd);
+		return false;
+	}
+
+	close(fd);
+	close(dirfd);
+
+	return true;
+}
+
+char *absolute_path(char *path)
+{
+	char *result = NULL;
+	char *tmppath, *tmpname;
+
+	result = realpath(path, NULL);
+	if (result != NULL)
+		return result;
+
+	tmppath = strdup(path);
+	tmpname = dirname(tmppath);
+	free(result);
+	result = realpath(tmpname, NULL);
+	free(tmppath);
+
+	if (result != NULL) {
+		tmppath = strdup(path);
+		tmpname = basename(tmppath);
+		strcat(result, "/");
+		strcat(result, tmpname);
+		free(tmppath);
+		return result;
+	}
+
+	free(result);
+	return NULL;
+}
+
+static char settings_filename[] = "metadata.txt";
+bool serialize_settings(struct settings *settings)
+{
+#define SERIALIZE_LINE(f, s, name, format) fprintf(f, "%s : " format "\n", #name, s->name)
+
+	int dirfd, fd;
+	FILE *f;
+
+	if (!settings->results_path) {
+		usage("No results-path set; this shouldn't happen", true);
+		return false;
+	}
+
+	if ((dirfd = open(settings->results_path, O_DIRECTORY | O_RDONLY)) < 0) {
+		mkdir(settings->results_path, 0755);
+		if ((dirfd = open(settings->results_path, O_DIRECTORY | O_RDONLY)) < 0) {
+			usage("Creating results-path failed", true);
+			return false;
+		}
+	}
+
+	if ((fd = openat(dirfd, settings_filename, O_RDONLY)) >= 0) {
+		close(fd);
+
+		if (!settings->overwrite) {
+			/* Serialization data already exists, not overwriting */
+			close(dirfd);
+			return false;
+		}
+
+		if (unlinkat(dirfd, settings_filename, 0) != 0) {
+			usage("Error overwriting old settings metadata", true);
+			close(dirfd);
+			return false;
+		}
+	}
+
+	if ((fd = openat(dirfd, settings_filename, O_CREAT | O_EXCL | O_WRONLY, 0666)) < 0) {
+		usage("Creating settings serialization file failed", true);
+		close(dirfd);
+		return false;
+	}
+
+	f = fdopen(fd, "w");
+	if (!f) {
+		close(fd);
+		close(dirfd);
+		return false;
+	}
+
+	SERIALIZE_LINE(f, settings, abort_on_error, "%d");
+	if (settings->test_list)
+		SERIALIZE_LINE(f, settings, test_list, "%s");
+	if (settings->name)
+		SERIALIZE_LINE(f, settings, name, "%s");
+	SERIALIZE_LINE(f, settings, dry_run, "%d");
+	SERIALIZE_LINE(f, settings, sync, "%d");
+	SERIALIZE_LINE(f, settings, log_level, "%d");
+	SERIALIZE_LINE(f, settings, overwrite, "%d");
+	SERIALIZE_LINE(f, settings, multiple_mode, "%d");
+	SERIALIZE_LINE(f, settings, inactivity_timeout, "%d");
+	SERIALIZE_LINE(f, settings, use_watchdog, "%d");
+	SERIALIZE_LINE(f, settings, test_root, "%s");
+	SERIALIZE_LINE(f, settings, results_path, "%s");
+
+	if (settings->sync) {
+		fsync(fd);
+		fsync(dirfd);
+	}
+
+	fclose(f);
+	close(dirfd);
+	return true;
+
+#undef SERIALIZE_LINE
+}
+
+static char *maybe_strdup(char *str)
+{
+	if (!str)
+		return NULL;
+
+	return strdup(str);
+}
+
+bool read_settings(struct settings *settings, int dirfd)
+{
+#define PARSE_LINE(s, name, val, field, write) \
+	if (!strcmp(name, #field)) {	       \
+		s->field = write;	       \
+		free(name);		       \
+		free(val);		       \
+		name = val = NULL;	       \
+		continue;		       \
+	}
+
+	int fd;
+	FILE *f;
+	char *name = NULL, *val = NULL;
+
+	free_settings(settings);
+
+	if ((fd = openat(dirfd, settings_filename, O_RDONLY)) < 0)
+		return false;
+
+	f = fdopen(fd, "r");
+	if (!f) {
+		close(fd);
+		return false;
+	}
+
+	while (fscanf(f, "%ms : %ms", &name, &val) == 2) {
+		int numval = atoi(val);
+		PARSE_LINE(settings, name, val, abort_on_error, numval);
+		PARSE_LINE(settings, name, val, test_list, maybe_strdup(val));
+		PARSE_LINE(settings, name, val, name, maybe_strdup(val));
+		PARSE_LINE(settings, name, val, dry_run, numval);
+		PARSE_LINE(settings, name, val, sync, numval);
+		PARSE_LINE(settings, name, val, log_level, numval);
+		PARSE_LINE(settings, name, val, overwrite, numval);
+		PARSE_LINE(settings, name, val, multiple_mode, numval);
+		PARSE_LINE(settings, name, val, inactivity_timeout, numval);
+		PARSE_LINE(settings, name, val, use_watchdog, numval);
+		PARSE_LINE(settings, name, val, test_root, maybe_strdup(val));
+		PARSE_LINE(settings, name, val, results_path, maybe_strdup(val));
+
+		printf("Warning: Unknown field in settings file: %s = %s\n",
+		       name, val);
+		free(name);
+		free(val);
+		name = val = NULL;
+	}
+
+	free(name);
+	free(val);
+	fclose(f);
+
+	return true;
+
+#undef PARSE_LINE
+}
diff --git a/runner/settings.h b/runner/settings.h
new file mode 100644
index 00000000..9d1f03fb
--- /dev/null
+++ b/runner/settings.h
@@ -0,0 +1,111 @@
+#ifndef RUNNER_SETTINGS_H
+#define RUNNER_SETTINGS_H
+
+#include <stdbool.h>
+#include <stddef.h>
+#include <sys/types.h>
+#include <regex.h>
+
+enum {
+	LOG_LEVEL_NORMAL = 0,
+	LOG_LEVEL_QUIET = -1,
+	LOG_LEVEL_VERBOSE = 1,
+};
+
+struct regex_list {
+	char **regex_strings;
+	regex_t** regexes;
+	size_t size;
+};
+
+struct settings {
+	bool abort_on_error;
+	char *test_list;
+	char *name;
+	bool dry_run;
+	struct regex_list include_regexes;
+	struct regex_list exclude_regexes;
+	bool sync;
+	int log_level;
+	bool overwrite;
+	bool multiple_mode;
+	int inactivity_timeout;
+	bool use_watchdog;
+	char *test_root;
+	char *results_path;
+};
+
+/**
+ * init_settings:
+ *
+ * Initializes a settings object to an empty state (all values NULL, 0
+ * or false).
+ *
+ * @settings: Object to initialize. Storage for it must exist.
+ */
+void init_settings(struct settings *settings);
+
+/**
+ * free_settings:
+ *
+ * Releases all allocated resources for a settings object and
+ * initializes it to an empty state (see #init_settings).
+ *
+ * @settings: Object to release and initialize.
+ */
+void free_settings(struct settings *settings);
+
+/**
+ * parse_options:
+ *
+ * Parses command line options and sets the settings object to
+ * designated values.
+ *
+ * The function can be called again on the same settings object. The
+ * old values will be properly released and cleared. On a parse
+ * failure, the settings object will be in an empty state (see
+ * #init_settings) and usage instructions will be printed with an
+ * error message.
+ *
+ * @argc: Argument count
+ * @argv: Argument array. First element is the program name.
+ * @settings: Settings object to fill with values. Must have proper
+ * storage.
+ *
+ * Returns: True on successful parse, false on error.
+ */
+bool parse_options(int argc, char **argv,
+		   struct settings *settings);
+
+/**
+ * validate_settings:
+ *
+ * Checks the settings object against the system to see if executing
+ * on it can be done. Checks pathnames for existence and access
+ * rights. Note that this function will not check that the designated
+ * job listing (through a test-list file or the -t/-x flags) yields a
+ * non-zero amount of testing to be done. On errors, usage
+ * instructions will be printed with an error message.
+ *
+ * @settings: Settings object to check.
+ *
+ * Returns: True on valid settings, false on any error.
+ */
+bool validate_settings(struct settings *settings);
+
+/* TODO: Better place for this */
+char *absolute_path(char *path);
+
+/**
+ * serialize_settings:
+ *
+ * Serializes the settings object to a file in the results_path
+ * directory.
+ *
+ * @settings: Settings object to serialize.
+ */
+bool serialize_settings(struct settings *settings);
+
+bool read_settings(struct settings *settings, int dirfd);
+
+#endif
-- 
2.14.1

_______________________________________________
igt-dev mailing list
igt-dev@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/igt-dev

Re: [igt-dev] [PATCH i-g-t 3/4] runner: New test runner

[Arkadiusz Hiler]

On Mon, Apr 30, 2018 at 12:28:47PM +0300, Petri Latvala wrote:
> This is a new test runner to replace piglit. Piglit has been very
> useful as a test runner, but certain improvements have been very
> difficult if possible at all in a generic test running framework.
> 
> Important improvements over piglit:
> 
> - Faster to launch. Being able to make assumptions about what we're
>   executing makes it possible to save significant amounts of time. For
>   example, a testlist file's line "igt@somebinary@somesubtest" already
>   has all the information we need to construct the correct command
>   line to execute that particular subtest, instead of listing all
>   subtests of all test binaries and mapping them to command
>   lines. Same goes for the regexp filters command line flags -t and
>   -x; If we use -x somebinaryname, we don't need to list subtests from
>   somebinaryname, we already know none of them will get executed.
> 
> - Logs of incomplete tests. Piglit collects test output to memory and
>   dumps them to a file when the test is complete. The new runner
>   writes all output to disk immediately.
> 
> - Ability to execute multiple subtests in one binary execution. This
>   was possible with piglit, but its semantics made it very hard to
>   implement in practice. For example, having a testlist file not only
>   selected a subset of tests to run, but also mandated that they be
>   executed in the same order.
> 
> - Flexible timeout support. Instead of mandating a time tests cannot
>   exceed, the new runner has a timeout on inactivity. Activity is
>   any output on the test's stdout or stderr, or kernel activity via
>   /dev/kmsg.
> 
> The runner is fairly piglit compatible. The command line is very
> similar, with a few additions. IGT_TEST_ROOT environment flag is still
> supported, but can also be set via command line (in place of igt.py in
> piglit command line).
> 
> The results are a set of log files, processed into a piglit-compatible
> results.json file (BZ2 compression TODO). There are some new fields in
> the json for extra information:
> 
> - "igt-version" contains the IGT version line. In
>   multiple-subtests-mode the version information is only printed once,
>   so it needs to be duplicated to all subtest results this way.
> - "dmesg-warnings" contains the dmesg lines that triggered a
>   dmesg-warn/dmesg-fail state.
> - Runtime information will be different. Piglit takes a timestamp at
>   the beginning and at the end of execution for runtime. The new
>   runner uses the subtest output text. The binary execution time will
>   also be included; The key "igt@somebinary" will have the runtime of
>   the binary "somebinary", whereas "igt@somebinary@a" etc will have
>   the runtime of the subtests. Substracting the subtest runtimes from
>   the binary runtime yields the total time spent doing setup in
>   igt_fixture blocks.
> 
> Signed-off-by: Petri Latvala <petri.latvala@intel.com>
> Cc: Maarten Lankhorst <maarten.lankhorst@linux.intel.com>
> Cc: Arkadiusz Hiler <arkadiusz.hiler@intel.com>
> Cc: Tomi Sarvela <tomi.p.sarvela@intel.com>
> Cc: Martin Peres <martin.peres@linux.intel.com>
> ---
>  meson.build        |    1 +
>  runner/executor.c  | 1076 ++++++++++++++++++++++++++++++++++++++++++++++++++++
>  runner/executor.h  |   42 ++
>  runner/job_list.c  |  492 ++++++++++++++++++++++++
>  runner/job_list.h  |   37 ++
>  runner/meson.build |   24 ++
>  runner/resultgen.c |  864 +++++++++++++++++++++++++++++++++++++++++
>  runner/resultgen.h |    9 +
>  runner/results.c   |   26 ++
>  runner/runner.c    |   40 ++
>  runner/settings.c  |  506 ++++++++++++++++++++++++
>  runner/settings.h  |  111 ++++++
>  12 files changed, 3228 insertions(+)
>  create mode 100644 runner/executor.c
>  create mode 100644 runner/executor.h
>  create mode 100644 runner/job_list.c
>  create mode 100644 runner/job_list.h
>  create mode 100644 runner/meson.build
>  create mode 100644 runner/resultgen.c
>  create mode 100644 runner/resultgen.h
>  create mode 100644 runner/results.c
>  create mode 100644 runner/runner.c
>  create mode 100644 runner/settings.c
>  create mode 100644 runner/settings.h
> 
> diff --git a/meson.build b/meson.build
> index 5b783e5d..35e42631 100644
> --- a/meson.build
> +++ b/meson.build
> @@ -136,6 +136,7 @@ subdir('lib')
>  subdir('tests')
>  subdir('benchmarks')
>  subdir('tools')
> +subdir('runner')
>  if libdrm_intel.found()
>  	subdir('assembler')
>  	if ['x86', 'x86_64'].contains(host_machine.cpu_family())
> diff --git a/runner/executor.c b/runner/executor.c
> new file mode 100644
> index 00000000..2754f5a2
> --- /dev/null
> +++ b/runner/executor.c
> @@ -0,0 +1,1076 @@
> +#include <errno.h>
> +#include <fcntl.h>
> +#include <linux/watchdog.h>
> +#include <signal.h>
> +#include <stdio.h>
> +#include <stdlib.h>
> +#include <string.h>
> +#include <sys/ioctl.h>
> +#include <sys/select.h>
> +#include <sys/signalfd.h>
> +#include <sys/stat.h>
> +#include <sys/types.h>
> +#include <sys/utsname.h>
> +#include <sys/wait.h>
> +#include <time.h>
> +#include <unistd.h>
> +
> +#include "executor.h"
> +
> +/* Clock handling copied from igt_core.c */

Maybe extract it to a some kind of meta-lib?

> +
> +static clockid_t igt_clock;
> +
> +#define time_valid(ts) ((ts)->tv_sec || (ts)->tv_nsec)
> +
> +static double
> +time_elapsed(struct timespec *then,
> +	     struct timespec *now)
> +{
> +	double elapsed = -1.;
> +
> +	if (time_valid(then) && time_valid(now)) {
> +		elapsed = now->tv_sec - then->tv_sec;
> +		elapsed += (now->tv_nsec - then->tv_nsec) * 1e-9;
> +	}
> +
> +	return elapsed;
> +}
> +
> +static int gettime(struct timespec *ts)
> +{
> +	memset(ts, 0, sizeof(*ts));
> +	errno = 0;
> +
> +	/* Stay on the same clock for consistency. */
> +	if (igt_clock != (clockid_t)-1) {
> +		if (clock_gettime(igt_clock, ts))
> +			goto error;
> +		return 0;
> +	}
> +
> +#ifdef CLOCK_MONOTONIC_RAW
> +	if (!clock_gettime(igt_clock = CLOCK_MONOTONIC_RAW, ts))
> +		return 0;
> +#endif
> +#ifdef CLOCK_MONOTONIC_COARSE
> +	if (!clock_gettime(igt_clock = CLOCK_MONOTONIC_COARSE, ts))
> +		return 0;
> +#endif
> +	if (!clock_gettime(igt_clock = CLOCK_MONOTONIC, ts))
> +		return 0;
> +error:
> +	fprintf(stderr, "Warning: Could not read monotonic time: %s\n",
> +		strerror(errno));
> +
> +	return -errno;
> +}
> +
> +struct watchdogs
> +{
> +	int *fds;
> +	size_t num_dogs;
> +} watchdogs;
> +
> +static void close_watchdogs(struct settings *settings)
> +{
> +	size_t i;
> +
> +	if (settings && settings->log_level >= LOG_LEVEL_VERBOSE)
> +		printf("Closing watchdogs\n");
> +
> +	for (i = 0; i < watchdogs.num_dogs; i++) {
> +		write(watchdogs.fds[i], "V", 1);
> +		close(watchdogs.fds[i]);
> +	}
> +}
> +
> +static void close_watchdogs_atexit()
> +{
> +	close_watchdogs(NULL);
> +}
> +
> +static void init_watchdogs(struct settings *settings)
> +{
> +	int i;
> +	char name[32];
> +	int fd;
> +
> +	memset(&watchdogs, 0, sizeof(watchdogs));
> +
> +	if (!settings->use_watchdog || settings->inactivity_timeout <= 0)
> +		return;
> +
> +	if (settings->log_level >= LOG_LEVEL_VERBOSE) {
> +		printf("Initializing watchdogs\n");
> +	}
> +
> +	atexit(close_watchdogs_atexit);
> +
> +	for (i = 0; ; i++) {
> +		snprintf(name, 32, "/dev/watchdog%d", i);

Sizeof instead of magic number repeated.

> +		if ((fd = open(name, O_RDWR | O_CLOEXEC)) < 0)
> +			break;
> +
> +		watchdogs.num_dogs++;
> +		watchdogs.fds = realloc(watchdogs.fds, watchdogs.num_dogs * sizeof(int));
> +		watchdogs.fds[i] = fd;
> +
> +		if (settings->log_level >= LOG_LEVEL_VERBOSE)
> +			printf(" %s\n", name);
> +	}
> +}
> +
> +static int watchdogs_set_timeout(int timeout)
> +{
> +	size_t i;
> +	int orig_timeout = timeout;
> +
> +	for (i = 0; i < watchdogs.num_dogs; i++) {
> +		if (ioctl(watchdogs.fds[i], WDIOC_SETTIMEOUT, &timeout)) {
> +			write(watchdogs.fds[i], "V", 1);
> +			close(watchdogs.fds[i]);
> +			watchdogs.fds[i] = -1;
> +			continue;
> +		}
> +
> +		if (timeout < orig_timeout) {
> +			/*
> +			 * Timeout of this caliber refused. We want to
> +			 * use the same timeout for all devices.
> +			 */
> +			return watchdogs_set_timeout(timeout);
> +		}
> +	}
> +
> +	return timeout;
> +}
> +
> +static void ping_watchdogs()
> +{
> +	size_t i;
> +
> +	for (i = 0; i < watchdogs.num_dogs; i++) {
> +		ioctl(watchdogs.fds[i], WDIOC_KEEPALIVE, 0);
> +	}
> +}
> +
> +static void prune_subtest(struct job_list_entry *entry, char *subtest)
> +{
> +	char *excl;
> +
> +	/*
> +	 * Subtest pruning is done by adding exclusion strings to the
> +	 * subtest list. The last matching item on the subtest
> +	 * selection command line flag decides whether to run a
> +	 * subtest, see igt_core.c for details.  If the list is empty,
> +	 * the expected subtest set is unknown, so we need to add '*'
> +	 * first so we can start excluding.
> +	 */
> +
> +	if (entry->subtest_count == 0) {
> +		entry->subtest_count++;
> +		entry->subtests = realloc(entry->subtests, entry->subtest_count * sizeof(*entry->subtests));
> +		entry->subtests[0] = strdup("*");
> +	}
> +
> +	excl = malloc(strlen(subtest) + 2);
> +	excl[0] = '!';
> +	strcpy(excl + 1, subtest);
> +
> +	entry->subtest_count++;
> +	entry->subtests = realloc(entry->subtests, entry->subtest_count * sizeof(*entry->subtests));
> +	entry->subtests[entry->subtest_count - 1] = excl;
> +}
> +
> +static bool prune_from_journal(struct job_list_entry *entry, int fd)
> +{
> +	char *subtest;
> +	FILE *f;
> +	bool any_pruned = false;
> +
> +	/*
> +	 * Each journal line is a subtest that has been started, or
> +	 * the line 'exit:$exitcode (time)', or 'timeout:$exitcode (time)'.
> +	 */
> +
> +	f = fdopen(fd, "r");
> +	if (!f)
> +		return false;
> +
> +	while (fscanf(f, "%ms", &subtest) == 1) {
> +		if (subtest[0] == '\0') {

So is this a convention, that we have the last line as an empty string or am
I missing something?

I am used to *scanf returning EOF to denote that.

> +			/* EOF */
> +			free(subtest);
> +			break;
> +		}
> +
> +		if (!strncmp(subtest, "exit:", 5)) {
> +			/* Fully done. Mark that by making the binary name invalid. */
> +			fscanf(f, " (%*fs)");
> +			entry->binary[0] = '\0';
> +			free(subtest);
> +			continue;
> +		}
> +
> +		if (!strncmp(subtest, "timeout:", 8)) {

Other option would be to define those strings and use sizeof,
but that may be an overkill.

> +			fscanf(f, " (%*fs)");
> +			free(subtest);
> +			continue;
> +		}
> +
> +		prune_subtest(entry, subtest);
> +
> +		free(subtest);
> +		any_pruned = true;
> +	}
> +
> +	fclose(f);
> +	return any_pruned;
> +}
> +
> +static char *filenames[] = {

It took me quite a while to relate this array to _F_LAST & friends.
Having them together, more explicit naming or using _F_LAST as array
size here could help.

> +	"journal.txt",
> +	"out.txt",
> +	"err.txt",
> +	"dmesg.txt",
> +};
> +
> +static int open_at_end(int dirfd, char *name)
> +{
> +	int fd = openat(dirfd, name, O_RDWR | O_CREAT | O_CLOEXEC, 0666);
> +	char last;
> +
> +	if (fd >= 0) {
> +		if (lseek(fd, -1, SEEK_END) >= 0 &&
> +		    read(fd, &last, 1) == 1 &&
> +		    last != '\n') {
> +			write(fd, "\n", 1);
> +		}
> +		lseek(fd, 0, SEEK_END);
> +	}
> +
> +	return fd;
> +}
> +
> +static int open_for_reading(int dirfd, char *name)
> +{
> +	return openat(dirfd, name, O_RDONLY);
> +}
> +
> +bool open_output_files(int dirfd, int *fds, bool write)
> +{
> +	int i;
> +	int (*openfunc)(int, char*) = write ? open_at_end : open_for_reading;
> +
> +	for (i = 0; i < _F_LAST; i++) {
> +		if ((fds[i] = openfunc(dirfd, filenames[i])) < 0) {
> +			while (--i >= 0)
> +				close(fds[i]);
> +			return false;
> +		}
> +	}
> +
> +	return true;
> +}
> +
> +void close_outputs(int *fds)
> +{
> +	int i;
> +
> +	for (i = 0; i < _F_LAST; i++) {
> +		close(fds[i]);
> +	}
> +}
> +
> +static void dump_dmesg(int kmsgfd, int outfd)
> +{
> +	/*
> +	 * Write kernel messages to the log file until we reach
> +	 * 'now'. Unfortunately, /dev/kmsg doesn't support seeking to
> +	 * -1 from SEEK_END so we need to use a second fd to read a
> +	 * message to match against, or stop when we reach EAGAIN.
> +	 */
> +
> +	int comparefd = open("/dev/kmsg", O_RDONLY | O_NONBLOCK);
> +	unsigned flags;
> +	unsigned long long seq, cmpseq, usec;
> +	char cont;
> +	char buf[256];
> +	ssize_t r;
> +
> +	if (comparefd < 0)
> +		return;
> +
> +	if (fcntl(kmsgfd, F_SETFL, O_NONBLOCK))
> +		return;
> +
> +	while (1) {
> +		if (comparefd >= 0) {
> +			r = read(comparefd, buf, sizeof(buf) - 1);
> +			if (r < 0) {
> +				if (errno != EAGAIN && errno != EPIPE)
> +					return;
> +			} else {
> +				buf[r] = '\0';
> +				if (sscanf(buf, "%u,%llu,%llu,%c;",
> +					   &flags, &cmpseq, &usec, &cont) == 4) {
> +					close(comparefd);
> +					comparefd = -1;
> +				}
> +			}
> +		}
> +
> +		r = read(kmsgfd, buf, sizeof(buf));
> +		if (r <= 0) {
> +			if (errno == EPIPE)
> +				continue;
> +
> +			/*
> +			 * If EAGAIN, we're done. If some other error,
> +			 * we can't do anything anyway.
> +			 */
> +			return;
> +		}
> +
> +		write(outfd, buf, r);
> +
> +		if (comparefd < 0 && sscanf(buf, "%u,%llu,%llu,%c;",
> +					    &flags, &seq, &usec, &cont) == 4) {
> +			if (seq >= cmpseq)
> +				return;
> +		}
> +	}
> +}

This is quite a tangled one, with two reads in the same loop from
different fds and the comparison. I would appreciate a little bit more
verbose comment about why (the kmsg oddities) and how, because I can't
think of anything more readable than this.

> +
> +static bool kill_child(bool use_sigkill,
> +		       struct settings *settings,
> +		       pid_t child)
> +{

Unused settings?

Why bool instead of just taking sig?
You have go to function to figure out what true/false mean.

> +	int sig = use_sigkill ? SIGKILL : SIGTERM;
> +
> +	/*
> +	 * Send the signal to the child directly, and to the child's
> +	 * process group.
> +	 */
> +	kill(-child, sig);
> +	if (kill(child, sig) && errno == ESRCH) {
> +		fprintf(stderr, "Child process does not exist. This shouldn't happen.\n");
> +		return false;
> +	}
> +
> +	return true;
> +}
> +
> +static char starting_subtest[] = "Starting subtest: ";
> +static size_t starting_len = sizeof(starting_subtest) - 1;
> +static char subtest_result_beg[] = "Subtest ";
> +static size_t subtest_result_len = sizeof(subtest_result_beg) - 1;
> +/*
> + * Returns:
> + *  >0 - Success
> + *  =0 - Failure executing
> + *  <0 - Timeout happened, need to recreate from journal
> + */

This is rather strange deviation from "the usual", and that's why you
need additional comment here.

What about 0 on success (cause we are not returning any non-error,
meaningful values anyway) and reusing some of the EXYZ error codes?

Same for execute_entry.

> +static int monitor_output(pid_t child,
> +			   int outfd, int errfd, int kmsgfd, int sigfd,
> +			   int *outputs,
> +			   struct settings *settings)
> +{
> +	fd_set set;
> +	char buf[256];
> +	char *outbuf = NULL;
> +	size_t outbufsize = 0;
> +	char current_subtest[256] = {};
> +	struct signalfd_siginfo siginfo;
> +	ssize_t s;
> +	int n, status;
> +	int nfds = outfd;
> +	int timeout = settings->inactivity_timeout;
> +	int timeout_intervals = 1, intervals_left;
> +	int wd_extra = 10;
> +	int killed = 0; /* 1 = sigterm sent, 2 = sigkill sent */
> +	struct timespec time_beg, time_end;
> +	bool aborting = false;
> +
> +	gettime(&time_beg);
> +
> +	if (errfd > nfds)
> +		nfds = errfd;
> +	if (kmsgfd > nfds)
> +		nfds = kmsgfd;
> +	if (sigfd > nfds)
> +		nfds = sigfd;
> +	nfds++;
> +
> +	if (timeout > 0) {
> +		/*
> +		 * Use original timeout plus some leeway. If we're still
> +		 * alive, we want to kill the test process instead of cutting
> +		 * power.
> +		 */
> +		int wd_timeout = watchdogs_set_timeout(timeout + wd_extra);
> +
> +		if (wd_timeout < timeout + wd_extra) {
> +			/* Watchdog timeout smaller, so ping it more often */
> +			if (wd_timeout - wd_extra < 0)
> +				wd_extra = wd_timeout / 2;
> +			timeout_intervals = timeout / (wd_timeout - wd_extra);
> +			intervals_left = timeout_intervals;
> +			timeout /= timeout_intervals;
> +
> +			if (settings->log_level >= LOG_LEVEL_VERBOSE) {
> +				printf("Watchdog doesn't support the timeout we requested (shortened to %d seconds).\n"
> +				       "Using %d intervals of %d seconds.\n",
> +				       wd_timeout, timeout_intervals, timeout);
> +			}
> +		}
> +	}
> +
> +	while (outfd >= 0 || errfd >= 0 || sigfd >= 0) {
> +		struct timeval tv = { .tv_sec = timeout };
> +
> +		FD_ZERO(&set);
> +		if (outfd >= 0)
> +			FD_SET(outfd, &set);
> +		if (errfd >= 0)
> +			FD_SET(errfd, &set);
> +		if (kmsgfd >= 0)
> +			FD_SET(kmsgfd, &set);
> +		if (sigfd >= 0)
> +			FD_SET(sigfd, &set);
> +
> +		n = select(nfds, &set, NULL, NULL, timeout == 0 ? NULL : &tv);
> +		if (n < 0) {
> +			/* TODO */
> +			return 0;
> +		}
> +
> +		if (n == 0) {
> +			intervals_left--;
> +			if (intervals_left) {
> +				continue;
> +			}

Why those shenanigans with first decrementing intervals_left, then ifing
here? Just leave it at zero once it is zeroed, and decrement later?

Then you have to do (in "case 0"):
intervals_left = timeout_intervals = 1;

And I think you've forgotten to do the same in "case 1", otherwise in
case of another timeout on select() (== 0), we will get negative
internals_left, and keep loopin'.

> +
> +			ping_watchdogs();
> +
> +			switch (killed) {

Why not using 0, SIGKILL and SIGTERM for values of killed instead of
assigning a meaning to arbitrary integers?

> +			case 0:
> +				if (settings->log_level >= LOG_LEVEL_NORMAL) {
> +					printf("Timeout. Killing the current test with SIGTERM.\n");
> +				}
> +
> +				if (!kill_child(false, settings, child))
> +					return 0;
> +				killed = 1;
> +
> +				/*
> +				 * Now continue the loop and let the
> +				 * dying child be handled normally.
> +				 */
> +				timeout = 2; /* Timeout for waiting selected by fair dice roll. */
> +				watchdogs_set_timeout(20);
> +				intervals_left = timeout_intervals = 1;

I am missing the point of setting timeout_intervals to 1 here.

> +				break;
> +			case 1:
> +				if (settings->log_level >= LOG_LEVEL_NORMAL) {
> +					printf("Timeout. Killing the current test with SIGKILL.\n");
> +				}
> +
> +				if (!kill_child(true, settings, child))
> +					return 0;
> +
> +				killed = 2;
> +				break;
> +			case 2:
> +				/* Nothing that can be done, really. Let's tell the caller we want to abort. */
> +				if (settings->log_level >= LOG_LEVEL_NORMAL) {
> +					fprintf(stderr, "Child refuses to die. Aborting.\n");
> +				}
> +				close_watchdogs(settings);
> +				free(outbuf);
> +				close(outfd);
> +				close(errfd);
> +				close(kmsgfd);
> +				close(sigfd);
> +				return 0;
> +			}
> +
> +			continue;
> +		}
> +
> +		intervals_left = timeout_intervals;
> +		ping_watchdogs();
> +
> +		if (outfd >= 0 && FD_ISSET(outfd, &set)) {
> +			char *newline;
> +
> +			s = read(outfd, buf, sizeof(buf));
> +			if (s <= 0) {
> +				if (s < 0) {
> +					fprintf(stderr, "Error reading test's stdout: %s\n",
> +						strerror(errno));
> +				}
> +
> +				close(outfd);
> +				outfd = -1;
> +				goto out_end;
> +			}
> +
> +			write(outputs[_F_OUT], buf, s);
> +			if (settings->sync) {
> +				fdatasync(outputs[_F_OUT]);
> +			}
> +
> +			outbuf = realloc(outbuf, outbufsize + s);
> +			memcpy(outbuf + outbufsize, buf, s);
> +			outbufsize += s;
> +
> +			while ((newline = memchr(outbuf, '\n', outbufsize)) != NULL) {
> +				size_t linelen = newline - outbuf + 1;
> +
> +				if (linelen > starting_len &&
> +				    !memcmp(outbuf, starting_subtest, starting_len)) {
> +					write(outputs[_F_JOURNAL], outbuf + starting_len,
> +					      linelen - starting_len);
> +					memcpy(current_subtest, outbuf + starting_len,
> +					       linelen - starting_len);
> +					current_subtest[linelen - starting_len] = '\0';
> +
> +					if (settings->log_level >= LOG_LEVEL_VERBOSE) {
> +						fwrite(outbuf, 1, linelen, stdout);
> +					}
> +				}
> +				if (linelen > subtest_result_len &&
> +				    !memcmp(outbuf, subtest_result_beg, subtest_result_len)) {
> +					char *delim = memchr(outbuf, ':', linelen);
> +
> +					if (delim != NULL) {
> +						size_t subtestlen = delim - outbuf - subtest_result_len;
> +						if (memcmp(current_subtest, outbuf + subtest_result_len,
> +							   subtestlen)) {
> +							/* Result for a test that didn't ever start */
> +							write(outputs[_F_JOURNAL],
> +							      outbuf + subtest_result_len,
> +							      subtestlen);
> +							write(outputs[_F_JOURNAL], "\n", 1);
> +							if (settings->sync) {
> +								fdatasync(outputs[_F_JOURNAL]);
> +							}
> +							current_subtest[0] = '\0';
> +						}
> +
> +						if (settings->log_level >= LOG_LEVEL_VERBOSE) {
> +							fwrite(outbuf, 1, linelen, stdout);
> +						}
> +					}
> +				}
> +
> +				memmove(outbuf, newline + 1, outbufsize - linelen);
> +				outbufsize -= linelen;
> +			}
> +		}
> +	out_end:
> +
> +		if (errfd >= 0 && FD_ISSET(errfd, &set)) {
> +			s = read(errfd, buf, sizeof(buf));
> +			if (s <= 0) {
> +				if (s < 0) {
> +					fprintf(stderr, "Error reading test's stderr: %s\n",
> +						strerror(errno));
> +				}
> +				close(errfd);
> +				errfd = -1;
> +			} else {
> +				write(outputs[_F_ERR], buf, s);
> +				if (settings->sync) {
> +					fdatasync(outputs[_F_ERR]);
> +				}
> +			}
> +		}
> +
> +		if (kmsgfd >= 0 && FD_ISSET(kmsgfd, &set)) {
> +			s = read(kmsgfd, buf, sizeof(buf));
> +			if (s < 0) {
> +				if (errno != EPIPE) {
> +					fprintf(stderr, "Error reading from kmsg, stopping monitoring: %s\n",
> +						strerror(errno));
> +					close(kmsgfd);
> +					kmsgfd = -1;
> +				}
> +			} else {
> +				write(outputs[_F_DMESG], buf, s);
> +				if (settings->sync) {
> +					fdatasync(outputs[_F_DMESG]);
> +				}
> +			}
> +		}
> +
> +		if (sigfd >= 0 && FD_ISSET(sigfd, &set)) {
> +			double time;
> +
> +			s = read(sigfd, &siginfo, sizeof(siginfo));
> +			if (s < 0) {
> +				fprintf(stderr, "Error reading from signalfd: %s\n",
> +					strerror(errno));
> +				continue;
> +			} else if (siginfo.ssi_signo == SIGCHLD) {
> +				if (child != waitpid(child, &status, WNOHANG)) {
> +					fprintf(stderr, "Failed to reap child\n");
> +					status = 9999;
> +				} else if (WIFEXITED(status)) {
> +					status = WEXITSTATUS(status);
> +					if (status >= 128) {
> +						status = 128 - status;
> +					}
> +				} else if (WIFSIGNALED(status)) {
> +					status = -WTERMSIG(status);
> +				} else {
> +					status = 9999;
> +				}
> +			} else {
> +				/* We're dying, so we're taking them with us */
> +				if (settings->log_level >= LOG_LEVEL_NORMAL)
> +					printf("Abort requested, terminating children\n");
> +
> +				if (!kill_child(false, settings, child))
> +					return 0;
> +				aborting = true;
> +				timeout = 2;
> +				killed = 1;
> +
> +				continue;
> +			}
> +
> +			gettime(&time_end);
> +
> +			time = time_elapsed(&time_beg, &time_end);
> +			if (time < 0.0)
> +				time = 0.0;
> +
> +			if (!aborting) {
> +				dprintf(outputs[_F_JOURNAL], "%s:%d (%.3fs)\n",
> +					killed ? "timeout" : "exit",
> +					status, time);
> +				if (settings->sync) {
> +					fdatasync(outputs[_F_JOURNAL]);
> +				}
> +			}
> +
> +			close(sigfd);
> +			sigfd = -1;
> +			child = 0;
> +		}
> +	}
> +
> +	dump_dmesg(kmsgfd, outputs[_F_DMESG]);
> +	if (settings->sync)
> +		fdatasync(outputs[_F_DMESG]);
> +
> +	free(outbuf);
> +	close(outfd);
> +	close(errfd);
> +	close(kmsgfd);
> +	close(sigfd);
> +
> +	if (aborting)
> +		return 0;
> +
> +	return killed ? -1 : 1;
> +}
> +
> +static void execute_test_process(int outfd, int errfd,
> +				 struct settings *settings,
> +				 struct job_list_entry *entry)
> +{
> +	char *argv[4] = {};
> +	size_t rootlen;
> +
> +	dup2(outfd, STDOUT_FILENO);
> +	dup2(errfd, STDERR_FILENO);
> +
> +	setpgid(0, 0);
> +
> +	rootlen = strlen(settings->test_root);
> +	argv[0] = malloc(rootlen + strlen(entry->binary) + 2);
> +	strcpy(argv[0], settings->test_root);
> +	argv[0][rootlen] = '/';
> +	strcpy(argv[0] + rootlen + 1, entry->binary);
> +
> +	if (entry->subtest_count) {
> +		size_t argsize;
> +		size_t i;
> +
> +		argv[1] = "--run-subtest";
> +		argsize = strlen(entry->subtests[0]);
> +		argv[2] = malloc(argsize + 1);
> +		strcpy(argv[2], entry->subtests[0]);
> +
> +		for (i = 1; i < entry->subtest_count; i++) {
> +			char *sub = entry->subtests[i];
> +			size_t sublen = strlen(sub);
> +
> +			argv[2] = realloc(argv[2], argsize + sublen + 2);
> +			argv[2][argsize] = ',';
> +			strcpy(argv[2] + argsize + 1, sub);
> +			argsize += sublen + 1;
> +		}
> +	}
> +
> +	execv(argv[0], argv);
> +	fprintf(stderr, "Cannot execute %s\n", argv[0]);
> +	exit(79);

Name it through #def?

> +}
> +
> +static int digits(size_t num)
> +{
> +	int ret = 0;
> +	while (num) {
> +		num /= 10;
> +		ret++;
> +	}
> +
> +	if (ret == 0) ret++;
> +	return ret;
> +}
> +
> +/*
> + * Returns:
> + *  >0 - Success
> + *  =0 - Failure executing
> + *  <0 - Timeout happened, need to recreate from journal
> + */
> +static int execute_entry(size_t idx,
> +			  size_t total,
> +			  struct settings *settings,
> +			  struct job_list_entry *entry,
> +			  int testdirfd, int resdirfd)
> +{
> +	int dirfd;
> +	int outputs[_F_LAST];
> +	int kmsgfd;
> +	int sigfd;
> +	sigset_t mask;
> +	int outpipe[2] = { -1, -1 };
> +	int errpipe[2] = { -1, -1 };
> +	char name[32];
> +	pid_t child;
> +	int result;
> +
> +	snprintf(name, 32, "%zd", idx);
> +	mkdirat(resdirfd, name, 0777);
> +	if ((dirfd = openat(resdirfd, name, O_DIRECTORY | O_RDONLY | O_CLOEXEC)) < 0) {
> +		fprintf(stderr, "Error accessing individual test result directory\n");
> +		return 0;
> +	}
> +
> +	if (!open_output_files(dirfd, outputs, true)) {
> +		close(dirfd);
> +		fprintf(stderr, "Error opening output files\n");
> +		return 0;
> +	}
> +
> +	if (settings->sync) {
> +		fsync(dirfd);
> +		fsync(resdirfd);
> +	}
> +
> +	if (pipe(outpipe) || pipe(errpipe)) {
> +		close_outputs(outputs);
> +		close(dirfd);
> +		close(outpipe[0]);
> +		close(outpipe[1]);
> +		close(errpipe[0]);
> +		close(errpipe[1]);
> +		fprintf(stderr, "Error creating pipes: %s\n", strerror(errno));
> +		return 0;
> +	}
> +
> +	if ((kmsgfd = open("/dev/kmsg", O_RDONLY | O_CLOEXEC)) < 0) {
> +		fprintf(stderr, "Warning: Cannot open /dev/kmsg\n");
> +	} else {
> +		/* TODO: Checking of abort conditions in pre-execute dmesg */
> +		lseek(kmsgfd, 0, SEEK_END);
> +	}
> +
> +	sigemptyset(&mask);
> +	sigaddset(&mask, SIGCHLD);
> +	sigaddset(&mask, SIGINT);
> +	sigaddset(&mask, SIGTERM);
> +	sigaddset(&mask, SIGQUIT);
> +	sigprocmask(SIG_BLOCK, &mask, NULL);
> +	sigfd = signalfd(-1, &mask, O_CLOEXEC);
> +
> +	if (sigfd < 0) {
> +		/* TODO: Handle better */
> +		fprintf(stderr, "Cannot monitor child process with signalfd\n");
> +		close(outpipe[0]);
> +		close(errpipe[0]);
> +		close(outpipe[1]);
> +		close(errpipe[1]);
> +		close(kmsgfd);
> +		close_outputs(outputs);
> +		close(dirfd);
> +		return 0;
> +	}
> +
> +	if (settings->log_level >= LOG_LEVEL_NORMAL) {
> +		int width = digits(total);
> +		printf("[%0*zd/%0*zd] %s", width, idx + 1, width, total, entry->binary);
> +		if (entry->subtest_count > 0) {
> +			size_t i;
> +			char *delim = "";
> +
> +			printf(" (");
> +			for (i = 0; i < entry->subtest_count; i++) {
> +				printf("%s%s", delim, entry->subtests[i]);
> +				delim = ", ";
> +			}
> +			printf(")");
> +		}
> +		printf("\n");
> +	}
> +
> +	if ((child = fork())) {
> +		int outfd = outpipe[0];
> +		int errfd = errpipe[0];
> +		close(outpipe[1]);
> +		close(errpipe[1]);
> +
> +		result = monitor_output(child, outfd, errfd, kmsgfd, sigfd,
> +					outputs, settings);
> +	} else {
> +		int outfd = outpipe[1];
> +		int errfd = errpipe[1];
> +		close(outpipe[0]);
> +		close(errpipe[0]);
> +
> +		sigprocmask(SIG_UNBLOCK, &mask, NULL);
> +
> +		setenv("IGT_SENTINEL_ON_STDERR", "1", 1);
> +
> +		execute_test_process(outfd, errfd, settings, entry);
> +	}
> +
> +	close(outpipe[1]);
> +	close(errpipe[1]);
> +	close(kmsgfd);
> +	close_outputs(outputs);
> +	close(dirfd);
> +
> +	return result;
> +}
> +
> +static bool clear_test_result_directory(int dirfd)
> +{
> +	if (unlinkat(dirfd, "out.txt", 0) ||
> +	    unlinkat(dirfd, "err.txt", 0) ||
> +	    unlinkat(dirfd, "dmesg.txt", 0) ||
> +	    unlinkat(dirfd, "journal.txt", 0)) {
> +		if (errno != ENOENT) {
> +			fprintf(stderr, "Error clearing test result directories: %s\n",
> +				strerror(errno));
> +			return false;
> +		}

rm err.txt

Then it bails out at it early (because of ||), but dmesg.txt and
journal.txt is still there.

> +	}
> +
> +	return true;
> +}
> +
> +static bool clear_old_results(char *path)
> +{
> +	int dirfd;
> +	size_t i;
> +
> +	if ((dirfd = open(path, O_DIRECTORY | O_RDONLY)) < 0) {
> +		if (errno == ENOENT) {
> +			/* Successfully cleared if it doesn't even exist */
> +			return true;
> +		}
> +
> +		fprintf(stderr, "Error clearing old results: %s\n", strerror(errno));
> +		return false;
> +	}
> +
> +	if (unlinkat(dirfd, "uname.txt", 0) && errno != ENOENT) {
> +		close(dirfd);
> +		fprintf(stderr, "Error clearing old results: %s\n", strerror(errno));
> +		return false;
> +	}
> +
> +	for (i = 0; true; i++) {
> +		char name[32];
> +		int resdirfd;
> +
> +		snprintf(name, 32, "%zd", i);
> +		if ((resdirfd = openat(dirfd, name, O_DIRECTORY | O_RDONLY)) < 0)
> +			break;
> +
> +		if (!clear_test_result_directory(resdirfd)) {
> +			close(resdirfd);
> +			close(dirfd);
> +			return false;
> +		}
> +		close(resdirfd);
> +		unlinkat(dirfd, name, AT_REMOVEDIR);

This will fail silently in the case explained above.

> +	}
> +
> +	close(dirfd);
> +
> +	return true;
> +}
> +
> +static bool initialize_execute_from_resume(struct execute_state *state,
> +					   struct settings *orig_settings,
> +					   struct job_list *orig_job_list)
> +{
> +	struct settings settings;
> +	struct job_list list;
> +	struct job_list_entry *entry;
> +	int dirfd, resdirfd, fd, i;
> +
> +	init_settings(&settings);
> +	init_job_list(&list);
> +
> +	if ((dirfd = open(orig_settings->results_path, O_DIRECTORY | O_RDONLY)) < 0)
> +		return false;
> +
> +	if (!read_settings(&settings, dirfd) ||
> +	    !read_job_list(&list, dirfd)) {
> +		close(dirfd);
> +		return false;
> +	}
> +
> +	for (i = list.size; i >= 0; i--) {
> +		char name[32];
> +
> +		snprintf(name, 32, "%d", i);
> +		if ((resdirfd = openat(dirfd, name, O_DIRECTORY | O_RDONLY)) >= 0) {
> +			break;
> +		}
> +	}
> +
> +	if (i < 0) {
> +		/* Nothing has been executed yet, state is fine as is */
> +		goto success;
> +	}
> +
> +	entry = &list.entries[i];
> +	state->next = i;
> +	if ((fd = openat(resdirfd, filenames[_F_JOURNAL], O_RDONLY)) >= 0) {
> +		if (!prune_from_journal(entry, fd)) {
> +			/*
> +			 * The test does not have subtests, or
> +			 * incompleted before the first subtest
> +			 * began. Either way, not suitable to
> +			 * re-run.
> +			 */
> +			state->next = i + 1;
> +		} else if (entry->binary[0] == '\0') {
> +			/* This test is fully completed */
> +			state->next = i + 1;
> +		}
> +
> +		close(fd);
> +	}
> +
> + success:
> +	free_settings(orig_settings);
> +	free_job_list(orig_job_list);
> +	*orig_settings = settings;
> +	*orig_job_list = list;
> +	close(resdirfd);
> +	close(dirfd);
> +
> +	return true;
> +}
> +
> +bool initialize_execute_state(struct execute_state *state,
> +			      struct settings *settings,
> +			      struct job_list *job_list)
> +{
> +	memset(state, 0, sizeof(*state));
> +
> +	if (!validate_settings(settings))
> +		return false;
> +
> +	if (settings->overwrite &&
> +	    !clear_old_results(settings->results_path))
> +		return false;
> +
> +	if (!serialize_settings(settings) ||
> +	    !serialize_job_list(job_list, settings))
> +		return initialize_execute_from_resume(state, settings, job_list);
> +
> +	return true;
> +}
> +
> +bool execute(struct execute_state *state,
> +	     struct settings *settings,
> +	     struct job_list *job_list)
> +{
> +	struct utsname unamebuf;
> +	int resdirfd, testdirfd, unamefd;
> +
> +	if ((resdirfd = open(settings->results_path, O_DIRECTORY | O_RDONLY)) < 0) {
> +		/* Initialize state should have done this */
> +		fprintf(stderr, "Error: Failure opening results path %s\n",
> +			settings->results_path);
> +		return false;
> +	}
> +
> +	if ((testdirfd = open(settings->test_root, O_DIRECTORY | O_RDONLY)) < 0) {
> +		fprintf(stderr, "Error: Failure opening test root %s\n",
> +			settings->test_root);
> +		close(resdirfd);
> +		return false;
> +	}
> +
> +	/* TODO: On resume, don't rewrite, verify that content matches current instead */
> +	if ((unamefd = openat(resdirfd, "uname.txt", O_CREAT | O_WRONLY | O_TRUNC, 0666)) < 0) {
> +		fprintf(stderr, "Error: Failure creating opening uname.txt: %s\n",
> +			strerror(errno));
> +		close(testdirfd);
> +		close(resdirfd);
> +		return false;
> +	}
> +
> +	init_watchdogs(settings);
> +
> +	if (!uname(&unamebuf)) {
> +		dprintf(unamefd, "%s %s %s %s %s\n",
> +			unamebuf.sysname,
> +			unamebuf.nodename,
> +			unamebuf.release,
> +			unamebuf.version,
> +			unamebuf.machine);
> +	} else {
> +		dprintf(unamefd, "uname() failed\n");
> +	}
> +	close(unamefd);
> +
> +	for (; state->next < job_list->size;
> +	     state->next++) {
> +		int result = execute_entry(state->next,
> +					   job_list->size,
> +					   settings,
> +					   &job_list->entries[state->next],
> +					   testdirfd, resdirfd);
> +		if (result <= 0) {
> +			close(testdirfd);
> +			close(resdirfd);
> +			close_watchdogs(settings);
> +			if (result < 0) {
> +				memset(state, 0, sizeof(*state));
> +				initialize_execute_from_resume(state, settings, job_list);
> +				return execute(state, settings, job_list);
> +			}
> +			return false;
> +		}
> +	}
> +
> +	close(testdirfd);
> +	close(resdirfd);
> +	close_watchdogs(settings);
> +	return true;
> +}

Awesome piece of work. Kudos for readability.
Tune in next week for the part two of the review ;-)

-Arek
_______________________________________________
igt-dev mailing list
igt-dev@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/igt-dev

Re: [igt-dev] [PATCH i-g-t 3/4] runner: New test runner

[Arkadiusz Hiler]

On Mon, Apr 30, 2018 at 12:28:47PM +0300, Petri Latvala wrote:
> diff --git a/runner/job_list.c b/runner/job_list.c
> new file mode 100644
> index 00000000..ec942481
> --- /dev/null
> +++ b/runner/job_list.c
> @@ -0,0 +1,492 @@
> +#include <errno.h>
> +#include <fcntl.h>
> +#include <linux/limits.h>
> +#include <stdio.h>
> +#include <stdlib.h>
> +#include <string.h>
> +#include <sys/stat.h>
> +#include <sys/types.h>
> +#include <unistd.h>
> +
> +#include "job_list.h"
> +#include "igt_core.h"
> +
> +static bool matches_any(const char *needle, struct regex_list *haystacks)
> +{
> +	size_t i;
> +
> +	for (i = 0; i < haystacks->size; i++) {
> +		if (regexec(haystacks->regexes[i], needle,
> +			    (size_t)0, NULL, 0) == 0) {
> +			return true;
> +		}
> +	}
> +
> +	return false;
> +}

This is really odd choice of naming convention. Needle and haystack...

I know it is widely used when you want to find an element in a set, but
it does not quite fit here - you are checking whether at least on regex,
from a list, matches given string.

The confusion is threefold:
1. You have string that is a haystack you search for a "needle" matching
   regex.
2. But you also have a haystack (why plural, BTW?) of regexes, that you
   check for a match with the "needle" string.
3. You do not care what you find, you are just interested in a having a
   confirmed hit.

What about something like this:
static bool matches_any_regex(rexex_list_t re_list, const char *string)

> +static void add_job_list_entry(struct job_list *job_list,
> +			       char *binary,
> +			       char **subtests,
> +			       size_t subtest_count)
> +{
> +	struct job_list_entry *entry;
> +
> +	job_list->size++;
> +	job_list->entries = realloc(job_list->entries, job_list->size * sizeof(*job_list->entries));
> +	entry = &job_list->entries[job_list->size - 1];
> +
> +	entry->binary = binary;
> +	entry->subtests = subtests;
> +	entry->subtest_count = subtest_count;
> +}
> +
> +static void add_subtests(struct job_list *job_list, struct settings *settings,
> +			 char *binary,
> +			 struct regex_list *include, struct regex_list *exclude)
> +{
> +	FILE *p;
> +	char cmd[256] = {};
> +	char *subtestname;
> +	char **subtests = NULL;
> +	size_t num_subtests = 0;
> +	size_t rootlen = strlen(settings->test_root);
> +	size_t binarylen = strlen(binary);
> +	int idx, s;
> +
> +	if (rootlen + binarylen + strlen(" --list-subtests") + 1 > 256) {

sizof(cmd) or #define CMD_MAX

Also we can go --l route here.

> +		/* This shouldn't happen */
> +		fprintf(stderr, "Path to binary too long, ignoring: %s/%s\n",
> +			settings->test_root, binary);
> +		return;
> +	}
> +
> +	strcpy(cmd, settings->test_root);
> +	idx = rootlen;
> +	cmd[idx++] = '/';

I think you haven't accounted for this slash in the check above. You
have +1 for NULL termination, but where's the space for the slash?

We can be more explicit with that as well:
rootlen + strlen("/") + binarylen + strlen(" --list-subtests") + 1 > CMD_MAX

> +	strcpy(cmd + idx, binary);
> +	idx += binarylen;
> +	strcpy(cmd + idx, " --list-subtests");

Reused magic string.

Also why not just compose is with snprintf and save yourself some hassle?
Then check whether ret < CMD_MAX instead of doing the thing above.

> +	p = popen(cmd, "r");
> +	if (!p) {
> +		fprintf(stderr, "popen failed when executing %s: %s\n",
> +			cmd,
> +			strerror(errno));
> +		return;
> +	}
> +
> +	while (fscanf(p, "%ms", &subtestname) == 1) {
> +		if (exclude && exclude->size && matches_any(subtestname, exclude)) {
> +			free(subtestname);
> +			continue;
> +		}
> +
> +		if (include && include->size && !matches_any(subtestname, include)) {
> +			free(subtestname);
> +			continue;
> +		}
> +
> +		if (settings->multiple_mode) {
> +			num_subtests++;
> +			subtests = realloc(subtests, num_subtests * sizeof(*subtests));
> +			subtests[num_subtests - 1] = strdup(subtestname);
> +		} else {
> +			subtests = malloc(sizeof(*subtests));
> +			*subtests = strdup(subtestname);
> +			add_job_list_entry(job_list, strdup(binary), subtests, 1);
> +			subtests = NULL;
> +		}
> +
> +		free(subtestname);
> +	}
> +
> +	if (num_subtests) {
> +		add_job_list_entry(job_list, strdup(binary), subtests, num_subtests);
> +	}
> +
> +	s = pclose(p);
> +	if (s == 0) {
> +		return;
> +	} else if (s == -1) {
> +		fprintf(stderr, "popen error when executing %s: %s\n", binary, strerror(errno));
> +	} else if (WIFEXITED(s)) {
> +		if (WEXITSTATUS(s) == IGT_EXIT_INVALID) {
> +			/* No subtests on this one */
> +			if (exclude && exclude->size && matches_any(binary, exclude)) {
> +				return;
> +			}
> +			if (!include || !include->size || matches_any(binary, include)) {
> +				add_job_list_entry(job_list, strdup(binary), NULL, 0);
> +				return;
> +			}
> +		}
> +	} else {
> +		fprintf(stderr, "Test binary %s died unexpectedly\n", binary);
> +	}
> +}
> +
> +static bool filtered_job_list(struct job_list *job_list,
> +			      struct settings *settings,
> +			      int fd)
> +{
> +	FILE *f;
> +	char buf[128];
> +
> +	if (job_list->entries != NULL) {
> +		fprintf(stderr, "Job list not cleared, this shouldn't happen\n");

Full invocation and/or error details please.

> +		exit(1);
> +	}
> +
> +	f = fdopen(fd, "r");
> +
> +	while (fscanf(f, "%127s", buf) == 1) {
> +		if (!strcmp(buf, "TESTLIST") || !(strcmp(buf, "END")))
> +			continue;
> +
> +		/*
> +		 * If the binary name matches exclude filters, no
> +		 * subtests are added.
> +		 */
> +		if (settings->exclude_regexes.size && matches_any(buf, &settings->exclude_regexes))
> +			continue;
> +
> +		/*
> +		 * If the binary name matches include filters (or include filters not present),
> +		 * all subtests except those matching exclude filters are added.
> +		 */
> +		if (!settings->include_regexes.size || matches_any(buf, &settings->include_regexes)) {
> +			if (settings->multiple_mode && !settings->exclude_regexes.size)
> +				/*
> +				 * Optimization; we know that all
> +				 * subtests will be included, so we
> +				 * get to omit executing
> +				 * --list-subtests.
> +				 */
> +				add_job_list_entry(job_list, strdup(buf), NULL, 0);
> +			else
> +				add_subtests(job_list, settings, buf,
> +					     NULL, &settings->exclude_regexes);
> +			continue;
> +		}
> +
> +		/*
> +		 * Binary name doesn't match exclude or include filters.
> +		 */
> +		add_subtests(job_list, settings, buf,
> +			     &settings->include_regexes,
> +			     &settings->exclude_regexes);
> +	}
> +
> +	return job_list->size != 0;
> +}
> +
> +static bool job_list_from_test_list(struct job_list *job_list,
> +				    struct settings *settings)
> +{
> +	FILE *f;
> +	ssize_t read;
> +	char *line = NULL;
> +	size_t line_len = 0;
> +	struct job_list_entry entry = {};
> +	bool any = false;
> +
> +	if ((f = fopen(settings->test_list, "r")) == NULL) {
> +		fprintf(stderr, "Cannot open test list file %s\n", settings->test_list);
> +		return false;
> +	}
> +
> +	while ((read = getline(&line, &line_len, f))) {
> +		char *binary;
> +		char *delim;
> +
> +		if (read < 0) {
> +			if (errno == EINTR)
> +				continue;
> +			else
> +				break;
> +		}
> +
> +		/* # starts a comment */
> +		if ((delim = strchr(line, '#')) != NULL) {
> +			*delim = '\0';
> +		}
> +
> +		if (sscanf(line, "igt@%ms", &binary) == 1) {
> +			if ((delim = strchr(binary, '@')) != NULL) {
> +				*delim++ = '\0';
> +			}
> +
> +			if (!settings->multiple_mode) {
> +				char **subtests = NULL;
> +				if (delim) {
> +					subtests = malloc(sizeof(char*));
> +					subtests[0] = strdup(delim);
> +				}
> +				add_job_list_entry(job_list, strdup(binary),
> +						   subtests, (size_t)(subtests != NULL));
> +				any = true;
> +				free(binary);
> +				binary = NULL;
> +				continue;
> +			}
> +
> +			/*
> +			 * If the currently built entry has the same
> +			 * binary, add a subtest. Otherwise submit
> +			 * what's already built and start a new one.
> +			 */
> +			if (entry.binary && !strcmp(entry.binary, binary)) {
> +				if (!delim) {
> +					/* ... except we didn't get a subtest */
> +					fprintf(stderr,
> +						"Error: Unexpected test without subtests "
> +						"after same test had subtests\n");
> +					free(binary);
> +					fclose(f);
> +					return false;
> +				}
> +				entry.subtest_count++;
> +				entry.subtests = realloc(entry.subtests,
> +							 entry.subtest_count *
> +							 sizeof(*entry.subtests));
> +				entry.subtests[entry.subtest_count - 1] = strdup(delim);
> +				free(binary);
> +				binary = NULL;
> +				continue;
> +			}
> +
> +			if (entry.binary) {
> +				add_job_list_entry(job_list, entry.binary, entry.subtests, entry.subtest_count);
> +				any = true;
> +			}
> +
> +			memset(&entry, 0, sizeof(entry));
> +			entry.binary = strdup(binary);
> +			if (delim) {
> +				entry.subtests = malloc(sizeof(*entry.subtests));
> +				entry.subtests[0] = strdup(delim);
> +				entry.subtest_count = 1;
> +			}
> +
> +			free(binary);
> +			binary = NULL;
> +		}
> +	}
> +
> +	if (entry.binary) {
> +		add_job_list_entry(job_list, entry.binary, entry.subtests, entry.subtest_count);
> +		any = true;
> +	}
> +
> +	free(line);
> +	fclose(f);
> +	return any;
> +}
> +
> +void init_job_list(struct job_list *job_list)
> +{
> +	memset(job_list, 0, sizeof(*job_list));
> +}
> +
> +void free_job_list(struct job_list *job_list)
> +{
> +	int i, k;
> +
> +	for (i = 0; i < job_list->size; i++) {
> +		struct job_list_entry *entry = &job_list->entries[i];
> +
> +		free(entry->binary);
> +		for (k = 0; k < entry->subtest_count; k++) {
> +			free(entry->subtests[k]);
> +		}
> +		free(entry->subtests);
> +	}
> +	free(job_list->entries);
> +	init_job_list(job_list);
> +}
> +
> +bool create_job_list(struct job_list *job_list,
> +		     struct settings *settings)
> +{
> +	int dirfd, fd;
> +	bool result;
> +
> +	if (!settings->test_root) {
> +		fprintf(stderr, "No test root set; this shouldn't happen\n");
> +		return false;
> +	}
> +
> +	free_job_list(job_list);
> +
> +	dirfd = open(settings->test_root, O_DIRECTORY | O_RDONLY);
> +	if (dirfd < 0) {
> +		fprintf(stderr, "Test directory %s cannot be opened\n", settings->test_root);
> +		return false;
> +	}
> +
> +	fd = openat(dirfd, "test-list.txt", O_RDONLY);
> +	if (fd < 0) {
> +		fprintf(stderr, "Cannot open %s/test-list.txt\n", settings->test_root);
> +		close(dirfd);
> +		return false;
> +	}
> +
> +	/*
> +	 * If a test_list is given (not to be confused with
> +	 * test-list.txt), we use it directly without making tests
> +	 * list their subtests. If include/exclude filters are given
> +	 * we filter them directly from the test_list.
> +	 */

Is is already to late to undo the damage and rename one of those two?

> +	if (settings->test_list)
> +		result = job_list_from_test_list(job_list, settings);
> +	else
> +		result = filtered_job_list(job_list, settings, fd);
> +
> +	close(fd);
> +	close(dirfd);
> +
> +	return result;
> +}
> +
> +static char joblist_filename[] = "joblist.txt";
> +bool serialize_job_list(struct job_list *job_list, struct settings *settings)
> +{
> +	int dirfd, fd;
> +	size_t i, k;
> +	FILE *f;
> +
> +	if (!settings->results_path) {
> +		fprintf(stderr, "No results-path set; this shouldn't happen\n");

> +		return false;
> +	}
> +
> +	if ((dirfd = open(settings->results_path, O_DIRECTORY | O_RDONLY)) < 0) {
> +		mkdir(settings->results_path, 0777);
> +		if ((dirfd = open(settings->results_path, O_DIRECTORY | O_RDONLY)) < 0) {
> +			fprintf(stderr, "Creating results-path failed\n");
> +			return false;
> +		}
> +	}
> +
> +	if ((fd = openat(dirfd, joblist_filename, O_RDONLY)) >= 0) {
> +		close(fd);
> +
> +		if (!settings->overwrite) {
> +			/* Serialization data already exists, not overwriting. */
> +			close(fd);
> +			close(dirfd);
> +			return false;
> +		}
> +
> +		if (unlinkat(dirfd, joblist_filename, 0) != 0) {
> +			fprintf(stderr, "Error overwriting old job list\n");
> +			close(dirfd);
> +			return false;
> +		}
> +	}
> +
> +	if ((fd = openat(dirfd, joblist_filename, O_CREAT | O_EXCL | O_WRONLY, 0666)) < 0) {
> +		fprintf(stderr, "Creating job list serialization file failed\n");
> +		return false;
> +	}
> +
> +	f = fdopen(fd, "w");
> +	if (!f) {
> +		close(fd);
> +		close(dirfd);
> +		return false;
> +	}
> +
> +	for (i = 0; i < job_list->size; i++) {
> +		struct job_list_entry *entry = &job_list->entries[i];
> +		fprintf(f, "%s", entry->binary);

Why not fputs if you are putting some strings without any formatting?

> +
> +		if (entry->subtest_count) {
> +			char *delim = "";
> +
> +			fprintf(f, " ");
> +
> +			for (k = 0; k < entry->subtest_count; k++) {
> +				fprintf(f, "%s%s", delim, entry->subtests[k]);
> +				delim = ",";
> +			}
> +		}
> +
> +		fprintf(f, "\n");
> +	}
> +
> +	if (settings->sync) {
> +		fsync(fd);
> +		fsync(dirfd);
> +	}
> +
> +	fclose(f);
> +	close(dirfd);
> +	return true;
> +}
> +
> +bool read_job_list(struct job_list *job_list, int dirfd)
> +{
> +	int fd;
> +	FILE *f;
> +	ssize_t read;
> +	char *line = NULL;
> +	size_t line_len = 0;
> +
> +	free_job_list(job_list);
> +
> +	if ((fd = openat(dirfd, joblist_filename, O_RDONLY)) < 0)
> +		return false;
> +
> +	f = fdopen(fd, "r");
> +	if (!f) {
> +		close(fd);
> +		return false;
> +	}
> +
> +	while ((read = getline(&line, &line_len, f))) {

There should be no case that getline() returns 0, so this is an obscure
way of doing an "infinite loop".

I guess more explicit way would be:

while (1) {
	if (getline(&line, &line_len, f) == -1)
	{
		if (errno == EINTR)
			continue;
		else
			break; /* EOF or worse */
	}

	/* ... */
}

This way you can also skip declaring `read`.
-Arek

> +		char *binary, *sublist, *comma;
> +		char **subtests = NULL;
> +		size_t num_subtests = 0, len;
> +
> +		if (read < 0) {
> +			if (errno == EINTR)
> +				continue;
> +			else
> +				break;
> +		}
> +
> +		len = strlen(line);
> +		if (len > 0 && line[len - 1] == '\n')
> +			line[len - 1] = '\0';
> +
> +		sublist = strchr(line, ' ');
> +		if (!sublist) {
> +			add_job_list_entry(job_list, strdup(line), NULL, 0);
> +			continue;
> +		}
> +
> +		*sublist++ = '\0';
> +		binary = strdup(line);
> +
> +		do {
> +			comma = strchr(sublist, ',');
> +			if (comma) {
> +				*comma++ = '\0';
> +			}
> +
> +			++num_subtests;
> +			subtests = realloc(subtests, num_subtests * sizeof(*subtests));
> +			subtests[num_subtests - 1] = strdup(sublist);
> +			sublist = comma;
> +		} while (comma != NULL);
> +
> +		add_job_list_entry(job_list, binary, subtests, num_subtests);
> +	}
> +
> +	free(line);
> +	fclose(f);
> +
> +	return true;
> +}
_______________________________________________
igt-dev mailing list
igt-dev@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/igt-dev

Re: [igt-dev] [PATCH i-g-t 3/4] runner: New test runner

[Daniel Vetter]

On Mon, Apr 30, 2018 at 12:28:47PM +0300, Petri Latvala wrote:
> This is a new test runner to replace piglit. Piglit has been very
> useful as a test runner, but certain improvements have been very
> difficult if possible at all in a generic test running framework.
> 
> Important improvements over piglit:
> 
> - Faster to launch. Being able to make assumptions about what we're
>   executing makes it possible to save significant amounts of time. For
>   example, a testlist file's line "igt@somebinary@somesubtest" already
>   has all the information we need to construct the correct command
>   line to execute that particular subtest, instead of listing all
>   subtests of all test binaries and mapping them to command
>   lines. Same goes for the regexp filters command line flags -t and
>   -x; If we use -x somebinaryname, we don't need to list subtests from
>   somebinaryname, we already know none of them will get executed.
> 
> - Logs of incomplete tests. Piglit collects test output to memory and
>   dumps them to a file when the test is complete. The new runner
>   writes all output to disk immediately.
> 
> - Ability to execute multiple subtests in one binary execution. This
>   was possible with piglit, but its semantics made it very hard to
>   implement in practice. For example, having a testlist file not only
>   selected a subset of tests to run, but also mandated that they be
>   executed in the same order.
> 
> - Flexible timeout support. Instead of mandating a time tests cannot
>   exceed, the new runner has a timeout on inactivity. Activity is
>   any output on the test's stdout or stderr, or kernel activity via
>   /dev/kmsg.
> 
> The runner is fairly piglit compatible. The command line is very
> similar, with a few additions. IGT_TEST_ROOT environment flag is still
> supported, but can also be set via command line (in place of igt.py in
> piglit command line).
> 
> The results are a set of log files, processed into a piglit-compatible
> results.json file (BZ2 compression TODO). There are some new fields in
> the json for extra information:
> 
> - "igt-version" contains the IGT version line. In
>   multiple-subtests-mode the version information is only printed once,
>   so it needs to be duplicated to all subtest results this way.
> - "dmesg-warnings" contains the dmesg lines that triggered a
>   dmesg-warn/dmesg-fail state.

Hm, I thought agreement was that if we do a new test runner then we
definitely don't want it to capture&analyze dmesg? That's why I've
originally done

https://patchwork.freedesktop.org/series/39263/

which you then took over. Is that still part of the overall plan, just not
yet there?
-Daniel

> - Runtime information will be different. Piglit takes a timestamp at
>   the beginning and at the end of execution for runtime. The new
>   runner uses the subtest output text. The binary execution time will
>   also be included; The key "igt@somebinary" will have the runtime of
>   the binary "somebinary", whereas "igt@somebinary@a" etc will have
>   the runtime of the subtests. Substracting the subtest runtimes from
>   the binary runtime yields the total time spent doing setup in
>   igt_fixture blocks.
> 
> Signed-off-by: Petri Latvala <petri.latvala@intel.com>
> Cc: Maarten Lankhorst <maarten.lankhorst@linux.intel.com>
> Cc: Arkadiusz Hiler <arkadiusz.hiler@intel.com>
> Cc: Tomi Sarvela <tomi.p.sarvela@intel.com>
> Cc: Martin Peres <martin.peres@linux.intel.com>
> ---

[ snip]

> +/*
> + * This regexp controls the kmsg handling. All kernel log records that
> + * have log level of warning or higher convert the result to
> + * dmesg-warn/dmesg-fail unless they match this regexp.
> + */
> +
> +#define _ "|"
> +static const char igt_dmesg_whitelist[] =
> +	"ACPI: button: The lid device is not compliant to SW_LID" _
> +	"ACPI: .*: Unable to dock!" _
> +	"IRQ [0-9]+: no longer affine to CPU[0-9]+" _
> +	"IRQ fixup: irq [0-9]+ move in progress, old vector [0-9]+" _
> +	/* i915 tests set module options, expected message */
> +	"Setting dangerous option [a-z_]+ - tainting kernel" _
> +	/* Raw printk() call, uses default log level (warn) */
> +	"Suspending console\\(s\\) \\(use no_console_suspend to debug\\)" _
> +	"atkbd serio[0-9]+: Failed to (deactivate|enable) keyboard on isa[0-9]+/serio[0-9]+" _
> +	"cache: parent cpu[0-9]+ should not be sleeping" _
> +	"hpet[0-9]+: lost [0-9]+ rtc interrupts" _
> +	/* i915 selftests terminate normally with ENODEV from the
> +	 * module load after the testing finishes, which produces this
> +	 * message.
> +	 */
> +	"i915: probe of [0-9:.]+ failed with error -25" _
> +	/* swiotbl warns even when asked not to */
> +	"mock: DMA: Out of SW-IOMMU space for [0-9]+ bytes" _
> +	"usb usb[0-9]+: root hub lost power or was reset"

The above really isn't stuff we should have in the runner :-)
-Daniel

> +	;
> +#undef _
> +
> +static regex_t re;
> +
> +static bool init_regex_whitelist()
> +{
> +	int status = -1;
> +
> +	if (status == -1) {
> +		if (regcomp(&re, igt_dmesg_whitelist, REG_EXTENDED | REG_NOSUB) != 0) {
> +			fprintf(stderr, "Cannot compile dmesg whitelist regexp\n");
> +			status = 0;
> +			return false;
> +		}
> +
> +		status = 1;
> +	}
> +
> +	return status;
> +}
> +
> +static bool fill_from_dmesg(int fd, char *binary,
> +			    struct subtests *subtests,
> +			    struct json_object *tests)
> +{
> +	char *line = NULL, *warnings = NULL, *dmesg = NULL;
> +	size_t linelen = 0, warningslen = 0, dmesglen = 0;
> +	struct json_object *current_test = NULL;
> +	FILE *f = fdopen(fd, "r");
> +	char *igt_name = NULL;
> +	ssize_t read;
> +	size_t i;
> +
> +	if (!f) {
> +		return false;
> +	}
> +
> +	if (init_regex_whitelist() != 1) {
> +		fclose(f);
> +		return false;
> +	}
> +
> +	while ((read = getline(&line, &linelen, f)) > 0) {
> +		char formatted[256];
> +		unsigned flags;
> +		unsigned long long seq, ts_usec;
> +		char continuation;
> +		char *message, *subtest;
> +		int s;
> +
> +		s = sscanf(line, "%u,%llu,%llu,%c;", &flags, &seq, &ts_usec, &continuation);
> +
> +		if (s != 4) {
> +			/*
> +			 * Machine readable key/value pairs begin with
> +			 * a space. We ignore them.
> +			 */
> +			if (line[0] != ' ') {
> +				fprintf(stderr, "Cannot parse kmsg record: %s\n", line);
> +			}
> +
> +			continue;
> +		}
> +
> +		message = strchr(line, ';');
> +		if (!message) {
> +			fprintf(stderr, "No ; found, this shouldn't happen\n");
> +			return false;
> +		}
> +		message++;
> +		snprintf(formatted, sizeof(formatted), "<%u> [%llu.%06llu] %s",
> +			 flags & 0x07, ts_usec / 1000000, ts_usec % 1000000, message);
> +
> +		if ((subtest = strstr(message, starting_subtest_dmesg)) != NULL) {
> +			if (current_test != NULL) {
> +				/* Done with the previous subtest, file up */
> +				json_object_object_add(current_test, "dmesg",
> +						       json_object_new_string_len(dmesg, dmesglen));
> +				if (warnings) {
> +					json_object_object_add(current_test, "dmesg-warnings",
> +							       json_object_new_string_len(warnings, warningslen));
> +					add_or_override_result(current_test, "pass", "dmesg-warn");
> +					add_or_override_result(current_test, "fail", "dmesg-fail");
> +					add_or_override_result(current_test, "warn", "dmesg-warn");
> +				}
> +				free(dmesg);
> +				free(warnings);
> +				dmesg = warnings = NULL;
> +				dmesglen = warningslen = 0;
> +			}
> +
> +			subtest += starting_subtest_dmesg_len;
> +			igt_name = gen_igt_name(binary, subtest);
> +			current_test = get_or_create_json_object(tests, igt_name);
> +		}
> +
> +		if ((flags & 0x07) <= 4 && continuation != 'c' &&
> +		    regexec(&re, message, (size_t)0, NULL, 0) == REG_NOMATCH) {
> +			append_line(&warnings, &warningslen, formatted);
> +		}
> +		append_line(&dmesg, &dmesglen, formatted);
> +	}
> +
> +	if (current_test != NULL) {
> +		json_object_object_add(current_test, "dmesg",
> +				       json_object_new_string_len(dmesg, dmesglen));
> +		if (warnings) {
> +			json_object_object_add(current_test, "dmesg-warnings",
> +					       json_object_new_string_len(warnings, warningslen));
> +			add_or_override_result(current_test, "pass", "dmesg-warn");
> +			add_or_override_result(current_test, "fail", "dmesg-fail");
> +			add_or_override_result(current_test, "warn", "dmesg-warn");
> +		}
> +	} else {
> +		/*
> +		 * Didn't get any subtest messages at all. If there
> +		 * are subtests, add all of the dmesg gotten to all of
> +		 * them.
> +		 */
> +		for (i = 0; i < subtests->size; i++) {
> +			igt_name = gen_igt_name(binary, subtests->names[i]);
> +			current_test = get_or_create_json_object(tests, igt_name);
> +			json_object_object_add(current_test, "dmesg",
> +					       json_object_new_string_len(dmesg, dmesglen));
> +			/*
> +			 * Don't bother with warnings, any subtests
> +			 * there are would have skip as their result
> +			 * anyway.
> +			 */
> +		}
> +
> +		if (i == 0) {
> +			/* There were no subtests */
> +			igt_name = gen_igt_name(binary, NULL);
> +			current_test = get_or_create_json_object(tests, igt_name);
> +			json_object_object_add(current_test, "dmesg",
> +					       json_object_new_string_len(dmesg, dmesglen));
> +			if (warnings) {
> +				json_object_object_add(current_test, "dmesg-warnings",
> +						       json_object_new_string_len(warnings, warningslen));
> +				add_or_override_result(current_test, "pass", "dmesg-warn");
> +				add_or_override_result(current_test, "fail", "dmesg-fail");
> +				add_or_override_result(current_test, "warn", "dmesg-warn");
> +			}
> +		}
> +	}
> +
> +	/*
> +	 * Add an empty string as the dmesg of all subtests that
> +	 * didn't get any dmesg yet.
> +	 */
> +	for (i = 0; i < subtests->size; i++) {
> +		igt_name = gen_igt_name(binary, subtests->names[i]);
> +		current_test = get_or_create_json_object(tests, igt_name);
> +		if (!json_object_object_get_ex(current_test, "dmesg", NULL)) {
> +			json_object_object_add(current_test, "dmesg",
> +					       json_object_new_string(""));
> +		}
> +	}
> +
> +	free(dmesg);
> +	free(warnings);
> +	fclose(f);
> +	return true;
> +}
> +
> +static char *result_from_exitcode(int exitcode)
> +{
> +	switch (exitcode) {
> +	case IGT_EXIT_TIMEOUT:
> +		return "timeout";
> +	case IGT_EXIT_SKIP:
> +		return "skip";
> +	case IGT_EXIT_SUCCESS:
> +		return "pass";
> +	case IGT_EXIT_INVALID:
> +		return "notrun";
> +	default:
> +		return "fail";
> +	}
> +}
> +
> +static void add_subtest(struct subtests *subtests, char *subtest)
> +{
> +	size_t len = strlen(subtest);
> +
> +	if (len == 0)
> +		return;
> +
> +	if (subtest[len - 1] == '\n')
> +		subtest[len - 1] = '\0';
> +
> +	subtests->size++;
> +	subtests->names = realloc(subtests->names, sizeof(*subtests->names) * subtests->size);
> +	subtests->names[subtests->size - 1] = subtest;
> +}
> +
> +static bool fill_from_journal(int fd, char *binary,
> +				  struct subtests *subtests,
> +				  struct json_object *tests)
> +{
> +	FILE *f = fdopen(fd, "r");
> +	char *line = NULL;
> +	size_t linelen = 0;
> +	ssize_t read;
> +	char exitline[] = "exit:";
> +	size_t exitlen = sizeof(exitline) - 1;
> +	char timeoutline[] = "timeout:";
> +	size_t timeoutlen = sizeof(timeoutline) - 1;
> +	int exitcode = 10000;
> +
> +	while ((read = getline(&line, &linelen, f)) > 0) {
> +		if (read >= exitlen && !memcmp(line, exitline, exitlen)) {
> +			char *p = strchr(line, '(');
> +			char *igt_name = gen_igt_name(binary, NULL);
> +			double time = 0.0;
> +			struct json_object *obj = get_or_create_json_object(tests, igt_name);
> +
> +			exitcode = atoi(line + exitlen);
> +
> +			if (p) {
> +				time = strtod(p + 1, NULL);
> +			}
> +
> +			add_runtime(obj, time);
> +		} else if (read >= timeoutlen && !memcmp(line, timeoutline, timeoutlen)) {
> +			if (subtests->size) {
> +				char *last_subtest = subtests->names[subtests->size - 1];
> +				char *igt_name = gen_igt_name(binary, last_subtest);
> +				char *p = strchr(line, '(');
> +				double time = 0.0;
> +				struct json_object *obj = get_or_create_json_object(tests, igt_name);
> +
> +				json_object_object_add(obj, "result",
> +						       json_object_new_string("timeout"));
> +
> +				if (p) {
> +					time = strtod(p + 1, NULL);
> +				}
> +
> +				add_runtime(obj, time);
> +
> +				igt_name = gen_igt_name(binary, NULL);
> +				obj = get_or_create_json_object(tests, igt_name);
> +				add_runtime(obj, time);
> +			}
> +		} else {
> +			add_subtest(subtests, strdup(line));
> +		}
> +	}
> +
> +	if (subtests->size == 0 && exitcode != 10000) {
> +		char *igt_name = gen_igt_name(binary, NULL);
> +		struct json_object *obj = get_or_create_json_object(tests, igt_name);
> +		char *result = result_from_exitcode(exitcode);
> +		json_object_object_add(obj, "result",
> +				       json_object_new_string(result));
> +	}
> +
> +	free(line);
> +	return true;
> +}
> +
> +static bool parse_test_directory(int dirfd, char *binary, struct json_object *tests)
> +{
> +	int fds[_F_LAST];
> +	struct subtests subtests = {};
> +
> +	if (!open_output_files(dirfd, fds, false)) {
> +		fprintf(stderr, "Error opening output files\n");
> +		return false;
> +	}
> +
> +	/* fill_from_journal also fills the subtests struct */
> +	if (!fill_from_journal(fds[_F_JOURNAL], binary, &subtests, tests)) {
> +		fprintf(stderr, "Error reading from journal\n");
> +		return false;
> +	}
> +
> +	/* Order of these is important */
> +	if (!fill_from_output(fds[_F_OUT], binary, true, &subtests, tests) ||
> +	    !fill_from_output(fds[_F_ERR], binary, false, &subtests, tests) ||
> +	    !fill_from_dmesg(fds[_F_DMESG], binary, &subtests, tests)) {
> +		fprintf(stderr, "Error parsing output files\n");
> +		return false;
> +	}
> +
> +	close_outputs(fds);
> +
> +	return true;
> +}
> +
> +bool generate_results(int dirfd)
> +{
> +	struct settings settings;
> +	struct job_list job_list;
> +	struct json_object *obj, *tests;
> +	int resultsfd, testdirfd, unamefd;
> +	const char *json_string;
> +	size_t i;
> +
> +	init_settings(&settings);
> +	init_job_list(&job_list);
> +
> +	if (!read_settings(&settings, dirfd)) {
> +		fprintf(stderr, "resultgen: Cannot parse settings\n");
> +		return false;
> +	}
> +
> +	if (!read_job_list(&job_list, dirfd)) {
> +		fprintf(stderr, "resultgen: Cannot parse job list\n");
> +		return false;
> +	}
> +
> +	/* TODO: settings.overwrite */
> +	if ((resultsfd = openat(dirfd, "results.json", O_WRONLY | O_CREAT | O_TRUNC, 0666)) < 0) {
> +		fprintf(stderr, "resultgen: Cannot create results file\n");
> +		return false;
> +	}
> +
> +	obj = json_object_new_object();
> +	json_object_object_add(obj, "__type__", json_object_new_string("TestrunResult"));
> +	json_object_object_add(obj, "results_version", json_object_new_int(9));
> +	json_object_object_add(obj, "name",
> +			       settings.name ?
> +			       json_object_new_string(settings.name) :
> +			       json_object_new_string(""));
> +
> +	if ((unamefd = openat(dirfd, "uname.txt", O_RDONLY)) >= 0) {
> +		char buf[128];
> +		ssize_t r = read(unamefd, buf, 128);
> +
> +		if (r > 0 && buf[r - 1] == '\n')
> +			r--;
> +
> +		json_object_object_add(obj, "uname",
> +				       json_object_new_string_len(buf, r));
> +		close(unamefd);
> +	}
> +
> +	/* lspci */
> +	/* results_version */
> +	/* glxinfo */
> +	/* wglinfo */
> +	/* clinfo */
> +	/* options */
> +	/* time_elapsed */
> +	/* totals */
> +
> +	tests = json_object_new_object();
> +	json_object_object_add(obj, "tests", tests);
> +
> +	for (i = 0; i < job_list.size; i++) {
> +		char name[16];
> +
> +		snprintf(name, 16, "%zd", i);
> +		if ((testdirfd = openat(dirfd, name, O_DIRECTORY | O_RDONLY)) < 0) {
> +			fprintf(stderr, "Warning: Cannot open result directory %s\n", name);
> +			break;
> +		}
> +
> +		if (!parse_test_directory(testdirfd, job_list.entries[i].binary, tests)) {
> +			close(resultsfd);
> +			return false;
> +		}
> +	}
> +
> +	json_string = json_object_to_json_string_ext(obj, JSON_C_TO_STRING_PRETTY);
> +	write(resultsfd, json_string, strlen(json_string));
> +	return true;
> +}
> +
> +bool generate_results_path(char *resultspath)
> +{
> +	int dirfd = open(resultspath, O_DIRECTORY | O_RDONLY);
> +
> +	if (dirfd < 0)
> +		return false;
> +
> +	return generate_results(dirfd);
> +}
> diff --git a/runner/resultgen.h b/runner/resultgen.h
> new file mode 100644
> index 00000000..83a0876b
> --- /dev/null
> +++ b/runner/resultgen.h
> @@ -0,0 +1,9 @@
> +#ifndef RUNNER_RESULTGEN_H
> +#define RUNNER_RESULTGEN_H
> +
> +#include <stdbool.h>
> +
> +bool generate_results(int dirfd);
> +bool generate_results_path(char *resultspath);
> +
> +#endif
> diff --git a/runner/results.c b/runner/results.c
> new file mode 100644
> index 00000000..3eb7cb15
> --- /dev/null
> +++ b/runner/results.c
> @@ -0,0 +1,26 @@
> +#include <fcntl.h>
> +#include <stdio.h>
> +#include <stdlib.h>
> +#include <sys/stat.h>
> +#include <sys/types.h>
> +
> +#include "resultgen.h"
> +
> +int main(int argc, char **argv)
> +{
> +	int dirfd;
> +
> +	if (argc < 2)
> +		exit(1);
> +
> +	dirfd = open(argv[1], O_DIRECTORY | O_RDONLY);
> +	if (dirfd < 0)
> +		exit(1);
> +
> +	if (generate_results(dirfd)) {
> +		printf("Results generated\n");
> +		exit(0);
> +	}
> +
> +	exit(1);
> +}
> diff --git a/runner/runner.c b/runner/runner.c
> new file mode 100644
> index 00000000..b685786a
> --- /dev/null
> +++ b/runner/runner.c
> @@ -0,0 +1,40 @@
> +#include <stdio.h>
> +#include <string.h>
> +
> +#include "settings.h"
> +#include "job_list.h"
> +#include "executor.h"
> +#include "resultgen.h"
> +
> +int main(int argc, char **argv)
> +{
> +	struct settings settings;
> +	struct job_list job_list;
> +	struct execute_state state;
> +
> +	init_settings(&settings);
> +	init_job_list(&job_list);
> +
> +	if (!parse_options(argc, argv, &settings)) {
> +		return 1;
> +	}
> +
> +	if (!create_job_list(&job_list, &settings)) {
> +		return 1;
> +	}
> +
> +	if (!initialize_execute_state(&state, &settings, &job_list)) {
> +		return 1;
> +	}
> +
> +	if (!execute(&state, &settings, &job_list)) {
> +		return 1;
> +	}
> +
> +	if (!generate_results_path(settings.results_path)) {
> +		return 1;
> +	}
> +
> +	printf("Done.\n");
> +	return 0;
> +}
> diff --git a/runner/settings.c b/runner/settings.c
> new file mode 100644
> index 00000000..50e4ea10
> --- /dev/null
> +++ b/runner/settings.c
> @@ -0,0 +1,506 @@
> +#include "settings.h"
> +
> +#include <fcntl.h>
> +#include <getopt.h>
> +#include <libgen.h>
> +#include <limits.h>
> +#include <stdio.h>
> +#include <stdlib.h>
> +#include <string.h>
> +#include <sys/types.h>
> +#include <sys/stat.h>
> +#include <unistd.h>
> +
> +enum {
> +	OPT_ABORT_ON_ERROR,
> +	OPT_TEST_LIST,
> +	OPT_IGNORE_MISSING,
> +	OPT_HELP = 'h',
> +	OPT_NAME = 'n',
> +	OPT_DRY_RUN = 'd',
> +	OPT_INCLUDE = 't',
> +	OPT_EXCLUDE = 'x',
> +	OPT_SYNC = 's',
> +	OPT_LOG_LEVEL = 'l',
> +	OPT_OVERWRITE = 'o',
> +	OPT_MULTIPLE = 'm',
> +	OPT_TIMEOUT = 'c',
> +	OPT_WATCHDOG = 'g',
> +};
> +
> +static struct {
> +	int level;
> +	const char *name;
> +} log_levels[] = {
> +	{ LOG_LEVEL_NORMAL, "normal" },
> +	{ LOG_LEVEL_QUIET, "quiet" },
> +	{ LOG_LEVEL_VERBOSE, "verbose" },
> +	{ 0, 0 },
> +};
> +
> +static bool set_log_level(struct settings* settings, const char *level)
> +{
> +	int c;
> +
> +	for (c = 0; log_levels[c].name != NULL; c++) {
> +		if (!strcmp(level, log_levels[c].name)) {
> +			settings->log_level = log_levels[c].level;
> +			return true;
> +		}
> +	}
> +
> +	return false;
> +}
> +
> +static const char *usage_str =
> +	"usage: runner [options] [test_root] results-path\n\n"
> +	"Options:\n"
> +	" Piglit compatible:\n"
> +	"  -h, --help            Show this help message and exit\n"
> +	"  -n <test name>, --name <test name>\n"
> +	"                        Name of this test run\n"
> +	"  -d, --dry-run         Do not execute the tests\n"
> +	"  -t <regex>, --include-tests <regex>\n"
> +	"                        Run only matching tests (can be used more than once)\n"
> +	"  -x <regex>, --exclude-tests <regex>\n"
> +	"                        Exclude matching tests (can be used more than once)\n"
> +	"  --abort-on-monitored-error\n"
> +	"                        Abort execution when a fatal condition is detected.\n"
> +	"                        <TODO>\n"
> +	"  -s, --sync            Sync results to disk after every test\n"
> +	"  -l {quiet,verbose,dummy}, --log-level {quiet,verbose,dummy}\n"
> +	"                        Set the logger verbosity level\n"
> +	"  --test-list TEST_LIST\n"
> +	"                        A file containing a list of tests to run\n"
> +	"  -o, --overwrite       If the results-path already exists, delete it\n"
> +	"  --ignore-missing      Ignored but accepted, for piglit compatibility\n"
> +	"\n"
> +	" Incompatible options:\n"
> +	"  -m, --multiple-mode   Run multiple subtests in the same binary execution.\n"
> +	"                        If a testlist file is given, consecutive subtests are\n"
> +	"                        run in the same execution if they are from the same\n"
> +	"                        binary. Note that in that case relative ordering of the\n"
> +	"                        subtest execution is dictated by the test binary, not\n"
> +	"                        the testlist\n"
> +	"  --inactivity-timeout <seconds>\n"
> +	"                        Kill the running test after <seconds> of inactivity in\n"
> +	"                        the test's stdout, stderr, or dmesg\n"
> +	"  --use-watchdog        Use hardware watchdog for lethal enforcement of the\n"
> +	"                        above timeout. Killing the test process is still\n"
> +	"                        attempted at timeout trigger.\n"
> +	"  [test_root]           Directory that contains the IGT tests. The environment\n"
> +	"                        variable IGT_TEST_ROOT will be used if set, overriding\n"
> +	"                        this option if given.\n"
> +	;
> +
> +static void usage(const char *extra_message, bool use_stderr)
> +{
> +	FILE *f = use_stderr ? stderr : stdout;
> +
> +	if (extra_message)
> +		fprintf(f, "%s\n\n", extra_message);
> +
> +	fprintf(f, "%s", usage_str);
> +}
> +
> +static bool add_regex(struct regex_list *list, char *new)
> +{
> +	regex_t *regex;
> +	size_t buflen;
> +	char *buf;
> +	int s;
> +
> +	regex = malloc(sizeof(*regex));
> +
> +	if ((s = regcomp(regex, new,
> +			 REG_EXTENDED | REG_NOSUB)) != 0) {
> +		buflen = regerror(s, regex, NULL, 0);
> +		buf = malloc(buflen);
> +		regerror(s, regex, buf, buflen);
> +		usage(buf, true);
> +
> +		free(buf);
> +		regfree(regex);
> +		free(regex);
> +		return false;
> +	}
> +
> +	list->regexes = realloc(list->regexes,
> +				(list->size + 1) * sizeof(*list->regexes));
> +	list->regex_strings = realloc(list->regex_strings,
> +				      (list->size + 1) * sizeof(*list->regex_strings));
> +	list->regexes[list->size] = regex;
> +	list->regex_strings[list->size] = new;
> +	list->size++;
> +
> +	return true;
> +}
> +
> +static void free_regexes(struct regex_list *regexes)
> +{
> +	size_t i;
> +
> +	for (i = 0; i < regexes->size; i++) {
> +		free(regexes->regex_strings[i]);
> +		regfree(regexes->regexes[i]);
> +		free(regexes->regexes[i]);
> +	}
> +	free(regexes->regex_strings);
> +	free(regexes->regexes);
> +}
> +
> +static bool readable_file(char *filename)
> +{
> +	return !access(filename, R_OK);
> +}
> +
> +void init_settings(struct settings *settings)
> +{
> +	memset(settings, 0, sizeof(*settings));
> +}
> +
> +void free_settings(struct settings *settings)
> +{
> +	free(settings->test_list);
> +	free(settings->name);
> +	free(settings->test_root);
> +	free(settings->results_path);
> +
> +	free_regexes(&settings->include_regexes);
> +	free_regexes(&settings->exclude_regexes);
> +
> +	init_settings(settings);
> +}
> +
> +bool parse_options(int argc, char **argv,
> +		   struct settings *settings)
> +{
> +	int c;
> +	char *env_test_root;
> +
> +	static struct option long_options[] = {
> +		{"help", no_argument, NULL, OPT_HELP},
> +		{"name", required_argument, NULL, OPT_NAME},
> +		{"dry-run", no_argument, NULL, OPT_DRY_RUN},
> +		{"include-tests", required_argument, NULL, OPT_INCLUDE},
> +		{"exclude-tests", required_argument, NULL, OPT_EXCLUDE},
> +		{"abort-on-monitored-error", no_argument, NULL, OPT_ABORT_ON_ERROR},
> +		{"sync", no_argument, NULL, OPT_SYNC},
> +		{"log-level", required_argument, NULL, OPT_LOG_LEVEL},
> +		{"test-list", required_argument, NULL, OPT_TEST_LIST},
> +		{"overwrite", no_argument, NULL, OPT_OVERWRITE},
> +		{"ignore-missing", no_argument, NULL, OPT_IGNORE_MISSING},
> +		{"multiple-mode", no_argument, NULL, OPT_MULTIPLE},
> +		{"inactivity-timeout", required_argument, NULL, OPT_TIMEOUT},
> +		{"use-watchdog", no_argument, NULL, OPT_WATCHDOG},
> +		{ 0, 0, 0, 0},
> +	};
> +
> +	free_settings(settings);
> +
> +	optind = 1;
> +
> +	while ((c = getopt_long(argc, argv, "hn:dt:x:sl:om", long_options, NULL)) != -1) {
> +		switch (c) {
> +		case OPT_HELP:
> +			usage(NULL, false);
> +			goto error;
> +		case OPT_NAME:
> +			settings->name = strdup(optarg);
> +			break;
> +		case OPT_DRY_RUN:
> +			settings->dry_run = true;
> +			break;
> +		case OPT_INCLUDE:
> +			if (!add_regex(&settings->include_regexes, strdup(optarg)))
> +				goto error;
> +			break;
> +		case OPT_EXCLUDE:
> +			if (!add_regex(&settings->exclude_regexes, strdup(optarg)))
> +				goto error;
> +			break;
> +		case OPT_ABORT_ON_ERROR:
> +			settings->abort_on_error = true;
> +			break;
> +		case OPT_SYNC:
> +			settings->sync = true;
> +			break;
> +		case OPT_LOG_LEVEL:
> +			if (!set_log_level(settings, optarg)) {
> +				usage("Cannot parse log level", true);
> +				goto error;
> +			}
> +			break;
> +		case OPT_TEST_LIST:
> +			settings->test_list = absolute_path(optarg);
> +			break;
> +		case OPT_OVERWRITE:
> +			settings->overwrite = true;
> +			break;
> +		case OPT_IGNORE_MISSING:
> +			/* Ignored, piglit compatibility */
> +			break;
> +		case OPT_MULTIPLE:
> +			settings->multiple_mode = true;
> +			break;
> +		case OPT_TIMEOUT:
> +			settings->inactivity_timeout = atoi(optarg);
> +			break;
> +		case OPT_WATCHDOG:
> +			settings->use_watchdog = true;
> +			break;
> +		case '?':
> +			usage(NULL, true);
> +			goto error;
> +		default:
> +			usage("Cannot parse options", true);
> +			goto error;
> +		}
> +	}
> +
> +	switch (argc - optind) {
> +	case 2:
> +		settings->test_root = absolute_path(argv[optind]);
> +		++optind;
> +		/* fallthrough */
> +	case 1:
> +		settings->results_path = absolute_path(argv[optind]);
> +		break;
> +	case 0:
> +		usage("Results-path missing", true);
> +		goto error;
> +	default:
> +		usage("Extra arguments after results-path", true);
> +		goto error;
> +	}
> +
> +	if ((env_test_root = getenv("IGT_TEST_ROOT")) != NULL) {
> +		free(settings->test_root);
> +		settings->test_root = absolute_path(env_test_root);
> +	}
> +
> +	if (!settings->test_root) {
> +		usage("Test root not set", true);
> +		goto error;
> +	}
> +
> +	if (!settings->name) {
> +		char *name = strdup(settings->results_path);
> +		settings->name = strdup(basename(name));
> +		free(name);
> +	}
> +
> +	return true;
> +
> + error:
> +	free_settings(settings);
> +	return false;
> +}
> +
> +bool validate_settings(struct settings *settings)
> +{
> +	int dirfd, fd;
> +
> +	if (settings->test_list && !readable_file(settings->test_list)) {
> +		usage("Cannot open test-list file", true);
> +		return false;
> +	}
> +
> +	if (!settings->results_path) {
> +		usage("No results-path set; this shouldn't happen", true);
> +		return false;
> +	}
> +
> +	if (!settings->test_root) {
> +		usage("No test root set; this shouldn't happen", true);
> +		return false;
> +	}
> +
> +	dirfd = open(settings->test_root, O_DIRECTORY | O_RDONLY);
> +	if (dirfd < 0) {
> +		fprintf(stderr, "Test directory %s cannot be opened\n", settings->test_root);
> +		return false;
> +	}
> +
> +	fd = openat(dirfd, "test-list.txt", O_RDONLY);
> +	if (fd < 0) {
> +		fprintf(stderr, "Cannot open %s/test-list.txt\n", settings->test_root);
> +		close(dirfd);
> +		return false;
> +	}
> +
> +	close(fd);
> +	close(dirfd);
> +
> +	return true;
> +}
> +
> +char *absolute_path(char *path)
> +{
> +	char *result = NULL;
> +	char *tmppath, *tmpname;
> +
> +	result = realpath(path, NULL);
> +	if (result != NULL)
> +		return result;
> +
> +	tmppath = strdup(path);
> +	tmpname = dirname(tmppath);
> +	free(result);
> +	result = realpath(tmpname, NULL);
> +	free(tmppath);
> +
> +	if (result != NULL) {
> +		tmppath = strdup(path);
> +		tmpname = basename(tmppath);
> +		strcat(result, "/");
> +		strcat(result, tmpname);
> +		free(tmppath);
> +		return result;
> +	}
> +
> +	free(result);
> +	return NULL;
> +}
> +
> +static char settings_filename[] = "metadata.txt";
> +bool serialize_settings(struct settings *settings)
> +{
> +#define SERIALIZE_LINE(f, s, name, format) fprintf(f, "%s : " format "\n", #name, s->name)
> +
> +	int dirfd, fd;
> +	FILE *f;
> +
> +	if (!settings->results_path) {
> +		usage("No results-path set; this shouldn't happen", true);
> +		return false;
> +	}
> +
> +	if ((dirfd = open(settings->results_path, O_DIRECTORY | O_RDONLY)) < 0) {
> +		mkdir(settings->results_path, 0755);
> +		if ((dirfd = open(settings->results_path, O_DIRECTORY | O_RDONLY)) < 0) {
> +			usage("Creating results-path failed", true);
> +			return false;
> +		}
> +	}
> +
> +	if ((fd = openat(dirfd, settings_filename, O_RDONLY)) >= 0) {
> +		close(fd);
> +
> +		if (!settings->overwrite) {
> +			/* Serialization data already exists, not overwriting */
> +			close(dirfd);
> +			return false;
> +		}
> +
> +		if (unlinkat(dirfd, settings_filename, 0) != 0) {
> +			usage("Error overwriting old settings metadata", true);
> +			close(dirfd);
> +			return false;
> +		}
> +	}
> +
> +	if ((fd = openat(dirfd, settings_filename, O_CREAT | O_EXCL | O_WRONLY, 0666)) < 0) {
> +		usage("Creating settings serialization file failed", true);
> +		close(dirfd);
> +		return false;
> +	}
> +
> +	f = fdopen(fd, "w");
> +	if (!f) {
> +		close(fd);
> +		close(dirfd);
> +		return false;
> +	}
> +
> +	SERIALIZE_LINE(f, settings, abort_on_error, "%d");
> +	if (settings->test_list)
> +		SERIALIZE_LINE(f, settings, test_list, "%s");
> +	if (settings->name)
> +		SERIALIZE_LINE(f, settings, name, "%s");
> +	SERIALIZE_LINE(f, settings, dry_run, "%d");
> +	SERIALIZE_LINE(f, settings, sync, "%d");
> +	SERIALIZE_LINE(f, settings, log_level, "%d");
> +	SERIALIZE_LINE(f, settings, overwrite, "%d");
> +	SERIALIZE_LINE(f, settings, multiple_mode, "%d");
> +	SERIALIZE_LINE(f, settings, inactivity_timeout, "%d");
> +	SERIALIZE_LINE(f, settings, use_watchdog, "%d");
> +	SERIALIZE_LINE(f, settings, test_root, "%s");
> +	SERIALIZE_LINE(f, settings, results_path, "%s");
> +
> +	if (settings->sync) {
> +		fsync(fd);
> +		fsync(dirfd);
> +	}
> +
> +	fclose(f);
> +	close(dirfd);
> +	return true;
> +
> +#undef SERIALIZE_LINE
> +}
> +
> +static char *maybe_strdup(char *str)
> +{
> +	if (!str)
> +		return NULL;
> +
> +	return strdup(str);
> +}
> +
> +bool read_settings(struct settings *settings, int dirfd)
> +{
> +#define PARSE_LINE(s, name, val, field, write) \
> +	if (!strcmp(name, #field)) {	       \
> +		s->field = write;	       \
> +		free(name);		       \
> +		free(val);		       \
> +		name = val = NULL;	       \
> +		continue;		       \
> +	}
> +
> +	int fd;
> +	FILE *f;
> +	char *name = NULL, *val = NULL;
> +
> +	free_settings(settings);
> +
> +	if ((fd = openat(dirfd, settings_filename, O_RDONLY)) < 0)
> +		return false;
> +
> +	f = fdopen(fd, "r");
> +	if (!f) {
> +		close(fd);
> +		return false;
> +	}
> +
> +	while (fscanf(f, "%ms : %ms", &name, &val) == 2) {
> +		int numval = atoi(val);
> +		PARSE_LINE(settings, name, val, abort_on_error, numval);
> +		PARSE_LINE(settings, name, val, test_list, maybe_strdup(val));
> +		PARSE_LINE(settings, name, val, name, maybe_strdup(val));
> +		PARSE_LINE(settings, name, val, dry_run, numval);
> +		PARSE_LINE(settings, name, val, sync, numval);
> +		PARSE_LINE(settings, name, val, log_level, numval);
> +		PARSE_LINE(settings, name, val, overwrite, numval);
> +		PARSE_LINE(settings, name, val, multiple_mode, numval);
> +		PARSE_LINE(settings, name, val, inactivity_timeout, numval);
> +		PARSE_LINE(settings, name, val, use_watchdog, numval);
> +		PARSE_LINE(settings, name, val, test_root, maybe_strdup(val));
> +		PARSE_LINE(settings, name, val, results_path, maybe_strdup(val));
> +
> +		printf("Warning: Unknown field in settings file: %s = %s\n",
> +		       name, val);
> +		free(name);
> +		free(val);
> +		name = val = NULL;
> +	}
> +
> +	free(name);
> +	free(val);
> +	fclose(f);
> +
> +	return true;
> +
> +#undef PARSE_LINE
> +}
> diff --git a/runner/settings.h b/runner/settings.h
> new file mode 100644
> index 00000000..9d1f03fb
> --- /dev/null
> +++ b/runner/settings.h
> @@ -0,0 +1,111 @@
> +#ifndef RUNNER_SETTINGS_H
> +#define RUNNER_SETTINGS_H
> +
> +#include <stdbool.h>
> +#include <stddef.h>
> +#include <sys/types.h>
> +#include <regex.h>
> +
> +enum {
> +	LOG_LEVEL_NORMAL = 0,
> +	LOG_LEVEL_QUIET = -1,
> +	LOG_LEVEL_VERBOSE = 1,
> +};
> +
> +struct regex_list {
> +	char **regex_strings;
> +	regex_t** regexes;
> +	size_t size;
> +};
> +
> +struct settings {
> +	bool abort_on_error;
> +	char *test_list;
> +	char *name;
> +	bool dry_run;
> +	struct regex_list include_regexes;
> +	struct regex_list exclude_regexes;
> +	bool sync;
> +	int log_level;
> +	bool overwrite;
> +	bool multiple_mode;
> +	int inactivity_timeout;
> +	bool use_watchdog;
> +	char *test_root;
> +	char *results_path;
> +};
> +
> +/**
> + * init_settings:
> + *
> + * Initializes a settings object to an empty state (all values NULL, 0
> + * or false).
> + *
> + * @settings: Object to initialize. Storage for it must exist.
> + */
> +void init_settings(struct settings *settings);
> +
> +/**
> + * free_settings:
> + *
> + * Releases all allocated resources for a settings object and
> + * initializes it to an empty state (see #init_settings).
> + *
> + * @settings: Object to release and initialize.
> + */
> +void free_settings(struct settings *settings);
> +
> +/**
> + * parse_options:
> + *
> + * Parses command line options and sets the settings object to
> + * designated values.
> + *
> + * The function can be called again on the same settings object. The
> + * old values will be properly released and cleared. On a parse
> + * failure, the settings object will be in an empty state (see
> + * #init_settings) and usage instructions will be printed with an
> + * error message.
> + *
> + * @argc: Argument count
> + * @argv: Argument array. First element is the program name.
> + * @settings: Settings object to fill with values. Must have proper
> + * storage.
> + *
> + * Returns: True on successful parse, false on error.
> + */
> +bool parse_options(int argc, char **argv,
> +		   struct settings *settings);
> +
> +/**
> + * validate_settings:
> + *
> + * Checks the settings object against the system to see if executing
> + * on it can be done. Checks pathnames for existence and access
> + * rights. Note that this function will not check that the designated
> + * job listing (through a test-list file or the -t/-x flags) yields a
> + * non-zero amount of testing to be done. On errors, usage
> + * instructions will be printed with an error message.
> + *
> + * @settings: Settings object to check.
> + *
> + * Returns: True on valid settings, false on any error.
> + */
> +bool validate_settings(struct settings *settings);
> +
> +/* TODO: Better place for this */
> +char *absolute_path(char *path);
> +
> +/**
> + * serialize_settings:
> + *
> + * Serializes the settings object to a file in the results_path
> + * directory.
> + *
> + * @settings: Settings object to serialize.
> + */
> +bool serialize_settings(struct settings *settings);
> +
> +bool read_settings(struct settings *settings, int dirfd);
> +
> +#endif
> -- 
> 2.14.1
> 
> _______________________________________________
> igt-dev mailing list
> igt-dev@lists.freedesktop.org
> https://lists.freedesktop.org/mailman/listinfo/igt-dev

-- 
Daniel Vetter
Software Engineer, Intel Corporation
http://blog.ffwll.ch
_______________________________________________
igt-dev mailing list
igt-dev@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/igt-dev

Re: [igt-dev] [PATCH i-g-t 3/4] runner: New test runner

[Petri Latvala]

On Mon, May 07, 2018 at 08:34:16PM +0200, Daniel Vetter wrote:
> On Mon, Apr 30, 2018 at 12:28:47PM +0300, Petri Latvala wrote:
> > This is a new test runner to replace piglit. Piglit has been very
> > useful as a test runner, but certain improvements have been very
> > difficult if possible at all in a generic test running framework.
> > 
> > Important improvements over piglit:
> > 
> > - Faster to launch. Being able to make assumptions about what we're
> >   executing makes it possible to save significant amounts of time. For
> >   example, a testlist file's line "igt@somebinary@somesubtest" already
> >   has all the information we need to construct the correct command
> >   line to execute that particular subtest, instead of listing all
> >   subtests of all test binaries and mapping them to command
> >   lines. Same goes for the regexp filters command line flags -t and
> >   -x; If we use -x somebinaryname, we don't need to list subtests from
> >   somebinaryname, we already know none of them will get executed.
> > 
> > - Logs of incomplete tests. Piglit collects test output to memory and
> >   dumps them to a file when the test is complete. The new runner
> >   writes all output to disk immediately.
> > 
> > - Ability to execute multiple subtests in one binary execution. This
> >   was possible with piglit, but its semantics made it very hard to
> >   implement in practice. For example, having a testlist file not only
> >   selected a subset of tests to run, but also mandated that they be
> >   executed in the same order.
> > 
> > - Flexible timeout support. Instead of mandating a time tests cannot
> >   exceed, the new runner has a timeout on inactivity. Activity is
> >   any output on the test's stdout or stderr, or kernel activity via
> >   /dev/kmsg.
> > 
> > The runner is fairly piglit compatible. The command line is very
> > similar, with a few additions. IGT_TEST_ROOT environment flag is still
> > supported, but can also be set via command line (in place of igt.py in
> > piglit command line).
> > 
> > The results are a set of log files, processed into a piglit-compatible
> > results.json file (BZ2 compression TODO). There are some new fields in
> > the json for extra information:
> > 
> > - "igt-version" contains the IGT version line. In
> >   multiple-subtests-mode the version information is only printed once,
> >   so it needs to be duplicated to all subtest results this way.
> > - "dmesg-warnings" contains the dmesg lines that triggered a
> >   dmesg-warn/dmesg-fail state.
> 
> Hm, I thought agreement was that if we do a new test runner then we
> definitely don't want it to capture&analyze dmesg? That's why I've
> originally done
> 
> https://patchwork.freedesktop.org/series/39263/
> 
> which you then took over. Is that still part of the overall plan, just not
> yet there?


That patch series is facing a deadend, albeit constructed with plywood
so could be punched through with enough violence used.

With the test itself monitoring /dev/kmsg and stuffing things to igt
log buffer, we get a race condition for capturing the last few kernel
messages before the test exits. That could be worked around by holding
the exit until we have read all of it (like this runner needs to do
anyway) but that means all tests incur an overhead at
exit. Considering we've been optimizing out milliseconds from test
runtimes...

Another problem is that just stuffing kernel messages to IGT log
buffer doesn't give any data for (automatic) processing of incomplete
tests, as the buffer is only dumped on a failure. Martin is eagerly
waiting for the ability to classify incompletes based on their dmesg.

Driver-specific dmesg filters are hard to do from the test
itself. Opening the driver happens at an arbitrary point in time, if
at all, an arbitrary amount of times.

Good news though is that we're able to do both styles of kmsg monitors
later on, active at the same time.


> -Daniel
> 
> > - Runtime information will be different. Piglit takes a timestamp at
> >   the beginning and at the end of execution for runtime. The new
> >   runner uses the subtest output text. The binary execution time will
> >   also be included; The key "igt@somebinary" will have the runtime of
> >   the binary "somebinary", whereas "igt@somebinary@a" etc will have
> >   the runtime of the subtests. Substracting the subtest runtimes from
> >   the binary runtime yields the total time spent doing setup in
> >   igt_fixture blocks.
> > 
> > Signed-off-by: Petri Latvala <petri.latvala@intel.com>
> > Cc: Maarten Lankhorst <maarten.lankhorst@linux.intel.com>
> > Cc: Arkadiusz Hiler <arkadiusz.hiler@intel.com>
> > Cc: Tomi Sarvela <tomi.p.sarvela@intel.com>
> > Cc: Martin Peres <martin.peres@linux.intel.com>
> > ---
> 
> [ snip]
> 
> > +/*
> > + * This regexp controls the kmsg handling. All kernel log records that
> > + * have log level of warning or higher convert the result to
> > + * dmesg-warn/dmesg-fail unless they match this regexp.
> > + */
> > +
> > +#define _ "|"
> > +static const char igt_dmesg_whitelist[] =
> > +	"ACPI: button: The lid device is not compliant to SW_LID" _
> > +	"ACPI: .*: Unable to dock!" _
> > +	"IRQ [0-9]+: no longer affine to CPU[0-9]+" _
> > +	"IRQ fixup: irq [0-9]+ move in progress, old vector [0-9]+" _
> > +	/* i915 tests set module options, expected message */
> > +	"Setting dangerous option [a-z_]+ - tainting kernel" _
> > +	/* Raw printk() call, uses default log level (warn) */
> > +	"Suspending console\\(s\\) \\(use no_console_suspend to debug\\)" _
> > +	"atkbd serio[0-9]+: Failed to (deactivate|enable) keyboard on isa[0-9]+/serio[0-9]+" _
> > +	"cache: parent cpu[0-9]+ should not be sleeping" _
> > +	"hpet[0-9]+: lost [0-9]+ rtc interrupts" _
> > +	/* i915 selftests terminate normally with ENODEV from the
> > +	 * module load after the testing finishes, which produces this
> > +	 * message.
> > +	 */
> > +	"i915: probe of [0-9:.]+ failed with error -25" _
> > +	/* swiotbl warns even when asked not to */
> > +	"mock: DMA: Out of SW-IOMMU space for [0-9]+ bytes" _
> > +	"usb usb[0-9]+: root hub lost power or was reset"
> 
> The above really isn't stuff we should have in the runner :-)

Yep, forgot a TODO. Those belong in general-suppressions.txt and
i915-suppressions.txt. Or tests/intel-ci/dmesg-suppressions.txt.


-- 
Petri Latvala
_______________________________________________
igt-dev mailing list
igt-dev@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/igt-dev

Re: [igt-dev] [PATCH i-g-t 3/4] runner: New test runner

[Arkadiusz Hiler]

On Mon, Apr 30, 2018 at 12:28:47PM +0300, Petri Latvala wrote:
> diff --git a/runner/resultgen.c b/runner/resultgen.c
> new file mode 100644
> index 00000000..978c07be
> --- /dev/null
> +++ b/runner/resultgen.c
> @@ -0,0 +1,864 @@
> +#include <ctype.h>
> +#include <fcntl.h>
> +#include <stdio.h>
> +#include <string.h>
> +#include <sys/mman.h>
> +#include <sys/stat.h>
> +#include <sys/types.h>
> +#include <unistd.h>
> +
> +#include <json.h>
> +
> +#include "resultgen.h"
> +#include "settings.h"
> +#include "executor.h"
> +#include "igt_core.h"
> +
> +static char subtest_result_beg[] = "Subtest ";
> +static size_t subtest_result_len = sizeof(subtest_result_beg) - 1;

_beg_

> +static char starting_subtest[] = "Starting subtest: ";
> +static size_t starting_len = sizeof(starting_subtest) - 1;

_subtest_

> +static char starting_subtest_dmesg[] = ": starting subtest ";
> +static size_t starting_subtest_dmesg_len = sizeof(starting_subtest_dmesg) - 1;

ALL_CAPS or something else for readability please. Otherwise it is easy
to get lost track of what is what in some of the parsing functions
down below, due to similar naming.

const?

> +struct subtests
> +{
> +	char **names;
> +	size_t size;
> +};

A comment that we are operating on a mmaped "buf" that may be not null
terminated would be much appreciated. Then the reason for passing end
pointer and the choice of mem* family of functions are much more
obvious.

> +static char *find_line(char *haystack, char *needle, size_t needle_size, char *end)
_starting_with?
> +{
> +	char *line = haystack;

Useless assignment. I don't think it increases readability.

> +
> +	while (line < end) {
> +		char *line_end = memchr(line, '\n', end - line);
> +
> +		if (end - line < needle_size)
> +			return NULL;
> +		if (!memcmp(line, needle, needle_size))
> +			return line;
> +		if (line_end == NULL)
> +			return NULL;
> +		line = line_end + 1;
> +	}
> +
> +	return NULL;
> +}
> +
> +static char *find_either_line(char *haystack,
> +			      char *needle1, size_t needle1_size,
> +			      char *needle2, size_t needle2_size,
> +			      char *end)
> +{
> +	char *line = haystack;
> +
> +	while (line < end) {
> +		char *line_end = memchr(line, '\n', end - line);
> +		size_t linelen = line_end != NULL ? line_end - line : end - line;
> +		if ((linelen >= needle1_size && !memcmp(line, needle1, needle1_size)) ||
> +		    (linelen >= needle2_size && !memcmp(line, needle2, needle2_size)))
> +			return line;
> +
> +		if (line_end == NULL)
> +			return NULL;
> +
> +		line = line_end + 1;
> +	}
> +
> +	return NULL;
> +}
> +
> +static char *next_line(char *line, char *bufend)
> +{
> +	char *ret;
> +
> +	if (!line)
> +		return NULL;
> +
> +	ret = memchr(line, '\n', bufend - line);
> +	if (ret)
> +		ret++;
> +
> +	return ret;
> +}

This suffers from being prone to
*bufend == '\n'

Then ret is one after the end. You always have to remember to check that.

In some cases you don't do that and defer the returning end of buffer
handling to find_* functions, which does not help with readability.

> +static size_t count_lines(char *buf, char *bufend)
> +{
> +	size_t ret = 0;
> +	while ((buf = next_line(buf, bufend)) != NULL)
> +		ret++;
> +
> +	return ret;
> +}

If you need intact next_line implementation for line counting, please
create __next_line that does not do the NULL check mentioned above.

or even:

while ((buf = memchr(line, '\n', bufend - buf)) != NULL)
	ret++;

return ret;

> +static char *lowercase(char *str)
> +{
> +	char *ret = malloc(strlen(str) + 1);
> +	char *q = ret;
> +
> +	while (*str) {
> +		if (isspace(*str))
> +			break;
> +
> +		*q++ = tolower(*str++);
> +	}
> +	*q = '\0';
> +
> +	return ret;
> +}
> +
> +static void append_line(char **buf, size_t *buflen, char *line)
> +{
> +	size_t linelen = strlen(line);
> +
> +	*buf = realloc(*buf, *buflen + linelen + 1);
> +	strcpy(*buf + *buflen, line);
> +	*buflen += linelen;
> +}
> +
> +static char *gen_igt_name(char *binary, char *subtest)

This clashes with gen, as in graphics, causing my brain to throw an
exception.

> +{
> +	static char namebuf[256];
> +
> +	char *lc_binary = lowercase(binary);
> +	char *lc_subtest = NULL;
> +
> +	if (!subtest) {
> +		snprintf(namebuf, 256, "igt@%s", lc_binary);

sizeof()

> +		free(lc_binary);
> +		return namebuf;
> +	}
> +
> +	lc_subtest = lowercase(subtest);
> +
> +	snprintf(namebuf, 256, "igt@%s@%s", lc_binary, lc_subtest);

sizeof()

> +
> +	free(lc_binary);
> +	free(lc_subtest);
> +	return namebuf;
> +}
> +
> +static struct {
> +	char *output_str;
> +	char *result_str;
> +} resultmap[] = {
> +	{ "SUCCESS", "pass" },
> +	{ "SKIP", "skip" },
> +	{ "FAIL", "fail" },
> +	{ "CRASH", "crash" },
> +	{ "TIMEOUT", "timeout" },
> +};
> +static void parse_result_string(char *resultstring, size_t len, char **result, double *time)
> +{
> +	size_t i;
> +	size_t wordlen = 0;
> +
> +	while (wordlen < len) {

&& !isspace(resultstring[wordlen])

> +		if (isspace(resultstring[wordlen]))
> +			break;
> +		wordlen++;
> +	}
> +
> +	for (i = 0; i < (sizeof(resultmap) / sizeof(resultmap[0])); i++) {
> +		if (!strncmp(resultstring, resultmap[i].output_str, wordlen)) {
> +			*result = resultmap[i].result_str;
> +			break;
> +		}
> +	}
> +
> +	/* If we don't find a result line, the test didn't finish. */
> +	if (!*result)

I would also reset result to NULL before previous loop, to make it
locally obvious that this will trigger, not having to go through its
uses.

> +		*result = "incomplete";
> +
> +	wordlen++;
> +	if (wordlen < len && resultstring[wordlen] == '(') {
> +		wordlen++;
> +		char *dup = malloc(len - wordlen);
> +		memcpy(dup, resultstring + wordlen, len - wordlen);
> +		dup[len - wordlen] = '\0';
> +		*time = strtod(dup, NULL);
> +
> +		free(dup);
> +	}
> +}
> +
> +static void parse_subtest_result(char *subtest, char **result, double *time, char *buf, char *bufend)
> +{
> +	char *line;
> +	char *line_end;
> +	char *resultstring;
> +	size_t linelen;
> +	size_t subtestlen = strlen(subtest);
> +
> +	*result = NULL;
> +	*time = 0.0;
> +
> +	if (!buf) return;
> +
> +	line = find_line(buf, subtest_result_beg, subtest_result_len, bufend);
> +	if (!line) {
> +		*result = "incomplete";
> +		return;
> +	}
> +
> +	line_end = memchr(line, '\n', bufend - line);
> +	linelen = line_end != NULL ? line_end - line : bufend - line;
> +
> +	if (subtest_result_len + subtestlen + 2 > linelen ||
> +	    strncmp(line + subtest_result_len, subtest, subtestlen))
> +		return parse_subtest_result(subtest, result, time, line + linelen, bufend);
> +
> +	resultstring = line + subtest_result_len + subtestlen + 2;
> +	parse_result_string(resultstring, linelen - (resultstring - line), result, time);
> +}

Please provide examples of input that his is parsing and add comments on
why +2.

> +static struct json_object *get_or_create_json_object(struct json_object *base,
> +						     char *key)
> +{
> +	struct json_object *ret;
> +
> +	if (json_object_object_get_ex(base, key, &ret))
> +		return ret;
> +
> +	ret = json_object_new_object();
> +	json_object_object_add(base, key, ret);
> +
> +	return ret;
> +}
> +
> +static void add_or_override_result(struct json_object *obj, char *from, char *to)
> +{
> +	struct json_object *current;
> +	const char *oldresult;
> +
> +	if (!json_object_object_get_ex(obj, "result", &current)) {
> +		json_object_object_add(obj, "result",
> +				       json_object_new_string(to));
> +		return;
> +	}
> +
> +	oldresult = json_object_get_string(current);
> +	if (from != NULL && !strcmp(oldresult, from))
> +		json_object_object_add(obj, "result",
> +				       json_object_new_string(to));
> +}
> +
> +static void add_runtime(struct json_object *obj, double time)
> +{
> +	double oldtime;
> +	struct json_object *timeobj = get_or_create_json_object(obj, "time");
> +	struct json_object *oldend;
> +
> +	json_object_object_add(timeobj, "__type__",
> +			       json_object_new_string("TimeAttribute"));
> +	json_object_object_add(timeobj, "start",
> +			       json_object_new_double(0.0));
> +
> +	if (!json_object_object_get_ex(timeobj, "end", &oldend)) {
> +		json_object_object_add(timeobj, "end",
> +				       json_object_new_double(time));
> +		return;
> +	}
> +
> +	oldtime = json_object_get_double(oldend);
> +	time += oldtime;
> +	json_object_object_add(timeobj, "end",
> +			       json_object_new_double(time));
> +}

I would prefer to have the logic promoting to dmesg-* and warns
separate, and in a single place, and possibly remove this conditional
overwrite from here.

> +static char versionstring[] = "IGT-Version: ";
> +static size_t versionlen = sizeof(versionstring) - 1;
> +static bool fill_from_output(int fd, char *binary, bool is_stdout,

Yet again, non-explicit, non-obvious mapping.

true = stdout
false = stderr

> +			     struct subtests *subtests,
> +			     struct json_object *tests)
> +{
> +	char *buf, *bufend;
> +	struct stat statbuf;
> +	char *igt_name = NULL;
> +	char *igt_version = NULL;
> +	size_t igt_version_len = 0;
> +	struct json_object *current_test = NULL;
> +	size_t i;
> +
> +	if (fstat(fd, &statbuf))
> +		return false;
> +
> +	buf = mmap(NULL, statbuf.st_size, PROT_READ, MAP_SHARED, fd, 0);
> +	if (!buf)
> +		return false;
> +
> +	bufend = buf + statbuf.st_size;
> +
> +	if (is_stdout) {
> +		char *newline;
> +
> +		igt_version = find_line(buf, versionstring, versionlen, bufend);
> +		if (igt_version) {
> +			newline = memchr(igt_version, '\n', bufend - igt_version);
> +			igt_version_len = newline - igt_version;
> +		} else {
> +			igt_version = NULL;
> +		}
> +	}
> +
> +	if (subtests->size == 0) {
> +		/* No subtests */
> +		igt_name = gen_igt_name(binary, NULL);
> +		current_test = get_or_create_json_object(tests, igt_name);
> +
> +		json_object_object_add(current_test, is_stdout ? "out" : "err",
> +				       json_object_new_string_len(buf, statbuf.st_size));
> +		if (!is_stdout && count_lines(buf, buf + statbuf.st_size) > 2) {
> +			add_or_override_result(current_test, "pass", "warn");
> +		}
> +		if (igt_version) {
> +			json_object_object_add(current_test, "igt-version",
> +					       json_object_new_string_len(igt_version,
> +									  igt_version_len));
> +		}

Unnecessary curly brackets.

> +		return true;
> +	}
> +
> +	for (i = 0; i < subtests->size; i++) {
> +		char *begin_subtest;
> +		char *subtest_result;
> +		char *beg, *end, *startline;
> +		int begin_len;
> +		int result_len;
> +
> +		igt_name = gen_igt_name(binary, subtests->names[i]);
> +		current_test = get_or_create_json_object(tests, igt_name);
> +
> +		begin_len = asprintf(&begin_subtest, "%s%s\n", starting_subtest, subtests->names[i]);
> +		result_len = asprintf(&subtest_result, "%s%s: ", subtest_result_beg, subtests->names[i]);
> +
> +		if (begin_len < 0 || result_len < 0) {
> +			fprintf(stderr, "Failure generating strings\n");
> +			return false;
> +		}

That's one of the very few places we look for allocation error.

> +
> +		beg = find_line(buf, begin_subtest, begin_len, bufend);
> +		end = find_line(buf, subtest_result, result_len, bufend);

Since we are always passing in NULL terminated strings to those find_
functions I would consider dropping the len from parameter list in favor
of being inefficient with strlen() inside.

> +		startline = beg;
> +
> +		free(begin_subtest);
> +		free(subtest_result);
> +
> +		if (beg == NULL && end == NULL) {
> +			/* No output at all */
> +			beg = bufend;
> +			end = bufend;
> +		}
> +
> +		if (beg == NULL) {
> +			/*
> +			 * Subtest didn't start, probably skipped from
> +			 * fixture already. Find the previous subtest
> +			 * output.
> +			 */

Extract this operation as a separate function (or at least the loop).

> +			beg = find_either_line(buf,
> +					       starting_subtest, starting_len,
> +					       subtest_result_beg, subtest_result_len,
> +					       end);
> +			if (beg == NULL) {
> +				beg = buf;
> +			} else {
> +				while (beg < end) {
> +					char *result;
> +					beg = next_line(beg, bufend);
> +					result = find_either_line(beg,
> +								  starting_subtest, starting_len,
> +								  subtest_result_beg, subtest_result_len,
> +								  bufend);
> +					if (result == NULL || result > end) {
> +						break;
> +					}
> +					beg = result;
> +				}
> +			}
> +		} else {
> +			/* Include the output after the previous subtest output */
> +			char *prevtest = find_either_line(buf,
> +							  starting_subtest, starting_len,
> +							  subtest_result_beg, subtest_result_len,
> +							  beg);
> +
> +			if (prevtest == NULL) {
> +				beg = buf;
> +			} else {
> +				while (prevtest != NULL && prevtest < beg) {

prevtest will be ever NULL only if prevtest is initally NULL, so it can be
turned into else if (...) (


> +					char *result;
> +					prevtest = next_line(prevtest, beg);

Never returns NULL.

> +					result = find_either_line(prevtest,
> +								  starting_subtest, starting_len,
> +								  subtest_result_beg, subtest_result_len,
> +								  beg);
> +					if (result == NULL) {
> +						beg = prevtest;
> +						break;
> +					}
> +					prevtest = result;
> +				}
> +			}
> +		}

So this copy and paste can be removed.

> +
> +		if (end == NULL) {
> +			/* Incomplete result. Find the next starting subtest or result. */
> +			end = next_line(startline, bufend);
> +			if (end != NULL) {
> +				end = find_either_line(end,
> +						       starting_subtest, starting_len,
> +						       subtest_result_beg, subtest_result_len,
> +						       bufend);
> +			}
> +			if (end == NULL) {
> +				end = bufend;
> +			}
> +		} else {
> +			/* Stretch onwards until the next subtest begins or ends */
> +			char *nexttest = next_line(end, bufend);
> +			if (nexttest != NULL) {
> +				nexttest = find_either_line(nexttest,
> +							    starting_subtest, starting_len,
> +							    subtest_result_beg, subtest_result_len,
> +							    bufend);
> +			}
> +			if (nexttest != NULL) {
> +				end = nexttest;
> +			} else {
> +				end = bufend;
> +			}
> +		}
> +
> +		json_object_object_add(current_test, is_stdout ? "out" : "err",
> +				       json_object_new_string_len(beg, end - beg));
> +
> +		if (is_stdout) {
> +			char *result;
> +			double time;
> +			parse_subtest_result(subtests->names[i], &result, &time, beg, end);
> +			add_or_override_result(current_test, NULL, result);
> +			add_runtime(current_test, time);
> +			if (igt_version) {
> +				json_object_object_add(current_test, "igt-version",
> +						       json_object_new_string_len(igt_version,
> +										  igt_version_len));
> +			}
> +		} else if (count_lines(startline, end) > 2) {
> +			add_or_override_result(current_test, "pass", "warn");
> +		}
> +	}
> +
> +	return true;
> +}

I would consider extracting chunks of this into separate functions, so
they are easier to inspect on themselves and don't clutter the body of
this function. Obvious candidates are the chunks that go through buffer
just using positions and the static patterns defined at the top.

> +
> +/*
> + * This regexp controls the kmsg handling. All kernel log records that
> + * have log level of warning or higher convert the result to
> + * dmesg-warn/dmesg-fail unless they match this regexp.
> + */
> +
> +#define _ "|"
> +static const char igt_dmesg_whitelist[] =
> +	"ACPI: button: The lid device is not compliant to SW_LID" _
> +	"ACPI: .*: Unable to dock!" _
> +	"IRQ [0-9]+: no longer affine to CPU[0-9]+" _
> +	"IRQ fixup: irq [0-9]+ move in progress, old vector [0-9]+" _
> +	/* i915 tests set module options, expected message */
> +	"Setting dangerous option [a-z_]+ - tainting kernel" _
> +	/* Raw printk() call, uses default log level (warn) */
> +	"Suspending console\\(s\\) \\(use no_console_suspend to debug\\)" _
> +	"atkbd serio[0-9]+: Failed to (deactivate|enable) keyboard on isa[0-9]+/serio[0-9]+" _
> +	"cache: parent cpu[0-9]+ should not be sleeping" _
> +	"hpet[0-9]+: lost [0-9]+ rtc interrupts" _
> +	/* i915 selftests terminate normally with ENODEV from the
> +	 * module load after the testing finishes, which produces this
> +	 * message.
> +	 */
> +	"i915: probe of [0-9:.]+ failed with error -25" _
> +	/* swiotbl warns even when asked not to */
> +	"mock: DMA: Out of SW-IOMMU space for [0-9]+ bytes" _
> +	"usb usb[0-9]+: root hub lost power or was reset"
> +	;
> +#undef _
> +
> +static regex_t re;
> +
> +static bool init_regex_whitelist()

You are mixing bools and integers here.

> +{
> +	int status = -1;

Missing static.

> +
> +	if (status == -1) {
> +		if (regcomp(&re, igt_dmesg_whitelist, REG_EXTENDED | REG_NOSUB) != 0) {
> +			fprintf(stderr, "Cannot compile dmesg whitelist regexp\n");
> +			status = 0;
> +			return false;
> +		}
> +
> +		status = 1;
> +	}
> +
> +	return status;
> +}
> +
> +static bool fill_from_dmesg(int fd, char *binary,
> +			    struct subtests *subtests,
> +			    struct json_object *tests)
> +{
> +	char *line = NULL, *warnings = NULL, *dmesg = NULL;
> +	size_t linelen = 0, warningslen = 0, dmesglen = 0;
> +	struct json_object *current_test = NULL;
> +	FILE *f = fdopen(fd, "r");
> +	char *igt_name = NULL;
> +	ssize_t read;
> +	size_t i;
> +
> +	if (!f) {
> +		return false;
> +	}
> +
> +	if (init_regex_whitelist() != 1) {

If you return bool, then make it bool comparison.

If you settle out on int - if you do not return meaningful integer as a
results, 0 should be considered as a success.

> +		fclose(f);
> +		return false;
> +	}
> +
> +	while ((read = getline(&line, &linelen, f)) > 0) {

The usual way is != -1, but that changes nothing here.

> +		char formatted[256];
> +		unsigned flags;
> +		unsigned long long seq, ts_usec;
> +		char continuation;
> +		char *message, *subtest;
> +		int s;
> +
> +		s = sscanf(line, "%u,%llu,%llu,%c;", &flags, &seq, &ts_usec, &continuation);
> +
> +		if (s != 4) {
> +			/*
> +			 * Machine readable key/value pairs begin with
> +			 * a space. We ignore them.
> +			 */
> +			if (line[0] != ' ') {
> +				fprintf(stderr, "Cannot parse kmsg record: %s\n", line);
> +			}
> +
> +			continue;
> +		}
> +
> +		message = strchr(line, ';');

if (NULL != message)
	message++

I would also extract this whole logic into parse_dmesg_line(*flags,
*ts_usec, *continuation) too offload some of the complexity and hide
seq.

And then:

if (parse_dmesg_line(&flags, &ts_usec, &continuation, &message))
	continue;

> +		if (!message) {
> +			fprintf(stderr, "No ; found, this shouldn't happen\n");
> +			return false;
> +		}
> +		snprintf(formatted, sizeof(formatted), "<%u> [%llu.%06llu] %s",
> +			 flags & 0x07, ts_usec / 1000000, ts_usec % 1000000, message);

This should return < sizeof(len(formatted)),
unless we don't care about possible truncating

> +
> +		if ((subtest = strstr(message, starting_subtest_dmesg)) != NULL) {
> +			if (current_test != NULL) {
> +				/* Done with the previous subtest, file up */

void add_dmesg(test, dmesg, warnings)
{

> +				json_object_object_add(current_test, "dmesg",
> +						       json_object_new_string_len(dmesg, dmesglen));
> +				if (warnings) {
> +					json_object_object_add(current_test, "dmesg-warnings",
> +							       json_object_new_string_len(warnings, warningslen));
> +					add_or_override_result(current_test, "pass", "dmesg-warn");
> +					add_or_override_result(current_test, "fail", "dmesg-fail");
> +					add_or_override_result(current_test, "warn", "dmesg-warn");
> +				}

}

times three.

> +				free(dmesg);
> +				free(warnings);
> +				dmesg = warnings = NULL;
> +				dmesglen = warningslen = 0;
> +			}
> +
> +			subtest += starting_subtest_dmesg_len;
> +			igt_name = gen_igt_name(binary, subtest);
> +			current_test = get_or_create_json_object(tests, igt_name);
> +		}
> +
> +		if ((flags & 0x07) <= 4 && continuation != 'c' &&
> +		    regexec(&re, message, (size_t)0, NULL, 0) == REG_NOMATCH) {
> +			append_line(&warnings, &warningslen, formatted);
> +		}
> +		append_line(&dmesg, &dmesglen, formatted);
> +	}
> +
> +	if (current_test != NULL) {
> +		json_object_object_add(current_test, "dmesg",
> +				       json_object_new_string_len(dmesg, dmesglen));
> +		if (warnings) {
> +			json_object_object_add(current_test, "dmesg-warnings",
> +					       json_object_new_string_len(warnings, warningslen));
> +			add_or_override_result(current_test, "pass", "dmesg-warn");
> +			add_or_override_result(current_test, "fail", "dmesg-fail");
> +			add_or_override_result(current_test, "warn", "dmesg-warn");
> +		}
> +	} else {
> +		/*
> +		 * Didn't get any subtest messages at all. If there
> +		 * are subtests, add all of the dmesg gotten to all of
> +		 * them.
> +		 */
> +		for (i = 0; i < subtests->size; i++) {
> +			igt_name = gen_igt_name(binary, subtests->names[i]);
> +			current_test = get_or_create_json_object(tests, igt_name);
> +			json_object_object_add(current_test, "dmesg",
> +					       json_object_new_string_len(dmesg, dmesglen));
> +			/*
> +			 * Don't bother with warnings, any subtests
> +			 * there are would have skip as their result
> +			 * anyway.
> +			 */
> +		}
> +
> +		if (i == 0) {
> +			/* There were no subtests */
> +			igt_name = gen_igt_name(binary, NULL);
> +			current_test = get_or_create_json_object(tests, igt_name);
> +			json_object_object_add(current_test, "dmesg",
> +					       json_object_new_string_len(dmesg, dmesglen));
> +			if (warnings) {
> +				json_object_object_add(current_test, "dmesg-warnings",
> +						       json_object_new_string_len(warnings, warningslen));
> +				add_or_override_result(current_test, "pass", "dmesg-warn");
> +				add_or_override_result(current_test, "fail", "dmesg-fail");
> +				add_or_override_result(current_test, "warn", "dmesg-warn");
> +			}
> +		}

Extract the whole else {} into a function, if on subtest->size, then
either iterate or do the other thing. Depending on the leftover value of
i is not explicit enough.

> +	}
> +
> +	/*
> +	 * Add an empty string as the dmesg of all subtests that
> +	 * didn't get any dmesg yet.
> +	 */
> +	for (i = 0; i < subtests->size; i++) {
> +		igt_name = gen_igt_name(binary, subtests->names[i]);
> +		current_test = get_or_create_json_object(tests, igt_name);
> +		if (!json_object_object_get_ex(current_test, "dmesg", NULL)) {
> +			json_object_object_add(current_test, "dmesg",
> +					       json_object_new_string(""));
> +		}
> +	}

Can be easily extracted into a clearly named function, so you can skip
the comment and save some screen estate taken by this function.

> +
> +	free(dmesg);
> +	free(warnings);
> +	fclose(f);
> +	return true;
> +}
> +
> +static char *result_from_exitcode(int exitcode)
> +{
> +	switch (exitcode) {
> +	case IGT_EXIT_TIMEOUT:
> +		return "timeout";
> +	case IGT_EXIT_SKIP:
> +		return "skip";
> +	case IGT_EXIT_SUCCESS:
> +		return "pass";
> +	case IGT_EXIT_INVALID:
> +		return "notrun";
> +	default:
> +		return "fail";
> +	}
> +}
> +
> +static void add_subtest(struct subtests *subtests, char *subtest)
> +{
> +	size_t len = strlen(subtest);
> +
> +	if (len == 0)
> +		return;
> +
> +	if (subtest[len - 1] == '\n')
> +		subtest[len - 1] = '\0';
> +
> +	subtests->size++;
> +	subtests->names = realloc(subtests->names, sizeof(*subtests->names) * subtests->size);
> +	subtests->names[subtests->size - 1] = subtest;
> +}
> +
> +static bool fill_from_journal(int fd, char *binary,
> +				  struct subtests *subtests,
> +				  struct json_object *tests)
> +{
> +	FILE *f = fdopen(fd, "r");
> +	char *line = NULL;
> +	size_t linelen = 0;
> +	ssize_t read;
> +	char exitline[] = "exit:";
> +	size_t exitlen = sizeof(exitline) - 1;
> +	char timeoutline[] = "timeout:";
> +	size_t timeoutlen = sizeof(timeoutline) - 1;
> +	int exitcode = 10000;
> +
> +	while ((read = getline(&line, &linelen, f)) > 0) {

!= -1

> +		if (read >= exitlen && !memcmp(line, exitline, exitlen)) {
> +			char *p = strchr(line, '(');
> +			char *igt_name = gen_igt_name(binary, NULL);
> +			double time = 0.0;
> +			struct json_object *obj = get_or_create_json_object(tests, igt_name);
> +
> +			exitcode = atoi(line + exitlen);
> +
> +			if (p) {
> +				time = strtod(p + 1, NULL);
> +			}
> +
> +			add_runtime(obj, time);
> +		} else if (read >= timeoutlen && !memcmp(line, timeoutline, timeoutlen)) {
> +			if (subtests->size) {
> +				char *last_subtest = subtests->names[subtests->size - 1];
> +				char *igt_name = gen_igt_name(binary, last_subtest);
> +				char *p = strchr(line, '(');
> +				double time = 0.0;
> +				struct json_object *obj = get_or_create_json_object(tests, igt_name);
> +
> +				json_object_object_add(obj, "result",
> +						       json_object_new_string("timeout"));
> +
> +				if (p) {
> +					time = strtod(p + 1, NULL);
> +				}
> +
> +				add_runtime(obj, time);
> +
> +				igt_name = gen_igt_name(binary, NULL);
> +				obj = get_or_create_json_object(tests, igt_name);
> +				add_runtime(obj, time);
> +			}
> +		} else {
> +			add_subtest(subtests, strdup(line));
> +		}
> +	}
> +
> +	if (subtests->size == 0 && exitcode != 10000) {
> +		char *igt_name = gen_igt_name(binary, NULL);
> +		struct json_object *obj = get_or_create_json_object(tests, igt_name);
> +		char *result = result_from_exitcode(exitcode);
> +		json_object_object_add(obj, "result",
> +				       json_object_new_string(result));
> +	}
> +
> +	free(line);
> +	return true;

void? It does not return anything but true.

> +}
> +
> +static bool parse_test_directory(int dirfd, char *binary, struct json_object *tests)
> +{
> +	int fds[_F_LAST];
> +	struct subtests subtests = {};
> +
> +	if (!open_output_files(dirfd, fds, false)) {
> +		fprintf(stderr, "Error opening output files\n");
> +		return false;
> +	}
> +
> +	/* fill_from_journal also fills the subtests struct */
> +	if (!fill_from_journal(fds[_F_JOURNAL], binary, &subtests, tests)) {
> +		fprintf(stderr, "Error reading from journal\n");
> +		return false;
> +	}
> +
> +	/* Order of these is important */

Why?

> +	if (!fill_from_output(fds[_F_OUT], binary, true, &subtests, tests) ||
> +	    !fill_from_output(fds[_F_ERR], binary, false, &subtests, tests) ||
> +	    !fill_from_dmesg(fds[_F_DMESG], binary, &subtests, tests)) {
> +		fprintf(stderr, "Error parsing output files\n");
> +		return false;
> +	}
> +
> +	close_outputs(fds);
> +
> +	return true;
> +}
> +
> +bool generate_results(int dirfd)
> +{
> +	struct settings settings;
> +	struct job_list job_list;
> +	struct json_object *obj, *tests;
> +	int resultsfd, testdirfd, unamefd;
> +	const char *json_string;
> +	size_t i;
> +
> +	init_settings(&settings);
> +	init_job_list(&job_list);
> +
> +	if (!read_settings(&settings, dirfd)) {
> +		fprintf(stderr, "resultgen: Cannot parse settings\n");
> +		return false;
> +	}
> +
> +	if (!read_job_list(&job_list, dirfd)) {
> +		fprintf(stderr, "resultgen: Cannot parse job list\n");
> +		return false;
> +	}
> +
> +	/* TODO: settings.overwrite */
> +	if ((resultsfd = openat(dirfd, "results.json", O_WRONLY | O_CREAT | O_TRUNC, 0666)) < 0) {
> +		fprintf(stderr, "resultgen: Cannot create results file\n");
> +		return false;
> +	}
> +
> +	obj = json_object_new_object();
> +	json_object_object_add(obj, "__type__", json_object_new_string("TestrunResult"));
> +	json_object_object_add(obj, "results_version", json_object_new_int(9));
> +	json_object_object_add(obj, "name",
> +			       settings.name ?
> +			       json_object_new_string(settings.name) :
> +			       json_object_new_string(""));
> +
> +	if ((unamefd = openat(dirfd, "uname.txt", O_RDONLY)) >= 0) {
> +		char buf[128];
> +		ssize_t r = read(unamefd, buf, 128);
> +
> +		if (r > 0 && buf[r - 1] == '\n')
> +			r--;
> +
> +		json_object_object_add(obj, "uname",
> +				       json_object_new_string_len(buf, r));
> +		close(unamefd);
> +	}
> +

/* TODO */

> +	/* lspci */
> +	/* results_version */
> +	/* glxinfo */
> +	/* wglinfo */
> +	/* clinfo */
> +	/* options */
> +	/* time_elapsed */
> +	/* totals */
> +
> +	tests = json_object_new_object();
> +	json_object_object_add(obj, "tests", tests);
> +
> +	for (i = 0; i < job_list.size; i++) {
> +		char name[16];
> +
> +		snprintf(name, 16, "%zd", i);
> +		if ((testdirfd = openat(dirfd, name, O_DIRECTORY | O_RDONLY)) < 0) {
> +			fprintf(stderr, "Warning: Cannot open result directory %s\n", name);
> +			break;
> +		}
> +
> +		if (!parse_test_directory(testdirfd, job_list.entries[i].binary, tests)) {
> +			close(resultsfd);
> +			return false;
> +		}
> +	}
> +
> +	json_string = json_object_to_json_string_ext(obj, JSON_C_TO_STRING_PRETTY);
> +	write(resultsfd, json_string, strlen(json_string));
> +	return true;
> +}
> +
> +bool generate_results_path(char *resultspath)
> +{
> +	int dirfd = open(resultspath, O_DIRECTORY | O_RDONLY);
> +
> +	if (dirfd < 0)
> +		return false;
> +
> +	return generate_results(dirfd);
> +}
_______________________________________________
igt-dev mailing list
igt-dev@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/igt-dev

Re: [igt-dev] [PATCH i-g-t 3/4] runner: New test runner

[Chris Wilson]

Quoting Arkadiusz Hiler (2018-05-08 09:47:45)
> On Mon, Apr 30, 2018 at 12:28:47PM +0300, Petri Latvala wrote:
> > diff --git a/runner/resultgen.c b/runner/resultgen.c
> > new file mode 100644
> > index 00000000..978c07be
> > --- /dev/null
> > +++ b/runner/resultgen.c
> > @@ -0,0 +1,864 @@
> > +#include <ctype.h>
> > +#include <fcntl.h>
> > +#include <stdio.h>
> > +#include <string.h>
> > +#include <sys/mman.h>
> > +#include <sys/stat.h>
> > +#include <sys/types.h>
> > +#include <unistd.h>
> > +
> > +#include <json.h>
> > +
> > +#include "resultgen.h"
> > +#include "settings.h"
> > +#include "executor.h"
> > +#include "igt_core.h"
> > +
> > +static char subtest_result_beg[] = "Subtest ";
> > +static size_t subtest_result_len = sizeof(subtest_result_beg) - 1;
> 
> _beg_
> 
> > +static char starting_subtest[] = "Starting subtest: ";
> > +static size_t starting_len = sizeof(starting_subtest) - 1;
> 
> _subtest_
> 
> > +static char starting_subtest_dmesg[] = ": starting subtest ";
> > +static size_t starting_subtest_dmesg_len = sizeof(starting_subtest_dmesg) - 1;
> 
> ALL_CAPS or something else for readability please. Otherwise it is easy
> to get lost track of what is what in some of the parsing functions
> down below, due to similar naming.

It's also the optimisation the compiler will make for
strlen(constant_sting), so don't bother, just make sure they are known
constant.
-Chris
_______________________________________________
igt-dev mailing list
igt-dev@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/igt-dev

Re: [igt-dev] [PATCH i-g-t 3/4] runner: New test runner

[Arkadiusz Hiler]

On Mon, Apr 30, 2018 at 12:28:47PM +0300, Petri Latvala wrote:
> diff --git a/runner/settings.c b/runner/settings.c
> new file mode 100644
> index 00000000..50e4ea10
> --- /dev/null
> +++ b/runner/settings.c
> @@ -0,0 +1,506 @@
> +#include "settings.h"
> +
> +#include <fcntl.h>
> +#include <getopt.h>
> +#include <libgen.h>
> +#include <limits.h>
> +#include <stdio.h>
> +#include <stdlib.h>
> +#include <string.h>
> +#include <sys/types.h>
> +#include <sys/stat.h>
> +#include <unistd.h>
> +
> +enum {
> +	OPT_ABORT_ON_ERROR,
> +	OPT_TEST_LIST,
> +	OPT_IGNORE_MISSING,
> +	OPT_HELP = 'h',
> +	OPT_NAME = 'n',
> +	OPT_DRY_RUN = 'd',
> +	OPT_INCLUDE = 't',
> +	OPT_EXCLUDE = 'x',
> +	OPT_SYNC = 's',
> +	OPT_LOG_LEVEL = 'l',
> +	OPT_OVERWRITE = 'o',
> +	OPT_MULTIPLE = 'm',
> +	OPT_TIMEOUT = 'c',
> +	OPT_WATCHDOG = 'g',
> +};
> +
> +static struct {
> +	int level;
> +	const char *name;
> +} log_levels[] = {
> +	{ LOG_LEVEL_NORMAL, "normal" },
> +	{ LOG_LEVEL_QUIET, "quiet" },
> +	{ LOG_LEVEL_VERBOSE, "verbose" },
> +	{ 0, 0 },
> +};
> +
> +static bool set_log_level(struct settings* settings, const char *level)
> +{
> +	int c;
> +
> +	for (c = 0; log_levels[c].name != NULL; c++) {

Since we are defining everything here, why are we hassling with adding
{ 0, 0} and then != NULL instead of using sizeof()?

> +		if (!strcmp(level, log_levels[c].name)) {
> +			settings->log_level = log_levels[c].level;
> +			return true;
> +		}
> +	}
> +
> +	return false;
> +}
> +
> +static const char *usage_str =
> +	"usage: runner [options] [test_root] results-path\n\n"
> +	"Options:\n"
> +	" Piglit compatible:\n"
> +	"  -h, --help            Show this help message and exit\n"
> +	"  -n <test name>, --name <test name>\n"
> +	"                        Name of this test run\n"
> +	"  -d, --dry-run         Do not execute the tests\n"
> +	"  -t <regex>, --include-tests <regex>\n"
> +	"                        Run only matching tests (can be used more than once)\n"
> +	"  -x <regex>, --exclude-tests <regex>\n"
> +	"                        Exclude matching tests (can be used more than once)\n"
> +	"  --abort-on-monitored-error\n"
> +	"                        Abort execution when a fatal condition is detected.\n"
> +	"                        <TODO>\n"
> +	"  -s, --sync            Sync results to disk after every test\n"
> +	"  -l {quiet,verbose,dummy}, --log-level {quiet,verbose,dummy}\n"
> +	"                        Set the logger verbosity level\n"
> +	"  --test-list TEST_LIST\n"
> +	"                        A file containing a list of tests to run\n"
> +	"  -o, --overwrite       If the results-path already exists, delete it\n"
> +	"  --ignore-missing      Ignored but accepted, for piglit compatibility\n"
> +	"\n"
> +	" Incompatible options:\n"
> +	"  -m, --multiple-mode   Run multiple subtests in the same binary execution.\n"
> +	"                        If a testlist file is given, consecutive subtests are\n"
> +	"                        run in the same execution if they are from the same\n"
> +	"                        binary. Note that in that case relative ordering of the\n"
> +	"                        subtest execution is dictated by the test binary, not\n"
> +	"                        the testlist\n"
> +	"  --inactivity-timeout <seconds>\n"
> +	"                        Kill the running test after <seconds> of inactivity in\n"
> +	"                        the test's stdout, stderr, or dmesg\n"
> +	"  --use-watchdog        Use hardware watchdog for lethal enforcement of the\n"
> +	"                        above timeout. Killing the test process is still\n"
> +	"                        attempted at timeout trigger.\n"
> +	"  [test_root]           Directory that contains the IGT tests. The environment\n"
> +	"                        variable IGT_TEST_ROOT will be used if set, overriding\n"
> +	"                        this option if given.\n"
> +	;
> +
> +static void usage(const char *extra_message, bool use_stderr)
> +{
> +	FILE *f = use_stderr ? stderr : stdout;

You can just pass stdout/stderr here, instead of another bool mapping.

> +
> +	if (extra_message)
> +		fprintf(f, "%s\n\n", extra_message);
> +
> +	fprintf(f, "%s", usage_str);
> +}
> +
> +static bool add_regex(struct regex_list *list, char *new)
> +{
> +	regex_t *regex;
> +	size_t buflen;
> +	char *buf;
> +	int s;
> +
> +	regex = malloc(sizeof(*regex));
> +
> +	if ((s = regcomp(regex, new,
> +			 REG_EXTENDED | REG_NOSUB)) != 0) {
> +		buflen = regerror(s, regex, NULL, 0);
> +		buf = malloc(buflen);
> +		regerror(s, regex, buf, buflen);
> +		usage(buf, true);
> +
> +		free(buf);
> +		regfree(regex);
> +		free(regex);
> +		return false;
> +	}
> +
> +	list->regexes = realloc(list->regexes,
> +				(list->size + 1) * sizeof(*list->regexes));
> +	list->regex_strings = realloc(list->regex_strings,
> +				      (list->size + 1) * sizeof(*list->regex_strings));
> +	list->regexes[list->size] = regex;
> +	list->regex_strings[list->size] = new;
> +	list->size++;
> +
> +	return true;
> +}
> +
> +static void free_regexes(struct regex_list *regexes)
> +{
> +	size_t i;
> +
> +	for (i = 0; i < regexes->size; i++) {
> +		free(regexes->regex_strings[i]);
> +		regfree(regexes->regexes[i]);
> +		free(regexes->regexes[i]);
> +	}
> +	free(regexes->regex_strings);
> +	free(regexes->regexes);
> +}
> +
> +static bool readable_file(char *filename)
> +{
> +	return !access(filename, R_OK);
> +}
> +
> +void init_settings(struct settings *settings)
> +{
> +	memset(settings, 0, sizeof(*settings));
> +}
> +
> +void free_settings(struct settings *settings)
> +{
> +	free(settings->test_list);
> +	free(settings->name);
> +	free(settings->test_root);
> +	free(settings->results_path);
> +
> +	free_regexes(&settings->include_regexes);
> +	free_regexes(&settings->exclude_regexes);
> +
> +	init_settings(settings);
> +}
> +
> +bool parse_options(int argc, char **argv,
> +		   struct settings *settings)
> +{
> +	int c;
> +	char *env_test_root;
> +
> +	static struct option long_options[] = {
> +		{"help", no_argument, NULL, OPT_HELP},
> +		{"name", required_argument, NULL, OPT_NAME},
> +		{"dry-run", no_argument, NULL, OPT_DRY_RUN},
> +		{"include-tests", required_argument, NULL, OPT_INCLUDE},
> +		{"exclude-tests", required_argument, NULL, OPT_EXCLUDE},
> +		{"abort-on-monitored-error", no_argument, NULL, OPT_ABORT_ON_ERROR},
> +		{"sync", no_argument, NULL, OPT_SYNC},
> +		{"log-level", required_argument, NULL, OPT_LOG_LEVEL},
> +		{"test-list", required_argument, NULL, OPT_TEST_LIST},
> +		{"overwrite", no_argument, NULL, OPT_OVERWRITE},
> +		{"ignore-missing", no_argument, NULL, OPT_IGNORE_MISSING},
> +		{"multiple-mode", no_argument, NULL, OPT_MULTIPLE},
> +		{"inactivity-timeout", required_argument, NULL, OPT_TIMEOUT},
> +		{"use-watchdog", no_argument, NULL, OPT_WATCHDOG},
> +		{ 0, 0, 0, 0},
> +	};
> +
> +	free_settings(settings);
> +
> +	optind = 1;
> +
> +	while ((c = getopt_long(argc, argv, "hn:dt:x:sl:om", long_options, NULL)) != -1) {
> +		switch (c) {
> +		case OPT_HELP:
> +			usage(NULL, false);
> +			goto error;
> +		case OPT_NAME:
> +			settings->name = strdup(optarg);
> +			break;
> +		case OPT_DRY_RUN:
> +			settings->dry_run = true;
> +			break;
> +		case OPT_INCLUDE:
> +			if (!add_regex(&settings->include_regexes, strdup(optarg)))
> +				goto error;
> +			break;
> +		case OPT_EXCLUDE:
> +			if (!add_regex(&settings->exclude_regexes, strdup(optarg)))
> +				goto error;
> +			break;
> +		case OPT_ABORT_ON_ERROR:
> +			settings->abort_on_error = true;
> +			break;
> +		case OPT_SYNC:
> +			settings->sync = true;
> +			break;
> +		case OPT_LOG_LEVEL:
> +			if (!set_log_level(settings, optarg)) {
> +				usage("Cannot parse log level", true);
> +				goto error;
> +			}
> +			break;
> +		case OPT_TEST_LIST:
> +			settings->test_list = absolute_path(optarg);
> +			break;
> +		case OPT_OVERWRITE:
> +			settings->overwrite = true;
> +			break;
> +		case OPT_IGNORE_MISSING:
> +			/* Ignored, piglit compatibility */
> +			break;
> +		case OPT_MULTIPLE:
> +			settings->multiple_mode = true;
> +			break;
> +		case OPT_TIMEOUT:
> +			settings->inactivity_timeout = atoi(optarg);
> +			break;
> +		case OPT_WATCHDOG:
> +			settings->use_watchdog = true;
> +			break;
> +		case '?':
> +			usage(NULL, true);
> +			goto error;
> +		default:
> +			usage("Cannot parse options", true);
> +			goto error;
> +		}
> +	}
> +
> +	switch (argc - optind) {
> +	case 2:
> +		settings->test_root = absolute_path(argv[optind]);
> +		++optind;
> +		/* fallthrough */
> +	case 1:
> +		settings->results_path = absolute_path(argv[optind]);
> +		break;
> +	case 0:
> +		usage("Results-path missing", true);
> +		goto error;
> +	default:
> +		usage("Extra arguments after results-path", true);
> +		goto error;
> +	}
> +
> +	if ((env_test_root = getenv("IGT_TEST_ROOT")) != NULL) {
> +		free(settings->test_root);
> +		settings->test_root = absolute_path(env_test_root);
> +	}
> +
> +	if (!settings->test_root) {
> +		usage("Test root not set", true);
> +		goto error;
> +	}
> +
> +	if (!settings->name) {
> +		char *name = strdup(settings->results_path);
> +		settings->name = strdup(basename(name));
> +		free(name);
> +	}
> +
> +	return true;
> +
> + error:
> +	free_settings(settings);
> +	return false;
> +}
> +
> +bool validate_settings(struct settings *settings)
> +{
> +	int dirfd, fd;
> +
> +	if (settings->test_list && !readable_file(settings->test_list)) {
> +		usage("Cannot open test-list file", true);
> +		return false;
> +	}
> +
> +	if (!settings->results_path) {
> +		usage("No results-path set; this shouldn't happen", true);
> +		return false;
> +	}
> +
> +	if (!settings->test_root) {
> +		usage("No test root set; this shouldn't happen", true);
> +		return false;
> +	}
> +
> +	dirfd = open(settings->test_root, O_DIRECTORY | O_RDONLY);
> +	if (dirfd < 0) {
> +		fprintf(stderr, "Test directory %s cannot be opened\n", settings->test_root);
> +		return false;
> +	}
> +
> +	fd = openat(dirfd, "test-list.txt", O_RDONLY);
> +	if (fd < 0) {
> +		fprintf(stderr, "Cannot open %s/test-list.txt\n", settings->test_root);
> +		close(dirfd);
> +		return false;
> +	}
> +
> +	close(fd);
> +	close(dirfd);
> +
> +	return true;
> +}
> +
> +char *absolute_path(char *path)
> +{
> +	char *result = NULL;
> +	char *tmppath, *tmpname;
> +
> +	result = realpath(path, NULL);
> +	if (result != NULL)
> +		return result;
> +
> +	tmppath = strdup(path);
> +	tmpname = dirname(tmppath);
> +	free(result);
> +	result = realpath(tmpname, NULL);
> +	free(tmppath);
> +
> +	if (result != NULL) {
> +		tmppath = strdup(path);
> +		tmpname = basename(tmppath);
> +		strcat(result, "/");
> +		strcat(result, tmpname);

An overflow. Result is a memory allocated by realpath() here, to quote
the manual, "up to PATH_MAX bytes", so it may be exact size of the
contained string.

Then we keep appedning to it without ever growing the allocated size.

> +		free(tmppath);
> +		return result;
> +	}
> +
> +	free(result);
> +	return NULL;
> +}
> +
> +static char settings_filename[] = "metadata.txt";
> +bool serialize_settings(struct settings *settings)
> +{
> +#define SERIALIZE_LINE(f, s, name, format) fprintf(f, "%s : " format "\n", #name, s->name)
> +
> +	int dirfd, fd;
> +	FILE *f;
> +
> +	if (!settings->results_path) {
> +		usage("No results-path set; this shouldn't happen", true);
> +		return false;
> +	}
> +
> +	if ((dirfd = open(settings->results_path, O_DIRECTORY | O_RDONLY)) < 0) {
> +		mkdir(settings->results_path, 0755);
> +		if ((dirfd = open(settings->results_path, O_DIRECTORY | O_RDONLY)) < 0) {
> +			usage("Creating results-path failed", true);
> +			return false;
> +		}
> +	}
> +
> +	if ((fd = openat(dirfd, settings_filename, O_RDONLY)) >= 0) {
> +		close(fd);
> +
> +		if (!settings->overwrite) {
> +			/* Serialization data already exists, not overwriting */
> +			close(dirfd);
> +			return false;
> +		}
> +
> +		if (unlinkat(dirfd, settings_filename, 0) != 0) {
> +			usage("Error overwriting old settings metadata", true);
> +			close(dirfd);
> +			return false;
> +		}

You can just try unlinking (if !overwrite) and on -1 check the errno for
ENOENT instead of hassling with openat() and close().

> +	}
> +
> +	if ((fd = openat(dirfd, settings_filename, O_CREAT | O_EXCL | O_WRONLY, 0666)) < 0) {
> +		usage("Creating settings serialization file failed", true);

Printing strerror(errno) could be helpful. It's easy to mess up the
permissions.

> +		close(dirfd);
> +		return false;
> +	}
> +
> +	f = fdopen(fd, "w");
> +	if (!f) {
> +		close(fd);
> +		close(dirfd);
> +		return false;
> +	}

I find the control flow of this, and it's caller in executor.c odd.
Should we just return from here like that? It will trigger
initialize_execute_from_resume() which seems strange.

I think that the runner should have couple more of "hard exits" with
loud complaints in scenarios like this.

> +
> +	SERIALIZE_LINE(f, settings, abort_on_error, "%d");
> +	if (settings->test_list)
> +		SERIALIZE_LINE(f, settings, test_list, "%s");
> +	if (settings->name)
> +		SERIALIZE_LINE(f, settings, name, "%s");
> +	SERIALIZE_LINE(f, settings, dry_run, "%d");
> +	SERIALIZE_LINE(f, settings, sync, "%d");
> +	SERIALIZE_LINE(f, settings, log_level, "%d");
> +	SERIALIZE_LINE(f, settings, overwrite, "%d");
> +	SERIALIZE_LINE(f, settings, multiple_mode, "%d");
> +	SERIALIZE_LINE(f, settings, inactivity_timeout, "%d");
> +	SERIALIZE_LINE(f, settings, use_watchdog, "%d");
> +	SERIALIZE_LINE(f, settings, test_root, "%s");
> +	SERIALIZE_LINE(f, settings, results_path, "%s");
> +
> +	if (settings->sync) {
> +		fsync(fd);
> +		fsync(dirfd);
> +	}
> +
> +	fclose(f);
> +	close(dirfd);
> +	return true;
> +
> +#undef SERIALIZE_LINE
> +}
> +
> +static char *maybe_strdup(char *str)
> +{
> +	if (!str)
> +		return NULL;
> +
> +	return strdup(str);
> +}

:D

> +
> +bool read_settings(struct settings *settings, int dirfd)
> +{
> +#define PARSE_LINE(s, name, val, field, write) \
> +	if (!strcmp(name, #field)) {	       \
> +		s->field = write;	       \
> +		free(name);		       \
> +		free(val);		       \
> +		name = val = NULL;	       \
> +		continue;		       \
> +	}
> +
> +	int fd;
> +	FILE *f;
> +	char *name = NULL, *val = NULL;
> +
> +	free_settings(settings);
> +
> +	if ((fd = openat(dirfd, settings_filename, O_RDONLY)) < 0)
> +		return false;
> +
> +	f = fdopen(fd, "r");
> +	if (!f) {
> +		close(fd);
> +		return false;
> +	}
> +
> +	while (fscanf(f, "%ms : %ms", &name, &val) == 2) {
> +		int numval = atoi(val);
> +		PARSE_LINE(settings, name, val, abort_on_error, numval);
> +		PARSE_LINE(settings, name, val, test_list, maybe_strdup(val));
> +		PARSE_LINE(settings, name, val, name, maybe_strdup(val));
> +		PARSE_LINE(settings, name, val, dry_run, numval);
> +		PARSE_LINE(settings, name, val, sync, numval);
> +		PARSE_LINE(settings, name, val, log_level, numval);
> +		PARSE_LINE(settings, name, val, overwrite, numval);
> +		PARSE_LINE(settings, name, val, multiple_mode, numval);
> +		PARSE_LINE(settings, name, val, inactivity_timeout, numval);
> +		PARSE_LINE(settings, name, val, use_watchdog, numval);
> +		PARSE_LINE(settings, name, val, test_root, maybe_strdup(val));
> +		PARSE_LINE(settings, name, val, results_path, maybe_strdup(val));
> +
> +		printf("Warning: Unknown field in settings file: %s = %s\n",
> +		       name, val);
> +		free(name);
> +		free(val);
> +		name = val = NULL;
> +	}
> +
> +	free(name);
> +	free(val);
> +	fclose(f);
> +
> +	return true;
> +
> +#undef PARSE_LINE
> +}

Thanks for the ride and good luck with going through this review :-P

Cheers,
Arek
_______________________________________________
igt-dev mailing list
igt-dev@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/igt-dev

Re: [igt-dev] [PATCH i-g-t 3/4] runner: New test runner

[Chris Wilson]

Quoting Arkadiusz Hiler (2018-05-11 11:45:04)
> On Mon, Apr 30, 2018 at 12:28:47PM +0300, Petri Latvala wrote:
> > +static struct {
> > +     int level;
> > +     const char *name;
> > +} log_levels[] = {
> > +     { LOG_LEVEL_NORMAL, "normal" },
> > +     { LOG_LEVEL_QUIET, "quiet" },
> > +     { LOG_LEVEL_VERBOSE, "verbose" },
> > +     { 0, 0 },
> > +};
> > +
> > +static bool set_log_level(struct settings* settings, const char *level)
> > +{
> > +     int c;
> > +
> > +     for (c = 0; log_levels[c].name != NULL; c++) {
> 
> Since we are defining everything here, why are we hassling with adding
> { 0, 0} and then != NULL instead of using sizeof()?

for (typeof(log_levels) it; it->name; it++) {
	if (strcmp(level, it->name))
		continue;
	
	settings->log_level = it->level;
	return true;
}

Because sometimes it's more hassle to use ARRAY_SIZE(). :)
-Chris
_______________________________________________
igt-dev mailing list
igt-dev@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/igt-dev

Re: [igt-dev] [PATCH i-g-t 3/4] runner: New test runner

[Chris Wilson]

Quoting Chris Wilson (2018-05-11 11:50:26)
> Quoting Arkadiusz Hiler (2018-05-11 11:45:04)
> > On Mon, Apr 30, 2018 at 12:28:47PM +0300, Petri Latvala wrote:
> > > +static struct {
> > > +     int level;
> > > +     const char *name;
> > > +} log_levels[] = {
> > > +     { LOG_LEVEL_NORMAL, "normal" },
> > > +     { LOG_LEVEL_QUIET, "quiet" },
> > > +     { LOG_LEVEL_VERBOSE, "verbose" },
> > > +     { 0, 0 },
> > > +};
> > > +
> > > +static bool set_log_level(struct settings* settings, const char *level)
> > > +{
> > > +     int c;
> > > +
> > > +     for (c = 0; log_levels[c].name != NULL; c++) {
> > 
> > Since we are defining everything here, why are we hassling with adding
> > { 0, 0} and then != NULL instead of using sizeof()?
> 
> for (typeof(log_levels) it; it->name; it++) {

for (typeof(log_levels) it = log_levels; it->name; it++) {

We're not rust yet.
-Chris
_______________________________________________
igt-dev mailing list
igt-dev@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/igt-dev

[igt-dev] [PATCH i-g-t 4/4] runner: Unit tests for the runner

[Petri Latvala]

TODO: Unit tests for the results.json file contents.

Signed-off-by: Petri Latvala <petri.latvala@intel.com>
Cc: Maarten Lankhorst <maarten.lankhorst@linux.intel.com>
Cc: Arkadiusz Hiler <arkadiusz.hiler@intel.com>
Cc: Tomi Sarvela <tomi.p.sarvela@intel.com>
Cc: Martin Peres <martin.peres@linux.intel.com>
---
 runner/meson.build            |  10 +
 runner/runner_tests.c         | 967 ++++++++++++++++++++++++++++++++++++++++++
 runner/testdata/meson.build   |  20 +
 runner/testdata/no-subtests.c |   6 +
 runner/testdata/skippers.c    |  14 +
 runner/testdata/successtest.c |  10 +
 6 files changed, 1027 insertions(+)
 create mode 100644 runner/runner_tests.c
 create mode 100644 runner/testdata/meson.build
 create mode 100644 runner/testdata/no-subtests.c
 create mode 100644 runner/testdata/skippers.c
 create mode 100644 runner/testdata/successtest.c

diff --git a/runner/meson.build b/runner/meson.build
index 773b4f3c..4e04d499 100644
--- a/runner/meson.build
+++ b/runner/meson.build
@@ -1,3 +1,5 @@
+subdir('testdata')
+
 jsonc = dependency('json-c', required: true)
 
 runnerlib_sources = [ 'settings.c',
@@ -8,6 +10,7 @@ runnerlib_sources = [ 'settings.c',
 
 runner_sources = [ 'runner.c',
 		 ]
+runner_test_sources = [ 'runner_tests.c' ]
 
 runnerlib = static_library('igt_runner', runnerlib_sources,
 			   include_directories : inc,
@@ -22,3 +25,10 @@ runner = executable('runner', runner_sources,
 results = executable('results', 'results.c',
 		     link_with : runnerlib,
 )
+
+runner_test = executable('runner_test', runner_test_sources,
+			 c_args : '-DTESTDATA_DIRECTORY="@0@"'.format(testdata_dir),
+			 link_with : runnerlib,
+			 install : false,
+			 dependencies : igt_deps)
+test('runner', runner_test)
diff --git a/runner/runner_tests.c b/runner/runner_tests.c
new file mode 100644
index 00000000..076f9995
--- /dev/null
+++ b/runner/runner_tests.c
@@ -0,0 +1,967 @@
+#include <dirent.h>
+#include <fcntl.h>
+#include <sys/stat.h>
+#include <sys/types.h>
+#include <unistd.h>
+
+#include "igt.h"
+
+#include "settings.h"
+#include "job_list.h"
+#include "executor.h"
+
+static char testdatadir[] = TESTDATA_DIRECTORY;
+
+static void igt_assert_eqstr(char *one, char *two)
+{
+	if (one == NULL && two == NULL)
+		return;
+
+	igt_assert_f(one != NULL && two != NULL, "Strings differ (one is NULL): %s vs %s\n", one, two);
+
+	igt_assert_f(!strcmp(one, two), "Strings differ: '%s' vs '%s'\n", one, two);
+}
+
+static void debug_print_executions(struct job_list *list)
+{
+	size_t i;
+	int k;
+
+	igt_debug("Executions:\n");
+	for (i = 0; i < list->size; i++) {
+		struct job_list_entry *entry = &list->entries[i];
+		igt_debug(" %s\n", entry->binary);
+		for (k = 0; k < entry->subtest_count; ++k) {
+			igt_debug("  %s\n", entry->subtests[k]);
+		}
+	}
+
+}
+
+static char *dump_file(int dirfd, char *name)
+{
+	int fd = openat(dirfd, name, O_RDONLY);
+	ssize_t s;
+	char *buf = malloc(256);
+
+	if (fd < 0) {
+		free(buf);
+		return NULL;
+	}
+
+	s = read(fd, buf, 255);
+	if (s < 0) {
+		free(buf);
+		return NULL;
+	}
+
+	buf[s] = '\0';
+	return buf;
+}
+
+static void job_list_filter_test(char *name, char *filterarg1, char *filterarg2,
+				 size_t expected_normal, size_t expected_multiple)
+{
+	int multiple;
+	struct settings settings;
+
+	igt_fixture
+		init_settings(&settings);
+
+	for (multiple = 0; multiple < 2; multiple++) {
+		igt_subtest_f("job-list-filters-%s-%s", name, multiple ? "multiple" : "normal") {
+			struct job_list list;
+			char *argv[] = { "runner",
+					 /* Ugly but does the trick */
+					 multiple ? "--multiple-mode" : "--sync",
+					 filterarg1, filterarg2,
+					 testdatadir,
+					 "path-to-results",
+			};
+			bool success = false;
+			size_t size;
+
+			init_job_list(&list);
+			igt_assert(parse_options(ARRAY_SIZE(argv), argv, &settings));
+
+			success = create_job_list(&list, &settings);
+			size = list.size;
+
+			if (success)
+				debug_print_executions(&list);
+
+			free_job_list(&list);
+
+			igt_assert_f(success, "Job list creation failed\n");
+			igt_assert_eq(size, multiple ? expected_multiple : expected_normal);
+		}
+	}
+
+	igt_fixture
+		free_settings(&settings);
+}
+
+static void clear_directory(int dirfd)
+{
+	DIR* d;
+	struct dirent *dirent;
+
+	d = fdopendir(dirfd);
+
+	if (dirfd < 0 || d == NULL) {
+		return;
+	}
+
+	while ((dirent = readdir(d)) != NULL) {
+		if (strcmp(dirent->d_name, ".") &&
+		    strcmp(dirent->d_name, "..")) {
+			if (dirent->d_type == DT_REG) {
+				unlinkat(dirfd, dirent->d_name, 0);
+			} else if (dirent->d_type == DT_DIR) {
+				clear_directory(openat(dirfd, dirent->d_name, O_DIRECTORY | O_RDONLY));
+				unlinkat(dirfd, dirent->d_name, AT_REMOVEDIR);
+			}
+		}
+	}
+
+	closedir(d);
+}
+
+static void assert_settings_equal(struct settings *one, struct settings *two)
+{
+	/*
+	 * Regex lists are not serialized, and thus won't be compared
+	 * here.
+	 */
+	igt_assert_eq(one->abort_on_error, two->abort_on_error);
+	igt_assert_eqstr(one->test_list, two->test_list);
+	igt_assert_eqstr(one->name, two->name);
+	igt_assert_eq(one->dry_run, two->dry_run);
+	igt_assert_eq(one->sync, two->sync);
+	igt_assert_eq(one->log_level, two->log_level);
+	igt_assert_eq(one->overwrite, two->overwrite);
+	igt_assert_eq(one->multiple_mode, two->multiple_mode);
+	igt_assert_eq(one->inactivity_timeout, two->inactivity_timeout);
+	igt_assert_eq(one->use_watchdog, two->use_watchdog);
+	igt_assert_eqstr(one->test_root, two->test_root);
+	igt_assert_eqstr(one->results_path, two->results_path);
+}
+
+static void assert_job_list_equal(struct job_list *one, struct job_list *two)
+{
+	size_t i, k;
+
+	igt_assert_eq(one->size, two->size);
+
+	for (i = 0; i < one->size; i++) {
+		struct job_list_entry *eone = &one->entries[i];
+		struct job_list_entry *etwo = &two->entries[i];
+
+		igt_assert_eqstr(eone->binary, etwo->binary);
+		igt_assert_eq(eone->subtest_count, etwo->subtest_count);
+
+		for (k = 0; k < eone->subtest_count; k++) {
+			igt_assert_eqstr(eone->subtests[k], etwo->subtests[k]);
+		}
+	}
+}
+
+static void assert_execution_created(int dirfd, char *name)
+{
+	int fd;
+
+	igt_assert_f((fd = openat(dirfd, name, O_RDONLY)) >= 0,
+		     "Execute didn't create %s\n", name);
+	close(fd);
+}
+
+static void assert_execution_results_exist(int dirfd)
+{
+	assert_execution_created(dirfd, "journal.txt");
+	assert_execution_created(dirfd, "out.txt");
+	assert_execution_created(dirfd, "err.txt");
+	assert_execution_created(dirfd, "dmesg.txt");
+}
+
+igt_main
+{
+	struct settings settings;
+
+	igt_fixture
+		init_settings(&settings);
+
+	igt_subtest("default-settings") {
+		char *argv[] = { "runner",
+				 "test-root-dir",
+				 "path-to-results",
+		};
+
+		igt_assert(parse_options(ARRAY_SIZE(argv), argv, &settings));
+
+		igt_assert(!settings.abort_on_error);
+		igt_assert(!settings.test_list);
+		igt_assert_eqstr(settings.name, "path-to-results");
+		igt_assert(!settings.dry_run);
+		igt_assert_eq(settings.include_regexes.size, 0);
+		igt_assert_eq(settings.exclude_regexes.size, 0);
+		igt_assert(!settings.sync);
+		igt_assert_eq(settings.log_level, LOG_LEVEL_NORMAL);
+		igt_assert(!settings.overwrite);
+		igt_assert(!settings.multiple_mode);
+		igt_assert_eq(settings.inactivity_timeout, 0);
+		igt_assert(!settings.use_watchdog);
+		igt_assert(strstr(settings.test_root, "test-root-dir") != NULL);
+		igt_assert(strstr(settings.results_path, "path-to-results") != NULL);
+	}
+
+	igt_subtest_group {
+		char *cwd;
+		char *path;
+
+		igt_fixture {
+			igt_require((cwd = realpath(".", NULL)) != NULL);
+			path = NULL;
+		}
+
+		igt_subtest("absolute-path-converter") {
+			struct {
+				char *path;
+				bool null;
+			} data[] = { { "simple-name", false },
+				     { "foo/bar", true },
+				     { ".", false },
+			};
+			size_t i;
+
+			for (i = 0; i < ARRAY_SIZE(data); i++) {
+				free(path);
+				path = absolute_path(data[i].path);
+				if (data[i].null) {
+					igt_assert(path == NULL);
+					continue;
+				}
+
+				igt_assert(path[0] == '/');
+				igt_debug("Got path %s for %s\n", path, data[i].path);
+				igt_assert(strstr(path, cwd) == path);
+				if (strcmp(data[i].path, ".")) {
+					igt_assert(strstr(path, data[i].path) != NULL);
+				}
+			}
+		}
+
+		igt_fixture {
+			free(cwd);
+			free(path);
+		}
+	}
+
+	igt_subtest_group {
+		char dirname[] = "tmpdirXXXXXX";
+		char tmptestlist[] = "tmp.testlist";
+		char pathtotestlist[64];
+		char *path;
+
+		igt_fixture {
+			int dirfd, fd;
+
+			path = NULL;
+
+			igt_require(mkdtemp(dirname) != NULL);
+			igt_require((dirfd = open(dirname, O_DIRECTORY | O_RDONLY)) >= 0);
+			igt_require((fd = openat(dirfd, tmptestlist, O_CREAT | O_EXCL | O_WRONLY, 0660)) >= 0);
+			close(fd);
+			close(dirfd);
+
+			strcpy(pathtotestlist, dirname);
+			strcat(pathtotestlist, "/");
+			strcat(pathtotestlist, tmptestlist);
+		}
+
+		igt_subtest("absolute-path-usage") {
+			char *argv[] = { "runner",
+					 "--test-list", pathtotestlist,
+					 testdatadir,
+					 dirname,
+			};
+
+			igt_assert(parse_options(ARRAY_SIZE(argv), argv, &settings));
+
+			path = realpath(testdatadir, NULL);
+			igt_assert(path != NULL);
+			igt_assert_eqstr(settings.test_root, path);
+			free(path);
+			path = realpath(dirname, NULL);
+			igt_assert(path != NULL);
+			igt_assert_eqstr(settings.results_path, path);
+			free(path);
+			path = realpath(pathtotestlist, NULL);
+			igt_assert(path != NULL);
+			igt_assert_eqstr(settings.test_list, path);
+		}
+
+		igt_fixture {
+			int dirfd;
+
+			igt_require((dirfd = open(dirname, O_DIRECTORY | O_RDONLY)) >= 0);
+			unlinkat(dirfd, tmptestlist, 0);
+			close(dirfd);
+			rmdir(dirname);
+
+			free(path);
+		}
+	}
+
+	igt_subtest("environment-overrides-test-root-flag") {
+		char *argv[] = { "runner",
+				 "test-root-dir",
+				 "path-to-results",
+		};
+
+		setenv("IGT_TEST_ROOT", testdatadir, 1);
+		igt_assert(parse_options(ARRAY_SIZE(argv), argv, &settings));
+
+		igt_assert(!settings.abort_on_error);
+		igt_assert(!settings.test_list);
+		igt_assert_eqstr(settings.name, "path-to-results");
+		igt_assert(!settings.dry_run);
+		igt_assert_eq(settings.include_regexes.size, 0);
+		igt_assert_eq(settings.exclude_regexes.size, 0);
+		igt_assert(!settings.sync);
+		igt_assert_eq(settings.log_level, LOG_LEVEL_NORMAL);
+		igt_assert(!settings.overwrite);
+		igt_assert(!settings.multiple_mode);
+		igt_assert_eq(settings.inactivity_timeout, 0);
+		igt_assert(!settings.use_watchdog);
+		igt_assert(strstr(settings.test_root, testdatadir) != NULL);
+		igt_assert(strstr(settings.results_path, "path-to-results") != NULL);
+	}
+
+	igt_fixture {
+		unsetenv("IGT_TEST_ROOT");
+	}
+
+	igt_subtest("parse-all-settings") {
+		char *argv[] = { "runner",
+				 "-n", "foo",
+				 "--abort-on-monitored-error",
+				 "--test-list", "path-to-test-list",
+				 "--ignore-missing",
+				 "--dry-run",
+				 "-t", "pattern1",
+				 "-t", "pattern2",
+				 "-x", "xpattern1",
+				 "-x", "xpattern2",
+				 "-s",
+				 "-l", "verbose",
+				 "--overwrite",
+				 "--multiple-mode",
+				 "--inactivity-timeout", "27",
+				 "--use-watchdog",
+				 "test-root-dir",
+				 "path-to-results",
+		};
+
+		igt_assert(parse_options(ARRAY_SIZE(argv), argv, &settings));
+
+		igt_assert(settings.abort_on_error);
+		igt_assert(strstr(settings.test_list, "path-to-test-list") != NULL);
+		igt_assert_eqstr(settings.name, "foo");
+		igt_assert(settings.dry_run);
+		igt_assert_eq(settings.include_regexes.size, 2);
+		igt_assert_eqstr(settings.include_regexes.regex_strings[0], "pattern1");
+		igt_assert_eqstr(settings.include_regexes.regex_strings[1], "pattern2");
+		igt_assert_eq(settings.exclude_regexes.size, 2);
+		igt_assert_eqstr(settings.exclude_regexes.regex_strings[0], "xpattern1");
+		igt_assert_eqstr(settings.exclude_regexes.regex_strings[1], "xpattern2");
+		igt_assert(settings.sync);
+		igt_assert_eq(settings.log_level, LOG_LEVEL_VERBOSE);
+		igt_assert(settings.overwrite);
+		igt_assert(settings.multiple_mode);
+		igt_assert_eq(settings.inactivity_timeout, 27);
+		igt_assert(settings.use_watchdog);
+		igt_assert(strstr(settings.test_root, "test-root-dir") != NULL);
+		igt_assert(strstr(settings.results_path, "path-to-results") != NULL);
+	}
+
+	igt_subtest("invalid-option") {
+		char *argv[] = { "runner",
+				 "--no-such-option",
+				 "test-root-dir",
+				 "results-path",
+		};
+
+		igt_assert(!parse_options(ARRAY_SIZE(argv), argv, &settings));
+	}
+
+	igt_subtest("paths-missing") {
+		char *argv[] = { "runner",
+				 "-o",
+		};
+		igt_assert(!parse_options(ARRAY_SIZE(argv), argv, &settings));
+	}
+
+	igt_subtest("log-levels") {
+		char *argv[] = { "runner",
+				 "-l", "normal",
+				 "test-root-dir",
+				 "results-path",
+		};
+
+		igt_assert(parse_options(ARRAY_SIZE(argv), argv, &settings));
+		igt_assert_eq(settings.log_level, LOG_LEVEL_NORMAL);
+
+		argv[2] = "quiet";
+		igt_assert(parse_options(ARRAY_SIZE(argv), argv, &settings));
+		igt_assert_eq(settings.log_level, LOG_LEVEL_QUIET);
+
+		argv[2] = "verbose";
+		igt_assert(parse_options(ARRAY_SIZE(argv), argv, &settings));
+		igt_assert_eq(settings.log_level, LOG_LEVEL_VERBOSE);
+	}
+
+	igt_subtest("parse-clears-old-data") {
+		char *argv[] = { "runner",
+				 "-n", "foo",
+				 "--dry-run",
+				 "test-root-dir",
+				 "results-path",
+		};
+
+		igt_assert(parse_options(ARRAY_SIZE(argv), argv, &settings));
+
+		igt_assert_eqstr(settings.name, "foo");
+		igt_assert(settings.dry_run);
+		igt_assert(!settings.test_list);
+		igt_assert(!settings.sync);
+
+		argv[1] = "--test-list";
+		argv[3] = "--sync";
+
+		igt_assert(parse_options(ARRAY_SIZE(argv), argv, &settings));
+
+		igt_assert_eqstr(settings.name, "results-path");
+		igt_assert(!settings.dry_run);
+		igt_assert(strstr(settings.test_list, "foo") != NULL);
+		igt_assert(settings.sync);
+	}
+
+	igt_subtest_group {
+		char filename[] = "tmplistXXXXXX";
+		int fd = -1;
+
+		igt_fixture {
+			igt_require((fd = mkstemp(filename)) >= 0);
+			close(fd);
+		}
+
+		igt_subtest("validate-ok") {
+			char *argv[] = { "runner",
+					 "--test-list", filename,
+					 testdatadir,
+					 "path-to-results",
+			};
+
+			igt_assert(parse_options(ARRAY_SIZE(argv), argv, &settings));
+
+			igt_assert(validate_settings(&settings));
+		}
+
+		igt_fixture {
+			unlink(filename);
+		}
+	}
+
+	igt_subtest("validate-no-test-list") {
+		char *nosuchfile = "no-such-file";
+		char *argv[] = { "runner",
+				 "--test-list", nosuchfile,
+				 testdatadir,
+				 "path-to-results",
+		};
+
+		igt_assert_lt(open(nosuchfile, O_RDONLY), 0);
+		igt_assert(parse_options(ARRAY_SIZE(argv), argv, &settings));
+
+		igt_assert(!validate_settings(&settings));
+	}
+
+	igt_subtest_group {
+		char dirname[] = "tmpdirXXXXXX";
+		struct job_list list;
+
+		igt_fixture {
+			igt_require(mkdtemp(dirname) != NULL);
+			init_job_list(&list);
+		}
+
+		igt_subtest("job-list-no-test-list-txt") {
+			char *argv[] = { "runner",
+					 dirname,
+					 "path-to-results",
+			};
+
+			igt_assert(parse_options(ARRAY_SIZE(argv), argv, &settings));
+
+			igt_assert(!create_job_list(&list, &settings));
+		}
+
+		igt_fixture {
+			rmdir(dirname);
+			free_job_list(&list);
+		}
+	}
+
+	job_list_filter_test("nofilters", "-n", "placeholderargs", 5, 3);
+	job_list_filter_test("binary-include", "-t", "successtest", 2, 1);
+	job_list_filter_test("binary-exclude", "-x", "successtest", 3, 2);
+	job_list_filter_test("subtest-include", "-t", "first-subtest", 1, 1);
+	job_list_filter_test("subtest-exclude", "-x", "second-subtest", 4, 3);
+
+	igt_subtest_group {
+		char filename[] = "tmplistXXXXXX";
+		char testlisttext[] = "igt@successtest@first-subtest\n"
+			"igt@successtest@second-subtest\n"
+			"igt@nosubtests\n";
+		int fd = -1, multiple;
+		struct job_list list;
+
+		igt_fixture {
+			igt_require((fd = mkstemp(filename)) >= 0);
+			igt_require(write(fd, testlisttext, sizeof(testlisttext)) == sizeof(testlisttext));
+			close(fd);
+			init_job_list(&list);
+		}
+
+		for (multiple = 0; multiple < 2; multiple++) {
+			igt_subtest_f("job-list-testlist-%s", multiple ? "multiple" : "normal") {
+				char *argv[] = { "runner",
+						 "--test-list", filename,
+						 multiple ? "--multiple-mode" : "--sync",
+						 testdatadir,
+						 "path-to-results",
+				};
+
+				igt_assert(parse_options(ARRAY_SIZE(argv), argv, &settings));
+				igt_assert(create_job_list(&list, &settings));
+
+				igt_assert_eq(list.size, multiple ? 2 : 3);
+
+				igt_assert_eqstr(list.entries[0].binary, "successtest");
+				if (!multiple) igt_assert_eqstr(list.entries[1].binary, "successtest");
+				igt_assert_eqstr(list.entries[multiple ? 1 : 2].binary, "nosubtests");
+
+				igt_assert_eq(list.entries[0].subtest_count, multiple ? 2 : 1);
+				igt_assert_eq(list.entries[1].subtest_count, multiple ? 0 : 1);
+				if (!multiple) igt_assert_eq(list.entries[2].subtest_count, 0);
+
+				igt_assert_eqstr(list.entries[0].subtests[0], "first-subtest");
+				igt_assert_eqstr(list.entries[multiple ? 0 : 1].subtests[multiple ? 1 : 0], "second-subtest");
+			}
+		}
+
+		igt_fixture {
+			unlink(filename);
+			free_job_list(&list);
+		}
+	}
+
+	igt_subtest_group {
+		char dirname[] = "tmpdirXXXXXX";
+		int dirfd = -1, fd = -1;
+		struct settings cmp_settings;
+
+		igt_fixture {
+			igt_require(mkdtemp(dirname) != NULL);
+			rmdir(dirname);
+			init_settings(&cmp_settings);
+		}
+
+		igt_subtest("settings-serialize") {
+			char *argv[] = { "runner",
+					 testdatadir,
+					 dirname,
+			};
+
+			igt_assert(parse_options(ARRAY_SIZE(argv), argv, &settings));
+
+			igt_assert(serialize_settings(&settings));
+
+			dirfd = open(dirname, O_DIRECTORY, O_RDONLY);
+			igt_assert_f(dirfd >= 0, "Serialization did not create the results directory\n");
+
+			igt_assert_f((fd = openat(dirfd, "metadata.txt", O_RDONLY)),
+				     "Opening %s/metadata.txt failed\n", dirname);
+			close(fd);
+
+			igt_assert_f(read_settings(&cmp_settings, dirfd), "Reading settings failed\n");
+			assert_settings_equal(&settings, &cmp_settings);
+		}
+
+		igt_fixture {
+			if (fd >= 0) close(fd);
+			if (dirfd >= 0)
+				clear_directory(dirfd);
+			rmdir(dirname);
+			free_settings(&cmp_settings);
+		}
+	}
+
+	igt_subtest_group {
+		char dirname[] = "tmpdirXXXXXX";
+		int dirfd = -1, fd = -1;
+		struct job_list list, cmp_list;
+		int multiple;
+
+		igt_fixture {
+			init_job_list(&list);
+			init_job_list(&cmp_list);
+			igt_require(mkdtemp(dirname) != NULL);
+			rmdir(dirname);
+		}
+
+		for (multiple = 0; multiple < 2; multiple++) {
+			igt_subtest_f("job-list-serialize-%s", multiple ? "multiple" : "normal") {
+				char *argv[] = { "runner",
+						 /* Ugly */
+						 multiple ? "--multiple-mode" : "--sync",
+						 testdatadir,
+						 dirname,
+				};
+
+				igt_assert(parse_options(ARRAY_SIZE(argv), argv, &settings));
+				igt_assert(create_job_list(&list, &settings));
+
+				igt_assert(serialize_settings(&settings));
+				igt_assert(serialize_job_list(&list, &settings));
+
+				dirfd = open(dirname, O_DIRECTORY, O_RDONLY);
+				igt_assert_f(dirfd >= 0, "Serialization did not create the results directory\n");
+
+				igt_assert_f((fd = openat(dirfd, "joblist.txt", O_RDONLY)) >= 0,
+					     "Opening %s/joblist.txt failed\n", dirname);
+				close(fd);
+				fd = -1;
+
+				igt_assert_f(read_job_list(&cmp_list, dirfd), "Reading job list failed\n");
+				assert_job_list_equal(&list, &cmp_list);
+			}
+
+			igt_fixture {
+				if (fd >= 0) close(fd);
+				if (dirfd >= 0)
+					clear_directory(dirfd);
+				rmdir(dirname);
+				free_job_list(&cmp_list);
+				free_job_list(&list);
+			}
+		}
+	}
+
+	igt_subtest_group {
+		char dirname[] = "tmpdirXXXXXX";
+		struct job_list list;
+		int dirfd = -1, fd = -1;
+
+		igt_fixture {
+			init_job_list(&list);
+			igt_require(mkdtemp(dirname) != NULL);
+			rmdir(dirname);
+		}
+
+		igt_subtest("execute-initialize-new-run") {
+			struct execute_state state;
+			char *argv[] = { "runner",
+					 testdatadir,
+					 dirname,
+			};
+
+			igt_assert(parse_options(ARRAY_SIZE(argv), argv, &settings));
+			igt_assert(create_job_list(&list, &settings));
+
+			igt_assert(initialize_execute_state(&state, &settings, &list));
+
+			igt_assert_eq(state.next, 0);
+			igt_assert_eq(list.size, 5);
+			igt_assert_f((dirfd = open(dirname, O_DIRECTORY | O_RDONLY)) >= 0,
+				     "Execute state initialization didn't create the results directory.\n");
+			igt_assert_f((fd = openat(dirfd, "metadata.txt", O_RDONLY)) >= 0,
+				     "Execute state initialization didn't serialize settings.\n");
+			close(fd);
+			igt_assert_f((fd = openat(dirfd, "joblist.txt", O_RDONLY)) >= 0,
+				     "Execute state initialization didn't serialize the job list.\n");
+			close(fd);
+			igt_assert_f((fd = openat(dirfd, "journal.txt", O_RDONLY)) < 0,
+				     "Execute state initialization created a journal.\n");
+			igt_assert_f((fd = openat(dirfd, "uname.txt", O_RDONLY)) < 0,
+				     "Execute state initialization created uname.txt.\n");
+		}
+
+		igt_fixture {
+			if (fd >= 0) close(fd);
+			if (dirfd >= 0)
+				clear_directory(dirfd);
+			rmdir(dirname);
+			free_job_list(&list);
+		}
+	}
+
+	igt_subtest_group {
+		char dirname[] = "tmpdirXXXXXX";
+		struct job_list list;
+		int dirfd = -1, subdirfd = -1, fd = -1;
+
+		igt_fixture {
+			init_job_list(&list);
+			igt_require(mkdtemp(dirname) != NULL);
+		}
+
+		igt_subtest("execute-initialize-subtest-started") {
+			struct execute_state state;
+			char *argv[] = { "runner",
+					 "--multiple-mode",
+					 "-t", "successtest",
+					 testdatadir,
+					 dirname,
+			};
+			char journaltext[] = "first-subtest\n";
+			char excludestring[] = "!first-subtest";
+
+			igt_assert(parse_options(ARRAY_SIZE(argv), argv, &settings));
+			igt_assert(create_job_list(&list, &settings));
+			igt_assert(list.size == 1);
+			igt_assert(list.entries[0].subtest_count == 0);
+
+			igt_assert(serialize_settings(&settings));
+			igt_assert(serialize_job_list(&list, &settings));
+
+			igt_assert((dirfd = open(dirname, O_DIRECTORY | O_RDONLY)) >= 0);
+			igt_assert(mkdirat(dirfd, "0", 0770) == 0);
+			igt_assert((subdirfd = openat(dirfd, "0", O_DIRECTORY | O_RDONLY)) >= 0);
+			igt_assert((fd = openat(subdirfd, "journal.txt", O_CREAT | O_WRONLY | O_EXCL, 0660)) >= 0);
+			igt_assert(write(fd, journaltext, sizeof(journaltext)) == sizeof(journaltext));
+
+			igt_assert(initialize_execute_state(&state, &settings, &list));
+
+			igt_assert_eq(state.next, 0);
+			igt_assert_eq(list.size, 1);
+			igt_assert_eq(list.entries[0].subtest_count, 2);
+			igt_assert_eqstr(list.entries[0].subtests[0], "*");
+			igt_assert_eqstr(list.entries[0].subtests[1], excludestring);
+		}
+
+		igt_fixture {
+			if (fd >= 0) close(fd);
+			if (subdirfd >= 0) close(subdirfd);
+			if (dirfd >= 0)
+				clear_directory(dirfd);
+			rmdir(dirname);
+			free_job_list(&list);
+		}
+	}
+
+	igt_subtest_group {
+		char dirname[] = "tmpdirXXXXXX";
+		struct job_list list;
+		int dirfd = -1, subdirfd = -1, fd = -1;
+
+		igt_fixture {
+			init_job_list(&list);
+			igt_require(mkdtemp(dirname) != NULL);
+		}
+
+		igt_subtest("execute-initialize-subtests-complete") {
+			struct execute_state state;
+			char *argv[] = { "runner",
+					 "--multiple-mode",
+					 testdatadir,
+					 dirname,
+			};
+			char journaltext[] = "first-subtest\nsecond-subtest\nexit:0\n";
+
+			igt_assert(parse_options(ARRAY_SIZE(argv), argv, &settings));
+			igt_assert(create_job_list(&list, &settings));
+			igt_assert(list.size == 3);
+
+			if (!strcmp(list.entries[0].binary, "no-subtests")) {
+				struct job_list_entry tmp = list.entries[0];
+				list.entries[0] = list.entries[1];
+				list.entries[1] = tmp;
+			}
+
+			igt_assert(list.entries[0].subtest_count == 0);
+
+			igt_assert(serialize_settings(&settings));
+			igt_assert(serialize_job_list(&list, &settings));
+
+			igt_assert_lte(0, dirfd = open(dirname, O_DIRECTORY | O_RDONLY));
+			igt_assert_eq(mkdirat(dirfd, "0", 0770), 0);
+			igt_assert((subdirfd = openat(dirfd, "0", O_DIRECTORY | O_RDONLY)) >= 0);
+			igt_assert_lte(0, fd = openat(subdirfd, "journal.txt", O_CREAT | O_WRONLY | O_EXCL, 0660));
+			igt_assert_eq(write(fd, journaltext, sizeof(journaltext)), sizeof(journaltext));
+
+			igt_assert(initialize_execute_state(&state, &settings, &list));
+
+			igt_assert_eq(state.next, 1);
+			igt_assert_eq(list.size, 3);
+		}
+
+		igt_fixture {
+			if (fd >= 0) close(fd);
+			if (subdirfd >= 0) close(subdirfd);
+			if (dirfd >= 0)
+				clear_directory(dirfd);
+			rmdir(dirname);
+			free_job_list(&list);
+		}
+	}
+
+	igt_subtest_group {
+		struct job_list list;
+		int dirfd = -1, subdirfd = -1, fd = -1;
+		int multiple;
+
+		igt_fixture {
+			init_job_list(&list);
+		}
+
+		for (multiple = 0; multiple < 2; multiple++) {
+			char dirname[] = "tmpdirXXXXXX";
+
+			igt_fixture {
+				igt_require(mkdtemp(dirname) != NULL);
+			}
+
+			igt_subtest_f("execute-subtests-%s", multiple ? "multiple" : "normal") {
+				struct execute_state state;
+				char *argv[] = { "runner",
+						 multiple ? "--multiple-mode" : "--sync",
+						 "-t", "-subtest",
+						 testdatadir,
+						 dirname,
+				};
+				char testdirname[16];
+				size_t expected_tests = multiple ? 2 : 3;
+				size_t i;
+
+				igt_assert(parse_options(ARRAY_SIZE(argv), argv, &settings));
+				igt_assert(create_job_list(&list, &settings));
+				igt_assert(initialize_execute_state(&state, &settings, &list));
+
+				igt_assert(execute(&state, &settings, &list));
+				igt_assert_f((dirfd = open(dirname, O_DIRECTORY | O_RDONLY)) >= 0,
+					     "Execute didn't create the results directory\n");
+
+				igt_assert_f((fd = openat(dirfd, "uname.txt", O_RDONLY)) >= 0,
+					     "Execute didn't create uname.txt\n");
+				close(fd);
+				fd = -1;
+
+				for (i = 0; i < expected_tests; i++) {
+					snprintf(testdirname, 16, "%zd", i);
+
+					igt_assert_f((subdirfd = openat(dirfd, testdirname, O_DIRECTORY | O_RDONLY)) >= 0,
+						     "Execute didn't create result directory '%s'\n", testdirname);
+					assert_execution_results_exist(subdirfd);
+					close(subdirfd);
+				}
+
+				snprintf(testdirname, 16, "%zd", expected_tests);
+				igt_assert_f((subdirfd = openat(dirfd, testdirname, O_DIRECTORY | O_RDONLY)) < 0,
+					     "Execute created too many directories\n");
+			}
+
+			igt_fixture {
+				if (fd >= 0) close(fd);
+				if (subdirfd >= 0) close(subdirfd);
+				if (dirfd >= 0)
+					clear_directory(dirfd);
+				rmdir(dirname);
+				free_job_list(&list);
+			}
+		}
+	}
+
+	igt_subtest_group {
+		struct job_list list;
+		int dirfd = -1, subdirfd = -1, fd = -1;
+		int multiple;
+
+		igt_fixture {
+			init_job_list(&list);
+		}
+
+		for (multiple = 0; multiple < 2; multiple++) {
+			char dirname[] = "tmpdirXXXXXX";
+
+			igt_fixture {
+				igt_require(mkdtemp(dirname) != NULL);
+			}
+
+			igt_subtest_f("execute-skipper-journal-%s", multiple ? "multiple" : "normal") {
+				struct execute_state state;
+				char *argv[] = { "runner",
+						 multiple ? "--multiple-mode" : "--sync",
+						 "-t", "skippers",
+						 testdatadir,
+						 dirname,
+				};
+				char *dump;
+				char *expected_0 = multiple ?
+					"skip-one\nskip-two\nexit:77 (" :
+					"skip-one\nexit:77 (";
+				char *expected_1 = "skip-two\nexit:77 (";
+
+				igt_assert(parse_options(ARRAY_SIZE(argv), argv, &settings));
+				igt_assert(create_job_list(&list, &settings));
+				igt_assert(initialize_execute_state(&state, &settings, &list));
+
+				igt_assert(execute(&state, &settings, &list));
+				igt_assert_f((dirfd = open(dirname, O_DIRECTORY | O_RDONLY)) >= 0,
+					     "Execute didn't create the results directory\n");
+
+				igt_assert_f((fd = openat(dirfd, "uname.txt", O_RDONLY)) >= 0,
+					     "Execute didn't create uname.txt\n");
+				close(fd);
+				fd = -1;
+
+				igt_assert_f((subdirfd = openat(dirfd, "0", O_DIRECTORY | O_RDONLY)) >= 0,
+					     "Execute didn't create result directory '0'\n");
+				dump = dump_file(subdirfd, "journal.txt");
+				igt_assert_f(dump != NULL,
+					     "Execute didn't create the journal\n");
+				/* Trim out the runtime */
+				dump[strlen(expected_0)] = '\0';
+				igt_assert_eqstr(dump, expected_0);
+				free(dump);
+				close(subdirfd);
+				subdirfd = -1;
+
+				if (!multiple) {
+					igt_assert_f((subdirfd = openat(dirfd, "1", O_DIRECTORY | O_RDONLY)) >= 0,
+						     "Execute didn't create result directory '1'\n");
+					dump = dump_file(subdirfd, "journal.txt");
+					igt_assert_f(dump != NULL,
+						     "Execute didn't create the journal\n");
+					dump[strlen(expected_1)] = '\0';
+					igt_assert_eqstr(dump, expected_1);
+					free(dump);
+					close(subdirfd);
+					subdirfd = -1;
+				}
+			}
+
+			igt_fixture {
+				if (fd >= 0) close(fd);
+				if (subdirfd >= 0) close(subdirfd);
+				if (dirfd >= 0)
+					clear_directory(dirfd);
+				rmdir(dirname);
+				free_job_list(&list);
+			}
+		}
+	}
+
+	igt_fixture
+		free_settings(&settings);
+}
diff --git a/runner/testdata/meson.build b/runner/testdata/meson.build
new file mode 100644
index 00000000..011eff8e
--- /dev/null
+++ b/runner/testdata/meson.build
@@ -0,0 +1,20 @@
+
+testdata_progs = [ 'successtest',
+		   'no-subtests',
+		   'skippers',
+		 ]
+
+testdata_executables = []
+
+foreach prog : testdata_progs
+	testdata_executables += executable(prog, prog + '.c',
+					   dependencies : igt_deps,
+					   install : false)
+endforeach
+
+testdata_list = custom_target('testdata_testlist',
+			      output : 'test-list.txt',
+			      command : [ gen_testlist, '@OUTPUT@', testdata_progs ],
+			      build_by_default : true)
+
+testdata_dir = meson.current_build_dir()
diff --git a/runner/testdata/no-subtests.c b/runner/testdata/no-subtests.c
new file mode 100644
index 00000000..00c92e25
--- /dev/null
+++ b/runner/testdata/no-subtests.c
@@ -0,0 +1,6 @@
+#include "igt.h"
+
+igt_simple_main
+{
+
+}
diff --git a/runner/testdata/skippers.c b/runner/testdata/skippers.c
new file mode 100644
index 00000000..be4a31b4
--- /dev/null
+++ b/runner/testdata/skippers.c
@@ -0,0 +1,14 @@
+#include "igt.h"
+
+igt_main
+{
+	igt_fixture {
+		igt_require_f(false, "Skipping from fixture\n");
+	}
+
+	igt_subtest("skip-one")
+		igt_debug("Should be skipped\n");
+
+	igt_subtest("skip-two")
+		igt_debug("Should be skipped\n");
+}
diff --git a/runner/testdata/successtest.c b/runner/testdata/successtest.c
new file mode 100644
index 00000000..bb411eb4
--- /dev/null
+++ b/runner/testdata/successtest.c
@@ -0,0 +1,10 @@
+#include "igt.h"
+
+igt_main
+{
+	igt_subtest("first-subtest")
+		igt_debug("Running first subtest\n");
+
+	igt_subtest("second-subtest")
+		igt_debug("Running second subtest\n");
+}
-- 
2.14.1

_______________________________________________
igt-dev mailing list
igt-dev@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/igt-dev

[igt-dev] ✓ Fi.CI.BAT: success for New runner to rule them all

[Patchwork]

== Series Details ==

Series: New runner to rule them all
URL   : https://patchwork.freedesktop.org/series/42467/
State : success

== Summary ==

= CI Bug Log - changes from CI_DRM_4113 -> IGTPW_1309 =

== Summary - SUCCESS ==

  No regressions found.

  External URL: https://patchwork.freedesktop.org/api/1.0/series/42467/revisions/1/mbox/

== Known issues ==

  Here are the changes found in IGTPW_1309 that come from known issues:

  === IGT changes ===

    ==== Issues hit ====

    igt@kms_pipe_crc_basic@suspend-read-crc-pipe-b:
      fi-snb-2520m:       PASS -> INCOMPLETE (fdo#103713)
      fi-ivb-3520m:       PASS -> DMESG-WARN (fdo#106084) +1

    igt@kms_pipe_crc_basic@suspend-read-crc-pipe-c:
      fi-bxt-dsi:         PASS -> INCOMPLETE (fdo#103927)

    
  fdo#103713 https://bugs.freedesktop.org/show_bug.cgi?id=103713
  fdo#103927 https://bugs.freedesktop.org/show_bug.cgi?id=103927
  fdo#106084 https://bugs.freedesktop.org/show_bug.cgi?id=106084


== Participating hosts (38 -> 35) ==

  Missing    (3): fi-ctg-p8600 fi-ilk-m540 fi-skl-6700hq 


== Build changes ==

    * IGT: IGT_4452 -> IGTPW_1309

  CI_DRM_4113: 1d2a421b1f9b47883b9d0eeb28dc4069e462dbe3 @ git://anongit.freedesktop.org/gfx-ci/linux
  IGTPW_1309: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_1309/
  IGT_4452: 29ae12bd764e3b1876356e7628a32192b4ec9066 @ git://anongit.freedesktop.org/xorg/app/intel-gpu-tools
  piglit_4452: 04a2952c5b3782eb03cb136bb16d89daaf243f14 @ git://anongit.freedesktop.org/piglit

== Logs ==

For more details see: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_1309/issues.html
_______________________________________________
igt-dev mailing list
igt-dev@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/igt-dev

[igt-dev] ✓ Fi.CI.IGT: success for New runner to rule them all

[Patchwork]

== Series Details ==

Series: New runner to rule them all
URL   : https://patchwork.freedesktop.org/series/42467/
State : success

== Summary ==

= CI Bug Log - changes from IGT_4452_full -> IGTPW_1309_full =

== Summary - WARNING ==

  Minor unknown changes coming with IGTPW_1309_full need to be verified
  manually.
  
  If you think the reported changes have nothing to do with the changes
  introduced in IGTPW_1309_full, please notify your bug team to allow them
  to document this new failure mode, which will reduce false positives in CI.

  External URL: https://patchwork.freedesktop.org/api/1.0/series/42467/revisions/1/mbox/

== Possible new issues ==

  Here are the unknown changes that may have been introduced in IGTPW_1309_full:

  === IGT changes ===

    ==== Warnings ====

    igt@kms_busy@extended-pageflip-hang-oldfb-render-b:
      shard-snb:          SKIP -> PASS

    
== Known issues ==

  Here are the changes found in IGTPW_1309_full that come from known issues:

  === IGT changes ===

    ==== Issues hit ====

    igt@kms_ccs@pipe-b-crc-primary-basic:
      shard-glk:          PASS -> DMESG-WARN (fdo#106247)

    igt@kms_flip@plain-flip-ts-check-interruptible:
      shard-glk:          PASS -> FAIL (fdo#100368)

    igt@kms_rotation_crc@primary-rotation-180:
      shard-snb:          PASS -> FAIL (fdo#103925)

    
    ==== Possible fixes ====

    igt@kms_flip@absolute-wf_vblank-interruptible:
      shard-glk:          FAIL (fdo#106087) -> PASS
      shard-apl:          FAIL (fdo#106087) -> PASS

    igt@kms_flip@dpms-vs-vblank-race-interruptible:
      shard-glk:          FAIL (fdo#103060) -> PASS

    igt@kms_flip@plain-flip-fb-recreate:
      shard-hsw:          FAIL (fdo#100368) -> PASS +1

    igt@kms_plane_multiple@atomic-pipe-a-tiling-x:
      shard-snb:          FAIL (fdo#103166) -> PASS

    
  fdo#100368 https://bugs.freedesktop.org/show_bug.cgi?id=100368
  fdo#103060 https://bugs.freedesktop.org/show_bug.cgi?id=103060
  fdo#103166 https://bugs.freedesktop.org/show_bug.cgi?id=103166
  fdo#103925 https://bugs.freedesktop.org/show_bug.cgi?id=103925
  fdo#106087 https://bugs.freedesktop.org/show_bug.cgi?id=106087
  fdo#106247 https://bugs.freedesktop.org/show_bug.cgi?id=106247


== Participating hosts (8 -> 4) ==

  Missing    (4): pig-skl-6600 pig-glk-j4005 pig-hsw-4770r shard-kbl 


== Build changes ==

    * IGT: IGT_4452 -> IGTPW_1309
    * Linux: CI_DRM_4109 -> CI_DRM_4113

  CI_DRM_4109: e701a0e6315dc85615f83b2ee14d9cb2f425d97d @ git://anongit.freedesktop.org/gfx-ci/linux
  CI_DRM_4113: 1d2a421b1f9b47883b9d0eeb28dc4069e462dbe3 @ git://anongit.freedesktop.org/gfx-ci/linux
  IGTPW_1309: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_1309/
  IGT_4452: 29ae12bd764e3b1876356e7628a32192b4ec9066 @ git://anongit.freedesktop.org/xorg/app/intel-gpu-tools
  piglit_4452: 04a2952c5b3782eb03cb136bb16d89daaf243f14 @ git://anongit.freedesktop.org/piglit

== Logs ==

For more details see: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_1309/shards.html
_______________________________________________
igt-dev mailing list
igt-dev@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/igt-dev

Re: [igt-dev] [PATCH i-g-t 0/4] New runner to rule them all

[David Weinehall]

On Mon, Apr 30, 2018 at 12:28:44PM +0300, Petri Latvala wrote:
> A new test running framework to replace piglit; Refer to patch 3/4 for
> a more thorough explanation.
> 
> Note, building it with autotools not done. Only meson hooked up atm.

Is autotools actually necessary? I can understand maintaining autotools
support for old projects, but implementing it for new projects?

[snip]

Kind regards, David
_______________________________________________
igt-dev mailing list
igt-dev@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/igt-dev

Re: [igt-dev] [PATCH i-g-t 0/4] New runner to rule them all

[Eric Anholt]

[-- Attachment #1.1: Type: text/plain, Size: 336 bytes --]

Petri Latvala <petri.latvala@intel.com> writes:

> A new test running framework to replace piglit; Refer to patch 3/4 for
> a more thorough explanation.
>
> Note, building it with autotools not done. Only meson hooked up atm.

I gave this a try on raspberry pi, and it cut the test runtime from 33s
to 17s.  Thanks for working on this!

[-- Attachment #1.2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

[-- Attachment #2: Type: text/plain, Size: 154 bytes --]

_______________________________________________
igt-dev mailing list
igt-dev@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/igt-dev