Re: [igt-dev] [PATCH i-g-t 3/4] runner: New test runner

[Daniel Vetter <daniel@ffwll.ch>]

On Mon, Apr 30, 2018 at 12:28:47PM +0300, Petri Latvala wrote:
> This is a new test runner to replace piglit. Piglit has been very
> useful as a test runner, but certain improvements have been very
> difficult if possible at all in a generic test running framework.
> 
> Important improvements over piglit:
> 
> - Faster to launch. Being able to make assumptions about what we're
>   executing makes it possible to save significant amounts of time. For
>   example, a testlist file's line "igt@somebinary@somesubtest" already
>   has all the information we need to construct the correct command
>   line to execute that particular subtest, instead of listing all
>   subtests of all test binaries and mapping them to command
>   lines. Same goes for the regexp filters command line flags -t and
>   -x; If we use -x somebinaryname, we don't need to list subtests from
>   somebinaryname, we already know none of them will get executed.
> 
> - Logs of incomplete tests. Piglit collects test output to memory and
>   dumps them to a file when the test is complete. The new runner
>   writes all output to disk immediately.
> 
> - Ability to execute multiple subtests in one binary execution. This
>   was possible with piglit, but its semantics made it very hard to
>   implement in practice. For example, having a testlist file not only
>   selected a subset of tests to run, but also mandated that they be
>   executed in the same order.
> 
> - Flexible timeout support. Instead of mandating a time tests cannot
>   exceed, the new runner has a timeout on inactivity. Activity is
>   any output on the test's stdout or stderr, or kernel activity via
>   /dev/kmsg.
> 
> The runner is fairly piglit compatible. The command line is very
> similar, with a few additions. IGT_TEST_ROOT environment flag is still
> supported, but can also be set via command line (in place of igt.py in
> piglit command line).
> 
> The results are a set of log files, processed into a piglit-compatible
> results.json file (BZ2 compression TODO). There are some new fields in
> the json for extra information:
> 
> - "igt-version" contains the IGT version line. In
>   multiple-subtests-mode the version information is only printed once,
>   so it needs to be duplicated to all subtest results this way.
> - "dmesg-warnings" contains the dmesg lines that triggered a
>   dmesg-warn/dmesg-fail state.

Hm, I thought agreement was that if we do a new test runner then we
definitely don't want it to capture&analyze dmesg? That's why I've
originally done

https://patchwork.freedesktop.org/series/39263/

which you then took over. Is that still part of the overall plan, just not
yet there?
-Daniel

> - Runtime information will be different. Piglit takes a timestamp at
>   the beginning and at the end of execution for runtime. The new
>   runner uses the subtest output text. The binary execution time will
>   also be included; The key "igt@somebinary" will have the runtime of
>   the binary "somebinary", whereas "igt@somebinary@a" etc will have
>   the runtime of the subtests. Substracting the subtest runtimes from
>   the binary runtime yields the total time spent doing setup in
>   igt_fixture blocks.
> 
> Signed-off-by: Petri Latvala <petri.latvala@intel.com>
> Cc: Maarten Lankhorst <maarten.lankhorst@linux.intel.com>
> Cc: Arkadiusz Hiler <arkadiusz.hiler@intel.com>
> Cc: Tomi Sarvela <tomi.p.sarvela@intel.com>
> Cc: Martin Peres <martin.peres@linux.intel.com>
> ---

[ snip]

> +/*
> + * This regexp controls the kmsg handling. All kernel log records that
> + * have log level of warning or higher convert the result to
> + * dmesg-warn/dmesg-fail unless they match this regexp.
> + */
> +
> +#define _ "|"
> +static const char igt_dmesg_whitelist[] =
> +	"ACPI: button: The lid device is not compliant to SW_LID" _
> +	"ACPI: .*: Unable to dock!" _
> +	"IRQ [0-9]+: no longer affine to CPU[0-9]+" _
> +	"IRQ fixup: irq [0-9]+ move in progress, old vector [0-9]+" _
> +	/* i915 tests set module options, expected message */
> +	"Setting dangerous option [a-z_]+ - tainting kernel" _
> +	/* Raw printk() call, uses default log level (warn) */
> +	"Suspending console\\(s\\) \\(use no_console_suspend to debug\\)" _
> +	"atkbd serio[0-9]+: Failed to (deactivate|enable) keyboard on isa[0-9]+/serio[0-9]+" _
> +	"cache: parent cpu[0-9]+ should not be sleeping" _
> +	"hpet[0-9]+: lost [0-9]+ rtc interrupts" _
> +	/* i915 selftests terminate normally with ENODEV from the
> +	 * module load after the testing finishes, which produces this
> +	 * message.
> +	 */
> +	"i915: probe of [0-9:.]+ failed with error -25" _
> +	/* swiotbl warns even when asked not to */
> +	"mock: DMA: Out of SW-IOMMU space for [0-9]+ bytes" _
> +	"usb usb[0-9]+: root hub lost power or was reset"

The above really isn't stuff we should have in the runner :-)
-Daniel

> +	;
> +#undef _
> +
> +static regex_t re;
> +
> +static bool init_regex_whitelist()
> +{
> +	int status = -1;
> +
> +	if (status == -1) {
> +		if (regcomp(&re, igt_dmesg_whitelist, REG_EXTENDED | REG_NOSUB) != 0) {
> +			fprintf(stderr, "Cannot compile dmesg whitelist regexp\n");
> +			status = 0;
> +			return false;
> +		}
> +
> +		status = 1;
> +	}
> +
> +	return status;
> +}
> +
> +static bool fill_from_dmesg(int fd, char *binary,
> +			    struct subtests *subtests,
> +			    struct json_object *tests)
> +{
> +	char *line = NULL, *warnings = NULL, *dmesg = NULL;
> +	size_t linelen = 0, warningslen = 0, dmesglen = 0;
> +	struct json_object *current_test = NULL;
> +	FILE *f = fdopen(fd, "r");
> +	char *igt_name = NULL;
> +	ssize_t read;
> +	size_t i;
> +
> +	if (!f) {
> +		return false;
> +	}
> +
> +	if (init_regex_whitelist() != 1) {
> +		fclose(f);
> +		return false;
> +	}
> +
> +	while ((read = getline(&line, &linelen, f)) > 0) {
> +		char formatted[256];
> +		unsigned flags;
> +		unsigned long long seq, ts_usec;
> +		char continuation;
> +		char *message, *subtest;
> +		int s;
> +
> +		s = sscanf(line, "%u,%llu,%llu,%c;", &flags, &seq, &ts_usec, &continuation);
> +
> +		if (s != 4) {
> +			/*
> +			 * Machine readable key/value pairs begin with
> +			 * a space. We ignore them.
> +			 */
> +			if (line[0] != ' ') {
> +				fprintf(stderr, "Cannot parse kmsg record: %s\n", line);
> +			}
> +
> +			continue;
> +		}
> +
> +		message = strchr(line, ';');
> +		if (!message) {
> +			fprintf(stderr, "No ; found, this shouldn't happen\n");
> +			return false;
> +		}
> +		message++;
> +		snprintf(formatted, sizeof(formatted), "<%u> [%llu.%06llu] %s",
> +			 flags & 0x07, ts_usec / 1000000, ts_usec % 1000000, message);
> +
> +		if ((subtest = strstr(message, starting_subtest_dmesg)) != NULL) {
> +			if (current_test != NULL) {
> +				/* Done with the previous subtest, file up */
> +				json_object_object_add(current_test, "dmesg",
> +						       json_object_new_string_len(dmesg, dmesglen));
> +				if (warnings) {
> +					json_object_object_add(current_test, "dmesg-warnings",
> +							       json_object_new_string_len(warnings, warningslen));
> +					add_or_override_result(current_test, "pass", "dmesg-warn");
> +					add_or_override_result(current_test, "fail", "dmesg-fail");
> +					add_or_override_result(current_test, "warn", "dmesg-warn");
> +				}
> +				free(dmesg);
> +				free(warnings);
> +				dmesg = warnings = NULL;
> +				dmesglen = warningslen = 0;
> +			}
> +
> +			subtest += starting_subtest_dmesg_len;
> +			igt_name = gen_igt_name(binary, subtest);
> +			current_test = get_or_create_json_object(tests, igt_name);
> +		}
> +
> +		if ((flags & 0x07) <= 4 && continuation != 'c' &&
> +		    regexec(&re, message, (size_t)0, NULL, 0) == REG_NOMATCH) {
> +			append_line(&warnings, &warningslen, formatted);
> +		}
> +		append_line(&dmesg, &dmesglen, formatted);
> +	}
> +
> +	if (current_test != NULL) {
> +		json_object_object_add(current_test, "dmesg",
> +				       json_object_new_string_len(dmesg, dmesglen));
> +		if (warnings) {
> +			json_object_object_add(current_test, "dmesg-warnings",
> +					       json_object_new_string_len(warnings, warningslen));
> +			add_or_override_result(current_test, "pass", "dmesg-warn");
> +			add_or_override_result(current_test, "fail", "dmesg-fail");
> +			add_or_override_result(current_test, "warn", "dmesg-warn");
> +		}
> +	} else {
> +		/*
> +		 * Didn't get any subtest messages at all. If there
> +		 * are subtests, add all of the dmesg gotten to all of
> +		 * them.
> +		 */
> +		for (i = 0; i < subtests->size; i++) {
> +			igt_name = gen_igt_name(binary, subtests->names[i]);
> +			current_test = get_or_create_json_object(tests, igt_name);
> +			json_object_object_add(current_test, "dmesg",
> +					       json_object_new_string_len(dmesg, dmesglen));
> +			/*
> +			 * Don't bother with warnings, any subtests
> +			 * there are would have skip as their result
> +			 * anyway.
> +			 */
> +		}
> +
> +		if (i == 0) {
> +			/* There were no subtests */
> +			igt_name = gen_igt_name(binary, NULL);
> +			current_test = get_or_create_json_object(tests, igt_name);
> +			json_object_object_add(current_test, "dmesg",
> +					       json_object_new_string_len(dmesg, dmesglen));
> +			if (warnings) {
> +				json_object_object_add(current_test, "dmesg-warnings",
> +						       json_object_new_string_len(warnings, warningslen));
> +				add_or_override_result(current_test, "pass", "dmesg-warn");
> +				add_or_override_result(current_test, "fail", "dmesg-fail");
> +				add_or_override_result(current_test, "warn", "dmesg-warn");
> +			}
> +		}
> +	}
> +
> +	/*
> +	 * Add an empty string as the dmesg of all subtests that
> +	 * didn't get any dmesg yet.
> +	 */
> +	for (i = 0; i < subtests->size; i++) {
> +		igt_name = gen_igt_name(binary, subtests->names[i]);
> +		current_test = get_or_create_json_object(tests, igt_name);
> +		if (!json_object_object_get_ex(current_test, "dmesg", NULL)) {
> +			json_object_object_add(current_test, "dmesg",
> +					       json_object_new_string(""));
> +		}
> +	}
> +
> +	free(dmesg);
> +	free(warnings);
> +	fclose(f);
> +	return true;
> +}
> +
> +static char *result_from_exitcode(int exitcode)
> +{
> +	switch (exitcode) {
> +	case IGT_EXIT_TIMEOUT:
> +		return "timeout";
> +	case IGT_EXIT_SKIP:
> +		return "skip";
> +	case IGT_EXIT_SUCCESS:
> +		return "pass";
> +	case IGT_EXIT_INVALID:
> +		return "notrun";
> +	default:
> +		return "fail";
> +	}
> +}
> +
> +static void add_subtest(struct subtests *subtests, char *subtest)
> +{
> +	size_t len = strlen(subtest);
> +
> +	if (len == 0)
> +		return;
> +
> +	if (subtest[len - 1] == '\n')
> +		subtest[len - 1] = '\0';
> +
> +	subtests->size++;
> +	subtests->names = realloc(subtests->names, sizeof(*subtests->names) * subtests->size);
> +	subtests->names[subtests->size - 1] = subtest;
> +}
> +
> +static bool fill_from_journal(int fd, char *binary,
> +				  struct subtests *subtests,
> +				  struct json_object *tests)
> +{
> +	FILE *f = fdopen(fd, "r");
> +	char *line = NULL;
> +	size_t linelen = 0;
> +	ssize_t read;
> +	char exitline[] = "exit:";
> +	size_t exitlen = sizeof(exitline) - 1;
> +	char timeoutline[] = "timeout:";
> +	size_t timeoutlen = sizeof(timeoutline) - 1;
> +	int exitcode = 10000;
> +
> +	while ((read = getline(&line, &linelen, f)) > 0) {
> +		if (read >= exitlen && !memcmp(line, exitline, exitlen)) {
> +			char *p = strchr(line, '(');
> +			char *igt_name = gen_igt_name(binary, NULL);
> +			double time = 0.0;
> +			struct json_object *obj = get_or_create_json_object(tests, igt_name);
> +
> +			exitcode = atoi(line + exitlen);
> +
> +			if (p) {
> +				time = strtod(p + 1, NULL);
> +			}
> +
> +			add_runtime(obj, time);
> +		} else if (read >= timeoutlen && !memcmp(line, timeoutline, timeoutlen)) {
> +			if (subtests->size) {
> +				char *last_subtest = subtests->names[subtests->size - 1];
> +				char *igt_name = gen_igt_name(binary, last_subtest);
> +				char *p = strchr(line, '(');
> +				double time = 0.0;
> +				struct json_object *obj = get_or_create_json_object(tests, igt_name);
> +
> +				json_object_object_add(obj, "result",
> +						       json_object_new_string("timeout"));
> +
> +				if (p) {
> +					time = strtod(p + 1, NULL);
> +				}
> +
> +				add_runtime(obj, time);
> +
> +				igt_name = gen_igt_name(binary, NULL);
> +				obj = get_or_create_json_object(tests, igt_name);
> +				add_runtime(obj, time);
> +			}
> +		} else {
> +			add_subtest(subtests, strdup(line));
> +		}
> +	}
> +
> +	if (subtests->size == 0 && exitcode != 10000) {
> +		char *igt_name = gen_igt_name(binary, NULL);
> +		struct json_object *obj = get_or_create_json_object(tests, igt_name);
> +		char *result = result_from_exitcode(exitcode);
> +		json_object_object_add(obj, "result",
> +				       json_object_new_string(result));
> +	}
> +
> +	free(line);
> +	return true;
> +}
> +
> +static bool parse_test_directory(int dirfd, char *binary, struct json_object *tests)
> +{
> +	int fds[_F_LAST];
> +	struct subtests subtests = {};
> +
> +	if (!open_output_files(dirfd, fds, false)) {
> +		fprintf(stderr, "Error opening output files\n");
> +		return false;
> +	}
> +
> +	/* fill_from_journal also fills the subtests struct */
> +	if (!fill_from_journal(fds[_F_JOURNAL], binary, &subtests, tests)) {
> +		fprintf(stderr, "Error reading from journal\n");
> +		return false;
> +	}
> +
> +	/* Order of these is important */
> +	if (!fill_from_output(fds[_F_OUT], binary, true, &subtests, tests) ||
> +	    !fill_from_output(fds[_F_ERR], binary, false, &subtests, tests) ||
> +	    !fill_from_dmesg(fds[_F_DMESG], binary, &subtests, tests)) {
> +		fprintf(stderr, "Error parsing output files\n");
> +		return false;
> +	}
> +
> +	close_outputs(fds);
> +
> +	return true;
> +}
> +
> +bool generate_results(int dirfd)
> +{
> +	struct settings settings;
> +	struct job_list job_list;
> +	struct json_object *obj, *tests;
> +	int resultsfd, testdirfd, unamefd;
> +	const char *json_string;
> +	size_t i;
> +
> +	init_settings(&settings);
> +	init_job_list(&job_list);
> +
> +	if (!read_settings(&settings, dirfd)) {
> +		fprintf(stderr, "resultgen: Cannot parse settings\n");
> +		return false;
> +	}
> +
> +	if (!read_job_list(&job_list, dirfd)) {
> +		fprintf(stderr, "resultgen: Cannot parse job list\n");
> +		return false;
> +	}
> +
> +	/* TODO: settings.overwrite */
> +	if ((resultsfd = openat(dirfd, "results.json", O_WRONLY | O_CREAT | O_TRUNC, 0666)) < 0) {
> +		fprintf(stderr, "resultgen: Cannot create results file\n");
> +		return false;
> +	}
> +
> +	obj = json_object_new_object();
> +	json_object_object_add(obj, "__type__", json_object_new_string("TestrunResult"));
> +	json_object_object_add(obj, "results_version", json_object_new_int(9));
> +	json_object_object_add(obj, "name",
> +			       settings.name ?
> +			       json_object_new_string(settings.name) :
> +			       json_object_new_string(""));
> +
> +	if ((unamefd = openat(dirfd, "uname.txt", O_RDONLY)) >= 0) {
> +		char buf[128];
> +		ssize_t r = read(unamefd, buf, 128);
> +
> +		if (r > 0 && buf[r - 1] == '\n')
> +			r--;
> +
> +		json_object_object_add(obj, "uname",
> +				       json_object_new_string_len(buf, r));
> +		close(unamefd);
> +	}
> +
> +	/* lspci */
> +	/* results_version */
> +	/* glxinfo */
> +	/* wglinfo */
> +	/* clinfo */
> +	/* options */
> +	/* time_elapsed */
> +	/* totals */
> +
> +	tests = json_object_new_object();
> +	json_object_object_add(obj, "tests", tests);
> +
> +	for (i = 0; i < job_list.size; i++) {
> +		char name[16];
> +
> +		snprintf(name, 16, "%zd", i);
> +		if ((testdirfd = openat(dirfd, name, O_DIRECTORY | O_RDONLY)) < 0) {
> +			fprintf(stderr, "Warning: Cannot open result directory %s\n", name);
> +			break;
> +		}
> +
> +		if (!parse_test_directory(testdirfd, job_list.entries[i].binary, tests)) {
> +			close(resultsfd);
> +			return false;
> +		}
> +	}
> +
> +	json_string = json_object_to_json_string_ext(obj, JSON_C_TO_STRING_PRETTY);
> +	write(resultsfd, json_string, strlen(json_string));
> +	return true;
> +}
> +
> +bool generate_results_path(char *resultspath)
> +{
> +	int dirfd = open(resultspath, O_DIRECTORY | O_RDONLY);
> +
> +	if (dirfd < 0)
> +		return false;
> +
> +	return generate_results(dirfd);
> +}
> diff --git a/runner/resultgen.h b/runner/resultgen.h
> new file mode 100644
> index 00000000..83a0876b
> --- /dev/null
> +++ b/runner/resultgen.h
> @@ -0,0 +1,9 @@
> +#ifndef RUNNER_RESULTGEN_H
> +#define RUNNER_RESULTGEN_H
> +
> +#include <stdbool.h>
> +
> +bool generate_results(int dirfd);
> +bool generate_results_path(char *resultspath);
> +
> +#endif
> diff --git a/runner/results.c b/runner/results.c
> new file mode 100644
> index 00000000..3eb7cb15
> --- /dev/null
> +++ b/runner/results.c
> @@ -0,0 +1,26 @@
> +#include <fcntl.h>
> +#include <stdio.h>
> +#include <stdlib.h>
> +#include <sys/stat.h>
> +#include <sys/types.h>
> +
> +#include "resultgen.h"
> +
> +int main(int argc, char **argv)
> +{
> +	int dirfd;
> +
> +	if (argc < 2)
> +		exit(1);
> +
> +	dirfd = open(argv[1], O_DIRECTORY | O_RDONLY);
> +	if (dirfd < 0)
> +		exit(1);
> +
> +	if (generate_results(dirfd)) {
> +		printf("Results generated\n");
> +		exit(0);
> +	}
> +
> +	exit(1);
> +}
> diff --git a/runner/runner.c b/runner/runner.c
> new file mode 100644
> index 00000000..b685786a
> --- /dev/null
> +++ b/runner/runner.c
> @@ -0,0 +1,40 @@
> +#include <stdio.h>
> +#include <string.h>
> +
> +#include "settings.h"
> +#include "job_list.h"
> +#include "executor.h"
> +#include "resultgen.h"
> +
> +int main(int argc, char **argv)
> +{
> +	struct settings settings;
> +	struct job_list job_list;
> +	struct execute_state state;
> +
> +	init_settings(&settings);
> +	init_job_list(&job_list);
> +
> +	if (!parse_options(argc, argv, &settings)) {
> +		return 1;
> +	}
> +
> +	if (!create_job_list(&job_list, &settings)) {
> +		return 1;
> +	}
> +
> +	if (!initialize_execute_state(&state, &settings, &job_list)) {
> +		return 1;
> +	}
> +
> +	if (!execute(&state, &settings, &job_list)) {
> +		return 1;
> +	}
> +
> +	if (!generate_results_path(settings.results_path)) {
> +		return 1;
> +	}
> +
> +	printf("Done.\n");
> +	return 0;
> +}
> diff --git a/runner/settings.c b/runner/settings.c
> new file mode 100644
> index 00000000..50e4ea10
> --- /dev/null
> +++ b/runner/settings.c
> @@ -0,0 +1,506 @@
> +#include "settings.h"
> +
> +#include <fcntl.h>
> +#include <getopt.h>
> +#include <libgen.h>
> +#include <limits.h>
> +#include <stdio.h>
> +#include <stdlib.h>
> +#include <string.h>
> +#include <sys/types.h>
> +#include <sys/stat.h>
> +#include <unistd.h>
> +
> +enum {
> +	OPT_ABORT_ON_ERROR,
> +	OPT_TEST_LIST,
> +	OPT_IGNORE_MISSING,
> +	OPT_HELP = 'h',
> +	OPT_NAME = 'n',
> +	OPT_DRY_RUN = 'd',
> +	OPT_INCLUDE = 't',
> +	OPT_EXCLUDE = 'x',
> +	OPT_SYNC = 's',
> +	OPT_LOG_LEVEL = 'l',
> +	OPT_OVERWRITE = 'o',
> +	OPT_MULTIPLE = 'm',
> +	OPT_TIMEOUT = 'c',
> +	OPT_WATCHDOG = 'g',
> +};
> +
> +static struct {
> +	int level;
> +	const char *name;
> +} log_levels[] = {
> +	{ LOG_LEVEL_NORMAL, "normal" },
> +	{ LOG_LEVEL_QUIET, "quiet" },
> +	{ LOG_LEVEL_VERBOSE, "verbose" },
> +	{ 0, 0 },
> +};
> +
> +static bool set_log_level(struct settings* settings, const char *level)
> +{
> +	int c;
> +
> +	for (c = 0; log_levels[c].name != NULL; c++) {
> +		if (!strcmp(level, log_levels[c].name)) {
> +			settings->log_level = log_levels[c].level;
> +			return true;
> +		}
> +	}
> +
> +	return false;
> +}
> +
> +static const char *usage_str =
> +	"usage: runner [options] [test_root] results-path\n\n"
> +	"Options:\n"
> +	" Piglit compatible:\n"
> +	"  -h, --help            Show this help message and exit\n"
> +	"  -n <test name>, --name <test name>\n"
> +	"                        Name of this test run\n"
> +	"  -d, --dry-run         Do not execute the tests\n"
> +	"  -t <regex>, --include-tests <regex>\n"
> +	"                        Run only matching tests (can be used more than once)\n"
> +	"  -x <regex>, --exclude-tests <regex>\n"
> +	"                        Exclude matching tests (can be used more than once)\n"
> +	"  --abort-on-monitored-error\n"
> +	"                        Abort execution when a fatal condition is detected.\n"
> +	"                        <TODO>\n"
> +	"  -s, --sync            Sync results to disk after every test\n"
> +	"  -l {quiet,verbose,dummy}, --log-level {quiet,verbose,dummy}\n"
> +	"                        Set the logger verbosity level\n"
> +	"  --test-list TEST_LIST\n"
> +	"                        A file containing a list of tests to run\n"
> +	"  -o, --overwrite       If the results-path already exists, delete it\n"
> +	"  --ignore-missing      Ignored but accepted, for piglit compatibility\n"
> +	"\n"
> +	" Incompatible options:\n"
> +	"  -m, --multiple-mode   Run multiple subtests in the same binary execution.\n"
> +	"                        If a testlist file is given, consecutive subtests are\n"
> +	"                        run in the same execution if they are from the same\n"
> +	"                        binary. Note that in that case relative ordering of the\n"
> +	"                        subtest execution is dictated by the test binary, not\n"
> +	"                        the testlist\n"
> +	"  --inactivity-timeout <seconds>\n"
> +	"                        Kill the running test after <seconds> of inactivity in\n"
> +	"                        the test's stdout, stderr, or dmesg\n"
> +	"  --use-watchdog        Use hardware watchdog for lethal enforcement of the\n"
> +	"                        above timeout. Killing the test process is still\n"
> +	"                        attempted at timeout trigger.\n"
> +	"  [test_root]           Directory that contains the IGT tests. The environment\n"
> +	"                        variable IGT_TEST_ROOT will be used if set, overriding\n"
> +	"                        this option if given.\n"
> +	;
> +
> +static void usage(const char *extra_message, bool use_stderr)
> +{
> +	FILE *f = use_stderr ? stderr : stdout;
> +
> +	if (extra_message)
> +		fprintf(f, "%s\n\n", extra_message);
> +
> +	fprintf(f, "%s", usage_str);
> +}
> +
> +static bool add_regex(struct regex_list *list, char *new)
> +{
> +	regex_t *regex;
> +	size_t buflen;
> +	char *buf;
> +	int s;
> +
> +	regex = malloc(sizeof(*regex));
> +
> +	if ((s = regcomp(regex, new,
> +			 REG_EXTENDED | REG_NOSUB)) != 0) {
> +		buflen = regerror(s, regex, NULL, 0);
> +		buf = malloc(buflen);
> +		regerror(s, regex, buf, buflen);
> +		usage(buf, true);
> +
> +		free(buf);
> +		regfree(regex);
> +		free(regex);
> +		return false;
> +	}
> +
> +	list->regexes = realloc(list->regexes,
> +				(list->size + 1) * sizeof(*list->regexes));
> +	list->regex_strings = realloc(list->regex_strings,
> +				      (list->size + 1) * sizeof(*list->regex_strings));
> +	list->regexes[list->size] = regex;
> +	list->regex_strings[list->size] = new;
> +	list->size++;
> +
> +	return true;
> +}
> +
> +static void free_regexes(struct regex_list *regexes)
> +{
> +	size_t i;
> +
> +	for (i = 0; i < regexes->size; i++) {
> +		free(regexes->regex_strings[i]);
> +		regfree(regexes->regexes[i]);
> +		free(regexes->regexes[i]);
> +	}
> +	free(regexes->regex_strings);
> +	free(regexes->regexes);
> +}
> +
> +static bool readable_file(char *filename)
> +{
> +	return !access(filename, R_OK);
> +}
> +
> +void init_settings(struct settings *settings)
> +{
> +	memset(settings, 0, sizeof(*settings));
> +}
> +
> +void free_settings(struct settings *settings)
> +{
> +	free(settings->test_list);
> +	free(settings->name);
> +	free(settings->test_root);
> +	free(settings->results_path);
> +
> +	free_regexes(&settings->include_regexes);
> +	free_regexes(&settings->exclude_regexes);
> +
> +	init_settings(settings);
> +}
> +
> +bool parse_options(int argc, char **argv,
> +		   struct settings *settings)
> +{
> +	int c;
> +	char *env_test_root;
> +
> +	static struct option long_options[] = {
> +		{"help", no_argument, NULL, OPT_HELP},
> +		{"name", required_argument, NULL, OPT_NAME},
> +		{"dry-run", no_argument, NULL, OPT_DRY_RUN},
> +		{"include-tests", required_argument, NULL, OPT_INCLUDE},
> +		{"exclude-tests", required_argument, NULL, OPT_EXCLUDE},
> +		{"abort-on-monitored-error", no_argument, NULL, OPT_ABORT_ON_ERROR},
> +		{"sync", no_argument, NULL, OPT_SYNC},
> +		{"log-level", required_argument, NULL, OPT_LOG_LEVEL},
> +		{"test-list", required_argument, NULL, OPT_TEST_LIST},
> +		{"overwrite", no_argument, NULL, OPT_OVERWRITE},
> +		{"ignore-missing", no_argument, NULL, OPT_IGNORE_MISSING},
> +		{"multiple-mode", no_argument, NULL, OPT_MULTIPLE},
> +		{"inactivity-timeout", required_argument, NULL, OPT_TIMEOUT},
> +		{"use-watchdog", no_argument, NULL, OPT_WATCHDOG},
> +		{ 0, 0, 0, 0},
> +	};
> +
> +	free_settings(settings);
> +
> +	optind = 1;
> +
> +	while ((c = getopt_long(argc, argv, "hn:dt:x:sl:om", long_options, NULL)) != -1) {
> +		switch (c) {
> +		case OPT_HELP:
> +			usage(NULL, false);
> +			goto error;
> +		case OPT_NAME:
> +			settings->name = strdup(optarg);
> +			break;
> +		case OPT_DRY_RUN:
> +			settings->dry_run = true;
> +			break;
> +		case OPT_INCLUDE:
> +			if (!add_regex(&settings->include_regexes, strdup(optarg)))
> +				goto error;
> +			break;
> +		case OPT_EXCLUDE:
> +			if (!add_regex(&settings->exclude_regexes, strdup(optarg)))
> +				goto error;
> +			break;
> +		case OPT_ABORT_ON_ERROR:
> +			settings->abort_on_error = true;
> +			break;
> +		case OPT_SYNC:
> +			settings->sync = true;
> +			break;
> +		case OPT_LOG_LEVEL:
> +			if (!set_log_level(settings, optarg)) {
> +				usage("Cannot parse log level", true);
> +				goto error;
> +			}
> +			break;
> +		case OPT_TEST_LIST:
> +			settings->test_list = absolute_path(optarg);
> +			break;
> +		case OPT_OVERWRITE:
> +			settings->overwrite = true;
> +			break;
> +		case OPT_IGNORE_MISSING:
> +			/* Ignored, piglit compatibility */
> +			break;
> +		case OPT_MULTIPLE:
> +			settings->multiple_mode = true;
> +			break;
> +		case OPT_TIMEOUT:
> +			settings->inactivity_timeout = atoi(optarg);
> +			break;
> +		case OPT_WATCHDOG:
> +			settings->use_watchdog = true;
> +			break;
> +		case '?':
> +			usage(NULL, true);
> +			goto error;
> +		default:
> +			usage("Cannot parse options", true);
> +			goto error;
> +		}
> +	}
> +
> +	switch (argc - optind) {
> +	case 2:
> +		settings->test_root = absolute_path(argv[optind]);
> +		++optind;
> +		/* fallthrough */
> +	case 1:
> +		settings->results_path = absolute_path(argv[optind]);
> +		break;
> +	case 0:
> +		usage("Results-path missing", true);
> +		goto error;
> +	default:
> +		usage("Extra arguments after results-path", true);
> +		goto error;
> +	}
> +
> +	if ((env_test_root = getenv("IGT_TEST_ROOT")) != NULL) {
> +		free(settings->test_root);
> +		settings->test_root = absolute_path(env_test_root);
> +	}
> +
> +	if (!settings->test_root) {
> +		usage("Test root not set", true);
> +		goto error;
> +	}
> +
> +	if (!settings->name) {
> +		char *name = strdup(settings->results_path);
> +		settings->name = strdup(basename(name));
> +		free(name);
> +	}
> +
> +	return true;
> +
> + error:
> +	free_settings(settings);
> +	return false;
> +}
> +
> +bool validate_settings(struct settings *settings)
> +{
> +	int dirfd, fd;
> +
> +	if (settings->test_list && !readable_file(settings->test_list)) {
> +		usage("Cannot open test-list file", true);
> +		return false;
> +	}
> +
> +	if (!settings->results_path) {
> +		usage("No results-path set; this shouldn't happen", true);
> +		return false;
> +	}
> +
> +	if (!settings->test_root) {
> +		usage("No test root set; this shouldn't happen", true);
> +		return false;
> +	}
> +
> +	dirfd = open(settings->test_root, O_DIRECTORY | O_RDONLY);
> +	if (dirfd < 0) {
> +		fprintf(stderr, "Test directory %s cannot be opened\n", settings->test_root);
> +		return false;
> +	}
> +
> +	fd = openat(dirfd, "test-list.txt", O_RDONLY);
> +	if (fd < 0) {
> +		fprintf(stderr, "Cannot open %s/test-list.txt\n", settings->test_root);
> +		close(dirfd);
> +		return false;
> +	}
> +
> +	close(fd);
> +	close(dirfd);
> +
> +	return true;
> +}
> +
> +char *absolute_path(char *path)
> +{
> +	char *result = NULL;
> +	char *tmppath, *tmpname;
> +
> +	result = realpath(path, NULL);
> +	if (result != NULL)
> +		return result;
> +
> +	tmppath = strdup(path);
> +	tmpname = dirname(tmppath);
> +	free(result);
> +	result = realpath(tmpname, NULL);
> +	free(tmppath);
> +
> +	if (result != NULL) {
> +		tmppath = strdup(path);
> +		tmpname = basename(tmppath);
> +		strcat(result, "/");
> +		strcat(result, tmpname);
> +		free(tmppath);
> +		return result;
> +	}
> +
> +	free(result);
> +	return NULL;
> +}
> +
> +static char settings_filename[] = "metadata.txt";
> +bool serialize_settings(struct settings *settings)
> +{
> +#define SERIALIZE_LINE(f, s, name, format) fprintf(f, "%s : " format "\n", #name, s->name)
> +
> +	int dirfd, fd;
> +	FILE *f;
> +
> +	if (!settings->results_path) {
> +		usage("No results-path set; this shouldn't happen", true);
> +		return false;
> +	}
> +
> +	if ((dirfd = open(settings->results_path, O_DIRECTORY | O_RDONLY)) < 0) {
> +		mkdir(settings->results_path, 0755);
> +		if ((dirfd = open(settings->results_path, O_DIRECTORY | O_RDONLY)) < 0) {
> +			usage("Creating results-path failed", true);
> +			return false;
> +		}
> +	}
> +
> +	if ((fd = openat(dirfd, settings_filename, O_RDONLY)) >= 0) {
> +		close(fd);
> +
> +		if (!settings->overwrite) {
> +			/* Serialization data already exists, not overwriting */
> +			close(dirfd);
> +			return false;
> +		}
> +
> +		if (unlinkat(dirfd, settings_filename, 0) != 0) {
> +			usage("Error overwriting old settings metadata", true);
> +			close(dirfd);
> +			return false;
> +		}
> +	}
> +
> +	if ((fd = openat(dirfd, settings_filename, O_CREAT | O_EXCL | O_WRONLY, 0666)) < 0) {
> +		usage("Creating settings serialization file failed", true);
> +		close(dirfd);
> +		return false;
> +	}
> +
> +	f = fdopen(fd, "w");
> +	if (!f) {
> +		close(fd);
> +		close(dirfd);
> +		return false;
> +	}
> +
> +	SERIALIZE_LINE(f, settings, abort_on_error, "%d");
> +	if (settings->test_list)
> +		SERIALIZE_LINE(f, settings, test_list, "%s");
> +	if (settings->name)
> +		SERIALIZE_LINE(f, settings, name, "%s");
> +	SERIALIZE_LINE(f, settings, dry_run, "%d");
> +	SERIALIZE_LINE(f, settings, sync, "%d");
> +	SERIALIZE_LINE(f, settings, log_level, "%d");
> +	SERIALIZE_LINE(f, settings, overwrite, "%d");
> +	SERIALIZE_LINE(f, settings, multiple_mode, "%d");
> +	SERIALIZE_LINE(f, settings, inactivity_timeout, "%d");
> +	SERIALIZE_LINE(f, settings, use_watchdog, "%d");
> +	SERIALIZE_LINE(f, settings, test_root, "%s");
> +	SERIALIZE_LINE(f, settings, results_path, "%s");
> +
> +	if (settings->sync) {
> +		fsync(fd);
> +		fsync(dirfd);
> +	}
> +
> +	fclose(f);
> +	close(dirfd);
> +	return true;
> +
> +#undef SERIALIZE_LINE
> +}
> +
> +static char *maybe_strdup(char *str)
> +{
> +	if (!str)
> +		return NULL;
> +
> +	return strdup(str);
> +}
> +
> +bool read_settings(struct settings *settings, int dirfd)
> +{
> +#define PARSE_LINE(s, name, val, field, write) \
> +	if (!strcmp(name, #field)) {	       \
> +		s->field = write;	       \
> +		free(name);		       \
> +		free(val);		       \
> +		name = val = NULL;	       \
> +		continue;		       \
> +	}
> +
> +	int fd;
> +	FILE *f;
> +	char *name = NULL, *val = NULL;
> +
> +	free_settings(settings);
> +
> +	if ((fd = openat(dirfd, settings_filename, O_RDONLY)) < 0)
> +		return false;
> +
> +	f = fdopen(fd, "r");
> +	if (!f) {
> +		close(fd);
> +		return false;
> +	}
> +
> +	while (fscanf(f, "%ms : %ms", &name, &val) == 2) {
> +		int numval = atoi(val);
> +		PARSE_LINE(settings, name, val, abort_on_error, numval);
> +		PARSE_LINE(settings, name, val, test_list, maybe_strdup(val));
> +		PARSE_LINE(settings, name, val, name, maybe_strdup(val));
> +		PARSE_LINE(settings, name, val, dry_run, numval);
> +		PARSE_LINE(settings, name, val, sync, numval);
> +		PARSE_LINE(settings, name, val, log_level, numval);
> +		PARSE_LINE(settings, name, val, overwrite, numval);
> +		PARSE_LINE(settings, name, val, multiple_mode, numval);
> +		PARSE_LINE(settings, name, val, inactivity_timeout, numval);
> +		PARSE_LINE(settings, name, val, use_watchdog, numval);
> +		PARSE_LINE(settings, name, val, test_root, maybe_strdup(val));
> +		PARSE_LINE(settings, name, val, results_path, maybe_strdup(val));
> +
> +		printf("Warning: Unknown field in settings file: %s = %s\n",
> +		       name, val);
> +		free(name);
> +		free(val);
> +		name = val = NULL;
> +	}
> +
> +	free(name);
> +	free(val);
> +	fclose(f);
> +
> +	return true;
> +
> +#undef PARSE_LINE
> +}
> diff --git a/runner/settings.h b/runner/settings.h
> new file mode 100644
> index 00000000..9d1f03fb
> --- /dev/null
> +++ b/runner/settings.h
> @@ -0,0 +1,111 @@
> +#ifndef RUNNER_SETTINGS_H
> +#define RUNNER_SETTINGS_H
> +
> +#include <stdbool.h>
> +#include <stddef.h>
> +#include <sys/types.h>
> +#include <regex.h>
> +
> +enum {
> +	LOG_LEVEL_NORMAL = 0,
> +	LOG_LEVEL_QUIET = -1,
> +	LOG_LEVEL_VERBOSE = 1,
> +};
> +
> +struct regex_list {
> +	char **regex_strings;
> +	regex_t** regexes;
> +	size_t size;
> +};
> +
> +struct settings {
> +	bool abort_on_error;
> +	char *test_list;
> +	char *name;
> +	bool dry_run;
> +	struct regex_list include_regexes;
> +	struct regex_list exclude_regexes;
> +	bool sync;
> +	int log_level;
> +	bool overwrite;
> +	bool multiple_mode;
> +	int inactivity_timeout;
> +	bool use_watchdog;
> +	char *test_root;
> +	char *results_path;
> +};
> +
> +/**
> + * init_settings:
> + *
> + * Initializes a settings object to an empty state (all values NULL, 0
> + * or false).
> + *
> + * @settings: Object to initialize. Storage for it must exist.
> + */
> +void init_settings(struct settings *settings);
> +
> +/**
> + * free_settings:
> + *
> + * Releases all allocated resources for a settings object and
> + * initializes it to an empty state (see #init_settings).
> + *
> + * @settings: Object to release and initialize.
> + */
> +void free_settings(struct settings *settings);
> +
> +/**
> + * parse_options:
> + *
> + * Parses command line options and sets the settings object to
> + * designated values.
> + *
> + * The function can be called again on the same settings object. The
> + * old values will be properly released and cleared. On a parse
> + * failure, the settings object will be in an empty state (see
> + * #init_settings) and usage instructions will be printed with an
> + * error message.
> + *
> + * @argc: Argument count
> + * @argv: Argument array. First element is the program name.
> + * @settings: Settings object to fill with values. Must have proper
> + * storage.
> + *
> + * Returns: True on successful parse, false on error.
> + */
> +bool parse_options(int argc, char **argv,
> +		   struct settings *settings);
> +
> +/**
> + * validate_settings:
> + *
> + * Checks the settings object against the system to see if executing
> + * on it can be done. Checks pathnames for existence and access
> + * rights. Note that this function will not check that the designated
> + * job listing (through a test-list file or the -t/-x flags) yields a
> + * non-zero amount of testing to be done. On errors, usage
> + * instructions will be printed with an error message.
> + *
> + * @settings: Settings object to check.
> + *
> + * Returns: True on valid settings, false on any error.
> + */
> +bool validate_settings(struct settings *settings);
> +
> +/* TODO: Better place for this */
> +char *absolute_path(char *path);
> +
> +/**
> + * serialize_settings:
> + *
> + * Serializes the settings object to a file in the results_path
> + * directory.
> + *
> + * @settings: Settings object to serialize.
> + */
> +bool serialize_settings(struct settings *settings);
> +
> +bool read_settings(struct settings *settings, int dirfd);
> +
> +#endif
> -- 
> 2.14.1
> 
> _______________________________________________
> igt-dev mailing list
> igt-dev@lists.freedesktop.org
> https://lists.freedesktop.org/mailman/listinfo/igt-dev

-- 
Daniel Vetter
Software Engineer, Intel Corporation
http://blog.ffwll.ch
_______________________________________________
igt-dev mailing list
igt-dev@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/igt-dev