[PATCH 13/20] ktest: Allow overriding bisect test results

[Steven Rostedt <rostedt@goodmis.org>]

[-- Attachment #1: Type: text/plain, Size: 5661 bytes --]

From: Steven Rostedt <srostedt@redhat.com>

When running the ktest git bisect test, if the BISECT_TYPE is "test",
the bisect is determined to be good or bad based off of the error
code of the test that is run. Currently, if the test returns 0,
it is considered a pass (good), a non-zero is considered a fail (bad).

But it has been requested to add more options, and also change
the meanings of the error codes of the test. For example, one may
want the test to detect if the commit is not good or bad,
(maybe the bisect came to a point where the code in question
does not exist). The test could report an error code that should tell
ktest to skip the commit.

Also, a test could detect that something is horribly wrong and the
biscet should just be aborted.

The new options:

 BISECT_RET_GOOD
 BISECT_RET_BAD
 BISECT_RET_SKIP
 BISECT_RET_ABORT
 BISECT_RET_DEFAULT

have been added. The first 4 take an integer value that will
represent if the test should be considered a pass, fail, neither
good nor bad, or abort respectively.

The BISECT_RET_DEFAULT will bo whatever is not defined by the
above codes. If only BISECT_RET_DEFAULT is defined, then all tests
will do the default.

Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
---
 tools/testing/ktest/ktest.pl    |   47 +++++++++++++++++++++++++++++++++++++++
 tools/testing/ktest/sample.conf |   36 +++++++++++++++++++++++++++++
 2 files changed, 83 insertions(+), 0 deletions(-)

diff --git a/tools/testing/ktest/ktest.pl b/tools/testing/ktest/ktest.pl
index 04a7bb5..47c2814 100755
--- a/tools/testing/ktest/ktest.pl
+++ b/tools/testing/ktest/ktest.pl
@@ -105,6 +105,11 @@ my $reverse_bisect;
 my $bisect_manual;
 my $bisect_skip;
 my $config_bisect_good;
+my $bisect_ret_good;
+my $bisect_ret_bad;
+my $bisect_ret_skip;
+my $bisect_ret_abort;
+my $bisect_ret_default;
 my $in_patchcheck = 0;
 my $run_test;
 my $redirect;
@@ -1854,6 +1859,43 @@ sub do_run_test {
     waitpid $child_pid, 0;
     $child_exit = $?;
 
+    if (!$bug && $in_bisect) {
+	if (defined($bisect_ret_good)) {
+	    if ($child_exit == $bisect_ret_good) {
+		return 1;
+	    }
+	}
+	if (defined($bisect_ret_skip)) {
+	    if ($child_exit == $bisect_ret_skip) {
+		return -1;
+	    }
+	}
+	if (defined($bisect_ret_abort)) {
+	    if ($child_exit == $bisect_ret_abort) {
+		fail "test abort" and return -2;
+	    }
+	}
+	if (defined($bisect_ret_bad)) {
+	    if ($child_exit == $bisect_ret_skip) {
+		return 0;
+	    }
+	}
+	if (defined($bisect_ret_default)) {
+	    if ($bisect_ret_default eq "good") {
+		return 1;
+	    } elsif ($bisect_ret_default eq "bad") {
+		return 0;
+	    } elsif ($bisect_ret_default eq "skip") {
+		return -1;
+	    } elsif ($bisect_ret_default eq "abort") {
+		return -2;
+	    } else {
+		fail "unknown default action: $bisect_ret_default"
+		    and return -2;
+	    }
+	}
+    }
+
     if ($bug || $child_exit) {
 	return 0 if $in_bisect;
 	fail "test failed" and return 0;
@@ -3284,6 +3326,11 @@ for (my $i = 1; $i <= $opt{"NUM_TESTS"}; $i++) {
     $bisect_manual = set_test_option("BISECT_MANUAL", $i);
     $bisect_skip = set_test_option("BISECT_SKIP", $i);
     $config_bisect_good = set_test_option("CONFIG_BISECT_GOOD", $i);
+    $bisect_ret_good = set_test_option("BISECT_RET_GOOD", $i);
+    $bisect_ret_bad = set_test_option("BISECT_RET_BAD", $i);
+    $bisect_ret_skip = set_test_option("BISECT_RET_SKIP", $i);
+    $bisect_ret_abort = set_test_option("BISECT_RET_ABORT", $i);
+    $bisect_ret_default = set_test_option("BISECT_RET_DEFAULT", $i);
     $store_failures = set_test_option("STORE_FAILURES", $i);
     $store_successes = set_test_option("STORE_SUCCESSES", $i);
     $test_name = set_test_option("TEST_NAME", $i);
diff --git a/tools/testing/ktest/sample.conf b/tools/testing/ktest/sample.conf
index 42e0eb9..2ff0f8c 100644
--- a/tools/testing/ktest/sample.conf
+++ b/tools/testing/ktest/sample.conf
@@ -868,6 +868,42 @@
 #   BISECT_BAD with BISECT_CHECK = good or
 #   BISECT_CHECK = bad, respectively.
 #
+# BISECT_RET_GOOD = 0 (optional, default undefined)
+#
+#   In case the specificed test returns something other than just
+#   0 for good, and non-zero for bad, you can override 0 being
+#   good by defining BISECT_RET_GOOD.
+#
+# BISECT_RET_BAD = 1 (optional, default undefined)
+#
+#   In case the specificed test returns something other than just
+#   0 for good, and non-zero for bad, you can override non-zero being
+#   bad by defining BISECT_RET_BAD.
+#
+# BISECT_RET_ABORT = 255 (optional, default undefined)
+#
+#   If you need to abort the bisect if the test discovers something
+#   that was wrong, you can define BISECT_RET_ABORT to be the error
+#   code returned by the test in order to abort the bisect.
+#
+# BISECT_RET_SKIP = 2 (optional, default undefined)
+#
+#   If the test detects that the current commit is neither good
+#   nor bad, but something else happened (another bug detected)
+#   you can specify BISECT_RET_SKIP to an error code that the
+#   test returns when it should skip the current commit.
+#
+# BISECT_RET_DEFAULT = good (optional, default undefined)
+#
+#   You can override the default of what to do when the above
+#   options are not hit. This may be one of, "good", "bad",
+#   "abort" or "skip" (without the quotes).
+#
+#   Note, if you do not define any of the previous BISECT_RET_*
+#   and define BISECT_RET_DEFAULT, all bisects results will do
+#   what the BISECT_RET_DEFAULT has.
+#
+#
 # Example:
 #   TEST_START
 #   TEST_TYPE = bisect
-- 
1.7.7.3



[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 836 bytes --]