[PATCH 0/6] ktest: Updates for ktest.pl

[PATCH 0/6] ktest: Updates for ktest.pl

[Steven Rostedt]

Linus,

Please pull the following patches from:

  git://git.kernel.org/pub/scm/linux/kernel/git/rostedt/linux-2.6-ktest.git

    branch: for-linus


Steven Rostedt (6):
      ktest: Fix post install command
      ktest: If test failed due to timeout, print that
      ktest: Reboot to good kernel after every bisect run
      ktest: Reboot after each patchcheck run
      ktest: Create variables for the ktest config files
      ktest: Allow options to be used by other options

----
 tools/testing/ktest/ktest.pl    |  156 +++++++++++++++++++++++++++++++++++++--
 tools/testing/ktest/sample.conf |   93 +++++++++++++++++++++++
 2 files changed, 242 insertions(+), 7 deletions(-)

[PATCH 1/6] ktest: Fix post install command

[Steven Rostedt]

[-- Attachment #1: 0001-ktest-Fix-post-install-command.patch --]
[-- Type: text/plain, Size: 880 bytes --]

From: Steven Rostedt <srostedt@redhat.com>

The command to run post install (for those that want initrds) was
broken. Instead of doing a substitution for the $KERNEL_VERSION
variable. It was replacing the entire command with nothing.

Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
---
 tools/testing/ktest/ktest.pl |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)

diff --git a/tools/testing/ktest/ktest.pl b/tools/testing/ktest/ktest.pl
index 8ce792e..7301d03 100755
--- a/tools/testing/ktest/ktest.pl
+++ b/tools/testing/ktest/ktest.pl
@@ -907,7 +907,7 @@ sub install {
     return if (!defined($post_install));
 
     my $cp_post_install = $post_install;
-    $cp_post_install = s/\$KERNEL_VERSION/$version/g;
+    $cp_post_install =~ s/\$KERNEL_VERSION/$version/g;
     run_command "$cp_post_install" or
 	dodie "Failed to run post install";
 }
-- 
1.7.4.4

[PATCH 2/6] ktest: If test failed due to timeout, print that

[Steven Rostedt]

[-- Attachment #1: 0002-ktest-If-test-failed-due-to-timeout-print-that.patch --]
[-- Type: text/plain, Size: 752 bytes --]

From: Steven Rostedt <srostedt@redhat.com>

If the test failed due to timeout for boot, print a message saying
so. Otherwise the user will be confused to why their test just failed.

Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
---
 tools/testing/ktest/ktest.pl |    1 +
 1 files changed, 1 insertions(+), 0 deletions(-)

diff --git a/tools/testing/ktest/ktest.pl b/tools/testing/ktest/ktest.pl
index 7301d03..0603e64 100755
--- a/tools/testing/ktest/ktest.pl
+++ b/tools/testing/ktest/ktest.pl
@@ -838,6 +838,7 @@ sub monitor {
 
 	if ($stop_test_after > 0 && !$booted && !$bug) {
 	    if (time - $monitor_start > $stop_test_after) {
+		doprint "STOP_TEST_AFTER ($stop_test_after seconds) timed out\n";
 		$done = 1;
 	    }
 	}
-- 
1.7.4.4

[PATCH 3/6] ktest: Reboot to good kernel after every bisect run

[Steven Rostedt]

[-- Attachment #1: 0003-ktest-Reboot-to-good-kernel-after-every-bisect-run.patch --]
[-- Type: text/plain, Size: 972 bytes --]

From: Steven Rostedt <srostedt@redhat.com>

Reboot after each bisect run regardless if the bisect passed
or failed. The test may just be to boot the kernel and that kernel
may not have a way to copy the next kerne to it. Reboot to a known
good kernel after each bisect run.

Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
---
 tools/testing/ktest/ktest.pl |   10 +++++-----
 1 files changed, 5 insertions(+), 5 deletions(-)

diff --git a/tools/testing/ktest/ktest.pl b/tools/testing/ktest/ktest.pl
index 0603e64..2011222 100755
--- a/tools/testing/ktest/ktest.pl
+++ b/tools/testing/ktest/ktest.pl
@@ -1248,14 +1248,14 @@ sub run_bisect_test {
 
     if ($failed) {
 	$result = 0;
-
-	# reboot the box to a good kernel
-	if ($type ne "build") {
-	    bisect_reboot;
-	}
     } else {
 	$result = 1;
     }
+
+    # reboot the box to a kernel we can ssh to
+    if ($type ne "build") {
+	bisect_reboot;
+    }
     $in_bisect = 0;
 
     return $result;
-- 
1.7.4.4

Re: [PATCH 3/6] ktest: Reboot to good kernel after every bisect run

[John Kacur]

On Mon, May 23, 2011 at 3:51 PM, Steven Rostedt <rostedt@goodmis.org> wrote:
> From: Steven Rostedt <srostedt@redhat.com>
>
> Reboot after each bisect run regardless if the bisect passed
> or failed. The test may just be to boot the kernel and that kernel
> may not have a way to copy the next kerne to it. Reboot to a known
> good kernel after each bisect run.
>

How about an option to not reboot for good kernels?

Re: [PATCH 3/6] ktest: Reboot to good kernel after every bisect run

[Steven Rostedt]

On Mon, 2011-05-23 at 17:04 +0200, John Kacur wrote:
> On Mon, May 23, 2011 at 3:51 PM, Steven Rostedt <rostedt@goodmis.org> wrote:
> > From: Steven Rostedt <srostedt@redhat.com>
> >
> > Reboot after each bisect run regardless if the bisect passed
> > or failed. The test may just be to boot the kernel and that kernel
> > may not have a way to copy the next kerne to it. Reboot to a known
> > good kernel after each bisect run.
> >
> 
> How about an option to not reboot for good kernels?

Would be trivial to add. But that would be for 2.6.41.

Thanks,

-- Steve

[PATCH 4/6] ktest: Reboot after each patchcheck run

[Steven Rostedt]

[-- Attachment #1: 0004-ktest-Reboot-after-each-patchcheck-run.patch --]
[-- Type: text/plain, Size: 2674 bytes --]

From: Steven Rostedt <srostedt@redhat.com>

The patches being checked may not leave the kernel in a state
that the next run will allow the new kernel to be copied to the
machine. Reboot to a known good kernel before continuing to the
next kernel to test.

Added option PATCHCHECK_SLEEP_TIME for the max time to sleep between
patchcheck reboots.

Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
---
 tools/testing/ktest/ktest.pl    |   13 +++++++++++++
 tools/testing/ktest/sample.conf |    4 ++++
 2 files changed, 17 insertions(+), 0 deletions(-)

diff --git a/tools/testing/ktest/ktest.pl b/tools/testing/ktest/ktest.pl
index 2011222..65003a1 100755
--- a/tools/testing/ktest/ktest.pl
+++ b/tools/testing/ktest/ktest.pl
@@ -36,6 +36,7 @@ $default{"REBOOT_ON_SUCCESS"}	= 1;
 $default{"POWEROFF_ON_SUCCESS"}	= 0;
 $default{"BUILD_OPTIONS"}	= "";
 $default{"BISECT_SLEEP_TIME"}	= 60;   # sleep time between bisects
+$default{"PATCHCHECK_SLEEP_TIME"} = 60; # sleep time between patch checks
 $default{"CLEAR_LOG"}		= 0;
 $default{"BISECT_MANUAL"}	= 0;
 $default{"BISECT_SKIP"}		= 1;
@@ -96,6 +97,7 @@ my $monitor_pid;
 my $monitor_cnt = 0;
 my $sleep_time;
 my $bisect_sleep_time;
+my $patchcheck_sleep_time;
 my $store_failures;
 my $timeout;
 my $booted_timeout;
@@ -1764,6 +1766,14 @@ sub config_bisect {
     success $i;
 }
 
+sub patchcheck_reboot {
+    doprint "Reboot and sleep $patchcheck_sleep_time seconds\n";
+    reboot;
+    start_monitor;
+    wait_for_monitor $patchcheck_sleep_time;
+    end_monitor;
+}
+
 sub patchcheck {
     my ($i) = @_;
 
@@ -1855,6 +1865,8 @@ sub patchcheck {
 	end_monitor;
 	return 0 if ($failed);
 
+	patchcheck_reboot;
+
     }
     $in_patchcheck = 0;
     success $i;
@@ -2004,6 +2016,7 @@ for (my $i = 1; $i <= $opt{"NUM_TESTS"}; $i++) {
     $poweroff_after_halt = set_test_option("POWEROFF_AFTER_HALT", $i);
     $sleep_time = set_test_option("SLEEP_TIME", $i);
     $bisect_sleep_time = set_test_option("BISECT_SLEEP_TIME", $i);
+    $patchcheck_sleep_time = set_test_option("PATCHCHECK_SLEEP_TIME", $i);
     $bisect_manual = set_test_option("BISECT_MANUAL", $i);
     $bisect_skip = set_test_option("BISECT_SKIP", $i);
     $store_failures = set_test_option("STORE_FAILURES", $i);
diff --git a/tools/testing/ktest/sample.conf b/tools/testing/ktest/sample.conf
index 4c5d6bd..87bf92a 100644
--- a/tools/testing/ktest/sample.conf
+++ b/tools/testing/ktest/sample.conf
@@ -366,6 +366,10 @@
 # (default 60)
 #BISECT_SLEEP_TIME = 60
 
+# The time in between patch checks to sleep (in seconds)
+# (default 60)
+#PATCHCHECK_SLEEP_TIME = 60
+
 # Reboot the target box on error (default 0)
 #REBOOT_ON_ERROR = 0
 
-- 
1.7.4.4

[PATCH 5/6] ktest: Create variables for the ktest config files

[Steven Rostedt]

[-- Attachment #1: 0005-ktest-Create-variables-for-the-ktest-config-files.patch --]
[-- Type: text/plain, Size: 5764 bytes --]

From: Steven Rostedt <srostedt@redhat.com>

I found that I constantly reuse information for each test case.
It would be nice to just define a variable to reuse.

For example I may have:

TEST_START
[...]
TEST = ssh root@mybox /path/to/my/script

TEST_START
[...]
TEST = ssh root@mybox /path/to/my/script

[etc]

The issue is, I may wont to change that script or one of the other
fields. Then I need to update each line individually.

With the addition of config variables (variables only used during parsing
the config) we can simplify the config files. These variables can
also be defined multiple times and each time the new value will
overwrite the old value.

The convention to use a config variable over a ktest option is to use :=
instead of =.

Now we could do:

USER := root
TARGET := mybox
TEST_SCRIPT := /path/to/my/script
TEST_CASE := ${USER}@${TARGET} ${TEST_SCRIPT}

TEST_START
[...]
TEST = ${TEST_CASE}

TEST_START
[...]
TEST = ${TEST_CASE}

[etc]

Now we just need to update the variables at the top.

Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
---
 tools/testing/ktest/ktest.pl    |   62 +++++++++++++++++++++++++++++++++++++++
 tools/testing/ktest/sample.conf |   59 +++++++++++++++++++++++++++++++++++++
 2 files changed, 121 insertions(+), 0 deletions(-)

diff --git a/tools/testing/ktest/ktest.pl b/tools/testing/ktest/ktest.pl
index 65003a1..18ef668 100755
--- a/tools/testing/ktest/ktest.pl
+++ b/tools/testing/ktest/ktest.pl
@@ -114,6 +114,7 @@ my $successes = 0;
 
 my %entered_configs;
 my %config_help;
+my %variable;
 
 $config_help{"MACHINE"} = << "EOF"
  The machine hostname that you will test.
@@ -262,6 +263,39 @@ sub get_ktest_configs {
     }
 }
 
+sub process_variables {
+    my ($value) = @_;
+    my $retval = "";
+
+    # We want to check for '\', and it is just easier
+    # to check the previous characet of '$' and not need
+    # to worry if '$' is the first character. By adding
+    # a space to $value, we can just check [^\\]\$ and
+    # it will still work.
+    $value = " $value";
+
+    while ($value =~ /(.*?[^\\])\$\{(.*?)\}(.*)/) {
+	my $begin = $1;
+	my $var = $2;
+	my $end = $3;
+	# append beginning of value to retval
+	$retval = "$retval$begin";
+	if (defined($variable{$var})) {
+	    $retval = "$retval$variable{$var}";
+	} else {
+	    # put back the origin piece.
+	    $retval = "$retval\$\{$var\}";
+	}
+	$value = $end;
+    }
+    $retval = "$retval$value";
+
+    # remove the space added in the beginning
+    $retval =~ s/ //;
+
+    return "$retval"
+}
+
 sub set_value {
     my ($lvalue, $rvalue) = @_;
 
@@ -271,10 +305,22 @@ sub set_value {
     if ($rvalue =~ /^\s*$/) {
 	delete $opt{$lvalue};
     } else {
+	$rvalue = process_variables($rvalue);
 	$opt{$lvalue} = $rvalue;
     }
 }
 
+sub set_variable {
+    my ($lvalue, $rvalue) = @_;
+
+    if ($rvalue =~ /^\s*$/) {
+	delete $variable{$lvalue};
+    } else {
+	$rvalue = process_variables($rvalue);
+	$variable{$lvalue} = $rvalue;
+    }
+}
+
 sub read_config {
     my ($config) = @_;
 
@@ -387,6 +433,22 @@ sub read_config {
 		    $repeats{$val} = $repeat;
 		}
 	    }
+	} elsif (/^\s*([A-Z_\[\]\d]+)\s*:=\s*(.*?)\s*$/) {
+	    next if ($skip);
+
+	    my $lvalue = $1;
+	    my $rvalue = $2;
+
+	    # process config variables.
+	    # Config variables are only active while reading the
+	    # config and can be defined anywhere. They also ignore
+	    # TEST_START and DEFAULTS, but are skipped if they are in
+	    # on of these sections that have SKIP defined.
+	    # The save variable can be
+	    # defined multiple times and the new one simply overrides
+	    # the prevous one.
+	    set_variable($lvalue, $rvalue);
+
 	} else {
 	    die "$name: $.: Garbage found in config\n$_";
 	}
diff --git a/tools/testing/ktest/sample.conf b/tools/testing/ktest/sample.conf
index 87bf92a..761079e 100644
--- a/tools/testing/ktest/sample.conf
+++ b/tools/testing/ktest/sample.conf
@@ -73,6 +73,65 @@
 # ktest will fail to execute, and no tests will run.
 #
 
+#### Config variables ####
+#
+# This config file can also contain "config variables".
+# These are assigned with ":=" instead of the ktest option
+# assigment "=".
+#
+# The difference between ktest options and config variables
+# is that config variables can be used multiple times,
+# where each instance will override the previous instance.
+# And that they only live at time of processing this config.
+#
+# The advantage to config variables are that they can be used
+# by any option or any other config variables to define thing
+# that you may use over and over again in the options.
+#
+# For example:
+#
+# USER      := root
+# TARGET    := mybox
+# TEST_CASE := ssh ${USER}@${TARGET} /path/to/my/test
+#
+# TEST_START
+# MIN_CONFIG = config1
+# TEST = ${TEST_CASE}
+#
+# TEST_START
+# MIN_CONFIG = config2
+# TEST = ${TEST_CASE}
+#
+# TEST_CASE := ssh ${USER}@${TARGET} /path/to/my/test2
+#
+# TEST_START
+# MIN_CONFIG = config1
+# TEST = ${TEST_CASE}
+#
+# TEST_START
+# MIN_CONFIG = config2
+# TEST = ${TEST_CASE}
+#
+# TEST_DIR := /home/me/test
+#
+# BUILD_DIR = ${TEST_DIR}/linux.git
+# OUTPUT_DIR = ${TEST_DIR}/test
+#
+# Note, the config variables are evaluated immediately, thus
+# updating TARGET after TEST_CASE has been assigned does nothing
+# to TEST_CASE.
+#
+# As shown in the example, to evaluate a config variable, you
+# use the ${X} convention. Simple $X will not work.
+#
+# If the config variable does not exist, the ${X} will not
+# be evaluated. Thus:
+#
+# MAKE_CMD = PATH=/mypath:${PATH} make
+#
+# If PATH is not a config variable, then the ${PATH} in
+# the MAKE_CMD option will be evaluated by the shell when
+# the MAKE_CMD option is passed into shell processing.
 
 #### Mandatory Default Options ####
 
-- 
1.7.4.4

[PATCH 6/6] ktest: Allow options to be used by other options

[Steven Rostedt]

[-- Attachment #1: 0006-ktest-Allow-options-to-be-used-by-other-options.patch --]
[-- Type: text/plain, Size: 3896 bytes --]

From: Steven Rostedt <srostedt@redhat.com>

There are cases where one ktest option may be used within another
ktest option. Allow them to be reused just like config variables
but there are evaluated at time of test not config processing time.

Thus having something like:

MAKE_CMD = make ARCH=${ARCH}

TEST_START
ARCH = powerpc

TEST_START
ARCH = arm

Will have the arch defined for each test iteration.

Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
---
 tools/testing/ktest/ktest.pl    |   68 ++++++++++++++++++++++++++++++++++++++-
 tools/testing/ktest/sample.conf |   30 +++++++++++++++++
 2 files changed, 97 insertions(+), 1 deletions(-)

diff --git a/tools/testing/ktest/ktest.pl b/tools/testing/ktest/ktest.pl
index 18ef668..1fd29b2 100755
--- a/tools/testing/ktest/ktest.pl
+++ b/tools/testing/ktest/ktest.pl
@@ -2019,7 +2019,7 @@ for (my $i = 0, my $repeat = 1; $i <= $opt{"NUM_TESTS"}; $i += $repeat) {
     }
 }
 
-sub set_test_option {
+sub __set_test_option {
     my ($name, $i) = @_;
 
     my $option = "$name\[$i\]";
@@ -2045,6 +2045,72 @@ sub set_test_option {
     return undef;
 }
 
+sub eval_option {
+    my ($option, $i) = @_;
+
+    # Add space to evaluate the character before $
+    $option = " $option";
+    my $retval = "";
+
+    while ($option =~ /(.*?[^\\])\$\{(.*?)\}(.*)/) {
+	my $start = $1;
+	my $var = $2;
+	my $end = $3;
+
+	# Append beginning of line
+	$retval = "$retval$start";
+
+	# If the iteration option OPT[$i] exists, then use that.
+	# otherwise see if the default OPT (without [$i]) exists.
+
+	my $o = "$var\[$i\]";
+
+	if (defined($opt{$o})) {
+	    $o = $opt{$o};
+	    $retval = "$retval$o";
+	} elsif (defined($opt{$var})) {
+	    $o = $opt{$var};
+	    $retval = "$retval$o";
+	} else {
+	    $retval = "$retval\$\{$var\}";
+	}
+
+	$option = $end;
+    }
+
+    $retval = "$retval$option";
+
+    $retval =~ s/^ //;
+
+    return $retval;
+}
+
+sub set_test_option {
+    my ($name, $i) = @_;
+
+    my $option = __set_test_option($name, $i);
+    return $option if (!defined($option));
+
+    my $prev = "";
+
+    # Since an option can evaluate to another option,
+    # keep iterating until we do not evaluate any more
+    # options.
+    my $r = 0;
+    while ($prev ne $option) {
+	# Check for recursive evaluations.
+	# 100 deep should be more than enough.
+	if ($r++ > 100) {
+	    die "Over 100 evaluations accurred with $name\n" .
+		"Check for recursive variables\n";
+	}
+	$prev = $option;
+	$option = eval_option($option, $i);
+    }
+
+    return $option;
+}
+
 # First we need to do is the builds
 for (my $i = 1; $i <= $opt{"NUM_TESTS"}; $i++) {
 
diff --git a/tools/testing/ktest/sample.conf b/tools/testing/ktest/sample.conf
index 761079e..48cbcc80 100644
--- a/tools/testing/ktest/sample.conf
+++ b/tools/testing/ktest/sample.conf
@@ -133,6 +133,36 @@
 # the MAKE_CMD option will be evaluated by the shell when
 # the MAKE_CMD option is passed into shell processing.
 
+#### Using options in other options ####
+#
+# Options that are defined in the config file may also be used
+# by other options. All options are evaulated at time of
+# use (except that config variables are evaluated at config
+# processing time).
+#
+# If an ktest option is used within another option, instead of
+# typing it again in that option you can simply use the option
+# just like you can config variables.
+#
+# MACHINE = mybox
+#
+# TEST = ssh root@${MACHINE} /path/to/test
+#
+# The option will be used per test case. Thus:
+#
+# TEST_TYPE = test
+# TEST = ssh root@{MACHINE}
+#
+# TEST_START
+# MACHINE = box1
+#
+# TEST_START
+# MACHINE = box2
+#
+# For both test cases, MACHINE will be evaluated at the time
+# of the test case. The first test will run ssh root@box1
+# and the second will run ssh root@box2.
+
 #### Mandatory Default Options ####
 
 # These options must be in the default section, although most
-- 
1.7.4.4