[KTEST PATCH 00/10] collection of ktest patches

[KTEST PATCH 00/10] collection of ktest patches

[Andrew Jones]

More than once I've had to drive git-bisect to bisect boot issues for
Xen guests. The last time I needed to do so I decided to give ktest a
whirl. While whirling I wrote a couple patches and added a few features.
This patchset is the collection.

Andrew Jones (10):
  ktest: create outputdir, if it doesn't exist
  ktest: small cleanup
  ktest: factor reboot code
  ktest: Introduce RESET_TIME
  ktest: refactor monitor/boot/test code
  ktest: make start_monitor_and_boot true to its name
  ktest: Introduce PASS_COUNT
  ktest: test faster, put REBOOT_ON_SUCCESS to more work
  ktest: test faster, favor rsync over the tarball method
  ktest: Introduce FAILURE_LINE

 tools/testing/ktest/ktest.pl    |  208 +++++++++++++++++++++++----------------
 tools/testing/ktest/sample.conf |   23 ++++-
 2 files changed, 147 insertions(+), 84 deletions(-)

-- 
1.7.4.1

[PATCH 01/10] ktest: create outputdir, if it doesn't exist

[Andrew Jones]

Signed-off-by: Andrew Jones <drjones@redhat.com>
---
 tools/testing/ktest/ktest.pl |    8 +++++---
 1 files changed, 5 insertions(+), 3 deletions(-)

diff --git a/tools/testing/ktest/ktest.pl b/tools/testing/ktest/ktest.pl
index 8d02ccb..7c0b201 100755
--- a/tools/testing/ktest/ktest.pl
+++ b/tools/testing/ktest/ktest.pl
@@ -2850,9 +2850,11 @@ for (my $i = 1; $i <= $opt{"NUM_TESTS"}; $i++) {
 
     chdir $builddir || die "can't change directory to $builddir";
 
-    if (!-d $tmpdir) {
-	mkpath($tmpdir) or
-	    die "can't create $tmpdir";
+    foreach my $dir ($tmpdir, $outputdir) {
+	if (!-d $dir) {
+	    mkpath($dir) or
+		die "can't create $dir";
+	}
     }
 
     $ENV{"SSH_USER"} = $ssh_user;
-- 
1.7.4.1

[PATCH 02/10] ktest: small cleanup

[Andrew Jones]

only save the .config file if we're doing mrproper

Signed-off-by: Andrew Jones <drjones@redhat.com>
---
 tools/testing/ktest/ktest.pl |   14 +++++++-------
 1 files changed, 7 insertions(+), 7 deletions(-)

diff --git a/tools/testing/ktest/ktest.pl b/tools/testing/ktest/ktest.pl
index 7c0b201..f1358c5 100755
--- a/tools/testing/ktest/ktest.pl
+++ b/tools/testing/ktest/ktest.pl
@@ -1272,15 +1272,15 @@ sub build {
 	# allow for empty configs
 	run_command "touch $output_config";
 
-	run_command "mv $output_config $outputdir/config_temp" or
-	    dodie "moving .config";
+	if (!$noclean) {
+	    run_command "mv $output_config $outputdir/config_temp" or
+		dodie "moving .config";
 
-	if (!$noclean && !run_command "$make mrproper") {
-	    dodie "make mrproper";
-	}
+	    run_command "$make mrproper" or dodie "make mrproper";
 
-	run_command "mv $outputdir/config_temp $output_config" or
-	    dodie "moving config_temp";
+	    run_command "mv $outputdir/config_temp $output_config" or
+		dodie "moving config_temp";
+	}
 
     } elsif (!$noclean) {
 	unlink "$output_config";
-- 
1.7.4.1

[PATCH 03/10] ktest: factor reboot code

[Andrew Jones]

clean up redundant code and prepare for reset time patch

Signed-off-by: Andrew Jones <drjones@redhat.com>
---
 tools/testing/ktest/ktest.pl |   36 ++++++++++++++++--------------------
 1 files changed, 16 insertions(+), 20 deletions(-)

diff --git a/tools/testing/ktest/ktest.pl b/tools/testing/ktest/ktest.pl
index f1358c5..12c392e 100755
--- a/tools/testing/ktest/ktest.pl
+++ b/tools/testing/ktest/ktest.pl
@@ -603,8 +603,13 @@ sub doprint {
 }
 
 sub run_command;
+sub start_monitor;
+sub end_monitor;
+sub wait_for_monitor;
 
 sub reboot {
+    my ($time) = @_;
+
     # try to reboot normally
     if (run_command $reboot) {
 	if (defined($powercycle_after_reboot)) {
@@ -615,6 +620,12 @@ sub reboot {
 	# nope? power cycle it.
 	run_command "$power_cycle";
     }
+
+    if (defined($time)) {
+	start_monitor;
+	wait_for_monitor $time;
+	end_monitor;
+    }
 }
 
 sub do_not_reboot {
@@ -719,10 +730,7 @@ sub fail {
 	# no need to reboot for just building.
 	if (!do_not_reboot) {
 	    doprint "REBOOTING\n";
-	    reboot;
-	    start_monitor;
-	    wait_for_monitor $sleep_time;
-	    end_monitor;
+	    reboot $sleep_time;
 	}
 
 	my $name = "";
@@ -1356,10 +1364,7 @@ sub success {
 
     if ($i != $opt{"NUM_TESTS"} && !do_not_reboot) {
 	doprint "Reboot and wait $sleep_time seconds\n";
-	reboot;
-	start_monitor;
-	wait_for_monitor $sleep_time;
-	end_monitor;
+	reboot $sleep_time;
     }
 }
 
@@ -1500,10 +1505,7 @@ sub run_git_bisect {
 
 sub bisect_reboot {
     doprint "Reboot and sleep $bisect_sleep_time seconds\n";
-    reboot;
-    start_monitor;
-    wait_for_monitor $bisect_sleep_time;
-    end_monitor;
+    reboot $bisect_sleep_time;
 }
 
 # returns 1 on success, 0 on failure, -1 on skip
@@ -2066,10 +2068,7 @@ sub config_bisect {
 
 sub patchcheck_reboot {
     doprint "Reboot and sleep $patchcheck_sleep_time seconds\n";
-    reboot;
-    start_monitor;
-    wait_for_monitor $patchcheck_sleep_time;
-    end_monitor;
+    reboot $patchcheck_sleep_time;
 }
 
 sub patchcheck {
@@ -2659,10 +2658,7 @@ sub make_min_config {
 	}
 
 	doprint "Reboot and wait $sleep_time seconds\n";
-	reboot;
-	start_monitor;
-	wait_for_monitor $sleep_time;
-	end_monitor;
+	reboot $sleep_time;
     }
 
     success $i;
-- 
1.7.4.1

[PATCH 04/10] ktest: Introduce RESET_TIME

[Andrew Jones]

When rebooting, some targets may lose their console connection. This
certainly happens with 'virsh console' when used with my xen guests.
Setting RESET_TIME will tell ktest to reconnect the console after
reboot.

Signed-off-by: Andrew Jones <drjones@redhat.com>
---
 tools/testing/ktest/ktest.pl    |   28 +++++++++++++++++++++++-----
 tools/testing/ktest/sample.conf |    8 ++++++++
 2 files changed, 31 insertions(+), 5 deletions(-)

diff --git a/tools/testing/ktest/ktest.pl b/tools/testing/ktest/ktest.pl
index 12c392e..d954564 100755
--- a/tools/testing/ktest/ktest.pl
+++ b/tools/testing/ktest/ktest.pl
@@ -108,6 +108,7 @@ my $monitor_cnt = 0;
 my $sleep_time;
 my $bisect_sleep_time;
 my $patchcheck_sleep_time;
+my $reset_time;
 my $ignore_warnings;
 my $store_failures;
 my $test_name;
@@ -606,6 +607,7 @@ sub run_command;
 sub start_monitor;
 sub end_monitor;
 sub wait_for_monitor;
+sub reset_monitor;
 
 sub reboot {
     my ($time) = @_;
@@ -626,6 +628,7 @@ sub reboot {
 	wait_for_monitor $time;
 	end_monitor;
     }
+    reset_monitor;
 }
 
 sub do_not_reboot {
@@ -685,7 +688,9 @@ sub close_console {
 }
 
 sub start_monitor {
-    if ($monitor_cnt++) {
+    my ($force) = @_;
+
+    if ($monitor_cnt++ && !defined($force)) {
 	return;
     }
     $monitor_fp = \*MONFD;
@@ -697,12 +702,23 @@ sub start_monitor {
 }
 
 sub end_monitor {
-    if (--$monitor_cnt) {
+    my ($force) = @_;
+
+    if (--$monitor_cnt && !defined($force)) {
 	return;
     }
     close_console($monitor_fp, $monitor_pid);
 }
 
+sub reset_monitor {
+    if ($monitor_cnt <= 0 || !defined($reset_time)) {
+	return;
+    }
+    end_monitor 'force';
+    sleep $reset_time;
+    start_monitor 'force';
+}
+
 sub wait_for_monitor {
     my ($time) = @_;
     my $line;
@@ -911,10 +927,11 @@ sub wait_for_input
 sub reboot_to {
     if ($reboot_type eq "grub") {
 	run_ssh "'(echo \"savedefault --default=$grub_number --once\" | grub --batch && reboot)'";
-	return;
+    } else {
+	run_command "$reboot_script";
     }
-
-    run_command "$reboot_script";
+    wait_for_monitor $sleep_time;
+    reset_monitor;
 }
 
 sub get_sha1 {
@@ -2817,6 +2834,7 @@ for (my $i = 1; $i <= $opt{"NUM_TESTS"}; $i++) {
     $sleep_time = set_test_option("SLEEP_TIME", $i);
     $bisect_sleep_time = set_test_option("BISECT_SLEEP_TIME", $i);
     $patchcheck_sleep_time = set_test_option("PATCHCHECK_SLEEP_TIME", $i);
+    $reset_time = set_test_option("RESET_TIME", $i);
     $ignore_warnings = set_test_option("IGNORE_WARNINGS", $i);
     $bisect_manual = set_test_option("BISECT_MANUAL", $i);
     $bisect_skip = set_test_option("BISECT_SKIP", $i);
diff --git a/tools/testing/ktest/sample.conf b/tools/testing/ktest/sample.conf
index b8bcd14..23b2fbb 100644
--- a/tools/testing/ktest/sample.conf
+++ b/tools/testing/ktest/sample.conf
@@ -491,6 +491,14 @@
 # (default 60)
 #PATCHCHECK_SLEEP_TIME = 60
 
+# If the console needs to be reset during a reboot cycle in
+# order to reestablish it's connection, then set this option
+# to the number of seconds ktest should wait between disconnect
+# and reconnect. This is needed when using 'virsh console' to
+# connect to guests.
+# (default undefined)
+#RESET_TIME = 5
+
 # Reboot the target box on error (default 0)
 #REBOOT_ON_ERROR = 0
 
-- 
1.7.4.1

Re: [PATCH 04/10] ktest: Introduce RESET_TIME

[Steven Rostedt]

Hi Andrew,


On Fri, 2011-08-12 at 15:32 +0200, Andrew Jones wrote:
 
> +# If the console needs to be reset during a reboot cycle in
> +# order to reestablish it's connection, then set this option
> +# to the number of seconds ktest should wait between disconnect
> +# and reconnect. This is needed when using 'virsh console' to
> +# connect to guests.
> +# (default undefined)
> +#RESET_TIME = 5

I think the name RESET_TIME is a bit ambiguous. Maybe it should be
called CONSOLE_RESET_TIME.

-- Steve

> +
>  # Reboot the target box on error (default 0)
>  #REBOOT_ON_ERROR = 0
>  

[PATCH 04/10 v2] ktest: Introduce CONSOLE_RESET_TIME

[Andrew Jones]

When rebooting, some targets may lose their console connection. This
certainly happens with 'virsh console' when used with my xen guests.
Setting CONSOLE_RESET_TIME will tell ktest to reconnect the console
after reboot.
---
 tools/testing/ktest/ktest.pl    |   28 +++++++++++++++++++++++-----
 tools/testing/ktest/sample.conf |    8 ++++++++
 2 files changed, 31 insertions(+), 5 deletions(-)

diff --git a/tools/testing/ktest/ktest.pl b/tools/testing/ktest/ktest.pl
index 12c392e..8bc6dd8 100755
--- a/tools/testing/ktest/ktest.pl
+++ b/tools/testing/ktest/ktest.pl
@@ -108,6 +108,7 @@ my $monitor_cnt = 0;
 my $sleep_time;
 my $bisect_sleep_time;
 my $patchcheck_sleep_time;
+my $console_reset_time;
 my $ignore_warnings;
 my $store_failures;
 my $test_name;
@@ -606,6 +607,7 @@ sub run_command;
 sub start_monitor;
 sub end_monitor;
 sub wait_for_monitor;
+sub reset_monitor;
 
 sub reboot {
     my ($time) = @_;
@@ -626,6 +628,7 @@ sub reboot {
 	wait_for_monitor $time;
 	end_monitor;
     }
+    reset_monitor;
 }
 
 sub do_not_reboot {
@@ -685,7 +688,9 @@ sub close_console {
 }
 
 sub start_monitor {
-    if ($monitor_cnt++) {
+    my ($force) = @_;
+
+    if ($monitor_cnt++ && !defined($force)) {
 	return;
     }
     $monitor_fp = \*MONFD;
@@ -697,12 +702,23 @@ sub start_monitor {
 }
 
 sub end_monitor {
-    if (--$monitor_cnt) {
+    my ($force) = @_;
+
+    if (--$monitor_cnt && !defined($force)) {
 	return;
     }
     close_console($monitor_fp, $monitor_pid);
 }
 
+sub reset_monitor {
+    if ($monitor_cnt <= 0 || !defined($console_reset_time)) {
+	return;
+    }
+    end_monitor 'force';
+    sleep $console_reset_time;
+    start_monitor 'force';
+}
+
 sub wait_for_monitor {
     my ($time) = @_;
     my $line;
@@ -911,10 +927,11 @@ sub wait_for_input
 sub reboot_to {
     if ($reboot_type eq "grub") {
 	run_ssh "'(echo \"savedefault --default=$grub_number --once\" | grub --batch && reboot)'";
-	return;
+    } else {
+	run_command "$reboot_script";
     }
-
-    run_command "$reboot_script";
+    wait_for_monitor $sleep_time;
+    reset_monitor;
 }
 
 sub get_sha1 {
@@ -2817,6 +2834,7 @@ for (my $i = 1; $i <= $opt{"NUM_TESTS"}; $i++) {
     $sleep_time = set_test_option("SLEEP_TIME", $i);
     $bisect_sleep_time = set_test_option("BISECT_SLEEP_TIME", $i);
     $patchcheck_sleep_time = set_test_option("PATCHCHECK_SLEEP_TIME", $i);
+    $console_reset_time = set_test_option("CONSOLE_RESET_TIME", $i);
     $ignore_warnings = set_test_option("IGNORE_WARNINGS", $i);
     $bisect_manual = set_test_option("BISECT_MANUAL", $i);
     $bisect_skip = set_test_option("BISECT_SKIP", $i);
diff --git a/tools/testing/ktest/sample.conf b/tools/testing/ktest/sample.conf
index b8bcd14..ba430a7 100644
--- a/tools/testing/ktest/sample.conf
+++ b/tools/testing/ktest/sample.conf
@@ -491,6 +491,14 @@
 # (default 60)
 #PATCHCHECK_SLEEP_TIME = 60
 
+# If the console needs to be reset during a reboot cycle in
+# order to reestablish it's connection, then set this option
+# to the number of seconds ktest should wait between disconnect
+# and reconnect. This is needed when using 'virsh console' to
+# connect to guests.
+# (default undefined)
+#CONSOLE_RESET_TIME = 5
+
 # Reboot the target box on error (default 0)
 #REBOOT_ON_ERROR = 0
 
-- 
1.7.4.1

Re: [PATCH 04/10 v2] ktest: Introduce CONSOLE_RESET_TIME

[Steven Rostedt]

On Fri, 2011-08-12 at 18:25 +0200, Andrew Jones wrote:
> When rebooting, some targets may lose their console connection. This
> certainly happens with 'virsh console' when used with my xen guests.
> Setting CONSOLE_RESET_TIME will tell ktest to reconnect the console
> after reboot.

Oops, you forgot to add your SOB to this version of the patch. Could you
just reply to this with your signed-off-by tag.

Thanks!

-- Steve

> ---
>  tools/testing/ktest/ktest.pl    |   28 +++++++++++++++++++++++-----
>  tools/testing/ktest/sample.conf |    8 ++++++++
>  2 files changed, 31 insertions(+), 5 deletions(-)
> 
> diff --git a/tools/testing/ktest/ktest.pl b/tools/testing/ktest/ktest.pl
> index 12c392e..8bc6dd8 100755
> --- a/tools/testing/ktest/ktest.pl
> +++ b/tools/testing/ktest/ktest.pl
> @@ -108,6 +108,7 @@ my $monitor_cnt = 0;
>  my $sleep_time;
>  my $bisect_sleep_time;
>  my $patchcheck_sleep_time;
> +my $console_reset_time;
>  my $ignore_warnings;
>  my $store_failures;
>  my $test_name;
> @@ -606,6 +607,7 @@ sub run_command;
>  sub start_monitor;
>  sub end_monitor;
>  sub wait_for_monitor;
> +sub reset_monitor;
>  
>  sub reboot {
>      my ($time) = @_;
> @@ -626,6 +628,7 @@ sub reboot {
>  	wait_for_monitor $time;
>  	end_monitor;
>      }
> +    reset_monitor;
>  }
>  
>  sub do_not_reboot {
> @@ -685,7 +688,9 @@ sub close_console {
>  }
>  
>  sub start_monitor {
> -    if ($monitor_cnt++) {
> +    my ($force) = @_;
> +
> +    if ($monitor_cnt++ && !defined($force)) {
>  	return;
>      }
>      $monitor_fp = \*MONFD;
> @@ -697,12 +702,23 @@ sub start_monitor {
>  }
>  
>  sub end_monitor {
> -    if (--$monitor_cnt) {
> +    my ($force) = @_;
> +
> +    if (--$monitor_cnt && !defined($force)) {
>  	return;
>      }
>      close_console($monitor_fp, $monitor_pid);
>  }
>  
> +sub reset_monitor {
> +    if ($monitor_cnt <= 0 || !defined($console_reset_time)) {
> +	return;
> +    }
> +    end_monitor 'force';
> +    sleep $console_reset_time;
> +    start_monitor 'force';
> +}
> +
>  sub wait_for_monitor {
>      my ($time) = @_;
>      my $line;
> @@ -911,10 +927,11 @@ sub wait_for_input
>  sub reboot_to {
>      if ($reboot_type eq "grub") {
>  	run_ssh "'(echo \"savedefault --default=$grub_number --once\" | grub --batch && reboot)'";
> -	return;
> +    } else {
> +	run_command "$reboot_script";
>      }
> -
> -    run_command "$reboot_script";
> +    wait_for_monitor $sleep_time;
> +    reset_monitor;
>  }
>  
>  sub get_sha1 {
> @@ -2817,6 +2834,7 @@ for (my $i = 1; $i <= $opt{"NUM_TESTS"}; $i++) {
>      $sleep_time = set_test_option("SLEEP_TIME", $i);
>      $bisect_sleep_time = set_test_option("BISECT_SLEEP_TIME", $i);
>      $patchcheck_sleep_time = set_test_option("PATCHCHECK_SLEEP_TIME", $i);
> +    $console_reset_time = set_test_option("CONSOLE_RESET_TIME", $i);
>      $ignore_warnings = set_test_option("IGNORE_WARNINGS", $i);
>      $bisect_manual = set_test_option("BISECT_MANUAL", $i);
>      $bisect_skip = set_test_option("BISECT_SKIP", $i);
> diff --git a/tools/testing/ktest/sample.conf b/tools/testing/ktest/sample.conf
> index b8bcd14..ba430a7 100644
> --- a/tools/testing/ktest/sample.conf
> +++ b/tools/testing/ktest/sample.conf
> @@ -491,6 +491,14 @@
>  # (default 60)
>  #PATCHCHECK_SLEEP_TIME = 60
>  
> +# If the console needs to be reset during a reboot cycle in
> +# order to reestablish it's connection, then set this option
> +# to the number of seconds ktest should wait between disconnect
> +# and reconnect. This is needed when using 'virsh console' to
> +# connect to guests.
> +# (default undefined)
> +#CONSOLE_RESET_TIME = 5
> +
>  # Reboot the target box on error (default 0)
>  #REBOOT_ON_ERROR = 0
>  

Re: [PATCH 04/10 v2] ktest: Introduce CONSOLE_RESET_TIME

[Andrew Jones]

On Thu, Aug 18, 2011 at 05:28:03PM -0400, Steven Rostedt wrote:
> On Fri, 2011-08-12 at 18:25 +0200, Andrew Jones wrote:
> > When rebooting, some targets may lose their console connection. This
> > certainly happens with 'virsh console' when used with my xen guests.
> > Setting CONSOLE_RESET_TIME will tell ktest to reconnect the console
> > after reboot.
> 
> Oops, you forgot to add your SOB to this version of the patch. Could you
> just reply to this with your signed-off-by tag.
>

Doh, I think I may have missed all my v2s.
 
> Thanks!
> 
> -- Steve
> 
> > ---
> >  tools/testing/ktest/ktest.pl    |   28 +++++++++++++++++++++++-----
> >  tools/testing/ktest/sample.conf |    8 ++++++++
> >  2 files changed, 31 insertions(+), 5 deletions(-)
> > 
> > diff --git a/tools/testing/ktest/ktest.pl b/tools/testing/ktest/ktest.pl
> > index 12c392e..8bc6dd8 100755
> > --- a/tools/testing/ktest/ktest.pl
> > +++ b/tools/testing/ktest/ktest.pl
> > @@ -108,6 +108,7 @@ my $monitor_cnt = 0;
> >  my $sleep_time;
> >  my $bisect_sleep_time;
> >  my $patchcheck_sleep_time;
> > +my $console_reset_time;
> >  my $ignore_warnings;
> >  my $store_failures;
> >  my $test_name;
> > @@ -606,6 +607,7 @@ sub run_command;
> >  sub start_monitor;
> >  sub end_monitor;
> >  sub wait_for_monitor;
> > +sub reset_monitor;
> >  
> >  sub reboot {
> >      my ($time) = @_;
> > @@ -626,6 +628,7 @@ sub reboot {
> >  	wait_for_monitor $time;
> >  	end_monitor;
> >      }
> > +    reset_monitor;
> >  }
> >  
> >  sub do_not_reboot {
> > @@ -685,7 +688,9 @@ sub close_console {
> >  }
> >  
> >  sub start_monitor {
> > -    if ($monitor_cnt++) {
> > +    my ($force) = @_;
> > +
> > +    if ($monitor_cnt++ && !defined($force)) {
> >  	return;
> >      }
> >      $monitor_fp = \*MONFD;
> > @@ -697,12 +702,23 @@ sub start_monitor {
> >  }
> >  
> >  sub end_monitor {
> > -    if (--$monitor_cnt) {
> > +    my ($force) = @_;
> > +
> > +    if (--$monitor_cnt && !defined($force)) {
> >  	return;
> >      }
> >      close_console($monitor_fp, $monitor_pid);
> >  }
> >  
> > +sub reset_monitor {
> > +    if ($monitor_cnt <= 0 || !defined($console_reset_time)) {
> > +	return;
> > +    }
> > +    end_monitor 'force';
> > +    sleep $console_reset_time;
> > +    start_monitor 'force';
> > +}
> > +
> >  sub wait_for_monitor {
> >      my ($time) = @_;
> >      my $line;
> > @@ -911,10 +927,11 @@ sub wait_for_input
> >  sub reboot_to {
> >      if ($reboot_type eq "grub") {
> >  	run_ssh "'(echo \"savedefault --default=$grub_number --once\" | grub --batch && reboot)'";
> > -	return;
> > +    } else {
> > +	run_command "$reboot_script";
> >      }
> > -
> > -    run_command "$reboot_script";
> > +    wait_for_monitor $sleep_time;
> > +    reset_monitor;
> >  }
> >  
> >  sub get_sha1 {
> > @@ -2817,6 +2834,7 @@ for (my $i = 1; $i <= $opt{"NUM_TESTS"}; $i++) {
> >      $sleep_time = set_test_option("SLEEP_TIME", $i);
> >      $bisect_sleep_time = set_test_option("BISECT_SLEEP_TIME", $i);
> >      $patchcheck_sleep_time = set_test_option("PATCHCHECK_SLEEP_TIME", $i);
> > +    $console_reset_time = set_test_option("CONSOLE_RESET_TIME", $i);
> >      $ignore_warnings = set_test_option("IGNORE_WARNINGS", $i);
> >      $bisect_manual = set_test_option("BISECT_MANUAL", $i);
> >      $bisect_skip = set_test_option("BISECT_SKIP", $i);
> > diff --git a/tools/testing/ktest/sample.conf b/tools/testing/ktest/sample.conf
> > index b8bcd14..ba430a7 100644
> > --- a/tools/testing/ktest/sample.conf
> > +++ b/tools/testing/ktest/sample.conf
> > @@ -491,6 +491,14 @@
> >  # (default 60)
> >  #PATCHCHECK_SLEEP_TIME = 60
> >  
> > +# If the console needs to be reset during a reboot cycle in
> > +# order to reestablish it's connection, then set this option
> > +# to the number of seconds ktest should wait between disconnect
> > +# and reconnect. This is needed when using 'virsh console' to
> > +# connect to guests.
> > +# (default undefined)
> > +#CONSOLE_RESET_TIME = 5
> > +
> >  # Reboot the target box on error (default 0)
> >  #REBOOT_ON_ERROR = 0
> >  
> 
>
Signed-off-by: Andrew Jones <drjones@redhat.com>

[PATCH 05/10] ktest: refactor monitor/boot/test code

[Andrew Jones]

The monitor/boot code was already refactored, but we need to pull the
test part of the monitor/boot/test code sequences in as well to prepare for
the next patch.

Signed-off-by: Andrew Jones <drjones@redhat.com>
---
 tools/testing/ktest/ktest.pl |   60 +++++++++++++++++------------------------
 1 files changed, 25 insertions(+), 35 deletions(-)

diff --git a/tools/testing/ktest/ktest.pl b/tools/testing/ktest/ktest.pl
index d954564..a42f8bf 100755
--- a/tools/testing/ktest/ktest.pl
+++ b/tools/testing/ktest/ktest.pl
@@ -1490,6 +1490,24 @@ sub do_run_test {
     return 1;
 }
 
+sub do_test {
+    my ($type) = @_;
+    my $failed = 0;
+
+    start_monitor_and_boot or $failed = 1;
+
+    if ($failed && $in_bisect && $type ne "boot") {
+	end_monitor;
+	return -1;
+    }
+
+    if (!$failed && $type ne "boot" && defined($run_test)) {
+	do_run_test or $failed = 1;
+    }
+    end_monitor;
+    return !$failed;
+}
+
 sub run_git_bisect {
     my ($command) = @_;
 
@@ -1531,8 +1549,6 @@ sub run_bisect_test {
 
     my $failed = 0;
     my $result;
-    my $output;
-    my $ret;
 
     $in_bisect = 1;
 
@@ -1545,27 +1561,17 @@ sub run_bisect_test {
 	}
 	dodie "Failed on build" if $failed;
 
-	# Now boot the box
-	start_monitor_and_boot or $failed = 1;
+	# Now boot the box and test
+	$result = do_test($type);
 
-	if ($type ne "boot") {
-	    if ($failed && $bisect_skip) {
-		end_monitor;
+	if ($result == -1) {
+	    if ($bisect_skip) {
 		bisect_reboot;
 		$in_bisect = 0;
 		return -1;
 	    }
-	    dodie "Failed on boot" if $failed;
-
-	    do_run_test or $failed = 1;
+	    dodie "Failed on boot";
 	}
-	end_monitor;
-    }
-
-    if ($failed) {
-	$result = 0;
-    } else {
-	$result = 1;
     }
 
     # reboot the box to a kernel we can ssh to
@@ -2174,18 +2180,9 @@ sub patchcheck {
 
 	next if ($type eq "build");
 
-	my $failed = 0;
-
-	start_monitor_and_boot or $failed = 1;
-
-	if (!$failed && $type ne "boot"){
-	    do_run_test or $failed = 1;
-	}
-	end_monitor;
-	return 0 if ($failed);
+	do_test($type) or return 0;
 
 	patchcheck_reboot;
-
     }
     $in_patchcheck = 0;
     success $i;
@@ -2946,14 +2943,7 @@ for (my $i = 1; $i <= $opt{"NUM_TESTS"}; $i++) {
     }
 
     if ($test_type ne "build") {
-	my $failed = 0;
-	start_monitor_and_boot or $failed = 1;
-
-	if (!$failed && $test_type ne "boot" && defined($run_test)) {
-	    do_run_test or $failed = 1;
-	}
-	end_monitor;
-	next if ($failed);
+	do_test($test_type) or next;
     }
 
     success $i;
-- 
1.7.4.1

[PATCH 06/10] ktest: make start_monitor_and_boot true to its name

[Andrew Jones]

Pull out the install parts of the monitor/boot function

Signed-off-by: Andrew Jones <drjones@redhat.com>
---
 tools/testing/ktest/ktest.pl |    6 +++++-
 1 files changed, 5 insertions(+), 1 deletions(-)

diff --git a/tools/testing/ktest/ktest.pl b/tools/testing/ktest/ktest.pl
index a42f8bf..4265652 100755
--- a/tools/testing/ktest/ktest.pl
+++ b/tools/testing/ktest/ktest.pl
@@ -1164,11 +1164,13 @@ sub get_version {
     doprint "$version\n";
 }
 
-sub start_monitor_and_boot {
+sub grub_install {
     get_grub_index;
     get_version;
     install;
+}
 
+sub start_monitor_and_boot {
     start_monitor;
     return monitor;
 }
@@ -1494,6 +1496,7 @@ sub do_test {
     my ($type) = @_;
     my $failed = 0;
 
+    grub_install;
     start_monitor_and_boot or $failed = 1;
 
     if ($failed && $in_bisect && $type ne "boot") {
@@ -2615,6 +2618,7 @@ sub make_min_config {
 
 	my $failed = 0;
 	build "oldconfig";
+	grub_install;
 	start_monitor_and_boot or $failed = 1;
 	end_monitor;
 
-- 
1.7.4.1

[PATCH 07/10] ktest: Introduce PASS_COUNT

[Andrew Jones]

Add another config variable that defines the number of times a test
must pass before it really passes. This is good for boot tests, where
the failure doesn't occur every time.

Signed-off-by: Andrew Jones <drjones@redhat.com>
---
 tools/testing/ktest/ktest.pl    |   23 +++++++++++++++--------
 tools/testing/ktest/sample.conf |    7 +++++++
 2 files changed, 22 insertions(+), 8 deletions(-)

diff --git a/tools/testing/ktest/ktest.pl b/tools/testing/ktest/ktest.pl
index 4265652..38c4532 100755
--- a/tools/testing/ktest/ktest.pl
+++ b/tools/testing/ktest/ktest.pl
@@ -24,6 +24,7 @@ my %default;
 $default{"NUM_TESTS"}		= 1;
 $default{"REBOOT_TYPE"}		= "grub";
 $default{"TEST_TYPE"}		= "test";
+$default{"PASS_COUNT"}		= 1;
 $default{"BUILD_TYPE"}		= "randconfig";
 $default{"MAKE_CMD"}		= "make";
 $default{"TIMEOUT"}		= 120;
@@ -61,6 +62,7 @@ my $builddir;
 my $outputdir;
 my $output_config;
 my $test_type;
+my $pass_count;
 my $build_type;
 my $build_options;
 my $pre_build;
@@ -1495,19 +1497,23 @@ sub do_run_test {
 sub do_test {
     my ($type) = @_;
     my $failed = 0;
+    my $count = $pass_count;
 
     grub_install;
-    start_monitor_and_boot or $failed = 1;
 
-    if ($failed && $in_bisect && $type ne "boot") {
-	end_monitor;
-	return -1;
-    }
+    while (!$failed && $count--) {
+	start_monitor_and_boot or $failed = 1;
 
-    if (!$failed && $type ne "boot" && defined($run_test)) {
-	do_run_test or $failed = 1;
+	if ($failed && $in_bisect && $type ne "boot") {
+	    end_monitor;
+	    return -1;
+	}
+
+	if (!$failed && $type ne "boot" && defined($run_test)) {
+	    do_run_test or $failed = 1;
+	}
+	end_monitor;
     }
-    end_monitor;
     return !$failed;
 }
 
@@ -2807,6 +2813,7 @@ for (my $i = 1; $i <= $opt{"NUM_TESTS"}; $i++) {
     $outputdir = set_test_option("OUTPUT_DIR", $i);
     $builddir = set_test_option("BUILD_DIR", $i);
     $test_type = set_test_option("TEST_TYPE", $i);
+    $pass_count = set_test_option("PASS_COUNT", $i);
     $build_type = set_test_option("BUILD_TYPE", $i);
     $build_options = set_test_option("BUILD_OPTIONS", $i);
     $pre_build = set_test_option("PRE_BUILD", $i);
diff --git a/tools/testing/ktest/sample.conf b/tools/testing/ktest/sample.conf
index 23b2fbb..fe7630a 100644
--- a/tools/testing/ktest/sample.conf
+++ b/tools/testing/ktest/sample.conf
@@ -266,6 +266,13 @@
 # default (undefined)
 #TEST = ssh user@machine /root/run_test
 
+# The number of times the test must pass before we really believe
+# it passes. Generally you could put a loop in the 'TEST' program,
+# but this option makes it easier to do boot testing for problems
+# that don't occur on every boot.
+# (default 1)
+#PASS_COUNT = 10
+
 # The build type is any make config type or special command
 #  (default randconfig)
 #   nobuild - skip the clean and build step
-- 
1.7.4.1

Re: [PATCH 07/10] ktest: Introduce PASS_COUNT

[Steven Rostedt]

On Fri, 2011-08-12 at 15:32 +0200, Andrew Jones wrote:
> Add another config variable that defines the number of times a test
> must pass before it really passes. This is good for boot tests, where
> the failure doesn't occur every time.

I'm curious to how this is really different than the ITERATE keyword.

That is, if we had:

TEST_START ITERATE 10
TEST_TYPE = test
TEST = ssh root@box "/work/runtest"

Hmm, is this to help in the bisects?

Maybe it should be called ITERATE as well, just to be consistent.

TEST_START
TEST_TYPE = bisect
ITERATE = 10

??

Or maybe that is confusing too, as that could be used to do the iterate
within the TEST_START (hmm, I may add that).

How about PASS_THRESHOLD? or PASS_COUNT_THRESHOLD? As I think that may
be more descriptive.

/me rereads his email and sees that he has a tendency to talk to
himself.


-- Steve

Re: [PATCH 07/10] ktest: Introduce PASS_COUNT

[Andrew Jones]

On Fri, Aug 12, 2011 at 12:49:45PM -0400, Steven Rostedt wrote:
> On Fri, 2011-08-12 at 15:32 +0200, Andrew Jones wrote:
> > Add another config variable that defines the number of times a test
> > must pass before it really passes. This is good for boot tests, where
> > the failure doesn't occur every time.
> 
> I'm curious to how this is really different than the ITERATE keyword.
> 
> That is, if we had:
> 
> TEST_START ITERATE 10
> TEST_TYPE = test
> TEST = ssh root@box "/work/runtest"
> 
> Hmm, is this to help in the bisects?

Right. I played with ITERATE, but it didn't look like it would work
for bisecting boot problems that may fail once in some number of
boots. Perhaps I missed something though.

> 
> Maybe it should be called ITERATE as well, just to be consistent.
> 
> TEST_START
> TEST_TYPE = bisect
> ITERATE = 10
> 
> ??
> 
> Or maybe that is confusing too, as that could be used to do the iterate
> within the TEST_START (hmm, I may add that).
> 
> How about PASS_THRESHOLD? or PASS_COUNT_THRESHOLD? As I think that may
> be more descriptive.

PASS_THRESHOLD sounds good. Should I respin with that change now? Or are
you still considering a different design?

> 
> /me rereads his email and sees that he has a tendency to talk to
> himself.
>

/me wonders if he answered himself with a different design idea, so will
hold off on the updated patch for the moment.

Drew
> 
> -- Steve
> 
> 

Re: [PATCH 07/10] ktest: Introduce PASS_COUNT

[Steven Rostedt]

On Fri, 2011-08-12 at 19:09 +0200, Andrew Jones wrote:

> > How about PASS_THRESHOLD? or PASS_COUNT_THRESHOLD? As I think that may
> > be more descriptive.
> 
> PASS_THRESHOLD sounds good. Should I respin with that change now? Or are
> you still considering a different design?
> 

Nah, thinking about it more, I don't like PASS_THRESHOLD either. Maybe
we should have a RERUN option. That is, RERUN the test for each
iteration?

-- Steve

[PATCH 07/10 v2] ktest: Introduce RERUN

[Andrew Jones]

Add another config variable that defines the number of additional
times a test must pass before it really passes. This is good for
bisecting boot problems, where the failure doesn't occur every time.
---
 tools/testing/ktest/ktest.pl    |   23 +++++++++++++++--------
 tools/testing/ktest/sample.conf |    7 +++++++
 2 files changed, 22 insertions(+), 8 deletions(-)

diff --git a/tools/testing/ktest/ktest.pl b/tools/testing/ktest/ktest.pl
index e37c67e..28658b4 100755
--- a/tools/testing/ktest/ktest.pl
+++ b/tools/testing/ktest/ktest.pl
@@ -24,6 +24,7 @@ my %default;
 $default{"NUM_TESTS"}		= 1;
 $default{"REBOOT_TYPE"}		= "grub";
 $default{"TEST_TYPE"}		= "test";
+$default{"RERUN"}		= 0;
 $default{"BUILD_TYPE"}		= "randconfig";
 $default{"MAKE_CMD"}		= "make";
 $default{"TIMEOUT"}		= 120;
@@ -61,6 +62,7 @@ my $builddir;
 my $outputdir;
 my $output_config;
 my $test_type;
+my $rerun;
 my $build_type;
 my $build_options;
 my $pre_build;
@@ -1495,19 +1497,23 @@ sub do_run_test {
 sub do_test {
     my ($type) = @_;
     my $failed = 0;
+    my $runs = $rerun + 1; # run at least once
 
     grub_install;
-    start_monitor_and_boot or $failed = 1;
 
-    if ($failed && $in_bisect && $type ne "boot") {
-	end_monitor;
-	return -1;
-    }
+    while (!$failed && $runs--) {
+	start_monitor_and_boot or $failed = 1;
 
-    if (!$failed && $type ne "boot" && defined($run_test)) {
-	do_run_test or $failed = 1;
+	if ($failed && $in_bisect && $type ne "boot") {
+	    end_monitor;
+	    return -1;
+	}
+
+	if (!$failed && $type ne "boot" && defined($run_test)) {
+	    do_run_test or $failed = 1;
+	}
+	end_monitor;
     }
-    end_monitor;
     return !$failed;
 }
 
@@ -2807,6 +2813,7 @@ for (my $i = 1; $i <= $opt{"NUM_TESTS"}; $i++) {
     $outputdir = set_test_option("OUTPUT_DIR", $i);
     $builddir = set_test_option("BUILD_DIR", $i);
     $test_type = set_test_option("TEST_TYPE", $i);
+    $rerun = set_test_option("RERUN", $i);
     $build_type = set_test_option("BUILD_TYPE", $i);
     $build_options = set_test_option("BUILD_OPTIONS", $i);
     $pre_build = set_test_option("PRE_BUILD", $i);
diff --git a/tools/testing/ktest/sample.conf b/tools/testing/ktest/sample.conf
index ba430a7..8d0d33e 100644
--- a/tools/testing/ktest/sample.conf
+++ b/tools/testing/ktest/sample.conf
@@ -266,6 +266,13 @@
 # default (undefined)
 #TEST = ssh user@machine /root/run_test
 
+# The number of times the test must pass before we really believe
+# it passes. Generally you could put a loop in the 'TEST' program,
+# but this option makes it easier to do boot testing for problems
+# that don't occur on every boot.
+# (default 0)
+#RERUN = 10
+
 # The build type is any make config type or special command
 #  (default randconfig)
 #   nobuild - skip the clean and build step
-- 
1.7.4.1

[PATCH 08/10] ktest: test faster, put REBOOT_ON_SUCCESS to more work

[Andrew Jones]

Allow the user to decide if it's necessary to reboot after each
successful test or not. If a successfully booted target supports
installation of another kernel, then we might as well save the
time to reboot (which may be long on some systems) to a known-good
kernel. We already have REBOOT_ON_SUCCESS, which defaults to true,
and was underutilized. It only controlled the rebooting of the
target after all testing completed successfully. Now, if the user
sets this to false, we assume that rebooting isn't necessary at
any stage of the testing, assuming the tests are successful. The
option is now also a per-test config option, so the top-level,
default behavior may remain the same.

Signed-off-by: Andrew Jones <drjones@redhat.com>
---
 tools/testing/ktest/ktest.pl    |    8 +++++---
 tools/testing/ktest/sample.conf |    1 -
 2 files changed, 5 insertions(+), 4 deletions(-)

diff --git a/tools/testing/ktest/ktest.pl b/tools/testing/ktest/ktest.pl
index 38c4532..719df19 100755
--- a/tools/testing/ktest/ktest.pl
+++ b/tools/testing/ktest/ktest.pl
@@ -74,6 +74,7 @@ my $reboot_script;
 my $power_cycle;
 my $reboot;
 my $reboot_on_error;
+my $reboot_on_success;
 my $poweroff_on_error;
 my $die_on_failure;
 my $powercycle_after_reboot;
@@ -1383,7 +1384,7 @@ sub success {
     doprint     "*******************************************\n";
     doprint     "*******************************************\n";
 
-    if ($i != $opt{"NUM_TESTS"} && !do_not_reboot) {
+    if ($i != $opt{"NUM_TESTS"} && !do_not_reboot && $reboot_on_success) {
 	doprint "Reboot and wait $sleep_time seconds\n";
 	reboot $sleep_time;
     }
@@ -1585,7 +1586,7 @@ sub run_bisect_test {
 
     # reboot the box to a kernel we can ssh to
     if ($type ne "build") {
-	bisect_reboot;
+	bisect_reboot if ($result == 0 || $reboot_on_success);
     }
     $in_bisect = 0;
 
@@ -2191,7 +2192,7 @@ sub patchcheck {
 
 	do_test($type) or return 0;
 
-	patchcheck_reboot;
+	patchcheck_reboot if ($reboot_on_success);
     }
     $in_patchcheck = 0;
     success $i;
@@ -2834,6 +2835,7 @@ for (my $i = 1; $i <= $opt{"NUM_TESTS"}; $i++) {
     $post_install = set_test_option("POST_INSTALL", $i);
     $reboot_script = set_test_option("REBOOT_SCRIPT", $i);
     $reboot_on_error = set_test_option("REBOOT_ON_ERROR", $i);
+    $reboot_on_success = set_test_option("REBOOT_ON_SUCCESS", $i);
     $poweroff_on_error = set_test_option("POWEROFF_ON_ERROR", $i);
     $die_on_failure = set_test_option("DIE_ON_FAILURE", $i);
     $power_off = set_test_option("POWER_OFF", $i);
diff --git a/tools/testing/ktest/sample.conf b/tools/testing/ktest/sample.conf
index fe7630a..36521c8 100644
--- a/tools/testing/ktest/sample.conf
+++ b/tools/testing/ktest/sample.conf
@@ -11,7 +11,6 @@
 #  LOG_FILE
 #  CLEAR_LOG
 #  POWEROFF_ON_SUCCESS
-#  REBOOT_ON_SUCCESS
 #
 # Test specific options are set after the label:
 #
-- 
1.7.4.1

[PATCH 09/10] ktest: test faster, favor rsync over the tarball method

[Andrew Jones]

If we have rsync on the host and target, then use that to install
modules. It's faster, doesn't require as much free disk space, and
is more environmentally friendly. OK, maybe not the last one, but
it's still better.

Signed-off-by: Andrew Jones <drjones@redhat.com>
---
 tools/testing/ktest/ktest.pl |   34 +++++++++++++++++++++++++---------
 1 files changed, 25 insertions(+), 9 deletions(-)

diff --git a/tools/testing/ktest/ktest.pl b/tools/testing/ktest/ktest.pl
index 719df19..ac0e688 100755
--- a/tools/testing/ktest/ktest.pl
+++ b/tools/testing/ktest/ktest.pl
@@ -81,6 +81,8 @@ my $powercycle_after_reboot;
 my $poweroff_after_halt;
 my $ssh_exec;
 my $scp_to_target;
+my $have_rsync;
+my %has_rsync = ();
 my $power_off;
 my $grub_menu;
 my $grub_number;
@@ -1137,25 +1139,39 @@ sub install {
 	dodie "Failed to install modules";
 
     my $modlib = "/lib/modules/$version";
-    my $modtar = "ktest-mods.tar.bz2";
 
-    run_ssh "rm -rf $modlib" or
+    if (!defined($have_rsync)) {
+	$have_rsync = run_command("which rsync");
+    }
+
+    if ($have_rsync && !defined($has_rsync{$machine})) {
+	$has_rsync{$machine} = run_ssh("which rsync");
+    }
+
+    if ($have_rsync && $has_rsync{$machine}) {
+	run_command
+	    "rsync -az --delete-before $tmpdir$modlib/ $target:$modlib" or
+		dodie "failed to rsync modules";
+    } else {
+	my $modtar = "ktest-mods.tar.bz2";
+
+	run_ssh "rm -rf $modlib" or
 	dodie "failed to remove old mods: $modlib";
 
-    # would be nice if scp -r did not follow symbolic links
-    run_command "cd $tmpdir && tar -cjf $modtar lib/modules/$version" or
+	# would be nice if scp -r did not follow symbolic links
+	run_command "cd $tmpdir && tar -cjf $modtar lib/modules/$version" or
 	dodie "making tarball";
 
-    run_scp "$tmpdir/$modtar", "/tmp" or
+	run_scp "$tmpdir/$modtar", "/tmp" or
 	dodie "failed to copy modules";
 
-    unlink "$tmpdir/$modtar";
+	unlink "$tmpdir/$modtar";
 
-    run_ssh "'(cd / && tar xjf /tmp/$modtar)'" or
+	run_ssh "'(cd / && tar xjf /tmp/$modtar)'" or
 	dodie "failed to tar modules";
 
-    run_ssh "rm -f /tmp/$modtar";
-
+	run_ssh "rm -f /tmp/$modtar";
+    }
     do_post_install;
 }
 
-- 
1.7.4.1

[PATCH 10/10] ktest: Introduce FAILURE_LINE

[Andrew Jones]

This is the counterpart to SUCCESS_LINE. In some cases the boot will
succeed, but the console will display a message that means that the
boot test failed (such as a warning) while it boots. Use this option
to detect this and fail the test.

Signed-off-by: Andrew Jones <drjones@redhat.com>
---
 tools/testing/ktest/ktest.pl    |    7 +++++++
 tools/testing/ktest/sample.conf |    7 +++++++
 2 files changed, 14 insertions(+), 0 deletions(-)

diff --git a/tools/testing/ktest/ktest.pl b/tools/testing/ktest/ktest.pl
index ac0e688..35af7e2 100755
--- a/tools/testing/ktest/ktest.pl
+++ b/tools/testing/ktest/ktest.pl
@@ -122,6 +122,7 @@ my $booted_timeout;
 my $detect_triplefault;
 my $console;
 my $success_line;
+my $failure_line;
 my $stop_after_success;
 my $stop_after_failure;
 my $stop_test_after;
@@ -1015,6 +1016,11 @@ sub monitor {
 	# we are not guaranteed to get a full line
 	$full_line .= $line;
 
+	if (defined($failure_line) && $full_line =~ /$failure_line/) {
+	    $bug = 1;
+	    last;
+	}
+
 	if ($full_line =~ /$success_line/) {
 	    $booted = 1;
 	    $success_start = time;
@@ -2872,6 +2878,7 @@ for (my $i = 1; $i <= $opt{"NUM_TESTS"}; $i++) {
     $console = set_test_option("CONSOLE", $i);
     $detect_triplefault = set_test_option("DETECT_TRIPLE_FAULT", $i);
     $success_line = set_test_option("SUCCESS_LINE", $i);
+    $failure_line = set_test_option("FAILURE_LINE", $i);
     $stop_after_success = set_test_option("STOP_AFTER_SUCCESS", $i);
     $stop_after_failure = set_test_option("STOP_AFTER_FAILURE", $i);
     $stop_test_after = set_test_option("STOP_TEST_AFTER", $i);
diff --git a/tools/testing/ktest/sample.conf b/tools/testing/ktest/sample.conf
index 36521c8..98df23c 100644
--- a/tools/testing/ktest/sample.conf
+++ b/tools/testing/ktest/sample.conf
@@ -421,6 +421,13 @@
 # (default "login:")
 #SUCCESS_LINE = login:
 
+# Line indicating a failed boot. This is what the line contains, not
+# the entire line. If you need the entire line to match, then use
+# regular expression syntax like with SUCCESS_LINE. This is useful
+# for testing boot issues that don't stop the boot from completing.
+# (default undefined)
+#FAILURE_LINE = WARNING
+
 # In case the console constantly fills the screen, having
 # a specified time to stop the test after success is recommended.
 # (in seconds)
-- 
1.7.4.1

Re: [PATCH 10/10] ktest: Introduce FAILURE_LINE

[Steven Rostedt]

On Fri, 2011-08-12 at 15:32 +0200, Andrew Jones wrote:
> This is the counterpart to SUCCESS_LINE. In some cases the boot will
> succeed, but the console will display a message that means that the
> boot test failed (such as a warning) while it boots. Use this option
> to detect this and fail the test.
> 
> Signed-off-by: Andrew Jones <drjones@redhat.com>
> ---
>  tools/testing/ktest/ktest.pl    |    7 +++++++
>  tools/testing/ktest/sample.conf |    7 +++++++
>  2 files changed, 14 insertions(+), 0 deletions(-)
> 
> diff --git a/tools/testing/ktest/ktest.pl b/tools/testing/ktest/ktest.pl
> index ac0e688..35af7e2 100755
> --- a/tools/testing/ktest/ktest.pl
> +++ b/tools/testing/ktest/ktest.pl
> @@ -122,6 +122,7 @@ my $booted_timeout;
>  my $detect_triplefault;
>  my $console;
>  my $success_line;
> +my $failure_line;
>  my $stop_after_success;
>  my $stop_after_failure;
>  my $stop_test_after;
> @@ -1015,6 +1016,11 @@ sub monitor {
>  	# we are not guaranteed to get a full line
>  	$full_line .= $line;
>  
> +	if (defined($failure_line) && $full_line =~ /$failure_line/) {
> +	    $bug = 1;
> +	    last;

Instead of last, it is better to use:

		$failure_start = time;

Otherwise you may lose out any informational messages that print after
it (call stack). If the user wants to break out early, they can make the
failure time shorter.

-- Steve


> +	}
> +
>  	if ($full_line =~ /$success_line/) {
>  	    $booted = 1;
>  	    $success_start = time;
> @@ -2872,6 +2878,7 @@ for (my $i = 1; $i <= $opt{"NUM_TESTS"}; $i++) {
>      $console = set_test_option("CONSOLE", $i);
>      $detect_triplefault = set_test_option("DETECT_TRIPLE_FAULT", $i);
>      $success_line = set_test_option("SUCCESS_LINE", $i);
> +    $failure_line = set_test_option("FAILURE_LINE", $i);
>      $stop_after_success = set_test_option("STOP_AFTER_SUCCESS", $i);
>      $stop_after_failure = set_test_option("STOP_AFTER_FAILURE", $i);
>      $stop_test_after = set_test_option("STOP_TEST_AFTER", $i);
> diff --git a/tools/testing/ktest/sample.conf b/tools/testing/ktest/sample.conf
> index 36521c8..98df23c 100644
> --- a/tools/testing/ktest/sample.conf
> +++ b/tools/testing/ktest/sample.conf
> @@ -421,6 +421,13 @@
>  # (default "login:")
>  #SUCCESS_LINE = login:
>  
> +# Line indicating a failed boot. This is what the line contains, not
> +# the entire line. If you need the entire line to match, then use
> +# regular expression syntax like with SUCCESS_LINE. This is useful
> +# for testing boot issues that don't stop the boot from completing.
> +# (default undefined)
> +#FAILURE_LINE = WARNING
> +
>  # In case the console constantly fills the screen, having
>  # a specified time to stop the test after success is recommended.
>  # (in seconds)

[PATCH 10/10 v2] ktest: Introduce FAILURE_LINE

[Andrew Jones]

This is the counterpart to SUCCESS_LINE. In some cases the boot will
succeed, but the console will display a message that means that the
boot test failed (such as a warning) while it boots. Use this option
to detect this and fail the test.
---
 tools/testing/ktest/ktest.pl    |    7 +++++++
 tools/testing/ktest/sample.conf |    7 +++++++
 2 files changed, 14 insertions(+), 0 deletions(-)

diff --git a/tools/testing/ktest/ktest.pl b/tools/testing/ktest/ktest.pl
index 6ea9fc9..918fd17 100755
--- a/tools/testing/ktest/ktest.pl
+++ b/tools/testing/ktest/ktest.pl
@@ -122,6 +122,7 @@ my $booted_timeout;
 my $detect_triplefault;
 my $console;
 my $success_line;
+my $failure_line;
 my $stop_after_success;
 my $stop_after_failure;
 my $stop_test_after;
@@ -1015,6 +1016,11 @@ sub monitor {
 	# we are not guaranteed to get a full line
 	$full_line .= $line;
 
+	if (defined($failure_line) && $full_line =~ /$failure_line/) {
+	    $bug = 1;
+	    $failure_start = time;
+	}
+
 	if ($full_line =~ /$success_line/) {
 	    $booted = 1;
 	    $success_start = time;
@@ -2872,6 +2878,7 @@ for (my $i = 1; $i <= $opt{"NUM_TESTS"}; $i++) {
     $console = set_test_option("CONSOLE", $i);
     $detect_triplefault = set_test_option("DETECT_TRIPLE_FAULT", $i);
     $success_line = set_test_option("SUCCESS_LINE", $i);
+    $failure_line = set_test_option("FAILURE_LINE", $i);
     $stop_after_success = set_test_option("STOP_AFTER_SUCCESS", $i);
     $stop_after_failure = set_test_option("STOP_AFTER_FAILURE", $i);
     $stop_test_after = set_test_option("STOP_TEST_AFTER", $i);
diff --git a/tools/testing/ktest/sample.conf b/tools/testing/ktest/sample.conf
index 99bc7d3..add9734 100644
--- a/tools/testing/ktest/sample.conf
+++ b/tools/testing/ktest/sample.conf
@@ -421,6 +421,13 @@
 # (default "login:")
 #SUCCESS_LINE = login:
 
+# Line indicating a failed boot. This is what the line contains, not
+# the entire line. If you need the entire line to match, then use
+# regular expression syntax like with SUCCESS_LINE. This is useful
+# for testing boot issues that don't stop the boot from completing.
+# (default undefined)
+#FAILURE_LINE = WARNING
+
 # In case the console constantly fills the screen, having
 # a specified time to stop the test after success is recommended.
 # (in seconds)
-- 
1.7.4.1

Re: [KTEST PATCH 00/10] collection of ktest patches

[Steven Rostedt]

On Fri, 2011-08-12 at 15:32 +0200, Andrew Jones wrote:
> More than once I've had to drive git-bisect to bisect boot issues for
> Xen guests. The last time I needed to do so I decided to give ktest a
> whirl. While whirling I wrote a couple patches and added a few features.
> This patchset is the collection.

I do like your changes and I'll start queuing them up soon. Probably
need to rename the PASS_COUNT though. What do you think of RERUN?

Anyway, I only post to Linus during the merge window (except for bug
fixes), which means they wont be in mainline until 3.2.

The best thing about this patchset, is that it proves that ktest is used
by more than 1 person. It's doubled its usage statistics!

-- Steve

> 
> Andrew Jones (10):
>   ktest: create outputdir, if it doesn't exist
>   ktest: small cleanup
>   ktest: factor reboot code
>   ktest: Introduce RESET_TIME
>   ktest: refactor monitor/boot/test code
>   ktest: make start_monitor_and_boot true to its name
>   ktest: Introduce PASS_COUNT
>   ktest: test faster, put REBOOT_ON_SUCCESS to more work
>   ktest: test faster, favor rsync over the tarball method
>   ktest: Introduce FAILURE_LINE
> 
>  tools/testing/ktest/ktest.pl    |  208 +++++++++++++++++++++++----------------
>  tools/testing/ktest/sample.conf |   23 ++++-
>  2 files changed, 147 insertions(+), 84 deletions(-)
> 

Re: [KTEST PATCH 00/10] collection of ktest patches

[Andrew Jones]

On Fri, Aug 12, 2011 at 01:44:18PM -0400, Steven Rostedt wrote:
> On Fri, 2011-08-12 at 15:32 +0200, Andrew Jones wrote:
> > More than once I've had to drive git-bisect to bisect boot issues for
> > Xen guests. The last time I needed to do so I decided to give ktest a
> > whirl. While whirling I wrote a couple patches and added a few features.
> > This patchset is the collection.
> 
> I do like your changes and I'll start queuing them up soon. Probably
> need to rename the PASS_COUNT though. What do you think of RERUN?

Thanks! I've sent the rerun patch.

> 
> Anyway, I only post to Linus during the merge window (except for bug
> fixes), which means they wont be in mainline until 3.2.
>

no problem
 
> The best thing about this patchset, is that it proves that ktest is used
> by more than 1 person. It's doubled its usage statistics!

lol

> 
> -- Steve
> 
> > 
> > Andrew Jones (10):
> >   ktest: create outputdir, if it doesn't exist
> >   ktest: small cleanup
> >   ktest: factor reboot code
> >   ktest: Introduce RESET_TIME
> >   ktest: refactor monitor/boot/test code
> >   ktest: make start_monitor_and_boot true to its name
> >   ktest: Introduce PASS_COUNT
> >   ktest: test faster, put REBOOT_ON_SUCCESS to more work
> >   ktest: test faster, favor rsync over the tarball method
> >   ktest: Introduce FAILURE_LINE
> > 
> >  tools/testing/ktest/ktest.pl    |  208 +++++++++++++++++++++++----------------
> >  tools/testing/ktest/sample.conf |   23 ++++-
> >  2 files changed, 147 insertions(+), 84 deletions(-)
> > 
> 
>