[PATCH 0/4] More test suite enchancements

[PATCH 0/4] More test suite enchancements

[Jes.Sorensen]

From: Jes Sorensen <Jes.Sorensen@redhat.com>

Hi,

Here's another set of test suite enhancements. In particular I have
added --logdir and --save-logs, plus --no-error  which allows for one
to run all the tests, without stopping after the first error. This
makes it easier to run the test suite in an automated fashion and
getting a report of all the tests that may have failed. I also did a
bit to make the --help output more useful.

Cheers,
Jes

Jes Sorensen (4):
  Add support for saving log files in test script
  Add --no-error argument to 'test'
  Improve --help message from test
  Move setup code to a function and introduce matching cleanup argument

 test |  113 ++++++++++++++++++++++++++++++++++++++++++++++++------------------
 1 files changed, 82 insertions(+), 31 deletions(-)

[PATCH 1/4] Add support for saving log files in test script

[Jes.Sorensen]

From: Jes Sorensen <Jes.Sorensen@redhat.com>

--logdir= specifies where to save, if different from default, and
--save-logs tells test to save all log files.

Signed-off-by: Jes Sorensen <Jes.Sorensen@redhat.com>
---
 test |   42 ++++++++++++++++++++++++++++++++++++------
 1 files changed, 36 insertions(+), 6 deletions(-)

diff --git a/test b/test
index 7882586..c7eb069 100755
--- a/test
+++ b/test
@@ -16,6 +16,10 @@ then
    echo >&2 "test: $mdadm isn't usable."
 fi
 
+testdir="tests"
+logdir="$testdir/logs"
+logsave=0
+
 # Check whether to run multipath tests
 modprobe multipath 2> /dev/null
 if grep -s 'Personalities : .*multipath' > /dev/null /proc/mdstat ; then
@@ -204,6 +208,7 @@ rotest() {
 
 do_test() {
   _script=$1
+  _basename=`basename $_script`
   if [ -f "$_script" ]
   then
     rm -f $targetdir/stderr
@@ -215,8 +220,17 @@ do_test() {
     # namespace, but cannot change it.
     echo -ne "$_script... "
     if ( set -ex ; . $_script )  2> $targetdir/log
-    then echo "succeeded"
-    else echo "FAILED - see $targetdir/log for details"
+    then
+      echo "succeeded"
+      _fail=0
+    else
+      echo "FAILED - see $targetdir/log for details"
+      _fail=1
+    fi
+    if [ "$savelogs" == "1" ]; then
+      cp $targetdir/log $logdir/$_basename.log
+    fi
+    if [ "$_fail" == "1" ]; then
       exit 1
     fi
   fi
@@ -224,7 +238,7 @@ do_test() {
 
 do_help() {
   echo "Usage: "
-  echo " $0 [--tests=<test1,test2,..>] [--disable-multipath] [setup] [prefix]"
+  echo " $0 [--tests=<test1,test2,..>] [--disable-multipath] [--logdir] [--save-logs] [setup] [prefix]"
 }
 
 parse_args() {
@@ -241,6 +255,18 @@ parse_args() {
     --tests=*)
       TESTLIST=`expr "x$i" : 'x[^=]*=\(.*\)' | sed -e 's/,/ /g'`
       ;;
+    --logdir=*)
+      logdir=`expr "x$i" : 'x[^=]*=\(.*\)'`
+      ;;
+    --save-logs)
+      savelogs=1
+      if [ ! -d $logdir ] ; then
+        mkdir $logdir
+        if [ $? -ne 0 ] ; then
+	  exit 1;
+	fi
+      fi
+      ;;
     --disable-multipath)
       unset MULTIPATH
       ;;
@@ -259,13 +285,17 @@ done
 
 parse_args $@
 
+if [ "$savelogs" == "1" ]; then
+  echo "Saving logs to $logdir"
+fi
+
 if [ "x$TESTLIST" != "x" ]; then
-  for script in $TESTLIST
+  for script in $testdir/$TESTLIST
   do
-    do_test tests/$script
+    do_test $script
   done
 else
-  for script in tests/$prefix tests/$prefix*[^~]
+  for script in $testdir/$prefix $testdir/$prefix*[^~]
   do
     do_test $script
   done
-- 
1.7.1

[PATCH 2/4] Add --no-error argument to 'test'

[Jes.Sorensen]

From: Jes Sorensen <Jes.Sorensen@redhat.com>

This allows the test suite to run to completion even if one test
fails.

Signed-off-by: Jes Sorensen <Jes.Sorensen@redhat.com>
---
 test |    8 ++++++--
 1 files changed, 6 insertions(+), 2 deletions(-)

diff --git a/test b/test
index c7eb069..a0561ab 100755
--- a/test
+++ b/test
@@ -19,6 +19,7 @@ fi
 testdir="tests"
 logdir="$testdir/logs"
 logsave=0
+exitonerror=1
 
 # Check whether to run multipath tests
 modprobe multipath 2> /dev/null
@@ -230,7 +231,7 @@ do_test() {
     if [ "$savelogs" == "1" ]; then
       cp $targetdir/log $logdir/$_basename.log
     fi
-    if [ "$_fail" == "1" ]; then
+    if [ "$_fail" == "1" -a $"exitonerror" == "1" ]; then
       exit 1
     fi
   fi
@@ -238,7 +239,7 @@ do_test() {
 
 do_help() {
   echo "Usage: "
-  echo " $0 [--tests=<test1,test2,..>] [--disable-multipath] [--logdir] [--save-logs] [setup] [prefix]"
+  echo " $0 [--tests=<test1,test2,..>] [--disable-multipath] [--logdir] [--save-logs] [--no-error] [setup] [prefix]"
 }
 
 parse_args() {
@@ -267,6 +268,9 @@ parse_args() {
 	fi
       fi
       ;;
+    --no-error)
+      exit_on_error=0
+      ;;
     --disable-multipath)
       unset MULTIPATH
       ;;
-- 
1.7.1

[PATCH 3/4] Improve --help message from test

[Jes.Sorensen]

From: Jes Sorensen <Jes.Sorensen@redhat.com>

Signed-off-by: Jes Sorensen <Jes.Sorensen@redhat.com>
---
 test |   11 +++++++++--
 1 files changed, 9 insertions(+), 2 deletions(-)

diff --git a/test b/test
index a0561ab..8e06ae2 100755
--- a/test
+++ b/test
@@ -238,8 +238,15 @@ do_test() {
 }
 
 do_help() {
-  echo "Usage: "
-  echo " $0 [--tests=<test1,test2,..>] [--disable-multipath] [--logdir] [--save-logs] [--no-error] [setup] [prefix]"
+  echo "Usage: $0 [options]"
+  echo " Options:"
+  echo "    --tests=<test1,test2,..>    Comma separated list of tests to run"
+  echo "    --disable-multipath         Disable any tests involving multipath"
+  echo "    --logdir=<directory>        Directory to save logfiles in"
+  echo "    --save-logs                 Save all logs in <logdir>"
+  echo "    --no-error                  Don't stop on error, ie. run all tests"
+  echo "    setup                       Setup test environment and exit"
+  echo "    <prefix>                    Run tests with <prefix>"
 }
 
 parse_args() {
-- 
1.7.1

[PATCH 4/4] Move setup code to a function and introduce matching cleanup argument

[Jes.Sorensen]

From: Jes Sorensen <Jes.Sorensen@redhat.com>

Signed-off-by: Jes Sorensen <Jes.Sorensen@redhat.com>
---
 test |   60 +++++++++++++++++++++++++++++++++++-------------------------
 1 files changed, 35 insertions(+), 25 deletions(-)

diff --git a/test b/test
index 8e06ae2..9532ea7 100755
--- a/test
+++ b/test
@@ -68,33 +68,35 @@ cleanup() {
 	done
 }
 
-trap cleanup 0 1 2 3 15
+do_setup() {
+  trap cleanup 0 1 2 3 15
 
-devlist=
-for d in 0 1 2 3 4 5 6 7 8 9 10 11 12
-do
-   sz=$size
-   if [ $d -gt 7 ]; then sz=$ddfsize ; fi
-   [ -f $targetdir/mdtest$d ] || dd if=/dev/zero of=$targetdir/mdtest$d count=$sz bs=1K > /dev/null 2>&1
-   [ -b /dev/loop$d ] || mknod /dev/loop$d b 7 $d
-   if [ $d -eq 7 ]
-   then
-     losetup /dev/loop$d $targetdir/mdtest6 # for multipath use
-   else
-     losetup /dev/loop$d $targetdir/mdtest$d
-   fi
-   eval dev$d=/dev/loop$d
-   eval file$d=$targetdir/mdtest$d
-   eval devlist=\"\$devlist \$dev$d\"
+  devlist=
+  for d in 0 1 2 3 4 5 6 7 8 9 10 11 12
+  do
+    sz=$size
+    if [ $d -gt 7 ]; then sz=$ddfsize ; fi
+    [ -f $targetdir/mdtest$d ] || dd if=/dev/zero of=$targetdir/mdtest$d count=$sz bs=1K > /dev/null 2>&1
+    [ -b /dev/loop$d ] || mknod /dev/loop$d b 7 $d
+    if [ $d -eq 7 ]
+    then
+      losetup /dev/loop$d $targetdir/mdtest6 # for multipath use
+    else
+      losetup /dev/loop$d $targetdir/mdtest$d
+    fi
+    eval dev$d=/dev/loop$d
+    eval file$d=$targetdir/mdtest$d
+    eval devlist=\"\$devlist \$dev$d\"
    #" <-- add this quote to un-confuse vim syntax highlighting
-done
-path0=$dev6
-path1=$dev7
+  done
+  path0=$dev6
+  path1=$dev7
 
-ulimit -c unlimited
-[ -f /proc/mdstat ] || modprobe md_mod
-echo 2000 > /proc/sys/dev/raid/speed_limit_max
-echo 0 > /sys/module/md_mod/parameters/start_ro
+  ulimit -c unlimited
+  [ -f /proc/mdstat ] || modprobe md_mod
+  echo 2000 > /proc/sys/dev/raid/speed_limit_max
+  echo 0 > /sys/module/md_mod/parameters/start_ro
+}
 
 # mdadm always adds --quiet, and we want to see any unexpected messages
 mdadm() {
@@ -231,7 +233,7 @@ do_test() {
     if [ "$savelogs" == "1" ]; then
       cp $targetdir/log $logdir/$_basename.log
     fi
-    if [ "$_fail" == "1" -a $"exitonerror" == "1" ]; then
+    if [ "$_fail" == "1" -a "$exitonerror" == "1" ]; then
       exit 1
     fi
   fi
@@ -246,6 +248,7 @@ do_help() {
   echo "    --save-logs                 Save all logs in <logdir>"
   echo "    --no-error                  Don't stop on error, ie. run all tests"
   echo "    setup                       Setup test environment and exit"
+  echo "    cleanup                     Cleanup test environment"
   echo "    <prefix>                    Run tests with <prefix>"
 }
 
@@ -258,8 +261,13 @@ parse_args() {
       ;;
     setup)
       echo "mdadm test environment setup"
+      do_setup
       trap 0; exit 0
       ;;
+    cleanup)
+      cleanup
+      exit 0
+      ;;
     --tests=*)
       TESTLIST=`expr "x$i" : 'x[^=]*=\(.*\)' | sed -e 's/,/ /g'`
       ;;
@@ -296,6 +304,8 @@ done
 
 parse_args $@
 
+do_setup
+
 if [ "$savelogs" == "1" ]; then
   echo "Saving logs to $logdir"
 fi
-- 
1.7.1

Re: [PATCH 1/4] Add support for saving log files in test script

[NeilBrown]

[-- Attachment #1: Type: text/plain, Size: 3009 bytes --]

On Fri, 25 May 2012 17:24:05 +0200 Jes.Sorensen@redhat.com wrote:

> From: Jes Sorensen <Jes.Sorensen@redhat.com>
> 
> --logdir= specifies where to save, if different from default, and
> --save-logs tells test to save all log files.
> 
> Signed-off-by: Jes Sorensen <Jes.Sorensen@redhat.com>
> ---
>  test |   42 ++++++++++++++++++++++++++++++++++++------
>  1 files changed, 36 insertions(+), 6 deletions(-)
> 
> diff --git a/test b/test
> index 7882586..c7eb069 100755
> --- a/test
> +++ b/test
> @@ -16,6 +16,10 @@ then
>     echo >&2 "test: $mdadm isn't usable."
>  fi
>  
> +testdir="tests"
> +logdir="$testdir/logs"
> +logsave=0
> +
>  # Check whether to run multipath tests
>  modprobe multipath 2> /dev/null
>  if grep -s 'Personalities : .*multipath' > /dev/null /proc/mdstat ; then
> @@ -204,6 +208,7 @@ rotest() {
>  
>  do_test() {
>    _script=$1
> +  _basename=`basename $_script`
>    if [ -f "$_script" ]
>    then
>      rm -f $targetdir/stderr
> @@ -215,8 +220,17 @@ do_test() {
>      # namespace, but cannot change it.
>      echo -ne "$_script... "
>      if ( set -ex ; . $_script )  2> $targetdir/log
> -    then echo "succeeded"
> -    else echo "FAILED - see $targetdir/log for details"
> +    then
> +      echo "succeeded"
> +      _fail=0
> +    else
> +      echo "FAILED - see $targetdir/log for details"
> +      _fail=1
> +    fi
> +    if [ "$savelogs" == "1" ]; then
> +      cp $targetdir/log $logdir/$_basename.log
> +    fi
> +    if [ "$_fail" == "1" ]; then
>        exit 1
>      fi
>    fi
> @@ -224,7 +238,7 @@ do_test() {
>  
>  do_help() {
>    echo "Usage: "
> -  echo " $0 [--tests=<test1,test2,..>] [--disable-multipath] [setup] [prefix]"
> +  echo " $0 [--tests=<test1,test2,..>] [--disable-multipath] [--logdir] [--save-logs] [setup] [prefix]"
>  }
>  
>  parse_args() {
> @@ -241,6 +255,18 @@ parse_args() {
>      --tests=*)
>        TESTLIST=`expr "x$i" : 'x[^=]*=\(.*\)' | sed -e 's/,/ /g'`
>        ;;
> +    --logdir=*)
> +      logdir=`expr "x$i" : 'x[^=]*=\(.*\)'`
> +      ;;
> +    --save-logs)
> +      savelogs=1
> +      if [ ! -d $logdir ] ; then
> +        mkdir $logdir
> +        if [ $? -ne 0 ] ; then
> +	  exit 1;
> +	fi
> +      fi
> +      ;;
>      --disable-multipath)
>        unset MULTIPATH
>        ;;
> @@ -259,13 +285,17 @@ done
>  
>  parse_args $@
>  
> +if [ "$savelogs" == "1" ]; then
> +  echo "Saving logs to $logdir"
> +fi
> +
>  if [ "x$TESTLIST" != "x" ]; then
> -  for script in $TESTLIST
> +  for script in $testdir/$TESTLIST

That's not really right, is it?
I fixed it to instead change...

>    do
> -    do_test tests/$script
> +    do_test $script

..this to
       do_text $testdir/$script

and applied.

Thanks,
NeilBrown


>    done
>  else
> -  for script in tests/$prefix tests/$prefix*[^~]
> +  for script in $testdir/$prefix $testdir/$prefix*[^~]
>    do
>      do_test $script
>    done


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 828 bytes --]

Re: [PATCH 4/4] Move setup code to a function and introduce matching cleanup argument

[NeilBrown]

[-- Attachment #1: Type: text/plain, Size: 548 bytes --]

On Fri, 25 May 2012 17:24:08 +0200 Jes.Sorensen@redhat.com wrote:

> @@ -231,7 +233,7 @@ do_test() {
>      if [ "$savelogs" == "1" ]; then
>        cp $targetdir/log $logdir/$_basename.log
>      fi
> -    if [ "$_fail" == "1" -a $"exitonerror" == "1" ]; then
> +    if [ "$_fail" == "1" -a "$exitonerror" == "1" ]; then
>        exit 1
>      fi
>    fi

I merged this fix back into the previous patch that introduced the bug :-)

Otherwise - all applied,
Thanks.

I'll definitely be using the --no-error option!

NeilBrown


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 828 bytes --]

Re: [PATCH 4/4] Move setup code to a function and introduce matching cleanup argument

[Jes Sorensen]

On 05/28/12 02:56, NeilBrown wrote:
> On Fri, 25 May 2012 17:24:08 +0200 Jes.Sorensen@redhat.com wrote:
> 
>> @@ -231,7 +233,7 @@ do_test() {
>>      if [ "$savelogs" == "1" ]; then
>>        cp $targetdir/log $logdir/$_basename.log
>>      fi
>> -    if [ "$_fail" == "1" -a $"exitonerror" == "1" ]; then
>> +    if [ "$_fail" == "1" -a "$exitonerror" == "1" ]; then
>>        exit 1
>>      fi
>>    fi
> 
> I merged this fix back into the previous patch that introduced the bug :-)
> 
> Otherwise - all applied,
> Thanks.

Sounds good!

> I'll definitely be using the --no-error option!

Excellent, glad it's turning out to be useful. I am trying to get to a
point where I can have the test suite run automatically, though I am
still hitting test errors on both Fedora and RHEL at this point which I
need to figure out first.

Cheers,
Jes