[LTP] Add option to skip tests

[LTP] Add option to skip tests

[Paul Larson]

[-- Attachment #1: Type: text/plain, Size: 1132 bytes --]

One problem that LTP has long suffered is that certain tests are known
to be broken, sometimes for long periods of time.  In an ideal world, I
think it best that these should be skipped, or at least removed from the
default list of tests to run, but this isn't always practical due to a
number of reasons.  Of course, another option is that a 'whitelist' of
tests to run can be created and passed to runltp, but this becomes hard
to maintain as there are often new tests that you may want to keep up
with, and with each release of LTP, the list would have to be reexamined.

So I wanted to see what people thought about a simple option, -S, to
specify a list of tests that you want to blacklist from the current run.
 This is handy if you have an automated testing environment with an
architecture that doesn't always get great attention and a list of tests
that you know to be broken on that arch.

The skipfile is simply a list of test names, one per line.

Thanks,
Paul Larson
---

Add a -S option to runltp that allows the user to specify a SKIPFILE of
tests to skip.

Signed-off-by: Paul Larson <paul.larson@canonical.com>
---

[-- Attachment #2: runltp-skipfile.patch --]
[-- Type: text/x-patch, Size: 1651 bytes --]

diff --git a/runltp b/runltp
index c26f4ab..746a1ec 100755
--- a/runltp
+++ b/runltp
@@ -141,6 +141,7 @@ usage()
     -q              Print less verbose output to screen.
     -r LTPROOT      Fully qualified path where testsuite is installed.
     -s PATTERN      Only run test cases which match PATTERN.
+    -S SKIPFILE     Skip tests specified in SKIPFILE
     -t DURATION     Execute the testsuite for given duration. Examples:
                       -t 60s = 60 seconds
                       -t 45m = 45 minutes
@@ -193,7 +194,7 @@ main()
     local DEFAULT_FILE_NAME_GENERATION_TIME=`date +"%Y_%b_%d-%Hh_%Mm_%Ss"`
     version_date=`head -n 1 $LTPROOT/ChangeLog`
 
-    while getopts a:c:C:d:D:f:ehi:g:l:m:Nno:pqr:s:t:T:vw:x:b:B: arg
+    while getopts a:c:C:d:D:f:ehi:g:l:m:Nno:pqr:s:S:t:T:vw:x:b:B: arg
     do  case $arg in
         a)  EMAIL_TO=$OPTARG
             ALT_EMAIL_OUT=1;;
@@ -353,6 +354,8 @@ main()
         r)  LTPROOT=$OPTARG;;
     
         s)  TAG_RESTRICT_STRING=$OPTARG;;
+
+	S)  SKIPFILE=$OPTARG;;
     
         t)  # In case you want to specify the time 
             # to run from the command line 
@@ -630,6 +633,14 @@ main()
         mv -f ${TMP}/alltests ${TMP}/alltests.orig
 	    grep $TAG_RESTRICT_STRING ${TMP}/alltests.orig > ${TMP}/alltests #Not worth checking return codes for this case
     fi
+
+    # Blacklist or skip tests if a SKIPFILE was specified with -S
+    if [ -n "$SKIPFILE" ]
+    then
+        for file in $( cat $SKIPFILE ); do
+            sed -i "/$file/d" ${TMP}/alltests
+        done
+    fi
     
     # check for required users and groups
     ${LTPROOT}/IDcheck.sh &>/dev/null || \

[-- Attachment #3: Type: text/plain, Size: 355 bytes --]

------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day 
trial. Simplify your report design, integration and deployment - and focus on 
what you do best, core application coding. Discover what's new with 
Crystal Reports now.  http://p.sf.net/sfu/bobj-july

[-- Attachment #4: Type: text/plain, Size: 155 bytes --]

_______________________________________________
Ltp-list mailing list
Ltp-list@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/ltp-list

Re: [LTP] Add option to skip tests

[Subrata Modak]

Hi Paul,

On Wed, 2009-08-12 at 16:50 -0500, Paul Larson wrote: 
> One problem that LTP has long suffered is that certain tests are known
> to be broken, sometimes for long periods of time.  In an ideal world, I
> think it best that these should be skipped, or at least removed from the
> default list of tests to run, but this isn't always practical due to a
> number of reasons.  Of course, another option is that a 'whitelist' of
> tests to run can be created and passed to runltp, but this becomes hard
> to maintain as there are often new tests that you may want to keep up
> with, and with each release of LTP, the list would have to be reexamined.
> 
> So I wanted to see what people thought about a simple option, -S, to
> specify a list of tests that you want to blacklist from the current run.
>  This is handy if you have an automated testing environment with an
> architecture that doesn't always get great attention and a list of tests
> that you know to be broken on that arch.

This indeed is a good approach. Each test engineer can maintain a list
of tests in afile which he is not interested to run or does not suit
his/her test environment. Then doing

./runltp -S <filename>

he can just skip to run these subset of test.

However, i am not able to achieve the same results post application of
this patch. I created my own version of runtest/math and runtest/math.x.
I want runltp to skip all tests mentioned mentioned in math.x and run
the remaining mentioned in math. However when i run the following i see
runltp still runs all the tests inside runtest/math:

$ cat runtest/math.x 
abs01 abs01
atof01 atof01
fptest01 fptest01
fptest02 fptest02

$ cat runtest/math
abs01 abs01
atof01 atof01
fptest01 fptest01
fptest02 fptest02
nextafter01 nextafter01

$./runltp -S math.x -f math

<<<test_start>>>
tag=abs01 stime=1250180349
cmdline="abs01"
contacts=""
analysis=exit
<<<test_output>>>
abs01       1  TPASS  :  Test passed
abs01       2  TPASS  :  Test passed
abs01       3  TPASS  :  Test passed
<<<execution_status>>>
initiation_status="ok"
duration=0 termination_type=exited termination_id=0 corefile=no
cutime=0 cstime=0
<<<test_end>>>
<<<test_start>>>
tag=atof01 stime=1250180349
cmdline="atof01"
contacts=""
analysis=exit
<<<test_output>>>
atof01      1  TPASS  :  Test passed
atof01      2  TPASS  :  Test passed
atof01      3  TPASS  :  Test passed
atof01      4  TPASS  :  Test passed
<<<execution_status>>>
initiation_status="ok"
duration=0 termination_type=exited termination_id=0 corefile=no
cutime=0 cstime=0
<<<test_end>>>
<<<test_start>>>
tag=fptest01 stime=1250180349
cmdline="fptest01"
contacts=""
analysis=exit
<<<test_output>>>
fptest01    1  TPASS  :  PASS
<<<execution_status>>>
initiation_status="ok"
duration=0 termination_type=exited termination_id=0 corefile=no
cutime=0 cstime=0
<<<test_end>>>
<<<test_start>>>
tag=fptest02 stime=1250180349
cmdline="fptest02"
contacts=""
analysis=exit
<<<test_output>>>
fptest02    1  TPASS  :  PASS
<<<execution_status>>>
initiation_status="ok"
duration=0 termination_type=exited termination_id=0 corefile=no
cutime=1 cstime=0
<<<test_end>>>
<<<test_start>>>
tag=nextafter01 stime=1250180349
cmdline="nextafter01"
contacts=""
analysis=exit
<<<test_output>>>
nextafter01    1  TPASS  :  Test passed
nextafter01    2  TPASS  :  Test passed
nextafter01    3  TPASS  :  Test passed
incrementing stop
<<<execution_status>>>
initiation_status="ok"
duration=0 termination_type=exited termination_id=0 corefile=no
cutime=0 cstime=0
<<<test_end>>>

I thought that math.x should just have the name of the test not to be
run. So, i just retained only the tag_names, and removed anything else,
but still i get the following output:

$ cat runtest/math.x.2 
abs01 
atof01
fptest01
fptest02

$ ./runltp -S math.x.2 -f math

<<<test_start>>>
tag=abs01 stime=1250180546
cmdline="abs01"
contacts=""
analysis=exit
<<<test_output>>>
abs01       1  TPASS  :  Test passed
abs01       2  TPASS  :  Test passed
abs01       3  TPASS  :  Test passed
<<<execution_status>>>
initiation_status="ok"
duration=0 termination_type=exited termination_id=0 corefile=no
cutime=0 cstime=0
<<<test_end>>>
<<<test_start>>>
tag=atof01 stime=1250180546
cmdline="atof01"
contacts=""
analysis=exit
<<<test_output>>>
atof01      1  TPASS  :  Test passed
atof01      2  TPASS  :  Test passed
atof01      3  TPASS  :  Test passed
atof01      4  TPASS  :  Test passed
<<<execution_status>>>
initiation_status="ok"
duration=0 termination_type=exited termination_id=0 corefile=no
cutime=0 cstime=0
<<<test_end>>>
<<<test_start>>>
tag=fptest01 stime=1250180546
cmdline="fptest01"
contacts=""
analysis=exit
<<<test_output>>>
fptest01    1  TPASS  :  PASS
<<<execution_status>>>
initiation_status="ok"
duration=0 termination_type=exited termination_id=0 corefile=no
cutime=0 cstime=0
<<<test_end>>>
<<<test_start>>>
tag=fptest02 stime=1250180546
cmdline="fptest02"
contacts=""
analysis=exit
<<<test_output>>>
fptest02    1  TPASS  :  PASS
<<<execution_status>>>
initiation_status="ok"
duration=0 termination_type=exited termination_id=0 corefile=no
cutime=0 cstime=0
<<<test_end>>>
<<<test_start>>>
tag=nextafter01 stime=1250180546
cmdline="nextafter01"
contacts=""
analysis=exit
<<<test_output>>>
nextafter01    1  TPASS  :  Test passed
nextafter01    2  TPASS  :  Test passed
nextafter01    3  TPASS  :  Test passed
incrementing stop
<<<execution_status>>>
initiation_status="ok"
duration=0 termination_type=exited termination_id=0 corefile=no
cutime=0 cstime=0
<<<test_end>>>

Is there something which i am missing out ?

Regards--
Subrata

> 
> The skipfile is simply a list of test names, one per line.
> 
> Thanks,
> Paul Larson
> ---
> 
> Add a -S option to runltp that allows the user to specify a SKIPFILE of
> tests to skip.
> 
> Signed-off-by: Paul Larson <paul.larson@canonical.com>
> ---
> ------------------------------------------------------------------------------
> Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day 
> trial. Simplify your report design, integration and deployment - and focus on 
> what you do best, core application coding. Discover what's new with 
> Crystal Reports now.  http://p.sf.net/sfu/bobj-july
> _______________________________________________ Ltp-list mailing list Ltp-list@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/ltp-list


------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day 
trial. Simplify your report design, integration and deployment - and focus on 
what you do best, core application coding. Discover what's new with 
Crystal Reports now.  http://p.sf.net/sfu/bobj-july
_______________________________________________
Ltp-list mailing list
Ltp-list@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/ltp-list

Re: [LTP] Add option to skip tests

[Paul Larson]

[-- Attachment #1: Type: text/plain, Size: 359 bytes --]

You were not providing the full path.  Here's a new version that should
let you either provide absolute or relative path (including no patch
assuming the file is in LTPROOT).  See if this works better for you.


Add a -S option to runltp that allows the user to specify a SKIPFILE of
tests to skip.

Signed-off-by: Paul Larson <paul.larson@canonical.com>
---

[-- Attachment #2: runltp-skipfile-2.patch --]
[-- Type: text/x-patch, Size: 1765 bytes --]

diff --git a/runltp b/runltp
index c26f4ab..15da5ab 100755
--- a/runltp
+++ b/runltp
@@ -141,6 +141,7 @@ usage()
     -q              Print less verbose output to screen.
     -r LTPROOT      Fully qualified path where testsuite is installed.
     -s PATTERN      Only run test cases which match PATTERN.
+    -S SKIPFILE     Skip tests specified in SKIPFILE
     -t DURATION     Execute the testsuite for given duration. Examples:
                       -t 60s = 60 seconds
                       -t 45m = 45 minutes
@@ -193,7 +194,7 @@ main()
     local DEFAULT_FILE_NAME_GENERATION_TIME=`date +"%Y_%b_%d-%Hh_%Mm_%Ss"`
     version_date=`head -n 1 $LTPROOT/ChangeLog`
 
-    while getopts a:c:C:d:D:f:ehi:g:l:m:Nno:pqr:s:t:T:vw:x:b:B: arg
+    while getopts a:c:C:d:D:f:ehi:g:l:m:Nno:pqr:s:S:t:T:vw:x:b:B: arg
     do  case $arg in
         a)  EMAIL_TO=$OPTARG
             ALT_EMAIL_OUT=1;;
@@ -353,6 +354,12 @@ main()
         r)  LTPROOT=$OPTARG;;
     
         s)  TAG_RESTRICT_STRING=$OPTARG;;
+
+	S)  case $OPTARG in
+            /*)
+                SKIPFILE=$OPTARG;;
+            *)
+                SKIPFILE="$LTPROOT/$OPTARG";;
     
         t)  # In case you want to specify the time 
             # to run from the command line 
@@ -630,6 +637,14 @@ main()
         mv -f ${TMP}/alltests ${TMP}/alltests.orig
 	    grep $TAG_RESTRICT_STRING ${TMP}/alltests.orig > ${TMP}/alltests #Not worth checking return codes for this case
     fi
+
+    # Blacklist or skip tests if a SKIPFILE was specified with -S
+    if [ -n "$SKIPFILE" ]
+    then
+        for file in $( cat $SKIPFILE ); do
+            sed -i "/$file/d" ${TMP}/alltests
+        done
+    fi
     
     # check for required users and groups
     ${LTPROOT}/IDcheck.sh &>/dev/null || \

[-- Attachment #3: Type: text/plain, Size: 355 bytes --]

------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day 
trial. Simplify your report design, integration and deployment - and focus on 
what you do best, core application coding. Discover what's new with 
Crystal Reports now.  http://p.sf.net/sfu/bobj-july

[-- Attachment #4: Type: text/plain, Size: 155 bytes --]

_______________________________________________
Ltp-list mailing list
Ltp-list@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/ltp-list

Re: [LTP] Add option to skip tests

[Garrett Cooper]

On Thu, Aug 13, 2009 at 10:54 AM, Paul Larson<paul.larson@canonical.com> wrote:
> You were not providing the full path.  Here's a new version that should
> let you either provide absolute or relative path (including no patch
> assuming the file is in LTPROOT).  See if this works better for you.
>
>
> Add a -S option to runltp that allows the user to specify a SKIPFILE of
> tests to skip.
>
> Signed-off-by: Paul Larson <paul.larson@canonical.com>

    Cool innovation Paul! I see potential for this providing us with
the shims necessary to automatically generate which testcases need to
be skipped after running autoconf and/or the compile end :). Also, I'd
say that if the basename is the same as the specified skipfile, and
the skipfile isn't in the base directory, then it should be assumed to
be in a predefined directory. Otherwise, it's not in the default
directory. That way we don't invite any confusing defaults...
Thanks,
-Garrett

------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day 
trial. Simplify your report design, integration and deployment - and focus on 
what you do best, core application coding. Discover what's new with 
Crystal Reports now.  http://p.sf.net/sfu/bobj-july
_______________________________________________
Ltp-list mailing list
Ltp-list@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/ltp-list

Re: [LTP] Add option to skip tests

[Subrata Modak]

Wow !! Thatś great and it works fine now.

On Thu, 2009-08-13 at 12:54 -0500, Paul Larson wrote:
> You were not providing the full path.  Here's a new version that
> should
> let you either provide absolute or relative path (including no patch
> assuming the file is in LTPROOT).  See if this works better for you.
> 
> 
> Add a -S option to runltp that allows the user to specify a SKIPFILE
> of
> tests to skip.
> 
> Signed-off-by: Paul Larson <paul.larson@canonical.com>
> ---
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> differences
> between files
> attachment
> (runltp-skipfile-2.patch)
> 
> 
> diff --git a/runltp b/runltp
> index c26f4ab..15da5ab 100755
> --- a/runltp
> +++ b/runltp
> @@ -141,6 +141,7 @@ usage()
>      -q              Print less verbose output to screen.
>      -r LTPROOT      Fully qualified path where testsuite is
> installed.
>      -s PATTERN      Only run test cases which match PATTERN.
> +    -S SKIPFILE     Skip tests specified in SKIPFILE
>      -t DURATION     Execute the testsuite for given duration.
> Examples:
>                        -t 60s = 60 seconds
>                        -t 45m = 45 minutes
> @@ -193,7 +194,7 @@ main()
>      local DEFAULT_FILE_NAME_GENERATION_TIME=`date +"%Y_%b_%d-%Hh_%Mm_
> %Ss"`
>      version_date=`head -n 1 $LTPROOT/ChangeLog`
> 
> -    while getopts a:c:C:d:D:f:ehi:g:l:m:Nno:pqr:s:t:T:vw:x:b:B: arg
> +    while getopts a:c:C:d:D:f:ehi:g:l:m:Nno:pqr:s:S:t:T:vw:x:b:B: arg
>      do  case $arg in
>          a)  EMAIL_TO=$OPTARG
>              ALT_EMAIL_OUT=1;;
> @@ -353,6 +354,12 @@ main()
>          r)  LTPROOT=$OPTARG;;
>      
>          s)  TAG_RESTRICT_STRING=$OPTARG;;
> +
> +       S)  case $OPTARG in
> +            /*)
> +                SKIPFILE=$OPTARG;;
> +            *)
> +                SKIPFILE="$LTPROOT/$OPTARG";;

We just needed an:
+ esac ;;
here.

Anyways this option will be used in LTP from now onwards.

Regards--
Subrata

>      
>          t)  # In case you want to specify the time 
>              # to run from the command line 
> @@ -630,6 +637,14 @@ main()
>          mv -f ${TMP}/alltests ${TMP}/alltests.orig
>             grep $TAG_RESTRICT_STRING ${TMP}/alltests.orig >
> ${TMP}/alltests #Not worth checking return codes for this case
>      fi
> +
> +    # Blacklist or skip tests if a SKIPFILE was specified with -S
> +    if [ -n "$SKIPFILE" ]
> +    then
> +        for file in $( cat $SKIPFILE ); do
> +            sed -i "/$file/d" ${TMP}/alltests
> +        done
> +    fi
>      
>      # check for required users and groups
>      ${LTPROOT}/IDcheck.sh &>/dev/null || \
> 


------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day 
trial. Simplify your report design, integration and deployment - and focus on 
what you do best, core application coding. Discover what's new with 
Crystal Reports now.  http://p.sf.net/sfu/bobj-july
_______________________________________________
Ltp-list mailing list
Ltp-list@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/ltp-list