[LTP] [RFC] Showing skipped tests in results?

[LTP] [RFC] Showing skipped tests in results?

[Dan Rue]

Hello!

Before I submit any patches, I wanted to run this idea by the community
for advice.

When we skip tests in LTP using "./runltp -S SKIPFILE", they get removed
from the test list completely, leaving no trace in the results.
Meanwhile, tests that are not able to run due to configuration show up
as a "CONF" in the results file.

From our perspective, this is not ideal because we want to see tests
that are explicitly skipped as skipped in the results. Ideally, tests
that are skipped will still show up in the results as a SKIP.

It looks like there's some precedent in runltp for prepending a string
to a test name that can then later be picked up and interpreted by
individual tests.

Specifically, I noticed that if you use -I ITERATIONS to run tests
multiple times, it will prepend an iteration count and number in
alltests:

$ cat alltests
1_ITERATION_gzip01 gzip_tests.sh
2_ITERATION_gzip01 gzip_tests.sh
3_ITERATION_gzip01 gzip_tests.sh

Which made me wonder if we could use a similar convention for SKIP,
rather than just removing the test from alltests (what currently
happens).

So for example, if you skip gzip01, the alltests file would end up with
something like: SKIP_gzip01 gzip_tests.sh

When I try that, it actually runs gzip01 still. I thought it would
fail. Which means that something is stripping the "SKIP_" (and
throwing it away).

I looked a bit and I'm guessing we'd need to patch lib/parse_opts.c
and then probably stuff in testcases/lib (but maybe not).

Any feedback on the concept or the implementation is really appreciated.
Thank you!

Dan

[LTP] [RFC] Showing skipped tests in results?

[Cyril Hrubis]

Hi!
> When we skip tests in LTP using "./runltp -S SKIPFILE", they get removed
> from the test list completely, leaving no trace in the results.
> Meanwhile, tests that are not able to run due to configuration show up
> as a "CONF" in the results file.
> 
> From our perspective, this is not ideal because we want to see tests
> that are explicitly skipped as skipped in the results. Ideally, tests
> that are skipped will still show up in the results as a SKIP.
> 
> It looks like there's some precedent in runltp for prepending a string
> to a test name that can then later be picked up and interpreted by
> individual tests.
> 
> Specifically, I noticed that if you use -I ITERATIONS to run tests
> multiple times, it will prepend an iteration count and number in
> alltests:
> 
> $ cat alltests
> 1_ITERATION_gzip01 gzip_tests.sh
> 2_ITERATION_gzip01 gzip_tests.sh
> 3_ITERATION_gzip01 gzip_tests.sh
> 
> Which made me wonder if we could use a similar convention for SKIP,
> rather than just removing the test from alltests (what currently
> happens).
> 
> So for example, if you skip gzip01, the alltests file would end up with
> something like: SKIP_gzip01 gzip_tests.sh
> 
> When I try that, it actually runs gzip01 still. I thought it would
> fail. Which means that something is stripping the "SKIP_" (and
> throwing it away).

I think that you misunderstand how things works, the first string on
each line in the alltests file that is parsed by ltp-pan is just test
name that appears in the test result file, it does not affect anything
but that.

I guess that the easiest solution would be replacing the actuall command
that is supposed to be executed with something that just reports
skipped status.

Quick hack would be replacing these with something that just returns the
right exit value:

gzip01 gzip_tests.hs -> gzip01 exit 32

Something like this should mark the gzip01 as TCONF in test results,
would that suffice for your purpose?

This should be as easy as changing the sed command from delete to
replace in the runltp script.

-- 
Cyril Hrubis
chrubis@suse.cz

[LTP] [RFC] Showing skipped tests in results?

[Naresh Kamboju]

On 6 December 2017 at 16:22, Cyril Hrubis <chrubis@suse.cz> wrote:
> Hi!
>> When we skip tests in LTP using "./runltp -S SKIPFILE", they get removed
>> from the test list completely, leaving no trace in the results.
>> Meanwhile, tests that are not able to run due to configuration show up
>> as a "CONF" in the results file.
>>
>> From our perspective, this is not ideal because we want to see tests
>> that are explicitly skipped as skipped in the results. Ideally, tests
>> that are skipped will still show up in the results as a SKIP.
>>
>> It looks like there's some precedent in runltp for prepending a string
>> to a test name that can then later be picked up and interpreted by
>> individual tests.
>>
>> Specifically, I noticed that if you use -I ITERATIONS to run tests
>> multiple times, it will prepend an iteration count and number in
>> alltests:
>>
>> $ cat alltests
>> 1_ITERATION_gzip01 gzip_tests.sh
>> 2_ITERATION_gzip01 gzip_tests.sh
>> 3_ITERATION_gzip01 gzip_tests.sh
>>
>> Which made me wonder if we could use a similar convention for SKIP,
>> rather than just removing the test from alltests (what currently
>> happens).
>>
>> So for example, if you skip gzip01, the alltests file would end up with
>> something like: SKIP_gzip01 gzip_tests.sh
>>
>> When I try that, it actually runs gzip01 still. I thought it would
>> fail. Which means that something is stripping the "SKIP_" (and
>> throwing it away).
>
> I think that you misunderstand how things works, the first string on
> each line in the alltests file that is parsed by ltp-pan is just test
> name that appears in the test result file, it does not affect anything
> but that.
>
> I guess that the easiest solution would be replacing the actuall command
> that is supposed to be executed with something that just reports
> skipped status.
>
> Quick hack would be replacing these with something that just returns the
> right exit value:
>
> gzip01 gzip_tests.hs -> gzip01 exit 32
Not working.
the second argument is expected to be a file.

gzip01 gzip_tests.hs -> gzip01 skip.sh
should work.
We have to create skip.sh inside ltp
skip.sh contains single line exit 32

Which is the right place holder to host skip.sh ?

>
> Something like this should mark the gzip01 as TCONF in test results,
> would that suffice for your purpose?
>
> This should be as easy as changing the sed command from delete to
> replace in the runltp script.


OTOH,
I have noticed one more problem with sed with updated script.


Issues:
1. Want to skip only "testcase" write inside skipfile and run ./ltp-skip.sh

 testcase is replaced by testcase skip.sh ( = TCONF ) which is expected.
 testcase_64 is replaced by testcase skip.sh which is incorrect testcase name
 testcase_16 is replaced by testcase skip.sh which is incorrect testcase name
 testcase_xxxx is replaced by testcase skip.sh which is incorrect testcase name

diff -Narub alltests.org  alltests
--- alltests.org 2017-12-07 22:41:46.373419820 +0530
+++ alltests 2017-12-07 23:54:53.188657734 +0530
@@ -272,8 +272,8 @@
 fcntl33_64 fcntl33_64
 fcntl34 fcntl34
 fcntl34_64 fcntl34_64
-fcntl35 fcntl35
-fcntl35_64 fcntl35_64
+fcntl35 skip.sh
+fcntl35 skip.sh
 fcntl36 fcntl36
 fcntl36_64 fcntl36_64


2. The skipfile have limitation of
testcase_name should be a single word in SKIPFILE

Feature is
We can added comments inside SKIPFILE starting with #
The script will  ignore line starting with #


From c3e77bbaf03e126de15524f817960740a8c94e1b Mon Sep 17 00:00:00 2001
From: Naresh Kamboju <naresh.kamboju@linaro.org>
Date: Thu, 7 Dec 2017 23:14:44 +0530
Subject: [PATCH] runltp: SKIPFILE: skip test cases should be visible as TCONF
 in results

When we skip tests in LTP using "./runltp -S SKIPFILE", they get removed
from the test list completely, leaving no trace in the results.

This patch will add SKIPFILE listed test cases names to results as TCONF
testcase_name -> testcase_name skip.sh

Signed-off-by: Naresh Kamboju <naresh.kamboju@linaro.org>
---
 runltp | 12 +++++++-----
 1 file changed, 7 insertions(+), 5 deletions(-)

diff --git a/runltp b/runltp
index 8e40d67..f206fe1 100755
--- a/runltp
+++ b/runltp
@@ -692,11 +692,13 @@ main()
     fi

     # Blacklist or skip tests if a SKIPFILE was specified with -S
-    if [ -n "$SKIPFILE" ]
-    then
-        for file in $( cat $SKIPFILE ); do
-            sed -i "/^$file[ \t]/d" ${TMP}/alltests
-        done
+    if [ -n "${SKIPFILE}" ]; then
+        while read -r test_name; do
+            case "${test_name}" in \#*) continue ;; esac
+            if [ ${test_name} ]; then
+                sed -i "/${test_name}/c\\${test_name} skip.sh" alltests
+            fi
+        done < ${SKIPFILE}
     fi

     # check for required users and groups
-- 
2.7.4

- Naresh

>
> --
> Cyril Hrubis
> chrubis@suse.cz
>
> --
> Mailing list info: https://lists.linux.it/listinfo/ltp

[LTP] [RFC] Showing skipped tests in results?

[Cyril Hrubis]

Hi!
> > I think that you misunderstand how things works, the first string on
> > each line in the alltests file that is parsed by ltp-pan is just test
> > name that appears in the test result file, it does not affect anything
> > but that.
> >
> > I guess that the easiest solution would be replacing the actuall command
> > that is supposed to be executed with something that just reports
> > skipped status.
> >
> > Quick hack would be replacing these with something that just returns the
> > right exit value:
> >
> > gzip01 gzip_tests.hs -> gzip01 exit 32
> Not working.
> the second argument is expected to be a file.

Not a true, I've looked to the ltp-pan.c and it does crude heuristic
that decides if the second part of the line is supposed to be binary
name or a shell script, if it finds any shell redirection characters or
semicolon then it's executed as a script.

> gzip01 gzip_tests.hs -> gzip01 skip.sh
> should work.
> We have to create skip.sh inside ltp
> skip.sh contains single line exit 32

gzip01 foo -> gzip01 exit 32;

That one actually works.

> OTOH,
> I have noticed one more problem with sed with updated script.
> 
> 
> Issues:
> 1. Want to skip only "testcase" write inside skipfile and run ./ltp-skip.sh

Which script are you referring to? There is no ltp-skip.sh in LTP source
code tree.

>  testcase is replaced by testcase skip.sh ( = TCONF ) which is expected.
>  testcase_64 is replaced by testcase skip.sh which is incorrect testcase name
>  testcase_16 is replaced by testcase skip.sh which is incorrect testcase name
>  testcase_xxxx is replaced by testcase skip.sh which is incorrect testcase name
> 
> diff -Narub alltests.org  alltests
> --- alltests.org 2017-12-07 22:41:46.373419820 +0530
> +++ alltests 2017-12-07 23:54:53.188657734 +0530
> @@ -272,8 +272,8 @@
>  fcntl33_64 fcntl33_64
>  fcntl34 fcntl34
>  fcntl34_64 fcntl34_64
> -fcntl35 fcntl35
> -fcntl35_64 fcntl35_64
> +fcntl35 skip.sh
> +fcntl35 skip.sh
>  fcntl36 fcntl36
>  fcntl36_64 fcntl36_64

That is beacuse you have broken the mathing pattern in your patch, the
original one that just deleted the line matched whitespace after the
name.

> 2. The skipfile have limitation of
> testcase_name should be a single word in SKIPFILE
> 
> Feature is
> We can added comments inside SKIPFILE starting with #
> The script will  ignore line starting with #

I'm okay with adding comments to skipfile.

> From c3e77bbaf03e126de15524f817960740a8c94e1b Mon Sep 17 00:00:00 2001
> From: Naresh Kamboju <naresh.kamboju@linaro.org>
> Date: Thu, 7 Dec 2017 23:14:44 +0530
> Subject: [PATCH] runltp: SKIPFILE: skip test cases should be visible as TCONF
>  in results
> 
> When we skip tests in LTP using "./runltp -S SKIPFILE", they get removed
> from the test list completely, leaving no trace in the results.
> 
> This patch will add SKIPFILE listed test cases names to results as TCONF
> testcase_name -> testcase_name skip.sh
> 
> Signed-off-by: Naresh Kamboju <naresh.kamboju@linaro.org>
> ---
>  runltp | 12 +++++++-----
>  1 file changed, 7 insertions(+), 5 deletions(-)
> 
> diff --git a/runltp b/runltp
> index 8e40d67..f206fe1 100755
> --- a/runltp
> +++ b/runltp
> @@ -692,11 +692,13 @@ main()
>      fi
> 
>      # Blacklist or skip tests if a SKIPFILE was specified with -S
> -    if [ -n "$SKIPFILE" ]
> -    then
> -        for file in $( cat $SKIPFILE ); do
> -            sed -i "/^$file[ \t]/d" ${TMP}/alltests
> -        done
> +    if [ -n "${SKIPFILE}" ]; then
> +        while read -r test_name; do
> +            case "${test_name}" in \#*) continue ;; esac
> +            if [ ${test_name} ]; then
> +                sed -i "/${test_name}/c\\${test_name} skip.sh" alltests
> +            fi
> +        done < ${SKIPFILE}

Apart from the broken sed matching regular expression and the skip.sh
that can be replaced with 'exit 32;' this looks reasonably good.

-- 
Cyril Hrubis
chrubis@suse.cz