[PATCH] check: annotate good and expunged tests in results

[PATCH] check: annotate good and expunged tests in results

[jeffm]

From: Jeff Mahoney <jeffm@suse.com>

Currently, we only create results files when a test has failed or was
supposed to run but some dependency wasn't met causing it not to run.
Short of saving the summary at the end of the run, there's no way to tell
which tests passed or which tests weren't run due to being excluded.

This patch moves successful test results to $seqres.out.good to annotate
good results.  It also adds tests excluded by group to the $tmp.xlist file
and adds annotations for every test excluded.  When a test is expunged
during execution, an expunged message will be issued and a
$seqres.expunged file will be created, both containing the reason for
the test being expunged.  Reasons can be "command line", "file $filename",
or "group <group> [group...]".

This makes the output more noisy in the expunged case and makes startup
take slightly longer, but ends up with results that can be more easily
parsed by automated tools.

Signed-off-by: Jeff Mahoney <jeffm@suse.com>
---
 check | 57 +++++++++++++++++++++++++++++++--------------------------
 1 file changed, 31 insertions(+), 26 deletions(-)

diff --git a/check b/check
index 546683c5..9a1f4e7e 100755
--- a/check
+++ b/check
@@ -173,28 +173,18 @@ get_all_tests()
 	done
 }
 
-# takes the list of tests to run in $tmp.list, and removes the tests passed to
-# the function from that list.
+# takes the list of tests to run in $tmp.list and adds them to the excluded
+# test list, annotated with the group that excluded each one
 trim_test_list()
 {
+	group=$1
+	shift
 	test_list="$*"
 
-	rm -f $tmp.grep
-	numsed=0
-	for t in $test_list
-	do
-	    if [ $numsed -gt 100 ]; then
-		grep -v -f $tmp.grep <$tmp.list >$tmp.tmp
-		mv $tmp.tmp $tmp.list
-		numsed=0
-		rm -f $tmp.grep
-	    fi
-	    echo "^$t\$" >>$tmp.grep
-	    numsed=`expr $numsed + 1`
+	for test in $test_list; do
+		test=${test##tests/}
+		echo "$test group $group" >> $tmp.xlist
 	done
-	grep -v -f $tmp.grep <$tmp.list >$tmp.tmp
-	mv $tmp.tmp $tmp.list
-	rm -f $tmp.grep
 }
 
 
@@ -246,7 +236,7 @@ _prepare_test_list()
 			exit 1
 		fi
 
-		trim_test_list $list
+		trim_test_list $xgroup $list
 	done
 
 	# sort the list of tests into numeric order
@@ -285,13 +275,15 @@ while [ $# -gt 0 ]; do
 		for d in $SRC_GROUPS $FSTYP; do
 			[ -f $SRC_DIR/$d/$xfile ] || continue
 			for f in `sed "s/#.*$//" $SRC_DIR/$d/$xfile`; do
-				echo $d/$f >> $tmp.xlist
+				echo "$d/$f command line" >> $tmp.xlist
 			done
 		done
 		;;
 	-E)	xfile=$2; shift ;
 		if [ -f $xfile ]; then
-			sed "s/#.*$//" "$xfile" >> $tmp.xlist
+			sed -e "s/#.*$//" \
+			    -e "s;$; file $xfile;" "$xfile" \
+			    >> $tmp.xlist
 	        fi
 		;;
 	-s)	RUN_SECTION="$RUN_SECTION $2"; shift ;;
@@ -491,11 +483,17 @@ _check_filesystems()
 _expunge_test()
 {
 	local TEST_ID="$1"
+	local OUTPUT="$2"
 	if [ -s $tmp.xlist ]; then
-		if grep -q $TEST_ID $tmp.xlist; then
-			echo "       [expunged]"
+		grep $TEST_ID $tmp.xlist > $tmp._expunge_test
+		if [ $? -eq 0 ]; then
+			sed -e "s;$TEST_ID ;;" $tmp._expunge_test | \
+				tr '\n' ' ' |  sed -e 's; group;;g'
+			echo
+			rm -f $tmp._expunge_test
 			return 1
 		fi
+		rm -f $tmp._expunge_test
 	fi
 	return 0
 }
@@ -670,8 +668,10 @@ for section in $HOST_OPTIONS_SECTIONS; do
 	    echo -n "$seqnum"
 
 	    if $showme; then
-		_expunge_test $seqnum
+		_expunge_test $seqnum > $tmp.xreason
 		if [ $? -eq 1 ]; then
+			echo -n "        [expunged] "
+			cat $tmp.xreason
 			continue
 		fi
 		echo
@@ -689,11 +689,15 @@ for section in $HOST_OPTIONS_SECTIONS; do
 	    else
 		# really going to try and run this one
 		#
-		rm -f $seqres.out.bad
+		rm -f $seqres.out.bad $seqres.out.good $seqres.expunged
+		rm -f $seqres.notrun
 
 		# check if we really should run it
-		_expunge_test $seqnum
+		_expunge_test $seqnum > $tmp.xreason
 		if [ $? -eq 1 ]; then
+			mv $tmp.xreason $seqres.expunged
+			echo -n "        [expunged] "
+			cat $seqres.expunged
 			continue
 		fi
 
@@ -704,7 +708,7 @@ for section in $HOST_OPTIONS_SECTIONS; do
 		else
 			echo -n "	"	# prettier output with timestamps.
 		fi
-		rm -f core $seqres.notrun
+		rm -f core
 
 		start=`_wallclock`
 		$timestamp && echo -n "	["`date "+%T"`"]"
@@ -770,6 +774,7 @@ for section in $HOST_OPTIONS_SECTIONS; do
 			    else
 				echo "$seqnum `expr $stop - $start`" >>$tmp.time
 				echo -n " `expr $stop - $start`s"
+				mv $tmp.out $seqres.out.good
 			    fi
 			    echo ""
 			else
-- 
2.15.1

Re: [PATCH] check: annotate good and expunged tests in results

[Eryu Guan]

On Thu, Apr 12, 2018 at 05:38:38PM -0400, jeffm@suse.com wrote:
> From: Jeff Mahoney <jeffm@suse.com>

[Sorry for the late review..]

> 
> Currently, we only create results files when a test has failed or was
> supposed to run but some dependency wasn't met causing it not to run.
> Short of saving the summary at the end of the run, there's no way to tell
> which tests passed or which tests weren't run due to being excluded.
> 
> This patch moves successful test results to $seqres.out.good to annotate
> good results.  It also adds tests excluded by group to the $tmp.xlist file
> and adds annotations for every test excluded.  When a test is expunged

I can see that annotating tests excluded by group is helpful, but I
don't think we need $seqres.out.good, passed tests are already shown by
check, and $seqres.out.good is same as the golden image, it looks
redundant.

> during execution, an expunged message will be issued and a
> $seqres.expunged file will be created, both containing the reason for
> the test being expunged.  Reasons can be "command line", "file $filename",
> or "group <group> [group...]".
> 
> This makes the output more noisy in the expunged case and makes startup
> take slightly longer, but ends up with results that can be more easily
> parsed by automated tools.
> 
> Signed-off-by: Jeff Mahoney <jeffm@suse.com>
> ---
>  check | 57 +++++++++++++++++++++++++++++++--------------------------
>  1 file changed, 31 insertions(+), 26 deletions(-)
> 
> diff --git a/check b/check
> index 546683c5..9a1f4e7e 100755
> --- a/check
> +++ b/check
> @@ -173,28 +173,18 @@ get_all_tests()
>  	done
>  }
>  
> -# takes the list of tests to run in $tmp.list, and removes the tests passed to
> -# the function from that list.
> +# takes the list of tests to run in $tmp.list and adds them to the excluded
> +# test list, annotated with the group that excluded each one

The comments don't seem quite right, the new behavior doesn't take tests
from $tmp.list, it just edits the $tmp.xlist file for later use.

>  trim_test_list()
>  {
> +	group=$1

Please declare local variable as 'local' when possible.

> +	shift
>  	test_list="$*"
>  
> -	rm -f $tmp.grep
> -	numsed=0
> -	for t in $test_list
> -	do
> -	    if [ $numsed -gt 100 ]; then
> -		grep -v -f $tmp.grep <$tmp.list >$tmp.tmp
> -		mv $tmp.tmp $tmp.list
> -		numsed=0
> -		rm -f $tmp.grep
> -	    fi
> -	    echo "^$t\$" >>$tmp.grep
> -	    numsed=`expr $numsed + 1`
> +	for test in $test_list; do
> +		test=${test##tests/}
> +		echo "$test group $group" >> $tmp.xlist
>  	done
> -	grep -v -f $tmp.grep <$tmp.list >$tmp.tmp
> -	mv $tmp.tmp $tmp.list
> -	rm -f $tmp.grep
>  }
>  
>  
> @@ -246,7 +236,7 @@ _prepare_test_list()
>  			exit 1
>  		fi
>  
> -		trim_test_list $list
> +		trim_test_list $xgroup $list
>  	done
>  
>  	# sort the list of tests into numeric order
> @@ -285,13 +275,15 @@ while [ $# -gt 0 ]; do
>  		for d in $SRC_GROUPS $FSTYP; do
>  			[ -f $SRC_DIR/$d/$xfile ] || continue
>  			for f in `sed "s/#.*$//" $SRC_DIR/$d/$xfile`; do
> -				echo $d/$f >> $tmp.xlist
> +				echo "$d/$f command line" >> $tmp.xlist
>  			done
>  		done
>  		;;
>  	-E)	xfile=$2; shift ;
>  		if [ -f $xfile ]; then
> -			sed "s/#.*$//" "$xfile" >> $tmp.xlist
> +			sed -e "s/#.*$//" \
> +			    -e "s;$; file $xfile;" "$xfile" \
> +			    >> $tmp.xlist

When using config section, e.g. ./check -s <sec_name>, and FSTYP is
different in new section, check will do _prepare_test_list again, so
tests selected by "-x <xgroup>" will be listed twice in $tmp.xlist file,
as a result, I see duplicated group names in the annotate message, e.g.

generic/458        [expunged] group metadata metadata

I think we should move above "-X/-E" handlings to _prepare_test_list,
only assign variable "xfile" here (use a new variable for -E option,
perhaps). So that we could simply remove $tmp.xlist before calling
_prepare_test_list again, and it could exclude all specified tests only
once. Maybe as patch 1/2 in a patchset.

>  	        fi
>  		;;
>  	-s)	RUN_SECTION="$RUN_SECTION $2"; shift ;;
> @@ -491,11 +483,17 @@ _check_filesystems()
>  _expunge_test()
>  {
>  	local TEST_ID="$1"
> +	local OUTPUT="$2"

OUTPUT is not used.

>  	if [ -s $tmp.xlist ]; then
> -		if grep -q $TEST_ID $tmp.xlist; then
> -			echo "       [expunged]"
> +		grep $TEST_ID $tmp.xlist > $tmp._expunge_test
> +		if [ $? -eq 0 ]; then
> +			sed -e "s;$TEST_ID ;;" $tmp._expunge_test | \
> +				tr '\n' ' ' |  sed -e 's; group;;g'
> +			echo
> +			rm -f $tmp._expunge_test
>  			return 1
>  		fi
> +		rm -f $tmp._expunge_test
>  	fi
>  	return 0
>  }
> @@ -670,8 +668,10 @@ for section in $HOST_OPTIONS_SECTIONS; do
>  	    echo -n "$seqnum"
>  
>  	    if $showme; then
> -		_expunge_test $seqnum
> +		_expunge_test $seqnum > $tmp.xreason
>  		if [ $? -eq 1 ]; then
> +			echo -n "        [expunged] "
> +			cat $tmp.xreason
>  			continue
>  		fi
>  		echo
> @@ -689,11 +689,15 @@ for section in $HOST_OPTIONS_SECTIONS; do
>  	    else
>  		# really going to try and run this one
>  		#
> -		rm -f $seqres.out.bad
> +		rm -f $seqres.out.bad $seqres.out.good $seqres.expunged
> +		rm -f $seqres.notrun
>  
>  		# check if we really should run it
> -		_expunge_test $seqnum
> +		_expunge_test $seqnum > $tmp.xreason
>  		if [ $? -eq 1 ]; then
> +			mv $tmp.xreason $seqres.expunged
> +			echo -n "        [expunged] "
> +			cat $seqres.expunged
>  			continue
>  		fi
>  
> @@ -704,7 +708,7 @@ for section in $HOST_OPTIONS_SECTIONS; do
>  		else
>  			echo -n "	"	# prettier output with timestamps.
>  		fi
> -		rm -f core $seqres.notrun
> +		rm -f core
>  
>  		start=`_wallclock`
>  		$timestamp && echo -n "	["`date "+%T"`"]"
> @@ -770,6 +774,7 @@ for section in $HOST_OPTIONS_SECTIONS; do
>  			    else
>  				echo "$seqnum `expr $stop - $start`" >>$tmp.time
>  				echo -n " `expr $stop - $start`s"
> +				mv $tmp.out $seqres.out.good

As said above, I don't think it's that useful.

Thanks,
Eryu

>  			    fi
>  			    echo ""
>  			else
> -- 
> 2.15.1
> 
> --
> To unsubscribe from this list: send the line "unsubscribe fstests" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH] check: annotate good and expunged tests in results

[Jeff Mahoney]

On 4/24/18 5:36 AM, Eryu Guan wrote:
> On Thu, Apr 12, 2018 at 05:38:38PM -0400, jeffm@suse.com wrote:
>> From: Jeff Mahoney <jeffm@suse.com>
> 
> [Sorry for the late review..]

Thanks for the review just the same.

>> Currently, we only create results files when a test has failed or was
>> supposed to run but some dependency wasn't met causing it not to run.
>> Short of saving the summary at the end of the run, there's no way to tell
>> which tests passed or which tests weren't run due to being excluded.
>>
>> This patch moves successful test results to $seqres.out.good to annotate
>> good results.  It also adds tests excluded by group to the $tmp.xlist file
>> and adds annotations for every test excluded.  When a test is expunged
> 
> I can see that annotating tests excluded by group is helpful, but I
> don't think we need $seqres.out.good, passed tests are already shown by
> check, and $seqres.out.good is same as the golden image, it looks
> redundant.

The contents are redundant, but the purpose of the file existing isn't.
My goal is to be able to point a script at the results directory for a
test run and autogenerate a table of results and/or compare them to
known results.  Right now, there's no way to tell whether a test was run
at all, only that it failed or was skipped due to some missing
dependency.  The expunge changes handle the explicitly skipped cases,
but without the .good files, there's nothing indicating a test actually
ran or when it last did.  The contents don't have to be the output.  If
you can think of something more useful there (or if we just create an
empty file), that's fine too.
>> -# takes the list of tests to run in $tmp.list, and removes the tests passed to
>> -# the function from that list.
>> +# takes the list of tests to run in $tmp.list and adds them to the excluded
>> +# test list, annotated with the group that excluded each one
> 
> The comments don't seem quite right, the new behavior doesn't take tests
> from $tmp.list, it just edits the $tmp.xlist file for later use.

Right.  I missed updating the comment after changing the behavior.

>>  trim_test_list()
>>  {
>> +	group=$1
> 
> Please declare local variable as 'local' when possible.

Ok.

> When using config section, e.g. ./check -s <sec_name>, and FSTYP is
> different in new section, check will do _prepare_test_list again, so
> tests selected by "-x <xgroup>" will be listed twice in $tmp.xlist file,
> as a result, I see duplicated group names in the annotate message, e.g.
> 
> generic/458        [expunged] group metadata metadata
> 
> I think we should move above "-X/-E" handlings to _prepare_test_list,
> only assign variable "xfile" here (use a new variable for -E option,
> perhaps). So that we could simply remove $tmp.xlist before calling
> _prepare_test_list again, and it could exclude all specified tests only
> once. Maybe as patch 1/2 in a patchset.

Ok, that's easy enough to do.

>>  	        fi
>>  		;;
>>  	-s)	RUN_SECTION="$RUN_SECTION $2"; shift ;;
>> @@ -491,11 +483,17 @@ _check_filesystems()
>>  _expunge_test()
>>  {
>>  	local TEST_ID="$1"
>> +	local OUTPUT="$2"
> 
> OUTPUT is not used.

Ack.

>> @@ -770,6 +774,7 @@ for section in $HOST_OPTIONS_SECTIONS; do
>>  			    else
>>  				echo "$seqnum `expr $stop - $start`" >>$tmp.time
>>  				echo -n " `expr $stop - $start`s"
>> +				mv $tmp.out $seqres.out.good
> 
> As said above, I don't think it's that useful.

It doesn't need to be the contents of the output, but I'd like it to be
*something* and moving a file that already exists is as easy as creating
a new one.

-Jeff

-- 
Jeff Mahoney
SUSE Labs

Re: [PATCH] check: annotate good and expunged tests in results

[Eryu Guan]

On Tue, Apr 24, 2018 at 02:03:24PM -0400, Jeff Mahoney wrote:
> On 4/24/18 5:36 AM, Eryu Guan wrote:
> > On Thu, Apr 12, 2018 at 05:38:38PM -0400, jeffm@suse.com wrote:
> >> From: Jeff Mahoney <jeffm@suse.com>
> > 
> > [Sorry for the late review..]
> 
> Thanks for the review just the same.
> 
> >> Currently, we only create results files when a test has failed or was
> >> supposed to run but some dependency wasn't met causing it not to run.
> >> Short of saving the summary at the end of the run, there's no way to tell
> >> which tests passed or which tests weren't run due to being excluded.
> >>
> >> This patch moves successful test results to $seqres.out.good to annotate
> >> good results.  It also adds tests excluded by group to the $tmp.xlist file
> >> and adds annotations for every test excluded.  When a test is expunged
> > 
> > I can see that annotating tests excluded by group is helpful, but I
> > don't think we need $seqres.out.good, passed tests are already shown by
> > check, and $seqres.out.good is same as the golden image, it looks
> > redundant.
> 
> The contents are redundant, but the purpose of the file existing isn't.
> My goal is to be able to point a script at the results directory for a
> test run and autogenerate a table of results and/or compare them to
> known results.  Right now, there's no way to tell whether a test was run

But the test run & pass info is already available from the check output
and the test result summary at the end of check. Is that sufficient for
you? Also, we already have mechanism to generate a test report in xunit
format, i.e. ./check -R xunit -g auto, which includes results for passed
& failed & notrun tests.

My concern is that saving .good files consume more and more space over
time, and they don't provide any additional information (even with empty
files that don't require much space).

Thanks,
Eryu

> at all, only that it failed or was skipped due to some missing
> dependency.  The expunge changes handle the explicitly skipped cases,
> but without the .good files, there's nothing indicating a test actually
> ran or when it last did.  The contents don't have to be the output.  If
> you can think of something more useful there (or if we just create an
> empty file), that's fine too.
> >> -# takes the list of tests to run in $tmp.list, and removes the tests passed to
> >> -# the function from that list.
> >> +# takes the list of tests to run in $tmp.list and adds them to the excluded
> >> +# test list, annotated with the group that excluded each one
> > 
> > The comments don't seem quite right, the new behavior doesn't take tests
> > from $tmp.list, it just edits the $tmp.xlist file for later use.
> 
> Right.  I missed updating the comment after changing the behavior.
> 
> >>  trim_test_list()
> >>  {
> >> +	group=$1
> > 
> > Please declare local variable as 'local' when possible.
> 
> Ok.
> 
> > When using config section, e.g. ./check -s <sec_name>, and FSTYP is
> > different in new section, check will do _prepare_test_list again, so
> > tests selected by "-x <xgroup>" will be listed twice in $tmp.xlist file,
> > as a result, I see duplicated group names in the annotate message, e.g.
> > 
> > generic/458        [expunged] group metadata metadata
> > 
> > I think we should move above "-X/-E" handlings to _prepare_test_list,
> > only assign variable "xfile" here (use a new variable for -E option,
> > perhaps). So that we could simply remove $tmp.xlist before calling
> > _prepare_test_list again, and it could exclude all specified tests only
> > once. Maybe as patch 1/2 in a patchset.
> 
> Ok, that's easy enough to do.
> 
> >>  	        fi
> >>  		;;
> >>  	-s)	RUN_SECTION="$RUN_SECTION $2"; shift ;;
> >> @@ -491,11 +483,17 @@ _check_filesystems()
> >>  _expunge_test()
> >>  {
> >>  	local TEST_ID="$1"
> >> +	local OUTPUT="$2"
> > 
> > OUTPUT is not used.
> 
> Ack.
> 
> >> @@ -770,6 +774,7 @@ for section in $HOST_OPTIONS_SECTIONS; do
> >>  			    else
> >>  				echo "$seqnum `expr $stop - $start`" >>$tmp.time
> >>  				echo -n " `expr $stop - $start`s"
> >> +				mv $tmp.out $seqres.out.good
> > 
> > As said above, I don't think it's that useful.
> 
> It doesn't need to be the contents of the output, but I'd like it to be
> *something* and moving a file that already exists is as easy as creating
> a new one.
> 
> -Jeff
> 
> -- 
> Jeff Mahoney
> SUSE Labs

Re: [PATCH] check: annotate good and expunged tests in results

[Luis R. Rodriguez]

On Fri, Apr 27, 2018 at 07:23:56PM +0800, Eryu Guan wrote:
> 
> But the test run & pass info is already available from the check output
> and the test result summary at the end of check. Is that sufficient for
> you? Also, we already have mechanism to generate a test report in xunit
> format, i.e. ./check -R xunit -g auto, which includes results for passed
> & failed & notrun tests.

Do we have a way to parse the results *after* a run? For instance,
if you forgot -R xunit ?

  Luis

Re: [PATCH] check: annotate good and expunged tests in results

[Eryu Guan]

On Fri, Apr 27, 2018 at 05:00:09PM +0000, Luis R. Rodriguez wrote:
> On Fri, Apr 27, 2018 at 07:23:56PM +0800, Eryu Guan wrote:
> > 
> > But the test run & pass info is already available from the check output
> > and the test result summary at the end of check. Is that sufficient for
> > you? Also, we already have mechanism to generate a test report in xunit
> > format, i.e. ./check -R xunit -g auto, which includes results for passed
> > & failed & notrun tests.
> 
> Do we have a way to parse the results *after* a run? For instance,
> if you forgot -R xunit ?

There's a tools/compare-failures script that takes the outputs of check
as inputs and compares the results. But, TBH, I never run it after
reviewing it.. Perhaps it could be enhanced somehow.

Thanks,
Eryu

Re: [PATCH] check: annotate good and expunged tests in results

[Jeff Mahoney]

On 4/27/18 10:45 PM, Eryu Guan wrote:
> On Fri, Apr 27, 2018 at 05:00:09PM +0000, Luis R. Rodriguez wrote:
>> On Fri, Apr 27, 2018 at 07:23:56PM +0800, Eryu Guan wrote:
>>>
>>> But the test run & pass info is already available from the check output
>>> and the test result summary at the end of check. Is that sufficient for
>>> you? Also, we already have mechanism to generate a test report in xunit
>>> format, i.e. ./check -R xunit -g auto, which includes results for passed
>>> & failed & notrun tests.
>>
>> Do we have a way to parse the results *after* a run? For instance,
>> if you forgot -R xunit ?
> 
> There's a tools/compare-failures script that takes the outputs of check
> as inputs and compares the results. But, TBH, I never run it after
> reviewing it.. Perhaps it could be enhanced somehow.

Yeah, that takes check output.  Without capturing it at runtime, it
can't compare anything.

The XML report may do most of what I want provided we can enable it by
default and write it someplace safe rather than clean it up
automatically when the test run is interrupted.

I already have test code that extends it to output expunged tests and to
add an explicit <pass/> element.  It saves the timestamps already, so
that's a plus.

-Jeff

-- 
Jeff Mahoney
SUSE Labs

Re: [PATCH] check: annotate good and expunged tests in results

[Jeff Mahoney]

On 4/30/18 4:48 PM, Jeff Mahoney wrote:
> On 4/27/18 10:45 PM, Eryu Guan wrote:
>> On Fri, Apr 27, 2018 at 05:00:09PM +0000, Luis R. Rodriguez wrote:
>>> On Fri, Apr 27, 2018 at 07:23:56PM +0800, Eryu Guan wrote:
>>>>
>>>> But the test run & pass info is already available from the check output
>>>> and the test result summary at the end of check. Is that sufficient for
>>>> you? Also, we already have mechanism to generate a test report in xunit
>>>> format, i.e. ./check -R xunit -g auto, which includes results for passed
>>>> & failed & notrun tests.
>>>
>>> Do we have a way to parse the results *after* a run? For instance,
>>> if you forgot -R xunit ?
>>
>> There's a tools/compare-failures script that takes the outputs of check
>> as inputs and compares the results. But, TBH, I never run it after
>> reviewing it.. Perhaps it could be enhanced somehow.
> 
> Yeah, that takes check output.  Without capturing it at runtime, it
> can't compare anything.
> 
> The XML report may do most of what I want provided we can enable it by
> default and write it someplace safe rather than clean it up
> automatically when the test run is interrupted.
> 
> I already have test code that extends it to output expunged tests and to
> add an explicit <pass/> element.  It saves the timestamps already, so
> that's a plus.

And shortly after writing this, I had a test run hang.  Since it hung,
even writing it on exit wouldn't have worked.  There's not enough
information leftover to generate it.  I think I'd still like the results
directory to contain the information required to generate it as a
post-mortem.

-Jeff

-- 
Jeff Mahoney
SUSE Labs

Re: [PATCH] check: annotate good and expunged tests in results

[Jeff Mahoney]

On 5/1/18 9:34 AM, Jeff Mahoney wrote:
> On 4/30/18 4:48 PM, Jeff Mahoney wrote:
>> On 4/27/18 10:45 PM, Eryu Guan wrote:
>>> On Fri, Apr 27, 2018 at 05:00:09PM +0000, Luis R. Rodriguez wrote:
>>>> On Fri, Apr 27, 2018 at 07:23:56PM +0800, Eryu Guan wrote:
>>>>>
>>>>> But the test run & pass info is already available from the check output
>>>>> and the test result summary at the end of check. Is that sufficient for
>>>>> you? Also, we already have mechanism to generate a test report in xunit
>>>>> format, i.e. ./check -R xunit -g auto, which includes results for passed
>>>>> & failed & notrun tests.
>>>>
>>>> Do we have a way to parse the results *after* a run? For instance,
>>>> if you forgot -R xunit ?
>>>
>>> There's a tools/compare-failures script that takes the outputs of check
>>> as inputs and compares the results. But, TBH, I never run it after
>>> reviewing it.. Perhaps it could be enhanced somehow.
>>
>> Yeah, that takes check output.  Without capturing it at runtime, it
>> can't compare anything.
>>
>> The XML report may do most of what I want provided we can enable it by
>> default and write it someplace safe rather than clean it up
>> automatically when the test run is interrupted.
>>
>> I already have test code that extends it to output expunged tests and to
>> add an explicit <pass/> element.  It saves the timestamps already, so
>> that's a plus.
> 
> And shortly after writing this, I had a test run hang.  Since it hung,
> even writing it on exit wouldn't have worked.  There's not enough
> information leftover to generate it.  I think I'd still like the results
> directory to contain the information required to generate it as a
> post-mortem.

After thinking on it a bit more, since you object to writing a bunch of
files by default, we could accomplish the same goal by adding a "files"
report type that does this without dropping files for everyone.  I'm
working that up now.

-Jeff

-- 
Jeff Mahoney
SUSE Labs

Re: [PATCH] check: annotate good and expunged tests in results

[Luis Chamberlain]

On Tue, May 1, 2018 at 8:44 AM Jeff Mahoney <jeffm@suse.com> wrote:
>
> On 5/1/18 9:34 AM, Jeff Mahoney wrote:
> > On 4/30/18 4:48 PM, Jeff Mahoney wrote:
> >> On 4/27/18 10:45 PM, Eryu Guan wrote:
> >>> On Fri, Apr 27, 2018 at 05:00:09PM +0000, Luis R. Rodriguez wrote:
> >>>> On Fri, Apr 27, 2018 at 07:23:56PM +0800, Eryu Guan wrote:
> >>>>>
> >>>>> But the test run & pass info is already available from the check output
> >>>>> and the test result summary at the end of check. Is that sufficient for
> >>>>> you? Also, we already have mechanism to generate a test report in xunit
> >>>>> format, i.e. ./check -R xunit -g auto, which includes results for passed
> >>>>> & failed & notrun tests.
> >>>>
> >>>> Do we have a way to parse the results *after* a run? For instance,
> >>>> if you forgot -R xunit ?
> >>>
> >>> There's a tools/compare-failures script that takes the outputs of check
> >>> as inputs and compares the results. But, TBH, I never run it after
> >>> reviewing it.. Perhaps it could be enhanced somehow.
> >>
> >> Yeah, that takes check output.  Without capturing it at runtime, it
> >> can't compare anything.
> >>
> >> The XML report may do most of what I want provided we can enable it by
> >> default and write it someplace safe rather than clean it up
> >> automatically when the test run is interrupted.
> >>
> >> I already have test code that extends it to output expunged tests and to
> >> add an explicit <pass/> element.  It saves the timestamps already, so
> >> that's a plus.
> >
> > And shortly after writing this, I had a test run hang.  Since it hung,
> > even writing it on exit wouldn't have worked.  There's not enough
> > information leftover to generate it.  I think I'd still like the results
> > directory to contain the information required to generate it as a
> > post-mortem.
>
> After thinking on it a bit more, since you object to writing a bunch of
> files by default, we could accomplish the same goal by adding a "files"
> report type that does this without dropping files for everyone.  I'm
> working that up now.

How's that going BTW? :)

 Luis

Re: [PATCH] check: annotate good and expunged tests in results

[Jeff Mahoney]

On 2/12/19 12:04 PM, Luis Chamberlain wrote:
> On Tue, May 1, 2018 at 8:44 AM Jeff Mahoney <jeffm@suse.com> wrote:
>>
>> On 5/1/18 9:34 AM, Jeff Mahoney wrote:
>>> On 4/30/18 4:48 PM, Jeff Mahoney wrote:
>>>> On 4/27/18 10:45 PM, Eryu Guan wrote:
>>>>> On Fri, Apr 27, 2018 at 05:00:09PM +0000, Luis R. Rodriguez wrote:
>>>>>> On Fri, Apr 27, 2018 at 07:23:56PM +0800, Eryu Guan wrote:
>>>>>>>
>>>>>>> But the test run & pass info is already available from the check output
>>>>>>> and the test result summary at the end of check. Is that sufficient for
>>>>>>> you? Also, we already have mechanism to generate a test report in xunit
>>>>>>> format, i.e. ./check -R xunit -g auto, which includes results for passed
>>>>>>> & failed & notrun tests.
>>>>>>
>>>>>> Do we have a way to parse the results *after* a run? For instance,
>>>>>> if you forgot -R xunit ?
>>>>>
>>>>> There's a tools/compare-failures script that takes the outputs of check
>>>>> as inputs and compares the results. But, TBH, I never run it after
>>>>> reviewing it.. Perhaps it could be enhanced somehow.
>>>>
>>>> Yeah, that takes check output.  Without capturing it at runtime, it
>>>> can't compare anything.
>>>>
>>>> The XML report may do most of what I want provided we can enable it by
>>>> default and write it someplace safe rather than clean it up
>>>> automatically when the test run is interrupted.
>>>>
>>>> I already have test code that extends it to output expunged tests and to
>>>> add an explicit <pass/> element.  It saves the timestamps already, so
>>>> that's a plus.
>>>
>>> And shortly after writing this, I had a test run hang.  Since it hung,
>>> even writing it on exit wouldn't have worked.  There's not enough
>>> information leftover to generate it.  I think I'd still like the results
>>> directory to contain the information required to generate it as a
>>> post-mortem.
>>
>> After thinking on it a bit more, since you object to writing a bunch of
>> files by default, we could accomplish the same goal by adding a "files"
>> report type that does this without dropping files for everyone.  I'm
>> working that up now.
> 
> How's that going BTW? :)

I've had it working for a while.  It's kind of messy at the moment and
drops files in the same output dir as the test results.  I'm reworking
it a bit to use a separate files directory under the report/section
directory so it's easier for scripts to work with.

-Jeff


-- 
Jeff Mahoney
SUSE Labs