[LTP] [PATCH v6 0/5] Run tests in CI

[LTP] [PATCH v6 0/5] Run tests in CI

[Petr Vorel]

Hi,

changes v5->v6:
* add new commit suggested by Cyril: "lib: Print Summary: into stderr"
* fix final error evaluation for C tests in runtest.sh
* run C and shell test separately (test-c and test-shell instead of
  test). Having them separate helps to easily navigate in CI. This
  required to add support to build.sh.

Hope this is a final version.

Kind regards,
Petr

Petr Vorel (5):
  lib: Print Summary: into stderr
  lib: Add script for running tests
  make: Add make test{, -c, -shell} targets
  build.sh: Add support for make test{,-c,-shell}
  CI: Run also make test-c, test-shell

 .github/workflows/ci.yml    |  10 ++
 Makefile                    |  23 +++++
 build.sh                    |  24 ++++-
 lib/newlib_tests/runtest.sh | 182 ++++++++++++++++++++++++++++++++++++
 lib/tst_test.c              |  12 +--
 5 files changed, 244 insertions(+), 7 deletions(-)
 create mode 100755 lib/newlib_tests/runtest.sh

-- 
2.32.0

[LTP] [PATCH v6 1/5] lib: Print Summary: into stderr

[Petr Vorel]

to follow tst_{brk,res}() being printed to stderr.

Found with GitHub actions where stdout and stderr are probably
block buffered and flushed at different times.

Suggested-by: Cyril Hrubis <chrubis@suse.cz>
Signed-off-by: Petr Vorel <pvorel@suse.cz>
---
* new in v6

 lib/tst_test.c | 12 ++++++------
 1 file changed, 6 insertions(+), 6 deletions(-)

diff --git a/lib/tst_test.c b/lib/tst_test.c
index f4d9f8e3b..084a83c9e 100644
--- a/lib/tst_test.c
+++ b/lib/tst_test.c
@@ -734,12 +734,12 @@ static void do_exit(int ret)
 		if (results->broken)
 			ret |= TBROK;
 
-		printf("\nSummary:\n");
-		printf("passed   %d\n", results->passed);
-		printf("failed   %d\n", results->failed);
-		printf("broken   %d\n", results->broken);
-		printf("skipped  %d\n", results->skipped);
-		printf("warnings %d\n", results->warnings);
+		fprintf(stderr, "\nSummary:\n");
+		fprintf(stderr, "passed   %d\n", results->passed);
+		fprintf(stderr, "failed   %d\n", results->failed);
+		fprintf(stderr, "broken   %d\n", results->broken);
+		fprintf(stderr, "skipped  %d\n", results->skipped);
+		fprintf(stderr, "warnings %d\n", results->warnings);
 	}
 
 	do_cleanup();
-- 
2.32.0

[LTP] [PATCH v6 2/5] lib: Add script for running tests

[Petr Vorel]

For now run only tests which TPASS or TCONF.

tst_fuzzy_sync01: sporadically fails, thus disabled:

../../include/tst_fuzzy_sync.h:685: TINFO: Exceeded execution loops, requesting exit
tst_fuzzy_sync01.c:227: TFAIL: acs:3  act:1  art:1  | =:23   -:46   +:2999931
...
Summary:
passed   21
failed   3

Use sync to workaround mangled output on GitHub Actions where stdout and
stderr are probably block buffered and flushed at different times.

Signed-off-by: Petr Vorel <pvorel@suse.cz>
---
changes v5->v6:
* fix final error evaluation for C tests in runtest.sh

 lib/newlib_tests/runtest.sh | 182 ++++++++++++++++++++++++++++++++++++
 1 file changed, 182 insertions(+)
 create mode 100755 lib/newlib_tests/runtest.sh

diff --git a/lib/newlib_tests/runtest.sh b/lib/newlib_tests/runtest.sh
new file mode 100755
index 000000000..bd7995f3b
--- /dev/null
+++ b/lib/newlib_tests/runtest.sh
@@ -0,0 +1,182 @@
+#!/bin/sh
+# Copyright (c) 2021 Petr Vorel <pvorel@suse.cz>
+
+LTP_C_API_TESTS="${LTP_C_API_TESTS:-test05 test07 test09 test12 test15 test18
+tst_bool_expr test_exec test_timer tst_res_hexd tst_strstatus tst_fuzzy_sync02
+tst_fuzzy_sync03}"
+
+LTP_SHELL_API_TESTS="${LTP_SHELL_API_TESTS:-shell/tst_check_driver.sh shell/net/*.sh}"
+
+cd $(dirname $0)
+PATH="$PWD/../../testcases/lib/:$PATH"
+
+. tst_ansi_color.sh
+
+usage()
+{
+	cat << EOF
+Usage: $0 [-b DIR ] [-c|-s]
+-b DIR  build directory (required for out-of-tree build)
+-c      run C API tests only
+-s      run shell API tests only
+-h      print this help
+EOF
+}
+
+tst_flag2mask()
+{
+	case "$1" in
+	TPASS) return 0;;
+	TFAIL) return 1;;
+	TBROK) return 2;;
+	TWARN) return 4;;
+	TINFO) return 16;;
+	TCONF) return 32;;
+	esac
+}
+
+runtest_res()
+{
+	if [ $# -eq 0 ]; then
+		echo >&2
+		return
+	fi
+
+	local res="$1"
+	shift
+
+	tst_color_enabled
+	local color=$?
+
+	printf "runtest " >&2
+	tst_print_colored $res "$res: " >&2
+	echo "$@" >&2
+
+}
+
+runtest_brk()
+{
+	local res="$1"
+	shift
+
+	tst_flag2mask "$res"
+	local mask=$?
+
+	runtest_res
+	runtest_res $res $@
+
+	exit $mask
+}
+
+run_tests()
+{
+	local target="$1"
+	local i res ret=0 tbrok tconf tfail tpass twarn vars
+
+	eval vars="\$LTP_${target}_API_TESTS"
+
+	runtest_res TINFO "=== Run $target tests ==="
+
+	for i in $vars; do
+		runtest_res TINFO "* $i"
+		./$i
+		res=$?
+
+		[ $res -ne 0 -a $res -ne 32 ] && ret=1
+
+		case $res in
+			0) tpass="$tpass $i";;
+			1) tfail="$tfail $i";;
+			2) tbrok="$tbrok $i";;
+			4) twarn="$twarn $i";;
+			32) tconf="$tconf $i";;
+			127) runtest_brk TBROK "Error: file not found (wrong PATH? out-of-tree build without -b?), exit code: $res";;
+			*) runtest_brk TBROK "Error: unknown failure, exit code: $res";;
+		esac
+		runtest_res
+		sync
+	done
+
+	runtest_res TINFO "=== $target TEST RESULTS ==="
+	runtest_res TINFO "$(echo $tpass | wc -w)x TPASS:$tpass"
+	runtest_res TINFO "$(echo $tfail | wc -w)x TFAIL:$tfail"
+	runtest_res TINFO "$(echo $tbrok | wc -w)x TBROK:$tbrok"
+	runtest_res TINFO "$(echo $twarn | wc -w)x TWARN:$twarn"
+	runtest_res TINFO "$(echo $tconf | wc -w)x TCONF:$tconf"
+	runtest_res
+
+	return $ret
+}
+
+run_c_tests()
+{
+	local ret
+
+	if [ "$builddir" ]; then
+		cd $builddir/lib/newlib_tests
+	fi
+
+	run_tests "C"
+	ret=$?
+
+	if [ "$builddir" ]; then
+		cd -
+	fi
+
+	return $ret
+}
+
+run_shell_tests()
+{
+	run_tests "SHELL"
+}
+
+
+print_result()
+{
+	local target="$1"
+	local res="$2"
+
+
+	if [ -z "$res" ]; then
+		runtest_res TCONF "$target tests skipped"
+	elif [ $res -eq 0 ]; then
+		runtest_res TPASS "All $target tests TCONF/TPASS"
+	else
+		runtest_res TFAIL "Some $target test(s) TBROK/TFAIL/TWARN"
+	fi
+}
+
+builddir=
+c_fail=
+run=
+shell_fail=
+
+while getopts b:chs opt; do
+	case $opt in
+		'h') usage; exit 0;;
+		'b') builddir=$OPTARG; PATH="$builddir/testcases/lib:$PATH";;
+		'c') run="c";;
+		's') run="s";;
+		*) usage; runtest_brk TBROK "Error: invalid option";;
+	esac
+done
+
+runtest_res TINFO "PATH='$PATH'"
+
+if [ -z "$run" -o "$run" = "c" ]; then
+	run_c_tests
+	c_fail=$?
+fi
+
+if [ -z "$run" -o "$run" = "s" ]; then
+	run_shell_tests
+	shell_fail=$?
+fi
+
+runtest_res TINFO "=== FINAL TEST RESULTS ==="
+
+print_result "C" "$c_fail"
+print_result "shell" "$shell_fail"
+
+exit $((c_fail|shell_fail))
-- 
2.32.0

[LTP] [PATCH v6 3/5] make: Add make test{, -c, -shell} targets

[Petr Vorel]

For testing C and shell API.

Reviewed-by: Cyril Hrubis <chrubis@suse.cz>
Signed-off-by: Petr Vorel <pvorel@suse.cz>
---
the same as in v5

 Makefile | 23 +++++++++++++++++++++++
 1 file changed, 23 insertions(+)

diff --git a/Makefile b/Makefile
index 56812d77b..ff4a70eec 100644
--- a/Makefile
+++ b/Makefile
@@ -192,6 +192,29 @@ $(INSTALL_TARGETS): $(INSTALL_DIR) $(DESTDIR)/$(bindir)
 ## Install
 install: $(INSTALL_TARGETS)
 
+## Test
+define _test
+	@set -e; $(top_srcdir)/lib/newlib_tests/runtest.sh -b $(abs_builddir) $(1)
+endef
+
+test: lib-all
+ifneq ($(build),$(host))
+	$(error running tests on cross-compile build not supported)
+endif
+	$(call _test)
+
+test-c: lib-all
+ifneq ($(build),$(host))
+	$(error running tests on cross-compile build not supported)
+endif
+	$(call _test,-c)
+
+test-shell: lib-all
+ifneq ($(build),$(host))
+	$(error running tests on cross-compile build not supported)
+endif
+	$(call _test,-s)
+
 ## Help
 .PHONY: help
 help:
-- 
2.32.0

[LTP] [PATCH v6 4/5] build.sh: Add support for make test{, -c, -shell}

[Petr Vorel]

Reviewed-by: Cyril Hrubis <chrubis@suse.cz>
Signed-off-by: Petr Vorel <pvorel@suse.cz>
---
changes v5->v6 (not yet reviewed by Cyril):
* add support to run also test-c and test-shell

 build.sh | 24 +++++++++++++++++++++++-
 1 file changed, 23 insertions(+), 1 deletion(-)

diff --git a/build.sh b/build.sh
index 240ce8e68..1767cc21b 100755
--- a/build.sh
+++ b/build.sh
@@ -119,6 +119,17 @@ build_out_tree()
 	make $MAKE_OPTS_OUT_TREE
 }
 
+test_in_tree()
+{
+	make $1
+}
+
+test_out_tree()
+{
+	cd $BUILD_DIR
+	make $MAKE_OPTS_OUT_TREE $1
+}
+
 install_in_tree()
 {
 	make $MAKE_OPTS install
@@ -165,6 +176,9 @@ RUN:
 autotools   run only 'make autotools'
 configure   run only 'configure'
 build       run only 'make'
+test        run only 'make test' (not supported for cross-compile build)
+test-c      run only 'make test-c' (not supported for cross-compile build)
+test-shell  run only 'make test-shell' (not supported for cross-compile build)
 install     run only 'make install'
 
 Default configure options:
@@ -192,7 +206,7 @@ while getopts "c:hio:p:r:t:" opt; do
 		esac;;
 	p) prefix="$OPTARG";;
 	r) case "$OPTARG" in
-		autotools|configure|build|install) run="$OPTARG";;
+		autotools|configure|build|test|test-c|test-shell|install) run="$OPTARG";;
 		*) echo "Wrong run type '$OPTARG'" >&2; usage; exit 1;;
 		esac;;
 	t) case "$OPTARG" in
@@ -218,6 +232,14 @@ if [ -z "$run" -o "$run" = "build" ]; then
 	eval build_${tree}_tree
 fi
 
+if [ -z "$run" -o "$run" = "test" -o "$run" = "test-c" -o "$run" = "test-shell" ]; then
+	if [ "$build" = "cross" ]; then
+		echo "cross-compile build, skipping running tests" >&2
+	else
+		eval test_${tree}_tree $run
+	fi
+fi
+
 if [ -z "$run" -o "$run" = "install" ]; then
 	if [ "$install" = 1 ]; then
 		eval install_${tree}_tree
-- 
2.32.0

[LTP] [PATCH v6 5/5] CI: Run also make test-c, test-shell

[Petr Vorel]

run C and shell test separately (test-c and test-shell instead of test).
Having them separate helps to easily navigate in CI.

It's run on all jobs, but on cross compile jobs it's obviously skipped.

Signed-off-by: Petr Vorel <pvorel@suse.cz>
---
changes v5->v6:
* run C and shell test separately (test-c and test-shell instead of
  test). Having them separate helps to easily navigate in CI. This
  required to add support to build.sh.

 .github/workflows/ci.yml | 10 ++++++++++
 1 file changed, 10 insertions(+)

diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
index f67f14927..9d37f49e4 100644
--- a/.github/workflows/ci.yml
+++ b/.github/workflows/ci.yml
@@ -151,6 +151,16 @@ jobs:
     - name: Compile
       run: ./build.sh -r build -o ${TREE:-in}
 
+    - name: Test C API
+      run: |
+        case "$VARIANT" in cross-compile*) BUILD="cross";; i386) BUILD="32";; *) BUILD="native";; esac
+        ./build.sh -r test-c -o ${TREE:-in} -t $BUILD
+
+    - name: Test shell API
+      run: |
+        case "$VARIANT" in cross-compile*) BUILD="cross";; i386) BUILD="32";; *) BUILD="native";; esac
+        ./build.sh -r test-shell -o ${TREE:-in} -t $BUILD
+
     - name: Install
       run: |
         if [ "$MAKE_INSTALL" = 1 ]; then INSTALL_OPT="-i"; fi
-- 
2.32.0

[LTP] [PATCH v6 0/5] Run tests in CI

[Petr Vorel]

Hi Cyril,

> Hi,

> changes v5->v6:
> * add new commit suggested by Cyril: "lib: Print Summary: into stderr"
> * fix final error evaluation for C tests in runtest.sh
> * run C and shell test separately (test-c and test-shell instead of
>   test). Having them separate helps to easily navigate in CI. This
>   required to add support to build.sh.

> Hope this is a final version.

Verified slightly modified version (see diff below):
https://github.com/pevik/ltp/actions/runs/1033374720

FYI I'd probably keep first commit "ib: Print Summary: into stderr" although it
should not be needed for CI as 1>&2 redirection fixed the output.

Can I merge it with your ack?

Kind regards,
Petr

diff --git ci/debian.sh ci/debian.sh
index a609da887..e929452ff 100755
--- ci/debian.sh
+++ ci/debian.sh
@@ -22,6 +22,7 @@ $apt \
 	clang \
 	gcc \
 	git \
+	iproute2 \
 	libacl1 \
 	libacl1-dev \
 	libaio-dev \
diff --git ci/fedora.sh ci/fedora.sh
index 959f3af20..dc1293aa5 100755
--- ci/fedora.sh
+++ ci/fedora.sh
@@ -13,6 +13,7 @@ $yum \
 	gcc \
 	git \
 	findutils \
+	iproute \
 	numactl-devel \
 	libtirpc \
 	libtirpc-devel \
diff --git ci/tumbleweed.sh ci/tumbleweed.sh
index ab622e05c..f1e7252f2 100755
--- ci/tumbleweed.sh
+++ ci/tumbleweed.sh
@@ -13,6 +13,7 @@ $zyp \
 	gcc \
 	git \
 	gzip \
+	iproute2 \
 	make \
 	kernel-default-devel \
 	keyutils-devel \
diff --git lib/newlib_tests/runtest.sh lib/newlib_tests/runtest.sh
index bd7995f3b..a11d6ed1a 100755
--- lib/newlib_tests/runtest.sh
+++ lib/newlib_tests/runtest.sh
@@ -79,7 +79,7 @@ run_tests()
 
 	for i in $vars; do
 		runtest_res TINFO "* $i"
-		./$i
+		./$i 1>&2
 		res=$?
 
 		[ $res -ne 0 -a $res -ne 32 ] && ret=1
@@ -94,7 +94,6 @@ run_tests()
 			*) runtest_brk TBROK "Error: unknown failure, exit code: $res";;
 		esac
 		runtest_res
-		sync
 	done
 
 	runtest_res TINFO "=== $target TEST RESULTS ==="

[LTP] [PATCH v6 0/5] Run tests in CI

[Petr Vorel]

Hi Cyril,

> Verified slightly modified version (see diff below):
> https://github.com/pevik/ltp/actions/runs/1033374720

Well, still some minor order issue with "In RPN: ..." (printed to stdout) being
at the end, but I give up and merge (unless you have explanation for it).
https://github.com/pevik/ltp/runs/3075175819?check_suite_focus=true#step:11:501

Kind regards,
Petr