[LTP] [PATCH v3 0/5] Run tests in CI

[LTP] [PATCH v3 0/5] Run tests in CI

[Petr Vorel]

Hi all,

changes v2->v3:
* going back to simplest changes in make
* adding lib/newlib_tests/runtest.sh (instead fighting with our build
  system I concentrated on runtest.sh)
* removing tst_fuzzy_sync01 as it sporadically fails

Tested:
https://github.com/pevik/ltp/actions/runs/984224611

TODO: Not what's wrong with PATH on CentOS 7:
/__w/ltp/ltp/lib/newlib_tests/runtest.sh: line 78: ./test05: No such file or directory
./shell/net/../../../../testcases/lib/tst_test.sh: line 149: tst_rod: command not found

Maybe it's in different directory?
It uses old make 3.82, there is something incompatible.

Kind regards,
Petr

Petr Vorel (5):
  tst_device: Require root
  lib: Add script for running tests
  make: Add make test{, -c, -shell} targets
  build.sh: Add support for make test
  CI: Run also make test

 .github/workflows/ci.yml      |   3 +
 Makefile                      |  10 +++
 build.sh                      |  18 ++++-
 lib/newlib_tests/runtest.sh   | 120 ++++++++++++++++++++++++++++++++++
 lib/newlib_tests/tst_device.c |   1 +
 5 files changed, 151 insertions(+), 1 deletion(-)
 create mode 100755 lib/newlib_tests/runtest.sh

-- 
2.32.0

[LTP] [PATCH v3 1/5] tst_device: Require root

[Petr Vorel]

as it's actually required for successful run:
tst_device.c:100: TINFO: Not allowed to open /dev/loop-control. Are you root?: EACCES (13)
tst_device.c:139: TINFO: No free devices found
tst_device.c:335: TBROK: Failed to acquire device

Signed-off-by: Petr Vorel <pvorel@suse.cz>
---
 lib/newlib_tests/tst_device.c | 1 +
 1 file changed, 1 insertion(+)

diff --git a/lib/newlib_tests/tst_device.c b/lib/newlib_tests/tst_device.c
index ad077affd..0bee0a939 100644
--- a/lib/newlib_tests/tst_device.c
+++ b/lib/newlib_tests/tst_device.c
@@ -40,6 +40,7 @@ static void do_test(void)
 }
 
 static struct tst_test test = {
+	.needs_root = 1,
 	.needs_device = 1,
 	.dev_min_size = 300,
 	.test_all = do_test,
-- 
2.32.0

[LTP] [PATCH v3 2/5] lib: Add script for running tests

[Petr Vorel]

For now run only tests which TPASS or TCONF.

Signed-off-by: Petr Vorel <pvorel@suse.cz>
---
Hi,

not sure if using same name for custom versions is a good idea.
Maybe I should have just used echo, but I plan to add other features to
runtest.sh, thus I try to keep usage similar to normal tests.
Maybe it's a wrong approach, making things less readable for new
contributors who will try to figure out why their change in test API
failed.

Kind regards,
Petr

 lib/newlib_tests/runtest.sh | 120 ++++++++++++++++++++++++++++++++++++
 1 file changed, 120 insertions(+)
 create mode 100755 lib/newlib_tests/runtest.sh

diff --git a/lib/newlib_tests/runtest.sh b/lib/newlib_tests/runtest.sh
new file mode 100755
index 000000000..b110a5876
--- /dev/null
+++ b/lib/newlib_tests/runtest.sh
@@ -0,0 +1,120 @@
+#!/bin/sh
+# Copyright (c) 2021 Petr Vorel <pvorel@suse.cz>
+
+LTP_C_API_TESTS="${LTP_C_API_TESTS:-test05 test07 test09 test12 test15 test16 test18
+test_exec test_timer tst_bool_expr tst_res_hexd tst_strstatus tst_fuzzy_sync02}"
+
+LTP_SHELL_API_TESTS="${LTP_SHELL_API_TESTS:-shell/tst_check_driver.sh shell/net/*.sh}"
+
+cd $(dirname $0)
+PATH="$PWD/../../testcases/lib/:$PATH"
+
+. tst_ansi_color.sh
+
+usage()
+{
+	echo "Usage: $0 [-c|-s]"
+	echo "-c    run C API tests only"
+	echo "-s    run shell API tests only"
+	echo "-h    print this help"
+}
+
+# custom version
+tst_flag2mask()
+{
+	case "$1" in
+	TPASS) return 0;;
+	TFAIL) return 1;;
+	TBROK) return 2;;
+	TWARN) return 4;;
+	TINFO) return 16;;
+	TCONF) return 32;;
+	esac
+}
+
+# custom version
+tst_res()
+{
+	if [ $# -eq 0 ]; then
+		echo >&2
+		return
+	fi
+
+	local res="$1"
+	shift
+
+	tst_color_enabled
+	local color=$?
+
+	printf "runtest " >&2
+	tst_print_colored $res "$res: " >&2
+	echo "$@" >&2
+
+}
+
+# custom version
+tst_brk()
+{
+	local res="$1"
+	shift
+
+	tst_res
+	tst_res $res $@
+
+	exit $(tst_flag2mask $res)
+}
+
+run_tests()
+{
+	local target="$1"
+	local i ret tconf tpass vars
+
+	eval vars="\$LTP_${target}_API_TESTS"
+
+	tst_res TINFO "=== Run $target tests ==="
+
+	for i in $vars; do
+		tst_res TINFO "* $i"
+		./$i
+		ret=$?
+
+		case $ret in
+			0) tpass="$tpass $i";;
+			1) tst_brk TFAIL "$i failed with TFAIL";;
+			2) tst_brk TFAIL "$i failed with TBROK";;
+			4) tst_brk TFAIL "$i failed with TWARN";;
+			32) tconf="$tconf $i";;
+		esac
+		tst_res
+	done
+
+	[ -z "$tpass" ] && tpass=" none"
+	[ -z "$tconf" ] && tconf=" none"
+
+	tst_res TINFO "=== $target TEST RESULTS ==="
+	tst_res TINFO "Tests exited with TPASS:$tpass"
+	tst_res TINFO "Tests exited with TCONF:$tconf"
+	tst_res
+}
+
+run=
+while getopts chs opt; do
+	case $opt in
+		'h') usage; exit 0;;
+		'c') run="c";;
+		's') run="s";;
+		*) usage; tst_brk TBROK "Error: invalid option";;
+	esac
+done
+
+tst_res TINFO "PATH='$PATH'"
+
+if [ -z "$run" -o "$run" = "c" ]; then
+	run_tests "C"
+fi
+
+if [ -z "$run" -o "$run" = "s" ]; then
+	run_tests "SHELL"
+fi
+
+tst_res TPASS "No test failed"
-- 
2.32.0

[LTP] [PATCH v3 3/5] make: Add make test{, -c, -shell} targets

[Petr Vorel]

For testing C and shell API.

Signed-off-by: Petr Vorel <pvorel@suse.cz>
---
Simplicity wins :).

 Makefile | 10 ++++++++++
 1 file changed, 10 insertions(+)

diff --git a/Makefile b/Makefile
index 56812d77b..7c1dba7e6 100644
--- a/Makefile
+++ b/Makefile
@@ -192,6 +192,16 @@ $(INSTALL_TARGETS): $(INSTALL_DIR) $(DESTDIR)/$(bindir)
 ## Install
 install: $(INSTALL_TARGETS)
 
+## Test
+test: lib-all
+	$(top_srcdir)/lib/newlib_tests/runtest.sh
+
+test-c: lib-all
+	$(top_srcdir)/lib/newlib_tests/runtest.sh -c
+
+test-shell: lib-all
+	$(top_srcdir)/lib/newlib_tests/runtest.sh -s
+
 ## Help
 .PHONY: help
 help:
-- 
2.32.0

[LTP] [PATCH v3 4/5] build.sh: Add support for make test

[Petr Vorel]

Signed-off-by: Petr Vorel <pvorel@suse.cz>
---
 build.sh | 18 +++++++++++++++++-
 1 file changed, 17 insertions(+), 1 deletion(-)

diff --git a/build.sh b/build.sh
index 240ce8e68..04459d6f9 100755
--- a/build.sh
+++ b/build.sh
@@ -119,6 +119,17 @@ build_out_tree()
 	make $MAKE_OPTS_OUT_TREE
 }
 
+test_in_tree()
+{
+	make test
+}
+
+test_out_tree()
+{
+	cd $BUILD_DIR
+	make $MAKE_OPTS_OUT_TREE test
+}
+
 install_in_tree()
 {
 	make $MAKE_OPTS install
@@ -165,6 +176,7 @@ RUN:
 autotools   run only 'make autotools'
 configure   run only 'configure'
 build       run only 'make'
+test        run only 'make test'
 install     run only 'make install'
 
 Default configure options:
@@ -192,7 +204,7 @@ while getopts "c:hio:p:r:t:" opt; do
 		esac;;
 	p) prefix="$OPTARG";;
 	r) case "$OPTARG" in
-		autotools|configure|build|install) run="$OPTARG";;
+		autotools|configure|build|test|install) run="$OPTARG";;
 		*) echo "Wrong run type '$OPTARG'" >&2; usage; exit 1;;
 		esac;;
 	t) case "$OPTARG" in
@@ -218,6 +230,10 @@ if [ -z "$run" -o "$run" = "build" ]; then
 	eval build_${tree}_tree
 fi
 
+if [ -z "$run" -o "$run" = "test" ]; then
+	eval test_${tree}_tree
+fi
+
 if [ -z "$run" -o "$run" = "install" ]; then
 	if [ "$install" = 1 ]; then
 		eval install_${tree}_tree
-- 
2.32.0

[LTP] [PATCH v3 5/5] CI: Run also make test

[Petr Vorel]

on all targets

Signed-off-by: Petr Vorel <pvorel@suse.cz>
---
 .github/workflows/ci.yml | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
index f67f14927..553d3c62b 100644
--- a/.github/workflows/ci.yml
+++ b/.github/workflows/ci.yml
@@ -151,6 +151,9 @@ jobs:
     - name: Compile
       run: ./build.sh -r build -o ${TREE:-in}
 
+    - name: Test
+      run: ./build.sh -r test -o ${TREE:-in}
+
     - name: Install
       run: |
         if [ "$MAKE_INSTALL" = 1 ]; then INSTALL_OPT="-i"; fi
-- 
2.32.0

[LTP] [PATCH v3 0/5] Run tests in CI

[Petr Vorel]

Hi,

> Hi all,

> changes v2->v3:
> * going back to simplest changes in make
> * adding lib/newlib_tests/runtest.sh (instead fighting with our build
>   system I concentrated on runtest.sh)
> * removing tst_fuzzy_sync01 as it sporadically fails

> Tested:
> https://github.com/pevik/ltp/actions/runs/984224611

> TODO: Not what's wrong with PATH on CentOS 7:
> /__w/ltp/ltp/lib/newlib_tests/runtest.sh: line 78: ./test05: No such file or directory
> ./shell/net/../../../../testcases/lib/tst_test.sh: line 149: tst_rod: command not found

> Maybe it's in different directory?
> It uses old make 3.82, there is something incompatible.
OK, whole out-of-tree build is broken. There are 2 issues:

1) probably PATH ../../testcases/lib/ points to srcdir instead of build dir).
It's just older make exit properly with 2, newer don't, not sure why.

2) The same issue with non-zero exit not being propagated on newer make
is also when runtest.sh quits with:
runtest TFAIL: shell/net/tst_ipaddr_un.sh failed with TFAIL

I guess I need to use @set -e; in Makefile (I didn't want to use set -e in the
runtest.sh itself, it would break printing printing what failed).

Kind regards,
Petr