[LTP] [RFC PATCH 0/2] make test: Run C API tests

[LTP] [RFC PATCH 0/2] make test: Run C API tests

[Petr Vorel]

Hi,

this is very simple way to add 'make test' target.  It seems tests are
outdated and some of them maybe not meant to be unit shell tests.  I
removed tst_safe_macros as it's failing and not sufficient enough. We
should write proper test for it.

Current implementation is still failing: after 'make test' run, there is
still failing tst_checkpoint_wait_timeout.

tst_checkpoint_wait_timeout    1  TBROK  :  tst_checkpoint.c:144: tst_checkpoint_wait_timeout.c:46: tst_checkpoint_wait(0, 10000): errno=ETIMEDOUT(110): Connection timed out
tst_checkpoint_wait_timeout    2  TBROK  :  tst_checkpoint.c:144: Remaining cases broken


Kind regards,
Petr

Petr Vorel (2):
  tests: Remove tst_safe_macros.c
  make: Add test target

 Makefile                    |  3 +++
 lib/tests/Makefile          |  3 +++
 lib/tests/test.sh           | 32 +++++++++++++++++++++++++++++
 lib/tests/tst_safe_macros.c | 40 -------------------------------------
 4 files changed, 38 insertions(+), 40 deletions(-)
 create mode 100755 lib/tests/test.sh
 delete mode 100644 lib/tests/tst_safe_macros.c

-- 
2.18.0

[LTP] [RFC PATCH 1/2] tests: Remove tst_safe_macros.c

[Petr Vorel]

It's broken and outdated (doesn't really test safe macros features).

Signed-off-by: Petr Vorel <pvorel@suse.cz>
---
 lib/tests/tst_safe_macros.c | 40 -------------------------------------
 1 file changed, 40 deletions(-)
 delete mode 100644 lib/tests/tst_safe_macros.c

diff --git a/lib/tests/tst_safe_macros.c b/lib/tests/tst_safe_macros.c
deleted file mode 100644
index b5809f40d..000000000
--- a/lib/tests/tst_safe_macros.c
+++ /dev/null
@@ -1,40 +0,0 @@
-#include "test.h"
-#include "safe_macros.h"
-
-char *TCID = "test_safe_macros";
-int TST_TOTAL = 1;
-
-int fd = -1;
-
-void cleanup(void)
-{
-	SAFE_CLOSE(NULL, fd);
-	SAFE_UNLINK(NULL, __FILE__ "~");
-	tst_resm(TINFO, "got here");
-}
-
-int main(int argc LTP_ATTRIBUTE_UNUSED, char **argv)
-{
-	char buf[10];
-	int fds[2];
-
-	buf[9] = '\0';
-
-	if (system("cp " __FILE__ " " __FILE__ "~")) {
-		fprintf(stderr, "error: could not cp file\n");
-		return 1;
-	}
-	printf("%s\n", SAFE_BASENAME(NULL, *argv));
-	printf("%s\n", SAFE_DIRNAME(NULL, *argv));
-	fd = SAFE_OPEN(cleanup, __FILE__ "~", O_RDWR);
-	SAFE_READ(cleanup, 0, fd, buf, 9);
-	printf("buf: %s\n", buf);
-	SAFE_READ(cleanup, 1, fd, buf, 9);
-	printf("buf: %s\n", buf);
-	SAFE_WRITE(cleanup, 0, -1, buf, 9);
-	SAFE_WRITE(NULL, 0, fd, buf, 9);
-	SAFE_WRITE(NULL, 1, fd, buf, 9);
-	SAFE_PIPE(NULL, fds);
-
-	return 0;
-}
-- 
2.18.0

[LTP] [RFC PATCH 2/2] make: Add test target

[Petr Vorel]

Signed-off-by: Petr Vorel <pvorel@suse.cz>
---
 Makefile           |  3 +++
 lib/tests/Makefile |  3 +++
 lib/tests/test.sh  | 32 ++++++++++++++++++++++++++++++++
 3 files changed, 38 insertions(+)
 create mode 100755 lib/tests/test.sh

diff --git a/Makefile b/Makefile
index b0368a472..f886ac350 100644
--- a/Makefile
+++ b/Makefile
@@ -89,6 +89,9 @@ MAKE_TARGETS		:= $(addsuffix -all,$(filter-out lib,$(COMMON_TARGETS)))
 # overtaxed one, or one where -j => 1 was specified.
 all: $(addsuffix -all,$(COMMON_TARGETS)) Version
 
+test:
+	cd lib/tests && make $@
+
 $(MAKE_TARGETS): lib-all
 
 .PHONY: include-all include-install
diff --git a/lib/tests/Makefile b/lib/tests/Makefile
index 73a0f1655..732fee2c4 100644
--- a/lib/tests/Makefile
+++ b/lib/tests/Makefile
@@ -7,4 +7,7 @@ LDLIBS			+= -lltp
 
 tst_cleanup_once: CFLAGS += -pthread
 
+test:
+	./test.sh $(MAKE_TARGETS)
+
 include $(top_srcdir)/include/mk/generic_leaf_target.mk
diff --git a/lib/tests/test.sh b/lib/tests/test.sh
new file mode 100755
index 000000000..81d57a81b
--- /dev/null
+++ b/lib/tests/test.sh
@@ -0,0 +1,32 @@
+#!/bin/sh
+# SPDX-License-Identifier: GPL-2.0-or-later
+# Copyright (c) 2016-2018 Petr Vorel <pvorel@suse.cz>
+
+set -e
+
+for i in $@; do
+    echo "=== Testing '$i' ==="
+    case $i in
+        tst_checkpoint_wake_timeout|tst_record_childstatus)
+        if [ "$i" = "tst_record_childstatus" ]; then
+            echo "NOTE: expecting fail the test"
+            ./$i || [ $? -eq 1 ]
+        fi
+        ;;
+        tst_device)
+            if [ $(id -u) -ne 0 ]; then
+                echo "WARN: not root, skip the test"
+            else
+                ./$i
+            fi
+        ;;
+        *)
+        ./$i
+        ;;
+    esac
+    echo
+done
+
+echo "END OF TESTING"
+
+# vim: set ft=sh ts=4 sts=4 sw=4 expandtab :
-- 
2.18.0

[LTP] [RFC PATCH 2/2] make: Add test target

[Jan Stancek]

----- Original Message -----
> Signed-off-by: Petr Vorel <pvorel@suse.cz>
> ---
>  Makefile           |  3 +++
>  lib/tests/Makefile |  3 +++
>  lib/tests/test.sh  | 32 ++++++++++++++++++++++++++++++++
>  3 files changed, 38 insertions(+)
>  create mode 100755 lib/tests/test.sh
> +
> +for i in $@; do
> +    echo "=== Testing '$i' ==="
> +    case $i in
> +        tst_checkpoint_wake_timeout|tst_record_childstatus)
> +        if [ "$i" = "tst_record_childstatus" ]; then
> +            echo "NOTE: expecting fail the test"
> +            ./$i || [ $? -eq 1 ]
> +        fi

I'd prefer we fix the tests and make them return 0 if everything goes as expected.

Regards,
Jan

[LTP] [RFC PATCH 2/2] make: Add test target

[Cyril Hrubis]

Hi!
> diff --git a/Makefile b/Makefile
> index b0368a472..f886ac350 100644
> --- a/Makefile
> +++ b/Makefile
> @@ -89,6 +89,9 @@ MAKE_TARGETS		:= $(addsuffix -all,$(filter-out lib,$(COMMON_TARGETS)))
>  # overtaxed one, or one where -j => 1 was specified.
>  all: $(addsuffix -all,$(COMMON_TARGETS)) Version
>  
> +test:
> +	cd lib/tests && make $@
> +
>  $(MAKE_TARGETS): lib-all
>  
>  .PHONY: include-all include-install
> diff --git a/lib/tests/Makefile b/lib/tests/Makefile
> index 73a0f1655..732fee2c4 100644
> --- a/lib/tests/Makefile
> +++ b/lib/tests/Makefile
> @@ -7,4 +7,7 @@ LDLIBS			+= -lltp
>  
>  tst_cleanup_once: CFLAGS += -pthread
>  
> +test:
> +	./test.sh $(MAKE_TARGETS)
> +
>  include $(top_srcdir)/include/mk/generic_leaf_target.mk
> diff --git a/lib/tests/test.sh b/lib/tests/test.sh
> new file mode 100755
> index 000000000..81d57a81b
> --- /dev/null
> +++ b/lib/tests/test.sh
> @@ -0,0 +1,32 @@
> +#!/bin/sh
> +# SPDX-License-Identifier: GPL-2.0-or-later
> +# Copyright (c) 2016-2018 Petr Vorel <pvorel@suse.cz>
> +
> +set -e
> +
> +for i in $@; do
> +    echo "=== Testing '$i' ==="
> +    case $i in
> +        tst_checkpoint_wake_timeout|tst_record_childstatus)
> +        if [ "$i" = "tst_record_childstatus" ]; then
> +            echo "NOTE: expecting fail the test"
> +            ./$i || [ $? -eq 1 ]
> +        fi
> +        ;;
> +        tst_device)
> +            if [ $(id -u) -ne 0 ]; then
> +                echo "WARN: not root, skip the test"
> +            else
> +                ./$i
> +            fi
> +        ;;

Listing the testcases does not scale that much. I wonder how we can do
better.

Also new library tests are stored in newlib_tests directory.

> +        *)
> +        ./$i
> +        ;;
> +    esac
> +    echo
> +done
> +
> +echo "END OF TESTING"
> +
> +# vim: set ft=sh ts=4 sts=4 sw=4 expandtab :

I wonder if we can try to check output of these tests against expected
output. Most of the testcases have nice and stable output that is always
the same. Some of the include pid in the test output that will change
with each iteration and some, for instance the thread safety related
tests for the new library, produce the output lines in random order.

So maybe the solution would be grepping for pre-defined patterns in the
test output in the case that the test output is not stable.

-- 
Cyril Hrubis
chrubis@suse.cz

[LTP] [RFC PATCH 2/2] make: Add test target

[Petr Vorel]

Hi Jan,

> ----- Original Message -----
> > Signed-off-by: Petr Vorel <pvorel@suse.cz>
> > ---
> >  Makefile           |  3 +++
> >  lib/tests/Makefile |  3 +++
> >  lib/tests/test.sh  | 32 ++++++++++++++++++++++++++++++++
> >  3 files changed, 38 insertions(+)
> >  create mode 100755 lib/tests/test.sh
> > +
> > +for i in $@; do
> > +    echo "=== Testing '$i' ==="
> > +    case $i in
> > +        tst_checkpoint_wake_timeout|tst_record_childstatus)
> > +        if [ "$i" = "tst_record_childstatus" ]; then
> > +            echo "NOTE: expecting fail the test"
> > +            ./$i || [ $? -eq 1 ]
> > +        fi

> I'd prefer we fix the tests and make them return 0 if everything goes as expected.
Agree. But I took different folder. As Cyril pointed out, tests in lib/tests/
are for legacy API. I'm going to test new ones from lib/newlib_tests/.

As Christian works on #312 "Unit testing the shell library" [1], I'd like if
both shell and C tests had the same approach of testing.

I wonder whether to keep these tests in lib/tests/. At least some of them aren't
meant to be for unit testing (due expecting failing), nobody runs them, ...

[1] https://github.com/linux-test-project/ltp/issues/312

> Regards,
> Jan