[LTP] [RFC PATCH 0/3] build: make check target

[LTP] [RFC PATCH 0/3] build: make check target

[Petr Vorel]

Hi,

actually, there are 3 targets: check, check-c, check-shell.

Previous attempt: make: Add make check target + run.sh [1].
This is built on the top of patchset tst_net.sh: Allow to skip
initialization [2]. Hope this time library check will made it :).

I plan to wrap tests with a script in the future, which will parse
expected output, i.e. approach from #312 [3].

It's not ready due several problems, but for curious here is a test run:
https://github.com/pevik/ltp/runs/2739826773

1) I made something wrong, I have no idea how to fix this error:
make[1]: Entering directory 'ltp/lib'
../include/mk/generic_trunk_target.inc:105: warning: overriding recipe for target 'check-c'
../include/mk/generic_leaf_target.inc:110: warning: ignoring old recipe for target 'check-c'
../include/mk/generic_trunk_target.inc:105: warning: overriding recipe for target 'check-shell'
../include/mk/generic_leaf_target.inc:118: warning: ignoring old recipe for target 'check-shell'

2) I obviously have problem with out-of-tree build
https://github.com/pevik/ltp/runs/2739826612?check_suite_focus=true
where path of shell tests (which are in src directory only) is obviously
broken:
make[2]: *** No rule to make target 'shell/tst_check_driver.sh', needed by 'check-shell'.  Stop.
20
make[1]: *** [/__w/ltp/ltp/include/mk/generic_trunk_target.inc:105: check-shell] Error 2
21
make[1]: Leaving directory '/__w/ltp/ltp-build/lib'

3)  tst_bool_expr fails on some systems when run via make check:
https://github.com/pevik/ltp/runs/2739826773?check_suite_focus=true
But it runs ok when running normally. It might be dash issue
(it's on Debian).

Kind regards,
Petr

[1] https://patchwork.ozlabs.org/project/ltp/patch/20190924182841.4528-1-pvorel@suse.cz/
[2] https://patchwork.ozlabs.org/project/ltp/list/?series=247078
[3] https://github.com/linux-test-project/ltp/issues/312

Petr Vorel (3):
  make: Add make check{,-c,-shell} targets
  build.sh: Add support for make check
  CI: Run also make check

 .github/workflows/ci.yml            |  3 +++
 Makefile                            | 21 ++++++++++++++++++++-
 build.sh                            | 18 +++++++++++++++++-
 include/mk/generic_leaf_target.inc  | 25 ++++++++++++++++++++++---
 include/mk/generic_trunk_target.inc |  4 ++--
 lib/newlib_tests/Makefile           |  8 ++++++++
 6 files changed, 72 insertions(+), 7 deletions(-)

-- 
2.31.1

[LTP] [RFC PATCH 1/3] make: Add make check{,-c,-shell} targets

[Petr Vorel]

For testing C and shell API.

Signed-off-by: Petr Vorel <pvorel@suse.cz>
---
 Makefile                            | 21 ++++++++++++++++++++-
 include/mk/generic_leaf_target.inc  | 25 ++++++++++++++++++++++---
 include/mk/generic_trunk_target.inc |  4 ++--
 lib/newlib_tests/Makefile           |  8 ++++++++
 4 files changed, 52 insertions(+), 6 deletions(-)

diff --git a/Makefile b/Makefile
index 56812d77b..acea4551e 100644
--- a/Makefile
+++ b/Makefile
@@ -75,10 +75,14 @@ endif
 INSTALL_TARGETS		+= $(COMMON_TARGETS)
 CLEAN_TARGETS		+= $(COMMON_TARGETS) lib libs
 BOOTSTRAP_TARGETS	:= $(sort $(COMMON_TARGETS) $(CLEAN_TARGETS) $(INSTALL_TARGETS))
+TEST_TARGETS		:= lib
+TEST_SHELL_TARGETS	:= lib
 
 CLEAN_TARGETS		:= $(addsuffix -clean,$(CLEAN_TARGETS))
 INSTALL_TARGETS		:= $(addsuffix -install,$(INSTALL_TARGETS))
 MAKE_TARGETS		:= $(addsuffix -all,$(filter-out lib,$(COMMON_TARGETS)))
+TEST_TARGETS		:= $(addsuffix -check-c,$(TEST_TARGETS))
+TEST_SHELL_TARGETS	:= $(addsuffix -check-shell,$(TEST_SHELL_TARGETS))
 
 # There's no reason why we should run `all' twice. Otherwise we're just wasting
 # 3+ mins of useful CPU cycles on a modern machine, and even more time on an
@@ -113,6 +117,14 @@ $(filter-out include-clean,$(CLEAN_TARGETS))::
 	-$(MAKE) -C "$(subst -clean,,$@)" \
 		 -f "$(abs_top_srcdir)/$(subst -clean,,$@)/Makefile" clean
 
+$(TEST_TARGETS): lib-all
+	$(MAKE) -C "$(subst -check-c,,$@)" \
+		-f "$(abs_top_srcdir)/$(subst -check-c,,$@)/Makefile" check-c
+
+$(TEST_SHELL_TARGETS):
+	$(MAKE) -C "$(subst -check-shell,,$@)" \
+		-f "$(abs_top_srcdir)/$(subst -check-shell,,$@)/Makefile" check-shell
+
 # Just like everything depends on include-all / -install, we need to get rid
 # of include last to ensure that things won't be monkey screwed up. Only do
 # this if we're invoking clean or a subclean directly though.
@@ -189,8 +201,15 @@ INSTALL_TARGETS		+= $(addprefix $(DESTDIR)/$(bindir)/,$(BINDIR_INSTALL_SCRIPTS))
 
 $(INSTALL_TARGETS): $(INSTALL_DIR) $(DESTDIR)/$(bindir)
 
+.PHONY: check check-c check-shell
+
+## Check
+check-c: $(TEST_TARGETS)
+check-shell: $(TEST_SHELL_TARGETS)
+check: check-c check-shell
+
 ## Install
-install: $(INSTALL_TARGETS)
+install: lib-all $(INSTALL_TARGETS)
 
 ## Help
 .PHONY: help
diff --git a/include/mk/generic_leaf_target.inc b/include/mk/generic_leaf_target.inc
index 64953f89a..d53623456 100644
--- a/include/mk/generic_leaf_target.inc
+++ b/include/mk/generic_leaf_target.inc
@@ -63,8 +63,12 @@
 #				  real files or directories). This will automatically append
 #				  adds the .o suffix to all files referenced by
 #				  $(MAKE_TARGETS)) to CLEAN_TARGETS, if MAKE_TARGETS wasn't
-#				  defined (see
-#				  $(MAKE_TARGETS)).
+#				  defined (see $(MAKE_TARGETS)).
+#
+# $(TEST_TARGETS)		: LTP C API tests.
+#
+# $(TEST_SHELL_TARGETS)		: LTP shell API tests.
+#
 # $(INSTALL_MODE)		: What mode should we using when calling
 # 				  install(1)?
 #
@@ -92,7 +96,7 @@
 # INSTALL_DIR			:= $(libdir)
 #
 
-.PHONY: all clean install
+.PHONY: all check-c check-shell clean install
 
 ifneq ($(strip $(MAKE_TARGETS)),)
 $(MAKE_TARGETS) += $(HOST_MAKE_TARGETS)
@@ -102,6 +106,21 @@ $(MAKE_TARGETS): | $(MAKE_DEPS)
 
 all: $(MAKE_TARGETS)
 
+check-c: $(TEST_TARGETS)
+	@set -e; echo; echo "===== Test C API ====="; \
+	for i in $(TEST_TARGETS); do \
+	    echo; echo "* $$i"; \
+	    echo "PATH $(top_srcdir)/testcases/lib:$$PATH"; \
+	    PATH="$(top_srcdir)/testcases/lib:$$PATH" ./$$i || [ $$? -eq 32 ]; \
+	done
+
+check-shell: $(TEST_SHELL_TARGETS)
+	@set -e; echo; echo "===== Test shell API ====="; \
+	for i in $(TEST_SHELL_TARGETS); do \
+	    echo; echo "* $$i"; \
+	    PATH="$(abs_top_srcdir)/testcases/lib:$(abs_top_builddir)/testcases/lib:$$PATH" $(abs_srcdir)/$$i || [ $$? -eq 32 ]; \
+	done
+
 clean:: $(CLEAN_DEPS)
 	-$(RM) -f -r $(CLEAN_TARGETS)
 
diff --git a/include/mk/generic_trunk_target.inc b/include/mk/generic_trunk_target.inc
index fc59f944f..d195fe3bd 100644
--- a/include/mk/generic_trunk_target.inc
+++ b/include/mk/generic_trunk_target.inc
@@ -48,7 +48,7 @@
 
 include $(top_srcdir)/include/mk/functions.mk
 
-RECURSIVE_TARGETS		?= all install
+RECURSIVE_TARGETS		?= all check check-c check-shell install
 
 $(eval $(get_make_dirs))
 
@@ -71,7 +71,7 @@ trunk-install: $(INSTALL_FILES)
 # Avoid creating duplicate .PHONY references to all, clean, and install. IIRC,
 # I've seen some indeterministic behavior when one does this in the past with
 # GNU Make...
-.PHONY: $(filter-out $(RECURSIVE_TARGETS),all clean install)
+.PHONY: $(filter-out $(RECURSIVE_TARGETS),all check clean install)
 all: trunk-all
 
 clean:: trunk-clean
diff --git a/lib/newlib_tests/Makefile b/lib/newlib_tests/Makefile
index 30ca6810c..182ef5689 100644
--- a/lib/newlib_tests/Makefile
+++ b/lib/newlib_tests/Makefile
@@ -19,5 +19,13 @@ ifeq ($(ANDROID),1)
 FILTER_OUT_MAKE_TARGETS	+= test08
 endif
 
+# C API tests (which exit 0)
+TEST_TARGETS += test0[579] test1[2568] test_exec test_timer
+TEST_TARGETS += tst_bool_expr tst_res_hexd tst_strstatus
+TEST_TARGETS += tst_fuzzy_sync0[1-2]
+
+# shell API tests (which exit 0)
+TEST_SHELL_TARGETS += shell/tst_check_driver.sh
+TEST_SHELL_TARGETS += shell/net/*.sh
 
 include $(top_srcdir)/include/mk/generic_leaf_target.mk
-- 
2.31.1

[LTP] [RFC PATCH 1/3] make: Add make check{,-c,-shell} targets

[Cyril Hrubis]

Hi!
> For testing C and shell API.
> 
> Signed-off-by: Petr Vorel <pvorel@suse.cz>
> ---
>  Makefile                            | 21 ++++++++++++++++++++-
>  include/mk/generic_leaf_target.inc  | 25 ++++++++++++++++++++++---
>  include/mk/generic_trunk_target.inc |  4 ++--
>  lib/newlib_tests/Makefile           |  8 ++++++++
>  4 files changed, 52 insertions(+), 6 deletions(-)
> 
> diff --git a/Makefile b/Makefile
> index 56812d77b..acea4551e 100644
> --- a/Makefile
> +++ b/Makefile
> @@ -75,10 +75,14 @@ endif
>  INSTALL_TARGETS		+= $(COMMON_TARGETS)
>  CLEAN_TARGETS		+= $(COMMON_TARGETS) lib libs
>  BOOTSTRAP_TARGETS	:= $(sort $(COMMON_TARGETS) $(CLEAN_TARGETS) $(INSTALL_TARGETS))
> +TEST_TARGETS		:= lib
> +TEST_SHELL_TARGETS	:= lib
>  
>  CLEAN_TARGETS		:= $(addsuffix -clean,$(CLEAN_TARGETS))
>  INSTALL_TARGETS		:= $(addsuffix -install,$(INSTALL_TARGETS))
>  MAKE_TARGETS		:= $(addsuffix -all,$(filter-out lib,$(COMMON_TARGETS)))
> +TEST_TARGETS		:= $(addsuffix -check-c,$(TEST_TARGETS))
> +TEST_SHELL_TARGETS	:= $(addsuffix -check-shell,$(TEST_SHELL_TARGETS))
>  
>  # There's no reason why we should run `all' twice. Otherwise we're just wasting
>  # 3+ mins of useful CPU cycles on a modern machine, and even more time on an
> @@ -113,6 +117,14 @@ $(filter-out include-clean,$(CLEAN_TARGETS))::
>  	-$(MAKE) -C "$(subst -clean,,$@)" \
>  		 -f "$(abs_top_srcdir)/$(subst -clean,,$@)/Makefile" clean
>  
> +$(TEST_TARGETS): lib-all
> +	$(MAKE) -C "$(subst -check-c,,$@)" \
> +		-f "$(abs_top_srcdir)/$(subst -check-c,,$@)/Makefile" check-c
> +
> +$(TEST_SHELL_TARGETS):
> +	$(MAKE) -C "$(subst -check-shell,,$@)" \
> +		-f "$(abs_top_srcdir)/$(subst -check-shell,,$@)/Makefile" check-shell
> +
>  # Just like everything depends on include-all / -install, we need to get rid
>  # of include last to ensure that things won't be monkey screwed up. Only do
>  # this if we're invoking clean or a subclean directly though.
> @@ -189,8 +201,15 @@ INSTALL_TARGETS		+= $(addprefix $(DESTDIR)/$(bindir)/,$(BINDIR_INSTALL_SCRIPTS))
>  
>  $(INSTALL_TARGETS): $(INSTALL_DIR) $(DESTDIR)/$(bindir)
>  
> +.PHONY: check check-c check-shell
> +
> +## Check
> +check-c: $(TEST_TARGETS)
> +check-shell: $(TEST_SHELL_TARGETS)
> +check: check-c check-shell
> +
>  ## Install
> -install: $(INSTALL_TARGETS)
> +install: lib-all $(INSTALL_TARGETS)
>  
>  ## Help
>  .PHONY: help
> diff --git a/include/mk/generic_leaf_target.inc b/include/mk/generic_leaf_target.inc
> index 64953f89a..d53623456 100644
> --- a/include/mk/generic_leaf_target.inc
> +++ b/include/mk/generic_leaf_target.inc
> @@ -63,8 +63,12 @@
>  #				  real files or directories). This will automatically append
>  #				  adds the .o suffix to all files referenced by
>  #				  $(MAKE_TARGETS)) to CLEAN_TARGETS, if MAKE_TARGETS wasn't
> -#				  defined (see
> -#				  $(MAKE_TARGETS)).
> +#				  defined (see $(MAKE_TARGETS)).
> +#
> +# $(TEST_TARGETS)		: LTP C API tests.
> +#
> +# $(TEST_SHELL_TARGETS)		: LTP shell API tests.
> +#
>  # $(INSTALL_MODE)		: What mode should we using when calling
>  # 				  install(1)?
>  #
> @@ -92,7 +96,7 @@
>  # INSTALL_DIR			:= $(libdir)
>  #
>  
> -.PHONY: all clean install
> +.PHONY: all check-c check-shell clean install
>  
>  ifneq ($(strip $(MAKE_TARGETS)),)
>  $(MAKE_TARGETS) += $(HOST_MAKE_TARGETS)
> @@ -102,6 +106,21 @@ $(MAKE_TARGETS): | $(MAKE_DEPS)
>  
>  all: $(MAKE_TARGETS)
>  
> +check-c: $(TEST_TARGETS)
> +	@set -e; echo; echo "===== Test C API ====="; \
> +	for i in $(TEST_TARGETS); do \
> +	    echo; echo "* $$i"; \
> +	    echo "PATH $(top_srcdir)/testcases/lib:$$PATH"; \
> +	    PATH="$(top_srcdir)/testcases/lib:$$PATH" ./$$i || [ $$? -eq 32 ]; \
> +	done
> +
> +check-shell: $(TEST_SHELL_TARGETS)
> +	@set -e; echo; echo "===== Test shell API ====="; \
> +	for i in $(TEST_SHELL_TARGETS); do \
> +	    echo; echo "* $$i"; \
> +	    PATH="$(abs_top_srcdir)/testcases/lib:$(abs_top_builddir)/testcases/lib:$$PATH" $(abs_srcdir)/$$i || [ $$? -eq 32 ]; \
> +	done
> +
>  clean:: $(CLEAN_DEPS)
>  	-$(RM) -f -r $(CLEAN_TARGETS)
>  
> diff --git a/include/mk/generic_trunk_target.inc b/include/mk/generic_trunk_target.inc
> index fc59f944f..d195fe3bd 100644
> --- a/include/mk/generic_trunk_target.inc
> +++ b/include/mk/generic_trunk_target.inc
> @@ -48,7 +48,7 @@
>  
>  include $(top_srcdir)/include/mk/functions.mk
>  
> -RECURSIVE_TARGETS		?= all install
> +RECURSIVE_TARGETS		?= all check check-c check-shell install

If you add a target to RECURSIVE_TARGETS such target automatically
traverses directories, that what the target is for. See the rule at the
end of the generic_trunk_target.inc. Which means that you redefine
targets if you define the same targets to loop and run the tests above.

I guess that you want add another target that depends on check targets
to actually run the tests?

Maybe:

check-c: check-c-run

check-c-run: $(TEST_TARGETS)
	for i in $(TEST_TARGETS); do
		...
	done

in order to run the tests.

In this case the check* targets would be used for directory traversal
and the check-c-run would be used to run the tests.

>  $(eval $(get_make_dirs))
>  
> @@ -71,7 +71,7 @@ trunk-install: $(INSTALL_FILES)
>  # Avoid creating duplicate .PHONY references to all, clean, and install. IIRC,
>  # I've seen some indeterministic behavior when one does this in the past with
>  # GNU Make...
> -.PHONY: $(filter-out $(RECURSIVE_TARGETS),all clean install)
> +.PHONY: $(filter-out $(RECURSIVE_TARGETS),all check clean install)

The check is already in RECURSIVE_TARGETS

>  all: trunk-all
>  
>  clean:: trunk-clean
> diff --git a/lib/newlib_tests/Makefile b/lib/newlib_tests/Makefile
> index 30ca6810c..182ef5689 100644
> --- a/lib/newlib_tests/Makefile
> +++ b/lib/newlib_tests/Makefile
> @@ -19,5 +19,13 @@ ifeq ($(ANDROID),1)
>  FILTER_OUT_MAKE_TARGETS	+= test08
>  endif
>  
> +# C API tests (which exit 0)
> +TEST_TARGETS += test0[579] test1[2568] test_exec test_timer
> +TEST_TARGETS += tst_bool_expr tst_res_hexd tst_strstatus
> +TEST_TARGETS += tst_fuzzy_sync0[1-2]
> +
> +# shell API tests (which exit 0)
> +TEST_SHELL_TARGETS += shell/tst_check_driver.sh
> +TEST_SHELL_TARGETS += shell/net/*.sh
>  
>  include $(top_srcdir)/include/mk/generic_leaf_target.mk
> -- 
> 2.31.1
> 

-- 
Cyril Hrubis
chrubis@suse.cz

[LTP] [RFC PATCH 1/3] make: Add make check{,-c,-shell} targets

[Petr Vorel]

Hi Cyril,

...
> > +++ b/include/mk/generic_trunk_target.inc
> > @@ -48,7 +48,7 @@

> >  include $(top_srcdir)/include/mk/functions.mk

> > -RECURSIVE_TARGETS		?= all install
> > +RECURSIVE_TARGETS		?= all check check-c check-shell install

> If you add a target to RECURSIVE_TARGETS such target automatically
> traverses directories, that what the target is for. See the rule at the
> end of the generic_trunk_target.inc. Which means that you redefine
> targets if you define the same targets to loop and run the tests above.
Ah, thanks!

> I guess that you want add another target that depends on check targets
> to actually run the tests?

> Maybe:

> check-c: check-c-run

> check-c-run: $(TEST_TARGETS)
> 	for i in $(TEST_TARGETS); do
> 		...
> 	done

> in order to run the tests.

> In this case the check* targets would be used for directory traversal
> and the check-c-run would be used to run the tests.

Sounds reasonable.

> >  $(eval $(get_make_dirs))

> > @@ -71,7 +71,7 @@ trunk-install: $(INSTALL_FILES)
> >  # Avoid creating duplicate .PHONY references to all, clean, and install. IIRC,
> >  # I've seen some indeterministic behavior when one does this in the past with
> >  # GNU Make...
> > -.PHONY: $(filter-out $(RECURSIVE_TARGETS),all clean install)
> > +.PHONY: $(filter-out $(RECURSIVE_TARGETS),all check clean install)

> The check is already in RECURSIVE_TARGETS
Thanks!

I'll send v2 on Monday, which also use test* as target name instead of check*
as we agreed on Richie's Libclang based analyzer patchset [1].

Kind regards,
Petr

[1] https://lists.linux.it/pipermail/ltp/2021-June/023008.html

[LTP] [RFC PATCH 1/3] make: Add make check{,-c,-shell} targets

[Enji Cooper]

> On Jun 3, 2021, at 11:38 AM, Petr Vorel <pvorel@suse.cz> wrote:
> 
> For testing C and shell API.

Why not just add a single ?make check? target that calls the C and shell targets to match what most open source projects do in terms of testing? Also, continuing on in light of errors seems like a good default behavior to have as it would prevent a single error/failure from gathering all other results.

My 2 cents,
-Enji
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: Message signed with OpenPGP
URL: <http://lists.linux.it/pipermail/ltp/attachments/20210604/1b27c9a2/attachment-0001.sig>

[LTP] [RFC PATCH 1/3] make: Add make check{,-c,-shell} targets

[Petr Vorel]

Hi Enji,

> > On Jun 3, 2021, at 11:38 AM, Petr Vorel <pvorel@suse.cz> wrote:

> > For testing C and shell API.

> Why not just add a single ?make check? target that calls the C and shell targets to match what most open source projects do in terms of testing? Also, continuing on in light of errors seems like a good default behavior to have as it would prevent a single error/failure from gathering all other results.

Yes, the original approach in this patchset is make check,
which consists of two make-c and make-shell. One of tst_fuzzy_sync checks is
quite long and in longer term the number of tests for both C and shell API might
grow to prolong the testing. Thus one may appreciate to test for local
development just one of these two.

Or, if you mean how it has been implemented (adding tests into variables),
I plan to add docparse tests (into docparse/tests/) - i.e. into different
directory.

Also, result of Richie's Libclang based analyzer patchset [1],
where he also added make check, we decided to use make test here [2].

> My 2 cents,
Thanks, comments are always welcome.
> -Enji

Kind regards,
Petr

[1] https://lists.linux.it/pipermail/ltp/2021-June/023008.html
[2] https://lists.linux.it/pipermail/ltp/2021-June/023021.html

[LTP] [RFC PATCH 2/3] build.sh: Add support for make check

[Petr Vorel]

Signed-off-by: Petr Vorel <pvorel@suse.cz>
---
 build.sh | 18 +++++++++++++++++-
 1 file changed, 17 insertions(+), 1 deletion(-)

diff --git a/build.sh b/build.sh
index 240ce8e68..c892fa87b 100755
--- a/build.sh
+++ b/build.sh
@@ -119,6 +119,17 @@ build_out_tree()
 	make $MAKE_OPTS_OUT_TREE
 }
 
+check_in_tree()
+{
+	make check
+}
+
+check_out_tree()
+{
+	cd $BUILD_DIR
+	make $MAKE_OPTS_OUT_TREE check
+}
+
 install_in_tree()
 {
 	make $MAKE_OPTS install
@@ -165,6 +176,7 @@ RUN:
 autotools   run only 'make autotools'
 configure   run only 'configure'
 build       run only 'make'
+check       run only 'make check'
 install     run only 'make install'
 
 Default configure options:
@@ -192,7 +204,7 @@ while getopts "c:hio:p:r:t:" opt; do
 		esac;;
 	p) prefix="$OPTARG";;
 	r) case "$OPTARG" in
-		autotools|configure|build|install) run="$OPTARG";;
+		autotools|configure|build|check|install) run="$OPTARG";;
 		*) echo "Wrong run type '$OPTARG'" >&2; usage; exit 1;;
 		esac;;
 	t) case "$OPTARG" in
@@ -218,6 +230,10 @@ if [ -z "$run" -o "$run" = "build" ]; then
 	eval build_${tree}_tree
 fi
 
+if [ -z "$run" -o "$run" = "check" ]; then
+	eval check_${tree}_tree
+fi
+
 if [ -z "$run" -o "$run" = "install" ]; then
 	if [ "$install" = 1 ]; then
 		eval install_${tree}_tree
-- 
2.31.1

[LTP] [RFC PATCH 3/3] CI: Run also make check

[Petr Vorel]

ona all targets

Signed-off-by: Petr Vorel <pvorel@suse.cz>
---
 .github/workflows/ci.yml | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
index 185f49e41..01e2578fa 100644
--- a/.github/workflows/ci.yml
+++ b/.github/workflows/ci.yml
@@ -151,6 +151,9 @@ jobs:
     - name: Compile
       run: ./build.sh -r build -o ${TREE:-in}
 
+    - name: Test
+      run: ./build.sh -r check -o ${TREE:-in}
+
     - name: Install
       run: |
         if [ "$MAKE_INSTALL" = 1 ]; then INSTALL_OPT="-i"; fi
-- 
2.31.1

[LTP] [RFC PATCH 0/3] build: make check target

[Richard Palethorpe]

Hello Petr,

Petr Vorel <pvorel@suse.cz> writes:

> Hi,
>
> actually, there are 3 targets: check, check-c, check-shell.

Usually, if you only want to make a sub group of tests. Then you cd into
the sub directory and do 'make'.

So why do we need three targets?

Just an idea, but maybe in a future patch this could run some actual
tests as well? A lot of the old syscall tests would be fine to run in
CI. More likely if they fail it is due to LTP library changes than a
kernel bug on the host.

>
> Previous attempt: make: Add make check target + run.sh [1].
> This is built on the top of patchset tst_net.sh: Allow to skip
> initialization [2]. Hope this time library check will made it :).
>
> I plan to wrap tests with a script in the future, which will parse
> expected output, i.e. approach from #312 [3].
>
> It's not ready due several problems, but for curious here is a test run:
> https://github.com/pevik/ltp/runs/2739826773
>
> 1) I made something wrong, I have no idea how to fix this error:
> make[1]: Entering directory 'ltp/lib'
> ../include/mk/generic_trunk_target.inc:105: warning: overriding recipe for target 'check-c'
> ../include/mk/generic_leaf_target.inc:110: warning: ignoring old recipe for target 'check-c'
> ../include/mk/generic_trunk_target.inc:105: warning: overriding recipe for target 'check-shell'
> ../include/mk/generic_leaf_target.inc:118: warning: ignoring old recipe for target 'check-shell'
>
> 2) I obviously have problem with out-of-tree build
> https://github.com/pevik/ltp/runs/2739826612?check_suite_focus=true
> where path of shell tests (which are in src directory only) is obviously
> broken:
> make[2]: *** No rule to make target 'shell/tst_check_driver.sh', needed by 'check-shell'.  Stop.
> 20
> make[1]: *** [/__w/ltp/ltp/include/mk/generic_trunk_target.inc:105: check-shell] Error 2
> 21
> make[1]: Leaving directory '/__w/ltp/ltp-build/lib'
>
> 3)  tst_bool_expr fails on some systems when run via make check:
> https://github.com/pevik/ltp/runs/2739826773?check_suite_focus=true
> But it runs ok when running normally. It might be dash issue
> (it's on Debian).
>
> Kind regards,
> Petr
>
> [1] https://patchwork.ozlabs.org/project/ltp/patch/20190924182841.4528-1-pvorel@suse.cz/
> [2] https://patchwork.ozlabs.org/project/ltp/list/?series=247078
> [3] https://github.com/linux-test-project/ltp/issues/312
>
> Petr Vorel (3):
>   make: Add make check{,-c,-shell} targets
>   build.sh: Add support for make check
>   CI: Run also make check
>
>  .github/workflows/ci.yml            |  3 +++
>  Makefile                            | 21 ++++++++++++++++++++-
>  build.sh                            | 18 +++++++++++++++++-
>  include/mk/generic_leaf_target.inc  | 25 ++++++++++++++++++++++---
>  include/mk/generic_trunk_target.inc |  4 ++--
>  lib/newlib_tests/Makefile           |  8 ++++++++
>  6 files changed, 72 insertions(+), 7 deletions(-)
>
> -- 
> 2.31.1


-- 
Thank you,
Richard.