[PATCH] selftests: use $SHELL to exec selftests

[PATCH] selftests: use $SHELL to exec selftests

[Luis R. Rodriguez]

Executing selftests is fragile as if someone forgot to set a secript
as executable the test will fail. Setting scripts as executable is
desirable to enable folks to execute tests as independent units,
however, we can avoid the fragile errors of forgetting to set the
script as executable by just invoking the $SHELL for running each
script.

Suggsted-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Luis R. Rodriguez <mcgrof@kernel.org>
---

Shuah, while the last two patches could be queued in for 4.13-final,
this one I think is more appropriate for v4.14-rc1 only.

 tools/testing/selftests/lib.mk | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/tools/testing/selftests/lib.mk b/tools/testing/selftests/lib.mk
index 959273c3a52e..2d6abb8037be 100644
--- a/tools/testing/selftests/lib.mk
+++ b/tools/testing/selftests/lib.mk
@@ -14,7 +14,7 @@ all: $(TEST_GEN_PROGS) $(TEST_GEN_PROGS_EXTENDED) $(TEST_GEN_FILES)
 define RUN_TESTS
 	@for TEST in $(TEST_GEN_PROGS) $(TEST_PROGS); do \
 		BASENAME_TEST=`basename $$TEST`;	\
-		cd `dirname $$TEST`; (./$$BASENAME_TEST && echo "selftests: $$BASENAME_TEST [PASS]") || echo "selftests:  $$BASENAME_TEST [FAIL]"; cd -;\
+		cd `dirname $$TEST`; ($$SHELL ./$$BASENAME_TEST && echo "selftests: $$BASENAME_TEST [PASS]") || echo "selftests:  $$BASENAME_TEST [FAIL]"; cd -;\
 	done;
 endef
 
-- 
2.11.0

Re: [PATCH] selftests: use $SHELL to exec selftests

[Kees Cook]

On Thu, Aug 3, 2017 at 9:59 AM, Luis R. Rodriguez <mcgrof@kernel.org> wrote:
> Executing selftests is fragile as if someone forgot to set a secript
> as executable the test will fail. Setting scripts as executable is
> desirable to enable folks to execute tests as independent units,
> however, we can avoid the fragile errors of forgetting to set the
> script as executable by just invoking the $SHELL for running each
> script.
>
> Suggsted-by: Andrew Morton <akpm@linux-foundation.org>
> Signed-off-by: Luis R. Rodriguez <mcgrof@kernel.org>
> ---
>
> Shuah, while the last two patches could be queued in for 4.13-final,
> this one I think is more appropriate for v4.14-rc1 only.
>
>  tools/testing/selftests/lib.mk | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/tools/testing/selftests/lib.mk b/tools/testing/selftests/lib.mk
> index 959273c3a52e..2d6abb8037be 100644
> --- a/tools/testing/selftests/lib.mk
> +++ b/tools/testing/selftests/lib.mk
> @@ -14,7 +14,7 @@ all: $(TEST_GEN_PROGS) $(TEST_GEN_PROGS_EXTENDED) $(TEST_GEN_FILES)
>  define RUN_TESTS
>         @for TEST in $(TEST_GEN_PROGS) $(TEST_PROGS); do \
>                 BASENAME_TEST=`basename $$TEST`;        \
> -               cd `dirname $$TEST`; (./$$BASENAME_TEST && echo "selftests: $$BASENAME_TEST [PASS]") || echo "selftests:  $$BASENAME_TEST [FAIL]"; cd -;\
> +               cd `dirname $$TEST`; ($$SHELL ./$$BASENAME_TEST && echo "selftests: $$BASENAME_TEST [PASS]") || echo "selftests:  $$BASENAME_TEST [FAIL]"; cd -;\
>         done;
>  endef

Is BASENAME_TEST always a script? Can't it be a built binary too?

-Kees

-- 
Kees Cook
Pixel Security

Re: [PATCH] selftests: use $SHELL to exec selftests

[Luis R. Rodriguez]

On Thu, Aug 03, 2017 at 10:27:33AM -0700, Kees Cook wrote:
> On Thu, Aug 3, 2017 at 9:59 AM, Luis R. Rodriguez <mcgrof@kernel.org> wrote:
> > Executing selftests is fragile as if someone forgot to set a secript
> > as executable the test will fail. Setting scripts as executable is
> > desirable to enable folks to execute tests as independent units,
> > however, we can avoid the fragile errors of forgetting to set the
> > script as executable by just invoking the $SHELL for running each
> > script.
> >
> > Suggsted-by: Andrew Morton <akpm@linux-foundation.org>
> > Signed-off-by: Luis R. Rodriguez <mcgrof@kernel.org>
> > ---
> >
> > Shuah, while the last two patches could be queued in for 4.13-final,
> > this one I think is more appropriate for v4.14-rc1 only.
> >
> >  tools/testing/selftests/lib.mk | 2 +-
> >  1 file changed, 1 insertion(+), 1 deletion(-)
> >
> > diff --git a/tools/testing/selftests/lib.mk b/tools/testing/selftests/lib.mk
> > index 959273c3a52e..2d6abb8037be 100644
> > --- a/tools/testing/selftests/lib.mk
> > +++ b/tools/testing/selftests/lib.mk
> > @@ -14,7 +14,7 @@ all: $(TEST_GEN_PROGS) $(TEST_GEN_PROGS_EXTENDED) $(TEST_GEN_FILES)
> >  define RUN_TESTS
> >         @for TEST in $(TEST_GEN_PROGS) $(TEST_PROGS); do \
> >                 BASENAME_TEST=`basename $$TEST`;        \
> > -               cd `dirname $$TEST`; (./$$BASENAME_TEST && echo "selftests: $$BASENAME_TEST [PASS]") || echo "selftests:  $$BASENAME_TEST [FAIL]"; cd -;\
> > +               cd `dirname $$TEST`; ($$SHELL ./$$BASENAME_TEST && echo "selftests: $$BASENAME_TEST [PASS]") || echo "selftests:  $$BASENAME_TEST [FAIL]"; cd -;\
> >         done;
> >  endef
> 
> Is BASENAME_TEST always a script? Can't it be a built binary too?

True, this should only be attempted then if the file is *not* a shell script.
Furthermore any interpreter could be used, so using a shell would not work
for all anyway. So this would be wrong too:

diff --git a/tools/testing/selftests/lib.mk b/tools/testing/selftests/lib.mk
index 959273c3a52e..e963ee375d77 100644
--- a/tools/testing/selftests/lib.mk
+++ b/tools/testing/selftests/lib.mk
@@ -14,7 +14,12 @@ all: $(TEST_GEN_PROGS) $(TEST_GEN_PROGS_EXTENDED) $(TEST_GEN_FILES)
 define RUN_TESTS
 	@for TEST in $(TEST_GEN_PROGS) $(TEST_PROGS); do \
 		BASENAME_TEST=`basename $$TEST`;	\
-		cd `dirname $$TEST`; (./$$BASENAME_TEST && echo "selftests: $$BASENAME_TEST [PASS]") || echo "selftests:  $$BASENAME_TEST [FAIL]"; cd -;\
+		USE_SHELL="";				\
+		if [ ! -x $$BASENAME_TEST ]; then	\
+			USE_SHELL="$$SHELL";		\
+			echo "Warning: file $$BASENAME_TEST is not executable, correct this. Assuming we can run it as a shell script";\
+		fi;					\
+		cd `dirname $$TEST`; ($$USE_SHELL ./$$BASENAME_TEST && echo "selftests: $$BASENAME_TEST [PASS]") || echo "selftests:  $$BASENAME_TEST [FAIL]"; cd -;\
 	done;
 endef
 
Just distinguishing between an ELF binary and an interpreted script seems
rather hacky here too -- even if say we use head -c 2 and grep for "#!" -- this
all just seems convoluted for a Makefile. As such for now I think its best we
just inform the user that the reason why a test failed was the lack of the
executable bit.

Will send patch follow up as v2.

  Luis