[PATCH trivial] gitlab-ci/build-oss-fuzz: print FAILED marker in case the test failed and run all tests

[PATCH trivial] gitlab-ci/build-oss-fuzz: print FAILED marker in case the test failed and run all tests

[Michael Tokarev]

currently, if an oss-fuzz fails, the script does just `exit 1`
without any additional output, and looking at the build log in
the gitlab ci it is not clear what actually failed, without
looking at build-oss-fuzz script and seeing this `exit 1`.

Print easily recognizable error message about test failure, so
it becomes obvious what exactly has failed.

While at it, continue running other tests even in case of
failure, and exit non-zero if at least one test failed.

Signed-off-by: Michael Tokarev <mjt@tls.msk.ru>
---
 .gitlab-ci.d/buildtest.yml | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/.gitlab-ci.d/buildtest.yml b/.gitlab-ci.d/buildtest.yml
index 87848c2ffe..25c481e43a 100644
--- a/.gitlab-ci.d/buildtest.yml
+++ b/.gitlab-ci.d/buildtest.yml
@@ -628,12 +628,15 @@ build-oss-fuzz:
     - CC="clang" CXX="clang++" CFLAGS="-fsanitize=address"
       ./scripts/oss-fuzz/build.sh
     - export ASAN_OPTIONS="fast_unwind_on_malloc=0"
+    - failures=0
     - for fuzzer in $(find ./build-oss-fuzz/DEST_DIR/ -executable -type f
                       | grep -v slirp); do
         grep "LLVMFuzzerTestOneInput" ${fuzzer} > /dev/null 2>&1 || continue ;
         echo Testing ${fuzzer} ... ;
-        "${fuzzer}" -runs=1 -seed=1 || exit 1 ;
+        "${fuzzer}" -runs=1 -seed=1 || { echo "FAILED:"" ${fuzzer} exit code is $?"; failures=$(($failures+1)); };
       done
+    - echo "Number of failures:"" $failures"
+    - test $failures = 0
 
 build-tci:
   extends: .native_build_job_template
-- 
2.39.5

Re: [PATCH trivial] gitlab-ci/build-oss-fuzz: print FAILED marker in case the test failed and run all tests

[Thomas Huth]

On 03/10/2024 14.16, Michael Tokarev wrote:
> currently, if an oss-fuzz fails, the script does just `exit 1`
> without any additional output, and looking at the build log in
> the gitlab ci it is not clear what actually failed, without
> looking at build-oss-fuzz script and seeing this `exit 1`.
> 
> Print easily recognizable error message about test failure, so
> it becomes obvious what exactly has failed.
> 
> While at it, continue running other tests even in case of
> failure, and exit non-zero if at least one test failed.
> 
> Signed-off-by: Michael Tokarev <mjt@tls.msk.ru>
> ---
>   .gitlab-ci.d/buildtest.yml | 5 ++++-
>   1 file changed, 4 insertions(+), 1 deletion(-)
> 
> diff --git a/.gitlab-ci.d/buildtest.yml b/.gitlab-ci.d/buildtest.yml
> index 87848c2ffe..25c481e43a 100644
> --- a/.gitlab-ci.d/buildtest.yml
> +++ b/.gitlab-ci.d/buildtest.yml
> @@ -628,12 +628,15 @@ build-oss-fuzz:
>       - CC="clang" CXX="clang++" CFLAGS="-fsanitize=address"
>         ./scripts/oss-fuzz/build.sh
>       - export ASAN_OPTIONS="fast_unwind_on_malloc=0"
> +    - failures=0
>       - for fuzzer in $(find ./build-oss-fuzz/DEST_DIR/ -executable -type f
>                         | grep -v slirp); do
>           grep "LLVMFuzzerTestOneInput" ${fuzzer} > /dev/null 2>&1 || continue ;
>           echo Testing ${fuzzer} ... ;
> -        "${fuzzer}" -runs=1 -seed=1 || exit 1 ;
> +        "${fuzzer}" -runs=1 -seed=1 || { echo "FAILED:"" ${fuzzer} exit code is $?"; failures=$(($failures+1)); };
>         done
> +    - echo "Number of failures:"" $failures"
> +    - test $failures = 0
>   
>   build-tci:
>     extends: .native_build_job_template

Reviewed-by: Thomas Huth <thuth@redhat.com>

Re: [PATCH trivial] gitlab-ci/build-oss-fuzz: print FAILED marker in case the test failed and run all tests

[Alex Bennée]

Michael Tokarev <mjt@tls.msk.ru> writes:

> currently, if an oss-fuzz fails, the script does just `exit 1`
> without any additional output, and looking at the build log in
> the gitlab ci it is not clear what actually failed, without
> looking at build-oss-fuzz script and seeing this `exit 1`.
>
> Print easily recognizable error message about test failure, so
> it becomes obvious what exactly has failed.
>
> While at it, continue running other tests even in case of
> failure, and exit non-zero if at least one test failed.

Queued to testing/next, thanks.

-- 
Alex Bennée
Virtualisation Tech Lead @ Linaro