[PATCH 0/7] selftests: report per-suite build results without failing the build

[Sasha Levin <sashal@kernel.org>]

Building the selftests succeeds as long as one target builds, so that a
single broken suite does not cost CI systems the coverage of all the
others. Commit cad6f32665cb ("selftests: Deescalate error reporting")
restored that policy after commit 7e47389142b8 ("selftests: Preserve
subtarget failures in all/install") had made any failure fatal. The
other side of the policy is that make cannot say which suites failed,
and CI systems fall back to guessing from the installed files, which
misses suites that installed their scripts but failed to compile and
flags suites that legitimately install nothing.

This series keeps the policy and adds an opt-in report instead:

 - Patch 1 fixes the exit status of the top-level loops, which wraps to
   success once eight or more targets fail.

 - Patch 2 adds KSFT_BUILD_REPORT. When it names an absolute file, the
   all and install rules append a "phase<TAB>target<TAB>status" record
   per target, without changing what is built or installed or the exit
   status of make.

 - Patches 3-7 apply the same policy to the suites that build their
   subdirectories in shell loops (futex, arm64, riscv, powerpc and
   sparc64). A failing subdirectory no longer hides behind the status
   of the last one, nor drops the whole suite from the installation,
   and each subdirectory is recorded as <suite>/<subdir>. FORCE_TARGETS
   keeps failing fast, now also inside these suites.

Patches 3-7 depend on patch 2 for the recording helper, so the series
is easiest to take as a whole; acks from the architecture and futex
maintainers would be appreciated.

Tested with mock suites for every converted loop and with native and
cross (arm64, riscv, powerpc) builds of the affected suites against the
base tree. A kselftest-install of 20 suites gives the same installation
and runlist as before, plus the report.

Sasha Levin (7):
  selftests: keep the build status from wrapping to success
  selftests: optionally record per-suite build and install results
  selftests/futex: report failures of the functional subdirectory
  kselftest/arm64: keep building and installing after a subdirectory
    fails
  selftests/riscv: keep building and installing after a subdirectory
    fails
  selftests/powerpc: report the outcome of each subdirectory
  selftests/sparc64: report failures of the drivers subdirectory

 Documentation/dev-tools/kselftest.rst        | 38 +++++++++++++++++
 tools/testing/selftests/Makefile             | 31 ++++++++++----
 tools/testing/selftests/arm64/Makefile       | 43 +++++++++++++-------
 tools/testing/selftests/build_report.mk      | 26 ++++++++++++
 tools/testing/selftests/futex/Makefile       | 32 ++++++++++-----
 tools/testing/selftests/powerpc/Makefile     | 20 ++++++---
 tools/testing/selftests/powerpc/pmu/Makefile | 20 ++++++---
 tools/testing/selftests/riscv/Makefile       | 39 +++++++++++++-----
 tools/testing/selftests/sparc64/Makefile     | 33 ++++++++++-----
 9 files changed, 218 insertions(+), 64 deletions(-)
 create mode 100644 tools/testing/selftests/build_report.mk


base-commit: 4eb2586bd40db27d77cc5b724ca9386ed0c71d53
-- 
2.53.0