[PATCH 0/7] selftests: report per-suite build results without failing the build

[PATCH 0/7] selftests: report per-suite build results without failing the build

[Sasha Levin]

Building the selftests succeeds as long as one target builds, so that a
single broken suite does not cost CI systems the coverage of all the
others. Commit cad6f32665cb ("selftests: Deescalate error reporting")
restored that policy after commit 7e47389142b8 ("selftests: Preserve
subtarget failures in all/install") had made any failure fatal. The
other side of the policy is that make cannot say which suites failed,
and CI systems fall back to guessing from the installed files, which
misses suites that installed their scripts but failed to compile and
flags suites that legitimately install nothing.

This series keeps the policy and adds an opt-in report instead:

 - Patch 1 fixes the exit status of the top-level loops, which wraps to
   success once eight or more targets fail.

 - Patch 2 adds KSFT_BUILD_REPORT. When it names an absolute file, the
   all and install rules append a "phase<TAB>target<TAB>status" record
   per target, without changing what is built or installed or the exit
   status of make.

 - Patches 3-7 apply the same policy to the suites that build their
   subdirectories in shell loops (futex, arm64, riscv, powerpc and
   sparc64). A failing subdirectory no longer hides behind the status
   of the last one, nor drops the whole suite from the installation,
   and each subdirectory is recorded as <suite>/<subdir>. FORCE_TARGETS
   keeps failing fast, now also inside these suites.

Patches 3-7 depend on patch 2 for the recording helper, so the series
is easiest to take as a whole; acks from the architecture and futex
maintainers would be appreciated.

Tested with mock suites for every converted loop and with native and
cross (arm64, riscv, powerpc) builds of the affected suites against the
base tree. A kselftest-install of 20 suites gives the same installation
and runlist as before, plus the report.

Sasha Levin (7):
  selftests: keep the build status from wrapping to success
  selftests: optionally record per-suite build and install results
  selftests/futex: report failures of the functional subdirectory
  kselftest/arm64: keep building and installing after a subdirectory
    fails
  selftests/riscv: keep building and installing after a subdirectory
    fails
  selftests/powerpc: report the outcome of each subdirectory
  selftests/sparc64: report failures of the drivers subdirectory

 Documentation/dev-tools/kselftest.rst        | 38 +++++++++++++++++
 tools/testing/selftests/Makefile             | 31 ++++++++++----
 tools/testing/selftests/arm64/Makefile       | 43 +++++++++++++-------
 tools/testing/selftests/build_report.mk      | 26 ++++++++++++
 tools/testing/selftests/futex/Makefile       | 32 ++++++++++-----
 tools/testing/selftests/powerpc/Makefile     | 20 ++++++---
 tools/testing/selftests/powerpc/pmu/Makefile | 20 ++++++---
 tools/testing/selftests/riscv/Makefile       | 39 +++++++++++++-----
 tools/testing/selftests/sparc64/Makefile     | 33 ++++++++++-----
 9 files changed, 218 insertions(+), 64 deletions(-)
 create mode 100644 tools/testing/selftests/build_report.mk


base-commit: 4eb2586bd40db27d77cc5b724ca9386ed0c71d53
-- 
2.53.0

[PATCH 6/7] selftests/powerpc: report the outcome of each subdirectory

[Sasha Levin]

The powerpc suite and its pmu subdirectory install and list their tests
in shell loops whose status is that of the last subdirectory, so a
failure in any other subdirectory is lost and FORCE_TARGETS has no
effect on them.

Make the installation and runlist loops fail only when no subdirectory
succeeds, which matches how the top-level selftests Makefile treats its
targets, and stop at the first failure when FORCE_TARGETS is set. When a
build report is requested, record each subdirectory as powerpc/<subdir>
or powerpc/pmu/<subdir>, including the builds of the subdirectories, so
that a failure is visible even when make itself succeeds.

The subdirectories are still built as prerequisites of "all", so a
subdirectory that fails to build still fails the build of the suite as
before; only the reporting of that failure is added.

Fixes: 6faeeea44b84 ("selftests: Add install support for the powerpc tests")
Fixes: 822a04957cc5 ("selftests/powerpc: Convert pmu Makefile to for loop style")
Signed-off-by: Sasha Levin <sashal@kernel.org>
---
 tools/testing/selftests/powerpc/Makefile     | 20 +++++++++++++++-----
 tools/testing/selftests/powerpc/pmu/Makefile | 20 +++++++++++++++-----
 2 files changed, 30 insertions(+), 10 deletions(-)

diff --git a/tools/testing/selftests/powerpc/Makefile b/tools/testing/selftests/powerpc/Makefile
index b175e94e1901d..8349196c934d5 100644
--- a/tools/testing/selftests/powerpc/Makefile
+++ b/tools/testing/selftests/powerpc/Makefile
@@ -37,9 +37,11 @@ endif
 all: $(SUB_DIRS)
 
 $(SUB_DIRS):
-	BUILD_TARGET=$(OUTPUT)/$@; mkdir -p $$BUILD_TARGET; $(MAKE) OUTPUT=$$BUILD_TARGET -k -C $@ all
+	BUILD_TARGET=$(OUTPUT)/$@; mkdir -p $$BUILD_TARGET && $(MAKE) OUTPUT=$$BUILD_TARGET -k -C $@ all; \
+	status=$$?; $(call ksft_record_build_status,all,powerpc/$@); exit $$status
 
 include ../lib.mk
+include $(selfdir)/build_report.mk
 include ./flags.mk
 
 override define RUN_TESTS
@@ -50,17 +52,25 @@ override define RUN_TESTS
 endef
 
 override define INSTALL_RULE
-	+@for TARGET in $(SUB_DIRS); do \
+	+@ret=$(if $(SUB_DIRS),1,0); for TARGET in $(SUB_DIRS); do \
 		BUILD_TARGET=$(OUTPUT)/$$TARGET;	\
 		$(MAKE) OUTPUT=$$BUILD_TARGET INSTALL_PATH=$$INSTALL_PATH/$$TARGET -C $$TARGET install;\
-	done;
+		status=$$?;				\
+		$(call ksft_record_build_status,install,powerpc/$$TARGET); \
+		$(if $(FORCE_TARGETS),[ $$status -eq 0 ] || exit $$status;) \
+		[ $$status -eq 0 ] && ret=0;		\
+	done; exit $$ret
 endef
 
 emit_tests:
-	+@for TARGET in $(SUB_DIRS); do \
+	+@ret=$(if $(SUB_DIRS),1,0); for TARGET in $(SUB_DIRS); do \
 		BUILD_TARGET=$(OUTPUT)/$$TARGET;	\
 		$(MAKE) OUTPUT=$$BUILD_TARGET COLLECTION=$(COLLECTION)/$$TARGET -s -C $$TARGET $@;\
-	done;
+		status=$$?;				\
+		$(call ksft_record_build_status,install,powerpc/$$TARGET); \
+		$(if $(FORCE_TARGETS),[ $$status -eq 0 ] || exit $$status;) \
+		[ $$status -eq 0 ] && ret=0;		\
+	done; exit $$ret
 
 override define CLEAN
 	+@for TARGET in $(SUB_DIRS); do \
diff --git a/tools/testing/selftests/powerpc/pmu/Makefile b/tools/testing/selftests/powerpc/pmu/Makefile
index 7e9dbf3d0d092..7b3ef1decbfbb 100644
--- a/tools/testing/selftests/powerpc/pmu/Makefile
+++ b/tools/testing/selftests/powerpc/pmu/Makefile
@@ -7,6 +7,7 @@ EXTRA_SOURCES := ../harness.c event.c lib.c ../utils.c
 
 top_srcdir = ../../../../..
 include ../../lib.mk
+include $(selfdir)/build_report.mk
 include ../flags.mk
 
 SUB_DIRS := ebb sampling_tests event_code_tests
@@ -26,7 +27,8 @@ $(OUTPUT)/count_stcx_fail: loop.S $(EXTRA_SOURCES)
 $(OUTPUT)/per_event_excludes: ../utils.c
 
 $(SUB_DIRS):
-	BUILD_TARGET=$(OUTPUT)/$@; mkdir -p $$BUILD_TARGET; $(MAKE) OUTPUT=$$BUILD_TARGET -k -C $@ all
+	BUILD_TARGET=$(OUTPUT)/$@; mkdir -p $$BUILD_TARGET && $(MAKE) OUTPUT=$$BUILD_TARGET -k -C $@ all; \
+	status=$$?; $(call ksft_record_build_status,all,powerpc/pmu/$@); exit $$status
 
 DEFAULT_RUN_TESTS := $(RUN_TESTS)
 override define RUN_TESTS
@@ -42,18 +44,26 @@ emit_tests:
 		BASENAME_TEST=`basename $$TEST`;	\
 		echo "$(COLLECTION):$$BASENAME_TEST";	\
 	done
-	+@for TARGET in $(SUB_DIRS); do \
+	+@ret=$(if $(SUB_DIRS),1,0); for TARGET in $(SUB_DIRS); do \
 		BUILD_TARGET=$(OUTPUT)/$$TARGET; \
 		$(MAKE) OUTPUT=$$BUILD_TARGET COLLECTION=$(COLLECTION)/$$TARGET -s -C $$TARGET emit_tests; \
-	done;
+		status=$$?; \
+		$(call ksft_record_build_status,install,powerpc/pmu/$$TARGET); \
+		$(if $(FORCE_TARGETS),[ $$status -eq 0 ] || exit $$status;) \
+		[ $$status -eq 0 ] && ret=0; \
+	done; exit $$ret
 
 DEFAULT_INSTALL_RULE := $(INSTALL_RULE)
 override define INSTALL_RULE
 	$(DEFAULT_INSTALL_RULE)
-	+@for TARGET in $(SUB_DIRS); do \
+	+@ret=$(if $(SUB_DIRS),1,0); for TARGET in $(SUB_DIRS); do \
 		BUILD_TARGET=$(OUTPUT)/$$TARGET; \
 		$(MAKE) OUTPUT=$$BUILD_TARGET INSTALL_PATH=$$INSTALL_PATH/$$TARGET -C $$TARGET install; \
-	done;
+		status=$$?; \
+		$(call ksft_record_build_status,install,powerpc/pmu/$$TARGET); \
+		$(if $(FORCE_TARGETS),[ $$status -eq 0 ] || exit $$status;) \
+		[ $$status -eq 0 ] && ret=0; \
+	done; exit $$ret
 endef
 
 DEFAULT_CLEAN := $(CLEAN)
-- 
2.53.0