[PATCH 0/7] selftests: report per-suite build results without failing the build

[PATCH 0/7] selftests: report per-suite build results without failing the build

[Sasha Levin]

Building the selftests succeeds as long as one target builds, so that a
single broken suite does not cost CI systems the coverage of all the
others. Commit cad6f32665cb ("selftests: Deescalate error reporting")
restored that policy after commit 7e47389142b8 ("selftests: Preserve
subtarget failures in all/install") had made any failure fatal. The
other side of the policy is that make cannot say which suites failed,
and CI systems fall back to guessing from the installed files, which
misses suites that installed their scripts but failed to compile and
flags suites that legitimately install nothing.

This series keeps the policy and adds an opt-in report instead:

 - Patch 1 fixes the exit status of the top-level loops, which wraps to
   success once eight or more targets fail.

 - Patch 2 adds KSFT_BUILD_REPORT. When it names an absolute file, the
   all and install rules append a "phase<TAB>target<TAB>status" record
   per target, without changing what is built or installed or the exit
   status of make.

 - Patches 3-7 apply the same policy to the suites that build their
   subdirectories in shell loops (futex, arm64, riscv, powerpc and
   sparc64). A failing subdirectory no longer hides behind the status
   of the last one, nor drops the whole suite from the installation,
   and each subdirectory is recorded as <suite>/<subdir>. FORCE_TARGETS
   keeps failing fast, now also inside these suites.

Patches 3-7 depend on patch 2 for the recording helper, so the series
is easiest to take as a whole; acks from the architecture and futex
maintainers would be appreciated.

Tested with mock suites for every converted loop and with native and
cross (arm64, riscv, powerpc) builds of the affected suites against the
base tree. A kselftest-install of 20 suites gives the same installation
and runlist as before, plus the report.

Sasha Levin (7):
  selftests: keep the build status from wrapping to success
  selftests: optionally record per-suite build and install results
  selftests/futex: report failures of the functional subdirectory
  kselftest/arm64: keep building and installing after a subdirectory
    fails
  selftests/riscv: keep building and installing after a subdirectory
    fails
  selftests/powerpc: report the outcome of each subdirectory
  selftests/sparc64: report failures of the drivers subdirectory

 Documentation/dev-tools/kselftest.rst        | 38 +++++++++++++++++
 tools/testing/selftests/Makefile             | 31 ++++++++++----
 tools/testing/selftests/arm64/Makefile       | 43 +++++++++++++-------
 tools/testing/selftests/build_report.mk      | 26 ++++++++++++
 tools/testing/selftests/futex/Makefile       | 32 ++++++++++-----
 tools/testing/selftests/powerpc/Makefile     | 20 ++++++---
 tools/testing/selftests/powerpc/pmu/Makefile | 20 ++++++---
 tools/testing/selftests/riscv/Makefile       | 39 +++++++++++++-----
 tools/testing/selftests/sparc64/Makefile     | 33 ++++++++++-----
 9 files changed, 218 insertions(+), 64 deletions(-)
 create mode 100644 tools/testing/selftests/build_report.mk


base-commit: 4eb2586bd40db27d77cc5b724ca9386ed0c71d53
-- 
2.53.0

[PATCH 1/7] selftests: keep the build status from wrapping to success

[Sasha Levin]

The top-level all and install rules return failure only when every
selected target fails, which commit cad6f32665cb ("selftests: Deescalate
error reporting") deliberately kept as the default. They implement this
by multiplying the exit statuses of the targets:

  ret=$$((ret * $$?))

A failing recursive make exits with status 2, so eight failing targets
produce 256, and "exit 256" reports success. Building eight or more
targets that all fail therefore succeeds, and install goes on to
package an empty installation.

Record a successful target instead of multiplying statuses. The rules
still fail only when no target succeeds, and FORCE_TARGETS still stops
at the first failure.

Fixes: 5f70bde26a48 ("selftests: fix build behaviour on targets' failures")
Signed-off-by: Sasha Levin <sashal@kernel.org>
---
 tools/testing/selftests/Makefile | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/tools/testing/selftests/Makefile b/tools/testing/selftests/Makefile
index 04cf073114318..a442b239f0610 100644
--- a/tools/testing/selftests/Makefile
+++ b/tools/testing/selftests/Makefile
@@ -236,7 +236,7 @@ all:
 		$(MAKE) OUTPUT=$$BUILD_TARGET -C $$TARGET	\
 				O=$(abs_objtree)		\
 				$(if $(FORCE_TARGETS),|| exit);	\
-		ret=$$((ret * $$?));				\
+		[ $$? -eq 0 ] && ret=0;				\
 	done; exit $$ret;
 
 run_tests: all
@@ -303,7 +303,7 @@ ifdef INSTALL_PATH
 				OBJ_PATH=$(INSTALL_PATH) \
 				O=$(abs_objtree)		\
 				$(if $(FORCE_TARGETS),|| exit);	\
-		ret=$$((ret * $$?));		\
+		[ $$? -eq 0 ] && ret=0;		\
 	done; exit $$ret;
 
 
-- 
2.53.0

[PATCH 2/7] selftests: optionally record per-suite build and install results

[Sasha Levin]

By default the selftests build and installation succeed as long as one
target succeeds, so that a single broken suite does not cost CI systems
the coverage of all others; commit cad6f32665cb ("selftests: Deescalate
error reporting") restored that policy after it had been tightened.
The exit status of make therefore cannot tell which suites failed, and
inspecting the installation is not reliable either: a suite whose
compilation failed can still install its scripts.

Add an opt-in report. When KSFT_BUILD_REPORT names an absolute file,
the all and install rules append one tab-separated record per target
with the phase, the target name and the exit status of its recursive
make. Generating the runlist is recorded as part of the install phase.
The records are written without changing which targets are built or
installed or the exit status of make, also with FORCE_TARGETS. Nothing
is recorded by make -n, -t or -q, including when the selftests are
built through the kernel's kselftest targets, which clear MAKEFLAGS.

The recording helper lives in a small fragment so that suites built
from subdirectories can add records for them, and the variable is
exported to sub-makes. A relative path is rejected, since every
recursive make runs in a different directory. Keep the exit status of
the runlist generation loop as before, and create an empty runlist for
installations in which no target emits tests.

Document the format, version 1, and how to read it.

Link: https://lore.kernel.org/r/20260416-selftests-deescalate-error-reporting-v1-1-38e7c0536227@kernel.org
Signed-off-by: Sasha Levin <sashal@kernel.org>
---
 Documentation/dev-tools/kselftest.rst   | 38 +++++++++++++++++++++++++
 tools/testing/selftests/Makefile        | 31 ++++++++++++++------
 tools/testing/selftests/build_report.mk | 26 +++++++++++++++++
 3 files changed, 87 insertions(+), 8 deletions(-)
 create mode 100644 tools/testing/selftests/build_report.mk

diff --git a/Documentation/dev-tools/kselftest.rst b/Documentation/dev-tools/kselftest.rst
index fc4dc1f9dc931..0772525cbc627 100644
--- a/Documentation/dev-tools/kselftest.rst
+++ b/Documentation/dev-tools/kselftest.rst
@@ -133,6 +133,44 @@ build successfully; make will abort as soon as any target fails::
 This applies to both the ``all`` and ``install`` targets and is useful in
 CI environments where a silent partial build would be misleading.
 
+Recording suite build results
+=============================
+
+Since the build succeeds as long as one target builds, the exit status of
+make does not show which targets failed. Set ``KSFT_BUILD_REPORT`` to the
+absolute path of a file to have the ``all`` and ``install`` targets append
+a tab-separated record for each target they process::
+
+  $ : > /tmp/ksft-build.tsv
+  $ make -C tools/testing/selftests O=/tmp/kernel-build \
+        KSFT_BUILD_REPORT=/tmp/ksft-build.tsv install
+
+Each record holds the phase (``all`` or ``install``), a name and the exit
+status of that step. Recording does not change which targets are built or
+installed, or the exit status of make, also with ``FORCE_TARGETS=1``. The
+caller creates or truncates the report before a build; the rules only
+append to it. Nothing is recorded by ``make -n``, ``-t`` or ``-q``.
+
+The same name can appear several times: ``install`` builds the targets
+again, and generating ``kselftest-list.txt`` is recorded as part of the
+``install`` phase. Targets that are added automatically as installation
+dependencies of other targets, such as ``net/lib``, are recorded too.
+Suites that are built from several subdirectories may also record each
+subdirectory under its path, for example ``arm64/tags``; a failure there
+means that the suite was built or installed only partially. A record
+belongs to the target with the longest name that is equal to the record
+name or followed by ``/`` in it.
+
+A target was built and installed successfully only if all of its records
+have a zero status. Missing records do not show that a target was built.
+A successful target that installs no tests has no entries in
+``kselftest-list.txt``. Install the kernel headers with ``make headers``
+before building: a suite that fails because the headers are missing is
+recorded as failed even if a later step installs them.
+
+``KSFT_BUILD_REPORT_VERSION`` in tools/testing/selftests/Makefile gives the
+version of this format, currently ``1``.
+
 Running the full range hotplug selftests
 ========================================
 
diff --git a/tools/testing/selftests/Makefile b/tools/testing/selftests/Makefile
index a442b239f0610..abbb854886cfb 100644
--- a/tools/testing/selftests/Makefile
+++ b/tools/testing/selftests/Makefile
@@ -180,6 +180,10 @@ endif
 # of the targets gets built.
 FORCE_TARGETS ?=
 
+# Remember dry runs for the build report before MAKEFLAGS may be cleared.
+KSFT_BUILD_REPORT_DRY_RUN := $(strip $(foreach f,n t q,$(findstring $(f),$(firstword -$(MAKEFLAGS)))))
+export KSFT_BUILD_REPORT_DRY_RUN
+
 # Clear LDFLAGS and MAKEFLAGS when implicit rules are missing.  This provides
 # implicit rules to sub-test Makefiles which avoids build failures in test
 # Makefile that don't have explicit build rules.
@@ -224,6 +228,10 @@ ARCH           ?= $(SUBARCH)
 export BUILD
 export KHDR_INCLUDES
 
+# Version of the optional KSFT_BUILD_REPORT format written by these rules.
+KSFT_BUILD_REPORT_VERSION := 1
+include $(top_srcdir)/tools/testing/selftests/build_report.mk
+
 # set default goal to all, so make without a target runs all, even when
 # all isn't the first target in the file.
 .DEFAULT_GOAL := all
@@ -232,11 +240,13 @@ all:
 	@ret=1;							\
 	for TARGET in $(TARGETS) $(INSTALL_DEP_TARGETS); do	\
 		BUILD_TARGET=$$BUILD/$$TARGET;			\
-		mkdir $$BUILD_TARGET  -p;			\
+		mkdir -p $$BUILD_TARGET &&			\
 		$(MAKE) OUTPUT=$$BUILD_TARGET -C $$TARGET	\
-				O=$(abs_objtree)		\
-				$(if $(FORCE_TARGETS),|| exit);	\
-		[ $$? -eq 0 ] && ret=0;				\
+				O=$(abs_objtree);		\
+		status=$$?;					\
+		$(call ksft_record_build_status,all,$$TARGET);	\
+		$(if $(FORCE_TARGETS),[ $$status -eq 0 ] || exit $$status;) \
+		[ $$status -eq 0 ] && ret=0;			\
 	done; exit $$ret;
 
 run_tests: all
@@ -293,7 +303,7 @@ ifdef INSTALL_PATH
 	install -m 744 kselftest/ktap_helpers.sh $(INSTALL_PATH)/kselftest/
 	install -m 744 kselftest/ksft.py $(INSTALL_PATH)/kselftest/
 	install -m 744 run_kselftest.sh $(INSTALL_PATH)/
-	rm -f $(TEST_LIST)
+	: > $(TEST_LIST)
 	@ret=1;	\
 	for TARGET in $(TARGETS) $(INSTALL_DEP_TARGETS); do \
 		BUILD_TARGET=$$BUILD/$$TARGET;	\
@@ -301,9 +311,11 @@ ifdef INSTALL_PATH
 				INSTALL_PATH=$(INSTALL_PATH)/$$TARGET \
 				SRC_PATH=$(shell readlink -e $$(pwd)) \
 				OBJ_PATH=$(INSTALL_PATH) \
-				O=$(abs_objtree)		\
-				$(if $(FORCE_TARGETS),|| exit);	\
-		[ $$? -eq 0 ] && ret=0;		\
+				O=$(abs_objtree);		\
+		status=$$?;				\
+		$(call ksft_record_build_status,install,$$TARGET); \
+		$(if $(FORCE_TARGETS),[ $$status -eq 0 ] || exit $$status;) \
+		[ $$status -eq 0 ] && ret=0;		\
 	done; exit $$ret;
 
 
@@ -317,6 +329,9 @@ ifdef INSTALL_PATH
 		printf "Emit Tests for $$TARGET\n"; \
 		$(MAKE) -s --no-print-directory OUTPUT=$$BUILD_TARGET COLLECTION=$$TARGET \
 			-C $$TARGET emit_tests >> $(TEST_LIST); \
+		status=$$?; \
+		$(call ksft_record_build_status,install,$$TARGET); \
+		(exit $$status); \
 	done;
 	@VERSION=$$(git describe HEAD 2>/dev/null); \
 	if [ -n "$$VERSION" ]; then \
diff --git a/tools/testing/selftests/build_report.mk b/tools/testing/selftests/build_report.mk
new file mode 100644
index 0000000000000..047c239925e16
--- /dev/null
+++ b/tools/testing/selftests/build_report.mk
@@ -0,0 +1,26 @@
+# SPDX-License-Identifier: GPL-2.0
+#
+# Optional per-suite build report, see "Recording suite build results" in
+# Documentation/dev-tools/kselftest.rst. Callers set KSFT_BUILD_REPORT to an
+# absolute filename and use ksft_record_build_status from a shell loop that
+# has stored the exit status of the recorded step in $status.
+
+ifneq (x$(KSFT_BUILD_REPORT)x,xx)
+ifeq ($(filter x/%,$(firstword x$(KSFT_BUILD_REPORT))),)
+$(error KSFT_BUILD_REPORT must be an absolute path: '$(KSFT_BUILD_REPORT)')
+endif
+endif
+export KSFT_BUILD_REPORT
+
+ksft_build_report_dry_run := $(KSFT_BUILD_REPORT_DRY_RUN) \
+	$(foreach f,n t q,$(findstring $(f),$(firstword -$(MAKEFLAGS))))
+
+ifneq ($(strip $(ksft_build_report_dry_run)),)
+ksft_record_build_status = :
+else
+define ksft_record_build_status
+if [ -n "$$KSFT_BUILD_REPORT" ]; then \
+	printf '%s\t%s\t%d\n' '$(1)' "$(2)" "$$status" >> "$$KSFT_BUILD_REPORT" || exit 1; \
+fi
+endef
+endif
-- 
2.53.0

[PATCH 3/7] selftests/futex: report failures of the functional subdirectory

[Sasha Levin]

The futex suite builds its tests in the functional subdirectory and then
copies run.sh into the output directory. The copy is the last command of
the loop, so its status replaces that of the build, and "make all"
succeeds even when the functional tests fail to compile.

Fold the copy into the status of the subdirectory, count a failure to
create the output directory as well, and let the build and installation
loops return failure when no subdirectory succeeds, which matches how the
top-level selftests Makefile treats its targets, or as soon as one fails
when FORCE_TARGETS is set. When a build report is requested, record each
subdirectory as futex/<subdir> so that a failure is visible even when
make itself succeeds.

Fixes: 9c3340ea7f5d ("selftests: futex: copy sub-dir test scripts for make O=dir run")
Fixes: 9705315b7562 ("selftests/futex: Update Makefile to use lib.mk")
Signed-off-by: Sasha Levin <sashal@kernel.org>
---
 tools/testing/selftests/futex/Makefile | 32 +++++++++++++++++---------
 1 file changed, 21 insertions(+), 11 deletions(-)

diff --git a/tools/testing/selftests/futex/Makefile b/tools/testing/selftests/futex/Makefile
index 78ab2cd111f60..d6923775b8d49 100644
--- a/tools/testing/selftests/futex/Makefile
+++ b/tools/testing/selftests/futex/Makefile
@@ -4,26 +4,36 @@ SUBDIRS := functional
 TEST_PROGS := run.sh
 
 include ../lib.mk
+include $(selfdir)/build_report.mk
 
 all:
-	@for DIR in $(SUBDIRS); do		\
-		BUILD_TARGET=$(OUTPUT)/$$DIR;	\
-		mkdir $$BUILD_TARGET  -p;	\
-		$(MAKE) OUTPUT=$$BUILD_TARGET -C $$DIR $@;\
-		if [ -e $$DIR/$(TEST_PROGS) ]; then \
-			rsync -a $$DIR/$(TEST_PROGS) $$BUILD_TARGET/; \
-		fi \
-	done
+	@ret=$(if $(SUBDIRS),1,0); for DIR in $(SUBDIRS); do	\
+		BUILD_TARGET=$(OUTPUT)/$$DIR;			\
+		mkdir -p $$BUILD_TARGET &&			\
+		$(MAKE) OUTPUT=$$BUILD_TARGET -C $$DIR $@;	\
+		status=$$?;					\
+		if [ -e $$DIR/$(TEST_PROGS) ]; then		\
+			rsync -a $$DIR/$(TEST_PROGS) $$BUILD_TARGET/ || \
+				[ $$status -ne 0 ] || status=1;	\
+		fi;						\
+		$(call ksft_record_build_status,all,futex/$$DIR); \
+		$(if $(FORCE_TARGETS),[ $$status -eq 0 ] || exit $$status;) \
+		[ $$status -eq 0 ] && ret=0;			\
+	done; exit $$ret
 
 override define INSTALL_RULE
 	mkdir -p $(INSTALL_PATH)
 	install -t $(INSTALL_PATH) $(TEST_PROGS) $(TEST_PROGS_EXTENDED) $(TEST_FILES)
 
-	@for SUBDIR in $(SUBDIRS); do \
+	@ret=$(if $(SUBDIRS),1,0); for SUBDIR in $(SUBDIRS); do \
 		BUILD_TARGET=$(OUTPUT)/$$SUBDIR;	\
-		mkdir $$BUILD_TARGET  -p;	\
+		mkdir -p $$BUILD_TARGET &&		\
 		$(MAKE) OUTPUT=$$BUILD_TARGET -C $$SUBDIR INSTALL_PATH=$(INSTALL_PATH)/$$SUBDIR install; \
-	done;
+		status=$$?;				\
+		$(call ksft_record_build_status,install,futex/$$SUBDIR); \
+		$(if $(FORCE_TARGETS),[ $$status -eq 0 ] || exit $$status;) \
+		[ $$status -eq 0 ] && ret=0;		\
+	done; exit $$ret
 endef
 
 override define CLEAN
-- 
2.53.0

[PATCH 4/7] kselftest/arm64: keep building and installing after a subdirectory fails

[Sasha Levin]

The arm64 suite builds and installs its subdirectories in shell loops
whose status is that of the last subdirectory. A failure in an earlier
subdirectory is therefore lost unless FORCE_TARGETS is set, while a
failure in the last one fails "make all" and, through "install: all",
drops the installation of every other arm64 test as well.

Make each loop fail only when no subdirectory succeeds, which matches
how the top-level selftests Makefile treats its targets, and keep
stopping at the first failure when FORCE_TARGETS is set. When a build
report is requested, record each subdirectory as arm64/<subdir> so that
a partially built suite is visible even though make succeeds.

Invoke the subdirectories with $(MAKE) rather than make. The command
line variables already reach them through MAKEFLAGS either way, but
$(MAKE) lets them use the parent's jobserver and makes -n, -t and -q
apply to them.

Fixes: 313a4db7f338 ("kselftest: arm64: extend toplevel skeleton Makefile")
Signed-off-by: Sasha Levin <sashal@kernel.org>
---
 tools/testing/selftests/arm64/Makefile | 43 +++++++++++++++++---------
 1 file changed, 29 insertions(+), 14 deletions(-)

diff --git a/tools/testing/selftests/arm64/Makefile b/tools/testing/selftests/arm64/Makefile
index e456f3b62fa13..c787e848d4718 100644
--- a/tools/testing/selftests/arm64/Makefile
+++ b/tools/testing/selftests/arm64/Makefile
@@ -26,38 +26,53 @@ OUTPUT ?= $(CURDIR)
 export CFLAGS
 export top_srcdir
 
+include $(top_srcdir)/tools/testing/selftests/build_report.mk
+
 all:
-	@for DIR in $(ARM64_SUBTARGETS); do				\
+	@ret=$(if $(ARM64_SUBTARGETS),1,0);				\
+	for DIR in $(ARM64_SUBTARGETS); do				\
 		BUILD_TARGET=$(OUTPUT)/$$DIR;			\
-		mkdir -p $$BUILD_TARGET;			\
-		make OUTPUT=$$BUILD_TARGET -C $$DIR $@		\
-			$(if $(FORCE_TARGETS),|| exit); \
-	done
+		mkdir -p $$BUILD_TARGET &&			\
+		$(MAKE) OUTPUT=$$BUILD_TARGET -C $$DIR $@;	\
+		status=$$?;					\
+		$(call ksft_record_build_status,all,arm64/$$DIR); \
+		$(if $(FORCE_TARGETS),[ $$status -eq 0 ] || exit $$status;) \
+		[ $$status -eq 0 ] && ret=0;			\
+	done; exit $$ret
 
 install: all
-	@for DIR in $(ARM64_SUBTARGETS); do				\
+	@ret=$(if $(ARM64_SUBTARGETS),1,0);				\
+	for DIR in $(ARM64_SUBTARGETS); do				\
 		BUILD_TARGET=$(OUTPUT)/$$DIR;			\
-		make OUTPUT=$$BUILD_TARGET -C $$DIR $@		\
-			$(if $(FORCE_TARGETS),|| exit); \
-	done
+		$(MAKE) OUTPUT=$$BUILD_TARGET -C $$DIR $@;	\
+		status=$$?;					\
+		$(call ksft_record_build_status,install,arm64/$$DIR); \
+		$(if $(FORCE_TARGETS),[ $$status -eq 0 ] || exit $$status;) \
+		[ $$status -eq 0 ] && ret=0;			\
+	done; exit $$ret
 
 run_tests: all
 	@for DIR in $(ARM64_SUBTARGETS); do				\
 		BUILD_TARGET=$(OUTPUT)/$$DIR;			\
-		make OUTPUT=$$BUILD_TARGET -C $$DIR $@;		\
+		$(MAKE) OUTPUT=$$BUILD_TARGET -C $$DIR $@;	\
 	done
 
 # Avoid any output on non arm64 on emit_tests
 emit_tests:
-	@for DIR in $(ARM64_SUBTARGETS); do				\
+	@ret=$(if $(ARM64_SUBTARGETS),1,0);				\
+	for DIR in $(ARM64_SUBTARGETS); do				\
 		BUILD_TARGET=$(OUTPUT)/$$DIR;			\
-		make OUTPUT=$$BUILD_TARGET -C $$DIR $@;		\
-	done
+		$(MAKE) OUTPUT=$$BUILD_TARGET -C $$DIR $@;	\
+		status=$$?;					\
+		$(call ksft_record_build_status,install,arm64/$$DIR); \
+		$(if $(FORCE_TARGETS),[ $$status -eq 0 ] || exit $$status;) \
+		[ $$status -eq 0 ] && ret=0;			\
+	done; exit $$ret
 
 clean:
 	@for DIR in $(ARM64_SUBTARGETS); do				\
 		BUILD_TARGET=$(OUTPUT)/$$DIR;			\
-		make OUTPUT=$$BUILD_TARGET -C $$DIR $@;		\
+		$(MAKE) OUTPUT=$$BUILD_TARGET -C $$DIR $@;	\
 	done
 
 .PHONY: all clean install run_tests emit_tests
-- 
2.53.0

[PATCH 5/7] selftests/riscv: keep building and installing after a subdirectory fails

[Sasha Levin]

The RISC-V suite builds and installs its subdirectories in shell loops
whose status is that of the last subdirectory. A failure in any other
subdirectory is lost, while a failure in the last one fails "make all"
and, through "install: all", drops the installation of every other
RISC-V test as well. FORCE_TARGETS has no effect on the loops either.

Make each loop fail only when no subdirectory succeeds, which matches
how the top-level selftests Makefile treats its targets, and stop at the
first failure when FORCE_TARGETS is set. When a build report is
requested, record each subdirectory as riscv/<subdir> so that a
partially built suite is visible even though make succeeds. Default
OUTPUT to the current directory, as the arm64 suite does, so that the
loops also work when the suite is built directly.

Fixes: 287dcc2b0c83 ("selftests: Test the new RISC-V hwprobe interface")
Signed-off-by: Sasha Levin <sashal@kernel.org>
---
 tools/testing/selftests/riscv/Makefile | 39 +++++++++++++++++++-------
 1 file changed, 29 insertions(+), 10 deletions(-)

diff --git a/tools/testing/selftests/riscv/Makefile b/tools/testing/selftests/riscv/Makefile
index 5671b4405a129..52d1228be8d03 100644
--- a/tools/testing/selftests/riscv/Makefile
+++ b/tools/testing/selftests/riscv/Makefile
@@ -20,21 +20,35 @@ CFLAGS += -I$(top_srcdir)/tools/testing/selftests/
 
 CFLAGS += $(KHDR_INCLUDES)
 
+OUTPUT ?= $(CURDIR)
+
 export CFLAGS
 export top_srcdir
 
+include $(top_srcdir)/tools/testing/selftests/build_report.mk
+
 all:
-	@for DIR in $(RISCV_SUBTARGETS); do				\
+	@ret=$(if $(RISCV_SUBTARGETS),1,0);				\
+	for DIR in $(RISCV_SUBTARGETS); do				\
 		BUILD_TARGET=$(OUTPUT)/$$DIR;			\
-		mkdir -p $$BUILD_TARGET;			\
-		$(MAKE) OUTPUT=$$BUILD_TARGET -C $$DIR $@;		\
-	done
+		mkdir -p $$BUILD_TARGET &&			\
+		$(MAKE) OUTPUT=$$BUILD_TARGET -C $$DIR $@;	\
+		status=$$?;					\
+		$(call ksft_record_build_status,all,riscv/$$DIR); \
+		$(if $(FORCE_TARGETS),[ $$status -eq 0 ] || exit $$status;) \
+		[ $$status -eq 0 ] && ret=0;			\
+	done; exit $$ret
 
 install: all
-	@for DIR in $(RISCV_SUBTARGETS); do				\
+	@ret=$(if $(RISCV_SUBTARGETS),1,0);				\
+	for DIR in $(RISCV_SUBTARGETS); do				\
 		BUILD_TARGET=$(OUTPUT)/$$DIR;			\
-		$(MAKE) OUTPUT=$$BUILD_TARGET -C $$DIR $@;		\
-	done
+		$(MAKE) OUTPUT=$$BUILD_TARGET -C $$DIR $@;	\
+		status=$$?;					\
+		$(call ksft_record_build_status,install,riscv/$$DIR); \
+		$(if $(FORCE_TARGETS),[ $$status -eq 0 ] || exit $$status;) \
+		[ $$status -eq 0 ] && ret=0;			\
+	done; exit $$ret
 
 run_tests: all
 	@for DIR in $(RISCV_SUBTARGETS); do				\
@@ -44,10 +58,15 @@ run_tests: all
 
 # Avoid any output on non riscv on emit_tests
 emit_tests:
-	@for DIR in $(RISCV_SUBTARGETS); do				\
+	@ret=$(if $(RISCV_SUBTARGETS),1,0);				\
+	for DIR in $(RISCV_SUBTARGETS); do				\
 		BUILD_TARGET=$(OUTPUT)/$$DIR;			\
-		$(MAKE) OUTPUT=$$BUILD_TARGET -C $$DIR $@;		\
-	done
+		$(MAKE) OUTPUT=$$BUILD_TARGET -C $$DIR $@;	\
+		status=$$?;					\
+		$(call ksft_record_build_status,install,riscv/$$DIR); \
+		$(if $(FORCE_TARGETS),[ $$status -eq 0 ] || exit $$status;) \
+		[ $$status -eq 0 ] && ret=0;			\
+	done; exit $$ret
 
 clean:
 	@for DIR in $(RISCV_SUBTARGETS); do				\
-- 
2.53.0

[PATCH 6/7] selftests/powerpc: report the outcome of each subdirectory

[Sasha Levin]

The powerpc suite and its pmu subdirectory install and list their tests
in shell loops whose status is that of the last subdirectory, so a
failure in any other subdirectory is lost and FORCE_TARGETS has no
effect on them.

Make the installation and runlist loops fail only when no subdirectory
succeeds, which matches how the top-level selftests Makefile treats its
targets, and stop at the first failure when FORCE_TARGETS is set. When a
build report is requested, record each subdirectory as powerpc/<subdir>
or powerpc/pmu/<subdir>, including the builds of the subdirectories, so
that a failure is visible even when make itself succeeds.

The subdirectories are still built as prerequisites of "all", so a
subdirectory that fails to build still fails the build of the suite as
before; only the reporting of that failure is added.

Fixes: 6faeeea44b84 ("selftests: Add install support for the powerpc tests")
Fixes: 822a04957cc5 ("selftests/powerpc: Convert pmu Makefile to for loop style")
Signed-off-by: Sasha Levin <sashal@kernel.org>
---
 tools/testing/selftests/powerpc/Makefile     | 20 +++++++++++++++-----
 tools/testing/selftests/powerpc/pmu/Makefile | 20 +++++++++++++++-----
 2 files changed, 30 insertions(+), 10 deletions(-)

diff --git a/tools/testing/selftests/powerpc/Makefile b/tools/testing/selftests/powerpc/Makefile
index b175e94e1901d..8349196c934d5 100644
--- a/tools/testing/selftests/powerpc/Makefile
+++ b/tools/testing/selftests/powerpc/Makefile
@@ -37,9 +37,11 @@ endif
 all: $(SUB_DIRS)
 
 $(SUB_DIRS):
-	BUILD_TARGET=$(OUTPUT)/$@; mkdir -p $$BUILD_TARGET; $(MAKE) OUTPUT=$$BUILD_TARGET -k -C $@ all
+	BUILD_TARGET=$(OUTPUT)/$@; mkdir -p $$BUILD_TARGET && $(MAKE) OUTPUT=$$BUILD_TARGET -k -C $@ all; \
+	status=$$?; $(call ksft_record_build_status,all,powerpc/$@); exit $$status
 
 include ../lib.mk
+include $(selfdir)/build_report.mk
 include ./flags.mk
 
 override define RUN_TESTS
@@ -50,17 +52,25 @@ override define RUN_TESTS
 endef
 
 override define INSTALL_RULE
-	+@for TARGET in $(SUB_DIRS); do \
+	+@ret=$(if $(SUB_DIRS),1,0); for TARGET in $(SUB_DIRS); do \
 		BUILD_TARGET=$(OUTPUT)/$$TARGET;	\
 		$(MAKE) OUTPUT=$$BUILD_TARGET INSTALL_PATH=$$INSTALL_PATH/$$TARGET -C $$TARGET install;\
-	done;
+		status=$$?;				\
+		$(call ksft_record_build_status,install,powerpc/$$TARGET); \
+		$(if $(FORCE_TARGETS),[ $$status -eq 0 ] || exit $$status;) \
+		[ $$status -eq 0 ] && ret=0;		\
+	done; exit $$ret
 endef
 
 emit_tests:
-	+@for TARGET in $(SUB_DIRS); do \
+	+@ret=$(if $(SUB_DIRS),1,0); for TARGET in $(SUB_DIRS); do \
 		BUILD_TARGET=$(OUTPUT)/$$TARGET;	\
 		$(MAKE) OUTPUT=$$BUILD_TARGET COLLECTION=$(COLLECTION)/$$TARGET -s -C $$TARGET $@;\
-	done;
+		status=$$?;				\
+		$(call ksft_record_build_status,install,powerpc/$$TARGET); \
+		$(if $(FORCE_TARGETS),[ $$status -eq 0 ] || exit $$status;) \
+		[ $$status -eq 0 ] && ret=0;		\
+	done; exit $$ret
 
 override define CLEAN
 	+@for TARGET in $(SUB_DIRS); do \
diff --git a/tools/testing/selftests/powerpc/pmu/Makefile b/tools/testing/selftests/powerpc/pmu/Makefile
index 7e9dbf3d0d092..7b3ef1decbfbb 100644
--- a/tools/testing/selftests/powerpc/pmu/Makefile
+++ b/tools/testing/selftests/powerpc/pmu/Makefile
@@ -7,6 +7,7 @@ EXTRA_SOURCES := ../harness.c event.c lib.c ../utils.c
 
 top_srcdir = ../../../../..
 include ../../lib.mk
+include $(selfdir)/build_report.mk
 include ../flags.mk
 
 SUB_DIRS := ebb sampling_tests event_code_tests
@@ -26,7 +27,8 @@ $(OUTPUT)/count_stcx_fail: loop.S $(EXTRA_SOURCES)
 $(OUTPUT)/per_event_excludes: ../utils.c
 
 $(SUB_DIRS):
-	BUILD_TARGET=$(OUTPUT)/$@; mkdir -p $$BUILD_TARGET; $(MAKE) OUTPUT=$$BUILD_TARGET -k -C $@ all
+	BUILD_TARGET=$(OUTPUT)/$@; mkdir -p $$BUILD_TARGET && $(MAKE) OUTPUT=$$BUILD_TARGET -k -C $@ all; \
+	status=$$?; $(call ksft_record_build_status,all,powerpc/pmu/$@); exit $$status
 
 DEFAULT_RUN_TESTS := $(RUN_TESTS)
 override define RUN_TESTS
@@ -42,18 +44,26 @@ emit_tests:
 		BASENAME_TEST=`basename $$TEST`;	\
 		echo "$(COLLECTION):$$BASENAME_TEST";	\
 	done
-	+@for TARGET in $(SUB_DIRS); do \
+	+@ret=$(if $(SUB_DIRS),1,0); for TARGET in $(SUB_DIRS); do \
 		BUILD_TARGET=$(OUTPUT)/$$TARGET; \
 		$(MAKE) OUTPUT=$$BUILD_TARGET COLLECTION=$(COLLECTION)/$$TARGET -s -C $$TARGET emit_tests; \
-	done;
+		status=$$?; \
+		$(call ksft_record_build_status,install,powerpc/pmu/$$TARGET); \
+		$(if $(FORCE_TARGETS),[ $$status -eq 0 ] || exit $$status;) \
+		[ $$status -eq 0 ] && ret=0; \
+	done; exit $$ret
 
 DEFAULT_INSTALL_RULE := $(INSTALL_RULE)
 override define INSTALL_RULE
 	$(DEFAULT_INSTALL_RULE)
-	+@for TARGET in $(SUB_DIRS); do \
+	+@ret=$(if $(SUB_DIRS),1,0); for TARGET in $(SUB_DIRS); do \
 		BUILD_TARGET=$(OUTPUT)/$$TARGET; \
 		$(MAKE) OUTPUT=$$BUILD_TARGET INSTALL_PATH=$$INSTALL_PATH/$$TARGET -C $$TARGET install; \
-	done;
+		status=$$?; \
+		$(call ksft_record_build_status,install,powerpc/pmu/$$TARGET); \
+		$(if $(FORCE_TARGETS),[ $$status -eq 0 ] || exit $$status;) \
+		[ $$status -eq 0 ] && ret=0; \
+	done; exit $$ret
 endef
 
 DEFAULT_CLEAN := $(CLEAN)
-- 
2.53.0

[PATCH 7/7] selftests/sparc64: report failures of the drivers subdirectory

[Sasha Levin]

The sparc64 suite builds and installs its subdirectories in shell loops
whose status is that of the last command. For the build that is the
copy of the subdirectory's test script, so "make all" succeeds even
when the drivers tests fail to compile, and FORCE_TARGETS has no effect.

Fold the copy into the status of the subdirectory and let both loops
return failure when no subdirectory succeeds, which matches how the
top-level selftests Makefile treats its targets, or as soon as one fails
when FORCE_TARGETS is set. When a build report is requested, record
each subdirectory as sparc64/<subdir>. Invoke the subdirectories with
$(MAKE) so that they use the parent's jobserver.

Fixes: 3c545084130c ("selftests: sparc64: char: Selftest for privileged ADI driver")
Signed-off-by: Sasha Levin <sashal@kernel.org>
---
 tools/testing/selftests/sparc64/Makefile | 33 ++++++++++++++++--------
 1 file changed, 22 insertions(+), 11 deletions(-)

diff --git a/tools/testing/selftests/sparc64/Makefile b/tools/testing/selftests/sparc64/Makefile
index 88f7be76f962d..a979c819348a0 100644
--- a/tools/testing/selftests/sparc64/Makefile
+++ b/tools/testing/selftests/sparc64/Makefile
@@ -2,6 +2,8 @@
 ARCH ?= $(shell uname -m 2>/dev/null || echo not)
 override ARCH := $(shell echo $(ARCH) | sed -e s/x86_64/x86/)
 
+include ../build_report.mk
+
 ifneq ($(ARCH),sparc64)
 nothing:
 .PHONY: all clean run_tests install
@@ -18,33 +20,42 @@ TEST_PROGS := run.sh
 include ../lib.mk
 
 all:
-	@for DIR in $(SUBDIRS); do		\
-		BUILD_TARGET=$(OUTPUT)/$$DIR;	\
-		mkdir $$BUILD_TARGET  -p;	\
-		make OUTPUT=$$BUILD_TARGET -C $$DIR $@;\
+	@ret=$(if $(SUBDIRS),1,0); for DIR in $(SUBDIRS); do	\
+		BUILD_TARGET=$(OUTPUT)/$$DIR;			\
+		mkdir -p $$BUILD_TARGET &&			\
+		$(MAKE) OUTPUT=$$BUILD_TARGET -C $$DIR $@;	\
+		status=$$?;					\
 		#SUBDIR test prog name should be in the form: SUBDIR_test.sh \
 		TEST=$$DIR"_test.sh"; \
 		if [ -e $$DIR/$$TEST ]; then \
-			rsync -a $$DIR/$$TEST $$BUILD_TARGET/; \
-		fi \
-	done
+			rsync -a $$DIR/$$TEST $$BUILD_TARGET/ || \
+				[ $$status -ne 0 ] || status=1;	\
+		fi; \
+		$(call ksft_record_build_status,all,sparc64/$$DIR); \
+		$(if $(FORCE_TARGETS),[ $$status -eq 0 ] || exit $$status;) \
+		[ $$status -eq 0 ] && ret=0;			\
+	done; exit $$ret
 
 override define INSTALL_RULE
 	mkdir -p $(INSTALL_PATH)
 	install -t $(INSTALL_PATH) $(TEST_PROGS) $(TEST_PROGS_EXTENDED) $(TEST_FILES)
 
-	@for SUBDIR in $(SUBDIRS); do \
+	@ret=$(if $(SUBDIRS),1,0); for SUBDIR in $(SUBDIRS); do \
 		BUILD_TARGET=$(OUTPUT)/$$SUBDIR;	\
-		mkdir $$BUILD_TARGET  -p;	\
+		mkdir -p $$BUILD_TARGET &&		\
 		$(MAKE) OUTPUT=$$BUILD_TARGET -C $$SUBDIR INSTALL_PATH=$(INSTALL_PATH)/$$SUBDIR install; \
-	done;
+		status=$$?;				\
+		$(call ksft_record_build_status,install,sparc64/$$SUBDIR); \
+		$(if $(FORCE_TARGETS),[ $$status -eq 0 ] || exit $$status;) \
+		[ $$status -eq 0 ] && ret=0;		\
+	done; exit $$ret
 endef
 
 override define CLEAN
 	@for DIR in $(SUBDIRS); do		\
 		BUILD_TARGET=$(OUTPUT)/$$DIR;	\
 		mkdir $$BUILD_TARGET  -p;	\
-		make OUTPUT=$$BUILD_TARGET -C $$DIR $@;\
+		$(MAKE) OUTPUT=$$BUILD_TARGET -C $$DIR $@;\
 	done
 endef
 endif
-- 
2.53.0