[PATCH 0/6] selftests: run_kselftest.sh cleanup and fixes

[PATCH 0/6] selftests: run_kselftest.sh cleanup and fixes

[Ricardo B. Marlière]

Hi Shuah,

Please consider merging the following patches. They are mostly small fixes
to run_kselftest.sh, plus one feature: adding the option of passing a
directory to the "-p --per-test-log" argument.

There is a small corner case spotted in tools/testing/selftests/Makefile
regarding failing sub-targets, and also the removal of the default install
directory for out of tree builds in the top-level Makefile.

Thank you!

Signed-off-by: Ricardo B. Marlière <rbm@suse.com>
---
Ricardo B. Marlière (6):
      selftests/run_kselftest.sh: Remove unused $ROOT
      selftests/run_kselftest.sh: Add missing quotes
      selftests/run_kselftest.sh: Resolve BASE_DIR with pwd -P
      selftests/run_kselftest.sh: Allow choosing per-test log directory
      selftests: Preserve subtarget failures in all/install
      kbuild: remove kselftest output in mrproper

 Makefile                                    |  1 +
 tools/testing/selftests/Makefile            |  8 ++---
 tools/testing/selftests/kselftest/runner.sh |  3 +-
 tools/testing/selftests/run_kselftest.sh    | 46 +++++++++++++++++++++--------
 4 files changed, 40 insertions(+), 18 deletions(-)
---
base-commit: c612261bedd6bbab7109f798715e449c9d20ff2f
change-id: 20260320-selftests-fixes-dad2b1df50d9

Best regards,
-- 
Ricardo B. Marlière <rbm@suse.com>

[PATCH 1/6] selftests/run_kselftest.sh: Remove unused $ROOT

[Ricardo B. Marlière]

Fix the following shellcheck warning:

ROOT appears unused. Verify use (or export if used externally). [SC2034]

Signed-off-by: Ricardo B. Marlière <rbm@suse.com>
---
 tools/testing/selftests/run_kselftest.sh | 1 -
 1 file changed, 1 deletion(-)

diff --git a/tools/testing/selftests/run_kselftest.sh b/tools/testing/selftests/run_kselftest.sh
index 84d45254675c..9abea79018d7 100755
--- a/tools/testing/selftests/run_kselftest.sh
+++ b/tools/testing/selftests/run_kselftest.sh
@@ -21,7 +21,6 @@ else
 fi
 
 . ./kselftest/runner.sh
-ROOT=$PWD
 
 usage()
 {

-- 
2.53.0

[PATCH 2/6] selftests/run_kselftest.sh: Add missing quotes

[Ricardo B. Marlière]

Fix the following shellcheck warning:

Double quote to prevent globbing and word splitting. [SC2086]

Signed-off-by: Ricardo B. Marlière <rbm@suse.com>
---
 tools/testing/selftests/run_kselftest.sh | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/tools/testing/selftests/run_kselftest.sh b/tools/testing/selftests/run_kselftest.sh
index 9abea79018d7..b782b025d002 100755
--- a/tools/testing/selftests/run_kselftest.sh
+++ b/tools/testing/selftests/run_kselftest.sh
@@ -11,7 +11,7 @@ else
         BASE_DIR=$(readlink -f $(dirname $0))
 fi
 
-cd $BASE_DIR
+cd "$BASE_DIR"
 TESTS="$BASE_DIR"/kselftest-list.txt
 if [ ! -r "$TESTS" ] ; then
 	echo "$0: Could not find list of tests to run ($TESTS)" >&2
@@ -38,7 +38,7 @@ Usage: $0 [OPTIONS]
   -h | --help			Show this usage info
   -o | --override-timeout	Number of seconds after which we timeout
 EOF
-	exit $1
+	exit "$1"
 }
 
 COLLECTIONS=""
@@ -51,7 +51,7 @@ while true; do
 	case "$1" in
 		-s | --summary)
 			logfile="$BASE_DIR"/output.log
-			cat /dev/null > $logfile
+			cat /dev/null > "$logfile"
 			shift ;;
 		-p | --per-test-log)
 			per_test_logging=1
@@ -127,7 +127,7 @@ collections=$(echo "$available" | cut -d: -f1 | sort | uniq)
 for collection in $collections ; do
 	[ -w /dev/kmsg ] && echo "kselftest: Running tests in $collection" >> /dev/kmsg
 	tests=$(echo "$available" | grep "^$collection:" | cut -d: -f2)
-	($dryrun cd "$collection" && $dryrun run_many $tests)
+	($dryrun cd "$collection" && $dryrun run_many "$tests")
 done
 
 failures="$(cat "$kselftest_failures_file")"

-- 
2.53.0

[PATCH 3/6] selftests/run_kselftest.sh: Resolve BASE_DIR with pwd -P

[Ricardo B. Marlière]

run_kselftest.sh only needs to canonicalize the directory containing the
script itself. Use shell-native path resolution for that by changing into
the directory and calling pwd -P.

This avoids depending on either realpath or readlink -f while still
producing a physical absolute path for BASE_DIR.

Signed-off-by: Ricardo B. Marlière <rbm@suse.com>
---
 tools/testing/selftests/run_kselftest.sh | 7 +------
 1 file changed, 1 insertion(+), 6 deletions(-)

diff --git a/tools/testing/selftests/run_kselftest.sh b/tools/testing/selftests/run_kselftest.sh
index b782b025d002..5f7bce9a640b 100755
--- a/tools/testing/selftests/run_kselftest.sh
+++ b/tools/testing/selftests/run_kselftest.sh
@@ -4,12 +4,7 @@
 # Run installed kselftest tests.
 #
 
-# Fallback to readlink if realpath is not available
-if which realpath > /dev/null; then
-        BASE_DIR=$(realpath $(dirname $0))
-else
-        BASE_DIR=$(readlink -f $(dirname $0))
-fi
+BASE_DIR=$(cd "$(dirname "$0")" && pwd -P)
 
 cd "$BASE_DIR"
 TESTS="$BASE_DIR"/kselftest-list.txt

-- 
2.53.0

[PATCH 4/6] selftests/run_kselftest.sh: Allow choosing per-test log directory

[Ricardo B. Marlière]

The --per-test-log option currently hard-codes /tmp. However, the system
under test will most likely have tmpfs mounted there. Since it's not clear
which filenames the log files will have, the user should be able to specify
a persistent directory to store the logs. Keeping those logs are important
because the run_kselftest.sh runner will only yield KTAP output, trimming
information that is otherwise available through running individual tests
directly.

Allow --per-test-log to take an optional directory argument. Keep the
existing behaviour when the option is passed without an argument, but if
a directory is provided, create it if needed, reject non-directory paths
and non-writable directories, canonicalize it, and have runner.sh write
per-test logs there instead of /tmp.

This also makes relative paths safe by resolving them before the runner
changes into a collection directory.

Signed-off-by: Ricardo B. Marlière <rbm@suse.com>
---
 tools/testing/selftests/kselftest/runner.sh |  3 ++-
 tools/testing/selftests/run_kselftest.sh    | 30 +++++++++++++++++++++++++++--
 2 files changed, 30 insertions(+), 3 deletions(-)

diff --git a/tools/testing/selftests/kselftest/runner.sh b/tools/testing/selftests/kselftest/runner.sh
index 3a62039fa621..6e13818fe433 100644
--- a/tools/testing/selftests/kselftest/runner.sh
+++ b/tools/testing/selftests/kselftest/runner.sh
@@ -6,6 +6,7 @@ export skip_rc=4
 export timeout_rc=124
 export logfile=/dev/stdout
 export per_test_logging=
+export per_test_log_dir=/tmp
 export RUN_IN_NETNS=
 
 # Defaults for "settings" file fields:
@@ -189,7 +190,7 @@ run_many()
 		BASENAME_TEST=$(basename $TEST)
 		test_num=$(( test_num + 1 ))
 		if [ -n "$per_test_logging" ]; then
-			logfile="/tmp/$BASENAME_TEST"
+			logfile="$per_test_log_dir/$BASENAME_TEST"
 			cat /dev/null > "$logfile"
 		fi
 		if [ -n "$RUN_IN_NETNS" ]; then
diff --git a/tools/testing/selftests/run_kselftest.sh b/tools/testing/selftests/run_kselftest.sh
index 5f7bce9a640b..fdbbe5b8b079 100755
--- a/tools/testing/selftests/run_kselftest.sh
+++ b/tools/testing/selftests/run_kselftest.sh
@@ -22,7 +22,7 @@ usage()
 	cat <<EOF
 Usage: $0 [OPTIONS]
   -s | --summary		Print summary with detailed log in output.log (conflict with -p)
-  -p | --per-test-log		Print test log in /tmp with each test name (conflict with -s)
+  -p | --per-test-log [DIR]	Print test log in /tmp or DIR with each test name (conflict with -s)
   -t | --test COLLECTION:TEST	Run TEST from COLLECTION
   -S | --skip COLLECTION:TEST	Skip TEST from COLLECTION
   -c | --collection COLLECTION	Run all tests from COLLECTION
@@ -50,7 +50,33 @@ while true; do
 			shift ;;
 		-p | --per-test-log)
 			per_test_logging=1
-			shift ;;
+			if [ -n "$2" ] && [ "${2#-}" = "$2" ]; then
+				per_test_log_dir="$2"
+				if [ -e "$per_test_log_dir" ] && [ ! -d "$per_test_log_dir" ]; then
+					echo "Per-test log path is not a dir:" \
+					     "$per_test_log_dir" >&2
+					exit 1
+				fi
+				if [ ! -d "$per_test_log_dir" ] && \
+				   ! mkdir -p "$per_test_log_dir"; then
+					echo "Could not create log dir:" \
+					     "$per_test_log_dir" >&2
+					exit 1
+				fi
+				per_test_log_dir=$(cd "$per_test_log_dir" && pwd -P)
+				if [ -z "$per_test_log_dir" ]; then
+					echo "Could not resolve per-test log directory" >&2
+					exit 1
+				fi
+				if [ ! -w "$per_test_log_dir" ]; then
+					echo "Per-test log dir is not writable:" \
+					     "$per_test_log_dir" >&2
+					exit 1
+				fi
+				shift 2
+			else
+				shift
+			fi ;;
 		-t | --test)
 			TESTS="$TESTS $2"
 			shift 2 ;;

-- 
2.53.0

[PATCH 5/6] selftests: Preserve subtarget failures in all/install

[Ricardo B. Marlière]

Track failures explicitly in the top-level selftests all/install loops.

The current code multiplies `ret` by each sub-make exit status. For
example, with `TARGETS=net`, the implicit `net/lib` dependency runs after
`net`, so a failed `net` build can be followed by a successful `net/lib`
build and reset the final result to success.

Set `ret` to 1 on any non-zero sub-make exit code and keep it sticky, so
the top-level make returns failure when any selected selftest target
fails.

Signed-off-by: Ricardo B. Marlière <rbm@suse.com>
---
 tools/testing/selftests/Makefile | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/tools/testing/selftests/Makefile b/tools/testing/selftests/Makefile
index 450f13ba4cca..0949f370ad78 100644
--- a/tools/testing/selftests/Makefile
+++ b/tools/testing/selftests/Makefile
@@ -209,14 +209,14 @@ export KHDR_INCLUDES
 .DEFAULT_GOAL := all
 
 all:
-	@ret=1;							\
+	@ret=0;							\
 	for TARGET in $(TARGETS) $(INSTALL_DEP_TARGETS); do	\
 		BUILD_TARGET=$$BUILD/$$TARGET;			\
 		mkdir $$BUILD_TARGET  -p;			\
 		$(MAKE) OUTPUT=$$BUILD_TARGET -C $$TARGET	\
 				O=$(abs_objtree)		\
 				$(if $(FORCE_TARGETS),|| exit);	\
-		ret=$$((ret * $$?));				\
+		[ $$? -eq 0 ] || ret=1;			\
 	done; exit $$ret;
 
 run_tests: all
@@ -274,7 +274,7 @@ ifdef INSTALL_PATH
 	install -m 744 kselftest/ksft.py $(INSTALL_PATH)/kselftest/
 	install -m 744 run_kselftest.sh $(INSTALL_PATH)/
 	rm -f $(TEST_LIST)
-	@ret=1;	\
+	@ret=0;	\
 	for TARGET in $(TARGETS) $(INSTALL_DEP_TARGETS); do \
 		BUILD_TARGET=$$BUILD/$$TARGET;	\
 		$(MAKE) OUTPUT=$$BUILD_TARGET -C $$TARGET install \
@@ -283,7 +283,7 @@ ifdef INSTALL_PATH
 				OBJ_PATH=$(INSTALL_PATH) \
 				O=$(abs_objtree)		\
 				$(if $(FORCE_TARGETS),|| exit);	\
-		ret=$$((ret * $$?));		\
+		[ $$? -eq 0 ] || ret=1;		\
 	done; exit $$ret;
 
 

-- 
2.53.0

[PATCH 6/6] kbuild: remove kselftest output in mrproper

[Ricardo B. Marlière]

Selftests built with O= store generated files by default under
$(objtree)/kselftest. Those files are not removed by mrproper today, so
stale generated artifacts can survive across builds and be reused
unexpectedly.

Add kselftest to MRPROPER_FILES, so that the default kselftest output
directory as part of mrproper target.

Signed-off-by: Ricardo B. Marlière <rbm@suse.com>
---
 Makefile | 1 +
 1 file changed, 1 insertion(+)

diff --git a/Makefile b/Makefile
index c9b7bee102e8..91e6ec790a0f 100644
--- a/Makefile
+++ b/Makefile
@@ -1662,6 +1662,7 @@ MRPROPER_FILES += include/config include/generated          \
 		  debian snap tar-install PKGBUILD pacman \
 		  .config .config.old .version \
 		  Module.symvers \
+		  kselftest \
 		  certs/signing_key.pem \
 		  certs/x509.genkey \
 		  vmlinux-gdb.py \

-- 
2.53.0

Re: [PATCH 0/6] selftests: run_kselftest.sh cleanup and fixes

[Shuah Khan]

On 3/20/26 12:29, Ricardo B. Marlière wrote:
> Hi Shuah,
> 
> Please consider merging the following patches. They are mostly small fixes
> to run_kselftest.sh, plus one feature: adding the option of passing a
> directory to the "-p --per-test-log" argument.
> 
> There is a small corner case spotted in tools/testing/selftests/Makefile
> regarding failing sub-targets, and also the removal of the default install
> directory for out of tree builds in the top-level Makefile.
> 
> Thank you!
> 
> Signed-off-by: Ricardo B. Marlière <rbm@suse.com>
> ---
> Ricardo B. Marlière (6):
>        selftests/run_kselftest.sh: Remove unused $ROOT
>        selftests/run_kselftest.sh: Add missing quotes
>        selftests/run_kselftest.sh: Resolve BASE_DIR with pwd -P
>        selftests/run_kselftest.sh: Allow choosing per-test log directory
>        selftests: Preserve subtarget failures in all/install
>        kbuild: remove kselftest output in mrproper
> 
>   Makefile                                    |  1 +
>   tools/testing/selftests/Makefile            |  8 ++---
>   tools/testing/selftests/kselftest/runner.sh |  3 +-
>   tools/testing/selftests/run_kselftest.sh    | 46 +++++++++++++++++++++--------
>   4 files changed, 40 insertions(+), 18 deletions(-)

Thank you Ricardo. I applied the series to linux_kselftest next

I had to fix a minor merge conflict when I applied patch v2.
Please check to see if everything looks sane.

thanks,
-- Shuah

Re: [PATCH 0/6] selftests: run_kselftest.sh cleanup and fixes

[Ricardo B. Marlière]

On Tue Mar 31, 2026 at 7:43 PM -03, Shuah Khan wrote:
> On 3/20/26 12:29, Ricardo B. Marlière wrote:
>> Hi Shuah,
>> 
>> Please consider merging the following patches. They are mostly small fixes
>> to run_kselftest.sh, plus one feature: adding the option of passing a
>> directory to the "-p --per-test-log" argument.
>> 
>> There is a small corner case spotted in tools/testing/selftests/Makefile
>> regarding failing sub-targets, and also the removal of the default install
>> directory for out of tree builds in the top-level Makefile.
>> 
>> Thank you!
>> 
>> Signed-off-by: Ricardo B. Marlière <rbm@suse.com>
>> ---
>> Ricardo B. Marlière (6):
>>        selftests/run_kselftest.sh: Remove unused $ROOT
>>        selftests/run_kselftest.sh: Add missing quotes
>>        selftests/run_kselftest.sh: Resolve BASE_DIR with pwd -P
>>        selftests/run_kselftest.sh: Allow choosing per-test log directory
>>        selftests: Preserve subtarget failures in all/install
>>        kbuild: remove kselftest output in mrproper
>> 
>>   Makefile                                    |  1 +
>>   tools/testing/selftests/Makefile            |  8 ++---
>>   tools/testing/selftests/kselftest/runner.sh |  3 +-
>>   tools/testing/selftests/run_kselftest.sh    | 46 +++++++++++++++++++++--------
>>   4 files changed, 40 insertions(+), 18 deletions(-)
>
> Thank you Ricardo. I applied the series to linux_kselftest next
>
> I had to fix a minor merge conflict when I applied patch v2.
> Please check to see if everything looks sane.

Can you please drop commit 33880424308d
("selftests/run_kselftest.sh: Add missing quotes") ? This actually
introduced a regression. I'm sorry about that.

>
> thanks,
> -- Shuah

Re: [PATCH 6/6] kbuild: remove kselftest output in mrproper

[Shuah Khan]

On 3/20/26 12:29, Ricardo B. Marlière wrote:
> Selftests built with O= store generated files by default under
> $(objtree)/kselftest. Those files are not removed by mrproper today, so
> stale generated artifacts can survive across builds and be reused
> unexpectedly.
> 
> Add kselftest to MRPROPER_FILES, so that the default kselftest output
> directory as part of mrproper target.

Does this change remove kselftest build objects in-tree builds?
It didn't for me - also there is kselftest directory under
tools/selftests that has several scripts.

Can you test this for in-tree builds - I would like this work
for both.

I dropped this from my next for now. This can go through kbuild
tree.

> 
> Signed-off-by: Ricardo B. Marlière <rbm@suse.com>
> ---
>   Makefile | 1 +
>   1 file changed, 1 insertion(+)
> 
> diff --git a/Makefile b/Makefile
> index c9b7bee102e8..91e6ec790a0f 100644
> --- a/Makefile
> +++ b/Makefile
> @@ -1662,6 +1662,7 @@ MRPROPER_FILES += include/config include/generated          \
>   		  debian snap tar-install PKGBUILD pacman \
>   		  .config .config.old .version \
>   		  Module.symvers \
> +		  kselftest \
>   		  certs/signing_key.pem \
>   		  certs/x509.genkey \
>   		  vmlinux-gdb.py \
> 

thanks,
-- Shuah

Re: [PATCH 6/6] kbuild: remove kselftest output in mrproper

[Ricardo B. Marlière]

On 4/10/26 4:19 PM, Shuah Khan wrote:
> On 3/20/26 12:29, Ricardo B. Marlière wrote:
>> Selftests built with O= store generated files by default under
>> $(objtree)/kselftest. Those files are not removed by mrproper today, so
>> stale generated artifacts can survive across builds and be reused
>> unexpectedly.
>>
>> Add kselftest to MRPROPER_FILES, so that the default kselftest output
>> directory as part of mrproper target.
>
> Does this change remove kselftest build objects in-tree builds?
> It didn't for me - also there is kselftest directory under
> tools/selftests that has several scripts.
>
> Can you test this for in-tree builds - I would like this work
> for both.
>
> I dropped this from my next for now. This can go through kbuild
> tree.

OK! But please also drop 
https://git.kernel.org/pub/scm/linux/kernel/git/shuah/linux-kselftest.git/commit/?h=next&id=33880424308df3c2e39fa88ea74f051205ebb7b4

Thanks!

>
>>
>> Signed-off-by: Ricardo B. Marlière <rbm@suse.com>
>> ---
>>   Makefile | 1 +
>>   1 file changed, 1 insertion(+)
>>
>> diff --git a/Makefile b/Makefile
>> index c9b7bee102e8..91e6ec790a0f 100644
>> --- a/Makefile
>> +++ b/Makefile
>> @@ -1662,6 +1662,7 @@ MRPROPER_FILES += include/config 
>> include/generated          \
>>             debian snap tar-install PKGBUILD pacman \
>>             .config .config.old .version \
>>             Module.symvers \
>> +          kselftest \
>>             certs/signing_key.pem \
>>             certs/x509.genkey \
>>             vmlinux-gdb.py \
>>
>
> thanks,
> -- Shuah

Re: [PATCH 6/6] kbuild: remove kselftest output in mrproper

[Shuah Khan]

On 4/11/26 06:49, Ricardo B. Marlière wrote:
> On 4/10/26 4:19 PM, Shuah Khan wrote:
>> On 3/20/26 12:29, Ricardo B. Marlière wrote:
>>> Selftests built with O= store generated files by default under
>>> $(objtree)/kselftest. Those files are not removed by mrproper today, so
>>> stale generated artifacts can survive across builds and be reused
>>> unexpectedly.
>>>
>>> Add kselftest to MRPROPER_FILES, so that the default kselftest output
>>> directory as part of mrproper target.
>>
>> Does this change remove kselftest build objects in-tree builds?
>> It didn't for me - also there is kselftest directory under
>> tools/selftests that has several scripts.
>>
>> Can you test this for in-tree builds - I would like this work
>> for both.
>>
>> I dropped this from my next for now. This can go through kbuild
>> tree.
> 
> OK! But please also drop https://git.kernel.org/pub/scm/linux/kernel/git/shuah/linux-kselftest.git/commit/?h=next&id=33880424308df3c2e39fa88ea74f051205ebb7b4
> 

I can do that - you didn't mention the reasons why you want me to drop
this one. Do other patches in this series depend on this one?

thanks,
-- Shuah

Re: [PATCH 6/6] kbuild: remove kselftest output in mrproper

[Ricardo B. Marlière]

On 4/13/26 12:58 PM, Shuah Khan wrote:
> On 4/11/26 06:49, Ricardo B. Marlière wrote:
>> On 4/10/26 4:19 PM, Shuah Khan wrote:
>>> On 3/20/26 12:29, Ricardo B. Marlière wrote:
>>>> Selftests built with O= store generated files by default under
>>>> $(objtree)/kselftest. Those files are not removed by mrproper 
>>>> today, so
>>>> stale generated artifacts can survive across builds and be reused
>>>> unexpectedly.
>>>>
>>>> Add kselftest to MRPROPER_FILES, so that the default kselftest output
>>>> directory as part of mrproper target.
>>>
>>> Does this change remove kselftest build objects in-tree builds?
>>> It didn't for me - also there is kselftest directory under
>>> tools/selftests that has several scripts.
>>>
>>> Can you test this for in-tree builds - I would like this work
>>> for both.
>>>
>>> I dropped this from my next for now. This can go through kbuild
>>> tree.
>>
>> OK! But please also drop 
>> https://git.kernel.org/pub/scm/linux/kernel/git/shuah/linux-kselftest.git/commit/?h=next&id=33880424308df3c2e39fa88ea74f051205ebb7b4
>>
>
> I can do that - you didn't mention the reasons why you want me to drop
> this one. Do other patches in this series depend on this one?

Others are fine. The problem with this one is this:


-    ($dryrun cd "$collection" && $dryrun run_many $tests)
+    ($dryrun cd "$collection" && $dryrun run_many "$tests")


but run_many expects word splitting.

>
> thanks,
> -- Shuah

Re: [PATCH 5/6] selftests: Preserve subtarget failures in all/install

[Mark Brown]

[-- Attachment #1: Type: text/plain, Size: 2321 bytes --]

On Fri, Mar 20, 2026 at 03:29:20PM -0300, Ricardo B. Marlière wrote:
> Track failures explicitly in the top-level selftests all/install loops.
> 
> The current code multiplies `ret` by each sub-make exit status. For
> example, with `TARGETS=net`, the implicit `net/lib` dependency runs after
> `net`, so a failed `net` build can be followed by a successful `net/lib`
> build and reset the final result to success.
> 
> Set `ret` to 1 on any non-zero sub-make exit code and keep it sticky, so
> the top-level make returns failure when any selected selftest target
> fails.

This patch, which is now in mainline as 7e47389142b8, is breaking a
bunch of CI systems - at least KernelCI, our Arm internal CI and my
personal stuff.  It causes the equivalent of FORCE_TARGETS behaviour in
the top level Makefile, the prior behaviour where the exit status of the
top level Makefile ignores failures from individual directories is
desirable since by default we try to build almost all the selftests but
between quality issues and build time dependencies it's very common for
at least one of them to fail.  With this commit unless the user has
configured a more restricted set of selftests it would be surprising if
we manage to get a successful build and install.

As well as being a poor default due to the very high likelyhood of build
failures this also has the undesirable effect of causing a build failure
in one selftest to cause the whole install target to fail, meaning that
the build failure is escallated to a complete lost of coverge for all
selftests in common CI usage.

This wasn't showing up in my -next build tests since I set FORCE_TARGETS
and explicitly choose a restricted set of kselftests which actually
build with my system and configuration.  It was less obvious than it
should have been with the other systems since they did not expect there
to be a complete failure to generate a kselftest tarball and variously
masked the error or reported it in a manner that looked like an
infrastructure issue.

It would be really nice to get to the point where we can reasonably do
this but we're simply not there at the current time.  At the moment if
people want to see build failures reported at the top level that really
needs to be opt in, we have FORCE_TARGETS for that.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [PATCH 5/6] selftests: Preserve subtarget failures in all/install

[Shuah Khan]

On 4/15/26 07:58, Mark Brown wrote:
> On Fri, Mar 20, 2026 at 03:29:20PM -0300, Ricardo B. Marlière wrote:
>> Track failures explicitly in the top-level selftests all/install loops.
>>
>> The current code multiplies `ret` by each sub-make exit status. For
>> example, with `TARGETS=net`, the implicit `net/lib` dependency runs after
>> `net`, so a failed `net` build can be followed by a successful `net/lib`
>> build and reset the final result to success.
>>
>> Set `ret` to 1 on any non-zero sub-make exit code and keep it sticky, so
>> the top-level make returns failure when any selected selftest target
>> fails.
> 
> This patch, which is now in mainline as 7e47389142b8, is breaking a
> bunch of CI systems - at least KernelCI, our Arm internal CI and my
> personal stuff.  It causes the equivalent of FORCE_TARGETS behaviour in
> the top level Makefile, the prior behaviour where the exit status of the
> top level Makefile ignores failures from individual directories is
> desirable since by default we try to build almost all the selftests but
> between quality issues and build time dependencies it's very common for
> at least one of them to fail.  With this commit unless the user has
> configured a more restricted set of selftests it would be surprising if
> we manage to get a successful build and install.
> 
> As well as being a poor default due to the very high likelyhood of build
> failures this also has the undesirable effect of causing a build failure
> in one selftest to cause the whole install target to fail, meaning that
> the build failure is escallated to a complete lost of coverge for all
> selftests in common CI usage.
> 
> This wasn't showing up in my -next build tests since I set FORCE_TARGETS
> and explicitly choose a restricted set of kselftests which actually
> build with my system and configuration.  It was less obvious than it
> should have been with the other systems since they did not expect there
> to be a complete failure to generate a kselftest tarball and variously
> masked the error or reported it in a manner that looked like an
> infrastructure issue.

I didn't see it when I did test on linux-next and my repo. I did install
to catch problems.

Sorry for not catching this. We can drop this patch.

> 
> It would be really nice to get to the point where we can reasonably do
> this but we're simply not there at the current time.  At the moment if
> people want to see build failures reported at the top level that really
> needs to be opt in, we have FORCE_TARGETS for that.

Good point - I will go look and see if we document this in kselftest doc
and add it.

Mark, would you like to a revert for this?

thanks,
-- Shuah

Re: [PATCH 5/6] selftests: Preserve subtarget failures in all/install

[Ricardo B. Marlière]

On Wed Apr 15, 2026 at 12:40 PM -03, Shuah Khan wrote:
> On 4/15/26 07:58, Mark Brown wrote:
>> On Fri, Mar 20, 2026 at 03:29:20PM -0300, Ricardo B. Marlière wrote:
>>> Track failures explicitly in the top-level selftests all/install loops.
>>>
>>> The current code multiplies `ret` by each sub-make exit status. For
>>> example, with `TARGETS=net`, the implicit `net/lib` dependency runs after
>>> `net`, so a failed `net` build can be followed by a successful `net/lib`
>>> build and reset the final result to success.
>>>
>>> Set `ret` to 1 on any non-zero sub-make exit code and keep it sticky, so
>>> the top-level make returns failure when any selected selftest target
>>> fails.
>> 
>> This patch, which is now in mainline as 7e47389142b8, is breaking a
>> bunch of CI systems - at least KernelCI, our Arm internal CI and my
>> personal stuff.  It causes the equivalent of FORCE_TARGETS behaviour in
>> the top level Makefile, the prior behaviour where the exit status of the
>> top level Makefile ignores failures from individual directories is
>> desirable since by default we try to build almost all the selftests but
>> between quality issues and build time dependencies it's very common for
>> at least one of them to fail.  With this commit unless the user has
>> configured a more restricted set of selftests it would be surprising if
>> we manage to get a successful build and install.
>> 
>> As well as being a poor default due to the very high likelyhood of build
>> failures this also has the undesirable effect of causing a build failure
>> in one selftest to cause the whole install target to fail, meaning that
>> the build failure is escallated to a complete lost of coverge for all
>> selftests in common CI usage.
>> 
>> This wasn't showing up in my -next build tests since I set FORCE_TARGETS
>> and explicitly choose a restricted set of kselftests which actually
>> build with my system and configuration.  It was less obvious than it
>> should have been with the other systems since they did not expect there
>> to be a complete failure to generate a kselftest tarball and variously
>> masked the error or reported it in a manner that looked like an
>> infrastructure issue.
>
> I didn't see it when I did test on linux-next and my repo. I did install
> to catch problems.
>
> Sorry for not catching this. We can drop this patch.
>
>> 
>> It would be really nice to get to the point where we can reasonably do
>> this but we're simply not there at the current time.  At the moment if
>> people want to see build failures reported at the top level that really
>> needs to be opt in, we have FORCE_TARGETS for that.
>
> Good point - I will go look and see if we document this in kselftest doc
> and add it.

It's not documented. It would have solved my issue, sorry for
overlooking this!

>
> Mark, would you like to a revert for this?
>
> thanks,
> -- Shuah

Re: [PATCH 5/6] selftests: Preserve subtarget failures in all/install

[Shuah Khan]

On 4/15/26 09:42, Ricardo B. Marlière wrote:
> On Wed Apr 15, 2026 at 12:40 PM -03, Shuah Khan wrote:
>> On 4/15/26 07:58, Mark Brown wrote:
>>> On Fri, Mar 20, 2026 at 03:29:20PM -0300, Ricardo B. Marlière wrote:
>>>> Track failures explicitly in the top-level selftests all/install loops.
>>>>
>>>> The current code multiplies `ret` by each sub-make exit status. For
>>>> example, with `TARGETS=net`, the implicit `net/lib` dependency runs after
>>>> `net`, so a failed `net` build can be followed by a successful `net/lib`
>>>> build and reset the final result to success.
>>>>
>>>> Set `ret` to 1 on any non-zero sub-make exit code and keep it sticky, so
>>>> the top-level make returns failure when any selected selftest target
>>>> fails.
>>>
>>> This patch, which is now in mainline as 7e47389142b8, is breaking a
>>> bunch of CI systems - at least KernelCI, our Arm internal CI and my
>>> personal stuff.  It causes the equivalent of FORCE_TARGETS behaviour in
>>> the top level Makefile, the prior behaviour where the exit status of the
>>> top level Makefile ignores failures from individual directories is
>>> desirable since by default we try to build almost all the selftests but
>>> between quality issues and build time dependencies it's very common for
>>> at least one of them to fail.  With this commit unless the user has
>>> configured a more restricted set of selftests it would be surprising if
>>> we manage to get a successful build and install.
>>>
>>> As well as being a poor default due to the very high likelyhood of build
>>> failures this also has the undesirable effect of causing a build failure
>>> in one selftest to cause the whole install target to fail, meaning that
>>> the build failure is escallated to a complete lost of coverge for all
>>> selftests in common CI usage.
>>>
>>> This wasn't showing up in my -next build tests since I set FORCE_TARGETS
>>> and explicitly choose a restricted set of kselftests which actually
>>> build with my system and configuration.  It was less obvious than it
>>> should have been with the other systems since they did not expect there
>>> to be a complete failure to generate a kselftest tarball and variously
>>> masked the error or reported it in a manner that looked like an
>>> infrastructure issue.
>>
>> I didn't see it when I did test on linux-next and my repo. I did install
>> to catch problems.
>>
>> Sorry for not catching this. We can drop this patch.
>>
>>>
>>> It would be really nice to get to the point where we can reasonably do
>>> this but we're simply not there at the current time.  At the moment if
>>> people want to see build failures reported at the top level that really
>>> needs to be opt in, we have FORCE_TARGETS for that.
>>
>> Good point - I will go look and see if we document this in kselftest doc
>> and add it.
> 
> It's not documented. It would have solved my issue, sorry for
> overlooking this!

Ricardo, Would you like to send me doc patch for this?

thanks,
-- Shuah

Re: [PATCH 5/6] selftests: Preserve subtarget failures in all/install

[Ricardo B. Marlière]

On Wed Apr 15, 2026 at 12:52 PM -03, Shuah Khan wrote:
> On 4/15/26 09:42, Ricardo B. Marlière wrote:
>> On Wed Apr 15, 2026 at 12:40 PM -03, Shuah Khan wrote:
>>> On 4/15/26 07:58, Mark Brown wrote:
>>>> On Fri, Mar 20, 2026 at 03:29:20PM -0300, Ricardo B. Marlière wrote:
>>>>> Track failures explicitly in the top-level selftests all/install loops.
>>>>>
>>>>> The current code multiplies `ret` by each sub-make exit status. For
>>>>> example, with `TARGETS=net`, the implicit `net/lib` dependency runs after
>>>>> `net`, so a failed `net` build can be followed by a successful `net/lib`
>>>>> build and reset the final result to success.
>>>>>
>>>>> Set `ret` to 1 on any non-zero sub-make exit code and keep it sticky, so
>>>>> the top-level make returns failure when any selected selftest target
>>>>> fails.
>>>>
>>>> This patch, which is now in mainline as 7e47389142b8, is breaking a
>>>> bunch of CI systems - at least KernelCI, our Arm internal CI and my
>>>> personal stuff.  It causes the equivalent of FORCE_TARGETS behaviour in
>>>> the top level Makefile, the prior behaviour where the exit status of the
>>>> top level Makefile ignores failures from individual directories is
>>>> desirable since by default we try to build almost all the selftests but
>>>> between quality issues and build time dependencies it's very common for
>>>> at least one of them to fail.  With this commit unless the user has
>>>> configured a more restricted set of selftests it would be surprising if
>>>> we manage to get a successful build and install.
>>>>
>>>> As well as being a poor default due to the very high likelyhood of build
>>>> failures this also has the undesirable effect of causing a build failure
>>>> in one selftest to cause the whole install target to fail, meaning that
>>>> the build failure is escallated to a complete lost of coverge for all
>>>> selftests in common CI usage.
>>>>
>>>> This wasn't showing up in my -next build tests since I set FORCE_TARGETS
>>>> and explicitly choose a restricted set of kselftests which actually
>>>> build with my system and configuration.  It was less obvious than it
>>>> should have been with the other systems since they did not expect there
>>>> to be a complete failure to generate a kselftest tarball and variously
>>>> masked the error or reported it in a manner that looked like an
>>>> infrastructure issue.
>>>
>>> I didn't see it when I did test on linux-next and my repo. I did install
>>> to catch problems.
>>>
>>> Sorry for not catching this. We can drop this patch.
>>>
>>>>
>>>> It would be really nice to get to the point where we can reasonably do
>>>> this but we're simply not there at the current time.  At the moment if
>>>> people want to see build failures reported at the top level that really
>>>> needs to be opt in, we have FORCE_TARGETS for that.
>>>
>>> Good point - I will go look and see if we document this in kselftest doc
>>> and add it.
>> 
>> It's not documented. It would have solved my issue, sorry for
>> overlooking this!
>
> Ricardo, Would you like to send me doc patch for this?

Sure, thanks!

>
> thanks,
> -- Shuah

Re: [PATCH 5/6] selftests: Preserve subtarget failures in all/install

[Mark Brown]

[-- Attachment #1: Type: text/plain, Size: 937 bytes --]

On Wed, Apr 15, 2026 at 09:40:37AM -0600, Shuah Khan wrote:
> On 4/15/26 07:58, Mark Brown wrote:

> > This wasn't showing up in my -next build tests since I set FORCE_TARGETS
> > and explicitly choose a restricted set of kselftests which actually
> > build with my system and configuration.  It was less obvious than it
> > should have been with the other systems since they did not expect there
> > to be a complete failure to generate a kselftest tarball and variously
> > masked the error or reported it in a manner that looked like an
> > infrastructure issue.

> I didn't see it when I did test on linux-next and my repo. I did install
> to catch problems.

> Sorry for not catching this. We can drop this patch.

Yeah, it's easy to miss if you're set up to build everything.

> Mark, would you like to a revert for this?

Yes, I'll massage the text from my report into a changelog send
something - it'll probably be tomorrow now.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [PATCH 5/6] selftests: Preserve subtarget failures in all/install

[Shuah Khan]

On 4/15/26 10:25, Mark Brown wrote:
> On Wed, Apr 15, 2026 at 09:40:37AM -0600, Shuah Khan wrote:
>> On 4/15/26 07:58, Mark Brown wrote:
> 
>>> This wasn't showing up in my -next build tests since I set FORCE_TARGETS
>>> and explicitly choose a restricted set of kselftests which actually
>>> build with my system and configuration.  It was less obvious than it
>>> should have been with the other systems since they did not expect there
>>> to be a complete failure to generate a kselftest tarball and variously
>>> masked the error or reported it in a manner that looked like an
>>> infrastructure issue.
> 
>> I didn't see it when I did test on linux-next and my repo. I did install
>> to catch problems.
> 
>> Sorry for not catching this. We can drop this patch.
> 
> Yeah, it's easy to miss if you're set up to build everything.
> 
>> Mark, would you like to a revert for this?
> 
> Yes, I'll massage the text from my report into a changelog send
> something - it'll probably be tomorrow now.

Thank you Mark

-- Shuah

Re: [PATCH 5/6] selftests: Preserve subtarget failures in all/install

[Mark Brown]

[-- Attachment #1: Type: text/plain, Size: 479 bytes --]

On Wed, Apr 15, 2026 at 05:25:40PM +0100, Mark Brown wrote:
> On Wed, Apr 15, 2026 at 09:40:37AM -0600, Shuah Khan wrote:

> > Mark, would you like to a revert for this?

> Yes, I'll massage the text from my report into a changelog send
> something - it'll probably be tomorrow now.

Sending shortly, FWIW I'm also seeing a buch of issues with:

./run_kselftest.sh: 5: ./kselftest/runner.sh: Bad substitution
./run_kselftest.sh: 5: .: cannot open ./ktap_helpers.sh: No such file

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [PATCH 5/6] selftests: Preserve subtarget failures in all/install

[Shuah Khan]

On 4/16/26 07:16, Mark Brown wrote:
> On Wed, Apr 15, 2026 at 05:25:40PM +0100, Mark Brown wrote:
>> On Wed, Apr 15, 2026 at 09:40:37AM -0600, Shuah Khan wrote:
> 
>>> Mark, would you like to a revert for this?
> 
>> Yes, I'll massage the text from my report into a changelog send
>> something - it'll probably be tomorrow now.
> 
> Sending shortly, FWIW I'm also seeing a buch of issues with:
> 
> ./run_kselftest.sh: 5: ./kselftest/runner.sh: Bad substitution
> ./run_kselftest.sh: 5: .: cannot open ./ktap_helpers.sh: No such file

I will take a look and isolate the two commits that changes
run_kselftest.sh - sorry for the trouble with CI this time
around.

thanks,
-- Shuah