[PATCH 0/5] selftests: harness: Detect illegal mixing of kselftest and harness functionality

[PATCH 0/5] selftests: harness: Detect illegal mixing of kselftest and harness functionality

[Thomas Weißschuh]

Users may accidentally use the kselftest_test_result_*() functions in
their harness tests. If ksft_finished() is not used, the results
reported in this way are silently ignored.

Detect such cases and fail the test.

This should probably only go in during the next cycle.

Signed-off-by: Thomas Weißschuh <thomas.weissschuh@linutronix.de>
---
Thomas Weißschuh (5):
      selftests: kselftest: Treat xpass as successful result
      selftests: harness: Validate that explicit kselftest exitcodes are handled
      selftests: kselftest: Add ksft_reset_state()
      selftests: harness: Detect illegal mixing of kselftest and harness functionality
      selftests: harness: Validate intermixing of kselftest and harness functionality

 tools/testing/selftests/kselftest.h                | 12 +++++
 tools/testing/selftests/kselftest_harness.h        |  9 ++++
 .../selftests/kselftest_harness/harness-selftest.c | 40 +++++++++++++++++
 .../kselftest_harness/harness-selftest.expected    | 52 ++++++++++++++++++----
 4 files changed, 104 insertions(+), 9 deletions(-)
---
base-commit: 6d6ad32e22f028c525d5df471c5522616e645a6b
change-id: 20260128-kselftest-harness-5e07ab07f58a

Best regards,
-- 
Thomas Weißschuh <thomas.weissschuh@linutronix.de>

[PATCH 1/5] selftests: kselftest: Treat xpass as successful result

[Thomas Weißschuh]

The harness treats these tests as successful, as does pytest.

Align kselftest.h to the rest of the ecosystem.

None of the Linux selftests seem to actually use this anyways.

Signed-off-by: Thomas Weißschuh <thomas.weissschuh@linutronix.de>
---
 tools/testing/selftests/kselftest.h | 1 +
 1 file changed, 1 insertion(+)

diff --git a/tools/testing/selftests/kselftest.h b/tools/testing/selftests/kselftest.h
index afbcf8412ae5..1e7e73f77fa7 100644
--- a/tools/testing/selftests/kselftest.h
+++ b/tools/testing/selftests/kselftest.h
@@ -399,6 +399,7 @@ static inline __noreturn void ksft_exit_fail(void)
 #define ksft_finished()			\
 	ksft_exit(ksft_plan ==		\
 		  ksft_cnt.ksft_pass +	\
+		  ksft_cnt.ksft_xpass +	\
 		  ksft_cnt.ksft_xfail +	\
 		  ksft_cnt.ksft_xskip)
 

-- 
2.52.0

[PATCH 2/5] selftests: harness: Validate that explicit kselftest exitcodes are handled

[Thomas Weißschuh]

The test programs can directly call exit with one of the KSFT_* constants.

Add tests for this functionality.

Signed-off-by: Thomas Weißschuh <thomas.weissschuh@linutronix.de>
---
 .../selftests/kselftest_harness/harness-selftest.c | 20 +++++++++++++
 .../kselftest_harness/harness-selftest.expected    | 35 ++++++++++++++++------
 2 files changed, 46 insertions(+), 9 deletions(-)

diff --git a/tools/testing/selftests/kselftest_harness/harness-selftest.c b/tools/testing/selftests/kselftest_harness/harness-selftest.c
index 7820bb5d0e6d..f109a21ed6f7 100644
--- a/tools/testing/selftests/kselftest_harness/harness-selftest.c
+++ b/tools/testing/selftests/kselftest_harness/harness-selftest.c
@@ -118,6 +118,26 @@ TEST_F(fixture_setup_failure, pass) {
 	TH_LOG("after");
 }
 
+TEST(exit_pass) {
+	exit(KSFT_PASS);
+}
+
+TEST(exit_xpass) {
+	exit(KSFT_XPASS);
+}
+
+TEST(exit_fail) {
+	exit(KSFT_FAIL);
+}
+
+TEST(exit_xfail) {
+	exit(KSFT_XFAIL);
+}
+
+TEST(exit_skip) {
+	exit(KSFT_SKIP);
+}
+
 int main(int argc, char **argv)
 {
 	/*
diff --git a/tools/testing/selftests/kselftest_harness/harness-selftest.expected b/tools/testing/selftests/kselftest_harness/harness-selftest.expected
index 97e1418c1c7e..da7febc11e10 100644
--- a/tools/testing/selftests/kselftest_harness/harness-selftest.expected
+++ b/tools/testing/selftests/kselftest_harness/harness-selftest.expected
@@ -1,6 +1,6 @@
 TAP version 13
-1..9
-# Starting 9 tests from 4 test cases.
+1..14
+# Starting 14 tests from 4 test cases.
 #  RUN           global.standalone_pass ...
 # harness-selftest.c:19:standalone_pass:before
 # harness-selftest.c:23:standalone_pass:after
@@ -24,6 +24,22 @@ ok 3 global.signal_pass
 # signal_fail: Test terminated by assertion
 #          FAIL  global.signal_fail
 not ok 4 global.signal_fail
+#  RUN           global.exit_pass ...
+#            OK  global.exit_pass
+ok 5 global.exit_pass
+#  RUN           global.exit_xpass ...
+#            OK  global.exit_xpass
+ok 6 global.exit_xpass # XPASS unknown
+#  RUN           global.exit_fail ...
+# exit_fail: Test failed
+#          FAIL  global.exit_fail
+not ok 7 global.exit_fail
+#  RUN           global.exit_xfail ...
+#            OK  global.exit_xfail
+ok 8 global.exit_xfail # XFAIL unknown
+#  RUN           global.exit_skip ...
+#            OK  global.exit_skip
+ok 9 global.exit_skip # SKIP unknown
 #  RUN           fixture.pass ...
 # harness-selftest.c:53:pass:setup
 # harness-selftest.c:62:pass:before
@@ -32,7 +48,7 @@ not ok 4 global.signal_fail
 # harness-selftest.c:66:pass:after
 # harness-selftest.c:58:pass:teardown same-process=1
 #            OK  fixture.pass
-ok 5 fixture.pass
+ok 10 fixture.pass
 #  RUN           fixture.fail ...
 # harness-selftest.c:53:fail:setup
 # harness-selftest.c:70:fail:before
@@ -40,25 +56,26 @@ ok 5 fixture.pass
 # harness-selftest.c:58:fail:teardown same-process=1
 # fail: Test terminated by assertion
 #          FAIL  fixture.fail
-not ok 6 fixture.fail
+not ok 11 fixture.fail
 #  RUN           fixture.timeout ...
 # harness-selftest.c:53:timeout:setup
 # harness-selftest.c:77:timeout:before
 # timeout: Test terminated by timeout
 #          FAIL  fixture.timeout
-not ok 7 fixture.timeout
+not ok 12 fixture.timeout
 #  RUN           fixture_parent.pass ...
 # harness-selftest.c:87:pass:setup
 # harness-selftest.c:96:pass:before
 # harness-selftest.c:98:pass:after
 # harness-selftest.c:92:pass:teardown same-process=0
 #            OK  fixture_parent.pass
-ok 8 fixture_parent.pass
+ok 13 fixture_parent.pass
 #  RUN           fixture_setup_failure.pass ...
 # harness-selftest.c:106:pass:setup
 # harness-selftest.c:108:pass:Expected 0 (0) == 1 (1)
 # pass: Test terminated by assertion
 #          FAIL  fixture_setup_failure.pass
-not ok 9 fixture_setup_failure.pass
-# FAILED: 4 / 9 tests passed.
-# Totals: pass:4 fail:5 xfail:0 xpass:0 skip:0 error:0
+not ok 14 fixture_setup_failure.pass
+# FAILED: 8 / 14 tests passed.
+# 1 skipped test(s) detected. Consider enabling relevant config options to improve coverage.
+# Totals: pass:5 fail:6 xfail:1 xpass:1 skip:1 error:0

-- 
2.52.0

[PATCH 3/5] selftests: kselftest: Add ksft_reset_state()

[Thomas Weißschuh]

Add a helper to reset the internal state of the kselftest framework.
It will be used by the selftest harness.

Signed-off-by: Thomas Weißschuh <thomas.weissschuh@linutronix.de>
---
 tools/testing/selftests/kselftest.h | 11 +++++++++++
 1 file changed, 11 insertions(+)

diff --git a/tools/testing/selftests/kselftest.h b/tools/testing/selftests/kselftest.h
index 1e7e73f77fa7..7979d6904ac6 100644
--- a/tools/testing/selftests/kselftest.h
+++ b/tools/testing/selftests/kselftest.h
@@ -476,4 +476,15 @@ static inline int ksft_min_kernel_version(unsigned int min_major,
 	return major > min_major || (major == min_major && minor >= min_minor);
 }
 
+static inline void ksft_reset_state(void)
+{
+	ksft_cnt.ksft_pass = 0;
+	ksft_cnt.ksft_fail = 0;
+	ksft_cnt.ksft_xfail = 0;
+	ksft_cnt.ksft_xpass = 0;
+	ksft_cnt.ksft_xskip = 0;
+	ksft_cnt.ksft_error = 0;
+	ksft_plan = 0;
+};
+
 #endif /* __KSELFTEST_H */

-- 
2.52.0

Re: [PATCH 3/5] selftests: kselftest: Add ksft_reset_state()

[kernel test robot]

Hi Thomas,

kernel test robot noticed the following build warnings:

[auto build test WARNING on 6d6ad32e22f028c525d5df471c5522616e645a6b]

url:    https://github.com/intel-lab-lkp/linux/commits/Thomas-Wei-schuh/selftests-kselftest-Treat-xpass-as-successful-result/20260128-182856
base:   6d6ad32e22f028c525d5df471c5522616e645a6b
patch link:    https://lore.kernel.org/r/20260128-kselftest-harness-v1-3-cb259fa81cd0%40linutronix.de
patch subject: [PATCH 3/5] selftests: kselftest: Add ksft_reset_state()
config: x86_64-allnoconfig-bpf (https://download.01.org/0day-ci/archive/20260128/202601282352.QFHPcLUZ-lkp@intel.com/config)
compiler: gcc-14 (Debian 14.2.0-19) 14.2.0
reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20260128/202601282352.QFHPcLUZ-lkp@intel.com/reproduce)

If you fix the issue in a separate patch/commit (i.e. not just a new version of
the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <lkp@intel.com>
| Closes: https://lore.kernel.org/oe-kbuild-all/202601282352.QFHPcLUZ-lkp@intel.com/

All warnings (new ones prefixed by >>):

   In file included from ovpn-cli.c:35:
>> tools/testing/selftests/../../../tools/testing/selftests/kselftest.h:488:2: warning: ISO C does not allow extra ';' outside of a function [-Wpedantic]
     488 | };
         |  ^


vim +488 tools/testing/selftests/../../../tools/testing/selftests/kselftest.h

   478	
   479	static inline void ksft_reset_state(void)
   480	{
   481		ksft_cnt.ksft_pass = 0;
   482		ksft_cnt.ksft_fail = 0;
   483		ksft_cnt.ksft_xfail = 0;
   484		ksft_cnt.ksft_xpass = 0;
   485		ksft_cnt.ksft_xskip = 0;
   486		ksft_cnt.ksft_error = 0;
   487		ksft_plan = 0;
 > 488	};
   489	

-- 
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki

Re: [PATCH 3/5] selftests: kselftest: Add ksft_reset_state()

[kernel test robot]

Hi Thomas,

kernel test robot noticed the following build warnings:

[auto build test WARNING on 6d6ad32e22f028c525d5df471c5522616e645a6b]

url:    https://github.com/intel-lab-lkp/linux/commits/Thomas-Wei-schuh/selftests-kselftest-Treat-xpass-as-successful-result/20260128-182856
base:   6d6ad32e22f028c525d5df471c5522616e645a6b
patch link:    https://lore.kernel.org/r/20260128-kselftest-harness-v1-3-cb259fa81cd0%40linutronix.de
patch subject: [PATCH 3/5] selftests: kselftest: Add ksft_reset_state()
config: i386-allnoconfig-bpf (https://download.01.org/0day-ci/archive/20260129/202601290039.37LnFCGx-lkp@intel.com/config)
compiler: clang version 20.1.8 (https://github.com/llvm/llvm-project 87f0227cb60147a26a1eeb4fb06e3b505e9c7261)
reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20260129/202601290039.37LnFCGx-lkp@intel.com/reproduce)

If you fix the issue in a separate patch/commit (i.e. not just a new version of
the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <lkp@intel.com>
| Closes: https://lore.kernel.org/oe-kbuild-all/202601290039.37LnFCGx-lkp@intel.com/

All warnings (new ones prefixed by >>):

   In file included from ovpn-cli.c:35:
>> tools/testing/selftests/../../../tools/testing/selftests/kselftest.h:488:2: warning: extra ';' outside of a function [-Wextra-semi]
     488 | };
         |  ^
   1 warning generated.


vim +488 tools/testing/selftests/../../../tools/testing/selftests/kselftest.h

   478	
   479	static inline void ksft_reset_state(void)
   480	{
   481		ksft_cnt.ksft_pass = 0;
   482		ksft_cnt.ksft_fail = 0;
   483		ksft_cnt.ksft_xfail = 0;
   484		ksft_cnt.ksft_xpass = 0;
   485		ksft_cnt.ksft_xskip = 0;
   486		ksft_cnt.ksft_error = 0;
   487		ksft_plan = 0;
 > 488	};
   489	

-- 
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki

[PATCH 4/5] selftests: harness: Detect illegal mixing of kselftest and harness functionality

[Thomas Weißschuh]

Users may accidentally use the kselftest_test_result_*() functions in
their harness tests. If ksft_finished() is not used, the results
reported in this way are silently ignored.

Detect such false-positive cases and fail the test.

A more correct test would be to reject *any* usage of the ksft APIs but
that would force code churn on users.

Correct usages, which do use ksft_finished() will not trigger this
validation as the test will exit before it.

Reported-by: Yuwen Chen <ywen.chen@foxmail.com>
Link: https://lore.kernel.org/lkml/tencent_56D79AF3D23CEFAF882E83A2196EC1F12107@qq.com/
Signed-off-by: Thomas Weißschuh <thomas.weissschuh@linutronix.de>

---
This is intentionally a Link tag over a Closes tag, as the real fix will
be to the selftests.

Signed-off-by: Thomas Weißschuh <thomas.weissschuh@linutronix.de>
---
 tools/testing/selftests/kselftest_harness.h | 9 +++++++++
 1 file changed, 9 insertions(+)

diff --git a/tools/testing/selftests/kselftest_harness.h b/tools/testing/selftests/kselftest_harness.h
index baae6b7ded41..93dcba095338 100644
--- a/tools/testing/selftests/kselftest_harness.h
+++ b/tools/testing/selftests/kselftest_harness.h
@@ -1216,7 +1216,16 @@ static void __run_test(struct __fixture_metadata *f,
 		t->exit_code = KSFT_FAIL;
 	} else if (child == 0) {
 		setpgrp();
+
+		/* Reset state inherited from the harness */
+		ksft_reset_state();
+
 		t->fn(t, variant);
+
+		if (__test_passed(t) && (ksft_get_fail_cnt() || ksft_get_error_cnt())) {
+			ksft_print_msg("Illegal usage of low-level ksft APIs in harness test\n");
+			t->exit_code = KSFT_FAIL;
+		}
 		_exit(t->exit_code);
 	} else {
 		t->pid = child;

-- 
2.52.0

[PATCH 5/5] selftests: harness: Validate intermixing of kselftest and harness functionality

[Thomas Weißschuh]

Make sure that calling ksft_test_result_*() functions from harness
tests work as expected.

Signed-off-by: Thomas Weißschuh <thomas.weissschuh@linutronix.de>
---
 .../selftests/kselftest_harness/harness-selftest.c | 20 +++++++++++++
 .../kselftest_harness/harness-selftest.expected    | 35 ++++++++++++++++------
 2 files changed, 46 insertions(+), 9 deletions(-)

diff --git a/tools/testing/selftests/kselftest_harness/harness-selftest.c b/tools/testing/selftests/kselftest_harness/harness-selftest.c
index f109a21ed6f7..3e3033d2745a 100644
--- a/tools/testing/selftests/kselftest_harness/harness-selftest.c
+++ b/tools/testing/selftests/kselftest_harness/harness-selftest.c
@@ -138,6 +138,26 @@ TEST(exit_skip) {
 	exit(KSFT_SKIP);
 }
 
+TEST(test_result_pass) {
+	ksft_test_result_pass("");
+}
+
+TEST(test_result_xpass) {
+	ksft_test_result_xpass("");
+}
+
+TEST(test_result_fail) {
+	ksft_test_result_fail("");
+}
+
+TEST(test_result_xfail) {
+	ksft_test_result_xfail("");
+}
+
+TEST(test_result_skip) {
+	ksft_test_result_skip("");
+}
+
 int main(int argc, char **argv)
 {
 	/*
diff --git a/tools/testing/selftests/kselftest_harness/harness-selftest.expected b/tools/testing/selftests/kselftest_harness/harness-selftest.expected
index da7febc11e10..c0a68ec124f8 100644
--- a/tools/testing/selftests/kselftest_harness/harness-selftest.expected
+++ b/tools/testing/selftests/kselftest_harness/harness-selftest.expected
@@ -1,6 +1,6 @@
 TAP version 13
-1..14
-# Starting 14 tests from 4 test cases.
+1..19
+# Starting 19 tests from 4 test cases.
 #  RUN           global.standalone_pass ...
 # harness-selftest.c:19:standalone_pass:before
 # harness-selftest.c:23:standalone_pass:after
@@ -40,6 +40,23 @@ ok 8 global.exit_xfail # XFAIL unknown
 #  RUN           global.exit_skip ...
 #            OK  global.exit_skip
 ok 9 global.exit_skip # SKIP unknown
+#  RUN           global.test_result_pass ...
+#            OK  global.test_result_pass
+ok 10 global.test_result_pass
+#  RUN           global.test_result_xpass ...
+#            OK  global.test_result_xpass
+ok 11 global.test_result_xpass
+#  RUN           global.test_result_fail ...
+not ok 1 # Illegal usage of low-level ksft APIs in harness test
+# test_result_fail: Test failed
+#          FAIL  global.test_result_fail
+not ok 12 global.test_result_fail
+#  RUN           global.test_result_xfail ...
+#            OK  global.test_result_xfail
+ok 13 global.test_result_xfail
+#  RUN           global.test_result_skip ...
+#            OK  global.test_result_skip
+ok 14 global.test_result_skip
 #  RUN           fixture.pass ...
 # harness-selftest.c:53:pass:setup
 # harness-selftest.c:62:pass:before
@@ -48,7 +65,7 @@ ok 9 global.exit_skip # SKIP unknown
 # harness-selftest.c:66:pass:after
 # harness-selftest.c:58:pass:teardown same-process=1
 #            OK  fixture.pass
-ok 10 fixture.pass
+ok 15 fixture.pass
 #  RUN           fixture.fail ...
 # harness-selftest.c:53:fail:setup
 # harness-selftest.c:70:fail:before
@@ -56,26 +73,26 @@ ok 10 fixture.pass
 # harness-selftest.c:58:fail:teardown same-process=1
 # fail: Test terminated by assertion
 #          FAIL  fixture.fail
-not ok 11 fixture.fail
+not ok 16 fixture.fail
 #  RUN           fixture.timeout ...
 # harness-selftest.c:53:timeout:setup
 # harness-selftest.c:77:timeout:before
 # timeout: Test terminated by timeout
 #          FAIL  fixture.timeout
-not ok 12 fixture.timeout
+not ok 17 fixture.timeout
 #  RUN           fixture_parent.pass ...
 # harness-selftest.c:87:pass:setup
 # harness-selftest.c:96:pass:before
 # harness-selftest.c:98:pass:after
 # harness-selftest.c:92:pass:teardown same-process=0
 #            OK  fixture_parent.pass
-ok 13 fixture_parent.pass
+ok 18 fixture_parent.pass
 #  RUN           fixture_setup_failure.pass ...
 # harness-selftest.c:106:pass:setup
 # harness-selftest.c:108:pass:Expected 0 (0) == 1 (1)
 # pass: Test terminated by assertion
 #          FAIL  fixture_setup_failure.pass
-not ok 14 fixture_setup_failure.pass
-# FAILED: 8 / 14 tests passed.
+not ok 19 fixture_setup_failure.pass
+# FAILED: 12 / 19 tests passed.
 # 1 skipped test(s) detected. Consider enabling relevant config options to improve coverage.
-# Totals: pass:5 fail:6 xfail:1 xpass:1 skip:1 error:0
+# Totals: pass:9 fail:7 xfail:1 xpass:1 skip:1 error:0

-- 
2.52.0