[PATCH v6 00/11] kunit: Introduce UAPI testing framework

["Thomas Weißschuh" <thomas.weissschuh@linutronix.de>]

Currently testing of userspace and in-kernel API use two different
frameworks. kselftests for the userspace ones and Kunit for the
in-kernel ones. Besides their different scopes, both have different
strengths and limitations:

Kunit:
* Tests are normal kernel code.
* They use the regular kernel toolchain.
* They can be packaged and distributed as modules conveniently.

Kselftests:
* Tests are normal userspace code
* They need a userspace toolchain.
  A kernel cross toolchain is likely not enough.
* A fair amout of userland is required to run the tests,
  which means a full distro or handcrafted rootfs.
* There is no way to conveniently package and run kselftests with a
  given kernel image.
* The kselftests makefiles are not as powerful as regular kbuild.
  For example they are missing proper header dependency tracking or more
  complex compiler option modifications.

Therefore kunit is much easier to run against different kernel
configurations and architectures.
This series aims to combine kselftests and kunit, avoiding both their
limitations. It works by compiling the userspace kselftests as part of
the regular kernel build, embedding them into the kunit kernel or module
and executing them from there. If the kernel toolchain is not fit to
produce userspace because of a missing libc, the kernel's own nolibc can
be used instead.
The structured TAP output from the kselftest is integrated into the
kunit KTAP output transparently, the kunit parser can parse the combined
logs together.

Further room for improvements:
* Call each test in its completely dedicated namespace
* Handle additional test files besides the test executable through
  archives. CPIO, cramfs, etc.
* Expose the blobs in debugfs
* Provide some convience wrappers around compat userprogs
* Figure out a migration path/coexistence solution for
  kunit UAPI and tools/testing/selftests/

Output from the kunit example testcase, note the output of
"example_uapi_tests".

$ ./tools/testing/kunit/kunit.py run --kunitconfig lib/kunit example
...
Running tests with:
$ .kunit/linux kunit.filter_glob=example kunit.enable=1 mem=1G console=tty kunit_shutdown=halt
[11:53:53] ================== example (10 subtests) ===================
[11:53:53] [PASSED] example_simple_test
[11:53:53] [SKIPPED] example_skip_test
[11:53:53] [SKIPPED] example_mark_skipped_test
[11:53:53] [PASSED] example_all_expect_macros_test
[11:53:53] [PASSED] example_static_stub_test
[11:53:53] [PASSED] example_static_stub_using_fn_ptr_test
[11:53:53] [PASSED] example_priv_test
[11:53:53] =================== example_params_test  ===================
[11:53:53] [SKIPPED] example value 3
[11:53:53] [PASSED] example value 2
[11:53:53] [PASSED] example value 1
[11:53:53] [SKIPPED] example value 0
[11:53:53] =============== [PASSED] example_params_test ===============
[11:53:53] [PASSED] example_slow_test
[11:53:53] ======================= (4 subtests) =======================
[11:53:53] [PASSED] procfs
[11:53:53] [PASSED] userspace test 2
[11:53:53] [SKIPPED] userspace test 3: some reason
[11:53:53] [PASSED] userspace test 4
[11:53:53] ================ [PASSED] example_uapi_test ================
[11:53:53] ===================== [PASSED] example =====================
[11:53:53] ============================================================
[11:53:53] Testing complete. Ran 16 tests: passed: 11, skipped: 5
[11:53:53] Elapsed time: 67.543s total, 1.823s configuring, 65.655s building, 0.058s running

---
Changes in v6:
- Use usermode helper framework
- Trim now unnecessary new module exports
- Report unhandled output line
- Add SUPERH to ARCH_HAS_NOLIBC
- Remove non-integral patches submitted elsewhere
- Drop CC_CAN_LINK_STATIC support. It is based on CC_CAN_LINK which is
  currently being reworked.
- Use more cleanup helpers
- Split stdout/stderr forwarding into its own patch
- Link to v5: https://lore.kernel.org/r/20250717-kunit-kselftests-v5-0-442b711cde2e@linutronix.de

Changes in v5:
- Initialize output variable of kernel_wait()
- Fix .incbin with in-tree builds
- Keep requirement of KTAP tests to have a number which was removed accidentally
- Only synthesize KTAP subtest failure if the outer one is TestStatus.FAILURE
- Use -I instead of -isystem in NOLIBC_USERCFLAGS to populate dependency files
- +To filesystem developers to all patches
- +To Luis Chamberlain for discussions about usage of usermodehelper
  (see patches 6 and 12)
- Link to v4: https://lore.kernel.org/r/20250626-kunit-kselftests-v4-0-48760534fef5@linutronix.de

Changes in v4:
- Move Kconfig.nolibc from tools/ to init/
- Drop generic userprogs nolibc integration
- Drop generic blob framework
- Pick up review tags from David
- Extend new kunit TAP parser tests
- Add MAINTAINERS entry
- Allow CONFIG_KUNIT_UAPI=m
- Split /proc validation into dedicated UAPI test
- Trim recipient list a bit
- Use KUNIT_FAIL_AND_ABORT() over KUNIT_FAIL()
- Link to v3: https://lore.kernel.org/r/20250611-kunit-kselftests-v3-0-55e3d148cbc6@linutronix.de

Changes in v3:
- Reintroduce CONFIG_CC_CAN_LINK_STATIC
- Enable CONFIG_ARCH_HAS_NOLIBC for m68k and SPARC
- Properly handle 'clean' target for userprogs
- Use ramfs over tmpfs to reduce dependencies
- Inherit userprogs byte order and ABI from kernel
- Drop now unnecessary "#ifndef NOLIBC"
- Pick up review tags
- Drop usage of __private in blob.h,
  sparse complains and it is not really necessary
- Fix execution on loongarch when using clang
- Drop userprogs libgcc handling, it was ugly and is not yet necessary
- Link to v2: https://lore.kernel.org/r/20250407-kunit-kselftests-v2-0-454114e287fd@linutronix.de

Changes in v2:
- Rebase onto v6.15-rc1
- Add documentation and kernel docs
- Resolve invalid kconfig breakages
- Drop already applied patch "kbuild: implement CONFIG_HEADERS_INSTALL for Usermode Linux"
- Drop userprogs CONFIG_WERROR integration, it doesn't need to be part of this series
- Replace patch prefix "kconfig" with "kbuild"
- Rename kunit_uapi_run_executable() to kunit_uapi_run_kselftest()
- Generate private, conflict-free symbols in the blob framework
- Handle kselftest exit codes
- Handle SIGABRT
- Forward output also to kunit debugfs log
- Install a fd=0 stdin filedescriptor
- Link to v1: https://lore.kernel.org/r/20250217-kunit-kselftests-v1-0-42b4524c3b0a@linutronix.de

To: Nathan Chancellor <nathan@kernel.org>
To: Andrew Morton <akpm@linux-foundation.org>
To: Willy Tarreau <w@1wt.eu>
To: Thomas Weißschuh <linux@weissschuh.net>
To: Brendan Higgins <brendan.higgins@linux.dev>
To: David Gow <davidgow@google.com>
To: Rae Moar <rmoar@google.com>
To: Shuah Khan <shuah@kernel.org>
To: Jonathan Corbet <corbet@lwn.net>
To: Nicolas Schier <nicolas.schier@linux.dev>
Cc: Christophe Leroy <christophe.leroy@csgroup.eu>
Cc: linux-kbuild@vger.kernel.org
Cc: linux-kernel@vger.kernel.org
Cc: linux-kselftest@vger.kernel.org
Cc: kunit-dev@googlegroups.com
Cc: linux-doc@vger.kernel.org
Cc: workflows@vger.kernel.org
To: Kees Cook <kees@kernel.org>
To: Alexander Viro <viro@zeniv.linux.org.uk>
To: Christian Brauner <brauner@kernel.org>
To: Jan Kara <jack@suse.cz>
To: Christoph Hellwig <hch@lst.de>
To: Luis Chamberlain <mcgrof@kernel.org>
Cc: linux-mm@kvack.org
Cc: linux-fsdevel@vger.kernel.org
Cc: Nicolas Schier <n.schier@avm.de>
Signed-off-by: Thomas Weißschuh <thomas.weissschuh@linutronix.de>

---
Thomas Weißschuh (11):
      kbuild: doc: add label for userprogs section
      exec: add dirfd parameter to kernel_execve()
      umh: add dirfd parameter
      mount: add support for __free(kern_unmount)
      init: add nolibc build support
      kunit: qemu_configs: loongarch: Enable LSX/LSAX
      kunit: Introduce UAPI testing framework
      kunit: uapi: Forward test executable output to KUnit log
      kunit: uapi: Add example for UAPI tests
      kunit: uapi: Introduce preinit executable
      kunit: uapi: Validate usability of /proc

 Documentation/dev-tools/kunit/api/index.rst   |   5 +
 Documentation/dev-tools/kunit/api/uapi.rst    |  14 +
 Documentation/kbuild/makefiles.rst            |   2 +
 MAINTAINERS                                   |  11 +
 fs/coredump.c                                 |   2 +-
 fs/exec.c                                     |   4 +-
 include/kunit/uapi.h                          |  77 ++++++
 include/linux/binfmts.h                       |   2 +-
 include/linux/mount.h                         |   2 +
 include/linux/umh.h                           |   3 +-
 init/Kconfig                                  |   2 +
 init/Kconfig.nolibc                           |  16 ++
 init/Makefile.nolibc                          |  13 +
 init/main.c                                   |   2 +-
 kernel/module/kmod.c                          |   2 +-
 kernel/umh.c                                  |   9 +-
 lib/kobject_uevent.c                          |   2 +-
 lib/kunit/Kconfig                             |  16 ++
 lib/kunit/Makefile                            |  26 ++
 lib/kunit/kunit-example-test.c                |  15 ++
 lib/kunit/kunit-example-uapi.c                |  22 ++
 lib/kunit/kunit-test-uapi.c                   |  51 ++++
 lib/kunit/kunit-test.c                        |  24 +-
 lib/kunit/kunit-uapi.c                        | 351 ++++++++++++++++++++++++++
 lib/kunit/uapi-preinit.c                      |  68 +++++
 security/keys/request_key.c                   |   2 +-
 tools/testing/kunit/qemu_configs/loongarch.py |   2 +
 27 files changed, 732 insertions(+), 13 deletions(-)
---
base-commit: 813d26ba48f9e3909d39966857421500faef508a
change-id: 20241015-kunit-kselftests-56273bc40442

Best regards,
--  
Thomas Weißschuh <thomas.weissschuh@linutronix.de>