Re: [PATCH v3 00/16] kunit: Introduce UAPI testing framework

[David Gow <davidgow@google.com>]

[-- Attachment #1: Type: text/plain, Size: 9697 bytes --]

On Wed, 11 Jun 2025 at 15:38, Thomas Weißschuh
<thomas.weissschuh@linutronix.de> wrote:
>
> Currently testing of userspace and in-kernel API use two different
> frameworks. kselftests for the userspace ones and Kunit for the
> in-kernel ones. Besides their different scopes, both have different
> strengths and limitations:
>
> Kunit:
> * Tests are normal kernel code.
> * They use the regular kernel toolchain.
> * They can be packaged and distributed as modules conveniently.
>
> Kselftests:
> * Tests are normal userspace code
> * They need a userspace toolchain.
>   A kernel cross toolchain is likely not enough.
> * A fair amout of userland is required to run the tests,
>   which means a full distro or handcrafted rootfs.
> * There is no way to conveniently package and run kselftests with a
>   given kernel image.
> * The kselftests makefiles are not as powerful as regular kbuild.
>   For example they are missing proper header dependency tracking or more
>   complex compiler option modifications.
>
> Therefore kunit is much easier to run against different kernel
> configurations and architectures.
> This series aims to combine kselftests and kunit, avoiding both their
> limitations. It works by compiling the userspace kselftests as part of
> the regular kernel build, embedding them into the kunit kernel or module
> and executing them from there. If the kernel toolchain is not fit to
> produce userspace because of a missing libc, the kernel's own nolibc can
> be used instead.
> The structured TAP output from the kselftest is integrated into the
> kunit KTAP output transparently, the kunit parser can parse the combined
> logs together.
>
> Further room for improvements:
> * Call each test in its completely dedicated namespace
> * Handle additional test files besides the test executable through
>   archives. CPIO, cramfs, etc.
> * Compatibility with kselftest_harness.h (in progress)
> * Expose the blobs in debugfs
> * Provide some convience wrappers around compat userprogs
> * Figure out a migration path/coexistence solution for
>   kunit UAPI and tools/testing/selftests/
>
> Output from the kunit example testcase, note the output of
> "example_uapi_tests".
>
> $ ./tools/testing/kunit/kunit.py run --kunitconfig lib/kunit example
> ...
> Running tests with:
> $ .kunit/linux kunit.filter_glob=example kunit.enable=1 mem=1G console=tty kunit_shutdown=halt
> [11:53:53] ================== example (10 subtests) ===================
> [11:53:53] [PASSED] example_simple_test
> [11:53:53] [SKIPPED] example_skip_test
> [11:53:53] [SKIPPED] example_mark_skipped_test
> [11:53:53] [PASSED] example_all_expect_macros_test
> [11:53:53] [PASSED] example_static_stub_test
> [11:53:53] [PASSED] example_static_stub_using_fn_ptr_test
> [11:53:53] [PASSED] example_priv_test
> [11:53:53] =================== example_params_test  ===================
> [11:53:53] [SKIPPED] example value 3
> [11:53:53] [PASSED] example value 2
> [11:53:53] [PASSED] example value 1
> [11:53:53] [SKIPPED] example value 0
> [11:53:53] =============== [PASSED] example_params_test ===============
> [11:53:53] [PASSED] example_slow_test
> [11:53:53] ======================= (4 subtests) =======================
> [11:53:53] [PASSED] procfs
> [11:53:53] [PASSED] userspace test 2
> [11:53:53] [SKIPPED] userspace test 3: some reason
> [11:53:53] [PASSED] userspace test 4
> [11:53:53] ================ [PASSED] example_uapi_test ================
> [11:53:53] ===================== [PASSED] example =====================
> [11:53:53] ============================================================
> [11:53:53] Testing complete. Ran 16 tests: passed: 11, skipped: 5
> [11:53:53] Elapsed time: 67.543s total, 1.823s configuring, 65.655s building, 0.058s running
>
> Based on v6.15-rc1.
>
> Signed-off-by: Thomas Weißschuh <thomas.weissschuh@linutronix.de>
> ---
> Changes in v3:
> - Reintroduce CONFIG_CC_CAN_LINK_STATIC
> - Enable CONFIG_ARCH_HAS_NOLIBC for m68k and SPARC
> - Properly handle 'clean' target for userprogs
> - Use ramfs over tmpfs to reduce dependencies
> - Inherit userprogs byte order and ABI from kernel
> - Drop now unnecessary "#ifndef NOLIBC"
> - Pick up review tags
> - Drop usage of __private in blob.h,
>   sparse complains and it is not really necessary
> - Fix execution on loongarch when using clang
> - Drop userprogs libgcc handling, it was ugly and is not yet necessary
> - Link to v2: https://lore.kernel.org/r/20250407-kunit-kselftests-v2-0-454114e287fd@linutronix.de
>
> Changes in v2:
> - Rebase onto v6.15-rc1
> - Add documentation and kernel docs
> - Resolve invalid kconfig breakages
> - Drop already applied patch "kbuild: implement CONFIG_HEADERS_INSTALL for Usermode Linux"
> - Drop userprogs CONFIG_WERROR integration, it doesn't need to be part of this series
> - Replace patch prefix "kconfig" with "kbuild"
> - Rename kunit_uapi_run_executable() to kunit_uapi_run_kselftest()
> - Generate private, conflict-free symbols in the blob framework
> - Handle kselftest exit codes
> - Handle SIGABRT
> - Forward output also to kunit debugfs log
> - Install a fd=0 stdin filedescriptor
> - Link to v1: https://lore.kernel.org/r/20250217-kunit-kselftests-v1-0-42b4524c3b0a@linutronix.de
>
> ---


Thanks very much for persisting with this, and sorry for the delayed response.

I've taken quite a liking to it: it'd definitely have made my life
easier more than once.

As a more general wrapping of selftests in KUnit tests, I do think
that there's still some risk of confusion as to when a KUnit UAPI test
makes sense versus a simple selftest. The UAPI tests are definitely
(IMO) easier to build and run, but won't be easier to debug, or to run
on an existing, non-test system as a part of troubleshooting (which
has been a complaint when selftests have been ported to KUnit in the
past).

Nevertheless, I'm pretty happy to have this be a part of KUnit, though
I have three slight reservations:
1. There's no real _user_ of this yet -- save for the small test of
/proc/self/comm in the example -- I'd like to see a real-world test
using this.
2. There's a fair bit of complexity here, and we're already a bit
behind with KUnit reviews. I'd love it if you could commit to helping
maintain the KUnit parts of this in MAINTAINERS.
3. We need to make sure that there's a clear understanding of when to
use this, versus in-kernel KUnit tests, versus kselftest. This'll
probably involve (a) making sure Shuah is on board -- or at least not
strongly opposed, and (b) updating
Documentation/dev-tools/testing-overview.rst.

But thanks very much -- it's working well in my testing here, and
running the tests is very pleasant.

Cheers,
-- David


> Thomas Weißschuh (16):
>       kbuild: userprogs: avoid duplicating of flags inherited from kernel
>       kbuild: userprogs: also inherit byte order and ABI from kernel
>       init: re-add CONFIG_CC_CAN_LINK_STATIC
>       kbuild: userprogs: add nolibc support
>       kbuild: introduce CONFIG_ARCH_HAS_NOLIBC
>       kbuild: doc: add label for userprogs section
>       kbuild: introduce blob framework
>       kunit: tool: Add test for nested test result reporting
>       kunit: tool: Don't overwrite test status based on subtest counts
>       kunit: tool: Parse skipped tests from kselftest.h
>       kunit: Always descend into kunit directory during build
>       kunit: qemu_configs: loongarch: Enable LSX/LSAX
>       kunit: Introduce UAPI testing framework
>       kunit: uapi: Add example for UAPI tests
>       kunit: uapi: Introduce preinit executable
>       kunit: uapi: Validate usability of /proc
>
>  Documentation/dev-tools/kunit/api/index.rst        |   5 +
>  Documentation/dev-tools/kunit/api/uapi.rst         |  12 +
>  Documentation/kbuild/makefiles.rst                 |  38 ++-
>  MAINTAINERS                                        |   2 +
>  Makefile                                           |   7 +-
>  include/kunit/uapi.h                               |  24 ++
>  include/linux/blob.h                               |  31 +++
>  init/Kconfig                                       |   7 +
>  lib/Makefile                                       |   4 -
>  lib/kunit/Kconfig                                  |  10 +
>  lib/kunit/Makefile                                 |  20 +-
>  lib/kunit/kunit-example-test.c                     |  15 ++
>  lib/kunit/kunit-example-uapi.c                     |  54 ++++
>  lib/kunit/uapi-preinit.c                           |  63 +++++
>  lib/kunit/uapi.c                                   | 294 +++++++++++++++++++++
>  scripts/Makefile.blobs                             |  19 ++
>  scripts/Makefile.build                             |   6 +
>  scripts/Makefile.clean                             |   2 +-
>  scripts/Makefile.userprogs                         |  13 +-
>  scripts/blob-wrap.c                                |  27 ++
>  tools/include/nolibc/Kconfig.nolibc                |  15 ++
>  tools/testing/kunit/kunit_parser.py                |  13 +-
>  tools/testing/kunit/kunit_tool_test.py             |   9 +
>  tools/testing/kunit/qemu_configs/loongarch.py      |   2 +
>  .../test_is_test_passed-failure-nested.log         |  10 +
>  .../test_data/test_is_test_passed-kselftest.log    |   3 +-
>  26 files changed, 686 insertions(+), 19 deletions(-)
> ---
> base-commit: f07a3558c4a5d76f3fea004075e5151c4516d055
> change-id: 20241015-kunit-kselftests-56273bc40442
>
> Best regards,
> --
> Thomas Weißschuh <thomas.weissschuh@linutronix.de>
>

[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 5281 bytes --]