Re: [Buildroot] [PATCH 2/5] support/testing: add libc-test runtime test

[Thomas Petazzoni via buildroot <buildroot@buildroot.org>]

Hello,

On Thu, Aug 27, 2026 at 06:59:49PM +0000, matt@thewebers.ws wrote:
> From: "Matthew Weber" <matthew.l.weber3@boeing.com>
> 
> Add a Buildroot runtime test that boots a minimal system with
> libc-test installed and exercises upstream functional tests across
> categories: string, stdio, stdlib, pthread, and misc POSIX APIs.
> 
> Includes skip tables for tests with known musl limitations:
> - wordexp (shell expansion)
> - strtold (long double rounding)
> - tls_* (need dynamic linker)
> - dlopen (needs shared libraries on target)
> 
> Tested on aarch64 (QEMU cortex-a57).
> 
> Signed-off-by: Matthew Weber <matthew.l.weber3@boeing.com>
> Signed-off-by: Clayton Shotwell <clayton.shotwell@boeing.com>

Thanks, always good to have tests for tests :-)

> diff --git a/support/testing/tests/package/test_libc_test.py b/support/testing/tests/package/test_libc_test.py
> new file mode 100644
> index 0000000000..e023b10e46
> --- /dev/null
> +++ b/support/testing/tests/package/test_libc_test.py
> @@ -0,0 +1,169 @@
> +# SPDX-License-Identifier: GPL-2.0+
> +import os
> +
> +import infra.basetest
> +
> +# libc-test runtime test for Buildroot
> +#
> +# Runs functional tests under QEMU aarch64-virt with musl libc.
> +# Not all tests from the suite can be exercised in this environment.
> +#
> +# Skipped tests — known broken / musl conformance issues:
> +# +---------------------+------------------------------------------------+
> +# | Test                | Reason                                         |
> +# +---------------------+------------------------------------------------+
> +# | strptime            | Known musl conformance issue                   |
> +# | utime               | QEMU virtfs timestamp resolution               |
> +# | mntent              | Known musl expected failure (from upstream CI)  |
> +# | tls_align           | Needs dynamic linker setup (not in initrd)     |
> +# | tls_init            | Needs dynamic linker setup (not in initrd)     |
> +# | tls_local_exec      | Needs dynamic linker setup (not in initrd)     |
> +# | dlopen              | Needs shared libraries on target               |
> +# | fma (math)          | Known musl precision issue                     |
> +# | fmal (math)         | Known musl precision issue                     |
> +# | powf (math)         | Known musl precision issue                     |
> +# | strtold             | Known musl long double rounding issue          |
> +# | wordexp             | musl wordexp limitations with shell expansion  |
> +# +---------------------+------------------------------------------------+
> +#
> +# Skipped tests — require non-root (test runs as root in QEMU):
> +# +---------------------+------------------------------------------------+
> +# | Test                | Reason                                         |
> +# +---------------------+------------------------------------------------+
> +# | fcntl               | Permission/lock tests bypassed by root         |
> +# +---------------------+------------------------------------------------+
> +
> +class TestLibcTestAarch64(infra.basetest.BRTest):
> +    config = \
> +        """
> +        BR2_aarch64=y
> +        BR2_TOOLCHAIN_EXTERNAL=y
> +        BR2_TOOLCHAIN_EXTERNAL_BOOTLIN=y
> +        BR2_TOOLCHAIN_EXTERNAL_BOOTLIN_AARCH64_MUSL_STABLE=y
> +        BR2_TARGET_GENERIC_GETTY_PORT="ttyAMA0"
> +        BR2_LINUX_KERNEL=y
> +        BR2_LINUX_KERNEL_CUSTOM_VERSION=y
> +        BR2_LINUX_KERNEL_CUSTOM_VERSION_VALUE="7.1.4"
> +        BR2_LINUX_KERNEL_USE_CUSTOM_CONFIG=y
> +        BR2_LINUX_KERNEL_CUSTOM_CONFIG_FILE="board/qemu/aarch64-virt/linux.config"
> +        BR2_LINUX_KERNEL_NEEDS_HOST_OPENSSL=y
> +        BR2_PACKAGE_LIBC_TEST=y
> +        BR2_TARGET_ROOTFS_CPIO=y
> +        BR2_TARGET_ROOTFS_CPIO_GZIP=y
> +        # BR2_TARGET_ROOTFS_TAR is not set
> +        """

For this particular package, I'd be inclined to have test cases for
musl, uclibc-ng and glibc. Indeed, because libc-test is primarily
developed by musl folks, I can easily see it being broken with other
libcs. Since the package itself is not tied to musl with a "depends
on" of any sort, it means our users will see "libc-test" in menuconfig
and might be tempted to use it regardless of the selected libc.

> +
> +    def test_run(self):
> +        img = os.path.join(self.builddir, "images", "rootfs.cpio.gz")
> +        kern = os.path.join(self.builddir, "images", "Image")
> +        self.emulator.boot(arch="aarch64",
> +                           kernel=kern,
> +                           kernel_cmdline=["console=ttyAMA0"],
> +                           options=["-M", "virt",
> +                                    "-cpu", "cortex-a57",
> +                                    "-m", "256M",
> +                                    "-initrd", img])
> +        self.emulator.login()
> +
> +        test_dir = "/usr/lib/libc-test"
> +
> +        # Verify test executables are installed
> +        self.assertRunOk(f"test -x {test_dir}/src/common/runtest.exe")
> +
> +        # --- Upstream functional tests ---

Rather than listing the tests to perform, can't we instead revert the
logic and run all tests that are available, except the ones that are
in an exclusion list?

This way, the exclusion list in your Python code will replace the
table above, i.e

	excluded_tests = [
		# Known musl conformance issue
		"strptime",
		# QEMU virtfs timestamp resolution
		"utime",
	]

this way, when we bump libc-test, new tests are automatically executed
(and we notice if one of the new tests fails in CI and needs to be
excluded).

> +        all_tests = (upstream_string + upstream_stdio +
> +                     upstream_stdlib + upstream_pthread +
> +                     upstream_misc)
> +
> +        for test in all_tests:
> +            exe = f"{test_dir}/src/functional/{test}.exe"
> +            cmd = f"{test_dir}/src/common/runtest.exe {exe}"

Just:

		cmd = f"{test_dir}/src/common/runtest.exe {test_dir}/src/functional/{test}.exe"

maybe? (This is very minor)

> +            self.assertRunOk(cmd)
> +
> +        # Run upstream regression tests
> +        out, ret = self.emulator.run(
> +            f"find {test_dir}/src/regression -name '*.exe'"
> +            " ! -name '*-static*' | sort | head -20")

Hm? What is happening here? Why head -20? Minor: instead of doing the
find on the target, can't we do it on the host in $(TARGET_DIR) if
it's all about figuring out the list of files/tests? Maybe with some
Python os.walk() whatever thingy?

> +        if ret == 0 and out:
> +            for line in out:
> +                exe = line.strip()
> +                if exe and exe.endswith(".exe"):
> +                    cmd = f"{test_dir}/src/common/runtest.exe {exe}"
> +                    self.assertRunOk(cmd)
> -- 
> 2.34.1
> 

Thanks!

Thomas
-- 
Thomas Petazzoni, co-owner and CEO, Bootlin
Embedded Linux and Kernel engineering and training
https://bootlin.com
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot