[PATCH 0/4] selftests/nolibc: integrate with kselftests

[PATCH 0/4] selftests/nolibc: integrate with kselftests

[Thomas Weißschuh]

Hook up nolibc-test with the kselftests framework.
This enables CI systems and developers to easily execute the tests.

Signed-off-by: Thomas Weißschuh <linux@weissschuh.net>
---
Thomas Weißschuh (4):
      selftests/nolibc: drop implicit defconfig executions
      selftests/nolibc: split out CFLAGS logic
      selftests/nolibc: rename Makefile
      selftests/nolibc: integrate with kselftests

 tools/testing/selftests/Makefile                |   1 +
 tools/testing/selftests/nolibc/Makefile         | 346 +-----------------------
 tools/testing/selftests/nolibc/Makefile.include |  10 +
 tools/testing/selftests/nolibc/Makefile.nolibc  | 340 +++++++++++++++++++++++
 tools/testing/selftests/nolibc/run-tests.sh     |   2 +-
 5 files changed, 363 insertions(+), 336 deletions(-)
---
base-commit: d7161bd24e41eee5a3cca5bd8caaf1afdf9120c9
change-id: 20250616-nolibc-selftests-39a774708272

Best regards,
-- 
Thomas Weißschuh <linux@weissschuh.net>

[PATCH 1/4] selftests/nolibc: drop implicit defconfig executions

[Thomas Weißschuh]

Commit d7d271ec30dd ("selftests/nolibc: execute defconfig before other targets")
accidentally introduced implicit executions of the defconfig target.
These executions were unintentional and come from a misunderstanding of
ordering dependencies.

Drop the dependencies again.

Reported-by: Mark Brown <broonie@kernel.org>
Closes: https://lore.kernel.org/all/3d5128b9-b4b6-4a8e-94ce-ea5ff4ea655b@sirena.org.uk/
Signed-off-by: Thomas Weißschuh <linux@weissschuh.net>
---
 tools/testing/selftests/nolibc/Makefile | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/tools/testing/selftests/nolibc/Makefile b/tools/testing/selftests/nolibc/Makefile
index 147ce411b46ac84ea3ee0f91a55a7bb6c0712626..41b97dfd02bff3fb57f4d2b73b718f5c389d25e9 100644
--- a/tools/testing/selftests/nolibc/Makefile
+++ b/tools/testing/selftests/nolibc/Makefile
@@ -307,10 +307,10 @@ defconfig:
 		$(MAKE) -C $(srctree) ARCH=$(ARCH) CC=$(CC) CROSS_COMPILE=$(CROSS_COMPILE) olddefconfig < /dev/null; \
 	fi
 
-kernel: | defconfig
+kernel:
 	$(Q)$(MAKE) -C $(srctree) ARCH=$(ARCH) CC=$(CC) CROSS_COMPILE=$(CROSS_COMPILE) $(IMAGE_NAME) < /dev/null
 
-kernel-standalone: initramfs | defconfig
+kernel-standalone: initramfs
 	$(Q)$(MAKE) -C $(srctree) ARCH=$(ARCH) CC=$(CC) CROSS_COMPILE=$(CROSS_COMPILE) $(IMAGE_NAME) CONFIG_INITRAMFS_SOURCE=$(CURDIR)/initramfs < /dev/null
 
 # run the tests after building the kernel

-- 
2.50.0

[PATCH 2/4] selftests/nolibc: split out CFLAGS logic

[Thomas Weißschuh]

Some upcoming changes will reuse the CFLAGS.

Split the computation into a reusable Makefile.

Signed-off-by: Thomas Weißschuh <linux@weissschuh.net>
---
 tools/testing/selftests/nolibc/Makefile         | 12 ++++--------
 tools/testing/selftests/nolibc/Makefile.include | 10 ++++++++++
 2 files changed, 14 insertions(+), 8 deletions(-)

diff --git a/tools/testing/selftests/nolibc/Makefile b/tools/testing/selftests/nolibc/Makefile
index 41b97dfd02bff3fb57f4d2b73b718f5c389d25e9..6d62f350d0c16405785a8aabc7f5741b82e55370 100644
--- a/tools/testing/selftests/nolibc/Makefile
+++ b/tools/testing/selftests/nolibc/Makefile
@@ -195,14 +195,10 @@ CFLAGS_sparc32 = $(call cc-option,-m32)
 ifeq ($(origin XARCH),command line)
 CFLAGS_XARCH = $(CFLAGS_$(XARCH))
 endif
-_CFLAGS_STACKPROTECTOR = $(call cc-option,-fstack-protector-all) $(call cc-option,-mstack-protector-guard=global)
-CFLAGS_STACKPROTECTOR ?= $(call try-run, \
-	echo 'void foo(void) {}' | $(CC) -x c - -o - -S $(CLANG_CROSS_FLAGS) $(_CFLAGS_STACKPROTECTOR) | grep -q __stack_chk_guard, \
-	$(_CFLAGS_STACKPROTECTOR))
-CFLAGS_SANITIZER ?= $(call cc-option,-fsanitize=undefined -fsanitize-trap=all)
-CFLAGS  ?= -Os -fno-ident -fno-asynchronous-unwind-tables -std=c89 -W -Wall -Wextra \
-		$(call cc-option,-fno-stack-protector) $(call cc-option,-Wmissing-prototypes) \
-		$(CFLAGS_XARCH) $(CFLAGS_STACKPROTECTOR) $(CFLAGS_SANITIZER) $(CFLAGS_EXTRA)
+
+include Makefile.include
+
+CFLAGS  ?= $(CFLAGS_NOLIBC_TEST) $(CFLAGS_XARCH) $(CFLAGS_EXTRA)
 LDFLAGS :=
 
 LIBGCC := -lgcc
diff --git a/tools/testing/selftests/nolibc/Makefile.include b/tools/testing/selftests/nolibc/Makefile.include
new file mode 100644
index 0000000000000000000000000000000000000000..66287fafbbe07e1750e31c3b2388ac4be1e7f8ae
--- /dev/null
+++ b/tools/testing/selftests/nolibc/Makefile.include
@@ -0,0 +1,10 @@
+# SPDX-License-Identifier: GPL-2.0
+
+__CFLAGS_STACKPROTECTOR = $(call cc-option,-fstack-protector-all) $(call cc-option,-mstack-protector-guard=global)
+_CFLAGS_STACKPROTECTOR ?= $(call try-run, \
+	echo 'void foo(void) {}' | $(CC) -x c - -o - -S $(CLANG_CROSS_FLAGS) $(__CFLAGS_STACKPROTECTOR) | grep -q __stack_chk_guard, \
+	$(__CFLAGS_STACKPROTECTOR))
+_CFLAGS_SANITIZER ?= $(call cc-option,-fsanitize=undefined -fsanitize-trap=all)
+CFLAGS_NOLIBC_TEST  ?= -Os -fno-ident -fno-asynchronous-unwind-tables -std=c89 -W -Wall -Wextra \
+		$(call cc-option,-fno-stack-protector) $(call cc-option,-Wmissing-prototypes) \
+		$(_CFLAGS_STACKPROTECTOR) $(_CFLAGS_SANITIZER)

-- 
2.50.0

[PATCH 3/4] selftests/nolibc: rename Makefile

[Thomas Weißschuh]

The nolibc tests are not real kselftests, they work differently and
provide a different interface. Users trying to use them like real
selftests may be confused and the tests are not executed by CI systems.

To make space for an integration with the kselftest framework, move the
custom tests out of the way.
The custom tests are still useful to keep as they provide functionality
not provided by kselftests.

Signed-off-by: Thomas Weißschuh <linux@weissschuh.net>
---
 tools/testing/selftests/nolibc/{Makefile => Makefile.nolibc} | 0
 tools/testing/selftests/nolibc/run-tests.sh                  | 2 +-
 2 files changed, 1 insertion(+), 1 deletion(-)

diff --git a/tools/testing/selftests/nolibc/Makefile b/tools/testing/selftests/nolibc/Makefile.nolibc
similarity index 100%
rename from tools/testing/selftests/nolibc/Makefile
rename to tools/testing/selftests/nolibc/Makefile.nolibc
diff --git a/tools/testing/selftests/nolibc/run-tests.sh b/tools/testing/selftests/nolibc/run-tests.sh
index 8277599e6441a933d9c1ec5003acf49b06df226f..279fbd93ef70497868689b7f1e14ddc6c5c1a15f 100755
--- a/tools/testing/selftests/nolibc/run-tests.sh
+++ b/tools/testing/selftests/nolibc/run-tests.sh
@@ -169,7 +169,7 @@ test_arch() {
 	if [ "$werror" -ne 0 ]; then
 		CFLAGS_EXTRA="$CFLAGS_EXTRA -Werror"
 	fi
-	MAKE=(make -j"${nproc}" XARCH="${arch}" CROSS_COMPILE="${cross_compile}" LLVM="${llvm}" O="${build_dir}")
+	MAKE=(make -f Makefile.nolibc -j"${nproc}" XARCH="${arch}" CROSS_COMPILE="${cross_compile}" LLVM="${llvm}" O="${build_dir}")
 
 	case "$test_mode" in
 		'system')

-- 
2.50.0

[PATCH 4/4] selftests/nolibc: integrate with kselftests

[Thomas Weißschuh]

Hook up nolibc-test with the kselftests framework.
This enables CI systems and developers to easily execute the tests.

While nolibc-test does not emit KTAP output itself that is not a problem,
as the kselftest executor will wrap the output in KTAP.

Signed-off-by: Thomas Weißschuh <linux@weissschuh.net>
---
 tools/testing/selftests/Makefile        |  1 +
 tools/testing/selftests/nolibc/Makefile | 20 ++++++++++++++++++++
 2 files changed, 21 insertions(+)

diff --git a/tools/testing/selftests/Makefile b/tools/testing/selftests/Makefile
index 339b31e6a6b592217eca1c03068f21728ea024b4..3a4c98102f6967c4a1586104a560418292b3f31d 100644
--- a/tools/testing/selftests/Makefile
+++ b/tools/testing/selftests/Makefile
@@ -77,6 +77,7 @@ TARGETS += net/ovpn
 TARGETS += net/packetdrill
 TARGETS += net/rds
 TARGETS += net/tcp_ao
+TARGETS += nolibc
 TARGETS += nsfs
 TARGETS += pci_endpoint
 TARGETS += pcie_bwctrl
diff --git a/tools/testing/selftests/nolibc/Makefile b/tools/testing/selftests/nolibc/Makefile
new file mode 100644
index 0000000000000000000000000000000000000000..fc7a65ca5fdd77576be49d6227939ec5451a814a
--- /dev/null
+++ b/tools/testing/selftests/nolibc/Makefile
@@ -0,0 +1,20 @@
+# SPDX-License-Identifier: GPL-2.0
+
+TEST_GEN_PROGS := nolibc-test
+
+include ../lib.mk
+include $(top_srcdir)/scripts/Makefile.compiler
+
+cc-option = $(call __cc-option, $(CC),,$(1),$(2))
+
+include Makefile.include
+
+CFLAGS = -nostdlib -nostdinc -static \
+	 -isystem $(top_srcdir)/tools/include/nolibc -isystem $(top_srcdir)/usr/include \
+	 $(CFLAGS_NOLIBC_TEST)
+
+ifeq ($(LLVM),)
+LDLIBS := -lgcc
+endif
+
+$(OUTPUT)/nolibc-test: nolibc-test.c nolibc-test-linkage.c | headers

-- 
2.50.0

Re: [PATCH 3/4] selftests/nolibc: rename Makefile

[Willy Tarreau]

Hi Thomas,

On Fri, Jun 20, 2025 at 11:39:32PM +0200, Thomas Weißschuh wrote:
> The nolibc tests are not real kselftests, they work differently and
> provide a different interface. Users trying to use them like real
> selftests may be confused and the tests are not executed by CI systems.
> 
> To make space for an integration with the kselftest framework, move the
> custom tests out of the way.
> The custom tests are still useful to keep as they provide functionality
> not provided by kselftests.

I'm wondering, what prevents us from merging the new rules into the
current makefile instead of renaming it, especially considering the
fact that we initially took care of not confiscating the "all" target ?
I'm asking because: 

  $ make -f Makefile.nolibc help

is clearly less convenient and intuitive than:

  $ make help

Regards,
Willy

Re: [PATCH 3/4] selftests/nolibc: rename Makefile

[Thomas Weißschuh]

On 2025-06-21 06:14:21+0200, Willy Tarreau wrote:
> Hi Thomas,
> 
> On Fri, Jun 20, 2025 at 11:39:32PM +0200, Thomas Weißschuh wrote:
> > The nolibc tests are not real kselftests, they work differently and
> > provide a different interface. Users trying to use them like real
> > selftests may be confused and the tests are not executed by CI systems.
> > 
> > To make space for an integration with the kselftest framework, move the
> > custom tests out of the way.
> > The custom tests are still useful to keep as they provide functionality
> > not provided by kselftests.
> 
> I'm wondering, what prevents us from merging the new rules into the
> current makefile instead of renaming it, especially considering the
> fact that we initially took care of not confiscating the "all" target ?

We'll have conflicts around CFLAGS, the nolibc-test target and probably
other things. It will also make everything harder to understand and may
break unexpectedly in the future.

> I'm asking because: 
> 
>   $ make -f Makefile.nolibc help
> 
> is clearly less convenient and intuitive than:
> 
>   $ make help

Is your issue specifically with the help target?
We should be able to show the help message from the main Makefile with a
hint to the Makefile.nolibc.

Another, more general, possibility would be to move the special Makefile
to tools/testing/nolibc/ and keep only the selftest parts in
tools/testing/selftests/nolibc/.

Re: [PATCH 3/4] selftests/nolibc: rename Makefile

[Willy Tarreau]

On Sat, Jun 21, 2025 at 10:34:38AM +0200, Thomas Weißschuh wrote:
> On 2025-06-21 06:14:21+0200, Willy Tarreau wrote:
> > Hi Thomas,
> > 
> > On Fri, Jun 20, 2025 at 11:39:32PM +0200, Thomas Weißschuh wrote:
> > > The nolibc tests are not real kselftests, they work differently and
> > > provide a different interface. Users trying to use them like real
> > > selftests may be confused and the tests are not executed by CI systems.
> > > 
> > > To make space for an integration with the kselftest framework, move the
> > > custom tests out of the way.
> > > The custom tests are still useful to keep as they provide functionality
> > > not provided by kselftests.
> > 
> > I'm wondering, what prevents us from merging the new rules into the
> > current makefile instead of renaming it, especially considering the
> > fact that we initially took care of not confiscating the "all" target ?
> 
> We'll have conflicts around CFLAGS, the nolibc-test target and probably
> other things.

OK I understand.

> It will also make everything harder to understand and may
> break unexpectedly in the future.
> 
> > I'm asking because: 
> > 
> >   $ make -f Makefile.nolibc help
> > 
> > is clearly less convenient and intuitive than:
> > 
> >   $ make help
> 
> Is your issue specifically with the help target?

Not just but that's an entry point. Admittedly it's not a big problem,
I was merely asking if there was a real reason for splitting them apart
or if it was just to keep the stuff clean.

> We should be able to show the help message from the main Makefile with a
> hint to the Makefile.nolibc.

I thought about it as well, we could have a help target in the main
makefile that just emits "Please run make -f Makefile.nolibc with the
following targets:", and then runs "make -f Makefile.nolibc help".

> Another, more general, possibility would be to move the special Makefile
> to tools/testing/nolibc/ and keep only the selftest parts in
> tools/testing/selftests/nolibc/.

I hadn't thought about this, but that could indeed make sense. Let's see
later how it goes and let's not add burden about this for now. Please just
keep your patch as-is.

Thanks,
Willy

Re: [PATCH 3/4] selftests/nolibc: rename Makefile

[Thomas Weißschuh]

On 2025-06-21 10:47:39+0200, Willy Tarreau wrote:
> On Sat, Jun 21, 2025 at 10:34:38AM +0200, Thomas Weißschuh wrote:
> > On 2025-06-21 06:14:21+0200, Willy Tarreau wrote:
> > > Hi Thomas,
> > > 
> > > On Fri, Jun 20, 2025 at 11:39:32PM +0200, Thomas Weißschuh wrote:
> > > > The nolibc tests are not real kselftests, they work differently and
> > > > provide a different interface. Users trying to use them like real
> > > > selftests may be confused and the tests are not executed by CI systems.
> > > > 
> > > > To make space for an integration with the kselftest framework, move the
> > > > custom tests out of the way.
> > > > The custom tests are still useful to keep as they provide functionality
> > > > not provided by kselftests.
> > > 
> > > I'm wondering, what prevents us from merging the new rules into the
> > > current makefile instead of renaming it, especially considering the
> > > fact that we initially took care of not confiscating the "all" target ?
> > 
> > We'll have conflicts around CFLAGS, the nolibc-test target and probably
> > other things.
> 
> OK I understand.
> 
> > It will also make everything harder to understand and may
> > break unexpectedly in the future.
> > 
> > > I'm asking because: 
> > > 
> > >   $ make -f Makefile.nolibc help
> > > 
> > > is clearly less convenient and intuitive than:
> > > 
> > >   $ make help
> > 
> > Is your issue specifically with the help target?
> 
> Not just but that's an entry point. Admittedly it's not a big problem,
> I was merely asking if there was a real reason for splitting them apart
> or if it was just to keep the stuff clean.
> 
> > We should be able to show the help message from the main Makefile with a
> > hint to the Makefile.nolibc.
> 
> I thought about it as well, we could have a help target in the main
> makefile that just emits "Please run make -f Makefile.nolibc with the
> following targets:", and then runs "make -f Makefile.nolibc help".

I'll do that.

> > Another, more general, possibility would be to move the special Makefile
> > to tools/testing/nolibc/ and keep only the selftest parts in
> > tools/testing/selftests/nolibc/.
> 
> I hadn't thought about this, but that could indeed make sense. Let's see
> later how it goes and let's not add burden about this for now. Please just
> keep your patch as-is.

Sounds good. Could you give a formal Ack for the patch/series?


Thomas

Re: [PATCH 3/4] selftests/nolibc: rename Makefile

[Willy Tarreau]

Hi Thomas,

On Mon, Jun 23, 2025 at 10:18:19PM +0200, Thomas Weißschuh wrote:
(...)
> > I hadn't thought about this, but that could indeed make sense. Let's see
> > later how it goes and let's not add burden about this for now. Please just
> > keep your patch as-is.
> 
> Sounds good. Could you give a formal Ack for the patch/series?

Sure, as usual, for the relevant parts of the series:

Acked-by: Willy Tarreau <w@1wt.eu>

Cheers,
Willy