# Toplevel Makefile doesn't process module build correctly on recursive make calls

# Toplevel Makefile doesn't process module build correctly on recursive make calls

[Marcos Paulo de Souza]

[-- Attachment #1: Type: text/plain, Size: 2692 bytes --]

Hi all,

I found an issue while moving the livepatch kselftest modules to be built on the
fly, instead of building them on kernel building.

If, for some reason, there is a recursive make invocation that starts from the
top level Makefile and in the leaf Makefile it tries to build a module (using M=
in the make invocation), it doesn't produce the module. This happens because the
toplevel Makefile checks for M= only once. This is controlled by the
sub_make_done variable, which is exported after checking the command line
options are passed to the top level Makefile. Once this variable is set it's
the M= setting is never checked again on the recursive call.

This can be observed when cleaning the bpf kselftest dir. When calling

	$ make TARGETS="bpf" SKIP_TARGETS="" kselftest-clean

What happens:

	1. It checks for some command line settings (like M=) was passed (it wasn't),
	set some definitions and exports sub_make_done.

	2. Jump into tools/testing/selftests/bpf, and calls the clean target.

	3. The clean target is overwritten to remove some files and then jump to
	bpf_testmod dir and call clean there

	4. On bpf_testmod/Makefile, the clean target will execute
		$(Q)make -C $(KDIR) M=$(BPF_TESTMOD_DIR) clean

	5. The KDIR is to toplevel dir. The top Makefile will check that sub_make_done was
	already set, ignoring the M= setting.

	6. As M= wasn't checked, KBUILD_EXTMOD isn't set, and the clean target applies
	to the kernel as a whole, making it clean all generated code/objects and
	everything.

One way to avoid it is to call "unexport sub_make_done" on
tools/testing/selftests/bpf/bpf_testmod/Makefile before processing the all
target, forcing the toplevel Makefile to process the M=, producing the module
file correctly.

If the M=dir points to /lib/modules/.../build, then it fails with "m2c: No such
file", which I already reported here[1]. At the time this problem was treated
like a problem with kselftest infrastructure.

Important: The process works fine if the initial make invocation is targeted to a
different directory (using -C), since it doesn't goes through the toplevel
Makefile, and sub_make_done variable is not set.

I attached a minimal reproducer, that can be used to better understand the
problem. The "make testmod" and "make testmod-clean" have the same effect that
can be seem with the bpf kselftests. There is a unexport call commented on
test-mods/Makefile, and once that is called the process works as expected.

Is there a better way to fix this? Is this really a problem, or am I missing
something?

Thanks in advance,
  Marcos

[1]:
https://lore.kernel.org/linux-kbuild/rgbi7vgqel5f2vtzeuxykeh7civd6xxbxkohsgwgsond3vwbbv@hebfs24hxzir/

[-- Attachment #2: 0001-TESTING-PoC-of-M-not-being-processed-on-module-build.patch --]
[-- Type: text/x-patch, Size: 2626 bytes --]

From fb99fa8553e7363d2f1f285bf6f2bc05b5c00196 Mon Sep 17 00:00:00 2001
From: Marcos Paulo de Souza <mpdesouza@suse.com>
Date: Mon, 9 Oct 2023 17:19:03 -0300
Subject: [PATCH] TESTING: PoC of M= not being processed on module building

After applying this patch, one can execute
  $ make testmod

And test-mod.ko won't be created. This happens because the toplevel
Makefile first checks if the initial invocation of make contains M=, and
then sets sub_make_done in order to avoid recursive make invocations.

If the unexport is uncommented from the test-mod/Makefile, the the
mod_test.ko is created, since the toplevel Makefile will run the make
for the module.

The same problem can be seen when calling the clean target for the
module. With the "unexport sub_make_done" commented it will execute the
clean on the entire git tree, instead of only cleaning up the modules
directory. The testing can be done by calling
  make testmod-clean

With and without the "unexport" call commented. The will produce different
outcomes.

Signed-off-by: Marcos Paulo de Souza <mpdesouza@suse.com>
---
 Makefile            |  6 ++++++
 test-mod/Makefile   | 12 ++++++++++++
 test-mod/mod_test.c | 15 +++++++++++++++
 3 files changed, 33 insertions(+)
 create mode 100644 test-mod/Makefile
 create mode 100644 test-mod/mod_test.c

diff --git a/Makefile b/Makefile
index 57698d048e2c..e487497feb04 100644
--- a/Makefile
+++ b/Makefile
@@ -1371,6 +1371,12 @@ kselftest-merge:
 		xargs $(srctree)/scripts/kconfig/merge_config.sh -m $(objtree)/.config
 	$(Q)$(MAKE) -f $(srctree)/Makefile olddefconfig
 
+PHONY += testmod testmod-clean
+testmod:
+	$(Q)$(MAKE) -C $(srctree)/test-mod
+testmod-clean:
+	$(Q)$(MAKE) -C $(srctree)/test-mod clean
+
 # ---------------------------------------------------------------------------
 # Devicetree files
 
diff --git a/test-mod/Makefile b/test-mod/Makefile
new file mode 100644
index 000000000000..d07460088773
--- /dev/null
+++ b/test-mod/Makefile
@@ -0,0 +1,12 @@
+TESTMODS_DIR := $(realpath $(dir $(abspath $(lastword $(MAKEFILE_LIST)))))
+KDIR ?= $(abs_srctree)
+
+obj-m += mod_test.o
+
+#unexport sub_make_done
+
+all:
+	make -C $(KDIR) M=$(TESTMODS_DIR) modules
+
+clean:
+	make -C $(KDIR) M=$(TESTMODS_DIR) clean
diff --git a/test-mod/mod_test.c b/test-mod/mod_test.c
new file mode 100644
index 000000000000..135f5d0b55b0
--- /dev/null
+++ b/test-mod/mod_test.c
@@ -0,0 +1,15 @@
+#include <linux/module.h>
+#include <linux/kernel.h>
+
+static int mod_init(void)
+{
+	return 0;
+}
+
+static void mod_exit(void)
+{
+}
+
+module_init(mod_init);
+module_exit(mod_exit);
+MODULE_LICENSE("GPL");
-- 
2.42.0

Re: # Toplevel Makefile doesn't process module build correctly on recursive make calls

[Masahiro Yamada]

On Tue, Oct 10, 2023 at 5:43 AM Marcos Paulo de Souza <mpdesouza@suse.de> wrote:
>
> Hi all,
>
> I found an issue while moving the livepatch kselftest modules to be built on the
> fly, instead of building them on kernel building.
>
> If, for some reason, there is a recursive make invocation that starts from the
> top level Makefile and in the leaf Makefile it tries to build a module (using M=
> in the make invocation), it doesn't produce the module. This happens because the
> toplevel Makefile checks for M= only once. This is controlled by the
> sub_make_done variable, which is exported after checking the command line
> options are passed to the top level Makefile. Once this variable is set it's
> the M= setting is never checked again on the recursive call.
>
> This can be observed when cleaning the bpf kselftest dir. When calling
>
>         $ make TARGETS="bpf" SKIP_TARGETS="" kselftest-clean
>
> What happens:
>
>         1. It checks for some command line settings (like M=) was passed (it wasn't),
>         set some definitions and exports sub_make_done.
>
>         2. Jump into tools/testing/selftests/bpf, and calls the clean target.
>
>         3. The clean target is overwritten to remove some files and then jump to
>         bpf_testmod dir and call clean there
>
>         4. On bpf_testmod/Makefile, the clean target will execute
>                 $(Q)make -C $(KDIR) M=$(BPF_TESTMOD_DIR) clean
>
>         5. The KDIR is to toplevel dir. The top Makefile will check that sub_make_done was
>         already set, ignoring the M= setting.
>
>         6. As M= wasn't checked, KBUILD_EXTMOD isn't set, and the clean target applies
>         to the kernel as a whole, making it clean all generated code/objects and
>         everything.
>
> One way to avoid it is to call "unexport sub_make_done" on
> tools/testing/selftests/bpf/bpf_testmod/Makefile before processing the all
> target, forcing the toplevel Makefile to process the M=, producing the module
> file correctly.
>
> If the M=dir points to /lib/modules/.../build, then it fails with "m2c: No such
> file", which I already reported here[1]. At the time this problem was treated
> like a problem with kselftest infrastructure.
>
> Important: The process works fine if the initial make invocation is targeted to a
> different directory (using -C), since it doesn't goes through the toplevel
> Makefile, and sub_make_done variable is not set.
>
> I attached a minimal reproducer, that can be used to better understand the
> problem. The "make testmod" and "make testmod-clean" have the same effect that
> can be seem with the bpf kselftests. There is a unexport call commented on
> test-mods/Makefile, and once that is called the process works as expected.
>
> Is there a better way to fix this? Is this really a problem, or am I missing
> something?


Or, using KBUILD_EXTMOD will work too.





--
Best Regards
Masahiro Yamada

Re: # Toplevel Makefile doesn't process module build correctly on recursive make calls

[Marcos Paulo de Souza]

[-- Attachment #1: Type: text/plain, Size: 5589 bytes --]

On Sat, Oct 14, 2023 at 05:35:55PM +0900, Masahiro Yamada wrote:
> On Tue, Oct 10, 2023 at 5:43 AM Marcos Paulo de Souza <mpdesouza@suse.de> wrote:
> >
> > Hi all,
> >
> > I found an issue while moving the livepatch kselftest modules to be built on the
> > fly, instead of building them on kernel building.
> >
> > If, for some reason, there is a recursive make invocation that starts from the
> > top level Makefile and in the leaf Makefile it tries to build a module (using M=
> > in the make invocation), it doesn't produce the module. This happens because the
> > toplevel Makefile checks for M= only once. This is controlled by the
> > sub_make_done variable, which is exported after checking the command line
> > options are passed to the top level Makefile. Once this variable is set it's
> > the M= setting is never checked again on the recursive call.
> >
> > This can be observed when cleaning the bpf kselftest dir. When calling
> >
> >         $ make TARGETS="bpf" SKIP_TARGETS="" kselftest-clean
> >
> > What happens:
> >
> >         1. It checks for some command line settings (like M=) was passed (it wasn't),
> >         set some definitions and exports sub_make_done.
> >
> >         2. Jump into tools/testing/selftests/bpf, and calls the clean target.
> >
> >         3. The clean target is overwritten to remove some files and then jump to
> >         bpf_testmod dir and call clean there
> >
> >         4. On bpf_testmod/Makefile, the clean target will execute
> >                 $(Q)make -C $(KDIR) M=$(BPF_TESTMOD_DIR) clean
> >
> >         5. The KDIR is to toplevel dir. The top Makefile will check that sub_make_done was
> >         already set, ignoring the M= setting.
> >
> >         6. As M= wasn't checked, KBUILD_EXTMOD isn't set, and the clean target applies
> >         to the kernel as a whole, making it clean all generated code/objects and
> >         everything.
> >
> > One way to avoid it is to call "unexport sub_make_done" on
> > tools/testing/selftests/bpf/bpf_testmod/Makefile before processing the all
> > target, forcing the toplevel Makefile to process the M=, producing the module
> > file correctly.
> >
> > If the M=dir points to /lib/modules/.../build, then it fails with "m2c: No such
> > file", which I already reported here[1]. At the time this problem was treated
> > like a problem with kselftest infrastructure.
> >
> > Important: The process works fine if the initial make invocation is targeted to a
> > different directory (using -C), since it doesn't goes through the toplevel
> > Makefile, and sub_make_done variable is not set.
> >
> > I attached a minimal reproducer, that can be used to better understand the
> > problem. The "make testmod" and "make testmod-clean" have the same effect that
> > can be seem with the bpf kselftests. There is a unexport call commented on
> > test-mods/Makefile, and once that is called the process works as expected.
> >
> > Is there a better way to fix this? Is this really a problem, or am I missing
> > something?
> 
> 
> Or, using KBUILD_EXTMOD will work too.

Yes, that works, only if set to /lib/modules:

$ make kselftest TARGETS=bpf SKIP_TARGETS=""
make[3]: Entering directory '/home/mpdesouza/git/linux/tools/testing/selftests/bpf'
  MOD      bpf_testmod.ko
warning: the compiler differs from the one used to build the kernel
  The kernel was built by: gcc (SUSE Linux) 13.2.1 20230803 [revision cc279d6c64562f05019e1d12d0d825f9391b5553]
  You are using:           gcc (SUSE Linux) 13.2.1 20230912 [revision b96e66fd4ef3e36983969fb8cdd1956f551a074b]
  CC [M]  /home/mpdesouza/git/linux/tools/testing/selftests/bpf/bpf_testmod/bpf_testmod.o
  MODPOST /home/mpdesouza/git/linux/tools/testing/selftests/bpf/bpf_testmod/Module.symvers
  CC [M]  /home/mpdesouza/git/linux/tools/testing/selftests/bpf/bpf_testmod/bpf_testmod.mod.o
  LD [M]  /home/mpdesouza/git/linux/tools/testing/selftests/bpf/bpf_testmod/bpf_testmod.ko
  BTF [M] /home/mpdesouza/git/linux/tools/testing/selftests/bpf/bpf_testmod/bpf_testmod.ko
Skipping BTF generation for /home/mpdesouza/git/linux/tools/testing/selftests/bpf/bpf_testmod/bpf_testmod.ko due to unavailability of vmlinux
  BINARY   xdp_synproxy
...

But if we set the KBUILD_EXTMOD to toplevel Makefile, it fails with a different
strange issue:

$ make kselftest TARGETS=bpf SKIP_TARGETS=""
  BINARY   urandom_read
  MOD      bpf_testmod.ko
m2c    -o scripts/Makefile.build -e scripts/Makefile.build scripts/Makefile.build.mod
make[6]: m2c: No such file or directory
make[6]: *** [<builtin>: scripts/Makefile.build] Error 127
make[5]: *** [Makefile:1913: /home/mpdesouza/git/linux/tools/testing/selftests/bpf/bpf_testmod] Error 2
make[4]: *** [Makefile:19: all] Error 2
make[3]: *** [Makefile:229: /home/mpdesouza/git/linux/tools/testing/selftests/bpf/bpf_testmod.ko] Error 2
make[3]: Leaving directory '/home/mpdesouza/git/linux/tools/testing/selftests/bpf'
make[2]: *** [Makefile:175: all] Error 2
make[1]: *** [/home/mpdesouza/git/linux/Makefile:1362: kselftest] Error 2

I attached a patch that can reproduce the case where it works, and the case
where it doesn't by changing the value of KDIR.

I understand that KBUILD_EXTMOD, as the name implies, was designed to build
"external" modules, and not ones that live inside kernel, but how could this be
solved? For the sake of my initial about livepatch kselftests, KBUILD_EXTMOD
will suffice, since we will target /lib/modules, but I would like to know what
we can do in this case. Do you have other suggestions?

Thanks in advance,
  Marcos

> 
> 
> 
> 
> 
> --
> Best Regards
> Masahiro Yamada

[-- Attachment #2: bpfmod.diff --]
[-- Type: text/x-patch, Size: 864 bytes --]

diff --git a/tools/testing/selftests/bpf/bpf_testmod/Makefile b/tools/testing/selftests/bpf/bpf_testmod/Makefile
index 15cb36c4483a..1dce76f35405 100644
--- a/tools/testing/selftests/bpf/bpf_testmod/Makefile
+++ b/tools/testing/selftests/bpf/bpf_testmod/Makefile
@@ -1,5 +1,6 @@
 BPF_TESTMOD_DIR := $(realpath $(dir $(abspath $(lastword $(MAKEFILE_LIST)))))
-KDIR ?= $(abspath $(BPF_TESTMOD_DIR)/../../../../..)
+#KDIR ?= $(abspath $(BPF_TESTMOD_DIR)/../../../../..)
+KDIR ?= /lib/modules/$(shell uname -r)/build
 
 ifeq ($(V),1)
 Q =
@@ -12,9 +13,10 @@ MODULES = bpf_testmod.ko
 obj-m += bpf_testmod.o
 CFLAGS_bpf_testmod.o = -I$(src)
 
+export KBUILD_EXTMOD := $(BPF_TESTMOD_DIR)
+
 all:
-	+$(Q)make -C $(KDIR) M=$(BPF_TESTMOD_DIR) modules
+	+$(Q)make -C $(KDIR) modules
 
 clean:
-	+$(Q)make -C $(KDIR) M=$(BPF_TESTMOD_DIR) clean
-
+	+$(Q)make -C $(KDIR) clean

Re: # Toplevel Makefile doesn't process module build correctly on recursive make calls

[Nicolas Schier]

On Thu, Oct 19, 2023 at 03:50:05PM -0300, Marcos Paulo de Souza wrote:
> On Sat, Oct 14, 2023 at 05:35:55PM +0900, Masahiro Yamada wrote:
> > On Tue, Oct 10, 2023 at 5:43 AM Marcos Paulo de Souza <mpdesouza@suse.de> wrote:
> > >
> > > Hi all,
> > >
> > > I found an issue while moving the livepatch kselftest modules to be built on the
> > > fly, instead of building them on kernel building.
> > >
> > > If, for some reason, there is a recursive make invocation that starts from the
> > > top level Makefile and in the leaf Makefile it tries to build a module (using M=
> > > in the make invocation), it doesn't produce the module. This happens because the
> > > toplevel Makefile checks for M= only once. This is controlled by the
> > > sub_make_done variable, which is exported after checking the command line
> > > options are passed to the top level Makefile. Once this variable is set it's
> > > the M= setting is never checked again on the recursive call.
> > >
> > > This can be observed when cleaning the bpf kselftest dir. When calling
> > >
> > >         $ make TARGETS="bpf" SKIP_TARGETS="" kselftest-clean
> > >
> > > What happens:
> > >
> > >         1. It checks for some command line settings (like M=) was passed (it wasn't),
> > >         set some definitions and exports sub_make_done.
> > >
> > >         2. Jump into tools/testing/selftests/bpf, and calls the clean target.
> > >
> > >         3. The clean target is overwritten to remove some files and then jump to
> > >         bpf_testmod dir and call clean there
> > >
> > >         4. On bpf_testmod/Makefile, the clean target will execute
> > >                 $(Q)make -C $(KDIR) M=$(BPF_TESTMOD_DIR) clean
> > >
> > >         5. The KDIR is to toplevel dir. The top Makefile will check that sub_make_done was
> > >         already set, ignoring the M= setting.
> > >
> > >         6. As M= wasn't checked, KBUILD_EXTMOD isn't set, and the clean target applies
> > >         to the kernel as a whole, making it clean all generated code/objects and
> > >         everything.
> > >
> > > One way to avoid it is to call "unexport sub_make_done" on
> > > tools/testing/selftests/bpf/bpf_testmod/Makefile before processing the all
> > > target, forcing the toplevel Makefile to process the M=, producing the module
> > > file correctly.
> > >
> > > If the M=dir points to /lib/modules/.../build, then it fails with "m2c: No such
> > > file", which I already reported here[1]. At the time this problem was treated
> > > like a problem with kselftest infrastructure.
> > >
> > > Important: The process works fine if the initial make invocation is targeted to a
> > > different directory (using -C), since it doesn't goes through the toplevel
> > > Makefile, and sub_make_done variable is not set.
> > >
> > > I attached a minimal reproducer, that can be used to better understand the
> > > problem. The "make testmod" and "make testmod-clean" have the same effect that
> > > can be seem with the bpf kselftests. There is a unexport call commented on
> > > test-mods/Makefile, and once that is called the process works as expected.
> > >
> > > Is there a better way to fix this? Is this really a problem, or am I missing
> > > something?
> > 
> > 
> > Or, using KBUILD_EXTMOD will work too.
> 
> Yes, that works, only if set to /lib/modules:
> 
> $ make kselftest TARGETS=bpf SKIP_TARGETS=""
> make[3]: Entering directory '/home/mpdesouza/git/linux/tools/testing/selftests/bpf'
>   MOD      bpf_testmod.ko
> warning: the compiler differs from the one used to build the kernel
>   The kernel was built by: gcc (SUSE Linux) 13.2.1 20230803 [revision cc279d6c64562f05019e1d12d0d825f9391b5553]
>   You are using:           gcc (SUSE Linux) 13.2.1 20230912 [revision b96e66fd4ef3e36983969fb8cdd1956f551a074b]
>   CC [M]  /home/mpdesouza/git/linux/tools/testing/selftests/bpf/bpf_testmod/bpf_testmod.o
>   MODPOST /home/mpdesouza/git/linux/tools/testing/selftests/bpf/bpf_testmod/Module.symvers
>   CC [M]  /home/mpdesouza/git/linux/tools/testing/selftests/bpf/bpf_testmod/bpf_testmod.mod.o
>   LD [M]  /home/mpdesouza/git/linux/tools/testing/selftests/bpf/bpf_testmod/bpf_testmod.ko
>   BTF [M] /home/mpdesouza/git/linux/tools/testing/selftests/bpf/bpf_testmod/bpf_testmod.ko
> Skipping BTF generation for /home/mpdesouza/git/linux/tools/testing/selftests/bpf/bpf_testmod/bpf_testmod.ko due to unavailability of vmlinux
>   BINARY   xdp_synproxy
> ...
> 
> But if we set the KBUILD_EXTMOD to toplevel Makefile, it fails with a different
> strange issue:
> 
> $ make kselftest TARGETS=bpf SKIP_TARGETS=""
>   BINARY   urandom_read
>   MOD      bpf_testmod.ko
> m2c    -o scripts/Makefile.build -e scripts/Makefile.build scripts/Makefile.build.mod
> make[6]: m2c: No such file or directory
> make[6]: *** [<builtin>: scripts/Makefile.build] Error 127
> make[5]: *** [Makefile:1913: /home/mpdesouza/git/linux/tools/testing/selftests/bpf/bpf_testmod] Error 2
> make[4]: *** [Makefile:19: all] Error 2
> make[3]: *** [Makefile:229: /home/mpdesouza/git/linux/tools/testing/selftests/bpf/bpf_testmod.ko] Error 2
> make[3]: Leaving directory '/home/mpdesouza/git/linux/tools/testing/selftests/bpf'
> make[2]: *** [Makefile:175: all] Error 2
> make[1]: *** [/home/mpdesouza/git/linux/Makefile:1362: kselftest] Error 2
> 
> I attached a patch that can reproduce the case where it works, and the case
> where it doesn't by changing the value of KDIR.
> 
> I understand that KBUILD_EXTMOD, as the name implies, was designed to build
> "external" modules, and not ones that live inside kernel, but how could this be
> solved?

It seems to me as if there is some confusion about in-tree vs.
out-of-tree kmods.

KBUILD_EXTMOD and M are almost the same and indicate that you want to
build _external_ (=out-of-tree) kernel modules.  In-tree modules are
only those that stay in-tree _and_ are built along with the kernel.
Thus, 'make modules KBUILD_EXTMOD=fs/ext4' could be used to build ext4
kmod as "out-of-tree" kernel module, that even taints the kernel if it
gets loaded.

If you want bpf_testmod.ko to be an in-tree kmod, it has to be build
during the usual kernel build, not by running 'make kselftest'.

If you use 'make -C $(KDIR)' for building out-of-tree kmods, KDIR has to
point to the kernel build directory.  (Or it may point to the source
tree if you give O=$(BUILDDIR) as well).

HTH.

Kind regards,
Nicolas


> For the sake of my initial about livepatch kselftests, KBUILD_EXTMOD
> will suffice, since we will target /lib/modules, but I would like to know what
> we can do in this case. Do you have other suggestions?
> 
> Thanks in advance,
>   Marcos
> 
> > 
> > 
> > 
> > 
> > 
> > --
> > Best Regards
> > Masahiro Yamada

> diff --git a/tools/testing/selftests/bpf/bpf_testmod/Makefile b/tools/testing/selftests/bpf/bpf_testmod/Makefile
> index 15cb36c4483a..1dce76f35405 100644
> --- a/tools/testing/selftests/bpf/bpf_testmod/Makefile
> +++ b/tools/testing/selftests/bpf/bpf_testmod/Makefile
> @@ -1,5 +1,6 @@
>  BPF_TESTMOD_DIR := $(realpath $(dir $(abspath $(lastword $(MAKEFILE_LIST)))))
> -KDIR ?= $(abspath $(BPF_TESTMOD_DIR)/../../../../..)
> +#KDIR ?= $(abspath $(BPF_TESTMOD_DIR)/../../../../..)
> +KDIR ?= /lib/modules/$(shell uname -r)/build
>  
>  ifeq ($(V),1)
>  Q =
> @@ -12,9 +13,10 @@ MODULES = bpf_testmod.ko
>  obj-m += bpf_testmod.o
>  CFLAGS_bpf_testmod.o = -I$(src)
>  
> +export KBUILD_EXTMOD := $(BPF_TESTMOD_DIR)
> +
>  all:
> -	+$(Q)make -C $(KDIR) M=$(BPF_TESTMOD_DIR) modules
> +	+$(Q)make -C $(KDIR) modules
>  
>  clean:
> -	+$(Q)make -C $(KDIR) M=$(BPF_TESTMOD_DIR) clean
> -
> +	+$(Q)make -C $(KDIR) clean

Re: # Toplevel Makefile doesn't process module build correctly on recursive make calls

[Marcos Paulo de Souza]

On Fri, Oct 20, 2023 at 06:21:40AM +0200, Nicolas Schier wrote:
> On Thu, Oct 19, 2023 at 03:50:05PM -0300, Marcos Paulo de Souza wrote:
> > On Sat, Oct 14, 2023 at 05:35:55PM +0900, Masahiro Yamada wrote:
> > > On Tue, Oct 10, 2023 at 5:43 AM Marcos Paulo de Souza <mpdesouza@suse.de> wrote:
> > > >
> > > > Hi all,
> > > >
> > > > I found an issue while moving the livepatch kselftest modules to be built on the
> > > > fly, instead of building them on kernel building.
> > > >
> > > > If, for some reason, there is a recursive make invocation that starts from the
> > > > top level Makefile and in the leaf Makefile it tries to build a module (using M=
> > > > in the make invocation), it doesn't produce the module. This happens because the
> > > > toplevel Makefile checks for M= only once. This is controlled by the
> > > > sub_make_done variable, which is exported after checking the command line
> > > > options are passed to the top level Makefile. Once this variable is set it's
> > > > the M= setting is never checked again on the recursive call.
> > > >
> > > > This can be observed when cleaning the bpf kselftest dir. When calling
> > > >
> > > >         $ make TARGETS="bpf" SKIP_TARGETS="" kselftest-clean
> > > >
> > > > What happens:
> > > >
> > > >         1. It checks for some command line settings (like M=) was passed (it wasn't),
> > > >         set some definitions and exports sub_make_done.
> > > >
> > > >         2. Jump into tools/testing/selftests/bpf, and calls the clean target.
> > > >
> > > >         3. The clean target is overwritten to remove some files and then jump to
> > > >         bpf_testmod dir and call clean there
> > > >
> > > >         4. On bpf_testmod/Makefile, the clean target will execute
> > > >                 $(Q)make -C $(KDIR) M=$(BPF_TESTMOD_DIR) clean
> > > >
> > > >         5. The KDIR is to toplevel dir. The top Makefile will check that sub_make_done was
> > > >         already set, ignoring the M= setting.
> > > >
> > > >         6. As M= wasn't checked, KBUILD_EXTMOD isn't set, and the clean target applies
> > > >         to the kernel as a whole, making it clean all generated code/objects and
> > > >         everything.
> > > >
> > > > One way to avoid it is to call "unexport sub_make_done" on
> > > > tools/testing/selftests/bpf/bpf_testmod/Makefile before processing the all
> > > > target, forcing the toplevel Makefile to process the M=, producing the module
> > > > file correctly.
> > > >
> > > > If the M=dir points to /lib/modules/.../build, then it fails with "m2c: No such
> > > > file", which I already reported here[1]. At the time this problem was treated
> > > > like a problem with kselftest infrastructure.
> > > >
> > > > Important: The process works fine if the initial make invocation is targeted to a
> > > > different directory (using -C), since it doesn't goes through the toplevel
> > > > Makefile, and sub_make_done variable is not set.
> > > >
> > > > I attached a minimal reproducer, that can be used to better understand the
> > > > problem. The "make testmod" and "make testmod-clean" have the same effect that
> > > > can be seem with the bpf kselftests. There is a unexport call commented on
> > > > test-mods/Makefile, and once that is called the process works as expected.
> > > >
> > > > Is there a better way to fix this? Is this really a problem, or am I missing
> > > > something?
> > > 
> > > 
> > > Or, using KBUILD_EXTMOD will work too.
> > 
> > Yes, that works, only if set to /lib/modules:
> > 
> > $ make kselftest TARGETS=bpf SKIP_TARGETS=""
> > make[3]: Entering directory '/home/mpdesouza/git/linux/tools/testing/selftests/bpf'
> >   MOD      bpf_testmod.ko
> > warning: the compiler differs from the one used to build the kernel
> >   The kernel was built by: gcc (SUSE Linux) 13.2.1 20230803 [revision cc279d6c64562f05019e1d12d0d825f9391b5553]
> >   You are using:           gcc (SUSE Linux) 13.2.1 20230912 [revision b96e66fd4ef3e36983969fb8cdd1956f551a074b]
> >   CC [M]  /home/mpdesouza/git/linux/tools/testing/selftests/bpf/bpf_testmod/bpf_testmod.o
> >   MODPOST /home/mpdesouza/git/linux/tools/testing/selftests/bpf/bpf_testmod/Module.symvers
> >   CC [M]  /home/mpdesouza/git/linux/tools/testing/selftests/bpf/bpf_testmod/bpf_testmod.mod.o
> >   LD [M]  /home/mpdesouza/git/linux/tools/testing/selftests/bpf/bpf_testmod/bpf_testmod.ko
> >   BTF [M] /home/mpdesouza/git/linux/tools/testing/selftests/bpf/bpf_testmod/bpf_testmod.ko
> > Skipping BTF generation for /home/mpdesouza/git/linux/tools/testing/selftests/bpf/bpf_testmod/bpf_testmod.ko due to unavailability of vmlinux
> >   BINARY   xdp_synproxy
> > ...
> > 
> > But if we set the KBUILD_EXTMOD to toplevel Makefile, it fails with a different
> > strange issue:
> > 
> > $ make kselftest TARGETS=bpf SKIP_TARGETS=""
> >   BINARY   urandom_read
> >   MOD      bpf_testmod.ko
> > m2c    -o scripts/Makefile.build -e scripts/Makefile.build scripts/Makefile.build.mod
> > make[6]: m2c: No such file or directory
> > make[6]: *** [<builtin>: scripts/Makefile.build] Error 127
> > make[5]: *** [Makefile:1913: /home/mpdesouza/git/linux/tools/testing/selftests/bpf/bpf_testmod] Error 2
> > make[4]: *** [Makefile:19: all] Error 2
> > make[3]: *** [Makefile:229: /home/mpdesouza/git/linux/tools/testing/selftests/bpf/bpf_testmod.ko] Error 2
> > make[3]: Leaving directory '/home/mpdesouza/git/linux/tools/testing/selftests/bpf'
> > make[2]: *** [Makefile:175: all] Error 2
> > make[1]: *** [/home/mpdesouza/git/linux/Makefile:1362: kselftest] Error 2
> > 
> > I attached a patch that can reproduce the case where it works, and the case
> > where it doesn't by changing the value of KDIR.
> > 
> > I understand that KBUILD_EXTMOD, as the name implies, was designed to build
> > "external" modules, and not ones that live inside kernel, but how could this be
> > solved?
> 
> It seems to me as if there is some confusion about in-tree vs.
> out-of-tree kmods.
> 
> KBUILD_EXTMOD and M are almost the same and indicate that you want to
> build _external_ (=out-of-tree) kernel modules.  In-tree modules are
> only those that stay in-tree _and_ are built along with the kernel.
> Thus, 'make modules KBUILD_EXTMOD=fs/ext4' could be used to build ext4
> kmod as "out-of-tree" kernel module, that even taints the kernel if it
> gets loaded.
> 
> If you want bpf_testmod.ko to be an in-tree kmod, it has to be build
> during the usual kernel build, not by running 'make kselftest'.
> 
> If you use 'make -C $(KDIR)' for building out-of-tree kmods, KDIR has to
> point to the kernel build directory.  (Or it may point to the source
> tree if you give O=$(BUILDDIR) as well).

Thanks for the explanation Nicolas. In this, I believe that the BPF module
should be moved into lib/, like lib/livepatch, when then be built along with
other in-tree modules.

Currently there is a bug when running the kselftests-clean target with bpf:

	make kselftest-clean TARGETS=bpf SKIP_TARGETS=""

As the M= argument is ignore on the toplevel Makefile, this make invocation
applies the clean to all built kernel objects/modules/everything, which is bug
IMO.

There is a statement in the BPF docs saying that the selftests should be run
inside the tools/testing/selftests/bpf directory. At the same time, kselftests
should comply with all the targets defined in the documention, like gen_tar, and
run_tests. In this case should the build process be fixed, or just make
kselftests less restrict?

(CCing kselftests and bpf ML)

> 
> HTH.
> 
> Kind regards,
> Nicolas
> 
> 
> > For the sake of my initial about livepatch kselftests, KBUILD_EXTMOD
> > will suffice, since we will target /lib/modules, but I would like to know what
> > we can do in this case. Do you have other suggestions?
> > 
> > Thanks in advance,
> >   Marcos
> > 
> > > 
> > > 
> > > 
> > > 
> > > 
> > > --
> > > Best Regards
> > > Masahiro Yamada
> 
> > diff --git a/tools/testing/selftests/bpf/bpf_testmod/Makefile b/tools/testing/selftests/bpf/bpf_testmod/Makefile
> > index 15cb36c4483a..1dce76f35405 100644
> > --- a/tools/testing/selftests/bpf/bpf_testmod/Makefile
> > +++ b/tools/testing/selftests/bpf/bpf_testmod/Makefile
> > @@ -1,5 +1,6 @@
> >  BPF_TESTMOD_DIR := $(realpath $(dir $(abspath $(lastword $(MAKEFILE_LIST)))))
> > -KDIR ?= $(abspath $(BPF_TESTMOD_DIR)/../../../../..)
> > +#KDIR ?= $(abspath $(BPF_TESTMOD_DIR)/../../../../..)
> > +KDIR ?= /lib/modules/$(shell uname -r)/build
> >  
> >  ifeq ($(V),1)
> >  Q =
> > @@ -12,9 +13,10 @@ MODULES = bpf_testmod.ko
> >  obj-m += bpf_testmod.o
> >  CFLAGS_bpf_testmod.o = -I$(src)
> >  
> > +export KBUILD_EXTMOD := $(BPF_TESTMOD_DIR)
> > +
> >  all:
> > -	+$(Q)make -C $(KDIR) M=$(BPF_TESTMOD_DIR) modules
> > +	+$(Q)make -C $(KDIR) modules
> >  
> >  clean:
> > -	+$(Q)make -C $(KDIR) M=$(BPF_TESTMOD_DIR) clean
> > -
> > +	+$(Q)make -C $(KDIR) clean
> 

Re: # Toplevel Makefile doesn't process module build correctly on recursive make calls

[Marcos Paulo de Souza]

On Sat, 14 Oct 2023 17:35:55 +0900 Masahiro Yamada <masahiroy@kernel.org> wrote:

> On Tue, Oct 10, 2023 at 5:43 AM Marcos Paulo de Souza <mpdesouza@suse.de> wrote:
> >
> > Hi all,
> >
> > I found an issue while moving the livepatch kselftest modules to be built on the
> > fly, instead of building them on kernel building.
> >
> > If, for some reason, there is a recursive make invocation that starts from the
> > top level Makefile and in the leaf Makefile it tries to build a module (using M=
> > in the make invocation), it doesn't produce the module. This happens because the
> > toplevel Makefile checks for M= only once. This is controlled by the
> > sub_make_done variable, which is exported after checking the command line
> > options are passed to the top level Makefile. Once this variable is set it's
> > the M= setting is never checked again on the recursive call.
> >
> > This can be observed when cleaning the bpf kselftest dir. When calling
> >
> >         $ make TARGETS="bpf" SKIP_TARGETS="" kselftest-clean
> >
> > What happens:
> >
> >         1. It checks for some command line settings (like M=) was passed (it wasn't),
> >         set some definitions and exports sub_make_done.
> >
> >         2. Jump into tools/testing/selftests/bpf, and calls the clean target.
> >
> >         3. The clean target is overwritten to remove some files and then jump to
> >         bpf_testmod dir and call clean there
> >
> >         4. On bpf_testmod/Makefile, the clean target will execute
> >                 $(Q)make -C $(KDIR) M=$(BPF_TESTMOD_DIR) clean
> >
> >         5. The KDIR is to toplevel dir. The top Makefile will check that sub_make_done was
> >         already set, ignoring the M= setting.
> >
> >         6. As M= wasn't checked, KBUILD_EXTMOD isn't set, and the clean target applies
> >         to the kernel as a whole, making it clean all generated code/objects and
> >         everything.
> >
> > One way to avoid it is to call "unexport sub_make_done" on
> > tools/testing/selftests/bpf/bpf_testmod/Makefile before processing the all
> > target, forcing the toplevel Makefile to process the M=, producing the module
> > file correctly.
> >
> > If the M=dir points to /lib/modules/.../build, then it fails with "m2c: No such
> > file", which I already reported here[1]. At the time this problem was treated
> > like a problem with kselftest infrastructure.
> >
> > Important: The process works fine if the initial make invocation is targeted to a
> > different directory (using -C), since it doesn't goes through the toplevel
> > Makefile, and sub_make_done variable is not set.
> >
> > I attached a minimal reproducer, that can be used to better understand the
> > problem. The "make testmod" and "make testmod-clean" have the same effect that
> > can be seem with the bpf kselftests. There is a unexport call commented on
> > test-mods/Makefile, and once that is called the process works as expected.
> >
> > Is there a better way to fix this? Is this really a problem, or am I missing
> > something?
> 
> 
> Or, using KBUILD_EXTMOD will work too.

So it worked until 13b25489b6f8bd73ed65f07928f7c27a481f1820 ("kbuild: change working
directory to external module directory with M="). Right now, it shows the same
error as reported before:

m2c    -o /home/mpdesouza/git/linux/scripts/Makefile.build -e /home/mpdesouza/git/linux/scripts/Makefile.build /home/mpdesouza/git/linux/scripts/Makefile.build.mod
make[6]: m2c: No such file or directory

To reproduce, it's just execute the livepatch ksefltests:

$ make kselftest TARGETS=livepatch KDIR=<kernel source top dir>

Is there something that we can do in this case to solve this issue?

I'm quite surprised that no bots found this problem yet.

> 
> 
> 
> 
> 
> --
> Best Regards
> Masahiro Yamada

/*
 * !REMOVE THIS COMMENT BLOCK BEFORE SENDING THIS MAIL!
 * below are signatures added by "hkml".
 * edit signatures below, or use "hkml signature".
 */

Sent using hkml (https://github.com/sjp38/hackermail)

Re: # Toplevel Makefile doesn't process module build correctly on recursive make calls

[Masahiro Yamada]

On Tue, Jan 14, 2025 at 4:38 AM Marcos Paulo de Souza
<mpdesouza@suse.com> wrote:
>
> On Sat, 14 Oct 2023 17:35:55 +0900 Masahiro Yamada <masahiroy@kernel.org> wrote:
>
> > On Tue, Oct 10, 2023 at 5:43 AM Marcos Paulo de Souza <mpdesouza@suse.de> wrote:
> > >
> > > Hi all,
> > >
> > > I found an issue while moving the livepatch kselftest modules to be built on the
> > > fly, instead of building them on kernel building.
> > >
> > > If, for some reason, there is a recursive make invocation that starts from the
> > > top level Makefile and in the leaf Makefile it tries to build a module (using M=
> > > in the make invocation), it doesn't produce the module. This happens because the
> > > toplevel Makefile checks for M= only once. This is controlled by the
> > > sub_make_done variable, which is exported after checking the command line
> > > options are passed to the top level Makefile. Once this variable is set it's
> > > the M= setting is never checked again on the recursive call.
> > >
> > > This can be observed when cleaning the bpf kselftest dir. When calling
> > >
> > >         $ make TARGETS="bpf" SKIP_TARGETS="" kselftest-clean
> > >
> > > What happens:
> > >
> > >         1. It checks for some command line settings (like M=) was passed (it wasn't),
> > >         set some definitions and exports sub_make_done.
> > >
> > >         2. Jump into tools/testing/selftests/bpf, and calls the clean target.
> > >
> > >         3. The clean target is overwritten to remove some files and then jump to
> > >         bpf_testmod dir and call clean there
> > >
> > >         4. On bpf_testmod/Makefile, the clean target will execute
> > >                 $(Q)make -C $(KDIR) M=$(BPF_TESTMOD_DIR) clean
> > >
> > >         5. The KDIR is to toplevel dir. The top Makefile will check that sub_make_done was
> > >         already set, ignoring the M= setting.
> > >
> > >         6. As M= wasn't checked, KBUILD_EXTMOD isn't set, and the clean target applies
> > >         to the kernel as a whole, making it clean all generated code/objects and
> > >         everything.
> > >
> > > One way to avoid it is to call "unexport sub_make_done" on
> > > tools/testing/selftests/bpf/bpf_testmod/Makefile before processing the all
> > > target, forcing the toplevel Makefile to process the M=, producing the module
> > > file correctly.
> > >
> > > If the M=dir points to /lib/modules/.../build, then it fails with "m2c: No such
> > > file", which I already reported here[1]. At the time this problem was treated
> > > like a problem with kselftest infrastructure.
> > >
> > > Important: The process works fine if the initial make invocation is targeted to a
> > > different directory (using -C), since it doesn't goes through the toplevel
> > > Makefile, and sub_make_done variable is not set.
> > >
> > > I attached a minimal reproducer, that can be used to better understand the
> > > problem. The "make testmod" and "make testmod-clean" have the same effect that
> > > can be seem with the bpf kselftests. There is a unexport call commented on
> > > test-mods/Makefile, and once that is called the process works as expected.
> > >
> > > Is there a better way to fix this? Is this really a problem, or am I missing
> > > something?
> >
> >
> > Or, using KBUILD_EXTMOD will work too.
>
> So it worked until 13b25489b6f8bd73ed65f07928f7c27a481f1820 ("kbuild: change working
> directory to external module directory with M="). Right now, it shows the same
> error as reported before:
>
> m2c    -o /home/mpdesouza/git/linux/scripts/Makefile.build -e /home/mpdesouza/git/linux/scripts/Makefile.build /home/mpdesouza/git/linux/scripts/Makefile.build.mod
> make[6]: m2c: No such file or directory
>
> To reproduce, it's just execute the livepatch ksefltests:
>
> $ make kselftest TARGETS=livepatch KDIR=<kernel source top dir>
>
> Is there something that we can do in this case to solve this issue?

The following may work although I did not test comprehensively.

I don't know why kselftests need to be started from multiple
places (top directory and tools/testing/selftests/).


diff --git a/tools/testing/selftests/lib.mk b/tools/testing/selftests/lib.mk
index d6edcfcb5be8..5587ace31b9e 100644
--- a/tools/testing/selftests/lib.mk
+++ b/tools/testing/selftests/lib.mk
@@ -143,7 +143,7 @@ else
 endif

 gen_mods_dir:
-       $(Q)$(MAKE) -C $(TEST_GEN_MODS_DIR)
+       $(Q)$(MAKE) sub_make_done= -C $(TEST_GEN_MODS_DIR)

 clean_mods_dir:
        $(Q)$(MAKE) -C $(TEST_GEN_MODS_DIR) clean











>
> I'm quite surprised that no bots found this problem yet.
>
> >
> >
> >
> >
> >
> > --
> > Best Regards
> > Masahiro Yamada
>
> /*
>  * !REMOVE THIS COMMENT BLOCK BEFORE SENDING THIS MAIL!
>  * below are signatures added by "hkml".
>  * edit signatures below, or use "hkml signature".
>  */
>
> Sent using hkml (https://github.com/sjp38/hackermail)



-- 
Best Regards
Masahiro Yamada

Re: # Toplevel Makefile doesn't process module build correctly on recursive make calls

[mpdesouza]

On Tue, 2025-01-14 at 21:36 +0900, Masahiro Yamada wrote:
> On Tue, Jan 14, 2025 at 4:38 AM Marcos Paulo de Souza
> <mpdesouza@suse.com> wrote:
> > 
> > On Sat, 14 Oct 2023 17:35:55 +0900 Masahiro Yamada
> > <masahiroy@kernel.org> wrote:
> > 
> > > On Tue, Oct 10, 2023 at 5:43 AM Marcos Paulo de Souza
> > > <mpdesouza@suse.de> wrote:
> > > > 
> > > > Hi all,
> > > > 
> > > > I found an issue while moving the livepatch kselftest modules
> > > > to be built on the
> > > > fly, instead of building them on kernel building.
> > > > 
> > > > If, for some reason, there is a recursive make invocation that
> > > > starts from the
> > > > top level Makefile and in the leaf Makefile it tries to build a
> > > > module (using M=
> > > > in the make invocation), it doesn't produce the module. This
> > > > happens because the
> > > > toplevel Makefile checks for M= only once. This is controlled
> > > > by the
> > > > sub_make_done variable, which is exported after checking the
> > > > command line
> > > > options are passed to the top level Makefile. Once this
> > > > variable is set it's
> > > > the M= setting is never checked again on the recursive call.
> > > > 
> > > > This can be observed when cleaning the bpf kselftest dir. When
> > > > calling
> > > > 
> > > >         $ make TARGETS="bpf" SKIP_TARGETS="" kselftest-clean
> > > > 
> > > > What happens:
> > > > 
> > > >         1. It checks for some command line settings (like M=)
> > > > was passed (it wasn't),
> > > >         set some definitions and exports sub_make_done.
> > > > 
> > > >         2. Jump into tools/testing/selftests/bpf, and calls the
> > > > clean target.
> > > > 
> > > >         3. The clean target is overwritten to remove some files
> > > > and then jump to
> > > >         bpf_testmod dir and call clean there
> > > > 
> > > >         4. On bpf_testmod/Makefile, the clean target will
> > > > execute
> > > >                 $(Q)make -C $(KDIR) M=$(BPF_TESTMOD_DIR) clean
> > > > 
> > > >         5. The KDIR is to toplevel dir. The top Makefile will
> > > > check that sub_make_done was
> > > >         already set, ignoring the M= setting.
> > > > 
> > > >         6. As M= wasn't checked, KBUILD_EXTMOD isn't set, and
> > > > the clean target applies
> > > >         to the kernel as a whole, making it clean all generated
> > > > code/objects and
> > > >         everything.
> > > > 
> > > > One way to avoid it is to call "unexport sub_make_done" on
> > > > tools/testing/selftests/bpf/bpf_testmod/Makefile before
> > > > processing the all
> > > > target, forcing the toplevel Makefile to process the M=,
> > > > producing the module
> > > > file correctly.
> > > > 
> > > > If the M=dir points to /lib/modules/.../build, then it fails
> > > > with "m2c: No such
> > > > file", which I already reported here[1]. At the time this
> > > > problem was treated
> > > > like a problem with kselftest infrastructure.
> > > > 
> > > > Important: The process works fine if the initial make
> > > > invocation is targeted to a
> > > > different directory (using -C), since it doesn't goes through
> > > > the toplevel
> > > > Makefile, and sub_make_done variable is not set.
> > > > 
> > > > I attached a minimal reproducer, that can be used to better
> > > > understand the
> > > > problem. The "make testmod" and "make testmod-clean" have the
> > > > same effect that
> > > > can be seem with the bpf kselftests. There is a unexport call
> > > > commented on
> > > > test-mods/Makefile, and once that is called the process works
> > > > as expected.
> > > > 
> > > > Is there a better way to fix this? Is this really a problem, or
> > > > am I missing
> > > > something?
> > > 
> > > 
> > > Or, using KBUILD_EXTMOD will work too.
> > 
> > So it worked until 13b25489b6f8bd73ed65f07928f7c27a481f1820
> > ("kbuild: change working
> > directory to external module directory with M="). Right now, it
> > shows the same
> > error as reported before:
> > 
> > m2c    -o /home/mpdesouza/git/linux/scripts/Makefile.build -e
> > /home/mpdesouza/git/linux/scripts/Makefile.build
> > /home/mpdesouza/git/linux/scripts/Makefile.build.mod
> > make[6]: m2c: No such file or directory
> > 
> > To reproduce, it's just execute the livepatch ksefltests:
> > 
> > $ make kselftest TARGETS=livepatch KDIR=<kernel source top dir>
> > 
> > Is there something that we can do in this case to solve this issue?
> 
> The following may work although I did not test comprehensively.

Unfortunately it didn't work:


$ make LLVM=1 kselftest TARGETS=livepatch
KDIR=/home/mpdesouza/git/linux
/home/mpdesouza/git/linux/Makefile:740:
/home/mpdesouza/git/linux/tools/testing/selftests/livepatch/test_module

s/include/config/auto.conf: No such file or directory
make[5]: *** [Makefile:243: __sub-make] Error 2
make[4]: *** [Makefile:19: modules] Error 2
make[3]: *** [../lib.mk:146: gen_mods_dir] Error 2
make[2]: *** [Makefile:197: all] Error 2
make[1]: *** [/home/mpdesouza/git/linux/Makefile:1402: kselftest] Error
2
make: *** [Makefile:243: __sub-make] Error 2


> 
> I don't know why kselftests need to be started from multiple
> places (top directory and tools/testing/selftests/).

Well, I'm not sure why as well, but it's a requirement that it should
work for all kselftests targets...

> 
> 
> diff --git a/tools/testing/selftests/lib.mk
> b/tools/testing/selftests/lib.mk
> index d6edcfcb5be8..5587ace31b9e 100644
> --- a/tools/testing/selftests/lib.mk
> +++ b/tools/testing/selftests/lib.mk
> @@ -143,7 +143,7 @@ else
>  endif
> 
>  gen_mods_dir:
> -       $(Q)$(MAKE) -C $(TEST_GEN_MODS_DIR)
> +       $(Q)$(MAKE) sub_make_done= -C $(TEST_GEN_MODS_DIR)
> 
>  clean_mods_dir:
>         $(Q)$(MAKE) -C $(TEST_GEN_MODS_DIR) clean
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> > 
> > I'm quite surprised that no bots found this problem yet.
> > 
> > > 
> > > 
> > > 
> > > 
> > > 
> > > --
> > > Best Regards
> > > Masahiro Yamada
> > 
> > /*
> >  * !REMOVE THIS COMMENT BLOCK BEFORE SENDING THIS MAIL!
> >  * below are signatures added by "hkml".
> >  * edit signatures below, or use "hkml signature".
> >  */
> > 
> > Sent using hkml (https://github.com/sjp38/hackermail)
> 
> 
>