Re: # Toplevel Makefile doesn't process module build correctly on recursive make calls

[Nicolas Schier <n.schier@avm.de>]

On Thu, Oct 19, 2023 at 03:50:05PM -0300, Marcos Paulo de Souza wrote:
> On Sat, Oct 14, 2023 at 05:35:55PM +0900, Masahiro Yamada wrote:
> > On Tue, Oct 10, 2023 at 5:43 AM Marcos Paulo de Souza <mpdesouza@suse.de> wrote:
> > >
> > > Hi all,
> > >
> > > I found an issue while moving the livepatch kselftest modules to be built on the
> > > fly, instead of building them on kernel building.
> > >
> > > If, for some reason, there is a recursive make invocation that starts from the
> > > top level Makefile and in the leaf Makefile it tries to build a module (using M=
> > > in the make invocation), it doesn't produce the module. This happens because the
> > > toplevel Makefile checks for M= only once. This is controlled by the
> > > sub_make_done variable, which is exported after checking the command line
> > > options are passed to the top level Makefile. Once this variable is set it's
> > > the M= setting is never checked again on the recursive call.
> > >
> > > This can be observed when cleaning the bpf kselftest dir. When calling
> > >
> > >         $ make TARGETS="bpf" SKIP_TARGETS="" kselftest-clean
> > >
> > > What happens:
> > >
> > >         1. It checks for some command line settings (like M=) was passed (it wasn't),
> > >         set some definitions and exports sub_make_done.
> > >
> > >         2. Jump into tools/testing/selftests/bpf, and calls the clean target.
> > >
> > >         3. The clean target is overwritten to remove some files and then jump to
> > >         bpf_testmod dir and call clean there
> > >
> > >         4. On bpf_testmod/Makefile, the clean target will execute
> > >                 $(Q)make -C $(KDIR) M=$(BPF_TESTMOD_DIR) clean
> > >
> > >         5. The KDIR is to toplevel dir. The top Makefile will check that sub_make_done was
> > >         already set, ignoring the M= setting.
> > >
> > >         6. As M= wasn't checked, KBUILD_EXTMOD isn't set, and the clean target applies
> > >         to the kernel as a whole, making it clean all generated code/objects and
> > >         everything.
> > >
> > > One way to avoid it is to call "unexport sub_make_done" on
> > > tools/testing/selftests/bpf/bpf_testmod/Makefile before processing the all
> > > target, forcing the toplevel Makefile to process the M=, producing the module
> > > file correctly.
> > >
> > > If the M=dir points to /lib/modules/.../build, then it fails with "m2c: No such
> > > file", which I already reported here[1]. At the time this problem was treated
> > > like a problem with kselftest infrastructure.
> > >
> > > Important: The process works fine if the initial make invocation is targeted to a
> > > different directory (using -C), since it doesn't goes through the toplevel
> > > Makefile, and sub_make_done variable is not set.
> > >
> > > I attached a minimal reproducer, that can be used to better understand the
> > > problem. The "make testmod" and "make testmod-clean" have the same effect that
> > > can be seem with the bpf kselftests. There is a unexport call commented on
> > > test-mods/Makefile, and once that is called the process works as expected.
> > >
> > > Is there a better way to fix this? Is this really a problem, or am I missing
> > > something?
> > 
> > 
> > Or, using KBUILD_EXTMOD will work too.
> 
> Yes, that works, only if set to /lib/modules:
> 
> $ make kselftest TARGETS=bpf SKIP_TARGETS=""
> make[3]: Entering directory '/home/mpdesouza/git/linux/tools/testing/selftests/bpf'
>   MOD      bpf_testmod.ko
> warning: the compiler differs from the one used to build the kernel
>   The kernel was built by: gcc (SUSE Linux) 13.2.1 20230803 [revision cc279d6c64562f05019e1d12d0d825f9391b5553]
>   You are using:           gcc (SUSE Linux) 13.2.1 20230912 [revision b96e66fd4ef3e36983969fb8cdd1956f551a074b]
>   CC [M]  /home/mpdesouza/git/linux/tools/testing/selftests/bpf/bpf_testmod/bpf_testmod.o
>   MODPOST /home/mpdesouza/git/linux/tools/testing/selftests/bpf/bpf_testmod/Module.symvers
>   CC [M]  /home/mpdesouza/git/linux/tools/testing/selftests/bpf/bpf_testmod/bpf_testmod.mod.o
>   LD [M]  /home/mpdesouza/git/linux/tools/testing/selftests/bpf/bpf_testmod/bpf_testmod.ko
>   BTF [M] /home/mpdesouza/git/linux/tools/testing/selftests/bpf/bpf_testmod/bpf_testmod.ko
> Skipping BTF generation for /home/mpdesouza/git/linux/tools/testing/selftests/bpf/bpf_testmod/bpf_testmod.ko due to unavailability of vmlinux
>   BINARY   xdp_synproxy
> ...
> 
> But if we set the KBUILD_EXTMOD to toplevel Makefile, it fails with a different
> strange issue:
> 
> $ make kselftest TARGETS=bpf SKIP_TARGETS=""
>   BINARY   urandom_read
>   MOD      bpf_testmod.ko
> m2c    -o scripts/Makefile.build -e scripts/Makefile.build scripts/Makefile.build.mod
> make[6]: m2c: No such file or directory
> make[6]: *** [<builtin>: scripts/Makefile.build] Error 127
> make[5]: *** [Makefile:1913: /home/mpdesouza/git/linux/tools/testing/selftests/bpf/bpf_testmod] Error 2
> make[4]: *** [Makefile:19: all] Error 2
> make[3]: *** [Makefile:229: /home/mpdesouza/git/linux/tools/testing/selftests/bpf/bpf_testmod.ko] Error 2
> make[3]: Leaving directory '/home/mpdesouza/git/linux/tools/testing/selftests/bpf'
> make[2]: *** [Makefile:175: all] Error 2
> make[1]: *** [/home/mpdesouza/git/linux/Makefile:1362: kselftest] Error 2
> 
> I attached a patch that can reproduce the case where it works, and the case
> where it doesn't by changing the value of KDIR.
> 
> I understand that KBUILD_EXTMOD, as the name implies, was designed to build
> "external" modules, and not ones that live inside kernel, but how could this be
> solved?

It seems to me as if there is some confusion about in-tree vs.
out-of-tree kmods.

KBUILD_EXTMOD and M are almost the same and indicate that you want to
build _external_ (=out-of-tree) kernel modules.  In-tree modules are
only those that stay in-tree _and_ are built along with the kernel.
Thus, 'make modules KBUILD_EXTMOD=fs/ext4' could be used to build ext4
kmod as "out-of-tree" kernel module, that even taints the kernel if it
gets loaded.

If you want bpf_testmod.ko to be an in-tree kmod, it has to be build
during the usual kernel build, not by running 'make kselftest'.

If you use 'make -C $(KDIR)' for building out-of-tree kmods, KDIR has to
point to the kernel build directory.  (Or it may point to the source
tree if you give O=$(BUILDDIR) as well).

HTH.

Kind regards,
Nicolas


> For the sake of my initial about livepatch kselftests, KBUILD_EXTMOD
> will suffice, since we will target /lib/modules, but I would like to know what
> we can do in this case. Do you have other suggestions?
> 
> Thanks in advance,
>   Marcos
> 
> > 
> > 
> > 
> > 
> > 
> > --
> > Best Regards
> > Masahiro Yamada

> diff --git a/tools/testing/selftests/bpf/bpf_testmod/Makefile b/tools/testing/selftests/bpf/bpf_testmod/Makefile
> index 15cb36c4483a..1dce76f35405 100644
> --- a/tools/testing/selftests/bpf/bpf_testmod/Makefile
> +++ b/tools/testing/selftests/bpf/bpf_testmod/Makefile
> @@ -1,5 +1,6 @@
>  BPF_TESTMOD_DIR := $(realpath $(dir $(abspath $(lastword $(MAKEFILE_LIST)))))
> -KDIR ?= $(abspath $(BPF_TESTMOD_DIR)/../../../../..)
> +#KDIR ?= $(abspath $(BPF_TESTMOD_DIR)/../../../../..)
> +KDIR ?= /lib/modules/$(shell uname -r)/build
>  
>  ifeq ($(V),1)
>  Q =
> @@ -12,9 +13,10 @@ MODULES = bpf_testmod.ko
>  obj-m += bpf_testmod.o
>  CFLAGS_bpf_testmod.o = -I$(src)
>  
> +export KBUILD_EXTMOD := $(BPF_TESTMOD_DIR)
> +
>  all:
> -	+$(Q)make -C $(KDIR) M=$(BPF_TESTMOD_DIR) modules
> +	+$(Q)make -C $(KDIR) modules
>  
>  clean:
> -	+$(Q)make -C $(KDIR) M=$(BPF_TESTMOD_DIR) clean
> -
> +	+$(Q)make -C $(KDIR) clean