Re: # Toplevel Makefile doesn't process module build correctly on recursive make calls

[mpdesouza@suse.com]

On Tue, 2025-01-14 at 21:36 +0900, Masahiro Yamada wrote:
> On Tue, Jan 14, 2025 at 4:38 AM Marcos Paulo de Souza
> <mpdesouza@suse.com> wrote:
> > 
> > On Sat, 14 Oct 2023 17:35:55 +0900 Masahiro Yamada
> > <masahiroy@kernel.org> wrote:
> > 
> > > On Tue, Oct 10, 2023 at 5:43 AM Marcos Paulo de Souza
> > > <mpdesouza@suse.de> wrote:
> > > > 
> > > > Hi all,
> > > > 
> > > > I found an issue while moving the livepatch kselftest modules
> > > > to be built on the
> > > > fly, instead of building them on kernel building.
> > > > 
> > > > If, for some reason, there is a recursive make invocation that
> > > > starts from the
> > > > top level Makefile and in the leaf Makefile it tries to build a
> > > > module (using M=
> > > > in the make invocation), it doesn't produce the module. This
> > > > happens because the
> > > > toplevel Makefile checks for M= only once. This is controlled
> > > > by the
> > > > sub_make_done variable, which is exported after checking the
> > > > command line
> > > > options are passed to the top level Makefile. Once this
> > > > variable is set it's
> > > > the M= setting is never checked again on the recursive call.
> > > > 
> > > > This can be observed when cleaning the bpf kselftest dir. When
> > > > calling
> > > > 
> > > >         $ make TARGETS="bpf" SKIP_TARGETS="" kselftest-clean
> > > > 
> > > > What happens:
> > > > 
> > > >         1. It checks for some command line settings (like M=)
> > > > was passed (it wasn't),
> > > >         set some definitions and exports sub_make_done.
> > > > 
> > > >         2. Jump into tools/testing/selftests/bpf, and calls the
> > > > clean target.
> > > > 
> > > >         3. The clean target is overwritten to remove some files
> > > > and then jump to
> > > >         bpf_testmod dir and call clean there
> > > > 
> > > >         4. On bpf_testmod/Makefile, the clean target will
> > > > execute
> > > >                 $(Q)make -C $(KDIR) M=$(BPF_TESTMOD_DIR) clean
> > > > 
> > > >         5. The KDIR is to toplevel dir. The top Makefile will
> > > > check that sub_make_done was
> > > >         already set, ignoring the M= setting.
> > > > 
> > > >         6. As M= wasn't checked, KBUILD_EXTMOD isn't set, and
> > > > the clean target applies
> > > >         to the kernel as a whole, making it clean all generated
> > > > code/objects and
> > > >         everything.
> > > > 
> > > > One way to avoid it is to call "unexport sub_make_done" on
> > > > tools/testing/selftests/bpf/bpf_testmod/Makefile before
> > > > processing the all
> > > > target, forcing the toplevel Makefile to process the M=,
> > > > producing the module
> > > > file correctly.
> > > > 
> > > > If the M=dir points to /lib/modules/.../build, then it fails
> > > > with "m2c: No such
> > > > file", which I already reported here[1]. At the time this
> > > > problem was treated
> > > > like a problem with kselftest infrastructure.
> > > > 
> > > > Important: The process works fine if the initial make
> > > > invocation is targeted to a
> > > > different directory (using -C), since it doesn't goes through
> > > > the toplevel
> > > > Makefile, and sub_make_done variable is not set.
> > > > 
> > > > I attached a minimal reproducer, that can be used to better
> > > > understand the
> > > > problem. The "make testmod" and "make testmod-clean" have the
> > > > same effect that
> > > > can be seem with the bpf kselftests. There is a unexport call
> > > > commented on
> > > > test-mods/Makefile, and once that is called the process works
> > > > as expected.
> > > > 
> > > > Is there a better way to fix this? Is this really a problem, or
> > > > am I missing
> > > > something?
> > > 
> > > 
> > > Or, using KBUILD_EXTMOD will work too.
> > 
> > So it worked until 13b25489b6f8bd73ed65f07928f7c27a481f1820
> > ("kbuild: change working
> > directory to external module directory with M="). Right now, it
> > shows the same
> > error as reported before:
> > 
> > m2c    -o /home/mpdesouza/git/linux/scripts/Makefile.build -e
> > /home/mpdesouza/git/linux/scripts/Makefile.build
> > /home/mpdesouza/git/linux/scripts/Makefile.build.mod
> > make[6]: m2c: No such file or directory
> > 
> > To reproduce, it's just execute the livepatch ksefltests:
> > 
> > $ make kselftest TARGETS=livepatch KDIR=<kernel source top dir>
> > 
> > Is there something that we can do in this case to solve this issue?
> 
> The following may work although I did not test comprehensively.

Unfortunately it didn't work:


$ make LLVM=1 kselftest TARGETS=livepatch
KDIR=/home/mpdesouza/git/linux
/home/mpdesouza/git/linux/Makefile:740:
/home/mpdesouza/git/linux/tools/testing/selftests/livepatch/test_module

s/include/config/auto.conf: No such file or directory
make[5]: *** [Makefile:243: __sub-make] Error 2
make[4]: *** [Makefile:19: modules] Error 2
make[3]: *** [../lib.mk:146: gen_mods_dir] Error 2
make[2]: *** [Makefile:197: all] Error 2
make[1]: *** [/home/mpdesouza/git/linux/Makefile:1402: kselftest] Error
2
make: *** [Makefile:243: __sub-make] Error 2


> 
> I don't know why kselftests need to be started from multiple
> places (top directory and tools/testing/selftests/).

Well, I'm not sure why as well, but it's a requirement that it should
work for all kselftests targets...

> 
> 
> diff --git a/tools/testing/selftests/lib.mk
> b/tools/testing/selftests/lib.mk
> index d6edcfcb5be8..5587ace31b9e 100644
> --- a/tools/testing/selftests/lib.mk
> +++ b/tools/testing/selftests/lib.mk
> @@ -143,7 +143,7 @@ else
>  endif
> 
>  gen_mods_dir:
> -       $(Q)$(MAKE) -C $(TEST_GEN_MODS_DIR)
> +       $(Q)$(MAKE) sub_make_done= -C $(TEST_GEN_MODS_DIR)
> 
>  clean_mods_dir:
>         $(Q)$(MAKE) -C $(TEST_GEN_MODS_DIR) clean
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> > 
> > I'm quite surprised that no bots found this problem yet.
> > 
> > > 
> > > 
> > > 
> > > 
> > > 
> > > --
> > > Best Regards
> > > Masahiro Yamada
> > 
> > /*
> >  * !REMOVE THIS COMMENT BLOCK BEFORE SENDING THIS MAIL!
> >  * below are signatures added by "hkml".
> >  * edit signatures below, or use "hkml signature".
> >  */
> > 
> > Sent using hkml (https://github.com/sjp38/hackermail)
> 
> 
>