* possible dependency error?
@ 2023-05-17 23:27 Bjorn Helgaas
2023-05-18 8:26 ` Masahiro Yamada
0 siblings, 1 reply; 7+ messages in thread
From: Bjorn Helgaas @ 2023-05-17 23:27 UTC (permalink / raw)
To: linux-kbuild
This is on v6.4-rc1. I fat-fingered the make target (I intended
"pciehp.o", not "pciehp.c"), then interrupted the build when I noticed
my mistake:
06:04:15 ~/linux (hotplug)$ make drivers/pci/hotplug/pciehp.c
SYNC include/config/auto.conf.cmd
^Cmake: *** [include/config/auto.conf.cmd] Deleting file 'include/generated/rustc_cfg'
make: *** [include/config/auto.conf.cmd] Deleting file 'include/generated/autoconf.h'
make[2]: *** [scripts/kconfig/Makefile:77: syncconfig] Interrupt
make[1]: *** [Makefile:692: syncconfig] Interrupt
make: *** [Makefile:793: include/config/auto.conf.cmd] Interrupt
Subsequent builds now fail ("pciehp.o" is *also* an incorrect target,
but doesn't seem related to the error):
06:04:22 ~/linux (hotplug)$ make drivers/pci/hotplug/pciehp.o
SYNC include/config/auto.conf.cmd
UPD include/config/kernel.release
UPD include/generated/utsrelease.h
UPD include/generated/compile.h
CC scripts/mod/empty.o
MKELF scripts/mod/elfconfig.h
HOSTCC scripts/mod/modpost.o
CC scripts/mod/devicetable-offsets.s
HOSTCC scripts/mod/file2alias.o
HOSTCC scripts/mod/sumversion.o
HOSTLD scripts/mod/modpost
CC kernel/bounds.s
CC arch/x86/kernel/asm-offsets.s
CALL scripts/checksyscalls.sh
DESCEND objtool
HOSTCC /home/bjorn/linux/tools/objtool/fixdep.o
HOSTLD /home/bjorn/linux/tools/objtool/fixdep-in.o
LINK /home/bjorn/linux/tools/objtool/fixdep
make[4]: *** No rule to make target '/usr/include/x86_64-linux-gnu/bits/sys_errlist.h', needed by '/home/bjorn/linux/tools/objtool/libsubcmd/exec-cmd.o'. Stop.
make[3]: *** [Makefile:80: /home/bjorn/linux/tools/objtool/libsubcmd/libsubcmd-in.o] Error 2
make[2]: *** [Makefile:78: /home/bjorn/linux/tools/objtool/libsubcmd/libsubcmd.a] Error 2
make[1]: *** [Makefile:73: objtool] Error 2
make: *** [Makefile:1440: tools/objtool] Error 2
I finally got the right target, but the build still fails:
06:04:39 ~/linux (hotplug)$ make drivers/pci/hotplug/
CALL scripts/checksyscalls.sh
DESCEND objtool
make[4]: *** No rule to make target '/usr/include/x86_64-linux-gnu/bits/sys_errlist.h', needed by '/home/bjorn/linux/tools/objtool/libsubcmd/exec-cmd.o'. Stop.
make[3]: *** [Makefile:80: /home/bjorn/linux/tools/objtool/libsubcmd/libsubcmd-in.o] Error 2
make[2]: *** [Makefile:78: /home/bjorn/linux/tools/objtool/libsubcmd/libsubcmd.a] Error 2
make[1]: *** [Makefile:73: objtool] Error 2
make: *** [Makefile:1440: tools/objtool] Error 2
After "make distclean", everything works as expected, so maybe this is
just the expected behavior after my initial user error? I dunno; it
seemed surprising. Just FYI.
Bjorn
^ permalink raw reply [flat|nested] 7+ messages in thread* Re: possible dependency error? 2023-05-17 23:27 possible dependency error? Bjorn Helgaas @ 2023-05-18 8:26 ` Masahiro Yamada 2024-05-23 17:54 ` Brian Norris 0 siblings, 1 reply; 7+ messages in thread From: Masahiro Yamada @ 2023-05-18 8:26 UTC (permalink / raw) To: Bjorn Helgaas; +Cc: linux-kbuild, Josh Poimboeuf, Peter Zijlstra (+CC: Josh Poimboeuf,Peter Zijlstra, objtool maintainer) On Thu, May 18, 2023 at 8:27 AM Bjorn Helgaas <helgaas@kernel.org> wrote: > > This is on v6.4-rc1. I fat-fingered the make target (I intended > "pciehp.o", not "pciehp.c"), then interrupted the build when I noticed > my mistake: > > 06:04:15 ~/linux (hotplug)$ make drivers/pci/hotplug/pciehp.c > SYNC include/config/auto.conf.cmd > ^Cmake: *** [include/config/auto.conf.cmd] Deleting file 'include/generated/rustc_cfg' > make: *** [include/config/auto.conf.cmd] Deleting file 'include/generated/autoconf.h' > make[2]: *** [scripts/kconfig/Makefile:77: syncconfig] Interrupt > make[1]: *** [Makefile:692: syncconfig] Interrupt > make: *** [Makefile:793: include/config/auto.conf.cmd] Interrupt > > Subsequent builds now fail ("pciehp.o" is *also* an incorrect target, > but doesn't seem related to the error): > > 06:04:22 ~/linux (hotplug)$ make drivers/pci/hotplug/pciehp.o > SYNC include/config/auto.conf.cmd > UPD include/config/kernel.release > UPD include/generated/utsrelease.h > UPD include/generated/compile.h > CC scripts/mod/empty.o > MKELF scripts/mod/elfconfig.h > HOSTCC scripts/mod/modpost.o > CC scripts/mod/devicetable-offsets.s > HOSTCC scripts/mod/file2alias.o > HOSTCC scripts/mod/sumversion.o > HOSTLD scripts/mod/modpost > CC kernel/bounds.s > CC arch/x86/kernel/asm-offsets.s > CALL scripts/checksyscalls.sh > DESCEND objtool > HOSTCC /home/bjorn/linux/tools/objtool/fixdep.o > HOSTLD /home/bjorn/linux/tools/objtool/fixdep-in.o > LINK /home/bjorn/linux/tools/objtool/fixdep > make[4]: *** No rule to make target '/usr/include/x86_64-linux-gnu/bits/sys_errlist.h', needed by '/home/bjorn/linux/tools/objtool/libsubcmd/exec-cmd.o'. Stop. > make[3]: *** [Makefile:80: /home/bjorn/linux/tools/objtool/libsubcmd/libsubcmd-in.o] Error 2 > make[2]: *** [Makefile:78: /home/bjorn/linux/tools/objtool/libsubcmd/libsubcmd.a] Error 2 > make[1]: *** [Makefile:73: objtool] Error 2 > make: *** [Makefile:1440: tools/objtool] Error 2 > > I finally got the right target, but the build still fails: > > 06:04:39 ~/linux (hotplug)$ make drivers/pci/hotplug/ > CALL scripts/checksyscalls.sh > DESCEND objtool > make[4]: *** No rule to make target '/usr/include/x86_64-linux-gnu/bits/sys_errlist.h', needed by '/home/bjorn/linux/tools/objtool/libsubcmd/exec-cmd.o'. Stop. > make[3]: *** [Makefile:80: /home/bjorn/linux/tools/objtool/libsubcmd/libsubcmd-in.o] Error 2 > make[2]: *** [Makefile:78: /home/bjorn/linux/tools/objtool/libsubcmd/libsubcmd.a] Error 2 > make[1]: *** [Makefile:73: objtool] Error 2 > make: *** [Makefile:1440: tools/objtool] Error 2 > > After "make distclean", everything works as expected, so maybe this is > just the expected behavior after my initial user error? I dunno; it > seemed surprising. Just FYI. > > Bjorn I do not know what is happening on your build machine, but judging from the error log, something went wrong while building objtool. objtool Makefile is not a part of Kbuild. The maintainers of objtool may have some insight. BTW, I could not reproduce the issue. I could not find anything weird from the Kbuild perspective so far. The errors in the single target builds are expected. This is what I did. $ git clean -fdx $ make defconfig $ make drivers/pci/hotplug/pciehp.c -> Fail. This is expected behavior $ make drivers/pci/hotplug/pciehp.o -> Fail. This is also expected behavior because CONFIG_HOTPLUG_PCI_PCIE is unset $ make drivers/pci/hotplug/ -> success for me Is this the same command sequence as you did? -- Best Regards Masahiro Yamada ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: possible dependency error? 2023-05-18 8:26 ` Masahiro Yamada @ 2024-05-23 17:54 ` Brian Norris 2024-05-25 16:35 ` Masahiro Yamada 0 siblings, 1 reply; 7+ messages in thread From: Brian Norris @ 2024-05-23 17:54 UTC (permalink / raw) To: Masahiro Yamada Cc: Bjorn Helgaas, linux-kbuild, Josh Poimboeuf, Peter Zijlstra Necromancing an old thread, since it identifies the same problem I've been poking at recently: On Thu, May 18, 2023 at 05:26:28PM +0900, Masahiro Yamada wrote: > (+CC: Josh Poimboeuf,Peter Zijlstra, objtool maintainer) > > On Thu, May 18, 2023 at 8:27 AM Bjorn Helgaas <helgaas@kernel.org> wrote: > > > > This is on v6.4-rc1. I fat-fingered the make target (I intended > > "pciehp.o", not "pciehp.c"), then interrupted the build when I noticed > > my mistake: > > > > 06:04:15 ~/linux (hotplug)$ make drivers/pci/hotplug/pciehp.c > > SYNC include/config/auto.conf.cmd > > ^Cmake: *** [include/config/auto.conf.cmd] Deleting file 'include/generated/rustc_cfg' > > make: *** [include/config/auto.conf.cmd] Deleting file 'include/generated/autoconf.h' > > make[2]: *** [scripts/kconfig/Makefile:77: syncconfig] Interrupt > > make[1]: *** [Makefile:692: syncconfig] Interrupt > > make: *** [Makefile:793: include/config/auto.conf.cmd] Interrupt > > > > Subsequent builds now fail ("pciehp.o" is *also* an incorrect target, > > but doesn't seem related to the error): > > > > 06:04:22 ~/linux (hotplug)$ make drivers/pci/hotplug/pciehp.o > > SYNC include/config/auto.conf.cmd > > UPD include/config/kernel.release > > UPD include/generated/utsrelease.h > > UPD include/generated/compile.h > > CC scripts/mod/empty.o > > MKELF scripts/mod/elfconfig.h > > HOSTCC scripts/mod/modpost.o > > CC scripts/mod/devicetable-offsets.s > > HOSTCC scripts/mod/file2alias.o > > HOSTCC scripts/mod/sumversion.o > > HOSTLD scripts/mod/modpost > > CC kernel/bounds.s > > CC arch/x86/kernel/asm-offsets.s > > CALL scripts/checksyscalls.sh > > DESCEND objtool > > HOSTCC /home/bjorn/linux/tools/objtool/fixdep.o > > HOSTLD /home/bjorn/linux/tools/objtool/fixdep-in.o > > LINK /home/bjorn/linux/tools/objtool/fixdep > > make[4]: *** No rule to make target '/usr/include/x86_64-linux-gnu/bits/sys_errlist.h', needed by '/home/bjorn/linux/tools/objtool/libsubcmd/exec-cmd.o'. Stop. > > make[3]: *** [Makefile:80: /home/bjorn/linux/tools/objtool/libsubcmd/libsubcmd-in.o] Error 2 > > make[2]: *** [Makefile:78: /home/bjorn/linux/tools/objtool/libsubcmd/libsubcmd.a] Error 2 > > make[1]: *** [Makefile:73: objtool] Error 2 > > make: *** [Makefile:1440: tools/objtool] Error 2 > > > > I finally got the right target, but the build still fails: > > > > 06:04:39 ~/linux (hotplug)$ make drivers/pci/hotplug/ > > CALL scripts/checksyscalls.sh > > DESCEND objtool > > make[4]: *** No rule to make target '/usr/include/x86_64-linux-gnu/bits/sys_errlist.h', needed by '/home/bjorn/linux/tools/objtool/libsubcmd/exec-cmd.o'. Stop. > > make[3]: *** [Makefile:80: /home/bjorn/linux/tools/objtool/libsubcmd/libsubcmd-in.o] Error 2 > > make[2]: *** [Makefile:78: /home/bjorn/linux/tools/objtool/libsubcmd/libsubcmd.a] Error 2 > > make[1]: *** [Makefile:73: objtool] Error 2 > > make: *** [Makefile:1440: tools/objtool] Error 2 > > > > After "make distclean", everything works as expected, so maybe this is > > just the expected behavior after my initial user error? I dunno; it > > seemed surprising. Just FYI. I believe we've been hitting a similar issue at $JOB, which looks like the following (this is on a 5.15-ish kernel, but AFAICT everything is still relevant): make[5]: *** No rule to make target '[...]/tools/include/linux/compiler.h', needed by '[...]/tools/bpf/resolve_btfids/libsubcmd/exec-cmd.o'. Stop. make[4]: *** [Makefile:59: [...]/tools/bpf/resolve_btfids/libsubcmd/libsubcmd-in.o] Error 2 make[3]: *** [Makefile:45: [...]/tools/bpf/resolve_btfids//libsubcmd/libsubcmd.a] Error 2 make[2]: *** [Makefile:72: bpf/resolve_btfids] Error 2 make[1]: *** [[...]/Makefile:1401: tools/bpf/resolve_btfids] Error 2 This particular case happens for us when the source tree is moving, but we're sharing an O= cache. This may or may not be a good idea, but AFAICT there still is a real bug underneath, which I explore below. > I do not know what is happening on your build machine, > but judging from the error log, something went wrong > while building objtool. > > objtool Makefile is not a part of Kbuild. > The maintainers of objtool may have some insight. I'm no maintainer, but I found that the .exec-cmd.o.cmd dep file is generated incorrectly due to improper fixdep dependencies: $ head -2 [...]/tools/bpf/resolve_btfids/libsubcmd/.exec-cmd.o.cmd # cannot find fixdep ([...]/tools/bpf/resolve_btfids/libsubcmd//fixdep) # using basic dep data Now, this soft error is normally OK, as long as you don't have any missing or moved headers. But if these moved around, then normally the fixdep'd dependencies would help rebuild (and regenerate .cmd files with new paths) silently. The bad .cmd file is reliably reproduced by: # for an easier-to-build target that also builds libsubcmd: cd tools/objtool # for maximum cleanliness: git clean -xfd .. make head -2 libsubcmd/.exec-cmd.o.cmd (NB: if you look hard enough, you'll notice that we have a similar "cannot find fixdep" error for tools/.../.fixdep.o.cmd too. I have some analysis at https://issuetracker.google.com/313508829#comment32 -- this one is publicly accessible -- but its solution would be more complex. I may raise a separate thread.) The following patch fixes libsubcmd stuff for me. I can resubmit in a proper patch form if desired, or feel free to scrape it as-is. Signed-off-by: Brian Norris <briannorris@chromium.org> --- diff --git a/tools/lib/subcmd/Makefile b/tools/lib/subcmd/Makefile index b87213263a5e..59b09f280e49 100644 --- a/tools/lib/subcmd/Makefile +++ b/tools/lib/subcmd/Makefile @@ -76,7 +76,7 @@ include $(srctree)/tools/build/Makefile.include all: fixdep $(LIBFILE) -$(SUBCMD_IN): FORCE +$(SUBCMD_IN): fixdep FORCE @$(MAKE) $(build)=libsubcmd $(LIBFILE): $(SUBCMD_IN) ^ permalink raw reply related [flat|nested] 7+ messages in thread
* Re: possible dependency error? 2024-05-23 17:54 ` Brian Norris @ 2024-05-25 16:35 ` Masahiro Yamada 2024-06-18 23:29 ` Brian Norris 0 siblings, 1 reply; 7+ messages in thread From: Masahiro Yamada @ 2024-05-25 16:35 UTC (permalink / raw) To: Brian Norris; +Cc: Bjorn Helgaas, linux-kbuild, Josh Poimboeuf, Peter Zijlstra On Fri, May 24, 2024 at 2:54 AM Brian Norris <briannorris@chromium.org> wrote: > > Necromancing an old thread, since it identifies the same problem I've > been poking at recently: > > On Thu, May 18, 2023 at 05:26:28PM +0900, Masahiro Yamada wrote: > > (+CC: Josh Poimboeuf,Peter Zijlstra, objtool maintainer) > > > > On Thu, May 18, 2023 at 8:27 AM Bjorn Helgaas <helgaas@kernel.org> wrote: > > > > > > This is on v6.4-rc1. I fat-fingered the make target (I intended > > > "pciehp.o", not "pciehp.c"), then interrupted the build when I noticed > > > my mistake: > > > > > > 06:04:15 ~/linux (hotplug)$ make drivers/pci/hotplug/pciehp.c > > > SYNC include/config/auto.conf.cmd > > > ^Cmake: *** [include/config/auto.conf.cmd] Deleting file 'include/generated/rustc_cfg' > > > make: *** [include/config/auto.conf.cmd] Deleting file 'include/generated/autoconf.h' > > > make[2]: *** [scripts/kconfig/Makefile:77: syncconfig] Interrupt > > > make[1]: *** [Makefile:692: syncconfig] Interrupt > > > make: *** [Makefile:793: include/config/auto.conf.cmd] Interrupt > > > > > > Subsequent builds now fail ("pciehp.o" is *also* an incorrect target, > > > but doesn't seem related to the error): > > > > > > 06:04:22 ~/linux (hotplug)$ make drivers/pci/hotplug/pciehp.o > > > SYNC include/config/auto.conf.cmd > > > UPD include/config/kernel.release > > > UPD include/generated/utsrelease.h > > > UPD include/generated/compile.h > > > CC scripts/mod/empty.o > > > MKELF scripts/mod/elfconfig.h > > > HOSTCC scripts/mod/modpost.o > > > CC scripts/mod/devicetable-offsets.s > > > HOSTCC scripts/mod/file2alias.o > > > HOSTCC scripts/mod/sumversion.o > > > HOSTLD scripts/mod/modpost > > > CC kernel/bounds.s > > > CC arch/x86/kernel/asm-offsets.s > > > CALL scripts/checksyscalls.sh > > > DESCEND objtool > > > HOSTCC /home/bjorn/linux/tools/objtool/fixdep.o > > > HOSTLD /home/bjorn/linux/tools/objtool/fixdep-in.o > > > LINK /home/bjorn/linux/tools/objtool/fixdep > > > make[4]: *** No rule to make target '/usr/include/x86_64-linux-gnu/bits/sys_errlist.h', needed by '/home/bjorn/linux/tools/objtool/libsubcmd/exec-cmd.o'. Stop. > > > make[3]: *** [Makefile:80: /home/bjorn/linux/tools/objtool/libsubcmd/libsubcmd-in.o] Error 2 > > > make[2]: *** [Makefile:78: /home/bjorn/linux/tools/objtool/libsubcmd/libsubcmd.a] Error 2 > > > make[1]: *** [Makefile:73: objtool] Error 2 > > > make: *** [Makefile:1440: tools/objtool] Error 2 > > > > > > I finally got the right target, but the build still fails: > > > > > > 06:04:39 ~/linux (hotplug)$ make drivers/pci/hotplug/ > > > CALL scripts/checksyscalls.sh > > > DESCEND objtool > > > make[4]: *** No rule to make target '/usr/include/x86_64-linux-gnu/bits/sys_errlist.h', needed by '/home/bjorn/linux/tools/objtool/libsubcmd/exec-cmd.o'. Stop. > > > make[3]: *** [Makefile:80: /home/bjorn/linux/tools/objtool/libsubcmd/libsubcmd-in.o] Error 2 > > > make[2]: *** [Makefile:78: /home/bjorn/linux/tools/objtool/libsubcmd/libsubcmd.a] Error 2 > > > make[1]: *** [Makefile:73: objtool] Error 2 > > > make: *** [Makefile:1440: tools/objtool] Error 2 > > > > > > After "make distclean", everything works as expected, so maybe this is > > > just the expected behavior after my initial user error? I dunno; it > > > seemed surprising. Just FYI. > > I believe we've been hitting a similar issue at $JOB, which looks like > the following (this is on a 5.15-ish kernel, but AFAICT everything is > still relevant): > > make[5]: *** No rule to make target '[...]/tools/include/linux/compiler.h', needed by '[...]/tools/bpf/resolve_btfids/libsubcmd/exec-cmd.o'. Stop. > make[4]: *** [Makefile:59: [...]/tools/bpf/resolve_btfids/libsubcmd/libsubcmd-in.o] Error 2 > make[3]: *** [Makefile:45: [...]/tools/bpf/resolve_btfids//libsubcmd/libsubcmd.a] Error 2 > make[2]: *** [Makefile:72: bpf/resolve_btfids] Error 2 > make[1]: *** [[...]/Makefile:1401: tools/bpf/resolve_btfids] Error 2 > > This particular case happens for us when the source tree is moving, but > we're sharing an O= cache. This may or may not be a good idea, but > AFAICT there still is a real bug underneath, which I explore below. > > > I do not know what is happening on your build machine, > > but judging from the error log, something went wrong > > while building objtool. > > > > objtool Makefile is not a part of Kbuild. > > The maintainers of objtool may have some insight. > > I'm no maintainer, but I found that the .exec-cmd.o.cmd dep file is > generated incorrectly due to improper fixdep dependencies: > > $ head -2 [...]/tools/bpf/resolve_btfids/libsubcmd/.exec-cmd.o.cmd > # cannot find fixdep ([...]/tools/bpf/resolve_btfids/libsubcmd//fixdep) > # using basic dep data > > Now, this soft error is normally OK, as long as you don't have any > missing or moved headers. But if these moved around, then normally the > fixdep'd dependencies would help rebuild (and regenerate .cmd files with > new paths) silently. > > The bad .cmd file is reliably reproduced by: > > # for an easier-to-build target that also builds libsubcmd: > cd tools/objtool > # for maximum cleanliness: > git clean -xfd .. > make > head -2 libsubcmd/.exec-cmd.o.cmd > > (NB: if you look hard enough, you'll notice that we have a similar > "cannot find fixdep" error for tools/.../.fixdep.o.cmd too. I have some > analysis at https://issuetracker.google.com/313508829#comment32 -- > this one is publicly accessible -- but its solution would be more > complex. I may raise a separate thread.) > > The following patch fixes libsubcmd stuff for me. I can resubmit in a > proper patch form if desired, or feel free to scrape it as-is. > > Signed-off-by: Brian Norris <briannorris@chromium.org> > --- > > diff --git a/tools/lib/subcmd/Makefile b/tools/lib/subcmd/Makefile > index b87213263a5e..59b09f280e49 100644 > --- a/tools/lib/subcmd/Makefile > +++ b/tools/lib/subcmd/Makefile > @@ -76,7 +76,7 @@ include $(srctree)/tools/build/Makefile.include > > all: fixdep $(LIBFILE) > > -$(SUBCMD_IN): FORCE > +$(SUBCMD_IN): fixdep FORCE > @$(MAKE) $(build)=libsubcmd > > $(LIBFILE): $(SUBCMD_IN) I may not fully understand the design policy of the tools/ build system, but this fix is presumably correct because the 'fixdep' binary is needed in each sub-directory for it to work correctly. tools/bpf/resolve_btfids/libsubcmd/.exec-cmd.o.cmd must be generated by tools/bpf/resolve_btfids/libsubcmd/fixdep instead of by tools/bpf/resolve_btfids/fixdep. But, fixing tools/lib/subcmd/Makefile is not enough. *.cmd files under tools/bpf/resolve_btfids/libbpf/staticobjs/ are broken for the same reason. So, this is fundamentally broken in many places. And, as you noted, there is no easy way to fix .fixdep.o.cmd Your description in https://issuetracker.google.com/issues/313508829#comment32 is all correct. "can we just use Kbuild?" is a good question. I do not understand why they use fragile build systems, where obviously they cannot do it correctly. In fact, I submitted a patch to migrate objtool to Kbuild because that fixes all the issues cleanly. The objtool maintainers rejected it. https://lore.kernel.org/linux-kbuild/1551764896-8453-3-git-send-email-yamada.masahiro@socionext.com/ Not only the build system. He also refused to participate in the standard Documentation directory. tools/objtool/Documentation/objtool.txt still resides in its own directory. -- Best Regards Masahiro Yamada ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: possible dependency error? 2024-05-25 16:35 ` Masahiro Yamada @ 2024-06-18 23:29 ` Brian Norris 2024-06-19 6:02 ` Masahiro Yamada 0 siblings, 1 reply; 7+ messages in thread From: Brian Norris @ 2024-06-18 23:29 UTC (permalink / raw) To: Masahiro Yamada Cc: Bjorn Helgaas, linux-kbuild, Josh Poimboeuf, Peter Zijlstra Hi Masahiro, Thanks for the reply. I've been away for a bit, but I've poked at a few more things now. On Sun, May 26, 2024 at 01:35:43AM +0900, Masahiro Yamada wrote: > On Fri, May 24, 2024 at 2:54 AM Brian Norris <briannorris@chromium.org> wrote: > > --- a/tools/lib/subcmd/Makefile > > +++ b/tools/lib/subcmd/Makefile > > @@ -76,7 +76,7 @@ include $(srctree)/tools/build/Makefile.include > > > > all: fixdep $(LIBFILE) > > > > -$(SUBCMD_IN): FORCE > > +$(SUBCMD_IN): fixdep FORCE > > @$(MAKE) $(build)=libsubcmd > > > > $(LIBFILE): $(SUBCMD_IN) > > > > I may not fully understand the design policy of the tools/ build system, > but this fix is presumably correct because the 'fixdep' binary > is needed in each sub-directory for it to work correctly. > > tools/bpf/resolve_btfids/libsubcmd/.exec-cmd.o.cmd must > be generated by tools/bpf/resolve_btfids/libsubcmd/fixdep > instead of by tools/bpf/resolve_btfids/fixdep. > > But, fixing tools/lib/subcmd/Makefile is not enough. > > *.cmd files under tools/bpf/resolve_btfids/libbpf/staticobjs/ > are broken for the same reason. > So, this is fundamentally broken in many places. I think I hacked something that works there too. It gets a bit uglier, but not too bad IMO. > And, as you noted, there is no easy way to fix .fixdep.o.cmd I haven't come up with a *great* solution, but I came up with something that works for the most part, by circumventing the normal Build / Makefile.build split. It's getting pretty ugly too though... > Your description in > https://issuetracker.google.com/issues/313508829#comment32 > is all correct. > > > "can we just use Kbuild?" is a good question. > I do not understand why they use fragile build systems, > where obviously they cannot do it correctly. > > > In fact, I submitted a patch to migrate objtool to Kbuild > because that fixes all the issues cleanly. > > The objtool maintainers rejected it. > https://lore.kernel.org/linux-kbuild/1551764896-8453-3-git-send-email-yamada.masahiro@socionext.com/ > > > Not only the build system. > He also refused to participate in the standard Documentation > directory. > tools/objtool/Documentation/objtool.txt still resides in its own directory. While I don't love having to solve the same problems repeatedly (once in Kbuild; potentially-many in tools/), I'm also OK with trying to hack fixes into the current duplicate build system if it's not prohibitively complex to do so. Here's what I have for now -- I might submit some or all of this as a proper patchset if I can fix a few rough edges. diff --git a/tools/build/Makefile b/tools/build/Makefile index 17cdf01e29a0..fad93f64035d 100644 --- a/tools/build/Makefile +++ b/tools/build/Makefile @@ -43,11 +43,8 @@ ifneq ($(wildcard $(TMP_O)),) $(Q)$(MAKE) -C feature OUTPUT=$(TMP_O) clean >/dev/null endif -$(OUTPUT)fixdep-in.o: FORCE - $(Q)$(MAKE) $(build)=fixdep - -$(OUTPUT)fixdep: $(OUTPUT)fixdep-in.o - $(QUIET_LINK)$(HOSTCC) $(KBUILD_HOSTLDFLAGS) -o $@ $< +$(OUTPUT)fixdep: $(srctree)/tools/build/fixdep.c + $(QUIET_CC)$(HOSTCC) $(KBUILD_HOSTLDFLAGS) -o $@ $< FORCE: diff --git a/tools/build/Makefile.include b/tools/build/Makefile.include index 8dadaa0fbb43..27b2090cb293 100644 --- a/tools/build/Makefile.include +++ b/tools/build/Makefile.include @@ -1,8 +1,16 @@ # SPDX-License-Identifier: GPL-2.0-only build := -f $(srctree)/tools/build/Makefile.build dir=. obj +# More than just $(Q), we sometimes want to suppress all command output from a +# recursive make -- even the 'up to date' printout. +ifeq ($(V),1) + SILENT_MAKE = $(Q)$(MAKE) +else + SILENT_MAKE = $(Q)$(MAKE) --silent +endif + fixdep: - $(Q)$(MAKE) -C $(srctree)/tools/build CFLAGS= LDFLAGS= $(OUTPUT)fixdep + $(SILENT_MAKE) -C $(srctree)/tools/build CFLAGS= LDFLAGS= $(OUTPUT)fixdep fixdep-clean: $(Q)$(MAKE) -C $(srctree)/tools/build clean diff --git a/tools/lib/bpf/Makefile b/tools/lib/bpf/Makefile index 2cf892774346..a8f34de1ca25 100644 --- a/tools/lib/bpf/Makefile +++ b/tools/lib/bpf/Makefile @@ -154,6 +154,8 @@ $(BPF_IN_SHARED): force $(BPF_GENERATED) $(Q)$(MAKE) $(build)=libbpf OUTPUT=$(SHARED_OBJDIR) CFLAGS="$(CFLAGS) $(SHLIB_FLAGS)" $(BPF_IN_STATIC): force $(BPF_GENERATED) + $(call rule_mkdir) + $(SILENT_MAKE) -C $(srctree)/tools/build CFLAGS= LDFLAGS= OUTPUT=$(STATIC_OBJDIR) $(STATIC_OBJDIR)fixdep $(Q)$(MAKE) $(build)=libbpf OUTPUT=$(STATIC_OBJDIR) $(BPF_HELPER_DEFS): $(srctree)/tools/include/uapi/linux/bpf.h diff --git a/tools/lib/subcmd/Makefile b/tools/lib/subcmd/Makefile index b87213263a5e..59b09f280e49 100644 --- a/tools/lib/subcmd/Makefile +++ b/tools/lib/subcmd/Makefile @@ -76,7 +76,7 @@ include $(srctree)/tools/build/Makefile.include all: fixdep $(LIBFILE) -$(SUBCMD_IN): FORCE +$(SUBCMD_IN): fixdep FORCE @$(MAKE) $(build)=libsubcmd $(LIBFILE): $(SUBCMD_IN) ^ permalink raw reply related [flat|nested] 7+ messages in thread
* Re: possible dependency error? 2024-06-18 23:29 ` Brian Norris @ 2024-06-19 6:02 ` Masahiro Yamada 2024-07-02 0:37 ` Brian Norris 0 siblings, 1 reply; 7+ messages in thread From: Masahiro Yamada @ 2024-06-19 6:02 UTC (permalink / raw) To: Brian Norris; +Cc: Bjorn Helgaas, linux-kbuild, Josh Poimboeuf, Peter Zijlstra On Wed, Jun 19, 2024 at 8:29 AM Brian Norris <briannorris@chromium.org> wrote: > > Hi Masahiro, > > Thanks for the reply. I've been away for a bit, but I've poked at a few > more things now. > > On Sun, May 26, 2024 at 01:35:43AM +0900, Masahiro Yamada wrote: > > On Fri, May 24, 2024 at 2:54 AM Brian Norris <briannorris@chromium.org> wrote: > > > --- a/tools/lib/subcmd/Makefile > > > +++ b/tools/lib/subcmd/Makefile > > > @@ -76,7 +76,7 @@ include $(srctree)/tools/build/Makefile.include > > > > > > all: fixdep $(LIBFILE) > > > > > > -$(SUBCMD_IN): FORCE > > > +$(SUBCMD_IN): fixdep FORCE > > > @$(MAKE) $(build)=libsubcmd > > > > > > $(LIBFILE): $(SUBCMD_IN) > > > > > > > > I may not fully understand the design policy of the tools/ build system, > > but this fix is presumably correct because the 'fixdep' binary > > is needed in each sub-directory for it to work correctly. > > > > tools/bpf/resolve_btfids/libsubcmd/.exec-cmd.o.cmd must > > be generated by tools/bpf/resolve_btfids/libsubcmd/fixdep > > instead of by tools/bpf/resolve_btfids/fixdep. > > > > But, fixing tools/lib/subcmd/Makefile is not enough. > > > > *.cmd files under tools/bpf/resolve_btfids/libbpf/staticobjs/ > > are broken for the same reason. > > So, this is fundamentally broken in many places. > > I think I hacked something that works there too. It gets a bit uglier, > but not too bad IMO. > > > And, as you noted, there is no easy way to fix .fixdep.o.cmd > > I haven't come up with a *great* solution, but I came up with something > that works for the most part, by circumventing the normal Build / > Makefile.build split. It's getting pretty ugly too though... > > > Your description in > > https://issuetracker.google.com/issues/313508829#comment32 > > is all correct. > > > > > > "can we just use Kbuild?" is a good question. > > I do not understand why they use fragile build systems, > > where obviously they cannot do it correctly. > > > > > > In fact, I submitted a patch to migrate objtool to Kbuild > > because that fixes all the issues cleanly. > > > > The objtool maintainers rejected it. > > https://lore.kernel.org/linux-kbuild/1551764896-8453-3-git-send-email-yamada.masahiro@socionext.com/ > > > > > > Not only the build system. > > He also refused to participate in the standard Documentation > > directory. > > tools/objtool/Documentation/objtool.txt still resides in its own directory. > > While I don't love having to solve the same problems repeatedly (once in > Kbuild; potentially-many in tools/), I'm also OK with trying to hack > fixes into the current duplicate build system if it's not prohibitively > complex to do so. > > Here's what I have for now -- I might submit some or all of this as a > proper patchset if I can fix a few rough edges. I do not see any maintainer for tools/build/, but at least you can find who is picking up the patches. masahiro@zoe:~/ref/linux(master)$ ./scripts/get_maintainer.pl -f tools/build/ Arnaldo Carvalho de Melo <acme@redhat.com> (commit_signer:7/10=70%,authored:1/10=10%) Namhyung Kim <namhyung@kernel.org> (commit_signer:5/10=50%,authored:2/10=20%) Ian Rogers <irogers@google.com> (commit_signer:3/10=30%,authored:1/10=10%) Thomas Richter <tmricht@linux.ibm.com> (commit_signer:2/10=20%,authored:2/10=20%) Quentin Monnet <qmo@kernel.org> (commit_signer:2/10=20%) Jiri Olsa <jolsa@kernel.org> (authored:1/10=10%) linux-kernel@vger.kernel.org (open list) > diff --git a/tools/build/Makefile b/tools/build/Makefile > index 17cdf01e29a0..fad93f64035d 100644 > --- a/tools/build/Makefile > +++ b/tools/build/Makefile > @@ -43,11 +43,8 @@ ifneq ($(wildcard $(TMP_O)),) > $(Q)$(MAKE) -C feature OUTPUT=$(TMP_O) clean >/dev/null > endif > > -$(OUTPUT)fixdep-in.o: FORCE > - $(Q)$(MAKE) $(build)=fixdep > - > -$(OUTPUT)fixdep: $(OUTPUT)fixdep-in.o > - $(QUIET_LINK)$(HOSTCC) $(KBUILD_HOSTLDFLAGS) -o $@ $< > +$(OUTPUT)fixdep: $(srctree)/tools/build/fixdep.c > + $(QUIET_CC)$(HOSTCC) $(KBUILD_HOSTLDFLAGS) -o $@ $< OK, you bypassed fixdep for fixdep itself. fixdep will not be rebuilt when the command line changes, but it may not be a big deal. It is not working that way already. BTW, did you have a chance to test your code with the -j<N> option? I quickly tested your change, and I observed new "jobserver unavailable" warnings. masahiro@zoe:~/ref/linux(master)$ make -j24 mkdir -p /home/masahiro/ref/linux/tools/objtool && make O=/home/masahiro/ref/linux subdir=tools/objtool --no-print-directory -C objtool make[4]: warning: jobserver unavailable: using -j1. Add '+' to parent make rule. make[5]: warning: jobserver unavailable: using -j1. Add '+' to parent make rule. The first line: mkdir -p /home/masahiro/ref/linux/tools/objtool && make O=/home/masahiro/ref/linux subdir=tools/objtool --no-print-directory -C objtool is an existing noise regardless of your change. (I do not know if anybody cares about this either) > FORCE: > > diff --git a/tools/build/Makefile.include b/tools/build/Makefile.include > index 8dadaa0fbb43..27b2090cb293 100644 > --- a/tools/build/Makefile.include > +++ b/tools/build/Makefile.include > @@ -1,8 +1,16 @@ > # SPDX-License-Identifier: GPL-2.0-only > build := -f $(srctree)/tools/build/Makefile.build dir=. obj > > +# More than just $(Q), we sometimes want to suppress all command output from a > +# recursive make -- even the 'up to date' printout. > +ifeq ($(V),1) > + SILENT_MAKE = $(Q)$(MAKE) > +else > + SILENT_MAKE = $(Q)$(MAKE) --silent > +endif > + > fixdep: > - $(Q)$(MAKE) -C $(srctree)/tools/build CFLAGS= LDFLAGS= $(OUTPUT)fixdep > + $(SILENT_MAKE) -C $(srctree)/tools/build CFLAGS= LDFLAGS= $(OUTPUT)fixdep > > fixdep-clean: > $(Q)$(MAKE) -C $(srctree)/tools/build clean > diff --git a/tools/lib/bpf/Makefile b/tools/lib/bpf/Makefile > index 2cf892774346..a8f34de1ca25 100644 > --- a/tools/lib/bpf/Makefile > +++ b/tools/lib/bpf/Makefile > @@ -154,6 +154,8 @@ $(BPF_IN_SHARED): force $(BPF_GENERATED) > $(Q)$(MAKE) $(build)=libbpf OUTPUT=$(SHARED_OBJDIR) CFLAGS="$(CFLAGS) $(SHLIB_FLAGS)" > > $(BPF_IN_STATIC): force $(BPF_GENERATED) > + $(call rule_mkdir) > + $(SILENT_MAKE) -C $(srctree)/tools/build CFLAGS= LDFLAGS= OUTPUT=$(STATIC_OBJDIR) $(STATIC_OBJDIR)fixdep > $(Q)$(MAKE) $(build)=libbpf OUTPUT=$(STATIC_OBJDIR) > > $(BPF_HELPER_DEFS): $(srctree)/tools/include/uapi/linux/bpf.h > diff --git a/tools/lib/subcmd/Makefile b/tools/lib/subcmd/Makefile > index b87213263a5e..59b09f280e49 100644 > --- a/tools/lib/subcmd/Makefile > +++ b/tools/lib/subcmd/Makefile > @@ -76,7 +76,7 @@ include $(srctree)/tools/build/Makefile.include > > all: fixdep $(LIBFILE) > > -$(SUBCMD_IN): FORCE > +$(SUBCMD_IN): fixdep FORCE > @$(MAKE) $(build)=libsubcmd > > $(LIBFILE): $(SUBCMD_IN) > -- Best Regards Masahiro Yamada ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: possible dependency error? 2024-06-19 6:02 ` Masahiro Yamada @ 2024-07-02 0:37 ` Brian Norris 0 siblings, 0 replies; 7+ messages in thread From: Brian Norris @ 2024-07-02 0:37 UTC (permalink / raw) To: Masahiro Yamada Cc: Bjorn Helgaas, linux-kbuild, Josh Poimboeuf, Peter Zijlstra Hi, On Wed, Jun 19, 2024 at 03:02:35PM +0900, Masahiro Yamada wrote: > I do not see any maintainer for tools/build/, but at least > you can find who is picking up the patches. > > > masahiro@zoe:~/ref/linux(master)$ ./scripts/get_maintainer.pl -f tools/build/ Thanks. I've factored some of this into my addressee list. > OK, you bypassed fixdep for fixdep itself. > > fixdep will not be rebuilt when the command line changes, > but it may not be a big deal. > It is not working that way already. Yeah, that was one rough edge. I didn't manage to smooth that one out, but like you say, it's not really much of a regression. (It *can* work today sometimes for just the right kind of incremental build, because you may have a pre-existing fixdep laying around that will be used if available. But it's not really *correct* still.) > BTW, did you have a chance to test your code > with the -j<N> option? > > > I quickly tested your change, and I observed new > "jobserver unavailable" warnings. > > > masahiro@zoe:~/ref/linux(master)$ make -j24 > mkdir -p /home/masahiro/ref/linux/tools/objtool && make > O=/home/masahiro/ref/linux subdir=tools/objtool --no-print-directory > -C objtool > make[4]: warning: jobserver unavailable: using -j1. Add '+' to parent > make rule. > make[5]: warning: jobserver unavailable: using -j1. Add '+' to parent > make rule. > > > > The first line: > > mkdir -p /home/masahiro/ref/linux/tools/objtool && make > O=/home/masahiro/ref/linux subdir=tools/objtool --no-print-directory > -C objtool > > is an existing noise regardless of your change. > (I do not know if anybody cares about this either) Thanks. I tried to tackle the existing and new noise in my posting here: Subject: [PATCH 0/3] tools build: Incorrect fixdep dependencies https://lore.kernel.org/lkml/20240702003119.3641219-1-briannorris@chromium.org/ Hopefully that's not horribly off the mark, although I'll admit tools/ is a bit of unfamiliar territory. Brian ^ permalink raw reply [flat|nested] 7+ messages in thread
end of thread, other threads:[~2024-07-02 0:37 UTC | newest] Thread overview: 7+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2023-05-17 23:27 possible dependency error? Bjorn Helgaas 2023-05-18 8:26 ` Masahiro Yamada 2024-05-23 17:54 ` Brian Norris 2024-05-25 16:35 ` Masahiro Yamada 2024-06-18 23:29 ` Brian Norris 2024-06-19 6:02 ` Masahiro Yamada 2024-07-02 0:37 ` Brian Norris
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox