Re: Issue with "kbuild: create a list of all built DTB files"

[Florian Fainelli <florian.fainelli@broadcom.com>]

[-- Attachment #1: Type: text/plain, Size: 17554 bytes --]

On 4/17/24 11:37, Florian Fainelli wrote:
> On 4/17/24 07:31, Masahiro Yamada wrote:
>> On Wed, Apr 17, 2024 at 2:22 AM Florian Fainelli
>> <florian.fainelli@broadcom.com> wrote:
>>>
>>> Hi Masahiro,
>>>
>>> Sorry about the late report, commit
>>> 24507871c3c6ae4f6b460b016da7ff434cd34149 ("kbuild: create a list of all
>>> built DTB files") is breaking the build when we are doing a rsync of the
>>> Linux sources into a build directory, which is how buildroot works when
>>> doing a source directory override.
>>>
>>> This does not happen when doing a build from the git directory
>>> containing the sources, and I cannot find a missing $(obj) reference, so
>>> I am left wondering what I am missing here. The build error looks 
>>> like this:
>>>
>>> cat: arch/arm64/boot/dts/actions/dtbs-list: No such file or directory
>>> host-make[3]: *** [scripts/Makefile.build:423:
>>> arch/arm64/boot/dts/dtbs-list] Error 1
>>> host-make[3]: *** Deleting file 'arch/arm64/boot/dts/dtbs-list'
>>> host-make[2]: *** [Makefile:1394: dtbs] Error 2
>>> host-make[2]: *** Waiting for unfinished jobs....
>>>
>>> running with verbose, we can see the rule not generating an empty file
>>> at all:
>>>
>>> /home/fainelli/work/buildroot/output/arm64/host/bin/host-make -f
>>> ./scripts/Makefile.build obj=scripts/dtc
>>> /home/fainelli/work/buildroot/output/arm64/host/bin/host-make -f
>>> ./scripts/Makefile.build obj=scripts
>>> /home/fainelli/work/buildroot/output/arm64/host/bin/host-make -f
>>> ./scripts/Makefile.build obj=arch/arm64/boot/dts need-dtbslist=1
>>> /home/fainelli/work/buildroot/output/arm64/host/bin/host-make -f
>>> ./scripts/Makefile.build obj=scripts/genksyms \
>>> need-builtin= \
>>> need-modorder= \
>>>
>>> /home/fainelli/work/buildroot/output/arm64/host/bin/host-make -f
>>> ./scripts/Makefile.build obj=arch/arm64/boot/dts/actions \
>>> need-builtin= \
>>> need-modorder= \
>>>
>>> /home/fainelli/work/buildroot/output/arm64/host/bin/host-make -f
>>> ./scripts/Makefile.build obj=arch/arm64/boot/dts/allwinner \
>>> need-builtin= \
>>> need-modorder= \
>>>
>>> /home/fainelli/work/buildroot/output/arm64/host/bin/host-make -f
>>> ./scripts/Makefile.build obj=arch/arm64/boot/dts/altera \
>>> need-builtin= \
>>> need-modorder= \
>>>
>>> /home/fainelli/work/buildroot/output/arm64/host/bin/host-make -f
>>> ./scripts/Makefile.build obj=arch/arm64/boot/dts/amazon \
>>> need-builtin= \
>>> need-modorder= \
>>>
>>> /home/fainelli/work/buildroot/output/arm64/host/bin/host-make -f
>>> ./scripts/Makefile.build obj=arch/arm64/tools kapi
>>> /home/fainelli/work/buildroot/output/arm64/host/bin/host-make -f
>>> ./scripts/Makefile.build obj=arch/arm64/boot/dts/amd \
>>> need-builtin= \
>>> need-modorder= \
>>>
>>> /home/fainelli/work/buildroot/output/arm64/host/bin/host-make -f
>>> ./scripts/Makefile.build obj=arch/arm64/boot/dts/amlogic \
>>> need-builtin= \
>>> need-modorder= \
>>>
>>> /home/fainelli/work/buildroot/output/arm64/host/bin/host-make -f
>>> ./scripts/Makefile.build obj=arch/arm64/boot/dts/apm \
>>> need-builtin= \
>>> need-modorder= \
>>>
>>> /home/fainelli/work/buildroot/output/arm64/host/bin/host-make -f
>>> ./scripts/Makefile.build obj=arch/arm64/boot/dts/apple \
>>> need-builtin= \
>>> need-modorder= \
>>>
>>> /home/fainelli/work/buildroot/output/arm64/host/bin/host-make -f
>>> ./scripts/Makefile.build obj=arch/arm64/boot/dts/arm \
>>> need-builtin= \
>>> need-modorder= \
>>>
>>> /home/fainelli/work/buildroot/output/arm64/host/bin/host-make -f
>>> ./scripts/Makefile.build obj=arch/arm64/boot/dts/bitmain \
>>> need-builtin= \
>>> need-modorder= \
>>>
>>> /home/fainelli/work/buildroot/output/arm64/host/bin/host-make -f
>>> ./scripts/Makefile.build obj=arch/arm64/boot/dts/broadcom \
>>> need-builtin= \
>>> need-modorder= \
>>>
>>> /home/fainelli/work/buildroot/output/arm64/host/bin/host-make -f
>>> ./scripts/Makefile.build obj=arch/arm64/boot/dts/cavium \
>>> need-builtin= \
>>> need-modorder= \
>>>
>>> /home/fainelli/work/buildroot/output/arm64/host/bin/host-make -f
>>> ./scripts/Makefile.build obj=arch/arm64/boot/dts/exynos \
>>> need-builtin= \
>>> need-modorder= \
>>>
>>> /home/fainelli/work/buildroot/output/arm64/host/bin/host-make -f
>>> ./scripts/Makefile.build obj=arch/arm64/boot/dts/freescale \
>>> need-builtin= \
>>> need-modorder= \
>>>
>>> /home/fainelli/work/buildroot/output/arm64/host/bin/host-make -f
>>> ./scripts/Makefile.build obj=arch/arm64/boot/dts/hisilicon \
>>> need-builtin= \
>>> need-modorder= \
>>>
>>> /home/fainelli/work/buildroot/output/arm64/host/bin/host-make -f
>>> ./scripts/Makefile.build obj=arch/arm64/boot/dts/intel \
>>> need-builtin= \
>>> need-modorder= \
>>>
>>> /home/fainelli/work/buildroot/output/arm64/host/bin/host-make -f
>>> ./scripts/Makefile.build obj=arch/arm64/boot/dts/lg \
>>> need-builtin= \
>>> need-modorder= \
>>>
>>> # GEN     arch/arm64/include/generated/asm/cpucap-defs.h
>>>     mkdir -p arch/arm64/include/generated/asm/; awk -f
>>> arch/arm64/tools/gen-cpucaps.awk arch/arm64/tools/cpucaps >
>>> arch/arm64/include/generated/asm/cpucap-defs.h
>>> /home/fainelli/work/buildroot/output/arm64/host/bin/host-make -f
>>> ./scripts/Makefile.build obj=arch/arm64/boot/dts/marvell \
>>> need-builtin= \
>>> need-modorder= \
>>>
>>> /home/fainelli/work/buildroot/output/arm64/host/bin/host-make -f
>>> ./scripts/Makefile.build obj=arch/arm64/boot/dts/mediatek \
>>> need-builtin= \
>>> need-modorder= \
>>>
>>> # GEN     arch/arm64/include/generated/asm/sysreg-defs.h
>>>     mkdir -p arch/arm64/include/generated/asm/; awk -f
>>> arch/arm64/tools/gen-sysreg.awk arch/arm64/tools/sysreg >
>>> arch/arm64/include/generated/asm/sysreg-defs.h
>>> /home/fainelli/work/buildroot/output/arm64/host/bin/host-make -f
>>> ./scripts/Makefile.build obj=arch/arm64/boot/dts/microchip \
>>> need-builtin= \
>>> need-modorder= \
>>>
>>> /home/fainelli/work/buildroot/output/arm64/host/bin/host-make -f
>>> ./scripts/Makefile.build obj=arch/arm64/boot/dts/broadcom/bcmbca \
>>> need-builtin= \
>>> need-modorder= \
>>>
>>> /home/fainelli/work/buildroot/output/arm64/host/bin/host-make -f
>>> ./scripts/Makefile.build obj=arch/arm64/boot/dts/nuvoton \
>>> need-builtin= \
>>> need-modorder= \
>>>
>>> /home/fainelli/work/buildroot/output/arm64/host/bin/host-make -f
>>> ./scripts/Makefile.build obj=arch/arm64/boot/dts/broadcom/northstar2 \
>>> need-builtin= \
>>> need-modorder= \
>>>
>>> /home/fainelli/work/buildroot/output/arm64/host/bin/host-make -f
>>> ./scripts/Makefile.build obj=arch/arm64/boot/dts/exynos/google \
>>> need-builtin= \
>>> need-modorder= \
>>>
>>> /home/fainelli/work/buildroot/output/arm64/host/bin/host-make -f
>>> ./scripts/Makefile.build obj=arch/arm64/boot/dts/nvidia \
>>> need-builtin= \
>>> need-modorder= \
>>>
>>> /home/fainelli/work/buildroot/output/arm64/host/bin/host-make -f
>>> ./scripts/Makefile.build obj=arch/arm64/boot/dts/broadcom/stingray \
>>> need-builtin= \
>>> need-modorder= \
>>>
>>> /home/fainelli/work/buildroot/output/arm64/host/bin/host-make -f
>>> ./scripts/Makefile.build obj=arch/arm64/boot/dts/qcom \
>>> need-builtin= \
>>> need-modorder= \
>>>
>>> /home/fainelli/work/buildroot/output/arm64/host/bin/host-make -f
>>> ./scripts/Makefile.build obj=arch/arm64/boot/dts/realtek \
>>> need-builtin= \
>>> need-modorder= \
>>>
>>> /home/fainelli/work/buildroot/output/arm64/host/bin/host-make -f
>>> ./scripts/Makefile.build obj=arch/arm64/boot/dts/renesas \
>>> need-builtin= \
>>> need-modorder= \
>>>
>>> /home/fainelli/work/buildroot/output/arm64/host/bin/host-make -f
>>> ./scripts/Makefile.build obj=arch/arm64/boot/dts/rockchip \
>>> need-builtin= \
>>> need-modorder= \
>>>
>>> /home/fainelli/work/buildroot/output/arm64/host/bin/host-make -f
>>> ./scripts/Makefile.build obj=arch/arm64/boot/dts/socionext \
>>> need-builtin= \
>>> need-modorder= \
>>>
>>> /home/fainelli/work/buildroot/output/arm64/host/bin/host-make -f
>>> ./scripts/Makefile.build obj=arch/arm64/boot/dts/sprd \
>>> need-builtin= \
>>> need-modorder= \
>>>
>>> /home/fainelli/work/buildroot/output/arm64/host/bin/host-make -f
>>> ./scripts/Makefile.build obj=arch/arm64/boot/dts/st \
>>> need-builtin= \
>>> need-modorder= \
>>>
>>> /home/fainelli/work/buildroot/output/arm64/host/bin/host-make -f
>>> ./scripts/Makefile.build obj=arch/arm64/boot/dts/synaptics \
>>> need-builtin= \
>>> need-modorder= \
>>>
>>> /home/fainelli/work/buildroot/output/arm64/host/bin/host-make -f
>>> ./scripts/Makefile.build obj=arch/arm64/boot/dts/tesla \
>>> need-builtin= \
>>> need-modorder= \
>>>
>>> /home/fainelli/work/buildroot/output/arm64/host/bin/host-make -f
>>> ./scripts/Makefile.build obj=arch/arm64/boot/dts/ti \
>>> need-builtin= \
>>> need-modorder= \
>>>
>>> /home/fainelli/work/buildroot/output/arm64/host/bin/host-make -f
>>> ./scripts/Makefile.build obj=arch/arm64/boot/dts/toshiba \
>>> need-builtin= \
>>> need-modorder= \
>>>
>>> /home/fainelli/work/buildroot/output/arm64/host/bin/host-make -f
>>> ./scripts/Makefile.build obj=arch/arm64/boot/dts/xilinx \
>>> need-builtin= \
>>> need-modorder= \
>>>
>>> # cmd_gen_order arch/arm64/boot/dts/dtbs-list
>>>     {   cat arch/arm64/boot/dts/actions/dtbs-list;   cat
>>> arch/arm64/boot/dts/allwinner/dtbs-list;   cat
>>> arch/arm64/boot/dts/altera/dtbs-list;   cat
>>> arch/arm64/boot/dts/amazon/dtbs-list;   cat
>>> arch/arm64/boot/dts/amd/dtbs-list;   cat
>>> arch/arm64/boot/dts/amlogic/dtbs-list;   cat
>>> arch/arm64/boot/dts/apm/dtbs-list;   cat
>>> arch/arm64/boot/dts/apple/dtbs-list;   cat
>>> arch/arm64/boot/dts/arm/dtbs-list;   cat
>>> arch/arm64/boot/dts/bitmain/dtbs-list;   cat
>>> arch/arm64/boot/dts/broadcom/dtbs-list;   cat
>>> arch/arm64/boot/dts/cavium/dtbs-list;   cat
>>> arch/arm64/boot/dts/exynos/dtbs-list;   cat
>>> arch/arm64/boot/dts/freescale/dtbs-list;   cat
>>> arch/arm64/boot/dts/hisilicon/dtbs-list;   cat
>>> arch/arm64/boot/dts/intel/dtbs-list;   cat
>>> arch/arm64/boot/dts/lg/dtbs-list;   cat
>>> arch/arm64/boot/dts/marvell/dtbs-list;   cat
>>> arch/arm64/boot/dts/mediatek/dtbs-list;   cat
>>> arch/arm64/boot/dts/microchip/dtbs-list;   cat
>>> arch/arm64/boot/dts/nuvoton/dtbs-list;   cat
>>> arch/arm64/boot/dts/nvidia/dtbs-list;   cat
>>> arch/arm64/boot/dts/qcom/dtbs-list;   cat
>>> arch/arm64/boot/dts/realtek/dtbs-list;   cat
>>> arch/arm64/boot/dts/renesas/dtbs-list;   cat
>>> arch/arm64/boot/dts/rockchip/dtbs-list;   cat
>>> arch/arm64/boot/dts/socionext/dtbs-list;   cat
>>> arch/arm64/boot/dts/sprd/dtbs-list;   cat
>>> arch/arm64/boot/dts/st/dtbs-list;   cat
>>> arch/arm64/boot/dts/synaptics/dtbs-list;   cat
>>> arch/arm64/boot/dts/tesla/dtbs-list;   cat
>>> arch/arm64/boot/dts/ti/dtbs-list;   cat
>>> arch/arm64/boot/dts/toshiba/dtbs-list;   cat
>>> arch/arm64/boot/dts/xilinx/dtbs-list; :; } > 
>>> arch/arm64/boot/dts/dtbs-list
>>> cat: arch/arm64/boot/dts/actions/dtbs-list: No such file or directory
>>> host-make[3]: *** [scripts/Makefile.build:423:
>>> arch/arm64/boot/dts/dtbs-list] Error 1
>>> host-make[3]: *** Deleting file 'arch/arm64/boot/dts/dtbs-list'
>>> host-make[2]: *** [Makefile:1394: dtbs] Error 2
>>> host-make[2]: *** Waiting for unfinished jobs....
>>> make[1]: *** [package/pkg-generic.mk:293:
>>> /home/fainelli/work/buildroot/output/arm64/build/linux-custom/.stamp_built]
>>> Error 2
>>> make: *** [Makefile:27: _all] Error 2
>>>
>>> Buildroot builds its own GNU Make version 4.4.1.
>>>
>>> Any clues what might be going on? My defconfig is such that only
>>> CONFIG_ARCH_BRCMSTB is enabled, and there are essentially no .dtbs file
>>> that will be generated.
>>
>>
>>
>> How to reproduce this in buildroot?
> 
> It is a bit involved since it would require having some intermediate 
> builds and as I was crafting an environment for you to take a look, it 
> stopped being reproducible...  I will let you know if this resurfaces 
> again.

Adding Markus since he helped me with a similar issue before that showed 
up entirely differently around last year. I had an alias that basically 
does the following to build a kernel using buildroot:

make DESTDIR=/path/to/tftp/$KVER linux-reconfigure all install -j48

down the road in buildroot this ends-up building the kernel. The first 
time I had a problem this is what happened:

ff944844@stbirv-lnx-2:/local/users/fainelli/buildroot/output/arm64$ make V=1
test -d images || mkdir images
test -r images/rootfs.cpio || touch images/rootfs.cpio
umask 0022 && make -C /local/users/fainelli/buildroot 
O=/local/users/fainelli/buildroot/output/arm64/.
 >>> host-skeleton  Executing pre-build script board/brcmstb/pre-build.sh
 >>> linux custom Building
board/brcmstb/linux-pre-build.sh
Removing previous kernel modules
/bin/sed -i -e '/^\(# \)\?CONFIG_GCC_PLUGINS\>/d' 
/local/users/fainelli/buildroot/output/arm64/build/linux-custom//.config 
&& echo '# CONFIG_GCC_PLUGINS is not set' >> 
/local/users/fainelli/buildroot/output/arm64/build/linux-custom//.config
PATH="/local/users/fainelli/buildroot/output/arm64/host/bin:/local/users/fainelli/buildroot/output/arm64/host/sbin:/home/ff944844/bin:/home/ff944844/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/opt/dell/srvadmin/bin" 
PKG_CONFIG="/local/users/fainelli/buildroot/output/arm64/host/bin/pkg-config" 
PKG_CONFIG_SYSROOT_DIR="/" PKG_CONFIG_ALLOW_SYSTEM_CFLAGS=1 
PKG_CONFIG_ALLOW_SYSTEM_LIBS=1 
PKG_CONFIG_LIBDIR="/local/users/fainelli/buildroot/output/arm64/host/lib/pkgconfig:/local/users/fainelli/buildroot/output/arm64/host/share/pkgconfig" 
BR_BINARIES_DIR=/local/users/fainelli/buildroot/output/arm64/images 
KCFLAGS=-Wno-attribute-alias 
/local/users/fainelli/buildroot/output/arm64/host/bin/host-make -j33 
HOSTCC="/local/users/fainelli/buildroot/output/arm64/host/bin/ccache 
/usr/bin/gcc -O2 -isystem 
/local/users/fainelli/buildroot/output/arm64/host/include -DNDEBUG 
-L/local/users/fainelli/buildroot/output/arm64/host/lib 
-Wl,-rpath,/local/users/fainelli/buildroot/output/arm64/host/lib" 
ARCH=arm64 
INSTALL_MOD_PATH=/local/users/fainelli/buildroot/output/arm64/target 
CROSS_COMPILE="/local/users/fainelli/buildroot/output/arm64/host/bin/aarch64-linux-" 
WERROR=0 REGENERATE_PARSERS=1 
DEPMOD=/local/users/fainelli/buildroot/output/arm64/host/sbin/depmod 
INSTALL_MOD_STRIP=1 -C 
/local/users/fainelli/buildroot/output/arm64/build/linux-custom all
/local/users/fainelli/buildroot/output/arm64/host/bin/host-make -f 
./Makefile syncconfig
/local/users/fainelli/buildroot/output/arm64/host/bin/host-make -f 
./scripts/Makefile.build obj=scripts/basic
rm -f .tmp_quiet_recordmcount
/local/users/fainelli/buildroot/output/arm64/host/bin/host-make -f 
./scripts/Makefile.build obj=scripts/kconfig syncconfig
scripts/kconfig/conf  --syncconfig Kconfig
/local/users/fainelli/buildroot/output/arm64/host/bin/host-make -f 
./Makefile syncconfig
/local/users/fainelli/buildroot/output/arm64/host/bin/host-make -f 
./scripts/Makefile.build obj=scripts/basic
rm -f .tmp_quiet_recordmcount
/local/users/fainelli/buildroot/output/arm64/host/bin/host-make -f 
./scripts/Makefile.build obj=scripts/kconfig syncconfig
scripts/kconfig/conf  --syncconfig Kconfig
/local/users/fainelli/buildroot/output/arm64/host/bin/host-make -f 
./Makefile syncconfig
/local/users/fainelli/buildroot/output/arm64/host/bin/host-make -f 
./scripts/Makefile.build obj=scripts/basic
rm -f .tmp_quiet_recordmcount
/local/users/fainelli/buildroot/output/arm64/host/bin/host-make -f 
./scripts/Makefile.build obj=scripts/kconfig syncconfig
scripts/kconfig/conf  --syncconfig Kconfig
/local/users/fainelli/buildroot/output/arm64/host/bin/host-make -f 
./Makefile syncconfig
/local/users/fainelli/buildroot/output/arm64/host/bin/host-make -f 
./scripts/Makefile.build obj=scripts/basic
rm -f .tmp_quiet_recordmcount
/local/users/fainelli/buildroot/output/arm64/host/bin/host-make -f 
./scripts/Makefile.build obj=scripts/kconfig syncconfig
scripts/kconfig/conf  --syncconfig Kconfig

we would basically end-up in an endless syncconfig make recipe and there 
would be no forward progress. Markus had done a great job at identifying 
that the problem, and he provided the following:

* The problem DOES happen with GNU make 4.4.1 and Linux 5.4 (and,
presumably, older kernels)
* The problem does NOT happen with GNU make 4.4 or older, irrespective
of the kernel version
* The problem also does NOT happen with Linux 5.15 (and, presumably
newer), irrespective of the GNU make version

So, this would seem to be some feature / change that was introduced in
GNU make that the kernel build system subsequently accounted for. Or
it could be independent changes where the kernel build system changed
first and GNU make changed later; 4.4.1 was only tagged on February
26, 2023.

And eventually he tracked it down to two commit of you, one in GNU Make:

https://git.savannah.gnu.org/cgit/make.git/commit/?id=a275f4e9ab14

and an innocent Linux change that ended up "fixing" the issue:

https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=77342a02ff6e

Sure enough, the machine on which I had the issue I reported to you with 
the dtbs-list being empty still used the old alias for building a kernel.

I would like to give you a minimal reproducer, because it seems like 
there is a genuine issue that just shows up in unexpected ways here. The 
key thing is really specifying a DESTDIR variable on the make command 
line of buildroot, which eventually makes it to Linux's build system and 
breaks in unexpected ways.

Thanks!
-- 
Florian


[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 4221 bytes --]