[Buildroot] [Bug 13141] New: Target-finalize fail with "depmod: ERROR: Bad version passed"

[Buildroot] [Bug 13141] New: Target-finalize fail with "depmod: ERROR: Bad version passed"

[bugzilla at busybox.net]

https://bugs.busybox.net/show_bug.cgi?id=13141

            Bug ID: 13141
           Summary: Target-finalize fail with "depmod: ERROR: Bad version
                    passed"
           Product: buildroot
           Version: 2020.02.1
          Hardware: Other
                OS: Linux
            Status: NEW
          Severity: normal
          Priority: P5
         Component: Other
          Assignee: unassigned at buildroot.uclibc.org
          Reporter: laurent at neko-labs.eu
                CC: buildroot at uclibc.org
  Target Milestone: ---

Hello,

I've successfully build my system up to 2020.02.

But when I started to try to build 2020.02.1 and onwards it always failed with
"depmod: ERROR: Bad version passed" message during TARGET_FINALIZE step.

I managed to find the commit that make my build fail:

8f5a610fe81676092a6960415717de731f43f8bf
linux: Allow modprobe for out-of-tree modules

When LINUX_TARGET_FINALIZE_HOOKS are call the new LINUX_RUN_DEPMOD function
fail in my case.

# Run depmod in a target-finalize hook, to encompass modules installed by
# packages.
define LINUX_RUN_DEPMOD
        $(HOST_DIR)/sbin/depmod -a -b $(TARGET_DIR) $(LINUX_VERSION_PROBED)
endef
LINUX_TARGET_FINALIZE_HOOKS += LINUX_RUN_DEPMOD

Here is my $(LINUX_VERSION_PROBED): 4.19.78-linux4sam-6.2 

I'm using a buildroot external tree for my board, still I've attached my
.config file if it can help

What I don't understand is that if I run manually the depmod command it pass
successfully :/

---> builroot log:
/home/buildroot/output/host/sbin/depmod -a -b /home/buildroot/output/target
`/usr/bin/make  HOSTCC="/home/buildroot/output/host/bin/ccache /usr/bin/gcc -O2
-I/home/buildroot/output/host/include -L/home/buildroot/output/host/lib
-Wl,-rpath,/home/buildroot/output/host/lib" ARCH=arm
INSTALL_MOD_PATH=/home/buildroot/output/target
CROSS_COMPILE="/home/buildroot/output/host/bin/arm-buildroot-linux-gnueabihf-"
DEPMOD=/home/buildroot/output/host/sbin/depmod INSTALL_MOD_STRIP=1 -C
/home/buildroot/output/build/linux-linux4sam_6.2 --no-print-directory -s
kernelrelease 2>/dev/null`

depmod: ERROR: Bad version passed make[1]:

make: *** [Makefile:747: target-finalize] Error 1


---> My manual commands (that succeed):
#/usr/bin/make  HOSTCC="/home/buildroot/output/host/bin/ccache /usr/bin/gcc -O2
-I/home/buildroot/output/host/include -L/home/buildroot/output/host/lib
-Wl,-rpath,/home/buildroot/output/host/lib" ARCH=arm
INSTALL_MOD_PATH=/home/buildroot/output/target
CROSS_COMPILE="/home/buildroot/output/host/bin/arm-buildroot-linux-gnueabihf-"
DEPMOD=/home/buildroot/output/host/sbin/depmod INSTALL_MOD_STRIP=1 -C
/home/buildroot/output/build/linux-linux4sam_6.2 --no-print-directory -s
kernelrelease 2>/dev/null

4.19.78-linux4sam-6.2

#/home/buildroot/output/host/sbin/depmod -a -b /home/buildroot/output/target
4.19.78-linux4sam-6.2

OK

Any help appreciated!

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Buildroot] [Bug 13141] Target-finalize fail with "depmod: ERROR: Bad version passed"

[bugzilla at busybox.net]

https://bugs.busybox.net/show_bug.cgi?id=13141

Yann E. MORIN <yann.morin.1998@free.fr> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |yann.morin.1998 at free.fr

--- Comment #1 from Yann E. MORIN <yann.morin.1998@free.fr> ---
Laurent, All,

Thanks for the report.

Could you try a few things for me, please:
 1- run:  make V=1
 2- in the LINUX_RUN_DEPMOD macro, add this as the first command:
        printf '"%s"\n' "$(LINUX_VERSION_PROBED)"
 3- try a build without ccache

Also, I noticed something weird in your report:


--> My manual commands (that succeed):
#/usr/bin/make  HOSTCC="/home/buildroot/output/host/bin/ccache /usr/bin/gcc -O2
-I/home/buildroot/output/host/include -L/home/buildroot/output/host/lib
-Wl,-rpath,/home/buildroot/output/host/lib" ARCH=arm
INSTALL_MOD_PATH=/home/buildroot/output/target
CROSS_COMPILE="/home/buildroot/output/host/bin/arm-buildroot-linux-gnueabihf-"
DEPMOD=/home/buildroot/output/host/sbin/depmod INSTALL_MOD_STRIP=1 -C
/home/buildroot/output/build/linux-linux4sam_6.2 --no-print-directory -s
kernelrelease 2>/dev/null

4.19.78-linux4sam-6.2


There is an empty line between the command and the version string. Is it
really what happens, or is it an artefact of your copy-paste?

Regards,
Yann E. MORIN.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Buildroot] [Bug 13141] Target-finalize fail with "depmod: ERROR: Bad version passed"

[bugzilla at busybox.net]

https://bugs.busybox.net/show_bug.cgi?id=13141

--- Comment #2 from Asaf Kahlon <asafka7@gmail.com> ---
(In reply to laurent from comment #0)

I encountered the same error few weeks ago.
I don't know if we're having the same case, but here are the results of my
little research:

1. I didn't run the 'make' command from Buildroot, I had something like: "make
-j8 -C buildroot/". If I run the `make` command directly from Buildroot, it
works fine.
* It always happens with compiling in parallel, doesn't matter whether it's -j2
or -j100. When I compiled with just make, it worked.
* It fails only on the first time! After the error occurred, I run "make -j8 -C
buildroot/" *again* and it worked.

Are those points relevant to your case?

Regards,
Asaf.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Buildroot] [Bug 13141] Target-finalize fail with "depmod: ERROR: Bad version passed"

[bugzilla at busybox.net]

https://bugs.busybox.net/show_bug.cgi?id=13141

--- Comment #3 from laurent at neko-labs.eu ---
Hi yann,

> make V=1:
if grep -q "CONFIG_MODULES=y"
/home/buildroot/output/build/linux-linux4sam_6.2/.config; then
/home/buildroot/output/host/sbin/depmod -a -b /home/buildroot/output/target
`/usr/bin/make  HOSTCC="/home/buildroot/output/host/bin/ccache /usr/bin/gcc -O2
-I/home/buildroot/output/host/include -L/home/buildroot/output/host/lib
-Wl,-rpath,/home/buildroot/output/host/lib" ARCH=arm
INSTALL_MOD_PATH=/home/buildroot/output/target
CROSS_COMPILE="/home/buildroot/output/host/bin/arm-buildroot-linux-gnueabihf-"
DEPMOD=/home/buildroot/output/host/sbin/depmod INSTALL_MOD_STRIP=1 -C
/home/buildroot/output/build/linux-linux4sam_6.2 --no-print-directory -s
kernelrelease 2>/dev/null`; fi
depmod: ERROR: Bad version passed make[1]:
make: *** [Makefile:747: target-finalize] Error 1
make : on quitte le r?pertoire ? /home/buildroot ?
make : on entre dans le r?pertoire ? /home/buildroot ?

> Adding printf 'probed linux version:"%s"\n'
>
># Run depmod in a target-finalize hook, to encompass modules installed by
># packages.
>define LINUX_RUN_DEPMOD
>        printf 'probed linux version:"%s"\n' "$(LINUX_VERSION_PROBED)"
>        if grep -q "CONFIG_MODULES=y" $(LINUX_DIR)/.config; then \
>                $(HOST_DIR)/sbin/depmod -a -b $(TARGET_DIR) >$(LINUX_VERSION_PROBED); \
>        fi
>endef
>LINUX_TARGET_FINALIZE_HOOKS += LINUX_RUN_DEPMOD

printf 'probed linux version:"%s"\n' "`/usr/bin/make 
HOSTCC="/home/buildroot/output/host/bin/ccache /usr/bin/gcc -O2
-I/home/buildroot/output/host/include -L/home/buildroot/output/host/lib
-Wl,-rpath,/home/buildroot/output/host/lib" ARCH=arm
INSTALL_MOD_PATH=/home/buildroot/output/target
CROSS_COMPILE="/home/buildroot/output/host/bin/arm-buildroot-linux-gnueabihf-"
DEPMOD=/home/buildroot/output/host/sbin/depmod INSTALL_MOD_STRIP=1 -C
/home/buildroot/output/build/linux-linux4sam_6.2 --no-print-directory -s
kernelrelease 2>/dev/null`"
probed linux version:"make[1]: Entering directory '/home/buildroot'
4.19.78-linux4sam-6.2
make[1]: Leaving directory '/home/buildroot/output/build/linux-linux4sam_6.2'"
if grep -q "CONFIG_MODULES=y"
/home/buildroot/output/build/linux-linux4sam_6.2/.config; then
/home/buildroot/output/host/sbin/depmod -a -b /home/buildroot/output/target
`/usr/bin/make  HOSTCC="/home/buildroot/output/host/bin/ccache /usr/bin/gcc -O2
-I/home/buildroot/output/host/include -L/home/buildroot/output/host/lib
-Wl,-rpath,/home/buildroot/output/host/lib" ARCH=arm
INSTALL_MOD_PATH=/home/buildroot/output/target
CROSS_COMPILE="/home/buildroot/output/host/bin/arm-buildroot-linux-gnueabihf-"
DEPMOD=/home/buildroot/output/host/sbin/depmod INSTALL_MOD_STRIP=1 -C
/home/buildroot/output/build/linux-linux4sam_6.2 --no-print-directory -s
kernelrelease 2>/dev/null`; fi
depmod: ERROR: Bad version passed make[1]:
make: *** [Makefile:747: target-finalize] Error 1
make?: on quitte le r?pertoire ??/home/buildroot??
make?: on entre dans le r?pertoire ??/home/buildroot??

>Removing ccache usage
printf 'probed linux version:"%s"\n' "`/usr/bin/make  HOSTCC="/usr/bin/gcc -O2
-I/home/buildroot/output/host/include -L/home/buildroot/output/host/lib
-Wl,-rpath,/home/buildroot/output/host/lib" ARCH=arm
INSTALL_MOD_PATH=/home/buildroot/output/target
CROSS_COMPILE="/home/buildroot/output/host/bin/arm-buildroot-linux-gnueabihf-"
DEPMOD=/home/buildroot/output/host/sbin/depmod INSTALL_MOD_STRIP=1 -C
/home/buildroot/output/build/linux-linux4sam_6.2 --no-print-directory -s
kernelrelease 2>/dev/null`"
probed linux version:"make[1]: Entering directory '/home/buildroot'
4.19.78-linux4sam-6.2
make[1]: Leaving directory '/home/buildroot/output/build/linux-linux4sam_6.2'"
if grep -q "CONFIG_MODULES=y"
/home/buildroot/output/build/linux-linux4sam_6.2/.config; then
/home/buildroot/output/host/sbin/depmod -a -b /home/buildroot/output/target
`/usr/bin/make  HOSTCC="/usr/bin/gcc -O2 -I/home/buildroot/output/host/include
-L/home/buildroot/output/host/lib -Wl,-rpath,/home/buildroot/output/host/lib"
ARCH=arm INSTALL_MOD_PATH=/home/buildroot/output/target
CROSS_COMPILE="/home/buildroot/output/host/bin/arm-buildroot-linux-gnueabihf-"
DEPMOD=/home/buildroot/output/host/sbin/depmod INSTALL_MOD_STRIP=1 -C
/home/buildroot/output/build/linux-linux4sam_6.2 --no-print-directory -s
kernelrelease 2>/dev/null`; fi
depmod: ERROR: Bad version passed make[1]:
make: *** [Makefile:747: target-finalize] Error 1


For the empty line, I added it to empathize the result, it was not there, you
are right.

Laurent

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Buildroot] [Bug 13141] Target-finalize fail with "depmod: ERROR: Bad version passed"

[bugzilla at busybox.net]

https://bugs.busybox.net/show_bug.cgi?id=13141

--- Comment #4 from laurent at neko-labs.eu ---
(In reply to Asaf Kahlon from comment #2)
Asaf, Yann,

1/As suggested by asaf I checked my build script and I had:
make -C $BR_SOURCE O=$BR_BUILD -j`nproc` V=1

Removing -j`nproc` seems to remove the issue.

--> I did the test during the night, I'll recheck tonight to be sure.

2/Asaf you are right first time I build it failed (with -j`nproc`), then next
time it succeed, but I was suspecting that buildroot somehow .stamp'ed the
TARGET_FINALIZE stage.

Laurent

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Buildroot] [Bug 13141] Target-finalize fail with "depmod: ERROR: Bad version passed"

[bugzilla at busybox.net]

https://bugs.busybox.net/show_bug.cgi?id=13141

--- Comment #5 from Yann E. MORIN <yann.morin.1998@free.fr> ---
Laurent, All,

> printf 'probed linux version:"%s"\n' "`/usr/bin/make  HOSTCC="/home/buildroot/output/host/bin/ccache /usr/bin/gcc -O2 -I/home/buildroot/output/host/include -L/home/buildroot/output/host/lib -Wl,-rpath,/home/buildroot/output/host/lib" ARCH=arm INSTALL_MOD_PATH=/home/buildroot/output/target CROSS_COMPILE="/home/buildroot/output/host/bin/arm-buildroot-linux-gnueabihf-" DEPMOD=/home/buildroot/output/host/sbin/depmod INSTALL_MOD_STRIP=1 -C /home/buildroot/output/build/linux-linux4sam_6.2 --no-print-directory -s kernelrelease 2>/dev/null`"
> probed linux version:"make[1]: Entering directory '/home/buildroot'
> 4.19.78-linux4sam-6.2
> make[1]: Leaving directory '/home/buildroot/output/build/linux-linux4sam_6.2'"

OK, so the bug is already visible here : see how the output contains make
messages around
the version value.

However, this is very strange, because:

1. we're telling make to not print those emssages: --no-print-directory -s
2. it is entering and leaving different directoriee: Entering directory
'/home/buildroot'
   but: Leaving directory '/home/buildroot/output/build/linux-linux4sam_6.2'

So I suspect a weird corner-case bug in make... 

Oh, by the way, top-level parallel build is not 100% supported, though, and at
the
very least, requires that you enable BR2_PER_PACKAGE_DIRECTORIES.

Regards,
Yann E. MORIN.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Buildroot] [Bug 13141] Target-finalize fail with "depmod: ERROR: Bad version passed"

[bugzilla at busybox.net]

https://bugs.busybox.net/show_bug.cgi?id=13141

--- Comment #6 from Yann E. MORIN <yann.morin.1998@free.fr> ---
Laurent, Asaf, All,

So I was initially not able to reproduce the issue, because my umask was
0002, which is not the umask Buildroot expects to work with. In this
situation, Buildroot respawns itself with a proper umask:

    https://git.buildroot.org/buildroot/tree/Makefile#n76

When I set a proper umask 0022, then I can indeed reproduce the issue,
but only in Ubuntu 18.04!

Still more investigations required...

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Buildroot] [Bug 13141] Target-finalize fail with "depmod: ERROR: Bad version passed"

[bugzilla at busybox.net]

https://bugs.busybox.net/show_bug.cgi?id=13141

--- Comment #7 from Yann E. MORIN <yann.morin.1998@free.fr> ---
Created attachment 8576
  --> https://bugs.busybox.net/attachment.cgi?id=8576&action=edit
0001-linux-workaround-make-4.1-bug-which-one.patch

Laurent, Asaf, All,

Could you please try to check if the attached patch solves the issue for you?

Regards,
Yann E. MORIN.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Buildroot] [Bug 13141] Target-finalize fail with "depmod: ERROR: Bad version passed"

[bugzilla at busybox.net]

https://bugs.busybox.net/show_bug.cgi?id=13141

--- Comment #8 from laurent at neko-labs.eu ---
Yann,
The patch is working!
I can activate all my procs :D, welcome back lightning fast compilation!

Thx!

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Buildroot] [Bug 13141] Target-finalize fail with "depmod: ERROR: Bad version passed"

[bugzilla at busybox.net]

https://bugs.busybox.net/show_bug.cgi?id=13141

--- Comment #9 from Yann E. MORIN <yann.morin.1998@free.fr> ---
(In reply to laurent from comment #8)
> The patch is working!

Great, thanks for the feedback.

I am not sure this will be the final form of the fix, or rather the workaround,
so maybe we'll need more testing...

Regards,
Yann E. MORIN.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Buildroot] [Bug 13141] Target-finalize fail with "depmod: ERROR: Bad version passed"

[bugzilla at busybox.net]

https://bugs.busybox.net/show_bug.cgi?id=13141

Peter Korsgaard <jacmet@uclibc.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
         Resolution|---                         |FIXED
             Status|NEW                         |RESOLVED

--- Comment #10 from Peter Korsgaard <jacmet@uclibc.org> ---
Worked around in git:
https://git.buildroot.org/buildroot/commit/?id=3f6a40e9fae92b3fe476d82449ddd6851cea03da

-- 
You are receiving this mail because:
You are on the CC list for the bug.