[PATCH] cmake.bbclass: Add ${COREBASE}/scripts to CMAKE_FIND_ROOT_PATH

[PATCH] cmake.bbclass: Add ${COREBASE}/scripts to CMAKE_FIND_ROOT_PATH

[Daniel Klauer]

${COREBASE}/scripts contains a "git" wrapper disabling fakeroot/pseudo.
This patch allows CMake to find ${COREBASE}/scripts/git instead of
${HOSTTOOLS_DIR}/git. This is needed for git invocations during do_install,
since do_install is a fakeroot task, and otherwise all git commands fail
with "fatal: detected dubious ownership in repository ...".

I don't know how common it is for CMake projects to invoke git during the
install phase intentionally. It's probably more common to do this during
the configure phase. However, the install step may re-run the configure
step, if some dependencies changed.

In my case, this happened in incremental Yocto builds which reran
do_install and repopulated parts of the recipe-sysroot during that,
without first rerunning do_configure or do_compile. One of the dependencies
changed (but only changing a file in some unrelated sub-package of it which
was not even installed into the recipe-sysroot), causing the dependant's
recipe-sysroot to be repopulated during do_install and thus causing the
CMake project to be reconfigured during do_install.

Signed-off-by: Daniel Klauer <daniel.klauer@gin.de>
---
 meta/classes-recipe/cmake.bbclass | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/meta/classes-recipe/cmake.bbclass b/meta/classes-recipe/cmake.bbclass
index 3d3781ef33..e1c3d7ddb5 100644
--- a/meta/classes-recipe/cmake.bbclass
+++ b/meta/classes-recipe/cmake.bbclass
@@ -124,7 +124,7 @@ set( CMAKE_CXX_LINK_FLAGS "${OECMAKE_CXX_LINK_FLAGS}" CACHE STRING "LDFLAGS" )
 
 # only search in the paths provided so cmake doesnt pick
 # up libraries and tools from the native build machine
-set( CMAKE_FIND_ROOT_PATH ${STAGING_DIR_HOST} ${STAGING_DIR_NATIVE} ${CROSS_DIR} ${OECMAKE_PERLNATIVE_DIR} ${OECMAKE_EXTRA_ROOT_PATH} ${EXTERNAL_TOOLCHAIN} ${HOSTTOOLS_DIR})
+set( CMAKE_FIND_ROOT_PATH ${STAGING_DIR_HOST} ${STAGING_DIR_NATIVE} ${CROSS_DIR} ${OECMAKE_PERLNATIVE_DIR} ${OECMAKE_EXTRA_ROOT_PATH} ${EXTERNAL_TOOLCHAIN} ${COREBASE}/scripts ${HOSTTOOLS_DIR} )
 set( CMAKE_FIND_ROOT_PATH_MODE_PACKAGE ONLY )
 set( CMAKE_FIND_ROOT_PATH_MODE_PROGRAM ${OECMAKE_FIND_ROOT_PATH_MODE_PROGRAM} )
 set( CMAKE_FIND_ROOT_PATH_MODE_LIBRARY ONLY )
-- 
2.34.1

Re: [OE-core] [PATCH] cmake.bbclass: Add ${COREBASE}/scripts to CMAKE_FIND_ROOT_PATH

[Khem Raj]

[-- Attachment #1: Type: text/plain, Size: 3236 bytes --]

On Wed, Apr 17, 2024 at 4:25 AM Daniel Klauer via lists.openembedded.org
<daniel.klauer=gin.de@lists.openembedded.org> wrote:

> ${COREBASE}/scripts contains a "git" wrapper disabling fakeroot/pseudo.
> This patch allows CMake to find ${COREBASE}/scripts/git instead of
> ${HOSTTOOLS_DIR}/git. This is needed for git invocations during do_install,
> since do_install is a fakeroot task, and otherwise all git commands fail
> with "fatal: detected dubious ownership in repository ...".
>
> I don't know how common it is for CMake projects to invoke git during the
> install phase intentionally. It's probably more common to do this during
> the configure phase. However, the install step may re-run the configure
> step, if some dependencies changed.


If a package build system has such dependencies they should be better
expressed in recipes and linked using bitbake task level dependency
mechanism

I wonder why git is needed during install phase
Can you explain a bit more on that

And since change it in a class it means it has to be a pattern


>
> In my case, this happened in incremental Yocto builds which reran
> do_install and repopulated parts of the recipe-sysroot during that,
> without first rerunning do_configure or do_compile. One of the dependencies
> changed (but only changing a file in some unrelated sub-package of it which
> was not even installed into the recipe-sysroot), causing the dependant's
> recipe-sysroot to be repopulated during do_install and thus causing the
> CMake project to be reconfigured during do_install.
>
> Signed-off-by: Daniel Klauer <daniel.klauer@gin.de>
> ---
>  meta/classes-recipe/cmake.bbclass | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/meta/classes-recipe/cmake.bbclass
> b/meta/classes-recipe/cmake.bbclass
> index 3d3781ef33..e1c3d7ddb5 100644
> --- a/meta/classes-recipe/cmake.bbclass
> +++ b/meta/classes-recipe/cmake.bbclass
> @@ -124,7 +124,7 @@ set( CMAKE_CXX_LINK_FLAGS "${OECMAKE_CXX_LINK_FLAGS}"
> CACHE STRING "LDFLAGS" )
>
>  # only search in the paths provided so cmake doesnt pick
>  # up libraries and tools from the native build machine
> -set( CMAKE_FIND_ROOT_PATH ${STAGING_DIR_HOST} ${STAGING_DIR_NATIVE}
> ${CROSS_DIR} ${OECMAKE_PERLNATIVE_DIR} ${OECMAKE_EXTRA_ROOT_PATH}
> ${EXTERNAL_TOOLCHAIN} ${HOSTTOOLS_DIR})
> +set( CMAKE_FIND_ROOT_PATH ${STAGING_DIR_HOST} ${STAGING_DIR_NATIVE}
> ${CROSS_DIR} ${OECMAKE_PERLNATIVE_DIR} ${OECMAKE_EXTRA_ROOT_PATH}
> ${EXTERNAL_TOOLCHAIN} ${COREBASE}/scripts ${HOSTTOOLS_DIR} )
>  set( CMAKE_FIND_ROOT_PATH_MODE_PACKAGE ONLY )
>  set( CMAKE_FIND_ROOT_PATH_MODE_PROGRAM
> ${OECMAKE_FIND_ROOT_PATH_MODE_PROGRAM} )
>  set( CMAKE_FIND_ROOT_PATH_MODE_LIBRARY ONLY )
> --
> 2.34.1
>
>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#198462):
> https://lists.openembedded.org/g/openembedded-core/message/198462
> Mute This Topic: https://lists.openembedded.org/mt/105575001/1997914
> Group Owner: openembedded-core+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [
> raj.khem@gmail.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>
>

[-- Attachment #2: Type: text/html, Size: 4513 bytes --]

Re: [OE-core] [PATCH] cmake.bbclass: Add ${COREBASE}/scripts to CMAKE_FIND_ROOT_PATH

[Chuck Wolber]

[-- Attachment #1: Type: text/plain, Size: 1506 bytes --]

On Wed, Apr 17, 2024 at 12:14 Khem Raj via lists.openembedded.org <raj.khem=
gmail.com@lists.openembedded.org> wrote:

>
>
> On Wed, Apr 17, 2024 at 4:25 AM Daniel Klauer via lists.openembedded.org
> <daniel.klauer=gin.de@lists.openembedded.org> wrote:
>
>> ${COREBASE}/scripts contains a "git" wrapper disabling fakeroot/pseudo.
>> This patch allows CMake to find ${COREBASE}/scripts/git instead of
>> ${HOSTTOOLS_DIR}/git. This is needed for git invocations during
>> do_install,
>> since do_install is a fakeroot task, and otherwise all git commands fail
>> with "fatal: detected dubious ownership in repository ...".
>>
>> I don't know how common it is for CMake projects to invoke git during the
>> install phase intentionally. It's probably more common to do this during
>> the configure phase. However, the install step may re-run the configure
>> step, if some dependencies changed.
>
>
> If a package build system has such dependencies they should be better
> expressed in recipes and linked using bitbake task level dependency
> mechanism
>
> I wonder why git is needed during install phase
> Can you explain a bit more on that
>
> And since change it in a class it means it has to be a pattern
>

I do not think dependencies will completely solve the problem. Such
patterns break builds when your distro is configured for shallow git
clones. I have run into at least one python library that operates this way
(the names escape me at the moment).

..Ch:W..

[-- Attachment #2: Type: text/html, Size: 2486 bytes --]

Re: [OE-core] [PATCH] cmake.bbclass: Add ${COREBASE}/scripts to CMAKE_FIND_ROOT_PATH

[Daniel Klauer]

Hi,

>> ${COREBASE}/scripts contains a "git" wrapper disabling fakeroot/pseudo.
>> This patch allows CMake to find ${COREBASE}/scripts/git instead of
>> ${HOSTTOOLS_DIR}/git. This is needed for git invocations during do_install,
>> since do_install is a fakeroot task, and otherwise all git commands fail
>> with "fatal: detected dubious ownership in repository ...".
>>
>> I don't know how common it is for CMake projects to invoke git during the
>> install phase intentionally. It's probably more common to do this during
>> the configure phase. However, the install step may re-run the configure
>> step, if some dependencies changed.
>
>
> If a package build system has such dependencies they should be better expressed in recipes and linked using bitbake task level dependency mechanism 
> 
> I wonder why git is needed during install phase
> Can you explain a bit more on that 
> 
> And since change it in a class it means it has to be a pattern 

It's just about running "git rev-parse HEAD" or similar, not acquiring external dependencies with "git clone", FetchContent download, etc., if that's what you meant. "dependencies" referred to timestamps of files checked by make/ninja.

The affected CMake projects are using code like the following to query Git revision information (for storing into header files and binaries):

	find_package(Git REQUIRED)
	execute_process(COMMAND "${GIT_EXECUTABLE}" rev-parse HEAD ...)
	set_property(DIRECTORY APPEND PROPERTY CMAKE_CONFIGURE_DEPENDS "${CMAKE_CURRENT_SOURCE_DIR}/.git/HEAD")

${S} is owned by the user that runs bitbake, so cmake succeeds running such git commands during do_configure, but they fail during do_install because it's under fakeroot/pseudo.

This can be fixed by allowing CMake to use the git wrapper script which temporarily disables fakeroot. It's already in PATH anyways (see PATH:prepend = "${COREBASE}/scripts..." in bitbake.conf), but CMake's find_package(Git) only respects CMAKE_FIND_ROOT_PATH. The description of commit af27c81e "scripts: Make git intercept global" hints at this type of situation.


As for why cmake re-runs during do_install, I'm still trying to piece it together and reduce it to a minimal example. I can try to explain what I know so far:

Basically, it's normal for a CMake project to be re-configured by "make" or "make install" (or ninja equivalents), if for example the CMakeLists.txt or files referenced by CMAKE_CONFIGURE_DEPENDS were modified, and an existing build directory was re-used. I'm not really surprised to see this case occurring in Yocto aswell. Presumably it can only happen in incremental Yocto builds if do_fetch/do_unpack/do_patch were already run previously and don't need to run again.

What I did:

1. have a recipe for a library
2. have a recipe for a program which depends on that library (and of courses uses DEPENDS += "library")
3. run bitbake to build the image
4. modify library:do_install (to install a different configuration file into /etc)
5. modify program:do_install (same)
6. run bitbake to build the image

My observations for the 2nd bitbake run (after I modified the two do_install functions):

6.1. library:do_configure did not run again (makes sense, since I didn't change anything there)
6.2. library:do_compile   did not run again (same)
6.3. library:do_install was re-run (due to the changes for the new config file in /etc)
6.4. library:do_package was re-run (but presumably only the main library package changed, not the library-dev or library-static packages)
6.5. program:do_configure did not run again (not sure why not, since library recipe changed, but maybe bitbake noticed through the library packaging that only /etc changed and not the library or its header files)
6.6. program:do_compile   did not run again (same)
6.7. program:do_install was re-run
6.7.1. The library was re-populated into the program's recipe-sysroot ("NOTE: <library> exists in sysroot, but is stale")
6.7.2. "ninja install" re-ran CMake to re-configure the project (I'm guessing due to the re-population of the recipe-sysroot), this is where the git commands failed.


Regards,
Daniel