[PATCH] gcc: remove paths to sysroot from configargs.h and checksum-options for gcc-cross-canadian

[PATCH] gcc: remove paths to sysroot from configargs.h and checksum-options for gcc-cross-canadian

[Oleksandr Hnatiuk]

Copy fixes for gcc-cross (84a78f46d594 and 0ead8cbdfb96) to gcc-cross-canadian.
This will improve (but not fix) reproducibility of gcc-cross-canadian.

Signed-off-by: Oleksandr Hnatiuk <ohnatiuk@cisco.com>
---
 meta/recipes-devtools/gcc/gcc-cross-canadian.inc | 14 ++++++++++++++
 1 file changed, 14 insertions(+)

diff --git a/meta/recipes-devtools/gcc/gcc-cross-canadian.inc b/meta/recipes-devtools/gcc/gcc-cross-canadian.inc
index 7c4233c21157..127f97473f91 100644
--- a/meta/recipes-devtools/gcc/gcc-cross-canadian.inc
+++ b/meta/recipes-devtools/gcc/gcc-cross-canadian.inc
@@ -64,6 +64,20 @@ do_configure () {
 }
 
 do_compile () {
+	# Prevent native/host sysroot path from being used in configargs.h header,
+	# as it will be rewritten when used by other sysroots preventing support
+	# for gcc plugins
+	oe_runmake configure-gcc
+	sed -i 's@${STAGING_DIR_TARGET}@/host@g' ${B}/gcc/configargs.h
+	sed -i 's@${STAGING_DIR_HOST}@/host@g' ${B}/gcc/configargs.h
+
+	# Prevent sysroot/workdir paths from being used in checksum-options.
+	# checksum-options is used to generate a checksum which is embedded into
+	# the output binary.
+	oe_runmake TARGET-gcc=checksum-options all-gcc
+	sed -i 's@${DEBUG_PREFIX_MAP}@@g' ${B}/gcc/checksum-options
+	sed -i 's@${STAGING_DIR_HOST}@/host@g' ${B}/gcc/checksum-options
+
 	oe_runmake all-host configure-target-libgcc
 	(cd ${B}/${TARGET_SYS}/libgcc; oe_runmake enable-execute-stack.c unwind.h md-unwind-support.h sfp-machine.h gthr-default.h)
 }
-- 
2.35.6

Re: [OE-core] [PATCH] gcc: remove paths to sysroot from configargs.h and checksum-options for gcc-cross-canadian

[Alexander Kanavin]

On Fri, 12 Jul 2024 at 13:01, Oleksandr Hnatiuk via
lists.openembedded.org <ohnatiuk=cisco.com@lists.openembedded.org>
wrote:
> +       sed -i 's@${DEBUG_PREFIX_MAP}@@g' ${B}/gcc/checksum-options
> +       sed -i 's@${STAGING_DIR_HOST}@/host@g' ${B}/gcc/checksum-options

The sed stuff here should also go to a function instead of being copy-pasted.

Alex

Re: [PATCH] gcc: remove paths to sysroot from configargs.h and checksum-options for gcc-cross-canadian

[Oleksandr Hnatiuk]

[-- Attachment #1: Type: text/plain, Size: 280 bytes --]

Hi Alex,

gcc recipe structure is quite complex. Where should I put these common functions? gcc-configure-common.inc ?
I see that all 3 files (gcc-target.inc, gcc-cross.inc, gcc-cross-canadian.inc) which use this code `require gcc-configure-common.inc`.

Thanks,
Oleksandr

[-- Attachment #2: Type: text/html, Size: 304 bytes --]

Re: [OE-core] [PATCH] gcc: remove paths to sysroot from configargs.h and checksum-options for gcc-cross-canadian

[Alexander Kanavin]

On Fri, 12 Jul 2024 at 21:22, Oleksandr Hnatiuk via
lists.openembedded.org <ohnatiuk=cisco.com@lists.openembedded.org>
wrote:
> gcc recipe structure is quite complex. Where should I put these common functions? gcc-configure-common.inc ?
> I see that all 3 files (gcc-target.inc, gcc-cross.inc, gcc-cross-canadian.inc) which use this code `require gcc-configure-common.inc`.

Yes, I think that'd be the best location.

Alex

[PATCH v2] gcc: remove paths to sysroot from configargs.h and checksum-options for gcc-cross-canadian

[Oleksandr Hnatiuk]

Apply fixes from gcc-cross (84a78f46d594 and 0ead8cbdfb96) to gcc-cross-canadian.
This will improve (but not fix) reproducibility of gcc-cross-canadian.
Also move this code to functions to avoid code duplication.

Signed-off-by: Oleksandr Hnatiuk <ohnatiuk@cisco.com>
---
 .../gcc/gcc-configure-common.inc              | 19 +++++++++++++++++++
 .../gcc/gcc-cross-canadian.inc                |  3 +++
 meta/recipes-devtools/gcc/gcc-cross.inc       | 15 ++-------------
 meta/recipes-devtools/gcc/gcc-target.inc      | 16 ++--------------
 4 files changed, 26 insertions(+), 27 deletions(-)

diff --git a/meta/recipes-devtools/gcc/gcc-configure-common.inc b/meta/recipes-devtools/gcc/gcc-configure-common.inc
index d77b2ec3d1de..c936d1a9bd7b 100644
--- a/meta/recipes-devtools/gcc/gcc-configure-common.inc
+++ b/meta/recipes-devtools/gcc/gcc-configure-common.inc
@@ -120,3 +120,22 @@ do_configure () {
 	oe_runconf
 }
 
+remove_sysroot_paths_from_configargs () {
+	# Prevent sysroot path from being used in configargs.h header, as it will
+	# be rewritten when used by other sysroots preventing support for gcc
+	# plugins. Additionally the path is embeddeded into the output binary, this
+	# prevents building a reproducible binary.
+	oe_runmake configure-gcc
+	sed -i 's@${STAGING_DIR_TARGET}@/host@g' ${B}/gcc/configargs.h
+	sed -i 's@${STAGING_DIR_HOST}@/host@g' ${B}/gcc/configargs.h
+}
+
+remove_sysroot_paths_from_checksum_options () {
+	# Prevent sysroot/workdir paths from being used in checksum-options.
+	# checksum-options is used to generate a checksum which is embedded into
+	# the output binary.
+	oe_runmake TARGET-gcc=checksum-options all-gcc
+	sed -i 's@${DEBUG_PREFIX_MAP}@@g' ${B}/gcc/checksum-options
+	sed -i 's@${1}@/host@g' ${B}/gcc/checksum-options
+}
+
diff --git a/meta/recipes-devtools/gcc/gcc-cross-canadian.inc b/meta/recipes-devtools/gcc/gcc-cross-canadian.inc
index 7c4233c21157..9528b71272dd 100644
--- a/meta/recipes-devtools/gcc/gcc-cross-canadian.inc
+++ b/meta/recipes-devtools/gcc/gcc-cross-canadian.inc
@@ -64,6 +64,9 @@ do_configure () {
 }
 
 do_compile () {
+	remove_sysroot_paths_from_configargs
+	remove_sysroot_paths_from_checksum_options '${STAGING_DIR_HOST}'
+
 	oe_runmake all-host configure-target-libgcc
 	(cd ${B}/${TARGET_SYS}/libgcc; oe_runmake enable-execute-stack.c unwind.h md-unwind-support.h sfp-machine.h gthr-default.h)
 }
diff --git a/meta/recipes-devtools/gcc/gcc-cross.inc b/meta/recipes-devtools/gcc/gcc-cross.inc
index 5b0ca15d4762..f8a7730cf322 100644
--- a/meta/recipes-devtools/gcc/gcc-cross.inc
+++ b/meta/recipes-devtools/gcc/gcc-cross.inc
@@ -62,19 +62,8 @@ do_compile () {
 	export CXXFLAGS_FOR_TARGET="${TARGET_CXXFLAGS}"
 	export LDFLAGS_FOR_TARGET="${TARGET_LDFLAGS}"
 
-	# Prevent native/host sysroot path from being used in configargs.h header,
-	# as it will be rewritten when used by other sysroots preventing support
-	# for gcc plugins
-	oe_runmake configure-gcc
-	sed -i 's@${STAGING_DIR_TARGET}@/host@g' ${B}/gcc/configargs.h
-	sed -i 's@${STAGING_DIR_HOST}@/host@g' ${B}/gcc/configargs.h
-
-	# Prevent sysroot/workdir paths from being used in checksum-options.
-	# checksum-options is used to generate a checksum which is embedded into
-	# the output binary.
-	oe_runmake TARGET-gcc=checksum-options all-gcc
-	sed -i 's@${DEBUG_PREFIX_MAP}@@g' ${B}/gcc/checksum-options
-	sed -i 's@${STAGING_DIR_HOST}@/host@g' ${B}/gcc/checksum-options
+	remove_sysroot_paths_from_configargs
+	remove_sysroot_paths_from_checksum_options '${STAGING_DIR_HOST}'
 
 	oe_runmake all-host configure-target-libgcc
 	(cd ${B}/${TARGET_SYS}/libgcc; oe_runmake enable-execute-stack.c unwind.h md-unwind-support.h sfp-machine.h gthr-default.h)
diff --git a/meta/recipes-devtools/gcc/gcc-target.inc b/meta/recipes-devtools/gcc/gcc-target.inc
index e9187fc444b9..bd7506fe3a55 100644
--- a/meta/recipes-devtools/gcc/gcc-target.inc
+++ b/meta/recipes-devtools/gcc/gcc-target.inc
@@ -132,20 +132,8 @@ FILES:${PN}-doc = "\
 "
 
 do_compile () {
-	# Prevent full target sysroot path from being used in configargs.h header,
-	# as it will be rewritten when used by other sysroots preventing support
-	# for gcc plugins. Additionally the path is embeddeded into the output
-	# binary, this prevents building a reproducible binary.
-	oe_runmake configure-gcc
-	sed -i 's@${STAGING_DIR_TARGET}@/@g' ${B}/gcc/configargs.h
-	sed -i 's@${STAGING_DIR_HOST}@/@g' ${B}/gcc/configargs.h
-
-	# Prevent sysroot/workdir paths from being used in checksum-options.
-	# checksum-options is used to generate a checksum which is embedded into
-	# the output binary.
-	oe_runmake TARGET-gcc=checksum-options all-gcc
-	sed -i 's@${DEBUG_PREFIX_MAP}@@g' ${B}/gcc/checksum-options
-	sed -i 's@${STAGING_DIR_TARGET}@/@g' ${B}/gcc/checksum-options
+	remove_sysroot_paths_from_configargs
+	remove_sysroot_paths_from_checksum_options '${STAGING_DIR_TARGET}'
 
 	oe_runmake all-host
 }
-- 
2.35.6

Re: [PATCH v2] gcc: remove paths to sysroot from configargs.h and checksum-options for gcc-cross-canadian

[Oleksandr Hnatiuk]

[-- Attachment #1: Type: text/plain, Size: 846 bytes --]

For some reason this created a new topic rather than post to existing one...

I have refactored the code from my previous submission to move the code from all 3 places to functions.
For both functions: I have chosen sed code from gcc-cross.inc (replaces the paths with "/host" rather than "/" as gcc-target did). This unification shouldn't affect the functionality, only what will be seen instead of sysroot paths.
For remove_sysroot_paths_from_checksum_options: I saw that gcc-target.inc and gcc-cross.inc erase STAGING_DIR_TARGET and STAGING_DIR_HOST respectively. To preserve this behavior, I pass them as parameters and use ${1} in sed.

Since I am not that familiar with possible differences between regular shell scripting and shell scripting inside of bitbake recipes, could you confirm that I parameterized the function correctly?

[-- Attachment #2: Type: text/html, Size: 955 bytes --]

Re: [OE-core] [PATCH v2] gcc: remove paths to sysroot from configargs.h and checksum-options for gcc-cross-canadian

[Denys Dmytriyenko]

FWIW, I'm testing master-next now with your patch applied and I'm also seeing 
the same buildpaths QA error in .../include-fixed/pthread.h

ERROR: gcc-cross-canadian-aarch64-14.1.0-r0 do_package_qa: QA Issue: File /usr/local/oe-sdk-hardcoded-buildpath/sysroots/x86_64-pokysdk-linux/usr/lib/aarch64-poky-linux/gcc/aarch64-poky-linux/14.1.0/include-fixed/pthread.h in package gcc-cross-canadian-aarch64 contains reference to TMPDIR [buildpaths]
ERROR: gcc-cross-canadian-aarch64-14.1.0-r0 do_package_qa: Fatal QA errors were found, failing task.
ERROR: Logfile of failure stored in: /OE/poky-master/build/tmp/work/x86_64-nativesdk-pokysdk-linux/gcc-cross-canadian-aarch64/14.1.0/temp/log.do_package_qa.820539
ERROR: Task (/OE/poky-master/meta/recipes-devtools/gcc/gcc-cross-canadian_14.1.bb:do_package_qa) failed with exit code '1'

The header file contains this:

/*  DO NOT EDIT THIS FILE.

    It has been auto-edited by fixincludes from:

        "/OE/poky-master/build/tmp/work/x86_64-nativesdk-pokysdk-linux/gcc-cross-canadian-aarch64/14.1.0/recipe-sysroot/usr/include/pthread.h"

    This had to be done to correct non-standard usages in the
    original, manufacturer supplied header file.  */

Both gcc-target and gcc-cross end up removing this header file:
https://git.openembedded.org/openembedded-core/commit/?id=e0af4b2c8f8e29ac6f8eccef401c7c004355359d
https://git.openembedded.org/openembedded-core/commit/?id=f0fcaa88b7b2977c2cb35b060747442ee9ff3dcd

I'm not sure if we should do the same for gcc-cross-canadian as well - I'm 
trying it locally now...



On Wed, Jul 17, 2024 at 02:56:06AM -0700, Oleksandr Hnatiuk via lists.openembedded.org wrote:
> Apply fixes from gcc-cross (84a78f46d594 and 0ead8cbdfb96) to gcc-cross-canadian.
> This will improve (but not fix) reproducibility of gcc-cross-canadian.
> Also move this code to functions to avoid code duplication.
> 
> Signed-off-by: Oleksandr Hnatiuk <ohnatiuk@cisco.com>
> ---
>  .../gcc/gcc-configure-common.inc              | 19 +++++++++++++++++++
>  .../gcc/gcc-cross-canadian.inc                |  3 +++
>  meta/recipes-devtools/gcc/gcc-cross.inc       | 15 ++-------------
>  meta/recipes-devtools/gcc/gcc-target.inc      | 16 ++--------------
>  4 files changed, 26 insertions(+), 27 deletions(-)
> 
> diff --git a/meta/recipes-devtools/gcc/gcc-configure-common.inc b/meta/recipes-devtools/gcc/gcc-configure-common.inc
> index d77b2ec3d1de..c936d1a9bd7b 100644
> --- a/meta/recipes-devtools/gcc/gcc-configure-common.inc
> +++ b/meta/recipes-devtools/gcc/gcc-configure-common.inc
> @@ -120,3 +120,22 @@ do_configure () {
>  	oe_runconf
>  }
>  
> +remove_sysroot_paths_from_configargs () {
> +	# Prevent sysroot path from being used in configargs.h header, as it will
> +	# be rewritten when used by other sysroots preventing support for gcc
> +	# plugins. Additionally the path is embeddeded into the output binary, this
> +	# prevents building a reproducible binary.
> +	oe_runmake configure-gcc
> +	sed -i 's@${STAGING_DIR_TARGET}@/host@g' ${B}/gcc/configargs.h
> +	sed -i 's@${STAGING_DIR_HOST}@/host@g' ${B}/gcc/configargs.h
> +}
> +
> +remove_sysroot_paths_from_checksum_options () {
> +	# Prevent sysroot/workdir paths from being used in checksum-options.
> +	# checksum-options is used to generate a checksum which is embedded into
> +	# the output binary.
> +	oe_runmake TARGET-gcc=checksum-options all-gcc
> +	sed -i 's@${DEBUG_PREFIX_MAP}@@g' ${B}/gcc/checksum-options
> +	sed -i 's@${1}@/host@g' ${B}/gcc/checksum-options
> +}
> +
> diff --git a/meta/recipes-devtools/gcc/gcc-cross-canadian.inc b/meta/recipes-devtools/gcc/gcc-cross-canadian.inc
> index 7c4233c21157..9528b71272dd 100644
> --- a/meta/recipes-devtools/gcc/gcc-cross-canadian.inc
> +++ b/meta/recipes-devtools/gcc/gcc-cross-canadian.inc
> @@ -64,6 +64,9 @@ do_configure () {
>  }
>  
>  do_compile () {
> +	remove_sysroot_paths_from_configargs
> +	remove_sysroot_paths_from_checksum_options '${STAGING_DIR_HOST}'
> +
>  	oe_runmake all-host configure-target-libgcc
>  	(cd ${B}/${TARGET_SYS}/libgcc; oe_runmake enable-execute-stack.c unwind.h md-unwind-support.h sfp-machine.h gthr-default.h)
>  }
> diff --git a/meta/recipes-devtools/gcc/gcc-cross.inc b/meta/recipes-devtools/gcc/gcc-cross.inc
> index 5b0ca15d4762..f8a7730cf322 100644
> --- a/meta/recipes-devtools/gcc/gcc-cross.inc
> +++ b/meta/recipes-devtools/gcc/gcc-cross.inc
> @@ -62,19 +62,8 @@ do_compile () {
>  	export CXXFLAGS_FOR_TARGET="${TARGET_CXXFLAGS}"
>  	export LDFLAGS_FOR_TARGET="${TARGET_LDFLAGS}"
>  
> -	# Prevent native/host sysroot path from being used in configargs.h header,
> -	# as it will be rewritten when used by other sysroots preventing support
> -	# for gcc plugins
> -	oe_runmake configure-gcc
> -	sed -i 's@${STAGING_DIR_TARGET}@/host@g' ${B}/gcc/configargs.h
> -	sed -i 's@${STAGING_DIR_HOST}@/host@g' ${B}/gcc/configargs.h
> -
> -	# Prevent sysroot/workdir paths from being used in checksum-options.
> -	# checksum-options is used to generate a checksum which is embedded into
> -	# the output binary.
> -	oe_runmake TARGET-gcc=checksum-options all-gcc
> -	sed -i 's@${DEBUG_PREFIX_MAP}@@g' ${B}/gcc/checksum-options
> -	sed -i 's@${STAGING_DIR_HOST}@/host@g' ${B}/gcc/checksum-options
> +	remove_sysroot_paths_from_configargs
> +	remove_sysroot_paths_from_checksum_options '${STAGING_DIR_HOST}'
>  
>  	oe_runmake all-host configure-target-libgcc
>  	(cd ${B}/${TARGET_SYS}/libgcc; oe_runmake enable-execute-stack.c unwind.h md-unwind-support.h sfp-machine.h gthr-default.h)
> diff --git a/meta/recipes-devtools/gcc/gcc-target.inc b/meta/recipes-devtools/gcc/gcc-target.inc
> index e9187fc444b9..bd7506fe3a55 100644
> --- a/meta/recipes-devtools/gcc/gcc-target.inc
> +++ b/meta/recipes-devtools/gcc/gcc-target.inc
> @@ -132,20 +132,8 @@ FILES:${PN}-doc = "\
>  "
>  
>  do_compile () {
> -	# Prevent full target sysroot path from being used in configargs.h header,
> -	# as it will be rewritten when used by other sysroots preventing support
> -	# for gcc plugins. Additionally the path is embeddeded into the output
> -	# binary, this prevents building a reproducible binary.
> -	oe_runmake configure-gcc
> -	sed -i 's@${STAGING_DIR_TARGET}@/@g' ${B}/gcc/configargs.h
> -	sed -i 's@${STAGING_DIR_HOST}@/@g' ${B}/gcc/configargs.h
> -
> -	# Prevent sysroot/workdir paths from being used in checksum-options.
> -	# checksum-options is used to generate a checksum which is embedded into
> -	# the output binary.
> -	oe_runmake TARGET-gcc=checksum-options all-gcc
> -	sed -i 's@${DEBUG_PREFIX_MAP}@@g' ${B}/gcc/checksum-options
> -	sed -i 's@${STAGING_DIR_TARGET}@/@g' ${B}/gcc/checksum-options
> +	remove_sysroot_paths_from_configargs
> +	remove_sysroot_paths_from_checksum_options '${STAGING_DIR_TARGET}'
>  
>  	oe_runmake all-host
>  }
> -- 
> 2.35.6

Re: [OE-core] [PATCH v2] gcc: remove paths to sysroot from configargs.h and checksum-options for gcc-cross-canadian

[Denys Dmytriyenko]

On Wed, Jul 17, 2024 at 08:21:31PM -0400, Denys Dmytriyenko wrote:
> FWIW, I'm testing master-next now with your patch applied and I'm also seeing 
> the same buildpaths QA error in .../include-fixed/pthread.h
> 
> ERROR: gcc-cross-canadian-aarch64-14.1.0-r0 do_package_qa: QA Issue: File /usr/local/oe-sdk-hardcoded-buildpath/sysroots/x86_64-pokysdk-linux/usr/lib/aarch64-poky-linux/gcc/aarch64-poky-linux/14.1.0/include-fixed/pthread.h in package gcc-cross-canadian-aarch64 contains reference to TMPDIR [buildpaths]
> ERROR: gcc-cross-canadian-aarch64-14.1.0-r0 do_package_qa: Fatal QA errors were found, failing task.
> ERROR: Logfile of failure stored in: /OE/poky-master/build/tmp/work/x86_64-nativesdk-pokysdk-linux/gcc-cross-canadian-aarch64/14.1.0/temp/log.do_package_qa.820539
> ERROR: Task (/OE/poky-master/meta/recipes-devtools/gcc/gcc-cross-canadian_14.1.bb:do_package_qa) failed with exit code '1'
> 
> The header file contains this:
> 
> /*  DO NOT EDIT THIS FILE.
> 
>     It has been auto-edited by fixincludes from:
> 
>         "/OE/poky-master/build/tmp/work/x86_64-nativesdk-pokysdk-linux/gcc-cross-canadian-aarch64/14.1.0/recipe-sysroot/usr/include/pthread.h"
> 
>     This had to be done to correct non-standard usages in the
>     original, manufacturer supplied header file.  */
> 
> Both gcc-target and gcc-cross end up removing this header file:
> https://git.openembedded.org/openembedded-core/commit/?id=e0af4b2c8f8e29ac6f8eccef401c7c004355359d
> https://git.openembedded.org/openembedded-core/commit/?id=f0fcaa88b7b2977c2cb35b060747442ee9ff3dcd
> 
> I'm not sure if we should do the same for gcc-cross-canadian as well - I'm 
> trying it locally now...

Well, similarly pruning include-fixed with find+rm makes all the builds 
succeed. But I haven't tried using the resulting gcc-canadian-cross yet.

Also, just a nitpick - I noticed the 2 new functions are used from 
do_compile(), while being added to gcc-configure-common.inc, not 
gcc-common.inc


> On Wed, Jul 17, 2024 at 02:56:06AM -0700, Oleksandr Hnatiuk via lists.openembedded.org wrote:
> > Apply fixes from gcc-cross (84a78f46d594 and 0ead8cbdfb96) to gcc-cross-canadian.
> > This will improve (but not fix) reproducibility of gcc-cross-canadian.
> > Also move this code to functions to avoid code duplication.
> > 
> > Signed-off-by: Oleksandr Hnatiuk <ohnatiuk@cisco.com>
> > ---
> >  .../gcc/gcc-configure-common.inc              | 19 +++++++++++++++++++
> >  .../gcc/gcc-cross-canadian.inc                |  3 +++
> >  meta/recipes-devtools/gcc/gcc-cross.inc       | 15 ++-------------
> >  meta/recipes-devtools/gcc/gcc-target.inc      | 16 ++--------------
> >  4 files changed, 26 insertions(+), 27 deletions(-)
> > 
> > diff --git a/meta/recipes-devtools/gcc/gcc-configure-common.inc b/meta/recipes-devtools/gcc/gcc-configure-common.inc
> > index d77b2ec3d1de..c936d1a9bd7b 100644
> > --- a/meta/recipes-devtools/gcc/gcc-configure-common.inc
> > +++ b/meta/recipes-devtools/gcc/gcc-configure-common.inc
> > @@ -120,3 +120,22 @@ do_configure () {
> >  	oe_runconf
> >  }
> >  
> > +remove_sysroot_paths_from_configargs () {
> > +	# Prevent sysroot path from being used in configargs.h header, as it will
> > +	# be rewritten when used by other sysroots preventing support for gcc
> > +	# plugins. Additionally the path is embeddeded into the output binary, this
> > +	# prevents building a reproducible binary.
> > +	oe_runmake configure-gcc
> > +	sed -i 's@${STAGING_DIR_TARGET}@/host@g' ${B}/gcc/configargs.h
> > +	sed -i 's@${STAGING_DIR_HOST}@/host@g' ${B}/gcc/configargs.h
> > +}
> > +
> > +remove_sysroot_paths_from_checksum_options () {
> > +	# Prevent sysroot/workdir paths from being used in checksum-options.
> > +	# checksum-options is used to generate a checksum which is embedded into
> > +	# the output binary.
> > +	oe_runmake TARGET-gcc=checksum-options all-gcc
> > +	sed -i 's@${DEBUG_PREFIX_MAP}@@g' ${B}/gcc/checksum-options
> > +	sed -i 's@${1}@/host@g' ${B}/gcc/checksum-options
> > +}
> > +
> > diff --git a/meta/recipes-devtools/gcc/gcc-cross-canadian.inc b/meta/recipes-devtools/gcc/gcc-cross-canadian.inc
> > index 7c4233c21157..9528b71272dd 100644
> > --- a/meta/recipes-devtools/gcc/gcc-cross-canadian.inc
> > +++ b/meta/recipes-devtools/gcc/gcc-cross-canadian.inc
> > @@ -64,6 +64,9 @@ do_configure () {
> >  }
> >  
> >  do_compile () {
> > +	remove_sysroot_paths_from_configargs
> > +	remove_sysroot_paths_from_checksum_options '${STAGING_DIR_HOST}'
> > +
> >  	oe_runmake all-host configure-target-libgcc
> >  	(cd ${B}/${TARGET_SYS}/libgcc; oe_runmake enable-execute-stack.c unwind.h md-unwind-support.h sfp-machine.h gthr-default.h)
> >  }
> > diff --git a/meta/recipes-devtools/gcc/gcc-cross.inc b/meta/recipes-devtools/gcc/gcc-cross.inc
> > index 5b0ca15d4762..f8a7730cf322 100644
> > --- a/meta/recipes-devtools/gcc/gcc-cross.inc
> > +++ b/meta/recipes-devtools/gcc/gcc-cross.inc
> > @@ -62,19 +62,8 @@ do_compile () {
> >  	export CXXFLAGS_FOR_TARGET="${TARGET_CXXFLAGS}"
> >  	export LDFLAGS_FOR_TARGET="${TARGET_LDFLAGS}"
> >  
> > -	# Prevent native/host sysroot path from being used in configargs.h header,
> > -	# as it will be rewritten when used by other sysroots preventing support
> > -	# for gcc plugins
> > -	oe_runmake configure-gcc
> > -	sed -i 's@${STAGING_DIR_TARGET}@/host@g' ${B}/gcc/configargs.h
> > -	sed -i 's@${STAGING_DIR_HOST}@/host@g' ${B}/gcc/configargs.h
> > -
> > -	# Prevent sysroot/workdir paths from being used in checksum-options.
> > -	# checksum-options is used to generate a checksum which is embedded into
> > -	# the output binary.
> > -	oe_runmake TARGET-gcc=checksum-options all-gcc
> > -	sed -i 's@${DEBUG_PREFIX_MAP}@@g' ${B}/gcc/checksum-options
> > -	sed -i 's@${STAGING_DIR_HOST}@/host@g' ${B}/gcc/checksum-options
> > +	remove_sysroot_paths_from_configargs
> > +	remove_sysroot_paths_from_checksum_options '${STAGING_DIR_HOST}'
> >  
> >  	oe_runmake all-host configure-target-libgcc
> >  	(cd ${B}/${TARGET_SYS}/libgcc; oe_runmake enable-execute-stack.c unwind.h md-unwind-support.h sfp-machine.h gthr-default.h)
> > diff --git a/meta/recipes-devtools/gcc/gcc-target.inc b/meta/recipes-devtools/gcc/gcc-target.inc
> > index e9187fc444b9..bd7506fe3a55 100644
> > --- a/meta/recipes-devtools/gcc/gcc-target.inc
> > +++ b/meta/recipes-devtools/gcc/gcc-target.inc
> > @@ -132,20 +132,8 @@ FILES:${PN}-doc = "\
> >  "
> >  
> >  do_compile () {
> > -	# Prevent full target sysroot path from being used in configargs.h header,
> > -	# as it will be rewritten when used by other sysroots preventing support
> > -	# for gcc plugins. Additionally the path is embeddeded into the output
> > -	# binary, this prevents building a reproducible binary.
> > -	oe_runmake configure-gcc
> > -	sed -i 's@${STAGING_DIR_TARGET}@/@g' ${B}/gcc/configargs.h
> > -	sed -i 's@${STAGING_DIR_HOST}@/@g' ${B}/gcc/configargs.h
> > -
> > -	# Prevent sysroot/workdir paths from being used in checksum-options.
> > -	# checksum-options is used to generate a checksum which is embedded into
> > -	# the output binary.
> > -	oe_runmake TARGET-gcc=checksum-options all-gcc
> > -	sed -i 's@${DEBUG_PREFIX_MAP}@@g' ${B}/gcc/checksum-options
> > -	sed -i 's@${STAGING_DIR_TARGET}@/@g' ${B}/gcc/checksum-options
> > +	remove_sysroot_paths_from_configargs
> > +	remove_sysroot_paths_from_checksum_options '${STAGING_DIR_TARGET}'
> >  
> >  	oe_runmake all-host
> >  }
> > -- 
> > 2.35.6

Re: [OE-core] [PATCH v2] gcc: remove paths to sysroot from configargs.h and checksum-options for gcc-cross-canadian

[Oleksandr Hnatiuk -X (ohnatiuk - GLOBALLOGIC INC at Cisco)]

[-- Attachment #1: Type: text/plain, Size: 2114 bytes --]

Hi Denys,

I am aware of this issue. As stated in my commit, it is designed to improve reproducibility of gcc-cross-canadian as opposed to fully fixing it.

I have another patch I was going to submit which fully fixes this package by modifying the code of gcc's fixincludes program to not include full source directory, only source file name. However, removing these files entirely as you suggest will also work.

Of course, it would be best to fix this in code and submit the same patch to gcc upstream to fix this issue for everyone. However, this would take more time and require us to maintain another patch in openembedded-core. Not sure which approach is preferred in OE.

Regards,
Oleksandr

________________________________
From: openembedded-core@lists.openembedded.org <openembedded-core@lists.openembedded.org> on behalf of Denys Dmytriyenko <denis@denix.org>
Sent: Thursday, July 18, 2024 3:21 AM
To: Oleksandr Hnatiuk -X (ohnatiuk - GLOBALLOGIC INC at Cisco) <ohnatiuk@cisco.com>; Richard Purdie <richard.purdie@linuxfoundation.org>
Cc: openembedded-core@lists.openembedded.org <openembedded-core@lists.openembedded.org>
Subject: Re: [OE-core] [PATCH v2] gcc: remove paths to sysroot from configargs.h and checksum-options for gcc-cross-canadian

FWIW, I'm testing master-next now with your patch applied and I'm also seeing
the same buildpaths QA error in .../include-fixed/pthread.h

ERROR: gcc-cross-canadian-aarch64-14.1.0-r0 do_package_qa: QA Issue: File /usr/local/oe-sdk-hardcoded-buildpath/sysroots/x86_64-pokysdk-linux/usr/lib/aarch64-poky-linux/gcc/aarch64-poky-linux/14.1.0/include-fixed/pthread.h in package gcc-cross-canadian-aarch64 contains reference to TMPDIR [buildpaths]
ERROR: gcc-cross-canadian-aarch64-14.1.0-r0 do_package_qa: Fatal QA errors were found, failing task.
ERROR: Logfile of failure stored in: /OE/poky-master/build/tmp/work/x86_64-nativesdk-pokysdk-linux/gcc-cross-canadian-aarch64/14.1.0/temp/log.do_package_qa.820539
ERROR: Task (/OE/poky-master/meta/recipes-devtools/gcc/gcc-cross-canadian_14.1.bb:do_package_qa) failed with exit code '1'

[-- Attachment #2: Type: text/html, Size: 4735 bytes --]

Re: [OE-core] [PATCH v2] gcc: remove paths to sysroot from configargs.h and checksum-options for gcc-cross-canadian

[Richard Purdie]

On Thu, 2024-07-18 at 07:40 +0000, Oleksandr Hnatiuk via lists.openembedded.org wrote:
> I am aware of this issue. As stated in my commit, it is designed to
> improve reproducibility of gcc-cross-canadian as opposed to fully
> fixing it.
> 
> I have another patch I was going to submit which fully fixes this
> package by modifying the code of gcc's fixincludes program to not
> include full source directory, only source file name. However,
> removing these files entirely as you suggest will also work.
> 
> Of course, it would be best to fix this in code and submit the same
> patch to gcc upstream to fix this issue for everyone. However, this
> would take more time and require us to maintain another patch in
> openembedded-core. Not sure which approach is preferred in OE.

We've now worked out why we didn't see this in our CI and have patches
to fix that but that does mean we need the patches to fix this error
more urgently! If you have something please do send it as I was
thinking I might have to look at this today.

Ideally we'd fix this with something acceptable to upstream, they'd
take that and then we'd drop our changes next time we upgrade gcc.
Maintaining a backported patch is fine, it is the ones that never get
discussed with upstream we've been pushing back against.

Re: deleting the files, the question is whether they're being used. If
they are, we need to keep them. If they are never used, they could be
deleted. I don't know which is the case here.

Cheers,

Richard

Re: [OE-core] [PATCH v2] gcc: remove paths to sysroot from configargs.h and checksum-options for gcc-cross-canadian

[Richard Purdie]

On Wed, 2024-07-17 at 02:56 -0700, Oleksandr Hnatiuk via lists.openembedded.org wrote:
> Apply fixes from gcc-cross (84a78f46d594 and 0ead8cbdfb96) to gcc-cross-canadian.
> This will improve (but not fix) reproducibility of gcc-cross-canadian.
> Also move this code to functions to avoid code duplication.
> 
> Signed-off-by: Oleksandr Hnatiuk <ohnatiuk@cisco.com>
> ---
>  .../gcc/gcc-configure-common.inc              | 19 +++++++++++++++++++
>  .../gcc/gcc-cross-canadian.inc                |  3 +++
>  meta/recipes-devtools/gcc/gcc-cross.inc       | 15 ++-------------
>  meta/recipes-devtools/gcc/gcc-target.inc      | 16 ++--------------
>  4 files changed, 26 insertions(+), 27 deletions(-)
> 
> diff --git a/meta/recipes-devtools/gcc/gcc-configure-common.inc b/meta/recipes-devtools/gcc/gcc-configure-common.inc
> index d77b2ec3d1de..c936d1a9bd7b 100644
> --- a/meta/recipes-devtools/gcc/gcc-configure-common.inc
> +++ b/meta/recipes-devtools/gcc/gcc-configure-common.inc
> @@ -120,3 +120,22 @@ do_configure () {
>  	oe_runconf
>  }
>  
> +remove_sysroot_paths_from_configargs () {
> +	# Prevent sysroot path from being used in configargs.h header, as it will
> +	# be rewritten when used by other sysroots preventing support for gcc
> +	# plugins. Additionally the path is embeddeded into the output binary, this
> +	# prevents building a reproducible binary.
> +	oe_runmake configure-gcc
> +	sed -i 's@${STAGING_DIR_TARGET}@/host@g' ${B}/gcc/configargs.h
> +	sed -i 's@${STAGING_DIR_HOST}@/host@g' ${B}/gcc/configargs.h
> +}
> +
> +remove_sysroot_paths_from_checksum_options () {
> +	# Prevent sysroot/workdir paths from being used in checksum-options.
> +	# checksum-options is used to generate a checksum which is embedded into
> +	# the output binary.
> +	oe_runmake TARGET-gcc=checksum-options all-gcc
> +	sed -i 's@${DEBUG_PREFIX_MAP}@@g' ${B}/gcc/checksum-options
> +	sed -i 's@${1}@/host@g' ${B}/gcc/checksum-options
> +}
> +
> diff --git a/meta/recipes-devtools/gcc/gcc-cross-canadian.inc b/meta/recipes-devtools/gcc/gcc-cross-canadian.inc
> index 7c4233c21157..9528b71272dd 100644
> --- a/meta/recipes-devtools/gcc/gcc-cross-canadian.inc
> +++ b/meta/recipes-devtools/gcc/gcc-cross-canadian.inc
> @@ -64,6 +64,9 @@ do_configure () {
>  }
>  
>  do_compile () {
> +	remove_sysroot_paths_from_configargs
> +	remove_sysroot_paths_from_checksum_options '${STAGING_DIR_HOST}'
> +
>  	oe_runmake all-host configure-target-libgcc
>  	(cd ${B}/${TARGET_SYS}/libgcc; oe_runmake enable-execute-stack.c unwind.h md-unwind-support.h sfp-machine.h gthr-default.h)
>  }
> diff --git a/meta/recipes-devtools/gcc/gcc-cross.inc b/meta/recipes-devtools/gcc/gcc-cross.inc
> index 5b0ca15d4762..f8a7730cf322 100644
> --- a/meta/recipes-devtools/gcc/gcc-cross.inc
> +++ b/meta/recipes-devtools/gcc/gcc-cross.inc
> @@ -62,19 +62,8 @@ do_compile () {
>  	export CXXFLAGS_FOR_TARGET="${TARGET_CXXFLAGS}"
>  	export LDFLAGS_FOR_TARGET="${TARGET_LDFLAGS}"
>  
> -	# Prevent native/host sysroot path from being used in configargs.h header,
> -	# as it will be rewritten when used by other sysroots preventing support
> -	# for gcc plugins
> -	oe_runmake configure-gcc
> -	sed -i 's@${STAGING_DIR_TARGET}@/host@g' ${B}/gcc/configargs.h
> -	sed -i 's@${STAGING_DIR_HOST}@/host@g' ${B}/gcc/configargs.h
> -
> -	# Prevent sysroot/workdir paths from being used in checksum-options.
> -	# checksum-options is used to generate a checksum which is embedded into
> -	# the output binary.
> -	oe_runmake TARGET-gcc=checksum-options all-gcc
> -	sed -i 's@${DEBUG_PREFIX_MAP}@@g' ${B}/gcc/checksum-options
> -	sed -i 's@${STAGING_DIR_HOST}@/host@g' ${B}/gcc/checksum-options
> +	remove_sysroot_paths_from_configargs
> +	remove_sysroot_paths_from_checksum_options '${STAGING_DIR_HOST}'
>  
>  	oe_runmake all-host configure-target-libgcc
>  	(cd ${B}/${TARGET_SYS}/libgcc; oe_runmake enable-execute-stack.c unwind.h md-unwind-support.h sfp-machine.h gthr-default.h)
> diff --git a/meta/recipes-devtools/gcc/gcc-target.inc b/meta/recipes-devtools/gcc/gcc-target.inc
> index e9187fc444b9..bd7506fe3a55 100644
> --- a/meta/recipes-devtools/gcc/gcc-target.inc
> +++ b/meta/recipes-devtools/gcc/gcc-target.inc
> @@ -132,20 +132,8 @@ FILES:${PN}-doc = "\
>  "
>  
>  do_compile () {
> -	# Prevent full target sysroot path from being used in configargs.h header,
> -	# as it will be rewritten when used by other sysroots preventing support
> -	# for gcc plugins. Additionally the path is embeddeded into the output
> -	# binary, this prevents building a reproducible binary.
> -	oe_runmake configure-gcc
> -	sed -i 's@${STAGING_DIR_TARGET}@/@g' ${B}/gcc/configargs.h
> -	sed -i 's@${STAGING_DIR_HOST}@/@g' ${B}/gcc/configargs.h
> -
> -	# Prevent sysroot/workdir paths from being used in checksum-options.
> -	# checksum-options is used to generate a checksum which is embedded into
> -	# the output binary.
> -	oe_runmake TARGET-gcc=checksum-options all-gcc
> -	sed -i 's@${DEBUG_PREFIX_MAP}@@g' ${B}/gcc/checksum-options
> -	sed -i 's@${STAGING_DIR_TARGET}@/@g' ${B}/gcc/checksum-options
> +	remove_sysroot_paths_from_configargs
> +	remove_sysroot_paths_from_checksum_options '${STAGING_DIR_TARGET}'
>  
>  	oe_runmake all-host
>  }

I think this also causes a reproducibility issue. In testing this patch seemed to cause:

https://autobuilder.yoctoproject.org/typhoon/#/builders/117/builds/5079/steps/13/logs/stdio

which leads to:

http://autobuilder.yocto.io/pub/repro-fail/oe-reproducible-20240717-49lmc_9g/packages/diff-html/

which is a difference in EXPORTED_CONST·unsigned·char·executable_checksum[16].

Cheers,

Richard