* Distro wide LTO broken?
@ 2025-09-29 1:22 William Kennington
2025-09-29 18:42 ` [OE-core] " Khem Raj
0 siblings, 1 reply; 8+ messages in thread
From: William Kennington @ 2025-09-29 1:22 UTC (permalink / raw)
To: openembedded-core
If I add the following to my distro, the build breaks for the openssl
and other core packages due to having build path impurities in the
static libraries created.
```
require conf/distro/include/lto.inc
DISTRO_FEATURES += "lto"
```
The errors I get are all like
```
ERROR: openssl-3.5.2-r0 do_package_qa: QA Issue: File
/usr/lib/libssl.a in package openssl-staticdev contains reference to
TMPDIR [buildpaths]
ERROR: openssl-3.5.2-r0 do_package_qa: QA Issue: File
/usr/lib/libcrypto.a in package openssl-staticdev contains reference
to TMPDIR [buildpaths]
ERROR: openssl-3.5.2-r0 do_package_qa: Fatal QA errors were found, failing task.
ERROR: Logfile of failure stored in:
/home/builder/yocto/build/tmp/work/cortexa76-cortexa55-crypto-oe-linux/openssl/3.5.2/temp/log.do_package_qa.118335
ERROR: Task (/srv/yocto/poky/meta/recipes-connectivity/openssl/openssl_3.5.2.bb:do_package_qa)
failed with exit code '1'
```
Probably there is just some kind of flag missing in the LTO build
options to filter out the build directory, as -ffile-prefix-map and
whatnot are already present in the builds. I haven't yet figured out
exactly what is contributing to this impurity.
Prior to rebasing on the latest master, LTO was working fine globally.
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [OE-core] Distro wide LTO broken?
2025-09-29 1:22 Distro wide LTO broken? William Kennington
@ 2025-09-29 18:42 ` Khem Raj
2025-09-30 0:19 ` William Kennington
0 siblings, 1 reply; 8+ messages in thread
From: Khem Raj @ 2025-09-29 18:42 UTC (permalink / raw)
To: william; +Cc: openembedded-core
On Mon, Sep 29, 2025 at 3:51 AM William Kennington via
lists.openembedded.org
<william=wkennington.com@lists.openembedded.org> wrote:
>
> If I add the following to my distro, the build breaks for the openssl
> and other core packages due to having build path impurities in the
> static libraries created.
>
> ```
> require conf/distro/include/lto.inc
> DISTRO_FEATURES += "lto"
> ```
>
> The errors I get are all like
> ```
> ERROR: openssl-3.5.2-r0 do_package_qa: QA Issue: File
> /usr/lib/libssl.a in package openssl-staticdev contains reference to
> TMPDIR [buildpaths]
> ERROR: openssl-3.5.2-r0 do_package_qa: QA Issue: File
> /usr/lib/libcrypto.a in package openssl-staticdev contains reference
> to TMPDIR [buildpaths]
> ERROR: openssl-3.5.2-r0 do_package_qa: Fatal QA errors were found, failing task.
> ERROR: Logfile of failure stored in:
> /home/builder/yocto/build/tmp/work/cortexa76-cortexa55-crypto-oe-linux/openssl/3.5.2/temp/log.do_package_qa.118335
> ERROR: Task (/srv/yocto/poky/meta/recipes-connectivity/openssl/openssl_3.5.2.bb:do_package_qa)
> failed with exit code '1'
> ```
>
> Probably there is just some kind of flag missing in the LTO build
> options to filter out the build directory, as -ffile-prefix-map and
> whatnot are already present in the builds. I haven't yet figured out
> exactly what is contributing to this impurity.
>
Inspect the .a files ( maybe using strings or binary viewing tools
like xxd ), with LTO they are flat objects with additional information
for
LTO to work and I think it will give more information.
> Prior to rebasing on the latest master, LTO was working fine globally.
what is the delta we are talking about here ?
>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#224137): https://lists.openembedded.org/g/openembedded-core/message/224137
> Mute This Topic: https://lists.openembedded.org/mt/115492623/1997914
> Group Owner: openembedded-core+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [raj.khem@gmail.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [OE-core] Distro wide LTO broken?
2025-09-29 18:42 ` [OE-core] " Khem Raj
@ 2025-09-30 0:19 ` William Kennington
2025-09-30 2:58 ` Khem Raj
0 siblings, 1 reply; 8+ messages in thread
From: William Kennington @ 2025-09-30 0:19 UTC (permalink / raw)
To: Khem Raj; +Cc: openembedded-core
On Mon, Sep 29, 2025 at 11:43 AM Khem Raj <raj.khem@gmail.com> wrote:
>
> On Mon, Sep 29, 2025 at 3:51 AM William Kennington via
> lists.openembedded.org
> <william=wkennington.com@lists.openembedded.org> wrote:
> >
> > If I add the following to my distro, the build breaks for the openssl
> > and other core packages due to having build path impurities in the
> > static libraries created.
> >
> > ```
> > require conf/distro/include/lto.inc
> > DISTRO_FEATURES += "lto"
> > ```
> >
> > The errors I get are all like
> > ```
> > ERROR: openssl-3.5.2-r0 do_package_qa: QA Issue: File
> > /usr/lib/libssl.a in package openssl-staticdev contains reference to
> > TMPDIR [buildpaths]
> > ERROR: openssl-3.5.2-r0 do_package_qa: QA Issue: File
> > /usr/lib/libcrypto.a in package openssl-staticdev contains reference
> > to TMPDIR [buildpaths]
> > ERROR: openssl-3.5.2-r0 do_package_qa: Fatal QA errors were found, failing task.
> > ERROR: Logfile of failure stored in:
> > /home/builder/yocto/build/tmp/work/cortexa76-cortexa55-crypto-oe-linux/openssl/3.5.2/temp/log.do_package_qa.118335
> > ERROR: Task (/srv/yocto/poky/meta/recipes-connectivity/openssl/openssl_3.5.2.bb:do_package_qa)
> > failed with exit code '1'
> > ```
> >
> > Probably there is just some kind of flag missing in the LTO build
> > options to filter out the build directory, as -ffile-prefix-map and
> > whatnot are already present in the builds. I haven't yet figured out
> > exactly what is contributing to this impurity.
> >
>
> Inspect the .a files ( maybe using strings or binary viewing tools
> like xxd ), with LTO they are flat objects with additional information
> for
> LTO to work and I think it will give more information.
Yeah, I believe it's just a single remapping we are missing somewhere
that is allowing the extra string to be added into the binary. It's
been a while since I've worked on this kind of impurity stripping, so
it will probably take a bit to get used to the tooling again.
Hopefully I find the root cause and we can strip it out
An example invocation looks like the following. All other object files
I looked at share the same impurity which ends up in the static
library later
```
aarch64-oe-linux-gcc -mcpu=cortex-a76.cortex-a55+crypto
-mbranch-protection=standard -fstack-protector-strong -O2
-D_FORTIFY_SOURCE=2 -Wformat -Wformat-security -Werror=format-security
--sysroot=/home/builder/yocto/build/tmp/work/cortexa76-cortexa55-crypto-oe-linux/openssl/3.5.2/recipe-sysroot
-I. -Iinclude -Iapps/include -I../sources/openssl-3.5.2
-I../sources/openssl-3.5.2/include
-I../sources/openssl-3.5.2/apps/include -fPIC -pthread
-Wa,--noexecstack -O2 -g -flto -ffat-lto-objects -fuse-linker-plugin
-ffile-prefix-map=/home/builder/yocto/build/tmp/work/cortexa76-cortexa55-crypto-oe-linux/openssl/3.5.2/s
ources/openssl-3.5.2=/usr/src/debug/openssl/3.5.2
-ffile-prefix-map=/home/builder/yocto/build/tmp/work/cortexa76-cortexa55-
crypto-oe-linux/openssl/3.5.2/build=/usr/src/debug/openssl/3.5.2
-ffile-prefix-map=/home/builder/yocto/build/tmp/work/cortexa76-cortexa55-crypto-oe-linux/openssl/3.5.2/recipe-sysroot=
-ffile-prefix-map=/home/builder/yocto/build/tmp/work/cortexa76-cortexa55-crypto-oe-linux/openssl/3.5.2/recipe-sysroot-native=
-pipe -DOPENSSL_USE_NODELETE -DOPENSSL_PIC
-DOPENSSLDIR="\"/usr/lib/ssl-3\""
-DENGINESDIR="\"/usr/lib/engines-3\""
-DMODULESDIR="\"/usr/lib/ossl-modules\"" -DOPENSSL_BUILDING_OPENSSL
-DNDEBUG -MMD -MF apps/lib/libapps-lib-app_x509.d.tmp -c -o
apps/lib/libapps-lib-app_x509.o ../sources/openssl-3.5.2/apps/l
ib/app_x509.c
```
and the partial strings dump with the impurity looks like
```
vtmp
unsigned int
/usr/src/debug/openssl/3.5.2
../sources/openssl-3.5.2/apps/lib/app_x509.c
/home/builder/yocto/build/tmp/work/cortexa76-cortexa55-crypto-oe-linux/openssl/3.5.2/build
include/openssl
/usr/lib/aarch64-oe-linux/gcc/aarch64-oe-linux/15.2.0/include
../sources/openssl-3.5.2/include/openssl
/usr/include
../sources/openssl-3.5.2/apps/lib
../sources/openssl-3.5.2/apps/include
app_x509.c
```
>
> > Prior to rebasing on the latest master, LTO was working fine globally.
>
> what is the delta we are talking about here ?
Sadly this was long ago `25d60ac6f61cc186b4c5a961554bee583736fd17`
from early 2024. Doing any kind of bisection is probably pointless so
I just need to figure out this new impurity (or maybe just the check
got added since then)
>
> >
> > -=-=-=-=-=-=-=-=-=-=-=-
> > Links: You receive all messages sent to this group.
> > View/Reply Online (#224137): https://lists.openembedded.org/g/openembedded-core/message/224137
> > Mute This Topic: https://lists.openembedded.org/mt/115492623/1997914
> > Group Owner: openembedded-core+owner@lists.openembedded.org
> > Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [raj.khem@gmail.com]
> > -=-=-=-=-=-=-=-=-=-=-=-
> >
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [OE-core] Distro wide LTO broken?
2025-09-30 0:19 ` William Kennington
@ 2025-09-30 2:58 ` Khem Raj
2025-09-30 7:49 ` William Kennington
0 siblings, 1 reply; 8+ messages in thread
From: Khem Raj @ 2025-09-30 2:58 UTC (permalink / raw)
To: William Kennington; +Cc: openembedded-core
On Mon, Sep 29, 2025 at 5:19 PM William Kennington
<william@wkennington.com> wrote:
>
> On Mon, Sep 29, 2025 at 11:43 AM Khem Raj <raj.khem@gmail.com> wrote:
> >
> > On Mon, Sep 29, 2025 at 3:51 AM William Kennington via
> > lists.openembedded.org
> > <william=wkennington.com@lists.openembedded.org> wrote:
> > >
> > > If I add the following to my distro, the build breaks for the openssl
> > > and other core packages due to having build path impurities in the
> > > static libraries created.
> > >
> > > ```
> > > require conf/distro/include/lto.inc
> > > DISTRO_FEATURES += "lto"
> > > ```
> > >
> > > The errors I get are all like
> > > ```
> > > ERROR: openssl-3.5.2-r0 do_package_qa: QA Issue: File
> > > /usr/lib/libssl.a in package openssl-staticdev contains reference to
> > > TMPDIR [buildpaths]
> > > ERROR: openssl-3.5.2-r0 do_package_qa: QA Issue: File
> > > /usr/lib/libcrypto.a in package openssl-staticdev contains reference
> > > to TMPDIR [buildpaths]
> > > ERROR: openssl-3.5.2-r0 do_package_qa: Fatal QA errors were found, failing task.
> > > ERROR: Logfile of failure stored in:
> > > /home/builder/yocto/build/tmp/work/cortexa76-cortexa55-crypto-oe-linux/openssl/3.5.2/temp/log.do_package_qa.118335
> > > ERROR: Task (/srv/yocto/poky/meta/recipes-connectivity/openssl/openssl_3.5.2.bb:do_package_qa)
> > > failed with exit code '1'
> > > ```
> > >
> > > Probably there is just some kind of flag missing in the LTO build
> > > options to filter out the build directory, as -ffile-prefix-map and
> > > whatnot are already present in the builds. I haven't yet figured out
> > > exactly what is contributing to this impurity.
> > >
> >
> > Inspect the .a files ( maybe using strings or binary viewing tools
> > like xxd ), with LTO they are flat objects with additional information
> > for
> > LTO to work and I think it will give more information.
>
> Yeah, I believe it's just a single remapping we are missing somewhere
> that is allowing the extra string to be added into the binary. It's
> been a while since I've worked on this kind of impurity stripping, so
> it will probably take a bit to get used to the tooling again.
> Hopefully I find the root cause and we can strip it out
>
> An example invocation looks like the following. All other object files
> I looked at share the same impurity which ends up in the static
> library later
> ```
> aarch64-oe-linux-gcc -mcpu=cortex-a76.cortex-a55+crypto
> -mbranch-protection=standard -fstack-protector-strong -O2
> -D_FORTIFY_SOURCE=2 -Wformat -Wformat-security -Werror=format-security
> --sysroot=/home/builder/yocto/build/tmp/work/cortexa76-cortexa55-crypto-oe-linux/openssl/3.5.2/recipe-sysroot
> -I. -Iinclude -Iapps/include -I../sources/openssl-3.5.2
> -I../sources/openssl-3.5.2/include
> -I../sources/openssl-3.5.2/apps/include -fPIC -pthread
> -Wa,--noexecstack -O2 -g -flto -ffat-lto-objects -fuse-linker-plugin
> -ffile-prefix-map=/home/builder/yocto/build/tmp/work/cortexa76-cortexa55-crypto-oe-linux/openssl/3.5.2/s
> ources/openssl-3.5.2=/usr/src/debug/openssl/3.5.2
> -ffile-prefix-map=/home/builder/yocto/build/tmp/work/cortexa76-cortexa55-
> crypto-oe-linux/openssl/3.5.2/build=/usr/src/debug/openssl/3.5.2
> -ffile-prefix-map=/home/builder/yocto/build/tmp/work/cortexa76-cortexa55-crypto-oe-linux/openssl/3.5.2/recipe-sysroot=
> -ffile-prefix-map=/home/builder/yocto/build/tmp/work/cortexa76-cortexa55-crypto-oe-linux/openssl/3.5.2/recipe-sysroot-native=
> -pipe -DOPENSSL_USE_NODELETE -DOPENSSL_PIC
> -DOPENSSLDIR="\"/usr/lib/ssl-3\""
> -DENGINESDIR="\"/usr/lib/engines-3\""
> -DMODULESDIR="\"/usr/lib/ossl-modules\"" -DOPENSSL_BUILDING_OPENSSL
> -DNDEBUG -MMD -MF apps/lib/libapps-lib-app_x509.d.tmp -c -o
> apps/lib/libapps-lib-app_x509.o ../sources/openssl-3.5.2/apps/l
> ib/app_x509.c
> ```
> and the partial strings dump with the impurity looks like
> ```
> vtmp
> unsigned int
> /usr/src/debug/openssl/3.5.2
> ../sources/openssl-3.5.2/apps/lib/app_x509.c
> /home/builder/yocto/build/tmp/work/cortexa76-cortexa55-crypto-oe-linux/openssl/3.5.2/build
> include/openssl
> /usr/lib/aarch64-oe-linux/gcc/aarch64-oe-linux/15.2.0/include
> ../sources/openssl-3.5.2/include/openssl
> /usr/include
> ../sources/openssl-3.5.2/apps/lib
> ../sources/openssl-3.5.2/apps/include
> app_x509.c
> ```
If you already know its coming from app_x509.c, then you have
narrowed it down a step. Next, would be
to build just this file with -dD -E option and inspect if it contains
any live string containing
"/home/builder/yocto/build/tmp/work/cortexa76-cortexa55-crypto-oe-linux/openssl/3.5.2/build"
if it does not find it. then perhaps build the assembly output with
-fverbose-asm -S
option and look for this string in the generated asm file.
> >
> > > Prior to rebasing on the latest master, LTO was working fine globally.
> >
> > what is the delta we are talking about here ?
>
> Sadly this was long ago `25d60ac6f61cc186b4c5a961554bee583736fd17`
> from early 2024. Doing any kind of bisection is probably pointless so
> I just need to figure out this new impurity (or maybe just the check
> got added since then)
Yeah that will take too long to bisect so leave that aside.
>
> >
> > >
> > > -=-=-=-=-=-=-=-=-=-=-=-
> > > Links: You receive all messages sent to this group.
> > > View/Reply Online (#224137): https://lists.openembedded.org/g/openembedded-core/message/224137
> > > Mute This Topic: https://lists.openembedded.org/mt/115492623/1997914
> > > Group Owner: openembedded-core+owner@lists.openembedded.org
> > > Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [raj.khem@gmail.com]
> > > -=-=-=-=-=-=-=-=-=-=-=-
> > >
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [OE-core] Distro wide LTO broken?
2025-09-30 2:58 ` Khem Raj
@ 2025-09-30 7:49 ` William Kennington
2025-09-30 14:55 ` Khem Raj
0 siblings, 1 reply; 8+ messages in thread
From: William Kennington @ 2025-09-30 7:49 UTC (permalink / raw)
To: Khem Raj; +Cc: openembedded-core
On Mon, Sep 29, 2025 at 7:58 PM Khem Raj <raj.khem@gmail.com> wrote:
>
> On Mon, Sep 29, 2025 at 5:19 PM William Kennington
> <william@wkennington.com> wrote:
> >
> > On Mon, Sep 29, 2025 at 11:43 AM Khem Raj <raj.khem@gmail.com> wrote:
> > >
> > > On Mon, Sep 29, 2025 at 3:51 AM William Kennington via
> > > lists.openembedded.org
> > > <william=wkennington.com@lists.openembedded.org> wrote:
> > > >
> > > > If I add the following to my distro, the build breaks for the openssl
> > > > and other core packages due to having build path impurities in the
> > > > static libraries created.
> > > >
> > > > ```
> > > > require conf/distro/include/lto.inc
> > > > DISTRO_FEATURES += "lto"
> > > > ```
> > > >
> > > > The errors I get are all like
> > > > ```
> > > > ERROR: openssl-3.5.2-r0 do_package_qa: QA Issue: File
> > > > /usr/lib/libssl.a in package openssl-staticdev contains reference to
> > > > TMPDIR [buildpaths]
> > > > ERROR: openssl-3.5.2-r0 do_package_qa: QA Issue: File
> > > > /usr/lib/libcrypto.a in package openssl-staticdev contains reference
> > > > to TMPDIR [buildpaths]
> > > > ERROR: openssl-3.5.2-r0 do_package_qa: Fatal QA errors were found, failing task.
> > > > ERROR: Logfile of failure stored in:
> > > > /home/builder/yocto/build/tmp/work/cortexa76-cortexa55-crypto-oe-linux/openssl/3.5.2/temp/log.do_package_qa.118335
> > > > ERROR: Task (/srv/yocto/poky/meta/recipes-connectivity/openssl/openssl_3.5.2.bb:do_package_qa)
> > > > failed with exit code '1'
> > > > ```
> > > >
> > > > Probably there is just some kind of flag missing in the LTO build
> > > > options to filter out the build directory, as -ffile-prefix-map and
> > > > whatnot are already present in the builds. I haven't yet figured out
> > > > exactly what is contributing to this impurity.
> > > >
> > >
> > > Inspect the .a files ( maybe using strings or binary viewing tools
> > > like xxd ), with LTO they are flat objects with additional information
> > > for
> > > LTO to work and I think it will give more information.
> >
> > Yeah, I believe it's just a single remapping we are missing somewhere
> > that is allowing the extra string to be added into the binary. It's
> > been a while since I've worked on this kind of impurity stripping, so
> > it will probably take a bit to get used to the tooling again.
> > Hopefully I find the root cause and we can strip it out
> >
> > An example invocation looks like the following. All other object files
> > I looked at share the same impurity which ends up in the static
> > library later
> > ```
> > aarch64-oe-linux-gcc -mcpu=cortex-a76.cortex-a55+crypto
> > -mbranch-protection=standard -fstack-protector-strong -O2
> > -D_FORTIFY_SOURCE=2 -Wformat -Wformat-security -Werror=format-security
> > --sysroot=/home/builder/yocto/build/tmp/work/cortexa76-cortexa55-crypto-oe-linux/openssl/3.5.2/recipe-sysroot
> > -I. -Iinclude -Iapps/include -I../sources/openssl-3.5.2
> > -I../sources/openssl-3.5.2/include
> > -I../sources/openssl-3.5.2/apps/include -fPIC -pthread
> > -Wa,--noexecstack -O2 -g -flto -ffat-lto-objects -fuse-linker-plugin
> > -ffile-prefix-map=/home/builder/yocto/build/tmp/work/cortexa76-cortexa55-crypto-oe-linux/openssl/3.5.2/s
> > ources/openssl-3.5.2=/usr/src/debug/openssl/3.5.2
> > -ffile-prefix-map=/home/builder/yocto/build/tmp/work/cortexa76-cortexa55-
> > crypto-oe-linux/openssl/3.5.2/build=/usr/src/debug/openssl/3.5.2
> > -ffile-prefix-map=/home/builder/yocto/build/tmp/work/cortexa76-cortexa55-crypto-oe-linux/openssl/3.5.2/recipe-sysroot=
> > -ffile-prefix-map=/home/builder/yocto/build/tmp/work/cortexa76-cortexa55-crypto-oe-linux/openssl/3.5.2/recipe-sysroot-native=
> > -pipe -DOPENSSL_USE_NODELETE -DOPENSSL_PIC
> > -DOPENSSLDIR="\"/usr/lib/ssl-3\""
> > -DENGINESDIR="\"/usr/lib/engines-3\""
> > -DMODULESDIR="\"/usr/lib/ossl-modules\"" -DOPENSSL_BUILDING_OPENSSL
> > -DNDEBUG -MMD -MF apps/lib/libapps-lib-app_x509.d.tmp -c -o
> > apps/lib/libapps-lib-app_x509.o ../sources/openssl-3.5.2/apps/l
> > ib/app_x509.c
> > ```
> > and the partial strings dump with the impurity looks like
> > ```
> > vtmp
> > unsigned int
> > /usr/src/debug/openssl/3.5.2
> > ../sources/openssl-3.5.2/apps/lib/app_x509.c
> > /home/builder/yocto/build/tmp/work/cortexa76-cortexa55-crypto-oe-linux/openssl/3.5.2/build
> > include/openssl
> > /usr/lib/aarch64-oe-linux/gcc/aarch64-oe-linux/15.2.0/include
> > ../sources/openssl-3.5.2/include/openssl
> > /usr/include
> > ../sources/openssl-3.5.2/apps/lib
> > ../sources/openssl-3.5.2/apps/include
> > app_x509.c
> > ```
>
> If you already know its coming from app_x509.c, then you have
> narrowed it down a step. Next, would be
To be fair, it's basically every invocation of `gcc ... -c` that
produces it. I just picked an arbitrary file that was easy to find
that happens to carry the issue.
> to build just this file with -dD -E option and inspect if it contains
> any live string containing
> "/home/builder/yocto/build/tmp/work/cortexa76-cortexa55-crypto-oe-linux/openssl/3.5.2/build"
> if it does not find it. then perhaps build the assembly output with
> -fverbose-asm -S
> option and look for this string in the generated asm file.
I ran this, and didn't see anything containing the path. I think it's
just a bug with the handling of `-ffile-prefix-map` because it looks
like the PWD being used is exactly listed
Tried the latest snapshot of gcc 15 too just to see if they fixed
anything. Nothing in the git log looks relevant and it doesn't fix the
issue.
Did some more digging and found out that if i remove
`-ffat-lto-objects` I don't have the issue anymore.
Seems like maybe this is related to
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=109805
>
> > >
> > > > Prior to rebasing on the latest master, LTO was working fine globally.
> > >
> > > what is the delta we are talking about here ?
> >
> > Sadly this was long ago `25d60ac6f61cc186b4c5a961554bee583736fd17`
> > from early 2024. Doing any kind of bisection is probably pointless so
> > I just need to figure out this new impurity (or maybe just the check
> > got added since then)
>
> Yeah that will take too long to bisect so leave that aside.
>
> >
> > >
> > > >
> > > > -=-=-=-=-=-=-=-=-=-=-=-
> > > > Links: You receive all messages sent to this group.
> > > > View/Reply Online (#224137): https://lists.openembedded.org/g/openembedded-core/message/224137
> > > > Mute This Topic: https://lists.openembedded.org/mt/115492623/1997914
> > > > Group Owner: openembedded-core+owner@lists.openembedded.org
> > > > Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [raj.khem@gmail.com]
> > > > -=-=-=-=-=-=-=-=-=-=-=-
> > > >
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [OE-core] Distro wide LTO broken?
2025-09-30 7:49 ` William Kennington
@ 2025-09-30 14:55 ` Khem Raj
2025-11-06 10:07 ` mark.yang
0 siblings, 1 reply; 8+ messages in thread
From: Khem Raj @ 2025-09-30 14:55 UTC (permalink / raw)
To: William Kennington; +Cc: openembedded-core
On Tue, Sep 30, 2025 at 12:50 AM William Kennington
<william@wkennington.com> wrote:
>
> On Mon, Sep 29, 2025 at 7:58 PM Khem Raj <raj.khem@gmail.com> wrote:
> >
> > On Mon, Sep 29, 2025 at 5:19 PM William Kennington
> > <william@wkennington.com> wrote:
> > >
> > > On Mon, Sep 29, 2025 at 11:43 AM Khem Raj <raj.khem@gmail.com> wrote:
> > > >
> > > > On Mon, Sep 29, 2025 at 3:51 AM William Kennington via
> > > > lists.openembedded.org
> > > > <william=wkennington.com@lists.openembedded.org> wrote:
> > > > >
> > > > > If I add the following to my distro, the build breaks for the openssl
> > > > > and other core packages due to having build path impurities in the
> > > > > static libraries created.
> > > > >
> > > > > ```
> > > > > require conf/distro/include/lto.inc
> > > > > DISTRO_FEATURES += "lto"
> > > > > ```
> > > > >
> > > > > The errors I get are all like
> > > > > ```
> > > > > ERROR: openssl-3.5.2-r0 do_package_qa: QA Issue: File
> > > > > /usr/lib/libssl.a in package openssl-staticdev contains reference to
> > > > > TMPDIR [buildpaths]
> > > > > ERROR: openssl-3.5.2-r0 do_package_qa: QA Issue: File
> > > > > /usr/lib/libcrypto.a in package openssl-staticdev contains reference
> > > > > to TMPDIR [buildpaths]
> > > > > ERROR: openssl-3.5.2-r0 do_package_qa: Fatal QA errors were found, failing task.
> > > > > ERROR: Logfile of failure stored in:
> > > > > /home/builder/yocto/build/tmp/work/cortexa76-cortexa55-crypto-oe-linux/openssl/3.5.2/temp/log.do_package_qa.118335
> > > > > ERROR: Task (/srv/yocto/poky/meta/recipes-connectivity/openssl/openssl_3.5.2.bb:do_package_qa)
> > > > > failed with exit code '1'
> > > > > ```
> > > > >
> > > > > Probably there is just some kind of flag missing in the LTO build
> > > > > options to filter out the build directory, as -ffile-prefix-map and
> > > > > whatnot are already present in the builds. I haven't yet figured out
> > > > > exactly what is contributing to this impurity.
> > > > >
> > > >
> > > > Inspect the .a files ( maybe using strings or binary viewing tools
> > > > like xxd ), with LTO they are flat objects with additional information
> > > > for
> > > > LTO to work and I think it will give more information.
> > >
> > > Yeah, I believe it's just a single remapping we are missing somewhere
> > > that is allowing the extra string to be added into the binary. It's
> > > been a while since I've worked on this kind of impurity stripping, so
> > > it will probably take a bit to get used to the tooling again.
> > > Hopefully I find the root cause and we can strip it out
> > >
> > > An example invocation looks like the following. All other object files
> > > I looked at share the same impurity which ends up in the static
> > > library later
> > > ```
> > > aarch64-oe-linux-gcc -mcpu=cortex-a76.cortex-a55+crypto
> > > -mbranch-protection=standard -fstack-protector-strong -O2
> > > -D_FORTIFY_SOURCE=2 -Wformat -Wformat-security -Werror=format-security
> > > --sysroot=/home/builder/yocto/build/tmp/work/cortexa76-cortexa55-crypto-oe-linux/openssl/3.5.2/recipe-sysroot
> > > -I. -Iinclude -Iapps/include -I../sources/openssl-3.5.2
> > > -I../sources/openssl-3.5.2/include
> > > -I../sources/openssl-3.5.2/apps/include -fPIC -pthread
> > > -Wa,--noexecstack -O2 -g -flto -ffat-lto-objects -fuse-linker-plugin
> > > -ffile-prefix-map=/home/builder/yocto/build/tmp/work/cortexa76-cortexa55-crypto-oe-linux/openssl/3.5.2/s
> > > ources/openssl-3.5.2=/usr/src/debug/openssl/3.5.2
> > > -ffile-prefix-map=/home/builder/yocto/build/tmp/work/cortexa76-cortexa55-
> > > crypto-oe-linux/openssl/3.5.2/build=/usr/src/debug/openssl/3.5.2
> > > -ffile-prefix-map=/home/builder/yocto/build/tmp/work/cortexa76-cortexa55-crypto-oe-linux/openssl/3.5.2/recipe-sysroot=
> > > -ffile-prefix-map=/home/builder/yocto/build/tmp/work/cortexa76-cortexa55-crypto-oe-linux/openssl/3.5.2/recipe-sysroot-native=
> > > -pipe -DOPENSSL_USE_NODELETE -DOPENSSL_PIC
> > > -DOPENSSLDIR="\"/usr/lib/ssl-3\""
> > > -DENGINESDIR="\"/usr/lib/engines-3\""
> > > -DMODULESDIR="\"/usr/lib/ossl-modules\"" -DOPENSSL_BUILDING_OPENSSL
> > > -DNDEBUG -MMD -MF apps/lib/libapps-lib-app_x509.d.tmp -c -o
> > > apps/lib/libapps-lib-app_x509.o ../sources/openssl-3.5.2/apps/l
> > > ib/app_x509.c
> > > ```
> > > and the partial strings dump with the impurity looks like
> > > ```
> > > vtmp
> > > unsigned int
> > > /usr/src/debug/openssl/3.5.2
> > > ../sources/openssl-3.5.2/apps/lib/app_x509.c
> > > /home/builder/yocto/build/tmp/work/cortexa76-cortexa55-crypto-oe-linux/openssl/3.5.2/build
> > > include/openssl
> > > /usr/lib/aarch64-oe-linux/gcc/aarch64-oe-linux/15.2.0/include
> > > ../sources/openssl-3.5.2/include/openssl
> > > /usr/include
> > > ../sources/openssl-3.5.2/apps/lib
> > > ../sources/openssl-3.5.2/apps/include
> > > app_x509.c
> > > ```
> >
> > If you already know its coming from app_x509.c, then you have
> > narrowed it down a step. Next, would be
> To be fair, it's basically every invocation of `gcc ... -c` that
> produces it. I just picked an arbitrary file that was easy to find
> that happens to carry the issue.
>
> > to build just this file with -dD -E option and inspect if it contains
> > any live string containing
> > "/home/builder/yocto/build/tmp/work/cortexa76-cortexa55-crypto-oe-linux/openssl/3.5.2/build"
> > if it does not find it. then perhaps build the assembly output with
> > -fverbose-asm -S
> > option and look for this string in the generated asm file.
>
> I ran this, and didn't see anything containing the path. I think it's
> just a bug with the handling of `-ffile-prefix-map` because it looks
> like the PWD being used is exactly listed
>
> Tried the latest snapshot of gcc 15 too just to see if they fixed
> anything. Nothing in the git log looks relevant and it doesn't fix the
> issue.
>
> Did some more digging and found out that if i remove
> `-ffat-lto-objects` I don't have the issue anymore.
> Seems like maybe this is related to
> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=109805
yeah seems to be the problem you are seeing as well. For tests try passing
LDFLAGS += "${DEBUG_PREFIX_MAP}"
and see if it helps.
>
> >
> > > >
> > > > > Prior to rebasing on the latest master, LTO was working fine globally.
> > > >
> > > > what is the delta we are talking about here ?
> > >
> > > Sadly this was long ago `25d60ac6f61cc186b4c5a961554bee583736fd17`
> > > from early 2024. Doing any kind of bisection is probably pointless so
> > > I just need to figure out this new impurity (or maybe just the check
> > > got added since then)
> >
> > Yeah that will take too long to bisect so leave that aside.
> >
> > >
> > > >
> > > > >
> > > > > -=-=-=-=-=-=-=-=-=-=-=-
> > > > > Links: You receive all messages sent to this group.
> > > > > View/Reply Online (#224137): https://lists.openembedded.org/g/openembedded-core/message/224137
> > > > > Mute This Topic: https://lists.openembedded.org/mt/115492623/1997914
> > > > > Group Owner: openembedded-core+owner@lists.openembedded.org
> > > > > Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [raj.khem@gmail.com]
> > > > > -=-=-=-=-=-=-=-=-=-=-=-
> > > > >
^ permalink raw reply [flat|nested] 8+ messages in thread* Re: Distro wide LTO broken?
2025-09-30 14:55 ` Khem Raj
@ 2025-11-06 10:07 ` mark.yang
0 siblings, 0 replies; 8+ messages in thread
From: mark.yang @ 2025-11-06 10:07 UTC (permalink / raw)
To: openembedded-core
[-- Attachment #1: Type: text/plain, Size: 7046 bytes --]
On Tue, Sep 30, 2025 at 11:55 PM, Khem Raj wrote:
>
> On Tue, Sep 30, 2025 at 12:50 AM William Kennington
> <william@wkennington.com> wrote:
>
>>
>> On Mon, Sep 29, 2025 at 7:58 PM Khem Raj <raj.khem@gmail.com> wrote:
>>
>>>
>>> On Mon, Sep 29, 2025 at 5:19 PM William Kennington
>>> <william@wkennington.com> wrote:
>>>
>>>>
>>>> On Mon, Sep 29, 2025 at 11:43 AM Khem Raj <raj.khem@gmail.com> wrote:
>>>>
>>>>>
>>>>> On Mon, Sep 29, 2025 at 3:51 AM William Kennington via
>>>>> lists.openembedded.org
>>>>> <william=wkennington.com@lists.openembedded.org> wrote:
>>>>>
>>>>>>
>>>>>> If I add the following to my distro, the build breaks for the openssl
>>>>>> and other core packages due to having build path impurities in the
>>>>>> static libraries created.
>>>>>>
>>>>>> ```
>>>>>> require conf/distro/include/lto.inc
>>>>>> DISTRO_FEATURES += "lto"
>>>>>> ```
>>>>>>
>>>>>> The errors I get are all like
>>>>>> ```
>>>>>> ERROR: openssl-3.5.2-r0 do_package_qa: QA Issue: File
>>>>>> /usr/lib/libssl.a in package openssl-staticdev contains reference to
>>>>>> TMPDIR [buildpaths]
>>>>>> ERROR: openssl-3.5.2-r0 do_package_qa: QA Issue: File
>>>>>> /usr/lib/libcrypto.a in package openssl-staticdev contains reference
>>>>>> to TMPDIR [buildpaths]
>>>>>> ERROR: openssl-3.5.2-r0 do_package_qa: Fatal QA errors were found, failing
>>>>>> task.
>>>>>> ERROR: Logfile of failure stored in:
>>>>>> /home/builder/yocto/build/tmp/work/cortexa76-cortexa55-crypto-oe-linux/openssl/3.5.2/temp/log.do_package_qa.118335
>>>>>>
>>>>>> ERROR: Task
>>>>>> (/srv/yocto/poky/meta/recipes-connectivity/openssl/openssl_3.5.2.bb:do_package_qa)
>>>>>>
>>>>>> failed with exit code '1'
>>>>>> ```
>>>>>>
>>>>>> Probably there is just some kind of flag missing in the LTO build
>>>>>> options to filter out the build directory, as -ffile-prefix-map and
>>>>>> whatnot are already present in the builds. I haven't yet figured out
>>>>>> exactly what is contributing to this impurity.
>>>>>
>>>>> Inspect the .a files ( maybe using strings or binary viewing tools
>>>>> like xxd ), with LTO they are flat objects with additional information
>>>>> for
>>>>> LTO to work and I think it will give more information.
>>>>
>>>> Yeah, I believe it's just a single remapping we are missing somewhere
>>>> that is allowing the extra string to be added into the binary. It's
>>>> been a while since I've worked on this kind of impurity stripping, so
>>>> it will probably take a bit to get used to the tooling again.
>>>> Hopefully I find the root cause and we can strip it out
>>>>
>>>> An example invocation looks like the following. All other object files
>>>> I looked at share the same impurity which ends up in the static
>>>> library later
>>>> ```
>>>> aarch64-oe-linux-gcc -mcpu=cortex-a76.cortex-a55+crypto
>>>> -mbranch-protection=standard -fstack-protector-strong -O2
>>>> -D_FORTIFY_SOURCE=2 -Wformat -Wformat-security -Werror=format-security
>>>> --sysroot=/home/builder/yocto/build/tmp/work/cortexa76-cortexa55-crypto-oe-linux/openssl/3.5.2/recipe-sysroot
>>>>
>>>> -I. -Iinclude -Iapps/include -I../sources/openssl-3.5.2
>>>> -I../sources/openssl-3.5.2/include
>>>> -I../sources/openssl-3.5.2/apps/include -fPIC -pthread
>>>> -Wa,--noexecstack -O2 -g -flto -ffat-lto-objects -fuse-linker-plugin
>>>> -ffile-prefix-map=/home/builder/yocto/build/tmp/work/cortexa76-cortexa55-crypto-oe-linux/openssl/3.5.2/s
>>>>
>>>> ources/openssl-3.5.2=/usr/src/debug/openssl/3.5.2
>>>> -ffile-prefix-map=/home/builder/yocto/build/tmp/work/cortexa76-cortexa55-
>>>> crypto-oe-linux/openssl/3.5.2/build=/usr/src/debug/openssl/3.5.2
>>>> -ffile-prefix-map=/home/builder/yocto/build/tmp/work/cortexa76-cortexa55-crypto-oe-linux/openssl/3.5.2/recipe-sysroot=
>>>>
>>>> -ffile-prefix-map=/home/builder/yocto/build/tmp/work/cortexa76-cortexa55-crypto-oe-linux/openssl/3.5.2/recipe-sysroot-native=
>>>>
>>>> -pipe -DOPENSSL_USE_NODELETE -DOPENSSL_PIC
>>>> -DOPENSSLDIR="\"/usr/lib/ssl-3\""
>>>> -DENGINESDIR="\"/usr/lib/engines-3\""
>>>> -DMODULESDIR="\"/usr/lib/ossl-modules\"" -DOPENSSL_BUILDING_OPENSSL
>>>> -DNDEBUG -MMD -MF apps/lib/libapps-lib-app_x509.d.tmp -c -o
>>>> apps/lib/libapps-lib-app_x509.o ../sources/openssl-3.5.2/apps/l
>>>> ib/app_x509.c
>>>> ```
>>>> and the partial strings dump with the impurity looks like
>>>> ```
>>>> vtmp
>>>> unsigned int
>>>> /usr/src/debug/openssl/3.5.2
>>>> ../sources/openssl-3.5.2/apps/lib/app_x509.c
>>>> /home/builder/yocto/build/tmp/work/cortexa76-cortexa55-crypto-oe-linux/openssl/3.5.2/build
>>>>
>>>> include/openssl
>>>> /usr/lib/aarch64-oe-linux/gcc/aarch64-oe-linux/15.2.0/include
>>>> ../sources/openssl-3.5.2/include/openssl
>>>> /usr/include
>>>> ../sources/openssl-3.5.2/apps/lib
>>>> ../sources/openssl-3.5.2/apps/include
>>>> app_x509.c
>>>> ```
>>>
>>> If you already know its coming from app_x509.c, then you have
>>> narrowed it down a step. Next, would be
>>
>> To be fair, it's basically every invocation of `gcc ... -c` that
>> produces it. I just picked an arbitrary file that was easy to find
>> that happens to carry the issue.
>>
>>
>>> to build just this file with -dD -E option and inspect if it contains
>>> any live string containing
>>> "/home/builder/yocto/build/tmp/work/cortexa76-cortexa55-crypto-oe-linux/openssl/3.5.2/build"
>>>
>>> if it does not find it. then perhaps build the assembly output with
>>> -fverbose-asm -S
>>> option and look for this string in the generated asm file.
>>
>> I ran this, and didn't see anything containing the path. I think it's
>> just a bug with the handling of `-ffile-prefix-map` because it looks
>> like the PWD being used is exactly listed
>>
>> Tried the latest snapshot of gcc 15 too just to see if they fixed
>> anything. Nothing in the git log looks relevant and it doesn't fix the
>> issue.
>>
>> Did some more digging and found out that if i remove
>> `-ffat-lto-objects` I don't have the issue anymore.
>> Seems like maybe this is related to
>> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=109805
>
> yeah seems to be the problem you are seeing as well. For tests try passing
>
>
> LDFLAGS += "${DEBUG_PREFIX_MAP}"
>
> and see if it helps.
>
>>
>>
>>>
>>>
>>>>
>>>>>
>>>>>
>>>>>> Prior to rebasing on the latest master, LTO was working fine globally.
>>>>>
>>>>> what is the delta we are talking about here ?
>>>>
>>>> Sadly this was long ago `25d60ac6f61cc186b4c5a961554bee583736fd17`
>>>> from early 2024. Doing any kind of bisection is probably pointless so
>>>> I just need to figure out this new impurity (or maybe just the check
>>>> got added since then)
>>>
>>> Yeah that will take too long to bisect so leave that aside.
>>>
>>>
>>>>
>>>>
>>>>>
>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>
>>>
>>
>>
>
>
I'm wondering if `-ffat-lto-objects` is strictly necessary for LTO.
[-- Attachment #2: Type: text/html, Size: 7209 bytes --]
^ permalink raw reply [flat|nested] 8+ messages in thread
* Distro Wide LTO Broken?
@ 2025-09-29 5:56 William Kennington
0 siblings, 0 replies; 8+ messages in thread
From: William Kennington @ 2025-09-29 5:56 UTC (permalink / raw)
To: openembedded-core
If I add the following to my distro, the build breaks for the openssl
and other core packages due to having build path impurities in the
static libraries created.
```
require conf/distro/include/lto.inc
DISTRO_FEATURES += "lto"
```
The errors I get are all like
```
ERROR: openssl-3.5.2-r0 do_package_qa: QA Issue: File
/usr/lib/libssl.a in package openssl-staticdev contains reference to
TMPDIR [buildpaths]
ERROR: openssl-3.5.2-r0 do_package_qa: QA Issue: File
/usr/lib/libcrypto.a in package openssl-staticdev contains reference
to TMPDIR [buildpaths]
ERROR: openssl-3.5.2-r0 do_package_qa: Fatal QA errors were found, failing task.
ERROR: Logfile of failure stored in:
/home/builder/yocto/build/tmp/work/cortexa76-cortexa55-crypto-oe-linux/openssl/3.5.2/temp/log.do_package_qa.118335
ERROR: Task (/srv/yocto/poky/meta/recipes-connectivity/openssl/openssl_3.5.2.bb:do_package_qa)
failed with exit code '1'
```
Probably there is just some kind of flag missing in the LTO build
options to filter out the build directory, as -ffile-prefix-map and
whatnot are already present in the builds. I haven't yet figured out
exactly what is contributing to this impurity.
Prior to rebasing on the latest master, LTO was working fine globally.
^ permalink raw reply [flat|nested] 8+ messages in thread
end of thread, other threads:[~2025-11-06 10:07 UTC | newest]
Thread overview: 8+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2025-09-29 1:22 Distro wide LTO broken? William Kennington
2025-09-29 18:42 ` [OE-core] " Khem Raj
2025-09-30 0:19 ` William Kennington
2025-09-30 2:58 ` Khem Raj
2025-09-30 7:49 ` William Kennington
2025-09-30 14:55 ` Khem Raj
2025-11-06 10:07 ` mark.yang
-- strict thread matches above, loose matches on Subject: below --
2025-09-29 5:56 Distro Wide LTO Broken? William Kennington
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox