Re: Distro wide LTO broken?

["mark.yang" <mark.yang@lge.com>]

[-- Attachment #1: Type: text/plain, Size: 7046 bytes --]

On Tue, Sep 30, 2025 at 11:55 PM, Khem Raj wrote:

> 
> On Tue, Sep 30, 2025 at 12:50 AM William Kennington
> <william@wkennington.com> wrote:
> 
>> 
>> On Mon, Sep 29, 2025 at 7:58 PM Khem Raj <raj.khem@gmail.com> wrote:
>> 
>>> 
>>> On Mon, Sep 29, 2025 at 5:19 PM William Kennington
>>> <william@wkennington.com> wrote:
>>> 
>>>> 
>>>> On Mon, Sep 29, 2025 at 11:43 AM Khem Raj <raj.khem@gmail.com> wrote:
>>>> 
>>>>> 
>>>>> On Mon, Sep 29, 2025 at 3:51 AM William Kennington via
>>>>> lists.openembedded.org
>>>>> <william=wkennington.com@lists.openembedded.org> wrote:
>>>>> 
>>>>>> 
>>>>>> If I add the following to my distro, the build breaks for the openssl
>>>>>> and other core packages due to having build path impurities in the
>>>>>> static libraries created.
>>>>>> 
>>>>>> ```
>>>>>> require conf/distro/include/lto.inc
>>>>>> DISTRO_FEATURES += "lto"
>>>>>> ```
>>>>>> 
>>>>>> The errors I get are all like
>>>>>> ```
>>>>>> ERROR: openssl-3.5.2-r0 do_package_qa: QA Issue: File
>>>>>> /usr/lib/libssl.a in package openssl-staticdev contains reference to
>>>>>> TMPDIR [buildpaths]
>>>>>> ERROR: openssl-3.5.2-r0 do_package_qa: QA Issue: File
>>>>>> /usr/lib/libcrypto.a in package openssl-staticdev contains reference
>>>>>> to TMPDIR [buildpaths]
>>>>>> ERROR: openssl-3.5.2-r0 do_package_qa: Fatal QA errors were found, failing
>>>>>> task.
>>>>>> ERROR: Logfile of failure stored in:
>>>>>> /home/builder/yocto/build/tmp/work/cortexa76-cortexa55-crypto-oe-linux/openssl/3.5.2/temp/log.do_package_qa.118335
>>>>>> 
>>>>>> ERROR: Task
>>>>>> (/srv/yocto/poky/meta/recipes-connectivity/openssl/openssl_3.5.2.bb:do_package_qa)
>>>>>> 
>>>>>> failed with exit code '1'
>>>>>> ```
>>>>>> 
>>>>>> Probably there is just some kind of flag missing in the LTO build
>>>>>> options to filter out the build directory, as -ffile-prefix-map and
>>>>>> whatnot are already present in the builds. I haven't yet figured out
>>>>>> exactly what is contributing to this impurity.
>>>>> 
>>>>> Inspect the .a files ( maybe using strings or binary viewing tools
>>>>> like xxd ), with LTO they are flat objects with additional information
>>>>> for
>>>>> LTO to work and I think it will give more information.
>>>> 
>>>> Yeah, I believe it's just a single remapping we are missing somewhere
>>>> that is allowing the extra string to be added into the binary. It's
>>>> been a while since I've worked on this kind of impurity stripping, so
>>>> it will probably take a bit to get used to the tooling again.
>>>> Hopefully I find the root cause and we can strip it out
>>>> 
>>>> An example invocation looks like the following. All other object files
>>>> I looked at share the same impurity which ends up in the static
>>>> library later
>>>> ```
>>>> aarch64-oe-linux-gcc -mcpu=cortex-a76.cortex-a55+crypto
>>>> -mbranch-protection=standard -fstack-protector-strong -O2
>>>> -D_FORTIFY_SOURCE=2 -Wformat -Wformat-security -Werror=format-security
>>>> --sysroot=/home/builder/yocto/build/tmp/work/cortexa76-cortexa55-crypto-oe-linux/openssl/3.5.2/recipe-sysroot
>>>> 
>>>> -I. -Iinclude -Iapps/include -I../sources/openssl-3.5.2
>>>> -I../sources/openssl-3.5.2/include
>>>> -I../sources/openssl-3.5.2/apps/include -fPIC -pthread
>>>> -Wa,--noexecstack -O2 -g -flto -ffat-lto-objects -fuse-linker-plugin
>>>> -ffile-prefix-map=/home/builder/yocto/build/tmp/work/cortexa76-cortexa55-crypto-oe-linux/openssl/3.5.2/s
>>>> 
>>>> ources/openssl-3.5.2=/usr/src/debug/openssl/3.5.2
>>>> -ffile-prefix-map=/home/builder/yocto/build/tmp/work/cortexa76-cortexa55-
>>>> crypto-oe-linux/openssl/3.5.2/build=/usr/src/debug/openssl/3.5.2
>>>> -ffile-prefix-map=/home/builder/yocto/build/tmp/work/cortexa76-cortexa55-crypto-oe-linux/openssl/3.5.2/recipe-sysroot=
>>>> 
>>>> -ffile-prefix-map=/home/builder/yocto/build/tmp/work/cortexa76-cortexa55-crypto-oe-linux/openssl/3.5.2/recipe-sysroot-native=
>>>> 
>>>> -pipe -DOPENSSL_USE_NODELETE -DOPENSSL_PIC
>>>> -DOPENSSLDIR="\"/usr/lib/ssl-3\""
>>>> -DENGINESDIR="\"/usr/lib/engines-3\""
>>>> -DMODULESDIR="\"/usr/lib/ossl-modules\"" -DOPENSSL_BUILDING_OPENSSL
>>>> -DNDEBUG -MMD -MF apps/lib/libapps-lib-app_x509.d.tmp -c -o
>>>> apps/lib/libapps-lib-app_x509.o ../sources/openssl-3.5.2/apps/l
>>>> ib/app_x509.c
>>>> ```
>>>> and the partial strings dump with the impurity looks like
>>>> ```
>>>> vtmp
>>>> unsigned int
>>>> /usr/src/debug/openssl/3.5.2
>>>> ../sources/openssl-3.5.2/apps/lib/app_x509.c
>>>> /home/builder/yocto/build/tmp/work/cortexa76-cortexa55-crypto-oe-linux/openssl/3.5.2/build
>>>> 
>>>> include/openssl
>>>> /usr/lib/aarch64-oe-linux/gcc/aarch64-oe-linux/15.2.0/include
>>>> ../sources/openssl-3.5.2/include/openssl
>>>> /usr/include
>>>> ../sources/openssl-3.5.2/apps/lib
>>>> ../sources/openssl-3.5.2/apps/include
>>>> app_x509.c
>>>> ```
>>> 
>>> If you already know its coming from app_x509.c, then you have
>>> narrowed it down a step. Next, would be
>> 
>> To be fair, it's basically every invocation of `gcc ... -c` that
>> produces it. I just picked an arbitrary file that was easy to find
>> that happens to carry the issue.
>> 
>> 
>>> to build just this file with -dD -E option and inspect if it contains
>>> any live string containing
>>> "/home/builder/yocto/build/tmp/work/cortexa76-cortexa55-crypto-oe-linux/openssl/3.5.2/build"
>>> 
>>> if it does not find it. then perhaps build the assembly output with
>>> -fverbose-asm -S
>>> option and look for this string in the generated asm file.
>> 
>> I ran this, and didn't see anything containing the path. I think it's
>> just a bug with the handling of `-ffile-prefix-map` because it looks
>> like the PWD being used is exactly listed
>> 
>> Tried the latest snapshot of gcc 15 too just to see if they fixed
>> anything. Nothing in the git log looks relevant and it doesn't fix the
>> issue.
>> 
>> Did some more digging and found out that if i remove
>> `-ffat-lto-objects` I don't have the issue anymore.
>> Seems like maybe this is related to
>> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=109805
> 
> yeah seems to be the problem you are seeing as well. For tests try passing
> 
> 
> LDFLAGS += "${DEBUG_PREFIX_MAP}"
> 
> and see if it helps.
> 
>> 
>> 
>>> 
>>> 
>>>> 
>>>>> 
>>>>> 
>>>>>> Prior to rebasing on the latest master, LTO was working fine globally.
>>>>> 
>>>>> what is the delta we are talking about here ?
>>>> 
>>>> Sadly this was long ago `25d60ac6f61cc186b4c5a961554bee583736fd17`
>>>> from early 2024. Doing any kind of bisection is probably pointless so
>>>> I just need to figure out this new impurity (or maybe just the check
>>>> got added since then)
>>> 
>>> Yeah that will take too long to bisect so leave that aside.
>>> 
>>> 
>>>> 
>>>> 
>>>>> 
>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>> 
>>>>> 
>>>> 
>>>> 
>>> 
>>> 
>> 
>> 
> 
> 

I'm wondering if `-ffat-lto-objects` is strictly necessary for LTO.

[-- Attachment #2: Type: text/html, Size: 7209 bytes --]