Yocto reproducibilty issue :: rust-llvm binary differences

Yocto reproducibilty issue :: rust-llvm binary differences

[Kokkonda, Sundeep]

[-- Attachment #1.1: Type: text/plain, Size: 1581 bytes --]

Hello,

The rust builds are not reproducible when the build path is changed. The issue details are in https://bugzilla.yoctoproject.org/show_bug.cgi?id=14875.

After debugging several ways, we tried to find at what build stage the Yocto build artifacts are deviating with rust sources generated artifacts.

I took the identical rust tarball sources (rustc-1.67.0-src.tar.xz) as Yocto rust sources and built in two different paths (buildA & buildB directories). In the rust sources the generated binaries are identical between 2 different builds.

But, In Yocto we found that the binaries '../tmp/work/x86_64-linux/rust-native/1.67.0-r0/recipe-sysroot-native/usr/lib/llvm-rust/lib/libLTO.so & libRemarks.so' are differed between 2 builds. See the differences in the attached 'libLTO-diff.png'.

And these objects are passed to compile all rust crates in later build stages (with option '-L native=/ala-lpggp31/skokkonda/875/poky/buildC/tmp/work/x86_64-linux/rust-native/1.67.0-r0/recipe-sysroot-native/usr/lib/llvm-rust/lib').

So, we suspect this could be the reason for this reproducibility issue in Yocto. I tried to find the reason for difference in these binaries but the '-vv' build log doesn't give enough info by linking which objects this libLTO.so is generated. I tried the disassembly of object files generated in ../x86_64-linux/rust-native/ but that didn't showup these binary differences.

Anyone who has expertise with rust-llvm can guide me here on how these *.so are generating and how to understand/fix these differences.



Thanks,
Sundeep K.

[-- Attachment #1.2: Type: text/html, Size: 7351 bytes --]

[-- Attachment #2: libLTO-diff.PNG --]
[-- Type: image/png, Size: 46741 bytes --]

Re: [OE-core] Yocto reproducibilty issue :: rust-llvm binary differences

[Alexander Kanavin]

Are the two different sets of object files available somewhere for
download and inspection? I could take a look and see if I can spot the
difference, but I need the files.

Alex

On Tue, 21 Feb 2023 at 17:47, Sundeep KOKKONDA
<sundeep.kokkonda@windriver.com> wrote:
>
> Hello,
>
> The rust builds are not reproducible when the build path is changed. The issue details are in https://bugzilla.yoctoproject.org/show_bug.cgi?id=14875.
>
> After debugging several ways, we tried to find at what build stage the Yocto build artifacts are deviating with rust sources generated artifacts.
>
> I took the identical rust tarball sources (rustc-1.67.0-src.tar.xz) as Yocto rust sources and built in two different paths (buildA & buildB directories). In the rust sources the generated binaries are identical between 2 different builds.
>
> But, In Yocto we found that the binaries '../tmp/work/x86_64-linux/rust-native/1.67.0-r0/recipe-sysroot-native/usr/lib/llvm-rust/lib/libLTO.so & libRemarks.so' are differed between 2 builds. See the differences in the attached 'libLTO-diff.png'.
>
> And these objects are passed to compile all rust crates in later build stages (with option '-L native=/ala-lpggp31/skokkonda/875/poky/buildC/tmp/work/x86_64-linux/rust-native/1.67.0-r0/recipe-sysroot-native/usr/lib/llvm-rust/lib').
>
> So, we suspect this could be the reason for this reproducibility issue in Yocto. I tried to find the reason for difference in these binaries but the '-vv' build log doesn't give enough info by linking which objects this libLTO.so is generated. I tried the disassembly of object files generated in ../x86_64-linux/rust-native/ but that didn't showup these binary differences.
>
> Anyone who has expertise with rust-llvm can guide me here on how these *.so are generating and how to understand/fix these differences.
>
>
>
> Thanks,
> Sundeep K.
>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#177528): https://lists.openembedded.org/g/openembedded-core/message/177528
> Mute This Topic: https://lists.openembedded.org/mt/97113152/1686489
> Group Owner: openembedded-core+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [alex.kanavin@gmail.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>

Re: Yocto reproducibilty issue :: rust-llvm binary differences

[Sundeep KOKKONDA]

[-- Attachment #1: Type: text/plain, Size: 84 bytes --]

Hello Alex,

files are here... https://we.tl/t-ijJJZnBvKh

Thanks,
Sundeep K.

[-- Attachment #2: Type: text/html, Size: 193 bytes --]

Re: Yocto reproducibilty issue :: rust-llvm binary differences

[Sundeep KOKKONDA]

[-- Attachment #1: Type: text/plain, Size: 1031 bytes --]

Additional object files link (if needed) -> https://we.tl/t-RvgWukdcFB

poky/build-llvm-A/tmp/work/x86_64-linux/rust-native/ -> rust-native-A-obj.tar
poky/build-llvm-B/tmp/work/x86_64-linux/rust-native/ -> rust-native-B-obj.tar
poky/build-llvm-A/tmp/work/x86_64-linux/rust-llvm-native/ -> rust-llvm-native-A-obj.tar
poky/build-llvm-B/tmp/work/x86_64-linux/rust-llvm-native/ -> rust-llvm-native-B-obj.tar

> 
> $ diff -ur rust-llvm-native-A-obj rust-llvm-native-B-obj
> Binary files rust-llvm-native-A-obj/llvm-config.cpp.o and
> rust-llvm-native-B-obj/llvm-config.cpp.o differ
> Binary files rust-llvm-native-A-obj/python.o and
> rust-llvm-native-B-obj/python.o differ
> 
> $ diff -ur rust-native-A-obj rust-native-B-obj
> Binary files rust-native-A-obj/1mg8n9voykv9ib2d.o and
> rust-native-B-obj/1mg8n9voykv9ib2d.o differ
> Binary files rust-native-A-obj/538dmycheh7dlbck.o and
> rust-native-B-obj/538dmycheh7dlbck.o differ
> Binary files rust-native-A-obj/python.o and rust-native-B-obj/python.o
> differ
>

[-- Attachment #2: Type: text/html, Size: 1352 bytes --]

Re: [OE-core] Yocto reproducibilty issue :: rust-llvm binary differences

[Alexander Kanavin]

I took a look at libRemarks.so and libLTO.so. If you run 'objdump -s'
on them, you'll see that they differ only in 'gnu.build-id' property,
and are otherwise identical. So you need to look into why the id is
different, I don't remember right now how it is created. Probably the
compile log can give a clue.

You can also confirm that it is indeed these two files that cause
divergence in target rust by for example copying them from the A build
into the B build just prior to building rust.

Alex

On Wed, 22 Feb 2023 at 04:58, Sundeep KOKKONDA
<sundeep.kokkonda@windriver.com> wrote:
>
> Hello Alex,
>
> files are here... https://we.tl/t-ijJJZnBvKh
>
>
>
> Thanks,
> Sundeep K.
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#177559): https://lists.openembedded.org/g/openembedded-core/message/177559
> Mute This Topic: https://lists.openembedded.org/mt/97113152/1686489
> Group Owner: openembedded-core+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [alex.kanavin@gmail.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>

Re: [OE-core] Yocto reproducibilty issue :: rust-llvm binary differences

[Khem Raj]

On Wed, Feb 22, 2023 at 3:10 AM Alexander Kanavin
<alex.kanavin@gmail.com> wrote:
>
> I took a look at libRemarks.so and libLTO.so. If you run 'objdump -s'
> on them, you'll see that they differ only in 'gnu.build-id' property,
> and are otherwise identical. So you need to look into why the id is
> different, I don't remember right now how it is created. Probably the
> compile log can give a clue.
>
> You can also confirm that it is indeed these two files that cause
> divergence in target rust by for example copying them from the A build
> into the B build just prior to building rust.

gnu.build-id in output means its using --build-id option during link so it will
be good to find out how this option is being constructed during link, usually
its SHA1 hash on parts of the output contents, so if these contents are
same then it should always come out to be same but it could be something
is changing in two cases perhaps some paths etc.

>
> Alex
>
> On Wed, 22 Feb 2023 at 04:58, Sundeep KOKKONDA
> <sundeep.kokkonda@windriver.com> wrote:
> >
> > Hello Alex,
> >
> > files are here... https://we.tl/t-ijJJZnBvKh
> >
> >
> >
> > Thanks,
> > Sundeep K.
> >
> >
>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#177565): https://lists.openembedded.org/g/openembedded-core/message/177565
> Mute This Topic: https://lists.openembedded.org/mt/97113152/1997914
> Group Owner: openembedded-core+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [raj.khem@gmail.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>

Re: Yocto reproducibilty issue :: rust-llvm binary differences

[Sundeep KOKKONDA]

[-- Attachment #1: Type: text/plain, Size: 3288 bytes --]

Hello,

I tried to copy the libLTO.so to ${D}. But this shows ERROR: rust-llvm-1.67.0-r0 do_package_qa: QA Issue: -dev package rust-llvm-dev contains non-symlink .so '/usr/lib/llvm-rust/lib/libLTO.so' [dev-elf] and I had a look into symlink where it points... The libLTO.so files RPATH in ${D} & sysroot-destdir are different as below,

$ objdump -x ./image/wdr/poky/buildA/tmp/work/x86_64-linux/rust-llvm-native/1.67.0-r0/recipe-sysroot-native/usr/lib/llvm-rust/lib/libLTO.so | grep 'R.*PATH'

RUNPATH /wdr/poky/buildA/tmp/work/x86_64-linux/rust-llvm-native/1.67.0-r0/recipe-sysroot-native/usr/lib:/wdr/poky/buildA/tmp/work/x86_64-linux/rust-llvm-native/1.67.0-r0/recipe-sysroot-native/lib::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::

$ objdump -x ./sysroot-destdir/wdr/poky/buildA/tmp/work/x86_64-linux/rust-llvm-native/1.67.0-r0/recipe-sysroot-native/usr/lib/llvm-rust/lib/libLTO.so | grep 'R.*PATH'

RUNPATH $ORIGIN/../..:$ORIGIN/../../../../lib:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.

Also, the libLTO.so is a symlink for libLTO.so.15. So, I tried copying both libLTO.so & libLTO.so.15 but again getting below QA error:

ERROR : rust-llvm-1.67.0-r0 do_package: QA Issue: File '/usr/lib/llvm-rust/lib/libLTO.so' from rust-llvm was already stripped, this will prevent future debugging! [already-stripped]

ERROR : rust-llvm-1.67.0-r0 do_package: QA Issue: File '/usr/lib/llvm-rust/lib/libLTO.so.15' from rust-llvm was already stripped, this will prevent future debugging! [already-stripped]

WARNING : rust-llvm-1.67.0-r0 do_package: rust-llvm-liblto-1.67.0 was registered as shlib provider for libLTO.so.15, changing it to rust-llvm-dev-1.67.0 because it was built later

ERROR : rust-llvm-1.67.0-r0 do_package: Fatal QA errors were found, failing task.

So, I disabled stripping of binaries in kernel.bbclass (@line 743 : #oe.package.runstrip((kernel_image_stripped, 8, strip, extra_sections))) but still facing below errors.

WARNING : rust-llvm-1.67.0-r0 do_package: rust-llvm-liblto-1.67.0 was registered as shlib provider for libLTO.so.15, changing it to rust-llvm-dev-1.67.0 because it was built later

WARNING : rust-llvm-1.67.0-r0 do_package_qa: QA Issue: File /usr/lib/llvm-rust/lib/libLTO.so.15 in package rust-llvm-liblto contains reference to TMPDIR [buildpaths]

WARNING : rust-llvm-1.67.0-r0 do_package_qa: QA Issue: File /usr/lib/llvm-rust/lib/libLTO.so in package rust-llvm-dev contains reference to TMPDIR [buildpaths]

ERROR : rust-llvm-1.67.0-r0 do_package_qa: QA Issue: -dev package rust-llvm-dev contains non-symlink .so '/usr/lib/llvm-rust/lib/libLTO.so' [dev-elf]

ERROR : rust-llvm-1.67.0-r0 do_package_qa: Fatal QA errors were found, failing task.

I tried to do the same with by adding my own task with ' addtask patchso after do_install ' but that is also having some issues while calling that task and that is not recommended way to do it by Richard.

Is there any way I can fix it or disable these checks for my testing?

Thanks,

Sundeep K.

[-- Attachment #2: Type: text/html, Size: 29163 bytes --]

Re: [OE-core] Yocto reproducibilty issue :: rust-llvm binary differences

[Alexander Kanavin]

You should not copy things between sysroot and ${D}. There is
post-processing between those steps, so do it only between sysroots
please.

Alex

On Wed, 1 Mar 2023 at 07:28, Sundeep KOKKONDA
<sundeep.kokkonda@windriver.com> wrote:
>
> Hello,
>
>
>
> I tried to copy the libLTO.so to ${D}. But this shows ERROR: rust-llvm-1.67.0-r0 do_package_qa: QA Issue: -dev package rust-llvm-dev contains non-symlink .so '/usr/lib/llvm-rust/lib/libLTO.so' [dev-elf] and I had a look into symlink where it points... The libLTO.so files RPATH in ${D} & sysroot-destdir are different as below,
>
>
>
> $ objdump -x ./image/wdr/poky/buildA/tmp/work/x86_64-linux/rust-llvm-native/1.67.0-r0/recipe-sysroot-native/usr/lib/llvm-rust/lib/libLTO.so | grep 'R.*PATH'
>
>   RUNPATH              /wdr/poky/buildA/tmp/work/x86_64-linux/rust-llvm-native/1.67.0-r0/recipe-sysroot-native/usr/lib:/wdr/poky/buildA/tmp/work/x86_64-linux/rust-llvm-native/1.67.0-r0/recipe-sysroot-native/lib::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
>
> $ objdump -x ./sysroot-destdir/wdr/poky/buildA/tmp/work/x86_64-linux/rust-llvm-native/1.67.0-r0/recipe-sysroot-native/usr/lib/llvm-rust/lib/libLTO.so | grep 'R.*PATH'
>
>   RUNPATH              $ORIGIN/../..:$ORIGIN/../../../../lib:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.
>
>
>
> Also, the libLTO.so is a symlink for libLTO.so.15. So, I tried copying both libLTO.so & libLTO.so.15 but again getting below QA error:
>
> ERROR: rust-llvm-1.67.0-r0 do_package: QA Issue: File '/usr/lib/llvm-rust/lib/libLTO.so' from rust-llvm was already stripped, this will prevent future debugging! [already-stripped]
>
> ERROR: rust-llvm-1.67.0-r0 do_package: QA Issue: File '/usr/lib/llvm-rust/lib/libLTO.so.15' from rust-llvm was already stripped, this will prevent future debugging! [already-stripped]
>
> WARNING: rust-llvm-1.67.0-r0 do_package: rust-llvm-liblto-1.67.0 was registered as shlib provider for libLTO.so.15, changing it to rust-llvm-dev-1.67.0 because it was built later
>
> ERROR: rust-llvm-1.67.0-r0 do_package: Fatal QA errors were found, failing task.
>
>
>
> So, I disabled stripping of binaries in kernel.bbclass (@line 743 : #oe.package.runstrip((kernel_image_stripped, 8, strip, extra_sections))) but still facing below errors.
>
> WARNING: rust-llvm-1.67.0-r0 do_package: rust-llvm-liblto-1.67.0 was registered as shlib provider for libLTO.so.15, changing it to rust-llvm-dev-1.67.0 because it was built later
>
> WARNING: rust-llvm-1.67.0-r0 do_package_qa: QA Issue: File /usr/lib/llvm-rust/lib/libLTO.so.15 in package rust-llvm-liblto contains reference to TMPDIR [buildpaths]
>
> WARNING: rust-llvm-1.67.0-r0 do_package_qa: QA Issue: File /usr/lib/llvm-rust/lib/libLTO.so in package rust-llvm-dev contains reference to TMPDIR [buildpaths]
>
> ERROR: rust-llvm-1.67.0-r0 do_package_qa: QA Issue: -dev package rust-llvm-dev contains non-symlink .so '/usr/lib/llvm-rust/lib/libLTO.so' [dev-elf]
>
> ERROR: rust-llvm-1.67.0-r0 do_package_qa: Fatal QA errors were found, failing task.
>
>
> I tried to do the same with by adding my own task with 'addtask patchso after do_install' but that is also having some issues while calling that task and that is not recommended way to do it by Richard.
>
> Is there any way I can fix it or disable these checks for my testing?
>
>
>
>
>
> Thanks,
>
> Sundeep K.
>
>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#177868): https://lists.openembedded.org/g/openembedded-core/message/177868
> Mute This Topic: https://lists.openembedded.org/mt/97113152/1686489
> Group Owner: openembedded-core+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [alex.kanavin@gmail.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>

Re: [OE-core] Yocto reproducibilty issue :: rust-llvm binary differences

[Richard Purdie]

On Wed, 2023-03-01 at 07:47 +0100, Alexander Kanavin wrote:
> You should not copy things between sysroot and ${D}. There is
> post-processing between those steps, so do it only between sysroots
> please.

I think Sundeep was trying to so something I'd suggested, which was to
inject a different copy of libLTO.so from an external build. The intent
was to check if this was the source of reproducibility problems or not.
It is unclear where the problems come from and this would isolate if it
really was the cause or not.

It looks like we may need to tweak the file being injected to have
safer paths though.

Cheers,

Richard

Re: [OE-core] Yocto reproducibilty issue :: rust-llvm binary differences

[Richard Purdie]

On Tue, 2023-02-28 at 22:28 -0800, Sundeep KOKKONDA wrote:
> Hello,
>  
> I tried to copy the libLTO.so to ${D}. But this shows ERROR: rust-
> llvm-1.67.0-r0 do_package_qa: QA Issue: -dev package rust-llvm-dev
> contains non-symlink .so '/usr/lib/llvm-rust/lib/libLTO.so' [dev-
> elf] and I had a look into symlink where it points... The libLTO.so
> files RPATH in ${D} & sysroot-destdir are different as below,
>  
> $ objdump -x ./image/wdr/poky/buildA/tmp/work/x86_64-linux/rust-llvm-native/1.67.0-r0/recipe-sysroot-native/usr/lib/llvm-rust/lib/libLTO.so | grep 'R.*PATH' 
>   RUNPATH              /wdr/poky/buildA/tmp/work/x86_64-linux/rust-llvm-native/1.67.0-r0/recipe-sysroot-native/usr/lib:/wdr/poky/buildA/tmp/work/x86_64-linux/rust-llvm-native/1.67.0-r0/recipe-sysroot-native/lib:::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::: 
> $ objdump -x ./sysroot-destdir/wdr/poky/buildA/tmp/work/x86_64-linux/rust-llvm-native/1.67.0-r0/recipe-sysroot-native/usr/lib/llvm-rust/lib/libLTO.so | grep 'R.*PATH' 
>   RUNPATH              $ORIGIN/../..:$ORIGIN/../../../../lib:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:. 

It is expected that the system would adjust the RUNPATH in the binary
to be relocatable so that isn't surprising.

> Also, the libLTO.so is a symlink for libLTO.so.15.

Right, so you only need to replace libTLO.so.15 and not the symlink.

>  So, I tried copying both libLTO.so & libLTO.so.15 but again getting below QA error:
> ERROR: rust-llvm-1.67.0-r0 do_package: QA Issue: File '/usr/lib/llvm-rust/lib/libLTO.so' from rust-llvm was already stripped, this will prevent future debugging! [already-stripped] 
> ERROR: rust-llvm-1.67.0-r0 do_package: QA Issue: File '/usr/lib/llvm-rust/lib/libLTO.so.15' from rust-llvm was already stripped, this will prevent future debugging! [already-stripped] 
> WARNING: rust-llvm-1.67.0-r0 do_package: rust-llvm-liblto-1.67.0 was registered as shlib provider for libLTO.so.15, changing it to rust-llvm-dev-1.67.0 because it was built later 
> ERROR: rust-llvm-1.67.0-r0 do_package: Fatal QA errors were found, failing task. 

You can avoid that with:

ERROR_QA:remove = "already-stripped"

> I tried to do the same with by adding my own task with 'addtask
> patchso after do_install' but that is also having some issues while
> calling that task and that is not recommended way to do it by
> Richard.

Adding to do_install will be much easier than trying to get the
dependencies right for that new task.

Cheers,

Richard