Rust reproducibility issue summary

Rust reproducibility issue summary

[Richard Purdie]

I've been asked what the reproducibility issue is and what data we
have. I ran a couple of test builds on the current autobuilder and the
test cluster. I added in the diff at the end of this email, just to
limit the test to what we're interested in.

The test on the new test cluster passed:

https://valkyrie.yoctoproject.org/#/builders/37/builds/36

However the build on the current autobuilder did not:

https://autobuilder.yoctoproject.org/typhoon/#/builders/117/builds/4876

and the different binaries generated are here:

http://autobuilder.yocto.io/pub/repro-fail/oe-reproducible-20240531-l872m0ip/packages/

We need to get to the bottom of why these are different and the test is
sometimes failing. Any help appreciated!

It is unclear what we need to do to reproduce the issue on demand.

Cheers,

Richard




diff --git a/meta/lib/oeqa/selftest/cases/reproducible.py b/meta/lib/oeqa/selftest/cases/reproducible.py
index 97a9c3da908..62b9f912af3 100644
--- a/meta/lib/oeqa/selftest/cases/reproducible.py
+++ b/meta/lib/oeqa/selftest/cases/reproducible.py
@@ -16,8 +16,6 @@ import os
 import datetime
 
 exclude_packages = [
-       'rust-rustdoc',
-       'rust-dbg'
        ]
 
 def is_excluded(package):
@@ -135,7 +133,7 @@ class ReproducibleTests(OESelftestTestCase):
     max_report_size = 250 * 1024 * 1024
 
     # targets are the things we want to test the reproducibility of
-    targets = ['core-image-minimal', 'core-image-sato', 'core-image-full-cmdline', 'core-image-weston', 'world']
+    targets = ['rust']
 
     # sstate targets are things to pull from sstate to potentially cut build/debugging time
     sstate_targets = []

Patchtest results for Rust reproducibility issue summary

[patchtest]

[-- Attachment #1: Type: text/plain, Size: 3071 bytes --]

Thank you for your submission. Patchtest identified one
or more issues with the patch. Please see the log below for
more information:

---
Testing patch /home/patchtest/share/mboxes/Rust-reproducibility-issue-summary.patch

FAIL: test Signed-off-by presence: Mbox is missing Signed-off-by. Add it manually or with "git commit --amend -s" (test_mbox.TestMbox.test_signed_off_by_presence)
FAIL: test shortlog format: Commit shortlog (first line of commit message) should follow the format "<target>: <summary>" (test_mbox.TestMbox.test_shortlog_format)

PASS: pretest pylint (test_python_pylint.PyLint.pretest_pylint)
PASS: test author valid (test_mbox.TestMbox.test_author_valid)
PASS: test commit message presence (test_mbox.TestMbox.test_commit_message_presence)
PASS: test max line length (test_metadata.TestMetadata.test_max_line_length)
PASS: test mbox format (test_mbox.TestMbox.test_mbox_format)
PASS: test non-AUH upgrade (test_mbox.TestMbox.test_non_auh_upgrade)
PASS: test pylint (test_python_pylint.PyLint.test_pylint)
PASS: test shortlog length (test_mbox.TestMbox.test_shortlog_length)
PASS: test target mailing list (test_mbox.TestMbox.test_target_mailing_list)

SKIP: pretest src uri left files: Patch cannot be merged (test_metadata.TestMetadata.pretest_src_uri_left_files)
SKIP: test CVE check ignore: No modified recipes or older target branch, skipping test (test_metadata.TestMetadata.test_cve_check_ignore)
SKIP: test CVE tag format: No new CVE patches introduced (test_patch.TestPatch.test_cve_tag_format)
SKIP: test Signed-off-by presence: No new CVE patches introduced (test_patch.TestPatch.test_signed_off_by_presence)
SKIP: test Upstream-Status presence: No new CVE patches introduced (test_patch.TestPatch.test_upstream_status_presence_format)
SKIP: test bugzilla entry format: No bug ID found (test_mbox.TestMbox.test_bugzilla_entry_format)
SKIP: test lic files chksum modified not mentioned: No modified recipes, skipping test (test_metadata.TestMetadata.test_lic_files_chksum_modified_not_mentioned)
SKIP: test lic files chksum presence: No added recipes, skipping test (test_metadata.TestMetadata.test_lic_files_chksum_presence)
SKIP: test license presence: No added recipes, skipping test (test_metadata.TestMetadata.test_license_presence)
SKIP: test series merge on head: Merge test is disabled for now (test_mbox.TestMbox.test_series_merge_on_head)
SKIP: test src uri left files: Patch cannot be merged (test_metadata.TestMetadata.test_src_uri_left_files)
SKIP: test summary presence: No added recipes, skipping test (test_metadata.TestMetadata.test_summary_presence)

---

Please address the issues identified and
submit a new revision of the patch, or alternatively, reply to this
email with an explanation of why the patch should be accepted. If you
believe these results are due to an error in patchtest, please submit a
bug at https://bugzilla.yoctoproject.org/ (use the 'Patchtest' category
under 'Yocto Project Subprojects'). For more information on specific
failures, see: https://wiki.yoctoproject.org/wiki/Patchtest. Thank
you!

Re: [OE-core] Rust reproducibility issue summary

[Alexander Kanavin]

On Fri, 31 May 2024 at 15:28, Richard Purdie via
lists.openembedded.org
<richard.purdie=linuxfoundation.org@lists.openembedded.org> wrote:
>
> I've been asked what the reproducibility issue is and what data we
> have. I ran a couple of test builds on the current autobuilder and the
> test cluster. I added in the diff at the end of this email, just to
> limit the test to what we're interested in.
>
> The test on the new test cluster passed:
>
> https://valkyrie.yoctoproject.org/#/builders/37/builds/36
>
> However the build on the current autobuilder did not:
>
> https://autobuilder.yoctoproject.org/typhoon/#/builders/117/builds/4876
>
> and the different binaries generated are here:
>
> http://autobuilder.yocto.io/pub/repro-fail/oe-reproducible-20240531-l872m0ip/packages/
>
> We need to get to the bottom of why these are different and the test is
> sometimes failing. Any help appreciated!
>
> It is unclear what we need to do to reproduce the issue on demand.

One possible lead is in the original attempt to fix this;
https://git.yoctoproject.org/poky/commit/?id=321aebfa281bd28e368c684ece57867f6bd0cbe7

It is not yet clear if disabling PGO makes things reproducible, as the
same commit also prevented the non reproducible binary (rustdoc) from
being installed (with the demangler line).

PGO as the culprit should be either confirmed, or ruled out first.

If PGO indeed helps then the next step is figure out how to disable
(or tweak) it in a way that doesn't cause issues with chromium:

https://git.yoctoproject.org/poky/commit/?id=694b85a217209787fe8c503a4b697dc6c2bfbe84

Alex

Re: [OE-core] Rust reproducibility issue summary

[Sundeep KOKKONDA]

[-- Attachment #1: Type: text/plain, Size: 2561 bytes --]


On 01-Jun-24 15:57, Alexander Kanavin wrote:
> CAUTION: This email comes from a non Wind River email account!
> Do not click links or open attachments unless you recognize the sender and know the content is safe.
>
> On Fri, 31 May 2024 at 15:28, Richard Purdie via
> lists.openembedded.org
> <richard.purdie=linuxfoundation.org@lists.openembedded.org>  wrote:
>> I've been asked what the reproducibility issue is and what data we
>> have. I ran a couple of test builds on the current autobuilder and the
>> test cluster. I added in the diff at the end of this email, just to
>> limit the test to what we're interested in.
>>
>> The test on the new test cluster passed:
>>
>> https://valkyrie.yoctoproject.org/#/builders/37/builds/36
>>
>> However the build on the current autobuilder did not:
>>
>> https://autobuilder.yoctoproject.org/typhoon/#/builders/117/builds/4876
>>
>> and the different binaries generated are here:
>>
>> http://autobuilder.yocto.io/pub/repro-fail/oe-reproducible-20240531-l872m0ip/packages/
>>
>> We need to get to the bottom of why these are different and the test is
>> sometimes failing. Any help appreciated!
>>
>> It is unclear what we need to do to reproduce the issue on demand.
> One possible lead is in the original attempt to fix this;
> https://git.yoctoproject.org/poky/commit/?id=321aebfa281bd28e368c684ece57867f6bd0cbe7
>
> It is not yet clear if disabling PGO makes things reproducible, as the
> same commit also prevented the non reproducible binary (rustdoc) from
> being installed (with the demangler line).
>
> PGO as the culprit should be either confirmed, or ruled out first.
>
> If PGO indeed helps then the next step is figure out how to disable
> (or tweak) it in a way that doesn't cause issues with chromium:
>
> https://git.yoctoproject.org/poky/commit/?id=694b85a217209787fe8c503a4b697dc6c2bfbe84
Hello Alex,

Initially we did enabled PGO & rust-demangler options for rustdoc 
reproducibility.
https://internals.rust-lang.org/t/rustdoc-binary-is-not-reproducible/20027/5

I did not get any info (from doc or community) on how and why optimation 
is affecting rustdoc binary.
https://internals.rust-lang.org/t/rustdoc-binary-is-not-reproducible/20027/6

However, later we learned that the '/rustdoc/' is reproducible without 
PGO also, and so removed for chromium project. And after removing PGO 
also we've not seen this rustdoc repro issue on autobuilder.

The issue is not reproduced on my end to confirm the PGO is root cause 
or any recent changes on rust recipe causing the issue?


/Sundeep
>
> Alex

[-- Attachment #2: Type: text/html, Size: 4271 bytes --]

Re: [OE-core] Rust reproducibility issue summary

[Sundeep KOKKONDA]

[-- Attachment #1: Type: text/plain, Size: 3662 bytes --]


On 03-Jun-24 9:53, Sundeep KOKKONDA via lists.openembedded.org wrote:
>
>
> On 01-Jun-24 15:57, Alexander Kanavin wrote:
>> CAUTION: This email comes from a non Wind River email account!
>> Do not click links or open attachments unless you recognize the sender and know the content is safe.
>>
>> On Fri, 31 May 2024 at 15:28, Richard Purdie via
>> lists.openembedded.org
>> <richard.purdie=linuxfoundation.org@lists.openembedded.org>  wrote:
>>> I've been asked what the reproducibility issue is and what data we
>>> have. I ran a couple of test builds on the current autobuilder and the
>>> test cluster. I added in the diff at the end of this email, just to
>>> limit the test to what we're interested in.
>>>
>>> The test on the new test cluster passed:
>>>
>>> https://valkyrie.yoctoproject.org/#/builders/37/builds/36
>>>
>>> However the build on the current autobuilder did not:
>>>
>>> https://autobuilder.yoctoproject.org/typhoon/#/builders/117/builds/4876
>>>
>>> and the different binaries generated are here:
>>>
>>> http://autobuilder.yocto.io/pub/repro-fail/oe-reproducible-20240531-l872m0ip/packages/
>>>
>>> We need to get to the bottom of why these are different and the test is
>>> sometimes failing. Any help appreciated!
>>>
>>> It is unclear what we need to do to reproduce the issue on demand.
>> One possible lead is in the original attempt to fix this;
>> https://git.yoctoproject.org/poky/commit/?id=321aebfa281bd28e368c684ece57867f6bd0cbe7
>>
>> It is not yet clear if disabling PGO makes things reproducible, as the
>> same commit also prevented the non reproducible binary (rustdoc) from
>> being installed (with the demangler line).
>>
>> PGO as the culprit should be either confirmed, or ruled out first.
>>
>> If PGO indeed helps then the next step is figure out how to disable
>> (or tweak) it in a way that doesn't cause issues with chromium:
>>
>> https://git.yoctoproject.org/poky/commit/?id=694b85a217209787fe8c503a4b697dc6c2bfbe84
> Hello Alex,
>
> Initially we did enabled PGO & rust-demangler options for rustdoc 
> reproducibility.
> https://internals.rust-lang.org/t/rustdoc-binary-is-not-reproducible/20027/5
>
> I did not get any info (from doc or community) on how and why 
> optimation is affecting rustdoc binary.
> https://internals.rust-lang.org/t/rustdoc-binary-is-not-reproducible/20027/6
>
> However, later we learned that the '/rustdoc/' is reproducible without 
> PGO also, and so removed for chromium project. And after removing PGO 
> also we've not seen this rustdoc repro issue on autobuilder.
>
> The issue is not reproduced on my end to confirm the PGO is root cause 
> or any recent changes on rust recipe causing the issue?


Further analysis on autobuilder binaries shows that the deviations are 
similar to that of earlier /rustdoc /issue. The deviations are in 
|.llvm.<hash>| suffixes.

As discussed in issue 
<https://internals.rust-lang.org/t/rustdoc-binary-is-not-reproducible/20027>, 
may be we should check by disabling PGO (by default it is enabled and as 
in comment <in 
https://internals.rust-lang.org/t/rustdoc-binary-is-not-reproducible/20027/3>the 
PGO profile generation is not deterministic...).

rustdoc-175 diff
>
>
> /Sundeep
>> Alex
>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#200231):https://lists.openembedded.org/g/openembedded-core/message/200231
> Mute This Topic:https://lists.openembedded.org/mt/106407787/7677583
> Group Owner:openembedded-core+owner@lists.openembedded.org
> Unsubscribe:https://lists.openembedded.org/g/openembedded-core/unsub  [sundeep.kokkonda@eng.windriver.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>

[-- Attachment #2.1: Type: text/html, Size: 6705 bytes --]

[-- Attachment #2.2: rustdoc-175.png --]
[-- Type: image/png, Size: 131505 bytes --]

Re: [OE-core] Rust reproducibility issue summary

[Alexander Kanavin]

[-- Attachment #1.1: Type: text/plain, Size: 4104 bytes --]

Yes please. This is what I've been trying to get across: try disabling the
PGO once more, but do not change what gets installed, and ensure that
rustdoc does get installed and packaged.

Then, if that solves the reproducibility issue we need to look at why
exactly chromium doesn't like it, so that it had to be reverted:
https://git.yoctoproject.org/poky/commit/?id=694b85a217209787fe8c503a4b697dc6c2bfbe84

Alex

On Mon, 3 Jun 2024 at 09:08, Sundeep KOKKONDA <
Sundeep.Kokkonda@windriver.com> wrote:

>
> On 03-Jun-24 9:53, Sundeep KOKKONDA via lists.openembedded.org wrote:
>
>
> On 01-Jun-24 15:57, Alexander Kanavin wrote:
>
> CAUTION: This email comes from a non Wind River email account!
> Do not click links or open attachments unless you recognize the sender and know the content is safe.
>
> On Fri, 31 May 2024 at 15:28, Richard Purdie vialists.openembedded.org<richard.purdie=linuxfoundation.org@lists.openembedded.org> <richard.purdie=linuxfoundation.org@lists.openembedded.org> wrote:
>
> I've been asked what the reproducibility issue is and what data we
> have. I ran a couple of test builds on the current autobuilder and the
> test cluster. I added in the diff at the end of this email, just to
> limit the test to what we're interested in.
>
> The test on the new test cluster passed:
> https://valkyrie.yoctoproject.org/#/builders/37/builds/36
>
> However the build on the current autobuilder did not:
> https://autobuilder.yoctoproject.org/typhoon/#/builders/117/builds/4876
>
> and the different binaries generated are here:
> http://autobuilder.yocto.io/pub/repro-fail/oe-reproducible-20240531-l872m0ip/packages/
>
> We need to get to the bottom of why these are different and the test is
> sometimes failing. Any help appreciated!
>
> It is unclear what we need to do to reproduce the issue on demand.
>
> One possible lead is in the original attempt to fix this;https://git.yoctoproject.org/poky/commit/?id=321aebfa281bd28e368c684ece57867f6bd0cbe7
>
> It is not yet clear if disabling PGO makes things reproducible, as the
> same commit also prevented the non reproducible binary (rustdoc) from
> being installed (with the demangler line).
>
> PGO as the culprit should be either confirmed, or ruled out first.
>
> If PGO indeed helps then the next step is figure out how to disable
> (or tweak) it in a way that doesn't cause issues with chromium:
> https://git.yoctoproject.org/poky/commit/?id=694b85a217209787fe8c503a4b697dc6c2bfbe84
>
> Hello Alex,
>
> Initially we did enabled PGO & rust-demangler options for rustdoc
> reproducibility.
>
> https://internals.rust-lang.org/t/rustdoc-binary-is-not-reproducible/20027/5
>
> I did not get any info (from doc or community) on how and why optimation
> is affecting rustdoc binary.
>
> https://internals.rust-lang.org/t/rustdoc-binary-is-not-reproducible/20027/6
>
> However, later we learned that the '*rustdoc*' is reproducible without
> PGO also, and so removed for chromium project. And after removing PGO also
> we've not seen this rustdoc repro issue on autobuilder.
>
> The issue is not reproduced on my end to confirm the PGO is root cause or
> any recent changes on rust recipe causing the issue?
>
>
> Further analysis on autobuilder binaries shows that the deviations are
> similar to that of earlier *rustdoc *issue. The deviations are in
> .llvm.<hash> suffixes.
>
> As discussed in issue
> <https://internals.rust-lang.org/t/rustdoc-binary-is-not-reproducible/20027>,
> may be we should check by disabling PGO (by default it is enabled and as in comment
> the PGO profile generation is not deterministic...).
> [image: rustdoc-175 diff]
>
>
> /Sundeep
>
> Alex
>
>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#200231): https://lists.openembedded.org/g/openembedded-core/message/200231
> Mute This Topic: https://lists.openembedded.org/mt/106407787/7677583
> Group Owner: openembedded-core+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [sundeep.kokkonda@eng.windriver.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>
>
>

[-- Attachment #1.2: Type: text/html, Size: 6836 bytes --]

[-- Attachment #2: rustdoc-175.png --]
[-- Type: image/png, Size: 131505 bytes --]

Re: Rust reproducibility issue summary

[Sundeep KOKKONDA]

[-- Attachment #1: Type: text/plain, Size: 2585 bytes --]

Hello,

I've a local reproducer for this issue. Below are a few observations with my analysis.
- The deviation in rustdoc binary are because of the different hash-id generated between the build (as explained above - https://lists.openembedded.org/g/openembedded-core/message/200238)
- The hash-id generates identically when the rust built from it's sources every time.
- During the oe-selftest triggered, when the issue reproduced I can see there are reproducibleA & reproducibleB directories are generated.
But, the rust is not built from sources on both directories beacuse of Setscene Tasks and Shared State. (reproducibleB directory has rust sources built and reproducibleA has setscene realted data copied the files into it from sstate and so, no rust sources are cloned & build into it)
- The above case not always happening with oe-selftest, when the test passed I can see the reproducibleA also has rust sources pulled and built, in this case rustdoc generates identically.
- When the issue reproduced, I did a check on the sstate sigdata to find any deviations but those sig's are identical.
- Also, when the test failed, I tried to find the unique hash-id's generated in rustdoc in the generated rlibs of rust. I could only find one hash-id in rlibs where the rust sources are actually built. The other hash-id not found in rlibs of reproducibleA/B.
- The ' diff .../reproducibleA/tmp/work/core2-64-poky-linux/rust/1.75.0/temp/depsig.do_package_write_deb .../reproducibleB/tmp/work/core2-64-poky-linux/rust/1.75.0/temp/depsig.do_package_write_deb ' shows -
8c8
< -rw-             15932884 6a6618ccfc171ea507ad39eb95bc239116b5f5c9e85a50ce9de3b43c73ff17d7./deploy-debs/core2-64/rust-dbg_1.75.0-r0_amd64.deb
---
> -rw-             15932940 4dcbe007aa8d46e658f95a3651938d67ca5cd55409c503861f365cb9c38f14e7./deploy-debs/core2-64/rust-dbg_1.75.0-r0_amd64.deb
11c11
< -rw-              2902068 564925753476842ea9052311afa27cbc099136113ea9f5394b5554eccb09e8f3./deploy-debs/core2-64/rust-rustdoc_1.75.0-r0_amd64.deb
---
> -rw-              2901612 c0cd05fe344400c0a9d31fecd808c3152481f35dacc0556c006b88a239209358./deploy-debs/core2-64/rust-rustdoc_1.75.0-r0_amd64.deb

I am not sure of what that diff indicates.

Can I get some pointers on -
1. Why only sometimes the setscene setting on reproducibleA (which causes rust sources not to rebuilt)
2. Is there any way/tool to analyze the above depsig data or what is going-on with setscene.

This helps to identify the issue is happening due to sstate? if not I can look into rust.

[-- Attachment #2: Type: text/html, Size: 3834 bytes --]

Re: [OE-core] Rust reproducibility issue summary

[Alexander Kanavin]

On Mon, 24 Jun 2024 at 06:00, Sundeep KOKKONDA via
lists.openembedded.org
<sundeep.kokkonda=windriver.com@lists.openembedded.org> wrote:
> - The hash-id generates identically when the rust built from it's sources every time.

This is not the case. I just built rust (without sstate, from sources)
in two different build directories on the same machine, and indeed
rustdoc is different:

alex@Zen2:/srv/storage/alex/yocto$ diff -uNr
build-64/tmp/work/core2-64-poky-linux/rust/1.75.0/image/
build-64-2/tmp/work/core2-64-poky-linux/rust/1.75.0/image/
Binary files build-64/tmp/work/core2-64-poky-linux/rust/1.75.0/image/usr/bin/rustdoc
and build-64-2/tmp/work/core2-64-poky-linux/rust/1.75.0/image/usr/bin/rustdoc
differ

Alex

Re: Rust reproducibility issue summary

[Sundeep KOKKONDA]

[-- Attachment #1: Type: text/plain, Size: 743 bytes --]

On Mon, Jun 24, 2024 at 01:50 PM, Alexander Kanavin wrote:

> 
> This is not the case. I just built rust (without sstate, from sources)
> in two different build directories on the same machine, and indeed

Thanks for sharing this... Unfortunately I couldn't see this happening during my testing. If this is the case, the issue might be there in rust.

for me diff is not there...
> diff -uNr build-b/tmp/work/core2-64-poky-linux/rust/1.75.0/image/ build-c/tmp/work/core2-64-poky-linux/rust/1.75.0/image/
> diff build-b/tmp/work/core2-64-poky-linux/rust/1.75.0/image/usr/bin/rustdoc build-c/tmp/work/core2-64-poky-linux/rust/1.75.0/image/usr/bin/rustdoc
> diff -uNr build-b/tmp/deploy/deb/core2-64/ build-c/tmp/deploy/deb/core2-64/

[-- Attachment #2: Type: text/html, Size: 819 bytes --]

Re: [OE-core] Rust reproducibility issue summary

[Alexander Kanavin]

On Mon, 24 Jun 2024 at 11:07, Sundeep KOKKONDA via
lists.openembedded.org
<sundeep.kokkonda=windriver.com@lists.openembedded.org> wrote:
> This is not the case. I just built rust (without sstate, from sources)
> in two different build directories on the same machine, and indeed
>
> Thanks for sharing this... Unfortunately I couldn't see this happening during my testing. If this is the case, the issue might be there in rust.
>
> for me diff is not there...
> > diff -uNr build-b/tmp/work/core2-64-poky-linux/rust/1.75.0/image/ build-c/tmp/work/core2-64-poky-linux/rust/1.75.0/image/
> > diff build-b/tmp/work/core2-64-poky-linux/rust/1.75.0/image/usr/bin/rustdoc build-c/tmp/work/core2-64-poky-linux/rust/1.75.0/image/usr/bin/rustdoc
> > diff -uNr build-b/tmp/deploy/deb/core2-64/ build-c/tmp/deploy/deb/core2-64/

I can share the two different build directories for your further
inspection perhaps? Let me upload the tarballs somewhere public.

Alex

Re: [OE-core] Rust reproducibility issue summary

[Sundeep KOKKONDA]

[-- Attachment #1: Type: text/plain, Size: 3543 bytes --]


On 28-Jun-24 22:57, Alexander Kanavin wrote:
> **
> *CAUTION: This email comes from a non Wind River email account!*
> Do not click links or open attachments unless you recognize the sender 
> and know the content is safe.
> Hello Sundeep,
>
> I ran both with 'bitbake -c install rust' which runs a complete build 
> up to install task, and then checked that rustdoc in image/ is 
> different between them. The other tasks aren't relevant, and you can 
> ignore lack of package/ etc.
>
> You need to check the difference in content of rustc-1.75.0-src/ (this 
> is where the build runs and intermediate artefacts are), and maybe 
> temp/log.do_install will give clues about rustdoc as well.
Hi Alex,

I am checking the build artifacts generated and finding the probable 
reasons to identify the issue. While looking onto /.../temp// I've seen 
a log - '/log.do_recipe_qa_setscene/', as per my understanding this task 
ensure that the build system can reuse previous build results to avoid 
unnecessary recompilation and to speed up the build process and which 
needs sharing of /sstate/. This log only seen in '/build-64-2/' 
directory (not in '/build-64/' dir).

Also, I've checked my previous builds where sstate not shared and this 
task was not executed. In may case, I made multiple builds and the issue 
not reproduced w/o sharing sstate. Can you check local.conf settings the 
sstate is disabled? and when will the '/do_recipe_qa_setscene' /task 
execute w/o sstate sharing?


Thanks,

Sundeep K.

>
> Alex
>
> On Fri, 28 Jun 2024 at 18:56, Sundeep KOKKONDA 
> <Sundeep.Kokkonda@windriver.com> wrote:
>
>     Hello Alex,
>
>     Need some info on the given builds... I can see the below attached
>     diffs in generated artifacts b/w the builds.
>
>     in /build-64/ there are /package & deploy/ dir's are there & in
>
>     /build-64-2/ there are /sstate-install-***/ dirs are present.
>
>
>     What is the diff b/w the builds? The sstate is not shared b/w the
>     builds right?
>
>
>     Thanks,
>
>     Sundeep K.
>
>     On 24-Jun-24 14:41, Alexander Kanavin wrote:
>>     CAUTION: This email comes from a non Wind River email account!
>>     Do not click links or open attachments unless you recognize the sender and know the content is safe.
>>
>>     On Mon, 24 Jun 2024 at 11:07, Sundeep KOKKONDA via
>>     lists.openembedded.org  <https://urldefense.com/v3/__http://lists.openembedded.org__;!!AjveYdw8EvQ!erzxpkS1tjGXNXDzGLxKnUsNCjmIRZq_UJXp8SrCd0r2DUSTD_lQkAQxAQ-kArkUUTbV82iNXIBB6wfiDmCagyK2JvJgOQ$>
>>     <sundeep.kokkonda=windriver.com@lists.openembedded.org>  <mailto:sundeep.kokkonda=windriver.com@lists.openembedded.org>  wrote:
>>>     This is not the case. I just built rust (without sstate, from sources)
>>>     in two different build directories on the same machine, and indeed
>>>
>>>     Thanks for sharing this... Unfortunately I couldn't see this happening during my testing. If this is the case, the issue might be there in rust.
>>>
>>>     for me diff is not there...
>>>>     diff -uNr build-b/tmp/work/core2-64-poky-linux/rust/1.75.0/image/ build-c/tmp/work/core2-64-poky-linux/rust/1.75.0/image/
>>>>     diff build-b/tmp/work/core2-64-poky-linux/rust/1.75.0/image/usr/bin/rustdoc build-c/tmp/work/core2-64-poky-linux/rust/1.75.0/image/usr/bin/rustdoc
>>>>     diff -uNr build-b/tmp/deploy/deb/core2-64/ build-c/tmp/deploy/deb/core2-64/
>>     I can share the two different build directories for your further
>>     inspection perhaps? Let me upload the tarballs somewhere public.
>>
>>     Alex
>

[-- Attachment #2.1: Type: text/html, Size: 6159 bytes --]

[-- Attachment #2.2: koj04EawYF28p0m6.png --]
[-- Type: image/png, Size: 112548 bytes --]

Re: [OE-core] Rust reproducibility issue summary

[Alexander Kanavin]

[-- Attachment #1: Type: text/plain, Size: 2138 bytes --]

On Wed, 3 Jul 2024 at 10:09, Sundeep KOKKONDA <
Sundeep.Kokkonda@windriver.com> wrote:

>
> I ran both with 'bitbake -c install rust' which runs a complete build up
> to install task, and then checked that rustdoc in image/ is different
> between them. The other tasks aren't relevant, and you can ignore lack of
> package/ etc.
>
> You need to check the difference in content of rustc-1.75.0-src/ (this is
> where the build runs and intermediate artefacts are), and maybe
> temp/log.do_install will give clues about rustdoc as well.
>
> Hi Alex,
>
> I am checking the build artifacts generated and finding the probable
> reasons to identify the issue. While looking onto *.../temp/* I've seen a
> log - '*log.do_recipe_qa_setscene*', as per my understanding this task
> ensure that the build system can reuse previous build results to avoid
> unnecessary recompilation and to speed up the build process and which needs
> sharing of *sstate*. This log only seen in '*build-64-2*' directory (not
> in '*build-64*' dir).
>
> Also, I've checked my previous builds where sstate not shared and this
> task was not executed. In may case, I made multiple builds and the issue
> not reproduced w/o sharing sstate. Can you check local.conf settings the
> sstate is disabled? and when will the '*do_recipe_qa_setscene'  *task
> execute w/o sstate sharing?
>

 Hello Sundeep,

recipe_qa task is only used for QA checks on the recipe and has no
influence on what gets built and installed. The actual build and
installation happens in do_install task, and both tarballs have a built
tree where do_install was fully executed, with results going into image/
(specifically, that directory has two different rustdoc executables).

So you can ignore any other task, regardless of whether it's fulfilled from
sstate or not. The key pieces in the tarballs are the content of image/
(where rustdoc differs), temp/log.do_install (the log for build and
installation which may give clues to where things diverge), and the content
of rustc-1.75.0-src/. The challenging part for you is to narrow this down
to the specific point where the two builds diverge.

Alex

[-- Attachment #2: Type: text/html, Size: 2974 bytes --]

Re: Rust reproducibility issue summary

[Sundeep KOKKONDA]

[-- Attachment #1: Type: text/plain, Size: 218 bytes --]

Hi Alex,

What distro you used to reproduce the issue & what pkgs were installed. You used any shared sstate of Yocto?
With this info I will retry to reproduce the issue on my local machine.

Thanks,
Sundeep K.

[-- Attachment #2: Type: text/html, Size: 252 bytes --]

Re: [OE-core] Rust reproducibility issue summary

[Alexander Kanavin]

Hello,

I simply ran two independent builds, each with their own sstate. The
host distro is not relevant, as it's been observed on yocto
autobuilder across many different distros.

I'm not sure why you insist on being able to reproduce the issue, as
you're simply going to get the same build trees as in the tarballs I
provided.

Alex

On Thu, 4 Jul 2024 at 17:13, Sundeep KOKKONDA via
lists.openembedded.org
<sundeep.kokkonda=windriver.com@lists.openembedded.org> wrote:
>
> Hi Alex,
>
> What distro you used to reproduce the issue & what pkgs were installed. You used any shared sstate of Yocto?
> With this info I will retry to reproduce the issue on my local machine.
>
>
> Thanks,
> Sundeep K.
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#201578): https://lists.openembedded.org/g/openembedded-core/message/201578
> Mute This Topic: https://lists.openembedded.org/mt/106407787/1686489
> Group Owner: openembedded-core+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [alex.kanavin@gmail.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>

Re: [OE-core] Rust reproducibility issue summary

[Randy MacLeod]

[-- Attachment #1: Type: text/plain, Size: 2129 bytes --]

On 2024-07-04 11:21 a.m., Alexander Kanavin via lists.openembedded.org 
wrote:
> Hello,
>
> I simply ran two independent builds, each with their own sstate. The
> host distro is not relevant, as it's been observed on yocto
> autobuilder across many different distros.
>
> I'm not sure why you insist on being able to reproduce the issue, as
> you're simply going to get the same build trees as in the tarballs I
> provided.

Hi Alex,

That's true, but it's always nicer to be able to reproduce a problem 
locally so one can tests fixes properly.
Richard and I suggested that we at least get the distro and list of 
packages identified so that Sundeep
could see if he can reproduce the problem independently.

You're right that it's not essential but we're also trying to avoid 
putting too much extra
load on the autobuilders and the people who run builds there.

It's a long-standing problem that the Rust updates are done  and pass 
all local tests only
to fail on one or more of the YP AB systems. We'd like to understand why 
that happens
so if you could share the info that Sundeep requested, that may help for 
this bug and for
future Rust work as well.

Thanks,

../Randy


>
> Alex
>
> On Thu, 4 Jul 2024 at 17:13, Sundeep KOKKONDA via
> lists.openembedded.org
> <sundeep.kokkonda=windriver.com@lists.openembedded.org>  wrote:
>> Hi Alex,
>>
>> What distro you used to reproduce the issue & what pkgs were installed. You used any shared sstate of Yocto?
>> With this info I will retry to reproduce the issue on my local machine.
>>
>>
>> Thanks,
>> Sundeep K.
>>
>>
>>
>> -=-=-=-=-=-=-=-=-=-=-=-
>> Links: You receive all messages sent to this group.
>> View/Reply Online (#201580):https://lists.openembedded.org/g/openembedded-core/message/201580
>> Mute This Topic:https://lists.openembedded.org/mt/106407787/3616765
>> Group Owner:openembedded-core+owner@lists.openembedded.org
>> Unsubscribe:https://lists.openembedded.org/g/openembedded-core/unsub  [randy.macleod@windriver.com]
>> -=-=-=-=-=-=-=-=-=-=-=-
>>

-- 
# Randy MacLeod
# Wind River Linux

[-- Attachment #2: Type: text/html, Size: 3649 bytes --]

Re: [OE-core] Rust reproducibility issue summary

[Alexander Kanavin]

On Thu, 4 Jul 2024 at 18:47, Randy MacLeod <randy.macleod@windriver.com> wrote:
> That's true, but it's always nicer to be able to reproduce a problem locally so one can tests fixes properly.
> Richard and I suggested that we at least get the distro and list of packages identified so that Sundeep
> could see if he can reproduce the problem independently.
>
> You're right that it's not essential but we're also trying to avoid putting too much extra
> load on the autobuilders and the people who run builds there.
>
> It's a long-standing problem that the Rust updates are done  and  pass all local tests only
> to fail on one or more of the YP AB systems. We'd like to understand why that happens
> so if you could share the info that Sundeep requested, that may help for this bug and for
> future Rust work as well.

The host distro is Debian 12, but I still do not think this is
relevant, here's why. I spent much of yesterday and today figuring out
how to trigger non-reproducibility reliably, and a few options were
ruled out:

- re-running the build in the same build directory (mv tmp tmpN &&
bitbake -c install rust) is reproducible

- running two builds in two adjacent build directories with names of
same length (build-1/ and build-2/) is reproducible

- what seems to be *not* reproducible is having two different build
directories with significantly different names (e.g. build/ and
build-rustreprolongdir/). I've seen this in build/ vs build-2/ as
well, but e.g. build-2/ and build-3/ are reproducible.

Such experiments are very expensive time-wise (each option needs to be
tried several times to ensure the issue is not intermittent with
reasonable confidence), so I'd appreciate if you try the third option
locally.

Cheers,
Alex

Re: Rust reproducibility issue summary

[Sundeep KOKKONDA]

[-- Attachment #1: Type: text/plain, Size: 584 bytes --]

On Fri, Jul 5, 2024 at 05:06 PM, Alexander Kanavin wrote:

> 
> - what seems to be *not* reproducible is having two different build
> directories with significantly different names (e.g. build/ and
> build-rustreprolongdir/). I've seen this in build/ vs build-2/ as
> well, but e.g. build-2/ and build-3/ are reproducible.

Hello Alex,

This case reproduces the issue for me and I haven't checked such case earlier... I am able to reproduce the issue when the build directory names are different in size ( build & build-somelongpath-rustdoc-repro ).

Thanks,
Sundeep K.

[-- Attachment #2: Type: text/html, Size: 663 bytes --]

Re: [OE-core] Rust reproducibility issue summary

[Alexander Kanavin]

On Mon, 24 Jun 2024 at 11:11, Alexander Kanavin via
lists.openembedded.org <alex.kanavin=gmail.com@lists.openembedded.org>
wrote:
> I can share the two different build directories for your further
> inspection perhaps? Let me upload the tarballs somewhere public.

They're here:
https://drive.google.com/file/d/1kKhHoC3oMFPFjvuLvM4glQzKX8JeXNue/view?usp=drive_link
https://drive.google.com/file/d/1RCoZipt6i9wWG3j-j2rilWCNW_NMNCk9/view?usp=drive_link

Please let me know when you have downloaded them because I'm nearly
out of space there, and need to delete them ASAP.

Alex

Re: [OE-core] Rust reproducibility issue summary

[Sundeep KOKKONDA]

[-- Attachment #1: Type: text/plain, Size: 913 bytes --]


On 24-Jun-24 17:52, Alexander Kanavin wrote:
> CAUTION: This email comes from a non Wind River email account!
> Do not click links or open attachments unless you recognize the sender and know the content is safe.
>
> On Mon, 24 Jun 2024 at 11:11, Alexander Kanavin via
> lists.openembedded.org<alex.kanavin=gmail.com@lists.openembedded.org>
> wrote:
>> I can share the two different build directories for your further
>> inspection perhaps? Let me upload the tarballs somewhere public.
> They're here:
> https://drive.google.com/file/d/1kKhHoC3oMFPFjvuLvM4glQzKX8JeXNue/view?usp=drive_link
> https://drive.google.com/file/d/1RCoZipt6i9wWG3j-j2rilWCNW_NMNCk9/view?usp=drive_link
>
> Please let me know when you have downloaded them because I'm nearly
> out of space there, and need to delete them ASAP.

Hi Alex,

Copied. Thnx.

I tried again with latest poky, still the /rustdoc /generated identical :(

>
> Alex

[-- Attachment #2: Type: text/html, Size: 2008 bytes --]

Re: [OE-core] Rust reproducibility issue summary

[Alexander Kanavin]

On Tue, 25 Jun 2024 at 05:02, Sundeep KOKKONDA
<Sundeep.Kokkonda@windriver.com> wrote:
> I can share the two different build directories for your further
> inspection perhaps? Let me upload the tarballs somewhere public.
>
> They're here:
> https://drive.google.com/file/d/1kKhHoC3oMFPFjvuLvM4glQzKX8JeXNue/view?usp=drive_link
> https://drive.google.com/file/d/1RCoZipt6i9wWG3j-j2rilWCNW_NMNCk9/view?usp=drive_link
>
> Please let me know when you have downloaded them because I'm nearly
> out of space there, and need to delete them ASAP.
>
> Hi Alex,
>
> Copied. Thnx.

The idea is that these tarballs contain a complete build directory of
rust, with all the intermediate artefacts (e.g. individual object
files). So you could try to look into it to see where things start to
diverge between them. It has to be something specific to rustdoc,
possibly iterating over a non-deterministic list, such as a file
listing that hasn't been sorted, or a list of keys in a hash table
that hasn't been sorted either.

Alex

Re: [OE-core] Rust reproducibility issue summary

[Alexander Kanavin]

[-- Attachment #1: Type: text/plain, Size: 974 bytes --]

On Mon, 3 Jun 2024 at 09:08, Sundeep KOKKONDA <
Sundeep.Kokkonda@windriver.com> wrote:

> Further analysis on autobuilder binaries shows that the deviations are
> similar to that of earlier *rustdoc *issue. The deviations are in
> .llvm.<hash> suffixes.
>
> As discussed in issue
> <https://internals.rust-lang.org/t/rustdoc-binary-is-not-reproducible/20027>,
> may be we should check by disabling PGO (by default it is enabled and as in comment
> the PGO profile generation is not deterministic...).
>

After reading the issue and poking at the rustdoc binaries on my local
disk, I think the key is in finding where those llvm.<hash> suffixes are
generated and *how*. Something (presumably rust-llvm) puts them there, so
narrowing down to the point where that non-reproducible value is decided
would help a lot in finding out why it sometimes changes. It does require
navigating and reading a mountain of unfamiliar llvm code, this is what
makes the issue difficult.

Alex

[-- Attachment #2: Type: text/html, Size: 1534 bytes --]

Re: Rust reproducibility issue summary

[Sundeep KOKKONDA]

[-- Attachment #1: Type: text/plain, Size: 956 bytes --]

On Fri, Jul 5, 2024 at 08:55 PM, Alexander Kanavin wrote:

> 
> After reading the issue and poking at the rustdoc binaries on my local
> disk, I think the key is in finding where those llvm.<hash> suffixes are
> generated and *how*. Something (presumably rust-llvm) puts them there, so
> narrowing down to the point where that non-reproducible value is decided
> would help a lot in finding out why it sometimes changes. It does require
> navigating and reading a mountain of unfamiliar llvm code, this is what
> makes the issue difficult.

The.llvm.<hash> are generated by ThinLTO optimizations, and this is enabled by default. I tried a build by disabling this optimization but that doesn't solved the problem. By disabling lto, the.llvm.<hash> are not seen in rustdoc binary but still the binaries are differed. I'll have to check with llvm community that how this hash depends/affects based on build-dir name/length.

Thanks,
Sundeep K.

[-- Attachment #2: Type: text/html, Size: 3657 bytes --]