[PATCH] rust: reproducibility issue fix

[PATCH] rust: reproducibility issue fix

[Sundeep KOKKONDA]

[Yocto#14875]
The '--remap-path-prefix' option removes all references to build directory structure in the debug information within the compiled output for Cargo dependencies and the project's binary.
However, some references to build directories remains in the final binary in .rustc section in the form of compressed metadata and this makes the build output dependent on the folder structure of the computer it's compiled on.
So, for reproducible builds, use the configuration option 'remap-debuginfo = true' along with the '--remap-path-prefix'.

Signed-off-by: Sundeep KOKKONDA <sundeep.kokkonda@windriver.com>
---
 meta/lib/oeqa/selftest/cases/reproducible.py | 2 --
 meta/recipes-devtools/rust/rust_1.70.0.bb    | 1 +
 2 files changed, 1 insertion(+), 2 deletions(-)

diff --git a/meta/lib/oeqa/selftest/cases/reproducible.py b/meta/lib/oeqa/selftest/cases/reproducible.py
index 84c6c3a05f..9b4a0887dd 100644
--- a/meta/lib/oeqa/selftest/cases/reproducible.py
+++ b/meta/lib/oeqa/selftest/cases/reproducible.py
@@ -16,8 +16,6 @@ import os
 import datetime
 
 exclude_packages = [
-	'rust',
-	'rust-dbg'
 	]
 
 def is_excluded(package):
diff --git a/meta/recipes-devtools/rust/rust_1.70.0.bb b/meta/recipes-devtools/rust/rust_1.70.0.bb
index 3b9c05a19f..16d433910f 100644
--- a/meta/recipes-devtools/rust/rust_1.70.0.bb
+++ b/meta/recipes-devtools/rust/rust_1.70.0.bb
@@ -130,6 +130,7 @@ python do_configure() {
     # [rust]
     config.add_section("rust")
     config.set("rust", "rpath", e(True))
+    config.set("rust", "remap-debuginfo", e(True))
     config.set("rust", "channel", e(d.expand("${RUST_CHANNEL}")))
 
     # Whether or not to optimize the compiler and standard library
-- 
2.39.0

Re: [OE-core] [PATCH] rust: reproducibility issue fix

[Alexander Kanavin]

Excellent, thanks!

Alex

On Wed, 11 Oct 2023 at 06:06, Sundeep KOKKONDA via
lists.openembedded.org
<sundeep.kokkonda=windriver.com@lists.openembedded.org> wrote:
>
> [Yocto#14875]
> The '--remap-path-prefix' option removes all references to build directory structure in the debug information within the compiled output for Cargo dependencies and the project's binary.
> However, some references to build directories remains in the final binary in .rustc section in the form of compressed metadata and this makes the build output dependent on the folder structure of the computer it's compiled on.
> So, for reproducible builds, use the configuration option 'remap-debuginfo = true' along with the '--remap-path-prefix'.
>
> Signed-off-by: Sundeep KOKKONDA <sundeep.kokkonda@windriver.com>
> ---
>  meta/lib/oeqa/selftest/cases/reproducible.py | 2 --
>  meta/recipes-devtools/rust/rust_1.70.0.bb    | 1 +
>  2 files changed, 1 insertion(+), 2 deletions(-)
>
> diff --git a/meta/lib/oeqa/selftest/cases/reproducible.py b/meta/lib/oeqa/selftest/cases/reproducible.py
> index 84c6c3a05f..9b4a0887dd 100644
> --- a/meta/lib/oeqa/selftest/cases/reproducible.py
> +++ b/meta/lib/oeqa/selftest/cases/reproducible.py
> @@ -16,8 +16,6 @@ import os
>  import datetime
>
>  exclude_packages = [
> -       'rust',
> -       'rust-dbg'
>         ]
>
>  def is_excluded(package):
> diff --git a/meta/recipes-devtools/rust/rust_1.70.0.bb b/meta/recipes-devtools/rust/rust_1.70.0.bb
> index 3b9c05a19f..16d433910f 100644
> --- a/meta/recipes-devtools/rust/rust_1.70.0.bb
> +++ b/meta/recipes-devtools/rust/rust_1.70.0.bb
> @@ -130,6 +130,7 @@ python do_configure() {
>      # [rust]
>      config.add_section("rust")
>      config.set("rust", "rpath", e(True))
> +    config.set("rust", "remap-debuginfo", e(True))
>      config.set("rust", "channel", e(d.expand("${RUST_CHANNEL}")))
>
>      # Whether or not to optimize the compiler and standard library
> --
> 2.39.0
>
>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#188940): https://lists.openembedded.org/g/openembedded-core/message/188940
> Mute This Topic: https://lists.openembedded.org/mt/101890610/1686489
> Group Owner: openembedded-core+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [alex.kanavin@gmail.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>

Re: [OE-core] [PATCH] rust: reproducibility issue fix

[Richard Purdie]

On Tue, 2023-10-10 at 21:05 -0700, Sundeep KOKKONDA via
lists.openembedded.org wrote:
> [Yocto#14875]
> The '--remap-path-prefix' option removes all references to build directory structure in the debug information within the compiled output for Cargo dependencies and the project's binary.
> However, some references to build directories remains in the final binary in .rustc section in the form of compressed metadata and this makes the build output dependent on the folder structure of the computer it's compiled on.
> So, for reproducible builds, use the configuration option 'remap-debuginfo = true' along with the '--remap-path-prefix'.
> 
> Signed-off-by: Sundeep KOKKONDA <sundeep.kokkonda@windriver.com>
> ---
>  meta/lib/oeqa/selftest/cases/reproducible.py | 2 --
>  meta/recipes-devtools/rust/rust_1.70.0.bb    | 1 +
>  2 files changed, 1 insertion(+), 2 deletions(-)

Thanks for that!

I know this has taken a lot of work to get to the bottom of. It is
great to have OE-Core at 100% reproducible again! :)

Cheers,

Richard

Re: [OE-core] [PATCH] rust: reproducibility issue fix

[Ross Burton]

On 11 Oct 2023, at 05:05, Sundeep KOKKONDA via lists.openembedded.org <sundeep.kokkonda=windriver.com@lists.openembedded.org> wrote:
> 
> [Yocto#14875]
> The '--remap-path-prefix' option removes all references to build directory structure in the debug information within the compiled output for Cargo dependencies and the project's binary.
> However, some references to build directories remains in the final binary in .rustc section in the form of compressed metadata and this makes the build output dependent on the folder structure of the computer it's compiled on.
> So, for reproducible builds, use the configuration option 'remap-debuginfo = true' along with the '--remap-path-prefix’.

Love it when literally months of work results in a one-liner…

So glad you managed to figure this out!

Ross

Re: [OE-core] [PATCH] rust: reproducibility issue fix

[Alex Kiernan]

On Wed, Oct 11, 2023 at 5:06 AM Sundeep KOKKONDA via
lists.openembedded.org
<sundeep.kokkonda=windriver.com@lists.openembedded.org> wrote:
>
> [Yocto#14875]
> The '--remap-path-prefix' option removes all references to build directory structure in the debug information within the compiled output for Cargo dependencies and the project's binary.
> However, some references to build directories remains in the final binary in .rustc section in the form of compressed metadata and this makes the build output dependent on the folder structure of the computer it's compiled on.
> So, for reproducible builds, use the configuration option 'remap-debuginfo = true' along with the '--remap-path-prefix'.
>
> Signed-off-by: Sundeep KOKKONDA <sundeep.kokkonda@windriver.com>
> ---
>  meta/lib/oeqa/selftest/cases/reproducible.py | 2 --
>  meta/recipes-devtools/rust/rust_1.70.0.bb    | 1 +
>  2 files changed, 1 insertion(+), 2 deletions(-)
>

Blimey, well done!

-- 
Alex Kiernan

Re: [OE-core] [PATCH] rust: reproducibility issue fix

[Frédéric Martinsons]

[-- Attachment #1: Type: text/plain, Size: 1621 bytes --]

very great news to read !


On Wed, 11 Oct 2023 at 13:03, Alex Kiernan <alex.kiernan@gmail.com> wrote:

> On Wed, Oct 11, 2023 at 5:06 AM Sundeep KOKKONDA via
> lists.openembedded.org
> <sundeep.kokkonda=windriver.com@lists.openembedded.org> wrote:
> >
> > [Yocto#14875]
> > The '--remap-path-prefix' option removes all references to build
> directory structure in the debug information within the compiled output for
> Cargo dependencies and the project's binary.
> > However, some references to build directories remains in the final
> binary in .rustc section in the form of compressed metadata and this makes
> the build output dependent on the folder structure of the computer it's
> compiled on.
> > So, for reproducible builds, use the configuration option
> 'remap-debuginfo = true' along with the '--remap-path-prefix'.
> >
> > Signed-off-by: Sundeep KOKKONDA <sundeep.kokkonda@windriver.com>
> > ---
> >  meta/lib/oeqa/selftest/cases/reproducible.py | 2 --
> >  meta/recipes-devtools/rust/rust_1.70.0.bb    | 1 +
> >  2 files changed, 1 insertion(+), 2 deletions(-)
> >
>
> Blimey, well done!
>
> --
> Alex Kiernan
>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#188954):
> https://lists.openembedded.org/g/openembedded-core/message/188954
> Mute This Topic: https://lists.openembedded.org/mt/101890610/6213388
> Group Owner: openembedded-core+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [
> frederic.martinsons@gmail.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>
>

[-- Attachment #2: Type: text/html, Size: 2789 bytes --]

Re: [OE-core] [PATCH] rust: reproducibility issue fix

[Frédéric Martinsons]

[-- Attachment #1: Type: text/plain, Size: 2153 bytes --]

On Wed, 11 Oct 2023 at 14:27, Frederic Martinsons via lists.openembedded.org
<frederic.martinsons=gmail.com@lists.openembedded.org> wrote:

> very great news to read !
>
>
> On Wed, 11 Oct 2023 at 13:03, Alex Kiernan <alex.kiernan@gmail.com> wrote:
>
>> On Wed, Oct 11, 2023 at 5:06 AM Sundeep KOKKONDA via
>> lists.openembedded.org
>> <sundeep.kokkonda=windriver.com@lists.openembedded.org> wrote:
>> >
>> > [Yocto#14875]
>> > The '--remap-path-prefix' option removes all references to build
>> directory structure in the debug information within the compiled output for
>> Cargo dependencies and the project's binary.
>> > However, some references to build directories remains in the final
>> binary in .rustc section in the form of compressed metadata and this makes
>> the build output dependent on the folder structure of the computer it's
>> compiled on.
>> > So, for reproducible builds, use the configuration option
>> 'remap-debuginfo = true' along with the '--remap-path-prefix'.
>> >
>> > Signed-off-by: Sundeep KOKKONDA <sundeep.kokkonda@windriver.com>
>> > ---
>> >  meta/lib/oeqa/selftest/cases/reproducible.py | 2 --
>> >  meta/recipes-devtools/rust/rust_1.70.0.bb    | 1 +
>> >  2 files changed, 1 insertion(+), 2 deletions(-)
>> >
>>
>> Blimey, well done!
>>
>> --
>> Alex Kiernan
>>
>>
>>
We may want to remove the  EXCLUDE_FROM_WORLD in zvariant recipe
<https://git.openembedded.org/openembedded-core/tree/meta-selftest/recipes-extended/zvariant/zvariant_3.12.0.bb>
since
it was introduced for reproducibility issue (#15090 which had been marked
as a duplicate of the main rust issue #14875).
What do you think sundeep ?


>>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#188956):
> https://lists.openembedded.org/g/openembedded-core/message/188956
> Mute This Topic: https://lists.openembedded.org/mt/101890610/6213388
> Group Owner: openembedded-core+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [
> frederic.martinsons@gmail.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>
>

[-- Attachment #2: Type: text/html, Size: 4022 bytes --]

Re: [OE-core] [PATCH] rust: reproducibility issue fix

[Richard Purdie]

On Wed, 2023-10-11 at 14:37 +0200, Frederic Martinsons wrote:
> We may want to remove the  EXCLUDE_FROM_WORLD in zvariant
> recipe since it was introduced for reproducibility issue (#15090
> which had been marked as a duplicate of the main rust issue #14875).
> What do you think sundeep ?

We should certainly test it, it really depends if there are further
issues present or not. This certainly is a good first step though!

Cheers,

Richard

Re: [PATCH] rust: reproducibility issue fix

[Sundeep KOKKONDA]

[-- Attachment #1: Type: text/plain, Size: 178 bytes --]

I looked briefly into the issue 15090 (zvariant recipe) description and as per me it should be tested before removing EXCLUDE_FROM_WORLD to ensure the builds are reproducible.

[-- Attachment #2: Type: text/html, Size: 182 bytes --]

Re: [OE-core] [PATCH] rust: reproducibility issue fix

[Frédéric Martinsons]

[-- Attachment #1: Type: text/plain, Size: 868 bytes --]

Le mer. 11 oct. 2023, 16:32, Sundeep KOKKONDA via lists.openembedded.org
<sundeep.kokkonda=windriver.com@lists.openembedded.org> a écrit :

> I looked briefly into the issue 15090 (zvariant recipe) description and as
> per me it should be tested before removing EXCLUDE_FROM_WORLD to ensure the
> builds are reproducible.


I'll try do that in the coming days and submit the patch if it is OK.


> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#188961):
> https://lists.openembedded.org/g/openembedded-core/message/188961
> Mute This Topic: https://lists.openembedded.org/mt/101890610/6213388
> Group Owner: openembedded-core+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [
> frederic.martinsons@gmail.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>
>

[-- Attachment #2: Type: text/html, Size: 1985 bytes --]

Re: [OE-core] [PATCH] rust: reproducibility issue fix

[Richard Purdie]

On Tue, 2023-10-10 at 21:05 -0700, Sundeep KOKKONDA via
lists.openembedded.org wrote:
> [Yocto#14875]
> The '--remap-path-prefix' option removes all references to build directory structure in the debug information within the compiled output for Cargo dependencies and the project's binary.
> However, some references to build directories remains in the final binary in .rustc section in the form of compressed metadata and this makes the build output dependent on the folder structure of the computer it's compiled on.
> So, for reproducible builds, use the configuration option 'remap-debuginfo = true' along with the '--remap-path-prefix'.
> 
> Signed-off-by: Sundeep KOKKONDA <sundeep.kokkonda@windriver.com>
> ---
>  meta/lib/oeqa/selftest/cases/reproducible.py | 2 --
>  meta/recipes-devtools/rust/rust_1.70.0.bb    | 1 +
>  2 files changed, 1 insertion(+), 2 deletions(-)
> 
> diff --git a/meta/lib/oeqa/selftest/cases/reproducible.py b/meta/lib/oeqa/selftest/cases/reproducible.py
> index 84c6c3a05f..9b4a0887dd 100644
> --- a/meta/lib/oeqa/selftest/cases/reproducible.py
> +++ b/meta/lib/oeqa/selftest/cases/reproducible.py
> @@ -16,8 +16,6 @@ import os
>  import datetime
>  
>  exclude_packages = [
> -	'rust',
> -	'rust-dbg'
>  	]
>  
>  def is_excluded(package):
> diff --git a/meta/recipes-devtools/rust/rust_1.70.0.bb b/meta/recipes-devtools/rust/rust_1.70.0.bb
> index 3b9c05a19f..16d433910f 100644
> --- a/meta/recipes-devtools/rust/rust_1.70.0.bb
> +++ b/meta/recipes-devtools/rust/rust_1.70.0.bb
> @@ -130,6 +130,7 @@ python do_configure() {
>      # [rust]
>      config.add_section("rust")
>      config.set("rust", "rpath", e(True))
> +    config.set("rust", "remap-debuginfo", e(True))
>      config.set("rust", "channel", e(d.expand("${RUST_CHANNEL}")))
>  
>      # Whether or not to optimize the compiler and standard library

Unfortunately I have some less good news, the rust reproducibility
tests do still have some kind of failure:

https://autobuilder.yoctoproject.org/typhoon/#/builders/117/builds/3736/steps/12/logs/stdio

http://autobuilder.yocto.io/pub/repro-fail/oe-reproducible-20231011-7wbq1syl/packages/diff-html/

This is a bit different to the previous issue since the rlib files are
not changing and this is rustdoc. At a quick glance, it looks like the
symbols are changing order in the file, which can sometimes be caused
by the link command not being in the same order on all builds. This
could be for example something doing:

$LD *.so

and hence the order of the linked objects depends on the order of the
files on disk and which order glob found them in. It could also be
something else.

So we've made great progress and this will help our SPDX issues as the
filenames are now consistent but we're not quite there yet I'm afraid.
Unless we can find it quickly we'll have to put rust back on the
exclusion list for the release.

Cheers,

Richard

Re: [OE-core] [PATCH] rust: reproducibility issue fix

[Frédéric Martinsons]

[-- Attachment #1: Type: text/plain, Size: 4013 bytes --]

On Thu, 12 Oct 2023 at 08:37, Richard Purdie <
richard.purdie@linuxfoundation.org> wrote:

> On Tue, 2023-10-10 at 21:05 -0700, Sundeep KOKKONDA via
> lists.openembedded.org wrote:
> > [Yocto#14875]
> > The '--remap-path-prefix' option removes all references to build
> directory structure in the debug information within the compiled output for
> Cargo dependencies and the project's binary.
> > However, some references to build directories remains in the final
> binary in .rustc section in the form of compressed metadata and this makes
> the build output dependent on the folder structure of the computer it's
> compiled on.
> > So, for reproducible builds, use the configuration option
> 'remap-debuginfo = true' along with the '--remap-path-prefix'.
> >
> > Signed-off-by: Sundeep KOKKONDA <sundeep.kokkonda@windriver.com>
> > ---
> >  meta/lib/oeqa/selftest/cases/reproducible.py | 2 --
> >  meta/recipes-devtools/rust/rust_1.70.0.bb    | 1 +
> >  2 files changed, 1 insertion(+), 2 deletions(-)
> >
> > diff --git a/meta/lib/oeqa/selftest/cases/reproducible.py
> b/meta/lib/oeqa/selftest/cases/reproducible.py
> > index 84c6c3a05f..9b4a0887dd 100644
> > --- a/meta/lib/oeqa/selftest/cases/reproducible.py
> > +++ b/meta/lib/oeqa/selftest/cases/reproducible.py
> > @@ -16,8 +16,6 @@ import os
> >  import datetime
> >
> >  exclude_packages = [
> > -     'rust',
> > -     'rust-dbg'
> >       ]
> >
> >  def is_excluded(package):
> > diff --git a/meta/recipes-devtools/rust/rust_1.70.0.bb
> b/meta/recipes-devtools/rust/rust_1.70.0.bb
> > index 3b9c05a19f..16d433910f 100644
> > --- a/meta/recipes-devtools/rust/rust_1.70.0.bb
> > +++ b/meta/recipes-devtools/rust/rust_1.70.0.bb
> > @@ -130,6 +130,7 @@ python do_configure() {
> >      # [rust]
> >      config.add_section("rust")
> >      config.set("rust", "rpath", e(True))
> > +    config.set("rust", "remap-debuginfo", e(True))
> >      config.set("rust", "channel", e(d.expand("${RUST_CHANNEL}")))
> >
> >      # Whether or not to optimize the compiler and standard library
>
> Unfortunately I have some less good news, the rust reproducibility
> tests do still have some kind of failure:
>
>
> https://autobuilder.yoctoproject.org/typhoon/#/builders/117/builds/3736/steps/12/logs/stdio
>
>
> http://autobuilder.yocto.io/pub/repro-fail/oe-reproducible-20231011-7wbq1syl/packages/diff-html/
>
> This is a bit different to the previous issue since the rlib files are
> not changing and this is rustdoc. At a quick glance, it looks like the
> symbols are changing order in the file, which can sometimes be caused
> by the link command not being in the same order on all builds. This
> could be for example something doing:
>
> $LD *.so
>
> and hence the order of the linked objects depends on the order of the
> files on disk and which order glob found them in. It could also be
> something else.
>
> So we've made great progress and this will help our SPDX issues as the
> filenames are now consistent but we're not quite there yet I'm afraid.
> Unless we can find it quickly we'll have to put rust back on the
> exclusion list for the release.
>
> Cheers,
>
> Richard
>
>
Hello, just to confirm that I still diff for zvariant package , I tested
with the followin setup on poky master:

local;conf:

SANITY_TESTED_DISTROS = ""
OEQA_REPRODUCIBLE_TEST_TARGET = "zvariant"
OEQA_REPRODUCIBLE_TEST_SSTATE_TARGETS = "rust-native cargo-native"

and

OEQA_DEBUGGING_SAVED_OUTPUT=/home/jenkins/yocto-poky-master/reproducible_saved_output
oe-selftest -r reproducible.ReproducibleTests.test_reproducible_builds


> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#188988):
> https://lists.openembedded.org/g/openembedded-core/message/188988
> Mute This Topic: https://lists.openembedded.org/mt/101890610/6213388
> Group Owner: openembedded-core+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [
> frederic.martinsons@gmail.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>
>

[-- Attachment #2: Type: text/html, Size: 6392 bytes --]