[Buildroot] [PATCH] Revert "support/download: generate even more reproducible tarballs"

[Buildroot] [PATCH] Revert "support/download: generate even more reproducible tarballs"

[Yann E. MORIN]

Commit 768f9f80f62c (support/download: generate even more reproducible
tarballs) causes non-reproducibility in tarballs we previousy
generated, especially the archives for two cargo-vendored packages,
ripgrep and sentry-cli.

The cause is that those two pakcages eventually vendor a file that has
the u+x bit set, but is otehrwise go-x. With 768f9f80f62c, the files are
now go+x, so the hash for those generated archives has changed.

Besides, that commit was wrong: it did not account for the 'r' bit for
go part, leaving some non-reproducibility still unaccounted for.

So, to generate really reproducible archives, we would need to fix that
read bit as well, and that has the potential to affect all the archives
we generated so far. If we wanted to do so, we'd need a way to version
all generated archives, like we do for git and svn, but now for all the
different CVSes, as well as for all the vendoring post-processes.

For 768f9f80f62c, all that was of conern was the working copies of CVSes
(i.e. git, svn, cvs...) that we cache in the Buildroot download dir, not
the temporary files during post-processing. Indeed, in that latter case,
the user has virtually no way to mangle with the mode of the
intermediate extract before repack.

And we do have a big fat warning that users should not attempt to meddle
with the git tree that Buildroot caches.

As 768f9f80f62c however demonstrates, is that it took quite a long time
between the introduction of the git caching, and the time someone
eventually discovered they could meddle in there. This shows that the
issue it not actually critical in most setups.

Also, the tar manual [0] hints at a better solution to handle
reproducibility, which even avoids touching the files on disk which is
even nicer:

    ‘--mode='go+u,go-w'’
        Omit irrelevant information about file permissions.

If we were to actually handle the mode bit for reproducibility, we'd
need to:
  - introduce archive versioning for all download backends and
    prost-processing
  - use the tar officially suggested method

So, revert that change, as it was incomplete, was not really fixing much
issues, and causes actual issues.

This reverts commit 768f9f80f62c1da6e298c680f0f4bfa887f38c78.

[0] https://www.gnu.org/software/tar/manual/tar.html#Reproducibility

Thanks to Vincent and Arnout for pointing at the tar manual.

Reported-by: Antoine Coutant <antoine.coutant@smile.fr>
Reported-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Vincent Fazio <vfazio@xes-inc.com>
Cc: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
---
 support/download/helpers | 3 ---
 1 file changed, 3 deletions(-)

diff --git a/support/download/helpers b/support/download/helpers
index 265685eff5..90a7d6c1ec 100755
--- a/support/download/helpers
+++ b/support/download/helpers
@@ -53,9 +53,6 @@ mk_tar_gz() {
     tmp="$(mktemp --tmpdir="$(pwd)")"
     pushd "${in_dir}" >/dev/null
 
-    # Enforce group/others mode bits
-    chmod -R go-wx+X .
-
     # Establish list
     find . -not -type d -and -not \( -false "${find_opts[@]}" \) >"${tmp}.list"
     # Sort list for reproducibility
-- 
2.43.0

_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH] Revert "support/download: generate even more reproducible tarballs"

[Romain Naour]

Hello Yann,

Le 01/01/2024 à 22:51, Yann E. MORIN a écrit :
> Commit 768f9f80f62c (support/download: generate even more reproducible
> tarballs) causes non-reproducibility in tarballs we previousy
> generated, especially the archives for two cargo-vendored packages,
> ripgrep and sentry-cli.
> 
> The cause is that those two pakcages eventually vendor a file that has
> the u+x bit set, but is otehrwise go-x. With 768f9f80f62c, the files are
> now go+x, so the hash for those generated archives has changed.
> 
> Besides, that commit was wrong: it did not account for the 'r' bit for
> go part, leaving some non-reproducibility still unaccounted for.
> 
> So, to generate really reproducible archives, we would need to fix that
> read bit as well, and that has the potential to affect all the archives
> we generated so far. If we wanted to do so, we'd need a way to version
> all generated archives, like we do for git and svn, but now for all the
> different CVSes, as well as for all the vendoring post-processes.
> 
> For 768f9f80f62c, all that was of conern was the working copies of CVSes
> (i.e. git, svn, cvs...) that we cache in the Buildroot download dir, not
> the temporary files during post-processing. Indeed, in that latter case,
> the user has virtually no way to mangle with the mode of the
> intermediate extract before repack.
> 
> And we do have a big fat warning that users should not attempt to meddle
> with the git tree that Buildroot caches.
> 
> As 768f9f80f62c however demonstrates, is that it took quite a long time
> between the introduction of the git caching, and the time someone
> eventually discovered they could meddle in there. This shows that the
> issue it not actually critical in most setups.
> 
> Also, the tar manual [0] hints at a better solution to handle
> reproducibility, which even avoids touching the files on disk which is
> even nicer:
> 
>     ‘--mode='go+u,go-w'’
>         Omit irrelevant information about file permissions.
> 
> If we were to actually handle the mode bit for reproducibility, we'd
> need to:
>   - introduce archive versioning for all download backends and
>     prost-processing
>   - use the tar officially suggested method
> 
> So, revert that change, as it was incomplete, was not really fixing much
> issues, and causes actual issues.
> 
> This reverts commit 768f9f80f62c1da6e298c680f0f4bfa887f38c78.
> 
> [0] https://www.gnu.org/software/tar/manual/tar.html#Reproducibility
> 
> Thanks to Vincent and Arnout for pointing at the tar manual.

Tested-by: Romain Naour <romain.naour@smile.fr>

Tested using TestRustVendoring from [1]:

With master the test fail due the wrong hash:

ERROR: while checking hashes from package/ripgrep//ripgrep.hash
ERROR: ripgrep-af6b6c543b224d348a8876f0c06245d9ea7929c5.tar.gz has wrong sha256
hash:
ERROR: expected: 6f1d4a8b653ce48d59ad777288b1257cbda607db29db19d031b7e622c60526f8
ERROR: got     : 9d9769e45ffe6089f58bc19fa39dd6b6299aa0c3ad90508d21dfa27a3d3416d5

With this patch applied the test pass.

[1] http://lists.busybox.net/pipermail/buildroot/2023-December/681765.html

Best regards,
Romain

> 
> Reported-by: Antoine Coutant <antoine.coutant@smile.fr>
> Reported-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
> Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
> Cc: Vincent Fazio <vfazio@xes-inc.com>
> Cc: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
> ---
>  support/download/helpers | 3 ---
>  1 file changed, 3 deletions(-)
> 
> diff --git a/support/download/helpers b/support/download/helpers
> index 265685eff5..90a7d6c1ec 100755
> --- a/support/download/helpers
> +++ b/support/download/helpers
> @@ -53,9 +53,6 @@ mk_tar_gz() {
>      tmp="$(mktemp --tmpdir="$(pwd)")"
>      pushd "${in_dir}" >/dev/null
>  
> -    # Enforce group/others mode bits
> -    chmod -R go-wx+X .
> -
>      # Establish list
>      find . -not -type d -and -not \( -false "${find_opts[@]}" \) >"${tmp}.list"
>      # Sort list for reproducibility

_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH] Revert "support/download: generate even more reproducible tarballs"

[Peter Korsgaard]

>>>>> "Yann" == Yann E MORIN <yann.morin.1998@free.fr> writes:

 > Commit 768f9f80f62c (support/download: generate even more reproducible
 > tarballs) causes non-reproducibility in tarballs we previousy
 > generated, especially the archives for two cargo-vendored packages,
 > ripgrep and sentry-cli.

 > The cause is that those two pakcages eventually vendor a file that has
 > the u+x bit set, but is otehrwise go-x. With 768f9f80f62c, the files are
 > now go+x, so the hash for those generated archives has changed.

 > Besides, that commit was wrong: it did not account for the 'r' bit for
 > go part, leaving some non-reproducibility still unaccounted for.

 > So, to generate really reproducible archives, we would need to fix that
 > read bit as well, and that has the potential to affect all the archives
 > we generated so far. If we wanted to do so, we'd need a way to version
 > all generated archives, like we do for git and svn, but now for all the
 > different CVSes, as well as for all the vendoring post-processes.

 > For 768f9f80f62c, all that was of conern was the working copies of CVSes
 > (i.e. git, svn, cvs...) that we cache in the Buildroot download dir, not
 > the temporary files during post-processing. Indeed, in that latter case,
 > the user has virtually no way to mangle with the mode of the
 > intermediate extract before repack.

 > And we do have a big fat warning that users should not attempt to meddle
 > with the git tree that Buildroot caches.

 > As 768f9f80f62c however demonstrates, is that it took quite a long time
 > between the introduction of the git caching, and the time someone
 > eventually discovered they could meddle in there. This shows that the
 > issue it not actually critical in most setups.

 > Also, the tar manual [0] hints at a better solution to handle
 > reproducibility, which even avoids touching the files on disk which is
 > even nicer:

 >     ‘--mode='go+u,go-w'’
 >         Omit irrelevant information about file permissions.

 > If we were to actually handle the mode bit for reproducibility, we'd
 > need to:
 >   - introduce archive versioning for all download backends and
 >     prost-processing
 >   - use the tar officially suggested method

 > So, revert that change, as it was incomplete, was not really fixing much
 > issues, and causes actual issues.

 > This reverts commit 768f9f80f62c1da6e298c680f0f4bfa887f38c78.

 > [0] https://www.gnu.org/software/tar/manual/tar.html#Reproducibility

 > Thanks to Vincent and Arnout for pointing at the tar manual.

 > Reported-by: Antoine Coutant <antoine.coutant@smile.fr>
 > Reported-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
 > Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
 > Cc: Vincent Fazio <vfazio@xes-inc.com>
 > Cc: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>

Committed to 2023.02.x and 2023.11.x, thanks.

-- 
Bye, Peter Korsgaard
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot