Re: [Buildroot] [PATCH] Revert "support/download: generate even more reproducible tarballs"

[Peter Korsgaard <peter@korsgaard.com>]

>>>>> "Yann" == Yann E MORIN <yann.morin.1998@free.fr> writes:

 > Commit 768f9f80f62c (support/download: generate even more reproducible
 > tarballs) causes non-reproducibility in tarballs we previousy
 > generated, especially the archives for two cargo-vendored packages,
 > ripgrep and sentry-cli.

 > The cause is that those two pakcages eventually vendor a file that has
 > the u+x bit set, but is otehrwise go-x. With 768f9f80f62c, the files are
 > now go+x, so the hash for those generated archives has changed.

 > Besides, that commit was wrong: it did not account for the 'r' bit for
 > go part, leaving some non-reproducibility still unaccounted for.

 > So, to generate really reproducible archives, we would need to fix that
 > read bit as well, and that has the potential to affect all the archives
 > we generated so far. If we wanted to do so, we'd need a way to version
 > all generated archives, like we do for git and svn, but now for all the
 > different CVSes, as well as for all the vendoring post-processes.

 > For 768f9f80f62c, all that was of conern was the working copies of CVSes
 > (i.e. git, svn, cvs...) that we cache in the Buildroot download dir, not
 > the temporary files during post-processing. Indeed, in that latter case,
 > the user has virtually no way to mangle with the mode of the
 > intermediate extract before repack.

 > And we do have a big fat warning that users should not attempt to meddle
 > with the git tree that Buildroot caches.

 > As 768f9f80f62c however demonstrates, is that it took quite a long time
 > between the introduction of the git caching, and the time someone
 > eventually discovered they could meddle in there. This shows that the
 > issue it not actually critical in most setups.

 > Also, the tar manual [0] hints at a better solution to handle
 > reproducibility, which even avoids touching the files on disk which is
 > even nicer:

 >     ‘--mode='go+u,go-w'’
 >         Omit irrelevant information about file permissions.

 > If we were to actually handle the mode bit for reproducibility, we'd
 > need to:
 >   - introduce archive versioning for all download backends and
 >     prost-processing
 >   - use the tar officially suggested method

 > So, revert that change, as it was incomplete, was not really fixing much
 > issues, and causes actual issues.

 > This reverts commit 768f9f80f62c1da6e298c680f0f4bfa887f38c78.

 > [0] https://www.gnu.org/software/tar/manual/tar.html#Reproducibility

 > Thanks to Vincent and Arnout for pointing at the tar manual.

 > Reported-by: Antoine Coutant <antoine.coutant@smile.fr>
 > Reported-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
 > Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
 > Cc: Vincent Fazio <vfazio@xes-inc.com>
 > Cc: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>

Committed to 2023.02.x and 2023.11.x, thanks.

-- 
Bye, Peter Korsgaard
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot