* [Buildroot] [PATCH 1/1] support/dependencies: bump minimal tar version to 1.35 @ 2023-10-18 14:11 Vincent Fazio 2023-10-18 17:43 ` Arnout Vandecappelle via buildroot 2023-11-08 21:01 ` Arnout Vandecappelle via buildroot 0 siblings, 2 replies; 7+ messages in thread From: Vincent Fazio @ 2023-10-18 14:11 UTC (permalink / raw) To: buildroot; +Cc: Yann E . MORIN, Vincent Fazio GNU tar 1.35 includes a breaking change [0] that changes the tar header created for each regular file even for "stable" formats like pax (which we use in support/download/helpers::mk_tar_gz). Previously, normal files had the devmajor/devminor fields of the header filled with ASCII zero (0x30). These fields are now null which also affects the checksum value in the header as well. $ diff <(head -n 100 xxd.1.34.output) <(head -n 100 xxd.1.35.output) 10c10 < 00000090: 3037 3500 3031 3531 3637 0020 3000 0000 075.015167. 0... --- > 00000090: 3037 3500 3031 3337 3237 0020 3000 0000 075.013727. 0... 21,22c21,22 < 00000140: 0000 0000 0000 0000 0030 3030 3030 3030 .........0000000 < 00000150: 0030 3030 3030 3030 0000 0000 0000 0000 .0000000........ --- > 00000140: 0000 0000 0000 0000 0000 0000 0000 0000 ................ > 00000150: 0000 0000 0000 0000 0000 0000 0000 0000 ................ This has the consequence of causing hash mismatches for any tarball created with a version prior to 1.35. Instead of reverting back to a host specific version (see 37a909cacf) and adding a cap to the maximal tar version (see ec50e407be), just establish that 1.35+ should be used going forward. This change requires an updated BR_FMT_VERSION for both git and svn. [0] https://git.savannah.gnu.org/cgit/tar.git/commit/?id=738de9ecdec45ebfeb999628742373b5f8253bd0 Signed-off-by: Vincent Fazio <vfazio@gmail.com> --- This patch should be coordinated with other patches [1] that update the BR_FMT_VERSION revision for git or svn as all of the tarball hashes will need to be regenerated tree-wide. [1] https://patchwork.ozlabs.org/project/buildroot/list/?series=373742 --- package/pkg-download.mk | 4 ++-- support/dependencies/check-host-tar.sh | 10 ++++++---- 2 files changed, 8 insertions(+), 6 deletions(-) diff --git a/package/pkg-download.mk b/package/pkg-download.mk index e5cd83d859..d33eb2d811 100644 --- a/package/pkg-download.mk +++ b/package/pkg-download.mk @@ -20,8 +20,8 @@ export LOCALFILES := $(call qstrip,$(BR2_LOCALFILES)) # Version of the format of the archives we generate in the corresponding # download backend: -BR_FMT_VERSION_git = -br1 -BR_FMT_VERSION_svn = -br3 +BR_FMT_VERSION_git = -br2 +BR_FMT_VERSION_svn = -br4 DL_WRAPPER = support/download/dl-wrapper diff --git a/support/dependencies/check-host-tar.sh b/support/dependencies/check-host-tar.sh index b7d607a47a..d56d0242a9 100755 --- a/support/dependencies/check-host-tar.sh +++ b/support/dependencies/check-host-tar.sh @@ -27,11 +27,13 @@ if [ -n "${version_bsd}" ] ; then exit 1 fi -# Minimal version = 1.27 (previous versions do not correctly unpack archives -# containing hard-links if the --strip-components option is used or create -# different gnu long link headers for path elements > 100 characters). +# Minimal version = 1.35 +# GNU tar upstream commit 738de9ecd introduced a "breaking" change that +# affects tarballs regardless of mode (pax/ustar/v7) such that tarballs +# generated with 1.35 will not hash to the same value as those generated +# by previous versions of tar. major_min=1 -minor_min=27 +minor_min=35 if [ $major -lt $major_min ]; then # echo nothing: no suitable tar found -- 2.34.1 _______________________________________________ buildroot mailing list buildroot@buildroot.org https://lists.buildroot.org/mailman/listinfo/buildroot ^ permalink raw reply related [flat|nested] 7+ messages in thread
* Re: [Buildroot] [PATCH 1/1] support/dependencies: bump minimal tar version to 1.35 2023-10-18 14:11 [Buildroot] [PATCH 1/1] support/dependencies: bump minimal tar version to 1.35 Vincent Fazio @ 2023-10-18 17:43 ` Arnout Vandecappelle via buildroot 2023-10-21 20:25 ` Yann E. MORIN 2023-11-08 21:01 ` Arnout Vandecappelle via buildroot 1 sibling, 1 reply; 7+ messages in thread From: Arnout Vandecappelle via buildroot @ 2023-10-18 17:43 UTC (permalink / raw) To: Vincent Fazio, buildroot; +Cc: Yann E . MORIN On 18/10/2023 16:11, Vincent Fazio wrote: > GNU tar 1.35 includes a breaking change [0] that changes the tar header > created for each regular file even for "stable" formats like pax (which > we use in support/download/helpers::mk_tar_gz). The funny thing is that the tar docs have a section about making the archive reproducible [1]. They call it "more reproducible", though, so perhaps they realize that they fail... > Previously, normal files had the devmajor/devminor fields of the header > filled with ASCII zero (0x30). These fields are now null which also > affects the checksum value in the header as well. > > $ diff <(head -n 100 xxd.1.34.output) <(head -n 100 xxd.1.35.output) > 10c10 > < 00000090: 3037 3500 3031 3531 3637 0020 3000 0000 075.015167. 0... > --- > > 00000090: 3037 3500 3031 3337 3237 0020 3000 0000 075.013727. 0... > 21,22c21,22 > < 00000140: 0000 0000 0000 0000 0030 3030 3030 3030 .........0000000 > < 00000150: 0030 3030 3030 3030 0000 0000 0000 0000 .0000000........ > --- > > 00000140: 0000 0000 0000 0000 0000 0000 0000 0000 ................ > > 00000150: 0000 0000 0000 0000 0000 0000 0000 0000 ................ > > This has the consequence of causing hash mismatches for any tarball > created with a version prior to 1.35. > > Instead of reverting back to a host specific version (see 37a909cacf) > and adding a cap to the maximal tar version (see ec50e407be), just > establish that 1.35+ should be used going forward. > > This change requires an updated BR_FMT_VERSION for both git and svn. > > [0] https://git.savannah.gnu.org/cgit/tar.git/commit/?id=738de9ecdec45ebfeb999628742373b5f8253bd0 > > Signed-off-by: Vincent Fazio <vfazio@gmail.com> > --- > This patch should be coordinated with other patches [1] that update the > BR_FMT_VERSION revision for git or svn as all of the tarball hashes will > need to be regenerated tree-wide. > > [1] https://patchwork.ozlabs.org/project/buildroot/list/?series=373742 > --- > package/pkg-download.mk | 4 ++-- > support/dependencies/check-host-tar.sh | 10 ++++++---- > 2 files changed, 8 insertions(+), 6 deletions(-) > > diff --git a/package/pkg-download.mk b/package/pkg-download.mk > index e5cd83d859..d33eb2d811 100644 > --- a/package/pkg-download.mk > +++ b/package/pkg-download.mk > @@ -20,8 +20,8 @@ export LOCALFILES := $(call qstrip,$(BR2_LOCALFILES)) > > # Version of the format of the archives we generate in the corresponding > # download backend: > -BR_FMT_VERSION_git = -br1 > -BR_FMT_VERSION_svn = -br3 > +BR_FMT_VERSION_git = -br2 > +BR_FMT_VERSION_svn = -br4 Vendored cargo and go packages will have the same issue, no? Those will need to be updated as well... Perhaps we should look for an alternative archive format that is actually reproducible. If we anyway have to do a mass update, then it's better if now is the last time... Though I don't know if an actually reproducible format exists. GNU cpio has the --reproducible option, but I don't know if it guarantees reproducibility across cpio versions. Also, we still have gzip to contend with. Regards, Arnout [1] https://www.gnu.org/software/tar/manual/html_chapter/Formats.html#Reproducibility > > DL_WRAPPER = support/download/dl-wrapper > > diff --git a/support/dependencies/check-host-tar.sh b/support/dependencies/check-host-tar.sh > index b7d607a47a..d56d0242a9 100755 > --- a/support/dependencies/check-host-tar.sh > +++ b/support/dependencies/check-host-tar.sh > @@ -27,11 +27,13 @@ if [ -n "${version_bsd}" ] ; then > exit 1 > fi > > -# Minimal version = 1.27 (previous versions do not correctly unpack archives > -# containing hard-links if the --strip-components option is used or create > -# different gnu long link headers for path elements > 100 characters). > +# Minimal version = 1.35 > +# GNU tar upstream commit 738de9ecd introduced a "breaking" change that > +# affects tarballs regardless of mode (pax/ustar/v7) such that tarballs > +# generated with 1.35 will not hash to the same value as those generated > +# by previous versions of tar. > major_min=1 > -minor_min=27 > +minor_min=35 > > if [ $major -lt $major_min ]; then > # echo nothing: no suitable tar found _______________________________________________ buildroot mailing list buildroot@buildroot.org https://lists.buildroot.org/mailman/listinfo/buildroot ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [Buildroot] [PATCH 1/1] support/dependencies: bump minimal tar version to 1.35 2023-10-18 17:43 ` Arnout Vandecappelle via buildroot @ 2023-10-21 20:25 ` Yann E. MORIN 2023-10-21 20:36 ` Yann E. MORIN 0 siblings, 1 reply; 7+ messages in thread From: Yann E. MORIN @ 2023-10-21 20:25 UTC (permalink / raw) To: Arnout Vandecappelle; +Cc: Vincent Fazio, buildroot Vincent, Arnout, All, On 2023-10-18 19:43 +0200, Arnout Vandecappelle via buildroot spake thusly: > On 18/10/2023 16:11, Vincent Fazio wrote: > >GNU tar 1.35 includes a breaking change [0] that changes the tar header > >created for each regular file even for "stable" formats like pax (which > >we use in support/download/helpers::mk_tar_gz). > The funny thing is that the tar docs have a section about making the > archive reproducible [1]. They call it "more reproducible", though, so > perhaps they realize that they fail... Yes, I think this is what served as a base for Vincent to come up with the options we have in the tar helper. [--SNIP--] > >diff --git a/package/pkg-download.mk b/package/pkg-download.mk > >index e5cd83d859..d33eb2d811 100644 > >--- a/package/pkg-download.mk > >+++ b/package/pkg-download.mk > >@@ -20,8 +20,8 @@ export LOCALFILES := $(call qstrip,$(BR2_LOCALFILES)) > > # Version of the format of the archives we generate in the corresponding > > # download backend: > >-BR_FMT_VERSION_git = -br1 > >-BR_FMT_VERSION_svn = -br3 > >+BR_FMT_VERSION_git = -br2 > >+BR_FMT_VERSION_svn = -br4 > Vendored cargo and go packages will have the same issue, no? Those will > need to be updated as well... Good catch, indeed. I'll try and resurect my patch to add a -br suffix to those... > Perhaps we should look for an alternative archive format that is actually > reproducible. If we anyway have to do a mass update, then it's better if now > is the last time... Though I don't know if an actually reproducible format > exists. There is a lost there: https://en.wikipedia.org/wiki/List_of_archive_formats but it does not provide any information about stability... We could of course come up with our own, but I would very much like that we do avoid that at all cost. The odds of us failing if we go that route are higher than those of us succeeding, I'd say... > GNU cpio has the --reproducible option, but I don't know if it > guarantees reproducibility across cpio versions. [0] states that --reproducible "is equivalent to ‘--ignore-devno --ignore-dirnlink --renumber-inodes’", which to me means that some metadata is not recorded (or is faked), not that there is any guarantee on the stability of the stream. [0] https://www.gnu.org/software/cpio/manual/cpio.html#Copy_002dpass-mode > Also, we still have gzip to > contend with. I did test that a while ago, and gzip *has* been ultra-super stable over the years, to the point that all the versions that I could build (we was almost back to the initial release 30 yerrs ago) did produce the same output as the latest version. So, if I had to worry about reproducibility, I would not look at gzip. Regards, Yann E. MORIN. > Regards, > Arnout > > [1] https://www.gnu.org/software/tar/manual/html_chapter/Formats.html#Reproducibility > > > > DL_WRAPPER = support/download/dl-wrapper > >diff --git a/support/dependencies/check-host-tar.sh b/support/dependencies/check-host-tar.sh > >index b7d607a47a..d56d0242a9 100755 > >--- a/support/dependencies/check-host-tar.sh > >+++ b/support/dependencies/check-host-tar.sh > >@@ -27,11 +27,13 @@ if [ -n "${version_bsd}" ] ; then > > exit 1 > > fi > >-# Minimal version = 1.27 (previous versions do not correctly unpack archives > >-# containing hard-links if the --strip-components option is used or create > >-# different gnu long link headers for path elements > 100 characters). > >+# Minimal version = 1.35 > >+# GNU tar upstream commit 738de9ecd introduced a "breaking" change that > >+# affects tarballs regardless of mode (pax/ustar/v7) such that tarballs > >+# generated with 1.35 will not hash to the same value as those generated > >+# by previous versions of tar. > > major_min=1 > >-minor_min=27 > >+minor_min=35 > > if [ $major -lt $major_min ]; then > > # echo nothing: no suitable tar found > _______________________________________________ > buildroot mailing list > buildroot@buildroot.org > https://lists.buildroot.org/mailman/listinfo/buildroot -- .-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 561 099 427 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------' _______________________________________________ buildroot mailing list buildroot@buildroot.org https://lists.buildroot.org/mailman/listinfo/buildroot ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [Buildroot] [PATCH 1/1] support/dependencies: bump minimal tar version to 1.35 2023-10-21 20:25 ` Yann E. MORIN @ 2023-10-21 20:36 ` Yann E. MORIN 2023-10-23 7:45 ` Arnout Vandecappelle via buildroot 0 siblings, 1 reply; 7+ messages in thread From: Yann E. MORIN @ 2023-10-21 20:36 UTC (permalink / raw) To: Arnout Vandecappelle; +Cc: Vincent Fazio, buildroot Vincent, Arnout, All, On 2023-10-21 22:25 +0200, Yann E. MORIN spake thusly: > On 2023-10-18 19:43 +0200, Arnout Vandecappelle via buildroot spake thusly: > > On 18/10/2023 16:11, Vincent Fazio wrote: > > >GNU tar 1.35 includes a breaking change [0] that changes the tar header > > >created for each regular file even for "stable" formats like pax (which > > >we use in support/download/helpers::mk_tar_gz). [--SNIP--] > > >diff --git a/package/pkg-download.mk b/package/pkg-download.mk > > >index e5cd83d859..d33eb2d811 100644 > > >--- a/package/pkg-download.mk > > >+++ b/package/pkg-download.mk > > >@@ -20,8 +20,8 @@ export LOCALFILES := $(call qstrip,$(BR2_LOCALFILES)) > > > # Version of the format of the archives we generate in the corresponding > > > # download backend: > > >-BR_FMT_VERSION_git = -br1 > > >-BR_FMT_VERSION_svn = -br3 > > >+BR_FMT_VERSION_git = -br2 > > >+BR_FMT_VERSION_svn = -br4 > > Vendored cargo and go packages will have the same issue, no? Those will > > need to be updated as well... > Good catch, indeed. I'll try and resurect my patch to add a -br suffix > to those... I think it was basically just that: b/package/pkg-download.mk | 1 1 0 0 + b/package/pkg-utils.mk | 2 1 1 0 +- 2 files changed, 2 insertions(+), 1 deletion(-) diff --git a/package/pkg-download.mk b/package/pkg-download.mk index e5cd83d859..5ad3f2baaf 100644 --- a/package/pkg-download.mk +++ b/package/pkg-download.mk @@ -22,6 +22,7 @@ export LOCALFILES := $(call qstrip,$(BR2_LOCALFILES)) # download backend: BR_FMT_VERSION_git = -br1 BR_FMT_VERSION_svn = -br3 +BR_FMT_VERSION_cargo = -cargo1 DL_WRAPPER = support/download/dl-wrapper diff --git a/package/pkg-utils.mk b/package/pkg-utils.mk index 7c670ee615..95a8452db8 100644 --- a/package/pkg-utils.mk +++ b/package/pkg-utils.mk @@ -48,7 +48,7 @@ pkgname = $(lastword $(subst /, ,$(pkgdir))) # Helper to build the extension for a package archive, based on various # conditions. # $(1): upper-case package name -pkg_source_ext = $(BR_FMT_VERSION_$($(1)_SITE_METHOD)).tar.gz +pkg_source_ext = $(BR_FMT_VERSION_$($(1)_SITE_METHOD))$(BR_FMT_VERSION_$($(1)_DOWNLOAD_POST_PROCESS)).tar.gz # Define extractors for different archive suffixes INFLATE.bz2 = $(BZCAT) Toally untested. Regards, Yann E. MORIN. -- .-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 561 099 427 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------' _______________________________________________ buildroot mailing list buildroot@buildroot.org https://lists.buildroot.org/mailman/listinfo/buildroot ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [Buildroot] [PATCH 1/1] support/dependencies: bump minimal tar version to 1.35 2023-10-21 20:36 ` Yann E. MORIN @ 2023-10-23 7:45 ` Arnout Vandecappelle via buildroot 0 siblings, 0 replies; 7+ messages in thread From: Arnout Vandecappelle via buildroot @ 2023-10-23 7:45 UTC (permalink / raw) To: Yann E. MORIN; +Cc: Vincent Fazio, buildroot On 21/10/2023 22:36, Yann E. MORIN wrote: > Vincent, Arnout, All, > > On 2023-10-21 22:25 +0200, Yann E. MORIN spake thusly: >> On 2023-10-18 19:43 +0200, Arnout Vandecappelle via buildroot spake thusly: >>> On 18/10/2023 16:11, Vincent Fazio wrote: >>>> GNU tar 1.35 includes a breaking change [0] that changes the tar header >>>> created for each regular file even for "stable" formats like pax (which >>>> we use in support/download/helpers::mk_tar_gz). > [--SNIP--] >>>> diff --git a/package/pkg-download.mk b/package/pkg-download.mk >>>> index e5cd83d859..d33eb2d811 100644 >>>> --- a/package/pkg-download.mk >>>> +++ b/package/pkg-download.mk >>>> @@ -20,8 +20,8 @@ export LOCALFILES := $(call qstrip,$(BR2_LOCALFILES)) >>>> # Version of the format of the archives we generate in the corresponding >>>> # download backend: >>>> -BR_FMT_VERSION_git = -br1 >>>> -BR_FMT_VERSION_svn = -br3 >>>> +BR_FMT_VERSION_git = -br2 >>>> +BR_FMT_VERSION_svn = -br4 >>> Vendored cargo and go packages will have the same issue, no? Those will >>> need to be updated as well... >> Good catch, indeed. I'll try and resurect my patch to add a -br suffix >> to those... > > I think it was basically just that: > > b/package/pkg-download.mk | 1 1 0 0 + > b/package/pkg-utils.mk | 2 1 1 0 +- > 2 files changed, 2 insertions(+), 1 deletion(-) > > diff --git a/package/pkg-download.mk b/package/pkg-download.mk > index e5cd83d859..5ad3f2baaf 100644 > --- a/package/pkg-download.mk > +++ b/package/pkg-download.mk > @@ -22,6 +22,7 @@ export LOCALFILES := $(call qstrip,$(BR2_LOCALFILES)) > # download backend: > BR_FMT_VERSION_git = -br1 > BR_FMT_VERSION_svn = -br3 > +BR_FMT_VERSION_cargo = -cargo1 > > DL_WRAPPER = support/download/dl-wrapper > > diff --git a/package/pkg-utils.mk b/package/pkg-utils.mk > index 7c670ee615..95a8452db8 100644 > --- a/package/pkg-utils.mk > +++ b/package/pkg-utils.mk > @@ -48,7 +48,7 @@ pkgname = $(lastword $(subst /, ,$(pkgdir))) > # Helper to build the extension for a package archive, based on various > # conditions. > # $(1): upper-case package name > -pkg_source_ext = $(BR_FMT_VERSION_$($(1)_SITE_METHOD)).tar.gz > +pkg_source_ext = $(BR_FMT_VERSION_$($(1)_SITE_METHOD))$(BR_FMT_VERSION_$($(1)_DOWNLOAD_POST_PROCESS)).tar.gz When POST_PROCESS is used, the SITE_METHOD is not actually relevant (we don't actually make a tarball of the git, or at least we don't hash it). So maybe: pkg_source_ext = $(or $(BR_FMT_VERSION_$($(1)_DOWNLOAD_POST_PROCESS)),$(BR_FMT_VERSION_$($(1)_SITE_METHOD))).tar.gz Then we an also make BR_FMT_VERSION_cargo = -br4 (not -br1 or -br3 to avoid conflict with already existing git or svn based tarballs). Regards, Arnout > > # Define extractors for different archive suffixes > INFLATE.bz2 = $(BZCAT) > > Toally untested. > > Regards, > Yann E. MORIN. > _______________________________________________ buildroot mailing list buildroot@buildroot.org https://lists.buildroot.org/mailman/listinfo/buildroot ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [Buildroot] [PATCH 1/1] support/dependencies: bump minimal tar version to 1.35 2023-10-18 14:11 [Buildroot] [PATCH 1/1] support/dependencies: bump minimal tar version to 1.35 Vincent Fazio 2023-10-18 17:43 ` Arnout Vandecappelle via buildroot @ 2023-11-08 21:01 ` Arnout Vandecappelle via buildroot 2023-11-09 15:18 ` Vincent Fazio 1 sibling, 1 reply; 7+ messages in thread From: Arnout Vandecappelle via buildroot @ 2023-11-08 21:01 UTC (permalink / raw) To: Vincent Fazio, buildroot; +Cc: Yann E . MORIN Hi Vincent, On 18/10/2023 16:11, Vincent Fazio wrote: > GNU tar 1.35 includes a breaking change [0] that changes the tar header > created for each regular file even for "stable" formats like pax (which > we use in support/download/helpers::mk_tar_gz). > > Previously, normal files had the devmajor/devminor fields of the header > filled with ASCII zero (0x30). These fields are now null which also > affects the checksum value in the header as well. > > $ diff <(head -n 100 xxd.1.34.output) <(head -n 100 xxd.1.35.output) > 10c10 > < 00000090: 3037 3500 3031 3531 3637 0020 3000 0000 075.015167. 0... > --- > > 00000090: 3037 3500 3031 3337 3237 0020 3000 0000 075.013727. 0... > 21,22c21,22 > < 00000140: 0000 0000 0000 0000 0030 3030 3030 3030 .........0000000 > < 00000150: 0030 3030 3030 3030 0000 0000 0000 0000 .0000000........ > --- > > 00000140: 0000 0000 0000 0000 0000 0000 0000 0000 ................ > > 00000150: 0000 0000 0000 0000 0000 0000 0000 0000 ................ > > This has the consequence of causing hash mismatches for any tarball > created with a version prior to 1.35. > > Instead of reverting back to a host specific version (see 37a909cacf) > and adding a cap to the maximal tar version (see ec50e407be), just > establish that 1.35+ should be used going forward. > > This change requires an updated BR_FMT_VERSION for both git and svn. > > [0] https://git.savannah.gnu.org/cgit/tar.git/commit/?id=738de9ecdec45ebfeb999628742373b5f8253bd0 > > Signed-off-by: Vincent Fazio <vfazio@gmail.com> We discussed with 3 more maintainers (Peter, Romain and myself) and came to the conclusion that it is really too annoying that almost everybody would have to build host-tar (unconditionally!). At the moment, of the main distros, only Fedora 39 (just released) and Arch Linux already have 1.35. The idea is to apply this "bump minimal tar" only when more major distros have switched to 1.35 or later. Or, perhaps with the additional time we find an alternative that is hopefully more stable. I was thinking of switching to sharutils, for example - last release is from 2015 so very stable! :-) So, for the time being, we can do the following: - in check-host-tar.sh, check that system tar < 1.35; - in package/tar, add a patch that reverts 738de9ecde (probably need to revert c3663838 as well). Regarding ec50e407be: we're indeed in a similar situation as back then. It's not ideal to have a maximal version, but for now it's better than putting the minimum at 1.35. Regards, Arnout > --- > This patch should be coordinated with other patches [1] that update the > BR_FMT_VERSION revision for git or svn as all of the tarball hashes will > need to be regenerated tree-wide. > > [1] https://patchwork.ozlabs.org/project/buildroot/list/?series=373742 > --- > package/pkg-download.mk | 4 ++-- > support/dependencies/check-host-tar.sh | 10 ++++++---- > 2 files changed, 8 insertions(+), 6 deletions(-) > > diff --git a/package/pkg-download.mk b/package/pkg-download.mk > index e5cd83d859..d33eb2d811 100644 > --- a/package/pkg-download.mk > +++ b/package/pkg-download.mk > @@ -20,8 +20,8 @@ export LOCALFILES := $(call qstrip,$(BR2_LOCALFILES)) > > # Version of the format of the archives we generate in the corresponding > # download backend: > -BR_FMT_VERSION_git = -br1 > -BR_FMT_VERSION_svn = -br3 > +BR_FMT_VERSION_git = -br2 > +BR_FMT_VERSION_svn = -br4 > > DL_WRAPPER = support/download/dl-wrapper > > diff --git a/support/dependencies/check-host-tar.sh b/support/dependencies/check-host-tar.sh > index b7d607a47a..d56d0242a9 100755 > --- a/support/dependencies/check-host-tar.sh > +++ b/support/dependencies/check-host-tar.sh > @@ -27,11 +27,13 @@ if [ -n "${version_bsd}" ] ; then > exit 1 > fi > > -# Minimal version = 1.27 (previous versions do not correctly unpack archives > -# containing hard-links if the --strip-components option is used or create > -# different gnu long link headers for path elements > 100 characters). > +# Minimal version = 1.35 > +# GNU tar upstream commit 738de9ecd introduced a "breaking" change that > +# affects tarballs regardless of mode (pax/ustar/v7) such that tarballs > +# generated with 1.35 will not hash to the same value as those generated > +# by previous versions of tar. > major_min=1 > -minor_min=27 > +minor_min=35 > > if [ $major -lt $major_min ]; then > # echo nothing: no suitable tar found _______________________________________________ buildroot mailing list buildroot@buildroot.org https://lists.buildroot.org/mailman/listinfo/buildroot ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [Buildroot] [PATCH 1/1] support/dependencies: bump minimal tar version to 1.35 2023-11-08 21:01 ` Arnout Vandecappelle via buildroot @ 2023-11-09 15:18 ` Vincent Fazio 0 siblings, 0 replies; 7+ messages in thread From: Vincent Fazio @ 2023-11-09 15:18 UTC (permalink / raw) To: Arnout Vandecappelle; +Cc: Yann E . MORIN, buildroot [-- Attachment #1.1: Type: text/plain, Size: 7231 bytes --] All On Wed, Nov 8, 2023 at 3:01 PM Arnout Vandecappelle <arnout@mind.be> wrote: > Hi Vincent, > > On 18/10/2023 16:11, Vincent Fazio wrote: > > GNU tar 1.35 includes a breaking change [0] that changes the tar header > > created for each regular file even for "stable" formats like pax (which > > we use in support/download/helpers::mk_tar_gz). > > > > Previously, normal files had the devmajor/devminor fields of the header > > filled with ASCII zero (0x30). These fields are now null which also > > affects the checksum value in the header as well. > > > > $ diff <(head -n 100 xxd.1.34.output) <(head -n 100 xxd.1.35.output) > > 10c10 > > < 00000090: 3037 3500 3031 3531 3637 0020 3000 0000 075.015167. 0... > > --- > > > 00000090: 3037 3500 3031 3337 3237 0020 3000 0000 075.013727. 0... > > 21,22c21,22 > > < 00000140: 0000 0000 0000 0000 0030 3030 3030 3030 .........0000000 > > < 00000150: 0030 3030 3030 3030 0000 0000 0000 0000 .0000000........ > > --- > > > 00000140: 0000 0000 0000 0000 0000 0000 0000 0000 ................ > > > 00000150: 0000 0000 0000 0000 0000 0000 0000 0000 ................ > > > > This has the consequence of causing hash mismatches for any tarball > > created with a version prior to 1.35. > > > > Instead of reverting back to a host specific version (see 37a909cacf) > > and adding a cap to the maximal tar version (see ec50e407be), just > > establish that 1.35+ should be used going forward. > > > > This change requires an updated BR_FMT_VERSION for both git and svn. > > > > [0] > https://git.savannah.gnu.org/cgit/tar.git/commit/?id=738de9ecdec45ebfeb999628742373b5f8253bd0 > > > > Signed-off-by: Vincent Fazio <vfazio@gmail.com> > > We discussed with 3 more maintainers (Peter, Romain and myself) and came > to > the conclusion that it is really too annoying that almost everybody would > have > to build host-tar (unconditionally!). At the moment, of the main distros, > only > The host-tar package takes between 20-45 seconds to build. I don't see that as overly burdensome on a build that can potentially take upwards of half an hour. > Fedora 39 (just released) and Arch Linux already have 1.35. > > We're already in a situation where people are using the latest Arch and encountering this (a colleague of Thomas in fact). It's what drove the investigation in October and this patch. They were using an older buildroot, so there's not much that can be done there. It's not ideal, but we can document the limitation with distributions using tar 1.35 and recommend utilizing a build container that uses a previous version of tar. > The idea is to apply this "bump minimal tar" only when more major > distros have > switched to 1.35 or later. Or, perhaps with the additional time we find an > alternative that is hopefully more stable. I was thinking of switching to > sharutils, for example - last release is from 2015 so very stable! :-) > > Based on testing Yann did back in the day (cbe95b1a455bbefcaa90a08cf3dd1a590630921e) the --format=posix format has been stable since at least GNU tar 1.27 released 10 years ago. I did open an issue in GNU tar for this change and asked about the guarantee of stability: https://savannah.gnu.org/bugs/index.php?64875 Pending the outcome of that issue, we can maybe not do anything for now except recommend against using distros with tar 1.35 I personally think it's better to do the bump and account for the behavior change now. The build of host-tar 1.35 will eventually not be required as the new version is adopted by distros vs sticking with 1.34 and then slowly forcing more builds of older tar on newer distros. > So, for the time being, we can do the following: > > - in check-host-tar.sh, check that system tar < 1.35; > - in package/tar, add a patch that reverts 738de9ecde (probably need to > revert > c3663838 as well). > I don't agree with reverting these commits in 1.35 as it causes functionally different behavior between BR built tar and distribution built tar on the same version. The only way to guarantee previous/stable behavior would be to _always_ use BR built tar with reverts, which you didn't want to do above. It would also make the recipe a bit funky, where we're specifically applying a patch to 1.35 for the host because we rely on the behavior or earlier versions but probably don't want to do that for the target. > > > Regarding ec50e407be: we're indeed in a similar situation as back then. > It's > not ideal to have a maximal version, but for now it's better than putting > the > minimum at 1.35. > > We could set the host tar version to 1.34, but it sounded like there were concerns about security? The only CVE I see at a glance is CVE-2022-48303. Upstream https://git.savannah.gnu.org/cgit/tar.git/commit/?id=3da78400eafcccb97e2f2fd4b227ea40d794ede8 Oddly, Debian hasn't ported the patch but Ubuntu has. I obviously defer to the decision of the maintainers, but wanted to voice my thoughts. > > Regards, > Arnout > > > > --- > > This patch should be coordinated with other patches [1] that update the > > BR_FMT_VERSION revision for git or svn as all of the tarball hashes will > > need to be regenerated tree-wide. > > > > [1] https://patchwork.ozlabs.org/project/buildroot/list/?series=373742 > > --- > > package/pkg-download.mk | 4 ++-- > > support/dependencies/check-host-tar.sh | 10 ++++++---- > > 2 files changed, 8 insertions(+), 6 deletions(-) > > > > diff --git a/package/pkg-download.mk b/package/pkg-download.mk > > index e5cd83d859..d33eb2d811 100644 > > --- a/package/pkg-download.mk > > +++ b/package/pkg-download.mk > > @@ -20,8 +20,8 @@ export LOCALFILES := $(call qstrip,$(BR2_LOCALFILES)) > > > > # Version of the format of the archives we generate in the > corresponding > > # download backend: > > -BR_FMT_VERSION_git = -br1 > > -BR_FMT_VERSION_svn = -br3 > > +BR_FMT_VERSION_git = -br2 > > +BR_FMT_VERSION_svn = -br4 > > > > DL_WRAPPER = support/download/dl-wrapper > > > > diff --git a/support/dependencies/check-host-tar.sh > b/support/dependencies/check-host-tar.sh > > index b7d607a47a..d56d0242a9 100755 > > --- a/support/dependencies/check-host-tar.sh > > +++ b/support/dependencies/check-host-tar.sh > > @@ -27,11 +27,13 @@ if [ -n "${version_bsd}" ] ; then > > exit 1 > > fi > > > > -# Minimal version = 1.27 (previous versions do not correctly unpack > archives > > -# containing hard-links if the --strip-components option is used or > create > > -# different gnu long link headers for path elements > 100 characters). > > +# Minimal version = 1.35 > > +# GNU tar upstream commit 738de9ecd introduced a "breaking" change that > > +# affects tarballs regardless of mode (pax/ustar/v7) such that tarballs > > +# generated with 1.35 will not hash to the same value as those generated > > +# by previous versions of tar. > > major_min=1 > > -minor_min=27 > > +minor_min=35 > > > > if [ $major -lt $major_min ]; then > > # echo nothing: no suitable tar found > [-- Attachment #1.2: Type: text/html, Size: 10248 bytes --] [-- Attachment #2: Type: text/plain, Size: 150 bytes --] _______________________________________________ buildroot mailing list buildroot@buildroot.org https://lists.buildroot.org/mailman/listinfo/buildroot ^ permalink raw reply [flat|nested] 7+ messages in thread
end of thread, other threads:[~2023-11-09 15:19 UTC | newest] Thread overview: 7+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2023-10-18 14:11 [Buildroot] [PATCH 1/1] support/dependencies: bump minimal tar version to 1.35 Vincent Fazio 2023-10-18 17:43 ` Arnout Vandecappelle via buildroot 2023-10-21 20:25 ` Yann E. MORIN 2023-10-21 20:36 ` Yann E. MORIN 2023-10-23 7:45 ` Arnout Vandecappelle via buildroot 2023-11-08 21:01 ` Arnout Vandecappelle via buildroot 2023-11-09 15:18 ` Vincent Fazio
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.