[Buildroot] Host tar 1.32 build issues

[Buildroot] Host tar 1.32 build issues

[Markus Mayer]

Hi all,

I noticed that host tar is no longer locked to 1.29 and has been
bumped to 1.32. The only problem is that the host tar build requires
tar in CPIO format, and that 1.32 does not exist as CPIO archive on
the official server. As per http://mirror.sergal.org/gnu/tar/, tar
1.31 is the latest that has a CPIO archive. 1.32 and 1.33 do not.

Naturally, that currently causes the build to fail if
BR2_FORCE_HOST_BUILD is enabled.

I don't know if the absence of CPIO archives is temporary or
permanent. It might be better to use tar-latest.cpio.gz as the host
tar version to prevent this issue from happening again in the future.

-HOST_TAR_SOURCE = tar-$(TAR_VERSION).cpio.gz
+HOST_TAR_SOURCE = tar-latest.cpio.gz

I tried this out and, while it would be nice, it doesn't seem to be
possible to do it this way, because the directory inside the tar-ball
is the actual version number. It would try to use
"build/host-tar-last/tar-last/*" which doesn't exist, and the build
still fails.

The only way I have been able to get past the problem was to
reintroduce the host specific TAR variables that were removed and to
hard-code them to 1.31 (rather than 1.29 as it was before).

What's your take on this?

Thanks,
-Markus

[Buildroot] Host tar 1.32 build issues

[Arnout Vandecappelle]

On 12/01/2021 00:45, Markus Mayer via buildroot wrote:
> Hi all,
> 
> I noticed that host tar is no longer locked to 1.29 and has been
> bumped to 1.32. The only problem is that the host tar build requires
> tar in CPIO format, and that 1.32 does not exist as CPIO archive on
> the official server. As per http://mirror.sergal.org/gnu/tar/, tar
> 1.31 is the latest that has a CPIO archive. 1.32 and 1.33 do not.
> 
> Naturally, that currently causes the build to fail if
> BR2_FORCE_HOST_BUILD is enabled.

 Good catch!

> I don't know if the absence of CPIO archives is temporary or
> permanent.

 AFAIU it's permanent :-(

> It might be better to use tar-latest.cpio.gz as the host
> tar version to prevent this issue from happening again in the future.
> 
> -HOST_TAR_SOURCE = tar-$(TAR_VERSION).cpio.gz
> +HOST_TAR_SOURCE = tar-latest.cpio.gz
> 
> I tried this out and, while it would be nice, it doesn't seem to be
> possible to do it this way, because the directory inside the tar-ball
> is the actual version number. It would try to use
> "build/host-tar-last/tar-last/*" which doesn't exist, and the build
> still fails.

 It should be possible to overcome this problem with:

-        mv $(@D)/tar-$(HOST_TAR_VERSION)/* $(@D)
+        mv $(@D)/tar-*/* $(@D)

(untested, of course).

 And for good measure, remove $(@D)/tar-* before extraction.


 An alternative is to download a tarball and extract it with custom extraction
commands that don't use --strip-components or --exclude or $(TAR_OPTIONS). That
way, we can extract it with any POSIX-y tar version.


 Regards,
 Arnout

> The only way I have been able to get past the problem was to
> reintroduce the host specific TAR variables that were removed and to
> hard-code them to 1.31 (rather than 1.29 as it was before).
> 
> What's your take on this?
> 
> Thanks,
> -Markus
> _______________________________________________
> buildroot mailing list
> buildroot at busybox.net
> http://lists.busybox.net/mailman/listinfo/buildroot
> 

[Buildroot] Host tar 1.32 build issues

[Yann E. MORIN]

Markus, Arnout, All,

On 2021-01-12 09:39 +0100, Arnout Vandecappelle spake thusly:
> On 12/01/2021 00:45, Markus Mayer via buildroot wrote:
> > I noticed that host tar is no longer locked to 1.29 and has been
> > bumped to 1.32. The only problem is that the host tar build requires
> > tar in CPIO format, and that 1.32 does not exist as CPIO archive on
> > the official server. As per http://mirror.sergal.org/gnu/tar/, tar
> > 1.31 is the latest that has a CPIO archive. 1.32 and 1.33 do not.
> >?
> > Naturally, that currently causes the build to fail if
> > BR2_FORCE_HOST_BUILD is enabled.
>?
> > I don't know if the absence of CPIO archives is temporary or
> > permanent.
>  AFAIU it's permanent :-(

Not so sure. 1.33 has been released 2021-01-07, but the cpio archives for
it are indeed present, and dated today, 2021-01-12.

So, maybe we can just point that out to the tar maintainers, that they
forgot the cpio for 1.32.

> > It might be better to use tar-latest.cpio.gz as the host
> > tar version to prevent this issue from happening again in the future.
> >?
> > -HOST_TAR_SOURCE = tar-$(TAR_VERSION).cpio.gz
> > +HOST_TAR_SOURCE = tar-latest.cpio.gz

That is a big no-no from me: since this archive is always pointing to
the latest version, that means it will be updated next time they do a
new release.

Which will be problematic as the hash we'd have for it will no longer
match.

>  An alternative is to download a tarball and extract it with custom extraction
> commands that don't use --strip-components or --exclude or $(TAR_OPTIONS). That
> way, we can extract it with any POSIX-y tar version.

I think this is the only good solution long-term.

Also, as a quick workaround for this issue, we can also bump to the
newly relased 1.33. ;-)

And I have tested with the Makefile in the commit log: 1.33 generates
archives that are reproducible.

I'll look more in-depth tonight (GMT+1)...

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

[Buildroot] Host tar 1.32 build issues

[Arnout Vandecappelle]

On 12/01/2021 10:19, Yann E. MORIN wrote:
> Markus, Arnout, All,
> 
> On 2021-01-12 09:39 +0100, Arnout Vandecappelle spake thusly:
>> On 12/01/2021 00:45, Markus Mayer via buildroot wrote:
>>> I noticed that host tar is no longer locked to 1.29 and has been
>>> bumped to 1.32. The only problem is that the host tar build requires
>>> tar in CPIO format, and that 1.32 does not exist as CPIO archive on
>>> the official server. As per http://mirror.sergal.org/gnu/tar/, tar
>>> 1.31 is the latest that has a CPIO archive. 1.32 and 1.33 do not.
>>> ?
>>> Naturally, that currently causes the build to fail if
>>> BR2_FORCE_HOST_BUILD is enabled.
>> ?
>>> I don't know if the absence of CPIO archives is temporary or
>>> permanent.
>>  AFAIU it's permanent :-(
> 
> Not so sure. 1.33 has been released 2021-01-07, but the cpio archives for
> it are indeed present, and dated today, 2021-01-12.
> 
> So, maybe we can just point that out to the tar maintainers, that they
> forgot the cpio for 1.32.

 And as Yann told me on IRC, the cpio for 1.32 appeared today as well. So the
problem should be solved.

 The only thing missing now is that the .hash file still refers to 1.29.

 Regards,
 Arnout

> 
>>> It might be better to use tar-latest.cpio.gz as the host
>>> tar version to prevent this issue from happening again in the future.
>>> ?
>>> -HOST_TAR_SOURCE = tar-$(TAR_VERSION).cpio.gz
>>> +HOST_TAR_SOURCE = tar-latest.cpio.gz
> 
> That is a big no-no from me: since this archive is always pointing to
> the latest version, that means it will be updated next time they do a
> new release.
> 
> Which will be problematic as the hash we'd have for it will no longer
> match.
> 
>>  An alternative is to download a tarball and extract it with custom extraction
>> commands that don't use --strip-components or --exclude or $(TAR_OPTIONS). That
>> way, we can extract it with any POSIX-y tar version.
> 
> I think this is the only good solution long-term.
> 
> Also, as a quick workaround for this issue, we can also bump to the
> newly relased 1.33. ;-)
> 
> And I have tested with the Makefile in the commit log: 1.33 generates
> archives that are reproducible.
> 
> I'll look more in-depth tonight (GMT+1)...
> 
> Regards,
> Yann E. MORIN.
> 

[Buildroot] Host tar 1.32 build issues

[Markus Mayer]

On Tue, 12 Jan 2021 at 02:54, Arnout Vandecappelle <arnout@mind.be> wrote:
>
> On 12/01/2021 10:19, Yann E. MORIN wrote:
> > Markus, Arnout, All,
> >
> > Not so sure. 1.33 has been released 2021-01-07, but the cpio archives for
> > it are indeed present, and dated today, 2021-01-12.
> >
> > So, maybe we can just point that out to the tar maintainers, that they
> > forgot the cpio for 1.32.
>
>  And as Yann told me on IRC, the cpio for 1.32 appeared today as well. So the
> problem should be solved.
>
>  The only thing missing now is that the .hash file still refers to 1.29.

The two CPIO archives are on the GNU server now. I asked about it on
the GNU tar mailing list last night (my time), and Sergey was so kind
to upload the two missing CPIO archives.

https://lists.gnu.org/archive/html/bug-tar/2021-01/msg00006.html

So everything is in place now.

Regards,
-Markus

[Buildroot] Host tar 1.32 build issues

[Yann E. MORIN]

Markus, Arnout, All,

On 2021-01-12 10:23 -0800, Markus Mayer spake thusly:
> The two CPIO archives are on the GNU server now. I asked about it on
> the GNU tar mailing list last night (my time), and Sergey was so kind
> to upload the two missing CPIO archives.
> 
> https://lists.gnu.org/archive/html/bug-tar/2021-01/msg00006.html
> 
> So everything is in place now.

There was still the update to the hash file that was missing. I've now
fixed it.

Thank you for noticing the issue and reporting so promptly! :-)

Now, tar 1.33 is out, care to send an update? ;-]

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

[Buildroot] Host tar 1.32 build issues

[Markus Mayer]

On Tue, 12 Jan 2021 at 13:04, Yann E. MORIN <yann.morin.1998@free.fr> wrote:
>
> Markus, Arnout, All,
>
> On 2021-01-12 10:23 -0800, Markus Mayer spake thusly:
> > The two CPIO archives are on the GNU server now. I asked about it on
> > the GNU tar mailing list last night (my time), and Sergey was so kind
> > to upload the two missing CPIO archives.
> >
> > https://lists.gnu.org/archive/html/bug-tar/2021-01/msg00006.html
> >
> > So everything is in place now.
>
> There was still the update to the hash file that was missing. I've now
> fixed it.
>
> Thank you for noticing the issue and reporting so promptly! :-)

I have an automated nightly build going that notices such things on my
behalf. :-)

> Now, tar 1.33 is out, care to send an update? ;-]

I did create a patch for this, but just as I was getting ready to
submit it to the list, I noticed that a regression has been found in
tar 1.33. Looks like it may only affect OpenBSD. At least I wasn't
able to reproduce it on Linux.

https://lists.gnu.org/archive/html/bug-tar/2021-01/msg00012.html

Still, I am wondering if we do want to go ahead with 1.33 or if we
should hold off and see whether 1.34 is released in the not too
distant future.

Regards,
-Markus




> Regards,
> Yann E. MORIN.
>
> --
> .-----------------.--------------------.------------------.--------------------.
> |  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
> | +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
> | +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
> | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
> '------------------------------^-------^------------------^--------------------'

[Buildroot] Host tar 1.32 build issues

[Markus Mayer]

On Tue, 12 Jan 2021 at 15:26, Markus Mayer <mmayer@broadcom.com> wrote:
>
> On Tue, 12 Jan 2021 at 13:04, Yann E. MORIN <yann.morin.1998@free.fr> wrote:
> >
> > Now, tar 1.33 is out, care to send an update? ;-]
>
[...]
>
> Still, I am wondering if we do want to go ahead with 1.33 or if we
> should hold off and see whether 1.34 is released in the not too
> distant future.

tar 1.34 is now out. The patch to update the package is forthcoming.

Regards,
-Markus