Re: [Buildroot] [PATCH v2 05/12] support/download/go-post-process: implement Go vendoring support

From: "Yann E. MORIN" <yann.morin.1998@free.fr>
To: Christian Stewart <christian@paral.in>
Cc: Matt Weber <Matthew.Weber@rockwellcollins.com>,
	Patrick Havelange <patrick.havelange@essensium.com>,
	Anisse Astier <anisse@astier.eu>,
	Ryan Barnett <ryan.barnett@rockwellcollins.com>,
	Thomas Petazzoni <thomas.petazzoni@bootlin.com>,
	Buildroot List <buildroot@buildroot.org>
Subject: Re: [Buildroot] [PATCH v2 05/12] support/download/go-post-process: implement Go vendoring support
Date: Sun, 1 Aug 2021 09:08:30 +0200	[thread overview]
Message-ID: <20210801070830.GP3189549@scaer> (raw)
In-Reply-To: <CA+h8R2onPMjOuvC0U6iM8QbhuAQQ9=aQ-yB-rWQkCbhpxcdiHw@mail.gmail.com>

Christian, All,

On 2021-07-31 21:59 -0700, Christian Stewart spake thusly:
> Vincent said:
> >I think Yann already calls out what may be going on in this patch
> >https://patchwork.ozlabs.org/project/buildroot/patch/20201219153525.1361175-5-thomas.petazzoni@bootlin.com/
> 
> I think this is probably correct, the .tar.gz needs to have
> deterministic formatting.

Can you try usign the tar helper in support/download/helpers instead?

> In this particular case the user was trying to build on Kali 2.
> 
> Doesn't buildroot compile host-gzip and host-tar for this? I'm
> wondering why it's not the same between host machines.

Since 2020.02, we are no longer building host-tar, unless for very old
versions (i.e. before 1.27), thanks to some tar trickery vy Vincent.
See:

    cbe95b1a455b support/download: add helper to generate a reproducible archive

We also only build host-gzip if the host gzip is pigz (or missing).

> On Thu, Jul 29, 2021 at 1:50 PM Thomas Petazzoni
> <thomas.petazzoni@bootlin.com> wrote:
> 
> > > This works fine in most cases, however, this causes inconsistent hashes.
> > Based on the difficulties, I would really like to understand a little
> > bit why the archive is not deterministic. Could you compare the
> > contents of the download stuff between your different machines, and the
> > contents of the tarball, and see what changes?
> The two versions are attached. The actual contents of the two are
> identical. But for some reason the formatting of
> embiggen-disk-bad.tar.gz is different:
> 
>   00000030: 8232 63de 4282 92dd acff fb79 6740 eae2
> 
> Starting at "2dd acff" - in "embiggen-disk-good" it's
> 
> 00000030: 8232 63de 4282 96dd acff fb79 6740 eae6

So, I zcat both archives, and the two tar are identical!

    $ sha1sum *.tar
    dbb8338ab4acaff1e232c67d3b46602d61114b53  embiggen-disk-bad.tar
    dbb8338ab4acaff1e232c67d3b46602d61114b53  embiggen-disk-good.tar

So I also hashed the archives you provided, and they too are identical:

    $ sha1sum *.tar.gz
    bd887a60c4ca60e55d062067132ccab7d85e7d95  embiggen-disk-bad.tar.gz
    bd887a60c4ca60e55d062067132ccab7d85e7d95  embiggen-disk-good.tar.gz

Did you mess up when sending the archives?

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
_______________________________________________
buildroot mailing list
buildroot@busybox.net
http://lists.busybox.net/mailman/listinfo/buildroot