[Buildroot] Hash verification from GitHub

[Buildroot] Hash verification from GitHub

[Danilo Bargen]

Hello folks

I'm trying to create a new buildroot package (my first one). This is
what the makefile (tealdeer.mk) looks like:

  TEALDEER_VERSION = 1.5.0
  TEALDEER_SITE = $(call github,dbrgn,tealdeer,v$(TEALDEER_VERSION))
  TEALDEER_LICENSE = Apache-2.0 or MIT
  TEALDEER_LICENSE_FILES = LICENSE-APACHE LICENSE-MIT
  $(eval $(cargo-package))

The URL should expand to
https://github.com/dbrgn/tealdeer/archive/v1.5.0/tealdeer-1.5.0.tar.gz.
To generate the checksum, I ran:

  $ sha256sum tealdeer-1.5.0.tar.gz
  00902a50373ab75fedec4578c6c2c02523fad435486918ad9a86ed01f804358a  tealdeer-1.5.0.tar.gz

I also added a hash file (tealdeer.hash):

  # Locally generated
  sha256  00902a50373ab75fedec4578c6c2c02523fad435486918ad9a86ed01f804358a  tealdeer-1.5.0.tar.gz
  sha256  62c7a1e35f56406896d7aa7ca52d0cc0d272ac022b5d2796e7d6905db8a3636a  LICENSE-APACHE
  sha256  a313b5e62b80a08f3aae0fa62ff3de8482ef55247299eb352ab44f87ef456b1b  LICENSE-MIT

When building this package, checksum verification fails every time.

  ERROR: tealdeer-1.5.0.tar.gz has wrong sha256 hash:
  ERROR: expected: 00902a50373ab75fedec4578c6c2c02523fad435486918ad9a86ed01f804358a
  ERROR: got     : 42febf9ee84721b9230077d62e2fc51201fd59624d3c776ccc1a634788768a60
  ERROR: Incomplete download, or man-in-the-middle (MITM) attack

No matter how I download the file (via wget, through the GitHub web UI,
etc), it always results in the SHA256 checksum starting with 009...,
but buildroot always thinks it should be 42f... I also tried changing
the TEALDEER_SITE variable as follows:

  TEALDEER_SITE = https://github.com/dbrgn/tealdeer/archive/v$(TEALDEER_VERSION)

...to ensure that this URL is *really* being downloaded, but it fails
every time.

Full build log can be found here:
https://gist.github.com/dbrgn/cc9e96051a079f5b63c531ca3c195954

Does someone have any pointers why the hash verification would fail in
this case? It must be some obvious mistake I'm making, but I cannot
figure out what it is.

Best regards,
Danilo Bargen
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] Hash verification from GitHub

[James Hilliard]

On Sun, Jan 16, 2022 at 3:04 PM Danilo Bargen <mail@dbrgn.ch> wrote:
>
> Hello folks
>
> I'm trying to create a new buildroot package (my first one). This is
> what the makefile (tealdeer.mk) looks like:
>
>   TEALDEER_VERSION = 1.5.0
>   TEALDEER_SITE = $(call github,dbrgn,tealdeer,v$(TEALDEER_VERSION))
>   TEALDEER_LICENSE = Apache-2.0 or MIT
>   TEALDEER_LICENSE_FILES = LICENSE-APACHE LICENSE-MIT
>   $(eval $(cargo-package))
>
> The URL should expand to
> https://github.com/dbrgn/tealdeer/archive/v1.5.0/tealdeer-1.5.0.tar.gz.
> To generate the checksum, I ran:
>
>   $ sha256sum tealdeer-1.5.0.tar.gz
>   00902a50373ab75fedec4578c6c2c02523fad435486918ad9a86ed01f804358a  tealdeer-1.5.0.tar.gz

Just use the hash buildroot generated for you, it should be correct.

>
> I also added a hash file (tealdeer.hash):
>
>   # Locally generated
>   sha256  00902a50373ab75fedec4578c6c2c02523fad435486918ad9a86ed01f804358a  tealdeer-1.5.0.tar.gz
>   sha256  62c7a1e35f56406896d7aa7ca52d0cc0d272ac022b5d2796e7d6905db8a3636a  LICENSE-APACHE
>   sha256  a313b5e62b80a08f3aae0fa62ff3de8482ef55247299eb352ab44f87ef456b1b  LICENSE-MIT
>
> When building this package, checksum verification fails every time.
>
>   ERROR: tealdeer-1.5.0.tar.gz has wrong sha256 hash:
>   ERROR: expected: 00902a50373ab75fedec4578c6c2c02523fad435486918ad9a86ed01f804358a
>   ERROR: got     : 42febf9ee84721b9230077d62e2fc51201fd59624d3c776ccc1a634788768a60

Use this hash.

>   ERROR: Incomplete download, or man-in-the-middle (MITM) attack
>
> No matter how I download the file (via wget, through the GitHub web UI,
> etc), it always results in the SHA256 checksum starting with 009...,
> but buildroot always thinks it should be 42f... I also tried changing
> the TEALDEER_SITE variable as follows:

Yeah, it's not going to match since buildroot is doing stuff like
vendoring the cargo deps
before generating the tarball(to ensure that the package can be built
from the tarball without
having to download anything additional from cargo):
https://github.com/buildroot/buildroot/blob/master/support/download/cargo-post-process

>
>   TEALDEER_SITE = https://github.com/dbrgn/tealdeer/archive/v$(TEALDEER_VERSION)
>
> ...to ensure that this URL is *really* being downloaded, but it fails
> every time.
>
> Full build log can be found here:
> https://gist.github.com/dbrgn/cc9e96051a079f5b63c531ca3c195954

You can see it vendoring the deps in the log here:
https://gist.github.com/dbrgn/cc9e96051a079f5b63c531ca3c195954#file-gistfile1-txt-L23-L200

>
> Does someone have any pointers why the hash verification would fail in
> this case? It must be some obvious mistake I'm making, but I cannot
> figure out what it is.
>
> Best regards,
> Danilo Bargen
> _______________________________________________
> buildroot mailing list
> buildroot@buildroot.org
> https://lists.buildroot.org/mailman/listinfo/buildroot
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] Hash verification from GitHub

[Danilo Bargen]

Hello James

> Just use the hash buildroot generated for you, it should be correct.
> (...)
> Yeah, it's not going to match since buildroot is doing stuff like
> vendoring the cargo deps before generating the tarball

Ahh, that explains it. Does this mean that for Cargo based packages, it
will always be TOFU and checksums provided by the project / maintainer
cannot be used (because they will mismatch)?

Danilo
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] Hash verification from GitHub

[James Hilliard]

On Sun, Jan 16, 2022 at 3:38 PM Danilo Bargen <mail@dbrgn.ch> wrote:
>
> Hello James
>
> > Just use the hash buildroot generated for you, it should be correct.
> > (...)
> > Yeah, it's not going to match since buildroot is doing stuff like
> > vendoring the cargo deps before generating the tarball
>
> Ahh, that explains it. Does this mean that for Cargo based packages, it
> will always be TOFU and checksums provided by the project / maintainer
> cannot be used (because they will mismatch)?

A maintainer could sign the tarball generated by buildroot and upload it to
github releases if they want.

As you can see the script will bail out if the tarball already has
vendored deps:
https://github.com/buildroot/buildroot/blob/6e4791b751c0f8e0ba218da2c22e71d3e1436b5d/support/download/cargo-post-process#L16-L19

So as long as the upstream release tarball has vendored deps the hash should
not change AFAIU.

>
> Danilo
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] Hash verification from GitHub

[Yann E. MORIN]

James, Danilo, All,

On 2022-01-16 15:51 -0700, James Hilliard spake thusly:
> On Sun, Jan 16, 2022 at 3:38 PM Danilo Bargen <mail@dbrgn.ch> wrote:
> > > Just use the hash buildroot generated for you, it should be correct.
> > > (...)
> > > Yeah, it's not going to match since buildroot is doing stuff like
> > > vendoring the cargo deps before generating the tarball
> > Ahh, that explains it. Does this mean that for Cargo based packages, it
> > will always be TOFU and checksums provided by the project / maintainer
> > cannot be used (because they will mismatch)?
> A maintainer could sign the tarball generated by buildroot and upload it to
> github releases if they want.

To be exact: an upstream maintainer (i.e. not a Buildroot maintainer).

> As you can see the script will bail out if the tarball already has
> vendored deps:
> https://github.com/buildroot/buildroot/blob/6e4791b751c0f8e0ba218da2c22e71d3e1436b5d/support/download/cargo-post-process#L16-L19

The script will not "bail out" on pre-vendored tarballs; it will just
exit without touching the tarball at all.

I.e. it means that we prefer using tarballs as-is from their upstreams,
when they are vendored; we only vendor packages which upstreams have
not.

> So as long as the upstream release tarball has vendored deps the hash should
> not change AFAIU.

Yes, this is the goal.

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] Hash verification from GitHub

[Danilo Bargen]

Hello James, Yann

On 1/17/22 11:17, Yann E. MORIN wrote:
>> A maintainer could sign the tarball generated by buildroot and upload it to
>> github releases if they want.
> 
> To be exact: an upstream maintainer (i.e. not a Buildroot maintainer).
> 
> (...)
> 
> I.e. it means that we prefer using tarballs as-is from their upstreams,
> when they are vendored; we only vendor packages which upstreams have
> not.
That makes sense! I am the maintainer of tealdeer, and I'll provide a 
vendored source tarball for the next release. (I've heard of "cargo 
vendor" before, but I haven't used it so far.)

Maybe a corresponding paragraph could be added to the docs at 
https://nightly.buildroot.org/#_integration_of_cargo_based_packages ?

Thanks for the helpful comments,
Danilo
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] Hash verification from GitHub

[Yann E. MORIN]

Danilo, All,

On 2022-01-17 11:24 +0100, Danilo Bargen spake thusly:
> On 1/17/22 11:17, Yann E. MORIN wrote:
> >>A maintainer could sign the tarball generated by buildroot and upload it to
> >>github releases if they want.
> >
> >To be exact: an upstream maintainer (i.e. not a Buildroot maintainer).
> >
> >(...)
> >
> >I.e. it means that we prefer using tarballs as-is from their upstreams,
> >when they are vendored; we only vendor packages which upstreams have
> >not.
> That makes sense! I am the maintainer of tealdeer, and I'll provide a
> vendored source tarball for the next release. (I've heard of "cargo vendor"
> before, but I haven't used it so far.)
> 
> Maybe a corresponding paragraph could be added to the docs at
> https://nightly.buildroot.org/#_integration_of_cargo_based_packages ?

Yeah, so I already noticed that the nightly manual is not up to date...

It indeed points to the correct sha1 in master, but the Cargo part is
not what it should be; it is still the older text from before we added
the cargo infra. We're looking into the issue...

You can hack on the manual too, it's in:    docs/manual/
And you can generate the manual locally:    make manual-html

Patches welcome! ;-)

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] Hash verification from GitHub

[Yann E. MORIN]

Danilo, All,

On 2022-01-17 11:32 +0100, Yann E. MORIN spake thusly:
> On 2022-01-17 11:24 +0100, Danilo Bargen spake thusly:
[--SNIP--]
> > Maybe a corresponding paragraph could be added to the docs at
> > https://nightly.buildroot.org/#_integration_of_cargo_based_packages ?
> Yeah, so I already noticed that the nightly manual is not up to date...

It appears that the local tree that serves to generate the manual was
somehow corrupted, and we were only ever building the same old manual
over and over again...

It has now presumably been resolved, and [0] now reads:

    A crate can depend on other libraries from crates.io or git
    repositories, listed in its Cargo.toml file. Buildroot automatically
    takes care of downloading such dependencies as part of the download
    step of packages that use the cargo-package infrastructure. Such
    dependencies are then kept together with the package source code in
    the tarball cached in Buildroot’s DL_DIR, and therefore the hash of
    the package’s tarball includes such dependencies.

[0] https://nightly.buildroot.org/#_infrastructure_for_cargo_based_packages

Maybe it is not so obvious and prominently advertised, indeed; patches
to improve the situation are most welcome! :-)

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] Hash verification from GitHub

[Yann E. MORIN]

Danilo, All,

On 2022-01-17 11:24 +0100, Danilo Bargen spake thusly:
> On 1/17/22 11:17, Yann E. MORIN wrote:
> >I.e. it means that we prefer using tarballs as-is from their upstreams,
> >when they are vendored; we only vendor packages which upstreams have
> >not.
> That makes sense! I am the maintainer of tealdeer, and I'll provide a
> vendored source tarball for the next release. (I've heard of "cargo vendor"
> before, but I haven't used it so far.)

Note that vendoring is definitely not a requirement we impose on
upstreams.

We do prefer when the vendoring has been done by upstream, because it
means (at least we hope it does!) that upstream has validated the fully
vendored package, and thus we have some confidence everything works as
expected.

It also avoids the case where an uppstreams for a dependency mucks
around with their releases: we already noticed the case where an
upstream for a dependency did a re-tag of their release, thus breaking
the vendoring of the dependees because it would no longer match the
expected hashes in the cargo.toml (or go.mod?). Also (but that's mostly
for go, IIRC), we also already noticed that some of the points of
distribution (goproxies?) are serving some incorrect archives, thus
causing download issues...

However, if an upstream decides to not vendor (for whatever reason),
then this is perfectly fine for Buildroot; this is exactly why the cargo
infra has been made to support doing the vendoring.

All in all, it is better that upstream vendors their releases, because
it avoids any of the pitfals I mention above.

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot