* [Buildroot] linux-firmware hash mismatch (tar-1.30) @ 2018-01-09 16:15 John Keeping 2018-01-10 19:15 ` Peter Korsgaard 0 siblings, 1 reply; 7+ messages in thread From: John Keeping @ 2018-01-09 16:15 UTC (permalink / raw) To: buildroot Hi, I'm getting the following error cloning linux-firmware: ERROR: linux-firmware-17e6288135d4500f9fe60224dce2b46d850c346b.tar.gz has wrong sha256 hash: ERROR: expected: 28d359523a36c1cdc3e85a8e148bb2d68b036d28b10f0e80a192f3dc29f02c16 ERROR: got : bf6fe8d7620949a3e771954cb6d9d18dcf000d37ecc910a7cf69723c1798e246 ERROR: Incomplete download, or man-in-the-middle (MITM) attack After a bit of digging, it looks like this is caused by tar-1.30 which includes the following fix: * --numeric-owner now affects private headers too. Comparing the tarball created on my system with one from sources.buildroot.net gives the output like this, which shows that the uname/gname fields are set for a GNUTYPE_LONGNAME record in the version from sources.buildroot.net and not set in the version created with tar-1.30: -- >8 -- @@ -669379,24 +669379,24 @@ 00a36c20: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00a36c30: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00a36c40: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00a36c50: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00a36c60: 0000 0000 3030 3030 3634 3400 3030 3030 ....0000644.0000 00a36c70: 3030 3000 3030 3030 3030 3000 3030 3030 000.0000000.0000 00a36c80: 3030 3030 3135 3600 3030 3030 3030 3030 0000156.00000000 -00a36c90: 3030 3000 3031 3136 3035 0020 4c00 0000 000.011605. L... +00a36c90: 3030 3000 3030 3737 3735 0020 4c00 0000 000.007775. L... 00a36ca0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00a36cb0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00a36cc0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00a36cd0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00a36ce0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00a36cf0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ -00a36d00: 0075 7374 6172 2020 0072 6f6f 7400 0000 .ustar .root... +00a36d00: 0075 7374 6172 2020 0000 0000 0000 0000 .ustar ........ 00a36d10: 0000 0000 0000 0000 0000 0000 0000 0000 ................ -00a36d20: 0000 0000 0000 0000 0072 6f6f 7400 0000 .........root... +00a36d20: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00a36d30: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00a36d40: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00a36d50: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00a36d60: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00a36d70: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00a36d80: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00a36d90: 0000 0000 0000 0000 0000 0000 0000 0000 ................ -- 8< -- I'm not sure whether anything can be done to avoid this problem, but hopefully reporting it will save the next person some debugging time. Regards, John ^ permalink raw reply [flat|nested] 7+ messages in thread
* [Buildroot] linux-firmware hash mismatch (tar-1.30) 2018-01-09 16:15 [Buildroot] linux-firmware hash mismatch (tar-1.30) John Keeping @ 2018-01-10 19:15 ` Peter Korsgaard 2018-01-11 10:55 ` John Keeping 0 siblings, 1 reply; 7+ messages in thread From: Peter Korsgaard @ 2018-01-10 19:15 UTC (permalink / raw) To: buildroot >>>>> "John" == John Keeping <john@metanate.com> writes: > Hi, > I'm getting the following error cloning linux-firmware: > ERROR: linux-firmware-17e6288135d4500f9fe60224dce2b46d850c346b.tar.gz has wrong sha256 hash: > ERROR: expected: 28d359523a36c1cdc3e85a8e148bb2d68b036d28b10f0e80a192f3dc29f02c16 > ERROR: got : bf6fe8d7620949a3e771954cb6d9d18dcf000d37ecc910a7cf69723c1798e246 > ERROR: Incomplete download, or man-in-the-middle (MITM) attack > After a bit of digging, it looks like this is caused by tar-1.30 which > includes the following fix: > * --numeric-owner now affects private headers too. > Comparing the tarball created on my system with one from > sources.buildroot.net gives the output like this, which shows that the > uname/gname fields are set for a GNUTYPE_LONGNAME record in the version > from sources.buildroot.net and not set in the version created with > tar-1.30: Gaah, what a mess :/ > I'm not sure whether anything can be done to avoid this problem, but > hopefully reporting it will save the next person some debugging time. I also don't quite see any good solutions either :/ Did you try bringing it up with tar upstream? Perhaps there is a way to disable this? Alternatively we can build tar-1.29 for the host and use that instead of whichever tar version is available on the build machine, but this will slow down the build. -- Bye, Peter Korsgaard ^ permalink raw reply [flat|nested] 7+ messages in thread
* [Buildroot] linux-firmware hash mismatch (tar-1.30) 2018-01-10 19:15 ` Peter Korsgaard @ 2018-01-11 10:55 ` John Keeping 2018-01-14 13:30 ` Yann E. MORIN 0 siblings, 1 reply; 7+ messages in thread From: John Keeping @ 2018-01-11 10:55 UTC (permalink / raw) To: buildroot On Wed, 10 Jan 2018 20:15:38 +0100, Peter Korsgaard wrote: > >>>>> "John" == John Keeping <john@metanate.com> writes: > > > Hi, > > I'm getting the following error cloning linux-firmware: > > > ERROR: linux-firmware-17e6288135d4500f9fe60224dce2b46d850c346b.tar.gz has wrong sha256 hash: > > ERROR: expected: 28d359523a36c1cdc3e85a8e148bb2d68b036d28b10f0e80a192f3dc29f02c16 > > ERROR: got : bf6fe8d7620949a3e771954cb6d9d18dcf000d37ecc910a7cf69723c1798e246 > > ERROR: Incomplete download, or man-in-the-middle (MITM) attack > > > > After a bit of digging, it looks like this is caused by tar-1.30 which > > includes the following fix: > > > * --numeric-owner now affects private headers too. > > > Comparing the tarball created on my system with one from > > sources.buildroot.net gives the output like this, which shows that the > > uname/gname fields are set for a GNUTYPE_LONGNAME record in the version > > from sources.buildroot.net and not set in the version created with > > tar-1.30: > > Gaah, what a mess :/ > > > I'm not sure whether anything can be done to avoid this problem, but > > hopefully reporting it will save the next person some debugging time. > > I also don't quite see any good solutions either :/ > > Did you try bringing it up with tar upstream? Perhaps there is a way to > disable this? Alternatively we can build tar-1.29 for the host and use > that instead of whichever tar version is available on the build machine, > but this will slow down the build. No, I didn't bring it up with tar upstream; I'm not sure there's much point since it is a clear bug fix to the --numeric-owner option. I don't think it's possible to reproducibly create bit-identical archives without using the same software version to produce the archive. ^ permalink raw reply [flat|nested] 7+ messages in thread
* [Buildroot] linux-firmware hash mismatch (tar-1.30) 2018-01-11 10:55 ` John Keeping @ 2018-01-14 13:30 ` Yann E. MORIN 2018-01-17 23:28 ` Arnout Vandecappelle 0 siblings, 1 reply; 7+ messages in thread From: Yann E. MORIN @ 2018-01-14 13:30 UTC (permalink / raw) To: buildroot John, Peter, All, (adding Thomas and Arnout) On 2018-01-11 10:55 +0000, John Keeping spake thusly: > On Wed, 10 Jan 2018 20:15:38 +0100, Peter Korsgaard wrote: > > >>>>> "John" == John Keeping <john@metanate.com> writes: > > > ERROR: linux-firmware-17e6288135d4500f9fe60224dce2b46d850c346b.tar.gz has wrong sha256 hash: > > > ERROR: expected: 28d359523a36c1cdc3e85a8e148bb2d68b036d28b10f0e80a192f3dc29f02c16 > > > ERROR: got : bf6fe8d7620949a3e771954cb6d9d18dcf000d37ecc910a7cf69723c1798e246 > > > ERROR: Incomplete download, or man-in-the-middle (MITM) attack > > > After a bit of digging, it looks like this is caused by tar-1.30 which > > > includes the following fix: > > > * --numeric-owner now affects private headers too. > > Gaah, what a mess :/ And that's not all. Between 1.26 and 1.28 [*] the header for "long links" or "long names" has changed. Previously, the "mode" field was al set to zero, but now it is set to some non-zero value (not sure which, though...) > > > I'm not sure whether anything can be done to avoid this problem, but > > > hopefully reporting it will save the next person some debugging time. > > > > I also don't quite see any good solutions either :/ > > > > Did you try bringing it up with tar upstream? Perhaps there is a way to > > disable this? Alternatively we can build tar-1.29 for the host and use > > that instead of whichever tar version is available on the build machine, > > but this will slow down the build. > > No, I didn't bring it up with tar upstream; I'm not sure there's much > point since it is a clear bug fix to the --numeric-owner option. > > I don't think it's possible to reproducibly create bit-identical > archives without using the same software version to produce the archive. Alas, this means that we can only depend on building our own tar... Even when we eventually support using a local git-clone cache, this we not solve the issue has the hashes we store are on the generated tarball... Regards, Yann E. MORIN. -- .-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 223 225 172 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------' ^ permalink raw reply [flat|nested] 7+ messages in thread
* [Buildroot] linux-firmware hash mismatch (tar-1.30) 2018-01-14 13:30 ` Yann E. MORIN @ 2018-01-17 23:28 ` Arnout Vandecappelle 2018-01-18 21:40 ` Peter Korsgaard 0 siblings, 1 reply; 7+ messages in thread From: Arnout Vandecappelle @ 2018-01-17 23:28 UTC (permalink / raw) To: buildroot On 14-01-18 14:30, Yann E. MORIN wrote: > John, Peter, All, > > (adding Thomas and Arnout) > > On 2018-01-11 10:55 +0000, John Keeping spake thusly: >> On Wed, 10 Jan 2018 20:15:38 +0100, Peter Korsgaard wrote: >>>>>>>> "John" == John Keeping <john@metanate.com> writes: >>> > ERROR: linux-firmware-17e6288135d4500f9fe60224dce2b46d850c346b.tar.gz has wrong sha256 hash: >>> > ERROR: expected: 28d359523a36c1cdc3e85a8e148bb2d68b036d28b10f0e80a192f3dc29f02c16 >>> > ERROR: got : bf6fe8d7620949a3e771954cb6d9d18dcf000d37ecc910a7cf69723c1798e246 >>> > ERROR: Incomplete download, or man-in-the-middle (MITM) attack >>> > After a bit of digging, it looks like this is caused by tar-1.30 which >>> > includes the following fix: >>> > * --numeric-owner now affects private headers too. >>> Gaah, what a mess :/ Note that this also means our github hashes will break again... [snip] >> I don't think it's possible to reproducibly create bit-identical >> archives without using the same software version to produce the archive. > > Alas, this means that we can only depend on building our own tar... Nah, let's not go there. > Even when we eventually support using a local git-clone cache, this we > not solve the issue has the hashes we store are on the generated > tarball... However, the way we create a git tarball can serve as a source of inspiration of how to solve it. Instead of hashing the tarball, we can hash the contents of the tarball instead, using --to-command and a support script (--to-command exists since 1.15.90 and our minimal tar version is 1.17). The support script would print the metadata and a hash of the contents. And the output of all that is piped into sort (to make sure the order of files in the tarball isn't relevant) and shasum. That way we don't rely on the way the user's tar behaves at all. We really check the contents of the tarball. It would also mean we can drop the complexity in the git download helper and directly use git archive. It does create additional overhead, since the tarball will be uncompressed twice: once for calculating the hash, and again when extracting. But normally calculating a shasum should take significantly longer than decompressing, so I don't expect this to have a big impact. Of course, introducing such a feature would break all existing hashes in one fell swoop... So to avoid that, I would introduce a new hash type, sha256-tar, that does this trickery. One small additional thing: we'll probably have to pass the result of suitable-extractor to the download helper and down check-hash; alternatively, we have to duplicate that logic, or move it to a script that is called from check-hash. Regards, Arnout -- Arnout Vandecappelle arnout at mind be Senior Embedded Software Architect +32-16-286500 Essensium/Mind http://www.mind.be G.Geenslaan 9, 3001 Leuven, Belgium BE 872 984 063 RPR Leuven LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle GPG fingerprint: 7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF ^ permalink raw reply [flat|nested] 7+ messages in thread
* [Buildroot] linux-firmware hash mismatch (tar-1.30) 2018-01-17 23:28 ` Arnout Vandecappelle @ 2018-01-18 21:40 ` Peter Korsgaard 2018-01-22 22:14 ` Arnout Vandecappelle 0 siblings, 1 reply; 7+ messages in thread From: Peter Korsgaard @ 2018-01-18 21:40 UTC (permalink / raw) To: buildroot >>>>> "Arnout" == Arnout Vandecappelle <arnout@mind.be> writes: Hi, >>>> Gaah, what a mess :/ > Note that this also means our github hashes will break again... Indeed, I also mentioned that earlier. >> Alas, this means that we can only depend on building our own tar... > Nah, let's not go there. >> Even when we eventually support using a local git-clone cache, this we >> not solve the issue has the hashes we store are on the generated >> tarball... > However, the way we create a git tarball can serve as a source of inspiration > of how to solve it. Instead of hashing the tarball, we can hash the contents of > the tarball instead, using --to-command and a support script (--to-command > exists since 1.15.90 and our minimal tar version is 1.17). The support script > would print the metadata and a hash of the contents. And the output of all that > is piped into sort (to make sure the order of files in the tarball isn't > relevant) and shasum. Hmm, that is perhaps an option. It does make it somewhat more cumbersome to calculate the expected hash by hand, but maybe that is ok. Alternatively we don't do anything and instead ensure that tarballs on s.b.o matches the hashes we commit, and people using incompatible tar versions will fall back to the tarballs on s.b.o. > That way we don't rely on the way the user's tar behaves at all. We really > check the contents of the tarball. It would also mean we can drop the complexity > in the git download helper and directly use git archive. I thought the main reason for our custom tarball handling was support for git submodules? I take it that git-archive still doesn't handle those? -- Bye, Peter Korsgaard ^ permalink raw reply [flat|nested] 7+ messages in thread
* [Buildroot] linux-firmware hash mismatch (tar-1.30) 2018-01-18 21:40 ` Peter Korsgaard @ 2018-01-22 22:14 ` Arnout Vandecappelle 0 siblings, 0 replies; 7+ messages in thread From: Arnout Vandecappelle @ 2018-01-22 22:14 UTC (permalink / raw) To: buildroot On 18-01-18 22:40, Peter Korsgaard wrote: >>>>>> "Arnout" == Arnout Vandecappelle <arnout@mind.be> writes: > > Hi, > > >>>> Gaah, what a mess :/ > > > Note that this also means our github hashes will break again... > > Indeed, I also mentioned that earlier. > > >> Alas, this means that we can only depend on building our own tar... > > > Nah, let's not go there. > > >> Even when we eventually support using a local git-clone cache, this we > >> not solve the issue has the hashes we store are on the generated > >> tarball... > > > However, the way we create a git tarball can serve as a source of inspiration > > of how to solve it. Instead of hashing the tarball, we can hash the contents of > > the tarball instead, using --to-command and a support script (--to-command > > exists since 1.15.90 and our minimal tar version is 1.17). The support script > > would print the metadata and a hash of the contents. And the output of all that > > is piped into sort (to make sure the order of files in the tarball isn't > > relevant) and shasum. > > Hmm, that is perhaps an option. It does make it somewhat more cumbersome > to calculate the expected hash by hand, but maybe that is ok. There's still a patch floating around that doesn't remove the downloaded file [1], and the calculated hash is already reported. So not much of an issue I think. > Alternatively we don't do anything and instead ensure that tarballs on > s.b.o matches the hashes we commit, and people using incompatible tar > versions will fall back to the tarballs on s.b.o. But then they'll still be downloaded twice. Not very nice. Let's discuss it at the meeting in two weeks. > > That way we don't rely on the way the user's tar behaves at all. We really > > check the contents of the tarball. It would also mean we can drop the complexity > > in the git download helper and directly use git archive. > > I thought the main reason for our custom tarball handling was support > for git submodules? I take it that git-archive still doesn't handle those? Right, I forgot about that. Regards, Arnout [1] http://patchwork.ozlabs.org/patch/834425/ -- Arnout Vandecappelle arnout at mind be Senior Embedded Software Architect +32-16-286500 Essensium/Mind http://www.mind.be G.Geenslaan 9, 3001 Leuven, Belgium BE 872 984 063 RPR Leuven LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle GPG fingerprint: 7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF ^ permalink raw reply [flat|nested] 7+ messages in thread
end of thread, other threads:[~2018-01-22 22:14 UTC | newest] Thread overview: 7+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2018-01-09 16:15 [Buildroot] linux-firmware hash mismatch (tar-1.30) John Keeping 2018-01-10 19:15 ` Peter Korsgaard 2018-01-11 10:55 ` John Keeping 2018-01-14 13:30 ` Yann E. MORIN 2018-01-17 23:28 ` Arnout Vandecappelle 2018-01-18 21:40 ` Peter Korsgaard 2018-01-22 22:14 ` Arnout Vandecappelle
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox