[Buildroot] [PATCH 0/3 v2] support/download: accept user-provided list of extra hash files (branch yem/multi-hash)

[Buildroot] [PATCH 0/3 v2] support/download: accept user-provided list of extra hash files (branch yem/multi-hash)

[Yann E. MORIN]

Hello All!

For packages where the version can be specified by the user (e.g. with a
custom version string, a custom tarball location, a custom git tree and
changeset...), Buildroot can't carry hashes for those, and thus does not
check the integritiy of the downloads.

Add the possibility for users to provide hashes for those, by leveraging
the global patch dir setting, to look up extra hash files in a way
similar to how extra patches are looked up in there.

Users who provide such extra hashes will most probably be interested in
ensuring that no download ever gets used without an actual integrity
check, so also add an option the requires all downloads to have at least
one valid hash (and no invalid ones, of course), rather than ignoring
downloads for custom versions.

Changes v1 -> v2:
  - restore legal-info  (Peter K.)
  - drop all.hash

Regards,
Yann E. MORIN.


----------------------------------------------------------------
Yann E. MORIN (3):
      support/download: teach dl-wrapper to handle more than one hash file
      package/pkg-download: lookup hash files in global-patch-dir
      pkg-download: add option to enforce hash checking

 Config.in                                  | 27 +++++++++++--
 Makefile                                   |  2 +-
 docs/manual/adding-packages-directory.adoc |  6 +++
 docs/manual/customize-patches.adoc         | 24 ++++++++++-
 package/pkg-download.mk                    |  7 ++--
 package/pkg-generic.mk                     | 14 ++++---
 package/pkg-utils.mk                       |  8 ++--
 support/download/check-hash                | 64 ++++++++++++++++--------------
 support/download/dl-wrapper                | 10 ++---
 9 files changed, 108 insertions(+), 54 deletions(-)

--
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

[Buildroot] [PATCH 1/3 v2] support/download: teach dl-wrapper to handle more than one hash file

[Yann E. MORIN]

Currently, we expect and only use hash files that lie within the package
directory, alongside the .mk file. Those hash files are thus bundeld
with Buildroot.

This implies that only what's known to Buildroot can ever get into those
hash files. For packages where the version is fixed (or a static
choice), then we can carry hashes for those known versions.

However, we do have a few packages for which the version is a free-form
entry, where the user can provide a custom location and/or version. like
a custom VCS tree and revision, or a custom tarball URL. This means that
Buildroot has no way to be able to cary hashes for such custom versions.

This means that there is no integrity check that what was downloaded is
what was expected. For a sha1 in a git tree, this is a minor issue,
because the sha1 by itself is already a hash of the expected content.
But for custom tarballs URLs, or for a tag in a VCS, there is indeed no
integrity check.

Buildroot can't provide such hashes, but interested users may want to
provide those, and currently there is no (easy) way to do so.

So, we need our download helpers to be able to accept more than one hash
file to lookup for hashes.

Extend the dl-wrapper and the check-hash helpers thusly, and update the
legal-info accordingly.

Note that, to be able to pass more than one hash file, we also need to
re-order the arguments passed to support/download/check-hash, which also
impies some shuffling in the three places it is called:
  - 2 in dl-wrapper
  - 1 in the legal-info infra

That in turn also requires that the legal-license-file macro args get
re-ordered to have the hash file last; we take the opportunity to also
move the HOST/TARGET arg to be first, like in the other legal-info
macros.

Reported-by: "Martin Zeiser (mzeiser)" <mzeiser@cisco.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Peter Korsgaard <peter@korsgaard.com>

---
Changes v1 -> v2:
  - restore legal-info  (Peter K.)
---
 Makefile                    |  2 +-
 package/pkg-generic.mk      |  2 +-
 package/pkg-utils.mk        |  8 ++---
 support/download/check-hash | 64 ++++++++++++++++++++-----------------
 support/download/dl-wrapper | 10 +++---
 5 files changed, 45 insertions(+), 41 deletions(-)

diff --git a/Makefile b/Makefile
index ada5b96f08..b9acd7e650 100644
--- a/Makefile
+++ b/Makefile
@@ -836,7 +836,7 @@ legal-info-clean:
 .PHONY: legal-info-prepare
 legal-info-prepare: $(LEGAL_INFO_DIR)
 	@$(call MESSAGE,"Buildroot $(BR2_VERSION_FULL) Collecting legal info")
-	@$(call legal-license-file,buildroot,buildroot,support/legal-info/buildroot.hash,COPYING,COPYING,HOST)
+	@$(call legal-license-file,HOST,buildroot,buildroot,COPYING,COPYING,support/legal-info/buildroot.hash)
 	@$(call legal-manifest,TARGET,PACKAGE,VERSION,LICENSE,LICENSE FILES,SOURCE ARCHIVE,SOURCE SITE,DEPENDENCIES WITH LICENSES)
 	@$(call legal-manifest,HOST,PACKAGE,VERSION,LICENSE,LICENSE FILES,SOURCE ARCHIVE,SOURCE SITE,DEPENDENCIES WITH LICENSES)
 	@$(call legal-manifest,HOST,buildroot,$(BR2_VERSION_FULL),GPL-2.0+,COPYING,not saved,not saved)
diff --git a/package/pkg-generic.mk b/package/pkg-generic.mk
index 943f8a08c0..6c24be7505 100644
--- a/package/pkg-generic.mk
+++ b/package/pkg-generic.mk
@@ -1138,7 +1138,7 @@ ifneq ($$(call qstrip,$$($(2)_SOURCE)),)
 ifeq ($$(call qstrip,$$($(2)_LICENSE_FILES)),)
 	$(Q)$$(call legal-warning-pkg,$$($(2)_BASENAME_RAW),cannot save license ($(2)_LICENSE_FILES not defined))
 else
-	$(Q)$$(foreach F,$$($(2)_LICENSE_FILES),$$(call legal-license-file,$$($(2)_RAWNAME),$$($(2)_BASENAME_RAW),$$($(2)_HASH_FILE),$$(F),$$($(2)_DIR)/$$(F),$$(call UPPERCASE,$(4)))$$(sep))
+	$(Q)$$(foreach F,$$($(2)_LICENSE_FILES),$$(call legal-license-file,$$(call UPPERCASE,$(4)),$$($(2)_RAWNAME),$$($(2)_BASENAME_RAW),$$(F),$$($(2)_DIR)/$$(F),$$($(2)_HASH_FILE))$$(sep))
 endif # license files
 
 ifeq ($$($(2)_REDISTRIBUTE),YES)
diff --git a/package/pkg-utils.mk b/package/pkg-utils.mk
index dcab7d9b2a..b91061a572 100644
--- a/package/pkg-utils.mk
+++ b/package/pkg-utils.mk
@@ -280,13 +280,13 @@ define legal-manifest # {HOST|TARGET}, pkg, version, license, license-files, sou
 	echo '"$(2)","$(3)","$(4)","$(5)","$(6)","$(7)","$(8)"' >>$(LEGAL_MANIFEST_CSV_$(1))
 endef
 
-define legal-license-file # pkgname, pkgname-pkgver, pkg-hashfile, filename, file-fullpath, {HOST|TARGET}
-	mkdir -p $(LICENSE_FILES_DIR_$(6))/$(2)/$(dir $(4)) && \
+define legal-license-file # {HOST|TARGET}, pkgname, pkgname-pkgver, filename, file-fullpath, pkg-hashfile
+	mkdir -p $(LICENSE_FILES_DIR_$(1))/$(3)/$(dir $(4)) && \
 	{ \
-		support/download/check-hash $(3) $(5) $(4); \
+		support/download/check-hash $(5) $(4) $(6); \
 		case $${?} in (0|3) ;; (*) exit 1;; esac; \
 	} && \
-	cp $(5) $(LICENSE_FILES_DIR_$(6))/$(2)/$(4)
+	cp $(5) $(LICENSE_FILES_DIR_$(1))/$(3)/$(4)
 endef
 
 non-virtual-deps = $(foreach p,$(1),$(if $($(call UPPERCASE,$(p))_IS_VIRTUAL),,$(p)))
diff --git a/support/download/check-hash b/support/download/check-hash
index 5a47f49bc3..03a6557187 100755
--- a/support/download/check-hash
+++ b/support/download/check-hash
@@ -3,12 +3,12 @@ set -e
 
 # Helper to check a file matches its known hash
 # Call it with:
-#   $1: the path of the file containing all the expected hashes
-#   $2: the full path to the temporary file that was downloaded, and
+#   $1: the full path to the temporary file that was downloaded, and
 #       that is to be checked
-#   $3: the final basename of the file, to which it will be ultimately
+#   $2: the final basename of the file, to which it will be ultimately
 #       saved as, to be able to match it to the corresponding hashes
 #       in the .hash file
+#   $*: the paths of the files containing all the expected hashes
 #
 # Exit codes:
 #   0:  the hash file exists and the file to check matches all its hashes,
@@ -27,28 +27,21 @@ while getopts :q OPT; do
 done
 shift $((OPTIND-1))
 
-h_file="${1}"
-file="${2}"
-base="${3}"
-
-# Bail early if no hash to check
-if [ -z "${h_file}" ]; then
-    exit 0
-fi
-# Does the hash-file exist?
-if [ ! -f "${h_file}" ]; then
-    printf "WARNING: no hash file for %s\n" "${base}" >&2
-    exit 0
-fi
+file="${1}"
+base="${2}"
+shift 2
+declare -a h_files=( "${@}" )
 
 # Check one hash for a file
 # $1: algo hash
 # $2: known hash
 # $3: file (full path)
+# $4: hash file (full path)
 check_one_hash() {
     _h="${1}"
     _known="${2}"
     _file="${3}"
+    _h_file="${4}"
 
     # Note: md5 is supported, but undocumented on purpose.
     # Note: sha3 is not supported, since there is currently no implementation
@@ -70,6 +63,7 @@ check_one_hash() {
         return 0
     fi
 
+    printf "ERROR: while checking hashes from %s\n" "${_h_file}" >&2
     printf "ERROR: %s has wrong %s hash:\n" "${base}" "${_h}" >&2
     printf "ERROR: expected: %s\n" "${_known}" >&2
     printf "ERROR: got     : %s\n" "${_hash}" >&2
@@ -79,21 +73,31 @@ check_one_hash() {
 }
 
 # Do we know one or more hashes for that file?
+nb_h_files=0
 nb_checks=0
-while read t h f; do
-    case "${t}" in
-        ''|'#'*)
-            # Skip comments and empty lines
-            continue
-            ;;
-        *)
-            if [ "${f}" = "${base}" ]; then
-                check_one_hash "${t}" "${h}" "${file}"
-                : $((nb_checks++))
-            fi
-            ;;
-    esac
-done <"${h_file}"
+for h_file in "${h_files[@]}"; do
+    [ -f "${h_file}" ] || continue
+    : $((nb_h_files++))
+    while read t h f; do
+        case "${t}" in
+            ''|'#'*)
+                # Skip comments and empty lines
+                continue
+                ;;
+            *)
+                if [ "${f}" = "${base}" ]; then
+                    check_one_hash "${t}" "${h}" "${file}" "${h_file}"
+                    : $((nb_checks++))
+                fi
+                ;;
+        esac
+    done <"${h_file}"
+done
+
+if [ ${nb_h_files} -eq 0 ]; then
+    printf "WARNING: no hash file for %s\n" "${base}" >&2
+    exit 0
+fi
 
 if [ ${nb_checks} -eq 0 ]; then
     case " ${BR_NO_CHECK_HASH_FOR} " in
diff --git a/support/download/dl-wrapper b/support/download/dl-wrapper
index 1e8d6058f6..35428faeef 100755
--- a/support/download/dl-wrapper
+++ b/support/download/dl-wrapper
@@ -21,8 +21,8 @@ export BR_BACKEND_DL_GETOPTS=":hc:d:o:n:N:H:lru:qf:e"
 
 main() {
     local OPT OPTARG
-    local backend output hfile large_file recurse quiet rc
-    local -a uris
+    local backend output large_file recurse quiet rc
+    local -a uris hfiles
 
     # Parse our options; anything after '--' is for the backend
     while getopts ":c:d:D:o:n:N:H:lrf:u:qp:" OPT; do
@@ -33,7 +33,7 @@ main() {
         o)  output="${OPTARG}";;
         n)  raw_base_name="${OPTARG}";;
         N)  base_name="${OPTARG}";;
-        H)  hfile="${OPTARG}";;
+        H)  hfiles+=( "${OPTARG}" );;
         l)  large_file="-l";;
         r)  recurse="-r";;
         f)  filename="${OPTARG}";;
@@ -70,7 +70,7 @@ main() {
     # - fails at least one of its hashes: force a re-download
     # - there's no hash (but a .hash file): consider it a hard error
     if [ -e "${output}" ]; then
-        if support/download/check-hash ${quiet} "${hfile}" "${output}" "${output##*/}"; then
+        if support/download/check-hash ${quiet} "${output}" "${output##*/}" "${hfiles[@]}"; then
             exit 0
         elif [ ${?} -ne 2 ]; then
             # Do not remove the file, otherwise it might get re-downloaded
@@ -154,7 +154,7 @@ main() {
 
         # Check if the downloaded file is sane, and matches the stored hashes
         # for that file
-        if support/download/check-hash ${quiet} "${hfile}" "${tmpf}" "${output##*/}"; then
+        if support/download/check-hash ${quiet} "${tmpf}" "${output##*/}" "${hfiles[@]}"; then
             rc=0
         else
             if [ ${?} -ne 3 ]; then
-- 
2.25.1

_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

[Buildroot] [PATCH 2/3 v2] package/pkg-download: lookup hash files in global-patch-dir

[Yann E. MORIN]

Currently, we expect and only use hash files that lie within the package
directory, alongside the .mk file. Those hash files are thus bundled
with Buildroot.

This implies that only what's known to Buildroot can ever get into those
hash files. For packages where the version is fixed (or a static
choice), then we can carry hashes for those known versions.

However, we do have a few packages for which the version is a free-form
entry, where the user can provide a custom location and/or version.  like
a custom VCS tree and revision, or a custom tarball URL. This means that
Buildroot has no way to be able to cary hashes for such custom versions.

This means that there is no integrity check that what was downloaded is
what was expected. For a sha1 in a git tree, this is a minor issue,
because the sha1 by itself is already a hash of the expected content.
But for custom tarballs URLs, or for a tag in a VCS, there is indeed no
integrity check.

Buildroot can't provide such hashes, but interested users may want to
provide those, and currently there is no (easy) way to do so.

We leverage the existng global-patch-dir mechanism to look for extra
hash files. We use the same heuristic that is used for bundled hash
files, and for each global patch directory <dir>, we use the first file
to exist among:
 1. look into <dir>/<package>/<version>/<package>.hash
 2. look into <dir>/<package>/<package>.hash

Reported-by: "Martin Zeiser (mzeiser)" <mzeiser@cisco.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>

---
Changes v1 -> v2:
  - drop <dir>/all.hash
---
 Config.in                                  | 10 +++++----
 docs/manual/adding-packages-directory.adoc |  6 ++++++
 docs/manual/customize-patches.adoc         | 24 +++++++++++++++++++++-
 package/pkg-download.mk                    |  2 +-
 package/pkg-generic.mk                     | 14 ++++++++-----
 package/pkg-utils.mk                       |  2 +-
 6 files changed, 46 insertions(+), 12 deletions(-)

diff --git a/Config.in b/Config.in
index 7dd49567f6..b344d0e80a 100644
--- a/Config.in
+++ b/Config.in
@@ -674,12 +674,12 @@ config BR2_PACKAGE_OVERRIDE_FILE
 	  documentation for more details on this feature.
 
 config BR2_GLOBAL_PATCH_DIR
-	string "global patch directories"
+	string "global patch and hash directories"
 	help
 	  You may specify a space separated list of one or more
-	  directories containing global package patches. For a specific
-	  version <packageversion> of a specific package <packagename>,
-	  patches are applied as follows:
+	  directories containing global package patches and/or hashes.
+	  For a specific version <packageversion> of a specific package
+	  <packagename>, patches are looked up as follows:
 
 	  First, the default Buildroot patch set for the package is
 	  applied from the package's directory in Buildroot.
@@ -693,6 +693,8 @@ config BR2_GLOBAL_PATCH_DIR
 	  exists, then all *.patch files in the directory will be
 	  applied.
 
+	  The hash files are looked up similarly to the patches.
+
 menu "Advanced"
 
 config BR2_FORCE_HOST_BUILD
diff --git a/docs/manual/adding-packages-directory.adoc b/docs/manual/adding-packages-directory.adoc
index 7150018a3b..5a0f298eb1 100644
--- a/docs/manual/adding-packages-directory.adoc
+++ b/docs/manual/adding-packages-directory.adoc
@@ -540,6 +540,12 @@ mercurial) because Buildroot currently does not generate reproducible
 tarballs when source code is fetched from such version control
 systems.
 
+Additionally, for packages for which it is possible to specify a custom
+version (e.g. a custom version string, a remote tarball URL, or a VCS
+repository location and changeset), Buildroot can't carry hashes for
+those. It is however possible to xref:customize-hashes[provide a list of
+extra hashes] that can cover such cases.
+
 Hashes should only be added in +.hash+ files for files that are
 guaranteed to be stable. For example, patches auto-generated by Github
 are not guaranteed to be stable, and therefore their hashes can change
diff --git a/docs/manual/customize-patches.adoc b/docs/manual/customize-patches.adoc
index eb98d1bea2..72e87c3c04 100644
--- a/docs/manual/customize-patches.adoc
+++ b/docs/manual/customize-patches.adoc
@@ -1,8 +1,10 @@
 // -*- mode:doc -*- ;
 // vim: set syntax=asciidoc:
 
+=== Adding project-specific patches and hashes
+
 [[customize-patches]]
-=== Adding project-specific patches
+==== Providing extra patches
 
 It is sometimes useful to apply 'extra' patches to packages - on top of
 those provided in Buildroot. This might be used to support custom
@@ -57,3 +59,23 @@ are available at a URL. *Note:* +BR2_LINUX_KERNEL_PATCH+ specifies kernel
 patches that are applied after patches available in +BR2_GLOBAL_PATCH_DIR+,
 as it is done from a post-patch hook of the Linux package.
 
+
+[[customize-hashes]]
+==== Providing extra hashes
+
+Buildroot bundles a xref:adding-packages-hash[list of hashes] against
+which it checks the integrity of the downloaded archives, or of those
+it generates locally from VCS checkouts. However, it can only do so
+for the known versions; for packages where it is possible to specify
+a custom version (e.g. a custom version string, a remote tarball URL,
+or a VCS repository location and changeset), Buildroot can't carry
+hashes for those.
+
+For users concerned with the integrity of such downloads, it is possible
+to provide a list of hashes that Buildroot can use to check arbitrary
+downloaded files. Those extra hashes are looked up similarly to the
+extra patches (above); for each directory in +BR2_GLOBAL_PATCH_DIR+,
+the first file to exist is used to check a package download:
+
+* +<global-patch-dir>/<packagename>/<packageversion>/<packagename>.hash+
+* +<global-patch-dir>/<packagename>/<packagename>.hash+
diff --git a/package/pkg-download.mk b/package/pkg-download.mk
index e5cd83d859..bf1c19e5ec 100644
--- a/package/pkg-download.mk
+++ b/package/pkg-download.mk
@@ -115,7 +115,7 @@ define DOWNLOAD
 		-d '$($(2)_DL_DIR)' \
 		-D '$(DL_DIR)' \
 		-f '$(notdir $(1))' \
-		-H '$($(2)_HASH_FILE)' \
+		$(foreach f,$($(2)_HASH_FILES),-H '$(f)') \
 		-n '$($(2)_BASENAME_RAW)' \
 		-N '$($(2)_RAWNAME)' \
 		-o '$($(2)_DL_DIR)/$(notdir $(1))' \
diff --git a/package/pkg-generic.mk b/package/pkg-generic.mk
index 6c24be7505..577a148c1e 100644
--- a/package/pkg-generic.mk
+++ b/package/pkg-generic.mk
@@ -509,11 +509,15 @@ else
 endif
 $(2)_VERSION := $$(call sanitize,$$($(2)_DL_VERSION))
 
-$(2)_HASH_FILE = \
+$(2)_HASH_FILES = \
 	$$(strip \
-		$$(if $$(wildcard $$($(2)_PKGDIR)/$$($(2)_VERSION)/$$($(2)_RAWNAME).hash),\
-			$$($(2)_PKGDIR)/$$($(2)_VERSION)/$$($(2)_RAWNAME).hash,\
-			$$($(2)_PKGDIR)/$$($(2)_RAWNAME).hash))
+		$$(foreach d, $$($(2)_PKGDIR) $$(addsuffix /$$($(2)_RAWNAME), $$(call qstrip,$$(BR2_GLOBAL_PATCH_DIR))),\
+			$$(if $$(wildcard $$(d)/$$($(2)_VERSION)/$$($(2)_RAWNAME).hash),\
+				$$(d)/$$($(2)_VERSION)/$$($(2)_RAWNAME).hash,\
+				$$(d)/$$($(2)_RAWNAME).hash\
+			)\
+		)\
+	)
 
 ifdef $(3)_OVERRIDE_SRCDIR
   $(2)_OVERRIDE_SRCDIR ?= $$($(3)_OVERRIDE_SRCDIR)
@@ -1138,7 +1142,7 @@ ifneq ($$(call qstrip,$$($(2)_SOURCE)),)
 ifeq ($$(call qstrip,$$($(2)_LICENSE_FILES)),)
 	$(Q)$$(call legal-warning-pkg,$$($(2)_BASENAME_RAW),cannot save license ($(2)_LICENSE_FILES not defined))
 else
-	$(Q)$$(foreach F,$$($(2)_LICENSE_FILES),$$(call legal-license-file,$$(call UPPERCASE,$(4)),$$($(2)_RAWNAME),$$($(2)_BASENAME_RAW),$$(F),$$($(2)_DIR)/$$(F),$$($(2)_HASH_FILE))$$(sep))
+	$(Q)$$(foreach F,$$($(2)_LICENSE_FILES),$$(call legal-license-file,$$(call UPPERCASE,$(4)),$$($(2)_RAWNAME),$$($(2)_BASENAME_RAW),$$(F),$$($(2)_DIR)/$$(F),$$($(2)_HASH_FILES))$$(sep))
 endif # license files
 
 ifeq ($$($(2)_REDISTRIBUTE),YES)
diff --git a/package/pkg-utils.mk b/package/pkg-utils.mk
index b91061a572..059e86ae0a 100644
--- a/package/pkg-utils.mk
+++ b/package/pkg-utils.mk
@@ -280,7 +280,7 @@ define legal-manifest # {HOST|TARGET}, pkg, version, license, license-files, sou
 	echo '"$(2)","$(3)","$(4)","$(5)","$(6)","$(7)","$(8)"' >>$(LEGAL_MANIFEST_CSV_$(1))
 endef
 
-define legal-license-file # {HOST|TARGET}, pkgname, pkgname-pkgver, filename, file-fullpath, pkg-hashfile
+define legal-license-file # {HOST|TARGET}, pkgname, pkgname-pkgver, filename, file-fullpath, pkg-hashfiles
 	mkdir -p $(LICENSE_FILES_DIR_$(1))/$(3)/$(dir $(4)) && \
 	{ \
 		support/download/check-hash $(5) $(4) $(6); \
-- 
2.25.1

_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

[Buildroot] [PATCH 3/3 v2] pkg-download: add option to enforce hash checking

[Yann E. MORIN]

Currently, when a pacakge is downloaded from a custom location or
version, Buildroot excludes such a package from the mandatory integrity
check with hashes, because it was until now not possible to have such
hashes.

We now have a mechanism which users can leverage to provide additional
hashes, and so custom versions or locations can now be checked too.

Buildroot has no way to know that hashes have indeed been provided for
a custom location/version, and so will still happilly ignore an
unchecked package.

However, users who do provide extra hashes most probably do expect that
no download is done without an integrity check, and thus expect that a
missing hash not be ignored.

Add an option that users can select to make Buildroot forcibly require
at least one valid hash, and no invalid hash, for all downloads.

Reported-by: "Martin Zeiser (mzeiser)" <mzeiser@cisco.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
---
 Config.in               | 17 +++++++++++++++++
 package/pkg-download.mk |  5 ++---
 2 files changed, 19 insertions(+), 3 deletions(-)

diff --git a/Config.in b/Config.in
index b344d0e80a..27f5deb16a 100644
--- a/Config.in
+++ b/Config.in
@@ -709,6 +709,23 @@ config BR2_FORCE_HOST_BUILD
 
 	  This option will increase build time.
 
+config BR2_DOWNLOAD_FORCE_CHECK_HASHES
+	bool "Force all downloads to have a valid hash"
+	depends on BR2_GLOBAL_PATCH_DIR != ""
+	help
+	  For packages where a custom version or location can be set,
+	  Buildroot does not carry a hash for those custom versions or
+	  locations, so the integrity of such downloads is not verified.
+
+	  Say 'y' here to enforce downloads to have at least one valid
+	  hash (and of course, that all hashes be valid).
+
+	  Those hashes are looked in files in BR2_GLOBAL_PATCH_DIR,
+	  see above.
+
+comment "Forcing all downloads to have a valid hash needs a global patch and hash directory"
+	depends on BR2_GLOBAL_PATCH_DIR = ""
+
 config BR2_REPRODUCIBLE
 	bool "Make the build reproducible (experimental)"
 	# SOURCE_DATE_EPOCH support in toolchain-wrapper requires GCC 4.4
diff --git a/package/pkg-download.mk b/package/pkg-download.mk
index bf1c19e5ec..30eeb6b1fc 100644
--- a/package/pkg-download.mk
+++ b/package/pkg-download.mk
@@ -66,9 +66,7 @@ github = https://github.com/$(1)/$(2)/archive/$(3)
 gitlab = https://gitlab.com/$(1)/$(2)/-/archive/$(3)
 
 # Expressly do not check hashes for those files
-# Exported variables default to immediately expanded in some versions of
-# make, but we need it to be recursively-epxanded, so explicitly assign it.
-export BR_NO_CHECK_HASH_FOR =
+BR_NO_CHECK_HASH_FOR =
 
 ################################################################################
 # DOWNLOAD_URIS - List the candidates URIs where to get the package from:
@@ -110,6 +108,7 @@ endif
 define DOWNLOAD
 	$(Q)mkdir -p $($(2)_DL_DIR)
 	$(Q)$(EXTRA_ENV) $($(2)_DL_ENV) \
+	BR_NO_CHECK_HASH_FOR="$(if $(BR2_DOWNLOAD_FORCE_CHECK_HASHES),,$(BR_NO_CHECK_HASH_FOR))" \
 		flock $($(2)_DL_DIR)/.lock $(DL_WRAPPER) \
 		-c '$($(2)_DL_VERSION)' \
 		-d '$($(2)_DL_DIR)' \
-- 
2.25.1

_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH 1/3 v2] support/download: teach dl-wrapper to handle more than one hash file

[Peter Korsgaard]

>>>>> "Yann" == Yann E MORIN <yann.morin.1998@free.fr> writes:

 > Currently, we expect and only use hash files that lie within the package
 > directory, alongside the .mk file. Those hash files are thus bundeld
 > with Buildroot.

 > This implies that only what's known to Buildroot can ever get into those
 > hash files. For packages where the version is fixed (or a static
 > choice), then we can carry hashes for those known versions.

 > However, we do have a few packages for which the version is a free-form
 > entry, where the user can provide a custom location and/or version. like
 > a custom VCS tree and revision, or a custom tarball URL. This means that
 > Buildroot has no way to be able to cary hashes for such custom versions.

 > This means that there is no integrity check that what was downloaded is
 > what was expected. For a sha1 in a git tree, this is a minor issue,
 > because the sha1 by itself is already a hash of the expected content.
 > But for custom tarballs URLs, or for a tag in a VCS, there is indeed no
 > integrity check.

 > Buildroot can't provide such hashes, but interested users may want to
 > provide those, and currently there is no (easy) way to do so.

 > So, we need our download helpers to be able to accept more than one hash
 > file to lookup for hashes.

 > Extend the dl-wrapper and the check-hash helpers thusly, and update the
 > legal-info accordingly.

 > Note that, to be able to pass more than one hash file, we also need to
 > re-order the arguments passed to support/download/check-hash, which also
 > impies some shuffling in the three places it is called:
 >   - 2 in dl-wrapper
 >   - 1 in the legal-info infra

 > That in turn also requires that the legal-license-file macro args get
 > re-ordered to have the hash file last; we take the opportunity to also
 > move the HOST/TARGET arg to be first, like in the other legal-info
 > macros.

 > Reported-by: "Martin Zeiser (mzeiser)" <mzeiser@cisco.com>
 > Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
 > Cc: Peter Korsgaard <peter@korsgaard.com>

 > ---
 > Changes v1 -> v2:
 >   - restore legal-info  (Peter K.)

Committed, thanks.

-- 
Bye, Peter Korsgaard
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH 2/3 v2] package/pkg-download: lookup hash files in global-patch-dir

[Peter Korsgaard]

>>>>> "Yann" == Yann E MORIN <yann.morin.1998@free.fr> writes:

 > Currently, we expect and only use hash files that lie within the package
 > directory, alongside the .mk file. Those hash files are thus bundled
 > with Buildroot.

 > This implies that only what's known to Buildroot can ever get into those
 > hash files. For packages where the version is fixed (or a static
 > choice), then we can carry hashes for those known versions.

 > However, we do have a few packages for which the version is a free-form
 > entry, where the user can provide a custom location and/or version.  like
 > a custom VCS tree and revision, or a custom tarball URL. This means that
 > Buildroot has no way to be able to cary hashes for such custom versions.

 > This means that there is no integrity check that what was downloaded is
 > what was expected. For a sha1 in a git tree, this is a minor issue,
 > because the sha1 by itself is already a hash of the expected content.
 > But for custom tarballs URLs, or for a tag in a VCS, there is indeed no
 > integrity check.

 > Buildroot can't provide such hashes, but interested users may want to
 > provide those, and currently there is no (easy) way to do so.

 > We leverage the existng global-patch-dir mechanism to look for extra
 > hash files. We use the same heuristic that is used for bundled hash
 > files, and for each global patch directory <dir>, we use the first file
 > to exist among:
 >  1. look into <dir>/<package>/<version>/<package>.hash
 >  2. look into <dir>/<package>/<package>.hash

 > Reported-by: "Martin Zeiser (mzeiser)" <mzeiser@cisco.com>
 > Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>

 > ---
 > Changes v1 -> v2:
 >   - drop <dir>/all.hash

Committed, thanks.

-- 
Bye, Peter Korsgaard
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH 3/3 v2] pkg-download: add option to enforce hash checking

[Peter Korsgaard]

>>>>> "Yann" == Yann E MORIN <yann.morin.1998@free.fr> writes:

 > Currently, when a pacakge is downloaded from a custom location or
 > version, Buildroot excludes such a package from the mandatory integrity
 > check with hashes, because it was until now not possible to have such
 > hashes.

 > We now have a mechanism which users can leverage to provide additional
 > hashes, and so custom versions or locations can now be checked too.

 > Buildroot has no way to know that hashes have indeed been provided for
 > a custom location/version, and so will still happilly ignore an
 > unchecked package.

 > However, users who do provide extra hashes most probably do expect that
 > no download is done without an integrity check, and thus expect that a
 > missing hash not be ignored.

 > Add an option that users can select to make Buildroot forcibly require
 > at least one valid hash, and no invalid hash, for all downloads.

 > Reported-by: "Martin Zeiser (mzeiser)" <mzeiser@cisco.com>
 > Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>

Committed, thanks.

-- 
Bye, Peter Korsgaard
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH 1/3 v2] support/download: teach dl-wrapper to handle more than one hash file

[Peter Korsgaard]

>>>>> "Peter" == Peter Korsgaard <peter@korsgaard.com> writes:

>>>>> "Yann" == Yann E MORIN <yann.morin.1998@free.fr> writes:
 >> Currently, we expect and only use hash files that lie within the package
 >> directory, alongside the .mk file. Those hash files are thus bundeld
 >> with Buildroot.

 >> This implies that only what's known to Buildroot can ever get into those
 >> hash files. For packages where the version is fixed (or a static
 >> choice), then we can carry hashes for those known versions.

 >> However, we do have a few packages for which the version is a free-form
 >> entry, where the user can provide a custom location and/or version. like
 >> a custom VCS tree and revision, or a custom tarball URL. This means that
 >> Buildroot has no way to be able to cary hashes for such custom versions.

 >> This means that there is no integrity check that what was downloaded is
 >> what was expected. For a sha1 in a git tree, this is a minor issue,
 >> because the sha1 by itself is already a hash of the expected content.
 >> But for custom tarballs URLs, or for a tag in a VCS, there is indeed no
 >> integrity check.

 >> Buildroot can't provide such hashes, but interested users may want to
 >> provide those, and currently there is no (easy) way to do so.

 >> So, we need our download helpers to be able to accept more than one hash
 >> file to lookup for hashes.

 >> Extend the dl-wrapper and the check-hash helpers thusly, and update the
 >> legal-info accordingly.

 >> Note that, to be able to pass more than one hash file, we also need to
 >> re-order the arguments passed to support/download/check-hash, which also
 >> impies some shuffling in the three places it is called:
 >> - 2 in dl-wrapper
 >> - 1 in the legal-info infra

 >> That in turn also requires that the legal-license-file macro args get
 >> re-ordered to have the hash file last; we take the opportunity to also
 >> move the HOST/TARGET arg to be first, like in the other legal-info
 >> macros.

 >> Reported-by: "Martin Zeiser (mzeiser)" <mzeiser@cisco.com>
 >> Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
 >> Cc: Peter Korsgaard <peter@korsgaard.com>

 >> ---
 >> Changes v1 -> v2:
 >> - restore legal-info  (Peter K.)

 > Committed, thanks.

Committed to 2023.02.x and 2023.08.x, thanks.

-- 
Bye, Peter Korsgaard
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH 2/3 v2] package/pkg-download: lookup hash files in global-patch-dir

[Peter Korsgaard]

>>>>> "Yann" == Yann E MORIN <yann.morin.1998@free.fr> writes:

 > Currently, we expect and only use hash files that lie within the package
 > directory, alongside the .mk file. Those hash files are thus bundled
 > with Buildroot.

 > This implies that only what's known to Buildroot can ever get into those
 > hash files. For packages where the version is fixed (or a static
 > choice), then we can carry hashes for those known versions.

 > However, we do have a few packages for which the version is a free-form
 > entry, where the user can provide a custom location and/or version.  like
 > a custom VCS tree and revision, or a custom tarball URL. This means that
 > Buildroot has no way to be able to cary hashes for such custom versions.

 > This means that there is no integrity check that what was downloaded is
 > what was expected. For a sha1 in a git tree, this is a minor issue,
 > because the sha1 by itself is already a hash of the expected content.
 > But for custom tarballs URLs, or for a tag in a VCS, there is indeed no
 > integrity check.

 > Buildroot can't provide such hashes, but interested users may want to
 > provide those, and currently there is no (easy) way to do so.

 > We leverage the existng global-patch-dir mechanism to look for extra
 > hash files. We use the same heuristic that is used for bundled hash
 > files, and for each global patch directory <dir>, we use the first file
 > to exist among:
 >  1. look into <dir>/<package>/<version>/<package>.hash
 >  2. look into <dir>/<package>/<package>.hash

 > Reported-by: "Martin Zeiser (mzeiser)" <mzeiser@cisco.com>
 > Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>

 > ---
 > Changes v1 -> v2:
 >   - drop <dir>/all.hash

Committed to 2023.02.x and 2023.08.x, thanks.

-- 
Bye, Peter Korsgaard
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH 3/3 v2] pkg-download: add option to enforce hash checking

[Peter Korsgaard]

>>>>> "Yann" == Yann E MORIN <yann.morin.1998@free.fr> writes:

 > Currently, when a pacakge is downloaded from a custom location or
 > version, Buildroot excludes such a package from the mandatory integrity
 > check with hashes, because it was until now not possible to have such
 > hashes.

 > We now have a mechanism which users can leverage to provide additional
 > hashes, and so custom versions or locations can now be checked too.

 > Buildroot has no way to know that hashes have indeed been provided for
 > a custom location/version, and so will still happilly ignore an
 > unchecked package.

 > However, users who do provide extra hashes most probably do expect that
 > no download is done without an integrity check, and thus expect that a
 > missing hash not be ignored.

 > Add an option that users can select to make Buildroot forcibly require
 > at least one valid hash, and no invalid hash, for all downloads.

 > Reported-by: "Martin Zeiser (mzeiser)" <mzeiser@cisco.com>
 > Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>

Committed to 2023.02.x and 2023.08.x, thanks.

-- 
Bye, Peter Korsgaard
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot