Buildroot Archive on lore.kernel.org
 help / color / mirror / Atom feed
* [Buildroot] [PATCH v2 1/3] support/download: add blake2b-256 hash support
@ 2026-06-17 17:42 James Hilliard
  2026-06-17 17:42 ` [Buildroot] [PATCH v2 2/3] package/pkg-python: derive PyPI site from blake2b-256 hash James Hilliard
                   ` (2 more replies)
  0 siblings, 3 replies; 4+ messages in thread
From: James Hilliard @ 2026-06-17 17:42 UTC (permalink / raw)
  To: buildroot
  Cc: Thomas Petazzoni, James Hilliard, Thomas Perale,
	Ricardo Martincoski

PyPI exposes a blake2b_256 digest for distribution files, and that
same digest is used in the hash-based files.pythonhosted.org download
paths. Add support for storing those hashes in Buildroot .hash files.

Teach the download hash checker to validate blake2b-256 entries using
b2sum -l 256, and update check-package so the new hash type and length
are accepted. Also add the CycloneDX hash spelling so generated SBOMs
can include blake2b-256 hashes from .hash files.

Update scanpypi to write the PyPI-provided blake2b_256 digest alongside
the existing md5 and sha256 entries.

Signed-off-by: James Hilliard <james.hilliard1@gmail.com>
---
 docs/manual/adding-packages-directory.adoc             | 3 ++-
 support/download/check-hash                            | 7 ++++++-
 support/testing/tests/utils/test_generate_cyclonedx.py | 9 +++++++++
 utils/checkpackagelib/lib_hash.py                      | 2 +-
 utils/checkpackagelib/test_lib_hash.py                 | 4 ++++
 utils/generate-cyclonedx                               | 1 +
 utils/scanpypi                                         | 7 ++++++-
 7 files changed, 29 insertions(+), 4 deletions(-)

diff --git a/docs/manual/adding-packages-directory.adoc b/docs/manual/adding-packages-directory.adoc
index 15de559deb..27f0fcd9a9 100644
--- a/docs/manual/adding-packages-directory.adoc
+++ b/docs/manual/adding-packages-directory.adoc
@@ -464,7 +464,7 @@ The format of this file is one line for each file for which to check the
 hash, each line with the following three fields separated by two spaces:
 
 * the type of hash, one of:
-** +md5+, +sha1+, +sha224+, +sha256+, +sha384+, +sha512+
+** +md5+, +sha1+, +sha224+, +sha256+, +sha384+, +sha512+, +blake2b-256+
 * the hash of the file:
 ** for +md5+, 32 hexadecimal characters
 ** for +sha1+, 40 hexadecimal characters
@@ -472,6 +472,7 @@ hash, each line with the following three fields separated by two spaces:
 ** for +sha256+, 64 hexadecimal characters
 ** for +sha384+, 96 hexadecimal characters
 ** for +sha512+, 128 hexadecimal characters
+** for +blake2b-256+, 64 hexadecimal characters
 * the name of the file:
 ** for a source archive: the basename of the file, without any directory
    component,
diff --git a/support/download/check-hash b/support/download/check-hash
index d18ec8b134..74079dac1a 100755
--- a/support/download/check-hash
+++ b/support/download/check-hash
@@ -49,6 +49,7 @@ check_one_hash() {
     case "${_h}" in
         md5|sha1)                       ;;
         sha224|sha256|sha384|sha512)    ;;
+        blake2b-256)                    ;;
         *) # Unknown hash, exit with error
             printf "ERROR: unknown hash '%s' for '%s'\n"  \
                    "${_h}" "${base}" >&2
@@ -57,7 +58,11 @@ check_one_hash() {
     esac
 
     # Do the hashes match?
-    _hash="$( "${_h}sum" "${_file}" |cut -d ' ' -f 1 )"
+    if [ "${_h}" = "blake2b-256" ]; then
+        _hash="$( b2sum -l 256 "${_file}" |cut -d ' ' -f 1 )"
+    else
+        _hash="$( "${_h}sum" "${_file}" |cut -d ' ' -f 1 )"
+    fi
     if [ "${_hash}" = "${_known}" ]; then
         printf "%s: OK (%s: %s)\n" "${base}" "${_h}" "${_hash}"
         return 0
diff --git a/support/testing/tests/utils/test_generate_cyclonedx.py b/support/testing/tests/utils/test_generate_cyclonedx.py
index e6640fbd0d..bb80a5ff23 100644
--- a/support/testing/tests/utils/test_generate_cyclonedx.py
+++ b/support/testing/tests/utils/test_generate_cyclonedx.py
@@ -186,6 +186,7 @@ class TestGenerateCycloneDX(unittest.TestCase):
                 "# source archive checksums\n"
                 "sha256 1111111111111111111111111111111111111111111111111111111111111111 foo-1.2.tar.gz\n"
                 "sha1 2222222222222222222222222222222222222222 foo-1.2.tar.gz\n"
+                "blake2b-256 3333333333333333333333333333333333333333333333333333333333333333 foo-1.2.tar.gz\n"
                 "sha256 aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa LICENSE\n"
             )
 
@@ -220,6 +221,10 @@ class TestGenerateCycloneDX(unittest.TestCase):
                             "alg": "SHA-1",
                             "content": "2222222222222222222222222222222222222222",
                         },
+                        {
+                            "alg": "BLAKE2b-256",
+                            "content": "3333333333333333333333333333333333333333333333333333333333333333",
+                        },
                     ]
                 },
                 {
@@ -234,6 +239,10 @@ class TestGenerateCycloneDX(unittest.TestCase):
                             "alg": "SHA-1",
                             "content": "2222222222222222222222222222222222222222",
                         },
+                        {
+                            "alg": "BLAKE2b-256",
+                            "content": "3333333333333333333333333333333333333333333333333333333333333333",
+                        },
                     ],
                 }
             ],
diff --git a/utils/checkpackagelib/lib_hash.py b/utils/checkpackagelib/lib_hash.py
index 5968c809bf..3eafe7d9e0 100644
--- a/utils/checkpackagelib/lib_hash.py
+++ b/utils/checkpackagelib/lib_hash.py
@@ -31,7 +31,7 @@ class HashNumberOfFields(_CheckFunction):
 
 class HashType(_CheckFunction):
     len_of_hash = {"md5": 32, "sha1": 40, "sha224": 56, "sha256": 64,
-                   "sha384": 96, "sha512": 128}
+                   "sha384": 96, "sha512": 128, "blake2b-256": 64}
 
     def check_line(self, lineno, text):
         if _empty_line_or_comment(text):
diff --git a/utils/checkpackagelib/test_lib_hash.py b/utils/checkpackagelib/test_lib_hash.py
index fdc6338189..cbccad647d 100644
--- a/utils/checkpackagelib/test_lib_hash.py
+++ b/utils/checkpackagelib/test_lib_hash.py
@@ -124,6 +124,10 @@ HashType = [
      'sha512 1234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678'
      '9012345678\n',
      []),
+    ('blake2b-256',
+     'any',
+     'blake2b-256 1234567890123456789012345678901234567890123456789012345678901234\n',
+     []),
     ]
 
 
diff --git a/utils/generate-cyclonedx b/utils/generate-cyclonedx
index bade018cd4..2f5981934f 100755
--- a/utils/generate-cyclonedx
+++ b/utils/generate-cyclonedx
@@ -298,6 +298,7 @@ def cyclonedx_source_hashes(comp, source):
         "sha256": "SHA-256",
         "sha512": "SHA-512",
         "md5": "MD5",
+        "blake2b-256": "BLAKE2b-256",
     }
 
     hashes = []
diff --git a/utils/scanpypi b/utils/scanpypi
index 61879e39d4..de658328fe 100755
--- a/utils/scanpypi
+++ b/utils/scanpypi
@@ -672,7 +672,7 @@ class BuildrootPackage():
         print('Creating {filename}...'.format(filename=path_to_hash))
         lines = []
         if self.used_url['digests']['md5'] and self.used_url['digests']['sha256']:
-            hash_header = '# md5, sha256 from {url}\n'.format(
+            hash_header = '# md5, sha256, blake2b-256 from {url}\n'.format(
                 url=self.metadata_url)
             lines.append(hash_header)
             hash_line = '{method}  {digest}  {filename}\n'.format(
@@ -685,6 +685,11 @@ class BuildrootPackage():
                 digest=self.used_url['digests']['sha256'],
                 filename=self.filename)
             lines.append(hash_line)
+            hash_line = '{method}  {digest}  {filename}\n'.format(
+                method='blake2b-256',
+                digest=self.used_url['digests']['blake2b_256'],
+                filename=self.filename)
+            lines.append(hash_line)
 
         if self.license_files:
             lines.append('# Locally computed sha256 checksums\n')
-- 
2.53.0

_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

^ permalink raw reply related	[flat|nested] 4+ messages in thread

* [Buildroot] [PATCH v2 2/3] package/pkg-python: derive PyPI site from blake2b-256 hash
  2026-06-17 17:42 [Buildroot] [PATCH v2 1/3] support/download: add blake2b-256 hash support James Hilliard
@ 2026-06-17 17:42 ` James Hilliard
  2026-06-17 17:42 ` [Buildroot] [PATCH v2 3/3] package/python-async-timeout: use hash-based PyPI site James Hilliard
  2026-06-18  5:51 ` [Buildroot] [PATCH v2 1/3] support/download: add blake2b-256 hash support yann.morin
  2 siblings, 0 replies; 4+ messages in thread
From: James Hilliard @ 2026-06-17 17:42 UTC (permalink / raw)
  To: buildroot
  Cc: Thomas Petazzoni, James Hilliard, Thomas Perale,
	Ricardo Martincoski

Python packages downloaded from PyPI currently store the complete
files.pythonhosted.org package path in their .mk file. That path is
hash-based, so it changes for every version bump and adds churn to
updates and backports.

When a Python package does not define _SITE, derive the hash-based
files.pythonhosted.org site from the blake2b-256 entry matching the
package source archive in the package .hash file before the generic
package infrastructure validates _SITE.

Follow the usual target/host fallback pattern for _SITE, and raise an
explicit error if no hash file is available or if the hash file does not
contain a matching blake2b-256 entry.

Update scanpypi to omit generated _SITE assignments so newly generated
PyPI packages can use the resolver.

Signed-off-by: James Hilliard <james.hilliard1@gmail.com>
---
Changes v1 -> v2:
  - Follow the usual target/host _SITE fallback pattern.
  - Drop the /dev/null awk input and guard missing hash files explicitly.
  - Error out when no matching blake2b-256 entry is found.
---
 docs/manual/adding-packages-python.adoc |  5 +++++
 package/pkg-python.mk                   | 14 ++++++++++++++
 utils/scanpypi                          |  9 ---------
 3 files changed, 19 insertions(+), 9 deletions(-)

diff --git a/docs/manual/adding-packages-python.adoc b/docs/manual/adding-packages-python.adoc
index d6bae299a0..a66af1e7e2 100644
--- a/docs/manual/adding-packages-python.adoc
+++ b/docs/manual/adding-packages-python.adoc
@@ -82,6 +82,11 @@ All the package metadata information variables that exist in the
 xref:generic-package-reference[generic package infrastructure] also
 exist in the Python infrastructure.
 
+For Python packages downloaded from PyPI, the +PYTHON_FOO_SITE+ variable
+can be omitted when the package +.hash+ file contains a +blake2b-256+
+hash for the source archive. Buildroot derives the hash-based
+files.pythonhosted.org URL from that hash.
+
 Note that:
 
  * It is not necessary to add +python+ or +host-python+ in the
diff --git a/package/pkg-python.mk b/package/pkg-python.mk
index 32ace4aac1..a301b0501c 100644
--- a/package/pkg-python.mk
+++ b/package/pkg-python.mk
@@ -454,6 +454,20 @@ endif
 
 endif # host / target
 
+ifndef $(2)_SITE
+ ifdef $(3)_SITE
+  $(2)_SITE = $$($(3)_SITE)
+ else
+  $(2)_SITE = $$(strip $$(if $$(strip $$($(2)_HASH_FILES)),\
+	$$(or $$(shell \
+		awk -v filename="$$(notdir $$($(2)_SOURCE))" \
+			'$$$$1 == "blake2b-256" && $$$$3 == filename { h = $$$$2; printf "https://files.pythonhosted.org/packages/%s/%s/%s", substr(h, 1, 2), substr(h, 3, 2), substr(h, 5); exit }' \
+			$$($(2)_HASH_FILES)),\
+		$$(error $(2)_SITE must be set or $$($(2)_HASH_FILES) must contain a blake2b-256 hash for $$($(2)_SOURCE))),\
+	$$(error $(2)_SITE must be set or a hash file must exist to derive the PyPI site)))
+ endif
+endif
+
 # Call the generic package infrastructure to generate the necessary
 # make targets
 $(call inner-generic-package,$(1),$(2),$(3),$(4))
diff --git a/utils/scanpypi b/utils/scanpypi
index de658328fe..63b8f8f4d0 100755
--- a/utils/scanpypi
+++ b/utils/scanpypi
@@ -495,15 +495,6 @@ class BuildrootPackage():
                 filename=targz)
             lines.append(targz_line)
 
-        if self.filename not in self.url:
-            # Sometimes the filename is in the url, sometimes it's not
-            site_url = self.url
-        else:
-            site_url = self.url[:self.url.find(self.filename)]
-        site_line = '{name}_SITE = {url}'.format(name=self.mk_name,
-                                                 url=site_url)
-        site_line = site_line.rstrip('/') + '\n'
-        lines.append(site_line)
         return lines
 
     def __create_mk_setup(self):
-- 
2.53.0

_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

^ permalink raw reply related	[flat|nested] 4+ messages in thread

* [Buildroot] [PATCH v2 3/3] package/python-async-timeout: use hash-based PyPI site
  2026-06-17 17:42 [Buildroot] [PATCH v2 1/3] support/download: add blake2b-256 hash support James Hilliard
  2026-06-17 17:42 ` [Buildroot] [PATCH v2 2/3] package/pkg-python: derive PyPI site from blake2b-256 hash James Hilliard
@ 2026-06-17 17:42 ` James Hilliard
  2026-06-18  5:51 ` [Buildroot] [PATCH v2 1/3] support/download: add blake2b-256 hash support yann.morin
  2 siblings, 0 replies; 4+ messages in thread
From: James Hilliard @ 2026-06-17 17:42 UTC (permalink / raw)
  To: buildroot
  Cc: Thomas Petazzoni, James Hilliard, Thomas Perale,
	Ricardo Martincoski

Add the blake2b-256 digest from the PyPI metadata and drop the explicit
PYTHON_ASYNC_TIMEOUT_SITE assignment.

The Python package infrastructure can now derive the hash-based
files.pythonhosted.org site from the package .hash file.

Signed-off-by: James Hilliard <james.hilliard1@gmail.com>
---
 package/python-async-timeout/python-async-timeout.hash | 3 ++-
 package/python-async-timeout/python-async-timeout.mk   | 1 -
 2 files changed, 2 insertions(+), 2 deletions(-)

diff --git a/package/python-async-timeout/python-async-timeout.hash b/package/python-async-timeout/python-async-timeout.hash
index 451c07de5d..c812b9c6c3 100644
--- a/package/python-async-timeout/python-async-timeout.hash
+++ b/package/python-async-timeout/python-async-timeout.hash
@@ -1,5 +1,6 @@
-# md5, sha256 from https://pypi.org/pypi/async-timeout/json
+# md5, sha256, blake2b-256 from https://pypi.org/pypi/async-timeout/json
 md5  566a39011e87cb8044ee75369e69b327  async_timeout-5.0.1.tar.gz
 sha256  d9321a7a3d5a6a5e187e824d2fa0793ce379a202935782d555d6e9d2735677d3  async_timeout-5.0.1.tar.gz
+blake2b-256  a5ae136395dfbfe00dfc94da3f3e136d0b13f394cba8f4841120e34226265780  async_timeout-5.0.1.tar.gz
 # Locally computed sha256 checksums
 sha256  e18d7bb8f513e2c46bb585c94b585bd30720dd3ccb21ddb0786f72d16658f92c  LICENSE
diff --git a/package/python-async-timeout/python-async-timeout.mk b/package/python-async-timeout/python-async-timeout.mk
index 2835368003..017ea548b5 100644
--- a/package/python-async-timeout/python-async-timeout.mk
+++ b/package/python-async-timeout/python-async-timeout.mk
@@ -6,7 +6,6 @@
 
 PYTHON_ASYNC_TIMEOUT_VERSION = 5.0.1
 PYTHON_ASYNC_TIMEOUT_SOURCE = async_timeout-$(PYTHON_ASYNC_TIMEOUT_VERSION).tar.gz
-PYTHON_ASYNC_TIMEOUT_SITE = https://files.pythonhosted.org/packages/a5/ae/136395dfbfe00dfc94da3f3e136d0b13f394cba8f4841120e34226265780
 PYTHON_ASYNC_TIMEOUT_LICENSE = Apache-2.0
 PYTHON_ASYNC_TIMEOUT_LICENSE_FILES = LICENSE
 PYTHON_ASYNC_TIMEOUT_SETUP_TYPE = setuptools
-- 
2.53.0

_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

^ permalink raw reply related	[flat|nested] 4+ messages in thread

* Re: [Buildroot] [PATCH v2 1/3] support/download: add blake2b-256 hash support
  2026-06-17 17:42 [Buildroot] [PATCH v2 1/3] support/download: add blake2b-256 hash support James Hilliard
  2026-06-17 17:42 ` [Buildroot] [PATCH v2 2/3] package/pkg-python: derive PyPI site from blake2b-256 hash James Hilliard
  2026-06-17 17:42 ` [Buildroot] [PATCH v2 3/3] package/python-async-timeout: use hash-based PyPI site James Hilliard
@ 2026-06-18  5:51 ` yann.morin
  2 siblings, 0 replies; 4+ messages in thread
From: yann.morin @ 2026-06-18  5:51 UTC (permalink / raw)
  To: James Hilliard
  Cc: buildroot, Thomas Petazzoni, Thomas Perale, Ricardo Martincoski

James, All,

On 2026-06-17 11:42 -0600, James Hilliard spake thusly:
[--SNIP--]
> Teach the download hash checker to validate blake2b-256 entries using
> b2sum -l 256, and update check-package so the new hash type and length
> are accepted. Also add the CycloneDX hash spelling so generated SBOMs
> can include blake2b-256 hashes from .hash files.
[--SNIP--]
> +    if [ "${_h}" = "blake2b-256" ]; then
> +        _hash="$( b2sum -l 256 "${_file}" |cut -d ' ' -f 1 )"

b2sum was only introduced with coreutils 8.26, but we support running on
systems that have an older version: we already check that ln supports
--relative (introduced in 8.16) and that realpath exists (introduced in
8.15).

So, we probably want to add b2sum to the list of tools to check in
support/dependencies/check-host-coreutils.s and then add
$(BR2_COREUTILS_HOST_DEPENDENCY) to the dependencies of packages that
have a blake2 hash (the python infra can do so, for example).

Regards,
Yann E. MORIN.

-- 
                                        ____________
.-----------------.--------------------:       _    :------------------.
|  Yann E. MORIN  | Real-Time Embedded |    __/ )   | /"\ ASCII RIBBON |
|                 | Software  Designer |  _/ - /'   | \ / CAMPAIGN     |
| +33 638.411.245 '--------------------: (_    `--, |  X  AGAINST      |
| yann.morin (at) orange.com           |_="    ,--' | / \ HTML MAIL    |
'--------------------------------------:______/_____:------------------'

____________________________________________________________________________________________________________
Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc
pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler
a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration,
Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci.

This message and its attachments may contain confidential or privileged information that may be protected by law;
they should not be distributed, used or copied without authorisation.
If you have received this email in error, please notify the sender and delete this message and its attachments.
As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified.
Thank you.
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

^ permalink raw reply	[flat|nested] 4+ messages in thread

end of thread, other threads:[~2026-06-18  5:52 UTC | newest]

Thread overview: 4+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2026-06-17 17:42 [Buildroot] [PATCH v2 1/3] support/download: add blake2b-256 hash support James Hilliard
2026-06-17 17:42 ` [Buildroot] [PATCH v2 2/3] package/pkg-python: derive PyPI site from blake2b-256 hash James Hilliard
2026-06-17 17:42 ` [Buildroot] [PATCH v2 3/3] package/python-async-timeout: use hash-based PyPI site James Hilliard
2026-06-18  5:51 ` [Buildroot] [PATCH v2 1/3] support/download: add blake2b-256 hash support yann.morin

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox