[Buildroot] [PATCH v3 2/3] package/pkg-python: derive PyPI site from blake2b-256 hash

[James Hilliard <james.hilliard1@gmail.com>]

Python packages downloaded from PyPI currently store the complete
files.pythonhosted.org package path in their .mk file. That path is
hash-based, so it changes for every version bump and adds churn to
updates and backports.

When a Python package does not define _SITE, derive the hash-based
files.pythonhosted.org site from the blake2b-256 entry matching the
package source archive in the package .hash file before the generic
package infrastructure validates _SITE.

Follow the usual target/host fallback pattern for _SITE, and raise an
explicit error if no hash file is available or if the hash file does not
contain a matching blake2b-256 entry. Add the host coreutils dependency
for packages using this resolver so b2sum is available when needed for
hash verification.

Update scanpypi to omit generated _SITE assignments so newly generated
PyPI packages can use the resolver.

Signed-off-by: James Hilliard <james.hilliard1@gmail.com>
---
Changes v2 -> v3:
  - Add the host coreutils dependency for packages using the hash-derived
    PyPI site.

Changes v1 -> v2:
  - Follow the usual target/host _SITE fallback pattern.
  - Drop the /dev/null awk input and guard missing hash files explicitly.
  - Error out when no matching blake2b-256 entry is found.
---
 docs/manual/adding-packages-python.adoc |  5 +++++
 package/pkg-python.mk                   | 15 +++++++++++++++
 utils/scanpypi                          |  9 ---------
 3 files changed, 20 insertions(+), 9 deletions(-)

diff --git a/docs/manual/adding-packages-python.adoc b/docs/manual/adding-packages-python.adoc
index d6bae299a0..a66af1e7e2 100644
--- a/docs/manual/adding-packages-python.adoc
+++ b/docs/manual/adding-packages-python.adoc
@@ -82,6 +82,11 @@ All the package metadata information variables that exist in the
 xref:generic-package-reference[generic package infrastructure] also
 exist in the Python infrastructure.
 
+For Python packages downloaded from PyPI, the +PYTHON_FOO_SITE+ variable
+can be omitted when the package +.hash+ file contains a +blake2b-256+
+hash for the source archive. Buildroot derives the hash-based
+files.pythonhosted.org URL from that hash.
+
 Note that:
 
  * It is not necessary to add +python+ or +host-python+ in the
diff --git a/package/pkg-python.mk b/package/pkg-python.mk
index 32ace4aac1..9ffb26b430 100644
--- a/package/pkg-python.mk
+++ b/package/pkg-python.mk
@@ -454,6 +454,21 @@ endif
 
 endif # host / target
 
+ifndef $(2)_SITE
+ ifdef $(3)_SITE
+  $(2)_SITE = $$($(3)_SITE)
+ else
+  $(2)_DOWNLOAD_DEPENDENCIES += $(BR2_COREUTILS_HOST_DEPENDENCY)
+  $(2)_SITE = $$(strip $$(if $$(strip $$($(2)_HASH_FILES)),\
+	$$(or $$(shell \
+		awk -v filename="$$(notdir $$($(2)_SOURCE))" \
+			'$$$$1 == "blake2b-256" && $$$$3 == filename { h = $$$$2; printf "https://files.pythonhosted.org/packages/%s/%s/%s", substr(h, 1, 2), substr(h, 3, 2), substr(h, 5); exit }' \
+			$$($(2)_HASH_FILES)),\
+		$$(error $(2)_SITE must be set or $$($(2)_HASH_FILES) must contain a blake2b-256 hash for $$($(2)_SOURCE))),\
+	$$(error $(2)_SITE must be set or a hash file must exist to derive the PyPI site)))
+ endif
+endif
+
 # Call the generic package infrastructure to generate the necessary
 # make targets
 $(call inner-generic-package,$(1),$(2),$(3),$(4))
diff --git a/utils/scanpypi b/utils/scanpypi
index de658328fe..63b8f8f4d0 100755
--- a/utils/scanpypi
+++ b/utils/scanpypi
@@ -495,15 +495,6 @@ class BuildrootPackage():
                 filename=targz)
             lines.append(targz_line)
 
-        if self.filename not in self.url:
-            # Sometimes the filename is in the url, sometimes it's not
-            site_url = self.url
-        else:
-            site_url = self.url[:self.url.find(self.filename)]
-        site_line = '{name}_SITE = {url}'.format(name=self.mk_name,
-                                                 url=site_url)
-        site_line = site_line.rstrip('/') + '\n'
-        lines.append(site_line)
         return lines
 
     def __create_mk_setup(self):
-- 
2.53.0

_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot