[Buildroot] [RFC PATCH 0/2] support/scripts/pkg-stats: add support for version prefixes

[Buildroot] [RFC PATCH 0/2] support/scripts/pkg-stats: add support for version prefixes

[Raphaël Mélotte via buildroot]

I started this series as an attempt at solving pkg-stats' issue with
packages using version prefixes.
Currently pkg-stats reports these packages as outdated, even when they
actually are up to date.

In the meantime I realized that a different approach was taken in 2019
(see [1]), so it's probably a good idea to follow that route
instead. I'm still sending the series in case in can be of any use in
the future or serve as a basis for new discussions.

A new per-package variable is introduced to configure the prefix for
each package, and a second commit configures it for every packages
that uses 'v' as a prefix.

This approach is not ideal as it requires a new per-package variable
to be introduced, but it gives us the flexibility to configure
different prefixes for different packages if needed while avoiding the
two copies in DL_DIR mentioned in [2].

[1]: https://lists.buildroot.org/pipermail/buildroot/2019-June/617217.html
[2]: https://lists.buildroot.org/pipermail/buildroot/2019-June/617575.html

Raphaël Mélotte (2):
  support/scripts/pkg-stats: add support for version prefixes
  package: add *_VERSION_PREFIX to packages using prefix 'v'

 docs/manual/adding-packages-generic.adoc         |  5 +++++
 package/aardvark-dns/aardvark-dns.mk             |  1 +
 .../amazon-ecr-credential-helper.mk              |  1 +
 package/bcc/bcc.mk                               |  1 +
 package/bpftool/bpftool.mk                       |  1 +
 package/brickd/brickd.mk                         |  1 +
 package/cutekeyboard/cutekeyboard.mk             |  1 +
 .../docker-credential-gcr.mk                     |  1 +
 package/embiggen-disk/embiggen-disk.mk           |  1 +
 .../gstreamer1/gst1-interpipe/gst1-interpipe.mk  |  1 +
 package/gstreamer1/gst1-shark/gst1-shark.mk      |  1 +
 package/libcamera/libcamera.mk                   |  1 +
 package/netavark/netavark.mk                     |  1 +
 package/ogre/ogre.mk                             |  1 +
 package/open62541/open62541.mk                   |  1 +
 package/podman/podman.mk                         |  1 +
 package/python-web2py/python-web2py.mk           |  1 +
 package/signal-estimator/signal-estimator.mk     |  1 +
 .../spirv-llvm-translator.mk                     |  1 +
 package/tl-expected/tl-expected.mk               |  1 +
 support/scripts/pkg-stats                        | 16 ++++++++++++++--
 21 files changed, 38 insertions(+), 2 deletions(-)

-- 
2.49.0

_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

[Buildroot] [RFC PATCH 1/2] support/scripts/pkg-stats: add support for version prefixes

[Raphaël Mélotte via buildroot]

Packages using a version prefix are problematic for pkg-stats, because
release-monitoring.org doesn't include the prefix in its versions.  As
a consequence, pkg-stat reports as outdated packages that actually
aren't.

Ideally release-monitoring.org would give us the prefix used by the
package since it stores this information per-package. Unfortunately,
the prefix is not part of the information we get from its API.

We cannot blindly remove prefixes from every package because some of
them use commit hashes (so for example a prefix 'v' to be removed
cannot be distinguished from the letter 'v' being the beginning of a
commit hash).

Instead, this commit introduces support for a new
'<PKG>_VERSION_PREFIX', which can be used to indicate the prefix used
by the package.
pkg-stats consumes this variable and removes the prefix from the
version when reporting or comparing versions.

The documentation is also updated accordingly.

Signed-off-by: Raphaël Mélotte <raphael.melotte@mind.be>
---
 docs/manual/adding-packages-generic.adoc |  5 +++++
 support/scripts/pkg-stats                | 16 ++++++++++++++--
 2 files changed, 19 insertions(+), 2 deletions(-)

diff --git a/docs/manual/adding-packages-generic.adoc b/docs/manual/adding-packages-generic.adoc
index 7e2e066c16..c0fbb83d55 100644
--- a/docs/manual/adding-packages-generic.adoc
+++ b/docs/manual/adding-packages-generic.adoc
@@ -219,6 +219,11 @@ not and can not work as people would expect it should:
      non reproducible, and people would be quite surprised and
      disappointed.
 
+* +LIBFOO_VERSION_PREFIX+ may contain the version prefix used by the
+  package. +pkg-stats+ uses this prefix when checking for the latest
+  version of packages.
+  Example: +LIBFOO_VERSION_PREFIX = v+
+
 * +LIBFOO_SOURCE+ may contain the name of the tarball of the package,
   which Buildroot will use to download the tarball from
   +LIBFOO_SITE+. If +HOST_LIBFOO_SOURCE+ is not specified, it defaults
diff --git a/support/scripts/pkg-stats b/support/scripts/pkg-stats
index c134e1ec06..f427cccc21 100755
--- a/support/scripts/pkg-stats
+++ b/support/scripts/pkg-stats
@@ -89,6 +89,7 @@ class Package:
     all_licenses = dict()
     all_license_files = list()
     all_versions = dict()
+    all_versions_prefixes = dict()
     all_ignored_cves = dict()
     all_cpeids = dict()
     # This is the list of all possible checks. Add new checks to this list so
@@ -258,10 +259,16 @@ class Package:
     def set_current_version(self):
         """
         Fills in the .current_version field
+        If needed, filter out the version prefix
         """
         var = self.pkgvar()
         if var in self.all_versions:
-            self.current_version = self.all_versions[var]
+            version = self.all_versions[var]
+
+            if var in self.all_versions_prefixes:
+                version = version.removeprefix(self.all_versions_prefixes[var])
+
+            self.current_version = version
 
     def set_cpeid(self):
         """
@@ -418,7 +425,8 @@ def package_init_make_info():
     # Fetch all variables at once
     variables = subprocess.check_output(["make", "--no-print-directory", "-s",
                                          "BR2_HAVE_DOT_CONFIG=y", "printvars",
-                                         "VARS=%_LICENSE %_LICENSE_FILES %_VERSION %_IGNORE_CVES %_CPE_ID"])
+                                         "VARS=%_LICENSE %_LICENSE_FILES %_VERSION %VERSION_PREFIX \
+                                         %_IGNORE_CVES %_CPE_ID"])
     variable_list = variables.decode().splitlines()
 
     # We process first the host package VERSION, and then the target
@@ -452,6 +460,10 @@ def package_init_make_info():
             pkgvar = pkgvar[:-8]
             Package.all_versions[pkgvar] = value
 
+        elif pkgvar.endswith("_VERSION_PREFIX"):
+            pkgvar = pkgvar[:-15]
+            Package.all_versions_prefixes[pkgvar] = value
+
         elif pkgvar.endswith("_IGNORE_CVES"):
             pkgvar = pkgvar[:-12]
             Package.all_ignored_cves[pkgvar] = value.split()
-- 
2.49.0

_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

[Buildroot] [RFC PATCH 2/2] package: add *_VERSION_PREFIX to packages using prefix 'v'

[Raphaël Mélotte via buildroot]

This makes use of the new variable introduced in pkg-stats in the
previous commit for every package using 'v' as a prefix.

With these changes, pkg-stats now correctly reports the affected
packages as up to date when they are.

Signed-off-by: Raphaël Mélotte <raphael.melotte@mind.be>
---
 package/aardvark-dns/aardvark-dns.mk                             | 1 +
 .../amazon-ecr-credential-helper/amazon-ecr-credential-helper.mk | 1 +
 package/bcc/bcc.mk                                               | 1 +
 package/bpftool/bpftool.mk                                       | 1 +
 package/brickd/brickd.mk                                         | 1 +
 package/cutekeyboard/cutekeyboard.mk                             | 1 +
 package/docker-credential-gcr/docker-credential-gcr.mk           | 1 +
 package/embiggen-disk/embiggen-disk.mk                           | 1 +
 package/gstreamer1/gst1-interpipe/gst1-interpipe.mk              | 1 +
 package/gstreamer1/gst1-shark/gst1-shark.mk                      | 1 +
 package/libcamera/libcamera.mk                                   | 1 +
 package/netavark/netavark.mk                                     | 1 +
 package/ogre/ogre.mk                                             | 1 +
 package/open62541/open62541.mk                                   | 1 +
 package/podman/podman.mk                                         | 1 +
 package/python-web2py/python-web2py.mk                           | 1 +
 package/signal-estimator/signal-estimator.mk                     | 1 +
 package/spirv-llvm-translator/spirv-llvm-translator.mk           | 1 +
 package/tl-expected/tl-expected.mk                               | 1 +
 19 files changed, 19 insertions(+)

diff --git a/package/aardvark-dns/aardvark-dns.mk b/package/aardvark-dns/aardvark-dns.mk
index a2e74a72bf..02fdfacb84 100644
--- a/package/aardvark-dns/aardvark-dns.mk
+++ b/package/aardvark-dns/aardvark-dns.mk
@@ -6,6 +6,7 @@
 
 # When updating the version here, also update netavark in lockstep
 AARDVARK_DNS_VERSION = v1.14.0
+AARDVARK_DNS_VERSION_PREFIX = v
 AARDVARK_DNS_SITE = https://github.com/containers/aardvark-dns
 AARDVARK_DNS_SITE_METHOD = git
 
diff --git a/package/amazon-ecr-credential-helper/amazon-ecr-credential-helper.mk b/package/amazon-ecr-credential-helper/amazon-ecr-credential-helper.mk
index acfeea16fb..64a70ce48e 100644
--- a/package/amazon-ecr-credential-helper/amazon-ecr-credential-helper.mk
+++ b/package/amazon-ecr-credential-helper/amazon-ecr-credential-helper.mk
@@ -5,6 +5,7 @@
 ################################################################################
 
 AMAZON_ECR_CREDENTIAL_HELPER_VERSION = v0.9.1
+AMAZON_ECR_CREDENTIAL_HELPER_VERSION_PREFIX = v
 AMAZON_ECR_CREDENTIAL_HELPER_SITE = https://github.com/awslabs/amazon-ecr-credential-helper
 AMAZON_ECR_CREDENTIAL_HELPER_SITE_METHOD = git
 
diff --git a/package/bcc/bcc.mk b/package/bcc/bcc.mk
index d5d72aa9c0..df5fbefe34 100644
--- a/package/bcc/bcc.mk
+++ b/package/bcc/bcc.mk
@@ -5,6 +5,7 @@
 ################################################################################
 
 BCC_VERSION = v0.31.0
+BCC_VERSION_PREFIX = v
 BCC_SITE = https://github.com/iovisor/bcc
 BCC_SITE_METHOD = git
 BCC_GIT_SUBMODULES = YES
diff --git a/package/bpftool/bpftool.mk b/package/bpftool/bpftool.mk
index cd68d89799..303b9872c9 100644
--- a/package/bpftool/bpftool.mk
+++ b/package/bpftool/bpftool.mk
@@ -5,6 +5,7 @@
 ################################################################################
 
 BPFTOOL_VERSION = v7.1.0
+BPFTOOL_VERSION_PREFIX = v
 BPFTOOL_SITE = https://github.com/libbpf/bpftool
 BPFTOOL_SITE_METHOD = git
 BPFTOOL_GIT_SUBMODULES = YES
diff --git a/package/brickd/brickd.mk b/package/brickd/brickd.mk
index f42b3c9956..b2e274feef 100644
--- a/package/brickd/brickd.mk
+++ b/package/brickd/brickd.mk
@@ -5,6 +5,7 @@
 ################################################################################
 
 BRICKD_VERSION = v1.2.2
+BRICKD_VERSION_PREFIX = v
 BRICKD_SITE = https://github.com/ev3dev/brickd
 BRICKD_SITE_METHOD = git
 BRICKD_GIT_SUBMODULES = YES
diff --git a/package/cutekeyboard/cutekeyboard.mk b/package/cutekeyboard/cutekeyboard.mk
index 775bed0fe9..5aa064ac2d 100644
--- a/package/cutekeyboard/cutekeyboard.mk
+++ b/package/cutekeyboard/cutekeyboard.mk
@@ -5,6 +5,7 @@
 ################################################################################
 
 CUTEKEYBOARD_VERSION = v1.3.0
+CUTEKEYBOARD_VERSION_PREFIX = v
 CUTEKEYBOARD_SITE = $(call github,amarula,cutekeyboard,$(CUTEKEYBOARD_VERSION))
 CUTEKEYBOARD_DEPENDENCIES = qt5declarative qt5quickcontrols2
 CUTEKEYBOARD_INSTALL_STAGING = YES
diff --git a/package/docker-credential-gcr/docker-credential-gcr.mk b/package/docker-credential-gcr/docker-credential-gcr.mk
index 658e9baf33..4a0c9aa9b5 100644
--- a/package/docker-credential-gcr/docker-credential-gcr.mk
+++ b/package/docker-credential-gcr/docker-credential-gcr.mk
@@ -5,6 +5,7 @@
 ################################################################################
 
 DOCKER_CREDENTIAL_GCR_VERSION = v2.1.26
+DOCKER_CREDENTIAL_GCR_VERSION_PREFIX = v
 DOCKER_CREDENTIAL_GCR_SITE = https://github.com/GoogleCloudPlatform/docker-credential-gcr
 DOCKER_CREDENTIAL_GCR_SITE_METHOD = git
 
diff --git a/package/embiggen-disk/embiggen-disk.mk b/package/embiggen-disk/embiggen-disk.mk
index 47b86c072b..d5228e20a2 100644
--- a/package/embiggen-disk/embiggen-disk.mk
+++ b/package/embiggen-disk/embiggen-disk.mk
@@ -5,6 +5,7 @@
 ################################################################################
 
 EMBIGGEN_DISK_VERSION = v20240521
+EMBIGGEN_DISK_VERSION_PREFIX = v
 EMBIGGEN_DISK_SITE = $(call github,skiffos,embiggen-disk,$(EMBIGGEN_DISK_VERSION))
 EMBIGGEN_DISK_LICENSE = Apache-2.0
 EMBIGGEN_DISK_LICENSE_FILES = LICENSE
diff --git a/package/gstreamer1/gst1-interpipe/gst1-interpipe.mk b/package/gstreamer1/gst1-interpipe/gst1-interpipe.mk
index cec23e7fa7..1ddb6d3d44 100644
--- a/package/gstreamer1/gst1-interpipe/gst1-interpipe.mk
+++ b/package/gstreamer1/gst1-interpipe/gst1-interpipe.mk
@@ -5,6 +5,7 @@
 ################################################################################
 
 GST1_INTERPIPE_VERSION = v1.1.8
+GST1_INTERPIPE_VERSION_PREFIX = v
 GST1_INTERPIPE_SITE = https://github.com/RidgeRun/gst-interpipe
 GST1_INTERPIPE_SITE_METHOD = git
 # fetch gst-interpipe/common sub module
diff --git a/package/gstreamer1/gst1-shark/gst1-shark.mk b/package/gstreamer1/gst1-shark/gst1-shark.mk
index a059d41cb8..65aba07713 100644
--- a/package/gstreamer1/gst1-shark/gst1-shark.mk
+++ b/package/gstreamer1/gst1-shark/gst1-shark.mk
@@ -5,6 +5,7 @@
 ################################################################################
 
 GST1_SHARK_VERSION = v0.8.1
+GST1_SHARK_VERSION_PREFIX = v
 GST1_SHARK_SITE =  https://github.com/RidgeRun/gst-shark.git
 GST1_SHARK_SITE_METHOD = git
 GST1_SHARK_GIT_SUBMODULES = YES
diff --git a/package/libcamera/libcamera.mk b/package/libcamera/libcamera.mk
index d732424015..89132e7ecc 100644
--- a/package/libcamera/libcamera.mk
+++ b/package/libcamera/libcamera.mk
@@ -6,6 +6,7 @@
 
 LIBCAMERA_SITE = https://git.linuxtv.org/libcamera.git
 LIBCAMERA_VERSION = v0.5.0
+LIBCAMERA_VERSION_PREFIX = v
 LIBCAMERA_SITE_METHOD = git
 LIBCAMERA_DEPENDENCIES = \
 	host-openssl \
diff --git a/package/netavark/netavark.mk b/package/netavark/netavark.mk
index 31d0b28b74..c44b37cdf9 100644
--- a/package/netavark/netavark.mk
+++ b/package/netavark/netavark.mk
@@ -6,6 +6,7 @@
 
 # When updating the version here, also update aardvark-dns in lockstep
 NETAVARK_VERSION = v1.14.0
+NETAVARK_VERSION_PREFIX = v
 NETAVARK_SITE = https://github.com/containers/netavark
 NETAVARK_SITE_METHOD = git
 
diff --git a/package/ogre/ogre.mk b/package/ogre/ogre.mk
index 3fd2fc4650..c4aa0c1ebb 100644
--- a/package/ogre/ogre.mk
+++ b/package/ogre/ogre.mk
@@ -5,6 +5,7 @@
 ################################################################################
 
 OGRE_VERSION = v1.12.12
+OGRE_VERSION_PREFIX = v
 OGRE_SITE = https://github.com/OGRECave/ogre
 OGRE_SITE_METHOD = git
 OGRE_LICENSE = MIT (main library, DeferredShadingMedia samples), Public Domain (samples and plugins)
diff --git a/package/open62541/open62541.mk b/package/open62541/open62541.mk
index 0d1abf724a..a817481fef 100644
--- a/package/open62541/open62541.mk
+++ b/package/open62541/open62541.mk
@@ -5,6 +5,7 @@
 ################################################################################
 
 OPEN62541_VERSION = v1.3.15
+OPEN62541_VERSION_PREFIX = v
 OPEN62541_SITE_METHOD = git
 OPEN62541_SITE = https://github.com/open62541/open62541.git
 OPEN62541_GIT_SUBMODULES = YES
diff --git a/package/podman/podman.mk b/package/podman/podman.mk
index 7ca9b9ddcb..59b2fb9603 100644
--- a/package/podman/podman.mk
+++ b/package/podman/podman.mk
@@ -5,6 +5,7 @@
 ################################################################################
 
 PODMAN_VERSION = v5.4.1
+PODMAN_VERSION_PREFIX = v
 PODMAN_SITE = https://github.com/containers/podman
 PODMAN_SITE_METHOD = git
 
diff --git a/package/python-web2py/python-web2py.mk b/package/python-web2py/python-web2py.mk
index 9e1de5c932..7f998b1b6b 100644
--- a/package/python-web2py/python-web2py.mk
+++ b/package/python-web2py/python-web2py.mk
@@ -5,6 +5,7 @@
 ################################################################################
 
 PYTHON_WEB2PY_VERSION = v3.0.11
+PYTHON_WEB2PY_VERSION_PREFIX = v
 PYTHON_WEB2PY_SITE = https://github.com/web2py/web2py.git
 PYTHON_WEB2PY_SITE_METHOD = git
 PYTHON_WEB2PY_GIT_SUBMODULES = YES
diff --git a/package/signal-estimator/signal-estimator.mk b/package/signal-estimator/signal-estimator.mk
index cb1594b58d..3b5069d185 100644
--- a/package/signal-estimator/signal-estimator.mk
+++ b/package/signal-estimator/signal-estimator.mk
@@ -5,6 +5,7 @@
 ################################################################################
 
 SIGNAL_ESTIMATOR_VERSION = v0.0.4
+SIGNAL_ESTIMATOR_VERSION_PREFIX = v
 SIGNAL_ESTIMATOR_SITE = https://github.com/gavv/signal-estimator
 SIGNAL_ESTIMATOR_SITE_METHOD = git
 SIGNAL_ESTIMATOR_GIT_SUBMODULES = YES
diff --git a/package/spirv-llvm-translator/spirv-llvm-translator.mk b/package/spirv-llvm-translator/spirv-llvm-translator.mk
index 523890a986..548087f862 100644
--- a/package/spirv-llvm-translator/spirv-llvm-translator.mk
+++ b/package/spirv-llvm-translator/spirv-llvm-translator.mk
@@ -7,6 +7,7 @@
 # Generate version string using:
 #   git describe --tags --match 'v15*' --abbrev=40 origin/llvm_release_150
 SPIRV_LLVM_TRANSLATOR_VERSION = v15.0.0-46-ge82ecc2bd7295604fcf1824e47c95fa6a09c6e63
+SPIRV_LLVM_TRANSLATOR_VERSION_PREFIX = v
 SPIRV_LLVM_TRANSLATOR_SITE = $(call github,KhronosGroup,SPIRV-LLVM-Translator,$(SPIRV_LLVM_TRANSLATOR_VERSION))
 SPIRV_LLVM_TRANSLATOR_LICENSE = NCSA
 SPIRV_LLVM_TRANSLATOR_LICENSE_FILES = LICENSE.TXT
diff --git a/package/tl-expected/tl-expected.mk b/package/tl-expected/tl-expected.mk
index 8d62c75fea..966fa826de 100644
--- a/package/tl-expected/tl-expected.mk
+++ b/package/tl-expected/tl-expected.mk
@@ -5,6 +5,7 @@
 ################################################################################
 
 TL_EXPECTED_VERSION = v1.0.0
+TL_EXPECTED_VERSION_PREFIX = v
 TL_EXPECTED_SITE = https://github.com/TartanLlama/expected
 TL_EXPECTED_SITE_METHOD = git
 TL_EXPECTED_GIT_SUBMODULES = YES
-- 
2.49.0

_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [RFC PATCH 2/2] package: add *_VERSION_PREFIX to packages using prefix 'v'

[Thomas Petazzoni via buildroot]

Hello Raphaël,

+Yann, Arnout in Cc.

On Wed, 23 Apr 2025 15:19:30 +0200
Raphaël Mélotte via buildroot <buildroot@buildroot.org> wrote:

> diff --git a/package/aardvark-dns/aardvark-dns.mk b/package/aardvark-dns/aardvark-dns.mk
> index a2e74a72bf..02fdfacb84 100644
> --- a/package/aardvark-dns/aardvark-dns.mk
> +++ b/package/aardvark-dns/aardvark-dns.mk
> @@ -6,6 +6,7 @@
>  
>  # When updating the version here, also update netavark in lockstep
>  AARDVARK_DNS_VERSION = v1.14.0
> +AARDVARK_DNS_VERSION_PREFIX = v

Thanks for this proposed change. We know this has been an issue for
quite some time, but I'm not sure your proposed solution is the one we
want (ideally, of course).

Indeed, there is already a <pkg>_DL_VERSION variable defined by the
package infrastructure, which we ideally would like to leverage, so
that we could write:

<foo>_VERSION = 1.14.0
<foo>_DL_VERSION = v$(<foo>_VERSION)

And of course, when <foo>_DL_VERSION is undefined, the infra assumes
that its value is the one of <foo>_VERSION.

I tried implementing this a long time ago:

  https://gitlab.com/tpetazzoni/buildroot/-/commit/a0b700fb08ca84b373247f3a52755fd4e8997d22

And I wrote some tests to verify the correct behavior:

  https://gitlab.com/tpetazzoni/buildroot/-/commit/3e32feb64135d5e4347f2a18e1648398efc1a6b1

but no matter what I tried, I couldn't find a solution that was passing
all tests.

I remember we discussed with with Yann and/or Arnout quite some time
ago, and I believe we reached some conclusion on how it should be
implemented, but of course I forgot what this conclusion was. Maybe
Yann and/or Arnout will have a better memory than I do?

Best regards,

Thomas
-- 
Thomas Petazzoni, co-owner and CEO, Bootlin
Embedded Linux and Kernel engineering and training
https://bootlin.com
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot