[Buildroot] [PATCH v2 0/6] Bugfixes for pkg-stats

[Buildroot] [PATCH v2 0/6] Bugfixes for pkg-stats

[Fiona Klute via buildroot]

Hi everyone,

this series is a collection of small bug fixes for
support/scripts/pkg-stats, fixing bugs I bumped into checking a fairly
simple build with it:

* Incorrect upstream URLs (most notably for the Linux kernel, and I
  had a wrong one for a package in an external tree due to an Emacs
  backup file), patch 1 + 3

* Entries for *missing* upstream URL would still be links with
  href="None", patch 2

* The "infrastructure" column would list "host-* (target)" when a
  package that supports a host build is actually built only for the
  target, patch 4

Patches 5 and 6 are small fixes for the script itself (don't
unnecessarily load whole package files, allow importing the script as
a module).

I see a bunch more things that should be fixed (notably virtual
packages reporting missing license/hash info when there simply is
none, and the package parsing process itself), this is what I had time
for.

One question for possible future improvements: Is the JSON output
format documented/defined anywhere, or simply "what the script
produces, subject to change"?

Best regards,
Fiona


Changes v1 -> v2:
* Avoid glob in patch 1
* Add patches 4-6

v1: https://patchwork.ozlabs.org/project/buildroot/list/?series=501442&state=*

Fiona Klute (6):
  support/scripts/pkg-stats: search only Config.in{,.host} for URL
  support/scripts/pkg-stats: format upstream URL info consistently in
    HTML
  linux/Config.in: add kernel.org URL as BR2_LINUX_KERNEL_LATEST_VERSION
    help
  support/scripts/pkg-stats: fix host/target infra filter
  support/scripts/pkg-stats: don't buffer whole file searching for infra
  support/scripts/pkg-stats: run main function only if called as script

 linux/Config.in           |  2 ++
 support/scripts/pkg-stats | 54 ++++++++++++++++++++++-----------------
 2 files changed, 32 insertions(+), 24 deletions(-)

-- 
2.53.0

_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

[Buildroot] [PATCH v2 1/6] support/scripts/pkg-stats: search only Config.in{, .host} for URL

[Fiona Klute via buildroot]

The previous Config.* glob also caught linux/Config.ext.in and
package/php/Config.ext, as well as some backup files created by
editors (e.g. Config.in~ after editing a Config.in file in Emacs),
leading to wrong results depending on directory listing order.

Also use "with" to automatically close the file when the block is
left, even on error.

Signed-off-by: Fiona Klute <fiona.klute@gmx.de>
---
This cannot solve the problem we have no structured URL information
and don't look at which option the URL is listed under, but at least
it's less wrong.

Changes v1 -> v2:
* Try fixed names instead of globbing all directory entries

 support/scripts/pkg-stats | 22 +++++++++++-----------
 1 file changed, 11 insertions(+), 11 deletions(-)

diff --git a/support/scripts/pkg-stats b/support/scripts/pkg-stats
index 55aa63c861..efb85a7405 100755
--- a/support/scripts/pkg-stats
+++ b/support/scripts/pkg-stats
@@ -148,17 +148,17 @@ class Package:
         Fills in the .url field
         """
         self.status['url'] = ("warning", "no Config.in")
-        for filename in os.listdir(self.pkgdir):
-            if fnmatch.fnmatch(filename, 'Config.*'):
-                fp = open(os.path.join(self.pkgdir, filename), "r")
-                for config_line in fp:
-                    if URL_RE.match(config_line):
-                        self.url = config_line.strip()
-                        self.status['url'] = ("ok", "found")
-                        fp.close()
-                        return
-                self.status['url'] = ("error", "missing")
-                fp.close()
+        for filename in ('Config.in', 'Config.in.host'):
+            try:
+                with open(os.path.join(self.pkgdir, filename), "r") as fp:
+                    for config_line in fp:
+                        if URL_RE.match(config_line):
+                            self.url = config_line.strip()
+                            self.status['url'] = ("ok", "found")
+                            return
+                    self.status['url'] = ("error", "missing")
+            except FileNotFoundError:
+                continue
 
     @property
     def patch_count(self):
-- 
2.53.0

_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

[Buildroot] [PATCH v2 2/6] support/scripts/pkg-stats: format upstream URL info consistently in HTML

[Fiona Klute via buildroot]

Use only one of the classes for "error" or "warning" status so they
look different, and format the error/warning text for both. Do not
make the text a link if the URL is None.

Signed-off-by: Fiona Klute <fiona.klute@gmx.de>
---
 support/scripts/pkg-stats | 17 +++++++++++------
 1 file changed, 11 insertions(+), 6 deletions(-)

diff --git a/support/scripts/pkg-stats b/support/scripts/pkg-stats
index efb85a7405..a87bb23a8e 100755
--- a/support/scripts/pkg-stats
+++ b/support/scripts/pkg-stats
@@ -1057,14 +1057,19 @@ def dump_html_pkg(f, pkg):
     div_class = ["centered upstream_url data"]
     div_class.append(f'_{pkg_css_class}')
     url_str = pkg.status['url'][1]
-    if pkg.status['url'][0] in ("error", "warning"):
-        div_class.append("missing_url")
-    if pkg.status['url'][0] == "error":
-        div_class.append("invalid_url")
-        url_str = f"""<a href="{pkg.url}">{pkg.status['url'][1]}</a>"""
-    else:
+    if pkg.status['url'][0] == "ok":
         div_class.append("good_url")
         url_str = f'<a href="{pkg.url}">Link</a>'
+    else:
+        if pkg.status['url'][0] == "warning":
+            div_class.append("missing_url")
+        else:
+            # "error" status
+            div_class.append("invalid_url")
+        if pkg.url is not None:
+            url_str = f"""<a href="{pkg.url}">{pkg.status['url'][1]}</a>"""
+        else:
+            url_str = pkg.status['url'][1]
     f.write(f'  <div id="{data_field_id}" class="{" ".join(div_class)}">{url_str}</div>\n')
 
     # CVEs
-- 
2.53.0

_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

[Buildroot] [PATCH v2 3/6] linux/Config.in: add kernel.org URL as BR2_LINUX_KERNEL_LATEST_VERSION help

[Fiona Klute via buildroot]

This makes it the first URL in the file, so it shows up correctly in
pkg-stats output.

Signed-off-by: Fiona Klute <fiona.klute@gmx.de>
---
 linux/Config.in | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/linux/Config.in b/linux/Config.in
index 8633ddbc82..1104902b4b 100644
--- a/linux/Config.in
+++ b/linux/Config.in
@@ -33,6 +33,8 @@ config BR2_LINUX_KERNEL_LATEST_VERSION
 	# mips always generates an ITB image
 	select BR2_PACKAGE_HOST_UBOOT_TOOLS if BR2_mips || BR2_mipsel || BR2_mips64 || BR2_mips64el
 	select BR2_PACKAGE_HOST_UBOOT_TOOLS_FIT_SUPPORT if BR2_mips || BR2_mipsel || BR2_mips64 || BR2_mips64el
+	help
+	  https://www.kernel.org/
 
 config BR2_LINUX_KERNEL_LATEST_CIP_VERSION
 	bool "Latest CIP SLTS version (5.10.246-cip66)"
-- 
2.53.0

_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

[Buildroot] [PATCH v2 4/6] support/scripts/pkg-stats: fix host/target infra filter

[Fiona Klute via buildroot]

The filter is supposed to exclude host/target infra from output if the
respective package is not built with the current
configuration.

However, excluding host packages did not work correctly: If keep_host
is False because the host package is not built, the next branch was
checked and included the host infra in output with "target" type if
the target package is built. For a package that support host and
target build, but gets built only for the target, this leads to output
like (Meson example):

meson (target)
host-meson (target)

Skip host infra in the target branch instead. Also include
Package.infra in Package.__str__() result, which was needed for
debugging this bug.

Signed-off-by: Fiona Klute <fiona.klute@gmx.de>
---
 support/scripts/pkg-stats | 7 ++++---
 1 file changed, 4 insertions(+), 3 deletions(-)

diff --git a/support/scripts/pkg-stats b/support/scripts/pkg-stats
index a87bb23a8e..bbd1e3e6f0 100755
--- a/support/scripts/pkg-stats
+++ b/support/scripts/pkg-stats
@@ -205,7 +205,7 @@ class Package:
                 infra = match.group(1)
                 if infra.startswith("host-") and keep_host:
                     self.infras.append(("host", infra[5:]))
-                elif keep_target:
+                elif not infra.startswith("host-") and keep_target:
                     self.infras.append(("target", infra))
 
     def set_license(self):
@@ -342,9 +342,10 @@ class Package:
         return self.path < other.path
 
     def __str__(self):
-        return "%s (path='%s', license='%s', license_files='%s', hash='%s', patches=%d)" % \
+        return "%s (path='%s', license='%s', license_files='%s', hash='%s', patches=%d, infras=%r)" % \
             (self.name, self.path, self.is_status_ok('license'),
-             self.is_status_ok('license-files'), self.status['hash'], self.patch_count)
+             self.is_status_ok('license-files'), self.status['hash'], self.patch_count,
+             self.infras)
 
 
 def get_pkglist(trees, npackages, package_list):
-- 
2.53.0

_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

[Buildroot] [PATCH v2 5/6] support/scripts/pkg-stats: don't buffer whole file searching for infra

[Fiona Klute via buildroot]

The file handle can be iterated over directly and each line is used
exactly once, so the only effect of reading all lines into a list
first was higher memory use and complexity.

Signed-off-by: Fiona Klute <fiona.klute@gmx.de>
---
 support/scripts/pkg-stats | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)

diff --git a/support/scripts/pkg-stats b/support/scripts/pkg-stats
index bbd1e3e6f0..5dcc812e2b 100755
--- a/support/scripts/pkg-stats
+++ b/support/scripts/pkg-stats
@@ -197,8 +197,7 @@ class Package:
 
         self.infras = list()
         with open(self.pkgfile, 'r') as f:
-            lines = f.readlines()
-            for line in lines:
+            for line in f:
                 match = INFRA_RE.match(line)
                 if not match:
                     continue
-- 
2.53.0

_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

[Buildroot] [PATCH v2 6/6] support/scripts/pkg-stats: run main function only if called as script

[Fiona Klute via buildroot]

The __name__ == '__main__' guard allows importing pkg-stats as a
module using importlib, circumventing the normal module filename
requirements. This in turn makes it possible to test/debug individual
functions.

Signed-off-by: Fiona Klute <fiona.klute@gmx.de>
---
 support/scripts/pkg-stats | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/support/scripts/pkg-stats b/support/scripts/pkg-stats
index 5dcc812e2b..2b2194b9a5 100755
--- a/support/scripts/pkg-stats
+++ b/support/scripts/pkg-stats
@@ -1334,7 +1334,7 @@ def is_newer_version_available(pkg):
     )
 
 
-def __main__():
+def main():
     global cvecheck
 
     args = parse_args()
@@ -1403,4 +1403,5 @@ def __main__():
         dump_json(packages, defconfigs, stats, date, commit, args.json)
 
 
-__main__()
+if __name__ == '__main__':
+    main()
-- 
2.53.0

_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH v2 0/6] Bugfixes for pkg-stats

[Arnout Vandecappelle via buildroot]

  Hi Fiona,

  I'm making an attempt to get this series merged :-)

  As a preliminary remark though: pkg-stats really should evolve to using 
show-info output instead of parsing files itself - that would solve the things 
like the incorrect patches and hash files. There's one big blocker for that 
though - pkg-stats supposed to be able to work without a .config, and show-info 
currently doesn't.

On 26/04/2026 17:33, Fiona Klute via buildroot wrote:
> Hi everyone,
> 
> this series is a collection of small bug fixes for
> support/scripts/pkg-stats, fixing bugs I bumped into checking a fairly
> simple build with it:
> 
> * Incorrect upstream URLs (most notably for the Linux kernel, and I
>    had a wrong one for a package in an external tree due to an Emacs
>    backup file), patch 1 + 3
> 
> * Entries for *missing* upstream URL would still be links with
>    href="None", patch 2
> 
> * The "infrastructure" column would list "host-* (target)" when a
>    package that supports a host build is actually built only for the
>    target, patch 4
> 
> Patches 5 and 6 are small fixes for the script itself (don't
> unnecessarily load whole package files, allow importing the script as
> a module).
> 
> I see a bunch more things that should be fixed (notably virtual
> packages reporting missing license/hash info when there simply is
> none, and the package parsing process itself), this is what I had time
> for.
> 
> One question for possible future improvements: Is the JSON output
> format documented/defined anywhere, or simply "what the script
> produces, subject to change"?

  Ideally pkg-stats would have documentation, including of its JSON output. And 
tests as well. We can only dream...

  Regards,
  Arnout

> 
> Best regards,
> Fiona
> 
> 
> Changes v1 -> v2:
> * Avoid glob in patch 1
> * Add patches 4-6
> 
> v1: https://patchwork.ozlabs.org/project/buildroot/list/?series=501442&state=*
> 
> Fiona Klute (6):
>    support/scripts/pkg-stats: search only Config.in{,.host} for URL
>    support/scripts/pkg-stats: format upstream URL info consistently in
>      HTML
>    linux/Config.in: add kernel.org URL as BR2_LINUX_KERNEL_LATEST_VERSION
>      help
>    support/scripts/pkg-stats: fix host/target infra filter
>    support/scripts/pkg-stats: don't buffer whole file searching for infra
>    support/scripts/pkg-stats: run main function only if called as script
> 
>   linux/Config.in           |  2 ++
>   support/scripts/pkg-stats | 54 ++++++++++++++++++++++-----------------
>   2 files changed, 32 insertions(+), 24 deletions(-)
> 

_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH v2 1/6] support/scripts/pkg-stats: search only Config.in{, .host} for URL

[Arnout Vandecappelle via buildroot]

On 26/04/2026 17:33, Fiona Klute via buildroot wrote:
> The previous Config.* glob also caught linux/Config.ext.in and
> package/php/Config.ext, as well as some backup files created by
> editors (e.g. Config.in~ after editing a Config.in file in Emacs),
> leading to wrong results depending on directory listing order.
> 
> Also use "with" to automatically close the file when the block is
> left, even on error.
> 
> Signed-off-by: Fiona Klute <fiona.klute@gmx.de>

  Applied to master, thanks.

> ---
> This cannot solve the problem we have no structured URL information
> and don't look at which option the URL is listed under, but at least
> it's less wrong.

  Does either of these actually cause problems?

  What is still incorrect is that it doesn't report when Config.in doesn't have 
an upstream URL while Config.in.host doesn't.

> 
> Changes v1 -> v2:
> * Try fixed names instead of globbing all directory entries
> 
>   support/scripts/pkg-stats | 22 +++++++++++-----------
>   1 file changed, 11 insertions(+), 11 deletions(-)
> 
> diff --git a/support/scripts/pkg-stats b/support/scripts/pkg-stats
> index 55aa63c861..efb85a7405 100755
> --- a/support/scripts/pkg-stats
> +++ b/support/scripts/pkg-stats
> @@ -148,17 +148,17 @@ class Package:
>           Fills in the .url field
>           """
>           self.status['url'] = ("warning", "no Config.in")
> -        for filename in os.listdir(self.pkgdir):
> -            if fnmatch.fnmatch(filename, 'Config.*'):
> -                fp = open(os.path.join(self.pkgdir, filename), "r")
> -                for config_line in fp:
> -                    if URL_RE.match(config_line):
> -                        self.url = config_line.strip()
> -                        self.status['url'] = ("ok", "found")
> -                        fp.close()
> -                        return
> -                self.status['url'] = ("error", "missing")
> -                fp.close()
> +        for filename in ('Config.in', 'Config.in.host'):
> +            try:
> +                with open(os.path.join(self.pkgdir, filename), "r") as fp:
> +                    for config_line in fp:
> +                        if URL_RE.match(config_line):
> +                            self.url = config_line.strip()
> +                            self.status['url'] = ("ok", "found")
> +                            return
> +                    self.status['url'] = ("error", "missing")
> +            except FileNotFoundError:

  We could modernise things by switching to pathlib and using .exists() a priori 
(and never mind the exception in case of TOCTOU or unreadable file).

  But that's further improvement :-)

  Regards,
  Arnout

> +                continue
>   
>       @property
>       def patch_count(self):

_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH v2 2/6] support/scripts/pkg-stats: format upstream URL info consistently in HTML

[Arnout Vandecappelle via buildroot]

On 26/04/2026 17:33, Fiona Klute via buildroot wrote:
> Use only one of the classes for "error" or "warning" status so they
> look different, and format the error/warning text for both. Do not
> make the text a link if the URL is None.
> 
> Signed-off-by: Fiona Klute <fiona.klute@gmx.de>

  Applied to master, thanks.

> ---
>   support/scripts/pkg-stats | 17 +++++++++++------
>   1 file changed, 11 insertions(+), 6 deletions(-)
> 
> diff --git a/support/scripts/pkg-stats b/support/scripts/pkg-stats
> index efb85a7405..a87bb23a8e 100755
> --- a/support/scripts/pkg-stats
> +++ b/support/scripts/pkg-stats
> @@ -1057,14 +1057,19 @@ def dump_html_pkg(f, pkg):
>       div_class = ["centered upstream_url data"]
>       div_class.append(f'_{pkg_css_class}')
>       url_str = pkg.status['url'][1]

  It looks like this could have been used below instead of repeating 
pkg.status['url'][1]. But I'm too lazy to test so I left it unchanged.

  Regards,
  Arnout

> -    if pkg.status['url'][0] in ("error", "warning"):
> -        div_class.append("missing_url")
> -    if pkg.status['url'][0] == "error":
> -        div_class.append("invalid_url")
> -        url_str = f"""<a href="{pkg.url}">{pkg.status['url'][1]}</a>"""
> -    else:
> +    if pkg.status['url'][0] == "ok":
>           div_class.append("good_url")
>           url_str = f'<a href="{pkg.url}">Link</a>'
> +    else:
> +        if pkg.status['url'][0] == "warning":
> +            div_class.append("missing_url")
> +        else:
> +            # "error" status
> +            div_class.append("invalid_url")
> +        if pkg.url is not None:
> +            url_str = f"""<a href="{pkg.url}">{pkg.status['url'][1]}</a>"""
> +        else:
> +            url_str = pkg.status['url'][1]
>       f.write(f'  <div id="{data_field_id}" class="{" ".join(div_class)}">{url_str}</div>\n')
>   
>       # CVEs

_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH v2 3/6] linux/Config.in: add kernel.org URL as BR2_LINUX_KERNEL_LATEST_VERSION help

[Arnout Vandecappelle via buildroot]

On 26/04/2026 17:33, Fiona Klute via buildroot wrote:
> This makes it the first URL in the file, so it shows up correctly in
> pkg-stats output.
> 
> Signed-off-by: Fiona Klute <fiona.klute@gmx.de>
> ---
>   linux/Config.in | 2 ++
>   1 file changed, 2 insertions(+)
> 
> diff --git a/linux/Config.in b/linux/Config.in
> index 8633ddbc82..1104902b4b 100644
> --- a/linux/Config.in
> +++ b/linux/Config.in
> @@ -33,6 +33,8 @@ config BR2_LINUX_KERNEL_LATEST_VERSION
>   	# mips always generates an ITB image
>   	select BR2_PACKAGE_HOST_UBOOT_TOOLS if BR2_mips || BR2_mipsel || BR2_mips64 || BR2_mips64el
>   	select BR2_PACKAGE_HOST_UBOOT_TOOLS_FIT_SUPPORT if BR2_mips || BR2_mipsel || BR2_mips64 || BR2_mips64el
> +	help
> +	  https://www.kernel.org/

  I don't see any reason to put it here rather than in the top-level option 
BR2_LINUX_KERNEL. So I did change that and put it there (and stripped most of 
the commit message).

  Regards,
  Arnout

>   
>   config BR2_LINUX_KERNEL_LATEST_CIP_VERSION
>   	bool "Latest CIP SLTS version (5.10.246-cip66)"

_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH v2 4/6] support/scripts/pkg-stats: fix host/target infra filter

[Arnout Vandecappelle via buildroot]

On 26/04/2026 17:33, Fiona Klute via buildroot wrote:
> The filter is supposed to exclude host/target infra from output if the
> respective package is not built with the current
> configuration.
> 
> However, excluding host packages did not work correctly: If keep_host
> is False because the host package is not built, the next branch was
> checked and included the host infra in output with "target" type if
> the target package is built. For a package that support host and
> target build, but gets built only for the target, this leads to output
> like (Meson example):
> 
> meson (target)
> host-meson (target)
> 
> Skip host infra in the target branch instead. Also include
> Package.infra in Package.__str__() result, which was needed for
> debugging this bug.
> 
> Signed-off-by: Fiona Klute <fiona.klute@gmx.de>

  Applied to master, thanks.
  Regards,
  Arnout

> ---
>   support/scripts/pkg-stats | 7 ++++---
>   1 file changed, 4 insertions(+), 3 deletions(-)
> 
> diff --git a/support/scripts/pkg-stats b/support/scripts/pkg-stats
> index a87bb23a8e..bbd1e3e6f0 100755
> --- a/support/scripts/pkg-stats
> +++ b/support/scripts/pkg-stats
> @@ -205,7 +205,7 @@ class Package:
>                   infra = match.group(1)
>                   if infra.startswith("host-") and keep_host:
>                       self.infras.append(("host", infra[5:]))
> -                elif keep_target:
> +                elif not infra.startswith("host-") and keep_target:
>                       self.infras.append(("target", infra))
>   
>       def set_license(self):
> @@ -342,9 +342,10 @@ class Package:
>           return self.path < other.path
>   
>       def __str__(self):
> -        return "%s (path='%s', license='%s', license_files='%s', hash='%s', patches=%d)" % \
> +        return "%s (path='%s', license='%s', license_files='%s', hash='%s', patches=%d, infras=%r)" % \
>               (self.name, self.path, self.is_status_ok('license'),
> -             self.is_status_ok('license-files'), self.status['hash'], self.patch_count)
> +             self.is_status_ok('license-files'), self.status['hash'], self.patch_count,
> +             self.infras)
>   
>   
>   def get_pkglist(trees, npackages, package_list):

_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH v2 5/6] support/scripts/pkg-stats: don't buffer whole file searching for infra

[Arnout Vandecappelle via buildroot]

On 26/04/2026 17:33, Fiona Klute via buildroot wrote:
> The file handle can be iterated over directly and each line is used
> exactly once, so the only effect of reading all lines into a list
> first was higher memory use and complexity.
> 
> Signed-off-by: Fiona Klute <fiona.klute@gmx.de>

  Applied to master, thanks.
  Regards,
  Arnout

> ---
>   support/scripts/pkg-stats | 3 +--
>   1 file changed, 1 insertion(+), 2 deletions(-)
> 
> diff --git a/support/scripts/pkg-stats b/support/scripts/pkg-stats
> index bbd1e3e6f0..5dcc812e2b 100755
> --- a/support/scripts/pkg-stats
> +++ b/support/scripts/pkg-stats
> @@ -197,8 +197,7 @@ class Package:
>   
>           self.infras = list()
>           with open(self.pkgfile, 'r') as f:
> -            lines = f.readlines()
> -            for line in lines:
> +            for line in f:
>                   match = INFRA_RE.match(line)
>                   if not match:
>                       continue

_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH v2 6/6] support/scripts/pkg-stats: run main function only if called as script

[Arnout Vandecappelle via buildroot]

On 26/04/2026 17:33, Fiona Klute via buildroot wrote:
> The __name__ == '__main__' guard allows importing pkg-stats as a
> module using importlib, circumventing the normal module filename
> requirements. This in turn makes it possible to test/debug individual
> functions.
> 
> Signed-off-by: Fiona Klute <fiona.klute@gmx.de>

  Applied to master, thanks.
  Regards,
  Arnout

> ---
>   support/scripts/pkg-stats | 5 +++--
>   1 file changed, 3 insertions(+), 2 deletions(-)
> 
> diff --git a/support/scripts/pkg-stats b/support/scripts/pkg-stats
> index 5dcc812e2b..2b2194b9a5 100755
> --- a/support/scripts/pkg-stats
> +++ b/support/scripts/pkg-stats
> @@ -1334,7 +1334,7 @@ def is_newer_version_available(pkg):
>       )
>   
>   
> -def __main__():
> +def main():
>       global cvecheck
>   
>       args = parse_args()
> @@ -1403,4 +1403,5 @@ def __main__():
>           dump_json(packages, defconfigs, stats, date, commit, args.json)
>   
>   
> -__main__()
> +if __name__ == '__main__':
> +    main()

_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH v2 1/6] support/scripts/pkg-stats: search only Config.in{, .host} for URL

[Fiona Klute via buildroot]

Am 05.05.26 um 21:32 schrieb Arnout Vandecappelle:
> On 26/04/2026 17:33, Fiona Klute via buildroot wrote:
>> The previous Config.* glob also caught linux/Config.ext.in and
>> package/php/Config.ext, as well as some backup files created by
>> editors (e.g. Config.in~ after editing a Config.in file in Emacs),
>> leading to wrong results depending on directory listing order.
>>
>> Also use "with" to automatically close the file when the block is
>> left, even on error.
>>
>> Signed-off-by: Fiona Klute <fiona.klute@gmx.de>
> 
>   Applied to master, thanks.
> 
>> ---
>> This cannot solve the problem we have no structured URL information
>> and don't look at which option the URL is listed under, but at least
>> it's less wrong.
> 
>   Does either of these actually cause problems?

Rarely. If there are multiple URLs in one config file, we simply pick 
the first one, so now for linux/Config.in you always get the kernel.org 
URL added in this series, even if you selected the CIP kernel which has 
its own URL in its help text. There probably aren't many packages where 
that's an issue.

Mostly it just feels awkward to probe the help text instead of having an 
explicit definition.

>   What is still incorrect is that it doesn't report when Config.in 
> doesn't have an upstream URL while Config.in.host doesn't.

True. Though in that case we'd also have to differentiate between that 
and "there is no Config.in" for host-only packages. So far pkg-stats 
treats host and target packages as one and only reports infra separately.

>> Changes v1 -> v2:
>> * Try fixed names instead of globbing all directory entries
>>
>>   support/scripts/pkg-stats | 22 +++++++++++-----------
>>   1 file changed, 11 insertions(+), 11 deletions(-)
>>
>> diff --git a/support/scripts/pkg-stats b/support/scripts/pkg-stats
>> index 55aa63c861..efb85a7405 100755
>> --- a/support/scripts/pkg-stats
>> +++ b/support/scripts/pkg-stats
>> @@ -148,17 +148,17 @@ class Package:
>>           Fills in the .url field
>>           """
>>           self.status['url'] = ("warning", "no Config.in")
>> -        for filename in os.listdir(self.pkgdir):
>> -            if fnmatch.fnmatch(filename, 'Config.*'):
>> -                fp = open(os.path.join(self.pkgdir, filename), "r")
>> -                for config_line in fp:
>> -                    if URL_RE.match(config_line):
>> -                        self.url = config_line.strip()
>> -                        self.status['url'] = ("ok", "found")
>> -                        fp.close()
>> -                        return
>> -                self.status['url'] = ("error", "missing")
>> -                fp.close()
>> +        for filename in ('Config.in', 'Config.in.host'):
>> +            try:
>> +                with open(os.path.join(self.pkgdir, filename), "r") 
>> as fp:
>> +                    for config_line in fp:
>> +                        if URL_RE.match(config_line):
>> +                            self.url = config_line.strip()
>> +                            self.status['url'] = ("ok", "found")
>> +                            return
>> +                    self.status['url'] = ("error", "missing")
>> +            except FileNotFoundError:
> 
>   We could modernise things by switching to pathlib and using .exists() 
> a priori (and never mind the exception in case of TOCTOU or unreadable 
> file).
> 
>   But that's further improvement :-)

I agree. I kept this series to simple and hopefully uncontroversial 
fixes, didn't want to get into full refactoring right away. ;-)

Best regards,
Fiona

_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH v2 0/6] Bugfixes for pkg-stats

[Fiona Klute via buildroot]

Hi Arnout!

Am 05.05.26 um 21:25 schrieb Arnout Vandecappelle:
>   I'm making an attempt to get this series merged :-)

I see you were successful, thank you! :-)

>   As a preliminary remark though: pkg-stats really should evolve to 
> using show-info output instead of parsing files itself - that would 
> solve the things like the incorrect patches and hash files. There's one 
> big blocker for that though - pkg-stats supposed to be able to work 
> without a .config, and show-info currently doesn't.

Interesting! Is that documented somewhere, e.g. in notes from a 
developer meetup or something?

[...]

>> I see a bunch more things that should be fixed (notably virtual
>> packages reporting missing license/hash info when there simply is
>> none, and the package parsing process itself), this is what I had time
>> for.
>>
>> One question for possible future improvements: Is the JSON output
>> format documented/defined anywhere, or simply "what the script
>> produces, subject to change"?
> 
>   Ideally pkg-stats would have documentation, including of its JSON 
> output. And tests as well. We can only dream...
I'd like a JSON schema with descriptions, that would also help with 
adding tests. ;-)

I was actually wondering if it'd be possible to apply for a grant for 
Buildroot work, e.g. with NLnet NGI Zero [1]. It would be easiser to 
tackle larger changes that way: There's this, maybe kind of related SBOM 
stuff I was involved with before for a client (left unfinished/pending 
because the project ended), and I have some ideas for an overhaul of the 
test system.

If that's an option (as in, maintainers approve of the idea) I'd like to 
collect requirements/ideas/issues as base for a possible grant application.

Best regards,
Fiona


[1] https://nlnet.nl/commonsfund/

_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH v2 0/6] Bugfixes for pkg-stats

[Arnout Vandecappelle via buildroot]

  [Added maintainers in Cc, see the end]

On 06/05/2026 19:49, Fiona Klute wrote:
> Hi Arnout!
> 
> Am 05.05.26 um 21:25 schrieb Arnout Vandecappelle:
>>   I'm making an attempt to get this series merged :-)
> 
> I see you were successful, thank you! :-)
> 
>>   As a preliminary remark though: pkg-stats really should evolve to using 
>> show-info output instead of parsing files itself - that would solve the things 
>> like the incorrect patches and hash files. There's one big blocker for that 
>> though - pkg-stats supposed to be able to work without a .config, and show- 
>> info currently doesn't.
> 
> Interesting! Is that documented somewhere, e.g. in notes from a developer meetup 
> or something?

  No, it's more a vibe than a real decision. When Thomas introduced show-info 
for the -c option of pkg-stats in 7a607dab3, the feedback was "that's great, we 
should always do that" but that turned out to be impossible so it got merged as is.

> 
> [...]
> 
>>> I see a bunch more things that should be fixed (notably virtual
>>> packages reporting missing license/hash info when there simply is
>>> none, and the package parsing process itself), this is what I had time
>>> for.
>>>
>>> One question for possible future improvements: Is the JSON output
>>> format documented/defined anywhere, or simply "what the script
>>> produces, subject to change"?
>>
>>   Ideally pkg-stats would have documentation, including of its JSON output. 
>> And tests as well. We can only dream...
> I'd like a JSON schema with descriptions, that would also help with adding 
> tests. ;-)
> 
> I was actually wondering if it'd be possible to apply for a grant for Buildroot 
> work, e.g. with NLnet NGI Zero [1]. It would be easiser to tackle larger changes 
> that way: There's this, maybe kind of related SBOM stuff I was involved with 
> before for a client (left unfinished/pending because the project ended), and I 
> have some ideas for an overhaul of the test system.
> 
> If that's an option (as in, maintainers approve of the idea) I'd like to collect 
> requirements/ideas/issues as base for a possible grant application.

  I wouldn't say no to that! We actually also have sponsors that could 
contribute more. I guess it would have to be routed through the Buildroot 
Association then.

  Regards,
  Arnout

_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH v2 0/6] Bugfixes for pkg-stats

[Thomas Petazzoni via buildroot]

On Thu, May 07, 2026 at 10:34:35AM +0200, Arnout Vandecappelle wrote:

> > I was actually wondering if it'd be possible to apply for a grant for
> > Buildroot work, e.g. with NLnet NGI Zero [1]. It would be easiser to
> > tackle larger changes that way: There's this, maybe kind of related SBOM
> > stuff I was involved with before for a client (left unfinished/pending
> > because the project ended), and I have some ideas for an overhaul of the
> > test system.
> > 
> > If that's an option (as in, maintainers approve of the idea) I'd like to
> > collect requirements/ideas/issues as base for a possible grant
> > application.
> 
>  I wouldn't say no to that! We actually also have sponsors that could
> contribute more. I guess it would have to be routed through the Buildroot
> Association then.

Totally agree: it's a great idea!

Thomas
-- 
Thomas Petazzoni, co-owner and CEO, Bootlin
Embedded Linux and Kernel engineering and training
https://bootlin.com
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH v2 0/6] Bugfixes for pkg-stats

[Peter Korsgaard]

>>>>> "Thomas" == Thomas Petazzoni <thomas.petazzoni@bootlin.com> writes:

Hi,

 >> I wouldn't say no to that! We actually also have sponsors that could
 >> contribute more. I guess it would have to be routed through the Buildroot
 >> Association then.

 > Totally agree: it's a great idea!

+1

-- 
Bye, Peter Korsgaard
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH v2 3/6] linux/Config.in: add kernel.org URL as BR2_LINUX_KERNEL_LATEST_VERSION help

[Thomas Perale via buildroot]

In reply of:
> This makes it the first URL in the file, so it shows up correctly in
> pkg-stats output.
> 
> Signed-off-by: Fiona Klute <fiona.klute@gmx.de>

Applied to 2025.02.x & 2026.02.x. Thanks

> ---
>  linux/Config.in | 2 ++
>  1 file changed, 2 insertions(+)
> 
> diff --git a/linux/Config.in b/linux/Config.in
> index 8633ddbc82..1104902b4b 100644
> --- a/linux/Config.in
> +++ b/linux/Config.in
> @@ -33,6 +33,8 @@ config BR2_LINUX_KERNEL_LATEST_VERSION
>  	# mips always generates an ITB image
>  	select BR2_PACKAGE_HOST_UBOOT_TOOLS if BR2_mips || BR2_mipsel || BR2_mips64 || BR2_mips64el
>  	select BR2_PACKAGE_HOST_UBOOT_TOOLS_FIT_SUPPORT if BR2_mips || BR2_mipsel || BR2_mips64 || BR2_mips64el
> +	help
> +	  https://www.kernel.org/
>  
>  config BR2_LINUX_KERNEL_LATEST_CIP_VERSION
>  	bool "Latest CIP SLTS version (5.10.246-cip66)"
> -- 
> 2.53.0
> 
> _______________________________________________
> buildroot mailing list
> buildroot@buildroot.org
> https://lists.buildroot.org/mailman/listinfo/buildroot
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH v2 5/6] support/scripts/pkg-stats: don't buffer whole file searching for infra

[Thomas Perale via buildroot]

In reply of:
> The file handle can be iterated over directly and each line is used
> exactly once, so the only effect of reading all lines into a list
> first was higher memory use and complexity.
> 
> Signed-off-by: Fiona Klute <fiona.klute@gmx.de>

Applied to 2025.02.x & 2026.02.x. Thanks

> ---
>  support/scripts/pkg-stats | 3 +--
>  1 file changed, 1 insertion(+), 2 deletions(-)
> 
> diff --git a/support/scripts/pkg-stats b/support/scripts/pkg-stats
> index bbd1e3e6f0..5dcc812e2b 100755
> --- a/support/scripts/pkg-stats
> +++ b/support/scripts/pkg-stats
> @@ -197,8 +197,7 @@ class Package:
>  
>          self.infras = list()
>          with open(self.pkgfile, 'r') as f:
> -            lines = f.readlines()
> -            for line in lines:
> +            for line in f:
>                  match = INFRA_RE.match(line)
>                  if not match:
>                      continue
> -- 
> 2.53.0
> 
> _______________________________________________
> buildroot mailing list
> buildroot@buildroot.org
> https://lists.buildroot.org/mailman/listinfo/buildroot
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH v2 4/6] support/scripts/pkg-stats: fix host/target infra filter

[Thomas Perale via buildroot]

In reply of:
> The filter is supposed to exclude host/target infra from output if the
> respective package is not built with the current
> configuration.
> 
> However, excluding host packages did not work correctly: If keep_host
> is False because the host package is not built, the next branch was
> checked and included the host infra in output with "target" type if
> the target package is built. For a package that support host and
> target build, but gets built only for the target, this leads to output
> like (Meson example):
> 
> meson (target)
> host-meson (target)
> 
> Skip host infra in the target branch instead. Also include
> Package.infra in Package.__str__() result, which was needed for
> debugging this bug.
> 
> Signed-off-by: Fiona Klute <fiona.klute@gmx.de>

Applied to 2025.02.x & 2026.02.x. Thanks

> ---
>  support/scripts/pkg-stats | 7 ++++---
>  1 file changed, 4 insertions(+), 3 deletions(-)
> 
> diff --git a/support/scripts/pkg-stats b/support/scripts/pkg-stats
> index a87bb23a8e..bbd1e3e6f0 100755
> --- a/support/scripts/pkg-stats
> +++ b/support/scripts/pkg-stats
> @@ -205,7 +205,7 @@ class Package:
>                  infra = match.group(1)
>                  if infra.startswith("host-") and keep_host:
>                      self.infras.append(("host", infra[5:]))
> -                elif keep_target:
> +                elif not infra.startswith("host-") and keep_target:
>                      self.infras.append(("target", infra))
>  
>      def set_license(self):
> @@ -342,9 +342,10 @@ class Package:
>          return self.path < other.path
>  
>      def __str__(self):
> -        return "%s (path='%s', license='%s', license_files='%s', hash='%s', patches=%d)" % \
> +        return "%s (path='%s', license='%s', license_files='%s', hash='%s', patches=%d, infras=%r)" % \
>              (self.name, self.path, self.is_status_ok('license'),
> -             self.is_status_ok('license-files'), self.status['hash'], self.patch_count)
> +             self.is_status_ok('license-files'), self.status['hash'], self.patch_count,
> +             self.infras)
>  
>  
>  def get_pkglist(trees, npackages, package_list):
> -- 
> 2.53.0
> 
> _______________________________________________
> buildroot mailing list
> buildroot@buildroot.org
> https://lists.buildroot.org/mailman/listinfo/buildroot
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH v2 2/6] support/scripts/pkg-stats: format upstream URL info consistently in HTML

[Thomas Perale via buildroot]

In reply of:
> Use only one of the classes for "error" or "warning" status so they
> look different, and format the error/warning text for both. Do not
> make the text a link if the URL is None.
> 
> Signed-off-by: Fiona Klute <fiona.klute@gmx.de>

Applied to 2025.02.x & 2026.02.x. Thanks

> ---
>  support/scripts/pkg-stats | 17 +++++++++++------
>  1 file changed, 11 insertions(+), 6 deletions(-)
> 
> diff --git a/support/scripts/pkg-stats b/support/scripts/pkg-stats
> index efb85a7405..a87bb23a8e 100755
> --- a/support/scripts/pkg-stats
> +++ b/support/scripts/pkg-stats
> @@ -1057,14 +1057,19 @@ def dump_html_pkg(f, pkg):
>      div_class = ["centered upstream_url data"]
>      div_class.append(f'_{pkg_css_class}')
>      url_str = pkg.status['url'][1]
> -    if pkg.status['url'][0] in ("error", "warning"):
> -        div_class.append("missing_url")
> -    if pkg.status['url'][0] == "error":
> -        div_class.append("invalid_url")
> -        url_str = f"""<a href="{pkg.url}">{pkg.status['url'][1]}</a>"""
> -    else:
> +    if pkg.status['url'][0] == "ok":
>          div_class.append("good_url")
>          url_str = f'<a href="{pkg.url}">Link</a>'
> +    else:
> +        if pkg.status['url'][0] == "warning":
> +            div_class.append("missing_url")
> +        else:
> +            # "error" status
> +            div_class.append("invalid_url")
> +        if pkg.url is not None:
> +            url_str = f"""<a href="{pkg.url}">{pkg.status['url'][1]}</a>"""
> +        else:
> +            url_str = pkg.status['url'][1]
>      f.write(f'  <div id="{data_field_id}" class="{" ".join(div_class)}">{url_str}</div>\n')
>  
>      # CVEs
> -- 
> 2.53.0
> 
> _______________________________________________
> buildroot mailing list
> buildroot@buildroot.org
> https://lists.buildroot.org/mailman/listinfo/buildroot
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH v2 1/6] support/scripts/pkg-stats: search only Config.in{, .host} for URL

[Thomas Perale via buildroot]

In reply of:
> The previous Config.* glob also caught linux/Config.ext.in and
> package/php/Config.ext, as well as some backup files created by
> editors (e.g. Config.in~ after editing a Config.in file in Emacs),
> leading to wrong results depending on directory listing order.
> 
> Also use "with" to automatically close the file when the block is
> left, even on error.
> 
> Signed-off-by: Fiona Klute <fiona.klute@gmx.de>

Applied to 2025.02.x & 2026.02.x. Thanks

> ---
> This cannot solve the problem we have no structured URL information
> and don't look at which option the URL is listed under, but at least
> it's less wrong.
> 
> Changes v1 -> v2:
> * Try fixed names instead of globbing all directory entries
> 
>  support/scripts/pkg-stats | 22 +++++++++++-----------
>  1 file changed, 11 insertions(+), 11 deletions(-)
> 
> diff --git a/support/scripts/pkg-stats b/support/scripts/pkg-stats
> index 55aa63c861..efb85a7405 100755
> --- a/support/scripts/pkg-stats
> +++ b/support/scripts/pkg-stats
> @@ -148,17 +148,17 @@ class Package:
>          Fills in the .url field
>          """
>          self.status['url'] = ("warning", "no Config.in")
> -        for filename in os.listdir(self.pkgdir):
> -            if fnmatch.fnmatch(filename, 'Config.*'):
> -                fp = open(os.path.join(self.pkgdir, filename), "r")
> -                for config_line in fp:
> -                    if URL_RE.match(config_line):
> -                        self.url = config_line.strip()
> -                        self.status['url'] = ("ok", "found")
> -                        fp.close()
> -                        return
> -                self.status['url'] = ("error", "missing")
> -                fp.close()
> +        for filename in ('Config.in', 'Config.in.host'):
> +            try:
> +                with open(os.path.join(self.pkgdir, filename), "r") as fp:
> +                    for config_line in fp:
> +                        if URL_RE.match(config_line):
> +                            self.url = config_line.strip()
> +                            self.status['url'] = ("ok", "found")
> +                            return
> +                    self.status['url'] = ("error", "missing")
> +            except FileNotFoundError:
> +                continue
>  
>      @property
>      def patch_count(self):
> -- 
> 2.53.0
> 
> _______________________________________________
> buildroot mailing list
> buildroot@buildroot.org
> https://lists.buildroot.org/mailman/listinfo/buildroot
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH v2 6/6] support/scripts/pkg-stats: run main function only if called as script

[Thomas Perale via buildroot]

In reply of:
> The __name__ == '__main__' guard allows importing pkg-stats as a
> module using importlib, circumventing the normal module filename
> requirements. This in turn makes it possible to test/debug individual
> functions.
> 
> Signed-off-by: Fiona Klute <fiona.klute@gmx.de>

Applied to 2025.02.x & 2026.02.x. Thanks

> ---
>  support/scripts/pkg-stats | 5 +++--
>  1 file changed, 3 insertions(+), 2 deletions(-)
> 
> diff --git a/support/scripts/pkg-stats b/support/scripts/pkg-stats
> index 5dcc812e2b..2b2194b9a5 100755
> --- a/support/scripts/pkg-stats
> +++ b/support/scripts/pkg-stats
> @@ -1334,7 +1334,7 @@ def is_newer_version_available(pkg):
>      )
>  
>  
> -def __main__():
> +def main():
>      global cvecheck
>  
>      args = parse_args()
> @@ -1403,4 +1403,5 @@ def __main__():
>          dump_json(packages, defconfigs, stats, date, commit, args.json)
>  
>  
> -__main__()
> +if __name__ == '__main__':
> +    main()
> -- 
> 2.53.0
> 
> _______________________________________________
> buildroot mailing list
> buildroot@buildroot.org
> https://lists.buildroot.org/mailman/listinfo/buildroot
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot