[Buildroot] [PATCH next v2 0/5] New pkg-stats script, with version information

[Buildroot] [PATCH next v2 0/5] New pkg-stats script, with version information

[Thomas Petazzoni]

Hello,

This series rewrites the pkg-stats script in Python, and adds two new
very useful information to it:

 - The current version of each package in Buildroot

 - The latest upstream version of each package, as provided by the
   release-monitoring.org web site.

The script then compares the current version in Buildroot with the
latest upstream version, and tells whether they are different.

You can see the script output at:

 https://bootlin.com/~thomas/stats.html

The code is available at:

 https://git.free-electrons.com/users/thomas-petazzoni/buildroot/log/?h=pkg-stats-v2

Changes since v1
================

This version mainly focuses on fixing the numerous comments and issues
pointed out by Ricardo Martincoski (many thanks for his very detailed
and useful review!), and also fixes a few other things:

 - Added copyright notice in the new script

 - Fixed all flake8 warnings (I still personally find the "let's force
   to have two empty lines between every function" a very silly rule,
   but well)

 - Pass the package path to the Package() constructor directly

 - Only introduce CSS properties when they are really needed (some
   were introduced in PATCH 1, but they were only really used by later
   patches)

 - Fix the broken #results internal link, which now properly points to
   the statistics table.

 - Remove double </table> tag.

 - Fix the sorting of the table by defining __eq__ and __lt__ in
   Package(), and make 'packages' a list rather than a dict.

 - Add the build date and git commit in the HTML page footer

 - Pass BR2_HAVE_DOT_CONFIG=y when calling make, in order to fake
   having a .config. This allows "printvars" to dump all variables
   even without a .config.

 - Add newlines in the HTML to make it readable.

More details
============

release-monitoring.org is a very useful web site, monitoring more than
16000 projects. It is also very easy to add new projects to be
monitored. It supports monitoring projects on popular hosting
platforms such as Github, but can also monitor plain HTTP folders, or
even web pages using a regexp to identify what is a version number
within the HTML blurb.

Projects can be found by regular search, but it is also possible to
add a mapping between the name of a package in a given distribution,
and the name of the package as known by release-monitoring.org. For
example in Buildroot "samba" is not named "samba" but "samba4", and
this mapping mechanism allows release-monitoring.org to reply to
requests for samba4 within the Buildroot distribution.

I had very good interactions with the release-monitoring.org
maintainers:

 - They are easily available on IRC

 - They created the "Buildroot" distribution within minutes,
   https://release-monitoring.org/distro/Buildroot/.

 - They have been very responsive to fix issues in existing packages.

It doesn't provide CVE related information for security, so it would
still be useful to extend this mechanism with another CVE related
database. But as we discussed during the last Buildroot meeting in
Brussels, the NIST database is not very up to date, while
release-monitoring.org, thanks to the process being fully automated.

Before people start sending gazillions of patches to update packages,
I would like us to focus on:

 - Adding missing projects on release-monitoring.org

 - Adding missing mappings for the Buildroot distribution on
   release-monitoring.org

 - Deciding how to handle packages such as all python-* packages or
   all x11r7 packages, for which the name never matches with the
   release-monitoring.org package name.

   Do we create a mapping for each of them on release-monitoring.org
   (which we would have to do for every new python package) or do we
   make the script smarter, and attempt to search the package without
   its python- prefix (which won't always work either) ?

Basically, I would like to focus on making the output of the script
more useful/relevant, and then only start getting gazillions of
patches updating packages.

Thanks for your review, and contributions!

Thomas


Thomas Petazzoni (5):
  support/scripts/pkg-stats-new: rewrite in Python
  support/scripts/pkg-stats-new: add -n and -p options
  support/scripts/pkg-stats-new: add current version information
  support/scripts/pkg-stats-new: add latest upstream version information
  support/scripts/pkg-stats: replace with new Python version

 support/scripts/pkg-stats | 1001 +++++++++++++++++++++++++++------------------
 1 file changed, 607 insertions(+), 394 deletions(-)

-- 
2.14.3

[Buildroot] [PATCH next v2 1/5] support/scripts/pkg-stats-new: rewrite in Python

[Thomas Petazzoni]

This commit adds a new version of the pkg-stats script, rewritten in
Python. It is for now implemented in a separate file called,
pkg-stats-new, in order to make the diff easily readable. A future
commit will rename it to pkg-stats.

Compared to the existing shell-based pkg-stats script, the
functionality and output is basically the same. The main difference is
that the output no longer goes to stdout, but to the file passed as
argument using the -o option. This allows stdout to be used for more
debugging related information.

The way the script works is that a first function get_pkglist()
creates a dict associating package names with an instance of a
Package() object, containing basic information about the package. Then
a number of other functions (add_infra_info, add_pkg_make_info,
add_hash_info, add_patch_count, add_check_package_warnings) will
calculate additional information about packages, and fill in fields in
the Package objects.

calculate_stats() then calculates global statistics (how packages have
license information, how packages have a hash file, etc.). Finally,
dump_html() produces the HTML output, using a number of sub-functions.

One improvement over the shell-based version is that we can use
regexps to exclude some .mk files. Thanks to this, we can exclude all
linux-ext-*.mk files, avoiding incorrect matches.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
---
Changes since v1: take into account Ricardo's comments:

- Added copyright notice
- Fix flake8 warnings
- Pass package path to Package() constructor
- Remove CSS properties not needed in this commit
- Fix broken #results link
- Remove double </table>
- Add newlines in the generated HTML
- Fix the sorting of the table by defining __eq__ and __lt__ in
  Package(), and making 'packages' a list rather than a dict.
- Add the build date and git commit in the HTML page footer
- Pass BR2_HAVE_DOT_CONFIG=y when calling make, in order to fake
  having a .config. This allows "printvars" to dump all variables even
  without a .config.
---
 support/scripts/pkg-stats-new | 478 ++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 478 insertions(+)
 create mode 100755 support/scripts/pkg-stats-new

diff --git a/support/scripts/pkg-stats-new b/support/scripts/pkg-stats-new
new file mode 100755
index 0000000000..8c74ff2a37
--- /dev/null
+++ b/support/scripts/pkg-stats-new
@@ -0,0 +1,478 @@
+#!/usr/bin/env python
+
+# Copyright (C) 2009 by Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+# General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+
+import argparse
+import datetime
+import fnmatch
+import os
+from collections import defaultdict
+import re
+import subprocess
+
+
+class Package:
+    def __init__(self, name, path):
+        self.name = name
+        self.path = path
+        self.infras = None
+        self.has_license = False
+        self.has_license_files = False
+        self.has_hash = False
+        self.patch_count = 0
+        self.warnings = 0
+
+    def __eq__(self, other):
+        return self.path == other.path
+
+    def __lt__(self, other):
+        return self.path < other.path
+
+    def __str__(self):
+        return "%s (path='%s', license='%s', license_files='%s', hash='%s', patches=%d)" % \
+            (self.name, self.path, self.has_license, self.has_license_files, self.has_hash, self.patch_count)
+
+
+def get_pkglist():
+    """
+    Builds the list of Buildroot packages, returning a list of Package
+    objects. Only the .name and .path fields of the Package object are
+    initialized.
+    """
+    WALK_USEFUL_SUBDIRS = ["boot", "linux", "package", "toolchain"]
+    WALK_EXCLUDES = ["boot/common.mk",
+                     "linux/linux-ext-.*.mk",
+                     "package/freescale-imx/freescale-imx.mk",
+                     "package/gcc/gcc.mk",
+                     "package/gstreamer/gstreamer.mk",
+                     "package/gstreamer1/gstreamer1.mk",
+                     "package/gtk2-themes/gtk2-themes.mk",
+                     "package/matchbox/matchbox.mk",
+                     "package/opengl/opengl.mk",
+                     "package/qt5/qt5.mk",
+                     "package/x11r7/x11r7.mk",
+                     "package/doc-asciidoc.mk",
+                     "package/pkg-.*.mk",
+                     "package/nvidia-tegra23/nvidia-tegra23.mk",
+                     "toolchain/toolchain-external/pkg-toolchain-external.mk",
+                     "toolchain/toolchain-external/toolchain-external.mk",
+                     "toolchain/toolchain.mk",
+                     "toolchain/helpers.mk",
+                     "toolchain/toolchain-wrapper.mk"]
+    packages = list()
+    for root, dirs, files in os.walk("."):
+        rootdir = root.split("/")
+        if len(rootdir) < 2:
+            continue
+        if rootdir[1] not in WALK_USEFUL_SUBDIRS:
+            continue
+        for f in files:
+            if not f.endswith(".mk"):
+                continue
+            # Strip ending ".mk"
+            pkgname = f[:-3]
+            pkgpath = os.path.join(root, f)
+            skip = False
+            for exclude in WALK_EXCLUDES:
+                # pkgpath[2:] strips the initial './'
+                if re.match(exclude, pkgpath[2:]):
+                    skip = True
+                    continue
+            if skip:
+                continue
+            p = Package(pkgname, pkgpath)
+            packages.append(p)
+    return packages
+
+
+INFRA_RE = re.compile("\$\(eval \$\(([a-z-]*)-package\)\)")
+
+
+def get_pkg_infra_info(pkgpath):
+    infras = list()
+    with open(pkgpath, 'r') as f:
+        lines = f.readlines()
+        for l in lines:
+            match = INFRA_RE.match(l)
+            if not match:
+                continue
+            infra = match.group(1)
+            if infra.startswith("host-"):
+                infras.append(("host", infra[5:]))
+            else:
+                infras.append(("target", infra))
+    return infras
+
+
+def add_infra_info(packages):
+
+    """
+    Fills in the .infras field of all Package objects
+    """
+    for pkg in packages:
+        pkg.infras = get_pkg_infra_info(pkg.path)
+
+
+def pkgname_to_pkgvar(pkgname):
+    return pkgname.upper().replace("-", "_")
+
+
+def add_pkg_make_info(packages):
+    """
+    Fills in the .has_license and .has_license_files fields of all
+    Package objects
+    """
+    licenses = list()
+    license_files = list()
+
+    # Licenses
+    o = subprocess.check_output(["make", "BR2_HAVE_DOT_CONFIG=y",
+                                 "-s", "printvars", "VARS=%_LICENSE"])
+    for l in o.splitlines():
+        # Get variable name and value
+        pkgvar, value = l.split("=")
+
+        # If present, strip HOST_ from variable name
+        if pkgvar.startswith("HOST_"):
+            pkgvar = pkgvar[5:]
+
+        # Strip _LICENSE
+        pkgvar = pkgvar[:-8]
+
+        # If value is "unknown", no license details available
+        if value == "unknown":
+            continue
+        licenses.append(pkgvar)
+
+    # License files
+    o = subprocess.check_output(["make", "BR2_HAVE_DOT_CONFIG=y",
+                                 "-s", "printvars", "VARS=%_LICENSE_FILES"])
+    for l in o.splitlines():
+        # Get variable name and value
+        pkgvar, value = l.split("=")
+
+        # If present, strip HOST_ from variable name
+        if pkgvar.startswith("HOST_"):
+            pkgvar = pkgvar[5:]
+
+        if pkgvar.endswith("_MANIFEST_LICENSE_FILES"):
+            continue
+
+        # Strip _LICENSE_FILES
+        pkgvar = pkgvar[:-14]
+
+        license_files.append(pkgvar)
+
+    for pkg in packages:
+        var = pkgname_to_pkgvar(pkg.name)
+        if var in licenses:
+            pkg.has_license = True
+        if var in license_files:
+            pkg.has_license_files = True
+
+
+def add_hash_info(packages):
+    """
+    Fills in the .has_hash field of all Package objects
+    """
+    for pkg in packages:
+        hashpath = pkg.path.replace(".mk", ".hash")
+        pkg.has_hash = os.path.exists(hashpath)
+
+
+def add_patch_count(packages):
+    """
+    Fills in the .patch_count field of all Package objects
+    """
+    for pkg in packages:
+        pkgdir = os.path.dirname(pkg.path)
+        pkg.patch_count = len(fnmatch.filter(os.listdir(pkgdir), '*.patch'))
+
+
+def get_check_package_warnings(pkgdir):
+    cmd = ["./utils/check-package"]
+    for root, dirs, files in os.walk(pkgdir):
+        for f in files:
+            if f.endswith(".mk") or f.endswith(".hash") or f == "Config.in" or f == "Config.in.host":
+                cmd.append(f)
+    o = subprocess.check_output(cmd, stderr=subprocess.STDOUT)
+    lines = o.splitlines()
+    for line in lines:
+        m = re.match("^([0-9]*) warnings generated", line)
+        if m:
+            return int(m.group(1))
+    return None
+
+
+def add_check_package_warnings(packages):
+    """
+    Fills in the .warnings field of all Package objects
+    """
+    for pkg in packages:
+        pkg.warnings = get_check_package_warnings(os.path.dirname(pkg.path))
+
+
+def calculate_stats(packages):
+    stats = defaultdict(int)
+    for pkg in packages:
+        # If packages have multiple infra, take the first one. For the
+        # vast majority of packages, the target and host infra are the
+        # same. There are very few packages that use a different infra
+        # for the host and target variants.
+        if len(pkg.infras) > 0:
+            infra = pkg.infras[0][1]
+            stats["infra-%s" % infra] += 1
+        else:
+            stats["infra-unknown"] += 1
+        if pkg.has_license:
+            stats["license"] += 1
+        else:
+            stats["no-license"] += 1
+        if pkg.has_license_files:
+            stats["license-files"] += 1
+        else:
+            stats["no-license-files"] += 1
+        if pkg.has_hash:
+            stats["hash"] += 1
+        else:
+            stats["no-hash"] += 1
+        stats["patches"] += pkg.patch_count
+    return stats
+
+
+html_header = """
+<head>
+<script src=\"https://www.kryogenix.org/code/browser/sorttable/sorttable.js\"></script>
+<style type=\"text/css\">
+table {
+  width: 100%;
+}
+td {
+  border: 1px solid black;
+}
+td.centered {
+  text-align: center;
+}
+td.wrong {
+  background: #ff9a69;
+}
+td.correct {
+  background: #d2ffc4;
+}
+td.nopatches {
+  background: #d2ffc4;
+}
+td.somepatches {
+  background: #ffd870;
+}
+td.lotsofpatches {
+  background: #ff9a69;
+}
+</style>
+<title>Statistics of Buildroot packages</title>
+</head>
+
+<a href=\"#results\">Results</a><br/>
+
+<p id=\"sortable_hint\"></p>
+"""
+
+
+html_footer = """
+</body>
+<script>
+if (typeof sorttable === \"object\") {
+  document.getElementById(\"sortable_hint\").innerHTML =
+  \"hint: the table can be sorted by clicking the column headers\"
+}
+</script>
+</html>
+"""
+
+
+def infra_str(infra_list):
+    if not infra_list:
+        return "Unknown"
+    elif len(infra_list) == 1:
+        return "<b>%s</b><br/>%s" % (infra_list[0][1], infra_list[0][0])
+    elif infra_list[0][1] == infra_list[1][1]:
+        return "<b>%s</b><br/>%s + %s" % \
+            (infra_list[0][1], infra_list[0][0], infra_list[1][0])
+    else:
+        return "<b>%s</b> (%s)<br/><b>%s</b> (%s)" % \
+            (infra_list[0][1], infra_list[0][0],
+             infra_list[1][1], infra_list[1][0])
+
+
+def boolean_str(b):
+    if b:
+        return "Yes"
+    else:
+        return "No"
+
+
+def dump_html_pkg(f, pkg):
+    f.write(" <tr>\n")
+    f.write("  <td>%s</td>\n" % pkg.path[2:])
+
+    # Patch count
+    td_class = ["centered"]
+    if pkg.patch_count == 0:
+        td_class.append("nopatches")
+    elif pkg.patch_count < 5:
+        td_class.append("somepatches")
+    else:
+        td_class.append("lotsofpatches")
+    f.write("  <td class=\"%s\">%s</td>\n" %
+            (" ".join(td_class), str(pkg.patch_count)))
+
+    # Infrastructure
+    infra = infra_str(pkg.infras)
+    td_class = ["centered"]
+    if infra == "Unknown":
+        td_class.append("wrong")
+    else:
+        td_class.append("correct")
+    f.write("  <td class=\"%s\">%s</td>\n" %
+            (" ".join(td_class), infra_str(pkg.infras)))
+
+    # License
+    td_class = ["centered"]
+    if pkg.has_license:
+        td_class.append("correct")
+    else:
+        td_class.append("wrong")
+    f.write("  <td class=\"%s\">%s</td>\n" %
+            (" ".join(td_class), boolean_str(pkg.has_license)))
+
+    # License files
+    td_class = ["centered"]
+    if pkg.has_license_files:
+        td_class.append("correct")
+    else:
+        td_class.append("wrong")
+    f.write("  <td class=\"%s\">%s</td>\n" %
+            (" ".join(td_class), boolean_str(pkg.has_license_files)))
+
+    # Hash
+    td_class = ["centered"]
+    if pkg.has_hash:
+        td_class.append("correct")
+    else:
+        td_class.append("wrong")
+    f.write("  <td class=\"%s\">%s</td>\n" %
+            (" ".join(td_class), boolean_str(pkg.has_hash)))
+
+    # Warnings
+    td_class = ["centered"]
+    if pkg.warnings == 0:
+        td_class.append("correct")
+    else:
+        td_class.append("wrong")
+    f.write("  <td class=\"%s\">%d</td>\n" %
+            (" ".join(td_class), pkg.warnings))
+
+    f.write(" </tr>\n")
+
+
+def dump_html_all_pkgs(f, packages):
+    f.write("""
+<table class=\"sortable\">
+<tr>
+<td>Package</td>
+<td class=\"centered\">Patch count</td>
+<td class=\"centered\">Infrastructure</td>
+<td class=\"centered\">License</td>
+<td class=\"centered\">License files</td>
+<td class=\"centered\">Hash file</td>
+<td class=\"centered\">Warnings</td>
+</tr>
+""")
+    for pkg in sorted(packages):
+        dump_html_pkg(f, pkg)
+    f.write("</table>")
+
+
+def dump_html_stats(f, stats):
+    f.write("<a id=\"results\"></a>\n")
+    f.write("<table>\n")
+    infras = [infra[6:] for infra in stats.keys() if infra.startswith("infra-")]
+    for infra in infras:
+        f.write(" <tr><td>Packages using the <i>%s</i> infrastructure</td><td>%s</td></tr>\n" %
+                (infra, stats["infra-%s" % infra]))
+    f.write(" <tr><td>Packages having license information</td><td>%s</td></tr>\n" %
+            stats["license"])
+    f.write(" <tr><td>Packages not having license information</td><td>%s</td></tr>\n" %
+            stats["no-license"])
+    f.write(" <tr><td>Packages having license files information</td><td>%s</td></tr>\n" %
+            stats["license-files"])
+    f.write(" <tr><td>Packages not having license files information</td><td>%s</td></tr>\n" %
+            stats["no-license-files"])
+    f.write(" <tr><td>Packages having a hash file</td><td>%s</td></tr>\n" %
+            stats["hash"])
+    f.write(" <tr><td>Packages not having a hash file</td><td>%s</td></tr>\n" %
+            stats["no-hash"])
+    f.write(" <tr><td>Total number of patches</td><td>%s</td></tr>\n" %
+            stats["patches"])
+    f.write("</table>\n")
+
+
+def dump_gen_info(f):
+    # Updated on Mon Feb 19 08:12:08 CET 2018, Git commit aa77030b8f5e41f1c53eb1c1ad664b8c814ba032
+    o = subprocess.check_output(["git", "log", "master", "-n", "1", "--pretty=format:%H"])
+    git_commit = o.splitlines()[0]
+    f.write("<p><i>Updated on %s, git commit %s</i></p>\n" %
+            (str(datetime.datetime.utcnow()), git_commit))
+
+
+def dump_html(packages, stats, output):
+    with open(output, 'w') as f:
+        f.write(html_header)
+        dump_html_all_pkgs(f, packages)
+        dump_html_stats(f, stats)
+        dump_gen_info(f)
+        f.write(html_footer)
+
+
+def parse_args():
+    parser = argparse.ArgumentParser()
+    parser.add_argument('-o', dest='output', action='store', required=True,
+                        help='HTML output file')
+    return parser.parse_args()
+
+
+def __main__():
+    args = parse_args()
+    print "Build package list ..."
+    packages = get_pkglist()
+    print "Get package infra ..."
+    add_infra_info(packages)
+    print "Get make info ..."
+    add_pkg_make_info(packages)
+    print "Get hash info ..."
+    add_hash_info(packages)
+    print "Get patch count ..."
+    add_patch_count(packages)
+    print "Get package warnings ..."
+    add_check_package_warnings(packages)
+    print "Calculate stats"
+    stats = calculate_stats(packages)
+    print "Write HTML"
+    dump_html(packages, stats, args.output)
+
+
+__main__()
-- 
2.14.3

[Buildroot] [PATCH next v2 1/5] support/scripts/pkg-stats-new: rewrite in Python

[Ricardo Martincoski]

Hello,

Sorry, I missed few items when reviewing v1.
I hope I find the time to review the other patches in the next days.

On Wed, Feb 21, 2018 at 07:13 PM, Thomas Petazzoni wrote:

[snip]
> +def add_patch_count(packages):
> +    """
> +    Fills in the .patch_count field of all Package objects
> +    """
> +    for pkg in packages:
> +        pkgdir = os.path.dirname(pkg.path)
> +        pkg.patch_count = len(fnmatch.filter(os.listdir(pkgdir), '*.patch'))

This method is missing patches in subdirs for binutils, gdb, ...

    for pkg in packages:
        pkg.patch_count = 0
        for subdir, _, _ in os.walk(pkgdir):
            pkg.patch_count += len(fnmatch.filter(os.listdir(subdir), '*.patch'))

> +
> +
> +def get_check_package_warnings(pkgdir):
> +    cmd = ["./utils/check-package"]
> +    for root, dirs, files in os.walk(pkgdir):
> +        for f in files:
> +            if f.endswith(".mk") or f.endswith(".hash") or f == "Config.in" or f == "Config.in.host":
> +                cmd.append(f)

Here you need:
                cmd.append(os.path.join(root, f))
otherwise the whole column is filled with zeros.

> +    o = subprocess.check_output(cmd, stderr=subprocess.STDOUT)

When above line is fixed, this command can return non-zero code that leads to
exception CalledProcessError. There are a few ways to solve it:

1)
    o = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE).communicate()[1]
2)
    try:
        ...
    except ...
3) change check-package, adding an option to always return code zero.

IMO, option 1 is the better one.

> +    lines = o.splitlines()
> +    for line in lines:
> +        m = re.match("^([0-9]*) warnings generated", line)
> +        if m:
> +            return int(m.group(1))
> +    return None
> +

Regards,
Ricardo

[Buildroot] [PATCH next v2 1/5] support/scripts/pkg-stats-new: rewrite in Python

[Thomas Petazzoni]

Hello,

On Wed, 21 Feb 2018 22:58:53 -0300, Ricardo Martincoski wrote:

> This method is missing patches in subdirs for binutils, gdb, ...
> 
>     for pkg in packages:
>         pkg.patch_count = 0
>         for subdir, _, _ in os.walk(pkgdir):
>             pkg.patch_count += len(fnmatch.filter(os.listdir(subdir), '*.patch'))

ACK, already fixed for v3.

> > +def get_check_package_warnings(pkgdir):
> > +    cmd = ["./utils/check-package"]
> > +    for root, dirs, files in os.walk(pkgdir):
> > +        for f in files:
> > +            if f.endswith(".mk") or f.endswith(".hash") or f == "Config.in" or f == "Config.in.host":
> > +                cmd.append(f)  
> 
> Here you need:
>                 cmd.append(os.path.join(root, f))
> otherwise the whole column is filled with zeros.

Indeed. Fixed for v3.

> 
> > +    o = subprocess.check_output(cmd, stderr=subprocess.STDOUT)  
> 
> When above line is fixed, this command can return non-zero code that leads to
> exception CalledProcessError. There are a few ways to solve it:
> 
> 1)
>     o = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE).communicate()[1]
> 2)
>     try:
>         ...
>     except ...
> 3) change check-package, adding an option to always return code zero.
> 
> IMO, option 1 is the better one.

Agreed, fixed for v3 as well.

Thanks for the review!

Thomas
-- 
Thomas Petazzoni, CTO, Bootlin (formerly Free Electrons)
Embedded Linux and Kernel engineering
http://bootlin.com

[Buildroot] [PATCH next v2 2/5] support/scripts/pkg-stats-new: add -n and -p options

[Thomas Petazzoni]

This commit adds the following options to the pkg-stats-new script:

 -n, to specify a number of packages to parse instead of all packages

 -p, to specify a list of packages (comma-separated) to parse instead
     of all packages

These options are basically only useful when debugging/developping
this script, but they are very useful, because the script is rather
slow to run completely with all 2000+ packages, especially once
upstream versions will be fetched from release-monitoring.org.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
---
Changes since v1:
- Fix flake8 warnings
---
 support/scripts/pkg-stats-new | 25 +++++++++++++++++++++++--
 1 file changed, 23 insertions(+), 2 deletions(-)

diff --git a/support/scripts/pkg-stats-new b/support/scripts/pkg-stats-new
index 8c74ff2a37..85a6caeeb9 100755
--- a/support/scripts/pkg-stats-new
+++ b/support/scripts/pkg-stats-new
@@ -23,6 +23,7 @@ import os
 from collections import defaultdict
 import re
 import subprocess
+import sys
 
 
 class Package:
@@ -47,11 +48,14 @@ class Package:
             (self.name, self.path, self.has_license, self.has_license_files, self.has_hash, self.patch_count)
 
 
-def get_pkglist():
+def get_pkglist(npackages, package_list):
     """
     Builds the list of Buildroot packages, returning a list of Package
     objects. Only the .name and .path fields of the Package object are
     initialized.
+
+    npackages: limit to N packages
+    package_list: limit to those packages in this list
     """
     WALK_USEFUL_SUBDIRS = ["boot", "linux", "package", "toolchain"]
     WALK_EXCLUDES = ["boot/common.mk",
@@ -74,6 +78,7 @@ def get_pkglist():
                      "toolchain/helpers.mk",
                      "toolchain/toolchain-wrapper.mk"]
     packages = list()
+    count = 0
     for root, dirs, files in os.walk("."):
         rootdir = root.split("/")
         if len(rootdir) < 2:
@@ -85,6 +90,8 @@ def get_pkglist():
                 continue
             # Strip ending ".mk"
             pkgname = f[:-3]
+            if package_list and pkgname not in package_list:
+                continue
             pkgpath = os.path.join(root, f)
             skip = False
             for exclude in WALK_EXCLUDES:
@@ -96,6 +103,9 @@ def get_pkglist():
                 continue
             p = Package(pkgname, pkgpath)
             packages.append(p)
+            count += 1
+            if npackages and count == npackages:
+                return packages
     return packages
 
 
@@ -452,13 +462,24 @@ def parse_args():
     parser = argparse.ArgumentParser()
     parser.add_argument('-o', dest='output', action='store', required=True,
                         help='HTML output file')
+    parser.add_argument('-n', dest='npackages', type=int, action='store',
+                        help='Number of packages')
+    parser.add_argument('-p', dest='packages', action='store',
+                        help='List of packages')
     return parser.parse_args()
 
 
 def __main__():
     args = parse_args()
+    if args.npackages and args.packages:
+        print "ERROR: -n and -p are mutually exclusive"
+        sys.exit(1)
+    if args.packages:
+        package_list = args.packages.split(",")
+    else:
+        package_list = None
     print "Build package list ..."
-    packages = get_pkglist()
+    packages = get_pkglist(args.npackages, package_list)
     print "Get package infra ..."
     add_infra_info(packages)
     print "Get make info ..."
-- 
2.14.3

[Buildroot] [PATCH next v2 2/5] support/scripts/pkg-stats-new: add -n and -p options

[Ricardo Martincoski]

Hello,

On Wed, Feb 21, 2018 at 07:13 PM, Thomas Petazzoni wrote:

> This commit adds the following options to the pkg-stats-new script:
> 
>  -n, to specify a number of packages to parse instead of all packages
> 
>  -p, to specify a list of packages (comma-separated) to parse instead
>      of all packages
> 
> These options are basically only useful when debugging/developping

typo:                                                    developing

> this script, but they are very useful, because the script is rather
> slow to run completely with all 2000+ packages, especially once
> upstream versions will be fetched from release-monitoring.org.
> 
> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>

With the typo fixed and regardless your decision about the help below:
Reviewed-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>

[snip]
> @@ -452,13 +462,24 @@ def parse_args():
>      parser = argparse.ArgumentParser()
>      parser.add_argument('-o', dest='output', action='store', required=True,
>                          help='HTML output file')
> +    parser.add_argument('-n', dest='npackages', type=int, action='store',
> +                        help='Number of packages')
> +    parser.add_argument('-p', dest='packages', action='store',
> +                        help='List of packages')

Should the help also mention the option expects a comma-separated list?

Regards,
Ricardo

[Buildroot] [PATCH next v2 2/5] support/scripts/pkg-stats-new: add -n and -p options

[Thomas Petazzoni]

Hello,

On Sat, 24 Feb 2018 01:54:27 -0300, Ricardo Martincoski wrote:
> Hello,
> 
> On Wed, Feb 21, 2018 at 07:13 PM, Thomas Petazzoni wrote:
> 
> > This commit adds the following options to the pkg-stats-new script:
> > 
> >  -n, to specify a number of packages to parse instead of all packages
> > 
> >  -p, to specify a list of packages (comma-separated) to parse instead
> >      of all packages
> > 
> > These options are basically only useful when debugging/developping  
> 
> typo:                                                    developing

Fixed.

> 
> > this script, but they are very useful, because the script is rather
> > slow to run completely with all 2000+ packages, especially once
> > upstream versions will be fetched from release-monitoring.org.
> > 
> > Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>  
> 
> With the typo fixed and regardless your decision about the help below:
> Reviewed-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>

Thanks!

> > +    parser.add_argument('-n', dest='npackages', type=int, action='store',
> > +                        help='Number of packages')
> > +    parser.add_argument('-p', dest='packages', action='store',
> > +                        help='List of packages')  
> 
> Should the help also mention the option expects a comma-separated list?

Absolutely, fixed in v3!

Thomas
-- 
Thomas Petazzoni, CTO, Bootlin (formerly Free Electrons)
Embedded Linux and Kernel engineering
http://bootlin.com

[Buildroot] [PATCH next v2 3/5] support/scripts/pkg-stats-new: add current version information

[Thomas Petazzoni]

This commit adds a new column in the HTML output containing the
current version of a package in Buildroot. As such, it isn't terribly
useful, but combined with the latest upstream version added in a
follow-up commit, it will become very useful.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
---
Changes since v1:
- Fix flake8 warnings
- Pass BR2_HAVE_DOT_CONFIG=y when calling make, in order to fake
  having a .config. This allows "printvars" to dump all variables even
  without a .config.
- Add missing newline in HTML code
---
 support/scripts/pkg-stats-new | 27 +++++++++++++++++++++++++++
 1 file changed, 27 insertions(+)

diff --git a/support/scripts/pkg-stats-new b/support/scripts/pkg-stats-new
index 85a6caeeb9..c4174877aa 100755
--- a/support/scripts/pkg-stats-new
+++ b/support/scripts/pkg-stats-new
@@ -36,6 +36,7 @@ class Package:
         self.has_hash = False
         self.patch_count = 0
         self.warnings = 0
+        self.current_version = None
 
     def __eq__(self, other):
         return self.path == other.path
@@ -148,6 +149,7 @@ def add_pkg_make_info(packages):
     """
     licenses = list()
     license_files = list()
+    versions = dict()
 
     # Licenses
     o = subprocess.check_output(["make", "BR2_HAVE_DOT_CONFIG=y",
@@ -187,12 +189,33 @@ def add_pkg_make_info(packages):
 
         license_files.append(pkgvar)
 
+    # Version
+    o = subprocess.check_output(["make", "BR2_HAVE_DOT_CONFIG=y",
+                                 "-s", "printvars", "VARS=%_VERSION"])
+    for l in o.splitlines():
+        # Get variable name and value
+        pkgvar, value = l.split("=")
+
+        # If present, strip HOST_ from variable name
+        if pkgvar.startswith("HOST_"):
+            pkgvar = pkgvar[5:]
+
+        if pkgvar.endswith("_DL_VERSION"):
+            continue
+
+        # Strip _VERSION
+        pkgvar = pkgvar[:-8]
+
+        versions[pkgvar] = value
+
     for pkg in packages:
         var = pkgname_to_pkgvar(pkg.name)
         if var in licenses:
             pkg.has_license = True
         if var in license_files:
             pkg.has_license_files = True
+        if var in versions:
+            pkg.current_version = versions[var]
 
 
 def add_hash_info(packages):
@@ -387,6 +410,9 @@ def dump_html_pkg(f, pkg):
     f.write("  <td class=\"%s\">%s</td>\n" %
             (" ".join(td_class), boolean_str(pkg.has_hash)))
 
+    # Current version
+    f.write("  <td class=\"centered\">%s</td>\n" % pkg.current_version)
+
     # Warnings
     td_class = ["centered"]
     if pkg.warnings == 0:
@@ -409,6 +435,7 @@ def dump_html_all_pkgs(f, packages):
 <td class=\"centered\">License</td>
 <td class=\"centered\">License files</td>
 <td class=\"centered\">Hash file</td>
+<td class=\"centered\">Current version</td>
 <td class=\"centered\">Warnings</td>
 </tr>
 """)
-- 
2.14.3

[Buildroot] [PATCH next v2 3/5] support/scripts/pkg-stats-new: add current version information

[Ricardo Martincoski]

Hello,

On Wed, Feb 21, 2018 at 07:13 PM, Thomas Petazzoni wrote:

[snip]
> @@ -187,12 +189,33 @@ def add_pkg_make_info(packages):
>  
>          license_files.append(pkgvar)
>  
> +    # Version
> +    o = subprocess.check_output(["make", "BR2_HAVE_DOT_CONFIG=y",
> +                                 "-s", "printvars", "VARS=%_VERSION"])
> +    for l in o.splitlines():
> +        # Get variable name and value
> +        pkgvar, value = l.split("=")
> +
> +        # If present, strip HOST_ from variable name
> +        if pkgvar.startswith("HOST_"):
> +            pkgvar = pkgvar[5:]

The output of printvars is sorted, so this makes the 'versions' dict to
contain at the very end:
 1) the target version when there is no host version
 2) the host version for packages alphabetically before hostapd
 3) the target version for packages alphabetically after hplip
 4) the host version when there is no target version

The inconsistency between 2 and 3 is not a problem right now because all
packages that have host and target versions use the same versions for both.
Maybe it will never be a problem?

I hope Yann can help us here (I added to CC).

The case 4 seems to serve a single package: lpc3250loader
We have other packages that are host-only, i.e. vboot-utils, they set _VERSION,
not HOST.*_VERSION.
Maybe it is something to fix in the package?
If yes, you could just ignore the entries starting with HOST_.

> +
> +        if pkgvar.endswith("_DL_VERSION"):
> +            continue
> +        if pkgvar.startswith("HOST_"):
> +            pkgvar = pkgvar[5:]
> +
> +        # Strip _VERSION
> +        pkgvar = pkgvar[:-8]
> +
> +        versions[pkgvar] = value
[snip]
> @@ -387,6 +410,9 @@ def dump_html_pkg(f, pkg):
>      f.write("  <td class=\"%s\">%s</td>\n" %
>              (" ".join(td_class), boolean_str(pkg.has_hash)))
>  
> +    # Current version
> +    f.write("  <td class=\"centered\">%s</td>\n" % pkg.current_version)

This column shows raw sha1 for git packages.
Wouldn't be visually better to show only the first, say 20, characters?
At the end, it's a matter of option.

If more people have the same opinion you could cut the string after checking
with a regexp for 40 or 39 (see sunxi-mali) hexa digits.

Anyway it can be done later.

Regards,
Ricardo

[Buildroot] [PATCH next v2 3/5] support/scripts/pkg-stats-new: add current version information

[Thomas Petazzoni]

Hello,

Thanks for your review!

On Sun, 25 Feb 2018 21:47:48 -0300, Ricardo Martincoski wrote:

> The output of printvars is sorted, so this makes the 'versions' dict to
> contain at the very end:
>  1) the target version when there is no host version
>  2) the host version for packages alphabetically before hostapd
>  3) the target version for packages alphabetically after hplip
>  4) the host version when there is no target version
> 
> The inconsistency between 2 and 3 is not a problem right now because all
> packages that have host and target versions use the same versions for both.
> Maybe it will never be a problem?
> 
> I hope Yann can help us here (I added to CC).
> 
> The case 4 seems to serve a single package: lpc3250loader
> We have other packages that are host-only, i.e. vboot-utils, they set _VERSION,
> not HOST.*_VERSION.
> Maybe it is something to fix in the package?
> If yes, you could just ignore the entries starting with HOST_.

I've addressed your concern by doing this:

+    # We process first the host package VERSION, and then the target
+    # package VERSION. This means that if a package exists in both
+    # target and host variants, with different version numbers
+    # (unlikely), we'll report the target version number.
+    version_list = o.splitlines()
+    version_list = [ x for x in version_list if x.startswith("HOST_") ] + \
+                   [ x for x in version_list if not x.startswith("HOST_") ]
+    for l in version_list:

This means that we will first process all the HOST_*_VERSION variables,
filling up the dict with the version numbers for the host packages, and
then we will process the *_VERSION variables.

With this:

 - If the package specifies only a target version, the target version
   is saved.

 - If the package specifies only a host version, the host version is
   saved.

 - If the package specifies both a target version and a host version,
   the target version always wins. So the scripts doesn't handle
   differentiating host and target versions for the same package, but
   at least we are consistent in the fact that we will always report
   the target version.

How does that sound ?

> This column shows raw sha1 for git packages.
> Wouldn't be visually better to show only the first, say 20, characters?
> At the end, it's a matter of option.
> 
> If more people have the same opinion you could cut the string after checking
> with a regexp for 40 or 39 (see sunxi-mali) hexa digits.

I'm just cutting after 20 characters, and adding "..." at the end, and
it seems good enough to me (compared to doing a regexp to try to guess
if it's a git hash or not).

Best regards,

Thomas
-- 
Thomas Petazzoni, CTO, Bootlin (formerly Free Electrons)
Embedded Linux and Kernel engineering
http://bootlin.com

[Buildroot] [PATCH next v2 3/5] support/scripts/pkg-stats-new: add current version information

[Ricardo Martincoski]

Hello,

On Wed, Mar 07, 2018 at 07:25 PM, Thomas Petazzoni wrote:

> On Sun, 25 Feb 2018 21:47:48 -0300, Ricardo Martincoski wrote:
> 
>> The output of printvars is sorted, so this makes the 'versions' dict to
>> contain at the very end:
>>  1) the target version when there is no host version
>>  2) the host version for packages alphabetically before hostapd
>>  3) the target version for packages alphabetically after hplip
>>  4) the host version when there is no target version
>> 
>> The inconsistency between 2 and 3 is not a problem right now because all
>> packages that have host and target versions use the same versions for both.
>> Maybe it will never be a problem?
>> 
>> I hope Yann can help us here (I added to CC).
>> 
>> The case 4 seems to serve a single package: lpc3250loader
>> We have other packages that are host-only, i.e. vboot-utils, they set _VERSION,
>> not HOST.*_VERSION.
>> Maybe it is something to fix in the package?
>> If yes, you could just ignore the entries starting with HOST_.
> 
> I've addressed your concern by doing this:
> 
> +    # We process first the host package VERSION, and then the target
> +    # package VERSION. This means that if a package exists in both
> +    # target and host variants, with different version numbers
> +    # (unlikely), we'll report the target version number.
> +    version_list = o.splitlines()
> +    version_list = [ x for x in version_list if x.startswith("HOST_") ] + \
> +                   [ x for x in version_list if not x.startswith("HOST_") ]
> +    for l in version_list:
> 
> This means that we will first process all the HOST_*_VERSION variables,
> filling up the dict with the version numbers for the host packages, and
> then we will process the *_VERSION variables.
> 
> With this:
> 
>  - If the package specifies only a target version, the target version
>    is saved.
> 
>  - If the package specifies only a host version, the host version is
>    saved.
> 
>  - If the package specifies both a target version and a host version,
>    the target version always wins. So the scripts doesn't handle
>    differentiating host and target versions for the same package, but
>    at least we are consistent in the fact that we will always report
>    the target version.
> 
> How does that sound ?

Sounds good.
Nit: please recheck with flake8.

> 
>> This column shows raw sha1 for git packages.
>> Wouldn't be visually better to show only the first, say 20, characters?
>> At the end, it's a matter of option.
>> 
>> If more people have the same opinion you could cut the string after checking
>> with a regexp for 40 or 39 (see sunxi-mali) hexa digits.
> 
> I'm just cutting after 20 characters, and adding "..." at the end, and
> it seems good enough to me (compared to doing a regexp to try to guess
> if it's a git hash or not).

OK since you are only changing the display of the version, not the string that
will be compared to the new column.

The few version strings in the tree that are not sha1 larger than 20 are:
2.0.0.alpha20140727b
kvm-unit-tests-20171020
rel_imx_4.9.x_1.0.0_ga

For me these are OK:
2.0.0.alpha20140727...
kvm-unit-tests-2017...
rel_imx_4.9.x_1.0.0...


Regards,
Ricardo

[Buildroot] [PATCH next v2 3/5] support/scripts/pkg-stats-new: add current version information

[Thomas Petazzoni]

Hello,

On Thu, 08 Mar 2018 00:14:51 -0300, Ricardo Martincoski wrote:

> > With this:
> > 
> >  - If the package specifies only a target version, the target version
> >    is saved.
> > 
> >  - If the package specifies only a host version, the host version is
> >    saved.
> > 
> >  - If the package specifies both a target version and a host version,
> >    the target version always wins. So the scripts doesn't handle
> >    differentiating host and target versions for the same package, but
> >    at least we are consistent in the fact that we will always report
> >    the target version.
> > 
> > How does that sound ?  
> 
> Sounds good.
> Nit: please recheck with flake8.

Will do.


> > I'm just cutting after 20 characters, and adding "..." at the end, and
> > it seems good enough to me (compared to doing a regexp to try to guess
> > if it's a git hash or not).  
> 
> OK since you are only changing the display of the version, not the string that
> will be compared to the new column.
> 
> The few version strings in the tree that are not sha1 larger than 20 are:
> 2.0.0.alpha20140727b
> kvm-unit-tests-20171020
> rel_imx_4.9.x_1.0.0_ga
> 
> For me these are OK:
> 2.0.0.alpha20140727...
> kvm-unit-tests-2017...
> rel_imx_4.9.x_1.0.0...

Yes, I'm only changing the display of the version, and I'm also adding
"..." which helps the reader understand that the version string has
been cut.

Best regards,

Thomas
-- 
Thomas Petazzoni, CTO, Bootlin (formerly Free Electrons)
Embedded Linux and Kernel engineering
http://bootlin.com

[Buildroot] [PATCH next v2 4/5] support/scripts/pkg-stats-new: add latest upstream version information

[Thomas Petazzoni]

This commit adds fetching the latest upstream version of each package
from release-monitoring.org.

The fetching process first tries to use the package mappings of the
"Buildroot" distribution [1]. If there is no result, then it does a
regular search, and within the search results, looks for a package
whose name matches the Buildroot name.

Since release-monitoring.org is a bit slow, we have 8 threads that
fetch information in parallel.

From an output point of view, the latest version column:

 - Is green when the version in Buildroot matches the latest upstream
   version

 - Is orange when the latest upstream version is unknown because the
   package was not found on release-monitoring.org

 - Is red when the version in Buildroot doesn't match the latest
   upstream version. Note that we are not doing anything smart here:
   we are just testing if the strings are equal or not.

 - The cell contains the link to the project on release-monitoring.org
   if found.

 - The cell indicates if the match was done using a distro mapping, or
   through a regular search.

[1] https://release-monitoring.org/distro/Buildroot/

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
---
Changes since v1:
- Fix flake8 warnings
- Add missing newline in HTML
---
 support/scripts/pkg-stats-new | 142 ++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 142 insertions(+)

diff --git a/support/scripts/pkg-stats-new b/support/scripts/pkg-stats-new
index c4174877aa..b27f4df9a8 100755
--- a/support/scripts/pkg-stats-new
+++ b/support/scripts/pkg-stats-new
@@ -24,6 +24,10 @@ from collections import defaultdict
 import re
 import subprocess
 import sys
+import json
+import urllib2
+from Queue import Queue
+from threading import Thread
 
 
 class Package:
@@ -37,6 +41,7 @@ class Package:
         self.patch_count = 0
         self.warnings = 0
         self.current_version = None
+        self.latest_version = None
 
     def __eq__(self, other):
         return self.path == other.path
@@ -259,6 +264,83 @@ def add_check_package_warnings(packages):
         pkg.warnings = get_check_package_warnings(os.path.dirname(pkg.path))
 
 
+RELEASE_MONITORING_API = "http://release-monitoring.org/api"
+
+
+def get_latest_version_by_distro(package):
+    req = urllib2.Request(os.path.join(RELEASE_MONITORING_API, "project", "Buildroot", package))
+    f = urllib2.urlopen(req)
+    data = json.loads(f.read())
+    if len(data['versions']) > 0:
+        return (True, data['versions'][0], data['id'])
+    else:
+        return (True, None, data['id'])
+
+
+def get_latest_version_by_guess(package):
+    req = urllib2.Request(os.path.join(RELEASE_MONITORING_API, "projects", "?pattern=%s" % package))
+    f = urllib2.urlopen(req)
+    data = json.loads(f.read())
+    for p in data['projects']:
+        if p['name'] == package and len(p['versions']) > 0:
+            return (False, p['versions'][0], p['id'])
+    return (False, None, None)
+
+
+def get_latest_version(package):
+    try:
+        # We first try by using the "Buildroot" distribution on
+        # release-monitoring.org, if it has a mapping for the current
+        # package name.
+        return get_latest_version_by_distro(package)
+    except urllib2.HTTPError, e:
+        # If that fails because there is no mapping, we try to search
+        # in all packages for a package of this name.
+        if e.code == 404:
+            return get_latest_version_by_guess(package)
+        else:
+            return (False, None, None)
+
+
+def get_version_worker(q):
+    while True:
+        pkg = q.get()
+        try:
+            pkg.latest_version = get_latest_version(pkg.name)
+            print " [%04d] %s => %s" % (q.qsize(), pkg.name, str(pkg.latest_version))
+        except ValueError:
+            pkg.latest_version = (False, None, None)
+            print " [%04d] %s => Value Error" % (q.qsize(), pkg.name)
+        q.task_done()
+
+
+def add_latest_version_info(packages):
+    """
+    Fills in the .latest_version field of all Package objects
+
+    This field has a special format:
+      (mapping, version, id)
+    with:
+    - mapping: boolean that indicates whether release-monitoring.org
+      has a mapping for this package name in the Buildroot distribution
+      or not
+    - version: string containing the latest version known by
+      release-monitoring.org for this package
+    - id: string containing the id of the project corresponding to this
+      package, as known by release-monitoring.org
+    """
+    q = Queue()
+    for pkg in packages:
+        q.put(pkg)
+    # Since release-monitoring.org is rather slow, we create 8 threads
+    # that do HTTP requests to the site.
+    for i in range(8):
+        t = Thread(target=get_version_worker, args=[q])
+        t.daemon = True
+        t.start()
+    q.join()
+
+
 def calculate_stats(packages):
     stats = defaultdict(int)
     for pkg in packages:
@@ -283,6 +365,16 @@ def calculate_stats(packages):
             stats["hash"] += 1
         else:
             stats["no-hash"] += 1
+        if pkg.latest_version[0]:
+            stats["rmo-mapping"] += 1
+        else:
+            stats["rmo-no-mapping"] += 1
+        if not pkg.latest_version[1]:
+            stats["version-unknown"] += 1
+        elif pkg.latest_version[1] == pkg.current_version:
+            stats["version-uptodate"] += 1
+        else:
+            stats["version-not-uptodate"] += 1
         stats["patches"] += pkg.patch_count
     return stats
 
@@ -315,6 +407,15 @@ td.somepatches {
 td.lotsofpatches {
   background: #ff9a69;
 }
+td.version-good {
+  background: #d2ffc4;
+}
+td.version-needs-update {
+  background: #ff9a69;
+}
+td.version-unknown {
+ background: #ffd870;
+}
 </style>
 <title>Statistics of Buildroot packages</title>
 </head>
@@ -413,6 +514,34 @@ def dump_html_pkg(f, pkg):
     # Current version
     f.write("  <td class=\"centered\">%s</td>\n" % pkg.current_version)
 
+    # Latest version
+    if pkg.latest_version[1] is None:
+        td_class.append("version-unknown")
+    elif pkg.latest_version[1] != pkg.current_version:
+        td_class.append("version-needs-update")
+    else:
+        td_class.append("version-good")
+
+    if pkg.latest_version[1] is None:
+        latest_version_text = "<b>Unknown</b>"
+    else:
+        latest_version_text = "<b>%s</b>" % str(pkg.latest_version[1])
+
+    latest_version_text += "<br/>"
+
+    if pkg.latest_version[2]:
+        latest_version_text += "<a href=\"https://release-monitoring.org/project/%s\">link</a>, " % pkg.latest_version[2]
+    else:
+        latest_version_text += "no link, "
+
+    if pkg.latest_version[0]:
+        latest_version_text += "has <a href=\"https://release-monitoring.org/distro/Buildroot/\">mapping</a>"
+    else:
+        latest_version_text += "has <a href=\"https://release-monitoring.org/distro/Buildroot/\">no mapping</a>"
+
+    f.write("  <td class=\"%s\">%s</td>\n" %
+            (" ".join(td_class), latest_version_text))
+
     # Warnings
     td_class = ["centered"]
     if pkg.warnings == 0:
@@ -436,6 +565,7 @@ def dump_html_all_pkgs(f, packages):
 <td class=\"centered\">License files</td>
 <td class=\"centered\">Hash file</td>
 <td class=\"centered\">Current version</td>
+<td class=\"centered\">Latest version</td>
 <td class=\"centered\">Warnings</td>
 </tr>
 """)
@@ -465,6 +595,16 @@ def dump_html_stats(f, stats):
             stats["no-hash"])
     f.write(" <tr><td>Total number of patches</td><td>%s</td></tr>\n" %
             stats["patches"])
+    f.write("<tr><td>Packages having a mapping on <i>release-monitoring.org</i></td><td>%s</td></tr>\n" %
+            stats["rmo-mapping"])
+    f.write("<tr><td>Packages lacking a mapping on <i>release-monitoring.org</i></td><td>%s</td></tr>\n" %
+            stats["rmo-no-mapping"])
+    f.write("<tr><td>Packages that are up-to-date</td><td>%s</td></tr>\n" %
+            stats["version-uptodate"])
+    f.write("<tr><td>Packages that are not up-to-date</td><td>%s</td></tr>\n" %
+            stats["version-not-uptodate"])
+    f.write("<tr><td>Packages with no known upstream version</td><td>%s</td></tr>\n" %
+            stats["version-unknown"])
     f.write("</table>\n")
 
 
@@ -517,6 +657,8 @@ def __main__():
     add_patch_count(packages)
     print "Get package warnings ..."
     add_check_package_warnings(packages)
+    print "Get latest version ..."
+    add_latest_version_info(packages)
     print "Calculate stats"
     stats = calculate_stats(packages)
     print "Write HTML"
-- 
2.14.3

[Buildroot] [PATCH next v2 4/5] support/scripts/pkg-stats-new: add latest upstream version information

[Ricardo Martincoski]

Hello,

This is the last part of my review for v2, as patch 5 is just the rename.

When running the script with a few packages it works well.
But when running the script for all packages, for me it often hangs at the end
(2 out of 5 tries).
It can be related to my setup (python module versions?, internet connection?)
But anyway see later in this email an alternative approach.

When it hangs, by scrolling back I see (in a different place for the each run
that hung):
[snip]
 [1453] python-autobahn => (False, None, None)
Exception in thread Thread-6:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 754, in run
    self.__target(*self.__args, **self.__kwargs)
  File "support/scripts/pkg-stats-new", line 309, in get_version_worker
    pkg.latest_version = get_latest_version(pkg.name)
  File "support/scripts/pkg-stats-new", line 300, in get_latest_version
    return get_latest_version_by_guess(package)
  File "support/scripts/pkg-stats-new", line 282, in get_latest_version_by_guess
    f = urllib2.urlopen(req)
  File "/usr/lib/python2.7/urllib2.py", line 154, in urlopen
    return opener.open(url, data, timeout)
  File "/usr/lib/python2.7/urllib2.py", line 435, in open
    response = meth(req, response)
  File "/usr/lib/python2.7/urllib2.py", line 548, in http_response
    'http', request, response, code, msg, hdrs)
  File "/usr/lib/python2.7/urllib2.py", line 467, in error
    result = self._call_chain(*args)
  File "/usr/lib/python2.7/urllib2.py", line 407, in _call_chain
    result = func(*args)
  File "/usr/lib/python2.7/urllib2.py", line 654, in http_error_302
    return self.parent.open(new, timeout=req.timeout)
  File "/usr/lib/python2.7/urllib2.py", line 435, in open
    response = meth(req, response)
  File "/usr/lib/python2.7/urllib2.py", line 548, in http_response
    'http', request, response, code, msg, hdrs)
  File "/usr/lib/python2.7/urllib2.py", line 473, in error
    return self._call_chain(*args)
  File "/usr/lib/python2.7/urllib2.py", line 407, in _call_chain
    result = func(*args)
  File "/usr/lib/python2.7/urllib2.py", line 556, in http_error_default
    raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
HTTPError: HTTP Error 404: NOT FOUND

 [1452] cups-filters => (False, u'1.20.1', 5541)
[snip]
 [0001] python-flask-login => (False, None, None)
 [0000] luabitop => (False, None, None)
 [0000] python-pyopenssl => (False, None, None)
 [0000] libsexy => (False, None, None)
 [0000] libnetfilter_conntrack => (False, u'1.0.6', 1675)
 [0000] doxygen => (False, u'1.8.14.windows.x64.bin', 457)
 [0000] vdr-plugin-vnsiserver => (False, None, None)
[hangs here, I waited 10 minutes]


On Wed, Feb 21, 2018 at 07:13 PM, Thomas Petazzoni wrote:

> This commit adds fetching the latest upstream version of each package
> from release-monitoring.org.
> 
> The fetching process first tries to use the package mappings of the
> "Buildroot" distribution [1]. If there is no result, then it does a
> regular search, and within the search results, looks for a package
> whose name matches the Buildroot name.
> 
> Since release-monitoring.org is a bit slow, we have 8 threads that
> fetch information in parallel.

For me:
patches 1 to 3:   1m37.798s
patches 1 to 4:  20m5.513s

Alternative:      3m7.943s

[snip]
> +RELEASE_MONITORING_API = "http://release-monitoring.org/api"
> +
> +
> +def get_latest_version_by_distro(package):
> +    req = urllib2.Request(os.path.join(RELEASE_MONITORING_API, "project", "Buildroot", package))

There is also an undocumented API:
http://release-monitoring.org/api/projects/?distro=Buildroot
It was implemented here:
https://github.com/release-monitoring/anitya/commit/f3d8be75a6643b5d8c55754b74ed4f74f71fe952
and will eventually be documented here:
https://github.com/release-monitoring/anitya/issues/420

When the Buildroot distro becomes 90% filled we could get something like this:
$ wget -O Alpine 'http://release-monitoring.org/api/projects/?distro=Alpine'
... 1MB, 15 seconds ...
$ grep -c '"name"' Alpine
1838

> +    f = urllib2.urlopen(req)
> +    data = json.loads(f.read())
> +    if len(data['versions']) > 0:
> +        return (True, data['versions'][0], data['id'])

Also there is a field 'version' that looks to contain the latest version (or
None if there is none). For the majority of projects it is exactly equal to
'versions'[0].
See the test below, but I didn't find a documentation to corroborate this.

$ wget http://release-monitoring.org/api/projects
... 7MB, 1 minute ...
$ cat check.py 
#!/usr/bin/env python
import json

with open("projects") as f:
    data = json.loads(f.read())
for p in data["projects"]:
    if p['version'] is None:
        continue
    if p['version'] == p['versions'][0]:
        continue
    print p['name'], p['version'], p['versions']
$ ./check.py | wc -l
109
$ grep -c '"name"' projects                                                                                        
16445

> +    else:
> +        return (True, None, data['id'])
[snip]
> +def add_latest_version_info(packages):
> +    """
> +    Fills in the .latest_version field of all Package objects
> +
> +    This field has a special format:
> +      (mapping, version, id)
> +    with:
> +    - mapping: boolean that indicates whether release-monitoring.org
> +      has a mapping for this package name in the Buildroot distribution
> +      or not
> +    - version: string containing the latest version known by
> +      release-monitoring.org for this package
> +    - id: string containing the id of the project corresponding to this
> +      package, as known by release-monitoring.org
> +    """

As an alternative approach we could first download 2 lists:
- all projects for distro (now 10, eventually 2000+)
- all projects (now 16000+)
It could even be saved to a file by one option and then loaded by other option
to speedup the development/maintenance.

But that would require:
The projects in the Buildroot distro to be named exactly as the buildroot
package, i.e. samba4, not samba. Or of course implementing the search by
similarity with regexp.

And be aware: it produces few different guesses (some better, some worst).

Here the sample code without Queue/threading:
    cache_latest_version_by_distro()
    cache_latest_version_by_guess()

    for pkg in packages:
        pkg.latest_version = get_latest_version(pkg.name)
        print " [%04d] %s => %s" % (packages.index(pkg), pkg.name, str(pkg.latest_version))


distro_list = None
guess_list = None


def cache_latest_version_by_distro():
    global distro_list
    req = urllib2.Request(os.path.join(RELEASE_MONITORING_API, "projects", "?distro=Buildroot"))
    f = urllib2.urlopen(req)
    distro_list = json.loads(f.read())


def cache_latest_version_by_guess():
    global guess_list
    req = urllib2.Request(os.path.join(RELEASE_MONITORING_API, "projects"))
    f = urllib2.urlopen(req)
    guess_list = json.loads(f.read())


def get_latest_version(package):
    # We first try by using the "Buildroot" distribution on
    # release-monitoring.org, if it has a mapping for the current
    # package name.
    for data in distro_list["projects"]:
        if data["name"] != package:
            continue
        if len(data['versions']) > 0:
            return (True, data['versions'][0], data['id'])
        else:
            return (True, None, data['id'])
    # If that fails because there is no mapping, we try to search
    # in all packages for a package of this name.
    for p in guess_list['projects']:
        if p['name'] == package and len(p['versions']) > 0:
            return (False, p['versions'][0], p['id'])
    return (False, None, None)

> +    q = Queue()
> +    for pkg in packages:
> +        q.put(pkg)
> +    # Since release-monitoring.org is rather slow, we create 8 threads
> +    # that do HTTP requests to the site.
> +    for i in range(8):
> +        t = Thread(target=get_version_worker, args=[q])
> +        t.daemon = True
> +        t.start()
> +    q.join()
[snip]
> @@ -413,6 +514,34 @@ def dump_html_pkg(f, pkg):
>      # Current version
>      f.write("  <td class=\"centered\">%s</td>\n" % pkg.current_version)
>  
> +    # Latest version
> +    if pkg.latest_version[1] is None:
> +        td_class.append("version-unknown")
> +    elif pkg.latest_version[1] != pkg.current_version:
> +        td_class.append("version-needs-update")
> +    else:
> +        td_class.append("version-good")
> +
> +    if pkg.latest_version[1] is None:
> +        latest_version_text = "<b>Unknown</b>"
> +    else:
> +        latest_version_text = "<b>%s</b>" % str(pkg.latest_version[1])
> +
> +    latest_version_text += "<br/>"
> +
> +    if pkg.latest_version[2]:
> +        latest_version_text += "<a href=\"https://release-monitoring.org/project/%s\">link</a>, " % pkg.latest_version[2]
> +    else:
> +        latest_version_text += "no link, "
> +
> +    if pkg.latest_version[0]:
> +        latest_version_text += "has <a href=\"https://release-monitoring.org/distro/Buildroot/\">mapping</a>"
> +    else:
> +        latest_version_text += "has <a href=\"https://release-monitoring.org/distro/Buildroot/\">no mapping</a>"

If you don't think it gets too ugly, you could invert the text, putting
'mapping'/'no mapping' as the first cell text. It should make sorting by this
column to show all 'mapping' together and 'no mapping' together. I did not
tested this.

The other way (the correct way) is to implement custom sort keys
https://www.kryogenix.org/code/browser/sorttable/#customkeys
to be done later, of course.

> +
> +    f.write("  <td class=\"%s\">%s</td>\n" %
> +            (" ".join(td_class), latest_version_text))
> +
[snip]

Regards,
Ricardo

[Buildroot] [PATCH next v2 4/5] support/scripts/pkg-stats-new: add latest upstream version information

[Thomas Petazzoni]

Hello,

On Wed, 28 Feb 2018 00:03:53 -0300, Ricardo Martincoski wrote:

> When running the script with a few packages it works well.
> But when running the script for all packages, for me it often hangs at the end
> (2 out of 5 tries).
> It can be related to my setup (python module versions?, internet connection?)
> But anyway see later in this email an alternative approach.

This could probably be solved by adding timeouts, but I find your
solution below interesting and useful.

> There is also an undocumented API:
> http://release-monitoring.org/api/projects/?distro=Buildroot
> It was implemented here:
> https://github.com/release-monitoring/anitya/commit/f3d8be75a6643b5d8c55754b74ed4f74f71fe952
> and will eventually be documented here:
> https://github.com/release-monitoring/anitya/issues/420
> 
> When the Buildroot distro becomes 90% filled we could get something like this:
> $ wget -O Alpine 'http://release-monitoring.org/api/projects/?distro=Alpine'
> ... 1MB, 15 seconds ...
> $ grep -c '"name"' Alpine
> 1838

Right, but it's going to take a while before we reach this :-)

> Also there is a field 'version' that looks to contain the latest version (or
> None if there is none). For the majority of projects it is exactly equal to
> 'versions'[0].
> See the test below, but I didn't find a documentation to corroborate this.

We can probably talk with the maintainers of release-monitoring.org
about this.

> As an alternative approach we could first download 2 lists:
> - all projects for distro (now 10, eventually 2000+)
> - all projects (now 16000+)
> It could even be saved to a file by one option and then loaded by other option
> to speedup the development/maintenance.

Yes, I'll do something like this: download both files to some
~/.release-monitoring-all-packages.cache and
~/.release-monitoring-buildroot-packages.cache, and use them if they
already exist. Then an option such as -r/--reload can force pkg-stats
to re-download the new version.

> But that would require:
> The projects in the Buildroot distro to be named exactly as the buildroot
> package, i.e. samba4, not samba.

Well, that is the whole point of "distros" on release-monitoring.org:
provide a mapping between the package name in the distribution and the
package name on release-monitoring.org.

The packages present in the "Buildroot" distro on
release-monitoring.org were added by me, often to fix a mismatch
between the Buildroot name and the release-monitoring.org name. samba4
(Buildroot) vs. samba (release-monitoring.org) is a good example.

> Or of course implementing the search by
> similarity with regexp.
> 
> And be aware: it produces few different guesses (some better, some worst).

I didn't understand this part. Why would the results be different ?

> > +    if pkg.latest_version[0]:
> > +        latest_version_text += "has <a href=\"https://release-monitoring.org/distro/Buildroot/\">mapping</a>"
> > +    else:
> > +        latest_version_text += "has <a href=\"https://release-monitoring.org/distro/Buildroot/\">no mapping</a>"  
> 
> If you don't think it gets too ugly, you could invert the text, putting
> 'mapping'/'no mapping' as the first cell text. It should make sorting by this
> column to show all 'mapping' together and 'no mapping' together. I did not
> tested this.

I understand the point, but I find it a bit ugly that this information
appears first in this cell. One option is to make this a separate
column, but we already have a lot of columns. Can we handle that later
on ?

Thanks!

Thomas
-- 
Thomas Petazzoni, CTO, Bootlin (formerly Free Electrons)
Embedded Linux and Kernel engineering
http://bootlin.com

[Buildroot] [PATCH next v2 4/5] support/scripts/pkg-stats-new: add latest upstream version information

[Ricardo Martincoski]

Hello,

On Wed, Mar 07, 2018 at 07:41 PM, Thomas Petazzoni wrote:

> On Wed, 28 Feb 2018 00:03:53 -0300, Ricardo Martincoski wrote:
> 
[snip]
>> As an alternative approach we could first download 2 lists:
>> - all projects for distro (now 10, eventually 2000+)
>> - all projects (now 16000+)
>> It could even be saved to a file by one option and then loaded by other option
>> to speedup the development/maintenance.
> 
> Yes, I'll do something like this: download both files to some
> ~/.release-monitoring-all-packages.cache and
> ~/.release-monitoring-buildroot-packages.cache, and use them if they
> already exist. Then an option such as -r/--reload can force pkg-stats
> to re-download the new version.

Sounds good.

[snip]
>> And be aware: it produces few different guesses (some better, some worst).
> 
> I didn't understand this part. Why would the results be different ?

Indeed, it shouldn't.

I am not sure why it does. It only happens for packages that have more than one
project with the exact same name. So we choose one guess over another.

You can check using this:

import json
import os
import subprocess
def print_id(list):
    for i in list['projects']:
        if i['name'] == 'readline':
            print(i['id'])
API = "http://release-monitoring.org/api"
url1 = os.path.join(API, "projects", "?pattern=%s" % 'readline')
url2 = os.path.join(API, "projects")
subprocess.call(['wget', '-O', 'readline', url1])
subprocess.call(['wget', '-O', 'projects', url2])
print_id(json.load(open('readline')))
print_id(json.load(open('projects')))

It gives to me most of times (3 out 4):
913
4173
4173
913
sometimes it gives me (1 out 4):
913
4173
913
4173
It seems to not always return the same order. If I run the above script few
times in a row the order is consistent. If I wait few hours the result can
change.

> 
>> > +    if pkg.latest_version[0]:
>> > +        latest_version_text += "has <a href=\"https://release-monitoring.org/distro/Buildroot/\">mapping</a>"
>> > +    else:
>> > +        latest_version_text += "has <a href=\"https://release-monitoring.org/distro/Buildroot/\">no mapping</a>"  
>> 
>> If you don't think it gets too ugly, you could invert the text, putting
>> 'mapping'/'no mapping' as the first cell text. It should make sorting by this
>> column to show all 'mapping' together and 'no mapping' together. I did not
>> tested this.
> 
> I understand the point, but I find it a bit ugly that this information
> appears first in this cell. One option is to make this a separate
> column, but we already have a lot of columns. Can we handle that later
> on ?

Sure.

Regards,
Ricardo

[Buildroot] [PATCH next v2 4/5] support/scripts/pkg-stats-new: add latest upstream version information

[Thomas Petazzoni]

Hello,

On Thu, 08 Mar 2018 06:52:49 -0300, Ricardo Martincoski wrote:

> > I didn't understand this part. Why would the results be different ?  
> 
> Indeed, it shouldn't.
> 
> I am not sure why it does. It only happens for packages that have more than one
> project with the exact same name. So we choose one guess over another.
> 
> You can check using this:
> 
> import json
> import os
> import subprocess
> def print_id(list):
>     for i in list['projects']:
>         if i['name'] == 'readline':
>             print(i['id'])
> API = "http://release-monitoring.org/api"
> url1 = os.path.join(API, "projects", "?pattern=%s" % 'readline')
> url2 = os.path.join(API, "projects")
> subprocess.call(['wget', '-O', 'readline', url1])
> subprocess.call(['wget', '-O', 'projects', url2])
> print_id(json.load(open('readline')))
> print_id(json.load(open('projects')))
> 
> It gives to me most of times (3 out 4):
> 913
> 4173
> 4173
> 913
> sometimes it gives me (1 out 4):
> 913
> 4173
> 913
> 4173
> It seems to not always return the same order. If I run the above script few
> times in a row the order is consistent. If I wait few hours the result can
> change.

Perhaps we should report this upstream to the people maintaining
release-monitoring.org ?

Thomas
-- 
Thomas Petazzoni, CTO, Bootlin (formerly Free Electrons)
Embedded Linux and Kernel engineering
http://bootlin.com

[Buildroot] [PATCH next v2 4/5] support/scripts/pkg-stats-new: add latest upstream version information

[Ricardo Martincoski]

Hello,

On Thu, Mar 08, 2018 at 06:56 AM, Thomas Petazzoni wrote:

[snip]
>> It seems to not always return the same order. If I run the above script few
>> times in a row the order is consistent. If I wait few hours the result can
>> change.
> 
> Perhaps we should report this upstream to the people maintaining
> release-monitoring.org ?

I created:
https://github.com/release-monitoring/anitya/issues/533


Also, regarding the 'version' field, I found these lines of code:

class Project(Base):
...
        latest_version (sa.Boolean): The latest version for the project, as determined
            by the version sorting algorithm.
...
    def __json__(self, detailed=False):
...
            version=self.latest_version,

https://github.com/release-monitoring/anitya/blob/a0a9343dd86d9b7d41487b241f8e102d3321deee/anitya/db/models.py#L415
(PS: it's not a boolean of course, the comment is wrong in this aspect)


Regards,
Ricardo

[Buildroot] [PATCH next v2 4/5] support/scripts/pkg-stats-new: add latest upstream version information

[Thomas Petazzoni]

Hello Ricardo,

On Wed, 28 Feb 2018 00:03:53 -0300, Ricardo Martincoski wrote:

> But that would require:
> The projects in the Buildroot distro to be named exactly as the buildroot
> package, i.e. samba4, not samba. Or of course implementing the search by
> similarity with regexp.
> 
> And be aware: it produces few different guesses (some better, some worst).
> 
> Here the sample code without Queue/threading:
>     cache_latest_version_by_distro()
>     cache_latest_version_by_guess()
> 
>     for pkg in packages:
>         pkg.latest_version = get_latest_version(pkg.name)
>         print " [%04d] %s => %s" % (packages.index(pkg), pkg.name, str(pkg.latest_version))
> 
> 
> distro_list = None
> guess_list = None
> 
> 
> def cache_latest_version_by_distro():
>     global distro_list
>     req = urllib2.Request(os.path.join(RELEASE_MONITORING_API, "projects", "?distro=Buildroot"))
>     f = urllib2.urlopen(req)
>     distro_list = json.loads(f.read())
> 
> 
> def cache_latest_version_by_guess():
>     global guess_list
>     req = urllib2.Request(os.path.join(RELEASE_MONITORING_API, "projects"))
>     f = urllib2.urlopen(req)
>     guess_list = json.loads(f.read())
> 
> 
> def get_latest_version(package):
>     # We first try by using the "Buildroot" distribution on
>     # release-monitoring.org, if it has a mapping for the current
>     # package name.
>     for data in distro_list["projects"]:
>         if data["name"] != package:
>             continue
>         if len(data['versions']) > 0:
>             return (True, data['versions'][0], data['id'])
>         else:
>             return (True, None, data['id'])
>     # If that fails because there is no mapping, we try to search
>     # in all packages for a package of this name.
>     for p in guess_list['projects']:
>         if p['name'] == package and len(p['versions']) > 0:
>             return (False, p['versions'][0], p['id'])
>     return (False, None, None)

I looked more closely into this, and unfortunately, it doesn't work.
Indeed, the data returned by the URL:

  RELEASE_MONITORING_API/projects/?distro=Buildroot

does not take into account the mapping between Buildroot package names
and release-monitoring.org package names, it only contains the
release-monitoring.org package names.

For example, we have a mapping samba4 -> samba, that allows
https://release-monitoring.org/api/project/Buildroot/samba4 to return
the expected result. But you won't find anything named "samba4" in
https://release-monitoring.org/api/projects/?distro=Buildroot. This
makes the entire concept of distro mapping useless.

Therefore, your proposal cannot work with the data that we can
currently download from release-monitoring.org. I have asked on the
#fedora-apps IRC channel about this.

Perhaps we could go with my v2 version on this aspect (i.e make HTTP
requests for each package), and change that later on if a better
solution is found ?

Best regards,

Thomas
-- 
Thomas Petazzoni, CTO, Bootlin (formerly Free Electrons)
Embedded Linux and Kernel engineering
https://bootlin.com

[Buildroot] [PATCH next v2 4/5] support/scripts/pkg-stats-new: add latest upstream version information

[Ricardo Martincoski]

Hello,

On Wed, Mar 21, 2018 at 05:58 PM, Thomas Petazzoni wrote:

> I looked more closely into this, and unfortunately, it doesn't work.
> Indeed, the data returned by the URL:
> 
>   RELEASE_MONITORING_API/projects/?distro=Buildroot
> 
> does not take into account the mapping between Buildroot package names
> and release-monitoring.org package names, it only contains the
> release-monitoring.org package names.

Indeed. I missed it was missing.

> 
> For example, we have a mapping samba4 -> samba, that allows
> https://release-monitoring.org/api/project/Buildroot/samba4 to return
> the expected result. But you won't find anything named "samba4" in
> https://release-monitoring.org/api/projects/?distro=Buildroot. This
> makes the entire concept of distro mapping useless.
> 
> Therefore, your proposal cannot work with the data that we can
> currently download from release-monitoring.org. I have asked on the
> #fedora-apps IRC channel about this.
> 
> Perhaps we could go with my v2 version on this aspect (i.e make HTTP
> requests for each package), and change that later on if a better
> solution is found ?

I dislike the fact we make 2000+ HTTP requests, but as you explained it is a
limitation from the current api exposed by the upstream. I even looked at the
code hoping the mapping could be easily introduced in api v1... but no, it is
not trivial.

So, yes. Let's do this incrementally, first let's make it work using one request
per package, then later we improve the script when the api v2 is deployed.

In the meanwhile, we will be able to start creating the mappings based on the
daily generated html.

Regards,
Ricardo

[Buildroot] [PATCH next v2 4/5] support/scripts/pkg-stats-new: add latest upstream version information

[Thomas Petazzoni]

Hello Ricardo,

On Thu, 22 Mar 2018 00:11:09 -0300, Ricardo Martincoski wrote:

> I dislike the fact we make 2000+ HTTP requests, but as you explained it is a
> limitation from the current api exposed by the upstream. I even looked at the
> code hoping the mapping could be easily introduced in api v1... but no, it is
> not trivial.

I think it is easy. It should just be a matter of the following patch:

diff --git a/anitya/api.py b/anitya/api.py
index a53be43..5f0be9e 100644
--- a/anitya/api.py
+++ b/anitya/api.py
@@ -156,7 +156,7 @@ def api_projects():
     else:
         project_objs = models.Project.all(Session)
 
-    projects = [project.__json__() for project in project_objs]
+    projects = [project.__json__(detailed=True) for project in project_objs]
 
     output = {
         'total': len(projects),

I've asked upstream if they were interested, and the reply I got this
morning is:

08:48 < pingou> kos_tom: sounds fine to me, but I know that jcline has been working on a v2 api, so maybe that'll help
08:49 < kos_tom> pingou: do you think you can apply the patch I provided ?
08:49 < kos_tom> pingou: alternatively, we could make getting the detailed information optional
08:49 < pingou> kos_tom: jcline is now the one maintaining it
08:52 < kos_tom> pingou: ok :)

> So, yes. Let's do this incrementally, first let's make it work using one request
> per package, then later we improve the script when the api v2 is deployed.

Right.

> In the meanwhile, we will be able to start creating the mappings based on the
> daily generated html.

Indeed.

Best regards,

Thomas
-- 
Thomas Petazzoni, CTO, Bootlin (formerly Free Electrons)
Embedded Linux and Kernel engineering
https://bootlin.com

[Buildroot] [PATCH next v2 4/5] support/scripts/pkg-stats-new: add latest upstream version information

[Thomas Petazzoni]

Hello,

On Wed, 28 Feb 2018 00:03:53 -0300, Ricardo Martincoski wrote:

> When it hangs, by scrolling back I see (in a different place for the each run
> that hung):
> [snip]
>  [1453] python-autobahn => (False, None, None)
> Exception in thread Thread-6:
> Traceback (most recent call last):
>   File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner
>     self.run()
>   File "/usr/lib/python2.7/threading.py", line 754, in run
>     self.__target(*self.__args, **self.__kwargs)
>   File "support/scripts/pkg-stats-new", line 309, in get_version_worker
>     pkg.latest_version = get_latest_version(pkg.name)
>   File "support/scripts/pkg-stats-new", line 300, in get_latest_version
>     return get_latest_version_by_guess(package)
>   File "support/scripts/pkg-stats-new", line 282, in get_latest_version_by_guess
>     f = urllib2.urlopen(req)
>   File "/usr/lib/python2.7/urllib2.py", line 154, in urlopen
>     return opener.open(url, data, timeout)
>   File "/usr/lib/python2.7/urllib2.py", line 435, in open
>     response = meth(req, response)
>   File "/usr/lib/python2.7/urllib2.py", line 548, in http_response
>     'http', request, response, code, msg, hdrs)
>   File "/usr/lib/python2.7/urllib2.py", line 467, in error
>     result = self._call_chain(*args)
>   File "/usr/lib/python2.7/urllib2.py", line 407, in _call_chain
>     result = func(*args)
>   File "/usr/lib/python2.7/urllib2.py", line 654, in http_error_302
>     return self.parent.open(new, timeout=req.timeout)
>   File "/usr/lib/python2.7/urllib2.py", line 435, in open
>     response = meth(req, response)
>   File "/usr/lib/python2.7/urllib2.py", line 548, in http_response
>     'http', request, response, code, msg, hdrs)
>   File "/usr/lib/python2.7/urllib2.py", line 473, in error
>     return self._call_chain(*args)
>   File "/usr/lib/python2.7/urllib2.py", line 407, in _call_chain
>     result = func(*args)
>   File "/usr/lib/python2.7/urllib2.py", line 556, in http_error_default
>     raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
> HTTPError: HTTP Error 404: NOT FOUND
> 
>  [1452] cups-filters => (False, u'1.20.1', 5541)
> [snip]
>  [0001] python-flask-login => (False, None, None)
>  [0000] luabitop => (False, None, None)
>  [0000] python-pyopenssl => (False, None, None)
>  [0000] libsexy => (False, None, None)
>  [0000] libnetfilter_conntrack => (False, u'1.0.6', 1675)
>  [0000] doxygen => (False, u'1.8.14.windows.x64.bin', 457)
>  [0000] vdr-plugin-vnsiserver => (False, None, None)
> [hangs here, I waited 10 minutes]

I should have fixed this, will be in v3. I'm now handling exceptions in
all cases, and I've added a timeout on the urllib2.urlopen() calls, that
will make it abort after 15 seconds if the HTTP request has not
returned.

This will allow to make sure the script terminates properly. However,
it means that the result of the script may be different from one run to
the other, as the HTTP request for a given package may sometimes take
more than 15 seconds, sometimes not.

I guess this is a good enough trade-off, until upstream provides us a
better way of retrieving the data.

Best regards,

Thomas
-- 
Thomas Petazzoni, CTO, Bootlin (formerly Free Electrons)
Embedded Linux and Kernel engineering
https://bootlin.com

[Buildroot] [PATCH next v2 4/5] support/scripts/pkg-stats-new: add latest upstream version information

[Ricardo Martincoski]

Hello,

On Wed, Mar 21, 2018 at 06:35 PM, Thomas Petazzoni wrote:

> On Wed, 28 Feb 2018 00:03:53 -0300, Ricardo Martincoski wrote:
> 
>>  [0000] vdr-plugin-vnsiserver => (False, None, None)
>> [hangs here, I waited 10 minutes]
> 
> I should have fixed this, will be in v3. I'm now handling exceptions in
> all cases, and I've added a timeout on the urllib2.urlopen() calls, that
> will make it abort after 15 seconds if the HTTP request has not
> returned.
> 
> This will allow to make sure the script terminates properly. However,
> it means that the result of the script may be different from one run to
> the other, as the HTTP request for a given package may sometimes take
> more than 15 seconds, sometimes not.

Do you mean the generated html will be different?
Will the script retry for a package that timeouts?
Well... I can wait you send v3 to know this.

> 
> I guess this is a good enough trade-off, until upstream provides us a
> better way of retrieving the data.

The upstream is working to provide api_v2 that will improve various aspects.
The main changes in the api are a new field ecosystem (it will contain pypi for
python packages, upstream url for custom projects, ...) and the paging system in
the web interface.
The new api will also provide the mapping:
url/api/v2/projects?distribution=Buildroot
I tested this by running a local server with a copy of the production database.
Sample output:
...
{
    "distribution": "Buildroot", 
    "ecosystem": "https://download.samba.org/pub/samba/", 
    "name": "samba4", 
    "project": "samba"
}, 
...

Most changes in the code look (to me) ready and the upstream is currently
setting up a staging server.
Api v2 is *not yet* deployed to release-monitoring. I don't know the upstream
timeline for this.

In the meanwhile, the solution you propose seems to be the best we can do using
api v1.

Right now there is a single direct user of the script (the server that
generates the html), so a time penalty is not critical.
All the other users (we) will access the already generated pkg-stats html.


But I think it is better to upgrade to api v2 before we provide this script as a
build target to generate a report based on the packages selected by .config .
I think you mentioned something similar in an e-mail I unfortunately can't find
right now.

Regards,
Ricardo

[Buildroot] [PATCH next v2 4/5] support/scripts/pkg-stats-new: add latest upstream version information

[Thomas Petazzoni]

Hello,

On Thu, 22 Mar 2018 00:17:26 -0300, Ricardo Martincoski wrote:

> > This will allow to make sure the script terminates properly. However,
> > it means that the result of the script may be different from one run to
> > the other, as the HTTP request for a given package may sometimes take
> > more than 15 seconds, sometimes not.  
> 
> Do you mean the generated html will be different?

Yes, the HTML would be different: if for a first run, the request for
package "foo" timeouts the HTML will say "unknown upstream version".
Then for a second run of the script, the request for package "foo"
doesn't timeout (release-monitoring.org was faster), then the HTML will
have a proper value for the upstream version.

> Will the script retry for a package that timeouts?

So far it doesn't.

> The upstream is working to provide api_v2 that will improve various aspects.
> The main changes in the api are a new field ecosystem (it will contain pypi for
> python packages, upstream url for custom projects, ...) and the paging system in
> the web interface.
> The new api will also provide the mapping:
> url/api/v2/projects?distribution=Buildroot
> I tested this by running a local server with a copy of the production database.
> Sample output:
> ...
> {
>     "distribution": "Buildroot", 
>     "ecosystem": "https://download.samba.org/pub/samba/", 
>     "name": "samba4", 
>     "project": "samba"
> }, 
> ...
> 
> Most changes in the code look (to me) ready and the upstream is currently
> setting up a staging server.
> Api v2 is *not yet* deployed to release-monitoring. I don't know the upstream
> timeline for this.
> 
> In the meanwhile, the solution you propose seems to be the best we can do using
> api v1.
> 
> Right now there is a single direct user of the script (the server that
> generates the html), so a time penalty is not critical.
> All the other users (we) will access the already generated pkg-stats html.

Yes, that was the idea.

> But I think it is better to upgrade to api v2 before we provide this script as a
> build target to generate a report based on the packages selected by .config .
> I think you mentioned something similar in an e-mail I unfortunately can't find
> right now.

Indeed, when the v2 API is available, we will definitely revisit this
and use pre-downloaded data.

Best regards,

Thomas
-- 
Thomas Petazzoni, CTO, Bootlin (formerly Free Electrons)
Embedded Linux and Kernel engineering
https://bootlin.com

[Buildroot] [PATCH next v2 5/5] support/scripts/pkg-stats: replace with new Python version

[Thomas Petazzoni]

Rename pkg-stats-new to pkg-stats.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
---
 support/scripts/pkg-stats     | 1001 +++++++++++++++++++++++++----------------
 support/scripts/pkg-stats-new |  668 ---------------------------
 2 files changed, 607 insertions(+), 1062 deletions(-)
 delete mode 100755 support/scripts/pkg-stats-new

diff --git a/support/scripts/pkg-stats b/support/scripts/pkg-stats
index 48a2cc29a1..b27f4df9a8 100755
--- a/support/scripts/pkg-stats
+++ b/support/scripts/pkg-stats
@@ -1,4 +1,4 @@
-#!/usr/bin/env bash
+#!/usr/bin/env python
 
 # Copyright (C) 2009 by Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
 #
@@ -16,16 +16,371 @@
 # along with this program; if not, write to the Free Software
 # Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
 
-# This script generates an HTML file that contains a report about all
-# Buildroot packages, their usage of the different package
-# infrastructure and possible cleanup actions
-#
-# Run the script from the Buildroot toplevel directory:
-#
-#  ./support/scripts/pkg-stats > /tmp/pkg.html
-#
-
-echo "<head>
+import argparse
+import datetime
+import fnmatch
+import os
+from collections import defaultdict
+import re
+import subprocess
+import sys
+import json
+import urllib2
+from Queue import Queue
+from threading import Thread
+
+
+class Package:
+    def __init__(self, name, path):
+        self.name = name
+        self.path = path
+        self.infras = None
+        self.has_license = False
+        self.has_license_files = False
+        self.has_hash = False
+        self.patch_count = 0
+        self.warnings = 0
+        self.current_version = None
+        self.latest_version = None
+
+    def __eq__(self, other):
+        return self.path == other.path
+
+    def __lt__(self, other):
+        return self.path < other.path
+
+    def __str__(self):
+        return "%s (path='%s', license='%s', license_files='%s', hash='%s', patches=%d)" % \
+            (self.name, self.path, self.has_license, self.has_license_files, self.has_hash, self.patch_count)
+
+
+def get_pkglist(npackages, package_list):
+    """
+    Builds the list of Buildroot packages, returning a list of Package
+    objects. Only the .name and .path fields of the Package object are
+    initialized.
+
+    npackages: limit to N packages
+    package_list: limit to those packages in this list
+    """
+    WALK_USEFUL_SUBDIRS = ["boot", "linux", "package", "toolchain"]
+    WALK_EXCLUDES = ["boot/common.mk",
+                     "linux/linux-ext-.*.mk",
+                     "package/freescale-imx/freescale-imx.mk",
+                     "package/gcc/gcc.mk",
+                     "package/gstreamer/gstreamer.mk",
+                     "package/gstreamer1/gstreamer1.mk",
+                     "package/gtk2-themes/gtk2-themes.mk",
+                     "package/matchbox/matchbox.mk",
+                     "package/opengl/opengl.mk",
+                     "package/qt5/qt5.mk",
+                     "package/x11r7/x11r7.mk",
+                     "package/doc-asciidoc.mk",
+                     "package/pkg-.*.mk",
+                     "package/nvidia-tegra23/nvidia-tegra23.mk",
+                     "toolchain/toolchain-external/pkg-toolchain-external.mk",
+                     "toolchain/toolchain-external/toolchain-external.mk",
+                     "toolchain/toolchain.mk",
+                     "toolchain/helpers.mk",
+                     "toolchain/toolchain-wrapper.mk"]
+    packages = list()
+    count = 0
+    for root, dirs, files in os.walk("."):
+        rootdir = root.split("/")
+        if len(rootdir) < 2:
+            continue
+        if rootdir[1] not in WALK_USEFUL_SUBDIRS:
+            continue
+        for f in files:
+            if not f.endswith(".mk"):
+                continue
+            # Strip ending ".mk"
+            pkgname = f[:-3]
+            if package_list and pkgname not in package_list:
+                continue
+            pkgpath = os.path.join(root, f)
+            skip = False
+            for exclude in WALK_EXCLUDES:
+                # pkgpath[2:] strips the initial './'
+                if re.match(exclude, pkgpath[2:]):
+                    skip = True
+                    continue
+            if skip:
+                continue
+            p = Package(pkgname, pkgpath)
+            packages.append(p)
+            count += 1
+            if npackages and count == npackages:
+                return packages
+    return packages
+
+
+INFRA_RE = re.compile("\$\(eval \$\(([a-z-]*)-package\)\)")
+
+
+def get_pkg_infra_info(pkgpath):
+    infras = list()
+    with open(pkgpath, 'r') as f:
+        lines = f.readlines()
+        for l in lines:
+            match = INFRA_RE.match(l)
+            if not match:
+                continue
+            infra = match.group(1)
+            if infra.startswith("host-"):
+                infras.append(("host", infra[5:]))
+            else:
+                infras.append(("target", infra))
+    return infras
+
+
+def add_infra_info(packages):
+
+    """
+    Fills in the .infras field of all Package objects
+    """
+    for pkg in packages:
+        pkg.infras = get_pkg_infra_info(pkg.path)
+
+
+def pkgname_to_pkgvar(pkgname):
+    return pkgname.upper().replace("-", "_")
+
+
+def add_pkg_make_info(packages):
+    """
+    Fills in the .has_license and .has_license_files fields of all
+    Package objects
+    """
+    licenses = list()
+    license_files = list()
+    versions = dict()
+
+    # Licenses
+    o = subprocess.check_output(["make", "BR2_HAVE_DOT_CONFIG=y",
+                                 "-s", "printvars", "VARS=%_LICENSE"])
+    for l in o.splitlines():
+        # Get variable name and value
+        pkgvar, value = l.split("=")
+
+        # If present, strip HOST_ from variable name
+        if pkgvar.startswith("HOST_"):
+            pkgvar = pkgvar[5:]
+
+        # Strip _LICENSE
+        pkgvar = pkgvar[:-8]
+
+        # If value is "unknown", no license details available
+        if value == "unknown":
+            continue
+        licenses.append(pkgvar)
+
+    # License files
+    o = subprocess.check_output(["make", "BR2_HAVE_DOT_CONFIG=y",
+                                 "-s", "printvars", "VARS=%_LICENSE_FILES"])
+    for l in o.splitlines():
+        # Get variable name and value
+        pkgvar, value = l.split("=")
+
+        # If present, strip HOST_ from variable name
+        if pkgvar.startswith("HOST_"):
+            pkgvar = pkgvar[5:]
+
+        if pkgvar.endswith("_MANIFEST_LICENSE_FILES"):
+            continue
+
+        # Strip _LICENSE_FILES
+        pkgvar = pkgvar[:-14]
+
+        license_files.append(pkgvar)
+
+    # Version
+    o = subprocess.check_output(["make", "BR2_HAVE_DOT_CONFIG=y",
+                                 "-s", "printvars", "VARS=%_VERSION"])
+    for l in o.splitlines():
+        # Get variable name and value
+        pkgvar, value = l.split("=")
+
+        # If present, strip HOST_ from variable name
+        if pkgvar.startswith("HOST_"):
+            pkgvar = pkgvar[5:]
+
+        if pkgvar.endswith("_DL_VERSION"):
+            continue
+
+        # Strip _VERSION
+        pkgvar = pkgvar[:-8]
+
+        versions[pkgvar] = value
+
+    for pkg in packages:
+        var = pkgname_to_pkgvar(pkg.name)
+        if var in licenses:
+            pkg.has_license = True
+        if var in license_files:
+            pkg.has_license_files = True
+        if var in versions:
+            pkg.current_version = versions[var]
+
+
+def add_hash_info(packages):
+    """
+    Fills in the .has_hash field of all Package objects
+    """
+    for pkg in packages:
+        hashpath = pkg.path.replace(".mk", ".hash")
+        pkg.has_hash = os.path.exists(hashpath)
+
+
+def add_patch_count(packages):
+    """
+    Fills in the .patch_count field of all Package objects
+    """
+    for pkg in packages:
+        pkgdir = os.path.dirname(pkg.path)
+        pkg.patch_count = len(fnmatch.filter(os.listdir(pkgdir), '*.patch'))
+
+
+def get_check_package_warnings(pkgdir):
+    cmd = ["./utils/check-package"]
+    for root, dirs, files in os.walk(pkgdir):
+        for f in files:
+            if f.endswith(".mk") or f.endswith(".hash") or f == "Config.in" or f == "Config.in.host":
+                cmd.append(f)
+    o = subprocess.check_output(cmd, stderr=subprocess.STDOUT)
+    lines = o.splitlines()
+    for line in lines:
+        m = re.match("^([0-9]*) warnings generated", line)
+        if m:
+            return int(m.group(1))
+    return None
+
+
+def add_check_package_warnings(packages):
+    """
+    Fills in the .warnings field of all Package objects
+    """
+    for pkg in packages:
+        pkg.warnings = get_check_package_warnings(os.path.dirname(pkg.path))
+
+
+RELEASE_MONITORING_API = "http://release-monitoring.org/api"
+
+
+def get_latest_version_by_distro(package):
+    req = urllib2.Request(os.path.join(RELEASE_MONITORING_API, "project", "Buildroot", package))
+    f = urllib2.urlopen(req)
+    data = json.loads(f.read())
+    if len(data['versions']) > 0:
+        return (True, data['versions'][0], data['id'])
+    else:
+        return (True, None, data['id'])
+
+
+def get_latest_version_by_guess(package):
+    req = urllib2.Request(os.path.join(RELEASE_MONITORING_API, "projects", "?pattern=%s" % package))
+    f = urllib2.urlopen(req)
+    data = json.loads(f.read())
+    for p in data['projects']:
+        if p['name'] == package and len(p['versions']) > 0:
+            return (False, p['versions'][0], p['id'])
+    return (False, None, None)
+
+
+def get_latest_version(package):
+    try:
+        # We first try by using the "Buildroot" distribution on
+        # release-monitoring.org, if it has a mapping for the current
+        # package name.
+        return get_latest_version_by_distro(package)
+    except urllib2.HTTPError, e:
+        # If that fails because there is no mapping, we try to search
+        # in all packages for a package of this name.
+        if e.code == 404:
+            return get_latest_version_by_guess(package)
+        else:
+            return (False, None, None)
+
+
+def get_version_worker(q):
+    while True:
+        pkg = q.get()
+        try:
+            pkg.latest_version = get_latest_version(pkg.name)
+            print " [%04d] %s => %s" % (q.qsize(), pkg.name, str(pkg.latest_version))
+        except ValueError:
+            pkg.latest_version = (False, None, None)
+            print " [%04d] %s => Value Error" % (q.qsize(), pkg.name)
+        q.task_done()
+
+
+def add_latest_version_info(packages):
+    """
+    Fills in the .latest_version field of all Package objects
+
+    This field has a special format:
+      (mapping, version, id)
+    with:
+    - mapping: boolean that indicates whether release-monitoring.org
+      has a mapping for this package name in the Buildroot distribution
+      or not
+    - version: string containing the latest version known by
+      release-monitoring.org for this package
+    - id: string containing the id of the project corresponding to this
+      package, as known by release-monitoring.org
+    """
+    q = Queue()
+    for pkg in packages:
+        q.put(pkg)
+    # Since release-monitoring.org is rather slow, we create 8 threads
+    # that do HTTP requests to the site.
+    for i in range(8):
+        t = Thread(target=get_version_worker, args=[q])
+        t.daemon = True
+        t.start()
+    q.join()
+
+
+def calculate_stats(packages):
+    stats = defaultdict(int)
+    for pkg in packages:
+        # If packages have multiple infra, take the first one. For the
+        # vast majority of packages, the target and host infra are the
+        # same. There are very few packages that use a different infra
+        # for the host and target variants.
+        if len(pkg.infras) > 0:
+            infra = pkg.infras[0][1]
+            stats["infra-%s" % infra] += 1
+        else:
+            stats["infra-unknown"] += 1
+        if pkg.has_license:
+            stats["license"] += 1
+        else:
+            stats["no-license"] += 1
+        if pkg.has_license_files:
+            stats["license-files"] += 1
+        else:
+            stats["no-license-files"] += 1
+        if pkg.has_hash:
+            stats["hash"] += 1
+        else:
+            stats["no-hash"] += 1
+        if pkg.latest_version[0]:
+            stats["rmo-mapping"] += 1
+        else:
+            stats["rmo-no-mapping"] += 1
+        if not pkg.latest_version[1]:
+            stats["version-unknown"] += 1
+        elif pkg.latest_version[1] == pkg.current_version:
+            stats["version-uptodate"] += 1
+        else:
+            stats["version-not-uptodate"] += 1
+        stats["patches"] += pkg.patch_count
+    return stats
+
+
+html_header = """
+<head>
 <script src=\"https://www.kryogenix.org/code/browser/sorttable/sorttable.js\"></script>
 <style type=\"text/css\">
 table {
@@ -46,14 +401,21 @@ td.correct {
 td.nopatches {
   background: #d2ffc4;
 }
-
 td.somepatches {
   background: #ffd870;
 }
-
 td.lotsofpatches {
   background: #ff9a69;
 }
+td.version-good {
+  background: #d2ffc4;
+}
+td.version-needs-update {
+  background: #ff9a69;
+}
+td.version-unknown {
+ background: #ffd870;
+}
 </style>
 <title>Statistics of Buildroot packages</title>
 </head>
@@ -61,395 +423,246 @@ td.lotsofpatches {
 <a href=\"#results\">Results</a><br/>
 
 <p id=\"sortable_hint\"></p>
+"""
 
+
+html_footer = """
+</body>
+<script>
+if (typeof sorttable === \"object\") {
+  document.getElementById(\"sortable_hint\").innerHTML =
+  \"hint: the table can be sorted by clicking the column headers\"
+}
+</script>
+</html>
+"""
+
+
+def infra_str(infra_list):
+    if not infra_list:
+        return "Unknown"
+    elif len(infra_list) == 1:
+        return "<b>%s</b><br/>%s" % (infra_list[0][1], infra_list[0][0])
+    elif infra_list[0][1] == infra_list[1][1]:
+        return "<b>%s</b><br/>%s + %s" % \
+            (infra_list[0][1], infra_list[0][0], infra_list[1][0])
+    else:
+        return "<b>%s</b> (%s)<br/><b>%s</b> (%s)" % \
+            (infra_list[0][1], infra_list[0][0],
+             infra_list[1][1], infra_list[1][0])
+
+
+def boolean_str(b):
+    if b:
+        return "Yes"
+    else:
+        return "No"
+
+
+def dump_html_pkg(f, pkg):
+    f.write(" <tr>\n")
+    f.write("  <td>%s</td>\n" % pkg.path[2:])
+
+    # Patch count
+    td_class = ["centered"]
+    if pkg.patch_count == 0:
+        td_class.append("nopatches")
+    elif pkg.patch_count < 5:
+        td_class.append("somepatches")
+    else:
+        td_class.append("lotsofpatches")
+    f.write("  <td class=\"%s\">%s</td>\n" %
+            (" ".join(td_class), str(pkg.patch_count)))
+
+    # Infrastructure
+    infra = infra_str(pkg.infras)
+    td_class = ["centered"]
+    if infra == "Unknown":
+        td_class.append("wrong")
+    else:
+        td_class.append("correct")
+    f.write("  <td class=\"%s\">%s</td>\n" %
+            (" ".join(td_class), infra_str(pkg.infras)))
+
+    # License
+    td_class = ["centered"]
+    if pkg.has_license:
+        td_class.append("correct")
+    else:
+        td_class.append("wrong")
+    f.write("  <td class=\"%s\">%s</td>\n" %
+            (" ".join(td_class), boolean_str(pkg.has_license)))
+
+    # License files
+    td_class = ["centered"]
+    if pkg.has_license_files:
+        td_class.append("correct")
+    else:
+        td_class.append("wrong")
+    f.write("  <td class=\"%s\">%s</td>\n" %
+            (" ".join(td_class), boolean_str(pkg.has_license_files)))
+
+    # Hash
+    td_class = ["centered"]
+    if pkg.has_hash:
+        td_class.append("correct")
+    else:
+        td_class.append("wrong")
+    f.write("  <td class=\"%s\">%s</td>\n" %
+            (" ".join(td_class), boolean_str(pkg.has_hash)))
+
+    # Current version
+    f.write("  <td class=\"centered\">%s</td>\n" % pkg.current_version)
+
+    # Latest version
+    if pkg.latest_version[1] is None:
+        td_class.append("version-unknown")
+    elif pkg.latest_version[1] != pkg.current_version:
+        td_class.append("version-needs-update")
+    else:
+        td_class.append("version-good")
+
+    if pkg.latest_version[1] is None:
+        latest_version_text = "<b>Unknown</b>"
+    else:
+        latest_version_text = "<b>%s</b>" % str(pkg.latest_version[1])
+
+    latest_version_text += "<br/>"
+
+    if pkg.latest_version[2]:
+        latest_version_text += "<a href=\"https://release-monitoring.org/project/%s\">link</a>, " % pkg.latest_version[2]
+    else:
+        latest_version_text += "no link, "
+
+    if pkg.latest_version[0]:
+        latest_version_text += "has <a href=\"https://release-monitoring.org/distro/Buildroot/\">mapping</a>"
+    else:
+        latest_version_text += "has <a href=\"https://release-monitoring.org/distro/Buildroot/\">no mapping</a>"
+
+    f.write("  <td class=\"%s\">%s</td>\n" %
+            (" ".join(td_class), latest_version_text))
+
+    # Warnings
+    td_class = ["centered"]
+    if pkg.warnings == 0:
+        td_class.append("correct")
+    else:
+        td_class.append("wrong")
+    f.write("  <td class=\"%s\">%d</td>\n" %
+            (" ".join(td_class), pkg.warnings))
+
+    f.write(" </tr>\n")
+
+
+def dump_html_all_pkgs(f, packages):
+    f.write("""
 <table class=\"sortable\">
 <tr>
-<td>Id</td>
 <td>Package</td>
 <td class=\"centered\">Patch count</td>
 <td class=\"centered\">Infrastructure</td>
 <td class=\"centered\">License</td>
 <td class=\"centered\">License files</td>
 <td class=\"centered\">Hash file</td>
+<td class=\"centered\">Current version</td>
+<td class=\"centered\">Latest version</td>
 <td class=\"centered\">Warnings</td>
 </tr>
-"
-
-autotools_packages=0
-cmake_packages=0
-kconfig_packages=0
-luarocks_package=0
-perl_packages=0
-python_packages=0
-rebar_packages=0
-virtual_packages=0
-generic_packages=0
-waf_packages=0
-manual_packages=0
-packages_with_licence=0
-packages_without_licence=0
-packages_with_license_files=0
-packages_without_license_files=0
-packages_with_hash_file=0
-packages_without_hash_file=0
-total_patch_count=0
-cnt=0
-
-for i in $(find boot/ linux/ package/ toolchain/ -name '*.mk' | sort) ; do
-
-    if test \
-	$i = "boot/common.mk" -o \
-	$i = "linux/linux-ext-ev3dev-linux-drivers.mk" -o \
-	$i = "linux/linux-ext-fbtft.mk" -o \
-	$i = "linux/linux-ext-xenomai.mk" -o \
-	$i = "linux/linux-ext-rtai.mk" -o \
-	$i = "package/freescale-imx/freescale-imx.mk" -o \
-	$i = "package/gcc/gcc.mk" -o \
-	$i = "package/gstreamer/gstreamer.mk" -o \
-	$i = "package/gstreamer1/gstreamer1.mk" -o \
-	$i = "package/gtk2-themes/gtk2-themes.mk" -o \
-	$i = "package/matchbox/matchbox.mk" -o \
-	$i = "package/opengl/opengl.mk" -o \
-	$i = "package/qt5/qt5.mk" -o \
-	$i = "package/x11r7/x11r7.mk" -o \
-	$i = "package/doc-asciidoc.mk" -o \
-	$i = "package/pkg-autotools.mk" -o \
-	$i = "package/pkg-cmake.mk" -o \
-	$i = "package/pkg-kconfig.mk" -o \
-	$i = "package/pkg-luarocks.mk" -o \
-	$i = "package/pkg-perl.mk" -o \
-	$i = "package/pkg-python.mk" -o \
-	$i = "package/pkg-rebar.mk" -o \
-	$i = "package/pkg-virtual.mk" -o \
-	$i = "package/pkg-download.mk" -o \
-	$i = "package/pkg-generic.mk" -o \
-	$i = "package/pkg-waf.mk" -o \
-	$i = "package/pkg-kernel-module.mk" -o \
-	$i = "package/pkg-utils.mk" -o \
-	$i = "package/nvidia-tegra23/nvidia-tegra23.mk" -o \
-	$i = "toolchain/toolchain-external/pkg-toolchain-external.mk" -o \
-	$i = "toolchain/toolchain-external/toolchain-external.mk" -o \
-	$i = "toolchain/toolchain.mk" -o \
-	$i = "toolchain/helpers.mk" -o \
-	$i = "toolchain/toolchain-wrapper.mk" ; then
-	echo "skipping $i" 1>&2
-	continue
-    fi
-
-    cnt=$((cnt+1))
-
-    hashost=0
-    hastarget=0
-    infratype=""
-
-    # Determine package infrastructure
-    if grep -E "\(host-autotools-package\)" $i > /dev/null ; then
-	infratype="autotools"
-	hashost=1
-    fi
-
-    if grep -E "\(autotools-package\)" $i > /dev/null ; then
-	infratype="autotools"
-	hastarget=1
-    fi
-
-    if grep -E "\(kconfig-package\)" $i > /dev/null ; then
-	infratype="kconfig"
-	hastarget=1
-    fi
-
-    if grep -E "\(host-luarocks-package\)" $i > /dev/null ; then
-	infratype="luarocks"
-	hashost=1
-    fi
-
-    if grep -E "\(luarocks-package\)" $i > /dev/null ; then
-	infratype="luarocks"
-	hastarget=1
-    fi
-
-    if grep -E "\(host-perl-package\)" $i > /dev/null ; then
-	infratype="perl"
-	hashost=1
-    fi
-
-    if grep -E "\(perl-package\)" $i > /dev/null ; then
-	infratype="perl"
-	hastarget=1
-    fi
-
-    if grep -E "\(host-python-package\)" $i > /dev/null ; then
-	infratype="python"
-	hashost=1
-    fi
-
-    if grep -E "\(python-package\)" $i > /dev/null ; then
-	infratype="python"
-	hastarget=1
-    fi
-
-    if grep -E "\(host-rebar-package\)" $i > /dev/null ; then
-	infratype="rebar"
-	hashost=1
-    fi
-
-    if grep -E "\(rebar-package\)" $i > /dev/null ; then
-	infratype="rebar"
-	hastarget=1
-    fi
-
-    if grep -E "\(host-virtual-package\)" $i > /dev/null ; then
-	infratype="virtual"
-	hashost=1
-    fi
-
-    if grep -E "\(virtual-package\)" $i > /dev/null ; then
-	infratype="virtual"
-	hastarget=1
-    fi
-
-    if grep -E "\(host-generic-package\)" $i > /dev/null ; then
-	infratype="generic"
-	hashost=1
-    fi
-
-    if grep -E "\(generic-package\)" $i > /dev/null ; then
-	infratype="generic"
-	hastarget=1
-    fi
-
-    if grep -E "\(host-cmake-package\)" $i > /dev/null ; then
-	infratype="cmake"
-	hashost=1
-    fi
-
-    if grep -E "\(cmake-package\)" $i > /dev/null ; then
-	infratype="cmake"
-	hastarget=1
-    fi
-
-    if grep -E "\(toolchain-external-package\)" $i > /dev/null ; then
-	infratype="toolchain-external"
-	hastarget=1
-    fi
-
-    if grep -E "\(waf-package\)" $i > /dev/null ; then
-	infratype="waf"
-	hastarget=1
-    fi
-
-    pkg=$(basename $i)
-    dir=$(dirname $i)
-    pkg=${pkg%.mk}
-    pkgvariable=$(echo ${pkg} | tr "a-z-" "A-Z_")
-
-
-    # Count packages per infrastructure
-    if [ -z ${infratype} ] ; then
-	infratype="manual"
-	manual_packages=$(($manual_packages+1))
-    elif [ ${infratype} = "autotools" ]; then
-	autotools_packages=$(($autotools_packages+1))
-    elif [ ${infratype} = "cmake" ]; then
-	cmake_packages=$(($cmake_packages+1))
-    elif [ ${infratype} = "kconfig" ]; then
-	kconfig_packages=$(($kconfig_packages+1))
-    elif [ ${infratype} = "luarocks" ]; then
-	luarocks_packages=$(($luarocks_packages+1))
-    elif [ ${infratype} = "perl" ]; then
-	perl_packages=$(($perl_packages+1))
-    elif [ ${infratype} = "python" ]; then
-	python_packages=$(($python_packages+1))
-    elif [ ${infratype} = "rebar" ]; then
-	rebar_packages=$(($rebar_packages+1))
-    elif [ ${infratype} = "virtual" ]; then
-	virtual_packages=$(($virtual_packages+1))
-    elif [ ${infratype} = "generic" ]; then
-	generic_packages=$(($generic_packages+1))
-    elif [ ${infratype} = "waf" ]; then
-	waf_packages=$(($waf_packages+1))
-    fi
-
-    if grep -qE "^${pkgvariable}_LICENSE[ ]*=" $i ; then
-	packages_with_license=$(($packages_with_license+1))
-	license=1
-    else
-	packages_without_license=$(($packages_without_license+1))
-	license=0
-    fi
-
-    if grep -qE "^${pkgvariable}_LICENSE_FILES[ ]*=" $i ; then
-	packages_with_license_files=$(($packages_with_license_files+1))
-	license_files=1
-    else
-	packages_without_license_files=$(($packages_without_license_files+1))
-	license_files=0
-    fi
-
-    if test -f ${dir}/${pkg}.hash; then
-	packages_with_hash_file=$(($packages_with_hash_file+1))
-	hash_file=1
-    else
-	packages_without_hash_file=$(($packages_without_hash_file+1))
-	hash_file=0
-    fi
-
-    echo "<tr>"
-
-    echo "<td>$cnt</td>"
-    echo "<td>$i</td>"
-
-    package_dir=$(dirname $i)
-    patch_count=$(find ${package_dir} -name '*.patch' | wc -l)
-    total_patch_count=$(($total_patch_count+$patch_count))
-
-    if test $patch_count -lt 1 ; then
-	patch_count_class="nopatches"
-    elif test $patch_count -lt 5 ; then
-	patch_count_class="somepatches"
-    else
-	patch_count_class="lotsofpatches"
-    fi
-
-    echo "<td class=\"centered ${patch_count_class}\">"
-    echo "<b>$patch_count</b>"
-    echo "</td>"
-
-    if [ ${infratype} = "manual" ] ; then
-	echo "<td class=\"centered wrong\"><b>manual</b></td>"
-    else
-	echo "<td class=\"centered correct\">"
-	echo "<b>${infratype}</b><br/>"
-	if [ ${hashost} -eq 1 -a ${hastarget} -eq 1 ]; then
-	    echo "target + host"
-	elif [ ${hashost} -eq 1 ]; then
-	    echo "host"
-	else
-	    echo "target"
-	fi
-	echo "</td>"
-    fi
-
-    if [ ${license} -eq 0 ] ; then
-	echo "<td class=\"centered wrong\">No</td>"
-    else
-	echo "<td class=\"centered correct\">Yes</td>"
-    fi
-
-    if [ ${license_files} -eq 0 ] ; then
-	echo "<td class=\"centered wrong\">No</td>"
-    else
-	echo "<td class=\"centered correct\">Yes</td>"
-    fi
-
-    if [ ${hash_file} -eq 0 ] ; then
-	echo "<td class=\"centered wrong\">No</td>"
-    else
-	echo "<td class=\"centered correct\">Yes</td>"
-    fi
-
-    file_list=$(find ${package_dir} -name '*.mk' -o -name '*.in*' -o -name '*.hash')
-    nwarnings=$(./utils/check-package ${file_list} 2>&1 | sed '/\([0-9]*\) warnings generated/!d; s//\1/')
-    if [ ${nwarnings} -eq 0 ] ; then
-	echo "<td class=\"centered correct\">${nwarnings}</td>"
-    else
-	echo "<td class=\"centered wrong\">${nwarnings}</td>"
-    fi
-
-    echo "</tr>"
-
-done
-echo "</table>"
-
-echo "<a id="results"></a>"
-echo "<table>"
-echo "<tr>"
-echo "<td>Packages using the <i>generic</i> infrastructure</td>"
-echo "<td>$generic_packages</td>"
-echo "</tr>"
-echo "<tr>"
-echo "<td>Packages using the <i>cmake</i> infrastructure</td>"
-echo "<td>$cmake_packages</td>"
-echo "</tr>"
-echo "<tr>"
-echo "<td>Packages using the <i>autotools</i> infrastructure</td>"
-echo "<td>$autotools_packages</td>"
-echo "</tr>"
-echo "<tr>"
-echo "<td>Packages using the <i>luarocks</i> infrastructure</td>"
-echo "<td>$luarocks_packages</td>"
-echo "</tr>"
-echo "<tr>"
-echo "<td>Packages using the <i>kconfig</i> infrastructure</td>"
-echo "<td>$kconfig_packages</td>"
-echo "</tr>"
-echo "<tr>"
-echo "<td>Packages using the <i>perl</i> infrastructure</td>"
-echo "<td>$perl_packages</td>"
-echo "</tr>"
-echo "<tr>"
-echo "<td>Packages using the <i>python</i> infrastructure</td>"
-echo "<td>$python_packages</td>"
-echo "</tr>"
-echo "<tr>"
-echo "<td>Packages using the <i>rebar</i> infrastructure</td>"
-echo "<td>$rebar_packages</td>"
-echo "</tr>"
-echo "<tr>"
-echo "<td>Packages using the <i>virtual</i> infrastructure</td>"
-echo "<td>$virtual_packages</td>"
-echo "</tr>"
-echo "<tr>"
-echo "<td>Packages using the <i>waf</i> infrastructure</td>"
-echo "<td>$waf_packages</td>"
-echo "</tr>"
-echo "<tr>"
-echo "<td>Packages not using any infrastructure</td>"
-echo "<td>$manual_packages</td>"
-echo "</tr>"
-echo "<tr>"
-echo "<td>Packages having license information</td>"
-echo "<td>$packages_with_license</td>"
-echo "</tr>"
-echo "<tr>"
-echo "<td>Packages not having licence information</td>"
-echo "<td>$packages_without_license</td>"
-echo "</tr>"
-echo "<tr>"
-echo "<td>Packages having license files information</td>"
-echo "<td>$packages_with_license_files</td>"
-echo "</tr>"
-echo "<tr>"
-echo "<td>Packages not having licence files information</td>"
-echo "<td>$packages_without_license_files</td>"
-echo "</tr>"
-echo "<tr>"
-echo "<td>Packages having hash file</td>"
-echo "<td>$packages_with_hash_file</td>"
-echo "</tr>"
-echo "<tr>"
-echo "<td>Packages not having hash file</td>"
-echo "<td>$packages_without_hash_file</td>"
-echo "</tr>"
-echo "<tr>"
-echo "<td>Number of patches in all packages</td>"
-echo "<td>$total_patch_count</td>"
-echo "</tr>"
-echo "<tr>"
-echo "<td>TOTAL</td>"
-echo "<td>$cnt</td>"
-echo "</tr>"
-echo "</table>"
-
-echo "<hr/>"
-echo "<i>Updated on $(LANG=C date), Git commit $(git log master -n 1 --pretty=format:%H)</i>"
-echo "</body>"
-
-echo "<script>
-if (typeof sorttable === \"object\") {
-  document.getElementById(\"sortable_hint\").innerHTML =
-  \"hint: the table can be sorted by clicking the column headers\"
-}
-</script>
-"
-echo "</html>"
+""")
+    for pkg in sorted(packages):
+        dump_html_pkg(f, pkg)
+    f.write("</table>")
+
+
+def dump_html_stats(f, stats):
+    f.write("<a id=\"results\"></a>\n")
+    f.write("<table>\n")
+    infras = [infra[6:] for infra in stats.keys() if infra.startswith("infra-")]
+    for infra in infras:
+        f.write(" <tr><td>Packages using the <i>%s</i> infrastructure</td><td>%s</td></tr>\n" %
+                (infra, stats["infra-%s" % infra]))
+    f.write(" <tr><td>Packages having license information</td><td>%s</td></tr>\n" %
+            stats["license"])
+    f.write(" <tr><td>Packages not having license information</td><td>%s</td></tr>\n" %
+            stats["no-license"])
+    f.write(" <tr><td>Packages having license files information</td><td>%s</td></tr>\n" %
+            stats["license-files"])
+    f.write(" <tr><td>Packages not having license files information</td><td>%s</td></tr>\n" %
+            stats["no-license-files"])
+    f.write(" <tr><td>Packages having a hash file</td><td>%s</td></tr>\n" %
+            stats["hash"])
+    f.write(" <tr><td>Packages not having a hash file</td><td>%s</td></tr>\n" %
+            stats["no-hash"])
+    f.write(" <tr><td>Total number of patches</td><td>%s</td></tr>\n" %
+            stats["patches"])
+    f.write("<tr><td>Packages having a mapping on <i>release-monitoring.org</i></td><td>%s</td></tr>\n" %
+            stats["rmo-mapping"])
+    f.write("<tr><td>Packages lacking a mapping on <i>release-monitoring.org</i></td><td>%s</td></tr>\n" %
+            stats["rmo-no-mapping"])
+    f.write("<tr><td>Packages that are up-to-date</td><td>%s</td></tr>\n" %
+            stats["version-uptodate"])
+    f.write("<tr><td>Packages that are not up-to-date</td><td>%s</td></tr>\n" %
+            stats["version-not-uptodate"])
+    f.write("<tr><td>Packages with no known upstream version</td><td>%s</td></tr>\n" %
+            stats["version-unknown"])
+    f.write("</table>\n")
+
+
+def dump_gen_info(f):
+    # Updated on Mon Feb 19 08:12:08 CET 2018, Git commit aa77030b8f5e41f1c53eb1c1ad664b8c814ba032
+    o = subprocess.check_output(["git", "log", "master", "-n", "1", "--pretty=format:%H"])
+    git_commit = o.splitlines()[0]
+    f.write("<p><i>Updated on %s, git commit %s</i></p>\n" %
+            (str(datetime.datetime.utcnow()), git_commit))
+
+
+def dump_html(packages, stats, output):
+    with open(output, 'w') as f:
+        f.write(html_header)
+        dump_html_all_pkgs(f, packages)
+        dump_html_stats(f, stats)
+        dump_gen_info(f)
+        f.write(html_footer)
+
+
+def parse_args():
+    parser = argparse.ArgumentParser()
+    parser.add_argument('-o', dest='output', action='store', required=True,
+                        help='HTML output file')
+    parser.add_argument('-n', dest='npackages', type=int, action='store',
+                        help='Number of packages')
+    parser.add_argument('-p', dest='packages', action='store',
+                        help='List of packages')
+    return parser.parse_args()
+
+
+def __main__():
+    args = parse_args()
+    if args.npackages and args.packages:
+        print "ERROR: -n and -p are mutually exclusive"
+        sys.exit(1)
+    if args.packages:
+        package_list = args.packages.split(",")
+    else:
+        package_list = None
+    print "Build package list ..."
+    packages = get_pkglist(args.npackages, package_list)
+    print "Get package infra ..."
+    add_infra_info(packages)
+    print "Get make info ..."
+    add_pkg_make_info(packages)
+    print "Get hash info ..."
+    add_hash_info(packages)
+    print "Get patch count ..."
+    add_patch_count(packages)
+    print "Get package warnings ..."
+    add_check_package_warnings(packages)
+    print "Get latest version ..."
+    add_latest_version_info(packages)
+    print "Calculate stats"
+    stats = calculate_stats(packages)
+    print "Write HTML"
+    dump_html(packages, stats, args.output)
+
+
+__main__()
diff --git a/support/scripts/pkg-stats-new b/support/scripts/pkg-stats-new
deleted file mode 100755
index b27f4df9a8..0000000000
--- a/support/scripts/pkg-stats-new
+++ /dev/null
@@ -1,668 +0,0 @@
-#!/usr/bin/env python
-
-# Copyright (C) 2009 by Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
-#
-# This program is free software; you can redistribute it and/or modify
-# it under the terms of the GNU General Public License as published by
-# the Free Software Foundation; either version 2 of the License, or
-# (at your option) any later version.
-#
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
-# General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License
-# along with this program; if not, write to the Free Software
-# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
-
-import argparse
-import datetime
-import fnmatch
-import os
-from collections import defaultdict
-import re
-import subprocess
-import sys
-import json
-import urllib2
-from Queue import Queue
-from threading import Thread
-
-
-class Package:
-    def __init__(self, name, path):
-        self.name = name
-        self.path = path
-        self.infras = None
-        self.has_license = False
-        self.has_license_files = False
-        self.has_hash = False
-        self.patch_count = 0
-        self.warnings = 0
-        self.current_version = None
-        self.latest_version = None
-
-    def __eq__(self, other):
-        return self.path == other.path
-
-    def __lt__(self, other):
-        return self.path < other.path
-
-    def __str__(self):
-        return "%s (path='%s', license='%s', license_files='%s', hash='%s', patches=%d)" % \
-            (self.name, self.path, self.has_license, self.has_license_files, self.has_hash, self.patch_count)
-
-
-def get_pkglist(npackages, package_list):
-    """
-    Builds the list of Buildroot packages, returning a list of Package
-    objects. Only the .name and .path fields of the Package object are
-    initialized.
-
-    npackages: limit to N packages
-    package_list: limit to those packages in this list
-    """
-    WALK_USEFUL_SUBDIRS = ["boot", "linux", "package", "toolchain"]
-    WALK_EXCLUDES = ["boot/common.mk",
-                     "linux/linux-ext-.*.mk",
-                     "package/freescale-imx/freescale-imx.mk",
-                     "package/gcc/gcc.mk",
-                     "package/gstreamer/gstreamer.mk",
-                     "package/gstreamer1/gstreamer1.mk",
-                     "package/gtk2-themes/gtk2-themes.mk",
-                     "package/matchbox/matchbox.mk",
-                     "package/opengl/opengl.mk",
-                     "package/qt5/qt5.mk",
-                     "package/x11r7/x11r7.mk",
-                     "package/doc-asciidoc.mk",
-                     "package/pkg-.*.mk",
-                     "package/nvidia-tegra23/nvidia-tegra23.mk",
-                     "toolchain/toolchain-external/pkg-toolchain-external.mk",
-                     "toolchain/toolchain-external/toolchain-external.mk",
-                     "toolchain/toolchain.mk",
-                     "toolchain/helpers.mk",
-                     "toolchain/toolchain-wrapper.mk"]
-    packages = list()
-    count = 0
-    for root, dirs, files in os.walk("."):
-        rootdir = root.split("/")
-        if len(rootdir) < 2:
-            continue
-        if rootdir[1] not in WALK_USEFUL_SUBDIRS:
-            continue
-        for f in files:
-            if not f.endswith(".mk"):
-                continue
-            # Strip ending ".mk"
-            pkgname = f[:-3]
-            if package_list and pkgname not in package_list:
-                continue
-            pkgpath = os.path.join(root, f)
-            skip = False
-            for exclude in WALK_EXCLUDES:
-                # pkgpath[2:] strips the initial './'
-                if re.match(exclude, pkgpath[2:]):
-                    skip = True
-                    continue
-            if skip:
-                continue
-            p = Package(pkgname, pkgpath)
-            packages.append(p)
-            count += 1
-            if npackages and count == npackages:
-                return packages
-    return packages
-
-
-INFRA_RE = re.compile("\$\(eval \$\(([a-z-]*)-package\)\)")
-
-
-def get_pkg_infra_info(pkgpath):
-    infras = list()
-    with open(pkgpath, 'r') as f:
-        lines = f.readlines()
-        for l in lines:
-            match = INFRA_RE.match(l)
-            if not match:
-                continue
-            infra = match.group(1)
-            if infra.startswith("host-"):
-                infras.append(("host", infra[5:]))
-            else:
-                infras.append(("target", infra))
-    return infras
-
-
-def add_infra_info(packages):
-
-    """
-    Fills in the .infras field of all Package objects
-    """
-    for pkg in packages:
-        pkg.infras = get_pkg_infra_info(pkg.path)
-
-
-def pkgname_to_pkgvar(pkgname):
-    return pkgname.upper().replace("-", "_")
-
-
-def add_pkg_make_info(packages):
-    """
-    Fills in the .has_license and .has_license_files fields of all
-    Package objects
-    """
-    licenses = list()
-    license_files = list()
-    versions = dict()
-
-    # Licenses
-    o = subprocess.check_output(["make", "BR2_HAVE_DOT_CONFIG=y",
-                                 "-s", "printvars", "VARS=%_LICENSE"])
-    for l in o.splitlines():
-        # Get variable name and value
-        pkgvar, value = l.split("=")
-
-        # If present, strip HOST_ from variable name
-        if pkgvar.startswith("HOST_"):
-            pkgvar = pkgvar[5:]
-
-        # Strip _LICENSE
-        pkgvar = pkgvar[:-8]
-
-        # If value is "unknown", no license details available
-        if value == "unknown":
-            continue
-        licenses.append(pkgvar)
-
-    # License files
-    o = subprocess.check_output(["make", "BR2_HAVE_DOT_CONFIG=y",
-                                 "-s", "printvars", "VARS=%_LICENSE_FILES"])
-    for l in o.splitlines():
-        # Get variable name and value
-        pkgvar, value = l.split("=")
-
-        # If present, strip HOST_ from variable name
-        if pkgvar.startswith("HOST_"):
-            pkgvar = pkgvar[5:]
-
-        if pkgvar.endswith("_MANIFEST_LICENSE_FILES"):
-            continue
-
-        # Strip _LICENSE_FILES
-        pkgvar = pkgvar[:-14]
-
-        license_files.append(pkgvar)
-
-    # Version
-    o = subprocess.check_output(["make", "BR2_HAVE_DOT_CONFIG=y",
-                                 "-s", "printvars", "VARS=%_VERSION"])
-    for l in o.splitlines():
-        # Get variable name and value
-        pkgvar, value = l.split("=")
-
-        # If present, strip HOST_ from variable name
-        if pkgvar.startswith("HOST_"):
-            pkgvar = pkgvar[5:]
-
-        if pkgvar.endswith("_DL_VERSION"):
-            continue
-
-        # Strip _VERSION
-        pkgvar = pkgvar[:-8]
-
-        versions[pkgvar] = value
-
-    for pkg in packages:
-        var = pkgname_to_pkgvar(pkg.name)
-        if var in licenses:
-            pkg.has_license = True
-        if var in license_files:
-            pkg.has_license_files = True
-        if var in versions:
-            pkg.current_version = versions[var]
-
-
-def add_hash_info(packages):
-    """
-    Fills in the .has_hash field of all Package objects
-    """
-    for pkg in packages:
-        hashpath = pkg.path.replace(".mk", ".hash")
-        pkg.has_hash = os.path.exists(hashpath)
-
-
-def add_patch_count(packages):
-    """
-    Fills in the .patch_count field of all Package objects
-    """
-    for pkg in packages:
-        pkgdir = os.path.dirname(pkg.path)
-        pkg.patch_count = len(fnmatch.filter(os.listdir(pkgdir), '*.patch'))
-
-
-def get_check_package_warnings(pkgdir):
-    cmd = ["./utils/check-package"]
-    for root, dirs, files in os.walk(pkgdir):
-        for f in files:
-            if f.endswith(".mk") or f.endswith(".hash") or f == "Config.in" or f == "Config.in.host":
-                cmd.append(f)
-    o = subprocess.check_output(cmd, stderr=subprocess.STDOUT)
-    lines = o.splitlines()
-    for line in lines:
-        m = re.match("^([0-9]*) warnings generated", line)
-        if m:
-            return int(m.group(1))
-    return None
-
-
-def add_check_package_warnings(packages):
-    """
-    Fills in the .warnings field of all Package objects
-    """
-    for pkg in packages:
-        pkg.warnings = get_check_package_warnings(os.path.dirname(pkg.path))
-
-
-RELEASE_MONITORING_API = "http://release-monitoring.org/api"
-
-
-def get_latest_version_by_distro(package):
-    req = urllib2.Request(os.path.join(RELEASE_MONITORING_API, "project", "Buildroot", package))
-    f = urllib2.urlopen(req)
-    data = json.loads(f.read())
-    if len(data['versions']) > 0:
-        return (True, data['versions'][0], data['id'])
-    else:
-        return (True, None, data['id'])
-
-
-def get_latest_version_by_guess(package):
-    req = urllib2.Request(os.path.join(RELEASE_MONITORING_API, "projects", "?pattern=%s" % package))
-    f = urllib2.urlopen(req)
-    data = json.loads(f.read())
-    for p in data['projects']:
-        if p['name'] == package and len(p['versions']) > 0:
-            return (False, p['versions'][0], p['id'])
-    return (False, None, None)
-
-
-def get_latest_version(package):
-    try:
-        # We first try by using the "Buildroot" distribution on
-        # release-monitoring.org, if it has a mapping for the current
-        # package name.
-        return get_latest_version_by_distro(package)
-    except urllib2.HTTPError, e:
-        # If that fails because there is no mapping, we try to search
-        # in all packages for a package of this name.
-        if e.code == 404:
-            return get_latest_version_by_guess(package)
-        else:
-            return (False, None, None)
-
-
-def get_version_worker(q):
-    while True:
-        pkg = q.get()
-        try:
-            pkg.latest_version = get_latest_version(pkg.name)
-            print " [%04d] %s => %s" % (q.qsize(), pkg.name, str(pkg.latest_version))
-        except ValueError:
-            pkg.latest_version = (False, None, None)
-            print " [%04d] %s => Value Error" % (q.qsize(), pkg.name)
-        q.task_done()
-
-
-def add_latest_version_info(packages):
-    """
-    Fills in the .latest_version field of all Package objects
-
-    This field has a special format:
-      (mapping, version, id)
-    with:
-    - mapping: boolean that indicates whether release-monitoring.org
-      has a mapping for this package name in the Buildroot distribution
-      or not
-    - version: string containing the latest version known by
-      release-monitoring.org for this package
-    - id: string containing the id of the project corresponding to this
-      package, as known by release-monitoring.org
-    """
-    q = Queue()
-    for pkg in packages:
-        q.put(pkg)
-    # Since release-monitoring.org is rather slow, we create 8 threads
-    # that do HTTP requests to the site.
-    for i in range(8):
-        t = Thread(target=get_version_worker, args=[q])
-        t.daemon = True
-        t.start()
-    q.join()
-
-
-def calculate_stats(packages):
-    stats = defaultdict(int)
-    for pkg in packages:
-        # If packages have multiple infra, take the first one. For the
-        # vast majority of packages, the target and host infra are the
-        # same. There are very few packages that use a different infra
-        # for the host and target variants.
-        if len(pkg.infras) > 0:
-            infra = pkg.infras[0][1]
-            stats["infra-%s" % infra] += 1
-        else:
-            stats["infra-unknown"] += 1
-        if pkg.has_license:
-            stats["license"] += 1
-        else:
-            stats["no-license"] += 1
-        if pkg.has_license_files:
-            stats["license-files"] += 1
-        else:
-            stats["no-license-files"] += 1
-        if pkg.has_hash:
-            stats["hash"] += 1
-        else:
-            stats["no-hash"] += 1
-        if pkg.latest_version[0]:
-            stats["rmo-mapping"] += 1
-        else:
-            stats["rmo-no-mapping"] += 1
-        if not pkg.latest_version[1]:
-            stats["version-unknown"] += 1
-        elif pkg.latest_version[1] == pkg.current_version:
-            stats["version-uptodate"] += 1
-        else:
-            stats["version-not-uptodate"] += 1
-        stats["patches"] += pkg.patch_count
-    return stats
-
-
-html_header = """
-<head>
-<script src=\"https://www.kryogenix.org/code/browser/sorttable/sorttable.js\"></script>
-<style type=\"text/css\">
-table {
-  width: 100%;
-}
-td {
-  border: 1px solid black;
-}
-td.centered {
-  text-align: center;
-}
-td.wrong {
-  background: #ff9a69;
-}
-td.correct {
-  background: #d2ffc4;
-}
-td.nopatches {
-  background: #d2ffc4;
-}
-td.somepatches {
-  background: #ffd870;
-}
-td.lotsofpatches {
-  background: #ff9a69;
-}
-td.version-good {
-  background: #d2ffc4;
-}
-td.version-needs-update {
-  background: #ff9a69;
-}
-td.version-unknown {
- background: #ffd870;
-}
-</style>
-<title>Statistics of Buildroot packages</title>
-</head>
-
-<a href=\"#results\">Results</a><br/>
-
-<p id=\"sortable_hint\"></p>
-"""
-
-
-html_footer = """
-</body>
-<script>
-if (typeof sorttable === \"object\") {
-  document.getElementById(\"sortable_hint\").innerHTML =
-  \"hint: the table can be sorted by clicking the column headers\"
-}
-</script>
-</html>
-"""
-
-
-def infra_str(infra_list):
-    if not infra_list:
-        return "Unknown"
-    elif len(infra_list) == 1:
-        return "<b>%s</b><br/>%s" % (infra_list[0][1], infra_list[0][0])
-    elif infra_list[0][1] == infra_list[1][1]:
-        return "<b>%s</b><br/>%s + %s" % \
-            (infra_list[0][1], infra_list[0][0], infra_list[1][0])
-    else:
-        return "<b>%s</b> (%s)<br/><b>%s</b> (%s)" % \
-            (infra_list[0][1], infra_list[0][0],
-             infra_list[1][1], infra_list[1][0])
-
-
-def boolean_str(b):
-    if b:
-        return "Yes"
-    else:
-        return "No"
-
-
-def dump_html_pkg(f, pkg):
-    f.write(" <tr>\n")
-    f.write("  <td>%s</td>\n" % pkg.path[2:])
-
-    # Patch count
-    td_class = ["centered"]
-    if pkg.patch_count == 0:
-        td_class.append("nopatches")
-    elif pkg.patch_count < 5:
-        td_class.append("somepatches")
-    else:
-        td_class.append("lotsofpatches")
-    f.write("  <td class=\"%s\">%s</td>\n" %
-            (" ".join(td_class), str(pkg.patch_count)))
-
-    # Infrastructure
-    infra = infra_str(pkg.infras)
-    td_class = ["centered"]
-    if infra == "Unknown":
-        td_class.append("wrong")
-    else:
-        td_class.append("correct")
-    f.write("  <td class=\"%s\">%s</td>\n" %
-            (" ".join(td_class), infra_str(pkg.infras)))
-
-    # License
-    td_class = ["centered"]
-    if pkg.has_license:
-        td_class.append("correct")
-    else:
-        td_class.append("wrong")
-    f.write("  <td class=\"%s\">%s</td>\n" %
-            (" ".join(td_class), boolean_str(pkg.has_license)))
-
-    # License files
-    td_class = ["centered"]
-    if pkg.has_license_files:
-        td_class.append("correct")
-    else:
-        td_class.append("wrong")
-    f.write("  <td class=\"%s\">%s</td>\n" %
-            (" ".join(td_class), boolean_str(pkg.has_license_files)))
-
-    # Hash
-    td_class = ["centered"]
-    if pkg.has_hash:
-        td_class.append("correct")
-    else:
-        td_class.append("wrong")
-    f.write("  <td class=\"%s\">%s</td>\n" %
-            (" ".join(td_class), boolean_str(pkg.has_hash)))
-
-    # Current version
-    f.write("  <td class=\"centered\">%s</td>\n" % pkg.current_version)
-
-    # Latest version
-    if pkg.latest_version[1] is None:
-        td_class.append("version-unknown")
-    elif pkg.latest_version[1] != pkg.current_version:
-        td_class.append("version-needs-update")
-    else:
-        td_class.append("version-good")
-
-    if pkg.latest_version[1] is None:
-        latest_version_text = "<b>Unknown</b>"
-    else:
-        latest_version_text = "<b>%s</b>" % str(pkg.latest_version[1])
-
-    latest_version_text += "<br/>"
-
-    if pkg.latest_version[2]:
-        latest_version_text += "<a href=\"https://release-monitoring.org/project/%s\">link</a>, " % pkg.latest_version[2]
-    else:
-        latest_version_text += "no link, "
-
-    if pkg.latest_version[0]:
-        latest_version_text += "has <a href=\"https://release-monitoring.org/distro/Buildroot/\">mapping</a>"
-    else:
-        latest_version_text += "has <a href=\"https://release-monitoring.org/distro/Buildroot/\">no mapping</a>"
-
-    f.write("  <td class=\"%s\">%s</td>\n" %
-            (" ".join(td_class), latest_version_text))
-
-    # Warnings
-    td_class = ["centered"]
-    if pkg.warnings == 0:
-        td_class.append("correct")
-    else:
-        td_class.append("wrong")
-    f.write("  <td class=\"%s\">%d</td>\n" %
-            (" ".join(td_class), pkg.warnings))
-
-    f.write(" </tr>\n")
-
-
-def dump_html_all_pkgs(f, packages):
-    f.write("""
-<table class=\"sortable\">
-<tr>
-<td>Package</td>
-<td class=\"centered\">Patch count</td>
-<td class=\"centered\">Infrastructure</td>
-<td class=\"centered\">License</td>
-<td class=\"centered\">License files</td>
-<td class=\"centered\">Hash file</td>
-<td class=\"centered\">Current version</td>
-<td class=\"centered\">Latest version</td>
-<td class=\"centered\">Warnings</td>
-</tr>
-""")
-    for pkg in sorted(packages):
-        dump_html_pkg(f, pkg)
-    f.write("</table>")
-
-
-def dump_html_stats(f, stats):
-    f.write("<a id=\"results\"></a>\n")
-    f.write("<table>\n")
-    infras = [infra[6:] for infra in stats.keys() if infra.startswith("infra-")]
-    for infra in infras:
-        f.write(" <tr><td>Packages using the <i>%s</i> infrastructure</td><td>%s</td></tr>\n" %
-                (infra, stats["infra-%s" % infra]))
-    f.write(" <tr><td>Packages having license information</td><td>%s</td></tr>\n" %
-            stats["license"])
-    f.write(" <tr><td>Packages not having license information</td><td>%s</td></tr>\n" %
-            stats["no-license"])
-    f.write(" <tr><td>Packages having license files information</td><td>%s</td></tr>\n" %
-            stats["license-files"])
-    f.write(" <tr><td>Packages not having license files information</td><td>%s</td></tr>\n" %
-            stats["no-license-files"])
-    f.write(" <tr><td>Packages having a hash file</td><td>%s</td></tr>\n" %
-            stats["hash"])
-    f.write(" <tr><td>Packages not having a hash file</td><td>%s</td></tr>\n" %
-            stats["no-hash"])
-    f.write(" <tr><td>Total number of patches</td><td>%s</td></tr>\n" %
-            stats["patches"])
-    f.write("<tr><td>Packages having a mapping on <i>release-monitoring.org</i></td><td>%s</td></tr>\n" %
-            stats["rmo-mapping"])
-    f.write("<tr><td>Packages lacking a mapping on <i>release-monitoring.org</i></td><td>%s</td></tr>\n" %
-            stats["rmo-no-mapping"])
-    f.write("<tr><td>Packages that are up-to-date</td><td>%s</td></tr>\n" %
-            stats["version-uptodate"])
-    f.write("<tr><td>Packages that are not up-to-date</td><td>%s</td></tr>\n" %
-            stats["version-not-uptodate"])
-    f.write("<tr><td>Packages with no known upstream version</td><td>%s</td></tr>\n" %
-            stats["version-unknown"])
-    f.write("</table>\n")
-
-
-def dump_gen_info(f):
-    # Updated on Mon Feb 19 08:12:08 CET 2018, Git commit aa77030b8f5e41f1c53eb1c1ad664b8c814ba032
-    o = subprocess.check_output(["git", "log", "master", "-n", "1", "--pretty=format:%H"])
-    git_commit = o.splitlines()[0]
-    f.write("<p><i>Updated on %s, git commit %s</i></p>\n" %
-            (str(datetime.datetime.utcnow()), git_commit))
-
-
-def dump_html(packages, stats, output):
-    with open(output, 'w') as f:
-        f.write(html_header)
-        dump_html_all_pkgs(f, packages)
-        dump_html_stats(f, stats)
-        dump_gen_info(f)
-        f.write(html_footer)
-
-
-def parse_args():
-    parser = argparse.ArgumentParser()
-    parser.add_argument('-o', dest='output', action='store', required=True,
-                        help='HTML output file')
-    parser.add_argument('-n', dest='npackages', type=int, action='store',
-                        help='Number of packages')
-    parser.add_argument('-p', dest='packages', action='store',
-                        help='List of packages')
-    return parser.parse_args()
-
-
-def __main__():
-    args = parse_args()
-    if args.npackages and args.packages:
-        print "ERROR: -n and -p are mutually exclusive"
-        sys.exit(1)
-    if args.packages:
-        package_list = args.packages.split(",")
-    else:
-        package_list = None
-    print "Build package list ..."
-    packages = get_pkglist(args.npackages, package_list)
-    print "Get package infra ..."
-    add_infra_info(packages)
-    print "Get make info ..."
-    add_pkg_make_info(packages)
-    print "Get hash info ..."
-    add_hash_info(packages)
-    print "Get patch count ..."
-    add_patch_count(packages)
-    print "Get package warnings ..."
-    add_check_package_warnings(packages)
-    print "Get latest version ..."
-    add_latest_version_info(packages)
-    print "Calculate stats"
-    stats = calculate_stats(packages)
-    print "Write HTML"
-    dump_html(packages, stats, args.output)
-
-
-__main__()
-- 
2.14.3

[Buildroot] [PATCH next v2 0/5] New pkg-stats script, with version information

[Arnout Vandecappelle]

On 21-02-18 23:13, Thomas Petazzoni wrote:
>  - Pass BR2_HAVE_DOT_CONFIG=y when calling make, in order to fake
>    having a .config. This allows "printvars" to dump all variables
>    even without a .config.

 Perhaps it would make more sense to do this directly in the top-level Makefile:

# Some targets require the package information but don't really require a
# .config. Force BR2_HAVE_DOT_CONFIG=y for these so the packages get included.
ifneq ($(filter printvars,$(MAKECMDGOALS),)
BR2_HAVE_DOT_CONFIG = y
endif


 There are probably more targets in addition to printvars that could benefit
from this (though I can't immediately think of any).

 Regards,
 Arnout

-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF