Buildroot Archive on lore.kernel.org
 help / color / mirror / Atom feed
* [Buildroot] [PATCH 0/5] Extend pkg-stats to replace cve-checker
@ 2020-11-05 16:30 Thomas Petazzoni
  2020-11-05 16:30 ` [Buildroot] [PATCH 1/5] support/scripts/pkg-stats: allow to run script outside of the top-level directory Thomas Petazzoni
                   ` (5 more replies)
  0 siblings, 6 replies; 12+ messages in thread
From: Thomas Petazzoni @ 2020-11-05 16:30 UTC (permalink / raw)
  To: buildroot

Hello,

We recently introduced cve-checker (commit
fafa3e4e293faabc0d38a714eb88a25252936a99). But thinking more about it,
what it does is in fact very similar to pkg-stats. It even largely
borrows from pkg-stats structure and logic.

The main difference is that pkg-stats was originally written as a
Buildroot maintainer-oriented tool, where the goal is to keep an eye
on the entire set of Buildroot packages. On the other hand,
cve-checker was written as a tool mainly for a Buildroot user, to keep
an eye on the CVEs affecting just the packages currently enabled in
the current configuration.

So, what this patch series does is extend pkg-stats so that instead of
producing its output only for all Buildroot packages, it can be done
just for the set of packages enabled in the current configuration.

Here is how it goes:

 - PATCH 1 makes pkg-stats usable outside of the Buildroot top-level
   directory. This will be useful to be able to run it from any output
   directory.

 - PATCH 2 really allows pkg-satts to generate its details based on
   the set of currently configured packages. This mode is enabled
   using the new -c option.

 - PATCH 3 drops cve-checker

 - PATCH 4 promotes the pkg-stats functionality as a Makefile
   target. Note that only the "pkg-stats -c" mode is used here: we
   target the use of pkg-stats by Buildroot users, who want results
   based on their configuration. The use of pkg-stats as a maintainer
   tool is different, and we assume maintainers will know how to run
   pkg-stats.

 - PATCH 5 adds some mentions of "make show-info" and "make pkg-stats"
   in the Buildroot manual.

Note: I think this series should be merged in master, not in
next. Indeed, cve-checker is new in 2020.11-rc1, so it would probably
be a bit silly to release 2020.11 with cve-checker and remove it right
after.

Thanks,

Thomas

Thomas Petazzoni (5):
  support/scripts/pkg-stats: allow to run script outside of the
    top-level directory
  support/scripts/pkg-stats: support generating stats based on
    configured packages
  support/scripts/cve-checker: remove script
  Makefile: add pkg-stats target
  docs/manual: add some minimal documentation about show-info and
    pkg-stats

 Makefile                     |   9 ++
 docs/manual/common-usage.txt |  23 ++++
 support/scripts/cve-checker  | 196 -----------------------------------
 support/scripts/pkg-stats    |  48 ++++++---
 4 files changed, 63 insertions(+), 213 deletions(-)
 delete mode 100755 support/scripts/cve-checker

-- 
2.26.2

^ permalink raw reply	[flat|nested] 12+ messages in thread

* [Buildroot] [PATCH 1/5] support/scripts/pkg-stats: allow to run script outside of the top-level directory
  2020-11-05 16:30 [Buildroot] [PATCH 0/5] Extend pkg-stats to replace cve-checker Thomas Petazzoni
@ 2020-11-05 16:30 ` Thomas Petazzoni
  2020-11-11 11:05   ` Peter Korsgaard
  2020-11-05 16:30 ` [Buildroot] [PATCH 2/5] support/scripts/pkg-stats: support generating stats based on configured packages Thomas Petazzoni
                   ` (4 subsequent siblings)
  5 siblings, 1 reply; 12+ messages in thread
From: Thomas Petazzoni @ 2020-11-05 16:30 UTC (permalink / raw)
  To: buildroot

Currently, pkg-stats expects being executed from Buildroot's top-level
source directory. As we are going to extend pkg-stats to cover only
the packages available in the current configuration, it makes sense to
be able to run it from the output directory, which can be anywhere
compared to Buildroot's top-level directory.

This commit adjusts pkg-stats to this, by inferring all Buildroot
paths based on the location of the pkg-stats script itself.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
---
 support/scripts/pkg-stats | 38 +++++++++++++++++++++-----------------
 1 file changed, 21 insertions(+), 17 deletions(-)

diff --git a/support/scripts/pkg-stats b/support/scripts/pkg-stats
index 503cc45c16..fd6e370c18 100755
--- a/support/scripts/pkg-stats
+++ b/support/scripts/pkg-stats
@@ -28,7 +28,9 @@ import subprocess
 import json
 import sys
 
-sys.path.append('utils/')
+brpath = os.path.normpath(os.path.join(os.path.dirname(__file__), "..", ".."))
+
+sys.path.append(os.path.join(brpath, "utils"))
 from getdeveloperlib import parse_developers  # noqa: E402
 import cve as cvecheck  # noqa: E402
 
@@ -66,7 +68,7 @@ def get_defconfig_list():
     """
     return [
         Defconfig(name[:-len('_defconfig')], os.path.join('configs', name))
-        for name in os.listdir('configs')
+        for name in os.listdir(os.path.join(brpath, 'configs'))
         if name.endswith('_defconfig')
     ]
 
@@ -108,9 +110,10 @@ class Package:
         Fills in the .url field
         """
         self.status['url'] = ("warning", "no Config.in")
-        for filename in os.listdir(os.path.dirname(self.path)):
+        pkgdir = os.path.dirname(os.path.join(brpath, self.path))
+        for filename in os.listdir(pkgdir):
             if fnmatch.fnmatch(filename, 'Config.*'):
-                fp = open(os.path.join(os.path.dirname(self.path), filename), "r")
+                fp = open(os.path.join(pkgdir, filename), "r")
                 for config_line in fp:
                     if URL_RE.match(config_line):
                         self.url = config_line.strip()
@@ -138,7 +141,7 @@ class Package:
         Fills in the .infras field
         """
         self.infras = list()
-        with open(self.path, 'r') as f:
+        with open(os.path.join(brpath, self.path), 'r') as f:
             lines = f.readlines()
             for l in lines:
                 match = INFRA_RE.match(l)
@@ -178,7 +181,7 @@ class Package:
             return
 
         hashpath = self.path.replace(".mk", ".hash")
-        if os.path.exists(hashpath):
+        if os.path.exists(os.path.join(brpath, hashpath)):
             self.status['hash'] = ("ok", "found")
         else:
             self.status['hash'] = ("error", "missing")
@@ -191,7 +194,7 @@ class Package:
             self.status['patches'] = ("na", "no valid package infra")
             return
 
-        pkgdir = os.path.dirname(self.path)
+        pkgdir = os.path.dirname(os.path.join(brpath, self.path))
         for subdir, _, _ in os.walk(pkgdir):
             self.patch_files = fnmatch.filter(os.listdir(subdir), '*.patch')
 
@@ -214,8 +217,8 @@ class Package:
         """
         Fills in the .warnings and .status['pkg-check'] fields
         """
-        cmd = ["./utils/check-package"]
-        pkgdir = os.path.dirname(self.path)
+        cmd = [os.path.join(brpath, "utils/check-package")]
+        pkgdir = os.path.dirname(os.path.join(brpath, self.path))
         self.status['pkg-check'] = ("error", "Missing")
         for root, dirs, files in os.walk(pkgdir):
             for f in files:
@@ -300,11 +303,12 @@ def get_pkglist(npackages, package_list):
                      "toolchain/toolchain-wrapper.mk"]
     packages = list()
     count = 0
-    for root, dirs, files in os.walk("."):
+    for root, dirs, files in os.walk(brpath):
+        root = os.path.relpath(root, brpath)
         rootdir = root.split("/")
-        if len(rootdir) < 2:
+        if len(rootdir) < 1:
             continue
-        if rootdir[1] not in WALK_USEFUL_SUBDIRS:
+        if rootdir[0] not in WALK_USEFUL_SUBDIRS:
             continue
         for f in files:
             if not f.endswith(".mk"):
@@ -316,8 +320,7 @@ def get_pkglist(npackages, package_list):
             pkgpath = os.path.join(root, f)
             skip = False
             for exclude in WALK_EXCLUDES:
-                # pkgpath[2:] strips the initial './'
-                if re.match(exclude, pkgpath[2:]):
+                if re.match(exclude, pkgpath):
                     skip = True
                     continue
             if skip:
@@ -678,7 +681,7 @@ def boolean_str(b):
 
 def dump_html_pkg(f, pkg):
     f.write(" <tr>\n")
-    f.write("  <td>%s</td>\n" % pkg.path[2:])
+    f.write("  <td>%s</td>\n" % pkg.path)
 
     # Patch count
     td_class = ["centered"]
@@ -945,12 +948,13 @@ def __main__():
     else:
         package_list = None
     date = datetime.datetime.utcnow()
-    commit = subprocess.check_output(['git', 'rev-parse',
+    commit = subprocess.check_output(['git', '-C', brpath,
+                                      'rev-parse',
                                       'HEAD']).splitlines()[0].decode()
     print("Build package list ...")
     packages = get_pkglist(args.npackages, package_list)
     print("Getting developers ...")
-    developers = parse_developers()
+    developers = parse_developers(brpath)
     print("Build defconfig list ...")
     defconfigs = get_defconfig_list()
     for d in defconfigs:
-- 
2.26.2

^ permalink raw reply related	[flat|nested] 12+ messages in thread

* [Buildroot] [PATCH 2/5] support/scripts/pkg-stats: support generating stats based on configured packages
  2020-11-05 16:30 [Buildroot] [PATCH 0/5] Extend pkg-stats to replace cve-checker Thomas Petazzoni
  2020-11-05 16:30 ` [Buildroot] [PATCH 1/5] support/scripts/pkg-stats: allow to run script outside of the top-level directory Thomas Petazzoni
@ 2020-11-05 16:30 ` Thomas Petazzoni
  2020-11-11 11:07   ` Peter Korsgaard
  2020-11-05 16:30 ` [Buildroot] [PATCH 3/5] support/scripts/cve-checker: remove script Thomas Petazzoni
                   ` (3 subsequent siblings)
  5 siblings, 1 reply; 12+ messages in thread
From: Thomas Petazzoni @ 2020-11-05 16:30 UTC (permalink / raw)
  To: buildroot

pkg-stats was initially a Buildroot maintenance oriented tool: it was
designed to examine all Buildroot packages and provide
statistics/details about them.

However, it turns out that a number of details provided by pkg-stats,
especially CVEs, are relevant also for Buildroot users, who would like
to check regularly if their specific Buildroot configuration is
affected by CVEs or not, and possibly check if all packages have
license information, license files, etc.

The cve-checker script was recently introduced to provide an output
relatively similar to pkg-stats, but focused on CVEs only.

But in fact, its main difference is on the set of packages that we
consider: pkg-stats considers all packages, while cve-checker uses
"make show-info" to only consider packages enabled in the current
configuration.

So, this commit introduces a -c option to pkg-stats, to tell pkg-stats
to generate its output based on the list of configured packages. -c is
mutually exclusive with the -p option (explicit list of packages) and
-n option (a number of packages, picked randomly).

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
---
 support/scripts/pkg-stats | 10 ++++++++++
 1 file changed, 10 insertions(+)

diff --git a/support/scripts/pkg-stats b/support/scripts/pkg-stats
index fd6e370c18..d44f8241c1 100755
--- a/support/scripts/pkg-stats
+++ b/support/scripts/pkg-stats
@@ -333,6 +333,12 @@ def get_pkglist(npackages, package_list):
     return packages
 
 
+def get_config_packages():
+    cmd = ["make", "--no-print-directory", "show-info"]
+    js = json.loads(subprocess.check_output(cmd))
+    return js.keys()
+
+
 def package_init_make_info():
     # Fetch all variables at once
     variables = subprocess.check_output(["make", "BR2_HAVE_DOT_CONFIG=y", "-s", "printvars",
@@ -929,6 +935,8 @@ def parse_args():
     output.add_argument('--json', dest='json', type=resolvepath,
                         help='JSON output file')
     packages = parser.add_mutually_exclusive_group()
+    packages.add_argument('-c', dest='configpackages', action='store_true',
+                          help='Apply to packages enabled in current configuration')
     packages.add_argument('-n', dest='npackages', type=int, action='store',
                           help='Number of packages')
     packages.add_argument('-p', dest='packages', action='store',
@@ -945,6 +953,8 @@ def __main__():
     args = parse_args()
     if args.packages:
         package_list = args.packages.split(",")
+    elif args.configpackages:
+        package_list = get_config_packages()
     else:
         package_list = None
     date = datetime.datetime.utcnow()
-- 
2.26.2

^ permalink raw reply related	[flat|nested] 12+ messages in thread

* [Buildroot] [PATCH 3/5] support/scripts/cve-checker: remove script
  2020-11-05 16:30 [Buildroot] [PATCH 0/5] Extend pkg-stats to replace cve-checker Thomas Petazzoni
  2020-11-05 16:30 ` [Buildroot] [PATCH 1/5] support/scripts/pkg-stats: allow to run script outside of the top-level directory Thomas Petazzoni
  2020-11-05 16:30 ` [Buildroot] [PATCH 2/5] support/scripts/pkg-stats: support generating stats based on configured packages Thomas Petazzoni
@ 2020-11-05 16:30 ` Thomas Petazzoni
  2020-11-11 11:07   ` Peter Korsgaard
  2020-11-05 16:30 ` [Buildroot] [PATCH 4/5] Makefile: add pkg-stats target Thomas Petazzoni
                   ` (2 subsequent siblings)
  5 siblings, 1 reply; 12+ messages in thread
From: Thomas Petazzoni @ 2020-11-05 16:30 UTC (permalink / raw)
  To: buildroot

Now that pkg-stats is able to generate its output based on the list of
packages enabled in the current configuration, cve-checker doesn't
serve any purpose.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
---
 support/scripts/cve-checker | 196 ------------------------------------
 1 file changed, 196 deletions(-)
 delete mode 100755 support/scripts/cve-checker

diff --git a/support/scripts/cve-checker b/support/scripts/cve-checker
deleted file mode 100755
index 998ea5b8af..0000000000
--- a/support/scripts/cve-checker
+++ /dev/null
@@ -1,196 +0,0 @@
-#!/usr/bin/env python
-
-# Copyright (C) 2009 by Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
-# Copyright (C) 2020 by Gregory CLEMENT <gregory.clement@bootlin.com>
-#
-# This program is free software; you can redistribute it and/or modify
-# it under the terms of the GNU General Public License as published by
-# the Free Software Foundation; either version 2 of the License, or
-# (at your option) any later version.
-#
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
-# General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License
-# along with this program; if not, write to the Free Software
-# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
-
-import argparse
-import datetime
-import os
-import json
-import sys
-import cve as cvecheck
-
-
-class Package:
-    def __init__(self, name, version, ignored_cves):
-        self.name = name
-        self.version = version
-        self.cves = list()
-        self.ignored_cves = ignored_cves
-
-
-def check_package_cves(nvd_path, packages):
-    if not os.path.isdir(nvd_path):
-        os.makedirs(nvd_path)
-
-    for cve in cvecheck.CVE.read_nvd_dir(nvd_path):
-        for pkg_name in cve.pkg_names:
-            pkg = packages.get(pkg_name, '')
-            if pkg and cve.affects(pkg.name, pkg.version, pkg.ignored_cves) == cve.CVE_AFFECTS:
-                pkg.cves.append(cve.identifier)
-
-
-html_header = """
-<head>
-<script src=\"https://www.kryogenix.org/code/browser/sorttable/sorttable.js\"></script>
-<style type=\"text/css\">
-table {
-  width: 100%;
-}
-td {
-  border: 1px solid black;
-}
-td.centered {
-  text-align: center;
-}
-td.wrong {
-  background: #ff9a69;
-}
-td.correct {
-  background: #d2ffc4;
-}
-
-</style>
-<title>CVE status for Buildroot configuration</title>
-</head>
-
-<p id=\"sortable_hint\"></p>
-"""
-
-
-html_footer = """
-</body>
-<script>
-if (typeof sorttable === \"object\") {
-  document.getElementById(\"sortable_hint\").innerHTML =
-  \"hint: the table can be sorted by clicking the column headers\"
-}
-</script>
-</html>
-"""
-
-
-def dump_html_pkg(f, pkg):
-    f.write(" <tr>\n")
-    f.write("  <td>%s</td>\n" % pkg.name)
-
-    # Current version
-    if len(pkg.version) > 20:
-        version = pkg.version[:20] + "..."
-    else:
-        version = pkg.version
-    f.write("  <td class=\"centered\">%s</td>\n" % version)
-
-    # CVEs
-    td_class = ["centered"]
-    if len(pkg.cves) == 0:
-        td_class.append("correct")
-    else:
-        td_class.append("wrong")
-    f.write("  <td class=\"%s\">\n" % " ".join(td_class))
-    for cve in pkg.cves:
-        f.write("   <a href=\"https://security-tracker.debian.org/tracker/%s\">%s<br/>\n" % (cve, cve))
-    f.write("  </td>\n")
-
-    f.write(" </tr>\n")
-
-
-def dump_html_all_pkgs(f, packages):
-    f.write("""
-<table class=\"sortable\">
-<tr>
-<td>Package</td>
-<td class=\"centered\">Version</td>
-<td class=\"centered\">CVEs</td>
-</tr>
-""")
-    for pkg in packages:
-        dump_html_pkg(f, pkg)
-    f.write("</table>")
-
-
-def dump_html_gen_info(f, date):
-    f.write("<p><i>Generated on %s</i></p>\n" % (str(date)))
-
-
-def dump_html(packages, date, output):
-    with open(output, 'w') as f:
-        f.write(html_header)
-        dump_html_all_pkgs(f, packages)
-        dump_html_gen_info(f, date)
-        f.write(html_footer)
-
-
-def dump_json(packages, date, output):
-    # Format packages as a dictionnary instead of a list
-    pkgs = {
-        pkg.name: {
-            "version": pkg.version,
-            "cves": pkg.cves,
-        } for pkg in packages
-    }
-    # The actual structure to dump, add date to it
-    final = {'packages': pkgs,
-             'date': str(date)}
-    with open(output, 'w') as f:
-        json.dump(final, f, indent=2, separators=(',', ': '))
-        f.write('\n')
-
-
-def resolvepath(path):
-        return os.path.abspath(os.path.expanduser(path))
-
-
-def parse_args():
-    parser = argparse.ArgumentParser()
-    output = parser.add_argument_group('output', 'Output file(s)')
-    output.add_argument('--html', dest='html', type=resolvepath,
-                        help='HTML output file')
-    output.add_argument('--json', dest='json', type=resolvepath,
-                        help='JSON output file')
-    parser.add_argument('--nvd-path', dest='nvd_path',
-                        help='Path to the local NVD database', type=resolvepath,
-                        required=True)
-    args = parser.parse_args()
-    if not args.html and not args.json:
-        parser.error('at least one of --html or --json (or both) is required')
-    return args
-
-
-def __main__():
-    packages = list()
-    content = json.load(sys.stdin)
-    for item in content:
-        pkg = content[item]
-        p = Package(item, pkg.get('version', ''), pkg.get('ignore_cves', ''))
-        packages.append(p)
-
-    args = parse_args()
-    date = datetime.datetime.utcnow()
-
-    print("Checking packages CVEs")
-    check_package_cves(args.nvd_path, {p.name: p for p in packages})
-
-    if args.html:
-        print("Write HTML")
-        dump_html(packages, date, args.html)
-    if args.json:
-        print("Write JSON")
-        dump_json(packages, date, args.json)
-
-
-__main__()
-- 
2.26.2

^ permalink raw reply related	[flat|nested] 12+ messages in thread

* [Buildroot] [PATCH 4/5] Makefile: add pkg-stats target
  2020-11-05 16:30 [Buildroot] [PATCH 0/5] Extend pkg-stats to replace cve-checker Thomas Petazzoni
                   ` (2 preceding siblings ...)
  2020-11-05 16:30 ` [Buildroot] [PATCH 3/5] support/scripts/cve-checker: remove script Thomas Petazzoni
@ 2020-11-05 16:30 ` Thomas Petazzoni
  2020-11-11 11:12   ` Peter Korsgaard
  2020-11-05 16:30 ` [Buildroot] [PATCH 5/5] docs/manual: add some minimal documentation about show-info and pkg-stats Thomas Petazzoni
  2020-11-06 14:59 ` [Buildroot] [PATCH 0/5] Extend pkg-stats to replace cve-checker Gregory CLEMENT
  5 siblings, 1 reply; 12+ messages in thread
From: Thomas Petazzoni @ 2020-11-05 16:30 UTC (permalink / raw)
  To: buildroot

Now that pkg-stats is not just a maintainer-oriented tool, but a tool
generally useful to users, introduce a make target to run
pkg-stats. Of course, it is run with the newly introduced -c option,
which produces a pkg-stats output for just the selection of packages
of the currently defined configuration.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
---
 Makefile | 9 +++++++++
 1 file changed, 9 insertions(+)

diff --git a/Makefile b/Makefile
index d4751e278e..ad5121a062 100644
--- a/Makefile
+++ b/Makefile
@@ -937,6 +937,14 @@ show-info:
 		) \
 	)
 
+.PHONY: pkg-stats
+pkg-stats:
+	@cd "$(CONFIG_DIR)" ; \
+	$(TOPDIR)/support/scripts/pkg-stats -c \
+		--json $(O)/pkg-stats.json \
+		--html $(O)/pkg-stats.html \
+		--nvd-path $(DL_DIR)/buildroot-nvd
+
 else # ifeq ($(BR2_HAVE_DOT_CONFIG),y)
 
 # Some subdirectories are also package names. To avoid that "make linux"
@@ -1156,6 +1164,7 @@ help:
 	@echo '  external-deps          - list external packages used'
 	@echo '  legal-info             - generate info about license compliance'
 	@echo '  show-info              - generate info about packages, as a JSON blurb'
+	@echo '  pkg-stats              - generate info about packages as JSON and HTML'
 	@echo '  printvars              - dump internal variables selected with VARS=...'
 	@echo
 	@echo '  make V=0|1             - 0 => quiet build (default), 1 => verbose build'
-- 
2.26.2

^ permalink raw reply related	[flat|nested] 12+ messages in thread

* [Buildroot] [PATCH 5/5] docs/manual: add some minimal documentation about show-info and pkg-stats
  2020-11-05 16:30 [Buildroot] [PATCH 0/5] Extend pkg-stats to replace cve-checker Thomas Petazzoni
                   ` (3 preceding siblings ...)
  2020-11-05 16:30 ` [Buildroot] [PATCH 4/5] Makefile: add pkg-stats target Thomas Petazzoni
@ 2020-11-05 16:30 ` Thomas Petazzoni
  2020-11-11 11:12   ` Peter Korsgaard
  2020-11-06 14:59 ` [Buildroot] [PATCH 0/5] Extend pkg-stats to replace cve-checker Gregory CLEMENT
  5 siblings, 1 reply; 12+ messages in thread
From: Thomas Petazzoni @ 2020-11-05 16:30 UTC (permalink / raw)
  To: buildroot

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
---
 docs/manual/common-usage.txt | 23 +++++++++++++++++++++++
 1 file changed, 23 insertions(+)

diff --git a/docs/manual/common-usage.txt b/docs/manual/common-usage.txt
index 7cfda10365..9ba87a8339 100644
--- a/docs/manual/common-usage.txt
+++ b/docs/manual/common-usage.txt
@@ -157,6 +157,29 @@ your filesystem, those parts may not be all-zeroes when read back). You
 should only use sparse files when handling files on the build machine, not
 when transferring them to an actual device that will be used on the target.
 
+=== Details about packages
+
+[[package-details]]
+
+Buildroot can produce a JSON blurb that describes the set of enabled
+packages in the current configuration, together with their
+dependencies, licenses and other metadata. This JSON blurb is produced
+by using the +show-info+ make target:
+
+------------------------
+make show-info
+------------------------
+
+Buildroot can also produce details about packages as HTML and JSON
+output using the +pkg-stats+ make target. Amongst other things, these
+details include whether known CVEs (security vulnerabilities) affect
+the packages in your current configuration. It also shows if there is
+a newer upstream version for those packages.
+
+------------------------
+make pkg-stats
+------------------------
+
 === Graphing the dependencies between packages
 
 [[graph-depends]]
-- 
2.26.2

^ permalink raw reply related	[flat|nested] 12+ messages in thread

* [Buildroot] [PATCH 0/5] Extend pkg-stats to replace cve-checker
  2020-11-05 16:30 [Buildroot] [PATCH 0/5] Extend pkg-stats to replace cve-checker Thomas Petazzoni
                   ` (4 preceding siblings ...)
  2020-11-05 16:30 ` [Buildroot] [PATCH 5/5] docs/manual: add some minimal documentation about show-info and pkg-stats Thomas Petazzoni
@ 2020-11-06 14:59 ` Gregory CLEMENT
  5 siblings, 0 replies; 12+ messages in thread
From: Gregory CLEMENT @ 2020-11-06 14:59 UTC (permalink / raw)
  To: buildroot

Hello Thomas,

> Hello,
>
> We recently introduced cve-checker (commit
> fafa3e4e293faabc0d38a714eb88a25252936a99). But thinking more about it,
> what it does is in fact very similar to pkg-stats. It even largely
> borrows from pkg-stats structure and logic.
>
> The main difference is that pkg-stats was originally written as a
> Buildroot maintainer-oriented tool, where the goal is to keep an eye
> on the entire set of Buildroot packages. On the other hand,
> cve-checker was written as a tool mainly for a Buildroot user, to keep
> an eye on the CVEs affecting just the packages currently enabled in
> the current configuration.
>
> So, what this patch series does is extend pkg-stats so that instead of
> producing its output only for all Buildroot packages, it can be done
> just for the set of packages enabled in the current configuration.


Your series looks good and I think it is a good things to finally remove
cve-checker. I introduced cve.py in order to share code between
cve-checker and pkg-stats but in the end there was sill a lot of
duplicate code, and each evolution was really painful, as it was needed
to duplicate it in each script but with very few difference.

Gregory

>
> Here is how it goes:
>
>  - PATCH 1 makes pkg-stats usable outside of the Buildroot top-level
>    directory. This will be useful to be able to run it from any output
>    directory.
>
>  - PATCH 2 really allows pkg-satts to generate its details based on
>    the set of currently configured packages. This mode is enabled
>    using the new -c option.
>
>  - PATCH 3 drops cve-checker
>
>  - PATCH 4 promotes the pkg-stats functionality as a Makefile
>    target. Note that only the "pkg-stats -c" mode is used here: we
>    target the use of pkg-stats by Buildroot users, who want results
>    based on their configuration. The use of pkg-stats as a maintainer
>    tool is different, and we assume maintainers will know how to run
>    pkg-stats.
>
>  - PATCH 5 adds some mentions of "make show-info" and "make pkg-stats"
>    in the Buildroot manual.
>
> Note: I think this series should be merged in master, not in
> next. Indeed, cve-checker is new in 2020.11-rc1, so it would probably
> be a bit silly to release 2020.11 with cve-checker and remove it right
> after.
>
> Thanks,
>
> Thomas
>
> Thomas Petazzoni (5):
>   support/scripts/pkg-stats: allow to run script outside of the
>     top-level directory
>   support/scripts/pkg-stats: support generating stats based on
>     configured packages
>   support/scripts/cve-checker: remove script
>   Makefile: add pkg-stats target
>   docs/manual: add some minimal documentation about show-info and
>     pkg-stats
>
>  Makefile                     |   9 ++
>  docs/manual/common-usage.txt |  23 ++++
>  support/scripts/cve-checker  | 196 -----------------------------------
>  support/scripts/pkg-stats    |  48 ++++++---
>  4 files changed, 63 insertions(+), 213 deletions(-)
>  delete mode 100755 support/scripts/cve-checker
>
> -- 
> 2.26.2
>

-- 
Gregory Clement, Bootlin
Embedded Linux and Kernel engineering
http://bootlin.com

^ permalink raw reply	[flat|nested] 12+ messages in thread

* [Buildroot] [PATCH 1/5] support/scripts/pkg-stats: allow to run script outside of the top-level directory
  2020-11-05 16:30 ` [Buildroot] [PATCH 1/5] support/scripts/pkg-stats: allow to run script outside of the top-level directory Thomas Petazzoni
@ 2020-11-11 11:05   ` Peter Korsgaard
  0 siblings, 0 replies; 12+ messages in thread
From: Peter Korsgaard @ 2020-11-11 11:05 UTC (permalink / raw)
  To: buildroot

>>>>> "Thomas" == Thomas Petazzoni <thomas.petazzoni@bootlin.com> writes:

 > Currently, pkg-stats expects being executed from Buildroot's top-level
 > source directory. As we are going to extend pkg-stats to cover only
 > the packages available in the current configuration, it makes sense to
 > be able to run it from the output directory, which can be anywhere
 > compared to Buildroot's top-level directory.

 > This commit adjusts pkg-stats to this, by inferring all Buildroot
 > paths based on the location of the pkg-stats script itself.

 > Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>

Committed, thanks.

-- 
Bye, Peter Korsgaard

^ permalink raw reply	[flat|nested] 12+ messages in thread

* [Buildroot] [PATCH 2/5] support/scripts/pkg-stats: support generating stats based on configured packages
  2020-11-05 16:30 ` [Buildroot] [PATCH 2/5] support/scripts/pkg-stats: support generating stats based on configured packages Thomas Petazzoni
@ 2020-11-11 11:07   ` Peter Korsgaard
  0 siblings, 0 replies; 12+ messages in thread
From: Peter Korsgaard @ 2020-11-11 11:07 UTC (permalink / raw)
  To: buildroot

>>>>> "Thomas" == Thomas Petazzoni <thomas.petazzoni@bootlin.com> writes:

 > pkg-stats was initially a Buildroot maintenance oriented tool: it was
 > designed to examine all Buildroot packages and provide
 > statistics/details about them.

 > However, it turns out that a number of details provided by pkg-stats,
 > especially CVEs, are relevant also for Buildroot users, who would like
 > to check regularly if their specific Buildroot configuration is
 > affected by CVEs or not, and possibly check if all packages have
 > license information, license files, etc.

 > The cve-checker script was recently introduced to provide an output
 > relatively similar to pkg-stats, but focused on CVEs only.

 > But in fact, its main difference is on the set of packages that we
 > consider: pkg-stats considers all packages, while cve-checker uses
 > "make show-info" to only consider packages enabled in the current
 > configuration.

 > So, this commit introduces a -c option to pkg-stats, to tell pkg-stats
 > to generate its output based on the list of configured packages. -c is
 > mutually exclusive with the -p option (explicit list of packages) and
 > -n option (a number of packages, picked randomly).
 > Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>

Committed, thanks.

-- 
Bye, Peter Korsgaard

^ permalink raw reply	[flat|nested] 12+ messages in thread

* [Buildroot] [PATCH 3/5] support/scripts/cve-checker: remove script
  2020-11-05 16:30 ` [Buildroot] [PATCH 3/5] support/scripts/cve-checker: remove script Thomas Petazzoni
@ 2020-11-11 11:07   ` Peter Korsgaard
  0 siblings, 0 replies; 12+ messages in thread
From: Peter Korsgaard @ 2020-11-11 11:07 UTC (permalink / raw)
  To: buildroot

>>>>> "Thomas" == Thomas Petazzoni <thomas.petazzoni@bootlin.com> writes:

 > Now that pkg-stats is able to generate its output based on the list of
 > packages enabled in the current configuration, cve-checker doesn't
 > serve any purpose.

 > Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>

Committed, thanks.

-- 
Bye, Peter Korsgaard

^ permalink raw reply	[flat|nested] 12+ messages in thread

* [Buildroot] [PATCH 4/5] Makefile: add pkg-stats target
  2020-11-05 16:30 ` [Buildroot] [PATCH 4/5] Makefile: add pkg-stats target Thomas Petazzoni
@ 2020-11-11 11:12   ` Peter Korsgaard
  0 siblings, 0 replies; 12+ messages in thread
From: Peter Korsgaard @ 2020-11-11 11:12 UTC (permalink / raw)
  To: buildroot

>>>>> "Thomas" == Thomas Petazzoni <thomas.petazzoni@bootlin.com> writes:

 > Now that pkg-stats is not just a maintainer-oriented tool, but a tool
 > generally useful to users, introduce a make target to run
 > pkg-stats. Of course, it is run with the newly introduced -c option,
 > which produces a pkg-stats output for just the selection of packages
 > of the currently defined configuration.

 > Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>

Committed, thanks.

-- 
Bye, Peter Korsgaard

^ permalink raw reply	[flat|nested] 12+ messages in thread

* [Buildroot] [PATCH 5/5] docs/manual: add some minimal documentation about show-info and pkg-stats
  2020-11-05 16:30 ` [Buildroot] [PATCH 5/5] docs/manual: add some minimal documentation about show-info and pkg-stats Thomas Petazzoni
@ 2020-11-11 11:12   ` Peter Korsgaard
  0 siblings, 0 replies; 12+ messages in thread
From: Peter Korsgaard @ 2020-11-11 11:12 UTC (permalink / raw)
  To: buildroot

>>>>> "Thomas" == Thomas Petazzoni <thomas.petazzoni@bootlin.com> writes:

 > Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>

Committed, thanks.

-- 
Bye, Peter Korsgaard

^ permalink raw reply	[flat|nested] 12+ messages in thread

end of thread, other threads:[~2020-11-11 11:12 UTC | newest]

Thread overview: 12+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2020-11-05 16:30 [Buildroot] [PATCH 0/5] Extend pkg-stats to replace cve-checker Thomas Petazzoni
2020-11-05 16:30 ` [Buildroot] [PATCH 1/5] support/scripts/pkg-stats: allow to run script outside of the top-level directory Thomas Petazzoni
2020-11-11 11:05   ` Peter Korsgaard
2020-11-05 16:30 ` [Buildroot] [PATCH 2/5] support/scripts/pkg-stats: support generating stats based on configured packages Thomas Petazzoni
2020-11-11 11:07   ` Peter Korsgaard
2020-11-05 16:30 ` [Buildroot] [PATCH 3/5] support/scripts/cve-checker: remove script Thomas Petazzoni
2020-11-11 11:07   ` Peter Korsgaard
2020-11-05 16:30 ` [Buildroot] [PATCH 4/5] Makefile: add pkg-stats target Thomas Petazzoni
2020-11-11 11:12   ` Peter Korsgaard
2020-11-05 16:30 ` [Buildroot] [PATCH 5/5] docs/manual: add some minimal documentation about show-info and pkg-stats Thomas Petazzoni
2020-11-11 11:12   ` Peter Korsgaard
2020-11-06 14:59 ` [Buildroot] [PATCH 0/5] Extend pkg-stats to replace cve-checker Gregory CLEMENT

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox