[PATCHv2 0/2] Add initial capability to check CVEs for recipes

[PATCHv2 0/2] Add initial capability to check CVEs for recipes

[mariano.lopez]

From: Mariano Lopez <mariano.lopez@linux.intel.com>

This series add the cve-check-tool recipe, a tool used to identify
potentially vulnerable software through version matching. It will
check if a vulnerability has been addressed by a patch.

Also add the new cve-check class that will add a task for all recipes
to check for CVEs using cve-check-tool. This tool can be used by recipe,
mage (will generate an image report in deploy dir), and with "world"
and "universe"

To run it just inherit the class and enter:

bitbake -c cve_check <recipe>

Changes in v2:

- Set the explicit dependency of cve-check-tool-native:do_populate_cve_db
  in the cve-check class
- Squashed patch 2 into patch 1

The following changes since commit 23056103c949b498c23b47579e8dd57ce78e6ed9:

  uclibc: Do not use immediate expansion operator (2016-02-22 20:42:48 +0000)

are available in the git repository at:

  git://git.yoctoproject.org/poky-contrib mariano/bug7515v2
  http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=mariano/bug7515v2

Mariano Lopez (2):
  cve-check-tool: Add recipe
  cve-check.bbclass: Add class

 meta/classes/cve-check.bbclass                     | 248 +++++++++++++++++++++
 .../change_logic_cve_get_file_parent.patch         |  45 ++++
 .../cve-check-tool/cve-check-tool_5.6.bb           |  54 +++++
 3 files changed, 347 insertions(+)
 create mode 100644 meta/classes/cve-check.bbclass
 create mode 100644 meta/recipes-devtools/cve-check-tool/cve-check-tool/change_logic_cve_get_file_parent.patch
 create mode 100644 meta/recipes-devtools/cve-check-tool/cve-check-tool_5.6.bb

-- 
2.6.2

[PATCHv2 1/2] cve-check-tool: Add recipe

[mariano.lopez]

From: Mariano Lopez <mariano.lopez@linux.intel.com>

cve-check-tool is a program to for checking public CVEs.
This tool also seek to determine if a vulnerability has
been addressed by a patch.

The recipe also includes the do_populate_cve_db task
that will populate the database used by the tool. This
task is added when the cve-check class has been inherited.

[YOCTO #7515]

Co-authored by Elena Reshetova & Mariano Lopez

Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
---
 .../change_logic_cve_get_file_parent.patch         | 45 ++++++++++++++++++
 .../cve-check-tool/cve-check-tool_5.6.bb           | 54 ++++++++++++++++++++++
 2 files changed, 99 insertions(+)
 create mode 100644 meta/recipes-devtools/cve-check-tool/cve-check-tool/change_logic_cve_get_file_parent.patch
 create mode 100644 meta/recipes-devtools/cve-check-tool/cve-check-tool_5.6.bb

diff --git a/meta/recipes-devtools/cve-check-tool/cve-check-tool/change_logic_cve_get_file_parent.patch b/meta/recipes-devtools/cve-check-tool/cve-check-tool/change_logic_cve_get_file_parent.patch
new file mode 100644
index 0000000..077de88
--- /dev/null
+++ b/meta/recipes-devtools/cve-check-tool/cve-check-tool/change_logic_cve_get_file_parent.patch
@@ -0,0 +1,45 @@
+From 22cc9186909f98f024d78a08504d0bf532806de0 Mon Sep 17 00:00:00 2001
+From: Mariano Lopez <mariano.lopez@linux.intel.com>
+Date: Thu, 18 Feb 2016 14:26:02 +0000
+Subject: [PATCH] util.c: Change logic in cve_get_file_parent()
+
+Function cve_get_file_parent() will try to get the
+realpath and the get the dirname. If the file used
+to get parent doesn't exist the call will fail.
+
+This problem is present when using another directory
+for the database, realpath() won't find the nvd.db
+file and the program will exit quitely.
+
+This patch will first get the dirname and the get
+the realpath to avoid failing when the doesn't exist.
+
+Upstream-Status: Accepted [Release v5.6.3]
+
+Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
+---
+ src/library/util.c | 8 +++-----
+ 1 file changed, 3 insertions(+), 5 deletions(-)
+
+diff --git a/src/library/util.c b/src/library/util.c
+index 8a20728..4d4a576 100644
+--- a/src/library/util.c
++++ b/src/library/util.c
+@@ -184,11 +184,9 @@ bool cve_is_dir(const char *p)
+ 
+ char *cve_get_file_parent(const char *p)
+ {
+-        char *r = realpath(p, NULL);
+-        if (!r) {
+-                return NULL;
+-        }
+-        return dirname(r);
++        autofree(char) *d = strdup(p);
++        char *r = realpath(dirname(d), NULL);
++        return r;
+ }
+ 
+ bool cve_file_set_text(const char *path, char *text)
+-- 
+2.6.2
+
diff --git a/meta/recipes-devtools/cve-check-tool/cve-check-tool_5.6.bb b/meta/recipes-devtools/cve-check-tool/cve-check-tool_5.6.bb
new file mode 100644
index 0000000..2315058
--- /dev/null
+++ b/meta/recipes-devtools/cve-check-tool/cve-check-tool_5.6.bb
@@ -0,0 +1,54 @@
+SUMMARY = "cve-check-tool"
+DESCRIPTION = "cve-check-tool is a tool for checking known (public) CVEs.\
+The tool will identify potentially vunlnerable software packages within Linux distributions through version matching."
+HOMEPAGE = "https://github.com/ikeydoherty/cve-check-tool"
+SECTION = "Development/Tools"
+LICENSE = "GPL-2.0"
+LIC_FILES_CHKSUM = "file://LICENSE;md5=e8c1458438ead3c34974bc0be3a03ed6"
+
+SRC_URI = "https://github.com/ikeydoherty/${BPN}/releases/download/v${PV}/${BP}.tar.xz \
+            file://change_logic_cve_get_file_parent.patch"
+
+SRC_URI[md5sum] = "30f32e6254580162eacfcc437a144463"
+SRC_URI[sha256sum] = "d35af2bfa014b9d7cdc9c59ec0bd7df40c22dfcd57244c9099c0aa9bdc9c0cb4"
+
+DEPENDS = "libcheck glib-2.0 json-glib curl libxml2 sqlite3 openssl"
+
+inherit pkgconfig autotools
+
+EXTRA_OECONF = "--disable-static"
+
+python do_populate_cve_db () {
+    import subprocess
+    import time
+    from bb.utils import export_proxies
+
+    export_proxies(d)
+    fail_text = "Failed to update database"
+    error_str = fail_text
+    cve_dir = d.getVar("CVE_CHECK_DB_DIR", True)
+    cmd = "cve-check-update -d %s" % cve_dir
+    bb.debug(1, "Updating cve-check-tool database located in %s" % cve_dir)
+    try:
+        popen = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
+        output, error = popen.communicate()
+        bb.debug(2, "Command %s returned:\n%s" % (cmd, output.decode()))
+        error_str = error.decode()
+        bb.debug(2, "Command %s errors:\n%s" % (cmd, error_str))
+    except:
+        bb.warn("Error in executing cve-check-update: %s" % str(sys.exc_info()))
+
+    if fail_text in error_str:
+        bb.warn("Failed to update cve-check-tool database, CVEs won't be checked")
+    else:
+        utc_time = time.gmtime(time.time())
+        format_time = "%Y-%m-%d %H:%M:%S"
+        with open(d.getVar("CVE_CHECK_TMP_FILE", True), "w") as f:
+            f.write("CVE database was updated on %s UTC\n\n"
+                    % time.strftime(format_time, utc_time))
+}
+
+addtask populate_cve_db after do_populate_sysroot
+do_populate_cve_db[nostamp] = "1"
+
+BBCLASSEXTEND = "native"
-- 
2.6.2

[PATCHv2 2/2] cve-check.bbclass: Add class

[mariano.lopez]

From: Mariano Lopez <mariano.lopez@linux.intel.com>

This class adds a new task for all the recipes to use
cve-check-tool in order to look for public CVEs affecting
the packages generated.

It is possible to use this class when building an image,
building a recipe, or using the "world" or "universe" cases.

In order to use this class it must be inherited at some
point and it will add the task automatically to every recipe.

[YOCTO #7515]

Co-authored by Ross Burton & Mariano Lopez

Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
---
 meta/classes/cve-check.bbclass | 248 +++++++++++++++++++++++++++++++++++++++++
 1 file changed, 248 insertions(+)
 create mode 100644 meta/classes/cve-check.bbclass

diff --git a/meta/classes/cve-check.bbclass b/meta/classes/cve-check.bbclass
new file mode 100644
index 0000000..ffa24c3
--- /dev/null
+++ b/meta/classes/cve-check.bbclass
@@ -0,0 +1,248 @@
+# This class is used to check recipes against public CVEs.
+#
+# In order to use this class just inherit the class in the
+# local.conf file and it will add the cve_check task for
+# every recipe. The task can be used per recipe, per image,
+# or using the special cases "world" and "universe". The
+# cve_check task will print a warning for every unpatched
+# CVE found and generate a file in the recipe WORKDIR/cve
+# directory. If an image is build it will generate a report
+# in DEPLOY_DIR_IMAGE for all the packages used.
+#
+# Example:
+#   bitbake -c cve_check openssl
+#   bitbake core-image-sato
+#   bitbake -k -c cve_check universe
+#
+# DISCLAIMER
+#
+# This class/tool is meant to be used as support and not
+# the only method to check against CVEs. Running this tool
+# doesn't guarantee your packages are free of CVEs.
+
+CVE_CHECK_DB_DIR ?= "${DL_DIR}/CVE_CHECK"
+CVE_CHECK_DB_FILE ?= "${CVE_CHECK_DB_DIR}/nvd.db"
+
+CVE_CHECK_LOCAL_DIR ?= "${WORKDIR}/cve"
+CVE_CHECK_LOCAL_FILE ?= "${CVE_CHECK_LOCAL_DIR}/cve.log"
+CVE_CHECK_TMP_FILE ?= "${TMPDIR}/cve_check"
+
+CVE_CHECK_DIR ??= "${DEPLOY_DIR}/cve"
+CVE_CHECK_MANIFEST ?= "${DEPLOY_DIR_IMAGE}/${IMAGE_NAME}.rootfs.cve"
+CVE_CHECK_COPY_FILES ??= "1"
+CVE_CHECK_CREATE_MANIFEST ??= "1"
+
+# Whitelist for packages (PN)
+cve_check_pn_whitelist () {
+    glibc-locale
+}
+
+# Whitelist for CVE and version of package
+python cve_check_cve_whitelist () {
+    {"CVE-2014-2524": ("6.3",), \
+    }
+}
+
+python do_cve_check () {
+    """
+    Check recipe for patched and unpatched CVEs
+    """
+
+    if os.path.exists(d.getVar("CVE_CHECK_TMP_FILE", True)):
+        patched_cves = get_patches_cves(d)
+        patched, unpatched = check_cves(d, patched_cves)
+        if patched or unpatched:
+            cve_data = get_cve_info(d, patched + unpatched)
+            cve_write_data(d, patched, unpatched, cve_data)
+    else:
+        bb.note("Failed to update CVE database, skipping CVE check")
+}
+
+addtask cve_check before do_build
+do_cve_check[depends] = "cve-check-tool-native:do_populate_cve_db"
+do_cve_check[nostamp] = "1"
+
+python cve_check_cleanup () {
+    """
+    Delete the file used to gather all the CVE information.
+    """
+    import bb.utils
+
+    tmp_file = e.data.getVar("CVE_CHECK_TMP_FILE", True)
+    bb.utils.remove(tmp_file)
+}
+
+addhandler cve_check_cleanup
+cve_check_cleanup[eventmask] = "bb.cooker.CookerExit"
+
+python cve_check_write_rootfs_manifest () {
+    """
+    Create CVE manifest when building an image
+    """
+
+    import shutil
+    from bb.utils import mkdirhier
+
+    if os.path.exists(d.getVar("CVE_CHECK_TMP_FILE", True)) and \
+            d.getVar("CVE_CHECK_CREATE_MANIFEST", True) == "1":
+        bb.note("Writing rootfs CVE manifest")
+        deploy_dir = d.getVar("DEPLOY_DIR_IMAGE", True)
+        link_name = d.getVar("IMAGE_LINK_NAME", True)
+        manifest_name = d.getVar("CVE_CHECK_MANIFEST", True)
+        cve_tmp_file = d.getVar("CVE_CHECK_TMP_FILE", True)
+
+        shutil.copyfile(cve_tmp_file, manifest_name)
+
+        if manifest_name is not None and os.path.exists(manifest_name):
+            manifest_link = os.path.join(deploy_dir, "%s.cve" % link_name)
+            if os.path.exists(manifest_link):
+                if d.getVar('RM_OLD_IMAGE', True) == "1" and \
+                        os.path.exists(os.path.realpath(manifest_link)):
+                    os.remove(os.path.realpath(manifest_link))
+                os.remove(manifest_link)
+            os.symlink(os.path.basename(manifest_name), manifest_link)
+            bb.plain("Image CVE report stored in: %s" % manifest_name)
+}
+
+ROOTFS_POSTPROCESS_COMMAND_prepend = "cve_check_write_rootfs_manifest; "
+
+def get_patches_cves(d):
+    """
+    Get patches that solve CVEs using the "CVE: " tag.
+    """
+
+    import re
+
+    pn = d.getVar("PN", True)
+    cve_match = re.compile("CVE:( CVE\-\d+\-\d+)+")
+    patched_cves = set()
+    for url in src_patches(d):
+        patch_file = bb.fetch.decodeurl(url)[2]
+        with open(patch_file, "r") as f:
+            patch_text = f.read()
+
+        # Search for the "CVE: " line
+        match = cve_match.search(patch_text)
+        if match:
+            # Get only the CVEs without the "CVE: " tag
+            cves = patch_text[match.start()+5:match.end()]
+            for cve in cves.split():
+                patched_cves.add(cve)
+
+    return patched_cves
+
+def check_cves(d, patched_cves):
+    """
+    Run cve-check-tool looking for patched and unpatched CVEs.
+    """
+
+    from bb.utils import export_proxies
+    import ast, csv, tempfile, subprocess, StringIO
+
+    cves_patched = []
+    cves_unpatched = []
+    bpn = d.getVar("BPN", True)
+    pv = d.getVar("PV", True)
+    cves = " ".join(patched_cves)
+    cve_dir = d.getVar("CVE_CHECK_DB_DIR", True)
+    cve_whitelist = ast.literal_eval(d.getVar("cve_check_cve_whitelist", True).strip())
+    cmd = "cve-check-tool --no-html --csv --not-affected -t faux -d %s" % cve_dir
+
+    # If the recipe has been whitlisted we return empty lists
+    if d.getVar("PN", True) in d.getVar("cve_check_pn_whitelist", True).split():
+        return ([], [])
+
+    # It is needed to export the proxies to download the database using HTTP
+    export_proxies(d)
+    # Write the faux CSV file to be used with cve-check-tool
+    fd, faux = tempfile.mkstemp(prefix="cve-faux-")
+    with os.fdopen(fd, "w") as f:
+        f.write("%s,%s,%s," % (bpn, pv, cves))
+
+    cmd += " %s" % faux
+    try:
+        popen = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
+        output, error = popen.communicate()
+    except:
+        bb.warn("Couldn't check CVEs %s" % str(sys.exc_info()))
+    finally:
+        os.remove(faux)
+
+    for row in csv.reader(StringIO.StringIO(output)):
+        if row[2]:
+            for cve in row[2].split():
+                # Skip if the CVE has been whitlisted for the current version
+                if pv not in cve_whitelist.get(cve,[]):
+                    cves_unpatched.append(cve)
+        if row[3]:
+            for cve in row[3].split():
+                cves_patched.append(cve)
+
+    return (cves_patched, cves_unpatched)
+
+def get_cve_info(d, cves):
+    """
+    Get CVE information from the database used by cve-check-tool.
+    """
+
+    try:
+        import sqlite3
+    except ImportError:
+        from pysqlite2 import dbapi2 as sqlite3
+
+    cve_data = {}
+    db_file = d.getVar("CVE_CHECK_DB_FILE", True)
+    placeholder = ",".join("?" * len(cves))
+    query = "SELECT * FROM NVD WHERE id IN (%s)" % placeholder
+    conn = sqlite3.connect(db_file)
+    cur = conn.cursor()
+    for row in cur.execute(query, tuple(cves)):
+        cve_data[row[0]] = {}
+        cve_data[row[0]]["summary"] = row[1]
+        cve_data[row[0]]["score"] = row[2]
+        cve_data[row[0]]["modified"] = row[3]
+        cve_data[row[0]]["vector"] = row[4]
+    conn.close()
+
+    return cve_data
+
+def cve_write_data(d, patched, unpatched, cve_data):
+    """
+    Write CVE information in WORKDIR; and to CVE_CHECK_DIR, and
+    CVE manifest if enabled.
+    """
+
+    from bb.utils import mkdirhier
+
+    cve_file = d.getVar("CVE_CHECK_LOCAL_FILE", True)
+    nvd_link = "https://web.nvd.nist.gov/view/vuln/detail?vulnId="
+    write_string = ""
+    mkdirhier(d.getVar("CVE_CHECK_LOCAL_DIR", True))
+
+    for cve in sorted(cve_data):
+        write_string += "PACKAGE NAME: %s\n" % d.getVar("PN", True)
+        write_string += "PACKAGE VERSION: %s\n" % d.getVar("PV", True)
+        write_string += "CVE: %s\n" % cve
+        if cve in patched:
+            write_string += "CVE STATUS: Patched\n"
+        else:
+            write_string += "CVE STATUS: Unpatched\n"
+            bb.warn("Found unpatched CVE, for more information check %s" % cve_file)
+        write_string += "CVE SUMMARY: %s\n" % cve_data[cve]["summary"]
+        write_string += "CVSS v2 BASE SCORE: %s\n" % cve_data[cve]["score"]
+        write_string += "VECTOR: %s\n" % cve_data[cve]["vector"]
+        write_string += "MORE INFORMATION: %s%s\n\n" % (nvd_link, cve)
+
+    with open(cve_file, "w") as f:
+        f.write(write_string)
+
+    if d.getVar("CVE_CHECK_COPY_FILES", True) == "1":
+        cve_dir = d.getVar("CVE_CHECK_DIR", True)
+        mkdirhier(cve_dir)
+        deploy_file = os.path.join(cve_dir, d.getVar("PN", True))
+        with open(deploy_file, "w") as f:
+            f.write(write_string)
+
+    if d.getVar("CVE_CHECK_CREATE_MANIFEST", True) == "1":
+        with open(d.getVar("CVE_CHECK_TMP_FILE", True), "a") as f:
+            f.write("%s" % write_string)
-- 
2.6.2

[PATCHv3 1/2] cve-check-tool: Add recipe

[mariano.lopez]

From: Mariano Lopez <mariano.lopez@linux.intel.com>

cve-check-tool is a program for public CVEs checking.
This tool also seek to determine if a vulnerability has
been addressed by a patch.

The recipe also includes the do_populate_cve_db task
that will populate the database used by the tool.

[YOCTO #7515]

Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
---
 .../cve-check-tool/cve-check-tool_5.6.4.bb         | 55 ++++++++++++++++++++++
 1 file changed, 55 insertions(+)
 create mode 100644 meta/recipes-devtools/cve-check-tool/cve-check-tool_5.6.4.bb

diff --git a/meta/recipes-devtools/cve-check-tool/cve-check-tool_5.6.4.bb b/meta/recipes-devtools/cve-check-tool/cve-check-tool_5.6.4.bb
new file mode 100644
index 0000000..0cf64e4
--- /dev/null
+++ b/meta/recipes-devtools/cve-check-tool/cve-check-tool_5.6.4.bb
@@ -0,0 +1,55 @@
+SUMMARY = "cve-check-tool"
+DESCRIPTION = "cve-check-tool is a tool for checking known (public) CVEs.\
+The tool will identify potentially vunlnerable software packages within Linux distributions through version matching."
+HOMEPAGE = "https://github.com/ikeydoherty/cve-check-tool"
+SECTION = "Development/Tools"
+LICENSE = "GPL-2.0"
+LIC_FILES_CHKSUM = "file://LICENSE;md5=e8c1458438ead3c34974bc0be3a03ed6"
+
+SRC_URI = "https://github.com/ikeydoherty/${BPN}/releases/download/v${PV}/${BP}.tar.xz"
+
+SRC_URI[md5sum] = "c5f4247140fc9be3bf41491d31a34155"
+SRC_URI[sha256sum] = "b8f283be718af8d31232ac1bfc10a0378fb958aaaa49af39168f8acf501e6a5b"
+
+DEPENDS = "libcheck glib-2.0 json-glib curl libxml2 sqlite3 openssl"
+
+inherit pkgconfig autotools
+
+EXTRA_OECONF = "--disable-static"
+
+python do_populate_cve_db () {
+    import subprocess
+    import time
+
+    if d.getVar("BB_NO_NETWORK", True) == "1":
+        bb.error("BB_NO_NETWORK is set; Can't update cve-check-tool database, "
+                  "CVEs won't be checked")
+        return
+
+    bb.utils.export_proxies(d)
+    # In case we don't inherit cve-check class, use default values defined in the class.
+    cve_dir = d.getVar("CVE_CHECK_DB_DIR", True) or d.expand("${DL_DIR}/CVE_CHECK")
+    cve_file = d.getVar("CVE_CHECK_TMP_FILE", True) or d.expand("${TMPDIR}/cve_check")
+    cve_cmd = "cve-check-update"
+    cmd = [cve_cmd, "-d", cve_dir]
+    bb.debug(1, "Updating cve-check-tool database located in %s" % cve_dir)
+    try:
+        output = subprocess.check_output(cmd, stderr=subprocess.STDOUT)
+        bb.debug(2, "Command '%s' returned:\n%s" % ("\n".join(cmd), output))
+        if bb.data.inherits_class('cve-check', d):
+            time_utc = time.gmtime(time.time())
+            time_format = "%Y-%m-%d %H:%M:%S"
+            with open(cve_file, "w") as f:
+                f.write("CVE database was updated on %s UTC\n\n"
+                        % time.strftime(time_format, time_utc))
+
+    except subprocess.CalledProcessError as e:
+        bb.warn("Error in executing cve-check-update: %s (output %s)" % (e, e.output))
+        if bb.data.inherits_class('cve-check', d):
+            bb.warn("Failed to update cve-check-tool database, CVEs won't be checked")
+}
+
+addtask populate_cve_db after do_populate_sysroot
+do_populate_cve_db[nostamp] = "1"
+
+BBCLASSEXTEND = "native"
-- 
2.6.6

Re: [PATCHv3 1/2] cve-check-tool: Add recipe

[akuster808]

Mariano,


On 07/11/2016 05:52 AM, mariano.lopez@linux.intel.com wrote:
> From: Mariano Lopez <mariano.lopez@linux.intel.com>
> 
> cve-check-tool is a program for public CVEs checking.
> This tool also seek to determine if a vulnerability has
> been addressed by a patch.

By tool do you mean the "cve-check-tool"? All the Nvd DB can tell you if
an CVE has been assigned, anything more than that is not guaranteed.

Look at https://web.nvd.nist.gov/view/vuln/detail?vulnId=CVE-2016-5320

> 
> The recipe also includes the do_populate_cve_db task
> that will populate the database used by the tool.

This DB is big. May want to add a note to that affect. Maybe a note
about how to share the DB across builds like with the AB.

time for me to play with this.

Thanks for driving this.
regards,
Armin

> 
> [YOCTO #7515]
> 
> Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
> ---
>  .../cve-check-tool/cve-check-tool_5.6.4.bb         | 55 ++++++++++++++++++++++
>  1 file changed, 55 insertions(+)
>  create mode 100644 meta/recipes-devtools/cve-check-tool/cve-check-tool_5.6.4.bb
> 
> diff --git a/meta/recipes-devtools/cve-check-tool/cve-check-tool_5.6.4.bb b/meta/recipes-devtools/cve-check-tool/cve-check-tool_5.6.4.bb
> new file mode 100644
> index 0000000..0cf64e4
> --- /dev/null
> +++ b/meta/recipes-devtools/cve-check-tool/cve-check-tool_5.6.4.bb
> @@ -0,0 +1,55 @@
> +SUMMARY = "cve-check-tool"
> +DESCRIPTION = "cve-check-tool is a tool for checking known (public) CVEs.\
> +The tool will identify potentially vunlnerable software packages within Linux distributions through version matching."
> +HOMEPAGE = "https://github.com/ikeydoherty/cve-check-tool"
> +SECTION = "Development/Tools"
> +LICENSE = "GPL-2.0"
> +LIC_FILES_CHKSUM = "file://LICENSE;md5=e8c1458438ead3c34974bc0be3a03ed6"
> +
> +SRC_URI = "https://github.com/ikeydoherty/${BPN}/releases/download/v${PV}/${BP}.tar.xz"
> +
> +SRC_URI[md5sum] = "c5f4247140fc9be3bf41491d31a34155"
> +SRC_URI[sha256sum] = "b8f283be718af8d31232ac1bfc10a0378fb958aaaa49af39168f8acf501e6a5b"
> +
> +DEPENDS = "libcheck glib-2.0 json-glib curl libxml2 sqlite3 openssl"
> +
> +inherit pkgconfig autotools
> +
> +EXTRA_OECONF = "--disable-static"
> +
> +python do_populate_cve_db () {
> +    import subprocess
> +    import time
> +
> +    if d.getVar("BB_NO_NETWORK", True) == "1":
> +        bb.error("BB_NO_NETWORK is set; Can't update cve-check-tool database, "
> +                  "CVEs won't be checked")
> +        return
> +
> +    bb.utils.export_proxies(d)
> +    # In case we don't inherit cve-check class, use default values defined in the class.
> +    cve_dir = d.getVar("CVE_CHECK_DB_DIR", True) or d.expand("${DL_DIR}/CVE_CHECK")
> +    cve_file = d.getVar("CVE_CHECK_TMP_FILE", True) or d.expand("${TMPDIR}/cve_check")
> +    cve_cmd = "cve-check-update"
> +    cmd = [cve_cmd, "-d", cve_dir]
> +    bb.debug(1, "Updating cve-check-tool database located in %s" % cve_dir)
> +    try:
> +        output = subprocess.check_output(cmd, stderr=subprocess.STDOUT)
> +        bb.debug(2, "Command '%s' returned:\n%s" % ("\n".join(cmd), output))
> +        if bb.data.inherits_class('cve-check', d):
> +            time_utc = time.gmtime(time.time())
> +            time_format = "%Y-%m-%d %H:%M:%S"
> +            with open(cve_file, "w") as f:
> +                f.write("CVE database was updated on %s UTC\n\n"
> +                        % time.strftime(time_format, time_utc))
> +
> +    except subprocess.CalledProcessError as e:
> +        bb.warn("Error in executing cve-check-update: %s (output %s)" % (e, e.output))
> +        if bb.data.inherits_class('cve-check', d):
> +            bb.warn("Failed to update cve-check-tool database, CVEs won't be checked")
> +}
> +
> +addtask populate_cve_db after do_populate_sysroot
> +do_populate_cve_db[nostamp] = "1"
> +
> +BBCLASSEXTEND = "native"
> 

Re: [PATCHv3 1/2] cve-check-tool: Add recipe

[Mariano Lopez]

On 07/12/2016 05:19 PM, akuster808 wrote:
> Mariano,
>
>
> On 07/11/2016 05:52 AM, mariano.lopez@linux.intel.com wrote:
>> From: Mariano Lopez <mariano.lopez@linux.intel.com>
>>
>> cve-check-tool is a program for public CVEs checking.
>> This tool also seek to determine if a vulnerability has
>> been addressed by a patch.
> By tool do you mean the "cve-check-tool"? All the Nvd DB can tell you if
> an CVE has been assigned, anything more than that is not guaranteed.
>
> Look at https://web.nvd.nist.gov/view/vuln/detail?vulnId=CVE-2016-5320

Sorry for the confusion, here I was referring to patches in OE that 
address the CVE, the class will look for the CVE tag for this.

>
>> The recipe also includes the do_populate_cve_db task
>> that will populate the database used by the tool.
> This DB is big. May want to add a note to that affect. Maybe a note
> about how to share the DB across builds like with the AB.

You are right, the DB is big and it will take some time to download. By 
default the tool will download the DB to DL_DIR, so if you have this dir 
shared, it will be downloaded just one time, and incremental updates later.

>
> time for me to play with this.
>
> Thanks for driving this.

Glad to be helping with this.

> regards,
> Armin
>
Mariano

[PATCHv3 2/2] cve-check.bbclass: Add class

[mariano.lopez]

From: Mariano Lopez <mariano.lopez@linux.intel.com>

This class adds a new task for all the recipes to use
cve-check-tool in order to look for public CVEs affecting
the packages generated.

It is possible to use this class when building an image,
building a recipe, or using the "world" or "universe" cases.

In order to use this class it must be inherited and it will
add the task automatically to every recipe.

[YOCTO #7515]

Co-authored by Ross Burton & Mariano Lopez

Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
---
 meta/classes/cve-check.bbclass | 267 +++++++++++++++++++++++++++++++++++++++++
 1 file changed, 267 insertions(+)
 create mode 100644 meta/classes/cve-check.bbclass

diff --git a/meta/classes/cve-check.bbclass b/meta/classes/cve-check.bbclass
new file mode 100644
index 0000000..8251ca7
--- /dev/null
+++ b/meta/classes/cve-check.bbclass
@@ -0,0 +1,267 @@
+# This class is used to check recipes against public CVEs.
+#
+# In order to use this class just inherit the class in the
+# local.conf file and it will add the cve_check task for
+# every recipe. The task can be used per recipe, per image,
+# or using the special cases "world" and "universe". The
+# cve_check task will print a warning for every unpatched
+# CVE found and generate a file in the recipe WORKDIR/cve
+# directory. If an image is build it will generate a report
+# in DEPLOY_DIR_IMAGE for all the packages used.
+#
+# Example:
+#   bitbake -c cve_check openssl
+#   bitbake core-image-sato
+#   bitbake -k -c cve_check universe
+#
+# DISCLAIMER
+#
+# This class/tool is meant to be used as support and not
+# the only method to check against CVEs. Running this tool
+# doesn't guarantee your packages are free of CVEs.
+
+CVE_CHECK_DB_DIR ?= "${DL_DIR}/CVE_CHECK"
+CVE_CHECK_DB_FILE ?= "${CVE_CHECK_DB_DIR}/nvd.db"
+
+CVE_CHECK_LOCAL_DIR ?= "${WORKDIR}/cve"
+CVE_CHECK_LOCAL_FILE ?= "${CVE_CHECK_LOCAL_DIR}/cve.log"
+CVE_CHECK_TMP_FILE ?= "${TMPDIR}/cve_check"
+
+CVE_CHECK_DIR ??= "${DEPLOY_DIR}/cve"
+CVE_CHECK_MANIFEST ?= "${DEPLOY_DIR_IMAGE}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.cve"
+CVE_CHECK_COPY_FILES ??= "1"
+CVE_CHECK_CREATE_MANIFEST ??= "1"
+
+# Whitelist for packages (PN)
+CVE_CHECK_PN_WHITELIST = "\
+    glibc-locale \
+"
+
+# Whitelist for CVE and version of package
+CVE_CHECK_CVE_WHITELIST = "{\
+    'CVE-2014-2524': ('6.3',), \
+}"
+
+python do_cve_check () {
+    """
+    Check recipe for patched and unpatched CVEs
+    """
+
+    if os.path.exists(d.getVar("CVE_CHECK_TMP_FILE", True)):
+        patched_cves = get_patches_cves(d)
+        patched, unpatched = check_cves(d, patched_cves)
+        if patched or unpatched:
+            cve_data = get_cve_info(d, patched + unpatched)
+            cve_write_data(d, patched, unpatched, cve_data)
+    else:
+        bb.note("Failed to update CVE database, skipping CVE check")
+}
+
+addtask cve_check after do_unpack before do_build
+do_cve_check[depends] = "cve-check-tool-native:do_populate_cve_db"
+do_cve_check[nostamp] = "1"
+
+python cve_check_cleanup () {
+    """
+    Delete the file used to gather all the CVE information.
+    """
+
+    bb.utils.remove(e.data.getVar("CVE_CHECK_TMP_FILE", True))
+}
+
+addhandler cve_check_cleanup
+cve_check_cleanup[eventmask] = "bb.cooker.CookerExit"
+
+python cve_check_write_rootfs_manifest () {
+    """
+    Create CVE manifest when building an image
+    """
+
+    import shutil
+
+    if os.path.exists(d.getVar("CVE_CHECK_TMP_FILE", True)):
+        bb.note("Writing rootfs CVE manifest")
+        deploy_dir = d.getVar("DEPLOY_DIR_IMAGE", True)
+        link_name = d.getVar("IMAGE_LINK_NAME", True)
+        manifest_name = d.getVar("CVE_CHECK_MANIFEST", True)
+        cve_tmp_file = d.getVar("CVE_CHECK_TMP_FILE", True)
+
+        shutil.copyfile(cve_tmp_file, manifest_name)
+
+        if manifest_name and os.path.exists(manifest_name):
+            manifest_link = os.path.join(deploy_dir, "%s.cve" % link_name)
+            # If we already have another manifest, update symlinks
+            if os.path.exists(os.path.realpath(manifest_link)):
+                if d.getVar('RM_OLD_IMAGE', True) == "1":
+                    os.remove(os.path.realpath(manifest_link))
+                os.remove(manifest_link)
+            os.symlink(os.path.basename(manifest_name), manifest_link)
+            bb.plain("Image CVE report stored in: %s" % manifest_name)
+}
+
+ROOTFS_POSTPROCESS_COMMAND_prepend = "${@'cve_check_write_rootfs_manifest; ' if d.getVar('CVE_CHECK_CREATE_MANIFEST', True) == '1' else ''}"
+
+def get_patches_cves(d):
+    """
+    Get patches that solve CVEs using the "CVE: " tag.
+    """
+
+    import re
+
+    pn = d.getVar("PN", True)
+    cve_match = re.compile("CVE:( CVE\-\d{4}\-\d+)+")
+    patched_cves = set()
+    bb.debug(2, "Looking for patches that solves CVEs for %s" % pn)
+    for url in src_patches(d):
+        patch_file = bb.fetch.decodeurl(url)[2]
+        with open(patch_file, "r", encoding="utf-8") as f:
+            try:
+                patch_text = f.read()
+            except UnicodeDecodeError:
+                bb.debug(1, "Failed to read patch %s using UTF-8 encoding"
+                        " trying with iso8859-1" %  patch_file)
+                f.close()
+                with open(patch_file, "r", encoding="iso8859-1") as f:
+                    patch_text = f.read()
+
+        # Search for the "CVE: " line
+        match = cve_match.search(patch_text)
+        if match:
+            # Get only the CVEs without the "CVE: " tag
+            cves = patch_text[match.start()+5:match.end()]
+            for cve in cves.split():
+                bb.debug(2, "Patch %s solves %s" % (patch_file, cve))
+                patched_cves.add(cve)
+        else:
+            bb.debug(2, "Patch %s doesn't solve CVEs" % patch_file)
+
+    return patched_cves
+
+def check_cves(d, patched_cves):
+    """
+    Run cve-check-tool looking for patched and unpatched CVEs.
+    """
+
+    import ast, csv, tempfile, subprocess, io
+
+    cves_patched = []
+    cves_unpatched = []
+    bpn = d.getVar("BPN", True)
+    pv = d.getVar("PV", True).split("git+")[0]
+    cves = " ".join(patched_cves)
+    cve_db_dir = d.getVar("CVE_CHECK_DB_DIR", True)
+    cve_whitelist = ast.literal_eval(d.getVar("CVE_CHECK_CVE_WHITELIST", True))
+    cve_cmd = "cve-check-tool"
+    cmd = [cve_cmd, "--no-html", "--csv", "--not-affected", "-t", "faux", "-d", cve_db_dir]
+
+    # If the recipe has been whitlisted we return empty lists
+    if d.getVar("PN", True) in d.getVar("CVE_CHECK_PN_WHITELIST", True).split():
+        bb.note("Recipe has been whitelisted, skipping check")
+        return ([], [])
+
+    # It is needed to export the proxies to download the database using HTTP
+    bb.utils.export_proxies(d)
+
+    try:
+        # Write the faux CSV file to be used with cve-check-tool
+        fd, faux = tempfile.mkstemp(prefix="cve-faux-")
+        with os.fdopen(fd, "w") as f:
+            f.write("%s,%s,%s," % (bpn, pv, cves))
+        cmd.append(faux)
+
+        output = subprocess.check_output(cmd, stderr=subprocess.STDOUT).decode("utf-8")
+        bb.debug(2, "Output of command %s:\n%s" % ("\n".join(cmd), output))
+    except subprocess.CalledProcessError as e:
+        bb.warn("Couldn't check for CVEs: %s (output %s)" % (e, e.output))
+    finally:
+        os.remove(faux)
+
+    for row in csv.reader(io.StringIO(output)):
+        # Third row has the unpatched CVEs
+        if row[2]:
+            for cve in row[2].split():
+                # Skip if the CVE has been whitlisted for the current version
+                if pv in cve_whitelist.get(cve,[]):
+                    bb.note("%s-%s has been whitelisted for %s" % (bpn, pv, cve))
+                else:
+                    cves_unpatched.append(cve)
+                    bb.debug(2, "%s-%s is not patched for %s" % (bpn, pv, cve))
+        # Fourth row has patched CVEs
+        if row[3]:
+            for cve in row[3].split():
+                cves_patched.append(cve)
+                bb.debug(2, "%s-%s is patched for %s" % (bpn, pv, cve))
+
+    return (cves_patched, cves_unpatched)
+
+def get_cve_info(d, cves):
+    """
+    Get CVE information from the database used by cve-check-tool.
+
+    Unfortunately the only way to get CVE info is set the output to
+    html (hard to parse) or query directly the database.
+    """
+
+    try:
+        import sqlite3
+    except ImportError:
+        from pysqlite2 import dbapi2 as sqlite3
+
+    cve_data = {}
+    db_file = d.getVar("CVE_CHECK_DB_FILE", True)
+    placeholder = ",".join("?" * len(cves))
+    query = "SELECT * FROM NVD WHERE id IN (%s)" % placeholder
+    conn = sqlite3.connect(db_file)
+    cur = conn.cursor()
+    for row in cur.execute(query, tuple(cves)):
+        cve_data[row[0]] = {}
+        cve_data[row[0]]["summary"] = row[1]
+        cve_data[row[0]]["score"] = row[2]
+        cve_data[row[0]]["modified"] = row[3]
+        cve_data[row[0]]["vector"] = row[4]
+    conn.close()
+
+    return cve_data
+
+def cve_write_data(d, patched, unpatched, cve_data):
+    """
+    Write CVE information in WORKDIR; and to CVE_CHECK_DIR, and
+    CVE manifest if enabled.
+    """
+
+    cve_file = d.getVar("CVE_CHECK_LOCAL_FILE", True)
+    nvd_link = "https://web.nvd.nist.gov/view/vuln/detail?vulnId="
+    write_string = ""
+    first_alert = True
+    bb.utils.mkdirhier(d.getVar("CVE_CHECK_LOCAL_DIR", True))
+
+    for cve in sorted(cve_data):
+        write_string += "PACKAGE NAME: %s\n" % d.getVar("PN", True)
+        write_string += "PACKAGE VERSION: %s\n" % d.getVar("PV", True)
+        write_string += "CVE: %s\n" % cve
+        if cve in patched:
+            write_string += "CVE STATUS: Patched\n"
+        else:
+            write_string += "CVE STATUS: Unpatched\n"
+            if first_alert:
+                bb.warn("Found unpatched CVE, for more information check %s" % cve_file)
+                first_alert = False
+        write_string += "CVE SUMMARY: %s\n" % cve_data[cve]["summary"]
+        write_string += "CVSS v2 BASE SCORE: %s\n" % cve_data[cve]["score"]
+        write_string += "VECTOR: %s\n" % cve_data[cve]["vector"]
+        write_string += "MORE INFORMATION: %s%s\n\n" % (nvd_link, cve)
+
+    with open(cve_file, "w") as f:
+        bb.note("Writing file %s with CVE information" % cve_file)
+        f.write(write_string)
+
+    if d.getVar("CVE_CHECK_COPY_FILES", True) == "1":
+        cve_dir = d.getVar("CVE_CHECK_DIR", True)
+        bb.utils.mkdirhier(cve_dir)
+        deploy_file = os.path.join(cve_dir, d.getVar("PN", True))
+        with open(deploy_file, "w") as f:
+            f.write(write_string)
+
+    if d.getVar("CVE_CHECK_CREATE_MANIFEST", True) == "1":
+        with open(d.getVar("CVE_CHECK_TMP_FILE", True), "a") as f:
+            f.write("%s" % write_string)
-- 
2.6.6