[Buildroot] [PATCH v4 1/6] support/scripts/pkg-stats: ignore more makefiles that aren't packages

[Buildroot] [PATCH v4 1/6] support/scripts/pkg-stats: ignore more makefiles that aren't packages

[Daniel Lang]

Some packages are grouped and have a general makefile that defines
reusable variables. These makefiles have no relevant information for
pkg-stats and should be excluded.

Signed-off-by: Daniel Lang <dalang@gmx.at>
---
v3 -> v4:
- Added more packages to ignore list, not just llvm-project
---
 support/scripts/pkg-stats | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)

diff --git a/support/scripts/pkg-stats b/support/scripts/pkg-stats
index 28f5a0789c..eca638f396 100755
--- a/support/scripts/pkg-stats
+++ b/support/scripts/pkg-stats
@@ -338,16 +338,21 @@ def get_pkglist(npackages, package_list):
     package_list: limit to those packages in this list
     """
     WALK_USEFUL_SUBDIRS = ["boot", "linux", "package", "toolchain"]
-    WALK_EXCLUDES = ["boot/common.mk",
+    WALK_EXCLUDES = ["boot/barebox/barebox.mk",
+                     "boot/common.mk",
                      "linux/linux-ext-.*.mk",
+                     "package/fftw/fftw.mk",
                      "package/freescale-imx/freescale-imx.mk",
                      "package/gcc/gcc.mk",
                      "package/gstreamer/gstreamer.mk",
                      "package/gstreamer1/gstreamer1.mk",
                      "package/gtk2-themes/gtk2-themes.mk",
+                     "package/kf5/kf5.mk",
+                     "package/llvm-project/llvm-project.mk",
                      "package/matchbox/matchbox.mk",
                      "package/opengl/opengl.mk",
                      "package/qt5/qt5.mk",
+                     "package/qt6/qt6.mk",
                      "package/x11r7/x11r7.mk",
                      "package/doc-asciidoc.mk",
                      "package/pkg-.*.mk",
-- 
2.42.0

_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

[Buildroot] [PATCH v4 2/6] support/scripts/pkg-stats: check all files for warnings

[Daniel Lang]

Instead of only checking .mk and Config.in{,.host}, check
all files in a package directory.
.checkpackageignore isn't considered here, therefore the shown number
includes ignored warnings as well.
Add another css class to signal some warning, compared to a lot (>5),
similar to patches.

Signed-off-by: Daniel Lang <dalang@gmx.at>
---
v3 -> v4:
- use yellowish bachground for "some" warnings
v2 -> v3:
- new patch

Signed-off-by: Daniel Lang <dalang@gmx.at>
---
 support/scripts/pkg-stats | 7 ++++---
 1 file changed, 4 insertions(+), 3 deletions(-)

diff --git a/support/scripts/pkg-stats b/support/scripts/pkg-stats
index eca638f396..9349a0df57 100755
--- a/support/scripts/pkg-stats
+++ b/support/scripts/pkg-stats
@@ -272,8 +272,7 @@ class Package:
         self.status['pkg-check'] = ("error", "Missing")
         for root, dirs, files in os.walk(pkgdir):
             for f in files:
-                if f.endswith(".mk") or f.endswith(".hash") or f == "Config.in" or f == "Config.in.host":
-                    cmd.append(os.path.join(root, f))
+                cmd.append(os.path.join(root, f))
         o = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE).communicate()[1]
         lines = o.splitlines()
         for line in lines:
@@ -876,7 +875,7 @@ function expandField(fieldId){
  .wrong, .lotsofpatches, .invalid_url, .version-needs-update, .cpe-nok, .cve-nok {
    background: #ff9a69;
  }
- .somepatches, .missing_url, .version-unknown, .cpe-unknown, .cve-unknown {
+ .somepatches, .somewarnings, .missing_url, .version-unknown, .cpe-unknown, .cve-unknown {
    background: #ffd870;
  }
  .cve_ignored, .version-error {
@@ -1033,6 +1032,8 @@ def dump_html_pkg(f, pkg):
     div_class.append(f'_{pkg_css_class}')
     if pkg.warnings == 0:
         div_class.append("correct")
+    elif pkg.warnings < 5:
+        div_class.append("somewarnings")
     else:
         div_class.append("wrong")
     f.write(f'  <div id="{data_field_id}" class="{" ".join(div_class)}">{pkg.warnings}</div>\n')
-- 
2.42.0

_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH v4 2/6] support/scripts/pkg-stats: check all files for warnings

[Thomas Petazzoni via buildroot]

On Fri,  1 Sep 2023 21:27:10 +0200
Daniel Lang <dalang@gmx.at> wrote:

> Instead of only checking .mk and Config.in{,.host}, check
> all files in a package directory.
> .checkpackageignore isn't considered here, therefore the shown number
> includes ignored warnings as well.
> Add another css class to signal some warning, compared to a lot (>5),
> similar to patches.
> 
> Signed-off-by: Daniel Lang <dalang@gmx.at>
> ---
> v3 -> v4:
> - use yellowish bachground for "some" warnings
> v2 -> v3:
> - new patch

Applied to master, thanks.

Thomas
-- 
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

[Buildroot] [PATCH v4 3/6] support/scripts/nvd_api_v2.py: new helper class

[Daniel Lang]

The current NVD data feeds used for CVE and CPE checking will be retired
by 2023-12-05 [0]. Both have to be switched to the new v2 API. Since
fetching data from both sources workes the same, a common base class is
used to handle the API interaction.
With the new API JSON pages are downloaded, meaning that we have to come
up with a storage solution ourselves. Therefore nvd_api_v2.py manages a
generic set of methods to initialize and update a sqlite database.

[0]: https://nvd.nist.gov/General/News/change-timeline

Signed-off-by: Daniel Lang <dalang@gmx.at>
---
v3 -> v4:
- use f-strings
- add meta table instead of tmp file
- drop error handling

Signed-off-by: Daniel Lang <dalang@gmx.at>
---
 DEVELOPERS                    |   1 +
 support/scripts/nvd_api_v2.py | 137 ++++++++++++++++++++++++++++++++++
 2 files changed, 138 insertions(+)
 create mode 100755 support/scripts/nvd_api_v2.py

diff --git a/DEVELOPERS b/DEVELOPERS
index 9b500f3701..046a022fc9 100644
--- a/DEVELOPERS
+++ b/DEVELOPERS
@@ -668,6 +668,7 @@ F:	package/paho-mqtt-cpp/
 F:	package/pangomm/
 F:	package/pangomm2_46/
 F:	package/sam-ba/
+F:	support/scripts/nvd_api_v2.py
 
 N:	Damien Lanson <damien@kal-host.com>
 F:	package/libvdpau/
diff --git a/support/scripts/nvd_api_v2.py b/support/scripts/nvd_api_v2.py
new file mode 100755
index 0000000000..f50a362f88
--- /dev/null
+++ b/support/scripts/nvd_api_v2.py
@@ -0,0 +1,137 @@
+#!/usr/bin/env python3
+
+from datetime import datetime, timedelta, timezone
+from pathlib import Path
+import requests
+import sqlite3
+import time
+
+NVD_API_VERSION = '2.0'
+NVD_API_BASE_URL = 'https://services.nvd.nist.gov/rest/json'
+# NVD rate limiting allows 5 requests per 30 seconds -> one every 6 seconds.
+NVD_SLEEP_TIME_SEC = 30 / 5
+# Number of minutes that are subtracted from the update timestap given by NVD.
+NVD_TIMESTAMP_SUBTRACT_MINUTES = 1
+# Only update the database if the given number of seconds elapsed.
+NVD_CHECK_TIMEOUT_SECONDS = 120 * 60
+
+
+class NVD_API:
+    """A helper class that fetches data from a NVD API and helps manage a sqlite database.
+
+    This class defines two functions that need to be implemented in a derived class.
+    init_db: Called to create database tables.
+    save_to_db: Called to actually analyse and save the downloaded data."""
+
+    def __init__(self, nvd_path, service, database_file_prefix):
+        """Initialize a new NVD API endpoint.
+
+        nvd_path             - The local path of the database, typically DL_DIR/buildroot-nvd.
+        service              - The service that gets appended to the API URL, should be 'CVEs' or 'CPEs'.
+        database_file_prefix - The prefix of the sqlite database, typically 'nvdcve' or 'nvdcpe'.
+        """
+        self.nvd_path = nvd_path
+        self.service = service
+        self.url = f'{NVD_API_BASE_URL}/{service.lower()}/{NVD_API_VERSION}'
+        self.db_file = Path(nvd_path, f'{database_file_prefix}-{NVD_API_VERSION}.sqlite')
+        self.db_connection = None
+
+    def init_db_meta(self) -> None:
+        """Create the internal meta table that stores the last update date."""
+        self.db_connection.execute('CREATE TABLE IF NOT EXISTS meta ( \
+            id INTEGER UNIQUE, \
+            last_update TIMESTAMP)').close()
+
+    def init_db(self) -> None:
+        """Used to make sure that database tables exist.
+
+        Database connection is active at this point (self.db_connection).
+        Needs to be implemented by derived classes.
+        """
+        pass
+
+    def save_to_db(self, content) -> bool:
+        """Used to save the data given by a single API request to the database.
+
+        content is the json downloaded from NVD.
+        Needs to be implemented by derived classes.
+        """
+        pass
+
+    def download(self, last_update) -> None:
+        """Download all entries from NVD since last_update (if not None).
+
+        For each downloaded page save_to_db is called.
+        """
+        args = {}
+        start_index = 0
+        total_results = 0
+        results_per_page = 0
+
+        print(f'Downloading new {self.service}')
+
+        if (last_update is not None):
+            args['lastModStartDate'] = last_update.isoformat()
+            args['lastModEndDate'] = datetime.now(tz=timezone.utc).isoformat()
+
+        while True:
+            args['startIndex'] = start_index
+
+            page = requests.get(self.url, params=args)
+            page.raise_for_status()
+            content = page.json()
+
+            if content is None:
+                # Nothing was downloaded
+                return False
+
+            results_per_page = content['resultsPerPage']
+            total_results = content['totalResults']
+            start_index = content['startIndex']
+            timestamp = content['timestamp']
+
+            start_index += results_per_page
+
+            # Call the save method of the derived class
+            if not self.save_to_db(content):
+                return False
+
+            print(f'[{start_index:0{len(str(total_results))}}/{total_results}]')
+
+            if start_index >= total_results:
+                # Update the meta table with the timestamp given by NVD but
+                # subtract a minute to make sure nothing is missed the next time.
+                timestamp = datetime.fromisoformat(timestamp) - timedelta(minutes=NVD_TIMESTAMP_SUBTRACT_MINUTES)
+                self.db_connection.execute('INSERT OR REPLACE INTO meta VALUES (0, ?)', (timestamp,)).close()
+                self.db_connection.commit()
+                return True
+
+            self.db_connection.commit()
+
+            # Otherwise rate limiting will be hit.
+            time.sleep(NVD_SLEEP_TIME_SEC)
+
+    def check_for_updates(self) -> None:
+        """Check if new data needs to be fetched from NVD.
+
+        If the last update happened more than 24 hours ago,
+        a new download is triggered.
+        """
+        self.db_connection = sqlite3.connect(self.db_file)
+        self.init_db_meta()
+        self.init_db()
+
+        last_update = None
+        result = self.db_connection.execute('SELECT last_update FROM meta WHERE id = 0').fetchone()
+        if result:
+            # NVD uses UTC without specifying the timezone on their timestamp
+            now = datetime.now(tz=timezone.utc)
+            last_update = datetime.fromisoformat(result[0])
+            delta = now - last_update.replace(tzinfo=timezone.utc)
+            if delta.total_seconds() < NVD_CHECK_TIMEOUT_SECONDS:
+                # Don't run the download if the last update
+                # is less than two hours ago.
+                return
+
+        if not self.download(last_update):
+            print('Update failed!')
-- 
2.42.0

_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH v4 3/6] support/scripts/nvd_api_v2.py: new helper class

[Thomas Petazzoni via buildroot]

Hello Daniel,

On Fri,  1 Sep 2023 21:27:11 +0200
Daniel Lang <dalang@gmx.at> wrote:

> The current NVD data feeds used for CVE and CPE checking will be retired
> by 2023-12-05 [0]. Both have to be switched to the new v2 API. Since
> fetching data from both sources workes the same, a common base class is
> used to handle the API interaction.
> With the new API JSON pages are downloaded, meaning that we have to come
> up with a storage solution ourselves. Therefore nvd_api_v2.py manages a
> generic set of methods to initialize and update a sqlite database.
> 
> [0]: https://nvd.nist.gov/General/News/change-timeline
> 
> Signed-off-by: Daniel Lang <dalang@gmx.at>

Thanks for working into this. I wanted to merge this, but taking a
fresh look at this, I don't quite understand the SW design choices
you've made when it comes to splitting between the NVD_API class and
the CVE_API class.

The NVD_API class implements
 - An actual init_db_meta() method
 - A dummy init_db() method
 - A dummy save_to_db() method
 - An actual download() method
 - An actual check_for_updates() method

Then the CVE_API class inherits from NVD_API, which provides the actual
implementation of init_db(), save_to_db() among others.

What is the reasoning behind this? One could think that the NVD_API
class was only related to calling the NVD HTTP API, but it does have
knowledge of the local sqlite database.

So I'm really not clear on this design. Could you provide some details?

Now I see that the CPE_API class also inherits from NVD_API, but I'm
still confused.

Thomas
-- 
Thomas Petazzoni, co-owner and CEO, Bootlin
Embedded Linux and Kernel engineering and training
https://bootlin.com
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH v4 3/6] support/scripts/nvd_api_v2.py: new helper class

[Daniel Lang via buildroot]

Hello Thomas,

sorry for the long silence.

On 08.11.23 22:55, Thomas Petazzoni wrote:
> Hello Daniel,
>
> On Fri,  1 Sep 2023 21:27:11 +0200
> Daniel Lang <dalang@gmx.at> wrote:
>
>> The current NVD data feeds used for CVE and CPE checking will be retired
>> by 2023-12-05 [0]. Both have to be switched to the new v2 API. Since
>> fetching data from both sources workes the same, a common base class is
>> used to handle the API interaction.
>> With the new API JSON pages are downloaded, meaning that we have to come
>> up with a storage solution ourselves. Therefore nvd_api_v2.py manages a
>> generic set of methods to initialize and update a sqlite database.
>>
>> [0]: https://nvd.nist.gov/General/News/change-timeline
>>
>> Signed-off-by: Daniel Lang <dalang@gmx.at>
>
> Thanks for working into this. I wanted to merge this, but taking a
> fresh look at this, I don't quite understand the SW design choices
> you've made when it comes to splitting between the NVD_API class and
> the CVE_API class.

My idea was to move code that is needed for CVEs and CPEs into a common
file since with the new API both endpoints are quit similar.

Therefore NVD_API contains code for querying the API, handling pagination,
querying meta information from the database. Additionally downloaded information
is passed to CVE_API or CPE_API via save_to_db().

Not sure, if that makes it clearer.

>
> The NVD_API class implements
>  - An actual init_db_meta() method
>  - A dummy init_db() method
>  - A dummy save_to_db() method
>  - An actual download() method
>  - An actual check_for_updates() method
>
> Then the CVE_API class inherits from NVD_API, which provides the actual
> implementation of init_db(), save_to_db() among others.
>
> What is the reasoning behind this? One could think that the NVD_API
> class was only related to calling the NVD HTTP API, but it does have
> knowledge of the local sqlite database.

Yes, it has knowledge of the local database to load and save meta information,
which is currently only "when was the last run".

>
> So I'm really not clear on this design. Could you provide some details?
>
> Now I see that the CPE_API class also inherits from NVD_API, but I'm
> still confused.

While implementing this I was going back and forth on whether to use a
database.
This version of the patch does, but I have since moved the code to GitLab
to try out GitLab Package Repositories for hosting the archives for each year.
During this move I switched back to using JSON files.
This makes it possible to query the data once for all buildroot users and filter
the data for the needed fields.
This means, that we end up with a couple of MB per year instead of a 400MB sqlite
database.
This approach could either be part of sources.buildroot.org or via GitLab.


>
> Thomas

Regards,
Daniel

_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

[Buildroot] [PATCH v4 4/6] support/scripts/cve.py: switch to NVD API v2

[Daniel Lang]

The currently used feed will be retired in December 2023 [0].
The only alternative is the new v2 API [1]. The API allows downloading
sets of CVEs (2k at a time) that were modified after a given
UTC timestamp. Rate limiting of 5 requests in a sliding 30 seconds
window is used [2].
nvd_api_v2.py is used to store the data in a local sqlite database.
Only a subset of fields get stored to keep the db smaller
(still ~400MB currently).

[0]: https://nvd.nist.gov/General/News/change-timeline
[1]: https://nvd.nist.gov/developers/vulnerabilities
[2]: https://nvd.nist.gov/developers/start-here

Signed-off-by: Daniel Lang <dalang@gmx.at>
---
v3 -> v4:
- progress output moved to nvd_api_v2
- self.connection was renamed to self.db_connection
v1 -> v2:
- switch to sqlite database for a more future proof storage
- CPE_ID class has been moved to a laster patch, therefore not used here

Signed-off-by: Daniel Lang <dalang@gmx.at>
---
 DEVELOPERS                |   1 +
 support/scripts/cve.py    | 413 ++++++++++++++++++++------------------
 support/scripts/pkg-stats |  31 ++-
 3 files changed, 236 insertions(+), 209 deletions(-)

diff --git a/DEVELOPERS b/DEVELOPERS
index 046a022fc9..87f5e816f2 100644
--- a/DEVELOPERS
+++ b/DEVELOPERS
@@ -668,6 +668,7 @@ F:	package/paho-mqtt-cpp/
 F:	package/pangomm/
 F:	package/pangomm2_46/
 F:	package/sam-ba/
+F:	support/scripts/cve.py
 F:	support/scripts/nvd_api_v2.py
 
 N:	Damien Lanson <damien@kal-host.com>
diff --git a/support/scripts/cve.py b/support/scripts/cve.py
index 7cd6fce4d8..b09ac64c81 100755
--- a/support/scripts/cve.py
+++ b/support/scripts/cve.py
@@ -17,40 +17,9 @@
 # along with this program; if not, write to the Free Software
 # Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
 
-import datetime
-import os
-import requests  # URL checking
 import distutils.version
-import time
-import gzip
-import sys
 import operator
-
-try:
-    import ijson
-    # backend is a module in < 2.5, a string in >= 2.5
-    if 'python' in getattr(ijson.backend, '__name__', ijson.backend):
-        try:
-            import ijson.backends.yajl2_cffi as ijson
-        except ImportError:
-            sys.stderr.write('Warning: Using slow ijson python backend\n')
-except ImportError:
-    sys.stderr.write("You need ijson to parse NVD for CVE check\n")
-    exit(1)
-
-sys.path.append('utils/')
-
-NVD_START_YEAR = 2002
-NVD_JSON_VERSION = "1.1"
-NVD_BASE_URL = "https://nvd.nist.gov/feeds/json/cve/" + NVD_JSON_VERSION
-
-ops = {
-    '>=': operator.ge,
-    '>': operator.gt,
-    '<=': operator.le,
-    '<': operator.lt,
-    '=': operator.eq
-}
+from nvd_api_v2 import NVD_API
 
 
 # Check if two CPE IDs match each other
@@ -77,141 +46,32 @@ class CVE:
     CVE_DOESNT_AFFECT = 2
     CVE_UNKNOWN = 3
 
-    def __init__(self, nvd_cve):
-        """Initialize a CVE from its NVD JSON representation"""
-        self.nvd_cve = nvd_cve
-
-    @staticmethod
-    def download_nvd_year(nvd_path, year):
-        metaf = "nvdcve-%s-%s.meta" % (NVD_JSON_VERSION, year)
-        path_metaf = os.path.join(nvd_path, metaf)
-        jsonf_gz = "nvdcve-%s-%s.json.gz" % (NVD_JSON_VERSION, year)
-        path_jsonf_gz = os.path.join(nvd_path, jsonf_gz)
-
-        # If the database file is less than a day old, we assume the NVD data
-        # locally available is recent enough.
-        if os.path.exists(path_jsonf_gz) and os.stat(path_jsonf_gz).st_mtime >= time.time() - 86400:
-            return path_jsonf_gz
-
-        # If not, we download the meta file
-        url = "%s/%s" % (NVD_BASE_URL, metaf)
-        print("Getting %s" % url)
-        page_meta = requests.get(url)
-        page_meta.raise_for_status()
-
-        # If the meta file already existed, we compare the existing
-        # one with the data newly downloaded. If they are different,
-        # we need to re-download the database.
-        # If the database does not exist locally, we need to redownload it in
-        # any case.
-        if os.path.exists(path_metaf) and os.path.exists(path_jsonf_gz):
-            meta_known = open(path_metaf, "r").read()
-            if page_meta.text == meta_known:
-                return path_jsonf_gz
-
-        # Grab the compressed JSON NVD, and write files to disk
-        url = "%s/%s" % (NVD_BASE_URL, jsonf_gz)
-        print("Getting %s" % url)
-        page_json = requests.get(url)
-        page_json.raise_for_status()
-        open(path_jsonf_gz, "wb").write(page_json.content)
-        open(path_metaf, "w").write(page_meta.text)
-        return path_jsonf_gz
-
-    @classmethod
-    def read_nvd_dir(cls, nvd_dir):
-        """
-        Iterate over all the CVEs contained in NIST Vulnerability Database
-        feeds since NVD_START_YEAR. If the files are missing or outdated in
-        nvd_dir, a fresh copy will be downloaded, and kept in .json.gz
-        """
-        for year in range(NVD_START_YEAR, datetime.datetime.now().year + 1):
-            filename = CVE.download_nvd_year(nvd_dir, year)
-            try:
-                content = ijson.items(gzip.GzipFile(filename), 'CVE_Items.item')
-            except:  # noqa: E722
-                print("ERROR: cannot read %s. Please remove the file then rerun this script" % filename)
-                raise
-            for cve in content:
-                yield cls(cve)
-
-    def each_product(self):
-        """Iterate over each product section of this cve"""
-        for vendor in self.nvd_cve['cve']['affects']['vendor']['vendor_data']:
-            for product in vendor['product']['product_data']:
-                yield product
-
-    def parse_node(self, node):
-        """
-        Parse the node inside the configurations section to extract the
-        cpe information usefull to know if a product is affected by
-        the CVE. Actually only the product name and the version
-        descriptor are needed, but we also provide the vendor name.
-        """
+    ops = {
+        '>=': operator.ge,
+        '>': operator.gt,
+        '<=': operator.le,
+        '<': operator.lt,
+        '=': operator.eq
+    }
 
-        # The node containing the cpe entries matching the CVE can also
-        # contain sub-nodes, so we need to manage it.
-        for child in node.get('children', ()):
-            for parsed_node in self.parse_node(child):
-                yield parsed_node
-
-        for cpe in node.get('cpe_match', ()):
-            if not cpe['vulnerable']:
-                return
-            product = cpe_product(cpe['cpe23Uri'])
-            version = cpe_version(cpe['cpe23Uri'])
-            # ignore when product is '-', which means N/A
-            if product == '-':
-                return
-            op_start = ''
-            op_end = ''
-            v_start = ''
-            v_end = ''
-
-            if version != '*' and version != '-':
-                # Version is defined, this is a '=' match
-                op_start = '='
-                v_start = version
-            else:
-                # Parse start version, end version and operators
-                if 'versionStartIncluding' in cpe:
-                    op_start = '>='
-                    v_start = cpe['versionStartIncluding']
-
-                if 'versionStartExcluding' in cpe:
-                    op_start = '>'
-                    v_start = cpe['versionStartExcluding']
-
-                if 'versionEndIncluding' in cpe:
-                    op_end = '<='
-                    v_end = cpe['versionEndIncluding']
-
-                if 'versionEndExcluding' in cpe:
-                    op_end = '<'
-                    v_end = cpe['versionEndExcluding']
-
-            yield {
-                'id': cpe['cpe23Uri'],
-                'v_start': v_start,
-                'op_start': op_start,
-                'v_end': v_end,
-                'op_end': op_end
-            }
-
-    def each_cpe(self):
-        for node in self.nvd_cve['configurations']['nodes']:
-            for cpe in self.parse_node(node):
-                yield cpe
+    def __init__(self, nvd_cve):
+        """Initialize a CVE from the database tuple representation"""
+        self.id = nvd_cve[0]
+        self.match_criteria = nvd_cve[2]
+        self.v_start = nvd_cve[3]
+        self.v_end = nvd_cve[4]
+        self.op_start = nvd_cve[5]
+        self.op_end = nvd_cve[6]
 
     @property
     def identifier(self):
         """The CVE unique identifier"""
-        return self.nvd_cve['cve']['CVE_data_meta']['ID']
+        return self.id
 
     @property
-    def affected_products(self):
-        """The set of CPE products referred by this CVE definition"""
-        return set(cpe_product(p['id']) for p in self.each_cpe())
+    def affected_product(self):
+        """Name of the affected product"""
+        return cpe_product(self.match_criteria)
 
     def affects(self, name, version, cve_ignore_list, cpeid=None):
         """
@@ -235,39 +95,206 @@ class CVE:
         else:
             pkg_version = distutils.version.LooseVersion(cpe_version(cpeid))
 
-        for cpe in self.each_cpe():
-            if not cpe_matches(cpe['id'], cpeid):
-                continue
-            if not cpe['v_start'] and not cpe['v_end']:
-                return self.CVE_AFFECTS
-            if not pkg_version:
+        if not cpe_matches(self.match_criteria, cpeid):
+            return self.CVE_DOESNT_AFFECT
+        if not self.v_start and not self.v_end:
+            return self.CVE_AFFECTS
+        if not pkg_version:
+            return self.CVE_DOESNT_AFFECT
+
+        if self.v_start:
+            try:
+                cve_affected_version = distutils.version.LooseVersion(self.v_start)
+                inrange = self.ops.get(self.op_start)(pkg_version, cve_affected_version)
+            except TypeError:
+                return self.CVE_UNKNOWN
+
+            # current package version is before v_start, so we're
+            # not affected by the CVE
+            if not inrange:
+                return self.CVE_DOESNT_AFFECT
+
+        if self.v_end:
+            try:
+                cve_affected_version = distutils.version.LooseVersion(self.v_end)
+                inrange = self.ops.get(self.op_end)(pkg_version, cve_affected_version)
+            except TypeError:
+                return self.CVE_UNKNOWN
+
+            # current package version is after v_end, so we're
+            # not affected by the CVE
+            if not inrange:
+                return self.CVE_DOESNT_AFFECT
+
+        # We're in the version range affected by this CVE
+        return self.CVE_AFFECTS
+
+
+class CVE_API(NVD_API):
+    """Download and manage CVEs in a sqlite database."""
+    def __init__(self, nvd_path):
+        """ Create a new API and database endpoint."""
+        NVD_API.__init__(self, nvd_path, 'CVEs', 'nvdcve')
+
+    def init_db(self):
+        """
+        Create all tables if the are missing.
+        """
+        cursor = self.db_connection.cursor()
+
+        cursor.execute('CREATE TABLE IF NOT EXISTS cves (\
+            id TEXT UNIQUE, \
+            description TEXT, \
+            metric2 REAL, \
+            metric3 REAL, \
+            severity TEXT)')
+
+        cursor.execute('CREATE TABLE IF NOT EXISTS cpe_matches (\
+            id TEXT UNIQUE, \
+            criteria TEXT, \
+            version_start TEXT, \
+            version_end TEXT, \
+            operator_start TEXT, \
+            operator_end TEXT)')
+
+        cursor.execute('CREATE TABLE IF NOT EXISTS configurations (\
+            cve_id TEXT, \
+            cpe_match_id TEXT, \
+            FOREIGN KEY (cve_id) REFERENCES cve (id) ON DELETE CASCADE, \
+            FOREIGN KEY (cpe_match_id) REFERENCES cpe_match (id) ON DELETE CASCADE, \
+            UNIQUE (cve_id, cpe_match_id))')
+
+        cursor.close()
+
+    def extract_cve_data(self, cve):
+        """Map CVE API data to database fields."""
+        description = ''
+        for d in cve['descriptions']:
+            if d['lang'] == 'en':
+                description = d['value']
+        metric2 = 0.0
+        metric3 = 0.0
+        severity = 'UNKNOWN'
+        if 'cvssMetricV31' in cve['metrics']:
+            metric3 = cve['metrics']['cvssMetricV31'][0]['cvssData']['baseScore']
+            severity = cve['metrics']['cvssMetricV31'][0]['cvssData']['baseSeverity']
+        elif 'cvssMetricV30' in cve['metrics']:
+            metric3 = cve['metrics']['cvssMetricV30'][0]['cvssData']['baseScore']
+            severity = cve['metrics']['cvssMetricV30'][0]['cvssData']['baseSeverity']
+        elif 'cvssMetricV2' in cve['metrics']:
+            metric2 = cve['metrics']['cvssMetricV2'][0]['cvssData']['baseScore']
+            severity = cve['metrics']['cvssMetricV2'][0]['baseSeverity']
+
+        return [cve['id'], description, metric2, metric3, severity]
+
+    def extract_cpe_match_data(self, cpe_match):
+        """Map CPE match information to database fields."""
+        product = cpe_product(cpe_match['criteria'])
+        version = cpe_version(cpe_match['criteria'])
+        # ignore when product is '-', which means N/A
+        if product == '-':
+            return
+        op_start = ''
+        op_end = ''
+        v_start = ''
+        v_end = ''
+
+        if version != '*' and version != '-':
+            # Version is defined, this is a '=' match
+            op_start = '='
+            v_start = version
+        else:
+            # Parse start version, end version and operators
+            if 'versionStartIncluding' in cpe_match:
+                op_start = '>='
+                v_start = cpe_match['versionStartIncluding']
+
+            if 'versionStartExcluding' in cpe_match:
+                op_start = '>'
+                v_start = cpe_match['versionStartExcluding']
+
+            if 'versionEndIncluding' in cpe_match:
+                op_end = '<='
+                v_end = cpe_match['versionEndIncluding']
+
+            if 'versionEndExcluding' in cpe_match:
+                op_end = '<'
+                v_end = cpe_match['versionEndExcluding']
+
+        return [
+            cpe_match['matchCriteriaId'],
+            cpe_match['criteria'],
+            v_start,
+            v_end,
+            op_start,
+            op_end
+        ]
+
+    def save_to_db(self, content) -> bool:
+        """
+        Save the response of a single API request to the database
+        and report the progress.
+        """
+        cve_ids_changed = list()
+        cve_ids_dropped = list()
+        cves = list()
+        cpe_matches = list()
+        configurations = list()
+
+        for vul in content['vulnerabilities']:
+            if vul['cve']['vulnStatus'] == 'Rejected':
+                cve_ids_dropped.append((vul['cve']['id'],))
                 continue
 
-            if cpe['v_start']:
-                try:
-                    cve_affected_version = distutils.version.LooseVersion(cpe['v_start'])
-                    inrange = ops.get(cpe['op_start'])(pkg_version, cve_affected_version)
-                except TypeError:
-                    return self.CVE_UNKNOWN
-
-                # current package version is before v_start, so we're
-                # not affected by the CVE
-                if not inrange:
-                    continue
-
-            if cpe['v_end']:
-                try:
-                    cve_affected_version = distutils.version.LooseVersion(cpe['v_end'])
-                    inrange = ops.get(cpe['op_end'])(pkg_version, cve_affected_version)
-                except TypeError:
-                    return self.CVE_UNKNOWN
-
-                # current package version is after v_end, so we're
-                # not affected by the CVE
-                if not inrange:
-                    continue
-
-            # We're in the version range affected by this CVE
-            return self.CVE_AFFECTS
+            cve_ids_changed.append((vul['cve']['id'],))
+            cves.append(self.extract_cve_data(vul['cve']))
+
+            for config in vul['cve'].get('configurations', ()):
+                for node in config['nodes']:
+                    for cpe_match in node['cpeMatch']:
+                        if not cpe_match['vulnerable']:
+                            continue
+                        match_data = self.extract_cpe_match_data(cpe_match)
+                        if not match_data:
+                            continue
+                        cpe_matches.append(match_data)
+                        configurations.append([vul['cve']['id'], match_data[0]])
+
+        cursor = self.db_connection.cursor()
+
+        # Drop all CVEs that are rejected, status might have changed
+        cursor.executemany('DELETE FROM cves WHERE id = ?', cve_ids_dropped)
+
+        # Delete configuration mapping for included CVEs, otherwise we can't detect
+        # upstream dropping configurations.
+        cursor.executemany('DELETE FROM configurations WHERE cve_id = ?', cve_ids_changed)
+        cursor.executemany('INSERT OR REPLACE INTO cves VALUES (?, ?, ?, ?, ?)', cves)
+        cursor.executemany('INSERT OR REPLACE INTO cpe_matches VALUES (?, ?, ?, ?, ?, ?)', cpe_matches)
+        cursor.executemany('INSERT OR REPLACE INTO configurations VALUES (?, ?)', configurations)
+
+        cursor.close()
+
+        return True
+
+    def load_all(self):
+        """
+        Load all entries from the database and use CVE class
+        to yield each result individually.
+        Each yielded object represents one configuration that
+        the included CVE is vulnerable for.
+        """
+        self.check_for_updates()
+
+        cursor = self.db_connection.cursor()
+        sql = 'SELECT c.id as cve_id, m.id, m.criteria, m.version_start, m.version_end, \
+            m.operator_start, m.operator_end \
+            FROM configurations \
+            INNER JOIN cves AS c ON c.id = configurations.cve_id \
+            INNER JOIN cpe_matches AS m ON m.id = configurations.cpe_match_id \
+            ORDER BY cve_id'
+
+        for row in cursor.execute(sql):
+            yield CVE(row)
 
-        return self.CVE_DOESNT_AFFECT
+        cursor.close()
+        self.db_connection.close()
diff --git a/support/scripts/pkg-stats b/support/scripts/pkg-stats
index 9349a0df57..b1f02c15b8 100755
--- a/support/scripts/pkg-stats
+++ b/support/scripts/pkg-stats
@@ -32,6 +32,7 @@ import time
 import gzip
 import xml.etree.ElementTree
 import requests
+from cve import CVE_API, cpe_product
 
 brpath = os.path.normpath(os.path.join(os.path.dirname(__file__), "..", ".."))
 
@@ -611,15 +612,17 @@ async def check_package_latest_version(packages):
 
 
 def check_package_cve_affects(cve, cpe_product_pkgs):
-    for product in cve.affected_products:
-        if product not in cpe_product_pkgs:
+    product = cve.affected_product
+    if product not in cpe_product_pkgs:
+        return
+    for pkg in cpe_product_pkgs[product]:
+        if cve.identifier in pkg.cves or cve.identifier in pkg.unsure_cves:
             continue
-        for pkg in cpe_product_pkgs[product]:
-            cve_status = cve.affects(pkg.name, pkg.current_version, pkg.ignored_cves, pkg.cpeid)
-            if cve_status == cve.CVE_AFFECTS:
-                pkg.cves.append(cve.identifier)
-            elif cve_status == cve.CVE_UNKNOWN:
-                pkg.unsure_cves.append(cve.identifier)
+        cve_status = cve.affects(pkg.name, pkg.current_version, pkg.ignored_cves, pkg.cpeid)
+        if cve_status == cve.CVE_AFFECTS:
+            pkg.cves.append(cve.identifier)
+        elif cve_status == cve.CVE_UNKNOWN:
+            pkg.unsure_cves.append(cve.identifier)
 
 
 def check_package_cves(nvd_path, packages):
@@ -635,12 +638,13 @@ def check_package_cves(nvd_path, packages):
             pkg.status['cve'] = ("na", "no version information available")
             continue
         if pkg.cpeid:
-            cpe_product = cvecheck.cpe_product(pkg.cpeid)
-            cpe_product_pkgs[cpe_product].append(pkg)
+            product = cpe_product(pkg.cpeid)
+            cpe_product_pkgs[product].append(pkg)
         else:
             cpe_product_pkgs[pkg.name].append(pkg)
 
-    for cve in cvecheck.CVE.read_nvd_dir(nvd_path):
+    cve_api = CVE_API(nvd_path)
+    for cve in cve_api.load_all():
         check_package_cve_affects(cve, cpe_product_pkgs)
 
     for pkg in packages:
@@ -1291,13 +1295,8 @@ def parse_args():
 
 
 def __main__():
-    global cvecheck
-
     args = parse_args()
 
-    if args.nvd_path:
-        import cve as cvecheck
-
     show_info_js = None
     if args.packages:
         package_list = args.packages.split(",")
-- 
2.42.0

_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH v4 4/6] support/scripts/cve.py: switch to NVD API v2

[Arnout Vandecappelle via buildroot]

  Hi Daniel,

  I'm finally getting around to this patch series... I have one little question 
(and a lot of little changes that I'm doing while applying).

On 01/09/2023 21:27, Daniel Lang wrote:
> +            for config in vul['cve'].get('configurations', ()):
> +                for node in config['nodes']:
> +                    for cpe_match in node['cpeMatch']:
> +                        if not cpe_match['vulnerable']:
> +                            continue
> +                        match_data = self.extract_cpe_match_data(cpe_match)
> +                        if not match_data:
> +                            continue
> +                        cpe_matches.append(match_data)
> +                        configurations.append([vul['cve']['id'], match_data[0]])
> +
> +        cursor = self.db_connection.cursor()
> +
> +        # Drop all CVEs that are rejected, status might have changed
> +        cursor.executemany('DELETE FROM cves WHERE id = ?', cve_ids_dropped)
> +
> +        # Delete configuration mapping for included CVEs, otherwise we can't detect
> +        # upstream dropping configurations.
> +        cursor.executemany('DELETE FROM configurations WHERE cve_id = ?', cve_ids_changed)
> +        cursor.executemany('INSERT OR REPLACE INTO cves VALUES (?, ?, ?, ?, ?)', cves)
> +        cursor.executemany('INSERT OR REPLACE INTO cpe_matches VALUES (?, ?, ?, ?, ?, ?)', cpe_matches)
> +        cursor.executemany('INSERT OR REPLACE INTO configurations VALUES (?, ?)', configurations)

  AFAIU, the configurations table is there to make an M-N relationship between 
CVEs and CPE_matches. However, I don't understand why this is needed or even 
possible - i.e., it necessarily has to be a 1-N relationship. Indeed, the code 
above always creates both a CPE_match line and a configuration line; the ID of 
the CPE_match line is UNIQUE, so the same match line cannot be reused for 
another CVE.

  So, I think this configurations table is entirely redundant, and instead the 
cpe_matches table should just contain a reference to the CVE ID.

  (Don't change and resend this patch, I've made a whole lot of changes myself 
and I can remove this configurations table too.)

  Regards,
  Arnout
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH v4 4/6] support/scripts/cve.py: switch to NVD API v2

[Daniel Lang via buildroot]

Hi Arnout,

On 12.01.24 20:39, Arnout Vandecappelle wrote:
>  Hi Daniel,
> 
>  I'm finally getting around to this patch series... I have one little question (and a lot of little changes that I'm doing while applying).

As I mentioned in my last email to Thomas [0], I have since moved away from the database approach back to json files.
Additionally I toyed with the idea of using GitLab for fetching and optimizing the data because passing it on to buildroot users.
A quick prove of concept can be found found here [1].

> 
> On 01/09/2023 21:27, Daniel Lang wrote:
>> +            for config in vul['cve'].get('configurations', ()):
>> +                for node in config['nodes']:
>> +                    for cpe_match in node['cpeMatch']:
>> +                        if not cpe_match['vulnerable']:
>> +                            continue
>> +                        match_data = self.extract_cpe_match_data(cpe_match)
>> +                        if not match_data:
>> +                            continue
>> +                        cpe_matches.append(match_data)
>> +                        configurations.append([vul['cve']['id'], match_data[0]])
>> +
>> +        cursor = self.db_connection.cursor()
>> +
>> +        # Drop all CVEs that are rejected, status might have changed
>> +        cursor.executemany('DELETE FROM cves WHERE id = ?', cve_ids_dropped)
>> +
>> +        # Delete configuration mapping for included CVEs, otherwise we can't detect
>> +        # upstream dropping configurations.
>> +        cursor.executemany('DELETE FROM configurations WHERE cve_id = ?', cve_ids_changed)
>> +        cursor.executemany('INSERT OR REPLACE INTO cves VALUES (?, ?, ?, ?, ?)', cves)
>> +        cursor.executemany('INSERT OR REPLACE INTO cpe_matches VALUES (?, ?, ?, ?, ?, ?)', cpe_matches)
>> +        cursor.executemany('INSERT OR REPLACE INTO configurations VALUES (?, ?)', configurations)
> 
>  AFAIU, the configurations table is there to make an M-N relationship between CVEs and CPE_matches. However, I don't understand why this is needed or even possible - i.e., it necessarily has to be a 1-N relationship. Indeed, the code above always creates both a CPE_match line and a configuration line; the ID of the CPE_match line is UNIQUE, so the same match line cannot be reused for another CVE.

Looking at the database when I last ran this version in September of 2023, the counts in the three tables looked like this:
- cves: 211066 rows
- cve_matches: 398178 rows
- configurations: 1971459 rows

So it seems that the same cve_matches ID is used for multiple cves.

> 
>  So, I think this configurations table is entirely redundant, and instead the cpe_matches table should just contain a reference to the CVE ID.
> 
>  (Don't change and resend this patch, I've made a whole lot of changes myself and I can remove this configurations table too.)
> 
>  Regards,
>  Arnout

Regards,
Daniel

[0]: https://lists.buildroot.org/pipermail/buildroot/2023-December/681888.html
[1]: https://gitlab.com/daniellang/nvd-v2.0/
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH v4 4/6] support/scripts/cve.py: switch to NVD API v2

[Arnout Vandecappelle via buildroot]

  Hi Daniel,

On 28/01/2024 20:40, Daniel Lang wrote:
> Hi Arnout,
> 
> On 12.01.24 20:39, Arnout Vandecappelle wrote:
>>   Hi Daniel,
>>
>>   I'm finally getting around to this patch series... I have one little question (and a lot of little changes that I'm doing while applying).
> 
> As I mentioned in my last email to Thomas [0], I have since moved away from the database approach back to json files.

  Oh, I wasn't aware of that. But now you mention it, I vaguely remember someone 
suggested to use json files instead of sqlite, indeed.

> Additionally I toyed with the idea of using GitLab for fetching and optimizing the data because passing it on to buildroot users.

  I think it makes more sense to use BR2_PRIMARY_SITE to store a cached version. 
However, if we do that, it should have a hash and everything, so it's perhaps 
not super practical...

> A quick prove of concept can be found found here [1].

  I'll take a look at it some time :-)

> 
>>
>> On 01/09/2023 21:27, Daniel Lang wrote:
>>> +            for config in vul['cve'].get('configurations', ()):
>>> +                for node in config['nodes']:
>>> +                    for cpe_match in node['cpeMatch']:
>>> +                        if not cpe_match['vulnerable']:
>>> +                            continue
>>> +                        match_data = self.extract_cpe_match_data(cpe_match)
>>> +                        if not match_data:
>>> +                            continue
>>> +                        cpe_matches.append(match_data)
>>> +                        configurations.append([vul['cve']['id'], match_data[0]])
>>> +
>>> +        cursor = self.db_connection.cursor()
>>> +
>>> +        # Drop all CVEs that are rejected, status might have changed
>>> +        cursor.executemany('DELETE FROM cves WHERE id = ?', cve_ids_dropped)
>>> +
>>> +        # Delete configuration mapping for included CVEs, otherwise we can't detect
>>> +        # upstream dropping configurations.
>>> +        cursor.executemany('DELETE FROM configurations WHERE cve_id = ?', cve_ids_changed)
>>> +        cursor.executemany('INSERT OR REPLACE INTO cves VALUES (?, ?, ?, ?, ?)', cves)
>>> +        cursor.executemany('INSERT OR REPLACE INTO cpe_matches VALUES (?, ?, ?, ?, ?, ?)', cpe_matches)
>>> +        cursor.executemany('INSERT OR REPLACE INTO configurations VALUES (?, ?)', configurations)
>>
>>   AFAIU, the configurations table is there to make an M-N relationship between CVEs and CPE_matches. However, I don't understand why this is needed or even possible - i.e., it necessarily has to be a 1-N relationship. Indeed, the code above always creates both a CPE_match line and a configuration line; the ID of the CPE_match line is UNIQUE, so the same match line cannot be reused for another CVE.
> 
> Looking at the database when I last ran this version in September of 2023, the counts in the three tables looked like this:
> - cves: 211066 rows
> - cve_matches: 398178 rows
> - configurations: 1971459 rows
> 
> So it seems that the same cve_matches ID is used for multiple cves.

  Right! I'm not sure though if the additional configurations table really saves 
space - since we only save a subset of the criteria, it's possible that 
duplicating the cpe_matches doesn't take (much) more space than the additional 
configurations table. Unfortunately I didn't implement it like that (I 
incorrectly overwrite the cpe_match's CVEID when the same one occurs several 
times), so I don't have the numbers to be sure.

  Anyway, as you say: if we go for storing JSON files after all, the point is 
probably moot.

  Regards,
  Arnout


> 
>>
>>   So, I think this configurations table is entirely redundant, and instead the cpe_matches table should just contain a reference to the CVE ID.
>>
>>   (Don't change and resend this patch, I've made a whole lot of changes myself and I can remove this configurations table too.)
>>
>>   Regards,
>>   Arnout
> 
> Regards,
> Daniel
> 
> [0]: https://lists.buildroot.org/pipermail/buildroot/2023-December/681888.html
> [1]: https://gitlab.com/daniellang/nvd-v2.0/
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

[Buildroot] [PATCH v4 5/6] support/scripts/pkg-stats: switch CPEs to NVD API v2

[Daniel Lang]

The currently used feed will be retired in December 2023 [0].
As an alternative the new v2 API [1]. The new API allows downloading
sets of CPEs (10k at a time) that were modified after a given
UTC timestamp. Rate limiting of 5 requests in a sliding 30 seconds
window is used [2].
nvd_api_v2.py is used to store the data in a local sqlite database.
Only a subset of fields is stored to keep the database "smaller"
(still ~200MB).

[0]: https://nvd.nist.gov/General/News/change-timeline
[1]: https://nvd.nist.gov/developers/products
[2]: https://nvd.nist.gov/developers/start-here

Signed-off-by: Daniel Lang <dalang@gmx.at>
---
v3 -> v4:
- progress output moved to nvd_api_v2
- self.connection was renamed to self.db_connection
v2 -> v3:
- drop XML handling for gen-missing-cpe
- Move creation of CPE_ID class into this patch

Signed-off-by: Daniel Lang <dalang@gmx.at>
---
 support/scripts/cpe.py    | 89 +++++++++++++++++++++++++++++++++++++++
 support/scripts/cve.py    | 29 +++----------
 support/scripts/pkg-stats | 42 +++---------------
 3 files changed, 100 insertions(+), 60 deletions(-)
 create mode 100755 support/scripts/cpe.py

diff --git a/support/scripts/cpe.py b/support/scripts/cpe.py
new file mode 100755
index 0000000000..671923ebff
--- /dev/null
+++ b/support/scripts/cpe.py
@@ -0,0 +1,89 @@
+#!/usr/bin/env python3
+
+from nvd_api_v2 import NVD_API
+
+
+class CPE_ID:
+    @staticmethod
+    def matches(cpe1, cpe2):
+        """Check if two CPE IDs match each other"""
+        cpe1_elems = cpe1.split(":")
+        cpe2_elems = cpe2.split(":")
+
+        remains = filter(lambda x: x[0] not in ["*", "-"] and x[1] not in ["*", "-"] and x[0] != x[1],
+                         zip(cpe1_elems, cpe2_elems))
+        return len(list(remains)) == 0
+
+    @staticmethod
+    def product(cpe):
+        return cpe.split(':')[4]
+
+    @staticmethod
+    def version(cpe):
+        return cpe.split(':')[5]
+
+    @staticmethod
+    def no_version(cpe):
+        return ":".join(cpe.split(":")[:5])
+
+
+class CPE_API(NVD_API):
+    def __init__(self, nvd_path):
+        NVD_API.__init__(self, nvd_path, 'CPEs', 'nvdcpe')
+        self.cpes = list()
+        self.cpes_without_version = dict()
+
+    def init_db(self):
+        cursor = self.db_connection.cursor()
+
+        cursor.execute('CREATE TABLE IF NOT EXISTS products ( \
+            id TEXT UNIQUE, \
+            name TEXT)')
+
+        cursor.close()
+
+    def save_to_db(self, content):
+        cpe_ids_dropped = list()
+        products = list()
+
+        for product in content['products']:
+            if product['cpe']['deprecated']:
+                cpe_ids_dropped.append((product['cpe']['cpeNameId'],))
+                continue
+
+            cpe = product['cpe']
+
+            products.append([cpe['cpeNameId'], cpe['cpeName']])
+
+        cursor = self.db_connection.cursor()
+
+        # Drop all CPEs that are deprecated, status might have changed
+        cursor.executemany('DELETE FROM products WHERE id = ?', cpe_ids_dropped)
+        cursor.executemany('INSERT OR REPLACE INTO products VALUES (?, ?)', products)
+
+        return True
+
+    def load_ids(self):
+        self.check_for_updates()
+
+        cursor = self.db_connection.cursor()
+
+        ids = list()
+        for row in cursor.execute('SELECT name FROM products'):
+            ids.append(row[0])
+
+        cursor.close()
+        self.db_connection.close()
+
+        self.cpes = ids
+        return ids
+
+    def generate_partials(self):
+        self.cpes_without_version = dict()
+        for cpe in self.cpes:
+            self.cpes_without_version[CPE_ID.no_version(cpe)] = cpe
+
+    def find_partial(self, cpe_id):
+        cpe_id_without_version = CPE_ID.no_version(cpe_id)
+        if cpe_id_without_version in self.cpes_without_version.keys():
+            return self.cpes_without_version[cpe_id_without_version]
diff --git a/support/scripts/cve.py b/support/scripts/cve.py
index b09ac64c81..ce951d60f6 100755
--- a/support/scripts/cve.py
+++ b/support/scripts/cve.py
@@ -20,24 +20,7 @@
 import distutils.version
 import operator
 from nvd_api_v2 import NVD_API
-
-
-# Check if two CPE IDs match each other
-def cpe_matches(cpe1, cpe2):
-    cpe1_elems = cpe1.split(":")
-    cpe2_elems = cpe2.split(":")
-
-    remains = filter(lambda x: x[0] not in ["*", "-"] and x[1] not in ["*", "-"] and x[0] != x[1],
-                     zip(cpe1_elems, cpe2_elems))
-    return len(list(remains)) == 0
-
-
-def cpe_product(cpe):
-    return cpe.split(':')[4]
-
-
-def cpe_version(cpe):
-    return cpe.split(':')[5]
+from cpe import CPE_ID
 
 
 class CVE:
@@ -71,7 +54,7 @@ class CVE:
     @property
     def affected_product(self):
         """Name of the affected product"""
-        return cpe_product(self.match_criteria)
+        return CPE_ID.product(self.match_criteria)
 
     def affects(self, name, version, cve_ignore_list, cpeid=None):
         """
@@ -93,9 +76,9 @@ class CVE:
         # version, as they might be different due to
         # <pkg>_CPE_ID_VERSION
         else:
-            pkg_version = distutils.version.LooseVersion(cpe_version(cpeid))
+            pkg_version = distutils.version.LooseVersion(CPE_ID.version(cpeid))
 
-        if not cpe_matches(self.match_criteria, cpeid):
+        if not CPE_ID.matches(self.match_criteria, cpeid):
             return self.CVE_DOESNT_AFFECT
         if not self.v_start and not self.v_end:
             return self.CVE_AFFECTS
@@ -189,8 +172,8 @@ class CVE_API(NVD_API):
 
     def extract_cpe_match_data(self, cpe_match):
         """Map CPE match information to database fields."""
-        product = cpe_product(cpe_match['criteria'])
-        version = cpe_version(cpe_match['criteria'])
+        product = CPE_ID.product(cpe_match['criteria'])
+        version = CPE_ID.version(cpe_match['criteria'])
         # ignore when product is '-', which means N/A
         if product == '-':
             return
diff --git a/support/scripts/pkg-stats b/support/scripts/pkg-stats
index b1f02c15b8..3d6c53a353 100755
--- a/support/scripts/pkg-stats
+++ b/support/scripts/pkg-stats
@@ -28,11 +28,8 @@ import re
 import subprocess
 import json
 import sys
-import time
-import gzip
-import xml.etree.ElementTree
-import requests
-from cve import CVE_API, cpe_product
+from cpe import CPE_API, CPE_ID
+from cve import CVE_API
 
 brpath = os.path.normpath(os.path.join(os.path.dirname(__file__), "..", ".."))
 
@@ -638,7 +635,7 @@ def check_package_cves(nvd_path, packages):
             pkg.status['cve'] = ("na", "no version information available")
             continue
         if pkg.cpeid:
-            product = cpe_product(pkg.cpeid)
+            product = CPE_ID.product(pkg.cpeid)
             cpe_product_pkgs[product].append(pkg)
         else:
             cpe_product_pkgs[pkg.name].append(pkg)
@@ -656,37 +653,8 @@ def check_package_cves(nvd_path, packages):
 
 
 def check_package_cpes(nvd_path, packages):
-    class CpeXmlParser:
-        cpes = []
-
-        def start(self, tag, attrib):
-            if tag == "{http://scap.nist.gov/schema/cpe-extension/2.3}cpe23-item":
-                self.cpes.append(attrib['name'])
-
-        def close(self):
-            return self.cpes
-
-    print("CPE: Setting up NIST dictionary")
-    if not os.path.exists(os.path.join(nvd_path, "cpe")):
-        os.makedirs(os.path.join(nvd_path, "cpe"))
-
-    cpe_dict_local = os.path.join(nvd_path, "cpe", os.path.basename(CPEDB_URL))
-    if not os.path.exists(cpe_dict_local) or os.stat(cpe_dict_local).st_mtime < time.time() - 86400:
-        print("CPE: Fetching xml manifest from [" + CPEDB_URL + "]")
-        cpe_dict = requests.get(CPEDB_URL)
-        open(cpe_dict_local, "wb").write(cpe_dict.content)
-
-    print("CPE: Unzipping xml manifest...")
-    nist_cpe_file = gzip.GzipFile(fileobj=open(cpe_dict_local, 'rb'))
-
-    parser = xml.etree.ElementTree.XMLParser(target=CpeXmlParser())
-    while True:
-        c = nist_cpe_file.read(1024*1024)
-        if not c:
-            break
-        parser.feed(c)
-    cpes = parser.close()
-
+    cpe_api = CPE_API(nvd_path)
+    cpes = cpe_api.load_ids()
     for p in packages:
         if not p.cpeid:
             continue
-- 
2.42.0

_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

[Buildroot] [PATCH v4 6/6] support/scripts/pkg-stats: Only match CPE vendor and product

[Daniel Lang]

Instead of checking if the whole CPE string is known in the database,
only check if a string with the same vendor and product exists.

Reported-by: Arnout Vandecappelle <arnout@mind.be>
Signed-off-by: Daniel Lang <dalang@gmx.at>
---
v1 -> v2:
- patch added
---
 support/scripts/pkg-stats | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/support/scripts/pkg-stats b/support/scripts/pkg-stats
index 3d6c53a353..d48582ea3b 100755
--- a/support/scripts/pkg-stats
+++ b/support/scripts/pkg-stats
@@ -654,11 +654,12 @@ def check_package_cves(nvd_path, packages):
 
 def check_package_cpes(nvd_path, packages):
     cpe_api = CPE_API(nvd_path)
-    cpes = cpe_api.load_ids()
+    cpe_api.load_ids()
+    cpe_api.generate_partials()
     for p in packages:
         if not p.cpeid:
             continue
-        if p.cpeid in cpes:
+        if cpe_api.find_partial(p.cpeid):
             p.status['cpe'] = ("ok", "verified CPE identifier")
         else:
             p.status['cpe'] = ("error", "CPE version unknown in CPE database")
-- 
2.42.0

_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH v4 1/6] support/scripts/pkg-stats: ignore more makefiles that aren't packages

[Thomas Petazzoni via buildroot]

On Fri,  1 Sep 2023 21:27:09 +0200
Daniel Lang <dalang@gmx.at> wrote:

> Some packages are grouped and have a general makefile that defines
> reusable variables. These makefiles have no relevant information for
> pkg-stats and should be excluded.
> 
> Signed-off-by: Daniel Lang <dalang@gmx.at>
> ---
> v3 -> v4:
> - Added more packages to ignore list, not just llvm-project
> ---
>  support/scripts/pkg-stats | 7 ++++++-
>  1 file changed, 6 insertions(+), 1 deletion(-)

Applied to master, thanks.

Thomas
-- 
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot