[PATCH 1/2] cve-update-db-native: update the CVE database once a day only

[PATCH 1/2] cve-update-db-native: update the CVE database once a day only

[Marta Rybczynska]

The update of the NVD database was expected to happen once per hour.
However, the database file date changes only if the content was actually
updated. In practice, the check worked for the first hour after the
new download.

As the NVD database changes usually only once a day, we can just
update it less frequently.

Signed-off-by: Marta Rybczynska <marta.rybczynska@huawei.com>
---
 meta/recipes-core/meta/cve-update-db-native.bb | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/meta/recipes-core/meta/cve-update-db-native.bb b/meta/recipes-core/meta/cve-update-db-native.bb
index e5822cee58..af39480dda 100644
--- a/meta/recipes-core/meta/cve-update-db-native.bb
+++ b/meta/recipes-core/meta/cve-update-db-native.bb
@@ -43,10 +43,10 @@ python do_fetch() {
         if os.path.exists(db_file):
             os.remove(db_file)
 
-    # Don't refresh the database more than once an hour
+    # The NVD database changes once a day, so no need to update more frequently
     try:
         import time
-        if time.time() - os.path.getmtime(db_file) < (60*60):
+        if time.time() - os.path.getmtime(db_file) < (24*60*60):
             bb.debug(2, "Recently updated, skipping")
             return
     except OSError:
-- 
2.33.0

[PATCH 2/2] cve-update-db-native: allow an option to force the CVE database update

[Marta Rybczynska]

Add a new variable FORCE_CVE_DB_UPDATE allowing the user to force
the database update, if the default update frequency is too low.

Signed-off-by: Marta Rybczynska <marta.rybczynska@huawei.com>
---
 meta/recipes-core/meta/cve-update-db-native.bb | 10 +++++++---
 1 file changed, 7 insertions(+), 3 deletions(-)

diff --git a/meta/recipes-core/meta/cve-update-db-native.bb b/meta/recipes-core/meta/cve-update-db-native.bb
index af39480dda..d89f79f310 100644
--- a/meta/recipes-core/meta/cve-update-db-native.bb
+++ b/meta/recipes-core/meta/cve-update-db-native.bb
@@ -13,6 +13,7 @@ deltask do_install
 deltask do_populate_sysroot
 
 NVDCVE_URL ?= "https://nvd.nist.gov/feeds/json/cve/1.1/nvdcve-1.1-"
+FORCE_CVE_DB_UPDATE ?= "0"
 
 python () {
     if not bb.data.inherits_class("cve-check", d):
@@ -44,11 +45,14 @@ python do_fetch() {
             os.remove(db_file)
 
     # The NVD database changes once a day, so no need to update more frequently
+    # Allow the user to force-update
     try:
         import time
-        if time.time() - os.path.getmtime(db_file) < (24*60*60):
-            bb.debug(2, "Recently updated, skipping")
-            return
+        if d.getVar("FORCE_CVE_DB_UPDATE") == "0":
+            if time.time() - os.path.getmtime(db_file) < (24*60*60):
+                bb.debug(2, "Recently updated, skipping")
+                return
+
     except OSError:
         pass
 
-- 
2.33.0

Re: [OE-core] [PATCH 2/2] cve-update-db-native: allow an option to force the CVE database update

[Ralph Siemsen]

Hi Marta,

This explains why the CVE database update seemed to happen far more
frequently than it should. Thanks for digging into it.

On Fri, Apr 29, 2022 at 2:32 AM Marta Rybczynska <rybczynska@gmail.com> wrote:
>
> Add a new variable FORCE_CVE_DB_UPDATE allowing the user to force
> the database update, if the default update frequency is too low.

Just an idea, maybe instead of a boolean, the variable could specify
the interval, eg:

# Interval between CVE database updates, in seconds.
# Set to "0" to to force an update of the database.
CVE_DATABASE_UPDATE_INTERVAL ?= "24*60*60"

Regards,
Ralph

Re: [OE-core] [PATCH 2/2] cve-update-db-native: allow an option to force the CVE database update

[Marta Rybczynska]

[-- Attachment #1: Type: text/plain, Size: 798 bytes --]

On Fri, Apr 29, 2022 at 5:53 PM Ralph Siemsen <ralph.siemsen@linaro.org>
wrote:

> Hi Marta,
>
> This explains why the CVE database update seemed to happen far more
> frequently than it should. Thanks for digging into it.
>
> On Fri, Apr 29, 2022 at 2:32 AM Marta Rybczynska <rybczynska@gmail.com>
> wrote:
> >
> > Add a new variable FORCE_CVE_DB_UPDATE allowing the user to force
> > the database update, if the default update frequency is too low.
>
> Just an idea, maybe instead of a boolean, the variable could specify
> the interval, eg:
>
> # Interval between CVE database updates, in seconds.
> # Set to "0" to to force an update of the database.
> CVE_DATABASE_UPDATE_INTERVAL ?= "24*60*60"
>
>
This is a good idea, thank you Ralph, I like it. I'll be sending a v2
shortly.

Regards,
Marta

[-- Attachment #2: Type: text/html, Size: 1329 bytes --]

Re: [OE-core] [PATCH 2/2] cve-update-db-native: allow an option to force the CVE database update

[Ralph Siemsen]

On Mon, May 2, 2022 at 9:23 AM Marta Rybczynska <rybczynska@gmail.com> wrote:
>
> On Fri, Apr 29, 2022 at 5:53 PM Ralph Siemsen <ralph.siemsen@linaro.org> wrote:
>>
>> # Interval between CVE database updates, in seconds.
>> # Set to "0" to to force an update of the database.
>> CVE_DATABASE_UPDATE_INTERVAL ?= "24*60*60"
>>
>
> This is a good idea, thank you Ralph, I like it. I'll be sending a v2
> shortly.

Thanks for this. I've tested it locally, on the fist run, the CVE
database was fetched (it took quite a while, for some reason). On
subsequent runs, no fetch occurs, so this seems to be working.

I then set CVE_DB_UPDATE_INTERVAL = "3600" in my local.conf, and ran
the build again. As it had been over an hour since the first build,
the database was downloaded again. The timestamp on nvdcve_1.1.db did
not change (as noted in one of your commit descriptions).

So, it seems to be working correctly. I will re-test tomorrow (eg.
after 24 hours) with the interval set back to default.

One minor point of confusion is that the log still shows "NOTE: recipe
cve-update-db-native-1.0-r0: task do_fetch: Started" even when the
download is skipped. This is of course understandable when looking at
the python code, the check is within the do_fetch function. There is
probably no simple way to avoid this from being displayed. And most
users won't notice anyway. But it initially confused me about what was
happening.

Regards,
Ralph

Re: [OE-core] [PATCH 2/2] cve-update-db-native: allow an option to force the CVE database update

[Khem Raj]

On Mon, May 2, 2022 at 12:43 PM Ralph Siemsen <ralph.siemsen@linaro.org> wrote:
>
> On Mon, May 2, 2022 at 9:23 AM Marta Rybczynska <rybczynska@gmail.com> wrote:
> >
> > On Fri, Apr 29, 2022 at 5:53 PM Ralph Siemsen <ralph.siemsen@linaro.org> wrote:
> >>
> >> # Interval between CVE database updates, in seconds.
> >> # Set to "0" to to force an update of the database.
> >> CVE_DATABASE_UPDATE_INTERVAL ?= "24*60*60"
> >>
> >
> > This is a good idea, thank you Ralph, I like it. I'll be sending a v2
> > shortly.
>
> Thanks for this. I've tested it locally, on the fist run, the CVE
> database was fetched (it took quite a while, for some reason). On
> subsequent runs, no fetch occurs, so this seems to be working.
>
> I then set CVE_DB_UPDATE_INTERVAL = "3600" in my local.conf, and ran
> the build again. As it had been over an hour since the first build,
> the database was downloaded again. The timestamp on nvdcve_1.1.db did
> not change (as noted in one of your commit descriptions).
>
> So, it seems to be working correctly. I will re-test tomorrow (eg.
> after 24 hours) with the interval set back to default.
>
> One minor point of confusion is that the log still shows "NOTE: recipe
> cve-update-db-native-1.0-r0: task do_fetch: Started" even when the
> download is skipped. This is of course understandable when looking at
> the python code, the check is within the do_fetch function. There is
> probably no simple way to avoid this from being displayed. And most
> users won't notice anyway. But it initially confused me about what was
> happening.


yeah you can avoid it unless another logic outside of this function is invoked.
perhaps we can add a diagnostic inside the do_fetch to spill out the
state information
and informing like "no fetch needed" or "database uptodate" or somesuch
>
> Regards,
> Ralph
>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#165172): https://lists.openembedded.org/g/openembedded-core/message/165172
> Mute This Topic: https://lists.openembedded.org/mt/90771095/1997914
> Group Owner: openembedded-core+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [raj.khem@gmail.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>