Re: [OE-core] [PATCH] cve-check-update-nvd2-native: Incremement DL_DIR database location

Re: [OE-core] [PATCH] cve-check-update-nvd2-native: Incremement DL_DIR database location

[Richard Purdie]

On Fri, 2024-10-11 at 11:56 +0100, Richard Purdie via lists.openembedded.org wrote:
> We're seeing a lot of sqlite database corruption issues in our automated
> testing. It is unclear why this is happening. There were process
> imrpovements implemented in master and it is unclear if older releases
> are somehow making those changes ineffective or if the problem is
> elsewhere.
> 
> By changing the location in DL_DIR, we split the two sets of accesses
> to be separate and can isolate whether the master changes really did
> improve things or not. If successful, we may consider backporting those
> changes to the stable releases.
> 
> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
> ---
>  meta/recipes-core/meta/cve-update-nvd2-native.bb | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/meta/recipes-core/meta/cve-update-nvd2-native.bb b/meta/recipes-core/meta/cve-update-nvd2-native.bb
> index 2d23d28c3e2..a7e568e3079 100644
> --- a/meta/recipes-core/meta/cve-update-nvd2-native.bb
> +++ b/meta/recipes-core/meta/cve-update-nvd2-native.bb
> @@ -34,7 +34,7 @@ CVE_DB_INCR_UPDATE_AGE_THRES ?= "10368000"
>  # Number of attempts for each http query to nvd server before giving up
>  CVE_DB_UPDATE_ATTEMPTS ?= "5"
>  
> -CVE_CHECK_DB_DLDIR_FILE ?= "${DL_DIR}/CVE_CHECK/${CVE_CHECK_DB_FILENAME}"
> +CVE_CHECK_DB_DLDIR_FILE ?= "${DL_DIR}/CVE_CHECK2/${CVE_CHECK_DB_FILENAME}"
>  CVE_CHECK_DB_DLDIR_LOCK ?= "${CVE_CHECK_DB_DLDIR_FILE}.lock"
>  CVE_CHECK_DB_TEMP_FILE ?= "${CVE_CHECK_DB_FILE}.tmp"

I now do have an example of a corrupted database:

https://valkyrie.yocto.io/pub/shared-failure-data/nvdcve_2-1.db

I'm not sure why this is happening but the older releases are all
accessing the DB over NFS which makes me nervous. We're seeing a lot of
failures on the new cluster, hopefully this will help isolate the
problem.

Cheers,

Richard

Re: [OE-core] [PATCH] cve-check-update-nvd2-native: Incremement DL_DIR database location

[Marta Rybczynska]

[-- Attachment #1: Type: text/plain, Size: 732 bytes --]

On Fri, Oct 11, 2024 at 1:02 PM Richard Purdie via lists.openembedded.org
<richard.purdie=linuxfoundation.org@lists.openembedded.org> wrote:

>
> I now do have an example of a corrupted database:
>
> https://valkyrie.yocto.io/pub/shared-failure-data/nvdcve_2-1.db
>
> I'm not sure why this is happening but the older releases are all
> accessing the DB over NFS which makes me nervous. We're seeing a lot of
> failures on the new cluster, hopefully this will help isolate the
> problem.
>

The database isn't completely corrupted, it looks like it is partial,
however. What was the error
that you got in that build? Any or just strange results? I'm running a
verification vs a fresh database.

Regards,
Marta

[-- Attachment #2: Type: text/html, Size: 1307 bytes --]

Re: [OE-core] [PATCH] cve-check-update-nvd2-native: Incremement DL_DIR database location

[Richard Purdie]

On Fri, 2024-10-11 at 13:17 +0200, Marta Rybczynska wrote:
> 
> On Fri, Oct 11, 2024 at 1:02 PM Richard Purdie via
> lists.openembedded.org
> <richard.purdie=linuxfoundation.org@lists.openembedded.org> wrote:
> > 
> > I now do have an example of a corrupted database:
> > 
> > https://valkyrie.yocto.io/pub/shared-failure-data/nvdcve_2-1.db
> > 
> > I'm not sure why this is happening but the older releases are all
> > accessing the DB over NFS which makes me nervous. We're seeing a
> > lot of
> > failures on the new cluster, hopefully this will help isolate the
> > problem.
> > 
> 
> 
> The database isn't completely corrupted, it looks like it is partial,
> however. What was the error
> that you got in that build? Any or just strange results? I'm running
> a verification vs a fresh database.

https://valkyrie.yoctoproject.org/#/builders/23/builds/238/steps/14/logs/stdio

Exception: sqlite3.DatabaseError: database disk image is malformed

Cheers,

Richard

Re: [OE-core] [PATCH] cve-check-update-nvd2-native: Incremement DL_DIR database location

[Marta Rybczynska]

[-- Attachment #1: Type: text/plain, Size: 1197 bytes --]

On Fri, Oct 11, 2024 at 1:19 PM Richard Purdie <
richard.purdie@linuxfoundation.org> wrote:

> On Fri, 2024-10-11 at 13:17 +0200, Marta Rybczynska wrote:
> >
> > On Fri, Oct 11, 2024 at 1:02 PM Richard Purdie via
> > lists.openembedded.org
> > <richard.purdie=linuxfoundation.org@lists.openembedded.org> wrote:
> > >
> > > I now do have an example of a corrupted database:
> > >
> > > https://valkyrie.yocto.io/pub/shared-failure-data/nvdcve_2-1.db
> > >
> > > I'm not sure why this is happening but the older releases are all
> > > accessing the DB over NFS which makes me nervous. We're seeing a
> > > lot of
> > > failures on the new cluster, hopefully this will help isolate the
> > > problem.
> > >
> >
> >
> > The database isn't completely corrupted, it looks like it is partial,
> > however. What was the error
> > that you got in that build? Any or just strange results? I'm running
> > a verification vs a fresh database.
>
>
> https://valkyrie.yoctoproject.org/#/builders/23/builds/238/steps/14/logs/stdio
>
> Exception: sqlite3.DatabaseError: database disk image is malformed
>
>
Managed to get the same error with the sqlite command line.

Marta

[-- Attachment #2: Type: text/html, Size: 2133 bytes --]

Re: [OE-core] [PATCH] cve-check-update-nvd2-native: Incremement DL_DIR database location

[Richard Purdie]

On Fri, 2024-10-11 at 13:24 +0200, Marta Rybczynska wrote:
> 
> 
> On Fri, Oct 11, 2024 at 1:19 PM Richard Purdie <richard.purdie@linuxfoundation.org> wrote:
> > On Fri, 2024-10-11 at 13:17 +0200, Marta Rybczynska wrote:
> > > 
> > > On Fri, Oct 11, 2024 at 1:02 PM Richard Purdie via
> > > lists.openembedded.org
> > > <richard.purdie=linuxfoundation.org@lists.openembedded.org> wrote:
> > > > 
> > > > I now do have an example of a corrupted database:
> > > > 
> > > > https://valkyrie.yocto.io/pub/shared-failure-data/nvdcve_2-1.db
> > > > 
> > > > I'm not sure why this is happening but the older releases are all
> > > > accessing the DB over NFS which makes me nervous. We're seeing a
> > > > lot of
> > > > failures on the new cluster, hopefully this will help isolate the
> > > > problem.
> > > > 
> > > 
> > > 
> > > The database isn't completely corrupted, it looks like it is partial,
> > > however. What was the error
> > > that you got in that build? Any or just strange results? I'm running
> > > a verification vs a fresh database.
> > 
> > https://valkyrie.yoctoproject.org/#/builders/23/builds/238/steps/14/logs/stdio
> > 
> > Exception: sqlite3.DatabaseError: database disk image is malformed
> > 
> > 
> 
> 
> Managed to get the same error with the sqlite command line.


I merged this patch as it would rule out a lot of things. The error recurred:

https://valkyrie.yoctoproject.org//#/builders/76/builds/235/steps/14/logs/stdio

This means that it happens with the access patterns in master alone.
I'm also noticing there seems to be a trend for this to happen on alma9
(older centos systems?).

Running those tests manually on that worker passed the tests ok.

It has python 3.9.18 with sqlite3 not available as an external command.

https://bugzilla.yoctoproject.org/show_bug.cgi?id=14899

Cheers,

Richard

Re: [OE-core] [PATCH] cve-check-update-nvd2-native: Incremement DL_DIR database location

[Richard Purdie]

On Sun, 2024-10-13 at 08:26 +0100, Richard Purdie via
lists.openembedded.org wrote:
> On Fri, 2024-10-11 at 13:24 +0200, Marta Rybczynska wrote:
> > 
> > 
> > On Fri, Oct 11, 2024 at 1:19 PM Richard Purdie
> > <richard.purdie@linuxfoundation.org> wrote:
> > > On Fri, 2024-10-11 at 13:17 +0200, Marta Rybczynska wrote:
> > > > 
> > > > On Fri, Oct 11, 2024 at 1:02 PM Richard Purdie via
> > > > lists.openembedded.org
> > > > <richard.purdie=linuxfoundation.org@lists.openembedded.org>
> > > > wrote:
> > > > > 
> > > > > I now do have an example of a corrupted database:
> > > > > 
> > > > > https://valkyrie.yocto.io/pub/shared-failure-data/nvdcve_2-1.db
> > > > > 
> > > > > I'm not sure why this is happening but the older releases are
> > > > > all
> > > > > accessing the DB over NFS which makes me nervous. We're
> > > > > seeing a
> > > > > lot of
> > > > > failures on the new cluster, hopefully this will help isolate
> > > > > the
> > > > > problem.
> > > > > 
> > > > 
> > > > 
> > > > The database isn't completely corrupted, it looks like it is
> > > > partial,
> > > > however. What was the error
> > > > that you got in that build? Any or just strange results? I'm
> > > > running
> > > > a verification vs a fresh database.
> > > 
> > > https://valkyrie.yoctoproject.org/#/builders/23/builds/238/steps/14/logs/stdio
> > > 
> > > Exception: sqlite3.DatabaseError: database disk image is
> > > malformed
> > > 
> > > 
> > 
> > 
> > Managed to get the same error with the sqlite command line.
> 
> 
> I merged this patch as it would rule out a lot of things.

It looks like I didn't do this. I will do so, then we can rule out a
few things.

Cheers,

Richard

Re: [OE-core] [PATCH] cve-check-update-nvd2-native: Incremement DL_DIR database location

[Marta Rybczynska]

[-- Attachment #1: Type: text/plain, Size: 2764 bytes --]

On Mon, Oct 14, 2024 at 6:07 PM Richard Purdie <
richard.purdie@linuxfoundation.org> wrote:

> On Sun, 2024-10-13 at 08:26 +0100, Richard Purdie via
> lists.openembedded.org wrote:
> > On Fri, 2024-10-11 at 13:24 +0200, Marta Rybczynska wrote:
> > >
> > >
> > > On Fri, Oct 11, 2024 at 1:19 PM Richard Purdie
> > > <richard.purdie@linuxfoundation.org> wrote:
> > > > On Fri, 2024-10-11 at 13:17 +0200, Marta Rybczynska wrote:
> > > > >
> > > > > On Fri, Oct 11, 2024 at 1:02 PM Richard Purdie via
> > > > > lists.openembedded.org
> > > > > <richard.purdie=linuxfoundation.org@lists.openembedded.org>
> > > > > wrote:
> > > > > >
> > > > > > I now do have an example of a corrupted database:
> > > > > >
> > > > > > https://valkyrie.yocto.io/pub/shared-failure-data/nvdcve_2-1.db
> > > > > >
> > > > > > I'm not sure why this is happening but the older releases are
> > > > > > all
> > > > > > accessing the DB over NFS which makes me nervous. We're
> > > > > > seeing a
> > > > > > lot of
> > > > > > failures on the new cluster, hopefully this will help isolate
> > > > > > the
> > > > > > problem.
> > > > > >
> > > > >
> > > > >
> > > > > The database isn't completely corrupted, it looks like it is
> > > > > partial,
> > > > > however. What was the error
> > > > > that you got in that build? Any or just strange results? I'm
> > > > > running
> > > > > a verification vs a fresh database.
> > > >
> > > >
> https://valkyrie.yoctoproject.org/#/builders/23/builds/238/steps/14/logs/stdio
> > > >
> > > > Exception: sqlite3.DatabaseError: database disk image is
> > > > malformed
> > > >
> > > >
> > >
> > >
> > > Managed to get the same error with the sqlite command line.
> >
> >
> > I merged this patch as it would rule out a lot of things.
>
> It looks like I didn't do this. I will do so, then we can rule out a
> few things.
>
>
I've analysed the corrupted file a bit. This is somewhat complex as I do
not have a "golden" copy with the
exact same content. However, what I can see:
1.  This is not a partial download, quite new CVEs from 2024 are there (as
from other years)
2. Damaged records come from various years (a theory to check: if they all
have been recently modified)
3. Only part of the database is broken and both NVD and PRODUCTS tables.
You can read various CVEs depending on how you format your SELECT (getting
all cve_ids works fine, for example)

If you suspect there are jobs accessing the file that shouldn't, what about
installing inotify hooks on the file?

As I've never seen such a corruption on my side, I could imagine either an
external job, or an effect of a re-download of the database during tests
(but this should use locks).

Regards,
Marta

[-- Attachment #2: Type: text/html, Size: 4421 bytes --]

Re: [OE-core] [PATCH] cve-check-update-nvd2-native: Incremement DL_DIR database location

[Richard Purdie]

On Mon, 2024-10-14 at 18:15 +0200, Marta Rybczynska wrote:
> I've analysed the corrupted file a bit. This is somewhat complex as I
> do not have a "golden" copy with the
> exact same content. However, what I can see:
> 1.  This is not a partial download, quite new CVEs from 2024 are
> there (as from other years)
> 2. Damaged records come from various years (a theory to check: if
> they all have been recently modified)
> 3. Only part of the database is broken and both NVD and PRODUCTS
> tables. You can read various CVEs depending on how you format your
> SELECT (getting all cve_ids works fine, for example)
> 
> If you suspect there are jobs accessing the file that shouldn't, what
> about installing inotify hooks on the file?
> 
> As I've never seen such a corruption on my side, I could imagine
> either an external job, or an effect of a re-download of the database
> during tests (but this should use locks).

That is all useful data, thanks.

I can add that on the autobuilders, the file is on NFS since DL_DIR is.
Any worker can therefore in theory access it and inotify could be
tricky to setup and monitor correctly in that setup.

In your tests, do you run multiple releases against the database? I'm
wondering if older sqlite versions in one of the older releases may be
triggering this somehow it if tries to update the database?

Cheers,

Richard