cve sqlite database 'malformed' issues are back

cve sqlite database 'malformed' issues are back

[Richard Purdie]

Unfortunately the malformed sqlite database issues are back. We had
hoped that moving off NFS would solve this but as we've backported the
changes, things have started to break again:

https://valkyrie.yoctoproject.org/#/builders/76/builds/594/steps/14/logs/stdio

This means we've ruled out a lot of things. It is interesting it is
only starting to happen as we ramped up the older LTS releases.

At this point I'm starting to suspect the older versions of sqlite in
the LTS are somehow not compatible with the newer versions of sqlite in
master :(.

This may be another sign we need to switch to something json based in
DL_DIR.

Cheers,

Richard

Re: cve sqlite database 'malformed' issues are back

[Colin McAllister]

[-- Attachment #1: Type: text/plain, Size: 1896 bytes --]

Hello,

Sorry, I've been away from a computer for the last week and am getting caught back up on cve-check developments.

One of the ideas I've been mulling over since the cve-check discussion in the engineering sync last Tuesday is why there's a local global CVE database to begin with? If the solution is to move to having something JSON-based in DL_DIR, couldn't there just be a smaller database specific to each recipe?

Instead of downloading a whole database, each recipe could run something like "do_cve_fetch" to download the CVE data associated with that recipe's packages. Based on the NIST NVD 2.0 API, this could be done fairly trivially, where the CPE is passed with the API call.

The major downside I see is that each recipe would have to make a NIST API call, which could make builds slower. I also don't see a way to specify the CPE in the CVE change history API, which I believe is leveraged to do incremental updates to the current sqlite database. So I'm not sure how a recipe-specific CVE database could be incrementally updated without just re-downloading all the CVE data. If the CVE change history API were to add the ability to specify a CPE, that would make this much more realistic, I think.

The advantages with this idea is that the CVE data will be more tailored to a specific build, where the whole database isn't needed to be downloaded. I'm guessing that this would reduce disk space, especially compared to the whole JSON dataset being saved in the DL_DIR. If the API were to become faster and more reliable, I think the overall network throughput would be less too.

I wanted to share these ideas with an email thread raising this issue with downloading the NIST database. Once again, I'd be happy to lend some time and what little experience I have to help address this issue, no matter what the best agreed-upon solution is. :)

Thanks,
Colin

[-- Attachment #2: Type: text/html, Size: 1962 bytes --]

Re: [OE-core] cve sqlite database 'malformed' issues are back

[Richard Purdie]

On Wed, 2024-12-18 at 13:22 -0800, Colin McAllister via lists.openembedded.org wrote:
> Sorry, I've been away from a computer for the last week and am
> getting caught back up on cve-check developments.
>
> One of the ideas I've been mulling over since the cve-check
> discussion in the engineering sync last Tuesday is why there's a
> local global CVE database to begin with? If the solution is to move
> to having something JSON-based in DL_DIR, couldn't there just be a
> smaller database specific to each recipe?
>
> Instead of downloading a whole database, each recipe could run
> something like "do_cve_fetch" to download the CVE data associated
> with that recipe's packages. Based on the NIST NVD 2.0 API, this
> could be done fairly trivially, where the CPE is passed with the API
> call.

In theory we could do that, yes. We've been trying to avoid making too
many API calls to NIST though. The current way we capture the data in
one go/recipe and then have the copy there which we can mirror/archive
as needed.

> The major downside I see is that each recipe would have to make a
> NIST API call, which could make builds slower. I also don't see a way
> to specify the CPE in the CVE change history API, which I believe is
> leveraged to do incremental updates to the current sqlite database.
> So I'm not sure how a recipe-specific CVE database could be
> incrementally updated without just re-downloading all the CVE data.

That may be a problem if we can't tell when we need to update or not
:/.

> If the CVE change history API were to add the ability to specify a
> CPE, that would make this much more realistic, I think.
> The advantages with this idea is that the CVE data will be more
> tailored to a specific build, where the whole database isn't needed
> to be downloaded. I'm guessing that this would reduce disk space,
> especially compared to the whole JSON dataset being saved in the
> DL_DIR. If the API were to become faster and more reliable, I think
> the overall network throughput would be less too.
> I wanted to share these ideas with an email thread raising this issue
> with downloading the NIST database. Once again, I'd be happy to lend
> some time and what little experience I have to help address this
> issue, no matter what the best agreed-upon solution is. :)

Just for the record, the malformed database issue has been tracked to
an unfortunate interaction between the way sqlite uses timestamps and
NFS which we use for DL_DIR. There is a simple fix of touching the
database file after we update it which might work around our build
failures. We shall see...

Cheers,

Richard