Re: [OE-core] cve sqlite database 'malformed' issues are back

[Richard Purdie <richard.purdie@linuxfoundation.org>]

On Wed, 2024-12-18 at 13:22 -0800, Colin McAllister via lists.openembedded.org wrote:
> Sorry, I've been away from a computer for the last week and am
> getting caught back up on cve-check developments.
>
> One of the ideas I've been mulling over since the cve-check
> discussion in the engineering sync last Tuesday is why there's a
> local global CVE database to begin with? If the solution is to move
> to having something JSON-based in DL_DIR, couldn't there just be a
> smaller database specific to each recipe?
>
> Instead of downloading a whole database, each recipe could run
> something like "do_cve_fetch" to download the CVE data associated
> with that recipe's packages. Based on the NIST NVD 2.0 API, this
> could be done fairly trivially, where the CPE is passed with the API
> call.

In theory we could do that, yes. We've been trying to avoid making too
many API calls to NIST though. The current way we capture the data in
one go/recipe and then have the copy there which we can mirror/archive
as needed.

> The major downside I see is that each recipe would have to make a
> NIST API call, which could make builds slower. I also don't see a way
> to specify the CPE in the CVE change history API, which I believe is
> leveraged to do incremental updates to the current sqlite database.
> So I'm not sure how a recipe-specific CVE database could be
> incrementally updated without just re-downloading all the CVE data.

That may be a problem if we can't tell when we need to update or not
:/.

> If the CVE change history API were to add the ability to specify a
> CPE, that would make this much more realistic, I think.
> The advantages with this idea is that the CVE data will be more
> tailored to a specific build, where the whole database isn't needed
> to be downloaded. I'm guessing that this would reduce disk space,
> especially compared to the whole JSON dataset being saved in the
> DL_DIR. If the API were to become faster and more reliable, I think
> the overall network throughput would be less too.
> I wanted to share these ideas with an email thread raising this issue
> with downloading the NIST database. Once again, I'd be happy to lend
> some time and what little experience I have to help address this
> issue, no matter what the best agreed-upon solution is. :)

Just for the record, the malformed database issue has been tracked to
an unfortunate interaction between the way sqlite uses timestamps and
NFS which we use for DL_DIR. There is a simple fix of touching the
database file after we update it which might work around our build
failures. We shall see...

Cheers,

Richard