Re: [Buildroot] [PATCH] support/scripts/cve.py: switch to NVD JSON version 2.0

From: Thomas Petazzoni via buildroot <buildroot@buildroot.org>
To: Daniel Lang <dalang@gmx.at>
Cc: buildroot@buildroot.org
Subject: Re: [Buildroot] [PATCH] support/scripts/cve.py: switch to NVD JSON version 2.0
Date: Tue, 1 Aug 2023 16:19:56 +0200	[thread overview]
Message-ID: <20230801161956.00715a06@windsurf> (raw)
In-Reply-To: <47519c2e-9b64-68b1-79b2-21a2ddea976b@gmx.at>

Hello Daniel,

On Tue, 1 Aug 2023 16:13:03 +0200
Daniel Lang <dalang@gmx.at> wrote:

> > Wow, thanks for working on this! Is the storing of 200k files workable,
> > or do we need to consider some other option like a local sqlite
> > database or something?  
> 
> From testing on my system I can say that it seems to be workable.
> Generating pkg-stats for all packages takes roughly the same time
> 
> old: ./support/scripts/pkg-stats --html old.html --nvd-path dl/buildroot-nvd/ --disable url,upstream,cpe   252,39s user 45,10s system 100% cpu 4:54,85 total
> new: ./support/scripts/pkg-stats --html new.html --nvd-path dl/buildroot-nvd/ --disable url,upstream,cpe   250,04s user 46,24s system 100% cpu 4:53,72 total

Nice!

I see you have --disable cpe. Is the CPE database unchanged on the NVD
side?

> I did consider a sqlite database given that that's the approach yocto uses.
> In the end I decided against it as I wasn't sure how future proof it would be.
> The current approach means that additional information (score, description,...)
> could be added or used for other purposes without having to download again.
> Whereas I thought I had to make a selection for the database.
> In hindsight I could have just added a column for every information available.

I'm not sure if trying to map all fields of the JSON into sqlite fields
would be relevant. In fact, we would only need some kind of key/value
store, where the key is the CVE identifier, and the value is the JSON
blob.

> If there is concern I can see with I have the time to also implement a database
> approach for comparison.

Not sure it's needed for now. The filesystem is also a good database :-)

> Not sure if updating would be faster with a database. It takes ~1.5 seconds
> on my system to save the batch of 2k CVEs to file. But I guess the main bottleneck
> is the API given that the initial download took upwards of 30 minutes during my
> test runs and only ~2.5 minutes are spend creating files.

OK.

> I did. For a 1:1 comparison the sorting on line 185 has to be changed to
> for cve_file in sorted(os.listdir(year_folder)):
> Otherwise CVEs within a package are sorted differently making a comparison
> very hard.
> Running pkg-stats with this change generates identical reports:
> 
> diff old.html new.html
> 57505c57505
> < <p><i>Updated on 2023-08-01 07:34:14.594976, git commit 22e476d7886163484d233803b42a2a4c2b588a5b</i></p>
> ---
> > <p><i>Updated on 2023-08-01 08:40:33.290711, git commit 22e476d7886163484d233803b42a2a4c2b588a5b</i></p>  

Excellent.

> One final note: I'm in no way a python expert, so any optimization or
> general input is welcome.

No problem, I'm also no a python expert at all :-)

Pending some feedback from you on the CPE question above, I think I'm
going to do some quick testing of your proposal and push it.

Thanks!

Thomas
-- 
Thomas Petazzoni, co-owner and CEO, Bootlin
Embedded Linux and Kernel engineering and training
https://bootlin.com
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot