Re: [Buildroot] [PATCH] support/scripts: use FKIE git tree

[Peter Korsgaard <peter@korsgaard.com>]

>>>>> "Yann" == Yann E MORIN <yann.morin.1998@free.fr> writes:

 > Currently, we grab the per-year CVE feeds, in two passes: first, we grab
 > the meta files, and check whether something has changed since last we
 > downloaded it; second, we download the feed proper, unless the meta file
 > has not changed, in which case we use the locally cached feed.

 > However, it has appeared that the FKIE releases no longer provide the
 > meta files, which means that (once again), our daily reports are broken.

 > The obvious fix would be to drop the use of the meta file, and always
 > and unconditionally download the feeds. That's relatively trivial to do,
 > but he feds are relatively big (even as xz-xompressed).

 > However, the CVE database from FKIE is available as a git tree. Git is
 > pretty good as only sending delta when updating a local copy. The git
 > tree, however, contains each CVE as individual files, so it is
 > relatively easier to scan and parse.

 > Switch to using a local git clone.

 > Slightly surprisingly (but not so much either), parsing the CVE files is
 > much faster when using the git working copy, than it is when parsing the
 > per-year feeds: indeed, the per-year feeds are xz-compressed, and even
 > if python is slow-ish to scan a directory and opening files therein, it
 > is still much faster than to decompress xz files. The timing delta [0]
 > is ~100s before and ~10s now, about a ten time improvement, over the
 > whole package set.

 > The drawback, however, is that the git tree is much bigger on-disk, from
 > ~55MiB for the per-year compressed feeds, to 2.1GiB for the git tree
 > (~366MiB) and a working copy (~1.8GiB)... Given very few people are
 > going to use that, that's considered acceptable...

 > Eventually, with a bit of hacking [1], the two pkg-stats, before and
 > after this change, yield the same data (except for the date and commit
 > hash).

 > [0] hacking support/scripts/pkg-stats to display the time before/after
 > the CVE scan, and hacking support/scripts/cve.py to do no download so
 > that only the CVE scan happens (and also because the meta files are no
 > longer available).

 > [1] sorting the CVE lists in json, sorting the json keys, and using the
 > commit from the FKIE git tree that was used for the current per-year
 > feeds.

 > Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
 > Cc: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
 > Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>

Committed to 2024.02.x, thanks.

-- 
Bye, Peter Korsgaard
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot