Re: [Buildroot] [PATCH] support/scripts/cve: fix running on older ijson versions

[Peter Korsgaard <peter@korsgaard.com>]

>>>>> "Yann" == Yann E MORIN <yann.morin.1998@free.fr> writes:

 > Commit 22b69455526f (support/scripts/cve.py: switch from NVD to FKIE for
 > the JSON files) had to change the decompressor from gz to xz, as the new
 > location is using xz compression.

 > That commit mentioned that it was spawning an external xz process to do
 > the decompression, on the pretence that "there is no xz decompressor in
 > Python stdlib."

 > ijson started to accept bytes() (and str()) only with version 3.1, and
 > using a subprocess means we are now passing bytes() to ijson, which it
 > is not expecting as input on such older versions, casuing build failures
 > such as:

 >     [...]
 >       File "/usr/lib/python3/dist-packages/ijson/backends/python.py", line 25, in Lexer
 >         if type(f.read(0)) == bytetype:
 >     AttributeError: 'bytes' object has no attribute 'read'

 > Ubuntu 20.04, on which the pkg-stats run to generate the daily report,
 > only has ijson 2.3. More recent distros have more recent versions of
 > ijson, like Fedora 39 that has 3.2.3, recent enough to supoprt being fed
 > bytes().

 > However, the reasonining in 22b69455526f is wrong: there *is* the lzma
 > module, at least since python 3.3, that is, aeons ago, which is able to
 > read xz-compressed files; it also has an API similar to the gzip module,
 > and can provide a file-like object that exposes the decompressed data.

 > So, do just that: provide an lzma-wrapped file-like object to ijson, so
 > that we can eventually recover our daily reports that everything is
 > broken! :-]

 > Note that this construct still works on recent versions!

 > Reported-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
 > Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
 > Cc: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
 > ---
 >  support/scripts/cve.py | 5 ++---
 >  1 file changed, 2 insertions(+), 3 deletions(-)

 > diff --git a/support/scripts/cve.py b/support/scripts/cve.py
 > index 1a3c307e12..7167ecbc6a 100755
 > --- a/support/scripts/cve.py
 > +++ b/support/scripts/cve.py
 > @@ -21,8 +21,8 @@ import datetime
 >  import os
 >  import requests  # URL checking
 >  import distutils.version
 > +import lzma
 >  import time
 > -import subprocess
 >  import sys
 >  import operator
 
 > @@ -134,8 +134,7 @@ class CVE:
 >          for year in range(NVD_START_YEAR, datetime.datetime.now().year + 1):
 >              filename = CVE.download_nvd_year(nvd_dir, year)
 >              try:
 > -                uncompressed = subprocess.check_output(["xz", "-d", "-c", filename])
 > -                content = ijson.items(uncompressed, 'cve_items.item')
 > +                content = ijson.items(lzma.LZMAFile(filename), 'cve_items.item')

Are you sure this provides str()?

xz GPL-2.0
python3
Python 3.11.2 (main, Mar 13 2023, 12:18:29) [GCC 12.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import lzma
>>> lzma.LZMAFile('GPL-2.0.xz').read(100)
b'Valid-License-Identifier: GPL-2.0\nValid-License-Identifier: GPL-2.0-only\nValid-License-Identifier: G'


Whereas lzma.open() accepts a 'rt' mode:

>>> lzma.open('GPL-2.0.xz', mode='rt').read(100)
'Valid-License-Identifier: GPL-2.0\nValid-License-Identifier: GPL-2.0-only\nValid-License-Identifier: G'

-- 
Bye, Peter Korsgaard
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot