[Buildroot] [PATCH] support/scripts/cve: fix running on older ijson versions

[Buildroot] [PATCH] support/scripts/cve: fix running on older ijson versions

[Yann E. MORIN]

Commit 22b69455526f (support/scripts/cve.py: switch from NVD to FKIE for
the JSON files) had to change the decompressor from gz to xz, as the new
location is using xz compression.

That commit mentioned that it was spawning an external xz process to do
the decompression, on the pretence that "there is no xz decompressor in
Python stdlib."

ijson started to accept bytes() (and str()) only with version 3.1, and
using a subprocess means we are now passing bytes() to ijson, which it
is not expecting as input on such older versions, casuing build failures
such as:

    [...]
      File "/usr/lib/python3/dist-packages/ijson/backends/python.py", line 25, in Lexer
        if type(f.read(0)) == bytetype:
    AttributeError: 'bytes' object has no attribute 'read'

Ubuntu 20.04, on which the pkg-stats run to generate the daily report,
only has ijson 2.3. More recent distros have more recent versions of
ijson, like Fedora 39 that has 3.2.3, recent enough to supoprt being fed
bytes().

However, the reasonining in 22b69455526f is wrong: there *is* the lzma
module, at least since python 3.3, that is, aeons ago, which is able to
read xz-compressed files; it also has an API similar to the gzip module,
and can provide a file-like object that exposes the decompressed data.

So, do just that: provide an lzma-wrapped file-like object to ijson, so
that we can eventually recover our daily reports that everything is
broken! :-]

Note that this construct still works on recent versions!

Reported-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
---
 support/scripts/cve.py | 5 ++---
 1 file changed, 2 insertions(+), 3 deletions(-)

diff --git a/support/scripts/cve.py b/support/scripts/cve.py
index 1a3c307e12..7167ecbc6a 100755
--- a/support/scripts/cve.py
+++ b/support/scripts/cve.py
@@ -21,8 +21,8 @@ import datetime
 import os
 import requests  # URL checking
 import distutils.version
+import lzma
 import time
-import subprocess
 import sys
 import operator
 
@@ -134,8 +134,7 @@ class CVE:
         for year in range(NVD_START_YEAR, datetime.datetime.now().year + 1):
             filename = CVE.download_nvd_year(nvd_dir, year)
             try:
-                uncompressed = subprocess.check_output(["xz", "-d", "-c", filename])
-                content = ijson.items(uncompressed, 'cve_items.item')
+                content = ijson.items(lzma.LZMAFile(filename), 'cve_items.item')
             except:  # noqa: E722
                 print("ERROR: cannot read %s. Please remove the file then rerun this script" % filename)
                 raise
-- 
2.43.2

_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH] support/scripts/cve: fix running on older ijson versions

[Peter Korsgaard]

>>>>> "Yann" == Yann E MORIN <yann.morin.1998@free.fr> writes:

 > Commit 22b69455526f (support/scripts/cve.py: switch from NVD to FKIE for
 > the JSON files) had to change the decompressor from gz to xz, as the new
 > location is using xz compression.

 > That commit mentioned that it was spawning an external xz process to do
 > the decompression, on the pretence that "there is no xz decompressor in
 > Python stdlib."

 > ijson started to accept bytes() (and str()) only with version 3.1, and
 > using a subprocess means we are now passing bytes() to ijson, which it
 > is not expecting as input on such older versions, casuing build failures
 > such as:

 >     [...]
 >       File "/usr/lib/python3/dist-packages/ijson/backends/python.py", line 25, in Lexer
 >         if type(f.read(0)) == bytetype:
 >     AttributeError: 'bytes' object has no attribute 'read'

 > Ubuntu 20.04, on which the pkg-stats run to generate the daily report,
 > only has ijson 2.3. More recent distros have more recent versions of
 > ijson, like Fedora 39 that has 3.2.3, recent enough to supoprt being fed
 > bytes().

 > However, the reasonining in 22b69455526f is wrong: there *is* the lzma
 > module, at least since python 3.3, that is, aeons ago, which is able to
 > read xz-compressed files; it also has an API similar to the gzip module,
 > and can provide a file-like object that exposes the decompressed data.

 > So, do just that: provide an lzma-wrapped file-like object to ijson, so
 > that we can eventually recover our daily reports that everything is
 > broken! :-]

 > Note that this construct still works on recent versions!

 > Reported-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
 > Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
 > Cc: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
 > ---
 >  support/scripts/cve.py | 5 ++---
 >  1 file changed, 2 insertions(+), 3 deletions(-)

 > diff --git a/support/scripts/cve.py b/support/scripts/cve.py
 > index 1a3c307e12..7167ecbc6a 100755
 > --- a/support/scripts/cve.py
 > +++ b/support/scripts/cve.py
 > @@ -21,8 +21,8 @@ import datetime
 >  import os
 >  import requests  # URL checking
 >  import distutils.version
 > +import lzma
 >  import time
 > -import subprocess
 >  import sys
 >  import operator
 
 > @@ -134,8 +134,7 @@ class CVE:
 >          for year in range(NVD_START_YEAR, datetime.datetime.now().year + 1):
 >              filename = CVE.download_nvd_year(nvd_dir, year)
 >              try:
 > -                uncompressed = subprocess.check_output(["xz", "-d", "-c", filename])
 > -                content = ijson.items(uncompressed, 'cve_items.item')
 > +                content = ijson.items(lzma.LZMAFile(filename), 'cve_items.item')

Are you sure this provides str()?

xz GPL-2.0
python3
Python 3.11.2 (main, Mar 13 2023, 12:18:29) [GCC 12.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import lzma
>>> lzma.LZMAFile('GPL-2.0.xz').read(100)
b'Valid-License-Identifier: GPL-2.0\nValid-License-Identifier: GPL-2.0-only\nValid-License-Identifier: G'


Whereas lzma.open() accepts a 'rt' mode:

>>> lzma.open('GPL-2.0.xz', mode='rt').read(100)
'Valid-License-Identifier: GPL-2.0\nValid-License-Identifier: GPL-2.0-only\nValid-License-Identifier: G'

-- 
Bye, Peter Korsgaard
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot