From: Peter Korsgaard <peter@korsgaard.com>
To: buildroot@busybox.net
Subject: [Buildroot] [PATCH] support/scripts/cve.py: use fast ijson backend if available on old ijson versions
Date: Fri, 9 Apr 2021 13:01:12 +0200 [thread overview]
Message-ID: <20210409110112.20087-1-peter@korsgaard.com> (raw)
ijson < 2.5 (as available in Debian 10) use the slow python backend by
default instead of the most efficient one available like modern ijson
versions, significantly slowing down cve checking. E.G.:
time ./support/scripts/pkg-stats --nvd-path ~/.nvd -p avahi --html foobar.html
Goes from
174,44s user 2,11s system 99% cpu 2:58,04 total
To
93,53s user 2,00s system 98% cpu 1:36,65 total
E.G. almost 2x as fast.
As a workaround, detect when the python backend is used and try to use a
more efficient one instead. Use the yajl2_cffi backend as recommended by
upstream, as it is most likely to work, and print a warning (and continue)
if we fail to load it.
The detection is slightly complicated by the fact that ijson.backends used
to be a reference to a backend module, but is nowadays a string (without the
ijson.backends prefix).
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
---
support/scripts/cve.py | 6 ++++++
1 file changed, 6 insertions(+)
diff --git a/support/scripts/cve.py b/support/scripts/cve.py
index 6e97ea193f..965fc2a466 100755
--- a/support/scripts/cve.py
+++ b/support/scripts/cve.py
@@ -28,6 +28,12 @@ import operator
try:
import ijson
+ # backend is a module in < 2.5, a string in >= 2.5
+ if 'python' in getattr(ijson.backend, '__name__', ijson.backend):
+ try:
+ import ijson.backends.yajl2_cffi as ijson
+ except ImportError:
+ sys.stderr.write('Warning: Using slow ijson python backend\n')
except ImportError:
sys.stderr.write("You need ijson to parse NVD for CVE check\n")
exit(1)
--
2.20.1
next reply other threads:[~2021-04-09 11:01 UTC|newest]
Thread overview: 3+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-04-09 11:01 Peter Korsgaard [this message]
2021-04-17 7:19 ` [Buildroot] [PATCH] support/scripts/cve.py: use fast ijson backend if available on old ijson versions Yann E. MORIN
2021-04-26 7:09 ` Peter Korsgaard
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20210409110112.20087-1-peter@korsgaard.com \
--to=peter@korsgaard.com \
--cc=buildroot@busybox.net \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.