From: Yann E. MORIN <yann.morin.1998@free.fr>
To: buildroot@busybox.net
Subject: [Buildroot] [PATCH] support/scripts/cve.py: use fast ijson backend if available on old ijson versions
Date: Sat, 17 Apr 2021 09:19:59 +0200 [thread overview]
Message-ID: <20210417071959.GP359705@scaer> (raw)
In-Reply-To: <20210409110112.20087-1-peter@korsgaard.com>
Peter, All,
On 2021-04-09 13:01 +0200, Peter Korsgaard spake thusly:
> ijson < 2.5 (as available in Debian 10) use the slow python backend by
> default instead of the most efficient one available like modern ijson
> versions, significantly slowing down cve checking. E.G.:
>
> time ./support/scripts/pkg-stats --nvd-path ~/.nvd -p avahi --html foobar.html
>
> Goes from
> 174,44s user 2,11s system 99% cpu 2:58,04 total
>
> To
> 93,53s user 2,00s system 98% cpu 1:36,65 total
>
> E.G. almost 2x as fast.
>
> As a workaround, detect when the python backend is used and try to use a
> more efficient one instead. Use the yajl2_cffi backend as recommended by
> upstream, as it is most likely to work, and print a warning (and continue)
> if we fail to load it.
>
> The detection is slightly complicated by the fact that ijson.backends used
> to be a reference to a backend module, but is nowadays a string (without the
> ijson.backends prefix).
>
> Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
Applied to master, thanks.
Regards,
Yann E. MORIN.
> ---
> support/scripts/cve.py | 6 ++++++
> 1 file changed, 6 insertions(+)
>
> diff --git a/support/scripts/cve.py b/support/scripts/cve.py
> index 6e97ea193f..965fc2a466 100755
> --- a/support/scripts/cve.py
> +++ b/support/scripts/cve.py
> @@ -28,6 +28,12 @@ import operator
>
> try:
> import ijson
> + # backend is a module in < 2.5, a string in >= 2.5
> + if 'python' in getattr(ijson.backend, '__name__', ijson.backend):
> + try:
> + import ijson.backends.yajl2_cffi as ijson
> + except ImportError:
> + sys.stderr.write('Warning: Using slow ijson python backend\n')
> except ImportError:
> sys.stderr.write("You need ijson to parse NVD for CVE check\n")
> exit(1)
> --
> 2.20.1
>
> _______________________________________________
> buildroot mailing list
> buildroot at busybox.net
> http://lists.busybox.net/mailman/listinfo/buildroot
--
.-----------------.--------------------.------------------.--------------------.
| Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ |
| +33 561 099 427 `------------.-------: X AGAINST | \e/ There is no |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. |
'------------------------------^-------^------------------^--------------------'
next prev parent reply other threads:[~2021-04-17 7:19 UTC|newest]
Thread overview: 3+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-04-09 11:01 [Buildroot] [PATCH] support/scripts/cve.py: use fast ijson backend if available on old ijson versions Peter Korsgaard
2021-04-17 7:19 ` Yann E. MORIN [this message]
2021-04-26 7:09 ` Peter Korsgaard
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20210417071959.GP359705@scaer \
--to=yann.morin.1998@free.fr \
--cc=buildroot@busybox.net \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.