From: Peter Marko <peter.marko@siemens.com>
To: openembedded-core@lists.openembedded.org
Cc: Peter Marko <peter.marko@siemens.com>
Subject: [OE-core][PATCH] cve-update-nvd2-native: add workaround for json5 style list
Date: Mon, 7 Apr 2025 11:35:57 +0200 [thread overview]
Message-ID: <20250407093557.50424-1-peter.marko@siemens.com> (raw)
From: Peter Marko <peter.marko@siemens.com>
NVD responses changed to an invalid json between:
* April 5, 2025 at 3:03:44 AM GMT+2
* April 5, 2025 at 4:19:48 AM GMT+2
The last response is since then in format
{
"resultsPerPage": 625,
"startIndex": 288000,
"totalResults": 288625,
"format": "NVD_CVE",
"version": "2.0",
"timestamp": "2025-04-07T07:17:17.534",
"vulnerabilities": [
{...},
...
{...},
]
}
Json does not allow trailing , in responses, that is json5 format.
So cve-update-nvd2-native do_Fetch task fails with log backtrace ending:
...
File: '/builds/ccp/meta-siemens/projects/ccp/../../poky/meta/recipes-core/meta/cve-update-nvd2-native.bb', lineno: 234, function: update_db_file
0230: if raw_data is None:
0231: # We haven't managed to download data
0232: return False
0233:
*** 0234: data = json.loads(raw_data)
0235:
0236: index = data["startIndex"]
0237: total = data["totalResults"]
0238: per_page = data["resultsPerPage"]
...
File: '/usr/lib/python3.11/json/decoder.py', lineno: 355, function: raw_decode
0351: """
0352: try:
0353: obj, end = self.scan_once(s, idx)
0354: except StopIteration as err:
*** 0355: raise JSONDecodeError("Expecting value", s, err.value) from None
0356: return obj, end
Exception: json.decoder.JSONDecodeError: Expecting value: line 1 column 1442633 (char 1442632)
...
There was no announcement about json format of API v2.0 by nvd.
Also this happens only if whole database is queried (database update is
fine, even when multiple pages as queried).
And lastly it's only the cve list, all other lists inside are fine.
So this looks like a bug in NVD 2.0 introduced with some update.
Patch this with simple character deletion for now and let's monitor the
situation and possibly switch to json5 in the future.
Note that there is no native json5 support in python, we'd have to use
one of external libraries for it.
Signed-off-by: Peter Marko <peter.marko@siemens.com>
---
meta/recipes-core/meta/cve-update-nvd2-native.bb | 5 +++++
1 file changed, 5 insertions(+)
diff --git a/meta/recipes-core/meta/cve-update-nvd2-native.bb b/meta/recipes-core/meta/cve-update-nvd2-native.bb
index b9c18bf6b6..32a14a932b 100644
--- a/meta/recipes-core/meta/cve-update-nvd2-native.bb
+++ b/meta/recipes-core/meta/cve-update-nvd2-native.bb
@@ -229,6 +229,11 @@ def update_db_file(db_tmp_file, d, database_time):
# We haven't managed to download data
return False
+ # hack for json5 style responses
+ if raw_data[-3:] == ',]}':
+ bb.note("Removing trailing ',' from nvd response")
+ raw_data = raw_data[:-3] + ']}'
+
data = json.loads(raw_data)
index = data["startIndex"]
next reply other threads:[~2025-04-07 9:37 UTC|newest]
Thread overview: 5+ messages / expand[flat|nested] mbox.gz Atom feed top
2025-04-07 9:35 Peter Marko [this message]
2025-04-07 9:45 ` [OE-core][PATCH] cve-update-nvd2-native: add workaround for json5 style list Marko, Peter
2025-04-09 16:32 ` Mark Hatle
2025-04-10 10:43 ` Marko, Peter
2025-04-10 10:51 ` Richard Purdie
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20250407093557.50424-1-peter.marko@siemens.com \
--to=peter.marko@siemens.com \
--cc=openembedded-core@lists.openembedded.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox