RE: [OE-core][PATCH] cve-update-nvd2-native: add workaround for json5 style list

["Marko, Peter" <Peter.Marko@siemens.com>]

The patch was just merged to master.
I'll send patches for LTS (kirkstone/scarthgap) in the evening, but maybe Steve can just pick them from master quicker as they should apply cleanly.

As I mention in the patch itself, it's hard to design a stable solution if we don't know the reasons for broken jsons.
For now I'm plugging the holes as they arise (monitored by nightly pipeline).
I'd give it a month or two to see stability of FKIE/NVD1 feeds and then we should revisit backports of these to LTS.

Peter

> -----Original Message-----
> From: Mark Hatle <mark.hatle@kernel.crashing.org>
> Sent: Wednesday, April 9, 2025 18:32
> To: Marko, Peter (FT D EU SK BFS1) <Peter.Marko@siemens.com>;
> richard.purdie@linuxfoundation.org; Marta Rybczynska <rybczynska@gmail.com>;
> Steve Sakoman <steve@sakoman.com>; Ross Burton <ross.burton@arm.com>
> Cc: openembedded-core@lists.openembedded.org
> Subject: Re: [OE-core][PATCH] cve-update-nvd2-native: add workaround for json5
> style list
> 
> We're definitely seeing the same failures now.  So we would like some sort of a
> solution back to scarthgap at least.
> 
> This hack can work, or a backport of the newer code.  For now I'm going to have
> to go with the hack for my own products (thanks for that), but I'd that we get a
> longer term solution for the LTS releases.
> 
> --Mark
> 
> On 4/7/25 4:45 AM, Peter Marko via lists.openembedded.org wrote:
> > Dear community,
> >
> > It looks like NVD introduces new bug in their API 2.0 responses every week.
> > (e.g. last week https://git.openembedded.org/openembedded-
> core/commit/?id=8ce06538c9cde0f09909a5a2e61ec10b0d35df49)
> >
> > I know that this is an ugly patch, but I propose it anyway.
> > We probably don't want to invest large effort in redesigning to json5 without
> official statement from NVD.
> >
> > For master this is a minor issue as it has already switched to FKIE as the default
> source.
> > But scarthgap/kirkstone this is currently the only source for cve-check feature.
> > Shall we consider backporting the FKIE to LTS branches?
> > And meanwhile backport this patch so that cve-check works again?
> >
> > Peter
> >
> >> -----Original Message-----
> >> From: Marko, Peter (FT D EU SK BFS1) <Peter.Marko@siemens.com>
> >> Sent: Monday, April 7, 2025 11:36
> >> To: openembedded-core@lists.openembedded.org
> >> Cc: Marko, Peter (FT D EU SK BFS1) <Peter.Marko@siemens.com>
> >> Subject: [OE-core][PATCH] cve-update-nvd2-native: add workaround for json5
> >> style list
> >>
> >> From: Peter Marko <peter.marko@siemens.com>
> >>
> >> NVD responses changed to an invalid json between:
> >> * April 5, 2025 at 3:03:44 AM GMT+2
> >> * April 5, 2025 at 4:19:48 AM GMT+2
> >>
> >> The last response is since then in format
> >> {
> >>    "resultsPerPage": 625,
> >>    "startIndex": 288000,
> >>    "totalResults": 288625,
> >>    "format": "NVD_CVE",
> >>    "version": "2.0",
> >>    "timestamp": "2025-04-07T07:17:17.534",
> >>    "vulnerabilities": [
> >>      {...},
> >>      ...
> >>      {...},
> >>    ]
> >> }
> >>
> >> Json does not allow trailing , in responses, that is json5 format.
> >> So cve-update-nvd2-native do_Fetch task fails with log backtrace ending:
> >>
> >> ...
> >> File: '/builds/ccp/meta-siemens/projects/ccp/../../poky/meta/recipes-
> core/meta/cve-
> >> update-nvd2-native.bb', lineno: 234, function: update_db_file
> >>       0230:            if raw_data is None:
> >>       0231:                # We haven't managed to download data
> >>       0232:                return False
> >>       0233:
> >>   *** 0234:            data = json.loads(raw_data)
> >>       0235:
> >>       0236:            index = data["startIndex"]
> >>       0237:            total = data["totalResults"]
> >>       0238:            per_page = data["resultsPerPage"]
> >> ...
> >> File: '/usr/lib/python3.11/json/decoder.py', lineno: 355, function: raw_decode
> >>       0351:        """
> >>       0352:        try:
> >>       0353:            obj, end = self.scan_once(s, idx)
> >>       0354:        except StopIteration as err:
> >>   *** 0355:            raise JSONDecodeError("Expecting value", s, err.value) from
> >> None
> >>       0356:        return obj, end
> >> Exception: json.decoder.JSONDecodeError: Expecting value: line 1 column
> >> 1442633 (char 1442632)
> >> ...
> >>
> >> There was no announcement about json format of API v2.0 by nvd.
> >> Also this happens only if whole database is queried (database update is
> >> fine, even when multiple pages as queried).
> >> And lastly it's only the cve list, all other lists inside are fine.
> >> So this looks like a bug in NVD 2.0 introduced with some update.
> >>
> >> Patch this with simple character deletion for now and let's monitor the
> >> situation and possibly switch to json5 in the future.
> >> Note that there is no native json5 support in python, we'd have to use
> >> one of external libraries for it.
> >>
> >> Signed-off-by: Peter Marko <peter.marko@siemens.com>
> >> ---
> >>   meta/recipes-core/meta/cve-update-nvd2-native.bb | 5 +++++
> >>   1 file changed, 5 insertions(+)
> >>
> >> diff --git a/meta/recipes-core/meta/cve-update-nvd2-native.bb b/meta/recipes-
> >> core/meta/cve-update-nvd2-native.bb
> >> index b9c18bf6b6..32a14a932b 100644
> >> --- a/meta/recipes-core/meta/cve-update-nvd2-native.bb
> >> +++ b/meta/recipes-core/meta/cve-update-nvd2-native.bb
> >> @@ -229,6 +229,11 @@ def update_db_file(db_tmp_file, d, database_time):
> >>                   # We haven't managed to download data
> >>                   return False
> >>
> >> +            # hack for json5 style responses
> >> +            if raw_data[-3:] == ',]}':
> >> +                bb.note("Removing trailing ',' from nvd response")
> >> +                raw_data = raw_data[:-3] + ']}'
> >> +
> >>               data = json.loads(raw_data)
> >>
> >>               index = data["startIndex"]
> >>
> >>
> >> -=-=-=-=-=-=-=-=-=-=-=-
> >> Links: You receive all messages sent to this group.
> >> View/Reply Online (#214428):
> https://lists.openembedded.org/g/openembedded-core/message/214428
> >> Mute This Topic: https://lists.openembedded.org/mt/112129465/3616948
> >> Group Owner: openembedded-core+owner@lists.openembedded.org
> >> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
> [mark.hatle@kernel.crashing.org]
> >> -=-=-=-=-=-=-=-=-=-=-=-
> >>