Re: [OE-core][PATCH] cve-update-nvd2-native: add workaround for json5 style list

[Mark Hatle <mark.hatle@kernel.crashing.org>]

We're definitely seeing the same failures now.  So we would like some sort of a 
solution back to scarthgap at least.

This hack can work, or a backport of the newer code.  For now I'm going to have 
to go with the hack for my own products (thanks for that), but I'd that we get a 
longer term solution for the LTS releases.

--Mark

On 4/7/25 4:45 AM, Peter Marko via lists.openembedded.org wrote:
> Dear community,
> 
> It looks like NVD introduces new bug in their API 2.0 responses every week.
> (e.g. last week https://git.openembedded.org/openembedded-core/commit/?id=8ce06538c9cde0f09909a5a2e61ec10b0d35df49)
> 
> I know that this is an ugly patch, but I propose it anyway.
> We probably don't want to invest large effort in redesigning to json5 without official statement from NVD.
> 
> For master this is a minor issue as it has already switched to FKIE as the default source.
> But scarthgap/kirkstone this is currently the only source for cve-check feature.
> Shall we consider backporting the FKIE to LTS branches?
> And meanwhile backport this patch so that cve-check works again?
> 
> Peter
> 
>> -----Original Message-----
>> From: Marko, Peter (FT D EU SK BFS1) <Peter.Marko@siemens.com>
>> Sent: Monday, April 7, 2025 11:36
>> To: openembedded-core@lists.openembedded.org
>> Cc: Marko, Peter (FT D EU SK BFS1) <Peter.Marko@siemens.com>
>> Subject: [OE-core][PATCH] cve-update-nvd2-native: add workaround for json5
>> style list
>>
>> From: Peter Marko <peter.marko@siemens.com>
>>
>> NVD responses changed to an invalid json between:
>> * April 5, 2025 at 3:03:44 AM GMT+2
>> * April 5, 2025 at 4:19:48 AM GMT+2
>>
>> The last response is since then in format
>> {
>>    "resultsPerPage": 625,
>>    "startIndex": 288000,
>>    "totalResults": 288625,
>>    "format": "NVD_CVE",
>>    "version": "2.0",
>>    "timestamp": "2025-04-07T07:17:17.534",
>>    "vulnerabilities": [
>>      {...},
>>      ...
>>      {...},
>>    ]
>> }
>>
>> Json does not allow trailing , in responses, that is json5 format.
>> So cve-update-nvd2-native do_Fetch task fails with log backtrace ending:
>>
>> ...
>> File: '/builds/ccp/meta-siemens/projects/ccp/../../poky/meta/recipes-core/meta/cve-
>> update-nvd2-native.bb', lineno: 234, function: update_db_file
>>       0230:            if raw_data is None:
>>       0231:                # We haven't managed to download data
>>       0232:                return False
>>       0233:
>>   *** 0234:            data = json.loads(raw_data)
>>       0235:
>>       0236:            index = data["startIndex"]
>>       0237:            total = data["totalResults"]
>>       0238:            per_page = data["resultsPerPage"]
>> ...
>> File: '/usr/lib/python3.11/json/decoder.py', lineno: 355, function: raw_decode
>>       0351:        """
>>       0352:        try:
>>       0353:            obj, end = self.scan_once(s, idx)
>>       0354:        except StopIteration as err:
>>   *** 0355:            raise JSONDecodeError("Expecting value", s, err.value) from
>> None
>>       0356:        return obj, end
>> Exception: json.decoder.JSONDecodeError: Expecting value: line 1 column
>> 1442633 (char 1442632)
>> ...
>>
>> There was no announcement about json format of API v2.0 by nvd.
>> Also this happens only if whole database is queried (database update is
>> fine, even when multiple pages as queried).
>> And lastly it's only the cve list, all other lists inside are fine.
>> So this looks like a bug in NVD 2.0 introduced with some update.
>>
>> Patch this with simple character deletion for now and let's monitor the
>> situation and possibly switch to json5 in the future.
>> Note that there is no native json5 support in python, we'd have to use
>> one of external libraries for it.
>>
>> Signed-off-by: Peter Marko <peter.marko@siemens.com>
>> ---
>>   meta/recipes-core/meta/cve-update-nvd2-native.bb | 5 +++++
>>   1 file changed, 5 insertions(+)
>>
>> diff --git a/meta/recipes-core/meta/cve-update-nvd2-native.bb b/meta/recipes-
>> core/meta/cve-update-nvd2-native.bb
>> index b9c18bf6b6..32a14a932b 100644
>> --- a/meta/recipes-core/meta/cve-update-nvd2-native.bb
>> +++ b/meta/recipes-core/meta/cve-update-nvd2-native.bb
>> @@ -229,6 +229,11 @@ def update_db_file(db_tmp_file, d, database_time):
>>                   # We haven't managed to download data
>>                   return False
>>
>> +            # hack for json5 style responses
>> +            if raw_data[-3:] == ',]}':
>> +                bb.note("Removing trailing ',' from nvd response")
>> +                raw_data = raw_data[:-3] + ']}'
>> +
>>               data = json.loads(raw_data)
>>
>>               index = data["startIndex"]
>>
>>
>> -=-=-=-=-=-=-=-=-=-=-=-
>> Links: You receive all messages sent to this group.
>> View/Reply Online (#214428): https://lists.openembedded.org/g/openembedded-core/message/214428
>> Mute This Topic: https://lists.openembedded.org/mt/112129465/3616948
>> Group Owner: openembedded-core+owner@lists.openembedded.org
>> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [mark.hatle@kernel.crashing.org]
>> -=-=-=-=-=-=-=-=-=-=-=-
>>