[OE-core][PATCH] cve-update-nvd2-native: add workaround for json5 style list

[OE-core][PATCH] cve-update-nvd2-native: add workaround for json5 style list

[Peter Marko]

From: Peter Marko <peter.marko@siemens.com>

NVD responses changed to an invalid json between:
* April 5, 2025 at 3:03:44 AM GMT+2
* April 5, 2025 at 4:19:48 AM GMT+2

The last response is since then in format
{
  "resultsPerPage": 625,
  "startIndex": 288000,
  "totalResults": 288625,
  "format": "NVD_CVE",
  "version": "2.0",
  "timestamp": "2025-04-07T07:17:17.534",
  "vulnerabilities": [
    {...},
    ...
    {...},
  ]
}

Json does not allow trailing , in responses, that is json5 format.
So cve-update-nvd2-native do_Fetch task fails with log backtrace ending:

...
File: '/builds/ccp/meta-siemens/projects/ccp/../../poky/meta/recipes-core/meta/cve-update-nvd2-native.bb', lineno: 234, function: update_db_file
     0230:            if raw_data is None:
     0231:                # We haven't managed to download data
     0232:                return False
     0233:
 *** 0234:            data = json.loads(raw_data)
     0235:
     0236:            index = data["startIndex"]
     0237:            total = data["totalResults"]
     0238:            per_page = data["resultsPerPage"]
...
File: '/usr/lib/python3.11/json/decoder.py', lineno: 355, function: raw_decode
     0351:        """
     0352:        try:
     0353:            obj, end = self.scan_once(s, idx)
     0354:        except StopIteration as err:
 *** 0355:            raise JSONDecodeError("Expecting value", s, err.value) from None
     0356:        return obj, end
Exception: json.decoder.JSONDecodeError: Expecting value: line 1 column 1442633 (char 1442632)
...

There was no announcement about json format of API v2.0 by nvd.
Also this happens only if whole database is queried (database update is
fine, even when multiple pages as queried).
And lastly it's only the cve list, all other lists inside are fine.
So this looks like a bug in NVD 2.0 introduced with some update.

Patch this with simple character deletion for now and let's monitor the
situation and possibly switch to json5 in the future.
Note that there is no native json5 support in python, we'd have to use
one of external libraries for it.

Signed-off-by: Peter Marko <peter.marko@siemens.com>
---
 meta/recipes-core/meta/cve-update-nvd2-native.bb | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/meta/recipes-core/meta/cve-update-nvd2-native.bb b/meta/recipes-core/meta/cve-update-nvd2-native.bb
index b9c18bf6b6..32a14a932b 100644
--- a/meta/recipes-core/meta/cve-update-nvd2-native.bb
+++ b/meta/recipes-core/meta/cve-update-nvd2-native.bb
@@ -229,6 +229,11 @@ def update_db_file(db_tmp_file, d, database_time):
                 # We haven't managed to download data
                 return False
 
+            # hack for json5 style responses
+            if raw_data[-3:] == ',]}':
+                bb.note("Removing trailing ',' from nvd response")
+                raw_data = raw_data[:-3] + ']}'
+
             data = json.loads(raw_data)
 
             index = data["startIndex"]

RE: [OE-core][PATCH] cve-update-nvd2-native: add workaround for json5 style list

[Marko, Peter]

Dear community,

It looks like NVD introduces new bug in their API 2.0 responses every week.
(e.g. last week https://git.openembedded.org/openembedded-core/commit/?id=8ce06538c9cde0f09909a5a2e61ec10b0d35df49)

I know that this is an ugly patch, but I propose it anyway.
We probably don't want to invest large effort in redesigning to json5 without official statement from NVD.

For master this is a minor issue as it has already switched to FKIE as the default source.
But scarthgap/kirkstone this is currently the only source for cve-check feature.
Shall we consider backporting the FKIE to LTS branches?
And meanwhile backport this patch so that cve-check works again?

Peter

> -----Original Message-----
> From: Marko, Peter (FT D EU SK BFS1) <Peter.Marko@siemens.com>
> Sent: Monday, April 7, 2025 11:36
> To: openembedded-core@lists.openembedded.org
> Cc: Marko, Peter (FT D EU SK BFS1) <Peter.Marko@siemens.com>
> Subject: [OE-core][PATCH] cve-update-nvd2-native: add workaround for json5
> style list
> 
> From: Peter Marko <peter.marko@siemens.com>
> 
> NVD responses changed to an invalid json between:
> * April 5, 2025 at 3:03:44 AM GMT+2
> * April 5, 2025 at 4:19:48 AM GMT+2
> 
> The last response is since then in format
> {
>   "resultsPerPage": 625,
>   "startIndex": 288000,
>   "totalResults": 288625,
>   "format": "NVD_CVE",
>   "version": "2.0",
>   "timestamp": "2025-04-07T07:17:17.534",
>   "vulnerabilities": [
>     {...},
>     ...
>     {...},
>   ]
> }
> 
> Json does not allow trailing , in responses, that is json5 format.
> So cve-update-nvd2-native do_Fetch task fails with log backtrace ending:
> 
> ...
> File: '/builds/ccp/meta-siemens/projects/ccp/../../poky/meta/recipes-core/meta/cve-
> update-nvd2-native.bb', lineno: 234, function: update_db_file
>      0230:            if raw_data is None:
>      0231:                # We haven't managed to download data
>      0232:                return False
>      0233:
>  *** 0234:            data = json.loads(raw_data)
>      0235:
>      0236:            index = data["startIndex"]
>      0237:            total = data["totalResults"]
>      0238:            per_page = data["resultsPerPage"]
> ...
> File: '/usr/lib/python3.11/json/decoder.py', lineno: 355, function: raw_decode
>      0351:        """
>      0352:        try:
>      0353:            obj, end = self.scan_once(s, idx)
>      0354:        except StopIteration as err:
>  *** 0355:            raise JSONDecodeError("Expecting value", s, err.value) from
> None
>      0356:        return obj, end
> Exception: json.decoder.JSONDecodeError: Expecting value: line 1 column
> 1442633 (char 1442632)
> ...
> 
> There was no announcement about json format of API v2.0 by nvd.
> Also this happens only if whole database is queried (database update is
> fine, even when multiple pages as queried).
> And lastly it's only the cve list, all other lists inside are fine.
> So this looks like a bug in NVD 2.0 introduced with some update.
> 
> Patch this with simple character deletion for now and let's monitor the
> situation and possibly switch to json5 in the future.
> Note that there is no native json5 support in python, we'd have to use
> one of external libraries for it.
> 
> Signed-off-by: Peter Marko <peter.marko@siemens.com>
> ---
>  meta/recipes-core/meta/cve-update-nvd2-native.bb | 5 +++++
>  1 file changed, 5 insertions(+)
> 
> diff --git a/meta/recipes-core/meta/cve-update-nvd2-native.bb b/meta/recipes-
> core/meta/cve-update-nvd2-native.bb
> index b9c18bf6b6..32a14a932b 100644
> --- a/meta/recipes-core/meta/cve-update-nvd2-native.bb
> +++ b/meta/recipes-core/meta/cve-update-nvd2-native.bb
> @@ -229,6 +229,11 @@ def update_db_file(db_tmp_file, d, database_time):
>                  # We haven't managed to download data
>                  return False
> 
> +            # hack for json5 style responses
> +            if raw_data[-3:] == ',]}':
> +                bb.note("Removing trailing ',' from nvd response")
> +                raw_data = raw_data[:-3] + ']}'
> +
>              data = json.loads(raw_data)
> 
>              index = data["startIndex"]

Re: [OE-core][PATCH] cve-update-nvd2-native: add workaround for json5 style list

[Mark Hatle]

We're definitely seeing the same failures now.  So we would like some sort of a 
solution back to scarthgap at least.

This hack can work, or a backport of the newer code.  For now I'm going to have 
to go with the hack for my own products (thanks for that), but I'd that we get a 
longer term solution for the LTS releases.

--Mark

On 4/7/25 4:45 AM, Peter Marko via lists.openembedded.org wrote:
> Dear community,
> 
> It looks like NVD introduces new bug in their API 2.0 responses every week.
> (e.g. last week https://git.openembedded.org/openembedded-core/commit/?id=8ce06538c9cde0f09909a5a2e61ec10b0d35df49)
> 
> I know that this is an ugly patch, but I propose it anyway.
> We probably don't want to invest large effort in redesigning to json5 without official statement from NVD.
> 
> For master this is a minor issue as it has already switched to FKIE as the default source.
> But scarthgap/kirkstone this is currently the only source for cve-check feature.
> Shall we consider backporting the FKIE to LTS branches?
> And meanwhile backport this patch so that cve-check works again?
> 
> Peter
> 
>> -----Original Message-----
>> From: Marko, Peter (FT D EU SK BFS1) <Peter.Marko@siemens.com>
>> Sent: Monday, April 7, 2025 11:36
>> To: openembedded-core@lists.openembedded.org
>> Cc: Marko, Peter (FT D EU SK BFS1) <Peter.Marko@siemens.com>
>> Subject: [OE-core][PATCH] cve-update-nvd2-native: add workaround for json5
>> style list
>>
>> From: Peter Marko <peter.marko@siemens.com>
>>
>> NVD responses changed to an invalid json between:
>> * April 5, 2025 at 3:03:44 AM GMT+2
>> * April 5, 2025 at 4:19:48 AM GMT+2
>>
>> The last response is since then in format
>> {
>>    "resultsPerPage": 625,
>>    "startIndex": 288000,
>>    "totalResults": 288625,
>>    "format": "NVD_CVE",
>>    "version": "2.0",
>>    "timestamp": "2025-04-07T07:17:17.534",
>>    "vulnerabilities": [
>>      {...},
>>      ...
>>      {...},
>>    ]
>> }
>>
>> Json does not allow trailing , in responses, that is json5 format.
>> So cve-update-nvd2-native do_Fetch task fails with log backtrace ending:
>>
>> ...
>> File: '/builds/ccp/meta-siemens/projects/ccp/../../poky/meta/recipes-core/meta/cve-
>> update-nvd2-native.bb', lineno: 234, function: update_db_file
>>       0230:            if raw_data is None:
>>       0231:                # We haven't managed to download data
>>       0232:                return False
>>       0233:
>>   *** 0234:            data = json.loads(raw_data)
>>       0235:
>>       0236:            index = data["startIndex"]
>>       0237:            total = data["totalResults"]
>>       0238:            per_page = data["resultsPerPage"]
>> ...
>> File: '/usr/lib/python3.11/json/decoder.py', lineno: 355, function: raw_decode
>>       0351:        """
>>       0352:        try:
>>       0353:            obj, end = self.scan_once(s, idx)
>>       0354:        except StopIteration as err:
>>   *** 0355:            raise JSONDecodeError("Expecting value", s, err.value) from
>> None
>>       0356:        return obj, end
>> Exception: json.decoder.JSONDecodeError: Expecting value: line 1 column
>> 1442633 (char 1442632)
>> ...
>>
>> There was no announcement about json format of API v2.0 by nvd.
>> Also this happens only if whole database is queried (database update is
>> fine, even when multiple pages as queried).
>> And lastly it's only the cve list, all other lists inside are fine.
>> So this looks like a bug in NVD 2.0 introduced with some update.
>>
>> Patch this with simple character deletion for now and let's monitor the
>> situation and possibly switch to json5 in the future.
>> Note that there is no native json5 support in python, we'd have to use
>> one of external libraries for it.
>>
>> Signed-off-by: Peter Marko <peter.marko@siemens.com>
>> ---
>>   meta/recipes-core/meta/cve-update-nvd2-native.bb | 5 +++++
>>   1 file changed, 5 insertions(+)
>>
>> diff --git a/meta/recipes-core/meta/cve-update-nvd2-native.bb b/meta/recipes-
>> core/meta/cve-update-nvd2-native.bb
>> index b9c18bf6b6..32a14a932b 100644
>> --- a/meta/recipes-core/meta/cve-update-nvd2-native.bb
>> +++ b/meta/recipes-core/meta/cve-update-nvd2-native.bb
>> @@ -229,6 +229,11 @@ def update_db_file(db_tmp_file, d, database_time):
>>                   # We haven't managed to download data
>>                   return False
>>
>> +            # hack for json5 style responses
>> +            if raw_data[-3:] == ',]}':
>> +                bb.note("Removing trailing ',' from nvd response")
>> +                raw_data = raw_data[:-3] + ']}'
>> +
>>               data = json.loads(raw_data)
>>
>>               index = data["startIndex"]
>>
>>
>> -=-=-=-=-=-=-=-=-=-=-=-
>> Links: You receive all messages sent to this group.
>> View/Reply Online (#214428): https://lists.openembedded.org/g/openembedded-core/message/214428
>> Mute This Topic: https://lists.openembedded.org/mt/112129465/3616948
>> Group Owner: openembedded-core+owner@lists.openembedded.org
>> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [mark.hatle@kernel.crashing.org]
>> -=-=-=-=-=-=-=-=-=-=-=-
>>

RE: [OE-core][PATCH] cve-update-nvd2-native: add workaround for json5 style list

[Marko, Peter]

The patch was just merged to master.
I'll send patches for LTS (kirkstone/scarthgap) in the evening, but maybe Steve can just pick them from master quicker as they should apply cleanly.

As I mention in the patch itself, it's hard to design a stable solution if we don't know the reasons for broken jsons.
For now I'm plugging the holes as they arise (monitored by nightly pipeline).
I'd give it a month or two to see stability of FKIE/NVD1 feeds and then we should revisit backports of these to LTS.

Peter

> -----Original Message-----
> From: Mark Hatle <mark.hatle@kernel.crashing.org>
> Sent: Wednesday, April 9, 2025 18:32
> To: Marko, Peter (FT D EU SK BFS1) <Peter.Marko@siemens.com>;
> richard.purdie@linuxfoundation.org; Marta Rybczynska <rybczynska@gmail.com>;
> Steve Sakoman <steve@sakoman.com>; Ross Burton <ross.burton@arm.com>
> Cc: openembedded-core@lists.openembedded.org
> Subject: Re: [OE-core][PATCH] cve-update-nvd2-native: add workaround for json5
> style list
> 
> We're definitely seeing the same failures now.  So we would like some sort of a
> solution back to scarthgap at least.
> 
> This hack can work, or a backport of the newer code.  For now I'm going to have
> to go with the hack for my own products (thanks for that), but I'd that we get a
> longer term solution for the LTS releases.
> 
> --Mark
> 
> On 4/7/25 4:45 AM, Peter Marko via lists.openembedded.org wrote:
> > Dear community,
> >
> > It looks like NVD introduces new bug in their API 2.0 responses every week.
> > (e.g. last week https://git.openembedded.org/openembedded-
> core/commit/?id=8ce06538c9cde0f09909a5a2e61ec10b0d35df49)
> >
> > I know that this is an ugly patch, but I propose it anyway.
> > We probably don't want to invest large effort in redesigning to json5 without
> official statement from NVD.
> >
> > For master this is a minor issue as it has already switched to FKIE as the default
> source.
> > But scarthgap/kirkstone this is currently the only source for cve-check feature.
> > Shall we consider backporting the FKIE to LTS branches?
> > And meanwhile backport this patch so that cve-check works again?
> >
> > Peter
> >
> >> -----Original Message-----
> >> From: Marko, Peter (FT D EU SK BFS1) <Peter.Marko@siemens.com>
> >> Sent: Monday, April 7, 2025 11:36
> >> To: openembedded-core@lists.openembedded.org
> >> Cc: Marko, Peter (FT D EU SK BFS1) <Peter.Marko@siemens.com>
> >> Subject: [OE-core][PATCH] cve-update-nvd2-native: add workaround for json5
> >> style list
> >>
> >> From: Peter Marko <peter.marko@siemens.com>
> >>
> >> NVD responses changed to an invalid json between:
> >> * April 5, 2025 at 3:03:44 AM GMT+2
> >> * April 5, 2025 at 4:19:48 AM GMT+2
> >>
> >> The last response is since then in format
> >> {
> >>    "resultsPerPage": 625,
> >>    "startIndex": 288000,
> >>    "totalResults": 288625,
> >>    "format": "NVD_CVE",
> >>    "version": "2.0",
> >>    "timestamp": "2025-04-07T07:17:17.534",
> >>    "vulnerabilities": [
> >>      {...},
> >>      ...
> >>      {...},
> >>    ]
> >> }
> >>
> >> Json does not allow trailing , in responses, that is json5 format.
> >> So cve-update-nvd2-native do_Fetch task fails with log backtrace ending:
> >>
> >> ...
> >> File: '/builds/ccp/meta-siemens/projects/ccp/../../poky/meta/recipes-
> core/meta/cve-
> >> update-nvd2-native.bb', lineno: 234, function: update_db_file
> >>       0230:            if raw_data is None:
> >>       0231:                # We haven't managed to download data
> >>       0232:                return False
> >>       0233:
> >>   *** 0234:            data = json.loads(raw_data)
> >>       0235:
> >>       0236:            index = data["startIndex"]
> >>       0237:            total = data["totalResults"]
> >>       0238:            per_page = data["resultsPerPage"]
> >> ...
> >> File: '/usr/lib/python3.11/json/decoder.py', lineno: 355, function: raw_decode
> >>       0351:        """
> >>       0352:        try:
> >>       0353:            obj, end = self.scan_once(s, idx)
> >>       0354:        except StopIteration as err:
> >>   *** 0355:            raise JSONDecodeError("Expecting value", s, err.value) from
> >> None
> >>       0356:        return obj, end
> >> Exception: json.decoder.JSONDecodeError: Expecting value: line 1 column
> >> 1442633 (char 1442632)
> >> ...
> >>
> >> There was no announcement about json format of API v2.0 by nvd.
> >> Also this happens only if whole database is queried (database update is
> >> fine, even when multiple pages as queried).
> >> And lastly it's only the cve list, all other lists inside are fine.
> >> So this looks like a bug in NVD 2.0 introduced with some update.
> >>
> >> Patch this with simple character deletion for now and let's monitor the
> >> situation and possibly switch to json5 in the future.
> >> Note that there is no native json5 support in python, we'd have to use
> >> one of external libraries for it.
> >>
> >> Signed-off-by: Peter Marko <peter.marko@siemens.com>
> >> ---
> >>   meta/recipes-core/meta/cve-update-nvd2-native.bb | 5 +++++
> >>   1 file changed, 5 insertions(+)
> >>
> >> diff --git a/meta/recipes-core/meta/cve-update-nvd2-native.bb b/meta/recipes-
> >> core/meta/cve-update-nvd2-native.bb
> >> index b9c18bf6b6..32a14a932b 100644
> >> --- a/meta/recipes-core/meta/cve-update-nvd2-native.bb
> >> +++ b/meta/recipes-core/meta/cve-update-nvd2-native.bb
> >> @@ -229,6 +229,11 @@ def update_db_file(db_tmp_file, d, database_time):
> >>                   # We haven't managed to download data
> >>                   return False
> >>
> >> +            # hack for json5 style responses
> >> +            if raw_data[-3:] == ',]}':
> >> +                bb.note("Removing trailing ',' from nvd response")
> >> +                raw_data = raw_data[:-3] + ']}'
> >> +
> >>               data = json.loads(raw_data)
> >>
> >>               index = data["startIndex"]
> >>
> >>
> >> -=-=-=-=-=-=-=-=-=-=-=-
> >> Links: You receive all messages sent to this group.
> >> View/Reply Online (#214428):
> https://lists.openembedded.org/g/openembedded-core/message/214428
> >> Mute This Topic: https://lists.openembedded.org/mt/112129465/3616948
> >> Group Owner: openembedded-core+owner@lists.openembedded.org
> >> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
> [mark.hatle@kernel.crashing.org]
> >> -=-=-=-=-=-=-=-=-=-=-=-
> >>

Re: [OE-core][PATCH] cve-update-nvd2-native: add workaround for json5 style list

[Richard Purdie]

On Thu, 2025-04-10 at 10:43 +0000, Marko, Peter wrote:
> The patch was just merged to master.
> I'll send patches for LTS (kirkstone/scarthgap) in the evening, but
> maybe Steve can just pick them from master quicker as they should
> apply cleanly.
> 
> As I mention in the patch itself, it's hard to design a stable
> solution if we don't know the reasons for broken jsons.
> For now I'm plugging the holes as they arise (monitored by nightly
> pipeline).
> I'd give it a month or two to see stability of FKIE/NVD1 feeds and
> then we should revisit backports of these to LTS.

FWIW this seems like the best plan to me. Things are very much in flux
and seem likely to change. I'll merge it to walnascar in the meantime.

Cheers,

Richard